[I 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.73: Saving environment info [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: options: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: dataset: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: aux_kw_path: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: batch_sampler: triplet_classif [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: batch_size: 100 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: database: lmdb [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: debug: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: dir: /home/mshukor/data/recipe1m [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: eval_split: val [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: freq_mismatch: 0.0 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: image_from: database [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: import: recipe1m.datasets.factory [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: kw_path: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: name: recipe1m [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: nb_classes: 1048 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: nb_threads: 4 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: randkw_p: 0.3 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: randkw_p_aux: 0.5 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: tokenized_raw_text: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: train_split: train [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: use_vcs: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: engine: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: debug: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: name: logger [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: nb_epochs: 120 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: print_freq: 10 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max'] [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: exp: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: checkpoint: /home/mshukor/logs/recipe1m/tfood_vlp_onlytext/ckpt_last_model.pth.tar [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: resume: None [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: misc: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cuda: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: data_parrallel: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: logs_name: None [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: overrite: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: seed: 1338 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: model: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: criterion: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cross_encoder: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cross_encoder_params: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: tri_cross_encoder: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: itm_loss_weight: 1 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: keep_background: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: retrieval_strategy: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: aggregation: valid [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: get_tokens: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: margin: 0.05 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: margin_params: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: increment: 0.005 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: increment_margin: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: max_margin: 0.3 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: name: triplet [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: nb_samples: 9999 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: sampling: max_negative [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: substrategy: ['IRR', 'RII', 'SIRR', 'SRII'] [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: substrategy_weights: [1.0, 1.0, 0.1, 0.1] [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: import: recipe1m.models.factory [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: metric: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: k_test: 10 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: nb_bags: 10 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: nb_matchs_per_bag: 1000 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: retrieval_dir: /tmp/recipe1m [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: save_ids: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: trijoint: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: network: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: activations: ['tanh', 'normalize'] [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: aux_kwords: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: aux_kwords_encoder: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: avg_concat: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cat_pos: 0 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: checkpoint: None [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: class_attention: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: class_attention_fus: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cls_token: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cls_token_fus: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cross_decoder_image: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cross_decoder_recipe: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cross_encoder: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cross_encoder_params: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: class_attention: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cls_token: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cross_attention: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cross_decoder: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: double_decoder: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: get_tokens: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: load_backbones: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: n_heads: 4 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: n_layers: 4 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: query_img: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: cross_transformer: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: dim_emb: 1024 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: dim_image_out: 2048 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: dim_ingr_out: 300 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: dim_instr_in: 1024 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: dim_instr_out: 1024 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: freeze_im: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: freeze_rec: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: get_tokens: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: get_tokens_cross_decoder_recipe: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: hidden_size: 512 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: image_backbone_name: clip [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: image_res: 224 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: kwords_same_level: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: last_cat_aux: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: n_heads: 4 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: n_heads_cross: 4 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: n_heads_cross_image: 4 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: n_heads_single: 4 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: n_layers: 2 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: n_layers_cross: 2 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: n_layers_cross_image: 1 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: n_layers_single: 2 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: name: trijoint [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: num_hidden_layers_kw: 2 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: old: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: recipe_encoder: tfoodvicha-transformer [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: remove_additional_embedding: ['proj_image'] [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: text_encoder: bert-base-uncased [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: vision_width: 768 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: vit_all_tokens: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: with_ingrs: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: with_instrs: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: with_titles: True [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: with_classif: False [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: optimizer: [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: batch_size_factor: None [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: clip_grad: 8.0 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: import: recipe1m.optimizers.factory [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: lr: 1e-05 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: lr_img: 1e-06 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: name: trijoint_fixed_fine_tune [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: switch_epoch: 20 [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml [S 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.75: view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm'] [I 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64') [I 2022-10-26 09:28:22] ...ytorch/bootstrap/run.py.80: Available GPUs: [0, 1] [I 2022-10-26 09:28:22] ...strap/engines/factory.py.9: Creating engine... [I 2022-10-26 09:28:22] ...trap/datasets/factory.py.8: Creating dataset... [I 2022-10-26 09:28:24] ...m/datasets/recipe1m.py.152: Load VCs... [I 2022-10-26 09:28:28] ...m/datasets/recipe1m.py.165: randkw_p... 0.3 [I 2022-10-26 09:29:54] ...m/datasets/recipe1m.py.370: Calculate indices by class... [I 2022-10-26 09:29:56] ...m/datasets/recipe1m.py.375: Done! [I 2022-10-26 09:29:57] ...m/datasets/recipe1m.py.152: Load VCs... [I 2022-10-26 09:29:58] ...m/datasets/recipe1m.py.165: randkw_p... None [I 2022-10-26 09:31:23] ...rap/datasets/factory.py.18: Training will take place on trainset (238399 items) [I 2022-10-26 09:31:23] ...rap/datasets/factory.py.21: Evaluation will take place on valset (51119 items) [I 2022-10-26 09:31:23] ...strap/models/factory.py.21: Creating model... [I 2022-10-26 09:31:24] ...e_networks/networks.py.203: Loading CLIP Visual model... [I 2022-10-26 09:31:27] ...e_networks/networks.py.145: cuda [I 2022-10-26 09:31:28] ...networks/tfood_vicha.py.49: Load VCs encoder... [I 2022-10-26 09:31:31] .../criterions/trijoint.py.30: ignore_index=0 [I 2022-10-26 09:31:31] ...els/metrics/trijoint.py.46: self.save_ids False [I 2022-10-26 09:31:31] ...els/metrics/trijoint.py.46: self.save_ids False [I 2022-10-26 09:31:31] ...strap/models/factory.py.44: Enabling CUDA mode... [I 2022-10-26 09:31:34] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-26 09:31:34] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-26 09:31:34] ...otstrap/views/factory.py.8: Creating views... [I 2022-10-26 09:31:34] ...trap/engines/engine.py.407: Loading model... /home/mshukor/logs/recipe1m/tfood_vlp_onlytext/ckpt_last_model.pth.tar [I 2022-10-26 09:31:34] ...trap/engines/engine.py.125: Launching training procedures [I 2022-10-26 09:31:34] ...trap/engines/engine.py.163: Training model on trainset for epoch 0 [I 2022-10-26 09:31:34] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 09:31:41] ...trap/engines/engine.py.224: train: epoch 0 | batch 0/2304 [I 2022-10-26 09:31:41] ...trap/engines/engine.py.228: elapsed: 0:00:06 | left: 4:03:01 [I 2022-10-26 09:31:41] ...trap/engines/engine.py.229: process: 6.32892 | load: 2.04562 [I 2022-10-26 09:31:41] ...trap/engines/engine.py.230: loss: 4.84076 [I 2022-10-26 09:31:41] ...optimizers/trijoint.py.151: total_norm: 171.835022 [I 2022-10-26 09:31:46] ...trap/engines/engine.py.224: train: epoch 0 | batch 10/2304 [I 2022-10-26 09:31:46] ...trap/engines/engine.py.228: elapsed: 0:00:11 | left: 0:44:15 [I 2022-10-26 09:31:46] ...trap/engines/engine.py.229: process: 0.51560 | load: 0.00484 [I 2022-10-26 09:31:46] ...trap/engines/engine.py.230: loss: 1.36239 [I 2022-10-26 09:31:46] ...optimizers/trijoint.py.151: total_norm: 8.495289 [I 2022-10-26 09:31:51] ...trap/engines/engine.py.224: train: epoch 0 | batch 20/2304 [I 2022-10-26 09:31:51] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:22:32 [I 2022-10-26 09:31:51] ...trap/engines/engine.py.229: process: 0.52778 | load: 0.00481 [I 2022-10-26 09:31:51] ...trap/engines/engine.py.230: loss: 1.33686 [I 2022-10-26 09:31:51] ...optimizers/trijoint.py.151: total_norm: 4.386299 [I 2022-10-26 09:31:57] ...trap/engines/engine.py.224: train: epoch 0 | batch 30/2304 [I 2022-10-26 09:31:57] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:20:11 [I 2022-10-26 09:31:57] ...trap/engines/engine.py.229: process: 0.50925 | load: 0.00420 [I 2022-10-26 09:31:57] ...trap/engines/engine.py.230: loss: 1.32001 [I 2022-10-26 09:31:57] ...optimizers/trijoint.py.151: total_norm: 0.662660 [I 2022-10-26 09:32:02] ...trap/engines/engine.py.224: train: epoch 0 | batch 40/2304 [I 2022-10-26 09:32:02] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:19:47 [I 2022-10-26 09:32:02] ...trap/engines/engine.py.229: process: 0.52958 | load: 0.00337 [I 2022-10-26 09:32:02] ...trap/engines/engine.py.230: loss: 1.31897 [I 2022-10-26 09:32:02] ...optimizers/trijoint.py.151: total_norm: 0.344177 [I 2022-10-26 09:32:07] ...trap/engines/engine.py.224: train: epoch 0 | batch 50/2304 [I 2022-10-26 09:32:07] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:19:33 [I 2022-10-26 09:32:07] ...trap/engines/engine.py.229: process: 0.52244 | load: 0.00340 [I 2022-10-26 09:32:07] ...trap/engines/engine.py.230: loss: 1.32151 [I 2022-10-26 09:32:07] ...optimizers/trijoint.py.151: total_norm: 0.798061 [I 2022-10-26 09:32:12] ...trap/engines/engine.py.224: train: epoch 0 | batch 60/2304 [I 2022-10-26 09:32:12] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:19:39 [I 2022-10-26 09:32:12] ...trap/engines/engine.py.229: process: 0.53246 | load: 0.00294 [I 2022-10-26 09:32:12] ...trap/engines/engine.py.230: loss: 1.31507 [I 2022-10-26 09:32:12] ...optimizers/trijoint.py.151: total_norm: 0.707173 [I 2022-10-26 09:32:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 70/2304 [I 2022-10-26 09:32:18] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:19:23 [I 2022-10-26 09:32:18] ...trap/engines/engine.py.229: process: 0.51433 | load: 0.00300 [I 2022-10-26 09:32:18] ...trap/engines/engine.py.230: loss: 1.31630 [I 2022-10-26 09:32:18] ...optimizers/trijoint.py.151: total_norm: 0.481085 [I 2022-10-26 09:32:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 80/2304 [I 2022-10-26 09:32:23] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:19:00 [I 2022-10-26 09:32:23] ...trap/engines/engine.py.229: process: 0.51153 | load: 0.00283 [I 2022-10-26 09:32:23] ...trap/engines/engine.py.230: loss: 1.31844 [I 2022-10-26 09:32:23] ...optimizers/trijoint.py.151: total_norm: 0.467728 [I 2022-10-26 09:32:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 90/2304 [I 2022-10-26 09:32:28] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:18:58 [I 2022-10-26 09:32:28] ...trap/engines/engine.py.229: process: 0.51501 | load: 0.00297 [I 2022-10-26 09:32:28] ...trap/engines/engine.py.230: loss: 1.30953 [I 2022-10-26 09:32:28] ...optimizers/trijoint.py.151: total_norm: 1.330758 [I 2022-10-26 09:32:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 100/2304 [I 2022-10-26 09:32:33] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:18:51 [I 2022-10-26 09:32:33] ...trap/engines/engine.py.229: process: 0.50316 | load: 0.00279 [I 2022-10-26 09:32:33] ...trap/engines/engine.py.230: loss: 1.30528 [I 2022-10-26 09:32:33] ...optimizers/trijoint.py.151: total_norm: 1.552766 [I 2022-10-26 09:32:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 110/2304 [I 2022-10-26 09:32:38] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:18:59 [I 2022-10-26 09:32:38] ...trap/engines/engine.py.229: process: 0.51092 | load: 0.00278 [I 2022-10-26 09:32:38] ...trap/engines/engine.py.230: loss: 1.28447 [I 2022-10-26 09:32:38] ...optimizers/trijoint.py.151: total_norm: 3.376291 [I 2022-10-26 09:32:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 120/2304 [I 2022-10-26 09:32:43] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:18:52 [I 2022-10-26 09:32:43] ...trap/engines/engine.py.229: process: 0.51346 | load: 0.00277 [I 2022-10-26 09:32:43] ...trap/engines/engine.py.230: loss: 1.26497 [I 2022-10-26 09:32:43] ...optimizers/trijoint.py.151: total_norm: 7.979314 [I 2022-10-26 09:32:49] ...trap/engines/engine.py.224: train: epoch 0 | batch 130/2304 [I 2022-10-26 09:32:49] ...trap/engines/engine.py.228: elapsed: 0:01:14 | left: 0:18:39 [I 2022-10-26 09:32:49] ...trap/engines/engine.py.229: process: 0.51060 | load: 0.00271 [I 2022-10-26 09:32:49] ...trap/engines/engine.py.230: loss: 1.20543 [I 2022-10-26 09:32:49] ...optimizers/trijoint.py.151: total_norm: 2.920342 [I 2022-10-26 09:32:54] ...trap/engines/engine.py.224: train: epoch 0 | batch 140/2304 [I 2022-10-26 09:32:54] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:18:47 [I 2022-10-26 09:32:54] ...trap/engines/engine.py.229: process: 0.53971 | load: 0.00265 [I 2022-10-26 09:32:54] ...trap/engines/engine.py.230: loss: 1.23201 [I 2022-10-26 09:32:54] ...optimizers/trijoint.py.151: total_norm: 3.995442 [I 2022-10-26 09:32:59] ...trap/engines/engine.py.224: train: epoch 0 | batch 150/2304 [I 2022-10-26 09:32:59] ...trap/engines/engine.py.228: elapsed: 0:01:24 | left: 0:18:38 [I 2022-10-26 09:32:59] ...trap/engines/engine.py.229: process: 0.51868 | load: 0.00274 [I 2022-10-26 09:32:59] ...trap/engines/engine.py.230: loss: 1.18603 [I 2022-10-26 09:32:59] ...optimizers/trijoint.py.151: total_norm: 5.684329 [I 2022-10-26 09:33:04] ...trap/engines/engine.py.224: train: epoch 0 | batch 160/2304 [I 2022-10-26 09:33:04] ...trap/engines/engine.py.228: elapsed: 0:01:30 | left: 0:18:30 [I 2022-10-26 09:33:04] ...trap/engines/engine.py.229: process: 0.51483 | load: 0.00273 [I 2022-10-26 09:33:04] ...trap/engines/engine.py.230: loss: 1.14636 [I 2022-10-26 09:33:04] ...optimizers/trijoint.py.151: total_norm: 7.336391 [I 2022-10-26 09:33:09] ...trap/engines/engine.py.224: train: epoch 0 | batch 170/2304 [I 2022-10-26 09:33:09] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:18:21 [I 2022-10-26 09:33:09] ...trap/engines/engine.py.229: process: 0.51414 | load: 0.00284 [I 2022-10-26 09:33:09] ...trap/engines/engine.py.230: loss: 1.08253 [I 2022-10-26 09:33:09] ...optimizers/trijoint.py.151: total_norm: 5.746813 [I 2022-10-26 09:33:15] ...trap/engines/engine.py.224: train: epoch 0 | batch 180/2304 [I 2022-10-26 09:33:15] ...trap/engines/engine.py.228: elapsed: 0:01:40 | left: 0:18:18 [I 2022-10-26 09:33:15] ...trap/engines/engine.py.229: process: 0.50923 | load: 0.00268 [I 2022-10-26 09:33:15] ...trap/engines/engine.py.230: loss: 1.08314 [I 2022-10-26 09:33:15] ...optimizers/trijoint.py.151: total_norm: 4.448725 [I 2022-10-26 09:33:20] ...trap/engines/engine.py.224: train: epoch 0 | batch 190/2304 [I 2022-10-26 09:33:20] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:18:15 [I 2022-10-26 09:33:20] ...trap/engines/engine.py.229: process: 0.51848 | load: 0.00272 [I 2022-10-26 09:33:20] ...trap/engines/engine.py.230: loss: 1.00779 [I 2022-10-26 09:33:20] ...optimizers/trijoint.py.151: total_norm: 4.116014 [I 2022-10-26 09:33:25] ...trap/engines/engine.py.224: train: epoch 0 | batch 200/2304 [I 2022-10-26 09:33:25] ...trap/engines/engine.py.228: elapsed: 0:01:50 | left: 0:18:14 [I 2022-10-26 09:33:25] ...trap/engines/engine.py.229: process: 0.51918 | load: 0.00265 [I 2022-10-26 09:33:25] ...trap/engines/engine.py.230: loss: 1.00935 [I 2022-10-26 09:33:25] ...optimizers/trijoint.py.151: total_norm: 7.501231 [I 2022-10-26 09:33:30] ...trap/engines/engine.py.224: train: epoch 0 | batch 210/2304 [I 2022-10-26 09:33:30] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:18:04 [I 2022-10-26 09:33:30] ...trap/engines/engine.py.229: process: 0.51181 | load: 0.00268 [I 2022-10-26 09:33:30] ...trap/engines/engine.py.230: loss: 1.04256 [I 2022-10-26 09:33:30] ...optimizers/trijoint.py.151: total_norm: 3.592602 [I 2022-10-26 09:33:35] ...trap/engines/engine.py.224: train: epoch 0 | batch 220/2304 [I 2022-10-26 09:33:35] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:18:01 [I 2022-10-26 09:33:35] ...trap/engines/engine.py.229: process: 0.52112 | load: 0.00267 [I 2022-10-26 09:33:35] ...trap/engines/engine.py.230: loss: 1.07377 [I 2022-10-26 09:33:35] ...optimizers/trijoint.py.151: total_norm: 5.490087 [I 2022-10-26 09:33:41] ...trap/engines/engine.py.224: train: epoch 0 | batch 230/2304 [I 2022-10-26 09:33:41] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:17:53 [I 2022-10-26 09:33:41] ...trap/engines/engine.py.229: process: 0.51128 | load: 0.00265 [I 2022-10-26 09:33:41] ...trap/engines/engine.py.230: loss: 1.06256 [I 2022-10-26 09:33:41] ...optimizers/trijoint.py.151: total_norm: 4.626555 [I 2022-10-26 09:33:46] ...trap/engines/engine.py.224: train: epoch 0 | batch 240/2304 [I 2022-10-26 09:33:46] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:17:49 [I 2022-10-26 09:33:46] ...trap/engines/engine.py.229: process: 0.51890 | load: 0.00268 [I 2022-10-26 09:33:46] ...trap/engines/engine.py.230: loss: 0.96383 [I 2022-10-26 09:33:46] ...optimizers/trijoint.py.151: total_norm: 4.868005 [I 2022-10-26 09:33:51] ...trap/engines/engine.py.224: train: epoch 0 | batch 250/2304 [I 2022-10-26 09:33:51] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:17:52 [I 2022-10-26 09:33:51] ...trap/engines/engine.py.229: process: 0.52726 | load: 0.00272 [I 2022-10-26 09:33:51] ...trap/engines/engine.py.230: loss: 0.95147 [I 2022-10-26 09:33:51] ...optimizers/trijoint.py.151: total_norm: 5.665173 [I 2022-10-26 09:33:56] ...trap/engines/engine.py.224: train: epoch 0 | batch 260/2304 [I 2022-10-26 09:33:56] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:17:41 [I 2022-10-26 09:33:56] ...trap/engines/engine.py.229: process: 0.51745 | load: 0.00272 [I 2022-10-26 09:33:56] ...trap/engines/engine.py.230: loss: 0.95951 [I 2022-10-26 09:33:56] ...optimizers/trijoint.py.151: total_norm: 7.947335 [I 2022-10-26 09:34:01] ...trap/engines/engine.py.224: train: epoch 0 | batch 270/2304 [I 2022-10-26 09:34:01] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:17:30 [I 2022-10-26 09:34:01] ...trap/engines/engine.py.229: process: 0.51670 | load: 0.00253 [I 2022-10-26 09:34:01] ...trap/engines/engine.py.230: loss: 1.02748 [I 2022-10-26 09:34:01] ...optimizers/trijoint.py.151: total_norm: 5.875740 [I 2022-10-26 09:34:07] ...trap/engines/engine.py.224: train: epoch 0 | batch 280/2304 [I 2022-10-26 09:34:07] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:17:24 [I 2022-10-26 09:34:07] ...trap/engines/engine.py.229: process: 0.50901 | load: 0.00273 [I 2022-10-26 09:34:07] ...trap/engines/engine.py.230: loss: 0.90155 [I 2022-10-26 09:34:07] ...optimizers/trijoint.py.151: total_norm: 6.865762 [I 2022-10-26 09:34:12] ...trap/engines/engine.py.224: train: epoch 0 | batch 290/2304 [I 2022-10-26 09:34:12] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:17:09 [I 2022-10-26 09:34:12] ...trap/engines/engine.py.229: process: 0.49973 | load: 0.00260 [I 2022-10-26 09:34:12] ...trap/engines/engine.py.230: loss: 0.94538 [I 2022-10-26 09:34:12] ...optimizers/trijoint.py.151: total_norm: 5.752509 [I 2022-10-26 09:34:17] ...trap/engines/engine.py.224: train: epoch 0 | batch 300/2304 [I 2022-10-26 09:34:17] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:17:27 [I 2022-10-26 09:34:17] ...trap/engines/engine.py.229: process: 0.53194 | load: 0.00267 [I 2022-10-26 09:34:17] ...trap/engines/engine.py.230: loss: 1.05160 [I 2022-10-26 09:34:17] ...optimizers/trijoint.py.151: total_norm: 10.708808 [I 2022-10-26 09:34:22] ...trap/engines/engine.py.224: train: epoch 0 | batch 310/2304 [I 2022-10-26 09:34:22] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:17:07 [I 2022-10-26 09:34:22] ...trap/engines/engine.py.229: process: 0.51036 | load: 0.00255 [I 2022-10-26 09:34:22] ...trap/engines/engine.py.230: loss: 0.93646 [I 2022-10-26 09:34:22] ...optimizers/trijoint.py.151: total_norm: 7.678205 [I 2022-10-26 09:34:27] ...trap/engines/engine.py.224: train: epoch 0 | batch 320/2304 [I 2022-10-26 09:34:27] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:16:51 [I 2022-10-26 09:34:27] ...trap/engines/engine.py.229: process: 0.50845 | load: 0.00279 [I 2022-10-26 09:34:27] ...trap/engines/engine.py.230: loss: 0.93211 [I 2022-10-26 09:34:27] ...optimizers/trijoint.py.151: total_norm: 4.768126 [I 2022-10-26 09:34:32] ...trap/engines/engine.py.224: train: epoch 0 | batch 330/2304 [I 2022-10-26 09:34:32] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:16:59 [I 2022-10-26 09:34:32] ...trap/engines/engine.py.229: process: 0.51837 | load: 0.00259 [I 2022-10-26 09:34:32] ...trap/engines/engine.py.230: loss: 0.83213 [I 2022-10-26 09:34:32] ...optimizers/trijoint.py.151: total_norm: 5.411397 [I 2022-10-26 09:34:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 340/2304 [I 2022-10-26 09:34:38] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:17:04 [I 2022-10-26 09:34:38] ...trap/engines/engine.py.229: process: 0.53689 | load: 0.00255 [I 2022-10-26 09:34:38] ...trap/engines/engine.py.230: loss: 0.82854 [I 2022-10-26 09:34:38] ...optimizers/trijoint.py.151: total_norm: 5.706120 [I 2022-10-26 09:34:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 350/2304 [I 2022-10-26 09:34:43] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:16:49 [I 2022-10-26 09:34:43] ...trap/engines/engine.py.229: process: 0.51822 | load: 0.00259 [I 2022-10-26 09:34:43] ...trap/engines/engine.py.230: loss: 0.81561 [I 2022-10-26 09:34:43] ...optimizers/trijoint.py.151: total_norm: 4.973293 [I 2022-10-26 09:34:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 360/2304 [I 2022-10-26 09:34:48] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:16:41 [I 2022-10-26 09:34:48] ...trap/engines/engine.py.229: process: 0.51090 | load: 0.00257 [I 2022-10-26 09:34:48] ...trap/engines/engine.py.230: loss: 0.87094 [I 2022-10-26 09:34:48] ...optimizers/trijoint.py.151: total_norm: 6.910635 [I 2022-10-26 09:34:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 370/2304 [I 2022-10-26 09:34:53] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:16:52 [I 2022-10-26 09:34:53] ...trap/engines/engine.py.229: process: 0.50565 | load: 0.00263 [I 2022-10-26 09:34:53] ...trap/engines/engine.py.230: loss: 0.72087 [I 2022-10-26 09:34:53] ...optimizers/trijoint.py.151: total_norm: 4.726211 [I 2022-10-26 09:34:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 380/2304 [I 2022-10-26 09:34:58] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:16:37 [I 2022-10-26 09:34:58] ...trap/engines/engine.py.229: process: 0.50217 | load: 0.00259 [I 2022-10-26 09:34:58] ...trap/engines/engine.py.230: loss: 0.88437 [I 2022-10-26 09:34:58] ...optimizers/trijoint.py.151: total_norm: 7.799954 [I 2022-10-26 09:35:04] ...trap/engines/engine.py.224: train: epoch 0 | batch 390/2304 [I 2022-10-26 09:35:04] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:16:42 [I 2022-10-26 09:35:04] ...trap/engines/engine.py.229: process: 0.52503 | load: 0.00266 [I 2022-10-26 09:35:04] ...trap/engines/engine.py.230: loss: 0.82155 [I 2022-10-26 09:35:04] ...optimizers/trijoint.py.151: total_norm: 4.609336 [I 2022-10-26 09:35:09] ...trap/engines/engine.py.224: train: epoch 0 | batch 400/2304 [I 2022-10-26 09:35:09] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:16:34 [I 2022-10-26 09:35:09] ...trap/engines/engine.py.229: process: 0.51184 | load: 0.00355 [I 2022-10-26 09:35:09] ...trap/engines/engine.py.230: loss: 0.83270 [I 2022-10-26 09:35:09] ...optimizers/trijoint.py.151: total_norm: 4.723934 [I 2022-10-26 09:35:14] ...trap/engines/engine.py.224: train: epoch 0 | batch 410/2304 [I 2022-10-26 09:35:14] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:16:25 [I 2022-10-26 09:35:14] ...trap/engines/engine.py.229: process: 0.51593 | load: 0.00307 [I 2022-10-26 09:35:14] ...trap/engines/engine.py.230: loss: 0.87628 [I 2022-10-26 09:35:14] ...optimizers/trijoint.py.151: total_norm: 5.051497 [I 2022-10-26 09:35:19] ...trap/engines/engine.py.224: train: epoch 0 | batch 420/2304 [I 2022-10-26 09:35:19] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:16:29 [I 2022-10-26 09:35:19] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00282 [I 2022-10-26 09:35:19] ...trap/engines/engine.py.230: loss: 0.94234 [I 2022-10-26 09:35:19] ...optimizers/trijoint.py.151: total_norm: 8.613985 [I 2022-10-26 09:35:25] ...trap/engines/engine.py.224: train: epoch 0 | batch 430/2304 [I 2022-10-26 09:35:25] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:16:02 [I 2022-10-26 09:35:25] ...trap/engines/engine.py.229: process: 0.50601 | load: 0.00292 [I 2022-10-26 09:35:25] ...trap/engines/engine.py.230: loss: 0.97225 [I 2022-10-26 09:35:25] ...optimizers/trijoint.py.151: total_norm: 6.305507 [I 2022-10-26 09:35:30] ...trap/engines/engine.py.224: train: epoch 0 | batch 440/2304 [I 2022-10-26 09:35:30] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:15:51 [I 2022-10-26 09:35:30] ...trap/engines/engine.py.229: process: 0.51221 | load: 0.00269 [I 2022-10-26 09:35:30] ...trap/engines/engine.py.230: loss: 0.90044 [I 2022-10-26 09:35:30] ...optimizers/trijoint.py.151: total_norm: 8.718062 [I 2022-10-26 09:35:35] ...trap/engines/engine.py.224: train: epoch 0 | batch 450/2304 [I 2022-10-26 09:35:35] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:15:47 [I 2022-10-26 09:35:35] ...trap/engines/engine.py.229: process: 0.50288 | load: 0.00306 [I 2022-10-26 09:35:35] ...trap/engines/engine.py.230: loss: 0.82629 [I 2022-10-26 09:35:35] ...optimizers/trijoint.py.151: total_norm: 7.310509 [I 2022-10-26 09:35:40] ...trap/engines/engine.py.224: train: epoch 0 | batch 460/2304 [I 2022-10-26 09:35:40] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:15:42 [I 2022-10-26 09:35:40] ...trap/engines/engine.py.229: process: 0.50867 | load: 0.00283 [I 2022-10-26 09:35:40] ...trap/engines/engine.py.230: loss: 0.82309 [I 2022-10-26 09:35:40] ...optimizers/trijoint.py.151: total_norm: 7.898829 [I 2022-10-26 09:35:45] ...trap/engines/engine.py.224: train: epoch 0 | batch 470/2304 [I 2022-10-26 09:35:45] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:15:32 [I 2022-10-26 09:35:45] ...trap/engines/engine.py.229: process: 0.49391 | load: 0.00282 [I 2022-10-26 09:35:45] ...trap/engines/engine.py.230: loss: 0.79472 [I 2022-10-26 09:35:45] ...optimizers/trijoint.py.151: total_norm: 7.570396 [I 2022-10-26 09:35:50] ...trap/engines/engine.py.224: train: epoch 0 | batch 480/2304 [I 2022-10-26 09:35:50] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:15:37 [I 2022-10-26 09:35:50] ...trap/engines/engine.py.229: process: 0.52235 | load: 0.00276 [I 2022-10-26 09:35:50] ...trap/engines/engine.py.230: loss: 0.88713 [I 2022-10-26 09:35:50] ...optimizers/trijoint.py.151: total_norm: 8.781833 [I 2022-10-26 09:35:55] ...trap/engines/engine.py.224: train: epoch 0 | batch 490/2304 [I 2022-10-26 09:35:55] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:15:26 [I 2022-10-26 09:35:55] ...trap/engines/engine.py.229: process: 0.52125 | load: 0.00268 [I 2022-10-26 09:35:55] ...trap/engines/engine.py.230: loss: 0.74268 [I 2022-10-26 09:35:55] ...optimizers/trijoint.py.151: total_norm: 4.513653 [I 2022-10-26 09:36:00] ...trap/engines/engine.py.224: train: epoch 0 | batch 500/2304 [I 2022-10-26 09:36:00] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:15:29 [I 2022-10-26 09:36:00] ...trap/engines/engine.py.229: process: 0.52888 | load: 0.00267 [I 2022-10-26 09:36:00] ...trap/engines/engine.py.230: loss: 0.86511 [I 2022-10-26 09:36:00] ...optimizers/trijoint.py.151: total_norm: 6.719504 [I 2022-10-26 09:36:06] ...trap/engines/engine.py.224: train: epoch 0 | batch 510/2304 [I 2022-10-26 09:36:06] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:15:24 [I 2022-10-26 09:36:06] ...trap/engines/engine.py.229: process: 0.51482 | load: 0.00297 [I 2022-10-26 09:36:06] ...trap/engines/engine.py.230: loss: 0.82820 [I 2022-10-26 09:36:06] ...optimizers/trijoint.py.151: total_norm: 4.999418 [I 2022-10-26 09:36:11] ...trap/engines/engine.py.224: train: epoch 0 | batch 520/2304 [I 2022-10-26 09:36:11] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:15:11 [I 2022-10-26 09:36:11] ...trap/engines/engine.py.229: process: 0.51642 | load: 0.00271 [I 2022-10-26 09:36:11] ...trap/engines/engine.py.230: loss: 0.83943 [I 2022-10-26 09:36:11] ...optimizers/trijoint.py.151: total_norm: 7.308864 [I 2022-10-26 09:36:16] ...trap/engines/engine.py.224: train: epoch 0 | batch 530/2304 [I 2022-10-26 09:36:16] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:14:59 [I 2022-10-26 09:36:16] ...trap/engines/engine.py.229: process: 0.50741 | load: 0.00279 [I 2022-10-26 09:36:16] ...trap/engines/engine.py.230: loss: 0.93837 [I 2022-10-26 09:36:16] ...optimizers/trijoint.py.151: total_norm: 10.310182 [I 2022-10-26 09:36:21] ...trap/engines/engine.py.224: train: epoch 0 | batch 540/2304 [I 2022-10-26 09:36:21] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:15:30 [I 2022-10-26 09:36:21] ...trap/engines/engine.py.229: process: 0.61850 | load: 0.00268 [I 2022-10-26 09:36:21] ...trap/engines/engine.py.230: loss: 0.73230 [I 2022-10-26 09:36:21] ...optimizers/trijoint.py.151: total_norm: 4.887022 [I 2022-10-26 09:36:26] ...trap/engines/engine.py.224: train: epoch 0 | batch 550/2304 [I 2022-10-26 09:36:26] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:14:59 [I 2022-10-26 09:36:26] ...trap/engines/engine.py.229: process: 0.50712 | load: 0.00272 [I 2022-10-26 09:36:26] ...trap/engines/engine.py.230: loss: 0.79706 [I 2022-10-26 09:36:26] ...optimizers/trijoint.py.151: total_norm: 6.169206 [I 2022-10-26 09:36:31] ...trap/engines/engine.py.224: train: epoch 0 | batch 560/2304 [I 2022-10-26 09:36:31] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:14:44 [I 2022-10-26 09:36:31] ...trap/engines/engine.py.229: process: 0.50683 | load: 0.00268 [I 2022-10-26 09:36:31] ...trap/engines/engine.py.230: loss: 0.78708 [I 2022-10-26 09:36:31] ...optimizers/trijoint.py.151: total_norm: 6.602774 [I 2022-10-26 09:36:36] ...trap/engines/engine.py.224: train: epoch 0 | batch 570/2304 [I 2022-10-26 09:36:36] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:14:33 [I 2022-10-26 09:36:36] ...trap/engines/engine.py.229: process: 0.50348 | load: 0.00267 [I 2022-10-26 09:36:36] ...trap/engines/engine.py.230: loss: 0.71873 [I 2022-10-26 09:36:36] ...optimizers/trijoint.py.151: total_norm: 7.217552 [I 2022-10-26 09:36:41] ...trap/engines/engine.py.224: train: epoch 0 | batch 580/2304 [I 2022-10-26 09:36:41] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:14:34 [I 2022-10-26 09:36:41] ...trap/engines/engine.py.229: process: 0.50625 | load: 0.00272 [I 2022-10-26 09:36:41] ...trap/engines/engine.py.230: loss: 0.66658 [I 2022-10-26 09:36:41] ...optimizers/trijoint.py.151: total_norm: 6.000564 [I 2022-10-26 09:36:46] ...trap/engines/engine.py.224: train: epoch 0 | batch 590/2304 [I 2022-10-26 09:36:46] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:14:27 [I 2022-10-26 09:36:46] ...trap/engines/engine.py.229: process: 0.49942 | load: 0.00260 [I 2022-10-26 09:36:46] ...trap/engines/engine.py.230: loss: 0.76433 [I 2022-10-26 09:36:46] ...optimizers/trijoint.py.151: total_norm: 5.948738 [I 2022-10-26 09:36:51] ...trap/engines/engine.py.224: train: epoch 0 | batch 600/2304 [I 2022-10-26 09:36:51] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:14:29 [I 2022-10-26 09:36:51] ...trap/engines/engine.py.229: process: 0.51292 | load: 0.00255 [I 2022-10-26 09:36:51] ...trap/engines/engine.py.230: loss: 0.75868 [I 2022-10-26 09:36:51] ...optimizers/trijoint.py.151: total_norm: 5.645348 [I 2022-10-26 09:36:56] ...trap/engines/engine.py.224: train: epoch 0 | batch 610/2304 [I 2022-10-26 09:36:56] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:14:18 [I 2022-10-26 09:36:56] ...trap/engines/engine.py.229: process: 0.50156 | load: 0.00265 [I 2022-10-26 09:36:56] ...trap/engines/engine.py.230: loss: 0.77036 [I 2022-10-26 09:36:56] ...optimizers/trijoint.py.151: total_norm: 6.108253 [I 2022-10-26 09:37:02] ...trap/engines/engine.py.224: train: epoch 0 | batch 620/2304 [I 2022-10-26 09:37:02] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:14:11 [I 2022-10-26 09:37:02] ...trap/engines/engine.py.229: process: 0.51253 | load: 0.00255 [I 2022-10-26 09:37:02] ...trap/engines/engine.py.230: loss: 0.72004 [I 2022-10-26 09:37:02] ...optimizers/trijoint.py.151: total_norm: 6.541828 [I 2022-10-26 09:37:07] ...trap/engines/engine.py.224: train: epoch 0 | batch 630/2304 [I 2022-10-26 09:37:07] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:14:06 [I 2022-10-26 09:37:07] ...trap/engines/engine.py.229: process: 0.50753 | load: 0.00263 [I 2022-10-26 09:37:07] ...trap/engines/engine.py.230: loss: 0.74080 [I 2022-10-26 09:37:07] ...optimizers/trijoint.py.151: total_norm: 5.662175 [I 2022-10-26 09:37:12] ...trap/engines/engine.py.224: train: epoch 0 | batch 640/2304 [I 2022-10-26 09:37:12] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:14:01 [I 2022-10-26 09:37:12] ...trap/engines/engine.py.229: process: 0.49967 | load: 0.00262 [I 2022-10-26 09:37:12] ...trap/engines/engine.py.230: loss: 0.68835 [I 2022-10-26 09:37:12] ...optimizers/trijoint.py.151: total_norm: 5.095282 [I 2022-10-26 09:37:17] ...trap/engines/engine.py.224: train: epoch 0 | batch 650/2304 [I 2022-10-26 09:37:17] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:13:58 [I 2022-10-26 09:37:17] ...trap/engines/engine.py.229: process: 0.51736 | load: 0.00259 [I 2022-10-26 09:37:17] ...trap/engines/engine.py.230: loss: 0.80816 [I 2022-10-26 09:37:17] ...optimizers/trijoint.py.151: total_norm: 8.498071 [I 2022-10-26 09:37:22] ...trap/engines/engine.py.224: train: epoch 0 | batch 660/2304 [I 2022-10-26 09:37:22] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:14:01 [I 2022-10-26 09:37:22] ...trap/engines/engine.py.229: process: 0.50966 | load: 0.00262 [I 2022-10-26 09:37:22] ...trap/engines/engine.py.230: loss: 0.84986 [I 2022-10-26 09:37:22] ...optimizers/trijoint.py.151: total_norm: 9.266282 [I 2022-10-26 09:37:27] ...trap/engines/engine.py.224: train: epoch 0 | batch 670/2304 [I 2022-10-26 09:37:27] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:13:46 [I 2022-10-26 09:37:27] ...trap/engines/engine.py.229: process: 0.50337 | load: 0.00261 [I 2022-10-26 09:37:27] ...trap/engines/engine.py.230: loss: 0.67614 [I 2022-10-26 09:37:27] ...optimizers/trijoint.py.151: total_norm: 3.967645 [I 2022-10-26 09:37:32] ...trap/engines/engine.py.224: train: epoch 0 | batch 680/2304 [I 2022-10-26 09:37:32] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:13:41 [I 2022-10-26 09:37:32] ...trap/engines/engine.py.229: process: 0.50640 | load: 0.00261 [I 2022-10-26 09:37:32] ...trap/engines/engine.py.230: loss: 0.75823 [I 2022-10-26 09:37:32] ...optimizers/trijoint.py.151: total_norm: 5.577765 [I 2022-10-26 09:37:37] ...trap/engines/engine.py.224: train: epoch 0 | batch 690/2304 [I 2022-10-26 09:37:37] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:13:36 [I 2022-10-26 09:37:37] ...trap/engines/engine.py.229: process: 0.51166 | load: 0.00262 [I 2022-10-26 09:37:37] ...trap/engines/engine.py.230: loss: 0.73611 [I 2022-10-26 09:37:37] ...optimizers/trijoint.py.151: total_norm: 8.299809 [I 2022-10-26 09:37:42] ...trap/engines/engine.py.224: train: epoch 0 | batch 700/2304 [I 2022-10-26 09:37:42] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:13:33 [I 2022-10-26 09:37:42] ...trap/engines/engine.py.229: process: 0.50379 | load: 0.00257 [I 2022-10-26 09:37:42] ...trap/engines/engine.py.230: loss: 0.77352 [I 2022-10-26 09:37:42] ...optimizers/trijoint.py.151: total_norm: 6.194364 [I 2022-10-26 09:37:47] ...trap/engines/engine.py.224: train: epoch 0 | batch 710/2304 [I 2022-10-26 09:37:47] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:13:24 [I 2022-10-26 09:37:47] ...trap/engines/engine.py.229: process: 0.50079 | load: 0.00260 [I 2022-10-26 09:37:47] ...trap/engines/engine.py.230: loss: 0.75674 [I 2022-10-26 09:37:47] ...optimizers/trijoint.py.151: total_norm: 4.299067 [I 2022-10-26 09:37:52] ...trap/engines/engine.py.224: train: epoch 0 | batch 720/2304 [I 2022-10-26 09:37:52] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:13:18 [I 2022-10-26 09:37:52] ...trap/engines/engine.py.229: process: 0.49711 | load: 0.00259 [I 2022-10-26 09:37:52] ...trap/engines/engine.py.230: loss: 0.66251 [I 2022-10-26 09:37:52] ...optimizers/trijoint.py.151: total_norm: 4.962493 [I 2022-10-26 09:37:57] ...trap/engines/engine.py.224: train: epoch 0 | batch 730/2304 [I 2022-10-26 09:37:57] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:13:09 [I 2022-10-26 09:37:57] ...trap/engines/engine.py.229: process: 0.50237 | load: 0.00255 [I 2022-10-26 09:37:57] ...trap/engines/engine.py.230: loss: 0.66034 [I 2022-10-26 09:37:57] ...optimizers/trijoint.py.151: total_norm: 5.652932 [I 2022-10-26 09:38:02] ...trap/engines/engine.py.224: train: epoch 0 | batch 740/2304 [I 2022-10-26 09:38:02] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:13:03 [I 2022-10-26 09:38:02] ...trap/engines/engine.py.229: process: 0.49997 | load: 0.00264 [I 2022-10-26 09:38:02] ...trap/engines/engine.py.230: loss: 0.77513 [I 2022-10-26 09:38:02] ...optimizers/trijoint.py.151: total_norm: 6.361018 [I 2022-10-26 09:38:07] ...trap/engines/engine.py.224: train: epoch 0 | batch 750/2304 [I 2022-10-26 09:38:07] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:13:05 [I 2022-10-26 09:38:07] ...trap/engines/engine.py.229: process: 0.50857 | load: 0.00252 [I 2022-10-26 09:38:07] ...trap/engines/engine.py.230: loss: 0.75280 [I 2022-10-26 09:38:07] ...optimizers/trijoint.py.151: total_norm: 5.259429 [I 2022-10-26 09:38:12] ...trap/engines/engine.py.224: train: epoch 0 | batch 760/2304 [I 2022-10-26 09:38:12] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:12:58 [I 2022-10-26 09:38:12] ...trap/engines/engine.py.229: process: 0.51806 | load: 0.00261 [I 2022-10-26 09:38:12] ...trap/engines/engine.py.230: loss: 0.65736 [I 2022-10-26 09:38:12] ...optimizers/trijoint.py.151: total_norm: 7.503713 [I 2022-10-26 09:38:17] ...trap/engines/engine.py.224: train: epoch 0 | batch 770/2304 [I 2022-10-26 09:38:17] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:12:49 [I 2022-10-26 09:38:17] ...trap/engines/engine.py.229: process: 0.49657 | load: 0.00252 [I 2022-10-26 09:38:17] ...trap/engines/engine.py.230: loss: 0.72407 [I 2022-10-26 09:38:17] ...optimizers/trijoint.py.151: total_norm: 6.118172 [I 2022-10-26 09:38:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 780/2304 [I 2022-10-26 09:38:23] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:12:54 [I 2022-10-26 09:38:23] ...trap/engines/engine.py.229: process: 0.49836 | load: 0.00257 [I 2022-10-26 09:38:23] ...trap/engines/engine.py.230: loss: 0.65544 [I 2022-10-26 09:38:23] ...optimizers/trijoint.py.151: total_norm: 4.683927 [I 2022-10-26 09:38:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 790/2304 [I 2022-10-26 09:38:28] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:12:36 [I 2022-10-26 09:38:28] ...trap/engines/engine.py.229: process: 0.50324 | load: 0.00252 [I 2022-10-26 09:38:28] ...trap/engines/engine.py.230: loss: 0.67112 [I 2022-10-26 09:38:28] ...optimizers/trijoint.py.151: total_norm: 6.845294 [I 2022-10-26 09:38:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 800/2304 [I 2022-10-26 09:38:33] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:12:33 [I 2022-10-26 09:38:33] ...trap/engines/engine.py.229: process: 0.50067 | load: 0.00259 [I 2022-10-26 09:38:33] ...trap/engines/engine.py.230: loss: 0.66221 [I 2022-10-26 09:38:33] ...optimizers/trijoint.py.151: total_norm: 5.052063 [I 2022-10-26 09:38:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 810/2304 [I 2022-10-26 09:38:38] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:12:21 [I 2022-10-26 09:38:38] ...trap/engines/engine.py.229: process: 0.50388 | load: 0.00259 [I 2022-10-26 09:38:38] ...trap/engines/engine.py.230: loss: 0.54071 [I 2022-10-26 09:38:38] ...optimizers/trijoint.py.151: total_norm: 6.276836 [I 2022-10-26 09:38:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 820/2304 [I 2022-10-26 09:38:43] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:12:25 [I 2022-10-26 09:38:43] ...trap/engines/engine.py.229: process: 0.50448 | load: 0.00264 [I 2022-10-26 09:38:43] ...trap/engines/engine.py.230: loss: 0.62636 [I 2022-10-26 09:38:43] ...optimizers/trijoint.py.151: total_norm: 5.376266 [I 2022-10-26 09:38:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 830/2304 [I 2022-10-26 09:38:48] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:12:17 [I 2022-10-26 09:38:48] ...trap/engines/engine.py.229: process: 0.49913 | load: 0.00257 [I 2022-10-26 09:38:48] ...trap/engines/engine.py.230: loss: 0.63758 [I 2022-10-26 09:38:48] ...optimizers/trijoint.py.151: total_norm: 4.471987 [I 2022-10-26 09:38:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 840/2304 [I 2022-10-26 09:38:53] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:12:09 [I 2022-10-26 09:38:53] ...trap/engines/engine.py.229: process: 0.49127 | load: 0.00260 [I 2022-10-26 09:38:53] ...trap/engines/engine.py.230: loss: 0.61794 [I 2022-10-26 09:38:53] ...optimizers/trijoint.py.151: total_norm: 4.960558 [I 2022-10-26 09:38:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 850/2304 [I 2022-10-26 09:38:58] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:12:11 [I 2022-10-26 09:38:58] ...trap/engines/engine.py.229: process: 0.51216 | load: 0.00259 [I 2022-10-26 09:38:58] ...trap/engines/engine.py.230: loss: 0.67488 [I 2022-10-26 09:38:58] ...optimizers/trijoint.py.151: total_norm: 6.536776 [I 2022-10-26 09:39:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 860/2304 [I 2022-10-26 09:39:03] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:12:02 [I 2022-10-26 09:39:03] ...trap/engines/engine.py.229: process: 0.50022 | load: 0.00255 [I 2022-10-26 09:39:03] ...trap/engines/engine.py.230: loss: 0.70083 [I 2022-10-26 09:39:03] ...optimizers/trijoint.py.151: total_norm: 5.220923 [I 2022-10-26 09:39:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 870/2304 [I 2022-10-26 09:39:08] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:11:56 [I 2022-10-26 09:39:08] ...trap/engines/engine.py.229: process: 0.49774 | load: 0.00257 [I 2022-10-26 09:39:08] ...trap/engines/engine.py.230: loss: 0.67800 [I 2022-10-26 09:39:08] ...optimizers/trijoint.py.151: total_norm: 7.685591 [I 2022-10-26 09:39:13] ...trap/engines/engine.py.224: train: epoch 0 | batch 880/2304 [I 2022-10-26 09:39:13] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:11:52 [I 2022-10-26 09:39:13] ...trap/engines/engine.py.229: process: 0.50054 | load: 0.00254 [I 2022-10-26 09:39:13] ...trap/engines/engine.py.230: loss: 0.70639 [I 2022-10-26 09:39:13] ...optimizers/trijoint.py.151: total_norm: 4.821180 [I 2022-10-26 09:39:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 890/2304 [I 2022-10-26 09:39:18] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:11:41 [I 2022-10-26 09:39:18] ...trap/engines/engine.py.229: process: 0.48939 | load: 0.00257 [I 2022-10-26 09:39:18] ...trap/engines/engine.py.230: loss: 0.72486 [I 2022-10-26 09:39:18] ...optimizers/trijoint.py.151: total_norm: 5.774707 [I 2022-10-26 09:39:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 900/2304 [I 2022-10-26 09:39:23] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:11:37 [I 2022-10-26 09:39:23] ...trap/engines/engine.py.229: process: 0.49965 | load: 0.00254 [I 2022-10-26 09:39:23] ...trap/engines/engine.py.230: loss: 0.66210 [I 2022-10-26 09:39:23] ...optimizers/trijoint.py.151: total_norm: 7.078920 [I 2022-10-26 09:39:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 910/2304 [I 2022-10-26 09:39:28] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:11:41 [I 2022-10-26 09:39:28] ...trap/engines/engine.py.229: process: 0.50075 | load: 0.00258 [I 2022-10-26 09:39:28] ...trap/engines/engine.py.230: loss: 0.62208 [I 2022-10-26 09:39:28] ...optimizers/trijoint.py.151: total_norm: 5.937192 [I 2022-10-26 09:39:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 920/2304 [I 2022-10-26 09:39:33] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:11:30 [I 2022-10-26 09:39:33] ...trap/engines/engine.py.229: process: 0.50432 | load: 0.00266 [I 2022-10-26 09:39:33] ...trap/engines/engine.py.230: loss: 0.57640 [I 2022-10-26 09:39:33] ...optimizers/trijoint.py.151: total_norm: 5.947730 [I 2022-10-26 09:39:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 930/2304 [I 2022-10-26 09:39:38] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:11:23 [I 2022-10-26 09:39:38] ...trap/engines/engine.py.229: process: 0.49644 | load: 0.00261 [I 2022-10-26 09:39:38] ...trap/engines/engine.py.230: loss: 0.63270 [I 2022-10-26 09:39:38] ...optimizers/trijoint.py.151: total_norm: 5.170531 [I 2022-10-26 09:39:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 940/2304 [I 2022-10-26 09:39:43] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:11:19 [I 2022-10-26 09:39:43] ...trap/engines/engine.py.229: process: 0.49768 | load: 0.00261 [I 2022-10-26 09:39:43] ...trap/engines/engine.py.230: loss: 0.74149 [I 2022-10-26 09:39:43] ...optimizers/trijoint.py.151: total_norm: 6.800480 [I 2022-10-26 09:39:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 950/2304 [I 2022-10-26 09:39:48] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:11:15 [I 2022-10-26 09:39:48] ...trap/engines/engine.py.229: process: 0.49497 | load: 0.00259 [I 2022-10-26 09:39:48] ...trap/engines/engine.py.230: loss: 0.68126 [I 2022-10-26 09:39:48] ...optimizers/trijoint.py.151: total_norm: 7.160139 [I 2022-10-26 09:39:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 960/2304 [I 2022-10-26 09:39:53] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:11:11 [I 2022-10-26 09:39:53] ...trap/engines/engine.py.229: process: 0.50543 | load: 0.00257 [I 2022-10-26 09:39:53] ...trap/engines/engine.py.230: loss: 0.68199 [I 2022-10-26 09:39:53] ...optimizers/trijoint.py.151: total_norm: 7.503321 [I 2022-10-26 09:39:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 970/2304 [I 2022-10-26 09:39:58] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:11:09 [I 2022-10-26 09:39:58] ...trap/engines/engine.py.229: process: 0.50490 | load: 0.00268 [I 2022-10-26 09:39:58] ...trap/engines/engine.py.230: loss: 0.60942 [I 2022-10-26 09:39:58] ...optimizers/trijoint.py.151: total_norm: 8.291856 [I 2022-10-26 09:40:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 980/2304 [I 2022-10-26 09:40:03] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:11:07 [I 2022-10-26 09:40:03] ...trap/engines/engine.py.229: process: 0.49932 | load: 0.00257 [I 2022-10-26 09:40:03] ...trap/engines/engine.py.230: loss: 0.54604 [I 2022-10-26 09:40:03] ...optimizers/trijoint.py.151: total_norm: 4.073378 [I 2022-10-26 09:40:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 990/2304 [I 2022-10-26 09:40:08] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:10:57 [I 2022-10-26 09:40:08] ...trap/engines/engine.py.229: process: 0.50237 | load: 0.00255 [I 2022-10-26 09:40:08] ...trap/engines/engine.py.230: loss: 0.67236 [I 2022-10-26 09:40:08] ...optimizers/trijoint.py.151: total_norm: 5.854854 [I 2022-10-26 09:40:13] ...trap/engines/engine.py.224: train: epoch 0 | batch 1000/2304 [I 2022-10-26 09:40:13] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:10:51 [I 2022-10-26 09:40:13] ...trap/engines/engine.py.229: process: 0.50028 | load: 0.00252 [I 2022-10-26 09:40:13] ...trap/engines/engine.py.230: loss: 0.62027 [I 2022-10-26 09:40:13] ...optimizers/trijoint.py.151: total_norm: 5.179504 [I 2022-10-26 09:40:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 1010/2304 [I 2022-10-26 09:40:18] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:10:39 [I 2022-10-26 09:40:18] ...trap/engines/engine.py.229: process: 0.48743 | load: 0.00254 [I 2022-10-26 09:40:18] ...trap/engines/engine.py.230: loss: 0.73107 [I 2022-10-26 09:40:18] ...optimizers/trijoint.py.151: total_norm: 5.447103 [I 2022-10-26 09:40:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 1020/2304 [I 2022-10-26 09:40:23] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:11:35 [I 2022-10-26 09:40:23] ...trap/engines/engine.py.229: process: 0.49786 | load: 0.00255 [I 2022-10-26 09:40:23] ...trap/engines/engine.py.230: loss: 0.59666 [I 2022-10-26 09:40:23] ...optimizers/trijoint.py.151: total_norm: 4.755252 [I 2022-10-26 09:40:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 1030/2304 [I 2022-10-26 09:40:28] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:10:40 [I 2022-10-26 09:40:28] ...trap/engines/engine.py.229: process: 0.48661 | load: 0.00273 [I 2022-10-26 09:40:28] ...trap/engines/engine.py.230: loss: 0.64649 [I 2022-10-26 09:40:28] ...optimizers/trijoint.py.151: total_norm: 6.408297 [I 2022-10-26 09:40:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 1040/2304 [I 2022-10-26 09:40:33] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:10:28 [I 2022-10-26 09:40:33] ...trap/engines/engine.py.229: process: 0.49798 | load: 0.00262 [I 2022-10-26 09:40:33] ...trap/engines/engine.py.230: loss: 0.61656 [I 2022-10-26 09:40:33] ...optimizers/trijoint.py.151: total_norm: 6.641245 [I 2022-10-26 09:40:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 1050/2304 [I 2022-10-26 09:40:38] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:10:26 [I 2022-10-26 09:40:38] ...trap/engines/engine.py.229: process: 0.50328 | load: 0.00257 [I 2022-10-26 09:40:38] ...trap/engines/engine.py.230: loss: 0.62330 [I 2022-10-26 09:40:38] ...optimizers/trijoint.py.151: total_norm: 4.900387 [I 2022-10-26 09:40:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 1060/2304 [I 2022-10-26 09:40:43] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:10:25 [I 2022-10-26 09:40:43] ...trap/engines/engine.py.229: process: 0.50310 | load: 0.00260 [I 2022-10-26 09:40:43] ...trap/engines/engine.py.230: loss: 0.62969 [I 2022-10-26 09:40:43] ...optimizers/trijoint.py.151: total_norm: 5.572310 [I 2022-10-26 09:40:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 1070/2304 [I 2022-10-26 09:40:48] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:10:17 [I 2022-10-26 09:40:48] ...trap/engines/engine.py.229: process: 0.49485 | load: 0.00255 [I 2022-10-26 09:40:48] ...trap/engines/engine.py.230: loss: 0.55801 [I 2022-10-26 09:40:48] ...optimizers/trijoint.py.151: total_norm: 5.772641 [I 2022-10-26 09:40:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 1080/2304 [I 2022-10-26 09:40:53] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:10:18 [I 2022-10-26 09:40:53] ...trap/engines/engine.py.229: process: 0.50622 | load: 0.00253 [I 2022-10-26 09:40:53] ...trap/engines/engine.py.230: loss: 0.72214 [I 2022-10-26 09:40:53] ...optimizers/trijoint.py.151: total_norm: 7.184116 [I 2022-10-26 09:40:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 1090/2304 [I 2022-10-26 09:40:58] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:10:07 [I 2022-10-26 09:40:58] ...trap/engines/engine.py.229: process: 0.49945 | load: 0.00257 [I 2022-10-26 09:40:58] ...trap/engines/engine.py.230: loss: 0.63168 [I 2022-10-26 09:40:58] ...optimizers/trijoint.py.151: total_norm: 6.263043 [I 2022-10-26 09:41:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 1100/2304 [I 2022-10-26 09:41:03] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:10:00 [I 2022-10-26 09:41:03] ...trap/engines/engine.py.229: process: 0.49726 | load: 0.00254 [I 2022-10-26 09:41:03] ...trap/engines/engine.py.230: loss: 0.60367 [I 2022-10-26 09:41:03] ...optimizers/trijoint.py.151: total_norm: 4.755497 [I 2022-10-26 09:41:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 1110/2304 [I 2022-10-26 09:41:08] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:09:55 [I 2022-10-26 09:41:08] ...trap/engines/engine.py.229: process: 0.49775 | load: 0.00251 [I 2022-10-26 09:41:08] ...trap/engines/engine.py.230: loss: 0.51709 [I 2022-10-26 09:41:08] ...optimizers/trijoint.py.151: total_norm: 4.812383 [I 2022-10-26 09:41:13] ...trap/engines/engine.py.224: train: epoch 0 | batch 1120/2304 [I 2022-10-26 09:41:13] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:09:50 [I 2022-10-26 09:41:13] ...trap/engines/engine.py.229: process: 0.49937 | load: 0.00249 [I 2022-10-26 09:41:13] ...trap/engines/engine.py.230: loss: 0.52106 [I 2022-10-26 09:41:13] ...optimizers/trijoint.py.151: total_norm: 4.172568 [I 2022-10-26 09:41:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 1130/2304 [I 2022-10-26 09:41:18] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:09:47 [I 2022-10-26 09:41:18] ...trap/engines/engine.py.229: process: 0.50484 | load: 0.00253 [I 2022-10-26 09:41:18] ...trap/engines/engine.py.230: loss: 0.49089 [I 2022-10-26 09:41:18] ...optimizers/trijoint.py.151: total_norm: 5.227578 [I 2022-10-26 09:41:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 1140/2304 [I 2022-10-26 09:41:23] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:09:45 [I 2022-10-26 09:41:23] ...trap/engines/engine.py.229: process: 0.50971 | load: 0.00258 [I 2022-10-26 09:41:23] ...trap/engines/engine.py.230: loss: 0.62379 [I 2022-10-26 09:41:23] ...optimizers/trijoint.py.151: total_norm: 7.855978 [I 2022-10-26 09:41:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 1150/2304 [I 2022-10-26 09:41:28] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:09:34 [I 2022-10-26 09:41:28] ...trap/engines/engine.py.229: process: 0.49814 | load: 0.00252 [I 2022-10-26 09:41:28] ...trap/engines/engine.py.230: loss: 0.52340 [I 2022-10-26 09:41:28] ...optimizers/trijoint.py.151: total_norm: 5.025939 [I 2022-10-26 09:41:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 1160/2304 [I 2022-10-26 09:41:33] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:09:30 [I 2022-10-26 09:41:33] ...trap/engines/engine.py.229: process: 0.50492 | load: 0.00261 [I 2022-10-26 09:41:33] ...trap/engines/engine.py.230: loss: 0.66525 [I 2022-10-26 09:41:33] ...optimizers/trijoint.py.151: total_norm: 5.873952 [I 2022-10-26 09:41:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 1170/2304 [I 2022-10-26 09:41:38] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:09:26 [I 2022-10-26 09:41:38] ...trap/engines/engine.py.229: process: 0.50485 | load: 0.00248 [I 2022-10-26 09:41:38] ...trap/engines/engine.py.230: loss: 0.59102 [I 2022-10-26 09:41:38] ...optimizers/trijoint.py.151: total_norm: 4.184215 [I 2022-10-26 09:41:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 1180/2304 [I 2022-10-26 09:41:43] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:09:18 [I 2022-10-26 09:41:43] ...trap/engines/engine.py.229: process: 0.50218 | load: 0.00260 [I 2022-10-26 09:41:43] ...trap/engines/engine.py.230: loss: 0.52951 [I 2022-10-26 09:41:43] ...optimizers/trijoint.py.151: total_norm: 4.404356 [I 2022-10-26 09:41:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 1190/2304 [I 2022-10-26 09:41:48] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:09:11 [I 2022-10-26 09:41:48] ...trap/engines/engine.py.229: process: 0.50293 | load: 0.00248 [I 2022-10-26 09:41:48] ...trap/engines/engine.py.230: loss: 0.64522 [I 2022-10-26 09:41:48] ...optimizers/trijoint.py.151: total_norm: 4.781116 [I 2022-10-26 09:41:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 1200/2304 [I 2022-10-26 09:41:53] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:09:12 [I 2022-10-26 09:41:53] ...trap/engines/engine.py.229: process: 0.49930 | load: 0.00252 [I 2022-10-26 09:41:53] ...trap/engines/engine.py.230: loss: 0.47200 [I 2022-10-26 09:41:53] ...optimizers/trijoint.py.151: total_norm: 4.905628 [I 2022-10-26 09:41:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 1210/2304 [I 2022-10-26 09:41:58] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:09:02 [I 2022-10-26 09:41:58] ...trap/engines/engine.py.229: process: 0.49854 | load: 0.00252 [I 2022-10-26 09:41:58] ...trap/engines/engine.py.230: loss: 0.53754 [I 2022-10-26 09:41:58] ...optimizers/trijoint.py.151: total_norm: 6.170385 [I 2022-10-26 09:42:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 1220/2304 [I 2022-10-26 09:42:03] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:09:02 [I 2022-10-26 09:42:03] ...trap/engines/engine.py.229: process: 0.50506 | load: 0.00270 [I 2022-10-26 09:42:03] ...trap/engines/engine.py.230: loss: 0.56027 [I 2022-10-26 09:42:03] ...optimizers/trijoint.py.151: total_norm: 5.350987 [I 2022-10-26 09:42:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 1230/2304 [I 2022-10-26 09:42:08] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:08:57 [I 2022-10-26 09:42:08] ...trap/engines/engine.py.229: process: 0.50918 | load: 0.00264 [I 2022-10-26 09:42:08] ...trap/engines/engine.py.230: loss: 0.59723 [I 2022-10-26 09:42:08] ...optimizers/trijoint.py.151: total_norm: 4.581063 [I 2022-10-26 09:42:13] ...trap/engines/engine.py.224: train: epoch 0 | batch 1240/2304 [I 2022-10-26 09:42:13] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:08:45 [I 2022-10-26 09:42:13] ...trap/engines/engine.py.229: process: 0.49115 | load: 0.00253 [I 2022-10-26 09:42:13] ...trap/engines/engine.py.230: loss: 0.56997 [I 2022-10-26 09:42:13] ...optimizers/trijoint.py.151: total_norm: 4.895467 [I 2022-10-26 09:42:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 1250/2304 [I 2022-10-26 09:42:18] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:08:47 [I 2022-10-26 09:42:18] ...trap/engines/engine.py.229: process: 0.49778 | load: 0.00255 [I 2022-10-26 09:42:18] ...trap/engines/engine.py.230: loss: 0.62206 [I 2022-10-26 09:42:18] ...optimizers/trijoint.py.151: total_norm: 5.643043 [I 2022-10-26 09:42:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 1260/2304 [I 2022-10-26 09:42:23] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:08:39 [I 2022-10-26 09:42:23] ...trap/engines/engine.py.229: process: 0.49795 | load: 0.00253 [I 2022-10-26 09:42:23] ...trap/engines/engine.py.230: loss: 0.44305 [I 2022-10-26 09:42:23] ...optimizers/trijoint.py.151: total_norm: 3.815744 [I 2022-10-26 09:42:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 1270/2304 [I 2022-10-26 09:42:28] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:08:39 [I 2022-10-26 09:42:28] ...trap/engines/engine.py.229: process: 0.49411 | load: 0.00250 [I 2022-10-26 09:42:28] ...trap/engines/engine.py.230: loss: 0.48483 [I 2022-10-26 09:42:28] ...optimizers/trijoint.py.151: total_norm: 5.522274 [I 2022-10-26 09:42:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 1280/2304 [I 2022-10-26 09:42:33] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:08:32 [I 2022-10-26 09:42:33] ...trap/engines/engine.py.229: process: 0.50056 | load: 0.00255 [I 2022-10-26 09:42:33] ...trap/engines/engine.py.230: loss: 0.54846 [I 2022-10-26 09:42:33] ...optimizers/trijoint.py.151: total_norm: 5.502153 [I 2022-10-26 09:42:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 1290/2304 [I 2022-10-26 09:42:38] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:08:24 [I 2022-10-26 09:42:38] ...trap/engines/engine.py.229: process: 0.48950 | load: 0.00254 [I 2022-10-26 09:42:38] ...trap/engines/engine.py.230: loss: 0.69847 [I 2022-10-26 09:42:38] ...optimizers/trijoint.py.151: total_norm: 5.966784 [I 2022-10-26 09:42:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 1300/2304 [I 2022-10-26 09:42:43] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:08:16 [I 2022-10-26 09:42:43] ...trap/engines/engine.py.229: process: 0.49383 | load: 0.00253 [I 2022-10-26 09:42:43] ...trap/engines/engine.py.230: loss: 0.54695 [I 2022-10-26 09:42:43] ...optimizers/trijoint.py.151: total_norm: 4.887349 [I 2022-10-26 09:42:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 1310/2304 [I 2022-10-26 09:42:48] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:08:12 [I 2022-10-26 09:42:48] ...trap/engines/engine.py.229: process: 0.49775 | load: 0.00256 [I 2022-10-26 09:42:48] ...trap/engines/engine.py.230: loss: 0.62830 [I 2022-10-26 09:42:48] ...optimizers/trijoint.py.151: total_norm: 5.162760 [I 2022-10-26 09:42:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 1320/2304 [I 2022-10-26 09:42:53] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:08:19 [I 2022-10-26 09:42:53] ...trap/engines/engine.py.229: process: 0.49728 | load: 0.00256 [I 2022-10-26 09:42:53] ...trap/engines/engine.py.230: loss: 0.48575 [I 2022-10-26 09:42:53] ...optimizers/trijoint.py.151: total_norm: 4.783540 [I 2022-10-26 09:42:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 1330/2304 [I 2022-10-26 09:42:58] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:08:06 [I 2022-10-26 09:42:58] ...trap/engines/engine.py.229: process: 0.49904 | load: 0.00264 [I 2022-10-26 09:42:58] ...trap/engines/engine.py.230: loss: 0.58755 [I 2022-10-26 09:42:58] ...optimizers/trijoint.py.151: total_norm: 5.106409 [I 2022-10-26 09:43:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 1340/2304 [I 2022-10-26 09:43:03] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:08:01 [I 2022-10-26 09:43:03] ...trap/engines/engine.py.229: process: 0.49928 | load: 0.00262 [I 2022-10-26 09:43:03] ...trap/engines/engine.py.230: loss: 0.58295 [I 2022-10-26 09:43:03] ...optimizers/trijoint.py.151: total_norm: 5.830812 [I 2022-10-26 09:43:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 1350/2304 [I 2022-10-26 09:43:08] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:07:54 [I 2022-10-26 09:43:08] ...trap/engines/engine.py.229: process: 0.50091 | load: 0.00257 [I 2022-10-26 09:43:08] ...trap/engines/engine.py.230: loss: 0.50125 [I 2022-10-26 09:43:08] ...optimizers/trijoint.py.151: total_norm: 4.451577 [I 2022-10-26 09:43:13] ...trap/engines/engine.py.224: train: epoch 0 | batch 1360/2304 [I 2022-10-26 09:43:13] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:08:02 [I 2022-10-26 09:43:13] ...trap/engines/engine.py.229: process: 0.51221 | load: 0.00259 [I 2022-10-26 09:43:13] ...trap/engines/engine.py.230: loss: 0.49509 [I 2022-10-26 09:43:13] ...optimizers/trijoint.py.151: total_norm: 3.900225 [I 2022-10-26 09:43:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 1370/2304 [I 2022-10-26 09:43:18] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:07:50 [I 2022-10-26 09:43:18] ...trap/engines/engine.py.229: process: 0.50086 | load: 0.00258 [I 2022-10-26 09:43:18] ...trap/engines/engine.py.230: loss: 0.54420 [I 2022-10-26 09:43:18] ...optimizers/trijoint.py.151: total_norm: 8.470966 [I 2022-10-26 09:43:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 1380/2304 [I 2022-10-26 09:43:23] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:07:45 [I 2022-10-26 09:43:23] ...trap/engines/engine.py.229: process: 0.50017 | load: 0.00257 [I 2022-10-26 09:43:23] ...trap/engines/engine.py.230: loss: 0.62003 [I 2022-10-26 09:43:23] ...optimizers/trijoint.py.151: total_norm: 4.902229 [I 2022-10-26 09:43:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 1390/2304 [I 2022-10-26 09:43:28] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:07:34 [I 2022-10-26 09:43:28] ...trap/engines/engine.py.229: process: 0.48446 | load: 0.00253 [I 2022-10-26 09:43:28] ...trap/engines/engine.py.230: loss: 0.52337 [I 2022-10-26 09:43:28] ...optimizers/trijoint.py.151: total_norm: 5.116558 [I 2022-10-26 09:43:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 1400/2304 [I 2022-10-26 09:43:33] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:07:29 [I 2022-10-26 09:43:33] ...trap/engines/engine.py.229: process: 0.49553 | load: 0.00267 [I 2022-10-26 09:43:33] ...trap/engines/engine.py.230: loss: 0.46521 [I 2022-10-26 09:43:33] ...optimizers/trijoint.py.151: total_norm: 4.521862 [I 2022-10-26 09:43:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 1410/2304 [I 2022-10-26 09:43:38] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:07:25 [I 2022-10-26 09:43:38] ...trap/engines/engine.py.229: process: 0.50469 | load: 0.00256 [I 2022-10-26 09:43:38] ...trap/engines/engine.py.230: loss: 0.60814 [I 2022-10-26 09:43:38] ...optimizers/trijoint.py.151: total_norm: 4.781235 [I 2022-10-26 09:43:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 1420/2304 [I 2022-10-26 09:43:43] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:07:21 [I 2022-10-26 09:43:43] ...trap/engines/engine.py.229: process: 0.50449 | load: 0.00252 [I 2022-10-26 09:43:43] ...trap/engines/engine.py.230: loss: 0.55976 [I 2022-10-26 09:43:43] ...optimizers/trijoint.py.151: total_norm: 5.576849 [I 2022-10-26 09:43:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 1430/2304 [I 2022-10-26 09:43:48] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:07:11 [I 2022-10-26 09:43:48] ...trap/engines/engine.py.229: process: 0.48911 | load: 0.00254 [I 2022-10-26 09:43:48] ...trap/engines/engine.py.230: loss: 0.59566 [I 2022-10-26 09:43:48] ...optimizers/trijoint.py.151: total_norm: 7.196237 [I 2022-10-26 09:43:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 1440/2304 [I 2022-10-26 09:43:53] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:07:14 [I 2022-10-26 09:43:53] ...trap/engines/engine.py.229: process: 0.49793 | load: 0.00259 [I 2022-10-26 09:43:53] ...trap/engines/engine.py.230: loss: 0.59437 [I 2022-10-26 09:43:53] ...optimizers/trijoint.py.151: total_norm: 5.152997 [I 2022-10-26 09:43:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 1450/2304 [I 2022-10-26 09:43:58] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:07:07 [I 2022-10-26 09:43:58] ...trap/engines/engine.py.229: process: 0.50579 | load: 0.00262 [I 2022-10-26 09:43:58] ...trap/engines/engine.py.230: loss: 0.50723 [I 2022-10-26 09:43:58] ...optimizers/trijoint.py.151: total_norm: 5.621633 [I 2022-10-26 09:44:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 1460/2304 [I 2022-10-26 09:44:03] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:07:00 [I 2022-10-26 09:44:03] ...trap/engines/engine.py.229: process: 0.48331 | load: 0.00250 [I 2022-10-26 09:44:03] ...trap/engines/engine.py.230: loss: 0.57923 [I 2022-10-26 09:44:03] ...optimizers/trijoint.py.151: total_norm: 4.966452 [I 2022-10-26 09:44:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 1470/2304 [I 2022-10-26 09:44:08] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:06:54 [I 2022-10-26 09:44:08] ...trap/engines/engine.py.229: process: 0.49956 | load: 0.00252 [I 2022-10-26 09:44:08] ...trap/engines/engine.py.230: loss: 0.49535 [I 2022-10-26 09:44:08] ...optimizers/trijoint.py.151: total_norm: 4.517986 [I 2022-10-26 09:44:13] ...trap/engines/engine.py.224: train: epoch 0 | batch 1480/2304 [I 2022-10-26 09:44:13] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:06:49 [I 2022-10-26 09:44:13] ...trap/engines/engine.py.229: process: 0.48670 | load: 0.00249 [I 2022-10-26 09:44:13] ...trap/engines/engine.py.230: loss: 0.49098 [I 2022-10-26 09:44:13] ...optimizers/trijoint.py.151: total_norm: 4.157776 [I 2022-10-26 09:44:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 1490/2304 [I 2022-10-26 09:44:18] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:06:43 [I 2022-10-26 09:44:18] ...trap/engines/engine.py.229: process: 0.49787 | load: 0.00263 [I 2022-10-26 09:44:18] ...trap/engines/engine.py.230: loss: 0.44120 [I 2022-10-26 09:44:18] ...optimizers/trijoint.py.151: total_norm: 6.131838 [I 2022-10-26 09:44:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 1500/2304 [I 2022-10-26 09:44:23] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:06:40 [I 2022-10-26 09:44:23] ...trap/engines/engine.py.229: process: 0.49830 | load: 0.00258 [I 2022-10-26 09:44:23] ...trap/engines/engine.py.230: loss: 0.58387 [I 2022-10-26 09:44:23] ...optimizers/trijoint.py.151: total_norm: 5.117280 [I 2022-10-26 09:44:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 1510/2304 [I 2022-10-26 09:44:28] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:06:35 [I 2022-10-26 09:44:28] ...trap/engines/engine.py.229: process: 0.50359 | load: 0.00255 [I 2022-10-26 09:44:28] ...trap/engines/engine.py.230: loss: 0.58484 [I 2022-10-26 09:44:28] ...optimizers/trijoint.py.151: total_norm: 5.700500 [I 2022-10-26 09:44:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 1520/2304 [I 2022-10-26 09:44:33] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:06:32 [I 2022-10-26 09:44:33] ...trap/engines/engine.py.229: process: 0.50365 | load: 0.00256 [I 2022-10-26 09:44:33] ...trap/engines/engine.py.230: loss: 0.59343 [I 2022-10-26 09:44:33] ...optimizers/trijoint.py.151: total_norm: 6.160998 [I 2022-10-26 09:44:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 1530/2304 [I 2022-10-26 09:44:38] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:06:29 [I 2022-10-26 09:44:38] ...trap/engines/engine.py.229: process: 0.51010 | load: 0.00260 [I 2022-10-26 09:44:38] ...trap/engines/engine.py.230: loss: 0.54768 [I 2022-10-26 09:44:38] ...optimizers/trijoint.py.151: total_norm: 4.221391 [I 2022-10-26 09:44:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 1540/2304 [I 2022-10-26 09:44:43] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:06:19 [I 2022-10-26 09:44:43] ...trap/engines/engine.py.229: process: 0.49264 | load: 0.00258 [I 2022-10-26 09:44:43] ...trap/engines/engine.py.230: loss: 0.44429 [I 2022-10-26 09:44:43] ...optimizers/trijoint.py.151: total_norm: 4.278655 [I 2022-10-26 09:44:48] ...trap/engines/engine.py.224: train: epoch 0 | batch 1550/2304 [I 2022-10-26 09:44:48] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:06:15 [I 2022-10-26 09:44:48] ...trap/engines/engine.py.229: process: 0.49484 | load: 0.00252 [I 2022-10-26 09:44:48] ...trap/engines/engine.py.230: loss: 0.61113 [I 2022-10-26 09:44:48] ...optimizers/trijoint.py.151: total_norm: 4.942786 [I 2022-10-26 09:44:53] ...trap/engines/engine.py.224: train: epoch 0 | batch 1560/2304 [I 2022-10-26 09:44:53] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:06:10 [I 2022-10-26 09:44:53] ...trap/engines/engine.py.229: process: 0.49244 | load: 0.00252 [I 2022-10-26 09:44:53] ...trap/engines/engine.py.230: loss: 0.47173 [I 2022-10-26 09:44:53] ...optimizers/trijoint.py.151: total_norm: 4.679360 [I 2022-10-26 09:44:58] ...trap/engines/engine.py.224: train: epoch 0 | batch 1570/2304 [I 2022-10-26 09:44:58] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:06:05 [I 2022-10-26 09:44:58] ...trap/engines/engine.py.229: process: 0.50025 | load: 0.00251 [I 2022-10-26 09:44:58] ...trap/engines/engine.py.230: loss: 0.58633 [I 2022-10-26 09:44:58] ...optimizers/trijoint.py.151: total_norm: 7.130239 [I 2022-10-26 09:45:03] ...trap/engines/engine.py.224: train: epoch 0 | batch 1580/2304 [I 2022-10-26 09:45:03] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:06:03 [I 2022-10-26 09:45:03] ...trap/engines/engine.py.229: process: 0.49486 | load: 0.00261 [I 2022-10-26 09:45:03] ...trap/engines/engine.py.230: loss: 0.52548 [I 2022-10-26 09:45:03] ...optimizers/trijoint.py.151: total_norm: 4.915854 [I 2022-10-26 09:45:08] ...trap/engines/engine.py.224: train: epoch 0 | batch 1590/2304 [I 2022-10-26 09:45:08] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:05:55 [I 2022-10-26 09:45:08] ...trap/engines/engine.py.229: process: 0.50095 | load: 0.00256 [I 2022-10-26 09:45:08] ...trap/engines/engine.py.230: loss: 0.47410 [I 2022-10-26 09:45:08] ...optimizers/trijoint.py.151: total_norm: 4.901070 [I 2022-10-26 09:45:13] ...trap/engines/engine.py.224: train: epoch 0 | batch 1600/2304 [I 2022-10-26 09:45:13] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:05:51 [I 2022-10-26 09:45:13] ...trap/engines/engine.py.229: process: 0.48813 | load: 0.00256 [I 2022-10-26 09:45:13] ...trap/engines/engine.py.230: loss: 0.54542 [I 2022-10-26 09:45:13] ...optimizers/trijoint.py.151: total_norm: 6.019722 [I 2022-10-26 09:45:18] ...trap/engines/engine.py.224: train: epoch 0 | batch 1610/2304 [I 2022-10-26 09:45:18] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:05:46 [I 2022-10-26 09:45:18] ...trap/engines/engine.py.229: process: 0.49004 | load: 0.00255 [I 2022-10-26 09:45:18] ...trap/engines/engine.py.230: loss: 0.58922 [I 2022-10-26 09:45:18] ...optimizers/trijoint.py.151: total_norm: 5.311929 [I 2022-10-26 09:45:23] ...trap/engines/engine.py.224: train: epoch 0 | batch 1620/2304 [I 2022-10-26 09:45:23] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:05:43 [I 2022-10-26 09:45:23] ...trap/engines/engine.py.229: process: 0.48807 | load: 0.00271 [I 2022-10-26 09:45:23] ...trap/engines/engine.py.230: loss: 0.49911 [I 2022-10-26 09:45:23] ...optimizers/trijoint.py.151: total_norm: 4.511446 [I 2022-10-26 09:45:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 1630/2304 [I 2022-10-26 09:45:28] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:05:39 [I 2022-10-26 09:45:28] ...trap/engines/engine.py.229: process: 0.50377 | load: 0.00256 [I 2022-10-26 09:45:28] ...trap/engines/engine.py.230: loss: 0.63536 [I 2022-10-26 09:45:28] ...optimizers/trijoint.py.151: total_norm: 6.227126 [I 2022-10-26 09:45:33] ...trap/engines/engine.py.224: train: epoch 0 | batch 1640/2304 [I 2022-10-26 09:45:33] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:05:32 [I 2022-10-26 09:45:33] ...trap/engines/engine.py.229: process: 0.50491 | load: 0.00254 [I 2022-10-26 09:45:33] ...trap/engines/engine.py.230: loss: 0.57513 [I 2022-10-26 09:45:33] ...optimizers/trijoint.py.151: total_norm: 5.665650 [I 2022-10-26 09:45:38] ...trap/engines/engine.py.224: train: epoch 0 | batch 1650/2304 [I 2022-10-26 09:45:38] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:05:23 [I 2022-10-26 09:45:38] ...trap/engines/engine.py.229: process: 0.49012 | load: 0.00253 [I 2022-10-26 09:45:38] ...trap/engines/engine.py.230: loss: 0.53220 [I 2022-10-26 09:45:38] ...optimizers/trijoint.py.151: total_norm: 4.278418 [I 2022-10-26 09:45:43] ...trap/engines/engine.py.224: train: epoch 0 | batch 1660/2304 [I 2022-10-26 09:45:43] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:05:23 [I 2022-10-26 09:45:43] ...trap/engines/engine.py.229: process: 0.51227 | load: 0.00255 [I 2022-10-26 09:45:43] ...trap/engines/engine.py.230: loss: 0.43050 [I 2022-10-26 09:45:43] ...optimizers/trijoint.py.151: total_norm: 4.245697 [I 2022-10-26 09:45:49] ...trap/engines/engine.py.224: train: epoch 0 | batch 1670/2304 [I 2022-10-26 09:45:49] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:05:27 [I 2022-10-26 09:45:49] ...trap/engines/engine.py.229: process: 0.49343 | load: 0.00262 [I 2022-10-26 09:45:49] ...trap/engines/engine.py.230: loss: 0.52288 [I 2022-10-26 09:45:49] ...optimizers/trijoint.py.151: total_norm: 5.116936 [I 2022-10-26 09:45:54] ...trap/engines/engine.py.224: train: epoch 0 | batch 1680/2304 [I 2022-10-26 09:45:54] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:05:11 [I 2022-10-26 09:45:54] ...trap/engines/engine.py.229: process: 0.50439 | load: 0.00259 [I 2022-10-26 09:45:54] ...trap/engines/engine.py.230: loss: 0.66217 [I 2022-10-26 09:45:54] ...optimizers/trijoint.py.151: total_norm: 6.346713 [I 2022-10-26 09:45:59] ...trap/engines/engine.py.224: train: epoch 0 | batch 1690/2304 [I 2022-10-26 09:45:59] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:05:04 [I 2022-10-26 09:45:59] ...trap/engines/engine.py.229: process: 0.49309 | load: 0.00256 [I 2022-10-26 09:45:59] ...trap/engines/engine.py.230: loss: 0.48332 [I 2022-10-26 09:45:59] ...optimizers/trijoint.py.151: total_norm: 5.768026 [I 2022-10-26 09:46:04] ...trap/engines/engine.py.224: train: epoch 0 | batch 1700/2304 [I 2022-10-26 09:46:04] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:05:01 [I 2022-10-26 09:46:04] ...trap/engines/engine.py.229: process: 0.49172 | load: 0.00258 [I 2022-10-26 09:46:04] ...trap/engines/engine.py.230: loss: 0.56838 [I 2022-10-26 09:46:04] ...optimizers/trijoint.py.151: total_norm: 5.742578 [I 2022-10-26 09:46:09] ...trap/engines/engine.py.224: train: epoch 0 | batch 1710/2304 [I 2022-10-26 09:46:09] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:04:59 [I 2022-10-26 09:46:09] ...trap/engines/engine.py.229: process: 0.49970 | load: 0.00267 [I 2022-10-26 09:46:09] ...trap/engines/engine.py.230: loss: 0.53555 [I 2022-10-26 09:46:09] ...optimizers/trijoint.py.151: total_norm: 5.281719 [I 2022-10-26 09:46:14] ...trap/engines/engine.py.224: train: epoch 0 | batch 1720/2304 [I 2022-10-26 09:46:14] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:04:53 [I 2022-10-26 09:46:14] ...trap/engines/engine.py.229: process: 0.50743 | load: 0.00261 [I 2022-10-26 09:46:14] ...trap/engines/engine.py.230: loss: 0.52116 [I 2022-10-26 09:46:14] ...optimizers/trijoint.py.151: total_norm: 6.293751 [I 2022-10-26 09:46:19] ...trap/engines/engine.py.224: train: epoch 0 | batch 1730/2304 [I 2022-10-26 09:46:19] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:04:45 [I 2022-10-26 09:46:19] ...trap/engines/engine.py.229: process: 0.49944 | load: 0.00258 [I 2022-10-26 09:46:19] ...trap/engines/engine.py.230: loss: 0.48822 [I 2022-10-26 09:46:19] ...optimizers/trijoint.py.151: total_norm: 6.666967 [I 2022-10-26 09:46:24] ...trap/engines/engine.py.224: train: epoch 0 | batch 1740/2304 [I 2022-10-26 09:46:24] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:04:41 [I 2022-10-26 09:46:24] ...trap/engines/engine.py.229: process: 0.50628 | load: 0.00260 [I 2022-10-26 09:46:24] ...trap/engines/engine.py.230: loss: 0.40569 [I 2022-10-26 09:46:24] ...optimizers/trijoint.py.151: total_norm: 4.513314 [I 2022-10-26 09:46:29] ...trap/engines/engine.py.224: train: epoch 0 | batch 1750/2304 [I 2022-10-26 09:46:29] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:04:36 [I 2022-10-26 09:46:29] ...trap/engines/engine.py.229: process: 0.49171 | load: 0.00265 [I 2022-10-26 09:46:29] ...trap/engines/engine.py.230: loss: 0.52922 [I 2022-10-26 09:46:29] ...optimizers/trijoint.py.151: total_norm: 6.302258 [I 2022-10-26 09:46:34] ...trap/engines/engine.py.224: train: epoch 0 | batch 1760/2304 [I 2022-10-26 09:46:34] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:04:31 [I 2022-10-26 09:46:34] ...trap/engines/engine.py.229: process: 0.50739 | load: 0.00251 [I 2022-10-26 09:46:34] ...trap/engines/engine.py.230: loss: 0.52987 [I 2022-10-26 09:46:34] ...optimizers/trijoint.py.151: total_norm: 5.577235 [I 2022-10-26 09:46:39] ...trap/engines/engine.py.224: train: epoch 0 | batch 1770/2304 [I 2022-10-26 09:46:39] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:04:28 [I 2022-10-26 09:46:39] ...trap/engines/engine.py.229: process: 0.50196 | load: 0.00266 [I 2022-10-26 09:46:39] ...trap/engines/engine.py.230: loss: 0.59587 [I 2022-10-26 09:46:39] ...optimizers/trijoint.py.151: total_norm: 6.685706 [I 2022-10-26 09:46:44] ...trap/engines/engine.py.224: train: epoch 0 | batch 1780/2304 [I 2022-10-26 09:46:44] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:04:22 [I 2022-10-26 09:46:44] ...trap/engines/engine.py.229: process: 0.50480 | load: 0.00261 [I 2022-10-26 09:46:44] ...trap/engines/engine.py.230: loss: 0.57071 [I 2022-10-26 09:46:44] ...optimizers/trijoint.py.151: total_norm: 6.842613 [I 2022-10-26 09:46:49] ...trap/engines/engine.py.224: train: epoch 0 | batch 1790/2304 [I 2022-10-26 09:46:49] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:04:17 [I 2022-10-26 09:46:49] ...trap/engines/engine.py.229: process: 0.49589 | load: 0.00261 [I 2022-10-26 09:46:49] ...trap/engines/engine.py.230: loss: 0.50976 [I 2022-10-26 09:46:49] ...optimizers/trijoint.py.151: total_norm: 5.660924 [I 2022-10-26 09:46:54] ...trap/engines/engine.py.224: train: epoch 0 | batch 1800/2304 [I 2022-10-26 09:46:54] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:04:11 [I 2022-10-26 09:46:54] ...trap/engines/engine.py.229: process: 0.50816 | load: 0.00248 [I 2022-10-26 09:46:54] ...trap/engines/engine.py.230: loss: 0.41540 [I 2022-10-26 09:46:54] ...optimizers/trijoint.py.151: total_norm: 4.576768 [I 2022-10-26 09:46:59] ...trap/engines/engine.py.224: train: epoch 0 | batch 1810/2304 [I 2022-10-26 09:46:59] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:04:20 [I 2022-10-26 09:46:59] ...trap/engines/engine.py.229: process: 0.50428 | load: 0.00259 [I 2022-10-26 09:46:59] ...trap/engines/engine.py.230: loss: 0.41696 [I 2022-10-26 09:46:59] ...optimizers/trijoint.py.151: total_norm: 5.164586 [I 2022-10-26 09:47:04] ...trap/engines/engine.py.224: train: epoch 0 | batch 1820/2304 [I 2022-10-26 09:47:04] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:04:00 [I 2022-10-26 09:47:04] ...trap/engines/engine.py.229: process: 0.49417 | load: 0.00254 [I 2022-10-26 09:47:04] ...trap/engines/engine.py.230: loss: 0.45954 [I 2022-10-26 09:47:04] ...optimizers/trijoint.py.151: total_norm: 5.849444 [I 2022-10-26 09:47:09] ...trap/engines/engine.py.224: train: epoch 0 | batch 1830/2304 [I 2022-10-26 09:47:09] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:03:57 [I 2022-10-26 09:47:09] ...trap/engines/engine.py.229: process: 0.50091 | load: 0.00259 [I 2022-10-26 09:47:09] ...trap/engines/engine.py.230: loss: 0.50932 [I 2022-10-26 09:47:09] ...optimizers/trijoint.py.151: total_norm: 5.338899 [I 2022-10-26 09:47:14] ...trap/engines/engine.py.224: train: epoch 0 | batch 1840/2304 [I 2022-10-26 09:47:14] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:03:51 [I 2022-10-26 09:47:14] ...trap/engines/engine.py.229: process: 0.49621 | load: 0.00258 [I 2022-10-26 09:47:14] ...trap/engines/engine.py.230: loss: 0.69587 [I 2022-10-26 09:47:14] ...optimizers/trijoint.py.151: total_norm: 5.915287 [I 2022-10-26 09:47:19] ...trap/engines/engine.py.224: train: epoch 0 | batch 1850/2304 [I 2022-10-26 09:47:19] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:03:48 [I 2022-10-26 09:47:19] ...trap/engines/engine.py.229: process: 0.50665 | load: 0.00258 [I 2022-10-26 09:47:19] ...trap/engines/engine.py.230: loss: 0.48921 [I 2022-10-26 09:47:19] ...optimizers/trijoint.py.151: total_norm: 6.101239 [I 2022-10-26 09:47:24] ...trap/engines/engine.py.224: train: epoch 0 | batch 1860/2304 [I 2022-10-26 09:47:24] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:03:41 [I 2022-10-26 09:47:24] ...trap/engines/engine.py.229: process: 0.49983 | load: 0.00261 [I 2022-10-26 09:47:24] ...trap/engines/engine.py.230: loss: 0.47634 [I 2022-10-26 09:47:24] ...optimizers/trijoint.py.151: total_norm: 5.185879 [I 2022-10-26 09:47:29] ...trap/engines/engine.py.224: train: epoch 0 | batch 1870/2304 [I 2022-10-26 09:47:29] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:03:36 [I 2022-10-26 09:47:29] ...trap/engines/engine.py.229: process: 0.49879 | load: 0.00286 [I 2022-10-26 09:47:29] ...trap/engines/engine.py.230: loss: 0.49227 [I 2022-10-26 09:47:29] ...optimizers/trijoint.py.151: total_norm: 5.162568 [I 2022-10-26 09:47:34] ...trap/engines/engine.py.224: train: epoch 0 | batch 1880/2304 [I 2022-10-26 09:47:34] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:03:32 [I 2022-10-26 09:47:34] ...trap/engines/engine.py.229: process: 0.49992 | load: 0.00256 [I 2022-10-26 09:47:34] ...trap/engines/engine.py.230: loss: 0.56324 [I 2022-10-26 09:47:34] ...optimizers/trijoint.py.151: total_norm: 5.168702 [I 2022-10-26 09:47:39] ...trap/engines/engine.py.224: train: epoch 0 | batch 1890/2304 [I 2022-10-26 09:47:39] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:03:29 [I 2022-10-26 09:47:39] ...trap/engines/engine.py.229: process: 0.50297 | load: 0.00254 [I 2022-10-26 09:47:39] ...trap/engines/engine.py.230: loss: 0.55376 [I 2022-10-26 09:47:39] ...optimizers/trijoint.py.151: total_norm: 5.625070 [I 2022-10-26 09:47:44] ...trap/engines/engine.py.224: train: epoch 0 | batch 1900/2304 [I 2022-10-26 09:47:44] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:03:23 [I 2022-10-26 09:47:44] ...trap/engines/engine.py.229: process: 0.49913 | load: 0.00250 [I 2022-10-26 09:47:44] ...trap/engines/engine.py.230: loss: 0.40284 [I 2022-10-26 09:47:44] ...optimizers/trijoint.py.151: total_norm: 3.804955 [I 2022-10-26 09:47:49] ...trap/engines/engine.py.224: train: epoch 0 | batch 1910/2304 [I 2022-10-26 09:47:49] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:03:16 [I 2022-10-26 09:47:49] ...trap/engines/engine.py.229: process: 0.49619 | load: 0.00254 [I 2022-10-26 09:47:49] ...trap/engines/engine.py.230: loss: 0.48452 [I 2022-10-26 09:47:49] ...optimizers/trijoint.py.151: total_norm: 5.621634 [I 2022-10-26 09:47:54] ...trap/engines/engine.py.224: train: epoch 0 | batch 1920/2304 [I 2022-10-26 09:47:54] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:03:10 [I 2022-10-26 09:47:54] ...trap/engines/engine.py.229: process: 0.50395 | load: 0.00259 [I 2022-10-26 09:47:54] ...trap/engines/engine.py.230: loss: 0.49835 [I 2022-10-26 09:47:54] ...optimizers/trijoint.py.151: total_norm: 5.167040 [I 2022-10-26 09:47:59] ...trap/engines/engine.py.224: train: epoch 0 | batch 1930/2304 [I 2022-10-26 09:47:59] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:03:05 [I 2022-10-26 09:47:59] ...trap/engines/engine.py.229: process: 0.48901 | load: 0.00251 [I 2022-10-26 09:47:59] ...trap/engines/engine.py.230: loss: 0.47453 [I 2022-10-26 09:47:59] ...optimizers/trijoint.py.151: total_norm: 4.685781 [I 2022-10-26 09:48:04] ...trap/engines/engine.py.224: train: epoch 0 | batch 1940/2304 [I 2022-10-26 09:48:04] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:03:02 [I 2022-10-26 09:48:04] ...trap/engines/engine.py.229: process: 0.50699 | load: 0.00248 [I 2022-10-26 09:48:04] ...trap/engines/engine.py.230: loss: 0.56593 [I 2022-10-26 09:48:04] ...optimizers/trijoint.py.151: total_norm: 5.167313 [I 2022-10-26 09:48:09] ...trap/engines/engine.py.224: train: epoch 0 | batch 1950/2304 [I 2022-10-26 09:48:09] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:02:57 [I 2022-10-26 09:48:09] ...trap/engines/engine.py.229: process: 0.50031 | load: 0.00250 [I 2022-10-26 09:48:09] ...trap/engines/engine.py.230: loss: 0.46573 [I 2022-10-26 09:48:09] ...optimizers/trijoint.py.151: total_norm: 7.208057 [I 2022-10-26 09:48:14] ...trap/engines/engine.py.224: train: epoch 0 | batch 1960/2304 [I 2022-10-26 09:48:14] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:03:00 [I 2022-10-26 09:48:14] ...trap/engines/engine.py.229: process: 0.57513 | load: 0.00277 [I 2022-10-26 09:48:14] ...trap/engines/engine.py.230: loss: 0.51752 [I 2022-10-26 09:48:14] ...optimizers/trijoint.py.151: total_norm: 5.372409 [I 2022-10-26 09:48:19] ...trap/engines/engine.py.224: train: epoch 0 | batch 1970/2304 [I 2022-10-26 09:48:19] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:02:47 [I 2022-10-26 09:48:19] ...trap/engines/engine.py.229: process: 0.50591 | load: 0.00259 [I 2022-10-26 09:48:19] ...trap/engines/engine.py.230: loss: 0.46804 [I 2022-10-26 09:48:19] ...optimizers/trijoint.py.151: total_norm: 3.983734 [I 2022-10-26 09:48:24] ...trap/engines/engine.py.224: train: epoch 0 | batch 1980/2304 [I 2022-10-26 09:48:24] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:02:42 [I 2022-10-26 09:48:24] ...trap/engines/engine.py.229: process: 0.50829 | load: 0.00255 [I 2022-10-26 09:48:24] ...trap/engines/engine.py.230: loss: 0.43288 [I 2022-10-26 09:48:24] ...optimizers/trijoint.py.151: total_norm: 4.564312 [I 2022-10-26 09:48:29] ...trap/engines/engine.py.224: train: epoch 0 | batch 1990/2304 [I 2022-10-26 09:48:29] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:02:36 [I 2022-10-26 09:48:29] ...trap/engines/engine.py.229: process: 0.50464 | load: 0.00259 [I 2022-10-26 09:48:29] ...trap/engines/engine.py.230: loss: 0.47425 [I 2022-10-26 09:48:29] ...optimizers/trijoint.py.151: total_norm: 5.575349 [I 2022-10-26 09:48:34] ...trap/engines/engine.py.224: train: epoch 0 | batch 2000/2304 [I 2022-10-26 09:48:34] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:02:32 [I 2022-10-26 09:48:34] ...trap/engines/engine.py.229: process: 0.49499 | load: 0.00248 [I 2022-10-26 09:48:34] ...trap/engines/engine.py.230: loss: 0.48092 [I 2022-10-26 09:48:34] ...optimizers/trijoint.py.151: total_norm: 5.449108 [I 2022-10-26 09:48:39] ...trap/engines/engine.py.224: train: epoch 0 | batch 2010/2304 [I 2022-10-26 09:48:39] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:02:26 [I 2022-10-26 09:48:39] ...trap/engines/engine.py.229: process: 0.49370 | load: 0.00258 [I 2022-10-26 09:48:39] ...trap/engines/engine.py.230: loss: 0.65814 [I 2022-10-26 09:48:39] ...optimizers/trijoint.py.151: total_norm: 5.328557 [I 2022-10-26 09:48:44] ...trap/engines/engine.py.224: train: epoch 0 | batch 2020/2304 [I 2022-10-26 09:48:44] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:02:21 [I 2022-10-26 09:48:44] ...trap/engines/engine.py.229: process: 0.49587 | load: 0.00252 [I 2022-10-26 09:48:44] ...trap/engines/engine.py.230: loss: 0.54095 [I 2022-10-26 09:48:44] ...optimizers/trijoint.py.151: total_norm: 5.809646 [I 2022-10-26 09:48:49] ...trap/engines/engine.py.224: train: epoch 0 | batch 2030/2304 [I 2022-10-26 09:48:49] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:02:19 [I 2022-10-26 09:48:49] ...trap/engines/engine.py.229: process: 0.50084 | load: 0.00252 [I 2022-10-26 09:48:49] ...trap/engines/engine.py.230: loss: 0.47063 [I 2022-10-26 09:48:49] ...optimizers/trijoint.py.151: total_norm: 4.900382 [I 2022-10-26 09:48:54] ...trap/engines/engine.py.224: train: epoch 0 | batch 2040/2304 [I 2022-10-26 09:48:54] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:02:12 [I 2022-10-26 09:48:54] ...trap/engines/engine.py.229: process: 0.50739 | load: 0.00256 [I 2022-10-26 09:48:54] ...trap/engines/engine.py.230: loss: 0.44493 [I 2022-10-26 09:48:54] ...optimizers/trijoint.py.151: total_norm: 5.732513 [I 2022-10-26 09:48:59] ...trap/engines/engine.py.224: train: epoch 0 | batch 2050/2304 [I 2022-10-26 09:48:59] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:02:07 [I 2022-10-26 09:48:59] ...trap/engines/engine.py.229: process: 0.49569 | load: 0.00254 [I 2022-10-26 09:48:59] ...trap/engines/engine.py.230: loss: 0.53937 [I 2022-10-26 09:48:59] ...optimizers/trijoint.py.151: total_norm: 5.203844 [I 2022-10-26 09:49:04] ...trap/engines/engine.py.224: train: epoch 0 | batch 2060/2304 [I 2022-10-26 09:49:04] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:02:02 [I 2022-10-26 09:49:04] ...trap/engines/engine.py.229: process: 0.51343 | load: 0.00257 [I 2022-10-26 09:49:04] ...trap/engines/engine.py.230: loss: 0.56677 [I 2022-10-26 09:49:04] ...optimizers/trijoint.py.151: total_norm: 6.399628 [I 2022-10-26 09:49:09] ...trap/engines/engine.py.224: train: epoch 0 | batch 2070/2304 [I 2022-10-26 09:49:09] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:01:55 [I 2022-10-26 09:49:09] ...trap/engines/engine.py.229: process: 0.49044 | load: 0.00244 [I 2022-10-26 09:49:09] ...trap/engines/engine.py.230: loss: 0.44040 [I 2022-10-26 09:49:09] ...optimizers/trijoint.py.151: total_norm: 4.947992 [I 2022-10-26 09:49:14] ...trap/engines/engine.py.224: train: epoch 0 | batch 2080/2304 [I 2022-10-26 09:49:14] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:01:50 [I 2022-10-26 09:49:14] ...trap/engines/engine.py.229: process: 0.48592 | load: 0.00251 [I 2022-10-26 09:49:14] ...trap/engines/engine.py.230: loss: 0.41900 [I 2022-10-26 09:49:14] ...optimizers/trijoint.py.151: total_norm: 4.648348 [I 2022-10-26 09:49:19] ...trap/engines/engine.py.224: train: epoch 0 | batch 2090/2304 [I 2022-10-26 09:49:19] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:01:47 [I 2022-10-26 09:49:19] ...trap/engines/engine.py.229: process: 0.49626 | load: 0.00259 [I 2022-10-26 09:49:19] ...trap/engines/engine.py.230: loss: 0.44152 [I 2022-10-26 09:49:19] ...optimizers/trijoint.py.151: total_norm: 4.274889 [I 2022-10-26 09:49:24] ...trap/engines/engine.py.224: train: epoch 0 | batch 2100/2304 [I 2022-10-26 09:49:24] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:01:43 [I 2022-10-26 09:49:24] ...trap/engines/engine.py.229: process: 0.51094 | load: 0.00260 [I 2022-10-26 09:49:24] ...trap/engines/engine.py.230: loss: 0.42850 [I 2022-10-26 09:49:24] ...optimizers/trijoint.py.151: total_norm: 4.720893 [I 2022-10-26 09:49:29] ...trap/engines/engine.py.224: train: epoch 0 | batch 2110/2304 [I 2022-10-26 09:49:29] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:01:36 [I 2022-10-26 09:49:29] ...trap/engines/engine.py.229: process: 0.49614 | load: 0.00259 [I 2022-10-26 09:49:29] ...trap/engines/engine.py.230: loss: 0.51738 [I 2022-10-26 09:49:29] ...optimizers/trijoint.py.151: total_norm: 6.005682 [I 2022-10-26 09:49:56] ...trap/engines/engine.py.224: train: epoch 0 | batch 2120/2304 [I 2022-10-26 09:49:56] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:14:48 [I 2022-10-26 09:49:56] ...trap/engines/engine.py.229: process: 22.14421 | load: 0.00253 [I 2022-10-26 09:49:56] ...trap/engines/engine.py.230: loss: 0.39885 [I 2022-10-26 09:49:56] ...optimizers/trijoint.py.151: total_norm: 3.622790 [I 2022-10-26 09:50:01] ...trap/engines/engine.py.224: train: epoch 0 | batch 2130/2304 [I 2022-10-26 09:50:01] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:02:48 [I 2022-10-26 09:50:01] ...trap/engines/engine.py.229: process: 0.49091 | load: 0.00247 [I 2022-10-26 09:50:01] ...trap/engines/engine.py.230: loss: 0.45198 [I 2022-10-26 09:50:01] ...optimizers/trijoint.py.151: total_norm: 5.039651 [I 2022-10-26 09:50:06] ...trap/engines/engine.py.224: train: epoch 0 | batch 2140/2304 [I 2022-10-26 09:50:06] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:01:30 [I 2022-10-26 09:50:06] ...trap/engines/engine.py.229: process: 0.50329 | load: 0.00247 [I 2022-10-26 09:50:06] ...trap/engines/engine.py.230: loss: 0.39120 [I 2022-10-26 09:50:06] ...optimizers/trijoint.py.151: total_norm: 3.844599 [I 2022-10-26 09:50:11] ...trap/engines/engine.py.224: train: epoch 0 | batch 2150/2304 [I 2022-10-26 09:50:11] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:01:17 [I 2022-10-26 09:50:11] ...trap/engines/engine.py.229: process: 0.50582 | load: 0.00244 [I 2022-10-26 09:50:11] ...trap/engines/engine.py.230: loss: 0.39086 [I 2022-10-26 09:50:11] ...optimizers/trijoint.py.151: total_norm: 4.062638 [I 2022-10-26 09:50:16] ...trap/engines/engine.py.224: train: epoch 0 | batch 2160/2304 [I 2022-10-26 09:50:16] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:01:12 [I 2022-10-26 09:50:16] ...trap/engines/engine.py.229: process: 0.52277 | load: 0.00252 [I 2022-10-26 09:50:16] ...trap/engines/engine.py.230: loss: 0.48415 [I 2022-10-26 09:50:16] ...optimizers/trijoint.py.151: total_norm: 6.744070 [I 2022-10-26 09:50:21] ...trap/engines/engine.py.224: train: epoch 0 | batch 2170/2304 [I 2022-10-26 09:50:21] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:01:06 [I 2022-10-26 09:50:21] ...trap/engines/engine.py.229: process: 0.49278 | load: 0.00245 [I 2022-10-26 09:50:21] ...trap/engines/engine.py.230: loss: 0.53813 [I 2022-10-26 09:50:21] ...optimizers/trijoint.py.151: total_norm: 5.200740 [I 2022-10-26 09:50:26] ...trap/engines/engine.py.224: train: epoch 0 | batch 2180/2304 [I 2022-10-26 09:50:26] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:01:01 [I 2022-10-26 09:50:26] ...trap/engines/engine.py.229: process: 0.49483 | load: 0.00244 [I 2022-10-26 09:50:26] ...trap/engines/engine.py.230: loss: 0.41271 [I 2022-10-26 09:50:26] ...optimizers/trijoint.py.151: total_norm: 5.065856 [I 2022-10-26 09:50:31] ...trap/engines/engine.py.224: train: epoch 0 | batch 2190/2304 [I 2022-10-26 09:50:31] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:00:57 [I 2022-10-26 09:50:31] ...trap/engines/engine.py.229: process: 0.51339 | load: 0.00247 [I 2022-10-26 09:50:31] ...trap/engines/engine.py.230: loss: 0.51076 [I 2022-10-26 09:50:31] ...optimizers/trijoint.py.151: total_norm: 5.871777 [I 2022-10-26 09:50:36] ...trap/engines/engine.py.224: train: epoch 0 | batch 2200/2304 [I 2022-10-26 09:50:36] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:00:52 [I 2022-10-26 09:50:36] ...trap/engines/engine.py.229: process: 0.50303 | load: 0.00258 [I 2022-10-26 09:50:36] ...trap/engines/engine.py.230: loss: 0.55669 [I 2022-10-26 09:50:36] ...optimizers/trijoint.py.151: total_norm: 7.086389 [I 2022-10-26 09:50:41] ...trap/engines/engine.py.224: train: epoch 0 | batch 2210/2304 [I 2022-10-26 09:50:41] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:00:47 [I 2022-10-26 09:50:41] ...trap/engines/engine.py.229: process: 0.49676 | load: 0.00251 [I 2022-10-26 09:50:41] ...trap/engines/engine.py.230: loss: 0.49638 [I 2022-10-26 09:50:41] ...optimizers/trijoint.py.151: total_norm: 4.731479 [I 2022-10-26 09:50:46] ...trap/engines/engine.py.224: train: epoch 0 | batch 2220/2304 [I 2022-10-26 09:50:46] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:00:42 [I 2022-10-26 09:50:46] ...trap/engines/engine.py.229: process: 0.49348 | load: 0.00241 [I 2022-10-26 09:50:46] ...trap/engines/engine.py.230: loss: 0.51182 [I 2022-10-26 09:50:46] ...optimizers/trijoint.py.151: total_norm: 4.827049 [I 2022-10-26 09:50:51] ...trap/engines/engine.py.224: train: epoch 0 | batch 2230/2304 [I 2022-10-26 09:50:51] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:00:36 [I 2022-10-26 09:50:51] ...trap/engines/engine.py.229: process: 0.49726 | load: 0.00243 [I 2022-10-26 09:50:51] ...trap/engines/engine.py.230: loss: 0.48692 [I 2022-10-26 09:50:51] ...optimizers/trijoint.py.151: total_norm: 4.660411 [I 2022-10-26 09:50:56] ...trap/engines/engine.py.224: train: epoch 0 | batch 2240/2304 [I 2022-10-26 09:50:56] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:00:31 [I 2022-10-26 09:50:56] ...trap/engines/engine.py.229: process: 0.49386 | load: 0.00240 [I 2022-10-26 09:50:56] ...trap/engines/engine.py.230: loss: 0.52417 [I 2022-10-26 09:50:56] ...optimizers/trijoint.py.151: total_norm: 5.017496 [I 2022-10-26 09:51:01] ...trap/engines/engine.py.224: train: epoch 0 | batch 2250/2304 [I 2022-10-26 09:51:01] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:00:26 [I 2022-10-26 09:51:01] ...trap/engines/engine.py.229: process: 0.49700 | load: 0.00245 [I 2022-10-26 09:51:01] ...trap/engines/engine.py.230: loss: 0.47783 [I 2022-10-26 09:51:01] ...optimizers/trijoint.py.151: total_norm: 5.034266 [I 2022-10-26 09:51:06] ...trap/engines/engine.py.224: train: epoch 0 | batch 2260/2304 [I 2022-10-26 09:51:06] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:00:21 [I 2022-10-26 09:51:06] ...trap/engines/engine.py.229: process: 0.49873 | load: 0.00259 [I 2022-10-26 09:51:06] ...trap/engines/engine.py.230: loss: 0.48874 [I 2022-10-26 09:51:06] ...optimizers/trijoint.py.151: total_norm: 5.036857 [I 2022-10-26 09:51:11] ...trap/engines/engine.py.224: train: epoch 0 | batch 2270/2304 [I 2022-10-26 09:51:11] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:00:16 [I 2022-10-26 09:51:11] ...trap/engines/engine.py.229: process: 0.49202 | load: 0.00255 [I 2022-10-26 09:51:11] ...trap/engines/engine.py.230: loss: 0.39550 [I 2022-10-26 09:51:11] ...optimizers/trijoint.py.151: total_norm: 4.000208 [I 2022-10-26 09:51:16] ...trap/engines/engine.py.224: train: epoch 0 | batch 2280/2304 [I 2022-10-26 09:51:16] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:00:11 [I 2022-10-26 09:51:16] ...trap/engines/engine.py.229: process: 0.50775 | load: 0.00252 [I 2022-10-26 09:51:16] ...trap/engines/engine.py.230: loss: 0.53652 [I 2022-10-26 09:51:16] ...optimizers/trijoint.py.151: total_norm: 4.845975 [I 2022-10-26 09:51:21] ...trap/engines/engine.py.224: train: epoch 0 | batch 2290/2304 [I 2022-10-26 09:51:21] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:00:06 [I 2022-10-26 09:51:21] ...trap/engines/engine.py.229: process: 0.50321 | load: 0.00271 [I 2022-10-26 09:51:21] ...trap/engines/engine.py.230: loss: 0.45279 [I 2022-10-26 09:51:21] ...optimizers/trijoint.py.151: total_norm: 5.709720 [I 2022-10-26 09:51:26] ...trap/engines/engine.py.224: train: epoch 0 | batch 2300/2304 [I 2022-10-26 09:51:26] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:00:02 [I 2022-10-26 09:51:26] ...trap/engines/engine.py.229: process: 0.49107 | load: 0.00019 [I 2022-10-26 09:51:26] ...trap/engines/engine.py.230: loss: 0.51818 [I 2022-10-26 09:51:26] ...optimizers/trijoint.py.151: total_norm: 5.608180 [I 2022-10-26 09:51:28] ...trap/engines/engine.py.224: train: epoch 0 | batch 2304/2304 [I 2022-10-26 09:51:28] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:00:00 [I 2022-10-26 09:51:28] ...trap/engines/engine.py.229: process: 0.50279 | load: 0.00013 [I 2022-10-26 09:51:28] ...trap/engines/engine.py.230: loss: 0.54386 [I 2022-10-26 09:51:28] ...optimizers/trijoint.py.151: total_norm: 5.518274 [S 2022-10-26 09:51:29] ...trap/engines/engine.py.129: train_epoch.epoch: 0 [S 2022-10-26 09:51:29] ...trap/engines/engine.py.129: train_epoch.loss: 0.6859714342818601 [I 2022-10-26 09:51:29] ...s/criterions/triplet.py.84: margin = 0.055 [I 2022-10-26 09:51:29] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 0 [I 2022-10-26 09:51:29] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.med_im2recipe_mean' not in 'logs.json' [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_1_im2recipe_mean' not in 'logs.json' [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_5_im2recipe_mean' not in 'logs.json' [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_10_im2recipe_mean' not in 'logs.json' [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.med_recipe2im_mean' not in 'logs.json' [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_1_recipe2im_mean' not in 'logs.json' [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_5_recipe2im_mean' not in 'logs.json' [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.87: View 'eval_epoch.metric.recall_at_10_recipe2im_mean' not in 'logs.json' [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 09:51:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 09:51:31] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 09:51:32] ...trap/engines/engine.py.321: eval: epoch 0 | batch 0/510 [I 2022-10-26 09:51:32] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:25:19 [I 2022-10-26 09:51:32] ...trap/engines/engine.py.326: process: 2.97995 | load: 2.11920 [I 2022-10-26 09:51:36] ...trap/engines/engine.py.321: eval: epoch 0 | batch 10/510 [I 2022-10-26 09:51:36] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:12 [I 2022-10-26 09:51:36] ...trap/engines/engine.py.326: process: 0.32418 | load: 0.00016 [I 2022-10-26 09:51:39] ...trap/engines/engine.py.321: eval: epoch 0 | batch 20/510 [I 2022-10-26 09:51:39] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:03:03 [I 2022-10-26 09:51:39] ...trap/engines/engine.py.326: process: 0.33518 | load: 0.00017 [I 2022-10-26 09:51:43] ...trap/engines/engine.py.321: eval: epoch 0 | batch 30/510 [I 2022-10-26 09:51:43] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:42 [I 2022-10-26 09:51:43] ...trap/engines/engine.py.326: process: 0.32073 | load: 0.00020 [I 2022-10-26 09:51:46] ...trap/engines/engine.py.321: eval: epoch 0 | batch 40/510 [I 2022-10-26 09:51:46] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:36 [I 2022-10-26 09:51:46] ...trap/engines/engine.py.326: process: 0.32854 | load: 0.00020 [I 2022-10-26 09:51:49] ...trap/engines/engine.py.321: eval: epoch 0 | batch 50/510 [I 2022-10-26 09:51:49] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-26 09:51:49] ...trap/engines/engine.py.326: process: 0.33586 | load: 0.00021 [I 2022-10-26 09:51:53] ...trap/engines/engine.py.321: eval: epoch 0 | batch 60/510 [I 2022-10-26 09:51:53] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:32 [I 2022-10-26 09:51:53] ...trap/engines/engine.py.326: process: 0.31990 | load: 0.00020 [I 2022-10-26 09:51:56] ...trap/engines/engine.py.321: eval: epoch 0 | batch 70/510 [I 2022-10-26 09:51:56] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-26 09:51:56] ...trap/engines/engine.py.326: process: 0.32853 | load: 0.00027 [I 2022-10-26 09:51:59] ...trap/engines/engine.py.321: eval: epoch 0 | batch 80/510 [I 2022-10-26 09:51:59] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:25 [I 2022-10-26 09:51:59] ...trap/engines/engine.py.326: process: 0.33187 | load: 0.00021 [I 2022-10-26 09:52:03] ...trap/engines/engine.py.321: eval: epoch 0 | batch 90/510 [I 2022-10-26 09:52:03] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:19 [I 2022-10-26 09:52:03] ...trap/engines/engine.py.326: process: 0.33297 | load: 0.00020 [I 2022-10-26 09:52:06] ...trap/engines/engine.py.321: eval: epoch 0 | batch 100/510 [I 2022-10-26 09:52:06] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:15 [I 2022-10-26 09:52:06] ...trap/engines/engine.py.326: process: 0.28686 | load: 0.00019 [I 2022-10-26 09:52:09] ...trap/engines/engine.py.321: eval: epoch 0 | batch 110/510 [I 2022-10-26 09:52:09] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:59 [I 2022-10-26 09:52:09] ...trap/engines/engine.py.326: process: 0.30653 | load: 0.00017 [I 2022-10-26 09:52:12] ...trap/engines/engine.py.321: eval: epoch 0 | batch 120/510 [I 2022-10-26 09:52:12] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 09:52:12] ...trap/engines/engine.py.326: process: 0.30644 | load: 0.00015 [I 2022-10-26 09:52:15] ...trap/engines/engine.py.321: eval: epoch 0 | batch 130/510 [I 2022-10-26 09:52:15] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-26 09:52:15] ...trap/engines/engine.py.326: process: 0.29809 | load: 0.00018 [I 2022-10-26 09:52:18] ...trap/engines/engine.py.321: eval: epoch 0 | batch 140/510 [I 2022-10-26 09:52:18] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-26 09:52:18] ...trap/engines/engine.py.326: process: 0.28537 | load: 0.00015 [I 2022-10-26 09:52:21] ...trap/engines/engine.py.321: eval: epoch 0 | batch 150/510 [I 2022-10-26 09:52:21] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 09:52:21] ...trap/engines/engine.py.326: process: 0.29061 | load: 0.00018 [I 2022-10-26 09:52:24] ...trap/engines/engine.py.321: eval: epoch 0 | batch 160/510 [I 2022-10-26 09:52:24] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-26 09:52:24] ...trap/engines/engine.py.326: process: 0.30335 | load: 0.00026 [I 2022-10-26 09:52:27] ...trap/engines/engine.py.321: eval: epoch 0 | batch 170/510 [I 2022-10-26 09:52:27] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:39 [I 2022-10-26 09:52:27] ...trap/engines/engine.py.326: process: 0.29119 | load: 0.00019 [I 2022-10-26 09:52:29] ...trap/engines/engine.py.321: eval: epoch 0 | batch 180/510 [I 2022-10-26 09:52:29] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-26 09:52:29] ...trap/engines/engine.py.326: process: 0.28796 | load: 0.00016 [I 2022-10-26 09:52:32] ...trap/engines/engine.py.321: eval: epoch 0 | batch 190/510 [I 2022-10-26 09:52:32] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:34 [I 2022-10-26 09:52:32] ...trap/engines/engine.py.326: process: 0.28993 | load: 0.00021 [I 2022-10-26 09:52:35] ...trap/engines/engine.py.321: eval: epoch 0 | batch 200/510 [I 2022-10-26 09:52:35] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:31 [I 2022-10-26 09:52:35] ...trap/engines/engine.py.326: process: 0.29566 | load: 0.00017 [I 2022-10-26 09:52:38] ...trap/engines/engine.py.321: eval: epoch 0 | batch 210/510 [I 2022-10-26 09:52:38] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 09:52:38] ...trap/engines/engine.py.326: process: 0.28674 | load: 0.00018 [I 2022-10-26 09:52:41] ...trap/engines/engine.py.321: eval: epoch 0 | batch 220/510 [I 2022-10-26 09:52:41] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 09:52:41] ...trap/engines/engine.py.326: process: 0.28667 | load: 0.00015 [I 2022-10-26 09:52:44] ...trap/engines/engine.py.321: eval: epoch 0 | batch 230/510 [I 2022-10-26 09:52:44] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 09:52:44] ...trap/engines/engine.py.326: process: 0.28534 | load: 0.00019 [I 2022-10-26 09:52:47] ...trap/engines/engine.py.321: eval: epoch 0 | batch 240/510 [I 2022-10-26 09:52:47] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 09:52:47] ...trap/engines/engine.py.326: process: 0.28336 | load: 0.00020 [I 2022-10-26 09:52:50] ...trap/engines/engine.py.321: eval: epoch 0 | batch 250/510 [I 2022-10-26 09:52:50] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-26 09:52:50] ...trap/engines/engine.py.326: process: 0.29138 | load: 0.00019 [I 2022-10-26 09:52:53] ...trap/engines/engine.py.321: eval: epoch 0 | batch 260/510 [I 2022-10-26 09:52:53] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-26 09:52:53] ...trap/engines/engine.py.326: process: 0.29126 | load: 0.00023 [I 2022-10-26 09:52:56] ...trap/engines/engine.py.321: eval: epoch 0 | batch 270/510 [I 2022-10-26 09:52:56] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:10 [I 2022-10-26 09:52:56] ...trap/engines/engine.py.326: process: 0.28718 | load: 0.00017 [I 2022-10-26 09:52:59] ...trap/engines/engine.py.321: eval: epoch 0 | batch 280/510 [I 2022-10-26 09:52:59] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:07 [I 2022-10-26 09:52:59] ...trap/engines/engine.py.326: process: 0.28983 | load: 0.00015 [I 2022-10-26 09:53:01] ...trap/engines/engine.py.321: eval: epoch 0 | batch 290/510 [I 2022-10-26 09:53:01] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-26 09:53:01] ...trap/engines/engine.py.326: process: 0.28456 | load: 0.00020 [I 2022-10-26 09:53:04] ...trap/engines/engine.py.321: eval: epoch 0 | batch 300/510 [I 2022-10-26 09:53:04] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-26 09:53:04] ...trap/engines/engine.py.326: process: 0.28694 | load: 0.00015 [I 2022-10-26 09:53:07] ...trap/engines/engine.py.321: eval: epoch 0 | batch 310/510 [I 2022-10-26 09:53:07] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 09:53:07] ...trap/engines/engine.py.326: process: 0.28901 | load: 0.00019 [I 2022-10-26 09:53:10] ...trap/engines/engine.py.321: eval: epoch 0 | batch 320/510 [I 2022-10-26 09:53:10] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 09:53:10] ...trap/engines/engine.py.326: process: 0.29012 | load: 0.00015 [I 2022-10-26 09:53:13] ...trap/engines/engine.py.321: eval: epoch 0 | batch 330/510 [I 2022-10-26 09:53:13] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 09:53:13] ...trap/engines/engine.py.326: process: 0.29604 | load: 0.00017 [I 2022-10-26 09:53:16] ...trap/engines/engine.py.321: eval: epoch 0 | batch 340/510 [I 2022-10-26 09:53:16] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 09:53:16] ...trap/engines/engine.py.326: process: 0.28045 | load: 0.00015 [I 2022-10-26 09:53:19] ...trap/engines/engine.py.321: eval: epoch 0 | batch 350/510 [I 2022-10-26 09:53:19] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-26 09:53:19] ...trap/engines/engine.py.326: process: 0.28970 | load: 0.00015 [I 2022-10-26 09:53:22] ...trap/engines/engine.py.321: eval: epoch 0 | batch 360/510 [I 2022-10-26 09:53:22] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 09:53:22] ...trap/engines/engine.py.326: process: 0.28290 | load: 0.00020 [I 2022-10-26 09:53:24] ...trap/engines/engine.py.321: eval: epoch 0 | batch 370/510 [I 2022-10-26 09:53:24] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-26 09:53:24] ...trap/engines/engine.py.326: process: 0.28555 | load: 0.00023 [I 2022-10-26 09:53:27] ...trap/engines/engine.py.321: eval: epoch 0 | batch 380/510 [I 2022-10-26 09:53:27] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-26 09:53:27] ...trap/engines/engine.py.326: process: 0.28684 | load: 0.00021 [I 2022-10-26 09:53:30] ...trap/engines/engine.py.321: eval: epoch 0 | batch 390/510 [I 2022-10-26 09:53:30] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 09:53:30] ...trap/engines/engine.py.326: process: 0.28279 | load: 0.00015 [I 2022-10-26 09:53:33] ...trap/engines/engine.py.321: eval: epoch 0 | batch 400/510 [I 2022-10-26 09:53:33] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 09:53:33] ...trap/engines/engine.py.326: process: 0.28043 | load: 0.00020 [I 2022-10-26 09:53:36] ...trap/engines/engine.py.321: eval: epoch 0 | batch 410/510 [I 2022-10-26 09:53:36] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 09:53:36] ...trap/engines/engine.py.326: process: 0.29051 | load: 0.00016 [I 2022-10-26 09:53:39] ...trap/engines/engine.py.321: eval: epoch 0 | batch 420/510 [I 2022-10-26 09:53:39] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 09:53:39] ...trap/engines/engine.py.326: process: 0.28757 | load: 0.00015 [I 2022-10-26 09:53:42] ...trap/engines/engine.py.321: eval: epoch 0 | batch 430/510 [I 2022-10-26 09:53:42] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 09:53:42] ...trap/engines/engine.py.326: process: 0.28682 | load: 0.00018 [I 2022-10-26 09:53:45] ...trap/engines/engine.py.321: eval: epoch 0 | batch 440/510 [I 2022-10-26 09:53:45] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 09:53:45] ...trap/engines/engine.py.326: process: 0.28480 | load: 0.00015 [I 2022-10-26 09:53:48] ...trap/engines/engine.py.321: eval: epoch 0 | batch 450/510 [I 2022-10-26 09:53:48] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 09:53:48] ...trap/engines/engine.py.326: process: 0.28609 | load: 0.00021 [I 2022-10-26 09:53:50] ...trap/engines/engine.py.321: eval: epoch 0 | batch 460/510 [I 2022-10-26 09:53:50] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-26 09:53:50] ...trap/engines/engine.py.326: process: 0.28277 | load: 0.00020 [I 2022-10-26 09:53:53] ...trap/engines/engine.py.321: eval: epoch 0 | batch 470/510 [I 2022-10-26 09:53:53] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 09:53:53] ...trap/engines/engine.py.326: process: 0.28750 | load: 0.00015 [I 2022-10-26 09:53:56] ...trap/engines/engine.py.321: eval: epoch 0 | batch 480/510 [I 2022-10-26 09:53:56] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 09:53:56] ...trap/engines/engine.py.326: process: 0.28800 | load: 0.00018 [I 2022-10-26 09:53:59] ...trap/engines/engine.py.321: eval: epoch 0 | batch 490/510 [I 2022-10-26 09:53:59] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 09:53:59] ...trap/engines/engine.py.326: process: 0.27994 | load: 0.00017 [I 2022-10-26 09:54:02] ...trap/engines/engine.py.321: eval: epoch 0 | batch 500/510 [I 2022-10-26 09:54:02] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 09:54:02] ...trap/engines/engine.py.326: process: 0.28587 | load: 0.00021 [I 2022-10-26 09:54:05] ...trap/engines/engine.py.321: eval: epoch 0 | batch 510/510 [I 2022-10-26 09:54:05] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 09:54:05] ...trap/engines/engine.py.326: process: 0.28289 | load: 0.00015 [S 2022-10-26 09:54:06] ...trap/engines/engine.py.132: eval_epoch.epoch: 0 [I 2022-10-26 09:54:06] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 5.70 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 5.00 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.205800 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.494700 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.637200 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.232000 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.522900 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.661000 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.458258 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.632456 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009796 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010277 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.018899 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012239 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.021045 [S 2022-10-26 09:54:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.021288 [I 2022-10-26 09:54:10] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 09:54:10] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 09:54:10] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 09:54:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 09:54:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 09:54:10] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 09:54:13] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 09:54:17] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 09:54:17] ...trap/engines/engine.py.163: Training model on trainset for epoch 1 [I 2022-10-26 09:54:17] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 09:54:21] ...trap/engines/engine.py.224: train: epoch 1 | batch 0/2304 [I 2022-10-26 09:54:21] ...trap/engines/engine.py.228: elapsed: 0:00:03 | left: 2:04:34 [I 2022-10-26 09:54:21] ...trap/engines/engine.py.229: process: 3.24427 | load: 2.17551 [I 2022-10-26 09:54:21] ...trap/engines/engine.py.230: loss: 0.51908 [I 2022-10-26 09:54:21] ...optimizers/trijoint.py.151: total_norm: 4.559384 [I 2022-10-26 09:54:26] ...trap/engines/engine.py.224: train: epoch 1 | batch 10/2304 [I 2022-10-26 09:54:26] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:21 [I 2022-10-26 09:54:26] ...trap/engines/engine.py.229: process: 0.49574 | load: 0.00484 [I 2022-10-26 09:54:26] ...trap/engines/engine.py.230: loss: 0.38475 [I 2022-10-26 09:54:26] ...optimizers/trijoint.py.151: total_norm: 5.812269 [I 2022-10-26 09:54:31] ...trap/engines/engine.py.224: train: epoch 1 | batch 20/2304 [I 2022-10-26 09:54:31] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:20:25 [I 2022-10-26 09:54:31] ...trap/engines/engine.py.229: process: 0.51560 | load: 0.00317 [I 2022-10-26 09:54:31] ...trap/engines/engine.py.230: loss: 0.48925 [I 2022-10-26 09:54:31] ...optimizers/trijoint.py.151: total_norm: 5.618195 [I 2022-10-26 09:54:36] ...trap/engines/engine.py.224: train: epoch 1 | batch 30/2304 [I 2022-10-26 09:54:36] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:19:21 [I 2022-10-26 09:54:36] ...trap/engines/engine.py.229: process: 0.49178 | load: 0.00334 [I 2022-10-26 09:54:36] ...trap/engines/engine.py.230: loss: 0.51536 [I 2022-10-26 09:54:36] ...optimizers/trijoint.py.151: total_norm: 5.520526 [I 2022-10-26 09:54:41] ...trap/engines/engine.py.224: train: epoch 1 | batch 40/2304 [I 2022-10-26 09:54:41] ...trap/engines/engine.py.228: elapsed: 0:00:23 | left: 0:19:03 [I 2022-10-26 09:54:41] ...trap/engines/engine.py.229: process: 0.50541 | load: 0.00317 [I 2022-10-26 09:54:41] ...trap/engines/engine.py.230: loss: 0.44564 [I 2022-10-26 09:54:41] ...optimizers/trijoint.py.151: total_norm: 4.846355 [I 2022-10-26 09:54:46] ...trap/engines/engine.py.224: train: epoch 1 | batch 50/2304 [I 2022-10-26 09:54:46] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:19:04 [I 2022-10-26 09:54:46] ...trap/engines/engine.py.229: process: 0.50650 | load: 0.00322 [I 2022-10-26 09:54:46] ...trap/engines/engine.py.230: loss: 0.47979 [I 2022-10-26 09:54:46] ...optimizers/trijoint.py.151: total_norm: 4.658461 [I 2022-10-26 09:54:51] ...trap/engines/engine.py.224: train: epoch 1 | batch 60/2304 [I 2022-10-26 09:54:51] ...trap/engines/engine.py.228: elapsed: 0:00:33 | left: 0:19:13 [I 2022-10-26 09:54:51] ...trap/engines/engine.py.229: process: 0.51104 | load: 0.00272 [I 2022-10-26 09:54:51] ...trap/engines/engine.py.230: loss: 0.52877 [I 2022-10-26 09:54:51] ...optimizers/trijoint.py.151: total_norm: 6.192705 [I 2022-10-26 09:54:56] ...trap/engines/engine.py.224: train: epoch 1 | batch 70/2304 [I 2022-10-26 09:54:56] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:19:09 [I 2022-10-26 09:54:56] ...trap/engines/engine.py.229: process: 0.52808 | load: 0.00264 [I 2022-10-26 09:54:56] ...trap/engines/engine.py.230: loss: 0.47065 [I 2022-10-26 09:54:56] ...optimizers/trijoint.py.151: total_norm: 5.339875 [I 2022-10-26 09:55:01] ...trap/engines/engine.py.224: train: epoch 1 | batch 80/2304 [I 2022-10-26 09:55:01] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:18:34 [I 2022-10-26 09:55:01] ...trap/engines/engine.py.229: process: 0.49483 | load: 0.00253 [I 2022-10-26 09:55:01] ...trap/engines/engine.py.230: loss: 0.58348 [I 2022-10-26 09:55:01] ...optimizers/trijoint.py.151: total_norm: 6.136307 [I 2022-10-26 09:55:07] ...trap/engines/engine.py.224: train: epoch 1 | batch 90/2304 [I 2022-10-26 09:55:07] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:18:44 [I 2022-10-26 09:55:07] ...trap/engines/engine.py.229: process: 0.50173 | load: 0.00268 [I 2022-10-26 09:55:07] ...trap/engines/engine.py.230: loss: 0.46685 [I 2022-10-26 09:55:07] ...optimizers/trijoint.py.151: total_norm: 6.451020 [I 2022-10-26 09:55:12] ...trap/engines/engine.py.224: train: epoch 1 | batch 100/2304 [I 2022-10-26 09:55:12] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:18:30 [I 2022-10-26 09:55:12] ...trap/engines/engine.py.229: process: 0.49907 | load: 0.00288 [I 2022-10-26 09:55:12] ...trap/engines/engine.py.230: loss: 0.49930 [I 2022-10-26 09:55:12] ...optimizers/trijoint.py.151: total_norm: 5.597136 [I 2022-10-26 09:55:17] ...trap/engines/engine.py.224: train: epoch 1 | batch 110/2304 [I 2022-10-26 09:55:17] ...trap/engines/engine.py.228: elapsed: 0:00:59 | left: 0:18:25 [I 2022-10-26 09:55:17] ...trap/engines/engine.py.229: process: 0.49309 | load: 0.00262 [I 2022-10-26 09:55:17] ...trap/engines/engine.py.230: loss: 0.36350 [I 2022-10-26 09:55:17] ...optimizers/trijoint.py.151: total_norm: 3.929482 [I 2022-10-26 09:55:22] ...trap/engines/engine.py.224: train: epoch 1 | batch 120/2304 [I 2022-10-26 09:55:22] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:18:16 [I 2022-10-26 09:55:22] ...trap/engines/engine.py.229: process: 0.49369 | load: 0.00274 [I 2022-10-26 09:55:22] ...trap/engines/engine.py.230: loss: 0.45868 [I 2022-10-26 09:55:22] ...optimizers/trijoint.py.151: total_norm: 5.159364 [I 2022-10-26 09:55:27] ...trap/engines/engine.py.224: train: epoch 1 | batch 130/2304 [I 2022-10-26 09:55:27] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:18:06 [I 2022-10-26 09:55:27] ...trap/engines/engine.py.229: process: 0.50501 | load: 0.00244 [I 2022-10-26 09:55:27] ...trap/engines/engine.py.230: loss: 0.56864 [I 2022-10-26 09:55:27] ...optimizers/trijoint.py.151: total_norm: 5.848804 [I 2022-10-26 09:55:32] ...trap/engines/engine.py.224: train: epoch 1 | batch 140/2304 [I 2022-10-26 09:55:32] ...trap/engines/engine.py.228: elapsed: 0:01:14 | left: 0:18:07 [I 2022-10-26 09:55:32] ...trap/engines/engine.py.229: process: 0.49512 | load: 0.00267 [I 2022-10-26 09:55:32] ...trap/engines/engine.py.230: loss: 0.63399 [I 2022-10-26 09:55:32] ...optimizers/trijoint.py.151: total_norm: 6.401259 [I 2022-10-26 09:55:37] ...trap/engines/engine.py.224: train: epoch 1 | batch 150/2304 [I 2022-10-26 09:55:37] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:18:02 [I 2022-10-26 09:55:37] ...trap/engines/engine.py.229: process: 0.48746 | load: 0.00243 [I 2022-10-26 09:55:37] ...trap/engines/engine.py.230: loss: 0.46787 [I 2022-10-26 09:55:37] ...optimizers/trijoint.py.151: total_norm: 5.665129 [I 2022-10-26 09:55:42] ...trap/engines/engine.py.224: train: epoch 1 | batch 160/2304 [I 2022-10-26 09:55:42] ...trap/engines/engine.py.228: elapsed: 0:01:24 | left: 0:17:56 [I 2022-10-26 09:55:42] ...trap/engines/engine.py.229: process: 0.50850 | load: 0.00267 [I 2022-10-26 09:55:42] ...trap/engines/engine.py.230: loss: 0.47162 [I 2022-10-26 09:55:42] ...optimizers/trijoint.py.151: total_norm: 6.098891 [I 2022-10-26 09:55:47] ...trap/engines/engine.py.224: train: epoch 1 | batch 170/2304 [I 2022-10-26 09:55:47] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:17:44 [I 2022-10-26 09:55:47] ...trap/engines/engine.py.229: process: 0.50040 | load: 0.00270 [I 2022-10-26 09:55:47] ...trap/engines/engine.py.230: loss: 0.43586 [I 2022-10-26 09:55:47] ...optimizers/trijoint.py.151: total_norm: 4.157384 [I 2022-10-26 09:55:52] ...trap/engines/engine.py.224: train: epoch 1 | batch 180/2304 [I 2022-10-26 09:55:52] ...trap/engines/engine.py.228: elapsed: 0:01:34 | left: 0:18:16 [I 2022-10-26 09:55:52] ...trap/engines/engine.py.229: process: 0.59929 | load: 0.00259 [I 2022-10-26 09:55:52] ...trap/engines/engine.py.230: loss: 0.48444 [I 2022-10-26 09:55:52] ...optimizers/trijoint.py.151: total_norm: 5.277218 [I 2022-10-26 09:55:57] ...trap/engines/engine.py.224: train: epoch 1 | batch 190/2304 [I 2022-10-26 09:55:57] ...trap/engines/engine.py.228: elapsed: 0:01:39 | left: 0:17:37 [I 2022-10-26 09:55:57] ...trap/engines/engine.py.229: process: 0.48304 | load: 0.00258 [I 2022-10-26 09:55:57] ...trap/engines/engine.py.230: loss: 0.41441 [I 2022-10-26 09:55:57] ...optimizers/trijoint.py.151: total_norm: 5.396059 [I 2022-10-26 09:56:02] ...trap/engines/engine.py.224: train: epoch 1 | batch 200/2304 [I 2022-10-26 09:56:02] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:17:30 [I 2022-10-26 09:56:02] ...trap/engines/engine.py.229: process: 0.49999 | load: 0.00262 [I 2022-10-26 09:56:02] ...trap/engines/engine.py.230: loss: 0.52402 [I 2022-10-26 09:56:02] ...optimizers/trijoint.py.151: total_norm: 3.896844 [I 2022-10-26 09:56:07] ...trap/engines/engine.py.224: train: epoch 1 | batch 210/2304 [I 2022-10-26 09:56:07] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:17:33 [I 2022-10-26 09:56:07] ...trap/engines/engine.py.229: process: 0.51850 | load: 0.00272 [I 2022-10-26 09:56:07] ...trap/engines/engine.py.230: loss: 0.43272 [I 2022-10-26 09:56:07] ...optimizers/trijoint.py.151: total_norm: 4.502170 [I 2022-10-26 09:56:12] ...trap/engines/engine.py.224: train: epoch 1 | batch 220/2304 [I 2022-10-26 09:56:12] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:17:34 [I 2022-10-26 09:56:12] ...trap/engines/engine.py.229: process: 0.51699 | load: 0.00261 [I 2022-10-26 09:56:12] ...trap/engines/engine.py.230: loss: 0.52996 [I 2022-10-26 09:56:12] ...optimizers/trijoint.py.151: total_norm: 5.457092 [I 2022-10-26 09:56:17] ...trap/engines/engine.py.224: train: epoch 1 | batch 230/2304 [I 2022-10-26 09:56:17] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:17:39 [I 2022-10-26 09:56:17] ...trap/engines/engine.py.229: process: 0.50861 | load: 0.00261 [I 2022-10-26 09:56:17] ...trap/engines/engine.py.230: loss: 0.53500 [I 2022-10-26 09:56:17] ...optimizers/trijoint.py.151: total_norm: 6.121262 [I 2022-10-26 09:56:22] ...trap/engines/engine.py.224: train: epoch 1 | batch 240/2304 [I 2022-10-26 09:56:22] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:17:27 [I 2022-10-26 09:56:22] ...trap/engines/engine.py.229: process: 0.52013 | load: 0.00257 [I 2022-10-26 09:56:22] ...trap/engines/engine.py.230: loss: 0.44026 [I 2022-10-26 09:56:22] ...optimizers/trijoint.py.151: total_norm: 4.594090 [I 2022-10-26 09:56:27] ...trap/engines/engine.py.224: train: epoch 1 | batch 250/2304 [I 2022-10-26 09:56:27] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:17:17 [I 2022-10-26 09:56:27] ...trap/engines/engine.py.229: process: 0.50464 | load: 0.00273 [I 2022-10-26 09:56:27] ...trap/engines/engine.py.230: loss: 0.39411 [I 2022-10-26 09:56:27] ...optimizers/trijoint.py.151: total_norm: 5.226319 [I 2022-10-26 09:56:33] ...trap/engines/engine.py.224: train: epoch 1 | batch 260/2304 [I 2022-10-26 09:56:33] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:17:25 [I 2022-10-26 09:56:33] ...trap/engines/engine.py.229: process: 0.51752 | load: 0.00246 [I 2022-10-26 09:56:33] ...trap/engines/engine.py.230: loss: 0.38607 [I 2022-10-26 09:56:33] ...optimizers/trijoint.py.151: total_norm: 4.391892 [I 2022-10-26 09:56:38] ...trap/engines/engine.py.224: train: epoch 1 | batch 270/2304 [I 2022-10-26 09:56:38] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:16:53 [I 2022-10-26 09:56:38] ...trap/engines/engine.py.229: process: 0.48534 | load: 0.00255 [I 2022-10-26 09:56:38] ...trap/engines/engine.py.230: loss: 0.39508 [I 2022-10-26 09:56:38] ...optimizers/trijoint.py.151: total_norm: 5.723965 [I 2022-10-26 09:56:43] ...trap/engines/engine.py.224: train: epoch 1 | batch 280/2304 [I 2022-10-26 09:56:43] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:17:04 [I 2022-10-26 09:56:43] ...trap/engines/engine.py.229: process: 0.49941 | load: 0.00244 [I 2022-10-26 09:56:43] ...trap/engines/engine.py.230: loss: 0.40578 [I 2022-10-26 09:56:43] ...optimizers/trijoint.py.151: total_norm: 6.048240 [I 2022-10-26 09:56:48] ...trap/engines/engine.py.224: train: epoch 1 | batch 290/2304 [I 2022-10-26 09:56:48] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:17:20 [I 2022-10-26 09:56:48] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00243 [I 2022-10-26 09:56:48] ...trap/engines/engine.py.230: loss: 0.43253 [I 2022-10-26 09:56:48] ...optimizers/trijoint.py.151: total_norm: 4.352730 [I 2022-10-26 09:56:53] ...trap/engines/engine.py.224: train: epoch 1 | batch 300/2304 [I 2022-10-26 09:56:53] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:17:01 [I 2022-10-26 09:56:53] ...trap/engines/engine.py.229: process: 0.51857 | load: 0.00246 [I 2022-10-26 09:56:53] ...trap/engines/engine.py.230: loss: 0.44500 [I 2022-10-26 09:56:53] ...optimizers/trijoint.py.151: total_norm: 4.626413 [I 2022-10-26 09:56:58] ...trap/engines/engine.py.224: train: epoch 1 | batch 310/2304 [I 2022-10-26 09:56:58] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:16:55 [I 2022-10-26 09:56:58] ...trap/engines/engine.py.229: process: 0.50014 | load: 0.00252 [I 2022-10-26 09:56:58] ...trap/engines/engine.py.230: loss: 0.41765 [I 2022-10-26 09:56:58] ...optimizers/trijoint.py.151: total_norm: 5.496317 [I 2022-10-26 09:57:03] ...trap/engines/engine.py.224: train: epoch 1 | batch 320/2304 [I 2022-10-26 09:57:03] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:16:48 [I 2022-10-26 09:57:03] ...trap/engines/engine.py.229: process: 0.50168 | load: 0.00254 [I 2022-10-26 09:57:03] ...trap/engines/engine.py.230: loss: 0.46445 [I 2022-10-26 09:57:03] ...optimizers/trijoint.py.151: total_norm: 6.148548 [I 2022-10-26 09:57:08] ...trap/engines/engine.py.224: train: epoch 1 | batch 330/2304 [I 2022-10-26 09:57:08] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:16:43 [I 2022-10-26 09:57:08] ...trap/engines/engine.py.229: process: 0.50546 | load: 0.00260 [I 2022-10-26 09:57:08] ...trap/engines/engine.py.230: loss: 0.42845 [I 2022-10-26 09:57:08] ...optimizers/trijoint.py.151: total_norm: 4.532191 [I 2022-10-26 09:57:13] ...trap/engines/engine.py.224: train: epoch 1 | batch 340/2304 [I 2022-10-26 09:57:13] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:16:28 [I 2022-10-26 09:57:13] ...trap/engines/engine.py.229: process: 0.50297 | load: 0.00253 [I 2022-10-26 09:57:13] ...trap/engines/engine.py.230: loss: 0.37913 [I 2022-10-26 09:57:13] ...optimizers/trijoint.py.151: total_norm: 4.355447 [I 2022-10-26 09:57:18] ...trap/engines/engine.py.224: train: epoch 1 | batch 350/2304 [I 2022-10-26 09:57:18] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:16:25 [I 2022-10-26 09:57:18] ...trap/engines/engine.py.229: process: 0.49436 | load: 0.00246 [I 2022-10-26 09:57:18] ...trap/engines/engine.py.230: loss: 0.48428 [I 2022-10-26 09:57:18] ...optimizers/trijoint.py.151: total_norm: 5.625874 [I 2022-10-26 09:57:23] ...trap/engines/engine.py.224: train: epoch 1 | batch 360/2304 [I 2022-10-26 09:57:23] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 0:16:18 [I 2022-10-26 09:57:23] ...trap/engines/engine.py.229: process: 0.50871 | load: 0.00248 [I 2022-10-26 09:57:23] ...trap/engines/engine.py.230: loss: 0.42147 [I 2022-10-26 09:57:23] ...optimizers/trijoint.py.151: total_norm: 5.059202 [I 2022-10-26 09:57:28] ...trap/engines/engine.py.224: train: epoch 1 | batch 370/2304 [I 2022-10-26 09:57:28] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:16:06 [I 2022-10-26 09:57:28] ...trap/engines/engine.py.229: process: 0.49634 | load: 0.00244 [I 2022-10-26 09:57:28] ...trap/engines/engine.py.230: loss: 0.42904 [I 2022-10-26 09:57:28] ...optimizers/trijoint.py.151: total_norm: 6.547608 [I 2022-10-26 09:57:33] ...trap/engines/engine.py.224: train: epoch 1 | batch 380/2304 [I 2022-10-26 09:57:33] ...trap/engines/engine.py.228: elapsed: 0:03:15 | left: 0:15:58 [I 2022-10-26 09:57:33] ...trap/engines/engine.py.229: process: 0.49016 | load: 0.00241 [I 2022-10-26 09:57:33] ...trap/engines/engine.py.230: loss: 0.48197 [I 2022-10-26 09:57:33] ...optimizers/trijoint.py.151: total_norm: 4.454890 [I 2022-10-26 09:57:39] ...trap/engines/engine.py.224: train: epoch 1 | batch 390/2304 [I 2022-10-26 09:57:39] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:16:07 [I 2022-10-26 09:57:39] ...trap/engines/engine.py.229: process: 0.50043 | load: 0.00249 [I 2022-10-26 09:57:39] ...trap/engines/engine.py.230: loss: 0.35712 [I 2022-10-26 09:57:39] ...optimizers/trijoint.py.151: total_norm: 3.912765 [I 2022-10-26 09:57:43] ...trap/engines/engine.py.224: train: epoch 1 | batch 400/2304 [I 2022-10-26 09:57:43] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:15:49 [I 2022-10-26 09:57:43] ...trap/engines/engine.py.229: process: 0.50775 | load: 0.00251 [I 2022-10-26 09:57:43] ...trap/engines/engine.py.230: loss: 0.46834 [I 2022-10-26 09:57:43] ...optimizers/trijoint.py.151: total_norm: 3.919188 [I 2022-10-26 09:57:48] ...trap/engines/engine.py.224: train: epoch 1 | batch 410/2304 [I 2022-10-26 09:57:48] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:15:40 [I 2022-10-26 09:57:48] ...trap/engines/engine.py.229: process: 0.48160 | load: 0.00252 [I 2022-10-26 09:57:48] ...trap/engines/engine.py.230: loss: 0.46379 [I 2022-10-26 09:57:48] ...optimizers/trijoint.py.151: total_norm: 5.108959 [I 2022-10-26 09:57:53] ...trap/engines/engine.py.224: train: epoch 1 | batch 420/2304 [I 2022-10-26 09:57:53] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:15:34 [I 2022-10-26 09:57:53] ...trap/engines/engine.py.229: process: 0.49332 | load: 0.00258 [I 2022-10-26 09:57:53] ...trap/engines/engine.py.230: loss: 0.54170 [I 2022-10-26 09:57:53] ...optimizers/trijoint.py.151: total_norm: 5.975724 [I 2022-10-26 09:57:58] ...trap/engines/engine.py.224: train: epoch 1 | batch 430/2304 [I 2022-10-26 09:57:58] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:15:32 [I 2022-10-26 09:57:58] ...trap/engines/engine.py.229: process: 0.49776 | load: 0.00253 [I 2022-10-26 09:57:58] ...trap/engines/engine.py.230: loss: 0.41881 [I 2022-10-26 09:57:58] ...optimizers/trijoint.py.151: total_norm: 5.038229 [I 2022-10-26 09:58:03] ...trap/engines/engine.py.224: train: epoch 1 | batch 440/2304 [I 2022-10-26 09:58:03] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:15:33 [I 2022-10-26 09:58:03] ...trap/engines/engine.py.229: process: 0.49873 | load: 0.00253 [I 2022-10-26 09:58:03] ...trap/engines/engine.py.230: loss: 0.43335 [I 2022-10-26 09:58:03] ...optimizers/trijoint.py.151: total_norm: 4.307209 [I 2022-10-26 09:58:08] ...trap/engines/engine.py.224: train: epoch 1 | batch 450/2304 [I 2022-10-26 09:58:08] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:15:25 [I 2022-10-26 09:58:08] ...trap/engines/engine.py.229: process: 0.50578 | load: 0.00242 [I 2022-10-26 09:58:08] ...trap/engines/engine.py.230: loss: 0.44894 [I 2022-10-26 09:58:08] ...optimizers/trijoint.py.151: total_norm: 4.793467 [I 2022-10-26 09:58:13] ...trap/engines/engine.py.224: train: epoch 1 | batch 460/2304 [I 2022-10-26 09:58:14] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:15:28 [I 2022-10-26 09:58:14] ...trap/engines/engine.py.229: process: 0.50801 | load: 0.00243 [I 2022-10-26 09:58:14] ...trap/engines/engine.py.230: loss: 0.45523 [I 2022-10-26 09:58:14] ...optimizers/trijoint.py.151: total_norm: 6.221380 [I 2022-10-26 09:58:19] ...trap/engines/engine.py.224: train: epoch 1 | batch 470/2304 [I 2022-10-26 09:58:19] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:15:15 [I 2022-10-26 09:58:19] ...trap/engines/engine.py.229: process: 0.49297 | load: 0.00250 [I 2022-10-26 09:58:19] ...trap/engines/engine.py.230: loss: 0.44972 [I 2022-10-26 09:58:19] ...optimizers/trijoint.py.151: total_norm: 4.825301 [I 2022-10-26 09:58:24] ...trap/engines/engine.py.224: train: epoch 1 | batch 480/2304 [I 2022-10-26 09:58:24] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:15:24 [I 2022-10-26 09:58:24] ...trap/engines/engine.py.229: process: 0.50459 | load: 0.00248 [I 2022-10-26 09:58:24] ...trap/engines/engine.py.230: loss: 0.38475 [I 2022-10-26 09:58:24] ...optimizers/trijoint.py.151: total_norm: 4.164514 [I 2022-10-26 09:58:29] ...trap/engines/engine.py.224: train: epoch 1 | batch 490/2304 [I 2022-10-26 09:58:29] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:15:21 [I 2022-10-26 09:58:29] ...trap/engines/engine.py.229: process: 0.49852 | load: 0.00254 [I 2022-10-26 09:58:29] ...trap/engines/engine.py.230: loss: 0.54788 [I 2022-10-26 09:58:29] ...optimizers/trijoint.py.151: total_norm: 5.190072 [I 2022-10-26 09:58:34] ...trap/engines/engine.py.224: train: epoch 1 | batch 500/2304 [I 2022-10-26 09:58:34] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:15:07 [I 2022-10-26 09:58:34] ...trap/engines/engine.py.229: process: 0.50406 | load: 0.00250 [I 2022-10-26 09:58:34] ...trap/engines/engine.py.230: loss: 0.39297 [I 2022-10-26 09:58:34] ...optimizers/trijoint.py.151: total_norm: 3.711251 [I 2022-10-26 09:58:39] ...trap/engines/engine.py.224: train: epoch 1 | batch 510/2304 [I 2022-10-26 09:58:39] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:14:58 [I 2022-10-26 09:58:39] ...trap/engines/engine.py.229: process: 0.51118 | load: 0.00249 [I 2022-10-26 09:58:39] ...trap/engines/engine.py.230: loss: 0.42058 [I 2022-10-26 09:58:39] ...optimizers/trijoint.py.151: total_norm: 4.163708 [I 2022-10-26 09:58:44] ...trap/engines/engine.py.224: train: epoch 1 | batch 520/2304 [I 2022-10-26 09:58:44] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:14:51 [I 2022-10-26 09:58:44] ...trap/engines/engine.py.229: process: 0.51099 | load: 0.00250 [I 2022-10-26 09:58:44] ...trap/engines/engine.py.230: loss: 0.39323 [I 2022-10-26 09:58:44] ...optimizers/trijoint.py.151: total_norm: 5.775640 [I 2022-10-26 09:58:49] ...trap/engines/engine.py.224: train: epoch 1 | batch 530/2304 [I 2022-10-26 09:58:49] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:15:14 [I 2022-10-26 09:58:49] ...trap/engines/engine.py.229: process: 0.50235 | load: 0.00243 [I 2022-10-26 09:58:49] ...trap/engines/engine.py.230: loss: 0.33064 [I 2022-10-26 09:58:49] ...optimizers/trijoint.py.151: total_norm: 4.791600 [I 2022-10-26 09:58:54] ...trap/engines/engine.py.224: train: epoch 1 | batch 540/2304 [I 2022-10-26 09:58:54] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:14:47 [I 2022-10-26 09:58:54] ...trap/engines/engine.py.229: process: 0.50817 | load: 0.00250 [I 2022-10-26 09:58:54] ...trap/engines/engine.py.230: loss: 0.41098 [I 2022-10-26 09:58:54] ...optimizers/trijoint.py.151: total_norm: 4.068395 [I 2022-10-26 09:58:59] ...trap/engines/engine.py.224: train: epoch 1 | batch 550/2304 [I 2022-10-26 09:58:59] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:14:30 [I 2022-10-26 09:58:59] ...trap/engines/engine.py.229: process: 0.50764 | load: 0.00256 [I 2022-10-26 09:58:59] ...trap/engines/engine.py.230: loss: 0.42728 [I 2022-10-26 09:58:59] ...optimizers/trijoint.py.151: total_norm: 4.167441 [I 2022-10-26 09:59:04] ...trap/engines/engine.py.224: train: epoch 1 | batch 560/2304 [I 2022-10-26 09:59:04] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:14:29 [I 2022-10-26 09:59:04] ...trap/engines/engine.py.229: process: 0.49777 | load: 0.00246 [I 2022-10-26 09:59:04] ...trap/engines/engine.py.230: loss: 0.42619 [I 2022-10-26 09:59:04] ...optimizers/trijoint.py.151: total_norm: 4.557531 [I 2022-10-26 09:59:09] ...trap/engines/engine.py.224: train: epoch 1 | batch 570/2304 [I 2022-10-26 09:59:09] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:14:21 [I 2022-10-26 09:59:09] ...trap/engines/engine.py.229: process: 0.49158 | load: 0.00249 [I 2022-10-26 09:59:09] ...trap/engines/engine.py.230: loss: 0.35090 [I 2022-10-26 09:59:09] ...optimizers/trijoint.py.151: total_norm: 3.578034 [I 2022-10-26 09:59:14] ...trap/engines/engine.py.224: train: epoch 1 | batch 580/2304 [I 2022-10-26 09:59:14] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:14:35 [I 2022-10-26 09:59:14] ...trap/engines/engine.py.229: process: 0.49846 | load: 0.00245 [I 2022-10-26 09:59:14] ...trap/engines/engine.py.230: loss: 0.44850 [I 2022-10-26 09:59:14] ...optimizers/trijoint.py.151: total_norm: 5.550544 [I 2022-10-26 09:59:19] ...trap/engines/engine.py.224: train: epoch 1 | batch 590/2304 [I 2022-10-26 09:59:19] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:14:24 [I 2022-10-26 09:59:19] ...trap/engines/engine.py.229: process: 0.51376 | load: 0.00249 [I 2022-10-26 09:59:19] ...trap/engines/engine.py.230: loss: 0.42825 [I 2022-10-26 09:59:19] ...optimizers/trijoint.py.151: total_norm: 5.496658 [I 2022-10-26 09:59:24] ...trap/engines/engine.py.224: train: epoch 1 | batch 600/2304 [I 2022-10-26 09:59:24] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:14:15 [I 2022-10-26 09:59:24] ...trap/engines/engine.py.229: process: 0.50488 | load: 0.00247 [I 2022-10-26 09:59:24] ...trap/engines/engine.py.230: loss: 0.34220 [I 2022-10-26 09:59:24] ...optimizers/trijoint.py.151: total_norm: 4.037503 [I 2022-10-26 09:59:29] ...trap/engines/engine.py.224: train: epoch 1 | batch 610/2304 [I 2022-10-26 09:59:29] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:14:04 [I 2022-10-26 09:59:29] ...trap/engines/engine.py.229: process: 0.50431 | load: 0.00242 [I 2022-10-26 09:59:29] ...trap/engines/engine.py.230: loss: 0.49692 [I 2022-10-26 09:59:29] ...optimizers/trijoint.py.151: total_norm: 5.284312 [I 2022-10-26 09:59:34] ...trap/engines/engine.py.224: train: epoch 1 | batch 620/2304 [I 2022-10-26 09:59:34] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:13:55 [I 2022-10-26 09:59:34] ...trap/engines/engine.py.229: process: 0.48545 | load: 0.00269 [I 2022-10-26 09:59:34] ...trap/engines/engine.py.230: loss: 0.40507 [I 2022-10-26 09:59:34] ...optimizers/trijoint.py.151: total_norm: 3.872633 [I 2022-10-26 09:59:39] ...trap/engines/engine.py.224: train: epoch 1 | batch 630/2304 [I 2022-10-26 09:59:39] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:13:47 [I 2022-10-26 09:59:39] ...trap/engines/engine.py.229: process: 0.48920 | load: 0.00243 [I 2022-10-26 09:59:39] ...trap/engines/engine.py.230: loss: 0.46816 [I 2022-10-26 09:59:39] ...optimizers/trijoint.py.151: total_norm: 5.502182 [I 2022-10-26 09:59:44] ...trap/engines/engine.py.224: train: epoch 1 | batch 640/2304 [I 2022-10-26 09:59:44] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:13:49 [I 2022-10-26 09:59:44] ...trap/engines/engine.py.229: process: 0.50108 | load: 0.00246 [I 2022-10-26 09:59:44] ...trap/engines/engine.py.230: loss: 0.43962 [I 2022-10-26 09:59:44] ...optimizers/trijoint.py.151: total_norm: 5.848121 [I 2022-10-26 09:59:49] ...trap/engines/engine.py.224: train: epoch 1 | batch 650/2304 [I 2022-10-26 09:59:49] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:13:43 [I 2022-10-26 09:59:49] ...trap/engines/engine.py.229: process: 0.49818 | load: 0.00243 [I 2022-10-26 09:59:49] ...trap/engines/engine.py.230: loss: 0.39535 [I 2022-10-26 09:59:49] ...optimizers/trijoint.py.151: total_norm: 4.534308 [I 2022-10-26 09:59:54] ...trap/engines/engine.py.224: train: epoch 1 | batch 660/2304 [I 2022-10-26 09:59:54] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:14:19 [I 2022-10-26 09:59:54] ...trap/engines/engine.py.229: process: 0.56309 | load: 0.00276 [I 2022-10-26 09:59:54] ...trap/engines/engine.py.230: loss: 0.50422 [I 2022-10-26 09:59:54] ...optimizers/trijoint.py.151: total_norm: 5.119494 [I 2022-10-26 09:59:59] ...trap/engines/engine.py.224: train: epoch 1 | batch 670/2304 [I 2022-10-26 09:59:59] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:13:36 [I 2022-10-26 09:59:59] ...trap/engines/engine.py.229: process: 0.49809 | load: 0.00242 [I 2022-10-26 09:59:59] ...trap/engines/engine.py.230: loss: 0.44435 [I 2022-10-26 09:59:59] ...optimizers/trijoint.py.151: total_norm: 5.346173 [I 2022-10-26 10:00:04] ...trap/engines/engine.py.224: train: epoch 1 | batch 680/2304 [I 2022-10-26 10:00:04] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:13:33 [I 2022-10-26 10:00:04] ...trap/engines/engine.py.229: process: 0.50998 | load: 0.00242 [I 2022-10-26 10:00:04] ...trap/engines/engine.py.230: loss: 0.46784 [I 2022-10-26 10:00:04] ...optimizers/trijoint.py.151: total_norm: 6.373725 [I 2022-10-26 10:00:09] ...trap/engines/engine.py.224: train: epoch 1 | batch 690/2304 [I 2022-10-26 10:00:09] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:13:29 [I 2022-10-26 10:00:09] ...trap/engines/engine.py.229: process: 0.49784 | load: 0.00251 [I 2022-10-26 10:00:09] ...trap/engines/engine.py.230: loss: 0.31023 [I 2022-10-26 10:00:09] ...optimizers/trijoint.py.151: total_norm: 5.067394 [I 2022-10-26 10:00:14] ...trap/engines/engine.py.224: train: epoch 1 | batch 700/2304 [I 2022-10-26 10:00:14] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:13:22 [I 2022-10-26 10:00:14] ...trap/engines/engine.py.229: process: 0.49735 | load: 0.00245 [I 2022-10-26 10:00:14] ...trap/engines/engine.py.230: loss: 0.34904 [I 2022-10-26 10:00:14] ...optimizers/trijoint.py.151: total_norm: 4.882132 [I 2022-10-26 10:00:19] ...trap/engines/engine.py.224: train: epoch 1 | batch 710/2304 [I 2022-10-26 10:00:19] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:13:18 [I 2022-10-26 10:00:19] ...trap/engines/engine.py.229: process: 0.48219 | load: 0.00243 [I 2022-10-26 10:00:19] ...trap/engines/engine.py.230: loss: 0.36307 [I 2022-10-26 10:00:19] ...optimizers/trijoint.py.151: total_norm: 4.927912 [I 2022-10-26 10:00:24] ...trap/engines/engine.py.224: train: epoch 1 | batch 720/2304 [I 2022-10-26 10:00:24] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:13:08 [I 2022-10-26 10:00:24] ...trap/engines/engine.py.229: process: 0.50073 | load: 0.00247 [I 2022-10-26 10:00:24] ...trap/engines/engine.py.230: loss: 0.39199 [I 2022-10-26 10:00:24] ...optimizers/trijoint.py.151: total_norm: 4.139427 [I 2022-10-26 10:00:29] ...trap/engines/engine.py.224: train: epoch 1 | batch 730/2304 [I 2022-10-26 10:00:29] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:13:13 [I 2022-10-26 10:00:29] ...trap/engines/engine.py.229: process: 0.50865 | load: 0.00247 [I 2022-10-26 10:00:29] ...trap/engines/engine.py.230: loss: 0.31238 [I 2022-10-26 10:00:29] ...optimizers/trijoint.py.151: total_norm: 4.141325 [I 2022-10-26 10:00:34] ...trap/engines/engine.py.224: train: epoch 1 | batch 740/2304 [I 2022-10-26 10:00:34] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:13:01 [I 2022-10-26 10:00:34] ...trap/engines/engine.py.229: process: 0.49623 | load: 0.00252 [I 2022-10-26 10:00:34] ...trap/engines/engine.py.230: loss: 0.49309 [I 2022-10-26 10:00:34] ...optimizers/trijoint.py.151: total_norm: 6.991801 [I 2022-10-26 10:00:39] ...trap/engines/engine.py.224: train: epoch 1 | batch 750/2304 [I 2022-10-26 10:00:39] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:13:13 [I 2022-10-26 10:00:39] ...trap/engines/engine.py.229: process: 0.52133 | load: 0.00247 [I 2022-10-26 10:00:39] ...trap/engines/engine.py.230: loss: 0.33171 [I 2022-10-26 10:00:39] ...optimizers/trijoint.py.151: total_norm: 4.283844 [I 2022-10-26 10:00:44] ...trap/engines/engine.py.224: train: epoch 1 | batch 760/2304 [I 2022-10-26 10:00:44] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:12:57 [I 2022-10-26 10:00:44] ...trap/engines/engine.py.229: process: 0.49584 | load: 0.00247 [I 2022-10-26 10:00:44] ...trap/engines/engine.py.230: loss: 0.46954 [I 2022-10-26 10:00:44] ...optimizers/trijoint.py.151: total_norm: 4.875780 [I 2022-10-26 10:00:49] ...trap/engines/engine.py.224: train: epoch 1 | batch 770/2304 [I 2022-10-26 10:00:49] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:12:53 [I 2022-10-26 10:00:49] ...trap/engines/engine.py.229: process: 0.50337 | load: 0.00253 [I 2022-10-26 10:00:49] ...trap/engines/engine.py.230: loss: 0.40693 [I 2022-10-26 10:00:49] ...optimizers/trijoint.py.151: total_norm: 4.608083 [I 2022-10-26 10:00:54] ...trap/engines/engine.py.224: train: epoch 1 | batch 780/2304 [I 2022-10-26 10:00:54] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:12:45 [I 2022-10-26 10:00:54] ...trap/engines/engine.py.229: process: 0.50070 | load: 0.00252 [I 2022-10-26 10:00:54] ...trap/engines/engine.py.230: loss: 0.41012 [I 2022-10-26 10:00:54] ...optimizers/trijoint.py.151: total_norm: 5.437197 [I 2022-10-26 10:00:59] ...trap/engines/engine.py.224: train: epoch 1 | batch 790/2304 [I 2022-10-26 10:00:59] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:12:36 [I 2022-10-26 10:00:59] ...trap/engines/engine.py.229: process: 0.48969 | load: 0.00246 [I 2022-10-26 10:00:59] ...trap/engines/engine.py.230: loss: 0.37723 [I 2022-10-26 10:00:59] ...optimizers/trijoint.py.151: total_norm: 4.603388 [I 2022-10-26 10:01:04] ...trap/engines/engine.py.224: train: epoch 1 | batch 800/2304 [I 2022-10-26 10:01:04] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:12:44 [I 2022-10-26 10:01:04] ...trap/engines/engine.py.229: process: 0.51089 | load: 0.00245 [I 2022-10-26 10:01:04] ...trap/engines/engine.py.230: loss: 0.47916 [I 2022-10-26 10:01:04] ...optimizers/trijoint.py.151: total_norm: 5.021913 [I 2022-10-26 10:01:09] ...trap/engines/engine.py.224: train: epoch 1 | batch 810/2304 [I 2022-10-26 10:01:09] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:12:22 [I 2022-10-26 10:01:09] ...trap/engines/engine.py.229: process: 0.49532 | load: 0.00245 [I 2022-10-26 10:01:09] ...trap/engines/engine.py.230: loss: 0.42777 [I 2022-10-26 10:01:09] ...optimizers/trijoint.py.151: total_norm: 3.920892 [I 2022-10-26 10:01:14] ...trap/engines/engine.py.224: train: epoch 1 | batch 820/2304 [I 2022-10-26 10:01:14] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:12:21 [I 2022-10-26 10:01:14] ...trap/engines/engine.py.229: process: 0.49039 | load: 0.00240 [I 2022-10-26 10:01:14] ...trap/engines/engine.py.230: loss: 0.33685 [I 2022-10-26 10:01:14] ...optimizers/trijoint.py.151: total_norm: 4.432399 [I 2022-10-26 10:01:19] ...trap/engines/engine.py.224: train: epoch 1 | batch 830/2304 [I 2022-10-26 10:01:19] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:12:17 [I 2022-10-26 10:01:19] ...trap/engines/engine.py.229: process: 0.50380 | load: 0.00243 [I 2022-10-26 10:01:19] ...trap/engines/engine.py.230: loss: 0.35176 [I 2022-10-26 10:01:19] ...optimizers/trijoint.py.151: total_norm: 4.202267 [I 2022-10-26 10:01:24] ...trap/engines/engine.py.224: train: epoch 1 | batch 840/2304 [I 2022-10-26 10:01:24] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:12:23 [I 2022-10-26 10:01:24] ...trap/engines/engine.py.229: process: 0.51815 | load: 0.00252 [I 2022-10-26 10:01:24] ...trap/engines/engine.py.230: loss: 0.41814 [I 2022-10-26 10:01:24] ...optimizers/trijoint.py.151: total_norm: 4.535090 [I 2022-10-26 10:01:29] ...trap/engines/engine.py.224: train: epoch 1 | batch 850/2304 [I 2022-10-26 10:01:29] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:12:05 [I 2022-10-26 10:01:29] ...trap/engines/engine.py.229: process: 0.50268 | load: 0.00252 [I 2022-10-26 10:01:29] ...trap/engines/engine.py.230: loss: 0.34328 [I 2022-10-26 10:01:29] ...optimizers/trijoint.py.151: total_norm: 4.536149 [I 2022-10-26 10:01:35] ...trap/engines/engine.py.224: train: epoch 1 | batch 860/2304 [I 2022-10-26 10:01:35] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:12:06 [I 2022-10-26 10:01:35] ...trap/engines/engine.py.229: process: 0.51691 | load: 0.00249 [I 2022-10-26 10:01:35] ...trap/engines/engine.py.230: loss: 0.48249 [I 2022-10-26 10:01:35] ...optimizers/trijoint.py.151: total_norm: 4.195716 [I 2022-10-26 10:01:40] ...trap/engines/engine.py.224: train: epoch 1 | batch 870/2304 [I 2022-10-26 10:01:40] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:12:05 [I 2022-10-26 10:01:40] ...trap/engines/engine.py.229: process: 0.52592 | load: 0.00248 [I 2022-10-26 10:01:40] ...trap/engines/engine.py.230: loss: 0.33255 [I 2022-10-26 10:01:40] ...optimizers/trijoint.py.151: total_norm: 3.552290 [I 2022-10-26 10:01:45] ...trap/engines/engine.py.224: train: epoch 1 | batch 880/2304 [I 2022-10-26 10:01:45] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:11:54 [I 2022-10-26 10:01:45] ...trap/engines/engine.py.229: process: 0.50077 | load: 0.00251 [I 2022-10-26 10:01:45] ...trap/engines/engine.py.230: loss: 0.37377 [I 2022-10-26 10:01:45] ...optimizers/trijoint.py.151: total_norm: 4.071578 [I 2022-10-26 10:01:50] ...trap/engines/engine.py.224: train: epoch 1 | batch 890/2304 [I 2022-10-26 10:01:50] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:11:45 [I 2022-10-26 10:01:50] ...trap/engines/engine.py.229: process: 0.49496 | load: 0.00246 [I 2022-10-26 10:01:50] ...trap/engines/engine.py.230: loss: 0.49031 [I 2022-10-26 10:01:50] ...optimizers/trijoint.py.151: total_norm: 4.895243 [I 2022-10-26 10:01:55] ...trap/engines/engine.py.224: train: epoch 1 | batch 900/2304 [I 2022-10-26 10:01:55] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:11:48 [I 2022-10-26 10:01:55] ...trap/engines/engine.py.229: process: 0.50958 | load: 0.00245 [I 2022-10-26 10:01:55] ...trap/engines/engine.py.230: loss: 0.50520 [I 2022-10-26 10:01:55] ...optimizers/trijoint.py.151: total_norm: 5.140042 [I 2022-10-26 10:02:00] ...trap/engines/engine.py.224: train: epoch 1 | batch 910/2304 [I 2022-10-26 10:02:00] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:11:36 [I 2022-10-26 10:02:00] ...trap/engines/engine.py.229: process: 0.49829 | load: 0.00250 [I 2022-10-26 10:02:00] ...trap/engines/engine.py.230: loss: 0.41164 [I 2022-10-26 10:02:00] ...optimizers/trijoint.py.151: total_norm: 5.226466 [I 2022-10-26 10:02:05] ...trap/engines/engine.py.224: train: epoch 1 | batch 920/2304 [I 2022-10-26 10:02:05] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:11:41 [I 2022-10-26 10:02:05] ...trap/engines/engine.py.229: process: 0.50130 | load: 0.00249 [I 2022-10-26 10:02:05] ...trap/engines/engine.py.230: loss: 0.39162 [I 2022-10-26 10:02:05] ...optimizers/trijoint.py.151: total_norm: 5.133657 [I 2022-10-26 10:02:10] ...trap/engines/engine.py.224: train: epoch 1 | batch 930/2304 [I 2022-10-26 10:02:10] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:11:33 [I 2022-10-26 10:02:10] ...trap/engines/engine.py.229: process: 0.49673 | load: 0.00254 [I 2022-10-26 10:02:10] ...trap/engines/engine.py.230: loss: 0.54729 [I 2022-10-26 10:02:10] ...optimizers/trijoint.py.151: total_norm: 4.812195 [I 2022-10-26 10:02:15] ...trap/engines/engine.py.224: train: epoch 1 | batch 940/2304 [I 2022-10-26 10:02:15] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:11:23 [I 2022-10-26 10:02:15] ...trap/engines/engine.py.229: process: 0.49835 | load: 0.00259 [I 2022-10-26 10:02:15] ...trap/engines/engine.py.230: loss: 0.50430 [I 2022-10-26 10:02:15] ...optimizers/trijoint.py.151: total_norm: 5.332110 [I 2022-10-26 10:02:20] ...trap/engines/engine.py.224: train: epoch 1 | batch 950/2304 [I 2022-10-26 10:02:20] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:11:16 [I 2022-10-26 10:02:20] ...trap/engines/engine.py.229: process: 0.49848 | load: 0.00254 [I 2022-10-26 10:02:20] ...trap/engines/engine.py.230: loss: 0.34519 [I 2022-10-26 10:02:20] ...optimizers/trijoint.py.151: total_norm: 4.190931 [I 2022-10-26 10:02:25] ...trap/engines/engine.py.224: train: epoch 1 | batch 960/2304 [I 2022-10-26 10:02:25] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:11:11 [I 2022-10-26 10:02:25] ...trap/engines/engine.py.229: process: 0.50878 | load: 0.00275 [I 2022-10-26 10:02:25] ...trap/engines/engine.py.230: loss: 0.49597 [I 2022-10-26 10:02:25] ...optimizers/trijoint.py.151: total_norm: 4.697143 [I 2022-10-26 10:02:30] ...trap/engines/engine.py.224: train: epoch 1 | batch 970/2304 [I 2022-10-26 10:02:30] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:11:14 [I 2022-10-26 10:02:30] ...trap/engines/engine.py.229: process: 0.51320 | load: 0.00253 [I 2022-10-26 10:02:30] ...trap/engines/engine.py.230: loss: 0.45224 [I 2022-10-26 10:02:30] ...optimizers/trijoint.py.151: total_norm: 5.784265 [I 2022-10-26 10:02:35] ...trap/engines/engine.py.224: train: epoch 1 | batch 980/2304 [I 2022-10-26 10:02:35] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:11:01 [I 2022-10-26 10:02:35] ...trap/engines/engine.py.229: process: 0.49880 | load: 0.00246 [I 2022-10-26 10:02:35] ...trap/engines/engine.py.230: loss: 0.53988 [I 2022-10-26 10:02:35] ...optimizers/trijoint.py.151: total_norm: 5.694337 [I 2022-10-26 10:02:40] ...trap/engines/engine.py.224: train: epoch 1 | batch 990/2304 [I 2022-10-26 10:02:40] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:10:57 [I 2022-10-26 10:02:40] ...trap/engines/engine.py.229: process: 0.50119 | load: 0.00250 [I 2022-10-26 10:02:40] ...trap/engines/engine.py.230: loss: 0.56559 [I 2022-10-26 10:02:40] ...optimizers/trijoint.py.151: total_norm: 5.823128 [I 2022-10-26 10:02:45] ...trap/engines/engine.py.224: train: epoch 1 | batch 1000/2304 [I 2022-10-26 10:02:45] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:10:54 [I 2022-10-26 10:02:45] ...trap/engines/engine.py.229: process: 0.51401 | load: 0.00249 [I 2022-10-26 10:02:45] ...trap/engines/engine.py.230: loss: 0.46857 [I 2022-10-26 10:02:45] ...optimizers/trijoint.py.151: total_norm: 5.138455 [I 2022-10-26 10:02:50] ...trap/engines/engine.py.224: train: epoch 1 | batch 1010/2304 [I 2022-10-26 10:02:50] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:10:51 [I 2022-10-26 10:02:50] ...trap/engines/engine.py.229: process: 0.49840 | load: 0.00246 [I 2022-10-26 10:02:50] ...trap/engines/engine.py.230: loss: 0.46191 [I 2022-10-26 10:02:50] ...optimizers/trijoint.py.151: total_norm: 5.062342 [I 2022-10-26 10:02:55] ...trap/engines/engine.py.224: train: epoch 1 | batch 1020/2304 [I 2022-10-26 10:02:55] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:10:52 [I 2022-10-26 10:02:55] ...trap/engines/engine.py.229: process: 0.50878 | load: 0.00254 [I 2022-10-26 10:02:55] ...trap/engines/engine.py.230: loss: 0.37892 [I 2022-10-26 10:02:55] ...optimizers/trijoint.py.151: total_norm: 4.043264 [I 2022-10-26 10:03:00] ...trap/engines/engine.py.224: train: epoch 1 | batch 1030/2304 [I 2022-10-26 10:03:00] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:10:34 [I 2022-10-26 10:03:00] ...trap/engines/engine.py.229: process: 0.48632 | load: 0.00248 [I 2022-10-26 10:03:00] ...trap/engines/engine.py.230: loss: 0.38051 [I 2022-10-26 10:03:00] ...optimizers/trijoint.py.151: total_norm: 5.067825 [I 2022-10-26 10:03:05] ...trap/engines/engine.py.224: train: epoch 1 | batch 1040/2304 [I 2022-10-26 10:03:05] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:10:30 [I 2022-10-26 10:03:05] ...trap/engines/engine.py.229: process: 0.50222 | load: 0.00246 [I 2022-10-26 10:03:05] ...trap/engines/engine.py.230: loss: 0.30847 [I 2022-10-26 10:03:05] ...optimizers/trijoint.py.151: total_norm: 4.425581 [I 2022-10-26 10:03:10] ...trap/engines/engine.py.224: train: epoch 1 | batch 1050/2304 [I 2022-10-26 10:03:10] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:10:28 [I 2022-10-26 10:03:10] ...trap/engines/engine.py.229: process: 0.49184 | load: 0.00246 [I 2022-10-26 10:03:10] ...trap/engines/engine.py.230: loss: 0.44110 [I 2022-10-26 10:03:10] ...optimizers/trijoint.py.151: total_norm: 4.942997 [I 2022-10-26 10:03:15] ...trap/engines/engine.py.224: train: epoch 1 | batch 1060/2304 [I 2022-10-26 10:03:15] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:10:20 [I 2022-10-26 10:03:15] ...trap/engines/engine.py.229: process: 0.49971 | load: 0.00248 [I 2022-10-26 10:03:15] ...trap/engines/engine.py.230: loss: 0.43999 [I 2022-10-26 10:03:15] ...optimizers/trijoint.py.151: total_norm: 5.417710 [I 2022-10-26 10:03:20] ...trap/engines/engine.py.224: train: epoch 1 | batch 1070/2304 [I 2022-10-26 10:03:20] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:10:11 [I 2022-10-26 10:03:20] ...trap/engines/engine.py.229: process: 0.49097 | load: 0.00265 [I 2022-10-26 10:03:20] ...trap/engines/engine.py.230: loss: 0.43833 [I 2022-10-26 10:03:20] ...optimizers/trijoint.py.151: total_norm: 4.594965 [I 2022-10-26 10:03:25] ...trap/engines/engine.py.224: train: epoch 1 | batch 1080/2304 [I 2022-10-26 10:03:25] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:10:09 [I 2022-10-26 10:03:25] ...trap/engines/engine.py.229: process: 0.49607 | load: 0.00254 [I 2022-10-26 10:03:25] ...trap/engines/engine.py.230: loss: 0.45873 [I 2022-10-26 10:03:25] ...optimizers/trijoint.py.151: total_norm: 4.510621 [I 2022-10-26 10:03:30] ...trap/engines/engine.py.224: train: epoch 1 | batch 1090/2304 [I 2022-10-26 10:03:30] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:10:09 [I 2022-10-26 10:03:30] ...trap/engines/engine.py.229: process: 0.48966 | load: 0.00253 [I 2022-10-26 10:03:30] ...trap/engines/engine.py.230: loss: 0.27049 [I 2022-10-26 10:03:30] ...optimizers/trijoint.py.151: total_norm: 4.158152 [I 2022-10-26 10:03:35] ...trap/engines/engine.py.224: train: epoch 1 | batch 1100/2304 [I 2022-10-26 10:03:35] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:10:07 [I 2022-10-26 10:03:35] ...trap/engines/engine.py.229: process: 0.49269 | load: 0.00270 [I 2022-10-26 10:03:35] ...trap/engines/engine.py.230: loss: 0.37577 [I 2022-10-26 10:03:35] ...optimizers/trijoint.py.151: total_norm: 4.866388 [I 2022-10-26 10:03:40] ...trap/engines/engine.py.224: train: epoch 1 | batch 1110/2304 [I 2022-10-26 10:03:40] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:09:53 [I 2022-10-26 10:03:40] ...trap/engines/engine.py.229: process: 0.49941 | load: 0.00274 [I 2022-10-26 10:03:40] ...trap/engines/engine.py.230: loss: 0.44453 [I 2022-10-26 10:03:40] ...optimizers/trijoint.py.151: total_norm: 4.607635 [I 2022-10-26 10:03:45] ...trap/engines/engine.py.224: train: epoch 1 | batch 1120/2304 [I 2022-10-26 10:03:45] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:09:52 [I 2022-10-26 10:03:45] ...trap/engines/engine.py.229: process: 0.50872 | load: 0.00248 [I 2022-10-26 10:03:45] ...trap/engines/engine.py.230: loss: 0.35970 [I 2022-10-26 10:03:45] ...optimizers/trijoint.py.151: total_norm: 4.601586 [I 2022-10-26 10:03:50] ...trap/engines/engine.py.224: train: epoch 1 | batch 1130/2304 [I 2022-10-26 10:03:50] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:09:45 [I 2022-10-26 10:03:50] ...trap/engines/engine.py.229: process: 0.50881 | load: 0.00245 [I 2022-10-26 10:03:50] ...trap/engines/engine.py.230: loss: 0.40801 [I 2022-10-26 10:03:50] ...optimizers/trijoint.py.151: total_norm: 4.178674 [I 2022-10-26 10:03:55] ...trap/engines/engine.py.224: train: epoch 1 | batch 1140/2304 [I 2022-10-26 10:03:55] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:09:45 [I 2022-10-26 10:03:55] ...trap/engines/engine.py.229: process: 0.49955 | load: 0.00247 [I 2022-10-26 10:03:55] ...trap/engines/engine.py.230: loss: 0.39285 [I 2022-10-26 10:03:55] ...optimizers/trijoint.py.151: total_norm: 3.995621 [I 2022-10-26 10:04:00] ...trap/engines/engine.py.224: train: epoch 1 | batch 1150/2304 [I 2022-10-26 10:04:00] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:09:40 [I 2022-10-26 10:04:00] ...trap/engines/engine.py.229: process: 0.49826 | load: 0.00249 [I 2022-10-26 10:04:00] ...trap/engines/engine.py.230: loss: 0.37677 [I 2022-10-26 10:04:00] ...optimizers/trijoint.py.151: total_norm: 4.509651 [I 2022-10-26 10:04:05] ...trap/engines/engine.py.224: train: epoch 1 | batch 1160/2304 [I 2022-10-26 10:04:05] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:09:46 [I 2022-10-26 10:04:05] ...trap/engines/engine.py.229: process: 0.51072 | load: 0.00247 [I 2022-10-26 10:04:05] ...trap/engines/engine.py.230: loss: 0.33169 [I 2022-10-26 10:04:05] ...optimizers/trijoint.py.151: total_norm: 3.716600 [I 2022-10-26 10:04:10] ...trap/engines/engine.py.224: train: epoch 1 | batch 1170/2304 [I 2022-10-26 10:04:10] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:09:26 [I 2022-10-26 10:04:10] ...trap/engines/engine.py.229: process: 0.49990 | load: 0.00246 [I 2022-10-26 10:04:10] ...trap/engines/engine.py.230: loss: 0.54861 [I 2022-10-26 10:04:10] ...optimizers/trijoint.py.151: total_norm: 5.532339 [I 2022-10-26 10:04:15] ...trap/engines/engine.py.224: train: epoch 1 | batch 1180/2304 [I 2022-10-26 10:04:15] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:09:17 [I 2022-10-26 10:04:15] ...trap/engines/engine.py.229: process: 0.48912 | load: 0.00249 [I 2022-10-26 10:04:15] ...trap/engines/engine.py.230: loss: 0.39537 [I 2022-10-26 10:04:15] ...optimizers/trijoint.py.151: total_norm: 3.972653 [I 2022-10-26 10:04:20] ...trap/engines/engine.py.224: train: epoch 1 | batch 1190/2304 [I 2022-10-26 10:04:20] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:09:19 [I 2022-10-26 10:04:20] ...trap/engines/engine.py.229: process: 0.50720 | load: 0.00254 [I 2022-10-26 10:04:20] ...trap/engines/engine.py.230: loss: 0.42812 [I 2022-10-26 10:04:20] ...optimizers/trijoint.py.151: total_norm: 4.847446 [I 2022-10-26 10:04:25] ...trap/engines/engine.py.224: train: epoch 1 | batch 1200/2304 [I 2022-10-26 10:04:25] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:09:17 [I 2022-10-26 10:04:25] ...trap/engines/engine.py.229: process: 0.50771 | load: 0.00294 [I 2022-10-26 10:04:25] ...trap/engines/engine.py.230: loss: 0.38008 [I 2022-10-26 10:04:25] ...optimizers/trijoint.py.151: total_norm: 4.600657 [I 2022-10-26 10:04:30] ...trap/engines/engine.py.224: train: epoch 1 | batch 1210/2304 [I 2022-10-26 10:04:30] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:09:12 [I 2022-10-26 10:04:30] ...trap/engines/engine.py.229: process: 0.49497 | load: 0.00246 [I 2022-10-26 10:04:30] ...trap/engines/engine.py.230: loss: 0.35396 [I 2022-10-26 10:04:30] ...optimizers/trijoint.py.151: total_norm: 4.763020 [I 2022-10-26 10:04:35] ...trap/engines/engine.py.224: train: epoch 1 | batch 1220/2304 [I 2022-10-26 10:04:35] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:09:03 [I 2022-10-26 10:04:35] ...trap/engines/engine.py.229: process: 0.50204 | load: 0.00246 [I 2022-10-26 10:04:35] ...trap/engines/engine.py.230: loss: 0.30333 [I 2022-10-26 10:04:35] ...optimizers/trijoint.py.151: total_norm: 3.766744 [I 2022-10-26 10:04:40] ...trap/engines/engine.py.224: train: epoch 1 | batch 1230/2304 [I 2022-10-26 10:04:40] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:08:57 [I 2022-10-26 10:04:40] ...trap/engines/engine.py.229: process: 0.49860 | load: 0.00245 [I 2022-10-26 10:04:40] ...trap/engines/engine.py.230: loss: 0.32804 [I 2022-10-26 10:04:40] ...optimizers/trijoint.py.151: total_norm: 4.432572 [I 2022-10-26 10:04:45] ...trap/engines/engine.py.224: train: epoch 1 | batch 1240/2304 [I 2022-10-26 10:04:45] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:08:56 [I 2022-10-26 10:04:45] ...trap/engines/engine.py.229: process: 0.49589 | load: 0.00250 [I 2022-10-26 10:04:45] ...trap/engines/engine.py.230: loss: 0.37628 [I 2022-10-26 10:04:45] ...optimizers/trijoint.py.151: total_norm: 4.518933 [I 2022-10-26 10:04:50] ...trap/engines/engine.py.224: train: epoch 1 | batch 1250/2304 [I 2022-10-26 10:04:50] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:08:46 [I 2022-10-26 10:04:50] ...trap/engines/engine.py.229: process: 0.50376 | load: 0.00247 [I 2022-10-26 10:04:50] ...trap/engines/engine.py.230: loss: 0.40035 [I 2022-10-26 10:04:50] ...optimizers/trijoint.py.151: total_norm: 5.404842 [I 2022-10-26 10:04:55] ...trap/engines/engine.py.224: train: epoch 1 | batch 1260/2304 [I 2022-10-26 10:04:55] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:08:38 [I 2022-10-26 10:04:55] ...trap/engines/engine.py.229: process: 0.48677 | load: 0.00246 [I 2022-10-26 10:04:55] ...trap/engines/engine.py.230: loss: 0.41635 [I 2022-10-26 10:04:55] ...optimizers/trijoint.py.151: total_norm: 5.595930 [I 2022-10-26 10:05:00] ...trap/engines/engine.py.224: train: epoch 1 | batch 1270/2304 [I 2022-10-26 10:05:00] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:08:31 [I 2022-10-26 10:05:00] ...trap/engines/engine.py.229: process: 0.48911 | load: 0.00243 [I 2022-10-26 10:05:00] ...trap/engines/engine.py.230: loss: 0.35601 [I 2022-10-26 10:05:00] ...optimizers/trijoint.py.151: total_norm: 4.127614 [I 2022-10-26 10:05:05] ...trap/engines/engine.py.224: train: epoch 1 | batch 1280/2304 [I 2022-10-26 10:05:05] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:08:25 [I 2022-10-26 10:05:05] ...trap/engines/engine.py.229: process: 0.48784 | load: 0.00246 [I 2022-10-26 10:05:05] ...trap/engines/engine.py.230: loss: 0.33611 [I 2022-10-26 10:05:05] ...optimizers/trijoint.py.151: total_norm: 4.299620 [I 2022-10-26 10:05:10] ...trap/engines/engine.py.224: train: epoch 1 | batch 1290/2304 [I 2022-10-26 10:05:10] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:08:26 [I 2022-10-26 10:05:10] ...trap/engines/engine.py.229: process: 0.50188 | load: 0.00254 [I 2022-10-26 10:05:10] ...trap/engines/engine.py.230: loss: 0.34224 [I 2022-10-26 10:05:10] ...optimizers/trijoint.py.151: total_norm: 4.166841 [I 2022-10-26 10:05:15] ...trap/engines/engine.py.224: train: epoch 1 | batch 1300/2304 [I 2022-10-26 10:05:15] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:08:30 [I 2022-10-26 10:05:15] ...trap/engines/engine.py.229: process: 0.52119 | load: 0.00265 [I 2022-10-26 10:05:15] ...trap/engines/engine.py.230: loss: 0.41825 [I 2022-10-26 10:05:15] ...optimizers/trijoint.py.151: total_norm: 4.362218 [I 2022-10-26 10:05:20] ...trap/engines/engine.py.224: train: epoch 1 | batch 1310/2304 [I 2022-10-26 10:05:20] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:08:22 [I 2022-10-26 10:05:20] ...trap/engines/engine.py.229: process: 0.50090 | load: 0.00245 [I 2022-10-26 10:05:20] ...trap/engines/engine.py.230: loss: 0.39493 [I 2022-10-26 10:05:20] ...optimizers/trijoint.py.151: total_norm: 4.471854 [I 2022-10-26 10:05:26] ...trap/engines/engine.py.224: train: epoch 1 | batch 1320/2304 [I 2022-10-26 10:05:26] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:08:33 [I 2022-10-26 10:05:26] ...trap/engines/engine.py.229: process: 0.50671 | load: 0.00252 [I 2022-10-26 10:05:26] ...trap/engines/engine.py.230: loss: 0.45143 [I 2022-10-26 10:05:26] ...optimizers/trijoint.py.151: total_norm: 4.688843 [I 2022-10-26 10:05:31] ...trap/engines/engine.py.224: train: epoch 1 | batch 1330/2304 [I 2022-10-26 10:05:31] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:08:12 [I 2022-10-26 10:05:31] ...trap/engines/engine.py.229: process: 0.49428 | load: 0.00245 [I 2022-10-26 10:05:31] ...trap/engines/engine.py.230: loss: 0.45928 [I 2022-10-26 10:05:31] ...optimizers/trijoint.py.151: total_norm: 4.966117 [I 2022-10-26 10:05:36] ...trap/engines/engine.py.224: train: epoch 1 | batch 1340/2304 [I 2022-10-26 10:05:36] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:08:08 [I 2022-10-26 10:05:36] ...trap/engines/engine.py.229: process: 0.49003 | load: 0.00250 [I 2022-10-26 10:05:36] ...trap/engines/engine.py.230: loss: 0.38060 [I 2022-10-26 10:05:36] ...optimizers/trijoint.py.151: total_norm: 6.775771 [I 2022-10-26 10:05:41] ...trap/engines/engine.py.224: train: epoch 1 | batch 1350/2304 [I 2022-10-26 10:05:41] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:07:57 [I 2022-10-26 10:05:41] ...trap/engines/engine.py.229: process: 0.49062 | load: 0.00252 [I 2022-10-26 10:05:41] ...trap/engines/engine.py.230: loss: 0.48650 [I 2022-10-26 10:05:41] ...optimizers/trijoint.py.151: total_norm: 6.890035 [I 2022-10-26 10:05:46] ...trap/engines/engine.py.224: train: epoch 1 | batch 1360/2304 [I 2022-10-26 10:05:46] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:07:47 [I 2022-10-26 10:05:46] ...trap/engines/engine.py.229: process: 0.48905 | load: 0.00246 [I 2022-10-26 10:05:46] ...trap/engines/engine.py.230: loss: 0.40478 [I 2022-10-26 10:05:46] ...optimizers/trijoint.py.151: total_norm: 3.987072 [I 2022-10-26 10:05:51] ...trap/engines/engine.py.224: train: epoch 1 | batch 1370/2304 [I 2022-10-26 10:05:51] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:07:43 [I 2022-10-26 10:05:51] ...trap/engines/engine.py.229: process: 0.48933 | load: 0.00243 [I 2022-10-26 10:05:51] ...trap/engines/engine.py.230: loss: 0.31055 [I 2022-10-26 10:05:51] ...optimizers/trijoint.py.151: total_norm: 4.431100 [I 2022-10-26 10:05:56] ...trap/engines/engine.py.224: train: epoch 1 | batch 1380/2304 [I 2022-10-26 10:05:56] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:07:45 [I 2022-10-26 10:05:56] ...trap/engines/engine.py.229: process: 0.51112 | load: 0.00244 [I 2022-10-26 10:05:56] ...trap/engines/engine.py.230: loss: 0.33631 [I 2022-10-26 10:05:56] ...optimizers/trijoint.py.151: total_norm: 3.999630 [I 2022-10-26 10:06:01] ...trap/engines/engine.py.224: train: epoch 1 | batch 1390/2304 [I 2022-10-26 10:06:01] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:07:39 [I 2022-10-26 10:06:01] ...trap/engines/engine.py.229: process: 0.48826 | load: 0.00243 [I 2022-10-26 10:06:01] ...trap/engines/engine.py.230: loss: 0.36856 [I 2022-10-26 10:06:01] ...optimizers/trijoint.py.151: total_norm: 4.130925 [I 2022-10-26 10:06:06] ...trap/engines/engine.py.224: train: epoch 1 | batch 1400/2304 [I 2022-10-26 10:06:06] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:07:35 [I 2022-10-26 10:06:06] ...trap/engines/engine.py.229: process: 0.49616 | load: 0.00245 [I 2022-10-26 10:06:06] ...trap/engines/engine.py.230: loss: 0.43999 [I 2022-10-26 10:06:06] ...optimizers/trijoint.py.151: total_norm: 4.712063 [I 2022-10-26 10:06:11] ...trap/engines/engine.py.224: train: epoch 1 | batch 1410/2304 [I 2022-10-26 10:06:11] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:07:32 [I 2022-10-26 10:06:11] ...trap/engines/engine.py.229: process: 0.51273 | load: 0.00253 [I 2022-10-26 10:06:11] ...trap/engines/engine.py.230: loss: 0.29273 [I 2022-10-26 10:06:11] ...optimizers/trijoint.py.151: total_norm: 3.657278 [I 2022-10-26 10:06:16] ...trap/engines/engine.py.224: train: epoch 1 | batch 1420/2304 [I 2022-10-26 10:06:16] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:07:32 [I 2022-10-26 10:06:16] ...trap/engines/engine.py.229: process: 0.50156 | load: 0.00246 [I 2022-10-26 10:06:16] ...trap/engines/engine.py.230: loss: 0.40716 [I 2022-10-26 10:06:16] ...optimizers/trijoint.py.151: total_norm: 4.360982 [I 2022-10-26 10:06:21] ...trap/engines/engine.py.224: train: epoch 1 | batch 1430/2304 [I 2022-10-26 10:06:21] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:07:18 [I 2022-10-26 10:06:21] ...trap/engines/engine.py.229: process: 0.50016 | load: 0.00253 [I 2022-10-26 10:06:21] ...trap/engines/engine.py.230: loss: 0.42092 [I 2022-10-26 10:06:21] ...optimizers/trijoint.py.151: total_norm: 4.670625 [I 2022-10-26 10:06:26] ...trap/engines/engine.py.224: train: epoch 1 | batch 1440/2304 [I 2022-10-26 10:06:26] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:07:12 [I 2022-10-26 10:06:26] ...trap/engines/engine.py.229: process: 0.50072 | load: 0.00252 [I 2022-10-26 10:06:26] ...trap/engines/engine.py.230: loss: 0.40490 [I 2022-10-26 10:06:26] ...optimizers/trijoint.py.151: total_norm: 4.763885 [I 2022-10-26 10:06:31] ...trap/engines/engine.py.224: train: epoch 1 | batch 1450/2304 [I 2022-10-26 10:06:31] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:07:09 [I 2022-10-26 10:06:31] ...trap/engines/engine.py.229: process: 0.50281 | load: 0.00252 [I 2022-10-26 10:06:31] ...trap/engines/engine.py.230: loss: 0.38022 [I 2022-10-26 10:06:31] ...optimizers/trijoint.py.151: total_norm: 4.739345 [I 2022-10-26 10:06:36] ...trap/engines/engine.py.224: train: epoch 1 | batch 1460/2304 [I 2022-10-26 10:06:36] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:06:56 [I 2022-10-26 10:06:36] ...trap/engines/engine.py.229: process: 0.49212 | load: 0.00250 [I 2022-10-26 10:06:36] ...trap/engines/engine.py.230: loss: 0.37229 [I 2022-10-26 10:06:36] ...optimizers/trijoint.py.151: total_norm: 4.291890 [I 2022-10-26 10:06:41] ...trap/engines/engine.py.224: train: epoch 1 | batch 1470/2304 [I 2022-10-26 10:06:41] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:06:56 [I 2022-10-26 10:06:41] ...trap/engines/engine.py.229: process: 0.50500 | load: 0.00244 [I 2022-10-26 10:06:41] ...trap/engines/engine.py.230: loss: 0.39659 [I 2022-10-26 10:06:41] ...optimizers/trijoint.py.151: total_norm: 4.453569 [I 2022-10-26 10:06:46] ...trap/engines/engine.py.224: train: epoch 1 | batch 1480/2304 [I 2022-10-26 10:06:46] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:06:52 [I 2022-10-26 10:06:46] ...trap/engines/engine.py.229: process: 0.50538 | load: 0.00249 [I 2022-10-26 10:06:46] ...trap/engines/engine.py.230: loss: 0.42207 [I 2022-10-26 10:06:46] ...optimizers/trijoint.py.151: total_norm: 4.912003 [I 2022-10-26 10:06:51] ...trap/engines/engine.py.224: train: epoch 1 | batch 1490/2304 [I 2022-10-26 10:06:51] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:06:49 [I 2022-10-26 10:06:51] ...trap/engines/engine.py.229: process: 0.49851 | load: 0.00250 [I 2022-10-26 10:06:51] ...trap/engines/engine.py.230: loss: 0.47441 [I 2022-10-26 10:06:51] ...optimizers/trijoint.py.151: total_norm: 5.961195 [I 2022-10-26 10:06:56] ...trap/engines/engine.py.224: train: epoch 1 | batch 1500/2304 [I 2022-10-26 10:06:56] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:06:40 [I 2022-10-26 10:06:56] ...trap/engines/engine.py.229: process: 0.50322 | load: 0.00252 [I 2022-10-26 10:06:56] ...trap/engines/engine.py.230: loss: 0.39548 [I 2022-10-26 10:06:56] ...optimizers/trijoint.py.151: total_norm: 4.907379 [I 2022-10-26 10:07:01] ...trap/engines/engine.py.224: train: epoch 1 | batch 1510/2304 [I 2022-10-26 10:07:01] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:06:36 [I 2022-10-26 10:07:01] ...trap/engines/engine.py.229: process: 0.50667 | load: 0.00247 [I 2022-10-26 10:07:01] ...trap/engines/engine.py.230: loss: 0.42148 [I 2022-10-26 10:07:01] ...optimizers/trijoint.py.151: total_norm: 4.426270 [I 2022-10-26 10:07:06] ...trap/engines/engine.py.224: train: epoch 1 | batch 1520/2304 [I 2022-10-26 10:07:06] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:06:31 [I 2022-10-26 10:07:06] ...trap/engines/engine.py.229: process: 0.50715 | load: 0.00248 [I 2022-10-26 10:07:06] ...trap/engines/engine.py.230: loss: 0.33908 [I 2022-10-26 10:07:06] ...optimizers/trijoint.py.151: total_norm: 3.668678 [I 2022-10-26 10:07:11] ...trap/engines/engine.py.224: train: epoch 1 | batch 1530/2304 [I 2022-10-26 10:07:11] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:06:24 [I 2022-10-26 10:07:11] ...trap/engines/engine.py.229: process: 0.49649 | load: 0.00247 [I 2022-10-26 10:07:11] ...trap/engines/engine.py.230: loss: 0.41122 [I 2022-10-26 10:07:11] ...optimizers/trijoint.py.151: total_norm: 4.216975 [I 2022-10-26 10:07:16] ...trap/engines/engine.py.224: train: epoch 1 | batch 1540/2304 [I 2022-10-26 10:07:16] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:06:23 [I 2022-10-26 10:07:16] ...trap/engines/engine.py.229: process: 0.49315 | load: 0.00246 [I 2022-10-26 10:07:16] ...trap/engines/engine.py.230: loss: 0.28930 [I 2022-10-26 10:07:16] ...optimizers/trijoint.py.151: total_norm: 3.894348 [I 2022-10-26 10:07:21] ...trap/engines/engine.py.224: train: epoch 1 | batch 1550/2304 [I 2022-10-26 10:07:21] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:06:15 [I 2022-10-26 10:07:21] ...trap/engines/engine.py.229: process: 0.49705 | load: 0.00250 [I 2022-10-26 10:07:21] ...trap/engines/engine.py.230: loss: 0.49912 [I 2022-10-26 10:07:21] ...optimizers/trijoint.py.151: total_norm: 5.018608 [I 2022-10-26 10:07:26] ...trap/engines/engine.py.224: train: epoch 1 | batch 1560/2304 [I 2022-10-26 10:07:26] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:06:09 [I 2022-10-26 10:07:26] ...trap/engines/engine.py.229: process: 0.50144 | load: 0.00243 [I 2022-10-26 10:07:26] ...trap/engines/engine.py.230: loss: 0.28886 [I 2022-10-26 10:07:26] ...optimizers/trijoint.py.151: total_norm: 4.233212 [I 2022-10-26 10:07:31] ...trap/engines/engine.py.224: train: epoch 1 | batch 1570/2304 [I 2022-10-26 10:07:31] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:06:04 [I 2022-10-26 10:07:31] ...trap/engines/engine.py.229: process: 0.49938 | load: 0.00242 [I 2022-10-26 10:07:31] ...trap/engines/engine.py.230: loss: 0.32764 [I 2022-10-26 10:07:31] ...optimizers/trijoint.py.151: total_norm: 5.472482 [I 2022-10-26 10:07:36] ...trap/engines/engine.py.224: train: epoch 1 | batch 1580/2304 [I 2022-10-26 10:07:36] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:06:02 [I 2022-10-26 10:07:36] ...trap/engines/engine.py.229: process: 0.49812 | load: 0.00243 [I 2022-10-26 10:07:36] ...trap/engines/engine.py.230: loss: 0.36701 [I 2022-10-26 10:07:36] ...optimizers/trijoint.py.151: total_norm: 4.486116 [I 2022-10-26 10:07:41] ...trap/engines/engine.py.224: train: epoch 1 | batch 1590/2304 [I 2022-10-26 10:07:41] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:05:56 [I 2022-10-26 10:07:41] ...trap/engines/engine.py.229: process: 0.49411 | load: 0.00241 [I 2022-10-26 10:07:41] ...trap/engines/engine.py.230: loss: 0.35297 [I 2022-10-26 10:07:41] ...optimizers/trijoint.py.151: total_norm: 4.133595 [I 2022-10-26 10:07:46] ...trap/engines/engine.py.224: train: epoch 1 | batch 1600/2304 [I 2022-10-26 10:07:46] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:06:01 [I 2022-10-26 10:07:46] ...trap/engines/engine.py.229: process: 0.49852 | load: 0.00257 [I 2022-10-26 10:07:46] ...trap/engines/engine.py.230: loss: 0.43349 [I 2022-10-26 10:07:46] ...optimizers/trijoint.py.151: total_norm: 5.167792 [I 2022-10-26 10:07:51] ...trap/engines/engine.py.224: train: epoch 1 | batch 1610/2304 [I 2022-10-26 10:07:51] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:05:47 [I 2022-10-26 10:07:51] ...trap/engines/engine.py.229: process: 0.50467 | load: 0.00252 [I 2022-10-26 10:07:51] ...trap/engines/engine.py.230: loss: 0.42772 [I 2022-10-26 10:07:51] ...optimizers/trijoint.py.151: total_norm: 4.465961 [I 2022-10-26 10:07:56] ...trap/engines/engine.py.224: train: epoch 1 | batch 1620/2304 [I 2022-10-26 10:07:57] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:05:45 [I 2022-10-26 10:07:57] ...trap/engines/engine.py.229: process: 0.51414 | load: 0.00247 [I 2022-10-26 10:07:57] ...trap/engines/engine.py.230: loss: 0.40195 [I 2022-10-26 10:07:57] ...optimizers/trijoint.py.151: total_norm: 4.580388 [I 2022-10-26 10:08:01] ...trap/engines/engine.py.224: train: epoch 1 | batch 1630/2304 [I 2022-10-26 10:08:01] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:05:36 [I 2022-10-26 10:08:01] ...trap/engines/engine.py.229: process: 0.50360 | load: 0.00269 [I 2022-10-26 10:08:01] ...trap/engines/engine.py.230: loss: 0.54741 [I 2022-10-26 10:08:01] ...optimizers/trijoint.py.151: total_norm: 5.425459 [I 2022-10-26 10:08:07] ...trap/engines/engine.py.224: train: epoch 1 | batch 1640/2304 [I 2022-10-26 10:08:07] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:05:36 [I 2022-10-26 10:08:07] ...trap/engines/engine.py.229: process: 0.50946 | load: 0.00245 [I 2022-10-26 10:08:07] ...trap/engines/engine.py.230: loss: 0.48868 [I 2022-10-26 10:08:07] ...optimizers/trijoint.py.151: total_norm: 4.727712 [I 2022-10-26 10:08:12] ...trap/engines/engine.py.224: train: epoch 1 | batch 1650/2304 [I 2022-10-26 10:08:12] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:05:31 [I 2022-10-26 10:08:12] ...trap/engines/engine.py.229: process: 0.49997 | load: 0.00252 [I 2022-10-26 10:08:12] ...trap/engines/engine.py.230: loss: 0.41989 [I 2022-10-26 10:08:12] ...optimizers/trijoint.py.151: total_norm: 5.893378 [I 2022-10-26 10:08:17] ...trap/engines/engine.py.224: train: epoch 1 | batch 1660/2304 [I 2022-10-26 10:08:17] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:05:21 [I 2022-10-26 10:08:17] ...trap/engines/engine.py.229: process: 0.50681 | load: 0.00248 [I 2022-10-26 10:08:17] ...trap/engines/engine.py.230: loss: 0.47139 [I 2022-10-26 10:08:17] ...optimizers/trijoint.py.151: total_norm: 4.943651 [I 2022-10-26 10:08:22] ...trap/engines/engine.py.224: train: epoch 1 | batch 1670/2304 [I 2022-10-26 10:08:22] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:05:20 [I 2022-10-26 10:08:22] ...trap/engines/engine.py.229: process: 0.49404 | load: 0.00247 [I 2022-10-26 10:08:22] ...trap/engines/engine.py.230: loss: 0.44128 [I 2022-10-26 10:08:22] ...optimizers/trijoint.py.151: total_norm: 4.833373 [I 2022-10-26 10:08:27] ...trap/engines/engine.py.224: train: epoch 1 | batch 1680/2304 [I 2022-10-26 10:08:27] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:05:19 [I 2022-10-26 10:08:27] ...trap/engines/engine.py.229: process: 0.52413 | load: 0.00245 [I 2022-10-26 10:08:27] ...trap/engines/engine.py.230: loss: 0.28107 [I 2022-10-26 10:08:27] ...optimizers/trijoint.py.151: total_norm: 3.302920 [I 2022-10-26 10:08:32] ...trap/engines/engine.py.224: train: epoch 1 | batch 1690/2304 [I 2022-10-26 10:08:32] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:05:13 [I 2022-10-26 10:08:32] ...trap/engines/engine.py.229: process: 0.50821 | load: 0.00257 [I 2022-10-26 10:08:32] ...trap/engines/engine.py.230: loss: 0.24951 [I 2022-10-26 10:08:32] ...optimizers/trijoint.py.151: total_norm: 4.407647 [I 2022-10-26 10:08:37] ...trap/engines/engine.py.224: train: epoch 1 | batch 1700/2304 [I 2022-10-26 10:08:37] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:05:08 [I 2022-10-26 10:08:37] ...trap/engines/engine.py.229: process: 0.51471 | load: 0.00260 [I 2022-10-26 10:08:37] ...trap/engines/engine.py.230: loss: 0.28622 [I 2022-10-26 10:08:37] ...optimizers/trijoint.py.151: total_norm: 4.080222 [I 2022-10-26 10:08:42] ...trap/engines/engine.py.224: train: epoch 1 | batch 1710/2304 [I 2022-10-26 10:08:42] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:04:58 [I 2022-10-26 10:08:42] ...trap/engines/engine.py.229: process: 0.50233 | load: 0.00246 [I 2022-10-26 10:08:42] ...trap/engines/engine.py.230: loss: 0.52148 [I 2022-10-26 10:08:42] ...optimizers/trijoint.py.151: total_norm: 5.549562 [I 2022-10-26 10:08:47] ...trap/engines/engine.py.224: train: epoch 1 | batch 1720/2304 [I 2022-10-26 10:08:47] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:04:50 [I 2022-10-26 10:08:47] ...trap/engines/engine.py.229: process: 0.50191 | load: 0.00246 [I 2022-10-26 10:08:47] ...trap/engines/engine.py.230: loss: 0.40965 [I 2022-10-26 10:08:47] ...optimizers/trijoint.py.151: total_norm: 4.427917 [I 2022-10-26 10:08:52] ...trap/engines/engine.py.224: train: epoch 1 | batch 1730/2304 [I 2022-10-26 10:08:52] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:04:48 [I 2022-10-26 10:08:52] ...trap/engines/engine.py.229: process: 0.50833 | load: 0.00244 [I 2022-10-26 10:08:52] ...trap/engines/engine.py.230: loss: 0.35867 [I 2022-10-26 10:08:52] ...optimizers/trijoint.py.151: total_norm: 4.699907 [I 2022-10-26 10:08:57] ...trap/engines/engine.py.224: train: epoch 1 | batch 1740/2304 [I 2022-10-26 10:08:57] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:04:48 [I 2022-10-26 10:08:57] ...trap/engines/engine.py.229: process: 0.49244 | load: 0.00253 [I 2022-10-26 10:08:57] ...trap/engines/engine.py.230: loss: 0.34446 [I 2022-10-26 10:08:57] ...optimizers/trijoint.py.151: total_norm: 4.158554 [I 2022-10-26 10:09:02] ...trap/engines/engine.py.224: train: epoch 1 | batch 1750/2304 [I 2022-10-26 10:09:02] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:04:34 [I 2022-10-26 10:09:02] ...trap/engines/engine.py.229: process: 0.49367 | load: 0.00246 [I 2022-10-26 10:09:02] ...trap/engines/engine.py.230: loss: 0.38809 [I 2022-10-26 10:09:02] ...optimizers/trijoint.py.151: total_norm: 4.581746 [I 2022-10-26 10:09:07] ...trap/engines/engine.py.224: train: epoch 1 | batch 1760/2304 [I 2022-10-26 10:09:07] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:04:32 [I 2022-10-26 10:09:07] ...trap/engines/engine.py.229: process: 0.50365 | load: 0.00244 [I 2022-10-26 10:09:07] ...trap/engines/engine.py.230: loss: 0.40953 [I 2022-10-26 10:09:07] ...optimizers/trijoint.py.151: total_norm: 4.201002 [I 2022-10-26 10:09:12] ...trap/engines/engine.py.224: train: epoch 1 | batch 1770/2304 [I 2022-10-26 10:09:12] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:04:24 [I 2022-10-26 10:09:12] ...trap/engines/engine.py.229: process: 0.48979 | load: 0.00246 [I 2022-10-26 10:09:12] ...trap/engines/engine.py.230: loss: 0.37846 [I 2022-10-26 10:09:12] ...optimizers/trijoint.py.151: total_norm: 4.890542 [I 2022-10-26 10:09:17] ...trap/engines/engine.py.224: train: epoch 1 | batch 1780/2304 [I 2022-10-26 10:09:17] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:04:21 [I 2022-10-26 10:09:17] ...trap/engines/engine.py.229: process: 0.49280 | load: 0.00244 [I 2022-10-26 10:09:17] ...trap/engines/engine.py.230: loss: 0.48232 [I 2022-10-26 10:09:17] ...optimizers/trijoint.py.151: total_norm: 5.558592 [I 2022-10-26 10:09:22] ...trap/engines/engine.py.224: train: epoch 1 | batch 1790/2304 [I 2022-10-26 10:09:22] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:04:15 [I 2022-10-26 10:09:22] ...trap/engines/engine.py.229: process: 0.50271 | load: 0.00240 [I 2022-10-26 10:09:22] ...trap/engines/engine.py.230: loss: 0.42050 [I 2022-10-26 10:09:22] ...optimizers/trijoint.py.151: total_norm: 5.022643 [I 2022-10-26 10:09:27] ...trap/engines/engine.py.224: train: epoch 1 | batch 1800/2304 [I 2022-10-26 10:09:27] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:04:09 [I 2022-10-26 10:09:27] ...trap/engines/engine.py.229: process: 0.49181 | load: 0.00266 [I 2022-10-26 10:09:27] ...trap/engines/engine.py.230: loss: 0.46539 [I 2022-10-26 10:09:27] ...optimizers/trijoint.py.151: total_norm: 4.779191 [I 2022-10-26 10:09:32] ...trap/engines/engine.py.224: train: epoch 1 | batch 1810/2304 [I 2022-10-26 10:09:32] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:04:07 [I 2022-10-26 10:09:32] ...trap/engines/engine.py.229: process: 0.49616 | load: 0.00254 [I 2022-10-26 10:09:32] ...trap/engines/engine.py.230: loss: 0.33260 [I 2022-10-26 10:09:32] ...optimizers/trijoint.py.151: total_norm: 3.459193 [I 2022-10-26 10:09:37] ...trap/engines/engine.py.224: train: epoch 1 | batch 1820/2304 [I 2022-10-26 10:09:37] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:04:05 [I 2022-10-26 10:09:37] ...trap/engines/engine.py.229: process: 0.50435 | load: 0.00281 [I 2022-10-26 10:09:37] ...trap/engines/engine.py.230: loss: 0.32663 [I 2022-10-26 10:09:37] ...optimizers/trijoint.py.151: total_norm: 3.931043 [I 2022-10-26 10:09:42] ...trap/engines/engine.py.224: train: epoch 1 | batch 1830/2304 [I 2022-10-26 10:09:42] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:03:58 [I 2022-10-26 10:09:42] ...trap/engines/engine.py.229: process: 0.50638 | load: 0.00246 [I 2022-10-26 10:09:42] ...trap/engines/engine.py.230: loss: 0.43258 [I 2022-10-26 10:09:42] ...optimizers/trijoint.py.151: total_norm: 5.758574 [I 2022-10-26 10:09:47] ...trap/engines/engine.py.224: train: epoch 1 | batch 1840/2304 [I 2022-10-26 10:09:47] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:03:54 [I 2022-10-26 10:09:47] ...trap/engines/engine.py.229: process: 0.53114 | load: 0.00242 [I 2022-10-26 10:09:47] ...trap/engines/engine.py.230: loss: 0.35404 [I 2022-10-26 10:09:47] ...optimizers/trijoint.py.151: total_norm: 6.028431 [I 2022-10-26 10:09:52] ...trap/engines/engine.py.224: train: epoch 1 | batch 1850/2304 [I 2022-10-26 10:09:52] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:03:48 [I 2022-10-26 10:09:52] ...trap/engines/engine.py.229: process: 0.49945 | load: 0.00249 [I 2022-10-26 10:09:52] ...trap/engines/engine.py.230: loss: 0.46684 [I 2022-10-26 10:09:52] ...optimizers/trijoint.py.151: total_norm: 5.674788 [I 2022-10-26 10:09:57] ...trap/engines/engine.py.224: train: epoch 1 | batch 1860/2304 [I 2022-10-26 10:09:57] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:03:48 [I 2022-10-26 10:09:57] ...trap/engines/engine.py.229: process: 0.50779 | load: 0.00264 [I 2022-10-26 10:09:57] ...trap/engines/engine.py.230: loss: 0.56390 [I 2022-10-26 10:09:57] ...optimizers/trijoint.py.151: total_norm: 5.813251 [I 2022-10-26 10:10:03] ...trap/engines/engine.py.224: train: epoch 1 | batch 1870/2304 [I 2022-10-26 10:10:03] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:03:38 [I 2022-10-26 10:10:03] ...trap/engines/engine.py.229: process: 0.49969 | load: 0.00246 [I 2022-10-26 10:10:03] ...trap/engines/engine.py.230: loss: 0.32354 [I 2022-10-26 10:10:03] ...optimizers/trijoint.py.151: total_norm: 3.093830 [I 2022-10-26 10:10:08] ...trap/engines/engine.py.224: train: epoch 1 | batch 1880/2304 [I 2022-10-26 10:10:08] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:03:32 [I 2022-10-26 10:10:08] ...trap/engines/engine.py.229: process: 0.50602 | load: 0.00247 [I 2022-10-26 10:10:08] ...trap/engines/engine.py.230: loss: 0.40368 [I 2022-10-26 10:10:08] ...optimizers/trijoint.py.151: total_norm: 5.503015 [I 2022-10-26 10:10:13] ...trap/engines/engine.py.224: train: epoch 1 | batch 1890/2304 [I 2022-10-26 10:10:13] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:03:26 [I 2022-10-26 10:10:13] ...trap/engines/engine.py.229: process: 0.50301 | load: 0.00243 [I 2022-10-26 10:10:13] ...trap/engines/engine.py.230: loss: 0.33698 [I 2022-10-26 10:10:13] ...optimizers/trijoint.py.151: total_norm: 5.067959 [I 2022-10-26 10:10:18] ...trap/engines/engine.py.224: train: epoch 1 | batch 1900/2304 [I 2022-10-26 10:10:18] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:03:21 [I 2022-10-26 10:10:18] ...trap/engines/engine.py.229: process: 0.50804 | load: 0.00272 [I 2022-10-26 10:10:18] ...trap/engines/engine.py.230: loss: 0.38367 [I 2022-10-26 10:10:18] ...optimizers/trijoint.py.151: total_norm: 3.950702 [I 2022-10-26 10:10:23] ...trap/engines/engine.py.224: train: epoch 1 | batch 1910/2304 [I 2022-10-26 10:10:23] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:03:17 [I 2022-10-26 10:10:23] ...trap/engines/engine.py.229: process: 0.51413 | load: 0.00254 [I 2022-10-26 10:10:23] ...trap/engines/engine.py.230: loss: 0.53197 [I 2022-10-26 10:10:23] ...optimizers/trijoint.py.151: total_norm: 5.324998 [I 2022-10-26 10:10:28] ...trap/engines/engine.py.224: train: epoch 1 | batch 1920/2304 [I 2022-10-26 10:10:28] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:03:10 [I 2022-10-26 10:10:28] ...trap/engines/engine.py.229: process: 0.49141 | load: 0.00249 [I 2022-10-26 10:10:28] ...trap/engines/engine.py.230: loss: 0.40699 [I 2022-10-26 10:10:28] ...optimizers/trijoint.py.151: total_norm: 4.952830 [I 2022-10-26 10:10:33] ...trap/engines/engine.py.224: train: epoch 1 | batch 1930/2304 [I 2022-10-26 10:10:33] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:03:06 [I 2022-10-26 10:10:33] ...trap/engines/engine.py.229: process: 0.50707 | load: 0.00246 [I 2022-10-26 10:10:33] ...trap/engines/engine.py.230: loss: 0.42422 [I 2022-10-26 10:10:33] ...optimizers/trijoint.py.151: total_norm: 6.800599 [I 2022-10-26 10:10:38] ...trap/engines/engine.py.224: train: epoch 1 | batch 1940/2304 [I 2022-10-26 10:10:38] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:03:01 [I 2022-10-26 10:10:38] ...trap/engines/engine.py.229: process: 0.49122 | load: 0.00244 [I 2022-10-26 10:10:38] ...trap/engines/engine.py.230: loss: 0.50369 [I 2022-10-26 10:10:38] ...optimizers/trijoint.py.151: total_norm: 5.166980 [I 2022-10-26 10:10:43] ...trap/engines/engine.py.224: train: epoch 1 | batch 1950/2304 [I 2022-10-26 10:10:43] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:02:57 [I 2022-10-26 10:10:43] ...trap/engines/engine.py.229: process: 0.50032 | load: 0.00250 [I 2022-10-26 10:10:43] ...trap/engines/engine.py.230: loss: 0.40364 [I 2022-10-26 10:10:43] ...optimizers/trijoint.py.151: total_norm: 4.916012 [I 2022-10-26 10:10:48] ...trap/engines/engine.py.224: train: epoch 1 | batch 1960/2304 [I 2022-10-26 10:10:48] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:02:50 [I 2022-10-26 10:10:48] ...trap/engines/engine.py.229: process: 0.49828 | load: 0.00245 [I 2022-10-26 10:10:48] ...trap/engines/engine.py.230: loss: 0.39353 [I 2022-10-26 10:10:48] ...optimizers/trijoint.py.151: total_norm: 4.083496 [I 2022-10-26 10:10:53] ...trap/engines/engine.py.224: train: epoch 1 | batch 1970/2304 [I 2022-10-26 10:10:53] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:02:46 [I 2022-10-26 10:10:53] ...trap/engines/engine.py.229: process: 0.50247 | load: 0.00246 [I 2022-10-26 10:10:53] ...trap/engines/engine.py.230: loss: 0.35214 [I 2022-10-26 10:10:53] ...optimizers/trijoint.py.151: total_norm: 3.846190 [I 2022-10-26 10:10:58] ...trap/engines/engine.py.224: train: epoch 1 | batch 1980/2304 [I 2022-10-26 10:10:58] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:02:41 [I 2022-10-26 10:10:58] ...trap/engines/engine.py.229: process: 0.48300 | load: 0.00244 [I 2022-10-26 10:10:58] ...trap/engines/engine.py.230: loss: 0.31225 [I 2022-10-26 10:10:58] ...optimizers/trijoint.py.151: total_norm: 3.871379 [I 2022-10-26 10:11:03] ...trap/engines/engine.py.224: train: epoch 1 | batch 1990/2304 [I 2022-10-26 10:11:03] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:02:36 [I 2022-10-26 10:11:03] ...trap/engines/engine.py.229: process: 0.50472 | load: 0.00245 [I 2022-10-26 10:11:03] ...trap/engines/engine.py.230: loss: 0.47684 [I 2022-10-26 10:11:03] ...optimizers/trijoint.py.151: total_norm: 4.832481 [I 2022-10-26 10:11:08] ...trap/engines/engine.py.224: train: epoch 1 | batch 2000/2304 [I 2022-10-26 10:11:08] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:02:31 [I 2022-10-26 10:11:08] ...trap/engines/engine.py.229: process: 0.49035 | load: 0.00238 [I 2022-10-26 10:11:08] ...trap/engines/engine.py.230: loss: 0.38425 [I 2022-10-26 10:11:08] ...optimizers/trijoint.py.151: total_norm: 4.184382 [I 2022-10-26 10:11:13] ...trap/engines/engine.py.224: train: epoch 1 | batch 2010/2304 [I 2022-10-26 10:11:13] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:02:26 [I 2022-10-26 10:11:13] ...trap/engines/engine.py.229: process: 0.50375 | load: 0.00253 [I 2022-10-26 10:11:13] ...trap/engines/engine.py.230: loss: 0.31839 [I 2022-10-26 10:11:13] ...optimizers/trijoint.py.151: total_norm: 5.201690 [I 2022-10-26 10:11:18] ...trap/engines/engine.py.224: train: epoch 1 | batch 2020/2304 [I 2022-10-26 10:11:18] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:02:21 [I 2022-10-26 10:11:18] ...trap/engines/engine.py.229: process: 0.49987 | load: 0.00246 [I 2022-10-26 10:11:18] ...trap/engines/engine.py.230: loss: 0.39232 [I 2022-10-26 10:11:18] ...optimizers/trijoint.py.151: total_norm: 4.758391 [I 2022-10-26 10:11:23] ...trap/engines/engine.py.224: train: epoch 1 | batch 2030/2304 [I 2022-10-26 10:11:23] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:02:17 [I 2022-10-26 10:11:23] ...trap/engines/engine.py.229: process: 0.50524 | load: 0.00247 [I 2022-10-26 10:11:23] ...trap/engines/engine.py.230: loss: 0.41883 [I 2022-10-26 10:11:23] ...optimizers/trijoint.py.151: total_norm: 4.115460 [I 2022-10-26 10:11:28] ...trap/engines/engine.py.224: train: epoch 1 | batch 2040/2304 [I 2022-10-26 10:11:28] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:02:11 [I 2022-10-26 10:11:28] ...trap/engines/engine.py.229: process: 0.49683 | load: 0.00244 [I 2022-10-26 10:11:28] ...trap/engines/engine.py.230: loss: 0.39243 [I 2022-10-26 10:11:28] ...optimizers/trijoint.py.151: total_norm: 6.408426 [I 2022-10-26 10:11:33] ...trap/engines/engine.py.224: train: epoch 1 | batch 2050/2304 [I 2022-10-26 10:11:33] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:02:07 [I 2022-10-26 10:11:33] ...trap/engines/engine.py.229: process: 0.50120 | load: 0.00248 [I 2022-10-26 10:11:33] ...trap/engines/engine.py.230: loss: 0.30959 [I 2022-10-26 10:11:33] ...optimizers/trijoint.py.151: total_norm: 4.114233 [I 2022-10-26 10:11:38] ...trap/engines/engine.py.224: train: epoch 1 | batch 2060/2304 [I 2022-10-26 10:11:38] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:02:02 [I 2022-10-26 10:11:38] ...trap/engines/engine.py.229: process: 0.50015 | load: 0.00253 [I 2022-10-26 10:11:38] ...trap/engines/engine.py.230: loss: 0.37101 [I 2022-10-26 10:11:38] ...optimizers/trijoint.py.151: total_norm: 4.527372 [I 2022-10-26 10:11:43] ...trap/engines/engine.py.224: train: epoch 1 | batch 2070/2304 [I 2022-10-26 10:11:43] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:01:55 [I 2022-10-26 10:11:43] ...trap/engines/engine.py.229: process: 0.48740 | load: 0.00245 [I 2022-10-26 10:11:43] ...trap/engines/engine.py.230: loss: 0.34961 [I 2022-10-26 10:11:43] ...optimizers/trijoint.py.151: total_norm: 3.833852 [I 2022-10-26 10:11:48] ...trap/engines/engine.py.224: train: epoch 1 | batch 2080/2304 [I 2022-10-26 10:11:48] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:01:51 [I 2022-10-26 10:11:48] ...trap/engines/engine.py.229: process: 0.50364 | load: 0.00246 [I 2022-10-26 10:11:48] ...trap/engines/engine.py.230: loss: 0.53553 [I 2022-10-26 10:11:48] ...optimizers/trijoint.py.151: total_norm: 5.464993 [I 2022-10-26 10:11:53] ...trap/engines/engine.py.224: train: epoch 1 | batch 2090/2304 [I 2022-10-26 10:11:53] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:01:46 [I 2022-10-26 10:11:53] ...trap/engines/engine.py.229: process: 0.49174 | load: 0.00244 [I 2022-10-26 10:11:53] ...trap/engines/engine.py.230: loss: 0.41709 [I 2022-10-26 10:11:53] ...optimizers/trijoint.py.151: total_norm: 4.050462 [I 2022-10-26 10:11:58] ...trap/engines/engine.py.224: train: epoch 1 | batch 2100/2304 [I 2022-10-26 10:11:58] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:01:41 [I 2022-10-26 10:11:58] ...trap/engines/engine.py.229: process: 0.49892 | load: 0.00247 [I 2022-10-26 10:11:58] ...trap/engines/engine.py.230: loss: 0.41300 [I 2022-10-26 10:11:58] ...optimizers/trijoint.py.151: total_norm: 3.763933 [I 2022-10-26 10:12:02] ...trap/engines/engine.py.224: train: epoch 1 | batch 2110/2304 [I 2022-10-26 10:12:02] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:01:36 [I 2022-10-26 10:12:02] ...trap/engines/engine.py.229: process: 0.51193 | load: 0.00252 [I 2022-10-26 10:12:02] ...trap/engines/engine.py.230: loss: 0.35804 [I 2022-10-26 10:12:02] ...optimizers/trijoint.py.151: total_norm: 4.084530 [I 2022-10-26 10:12:07] ...trap/engines/engine.py.224: train: epoch 1 | batch 2120/2304 [I 2022-10-26 10:12:07] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:01:31 [I 2022-10-26 10:12:07] ...trap/engines/engine.py.229: process: 0.50340 | load: 0.00250 [I 2022-10-26 10:12:07] ...trap/engines/engine.py.230: loss: 0.40844 [I 2022-10-26 10:12:07] ...optimizers/trijoint.py.151: total_norm: 4.195307 [I 2022-10-26 10:12:13] ...trap/engines/engine.py.224: train: epoch 1 | batch 2130/2304 [I 2022-10-26 10:12:13] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:01:27 [I 2022-10-26 10:12:13] ...trap/engines/engine.py.229: process: 0.50822 | load: 0.00255 [I 2022-10-26 10:12:13] ...trap/engines/engine.py.230: loss: 0.38807 [I 2022-10-26 10:12:13] ...optimizers/trijoint.py.151: total_norm: 4.204256 [I 2022-10-26 10:12:18] ...trap/engines/engine.py.224: train: epoch 1 | batch 2140/2304 [I 2022-10-26 10:12:18] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:01:22 [I 2022-10-26 10:12:18] ...trap/engines/engine.py.229: process: 0.51476 | load: 0.00258 [I 2022-10-26 10:12:18] ...trap/engines/engine.py.230: loss: 0.30683 [I 2022-10-26 10:12:18] ...optimizers/trijoint.py.151: total_norm: 3.251572 [I 2022-10-26 10:12:23] ...trap/engines/engine.py.224: train: epoch 1 | batch 2150/2304 [I 2022-10-26 10:12:23] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:01:17 [I 2022-10-26 10:12:23] ...trap/engines/engine.py.229: process: 0.52286 | load: 0.00255 [I 2022-10-26 10:12:23] ...trap/engines/engine.py.230: loss: 0.35112 [I 2022-10-26 10:12:23] ...optimizers/trijoint.py.151: total_norm: 4.325217 [I 2022-10-26 10:12:28] ...trap/engines/engine.py.224: train: epoch 1 | batch 2160/2304 [I 2022-10-26 10:12:28] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:01:11 [I 2022-10-26 10:12:28] ...trap/engines/engine.py.229: process: 0.49890 | load: 0.00258 [I 2022-10-26 10:12:28] ...trap/engines/engine.py.230: loss: 0.40896 [I 2022-10-26 10:12:28] ...optimizers/trijoint.py.151: total_norm: 5.167802 [I 2022-10-26 10:12:33] ...trap/engines/engine.py.224: train: epoch 1 | batch 2170/2304 [I 2022-10-26 10:12:33] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:01:06 [I 2022-10-26 10:12:33] ...trap/engines/engine.py.229: process: 0.48800 | load: 0.00257 [I 2022-10-26 10:12:33] ...trap/engines/engine.py.230: loss: 0.37578 [I 2022-10-26 10:12:33] ...optimizers/trijoint.py.151: total_norm: 3.748184 [I 2022-10-26 10:12:38] ...trap/engines/engine.py.224: train: epoch 1 | batch 2180/2304 [I 2022-10-26 10:12:38] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:01:02 [I 2022-10-26 10:12:38] ...trap/engines/engine.py.229: process: 0.50451 | load: 0.00257 [I 2022-10-26 10:12:38] ...trap/engines/engine.py.230: loss: 0.40665 [I 2022-10-26 10:12:38] ...optimizers/trijoint.py.151: total_norm: 4.791046 [I 2022-10-26 10:12:43] ...trap/engines/engine.py.224: train: epoch 1 | batch 2190/2304 [I 2022-10-26 10:12:43] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:00:57 [I 2022-10-26 10:12:43] ...trap/engines/engine.py.229: process: 0.50904 | load: 0.00249 [I 2022-10-26 10:12:43] ...trap/engines/engine.py.230: loss: 0.31512 [I 2022-10-26 10:12:43] ...optimizers/trijoint.py.151: total_norm: 3.855683 [I 2022-10-26 10:12:48] ...trap/engines/engine.py.224: train: epoch 1 | batch 2200/2304 [I 2022-10-26 10:12:48] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:00:52 [I 2022-10-26 10:12:48] ...trap/engines/engine.py.229: process: 0.50068 | load: 0.00254 [I 2022-10-26 10:12:48] ...trap/engines/engine.py.230: loss: 0.33168 [I 2022-10-26 10:12:48] ...optimizers/trijoint.py.151: total_norm: 3.843516 [I 2022-10-26 10:12:53] ...trap/engines/engine.py.224: train: epoch 1 | batch 2210/2304 [I 2022-10-26 10:12:53] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:00:47 [I 2022-10-26 10:12:53] ...trap/engines/engine.py.229: process: 0.50309 | load: 0.00241 [I 2022-10-26 10:12:53] ...trap/engines/engine.py.230: loss: 0.42691 [I 2022-10-26 10:12:53] ...optimizers/trijoint.py.151: total_norm: 4.237589 [I 2022-10-26 10:12:58] ...trap/engines/engine.py.224: train: epoch 1 | batch 2220/2304 [I 2022-10-26 10:12:58] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:00:41 [I 2022-10-26 10:12:58] ...trap/engines/engine.py.229: process: 0.48930 | load: 0.00252 [I 2022-10-26 10:12:58] ...trap/engines/engine.py.230: loss: 0.43502 [I 2022-10-26 10:12:58] ...optimizers/trijoint.py.151: total_norm: 4.786369 [I 2022-10-26 10:13:03] ...trap/engines/engine.py.224: train: epoch 1 | batch 2230/2304 [I 2022-10-26 10:13:03] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:00:37 [I 2022-10-26 10:13:03] ...trap/engines/engine.py.229: process: 0.49226 | load: 0.00266 [I 2022-10-26 10:13:03] ...trap/engines/engine.py.230: loss: 0.31436 [I 2022-10-26 10:13:03] ...optimizers/trijoint.py.151: total_norm: 3.963629 [I 2022-10-26 10:13:08] ...trap/engines/engine.py.224: train: epoch 1 | batch 2240/2304 [I 2022-10-26 10:13:08] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:00:31 [I 2022-10-26 10:13:08] ...trap/engines/engine.py.229: process: 0.50975 | load: 0.00247 [I 2022-10-26 10:13:08] ...trap/engines/engine.py.230: loss: 0.35434 [I 2022-10-26 10:13:08] ...optimizers/trijoint.py.151: total_norm: 4.311651 [I 2022-10-26 10:13:13] ...trap/engines/engine.py.224: train: epoch 1 | batch 2250/2304 [I 2022-10-26 10:13:13] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:00:26 [I 2022-10-26 10:13:13] ...trap/engines/engine.py.229: process: 0.48824 | load: 0.00246 [I 2022-10-26 10:13:13] ...trap/engines/engine.py.230: loss: 0.44034 [I 2022-10-26 10:13:13] ...optimizers/trijoint.py.151: total_norm: 5.566633 [I 2022-10-26 10:13:18] ...trap/engines/engine.py.224: train: epoch 1 | batch 2260/2304 [I 2022-10-26 10:13:18] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:00:21 [I 2022-10-26 10:13:18] ...trap/engines/engine.py.229: process: 0.48566 | load: 0.00243 [I 2022-10-26 10:13:18] ...trap/engines/engine.py.230: loss: 0.39082 [I 2022-10-26 10:13:18] ...optimizers/trijoint.py.151: total_norm: 4.899796 [I 2022-10-26 10:13:23] ...trap/engines/engine.py.224: train: epoch 1 | batch 2270/2304 [I 2022-10-26 10:13:23] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:00:17 [I 2022-10-26 10:13:23] ...trap/engines/engine.py.229: process: 0.51359 | load: 0.00246 [I 2022-10-26 10:13:23] ...trap/engines/engine.py.230: loss: 0.52227 [I 2022-10-26 10:13:23] ...optimizers/trijoint.py.151: total_norm: 4.995454 [I 2022-10-26 10:13:28] ...trap/engines/engine.py.224: train: epoch 1 | batch 2280/2304 [I 2022-10-26 10:13:28] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:00:11 [I 2022-10-26 10:13:28] ...trap/engines/engine.py.229: process: 0.49973 | load: 0.00242 [I 2022-10-26 10:13:28] ...trap/engines/engine.py.230: loss: 0.37516 [I 2022-10-26 10:13:28] ...optimizers/trijoint.py.151: total_norm: 4.573330 [I 2022-10-26 10:13:33] ...trap/engines/engine.py.224: train: epoch 1 | batch 2290/2304 [I 2022-10-26 10:13:33] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:00:06 [I 2022-10-26 10:13:33] ...trap/engines/engine.py.229: process: 0.48602 | load: 0.00245 [I 2022-10-26 10:13:33] ...trap/engines/engine.py.230: loss: 0.37922 [I 2022-10-26 10:13:33] ...optimizers/trijoint.py.151: total_norm: 3.623217 [I 2022-10-26 10:13:38] ...trap/engines/engine.py.224: train: epoch 1 | batch 2300/2304 [I 2022-10-26 10:13:38] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:00:02 [I 2022-10-26 10:13:38] ...trap/engines/engine.py.229: process: 0.48246 | load: 0.00014 [I 2022-10-26 10:13:38] ...trap/engines/engine.py.230: loss: 0.39522 [I 2022-10-26 10:13:38] ...optimizers/trijoint.py.151: total_norm: 4.398092 [I 2022-10-26 10:13:40] ...trap/engines/engine.py.224: train: epoch 1 | batch 2304/2304 [I 2022-10-26 10:13:40] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:00:00 [I 2022-10-26 10:13:40] ...trap/engines/engine.py.229: process: 0.48865 | load: 0.00013 [I 2022-10-26 10:13:40] ...trap/engines/engine.py.230: loss: 0.38801 [I 2022-10-26 10:13:40] ...optimizers/trijoint.py.151: total_norm: 4.422812 [S 2022-10-26 10:13:41] ...trap/engines/engine.py.129: train_epoch.epoch: 1 [S 2022-10-26 10:13:41] ...trap/engines/engine.py.129: train_epoch.loss: 0.41588613644339256 [I 2022-10-26 10:13:41] ...s/criterions/triplet.py.84: margin = 0.06 [I 2022-10-26 10:13:41] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 1 [I 2022-10-26 10:13:41] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 10:13:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 10:13:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 10:13:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 10:13:43] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 10:13:43] ...trap/engines/engine.py.321: eval: epoch 1 | batch 0/510 [I 2022-10-26 10:13:43] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:21:37 [I 2022-10-26 10:13:43] ...trap/engines/engine.py.326: process: 2.54351 | load: 2.17404 [I 2022-10-26 10:13:47] ...trap/engines/engine.py.321: eval: epoch 1 | batch 10/510 [I 2022-10-26 10:13:47] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:04:43 [I 2022-10-26 10:13:47] ...trap/engines/engine.py.326: process: 0.31858 | load: 0.00019 [I 2022-10-26 10:13:50] ...trap/engines/engine.py.321: eval: epoch 1 | batch 20/510 [I 2022-10-26 10:13:50] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:54 [I 2022-10-26 10:13:50] ...trap/engines/engine.py.326: process: 0.33299 | load: 0.00018 [I 2022-10-26 10:13:53] ...trap/engines/engine.py.321: eval: epoch 1 | batch 30/510 [I 2022-10-26 10:13:53] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:44 [I 2022-10-26 10:13:53] ...trap/engines/engine.py.326: process: 0.32434 | load: 0.00019 [I 2022-10-26 10:13:57] ...trap/engines/engine.py.321: eval: epoch 1 | batch 40/510 [I 2022-10-26 10:13:57] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:41 [I 2022-10-26 10:13:57] ...trap/engines/engine.py.326: process: 0.35274 | load: 0.00020 [I 2022-10-26 10:14:00] ...trap/engines/engine.py.321: eval: epoch 1 | batch 50/510 [I 2022-10-26 10:14:00] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:34 [I 2022-10-26 10:14:00] ...trap/engines/engine.py.326: process: 0.31831 | load: 0.00021 [I 2022-10-26 10:14:03] ...trap/engines/engine.py.321: eval: epoch 1 | batch 60/510 [I 2022-10-26 10:14:03] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:31 [I 2022-10-26 10:14:03] ...trap/engines/engine.py.326: process: 0.34150 | load: 0.00016 [I 2022-10-26 10:14:07] ...trap/engines/engine.py.321: eval: epoch 1 | batch 70/510 [I 2022-10-26 10:14:07] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-26 10:14:07] ...trap/engines/engine.py.326: process: 0.32866 | load: 0.00018 [I 2022-10-26 10:14:10] ...trap/engines/engine.py.321: eval: epoch 1 | batch 80/510 [I 2022-10-26 10:14:10] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:22 [I 2022-10-26 10:14:10] ...trap/engines/engine.py.326: process: 0.32874 | load: 0.00022 [I 2022-10-26 10:14:14] ...trap/engines/engine.py.321: eval: epoch 1 | batch 90/510 [I 2022-10-26 10:14:14] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:21 [I 2022-10-26 10:14:14] ...trap/engines/engine.py.326: process: 0.33094 | load: 0.00019 [I 2022-10-26 10:14:17] ...trap/engines/engine.py.321: eval: epoch 1 | batch 100/510 [I 2022-10-26 10:14:17] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:19 [I 2022-10-26 10:14:17] ...trap/engines/engine.py.326: process: 0.31174 | load: 0.00024 [I 2022-10-26 10:14:20] ...trap/engines/engine.py.321: eval: epoch 1 | batch 110/510 [I 2022-10-26 10:14:20] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:58 [I 2022-10-26 10:14:20] ...trap/engines/engine.py.326: process: 0.28607 | load: 0.00021 [I 2022-10-26 10:14:23] ...trap/engines/engine.py.321: eval: epoch 1 | batch 120/510 [I 2022-10-26 10:14:23] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-26 10:14:23] ...trap/engines/engine.py.326: process: 0.29276 | load: 0.00014 [I 2022-10-26 10:14:26] ...trap/engines/engine.py.321: eval: epoch 1 | batch 130/510 [I 2022-10-26 10:14:26] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 10:14:26] ...trap/engines/engine.py.326: process: 0.29272 | load: 0.00016 [I 2022-10-26 10:14:29] ...trap/engines/engine.py.321: eval: epoch 1 | batch 140/510 [I 2022-10-26 10:14:29] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:46 [I 2022-10-26 10:14:29] ...trap/engines/engine.py.326: process: 0.28963 | load: 0.00015 [I 2022-10-26 10:14:31] ...trap/engines/engine.py.321: eval: epoch 1 | batch 150/510 [I 2022-10-26 10:14:31] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:42 [I 2022-10-26 10:14:31] ...trap/engines/engine.py.326: process: 0.28644 | load: 0.00014 [I 2022-10-26 10:14:34] ...trap/engines/engine.py.321: eval: epoch 1 | batch 160/510 [I 2022-10-26 10:14:34] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:41 [I 2022-10-26 10:14:34] ...trap/engines/engine.py.326: process: 0.30466 | load: 0.00016 [I 2022-10-26 10:14:37] ...trap/engines/engine.py.321: eval: epoch 1 | batch 170/510 [I 2022-10-26 10:14:37] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 10:14:37] ...trap/engines/engine.py.326: process: 0.28986 | load: 0.00015 [I 2022-10-26 10:14:40] ...trap/engines/engine.py.321: eval: epoch 1 | batch 180/510 [I 2022-10-26 10:14:40] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 10:14:40] ...trap/engines/engine.py.326: process: 0.28462 | load: 0.00014 [I 2022-10-26 10:14:43] ...trap/engines/engine.py.321: eval: epoch 1 | batch 190/510 [I 2022-10-26 10:14:43] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 10:14:43] ...trap/engines/engine.py.326: process: 0.28064 | load: 0.00014 [I 2022-10-26 10:14:46] ...trap/engines/engine.py.321: eval: epoch 1 | batch 200/510 [I 2022-10-26 10:14:46] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 10:14:46] ...trap/engines/engine.py.326: process: 0.28543 | load: 0.00020 [I 2022-10-26 10:14:49] ...trap/engines/engine.py.321: eval: epoch 1 | batch 210/510 [I 2022-10-26 10:14:49] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 10:14:49] ...trap/engines/engine.py.326: process: 0.28627 | load: 0.00016 [I 2022-10-26 10:14:52] ...trap/engines/engine.py.321: eval: epoch 1 | batch 220/510 [I 2022-10-26 10:14:52] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 10:14:52] ...trap/engines/engine.py.326: process: 0.29024 | load: 0.00056 [I 2022-10-26 10:14:55] ...trap/engines/engine.py.321: eval: epoch 1 | batch 230/510 [I 2022-10-26 10:14:55] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:22 [I 2022-10-26 10:14:55] ...trap/engines/engine.py.326: process: 0.28563 | load: 0.00014 [I 2022-10-26 10:14:58] ...trap/engines/engine.py.321: eval: epoch 1 | batch 240/510 [I 2022-10-26 10:14:58] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:18 [I 2022-10-26 10:14:58] ...trap/engines/engine.py.326: process: 0.29019 | load: 0.00015 [I 2022-10-26 10:15:01] ...trap/engines/engine.py.321: eval: epoch 1 | batch 250/510 [I 2022-10-26 10:15:01] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:14 [I 2022-10-26 10:15:01] ...trap/engines/engine.py.326: process: 0.27976 | load: 0.00016 [I 2022-10-26 10:15:04] ...trap/engines/engine.py.321: eval: epoch 1 | batch 260/510 [I 2022-10-26 10:15:04] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:13 [I 2022-10-26 10:15:04] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00016 [I 2022-10-26 10:15:06] ...trap/engines/engine.py.321: eval: epoch 1 | batch 270/510 [I 2022-10-26 10:15:06] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 10:15:06] ...trap/engines/engine.py.326: process: 0.28774 | load: 0.00016 [I 2022-10-26 10:15:09] ...trap/engines/engine.py.321: eval: epoch 1 | batch 280/510 [I 2022-10-26 10:15:09] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:05 [I 2022-10-26 10:15:09] ...trap/engines/engine.py.326: process: 0.28585 | load: 0.00015 [I 2022-10-26 10:15:12] ...trap/engines/engine.py.321: eval: epoch 1 | batch 290/510 [I 2022-10-26 10:15:12] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 10:15:12] ...trap/engines/engine.py.326: process: 0.29053 | load: 0.00015 [I 2022-10-26 10:15:15] ...trap/engines/engine.py.321: eval: epoch 1 | batch 300/510 [I 2022-10-26 10:15:15] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 10:15:15] ...trap/engines/engine.py.326: process: 0.28909 | load: 0.00015 [I 2022-10-26 10:15:18] ...trap/engines/engine.py.321: eval: epoch 1 | batch 310/510 [I 2022-10-26 10:15:18] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 10:15:18] ...trap/engines/engine.py.326: process: 0.28985 | load: 0.00015 [I 2022-10-26 10:15:21] ...trap/engines/engine.py.321: eval: epoch 1 | batch 320/510 [I 2022-10-26 10:15:21] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 10:15:21] ...trap/engines/engine.py.326: process: 0.28953 | load: 0.00015 [I 2022-10-26 10:15:24] ...trap/engines/engine.py.321: eval: epoch 1 | batch 330/510 [I 2022-10-26 10:15:24] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:51 [I 2022-10-26 10:15:24] ...trap/engines/engine.py.326: process: 0.28874 | load: 0.00015 [I 2022-10-26 10:15:27] ...trap/engines/engine.py.321: eval: epoch 1 | batch 340/510 [I 2022-10-26 10:15:27] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:49 [I 2022-10-26 10:15:27] ...trap/engines/engine.py.326: process: 0.28680 | load: 0.00019 [I 2022-10-26 10:15:29] ...trap/engines/engine.py.321: eval: epoch 1 | batch 350/510 [I 2022-10-26 10:15:29] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:45 [I 2022-10-26 10:15:29] ...trap/engines/engine.py.326: process: 0.28454 | load: 0.00024 [I 2022-10-26 10:15:32] ...trap/engines/engine.py.321: eval: epoch 1 | batch 360/510 [I 2022-10-26 10:15:32] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 10:15:32] ...trap/engines/engine.py.326: process: 0.29310 | load: 0.00014 [I 2022-10-26 10:15:35] ...trap/engines/engine.py.321: eval: epoch 1 | batch 370/510 [I 2022-10-26 10:15:35] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 10:15:35] ...trap/engines/engine.py.326: process: 0.30404 | load: 0.00020 [I 2022-10-26 10:15:38] ...trap/engines/engine.py.321: eval: epoch 1 | batch 380/510 [I 2022-10-26 10:15:38] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 10:15:38] ...trap/engines/engine.py.326: process: 0.28945 | load: 0.00015 [I 2022-10-26 10:15:41] ...trap/engines/engine.py.321: eval: epoch 1 | batch 390/510 [I 2022-10-26 10:15:41] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 10:15:41] ...trap/engines/engine.py.326: process: 0.29080 | load: 0.00016 [I 2022-10-26 10:15:44] ...trap/engines/engine.py.321: eval: epoch 1 | batch 400/510 [I 2022-10-26 10:15:44] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:32 [I 2022-10-26 10:15:44] ...trap/engines/engine.py.326: process: 0.28329 | load: 0.00019 [I 2022-10-26 10:15:47] ...trap/engines/engine.py.321: eval: epoch 1 | batch 410/510 [I 2022-10-26 10:15:47] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 10:15:47] ...trap/engines/engine.py.326: process: 0.28837 | load: 0.00018 [I 2022-10-26 10:15:50] ...trap/engines/engine.py.321: eval: epoch 1 | batch 420/510 [I 2022-10-26 10:15:50] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-26 10:15:50] ...trap/engines/engine.py.326: process: 0.29228 | load: 0.00015 [I 2022-10-26 10:15:53] ...trap/engines/engine.py.321: eval: epoch 1 | batch 430/510 [I 2022-10-26 10:15:53] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:22 [I 2022-10-26 10:15:53] ...trap/engines/engine.py.326: process: 0.28546 | load: 0.00015 [I 2022-10-26 10:15:55] ...trap/engines/engine.py.321: eval: epoch 1 | batch 440/510 [I 2022-10-26 10:15:55] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:19 [I 2022-10-26 10:15:55] ...trap/engines/engine.py.326: process: 0.27880 | load: 0.00015 [I 2022-10-26 10:15:58] ...trap/engines/engine.py.321: eval: epoch 1 | batch 450/510 [I 2022-10-26 10:15:58] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 10:15:58] ...trap/engines/engine.py.326: process: 0.28862 | load: 0.00014 [I 2022-10-26 10:16:01] ...trap/engines/engine.py.321: eval: epoch 1 | batch 460/510 [I 2022-10-26 10:16:01] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 10:16:01] ...trap/engines/engine.py.326: process: 0.28877 | load: 0.00015 [I 2022-10-26 10:16:04] ...trap/engines/engine.py.321: eval: epoch 1 | batch 470/510 [I 2022-10-26 10:16:04] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 10:16:04] ...trap/engines/engine.py.326: process: 0.28673 | load: 0.00015 [I 2022-10-26 10:16:07] ...trap/engines/engine.py.321: eval: epoch 1 | batch 480/510 [I 2022-10-26 10:16:07] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 10:16:07] ...trap/engines/engine.py.326: process: 0.28610 | load: 0.00014 [I 2022-10-26 10:16:10] ...trap/engines/engine.py.321: eval: epoch 1 | batch 490/510 [I 2022-10-26 10:16:10] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 10:16:10] ...trap/engines/engine.py.326: process: 0.28677 | load: 0.00021 [I 2022-10-26 10:16:13] ...trap/engines/engine.py.321: eval: epoch 1 | batch 500/510 [I 2022-10-26 10:16:13] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-26 10:16:13] ...trap/engines/engine.py.326: process: 0.28184 | load: 0.00016 [I 2022-10-26 10:16:15] ...trap/engines/engine.py.321: eval: epoch 1 | batch 510/510 [I 2022-10-26 10:16:15] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 10:16:15] ...trap/engines/engine.py.326: process: 0.28876 | load: 0.00011 [S 2022-10-26 10:16:16] ...trap/engines/engine.py.132: eval_epoch.epoch: 1 [I 2022-10-26 10:16:16] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 3.10 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 3.00 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.302800 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.622100 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.737900 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.323900 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.636800 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.746400 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.300000 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009631 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.015852 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007556 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013034 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.014134 [S 2022-10-26 10:16:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.014403 [I 2022-10-26 10:16:21] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-26 10:16:21] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 10:16:21] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 10:16:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 10:16:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 10:16:21] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 10:16:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 10:16:29] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 10:16:29] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 10:16:29] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 10:16:31] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 10:16:36] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 10:16:36] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 10:16:36] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 10:16:38] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 10:16:43] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 10:16:43] ...trap/engines/engine.py.163: Training model on trainset for epoch 2 [I 2022-10-26 10:16:43] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 10:16:46] ...trap/engines/engine.py.224: train: epoch 2 | batch 0/2304 [I 2022-10-26 10:16:46] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:17 [I 2022-10-26 10:16:46] ...trap/engines/engine.py.229: process: 2.71589 | load: 2.15640 [I 2022-10-26 10:16:46] ...trap/engines/engine.py.230: loss: 0.37574 [I 2022-10-26 10:16:46] ...optimizers/trijoint.py.151: total_norm: 4.023621 [I 2022-10-26 10:16:51] ...trap/engines/engine.py.224: train: epoch 2 | batch 10/2304 [I 2022-10-26 10:16:51] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:14 [I 2022-10-26 10:16:51] ...trap/engines/engine.py.229: process: 0.49662 | load: 0.00425 [I 2022-10-26 10:16:51] ...trap/engines/engine.py.230: loss: 0.45433 [I 2022-10-26 10:16:51] ...optimizers/trijoint.py.151: total_norm: 4.739419 [I 2022-10-26 10:16:56] ...trap/engines/engine.py.224: train: epoch 2 | batch 20/2304 [I 2022-10-26 10:16:56] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:25 [I 2022-10-26 10:16:56] ...trap/engines/engine.py.229: process: 0.49865 | load: 0.00374 [I 2022-10-26 10:16:56] ...trap/engines/engine.py.230: loss: 0.40543 [I 2022-10-26 10:16:56] ...optimizers/trijoint.py.151: total_norm: 4.490154 [I 2022-10-26 10:17:01] ...trap/engines/engine.py.224: train: epoch 2 | batch 30/2304 [I 2022-10-26 10:17:01] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:19 [I 2022-10-26 10:17:01] ...trap/engines/engine.py.229: process: 0.50115 | load: 0.00317 [I 2022-10-26 10:17:01] ...trap/engines/engine.py.230: loss: 0.33766 [I 2022-10-26 10:17:01] ...optimizers/trijoint.py.151: total_norm: 3.936001 [I 2022-10-26 10:17:06] ...trap/engines/engine.py.224: train: epoch 2 | batch 40/2304 [I 2022-10-26 10:17:06] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:19:11 [I 2022-10-26 10:17:06] ...trap/engines/engine.py.229: process: 0.51041 | load: 0.00299 [I 2022-10-26 10:17:06] ...trap/engines/engine.py.230: loss: 0.26216 [I 2022-10-26 10:17:06] ...optimizers/trijoint.py.151: total_norm: 3.568606 [I 2022-10-26 10:17:11] ...trap/engines/engine.py.224: train: epoch 2 | batch 50/2304 [I 2022-10-26 10:17:11] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:19:02 [I 2022-10-26 10:17:11] ...trap/engines/engine.py.229: process: 0.50414 | load: 0.00309 [I 2022-10-26 10:17:11] ...trap/engines/engine.py.230: loss: 0.35845 [I 2022-10-26 10:17:11] ...optimizers/trijoint.py.151: total_norm: 4.494561 [I 2022-10-26 10:17:17] ...trap/engines/engine.py.224: train: epoch 2 | batch 60/2304 [I 2022-10-26 10:17:17] ...trap/engines/engine.py.228: elapsed: 0:00:33 | left: 0:19:14 [I 2022-10-26 10:17:17] ...trap/engines/engine.py.229: process: 0.52734 | load: 0.00283 [I 2022-10-26 10:17:17] ...trap/engines/engine.py.230: loss: 0.45103 [I 2022-10-26 10:17:17] ...optimizers/trijoint.py.151: total_norm: 5.946507 [I 2022-10-26 10:17:22] ...trap/engines/engine.py.224: train: epoch 2 | batch 70/2304 [I 2022-10-26 10:17:22] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:18:41 [I 2022-10-26 10:17:22] ...trap/engines/engine.py.229: process: 0.49716 | load: 0.00267 [I 2022-10-26 10:17:22] ...trap/engines/engine.py.230: loss: 0.40337 [I 2022-10-26 10:17:22] ...optimizers/trijoint.py.151: total_norm: 4.147094 [I 2022-10-26 10:17:27] ...trap/engines/engine.py.224: train: epoch 2 | batch 80/2304 [I 2022-10-26 10:17:27] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:18:46 [I 2022-10-26 10:17:27] ...trap/engines/engine.py.229: process: 0.51711 | load: 0.00307 [I 2022-10-26 10:17:27] ...trap/engines/engine.py.230: loss: 0.47011 [I 2022-10-26 10:17:27] ...optimizers/trijoint.py.151: total_norm: 6.794570 [I 2022-10-26 10:17:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 90/2304 [I 2022-10-26 10:17:32] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:18:59 [I 2022-10-26 10:17:32] ...trap/engines/engine.py.229: process: 0.51936 | load: 0.00261 [I 2022-10-26 10:17:32] ...trap/engines/engine.py.230: loss: 0.32550 [I 2022-10-26 10:17:32] ...optimizers/trijoint.py.151: total_norm: 4.005579 [I 2022-10-26 10:17:37] ...trap/engines/engine.py.224: train: epoch 2 | batch 100/2304 [I 2022-10-26 10:17:37] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:19:00 [I 2022-10-26 10:17:37] ...trap/engines/engine.py.229: process: 0.52211 | load: 0.00291 [I 2022-10-26 10:17:37] ...trap/engines/engine.py.230: loss: 0.45823 [I 2022-10-26 10:17:37] ...optimizers/trijoint.py.151: total_norm: 4.606919 [I 2022-10-26 10:17:42] ...trap/engines/engine.py.224: train: epoch 2 | batch 110/2304 [I 2022-10-26 10:17:42] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:18:28 [I 2022-10-26 10:17:42] ...trap/engines/engine.py.229: process: 0.50626 | load: 0.00275 [I 2022-10-26 10:17:42] ...trap/engines/engine.py.230: loss: 0.43413 [I 2022-10-26 10:17:42] ...optimizers/trijoint.py.151: total_norm: 4.769399 [I 2022-10-26 10:17:47] ...trap/engines/engine.py.224: train: epoch 2 | batch 120/2304 [I 2022-10-26 10:17:47] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:04 [I 2022-10-26 10:17:47] ...trap/engines/engine.py.229: process: 0.49385 | load: 0.00273 [I 2022-10-26 10:17:47] ...trap/engines/engine.py.230: loss: 0.34024 [I 2022-10-26 10:17:47] ...optimizers/trijoint.py.151: total_norm: 4.316635 [I 2022-10-26 10:17:52] ...trap/engines/engine.py.224: train: epoch 2 | batch 130/2304 [I 2022-10-26 10:17:52] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:02 [I 2022-10-26 10:17:52] ...trap/engines/engine.py.229: process: 0.48962 | load: 0.00256 [I 2022-10-26 10:17:52] ...trap/engines/engine.py.230: loss: 0.30622 [I 2022-10-26 10:17:52] ...optimizers/trijoint.py.151: total_norm: 4.825583 [I 2022-10-26 10:17:57] ...trap/engines/engine.py.224: train: epoch 2 | batch 140/2304 [I 2022-10-26 10:17:57] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:18:03 [I 2022-10-26 10:17:57] ...trap/engines/engine.py.229: process: 0.50193 | load: 0.00254 [I 2022-10-26 10:17:57] ...trap/engines/engine.py.230: loss: 0.26963 [I 2022-10-26 10:17:57] ...optimizers/trijoint.py.151: total_norm: 3.254874 [I 2022-10-26 10:18:02] ...trap/engines/engine.py.224: train: epoch 2 | batch 150/2304 [I 2022-10-26 10:18:02] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:17:48 [I 2022-10-26 10:18:02] ...trap/engines/engine.py.229: process: 0.49504 | load: 0.00257 [I 2022-10-26 10:18:02] ...trap/engines/engine.py.230: loss: 0.27968 [I 2022-10-26 10:18:02] ...optimizers/trijoint.py.151: total_norm: 3.790067 [I 2022-10-26 10:18:07] ...trap/engines/engine.py.224: train: epoch 2 | batch 160/2304 [I 2022-10-26 10:18:07] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:17:47 [I 2022-10-26 10:18:07] ...trap/engines/engine.py.229: process: 0.49736 | load: 0.00265 [I 2022-10-26 10:18:07] ...trap/engines/engine.py.230: loss: 0.48821 [I 2022-10-26 10:18:07] ...optimizers/trijoint.py.151: total_norm: 6.062872 [I 2022-10-26 10:18:12] ...trap/engines/engine.py.224: train: epoch 2 | batch 170/2304 [I 2022-10-26 10:18:12] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:51 [I 2022-10-26 10:18:12] ...trap/engines/engine.py.229: process: 0.49520 | load: 0.00257 [I 2022-10-26 10:18:12] ...trap/engines/engine.py.230: loss: 0.31086 [I 2022-10-26 10:18:12] ...optimizers/trijoint.py.151: total_norm: 4.460688 [I 2022-10-26 10:18:17] ...trap/engines/engine.py.224: train: epoch 2 | batch 180/2304 [I 2022-10-26 10:18:17] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:17:48 [I 2022-10-26 10:18:17] ...trap/engines/engine.py.229: process: 0.50631 | load: 0.00251 [I 2022-10-26 10:18:17] ...trap/engines/engine.py.230: loss: 0.39249 [I 2022-10-26 10:18:17] ...optimizers/trijoint.py.151: total_norm: 5.300549 [I 2022-10-26 10:18:22] ...trap/engines/engine.py.224: train: epoch 2 | batch 190/2304 [I 2022-10-26 10:18:22] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:17:41 [I 2022-10-26 10:18:22] ...trap/engines/engine.py.229: process: 0.50139 | load: 0.00255 [I 2022-10-26 10:18:22] ...trap/engines/engine.py.230: loss: 0.40381 [I 2022-10-26 10:18:22] ...optimizers/trijoint.py.151: total_norm: 4.163960 [I 2022-10-26 10:18:27] ...trap/engines/engine.py.224: train: epoch 2 | batch 200/2304 [I 2022-10-26 10:18:27] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:49 [I 2022-10-26 10:18:27] ...trap/engines/engine.py.229: process: 0.51495 | load: 0.00254 [I 2022-10-26 10:18:27] ...trap/engines/engine.py.230: loss: 0.33949 [I 2022-10-26 10:18:27] ...optimizers/trijoint.py.151: total_norm: 4.478261 [I 2022-10-26 10:18:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 210/2304 [I 2022-10-26 10:18:33] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:17:55 [I 2022-10-26 10:18:33] ...trap/engines/engine.py.229: process: 0.51195 | load: 0.00248 [I 2022-10-26 10:18:33] ...trap/engines/engine.py.230: loss: 0.37455 [I 2022-10-26 10:18:33] ...optimizers/trijoint.py.151: total_norm: 4.148418 [I 2022-10-26 10:18:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 220/2304 [I 2022-10-26 10:18:38] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:17:33 [I 2022-10-26 10:18:38] ...trap/engines/engine.py.229: process: 0.50632 | load: 0.00250 [I 2022-10-26 10:18:38] ...trap/engines/engine.py.230: loss: 0.27278 [I 2022-10-26 10:18:38] ...optimizers/trijoint.py.151: total_norm: 3.660230 [I 2022-10-26 10:18:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 230/2304 [I 2022-10-26 10:18:43] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:17:41 [I 2022-10-26 10:18:43] ...trap/engines/engine.py.229: process: 0.50653 | load: 0.00248 [I 2022-10-26 10:18:43] ...trap/engines/engine.py.230: loss: 0.29962 [I 2022-10-26 10:18:43] ...optimizers/trijoint.py.151: total_norm: 4.160870 [I 2022-10-26 10:18:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 240/2304 [I 2022-10-26 10:18:48] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:17:30 [I 2022-10-26 10:18:48] ...trap/engines/engine.py.229: process: 0.50797 | load: 0.00286 [I 2022-10-26 10:18:48] ...trap/engines/engine.py.230: loss: 0.36867 [I 2022-10-26 10:18:48] ...optimizers/trijoint.py.151: total_norm: 4.927149 [I 2022-10-26 10:18:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 250/2304 [I 2022-10-26 10:18:53] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:17:09 [I 2022-10-26 10:18:53] ...trap/engines/engine.py.229: process: 0.50365 | load: 0.00248 [I 2022-10-26 10:18:53] ...trap/engines/engine.py.230: loss: 0.36727 [I 2022-10-26 10:18:53] ...optimizers/trijoint.py.151: total_norm: 5.253710 [I 2022-10-26 10:18:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 260/2304 [I 2022-10-26 10:18:58] ...trap/engines/engine.py.228: elapsed: 0:02:14 | left: 0:16:55 [I 2022-10-26 10:18:58] ...trap/engines/engine.py.229: process: 0.49339 | load: 0.00252 [I 2022-10-26 10:18:58] ...trap/engines/engine.py.230: loss: 0.32644 [I 2022-10-26 10:18:58] ...optimizers/trijoint.py.151: total_norm: 5.207793 [I 2022-10-26 10:19:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 270/2304 [I 2022-10-26 10:19:03] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:16:55 [I 2022-10-26 10:19:03] ...trap/engines/engine.py.229: process: 0.50240 | load: 0.00244 [I 2022-10-26 10:19:03] ...trap/engines/engine.py.230: loss: 0.31922 [I 2022-10-26 10:19:03] ...optimizers/trijoint.py.151: total_norm: 4.086229 [I 2022-10-26 10:19:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 280/2304 [I 2022-10-26 10:19:08] ...trap/engines/engine.py.228: elapsed: 0:02:24 | left: 0:16:53 [I 2022-10-26 10:19:08] ...trap/engines/engine.py.229: process: 0.50344 | load: 0.00245 [I 2022-10-26 10:19:08] ...trap/engines/engine.py.230: loss: 0.29374 [I 2022-10-26 10:19:08] ...optimizers/trijoint.py.151: total_norm: 3.643251 [I 2022-10-26 10:19:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 290/2304 [I 2022-10-26 10:19:13] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:17:00 [I 2022-10-26 10:19:13] ...trap/engines/engine.py.229: process: 0.51737 | load: 0.00244 [I 2022-10-26 10:19:13] ...trap/engines/engine.py.230: loss: 0.40202 [I 2022-10-26 10:19:13] ...optimizers/trijoint.py.151: total_norm: 4.071835 [I 2022-10-26 10:19:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 300/2304 [I 2022-10-26 10:19:18] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:16:48 [I 2022-10-26 10:19:18] ...trap/engines/engine.py.229: process: 0.48725 | load: 0.00285 [I 2022-10-26 10:19:18] ...trap/engines/engine.py.230: loss: 0.30436 [I 2022-10-26 10:19:18] ...optimizers/trijoint.py.151: total_norm: 4.083552 [I 2022-10-26 10:19:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 310/2304 [I 2022-10-26 10:19:23] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:16:51 [I 2022-10-26 10:19:23] ...trap/engines/engine.py.229: process: 0.49566 | load: 0.00262 [I 2022-10-26 10:19:23] ...trap/engines/engine.py.230: loss: 0.41360 [I 2022-10-26 10:19:23] ...optimizers/trijoint.py.151: total_norm: 6.258076 [I 2022-10-26 10:19:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 320/2304 [I 2022-10-26 10:19:28] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:16:37 [I 2022-10-26 10:19:28] ...trap/engines/engine.py.229: process: 0.50503 | load: 0.00246 [I 2022-10-26 10:19:28] ...trap/engines/engine.py.230: loss: 0.33039 [I 2022-10-26 10:19:28] ...optimizers/trijoint.py.151: total_norm: 3.439393 [I 2022-10-26 10:19:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 330/2304 [I 2022-10-26 10:19:33] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:16:18 [I 2022-10-26 10:19:33] ...trap/engines/engine.py.229: process: 0.48053 | load: 0.00256 [I 2022-10-26 10:19:33] ...trap/engines/engine.py.230: loss: 0.38184 [I 2022-10-26 10:19:33] ...optimizers/trijoint.py.151: total_norm: 5.273246 [I 2022-10-26 10:19:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 340/2304 [I 2022-10-26 10:19:38] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:16:16 [I 2022-10-26 10:19:38] ...trap/engines/engine.py.229: process: 0.49935 | load: 0.00244 [I 2022-10-26 10:19:38] ...trap/engines/engine.py.230: loss: 0.39463 [I 2022-10-26 10:19:38] ...optimizers/trijoint.py.151: total_norm: 4.174388 [I 2022-10-26 10:19:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 350/2304 [I 2022-10-26 10:19:43] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:16:11 [I 2022-10-26 10:19:43] ...trap/engines/engine.py.229: process: 0.49188 | load: 0.00246 [I 2022-10-26 10:19:43] ...trap/engines/engine.py.230: loss: 0.34993 [I 2022-10-26 10:19:43] ...optimizers/trijoint.py.151: total_norm: 4.315186 [I 2022-10-26 10:19:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 360/2304 [I 2022-10-26 10:19:48] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:16:02 [I 2022-10-26 10:19:48] ...trap/engines/engine.py.229: process: 0.49394 | load: 0.00244 [I 2022-10-26 10:19:48] ...trap/engines/engine.py.230: loss: 0.37737 [I 2022-10-26 10:19:48] ...optimizers/trijoint.py.151: total_norm: 3.753354 [I 2022-10-26 10:19:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 370/2304 [I 2022-10-26 10:19:53] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:16:13 [I 2022-10-26 10:19:53] ...trap/engines/engine.py.229: process: 0.49987 | load: 0.00254 [I 2022-10-26 10:19:53] ...trap/engines/engine.py.230: loss: 0.33143 [I 2022-10-26 10:19:53] ...optimizers/trijoint.py.151: total_norm: 4.570277 [I 2022-10-26 10:19:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 380/2304 [I 2022-10-26 10:19:58] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:16:17 [I 2022-10-26 10:19:58] ...trap/engines/engine.py.229: process: 0.50229 | load: 0.00248 [I 2022-10-26 10:19:58] ...trap/engines/engine.py.230: loss: 0.33487 [I 2022-10-26 10:19:58] ...optimizers/trijoint.py.151: total_norm: 4.740717 [I 2022-10-26 10:20:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 390/2304 [I 2022-10-26 10:20:03] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:16:00 [I 2022-10-26 10:20:03] ...trap/engines/engine.py.229: process: 0.51650 | load: 0.00246 [I 2022-10-26 10:20:03] ...trap/engines/engine.py.230: loss: 0.30363 [I 2022-10-26 10:20:03] ...optimizers/trijoint.py.151: total_norm: 4.775984 [I 2022-10-26 10:20:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 400/2304 [I 2022-10-26 10:20:08] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:15:47 [I 2022-10-26 10:20:08] ...trap/engines/engine.py.229: process: 0.49724 | load: 0.00244 [I 2022-10-26 10:20:08] ...trap/engines/engine.py.230: loss: 0.44589 [I 2022-10-26 10:20:08] ...optimizers/trijoint.py.151: total_norm: 4.908936 [I 2022-10-26 10:20:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 410/2304 [I 2022-10-26 10:20:13] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:15:29 [I 2022-10-26 10:20:13] ...trap/engines/engine.py.229: process: 0.49117 | load: 0.00241 [I 2022-10-26 10:20:13] ...trap/engines/engine.py.230: loss: 0.30002 [I 2022-10-26 10:20:13] ...optimizers/trijoint.py.151: total_norm: 4.051209 [I 2022-10-26 10:20:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 420/2304 [I 2022-10-26 10:20:18] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:15:30 [I 2022-10-26 10:20:18] ...trap/engines/engine.py.229: process: 0.49344 | load: 0.00249 [I 2022-10-26 10:20:18] ...trap/engines/engine.py.230: loss: 0.35306 [I 2022-10-26 10:20:18] ...optimizers/trijoint.py.151: total_norm: 3.281825 [I 2022-10-26 10:20:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 430/2304 [I 2022-10-26 10:20:23] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:15:31 [I 2022-10-26 10:20:23] ...trap/engines/engine.py.229: process: 0.50292 | load: 0.00238 [I 2022-10-26 10:20:23] ...trap/engines/engine.py.230: loss: 0.26657 [I 2022-10-26 10:20:23] ...optimizers/trijoint.py.151: total_norm: 3.380479 [I 2022-10-26 10:20:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 440/2304 [I 2022-10-26 10:20:28] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:15:33 [I 2022-10-26 10:20:28] ...trap/engines/engine.py.229: process: 0.49826 | load: 0.00252 [I 2022-10-26 10:20:28] ...trap/engines/engine.py.230: loss: 0.27440 [I 2022-10-26 10:20:28] ...optimizers/trijoint.py.151: total_norm: 4.213415 [I 2022-10-26 10:20:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 450/2304 [I 2022-10-26 10:20:33] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:15:18 [I 2022-10-26 10:20:33] ...trap/engines/engine.py.229: process: 0.50204 | load: 0.00247 [I 2022-10-26 10:20:33] ...trap/engines/engine.py.230: loss: 0.38359 [I 2022-10-26 10:20:33] ...optimizers/trijoint.py.151: total_norm: 4.678886 [I 2022-10-26 10:20:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 460/2304 [I 2022-10-26 10:20:38] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:15:16 [I 2022-10-26 10:20:38] ...trap/engines/engine.py.229: process: 0.49276 | load: 0.00245 [I 2022-10-26 10:20:38] ...trap/engines/engine.py.230: loss: 0.30037 [I 2022-10-26 10:20:38] ...optimizers/trijoint.py.151: total_norm: 4.391195 [I 2022-10-26 10:20:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 470/2304 [I 2022-10-26 10:20:43] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:15:07 [I 2022-10-26 10:20:43] ...trap/engines/engine.py.229: process: 0.47559 | load: 0.00245 [I 2022-10-26 10:20:43] ...trap/engines/engine.py.230: loss: 0.33483 [I 2022-10-26 10:20:43] ...optimizers/trijoint.py.151: total_norm: 5.051487 [I 2022-10-26 10:20:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 480/2304 [I 2022-10-26 10:20:48] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:15:10 [I 2022-10-26 10:20:48] ...trap/engines/engine.py.229: process: 0.50176 | load: 0.00246 [I 2022-10-26 10:20:48] ...trap/engines/engine.py.230: loss: 0.36920 [I 2022-10-26 10:20:48] ...optimizers/trijoint.py.151: total_norm: 4.495469 [I 2022-10-26 10:20:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 490/2304 [I 2022-10-26 10:20:53] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:15:12 [I 2022-10-26 10:20:53] ...trap/engines/engine.py.229: process: 0.50553 | load: 0.00246 [I 2022-10-26 10:20:53] ...trap/engines/engine.py.230: loss: 0.48488 [I 2022-10-26 10:20:53] ...optimizers/trijoint.py.151: total_norm: 4.920585 [I 2022-10-26 10:20:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 500/2304 [I 2022-10-26 10:20:58] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:15:07 [I 2022-10-26 10:20:58] ...trap/engines/engine.py.229: process: 0.50619 | load: 0.00247 [I 2022-10-26 10:20:58] ...trap/engines/engine.py.230: loss: 0.34203 [I 2022-10-26 10:20:58] ...optimizers/trijoint.py.151: total_norm: 4.892538 [I 2022-10-26 10:21:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 510/2304 [I 2022-10-26 10:21:03] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:14:47 [I 2022-10-26 10:21:03] ...trap/engines/engine.py.229: process: 0.50036 | load: 0.00255 [I 2022-10-26 10:21:03] ...trap/engines/engine.py.230: loss: 0.46315 [I 2022-10-26 10:21:03] ...optimizers/trijoint.py.151: total_norm: 5.388461 [I 2022-10-26 10:21:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 520/2304 [I 2022-10-26 10:21:08] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:14:49 [I 2022-10-26 10:21:08] ...trap/engines/engine.py.229: process: 0.50490 | load: 0.00243 [I 2022-10-26 10:21:08] ...trap/engines/engine.py.230: loss: 0.31215 [I 2022-10-26 10:21:08] ...optimizers/trijoint.py.151: total_norm: 4.532295 [I 2022-10-26 10:21:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 530/2304 [I 2022-10-26 10:21:13] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:14:46 [I 2022-10-26 10:21:13] ...trap/engines/engine.py.229: process: 0.50858 | load: 0.00245 [I 2022-10-26 10:21:13] ...trap/engines/engine.py.230: loss: 0.34650 [I 2022-10-26 10:21:13] ...optimizers/trijoint.py.151: total_norm: 5.404676 [I 2022-10-26 10:21:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 540/2304 [I 2022-10-26 10:21:18] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:14:37 [I 2022-10-26 10:21:18] ...trap/engines/engine.py.229: process: 0.49414 | load: 0.00251 [I 2022-10-26 10:21:18] ...trap/engines/engine.py.230: loss: 0.36451 [I 2022-10-26 10:21:18] ...optimizers/trijoint.py.151: total_norm: 4.733621 [I 2022-10-26 10:21:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 550/2304 [I 2022-10-26 10:21:23] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:14:34 [I 2022-10-26 10:21:23] ...trap/engines/engine.py.229: process: 0.50083 | load: 0.00247 [I 2022-10-26 10:21:23] ...trap/engines/engine.py.230: loss: 0.31265 [I 2022-10-26 10:21:23] ...optimizers/trijoint.py.151: total_norm: 3.646485 [I 2022-10-26 10:21:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 560/2304 [I 2022-10-26 10:21:28] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:14:27 [I 2022-10-26 10:21:28] ...trap/engines/engine.py.229: process: 0.49735 | load: 0.00244 [I 2022-10-26 10:21:28] ...trap/engines/engine.py.230: loss: 0.37084 [I 2022-10-26 10:21:28] ...optimizers/trijoint.py.151: total_norm: 4.291998 [I 2022-10-26 10:21:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 570/2304 [I 2022-10-26 10:21:33] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:14:17 [I 2022-10-26 10:21:33] ...trap/engines/engine.py.229: process: 0.49444 | load: 0.00245 [I 2022-10-26 10:21:33] ...trap/engines/engine.py.230: loss: 0.32580 [I 2022-10-26 10:21:33] ...optimizers/trijoint.py.151: total_norm: 3.824925 [I 2022-10-26 10:21:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 580/2304 [I 2022-10-26 10:21:38] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:14:15 [I 2022-10-26 10:21:38] ...trap/engines/engine.py.229: process: 0.50009 | load: 0.00243 [I 2022-10-26 10:21:38] ...trap/engines/engine.py.230: loss: 0.31809 [I 2022-10-26 10:21:38] ...optimizers/trijoint.py.151: total_norm: 4.046407 [I 2022-10-26 10:21:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 590/2304 [I 2022-10-26 10:21:43] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:14:10 [I 2022-10-26 10:21:43] ...trap/engines/engine.py.229: process: 0.49618 | load: 0.00244 [I 2022-10-26 10:21:43] ...trap/engines/engine.py.230: loss: 0.35293 [I 2022-10-26 10:21:43] ...optimizers/trijoint.py.151: total_norm: 4.389876 [I 2022-10-26 10:21:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 600/2304 [I 2022-10-26 10:21:48] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:14:16 [I 2022-10-26 10:21:48] ...trap/engines/engine.py.229: process: 0.50853 | load: 0.00256 [I 2022-10-26 10:21:48] ...trap/engines/engine.py.230: loss: 0.29626 [I 2022-10-26 10:21:48] ...optimizers/trijoint.py.151: total_norm: 3.967680 [I 2022-10-26 10:21:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 610/2304 [I 2022-10-26 10:21:53] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:14:01 [I 2022-10-26 10:21:53] ...trap/engines/engine.py.229: process: 0.49305 | load: 0.00245 [I 2022-10-26 10:21:53] ...trap/engines/engine.py.230: loss: 0.29359 [I 2022-10-26 10:21:53] ...optimizers/trijoint.py.151: total_norm: 4.259235 [I 2022-10-26 10:21:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 620/2304 [I 2022-10-26 10:21:58] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:14:01 [I 2022-10-26 10:21:58] ...trap/engines/engine.py.229: process: 0.50226 | load: 0.00245 [I 2022-10-26 10:21:58] ...trap/engines/engine.py.230: loss: 0.38227 [I 2022-10-26 10:21:58] ...optimizers/trijoint.py.151: total_norm: 3.832661 [I 2022-10-26 10:22:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 630/2304 [I 2022-10-26 10:22:03] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:14:00 [I 2022-10-26 10:22:03] ...trap/engines/engine.py.229: process: 0.50771 | load: 0.00242 [I 2022-10-26 10:22:03] ...trap/engines/engine.py.230: loss: 0.41102 [I 2022-10-26 10:22:03] ...optimizers/trijoint.py.151: total_norm: 4.649594 [I 2022-10-26 10:22:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 640/2304 [I 2022-10-26 10:22:08] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:13:47 [I 2022-10-26 10:22:08] ...trap/engines/engine.py.229: process: 0.50142 | load: 0.00242 [I 2022-10-26 10:22:08] ...trap/engines/engine.py.230: loss: 0.36388 [I 2022-10-26 10:22:08] ...optimizers/trijoint.py.151: total_norm: 5.106826 [I 2022-10-26 10:22:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 650/2304 [I 2022-10-26 10:22:13] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:13:41 [I 2022-10-26 10:22:13] ...trap/engines/engine.py.229: process: 0.50354 | load: 0.00242 [I 2022-10-26 10:22:13] ...trap/engines/engine.py.230: loss: 0.34617 [I 2022-10-26 10:22:13] ...optimizers/trijoint.py.151: total_norm: 4.971793 [I 2022-10-26 10:22:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 660/2304 [I 2022-10-26 10:22:18] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:13:41 [I 2022-10-26 10:22:18] ...trap/engines/engine.py.229: process: 0.50621 | load: 0.00242 [I 2022-10-26 10:22:18] ...trap/engines/engine.py.230: loss: 0.41983 [I 2022-10-26 10:22:18] ...optimizers/trijoint.py.151: total_norm: 4.460811 [I 2022-10-26 10:22:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 670/2304 [I 2022-10-26 10:22:23] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:13:34 [I 2022-10-26 10:22:23] ...trap/engines/engine.py.229: process: 0.50039 | load: 0.00242 [I 2022-10-26 10:22:23] ...trap/engines/engine.py.230: loss: 0.40844 [I 2022-10-26 10:22:23] ...optimizers/trijoint.py.151: total_norm: 4.252997 [I 2022-10-26 10:22:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 680/2304 [I 2022-10-26 10:22:28] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:13:25 [I 2022-10-26 10:22:28] ...trap/engines/engine.py.229: process: 0.49595 | load: 0.00243 [I 2022-10-26 10:22:28] ...trap/engines/engine.py.230: loss: 0.40854 [I 2022-10-26 10:22:28] ...optimizers/trijoint.py.151: total_norm: 3.923782 [I 2022-10-26 10:22:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 690/2304 [I 2022-10-26 10:22:33] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:13:19 [I 2022-10-26 10:22:33] ...trap/engines/engine.py.229: process: 0.48228 | load: 0.00240 [I 2022-10-26 10:22:33] ...trap/engines/engine.py.230: loss: 0.35142 [I 2022-10-26 10:22:33] ...optimizers/trijoint.py.151: total_norm: 4.618488 [I 2022-10-26 10:22:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 700/2304 [I 2022-10-26 10:22:38] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:13:15 [I 2022-10-26 10:22:38] ...trap/engines/engine.py.229: process: 0.47916 | load: 0.00249 [I 2022-10-26 10:22:38] ...trap/engines/engine.py.230: loss: 0.44264 [I 2022-10-26 10:22:38] ...optimizers/trijoint.py.151: total_norm: 4.454980 [I 2022-10-26 10:22:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 710/2304 [I 2022-10-26 10:22:43] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:13:12 [I 2022-10-26 10:22:43] ...trap/engines/engine.py.229: process: 0.49587 | load: 0.00243 [I 2022-10-26 10:22:43] ...trap/engines/engine.py.230: loss: 0.36201 [I 2022-10-26 10:22:43] ...optimizers/trijoint.py.151: total_norm: 3.415882 [I 2022-10-26 10:22:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 720/2304 [I 2022-10-26 10:22:48] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:12:59 [I 2022-10-26 10:22:48] ...trap/engines/engine.py.229: process: 0.47806 | load: 0.00242 [I 2022-10-26 10:22:48] ...trap/engines/engine.py.230: loss: 0.42480 [I 2022-10-26 10:22:48] ...optimizers/trijoint.py.151: total_norm: 5.143857 [I 2022-10-26 10:22:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 730/2304 [I 2022-10-26 10:22:53] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:13:07 [I 2022-10-26 10:22:53] ...trap/engines/engine.py.229: process: 0.50082 | load: 0.00244 [I 2022-10-26 10:22:53] ...trap/engines/engine.py.230: loss: 0.25315 [I 2022-10-26 10:22:53] ...optimizers/trijoint.py.151: total_norm: 3.393831 [I 2022-10-26 10:22:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 740/2304 [I 2022-10-26 10:22:58] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:12:59 [I 2022-10-26 10:22:58] ...trap/engines/engine.py.229: process: 0.50940 | load: 0.00245 [I 2022-10-26 10:22:58] ...trap/engines/engine.py.230: loss: 0.32029 [I 2022-10-26 10:22:58] ...optimizers/trijoint.py.151: total_norm: 3.752808 [I 2022-10-26 10:23:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 750/2304 [I 2022-10-26 10:23:03] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:12:51 [I 2022-10-26 10:23:03] ...trap/engines/engine.py.229: process: 0.49364 | load: 0.00242 [I 2022-10-26 10:23:03] ...trap/engines/engine.py.230: loss: 0.36510 [I 2022-10-26 10:23:03] ...optimizers/trijoint.py.151: total_norm: 4.911533 [I 2022-10-26 10:23:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 760/2304 [I 2022-10-26 10:23:08] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:12:58 [I 2022-10-26 10:23:08] ...trap/engines/engine.py.229: process: 0.51557 | load: 0.00243 [I 2022-10-26 10:23:08] ...trap/engines/engine.py.230: loss: 0.39973 [I 2022-10-26 10:23:08] ...optimizers/trijoint.py.151: total_norm: 4.226429 [I 2022-10-26 10:23:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 770/2304 [I 2022-10-26 10:23:13] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:12:44 [I 2022-10-26 10:23:13] ...trap/engines/engine.py.229: process: 0.50504 | load: 0.00249 [I 2022-10-26 10:23:13] ...trap/engines/engine.py.230: loss: 0.42111 [I 2022-10-26 10:23:13] ...optimizers/trijoint.py.151: total_norm: 4.163265 [I 2022-10-26 10:23:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 780/2304 [I 2022-10-26 10:23:18] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:12:35 [I 2022-10-26 10:23:18] ...trap/engines/engine.py.229: process: 0.49979 | load: 0.00245 [I 2022-10-26 10:23:18] ...trap/engines/engine.py.230: loss: 0.45995 [I 2022-10-26 10:23:18] ...optimizers/trijoint.py.151: total_norm: 4.320973 [I 2022-10-26 10:23:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 790/2304 [I 2022-10-26 10:23:23] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:12:30 [I 2022-10-26 10:23:23] ...trap/engines/engine.py.229: process: 0.49748 | load: 0.00248 [I 2022-10-26 10:23:23] ...trap/engines/engine.py.230: loss: 0.32163 [I 2022-10-26 10:23:23] ...optimizers/trijoint.py.151: total_norm: 4.198467 [I 2022-10-26 10:23:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 800/2304 [I 2022-10-26 10:23:28] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:12:31 [I 2022-10-26 10:23:28] ...trap/engines/engine.py.229: process: 0.49498 | load: 0.00244 [I 2022-10-26 10:23:28] ...trap/engines/engine.py.230: loss: 0.29873 [I 2022-10-26 10:23:28] ...optimizers/trijoint.py.151: total_norm: 4.061179 [I 2022-10-26 10:23:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 810/2304 [I 2022-10-26 10:23:33] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:12:21 [I 2022-10-26 10:23:33] ...trap/engines/engine.py.229: process: 0.49578 | load: 0.00241 [I 2022-10-26 10:23:33] ...trap/engines/engine.py.230: loss: 0.28908 [I 2022-10-26 10:23:33] ...optimizers/trijoint.py.151: total_norm: 3.634864 [I 2022-10-26 10:23:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 820/2304 [I 2022-10-26 10:23:38] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:12:13 [I 2022-10-26 10:23:38] ...trap/engines/engine.py.229: process: 0.48793 | load: 0.00250 [I 2022-10-26 10:23:38] ...trap/engines/engine.py.230: loss: 0.38204 [I 2022-10-26 10:23:38] ...optimizers/trijoint.py.151: total_norm: 4.376987 [I 2022-10-26 10:23:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 830/2304 [I 2022-10-26 10:23:43] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:12:12 [I 2022-10-26 10:23:43] ...trap/engines/engine.py.229: process: 0.49760 | load: 0.00240 [I 2022-10-26 10:23:43] ...trap/engines/engine.py.230: loss: 0.31920 [I 2022-10-26 10:23:43] ...optimizers/trijoint.py.151: total_norm: 4.347280 [I 2022-10-26 10:23:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 840/2304 [I 2022-10-26 10:23:48] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:12:10 [I 2022-10-26 10:23:48] ...trap/engines/engine.py.229: process: 0.49741 | load: 0.00249 [I 2022-10-26 10:23:48] ...trap/engines/engine.py.230: loss: 0.31690 [I 2022-10-26 10:23:48] ...optimizers/trijoint.py.151: total_norm: 4.341025 [I 2022-10-26 10:23:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 850/2304 [I 2022-10-26 10:23:53] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:12:03 [I 2022-10-26 10:23:53] ...trap/engines/engine.py.229: process: 0.49652 | load: 0.00246 [I 2022-10-26 10:23:53] ...trap/engines/engine.py.230: loss: 0.44751 [I 2022-10-26 10:23:53] ...optimizers/trijoint.py.151: total_norm: 5.314078 [I 2022-10-26 10:23:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 860/2304 [I 2022-10-26 10:23:58] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:12:01 [I 2022-10-26 10:23:58] ...trap/engines/engine.py.229: process: 0.50384 | load: 0.00244 [I 2022-10-26 10:23:58] ...trap/engines/engine.py.230: loss: 0.39120 [I 2022-10-26 10:23:58] ...optimizers/trijoint.py.151: total_norm: 4.104160 [I 2022-10-26 10:24:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 870/2304 [I 2022-10-26 10:24:03] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:11:58 [I 2022-10-26 10:24:03] ...trap/engines/engine.py.229: process: 0.50856 | load: 0.00244 [I 2022-10-26 10:24:03] ...trap/engines/engine.py.230: loss: 0.37298 [I 2022-10-26 10:24:03] ...optimizers/trijoint.py.151: total_norm: 4.175174 [I 2022-10-26 10:24:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 880/2304 [I 2022-10-26 10:24:08] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:11:47 [I 2022-10-26 10:24:08] ...trap/engines/engine.py.229: process: 0.50613 | load: 0.00244 [I 2022-10-26 10:24:08] ...trap/engines/engine.py.230: loss: 0.31988 [I 2022-10-26 10:24:08] ...optimizers/trijoint.py.151: total_norm: 3.847762 [I 2022-10-26 10:24:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 890/2304 [I 2022-10-26 10:24:13] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:11:40 [I 2022-10-26 10:24:13] ...trap/engines/engine.py.229: process: 0.49250 | load: 0.00243 [I 2022-10-26 10:24:13] ...trap/engines/engine.py.230: loss: 0.29880 [I 2022-10-26 10:24:13] ...optimizers/trijoint.py.151: total_norm: 4.282523 [I 2022-10-26 10:24:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 900/2304 [I 2022-10-26 10:24:18] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:11:41 [I 2022-10-26 10:24:18] ...trap/engines/engine.py.229: process: 0.50275 | load: 0.00244 [I 2022-10-26 10:24:18] ...trap/engines/engine.py.230: loss: 0.37331 [I 2022-10-26 10:24:18] ...optimizers/trijoint.py.151: total_norm: 5.018770 [I 2022-10-26 10:24:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 910/2304 [I 2022-10-26 10:24:23] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:11:37 [I 2022-10-26 10:24:23] ...trap/engines/engine.py.229: process: 0.49632 | load: 0.00246 [I 2022-10-26 10:24:23] ...trap/engines/engine.py.230: loss: 0.34855 [I 2022-10-26 10:24:23] ...optimizers/trijoint.py.151: total_norm: 3.756971 [I 2022-10-26 10:24:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 920/2304 [I 2022-10-26 10:24:28] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:11:29 [I 2022-10-26 10:24:28] ...trap/engines/engine.py.229: process: 0.49726 | load: 0.00245 [I 2022-10-26 10:24:28] ...trap/engines/engine.py.230: loss: 0.39581 [I 2022-10-26 10:24:28] ...optimizers/trijoint.py.151: total_norm: 5.082807 [I 2022-10-26 10:24:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 930/2304 [I 2022-10-26 10:24:33] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:11:27 [I 2022-10-26 10:24:33] ...trap/engines/engine.py.229: process: 0.50305 | load: 0.00262 [I 2022-10-26 10:24:33] ...trap/engines/engine.py.230: loss: 0.34275 [I 2022-10-26 10:24:33] ...optimizers/trijoint.py.151: total_norm: 4.351330 [I 2022-10-26 10:24:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 940/2304 [I 2022-10-26 10:24:38] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:11:18 [I 2022-10-26 10:24:38] ...trap/engines/engine.py.229: process: 0.50764 | load: 0.00243 [I 2022-10-26 10:24:38] ...trap/engines/engine.py.230: loss: 0.36439 [I 2022-10-26 10:24:38] ...optimizers/trijoint.py.151: total_norm: 5.652126 [I 2022-10-26 10:24:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 950/2304 [I 2022-10-26 10:24:43] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:11:10 [I 2022-10-26 10:24:43] ...trap/engines/engine.py.229: process: 0.50047 | load: 0.00243 [I 2022-10-26 10:24:43] ...trap/engines/engine.py.230: loss: 0.36157 [I 2022-10-26 10:24:43] ...optimizers/trijoint.py.151: total_norm: 3.757248 [I 2022-10-26 10:24:47] ...trap/engines/engine.py.224: train: epoch 2 | batch 960/2304 [I 2022-10-26 10:24:47] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:11:02 [I 2022-10-26 10:24:47] ...trap/engines/engine.py.229: process: 0.48988 | load: 0.00251 [I 2022-10-26 10:24:47] ...trap/engines/engine.py.230: loss: 0.42355 [I 2022-10-26 10:24:47] ...optimizers/trijoint.py.151: total_norm: 4.447293 [I 2022-10-26 10:24:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 970/2304 [I 2022-10-26 10:24:53] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:11:11 [I 2022-10-26 10:24:53] ...trap/engines/engine.py.229: process: 0.50261 | load: 0.00249 [I 2022-10-26 10:24:53] ...trap/engines/engine.py.230: loss: 0.34137 [I 2022-10-26 10:24:53] ...optimizers/trijoint.py.151: total_norm: 3.591470 [I 2022-10-26 10:24:57] ...trap/engines/engine.py.224: train: epoch 2 | batch 980/2304 [I 2022-10-26 10:24:57] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:11:01 [I 2022-10-26 10:24:57] ...trap/engines/engine.py.229: process: 0.50518 | load: 0.00246 [I 2022-10-26 10:24:57] ...trap/engines/engine.py.230: loss: 0.29301 [I 2022-10-26 10:24:57] ...optimizers/trijoint.py.151: total_norm: 4.235028 [I 2022-10-26 10:25:02] ...trap/engines/engine.py.224: train: epoch 2 | batch 990/2304 [I 2022-10-26 10:25:02] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:10:54 [I 2022-10-26 10:25:02] ...trap/engines/engine.py.229: process: 0.49971 | load: 0.00245 [I 2022-10-26 10:25:02] ...trap/engines/engine.py.230: loss: 0.28736 [I 2022-10-26 10:25:02] ...optimizers/trijoint.py.151: total_norm: 3.387936 [I 2022-10-26 10:25:07] ...trap/engines/engine.py.224: train: epoch 2 | batch 1000/2304 [I 2022-10-26 10:25:07] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:10:50 [I 2022-10-26 10:25:07] ...trap/engines/engine.py.229: process: 0.49319 | load: 0.00241 [I 2022-10-26 10:25:07] ...trap/engines/engine.py.230: loss: 0.38236 [I 2022-10-26 10:25:07] ...optimizers/trijoint.py.151: total_norm: 4.098517 [I 2022-10-26 10:25:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 1010/2304 [I 2022-10-26 10:25:13] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:10:52 [I 2022-10-26 10:25:13] ...trap/engines/engine.py.229: process: 0.49292 | load: 0.00268 [I 2022-10-26 10:25:13] ...trap/engines/engine.py.230: loss: 0.41305 [I 2022-10-26 10:25:13] ...optimizers/trijoint.py.151: total_norm: 4.949179 [I 2022-10-26 10:25:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 1020/2304 [I 2022-10-26 10:25:18] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:10:37 [I 2022-10-26 10:25:18] ...trap/engines/engine.py.229: process: 0.49309 | load: 0.00245 [I 2022-10-26 10:25:18] ...trap/engines/engine.py.230: loss: 0.32998 [I 2022-10-26 10:25:18] ...optimizers/trijoint.py.151: total_norm: 4.122078 [I 2022-10-26 10:25:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 1030/2304 [I 2022-10-26 10:25:23] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:10:34 [I 2022-10-26 10:25:23] ...trap/engines/engine.py.229: process: 0.49464 | load: 0.00245 [I 2022-10-26 10:25:23] ...trap/engines/engine.py.230: loss: 0.38807 [I 2022-10-26 10:25:23] ...optimizers/trijoint.py.151: total_norm: 4.818361 [I 2022-10-26 10:25:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 1040/2304 [I 2022-10-26 10:25:28] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:10:25 [I 2022-10-26 10:25:28] ...trap/engines/engine.py.229: process: 0.48871 | load: 0.00244 [I 2022-10-26 10:25:28] ...trap/engines/engine.py.230: loss: 0.36269 [I 2022-10-26 10:25:28] ...optimizers/trijoint.py.151: total_norm: 3.920677 [I 2022-10-26 10:25:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 1050/2304 [I 2022-10-26 10:25:33] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:10:22 [I 2022-10-26 10:25:33] ...trap/engines/engine.py.229: process: 0.49291 | load: 0.00240 [I 2022-10-26 10:25:33] ...trap/engines/engine.py.230: loss: 0.30944 [I 2022-10-26 10:25:33] ...optimizers/trijoint.py.151: total_norm: 3.719637 [I 2022-10-26 10:25:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 1060/2304 [I 2022-10-26 10:25:38] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:10:19 [I 2022-10-26 10:25:38] ...trap/engines/engine.py.229: process: 0.49500 | load: 0.00240 [I 2022-10-26 10:25:38] ...trap/engines/engine.py.230: loss: 0.34733 [I 2022-10-26 10:25:38] ...optimizers/trijoint.py.151: total_norm: 5.478983 [I 2022-10-26 10:25:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 1070/2304 [I 2022-10-26 10:25:43] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:10:13 [I 2022-10-26 10:25:43] ...trap/engines/engine.py.229: process: 0.49300 | load: 0.00253 [I 2022-10-26 10:25:43] ...trap/engines/engine.py.230: loss: 0.37125 [I 2022-10-26 10:25:43] ...optimizers/trijoint.py.151: total_norm: 4.395055 [I 2022-10-26 10:25:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 1080/2304 [I 2022-10-26 10:25:48] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:10:06 [I 2022-10-26 10:25:48] ...trap/engines/engine.py.229: process: 0.49600 | load: 0.00248 [I 2022-10-26 10:25:48] ...trap/engines/engine.py.230: loss: 0.33472 [I 2022-10-26 10:25:48] ...optimizers/trijoint.py.151: total_norm: 4.931963 [I 2022-10-26 10:25:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 1090/2304 [I 2022-10-26 10:25:53] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:10:06 [I 2022-10-26 10:25:53] ...trap/engines/engine.py.229: process: 0.50620 | load: 0.00245 [I 2022-10-26 10:25:53] ...trap/engines/engine.py.230: loss: 0.27882 [I 2022-10-26 10:25:53] ...optimizers/trijoint.py.151: total_norm: 4.274022 [I 2022-10-26 10:25:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 1100/2304 [I 2022-10-26 10:25:58] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:10:13 [I 2022-10-26 10:25:58] ...trap/engines/engine.py.229: process: 0.50096 | load: 0.00246 [I 2022-10-26 10:25:58] ...trap/engines/engine.py.230: loss: 0.40227 [I 2022-10-26 10:25:58] ...optimizers/trijoint.py.151: total_norm: 4.617233 [I 2022-10-26 10:26:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 1110/2304 [I 2022-10-26 10:26:03] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:09:53 [I 2022-10-26 10:26:03] ...trap/engines/engine.py.229: process: 0.49529 | load: 0.00246 [I 2022-10-26 10:26:03] ...trap/engines/engine.py.230: loss: 0.36761 [I 2022-10-26 10:26:03] ...optimizers/trijoint.py.151: total_norm: 4.630520 [I 2022-10-26 10:26:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 1120/2304 [I 2022-10-26 10:26:08] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:09:50 [I 2022-10-26 10:26:08] ...trap/engines/engine.py.229: process: 0.50193 | load: 0.00244 [I 2022-10-26 10:26:08] ...trap/engines/engine.py.230: loss: 0.27039 [I 2022-10-26 10:26:08] ...optimizers/trijoint.py.151: total_norm: 3.707346 [I 2022-10-26 10:26:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 1130/2304 [I 2022-10-26 10:26:13] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:09:45 [I 2022-10-26 10:26:13] ...trap/engines/engine.py.229: process: 0.50137 | load: 0.00240 [I 2022-10-26 10:26:13] ...trap/engines/engine.py.230: loss: 0.28622 [I 2022-10-26 10:26:13] ...optimizers/trijoint.py.151: total_norm: 3.750507 [I 2022-10-26 10:26:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 1140/2304 [I 2022-10-26 10:26:18] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:09:44 [I 2022-10-26 10:26:18] ...trap/engines/engine.py.229: process: 0.49259 | load: 0.00243 [I 2022-10-26 10:26:18] ...trap/engines/engine.py.230: loss: 0.29526 [I 2022-10-26 10:26:18] ...optimizers/trijoint.py.151: total_norm: 3.489421 [I 2022-10-26 10:26:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 1150/2304 [I 2022-10-26 10:26:23] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:09:34 [I 2022-10-26 10:26:23] ...trap/engines/engine.py.229: process: 0.50168 | load: 0.00247 [I 2022-10-26 10:26:23] ...trap/engines/engine.py.230: loss: 0.23855 [I 2022-10-26 10:26:23] ...optimizers/trijoint.py.151: total_norm: 3.426159 [I 2022-10-26 10:26:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 1160/2304 [I 2022-10-26 10:26:28] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:09:25 [I 2022-10-26 10:26:28] ...trap/engines/engine.py.229: process: 0.49006 | load: 0.00247 [I 2022-10-26 10:26:28] ...trap/engines/engine.py.230: loss: 0.24423 [I 2022-10-26 10:26:28] ...optimizers/trijoint.py.151: total_norm: 3.815701 [I 2022-10-26 10:26:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 1170/2304 [I 2022-10-26 10:26:33] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:09:23 [I 2022-10-26 10:26:33] ...trap/engines/engine.py.229: process: 0.50648 | load: 0.00240 [I 2022-10-26 10:26:33] ...trap/engines/engine.py.230: loss: 0.40923 [I 2022-10-26 10:26:33] ...optimizers/trijoint.py.151: total_norm: 4.581362 [I 2022-10-26 10:26:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 1180/2304 [I 2022-10-26 10:26:38] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:09:21 [I 2022-10-26 10:26:38] ...trap/engines/engine.py.229: process: 0.50186 | load: 0.00242 [I 2022-10-26 10:26:38] ...trap/engines/engine.py.230: loss: 0.34423 [I 2022-10-26 10:26:38] ...optimizers/trijoint.py.151: total_norm: 4.140520 [I 2022-10-26 10:26:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 1190/2304 [I 2022-10-26 10:26:43] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:09:14 [I 2022-10-26 10:26:43] ...trap/engines/engine.py.229: process: 0.50122 | load: 0.00242 [I 2022-10-26 10:26:43] ...trap/engines/engine.py.230: loss: 0.33358 [I 2022-10-26 10:26:43] ...optimizers/trijoint.py.151: total_norm: 3.822365 [I 2022-10-26 10:26:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 1200/2304 [I 2022-10-26 10:26:48] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:09:10 [I 2022-10-26 10:26:48] ...trap/engines/engine.py.229: process: 0.50528 | load: 0.00242 [I 2022-10-26 10:26:48] ...trap/engines/engine.py.230: loss: 0.43608 [I 2022-10-26 10:26:48] ...optimizers/trijoint.py.151: total_norm: 6.130070 [I 2022-10-26 10:26:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 1210/2304 [I 2022-10-26 10:26:53] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:09:03 [I 2022-10-26 10:26:53] ...trap/engines/engine.py.229: process: 0.49528 | load: 0.00245 [I 2022-10-26 10:26:53] ...trap/engines/engine.py.230: loss: 0.25578 [I 2022-10-26 10:26:53] ...optimizers/trijoint.py.151: total_norm: 3.673159 [I 2022-10-26 10:26:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 1220/2304 [I 2022-10-26 10:26:58] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:08:57 [I 2022-10-26 10:26:58] ...trap/engines/engine.py.229: process: 0.49651 | load: 0.00245 [I 2022-10-26 10:26:58] ...trap/engines/engine.py.230: loss: 0.30715 [I 2022-10-26 10:26:58] ...optimizers/trijoint.py.151: total_norm: 4.777792 [I 2022-10-26 10:27:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 1230/2304 [I 2022-10-26 10:27:03] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:08:54 [I 2022-10-26 10:27:03] ...trap/engines/engine.py.229: process: 0.49904 | load: 0.00241 [I 2022-10-26 10:27:03] ...trap/engines/engine.py.230: loss: 0.34450 [I 2022-10-26 10:27:03] ...optimizers/trijoint.py.151: total_norm: 3.349348 [I 2022-10-26 10:27:07] ...trap/engines/engine.py.224: train: epoch 2 | batch 1240/2304 [I 2022-10-26 10:27:07] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:08:45 [I 2022-10-26 10:27:07] ...trap/engines/engine.py.229: process: 0.49176 | load: 0.00249 [I 2022-10-26 10:27:07] ...trap/engines/engine.py.230: loss: 0.36959 [I 2022-10-26 10:27:07] ...optimizers/trijoint.py.151: total_norm: 4.843153 [I 2022-10-26 10:27:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 1250/2304 [I 2022-10-26 10:27:13] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:08:51 [I 2022-10-26 10:27:13] ...trap/engines/engine.py.229: process: 0.50664 | load: 0.00248 [I 2022-10-26 10:27:13] ...trap/engines/engine.py.230: loss: 0.35750 [I 2022-10-26 10:27:13] ...optimizers/trijoint.py.151: total_norm: 3.754816 [I 2022-10-26 10:27:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 1260/2304 [I 2022-10-26 10:27:18] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:08:54 [I 2022-10-26 10:27:18] ...trap/engines/engine.py.229: process: 0.51526 | load: 0.00242 [I 2022-10-26 10:27:18] ...trap/engines/engine.py.230: loss: 0.27611 [I 2022-10-26 10:27:18] ...optimizers/trijoint.py.151: total_norm: 3.619685 [I 2022-10-26 10:27:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 1270/2304 [I 2022-10-26 10:27:23] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:08:46 [I 2022-10-26 10:27:23] ...trap/engines/engine.py.229: process: 0.49563 | load: 0.00241 [I 2022-10-26 10:27:23] ...trap/engines/engine.py.230: loss: 0.33958 [I 2022-10-26 10:27:23] ...optimizers/trijoint.py.151: total_norm: 3.699036 [I 2022-10-26 10:27:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 1280/2304 [I 2022-10-26 10:27:28] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:08:29 [I 2022-10-26 10:27:28] ...trap/engines/engine.py.229: process: 0.50924 | load: 0.00250 [I 2022-10-26 10:27:28] ...trap/engines/engine.py.230: loss: 0.34625 [I 2022-10-26 10:27:28] ...optimizers/trijoint.py.151: total_norm: 3.980020 [I 2022-10-26 10:27:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 1290/2304 [I 2022-10-26 10:27:33] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:08:30 [I 2022-10-26 10:27:33] ...trap/engines/engine.py.229: process: 0.50002 | load: 0.00244 [I 2022-10-26 10:27:33] ...trap/engines/engine.py.230: loss: 0.43707 [I 2022-10-26 10:27:33] ...optimizers/trijoint.py.151: total_norm: 4.363693 [I 2022-10-26 10:27:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 1300/2304 [I 2022-10-26 10:27:38] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:08:18 [I 2022-10-26 10:27:38] ...trap/engines/engine.py.229: process: 0.49039 | load: 0.00247 [I 2022-10-26 10:27:38] ...trap/engines/engine.py.230: loss: 0.29471 [I 2022-10-26 10:27:38] ...optimizers/trijoint.py.151: total_norm: 4.004306 [I 2022-10-26 10:27:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 1310/2304 [I 2022-10-26 10:27:43] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:08:10 [I 2022-10-26 10:27:43] ...trap/engines/engine.py.229: process: 0.48753 | load: 0.00248 [I 2022-10-26 10:27:43] ...trap/engines/engine.py.230: loss: 0.30300 [I 2022-10-26 10:27:43] ...optimizers/trijoint.py.151: total_norm: 3.787084 [I 2022-10-26 10:27:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 1320/2304 [I 2022-10-26 10:27:48] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:08:08 [I 2022-10-26 10:27:48] ...trap/engines/engine.py.229: process: 0.49271 | load: 0.00247 [I 2022-10-26 10:27:48] ...trap/engines/engine.py.230: loss: 0.36163 [I 2022-10-26 10:27:48] ...optimizers/trijoint.py.151: total_norm: 3.907296 [I 2022-10-26 10:27:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 1330/2304 [I 2022-10-26 10:27:53] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:08:01 [I 2022-10-26 10:27:53] ...trap/engines/engine.py.229: process: 0.49019 | load: 0.00240 [I 2022-10-26 10:27:53] ...trap/engines/engine.py.230: loss: 0.33028 [I 2022-10-26 10:27:53] ...optimizers/trijoint.py.151: total_norm: 3.641616 [I 2022-10-26 10:27:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 1340/2304 [I 2022-10-26 10:27:58] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:07:57 [I 2022-10-26 10:27:58] ...trap/engines/engine.py.229: process: 0.49476 | load: 0.00245 [I 2022-10-26 10:27:58] ...trap/engines/engine.py.230: loss: 0.25809 [I 2022-10-26 10:27:58] ...optimizers/trijoint.py.151: total_norm: 4.002836 [I 2022-10-26 10:28:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 1350/2304 [I 2022-10-26 10:28:03] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:07:52 [I 2022-10-26 10:28:03] ...trap/engines/engine.py.229: process: 0.48981 | load: 0.00252 [I 2022-10-26 10:28:03] ...trap/engines/engine.py.230: loss: 0.35555 [I 2022-10-26 10:28:03] ...optimizers/trijoint.py.151: total_norm: 4.475904 [I 2022-10-26 10:28:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 1360/2304 [I 2022-10-26 10:28:08] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:07:49 [I 2022-10-26 10:28:08] ...trap/engines/engine.py.229: process: 0.49497 | load: 0.00253 [I 2022-10-26 10:28:08] ...trap/engines/engine.py.230: loss: 0.27148 [I 2022-10-26 10:28:08] ...optimizers/trijoint.py.151: total_norm: 3.690212 [I 2022-10-26 10:28:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 1370/2304 [I 2022-10-26 10:28:13] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:07:47 [I 2022-10-26 10:28:13] ...trap/engines/engine.py.229: process: 0.49609 | load: 0.00247 [I 2022-10-26 10:28:13] ...trap/engines/engine.py.230: loss: 0.47988 [I 2022-10-26 10:28:13] ...optimizers/trijoint.py.151: total_norm: 4.925137 [I 2022-10-26 10:28:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 1380/2304 [I 2022-10-26 10:28:18] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:07:38 [I 2022-10-26 10:28:18] ...trap/engines/engine.py.229: process: 0.48973 | load: 0.00241 [I 2022-10-26 10:28:18] ...trap/engines/engine.py.230: loss: 0.43528 [I 2022-10-26 10:28:18] ...optimizers/trijoint.py.151: total_norm: 4.568405 [I 2022-10-26 10:28:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 1390/2304 [I 2022-10-26 10:28:23] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:07:36 [I 2022-10-26 10:28:23] ...trap/engines/engine.py.229: process: 0.49648 | load: 0.00246 [I 2022-10-26 10:28:23] ...trap/engines/engine.py.230: loss: 0.38005 [I 2022-10-26 10:28:23] ...optimizers/trijoint.py.151: total_norm: 5.862741 [I 2022-10-26 10:28:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 1400/2304 [I 2022-10-26 10:28:28] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:07:40 [I 2022-10-26 10:28:28] ...trap/engines/engine.py.229: process: 0.49805 | load: 0.00243 [I 2022-10-26 10:28:28] ...trap/engines/engine.py.230: loss: 0.40812 [I 2022-10-26 10:28:28] ...optimizers/trijoint.py.151: total_norm: 5.650739 [I 2022-10-26 10:28:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 1410/2304 [I 2022-10-26 10:28:33] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:07:25 [I 2022-10-26 10:28:33] ...trap/engines/engine.py.229: process: 0.49191 | load: 0.00256 [I 2022-10-26 10:28:33] ...trap/engines/engine.py.230: loss: 0.32434 [I 2022-10-26 10:28:33] ...optimizers/trijoint.py.151: total_norm: 3.600701 [I 2022-10-26 10:28:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 1420/2304 [I 2022-10-26 10:28:38] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:07:20 [I 2022-10-26 10:28:38] ...trap/engines/engine.py.229: process: 0.49965 | load: 0.00248 [I 2022-10-26 10:28:38] ...trap/engines/engine.py.230: loss: 0.38385 [I 2022-10-26 10:28:38] ...optimizers/trijoint.py.151: total_norm: 4.235540 [I 2022-10-26 10:28:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 1430/2304 [I 2022-10-26 10:28:43] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:07:09 [I 2022-10-26 10:28:43] ...trap/engines/engine.py.229: process: 0.48084 | load: 0.00247 [I 2022-10-26 10:28:43] ...trap/engines/engine.py.230: loss: 0.31025 [I 2022-10-26 10:28:43] ...optimizers/trijoint.py.151: total_norm: 3.750202 [I 2022-10-26 10:28:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 1440/2304 [I 2022-10-26 10:28:48] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:07:09 [I 2022-10-26 10:28:48] ...trap/engines/engine.py.229: process: 0.50415 | load: 0.00242 [I 2022-10-26 10:28:48] ...trap/engines/engine.py.230: loss: 0.37012 [I 2022-10-26 10:28:48] ...optimizers/trijoint.py.151: total_norm: 5.018806 [I 2022-10-26 10:28:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 1450/2304 [I 2022-10-26 10:28:53] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:07:01 [I 2022-10-26 10:28:53] ...trap/engines/engine.py.229: process: 0.49033 | load: 0.00242 [I 2022-10-26 10:28:53] ...trap/engines/engine.py.230: loss: 0.45339 [I 2022-10-26 10:28:53] ...optimizers/trijoint.py.151: total_norm: 4.870852 [I 2022-10-26 10:28:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 1460/2304 [I 2022-10-26 10:28:58] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:07:00 [I 2022-10-26 10:28:58] ...trap/engines/engine.py.229: process: 0.49815 | load: 0.00243 [I 2022-10-26 10:28:58] ...trap/engines/engine.py.230: loss: 0.48292 [I 2022-10-26 10:28:58] ...optimizers/trijoint.py.151: total_norm: 6.221889 [I 2022-10-26 10:29:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 1470/2304 [I 2022-10-26 10:29:03] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:06:55 [I 2022-10-26 10:29:03] ...trap/engines/engine.py.229: process: 0.50752 | load: 0.00248 [I 2022-10-26 10:29:03] ...trap/engines/engine.py.230: loss: 0.45605 [I 2022-10-26 10:29:03] ...optimizers/trijoint.py.151: total_norm: 4.721269 [I 2022-10-26 10:29:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 1480/2304 [I 2022-10-26 10:29:08] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:06:53 [I 2022-10-26 10:29:08] ...trap/engines/engine.py.229: process: 0.51368 | load: 0.00247 [I 2022-10-26 10:29:08] ...trap/engines/engine.py.230: loss: 0.34506 [I 2022-10-26 10:29:08] ...optimizers/trijoint.py.151: total_norm: 4.654481 [I 2022-10-26 10:29:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 1490/2304 [I 2022-10-26 10:29:13] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:06:45 [I 2022-10-26 10:29:13] ...trap/engines/engine.py.229: process: 0.49801 | load: 0.00246 [I 2022-10-26 10:29:13] ...trap/engines/engine.py.230: loss: 0.39851 [I 2022-10-26 10:29:13] ...optimizers/trijoint.py.151: total_norm: 4.322958 [I 2022-10-26 10:29:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 1500/2304 [I 2022-10-26 10:29:18] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:06:41 [I 2022-10-26 10:29:18] ...trap/engines/engine.py.229: process: 0.50292 | load: 0.00250 [I 2022-10-26 10:29:18] ...trap/engines/engine.py.230: loss: 0.24947 [I 2022-10-26 10:29:18] ...optimizers/trijoint.py.151: total_norm: 3.673500 [I 2022-10-26 10:29:23] ...trap/engines/engine.py.224: train: epoch 2 | batch 1510/2304 [I 2022-10-26 10:29:23] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:06:36 [I 2022-10-26 10:29:23] ...trap/engines/engine.py.229: process: 0.51006 | load: 0.00243 [I 2022-10-26 10:29:23] ...trap/engines/engine.py.230: loss: 0.31245 [I 2022-10-26 10:29:23] ...optimizers/trijoint.py.151: total_norm: 4.002651 [I 2022-10-26 10:29:28] ...trap/engines/engine.py.224: train: epoch 2 | batch 1520/2304 [I 2022-10-26 10:29:28] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:06:29 [I 2022-10-26 10:29:28] ...trap/engines/engine.py.229: process: 0.49286 | load: 0.00242 [I 2022-10-26 10:29:28] ...trap/engines/engine.py.230: loss: 0.31953 [I 2022-10-26 10:29:28] ...optimizers/trijoint.py.151: total_norm: 4.416693 [I 2022-10-26 10:29:33] ...trap/engines/engine.py.224: train: epoch 2 | batch 1530/2304 [I 2022-10-26 10:29:33] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:06:31 [I 2022-10-26 10:29:33] ...trap/engines/engine.py.229: process: 0.49373 | load: 0.00253 [I 2022-10-26 10:29:33] ...trap/engines/engine.py.230: loss: 0.32330 [I 2022-10-26 10:29:33] ...optimizers/trijoint.py.151: total_norm: 4.327767 [I 2022-10-26 10:29:38] ...trap/engines/engine.py.224: train: epoch 2 | batch 1540/2304 [I 2022-10-26 10:29:38] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:06:18 [I 2022-10-26 10:29:38] ...trap/engines/engine.py.229: process: 0.48373 | load: 0.00245 [I 2022-10-26 10:29:38] ...trap/engines/engine.py.230: loss: 0.35928 [I 2022-10-26 10:29:38] ...optimizers/trijoint.py.151: total_norm: 4.282040 [I 2022-10-26 10:29:43] ...trap/engines/engine.py.224: train: epoch 2 | batch 1550/2304 [I 2022-10-26 10:29:43] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:06:16 [I 2022-10-26 10:29:43] ...trap/engines/engine.py.229: process: 0.50349 | load: 0.00243 [I 2022-10-26 10:29:43] ...trap/engines/engine.py.230: loss: 0.36022 [I 2022-10-26 10:29:43] ...optimizers/trijoint.py.151: total_norm: 3.565710 [I 2022-10-26 10:29:48] ...trap/engines/engine.py.224: train: epoch 2 | batch 1560/2304 [I 2022-10-26 10:29:48] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:06:14 [I 2022-10-26 10:29:48] ...trap/engines/engine.py.229: process: 0.49399 | load: 0.00251 [I 2022-10-26 10:29:48] ...trap/engines/engine.py.230: loss: 0.31342 [I 2022-10-26 10:29:48] ...optimizers/trijoint.py.151: total_norm: 3.816478 [I 2022-10-26 10:29:53] ...trap/engines/engine.py.224: train: epoch 2 | batch 1570/2304 [I 2022-10-26 10:29:53] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:06:10 [I 2022-10-26 10:29:53] ...trap/engines/engine.py.229: process: 0.50695 | load: 0.00244 [I 2022-10-26 10:29:53] ...trap/engines/engine.py.230: loss: 0.28796 [I 2022-10-26 10:29:53] ...optimizers/trijoint.py.151: total_norm: 3.791007 [I 2022-10-26 10:29:58] ...trap/engines/engine.py.224: train: epoch 2 | batch 1580/2304 [I 2022-10-26 10:29:58] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:06:05 [I 2022-10-26 10:29:58] ...trap/engines/engine.py.229: process: 0.49554 | load: 0.00245 [I 2022-10-26 10:29:58] ...trap/engines/engine.py.230: loss: 0.22666 [I 2022-10-26 10:29:58] ...optimizers/trijoint.py.151: total_norm: 3.548635 [I 2022-10-26 10:30:03] ...trap/engines/engine.py.224: train: epoch 2 | batch 1590/2304 [I 2022-10-26 10:30:03] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:06:01 [I 2022-10-26 10:30:03] ...trap/engines/engine.py.229: process: 0.49623 | load: 0.00244 [I 2022-10-26 10:30:03] ...trap/engines/engine.py.230: loss: 0.38739 [I 2022-10-26 10:30:03] ...optimizers/trijoint.py.151: total_norm: 4.269332 [I 2022-10-26 10:30:08] ...trap/engines/engine.py.224: train: epoch 2 | batch 1600/2304 [I 2022-10-26 10:30:08] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:05:53 [I 2022-10-26 10:30:08] ...trap/engines/engine.py.229: process: 0.50378 | load: 0.00247 [I 2022-10-26 10:30:08] ...trap/engines/engine.py.230: loss: 0.40881 [I 2022-10-26 10:30:08] ...optimizers/trijoint.py.151: total_norm: 4.935067 [I 2022-10-26 10:30:13] ...trap/engines/engine.py.224: train: epoch 2 | batch 1610/2304 [I 2022-10-26 10:30:13] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:05:52 [I 2022-10-26 10:30:13] ...trap/engines/engine.py.229: process: 0.50506 | load: 0.00245 [I 2022-10-26 10:30:13] ...trap/engines/engine.py.230: loss: 0.31488 [I 2022-10-26 10:30:13] ...optimizers/trijoint.py.151: total_norm: 5.027773 [I 2022-10-26 10:30:18] ...trap/engines/engine.py.224: train: epoch 2 | batch 1620/2304 [I 2022-10-26 10:30:18] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:05:50 [I 2022-10-26 10:30:18] ...trap/engines/engine.py.229: process: 0.51993 | load: 0.00257 [I 2022-10-26 10:30:18] ...trap/engines/engine.py.230: loss: 0.49175 [I 2022-10-26 10:30:18] ...optimizers/trijoint.py.151: total_norm: 4.078733 [I 2022-10-26 10:30:24] ...trap/engines/engine.py.224: train: epoch 2 | batch 1630/2304 [I 2022-10-26 10:30:24] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:06:22 [I 2022-10-26 10:30:24] ...trap/engines/engine.py.229: process: 0.53158 | load: 0.00261 [I 2022-10-26 10:30:24] ...trap/engines/engine.py.230: loss: 0.29496 [I 2022-10-26 10:30:24] ...optimizers/trijoint.py.151: total_norm: 3.975109 [I 2022-10-26 10:30:30] ...trap/engines/engine.py.224: train: epoch 2 | batch 1640/2304 [I 2022-10-26 10:30:30] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:06:16 [I 2022-10-26 10:30:30] ...trap/engines/engine.py.229: process: 0.51063 | load: 0.00249 [I 2022-10-26 10:30:30] ...trap/engines/engine.py.230: loss: 0.33116 [I 2022-10-26 10:30:30] ...optimizers/trijoint.py.151: total_norm: 3.113629 [I 2022-10-26 10:30:35] ...trap/engines/engine.py.224: train: epoch 2 | batch 1650/2304 [I 2022-10-26 10:30:35] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:05:35 [I 2022-10-26 10:30:35] ...trap/engines/engine.py.229: process: 0.51100 | load: 0.00243 [I 2022-10-26 10:30:35] ...trap/engines/engine.py.230: loss: 0.33135 [I 2022-10-26 10:30:35] ...optimizers/trijoint.py.151: total_norm: 3.946239 [I 2022-10-26 10:30:40] ...trap/engines/engine.py.224: train: epoch 2 | batch 1660/2304 [I 2022-10-26 10:30:40] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:05:27 [I 2022-10-26 10:30:40] ...trap/engines/engine.py.229: process: 0.51493 | load: 0.00243 [I 2022-10-26 10:30:40] ...trap/engines/engine.py.230: loss: 0.36328 [I 2022-10-26 10:30:40] ...optimizers/trijoint.py.151: total_norm: 4.564573 [I 2022-10-26 10:30:45] ...trap/engines/engine.py.224: train: epoch 2 | batch 1670/2304 [I 2022-10-26 10:30:45] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:05:22 [I 2022-10-26 10:30:45] ...trap/engines/engine.py.229: process: 0.51969 | load: 0.00246 [I 2022-10-26 10:30:45] ...trap/engines/engine.py.230: loss: 0.33942 [I 2022-10-26 10:30:45] ...optimizers/trijoint.py.151: total_norm: 4.346745 [I 2022-10-26 10:30:50] ...trap/engines/engine.py.224: train: epoch 2 | batch 1680/2304 [I 2022-10-26 10:30:50] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:05:23 [I 2022-10-26 10:30:50] ...trap/engines/engine.py.229: process: 0.52013 | load: 0.00259 [I 2022-10-26 10:30:50] ...trap/engines/engine.py.230: loss: 0.35950 [I 2022-10-26 10:30:50] ...optimizers/trijoint.py.151: total_norm: 3.506586 [I 2022-10-26 10:30:55] ...trap/engines/engine.py.224: train: epoch 2 | batch 1690/2304 [I 2022-10-26 10:30:55] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:05:11 [I 2022-10-26 10:30:55] ...trap/engines/engine.py.229: process: 0.50667 | load: 0.00246 [I 2022-10-26 10:30:55] ...trap/engines/engine.py.230: loss: 0.33585 [I 2022-10-26 10:30:55] ...optimizers/trijoint.py.151: total_norm: 3.706125 [I 2022-10-26 10:31:00] ...trap/engines/engine.py.224: train: epoch 2 | batch 1700/2304 [I 2022-10-26 10:31:00] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:05:05 [I 2022-10-26 10:31:00] ...trap/engines/engine.py.229: process: 0.50942 | load: 0.00247 [I 2022-10-26 10:31:00] ...trap/engines/engine.py.230: loss: 0.42205 [I 2022-10-26 10:31:00] ...optimizers/trijoint.py.151: total_norm: 5.031879 [I 2022-10-26 10:31:06] ...trap/engines/engine.py.224: train: epoch 2 | batch 1710/2304 [I 2022-10-26 10:31:06] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:05:02 [I 2022-10-26 10:31:06] ...trap/engines/engine.py.229: process: 0.51315 | load: 0.00244 [I 2022-10-26 10:31:06] ...trap/engines/engine.py.230: loss: 0.37969 [I 2022-10-26 10:31:06] ...optimizers/trijoint.py.151: total_norm: 4.474727 [I 2022-10-26 10:31:11] ...trap/engines/engine.py.224: train: epoch 2 | batch 1720/2304 [I 2022-10-26 10:31:11] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:04:56 [I 2022-10-26 10:31:11] ...trap/engines/engine.py.229: process: 0.52090 | load: 0.00264 [I 2022-10-26 10:31:11] ...trap/engines/engine.py.230: loss: 0.33923 [I 2022-10-26 10:31:11] ...optimizers/trijoint.py.151: total_norm: 3.804897 [I 2022-10-26 10:31:16] ...trap/engines/engine.py.224: train: epoch 2 | batch 1730/2304 [I 2022-10-26 10:31:16] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:04:53 [I 2022-10-26 10:31:16] ...trap/engines/engine.py.229: process: 0.52075 | load: 0.00252 [I 2022-10-26 10:31:16] ...trap/engines/engine.py.230: loss: 0.36651 [I 2022-10-26 10:31:16] ...optimizers/trijoint.py.151: total_norm: 3.690793 [I 2022-10-26 10:31:21] ...trap/engines/engine.py.224: train: epoch 2 | batch 1740/2304 [I 2022-10-26 10:31:21] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:04:44 [I 2022-10-26 10:31:21] ...trap/engines/engine.py.229: process: 0.50794 | load: 0.00251 [I 2022-10-26 10:31:21] ...trap/engines/engine.py.230: loss: 0.35586 [I 2022-10-26 10:31:21] ...optimizers/trijoint.py.151: total_norm: 4.519928 [I 2022-10-26 10:31:26] ...trap/engines/engine.py.224: train: epoch 2 | batch 1750/2304 [I 2022-10-26 10:31:26] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:04:37 [I 2022-10-26 10:31:26] ...trap/engines/engine.py.229: process: 0.50425 | load: 0.00244 [I 2022-10-26 10:31:26] ...trap/engines/engine.py.230: loss: 0.24489 [I 2022-10-26 10:31:26] ...optimizers/trijoint.py.151: total_norm: 2.857010 [I 2022-10-26 10:31:31] ...trap/engines/engine.py.224: train: epoch 2 | batch 1760/2304 [I 2022-10-26 10:31:31] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:04:34 [I 2022-10-26 10:31:31] ...trap/engines/engine.py.229: process: 0.50527 | load: 0.00246 [I 2022-10-26 10:31:31] ...trap/engines/engine.py.230: loss: 0.38261 [I 2022-10-26 10:31:31] ...optimizers/trijoint.py.151: total_norm: 3.212566 [I 2022-10-26 10:31:36] ...trap/engines/engine.py.224: train: epoch 2 | batch 1770/2304 [I 2022-10-26 10:31:36] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:04:31 [I 2022-10-26 10:31:36] ...trap/engines/engine.py.229: process: 0.49065 | load: 0.00246 [I 2022-10-26 10:31:36] ...trap/engines/engine.py.230: loss: 0.20147 [I 2022-10-26 10:31:36] ...optimizers/trijoint.py.151: total_norm: 2.391807 [I 2022-10-26 10:31:41] ...trap/engines/engine.py.224: train: epoch 2 | batch 1780/2304 [I 2022-10-26 10:31:41] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:04:25 [I 2022-10-26 10:31:41] ...trap/engines/engine.py.229: process: 0.50436 | load: 0.00249 [I 2022-10-26 10:31:41] ...trap/engines/engine.py.230: loss: 0.32615 [I 2022-10-26 10:31:41] ...optimizers/trijoint.py.151: total_norm: 3.938965 [I 2022-10-26 10:31:46] ...trap/engines/engine.py.224: train: epoch 2 | batch 1790/2304 [I 2022-10-26 10:31:46] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:04:23 [I 2022-10-26 10:31:46] ...trap/engines/engine.py.229: process: 0.50791 | load: 0.00254 [I 2022-10-26 10:31:46] ...trap/engines/engine.py.230: loss: 0.32136 [I 2022-10-26 10:31:46] ...optimizers/trijoint.py.151: total_norm: 4.273423 [I 2022-10-26 10:31:51] ...trap/engines/engine.py.224: train: epoch 2 | batch 1800/2304 [I 2022-10-26 10:31:51] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:04:17 [I 2022-10-26 10:31:51] ...trap/engines/engine.py.229: process: 0.51334 | load: 0.00260 [I 2022-10-26 10:31:51] ...trap/engines/engine.py.230: loss: 0.35093 [I 2022-10-26 10:31:51] ...optimizers/trijoint.py.151: total_norm: 4.426060 [I 2022-10-26 10:31:56] ...trap/engines/engine.py.224: train: epoch 2 | batch 1810/2304 [I 2022-10-26 10:31:56] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:04:11 [I 2022-10-26 10:31:56] ...trap/engines/engine.py.229: process: 0.51407 | load: 0.00245 [I 2022-10-26 10:31:56] ...trap/engines/engine.py.230: loss: 0.35061 [I 2022-10-26 10:31:56] ...optimizers/trijoint.py.151: total_norm: 4.338614 [I 2022-10-26 10:32:01] ...trap/engines/engine.py.224: train: epoch 2 | batch 1820/2304 [I 2022-10-26 10:32:01] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:04:05 [I 2022-10-26 10:32:01] ...trap/engines/engine.py.229: process: 0.51086 | load: 0.00261 [I 2022-10-26 10:32:01] ...trap/engines/engine.py.230: loss: 0.31368 [I 2022-10-26 10:32:01] ...optimizers/trijoint.py.151: total_norm: 4.534160 [I 2022-10-26 10:32:07] ...trap/engines/engine.py.224: train: epoch 2 | batch 1830/2304 [I 2022-10-26 10:32:07] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:04:00 [I 2022-10-26 10:32:07] ...trap/engines/engine.py.229: process: 0.50221 | load: 0.00243 [I 2022-10-26 10:32:07] ...trap/engines/engine.py.230: loss: 0.47037 [I 2022-10-26 10:32:07] ...optimizers/trijoint.py.151: total_norm: 5.031551 [I 2022-10-26 10:32:12] ...trap/engines/engine.py.224: train: epoch 2 | batch 1840/2304 [I 2022-10-26 10:32:12] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:03:55 [I 2022-10-26 10:32:12] ...trap/engines/engine.py.229: process: 0.51060 | load: 0.00244 [I 2022-10-26 10:32:12] ...trap/engines/engine.py.230: loss: 0.36410 [I 2022-10-26 10:32:12] ...optimizers/trijoint.py.151: total_norm: 3.921691 [I 2022-10-26 10:32:17] ...trap/engines/engine.py.224: train: epoch 2 | batch 1850/2304 [I 2022-10-26 10:32:17] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:03:51 [I 2022-10-26 10:32:17] ...trap/engines/engine.py.229: process: 0.50182 | load: 0.00253 [I 2022-10-26 10:32:17] ...trap/engines/engine.py.230: loss: 0.34806 [I 2022-10-26 10:32:17] ...optimizers/trijoint.py.151: total_norm: 4.090661 [I 2022-10-26 10:32:22] ...trap/engines/engine.py.224: train: epoch 2 | batch 1860/2304 [I 2022-10-26 10:32:22] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:03:43 [I 2022-10-26 10:32:22] ...trap/engines/engine.py.229: process: 0.50284 | load: 0.00248 [I 2022-10-26 10:32:22] ...trap/engines/engine.py.230: loss: 0.39794 [I 2022-10-26 10:32:22] ...optimizers/trijoint.py.151: total_norm: 4.124835 [I 2022-10-26 10:32:27] ...trap/engines/engine.py.224: train: epoch 2 | batch 1870/2304 [I 2022-10-26 10:32:27] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:03:35 [I 2022-10-26 10:32:27] ...trap/engines/engine.py.229: process: 0.48768 | load: 0.00242 [I 2022-10-26 10:32:27] ...trap/engines/engine.py.230: loss: 0.27362 [I 2022-10-26 10:32:27] ...optimizers/trijoint.py.151: total_norm: 3.596622 [I 2022-10-26 10:32:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 1880/2304 [I 2022-10-26 10:32:32] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:03:29 [I 2022-10-26 10:32:32] ...trap/engines/engine.py.229: process: 0.49331 | load: 0.00246 [I 2022-10-26 10:32:32] ...trap/engines/engine.py.230: loss: 0.33703 [I 2022-10-26 10:32:32] ...optimizers/trijoint.py.151: total_norm: 3.889217 [I 2022-10-26 10:32:37] ...trap/engines/engine.py.224: train: epoch 2 | batch 1890/2304 [I 2022-10-26 10:32:37] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:03:27 [I 2022-10-26 10:32:37] ...trap/engines/engine.py.229: process: 0.49947 | load: 0.00247 [I 2022-10-26 10:32:37] ...trap/engines/engine.py.230: loss: 0.31587 [I 2022-10-26 10:32:37] ...optimizers/trijoint.py.151: total_norm: 4.245175 [I 2022-10-26 10:32:42] ...trap/engines/engine.py.224: train: epoch 2 | batch 1900/2304 [I 2022-10-26 10:32:42] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:03:21 [I 2022-10-26 10:32:42] ...trap/engines/engine.py.229: process: 0.49998 | load: 0.00247 [I 2022-10-26 10:32:42] ...trap/engines/engine.py.230: loss: 0.23310 [I 2022-10-26 10:32:42] ...optimizers/trijoint.py.151: total_norm: 3.207390 [I 2022-10-26 10:32:47] ...trap/engines/engine.py.224: train: epoch 2 | batch 1910/2304 [I 2022-10-26 10:32:47] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:03:15 [I 2022-10-26 10:32:47] ...trap/engines/engine.py.229: process: 0.49328 | load: 0.00247 [I 2022-10-26 10:32:47] ...trap/engines/engine.py.230: loss: 0.28964 [I 2022-10-26 10:32:47] ...optimizers/trijoint.py.151: total_norm: 3.526493 [I 2022-10-26 10:32:52] ...trap/engines/engine.py.224: train: epoch 2 | batch 1920/2304 [I 2022-10-26 10:32:52] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:03:11 [I 2022-10-26 10:32:52] ...trap/engines/engine.py.229: process: 0.49790 | load: 0.00244 [I 2022-10-26 10:32:52] ...trap/engines/engine.py.230: loss: 0.37289 [I 2022-10-26 10:32:52] ...optimizers/trijoint.py.151: total_norm: 4.531589 [I 2022-10-26 10:32:57] ...trap/engines/engine.py.224: train: epoch 2 | batch 1930/2304 [I 2022-10-26 10:32:57] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:03:06 [I 2022-10-26 10:32:57] ...trap/engines/engine.py.229: process: 0.49580 | load: 0.00244 [I 2022-10-26 10:32:57] ...trap/engines/engine.py.230: loss: 0.29690 [I 2022-10-26 10:32:57] ...optimizers/trijoint.py.151: total_norm: 4.392467 [I 2022-10-26 10:33:02] ...trap/engines/engine.py.224: train: epoch 2 | batch 1940/2304 [I 2022-10-26 10:33:02] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:03:01 [I 2022-10-26 10:33:02] ...trap/engines/engine.py.229: process: 0.49947 | load: 0.00243 [I 2022-10-26 10:33:02] ...trap/engines/engine.py.230: loss: 0.30096 [I 2022-10-26 10:33:02] ...optimizers/trijoint.py.151: total_norm: 5.079043 [I 2022-10-26 10:33:07] ...trap/engines/engine.py.224: train: epoch 2 | batch 1950/2304 [I 2022-10-26 10:33:07] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:02:56 [I 2022-10-26 10:33:07] ...trap/engines/engine.py.229: process: 0.50035 | load: 0.00252 [I 2022-10-26 10:33:07] ...trap/engines/engine.py.230: loss: 0.35070 [I 2022-10-26 10:33:07] ...optimizers/trijoint.py.151: total_norm: 4.384872 [I 2022-10-26 10:33:12] ...trap/engines/engine.py.224: train: epoch 2 | batch 1960/2304 [I 2022-10-26 10:33:12] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:02:51 [I 2022-10-26 10:33:12] ...trap/engines/engine.py.229: process: 0.49281 | load: 0.00254 [I 2022-10-26 10:33:12] ...trap/engines/engine.py.230: loss: 0.24517 [I 2022-10-26 10:33:12] ...optimizers/trijoint.py.151: total_norm: 3.423934 [I 2022-10-26 10:33:17] ...trap/engines/engine.py.224: train: epoch 2 | batch 1970/2304 [I 2022-10-26 10:33:17] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:02:47 [I 2022-10-26 10:33:17] ...trap/engines/engine.py.229: process: 0.50245 | load: 0.00244 [I 2022-10-26 10:33:17] ...trap/engines/engine.py.230: loss: 0.34132 [I 2022-10-26 10:33:17] ...optimizers/trijoint.py.151: total_norm: 4.122341 [I 2022-10-26 10:33:22] ...trap/engines/engine.py.224: train: epoch 2 | batch 1980/2304 [I 2022-10-26 10:33:22] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:02:42 [I 2022-10-26 10:33:22] ...trap/engines/engine.py.229: process: 0.49559 | load: 0.00244 [I 2022-10-26 10:33:22] ...trap/engines/engine.py.230: loss: 0.32213 [I 2022-10-26 10:33:22] ...optimizers/trijoint.py.151: total_norm: 3.901689 [I 2022-10-26 10:33:27] ...trap/engines/engine.py.224: train: epoch 2 | batch 1990/2304 [I 2022-10-26 10:33:27] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:02:37 [I 2022-10-26 10:33:27] ...trap/engines/engine.py.229: process: 0.50722 | load: 0.00244 [I 2022-10-26 10:33:27] ...trap/engines/engine.py.230: loss: 0.32279 [I 2022-10-26 10:33:27] ...optimizers/trijoint.py.151: total_norm: 3.703617 [I 2022-10-26 10:33:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 2000/2304 [I 2022-10-26 10:33:32] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:02:29 [I 2022-10-26 10:33:32] ...trap/engines/engine.py.229: process: 0.48671 | load: 0.00246 [I 2022-10-26 10:33:32] ...trap/engines/engine.py.230: loss: 0.35169 [I 2022-10-26 10:33:32] ...optimizers/trijoint.py.151: total_norm: 4.553251 [I 2022-10-26 10:33:37] ...trap/engines/engine.py.224: train: epoch 2 | batch 2010/2304 [I 2022-10-26 10:33:37] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:02:27 [I 2022-10-26 10:33:37] ...trap/engines/engine.py.229: process: 0.50147 | load: 0.00246 [I 2022-10-26 10:33:37] ...trap/engines/engine.py.230: loss: 0.35887 [I 2022-10-26 10:33:37] ...optimizers/trijoint.py.151: total_norm: 3.774838 [I 2022-10-26 10:33:42] ...trap/engines/engine.py.224: train: epoch 2 | batch 2020/2304 [I 2022-10-26 10:33:42] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:02:22 [I 2022-10-26 10:33:42] ...trap/engines/engine.py.229: process: 0.50622 | load: 0.00250 [I 2022-10-26 10:33:42] ...trap/engines/engine.py.230: loss: 0.27976 [I 2022-10-26 10:33:42] ...optimizers/trijoint.py.151: total_norm: 4.213994 [I 2022-10-26 10:33:47] ...trap/engines/engine.py.224: train: epoch 2 | batch 2030/2304 [I 2022-10-26 10:33:47] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:02:17 [I 2022-10-26 10:33:47] ...trap/engines/engine.py.229: process: 0.50349 | load: 0.00247 [I 2022-10-26 10:33:47] ...trap/engines/engine.py.230: loss: 0.26878 [I 2022-10-26 10:33:47] ...optimizers/trijoint.py.151: total_norm: 3.824134 [I 2022-10-26 10:33:52] ...trap/engines/engine.py.224: train: epoch 2 | batch 2040/2304 [I 2022-10-26 10:33:52] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:02:11 [I 2022-10-26 10:33:52] ...trap/engines/engine.py.229: process: 0.49950 | load: 0.00241 [I 2022-10-26 10:33:52] ...trap/engines/engine.py.230: loss: 0.28983 [I 2022-10-26 10:33:52] ...optimizers/trijoint.py.151: total_norm: 4.158731 [I 2022-10-26 10:33:57] ...trap/engines/engine.py.224: train: epoch 2 | batch 2050/2304 [I 2022-10-26 10:33:57] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:02:05 [I 2022-10-26 10:33:57] ...trap/engines/engine.py.229: process: 0.49079 | load: 0.00243 [I 2022-10-26 10:33:57] ...trap/engines/engine.py.230: loss: 0.36638 [I 2022-10-26 10:33:57] ...optimizers/trijoint.py.151: total_norm: 3.593614 [I 2022-10-26 10:34:02] ...trap/engines/engine.py.224: train: epoch 2 | batch 2060/2304 [I 2022-10-26 10:34:02] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:02:00 [I 2022-10-26 10:34:02] ...trap/engines/engine.py.229: process: 0.49345 | load: 0.00243 [I 2022-10-26 10:34:02] ...trap/engines/engine.py.230: loss: 0.30692 [I 2022-10-26 10:34:02] ...optimizers/trijoint.py.151: total_norm: 4.617069 [I 2022-10-26 10:34:07] ...trap/engines/engine.py.224: train: epoch 2 | batch 2070/2304 [I 2022-10-26 10:34:07] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:01:55 [I 2022-10-26 10:34:07] ...trap/engines/engine.py.229: process: 0.48787 | load: 0.00242 [I 2022-10-26 10:34:07] ...trap/engines/engine.py.230: loss: 0.36876 [I 2022-10-26 10:34:07] ...optimizers/trijoint.py.151: total_norm: 3.384065 [I 2022-10-26 10:34:12] ...trap/engines/engine.py.224: train: epoch 2 | batch 2080/2304 [I 2022-10-26 10:34:12] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:01:51 [I 2022-10-26 10:34:12] ...trap/engines/engine.py.229: process: 0.49804 | load: 0.00239 [I 2022-10-26 10:34:12] ...trap/engines/engine.py.230: loss: 0.30083 [I 2022-10-26 10:34:12] ...optimizers/trijoint.py.151: total_norm: 3.345804 [I 2022-10-26 10:34:17] ...trap/engines/engine.py.224: train: epoch 2 | batch 2090/2304 [I 2022-10-26 10:34:17] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:01:47 [I 2022-10-26 10:34:17] ...trap/engines/engine.py.229: process: 0.50157 | load: 0.00251 [I 2022-10-26 10:34:17] ...trap/engines/engine.py.230: loss: 0.39878 [I 2022-10-26 10:34:17] ...optimizers/trijoint.py.151: total_norm: 4.158276 [I 2022-10-26 10:34:22] ...trap/engines/engine.py.224: train: epoch 2 | batch 2100/2304 [I 2022-10-26 10:34:22] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:01:40 [I 2022-10-26 10:34:22] ...trap/engines/engine.py.229: process: 0.48973 | load: 0.00246 [I 2022-10-26 10:34:22] ...trap/engines/engine.py.230: loss: 0.35723 [I 2022-10-26 10:34:22] ...optimizers/trijoint.py.151: total_norm: 4.967892 [I 2022-10-26 10:34:27] ...trap/engines/engine.py.224: train: epoch 2 | batch 2110/2304 [I 2022-10-26 10:34:27] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:01:36 [I 2022-10-26 10:34:27] ...trap/engines/engine.py.229: process: 0.50293 | load: 0.00244 [I 2022-10-26 10:34:27] ...trap/engines/engine.py.230: loss: 0.35428 [I 2022-10-26 10:34:27] ...optimizers/trijoint.py.151: total_norm: 4.160800 [I 2022-10-26 10:34:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 2120/2304 [I 2022-10-26 10:34:32] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:01:31 [I 2022-10-26 10:34:32] ...trap/engines/engine.py.229: process: 0.49355 | load: 0.00243 [I 2022-10-26 10:34:32] ...trap/engines/engine.py.230: loss: 0.34217 [I 2022-10-26 10:34:32] ...optimizers/trijoint.py.151: total_norm: 4.369987 [I 2022-10-26 10:34:37] ...trap/engines/engine.py.224: train: epoch 2 | batch 2130/2304 [I 2022-10-26 10:34:37] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:01:26 [I 2022-10-26 10:34:37] ...trap/engines/engine.py.229: process: 0.49424 | load: 0.00244 [I 2022-10-26 10:34:37] ...trap/engines/engine.py.230: loss: 0.23978 [I 2022-10-26 10:34:37] ...optimizers/trijoint.py.151: total_norm: 3.488546 [I 2022-10-26 10:34:42] ...trap/engines/engine.py.224: train: epoch 2 | batch 2140/2304 [I 2022-10-26 10:34:42] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:01:22 [I 2022-10-26 10:34:42] ...trap/engines/engine.py.229: process: 0.50090 | load: 0.00258 [I 2022-10-26 10:34:42] ...trap/engines/engine.py.230: loss: 0.30371 [I 2022-10-26 10:34:42] ...optimizers/trijoint.py.151: total_norm: 4.355774 [I 2022-10-26 10:34:47] ...trap/engines/engine.py.224: train: epoch 2 | batch 2150/2304 [I 2022-10-26 10:34:47] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:01:17 [I 2022-10-26 10:34:47] ...trap/engines/engine.py.229: process: 0.48525 | load: 0.00250 [I 2022-10-26 10:34:47] ...trap/engines/engine.py.230: loss: 0.39283 [I 2022-10-26 10:34:47] ...optimizers/trijoint.py.151: total_norm: 4.734727 [I 2022-10-26 10:34:52] ...trap/engines/engine.py.224: train: epoch 2 | batch 2160/2304 [I 2022-10-26 10:34:52] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:01:12 [I 2022-10-26 10:34:52] ...trap/engines/engine.py.229: process: 0.49930 | load: 0.00308 [I 2022-10-26 10:34:52] ...trap/engines/engine.py.230: loss: 0.29828 [I 2022-10-26 10:34:52] ...optimizers/trijoint.py.151: total_norm: 3.755247 [I 2022-10-26 10:34:57] ...trap/engines/engine.py.224: train: epoch 2 | batch 2170/2304 [I 2022-10-26 10:34:57] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:01:07 [I 2022-10-26 10:34:57] ...trap/engines/engine.py.229: process: 0.49790 | load: 0.00260 [I 2022-10-26 10:34:57] ...trap/engines/engine.py.230: loss: 0.34703 [I 2022-10-26 10:34:57] ...optimizers/trijoint.py.151: total_norm: 4.759292 [I 2022-10-26 10:35:02] ...trap/engines/engine.py.224: train: epoch 2 | batch 2180/2304 [I 2022-10-26 10:35:02] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:01:01 [I 2022-10-26 10:35:02] ...trap/engines/engine.py.229: process: 0.49428 | load: 0.00253 [I 2022-10-26 10:35:02] ...trap/engines/engine.py.230: loss: 0.39833 [I 2022-10-26 10:35:02] ...optimizers/trijoint.py.151: total_norm: 3.928720 [I 2022-10-26 10:35:07] ...trap/engines/engine.py.224: train: epoch 2 | batch 2190/2304 [I 2022-10-26 10:35:07] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:00:57 [I 2022-10-26 10:35:07] ...trap/engines/engine.py.229: process: 0.50199 | load: 0.00254 [I 2022-10-26 10:35:07] ...trap/engines/engine.py.230: loss: 0.36801 [I 2022-10-26 10:35:07] ...optimizers/trijoint.py.151: total_norm: 3.874548 [I 2022-10-26 10:35:12] ...trap/engines/engine.py.224: train: epoch 2 | batch 2200/2304 [I 2022-10-26 10:35:12] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:00:53 [I 2022-10-26 10:35:12] ...trap/engines/engine.py.229: process: 0.49928 | load: 0.00278 [I 2022-10-26 10:35:12] ...trap/engines/engine.py.230: loss: 0.38543 [I 2022-10-26 10:35:12] ...optimizers/trijoint.py.151: total_norm: 3.902645 [I 2022-10-26 10:35:17] ...trap/engines/engine.py.224: train: epoch 2 | batch 2210/2304 [I 2022-10-26 10:35:17] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:00:47 [I 2022-10-26 10:35:17] ...trap/engines/engine.py.229: process: 0.49807 | load: 0.00272 [I 2022-10-26 10:35:17] ...trap/engines/engine.py.230: loss: 0.29740 [I 2022-10-26 10:35:17] ...optimizers/trijoint.py.151: total_norm: 3.372926 [I 2022-10-26 10:35:22] ...trap/engines/engine.py.224: train: epoch 2 | batch 2220/2304 [I 2022-10-26 10:35:22] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:00:41 [I 2022-10-26 10:35:22] ...trap/engines/engine.py.229: process: 0.50233 | load: 0.00252 [I 2022-10-26 10:35:22] ...trap/engines/engine.py.230: loss: 0.27201 [I 2022-10-26 10:35:22] ...optimizers/trijoint.py.151: total_norm: 3.581305 [I 2022-10-26 10:35:27] ...trap/engines/engine.py.224: train: epoch 2 | batch 2230/2304 [I 2022-10-26 10:35:27] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:00:36 [I 2022-10-26 10:35:27] ...trap/engines/engine.py.229: process: 0.50029 | load: 0.00252 [I 2022-10-26 10:35:27] ...trap/engines/engine.py.230: loss: 0.33239 [I 2022-10-26 10:35:27] ...optimizers/trijoint.py.151: total_norm: 4.438666 [I 2022-10-26 10:35:32] ...trap/engines/engine.py.224: train: epoch 2 | batch 2240/2304 [I 2022-10-26 10:35:32] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:00:32 [I 2022-10-26 10:35:32] ...trap/engines/engine.py.229: process: 0.50556 | load: 0.00269 [I 2022-10-26 10:35:32] ...trap/engines/engine.py.230: loss: 0.32381 [I 2022-10-26 10:35:32] ...optimizers/trijoint.py.151: total_norm: 4.000589 [I 2022-10-26 10:35:37] ...trap/engines/engine.py.224: train: epoch 2 | batch 2250/2304 [I 2022-10-26 10:35:37] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:00:27 [I 2022-10-26 10:35:37] ...trap/engines/engine.py.229: process: 0.51006 | load: 0.00258 [I 2022-10-26 10:35:37] ...trap/engines/engine.py.230: loss: 0.31869 [I 2022-10-26 10:35:37] ...optimizers/trijoint.py.151: total_norm: 3.657992 [I 2022-10-26 10:35:42] ...trap/engines/engine.py.224: train: epoch 2 | batch 2260/2304 [I 2022-10-26 10:35:42] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:00:22 [I 2022-10-26 10:35:42] ...trap/engines/engine.py.229: process: 0.50702 | load: 0.00261 [I 2022-10-26 10:35:42] ...trap/engines/engine.py.230: loss: 0.27786 [I 2022-10-26 10:35:42] ...optimizers/trijoint.py.151: total_norm: 3.509915 [I 2022-10-26 10:35:47] ...trap/engines/engine.py.224: train: epoch 2 | batch 2270/2304 [I 2022-10-26 10:35:47] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:00:17 [I 2022-10-26 10:35:47] ...trap/engines/engine.py.229: process: 0.50546 | load: 0.00258 [I 2022-10-26 10:35:47] ...trap/engines/engine.py.230: loss: 0.37010 [I 2022-10-26 10:35:47] ...optimizers/trijoint.py.151: total_norm: 3.863103 [I 2022-10-26 10:35:52] ...trap/engines/engine.py.224: train: epoch 2 | batch 2280/2304 [I 2022-10-26 10:35:52] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:00:11 [I 2022-10-26 10:35:52] ...trap/engines/engine.py.229: process: 0.49702 | load: 0.00256 [I 2022-10-26 10:35:52] ...trap/engines/engine.py.230: loss: 0.30886 [I 2022-10-26 10:35:52] ...optimizers/trijoint.py.151: total_norm: 4.770707 [I 2022-10-26 10:35:57] ...trap/engines/engine.py.224: train: epoch 2 | batch 2290/2304 [I 2022-10-26 10:35:57] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:00:07 [I 2022-10-26 10:35:57] ...trap/engines/engine.py.229: process: 0.49212 | load: 0.00254 [I 2022-10-26 10:35:57] ...trap/engines/engine.py.230: loss: 0.38015 [I 2022-10-26 10:35:57] ...optimizers/trijoint.py.151: total_norm: 3.752532 [I 2022-10-26 10:36:02] ...trap/engines/engine.py.224: train: epoch 2 | batch 2300/2304 [I 2022-10-26 10:36:02] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:00:02 [I 2022-10-26 10:36:02] ...trap/engines/engine.py.229: process: 0.49922 | load: 0.00015 [I 2022-10-26 10:36:02] ...trap/engines/engine.py.230: loss: 0.24847 [I 2022-10-26 10:36:02] ...optimizers/trijoint.py.151: total_norm: 2.813234 [I 2022-10-26 10:36:04] ...trap/engines/engine.py.224: train: epoch 2 | batch 2304/2304 [I 2022-10-26 10:36:04] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:00:00 [I 2022-10-26 10:36:04] ...trap/engines/engine.py.229: process: 0.49370 | load: 0.00014 [I 2022-10-26 10:36:04] ...trap/engines/engine.py.230: loss: 0.31667 [I 2022-10-26 10:36:04] ...optimizers/trijoint.py.151: total_norm: 4.194236 [S 2022-10-26 10:36:05] ...trap/engines/engine.py.129: train_epoch.epoch: 2 [S 2022-10-26 10:36:05] ...trap/engines/engine.py.129: train_epoch.loss: 0.3479859422318589 [I 2022-10-26 10:36:05] ...s/criterions/triplet.py.84: margin = 0.065 [I 2022-10-26 10:36:05] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 2 [I 2022-10-26 10:36:05] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 10:36:07] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 10:36:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 10:36:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 10:36:07] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 10:36:08] ...trap/engines/engine.py.321: eval: epoch 2 | batch 0/510 [I 2022-10-26 10:36:08] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:08 [I 2022-10-26 10:36:08] ...trap/engines/engine.py.326: process: 2.60438 | load: 2.25674 [I 2022-10-26 10:36:11] ...trap/engines/engine.py.321: eval: epoch 2 | batch 10/510 [I 2022-10-26 10:36:11] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:49 [I 2022-10-26 10:36:11] ...trap/engines/engine.py.326: process: 0.32217 | load: 0.00015 [I 2022-10-26 10:36:15] ...trap/engines/engine.py.321: eval: epoch 2 | batch 20/510 [I 2022-10-26 10:36:15] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:57 [I 2022-10-26 10:36:15] ...trap/engines/engine.py.326: process: 0.33590 | load: 0.00017 [I 2022-10-26 10:36:18] ...trap/engines/engine.py.321: eval: epoch 2 | batch 30/510 [I 2022-10-26 10:36:18] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:41 [I 2022-10-26 10:36:18] ...trap/engines/engine.py.326: process: 0.32677 | load: 0.00018 [I 2022-10-26 10:36:21] ...trap/engines/engine.py.321: eval: epoch 2 | batch 40/510 [I 2022-10-26 10:36:21] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:35 [I 2022-10-26 10:36:21] ...trap/engines/engine.py.326: process: 0.32000 | load: 0.00016 [I 2022-10-26 10:36:25] ...trap/engines/engine.py.321: eval: epoch 2 | batch 50/510 [I 2022-10-26 10:36:25] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:34 [I 2022-10-26 10:36:25] ...trap/engines/engine.py.326: process: 0.33764 | load: 0.00020 [I 2022-10-26 10:36:28] ...trap/engines/engine.py.321: eval: epoch 2 | batch 60/510 [I 2022-10-26 10:36:28] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:34 [I 2022-10-26 10:36:28] ...trap/engines/engine.py.326: process: 0.32376 | load: 0.00020 [I 2022-10-26 10:36:32] ...trap/engines/engine.py.321: eval: epoch 2 | batch 70/510 [I 2022-10-26 10:36:32] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:32 [I 2022-10-26 10:36:32] ...trap/engines/engine.py.326: process: 0.33254 | load: 0.00017 [I 2022-10-26 10:36:35] ...trap/engines/engine.py.321: eval: epoch 2 | batch 80/510 [I 2022-10-26 10:36:35] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:22 [I 2022-10-26 10:36:35] ...trap/engines/engine.py.326: process: 0.32256 | load: 0.00017 [I 2022-10-26 10:36:38] ...trap/engines/engine.py.321: eval: epoch 2 | batch 90/510 [I 2022-10-26 10:36:38] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:18 [I 2022-10-26 10:36:38] ...trap/engines/engine.py.326: process: 0.33011 | load: 0.00018 [I 2022-10-26 10:36:42] ...trap/engines/engine.py.321: eval: epoch 2 | batch 100/510 [I 2022-10-26 10:36:42] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:15 [I 2022-10-26 10:36:42] ...trap/engines/engine.py.326: process: 0.28602 | load: 0.00020 [I 2022-10-26 10:36:45] ...trap/engines/engine.py.321: eval: epoch 2 | batch 110/510 [I 2022-10-26 10:36:45] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:59 [I 2022-10-26 10:36:45] ...trap/engines/engine.py.326: process: 0.29404 | load: 0.00015 [I 2022-10-26 10:36:48] ...trap/engines/engine.py.321: eval: epoch 2 | batch 120/510 [I 2022-10-26 10:36:48] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:56 [I 2022-10-26 10:36:48] ...trap/engines/engine.py.326: process: 0.31231 | load: 0.00020 [I 2022-10-26 10:36:51] ...trap/engines/engine.py.321: eval: epoch 2 | batch 130/510 [I 2022-10-26 10:36:51] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-26 10:36:51] ...trap/engines/engine.py.326: process: 0.29123 | load: 0.00015 [I 2022-10-26 10:36:53] ...trap/engines/engine.py.321: eval: epoch 2 | batch 140/510 [I 2022-10-26 10:36:53] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:48 [I 2022-10-26 10:36:53] ...trap/engines/engine.py.326: process: 0.28802 | load: 0.00019 [I 2022-10-26 10:36:56] ...trap/engines/engine.py.321: eval: epoch 2 | batch 150/510 [I 2022-10-26 10:36:56] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 10:36:56] ...trap/engines/engine.py.326: process: 0.28772 | load: 0.00020 [I 2022-10-26 10:36:59] ...trap/engines/engine.py.321: eval: epoch 2 | batch 160/510 [I 2022-10-26 10:36:59] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:41 [I 2022-10-26 10:36:59] ...trap/engines/engine.py.326: process: 0.29017 | load: 0.00018 [I 2022-10-26 10:37:02] ...trap/engines/engine.py.321: eval: epoch 2 | batch 170/510 [I 2022-10-26 10:37:02] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 10:37:02] ...trap/engines/engine.py.326: process: 0.28937 | load: 0.00015 [I 2022-10-26 10:37:05] ...trap/engines/engine.py.321: eval: epoch 2 | batch 180/510 [I 2022-10-26 10:37:05] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:34 [I 2022-10-26 10:37:05] ...trap/engines/engine.py.326: process: 0.28925 | load: 0.00015 [I 2022-10-26 10:37:08] ...trap/engines/engine.py.321: eval: epoch 2 | batch 190/510 [I 2022-10-26 10:37:08] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:36 [I 2022-10-26 10:37:08] ...trap/engines/engine.py.326: process: 0.29008 | load: 0.00015 [I 2022-10-26 10:37:11] ...trap/engines/engine.py.321: eval: epoch 2 | batch 200/510 [I 2022-10-26 10:37:11] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-26 10:37:11] ...trap/engines/engine.py.326: process: 0.28989 | load: 0.00016 [I 2022-10-26 10:37:14] ...trap/engines/engine.py.321: eval: epoch 2 | batch 210/510 [I 2022-10-26 10:37:14] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 10:37:14] ...trap/engines/engine.py.326: process: 0.28596 | load: 0.00020 [I 2022-10-26 10:37:17] ...trap/engines/engine.py.321: eval: epoch 2 | batch 220/510 [I 2022-10-26 10:37:17] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 10:37:17] ...trap/engines/engine.py.326: process: 0.28213 | load: 0.00024 [I 2022-10-26 10:37:20] ...trap/engines/engine.py.321: eval: epoch 2 | batch 230/510 [I 2022-10-26 10:37:20] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 10:37:20] ...trap/engines/engine.py.326: process: 0.28518 | load: 0.00014 [I 2022-10-26 10:37:22] ...trap/engines/engine.py.321: eval: epoch 2 | batch 240/510 [I 2022-10-26 10:37:22] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 10:37:22] ...trap/engines/engine.py.326: process: 0.28743 | load: 0.00015 [I 2022-10-26 10:37:25] ...trap/engines/engine.py.321: eval: epoch 2 | batch 250/510 [I 2022-10-26 10:37:25] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-26 10:37:25] ...trap/engines/engine.py.326: process: 0.28410 | load: 0.00014 [I 2022-10-26 10:37:28] ...trap/engines/engine.py.321: eval: epoch 2 | batch 260/510 [I 2022-10-26 10:37:28] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:12 [I 2022-10-26 10:37:28] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00021 [I 2022-10-26 10:37:31] ...trap/engines/engine.py.321: eval: epoch 2 | batch 270/510 [I 2022-10-26 10:37:31] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 10:37:31] ...trap/engines/engine.py.326: process: 0.29585 | load: 0.00014 [I 2022-10-26 10:37:34] ...trap/engines/engine.py.321: eval: epoch 2 | batch 280/510 [I 2022-10-26 10:37:34] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:05 [I 2022-10-26 10:37:34] ...trap/engines/engine.py.326: process: 0.28472 | load: 0.00015 [I 2022-10-26 10:37:37] ...trap/engines/engine.py.321: eval: epoch 2 | batch 290/510 [I 2022-10-26 10:37:37] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 10:37:37] ...trap/engines/engine.py.326: process: 0.28606 | load: 0.00014 [I 2022-10-26 10:37:40] ...trap/engines/engine.py.321: eval: epoch 2 | batch 300/510 [I 2022-10-26 10:37:40] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 10:37:40] ...trap/engines/engine.py.326: process: 0.29478 | load: 0.00015 [I 2022-10-26 10:37:43] ...trap/engines/engine.py.321: eval: epoch 2 | batch 310/510 [I 2022-10-26 10:37:43] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:59 [I 2022-10-26 10:37:43] ...trap/engines/engine.py.326: process: 0.28799 | load: 0.00016 [I 2022-10-26 10:37:46] ...trap/engines/engine.py.321: eval: epoch 2 | batch 320/510 [I 2022-10-26 10:37:46] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 10:37:46] ...trap/engines/engine.py.326: process: 0.28525 | load: 0.00015 [I 2022-10-26 10:37:49] ...trap/engines/engine.py.321: eval: epoch 2 | batch 330/510 [I 2022-10-26 10:37:49] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 10:37:49] ...trap/engines/engine.py.326: process: 0.28747 | load: 0.00026 [I 2022-10-26 10:37:51] ...trap/engines/engine.py.321: eval: epoch 2 | batch 340/510 [I 2022-10-26 10:37:51] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 10:37:51] ...trap/engines/engine.py.326: process: 0.29281 | load: 0.00015 [I 2022-10-26 10:37:54] ...trap/engines/engine.py.321: eval: epoch 2 | batch 350/510 [I 2022-10-26 10:37:54] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 10:37:54] ...trap/engines/engine.py.326: process: 0.29266 | load: 0.00017 [I 2022-10-26 10:37:57] ...trap/engines/engine.py.321: eval: epoch 2 | batch 360/510 [I 2022-10-26 10:37:57] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 10:37:57] ...trap/engines/engine.py.326: process: 0.28591 | load: 0.00015 [I 2022-10-26 10:38:00] ...trap/engines/engine.py.321: eval: epoch 2 | batch 370/510 [I 2022-10-26 10:38:00] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 10:38:00] ...trap/engines/engine.py.326: process: 0.28966 | load: 0.00015 [I 2022-10-26 10:38:03] ...trap/engines/engine.py.321: eval: epoch 2 | batch 380/510 [I 2022-10-26 10:38:03] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 10:38:03] ...trap/engines/engine.py.326: process: 0.29213 | load: 0.00018 [I 2022-10-26 10:38:06] ...trap/engines/engine.py.321: eval: epoch 2 | batch 390/510 [I 2022-10-26 10:38:06] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 10:38:06] ...trap/engines/engine.py.326: process: 0.28593 | load: 0.00015 [I 2022-10-26 10:38:09] ...trap/engines/engine.py.321: eval: epoch 2 | batch 400/510 [I 2022-10-26 10:38:09] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 10:38:09] ...trap/engines/engine.py.326: process: 0.28681 | load: 0.00016 [I 2022-10-26 10:38:12] ...trap/engines/engine.py.321: eval: epoch 2 | batch 410/510 [I 2022-10-26 10:38:12] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 10:38:12] ...trap/engines/engine.py.326: process: 0.28902 | load: 0.00019 [I 2022-10-26 10:38:15] ...trap/engines/engine.py.321: eval: epoch 2 | batch 420/510 [I 2022-10-26 10:38:15] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 10:38:15] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00015 [I 2022-10-26 10:38:17] ...trap/engines/engine.py.321: eval: epoch 2 | batch 430/510 [I 2022-10-26 10:38:17] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:24 [I 2022-10-26 10:38:17] ...trap/engines/engine.py.326: process: 0.31917 | load: 0.00019 [I 2022-10-26 10:38:20] ...trap/engines/engine.py.321: eval: epoch 2 | batch 440/510 [I 2022-10-26 10:38:20] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 10:38:20] ...trap/engines/engine.py.326: process: 0.28102 | load: 0.00015 [I 2022-10-26 10:38:23] ...trap/engines/engine.py.321: eval: epoch 2 | batch 450/510 [I 2022-10-26 10:38:23] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 10:38:23] ...trap/engines/engine.py.326: process: 0.28522 | load: 0.00017 [I 2022-10-26 10:38:26] ...trap/engines/engine.py.321: eval: epoch 2 | batch 460/510 [I 2022-10-26 10:38:26] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 10:38:26] ...trap/engines/engine.py.326: process: 0.30575 | load: 0.00015 [I 2022-10-26 10:38:29] ...trap/engines/engine.py.321: eval: epoch 2 | batch 470/510 [I 2022-10-26 10:38:29] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 10:38:29] ...trap/engines/engine.py.326: process: 0.28684 | load: 0.00015 [I 2022-10-26 10:38:32] ...trap/engines/engine.py.321: eval: epoch 2 | batch 480/510 [I 2022-10-26 10:38:32] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 10:38:32] ...trap/engines/engine.py.326: process: 0.28597 | load: 0.00021 [I 2022-10-26 10:38:35] ...trap/engines/engine.py.321: eval: epoch 2 | batch 490/510 [I 2022-10-26 10:38:35] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 10:38:35] ...trap/engines/engine.py.326: process: 0.28736 | load: 0.00015 [I 2022-10-26 10:38:38] ...trap/engines/engine.py.321: eval: epoch 2 | batch 500/510 [I 2022-10-26 10:38:38] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 10:38:38] ...trap/engines/engine.py.326: process: 0.29157 | load: 0.00019 [I 2022-10-26 10:38:41] ...trap/engines/engine.py.321: eval: epoch 2 | batch 510/510 [I 2022-10-26 10:38:41] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 10:38:41] ...trap/engines/engine.py.326: process: 0.28732 | load: 0.00011 [S 2022-10-26 10:38:41] ...trap/engines/engine.py.132: eval_epoch.epoch: 2 [I 2022-10-26 10:38:41] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.60 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.10 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.355500 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.680000 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.788200 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.372300 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.696900 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.796100 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.435890 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.300000 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.005869 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013161 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.013151 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015007 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012078 [S 2022-10-26 10:38:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.011519 [I 2022-10-26 10:38:46] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-26 10:38:46] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 10:38:46] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 10:38:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 10:38:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 10:38:46] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 10:38:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 10:38:55] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 10:38:55] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 10:38:55] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 10:38:58] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 10:39:03] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 10:39:03] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 10:39:03] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 10:39:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 10:39:11] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 10:39:11] ...trap/engines/engine.py.163: Training model on trainset for epoch 3 [I 2022-10-26 10:39:11] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 10:39:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 0/2304 [I 2022-10-26 10:39:14] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:51 [I 2022-10-26 10:39:14] ...trap/engines/engine.py.229: process: 2.78288 | load: 2.21854 [I 2022-10-26 10:39:14] ...trap/engines/engine.py.230: loss: 0.26751 [I 2022-10-26 10:39:14] ...optimizers/trijoint.py.151: total_norm: 3.518784 [I 2022-10-26 10:39:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 10/2304 [I 2022-10-26 10:39:19] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:26 [I 2022-10-26 10:39:19] ...trap/engines/engine.py.229: process: 0.49352 | load: 0.00471 [I 2022-10-26 10:39:19] ...trap/engines/engine.py.230: loss: 0.34315 [I 2022-10-26 10:39:19] ...optimizers/trijoint.py.151: total_norm: 3.743011 [I 2022-10-26 10:39:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 20/2304 [I 2022-10-26 10:39:24] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:19:59 [I 2022-10-26 10:39:24] ...trap/engines/engine.py.229: process: 0.50993 | load: 0.00364 [I 2022-10-26 10:39:24] ...trap/engines/engine.py.230: loss: 0.37762 [I 2022-10-26 10:39:24] ...optimizers/trijoint.py.151: total_norm: 4.958694 [I 2022-10-26 10:39:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 30/2304 [I 2022-10-26 10:39:29] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:06 [I 2022-10-26 10:39:29] ...trap/engines/engine.py.229: process: 0.51284 | load: 0.00309 [I 2022-10-26 10:39:29] ...trap/engines/engine.py.230: loss: 0.32546 [I 2022-10-26 10:39:29] ...optimizers/trijoint.py.151: total_norm: 4.600086 [I 2022-10-26 10:39:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 40/2304 [I 2022-10-26 10:39:34] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:19:25 [I 2022-10-26 10:39:34] ...trap/engines/engine.py.229: process: 0.48153 | load: 0.00275 [I 2022-10-26 10:39:34] ...trap/engines/engine.py.230: loss: 0.31508 [I 2022-10-26 10:39:34] ...optimizers/trijoint.py.151: total_norm: 3.827806 [I 2022-10-26 10:39:39] ...trap/engines/engine.py.224: train: epoch 3 | batch 50/2304 [I 2022-10-26 10:39:39] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:51 [I 2022-10-26 10:39:39] ...trap/engines/engine.py.229: process: 0.49471 | load: 0.00269 [I 2022-10-26 10:39:39] ...trap/engines/engine.py.230: loss: 0.33359 [I 2022-10-26 10:39:39] ...optimizers/trijoint.py.151: total_norm: 4.128932 [I 2022-10-26 10:39:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 60/2304 [I 2022-10-26 10:39:44] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:52 [I 2022-10-26 10:39:44] ...trap/engines/engine.py.229: process: 0.49496 | load: 0.00272 [I 2022-10-26 10:39:44] ...trap/engines/engine.py.230: loss: 0.24501 [I 2022-10-26 10:39:44] ...optimizers/trijoint.py.151: total_norm: 3.271328 [I 2022-10-26 10:39:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 70/2304 [I 2022-10-26 10:39:49] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:18:51 [I 2022-10-26 10:39:49] ...trap/engines/engine.py.229: process: 0.50494 | load: 0.00280 [I 2022-10-26 10:39:49] ...trap/engines/engine.py.230: loss: 0.32031 [I 2022-10-26 10:39:49] ...optimizers/trijoint.py.151: total_norm: 4.013525 [I 2022-10-26 10:39:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 80/2304 [I 2022-10-26 10:39:54] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:18:39 [I 2022-10-26 10:39:54] ...trap/engines/engine.py.229: process: 0.52380 | load: 0.00330 [I 2022-10-26 10:39:54] ...trap/engines/engine.py.230: loss: 0.28805 [I 2022-10-26 10:39:54] ...optimizers/trijoint.py.151: total_norm: 3.524510 [I 2022-10-26 10:39:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 90/2304 [I 2022-10-26 10:39:59] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:18:27 [I 2022-10-26 10:39:59] ...trap/engines/engine.py.229: process: 0.48955 | load: 0.00258 [I 2022-10-26 10:39:59] ...trap/engines/engine.py.230: loss: 0.31083 [I 2022-10-26 10:39:59] ...optimizers/trijoint.py.151: total_norm: 4.488346 [I 2022-10-26 10:40:04] ...trap/engines/engine.py.224: train: epoch 3 | batch 100/2304 [I 2022-10-26 10:40:04] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:18:46 [I 2022-10-26 10:40:04] ...trap/engines/engine.py.229: process: 0.53551 | load: 0.00267 [I 2022-10-26 10:40:04] ...trap/engines/engine.py.230: loss: 0.44847 [I 2022-10-26 10:40:04] ...optimizers/trijoint.py.151: total_norm: 5.611503 [I 2022-10-26 10:40:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 110/2304 [I 2022-10-26 10:40:09] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:18:43 [I 2022-10-26 10:40:09] ...trap/engines/engine.py.229: process: 0.49631 | load: 0.00266 [I 2022-10-26 10:40:09] ...trap/engines/engine.py.230: loss: 0.36365 [I 2022-10-26 10:40:09] ...optimizers/trijoint.py.151: total_norm: 4.307098 [I 2022-10-26 10:40:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 120/2304 [I 2022-10-26 10:40:14] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:22 [I 2022-10-26 10:40:14] ...trap/engines/engine.py.229: process: 0.49198 | load: 0.00267 [I 2022-10-26 10:40:14] ...trap/engines/engine.py.230: loss: 0.32299 [I 2022-10-26 10:40:14] ...optimizers/trijoint.py.151: total_norm: 4.476916 [I 2022-10-26 10:40:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 130/2304 [I 2022-10-26 10:40:19] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:24 [I 2022-10-26 10:40:19] ...trap/engines/engine.py.229: process: 0.50437 | load: 0.00292 [I 2022-10-26 10:40:19] ...trap/engines/engine.py.230: loss: 0.27799 [I 2022-10-26 10:40:19] ...optimizers/trijoint.py.151: total_norm: 3.774602 [I 2022-10-26 10:40:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 140/2304 [I 2022-10-26 10:40:24] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:18:20 [I 2022-10-26 10:40:24] ...trap/engines/engine.py.229: process: 0.53936 | load: 0.00263 [I 2022-10-26 10:40:24] ...trap/engines/engine.py.230: loss: 0.26805 [I 2022-10-26 10:40:24] ...optimizers/trijoint.py.151: total_norm: 4.418965 [I 2022-10-26 10:40:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 150/2304 [I 2022-10-26 10:40:29] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:18:12 [I 2022-10-26 10:40:29] ...trap/engines/engine.py.229: process: 0.51087 | load: 0.00275 [I 2022-10-26 10:40:29] ...trap/engines/engine.py.230: loss: 0.33422 [I 2022-10-26 10:40:29] ...optimizers/trijoint.py.151: total_norm: 3.636553 [I 2022-10-26 10:40:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 160/2304 [I 2022-10-26 10:40:35] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:18:00 [I 2022-10-26 10:40:35] ...trap/engines/engine.py.229: process: 0.48863 | load: 0.00255 [I 2022-10-26 10:40:35] ...trap/engines/engine.py.230: loss: 0.29098 [I 2022-10-26 10:40:35] ...optimizers/trijoint.py.151: total_norm: 4.051325 [I 2022-10-26 10:40:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 170/2304 [I 2022-10-26 10:40:40] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:58 [I 2022-10-26 10:40:40] ...trap/engines/engine.py.229: process: 0.49987 | load: 0.00274 [I 2022-10-26 10:40:40] ...trap/engines/engine.py.230: loss: 0.34333 [I 2022-10-26 10:40:40] ...optimizers/trijoint.py.151: total_norm: 3.310761 [I 2022-10-26 10:40:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 180/2304 [I 2022-10-26 10:40:45] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:17:48 [I 2022-10-26 10:40:45] ...trap/engines/engine.py.229: process: 0.50217 | load: 0.00275 [I 2022-10-26 10:40:45] ...trap/engines/engine.py.230: loss: 0.25104 [I 2022-10-26 10:40:45] ...optimizers/trijoint.py.151: total_norm: 2.995949 [I 2022-10-26 10:40:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 190/2304 [I 2022-10-26 10:40:50] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:17:37 [I 2022-10-26 10:40:50] ...trap/engines/engine.py.229: process: 0.49622 | load: 0.00271 [I 2022-10-26 10:40:50] ...trap/engines/engine.py.230: loss: 0.40512 [I 2022-10-26 10:40:50] ...optimizers/trijoint.py.151: total_norm: 4.170757 [I 2022-10-26 10:40:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 200/2304 [I 2022-10-26 10:40:55] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:41 [I 2022-10-26 10:40:55] ...trap/engines/engine.py.229: process: 0.49760 | load: 0.00264 [I 2022-10-26 10:40:55] ...trap/engines/engine.py.230: loss: 0.40655 [I 2022-10-26 10:40:55] ...optimizers/trijoint.py.151: total_norm: 4.961406 [I 2022-10-26 10:41:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 210/2304 [I 2022-10-26 10:41:00] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:19 [I 2022-10-26 10:41:00] ...trap/engines/engine.py.229: process: 0.47888 | load: 0.00288 [I 2022-10-26 10:41:00] ...trap/engines/engine.py.230: loss: 0.25833 [I 2022-10-26 10:41:00] ...optimizers/trijoint.py.151: total_norm: 3.204866 [I 2022-10-26 10:41:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 220/2304 [I 2022-10-26 10:41:05] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:27 [I 2022-10-26 10:41:05] ...trap/engines/engine.py.229: process: 0.51199 | load: 0.00266 [I 2022-10-26 10:41:05] ...trap/engines/engine.py.230: loss: 0.28755 [I 2022-10-26 10:41:05] ...optimizers/trijoint.py.151: total_norm: 4.271145 [I 2022-10-26 10:41:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 230/2304 [I 2022-10-26 10:41:10] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:16 [I 2022-10-26 10:41:10] ...trap/engines/engine.py.229: process: 0.49025 | load: 0.00289 [I 2022-10-26 10:41:10] ...trap/engines/engine.py.230: loss: 0.32395 [I 2022-10-26 10:41:10] ...optimizers/trijoint.py.151: total_norm: 3.791663 [I 2022-10-26 10:41:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 240/2304 [I 2022-10-26 10:41:15] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:05 [I 2022-10-26 10:41:15] ...trap/engines/engine.py.229: process: 0.50018 | load: 0.00288 [I 2022-10-26 10:41:15] ...trap/engines/engine.py.230: loss: 0.34569 [I 2022-10-26 10:41:15] ...optimizers/trijoint.py.151: total_norm: 4.716081 [I 2022-10-26 10:41:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 250/2304 [I 2022-10-26 10:41:20] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:03 [I 2022-10-26 10:41:20] ...trap/engines/engine.py.229: process: 0.49400 | load: 0.00265 [I 2022-10-26 10:41:20] ...trap/engines/engine.py.230: loss: 0.24180 [I 2022-10-26 10:41:20] ...optimizers/trijoint.py.151: total_norm: 3.846487 [I 2022-10-26 10:41:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 260/2304 [I 2022-10-26 10:41:25] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:04 [I 2022-10-26 10:41:25] ...trap/engines/engine.py.229: process: 0.50926 | load: 0.00266 [I 2022-10-26 10:41:25] ...trap/engines/engine.py.230: loss: 0.41941 [I 2022-10-26 10:41:25] ...optimizers/trijoint.py.151: total_norm: 4.949699 [I 2022-10-26 10:41:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 270/2304 [I 2022-10-26 10:41:30] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:17:00 [I 2022-10-26 10:41:30] ...trap/engines/engine.py.229: process: 0.50541 | load: 0.00273 [I 2022-10-26 10:41:30] ...trap/engines/engine.py.230: loss: 0.30598 [I 2022-10-26 10:41:30] ...optimizers/trijoint.py.151: total_norm: 4.308398 [I 2022-10-26 10:41:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 280/2304 [I 2022-10-26 10:41:35] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:16:55 [I 2022-10-26 10:41:35] ...trap/engines/engine.py.229: process: 0.51601 | load: 0.00277 [I 2022-10-26 10:41:35] ...trap/engines/engine.py.230: loss: 0.38674 [I 2022-10-26 10:41:35] ...optimizers/trijoint.py.151: total_norm: 4.386724 [I 2022-10-26 10:41:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 290/2304 [I 2022-10-26 10:41:40] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:16:52 [I 2022-10-26 10:41:40] ...trap/engines/engine.py.229: process: 0.50121 | load: 0.00251 [I 2022-10-26 10:41:40] ...trap/engines/engine.py.230: loss: 0.33167 [I 2022-10-26 10:41:40] ...optimizers/trijoint.py.151: total_norm: 3.768284 [I 2022-10-26 10:41:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 300/2304 [I 2022-10-26 10:41:45] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:16:35 [I 2022-10-26 10:41:45] ...trap/engines/engine.py.229: process: 0.49575 | load: 0.00278 [I 2022-10-26 10:41:45] ...trap/engines/engine.py.230: loss: 0.32719 [I 2022-10-26 10:41:45] ...optimizers/trijoint.py.151: total_norm: 4.177757 [I 2022-10-26 10:41:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 310/2304 [I 2022-10-26 10:41:50] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:16:36 [I 2022-10-26 10:41:50] ...trap/engines/engine.py.229: process: 0.49354 | load: 0.00270 [I 2022-10-26 10:41:50] ...trap/engines/engine.py.230: loss: 0.24066 [I 2022-10-26 10:41:50] ...optimizers/trijoint.py.151: total_norm: 3.428684 [I 2022-10-26 10:41:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 320/2304 [I 2022-10-26 10:41:55] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:16:37 [I 2022-10-26 10:41:55] ...trap/engines/engine.py.229: process: 0.50930 | load: 0.00252 [I 2022-10-26 10:41:55] ...trap/engines/engine.py.230: loss: 0.26448 [I 2022-10-26 10:41:55] ...optimizers/trijoint.py.151: total_norm: 3.900478 [I 2022-10-26 10:42:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 330/2304 [I 2022-10-26 10:42:00] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:16:33 [I 2022-10-26 10:42:00] ...trap/engines/engine.py.229: process: 0.51334 | load: 0.00272 [I 2022-10-26 10:42:00] ...trap/engines/engine.py.230: loss: 0.32870 [I 2022-10-26 10:42:00] ...optimizers/trijoint.py.151: total_norm: 4.622840 [I 2022-10-26 10:42:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 340/2304 [I 2022-10-26 10:42:05] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:16:25 [I 2022-10-26 10:42:05] ...trap/engines/engine.py.229: process: 0.49967 | load: 0.00253 [I 2022-10-26 10:42:05] ...trap/engines/engine.py.230: loss: 0.27277 [I 2022-10-26 10:42:05] ...optimizers/trijoint.py.151: total_norm: 4.343641 [I 2022-10-26 10:42:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 350/2304 [I 2022-10-26 10:42:10] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:16:26 [I 2022-10-26 10:42:10] ...trap/engines/engine.py.229: process: 0.51202 | load: 0.00249 [I 2022-10-26 10:42:10] ...trap/engines/engine.py.230: loss: 0.33416 [I 2022-10-26 10:42:10] ...optimizers/trijoint.py.151: total_norm: 4.001723 [I 2022-10-26 10:42:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 360/2304 [I 2022-10-26 10:42:15] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:16:14 [I 2022-10-26 10:42:15] ...trap/engines/engine.py.229: process: 0.50473 | load: 0.00262 [I 2022-10-26 10:42:15] ...trap/engines/engine.py.230: loss: 0.36525 [I 2022-10-26 10:42:15] ...optimizers/trijoint.py.151: total_norm: 4.000021 [I 2022-10-26 10:42:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 370/2304 [I 2022-10-26 10:42:20] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:15:55 [I 2022-10-26 10:42:20] ...trap/engines/engine.py.229: process: 0.49421 | load: 0.00259 [I 2022-10-26 10:42:20] ...trap/engines/engine.py.230: loss: 0.30528 [I 2022-10-26 10:42:20] ...optimizers/trijoint.py.151: total_norm: 3.904172 [I 2022-10-26 10:42:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 380/2304 [I 2022-10-26 10:42:25] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:15:57 [I 2022-10-26 10:42:25] ...trap/engines/engine.py.229: process: 0.48890 | load: 0.00249 [I 2022-10-26 10:42:25] ...trap/engines/engine.py.230: loss: 0.30315 [I 2022-10-26 10:42:25] ...optimizers/trijoint.py.151: total_norm: 3.796824 [I 2022-10-26 10:42:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 390/2304 [I 2022-10-26 10:42:30] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:16:00 [I 2022-10-26 10:42:30] ...trap/engines/engine.py.229: process: 0.52077 | load: 0.00249 [I 2022-10-26 10:42:30] ...trap/engines/engine.py.230: loss: 0.45191 [I 2022-10-26 10:42:30] ...optimizers/trijoint.py.151: total_norm: 5.633782 [I 2022-10-26 10:42:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 400/2304 [I 2022-10-26 10:42:35] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:15:57 [I 2022-10-26 10:42:35] ...trap/engines/engine.py.229: process: 0.50163 | load: 0.00246 [I 2022-10-26 10:42:35] ...trap/engines/engine.py.230: loss: 0.34587 [I 2022-10-26 10:42:35] ...optimizers/trijoint.py.151: total_norm: 4.362685 [I 2022-10-26 10:42:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 410/2304 [I 2022-10-26 10:42:40] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:15:50 [I 2022-10-26 10:42:40] ...trap/engines/engine.py.229: process: 0.50802 | load: 0.00254 [I 2022-10-26 10:42:40] ...trap/engines/engine.py.230: loss: 0.31474 [I 2022-10-26 10:42:40] ...optimizers/trijoint.py.151: total_norm: 4.121575 [I 2022-10-26 10:42:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 420/2304 [I 2022-10-26 10:42:45] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:15:41 [I 2022-10-26 10:42:45] ...trap/engines/engine.py.229: process: 0.49358 | load: 0.00253 [I 2022-10-26 10:42:45] ...trap/engines/engine.py.230: loss: 0.35472 [I 2022-10-26 10:42:45] ...optimizers/trijoint.py.151: total_norm: 4.512817 [I 2022-10-26 10:42:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 430/2304 [I 2022-10-26 10:42:50] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:15:34 [I 2022-10-26 10:42:50] ...trap/engines/engine.py.229: process: 0.50089 | load: 0.00244 [I 2022-10-26 10:42:50] ...trap/engines/engine.py.230: loss: 0.42058 [I 2022-10-26 10:42:50] ...optimizers/trijoint.py.151: total_norm: 5.106207 [I 2022-10-26 10:42:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 440/2304 [I 2022-10-26 10:42:55] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:15:33 [I 2022-10-26 10:42:55] ...trap/engines/engine.py.229: process: 0.50137 | load: 0.00249 [I 2022-10-26 10:42:55] ...trap/engines/engine.py.230: loss: 0.37115 [I 2022-10-26 10:42:55] ...optimizers/trijoint.py.151: total_norm: 4.124381 [I 2022-10-26 10:43:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 450/2304 [I 2022-10-26 10:43:00] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:15:23 [I 2022-10-26 10:43:00] ...trap/engines/engine.py.229: process: 0.50418 | load: 0.00246 [I 2022-10-26 10:43:00] ...trap/engines/engine.py.230: loss: 0.35116 [I 2022-10-26 10:43:00] ...optimizers/trijoint.py.151: total_norm: 3.867568 [I 2022-10-26 10:43:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 460/2304 [I 2022-10-26 10:43:05] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:15:14 [I 2022-10-26 10:43:05] ...trap/engines/engine.py.229: process: 0.50517 | load: 0.00240 [I 2022-10-26 10:43:05] ...trap/engines/engine.py.230: loss: 0.28684 [I 2022-10-26 10:43:05] ...optimizers/trijoint.py.151: total_norm: 3.779252 [I 2022-10-26 10:43:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 470/2304 [I 2022-10-26 10:43:10] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:15:15 [I 2022-10-26 10:43:10] ...trap/engines/engine.py.229: process: 0.51230 | load: 0.00245 [I 2022-10-26 10:43:10] ...trap/engines/engine.py.230: loss: 0.34583 [I 2022-10-26 10:43:10] ...optimizers/trijoint.py.151: total_norm: 4.362070 [I 2022-10-26 10:43:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 480/2304 [I 2022-10-26 10:43:15] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:14:55 [I 2022-10-26 10:43:15] ...trap/engines/engine.py.229: process: 0.47837 | load: 0.00275 [I 2022-10-26 10:43:15] ...trap/engines/engine.py.230: loss: 0.36498 [I 2022-10-26 10:43:15] ...optimizers/trijoint.py.151: total_norm: 3.773156 [I 2022-10-26 10:43:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 490/2304 [I 2022-10-26 10:43:20] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:15:01 [I 2022-10-26 10:43:20] ...trap/engines/engine.py.229: process: 0.49708 | load: 0.00246 [I 2022-10-26 10:43:20] ...trap/engines/engine.py.230: loss: 0.32219 [I 2022-10-26 10:43:20] ...optimizers/trijoint.py.151: total_norm: 4.137393 [I 2022-10-26 10:43:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 500/2304 [I 2022-10-26 10:43:25] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:15:01 [I 2022-10-26 10:43:25] ...trap/engines/engine.py.229: process: 0.49720 | load: 0.00253 [I 2022-10-26 10:43:25] ...trap/engines/engine.py.230: loss: 0.27002 [I 2022-10-26 10:43:25] ...optimizers/trijoint.py.151: total_norm: 4.357494 [I 2022-10-26 10:43:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 510/2304 [I 2022-10-26 10:43:30] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:14:55 [I 2022-10-26 10:43:30] ...trap/engines/engine.py.229: process: 0.50331 | load: 0.00251 [I 2022-10-26 10:43:30] ...trap/engines/engine.py.230: loss: 0.25664 [I 2022-10-26 10:43:30] ...optimizers/trijoint.py.151: total_norm: 3.365204 [I 2022-10-26 10:43:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 520/2304 [I 2022-10-26 10:43:35] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:14:50 [I 2022-10-26 10:43:35] ...trap/engines/engine.py.229: process: 0.50382 | load: 0.00252 [I 2022-10-26 10:43:35] ...trap/engines/engine.py.230: loss: 0.41027 [I 2022-10-26 10:43:35] ...optimizers/trijoint.py.151: total_norm: 4.709436 [I 2022-10-26 10:43:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 530/2304 [I 2022-10-26 10:43:40] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:14:44 [I 2022-10-26 10:43:40] ...trap/engines/engine.py.229: process: 0.50135 | load: 0.00242 [I 2022-10-26 10:43:40] ...trap/engines/engine.py.230: loss: 0.29431 [I 2022-10-26 10:43:40] ...optimizers/trijoint.py.151: total_norm: 3.471824 [I 2022-10-26 10:43:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 540/2304 [I 2022-10-26 10:43:45] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:14:38 [I 2022-10-26 10:43:45] ...trap/engines/engine.py.229: process: 0.50636 | load: 0.00253 [I 2022-10-26 10:43:45] ...trap/engines/engine.py.230: loss: 0.35300 [I 2022-10-26 10:43:45] ...optimizers/trijoint.py.151: total_norm: 4.415892 [I 2022-10-26 10:43:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 550/2304 [I 2022-10-26 10:43:50] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:14:29 [I 2022-10-26 10:43:50] ...trap/engines/engine.py.229: process: 0.48647 | load: 0.00257 [I 2022-10-26 10:43:50] ...trap/engines/engine.py.230: loss: 0.32444 [I 2022-10-26 10:43:50] ...optimizers/trijoint.py.151: total_norm: 3.856771 [I 2022-10-26 10:43:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 560/2304 [I 2022-10-26 10:43:55] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:14:31 [I 2022-10-26 10:43:55] ...trap/engines/engine.py.229: process: 0.50948 | load: 0.00254 [I 2022-10-26 10:43:55] ...trap/engines/engine.py.230: loss: 0.26106 [I 2022-10-26 10:43:55] ...optimizers/trijoint.py.151: total_norm: 4.629165 [I 2022-10-26 10:44:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 570/2304 [I 2022-10-26 10:44:00] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:14:14 [I 2022-10-26 10:44:00] ...trap/engines/engine.py.229: process: 0.48514 | load: 0.00250 [I 2022-10-26 10:44:00] ...trap/engines/engine.py.230: loss: 0.28800 [I 2022-10-26 10:44:00] ...optimizers/trijoint.py.151: total_norm: 3.605556 [I 2022-10-26 10:44:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 580/2304 [I 2022-10-26 10:44:05] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:14:13 [I 2022-10-26 10:44:05] ...trap/engines/engine.py.229: process: 0.50348 | load: 0.00248 [I 2022-10-26 10:44:05] ...trap/engines/engine.py.230: loss: 0.24851 [I 2022-10-26 10:44:05] ...optimizers/trijoint.py.151: total_norm: 4.055263 [I 2022-10-26 10:44:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 590/2304 [I 2022-10-26 10:44:10] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:14:15 [I 2022-10-26 10:44:10] ...trap/engines/engine.py.229: process: 0.49180 | load: 0.00249 [I 2022-10-26 10:44:10] ...trap/engines/engine.py.230: loss: 0.24755 [I 2022-10-26 10:44:10] ...optimizers/trijoint.py.151: total_norm: 5.138619 [I 2022-10-26 10:44:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 600/2304 [I 2022-10-26 10:44:15] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:14:15 [I 2022-10-26 10:44:15] ...trap/engines/engine.py.229: process: 0.49512 | load: 0.00267 [I 2022-10-26 10:44:15] ...trap/engines/engine.py.230: loss: 0.43673 [I 2022-10-26 10:44:15] ...optimizers/trijoint.py.151: total_norm: 4.866148 [I 2022-10-26 10:44:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 610/2304 [I 2022-10-26 10:44:20] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:14:02 [I 2022-10-26 10:44:20] ...trap/engines/engine.py.229: process: 0.49686 | load: 0.00250 [I 2022-10-26 10:44:20] ...trap/engines/engine.py.230: loss: 0.25129 [I 2022-10-26 10:44:20] ...optimizers/trijoint.py.151: total_norm: 3.770300 [I 2022-10-26 10:44:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 620/2304 [I 2022-10-26 10:44:25] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:13:53 [I 2022-10-26 10:44:25] ...trap/engines/engine.py.229: process: 0.48700 | load: 0.00246 [I 2022-10-26 10:44:25] ...trap/engines/engine.py.230: loss: 0.30536 [I 2022-10-26 10:44:25] ...optimizers/trijoint.py.151: total_norm: 3.793727 [I 2022-10-26 10:44:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 630/2304 [I 2022-10-26 10:44:30] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:13:51 [I 2022-10-26 10:44:30] ...trap/engines/engine.py.229: process: 0.50117 | load: 0.00251 [I 2022-10-26 10:44:30] ...trap/engines/engine.py.230: loss: 0.34169 [I 2022-10-26 10:44:30] ...optimizers/trijoint.py.151: total_norm: 3.781406 [I 2022-10-26 10:44:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 640/2304 [I 2022-10-26 10:44:35] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:13:52 [I 2022-10-26 10:44:35] ...trap/engines/engine.py.229: process: 0.49859 | load: 0.00245 [I 2022-10-26 10:44:35] ...trap/engines/engine.py.230: loss: 0.29257 [I 2022-10-26 10:44:35] ...optimizers/trijoint.py.151: total_norm: 3.587916 [I 2022-10-26 10:44:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 650/2304 [I 2022-10-26 10:44:40] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:13:43 [I 2022-10-26 10:44:40] ...trap/engines/engine.py.229: process: 0.48999 | load: 0.00246 [I 2022-10-26 10:44:40] ...trap/engines/engine.py.230: loss: 0.29912 [I 2022-10-26 10:44:40] ...optimizers/trijoint.py.151: total_norm: 3.651933 [I 2022-10-26 10:44:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 660/2304 [I 2022-10-26 10:44:45] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:13:31 [I 2022-10-26 10:44:45] ...trap/engines/engine.py.229: process: 0.48251 | load: 0.00249 [I 2022-10-26 10:44:45] ...trap/engines/engine.py.230: loss: 0.31492 [I 2022-10-26 10:44:45] ...optimizers/trijoint.py.151: total_norm: 3.409606 [I 2022-10-26 10:44:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 670/2304 [I 2022-10-26 10:44:50] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:13:25 [I 2022-10-26 10:44:50] ...trap/engines/engine.py.229: process: 0.49107 | load: 0.00249 [I 2022-10-26 10:44:50] ...trap/engines/engine.py.230: loss: 0.31528 [I 2022-10-26 10:44:50] ...optimizers/trijoint.py.151: total_norm: 4.378427 [I 2022-10-26 10:44:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 680/2304 [I 2022-10-26 10:44:55] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:13:33 [I 2022-10-26 10:44:55] ...trap/engines/engine.py.229: process: 0.50348 | load: 0.00244 [I 2022-10-26 10:44:55] ...trap/engines/engine.py.230: loss: 0.35492 [I 2022-10-26 10:44:55] ...optimizers/trijoint.py.151: total_norm: 4.586299 [I 2022-10-26 10:45:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 690/2304 [I 2022-10-26 10:45:00] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:13:27 [I 2022-10-26 10:45:00] ...trap/engines/engine.py.229: process: 0.50715 | load: 0.00242 [I 2022-10-26 10:45:00] ...trap/engines/engine.py.230: loss: 0.38504 [I 2022-10-26 10:45:00] ...optimizers/trijoint.py.151: total_norm: 5.718184 [I 2022-10-26 10:45:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 700/2304 [I 2022-10-26 10:45:05] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:13:31 [I 2022-10-26 10:45:05] ...trap/engines/engine.py.229: process: 0.49179 | load: 0.00241 [I 2022-10-26 10:45:05] ...trap/engines/engine.py.230: loss: 0.33540 [I 2022-10-26 10:45:05] ...optimizers/trijoint.py.151: total_norm: 4.521776 [I 2022-10-26 10:45:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 710/2304 [I 2022-10-26 10:45:10] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:13:11 [I 2022-10-26 10:45:10] ...trap/engines/engine.py.229: process: 0.48450 | load: 0.00246 [I 2022-10-26 10:45:10] ...trap/engines/engine.py.230: loss: 0.29046 [I 2022-10-26 10:45:10] ...optimizers/trijoint.py.151: total_norm: 4.195248 [I 2022-10-26 10:45:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 720/2304 [I 2022-10-26 10:45:15] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:13:21 [I 2022-10-26 10:45:15] ...trap/engines/engine.py.229: process: 0.49522 | load: 0.00240 [I 2022-10-26 10:45:15] ...trap/engines/engine.py.230: loss: 0.31622 [I 2022-10-26 10:45:15] ...optimizers/trijoint.py.151: total_norm: 3.857481 [I 2022-10-26 10:45:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 730/2304 [I 2022-10-26 10:45:20] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:13:06 [I 2022-10-26 10:45:20] ...trap/engines/engine.py.229: process: 0.48793 | load: 0.00239 [I 2022-10-26 10:45:20] ...trap/engines/engine.py.230: loss: 0.36805 [I 2022-10-26 10:45:20] ...optimizers/trijoint.py.151: total_norm: 3.848441 [I 2022-10-26 10:45:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 740/2304 [I 2022-10-26 10:45:25] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:13:05 [I 2022-10-26 10:45:25] ...trap/engines/engine.py.229: process: 0.50490 | load: 0.00240 [I 2022-10-26 10:45:25] ...trap/engines/engine.py.230: loss: 0.33434 [I 2022-10-26 10:45:25] ...optimizers/trijoint.py.151: total_norm: 4.795093 [I 2022-10-26 10:45:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 750/2304 [I 2022-10-26 10:45:30] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:12:52 [I 2022-10-26 10:45:30] ...trap/engines/engine.py.229: process: 0.48909 | load: 0.00251 [I 2022-10-26 10:45:30] ...trap/engines/engine.py.230: loss: 0.26658 [I 2022-10-26 10:45:30] ...optimizers/trijoint.py.151: total_norm: 3.347032 [I 2022-10-26 10:45:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 760/2304 [I 2022-10-26 10:45:35] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:12:50 [I 2022-10-26 10:45:35] ...trap/engines/engine.py.229: process: 0.50190 | load: 0.00245 [I 2022-10-26 10:45:35] ...trap/engines/engine.py.230: loss: 0.30260 [I 2022-10-26 10:45:35] ...optimizers/trijoint.py.151: total_norm: 4.131788 [I 2022-10-26 10:45:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 770/2304 [I 2022-10-26 10:45:40] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:12:53 [I 2022-10-26 10:45:40] ...trap/engines/engine.py.229: process: 0.50253 | load: 0.00291 [I 2022-10-26 10:45:40] ...trap/engines/engine.py.230: loss: 0.32918 [I 2022-10-26 10:45:40] ...optimizers/trijoint.py.151: total_norm: 4.571667 [I 2022-10-26 10:45:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 780/2304 [I 2022-10-26 10:45:45] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:12:43 [I 2022-10-26 10:45:45] ...trap/engines/engine.py.229: process: 0.50517 | load: 0.00241 [I 2022-10-26 10:45:45] ...trap/engines/engine.py.230: loss: 0.25593 [I 2022-10-26 10:45:45] ...optimizers/trijoint.py.151: total_norm: 3.128931 [I 2022-10-26 10:45:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 790/2304 [I 2022-10-26 10:45:50] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:12:26 [I 2022-10-26 10:45:50] ...trap/engines/engine.py.229: process: 0.48848 | load: 0.00242 [I 2022-10-26 10:45:50] ...trap/engines/engine.py.230: loss: 0.35610 [I 2022-10-26 10:45:50] ...optimizers/trijoint.py.151: total_norm: 3.762307 [I 2022-10-26 10:45:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 800/2304 [I 2022-10-26 10:45:55] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:12:32 [I 2022-10-26 10:45:55] ...trap/engines/engine.py.229: process: 0.49656 | load: 0.00244 [I 2022-10-26 10:45:55] ...trap/engines/engine.py.230: loss: 0.37227 [I 2022-10-26 10:45:55] ...optimizers/trijoint.py.151: total_norm: 4.344708 [I 2022-10-26 10:46:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 810/2304 [I 2022-10-26 10:46:00] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:12:26 [I 2022-10-26 10:46:00] ...trap/engines/engine.py.229: process: 0.50274 | load: 0.00246 [I 2022-10-26 10:46:00] ...trap/engines/engine.py.230: loss: 0.26005 [I 2022-10-26 10:46:00] ...optimizers/trijoint.py.151: total_norm: 3.600539 [I 2022-10-26 10:46:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 820/2304 [I 2022-10-26 10:46:05] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:12:22 [I 2022-10-26 10:46:05] ...trap/engines/engine.py.229: process: 0.51051 | load: 0.00254 [I 2022-10-26 10:46:05] ...trap/engines/engine.py.230: loss: 0.26925 [I 2022-10-26 10:46:05] ...optimizers/trijoint.py.151: total_norm: 4.858683 [I 2022-10-26 10:46:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 830/2304 [I 2022-10-26 10:46:10] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:12:09 [I 2022-10-26 10:46:10] ...trap/engines/engine.py.229: process: 0.48677 | load: 0.00245 [I 2022-10-26 10:46:10] ...trap/engines/engine.py.230: loss: 0.34633 [I 2022-10-26 10:46:10] ...optimizers/trijoint.py.151: total_norm: 5.124446 [I 2022-10-26 10:46:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 840/2304 [I 2022-10-26 10:46:15] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:12:07 [I 2022-10-26 10:46:15] ...trap/engines/engine.py.229: process: 0.49033 | load: 0.00245 [I 2022-10-26 10:46:15] ...trap/engines/engine.py.230: loss: 0.28268 [I 2022-10-26 10:46:15] ...optimizers/trijoint.py.151: total_norm: 3.579961 [I 2022-10-26 10:46:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 850/2304 [I 2022-10-26 10:46:20] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:12:01 [I 2022-10-26 10:46:20] ...trap/engines/engine.py.229: process: 0.50805 | load: 0.00246 [I 2022-10-26 10:46:20] ...trap/engines/engine.py.230: loss: 0.34070 [I 2022-10-26 10:46:20] ...optimizers/trijoint.py.151: total_norm: 3.943244 [I 2022-10-26 10:46:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 860/2304 [I 2022-10-26 10:46:25] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:11:55 [I 2022-10-26 10:46:25] ...trap/engines/engine.py.229: process: 0.49037 | load: 0.00246 [I 2022-10-26 10:46:25] ...trap/engines/engine.py.230: loss: 0.37516 [I 2022-10-26 10:46:25] ...optimizers/trijoint.py.151: total_norm: 4.402276 [I 2022-10-26 10:46:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 870/2304 [I 2022-10-26 10:46:30] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:12:01 [I 2022-10-26 10:46:30] ...trap/engines/engine.py.229: process: 0.49153 | load: 0.00248 [I 2022-10-26 10:46:30] ...trap/engines/engine.py.230: loss: 0.28837 [I 2022-10-26 10:46:30] ...optimizers/trijoint.py.151: total_norm: 3.914248 [I 2022-10-26 10:46:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 880/2304 [I 2022-10-26 10:46:35] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:11:54 [I 2022-10-26 10:46:35] ...trap/engines/engine.py.229: process: 0.50125 | load: 0.00242 [I 2022-10-26 10:46:35] ...trap/engines/engine.py.230: loss: 0.25029 [I 2022-10-26 10:46:35] ...optimizers/trijoint.py.151: total_norm: 3.322346 [I 2022-10-26 10:46:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 890/2304 [I 2022-10-26 10:46:40] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:11:43 [I 2022-10-26 10:46:40] ...trap/engines/engine.py.229: process: 0.49695 | load: 0.00270 [I 2022-10-26 10:46:40] ...trap/engines/engine.py.230: loss: 0.24775 [I 2022-10-26 10:46:40] ...optimizers/trijoint.py.151: total_norm: 3.440675 [I 2022-10-26 10:46:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 900/2304 [I 2022-10-26 10:46:45] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:11:46 [I 2022-10-26 10:46:45] ...trap/engines/engine.py.229: process: 0.50408 | load: 0.00260 [I 2022-10-26 10:46:45] ...trap/engines/engine.py.230: loss: 0.33882 [I 2022-10-26 10:46:45] ...optimizers/trijoint.py.151: total_norm: 3.610318 [I 2022-10-26 10:46:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 910/2304 [I 2022-10-26 10:46:50] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:11:39 [I 2022-10-26 10:46:50] ...trap/engines/engine.py.229: process: 0.50712 | load: 0.00250 [I 2022-10-26 10:46:50] ...trap/engines/engine.py.230: loss: 0.28330 [I 2022-10-26 10:46:50] ...optimizers/trijoint.py.151: total_norm: 3.616642 [I 2022-10-26 10:46:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 920/2304 [I 2022-10-26 10:46:55] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:11:31 [I 2022-10-26 10:46:55] ...trap/engines/engine.py.229: process: 0.50027 | load: 0.00261 [I 2022-10-26 10:46:55] ...trap/engines/engine.py.230: loss: 0.33759 [I 2022-10-26 10:46:55] ...optimizers/trijoint.py.151: total_norm: 5.365242 [I 2022-10-26 10:47:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 930/2304 [I 2022-10-26 10:47:00] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:11:20 [I 2022-10-26 10:47:00] ...trap/engines/engine.py.229: process: 0.49932 | load: 0.00257 [I 2022-10-26 10:47:00] ...trap/engines/engine.py.230: loss: 0.25670 [I 2022-10-26 10:47:00] ...optimizers/trijoint.py.151: total_norm: 3.820573 [I 2022-10-26 10:47:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 940/2304 [I 2022-10-26 10:47:05] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:11:27 [I 2022-10-26 10:47:05] ...trap/engines/engine.py.229: process: 0.50150 | load: 0.00247 [I 2022-10-26 10:47:05] ...trap/engines/engine.py.230: loss: 0.35053 [I 2022-10-26 10:47:05] ...optimizers/trijoint.py.151: total_norm: 4.246661 [I 2022-10-26 10:47:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 950/2304 [I 2022-10-26 10:47:10] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:11:18 [I 2022-10-26 10:47:10] ...trap/engines/engine.py.229: process: 0.48098 | load: 0.00292 [I 2022-10-26 10:47:10] ...trap/engines/engine.py.230: loss: 0.25502 [I 2022-10-26 10:47:10] ...optimizers/trijoint.py.151: total_norm: 4.612164 [I 2022-10-26 10:47:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 960/2304 [I 2022-10-26 10:47:15] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:11:13 [I 2022-10-26 10:47:15] ...trap/engines/engine.py.229: process: 0.51179 | load: 0.00260 [I 2022-10-26 10:47:15] ...trap/engines/engine.py.230: loss: 0.21536 [I 2022-10-26 10:47:15] ...optimizers/trijoint.py.151: total_norm: 3.318140 [I 2022-10-26 10:47:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 970/2304 [I 2022-10-26 10:47:20] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:11:02 [I 2022-10-26 10:47:20] ...trap/engines/engine.py.229: process: 0.49370 | load: 0.00260 [I 2022-10-26 10:47:20] ...trap/engines/engine.py.230: loss: 0.26464 [I 2022-10-26 10:47:20] ...optimizers/trijoint.py.151: total_norm: 4.826906 [I 2022-10-26 10:47:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 980/2304 [I 2022-10-26 10:47:25] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:11:00 [I 2022-10-26 10:47:25] ...trap/engines/engine.py.229: process: 0.50457 | load: 0.00247 [I 2022-10-26 10:47:25] ...trap/engines/engine.py.230: loss: 0.36785 [I 2022-10-26 10:47:25] ...optimizers/trijoint.py.151: total_norm: 5.142892 [I 2022-10-26 10:47:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 990/2304 [I 2022-10-26 10:47:30] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:11:20 [I 2022-10-26 10:47:30] ...trap/engines/engine.py.229: process: 0.49685 | load: 0.00242 [I 2022-10-26 10:47:30] ...trap/engines/engine.py.230: loss: 0.27540 [I 2022-10-26 10:47:30] ...optimizers/trijoint.py.151: total_norm: 4.775414 [I 2022-10-26 10:47:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 1000/2304 [I 2022-10-26 10:47:35] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:10:57 [I 2022-10-26 10:47:35] ...trap/engines/engine.py.229: process: 0.50410 | load: 0.00253 [I 2022-10-26 10:47:35] ...trap/engines/engine.py.230: loss: 0.19790 [I 2022-10-26 10:47:35] ...optimizers/trijoint.py.151: total_norm: 3.895298 [I 2022-10-26 10:47:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 1010/2304 [I 2022-10-26 10:47:40] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:10:45 [I 2022-10-26 10:47:40] ...trap/engines/engine.py.229: process: 0.49424 | load: 0.00250 [I 2022-10-26 10:47:40] ...trap/engines/engine.py.230: loss: 0.29008 [I 2022-10-26 10:47:40] ...optimizers/trijoint.py.151: total_norm: 3.847707 [I 2022-10-26 10:47:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 1020/2304 [I 2022-10-26 10:47:45] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:10:40 [I 2022-10-26 10:47:45] ...trap/engines/engine.py.229: process: 0.49430 | load: 0.00327 [I 2022-10-26 10:47:45] ...trap/engines/engine.py.230: loss: 0.23257 [I 2022-10-26 10:47:45] ...optimizers/trijoint.py.151: total_norm: 3.351121 [I 2022-10-26 10:47:51] ...trap/engines/engine.py.224: train: epoch 3 | batch 1030/2304 [I 2022-10-26 10:47:51] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:10:38 [I 2022-10-26 10:47:51] ...trap/engines/engine.py.229: process: 0.49310 | load: 0.00274 [I 2022-10-26 10:47:51] ...trap/engines/engine.py.230: loss: 0.40160 [I 2022-10-26 10:47:51] ...optimizers/trijoint.py.151: total_norm: 5.289288 [I 2022-10-26 10:47:56] ...trap/engines/engine.py.224: train: epoch 3 | batch 1040/2304 [I 2022-10-26 10:47:56] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:10:32 [I 2022-10-26 10:47:56] ...trap/engines/engine.py.229: process: 0.48630 | load: 0.00264 [I 2022-10-26 10:47:56] ...trap/engines/engine.py.230: loss: 0.30882 [I 2022-10-26 10:47:56] ...optimizers/trijoint.py.151: total_norm: 3.727652 [I 2022-10-26 10:48:01] ...trap/engines/engine.py.224: train: epoch 3 | batch 1050/2304 [I 2022-10-26 10:48:01] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:10:26 [I 2022-10-26 10:48:01] ...trap/engines/engine.py.229: process: 0.49853 | load: 0.00252 [I 2022-10-26 10:48:01] ...trap/engines/engine.py.230: loss: 0.28433 [I 2022-10-26 10:48:01] ...optimizers/trijoint.py.151: total_norm: 3.467481 [I 2022-10-26 10:48:06] ...trap/engines/engine.py.224: train: epoch 3 | batch 1060/2304 [I 2022-10-26 10:48:06] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:10:15 [I 2022-10-26 10:48:06] ...trap/engines/engine.py.229: process: 0.50344 | load: 0.00246 [I 2022-10-26 10:48:06] ...trap/engines/engine.py.230: loss: 0.36790 [I 2022-10-26 10:48:06] ...optimizers/trijoint.py.151: total_norm: 4.656795 [I 2022-10-26 10:48:11] ...trap/engines/engine.py.224: train: epoch 3 | batch 1070/2304 [I 2022-10-26 10:48:11] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:10:15 [I 2022-10-26 10:48:11] ...trap/engines/engine.py.229: process: 0.50029 | load: 0.00245 [I 2022-10-26 10:48:11] ...trap/engines/engine.py.230: loss: 0.34083 [I 2022-10-26 10:48:11] ...optimizers/trijoint.py.151: total_norm: 4.597493 [I 2022-10-26 10:48:16] ...trap/engines/engine.py.224: train: epoch 3 | batch 1080/2304 [I 2022-10-26 10:48:16] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:10:15 [I 2022-10-26 10:48:16] ...trap/engines/engine.py.229: process: 0.51311 | load: 0.00252 [I 2022-10-26 10:48:16] ...trap/engines/engine.py.230: loss: 0.26171 [I 2022-10-26 10:48:16] ...optimizers/trijoint.py.151: total_norm: 3.716474 [I 2022-10-26 10:48:21] ...trap/engines/engine.py.224: train: epoch 3 | batch 1090/2304 [I 2022-10-26 10:48:21] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:10:08 [I 2022-10-26 10:48:21] ...trap/engines/engine.py.229: process: 0.50677 | load: 0.00248 [I 2022-10-26 10:48:21] ...trap/engines/engine.py.230: loss: 0.37341 [I 2022-10-26 10:48:21] ...optimizers/trijoint.py.151: total_norm: 4.186347 [I 2022-10-26 10:48:26] ...trap/engines/engine.py.224: train: epoch 3 | batch 1100/2304 [I 2022-10-26 10:48:26] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:10:05 [I 2022-10-26 10:48:26] ...trap/engines/engine.py.229: process: 0.49754 | load: 0.00247 [I 2022-10-26 10:48:26] ...trap/engines/engine.py.230: loss: 0.31635 [I 2022-10-26 10:48:26] ...optimizers/trijoint.py.151: total_norm: 4.383731 [I 2022-10-26 10:48:31] ...trap/engines/engine.py.224: train: epoch 3 | batch 1110/2304 [I 2022-10-26 10:48:31] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:10:20 [I 2022-10-26 10:48:31] ...trap/engines/engine.py.229: process: 0.60179 | load: 0.00243 [I 2022-10-26 10:48:31] ...trap/engines/engine.py.230: loss: 0.30070 [I 2022-10-26 10:48:31] ...optimizers/trijoint.py.151: total_norm: 3.241376 [I 2022-10-26 10:48:36] ...trap/engines/engine.py.224: train: epoch 3 | batch 1120/2304 [I 2022-10-26 10:48:36] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:09:49 [I 2022-10-26 10:48:36] ...trap/engines/engine.py.229: process: 0.49505 | load: 0.00243 [I 2022-10-26 10:48:36] ...trap/engines/engine.py.230: loss: 0.28453 [I 2022-10-26 10:48:36] ...optimizers/trijoint.py.151: total_norm: 3.797685 [I 2022-10-26 10:48:41] ...trap/engines/engine.py.224: train: epoch 3 | batch 1130/2304 [I 2022-10-26 10:48:41] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:09:40 [I 2022-10-26 10:48:41] ...trap/engines/engine.py.229: process: 0.48950 | load: 0.00247 [I 2022-10-26 10:48:41] ...trap/engines/engine.py.230: loss: 0.36512 [I 2022-10-26 10:48:41] ...optimizers/trijoint.py.151: total_norm: 4.817341 [I 2022-10-26 10:48:46] ...trap/engines/engine.py.224: train: epoch 3 | batch 1140/2304 [I 2022-10-26 10:48:46] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:09:43 [I 2022-10-26 10:48:46] ...trap/engines/engine.py.229: process: 0.50841 | load: 0.00244 [I 2022-10-26 10:48:46] ...trap/engines/engine.py.230: loss: 0.25452 [I 2022-10-26 10:48:46] ...optimizers/trijoint.py.151: total_norm: 3.409524 [I 2022-10-26 10:48:51] ...trap/engines/engine.py.224: train: epoch 3 | batch 1150/2304 [I 2022-10-26 10:48:51] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:09:32 [I 2022-10-26 10:48:51] ...trap/engines/engine.py.229: process: 0.49201 | load: 0.00241 [I 2022-10-26 10:48:51] ...trap/engines/engine.py.230: loss: 0.38689 [I 2022-10-26 10:48:51] ...optimizers/trijoint.py.151: total_norm: 4.513097 [I 2022-10-26 10:48:56] ...trap/engines/engine.py.224: train: epoch 3 | batch 1160/2304 [I 2022-10-26 10:48:56] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:09:26 [I 2022-10-26 10:48:56] ...trap/engines/engine.py.229: process: 0.49477 | load: 0.00245 [I 2022-10-26 10:48:56] ...trap/engines/engine.py.230: loss: 0.37776 [I 2022-10-26 10:48:56] ...optimizers/trijoint.py.151: total_norm: 4.677588 [I 2022-10-26 10:49:01] ...trap/engines/engine.py.224: train: epoch 3 | batch 1170/2304 [I 2022-10-26 10:49:01] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:09:22 [I 2022-10-26 10:49:01] ...trap/engines/engine.py.229: process: 0.50187 | load: 0.00241 [I 2022-10-26 10:49:01] ...trap/engines/engine.py.230: loss: 0.43410 [I 2022-10-26 10:49:01] ...optimizers/trijoint.py.151: total_norm: 5.370604 [I 2022-10-26 10:49:06] ...trap/engines/engine.py.224: train: epoch 3 | batch 1180/2304 [I 2022-10-26 10:49:06] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:09:21 [I 2022-10-26 10:49:06] ...trap/engines/engine.py.229: process: 0.50247 | load: 0.00242 [I 2022-10-26 10:49:06] ...trap/engines/engine.py.230: loss: 0.35231 [I 2022-10-26 10:49:06] ...optimizers/trijoint.py.151: total_norm: 3.455196 [I 2022-10-26 10:49:11] ...trap/engines/engine.py.224: train: epoch 3 | batch 1190/2304 [I 2022-10-26 10:49:11] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:09:10 [I 2022-10-26 10:49:11] ...trap/engines/engine.py.229: process: 0.48645 | load: 0.00243 [I 2022-10-26 10:49:11] ...trap/engines/engine.py.230: loss: 0.28075 [I 2022-10-26 10:49:11] ...optimizers/trijoint.py.151: total_norm: 3.598725 [I 2022-10-26 10:49:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 1200/2304 [I 2022-10-26 10:49:15] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:09:06 [I 2022-10-26 10:49:15] ...trap/engines/engine.py.229: process: 0.49969 | load: 0.00241 [I 2022-10-26 10:49:15] ...trap/engines/engine.py.230: loss: 0.35929 [I 2022-10-26 10:49:15] ...optimizers/trijoint.py.151: total_norm: 4.753610 [I 2022-10-26 10:49:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 1210/2304 [I 2022-10-26 10:49:20] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:09:01 [I 2022-10-26 10:49:20] ...trap/engines/engine.py.229: process: 0.48211 | load: 0.00251 [I 2022-10-26 10:49:20] ...trap/engines/engine.py.230: loss: 0.37524 [I 2022-10-26 10:49:20] ...optimizers/trijoint.py.151: total_norm: 4.138712 [I 2022-10-26 10:49:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 1220/2304 [I 2022-10-26 10:49:25] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:09:00 [I 2022-10-26 10:49:25] ...trap/engines/engine.py.229: process: 0.49685 | load: 0.00249 [I 2022-10-26 10:49:25] ...trap/engines/engine.py.230: loss: 0.27311 [I 2022-10-26 10:49:25] ...optimizers/trijoint.py.151: total_norm: 3.501186 [I 2022-10-26 10:49:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 1230/2304 [I 2022-10-26 10:49:30] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:08:50 [I 2022-10-26 10:49:30] ...trap/engines/engine.py.229: process: 0.48918 | load: 0.00245 [I 2022-10-26 10:49:30] ...trap/engines/engine.py.230: loss: 0.29250 [I 2022-10-26 10:49:30] ...optimizers/trijoint.py.151: total_norm: 3.369260 [I 2022-10-26 10:49:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 1240/2304 [I 2022-10-26 10:49:35] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:08:51 [I 2022-10-26 10:49:35] ...trap/engines/engine.py.229: process: 0.48527 | load: 0.00242 [I 2022-10-26 10:49:35] ...trap/engines/engine.py.230: loss: 0.30397 [I 2022-10-26 10:49:35] ...optimizers/trijoint.py.151: total_norm: 3.598985 [I 2022-10-26 10:49:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 1250/2304 [I 2022-10-26 10:49:40] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:08:39 [I 2022-10-26 10:49:40] ...trap/engines/engine.py.229: process: 0.48180 | load: 0.00244 [I 2022-10-26 10:49:40] ...trap/engines/engine.py.230: loss: 0.20550 [I 2022-10-26 10:49:40] ...optimizers/trijoint.py.151: total_norm: 3.063443 [I 2022-10-26 10:49:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 1260/2304 [I 2022-10-26 10:49:45] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:08:36 [I 2022-10-26 10:49:45] ...trap/engines/engine.py.229: process: 0.50217 | load: 0.00242 [I 2022-10-26 10:49:45] ...trap/engines/engine.py.230: loss: 0.37702 [I 2022-10-26 10:49:45] ...optimizers/trijoint.py.151: total_norm: 4.516772 [I 2022-10-26 10:49:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 1270/2304 [I 2022-10-26 10:49:50] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:08:33 [I 2022-10-26 10:49:50] ...trap/engines/engine.py.229: process: 0.50270 | load: 0.00247 [I 2022-10-26 10:49:50] ...trap/engines/engine.py.230: loss: 0.24946 [I 2022-10-26 10:49:50] ...optimizers/trijoint.py.151: total_norm: 3.184981 [I 2022-10-26 10:49:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 1280/2304 [I 2022-10-26 10:49:55] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:08:28 [I 2022-10-26 10:49:55] ...trap/engines/engine.py.229: process: 0.49577 | load: 0.00253 [I 2022-10-26 10:49:55] ...trap/engines/engine.py.230: loss: 0.30005 [I 2022-10-26 10:49:55] ...optimizers/trijoint.py.151: total_norm: 3.956112 [I 2022-10-26 10:50:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 1290/2304 [I 2022-10-26 10:50:00] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:08:20 [I 2022-10-26 10:50:00] ...trap/engines/engine.py.229: process: 0.49093 | load: 0.00247 [I 2022-10-26 10:50:00] ...trap/engines/engine.py.230: loss: 0.41295 [I 2022-10-26 10:50:00] ...optimizers/trijoint.py.151: total_norm: 4.023728 [I 2022-10-26 10:50:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 1300/2304 [I 2022-10-26 10:50:05] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:08:17 [I 2022-10-26 10:50:05] ...trap/engines/engine.py.229: process: 0.49812 | load: 0.00247 [I 2022-10-26 10:50:05] ...trap/engines/engine.py.230: loss: 0.24108 [I 2022-10-26 10:50:05] ...optimizers/trijoint.py.151: total_norm: 3.722191 [I 2022-10-26 10:50:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 1310/2304 [I 2022-10-26 10:50:10] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:08:11 [I 2022-10-26 10:50:10] ...trap/engines/engine.py.229: process: 0.48615 | load: 0.00243 [I 2022-10-26 10:50:10] ...trap/engines/engine.py.230: loss: 0.23133 [I 2022-10-26 10:50:10] ...optimizers/trijoint.py.151: total_norm: 4.813662 [I 2022-10-26 10:50:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 1320/2304 [I 2022-10-26 10:50:15] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:08:15 [I 2022-10-26 10:50:15] ...trap/engines/engine.py.229: process: 0.49399 | load: 0.00242 [I 2022-10-26 10:50:15] ...trap/engines/engine.py.230: loss: 0.29880 [I 2022-10-26 10:50:15] ...optimizers/trijoint.py.151: total_norm: 3.581188 [I 2022-10-26 10:50:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 1330/2304 [I 2022-10-26 10:50:20] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:08:05 [I 2022-10-26 10:50:20] ...trap/engines/engine.py.229: process: 0.50242 | load: 0.00251 [I 2022-10-26 10:50:20] ...trap/engines/engine.py.230: loss: 0.43494 [I 2022-10-26 10:50:20] ...optimizers/trijoint.py.151: total_norm: 4.521018 [I 2022-10-26 10:50:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 1340/2304 [I 2022-10-26 10:50:25] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:07:59 [I 2022-10-26 10:50:25] ...trap/engines/engine.py.229: process: 0.49228 | load: 0.00263 [I 2022-10-26 10:50:25] ...trap/engines/engine.py.230: loss: 0.29201 [I 2022-10-26 10:50:25] ...optimizers/trijoint.py.151: total_norm: 3.866321 [I 2022-10-26 10:50:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 1350/2304 [I 2022-10-26 10:50:30] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:07:53 [I 2022-10-26 10:50:30] ...trap/engines/engine.py.229: process: 0.49908 | load: 0.00245 [I 2022-10-26 10:50:30] ...trap/engines/engine.py.230: loss: 0.41052 [I 2022-10-26 10:50:30] ...optimizers/trijoint.py.151: total_norm: 4.480718 [I 2022-10-26 10:50:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 1360/2304 [I 2022-10-26 10:50:35] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:07:45 [I 2022-10-26 10:50:35] ...trap/engines/engine.py.229: process: 0.49018 | load: 0.00246 [I 2022-10-26 10:50:35] ...trap/engines/engine.py.230: loss: 0.32316 [I 2022-10-26 10:50:35] ...optimizers/trijoint.py.151: total_norm: 3.284195 [I 2022-10-26 10:50:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 1370/2304 [I 2022-10-26 10:50:40] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:07:46 [I 2022-10-26 10:50:40] ...trap/engines/engine.py.229: process: 0.49982 | load: 0.00241 [I 2022-10-26 10:50:40] ...trap/engines/engine.py.230: loss: 0.29239 [I 2022-10-26 10:50:40] ...optimizers/trijoint.py.151: total_norm: 3.408463 [I 2022-10-26 10:50:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 1380/2304 [I 2022-10-26 10:50:45] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:07:38 [I 2022-10-26 10:50:45] ...trap/engines/engine.py.229: process: 0.48916 | load: 0.00248 [I 2022-10-26 10:50:45] ...trap/engines/engine.py.230: loss: 0.35218 [I 2022-10-26 10:50:45] ...optimizers/trijoint.py.151: total_norm: 3.689927 [I 2022-10-26 10:50:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 1390/2304 [I 2022-10-26 10:50:50] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:07:33 [I 2022-10-26 10:50:50] ...trap/engines/engine.py.229: process: 0.50197 | load: 0.00246 [I 2022-10-26 10:50:50] ...trap/engines/engine.py.230: loss: 0.32886 [I 2022-10-26 10:50:50] ...optimizers/trijoint.py.151: total_norm: 4.189136 [I 2022-10-26 10:50:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 1400/2304 [I 2022-10-26 10:50:55] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:07:22 [I 2022-10-26 10:50:55] ...trap/engines/engine.py.229: process: 0.48380 | load: 0.00245 [I 2022-10-26 10:50:55] ...trap/engines/engine.py.230: loss: 0.25869 [I 2022-10-26 10:50:55] ...optimizers/trijoint.py.151: total_norm: 3.409130 [I 2022-10-26 10:51:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 1410/2304 [I 2022-10-26 10:51:00] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:07:21 [I 2022-10-26 10:51:00] ...trap/engines/engine.py.229: process: 0.48734 | load: 0.00244 [I 2022-10-26 10:51:00] ...trap/engines/engine.py.230: loss: 0.27191 [I 2022-10-26 10:51:00] ...optimizers/trijoint.py.151: total_norm: 3.570301 [I 2022-10-26 10:51:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 1420/2304 [I 2022-10-26 10:51:05] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:07:16 [I 2022-10-26 10:51:05] ...trap/engines/engine.py.229: process: 0.49632 | load: 0.00241 [I 2022-10-26 10:51:05] ...trap/engines/engine.py.230: loss: 0.25551 [I 2022-10-26 10:51:05] ...optimizers/trijoint.py.151: total_norm: 3.267364 [I 2022-10-26 10:51:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 1430/2304 [I 2022-10-26 10:51:10] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:07:11 [I 2022-10-26 10:51:10] ...trap/engines/engine.py.229: process: 0.49505 | load: 0.00245 [I 2022-10-26 10:51:10] ...trap/engines/engine.py.230: loss: 0.40128 [I 2022-10-26 10:51:10] ...optimizers/trijoint.py.151: total_norm: 4.371633 [I 2022-10-26 10:51:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 1440/2304 [I 2022-10-26 10:51:15] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:07:04 [I 2022-10-26 10:51:15] ...trap/engines/engine.py.229: process: 0.49225 | load: 0.00242 [I 2022-10-26 10:51:15] ...trap/engines/engine.py.230: loss: 0.20229 [I 2022-10-26 10:51:15] ...optimizers/trijoint.py.151: total_norm: 3.291367 [I 2022-10-26 10:51:20] ...trap/engines/engine.py.224: train: epoch 3 | batch 1450/2304 [I 2022-10-26 10:51:20] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:07:04 [I 2022-10-26 10:51:20] ...trap/engines/engine.py.229: process: 0.49687 | load: 0.00258 [I 2022-10-26 10:51:20] ...trap/engines/engine.py.230: loss: 0.34619 [I 2022-10-26 10:51:20] ...optimizers/trijoint.py.151: total_norm: 3.778196 [I 2022-10-26 10:51:25] ...trap/engines/engine.py.224: train: epoch 3 | batch 1460/2304 [I 2022-10-26 10:51:25] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:07:00 [I 2022-10-26 10:51:25] ...trap/engines/engine.py.229: process: 0.50178 | load: 0.00246 [I 2022-10-26 10:51:25] ...trap/engines/engine.py.230: loss: 0.25718 [I 2022-10-26 10:51:25] ...optimizers/trijoint.py.151: total_norm: 2.744503 [I 2022-10-26 10:51:30] ...trap/engines/engine.py.224: train: epoch 3 | batch 1470/2304 [I 2022-10-26 10:51:30] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:06:53 [I 2022-10-26 10:51:30] ...trap/engines/engine.py.229: process: 0.49735 | load: 0.00246 [I 2022-10-26 10:51:30] ...trap/engines/engine.py.230: loss: 0.31256 [I 2022-10-26 10:51:30] ...optimizers/trijoint.py.151: total_norm: 4.015777 [I 2022-10-26 10:51:35] ...trap/engines/engine.py.224: train: epoch 3 | batch 1480/2304 [I 2022-10-26 10:51:35] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:06:48 [I 2022-10-26 10:51:35] ...trap/engines/engine.py.229: process: 0.49643 | load: 0.00244 [I 2022-10-26 10:51:35] ...trap/engines/engine.py.230: loss: 0.27132 [I 2022-10-26 10:51:35] ...optimizers/trijoint.py.151: total_norm: 4.114854 [I 2022-10-26 10:51:40] ...trap/engines/engine.py.224: train: epoch 3 | batch 1490/2304 [I 2022-10-26 10:51:40] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:06:43 [I 2022-10-26 10:51:40] ...trap/engines/engine.py.229: process: 0.49536 | load: 0.00243 [I 2022-10-26 10:51:40] ...trap/engines/engine.py.230: loss: 0.30522 [I 2022-10-26 10:51:40] ...optimizers/trijoint.py.151: total_norm: 3.369683 [I 2022-10-26 10:51:45] ...trap/engines/engine.py.224: train: epoch 3 | batch 1500/2304 [I 2022-10-26 10:51:45] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:06:36 [I 2022-10-26 10:51:45] ...trap/engines/engine.py.229: process: 0.49199 | load: 0.00241 [I 2022-10-26 10:51:45] ...trap/engines/engine.py.230: loss: 0.32868 [I 2022-10-26 10:51:45] ...optimizers/trijoint.py.151: total_norm: 4.331511 [I 2022-10-26 10:51:50] ...trap/engines/engine.py.224: train: epoch 3 | batch 1510/2304 [I 2022-10-26 10:51:50] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:06:32 [I 2022-10-26 10:51:50] ...trap/engines/engine.py.229: process: 0.49575 | load: 0.00244 [I 2022-10-26 10:51:50] ...trap/engines/engine.py.230: loss: 0.32640 [I 2022-10-26 10:51:50] ...optimizers/trijoint.py.151: total_norm: 3.932878 [I 2022-10-26 10:51:55] ...trap/engines/engine.py.224: train: epoch 3 | batch 1520/2304 [I 2022-10-26 10:51:55] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:06:31 [I 2022-10-26 10:51:55] ...trap/engines/engine.py.229: process: 0.48966 | load: 0.00243 [I 2022-10-26 10:51:55] ...trap/engines/engine.py.230: loss: 0.37637 [I 2022-10-26 10:51:55] ...optimizers/trijoint.py.151: total_norm: 4.395252 [I 2022-10-26 10:52:00] ...trap/engines/engine.py.224: train: epoch 3 | batch 1530/2304 [I 2022-10-26 10:52:00] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:06:25 [I 2022-10-26 10:52:00] ...trap/engines/engine.py.229: process: 0.50411 | load: 0.00255 [I 2022-10-26 10:52:00] ...trap/engines/engine.py.230: loss: 0.43704 [I 2022-10-26 10:52:00] ...optimizers/trijoint.py.151: total_norm: 5.535292 [I 2022-10-26 10:52:05] ...trap/engines/engine.py.224: train: epoch 3 | batch 1540/2304 [I 2022-10-26 10:52:05] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:06:15 [I 2022-10-26 10:52:05] ...trap/engines/engine.py.229: process: 0.48225 | load: 0.00243 [I 2022-10-26 10:52:05] ...trap/engines/engine.py.230: loss: 0.30281 [I 2022-10-26 10:52:05] ...optimizers/trijoint.py.151: total_norm: 4.363162 [I 2022-10-26 10:52:10] ...trap/engines/engine.py.224: train: epoch 3 | batch 1550/2304 [I 2022-10-26 10:52:10] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:06:12 [I 2022-10-26 10:52:10] ...trap/engines/engine.py.229: process: 0.49512 | load: 0.00241 [I 2022-10-26 10:52:10] ...trap/engines/engine.py.230: loss: 0.33860 [I 2022-10-26 10:52:10] ...optimizers/trijoint.py.151: total_norm: 3.841261 [I 2022-10-26 10:52:15] ...trap/engines/engine.py.224: train: epoch 3 | batch 1560/2304 [I 2022-10-26 10:52:15] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:06:11 [I 2022-10-26 10:52:15] ...trap/engines/engine.py.229: process: 0.50861 | load: 0.00243 [I 2022-10-26 10:52:15] ...trap/engines/engine.py.230: loss: 0.23202 [I 2022-10-26 10:52:15] ...optimizers/trijoint.py.151: total_norm: 2.797324 [I 2022-10-26 10:52:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 1570/2304 [I 2022-10-26 10:52:19] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:06:04 [I 2022-10-26 10:52:19] ...trap/engines/engine.py.229: process: 0.50062 | load: 0.00244 [I 2022-10-26 10:52:19] ...trap/engines/engine.py.230: loss: 0.33835 [I 2022-10-26 10:52:19] ...optimizers/trijoint.py.151: total_norm: 3.504319 [I 2022-10-26 10:52:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 1580/2304 [I 2022-10-26 10:52:24] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:05:56 [I 2022-10-26 10:52:24] ...trap/engines/engine.py.229: process: 0.48915 | load: 0.00243 [I 2022-10-26 10:52:24] ...trap/engines/engine.py.230: loss: 0.27471 [I 2022-10-26 10:52:24] ...optimizers/trijoint.py.151: total_norm: 2.993283 [I 2022-10-26 10:52:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 1590/2304 [I 2022-10-26 10:52:29] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:05:52 [I 2022-10-26 10:52:29] ...trap/engines/engine.py.229: process: 0.49686 | load: 0.00242 [I 2022-10-26 10:52:29] ...trap/engines/engine.py.230: loss: 0.18595 [I 2022-10-26 10:52:29] ...optimizers/trijoint.py.151: total_norm: 2.533073 [I 2022-10-26 10:52:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 1600/2304 [I 2022-10-26 10:52:34] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:05:51 [I 2022-10-26 10:52:34] ...trap/engines/engine.py.229: process: 0.50433 | load: 0.00242 [I 2022-10-26 10:52:34] ...trap/engines/engine.py.230: loss: 0.31564 [I 2022-10-26 10:52:34] ...optimizers/trijoint.py.151: total_norm: 4.157302 [I 2022-10-26 10:52:39] ...trap/engines/engine.py.224: train: epoch 3 | batch 1610/2304 [I 2022-10-26 10:52:39] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:05:45 [I 2022-10-26 10:52:39] ...trap/engines/engine.py.229: process: 0.49889 | load: 0.00239 [I 2022-10-26 10:52:39] ...trap/engines/engine.py.230: loss: 0.35290 [I 2022-10-26 10:52:39] ...optimizers/trijoint.py.151: total_norm: 3.830933 [I 2022-10-26 10:52:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 1620/2304 [I 2022-10-26 10:52:44] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:05:38 [I 2022-10-26 10:52:44] ...trap/engines/engine.py.229: process: 0.49021 | load: 0.00240 [I 2022-10-26 10:52:44] ...trap/engines/engine.py.230: loss: 0.26205 [I 2022-10-26 10:52:44] ...optimizers/trijoint.py.151: total_norm: 4.953189 [I 2022-10-26 10:52:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 1630/2304 [I 2022-10-26 10:52:49] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:05:33 [I 2022-10-26 10:52:49] ...trap/engines/engine.py.229: process: 0.48682 | load: 0.00240 [I 2022-10-26 10:52:49] ...trap/engines/engine.py.230: loss: 0.24564 [I 2022-10-26 10:52:49] ...optimizers/trijoint.py.151: total_norm: 3.236665 [I 2022-10-26 10:52:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 1640/2304 [I 2022-10-26 10:52:54] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:05:30 [I 2022-10-26 10:52:54] ...trap/engines/engine.py.229: process: 0.50204 | load: 0.00240 [I 2022-10-26 10:52:54] ...trap/engines/engine.py.230: loss: 0.27079 [I 2022-10-26 10:52:54] ...optimizers/trijoint.py.151: total_norm: 3.119426 [I 2022-10-26 10:52:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 1650/2304 [I 2022-10-26 10:52:59] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:05:22 [I 2022-10-26 10:52:59] ...trap/engines/engine.py.229: process: 0.49812 | load: 0.00240 [I 2022-10-26 10:52:59] ...trap/engines/engine.py.230: loss: 0.24804 [I 2022-10-26 10:52:59] ...optimizers/trijoint.py.151: total_norm: 3.466458 [I 2022-10-26 10:53:04] ...trap/engines/engine.py.224: train: epoch 3 | batch 1660/2304 [I 2022-10-26 10:53:04] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:05:18 [I 2022-10-26 10:53:04] ...trap/engines/engine.py.229: process: 0.48563 | load: 0.00250 [I 2022-10-26 10:53:04] ...trap/engines/engine.py.230: loss: 0.31043 [I 2022-10-26 10:53:04] ...optimizers/trijoint.py.151: total_norm: 3.664551 [I 2022-10-26 10:53:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 1670/2304 [I 2022-10-26 10:53:09] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:05:14 [I 2022-10-26 10:53:09] ...trap/engines/engine.py.229: process: 0.49649 | load: 0.00247 [I 2022-10-26 10:53:09] ...trap/engines/engine.py.230: loss: 0.36090 [I 2022-10-26 10:53:09] ...optimizers/trijoint.py.151: total_norm: 3.771563 [I 2022-10-26 10:53:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 1680/2304 [I 2022-10-26 10:53:14] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:05:09 [I 2022-10-26 10:53:14] ...trap/engines/engine.py.229: process: 0.48482 | load: 0.00244 [I 2022-10-26 10:53:14] ...trap/engines/engine.py.230: loss: 0.32601 [I 2022-10-26 10:53:14] ...optimizers/trijoint.py.151: total_norm: 4.374679 [I 2022-10-26 10:53:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 1690/2304 [I 2022-10-26 10:53:19] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:05:05 [I 2022-10-26 10:53:19] ...trap/engines/engine.py.229: process: 0.49910 | load: 0.00246 [I 2022-10-26 10:53:19] ...trap/engines/engine.py.230: loss: 0.31324 [I 2022-10-26 10:53:19] ...optimizers/trijoint.py.151: total_norm: 3.644497 [I 2022-10-26 10:53:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 1700/2304 [I 2022-10-26 10:53:24] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:04:58 [I 2022-10-26 10:53:24] ...trap/engines/engine.py.229: process: 0.48660 | load: 0.00243 [I 2022-10-26 10:53:24] ...trap/engines/engine.py.230: loss: 0.29140 [I 2022-10-26 10:53:24] ...optimizers/trijoint.py.151: total_norm: 3.527965 [I 2022-10-26 10:53:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 1710/2304 [I 2022-10-26 10:53:29] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:04:55 [I 2022-10-26 10:53:29] ...trap/engines/engine.py.229: process: 0.49440 | load: 0.00245 [I 2022-10-26 10:53:29] ...trap/engines/engine.py.230: loss: 0.40386 [I 2022-10-26 10:53:29] ...optimizers/trijoint.py.151: total_norm: 4.299681 [I 2022-10-26 10:53:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 1720/2304 [I 2022-10-26 10:53:34] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:04:48 [I 2022-10-26 10:53:34] ...trap/engines/engine.py.229: process: 0.48472 | load: 0.00241 [I 2022-10-26 10:53:34] ...trap/engines/engine.py.230: loss: 0.31169 [I 2022-10-26 10:53:34] ...optimizers/trijoint.py.151: total_norm: 4.454794 [I 2022-10-26 10:53:39] ...trap/engines/engine.py.224: train: epoch 3 | batch 1730/2304 [I 2022-10-26 10:53:39] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:04:46 [I 2022-10-26 10:53:39] ...trap/engines/engine.py.229: process: 0.51267 | load: 0.00244 [I 2022-10-26 10:53:39] ...trap/engines/engine.py.230: loss: 0.23468 [I 2022-10-26 10:53:39] ...optimizers/trijoint.py.151: total_norm: 4.189833 [I 2022-10-26 10:53:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 1740/2304 [I 2022-10-26 10:53:44] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:04:38 [I 2022-10-26 10:53:44] ...trap/engines/engine.py.229: process: 0.48833 | load: 0.00246 [I 2022-10-26 10:53:44] ...trap/engines/engine.py.230: loss: 0.33150 [I 2022-10-26 10:53:44] ...optimizers/trijoint.py.151: total_norm: 4.190816 [I 2022-10-26 10:53:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 1750/2304 [I 2022-10-26 10:53:49] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:04:35 [I 2022-10-26 10:53:49] ...trap/engines/engine.py.229: process: 0.48929 | load: 0.00245 [I 2022-10-26 10:53:49] ...trap/engines/engine.py.230: loss: 0.34653 [I 2022-10-26 10:53:49] ...optimizers/trijoint.py.151: total_norm: 4.709126 [I 2022-10-26 10:53:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 1760/2304 [I 2022-10-26 10:53:54] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:04:29 [I 2022-10-26 10:53:54] ...trap/engines/engine.py.229: process: 0.49046 | load: 0.00243 [I 2022-10-26 10:53:54] ...trap/engines/engine.py.230: loss: 0.30701 [I 2022-10-26 10:53:54] ...optimizers/trijoint.py.151: total_norm: 3.855257 [I 2022-10-26 10:53:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 1770/2304 [I 2022-10-26 10:53:59] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:04:25 [I 2022-10-26 10:53:59] ...trap/engines/engine.py.229: process: 0.49611 | load: 0.00243 [I 2022-10-26 10:53:59] ...trap/engines/engine.py.230: loss: 0.32382 [I 2022-10-26 10:53:59] ...optimizers/trijoint.py.151: total_norm: 4.095091 [I 2022-10-26 10:54:04] ...trap/engines/engine.py.224: train: epoch 3 | batch 1780/2304 [I 2022-10-26 10:54:04] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:04:20 [I 2022-10-26 10:54:04] ...trap/engines/engine.py.229: process: 0.49320 | load: 0.00258 [I 2022-10-26 10:54:04] ...trap/engines/engine.py.230: loss: 0.28189 [I 2022-10-26 10:54:04] ...optimizers/trijoint.py.151: total_norm: 3.252006 [I 2022-10-26 10:54:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 1790/2304 [I 2022-10-26 10:54:09] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:04:15 [I 2022-10-26 10:54:09] ...trap/engines/engine.py.229: process: 0.49392 | load: 0.00249 [I 2022-10-26 10:54:09] ...trap/engines/engine.py.230: loss: 0.34036 [I 2022-10-26 10:54:09] ...optimizers/trijoint.py.151: total_norm: 4.385663 [I 2022-10-26 10:54:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 1800/2304 [I 2022-10-26 10:54:14] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:04:10 [I 2022-10-26 10:54:14] ...trap/engines/engine.py.229: process: 0.50691 | load: 0.00246 [I 2022-10-26 10:54:14] ...trap/engines/engine.py.230: loss: 0.24038 [I 2022-10-26 10:54:14] ...optimizers/trijoint.py.151: total_norm: 3.061443 [I 2022-10-26 10:54:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 1810/2304 [I 2022-10-26 10:54:19] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:04:05 [I 2022-10-26 10:54:19] ...trap/engines/engine.py.229: process: 0.49810 | load: 0.00246 [I 2022-10-26 10:54:19] ...trap/engines/engine.py.230: loss: 0.30319 [I 2022-10-26 10:54:19] ...optimizers/trijoint.py.151: total_norm: 4.479308 [I 2022-10-26 10:54:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 1820/2304 [I 2022-10-26 10:54:24] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:04:03 [I 2022-10-26 10:54:24] ...trap/engines/engine.py.229: process: 0.52750 | load: 0.00243 [I 2022-10-26 10:54:24] ...trap/engines/engine.py.230: loss: 0.37240 [I 2022-10-26 10:54:24] ...optimizers/trijoint.py.151: total_norm: 4.757746 [I 2022-10-26 10:54:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 1830/2304 [I 2022-10-26 10:54:29] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:03:56 [I 2022-10-26 10:54:29] ...trap/engines/engine.py.229: process: 0.49919 | load: 0.00246 [I 2022-10-26 10:54:29] ...trap/engines/engine.py.230: loss: 0.25777 [I 2022-10-26 10:54:29] ...optimizers/trijoint.py.151: total_norm: 3.656775 [I 2022-10-26 10:54:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 1840/2304 [I 2022-10-26 10:54:34] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:03:52 [I 2022-10-26 10:54:34] ...trap/engines/engine.py.229: process: 0.50073 | load: 0.00257 [I 2022-10-26 10:54:34] ...trap/engines/engine.py.230: loss: 0.24569 [I 2022-10-26 10:54:34] ...optimizers/trijoint.py.151: total_norm: 3.637363 [I 2022-10-26 10:54:39] ...trap/engines/engine.py.224: train: epoch 3 | batch 1850/2304 [I 2022-10-26 10:54:39] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:03:46 [I 2022-10-26 10:54:39] ...trap/engines/engine.py.229: process: 0.50081 | load: 0.00250 [I 2022-10-26 10:54:39] ...trap/engines/engine.py.230: loss: 0.26242 [I 2022-10-26 10:54:39] ...optimizers/trijoint.py.151: total_norm: 3.500561 [I 2022-10-26 10:54:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 1860/2304 [I 2022-10-26 10:54:44] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:03:41 [I 2022-10-26 10:54:44] ...trap/engines/engine.py.229: process: 0.50443 | load: 0.00253 [I 2022-10-26 10:54:44] ...trap/engines/engine.py.230: loss: 0.32161 [I 2022-10-26 10:54:44] ...optimizers/trijoint.py.151: total_norm: 4.356975 [I 2022-10-26 10:54:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 1870/2304 [I 2022-10-26 10:54:49] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:03:36 [I 2022-10-26 10:54:49] ...trap/engines/engine.py.229: process: 0.49738 | load: 0.00265 [I 2022-10-26 10:54:49] ...trap/engines/engine.py.230: loss: 0.26813 [I 2022-10-26 10:54:49] ...optimizers/trijoint.py.151: total_norm: 4.785460 [I 2022-10-26 10:54:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 1880/2304 [I 2022-10-26 10:54:54] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:03:34 [I 2022-10-26 10:54:54] ...trap/engines/engine.py.229: process: 0.50591 | load: 0.00250 [I 2022-10-26 10:54:54] ...trap/engines/engine.py.230: loss: 0.38393 [I 2022-10-26 10:54:54] ...optimizers/trijoint.py.151: total_norm: 4.066545 [I 2022-10-26 10:54:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 1890/2304 [I 2022-10-26 10:54:59] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:03:28 [I 2022-10-26 10:54:59] ...trap/engines/engine.py.229: process: 0.49321 | load: 0.00247 [I 2022-10-26 10:54:59] ...trap/engines/engine.py.230: loss: 0.30386 [I 2022-10-26 10:54:59] ...optimizers/trijoint.py.151: total_norm: 3.907422 [I 2022-10-26 10:55:04] ...trap/engines/engine.py.224: train: epoch 3 | batch 1900/2304 [I 2022-10-26 10:55:04] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:03:22 [I 2022-10-26 10:55:04] ...trap/engines/engine.py.229: process: 0.49947 | load: 0.00246 [I 2022-10-26 10:55:04] ...trap/engines/engine.py.230: loss: 0.31163 [I 2022-10-26 10:55:04] ...optimizers/trijoint.py.151: total_norm: 4.410619 [I 2022-10-26 10:55:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 1910/2304 [I 2022-10-26 10:55:09] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:03:18 [I 2022-10-26 10:55:09] ...trap/engines/engine.py.229: process: 0.50326 | load: 0.00245 [I 2022-10-26 10:55:09] ...trap/engines/engine.py.230: loss: 0.22757 [I 2022-10-26 10:55:09] ...optimizers/trijoint.py.151: total_norm: 3.051540 [I 2022-10-26 10:55:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 1920/2304 [I 2022-10-26 10:55:14] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:03:19 [I 2022-10-26 10:55:14] ...trap/engines/engine.py.229: process: 0.60601 | load: 0.00265 [I 2022-10-26 10:55:14] ...trap/engines/engine.py.230: loss: 0.35244 [I 2022-10-26 10:55:14] ...optimizers/trijoint.py.151: total_norm: 4.872014 [I 2022-10-26 10:55:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 1930/2304 [I 2022-10-26 10:55:19] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:03:07 [I 2022-10-26 10:55:19] ...trap/engines/engine.py.229: process: 0.48566 | load: 0.00244 [I 2022-10-26 10:55:19] ...trap/engines/engine.py.230: loss: 0.32351 [I 2022-10-26 10:55:19] ...optimizers/trijoint.py.151: total_norm: 4.036644 [I 2022-10-26 10:55:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 1940/2304 [I 2022-10-26 10:55:24] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:03:02 [I 2022-10-26 10:55:24] ...trap/engines/engine.py.229: process: 0.49603 | load: 0.00243 [I 2022-10-26 10:55:24] ...trap/engines/engine.py.230: loss: 0.35286 [I 2022-10-26 10:55:24] ...optimizers/trijoint.py.151: total_norm: 4.232891 [I 2022-10-26 10:55:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 1950/2304 [I 2022-10-26 10:55:29] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:02:55 [I 2022-10-26 10:55:29] ...trap/engines/engine.py.229: process: 0.49178 | load: 0.00240 [I 2022-10-26 10:55:29] ...trap/engines/engine.py.230: loss: 0.31267 [I 2022-10-26 10:55:29] ...optimizers/trijoint.py.151: total_norm: 3.287278 [I 2022-10-26 10:55:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 1960/2304 [I 2022-10-26 10:55:34] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:02:50 [I 2022-10-26 10:55:34] ...trap/engines/engine.py.229: process: 0.49748 | load: 0.00247 [I 2022-10-26 10:55:34] ...trap/engines/engine.py.230: loss: 0.25398 [I 2022-10-26 10:55:34] ...optimizers/trijoint.py.151: total_norm: 3.710576 [I 2022-10-26 10:55:39] ...trap/engines/engine.py.224: train: epoch 3 | batch 1970/2304 [I 2022-10-26 10:55:39] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:02:44 [I 2022-10-26 10:55:39] ...trap/engines/engine.py.229: process: 0.49224 | load: 0.00244 [I 2022-10-26 10:55:39] ...trap/engines/engine.py.230: loss: 0.27458 [I 2022-10-26 10:55:39] ...optimizers/trijoint.py.151: total_norm: 3.359241 [I 2022-10-26 10:55:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 1980/2304 [I 2022-10-26 10:55:44] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:02:40 [I 2022-10-26 10:55:44] ...trap/engines/engine.py.229: process: 0.49448 | load: 0.00247 [I 2022-10-26 10:55:44] ...trap/engines/engine.py.230: loss: 0.28753 [I 2022-10-26 10:55:44] ...optimizers/trijoint.py.151: total_norm: 3.869576 [I 2022-10-26 10:55:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 1990/2304 [I 2022-10-26 10:55:49] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:02:37 [I 2022-10-26 10:55:49] ...trap/engines/engine.py.229: process: 0.50876 | load: 0.00246 [I 2022-10-26 10:55:49] ...trap/engines/engine.py.230: loss: 0.33748 [I 2022-10-26 10:55:49] ...optimizers/trijoint.py.151: total_norm: 3.936691 [I 2022-10-26 10:55:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 2000/2304 [I 2022-10-26 10:55:54] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:02:37 [I 2022-10-26 10:55:54] ...trap/engines/engine.py.229: process: 0.49459 | load: 0.00247 [I 2022-10-26 10:55:54] ...trap/engines/engine.py.230: loss: 0.34904 [I 2022-10-26 10:55:54] ...optimizers/trijoint.py.151: total_norm: 3.523772 [I 2022-10-26 10:55:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 2010/2304 [I 2022-10-26 10:55:59] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:02:26 [I 2022-10-26 10:55:59] ...trap/engines/engine.py.229: process: 0.49440 | load: 0.00246 [I 2022-10-26 10:55:59] ...trap/engines/engine.py.230: loss: 0.27113 [I 2022-10-26 10:55:59] ...optimizers/trijoint.py.151: total_norm: 3.342476 [I 2022-10-26 10:56:04] ...trap/engines/engine.py.224: train: epoch 3 | batch 2020/2304 [I 2022-10-26 10:56:04] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:02:22 [I 2022-10-26 10:56:04] ...trap/engines/engine.py.229: process: 0.49425 | load: 0.00242 [I 2022-10-26 10:56:04] ...trap/engines/engine.py.230: loss: 0.30411 [I 2022-10-26 10:56:04] ...optimizers/trijoint.py.151: total_norm: 4.652636 [I 2022-10-26 10:56:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 2030/2304 [I 2022-10-26 10:56:09] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:02:16 [I 2022-10-26 10:56:09] ...trap/engines/engine.py.229: process: 0.49407 | load: 0.00242 [I 2022-10-26 10:56:09] ...trap/engines/engine.py.230: loss: 0.26798 [I 2022-10-26 10:56:09] ...optimizers/trijoint.py.151: total_norm: 4.221336 [I 2022-10-26 10:56:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 2040/2304 [I 2022-10-26 10:56:14] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:02:10 [I 2022-10-26 10:56:14] ...trap/engines/engine.py.229: process: 0.49218 | load: 0.00245 [I 2022-10-26 10:56:14] ...trap/engines/engine.py.230: loss: 0.28697 [I 2022-10-26 10:56:14] ...optimizers/trijoint.py.151: total_norm: 4.260954 [I 2022-10-26 10:56:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 2050/2304 [I 2022-10-26 10:56:19] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:02:06 [I 2022-10-26 10:56:19] ...trap/engines/engine.py.229: process: 0.50240 | load: 0.00248 [I 2022-10-26 10:56:19] ...trap/engines/engine.py.230: loss: 0.25147 [I 2022-10-26 10:56:19] ...optimizers/trijoint.py.151: total_norm: 3.727212 [I 2022-10-26 10:56:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 2060/2304 [I 2022-10-26 10:56:24] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:02:02 [I 2022-10-26 10:56:24] ...trap/engines/engine.py.229: process: 0.50723 | load: 0.00244 [I 2022-10-26 10:56:24] ...trap/engines/engine.py.230: loss: 0.34309 [I 2022-10-26 10:56:24] ...optimizers/trijoint.py.151: total_norm: 4.309031 [I 2022-10-26 10:56:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 2070/2304 [I 2022-10-26 10:56:29] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:01:56 [I 2022-10-26 10:56:29] ...trap/engines/engine.py.229: process: 0.50382 | load: 0.00245 [I 2022-10-26 10:56:29] ...trap/engines/engine.py.230: loss: 0.22505 [I 2022-10-26 10:56:29] ...optimizers/trijoint.py.151: total_norm: 3.223204 [I 2022-10-26 10:56:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 2080/2304 [I 2022-10-26 10:56:34] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:01:51 [I 2022-10-26 10:56:34] ...trap/engines/engine.py.229: process: 0.48846 | load: 0.00246 [I 2022-10-26 10:56:34] ...trap/engines/engine.py.230: loss: 0.36462 [I 2022-10-26 10:56:34] ...optimizers/trijoint.py.151: total_norm: 4.227733 [I 2022-10-26 10:56:39] ...trap/engines/engine.py.224: train: epoch 3 | batch 2090/2304 [I 2022-10-26 10:56:39] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:01:46 [I 2022-10-26 10:56:39] ...trap/engines/engine.py.229: process: 0.50156 | load: 0.00241 [I 2022-10-26 10:56:39] ...trap/engines/engine.py.230: loss: 0.28717 [I 2022-10-26 10:56:39] ...optimizers/trijoint.py.151: total_norm: 3.737385 [I 2022-10-26 10:56:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 2100/2304 [I 2022-10-26 10:56:44] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:01:42 [I 2022-10-26 10:56:44] ...trap/engines/engine.py.229: process: 0.50129 | load: 0.00252 [I 2022-10-26 10:56:44] ...trap/engines/engine.py.230: loss: 0.28803 [I 2022-10-26 10:56:44] ...optimizers/trijoint.py.151: total_norm: 4.147923 [I 2022-10-26 10:56:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 2110/2304 [I 2022-10-26 10:56:49] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:01:36 [I 2022-10-26 10:56:49] ...trap/engines/engine.py.229: process: 0.49162 | load: 0.00247 [I 2022-10-26 10:56:49] ...trap/engines/engine.py.230: loss: 0.37123 [I 2022-10-26 10:56:49] ...optimizers/trijoint.py.151: total_norm: 4.024765 [I 2022-10-26 10:56:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 2120/2304 [I 2022-10-26 10:56:54] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:01:32 [I 2022-10-26 10:56:54] ...trap/engines/engine.py.229: process: 0.50471 | load: 0.00249 [I 2022-10-26 10:56:54] ...trap/engines/engine.py.230: loss: 0.38760 [I 2022-10-26 10:56:54] ...optimizers/trijoint.py.151: total_norm: 4.244510 [I 2022-10-26 10:56:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 2130/2304 [I 2022-10-26 10:56:59] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:01:26 [I 2022-10-26 10:56:59] ...trap/engines/engine.py.229: process: 0.48692 | load: 0.00245 [I 2022-10-26 10:56:59] ...trap/engines/engine.py.230: loss: 0.34143 [I 2022-10-26 10:56:59] ...optimizers/trijoint.py.151: total_norm: 3.175216 [I 2022-10-26 10:57:04] ...trap/engines/engine.py.224: train: epoch 3 | batch 2140/2304 [I 2022-10-26 10:57:04] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:01:21 [I 2022-10-26 10:57:04] ...trap/engines/engine.py.229: process: 0.50614 | load: 0.00250 [I 2022-10-26 10:57:04] ...trap/engines/engine.py.230: loss: 0.27192 [I 2022-10-26 10:57:04] ...optimizers/trijoint.py.151: total_norm: 3.038549 [I 2022-10-26 10:57:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 2150/2304 [I 2022-10-26 10:57:09] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:01:17 [I 2022-10-26 10:57:09] ...trap/engines/engine.py.229: process: 0.48689 | load: 0.00251 [I 2022-10-26 10:57:09] ...trap/engines/engine.py.230: loss: 0.44877 [I 2022-10-26 10:57:09] ...optimizers/trijoint.py.151: total_norm: 4.692389 [I 2022-10-26 10:57:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 2160/2304 [I 2022-10-26 10:57:14] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:01:11 [I 2022-10-26 10:57:14] ...trap/engines/engine.py.229: process: 0.49957 | load: 0.00245 [I 2022-10-26 10:57:14] ...trap/engines/engine.py.230: loss: 0.40531 [I 2022-10-26 10:57:14] ...optimizers/trijoint.py.151: total_norm: 4.877255 [I 2022-10-26 10:57:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 2170/2304 [I 2022-10-26 10:57:19] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:01:06 [I 2022-10-26 10:57:19] ...trap/engines/engine.py.229: process: 0.49744 | load: 0.00245 [I 2022-10-26 10:57:19] ...trap/engines/engine.py.230: loss: 0.28945 [I 2022-10-26 10:57:19] ...optimizers/trijoint.py.151: total_norm: 3.157675 [I 2022-10-26 10:57:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 2180/2304 [I 2022-10-26 10:57:24] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:01:01 [I 2022-10-26 10:57:24] ...trap/engines/engine.py.229: process: 0.48687 | load: 0.00246 [I 2022-10-26 10:57:24] ...trap/engines/engine.py.230: loss: 0.34739 [I 2022-10-26 10:57:24] ...optimizers/trijoint.py.151: total_norm: 5.082944 [I 2022-10-26 10:57:29] ...trap/engines/engine.py.224: train: epoch 3 | batch 2190/2304 [I 2022-10-26 10:57:29] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:00:56 [I 2022-10-26 10:57:29] ...trap/engines/engine.py.229: process: 0.48925 | load: 0.00269 [I 2022-10-26 10:57:29] ...trap/engines/engine.py.230: loss: 0.28212 [I 2022-10-26 10:57:29] ...optimizers/trijoint.py.151: total_norm: 3.806728 [I 2022-10-26 10:57:34] ...trap/engines/engine.py.224: train: epoch 3 | batch 2200/2304 [I 2022-10-26 10:57:34] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:00:51 [I 2022-10-26 10:57:34] ...trap/engines/engine.py.229: process: 0.49956 | load: 0.00244 [I 2022-10-26 10:57:34] ...trap/engines/engine.py.230: loss: 0.44512 [I 2022-10-26 10:57:34] ...optimizers/trijoint.py.151: total_norm: 4.330037 [I 2022-10-26 10:57:39] ...trap/engines/engine.py.224: train: epoch 3 | batch 2210/2304 [I 2022-10-26 10:57:39] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:00:46 [I 2022-10-26 10:57:39] ...trap/engines/engine.py.229: process: 0.50373 | load: 0.00242 [I 2022-10-26 10:57:39] ...trap/engines/engine.py.230: loss: 0.24656 [I 2022-10-26 10:57:39] ...optimizers/trijoint.py.151: total_norm: 3.537638 [I 2022-10-26 10:57:44] ...trap/engines/engine.py.224: train: epoch 3 | batch 2220/2304 [I 2022-10-26 10:57:44] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:00:42 [I 2022-10-26 10:57:44] ...trap/engines/engine.py.229: process: 0.49808 | load: 0.00244 [I 2022-10-26 10:57:44] ...trap/engines/engine.py.230: loss: 0.28404 [I 2022-10-26 10:57:44] ...optimizers/trijoint.py.151: total_norm: 3.540091 [I 2022-10-26 10:57:49] ...trap/engines/engine.py.224: train: epoch 3 | batch 2230/2304 [I 2022-10-26 10:57:49] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:00:36 [I 2022-10-26 10:57:49] ...trap/engines/engine.py.229: process: 0.50295 | load: 0.00241 [I 2022-10-26 10:57:49] ...trap/engines/engine.py.230: loss: 0.21405 [I 2022-10-26 10:57:49] ...optimizers/trijoint.py.151: total_norm: 4.223160 [I 2022-10-26 10:57:54] ...trap/engines/engine.py.224: train: epoch 3 | batch 2240/2304 [I 2022-10-26 10:57:54] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:00:31 [I 2022-10-26 10:57:54] ...trap/engines/engine.py.229: process: 0.49100 | load: 0.00244 [I 2022-10-26 10:57:54] ...trap/engines/engine.py.230: loss: 0.20221 [I 2022-10-26 10:57:54] ...optimizers/trijoint.py.151: total_norm: 2.815905 [I 2022-10-26 10:57:59] ...trap/engines/engine.py.224: train: epoch 3 | batch 2250/2304 [I 2022-10-26 10:57:59] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:00:26 [I 2022-10-26 10:57:59] ...trap/engines/engine.py.229: process: 0.48991 | load: 0.00249 [I 2022-10-26 10:57:59] ...trap/engines/engine.py.230: loss: 0.33779 [I 2022-10-26 10:57:59] ...optimizers/trijoint.py.151: total_norm: 3.317611 [I 2022-10-26 10:58:04] ...trap/engines/engine.py.224: train: epoch 3 | batch 2260/2304 [I 2022-10-26 10:58:04] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:00:21 [I 2022-10-26 10:58:04] ...trap/engines/engine.py.229: process: 0.49803 | load: 0.00244 [I 2022-10-26 10:58:04] ...trap/engines/engine.py.230: loss: 0.27258 [I 2022-10-26 10:58:04] ...optimizers/trijoint.py.151: total_norm: 3.153216 [I 2022-10-26 10:58:09] ...trap/engines/engine.py.224: train: epoch 3 | batch 2270/2304 [I 2022-10-26 10:58:09] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:00:16 [I 2022-10-26 10:58:09] ...trap/engines/engine.py.229: process: 0.49951 | load: 0.00243 [I 2022-10-26 10:58:09] ...trap/engines/engine.py.230: loss: 0.31777 [I 2022-10-26 10:58:09] ...optimizers/trijoint.py.151: total_norm: 3.865536 [I 2022-10-26 10:58:14] ...trap/engines/engine.py.224: train: epoch 3 | batch 2280/2304 [I 2022-10-26 10:58:14] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:00:11 [I 2022-10-26 10:58:14] ...trap/engines/engine.py.229: process: 0.49986 | load: 0.00243 [I 2022-10-26 10:58:14] ...trap/engines/engine.py.230: loss: 0.36434 [I 2022-10-26 10:58:14] ...optimizers/trijoint.py.151: total_norm: 4.491561 [I 2022-10-26 10:58:19] ...trap/engines/engine.py.224: train: epoch 3 | batch 2290/2304 [I 2022-10-26 10:58:19] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:00:07 [I 2022-10-26 10:58:19] ...trap/engines/engine.py.229: process: 0.49230 | load: 0.00241 [I 2022-10-26 10:58:19] ...trap/engines/engine.py.230: loss: 0.27121 [I 2022-10-26 10:58:19] ...optimizers/trijoint.py.151: total_norm: 2.992890 [I 2022-10-26 10:58:24] ...trap/engines/engine.py.224: train: epoch 3 | batch 2300/2304 [I 2022-10-26 10:58:24] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:00:02 [I 2022-10-26 10:58:24] ...trap/engines/engine.py.229: process: 0.48713 | load: 0.00014 [I 2022-10-26 10:58:24] ...trap/engines/engine.py.230: loss: 0.29827 [I 2022-10-26 10:58:24] ...optimizers/trijoint.py.151: total_norm: 3.888499 [I 2022-10-26 10:58:26] ...trap/engines/engine.py.224: train: epoch 3 | batch 2304/2304 [I 2022-10-26 10:58:26] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:00:00 [I 2022-10-26 10:58:26] ...trap/engines/engine.py.229: process: 0.48828 | load: 0.00024 [I 2022-10-26 10:58:26] ...trap/engines/engine.py.230: loss: 0.16898 [I 2022-10-26 10:58:26] ...optimizers/trijoint.py.151: total_norm: 3.148767 [S 2022-10-26 10:58:27] ...trap/engines/engine.py.129: train_epoch.epoch: 3 [S 2022-10-26 10:58:27] ...trap/engines/engine.py.129: train_epoch.loss: 0.3146571084048898 [I 2022-10-26 10:58:27] ...s/criterions/triplet.py.84: margin = 0.07 [I 2022-10-26 10:58:28] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 3 [I 2022-10-26 10:58:28] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 10:58:29] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 10:58:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 10:58:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 10:58:29] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 10:58:30] ...trap/engines/engine.py.321: eval: epoch 3 | batch 0/510 [I 2022-10-26 10:58:30] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:21:43 [I 2022-10-26 10:58:30] ...trap/engines/engine.py.326: process: 2.55585 | load: 2.17402 [I 2022-10-26 10:58:33] ...trap/engines/engine.py.321: eval: epoch 3 | batch 10/510 [I 2022-10-26 10:58:33] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:04:49 [I 2022-10-26 10:58:33] ...trap/engines/engine.py.326: process: 0.35241 | load: 0.00018 [I 2022-10-26 10:58:37] ...trap/engines/engine.py.321: eval: epoch 3 | batch 20/510 [I 2022-10-26 10:58:37] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:56 [I 2022-10-26 10:58:37] ...trap/engines/engine.py.326: process: 0.33019 | load: 0.00021 [I 2022-10-26 10:58:40] ...trap/engines/engine.py.321: eval: epoch 3 | batch 30/510 [I 2022-10-26 10:58:40] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:46 [I 2022-10-26 10:58:40] ...trap/engines/engine.py.326: process: 0.32651 | load: 0.00021 [I 2022-10-26 10:58:44] ...trap/engines/engine.py.321: eval: epoch 3 | batch 40/510 [I 2022-10-26 10:58:44] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:38 [I 2022-10-26 10:58:44] ...trap/engines/engine.py.326: process: 0.32558 | load: 0.00028 [I 2022-10-26 10:58:47] ...trap/engines/engine.py.321: eval: epoch 3 | batch 50/510 [I 2022-10-26 10:58:47] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-26 10:58:47] ...trap/engines/engine.py.326: process: 0.33752 | load: 0.00017 [I 2022-10-26 10:58:50] ...trap/engines/engine.py.321: eval: epoch 3 | batch 60/510 [I 2022-10-26 10:58:50] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:34 [I 2022-10-26 10:58:50] ...trap/engines/engine.py.326: process: 0.33297 | load: 0.00017 [I 2022-10-26 10:58:54] ...trap/engines/engine.py.321: eval: epoch 3 | batch 70/510 [I 2022-10-26 10:58:54] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:39 [I 2022-10-26 10:58:54] ...trap/engines/engine.py.326: process: 0.40763 | load: 0.00018 [I 2022-10-26 10:58:57] ...trap/engines/engine.py.321: eval: epoch 3 | batch 80/510 [I 2022-10-26 10:58:57] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:26 [I 2022-10-26 10:58:57] ...trap/engines/engine.py.326: process: 0.34011 | load: 0.00017 [I 2022-10-26 10:59:01] ...trap/engines/engine.py.321: eval: epoch 3 | batch 90/510 [I 2022-10-26 10:59:01] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:22 [I 2022-10-26 10:59:01] ...trap/engines/engine.py.326: process: 0.33078 | load: 0.00018 [I 2022-10-26 10:59:04] ...trap/engines/engine.py.321: eval: epoch 3 | batch 100/510 [I 2022-10-26 10:59:04] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:14 [I 2022-10-26 10:59:04] ...trap/engines/engine.py.326: process: 0.28614 | load: 0.00028 [I 2022-10-26 10:59:07] ...trap/engines/engine.py.321: eval: epoch 3 | batch 110/510 [I 2022-10-26 10:59:07] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 10:59:07] ...trap/engines/engine.py.326: process: 0.28231 | load: 0.00017 [I 2022-10-26 10:59:10] ...trap/engines/engine.py.321: eval: epoch 3 | batch 120/510 [I 2022-10-26 10:59:10] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-26 10:59:10] ...trap/engines/engine.py.326: process: 0.28811 | load: 0.00017 [I 2022-10-26 10:59:13] ...trap/engines/engine.py.321: eval: epoch 3 | batch 130/510 [I 2022-10-26 10:59:13] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 10:59:13] ...trap/engines/engine.py.326: process: 0.31118 | load: 0.00014 [I 2022-10-26 10:59:16] ...trap/engines/engine.py.321: eval: epoch 3 | batch 140/510 [I 2022-10-26 10:59:16] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:48 [I 2022-10-26 10:59:16] ...trap/engines/engine.py.326: process: 0.29021 | load: 0.00014 [I 2022-10-26 10:59:19] ...trap/engines/engine.py.321: eval: epoch 3 | batch 150/510 [I 2022-10-26 10:59:19] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 10:59:19] ...trap/engines/engine.py.326: process: 0.28711 | load: 0.00025 [I 2022-10-26 10:59:22] ...trap/engines/engine.py.321: eval: epoch 3 | batch 160/510 [I 2022-10-26 10:59:22] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:43 [I 2022-10-26 10:59:22] ...trap/engines/engine.py.326: process: 0.31156 | load: 0.00013 [I 2022-10-26 10:59:24] ...trap/engines/engine.py.321: eval: epoch 3 | batch 170/510 [I 2022-10-26 10:59:24] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:39 [I 2022-10-26 10:59:24] ...trap/engines/engine.py.326: process: 0.28789 | load: 0.00014 [I 2022-10-26 10:59:27] ...trap/engines/engine.py.321: eval: epoch 3 | batch 180/510 [I 2022-10-26 10:59:27] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 10:59:27] ...trap/engines/engine.py.326: process: 0.29474 | load: 0.00014 [I 2022-10-26 10:59:30] ...trap/engines/engine.py.321: eval: epoch 3 | batch 190/510 [I 2022-10-26 10:59:30] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-26 10:59:30] ...trap/engines/engine.py.326: process: 0.32957 | load: 0.00017 [I 2022-10-26 10:59:33] ...trap/engines/engine.py.321: eval: epoch 3 | batch 200/510 [I 2022-10-26 10:59:33] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:28 [I 2022-10-26 10:59:33] ...trap/engines/engine.py.326: process: 0.28163 | load: 0.00015 [I 2022-10-26 10:59:36] ...trap/engines/engine.py.321: eval: epoch 3 | batch 210/510 [I 2022-10-26 10:59:36] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 10:59:36] ...trap/engines/engine.py.326: process: 0.28320 | load: 0.00017 [I 2022-10-26 10:59:39] ...trap/engines/engine.py.321: eval: epoch 3 | batch 220/510 [I 2022-10-26 10:59:39] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 10:59:39] ...trap/engines/engine.py.326: process: 0.28591 | load: 0.00017 [I 2022-10-26 10:59:42] ...trap/engines/engine.py.321: eval: epoch 3 | batch 230/510 [I 2022-10-26 10:59:42] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 10:59:42] ...trap/engines/engine.py.326: process: 0.28745 | load: 0.00015 [I 2022-10-26 10:59:45] ...trap/engines/engine.py.321: eval: epoch 3 | batch 240/510 [I 2022-10-26 10:59:45] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 10:59:45] ...trap/engines/engine.py.326: process: 0.28760 | load: 0.00014 [I 2022-10-26 10:59:48] ...trap/engines/engine.py.321: eval: epoch 3 | batch 250/510 [I 2022-10-26 10:59:48] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-26 10:59:48] ...trap/engines/engine.py.326: process: 0.28771 | load: 0.00020 [I 2022-10-26 10:59:50] ...trap/engines/engine.py.321: eval: epoch 3 | batch 260/510 [I 2022-10-26 10:59:50] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:12 [I 2022-10-26 10:59:50] ...trap/engines/engine.py.326: process: 0.28740 | load: 0.00017 [I 2022-10-26 10:59:53] ...trap/engines/engine.py.321: eval: epoch 3 | batch 270/510 [I 2022-10-26 10:59:53] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 10:59:53] ...trap/engines/engine.py.326: process: 0.28524 | load: 0.00015 [I 2022-10-26 10:59:56] ...trap/engines/engine.py.321: eval: epoch 3 | batch 280/510 [I 2022-10-26 10:59:56] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:07 [I 2022-10-26 10:59:56] ...trap/engines/engine.py.326: process: 0.30260 | load: 0.00015 [I 2022-10-26 10:59:59] ...trap/engines/engine.py.321: eval: epoch 3 | batch 290/510 [I 2022-10-26 10:59:59] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 10:59:59] ...trap/engines/engine.py.326: process: 0.28533 | load: 0.00016 [I 2022-10-26 11:00:02] ...trap/engines/engine.py.321: eval: epoch 3 | batch 300/510 [I 2022-10-26 11:00:02] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 11:00:02] ...trap/engines/engine.py.326: process: 0.28089 | load: 0.00015 [I 2022-10-26 11:00:05] ...trap/engines/engine.py.321: eval: epoch 3 | batch 310/510 [I 2022-10-26 11:00:05] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 11:00:05] ...trap/engines/engine.py.326: process: 0.28929 | load: 0.00018 [I 2022-10-26 11:00:08] ...trap/engines/engine.py.321: eval: epoch 3 | batch 320/510 [I 2022-10-26 11:00:08] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:55 [I 2022-10-26 11:00:08] ...trap/engines/engine.py.326: process: 0.28833 | load: 0.00015 [I 2022-10-26 11:00:11] ...trap/engines/engine.py.321: eval: epoch 3 | batch 330/510 [I 2022-10-26 11:00:11] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:52 [I 2022-10-26 11:00:11] ...trap/engines/engine.py.326: process: 0.29066 | load: 0.00014 [I 2022-10-26 11:00:14] ...trap/engines/engine.py.321: eval: epoch 3 | batch 340/510 [I 2022-10-26 11:00:14] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 11:00:14] ...trap/engines/engine.py.326: process: 0.28495 | load: 0.00014 [I 2022-10-26 11:00:16] ...trap/engines/engine.py.321: eval: epoch 3 | batch 350/510 [I 2022-10-26 11:00:16] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:45 [I 2022-10-26 11:00:16] ...trap/engines/engine.py.326: process: 0.28329 | load: 0.00015 [I 2022-10-26 11:00:19] ...trap/engines/engine.py.321: eval: epoch 3 | batch 360/510 [I 2022-10-26 11:00:19] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:42 [I 2022-10-26 11:00:19] ...trap/engines/engine.py.326: process: 0.28101 | load: 0.00014 [I 2022-10-26 11:00:22] ...trap/engines/engine.py.321: eval: epoch 3 | batch 370/510 [I 2022-10-26 11:00:22] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 11:00:22] ...trap/engines/engine.py.326: process: 0.28213 | load: 0.00015 [I 2022-10-26 11:00:25] ...trap/engines/engine.py.321: eval: epoch 3 | batch 380/510 [I 2022-10-26 11:00:25] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 11:00:25] ...trap/engines/engine.py.326: process: 0.28714 | load: 0.00014 [I 2022-10-26 11:00:28] ...trap/engines/engine.py.321: eval: epoch 3 | batch 390/510 [I 2022-10-26 11:00:28] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 11:00:28] ...trap/engines/engine.py.326: process: 0.28998 | load: 0.00015 [I 2022-10-26 11:00:31] ...trap/engines/engine.py.321: eval: epoch 3 | batch 400/510 [I 2022-10-26 11:00:31] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 11:00:31] ...trap/engines/engine.py.326: process: 0.28495 | load: 0.00016 [I 2022-10-26 11:00:34] ...trap/engines/engine.py.321: eval: epoch 3 | batch 410/510 [I 2022-10-26 11:00:34] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 11:00:34] ...trap/engines/engine.py.326: process: 0.28452 | load: 0.00015 [I 2022-10-26 11:00:37] ...trap/engines/engine.py.321: eval: epoch 3 | batch 420/510 [I 2022-10-26 11:00:37] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 11:00:37] ...trap/engines/engine.py.326: process: 0.28600 | load: 0.00015 [I 2022-10-26 11:00:39] ...trap/engines/engine.py.321: eval: epoch 3 | batch 430/510 [I 2022-10-26 11:00:39] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-26 11:00:39] ...trap/engines/engine.py.326: process: 0.29202 | load: 0.00014 [I 2022-10-26 11:00:42] ...trap/engines/engine.py.321: eval: epoch 3 | batch 440/510 [I 2022-10-26 11:00:42] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 11:00:42] ...trap/engines/engine.py.326: process: 0.28986 | load: 0.00014 [I 2022-10-26 11:00:45] ...trap/engines/engine.py.321: eval: epoch 3 | batch 450/510 [I 2022-10-26 11:00:45] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 11:00:45] ...trap/engines/engine.py.326: process: 0.28696 | load: 0.00030 [I 2022-10-26 11:00:48] ...trap/engines/engine.py.321: eval: epoch 3 | batch 460/510 [I 2022-10-26 11:00:48] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 11:00:48] ...trap/engines/engine.py.326: process: 0.27994 | load: 0.00015 [I 2022-10-26 11:00:51] ...trap/engines/engine.py.321: eval: epoch 3 | batch 470/510 [I 2022-10-26 11:00:51] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 11:00:51] ...trap/engines/engine.py.326: process: 0.29132 | load: 0.00015 [I 2022-10-26 11:00:54] ...trap/engines/engine.py.321: eval: epoch 3 | batch 480/510 [I 2022-10-26 11:00:54] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 11:00:54] ...trap/engines/engine.py.326: process: 0.28437 | load: 0.00015 [I 2022-10-26 11:00:57] ...trap/engines/engine.py.321: eval: epoch 3 | batch 490/510 [I 2022-10-26 11:00:57] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 11:00:57] ...trap/engines/engine.py.326: process: 0.28278 | load: 0.00015 [I 2022-10-26 11:01:00] ...trap/engines/engine.py.321: eval: epoch 3 | batch 500/510 [I 2022-10-26 11:01:00] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 11:01:00] ...trap/engines/engine.py.326: process: 0.28451 | load: 0.00020 [I 2022-10-26 11:01:02] ...trap/engines/engine.py.321: eval: epoch 3 | batch 510/510 [I 2022-10-26 11:01:02] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 11:01:02] ...trap/engines/engine.py.326: process: 0.28939 | load: 0.00010 [S 2022-10-26 11:01:03] ...trap/engines/engine.py.132: eval_epoch.epoch: 3 [I 2022-10-26 11:01:03] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.00 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.395700 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.712500 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.818600 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.411400 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.723600 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.823900 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012760 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.018954 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010920 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.018757 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.013116 [S 2022-10-26 11:01:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.012708 [I 2022-10-26 11:01:08] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-26 11:01:08] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 11:01:08] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 11:01:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 11:01:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 11:01:08] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 11:01:11] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 11:01:16] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 11:01:16] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 11:01:16] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 11:01:20] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 11:01:25] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 11:01:25] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 11:01:25] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 11:01:28] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 11:01:33] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 11:01:33] ...trap/engines/engine.py.163: Training model on trainset for epoch 4 [I 2022-10-26 11:01:33] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 11:01:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 0/2304 [I 2022-10-26 11:01:36] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:48:32 [I 2022-10-26 11:01:36] ...trap/engines/engine.py.229: process: 2.82673 | load: 2.23388 [I 2022-10-26 11:01:36] ...trap/engines/engine.py.230: loss: 0.28885 [I 2022-10-26 11:01:36] ...optimizers/trijoint.py.151: total_norm: 3.760218 [I 2022-10-26 11:01:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 10/2304 [I 2022-10-26 11:01:41] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:48 [I 2022-10-26 11:01:41] ...trap/engines/engine.py.229: process: 0.51422 | load: 0.00464 [I 2022-10-26 11:01:41] ...trap/engines/engine.py.230: loss: 0.29172 [I 2022-10-26 11:01:41] ...optimizers/trijoint.py.151: total_norm: 3.743131 [I 2022-10-26 11:01:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 20/2304 [I 2022-10-26 11:01:46] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:10 [I 2022-10-26 11:01:46] ...trap/engines/engine.py.229: process: 0.50435 | load: 0.00358 [I 2022-10-26 11:01:46] ...trap/engines/engine.py.230: loss: 0.33160 [I 2022-10-26 11:01:46] ...optimizers/trijoint.py.151: total_norm: 4.146134 [I 2022-10-26 11:01:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 30/2304 [I 2022-10-26 11:01:51] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:06 [I 2022-10-26 11:01:51] ...trap/engines/engine.py.229: process: 0.48793 | load: 0.00301 [I 2022-10-26 11:01:51] ...trap/engines/engine.py.230: loss: 0.23934 [I 2022-10-26 11:01:51] ...optimizers/trijoint.py.151: total_norm: 2.960307 [I 2022-10-26 11:01:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 40/2304 [I 2022-10-26 11:01:56] ...trap/engines/engine.py.228: elapsed: 0:00:23 | left: 0:19:18 [I 2022-10-26 11:01:56] ...trap/engines/engine.py.229: process: 0.53989 | load: 0.00304 [I 2022-10-26 11:01:56] ...trap/engines/engine.py.230: loss: 0.29300 [I 2022-10-26 11:01:56] ...optimizers/trijoint.py.151: total_norm: 3.818268 [I 2022-10-26 11:02:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 50/2304 [I 2022-10-26 11:02:01] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:19:04 [I 2022-10-26 11:02:01] ...trap/engines/engine.py.229: process: 0.52612 | load: 0.00293 [I 2022-10-26 11:02:01] ...trap/engines/engine.py.230: loss: 0.30513 [I 2022-10-26 11:02:01] ...optimizers/trijoint.py.151: total_norm: 4.281054 [I 2022-10-26 11:02:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 60/2304 [I 2022-10-26 11:02:06] ...trap/engines/engine.py.228: elapsed: 0:00:33 | left: 0:18:59 [I 2022-10-26 11:02:06] ...trap/engines/engine.py.229: process: 0.50711 | load: 0.00342 [I 2022-10-26 11:02:06] ...trap/engines/engine.py.230: loss: 0.30065 [I 2022-10-26 11:02:06] ...optimizers/trijoint.py.151: total_norm: 3.165677 [I 2022-10-26 11:02:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 70/2304 [I 2022-10-26 11:02:11] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:18:39 [I 2022-10-26 11:02:11] ...trap/engines/engine.py.229: process: 0.50144 | load: 0.00286 [I 2022-10-26 11:02:11] ...trap/engines/engine.py.230: loss: 0.35047 [I 2022-10-26 11:02:11] ...optimizers/trijoint.py.151: total_norm: 3.541531 [I 2022-10-26 11:02:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 80/2304 [I 2022-10-26 11:02:16] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:18:38 [I 2022-10-26 11:02:16] ...trap/engines/engine.py.229: process: 0.50821 | load: 0.00299 [I 2022-10-26 11:02:16] ...trap/engines/engine.py.230: loss: 0.30114 [I 2022-10-26 11:02:16] ...optimizers/trijoint.py.151: total_norm: 3.780303 [I 2022-10-26 11:02:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 90/2304 [I 2022-10-26 11:02:21] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:18:24 [I 2022-10-26 11:02:21] ...trap/engines/engine.py.229: process: 0.49829 | load: 0.00266 [I 2022-10-26 11:02:21] ...trap/engines/engine.py.230: loss: 0.36539 [I 2022-10-26 11:02:21] ...optimizers/trijoint.py.151: total_norm: 3.911727 [I 2022-10-26 11:02:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 100/2304 [I 2022-10-26 11:02:26] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:18:25 [I 2022-10-26 11:02:26] ...trap/engines/engine.py.229: process: 0.49812 | load: 0.00288 [I 2022-10-26 11:02:26] ...trap/engines/engine.py.230: loss: 0.25840 [I 2022-10-26 11:02:26] ...optimizers/trijoint.py.151: total_norm: 3.222230 [I 2022-10-26 11:02:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 110/2304 [I 2022-10-26 11:02:31] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:18:30 [I 2022-10-26 11:02:31] ...trap/engines/engine.py.229: process: 0.50616 | load: 0.00265 [I 2022-10-26 11:02:31] ...trap/engines/engine.py.230: loss: 0.35651 [I 2022-10-26 11:02:31] ...optimizers/trijoint.py.151: total_norm: 3.069616 [I 2022-10-26 11:02:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 120/2304 [I 2022-10-26 11:02:36] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:26 [I 2022-10-26 11:02:36] ...trap/engines/engine.py.229: process: 0.49352 | load: 0.00264 [I 2022-10-26 11:02:36] ...trap/engines/engine.py.230: loss: 0.29961 [I 2022-10-26 11:02:36] ...optimizers/trijoint.py.151: total_norm: 3.326259 [I 2022-10-26 11:02:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 130/2304 [I 2022-10-26 11:02:41] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:18 [I 2022-10-26 11:02:41] ...trap/engines/engine.py.229: process: 0.51216 | load: 0.00260 [I 2022-10-26 11:02:41] ...trap/engines/engine.py.230: loss: 0.29390 [I 2022-10-26 11:02:41] ...optimizers/trijoint.py.151: total_norm: 4.015908 [I 2022-10-26 11:02:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 140/2304 [I 2022-10-26 11:02:47] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:18:22 [I 2022-10-26 11:02:47] ...trap/engines/engine.py.229: process: 0.50172 | load: 0.00281 [I 2022-10-26 11:02:47] ...trap/engines/engine.py.230: loss: 0.28050 [I 2022-10-26 11:02:47] ...optimizers/trijoint.py.151: total_norm: 3.944850 [I 2022-10-26 11:02:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 150/2304 [I 2022-10-26 11:02:52] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:18:02 [I 2022-10-26 11:02:52] ...trap/engines/engine.py.229: process: 0.50710 | load: 0.00272 [I 2022-10-26 11:02:52] ...trap/engines/engine.py.230: loss: 0.20881 [I 2022-10-26 11:02:52] ...optimizers/trijoint.py.151: total_norm: 3.408262 [I 2022-10-26 11:02:57] ...trap/engines/engine.py.224: train: epoch 4 | batch 160/2304 [I 2022-10-26 11:02:57] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:17:39 [I 2022-10-26 11:02:57] ...trap/engines/engine.py.229: process: 0.49101 | load: 0.00264 [I 2022-10-26 11:02:57] ...trap/engines/engine.py.230: loss: 0.32782 [I 2022-10-26 11:02:57] ...optimizers/trijoint.py.151: total_norm: 4.436814 [I 2022-10-26 11:03:02] ...trap/engines/engine.py.224: train: epoch 4 | batch 170/2304 [I 2022-10-26 11:03:02] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:41 [I 2022-10-26 11:03:02] ...trap/engines/engine.py.229: process: 0.50802 | load: 0.00268 [I 2022-10-26 11:03:02] ...trap/engines/engine.py.230: loss: 0.27531 [I 2022-10-26 11:03:02] ...optimizers/trijoint.py.151: total_norm: 3.226898 [I 2022-10-26 11:03:07] ...trap/engines/engine.py.224: train: epoch 4 | batch 180/2304 [I 2022-10-26 11:03:07] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:17:49 [I 2022-10-26 11:03:07] ...trap/engines/engine.py.229: process: 0.51380 | load: 0.00272 [I 2022-10-26 11:03:07] ...trap/engines/engine.py.230: loss: 0.24845 [I 2022-10-26 11:03:07] ...optimizers/trijoint.py.151: total_norm: 4.318590 [I 2022-10-26 11:03:12] ...trap/engines/engine.py.224: train: epoch 4 | batch 190/2304 [I 2022-10-26 11:03:12] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:17:33 [I 2022-10-26 11:03:12] ...trap/engines/engine.py.229: process: 0.51115 | load: 0.00246 [I 2022-10-26 11:03:12] ...trap/engines/engine.py.230: loss: 0.32428 [I 2022-10-26 11:03:12] ...optimizers/trijoint.py.151: total_norm: 4.267913 [I 2022-10-26 11:03:17] ...trap/engines/engine.py.224: train: epoch 4 | batch 200/2304 [I 2022-10-26 11:03:17] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:34 [I 2022-10-26 11:03:17] ...trap/engines/engine.py.229: process: 0.49929 | load: 0.00255 [I 2022-10-26 11:03:17] ...trap/engines/engine.py.230: loss: 0.30473 [I 2022-10-26 11:03:17] ...optimizers/trijoint.py.151: total_norm: 3.526958 [I 2022-10-26 11:03:22] ...trap/engines/engine.py.224: train: epoch 4 | batch 210/2304 [I 2022-10-26 11:03:22] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:35 [I 2022-10-26 11:03:22] ...trap/engines/engine.py.229: process: 0.50692 | load: 0.00255 [I 2022-10-26 11:03:22] ...trap/engines/engine.py.230: loss: 0.27681 [I 2022-10-26 11:03:22] ...optimizers/trijoint.py.151: total_norm: 3.092136 [I 2022-10-26 11:03:27] ...trap/engines/engine.py.224: train: epoch 4 | batch 220/2304 [I 2022-10-26 11:03:27] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:27 [I 2022-10-26 11:03:27] ...trap/engines/engine.py.229: process: 0.49970 | load: 0.00250 [I 2022-10-26 11:03:27] ...trap/engines/engine.py.230: loss: 0.24287 [I 2022-10-26 11:03:27] ...optimizers/trijoint.py.151: total_norm: 4.293203 [I 2022-10-26 11:03:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 230/2304 [I 2022-10-26 11:03:32] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:22 [I 2022-10-26 11:03:32] ...trap/engines/engine.py.229: process: 0.50587 | load: 0.00247 [I 2022-10-26 11:03:32] ...trap/engines/engine.py.230: loss: 0.27215 [I 2022-10-26 11:03:32] ...optimizers/trijoint.py.151: total_norm: 3.258912 [I 2022-10-26 11:03:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 240/2304 [I 2022-10-26 11:03:37] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:12 [I 2022-10-26 11:03:37] ...trap/engines/engine.py.229: process: 0.50440 | load: 0.00261 [I 2022-10-26 11:03:37] ...trap/engines/engine.py.230: loss: 0.18934 [I 2022-10-26 11:03:37] ...optimizers/trijoint.py.151: total_norm: 2.857330 [I 2022-10-26 11:03:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 250/2304 [I 2022-10-26 11:03:42] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:12 [I 2022-10-26 11:03:42] ...trap/engines/engine.py.229: process: 0.50545 | load: 0.00252 [I 2022-10-26 11:03:42] ...trap/engines/engine.py.230: loss: 0.28187 [I 2022-10-26 11:03:42] ...optimizers/trijoint.py.151: total_norm: 4.701517 [I 2022-10-26 11:03:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 260/2304 [I 2022-10-26 11:03:47] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:02 [I 2022-10-26 11:03:47] ...trap/engines/engine.py.229: process: 0.49910 | load: 0.00246 [I 2022-10-26 11:03:47] ...trap/engines/engine.py.230: loss: 0.36684 [I 2022-10-26 11:03:47] ...optimizers/trijoint.py.151: total_norm: 4.253075 [I 2022-10-26 11:03:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 270/2304 [I 2022-10-26 11:03:52] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:17:07 [I 2022-10-26 11:03:52] ...trap/engines/engine.py.229: process: 0.50019 | load: 0.00248 [I 2022-10-26 11:03:52] ...trap/engines/engine.py.230: loss: 0.23387 [I 2022-10-26 11:03:52] ...optimizers/trijoint.py.151: total_norm: 3.392167 [I 2022-10-26 11:03:57] ...trap/engines/engine.py.224: train: epoch 4 | batch 280/2304 [I 2022-10-26 11:03:57] ...trap/engines/engine.py.228: elapsed: 0:02:24 | left: 0:16:56 [I 2022-10-26 11:03:57] ...trap/engines/engine.py.229: process: 0.48239 | load: 0.00260 [I 2022-10-26 11:03:57] ...trap/engines/engine.py.230: loss: 0.22429 [I 2022-10-26 11:03:57] ...optimizers/trijoint.py.151: total_norm: 2.942237 [I 2022-10-26 11:04:02] ...trap/engines/engine.py.224: train: epoch 4 | batch 290/2304 [I 2022-10-26 11:04:02] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:16:55 [I 2022-10-26 11:04:02] ...trap/engines/engine.py.229: process: 0.49813 | load: 0.00252 [I 2022-10-26 11:04:02] ...trap/engines/engine.py.230: loss: 0.27408 [I 2022-10-26 11:04:02] ...optimizers/trijoint.py.151: total_norm: 3.801992 [I 2022-10-26 11:04:07] ...trap/engines/engine.py.224: train: epoch 4 | batch 300/2304 [I 2022-10-26 11:04:07] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:16:54 [I 2022-10-26 11:04:07] ...trap/engines/engine.py.229: process: 0.51507 | load: 0.00259 [I 2022-10-26 11:04:07] ...trap/engines/engine.py.230: loss: 0.35444 [I 2022-10-26 11:04:07] ...optimizers/trijoint.py.151: total_norm: 5.429807 [I 2022-10-26 11:04:12] ...trap/engines/engine.py.224: train: epoch 4 | batch 310/2304 [I 2022-10-26 11:04:12] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:16:43 [I 2022-10-26 11:04:12] ...trap/engines/engine.py.229: process: 0.50418 | load: 0.00249 [I 2022-10-26 11:04:12] ...trap/engines/engine.py.230: loss: 0.33948 [I 2022-10-26 11:04:12] ...optimizers/trijoint.py.151: total_norm: 4.557414 [I 2022-10-26 11:04:17] ...trap/engines/engine.py.224: train: epoch 4 | batch 320/2304 [I 2022-10-26 11:04:17] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:16:38 [I 2022-10-26 11:04:17] ...trap/engines/engine.py.229: process: 0.49258 | load: 0.00269 [I 2022-10-26 11:04:17] ...trap/engines/engine.py.230: loss: 0.29003 [I 2022-10-26 11:04:17] ...optimizers/trijoint.py.151: total_norm: 3.648724 [I 2022-10-26 11:04:22] ...trap/engines/engine.py.224: train: epoch 4 | batch 330/2304 [I 2022-10-26 11:04:22] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:16:37 [I 2022-10-26 11:04:22] ...trap/engines/engine.py.229: process: 0.51223 | load: 0.00253 [I 2022-10-26 11:04:22] ...trap/engines/engine.py.230: loss: 0.23133 [I 2022-10-26 11:04:22] ...optimizers/trijoint.py.151: total_norm: 2.664006 [I 2022-10-26 11:04:27] ...trap/engines/engine.py.224: train: epoch 4 | batch 340/2304 [I 2022-10-26 11:04:27] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:16:19 [I 2022-10-26 11:04:27] ...trap/engines/engine.py.229: process: 0.49615 | load: 0.00252 [I 2022-10-26 11:04:27] ...trap/engines/engine.py.230: loss: 0.26470 [I 2022-10-26 11:04:27] ...optimizers/trijoint.py.151: total_norm: 3.761774 [I 2022-10-26 11:04:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 350/2304 [I 2022-10-26 11:04:32] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:16:14 [I 2022-10-26 11:04:32] ...trap/engines/engine.py.229: process: 0.50195 | load: 0.00258 [I 2022-10-26 11:04:32] ...trap/engines/engine.py.230: loss: 0.32104 [I 2022-10-26 11:04:32] ...optimizers/trijoint.py.151: total_norm: 3.869668 [I 2022-10-26 11:04:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 360/2304 [I 2022-10-26 11:04:37] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:16:25 [I 2022-10-26 11:04:37] ...trap/engines/engine.py.229: process: 0.50800 | load: 0.00250 [I 2022-10-26 11:04:37] ...trap/engines/engine.py.230: loss: 0.30836 [I 2022-10-26 11:04:37] ...optimizers/trijoint.py.151: total_norm: 3.540564 [I 2022-10-26 11:04:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 370/2304 [I 2022-10-26 11:04:42] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:16:19 [I 2022-10-26 11:04:42] ...trap/engines/engine.py.229: process: 0.50001 | load: 0.00248 [I 2022-10-26 11:04:42] ...trap/engines/engine.py.230: loss: 0.31236 [I 2022-10-26 11:04:42] ...optimizers/trijoint.py.151: total_norm: 3.857081 [I 2022-10-26 11:04:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 380/2304 [I 2022-10-26 11:04:47] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:16:00 [I 2022-10-26 11:04:47] ...trap/engines/engine.py.229: process: 0.50602 | load: 0.00258 [I 2022-10-26 11:04:47] ...trap/engines/engine.py.230: loss: 0.34516 [I 2022-10-26 11:04:47] ...optimizers/trijoint.py.151: total_norm: 3.730016 [I 2022-10-26 11:04:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 390/2304 [I 2022-10-26 11:04:52] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:15:50 [I 2022-10-26 11:04:52] ...trap/engines/engine.py.229: process: 0.49506 | load: 0.00248 [I 2022-10-26 11:04:52] ...trap/engines/engine.py.230: loss: 0.24493 [I 2022-10-26 11:04:52] ...optimizers/trijoint.py.151: total_norm: 3.699025 [I 2022-10-26 11:04:57] ...trap/engines/engine.py.224: train: epoch 4 | batch 400/2304 [I 2022-10-26 11:04:57] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:15:57 [I 2022-10-26 11:04:57] ...trap/engines/engine.py.229: process: 0.50932 | load: 0.00246 [I 2022-10-26 11:04:57] ...trap/engines/engine.py.230: loss: 0.29229 [I 2022-10-26 11:04:57] ...optimizers/trijoint.py.151: total_norm: 3.984017 [I 2022-10-26 11:05:03] ...trap/engines/engine.py.224: train: epoch 4 | batch 410/2304 [I 2022-10-26 11:05:03] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:15:59 [I 2022-10-26 11:05:03] ...trap/engines/engine.py.229: process: 0.50358 | load: 0.00265 [I 2022-10-26 11:05:03] ...trap/engines/engine.py.230: loss: 0.25582 [I 2022-10-26 11:05:03] ...optimizers/trijoint.py.151: total_norm: 3.582630 [I 2022-10-26 11:05:08] ...trap/engines/engine.py.224: train: epoch 4 | batch 420/2304 [I 2022-10-26 11:05:08] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:15:41 [I 2022-10-26 11:05:08] ...trap/engines/engine.py.229: process: 0.51196 | load: 0.00263 [I 2022-10-26 11:05:08] ...trap/engines/engine.py.230: loss: 0.30826 [I 2022-10-26 11:05:08] ...optimizers/trijoint.py.151: total_norm: 4.276845 [I 2022-10-26 11:05:13] ...trap/engines/engine.py.224: train: epoch 4 | batch 430/2304 [I 2022-10-26 11:05:13] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:15:53 [I 2022-10-26 11:05:13] ...trap/engines/engine.py.229: process: 0.50091 | load: 0.00269 [I 2022-10-26 11:05:13] ...trap/engines/engine.py.230: loss: 0.36928 [I 2022-10-26 11:05:13] ...optimizers/trijoint.py.151: total_norm: 4.911595 [I 2022-10-26 11:05:18] ...trap/engines/engine.py.224: train: epoch 4 | batch 440/2304 [I 2022-10-26 11:05:18] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:15:21 [I 2022-10-26 11:05:18] ...trap/engines/engine.py.229: process: 0.48757 | load: 0.00262 [I 2022-10-26 11:05:18] ...trap/engines/engine.py.230: loss: 0.25538 [I 2022-10-26 11:05:18] ...optimizers/trijoint.py.151: total_norm: 3.466836 [I 2022-10-26 11:05:23] ...trap/engines/engine.py.224: train: epoch 4 | batch 450/2304 [I 2022-10-26 11:05:23] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:15:35 [I 2022-10-26 11:05:23] ...trap/engines/engine.py.229: process: 0.49709 | load: 0.00261 [I 2022-10-26 11:05:23] ...trap/engines/engine.py.230: loss: 0.24519 [I 2022-10-26 11:05:23] ...optimizers/trijoint.py.151: total_norm: 3.117728 [I 2022-10-26 11:05:28] ...trap/engines/engine.py.224: train: epoch 4 | batch 460/2304 [I 2022-10-26 11:05:28] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:15:16 [I 2022-10-26 11:05:28] ...trap/engines/engine.py.229: process: 0.50462 | load: 0.00255 [I 2022-10-26 11:05:28] ...trap/engines/engine.py.230: loss: 0.26842 [I 2022-10-26 11:05:28] ...optimizers/trijoint.py.151: total_norm: 3.865620 [I 2022-10-26 11:05:33] ...trap/engines/engine.py.224: train: epoch 4 | batch 470/2304 [I 2022-10-26 11:05:33] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:15:27 [I 2022-10-26 11:05:33] ...trap/engines/engine.py.229: process: 0.49025 | load: 0.00258 [I 2022-10-26 11:05:33] ...trap/engines/engine.py.230: loss: 0.34708 [I 2022-10-26 11:05:33] ...optimizers/trijoint.py.151: total_norm: 4.114746 [I 2022-10-26 11:05:38] ...trap/engines/engine.py.224: train: epoch 4 | batch 480/2304 [I 2022-10-26 11:05:38] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:15:15 [I 2022-10-26 11:05:38] ...trap/engines/engine.py.229: process: 0.52502 | load: 0.00273 [I 2022-10-26 11:05:38] ...trap/engines/engine.py.230: loss: 0.32428 [I 2022-10-26 11:05:38] ...optimizers/trijoint.py.151: total_norm: 4.302448 [I 2022-10-26 11:05:43] ...trap/engines/engine.py.224: train: epoch 4 | batch 490/2304 [I 2022-10-26 11:05:43] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:15:03 [I 2022-10-26 11:05:43] ...trap/engines/engine.py.229: process: 0.50165 | load: 0.00353 [I 2022-10-26 11:05:43] ...trap/engines/engine.py.230: loss: 0.24005 [I 2022-10-26 11:05:43] ...optimizers/trijoint.py.151: total_norm: 3.359154 [I 2022-10-26 11:05:48] ...trap/engines/engine.py.224: train: epoch 4 | batch 500/2304 [I 2022-10-26 11:05:48] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:15:06 [I 2022-10-26 11:05:48] ...trap/engines/engine.py.229: process: 0.50145 | load: 0.00250 [I 2022-10-26 11:05:48] ...trap/engines/engine.py.230: loss: 0.28525 [I 2022-10-26 11:05:48] ...optimizers/trijoint.py.151: total_norm: 3.525519 [I 2022-10-26 11:05:53] ...trap/engines/engine.py.224: train: epoch 4 | batch 510/2304 [I 2022-10-26 11:05:53] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:15:07 [I 2022-10-26 11:05:53] ...trap/engines/engine.py.229: process: 0.50839 | load: 0.00248 [I 2022-10-26 11:05:53] ...trap/engines/engine.py.230: loss: 0.23672 [I 2022-10-26 11:05:53] ...optimizers/trijoint.py.151: total_norm: 3.280454 [I 2022-10-26 11:05:58] ...trap/engines/engine.py.224: train: epoch 4 | batch 520/2304 [I 2022-10-26 11:05:58] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:15:08 [I 2022-10-26 11:05:58] ...trap/engines/engine.py.229: process: 0.50680 | load: 0.00246 [I 2022-10-26 11:05:58] ...trap/engines/engine.py.230: loss: 0.29562 [I 2022-10-26 11:05:58] ...optimizers/trijoint.py.151: total_norm: 3.816673 [I 2022-10-26 11:06:03] ...trap/engines/engine.py.224: train: epoch 4 | batch 530/2304 [I 2022-10-26 11:06:03] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:14:44 [I 2022-10-26 11:06:03] ...trap/engines/engine.py.229: process: 0.49223 | load: 0.00269 [I 2022-10-26 11:06:03] ...trap/engines/engine.py.230: loss: 0.35082 [I 2022-10-26 11:06:03] ...optimizers/trijoint.py.151: total_norm: 4.630788 [I 2022-10-26 11:06:08] ...trap/engines/engine.py.224: train: epoch 4 | batch 540/2304 [I 2022-10-26 11:06:08] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:14:41 [I 2022-10-26 11:06:08] ...trap/engines/engine.py.229: process: 0.49836 | load: 0.00257 [I 2022-10-26 11:06:08] ...trap/engines/engine.py.230: loss: 0.21458 [I 2022-10-26 11:06:08] ...optimizers/trijoint.py.151: total_norm: 2.893582 [I 2022-10-26 11:06:13] ...trap/engines/engine.py.224: train: epoch 4 | batch 550/2304 [I 2022-10-26 11:06:13] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:14:36 [I 2022-10-26 11:06:13] ...trap/engines/engine.py.229: process: 0.49927 | load: 0.00250 [I 2022-10-26 11:06:13] ...trap/engines/engine.py.230: loss: 0.30243 [I 2022-10-26 11:06:13] ...optimizers/trijoint.py.151: total_norm: 3.478439 [I 2022-10-26 11:06:18] ...trap/engines/engine.py.224: train: epoch 4 | batch 560/2304 [I 2022-10-26 11:06:18] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:14:34 [I 2022-10-26 11:06:18] ...trap/engines/engine.py.229: process: 0.49980 | load: 0.00267 [I 2022-10-26 11:06:18] ...trap/engines/engine.py.230: loss: 0.25088 [I 2022-10-26 11:06:18] ...optimizers/trijoint.py.151: total_norm: 3.299564 [I 2022-10-26 11:06:23] ...trap/engines/engine.py.224: train: epoch 4 | batch 570/2304 [I 2022-10-26 11:06:23] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:14:30 [I 2022-10-26 11:06:23] ...trap/engines/engine.py.229: process: 0.50000 | load: 0.00253 [I 2022-10-26 11:06:23] ...trap/engines/engine.py.230: loss: 0.34882 [I 2022-10-26 11:06:23] ...optimizers/trijoint.py.151: total_norm: 5.150045 [I 2022-10-26 11:06:28] ...trap/engines/engine.py.224: train: epoch 4 | batch 580/2304 [I 2022-10-26 11:06:28] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:14:16 [I 2022-10-26 11:06:28] ...trap/engines/engine.py.229: process: 0.49867 | load: 0.00255 [I 2022-10-26 11:06:28] ...trap/engines/engine.py.230: loss: 0.32417 [I 2022-10-26 11:06:28] ...optimizers/trijoint.py.151: total_norm: 5.930930 [I 2022-10-26 11:06:33] ...trap/engines/engine.py.224: train: epoch 4 | batch 590/2304 [I 2022-10-26 11:06:33] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:14:19 [I 2022-10-26 11:06:33] ...trap/engines/engine.py.229: process: 0.51147 | load: 0.00251 [I 2022-10-26 11:06:33] ...trap/engines/engine.py.230: loss: 0.41568 [I 2022-10-26 11:06:33] ...optimizers/trijoint.py.151: total_norm: 4.389794 [I 2022-10-26 11:06:38] ...trap/engines/engine.py.224: train: epoch 4 | batch 600/2304 [I 2022-10-26 11:06:38] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:14:15 [I 2022-10-26 11:06:38] ...trap/engines/engine.py.229: process: 0.49896 | load: 0.00242 [I 2022-10-26 11:06:38] ...trap/engines/engine.py.230: loss: 0.28268 [I 2022-10-26 11:06:38] ...optimizers/trijoint.py.151: total_norm: 3.556362 [I 2022-10-26 11:06:43] ...trap/engines/engine.py.224: train: epoch 4 | batch 610/2304 [I 2022-10-26 11:06:43] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:14:13 [I 2022-10-26 11:06:43] ...trap/engines/engine.py.229: process: 0.50970 | load: 0.00262 [I 2022-10-26 11:06:43] ...trap/engines/engine.py.230: loss: 0.35855 [I 2022-10-26 11:06:43] ...optimizers/trijoint.py.151: total_norm: 4.339309 [I 2022-10-26 11:06:48] ...trap/engines/engine.py.224: train: epoch 4 | batch 620/2304 [I 2022-10-26 11:06:48] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:13:58 [I 2022-10-26 11:06:48] ...trap/engines/engine.py.229: process: 0.49670 | load: 0.00256 [I 2022-10-26 11:06:48] ...trap/engines/engine.py.230: loss: 0.35296 [I 2022-10-26 11:06:48] ...optimizers/trijoint.py.151: total_norm: 4.635828 [I 2022-10-26 11:06:53] ...trap/engines/engine.py.224: train: epoch 4 | batch 630/2304 [I 2022-10-26 11:06:53] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:13:53 [I 2022-10-26 11:06:53] ...trap/engines/engine.py.229: process: 0.50618 | load: 0.00247 [I 2022-10-26 11:06:53] ...trap/engines/engine.py.230: loss: 0.15538 [I 2022-10-26 11:06:53] ...optimizers/trijoint.py.151: total_norm: 2.534387 [I 2022-10-26 11:06:58] ...trap/engines/engine.py.224: train: epoch 4 | batch 640/2304 [I 2022-10-26 11:06:58] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:13:52 [I 2022-10-26 11:06:58] ...trap/engines/engine.py.229: process: 0.50016 | load: 0.00242 [I 2022-10-26 11:06:58] ...trap/engines/engine.py.230: loss: 0.25564 [I 2022-10-26 11:06:58] ...optimizers/trijoint.py.151: total_norm: 3.949230 [I 2022-10-26 11:07:03] ...trap/engines/engine.py.224: train: epoch 4 | batch 650/2304 [I 2022-10-26 11:07:03] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:13:50 [I 2022-10-26 11:07:03] ...trap/engines/engine.py.229: process: 0.49643 | load: 0.00267 [I 2022-10-26 11:07:03] ...trap/engines/engine.py.230: loss: 0.23388 [I 2022-10-26 11:07:03] ...optimizers/trijoint.py.151: total_norm: 3.907440 [I 2022-10-26 11:07:08] ...trap/engines/engine.py.224: train: epoch 4 | batch 660/2304 [I 2022-10-26 11:07:08] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:13:43 [I 2022-10-26 11:07:08] ...trap/engines/engine.py.229: process: 0.51022 | load: 0.00258 [I 2022-10-26 11:07:08] ...trap/engines/engine.py.230: loss: 0.26933 [I 2022-10-26 11:07:08] ...optimizers/trijoint.py.151: total_norm: 3.449156 [I 2022-10-26 11:07:13] ...trap/engines/engine.py.224: train: epoch 4 | batch 670/2304 [I 2022-10-26 11:07:13] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:13:37 [I 2022-10-26 11:07:13] ...trap/engines/engine.py.229: process: 0.50530 | load: 0.00252 [I 2022-10-26 11:07:13] ...trap/engines/engine.py.230: loss: 0.29861 [I 2022-10-26 11:07:13] ...optimizers/trijoint.py.151: total_norm: 2.950304 [I 2022-10-26 11:07:18] ...trap/engines/engine.py.224: train: epoch 4 | batch 680/2304 [I 2022-10-26 11:07:18] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:13:37 [I 2022-10-26 11:07:18] ...trap/engines/engine.py.229: process: 0.49464 | load: 0.00272 [I 2022-10-26 11:07:18] ...trap/engines/engine.py.230: loss: 0.31745 [I 2022-10-26 11:07:18] ...optimizers/trijoint.py.151: total_norm: 3.910044 [I 2022-10-26 11:07:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 690/2304 [I 2022-10-26 11:07:46] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 1:29:52 [I 2022-10-26 11:07:46] ...trap/engines/engine.py.229: process: 0.49586 | load: 0.00249 [I 2022-10-26 11:07:46] ...trap/engines/engine.py.230: loss: 0.22755 [I 2022-10-26 11:07:46] ...optimizers/trijoint.py.151: total_norm: 3.187625 [I 2022-10-26 11:07:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 700/2304 [I 2022-10-26 11:07:51] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:21:34 [I 2022-10-26 11:07:51] ...trap/engines/engine.py.229: process: 0.49928 | load: 0.00258 [I 2022-10-26 11:07:51] ...trap/engines/engine.py.230: loss: 0.31635 [I 2022-10-26 11:07:51] ...optimizers/trijoint.py.151: total_norm: 4.009079 [I 2022-10-26 11:07:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 710/2304 [I 2022-10-26 11:07:56] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:14:02 [I 2022-10-26 11:07:56] ...trap/engines/engine.py.229: process: 0.47950 | load: 0.00247 [I 2022-10-26 11:07:56] ...trap/engines/engine.py.230: loss: 0.29921 [I 2022-10-26 11:07:56] ...optimizers/trijoint.py.151: total_norm: 3.968884 [I 2022-10-26 11:08:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 720/2304 [I 2022-10-26 11:08:01] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:13:17 [I 2022-10-26 11:08:01] ...trap/engines/engine.py.229: process: 0.50082 | load: 0.00247 [I 2022-10-26 11:08:01] ...trap/engines/engine.py.230: loss: 0.27096 [I 2022-10-26 11:08:01] ...optimizers/trijoint.py.151: total_norm: 3.616850 [I 2022-10-26 11:08:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 730/2304 [I 2022-10-26 11:08:06] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:13:06 [I 2022-10-26 11:08:06] ...trap/engines/engine.py.229: process: 0.49819 | load: 0.00243 [I 2022-10-26 11:08:06] ...trap/engines/engine.py.230: loss: 0.30441 [I 2022-10-26 11:08:06] ...optimizers/trijoint.py.151: total_norm: 5.218547 [I 2022-10-26 11:08:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 740/2304 [I 2022-10-26 11:08:11] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:13:01 [I 2022-10-26 11:08:11] ...trap/engines/engine.py.229: process: 0.50241 | load: 0.00241 [I 2022-10-26 11:08:11] ...trap/engines/engine.py.230: loss: 0.27779 [I 2022-10-26 11:08:11] ...optimizers/trijoint.py.151: total_norm: 3.298520 [I 2022-10-26 11:08:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 750/2304 [I 2022-10-26 11:08:16] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:12:57 [I 2022-10-26 11:08:16] ...trap/engines/engine.py.229: process: 0.50280 | load: 0.00241 [I 2022-10-26 11:08:16] ...trap/engines/engine.py.230: loss: 0.31746 [I 2022-10-26 11:08:16] ...optimizers/trijoint.py.151: total_norm: 4.574115 [I 2022-10-26 11:08:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 760/2304 [I 2022-10-26 11:08:21] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:12:46 [I 2022-10-26 11:08:21] ...trap/engines/engine.py.229: process: 0.49092 | load: 0.00251 [I 2022-10-26 11:08:21] ...trap/engines/engine.py.230: loss: 0.29266 [I 2022-10-26 11:08:21] ...optimizers/trijoint.py.151: total_norm: 3.515711 [I 2022-10-26 11:08:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 770/2304 [I 2022-10-26 11:08:26] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:12:45 [I 2022-10-26 11:08:26] ...trap/engines/engine.py.229: process: 0.49417 | load: 0.00245 [I 2022-10-26 11:08:26] ...trap/engines/engine.py.230: loss: 0.25946 [I 2022-10-26 11:08:26] ...optimizers/trijoint.py.151: total_norm: 3.076915 [I 2022-10-26 11:08:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 780/2304 [I 2022-10-26 11:08:31] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:12:36 [I 2022-10-26 11:08:31] ...trap/engines/engine.py.229: process: 0.48267 | load: 0.00240 [I 2022-10-26 11:08:31] ...trap/engines/engine.py.230: loss: 0.30635 [I 2022-10-26 11:08:31] ...optimizers/trijoint.py.151: total_norm: 3.596497 [I 2022-10-26 11:08:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 790/2304 [I 2022-10-26 11:08:36] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:12:34 [I 2022-10-26 11:08:36] ...trap/engines/engine.py.229: process: 0.49949 | load: 0.00247 [I 2022-10-26 11:08:36] ...trap/engines/engine.py.230: loss: 0.26271 [I 2022-10-26 11:08:36] ...optimizers/trijoint.py.151: total_norm: 4.202362 [I 2022-10-26 11:08:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 800/2304 [I 2022-10-26 11:08:41] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:12:32 [I 2022-10-26 11:08:41] ...trap/engines/engine.py.229: process: 0.50644 | load: 0.00249 [I 2022-10-26 11:08:41] ...trap/engines/engine.py.230: loss: 0.30673 [I 2022-10-26 11:08:41] ...optimizers/trijoint.py.151: total_norm: 4.055570 [I 2022-10-26 11:08:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 810/2304 [I 2022-10-26 11:08:46] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:12:48 [I 2022-10-26 11:08:46] ...trap/engines/engine.py.229: process: 0.49420 | load: 0.00264 [I 2022-10-26 11:08:46] ...trap/engines/engine.py.230: loss: 0.24956 [I 2022-10-26 11:08:46] ...optimizers/trijoint.py.151: total_norm: 3.495850 [I 2022-10-26 11:08:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 820/2304 [I 2022-10-26 11:08:51] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:12:25 [I 2022-10-26 11:08:51] ...trap/engines/engine.py.229: process: 0.49288 | load: 0.00251 [I 2022-10-26 11:08:51] ...trap/engines/engine.py.230: loss: 0.29577 [I 2022-10-26 11:08:51] ...optimizers/trijoint.py.151: total_norm: 3.901305 [I 2022-10-26 11:08:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 830/2304 [I 2022-10-26 11:08:56] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:12:09 [I 2022-10-26 11:08:56] ...trap/engines/engine.py.229: process: 0.48924 | load: 0.00273 [I 2022-10-26 11:08:56] ...trap/engines/engine.py.230: loss: 0.39715 [I 2022-10-26 11:08:56] ...optimizers/trijoint.py.151: total_norm: 5.730721 [I 2022-10-26 11:09:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 840/2304 [I 2022-10-26 11:09:01] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:12:13 [I 2022-10-26 11:09:01] ...trap/engines/engine.py.229: process: 0.51859 | load: 0.00245 [I 2022-10-26 11:09:01] ...trap/engines/engine.py.230: loss: 0.34395 [I 2022-10-26 11:09:01] ...optimizers/trijoint.py.151: total_norm: 4.167847 [I 2022-10-26 11:09:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 850/2304 [I 2022-10-26 11:09:06] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:12:05 [I 2022-10-26 11:09:06] ...trap/engines/engine.py.229: process: 0.50281 | load: 0.00248 [I 2022-10-26 11:09:06] ...trap/engines/engine.py.230: loss: 0.25126 [I 2022-10-26 11:09:06] ...optimizers/trijoint.py.151: total_norm: 3.755567 [I 2022-10-26 11:09:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 860/2304 [I 2022-10-26 11:09:11] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:12:12 [I 2022-10-26 11:09:11] ...trap/engines/engine.py.229: process: 0.51740 | load: 0.00243 [I 2022-10-26 11:09:11] ...trap/engines/engine.py.230: loss: 0.31917 [I 2022-10-26 11:09:11] ...optimizers/trijoint.py.151: total_norm: 3.674547 [I 2022-10-26 11:09:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 870/2304 [I 2022-10-26 11:09:16] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:11:58 [I 2022-10-26 11:09:16] ...trap/engines/engine.py.229: process: 0.50630 | load: 0.00243 [I 2022-10-26 11:09:16] ...trap/engines/engine.py.230: loss: 0.44463 [I 2022-10-26 11:09:16] ...optimizers/trijoint.py.151: total_norm: 4.763385 [I 2022-10-26 11:09:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 880/2304 [I 2022-10-26 11:09:21] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:11:45 [I 2022-10-26 11:09:21] ...trap/engines/engine.py.229: process: 0.49107 | load: 0.00242 [I 2022-10-26 11:09:21] ...trap/engines/engine.py.230: loss: 0.21852 [I 2022-10-26 11:09:21] ...optimizers/trijoint.py.151: total_norm: 4.868185 [I 2022-10-26 11:09:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 890/2304 [I 2022-10-26 11:09:26] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:11:43 [I 2022-10-26 11:09:26] ...trap/engines/engine.py.229: process: 0.49529 | load: 0.00256 [I 2022-10-26 11:09:26] ...trap/engines/engine.py.230: loss: 0.34404 [I 2022-10-26 11:09:26] ...optimizers/trijoint.py.151: total_norm: 4.824641 [I 2022-10-26 11:09:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 900/2304 [I 2022-10-26 11:09:31] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:11:43 [I 2022-10-26 11:09:31] ...trap/engines/engine.py.229: process: 0.50471 | load: 0.00250 [I 2022-10-26 11:09:31] ...trap/engines/engine.py.230: loss: 0.27660 [I 2022-10-26 11:09:31] ...optimizers/trijoint.py.151: total_norm: 3.137760 [I 2022-10-26 11:09:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 910/2304 [I 2022-10-26 11:09:36] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:11:42 [I 2022-10-26 11:09:36] ...trap/engines/engine.py.229: process: 0.50778 | load: 0.00246 [I 2022-10-26 11:09:36] ...trap/engines/engine.py.230: loss: 0.40883 [I 2022-10-26 11:09:36] ...optimizers/trijoint.py.151: total_norm: 4.246687 [I 2022-10-26 11:09:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 920/2304 [I 2022-10-26 11:09:41] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:11:26 [I 2022-10-26 11:09:41] ...trap/engines/engine.py.229: process: 0.48344 | load: 0.00249 [I 2022-10-26 11:09:41] ...trap/engines/engine.py.230: loss: 0.22793 [I 2022-10-26 11:09:41] ...optimizers/trijoint.py.151: total_norm: 3.224732 [I 2022-10-26 11:09:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 930/2304 [I 2022-10-26 11:09:46] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:11:30 [I 2022-10-26 11:09:46] ...trap/engines/engine.py.229: process: 0.49955 | load: 0.00243 [I 2022-10-26 11:09:46] ...trap/engines/engine.py.230: loss: 0.25524 [I 2022-10-26 11:09:46] ...optimizers/trijoint.py.151: total_norm: 3.766887 [I 2022-10-26 11:09:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 940/2304 [I 2022-10-26 11:09:51] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:11:33 [I 2022-10-26 11:09:51] ...trap/engines/engine.py.229: process: 0.51731 | load: 0.00248 [I 2022-10-26 11:09:51] ...trap/engines/engine.py.230: loss: 0.33603 [I 2022-10-26 11:09:51] ...optimizers/trijoint.py.151: total_norm: 3.706997 [I 2022-10-26 11:09:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 950/2304 [I 2022-10-26 11:09:56] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:11:16 [I 2022-10-26 11:09:56] ...trap/engines/engine.py.229: process: 0.50501 | load: 0.00244 [I 2022-10-26 11:09:56] ...trap/engines/engine.py.230: loss: 0.23636 [I 2022-10-26 11:09:56] ...optimizers/trijoint.py.151: total_norm: 3.982139 [I 2022-10-26 11:10:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 960/2304 [I 2022-10-26 11:10:01] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:11:05 [I 2022-10-26 11:10:01] ...trap/engines/engine.py.229: process: 0.49742 | load: 0.00248 [I 2022-10-26 11:10:01] ...trap/engines/engine.py.230: loss: 0.24506 [I 2022-10-26 11:10:01] ...optimizers/trijoint.py.151: total_norm: 3.536537 [I 2022-10-26 11:10:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 970/2304 [I 2022-10-26 11:10:06] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:11:03 [I 2022-10-26 11:10:06] ...trap/engines/engine.py.229: process: 0.50012 | load: 0.00244 [I 2022-10-26 11:10:06] ...trap/engines/engine.py.230: loss: 0.20044 [I 2022-10-26 11:10:06] ...optimizers/trijoint.py.151: total_norm: 2.999313 [I 2022-10-26 11:10:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 980/2304 [I 2022-10-26 11:10:11] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:10:55 [I 2022-10-26 11:10:11] ...trap/engines/engine.py.229: process: 0.48433 | load: 0.00265 [I 2022-10-26 11:10:11] ...trap/engines/engine.py.230: loss: 0.35998 [I 2022-10-26 11:10:11] ...optimizers/trijoint.py.151: total_norm: 3.662685 [I 2022-10-26 11:10:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 990/2304 [I 2022-10-26 11:10:16] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:10:54 [I 2022-10-26 11:10:16] ...trap/engines/engine.py.229: process: 0.50170 | load: 0.00276 [I 2022-10-26 11:10:16] ...trap/engines/engine.py.230: loss: 0.33292 [I 2022-10-26 11:10:16] ...optimizers/trijoint.py.151: total_norm: 3.742424 [I 2022-10-26 11:10:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 1000/2304 [I 2022-10-26 11:10:21] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:10:45 [I 2022-10-26 11:10:21] ...trap/engines/engine.py.229: process: 0.49659 | load: 0.00253 [I 2022-10-26 11:10:21] ...trap/engines/engine.py.230: loss: 0.20415 [I 2022-10-26 11:10:21] ...optimizers/trijoint.py.151: total_norm: 3.320782 [I 2022-10-26 11:10:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 1010/2304 [I 2022-10-26 11:10:26] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:10:56 [I 2022-10-26 11:10:26] ...trap/engines/engine.py.229: process: 0.49561 | load: 0.00252 [I 2022-10-26 11:10:26] ...trap/engines/engine.py.230: loss: 0.25553 [I 2022-10-26 11:10:26] ...optimizers/trijoint.py.151: total_norm: 2.741456 [I 2022-10-26 11:10:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 1020/2304 [I 2022-10-26 11:10:31] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:10:55 [I 2022-10-26 11:10:31] ...trap/engines/engine.py.229: process: 0.51199 | load: 0.00250 [I 2022-10-26 11:10:31] ...trap/engines/engine.py.230: loss: 0.36489 [I 2022-10-26 11:10:31] ...optimizers/trijoint.py.151: total_norm: 4.300326 [I 2022-10-26 11:10:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 1030/2304 [I 2022-10-26 11:10:36] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:10:45 [I 2022-10-26 11:10:36] ...trap/engines/engine.py.229: process: 0.49276 | load: 0.00270 [I 2022-10-26 11:10:36] ...trap/engines/engine.py.230: loss: 0.28922 [I 2022-10-26 11:10:36] ...optimizers/trijoint.py.151: total_norm: 4.890089 [I 2022-10-26 11:10:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 1040/2304 [I 2022-10-26 11:10:41] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:10:28 [I 2022-10-26 11:10:41] ...trap/engines/engine.py.229: process: 0.48956 | load: 0.00254 [I 2022-10-26 11:10:41] ...trap/engines/engine.py.230: loss: 0.32919 [I 2022-10-26 11:10:41] ...optimizers/trijoint.py.151: total_norm: 3.679198 [I 2022-10-26 11:10:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 1050/2304 [I 2022-10-26 11:10:46] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:10:30 [I 2022-10-26 11:10:46] ...trap/engines/engine.py.229: process: 0.48628 | load: 0.00248 [I 2022-10-26 11:10:46] ...trap/engines/engine.py.230: loss: 0.37570 [I 2022-10-26 11:10:46] ...optimizers/trijoint.py.151: total_norm: 4.473973 [I 2022-10-26 11:10:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 1060/2304 [I 2022-10-26 11:10:51] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:10:18 [I 2022-10-26 11:10:51] ...trap/engines/engine.py.229: process: 0.49521 | load: 0.00243 [I 2022-10-26 11:10:51] ...trap/engines/engine.py.230: loss: 0.31965 [I 2022-10-26 11:10:51] ...optimizers/trijoint.py.151: total_norm: 3.430471 [I 2022-10-26 11:10:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 1070/2304 [I 2022-10-26 11:10:56] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:10:15 [I 2022-10-26 11:10:56] ...trap/engines/engine.py.229: process: 0.49743 | load: 0.00249 [I 2022-10-26 11:10:56] ...trap/engines/engine.py.230: loss: 0.45742 [I 2022-10-26 11:10:56] ...optimizers/trijoint.py.151: total_norm: 5.391337 [I 2022-10-26 11:11:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 1080/2304 [I 2022-10-26 11:11:01] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:10:11 [I 2022-10-26 11:11:01] ...trap/engines/engine.py.229: process: 0.50358 | load: 0.00255 [I 2022-10-26 11:11:01] ...trap/engines/engine.py.230: loss: 0.32262 [I 2022-10-26 11:11:01] ...optimizers/trijoint.py.151: total_norm: 3.666235 [I 2022-10-26 11:11:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 1090/2304 [I 2022-10-26 11:11:06] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:10:07 [I 2022-10-26 11:11:06] ...trap/engines/engine.py.229: process: 0.49798 | load: 0.00251 [I 2022-10-26 11:11:06] ...trap/engines/engine.py.230: loss: 0.42108 [I 2022-10-26 11:11:06] ...optimizers/trijoint.py.151: total_norm: 3.885788 [I 2022-10-26 11:11:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 1100/2304 [I 2022-10-26 11:11:11] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:10:05 [I 2022-10-26 11:11:11] ...trap/engines/engine.py.229: process: 0.49980 | load: 0.00251 [I 2022-10-26 11:11:11] ...trap/engines/engine.py.230: loss: 0.24049 [I 2022-10-26 11:11:11] ...optimizers/trijoint.py.151: total_norm: 3.609342 [I 2022-10-26 11:11:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 1110/2304 [I 2022-10-26 11:11:16] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:09:56 [I 2022-10-26 11:11:16] ...trap/engines/engine.py.229: process: 0.49941 | load: 0.00252 [I 2022-10-26 11:11:16] ...trap/engines/engine.py.230: loss: 0.22458 [I 2022-10-26 11:11:16] ...optimizers/trijoint.py.151: total_norm: 2.288220 [I 2022-10-26 11:11:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 1120/2304 [I 2022-10-26 11:11:21] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:09:54 [I 2022-10-26 11:11:21] ...trap/engines/engine.py.229: process: 0.51277 | load: 0.00246 [I 2022-10-26 11:11:21] ...trap/engines/engine.py.230: loss: 0.29544 [I 2022-10-26 11:11:21] ...optimizers/trijoint.py.151: total_norm: 3.073758 [I 2022-10-26 11:11:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 1130/2304 [I 2022-10-26 11:11:26] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:09:48 [I 2022-10-26 11:11:26] ...trap/engines/engine.py.229: process: 0.50162 | load: 0.00252 [I 2022-10-26 11:11:26] ...trap/engines/engine.py.230: loss: 0.34669 [I 2022-10-26 11:11:26] ...optimizers/trijoint.py.151: total_norm: 4.101739 [I 2022-10-26 11:11:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 1140/2304 [I 2022-10-26 11:11:31] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:09:44 [I 2022-10-26 11:11:31] ...trap/engines/engine.py.229: process: 0.50229 | load: 0.00246 [I 2022-10-26 11:11:31] ...trap/engines/engine.py.230: loss: 0.28396 [I 2022-10-26 11:11:31] ...optimizers/trijoint.py.151: total_norm: 4.473515 [I 2022-10-26 11:11:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 1150/2304 [I 2022-10-26 11:11:36] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:09:38 [I 2022-10-26 11:11:36] ...trap/engines/engine.py.229: process: 0.50281 | load: 0.00251 [I 2022-10-26 11:11:36] ...trap/engines/engine.py.230: loss: 0.36428 [I 2022-10-26 11:11:36] ...optimizers/trijoint.py.151: total_norm: 4.477778 [I 2022-10-26 11:11:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 1160/2304 [I 2022-10-26 11:11:41] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:09:31 [I 2022-10-26 11:11:41] ...trap/engines/engine.py.229: process: 0.50191 | load: 0.00245 [I 2022-10-26 11:11:41] ...trap/engines/engine.py.230: loss: 0.31671 [I 2022-10-26 11:11:41] ...optimizers/trijoint.py.151: total_norm: 4.181246 [I 2022-10-26 11:11:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 1170/2304 [I 2022-10-26 11:11:46] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:09:25 [I 2022-10-26 11:11:46] ...trap/engines/engine.py.229: process: 0.50202 | load: 0.00300 [I 2022-10-26 11:11:46] ...trap/engines/engine.py.230: loss: 0.22705 [I 2022-10-26 11:11:46] ...optimizers/trijoint.py.151: total_norm: 3.857971 [I 2022-10-26 11:11:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 1180/2304 [I 2022-10-26 11:11:51] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:09:15 [I 2022-10-26 11:11:51] ...trap/engines/engine.py.229: process: 0.48375 | load: 0.00274 [I 2022-10-26 11:11:51] ...trap/engines/engine.py.230: loss: 0.37695 [I 2022-10-26 11:11:51] ...optimizers/trijoint.py.151: total_norm: 4.341044 [I 2022-10-26 11:11:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 1190/2304 [I 2022-10-26 11:11:56] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:09:19 [I 2022-10-26 11:11:56] ...trap/engines/engine.py.229: process: 0.50745 | load: 0.00252 [I 2022-10-26 11:11:56] ...trap/engines/engine.py.230: loss: 0.33350 [I 2022-10-26 11:11:56] ...optimizers/trijoint.py.151: total_norm: 3.782372 [I 2022-10-26 11:12:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 1200/2304 [I 2022-10-26 11:12:01] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:09:14 [I 2022-10-26 11:12:02] ...trap/engines/engine.py.229: process: 0.50037 | load: 0.00248 [I 2022-10-26 11:12:02] ...trap/engines/engine.py.230: loss: 0.26552 [I 2022-10-26 11:12:02] ...optimizers/trijoint.py.151: total_norm: 3.428734 [I 2022-10-26 11:12:07] ...trap/engines/engine.py.224: train: epoch 4 | batch 1210/2304 [I 2022-10-26 11:12:07] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:09:11 [I 2022-10-26 11:12:07] ...trap/engines/engine.py.229: process: 0.49573 | load: 0.00246 [I 2022-10-26 11:12:07] ...trap/engines/engine.py.230: loss: 0.24035 [I 2022-10-26 11:12:07] ...optimizers/trijoint.py.151: total_norm: 2.910754 [I 2022-10-26 11:12:12] ...trap/engines/engine.py.224: train: epoch 4 | batch 1220/2304 [I 2022-10-26 11:12:12] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:08:58 [I 2022-10-26 11:12:12] ...trap/engines/engine.py.229: process: 0.49767 | load: 0.00249 [I 2022-10-26 11:12:12] ...trap/engines/engine.py.230: loss: 0.32063 [I 2022-10-26 11:12:12] ...optimizers/trijoint.py.151: total_norm: 4.418893 [I 2022-10-26 11:12:17] ...trap/engines/engine.py.224: train: epoch 4 | batch 1230/2304 [I 2022-10-26 11:12:17] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:08:59 [I 2022-10-26 11:12:17] ...trap/engines/engine.py.229: process: 0.50565 | load: 0.00248 [I 2022-10-26 11:12:17] ...trap/engines/engine.py.230: loss: 0.24984 [I 2022-10-26 11:12:17] ...optimizers/trijoint.py.151: total_norm: 3.528030 [I 2022-10-26 11:12:22] ...trap/engines/engine.py.224: train: epoch 4 | batch 1240/2304 [I 2022-10-26 11:12:22] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:08:47 [I 2022-10-26 11:12:22] ...trap/engines/engine.py.229: process: 0.50448 | load: 0.00241 [I 2022-10-26 11:12:22] ...trap/engines/engine.py.230: loss: 0.27483 [I 2022-10-26 11:12:22] ...optimizers/trijoint.py.151: total_norm: 3.509731 [I 2022-10-26 11:12:27] ...trap/engines/engine.py.224: train: epoch 4 | batch 1250/2304 [I 2022-10-26 11:12:27] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:08:45 [I 2022-10-26 11:12:27] ...trap/engines/engine.py.229: process: 0.49742 | load: 0.00243 [I 2022-10-26 11:12:27] ...trap/engines/engine.py.230: loss: 0.31598 [I 2022-10-26 11:12:27] ...optimizers/trijoint.py.151: total_norm: 3.293041 [I 2022-10-26 11:12:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 1260/2304 [I 2022-10-26 11:12:32] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:08:44 [I 2022-10-26 11:12:32] ...trap/engines/engine.py.229: process: 0.50732 | load: 0.00252 [I 2022-10-26 11:12:32] ...trap/engines/engine.py.230: loss: 0.26094 [I 2022-10-26 11:12:32] ...optimizers/trijoint.py.151: total_norm: 3.368524 [I 2022-10-26 11:12:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 1270/2304 [I 2022-10-26 11:12:37] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:08:32 [I 2022-10-26 11:12:37] ...trap/engines/engine.py.229: process: 0.49403 | load: 0.00264 [I 2022-10-26 11:12:37] ...trap/engines/engine.py.230: loss: 0.32409 [I 2022-10-26 11:12:37] ...optimizers/trijoint.py.151: total_norm: 3.461354 [I 2022-10-26 11:12:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 1280/2304 [I 2022-10-26 11:12:41] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:08:29 [I 2022-10-26 11:12:41] ...trap/engines/engine.py.229: process: 0.50755 | load: 0.00246 [I 2022-10-26 11:12:41] ...trap/engines/engine.py.230: loss: 0.31655 [I 2022-10-26 11:12:41] ...optimizers/trijoint.py.151: total_norm: 3.342995 [I 2022-10-26 11:12:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 1290/2304 [I 2022-10-26 11:12:46] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:08:26 [I 2022-10-26 11:12:46] ...trap/engines/engine.py.229: process: 0.50476 | load: 0.00249 [I 2022-10-26 11:12:46] ...trap/engines/engine.py.230: loss: 0.29501 [I 2022-10-26 11:12:46] ...optimizers/trijoint.py.151: total_norm: 3.998553 [I 2022-10-26 11:12:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 1300/2304 [I 2022-10-26 11:12:52] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:08:23 [I 2022-10-26 11:12:52] ...trap/engines/engine.py.229: process: 0.50067 | load: 0.00242 [I 2022-10-26 11:12:52] ...trap/engines/engine.py.230: loss: 0.44411 [I 2022-10-26 11:12:52] ...optimizers/trijoint.py.151: total_norm: 5.716559 [I 2022-10-26 11:12:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 1310/2304 [I 2022-10-26 11:12:56] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:08:15 [I 2022-10-26 11:12:56] ...trap/engines/engine.py.229: process: 0.51046 | load: 0.00246 [I 2022-10-26 11:12:56] ...trap/engines/engine.py.230: loss: 0.34095 [I 2022-10-26 11:12:56] ...optimizers/trijoint.py.151: total_norm: 3.985017 [I 2022-10-26 11:13:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 1320/2304 [I 2022-10-26 11:13:01] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:08:06 [I 2022-10-26 11:13:01] ...trap/engines/engine.py.229: process: 0.49727 | load: 0.00245 [I 2022-10-26 11:13:01] ...trap/engines/engine.py.230: loss: 0.35135 [I 2022-10-26 11:13:01] ...optimizers/trijoint.py.151: total_norm: 3.800006 [I 2022-10-26 11:13:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 1330/2304 [I 2022-10-26 11:13:06] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:08:05 [I 2022-10-26 11:13:06] ...trap/engines/engine.py.229: process: 0.49319 | load: 0.00241 [I 2022-10-26 11:13:06] ...trap/engines/engine.py.230: loss: 0.24662 [I 2022-10-26 11:13:06] ...optimizers/trijoint.py.151: total_norm: 3.147089 [I 2022-10-26 11:13:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 1340/2304 [I 2022-10-26 11:13:11] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:07:55 [I 2022-10-26 11:13:11] ...trap/engines/engine.py.229: process: 0.49279 | load: 0.00251 [I 2022-10-26 11:13:11] ...trap/engines/engine.py.230: loss: 0.24438 [I 2022-10-26 11:13:11] ...optimizers/trijoint.py.151: total_norm: 3.482700 [I 2022-10-26 11:13:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 1350/2304 [I 2022-10-26 11:13:16] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:07:55 [I 2022-10-26 11:13:16] ...trap/engines/engine.py.229: process: 0.50348 | load: 0.00254 [I 2022-10-26 11:13:16] ...trap/engines/engine.py.230: loss: 0.27473 [I 2022-10-26 11:13:16] ...optimizers/trijoint.py.151: total_norm: 3.657472 [I 2022-10-26 11:13:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 1360/2304 [I 2022-10-26 11:13:21] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:07:49 [I 2022-10-26 11:13:21] ...trap/engines/engine.py.229: process: 0.49284 | load: 0.00246 [I 2022-10-26 11:13:21] ...trap/engines/engine.py.230: loss: 0.23086 [I 2022-10-26 11:13:21] ...optimizers/trijoint.py.151: total_norm: 3.333765 [I 2022-10-26 11:13:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 1370/2304 [I 2022-10-26 11:13:26] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:07:47 [I 2022-10-26 11:13:26] ...trap/engines/engine.py.229: process: 0.50022 | load: 0.00248 [I 2022-10-26 11:13:26] ...trap/engines/engine.py.230: loss: 0.36073 [I 2022-10-26 11:13:26] ...optimizers/trijoint.py.151: total_norm: 4.192193 [I 2022-10-26 11:13:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 1380/2304 [I 2022-10-26 11:13:31] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:07:35 [I 2022-10-26 11:13:31] ...trap/engines/engine.py.229: process: 0.48641 | load: 0.00245 [I 2022-10-26 11:13:31] ...trap/engines/engine.py.230: loss: 0.28795 [I 2022-10-26 11:13:31] ...optimizers/trijoint.py.151: total_norm: 4.028944 [I 2022-10-26 11:13:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 1390/2304 [I 2022-10-26 11:13:36] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:07:37 [I 2022-10-26 11:13:36] ...trap/engines/engine.py.229: process: 0.50402 | load: 0.00243 [I 2022-10-26 11:13:36] ...trap/engines/engine.py.230: loss: 0.25180 [I 2022-10-26 11:13:36] ...optimizers/trijoint.py.151: total_norm: 3.888797 [I 2022-10-26 11:13:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 1400/2304 [I 2022-10-26 11:13:41] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:07:31 [I 2022-10-26 11:13:41] ...trap/engines/engine.py.229: process: 0.49662 | load: 0.00243 [I 2022-10-26 11:13:41] ...trap/engines/engine.py.230: loss: 0.26833 [I 2022-10-26 11:13:41] ...optimizers/trijoint.py.151: total_norm: 3.242588 [I 2022-10-26 11:13:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 1410/2304 [I 2022-10-26 11:13:46] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:07:23 [I 2022-10-26 11:13:46] ...trap/engines/engine.py.229: process: 0.49773 | load: 0.00245 [I 2022-10-26 11:13:46] ...trap/engines/engine.py.230: loss: 0.30391 [I 2022-10-26 11:13:46] ...optimizers/trijoint.py.151: total_norm: 3.609895 [I 2022-10-26 11:13:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 1420/2304 [I 2022-10-26 11:13:51] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:07:17 [I 2022-10-26 11:13:51] ...trap/engines/engine.py.229: process: 0.48628 | load: 0.00242 [I 2022-10-26 11:13:51] ...trap/engines/engine.py.230: loss: 0.26023 [I 2022-10-26 11:13:51] ...optimizers/trijoint.py.151: total_norm: 3.431021 [I 2022-10-26 11:13:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 1430/2304 [I 2022-10-26 11:13:56] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:07:13 [I 2022-10-26 11:13:56] ...trap/engines/engine.py.229: process: 0.50026 | load: 0.00244 [I 2022-10-26 11:13:56] ...trap/engines/engine.py.230: loss: 0.31143 [I 2022-10-26 11:13:56] ...optimizers/trijoint.py.151: total_norm: 4.023187 [I 2022-10-26 11:14:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 1440/2304 [I 2022-10-26 11:14:01] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:07:10 [I 2022-10-26 11:14:01] ...trap/engines/engine.py.229: process: 0.50664 | load: 0.00240 [I 2022-10-26 11:14:01] ...trap/engines/engine.py.230: loss: 0.31860 [I 2022-10-26 11:14:01] ...optimizers/trijoint.py.151: total_norm: 4.311879 [I 2022-10-26 11:14:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 1450/2304 [I 2022-10-26 11:14:06] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:07:02 [I 2022-10-26 11:14:06] ...trap/engines/engine.py.229: process: 0.49633 | load: 0.00251 [I 2022-10-26 11:14:06] ...trap/engines/engine.py.230: loss: 0.24520 [I 2022-10-26 11:14:06] ...optimizers/trijoint.py.151: total_norm: 3.443894 [I 2022-10-26 11:14:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 1460/2304 [I 2022-10-26 11:14:11] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:06:59 [I 2022-10-26 11:14:11] ...trap/engines/engine.py.229: process: 0.49443 | load: 0.00246 [I 2022-10-26 11:14:11] ...trap/engines/engine.py.230: loss: 0.27179 [I 2022-10-26 11:14:11] ...optimizers/trijoint.py.151: total_norm: 4.351238 [I 2022-10-26 11:14:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 1470/2304 [I 2022-10-26 11:14:16] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:06:57 [I 2022-10-26 11:14:16] ...trap/engines/engine.py.229: process: 0.50463 | load: 0.00248 [I 2022-10-26 11:14:16] ...trap/engines/engine.py.230: loss: 0.31966 [I 2022-10-26 11:14:16] ...optimizers/trijoint.py.151: total_norm: 4.328204 [I 2022-10-26 11:14:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 1480/2304 [I 2022-10-26 11:14:21] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:06:52 [I 2022-10-26 11:14:21] ...trap/engines/engine.py.229: process: 0.50077 | load: 0.00242 [I 2022-10-26 11:14:21] ...trap/engines/engine.py.230: loss: 0.24098 [I 2022-10-26 11:14:21] ...optimizers/trijoint.py.151: total_norm: 3.414154 [I 2022-10-26 11:14:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 1490/2304 [I 2022-10-26 11:14:26] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:06:44 [I 2022-10-26 11:14:26] ...trap/engines/engine.py.229: process: 0.48835 | load: 0.00242 [I 2022-10-26 11:14:26] ...trap/engines/engine.py.230: loss: 0.35324 [I 2022-10-26 11:14:26] ...optimizers/trijoint.py.151: total_norm: 3.937383 [I 2022-10-26 11:14:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 1500/2304 [I 2022-10-26 11:14:31] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:06:41 [I 2022-10-26 11:14:31] ...trap/engines/engine.py.229: process: 0.49513 | load: 0.00270 [I 2022-10-26 11:14:31] ...trap/engines/engine.py.230: loss: 0.35438 [I 2022-10-26 11:14:31] ...optimizers/trijoint.py.151: total_norm: 4.192370 [I 2022-10-26 11:14:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 1510/2304 [I 2022-10-26 11:14:36] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:06:34 [I 2022-10-26 11:14:36] ...trap/engines/engine.py.229: process: 0.48198 | load: 0.00246 [I 2022-10-26 11:14:36] ...trap/engines/engine.py.230: loss: 0.36961 [I 2022-10-26 11:14:36] ...optimizers/trijoint.py.151: total_norm: 4.009475 [I 2022-10-26 11:14:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 1520/2304 [I 2022-10-26 11:14:41] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:06:26 [I 2022-10-26 11:14:41] ...trap/engines/engine.py.229: process: 0.48793 | load: 0.00246 [I 2022-10-26 11:14:41] ...trap/engines/engine.py.230: loss: 0.28573 [I 2022-10-26 11:14:41] ...optimizers/trijoint.py.151: total_norm: 4.174740 [I 2022-10-26 11:14:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 1530/2304 [I 2022-10-26 11:14:46] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:06:20 [I 2022-10-26 11:14:46] ...trap/engines/engine.py.229: process: 0.49735 | load: 0.00243 [I 2022-10-26 11:14:46] ...trap/engines/engine.py.230: loss: 0.30364 [I 2022-10-26 11:14:46] ...optimizers/trijoint.py.151: total_norm: 3.462885 [I 2022-10-26 11:14:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 1540/2304 [I 2022-10-26 11:14:51] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:06:19 [I 2022-10-26 11:14:51] ...trap/engines/engine.py.229: process: 0.49964 | load: 0.00241 [I 2022-10-26 11:14:51] ...trap/engines/engine.py.230: loss: 0.31525 [I 2022-10-26 11:14:51] ...optimizers/trijoint.py.151: total_norm: 4.259511 [I 2022-10-26 11:14:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 1550/2304 [I 2022-10-26 11:14:56] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:06:13 [I 2022-10-26 11:14:56] ...trap/engines/engine.py.229: process: 0.49585 | load: 0.00253 [I 2022-10-26 11:14:56] ...trap/engines/engine.py.230: loss: 0.24392 [I 2022-10-26 11:14:56] ...optimizers/trijoint.py.151: total_norm: 3.610771 [I 2022-10-26 11:15:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 1560/2304 [I 2022-10-26 11:15:01] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:06:10 [I 2022-10-26 11:15:01] ...trap/engines/engine.py.229: process: 0.50024 | load: 0.00246 [I 2022-10-26 11:15:01] ...trap/engines/engine.py.230: loss: 0.21739 [I 2022-10-26 11:15:01] ...optimizers/trijoint.py.151: total_norm: 3.325210 [I 2022-10-26 11:15:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 1570/2304 [I 2022-10-26 11:15:06] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:06:03 [I 2022-10-26 11:15:06] ...trap/engines/engine.py.229: process: 0.49503 | load: 0.00247 [I 2022-10-26 11:15:06] ...trap/engines/engine.py.230: loss: 0.27676 [I 2022-10-26 11:15:06] ...optimizers/trijoint.py.151: total_norm: 3.967037 [I 2022-10-26 11:15:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 1580/2304 [I 2022-10-26 11:15:11] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:06:07 [I 2022-10-26 11:15:11] ...trap/engines/engine.py.229: process: 0.49679 | load: 0.00243 [I 2022-10-26 11:15:11] ...trap/engines/engine.py.230: loss: 0.31032 [I 2022-10-26 11:15:11] ...optimizers/trijoint.py.151: total_norm: 3.612238 [I 2022-10-26 11:15:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 1590/2304 [I 2022-10-26 11:15:16] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:05:59 [I 2022-10-26 11:15:16] ...trap/engines/engine.py.229: process: 0.51466 | load: 0.00247 [I 2022-10-26 11:15:16] ...trap/engines/engine.py.230: loss: 0.30528 [I 2022-10-26 11:15:16] ...optimizers/trijoint.py.151: total_norm: 4.021371 [I 2022-10-26 11:15:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 1600/2304 [I 2022-10-26 11:15:21] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:05:54 [I 2022-10-26 11:15:21] ...trap/engines/engine.py.229: process: 0.49334 | load: 0.00249 [I 2022-10-26 11:15:21] ...trap/engines/engine.py.230: loss: 0.33198 [I 2022-10-26 11:15:21] ...optimizers/trijoint.py.151: total_norm: 3.192971 [I 2022-10-26 11:15:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 1610/2304 [I 2022-10-26 11:15:26] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:05:45 [I 2022-10-26 11:15:26] ...trap/engines/engine.py.229: process: 0.49286 | load: 0.00244 [I 2022-10-26 11:15:26] ...trap/engines/engine.py.230: loss: 0.38877 [I 2022-10-26 11:15:26] ...optimizers/trijoint.py.151: total_norm: 4.764105 [I 2022-10-26 11:15:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 1620/2304 [I 2022-10-26 11:15:31] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:05:38 [I 2022-10-26 11:15:31] ...trap/engines/engine.py.229: process: 0.49998 | load: 0.00244 [I 2022-10-26 11:15:31] ...trap/engines/engine.py.230: loss: 0.39878 [I 2022-10-26 11:15:31] ...optimizers/trijoint.py.151: total_norm: 4.880831 [I 2022-10-26 11:15:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 1630/2304 [I 2022-10-26 11:15:36] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:05:32 [I 2022-10-26 11:15:36] ...trap/engines/engine.py.229: process: 0.48010 | load: 0.00264 [I 2022-10-26 11:15:36] ...trap/engines/engine.py.230: loss: 0.22009 [I 2022-10-26 11:15:36] ...optimizers/trijoint.py.151: total_norm: 3.004152 [I 2022-10-26 11:15:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 1640/2304 [I 2022-10-26 11:15:41] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:05:28 [I 2022-10-26 11:15:41] ...trap/engines/engine.py.229: process: 0.49952 | load: 0.00244 [I 2022-10-26 11:15:41] ...trap/engines/engine.py.230: loss: 0.28462 [I 2022-10-26 11:15:41] ...optimizers/trijoint.py.151: total_norm: 2.908793 [I 2022-10-26 11:15:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 1650/2304 [I 2022-10-26 11:15:46] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:05:24 [I 2022-10-26 11:15:46] ...trap/engines/engine.py.229: process: 0.49429 | load: 0.00245 [I 2022-10-26 11:15:46] ...trap/engines/engine.py.230: loss: 0.34867 [I 2022-10-26 11:15:46] ...optimizers/trijoint.py.151: total_norm: 3.461858 [I 2022-10-26 11:15:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 1660/2304 [I 2022-10-26 11:15:51] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:05:20 [I 2022-10-26 11:15:51] ...trap/engines/engine.py.229: process: 0.49558 | load: 0.00279 [I 2022-10-26 11:15:51] ...trap/engines/engine.py.230: loss: 0.29532 [I 2022-10-26 11:15:51] ...optimizers/trijoint.py.151: total_norm: 3.956685 [I 2022-10-26 11:15:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 1670/2304 [I 2022-10-26 11:15:56] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:05:26 [I 2022-10-26 11:15:56] ...trap/engines/engine.py.229: process: 0.49443 | load: 0.00241 [I 2022-10-26 11:15:56] ...trap/engines/engine.py.230: loss: 0.29152 [I 2022-10-26 11:15:56] ...optimizers/trijoint.py.151: total_norm: 3.005291 [I 2022-10-26 11:16:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 1680/2304 [I 2022-10-26 11:16:01] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:05:11 [I 2022-10-26 11:16:01] ...trap/engines/engine.py.229: process: 0.48863 | load: 0.00272 [I 2022-10-26 11:16:01] ...trap/engines/engine.py.230: loss: 0.37671 [I 2022-10-26 11:16:01] ...optimizers/trijoint.py.151: total_norm: 4.058781 [I 2022-10-26 11:16:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 1690/2304 [I 2022-10-26 11:16:06] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:05:05 [I 2022-10-26 11:16:06] ...trap/engines/engine.py.229: process: 0.50195 | load: 0.00248 [I 2022-10-26 11:16:06] ...trap/engines/engine.py.230: loss: 0.27338 [I 2022-10-26 11:16:06] ...optimizers/trijoint.py.151: total_norm: 3.788258 [I 2022-10-26 11:16:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 1700/2304 [I 2022-10-26 11:16:11] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:05:00 [I 2022-10-26 11:16:11] ...trap/engines/engine.py.229: process: 0.49676 | load: 0.00246 [I 2022-10-26 11:16:11] ...trap/engines/engine.py.230: loss: 0.32545 [I 2022-10-26 11:16:11] ...optimizers/trijoint.py.151: total_norm: 3.657541 [I 2022-10-26 11:16:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 1710/2304 [I 2022-10-26 11:16:16] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:04:56 [I 2022-10-26 11:16:16] ...trap/engines/engine.py.229: process: 0.49417 | load: 0.00248 [I 2022-10-26 11:16:16] ...trap/engines/engine.py.230: loss: 0.21986 [I 2022-10-26 11:16:16] ...optimizers/trijoint.py.151: total_norm: 2.836412 [I 2022-10-26 11:16:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 1720/2304 [I 2022-10-26 11:16:21] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:04:51 [I 2022-10-26 11:16:21] ...trap/engines/engine.py.229: process: 0.50548 | load: 0.00252 [I 2022-10-26 11:16:21] ...trap/engines/engine.py.230: loss: 0.32574 [I 2022-10-26 11:16:21] ...optimizers/trijoint.py.151: total_norm: 4.149309 [I 2022-10-26 11:16:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 1730/2304 [I 2022-10-26 11:16:26] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:04:46 [I 2022-10-26 11:16:26] ...trap/engines/engine.py.229: process: 0.49282 | load: 0.00243 [I 2022-10-26 11:16:26] ...trap/engines/engine.py.230: loss: 0.24171 [I 2022-10-26 11:16:26] ...optimizers/trijoint.py.151: total_norm: 3.804442 [I 2022-10-26 11:16:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 1740/2304 [I 2022-10-26 11:16:31] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:04:40 [I 2022-10-26 11:16:31] ...trap/engines/engine.py.229: process: 0.49294 | load: 0.00247 [I 2022-10-26 11:16:31] ...trap/engines/engine.py.230: loss: 0.36303 [I 2022-10-26 11:16:31] ...optimizers/trijoint.py.151: total_norm: 4.698232 [I 2022-10-26 11:16:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 1750/2304 [I 2022-10-26 11:16:36] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:04:40 [I 2022-10-26 11:16:36] ...trap/engines/engine.py.229: process: 0.50320 | load: 0.00245 [I 2022-10-26 11:16:36] ...trap/engines/engine.py.230: loss: 0.23066 [I 2022-10-26 11:16:36] ...optimizers/trijoint.py.151: total_norm: 3.655604 [I 2022-10-26 11:16:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 1760/2304 [I 2022-10-26 11:16:41] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:04:31 [I 2022-10-26 11:16:41] ...trap/engines/engine.py.229: process: 0.49894 | load: 0.00242 [I 2022-10-26 11:16:41] ...trap/engines/engine.py.230: loss: 0.24818 [I 2022-10-26 11:16:41] ...optimizers/trijoint.py.151: total_norm: 3.309088 [I 2022-10-26 11:16:46] ...trap/engines/engine.py.224: train: epoch 4 | batch 1770/2304 [I 2022-10-26 11:16:46] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:04:25 [I 2022-10-26 11:16:46] ...trap/engines/engine.py.229: process: 0.48284 | load: 0.00242 [I 2022-10-26 11:16:46] ...trap/engines/engine.py.230: loss: 0.24433 [I 2022-10-26 11:16:46] ...optimizers/trijoint.py.151: total_norm: 3.179937 [I 2022-10-26 11:16:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 1780/2304 [I 2022-10-26 11:16:51] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:04:20 [I 2022-10-26 11:16:51] ...trap/engines/engine.py.229: process: 0.50304 | load: 0.00241 [I 2022-10-26 11:16:51] ...trap/engines/engine.py.230: loss: 0.26036 [I 2022-10-26 11:16:51] ...optimizers/trijoint.py.151: total_norm: 4.063525 [I 2022-10-26 11:16:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 1790/2304 [I 2022-10-26 11:16:56] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:04:14 [I 2022-10-26 11:16:56] ...trap/engines/engine.py.229: process: 0.49449 | load: 0.00253 [I 2022-10-26 11:16:56] ...trap/engines/engine.py.230: loss: 0.32227 [I 2022-10-26 11:16:56] ...optimizers/trijoint.py.151: total_norm: 4.186669 [I 2022-10-26 11:17:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 1800/2304 [I 2022-10-26 11:17:01] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:04:09 [I 2022-10-26 11:17:01] ...trap/engines/engine.py.229: process: 0.49532 | load: 0.00247 [I 2022-10-26 11:17:01] ...trap/engines/engine.py.230: loss: 0.21897 [I 2022-10-26 11:17:01] ...optimizers/trijoint.py.151: total_norm: 3.281135 [I 2022-10-26 11:17:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 1810/2304 [I 2022-10-26 11:17:06] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:04:06 [I 2022-10-26 11:17:06] ...trap/engines/engine.py.229: process: 0.51088 | load: 0.00247 [I 2022-10-26 11:17:06] ...trap/engines/engine.py.230: loss: 0.22955 [I 2022-10-26 11:17:06] ...optimizers/trijoint.py.151: total_norm: 3.517903 [I 2022-10-26 11:17:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 1820/2304 [I 2022-10-26 11:17:11] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:04:03 [I 2022-10-26 11:17:11] ...trap/engines/engine.py.229: process: 0.49018 | load: 0.00250 [I 2022-10-26 11:17:11] ...trap/engines/engine.py.230: loss: 0.21548 [I 2022-10-26 11:17:11] ...optimizers/trijoint.py.151: total_norm: 3.006627 [I 2022-10-26 11:17:16] ...trap/engines/engine.py.224: train: epoch 4 | batch 1830/2304 [I 2022-10-26 11:17:16] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:03:56 [I 2022-10-26 11:17:16] ...trap/engines/engine.py.229: process: 0.50067 | load: 0.00244 [I 2022-10-26 11:17:16] ...trap/engines/engine.py.230: loss: 0.30305 [I 2022-10-26 11:17:16] ...optimizers/trijoint.py.151: total_norm: 3.821606 [I 2022-10-26 11:17:21] ...trap/engines/engine.py.224: train: epoch 4 | batch 1840/2304 [I 2022-10-26 11:17:21] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:03:54 [I 2022-10-26 11:17:21] ...trap/engines/engine.py.229: process: 0.50753 | load: 0.00282 [I 2022-10-26 11:17:21] ...trap/engines/engine.py.230: loss: 0.29261 [I 2022-10-26 11:17:21] ...optimizers/trijoint.py.151: total_norm: 3.415993 [I 2022-10-26 11:17:26] ...trap/engines/engine.py.224: train: epoch 4 | batch 1850/2304 [I 2022-10-26 11:17:26] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:03:47 [I 2022-10-26 11:17:26] ...trap/engines/engine.py.229: process: 0.50394 | load: 0.00242 [I 2022-10-26 11:17:26] ...trap/engines/engine.py.230: loss: 0.34030 [I 2022-10-26 11:17:26] ...optimizers/trijoint.py.151: total_norm: 4.111091 [I 2022-10-26 11:17:31] ...trap/engines/engine.py.224: train: epoch 4 | batch 1860/2304 [I 2022-10-26 11:17:31] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:03:40 [I 2022-10-26 11:17:31] ...trap/engines/engine.py.229: process: 0.49464 | load: 0.00243 [I 2022-10-26 11:17:31] ...trap/engines/engine.py.230: loss: 0.25469 [I 2022-10-26 11:17:31] ...optimizers/trijoint.py.151: total_norm: 4.359564 [I 2022-10-26 11:17:36] ...trap/engines/engine.py.224: train: epoch 4 | batch 1870/2304 [I 2022-10-26 11:17:36] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:03:37 [I 2022-10-26 11:17:36] ...trap/engines/engine.py.229: process: 0.50945 | load: 0.00242 [I 2022-10-26 11:17:36] ...trap/engines/engine.py.230: loss: 0.25021 [I 2022-10-26 11:17:36] ...optimizers/trijoint.py.151: total_norm: 3.319418 [I 2022-10-26 11:17:41] ...trap/engines/engine.py.224: train: epoch 4 | batch 1880/2304 [I 2022-10-26 11:17:41] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:03:32 [I 2022-10-26 11:17:41] ...trap/engines/engine.py.229: process: 0.50309 | load: 0.00244 [I 2022-10-26 11:17:41] ...trap/engines/engine.py.230: loss: 0.26959 [I 2022-10-26 11:17:41] ...optimizers/trijoint.py.151: total_norm: 3.431398 [I 2022-10-26 11:17:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 1890/2304 [I 2022-10-26 11:17:47] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:03:29 [I 2022-10-26 11:17:47] ...trap/engines/engine.py.229: process: 0.50831 | load: 0.00242 [I 2022-10-26 11:17:47] ...trap/engines/engine.py.230: loss: 0.36243 [I 2022-10-26 11:17:47] ...optimizers/trijoint.py.151: total_norm: 4.302494 [I 2022-10-26 11:17:51] ...trap/engines/engine.py.224: train: epoch 4 | batch 1900/2304 [I 2022-10-26 11:17:51] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:03:20 [I 2022-10-26 11:17:51] ...trap/engines/engine.py.229: process: 0.48397 | load: 0.00253 [I 2022-10-26 11:17:51] ...trap/engines/engine.py.230: loss: 0.35129 [I 2022-10-26 11:17:51] ...optimizers/trijoint.py.151: total_norm: 3.607842 [I 2022-10-26 11:17:56] ...trap/engines/engine.py.224: train: epoch 4 | batch 1910/2304 [I 2022-10-26 11:17:56] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:03:15 [I 2022-10-26 11:17:56] ...trap/engines/engine.py.229: process: 0.50348 | load: 0.00250 [I 2022-10-26 11:17:56] ...trap/engines/engine.py.230: loss: 0.32371 [I 2022-10-26 11:17:56] ...optimizers/trijoint.py.151: total_norm: 4.310977 [I 2022-10-26 11:18:01] ...trap/engines/engine.py.224: train: epoch 4 | batch 1920/2304 [I 2022-10-26 11:18:01] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:03:10 [I 2022-10-26 11:18:01] ...trap/engines/engine.py.229: process: 0.50407 | load: 0.00248 [I 2022-10-26 11:18:01] ...trap/engines/engine.py.230: loss: 0.32540 [I 2022-10-26 11:18:01] ...optimizers/trijoint.py.151: total_norm: 4.613842 [I 2022-10-26 11:18:06] ...trap/engines/engine.py.224: train: epoch 4 | batch 1930/2304 [I 2022-10-26 11:18:06] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:03:07 [I 2022-10-26 11:18:06] ...trap/engines/engine.py.229: process: 0.51174 | load: 0.00244 [I 2022-10-26 11:18:06] ...trap/engines/engine.py.230: loss: 0.45653 [I 2022-10-26 11:18:06] ...optimizers/trijoint.py.151: total_norm: 3.849960 [I 2022-10-26 11:18:11] ...trap/engines/engine.py.224: train: epoch 4 | batch 1940/2304 [I 2022-10-26 11:18:11] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:03:02 [I 2022-10-26 11:18:11] ...trap/engines/engine.py.229: process: 0.49414 | load: 0.00241 [I 2022-10-26 11:18:11] ...trap/engines/engine.py.230: loss: 0.35413 [I 2022-10-26 11:18:11] ...optimizers/trijoint.py.151: total_norm: 4.925144 [I 2022-10-26 11:18:17] ...trap/engines/engine.py.224: train: epoch 4 | batch 1950/2304 [I 2022-10-26 11:18:17] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:02:58 [I 2022-10-26 11:18:17] ...trap/engines/engine.py.229: process: 0.50049 | load: 0.00246 [I 2022-10-26 11:18:17] ...trap/engines/engine.py.230: loss: 0.15867 [I 2022-10-26 11:18:17] ...optimizers/trijoint.py.151: total_norm: 3.018731 [I 2022-10-26 11:18:22] ...trap/engines/engine.py.224: train: epoch 4 | batch 1960/2304 [I 2022-10-26 11:18:22] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:02:53 [I 2022-10-26 11:18:22] ...trap/engines/engine.py.229: process: 0.50828 | load: 0.00242 [I 2022-10-26 11:18:22] ...trap/engines/engine.py.230: loss: 0.25462 [I 2022-10-26 11:18:22] ...optimizers/trijoint.py.151: total_norm: 2.737376 [I 2022-10-26 11:18:27] ...trap/engines/engine.py.224: train: epoch 4 | batch 1970/2304 [I 2022-10-26 11:18:27] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:02:47 [I 2022-10-26 11:18:27] ...trap/engines/engine.py.229: process: 0.50665 | load: 0.00246 [I 2022-10-26 11:18:27] ...trap/engines/engine.py.230: loss: 0.23814 [I 2022-10-26 11:18:27] ...optimizers/trijoint.py.151: total_norm: 4.209979 [I 2022-10-26 11:18:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 1980/2304 [I 2022-10-26 11:18:32] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:02:40 [I 2022-10-26 11:18:32] ...trap/engines/engine.py.229: process: 0.49420 | load: 0.00241 [I 2022-10-26 11:18:32] ...trap/engines/engine.py.230: loss: 0.34736 [I 2022-10-26 11:18:32] ...optimizers/trijoint.py.151: total_norm: 3.829628 [I 2022-10-26 11:18:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 1990/2304 [I 2022-10-26 11:18:37] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:02:37 [I 2022-10-26 11:18:37] ...trap/engines/engine.py.229: process: 0.49678 | load: 0.00242 [I 2022-10-26 11:18:37] ...trap/engines/engine.py.230: loss: 0.27337 [I 2022-10-26 11:18:37] ...optimizers/trijoint.py.151: total_norm: 4.459150 [I 2022-10-26 11:18:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 2000/2304 [I 2022-10-26 11:18:42] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:02:32 [I 2022-10-26 11:18:42] ...trap/engines/engine.py.229: process: 0.49797 | load: 0.00251 [I 2022-10-26 11:18:42] ...trap/engines/engine.py.230: loss: 0.32590 [I 2022-10-26 11:18:42] ...optimizers/trijoint.py.151: total_norm: 4.439370 [I 2022-10-26 11:18:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 2010/2304 [I 2022-10-26 11:18:47] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:02:25 [I 2022-10-26 11:18:47] ...trap/engines/engine.py.229: process: 0.50022 | load: 0.00246 [I 2022-10-26 11:18:47] ...trap/engines/engine.py.230: loss: 0.26935 [I 2022-10-26 11:18:47] ...optimizers/trijoint.py.151: total_norm: 3.100965 [I 2022-10-26 11:18:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 2020/2304 [I 2022-10-26 11:18:52] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:02:20 [I 2022-10-26 11:18:52] ...trap/engines/engine.py.229: process: 0.48900 | load: 0.00245 [I 2022-10-26 11:18:52] ...trap/engines/engine.py.230: loss: 0.21504 [I 2022-10-26 11:18:52] ...optimizers/trijoint.py.151: total_norm: 3.355964 [I 2022-10-26 11:18:57] ...trap/engines/engine.py.224: train: epoch 4 | batch 2030/2304 [I 2022-10-26 11:18:57] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:02:17 [I 2022-10-26 11:18:57] ...trap/engines/engine.py.229: process: 0.49031 | load: 0.00247 [I 2022-10-26 11:18:57] ...trap/engines/engine.py.230: loss: 0.29660 [I 2022-10-26 11:18:57] ...optimizers/trijoint.py.151: total_norm: 3.435297 [I 2022-10-26 11:19:02] ...trap/engines/engine.py.224: train: epoch 4 | batch 2040/2304 [I 2022-10-26 11:19:02] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:02:12 [I 2022-10-26 11:19:02] ...trap/engines/engine.py.229: process: 0.50060 | load: 0.00243 [I 2022-10-26 11:19:02] ...trap/engines/engine.py.230: loss: 0.35055 [I 2022-10-26 11:19:02] ...optimizers/trijoint.py.151: total_norm: 4.291605 [I 2022-10-26 11:19:07] ...trap/engines/engine.py.224: train: epoch 4 | batch 2050/2304 [I 2022-10-26 11:19:07] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:02:06 [I 2022-10-26 11:19:07] ...trap/engines/engine.py.229: process: 0.49702 | load: 0.00256 [I 2022-10-26 11:19:07] ...trap/engines/engine.py.230: loss: 0.23992 [I 2022-10-26 11:19:07] ...optimizers/trijoint.py.151: total_norm: 3.856994 [I 2022-10-26 11:19:12] ...trap/engines/engine.py.224: train: epoch 4 | batch 2060/2304 [I 2022-10-26 11:19:12] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:02:01 [I 2022-10-26 11:19:12] ...trap/engines/engine.py.229: process: 0.50639 | load: 0.00242 [I 2022-10-26 11:19:12] ...trap/engines/engine.py.230: loss: 0.24168 [I 2022-10-26 11:19:12] ...optimizers/trijoint.py.151: total_norm: 3.381026 [I 2022-10-26 11:19:17] ...trap/engines/engine.py.224: train: epoch 4 | batch 2070/2304 [I 2022-10-26 11:19:17] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:01:55 [I 2022-10-26 11:19:17] ...trap/engines/engine.py.229: process: 0.49836 | load: 0.00242 [I 2022-10-26 11:19:17] ...trap/engines/engine.py.230: loss: 0.27479 [I 2022-10-26 11:19:17] ...optimizers/trijoint.py.151: total_norm: 3.770941 [I 2022-10-26 11:19:22] ...trap/engines/engine.py.224: train: epoch 4 | batch 2080/2304 [I 2022-10-26 11:19:22] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:01:50 [I 2022-10-26 11:19:22] ...trap/engines/engine.py.229: process: 0.49383 | load: 0.00245 [I 2022-10-26 11:19:22] ...trap/engines/engine.py.230: loss: 0.31229 [I 2022-10-26 11:19:22] ...optimizers/trijoint.py.151: total_norm: 3.978054 [I 2022-10-26 11:19:27] ...trap/engines/engine.py.224: train: epoch 4 | batch 2090/2304 [I 2022-10-26 11:19:27] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:01:46 [I 2022-10-26 11:19:27] ...trap/engines/engine.py.229: process: 0.50520 | load: 0.00240 [I 2022-10-26 11:19:27] ...trap/engines/engine.py.230: loss: 0.26705 [I 2022-10-26 11:19:27] ...optimizers/trijoint.py.151: total_norm: 3.903678 [I 2022-10-26 11:19:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 2100/2304 [I 2022-10-26 11:19:32] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:01:43 [I 2022-10-26 11:19:32] ...trap/engines/engine.py.229: process: 0.49107 | load: 0.00242 [I 2022-10-26 11:19:32] ...trap/engines/engine.py.230: loss: 0.26257 [I 2022-10-26 11:19:32] ...optimizers/trijoint.py.151: total_norm: 3.383372 [I 2022-10-26 11:19:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 2110/2304 [I 2022-10-26 11:19:37] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:01:36 [I 2022-10-26 11:19:37] ...trap/engines/engine.py.229: process: 0.48560 | load: 0.00249 [I 2022-10-26 11:19:37] ...trap/engines/engine.py.230: loss: 0.21761 [I 2022-10-26 11:19:37] ...optimizers/trijoint.py.151: total_norm: 3.408775 [I 2022-10-26 11:19:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 2120/2304 [I 2022-10-26 11:19:42] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:01:31 [I 2022-10-26 11:19:42] ...trap/engines/engine.py.229: process: 0.49434 | load: 0.00244 [I 2022-10-26 11:19:42] ...trap/engines/engine.py.230: loss: 0.21029 [I 2022-10-26 11:19:42] ...optimizers/trijoint.py.151: total_norm: 3.963093 [I 2022-10-26 11:19:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 2130/2304 [I 2022-10-26 11:19:47] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:01:26 [I 2022-10-26 11:19:47] ...trap/engines/engine.py.229: process: 0.47859 | load: 0.00247 [I 2022-10-26 11:19:47] ...trap/engines/engine.py.230: loss: 0.27565 [I 2022-10-26 11:19:47] ...optimizers/trijoint.py.151: total_norm: 3.521951 [I 2022-10-26 11:19:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 2140/2304 [I 2022-10-26 11:19:52] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:01:22 [I 2022-10-26 11:19:52] ...trap/engines/engine.py.229: process: 0.53134 | load: 0.00246 [I 2022-10-26 11:19:52] ...trap/engines/engine.py.230: loss: 0.24446 [I 2022-10-26 11:19:52] ...optimizers/trijoint.py.151: total_norm: 4.150137 [I 2022-10-26 11:19:57] ...trap/engines/engine.py.224: train: epoch 4 | batch 2150/2304 [I 2022-10-26 11:19:57] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:01:17 [I 2022-10-26 11:19:57] ...trap/engines/engine.py.229: process: 0.50013 | load: 0.00251 [I 2022-10-26 11:19:57] ...trap/engines/engine.py.230: loss: 0.33588 [I 2022-10-26 11:19:57] ...optimizers/trijoint.py.151: total_norm: 4.867143 [I 2022-10-26 11:20:02] ...trap/engines/engine.py.224: train: epoch 4 | batch 2160/2304 [I 2022-10-26 11:20:02] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:01:11 [I 2022-10-26 11:20:02] ...trap/engines/engine.py.229: process: 0.49770 | load: 0.00245 [I 2022-10-26 11:20:02] ...trap/engines/engine.py.230: loss: 0.30319 [I 2022-10-26 11:20:02] ...optimizers/trijoint.py.151: total_norm: 3.977938 [I 2022-10-26 11:20:07] ...trap/engines/engine.py.224: train: epoch 4 | batch 2170/2304 [I 2022-10-26 11:20:07] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:01:09 [I 2022-10-26 11:20:07] ...trap/engines/engine.py.229: process: 0.49295 | load: 0.00242 [I 2022-10-26 11:20:07] ...trap/engines/engine.py.230: loss: 0.21726 [I 2022-10-26 11:20:07] ...optimizers/trijoint.py.151: total_norm: 3.067268 [I 2022-10-26 11:20:12] ...trap/engines/engine.py.224: train: epoch 4 | batch 2180/2304 [I 2022-10-26 11:20:12] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:01:02 [I 2022-10-26 11:20:12] ...trap/engines/engine.py.229: process: 0.49589 | load: 0.00246 [I 2022-10-26 11:20:12] ...trap/engines/engine.py.230: loss: 0.23328 [I 2022-10-26 11:20:12] ...optimizers/trijoint.py.151: total_norm: 3.644489 [I 2022-10-26 11:20:17] ...trap/engines/engine.py.224: train: epoch 4 | batch 2190/2304 [I 2022-10-26 11:20:17] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:00:57 [I 2022-10-26 11:20:17] ...trap/engines/engine.py.229: process: 0.50062 | load: 0.00258 [I 2022-10-26 11:20:17] ...trap/engines/engine.py.230: loss: 0.37191 [I 2022-10-26 11:20:17] ...optimizers/trijoint.py.151: total_norm: 3.759424 [I 2022-10-26 11:20:22] ...trap/engines/engine.py.224: train: epoch 4 | batch 2200/2304 [I 2022-10-26 11:20:22] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:00:51 [I 2022-10-26 11:20:22] ...trap/engines/engine.py.229: process: 0.50381 | load: 0.00251 [I 2022-10-26 11:20:22] ...trap/engines/engine.py.230: loss: 0.25281 [I 2022-10-26 11:20:22] ...optimizers/trijoint.py.151: total_norm: 2.590159 [I 2022-10-26 11:20:27] ...trap/engines/engine.py.224: train: epoch 4 | batch 2210/2304 [I 2022-10-26 11:20:27] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:00:46 [I 2022-10-26 11:20:27] ...trap/engines/engine.py.229: process: 0.49213 | load: 0.00250 [I 2022-10-26 11:20:27] ...trap/engines/engine.py.230: loss: 0.32228 [I 2022-10-26 11:20:27] ...optimizers/trijoint.py.151: total_norm: 3.658482 [I 2022-10-26 11:20:32] ...trap/engines/engine.py.224: train: epoch 4 | batch 2220/2304 [I 2022-10-26 11:20:32] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:00:41 [I 2022-10-26 11:20:32] ...trap/engines/engine.py.229: process: 0.50119 | load: 0.00249 [I 2022-10-26 11:20:32] ...trap/engines/engine.py.230: loss: 0.21079 [I 2022-10-26 11:20:32] ...optimizers/trijoint.py.151: total_norm: 3.137305 [I 2022-10-26 11:20:37] ...trap/engines/engine.py.224: train: epoch 4 | batch 2230/2304 [I 2022-10-26 11:20:37] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:00:37 [I 2022-10-26 11:20:37] ...trap/engines/engine.py.229: process: 0.53599 | load: 0.00243 [I 2022-10-26 11:20:37] ...trap/engines/engine.py.230: loss: 0.24878 [I 2022-10-26 11:20:37] ...optimizers/trijoint.py.151: total_norm: 3.343382 [I 2022-10-26 11:20:42] ...trap/engines/engine.py.224: train: epoch 4 | batch 2240/2304 [I 2022-10-26 11:20:42] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:00:32 [I 2022-10-26 11:20:42] ...trap/engines/engine.py.229: process: 0.50777 | load: 0.00251 [I 2022-10-26 11:20:42] ...trap/engines/engine.py.230: loss: 0.23117 [I 2022-10-26 11:20:42] ...optimizers/trijoint.py.151: total_norm: 3.562478 [I 2022-10-26 11:20:47] ...trap/engines/engine.py.224: train: epoch 4 | batch 2250/2304 [I 2022-10-26 11:20:47] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:00:26 [I 2022-10-26 11:20:47] ...trap/engines/engine.py.229: process: 0.49343 | load: 0.00245 [I 2022-10-26 11:20:47] ...trap/engines/engine.py.230: loss: 0.29228 [I 2022-10-26 11:20:47] ...optimizers/trijoint.py.151: total_norm: 3.660161 [I 2022-10-26 11:20:52] ...trap/engines/engine.py.224: train: epoch 4 | batch 2260/2304 [I 2022-10-26 11:20:52] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:00:21 [I 2022-10-26 11:20:52] ...trap/engines/engine.py.229: process: 0.50093 | load: 0.00246 [I 2022-10-26 11:20:52] ...trap/engines/engine.py.230: loss: 0.27452 [I 2022-10-26 11:20:52] ...optimizers/trijoint.py.151: total_norm: 3.440088 [I 2022-10-26 11:20:57] ...trap/engines/engine.py.224: train: epoch 4 | batch 2270/2304 [I 2022-10-26 11:20:57] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:00:16 [I 2022-10-26 11:20:57] ...trap/engines/engine.py.229: process: 0.50053 | load: 0.00240 [I 2022-10-26 11:20:57] ...trap/engines/engine.py.230: loss: 0.20196 [I 2022-10-26 11:20:57] ...optimizers/trijoint.py.151: total_norm: 3.162028 [I 2022-10-26 11:21:02] ...trap/engines/engine.py.224: train: epoch 4 | batch 2280/2304 [I 2022-10-26 11:21:02] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:00:12 [I 2022-10-26 11:21:02] ...trap/engines/engine.py.229: process: 0.50290 | load: 0.00243 [I 2022-10-26 11:21:02] ...trap/engines/engine.py.230: loss: 0.32077 [I 2022-10-26 11:21:02] ...optimizers/trijoint.py.151: total_norm: 3.547247 [I 2022-10-26 11:21:07] ...trap/engines/engine.py.224: train: epoch 4 | batch 2290/2304 [I 2022-10-26 11:21:07] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:00:07 [I 2022-10-26 11:21:07] ...trap/engines/engine.py.229: process: 0.50795 | load: 0.00250 [I 2022-10-26 11:21:07] ...trap/engines/engine.py.230: loss: 0.44192 [I 2022-10-26 11:21:07] ...optimizers/trijoint.py.151: total_norm: 5.090762 [I 2022-10-26 11:21:12] ...trap/engines/engine.py.224: train: epoch 4 | batch 2300/2304 [I 2022-10-26 11:21:12] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:00:02 [I 2022-10-26 11:21:12] ...trap/engines/engine.py.229: process: 0.47839 | load: 0.00014 [I 2022-10-26 11:21:12] ...trap/engines/engine.py.230: loss: 0.21575 [I 2022-10-26 11:21:12] ...optimizers/trijoint.py.151: total_norm: 3.508178 [I 2022-10-26 11:21:14] ...trap/engines/engine.py.224: train: epoch 4 | batch 2304/2304 [I 2022-10-26 11:21:14] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:00:00 [I 2022-10-26 11:21:14] ...trap/engines/engine.py.229: process: 0.48751 | load: 0.00014 [I 2022-10-26 11:21:14] ...trap/engines/engine.py.230: loss: 0.24265 [I 2022-10-26 11:21:14] ...optimizers/trijoint.py.151: total_norm: 2.749809 [S 2022-10-26 11:21:15] ...trap/engines/engine.py.129: train_epoch.epoch: 4 [S 2022-10-26 11:21:15] ...trap/engines/engine.py.129: train_epoch.loss: 0.291718706630833 [I 2022-10-26 11:21:15] ...s/criterions/triplet.py.84: margin = 0.07500000000000001 [I 2022-10-26 11:21:15] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 4 [I 2022-10-26 11:21:15] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 11:21:17] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 11:21:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 11:21:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 11:21:17] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 11:21:18] ...trap/engines/engine.py.321: eval: epoch 4 | batch 0/510 [I 2022-10-26 11:21:18] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:21:49 [I 2022-10-26 11:21:18] ...trap/engines/engine.py.326: process: 2.56862 | load: 2.19838 [I 2022-10-26 11:21:21] ...trap/engines/engine.py.321: eval: epoch 4 | batch 10/510 [I 2022-10-26 11:21:21] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:04:50 [I 2022-10-26 11:21:21] ...trap/engines/engine.py.326: process: 0.35595 | load: 0.00018 [I 2022-10-26 11:21:24] ...trap/engines/engine.py.321: eval: epoch 4 | batch 20/510 [I 2022-10-26 11:21:24] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:57 [I 2022-10-26 11:21:24] ...trap/engines/engine.py.326: process: 0.31979 | load: 0.00022 [I 2022-10-26 11:21:28] ...trap/engines/engine.py.321: eval: epoch 4 | batch 30/510 [I 2022-10-26 11:21:28] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:43 [I 2022-10-26 11:21:28] ...trap/engines/engine.py.326: process: 0.32420 | load: 0.00021 [I 2022-10-26 11:21:31] ...trap/engines/engine.py.321: eval: epoch 4 | batch 40/510 [I 2022-10-26 11:21:31] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:36 [I 2022-10-26 11:21:31] ...trap/engines/engine.py.326: process: 0.32938 | load: 0.00018 [I 2022-10-26 11:21:34] ...trap/engines/engine.py.321: eval: epoch 4 | batch 50/510 [I 2022-10-26 11:21:34] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:37 [I 2022-10-26 11:21:34] ...trap/engines/engine.py.326: process: 0.35668 | load: 0.00019 [I 2022-10-26 11:21:38] ...trap/engines/engine.py.321: eval: epoch 4 | batch 60/510 [I 2022-10-26 11:21:38] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:31 [I 2022-10-26 11:21:38] ...trap/engines/engine.py.326: process: 0.33170 | load: 0.00024 [I 2022-10-26 11:21:41] ...trap/engines/engine.py.321: eval: epoch 4 | batch 70/510 [I 2022-10-26 11:21:41] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-26 11:21:41] ...trap/engines/engine.py.326: process: 0.32861 | load: 0.00017 [I 2022-10-26 11:21:45] ...trap/engines/engine.py.321: eval: epoch 4 | batch 80/510 [I 2022-10-26 11:21:45] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:23 [I 2022-10-26 11:21:45] ...trap/engines/engine.py.326: process: 0.32636 | load: 0.00019 [I 2022-10-26 11:21:48] ...trap/engines/engine.py.321: eval: epoch 4 | batch 90/510 [I 2022-10-26 11:21:48] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:20 [I 2022-10-26 11:21:48] ...trap/engines/engine.py.326: process: 0.32794 | load: 0.00018 [I 2022-10-26 11:21:51] ...trap/engines/engine.py.321: eval: epoch 4 | batch 100/510 [I 2022-10-26 11:21:51] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:11 [I 2022-10-26 11:21:51] ...trap/engines/engine.py.326: process: 0.28161 | load: 0.00020 [I 2022-10-26 11:21:54] ...trap/engines/engine.py.321: eval: epoch 4 | batch 110/510 [I 2022-10-26 11:21:54] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:58 [I 2022-10-26 11:21:54] ...trap/engines/engine.py.326: process: 0.29230 | load: 0.00017 [I 2022-10-26 11:21:57] ...trap/engines/engine.py.321: eval: epoch 4 | batch 120/510 [I 2022-10-26 11:21:57] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 11:21:57] ...trap/engines/engine.py.326: process: 0.30919 | load: 0.00017 [I 2022-10-26 11:22:00] ...trap/engines/engine.py.321: eval: epoch 4 | batch 130/510 [I 2022-10-26 11:22:00] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:50 [I 2022-10-26 11:22:00] ...trap/engines/engine.py.326: process: 0.28764 | load: 0.00019 [I 2022-10-26 11:22:03] ...trap/engines/engine.py.321: eval: epoch 4 | batch 140/510 [I 2022-10-26 11:22:03] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:47 [I 2022-10-26 11:22:03] ...trap/engines/engine.py.326: process: 0.28770 | load: 0.00022 [I 2022-10-26 11:22:06] ...trap/engines/engine.py.321: eval: epoch 4 | batch 150/510 [I 2022-10-26 11:22:06] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:44 [I 2022-10-26 11:22:06] ...trap/engines/engine.py.326: process: 0.30275 | load: 0.00016 [I 2022-10-26 11:22:09] ...trap/engines/engine.py.321: eval: epoch 4 | batch 160/510 [I 2022-10-26 11:22:09] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 11:22:09] ...trap/engines/engine.py.326: process: 0.28687 | load: 0.00018 [I 2022-10-26 11:22:12] ...trap/engines/engine.py.321: eval: epoch 4 | batch 170/510 [I 2022-10-26 11:22:12] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 11:22:12] ...trap/engines/engine.py.326: process: 0.28865 | load: 0.00015 [I 2022-10-26 11:22:14] ...trap/engines/engine.py.321: eval: epoch 4 | batch 180/510 [I 2022-10-26 11:22:14] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:36 [I 2022-10-26 11:22:14] ...trap/engines/engine.py.326: process: 0.30490 | load: 0.00014 [I 2022-10-26 11:22:17] ...trap/engines/engine.py.321: eval: epoch 4 | batch 190/510 [I 2022-10-26 11:22:17] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:33 [I 2022-10-26 11:22:17] ...trap/engines/engine.py.326: process: 0.29703 | load: 0.00018 [I 2022-10-26 11:22:20] ...trap/engines/engine.py.321: eval: epoch 4 | batch 200/510 [I 2022-10-26 11:22:20] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-26 11:22:20] ...trap/engines/engine.py.326: process: 0.29045 | load: 0.00022 [I 2022-10-26 11:22:23] ...trap/engines/engine.py.321: eval: epoch 4 | batch 210/510 [I 2022-10-26 11:22:23] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-26 11:22:23] ...trap/engines/engine.py.326: process: 0.28554 | load: 0.00019 [I 2022-10-26 11:22:26] ...trap/engines/engine.py.321: eval: epoch 4 | batch 220/510 [I 2022-10-26 11:22:26] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 11:22:26] ...trap/engines/engine.py.326: process: 0.28576 | load: 0.00015 [I 2022-10-26 11:22:29] ...trap/engines/engine.py.321: eval: epoch 4 | batch 230/510 [I 2022-10-26 11:22:29] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:20 [I 2022-10-26 11:22:29] ...trap/engines/engine.py.326: process: 0.29085 | load: 0.00021 [I 2022-10-26 11:22:32] ...trap/engines/engine.py.321: eval: epoch 4 | batch 240/510 [I 2022-10-26 11:22:32] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:17 [I 2022-10-26 11:22:32] ...trap/engines/engine.py.326: process: 0.29208 | load: 0.00014 [I 2022-10-26 11:22:35] ...trap/engines/engine.py.321: eval: epoch 4 | batch 250/510 [I 2022-10-26 11:22:35] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:15 [I 2022-10-26 11:22:35] ...trap/engines/engine.py.326: process: 0.28732 | load: 0.00015 [I 2022-10-26 11:22:38] ...trap/engines/engine.py.321: eval: epoch 4 | batch 260/510 [I 2022-10-26 11:22:38] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:12 [I 2022-10-26 11:22:38] ...trap/engines/engine.py.326: process: 0.28269 | load: 0.00020 [I 2022-10-26 11:22:41] ...trap/engines/engine.py.321: eval: epoch 4 | batch 270/510 [I 2022-10-26 11:22:41] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 11:22:41] ...trap/engines/engine.py.326: process: 0.28372 | load: 0.00014 [I 2022-10-26 11:22:43] ...trap/engines/engine.py.321: eval: epoch 4 | batch 280/510 [I 2022-10-26 11:22:43] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 11:22:43] ...trap/engines/engine.py.326: process: 0.28733 | load: 0.00014 [I 2022-10-26 11:22:46] ...trap/engines/engine.py.321: eval: epoch 4 | batch 290/510 [I 2022-10-26 11:22:46] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 11:22:46] ...trap/engines/engine.py.326: process: 0.28877 | load: 0.00015 [I 2022-10-26 11:22:49] ...trap/engines/engine.py.321: eval: epoch 4 | batch 300/510 [I 2022-10-26 11:22:49] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:00:59 [I 2022-10-26 11:22:49] ...trap/engines/engine.py.326: process: 0.28196 | load: 0.00016 [I 2022-10-26 11:22:52] ...trap/engines/engine.py.321: eval: epoch 4 | batch 310/510 [I 2022-10-26 11:22:52] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:56 [I 2022-10-26 11:22:52] ...trap/engines/engine.py.326: process: 0.28375 | load: 0.00016 [I 2022-10-26 11:22:55] ...trap/engines/engine.py.321: eval: epoch 4 | batch 320/510 [I 2022-10-26 11:22:55] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:54 [I 2022-10-26 11:22:55] ...trap/engines/engine.py.326: process: 0.28783 | load: 0.00024 [I 2022-10-26 11:22:58] ...trap/engines/engine.py.321: eval: epoch 4 | batch 330/510 [I 2022-10-26 11:22:58] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:51 [I 2022-10-26 11:22:58] ...trap/engines/engine.py.326: process: 0.29077 | load: 0.00016 [I 2022-10-26 11:23:01] ...trap/engines/engine.py.321: eval: epoch 4 | batch 340/510 [I 2022-10-26 11:23:01] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:49 [I 2022-10-26 11:23:01] ...trap/engines/engine.py.326: process: 0.28555 | load: 0.00021 [I 2022-10-26 11:23:04] ...trap/engines/engine.py.321: eval: epoch 4 | batch 350/510 [I 2022-10-26 11:23:04] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 11:23:04] ...trap/engines/engine.py.326: process: 0.28938 | load: 0.00016 [I 2022-10-26 11:23:06] ...trap/engines/engine.py.321: eval: epoch 4 | batch 360/510 [I 2022-10-26 11:23:06] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 11:23:06] ...trap/engines/engine.py.326: process: 0.28738 | load: 0.00018 [I 2022-10-26 11:23:09] ...trap/engines/engine.py.321: eval: epoch 4 | batch 370/510 [I 2022-10-26 11:23:09] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 11:23:09] ...trap/engines/engine.py.326: process: 0.28508 | load: 0.00014 [I 2022-10-26 11:23:12] ...trap/engines/engine.py.321: eval: epoch 4 | batch 380/510 [I 2022-10-26 11:23:12] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 11:23:12] ...trap/engines/engine.py.326: process: 0.29040 | load: 0.00016 [I 2022-10-26 11:23:15] ...trap/engines/engine.py.321: eval: epoch 4 | batch 390/510 [I 2022-10-26 11:23:15] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 11:23:15] ...trap/engines/engine.py.326: process: 0.28757 | load: 0.00017 [I 2022-10-26 11:23:18] ...trap/engines/engine.py.321: eval: epoch 4 | batch 400/510 [I 2022-10-26 11:23:18] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 11:23:18] ...trap/engines/engine.py.326: process: 0.28756 | load: 0.00015 [I 2022-10-26 11:23:21] ...trap/engines/engine.py.321: eval: epoch 4 | batch 410/510 [I 2022-10-26 11:23:21] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:28 [I 2022-10-26 11:23:21] ...trap/engines/engine.py.326: process: 0.28971 | load: 0.00016 [I 2022-10-26 11:23:24] ...trap/engines/engine.py.321: eval: epoch 4 | batch 420/510 [I 2022-10-26 11:23:24] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-26 11:23:24] ...trap/engines/engine.py.326: process: 0.28530 | load: 0.00016 [I 2022-10-26 11:23:27] ...trap/engines/engine.py.321: eval: epoch 4 | batch 430/510 [I 2022-10-26 11:23:27] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:22 [I 2022-10-26 11:23:27] ...trap/engines/engine.py.326: process: 0.28415 | load: 0.00019 [I 2022-10-26 11:23:30] ...trap/engines/engine.py.321: eval: epoch 4 | batch 440/510 [I 2022-10-26 11:23:30] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 11:23:30] ...trap/engines/engine.py.326: process: 0.28821 | load: 0.00022 [I 2022-10-26 11:23:32] ...trap/engines/engine.py.321: eval: epoch 4 | batch 450/510 [I 2022-10-26 11:23:32] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 11:23:32] ...trap/engines/engine.py.326: process: 0.28813 | load: 0.00019 [I 2022-10-26 11:23:35] ...trap/engines/engine.py.321: eval: epoch 4 | batch 460/510 [I 2022-10-26 11:23:35] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 11:23:35] ...trap/engines/engine.py.326: process: 0.28259 | load: 0.00015 [I 2022-10-26 11:23:38] ...trap/engines/engine.py.321: eval: epoch 4 | batch 470/510 [I 2022-10-26 11:23:38] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 11:23:38] ...trap/engines/engine.py.326: process: 0.28503 | load: 0.00016 [I 2022-10-26 11:23:41] ...trap/engines/engine.py.321: eval: epoch 4 | batch 480/510 [I 2022-10-26 11:23:41] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 11:23:41] ...trap/engines/engine.py.326: process: 0.27797 | load: 0.00018 [I 2022-10-26 11:23:44] ...trap/engines/engine.py.321: eval: epoch 4 | batch 490/510 [I 2022-10-26 11:23:44] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:05 [I 2022-10-26 11:23:44] ...trap/engines/engine.py.326: process: 0.28706 | load: 0.00017 [I 2022-10-26 11:23:47] ...trap/engines/engine.py.321: eval: epoch 4 | batch 500/510 [I 2022-10-26 11:23:47] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-26 11:23:47] ...trap/engines/engine.py.326: process: 0.28349 | load: 0.00015 [I 2022-10-26 11:23:50] ...trap/engines/engine.py.321: eval: epoch 4 | batch 510/510 [I 2022-10-26 11:23:50] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 11:23:50] ...trap/engines/engine.py.326: process: 0.28445 | load: 0.00010 [S 2022-10-26 11:23:50] ...trap/engines/engine.py.132: eval_epoch.epoch: 4 [I 2022-10-26 11:23:50] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.00 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.424100 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.735500 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.833900 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.437300 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.750500 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.840200 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013575 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012746 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009321 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013327 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011758 [S 2022-10-26 11:23:55] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.011016 [I 2022-10-26 11:23:55] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 11:23:55] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 11:23:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 11:23:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 11:23:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 11:23:55] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 11:23:58] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 11:24:03] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 11:24:03] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 11:24:03] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 11:24:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 11:24:11] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 11:24:11] ...trap/engines/engine.py.163: Training model on trainset for epoch 5 [I 2022-10-26 11:24:11] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 11:24:14] ...trap/engines/engine.py.224: train: epoch 5 | batch 0/2304 [I 2022-10-26 11:24:14] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:28 [I 2022-10-26 11:24:14] ...trap/engines/engine.py.229: process: 2.77264 | load: 2.20502 [I 2022-10-26 11:24:14] ...trap/engines/engine.py.230: loss: 0.28180 [I 2022-10-26 11:24:14] ...optimizers/trijoint.py.151: total_norm: 3.755747 [I 2022-10-26 11:24:19] ...trap/engines/engine.py.224: train: epoch 5 | batch 10/2304 [I 2022-10-26 11:24:19] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:21 [I 2022-10-26 11:24:19] ...trap/engines/engine.py.229: process: 0.49536 | load: 0.00510 [I 2022-10-26 11:24:19] ...trap/engines/engine.py.230: loss: 0.32137 [I 2022-10-26 11:24:19] ...optimizers/trijoint.py.151: total_norm: 4.823385 [I 2022-10-26 11:24:24] ...trap/engines/engine.py.224: train: epoch 5 | batch 20/2304 [I 2022-10-26 11:24:24] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:00 [I 2022-10-26 11:24:24] ...trap/engines/engine.py.229: process: 0.50167 | load: 0.00327 [I 2022-10-26 11:24:24] ...trap/engines/engine.py.230: loss: 0.21375 [I 2022-10-26 11:24:24] ...optimizers/trijoint.py.151: total_norm: 3.799538 [I 2022-10-26 11:24:29] ...trap/engines/engine.py.224: train: epoch 5 | batch 30/2304 [I 2022-10-26 11:24:29] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:02 [I 2022-10-26 11:24:29] ...trap/engines/engine.py.229: process: 0.49032 | load: 0.00292 [I 2022-10-26 11:24:29] ...trap/engines/engine.py.230: loss: 0.22712 [I 2022-10-26 11:24:29] ...optimizers/trijoint.py.151: total_norm: 3.568820 [I 2022-10-26 11:24:34] ...trap/engines/engine.py.224: train: epoch 5 | batch 40/2304 [I 2022-10-26 11:24:34] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:43 [I 2022-10-26 11:24:34] ...trap/engines/engine.py.229: process: 0.49272 | load: 0.00314 [I 2022-10-26 11:24:34] ...trap/engines/engine.py.230: loss: 0.20656 [I 2022-10-26 11:24:34] ...optimizers/trijoint.py.151: total_norm: 3.061484 [I 2022-10-26 11:24:39] ...trap/engines/engine.py.224: train: epoch 5 | batch 50/2304 [I 2022-10-26 11:24:39] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:37 [I 2022-10-26 11:24:39] ...trap/engines/engine.py.229: process: 0.49471 | load: 0.00278 [I 2022-10-26 11:24:39] ...trap/engines/engine.py.230: loss: 0.21909 [I 2022-10-26 11:24:39] ...optimizers/trijoint.py.151: total_norm: 3.306192 [I 2022-10-26 11:24:44] ...trap/engines/engine.py.224: train: epoch 5 | batch 60/2304 [I 2022-10-26 11:24:44] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:40 [I 2022-10-26 11:24:44] ...trap/engines/engine.py.229: process: 0.48899 | load: 0.00282 [I 2022-10-26 11:24:44] ...trap/engines/engine.py.230: loss: 0.26730 [I 2022-10-26 11:24:44] ...optimizers/trijoint.py.151: total_norm: 3.185157 [I 2022-10-26 11:24:49] ...trap/engines/engine.py.224: train: epoch 5 | batch 70/2304 [I 2022-10-26 11:24:49] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:34 [I 2022-10-26 11:24:49] ...trap/engines/engine.py.229: process: 0.50374 | load: 0.00296 [I 2022-10-26 11:24:49] ...trap/engines/engine.py.230: loss: 0.21270 [I 2022-10-26 11:24:49] ...optimizers/trijoint.py.151: total_norm: 3.216686 [I 2022-10-26 11:24:54] ...trap/engines/engine.py.224: train: epoch 5 | batch 80/2304 [I 2022-10-26 11:24:54] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:27 [I 2022-10-26 11:24:54] ...trap/engines/engine.py.229: process: 0.49603 | load: 0.00304 [I 2022-10-26 11:24:54] ...trap/engines/engine.py.230: loss: 0.23182 [I 2022-10-26 11:24:54] ...optimizers/trijoint.py.151: total_norm: 3.797624 [I 2022-10-26 11:24:59] ...trap/engines/engine.py.224: train: epoch 5 | batch 90/2304 [I 2022-10-26 11:24:59] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:18:20 [I 2022-10-26 11:24:59] ...trap/engines/engine.py.229: process: 0.50126 | load: 0.00258 [I 2022-10-26 11:24:59] ...trap/engines/engine.py.230: loss: 0.20934 [I 2022-10-26 11:24:59] ...optimizers/trijoint.py.151: total_norm: 2.861101 [I 2022-10-26 11:25:04] ...trap/engines/engine.py.224: train: epoch 5 | batch 100/2304 [I 2022-10-26 11:25:04] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:18:40 [I 2022-10-26 11:25:04] ...trap/engines/engine.py.229: process: 0.52670 | load: 0.00275 [I 2022-10-26 11:25:04] ...trap/engines/engine.py.230: loss: 0.24372 [I 2022-10-26 11:25:04] ...optimizers/trijoint.py.151: total_norm: 3.392499 [I 2022-10-26 11:25:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 110/2304 [I 2022-10-26 11:25:09] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:18:21 [I 2022-10-26 11:25:09] ...trap/engines/engine.py.229: process: 0.51067 | load: 0.00276 [I 2022-10-26 11:25:09] ...trap/engines/engine.py.230: loss: 0.26330 [I 2022-10-26 11:25:09] ...optimizers/trijoint.py.151: total_norm: 3.122722 [I 2022-10-26 11:25:14] ...trap/engines/engine.py.224: train: epoch 5 | batch 120/2304 [I 2022-10-26 11:25:14] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:18:08 [I 2022-10-26 11:25:14] ...trap/engines/engine.py.229: process: 0.49716 | load: 0.00247 [I 2022-10-26 11:25:14] ...trap/engines/engine.py.230: loss: 0.22977 [I 2022-10-26 11:25:14] ...optimizers/trijoint.py.151: total_norm: 2.839210 [I 2022-10-26 11:25:19] ...trap/engines/engine.py.224: train: epoch 5 | batch 130/2304 [I 2022-10-26 11:25:19] ...trap/engines/engine.py.228: elapsed: 0:01:07 | left: 0:18:21 [I 2022-10-26 11:25:19] ...trap/engines/engine.py.229: process: 0.49104 | load: 0.00243 [I 2022-10-26 11:25:19] ...trap/engines/engine.py.230: loss: 0.35162 [I 2022-10-26 11:25:19] ...optimizers/trijoint.py.151: total_norm: 3.900760 [I 2022-10-26 11:25:24] ...trap/engines/engine.py.224: train: epoch 5 | batch 140/2304 [I 2022-10-26 11:25:24] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:18:12 [I 2022-10-26 11:25:24] ...trap/engines/engine.py.229: process: 0.50758 | load: 0.00278 [I 2022-10-26 11:25:24] ...trap/engines/engine.py.230: loss: 0.33134 [I 2022-10-26 11:25:24] ...optimizers/trijoint.py.151: total_norm: 4.096757 [I 2022-10-26 11:25:29] ...trap/engines/engine.py.224: train: epoch 5 | batch 150/2304 [I 2022-10-26 11:25:29] ...trap/engines/engine.py.228: elapsed: 0:01:17 | left: 0:17:47 [I 2022-10-26 11:25:29] ...trap/engines/engine.py.229: process: 0.47900 | load: 0.00251 [I 2022-10-26 11:25:29] ...trap/engines/engine.py.230: loss: 0.25677 [I 2022-10-26 11:25:29] ...optimizers/trijoint.py.151: total_norm: 3.497740 [I 2022-10-26 11:25:34] ...trap/engines/engine.py.224: train: epoch 5 | batch 160/2304 [I 2022-10-26 11:25:34] ...trap/engines/engine.py.228: elapsed: 0:01:22 | left: 0:17:48 [I 2022-10-26 11:25:34] ...trap/engines/engine.py.229: process: 0.51593 | load: 0.00259 [I 2022-10-26 11:25:34] ...trap/engines/engine.py.230: loss: 0.28759 [I 2022-10-26 11:25:34] ...optimizers/trijoint.py.151: total_norm: 3.642903 [I 2022-10-26 11:25:39] ...trap/engines/engine.py.224: train: epoch 5 | batch 170/2304 [I 2022-10-26 11:25:39] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:17:40 [I 2022-10-26 11:25:39] ...trap/engines/engine.py.229: process: 0.50516 | load: 0.00262 [I 2022-10-26 11:25:39] ...trap/engines/engine.py.230: loss: 0.27099 [I 2022-10-26 11:25:39] ...optimizers/trijoint.py.151: total_norm: 4.038581 [I 2022-10-26 11:25:44] ...trap/engines/engine.py.224: train: epoch 5 | batch 180/2304 [I 2022-10-26 11:25:44] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:17:43 [I 2022-10-26 11:25:44] ...trap/engines/engine.py.229: process: 0.50817 | load: 0.00265 [I 2022-10-26 11:25:44] ...trap/engines/engine.py.230: loss: 0.29111 [I 2022-10-26 11:25:44] ...optimizers/trijoint.py.151: total_norm: 3.622817 [I 2022-10-26 11:25:49] ...trap/engines/engine.py.224: train: epoch 5 | batch 190/2304 [I 2022-10-26 11:25:49] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:17:29 [I 2022-10-26 11:25:49] ...trap/engines/engine.py.229: process: 0.49105 | load: 0.00267 [I 2022-10-26 11:25:49] ...trap/engines/engine.py.230: loss: 0.26819 [I 2022-10-26 11:25:49] ...optimizers/trijoint.py.151: total_norm: 3.545894 [I 2022-10-26 11:25:54] ...trap/engines/engine.py.224: train: epoch 5 | batch 200/2304 [I 2022-10-26 11:25:54] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:58 [I 2022-10-26 11:25:54] ...trap/engines/engine.py.229: process: 0.50216 | load: 0.00257 [I 2022-10-26 11:25:54] ...trap/engines/engine.py.230: loss: 0.27866 [I 2022-10-26 11:25:54] ...optimizers/trijoint.py.151: total_norm: 3.647770 [I 2022-10-26 11:25:59] ...trap/engines/engine.py.224: train: epoch 5 | batch 210/2304 [I 2022-10-26 11:25:59] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:21 [I 2022-10-26 11:25:59] ...trap/engines/engine.py.229: process: 0.49378 | load: 0.00245 [I 2022-10-26 11:25:59] ...trap/engines/engine.py.230: loss: 0.22245 [I 2022-10-26 11:25:59] ...optimizers/trijoint.py.151: total_norm: 3.032911 [I 2022-10-26 11:26:04] ...trap/engines/engine.py.224: train: epoch 5 | batch 220/2304 [I 2022-10-26 11:26:04] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:12 [I 2022-10-26 11:26:04] ...trap/engines/engine.py.229: process: 0.49212 | load: 0.00251 [I 2022-10-26 11:26:04] ...trap/engines/engine.py.230: loss: 0.28856 [I 2022-10-26 11:26:04] ...optimizers/trijoint.py.151: total_norm: 3.680014 [I 2022-10-26 11:26:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 230/2304 [I 2022-10-26 11:26:09] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:14 [I 2022-10-26 11:26:09] ...trap/engines/engine.py.229: process: 0.50365 | load: 0.00248 [I 2022-10-26 11:26:09] ...trap/engines/engine.py.230: loss: 0.33785 [I 2022-10-26 11:26:09] ...optimizers/trijoint.py.151: total_norm: 3.593427 [I 2022-10-26 11:26:14] ...trap/engines/engine.py.224: train: epoch 5 | batch 240/2304 [I 2022-10-26 11:26:14] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:11 [I 2022-10-26 11:26:14] ...trap/engines/engine.py.229: process: 0.50715 | load: 0.00253 [I 2022-10-26 11:26:14] ...trap/engines/engine.py.230: loss: 0.30615 [I 2022-10-26 11:26:14] ...optimizers/trijoint.py.151: total_norm: 3.779968 [I 2022-10-26 11:26:19] ...trap/engines/engine.py.224: train: epoch 5 | batch 250/2304 [I 2022-10-26 11:26:19] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:23 [I 2022-10-26 11:26:19] ...trap/engines/engine.py.229: process: 0.50923 | load: 0.00246 [I 2022-10-26 11:26:19] ...trap/engines/engine.py.230: loss: 0.27614 [I 2022-10-26 11:26:19] ...optimizers/trijoint.py.151: total_norm: 3.011224 [I 2022-10-26 11:26:24] ...trap/engines/engine.py.224: train: epoch 5 | batch 260/2304 [I 2022-10-26 11:26:24] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:12 [I 2022-10-26 11:26:24] ...trap/engines/engine.py.229: process: 0.49660 | load: 0.00247 [I 2022-10-26 11:26:24] ...trap/engines/engine.py.230: loss: 0.23522 [I 2022-10-26 11:26:24] ...optimizers/trijoint.py.151: total_norm: 2.784970 [I 2022-10-26 11:26:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 270/2304 [I 2022-10-26 11:26:30] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:17:27 [I 2022-10-26 11:26:30] ...trap/engines/engine.py.229: process: 0.50112 | load: 0.00254 [I 2022-10-26 11:26:30] ...trap/engines/engine.py.230: loss: 0.25908 [I 2022-10-26 11:26:30] ...optimizers/trijoint.py.151: total_norm: 4.180307 [I 2022-10-26 11:26:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 280/2304 [I 2022-10-26 11:26:35] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:16:50 [I 2022-10-26 11:26:35] ...trap/engines/engine.py.229: process: 0.49519 | load: 0.00247 [I 2022-10-26 11:26:35] ...trap/engines/engine.py.230: loss: 0.19536 [I 2022-10-26 11:26:35] ...optimizers/trijoint.py.151: total_norm: 3.124995 [I 2022-10-26 11:26:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 290/2304 [I 2022-10-26 11:26:40] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:16:45 [I 2022-10-26 11:26:40] ...trap/engines/engine.py.229: process: 0.49886 | load: 0.00254 [I 2022-10-26 11:26:40] ...trap/engines/engine.py.230: loss: 0.34643 [I 2022-10-26 11:26:40] ...optimizers/trijoint.py.151: total_norm: 3.399101 [I 2022-10-26 11:26:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 300/2304 [I 2022-10-26 11:26:45] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:16:45 [I 2022-10-26 11:26:45] ...trap/engines/engine.py.229: process: 0.52002 | load: 0.00247 [I 2022-10-26 11:26:45] ...trap/engines/engine.py.230: loss: 0.23598 [I 2022-10-26 11:26:45] ...optimizers/trijoint.py.151: total_norm: 3.769417 [I 2022-10-26 11:26:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 310/2304 [I 2022-10-26 11:26:50] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:16:38 [I 2022-10-26 11:26:50] ...trap/engines/engine.py.229: process: 0.50523 | load: 0.00252 [I 2022-10-26 11:26:50] ...trap/engines/engine.py.230: loss: 0.28746 [I 2022-10-26 11:26:50] ...optimizers/trijoint.py.151: total_norm: 4.412879 [I 2022-10-26 11:26:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 320/2304 [I 2022-10-26 11:26:55] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:16:29 [I 2022-10-26 11:26:55] ...trap/engines/engine.py.229: process: 0.50061 | load: 0.00245 [I 2022-10-26 11:26:55] ...trap/engines/engine.py.230: loss: 0.35696 [I 2022-10-26 11:26:55] ...optimizers/trijoint.py.151: total_norm: 4.864490 [I 2022-10-26 11:27:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 330/2304 [I 2022-10-26 11:27:00] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:16:23 [I 2022-10-26 11:27:00] ...trap/engines/engine.py.229: process: 0.49641 | load: 0.00269 [I 2022-10-26 11:27:00] ...trap/engines/engine.py.230: loss: 0.34106 [I 2022-10-26 11:27:00] ...optimizers/trijoint.py.151: total_norm: 3.727069 [I 2022-10-26 11:27:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 340/2304 [I 2022-10-26 11:27:05] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:16:19 [I 2022-10-26 11:27:05] ...trap/engines/engine.py.229: process: 0.49846 | load: 0.00256 [I 2022-10-26 11:27:05] ...trap/engines/engine.py.230: loss: 0.29617 [I 2022-10-26 11:27:05] ...optimizers/trijoint.py.151: total_norm: 3.983997 [I 2022-10-26 11:27:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 350/2304 [I 2022-10-26 11:27:10] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:16:19 [I 2022-10-26 11:27:10] ...trap/engines/engine.py.229: process: 0.48678 | load: 0.00246 [I 2022-10-26 11:27:10] ...trap/engines/engine.py.230: loss: 0.26628 [I 2022-10-26 11:27:10] ...optimizers/trijoint.py.151: total_norm: 3.620519 [I 2022-10-26 11:27:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 360/2304 [I 2022-10-26 11:27:15] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:16:09 [I 2022-10-26 11:27:15] ...trap/engines/engine.py.229: process: 0.50369 | load: 0.00249 [I 2022-10-26 11:27:15] ...trap/engines/engine.py.230: loss: 0.28789 [I 2022-10-26 11:27:15] ...optimizers/trijoint.py.151: total_norm: 4.052470 [I 2022-10-26 11:27:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 370/2304 [I 2022-10-26 11:27:20] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:15:58 [I 2022-10-26 11:27:20] ...trap/engines/engine.py.229: process: 0.50103 | load: 0.00253 [I 2022-10-26 11:27:20] ...trap/engines/engine.py.230: loss: 0.28702 [I 2022-10-26 11:27:20] ...optimizers/trijoint.py.151: total_norm: 3.558577 [I 2022-10-26 11:27:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 380/2304 [I 2022-10-26 11:27:25] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:16:07 [I 2022-10-26 11:27:25] ...trap/engines/engine.py.229: process: 0.52626 | load: 0.00242 [I 2022-10-26 11:27:25] ...trap/engines/engine.py.230: loss: 0.34344 [I 2022-10-26 11:27:25] ...optimizers/trijoint.py.151: total_norm: 5.033873 [I 2022-10-26 11:27:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 390/2304 [I 2022-10-26 11:27:30] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:15:58 [I 2022-10-26 11:27:30] ...trap/engines/engine.py.229: process: 0.49860 | load: 0.00255 [I 2022-10-26 11:27:30] ...trap/engines/engine.py.230: loss: 0.25340 [I 2022-10-26 11:27:30] ...optimizers/trijoint.py.151: total_norm: 4.911703 [I 2022-10-26 11:27:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 400/2304 [I 2022-10-26 11:27:35] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:16:22 [I 2022-10-26 11:27:35] ...trap/engines/engine.py.229: process: 0.51703 | load: 0.00270 [I 2022-10-26 11:27:35] ...trap/engines/engine.py.230: loss: 0.29057 [I 2022-10-26 11:27:35] ...optimizers/trijoint.py.151: total_norm: 3.643254 [I 2022-10-26 11:27:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 410/2304 [I 2022-10-26 11:27:40] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:15:43 [I 2022-10-26 11:27:40] ...trap/engines/engine.py.229: process: 0.48642 | load: 0.00252 [I 2022-10-26 11:27:40] ...trap/engines/engine.py.230: loss: 0.27516 [I 2022-10-26 11:27:40] ...optimizers/trijoint.py.151: total_norm: 4.167837 [I 2022-10-26 11:27:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 420/2304 [I 2022-10-26 11:27:45] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:15:28 [I 2022-10-26 11:27:45] ...trap/engines/engine.py.229: process: 0.48684 | load: 0.00250 [I 2022-10-26 11:27:45] ...trap/engines/engine.py.230: loss: 0.19229 [I 2022-10-26 11:27:45] ...optimizers/trijoint.py.151: total_norm: 3.515428 [I 2022-10-26 11:27:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 430/2304 [I 2022-10-26 11:27:50] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:15:24 [I 2022-10-26 11:27:50] ...trap/engines/engine.py.229: process: 0.49563 | load: 0.00251 [I 2022-10-26 11:27:50] ...trap/engines/engine.py.230: loss: 0.23412 [I 2022-10-26 11:27:50] ...optimizers/trijoint.py.151: total_norm: 3.679183 [I 2022-10-26 11:27:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 440/2304 [I 2022-10-26 11:27:55] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:15:38 [I 2022-10-26 11:27:55] ...trap/engines/engine.py.229: process: 0.49945 | load: 0.00247 [I 2022-10-26 11:27:55] ...trap/engines/engine.py.230: loss: 0.40182 [I 2022-10-26 11:27:55] ...optimizers/trijoint.py.151: total_norm: 3.936336 [I 2022-10-26 11:28:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 450/2304 [I 2022-10-26 11:28:00] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:15:25 [I 2022-10-26 11:28:00] ...trap/engines/engine.py.229: process: 0.48901 | load: 0.00248 [I 2022-10-26 11:28:00] ...trap/engines/engine.py.230: loss: 0.22648 [I 2022-10-26 11:28:00] ...optimizers/trijoint.py.151: total_norm: 3.561379 [I 2022-10-26 11:28:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 460/2304 [I 2022-10-26 11:28:05] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:15:17 [I 2022-10-26 11:28:05] ...trap/engines/engine.py.229: process: 0.49107 | load: 0.00251 [I 2022-10-26 11:28:05] ...trap/engines/engine.py.230: loss: 0.30174 [I 2022-10-26 11:28:05] ...optimizers/trijoint.py.151: total_norm: 3.516040 [I 2022-10-26 11:28:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 470/2304 [I 2022-10-26 11:28:10] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:15:16 [I 2022-10-26 11:28:10] ...trap/engines/engine.py.229: process: 0.50561 | load: 0.00246 [I 2022-10-26 11:28:10] ...trap/engines/engine.py.230: loss: 0.26844 [I 2022-10-26 11:28:10] ...optimizers/trijoint.py.151: total_norm: 4.315770 [I 2022-10-26 11:28:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 480/2304 [I 2022-10-26 11:28:15] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:15:09 [I 2022-10-26 11:28:15] ...trap/engines/engine.py.229: process: 0.50140 | load: 0.00252 [I 2022-10-26 11:28:15] ...trap/engines/engine.py.230: loss: 0.23637 [I 2022-10-26 11:28:15] ...optimizers/trijoint.py.151: total_norm: 3.668499 [I 2022-10-26 11:28:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 490/2304 [I 2022-10-26 11:28:20] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:15:06 [I 2022-10-26 11:28:20] ...trap/engines/engine.py.229: process: 0.49735 | load: 0.00247 [I 2022-10-26 11:28:20] ...trap/engines/engine.py.230: loss: 0.35892 [I 2022-10-26 11:28:20] ...optimizers/trijoint.py.151: total_norm: 4.628178 [I 2022-10-26 11:28:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 500/2304 [I 2022-10-26 11:28:25] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:15:00 [I 2022-10-26 11:28:25] ...trap/engines/engine.py.229: process: 0.48885 | load: 0.00281 [I 2022-10-26 11:28:25] ...trap/engines/engine.py.230: loss: 0.24024 [I 2022-10-26 11:28:25] ...optimizers/trijoint.py.151: total_norm: 3.882194 [I 2022-10-26 11:28:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 510/2304 [I 2022-10-26 11:28:30] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:14:56 [I 2022-10-26 11:28:30] ...trap/engines/engine.py.229: process: 0.49390 | load: 0.00245 [I 2022-10-26 11:28:30] ...trap/engines/engine.py.230: loss: 0.21357 [I 2022-10-26 11:28:30] ...optimizers/trijoint.py.151: total_norm: 3.091514 [I 2022-10-26 11:28:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 520/2304 [I 2022-10-26 11:28:35] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:14:55 [I 2022-10-26 11:28:35] ...trap/engines/engine.py.229: process: 0.50264 | load: 0.00263 [I 2022-10-26 11:28:35] ...trap/engines/engine.py.230: loss: 0.30360 [I 2022-10-26 11:28:35] ...optimizers/trijoint.py.151: total_norm: 3.633320 [I 2022-10-26 11:28:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 530/2304 [I 2022-10-26 11:28:40] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:15:37 [I 2022-10-26 11:28:40] ...trap/engines/engine.py.229: process: 0.58976 | load: 0.00275 [I 2022-10-26 11:28:40] ...trap/engines/engine.py.230: loss: 0.23936 [I 2022-10-26 11:28:40] ...optimizers/trijoint.py.151: total_norm: 2.955057 [I 2022-10-26 11:28:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 540/2304 [I 2022-10-26 11:28:45] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:14:36 [I 2022-10-26 11:28:45] ...trap/engines/engine.py.229: process: 0.49027 | load: 0.00245 [I 2022-10-26 11:28:45] ...trap/engines/engine.py.230: loss: 0.24783 [I 2022-10-26 11:28:45] ...optimizers/trijoint.py.151: total_norm: 3.761054 [I 2022-10-26 11:28:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 550/2304 [I 2022-10-26 11:28:50] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:14:25 [I 2022-10-26 11:28:50] ...trap/engines/engine.py.229: process: 0.49925 | load: 0.00250 [I 2022-10-26 11:28:50] ...trap/engines/engine.py.230: loss: 0.28411 [I 2022-10-26 11:28:50] ...optimizers/trijoint.py.151: total_norm: 4.761456 [I 2022-10-26 11:28:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 560/2304 [I 2022-10-26 11:28:55] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:14:20 [I 2022-10-26 11:28:55] ...trap/engines/engine.py.229: process: 0.48388 | load: 0.00245 [I 2022-10-26 11:28:55] ...trap/engines/engine.py.230: loss: 0.27736 [I 2022-10-26 11:28:55] ...optimizers/trijoint.py.151: total_norm: 3.690775 [I 2022-10-26 11:29:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 570/2304 [I 2022-10-26 11:29:00] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:14:19 [I 2022-10-26 11:29:00] ...trap/engines/engine.py.229: process: 0.49436 | load: 0.00248 [I 2022-10-26 11:29:00] ...trap/engines/engine.py.230: loss: 0.24607 [I 2022-10-26 11:29:00] ...optimizers/trijoint.py.151: total_norm: 3.233850 [I 2022-10-26 11:29:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 580/2304 [I 2022-10-26 11:29:05] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:14:21 [I 2022-10-26 11:29:05] ...trap/engines/engine.py.229: process: 0.50297 | load: 0.00242 [I 2022-10-26 11:29:05] ...trap/engines/engine.py.230: loss: 0.20425 [I 2022-10-26 11:29:05] ...optimizers/trijoint.py.151: total_norm: 4.172549 [I 2022-10-26 11:29:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 590/2304 [I 2022-10-26 11:29:10] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:14:20 [I 2022-10-26 11:29:10] ...trap/engines/engine.py.229: process: 0.50420 | load: 0.00245 [I 2022-10-26 11:29:10] ...trap/engines/engine.py.230: loss: 0.34157 [I 2022-10-26 11:29:10] ...optimizers/trijoint.py.151: total_norm: 3.938782 [I 2022-10-26 11:29:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 600/2304 [I 2022-10-26 11:29:15] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:14:12 [I 2022-10-26 11:29:15] ...trap/engines/engine.py.229: process: 0.49827 | load: 0.00241 [I 2022-10-26 11:29:15] ...trap/engines/engine.py.230: loss: 0.30937 [I 2022-10-26 11:29:15] ...optimizers/trijoint.py.151: total_norm: 4.500381 [I 2022-10-26 11:29:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 610/2304 [I 2022-10-26 11:29:20] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:14:13 [I 2022-10-26 11:29:20] ...trap/engines/engine.py.229: process: 0.50312 | load: 0.00243 [I 2022-10-26 11:29:20] ...trap/engines/engine.py.230: loss: 0.27999 [I 2022-10-26 11:29:20] ...optimizers/trijoint.py.151: total_norm: 3.625842 [I 2022-10-26 11:29:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 620/2304 [I 2022-10-26 11:29:25] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:13:57 [I 2022-10-26 11:29:25] ...trap/engines/engine.py.229: process: 0.49259 | load: 0.00242 [I 2022-10-26 11:29:25] ...trap/engines/engine.py.230: loss: 0.18255 [I 2022-10-26 11:29:25] ...optimizers/trijoint.py.151: total_norm: 3.565203 [I 2022-10-26 11:29:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 630/2304 [I 2022-10-26 11:29:30] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:13:53 [I 2022-10-26 11:29:30] ...trap/engines/engine.py.229: process: 0.49403 | load: 0.00244 [I 2022-10-26 11:29:30] ...trap/engines/engine.py.230: loss: 0.17306 [I 2022-10-26 11:29:30] ...optimizers/trijoint.py.151: total_norm: 2.650575 [I 2022-10-26 11:29:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 640/2304 [I 2022-10-26 11:29:35] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:13:51 [I 2022-10-26 11:29:35] ...trap/engines/engine.py.229: process: 0.50035 | load: 0.00252 [I 2022-10-26 11:29:35] ...trap/engines/engine.py.230: loss: 0.24741 [I 2022-10-26 11:29:35] ...optimizers/trijoint.py.151: total_norm: 4.058597 [I 2022-10-26 11:29:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 650/2304 [I 2022-10-26 11:29:40] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:13:37 [I 2022-10-26 11:29:40] ...trap/engines/engine.py.229: process: 0.49504 | load: 0.00246 [I 2022-10-26 11:29:40] ...trap/engines/engine.py.230: loss: 0.23052 [I 2022-10-26 11:29:40] ...optimizers/trijoint.py.151: total_norm: 3.387769 [I 2022-10-26 11:29:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 660/2304 [I 2022-10-26 11:29:45] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:14:02 [I 2022-10-26 11:29:45] ...trap/engines/engine.py.229: process: 0.49500 | load: 0.00264 [I 2022-10-26 11:29:45] ...trap/engines/engine.py.230: loss: 0.22905 [I 2022-10-26 11:29:45] ...optimizers/trijoint.py.151: total_norm: 3.462395 [I 2022-10-26 11:29:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 670/2304 [I 2022-10-26 11:29:50] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:13:39 [I 2022-10-26 11:29:50] ...trap/engines/engine.py.229: process: 0.52311 | load: 0.00246 [I 2022-10-26 11:29:50] ...trap/engines/engine.py.230: loss: 0.16762 [I 2022-10-26 11:29:50] ...optimizers/trijoint.py.151: total_norm: 3.178137 [I 2022-10-26 11:29:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 680/2304 [I 2022-10-26 11:29:55] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:13:33 [I 2022-10-26 11:29:55] ...trap/engines/engine.py.229: process: 0.50037 | load: 0.00252 [I 2022-10-26 11:29:55] ...trap/engines/engine.py.230: loss: 0.34665 [I 2022-10-26 11:29:55] ...optimizers/trijoint.py.151: total_norm: 4.719666 [I 2022-10-26 11:30:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 690/2304 [I 2022-10-26 11:30:00] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:13:26 [I 2022-10-26 11:30:00] ...trap/engines/engine.py.229: process: 0.50002 | load: 0.00249 [I 2022-10-26 11:30:00] ...trap/engines/engine.py.230: loss: 0.28226 [I 2022-10-26 11:30:00] ...optimizers/trijoint.py.151: total_norm: 3.335311 [I 2022-10-26 11:30:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 700/2304 [I 2022-10-26 11:30:05] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:13:21 [I 2022-10-26 11:30:05] ...trap/engines/engine.py.229: process: 0.50005 | load: 0.00245 [I 2022-10-26 11:30:05] ...trap/engines/engine.py.230: loss: 0.28721 [I 2022-10-26 11:30:05] ...optimizers/trijoint.py.151: total_norm: 3.692451 [I 2022-10-26 11:30:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 710/2304 [I 2022-10-26 11:30:10] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:13:14 [I 2022-10-26 11:30:10] ...trap/engines/engine.py.229: process: 0.50494 | load: 0.00243 [I 2022-10-26 11:30:10] ...trap/engines/engine.py.230: loss: 0.30859 [I 2022-10-26 11:30:10] ...optimizers/trijoint.py.151: total_norm: 3.814741 [I 2022-10-26 11:30:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 720/2304 [I 2022-10-26 11:30:15] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:13:05 [I 2022-10-26 11:30:15] ...trap/engines/engine.py.229: process: 0.49159 | load: 0.00256 [I 2022-10-26 11:30:15] ...trap/engines/engine.py.230: loss: 0.23943 [I 2022-10-26 11:30:15] ...optimizers/trijoint.py.151: total_norm: 4.462694 [I 2022-10-26 11:30:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 730/2304 [I 2022-10-26 11:30:20] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:13:03 [I 2022-10-26 11:30:20] ...trap/engines/engine.py.229: process: 0.49790 | load: 0.00252 [I 2022-10-26 11:30:20] ...trap/engines/engine.py.230: loss: 0.32462 [I 2022-10-26 11:30:20] ...optimizers/trijoint.py.151: total_norm: 3.792692 [I 2022-10-26 11:30:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 740/2304 [I 2022-10-26 11:30:25] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:12:56 [I 2022-10-26 11:30:25] ...trap/engines/engine.py.229: process: 0.50072 | load: 0.00242 [I 2022-10-26 11:30:25] ...trap/engines/engine.py.230: loss: 0.32237 [I 2022-10-26 11:30:25] ...optimizers/trijoint.py.151: total_norm: 3.838084 [I 2022-10-26 11:30:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 750/2304 [I 2022-10-26 11:30:30] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:12:52 [I 2022-10-26 11:30:30] ...trap/engines/engine.py.229: process: 0.49671 | load: 0.00245 [I 2022-10-26 11:30:30] ...trap/engines/engine.py.230: loss: 0.25514 [I 2022-10-26 11:30:30] ...optimizers/trijoint.py.151: total_norm: 3.064433 [I 2022-10-26 11:30:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 760/2304 [I 2022-10-26 11:30:35] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:12:45 [I 2022-10-26 11:30:35] ...trap/engines/engine.py.229: process: 0.48663 | load: 0.00245 [I 2022-10-26 11:30:35] ...trap/engines/engine.py.230: loss: 0.30756 [I 2022-10-26 11:30:35] ...optimizers/trijoint.py.151: total_norm: 3.418147 [I 2022-10-26 11:30:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 770/2304 [I 2022-10-26 11:30:40] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:12:47 [I 2022-10-26 11:30:40] ...trap/engines/engine.py.229: process: 0.52041 | load: 0.00241 [I 2022-10-26 11:30:40] ...trap/engines/engine.py.230: loss: 0.26224 [I 2022-10-26 11:30:40] ...optimizers/trijoint.py.151: total_norm: 3.744776 [I 2022-10-26 11:30:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 780/2304 [I 2022-10-26 11:30:45] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:12:42 [I 2022-10-26 11:30:45] ...trap/engines/engine.py.229: process: 0.50125 | load: 0.00247 [I 2022-10-26 11:30:45] ...trap/engines/engine.py.230: loss: 0.26226 [I 2022-10-26 11:30:45] ...optimizers/trijoint.py.151: total_norm: 3.246289 [I 2022-10-26 11:30:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 790/2304 [I 2022-10-26 11:30:50] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:12:30 [I 2022-10-26 11:30:50] ...trap/engines/engine.py.229: process: 0.50249 | load: 0.00243 [I 2022-10-26 11:30:50] ...trap/engines/engine.py.230: loss: 0.26010 [I 2022-10-26 11:30:50] ...optimizers/trijoint.py.151: total_norm: 3.563407 [I 2022-10-26 11:30:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 800/2304 [I 2022-10-26 11:30:55] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:12:36 [I 2022-10-26 11:30:55] ...trap/engines/engine.py.229: process: 0.50408 | load: 0.00244 [I 2022-10-26 11:30:55] ...trap/engines/engine.py.230: loss: 0.26109 [I 2022-10-26 11:30:55] ...optimizers/trijoint.py.151: total_norm: 3.235312 [I 2022-10-26 11:31:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 810/2304 [I 2022-10-26 11:31:00] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:12:21 [I 2022-10-26 11:31:00] ...trap/engines/engine.py.229: process: 0.49080 | load: 0.00244 [I 2022-10-26 11:31:00] ...trap/engines/engine.py.230: loss: 0.25133 [I 2022-10-26 11:31:00] ...optimizers/trijoint.py.151: total_norm: 3.544682 [I 2022-10-26 11:31:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 820/2304 [I 2022-10-26 11:31:05] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:12:22 [I 2022-10-26 11:31:05] ...trap/engines/engine.py.229: process: 0.49793 | load: 0.00246 [I 2022-10-26 11:31:05] ...trap/engines/engine.py.230: loss: 0.26314 [I 2022-10-26 11:31:05] ...optimizers/trijoint.py.151: total_norm: 3.109204 [I 2022-10-26 11:31:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 830/2304 [I 2022-10-26 11:31:10] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:12:15 [I 2022-10-26 11:31:10] ...trap/engines/engine.py.229: process: 0.49176 | load: 0.00247 [I 2022-10-26 11:31:10] ...trap/engines/engine.py.230: loss: 0.32481 [I 2022-10-26 11:31:10] ...optimizers/trijoint.py.151: total_norm: 4.080914 [I 2022-10-26 11:31:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 840/2304 [I 2022-10-26 11:31:15] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:12:08 [I 2022-10-26 11:31:15] ...trap/engines/engine.py.229: process: 0.49157 | load: 0.00245 [I 2022-10-26 11:31:15] ...trap/engines/engine.py.230: loss: 0.24866 [I 2022-10-26 11:31:15] ...optimizers/trijoint.py.151: total_norm: 4.099703 [I 2022-10-26 11:31:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 850/2304 [I 2022-10-26 11:31:20] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:12:02 [I 2022-10-26 11:31:20] ...trap/engines/engine.py.229: process: 0.49934 | load: 0.00248 [I 2022-10-26 11:31:20] ...trap/engines/engine.py.230: loss: 0.20178 [I 2022-10-26 11:31:20] ...optimizers/trijoint.py.151: total_norm: 3.099889 [I 2022-10-26 11:31:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 860/2304 [I 2022-10-26 11:31:25] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:11:50 [I 2022-10-26 11:31:25] ...trap/engines/engine.py.229: process: 0.47981 | load: 0.00245 [I 2022-10-26 11:31:25] ...trap/engines/engine.py.230: loss: 0.27227 [I 2022-10-26 11:31:25] ...optimizers/trijoint.py.151: total_norm: 3.144595 [I 2022-10-26 11:31:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 870/2304 [I 2022-10-26 11:31:30] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:11:50 [I 2022-10-26 11:31:30] ...trap/engines/engine.py.229: process: 0.49454 | load: 0.00242 [I 2022-10-26 11:31:30] ...trap/engines/engine.py.230: loss: 0.19370 [I 2022-10-26 11:31:30] ...optimizers/trijoint.py.151: total_norm: 3.007309 [I 2022-10-26 11:31:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 880/2304 [I 2022-10-26 11:31:35] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:11:47 [I 2022-10-26 11:31:35] ...trap/engines/engine.py.229: process: 0.48936 | load: 0.00248 [I 2022-10-26 11:31:35] ...trap/engines/engine.py.230: loss: 0.21481 [I 2022-10-26 11:31:35] ...optimizers/trijoint.py.151: total_norm: 3.461344 [I 2022-10-26 11:31:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 890/2304 [I 2022-10-26 11:31:40] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:11:44 [I 2022-10-26 11:31:40] ...trap/engines/engine.py.229: process: 0.50060 | load: 0.00244 [I 2022-10-26 11:31:40] ...trap/engines/engine.py.230: loss: 0.24230 [I 2022-10-26 11:31:40] ...optimizers/trijoint.py.151: total_norm: 3.560712 [I 2022-10-26 11:31:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 900/2304 [I 2022-10-26 11:31:45] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:11:36 [I 2022-10-26 11:31:45] ...trap/engines/engine.py.229: process: 0.49928 | load: 0.00244 [I 2022-10-26 11:31:45] ...trap/engines/engine.py.230: loss: 0.33303 [I 2022-10-26 11:31:45] ...optimizers/trijoint.py.151: total_norm: 4.112177 [I 2022-10-26 11:31:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 910/2304 [I 2022-10-26 11:31:50] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:11:36 [I 2022-10-26 11:31:50] ...trap/engines/engine.py.229: process: 0.50093 | load: 0.00245 [I 2022-10-26 11:31:50] ...trap/engines/engine.py.230: loss: 0.26202 [I 2022-10-26 11:31:50] ...optimizers/trijoint.py.151: total_norm: 3.902564 [I 2022-10-26 11:31:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 920/2304 [I 2022-10-26 11:31:55] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:11:29 [I 2022-10-26 11:31:55] ...trap/engines/engine.py.229: process: 0.49850 | load: 0.00242 [I 2022-10-26 11:31:55] ...trap/engines/engine.py.230: loss: 0.29674 [I 2022-10-26 11:31:55] ...optimizers/trijoint.py.151: total_norm: 3.690076 [I 2022-10-26 11:32:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 930/2304 [I 2022-10-26 11:32:00] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:11:24 [I 2022-10-26 11:32:00] ...trap/engines/engine.py.229: process: 0.50838 | load: 0.00250 [I 2022-10-26 11:32:00] ...trap/engines/engine.py.230: loss: 0.23586 [I 2022-10-26 11:32:00] ...optimizers/trijoint.py.151: total_norm: 3.097134 [I 2022-10-26 11:32:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 940/2304 [I 2022-10-26 11:32:05] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:11:19 [I 2022-10-26 11:32:05] ...trap/engines/engine.py.229: process: 0.49852 | load: 0.00248 [I 2022-10-26 11:32:05] ...trap/engines/engine.py.230: loss: 0.26006 [I 2022-10-26 11:32:05] ...optimizers/trijoint.py.151: total_norm: 2.949271 [I 2022-10-26 11:32:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 950/2304 [I 2022-10-26 11:32:10] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:11:14 [I 2022-10-26 11:32:10] ...trap/engines/engine.py.229: process: 0.50034 | load: 0.00248 [I 2022-10-26 11:32:10] ...trap/engines/engine.py.230: loss: 0.25777 [I 2022-10-26 11:32:10] ...optimizers/trijoint.py.151: total_norm: 3.946560 [I 2022-10-26 11:32:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 960/2304 [I 2022-10-26 11:32:15] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:11:09 [I 2022-10-26 11:32:15] ...trap/engines/engine.py.229: process: 0.49658 | load: 0.00245 [I 2022-10-26 11:32:15] ...trap/engines/engine.py.230: loss: 0.25050 [I 2022-10-26 11:32:15] ...optimizers/trijoint.py.151: total_norm: 3.546809 [I 2022-10-26 11:32:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 970/2304 [I 2022-10-26 11:32:20] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:11:03 [I 2022-10-26 11:32:20] ...trap/engines/engine.py.229: process: 0.49581 | load: 0.00249 [I 2022-10-26 11:32:20] ...trap/engines/engine.py.230: loss: 0.31864 [I 2022-10-26 11:32:20] ...optimizers/trijoint.py.151: total_norm: 4.100041 [I 2022-10-26 11:32:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 980/2304 [I 2022-10-26 11:32:25] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:10:58 [I 2022-10-26 11:32:25] ...trap/engines/engine.py.229: process: 0.50354 | load: 0.00245 [I 2022-10-26 11:32:25] ...trap/engines/engine.py.230: loss: 0.27947 [I 2022-10-26 11:32:25] ...optimizers/trijoint.py.151: total_norm: 3.734560 [I 2022-10-26 11:32:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 990/2304 [I 2022-10-26 11:32:30] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:10:57 [I 2022-10-26 11:32:30] ...trap/engines/engine.py.229: process: 0.50314 | load: 0.00243 [I 2022-10-26 11:32:30] ...trap/engines/engine.py.230: loss: 0.32656 [I 2022-10-26 11:32:30] ...optimizers/trijoint.py.151: total_norm: 4.592731 [I 2022-10-26 11:32:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 1000/2304 [I 2022-10-26 11:32:35] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:10:45 [I 2022-10-26 11:32:35] ...trap/engines/engine.py.229: process: 0.47202 | load: 0.00246 [I 2022-10-26 11:32:35] ...trap/engines/engine.py.230: loss: 0.22736 [I 2022-10-26 11:32:35] ...optimizers/trijoint.py.151: total_norm: 3.560294 [I 2022-10-26 11:32:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 1010/2304 [I 2022-10-26 11:32:40] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:10:47 [I 2022-10-26 11:32:40] ...trap/engines/engine.py.229: process: 0.49123 | load: 0.00247 [I 2022-10-26 11:32:40] ...trap/engines/engine.py.230: loss: 0.30266 [I 2022-10-26 11:32:40] ...optimizers/trijoint.py.151: total_norm: 3.483556 [I 2022-10-26 11:32:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 1020/2304 [I 2022-10-26 11:32:45] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:10:41 [I 2022-10-26 11:32:45] ...trap/engines/engine.py.229: process: 0.49623 | load: 0.00249 [I 2022-10-26 11:32:45] ...trap/engines/engine.py.230: loss: 0.33706 [I 2022-10-26 11:32:45] ...optimizers/trijoint.py.151: total_norm: 3.725672 [I 2022-10-26 11:32:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 1030/2304 [I 2022-10-26 11:32:50] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:10:32 [I 2022-10-26 11:32:50] ...trap/engines/engine.py.229: process: 0.49884 | load: 0.00247 [I 2022-10-26 11:32:50] ...trap/engines/engine.py.230: loss: 0.24247 [I 2022-10-26 11:32:50] ...optimizers/trijoint.py.151: total_norm: 3.328015 [I 2022-10-26 11:32:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 1040/2304 [I 2022-10-26 11:32:55] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:10:42 [I 2022-10-26 11:32:55] ...trap/engines/engine.py.229: process: 0.52257 | load: 0.00257 [I 2022-10-26 11:32:55] ...trap/engines/engine.py.230: loss: 0.34930 [I 2022-10-26 11:32:55] ...optimizers/trijoint.py.151: total_norm: 3.266311 [I 2022-10-26 11:33:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 1050/2304 [I 2022-10-26 11:33:00] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:10:29 [I 2022-10-26 11:33:00] ...trap/engines/engine.py.229: process: 0.49867 | load: 0.00246 [I 2022-10-26 11:33:00] ...trap/engines/engine.py.230: loss: 0.33880 [I 2022-10-26 11:33:00] ...optimizers/trijoint.py.151: total_norm: 4.219313 [I 2022-10-26 11:33:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 1060/2304 [I 2022-10-26 11:33:05] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:10:21 [I 2022-10-26 11:33:05] ...trap/engines/engine.py.229: process: 0.49970 | load: 0.00244 [I 2022-10-26 11:33:05] ...trap/engines/engine.py.230: loss: 0.27638 [I 2022-10-26 11:33:05] ...optimizers/trijoint.py.151: total_norm: 4.340180 [I 2022-10-26 11:33:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 1070/2304 [I 2022-10-26 11:33:10] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:10:12 [I 2022-10-26 11:33:10] ...trap/engines/engine.py.229: process: 0.50013 | load: 0.00243 [I 2022-10-26 11:33:10] ...trap/engines/engine.py.230: loss: 0.33257 [I 2022-10-26 11:33:10] ...optimizers/trijoint.py.151: total_norm: 3.698012 [I 2022-10-26 11:33:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 1080/2304 [I 2022-10-26 11:33:15] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:10:10 [I 2022-10-26 11:33:15] ...trap/engines/engine.py.229: process: 0.49586 | load: 0.00248 [I 2022-10-26 11:33:15] ...trap/engines/engine.py.230: loss: 0.22178 [I 2022-10-26 11:33:15] ...optimizers/trijoint.py.151: total_norm: 3.123175 [I 2022-10-26 11:33:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 1090/2304 [I 2022-10-26 11:33:20] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:10:05 [I 2022-10-26 11:33:20] ...trap/engines/engine.py.229: process: 0.50313 | load: 0.00250 [I 2022-10-26 11:33:20] ...trap/engines/engine.py.230: loss: 0.18709 [I 2022-10-26 11:33:20] ...optimizers/trijoint.py.151: total_norm: 3.710800 [I 2022-10-26 11:33:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 1100/2304 [I 2022-10-26 11:33:25] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:09:57 [I 2022-10-26 11:33:25] ...trap/engines/engine.py.229: process: 0.49288 | load: 0.00245 [I 2022-10-26 11:33:25] ...trap/engines/engine.py.230: loss: 0.27173 [I 2022-10-26 11:33:25] ...optimizers/trijoint.py.151: total_norm: 4.891082 [I 2022-10-26 11:33:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 1110/2304 [I 2022-10-26 11:33:30] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:09:57 [I 2022-10-26 11:33:30] ...trap/engines/engine.py.229: process: 0.50452 | load: 0.00253 [I 2022-10-26 11:33:30] ...trap/engines/engine.py.230: loss: 0.32917 [I 2022-10-26 11:33:30] ...optimizers/trijoint.py.151: total_norm: 3.590385 [I 2022-10-26 11:33:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 1120/2304 [I 2022-10-26 11:33:35] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:09:50 [I 2022-10-26 11:33:35] ...trap/engines/engine.py.229: process: 0.48813 | load: 0.00244 [I 2022-10-26 11:33:35] ...trap/engines/engine.py.230: loss: 0.31069 [I 2022-10-26 11:33:35] ...optimizers/trijoint.py.151: total_norm: 3.751722 [I 2022-10-26 11:33:40] ...trap/engines/engine.py.224: train: epoch 5 | batch 1130/2304 [I 2022-10-26 11:33:40] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:09:46 [I 2022-10-26 11:33:40] ...trap/engines/engine.py.229: process: 0.50286 | load: 0.00245 [I 2022-10-26 11:33:40] ...trap/engines/engine.py.230: loss: 0.27119 [I 2022-10-26 11:33:40] ...optimizers/trijoint.py.151: total_norm: 3.726280 [I 2022-10-26 11:33:45] ...trap/engines/engine.py.224: train: epoch 5 | batch 1140/2304 [I 2022-10-26 11:33:45] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:09:39 [I 2022-10-26 11:33:45] ...trap/engines/engine.py.229: process: 0.49126 | load: 0.00245 [I 2022-10-26 11:33:45] ...trap/engines/engine.py.230: loss: 0.24735 [I 2022-10-26 11:33:45] ...optimizers/trijoint.py.151: total_norm: 4.433414 [I 2022-10-26 11:33:50] ...trap/engines/engine.py.224: train: epoch 5 | batch 1150/2304 [I 2022-10-26 11:33:50] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:09:39 [I 2022-10-26 11:33:50] ...trap/engines/engine.py.229: process: 0.49316 | load: 0.00243 [I 2022-10-26 11:33:50] ...trap/engines/engine.py.230: loss: 0.30726 [I 2022-10-26 11:33:50] ...optimizers/trijoint.py.151: total_norm: 3.576212 [I 2022-10-26 11:33:55] ...trap/engines/engine.py.224: train: epoch 5 | batch 1160/2304 [I 2022-10-26 11:33:55] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:09:31 [I 2022-10-26 11:33:55] ...trap/engines/engine.py.229: process: 0.49728 | load: 0.00259 [I 2022-10-26 11:33:55] ...trap/engines/engine.py.230: loss: 0.34183 [I 2022-10-26 11:33:55] ...optimizers/trijoint.py.151: total_norm: 3.814058 [I 2022-10-26 11:34:00] ...trap/engines/engine.py.224: train: epoch 5 | batch 1170/2304 [I 2022-10-26 11:34:00] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:09:25 [I 2022-10-26 11:34:00] ...trap/engines/engine.py.229: process: 0.50269 | load: 0.00249 [I 2022-10-26 11:34:00] ...trap/engines/engine.py.230: loss: 0.37201 [I 2022-10-26 11:34:00] ...optimizers/trijoint.py.151: total_norm: 4.185572 [I 2022-10-26 11:34:05] ...trap/engines/engine.py.224: train: epoch 5 | batch 1180/2304 [I 2022-10-26 11:34:05] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:09:21 [I 2022-10-26 11:34:05] ...trap/engines/engine.py.229: process: 0.49840 | load: 0.00246 [I 2022-10-26 11:34:05] ...trap/engines/engine.py.230: loss: 0.26282 [I 2022-10-26 11:34:05] ...optimizers/trijoint.py.151: total_norm: 3.136150 [I 2022-10-26 11:34:10] ...trap/engines/engine.py.224: train: epoch 5 | batch 1190/2304 [I 2022-10-26 11:34:10] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:09:17 [I 2022-10-26 11:34:10] ...trap/engines/engine.py.229: process: 0.50165 | load: 0.00282 [I 2022-10-26 11:34:10] ...trap/engines/engine.py.230: loss: 0.23254 [I 2022-10-26 11:34:10] ...optimizers/trijoint.py.151: total_norm: 3.029479 [I 2022-10-26 11:34:15] ...trap/engines/engine.py.224: train: epoch 5 | batch 1200/2304 [I 2022-10-26 11:34:15] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:09:13 [I 2022-10-26 11:34:15] ...trap/engines/engine.py.229: process: 0.50087 | load: 0.00250 [I 2022-10-26 11:34:15] ...trap/engines/engine.py.230: loss: 0.28243 [I 2022-10-26 11:34:15] ...optimizers/trijoint.py.151: total_norm: 2.669073 [I 2022-10-26 11:34:20] ...trap/engines/engine.py.224: train: epoch 5 | batch 1210/2304 [I 2022-10-26 11:34:20] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:09:10 [I 2022-10-26 11:34:20] ...trap/engines/engine.py.229: process: 0.50608 | load: 0.00244 [I 2022-10-26 11:34:20] ...trap/engines/engine.py.230: loss: 0.21240 [I 2022-10-26 11:34:20] ...optimizers/trijoint.py.151: total_norm: 2.845297 [I 2022-10-26 11:34:25] ...trap/engines/engine.py.224: train: epoch 5 | batch 1220/2304 [I 2022-10-26 11:34:25] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:08:58 [I 2022-10-26 11:34:25] ...trap/engines/engine.py.229: process: 0.48972 | load: 0.00244 [I 2022-10-26 11:34:25] ...trap/engines/engine.py.230: loss: 0.21213 [I 2022-10-26 11:34:25] ...optimizers/trijoint.py.151: total_norm: 2.996111 [I 2022-10-26 11:34:30] ...trap/engines/engine.py.224: train: epoch 5 | batch 1230/2304 [I 2022-10-26 11:34:30] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:08:55 [I 2022-10-26 11:34:30] ...trap/engines/engine.py.229: process: 0.50060 | load: 0.00243 [I 2022-10-26 11:34:30] ...trap/engines/engine.py.230: loss: 0.25561 [I 2022-10-26 11:34:30] ...optimizers/trijoint.py.151: total_norm: 3.281155 [I 2022-10-26 11:34:35] ...trap/engines/engine.py.224: train: epoch 5 | batch 1240/2304 [I 2022-10-26 11:34:35] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:08:47 [I 2022-10-26 11:34:35] ...trap/engines/engine.py.229: process: 0.48696 | load: 0.00241 [I 2022-10-26 11:34:35] ...trap/engines/engine.py.230: loss: 0.22218 [I 2022-10-26 11:34:35] ...optimizers/trijoint.py.151: total_norm: 2.871221 [I 2022-10-26 11:35:02] ...trap/engines/engine.py.224: train: epoch 5 | batch 1250/2304 [I 2022-10-26 11:35:02] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:19:28 [I 2022-10-26 11:35:02] ...trap/engines/engine.py.229: process: 0.51784 | load: 0.00239 [I 2022-10-26 11:35:02] ...trap/engines/engine.py.230: loss: 0.31502 [I 2022-10-26 11:35:02] ...optimizers/trijoint.py.151: total_norm: 4.470117 [I 2022-10-26 11:35:07] ...trap/engines/engine.py.224: train: epoch 5 | batch 1260/2304 [I 2022-10-26 11:35:07] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:09:53 [I 2022-10-26 11:35:07] ...trap/engines/engine.py.229: process: 0.50038 | load: 0.00238 [I 2022-10-26 11:35:07] ...trap/engines/engine.py.230: loss: 0.30991 [I 2022-10-26 11:35:07] ...optimizers/trijoint.py.151: total_norm: 3.759821 [I 2022-10-26 11:35:12] ...trap/engines/engine.py.224: train: epoch 5 | batch 1270/2304 [I 2022-10-26 11:35:12] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:08:58 [I 2022-10-26 11:35:12] ...trap/engines/engine.py.229: process: 0.49147 | load: 0.00250 [I 2022-10-26 11:35:12] ...trap/engines/engine.py.230: loss: 0.28855 [I 2022-10-26 11:35:12] ...optimizers/trijoint.py.151: total_norm: 4.523794 [I 2022-10-26 11:35:17] ...trap/engines/engine.py.224: train: epoch 5 | batch 1280/2304 [I 2022-10-26 11:35:17] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:08:34 [I 2022-10-26 11:35:17] ...trap/engines/engine.py.229: process: 0.50348 | load: 0.00243 [I 2022-10-26 11:35:17] ...trap/engines/engine.py.230: loss: 0.24314 [I 2022-10-26 11:35:17] ...optimizers/trijoint.py.151: total_norm: 3.893332 [I 2022-10-26 11:35:22] ...trap/engines/engine.py.224: train: epoch 5 | batch 1290/2304 [I 2022-10-26 11:35:22] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:08:24 [I 2022-10-26 11:35:22] ...trap/engines/engine.py.229: process: 0.49633 | load: 0.00245 [I 2022-10-26 11:35:22] ...trap/engines/engine.py.230: loss: 0.26737 [I 2022-10-26 11:35:22] ...optimizers/trijoint.py.151: total_norm: 2.784303 [I 2022-10-26 11:35:27] ...trap/engines/engine.py.224: train: epoch 5 | batch 1300/2304 [I 2022-10-26 11:35:27] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:08:22 [I 2022-10-26 11:35:27] ...trap/engines/engine.py.229: process: 0.50050 | load: 0.00240 [I 2022-10-26 11:35:27] ...trap/engines/engine.py.230: loss: 0.41999 [I 2022-10-26 11:35:27] ...optimizers/trijoint.py.151: total_norm: 4.697162 [I 2022-10-26 11:35:32] ...trap/engines/engine.py.224: train: epoch 5 | batch 1310/2304 [I 2022-10-26 11:35:32] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:08:21 [I 2022-10-26 11:35:32] ...trap/engines/engine.py.229: process: 0.50931 | load: 0.00242 [I 2022-10-26 11:35:32] ...trap/engines/engine.py.230: loss: 0.26185 [I 2022-10-26 11:35:32] ...optimizers/trijoint.py.151: total_norm: 3.463600 [I 2022-10-26 11:35:37] ...trap/engines/engine.py.224: train: epoch 5 | batch 1320/2304 [I 2022-10-26 11:35:37] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:08:10 [I 2022-10-26 11:35:37] ...trap/engines/engine.py.229: process: 0.50196 | load: 0.00241 [I 2022-10-26 11:35:37] ...trap/engines/engine.py.230: loss: 0.28503 [I 2022-10-26 11:35:37] ...optimizers/trijoint.py.151: total_norm: 4.014185 [I 2022-10-26 11:35:42] ...trap/engines/engine.py.224: train: epoch 5 | batch 1330/2304 [I 2022-10-26 11:35:42] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:08:07 [I 2022-10-26 11:35:42] ...trap/engines/engine.py.229: process: 0.50759 | load: 0.00247 [I 2022-10-26 11:35:42] ...trap/engines/engine.py.230: loss: 0.24925 [I 2022-10-26 11:35:42] ...optimizers/trijoint.py.151: total_norm: 3.417650 [I 2022-10-26 11:35:47] ...trap/engines/engine.py.224: train: epoch 5 | batch 1340/2304 [I 2022-10-26 11:35:47] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:08:04 [I 2022-10-26 11:35:47] ...trap/engines/engine.py.229: process: 0.50536 | load: 0.00248 [I 2022-10-26 11:35:47] ...trap/engines/engine.py.230: loss: 0.31257 [I 2022-10-26 11:35:47] ...optimizers/trijoint.py.151: total_norm: 3.519181 [I 2022-10-26 11:35:52] ...trap/engines/engine.py.224: train: epoch 5 | batch 1350/2304 [I 2022-10-26 11:35:52] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:08:02 [I 2022-10-26 11:35:52] ...trap/engines/engine.py.229: process: 0.50110 | load: 0.00247 [I 2022-10-26 11:35:52] ...trap/engines/engine.py.230: loss: 0.28404 [I 2022-10-26 11:35:52] ...optimizers/trijoint.py.151: total_norm: 3.628379 [I 2022-10-26 11:35:57] ...trap/engines/engine.py.224: train: epoch 5 | batch 1360/2304 [I 2022-10-26 11:35:57] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:07:53 [I 2022-10-26 11:35:57] ...trap/engines/engine.py.229: process: 0.48675 | load: 0.00255 [I 2022-10-26 11:35:57] ...trap/engines/engine.py.230: loss: 0.31987 [I 2022-10-26 11:35:57] ...optimizers/trijoint.py.151: total_norm: 3.205994 [I 2022-10-26 11:36:02] ...trap/engines/engine.py.224: train: epoch 5 | batch 1370/2304 [I 2022-10-26 11:36:02] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:07:46 [I 2022-10-26 11:36:02] ...trap/engines/engine.py.229: process: 0.49879 | load: 0.00248 [I 2022-10-26 11:36:02] ...trap/engines/engine.py.230: loss: 0.27450 [I 2022-10-26 11:36:02] ...optimizers/trijoint.py.151: total_norm: 4.202867 [I 2022-10-26 11:36:07] ...trap/engines/engine.py.224: train: epoch 5 | batch 1380/2304 [I 2022-10-26 11:36:07] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:07:41 [I 2022-10-26 11:36:07] ...trap/engines/engine.py.229: process: 0.49614 | load: 0.00263 [I 2022-10-26 11:36:07] ...trap/engines/engine.py.230: loss: 0.17531 [I 2022-10-26 11:36:07] ...optimizers/trijoint.py.151: total_norm: 2.651028 [I 2022-10-26 11:36:12] ...trap/engines/engine.py.224: train: epoch 5 | batch 1390/2304 [I 2022-10-26 11:36:12] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:07:37 [I 2022-10-26 11:36:12] ...trap/engines/engine.py.229: process: 0.50009 | load: 0.00247 [I 2022-10-26 11:36:12] ...trap/engines/engine.py.230: loss: 0.37527 [I 2022-10-26 11:36:12] ...optimizers/trijoint.py.151: total_norm: 3.852969 [I 2022-10-26 11:36:17] ...trap/engines/engine.py.224: train: epoch 5 | batch 1400/2304 [I 2022-10-26 11:36:17] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:07:33 [I 2022-10-26 11:36:17] ...trap/engines/engine.py.229: process: 0.50816 | load: 0.00249 [I 2022-10-26 11:36:17] ...trap/engines/engine.py.230: loss: 0.31058 [I 2022-10-26 11:36:17] ...optimizers/trijoint.py.151: total_norm: 3.216189 [I 2022-10-26 11:36:22] ...trap/engines/engine.py.224: train: epoch 5 | batch 1410/2304 [I 2022-10-26 11:36:22] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:07:27 [I 2022-10-26 11:36:22] ...trap/engines/engine.py.229: process: 0.48661 | load: 0.00269 [I 2022-10-26 11:36:22] ...trap/engines/engine.py.230: loss: 0.17986 [I 2022-10-26 11:36:22] ...optimizers/trijoint.py.151: total_norm: 2.849334 [I 2022-10-26 11:36:27] ...trap/engines/engine.py.224: train: epoch 5 | batch 1420/2304 [I 2022-10-26 11:36:27] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:07:20 [I 2022-10-26 11:36:27] ...trap/engines/engine.py.229: process: 0.50480 | load: 0.00254 [I 2022-10-26 11:36:27] ...trap/engines/engine.py.230: loss: 0.29827 [I 2022-10-26 11:36:27] ...optimizers/trijoint.py.151: total_norm: 3.329766 [I 2022-10-26 11:36:32] ...trap/engines/engine.py.224: train: epoch 5 | batch 1430/2304 [I 2022-10-26 11:36:32] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:07:18 [I 2022-10-26 11:36:32] ...trap/engines/engine.py.229: process: 0.49079 | load: 0.00247 [I 2022-10-26 11:36:32] ...trap/engines/engine.py.230: loss: 0.29700 [I 2022-10-26 11:36:32] ...optimizers/trijoint.py.151: total_norm: 3.674783 [I 2022-10-26 11:36:37] ...trap/engines/engine.py.224: train: epoch 5 | batch 1440/2304 [I 2022-10-26 11:36:37] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:07:12 [I 2022-10-26 11:36:37] ...trap/engines/engine.py.229: process: 0.49682 | load: 0.00244 [I 2022-10-26 11:36:37] ...trap/engines/engine.py.230: loss: 0.27149 [I 2022-10-26 11:36:37] ...optimizers/trijoint.py.151: total_norm: 3.079194 [I 2022-10-26 11:36:43] ...trap/engines/engine.py.224: train: epoch 5 | batch 1450/2304 [I 2022-10-26 11:36:43] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:07:08 [I 2022-10-26 11:36:43] ...trap/engines/engine.py.229: process: 0.49570 | load: 0.00271 [I 2022-10-26 11:36:43] ...trap/engines/engine.py.230: loss: 0.20113 [I 2022-10-26 11:36:43] ...optimizers/trijoint.py.151: total_norm: 3.195238 [I 2022-10-26 11:36:48] ...trap/engines/engine.py.224: train: epoch 5 | batch 1460/2304 [I 2022-10-26 11:36:48] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:07:04 [I 2022-10-26 11:36:48] ...trap/engines/engine.py.229: process: 0.50536 | load: 0.00253 [I 2022-10-26 11:36:48] ...trap/engines/engine.py.230: loss: 0.25326 [I 2022-10-26 11:36:48] ...optimizers/trijoint.py.151: total_norm: 3.256660 [I 2022-10-26 11:36:53] ...trap/engines/engine.py.224: train: epoch 5 | batch 1470/2304 [I 2022-10-26 11:36:53] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:07:00 [I 2022-10-26 11:36:53] ...trap/engines/engine.py.229: process: 0.48119 | load: 0.00249 [I 2022-10-26 11:36:53] ...trap/engines/engine.py.230: loss: 0.27254 [I 2022-10-26 11:36:53] ...optimizers/trijoint.py.151: total_norm: 2.880069 [I 2022-10-26 11:36:58] ...trap/engines/engine.py.224: train: epoch 5 | batch 1480/2304 [I 2022-10-26 11:36:58] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:06:53 [I 2022-10-26 11:36:58] ...trap/engines/engine.py.229: process: 0.49689 | load: 0.00251 [I 2022-10-26 11:36:58] ...trap/engines/engine.py.230: loss: 0.39621 [I 2022-10-26 11:36:58] ...optimizers/trijoint.py.151: total_norm: 3.914773 [I 2022-10-26 11:37:03] ...trap/engines/engine.py.224: train: epoch 5 | batch 1490/2304 [I 2022-10-26 11:37:03] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:06:47 [I 2022-10-26 11:37:03] ...trap/engines/engine.py.229: process: 0.49586 | load: 0.00245 [I 2022-10-26 11:37:03] ...trap/engines/engine.py.230: loss: 0.31153 [I 2022-10-26 11:37:03] ...optimizers/trijoint.py.151: total_norm: 3.697358 [I 2022-10-26 11:37:08] ...trap/engines/engine.py.224: train: epoch 5 | batch 1500/2304 [I 2022-10-26 11:37:08] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:06:40 [I 2022-10-26 11:37:08] ...trap/engines/engine.py.229: process: 0.49800 | load: 0.00254 [I 2022-10-26 11:37:08] ...trap/engines/engine.py.230: loss: 0.25400 [I 2022-10-26 11:37:08] ...optimizers/trijoint.py.151: total_norm: 3.387510 [I 2022-10-26 11:37:13] ...trap/engines/engine.py.224: train: epoch 5 | batch 1510/2304 [I 2022-10-26 11:37:13] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:06:36 [I 2022-10-26 11:37:13] ...trap/engines/engine.py.229: process: 0.49938 | load: 0.00250 [I 2022-10-26 11:37:13] ...trap/engines/engine.py.230: loss: 0.31828 [I 2022-10-26 11:37:13] ...optimizers/trijoint.py.151: total_norm: 2.901852 [I 2022-10-26 11:37:18] ...trap/engines/engine.py.224: train: epoch 5 | batch 1520/2304 [I 2022-10-26 11:37:18] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:06:33 [I 2022-10-26 11:37:18] ...trap/engines/engine.py.229: process: 0.50152 | load: 0.00246 [I 2022-10-26 11:37:18] ...trap/engines/engine.py.230: loss: 0.25771 [I 2022-10-26 11:37:18] ...optimizers/trijoint.py.151: total_norm: 4.224974 [I 2022-10-26 11:37:23] ...trap/engines/engine.py.224: train: epoch 5 | batch 1530/2304 [I 2022-10-26 11:37:23] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:06:42 [I 2022-10-26 11:37:23] ...trap/engines/engine.py.229: process: 0.58552 | load: 0.00262 [I 2022-10-26 11:37:23] ...trap/engines/engine.py.230: loss: 0.30537 [I 2022-10-26 11:37:23] ...optimizers/trijoint.py.151: total_norm: 4.139788 [I 2022-10-26 11:37:28] ...trap/engines/engine.py.224: train: epoch 5 | batch 1540/2304 [I 2022-10-26 11:37:28] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:06:20 [I 2022-10-26 11:37:28] ...trap/engines/engine.py.229: process: 0.48484 | load: 0.00252 [I 2022-10-26 11:37:28] ...trap/engines/engine.py.230: loss: 0.31582 [I 2022-10-26 11:37:28] ...optimizers/trijoint.py.151: total_norm: 3.604992 [I 2022-10-26 11:37:33] ...trap/engines/engine.py.224: train: epoch 5 | batch 1550/2304 [I 2022-10-26 11:37:33] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:06:17 [I 2022-10-26 11:37:33] ...trap/engines/engine.py.229: process: 0.50478 | load: 0.00255 [I 2022-10-26 11:37:33] ...trap/engines/engine.py.230: loss: 0.23178 [I 2022-10-26 11:37:33] ...optimizers/trijoint.py.151: total_norm: 3.428903 [I 2022-10-26 11:37:38] ...trap/engines/engine.py.224: train: epoch 5 | batch 1560/2304 [I 2022-10-26 11:37:38] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:06:08 [I 2022-10-26 11:37:38] ...trap/engines/engine.py.229: process: 0.49279 | load: 0.00248 [I 2022-10-26 11:37:38] ...trap/engines/engine.py.230: loss: 0.24905 [I 2022-10-26 11:37:38] ...optimizers/trijoint.py.151: total_norm: 2.971141 [I 2022-10-26 11:37:43] ...trap/engines/engine.py.224: train: epoch 5 | batch 1570/2304 [I 2022-10-26 11:37:43] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:06:04 [I 2022-10-26 11:37:43] ...trap/engines/engine.py.229: process: 0.48981 | load: 0.00250 [I 2022-10-26 11:37:43] ...trap/engines/engine.py.230: loss: 0.25088 [I 2022-10-26 11:37:43] ...optimizers/trijoint.py.151: total_norm: 3.318656 [I 2022-10-26 11:37:48] ...trap/engines/engine.py.224: train: epoch 5 | batch 1580/2304 [I 2022-10-26 11:37:48] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:06:01 [I 2022-10-26 11:37:48] ...trap/engines/engine.py.229: process: 0.50264 | load: 0.00257 [I 2022-10-26 11:37:48] ...trap/engines/engine.py.230: loss: 0.23892 [I 2022-10-26 11:37:48] ...optimizers/trijoint.py.151: total_norm: 3.158040 [I 2022-10-26 11:37:53] ...trap/engines/engine.py.224: train: epoch 5 | batch 1590/2304 [I 2022-10-26 11:37:53] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:05:55 [I 2022-10-26 11:37:53] ...trap/engines/engine.py.229: process: 0.48486 | load: 0.00250 [I 2022-10-26 11:37:53] ...trap/engines/engine.py.230: loss: 0.30973 [I 2022-10-26 11:37:53] ...optimizers/trijoint.py.151: total_norm: 3.955114 [I 2022-10-26 11:37:58] ...trap/engines/engine.py.224: train: epoch 5 | batch 1600/2304 [I 2022-10-26 11:37:58] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:05:51 [I 2022-10-26 11:37:58] ...trap/engines/engine.py.229: process: 0.49181 | load: 0.00248 [I 2022-10-26 11:37:58] ...trap/engines/engine.py.230: loss: 0.27927 [I 2022-10-26 11:37:58] ...optimizers/trijoint.py.151: total_norm: 3.188869 [I 2022-10-26 11:38:03] ...trap/engines/engine.py.224: train: epoch 5 | batch 1610/2304 [I 2022-10-26 11:38:03] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:05:52 [I 2022-10-26 11:38:03] ...trap/engines/engine.py.229: process: 0.53660 | load: 0.00263 [I 2022-10-26 11:38:03] ...trap/engines/engine.py.230: loss: 0.31767 [I 2022-10-26 11:38:03] ...optimizers/trijoint.py.151: total_norm: 4.005391 [I 2022-10-26 11:38:08] ...trap/engines/engine.py.224: train: epoch 5 | batch 1620/2304 [I 2022-10-26 11:38:08] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:05:46 [I 2022-10-26 11:38:08] ...trap/engines/engine.py.229: process: 0.50365 | load: 0.00258 [I 2022-10-26 11:38:08] ...trap/engines/engine.py.230: loss: 0.32816 [I 2022-10-26 11:38:08] ...optimizers/trijoint.py.151: total_norm: 4.050399 [I 2022-10-26 11:38:13] ...trap/engines/engine.py.224: train: epoch 5 | batch 1630/2304 [I 2022-10-26 11:38:13] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:05:37 [I 2022-10-26 11:38:13] ...trap/engines/engine.py.229: process: 0.50491 | load: 0.00249 [I 2022-10-26 11:38:13] ...trap/engines/engine.py.230: loss: 0.33765 [I 2022-10-26 11:38:13] ...optimizers/trijoint.py.151: total_norm: 3.084738 [I 2022-10-26 11:38:18] ...trap/engines/engine.py.224: train: epoch 5 | batch 1640/2304 [I 2022-10-26 11:38:18] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:05:34 [I 2022-10-26 11:38:18] ...trap/engines/engine.py.229: process: 0.50166 | load: 0.00246 [I 2022-10-26 11:38:18] ...trap/engines/engine.py.230: loss: 0.30561 [I 2022-10-26 11:38:18] ...optimizers/trijoint.py.151: total_norm: 3.527257 [I 2022-10-26 11:38:23] ...trap/engines/engine.py.224: train: epoch 5 | batch 1650/2304 [I 2022-10-26 11:38:23] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:05:31 [I 2022-10-26 11:38:23] ...trap/engines/engine.py.229: process: 0.50051 | load: 0.00262 [I 2022-10-26 11:38:23] ...trap/engines/engine.py.230: loss: 0.22425 [I 2022-10-26 11:38:23] ...optimizers/trijoint.py.151: total_norm: 3.113599 [I 2022-10-26 11:38:28] ...trap/engines/engine.py.224: train: epoch 5 | batch 1660/2304 [I 2022-10-26 11:38:28] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:05:22 [I 2022-10-26 11:38:28] ...trap/engines/engine.py.229: process: 0.50126 | load: 0.00252 [I 2022-10-26 11:38:28] ...trap/engines/engine.py.230: loss: 0.22759 [I 2022-10-26 11:38:28] ...optimizers/trijoint.py.151: total_norm: 3.473295 [I 2022-10-26 11:38:33] ...trap/engines/engine.py.224: train: epoch 5 | batch 1670/2304 [I 2022-10-26 11:38:33] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:05:16 [I 2022-10-26 11:38:33] ...trap/engines/engine.py.229: process: 0.50240 | load: 0.00269 [I 2022-10-26 11:38:33] ...trap/engines/engine.py.230: loss: 0.31934 [I 2022-10-26 11:38:33] ...optimizers/trijoint.py.151: total_norm: 3.752689 [I 2022-10-26 11:38:38] ...trap/engines/engine.py.224: train: epoch 5 | batch 1680/2304 [I 2022-10-26 11:38:38] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:05:13 [I 2022-10-26 11:38:38] ...trap/engines/engine.py.229: process: 0.49496 | load: 0.00267 [I 2022-10-26 11:38:38] ...trap/engines/engine.py.230: loss: 0.31960 [I 2022-10-26 11:38:38] ...optimizers/trijoint.py.151: total_norm: 5.118402 [I 2022-10-26 11:38:43] ...trap/engines/engine.py.224: train: epoch 5 | batch 1690/2304 [I 2022-10-26 11:38:43] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:05:08 [I 2022-10-26 11:38:43] ...trap/engines/engine.py.229: process: 0.50143 | load: 0.00262 [I 2022-10-26 11:38:43] ...trap/engines/engine.py.230: loss: 0.24712 [I 2022-10-26 11:38:43] ...optimizers/trijoint.py.151: total_norm: 3.490695 [I 2022-10-26 11:38:48] ...trap/engines/engine.py.224: train: epoch 5 | batch 1700/2304 [I 2022-10-26 11:38:48] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:05:00 [I 2022-10-26 11:38:48] ...trap/engines/engine.py.229: process: 0.48939 | load: 0.00255 [I 2022-10-26 11:38:48] ...trap/engines/engine.py.230: loss: 0.21151 [I 2022-10-26 11:38:48] ...optimizers/trijoint.py.151: total_norm: 3.182890 [I 2022-10-26 11:38:53] ...trap/engines/engine.py.224: train: epoch 5 | batch 1710/2304 [I 2022-10-26 11:38:53] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:04:57 [I 2022-10-26 11:38:53] ...trap/engines/engine.py.229: process: 0.50282 | load: 0.00273 [I 2022-10-26 11:38:53] ...trap/engines/engine.py.230: loss: 0.29780 [I 2022-10-26 11:38:53] ...optimizers/trijoint.py.151: total_norm: 3.773096 [I 2022-10-26 11:38:58] ...trap/engines/engine.py.224: train: epoch 5 | batch 1720/2304 [I 2022-10-26 11:38:58] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:04:51 [I 2022-10-26 11:38:58] ...trap/engines/engine.py.229: process: 0.49936 | load: 0.00252 [I 2022-10-26 11:38:58] ...trap/engines/engine.py.230: loss: 0.29445 [I 2022-10-26 11:38:58] ...optimizers/trijoint.py.151: total_norm: 3.681019 [I 2022-10-26 11:39:03] ...trap/engines/engine.py.224: train: epoch 5 | batch 1730/2304 [I 2022-10-26 11:39:03] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:04:49 [I 2022-10-26 11:39:03] ...trap/engines/engine.py.229: process: 0.50811 | load: 0.00250 [I 2022-10-26 11:39:03] ...trap/engines/engine.py.230: loss: 0.23870 [I 2022-10-26 11:39:03] ...optimizers/trijoint.py.151: total_norm: 3.388271 [I 2022-10-26 11:39:08] ...trap/engines/engine.py.224: train: epoch 5 | batch 1740/2304 [I 2022-10-26 11:39:08] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:04:39 [I 2022-10-26 11:39:08] ...trap/engines/engine.py.229: process: 0.48812 | load: 0.00253 [I 2022-10-26 11:39:08] ...trap/engines/engine.py.230: loss: 0.22858 [I 2022-10-26 11:39:08] ...optimizers/trijoint.py.151: total_norm: 2.431413 [I 2022-10-26 11:39:13] ...trap/engines/engine.py.224: train: epoch 5 | batch 1750/2304 [I 2022-10-26 11:39:13] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:04:36 [I 2022-10-26 11:39:13] ...trap/engines/engine.py.229: process: 0.50323 | load: 0.00250 [I 2022-10-26 11:39:13] ...trap/engines/engine.py.230: loss: 0.30055 [I 2022-10-26 11:39:13] ...optimizers/trijoint.py.151: total_norm: 4.037104 [I 2022-10-26 11:39:18] ...trap/engines/engine.py.224: train: epoch 5 | batch 1760/2304 [I 2022-10-26 11:39:18] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:04:30 [I 2022-10-26 11:39:18] ...trap/engines/engine.py.229: process: 0.48851 | load: 0.00247 [I 2022-10-26 11:39:18] ...trap/engines/engine.py.230: loss: 0.24069 [I 2022-10-26 11:39:18] ...optimizers/trijoint.py.151: total_norm: 3.496932 [I 2022-10-26 11:39:23] ...trap/engines/engine.py.224: train: epoch 5 | batch 1770/2304 [I 2022-10-26 11:39:23] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:04:26 [I 2022-10-26 11:39:23] ...trap/engines/engine.py.229: process: 0.50629 | load: 0.00248 [I 2022-10-26 11:39:23] ...trap/engines/engine.py.230: loss: 0.32406 [I 2022-10-26 11:39:23] ...optimizers/trijoint.py.151: total_norm: 4.477784 [I 2022-10-26 11:39:28] ...trap/engines/engine.py.224: train: epoch 5 | batch 1780/2304 [I 2022-10-26 11:39:28] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:04:28 [I 2022-10-26 11:39:28] ...trap/engines/engine.py.229: process: 0.50484 | load: 0.00254 [I 2022-10-26 11:39:28] ...trap/engines/engine.py.230: loss: 0.37146 [I 2022-10-26 11:39:28] ...optimizers/trijoint.py.151: total_norm: 3.860523 [I 2022-10-26 11:39:33] ...trap/engines/engine.py.224: train: epoch 5 | batch 1790/2304 [I 2022-10-26 11:39:33] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:04:17 [I 2022-10-26 11:39:33] ...trap/engines/engine.py.229: process: 0.50180 | load: 0.00249 [I 2022-10-26 11:39:33] ...trap/engines/engine.py.230: loss: 0.28175 [I 2022-10-26 11:39:33] ...optimizers/trijoint.py.151: total_norm: 4.138397 [I 2022-10-26 11:39:38] ...trap/engines/engine.py.224: train: epoch 5 | batch 1800/2304 [I 2022-10-26 11:39:38] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:04:11 [I 2022-10-26 11:39:38] ...trap/engines/engine.py.229: process: 0.49347 | load: 0.00246 [I 2022-10-26 11:39:38] ...trap/engines/engine.py.230: loss: 0.25743 [I 2022-10-26 11:39:38] ...optimizers/trijoint.py.151: total_norm: 3.254634 [I 2022-10-26 11:39:43] ...trap/engines/engine.py.224: train: epoch 5 | batch 1810/2304 [I 2022-10-26 11:39:43] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:04:04 [I 2022-10-26 11:39:43] ...trap/engines/engine.py.229: process: 0.50389 | load: 0.00244 [I 2022-10-26 11:39:43] ...trap/engines/engine.py.230: loss: 0.19296 [I 2022-10-26 11:39:43] ...optimizers/trijoint.py.151: total_norm: 2.890976 [I 2022-10-26 11:39:48] ...trap/engines/engine.py.224: train: epoch 5 | batch 1820/2304 [I 2022-10-26 11:39:48] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:04:02 [I 2022-10-26 11:39:48] ...trap/engines/engine.py.229: process: 0.49893 | load: 0.00253 [I 2022-10-26 11:39:48] ...trap/engines/engine.py.230: loss: 0.29189 [I 2022-10-26 11:39:48] ...optimizers/trijoint.py.151: total_norm: 4.427865 [I 2022-10-26 11:39:53] ...trap/engines/engine.py.224: train: epoch 5 | batch 1830/2304 [I 2022-10-26 11:39:53] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:03:56 [I 2022-10-26 11:39:53] ...trap/engines/engine.py.229: process: 0.50805 | load: 0.00248 [I 2022-10-26 11:39:53] ...trap/engines/engine.py.230: loss: 0.29360 [I 2022-10-26 11:39:53] ...optimizers/trijoint.py.151: total_norm: 4.238403 [I 2022-10-26 11:39:58] ...trap/engines/engine.py.224: train: epoch 5 | batch 1840/2304 [I 2022-10-26 11:39:58] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:03:52 [I 2022-10-26 11:39:58] ...trap/engines/engine.py.229: process: 0.50500 | load: 0.00245 [I 2022-10-26 11:39:58] ...trap/engines/engine.py.230: loss: 0.29661 [I 2022-10-26 11:39:58] ...optimizers/trijoint.py.151: total_norm: 4.098897 [I 2022-10-26 11:40:04] ...trap/engines/engine.py.224: train: epoch 5 | batch 1850/2304 [I 2022-10-26 11:40:04] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:03:49 [I 2022-10-26 11:40:04] ...trap/engines/engine.py.229: process: 0.49502 | load: 0.00243 [I 2022-10-26 11:40:04] ...trap/engines/engine.py.230: loss: 0.31401 [I 2022-10-26 11:40:04] ...optimizers/trijoint.py.151: total_norm: 4.448437 [I 2022-10-26 11:40:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 1860/2304 [I 2022-10-26 11:40:09] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:03:42 [I 2022-10-26 11:40:09] ...trap/engines/engine.py.229: process: 0.50403 | load: 0.00243 [I 2022-10-26 11:40:09] ...trap/engines/engine.py.230: loss: 0.32998 [I 2022-10-26 11:40:09] ...optimizers/trijoint.py.151: total_norm: 4.281487 [I 2022-10-26 11:40:14] ...trap/engines/engine.py.224: train: epoch 5 | batch 1870/2304 [I 2022-10-26 11:40:14] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:03:36 [I 2022-10-26 11:40:14] ...trap/engines/engine.py.229: process: 0.49548 | load: 0.00253 [I 2022-10-26 11:40:14] ...trap/engines/engine.py.230: loss: 0.25903 [I 2022-10-26 11:40:14] ...optimizers/trijoint.py.151: total_norm: 2.839414 [I 2022-10-26 11:40:19] ...trap/engines/engine.py.224: train: epoch 5 | batch 1880/2304 [I 2022-10-26 11:40:19] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:03:32 [I 2022-10-26 11:40:19] ...trap/engines/engine.py.229: process: 0.50710 | load: 0.00266 [I 2022-10-26 11:40:19] ...trap/engines/engine.py.230: loss: 0.29763 [I 2022-10-26 11:40:19] ...optimizers/trijoint.py.151: total_norm: 3.974993 [I 2022-10-26 11:40:24] ...trap/engines/engine.py.224: train: epoch 5 | batch 1890/2304 [I 2022-10-26 11:40:24] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:03:27 [I 2022-10-26 11:40:24] ...trap/engines/engine.py.229: process: 0.49625 | load: 0.00250 [I 2022-10-26 11:40:24] ...trap/engines/engine.py.230: loss: 0.23672 [I 2022-10-26 11:40:24] ...optimizers/trijoint.py.151: total_norm: 3.340015 [I 2022-10-26 11:40:29] ...trap/engines/engine.py.224: train: epoch 5 | batch 1900/2304 [I 2022-10-26 11:40:29] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:03:21 [I 2022-10-26 11:40:29] ...trap/engines/engine.py.229: process: 0.50550 | load: 0.00243 [I 2022-10-26 11:40:29] ...trap/engines/engine.py.230: loss: 0.25787 [I 2022-10-26 11:40:29] ...optimizers/trijoint.py.151: total_norm: 4.256468 [I 2022-10-26 11:40:34] ...trap/engines/engine.py.224: train: epoch 5 | batch 1910/2304 [I 2022-10-26 11:40:34] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:03:15 [I 2022-10-26 11:40:34] ...trap/engines/engine.py.229: process: 0.50024 | load: 0.00245 [I 2022-10-26 11:40:34] ...trap/engines/engine.py.230: loss: 0.22758 [I 2022-10-26 11:40:34] ...optimizers/trijoint.py.151: total_norm: 3.241660 [I 2022-10-26 11:40:39] ...trap/engines/engine.py.224: train: epoch 5 | batch 1920/2304 [I 2022-10-26 11:40:39] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:03:10 [I 2022-10-26 11:40:39] ...trap/engines/engine.py.229: process: 0.50421 | load: 0.00244 [I 2022-10-26 11:40:39] ...trap/engines/engine.py.230: loss: 0.23524 [I 2022-10-26 11:40:39] ...optimizers/trijoint.py.151: total_norm: 2.820237 [I 2022-10-26 11:40:44] ...trap/engines/engine.py.224: train: epoch 5 | batch 1930/2304 [I 2022-10-26 11:40:44] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:03:06 [I 2022-10-26 11:40:44] ...trap/engines/engine.py.229: process: 0.49906 | load: 0.00255 [I 2022-10-26 11:40:44] ...trap/engines/engine.py.230: loss: 0.29964 [I 2022-10-26 11:40:44] ...optimizers/trijoint.py.151: total_norm: 3.539238 [I 2022-10-26 11:40:49] ...trap/engines/engine.py.224: train: epoch 5 | batch 1940/2304 [I 2022-10-26 11:40:49] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:02:59 [I 2022-10-26 11:40:49] ...trap/engines/engine.py.229: process: 0.48662 | load: 0.00246 [I 2022-10-26 11:40:49] ...trap/engines/engine.py.230: loss: 0.35824 [I 2022-10-26 11:40:49] ...optimizers/trijoint.py.151: total_norm: 4.386959 [I 2022-10-26 11:40:54] ...trap/engines/engine.py.224: train: epoch 5 | batch 1950/2304 [I 2022-10-26 11:40:54] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:02:56 [I 2022-10-26 11:40:54] ...trap/engines/engine.py.229: process: 0.50262 | load: 0.00249 [I 2022-10-26 11:40:54] ...trap/engines/engine.py.230: loss: 0.30681 [I 2022-10-26 11:40:54] ...optimizers/trijoint.py.151: total_norm: 3.973942 [I 2022-10-26 11:40:59] ...trap/engines/engine.py.224: train: epoch 5 | batch 1960/2304 [I 2022-10-26 11:40:59] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:02:51 [I 2022-10-26 11:40:59] ...trap/engines/engine.py.229: process: 0.49501 | load: 0.00245 [I 2022-10-26 11:40:59] ...trap/engines/engine.py.230: loss: 0.32045 [I 2022-10-26 11:40:59] ...optimizers/trijoint.py.151: total_norm: 3.353336 [I 2022-10-26 11:41:04] ...trap/engines/engine.py.224: train: epoch 5 | batch 1970/2304 [I 2022-10-26 11:41:04] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:02:46 [I 2022-10-26 11:41:04] ...trap/engines/engine.py.229: process: 0.49911 | load: 0.00243 [I 2022-10-26 11:41:04] ...trap/engines/engine.py.230: loss: 0.24091 [I 2022-10-26 11:41:04] ...optimizers/trijoint.py.151: total_norm: 3.139890 [I 2022-10-26 11:41:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 1980/2304 [I 2022-10-26 11:41:09] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:02:41 [I 2022-10-26 11:41:09] ...trap/engines/engine.py.229: process: 0.49676 | load: 0.00255 [I 2022-10-26 11:41:09] ...trap/engines/engine.py.230: loss: 0.16581 [I 2022-10-26 11:41:09] ...optimizers/trijoint.py.151: total_norm: 2.444229 [I 2022-10-26 11:41:14] ...trap/engines/engine.py.224: train: epoch 5 | batch 1990/2304 [I 2022-10-26 11:41:14] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:02:36 [I 2022-10-26 11:41:14] ...trap/engines/engine.py.229: process: 0.49924 | load: 0.00250 [I 2022-10-26 11:41:14] ...trap/engines/engine.py.230: loss: 0.30916 [I 2022-10-26 11:41:14] ...optimizers/trijoint.py.151: total_norm: 3.119462 [I 2022-10-26 11:41:19] ...trap/engines/engine.py.224: train: epoch 5 | batch 2000/2304 [I 2022-10-26 11:41:19] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:02:33 [I 2022-10-26 11:41:19] ...trap/engines/engine.py.229: process: 0.50288 | load: 0.00249 [I 2022-10-26 11:41:19] ...trap/engines/engine.py.230: loss: 0.24127 [I 2022-10-26 11:41:19] ...optimizers/trijoint.py.151: total_norm: 3.327284 [I 2022-10-26 11:41:24] ...trap/engines/engine.py.224: train: epoch 5 | batch 2010/2304 [I 2022-10-26 11:41:24] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:02:26 [I 2022-10-26 11:41:24] ...trap/engines/engine.py.229: process: 0.49538 | load: 0.00250 [I 2022-10-26 11:41:24] ...trap/engines/engine.py.230: loss: 0.26095 [I 2022-10-26 11:41:24] ...optimizers/trijoint.py.151: total_norm: 4.007466 [I 2022-10-26 11:41:29] ...trap/engines/engine.py.224: train: epoch 5 | batch 2020/2304 [I 2022-10-26 11:41:29] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:02:23 [I 2022-10-26 11:41:29] ...trap/engines/engine.py.229: process: 0.49737 | load: 0.00246 [I 2022-10-26 11:41:29] ...trap/engines/engine.py.230: loss: 0.28900 [I 2022-10-26 11:41:29] ...optimizers/trijoint.py.151: total_norm: 5.161914 [I 2022-10-26 11:41:34] ...trap/engines/engine.py.224: train: epoch 5 | batch 2030/2304 [I 2022-10-26 11:41:34] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:02:17 [I 2022-10-26 11:41:34] ...trap/engines/engine.py.229: process: 0.49656 | load: 0.00254 [I 2022-10-26 11:41:34] ...trap/engines/engine.py.230: loss: 0.38361 [I 2022-10-26 11:41:34] ...optimizers/trijoint.py.151: total_norm: 4.812560 [I 2022-10-26 11:41:39] ...trap/engines/engine.py.224: train: epoch 5 | batch 2040/2304 [I 2022-10-26 11:41:39] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:02:12 [I 2022-10-26 11:41:39] ...trap/engines/engine.py.229: process: 0.49165 | load: 0.00245 [I 2022-10-26 11:41:39] ...trap/engines/engine.py.230: loss: 0.34040 [I 2022-10-26 11:41:39] ...optimizers/trijoint.py.151: total_norm: 3.970344 [I 2022-10-26 11:41:44] ...trap/engines/engine.py.224: train: epoch 5 | batch 2050/2304 [I 2022-10-26 11:41:44] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:02:07 [I 2022-10-26 11:41:44] ...trap/engines/engine.py.229: process: 0.49689 | load: 0.00248 [I 2022-10-26 11:41:44] ...trap/engines/engine.py.230: loss: 0.25827 [I 2022-10-26 11:41:44] ...optimizers/trijoint.py.151: total_norm: 2.476398 [I 2022-10-26 11:41:49] ...trap/engines/engine.py.224: train: epoch 5 | batch 2060/2304 [I 2022-10-26 11:41:49] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:02:02 [I 2022-10-26 11:41:49] ...trap/engines/engine.py.229: process: 0.50460 | load: 0.00300 [I 2022-10-26 11:41:49] ...trap/engines/engine.py.230: loss: 0.29426 [I 2022-10-26 11:41:49] ...optimizers/trijoint.py.151: total_norm: 3.532506 [I 2022-10-26 11:41:54] ...trap/engines/engine.py.224: train: epoch 5 | batch 2070/2304 [I 2022-10-26 11:41:54] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:01:57 [I 2022-10-26 11:41:54] ...trap/engines/engine.py.229: process: 0.49865 | load: 0.00254 [I 2022-10-26 11:41:54] ...trap/engines/engine.py.230: loss: 0.29694 [I 2022-10-26 11:41:54] ...optimizers/trijoint.py.151: total_norm: 3.548334 [I 2022-10-26 11:41:59] ...trap/engines/engine.py.224: train: epoch 5 | batch 2080/2304 [I 2022-10-26 11:41:59] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:01:51 [I 2022-10-26 11:41:59] ...trap/engines/engine.py.229: process: 0.51055 | load: 0.00262 [I 2022-10-26 11:41:59] ...trap/engines/engine.py.230: loss: 0.21544 [I 2022-10-26 11:41:59] ...optimizers/trijoint.py.151: total_norm: 3.270838 [I 2022-10-26 11:42:04] ...trap/engines/engine.py.224: train: epoch 5 | batch 2090/2304 [I 2022-10-26 11:42:04] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:01:47 [I 2022-10-26 11:42:04] ...trap/engines/engine.py.229: process: 0.50301 | load: 0.00248 [I 2022-10-26 11:42:04] ...trap/engines/engine.py.230: loss: 0.27383 [I 2022-10-26 11:42:04] ...optimizers/trijoint.py.151: total_norm: 3.523932 [I 2022-10-26 11:42:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 2100/2304 [I 2022-10-26 11:42:09] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:01:42 [I 2022-10-26 11:42:09] ...trap/engines/engine.py.229: process: 0.50100 | load: 0.00254 [I 2022-10-26 11:42:09] ...trap/engines/engine.py.230: loss: 0.25708 [I 2022-10-26 11:42:09] ...optimizers/trijoint.py.151: total_norm: 3.295598 [I 2022-10-26 11:42:14] ...trap/engines/engine.py.224: train: epoch 5 | batch 2110/2304 [I 2022-10-26 11:42:14] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:01:37 [I 2022-10-26 11:42:14] ...trap/engines/engine.py.229: process: 0.50192 | load: 0.00256 [I 2022-10-26 11:42:14] ...trap/engines/engine.py.230: loss: 0.28363 [I 2022-10-26 11:42:14] ...optimizers/trijoint.py.151: total_norm: 3.569071 [I 2022-10-26 11:42:19] ...trap/engines/engine.py.224: train: epoch 5 | batch 2120/2304 [I 2022-10-26 11:42:19] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:01:31 [I 2022-10-26 11:42:19] ...trap/engines/engine.py.229: process: 0.49975 | load: 0.00247 [I 2022-10-26 11:42:19] ...trap/engines/engine.py.230: loss: 0.29204 [I 2022-10-26 11:42:19] ...optimizers/trijoint.py.151: total_norm: 3.190688 [I 2022-10-26 11:42:24] ...trap/engines/engine.py.224: train: epoch 5 | batch 2130/2304 [I 2022-10-26 11:42:24] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:01:26 [I 2022-10-26 11:42:24] ...trap/engines/engine.py.229: process: 0.48621 | load: 0.00244 [I 2022-10-26 11:42:24] ...trap/engines/engine.py.230: loss: 0.28276 [I 2022-10-26 11:42:24] ...optimizers/trijoint.py.151: total_norm: 4.412430 [I 2022-10-26 11:42:29] ...trap/engines/engine.py.224: train: epoch 5 | batch 2140/2304 [I 2022-10-26 11:42:29] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:01:21 [I 2022-10-26 11:42:29] ...trap/engines/engine.py.229: process: 0.50035 | load: 0.00244 [I 2022-10-26 11:42:29] ...trap/engines/engine.py.230: loss: 0.25963 [I 2022-10-26 11:42:29] ...optimizers/trijoint.py.151: total_norm: 3.834741 [I 2022-10-26 11:42:34] ...trap/engines/engine.py.224: train: epoch 5 | batch 2150/2304 [I 2022-10-26 11:42:34] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:01:16 [I 2022-10-26 11:42:34] ...trap/engines/engine.py.229: process: 0.48761 | load: 0.00251 [I 2022-10-26 11:42:34] ...trap/engines/engine.py.230: loss: 0.31125 [I 2022-10-26 11:42:34] ...optimizers/trijoint.py.151: total_norm: 3.429585 [I 2022-10-26 11:42:39] ...trap/engines/engine.py.224: train: epoch 5 | batch 2160/2304 [I 2022-10-26 11:42:39] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:01:11 [I 2022-10-26 11:42:39] ...trap/engines/engine.py.229: process: 0.48834 | load: 0.00248 [I 2022-10-26 11:42:39] ...trap/engines/engine.py.230: loss: 0.32632 [I 2022-10-26 11:42:39] ...optimizers/trijoint.py.151: total_norm: 3.473933 [I 2022-10-26 11:42:44] ...trap/engines/engine.py.224: train: epoch 5 | batch 2170/2304 [I 2022-10-26 11:42:44] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:01:06 [I 2022-10-26 11:42:44] ...trap/engines/engine.py.229: process: 0.49658 | load: 0.00242 [I 2022-10-26 11:42:44] ...trap/engines/engine.py.230: loss: 0.27332 [I 2022-10-26 11:42:44] ...optimizers/trijoint.py.151: total_norm: 3.652030 [I 2022-10-26 11:42:49] ...trap/engines/engine.py.224: train: epoch 5 | batch 2180/2304 [I 2022-10-26 11:42:49] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:01:01 [I 2022-10-26 11:42:49] ...trap/engines/engine.py.229: process: 0.48922 | load: 0.00242 [I 2022-10-26 11:42:49] ...trap/engines/engine.py.230: loss: 0.34743 [I 2022-10-26 11:42:49] ...optimizers/trijoint.py.151: total_norm: 3.297950 [I 2022-10-26 11:42:54] ...trap/engines/engine.py.224: train: epoch 5 | batch 2190/2304 [I 2022-10-26 11:42:54] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:00:56 [I 2022-10-26 11:42:54] ...trap/engines/engine.py.229: process: 0.48875 | load: 0.00242 [I 2022-10-26 11:42:54] ...trap/engines/engine.py.230: loss: 0.27898 [I 2022-10-26 11:42:54] ...optimizers/trijoint.py.151: total_norm: 3.355265 [I 2022-10-26 11:42:59] ...trap/engines/engine.py.224: train: epoch 5 | batch 2200/2304 [I 2022-10-26 11:42:59] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:00:51 [I 2022-10-26 11:42:59] ...trap/engines/engine.py.229: process: 0.49428 | load: 0.00250 [I 2022-10-26 11:42:59] ...trap/engines/engine.py.230: loss: 0.26948 [I 2022-10-26 11:42:59] ...optimizers/trijoint.py.151: total_norm: 3.965106 [I 2022-10-26 11:43:04] ...trap/engines/engine.py.224: train: epoch 5 | batch 2210/2304 [I 2022-10-26 11:43:04] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:00:46 [I 2022-10-26 11:43:04] ...trap/engines/engine.py.229: process: 0.50368 | load: 0.00247 [I 2022-10-26 11:43:04] ...trap/engines/engine.py.230: loss: 0.23492 [I 2022-10-26 11:43:04] ...optimizers/trijoint.py.151: total_norm: 3.686142 [I 2022-10-26 11:43:09] ...trap/engines/engine.py.224: train: epoch 5 | batch 2220/2304 [I 2022-10-26 11:43:09] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:00:42 [I 2022-10-26 11:43:09] ...trap/engines/engine.py.229: process: 0.49205 | load: 0.00251 [I 2022-10-26 11:43:09] ...trap/engines/engine.py.230: loss: 0.34148 [I 2022-10-26 11:43:09] ...optimizers/trijoint.py.151: total_norm: 3.371651 [I 2022-10-26 11:43:14] ...trap/engines/engine.py.224: train: epoch 5 | batch 2230/2304 [I 2022-10-26 11:43:14] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:00:37 [I 2022-10-26 11:43:14] ...trap/engines/engine.py.229: process: 0.50799 | load: 0.00250 [I 2022-10-26 11:43:14] ...trap/engines/engine.py.230: loss: 0.24681 [I 2022-10-26 11:43:14] ...optimizers/trijoint.py.151: total_norm: 2.664953 [I 2022-10-26 11:43:19] ...trap/engines/engine.py.224: train: epoch 5 | batch 2240/2304 [I 2022-10-26 11:43:19] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:00:31 [I 2022-10-26 11:43:19] ...trap/engines/engine.py.229: process: 0.49931 | load: 0.00245 [I 2022-10-26 11:43:19] ...trap/engines/engine.py.230: loss: 0.34279 [I 2022-10-26 11:43:19] ...optimizers/trijoint.py.151: total_norm: 3.598917 [I 2022-10-26 11:43:24] ...trap/engines/engine.py.224: train: epoch 5 | batch 2250/2304 [I 2022-10-26 11:43:24] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:00:26 [I 2022-10-26 11:43:24] ...trap/engines/engine.py.229: process: 0.49716 | load: 0.00241 [I 2022-10-26 11:43:24] ...trap/engines/engine.py.230: loss: 0.23322 [I 2022-10-26 11:43:24] ...optimizers/trijoint.py.151: total_norm: 3.349577 [I 2022-10-26 11:43:29] ...trap/engines/engine.py.224: train: epoch 5 | batch 2260/2304 [I 2022-10-26 11:43:29] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:00:22 [I 2022-10-26 11:43:29] ...trap/engines/engine.py.229: process: 0.50690 | load: 0.00243 [I 2022-10-26 11:43:29] ...trap/engines/engine.py.230: loss: 0.24059 [I 2022-10-26 11:43:29] ...optimizers/trijoint.py.151: total_norm: 3.087265 [I 2022-10-26 11:43:34] ...trap/engines/engine.py.224: train: epoch 5 | batch 2270/2304 [I 2022-10-26 11:43:34] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:00:16 [I 2022-10-26 11:43:34] ...trap/engines/engine.py.229: process: 0.50244 | load: 0.00244 [I 2022-10-26 11:43:34] ...trap/engines/engine.py.230: loss: 0.28635 [I 2022-10-26 11:43:34] ...optimizers/trijoint.py.151: total_norm: 4.193513 [I 2022-10-26 11:43:39] ...trap/engines/engine.py.224: train: epoch 5 | batch 2280/2304 [I 2022-10-26 11:43:39] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:00:11 [I 2022-10-26 11:43:39] ...trap/engines/engine.py.229: process: 0.49806 | load: 0.00246 [I 2022-10-26 11:43:39] ...trap/engines/engine.py.230: loss: 0.23746 [I 2022-10-26 11:43:39] ...optimizers/trijoint.py.151: total_norm: 3.621052 [I 2022-10-26 11:43:44] ...trap/engines/engine.py.224: train: epoch 5 | batch 2290/2304 [I 2022-10-26 11:43:44] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:00:07 [I 2022-10-26 11:43:44] ...trap/engines/engine.py.229: process: 0.50297 | load: 0.00242 [I 2022-10-26 11:43:44] ...trap/engines/engine.py.230: loss: 0.30269 [I 2022-10-26 11:43:44] ...optimizers/trijoint.py.151: total_norm: 3.644756 [I 2022-10-26 11:43:49] ...trap/engines/engine.py.224: train: epoch 5 | batch 2300/2304 [I 2022-10-26 11:43:49] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:00:02 [I 2022-10-26 11:43:49] ...trap/engines/engine.py.229: process: 0.49963 | load: 0.00015 [I 2022-10-26 11:43:49] ...trap/engines/engine.py.230: loss: 0.33469 [I 2022-10-26 11:43:49] ...optimizers/trijoint.py.151: total_norm: 3.521852 [I 2022-10-26 11:43:51] ...trap/engines/engine.py.224: train: epoch 5 | batch 2304/2304 [I 2022-10-26 11:43:51] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:00:00 [I 2022-10-26 11:43:51] ...trap/engines/engine.py.229: process: 0.49892 | load: 0.00015 [I 2022-10-26 11:43:51] ...trap/engines/engine.py.230: loss: 0.30036 [I 2022-10-26 11:43:51] ...optimizers/trijoint.py.151: total_norm: 3.036632 [S 2022-10-26 11:43:52] ...trap/engines/engine.py.129: train_epoch.epoch: 5 [S 2022-10-26 11:43:52] ...trap/engines/engine.py.129: train_epoch.loss: 0.2771904367945458 [I 2022-10-26 11:43:52] ...s/criterions/triplet.py.84: margin = 0.08000000000000002 [I 2022-10-26 11:43:52] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 5 [I 2022-10-26 11:43:52] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 11:43:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 11:43:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 11:43:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 11:43:54] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 11:43:55] ...trap/engines/engine.py.321: eval: epoch 5 | batch 0/510 [I 2022-10-26 11:43:55] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:16 [I 2022-10-26 11:43:55] ...trap/engines/engine.py.326: process: 2.62132 | load: 2.25883 [I 2022-10-26 11:43:58] ...trap/engines/engine.py.321: eval: epoch 5 | batch 10/510 [I 2022-10-26 11:43:58] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:04:50 [I 2022-10-26 11:43:58] ...trap/engines/engine.py.326: process: 0.35410 | load: 0.00018 [I 2022-10-26 11:44:02] ...trap/engines/engine.py.321: eval: epoch 5 | batch 20/510 [I 2022-10-26 11:44:02] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:56 [I 2022-10-26 11:44:02] ...trap/engines/engine.py.326: process: 0.32467 | load: 0.00017 [I 2022-10-26 11:44:05] ...trap/engines/engine.py.321: eval: epoch 5 | batch 30/510 [I 2022-10-26 11:44:05] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:40 [I 2022-10-26 11:44:05] ...trap/engines/engine.py.326: process: 0.32811 | load: 0.00017 [I 2022-10-26 11:44:08] ...trap/engines/engine.py.321: eval: epoch 5 | batch 40/510 [I 2022-10-26 11:44:08] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:36 [I 2022-10-26 11:44:08] ...trap/engines/engine.py.326: process: 0.33298 | load: 0.00018 [I 2022-10-26 11:44:12] ...trap/engines/engine.py.321: eval: epoch 5 | batch 50/510 [I 2022-10-26 11:44:12] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:37 [I 2022-10-26 11:44:12] ...trap/engines/engine.py.326: process: 0.33544 | load: 0.00018 [I 2022-10-26 11:44:15] ...trap/engines/engine.py.321: eval: epoch 5 | batch 60/510 [I 2022-10-26 11:44:15] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:35 [I 2022-10-26 11:44:15] ...trap/engines/engine.py.326: process: 0.36402 | load: 0.00018 [I 2022-10-26 11:44:19] ...trap/engines/engine.py.321: eval: epoch 5 | batch 70/510 [I 2022-10-26 11:44:19] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-26 11:44:19] ...trap/engines/engine.py.326: process: 0.32700 | load: 0.00018 [I 2022-10-26 11:44:22] ...trap/engines/engine.py.321: eval: epoch 5 | batch 80/510 [I 2022-10-26 11:44:22] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:25 [I 2022-10-26 11:44:22] ...trap/engines/engine.py.326: process: 0.32218 | load: 0.00021 [I 2022-10-26 11:44:25] ...trap/engines/engine.py.321: eval: epoch 5 | batch 90/510 [I 2022-10-26 11:44:25] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:24 [I 2022-10-26 11:44:25] ...trap/engines/engine.py.326: process: 0.35394 | load: 0.00020 [I 2022-10-26 11:44:29] ...trap/engines/engine.py.321: eval: epoch 5 | batch 100/510 [I 2022-10-26 11:44:29] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:16 [I 2022-10-26 11:44:29] ...trap/engines/engine.py.326: process: 0.29273 | load: 0.00017 [I 2022-10-26 11:44:32] ...trap/engines/engine.py.321: eval: epoch 5 | batch 110/510 [I 2022-10-26 11:44:32] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:58 [I 2022-10-26 11:44:32] ...trap/engines/engine.py.326: process: 0.28426 | load: 0.00022 [I 2022-10-26 11:44:35] ...trap/engines/engine.py.321: eval: epoch 5 | batch 120/510 [I 2022-10-26 11:44:35] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-26 11:44:35] ...trap/engines/engine.py.326: process: 0.29063 | load: 0.00015 [I 2022-10-26 11:44:37] ...trap/engines/engine.py.321: eval: epoch 5 | batch 130/510 [I 2022-10-26 11:44:37] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 11:44:37] ...trap/engines/engine.py.326: process: 0.28208 | load: 0.00019 [I 2022-10-26 11:44:40] ...trap/engines/engine.py.321: eval: epoch 5 | batch 140/510 [I 2022-10-26 11:44:40] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:48 [I 2022-10-26 11:44:40] ...trap/engines/engine.py.326: process: 0.28615 | load: 0.00016 [I 2022-10-26 11:44:43] ...trap/engines/engine.py.321: eval: epoch 5 | batch 150/510 [I 2022-10-26 11:44:43] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:45 [I 2022-10-26 11:44:43] ...trap/engines/engine.py.326: process: 0.28787 | load: 0.00019 [I 2022-10-26 11:44:46] ...trap/engines/engine.py.321: eval: epoch 5 | batch 160/510 [I 2022-10-26 11:44:46] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 11:44:46] ...trap/engines/engine.py.326: process: 0.28505 | load: 0.00014 [I 2022-10-26 11:44:49] ...trap/engines/engine.py.321: eval: epoch 5 | batch 170/510 [I 2022-10-26 11:44:49] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 11:44:49] ...trap/engines/engine.py.326: process: 0.29060 | load: 0.00018 [I 2022-10-26 11:44:52] ...trap/engines/engine.py.321: eval: epoch 5 | batch 180/510 [I 2022-10-26 11:44:52] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:36 [I 2022-10-26 11:44:52] ...trap/engines/engine.py.326: process: 0.29279 | load: 0.00017 [I 2022-10-26 11:44:55] ...trap/engines/engine.py.321: eval: epoch 5 | batch 190/510 [I 2022-10-26 11:44:55] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-26 11:44:55] ...trap/engines/engine.py.326: process: 0.30405 | load: 0.00038 [I 2022-10-26 11:44:58] ...trap/engines/engine.py.321: eval: epoch 5 | batch 200/510 [I 2022-10-26 11:44:58] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 11:44:58] ...trap/engines/engine.py.326: process: 0.28230 | load: 0.00014 [I 2022-10-26 11:45:01] ...trap/engines/engine.py.321: eval: epoch 5 | batch 210/510 [I 2022-10-26 11:45:01] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:28 [I 2022-10-26 11:45:01] ...trap/engines/engine.py.326: process: 0.30037 | load: 0.00019 [I 2022-10-26 11:45:04] ...trap/engines/engine.py.321: eval: epoch 5 | batch 220/510 [I 2022-10-26 11:45:04] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-26 11:45:04] ...trap/engines/engine.py.326: process: 0.31131 | load: 0.00020 [I 2022-10-26 11:45:07] ...trap/engines/engine.py.321: eval: epoch 5 | batch 230/510 [I 2022-10-26 11:45:07] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-26 11:45:07] ...trap/engines/engine.py.326: process: 0.28829 | load: 0.00019 [I 2022-10-26 11:45:10] ...trap/engines/engine.py.321: eval: epoch 5 | batch 240/510 [I 2022-10-26 11:45:10] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:19 [I 2022-10-26 11:45:10] ...trap/engines/engine.py.326: process: 0.29774 | load: 0.00018 [I 2022-10-26 11:45:13] ...trap/engines/engine.py.321: eval: epoch 5 | batch 250/510 [I 2022-10-26 11:45:13] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-26 11:45:13] ...trap/engines/engine.py.326: process: 0.29094 | load: 0.00021 [I 2022-10-26 11:45:16] ...trap/engines/engine.py.321: eval: epoch 5 | batch 260/510 [I 2022-10-26 11:45:16] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:13 [I 2022-10-26 11:45:16] ...trap/engines/engine.py.326: process: 0.28817 | load: 0.00017 [I 2022-10-26 11:45:18] ...trap/engines/engine.py.321: eval: epoch 5 | batch 270/510 [I 2022-10-26 11:45:18] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-26 11:45:18] ...trap/engines/engine.py.326: process: 0.28593 | load: 0.00016 [I 2022-10-26 11:45:21] ...trap/engines/engine.py.321: eval: epoch 5 | batch 280/510 [I 2022-10-26 11:45:21] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-26 11:45:21] ...trap/engines/engine.py.326: process: 0.29316 | load: 0.00029 [I 2022-10-26 11:45:24] ...trap/engines/engine.py.321: eval: epoch 5 | batch 290/510 [I 2022-10-26 11:45:24] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 11:45:24] ...trap/engines/engine.py.326: process: 0.28322 | load: 0.00015 [I 2022-10-26 11:45:27] ...trap/engines/engine.py.321: eval: epoch 5 | batch 300/510 [I 2022-10-26 11:45:27] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 11:45:27] ...trap/engines/engine.py.326: process: 0.28131 | load: 0.00015 [I 2022-10-26 11:45:30] ...trap/engines/engine.py.321: eval: epoch 5 | batch 310/510 [I 2022-10-26 11:45:30] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:58 [I 2022-10-26 11:45:30] ...trap/engines/engine.py.326: process: 0.30536 | load: 0.00016 [I 2022-10-26 11:45:33] ...trap/engines/engine.py.321: eval: epoch 5 | batch 320/510 [I 2022-10-26 11:45:33] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 11:45:33] ...trap/engines/engine.py.326: process: 0.29237 | load: 0.00020 [I 2022-10-26 11:45:36] ...trap/engines/engine.py.321: eval: epoch 5 | batch 330/510 [I 2022-10-26 11:45:36] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 11:45:36] ...trap/engines/engine.py.326: process: 0.28483 | load: 0.00019 [I 2022-10-26 11:45:39] ...trap/engines/engine.py.321: eval: epoch 5 | batch 340/510 [I 2022-10-26 11:45:39] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 11:45:39] ...trap/engines/engine.py.326: process: 0.28538 | load: 0.00019 [I 2022-10-26 11:45:42] ...trap/engines/engine.py.321: eval: epoch 5 | batch 350/510 [I 2022-10-26 11:45:42] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 11:45:42] ...trap/engines/engine.py.326: process: 0.28587 | load: 0.00028 [I 2022-10-26 11:45:45] ...trap/engines/engine.py.321: eval: epoch 5 | batch 360/510 [I 2022-10-26 11:45:45] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 11:45:45] ...trap/engines/engine.py.326: process: 0.29102 | load: 0.00015 [I 2022-10-26 11:45:47] ...trap/engines/engine.py.321: eval: epoch 5 | batch 370/510 [I 2022-10-26 11:45:47] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-26 11:45:47] ...trap/engines/engine.py.326: process: 0.29079 | load: 0.00016 [I 2022-10-26 11:45:50] ...trap/engines/engine.py.321: eval: epoch 5 | batch 380/510 [I 2022-10-26 11:45:50] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 11:45:50] ...trap/engines/engine.py.326: process: 0.29030 | load: 0.00014 [I 2022-10-26 11:45:53] ...trap/engines/engine.py.321: eval: epoch 5 | batch 390/510 [I 2022-10-26 11:45:53] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:35 [I 2022-10-26 11:45:53] ...trap/engines/engine.py.326: process: 0.31255 | load: 0.00016 [I 2022-10-26 11:45:56] ...trap/engines/engine.py.321: eval: epoch 5 | batch 400/510 [I 2022-10-26 11:45:56] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 11:45:56] ...trap/engines/engine.py.326: process: 0.29154 | load: 0.00017 [I 2022-10-26 11:45:59] ...trap/engines/engine.py.321: eval: epoch 5 | batch 410/510 [I 2022-10-26 11:45:59] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 11:45:59] ...trap/engines/engine.py.326: process: 0.28811 | load: 0.00019 [I 2022-10-26 11:46:02] ...trap/engines/engine.py.321: eval: epoch 5 | batch 420/510 [I 2022-10-26 11:46:02] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 11:46:02] ...trap/engines/engine.py.326: process: 0.28778 | load: 0.00020 [I 2022-10-26 11:46:05] ...trap/engines/engine.py.321: eval: epoch 5 | batch 430/510 [I 2022-10-26 11:46:05] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 11:46:05] ...trap/engines/engine.py.326: process: 0.27962 | load: 0.00019 [I 2022-10-26 11:46:08] ...trap/engines/engine.py.321: eval: epoch 5 | batch 440/510 [I 2022-10-26 11:46:08] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 11:46:08] ...trap/engines/engine.py.326: process: 0.28696 | load: 0.00021 [I 2022-10-26 11:46:10] ...trap/engines/engine.py.321: eval: epoch 5 | batch 450/510 [I 2022-10-26 11:46:10] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 11:46:10] ...trap/engines/engine.py.326: process: 0.28553 | load: 0.00035 [I 2022-10-26 11:46:13] ...trap/engines/engine.py.321: eval: epoch 5 | batch 460/510 [I 2022-10-26 11:46:13] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-26 11:46:13] ...trap/engines/engine.py.326: process: 0.28715 | load: 0.00016 [I 2022-10-26 11:46:16] ...trap/engines/engine.py.321: eval: epoch 5 | batch 470/510 [I 2022-10-26 11:46:16] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 11:46:16] ...trap/engines/engine.py.326: process: 0.29533 | load: 0.00019 [I 2022-10-26 11:46:19] ...trap/engines/engine.py.321: eval: epoch 5 | batch 480/510 [I 2022-10-26 11:46:19] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 11:46:19] ...trap/engines/engine.py.326: process: 0.28706 | load: 0.00021 [I 2022-10-26 11:46:22] ...trap/engines/engine.py.321: eval: epoch 5 | batch 490/510 [I 2022-10-26 11:46:22] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 11:46:22] ...trap/engines/engine.py.326: process: 0.28541 | load: 0.00016 [I 2022-10-26 11:46:25] ...trap/engines/engine.py.321: eval: epoch 5 | batch 500/510 [I 2022-10-26 11:46:25] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 11:46:25] ...trap/engines/engine.py.326: process: 0.28702 | load: 0.00020 [I 2022-10-26 11:46:28] ...trap/engines/engine.py.321: eval: epoch 5 | batch 510/510 [I 2022-10-26 11:46:28] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 11:46:28] ...trap/engines/engine.py.326: process: 0.28988 | load: 0.00011 [S 2022-10-26 11:46:29] ...trap/engines/engine.py.132: eval_epoch.epoch: 5 [I 2022-10-26 11:46:29] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.00 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.438100 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.752500 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.844700 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.449600 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.765800 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.847800 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014686 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011111 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010354 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010716 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011347 [S 2022-10-26 11:46:33] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.011948 [I 2022-10-26 11:46:34] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 11:46:34] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 11:46:34] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 11:46:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 11:46:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 11:46:34] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 11:46:37] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 11:46:42] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 11:46:42] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 11:46:42] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 11:46:45] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 11:46:50] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 11:46:50] ...trap/engines/engine.py.163: Training model on trainset for epoch 6 [I 2022-10-26 11:46:50] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 11:46:53] ...trap/engines/engine.py.224: train: epoch 6 | batch 0/2304 [I 2022-10-26 11:46:53] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:31 [I 2022-10-26 11:46:53] ...trap/engines/engine.py.229: process: 2.77423 | load: 2.15993 [I 2022-10-26 11:46:53] ...trap/engines/engine.py.230: loss: 0.22975 [I 2022-10-26 11:46:53] ...optimizers/trijoint.py.151: total_norm: 2.894983 [I 2022-10-26 11:46:58] ...trap/engines/engine.py.224: train: epoch 6 | batch 10/2304 [I 2022-10-26 11:46:58] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:36 [I 2022-10-26 11:46:58] ...trap/engines/engine.py.229: process: 0.50428 | load: 0.00345 [I 2022-10-26 11:46:58] ...trap/engines/engine.py.230: loss: 0.24208 [I 2022-10-26 11:46:58] ...optimizers/trijoint.py.151: total_norm: 3.349245 [I 2022-10-26 11:47:03] ...trap/engines/engine.py.224: train: epoch 6 | batch 20/2304 [I 2022-10-26 11:47:03] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:19:55 [I 2022-10-26 11:47:03] ...trap/engines/engine.py.229: process: 0.49693 | load: 0.00353 [I 2022-10-26 11:47:03] ...trap/engines/engine.py.230: loss: 0.21662 [I 2022-10-26 11:47:03] ...optimizers/trijoint.py.151: total_norm: 2.720183 [I 2022-10-26 11:47:08] ...trap/engines/engine.py.224: train: epoch 6 | batch 30/2304 [I 2022-10-26 11:47:08] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:18:51 [I 2022-10-26 11:47:08] ...trap/engines/engine.py.229: process: 0.47854 | load: 0.00300 [I 2022-10-26 11:47:08] ...trap/engines/engine.py.230: loss: 0.35866 [I 2022-10-26 11:47:08] ...optimizers/trijoint.py.151: total_norm: 3.025339 [I 2022-10-26 11:47:13] ...trap/engines/engine.py.224: train: epoch 6 | batch 40/2304 [I 2022-10-26 11:47:13] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:44 [I 2022-10-26 11:47:13] ...trap/engines/engine.py.229: process: 0.48653 | load: 0.00302 [I 2022-10-26 11:47:13] ...trap/engines/engine.py.230: loss: 0.21489 [I 2022-10-26 11:47:13] ...optimizers/trijoint.py.151: total_norm: 3.791904 [I 2022-10-26 11:47:18] ...trap/engines/engine.py.224: train: epoch 6 | batch 50/2304 [I 2022-10-26 11:47:18] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:48 [I 2022-10-26 11:47:18] ...trap/engines/engine.py.229: process: 0.50034 | load: 0.00289 [I 2022-10-26 11:47:18] ...trap/engines/engine.py.230: loss: 0.26713 [I 2022-10-26 11:47:18] ...optimizers/trijoint.py.151: total_norm: 3.797714 [I 2022-10-26 11:47:23] ...trap/engines/engine.py.224: train: epoch 6 | batch 60/2304 [I 2022-10-26 11:47:23] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:41 [I 2022-10-26 11:47:23] ...trap/engines/engine.py.229: process: 0.48919 | load: 0.00294 [I 2022-10-26 11:47:23] ...trap/engines/engine.py.230: loss: 0.31769 [I 2022-10-26 11:47:23] ...optimizers/trijoint.py.151: total_norm: 3.693207 [I 2022-10-26 11:47:28] ...trap/engines/engine.py.224: train: epoch 6 | batch 70/2304 [I 2022-10-26 11:47:28] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:30 [I 2022-10-26 11:47:28] ...trap/engines/engine.py.229: process: 0.50057 | load: 0.00300 [I 2022-10-26 11:47:28] ...trap/engines/engine.py.230: loss: 0.19705 [I 2022-10-26 11:47:28] ...optimizers/trijoint.py.151: total_norm: 3.470104 [I 2022-10-26 11:47:33] ...trap/engines/engine.py.224: train: epoch 6 | batch 80/2304 [I 2022-10-26 11:47:33] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:18:37 [I 2022-10-26 11:47:33] ...trap/engines/engine.py.229: process: 0.48758 | load: 0.00277 [I 2022-10-26 11:47:33] ...trap/engines/engine.py.230: loss: 0.21942 [I 2022-10-26 11:47:33] ...optimizers/trijoint.py.151: total_norm: 3.441072 [I 2022-10-26 11:47:38] ...trap/engines/engine.py.224: train: epoch 6 | batch 90/2304 [I 2022-10-26 11:47:38] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:18:27 [I 2022-10-26 11:47:38] ...trap/engines/engine.py.229: process: 0.49977 | load: 0.00279 [I 2022-10-26 11:47:38] ...trap/engines/engine.py.230: loss: 0.30294 [I 2022-10-26 11:47:38] ...optimizers/trijoint.py.151: total_norm: 4.451406 [I 2022-10-26 11:47:43] ...trap/engines/engine.py.224: train: epoch 6 | batch 100/2304 [I 2022-10-26 11:47:43] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:18:16 [I 2022-10-26 11:47:43] ...trap/engines/engine.py.229: process: 0.49471 | load: 0.00260 [I 2022-10-26 11:47:43] ...trap/engines/engine.py.230: loss: 0.35610 [I 2022-10-26 11:47:43] ...optimizers/trijoint.py.151: total_norm: 3.710003 [I 2022-10-26 11:47:48] ...trap/engines/engine.py.224: train: epoch 6 | batch 110/2304 [I 2022-10-26 11:47:48] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:18:10 [I 2022-10-26 11:47:48] ...trap/engines/engine.py.229: process: 0.49442 | load: 0.00263 [I 2022-10-26 11:47:48] ...trap/engines/engine.py.230: loss: 0.25109 [I 2022-10-26 11:47:48] ...optimizers/trijoint.py.151: total_norm: 4.270509 [I 2022-10-26 11:47:53] ...trap/engines/engine.py.224: train: epoch 6 | batch 120/2304 [I 2022-10-26 11:47:53] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:18:11 [I 2022-10-26 11:47:53] ...trap/engines/engine.py.229: process: 0.50323 | load: 0.00267 [I 2022-10-26 11:47:53] ...trap/engines/engine.py.230: loss: 0.25609 [I 2022-10-26 11:47:53] ...optimizers/trijoint.py.151: total_norm: 3.333663 [I 2022-10-26 11:47:58] ...trap/engines/engine.py.224: train: epoch 6 | batch 130/2304 [I 2022-10-26 11:47:58] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:15 [I 2022-10-26 11:47:58] ...trap/engines/engine.py.229: process: 0.50193 | load: 0.00281 [I 2022-10-26 11:47:58] ...trap/engines/engine.py.230: loss: 0.31770 [I 2022-10-26 11:47:58] ...optimizers/trijoint.py.151: total_norm: 3.598657 [I 2022-10-26 11:48:03] ...trap/engines/engine.py.224: train: epoch 6 | batch 140/2304 [I 2022-10-26 11:48:03] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:17:56 [I 2022-10-26 11:48:03] ...trap/engines/engine.py.229: process: 0.49080 | load: 0.00251 [I 2022-10-26 11:48:03] ...trap/engines/engine.py.230: loss: 0.33961 [I 2022-10-26 11:48:03] ...optimizers/trijoint.py.151: total_norm: 3.933886 [I 2022-10-26 11:48:08] ...trap/engines/engine.py.224: train: epoch 6 | batch 150/2304 [I 2022-10-26 11:48:08] ...trap/engines/engine.py.228: elapsed: 0:01:17 | left: 0:17:48 [I 2022-10-26 11:48:08] ...trap/engines/engine.py.229: process: 0.49973 | load: 0.00259 [I 2022-10-26 11:48:08] ...trap/engines/engine.py.230: loss: 0.21495 [I 2022-10-26 11:48:08] ...optimizers/trijoint.py.151: total_norm: 3.333372 [I 2022-10-26 11:48:13] ...trap/engines/engine.py.224: train: epoch 6 | batch 160/2304 [I 2022-10-26 11:48:13] ...trap/engines/engine.py.228: elapsed: 0:01:22 | left: 0:17:55 [I 2022-10-26 11:48:13] ...trap/engines/engine.py.229: process: 0.52323 | load: 0.00267 [I 2022-10-26 11:48:13] ...trap/engines/engine.py.230: loss: 0.22008 [I 2022-10-26 11:48:13] ...optimizers/trijoint.py.151: total_norm: 3.080223 [I 2022-10-26 11:48:18] ...trap/engines/engine.py.224: train: epoch 6 | batch 170/2304 [I 2022-10-26 11:48:18] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:17:51 [I 2022-10-26 11:48:18] ...trap/engines/engine.py.229: process: 0.51700 | load: 0.00254 [I 2022-10-26 11:48:18] ...trap/engines/engine.py.230: loss: 0.25527 [I 2022-10-26 11:48:18] ...optimizers/trijoint.py.151: total_norm: 3.740395 [I 2022-10-26 11:48:23] ...trap/engines/engine.py.224: train: epoch 6 | batch 180/2304 [I 2022-10-26 11:48:23] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:17:33 [I 2022-10-26 11:48:23] ...trap/engines/engine.py.229: process: 0.49291 | load: 0.00251 [I 2022-10-26 11:48:23] ...trap/engines/engine.py.230: loss: 0.28785 [I 2022-10-26 11:48:23] ...optimizers/trijoint.py.151: total_norm: 3.702152 [I 2022-10-26 11:48:28] ...trap/engines/engine.py.224: train: epoch 6 | batch 190/2304 [I 2022-10-26 11:48:28] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:17:33 [I 2022-10-26 11:48:28] ...trap/engines/engine.py.229: process: 0.50176 | load: 0.00266 [I 2022-10-26 11:48:28] ...trap/engines/engine.py.230: loss: 0.37135 [I 2022-10-26 11:48:28] ...optimizers/trijoint.py.151: total_norm: 4.167512 [I 2022-10-26 11:48:33] ...trap/engines/engine.py.224: train: epoch 6 | batch 200/2304 [I 2022-10-26 11:48:33] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:17:37 [I 2022-10-26 11:48:33] ...trap/engines/engine.py.229: process: 0.50698 | load: 0.00255 [I 2022-10-26 11:48:33] ...trap/engines/engine.py.230: loss: 0.28270 [I 2022-10-26 11:48:33] ...optimizers/trijoint.py.151: total_norm: 3.755350 [I 2022-10-26 11:48:38] ...trap/engines/engine.py.224: train: epoch 6 | batch 210/2304 [I 2022-10-26 11:48:38] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:38 [I 2022-10-26 11:48:38] ...trap/engines/engine.py.229: process: 0.50105 | load: 0.00244 [I 2022-10-26 11:48:38] ...trap/engines/engine.py.230: loss: 0.25521 [I 2022-10-26 11:48:38] ...optimizers/trijoint.py.151: total_norm: 3.753439 [I 2022-10-26 11:48:43] ...trap/engines/engine.py.224: train: epoch 6 | batch 220/2304 [I 2022-10-26 11:48:43] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:22 [I 2022-10-26 11:48:43] ...trap/engines/engine.py.229: process: 0.50142 | load: 0.00257 [I 2022-10-26 11:48:43] ...trap/engines/engine.py.230: loss: 0.33087 [I 2022-10-26 11:48:43] ...optimizers/trijoint.py.151: total_norm: 3.944459 [I 2022-10-26 11:48:48] ...trap/engines/engine.py.224: train: epoch 6 | batch 230/2304 [I 2022-10-26 11:48:48] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:20 [I 2022-10-26 11:48:48] ...trap/engines/engine.py.229: process: 0.49982 | load: 0.00248 [I 2022-10-26 11:48:48] ...trap/engines/engine.py.230: loss: 0.30009 [I 2022-10-26 11:48:48] ...optimizers/trijoint.py.151: total_norm: 3.725061 [I 2022-10-26 11:48:53] ...trap/engines/engine.py.224: train: epoch 6 | batch 240/2304 [I 2022-10-26 11:48:53] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:10 [I 2022-10-26 11:48:53] ...trap/engines/engine.py.229: process: 0.49500 | load: 0.00252 [I 2022-10-26 11:48:53] ...trap/engines/engine.py.230: loss: 0.25443 [I 2022-10-26 11:48:53] ...optimizers/trijoint.py.151: total_norm: 3.522079 [I 2022-10-26 11:48:58] ...trap/engines/engine.py.224: train: epoch 6 | batch 250/2304 [I 2022-10-26 11:48:58] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:05 [I 2022-10-26 11:48:58] ...trap/engines/engine.py.229: process: 0.50896 | load: 0.00249 [I 2022-10-26 11:48:58] ...trap/engines/engine.py.230: loss: 0.15979 [I 2022-10-26 11:48:58] ...optimizers/trijoint.py.151: total_norm: 2.612931 [I 2022-10-26 11:49:03] ...trap/engines/engine.py.224: train: epoch 6 | batch 260/2304 [I 2022-10-26 11:49:03] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:03 [I 2022-10-26 11:49:03] ...trap/engines/engine.py.229: process: 0.50382 | load: 0.00251 [I 2022-10-26 11:49:03] ...trap/engines/engine.py.230: loss: 0.15619 [I 2022-10-26 11:49:03] ...optimizers/trijoint.py.151: total_norm: 2.652839 [I 2022-10-26 11:49:08] ...trap/engines/engine.py.224: train: epoch 6 | batch 270/2304 [I 2022-10-26 11:49:08] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:17:05 [I 2022-10-26 11:49:08] ...trap/engines/engine.py.229: process: 0.50541 | load: 0.00270 [I 2022-10-26 11:49:08] ...trap/engines/engine.py.230: loss: 0.34059 [I 2022-10-26 11:49:08] ...optimizers/trijoint.py.151: total_norm: 3.747576 [I 2022-10-26 11:49:14] ...trap/engines/engine.py.224: train: epoch 6 | batch 280/2304 [I 2022-10-26 11:49:14] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:17:02 [I 2022-10-26 11:49:14] ...trap/engines/engine.py.229: process: 0.50149 | load: 0.00259 [I 2022-10-26 11:49:14] ...trap/engines/engine.py.230: loss: 0.23691 [I 2022-10-26 11:49:14] ...optimizers/trijoint.py.151: total_norm: 2.138391 [I 2022-10-26 11:49:19] ...trap/engines/engine.py.224: train: epoch 6 | batch 290/2304 [I 2022-10-26 11:49:19] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:16:46 [I 2022-10-26 11:49:19] ...trap/engines/engine.py.229: process: 0.49570 | load: 0.00320 [I 2022-10-26 11:49:19] ...trap/engines/engine.py.230: loss: 0.22672 [I 2022-10-26 11:49:19] ...optimizers/trijoint.py.151: total_norm: 3.426965 [I 2022-10-26 11:49:24] ...trap/engines/engine.py.224: train: epoch 6 | batch 300/2304 [I 2022-10-26 11:49:24] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:16:44 [I 2022-10-26 11:49:24] ...trap/engines/engine.py.229: process: 0.50020 | load: 0.00248 [I 2022-10-26 11:49:24] ...trap/engines/engine.py.230: loss: 0.21851 [I 2022-10-26 11:49:24] ...optimizers/trijoint.py.151: total_norm: 3.938632 [I 2022-10-26 11:49:29] ...trap/engines/engine.py.224: train: epoch 6 | batch 310/2304 [I 2022-10-26 11:49:29] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:16:43 [I 2022-10-26 11:49:29] ...trap/engines/engine.py.229: process: 0.51760 | load: 0.00265 [I 2022-10-26 11:49:29] ...trap/engines/engine.py.230: loss: 0.24025 [I 2022-10-26 11:49:29] ...optimizers/trijoint.py.151: total_norm: 4.119368 [I 2022-10-26 11:49:34] ...trap/engines/engine.py.224: train: epoch 6 | batch 320/2304 [I 2022-10-26 11:49:34] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:16:29 [I 2022-10-26 11:49:34] ...trap/engines/engine.py.229: process: 0.48775 | load: 0.00260 [I 2022-10-26 11:49:34] ...trap/engines/engine.py.230: loss: 0.23341 [I 2022-10-26 11:49:34] ...optimizers/trijoint.py.151: total_norm: 3.708037 [I 2022-10-26 11:49:39] ...trap/engines/engine.py.224: train: epoch 6 | batch 330/2304 [I 2022-10-26 11:49:39] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:16:35 [I 2022-10-26 11:49:39] ...trap/engines/engine.py.229: process: 0.51894 | load: 0.00292 [I 2022-10-26 11:49:39] ...trap/engines/engine.py.230: loss: 0.30961 [I 2022-10-26 11:49:39] ...optimizers/trijoint.py.151: total_norm: 4.140604 [I 2022-10-26 11:49:44] ...trap/engines/engine.py.224: train: epoch 6 | batch 340/2304 [I 2022-10-26 11:49:44] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:16:36 [I 2022-10-26 11:49:44] ...trap/engines/engine.py.229: process: 0.51670 | load: 0.00258 [I 2022-10-26 11:49:44] ...trap/engines/engine.py.230: loss: 0.24558 [I 2022-10-26 11:49:44] ...optimizers/trijoint.py.151: total_norm: 3.294353 [I 2022-10-26 11:49:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 350/2304 [I 2022-10-26 11:49:49] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:16:24 [I 2022-10-26 11:49:49] ...trap/engines/engine.py.229: process: 0.51146 | load: 0.00252 [I 2022-10-26 11:49:49] ...trap/engines/engine.py.230: loss: 0.25524 [I 2022-10-26 11:49:49] ...optimizers/trijoint.py.151: total_norm: 2.831619 [I 2022-10-26 11:49:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 360/2304 [I 2022-10-26 11:49:54] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:16:11 [I 2022-10-26 11:49:54] ...trap/engines/engine.py.229: process: 0.50600 | load: 0.00261 [I 2022-10-26 11:49:54] ...trap/engines/engine.py.230: loss: 0.31098 [I 2022-10-26 11:49:54] ...optimizers/trijoint.py.151: total_norm: 3.719751 [I 2022-10-26 11:49:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 370/2304 [I 2022-10-26 11:49:59] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:16:01 [I 2022-10-26 11:49:59] ...trap/engines/engine.py.229: process: 0.49158 | load: 0.00263 [I 2022-10-26 11:49:59] ...trap/engines/engine.py.230: loss: 0.19122 [I 2022-10-26 11:49:59] ...optimizers/trijoint.py.151: total_norm: 2.993364 [I 2022-10-26 11:50:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 380/2304 [I 2022-10-26 11:50:04] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:15:58 [I 2022-10-26 11:50:04] ...trap/engines/engine.py.229: process: 0.50174 | load: 0.00261 [I 2022-10-26 11:50:04] ...trap/engines/engine.py.230: loss: 0.21246 [I 2022-10-26 11:50:04] ...optimizers/trijoint.py.151: total_norm: 3.404096 [I 2022-10-26 11:50:09] ...trap/engines/engine.py.224: train: epoch 6 | batch 390/2304 [I 2022-10-26 11:50:09] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:15:56 [I 2022-10-26 11:50:09] ...trap/engines/engine.py.229: process: 0.51456 | load: 0.00284 [I 2022-10-26 11:50:09] ...trap/engines/engine.py.230: loss: 0.26182 [I 2022-10-26 11:50:09] ...optimizers/trijoint.py.151: total_norm: 3.356113 [I 2022-10-26 11:50:14] ...trap/engines/engine.py.224: train: epoch 6 | batch 400/2304 [I 2022-10-26 11:50:14] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:15:52 [I 2022-10-26 11:50:14] ...trap/engines/engine.py.229: process: 0.50268 | load: 0.00253 [I 2022-10-26 11:50:14] ...trap/engines/engine.py.230: loss: 0.27085 [I 2022-10-26 11:50:14] ...optimizers/trijoint.py.151: total_norm: 3.718076 [I 2022-10-26 11:50:19] ...trap/engines/engine.py.224: train: epoch 6 | batch 410/2304 [I 2022-10-26 11:50:19] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:15:52 [I 2022-10-26 11:50:19] ...trap/engines/engine.py.229: process: 0.49874 | load: 0.00257 [I 2022-10-26 11:50:19] ...trap/engines/engine.py.230: loss: 0.30892 [I 2022-10-26 11:50:19] ...optimizers/trijoint.py.151: total_norm: 3.554291 [I 2022-10-26 11:50:24] ...trap/engines/engine.py.224: train: epoch 6 | batch 420/2304 [I 2022-10-26 11:50:24] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:15:43 [I 2022-10-26 11:50:24] ...trap/engines/engine.py.229: process: 0.51040 | load: 0.00287 [I 2022-10-26 11:50:24] ...trap/engines/engine.py.230: loss: 0.27015 [I 2022-10-26 11:50:24] ...optimizers/trijoint.py.151: total_norm: 3.298760 [I 2022-10-26 11:50:29] ...trap/engines/engine.py.224: train: epoch 6 | batch 430/2304 [I 2022-10-26 11:50:29] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:15:43 [I 2022-10-26 11:50:29] ...trap/engines/engine.py.229: process: 0.50645 | load: 0.00254 [I 2022-10-26 11:50:29] ...trap/engines/engine.py.230: loss: 0.24768 [I 2022-10-26 11:50:29] ...optimizers/trijoint.py.151: total_norm: 3.386895 [I 2022-10-26 11:50:34] ...trap/engines/engine.py.224: train: epoch 6 | batch 440/2304 [I 2022-10-26 11:50:34] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:15:33 [I 2022-10-26 11:50:34] ...trap/engines/engine.py.229: process: 0.49397 | load: 0.00252 [I 2022-10-26 11:50:34] ...trap/engines/engine.py.230: loss: 0.23452 [I 2022-10-26 11:50:34] ...optimizers/trijoint.py.151: total_norm: 3.454288 [I 2022-10-26 11:50:39] ...trap/engines/engine.py.224: train: epoch 6 | batch 450/2304 [I 2022-10-26 11:50:39] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:15:27 [I 2022-10-26 11:50:39] ...trap/engines/engine.py.229: process: 0.50078 | load: 0.00256 [I 2022-10-26 11:50:39] ...trap/engines/engine.py.230: loss: 0.19990 [I 2022-10-26 11:50:39] ...optimizers/trijoint.py.151: total_norm: 3.523952 [I 2022-10-26 11:50:44] ...trap/engines/engine.py.224: train: epoch 6 | batch 460/2304 [I 2022-10-26 11:50:44] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:15:19 [I 2022-10-26 11:50:44] ...trap/engines/engine.py.229: process: 0.49390 | load: 0.00249 [I 2022-10-26 11:50:44] ...trap/engines/engine.py.230: loss: 0.34378 [I 2022-10-26 11:50:44] ...optimizers/trijoint.py.151: total_norm: 4.087602 [I 2022-10-26 11:50:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 470/2304 [I 2022-10-26 11:50:49] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:15:22 [I 2022-10-26 11:50:49] ...trap/engines/engine.py.229: process: 0.49385 | load: 0.00254 [I 2022-10-26 11:50:49] ...trap/engines/engine.py.230: loss: 0.23238 [I 2022-10-26 11:50:49] ...optimizers/trijoint.py.151: total_norm: 2.991907 [I 2022-10-26 11:50:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 480/2304 [I 2022-10-26 11:50:54] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:15:10 [I 2022-10-26 11:50:54] ...trap/engines/engine.py.229: process: 0.49899 | load: 0.00250 [I 2022-10-26 11:50:54] ...trap/engines/engine.py.230: loss: 0.28316 [I 2022-10-26 11:50:54] ...optimizers/trijoint.py.151: total_norm: 4.059545 [I 2022-10-26 11:50:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 490/2304 [I 2022-10-26 11:50:59] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:15:11 [I 2022-10-26 11:50:59] ...trap/engines/engine.py.229: process: 0.50211 | load: 0.00286 [I 2022-10-26 11:50:59] ...trap/engines/engine.py.230: loss: 0.21524 [I 2022-10-26 11:50:59] ...optimizers/trijoint.py.151: total_norm: 2.995945 [I 2022-10-26 11:51:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 500/2304 [I 2022-10-26 11:51:04] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:15:04 [I 2022-10-26 11:51:04] ...trap/engines/engine.py.229: process: 0.50050 | load: 0.00263 [I 2022-10-26 11:51:04] ...trap/engines/engine.py.230: loss: 0.29073 [I 2022-10-26 11:51:04] ...optimizers/trijoint.py.151: total_norm: 3.920671 [I 2022-10-26 11:51:09] ...trap/engines/engine.py.224: train: epoch 6 | batch 510/2304 [I 2022-10-26 11:51:09] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:15:04 [I 2022-10-26 11:51:09] ...trap/engines/engine.py.229: process: 0.49459 | load: 0.00257 [I 2022-10-26 11:51:09] ...trap/engines/engine.py.230: loss: 0.20603 [I 2022-10-26 11:51:09] ...optimizers/trijoint.py.151: total_norm: 3.456615 [I 2022-10-26 11:51:14] ...trap/engines/engine.py.224: train: epoch 6 | batch 520/2304 [I 2022-10-26 11:51:14] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:14:48 [I 2022-10-26 11:51:14] ...trap/engines/engine.py.229: process: 0.48687 | load: 0.00252 [I 2022-10-26 11:51:14] ...trap/engines/engine.py.230: loss: 0.32952 [I 2022-10-26 11:51:14] ...optimizers/trijoint.py.151: total_norm: 4.711071 [I 2022-10-26 11:51:19] ...trap/engines/engine.py.224: train: epoch 6 | batch 530/2304 [I 2022-10-26 11:51:19] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:14:53 [I 2022-10-26 11:51:19] ...trap/engines/engine.py.229: process: 0.51072 | load: 0.00247 [I 2022-10-26 11:51:19] ...trap/engines/engine.py.230: loss: 0.20745 [I 2022-10-26 11:51:19] ...optimizers/trijoint.py.151: total_norm: 3.464580 [I 2022-10-26 11:51:24] ...trap/engines/engine.py.224: train: epoch 6 | batch 540/2304 [I 2022-10-26 11:51:24] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:14:37 [I 2022-10-26 11:51:24] ...trap/engines/engine.py.229: process: 0.49820 | load: 0.00248 [I 2022-10-26 11:51:24] ...trap/engines/engine.py.230: loss: 0.25398 [I 2022-10-26 11:51:24] ...optimizers/trijoint.py.151: total_norm: 3.500772 [I 2022-10-26 11:51:29] ...trap/engines/engine.py.224: train: epoch 6 | batch 550/2304 [I 2022-10-26 11:51:29] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:14:36 [I 2022-10-26 11:51:29] ...trap/engines/engine.py.229: process: 0.49857 | load: 0.00253 [I 2022-10-26 11:51:29] ...trap/engines/engine.py.230: loss: 0.26199 [I 2022-10-26 11:51:29] ...optimizers/trijoint.py.151: total_norm: 3.344685 [I 2022-10-26 11:51:34] ...trap/engines/engine.py.224: train: epoch 6 | batch 560/2304 [I 2022-10-26 11:51:34] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:14:40 [I 2022-10-26 11:51:34] ...trap/engines/engine.py.229: process: 0.49917 | load: 0.00249 [I 2022-10-26 11:51:34] ...trap/engines/engine.py.230: loss: 0.30074 [I 2022-10-26 11:51:34] ...optimizers/trijoint.py.151: total_norm: 4.410790 [I 2022-10-26 11:51:39] ...trap/engines/engine.py.224: train: epoch 6 | batch 570/2304 [I 2022-10-26 11:51:39] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:14:26 [I 2022-10-26 11:51:39] ...trap/engines/engine.py.229: process: 0.50235 | load: 0.00268 [I 2022-10-26 11:51:39] ...trap/engines/engine.py.230: loss: 0.35912 [I 2022-10-26 11:51:39] ...optimizers/trijoint.py.151: total_norm: 3.375697 [I 2022-10-26 11:51:44] ...trap/engines/engine.py.224: train: epoch 6 | batch 580/2304 [I 2022-10-26 11:51:44] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:14:12 [I 2022-10-26 11:51:44] ...trap/engines/engine.py.229: process: 0.50228 | load: 0.00264 [I 2022-10-26 11:51:44] ...trap/engines/engine.py.230: loss: 0.28983 [I 2022-10-26 11:51:44] ...optimizers/trijoint.py.151: total_norm: 4.106347 [I 2022-10-26 11:51:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 590/2304 [I 2022-10-26 11:51:49] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:14:12 [I 2022-10-26 11:51:49] ...trap/engines/engine.py.229: process: 0.49255 | load: 0.00255 [I 2022-10-26 11:51:49] ...trap/engines/engine.py.230: loss: 0.33077 [I 2022-10-26 11:51:49] ...optimizers/trijoint.py.151: total_norm: 3.501884 [I 2022-10-26 11:51:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 600/2304 [I 2022-10-26 11:51:54] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:14:09 [I 2022-10-26 11:51:54] ...trap/engines/engine.py.229: process: 0.50050 | load: 0.00260 [I 2022-10-26 11:51:54] ...trap/engines/engine.py.230: loss: 0.28260 [I 2022-10-26 11:51:54] ...optimizers/trijoint.py.151: total_norm: 2.824991 [I 2022-10-26 11:51:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 610/2304 [I 2022-10-26 11:51:59] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:14:13 [I 2022-10-26 11:51:59] ...trap/engines/engine.py.229: process: 0.51660 | load: 0.00254 [I 2022-10-26 11:51:59] ...trap/engines/engine.py.230: loss: 0.35227 [I 2022-10-26 11:51:59] ...optimizers/trijoint.py.151: total_norm: 3.957617 [I 2022-10-26 11:52:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 620/2304 [I 2022-10-26 11:52:04] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:13:49 [I 2022-10-26 11:52:04] ...trap/engines/engine.py.229: process: 0.49074 | load: 0.00284 [I 2022-10-26 11:52:04] ...trap/engines/engine.py.230: loss: 0.30934 [I 2022-10-26 11:52:04] ...optimizers/trijoint.py.151: total_norm: 3.449102 [I 2022-10-26 11:52:09] ...trap/engines/engine.py.224: train: epoch 6 | batch 630/2304 [I 2022-10-26 11:52:09] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:14:03 [I 2022-10-26 11:52:09] ...trap/engines/engine.py.229: process: 0.51551 | load: 0.00264 [I 2022-10-26 11:52:09] ...trap/engines/engine.py.230: loss: 0.22326 [I 2022-10-26 11:52:09] ...optimizers/trijoint.py.151: total_norm: 3.427022 [I 2022-10-26 11:52:14] ...trap/engines/engine.py.224: train: epoch 6 | batch 640/2304 [I 2022-10-26 11:52:14] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:13:57 [I 2022-10-26 11:52:14] ...trap/engines/engine.py.229: process: 0.49799 | load: 0.00255 [I 2022-10-26 11:52:14] ...trap/engines/engine.py.230: loss: 0.26363 [I 2022-10-26 11:52:14] ...optimizers/trijoint.py.151: total_norm: 3.332261 [I 2022-10-26 11:52:19] ...trap/engines/engine.py.224: train: epoch 6 | batch 650/2304 [I 2022-10-26 11:52:19] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:13:53 [I 2022-10-26 11:52:19] ...trap/engines/engine.py.229: process: 0.51506 | load: 0.00279 [I 2022-10-26 11:52:19] ...trap/engines/engine.py.230: loss: 0.36144 [I 2022-10-26 11:52:19] ...optimizers/trijoint.py.151: total_norm: 3.923292 [I 2022-10-26 11:52:24] ...trap/engines/engine.py.224: train: epoch 6 | batch 660/2304 [I 2022-10-26 11:52:24] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:13:48 [I 2022-10-26 11:52:24] ...trap/engines/engine.py.229: process: 0.49895 | load: 0.00252 [I 2022-10-26 11:52:24] ...trap/engines/engine.py.230: loss: 0.31669 [I 2022-10-26 11:52:24] ...optimizers/trijoint.py.151: total_norm: 3.180237 [I 2022-10-26 11:52:29] ...trap/engines/engine.py.224: train: epoch 6 | batch 670/2304 [I 2022-10-26 11:52:29] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:13:42 [I 2022-10-26 11:52:29] ...trap/engines/engine.py.229: process: 0.49567 | load: 0.00263 [I 2022-10-26 11:52:29] ...trap/engines/engine.py.230: loss: 0.26078 [I 2022-10-26 11:52:29] ...optimizers/trijoint.py.151: total_norm: 4.220042 [I 2022-10-26 11:52:34] ...trap/engines/engine.py.224: train: epoch 6 | batch 680/2304 [I 2022-10-26 11:52:34] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:13:36 [I 2022-10-26 11:52:34] ...trap/engines/engine.py.229: process: 0.50356 | load: 0.00265 [I 2022-10-26 11:52:34] ...trap/engines/engine.py.230: loss: 0.29745 [I 2022-10-26 11:52:34] ...optimizers/trijoint.py.151: total_norm: 3.963385 [I 2022-10-26 11:52:39] ...trap/engines/engine.py.224: train: epoch 6 | batch 690/2304 [I 2022-10-26 11:52:39] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:13:26 [I 2022-10-26 11:52:39] ...trap/engines/engine.py.229: process: 0.49451 | load: 0.00262 [I 2022-10-26 11:52:39] ...trap/engines/engine.py.230: loss: 0.28265 [I 2022-10-26 11:52:39] ...optimizers/trijoint.py.151: total_norm: 4.149063 [I 2022-10-26 11:52:44] ...trap/engines/engine.py.224: train: epoch 6 | batch 700/2304 [I 2022-10-26 11:52:44] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:13:25 [I 2022-10-26 11:52:44] ...trap/engines/engine.py.229: process: 0.49634 | load: 0.00249 [I 2022-10-26 11:52:44] ...trap/engines/engine.py.230: loss: 0.18869 [I 2022-10-26 11:52:44] ...optimizers/trijoint.py.151: total_norm: 2.650471 [I 2022-10-26 11:52:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 710/2304 [I 2022-10-26 11:52:49] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:13:15 [I 2022-10-26 11:52:49] ...trap/engines/engine.py.229: process: 0.49818 | load: 0.00261 [I 2022-10-26 11:52:49] ...trap/engines/engine.py.230: loss: 0.24616 [I 2022-10-26 11:52:49] ...optimizers/trijoint.py.151: total_norm: 2.875318 [I 2022-10-26 11:52:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 720/2304 [I 2022-10-26 11:52:54] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:13:11 [I 2022-10-26 11:52:54] ...trap/engines/engine.py.229: process: 0.50035 | load: 0.00249 [I 2022-10-26 11:52:54] ...trap/engines/engine.py.230: loss: 0.28587 [I 2022-10-26 11:52:54] ...optimizers/trijoint.py.151: total_norm: 3.587174 [I 2022-10-26 11:53:00] ...trap/engines/engine.py.224: train: epoch 6 | batch 730/2304 [I 2022-10-26 11:53:00] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:13:15 [I 2022-10-26 11:53:00] ...trap/engines/engine.py.229: process: 0.50882 | load: 0.00282 [I 2022-10-26 11:53:00] ...trap/engines/engine.py.230: loss: 0.23140 [I 2022-10-26 11:53:00] ...optimizers/trijoint.py.151: total_norm: 3.982243 [I 2022-10-26 11:53:05] ...trap/engines/engine.py.224: train: epoch 6 | batch 740/2304 [I 2022-10-26 11:53:05] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:13:00 [I 2022-10-26 11:53:05] ...trap/engines/engine.py.229: process: 0.49327 | load: 0.00261 [I 2022-10-26 11:53:05] ...trap/engines/engine.py.230: loss: 0.27285 [I 2022-10-26 11:53:05] ...optimizers/trijoint.py.151: total_norm: 3.531972 [I 2022-10-26 11:53:10] ...trap/engines/engine.py.224: train: epoch 6 | batch 750/2304 [I 2022-10-26 11:53:10] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:12:58 [I 2022-10-26 11:53:10] ...trap/engines/engine.py.229: process: 0.49956 | load: 0.00253 [I 2022-10-26 11:53:10] ...trap/engines/engine.py.230: loss: 0.27223 [I 2022-10-26 11:53:10] ...optimizers/trijoint.py.151: total_norm: 3.981380 [I 2022-10-26 11:53:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 760/2304 [I 2022-10-26 11:53:15] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:13:03 [I 2022-10-26 11:53:15] ...trap/engines/engine.py.229: process: 0.51114 | load: 0.00257 [I 2022-10-26 11:53:15] ...trap/engines/engine.py.230: loss: 0.28589 [I 2022-10-26 11:53:15] ...optimizers/trijoint.py.151: total_norm: 3.633719 [I 2022-10-26 11:53:20] ...trap/engines/engine.py.224: train: epoch 6 | batch 770/2304 [I 2022-10-26 11:53:20] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:12:41 [I 2022-10-26 11:53:20] ...trap/engines/engine.py.229: process: 0.49390 | load: 0.00246 [I 2022-10-26 11:53:20] ...trap/engines/engine.py.230: loss: 0.29928 [I 2022-10-26 11:53:20] ...optimizers/trijoint.py.151: total_norm: 3.893520 [I 2022-10-26 11:53:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 780/2304 [I 2022-10-26 11:53:25] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:12:47 [I 2022-10-26 11:53:25] ...trap/engines/engine.py.229: process: 0.49361 | load: 0.00251 [I 2022-10-26 11:53:25] ...trap/engines/engine.py.230: loss: 0.28044 [I 2022-10-26 11:53:25] ...optimizers/trijoint.py.151: total_norm: 3.874915 [I 2022-10-26 11:53:30] ...trap/engines/engine.py.224: train: epoch 6 | batch 790/2304 [I 2022-10-26 11:53:30] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:12:41 [I 2022-10-26 11:53:30] ...trap/engines/engine.py.229: process: 0.50523 | load: 0.00256 [I 2022-10-26 11:53:30] ...trap/engines/engine.py.230: loss: 0.21033 [I 2022-10-26 11:53:30] ...optimizers/trijoint.py.151: total_norm: 3.607545 [I 2022-10-26 11:53:35] ...trap/engines/engine.py.224: train: epoch 6 | batch 800/2304 [I 2022-10-26 11:53:35] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:12:32 [I 2022-10-26 11:53:35] ...trap/engines/engine.py.229: process: 0.50044 | load: 0.00253 [I 2022-10-26 11:53:35] ...trap/engines/engine.py.230: loss: 0.23861 [I 2022-10-26 11:53:35] ...optimizers/trijoint.py.151: total_norm: 3.173575 [I 2022-10-26 11:53:40] ...trap/engines/engine.py.224: train: epoch 6 | batch 810/2304 [I 2022-10-26 11:53:40] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:12:31 [I 2022-10-26 11:53:40] ...trap/engines/engine.py.229: process: 0.51322 | load: 0.00258 [I 2022-10-26 11:53:40] ...trap/engines/engine.py.230: loss: 0.28336 [I 2022-10-26 11:53:40] ...optimizers/trijoint.py.151: total_norm: 4.020901 [I 2022-10-26 11:53:45] ...trap/engines/engine.py.224: train: epoch 6 | batch 820/2304 [I 2022-10-26 11:53:45] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:12:24 [I 2022-10-26 11:53:45] ...trap/engines/engine.py.229: process: 0.50663 | load: 0.00257 [I 2022-10-26 11:53:45] ...trap/engines/engine.py.230: loss: 0.26087 [I 2022-10-26 11:53:45] ...optimizers/trijoint.py.151: total_norm: 3.301664 [I 2022-10-26 11:53:50] ...trap/engines/engine.py.224: train: epoch 6 | batch 830/2304 [I 2022-10-26 11:53:50] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:12:15 [I 2022-10-26 11:53:50] ...trap/engines/engine.py.229: process: 0.48717 | load: 0.00261 [I 2022-10-26 11:53:50] ...trap/engines/engine.py.230: loss: 0.24812 [I 2022-10-26 11:53:50] ...optimizers/trijoint.py.151: total_norm: 3.691334 [I 2022-10-26 11:53:55] ...trap/engines/engine.py.224: train: epoch 6 | batch 840/2304 [I 2022-10-26 11:53:55] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:12:09 [I 2022-10-26 11:53:55] ...trap/engines/engine.py.229: process: 0.49032 | load: 0.00279 [I 2022-10-26 11:53:55] ...trap/engines/engine.py.230: loss: 0.27330 [I 2022-10-26 11:53:55] ...optimizers/trijoint.py.151: total_norm: 3.976381 [I 2022-10-26 11:54:00] ...trap/engines/engine.py.224: train: epoch 6 | batch 850/2304 [I 2022-10-26 11:54:00] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:12:10 [I 2022-10-26 11:54:00] ...trap/engines/engine.py.229: process: 0.50710 | load: 0.00255 [I 2022-10-26 11:54:00] ...trap/engines/engine.py.230: loss: 0.26104 [I 2022-10-26 11:54:00] ...optimizers/trijoint.py.151: total_norm: 3.327097 [I 2022-10-26 11:54:05] ...trap/engines/engine.py.224: train: epoch 6 | batch 860/2304 [I 2022-10-26 11:54:05] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:11:59 [I 2022-10-26 11:54:05] ...trap/engines/engine.py.229: process: 0.50732 | load: 0.00262 [I 2022-10-26 11:54:05] ...trap/engines/engine.py.230: loss: 0.33991 [I 2022-10-26 11:54:05] ...optimizers/trijoint.py.151: total_norm: 3.438541 [I 2022-10-26 11:54:10] ...trap/engines/engine.py.224: train: epoch 6 | batch 870/2304 [I 2022-10-26 11:54:10] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:12:04 [I 2022-10-26 11:54:10] ...trap/engines/engine.py.229: process: 0.49419 | load: 0.00254 [I 2022-10-26 11:54:10] ...trap/engines/engine.py.230: loss: 0.26486 [I 2022-10-26 11:54:10] ...optimizers/trijoint.py.151: total_norm: 4.113139 [I 2022-10-26 11:54:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 880/2304 [I 2022-10-26 11:54:15] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:12:02 [I 2022-10-26 11:54:15] ...trap/engines/engine.py.229: process: 0.51982 | load: 0.00292 [I 2022-10-26 11:54:15] ...trap/engines/engine.py.230: loss: 0.15892 [I 2022-10-26 11:54:15] ...optimizers/trijoint.py.151: total_norm: 2.581519 [I 2022-10-26 11:54:20] ...trap/engines/engine.py.224: train: epoch 6 | batch 890/2304 [I 2022-10-26 11:54:20] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:11:44 [I 2022-10-26 11:54:20] ...trap/engines/engine.py.229: process: 0.49261 | load: 0.00293 [I 2022-10-26 11:54:20] ...trap/engines/engine.py.230: loss: 0.30199 [I 2022-10-26 11:54:20] ...optimizers/trijoint.py.151: total_norm: 3.850068 [I 2022-10-26 11:54:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 900/2304 [I 2022-10-26 11:54:25] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:11:38 [I 2022-10-26 11:54:25] ...trap/engines/engine.py.229: process: 0.49969 | load: 0.00259 [I 2022-10-26 11:54:25] ...trap/engines/engine.py.230: loss: 0.34545 [I 2022-10-26 11:54:25] ...optimizers/trijoint.py.151: total_norm: 4.765612 [I 2022-10-26 11:54:30] ...trap/engines/engine.py.224: train: epoch 6 | batch 910/2304 [I 2022-10-26 11:54:30] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:11:34 [I 2022-10-26 11:54:30] ...trap/engines/engine.py.229: process: 0.49768 | load: 0.00253 [I 2022-10-26 11:54:30] ...trap/engines/engine.py.230: loss: 0.25486 [I 2022-10-26 11:54:30] ...optimizers/trijoint.py.151: total_norm: 3.347682 [I 2022-10-26 11:54:35] ...trap/engines/engine.py.224: train: epoch 6 | batch 920/2304 [I 2022-10-26 11:54:35] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:11:34 [I 2022-10-26 11:54:35] ...trap/engines/engine.py.229: process: 0.50673 | load: 0.00252 [I 2022-10-26 11:54:35] ...trap/engines/engine.py.230: loss: 0.23905 [I 2022-10-26 11:54:35] ...optimizers/trijoint.py.151: total_norm: 3.006259 [I 2022-10-26 11:54:40] ...trap/engines/engine.py.224: train: epoch 6 | batch 930/2304 [I 2022-10-26 11:54:40] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:11:33 [I 2022-10-26 11:54:40] ...trap/engines/engine.py.229: process: 0.51037 | load: 0.00265 [I 2022-10-26 11:54:40] ...trap/engines/engine.py.230: loss: 0.29626 [I 2022-10-26 11:54:40] ...optimizers/trijoint.py.151: total_norm: 3.920227 [I 2022-10-26 11:54:45] ...trap/engines/engine.py.224: train: epoch 6 | batch 940/2304 [I 2022-10-26 11:54:45] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:11:26 [I 2022-10-26 11:54:45] ...trap/engines/engine.py.229: process: 0.49893 | load: 0.00256 [I 2022-10-26 11:54:45] ...trap/engines/engine.py.230: loss: 0.31044 [I 2022-10-26 11:54:45] ...optimizers/trijoint.py.151: total_norm: 3.141927 [I 2022-10-26 11:54:50] ...trap/engines/engine.py.224: train: epoch 6 | batch 950/2304 [I 2022-10-26 11:54:50] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:11:22 [I 2022-10-26 11:54:50] ...trap/engines/engine.py.229: process: 0.50660 | load: 0.00245 [I 2022-10-26 11:54:50] ...trap/engines/engine.py.230: loss: 0.29955 [I 2022-10-26 11:54:50] ...optimizers/trijoint.py.151: total_norm: 4.559610 [I 2022-10-26 11:54:55] ...trap/engines/engine.py.224: train: epoch 6 | batch 960/2304 [I 2022-10-26 11:54:55] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:11:13 [I 2022-10-26 11:54:55] ...trap/engines/engine.py.229: process: 0.49515 | load: 0.00243 [I 2022-10-26 11:54:55] ...trap/engines/engine.py.230: loss: 0.18880 [I 2022-10-26 11:54:55] ...optimizers/trijoint.py.151: total_norm: 3.010818 [I 2022-10-26 11:55:00] ...trap/engines/engine.py.224: train: epoch 6 | batch 970/2304 [I 2022-10-26 11:55:00] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:11:04 [I 2022-10-26 11:55:00] ...trap/engines/engine.py.229: process: 0.49948 | load: 0.00266 [I 2022-10-26 11:55:00] ...trap/engines/engine.py.230: loss: 0.26426 [I 2022-10-26 11:55:00] ...optimizers/trijoint.py.151: total_norm: 3.068857 [I 2022-10-26 11:55:05] ...trap/engines/engine.py.224: train: epoch 6 | batch 980/2304 [I 2022-10-26 11:55:05] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:11:16 [I 2022-10-26 11:55:05] ...trap/engines/engine.py.229: process: 0.51249 | load: 0.00254 [I 2022-10-26 11:55:05] ...trap/engines/engine.py.230: loss: 0.25926 [I 2022-10-26 11:55:05] ...optimizers/trijoint.py.151: total_norm: 4.711819 [I 2022-10-26 11:55:10] ...trap/engines/engine.py.224: train: epoch 6 | batch 990/2304 [I 2022-10-26 11:55:10] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:10:56 [I 2022-10-26 11:55:10] ...trap/engines/engine.py.229: process: 0.49273 | load: 0.00250 [I 2022-10-26 11:55:10] ...trap/engines/engine.py.230: loss: 0.34897 [I 2022-10-26 11:55:10] ...optimizers/trijoint.py.151: total_norm: 3.829904 [I 2022-10-26 11:55:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 1000/2304 [I 2022-10-26 11:55:15] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:11:22 [I 2022-10-26 11:55:15] ...trap/engines/engine.py.229: process: 0.57641 | load: 0.00273 [I 2022-10-26 11:55:15] ...trap/engines/engine.py.230: loss: 0.24276 [I 2022-10-26 11:55:15] ...optimizers/trijoint.py.151: total_norm: 3.605217 [I 2022-10-26 11:55:20] ...trap/engines/engine.py.224: train: epoch 6 | batch 1010/2304 [I 2022-10-26 11:55:20] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:10:46 [I 2022-10-26 11:55:20] ...trap/engines/engine.py.229: process: 0.49110 | load: 0.00256 [I 2022-10-26 11:55:20] ...trap/engines/engine.py.230: loss: 0.27560 [I 2022-10-26 11:55:20] ...optimizers/trijoint.py.151: total_norm: 3.364440 [I 2022-10-26 11:55:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 1020/2304 [I 2022-10-26 11:55:25] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:10:39 [I 2022-10-26 11:55:25] ...trap/engines/engine.py.229: process: 0.49724 | load: 0.00268 [I 2022-10-26 11:55:25] ...trap/engines/engine.py.230: loss: 0.23791 [I 2022-10-26 11:55:25] ...optimizers/trijoint.py.151: total_norm: 2.962295 [I 2022-10-26 11:55:30] ...trap/engines/engine.py.224: train: epoch 6 | batch 1030/2304 [I 2022-10-26 11:55:30] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:10:36 [I 2022-10-26 11:55:30] ...trap/engines/engine.py.229: process: 0.49678 | load: 0.00283 [I 2022-10-26 11:55:30] ...trap/engines/engine.py.230: loss: 0.35281 [I 2022-10-26 11:55:30] ...optimizers/trijoint.py.151: total_norm: 3.713677 [I 2022-10-26 11:55:35] ...trap/engines/engine.py.224: train: epoch 6 | batch 1040/2304 [I 2022-10-26 11:55:35] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:10:35 [I 2022-10-26 11:55:35] ...trap/engines/engine.py.229: process: 0.50924 | load: 0.00264 [I 2022-10-26 11:55:35] ...trap/engines/engine.py.230: loss: 0.23224 [I 2022-10-26 11:55:35] ...optimizers/trijoint.py.151: total_norm: 3.237841 [I 2022-10-26 11:55:40] ...trap/engines/engine.py.224: train: epoch 6 | batch 1050/2304 [I 2022-10-26 11:55:40] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:10:27 [I 2022-10-26 11:55:40] ...trap/engines/engine.py.229: process: 0.49899 | load: 0.00262 [I 2022-10-26 11:55:40] ...trap/engines/engine.py.230: loss: 0.29319 [I 2022-10-26 11:55:40] ...optimizers/trijoint.py.151: total_norm: 3.781688 [I 2022-10-26 11:55:45] ...trap/engines/engine.py.224: train: epoch 6 | batch 1060/2304 [I 2022-10-26 11:55:45] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:10:23 [I 2022-10-26 11:55:45] ...trap/engines/engine.py.229: process: 0.50074 | load: 0.00250 [I 2022-10-26 11:55:45] ...trap/engines/engine.py.230: loss: 0.21552 [I 2022-10-26 11:55:45] ...optimizers/trijoint.py.151: total_norm: 3.202997 [I 2022-10-26 11:55:50] ...trap/engines/engine.py.224: train: epoch 6 | batch 1070/2304 [I 2022-10-26 11:55:50] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:10:17 [I 2022-10-26 11:55:50] ...trap/engines/engine.py.229: process: 0.51219 | load: 0.00248 [I 2022-10-26 11:55:50] ...trap/engines/engine.py.230: loss: 0.19484 [I 2022-10-26 11:55:50] ...optimizers/trijoint.py.151: total_norm: 2.248682 [I 2022-10-26 11:55:56] ...trap/engines/engine.py.224: train: epoch 6 | batch 1080/2304 [I 2022-10-26 11:55:56] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:10:12 [I 2022-10-26 11:55:56] ...trap/engines/engine.py.229: process: 0.48379 | load: 0.00245 [I 2022-10-26 11:55:56] ...trap/engines/engine.py.230: loss: 0.26161 [I 2022-10-26 11:55:56] ...optimizers/trijoint.py.151: total_norm: 3.543328 [I 2022-10-26 11:56:23] ...trap/engines/engine.py.224: train: epoch 6 | batch 1090/2304 [I 2022-10-26 11:56:23] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 1:39:44 [I 2022-10-26 11:56:23] ...trap/engines/engine.py.229: process: 22.64482 | load: 0.00290 [I 2022-10-26 11:56:23] ...trap/engines/engine.py.230: loss: 0.36656 [I 2022-10-26 11:56:23] ...optimizers/trijoint.py.151: total_norm: 4.986342 [I 2022-10-26 11:56:28] ...trap/engines/engine.py.224: train: epoch 6 | batch 1100/2304 [I 2022-10-26 11:56:28] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:19:34 [I 2022-10-26 11:56:28] ...trap/engines/engine.py.229: process: 0.50297 | load: 0.00238 [I 2022-10-26 11:56:28] ...trap/engines/engine.py.230: loss: 0.23827 [I 2022-10-26 11:56:28] ...optimizers/trijoint.py.151: total_norm: 3.072451 [I 2022-10-26 11:56:33] ...trap/engines/engine.py.224: train: epoch 6 | batch 1110/2304 [I 2022-10-26 11:56:33] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:11:01 [I 2022-10-26 11:56:33] ...trap/engines/engine.py.229: process: 0.49118 | load: 0.00248 [I 2022-10-26 11:56:33] ...trap/engines/engine.py.230: loss: 0.29920 [I 2022-10-26 11:56:33] ...optimizers/trijoint.py.151: total_norm: 4.131122 [I 2022-10-26 11:56:38] ...trap/engines/engine.py.224: train: epoch 6 | batch 1120/2304 [I 2022-10-26 11:56:38] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:09:59 [I 2022-10-26 11:56:38] ...trap/engines/engine.py.229: process: 0.50345 | load: 0.00243 [I 2022-10-26 11:56:38] ...trap/engines/engine.py.230: loss: 0.25312 [I 2022-10-26 11:56:38] ...optimizers/trijoint.py.151: total_norm: 2.828771 [I 2022-10-26 11:56:43] ...trap/engines/engine.py.224: train: epoch 6 | batch 1130/2304 [I 2022-10-26 11:56:43] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:09:46 [I 2022-10-26 11:56:43] ...trap/engines/engine.py.229: process: 0.49720 | load: 0.00244 [I 2022-10-26 11:56:43] ...trap/engines/engine.py.230: loss: 0.32104 [I 2022-10-26 11:56:43] ...optimizers/trijoint.py.151: total_norm: 3.611644 [I 2022-10-26 11:56:48] ...trap/engines/engine.py.224: train: epoch 6 | batch 1140/2304 [I 2022-10-26 11:56:48] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:09:41 [I 2022-10-26 11:56:48] ...trap/engines/engine.py.229: process: 0.50227 | load: 0.00243 [I 2022-10-26 11:56:48] ...trap/engines/engine.py.230: loss: 0.27473 [I 2022-10-26 11:56:48] ...optimizers/trijoint.py.151: total_norm: 4.019435 [I 2022-10-26 11:56:53] ...trap/engines/engine.py.224: train: epoch 6 | batch 1150/2304 [I 2022-10-26 11:56:53] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:09:39 [I 2022-10-26 11:56:53] ...trap/engines/engine.py.229: process: 0.49102 | load: 0.00241 [I 2022-10-26 11:56:53] ...trap/engines/engine.py.230: loss: 0.14561 [I 2022-10-26 11:56:53] ...optimizers/trijoint.py.151: total_norm: 2.638108 [I 2022-10-26 11:56:58] ...trap/engines/engine.py.224: train: epoch 6 | batch 1160/2304 [I 2022-10-26 11:56:58] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:09:34 [I 2022-10-26 11:56:58] ...trap/engines/engine.py.229: process: 0.50806 | load: 0.00239 [I 2022-10-26 11:56:58] ...trap/engines/engine.py.230: loss: 0.21101 [I 2022-10-26 11:56:58] ...optimizers/trijoint.py.151: total_norm: 3.389114 [I 2022-10-26 11:57:03] ...trap/engines/engine.py.224: train: epoch 6 | batch 1170/2304 [I 2022-10-26 11:57:03] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:09:27 [I 2022-10-26 11:57:03] ...trap/engines/engine.py.229: process: 0.49797 | load: 0.00243 [I 2022-10-26 11:57:03] ...trap/engines/engine.py.230: loss: 0.27649 [I 2022-10-26 11:57:03] ...optimizers/trijoint.py.151: total_norm: 4.008129 [I 2022-10-26 11:57:08] ...trap/engines/engine.py.224: train: epoch 6 | batch 1180/2304 [I 2022-10-26 11:57:08] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:09:20 [I 2022-10-26 11:57:08] ...trap/engines/engine.py.229: process: 0.50268 | load: 0.00239 [I 2022-10-26 11:57:08] ...trap/engines/engine.py.230: loss: 0.27961 [I 2022-10-26 11:57:08] ...optimizers/trijoint.py.151: total_norm: 4.072676 [I 2022-10-26 11:57:13] ...trap/engines/engine.py.224: train: epoch 6 | batch 1190/2304 [I 2022-10-26 11:57:13] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:09:14 [I 2022-10-26 11:57:13] ...trap/engines/engine.py.229: process: 0.50012 | load: 0.00249 [I 2022-10-26 11:57:13] ...trap/engines/engine.py.230: loss: 0.26699 [I 2022-10-26 11:57:13] ...optimizers/trijoint.py.151: total_norm: 3.414892 [I 2022-10-26 11:57:18] ...trap/engines/engine.py.224: train: epoch 6 | batch 1200/2304 [I 2022-10-26 11:57:18] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:09:08 [I 2022-10-26 11:57:18] ...trap/engines/engine.py.229: process: 0.49846 | load: 0.00246 [I 2022-10-26 11:57:18] ...trap/engines/engine.py.230: loss: 0.21213 [I 2022-10-26 11:57:18] ...optimizers/trijoint.py.151: total_norm: 2.826430 [I 2022-10-26 11:57:23] ...trap/engines/engine.py.224: train: epoch 6 | batch 1210/2304 [I 2022-10-26 11:57:23] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:09:17 [I 2022-10-26 11:57:23] ...trap/engines/engine.py.229: process: 0.52637 | load: 0.00250 [I 2022-10-26 11:57:23] ...trap/engines/engine.py.230: loss: 0.25107 [I 2022-10-26 11:57:23] ...optimizers/trijoint.py.151: total_norm: 3.938875 [I 2022-10-26 11:57:28] ...trap/engines/engine.py.224: train: epoch 6 | batch 1220/2304 [I 2022-10-26 11:57:28] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:09:13 [I 2022-10-26 11:57:28] ...trap/engines/engine.py.229: process: 0.51095 | load: 0.00248 [I 2022-10-26 11:57:28] ...trap/engines/engine.py.230: loss: 0.28274 [I 2022-10-26 11:57:28] ...optimizers/trijoint.py.151: total_norm: 4.103403 [I 2022-10-26 11:57:33] ...trap/engines/engine.py.224: train: epoch 6 | batch 1230/2304 [I 2022-10-26 11:57:33] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:09:13 [I 2022-10-26 11:57:33] ...trap/engines/engine.py.229: process: 0.51203 | load: 0.00253 [I 2022-10-26 11:57:33] ...trap/engines/engine.py.230: loss: 0.33283 [I 2022-10-26 11:57:33] ...optimizers/trijoint.py.151: total_norm: 3.460064 [I 2022-10-26 11:57:38] ...trap/engines/engine.py.224: train: epoch 6 | batch 1240/2304 [I 2022-10-26 11:57:38] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:09:04 [I 2022-10-26 11:57:38] ...trap/engines/engine.py.229: process: 0.50475 | load: 0.00246 [I 2022-10-26 11:57:38] ...trap/engines/engine.py.230: loss: 0.22492 [I 2022-10-26 11:57:38] ...optimizers/trijoint.py.151: total_norm: 3.660286 [I 2022-10-26 11:57:44] ...trap/engines/engine.py.224: train: epoch 6 | batch 1250/2304 [I 2022-10-26 11:57:44] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:09:13 [I 2022-10-26 11:57:44] ...trap/engines/engine.py.229: process: 0.52080 | load: 0.00247 [I 2022-10-26 11:57:44] ...trap/engines/engine.py.230: loss: 0.31225 [I 2022-10-26 11:57:44] ...optimizers/trijoint.py.151: total_norm: 3.598361 [I 2022-10-26 11:57:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 1260/2304 [I 2022-10-26 11:57:49] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:08:52 [I 2022-10-26 11:57:49] ...trap/engines/engine.py.229: process: 0.50081 | load: 0.00266 [I 2022-10-26 11:57:49] ...trap/engines/engine.py.230: loss: 0.27815 [I 2022-10-26 11:57:49] ...optimizers/trijoint.py.151: total_norm: 3.696326 [I 2022-10-26 11:57:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 1270/2304 [I 2022-10-26 11:57:54] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:08:39 [I 2022-10-26 11:57:54] ...trap/engines/engine.py.229: process: 0.50336 | load: 0.00247 [I 2022-10-26 11:57:54] ...trap/engines/engine.py.230: loss: 0.20093 [I 2022-10-26 11:57:54] ...optimizers/trijoint.py.151: total_norm: 3.087947 [I 2022-10-26 11:57:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 1280/2304 [I 2022-10-26 11:57:59] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:08:35 [I 2022-10-26 11:57:59] ...trap/engines/engine.py.229: process: 0.50673 | load: 0.00248 [I 2022-10-26 11:57:59] ...trap/engines/engine.py.230: loss: 0.34435 [I 2022-10-26 11:57:59] ...optimizers/trijoint.py.151: total_norm: 3.769508 [I 2022-10-26 11:58:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 1290/2304 [I 2022-10-26 11:58:04] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:08:26 [I 2022-10-26 11:58:04] ...trap/engines/engine.py.229: process: 0.50560 | load: 0.00250 [I 2022-10-26 11:58:04] ...trap/engines/engine.py.230: loss: 0.30200 [I 2022-10-26 11:58:04] ...optimizers/trijoint.py.151: total_norm: 3.557555 [I 2022-10-26 11:58:09] ...trap/engines/engine.py.224: train: epoch 6 | batch 1300/2304 [I 2022-10-26 11:58:09] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:08:22 [I 2022-10-26 11:58:09] ...trap/engines/engine.py.229: process: 0.49821 | load: 0.00249 [I 2022-10-26 11:58:09] ...trap/engines/engine.py.230: loss: 0.31133 [I 2022-10-26 11:58:09] ...optimizers/trijoint.py.151: total_norm: 3.438406 [I 2022-10-26 11:58:14] ...trap/engines/engine.py.224: train: epoch 6 | batch 1310/2304 [I 2022-10-26 11:58:14] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:08:17 [I 2022-10-26 11:58:14] ...trap/engines/engine.py.229: process: 0.50120 | load: 0.00267 [I 2022-10-26 11:58:14] ...trap/engines/engine.py.230: loss: 0.22033 [I 2022-10-26 11:58:14] ...optimizers/trijoint.py.151: total_norm: 4.134588 [I 2022-10-26 11:58:19] ...trap/engines/engine.py.224: train: epoch 6 | batch 1320/2304 [I 2022-10-26 11:58:19] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:08:10 [I 2022-10-26 11:58:19] ...trap/engines/engine.py.229: process: 0.50415 | load: 0.00248 [I 2022-10-26 11:58:19] ...trap/engines/engine.py.230: loss: 0.30147 [I 2022-10-26 11:58:19] ...optimizers/trijoint.py.151: total_norm: 3.031886 [I 2022-10-26 11:58:24] ...trap/engines/engine.py.224: train: epoch 6 | batch 1330/2304 [I 2022-10-26 11:58:24] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:08:03 [I 2022-10-26 11:58:24] ...trap/engines/engine.py.229: process: 0.48747 | load: 0.00246 [I 2022-10-26 11:58:24] ...trap/engines/engine.py.230: loss: 0.20904 [I 2022-10-26 11:58:24] ...optimizers/trijoint.py.151: total_norm: 2.503435 [I 2022-10-26 11:58:29] ...trap/engines/engine.py.224: train: epoch 6 | batch 1340/2304 [I 2022-10-26 11:58:29] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:08:04 [I 2022-10-26 11:58:29] ...trap/engines/engine.py.229: process: 0.52141 | load: 0.00240 [I 2022-10-26 11:58:29] ...trap/engines/engine.py.230: loss: 0.26258 [I 2022-10-26 11:58:29] ...optimizers/trijoint.py.151: total_norm: 4.599281 [I 2022-10-26 11:58:34] ...trap/engines/engine.py.224: train: epoch 6 | batch 1350/2304 [I 2022-10-26 11:58:34] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:08:00 [I 2022-10-26 11:58:34] ...trap/engines/engine.py.229: process: 0.50054 | load: 0.00258 [I 2022-10-26 11:58:34] ...trap/engines/engine.py.230: loss: 0.26005 [I 2022-10-26 11:58:34] ...optimizers/trijoint.py.151: total_norm: 2.989114 [I 2022-10-26 11:58:39] ...trap/engines/engine.py.224: train: epoch 6 | batch 1360/2304 [I 2022-10-26 11:58:39] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:08:12 [I 2022-10-26 11:58:39] ...trap/engines/engine.py.229: process: 0.56548 | load: 0.00266 [I 2022-10-26 11:58:39] ...trap/engines/engine.py.230: loss: 0.24957 [I 2022-10-26 11:58:39] ...optimizers/trijoint.py.151: total_norm: 3.901156 [I 2022-10-26 11:58:44] ...trap/engines/engine.py.224: train: epoch 6 | batch 1370/2304 [I 2022-10-26 11:58:44] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:07:51 [I 2022-10-26 11:58:44] ...trap/engines/engine.py.229: process: 0.50194 | load: 0.00254 [I 2022-10-26 11:58:44] ...trap/engines/engine.py.230: loss: 0.29686 [I 2022-10-26 11:58:44] ...optimizers/trijoint.py.151: total_norm: 3.912042 [I 2022-10-26 11:58:49] ...trap/engines/engine.py.224: train: epoch 6 | batch 1380/2304 [I 2022-10-26 11:58:49] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:07:43 [I 2022-10-26 11:58:49] ...trap/engines/engine.py.229: process: 0.50361 | load: 0.00247 [I 2022-10-26 11:58:49] ...trap/engines/engine.py.230: loss: 0.39526 [I 2022-10-26 11:58:49] ...optimizers/trijoint.py.151: total_norm: 5.007086 [I 2022-10-26 11:58:54] ...trap/engines/engine.py.224: train: epoch 6 | batch 1390/2304 [I 2022-10-26 11:58:54] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:07:36 [I 2022-10-26 11:58:54] ...trap/engines/engine.py.229: process: 0.49978 | load: 0.00247 [I 2022-10-26 11:58:54] ...trap/engines/engine.py.230: loss: 0.21444 [I 2022-10-26 11:58:54] ...optimizers/trijoint.py.151: total_norm: 3.365426 [I 2022-10-26 11:58:59] ...trap/engines/engine.py.224: train: epoch 6 | batch 1400/2304 [I 2022-10-26 11:58:59] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:07:36 [I 2022-10-26 11:58:59] ...trap/engines/engine.py.229: process: 0.49761 | load: 0.00262 [I 2022-10-26 11:58:59] ...trap/engines/engine.py.230: loss: 0.25433 [I 2022-10-26 11:58:59] ...optimizers/trijoint.py.151: total_norm: 2.312571 [I 2022-10-26 11:59:04] ...trap/engines/engine.py.224: train: epoch 6 | batch 1410/2304 [I 2022-10-26 11:59:04] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:07:28 [I 2022-10-26 11:59:04] ...trap/engines/engine.py.229: process: 0.50601 | load: 0.00251 [I 2022-10-26 11:59:04] ...trap/engines/engine.py.230: loss: 0.21438 [I 2022-10-26 11:59:04] ...optimizers/trijoint.py.151: total_norm: 4.264822 [I 2022-10-26 11:59:09] ...trap/engines/engine.py.224: train: epoch 6 | batch 1420/2304 [I 2022-10-26 11:59:09] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:07:21 [I 2022-10-26 11:59:09] ...trap/engines/engine.py.229: process: 0.48895 | load: 0.00245 [I 2022-10-26 11:59:09] ...trap/engines/engine.py.230: loss: 0.25893 [I 2022-10-26 11:59:09] ...optimizers/trijoint.py.151: total_norm: 3.522391 [I 2022-10-26 11:59:14] ...trap/engines/engine.py.224: train: epoch 6 | batch 1430/2304 [I 2022-10-26 11:59:14] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:07:18 [I 2022-10-26 11:59:14] ...trap/engines/engine.py.229: process: 0.50092 | load: 0.00252 [I 2022-10-26 11:59:14] ...trap/engines/engine.py.230: loss: 0.25502 [I 2022-10-26 11:59:14] ...optimizers/trijoint.py.151: total_norm: 3.894094 [I 2022-10-26 11:59:19] ...trap/engines/engine.py.224: train: epoch 6 | batch 1440/2304 [I 2022-10-26 11:59:19] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:07:16 [I 2022-10-26 11:59:19] ...trap/engines/engine.py.229: process: 0.50584 | load: 0.00248 [I 2022-10-26 11:59:19] ...trap/engines/engine.py.230: loss: 0.26974 [I 2022-10-26 11:59:19] ...optimizers/trijoint.py.151: total_norm: 4.487853 [I 2022-10-26 11:59:24] ...trap/engines/engine.py.224: train: epoch 6 | batch 1450/2304 [I 2022-10-26 11:59:24] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:07:08 [I 2022-10-26 11:59:24] ...trap/engines/engine.py.229: process: 0.49788 | load: 0.00265 [I 2022-10-26 11:59:24] ...trap/engines/engine.py.230: loss: 0.27723 [I 2022-10-26 11:59:24] ...optimizers/trijoint.py.151: total_norm: 2.777473 [I 2022-10-26 11:59:29] ...trap/engines/engine.py.224: train: epoch 6 | batch 1460/2304 [I 2022-10-26 11:59:29] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:07:01 [I 2022-10-26 11:59:29] ...trap/engines/engine.py.229: process: 0.49914 | load: 0.00249 [I 2022-10-26 11:59:29] ...trap/engines/engine.py.230: loss: 0.23600 [I 2022-10-26 11:59:29] ...optimizers/trijoint.py.151: total_norm: 3.647929 [I 2022-10-26 11:59:34] ...trap/engines/engine.py.224: train: epoch 6 | batch 1470/2304 [I 2022-10-26 11:59:34] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:06:59 [I 2022-10-26 11:59:34] ...trap/engines/engine.py.229: process: 0.50229 | load: 0.00248 [I 2022-10-26 11:59:34] ...trap/engines/engine.py.230: loss: 0.27091 [I 2022-10-26 11:59:34] ...optimizers/trijoint.py.151: total_norm: 3.561135 [I 2022-10-26 11:59:39] ...trap/engines/engine.py.224: train: epoch 6 | batch 1480/2304 [I 2022-10-26 11:59:39] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:06:55 [I 2022-10-26 11:59:39] ...trap/engines/engine.py.229: process: 0.50337 | load: 0.00255 [I 2022-10-26 11:59:39] ...trap/engines/engine.py.230: loss: 0.22284 [I 2022-10-26 11:59:39] ...optimizers/trijoint.py.151: total_norm: 3.644784 [I 2022-10-26 11:59:45] ...trap/engines/engine.py.224: train: epoch 6 | batch 1490/2304 [I 2022-10-26 11:59:45] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:07:00 [I 2022-10-26 11:59:45] ...trap/engines/engine.py.229: process: 0.52434 | load: 0.00252 [I 2022-10-26 11:59:45] ...trap/engines/engine.py.230: loss: 0.23315 [I 2022-10-26 11:59:45] ...optimizers/trijoint.py.151: total_norm: 3.155434 [I 2022-10-26 11:59:50] ...trap/engines/engine.py.224: train: epoch 6 | batch 1500/2304 [I 2022-10-26 11:59:50] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:06:46 [I 2022-10-26 11:59:50] ...trap/engines/engine.py.229: process: 0.50164 | load: 0.00258 [I 2022-10-26 11:59:50] ...trap/engines/engine.py.230: loss: 0.23266 [I 2022-10-26 11:59:50] ...optimizers/trijoint.py.151: total_norm: 3.524416 [I 2022-10-26 11:59:55] ...trap/engines/engine.py.224: train: epoch 6 | batch 1510/2304 [I 2022-10-26 11:59:55] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:06:34 [I 2022-10-26 11:59:55] ...trap/engines/engine.py.229: process: 0.48047 | load: 0.00251 [I 2022-10-26 11:59:55] ...trap/engines/engine.py.230: loss: 0.27107 [I 2022-10-26 11:59:55] ...optimizers/trijoint.py.151: total_norm: 2.954680 [I 2022-10-26 12:00:00] ...trap/engines/engine.py.224: train: epoch 6 | batch 1520/2304 [I 2022-10-26 12:00:00] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:06:33 [I 2022-10-26 12:00:00] ...trap/engines/engine.py.229: process: 0.50592 | load: 0.00255 [I 2022-10-26 12:00:00] ...trap/engines/engine.py.230: loss: 0.28558 [I 2022-10-26 12:00:00] ...optimizers/trijoint.py.151: total_norm: 4.936705 [I 2022-10-26 12:00:05] ...trap/engines/engine.py.224: train: epoch 6 | batch 1530/2304 [I 2022-10-26 12:00:05] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:06:24 [I 2022-10-26 12:00:05] ...trap/engines/engine.py.229: process: 0.49357 | load: 0.00256 [I 2022-10-26 12:00:05] ...trap/engines/engine.py.230: loss: 0.36128 [I 2022-10-26 12:00:05] ...optimizers/trijoint.py.151: total_norm: 4.000785 [I 2022-10-26 12:00:10] ...trap/engines/engine.py.224: train: epoch 6 | batch 1540/2304 [I 2022-10-26 12:00:10] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:06:24 [I 2022-10-26 12:00:10] ...trap/engines/engine.py.229: process: 0.50737 | load: 0.00246 [I 2022-10-26 12:00:10] ...trap/engines/engine.py.230: loss: 0.26759 [I 2022-10-26 12:00:10] ...optimizers/trijoint.py.151: total_norm: 3.691051 [I 2022-10-26 12:00:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 1550/2304 [I 2022-10-26 12:00:15] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:06:22 [I 2022-10-26 12:00:15] ...trap/engines/engine.py.229: process: 0.51308 | load: 0.00256 [I 2022-10-26 12:00:15] ...trap/engines/engine.py.230: loss: 0.25030 [I 2022-10-26 12:00:15] ...optimizers/trijoint.py.151: total_norm: 3.369883 [I 2022-10-26 12:00:20] ...trap/engines/engine.py.224: train: epoch 6 | batch 1560/2304 [I 2022-10-26 12:00:20] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:06:13 [I 2022-10-26 12:00:20] ...trap/engines/engine.py.229: process: 0.49811 | load: 0.00248 [I 2022-10-26 12:00:20] ...trap/engines/engine.py.230: loss: 0.25894 [I 2022-10-26 12:00:20] ...optimizers/trijoint.py.151: total_norm: 3.463216 [I 2022-10-26 12:00:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 1570/2304 [I 2022-10-26 12:00:25] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:06:10 [I 2022-10-26 12:00:25] ...trap/engines/engine.py.229: process: 0.50913 | load: 0.00247 [I 2022-10-26 12:00:25] ...trap/engines/engine.py.230: loss: 0.22266 [I 2022-10-26 12:00:25] ...optimizers/trijoint.py.151: total_norm: 3.064294 [I 2022-10-26 12:00:30] ...trap/engines/engine.py.224: train: epoch 6 | batch 1580/2304 [I 2022-10-26 12:00:30] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:06:06 [I 2022-10-26 12:00:30] ...trap/engines/engine.py.229: process: 0.50409 | load: 0.00255 [I 2022-10-26 12:00:30] ...trap/engines/engine.py.230: loss: 0.26607 [I 2022-10-26 12:00:30] ...optimizers/trijoint.py.151: total_norm: 3.343960 [I 2022-10-26 12:00:35] ...trap/engines/engine.py.224: train: epoch 6 | batch 1590/2304 [I 2022-10-26 12:00:35] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:05:55 [I 2022-10-26 12:00:35] ...trap/engines/engine.py.229: process: 0.48763 | load: 0.00253 [I 2022-10-26 12:00:35] ...trap/engines/engine.py.230: loss: 0.35064 [I 2022-10-26 12:00:35] ...optimizers/trijoint.py.151: total_norm: 3.689898 [I 2022-10-26 12:00:40] ...trap/engines/engine.py.224: train: epoch 6 | batch 1600/2304 [I 2022-10-26 12:00:40] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:05:55 [I 2022-10-26 12:00:40] ...trap/engines/engine.py.229: process: 0.50079 | load: 0.00250 [I 2022-10-26 12:00:40] ...trap/engines/engine.py.230: loss: 0.31118 [I 2022-10-26 12:00:40] ...optimizers/trijoint.py.151: total_norm: 3.655808 [I 2022-10-26 12:00:45] ...trap/engines/engine.py.224: train: epoch 6 | batch 1610/2304 [I 2022-10-26 12:00:45] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:05:46 [I 2022-10-26 12:00:45] ...trap/engines/engine.py.229: process: 0.50290 | load: 0.00255 [I 2022-10-26 12:00:45] ...trap/engines/engine.py.230: loss: 0.20011 [I 2022-10-26 12:00:45] ...optimizers/trijoint.py.151: total_norm: 3.189142 [I 2022-10-26 12:00:50] ...trap/engines/engine.py.224: train: epoch 6 | batch 1620/2304 [I 2022-10-26 12:00:50] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:05:39 [I 2022-10-26 12:00:50] ...trap/engines/engine.py.229: process: 0.49836 | load: 0.00247 [I 2022-10-26 12:00:50] ...trap/engines/engine.py.230: loss: 0.29362 [I 2022-10-26 12:00:50] ...optimizers/trijoint.py.151: total_norm: 3.986232 [I 2022-10-26 12:00:55] ...trap/engines/engine.py.224: train: epoch 6 | batch 1630/2304 [I 2022-10-26 12:00:55] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:05:37 [I 2022-10-26 12:00:55] ...trap/engines/engine.py.229: process: 0.49855 | load: 0.00250 [I 2022-10-26 12:00:55] ...trap/engines/engine.py.230: loss: 0.24479 [I 2022-10-26 12:00:55] ...optimizers/trijoint.py.151: total_norm: 2.989003 [I 2022-10-26 12:01:00] ...trap/engines/engine.py.224: train: epoch 6 | batch 1640/2304 [I 2022-10-26 12:01:00] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:05:31 [I 2022-10-26 12:01:00] ...trap/engines/engine.py.229: process: 0.50359 | load: 0.00254 [I 2022-10-26 12:01:00] ...trap/engines/engine.py.230: loss: 0.28671 [I 2022-10-26 12:01:00] ...optimizers/trijoint.py.151: total_norm: 3.644640 [I 2022-10-26 12:01:05] ...trap/engines/engine.py.224: train: epoch 6 | batch 1650/2304 [I 2022-10-26 12:01:05] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:05:26 [I 2022-10-26 12:01:05] ...trap/engines/engine.py.229: process: 0.49127 | load: 0.00246 [I 2022-10-26 12:01:05] ...trap/engines/engine.py.230: loss: 0.31517 [I 2022-10-26 12:01:05] ...optimizers/trijoint.py.151: total_norm: 3.647323 [I 2022-10-26 12:01:10] ...trap/engines/engine.py.224: train: epoch 6 | batch 1660/2304 [I 2022-10-26 12:01:10] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:05:22 [I 2022-10-26 12:01:10] ...trap/engines/engine.py.229: process: 0.50015 | load: 0.00250 [I 2022-10-26 12:01:10] ...trap/engines/engine.py.230: loss: 0.25687 [I 2022-10-26 12:01:10] ...optimizers/trijoint.py.151: total_norm: 3.017357 [I 2022-10-26 12:01:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 1670/2304 [I 2022-10-26 12:01:15] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:05:18 [I 2022-10-26 12:01:15] ...trap/engines/engine.py.229: process: 0.50637 | load: 0.00257 [I 2022-10-26 12:01:15] ...trap/engines/engine.py.230: loss: 0.33493 [I 2022-10-26 12:01:15] ...optimizers/trijoint.py.151: total_norm: 4.657517 [I 2022-10-26 12:01:20] ...trap/engines/engine.py.224: train: epoch 6 | batch 1680/2304 [I 2022-10-26 12:01:20] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:05:13 [I 2022-10-26 12:01:20] ...trap/engines/engine.py.229: process: 0.50403 | load: 0.00248 [I 2022-10-26 12:01:20] ...trap/engines/engine.py.230: loss: 0.23121 [I 2022-10-26 12:01:20] ...optimizers/trijoint.py.151: total_norm: 3.346101 [I 2022-10-26 12:01:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 1690/2304 [I 2022-10-26 12:01:25] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:05:07 [I 2022-10-26 12:01:25] ...trap/engines/engine.py.229: process: 0.50920 | load: 0.00249 [I 2022-10-26 12:01:25] ...trap/engines/engine.py.230: loss: 0.27324 [I 2022-10-26 12:01:25] ...optimizers/trijoint.py.151: total_norm: 4.142156 [I 2022-10-26 12:01:30] ...trap/engines/engine.py.224: train: epoch 6 | batch 1700/2304 [I 2022-10-26 12:01:30] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:05:03 [I 2022-10-26 12:01:30] ...trap/engines/engine.py.229: process: 0.50081 | load: 0.00244 [I 2022-10-26 12:01:30] ...trap/engines/engine.py.230: loss: 0.28935 [I 2022-10-26 12:01:30] ...optimizers/trijoint.py.151: total_norm: 3.566478 [I 2022-10-26 12:01:35] ...trap/engines/engine.py.224: train: epoch 6 | batch 1710/2304 [I 2022-10-26 12:01:35] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:04:54 [I 2022-10-26 12:01:35] ...trap/engines/engine.py.229: process: 0.49757 | load: 0.00244 [I 2022-10-26 12:01:35] ...trap/engines/engine.py.230: loss: 0.36223 [I 2022-10-26 12:01:35] ...optimizers/trijoint.py.151: total_norm: 5.060033 [I 2022-10-26 12:01:40] ...trap/engines/engine.py.224: train: epoch 6 | batch 1720/2304 [I 2022-10-26 12:01:40] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:04:50 [I 2022-10-26 12:01:40] ...trap/engines/engine.py.229: process: 0.50516 | load: 0.00244 [I 2022-10-26 12:01:40] ...trap/engines/engine.py.230: loss: 0.31407 [I 2022-10-26 12:01:40] ...optimizers/trijoint.py.151: total_norm: 3.689897 [I 2022-10-26 12:01:45] ...trap/engines/engine.py.224: train: epoch 6 | batch 1730/2304 [I 2022-10-26 12:01:45] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:04:46 [I 2022-10-26 12:01:45] ...trap/engines/engine.py.229: process: 0.50256 | load: 0.00245 [I 2022-10-26 12:01:45] ...trap/engines/engine.py.230: loss: 0.33142 [I 2022-10-26 12:01:45] ...optimizers/trijoint.py.151: total_norm: 3.875755 [I 2022-10-26 12:01:50] ...trap/engines/engine.py.224: train: epoch 6 | batch 1740/2304 [I 2022-10-26 12:01:50] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:04:42 [I 2022-10-26 12:01:50] ...trap/engines/engine.py.229: process: 0.49941 | load: 0.00322 [I 2022-10-26 12:01:50] ...trap/engines/engine.py.230: loss: 0.37570 [I 2022-10-26 12:01:50] ...optimizers/trijoint.py.151: total_norm: 3.983339 [I 2022-10-26 12:01:55] ...trap/engines/engine.py.224: train: epoch 6 | batch 1750/2304 [I 2022-10-26 12:01:55] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:04:38 [I 2022-10-26 12:01:55] ...trap/engines/engine.py.229: process: 0.50729 | load: 0.00255 [I 2022-10-26 12:01:55] ...trap/engines/engine.py.230: loss: 0.25966 [I 2022-10-26 12:01:55] ...optimizers/trijoint.py.151: total_norm: 3.355225 [I 2022-10-26 12:02:00] ...trap/engines/engine.py.224: train: epoch 6 | batch 1760/2304 [I 2022-10-26 12:02:00] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:04:32 [I 2022-10-26 12:02:00] ...trap/engines/engine.py.229: process: 0.49897 | load: 0.00251 [I 2022-10-26 12:02:00] ...trap/engines/engine.py.230: loss: 0.22428 [I 2022-10-26 12:02:00] ...optimizers/trijoint.py.151: total_norm: 2.803161 [I 2022-10-26 12:02:05] ...trap/engines/engine.py.224: train: epoch 6 | batch 1770/2304 [I 2022-10-26 12:02:05] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:04:28 [I 2022-10-26 12:02:05] ...trap/engines/engine.py.229: process: 0.50429 | load: 0.00247 [I 2022-10-26 12:02:05] ...trap/engines/engine.py.230: loss: 0.23238 [I 2022-10-26 12:02:05] ...optimizers/trijoint.py.151: total_norm: 4.191577 [I 2022-10-26 12:02:10] ...trap/engines/engine.py.224: train: epoch 6 | batch 1780/2304 [I 2022-10-26 12:02:10] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:04:22 [I 2022-10-26 12:02:10] ...trap/engines/engine.py.229: process: 0.49943 | load: 0.00246 [I 2022-10-26 12:02:10] ...trap/engines/engine.py.230: loss: 0.21124 [I 2022-10-26 12:02:10] ...optimizers/trijoint.py.151: total_norm: 4.476735 [I 2022-10-26 12:02:15] ...trap/engines/engine.py.224: train: epoch 6 | batch 1790/2304 [I 2022-10-26 12:02:15] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:04:17 [I 2022-10-26 12:02:15] ...trap/engines/engine.py.229: process: 0.50462 | load: 0.00246 [I 2022-10-26 12:02:15] ...trap/engines/engine.py.230: loss: 0.25706 [I 2022-10-26 12:02:15] ...optimizers/trijoint.py.151: total_norm: 2.234512 [I 2022-10-26 12:02:20] ...trap/engines/engine.py.224: train: epoch 6 | batch 1800/2304 [I 2022-10-26 12:02:20] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:04:12 [I 2022-10-26 12:02:20] ...trap/engines/engine.py.229: process: 0.49673 | load: 0.00249 [I 2022-10-26 12:02:20] ...trap/engines/engine.py.230: loss: 0.24498 [I 2022-10-26 12:02:20] ...optimizers/trijoint.py.151: total_norm: 3.707538 [I 2022-10-26 12:02:25] ...trap/engines/engine.py.224: train: epoch 6 | batch 1810/2304 [I 2022-10-26 12:02:25] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:04:06 [I 2022-10-26 12:02:25] ...trap/engines/engine.py.229: process: 0.50331 | load: 0.00247 [I 2022-10-26 12:02:25] ...trap/engines/engine.py.230: loss: 0.17899 [I 2022-10-26 12:02:25] ...optimizers/trijoint.py.151: total_norm: 1.906829 [I 2022-10-26 12:02:30] ...trap/engines/engine.py.224: train: epoch 6 | batch 1820/2304 [I 2022-10-26 12:02:30] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:04:01 [I 2022-10-26 12:02:30] ...trap/engines/engine.py.229: process: 0.49871 | load: 0.00246 [I 2022-10-26 12:02:30] ...trap/engines/engine.py.230: loss: 0.23048 [I 2022-10-26 12:02:30] ...optimizers/trijoint.py.151: total_norm: 3.380818 [I 2022-10-26 12:02:35] ...trap/engines/engine.py.224: train: epoch 6 | batch 1830/2304 [I 2022-10-26 12:02:35] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:03:55 [I 2022-10-26 12:02:35] ...trap/engines/engine.py.229: process: 0.49440 | load: 0.00256 [I 2022-10-26 12:02:35] ...trap/engines/engine.py.230: loss: 0.31777 [I 2022-10-26 12:02:35] ...optimizers/trijoint.py.151: total_norm: 4.293699 [I 2022-10-26 12:02:41] ...trap/engines/engine.py.224: train: epoch 6 | batch 1840/2304 [I 2022-10-26 12:02:41] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:03:51 [I 2022-10-26 12:02:41] ...trap/engines/engine.py.229: process: 0.49979 | load: 0.00251 [I 2022-10-26 12:02:41] ...trap/engines/engine.py.230: loss: 0.28982 [I 2022-10-26 12:02:41] ...optimizers/trijoint.py.151: total_norm: 4.392007 [I 2022-10-26 12:02:46] ...trap/engines/engine.py.224: train: epoch 6 | batch 1850/2304 [I 2022-10-26 12:02:46] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:03:48 [I 2022-10-26 12:02:46] ...trap/engines/engine.py.229: process: 0.51521 | load: 0.00265 [I 2022-10-26 12:02:46] ...trap/engines/engine.py.230: loss: 0.20401 [I 2022-10-26 12:02:46] ...optimizers/trijoint.py.151: total_norm: 2.637777 [I 2022-10-26 12:02:51] ...trap/engines/engine.py.224: train: epoch 6 | batch 1860/2304 [I 2022-10-26 12:02:51] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:03:41 [I 2022-10-26 12:02:51] ...trap/engines/engine.py.229: process: 0.49074 | load: 0.00266 [I 2022-10-26 12:02:51] ...trap/engines/engine.py.230: loss: 0.23990 [I 2022-10-26 12:02:51] ...optimizers/trijoint.py.151: total_norm: 2.320799 [I 2022-10-26 12:02:56] ...trap/engines/engine.py.224: train: epoch 6 | batch 1870/2304 [I 2022-10-26 12:02:56] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:03:38 [I 2022-10-26 12:02:56] ...trap/engines/engine.py.229: process: 0.49422 | load: 0.00266 [I 2022-10-26 12:02:56] ...trap/engines/engine.py.230: loss: 0.34810 [I 2022-10-26 12:02:56] ...optimizers/trijoint.py.151: total_norm: 4.367723 [I 2022-10-26 12:03:01] ...trap/engines/engine.py.224: train: epoch 6 | batch 1880/2304 [I 2022-10-26 12:03:01] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:03:32 [I 2022-10-26 12:03:01] ...trap/engines/engine.py.229: process: 0.50728 | load: 0.00271 [I 2022-10-26 12:03:01] ...trap/engines/engine.py.230: loss: 0.32332 [I 2022-10-26 12:03:01] ...optimizers/trijoint.py.151: total_norm: 4.092729 [I 2022-10-26 12:03:06] ...trap/engines/engine.py.224: train: epoch 6 | batch 1890/2304 [I 2022-10-26 12:03:06] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:03:27 [I 2022-10-26 12:03:06] ...trap/engines/engine.py.229: process: 0.50581 | load: 0.00279 [I 2022-10-26 12:03:06] ...trap/engines/engine.py.230: loss: 0.23993 [I 2022-10-26 12:03:06] ...optimizers/trijoint.py.151: total_norm: 2.908127 [I 2022-10-26 12:03:11] ...trap/engines/engine.py.224: train: epoch 6 | batch 1900/2304 [I 2022-10-26 12:03:11] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:03:22 [I 2022-10-26 12:03:11] ...trap/engines/engine.py.229: process: 0.50618 | load: 0.00274 [I 2022-10-26 12:03:11] ...trap/engines/engine.py.230: loss: 0.22201 [I 2022-10-26 12:03:11] ...optimizers/trijoint.py.151: total_norm: 3.294065 [I 2022-10-26 12:03:16] ...trap/engines/engine.py.224: train: epoch 6 | batch 1910/2304 [I 2022-10-26 12:03:16] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:03:17 [I 2022-10-26 12:03:16] ...trap/engines/engine.py.229: process: 0.51181 | load: 0.00269 [I 2022-10-26 12:03:16] ...trap/engines/engine.py.230: loss: 0.21243 [I 2022-10-26 12:03:16] ...optimizers/trijoint.py.151: total_norm: 2.568849 [I 2022-10-26 12:03:21] ...trap/engines/engine.py.224: train: epoch 6 | batch 1920/2304 [I 2022-10-26 12:03:21] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:03:12 [I 2022-10-26 12:03:21] ...trap/engines/engine.py.229: process: 0.50695 | load: 0.00263 [I 2022-10-26 12:03:21] ...trap/engines/engine.py.230: loss: 0.21250 [I 2022-10-26 12:03:21] ...optimizers/trijoint.py.151: total_norm: 3.441904 [I 2022-10-26 12:03:26] ...trap/engines/engine.py.224: train: epoch 6 | batch 1930/2304 [I 2022-10-26 12:03:26] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:03:06 [I 2022-10-26 12:03:26] ...trap/engines/engine.py.229: process: 0.49230 | load: 0.00262 [I 2022-10-26 12:03:26] ...trap/engines/engine.py.230: loss: 0.21881 [I 2022-10-26 12:03:26] ...optimizers/trijoint.py.151: total_norm: 3.600059 [I 2022-10-26 12:03:31] ...trap/engines/engine.py.224: train: epoch 6 | batch 1940/2304 [I 2022-10-26 12:03:31] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:03:02 [I 2022-10-26 12:03:31] ...trap/engines/engine.py.229: process: 0.50943 | load: 0.00289 [I 2022-10-26 12:03:31] ...trap/engines/engine.py.230: loss: 0.31831 [I 2022-10-26 12:03:31] ...optimizers/trijoint.py.151: total_norm: 3.463091 [I 2022-10-26 12:03:36] ...trap/engines/engine.py.224: train: epoch 6 | batch 1950/2304 [I 2022-10-26 12:03:36] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:02:57 [I 2022-10-26 12:03:36] ...trap/engines/engine.py.229: process: 0.50207 | load: 0.00250 [I 2022-10-26 12:03:36] ...trap/engines/engine.py.230: loss: 0.27199 [I 2022-10-26 12:03:36] ...optimizers/trijoint.py.151: total_norm: 3.257566 [I 2022-10-26 12:03:41] ...trap/engines/engine.py.224: train: epoch 6 | batch 1960/2304 [I 2022-10-26 12:03:41] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:02:53 [I 2022-10-26 12:03:41] ...trap/engines/engine.py.229: process: 0.49928 | load: 0.00250 [I 2022-10-26 12:03:41] ...trap/engines/engine.py.230: loss: 0.27312 [I 2022-10-26 12:03:41] ...optimizers/trijoint.py.151: total_norm: 3.321138 [I 2022-10-26 12:03:46] ...trap/engines/engine.py.224: train: epoch 6 | batch 1970/2304 [I 2022-10-26 12:03:46] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:02:48 [I 2022-10-26 12:03:46] ...trap/engines/engine.py.229: process: 0.49884 | load: 0.00245 [I 2022-10-26 12:03:46] ...trap/engines/engine.py.230: loss: 0.24055 [I 2022-10-26 12:03:46] ...optimizers/trijoint.py.151: total_norm: 3.001843 [I 2022-10-26 12:03:51] ...trap/engines/engine.py.224: train: epoch 6 | batch 1980/2304 [I 2022-10-26 12:03:51] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:02:43 [I 2022-10-26 12:03:51] ...trap/engines/engine.py.229: process: 0.49143 | load: 0.00247 [I 2022-10-26 12:03:51] ...trap/engines/engine.py.230: loss: 0.22786 [I 2022-10-26 12:03:51] ...optimizers/trijoint.py.151: total_norm: 3.347538 [I 2022-10-26 12:03:56] ...trap/engines/engine.py.224: train: epoch 6 | batch 1990/2304 [I 2022-10-26 12:03:56] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:02:54 [I 2022-10-26 12:03:56] ...trap/engines/engine.py.229: process: 0.50831 | load: 0.00256 [I 2022-10-26 12:03:56] ...trap/engines/engine.py.230: loss: 0.16901 [I 2022-10-26 12:03:56] ...optimizers/trijoint.py.151: total_norm: 3.152612 [I 2022-10-26 12:04:01] ...trap/engines/engine.py.224: train: epoch 6 | batch 2000/2304 [I 2022-10-26 12:04:01] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:02:33 [I 2022-10-26 12:04:01] ...trap/engines/engine.py.229: process: 0.49539 | load: 0.00252 [I 2022-10-26 12:04:01] ...trap/engines/engine.py.230: loss: 0.23176 [I 2022-10-26 12:04:01] ...optimizers/trijoint.py.151: total_norm: 3.460642 [I 2022-10-26 12:04:06] ...trap/engines/engine.py.224: train: epoch 6 | batch 2010/2304 [I 2022-10-26 12:04:06] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:02:27 [I 2022-10-26 12:04:06] ...trap/engines/engine.py.229: process: 0.51179 | load: 0.00246 [I 2022-10-26 12:04:06] ...trap/engines/engine.py.230: loss: 0.25172 [I 2022-10-26 12:04:06] ...optimizers/trijoint.py.151: total_norm: 4.638733 [I 2022-10-26 12:04:11] ...trap/engines/engine.py.224: train: epoch 6 | batch 2020/2304 [I 2022-10-26 12:04:11] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:02:22 [I 2022-10-26 12:04:11] ...trap/engines/engine.py.229: process: 0.49536 | load: 0.00247 [I 2022-10-26 12:04:11] ...trap/engines/engine.py.230: loss: 0.25999 [I 2022-10-26 12:04:11] ...optimizers/trijoint.py.151: total_norm: 3.503677 [I 2022-10-26 12:04:16] ...trap/engines/engine.py.224: train: epoch 6 | batch 2030/2304 [I 2022-10-26 12:04:16] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:02:17 [I 2022-10-26 12:04:16] ...trap/engines/engine.py.229: process: 0.50100 | load: 0.00260 [I 2022-10-26 12:04:16] ...trap/engines/engine.py.230: loss: 0.26450 [I 2022-10-26 12:04:16] ...optimizers/trijoint.py.151: total_norm: 3.618544 [I 2022-10-26 12:04:21] ...trap/engines/engine.py.224: train: epoch 6 | batch 2040/2304 [I 2022-10-26 12:04:21] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:02:10 [I 2022-10-26 12:04:21] ...trap/engines/engine.py.229: process: 0.49226 | load: 0.00243 [I 2022-10-26 12:04:21] ...trap/engines/engine.py.230: loss: 0.25544 [I 2022-10-26 12:04:21] ...optimizers/trijoint.py.151: total_norm: 3.668625 [I 2022-10-26 12:04:26] ...trap/engines/engine.py.224: train: epoch 6 | batch 2050/2304 [I 2022-10-26 12:04:26] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:02:07 [I 2022-10-26 12:04:26] ...trap/engines/engine.py.229: process: 0.49727 | load: 0.00253 [I 2022-10-26 12:04:26] ...trap/engines/engine.py.230: loss: 0.24212 [I 2022-10-26 12:04:26] ...optimizers/trijoint.py.151: total_norm: 3.509558 [I 2022-10-26 12:04:31] ...trap/engines/engine.py.224: train: epoch 6 | batch 2060/2304 [I 2022-10-26 12:04:31] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:02:02 [I 2022-10-26 12:04:31] ...trap/engines/engine.py.229: process: 0.50095 | load: 0.00249 [I 2022-10-26 12:04:31] ...trap/engines/engine.py.230: loss: 0.30130 [I 2022-10-26 12:04:31] ...optimizers/trijoint.py.151: total_norm: 4.007177 [I 2022-10-26 12:04:36] ...trap/engines/engine.py.224: train: epoch 6 | batch 2070/2304 [I 2022-10-26 12:04:36] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:01:57 [I 2022-10-26 12:04:36] ...trap/engines/engine.py.229: process: 0.50040 | load: 0.00245 [I 2022-10-26 12:04:36] ...trap/engines/engine.py.230: loss: 0.26388 [I 2022-10-26 12:04:36] ...optimizers/trijoint.py.151: total_norm: 3.109192 [I 2022-10-26 12:04:41] ...trap/engines/engine.py.224: train: epoch 6 | batch 2080/2304 [I 2022-10-26 12:04:41] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:01:51 [I 2022-10-26 12:04:41] ...trap/engines/engine.py.229: process: 0.50268 | load: 0.00249 [I 2022-10-26 12:04:41] ...trap/engines/engine.py.230: loss: 0.22489 [I 2022-10-26 12:04:41] ...optimizers/trijoint.py.151: total_norm: 3.038409 [I 2022-10-26 12:04:46] ...trap/engines/engine.py.224: train: epoch 6 | batch 2090/2304 [I 2022-10-26 12:04:46] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:01:46 [I 2022-10-26 12:04:46] ...trap/engines/engine.py.229: process: 0.49984 | load: 0.00245 [I 2022-10-26 12:04:46] ...trap/engines/engine.py.230: loss: 0.33706 [I 2022-10-26 12:04:46] ...optimizers/trijoint.py.151: total_norm: 3.998341 [I 2022-10-26 12:04:52] ...trap/engines/engine.py.224: train: epoch 6 | batch 2100/2304 [I 2022-10-26 12:04:52] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:01:42 [I 2022-10-26 12:04:52] ...trap/engines/engine.py.229: process: 0.49713 | load: 0.00247 [I 2022-10-26 12:04:52] ...trap/engines/engine.py.230: loss: 0.24881 [I 2022-10-26 12:04:52] ...optimizers/trijoint.py.151: total_norm: 3.622774 [I 2022-10-26 12:04:57] ...trap/engines/engine.py.224: train: epoch 6 | batch 2110/2304 [I 2022-10-26 12:04:57] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:01:37 [I 2022-10-26 12:04:57] ...trap/engines/engine.py.229: process: 0.50454 | load: 0.00253 [I 2022-10-26 12:04:57] ...trap/engines/engine.py.230: loss: 0.24564 [I 2022-10-26 12:04:57] ...optimizers/trijoint.py.151: total_norm: 2.999803 [I 2022-10-26 12:05:02] ...trap/engines/engine.py.224: train: epoch 6 | batch 2120/2304 [I 2022-10-26 12:05:02] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:01:34 [I 2022-10-26 12:05:02] ...trap/engines/engine.py.229: process: 0.50347 | load: 0.00259 [I 2022-10-26 12:05:02] ...trap/engines/engine.py.230: loss: 0.27020 [I 2022-10-26 12:05:02] ...optimizers/trijoint.py.151: total_norm: 3.369636 [I 2022-10-26 12:05:07] ...trap/engines/engine.py.224: train: epoch 6 | batch 2130/2304 [I 2022-10-26 12:05:07] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:01:27 [I 2022-10-26 12:05:07] ...trap/engines/engine.py.229: process: 0.49211 | load: 0.00244 [I 2022-10-26 12:05:07] ...trap/engines/engine.py.230: loss: 0.27971 [I 2022-10-26 12:05:07] ...optimizers/trijoint.py.151: total_norm: 3.520955 [I 2022-10-26 12:05:12] ...trap/engines/engine.py.224: train: epoch 6 | batch 2140/2304 [I 2022-10-26 12:05:12] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:01:26 [I 2022-10-26 12:05:12] ...trap/engines/engine.py.229: process: 0.64647 | load: 0.00258 [I 2022-10-26 12:05:12] ...trap/engines/engine.py.230: loss: 0.32785 [I 2022-10-26 12:05:12] ...optimizers/trijoint.py.151: total_norm: 4.045518 [I 2022-10-26 12:05:17] ...trap/engines/engine.py.224: train: epoch 6 | batch 2150/2304 [I 2022-10-26 12:05:17] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:01:17 [I 2022-10-26 12:05:17] ...trap/engines/engine.py.229: process: 0.49142 | load: 0.00246 [I 2022-10-26 12:05:17] ...trap/engines/engine.py.230: loss: 0.19271 [I 2022-10-26 12:05:17] ...optimizers/trijoint.py.151: total_norm: 3.352317 [I 2022-10-26 12:05:22] ...trap/engines/engine.py.224: train: epoch 6 | batch 2160/2304 [I 2022-10-26 12:05:22] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:01:11 [I 2022-10-26 12:05:22] ...trap/engines/engine.py.229: process: 0.49970 | load: 0.00244 [I 2022-10-26 12:05:22] ...trap/engines/engine.py.230: loss: 0.20327 [I 2022-10-26 12:05:22] ...optimizers/trijoint.py.151: total_norm: 2.820635 [I 2022-10-26 12:05:27] ...trap/engines/engine.py.224: train: epoch 6 | batch 2170/2304 [I 2022-10-26 12:05:27] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:01:06 [I 2022-10-26 12:05:27] ...trap/engines/engine.py.229: process: 0.49825 | load: 0.00246 [I 2022-10-26 12:05:27] ...trap/engines/engine.py.230: loss: 0.20764 [I 2022-10-26 12:05:27] ...optimizers/trijoint.py.151: total_norm: 3.656207 [I 2022-10-26 12:05:32] ...trap/engines/engine.py.224: train: epoch 6 | batch 2180/2304 [I 2022-10-26 12:05:32] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:01:01 [I 2022-10-26 12:05:32] ...trap/engines/engine.py.229: process: 0.50107 | load: 0.00254 [I 2022-10-26 12:05:32] ...trap/engines/engine.py.230: loss: 0.28403 [I 2022-10-26 12:05:32] ...optimizers/trijoint.py.151: total_norm: 4.092959 [I 2022-10-26 12:05:37] ...trap/engines/engine.py.224: train: epoch 6 | batch 2190/2304 [I 2022-10-26 12:05:37] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:00:56 [I 2022-10-26 12:05:37] ...trap/engines/engine.py.229: process: 0.49570 | load: 0.00250 [I 2022-10-26 12:05:37] ...trap/engines/engine.py.230: loss: 0.31622 [I 2022-10-26 12:05:37] ...optimizers/trijoint.py.151: total_norm: 4.082462 [I 2022-10-26 12:05:42] ...trap/engines/engine.py.224: train: epoch 6 | batch 2200/2304 [I 2022-10-26 12:05:42] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:00:51 [I 2022-10-26 12:05:42] ...trap/engines/engine.py.229: process: 0.49587 | load: 0.00248 [I 2022-10-26 12:05:42] ...trap/engines/engine.py.230: loss: 0.19798 [I 2022-10-26 12:05:42] ...optimizers/trijoint.py.151: total_norm: 2.770709 [I 2022-10-26 12:05:47] ...trap/engines/engine.py.224: train: epoch 6 | batch 2210/2304 [I 2022-10-26 12:05:47] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:00:46 [I 2022-10-26 12:05:47] ...trap/engines/engine.py.229: process: 0.49557 | load: 0.00245 [I 2022-10-26 12:05:47] ...trap/engines/engine.py.230: loss: 0.26712 [I 2022-10-26 12:05:47] ...optimizers/trijoint.py.151: total_norm: 3.527416 [I 2022-10-26 12:05:52] ...trap/engines/engine.py.224: train: epoch 6 | batch 2220/2304 [I 2022-10-26 12:05:52] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:00:42 [I 2022-10-26 12:05:52] ...trap/engines/engine.py.229: process: 0.48404 | load: 0.00244 [I 2022-10-26 12:05:52] ...trap/engines/engine.py.230: loss: 0.22544 [I 2022-10-26 12:05:52] ...optimizers/trijoint.py.151: total_norm: 2.567481 [I 2022-10-26 12:05:57] ...trap/engines/engine.py.224: train: epoch 6 | batch 2230/2304 [I 2022-10-26 12:05:57] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:00:37 [I 2022-10-26 12:05:57] ...trap/engines/engine.py.229: process: 0.49526 | load: 0.00243 [I 2022-10-26 12:05:57] ...trap/engines/engine.py.230: loss: 0.32972 [I 2022-10-26 12:05:57] ...optimizers/trijoint.py.151: total_norm: 4.139964 [I 2022-10-26 12:06:02] ...trap/engines/engine.py.224: train: epoch 6 | batch 2240/2304 [I 2022-10-26 12:06:02] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:00:31 [I 2022-10-26 12:06:02] ...trap/engines/engine.py.229: process: 0.50073 | load: 0.00247 [I 2022-10-26 12:06:02] ...trap/engines/engine.py.230: loss: 0.29273 [I 2022-10-26 12:06:02] ...optimizers/trijoint.py.151: total_norm: 4.367486 [I 2022-10-26 12:06:07] ...trap/engines/engine.py.224: train: epoch 6 | batch 2250/2304 [I 2022-10-26 12:06:07] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:00:27 [I 2022-10-26 12:06:07] ...trap/engines/engine.py.229: process: 0.50317 | load: 0.00252 [I 2022-10-26 12:06:07] ...trap/engines/engine.py.230: loss: 0.32470 [I 2022-10-26 12:06:07] ...optimizers/trijoint.py.151: total_norm: 3.306624 [I 2022-10-26 12:06:12] ...trap/engines/engine.py.224: train: epoch 6 | batch 2260/2304 [I 2022-10-26 12:06:12] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:00:22 [I 2022-10-26 12:06:12] ...trap/engines/engine.py.229: process: 0.49362 | load: 0.00247 [I 2022-10-26 12:06:12] ...trap/engines/engine.py.230: loss: 0.32380 [I 2022-10-26 12:06:12] ...optimizers/trijoint.py.151: total_norm: 4.018484 [I 2022-10-26 12:06:17] ...trap/engines/engine.py.224: train: epoch 6 | batch 2270/2304 [I 2022-10-26 12:06:17] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:00:16 [I 2022-10-26 12:06:17] ...trap/engines/engine.py.229: process: 0.50788 | load: 0.00247 [I 2022-10-26 12:06:17] ...trap/engines/engine.py.230: loss: 0.35148 [I 2022-10-26 12:06:17] ...optimizers/trijoint.py.151: total_norm: 4.190016 [I 2022-10-26 12:06:22] ...trap/engines/engine.py.224: train: epoch 6 | batch 2280/2304 [I 2022-10-26 12:06:22] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:00:11 [I 2022-10-26 12:06:22] ...trap/engines/engine.py.229: process: 0.49882 | load: 0.00242 [I 2022-10-26 12:06:22] ...trap/engines/engine.py.230: loss: 0.24775 [I 2022-10-26 12:06:22] ...optimizers/trijoint.py.151: total_norm: 3.245196 [I 2022-10-26 12:06:27] ...trap/engines/engine.py.224: train: epoch 6 | batch 2290/2304 [I 2022-10-26 12:06:27] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:00:07 [I 2022-10-26 12:06:27] ...trap/engines/engine.py.229: process: 0.50130 | load: 0.00242 [I 2022-10-26 12:06:27] ...trap/engines/engine.py.230: loss: 0.23674 [I 2022-10-26 12:06:27] ...optimizers/trijoint.py.151: total_norm: 2.967310 [I 2022-10-26 12:06:32] ...trap/engines/engine.py.224: train: epoch 6 | batch 2300/2304 [I 2022-10-26 12:06:32] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:00:02 [I 2022-10-26 12:06:32] ...trap/engines/engine.py.229: process: 0.49218 | load: 0.00015 [I 2022-10-26 12:06:32] ...trap/engines/engine.py.230: loss: 0.23897 [I 2022-10-26 12:06:32] ...optimizers/trijoint.py.151: total_norm: 2.447505 [I 2022-10-26 12:06:34] ...trap/engines/engine.py.224: train: epoch 6 | batch 2304/2304 [I 2022-10-26 12:06:34] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:00:00 [I 2022-10-26 12:06:34] ...trap/engines/engine.py.229: process: 0.49373 | load: 0.00016 [I 2022-10-26 12:06:34] ...trap/engines/engine.py.230: loss: 0.38349 [I 2022-10-26 12:06:34] ...optimizers/trijoint.py.151: total_norm: 4.292707 [S 2022-10-26 12:06:35] ...trap/engines/engine.py.129: train_epoch.epoch: 6 [S 2022-10-26 12:06:35] ...trap/engines/engine.py.129: train_epoch.loss: 0.2651136391139599 [I 2022-10-26 12:06:35] ...s/criterions/triplet.py.84: margin = 0.08500000000000002 [I 2022-10-26 12:06:35] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 6 [I 2022-10-26 12:06:36] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 12:06:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 12:06:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 12:06:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 12:06:38] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 12:06:38] ...trap/engines/engine.py.321: eval: epoch 6 | batch 0/510 [I 2022-10-26 12:06:38] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:26 [I 2022-10-26 12:06:38] ...trap/engines/engine.py.326: process: 2.63971 | load: 2.25180 [I 2022-10-26 12:06:41] ...trap/engines/engine.py.321: eval: epoch 6 | batch 10/510 [I 2022-10-26 12:06:41] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:04:51 [I 2022-10-26 12:06:41] ...trap/engines/engine.py.326: process: 0.33729 | load: 0.00019 [I 2022-10-26 12:06:45] ...trap/engines/engine.py.321: eval: epoch 6 | batch 20/510 [I 2022-10-26 12:06:45] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:58 [I 2022-10-26 12:06:45] ...trap/engines/engine.py.326: process: 0.33968 | load: 0.00017 [I 2022-10-26 12:06:48] ...trap/engines/engine.py.321: eval: epoch 6 | batch 30/510 [I 2022-10-26 12:06:48] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:44 [I 2022-10-26 12:06:48] ...trap/engines/engine.py.326: process: 0.33024 | load: 0.00018 [I 2022-10-26 12:06:52] ...trap/engines/engine.py.321: eval: epoch 6 | batch 40/510 [I 2022-10-26 12:06:52] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:40 [I 2022-10-26 12:06:52] ...trap/engines/engine.py.326: process: 0.32733 | load: 0.00034 [I 2022-10-26 12:06:55] ...trap/engines/engine.py.321: eval: epoch 6 | batch 50/510 [I 2022-10-26 12:06:55] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-26 12:06:55] ...trap/engines/engine.py.326: process: 0.34361 | load: 0.00018 [I 2022-10-26 12:06:59] ...trap/engines/engine.py.321: eval: epoch 6 | batch 60/510 [I 2022-10-26 12:06:59] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:33 [I 2022-10-26 12:06:59] ...trap/engines/engine.py.326: process: 0.32920 | load: 0.00017 [I 2022-10-26 12:07:02] ...trap/engines/engine.py.321: eval: epoch 6 | batch 70/510 [I 2022-10-26 12:07:02] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:38 [I 2022-10-26 12:07:02] ...trap/engines/engine.py.326: process: 0.35962 | load: 0.00017 [I 2022-10-26 12:07:05] ...trap/engines/engine.py.321: eval: epoch 6 | batch 80/510 [I 2022-10-26 12:07:05] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:26 [I 2022-10-26 12:07:05] ...trap/engines/engine.py.326: process: 0.32793 | load: 0.00017 [I 2022-10-26 12:07:09] ...trap/engines/engine.py.321: eval: epoch 6 | batch 90/510 [I 2022-10-26 12:07:09] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-26 12:07:09] ...trap/engines/engine.py.326: process: 0.33681 | load: 0.00018 [I 2022-10-26 12:07:12] ...trap/engines/engine.py.321: eval: epoch 6 | batch 100/510 [I 2022-10-26 12:07:12] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 12:07:12] ...trap/engines/engine.py.326: process: 0.28586 | load: 0.00019 [I 2022-10-26 12:07:15] ...trap/engines/engine.py.321: eval: epoch 6 | batch 110/510 [I 2022-10-26 12:07:15] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-26 12:07:15] ...trap/engines/engine.py.326: process: 0.28864 | load: 0.00020 [I 2022-10-26 12:07:18] ...trap/engines/engine.py.321: eval: epoch 6 | batch 120/510 [I 2022-10-26 12:07:18] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-26 12:07:18] ...trap/engines/engine.py.326: process: 0.28436 | load: 0.00041 [I 2022-10-26 12:07:21] ...trap/engines/engine.py.321: eval: epoch 6 | batch 130/510 [I 2022-10-26 12:07:21] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 12:07:21] ...trap/engines/engine.py.326: process: 0.28383 | load: 0.00014 [I 2022-10-26 12:07:24] ...trap/engines/engine.py.321: eval: epoch 6 | batch 140/510 [I 2022-10-26 12:07:24] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-26 12:07:24] ...trap/engines/engine.py.326: process: 0.29142 | load: 0.00020 [I 2022-10-26 12:07:27] ...trap/engines/engine.py.321: eval: epoch 6 | batch 150/510 [I 2022-10-26 12:07:27] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 12:07:27] ...trap/engines/engine.py.326: process: 0.28338 | load: 0.00018 [I 2022-10-26 12:07:30] ...trap/engines/engine.py.321: eval: epoch 6 | batch 160/510 [I 2022-10-26 12:07:30] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:39 [I 2022-10-26 12:07:30] ...trap/engines/engine.py.326: process: 0.28568 | load: 0.00021 [I 2022-10-26 12:07:32] ...trap/engines/engine.py.321: eval: epoch 6 | batch 170/510 [I 2022-10-26 12:07:32] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:37 [I 2022-10-26 12:07:32] ...trap/engines/engine.py.326: process: 0.28362 | load: 0.00018 [I 2022-10-26 12:07:35] ...trap/engines/engine.py.321: eval: epoch 6 | batch 180/510 [I 2022-10-26 12:07:35] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:34 [I 2022-10-26 12:07:35] ...trap/engines/engine.py.326: process: 0.28085 | load: 0.00016 [I 2022-10-26 12:07:38] ...trap/engines/engine.py.321: eval: epoch 6 | batch 190/510 [I 2022-10-26 12:07:38] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:36 [I 2022-10-26 12:07:38] ...trap/engines/engine.py.326: process: 0.31055 | load: 0.00017 [I 2022-10-26 12:07:41] ...trap/engines/engine.py.321: eval: epoch 6 | batch 200/510 [I 2022-10-26 12:07:41] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 12:07:41] ...trap/engines/engine.py.326: process: 0.29359 | load: 0.00022 [I 2022-10-26 12:07:44] ...trap/engines/engine.py.321: eval: epoch 6 | batch 210/510 [I 2022-10-26 12:07:44] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 12:07:44] ...trap/engines/engine.py.326: process: 0.28981 | load: 0.00014 [I 2022-10-26 12:07:47] ...trap/engines/engine.py.321: eval: epoch 6 | batch 220/510 [I 2022-10-26 12:07:47] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-26 12:07:47] ...trap/engines/engine.py.326: process: 0.28977 | load: 0.00019 [I 2022-10-26 12:07:50] ...trap/engines/engine.py.321: eval: epoch 6 | batch 230/510 [I 2022-10-26 12:07:50] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 12:07:50] ...trap/engines/engine.py.326: process: 0.29032 | load: 0.00023 [I 2022-10-26 12:07:53] ...trap/engines/engine.py.321: eval: epoch 6 | batch 240/510 [I 2022-10-26 12:07:53] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 12:07:53] ...trap/engines/engine.py.326: process: 0.28800 | load: 0.00016 [I 2022-10-26 12:07:56] ...trap/engines/engine.py.321: eval: epoch 6 | batch 250/510 [I 2022-10-26 12:07:56] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-26 12:07:56] ...trap/engines/engine.py.326: process: 0.28817 | load: 0.00014 [I 2022-10-26 12:07:58] ...trap/engines/engine.py.321: eval: epoch 6 | batch 260/510 [I 2022-10-26 12:07:58] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-26 12:07:58] ...trap/engines/engine.py.326: process: 0.29057 | load: 0.00016 [I 2022-10-26 12:08:01] ...trap/engines/engine.py.321: eval: epoch 6 | batch 270/510 [I 2022-10-26 12:08:01] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 12:08:01] ...trap/engines/engine.py.326: process: 0.28836 | load: 0.00020 [I 2022-10-26 12:08:04] ...trap/engines/engine.py.321: eval: epoch 6 | batch 280/510 [I 2022-10-26 12:08:04] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:05 [I 2022-10-26 12:08:04] ...trap/engines/engine.py.326: process: 0.28135 | load: 0.00015 [I 2022-10-26 12:08:07] ...trap/engines/engine.py.321: eval: epoch 6 | batch 290/510 [I 2022-10-26 12:08:07] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 12:08:07] ...trap/engines/engine.py.326: process: 0.29045 | load: 0.00016 [I 2022-10-26 12:08:10] ...trap/engines/engine.py.321: eval: epoch 6 | batch 300/510 [I 2022-10-26 12:08:10] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 12:08:10] ...trap/engines/engine.py.326: process: 0.28790 | load: 0.00024 [I 2022-10-26 12:08:13] ...trap/engines/engine.py.321: eval: epoch 6 | batch 310/510 [I 2022-10-26 12:08:13] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 12:08:13] ...trap/engines/engine.py.326: process: 0.28335 | load: 0.00016 [I 2022-10-26 12:08:16] ...trap/engines/engine.py.321: eval: epoch 6 | batch 320/510 [I 2022-10-26 12:08:16] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:56 [I 2022-10-26 12:08:16] ...trap/engines/engine.py.326: process: 0.28982 | load: 0.00015 [I 2022-10-26 12:08:19] ...trap/engines/engine.py.321: eval: epoch 6 | batch 330/510 [I 2022-10-26 12:08:19] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 12:08:19] ...trap/engines/engine.py.326: process: 0.28728 | load: 0.00015 [I 2022-10-26 12:08:22] ...trap/engines/engine.py.321: eval: epoch 6 | batch 340/510 [I 2022-10-26 12:08:22] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 12:08:22] ...trap/engines/engine.py.326: process: 0.29075 | load: 0.00018 [I 2022-10-26 12:08:24] ...trap/engines/engine.py.321: eval: epoch 6 | batch 350/510 [I 2022-10-26 12:08:24] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-26 12:08:24] ...trap/engines/engine.py.326: process: 0.28912 | load: 0.00015 [I 2022-10-26 12:08:27] ...trap/engines/engine.py.321: eval: epoch 6 | batch 360/510 [I 2022-10-26 12:08:27] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:42 [I 2022-10-26 12:08:27] ...trap/engines/engine.py.326: process: 0.28875 | load: 0.00016 [I 2022-10-26 12:08:30] ...trap/engines/engine.py.321: eval: epoch 6 | batch 370/510 [I 2022-10-26 12:08:30] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 12:08:30] ...trap/engines/engine.py.326: process: 0.28592 | load: 0.00017 [I 2022-10-26 12:08:33] ...trap/engines/engine.py.321: eval: epoch 6 | batch 380/510 [I 2022-10-26 12:08:33] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 12:08:33] ...trap/engines/engine.py.326: process: 0.28912 | load: 0.00016 [I 2022-10-26 12:08:36] ...trap/engines/engine.py.321: eval: epoch 6 | batch 390/510 [I 2022-10-26 12:08:36] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 12:08:36] ...trap/engines/engine.py.326: process: 0.28552 | load: 0.00020 [I 2022-10-26 12:08:39] ...trap/engines/engine.py.321: eval: epoch 6 | batch 400/510 [I 2022-10-26 12:08:39] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 12:08:39] ...trap/engines/engine.py.326: process: 0.29248 | load: 0.00017 [I 2022-10-26 12:08:42] ...trap/engines/engine.py.321: eval: epoch 6 | batch 410/510 [I 2022-10-26 12:08:42] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 12:08:42] ...trap/engines/engine.py.326: process: 0.28728 | load: 0.00019 [I 2022-10-26 12:08:45] ...trap/engines/engine.py.321: eval: epoch 6 | batch 420/510 [I 2022-10-26 12:08:45] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 12:08:45] ...trap/engines/engine.py.326: process: 0.28392 | load: 0.00015 [I 2022-10-26 12:08:47] ...trap/engines/engine.py.321: eval: epoch 6 | batch 430/510 [I 2022-10-26 12:08:47] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:22 [I 2022-10-26 12:08:47] ...trap/engines/engine.py.326: process: 0.28486 | load: 0.00019 [I 2022-10-26 12:08:50] ...trap/engines/engine.py.321: eval: epoch 6 | batch 440/510 [I 2022-10-26 12:08:50] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:21 [I 2022-10-26 12:08:50] ...trap/engines/engine.py.326: process: 0.28906 | load: 0.00015 [I 2022-10-26 12:08:53] ...trap/engines/engine.py.321: eval: epoch 6 | batch 450/510 [I 2022-10-26 12:08:53] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 12:08:53] ...trap/engines/engine.py.326: process: 0.28363 | load: 0.00018 [I 2022-10-26 12:08:56] ...trap/engines/engine.py.321: eval: epoch 6 | batch 460/510 [I 2022-10-26 12:08:56] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 12:08:56] ...trap/engines/engine.py.326: process: 0.29465 | load: 0.00015 [I 2022-10-26 12:08:59] ...trap/engines/engine.py.321: eval: epoch 6 | batch 470/510 [I 2022-10-26 12:08:59] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 12:08:59] ...trap/engines/engine.py.326: process: 0.28613 | load: 0.00022 [I 2022-10-26 12:09:02] ...trap/engines/engine.py.321: eval: epoch 6 | batch 480/510 [I 2022-10-26 12:09:02] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 12:09:02] ...trap/engines/engine.py.326: process: 0.29146 | load: 0.00017 [I 2022-10-26 12:09:05] ...trap/engines/engine.py.321: eval: epoch 6 | batch 490/510 [I 2022-10-26 12:09:05] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 12:09:05] ...trap/engines/engine.py.326: process: 0.29195 | load: 0.00019 [I 2022-10-26 12:09:08] ...trap/engines/engine.py.321: eval: epoch 6 | batch 500/510 [I 2022-10-26 12:09:08] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 12:09:08] ...trap/engines/engine.py.326: process: 0.28334 | load: 0.00016 [I 2022-10-26 12:09:11] ...trap/engines/engine.py.321: eval: epoch 6 | batch 510/510 [I 2022-10-26 12:09:11] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 12:09:11] ...trap/engines/engine.py.326: process: 0.30152 | load: 0.00011 [S 2022-10-26 12:09:11] ...trap/engines/engine.py.132: eval_epoch.epoch: 6 [I 2022-10-26 12:09:11] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.00 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.452700 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.766800 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.850100 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.468100 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.771700 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.854000 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015278 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013288 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011049 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.018168 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.014758 [S 2022-10-26 12:09:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009077 [I 2022-10-26 12:09:16] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 12:09:16] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 12:09:16] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 12:09:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 12:09:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 12:09:16] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 12:09:19] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 12:09:25] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 12:09:25] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 12:09:25] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 12:09:28] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 12:09:34] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 12:09:34] ...trap/engines/engine.py.163: Training model on trainset for epoch 7 [I 2022-10-26 12:09:34] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 12:09:36] ...trap/engines/engine.py.224: train: epoch 7 | batch 0/2304 [I 2022-10-26 12:09:36] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:47 [I 2022-10-26 12:09:36] ...trap/engines/engine.py.229: process: 2.78124 | load: 2.21948 [I 2022-10-26 12:09:36] ...trap/engines/engine.py.230: loss: 0.30495 [I 2022-10-26 12:09:36] ...optimizers/trijoint.py.151: total_norm: 3.864992 [I 2022-10-26 12:09:41] ...trap/engines/engine.py.224: train: epoch 7 | batch 10/2304 [I 2022-10-26 12:09:41] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:29 [I 2022-10-26 12:09:41] ...trap/engines/engine.py.229: process: 0.49815 | load: 0.00454 [I 2022-10-26 12:09:41] ...trap/engines/engine.py.230: loss: 0.17562 [I 2022-10-26 12:09:41] ...optimizers/trijoint.py.151: total_norm: 2.847054 [I 2022-10-26 12:09:46] ...trap/engines/engine.py.224: train: epoch 7 | batch 20/2304 [I 2022-10-26 12:09:46] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:03 [I 2022-10-26 12:09:46] ...trap/engines/engine.py.229: process: 0.50589 | load: 0.00353 [I 2022-10-26 12:09:46] ...trap/engines/engine.py.230: loss: 0.21961 [I 2022-10-26 12:09:46] ...optimizers/trijoint.py.151: total_norm: 3.894209 [I 2022-10-26 12:09:51] ...trap/engines/engine.py.224: train: epoch 7 | batch 30/2304 [I 2022-10-26 12:09:51] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:04 [I 2022-10-26 12:09:51] ...trap/engines/engine.py.229: process: 0.49828 | load: 0.00310 [I 2022-10-26 12:09:51] ...trap/engines/engine.py.230: loss: 0.28844 [I 2022-10-26 12:09:51] ...optimizers/trijoint.py.151: total_norm: 3.588517 [I 2022-10-26 12:09:56] ...trap/engines/engine.py.224: train: epoch 7 | batch 40/2304 [I 2022-10-26 12:09:56] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:54 [I 2022-10-26 12:09:56] ...trap/engines/engine.py.229: process: 0.49717 | load: 0.00342 [I 2022-10-26 12:09:56] ...trap/engines/engine.py.230: loss: 0.26930 [I 2022-10-26 12:09:56] ...optimizers/trijoint.py.151: total_norm: 3.340342 [I 2022-10-26 12:10:01] ...trap/engines/engine.py.224: train: epoch 7 | batch 50/2304 [I 2022-10-26 12:10:01] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:51 [I 2022-10-26 12:10:01] ...trap/engines/engine.py.229: process: 0.49562 | load: 0.00289 [I 2022-10-26 12:10:01] ...trap/engines/engine.py.230: loss: 0.23996 [I 2022-10-26 12:10:01] ...optimizers/trijoint.py.151: total_norm: 3.592129 [I 2022-10-26 12:10:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 60/2304 [I 2022-10-26 12:10:07] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:32 [I 2022-10-26 12:10:07] ...trap/engines/engine.py.229: process: 0.49018 | load: 0.00309 [I 2022-10-26 12:10:07] ...trap/engines/engine.py.230: loss: 0.21789 [I 2022-10-26 12:10:07] ...optimizers/trijoint.py.151: total_norm: 3.289459 [I 2022-10-26 12:10:12] ...trap/engines/engine.py.224: train: epoch 7 | batch 70/2304 [I 2022-10-26 12:10:12] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:36 [I 2022-10-26 12:10:12] ...trap/engines/engine.py.229: process: 0.49289 | load: 0.00267 [I 2022-10-26 12:10:12] ...trap/engines/engine.py.230: loss: 0.20286 [I 2022-10-26 12:10:12] ...optimizers/trijoint.py.151: total_norm: 4.064885 [I 2022-10-26 12:10:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 80/2304 [I 2022-10-26 12:10:17] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:23 [I 2022-10-26 12:10:17] ...trap/engines/engine.py.229: process: 0.49885 | load: 0.00266 [I 2022-10-26 12:10:17] ...trap/engines/engine.py.230: loss: 0.29883 [I 2022-10-26 12:10:17] ...optimizers/trijoint.py.151: total_norm: 3.324949 [I 2022-10-26 12:10:21] ...trap/engines/engine.py.224: train: epoch 7 | batch 90/2304 [I 2022-10-26 12:10:21] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:18:13 [I 2022-10-26 12:10:21] ...trap/engines/engine.py.229: process: 0.48302 | load: 0.00268 [I 2022-10-26 12:10:21] ...trap/engines/engine.py.230: loss: 0.28921 [I 2022-10-26 12:10:21] ...optimizers/trijoint.py.151: total_norm: 3.845392 [I 2022-10-26 12:10:26] ...trap/engines/engine.py.224: train: epoch 7 | batch 100/2304 [I 2022-10-26 12:10:26] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:18:12 [I 2022-10-26 12:10:26] ...trap/engines/engine.py.229: process: 0.49076 | load: 0.00272 [I 2022-10-26 12:10:26] ...trap/engines/engine.py.230: loss: 0.23804 [I 2022-10-26 12:10:26] ...optimizers/trijoint.py.151: total_norm: 3.079543 [I 2022-10-26 12:10:31] ...trap/engines/engine.py.224: train: epoch 7 | batch 110/2304 [I 2022-10-26 12:10:31] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:18:12 [I 2022-10-26 12:10:31] ...trap/engines/engine.py.229: process: 0.50173 | load: 0.00289 [I 2022-10-26 12:10:31] ...trap/engines/engine.py.230: loss: 0.22650 [I 2022-10-26 12:10:31] ...optimizers/trijoint.py.151: total_norm: 3.297521 [I 2022-10-26 12:10:36] ...trap/engines/engine.py.224: train: epoch 7 | batch 120/2304 [I 2022-10-26 12:10:36] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:18:18 [I 2022-10-26 12:10:36] ...trap/engines/engine.py.229: process: 0.50523 | load: 0.00291 [I 2022-10-26 12:10:36] ...trap/engines/engine.py.230: loss: 0.24119 [I 2022-10-26 12:10:36] ...optimizers/trijoint.py.151: total_norm: 3.239463 [I 2022-10-26 12:10:41] ...trap/engines/engine.py.224: train: epoch 7 | batch 130/2304 [I 2022-10-26 12:10:41] ...trap/engines/engine.py.228: elapsed: 0:01:07 | left: 0:18:13 [I 2022-10-26 12:10:41] ...trap/engines/engine.py.229: process: 0.52989 | load: 0.00275 [I 2022-10-26 12:10:41] ...trap/engines/engine.py.230: loss: 0.29326 [I 2022-10-26 12:10:41] ...optimizers/trijoint.py.151: total_norm: 3.075544 [I 2022-10-26 12:10:46] ...trap/engines/engine.py.224: train: epoch 7 | batch 140/2304 [I 2022-10-26 12:10:46] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:18:01 [I 2022-10-26 12:10:46] ...trap/engines/engine.py.229: process: 0.49838 | load: 0.00263 [I 2022-10-26 12:10:46] ...trap/engines/engine.py.230: loss: 0.30393 [I 2022-10-26 12:10:46] ...optimizers/trijoint.py.151: total_norm: 3.948577 [I 2022-10-26 12:10:51] ...trap/engines/engine.py.224: train: epoch 7 | batch 150/2304 [I 2022-10-26 12:10:51] ...trap/engines/engine.py.228: elapsed: 0:01:17 | left: 0:17:54 [I 2022-10-26 12:10:51] ...trap/engines/engine.py.229: process: 0.49819 | load: 0.00275 [I 2022-10-26 12:10:51] ...trap/engines/engine.py.230: loss: 0.25514 [I 2022-10-26 12:10:51] ...optimizers/trijoint.py.151: total_norm: 3.526267 [I 2022-10-26 12:10:56] ...trap/engines/engine.py.224: train: epoch 7 | batch 160/2304 [I 2022-10-26 12:10:56] ...trap/engines/engine.py.228: elapsed: 0:01:22 | left: 0:17:52 [I 2022-10-26 12:10:56] ...trap/engines/engine.py.229: process: 0.50324 | load: 0.00268 [I 2022-10-26 12:10:56] ...trap/engines/engine.py.230: loss: 0.36102 [I 2022-10-26 12:10:56] ...optimizers/trijoint.py.151: total_norm: 3.806011 [I 2022-10-26 12:11:01] ...trap/engines/engine.py.224: train: epoch 7 | batch 170/2304 [I 2022-10-26 12:11:01] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:17:36 [I 2022-10-26 12:11:01] ...trap/engines/engine.py.229: process: 0.48560 | load: 0.00253 [I 2022-10-26 12:11:01] ...trap/engines/engine.py.230: loss: 0.33121 [I 2022-10-26 12:11:01] ...optimizers/trijoint.py.151: total_norm: 3.488178 [I 2022-10-26 12:11:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 180/2304 [I 2022-10-26 12:11:07] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:17:43 [I 2022-10-26 12:11:07] ...trap/engines/engine.py.229: process: 0.49717 | load: 0.00263 [I 2022-10-26 12:11:07] ...trap/engines/engine.py.230: loss: 0.37130 [I 2022-10-26 12:11:07] ...optimizers/trijoint.py.151: total_norm: 5.068488 [I 2022-10-26 12:11:12] ...trap/engines/engine.py.224: train: epoch 7 | batch 190/2304 [I 2022-10-26 12:11:12] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:17:43 [I 2022-10-26 12:11:12] ...trap/engines/engine.py.229: process: 0.49964 | load: 0.00289 [I 2022-10-26 12:11:12] ...trap/engines/engine.py.230: loss: 0.23504 [I 2022-10-26 12:11:12] ...optimizers/trijoint.py.151: total_norm: 3.557592 [I 2022-10-26 12:11:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 200/2304 [I 2022-10-26 12:11:17] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:17:26 [I 2022-10-26 12:11:17] ...trap/engines/engine.py.229: process: 0.49567 | load: 0.00246 [I 2022-10-26 12:11:17] ...trap/engines/engine.py.230: loss: 0.19641 [I 2022-10-26 12:11:17] ...optimizers/trijoint.py.151: total_norm: 3.307601 [I 2022-10-26 12:11:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 210/2304 [I 2022-10-26 12:11:22] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:17:20 [I 2022-10-26 12:11:22] ...trap/engines/engine.py.229: process: 0.49718 | load: 0.00256 [I 2022-10-26 12:11:22] ...trap/engines/engine.py.230: loss: 0.29307 [I 2022-10-26 12:11:22] ...optimizers/trijoint.py.151: total_norm: 4.597818 [I 2022-10-26 12:11:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 220/2304 [I 2022-10-26 12:11:27] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:17:29 [I 2022-10-26 12:11:27] ...trap/engines/engine.py.229: process: 0.50262 | load: 0.00260 [I 2022-10-26 12:11:27] ...trap/engines/engine.py.230: loss: 0.25688 [I 2022-10-26 12:11:27] ...optimizers/trijoint.py.151: total_norm: 3.308494 [I 2022-10-26 12:11:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 230/2304 [I 2022-10-26 12:11:32] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:19 [I 2022-10-26 12:11:32] ...trap/engines/engine.py.229: process: 0.49405 | load: 0.00254 [I 2022-10-26 12:11:32] ...trap/engines/engine.py.230: loss: 0.25582 [I 2022-10-26 12:11:32] ...optimizers/trijoint.py.151: total_norm: 3.477560 [I 2022-10-26 12:11:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 240/2304 [I 2022-10-26 12:11:37] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:10 [I 2022-10-26 12:11:37] ...trap/engines/engine.py.229: process: 0.49711 | load: 0.00269 [I 2022-10-26 12:11:37] ...trap/engines/engine.py.230: loss: 0.16642 [I 2022-10-26 12:11:37] ...optimizers/trijoint.py.151: total_norm: 2.594873 [I 2022-10-26 12:11:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 250/2304 [I 2022-10-26 12:11:42] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:21 [I 2022-10-26 12:11:42] ...trap/engines/engine.py.229: process: 0.50624 | load: 0.00251 [I 2022-10-26 12:11:42] ...trap/engines/engine.py.230: loss: 0.27543 [I 2022-10-26 12:11:42] ...optimizers/trijoint.py.151: total_norm: 4.024837 [I 2022-10-26 12:11:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 260/2304 [I 2022-10-26 12:11:47] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:13 [I 2022-10-26 12:11:47] ...trap/engines/engine.py.229: process: 0.50005 | load: 0.00257 [I 2022-10-26 12:11:47] ...trap/engines/engine.py.230: loss: 0.28870 [I 2022-10-26 12:11:47] ...optimizers/trijoint.py.151: total_norm: 3.016786 [I 2022-10-26 12:11:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 270/2304 [I 2022-10-26 12:11:52] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:17:14 [I 2022-10-26 12:11:52] ...trap/engines/engine.py.229: process: 0.50719 | load: 0.00256 [I 2022-10-26 12:11:52] ...trap/engines/engine.py.230: loss: 0.20353 [I 2022-10-26 12:11:52] ...optimizers/trijoint.py.151: total_norm: 2.864831 [I 2022-10-26 12:11:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 280/2304 [I 2022-10-26 12:11:57] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:16:56 [I 2022-10-26 12:11:57] ...trap/engines/engine.py.229: process: 0.48217 | load: 0.00260 [I 2022-10-26 12:11:57] ...trap/engines/engine.py.230: loss: 0.22492 [I 2022-10-26 12:11:57] ...optimizers/trijoint.py.151: total_norm: 3.952363 [I 2022-10-26 12:12:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 290/2304 [I 2022-10-26 12:12:02] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:16:36 [I 2022-10-26 12:12:02] ...trap/engines/engine.py.229: process: 0.49881 | load: 0.00248 [I 2022-10-26 12:12:02] ...trap/engines/engine.py.230: loss: 0.18444 [I 2022-10-26 12:12:02] ...optimizers/trijoint.py.151: total_norm: 2.989873 [I 2022-10-26 12:12:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 300/2304 [I 2022-10-26 12:12:07] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:16:30 [I 2022-10-26 12:12:07] ...trap/engines/engine.py.229: process: 0.48188 | load: 0.00245 [I 2022-10-26 12:12:07] ...trap/engines/engine.py.230: loss: 0.20536 [I 2022-10-26 12:12:07] ...optimizers/trijoint.py.151: total_norm: 3.117396 [I 2022-10-26 12:12:12] ...trap/engines/engine.py.224: train: epoch 7 | batch 310/2304 [I 2022-10-26 12:12:12] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:16:39 [I 2022-10-26 12:12:12] ...trap/engines/engine.py.229: process: 0.50939 | load: 0.00247 [I 2022-10-26 12:12:12] ...trap/engines/engine.py.230: loss: 0.39189 [I 2022-10-26 12:12:12] ...optimizers/trijoint.py.151: total_norm: 3.642584 [I 2022-10-26 12:12:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 320/2304 [I 2022-10-26 12:12:17] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:16:41 [I 2022-10-26 12:12:17] ...trap/engines/engine.py.229: process: 0.52036 | load: 0.00251 [I 2022-10-26 12:12:17] ...trap/engines/engine.py.230: loss: 0.20378 [I 2022-10-26 12:12:17] ...optimizers/trijoint.py.151: total_norm: 2.342240 [I 2022-10-26 12:12:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 330/2304 [I 2022-10-26 12:12:22] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:16:23 [I 2022-10-26 12:12:22] ...trap/engines/engine.py.229: process: 0.49854 | load: 0.00268 [I 2022-10-26 12:12:22] ...trap/engines/engine.py.230: loss: 0.26639 [I 2022-10-26 12:12:22] ...optimizers/trijoint.py.151: total_norm: 3.185177 [I 2022-10-26 12:12:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 340/2304 [I 2022-10-26 12:12:27] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:16:05 [I 2022-10-26 12:12:27] ...trap/engines/engine.py.229: process: 0.48520 | load: 0.00252 [I 2022-10-26 12:12:27] ...trap/engines/engine.py.230: loss: 0.18546 [I 2022-10-26 12:12:27] ...optimizers/trijoint.py.151: total_norm: 2.951250 [I 2022-10-26 12:12:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 350/2304 [I 2022-10-26 12:12:32] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:16:08 [I 2022-10-26 12:12:32] ...trap/engines/engine.py.229: process: 0.49195 | load: 0.00259 [I 2022-10-26 12:12:32] ...trap/engines/engine.py.230: loss: 0.21781 [I 2022-10-26 12:12:32] ...optimizers/trijoint.py.151: total_norm: 3.983707 [I 2022-10-26 12:12:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 360/2304 [I 2022-10-26 12:12:37] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:16:18 [I 2022-10-26 12:12:37] ...trap/engines/engine.py.229: process: 0.51308 | load: 0.00245 [I 2022-10-26 12:12:37] ...trap/engines/engine.py.230: loss: 0.15092 [I 2022-10-26 12:12:37] ...optimizers/trijoint.py.151: total_norm: 2.412764 [I 2022-10-26 12:12:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 370/2304 [I 2022-10-26 12:12:42] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:16:10 [I 2022-10-26 12:12:42] ...trap/engines/engine.py.229: process: 0.49927 | load: 0.00257 [I 2022-10-26 12:12:42] ...trap/engines/engine.py.230: loss: 0.23028 [I 2022-10-26 12:12:42] ...optimizers/trijoint.py.151: total_norm: 4.300151 [I 2022-10-26 12:12:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 380/2304 [I 2022-10-26 12:12:47] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:16:01 [I 2022-10-26 12:12:47] ...trap/engines/engine.py.229: process: 0.49213 | load: 0.00247 [I 2022-10-26 12:12:47] ...trap/engines/engine.py.230: loss: 0.20515 [I 2022-10-26 12:12:47] ...optimizers/trijoint.py.151: total_norm: 3.676671 [I 2022-10-26 12:12:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 390/2304 [I 2022-10-26 12:12:52] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:15:54 [I 2022-10-26 12:12:52] ...trap/engines/engine.py.229: process: 0.49528 | load: 0.00254 [I 2022-10-26 12:12:52] ...trap/engines/engine.py.230: loss: 0.16632 [I 2022-10-26 12:12:52] ...optimizers/trijoint.py.151: total_norm: 2.037539 [I 2022-10-26 12:12:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 400/2304 [I 2022-10-26 12:12:57] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:15:42 [I 2022-10-26 12:12:57] ...trap/engines/engine.py.229: process: 0.49603 | load: 0.00255 [I 2022-10-26 12:12:57] ...trap/engines/engine.py.230: loss: 0.31070 [I 2022-10-26 12:12:57] ...optimizers/trijoint.py.151: total_norm: 3.348737 [I 2022-10-26 12:13:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 410/2304 [I 2022-10-26 12:13:02] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:15:45 [I 2022-10-26 12:13:02] ...trap/engines/engine.py.229: process: 0.50196 | load: 0.00247 [I 2022-10-26 12:13:02] ...trap/engines/engine.py.230: loss: 0.22073 [I 2022-10-26 12:13:02] ...optimizers/trijoint.py.151: total_norm: 3.211037 [I 2022-10-26 12:13:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 420/2304 [I 2022-10-26 12:13:07] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:15:41 [I 2022-10-26 12:13:07] ...trap/engines/engine.py.229: process: 0.49942 | load: 0.00257 [I 2022-10-26 12:13:07] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-26 12:13:07] ...optimizers/trijoint.py.151: total_norm: 2.559741 [I 2022-10-26 12:13:12] ...trap/engines/engine.py.224: train: epoch 7 | batch 430/2304 [I 2022-10-26 12:13:12] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:15:29 [I 2022-10-26 12:13:12] ...trap/engines/engine.py.229: process: 0.50162 | load: 0.00244 [I 2022-10-26 12:13:12] ...trap/engines/engine.py.230: loss: 0.19665 [I 2022-10-26 12:13:12] ...optimizers/trijoint.py.151: total_norm: 2.780066 [I 2022-10-26 12:13:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 440/2304 [I 2022-10-26 12:13:17] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:15:29 [I 2022-10-26 12:13:17] ...trap/engines/engine.py.229: process: 0.50212 | load: 0.00246 [I 2022-10-26 12:13:17] ...trap/engines/engine.py.230: loss: 0.24602 [I 2022-10-26 12:13:17] ...optimizers/trijoint.py.151: total_norm: 3.844184 [I 2022-10-26 12:13:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 450/2304 [I 2022-10-26 12:13:22] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:15:23 [I 2022-10-26 12:13:22] ...trap/engines/engine.py.229: process: 0.50736 | load: 0.00253 [I 2022-10-26 12:13:22] ...trap/engines/engine.py.230: loss: 0.29006 [I 2022-10-26 12:13:22] ...optimizers/trijoint.py.151: total_norm: 4.165448 [I 2022-10-26 12:13:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 460/2304 [I 2022-10-26 12:13:27] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:15:16 [I 2022-10-26 12:13:27] ...trap/engines/engine.py.229: process: 0.48949 | load: 0.00253 [I 2022-10-26 12:13:27] ...trap/engines/engine.py.230: loss: 0.21477 [I 2022-10-26 12:13:27] ...optimizers/trijoint.py.151: total_norm: 3.001533 [I 2022-10-26 12:13:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 470/2304 [I 2022-10-26 12:13:32] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:15:21 [I 2022-10-26 12:13:32] ...trap/engines/engine.py.229: process: 0.49551 | load: 0.00242 [I 2022-10-26 12:13:32] ...trap/engines/engine.py.230: loss: 0.30601 [I 2022-10-26 12:13:32] ...optimizers/trijoint.py.151: total_norm: 3.656652 [I 2022-10-26 12:13:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 480/2304 [I 2022-10-26 12:13:37] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:15:18 [I 2022-10-26 12:13:37] ...trap/engines/engine.py.229: process: 0.50966 | load: 0.00251 [I 2022-10-26 12:13:37] ...trap/engines/engine.py.230: loss: 0.30132 [I 2022-10-26 12:13:37] ...optimizers/trijoint.py.151: total_norm: 3.771904 [I 2022-10-26 12:13:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 490/2304 [I 2022-10-26 12:13:42] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:15:06 [I 2022-10-26 12:13:42] ...trap/engines/engine.py.229: process: 0.49395 | load: 0.00248 [I 2022-10-26 12:13:42] ...trap/engines/engine.py.230: loss: 0.25890 [I 2022-10-26 12:13:42] ...optimizers/trijoint.py.151: total_norm: 3.317164 [I 2022-10-26 12:13:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 500/2304 [I 2022-10-26 12:13:47] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:15:05 [I 2022-10-26 12:13:47] ...trap/engines/engine.py.229: process: 0.50362 | load: 0.00251 [I 2022-10-26 12:13:47] ...trap/engines/engine.py.230: loss: 0.19008 [I 2022-10-26 12:13:47] ...optimizers/trijoint.py.151: total_norm: 2.659555 [I 2022-10-26 12:13:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 510/2304 [I 2022-10-26 12:13:52] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:14:54 [I 2022-10-26 12:13:52] ...trap/engines/engine.py.229: process: 0.49331 | load: 0.00245 [I 2022-10-26 12:13:52] ...trap/engines/engine.py.230: loss: 0.21914 [I 2022-10-26 12:13:52] ...optimizers/trijoint.py.151: total_norm: 3.467216 [I 2022-10-26 12:13:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 520/2304 [I 2022-10-26 12:13:57] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:14:47 [I 2022-10-26 12:13:57] ...trap/engines/engine.py.229: process: 0.50035 | load: 0.00247 [I 2022-10-26 12:13:57] ...trap/engines/engine.py.230: loss: 0.19101 [I 2022-10-26 12:13:57] ...optimizers/trijoint.py.151: total_norm: 3.109954 [I 2022-10-26 12:14:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 530/2304 [I 2022-10-26 12:14:02] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:14:51 [I 2022-10-26 12:14:02] ...trap/engines/engine.py.229: process: 0.49235 | load: 0.00254 [I 2022-10-26 12:14:02] ...trap/engines/engine.py.230: loss: 0.19778 [I 2022-10-26 12:14:02] ...optimizers/trijoint.py.151: total_norm: 3.674214 [I 2022-10-26 12:14:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 540/2304 [I 2022-10-26 12:14:07] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:14:49 [I 2022-10-26 12:14:07] ...trap/engines/engine.py.229: process: 0.51744 | load: 0.00252 [I 2022-10-26 12:14:07] ...trap/engines/engine.py.230: loss: 0.25037 [I 2022-10-26 12:14:07] ...optimizers/trijoint.py.151: total_norm: 3.665344 [I 2022-10-26 12:14:12] ...trap/engines/engine.py.224: train: epoch 7 | batch 550/2304 [I 2022-10-26 12:14:12] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:14:39 [I 2022-10-26 12:14:12] ...trap/engines/engine.py.229: process: 0.51236 | load: 0.00246 [I 2022-10-26 12:14:12] ...trap/engines/engine.py.230: loss: 0.22750 [I 2022-10-26 12:14:12] ...optimizers/trijoint.py.151: total_norm: 3.965905 [I 2022-10-26 12:14:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 560/2304 [I 2022-10-26 12:14:17] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:14:31 [I 2022-10-26 12:14:17] ...trap/engines/engine.py.229: process: 0.50266 | load: 0.00243 [I 2022-10-26 12:14:17] ...trap/engines/engine.py.230: loss: 0.23914 [I 2022-10-26 12:14:17] ...optimizers/trijoint.py.151: total_norm: 3.748571 [I 2022-10-26 12:14:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 570/2304 [I 2022-10-26 12:14:22] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:14:26 [I 2022-10-26 12:14:22] ...trap/engines/engine.py.229: process: 0.49860 | load: 0.00246 [I 2022-10-26 12:14:22] ...trap/engines/engine.py.230: loss: 0.23842 [I 2022-10-26 12:14:22] ...optimizers/trijoint.py.151: total_norm: 2.993183 [I 2022-10-26 12:14:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 580/2304 [I 2022-10-26 12:14:27] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:14:20 [I 2022-10-26 12:14:27] ...trap/engines/engine.py.229: process: 0.49937 | load: 0.00256 [I 2022-10-26 12:14:27] ...trap/engines/engine.py.230: loss: 0.24428 [I 2022-10-26 12:14:27] ...optimizers/trijoint.py.151: total_norm: 2.675122 [I 2022-10-26 12:14:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 590/2304 [I 2022-10-26 12:14:32] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:14:09 [I 2022-10-26 12:14:32] ...trap/engines/engine.py.229: process: 0.49146 | load: 0.00251 [I 2022-10-26 12:14:32] ...trap/engines/engine.py.230: loss: 0.29638 [I 2022-10-26 12:14:32] ...optimizers/trijoint.py.151: total_norm: 3.212865 [I 2022-10-26 12:14:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 600/2304 [I 2022-10-26 12:14:37] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:14:05 [I 2022-10-26 12:14:37] ...trap/engines/engine.py.229: process: 0.49490 | load: 0.00247 [I 2022-10-26 12:14:37] ...trap/engines/engine.py.230: loss: 0.19992 [I 2022-10-26 12:14:37] ...optimizers/trijoint.py.151: total_norm: 3.945717 [I 2022-10-26 12:14:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 610/2304 [I 2022-10-26 12:14:42] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:14:02 [I 2022-10-26 12:14:42] ...trap/engines/engine.py.229: process: 0.50290 | load: 0.00244 [I 2022-10-26 12:14:42] ...trap/engines/engine.py.230: loss: 0.27829 [I 2022-10-26 12:14:42] ...optimizers/trijoint.py.151: total_norm: 4.140086 [I 2022-10-26 12:14:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 620/2304 [I 2022-10-26 12:14:47] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:13:58 [I 2022-10-26 12:14:47] ...trap/engines/engine.py.229: process: 0.49495 | load: 0.00243 [I 2022-10-26 12:14:47] ...trap/engines/engine.py.230: loss: 0.30427 [I 2022-10-26 12:14:47] ...optimizers/trijoint.py.151: total_norm: 2.982697 [I 2022-10-26 12:14:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 630/2304 [I 2022-10-26 12:14:52] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:13:55 [I 2022-10-26 12:14:52] ...trap/engines/engine.py.229: process: 0.50612 | load: 0.00243 [I 2022-10-26 12:14:52] ...trap/engines/engine.py.230: loss: 0.31215 [I 2022-10-26 12:14:52] ...optimizers/trijoint.py.151: total_norm: 3.170970 [I 2022-10-26 12:14:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 640/2304 [I 2022-10-26 12:14:57] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:13:48 [I 2022-10-26 12:14:57] ...trap/engines/engine.py.229: process: 0.48562 | load: 0.00241 [I 2022-10-26 12:14:57] ...trap/engines/engine.py.230: loss: 0.23976 [I 2022-10-26 12:14:57] ...optimizers/trijoint.py.151: total_norm: 3.554823 [I 2022-10-26 12:15:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 650/2304 [I 2022-10-26 12:15:02] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:13:43 [I 2022-10-26 12:15:02] ...trap/engines/engine.py.229: process: 0.51673 | load: 0.00361 [I 2022-10-26 12:15:02] ...trap/engines/engine.py.230: loss: 0.15462 [I 2022-10-26 12:15:02] ...optimizers/trijoint.py.151: total_norm: 2.903199 [I 2022-10-26 12:15:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 660/2304 [I 2022-10-26 12:15:07] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:13:38 [I 2022-10-26 12:15:07] ...trap/engines/engine.py.229: process: 0.48674 | load: 0.00243 [I 2022-10-26 12:15:07] ...trap/engines/engine.py.230: loss: 0.24012 [I 2022-10-26 12:15:07] ...optimizers/trijoint.py.151: total_norm: 3.062858 [I 2022-10-26 12:15:12] ...trap/engines/engine.py.224: train: epoch 7 | batch 670/2304 [I 2022-10-26 12:15:12] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:13:37 [I 2022-10-26 12:15:12] ...trap/engines/engine.py.229: process: 0.48404 | load: 0.00246 [I 2022-10-26 12:15:12] ...trap/engines/engine.py.230: loss: 0.24634 [I 2022-10-26 12:15:12] ...optimizers/trijoint.py.151: total_norm: 4.338156 [I 2022-10-26 12:15:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 680/2304 [I 2022-10-26 12:15:17] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:13:30 [I 2022-10-26 12:15:17] ...trap/engines/engine.py.229: process: 0.49878 | load: 0.00241 [I 2022-10-26 12:15:17] ...trap/engines/engine.py.230: loss: 0.26549 [I 2022-10-26 12:15:17] ...optimizers/trijoint.py.151: total_norm: 4.151420 [I 2022-10-26 12:15:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 690/2304 [I 2022-10-26 12:15:22] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:13:44 [I 2022-10-26 12:15:22] ...trap/engines/engine.py.229: process: 0.49581 | load: 0.00242 [I 2022-10-26 12:15:22] ...trap/engines/engine.py.230: loss: 0.28523 [I 2022-10-26 12:15:22] ...optimizers/trijoint.py.151: total_norm: 3.836306 [I 2022-10-26 12:15:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 700/2304 [I 2022-10-26 12:15:27] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:13:25 [I 2022-10-26 12:15:27] ...trap/engines/engine.py.229: process: 0.49257 | load: 0.00286 [I 2022-10-26 12:15:27] ...trap/engines/engine.py.230: loss: 0.24342 [I 2022-10-26 12:15:27] ...optimizers/trijoint.py.151: total_norm: 3.843972 [I 2022-10-26 12:15:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 710/2304 [I 2022-10-26 12:15:32] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:13:25 [I 2022-10-26 12:15:32] ...trap/engines/engine.py.229: process: 0.51407 | load: 0.00247 [I 2022-10-26 12:15:32] ...trap/engines/engine.py.230: loss: 0.17665 [I 2022-10-26 12:15:32] ...optimizers/trijoint.py.151: total_norm: 2.684619 [I 2022-10-26 12:15:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 720/2304 [I 2022-10-26 12:15:37] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:13:03 [I 2022-10-26 12:15:37] ...trap/engines/engine.py.229: process: 0.47933 | load: 0.00245 [I 2022-10-26 12:15:37] ...trap/engines/engine.py.230: loss: 0.27765 [I 2022-10-26 12:15:37] ...optimizers/trijoint.py.151: total_norm: 3.562454 [I 2022-10-26 12:15:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 730/2304 [I 2022-10-26 12:15:42] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:13:02 [I 2022-10-26 12:15:42] ...trap/engines/engine.py.229: process: 0.48881 | load: 0.00244 [I 2022-10-26 12:15:42] ...trap/engines/engine.py.230: loss: 0.31982 [I 2022-10-26 12:15:42] ...optimizers/trijoint.py.151: total_norm: 3.449332 [I 2022-10-26 12:15:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 740/2304 [I 2022-10-26 12:15:47] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:12:55 [I 2022-10-26 12:15:47] ...trap/engines/engine.py.229: process: 0.49587 | load: 0.00246 [I 2022-10-26 12:15:47] ...trap/engines/engine.py.230: loss: 0.21707 [I 2022-10-26 12:15:47] ...optimizers/trijoint.py.151: total_norm: 3.047309 [I 2022-10-26 12:15:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 750/2304 [I 2022-10-26 12:15:52] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:12:44 [I 2022-10-26 12:15:52] ...trap/engines/engine.py.229: process: 0.49206 | load: 0.00246 [I 2022-10-26 12:15:52] ...trap/engines/engine.py.230: loss: 0.32422 [I 2022-10-26 12:15:52] ...optimizers/trijoint.py.151: total_norm: 3.990373 [I 2022-10-26 12:15:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 760/2304 [I 2022-10-26 12:15:57] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:13:15 [I 2022-10-26 12:15:57] ...trap/engines/engine.py.229: process: 0.59279 | load: 0.00242 [I 2022-10-26 12:15:57] ...trap/engines/engine.py.230: loss: 0.23256 [I 2022-10-26 12:15:57] ...optimizers/trijoint.py.151: total_norm: 3.273910 [I 2022-10-26 12:16:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 770/2304 [I 2022-10-26 12:16:02] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:12:51 [I 2022-10-26 12:16:02] ...trap/engines/engine.py.229: process: 0.49726 | load: 0.00246 [I 2022-10-26 12:16:02] ...trap/engines/engine.py.230: loss: 0.33000 [I 2022-10-26 12:16:02] ...optimizers/trijoint.py.151: total_norm: 3.891091 [I 2022-10-26 12:16:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 780/2304 [I 2022-10-26 12:16:07] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:12:47 [I 2022-10-26 12:16:07] ...trap/engines/engine.py.229: process: 0.51434 | load: 0.00249 [I 2022-10-26 12:16:07] ...trap/engines/engine.py.230: loss: 0.33265 [I 2022-10-26 12:16:07] ...optimizers/trijoint.py.151: total_norm: 4.143903 [I 2022-10-26 12:16:12] ...trap/engines/engine.py.224: train: epoch 7 | batch 790/2304 [I 2022-10-26 12:16:12] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:12:38 [I 2022-10-26 12:16:12] ...trap/engines/engine.py.229: process: 0.50400 | load: 0.00246 [I 2022-10-26 12:16:12] ...trap/engines/engine.py.230: loss: 0.29734 [I 2022-10-26 12:16:12] ...optimizers/trijoint.py.151: total_norm: 3.849170 [I 2022-10-26 12:16:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 800/2304 [I 2022-10-26 12:16:17] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:12:33 [I 2022-10-26 12:16:17] ...trap/engines/engine.py.229: process: 0.49622 | load: 0.00243 [I 2022-10-26 12:16:17] ...trap/engines/engine.py.230: loss: 0.24391 [I 2022-10-26 12:16:17] ...optimizers/trijoint.py.151: total_norm: 4.326157 [I 2022-10-26 12:16:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 810/2304 [I 2022-10-26 12:16:22] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:12:29 [I 2022-10-26 12:16:22] ...trap/engines/engine.py.229: process: 0.49730 | load: 0.00264 [I 2022-10-26 12:16:22] ...trap/engines/engine.py.230: loss: 0.27620 [I 2022-10-26 12:16:22] ...optimizers/trijoint.py.151: total_norm: 3.511825 [I 2022-10-26 12:16:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 820/2304 [I 2022-10-26 12:16:27] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:12:12 [I 2022-10-26 12:16:27] ...trap/engines/engine.py.229: process: 0.49053 | load: 0.00249 [I 2022-10-26 12:16:27] ...trap/engines/engine.py.230: loss: 0.29509 [I 2022-10-26 12:16:27] ...optimizers/trijoint.py.151: total_norm: 3.169281 [I 2022-10-26 12:16:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 830/2304 [I 2022-10-26 12:16:32] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:12:13 [I 2022-10-26 12:16:32] ...trap/engines/engine.py.229: process: 0.50577 | load: 0.00247 [I 2022-10-26 12:16:32] ...trap/engines/engine.py.230: loss: 0.33436 [I 2022-10-26 12:16:32] ...optimizers/trijoint.py.151: total_norm: 5.108241 [I 2022-10-26 12:16:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 840/2304 [I 2022-10-26 12:16:37] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:12:16 [I 2022-10-26 12:16:37] ...trap/engines/engine.py.229: process: 0.49907 | load: 0.00245 [I 2022-10-26 12:16:37] ...trap/engines/engine.py.230: loss: 0.33837 [I 2022-10-26 12:16:37] ...optimizers/trijoint.py.151: total_norm: 3.935182 [I 2022-10-26 12:16:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 850/2304 [I 2022-10-26 12:16:42] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:12:01 [I 2022-10-26 12:16:42] ...trap/engines/engine.py.229: process: 0.50105 | load: 0.00243 [I 2022-10-26 12:16:42] ...trap/engines/engine.py.230: loss: 0.16968 [I 2022-10-26 12:16:42] ...optimizers/trijoint.py.151: total_norm: 3.277562 [I 2022-10-26 12:16:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 860/2304 [I 2022-10-26 12:16:47] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:11:54 [I 2022-10-26 12:16:47] ...trap/engines/engine.py.229: process: 0.49023 | load: 0.00243 [I 2022-10-26 12:16:47] ...trap/engines/engine.py.230: loss: 0.21447 [I 2022-10-26 12:16:47] ...optimizers/trijoint.py.151: total_norm: 2.752640 [I 2022-10-26 12:16:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 870/2304 [I 2022-10-26 12:16:52] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:11:55 [I 2022-10-26 12:16:52] ...trap/engines/engine.py.229: process: 0.50740 | load: 0.00249 [I 2022-10-26 12:16:52] ...trap/engines/engine.py.230: loss: 0.31872 [I 2022-10-26 12:16:52] ...optimizers/trijoint.py.151: total_norm: 4.135787 [I 2022-10-26 12:16:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 880/2304 [I 2022-10-26 12:16:57] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:11:48 [I 2022-10-26 12:16:57] ...trap/engines/engine.py.229: process: 0.49377 | load: 0.00244 [I 2022-10-26 12:16:57] ...trap/engines/engine.py.230: loss: 0.21363 [I 2022-10-26 12:16:57] ...optimizers/trijoint.py.151: total_norm: 3.640152 [I 2022-10-26 12:17:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 890/2304 [I 2022-10-26 12:17:02] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:11:42 [I 2022-10-26 12:17:02] ...trap/engines/engine.py.229: process: 0.49459 | load: 0.00246 [I 2022-10-26 12:17:02] ...trap/engines/engine.py.230: loss: 0.29006 [I 2022-10-26 12:17:02] ...optimizers/trijoint.py.151: total_norm: 5.175260 [I 2022-10-26 12:17:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 900/2304 [I 2022-10-26 12:17:07] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:11:38 [I 2022-10-26 12:17:07] ...trap/engines/engine.py.229: process: 0.50744 | load: 0.00246 [I 2022-10-26 12:17:07] ...trap/engines/engine.py.230: loss: 0.33705 [I 2022-10-26 12:17:07] ...optimizers/trijoint.py.151: total_norm: 3.276283 [I 2022-10-26 12:17:12] ...trap/engines/engine.py.224: train: epoch 7 | batch 910/2304 [I 2022-10-26 12:17:12] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:11:55 [I 2022-10-26 12:17:12] ...trap/engines/engine.py.229: process: 0.50290 | load: 0.00242 [I 2022-10-26 12:17:12] ...trap/engines/engine.py.230: loss: 0.30141 [I 2022-10-26 12:17:12] ...optimizers/trijoint.py.151: total_norm: 3.866482 [I 2022-10-26 12:17:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 920/2304 [I 2022-10-26 12:17:17] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:11:37 [I 2022-10-26 12:17:17] ...trap/engines/engine.py.229: process: 0.51729 | load: 0.00245 [I 2022-10-26 12:17:17] ...trap/engines/engine.py.230: loss: 0.27698 [I 2022-10-26 12:17:17] ...optimizers/trijoint.py.151: total_norm: 3.761344 [I 2022-10-26 12:17:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 930/2304 [I 2022-10-26 12:17:22] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:11:28 [I 2022-10-26 12:17:22] ...trap/engines/engine.py.229: process: 0.50295 | load: 0.00263 [I 2022-10-26 12:17:22] ...trap/engines/engine.py.230: loss: 0.19767 [I 2022-10-26 12:17:22] ...optimizers/trijoint.py.151: total_norm: 3.350959 [I 2022-10-26 12:17:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 940/2304 [I 2022-10-26 12:17:27] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:11:10 [I 2022-10-26 12:17:27] ...trap/engines/engine.py.229: process: 0.48073 | load: 0.00248 [I 2022-10-26 12:17:27] ...trap/engines/engine.py.230: loss: 0.25334 [I 2022-10-26 12:17:27] ...optimizers/trijoint.py.151: total_norm: 3.778584 [I 2022-10-26 12:17:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 950/2304 [I 2022-10-26 12:17:32] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:11:12 [I 2022-10-26 12:17:32] ...trap/engines/engine.py.229: process: 0.50349 | load: 0.00247 [I 2022-10-26 12:17:32] ...trap/engines/engine.py.230: loss: 0.32015 [I 2022-10-26 12:17:32] ...optimizers/trijoint.py.151: total_norm: 2.830006 [I 2022-10-26 12:17:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 960/2304 [I 2022-10-26 12:17:37] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:11:07 [I 2022-10-26 12:17:37] ...trap/engines/engine.py.229: process: 0.49535 | load: 0.00246 [I 2022-10-26 12:17:37] ...trap/engines/engine.py.230: loss: 0.25091 [I 2022-10-26 12:17:37] ...optimizers/trijoint.py.151: total_norm: 3.597044 [I 2022-10-26 12:17:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 970/2304 [I 2022-10-26 12:17:42] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:11:05 [I 2022-10-26 12:17:42] ...trap/engines/engine.py.229: process: 0.49361 | load: 0.00263 [I 2022-10-26 12:17:42] ...trap/engines/engine.py.230: loss: 0.19984 [I 2022-10-26 12:17:42] ...optimizers/trijoint.py.151: total_norm: 4.231494 [I 2022-10-26 12:17:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 980/2304 [I 2022-10-26 12:17:47] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:11:01 [I 2022-10-26 12:17:47] ...trap/engines/engine.py.229: process: 0.50081 | load: 0.00244 [I 2022-10-26 12:17:47] ...trap/engines/engine.py.230: loss: 0.28722 [I 2022-10-26 12:17:47] ...optimizers/trijoint.py.151: total_norm: 4.263769 [I 2022-10-26 12:17:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 990/2304 [I 2022-10-26 12:17:52] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:10:57 [I 2022-10-26 12:17:52] ...trap/engines/engine.py.229: process: 0.50310 | load: 0.00250 [I 2022-10-26 12:17:52] ...trap/engines/engine.py.230: loss: 0.21097 [I 2022-10-26 12:17:52] ...optimizers/trijoint.py.151: total_norm: 3.302139 [I 2022-10-26 12:17:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 1000/2304 [I 2022-10-26 12:17:57] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:10:47 [I 2022-10-26 12:17:57] ...trap/engines/engine.py.229: process: 0.50161 | load: 0.00243 [I 2022-10-26 12:17:57] ...trap/engines/engine.py.230: loss: 0.19534 [I 2022-10-26 12:17:57] ...optimizers/trijoint.py.151: total_norm: 2.731064 [I 2022-10-26 12:18:02] ...trap/engines/engine.py.224: train: epoch 7 | batch 1010/2304 [I 2022-10-26 12:18:02] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:10:48 [I 2022-10-26 12:18:02] ...trap/engines/engine.py.229: process: 0.50944 | load: 0.00246 [I 2022-10-26 12:18:02] ...trap/engines/engine.py.230: loss: 0.35833 [I 2022-10-26 12:18:02] ...optimizers/trijoint.py.151: total_norm: 3.810225 [I 2022-10-26 12:18:07] ...trap/engines/engine.py.224: train: epoch 7 | batch 1020/2304 [I 2022-10-26 12:18:07] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:10:45 [I 2022-10-26 12:18:07] ...trap/engines/engine.py.229: process: 0.50111 | load: 0.00256 [I 2022-10-26 12:18:07] ...trap/engines/engine.py.230: loss: 0.31752 [I 2022-10-26 12:18:07] ...optimizers/trijoint.py.151: total_norm: 3.635948 [I 2022-10-26 12:18:12] ...trap/engines/engine.py.224: train: epoch 7 | batch 1030/2304 [I 2022-10-26 12:18:12] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:10:32 [I 2022-10-26 12:18:12] ...trap/engines/engine.py.229: process: 0.48133 | load: 0.00258 [I 2022-10-26 12:18:12] ...trap/engines/engine.py.230: loss: 0.32244 [I 2022-10-26 12:18:12] ...optimizers/trijoint.py.151: total_norm: 3.687100 [I 2022-10-26 12:18:17] ...trap/engines/engine.py.224: train: epoch 7 | batch 1040/2304 [I 2022-10-26 12:18:17] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:10:32 [I 2022-10-26 12:18:17] ...trap/engines/engine.py.229: process: 0.50569 | load: 0.00252 [I 2022-10-26 12:18:17] ...trap/engines/engine.py.230: loss: 0.22401 [I 2022-10-26 12:18:17] ...optimizers/trijoint.py.151: total_norm: 2.581503 [I 2022-10-26 12:18:22] ...trap/engines/engine.py.224: train: epoch 7 | batch 1050/2304 [I 2022-10-26 12:18:22] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:10:22 [I 2022-10-26 12:18:22] ...trap/engines/engine.py.229: process: 0.50133 | load: 0.00263 [I 2022-10-26 12:18:22] ...trap/engines/engine.py.230: loss: 0.27400 [I 2022-10-26 12:18:22] ...optimizers/trijoint.py.151: total_norm: 3.025803 [I 2022-10-26 12:18:27] ...trap/engines/engine.py.224: train: epoch 7 | batch 1060/2304 [I 2022-10-26 12:18:27] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:10:20 [I 2022-10-26 12:18:27] ...trap/engines/engine.py.229: process: 0.50236 | load: 0.00262 [I 2022-10-26 12:18:27] ...trap/engines/engine.py.230: loss: 0.24982 [I 2022-10-26 12:18:27] ...optimizers/trijoint.py.151: total_norm: 3.538731 [I 2022-10-26 12:18:32] ...trap/engines/engine.py.224: train: epoch 7 | batch 1070/2304 [I 2022-10-26 12:18:32] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:10:14 [I 2022-10-26 12:18:32] ...trap/engines/engine.py.229: process: 0.49295 | load: 0.00275 [I 2022-10-26 12:18:32] ...trap/engines/engine.py.230: loss: 0.16373 [I 2022-10-26 12:18:32] ...optimizers/trijoint.py.151: total_norm: 2.451412 [I 2022-10-26 12:18:37] ...trap/engines/engine.py.224: train: epoch 7 | batch 1080/2304 [I 2022-10-26 12:18:37] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:10:13 [I 2022-10-26 12:18:37] ...trap/engines/engine.py.229: process: 0.50080 | load: 0.00261 [I 2022-10-26 12:18:37] ...trap/engines/engine.py.230: loss: 0.30160 [I 2022-10-26 12:18:37] ...optimizers/trijoint.py.151: total_norm: 3.533737 [I 2022-10-26 12:18:42] ...trap/engines/engine.py.224: train: epoch 7 | batch 1090/2304 [I 2022-10-26 12:18:42] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:10:05 [I 2022-10-26 12:18:42] ...trap/engines/engine.py.229: process: 0.50283 | load: 0.00256 [I 2022-10-26 12:18:42] ...trap/engines/engine.py.230: loss: 0.23595 [I 2022-10-26 12:18:42] ...optimizers/trijoint.py.151: total_norm: 3.622083 [I 2022-10-26 12:18:47] ...trap/engines/engine.py.224: train: epoch 7 | batch 1100/2304 [I 2022-10-26 12:18:47] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:10:09 [I 2022-10-26 12:18:47] ...trap/engines/engine.py.229: process: 0.50782 | load: 0.00260 [I 2022-10-26 12:18:47] ...trap/engines/engine.py.230: loss: 0.20684 [I 2022-10-26 12:18:47] ...optimizers/trijoint.py.151: total_norm: 2.762197 [I 2022-10-26 12:18:52] ...trap/engines/engine.py.224: train: epoch 7 | batch 1110/2304 [I 2022-10-26 12:18:52] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:09:53 [I 2022-10-26 12:18:52] ...trap/engines/engine.py.229: process: 0.49791 | load: 0.00578 [I 2022-10-26 12:18:52] ...trap/engines/engine.py.230: loss: 0.28917 [I 2022-10-26 12:18:52] ...optimizers/trijoint.py.151: total_norm: 4.191380 [I 2022-10-26 12:18:57] ...trap/engines/engine.py.224: train: epoch 7 | batch 1120/2304 [I 2022-10-26 12:18:57] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:09:47 [I 2022-10-26 12:18:57] ...trap/engines/engine.py.229: process: 0.49013 | load: 0.00263 [I 2022-10-26 12:18:57] ...trap/engines/engine.py.230: loss: 0.23599 [I 2022-10-26 12:18:57] ...optimizers/trijoint.py.151: total_norm: 3.470483 [I 2022-10-26 12:19:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 1130/2304 [I 2022-10-26 12:19:03] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:09:52 [I 2022-10-26 12:19:03] ...trap/engines/engine.py.229: process: 0.50474 | load: 0.00349 [I 2022-10-26 12:19:03] ...trap/engines/engine.py.230: loss: 0.38774 [I 2022-10-26 12:19:03] ...optimizers/trijoint.py.151: total_norm: 3.975601 [I 2022-10-26 12:19:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 1140/2304 [I 2022-10-26 12:19:08] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:09:45 [I 2022-10-26 12:19:08] ...trap/engines/engine.py.229: process: 0.50190 | load: 0.00289 [I 2022-10-26 12:19:08] ...trap/engines/engine.py.230: loss: 0.19442 [I 2022-10-26 12:19:08] ...optimizers/trijoint.py.151: total_norm: 2.409451 [I 2022-10-26 12:19:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1150/2304 [I 2022-10-26 12:19:13] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:09:39 [I 2022-10-26 12:19:13] ...trap/engines/engine.py.229: process: 0.50888 | load: 0.00268 [I 2022-10-26 12:19:13] ...trap/engines/engine.py.230: loss: 0.21414 [I 2022-10-26 12:19:13] ...optimizers/trijoint.py.151: total_norm: 3.154883 [I 2022-10-26 12:19:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 1160/2304 [I 2022-10-26 12:19:18] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:09:33 [I 2022-10-26 12:19:18] ...trap/engines/engine.py.229: process: 0.50191 | load: 0.00272 [I 2022-10-26 12:19:18] ...trap/engines/engine.py.230: loss: 0.29847 [I 2022-10-26 12:19:18] ...optimizers/trijoint.py.151: total_norm: 3.324598 [I 2022-10-26 12:19:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1170/2304 [I 2022-10-26 12:19:23] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:09:30 [I 2022-10-26 12:19:23] ...trap/engines/engine.py.229: process: 0.51875 | load: 0.00262 [I 2022-10-26 12:19:23] ...trap/engines/engine.py.230: loss: 0.25022 [I 2022-10-26 12:19:23] ...optimizers/trijoint.py.151: total_norm: 3.746983 [I 2022-10-26 12:19:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1180/2304 [I 2022-10-26 12:19:28] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:09:19 [I 2022-10-26 12:19:28] ...trap/engines/engine.py.229: process: 0.50124 | load: 0.00253 [I 2022-10-26 12:19:28] ...trap/engines/engine.py.230: loss: 0.20910 [I 2022-10-26 12:19:28] ...optimizers/trijoint.py.151: total_norm: 3.296405 [I 2022-10-26 12:19:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 1190/2304 [I 2022-10-26 12:19:33] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:09:19 [I 2022-10-26 12:19:33] ...trap/engines/engine.py.229: process: 0.50733 | load: 0.00309 [I 2022-10-26 12:19:33] ...trap/engines/engine.py.230: loss: 0.38236 [I 2022-10-26 12:19:33] ...optimizers/trijoint.py.151: total_norm: 5.038955 [I 2022-10-26 12:19:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 1200/2304 [I 2022-10-26 12:19:38] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:09:17 [I 2022-10-26 12:19:38] ...trap/engines/engine.py.229: process: 0.50149 | load: 0.00281 [I 2022-10-26 12:19:38] ...trap/engines/engine.py.230: loss: 0.30164 [I 2022-10-26 12:19:38] ...optimizers/trijoint.py.151: total_norm: 3.648247 [I 2022-10-26 12:19:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 1210/2304 [I 2022-10-26 12:19:43] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:09:07 [I 2022-10-26 12:19:43] ...trap/engines/engine.py.229: process: 0.48858 | load: 0.00259 [I 2022-10-26 12:19:43] ...trap/engines/engine.py.230: loss: 0.28315 [I 2022-10-26 12:19:43] ...optimizers/trijoint.py.151: total_norm: 4.149284 [I 2022-10-26 12:19:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 1220/2304 [I 2022-10-26 12:19:48] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:09:00 [I 2022-10-26 12:19:48] ...trap/engines/engine.py.229: process: 0.50551 | load: 0.00248 [I 2022-10-26 12:19:48] ...trap/engines/engine.py.230: loss: 0.33519 [I 2022-10-26 12:19:48] ...optimizers/trijoint.py.151: total_norm: 4.254197 [I 2022-10-26 12:19:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 1230/2304 [I 2022-10-26 12:19:53] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:08:54 [I 2022-10-26 12:19:53] ...trap/engines/engine.py.229: process: 0.49375 | load: 0.00254 [I 2022-10-26 12:19:53] ...trap/engines/engine.py.230: loss: 0.33841 [I 2022-10-26 12:19:53] ...optimizers/trijoint.py.151: total_norm: 4.341182 [I 2022-10-26 12:19:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 1240/2304 [I 2022-10-26 12:19:58] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:08:50 [I 2022-10-26 12:19:58] ...trap/engines/engine.py.229: process: 0.49669 | load: 0.00259 [I 2022-10-26 12:19:58] ...trap/engines/engine.py.230: loss: 0.25563 [I 2022-10-26 12:19:58] ...optimizers/trijoint.py.151: total_norm: 2.926406 [I 2022-10-26 12:20:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 1250/2304 [I 2022-10-26 12:20:03] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:08:41 [I 2022-10-26 12:20:03] ...trap/engines/engine.py.229: process: 0.49948 | load: 0.00255 [I 2022-10-26 12:20:03] ...trap/engines/engine.py.230: loss: 0.20016 [I 2022-10-26 12:20:03] ...optimizers/trijoint.py.151: total_norm: 3.177737 [I 2022-10-26 12:20:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 1260/2304 [I 2022-10-26 12:20:08] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:08:36 [I 2022-10-26 12:20:08] ...trap/engines/engine.py.229: process: 0.49490 | load: 0.00250 [I 2022-10-26 12:20:08] ...trap/engines/engine.py.230: loss: 0.26531 [I 2022-10-26 12:20:08] ...optimizers/trijoint.py.151: total_norm: 3.751558 [I 2022-10-26 12:20:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1270/2304 [I 2022-10-26 12:20:13] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:08:58 [I 2022-10-26 12:20:13] ...trap/engines/engine.py.229: process: 0.56547 | load: 0.00271 [I 2022-10-26 12:20:13] ...trap/engines/engine.py.230: loss: 0.35869 [I 2022-10-26 12:20:13] ...optimizers/trijoint.py.151: total_norm: 4.441578 [I 2022-10-26 12:20:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 1280/2304 [I 2022-10-26 12:20:18] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:08:38 [I 2022-10-26 12:20:18] ...trap/engines/engine.py.229: process: 0.50807 | load: 0.00247 [I 2022-10-26 12:20:18] ...trap/engines/engine.py.230: loss: 0.25148 [I 2022-10-26 12:20:18] ...optimizers/trijoint.py.151: total_norm: 3.285463 [I 2022-10-26 12:20:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1290/2304 [I 2022-10-26 12:20:23] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:08:29 [I 2022-10-26 12:20:23] ...trap/engines/engine.py.229: process: 0.49744 | load: 0.00248 [I 2022-10-26 12:20:23] ...trap/engines/engine.py.230: loss: 0.26212 [I 2022-10-26 12:20:23] ...optimizers/trijoint.py.151: total_norm: 3.726750 [I 2022-10-26 12:20:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1300/2304 [I 2022-10-26 12:20:28] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:08:18 [I 2022-10-26 12:20:28] ...trap/engines/engine.py.229: process: 0.48699 | load: 0.00246 [I 2022-10-26 12:20:28] ...trap/engines/engine.py.230: loss: 0.30214 [I 2022-10-26 12:20:28] ...optimizers/trijoint.py.151: total_norm: 3.705213 [I 2022-10-26 12:20:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 1310/2304 [I 2022-10-26 12:20:33] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:08:16 [I 2022-10-26 12:20:33] ...trap/engines/engine.py.229: process: 0.50072 | load: 0.00245 [I 2022-10-26 12:20:33] ...trap/engines/engine.py.230: loss: 0.22639 [I 2022-10-26 12:20:33] ...optimizers/trijoint.py.151: total_norm: 3.543152 [I 2022-10-26 12:20:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 1320/2304 [I 2022-10-26 12:20:38] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:08:08 [I 2022-10-26 12:20:38] ...trap/engines/engine.py.229: process: 0.49474 | load: 0.00246 [I 2022-10-26 12:20:38] ...trap/engines/engine.py.230: loss: 0.30621 [I 2022-10-26 12:20:38] ...optimizers/trijoint.py.151: total_norm: 3.537688 [I 2022-10-26 12:20:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 1330/2304 [I 2022-10-26 12:20:43] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:08:10 [I 2022-10-26 12:20:43] ...trap/engines/engine.py.229: process: 0.49675 | load: 0.00244 [I 2022-10-26 12:20:43] ...trap/engines/engine.py.230: loss: 0.25743 [I 2022-10-26 12:20:43] ...optimizers/trijoint.py.151: total_norm: 3.875162 [I 2022-10-26 12:20:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 1340/2304 [I 2022-10-26 12:20:48] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:08:03 [I 2022-10-26 12:20:48] ...trap/engines/engine.py.229: process: 0.50102 | load: 0.00249 [I 2022-10-26 12:20:48] ...trap/engines/engine.py.230: loss: 0.17079 [I 2022-10-26 12:20:48] ...optimizers/trijoint.py.151: total_norm: 2.808680 [I 2022-10-26 12:20:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 1350/2304 [I 2022-10-26 12:20:53] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:07:55 [I 2022-10-26 12:20:53] ...trap/engines/engine.py.229: process: 0.50497 | load: 0.00245 [I 2022-10-26 12:20:53] ...trap/engines/engine.py.230: loss: 0.21105 [I 2022-10-26 12:20:53] ...optimizers/trijoint.py.151: total_norm: 3.439022 [I 2022-10-26 12:20:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 1360/2304 [I 2022-10-26 12:20:58] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:07:50 [I 2022-10-26 12:20:58] ...trap/engines/engine.py.229: process: 0.49991 | load: 0.00245 [I 2022-10-26 12:20:58] ...trap/engines/engine.py.230: loss: 0.24992 [I 2022-10-26 12:20:58] ...optimizers/trijoint.py.151: total_norm: 3.459980 [I 2022-10-26 12:21:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 1370/2304 [I 2022-10-26 12:21:03] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:07:48 [I 2022-10-26 12:21:03] ...trap/engines/engine.py.229: process: 0.49836 | load: 0.00241 [I 2022-10-26 12:21:03] ...trap/engines/engine.py.230: loss: 0.25236 [I 2022-10-26 12:21:03] ...optimizers/trijoint.py.151: total_norm: 2.760850 [I 2022-10-26 12:21:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 1380/2304 [I 2022-10-26 12:21:08] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:07:39 [I 2022-10-26 12:21:08] ...trap/engines/engine.py.229: process: 0.50127 | load: 0.00246 [I 2022-10-26 12:21:08] ...trap/engines/engine.py.230: loss: 0.29277 [I 2022-10-26 12:21:08] ...optimizers/trijoint.py.151: total_norm: 4.378198 [I 2022-10-26 12:21:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1390/2304 [I 2022-10-26 12:21:13] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:07:39 [I 2022-10-26 12:21:13] ...trap/engines/engine.py.229: process: 0.50757 | load: 0.00255 [I 2022-10-26 12:21:13] ...trap/engines/engine.py.230: loss: 0.21930 [I 2022-10-26 12:21:13] ...optimizers/trijoint.py.151: total_norm: 3.707766 [I 2022-10-26 12:21:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 1400/2304 [I 2022-10-26 12:21:18] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:07:30 [I 2022-10-26 12:21:18] ...trap/engines/engine.py.229: process: 0.50382 | load: 0.00244 [I 2022-10-26 12:21:18] ...trap/engines/engine.py.230: loss: 0.30372 [I 2022-10-26 12:21:18] ...optimizers/trijoint.py.151: total_norm: 3.926374 [I 2022-10-26 12:21:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1410/2304 [I 2022-10-26 12:21:23] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:07:23 [I 2022-10-26 12:21:23] ...trap/engines/engine.py.229: process: 0.48939 | load: 0.00251 [I 2022-10-26 12:21:23] ...trap/engines/engine.py.230: loss: 0.27049 [I 2022-10-26 12:21:23] ...optimizers/trijoint.py.151: total_norm: 3.229999 [I 2022-10-26 12:21:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1420/2304 [I 2022-10-26 12:21:28] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:07:18 [I 2022-10-26 12:21:28] ...trap/engines/engine.py.229: process: 0.50414 | load: 0.00244 [I 2022-10-26 12:21:28] ...trap/engines/engine.py.230: loss: 0.23485 [I 2022-10-26 12:21:28] ...optimizers/trijoint.py.151: total_norm: 3.625882 [I 2022-10-26 12:21:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 1430/2304 [I 2022-10-26 12:21:33] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:07:11 [I 2022-10-26 12:21:33] ...trap/engines/engine.py.229: process: 0.49817 | load: 0.00264 [I 2022-10-26 12:21:33] ...trap/engines/engine.py.230: loss: 0.19399 [I 2022-10-26 12:21:33] ...optimizers/trijoint.py.151: total_norm: 2.565153 [I 2022-10-26 12:21:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 1440/2304 [I 2022-10-26 12:21:38] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:07:09 [I 2022-10-26 12:21:38] ...trap/engines/engine.py.229: process: 0.50132 | load: 0.00246 [I 2022-10-26 12:21:38] ...trap/engines/engine.py.230: loss: 0.35003 [I 2022-10-26 12:21:38] ...optimizers/trijoint.py.151: total_norm: 3.675645 [I 2022-10-26 12:21:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 1450/2304 [I 2022-10-26 12:21:43] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:07:08 [I 2022-10-26 12:21:43] ...trap/engines/engine.py.229: process: 0.50274 | load: 0.00243 [I 2022-10-26 12:21:43] ...trap/engines/engine.py.230: loss: 0.25074 [I 2022-10-26 12:21:43] ...optimizers/trijoint.py.151: total_norm: 4.527509 [I 2022-10-26 12:21:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 1460/2304 [I 2022-10-26 12:21:48] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:06:58 [I 2022-10-26 12:21:48] ...trap/engines/engine.py.229: process: 0.50702 | load: 0.00244 [I 2022-10-26 12:21:48] ...trap/engines/engine.py.230: loss: 0.29853 [I 2022-10-26 12:21:48] ...optimizers/trijoint.py.151: total_norm: 3.777978 [I 2022-10-26 12:21:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 1470/2304 [I 2022-10-26 12:21:53] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:06:53 [I 2022-10-26 12:21:53] ...trap/engines/engine.py.229: process: 0.49091 | load: 0.00245 [I 2022-10-26 12:21:53] ...trap/engines/engine.py.230: loss: 0.26221 [I 2022-10-26 12:21:53] ...optimizers/trijoint.py.151: total_norm: 2.548331 [I 2022-10-26 12:21:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 1480/2304 [I 2022-10-26 12:21:58] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:06:49 [I 2022-10-26 12:21:58] ...trap/engines/engine.py.229: process: 0.49178 | load: 0.00251 [I 2022-10-26 12:21:58] ...trap/engines/engine.py.230: loss: 0.35549 [I 2022-10-26 12:21:58] ...optimizers/trijoint.py.151: total_norm: 4.041587 [I 2022-10-26 12:22:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 1490/2304 [I 2022-10-26 12:22:03] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:06:45 [I 2022-10-26 12:22:03] ...trap/engines/engine.py.229: process: 0.49807 | load: 0.00248 [I 2022-10-26 12:22:03] ...trap/engines/engine.py.230: loss: 0.26668 [I 2022-10-26 12:22:03] ...optimizers/trijoint.py.151: total_norm: 3.328261 [I 2022-10-26 12:22:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 1500/2304 [I 2022-10-26 12:22:08] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:06:42 [I 2022-10-26 12:22:08] ...trap/engines/engine.py.229: process: 0.50667 | load: 0.00248 [I 2022-10-26 12:22:08] ...trap/engines/engine.py.230: loss: 0.21074 [I 2022-10-26 12:22:08] ...optimizers/trijoint.py.151: total_norm: 2.844255 [I 2022-10-26 12:22:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1510/2304 [I 2022-10-26 12:22:13] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:06:37 [I 2022-10-26 12:22:13] ...trap/engines/engine.py.229: process: 0.50262 | load: 0.00262 [I 2022-10-26 12:22:13] ...trap/engines/engine.py.230: loss: 0.23026 [I 2022-10-26 12:22:13] ...optimizers/trijoint.py.151: total_norm: 3.024975 [I 2022-10-26 12:22:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 1520/2304 [I 2022-10-26 12:22:18] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:06:30 [I 2022-10-26 12:22:18] ...trap/engines/engine.py.229: process: 0.49876 | load: 0.00251 [I 2022-10-26 12:22:18] ...trap/engines/engine.py.230: loss: 0.16374 [I 2022-10-26 12:22:18] ...optimizers/trijoint.py.151: total_norm: 2.751289 [I 2022-10-26 12:22:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1530/2304 [I 2022-10-26 12:22:23] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:06:24 [I 2022-10-26 12:22:23] ...trap/engines/engine.py.229: process: 0.50081 | load: 0.00249 [I 2022-10-26 12:22:23] ...trap/engines/engine.py.230: loss: 0.35878 [I 2022-10-26 12:22:23] ...optimizers/trijoint.py.151: total_norm: 3.769307 [I 2022-10-26 12:22:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1540/2304 [I 2022-10-26 12:22:28] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:06:21 [I 2022-10-26 12:22:28] ...trap/engines/engine.py.229: process: 0.49680 | load: 0.00251 [I 2022-10-26 12:22:28] ...trap/engines/engine.py.230: loss: 0.13772 [I 2022-10-26 12:22:28] ...optimizers/trijoint.py.151: total_norm: 2.400344 [I 2022-10-26 12:22:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 1550/2304 [I 2022-10-26 12:22:33] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:06:14 [I 2022-10-26 12:22:33] ...trap/engines/engine.py.229: process: 0.49850 | load: 0.00247 [I 2022-10-26 12:22:33] ...trap/engines/engine.py.230: loss: 0.29685 [I 2022-10-26 12:22:33] ...optimizers/trijoint.py.151: total_norm: 3.590945 [I 2022-10-26 12:22:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 1560/2304 [I 2022-10-26 12:22:38] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:06:13 [I 2022-10-26 12:22:38] ...trap/engines/engine.py.229: process: 0.49188 | load: 0.00256 [I 2022-10-26 12:22:38] ...trap/engines/engine.py.230: loss: 0.36298 [I 2022-10-26 12:22:38] ...optimizers/trijoint.py.151: total_norm: 3.625420 [I 2022-10-26 12:22:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 1570/2304 [I 2022-10-26 12:22:43] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:06:07 [I 2022-10-26 12:22:43] ...trap/engines/engine.py.229: process: 0.50127 | load: 0.00250 [I 2022-10-26 12:22:43] ...trap/engines/engine.py.230: loss: 0.24551 [I 2022-10-26 12:22:43] ...optimizers/trijoint.py.151: total_norm: 3.283216 [I 2022-10-26 12:22:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 1580/2304 [I 2022-10-26 12:22:48] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:06:01 [I 2022-10-26 12:22:48] ...trap/engines/engine.py.229: process: 0.50254 | load: 0.00267 [I 2022-10-26 12:22:48] ...trap/engines/engine.py.230: loss: 0.25946 [I 2022-10-26 12:22:48] ...optimizers/trijoint.py.151: total_norm: 3.173354 [I 2022-10-26 12:22:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 1590/2304 [I 2022-10-26 12:22:53] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:05:57 [I 2022-10-26 12:22:53] ...trap/engines/engine.py.229: process: 0.51182 | load: 0.00246 [I 2022-10-26 12:22:53] ...trap/engines/engine.py.230: loss: 0.20371 [I 2022-10-26 12:22:53] ...optimizers/trijoint.py.151: total_norm: 2.491420 [I 2022-10-26 12:22:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 1600/2304 [I 2022-10-26 12:22:58] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:05:50 [I 2022-10-26 12:22:58] ...trap/engines/engine.py.229: process: 0.50202 | load: 0.00252 [I 2022-10-26 12:22:58] ...trap/engines/engine.py.230: loss: 0.24153 [I 2022-10-26 12:22:58] ...optimizers/trijoint.py.151: total_norm: 2.828554 [I 2022-10-26 12:23:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 1610/2304 [I 2022-10-26 12:23:03] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:05:46 [I 2022-10-26 12:23:03] ...trap/engines/engine.py.229: process: 0.50556 | load: 0.00246 [I 2022-10-26 12:23:03] ...trap/engines/engine.py.230: loss: 0.20584 [I 2022-10-26 12:23:03] ...optimizers/trijoint.py.151: total_norm: 2.574648 [I 2022-10-26 12:23:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 1620/2304 [I 2022-10-26 12:23:08] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:05:41 [I 2022-10-26 12:23:08] ...trap/engines/engine.py.229: process: 0.49433 | load: 0.00246 [I 2022-10-26 12:23:08] ...trap/engines/engine.py.230: loss: 0.30048 [I 2022-10-26 12:23:08] ...optimizers/trijoint.py.151: total_norm: 3.497072 [I 2022-10-26 12:23:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1630/2304 [I 2022-10-26 12:23:13] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:05:35 [I 2022-10-26 12:23:13] ...trap/engines/engine.py.229: process: 0.49984 | load: 0.00246 [I 2022-10-26 12:23:13] ...trap/engines/engine.py.230: loss: 0.26963 [I 2022-10-26 12:23:13] ...optimizers/trijoint.py.151: total_norm: 3.927297 [I 2022-10-26 12:23:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 1640/2304 [I 2022-10-26 12:23:18] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:05:33 [I 2022-10-26 12:23:18] ...trap/engines/engine.py.229: process: 0.50196 | load: 0.00247 [I 2022-10-26 12:23:18] ...trap/engines/engine.py.230: loss: 0.21008 [I 2022-10-26 12:23:18] ...optimizers/trijoint.py.151: total_norm: 3.798540 [I 2022-10-26 12:23:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1650/2304 [I 2022-10-26 12:23:23] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:05:28 [I 2022-10-26 12:23:23] ...trap/engines/engine.py.229: process: 0.50693 | load: 0.00247 [I 2022-10-26 12:23:23] ...trap/engines/engine.py.230: loss: 0.26126 [I 2022-10-26 12:23:23] ...optimizers/trijoint.py.151: total_norm: 3.696059 [I 2022-10-26 12:23:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1660/2304 [I 2022-10-26 12:23:28] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:05:21 [I 2022-10-26 12:23:28] ...trap/engines/engine.py.229: process: 0.50027 | load: 0.00251 [I 2022-10-26 12:23:28] ...trap/engines/engine.py.230: loss: 0.27230 [I 2022-10-26 12:23:28] ...optimizers/trijoint.py.151: total_norm: 4.463497 [I 2022-10-26 12:23:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 1670/2304 [I 2022-10-26 12:23:33] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:05:15 [I 2022-10-26 12:23:33] ...trap/engines/engine.py.229: process: 0.49598 | load: 0.00248 [I 2022-10-26 12:23:33] ...trap/engines/engine.py.230: loss: 0.28263 [I 2022-10-26 12:23:33] ...optimizers/trijoint.py.151: total_norm: 3.509473 [I 2022-10-26 12:23:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 1680/2304 [I 2022-10-26 12:23:38] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:05:11 [I 2022-10-26 12:23:38] ...trap/engines/engine.py.229: process: 0.50593 | load: 0.00262 [I 2022-10-26 12:23:38] ...trap/engines/engine.py.230: loss: 0.30990 [I 2022-10-26 12:23:38] ...optimizers/trijoint.py.151: total_norm: 3.471569 [I 2022-10-26 12:23:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 1690/2304 [I 2022-10-26 12:23:43] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:05:05 [I 2022-10-26 12:23:43] ...trap/engines/engine.py.229: process: 0.49523 | load: 0.00245 [I 2022-10-26 12:23:43] ...trap/engines/engine.py.230: loss: 0.25628 [I 2022-10-26 12:23:43] ...optimizers/trijoint.py.151: total_norm: 4.118742 [I 2022-10-26 12:23:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 1700/2304 [I 2022-10-26 12:23:48] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:04:59 [I 2022-10-26 12:23:48] ...trap/engines/engine.py.229: process: 0.49916 | load: 0.00245 [I 2022-10-26 12:23:48] ...trap/engines/engine.py.230: loss: 0.35626 [I 2022-10-26 12:23:48] ...optimizers/trijoint.py.151: total_norm: 4.202381 [I 2022-10-26 12:23:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 1710/2304 [I 2022-10-26 12:23:53] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:04:53 [I 2022-10-26 12:23:53] ...trap/engines/engine.py.229: process: 0.48556 | load: 0.00241 [I 2022-10-26 12:23:53] ...trap/engines/engine.py.230: loss: 0.26273 [I 2022-10-26 12:23:53] ...optimizers/trijoint.py.151: total_norm: 3.340173 [I 2022-10-26 12:23:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 1720/2304 [I 2022-10-26 12:23:58] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:04:50 [I 2022-10-26 12:23:58] ...trap/engines/engine.py.229: process: 0.50099 | load: 0.00244 [I 2022-10-26 12:23:58] ...trap/engines/engine.py.230: loss: 0.22414 [I 2022-10-26 12:23:58] ...optimizers/trijoint.py.151: total_norm: 2.985254 [I 2022-10-26 12:24:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 1730/2304 [I 2022-10-26 12:24:03] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:04:45 [I 2022-10-26 12:24:03] ...trap/engines/engine.py.229: process: 0.49837 | load: 0.00243 [I 2022-10-26 12:24:03] ...trap/engines/engine.py.230: loss: 0.20461 [I 2022-10-26 12:24:03] ...optimizers/trijoint.py.151: total_norm: 3.417218 [I 2022-10-26 12:24:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 1740/2304 [I 2022-10-26 12:24:08] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:04:42 [I 2022-10-26 12:24:08] ...trap/engines/engine.py.229: process: 0.50225 | load: 0.00250 [I 2022-10-26 12:24:08] ...trap/engines/engine.py.230: loss: 0.28686 [I 2022-10-26 12:24:08] ...optimizers/trijoint.py.151: total_norm: 2.859709 [I 2022-10-26 12:24:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1750/2304 [I 2022-10-26 12:24:13] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:04:37 [I 2022-10-26 12:24:13] ...trap/engines/engine.py.229: process: 0.49689 | load: 0.00246 [I 2022-10-26 12:24:13] ...trap/engines/engine.py.230: loss: 0.31523 [I 2022-10-26 12:24:13] ...optimizers/trijoint.py.151: total_norm: 3.698940 [I 2022-10-26 12:24:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 1760/2304 [I 2022-10-26 12:24:18] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:04:30 [I 2022-10-26 12:24:18] ...trap/engines/engine.py.229: process: 0.49715 | load: 0.00249 [I 2022-10-26 12:24:18] ...trap/engines/engine.py.230: loss: 0.25381 [I 2022-10-26 12:24:18] ...optimizers/trijoint.py.151: total_norm: 3.839655 [I 2022-10-26 12:24:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1770/2304 [I 2022-10-26 12:24:23] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:04:24 [I 2022-10-26 12:24:23] ...trap/engines/engine.py.229: process: 0.49352 | load: 0.00244 [I 2022-10-26 12:24:23] ...trap/engines/engine.py.230: loss: 0.18632 [I 2022-10-26 12:24:23] ...optimizers/trijoint.py.151: total_norm: 2.385701 [I 2022-10-26 12:24:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1780/2304 [I 2022-10-26 12:24:28] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:04:20 [I 2022-10-26 12:24:28] ...trap/engines/engine.py.229: process: 0.50185 | load: 0.00247 [I 2022-10-26 12:24:28] ...trap/engines/engine.py.230: loss: 0.39899 [I 2022-10-26 12:24:28] ...optimizers/trijoint.py.151: total_norm: 5.796701 [I 2022-10-26 12:24:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 1790/2304 [I 2022-10-26 12:24:33] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:04:16 [I 2022-10-26 12:24:33] ...trap/engines/engine.py.229: process: 0.50435 | load: 0.00242 [I 2022-10-26 12:24:33] ...trap/engines/engine.py.230: loss: 0.27963 [I 2022-10-26 12:24:33] ...optimizers/trijoint.py.151: total_norm: 2.905468 [I 2022-10-26 12:24:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 1800/2304 [I 2022-10-26 12:24:38] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:04:11 [I 2022-10-26 12:24:38] ...trap/engines/engine.py.229: process: 0.50037 | load: 0.00242 [I 2022-10-26 12:24:38] ...trap/engines/engine.py.230: loss: 0.24757 [I 2022-10-26 12:24:38] ...optimizers/trijoint.py.151: total_norm: 3.130431 [I 2022-10-26 12:24:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 1810/2304 [I 2022-10-26 12:24:43] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:04:06 [I 2022-10-26 12:24:43] ...trap/engines/engine.py.229: process: 0.49492 | load: 0.00242 [I 2022-10-26 12:24:43] ...trap/engines/engine.py.230: loss: 0.28258 [I 2022-10-26 12:24:43] ...optimizers/trijoint.py.151: total_norm: 3.763498 [I 2022-10-26 12:24:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 1820/2304 [I 2022-10-26 12:24:48] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:04:02 [I 2022-10-26 12:24:48] ...trap/engines/engine.py.229: process: 0.49303 | load: 0.00254 [I 2022-10-26 12:24:48] ...trap/engines/engine.py.230: loss: 0.24652 [I 2022-10-26 12:24:48] ...optimizers/trijoint.py.151: total_norm: 2.757220 [I 2022-10-26 12:24:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 1830/2304 [I 2022-10-26 12:24:53] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:03:55 [I 2022-10-26 12:24:53] ...trap/engines/engine.py.229: process: 0.48930 | load: 0.00247 [I 2022-10-26 12:24:53] ...trap/engines/engine.py.230: loss: 0.22697 [I 2022-10-26 12:24:53] ...optimizers/trijoint.py.151: total_norm: 2.933398 [I 2022-10-26 12:24:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 1840/2304 [I 2022-10-26 12:24:58] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:03:51 [I 2022-10-26 12:24:58] ...trap/engines/engine.py.229: process: 0.50196 | load: 0.00246 [I 2022-10-26 12:24:58] ...trap/engines/engine.py.230: loss: 0.28463 [I 2022-10-26 12:24:58] ...optimizers/trijoint.py.151: total_norm: 4.696560 [I 2022-10-26 12:25:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 1850/2304 [I 2022-10-26 12:25:03] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:03:46 [I 2022-10-26 12:25:03] ...trap/engines/engine.py.229: process: 0.50594 | load: 0.00247 [I 2022-10-26 12:25:03] ...trap/engines/engine.py.230: loss: 0.27990 [I 2022-10-26 12:25:03] ...optimizers/trijoint.py.151: total_norm: 3.004370 [I 2022-10-26 12:25:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 1860/2304 [I 2022-10-26 12:25:08] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:03:39 [I 2022-10-26 12:25:08] ...trap/engines/engine.py.229: process: 0.49428 | load: 0.00243 [I 2022-10-26 12:25:08] ...trap/engines/engine.py.230: loss: 0.28806 [I 2022-10-26 12:25:08] ...optimizers/trijoint.py.151: total_norm: 2.974239 [I 2022-10-26 12:25:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1870/2304 [I 2022-10-26 12:25:13] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:03:38 [I 2022-10-26 12:25:13] ...trap/engines/engine.py.229: process: 0.50183 | load: 0.00262 [I 2022-10-26 12:25:13] ...trap/engines/engine.py.230: loss: 0.31213 [I 2022-10-26 12:25:13] ...optimizers/trijoint.py.151: total_norm: 3.558973 [I 2022-10-26 12:25:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 1880/2304 [I 2022-10-26 12:25:18] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:03:32 [I 2022-10-26 12:25:18] ...trap/engines/engine.py.229: process: 0.50359 | load: 0.00250 [I 2022-10-26 12:25:18] ...trap/engines/engine.py.230: loss: 0.38277 [I 2022-10-26 12:25:18] ...optimizers/trijoint.py.151: total_norm: 4.210283 [I 2022-10-26 12:25:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 1890/2304 [I 2022-10-26 12:25:23] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:03:26 [I 2022-10-26 12:25:23] ...trap/engines/engine.py.229: process: 0.50345 | load: 0.00251 [I 2022-10-26 12:25:23] ...trap/engines/engine.py.230: loss: 0.26827 [I 2022-10-26 12:25:23] ...optimizers/trijoint.py.151: total_norm: 3.803695 [I 2022-10-26 12:25:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 1900/2304 [I 2022-10-26 12:25:28] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:03:22 [I 2022-10-26 12:25:28] ...trap/engines/engine.py.229: process: 0.49477 | load: 0.00247 [I 2022-10-26 12:25:28] ...trap/engines/engine.py.230: loss: 0.25592 [I 2022-10-26 12:25:28] ...optimizers/trijoint.py.151: total_norm: 3.469436 [I 2022-10-26 12:25:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 1910/2304 [I 2022-10-26 12:25:33] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:03:15 [I 2022-10-26 12:25:33] ...trap/engines/engine.py.229: process: 0.49614 | load: 0.00250 [I 2022-10-26 12:25:33] ...trap/engines/engine.py.230: loss: 0.23879 [I 2022-10-26 12:25:33] ...optimizers/trijoint.py.151: total_norm: 3.321204 [I 2022-10-26 12:25:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 1920/2304 [I 2022-10-26 12:25:38] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:03:11 [I 2022-10-26 12:25:38] ...trap/engines/engine.py.229: process: 0.48937 | load: 0.00247 [I 2022-10-26 12:25:38] ...trap/engines/engine.py.230: loss: 0.18734 [I 2022-10-26 12:25:38] ...optimizers/trijoint.py.151: total_norm: 2.764823 [I 2022-10-26 12:25:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 1930/2304 [I 2022-10-26 12:25:43] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:03:06 [I 2022-10-26 12:25:43] ...trap/engines/engine.py.229: process: 0.49857 | load: 0.00244 [I 2022-10-26 12:25:43] ...trap/engines/engine.py.230: loss: 0.36398 [I 2022-10-26 12:25:43] ...optimizers/trijoint.py.151: total_norm: 3.216141 [I 2022-10-26 12:25:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 1940/2304 [I 2022-10-26 12:25:48] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:03:01 [I 2022-10-26 12:25:48] ...trap/engines/engine.py.229: process: 0.49330 | load: 0.00251 [I 2022-10-26 12:25:48] ...trap/engines/engine.py.230: loss: 0.23564 [I 2022-10-26 12:25:48] ...optimizers/trijoint.py.151: total_norm: 3.124374 [I 2022-10-26 12:25:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 1950/2304 [I 2022-10-26 12:25:53] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:02:55 [I 2022-10-26 12:25:53] ...trap/engines/engine.py.229: process: 0.48329 | load: 0.00245 [I 2022-10-26 12:25:53] ...trap/engines/engine.py.230: loss: 0.17942 [I 2022-10-26 12:25:53] ...optimizers/trijoint.py.151: total_norm: 2.368539 [I 2022-10-26 12:25:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 1960/2304 [I 2022-10-26 12:25:58] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:02:50 [I 2022-10-26 12:25:58] ...trap/engines/engine.py.229: process: 0.49046 | load: 0.00251 [I 2022-10-26 12:25:58] ...trap/engines/engine.py.230: loss: 0.19794 [I 2022-10-26 12:25:58] ...optimizers/trijoint.py.151: total_norm: 3.002512 [I 2022-10-26 12:26:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 1970/2304 [I 2022-10-26 12:26:03] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:02:47 [I 2022-10-26 12:26:03] ...trap/engines/engine.py.229: process: 0.49025 | load: 0.00248 [I 2022-10-26 12:26:03] ...trap/engines/engine.py.230: loss: 0.19702 [I 2022-10-26 12:26:03] ...optimizers/trijoint.py.151: total_norm: 3.076585 [I 2022-10-26 12:26:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 1980/2304 [I 2022-10-26 12:26:08] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:02:41 [I 2022-10-26 12:26:08] ...trap/engines/engine.py.229: process: 0.49902 | load: 0.00240 [I 2022-10-26 12:26:08] ...trap/engines/engine.py.230: loss: 0.24402 [I 2022-10-26 12:26:08] ...optimizers/trijoint.py.151: total_norm: 3.977573 [I 2022-10-26 12:26:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 1990/2304 [I 2022-10-26 12:26:13] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:02:35 [I 2022-10-26 12:26:13] ...trap/engines/engine.py.229: process: 0.48753 | load: 0.00244 [I 2022-10-26 12:26:13] ...trap/engines/engine.py.230: loss: 0.35112 [I 2022-10-26 12:26:13] ...optimizers/trijoint.py.151: total_norm: 3.597223 [I 2022-10-26 12:26:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 2000/2304 [I 2022-10-26 12:26:18] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:02:32 [I 2022-10-26 12:26:18] ...trap/engines/engine.py.229: process: 0.50252 | load: 0.00243 [I 2022-10-26 12:26:18] ...trap/engines/engine.py.230: loss: 0.22525 [I 2022-10-26 12:26:18] ...optimizers/trijoint.py.151: total_norm: 3.179343 [I 2022-10-26 12:26:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 2010/2304 [I 2022-10-26 12:26:23] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:02:26 [I 2022-10-26 12:26:23] ...trap/engines/engine.py.229: process: 0.48870 | load: 0.00256 [I 2022-10-26 12:26:23] ...trap/engines/engine.py.230: loss: 0.28272 [I 2022-10-26 12:26:23] ...optimizers/trijoint.py.151: total_norm: 4.129572 [I 2022-10-26 12:26:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 2020/2304 [I 2022-10-26 12:26:28] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:02:21 [I 2022-10-26 12:26:28] ...trap/engines/engine.py.229: process: 0.50348 | load: 0.00250 [I 2022-10-26 12:26:28] ...trap/engines/engine.py.230: loss: 0.29751 [I 2022-10-26 12:26:28] ...optimizers/trijoint.py.151: total_norm: 4.127906 [I 2022-10-26 12:26:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 2030/2304 [I 2022-10-26 12:26:33] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:02:16 [I 2022-10-26 12:26:33] ...trap/engines/engine.py.229: process: 0.48376 | load: 0.00268 [I 2022-10-26 12:26:33] ...trap/engines/engine.py.230: loss: 0.17413 [I 2022-10-26 12:26:33] ...optimizers/trijoint.py.151: total_norm: 1.924357 [I 2022-10-26 12:26:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 2040/2304 [I 2022-10-26 12:26:38] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:02:11 [I 2022-10-26 12:26:38] ...trap/engines/engine.py.229: process: 0.49644 | load: 0.00248 [I 2022-10-26 12:26:38] ...trap/engines/engine.py.230: loss: 0.24566 [I 2022-10-26 12:26:38] ...optimizers/trijoint.py.151: total_norm: 3.300673 [I 2022-10-26 12:26:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 2050/2304 [I 2022-10-26 12:26:43] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:02:06 [I 2022-10-26 12:26:43] ...trap/engines/engine.py.229: process: 0.49458 | load: 0.00260 [I 2022-10-26 12:26:43] ...trap/engines/engine.py.230: loss: 0.24756 [I 2022-10-26 12:26:43] ...optimizers/trijoint.py.151: total_norm: 3.243231 [I 2022-10-26 12:26:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 2060/2304 [I 2022-10-26 12:26:48] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:02:01 [I 2022-10-26 12:26:48] ...trap/engines/engine.py.229: process: 0.49774 | load: 0.00256 [I 2022-10-26 12:26:48] ...trap/engines/engine.py.230: loss: 0.22917 [I 2022-10-26 12:26:48] ...optimizers/trijoint.py.151: total_norm: 3.442814 [I 2022-10-26 12:26:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 2070/2304 [I 2022-10-26 12:26:53] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:01:57 [I 2022-10-26 12:26:53] ...trap/engines/engine.py.229: process: 0.50354 | load: 0.00254 [I 2022-10-26 12:26:53] ...trap/engines/engine.py.230: loss: 0.26271 [I 2022-10-26 12:26:53] ...optimizers/trijoint.py.151: total_norm: 3.296412 [I 2022-10-26 12:26:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 2080/2304 [I 2022-10-26 12:26:58] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:01:50 [I 2022-10-26 12:26:58] ...trap/engines/engine.py.229: process: 0.50229 | load: 0.00249 [I 2022-10-26 12:26:58] ...trap/engines/engine.py.230: loss: 0.36777 [I 2022-10-26 12:26:58] ...optimizers/trijoint.py.151: total_norm: 4.384508 [I 2022-10-26 12:27:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 2090/2304 [I 2022-10-26 12:27:03] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:01:47 [I 2022-10-26 12:27:03] ...trap/engines/engine.py.229: process: 0.50636 | load: 0.00251 [I 2022-10-26 12:27:03] ...trap/engines/engine.py.230: loss: 0.32027 [I 2022-10-26 12:27:03] ...optimizers/trijoint.py.151: total_norm: 3.557577 [I 2022-10-26 12:27:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 2100/2304 [I 2022-10-26 12:27:08] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:01:41 [I 2022-10-26 12:27:08] ...trap/engines/engine.py.229: process: 0.49977 | load: 0.00252 [I 2022-10-26 12:27:08] ...trap/engines/engine.py.230: loss: 0.25035 [I 2022-10-26 12:27:08] ...optimizers/trijoint.py.151: total_norm: 3.324785 [I 2022-10-26 12:27:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 2110/2304 [I 2022-10-26 12:27:13] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:01:39 [I 2022-10-26 12:27:13] ...trap/engines/engine.py.229: process: 0.49007 | load: 0.00260 [I 2022-10-26 12:27:13] ...trap/engines/engine.py.230: loss: 0.21700 [I 2022-10-26 12:27:13] ...optimizers/trijoint.py.151: total_norm: 2.280995 [I 2022-10-26 12:27:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 2120/2304 [I 2022-10-26 12:27:18] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:01:31 [I 2022-10-26 12:27:18] ...trap/engines/engine.py.229: process: 0.49239 | load: 0.00258 [I 2022-10-26 12:27:18] ...trap/engines/engine.py.230: loss: 0.20777 [I 2022-10-26 12:27:18] ...optimizers/trijoint.py.151: total_norm: 4.092213 [I 2022-10-26 12:27:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 2130/2304 [I 2022-10-26 12:27:23] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:01:26 [I 2022-10-26 12:27:23] ...trap/engines/engine.py.229: process: 0.48883 | load: 0.00254 [I 2022-10-26 12:27:23] ...trap/engines/engine.py.230: loss: 0.20910 [I 2022-10-26 12:27:23] ...optimizers/trijoint.py.151: total_norm: 2.950120 [I 2022-10-26 12:27:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 2140/2304 [I 2022-10-26 12:27:28] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:01:21 [I 2022-10-26 12:27:28] ...trap/engines/engine.py.229: process: 0.49976 | load: 0.00243 [I 2022-10-26 12:27:28] ...trap/engines/engine.py.230: loss: 0.26872 [I 2022-10-26 12:27:28] ...optimizers/trijoint.py.151: total_norm: 3.166682 [I 2022-10-26 12:27:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 2150/2304 [I 2022-10-26 12:27:33] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:01:16 [I 2022-10-26 12:27:33] ...trap/engines/engine.py.229: process: 0.49686 | load: 0.00242 [I 2022-10-26 12:27:33] ...trap/engines/engine.py.230: loss: 0.23195 [I 2022-10-26 12:27:33] ...optimizers/trijoint.py.151: total_norm: 2.840898 [I 2022-10-26 12:27:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 2160/2304 [I 2022-10-26 12:27:38] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:01:11 [I 2022-10-26 12:27:38] ...trap/engines/engine.py.229: process: 0.48467 | load: 0.00250 [I 2022-10-26 12:27:38] ...trap/engines/engine.py.230: loss: 0.18456 [I 2022-10-26 12:27:38] ...optimizers/trijoint.py.151: total_norm: 3.040089 [I 2022-10-26 12:27:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 2170/2304 [I 2022-10-26 12:27:43] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:01:06 [I 2022-10-26 12:27:43] ...trap/engines/engine.py.229: process: 0.49148 | load: 0.00246 [I 2022-10-26 12:27:43] ...trap/engines/engine.py.230: loss: 0.33205 [I 2022-10-26 12:27:43] ...optimizers/trijoint.py.151: total_norm: 4.092090 [I 2022-10-26 12:27:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 2180/2304 [I 2022-10-26 12:27:48] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:01:03 [I 2022-10-26 12:27:48] ...trap/engines/engine.py.229: process: 0.49457 | load: 0.00248 [I 2022-10-26 12:27:48] ...trap/engines/engine.py.230: loss: 0.22735 [I 2022-10-26 12:27:48] ...optimizers/trijoint.py.151: total_norm: 3.791019 [I 2022-10-26 12:27:53] ...trap/engines/engine.py.224: train: epoch 7 | batch 2190/2304 [I 2022-10-26 12:27:53] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:00:56 [I 2022-10-26 12:27:53] ...trap/engines/engine.py.229: process: 0.49728 | load: 0.00247 [I 2022-10-26 12:27:53] ...trap/engines/engine.py.230: loss: 0.33119 [I 2022-10-26 12:27:53] ...optimizers/trijoint.py.151: total_norm: 3.564225 [I 2022-10-26 12:27:58] ...trap/engines/engine.py.224: train: epoch 7 | batch 2200/2304 [I 2022-10-26 12:27:58] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:00:51 [I 2022-10-26 12:27:58] ...trap/engines/engine.py.229: process: 0.50381 | load: 0.00247 [I 2022-10-26 12:27:58] ...trap/engines/engine.py.230: loss: 0.26151 [I 2022-10-26 12:27:58] ...optimizers/trijoint.py.151: total_norm: 2.812115 [I 2022-10-26 12:28:03] ...trap/engines/engine.py.224: train: epoch 7 | batch 2210/2304 [I 2022-10-26 12:28:03] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:00:47 [I 2022-10-26 12:28:03] ...trap/engines/engine.py.229: process: 0.49574 | load: 0.00249 [I 2022-10-26 12:28:03] ...trap/engines/engine.py.230: loss: 0.22559 [I 2022-10-26 12:28:03] ...optimizers/trijoint.py.151: total_norm: 3.328868 [I 2022-10-26 12:28:08] ...trap/engines/engine.py.224: train: epoch 7 | batch 2220/2304 [I 2022-10-26 12:28:08] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:00:41 [I 2022-10-26 12:28:08] ...trap/engines/engine.py.229: process: 0.49622 | load: 0.00246 [I 2022-10-26 12:28:08] ...trap/engines/engine.py.230: loss: 0.26953 [I 2022-10-26 12:28:08] ...optimizers/trijoint.py.151: total_norm: 4.134007 [I 2022-10-26 12:28:13] ...trap/engines/engine.py.224: train: epoch 7 | batch 2230/2304 [I 2022-10-26 12:28:13] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:00:36 [I 2022-10-26 12:28:13] ...trap/engines/engine.py.229: process: 0.50049 | load: 0.00251 [I 2022-10-26 12:28:13] ...trap/engines/engine.py.230: loss: 0.33085 [I 2022-10-26 12:28:13] ...optimizers/trijoint.py.151: total_norm: 4.310568 [I 2022-10-26 12:28:18] ...trap/engines/engine.py.224: train: epoch 7 | batch 2240/2304 [I 2022-10-26 12:28:18] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:00:31 [I 2022-10-26 12:28:18] ...trap/engines/engine.py.229: process: 0.50196 | load: 0.00246 [I 2022-10-26 12:28:18] ...trap/engines/engine.py.230: loss: 0.28696 [I 2022-10-26 12:28:18] ...optimizers/trijoint.py.151: total_norm: 3.227141 [I 2022-10-26 12:28:23] ...trap/engines/engine.py.224: train: epoch 7 | batch 2250/2304 [I 2022-10-26 12:28:23] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:00:27 [I 2022-10-26 12:28:23] ...trap/engines/engine.py.229: process: 0.57460 | load: 0.00248 [I 2022-10-26 12:28:23] ...trap/engines/engine.py.230: loss: 0.19644 [I 2022-10-26 12:28:23] ...optimizers/trijoint.py.151: total_norm: 3.118520 [I 2022-10-26 12:28:28] ...trap/engines/engine.py.224: train: epoch 7 | batch 2260/2304 [I 2022-10-26 12:28:28] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:00:21 [I 2022-10-26 12:28:28] ...trap/engines/engine.py.229: process: 0.49121 | load: 0.00254 [I 2022-10-26 12:28:28] ...trap/engines/engine.py.230: loss: 0.29396 [I 2022-10-26 12:28:28] ...optimizers/trijoint.py.151: total_norm: 4.721076 [I 2022-10-26 12:28:33] ...trap/engines/engine.py.224: train: epoch 7 | batch 2270/2304 [I 2022-10-26 12:28:33] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:00:16 [I 2022-10-26 12:28:33] ...trap/engines/engine.py.229: process: 0.49857 | load: 0.00256 [I 2022-10-26 12:28:33] ...trap/engines/engine.py.230: loss: 0.20783 [I 2022-10-26 12:28:33] ...optimizers/trijoint.py.151: total_norm: 3.171083 [I 2022-10-26 12:28:38] ...trap/engines/engine.py.224: train: epoch 7 | batch 2280/2304 [I 2022-10-26 12:28:38] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:00:11 [I 2022-10-26 12:28:38] ...trap/engines/engine.py.229: process: 0.48641 | load: 0.00255 [I 2022-10-26 12:28:38] ...trap/engines/engine.py.230: loss: 0.30096 [I 2022-10-26 12:28:38] ...optimizers/trijoint.py.151: total_norm: 3.518287 [I 2022-10-26 12:28:43] ...trap/engines/engine.py.224: train: epoch 7 | batch 2290/2304 [I 2022-10-26 12:28:43] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:00:06 [I 2022-10-26 12:28:43] ...trap/engines/engine.py.229: process: 0.48795 | load: 0.00250 [I 2022-10-26 12:28:43] ...trap/engines/engine.py.230: loss: 0.22896 [I 2022-10-26 12:28:43] ...optimizers/trijoint.py.151: total_norm: 3.487776 [I 2022-10-26 12:28:48] ...trap/engines/engine.py.224: train: epoch 7 | batch 2300/2304 [I 2022-10-26 12:28:48] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:00:02 [I 2022-10-26 12:28:48] ...trap/engines/engine.py.229: process: 0.49627 | load: 0.00014 [I 2022-10-26 12:28:48] ...trap/engines/engine.py.230: loss: 0.31698 [I 2022-10-26 12:28:48] ...optimizers/trijoint.py.151: total_norm: 3.937211 [I 2022-10-26 12:28:50] ...trap/engines/engine.py.224: train: epoch 7 | batch 2304/2304 [I 2022-10-26 12:28:50] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:00:00 [I 2022-10-26 12:28:50] ...trap/engines/engine.py.229: process: 0.49174 | load: 0.00014 [I 2022-10-26 12:28:50] ...trap/engines/engine.py.230: loss: 0.25729 [I 2022-10-26 12:28:50] ...optimizers/trijoint.py.151: total_norm: 3.983340 [S 2022-10-26 12:28:51] ...trap/engines/engine.py.129: train_epoch.epoch: 7 [S 2022-10-26 12:28:51] ...trap/engines/engine.py.129: train_epoch.loss: 0.2560569995072814 [I 2022-10-26 12:28:51] ...s/criterions/triplet.py.84: margin = 0.09000000000000002 [I 2022-10-26 12:28:51] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 7 [I 2022-10-26 12:28:52] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 12:28:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 12:28:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 12:28:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 12:28:54] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 12:28:54] ...trap/engines/engine.py.321: eval: epoch 7 | batch 0/510 [I 2022-10-26 12:28:54] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:03 [I 2022-10-26 12:28:54] ...trap/engines/engine.py.326: process: 2.59421 | load: 2.22959 [I 2022-10-26 12:28:57] ...trap/engines/engine.py.321: eval: epoch 7 | batch 10/510 [I 2022-10-26 12:28:57] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:04:46 [I 2022-10-26 12:28:57] ...trap/engines/engine.py.326: process: 0.32419 | load: 0.00020 [I 2022-10-26 12:29:01] ...trap/engines/engine.py.321: eval: epoch 7 | batch 20/510 [I 2022-10-26 12:29:01] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:59 [I 2022-10-26 12:29:01] ...trap/engines/engine.py.326: process: 0.32463 | load: 0.00018 [I 2022-10-26 12:29:04] ...trap/engines/engine.py.321: eval: epoch 7 | batch 30/510 [I 2022-10-26 12:29:04] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:42 [I 2022-10-26 12:29:04] ...trap/engines/engine.py.326: process: 0.34143 | load: 0.00017 [I 2022-10-26 12:29:08] ...trap/engines/engine.py.321: eval: epoch 7 | batch 40/510 [I 2022-10-26 12:29:08] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:38 [I 2022-10-26 12:29:08] ...trap/engines/engine.py.326: process: 0.32323 | load: 0.00017 [I 2022-10-26 12:29:11] ...trap/engines/engine.py.321: eval: epoch 7 | batch 50/510 [I 2022-10-26 12:29:11] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-26 12:29:11] ...trap/engines/engine.py.326: process: 0.34605 | load: 0.00016 [I 2022-10-26 12:29:14] ...trap/engines/engine.py.321: eval: epoch 7 | batch 60/510 [I 2022-10-26 12:29:14] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:29 [I 2022-10-26 12:29:14] ...trap/engines/engine.py.326: process: 0.32667 | load: 0.00019 [I 2022-10-26 12:29:18] ...trap/engines/engine.py.321: eval: epoch 7 | batch 70/510 [I 2022-10-26 12:29:18] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:26 [I 2022-10-26 12:29:18] ...trap/engines/engine.py.326: process: 0.33936 | load: 0.00018 [I 2022-10-26 12:29:21] ...trap/engines/engine.py.321: eval: epoch 7 | batch 80/510 [I 2022-10-26 12:29:21] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:22 [I 2022-10-26 12:29:21] ...trap/engines/engine.py.326: process: 0.32293 | load: 0.00021 [I 2022-10-26 12:29:24] ...trap/engines/engine.py.321: eval: epoch 7 | batch 90/510 [I 2022-10-26 12:29:24] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:22 [I 2022-10-26 12:29:24] ...trap/engines/engine.py.326: process: 0.32312 | load: 0.00019 [I 2022-10-26 12:29:28] ...trap/engines/engine.py.321: eval: epoch 7 | batch 100/510 [I 2022-10-26 12:29:28] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:15 [I 2022-10-26 12:29:28] ...trap/engines/engine.py.326: process: 0.30543 | load: 0.00021 [I 2022-10-26 12:29:31] ...trap/engines/engine.py.321: eval: epoch 7 | batch 110/510 [I 2022-10-26 12:29:31] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:58 [I 2022-10-26 12:29:31] ...trap/engines/engine.py.326: process: 0.28742 | load: 0.00023 [I 2022-10-26 12:29:33] ...trap/engines/engine.py.321: eval: epoch 7 | batch 120/510 [I 2022-10-26 12:29:33] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-26 12:29:33] ...trap/engines/engine.py.326: process: 0.28797 | load: 0.00015 [I 2022-10-26 12:29:36] ...trap/engines/engine.py.321: eval: epoch 7 | batch 130/510 [I 2022-10-26 12:29:36] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:50 [I 2022-10-26 12:29:36] ...trap/engines/engine.py.326: process: 0.28643 | load: 0.00016 [I 2022-10-26 12:29:39] ...trap/engines/engine.py.321: eval: epoch 7 | batch 140/510 [I 2022-10-26 12:29:39] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-26 12:29:39] ...trap/engines/engine.py.326: process: 0.29081 | load: 0.00021 [I 2022-10-26 12:29:42] ...trap/engines/engine.py.321: eval: epoch 7 | batch 150/510 [I 2022-10-26 12:29:42] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:43 [I 2022-10-26 12:29:42] ...trap/engines/engine.py.326: process: 0.28457 | load: 0.00017 [I 2022-10-26 12:29:45] ...trap/engines/engine.py.321: eval: epoch 7 | batch 160/510 [I 2022-10-26 12:29:45] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 12:29:45] ...trap/engines/engine.py.326: process: 0.28215 | load: 0.00018 [I 2022-10-26 12:29:48] ...trap/engines/engine.py.321: eval: epoch 7 | batch 170/510 [I 2022-10-26 12:29:48] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:37 [I 2022-10-26 12:29:48] ...trap/engines/engine.py.326: process: 0.28396 | load: 0.00020 [I 2022-10-26 12:29:51] ...trap/engines/engine.py.321: eval: epoch 7 | batch 180/510 [I 2022-10-26 12:29:51] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 12:29:51] ...trap/engines/engine.py.326: process: 0.28384 | load: 0.00015 [I 2022-10-26 12:29:54] ...trap/engines/engine.py.321: eval: epoch 7 | batch 190/510 [I 2022-10-26 12:29:54] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-26 12:29:54] ...trap/engines/engine.py.326: process: 0.30042 | load: 0.00015 [I 2022-10-26 12:29:57] ...trap/engines/engine.py.321: eval: epoch 7 | batch 200/510 [I 2022-10-26 12:29:57] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 12:29:57] ...trap/engines/engine.py.326: process: 0.27538 | load: 0.00016 [I 2022-10-26 12:30:00] ...trap/engines/engine.py.321: eval: epoch 7 | batch 210/510 [I 2022-10-26 12:30:00] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 12:30:00] ...trap/engines/engine.py.326: process: 0.27757 | load: 0.00016 [I 2022-10-26 12:30:03] ...trap/engines/engine.py.321: eval: epoch 7 | batch 220/510 [I 2022-10-26 12:30:03] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-26 12:30:03] ...trap/engines/engine.py.326: process: 0.28737 | load: 0.00021 [I 2022-10-26 12:30:06] ...trap/engines/engine.py.321: eval: epoch 7 | batch 230/510 [I 2022-10-26 12:30:06] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 12:30:06] ...trap/engines/engine.py.326: process: 0.28702 | load: 0.00019 [I 2022-10-26 12:30:08] ...trap/engines/engine.py.321: eval: epoch 7 | batch 240/510 [I 2022-10-26 12:30:08] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:17 [I 2022-10-26 12:30:08] ...trap/engines/engine.py.326: process: 0.28832 | load: 0.00015 [I 2022-10-26 12:30:11] ...trap/engines/engine.py.321: eval: epoch 7 | batch 250/510 [I 2022-10-26 12:30:11] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:14 [I 2022-10-26 12:30:11] ...trap/engines/engine.py.326: process: 0.28067 | load: 0.00018 [I 2022-10-26 12:30:14] ...trap/engines/engine.py.321: eval: epoch 7 | batch 260/510 [I 2022-10-26 12:30:14] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:13 [I 2022-10-26 12:30:14] ...trap/engines/engine.py.326: process: 0.29162 | load: 0.00015 [I 2022-10-26 12:30:17] ...trap/engines/engine.py.321: eval: epoch 7 | batch 270/510 [I 2022-10-26 12:30:17] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-26 12:30:17] ...trap/engines/engine.py.326: process: 0.28887 | load: 0.00016 [I 2022-10-26 12:30:20] ...trap/engines/engine.py.321: eval: epoch 7 | batch 280/510 [I 2022-10-26 12:30:20] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 12:30:20] ...trap/engines/engine.py.326: process: 0.29163 | load: 0.00016 [I 2022-10-26 12:30:23] ...trap/engines/engine.py.321: eval: epoch 7 | batch 290/510 [I 2022-10-26 12:30:23] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 12:30:23] ...trap/engines/engine.py.326: process: 0.28782 | load: 0.00016 [I 2022-10-26 12:30:26] ...trap/engines/engine.py.321: eval: epoch 7 | batch 300/510 [I 2022-10-26 12:30:26] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 12:30:26] ...trap/engines/engine.py.326: process: 0.28750 | load: 0.00015 [I 2022-10-26 12:30:29] ...trap/engines/engine.py.321: eval: epoch 7 | batch 310/510 [I 2022-10-26 12:30:29] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 12:30:29] ...trap/engines/engine.py.326: process: 0.29200 | load: 0.00022 [I 2022-10-26 12:30:32] ...trap/engines/engine.py.321: eval: epoch 7 | batch 320/510 [I 2022-10-26 12:30:32] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:55 [I 2022-10-26 12:30:32] ...trap/engines/engine.py.326: process: 0.29096 | load: 0.00015 [I 2022-10-26 12:30:34] ...trap/engines/engine.py.321: eval: epoch 7 | batch 330/510 [I 2022-10-26 12:30:34] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:52 [I 2022-10-26 12:30:34] ...trap/engines/engine.py.326: process: 0.28485 | load: 0.00015 [I 2022-10-26 12:30:37] ...trap/engines/engine.py.321: eval: epoch 7 | batch 340/510 [I 2022-10-26 12:30:37] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:48 [I 2022-10-26 12:30:37] ...trap/engines/engine.py.326: process: 0.28542 | load: 0.00019 [I 2022-10-26 12:30:40] ...trap/engines/engine.py.321: eval: epoch 7 | batch 350/510 [I 2022-10-26 12:30:40] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:45 [I 2022-10-26 12:30:40] ...trap/engines/engine.py.326: process: 0.28441 | load: 0.00019 [I 2022-10-26 12:30:43] ...trap/engines/engine.py.321: eval: epoch 7 | batch 360/510 [I 2022-10-26 12:30:43] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 12:30:43] ...trap/engines/engine.py.326: process: 0.28739 | load: 0.00015 [I 2022-10-26 12:30:46] ...trap/engines/engine.py.321: eval: epoch 7 | batch 370/510 [I 2022-10-26 12:30:46] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 12:30:46] ...trap/engines/engine.py.326: process: 0.28709 | load: 0.00016 [I 2022-10-26 12:30:49] ...trap/engines/engine.py.321: eval: epoch 7 | batch 380/510 [I 2022-10-26 12:30:49] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 12:30:49] ...trap/engines/engine.py.326: process: 0.29000 | load: 0.00016 [I 2022-10-26 12:30:52] ...trap/engines/engine.py.321: eval: epoch 7 | batch 390/510 [I 2022-10-26 12:30:52] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 12:30:52] ...trap/engines/engine.py.326: process: 0.28089 | load: 0.00015 [I 2022-10-26 12:30:55] ...trap/engines/engine.py.321: eval: epoch 7 | batch 400/510 [I 2022-10-26 12:30:55] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 12:30:55] ...trap/engines/engine.py.326: process: 0.28734 | load: 0.00021 [I 2022-10-26 12:30:57] ...trap/engines/engine.py.321: eval: epoch 7 | batch 410/510 [I 2022-10-26 12:30:57] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:28 [I 2022-10-26 12:30:57] ...trap/engines/engine.py.326: process: 0.28705 | load: 0.00018 [I 2022-10-26 12:31:00] ...trap/engines/engine.py.321: eval: epoch 7 | batch 420/510 [I 2022-10-26 12:31:00] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-26 12:31:00] ...trap/engines/engine.py.326: process: 0.28277 | load: 0.00015 [I 2022-10-26 12:31:03] ...trap/engines/engine.py.321: eval: epoch 7 | batch 430/510 [I 2022-10-26 12:31:03] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:22 [I 2022-10-26 12:31:03] ...trap/engines/engine.py.326: process: 0.28637 | load: 0.00018 [I 2022-10-26 12:31:06] ...trap/engines/engine.py.321: eval: epoch 7 | batch 440/510 [I 2022-10-26 12:31:06] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 12:31:06] ...trap/engines/engine.py.326: process: 0.29045 | load: 0.00018 [I 2022-10-26 12:31:09] ...trap/engines/engine.py.321: eval: epoch 7 | batch 450/510 [I 2022-10-26 12:31:09] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 12:31:09] ...trap/engines/engine.py.326: process: 0.29270 | load: 0.00016 [I 2022-10-26 12:31:12] ...trap/engines/engine.py.321: eval: epoch 7 | batch 460/510 [I 2022-10-26 12:31:12] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 12:31:12] ...trap/engines/engine.py.326: process: 0.29205 | load: 0.00016 [I 2022-10-26 12:31:15] ...trap/engines/engine.py.321: eval: epoch 7 | batch 470/510 [I 2022-10-26 12:31:15] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 12:31:15] ...trap/engines/engine.py.326: process: 0.28363 | load: 0.00019 [I 2022-10-26 12:31:18] ...trap/engines/engine.py.321: eval: epoch 7 | batch 480/510 [I 2022-10-26 12:31:18] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 12:31:18] ...trap/engines/engine.py.326: process: 0.28916 | load: 0.00021 [I 2022-10-26 12:31:20] ...trap/engines/engine.py.321: eval: epoch 7 | batch 490/510 [I 2022-10-26 12:31:20] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:05 [I 2022-10-26 12:31:20] ...trap/engines/engine.py.326: process: 0.29027 | load: 0.00016 [I 2022-10-26 12:31:23] ...trap/engines/engine.py.321: eval: epoch 7 | batch 500/510 [I 2022-10-26 12:31:23] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-26 12:31:23] ...trap/engines/engine.py.326: process: 0.28668 | load: 0.00023 [I 2022-10-26 12:31:26] ...trap/engines/engine.py.321: eval: epoch 7 | batch 510/510 [I 2022-10-26 12:31:26] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 12:31:26] ...trap/engines/engine.py.326: process: 0.27786 | load: 0.00014 [S 2022-10-26 12:31:27] ...trap/engines/engine.py.132: eval_epoch.epoch: 7 [I 2022-10-26 12:31:27] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.90 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.70 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.486500 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.783600 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.860100 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.491000 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.787000 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.862800 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.300000 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.458258 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008523 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006621 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011414 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017000 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010286 [S 2022-10-26 12:31:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009558 [I 2022-10-26 12:31:32] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-26 12:31:32] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 12:31:32] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 12:31:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 12:31:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 12:31:32] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 12:31:35] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 12:31:40] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 12:31:40] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 12:31:40] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 12:31:43] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 12:31:48] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 12:31:48] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 12:31:48] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 12:31:51] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 12:31:56] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 12:31:56] ...trap/engines/engine.py.163: Training model on trainset for epoch 8 [I 2022-10-26 12:31:56] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 12:31:59] ...trap/engines/engine.py.224: train: epoch 8 | batch 0/2304 [I 2022-10-26 12:31:59] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:17 [I 2022-10-26 12:31:59] ...trap/engines/engine.py.229: process: 2.79416 | load: 2.22848 [I 2022-10-26 12:31:59] ...trap/engines/engine.py.230: loss: 0.23693 [I 2022-10-26 12:31:59] ...optimizers/trijoint.py.151: total_norm: 2.232200 [I 2022-10-26 12:32:04] ...trap/engines/engine.py.224: train: epoch 8 | batch 10/2304 [I 2022-10-26 12:32:04] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:36 [I 2022-10-26 12:32:04] ...trap/engines/engine.py.229: process: 0.51199 | load: 0.00379 [I 2022-10-26 12:32:04] ...trap/engines/engine.py.230: loss: 0.20984 [I 2022-10-26 12:32:04] ...optimizers/trijoint.py.151: total_norm: 2.270966 [I 2022-10-26 12:32:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 20/2304 [I 2022-10-26 12:32:09] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:05 [I 2022-10-26 12:32:09] ...trap/engines/engine.py.229: process: 0.50413 | load: 0.00347 [I 2022-10-26 12:32:09] ...trap/engines/engine.py.230: loss: 0.22538 [I 2022-10-26 12:32:09] ...optimizers/trijoint.py.151: total_norm: 2.922129 [I 2022-10-26 12:32:14] ...trap/engines/engine.py.224: train: epoch 8 | batch 30/2304 [I 2022-10-26 12:32:14] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:18:57 [I 2022-10-26 12:32:14] ...trap/engines/engine.py.229: process: 0.48348 | load: 0.00304 [I 2022-10-26 12:32:14] ...trap/engines/engine.py.230: loss: 0.25666 [I 2022-10-26 12:32:14] ...optimizers/trijoint.py.151: total_norm: 3.727711 [I 2022-10-26 12:32:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 40/2304 [I 2022-10-26 12:32:19] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:54 [I 2022-10-26 12:32:19] ...trap/engines/engine.py.229: process: 0.50280 | load: 0.00312 [I 2022-10-26 12:32:19] ...trap/engines/engine.py.230: loss: 0.22637 [I 2022-10-26 12:32:19] ...optimizers/trijoint.py.151: total_norm: 2.921019 [I 2022-10-26 12:32:24] ...trap/engines/engine.py.224: train: epoch 8 | batch 50/2304 [I 2022-10-26 12:32:24] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:44 [I 2022-10-26 12:32:24] ...trap/engines/engine.py.229: process: 0.49842 | load: 0.00333 [I 2022-10-26 12:32:24] ...trap/engines/engine.py.230: loss: 0.22514 [I 2022-10-26 12:32:24] ...optimizers/trijoint.py.151: total_norm: 3.972777 [I 2022-10-26 12:32:29] ...trap/engines/engine.py.224: train: epoch 8 | batch 60/2304 [I 2022-10-26 12:32:29] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:43 [I 2022-10-26 12:32:29] ...trap/engines/engine.py.229: process: 0.49386 | load: 0.00282 [I 2022-10-26 12:32:29] ...trap/engines/engine.py.230: loss: 0.23009 [I 2022-10-26 12:32:29] ...optimizers/trijoint.py.151: total_norm: 3.293619 [I 2022-10-26 12:32:34] ...trap/engines/engine.py.224: train: epoch 8 | batch 70/2304 [I 2022-10-26 12:32:34] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:31 [I 2022-10-26 12:32:34] ...trap/engines/engine.py.229: process: 0.49186 | load: 0.00302 [I 2022-10-26 12:32:34] ...trap/engines/engine.py.230: loss: 0.28774 [I 2022-10-26 12:32:34] ...optimizers/trijoint.py.151: total_norm: 4.193452 [I 2022-10-26 12:32:39] ...trap/engines/engine.py.224: train: epoch 8 | batch 80/2304 [I 2022-10-26 12:32:39] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:33 [I 2022-10-26 12:32:39] ...trap/engines/engine.py.229: process: 0.50334 | load: 0.00270 [I 2022-10-26 12:32:39] ...trap/engines/engine.py.230: loss: 0.28534 [I 2022-10-26 12:32:39] ...optimizers/trijoint.py.151: total_norm: 3.048794 [I 2022-10-26 12:32:44] ...trap/engines/engine.py.224: train: epoch 8 | batch 90/2304 [I 2022-10-26 12:32:44] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:18:21 [I 2022-10-26 12:32:44] ...trap/engines/engine.py.229: process: 0.49300 | load: 0.00266 [I 2022-10-26 12:32:44] ...trap/engines/engine.py.230: loss: 0.36375 [I 2022-10-26 12:32:44] ...optimizers/trijoint.py.151: total_norm: 4.261285 [I 2022-10-26 12:32:49] ...trap/engines/engine.py.224: train: epoch 8 | batch 100/2304 [I 2022-10-26 12:32:49] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:18:19 [I 2022-10-26 12:32:49] ...trap/engines/engine.py.229: process: 0.50183 | load: 0.00276 [I 2022-10-26 12:32:49] ...trap/engines/engine.py.230: loss: 0.31559 [I 2022-10-26 12:32:49] ...optimizers/trijoint.py.151: total_norm: 4.231431 [I 2022-10-26 12:32:54] ...trap/engines/engine.py.224: train: epoch 8 | batch 110/2304 [I 2022-10-26 12:32:54] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:18:20 [I 2022-10-26 12:32:54] ...trap/engines/engine.py.229: process: 0.49301 | load: 0.00278 [I 2022-10-26 12:32:54] ...trap/engines/engine.py.230: loss: 0.18578 [I 2022-10-26 12:32:54] ...optimizers/trijoint.py.151: total_norm: 2.789191 [I 2022-10-26 12:32:59] ...trap/engines/engine.py.224: train: epoch 8 | batch 120/2304 [I 2022-10-26 12:32:59] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:07 [I 2022-10-26 12:32:59] ...trap/engines/engine.py.229: process: 0.49348 | load: 0.00256 [I 2022-10-26 12:32:59] ...trap/engines/engine.py.230: loss: 0.29709 [I 2022-10-26 12:32:59] ...optimizers/trijoint.py.151: total_norm: 3.219699 [I 2022-10-26 12:33:04] ...trap/engines/engine.py.224: train: epoch 8 | batch 130/2304 [I 2022-10-26 12:33:04] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:17:59 [I 2022-10-26 12:33:04] ...trap/engines/engine.py.229: process: 0.50094 | load: 0.00265 [I 2022-10-26 12:33:04] ...trap/engines/engine.py.230: loss: 0.30310 [I 2022-10-26 12:33:04] ...optimizers/trijoint.py.151: total_norm: 4.155420 [I 2022-10-26 12:33:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 140/2304 [I 2022-10-26 12:33:09] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:17:58 [I 2022-10-26 12:33:09] ...trap/engines/engine.py.229: process: 0.49917 | load: 0.00258 [I 2022-10-26 12:33:09] ...trap/engines/engine.py.230: loss: 0.28685 [I 2022-10-26 12:33:09] ...optimizers/trijoint.py.151: total_norm: 3.122946 [I 2022-10-26 12:33:14] ...trap/engines/engine.py.224: train: epoch 8 | batch 150/2304 [I 2022-10-26 12:33:14] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:17:54 [I 2022-10-26 12:33:14] ...trap/engines/engine.py.229: process: 0.49458 | load: 0.00257 [I 2022-10-26 12:33:14] ...trap/engines/engine.py.230: loss: 0.19089 [I 2022-10-26 12:33:14] ...optimizers/trijoint.py.151: total_norm: 2.832718 [I 2022-10-26 12:33:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 160/2304 [I 2022-10-26 12:33:19] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:17:57 [I 2022-10-26 12:33:19] ...trap/engines/engine.py.229: process: 0.51757 | load: 0.00257 [I 2022-10-26 12:33:19] ...trap/engines/engine.py.230: loss: 0.21601 [I 2022-10-26 12:33:19] ...optimizers/trijoint.py.151: total_norm: 3.746289 [I 2022-10-26 12:33:24] ...trap/engines/engine.py.224: train: epoch 8 | batch 170/2304 [I 2022-10-26 12:33:24] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:47 [I 2022-10-26 12:33:24] ...trap/engines/engine.py.229: process: 0.49818 | load: 0.00273 [I 2022-10-26 12:33:24] ...trap/engines/engine.py.230: loss: 0.21148 [I 2022-10-26 12:33:24] ...optimizers/trijoint.py.151: total_norm: 3.316182 [I 2022-10-26 12:33:29] ...trap/engines/engine.py.224: train: epoch 8 | batch 180/2304 [I 2022-10-26 12:33:29] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:17:41 [I 2022-10-26 12:33:29] ...trap/engines/engine.py.229: process: 0.48921 | load: 0.00244 [I 2022-10-26 12:33:29] ...trap/engines/engine.py.230: loss: 0.31227 [I 2022-10-26 12:33:29] ...optimizers/trijoint.py.151: total_norm: 4.728972 [I 2022-10-26 12:33:34] ...trap/engines/engine.py.224: train: epoch 8 | batch 190/2304 [I 2022-10-26 12:33:34] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:17:29 [I 2022-10-26 12:33:34] ...trap/engines/engine.py.229: process: 0.50312 | load: 0.00298 [I 2022-10-26 12:33:34] ...trap/engines/engine.py.230: loss: 0.30247 [I 2022-10-26 12:33:34] ...optimizers/trijoint.py.151: total_norm: 3.489239 [I 2022-10-26 12:33:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 200/2304 [I 2022-10-26 12:33:40] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:29 [I 2022-10-26 12:33:40] ...trap/engines/engine.py.229: process: 0.49503 | load: 0.00261 [I 2022-10-26 12:33:40] ...trap/engines/engine.py.230: loss: 0.19504 [I 2022-10-26 12:33:40] ...optimizers/trijoint.py.151: total_norm: 2.654005 [I 2022-10-26 12:33:44] ...trap/engines/engine.py.224: train: epoch 8 | batch 210/2304 [I 2022-10-26 12:33:44] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:19 [I 2022-10-26 12:33:44] ...trap/engines/engine.py.229: process: 0.49615 | load: 0.00261 [I 2022-10-26 12:33:44] ...trap/engines/engine.py.230: loss: 0.25939 [I 2022-10-26 12:33:44] ...optimizers/trijoint.py.151: total_norm: 4.042603 [I 2022-10-26 12:33:49] ...trap/engines/engine.py.224: train: epoch 8 | batch 220/2304 [I 2022-10-26 12:33:49] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:12 [I 2022-10-26 12:33:49] ...trap/engines/engine.py.229: process: 0.49401 | load: 0.00250 [I 2022-10-26 12:33:49] ...trap/engines/engine.py.230: loss: 0.22657 [I 2022-10-26 12:33:49] ...optimizers/trijoint.py.151: total_norm: 3.036465 [I 2022-10-26 12:33:54] ...trap/engines/engine.py.224: train: epoch 8 | batch 230/2304 [I 2022-10-26 12:33:54] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:19 [I 2022-10-26 12:33:54] ...trap/engines/engine.py.229: process: 0.50880 | load: 0.00254 [I 2022-10-26 12:33:54] ...trap/engines/engine.py.230: loss: 0.31209 [I 2022-10-26 12:33:54] ...optimizers/trijoint.py.151: total_norm: 3.685797 [I 2022-10-26 12:34:00] ...trap/engines/engine.py.224: train: epoch 8 | batch 240/2304 [I 2022-10-26 12:34:00] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:18 [I 2022-10-26 12:34:00] ...trap/engines/engine.py.229: process: 0.49246 | load: 0.00246 [I 2022-10-26 12:34:00] ...trap/engines/engine.py.230: loss: 0.27291 [I 2022-10-26 12:34:00] ...optimizers/trijoint.py.151: total_norm: 3.642296 [I 2022-10-26 12:34:05] ...trap/engines/engine.py.224: train: epoch 8 | batch 250/2304 [I 2022-10-26 12:34:05] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:16:58 [I 2022-10-26 12:34:05] ...trap/engines/engine.py.229: process: 0.49251 | load: 0.00261 [I 2022-10-26 12:34:05] ...trap/engines/engine.py.230: loss: 0.25864 [I 2022-10-26 12:34:05] ...optimizers/trijoint.py.151: total_norm: 3.503309 [I 2022-10-26 12:34:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 260/2304 [I 2022-10-26 12:34:09] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:16:53 [I 2022-10-26 12:34:09] ...trap/engines/engine.py.229: process: 0.49152 | load: 0.00263 [I 2022-10-26 12:34:09] ...trap/engines/engine.py.230: loss: 0.14047 [I 2022-10-26 12:34:10] ...optimizers/trijoint.py.151: total_norm: 2.197412 [I 2022-10-26 12:34:14] ...trap/engines/engine.py.224: train: epoch 8 | batch 270/2304 [I 2022-10-26 12:34:14] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:16:52 [I 2022-10-26 12:34:14] ...trap/engines/engine.py.229: process: 0.49278 | load: 0.00264 [I 2022-10-26 12:34:14] ...trap/engines/engine.py.230: loss: 0.35732 [I 2022-10-26 12:34:14] ...optimizers/trijoint.py.151: total_norm: 3.499274 [I 2022-10-26 12:34:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 280/2304 [I 2022-10-26 12:34:19] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:16:48 [I 2022-10-26 12:34:19] ...trap/engines/engine.py.229: process: 0.49952 | load: 0.00276 [I 2022-10-26 12:34:19] ...trap/engines/engine.py.230: loss: 0.25911 [I 2022-10-26 12:34:19] ...optimizers/trijoint.py.151: total_norm: 3.384321 [I 2022-10-26 12:34:25] ...trap/engines/engine.py.224: train: epoch 8 | batch 290/2304 [I 2022-10-26 12:34:25] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:16:52 [I 2022-10-26 12:34:25] ...trap/engines/engine.py.229: process: 0.49467 | load: 0.00254 [I 2022-10-26 12:34:25] ...trap/engines/engine.py.230: loss: 0.25571 [I 2022-10-26 12:34:25] ...optimizers/trijoint.py.151: total_norm: 3.250455 [I 2022-10-26 12:34:30] ...trap/engines/engine.py.224: train: epoch 8 | batch 300/2304 [I 2022-10-26 12:34:30] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:16:40 [I 2022-10-26 12:34:30] ...trap/engines/engine.py.229: process: 0.49989 | load: 0.00264 [I 2022-10-26 12:34:30] ...trap/engines/engine.py.230: loss: 0.13361 [I 2022-10-26 12:34:30] ...optimizers/trijoint.py.151: total_norm: 1.960130 [I 2022-10-26 12:34:35] ...trap/engines/engine.py.224: train: epoch 8 | batch 310/2304 [I 2022-10-26 12:34:35] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:16:35 [I 2022-10-26 12:34:35] ...trap/engines/engine.py.229: process: 0.49507 | load: 0.00254 [I 2022-10-26 12:34:35] ...trap/engines/engine.py.230: loss: 0.18648 [I 2022-10-26 12:34:35] ...optimizers/trijoint.py.151: total_norm: 3.034179 [I 2022-10-26 12:34:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 320/2304 [I 2022-10-26 12:34:40] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:16:31 [I 2022-10-26 12:34:40] ...trap/engines/engine.py.229: process: 0.49994 | load: 0.00252 [I 2022-10-26 12:34:40] ...trap/engines/engine.py.230: loss: 0.24759 [I 2022-10-26 12:34:40] ...optimizers/trijoint.py.151: total_norm: 4.104276 [I 2022-10-26 12:34:45] ...trap/engines/engine.py.224: train: epoch 8 | batch 330/2304 [I 2022-10-26 12:34:45] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:16:26 [I 2022-10-26 12:34:45] ...trap/engines/engine.py.229: process: 0.48122 | load: 0.00256 [I 2022-10-26 12:34:45] ...trap/engines/engine.py.230: loss: 0.26662 [I 2022-10-26 12:34:45] ...optimizers/trijoint.py.151: total_norm: 2.725955 [I 2022-10-26 12:34:50] ...trap/engines/engine.py.224: train: epoch 8 | batch 340/2304 [I 2022-10-26 12:34:50] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:16:18 [I 2022-10-26 12:34:50] ...trap/engines/engine.py.229: process: 0.50152 | load: 0.00254 [I 2022-10-26 12:34:50] ...trap/engines/engine.py.230: loss: 0.26507 [I 2022-10-26 12:34:50] ...optimizers/trijoint.py.151: total_norm: 3.558130 [I 2022-10-26 12:34:55] ...trap/engines/engine.py.224: train: epoch 8 | batch 350/2304 [I 2022-10-26 12:34:55] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:16:16 [I 2022-10-26 12:34:55] ...trap/engines/engine.py.229: process: 0.50228 | load: 0.00258 [I 2022-10-26 12:34:55] ...trap/engines/engine.py.230: loss: 0.23409 [I 2022-10-26 12:34:55] ...optimizers/trijoint.py.151: total_norm: 3.715462 [I 2022-10-26 12:35:00] ...trap/engines/engine.py.224: train: epoch 8 | batch 360/2304 [I 2022-10-26 12:35:00] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:16:19 [I 2022-10-26 12:35:00] ...trap/engines/engine.py.229: process: 0.50589 | load: 0.00254 [I 2022-10-26 12:35:00] ...trap/engines/engine.py.230: loss: 0.19257 [I 2022-10-26 12:35:00] ...optimizers/trijoint.py.151: total_norm: 2.596135 [I 2022-10-26 12:35:05] ...trap/engines/engine.py.224: train: epoch 8 | batch 370/2304 [I 2022-10-26 12:35:05] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:16:16 [I 2022-10-26 12:35:05] ...trap/engines/engine.py.229: process: 0.51086 | load: 0.00253 [I 2022-10-26 12:35:05] ...trap/engines/engine.py.230: loss: 0.20953 [I 2022-10-26 12:35:05] ...optimizers/trijoint.py.151: total_norm: 3.067398 [I 2022-10-26 12:35:10] ...trap/engines/engine.py.224: train: epoch 8 | batch 380/2304 [I 2022-10-26 12:35:10] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:16:08 [I 2022-10-26 12:35:10] ...trap/engines/engine.py.229: process: 0.51142 | load: 0.00258 [I 2022-10-26 12:35:10] ...trap/engines/engine.py.230: loss: 0.20002 [I 2022-10-26 12:35:10] ...optimizers/trijoint.py.151: total_norm: 3.065160 [I 2022-10-26 12:35:15] ...trap/engines/engine.py.224: train: epoch 8 | batch 390/2304 [I 2022-10-26 12:35:15] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:16:08 [I 2022-10-26 12:35:15] ...trap/engines/engine.py.229: process: 0.52589 | load: 0.00265 [I 2022-10-26 12:35:15] ...trap/engines/engine.py.230: loss: 0.21103 [I 2022-10-26 12:35:15] ...optimizers/trijoint.py.151: total_norm: 4.129525 [I 2022-10-26 12:35:20] ...trap/engines/engine.py.224: train: epoch 8 | batch 400/2304 [I 2022-10-26 12:35:20] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:16:26 [I 2022-10-26 12:35:20] ...trap/engines/engine.py.229: process: 0.53126 | load: 0.00277 [I 2022-10-26 12:35:20] ...trap/engines/engine.py.230: loss: 0.38207 [I 2022-10-26 12:35:20] ...optimizers/trijoint.py.151: total_norm: 4.641836 [I 2022-10-26 12:35:25] ...trap/engines/engine.py.224: train: epoch 8 | batch 410/2304 [I 2022-10-26 12:35:25] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:15:57 [I 2022-10-26 12:35:25] ...trap/engines/engine.py.229: process: 0.49529 | load: 0.00275 [I 2022-10-26 12:35:25] ...trap/engines/engine.py.230: loss: 0.22603 [I 2022-10-26 12:35:25] ...optimizers/trijoint.py.151: total_norm: 2.541773 [I 2022-10-26 12:35:30] ...trap/engines/engine.py.224: train: epoch 8 | batch 420/2304 [I 2022-10-26 12:35:30] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:15:50 [I 2022-10-26 12:35:30] ...trap/engines/engine.py.229: process: 0.50652 | load: 0.00279 [I 2022-10-26 12:35:30] ...trap/engines/engine.py.230: loss: 0.19983 [I 2022-10-26 12:35:30] ...optimizers/trijoint.py.151: total_norm: 2.919301 [I 2022-10-26 12:35:35] ...trap/engines/engine.py.224: train: epoch 8 | batch 430/2304 [I 2022-10-26 12:35:35] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:15:47 [I 2022-10-26 12:35:35] ...trap/engines/engine.py.229: process: 0.49869 | load: 0.00254 [I 2022-10-26 12:35:35] ...trap/engines/engine.py.230: loss: 0.21144 [I 2022-10-26 12:35:35] ...optimizers/trijoint.py.151: total_norm: 3.401485 [I 2022-10-26 12:35:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 440/2304 [I 2022-10-26 12:35:40] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:15:39 [I 2022-10-26 12:35:40] ...trap/engines/engine.py.229: process: 0.50128 | load: 0.00259 [I 2022-10-26 12:35:40] ...trap/engines/engine.py.230: loss: 0.30065 [I 2022-10-26 12:35:40] ...optimizers/trijoint.py.151: total_norm: 3.533259 [I 2022-10-26 12:35:45] ...trap/engines/engine.py.224: train: epoch 8 | batch 450/2304 [I 2022-10-26 12:35:45] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:15:28 [I 2022-10-26 12:35:45] ...trap/engines/engine.py.229: process: 0.50256 | load: 0.00266 [I 2022-10-26 12:35:45] ...trap/engines/engine.py.230: loss: 0.27376 [I 2022-10-26 12:35:45] ...optimizers/trijoint.py.151: total_norm: 4.428073 [I 2022-10-26 12:35:50] ...trap/engines/engine.py.224: train: epoch 8 | batch 460/2304 [I 2022-10-26 12:35:50] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:15:22 [I 2022-10-26 12:35:50] ...trap/engines/engine.py.229: process: 0.49272 | load: 0.00308 [I 2022-10-26 12:35:50] ...trap/engines/engine.py.230: loss: 0.24910 [I 2022-10-26 12:35:50] ...optimizers/trijoint.py.151: total_norm: 3.433122 [I 2022-10-26 12:35:55] ...trap/engines/engine.py.224: train: epoch 8 | batch 470/2304 [I 2022-10-26 12:35:55] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:15:23 [I 2022-10-26 12:35:55] ...trap/engines/engine.py.229: process: 0.50709 | load: 0.00262 [I 2022-10-26 12:35:55] ...trap/engines/engine.py.230: loss: 0.22558 [I 2022-10-26 12:35:55] ...optimizers/trijoint.py.151: total_norm: 3.245680 [I 2022-10-26 12:36:00] ...trap/engines/engine.py.224: train: epoch 8 | batch 480/2304 [I 2022-10-26 12:36:00] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:15:17 [I 2022-10-26 12:36:00] ...trap/engines/engine.py.229: process: 0.50476 | load: 0.00286 [I 2022-10-26 12:36:00] ...trap/engines/engine.py.230: loss: 0.26708 [I 2022-10-26 12:36:00] ...optimizers/trijoint.py.151: total_norm: 4.135664 [I 2022-10-26 12:36:05] ...trap/engines/engine.py.224: train: epoch 8 | batch 490/2304 [I 2022-10-26 12:36:05] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:15:21 [I 2022-10-26 12:36:05] ...trap/engines/engine.py.229: process: 0.51870 | load: 0.00278 [I 2022-10-26 12:36:05] ...trap/engines/engine.py.230: loss: 0.21411 [I 2022-10-26 12:36:05] ...optimizers/trijoint.py.151: total_norm: 2.839086 [I 2022-10-26 12:36:11] ...trap/engines/engine.py.224: train: epoch 8 | batch 500/2304 [I 2022-10-26 12:36:11] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:15:29 [I 2022-10-26 12:36:11] ...trap/engines/engine.py.229: process: 0.50141 | load: 0.00258 [I 2022-10-26 12:36:11] ...trap/engines/engine.py.230: loss: 0.31884 [I 2022-10-26 12:36:11] ...optimizers/trijoint.py.151: total_norm: 3.504434 [I 2022-10-26 12:36:16] ...trap/engines/engine.py.224: train: epoch 8 | batch 510/2304 [I 2022-10-26 12:36:16] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:15:04 [I 2022-10-26 12:36:16] ...trap/engines/engine.py.229: process: 0.50001 | load: 0.00255 [I 2022-10-26 12:36:16] ...trap/engines/engine.py.230: loss: 0.25664 [I 2022-10-26 12:36:16] ...optimizers/trijoint.py.151: total_norm: 2.514513 [I 2022-10-26 12:36:21] ...trap/engines/engine.py.224: train: epoch 8 | batch 520/2304 [I 2022-10-26 12:36:21] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:15:02 [I 2022-10-26 12:36:21] ...trap/engines/engine.py.229: process: 0.50155 | load: 0.00271 [I 2022-10-26 12:36:21] ...trap/engines/engine.py.230: loss: 0.21135 [I 2022-10-26 12:36:21] ...optimizers/trijoint.py.151: total_norm: 3.356058 [I 2022-10-26 12:36:26] ...trap/engines/engine.py.224: train: epoch 8 | batch 530/2304 [I 2022-10-26 12:36:26] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:15:00 [I 2022-10-26 12:36:26] ...trap/engines/engine.py.229: process: 0.50317 | load: 0.00257 [I 2022-10-26 12:36:26] ...trap/engines/engine.py.230: loss: 0.15900 [I 2022-10-26 12:36:26] ...optimizers/trijoint.py.151: total_norm: 2.570350 [I 2022-10-26 12:36:31] ...trap/engines/engine.py.224: train: epoch 8 | batch 540/2304 [I 2022-10-26 12:36:31] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:14:43 [I 2022-10-26 12:36:31] ...trap/engines/engine.py.229: process: 0.50455 | load: 0.00276 [I 2022-10-26 12:36:31] ...trap/engines/engine.py.230: loss: 0.22035 [I 2022-10-26 12:36:31] ...optimizers/trijoint.py.151: total_norm: 3.967416 [I 2022-10-26 12:36:36] ...trap/engines/engine.py.224: train: epoch 8 | batch 550/2304 [I 2022-10-26 12:36:36] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:14:37 [I 2022-10-26 12:36:36] ...trap/engines/engine.py.229: process: 0.49144 | load: 0.00264 [I 2022-10-26 12:36:36] ...trap/engines/engine.py.230: loss: 0.22660 [I 2022-10-26 12:36:36] ...optimizers/trijoint.py.151: total_norm: 3.185004 [I 2022-10-26 12:36:41] ...trap/engines/engine.py.224: train: epoch 8 | batch 560/2304 [I 2022-10-26 12:36:41] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:14:37 [I 2022-10-26 12:36:41] ...trap/engines/engine.py.229: process: 0.50259 | load: 0.00250 [I 2022-10-26 12:36:41] ...trap/engines/engine.py.230: loss: 0.19274 [I 2022-10-26 12:36:41] ...optimizers/trijoint.py.151: total_norm: 3.265001 [I 2022-10-26 12:36:46] ...trap/engines/engine.py.224: train: epoch 8 | batch 570/2304 [I 2022-10-26 12:36:46] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:14:43 [I 2022-10-26 12:36:46] ...trap/engines/engine.py.229: process: 0.49673 | load: 0.00274 [I 2022-10-26 12:36:46] ...trap/engines/engine.py.230: loss: 0.27236 [I 2022-10-26 12:36:46] ...optimizers/trijoint.py.151: total_norm: 3.321131 [I 2022-10-26 12:36:51] ...trap/engines/engine.py.224: train: epoch 8 | batch 580/2304 [I 2022-10-26 12:36:51] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:14:35 [I 2022-10-26 12:36:51] ...trap/engines/engine.py.229: process: 0.49623 | load: 0.00316 [I 2022-10-26 12:36:51] ...trap/engines/engine.py.230: loss: 0.17616 [I 2022-10-26 12:36:51] ...optimizers/trijoint.py.151: total_norm: 2.703763 [I 2022-10-26 12:36:56] ...trap/engines/engine.py.224: train: epoch 8 | batch 590/2304 [I 2022-10-26 12:36:56] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:14:22 [I 2022-10-26 12:36:56] ...trap/engines/engine.py.229: process: 0.50041 | load: 0.00263 [I 2022-10-26 12:36:56] ...trap/engines/engine.py.230: loss: 0.19789 [I 2022-10-26 12:36:56] ...optimizers/trijoint.py.151: total_norm: 3.082126 [I 2022-10-26 12:37:01] ...trap/engines/engine.py.224: train: epoch 8 | batch 600/2304 [I 2022-10-26 12:37:01] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:14:11 [I 2022-10-26 12:37:01] ...trap/engines/engine.py.229: process: 0.49760 | load: 0.00289 [I 2022-10-26 12:37:01] ...trap/engines/engine.py.230: loss: 0.29797 [I 2022-10-26 12:37:01] ...optimizers/trijoint.py.151: total_norm: 3.214234 [I 2022-10-26 12:37:06] ...trap/engines/engine.py.224: train: epoch 8 | batch 610/2304 [I 2022-10-26 12:37:06] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:14:07 [I 2022-10-26 12:37:06] ...trap/engines/engine.py.229: process: 0.50597 | load: 0.00259 [I 2022-10-26 12:37:06] ...trap/engines/engine.py.230: loss: 0.29570 [I 2022-10-26 12:37:06] ...optimizers/trijoint.py.151: total_norm: 4.317130 [I 2022-10-26 12:37:11] ...trap/engines/engine.py.224: train: epoch 8 | batch 620/2304 [I 2022-10-26 12:37:11] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:14:03 [I 2022-10-26 12:37:11] ...trap/engines/engine.py.229: process: 0.49971 | load: 0.00260 [I 2022-10-26 12:37:11] ...trap/engines/engine.py.230: loss: 0.16340 [I 2022-10-26 12:37:11] ...optimizers/trijoint.py.151: total_norm: 2.613465 [I 2022-10-26 12:37:16] ...trap/engines/engine.py.224: train: epoch 8 | batch 630/2304 [I 2022-10-26 12:37:16] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:13:56 [I 2022-10-26 12:37:16] ...trap/engines/engine.py.229: process: 0.50517 | load: 0.00253 [I 2022-10-26 12:37:16] ...trap/engines/engine.py.230: loss: 0.29512 [I 2022-10-26 12:37:16] ...optimizers/trijoint.py.151: total_norm: 3.482699 [I 2022-10-26 12:37:21] ...trap/engines/engine.py.224: train: epoch 8 | batch 640/2304 [I 2022-10-26 12:37:21] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:13:59 [I 2022-10-26 12:37:21] ...trap/engines/engine.py.229: process: 0.50243 | load: 0.00254 [I 2022-10-26 12:37:21] ...trap/engines/engine.py.230: loss: 0.25032 [I 2022-10-26 12:37:21] ...optimizers/trijoint.py.151: total_norm: 3.497311 [I 2022-10-26 12:37:26] ...trap/engines/engine.py.224: train: epoch 8 | batch 650/2304 [I 2022-10-26 12:37:26] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:13:56 [I 2022-10-26 12:37:26] ...trap/engines/engine.py.229: process: 0.50460 | load: 0.00260 [I 2022-10-26 12:37:26] ...trap/engines/engine.py.230: loss: 0.25993 [I 2022-10-26 12:37:26] ...optimizers/trijoint.py.151: total_norm: 3.890856 [I 2022-10-26 12:37:31] ...trap/engines/engine.py.224: train: epoch 8 | batch 660/2304 [I 2022-10-26 12:37:31] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:13:41 [I 2022-10-26 12:37:31] ...trap/engines/engine.py.229: process: 0.49105 | load: 0.00245 [I 2022-10-26 12:37:31] ...trap/engines/engine.py.230: loss: 0.23349 [I 2022-10-26 12:37:31] ...optimizers/trijoint.py.151: total_norm: 3.710550 [I 2022-10-26 12:37:36] ...trap/engines/engine.py.224: train: epoch 8 | batch 670/2304 [I 2022-10-26 12:37:36] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:13:36 [I 2022-10-26 12:37:36] ...trap/engines/engine.py.229: process: 0.49485 | load: 0.00251 [I 2022-10-26 12:37:36] ...trap/engines/engine.py.230: loss: 0.24278 [I 2022-10-26 12:37:36] ...optimizers/trijoint.py.151: total_norm: 3.427655 [I 2022-10-26 12:37:41] ...trap/engines/engine.py.224: train: epoch 8 | batch 680/2304 [I 2022-10-26 12:37:41] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:13:29 [I 2022-10-26 12:37:41] ...trap/engines/engine.py.229: process: 0.50215 | load: 0.00255 [I 2022-10-26 12:37:41] ...trap/engines/engine.py.230: loss: 0.23101 [I 2022-10-26 12:37:41] ...optimizers/trijoint.py.151: total_norm: 3.519356 [I 2022-10-26 12:37:46] ...trap/engines/engine.py.224: train: epoch 8 | batch 690/2304 [I 2022-10-26 12:37:46] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:13:49 [I 2022-10-26 12:37:46] ...trap/engines/engine.py.229: process: 0.50766 | load: 0.00250 [I 2022-10-26 12:37:46] ...trap/engines/engine.py.230: loss: 0.23291 [I 2022-10-26 12:37:46] ...optimizers/trijoint.py.151: total_norm: 4.200334 [I 2022-10-26 12:37:51] ...trap/engines/engine.py.224: train: epoch 8 | batch 700/2304 [I 2022-10-26 12:37:51] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:13:15 [I 2022-10-26 12:37:51] ...trap/engines/engine.py.229: process: 0.50660 | load: 0.00247 [I 2022-10-26 12:37:51] ...trap/engines/engine.py.230: loss: 0.31443 [I 2022-10-26 12:37:51] ...optimizers/trijoint.py.151: total_norm: 3.442735 [I 2022-10-26 12:37:56] ...trap/engines/engine.py.224: train: epoch 8 | batch 710/2304 [I 2022-10-26 12:37:56] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:13:05 [I 2022-10-26 12:37:56] ...trap/engines/engine.py.229: process: 0.48648 | load: 0.00249 [I 2022-10-26 12:37:56] ...trap/engines/engine.py.230: loss: 0.27102 [I 2022-10-26 12:37:56] ...optimizers/trijoint.py.151: total_norm: 5.108930 [I 2022-10-26 12:38:01] ...trap/engines/engine.py.224: train: epoch 8 | batch 720/2304 [I 2022-10-26 12:38:01] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:13:11 [I 2022-10-26 12:38:01] ...trap/engines/engine.py.229: process: 0.49608 | load: 0.00247 [I 2022-10-26 12:38:01] ...trap/engines/engine.py.230: loss: 0.23306 [I 2022-10-26 12:38:01] ...optimizers/trijoint.py.151: total_norm: 3.396973 [I 2022-10-26 12:38:06] ...trap/engines/engine.py.224: train: epoch 8 | batch 730/2304 [I 2022-10-26 12:38:06] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:13:06 [I 2022-10-26 12:38:06] ...trap/engines/engine.py.229: process: 0.50590 | load: 0.00240 [I 2022-10-26 12:38:06] ...trap/engines/engine.py.230: loss: 0.26498 [I 2022-10-26 12:38:06] ...optimizers/trijoint.py.151: total_norm: 3.784117 [I 2022-10-26 12:38:11] ...trap/engines/engine.py.224: train: epoch 8 | batch 740/2304 [I 2022-10-26 12:38:11] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:13:14 [I 2022-10-26 12:38:11] ...trap/engines/engine.py.229: process: 0.50925 | load: 0.00246 [I 2022-10-26 12:38:11] ...trap/engines/engine.py.230: loss: 0.25761 [I 2022-10-26 12:38:11] ...optimizers/trijoint.py.151: total_norm: 2.995194 [I 2022-10-26 12:38:16] ...trap/engines/engine.py.224: train: epoch 8 | batch 750/2304 [I 2022-10-26 12:38:16] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:13:02 [I 2022-10-26 12:38:16] ...trap/engines/engine.py.229: process: 0.52279 | load: 0.00249 [I 2022-10-26 12:38:16] ...trap/engines/engine.py.230: loss: 0.23039 [I 2022-10-26 12:38:16] ...optimizers/trijoint.py.151: total_norm: 3.836035 [I 2022-10-26 12:38:21] ...trap/engines/engine.py.224: train: epoch 8 | batch 760/2304 [I 2022-10-26 12:38:21] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:12:51 [I 2022-10-26 12:38:21] ...trap/engines/engine.py.229: process: 0.49581 | load: 0.00255 [I 2022-10-26 12:38:21] ...trap/engines/engine.py.230: loss: 0.24233 [I 2022-10-26 12:38:21] ...optimizers/trijoint.py.151: total_norm: 3.597780 [I 2022-10-26 12:38:26] ...trap/engines/engine.py.224: train: epoch 8 | batch 770/2304 [I 2022-10-26 12:38:26] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:12:42 [I 2022-10-26 12:38:26] ...trap/engines/engine.py.229: process: 0.49649 | load: 0.00256 [I 2022-10-26 12:38:26] ...trap/engines/engine.py.230: loss: 0.22670 [I 2022-10-26 12:38:26] ...optimizers/trijoint.py.151: total_norm: 3.655350 [I 2022-10-26 12:38:31] ...trap/engines/engine.py.224: train: epoch 8 | batch 780/2304 [I 2022-10-26 12:38:31] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:12:44 [I 2022-10-26 12:38:31] ...trap/engines/engine.py.229: process: 0.50787 | load: 0.00251 [I 2022-10-26 12:38:31] ...trap/engines/engine.py.230: loss: 0.30226 [I 2022-10-26 12:38:31] ...optimizers/trijoint.py.151: total_norm: 3.151946 [I 2022-10-26 12:38:36] ...trap/engines/engine.py.224: train: epoch 8 | batch 790/2304 [I 2022-10-26 12:38:36] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:12:41 [I 2022-10-26 12:38:36] ...trap/engines/engine.py.229: process: 0.51146 | load: 0.00252 [I 2022-10-26 12:38:36] ...trap/engines/engine.py.230: loss: 0.25401 [I 2022-10-26 12:38:36] ...optimizers/trijoint.py.151: total_norm: 3.062578 [I 2022-10-26 12:38:41] ...trap/engines/engine.py.224: train: epoch 8 | batch 800/2304 [I 2022-10-26 12:38:41] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:12:31 [I 2022-10-26 12:38:41] ...trap/engines/engine.py.229: process: 0.49054 | load: 0.00250 [I 2022-10-26 12:38:41] ...trap/engines/engine.py.230: loss: 0.28715 [I 2022-10-26 12:38:41] ...optimizers/trijoint.py.151: total_norm: 3.486438 [I 2022-10-26 12:38:46] ...trap/engines/engine.py.224: train: epoch 8 | batch 810/2304 [I 2022-10-26 12:38:46] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:12:29 [I 2022-10-26 12:38:46] ...trap/engines/engine.py.229: process: 0.50287 | load: 0.00258 [I 2022-10-26 12:38:46] ...trap/engines/engine.py.230: loss: 0.23925 [I 2022-10-26 12:38:46] ...optimizers/trijoint.py.151: total_norm: 3.982354 [I 2022-10-26 12:38:52] ...trap/engines/engine.py.224: train: epoch 8 | batch 820/2304 [I 2022-10-26 12:38:52] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:12:57 [I 2022-10-26 12:38:52] ...trap/engines/engine.py.229: process: 0.57991 | load: 0.00265 [I 2022-10-26 12:38:52] ...trap/engines/engine.py.230: loss: 0.19027 [I 2022-10-26 12:38:52] ...optimizers/trijoint.py.151: total_norm: 3.386052 [I 2022-10-26 12:38:57] ...trap/engines/engine.py.224: train: epoch 8 | batch 830/2304 [I 2022-10-26 12:38:57] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:12:30 [I 2022-10-26 12:38:57] ...trap/engines/engine.py.229: process: 0.48560 | load: 0.00254 [I 2022-10-26 12:38:57] ...trap/engines/engine.py.230: loss: 0.32554 [I 2022-10-26 12:38:57] ...optimizers/trijoint.py.151: total_norm: 4.424922 [I 2022-10-26 12:39:02] ...trap/engines/engine.py.224: train: epoch 8 | batch 840/2304 [I 2022-10-26 12:39:02] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:12:11 [I 2022-10-26 12:39:02] ...trap/engines/engine.py.229: process: 0.50386 | load: 0.00253 [I 2022-10-26 12:39:02] ...trap/engines/engine.py.230: loss: 0.15875 [I 2022-10-26 12:39:02] ...optimizers/trijoint.py.151: total_norm: 2.707757 [I 2022-10-26 12:39:07] ...trap/engines/engine.py.224: train: epoch 8 | batch 850/2304 [I 2022-10-26 12:39:07] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:12:02 [I 2022-10-26 12:39:07] ...trap/engines/engine.py.229: process: 0.49207 | load: 0.00251 [I 2022-10-26 12:39:07] ...trap/engines/engine.py.230: loss: 0.29168 [I 2022-10-26 12:39:07] ...optimizers/trijoint.py.151: total_norm: 2.871614 [I 2022-10-26 12:39:12] ...trap/engines/engine.py.224: train: epoch 8 | batch 860/2304 [I 2022-10-26 12:39:12] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:12:06 [I 2022-10-26 12:39:12] ...trap/engines/engine.py.229: process: 0.50113 | load: 0.00253 [I 2022-10-26 12:39:12] ...trap/engines/engine.py.230: loss: 0.19934 [I 2022-10-26 12:39:12] ...optimizers/trijoint.py.151: total_norm: 2.241147 [I 2022-10-26 12:39:17] ...trap/engines/engine.py.224: train: epoch 8 | batch 870/2304 [I 2022-10-26 12:39:17] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:11:57 [I 2022-10-26 12:39:17] ...trap/engines/engine.py.229: process: 0.49310 | load: 0.00248 [I 2022-10-26 12:39:17] ...trap/engines/engine.py.230: loss: 0.25124 [I 2022-10-26 12:39:17] ...optimizers/trijoint.py.151: total_norm: 3.371654 [I 2022-10-26 12:39:22] ...trap/engines/engine.py.224: train: epoch 8 | batch 880/2304 [I 2022-10-26 12:39:22] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:11:56 [I 2022-10-26 12:39:22] ...trap/engines/engine.py.229: process: 0.50592 | load: 0.00253 [I 2022-10-26 12:39:22] ...trap/engines/engine.py.230: loss: 0.24694 [I 2022-10-26 12:39:22] ...optimizers/trijoint.py.151: total_norm: 3.397806 [I 2022-10-26 12:39:27] ...trap/engines/engine.py.224: train: epoch 8 | batch 890/2304 [I 2022-10-26 12:39:27] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:11:54 [I 2022-10-26 12:39:27] ...trap/engines/engine.py.229: process: 0.52391 | load: 0.00288 [I 2022-10-26 12:39:27] ...trap/engines/engine.py.230: loss: 0.25318 [I 2022-10-26 12:39:27] ...optimizers/trijoint.py.151: total_norm: 2.931647 [I 2022-10-26 12:39:32] ...trap/engines/engine.py.224: train: epoch 8 | batch 900/2304 [I 2022-10-26 12:39:32] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:11:44 [I 2022-10-26 12:39:32] ...trap/engines/engine.py.229: process: 0.50683 | load: 0.00250 [I 2022-10-26 12:39:32] ...trap/engines/engine.py.230: loss: 0.18302 [I 2022-10-26 12:39:32] ...optimizers/trijoint.py.151: total_norm: 2.594114 [I 2022-10-26 12:39:37] ...trap/engines/engine.py.224: train: epoch 8 | batch 910/2304 [I 2022-10-26 12:39:37] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:11:45 [I 2022-10-26 12:39:37] ...trap/engines/engine.py.229: process: 0.52012 | load: 0.00272 [I 2022-10-26 12:39:37] ...trap/engines/engine.py.230: loss: 0.26571 [I 2022-10-26 12:39:37] ...optimizers/trijoint.py.151: total_norm: 3.094992 [I 2022-10-26 12:39:42] ...trap/engines/engine.py.224: train: epoch 8 | batch 920/2304 [I 2022-10-26 12:39:42] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:11:39 [I 2022-10-26 12:39:42] ...trap/engines/engine.py.229: process: 0.50866 | load: 0.00250 [I 2022-10-26 12:39:42] ...trap/engines/engine.py.230: loss: 0.33657 [I 2022-10-26 12:39:42] ...optimizers/trijoint.py.151: total_norm: 3.289369 [I 2022-10-26 12:39:47] ...trap/engines/engine.py.224: train: epoch 8 | batch 930/2304 [I 2022-10-26 12:39:47] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:11:24 [I 2022-10-26 12:39:47] ...trap/engines/engine.py.229: process: 0.49895 | load: 0.00249 [I 2022-10-26 12:39:47] ...trap/engines/engine.py.230: loss: 0.28958 [I 2022-10-26 12:39:47] ...optimizers/trijoint.py.151: total_norm: 3.054127 [I 2022-10-26 12:39:52] ...trap/engines/engine.py.224: train: epoch 8 | batch 940/2304 [I 2022-10-26 12:39:52] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:11:25 [I 2022-10-26 12:39:52] ...trap/engines/engine.py.229: process: 0.51245 | load: 0.00244 [I 2022-10-26 12:39:52] ...trap/engines/engine.py.230: loss: 0.25004 [I 2022-10-26 12:39:52] ...optimizers/trijoint.py.151: total_norm: 3.283857 [I 2022-10-26 12:39:57] ...trap/engines/engine.py.224: train: epoch 8 | batch 950/2304 [I 2022-10-26 12:39:57] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:11:15 [I 2022-10-26 12:39:57] ...trap/engines/engine.py.229: process: 0.50316 | load: 0.00258 [I 2022-10-26 12:39:57] ...trap/engines/engine.py.230: loss: 0.35616 [I 2022-10-26 12:39:57] ...optimizers/trijoint.py.151: total_norm: 4.253070 [I 2022-10-26 12:40:02] ...trap/engines/engine.py.224: train: epoch 8 | batch 960/2304 [I 2022-10-26 12:40:02] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:11:20 [I 2022-10-26 12:40:02] ...trap/engines/engine.py.229: process: 0.50110 | load: 0.00254 [I 2022-10-26 12:40:02] ...trap/engines/engine.py.230: loss: 0.21721 [I 2022-10-26 12:40:02] ...optimizers/trijoint.py.151: total_norm: 2.715354 [I 2022-10-26 12:40:07] ...trap/engines/engine.py.224: train: epoch 8 | batch 970/2304 [I 2022-10-26 12:40:07] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:11:10 [I 2022-10-26 12:40:07] ...trap/engines/engine.py.229: process: 0.50351 | load: 0.00248 [I 2022-10-26 12:40:07] ...trap/engines/engine.py.230: loss: 0.20615 [I 2022-10-26 12:40:07] ...optimizers/trijoint.py.151: total_norm: 3.122381 [I 2022-10-26 12:40:12] ...trap/engines/engine.py.224: train: epoch 8 | batch 980/2304 [I 2022-10-26 12:40:12] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:11:03 [I 2022-10-26 12:40:12] ...trap/engines/engine.py.229: process: 0.49707 | load: 0.00254 [I 2022-10-26 12:40:12] ...trap/engines/engine.py.230: loss: 0.25075 [I 2022-10-26 12:40:12] ...optimizers/trijoint.py.151: total_norm: 2.872501 [I 2022-10-26 12:40:17] ...trap/engines/engine.py.224: train: epoch 8 | batch 990/2304 [I 2022-10-26 12:40:17] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:10:51 [I 2022-10-26 12:40:17] ...trap/engines/engine.py.229: process: 0.48769 | load: 0.00254 [I 2022-10-26 12:40:17] ...trap/engines/engine.py.230: loss: 0.26908 [I 2022-10-26 12:40:17] ...optimizers/trijoint.py.151: total_norm: 4.427482 [I 2022-10-26 12:40:22] ...trap/engines/engine.py.224: train: epoch 8 | batch 1000/2304 [I 2022-10-26 12:40:22] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:10:46 [I 2022-10-26 12:40:22] ...trap/engines/engine.py.229: process: 0.49273 | load: 0.00252 [I 2022-10-26 12:40:22] ...trap/engines/engine.py.230: loss: 0.23446 [I 2022-10-26 12:40:22] ...optimizers/trijoint.py.151: total_norm: 3.648078 [I 2022-10-26 12:40:27] ...trap/engines/engine.py.224: train: epoch 8 | batch 1010/2304 [I 2022-10-26 12:40:27] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:10:50 [I 2022-10-26 12:40:27] ...trap/engines/engine.py.229: process: 0.50471 | load: 0.00245 [I 2022-10-26 12:40:27] ...trap/engines/engine.py.230: loss: 0.27218 [I 2022-10-26 12:40:27] ...optimizers/trijoint.py.151: total_norm: 3.902746 [I 2022-10-26 12:40:32] ...trap/engines/engine.py.224: train: epoch 8 | batch 1020/2304 [I 2022-10-26 12:40:32] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:10:39 [I 2022-10-26 12:40:32] ...trap/engines/engine.py.229: process: 0.50257 | load: 0.00248 [I 2022-10-26 12:40:32] ...trap/engines/engine.py.230: loss: 0.19593 [I 2022-10-26 12:40:32] ...optimizers/trijoint.py.151: total_norm: 2.659590 [I 2022-10-26 12:40:37] ...trap/engines/engine.py.224: train: epoch 8 | batch 1030/2304 [I 2022-10-26 12:40:37] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:10:38 [I 2022-10-26 12:40:37] ...trap/engines/engine.py.229: process: 0.50346 | load: 0.00250 [I 2022-10-26 12:40:37] ...trap/engines/engine.py.230: loss: 0.28252 [I 2022-10-26 12:40:37] ...optimizers/trijoint.py.151: total_norm: 4.352686 [I 2022-10-26 12:40:42] ...trap/engines/engine.py.224: train: epoch 8 | batch 1040/2304 [I 2022-10-26 12:40:42] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:10:31 [I 2022-10-26 12:40:42] ...trap/engines/engine.py.229: process: 0.49249 | load: 0.00251 [I 2022-10-26 12:40:42] ...trap/engines/engine.py.230: loss: 0.35297 [I 2022-10-26 12:40:42] ...optimizers/trijoint.py.151: total_norm: 4.047441 [I 2022-10-26 12:40:47] ...trap/engines/engine.py.224: train: epoch 8 | batch 1050/2304 [I 2022-10-26 12:40:47] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:10:31 [I 2022-10-26 12:40:47] ...trap/engines/engine.py.229: process: 0.51508 | load: 0.00259 [I 2022-10-26 12:40:47] ...trap/engines/engine.py.230: loss: 0.26341 [I 2022-10-26 12:40:47] ...optimizers/trijoint.py.151: total_norm: 3.981695 [I 2022-10-26 12:40:52] ...trap/engines/engine.py.224: train: epoch 8 | batch 1060/2304 [I 2022-10-26 12:40:52] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:10:23 [I 2022-10-26 12:40:52] ...trap/engines/engine.py.229: process: 0.49453 | load: 0.00253 [I 2022-10-26 12:40:52] ...trap/engines/engine.py.230: loss: 0.19374 [I 2022-10-26 12:40:52] ...optimizers/trijoint.py.151: total_norm: 2.806856 [I 2022-10-26 12:40:57] ...trap/engines/engine.py.224: train: epoch 8 | batch 1070/2304 [I 2022-10-26 12:40:57] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:10:17 [I 2022-10-26 12:40:57] ...trap/engines/engine.py.229: process: 0.49683 | load: 0.00247 [I 2022-10-26 12:40:57] ...trap/engines/engine.py.230: loss: 0.26729 [I 2022-10-26 12:40:57] ...optimizers/trijoint.py.151: total_norm: 3.596007 [I 2022-10-26 12:41:02] ...trap/engines/engine.py.224: train: epoch 8 | batch 1080/2304 [I 2022-10-26 12:41:02] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:10:12 [I 2022-10-26 12:41:02] ...trap/engines/engine.py.229: process: 0.50330 | load: 0.00244 [I 2022-10-26 12:41:02] ...trap/engines/engine.py.230: loss: 0.21345 [I 2022-10-26 12:41:02] ...optimizers/trijoint.py.151: total_norm: 2.965557 [I 2022-10-26 12:41:07] ...trap/engines/engine.py.224: train: epoch 8 | batch 1090/2304 [I 2022-10-26 12:41:07] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:10:06 [I 2022-10-26 12:41:07] ...trap/engines/engine.py.229: process: 0.49636 | load: 0.00243 [I 2022-10-26 12:41:07] ...trap/engines/engine.py.230: loss: 0.19601 [I 2022-10-26 12:41:07] ...optimizers/trijoint.py.151: total_norm: 2.827064 [I 2022-10-26 12:41:12] ...trap/engines/engine.py.224: train: epoch 8 | batch 1100/2304 [I 2022-10-26 12:41:12] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:10:02 [I 2022-10-26 12:41:12] ...trap/engines/engine.py.229: process: 0.50117 | load: 0.00245 [I 2022-10-26 12:41:12] ...trap/engines/engine.py.230: loss: 0.14818 [I 2022-10-26 12:41:12] ...optimizers/trijoint.py.151: total_norm: 2.408293 [I 2022-10-26 12:41:17] ...trap/engines/engine.py.224: train: epoch 8 | batch 1110/2304 [I 2022-10-26 12:41:17] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:10:02 [I 2022-10-26 12:41:17] ...trap/engines/engine.py.229: process: 0.52201 | load: 0.00253 [I 2022-10-26 12:41:17] ...trap/engines/engine.py.230: loss: 0.20747 [I 2022-10-26 12:41:17] ...optimizers/trijoint.py.151: total_norm: 2.718767 [I 2022-10-26 12:41:22] ...trap/engines/engine.py.224: train: epoch 8 | batch 1120/2304 [I 2022-10-26 12:41:22] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:09:53 [I 2022-10-26 12:41:22] ...trap/engines/engine.py.229: process: 0.49587 | load: 0.00248 [I 2022-10-26 12:41:22] ...trap/engines/engine.py.230: loss: 0.30642 [I 2022-10-26 12:41:22] ...optimizers/trijoint.py.151: total_norm: 3.437502 [I 2022-10-26 12:41:27] ...trap/engines/engine.py.224: train: epoch 8 | batch 1130/2304 [I 2022-10-26 12:41:27] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:09:46 [I 2022-10-26 12:41:27] ...trap/engines/engine.py.229: process: 0.49137 | load: 0.00251 [I 2022-10-26 12:41:27] ...trap/engines/engine.py.230: loss: 0.32992 [I 2022-10-26 12:41:27] ...optimizers/trijoint.py.151: total_norm: 4.300176 [I 2022-10-26 12:41:32] ...trap/engines/engine.py.224: train: epoch 8 | batch 1140/2304 [I 2022-10-26 12:41:32] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:09:40 [I 2022-10-26 12:41:32] ...trap/engines/engine.py.229: process: 0.49288 | load: 0.00243 [I 2022-10-26 12:41:32] ...trap/engines/engine.py.230: loss: 0.27482 [I 2022-10-26 12:41:32] ...optimizers/trijoint.py.151: total_norm: 4.087392 [I 2022-10-26 12:41:37] ...trap/engines/engine.py.224: train: epoch 8 | batch 1150/2304 [I 2022-10-26 12:41:37] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:09:32 [I 2022-10-26 12:41:37] ...trap/engines/engine.py.229: process: 0.49245 | load: 0.00250 [I 2022-10-26 12:41:37] ...trap/engines/engine.py.230: loss: 0.26858 [I 2022-10-26 12:41:37] ...optimizers/trijoint.py.151: total_norm: 4.125876 [I 2022-10-26 12:41:42] ...trap/engines/engine.py.224: train: epoch 8 | batch 1160/2304 [I 2022-10-26 12:41:42] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:09:33 [I 2022-10-26 12:41:42] ...trap/engines/engine.py.229: process: 0.50739 | load: 0.00246 [I 2022-10-26 12:41:42] ...trap/engines/engine.py.230: loss: 0.33006 [I 2022-10-26 12:41:42] ...optimizers/trijoint.py.151: total_norm: 4.213014 [I 2022-10-26 12:41:47] ...trap/engines/engine.py.224: train: epoch 8 | batch 1170/2304 [I 2022-10-26 12:41:47] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:09:25 [I 2022-10-26 12:41:47] ...trap/engines/engine.py.229: process: 0.49895 | load: 0.00244 [I 2022-10-26 12:41:47] ...trap/engines/engine.py.230: loss: 0.32935 [I 2022-10-26 12:41:47] ...optimizers/trijoint.py.151: total_norm: 3.466421 [I 2022-10-26 12:41:52] ...trap/engines/engine.py.224: train: epoch 8 | batch 1180/2304 [I 2022-10-26 12:41:52] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:09:22 [I 2022-10-26 12:41:52] ...trap/engines/engine.py.229: process: 0.50395 | load: 0.00248 [I 2022-10-26 12:41:52] ...trap/engines/engine.py.230: loss: 0.22003 [I 2022-10-26 12:41:52] ...optimizers/trijoint.py.151: total_norm: 4.037739 [I 2022-10-26 12:41:57] ...trap/engines/engine.py.224: train: epoch 8 | batch 1190/2304 [I 2022-10-26 12:41:57] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:09:15 [I 2022-10-26 12:41:57] ...trap/engines/engine.py.229: process: 0.49066 | load: 0.00248 [I 2022-10-26 12:41:57] ...trap/engines/engine.py.230: loss: 0.23497 [I 2022-10-26 12:41:57] ...optimizers/trijoint.py.151: total_norm: 3.813572 [I 2022-10-26 12:42:02] ...trap/engines/engine.py.224: train: epoch 8 | batch 1200/2304 [I 2022-10-26 12:42:02] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:09:11 [I 2022-10-26 12:42:02] ...trap/engines/engine.py.229: process: 0.49103 | load: 0.00242 [I 2022-10-26 12:42:02] ...trap/engines/engine.py.230: loss: 0.17397 [I 2022-10-26 12:42:02] ...optimizers/trijoint.py.151: total_norm: 2.792155 [I 2022-10-26 12:42:08] ...trap/engines/engine.py.224: train: epoch 8 | batch 1210/2304 [I 2022-10-26 12:42:08] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:09:14 [I 2022-10-26 12:42:08] ...trap/engines/engine.py.229: process: 0.51308 | load: 0.00245 [I 2022-10-26 12:42:08] ...trap/engines/engine.py.230: loss: 0.28759 [I 2022-10-26 12:42:08] ...optimizers/trijoint.py.151: total_norm: 4.071457 [I 2022-10-26 12:42:13] ...trap/engines/engine.py.224: train: epoch 8 | batch 1220/2304 [I 2022-10-26 12:42:13] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:09:06 [I 2022-10-26 12:42:13] ...trap/engines/engine.py.229: process: 0.50066 | load: 0.00247 [I 2022-10-26 12:42:13] ...trap/engines/engine.py.230: loss: 0.23069 [I 2022-10-26 12:42:13] ...optimizers/trijoint.py.151: total_norm: 3.499593 [I 2022-10-26 12:42:18] ...trap/engines/engine.py.224: train: epoch 8 | batch 1230/2304 [I 2022-10-26 12:42:18] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:08:53 [I 2022-10-26 12:42:18] ...trap/engines/engine.py.229: process: 0.49096 | load: 0.00245 [I 2022-10-26 12:42:18] ...trap/engines/engine.py.230: loss: 0.22602 [I 2022-10-26 12:42:18] ...optimizers/trijoint.py.151: total_norm: 3.556997 [I 2022-10-26 12:42:23] ...trap/engines/engine.py.224: train: epoch 8 | batch 1240/2304 [I 2022-10-26 12:42:23] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:08:53 [I 2022-10-26 12:42:23] ...trap/engines/engine.py.229: process: 0.50854 | load: 0.00249 [I 2022-10-26 12:42:23] ...trap/engines/engine.py.230: loss: 0.22168 [I 2022-10-26 12:42:23] ...optimizers/trijoint.py.151: total_norm: 3.373466 [I 2022-10-26 12:42:28] ...trap/engines/engine.py.224: train: epoch 8 | batch 1250/2304 [I 2022-10-26 12:42:28] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:08:44 [I 2022-10-26 12:42:28] ...trap/engines/engine.py.229: process: 0.50334 | load: 0.00249 [I 2022-10-26 12:42:28] ...trap/engines/engine.py.230: loss: 0.19456 [I 2022-10-26 12:42:28] ...optimizers/trijoint.py.151: total_norm: 2.486660 [I 2022-10-26 12:42:33] ...trap/engines/engine.py.224: train: epoch 8 | batch 1260/2304 [I 2022-10-26 12:42:33] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:08:37 [I 2022-10-26 12:42:33] ...trap/engines/engine.py.229: process: 0.49087 | load: 0.00246 [I 2022-10-26 12:42:33] ...trap/engines/engine.py.230: loss: 0.18690 [I 2022-10-26 12:42:33] ...optimizers/trijoint.py.151: total_norm: 2.640289 [I 2022-10-26 12:42:38] ...trap/engines/engine.py.224: train: epoch 8 | batch 1270/2304 [I 2022-10-26 12:42:38] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:08:35 [I 2022-10-26 12:42:38] ...trap/engines/engine.py.229: process: 0.50112 | load: 0.00245 [I 2022-10-26 12:42:38] ...trap/engines/engine.py.230: loss: 0.24060 [I 2022-10-26 12:42:38] ...optimizers/trijoint.py.151: total_norm: 3.040605 [I 2022-10-26 12:42:43] ...trap/engines/engine.py.224: train: epoch 8 | batch 1280/2304 [I 2022-10-26 12:42:43] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:08:27 [I 2022-10-26 12:42:43] ...trap/engines/engine.py.229: process: 0.49794 | load: 0.00242 [I 2022-10-26 12:42:43] ...trap/engines/engine.py.230: loss: 0.22059 [I 2022-10-26 12:42:43] ...optimizers/trijoint.py.151: total_norm: 3.184967 [I 2022-10-26 12:42:48] ...trap/engines/engine.py.224: train: epoch 8 | batch 1290/2304 [I 2022-10-26 12:42:48] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:08:23 [I 2022-10-26 12:42:48] ...trap/engines/engine.py.229: process: 0.49266 | load: 0.00243 [I 2022-10-26 12:42:48] ...trap/engines/engine.py.230: loss: 0.18851 [I 2022-10-26 12:42:48] ...optimizers/trijoint.py.151: total_norm: 3.433446 [I 2022-10-26 12:42:53] ...trap/engines/engine.py.224: train: epoch 8 | batch 1300/2304 [I 2022-10-26 12:42:53] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:08:20 [I 2022-10-26 12:42:53] ...trap/engines/engine.py.229: process: 0.50404 | load: 0.00245 [I 2022-10-26 12:42:53] ...trap/engines/engine.py.230: loss: 0.36340 [I 2022-10-26 12:42:53] ...optimizers/trijoint.py.151: total_norm: 4.455652 [I 2022-10-26 12:42:58] ...trap/engines/engine.py.224: train: epoch 8 | batch 1310/2304 [I 2022-10-26 12:42:58] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:08:22 [I 2022-10-26 12:42:58] ...trap/engines/engine.py.229: process: 0.51421 | load: 0.00245 [I 2022-10-26 12:42:58] ...trap/engines/engine.py.230: loss: 0.23994 [I 2022-10-26 12:42:58] ...optimizers/trijoint.py.151: total_norm: 5.341235 [I 2022-10-26 12:43:03] ...trap/engines/engine.py.224: train: epoch 8 | batch 1320/2304 [I 2022-10-26 12:43:03] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:08:16 [I 2022-10-26 12:43:03] ...trap/engines/engine.py.229: process: 0.50789 | load: 0.00242 [I 2022-10-26 12:43:03] ...trap/engines/engine.py.230: loss: 0.16700 [I 2022-10-26 12:43:03] ...optimizers/trijoint.py.151: total_norm: 2.541737 [I 2022-10-26 12:43:08] ...trap/engines/engine.py.224: train: epoch 8 | batch 1330/2304 [I 2022-10-26 12:43:08] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:08:02 [I 2022-10-26 12:43:08] ...trap/engines/engine.py.229: process: 0.48873 | load: 0.00242 [I 2022-10-26 12:43:08] ...trap/engines/engine.py.230: loss: 0.25197 [I 2022-10-26 12:43:08] ...optimizers/trijoint.py.151: total_norm: 3.503048 [I 2022-10-26 12:43:13] ...trap/engines/engine.py.224: train: epoch 8 | batch 1340/2304 [I 2022-10-26 12:43:13] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:08:00 [I 2022-10-26 12:43:13] ...trap/engines/engine.py.229: process: 0.49843 | load: 0.00245 [I 2022-10-26 12:43:13] ...trap/engines/engine.py.230: loss: 0.29106 [I 2022-10-26 12:43:13] ...optimizers/trijoint.py.151: total_norm: 3.202558 [I 2022-10-26 12:43:18] ...trap/engines/engine.py.224: train: epoch 8 | batch 1350/2304 [I 2022-10-26 12:43:18] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:07:58 [I 2022-10-26 12:43:18] ...trap/engines/engine.py.229: process: 0.50253 | load: 0.00255 [I 2022-10-26 12:43:18] ...trap/engines/engine.py.230: loss: 0.19438 [I 2022-10-26 12:43:18] ...optimizers/trijoint.py.151: total_norm: 2.622023 [I 2022-10-26 12:43:23] ...trap/engines/engine.py.224: train: epoch 8 | batch 1360/2304 [I 2022-10-26 12:43:23] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:07:51 [I 2022-10-26 12:43:23] ...trap/engines/engine.py.229: process: 0.49616 | load: 0.00247 [I 2022-10-26 12:43:23] ...trap/engines/engine.py.230: loss: 0.27224 [I 2022-10-26 12:43:23] ...optimizers/trijoint.py.151: total_norm: 3.014400 [I 2022-10-26 12:43:28] ...trap/engines/engine.py.224: train: epoch 8 | batch 1370/2304 [I 2022-10-26 12:43:28] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:07:48 [I 2022-10-26 12:43:28] ...trap/engines/engine.py.229: process: 0.49979 | load: 0.00244 [I 2022-10-26 12:43:28] ...trap/engines/engine.py.230: loss: 0.19360 [I 2022-10-26 12:43:28] ...optimizers/trijoint.py.151: total_norm: 3.263892 [I 2022-10-26 12:43:33] ...trap/engines/engine.py.224: train: epoch 8 | batch 1380/2304 [I 2022-10-26 12:43:33] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:07:38 [I 2022-10-26 12:43:33] ...trap/engines/engine.py.229: process: 0.49004 | load: 0.00244 [I 2022-10-26 12:43:33] ...trap/engines/engine.py.230: loss: 0.24799 [I 2022-10-26 12:43:33] ...optimizers/trijoint.py.151: total_norm: 3.150226 [I 2022-10-26 12:43:38] ...trap/engines/engine.py.224: train: epoch 8 | batch 1390/2304 [I 2022-10-26 12:43:38] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:07:36 [I 2022-10-26 12:43:38] ...trap/engines/engine.py.229: process: 0.50498 | load: 0.00247 [I 2022-10-26 12:43:38] ...trap/engines/engine.py.230: loss: 0.24423 [I 2022-10-26 12:43:38] ...optimizers/trijoint.py.151: total_norm: 3.530431 [I 2022-10-26 12:43:43] ...trap/engines/engine.py.224: train: epoch 8 | batch 1400/2304 [I 2022-10-26 12:43:43] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:07:27 [I 2022-10-26 12:43:43] ...trap/engines/engine.py.229: process: 0.48477 | load: 0.00246 [I 2022-10-26 12:43:43] ...trap/engines/engine.py.230: loss: 0.18567 [I 2022-10-26 12:43:43] ...optimizers/trijoint.py.151: total_norm: 3.694274 [I 2022-10-26 12:43:48] ...trap/engines/engine.py.224: train: epoch 8 | batch 1410/2304 [I 2022-10-26 12:43:48] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:07:27 [I 2022-10-26 12:43:48] ...trap/engines/engine.py.229: process: 0.50652 | load: 0.00241 [I 2022-10-26 12:43:48] ...trap/engines/engine.py.230: loss: 0.28499 [I 2022-10-26 12:43:48] ...optimizers/trijoint.py.151: total_norm: 4.124911 [I 2022-10-26 12:43:53] ...trap/engines/engine.py.224: train: epoch 8 | batch 1420/2304 [I 2022-10-26 12:43:53] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:07:16 [I 2022-10-26 12:43:53] ...trap/engines/engine.py.229: process: 0.49607 | load: 0.00265 [I 2022-10-26 12:43:53] ...trap/engines/engine.py.230: loss: 0.32046 [I 2022-10-26 12:43:53] ...optimizers/trijoint.py.151: total_norm: 3.382805 [I 2022-10-26 12:43:58] ...trap/engines/engine.py.224: train: epoch 8 | batch 1430/2304 [I 2022-10-26 12:43:58] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:07:12 [I 2022-10-26 12:43:58] ...trap/engines/engine.py.229: process: 0.49300 | load: 0.00245 [I 2022-10-26 12:43:58] ...trap/engines/engine.py.230: loss: 0.14924 [I 2022-10-26 12:43:58] ...optimizers/trijoint.py.151: total_norm: 2.317974 [I 2022-10-26 12:44:03] ...trap/engines/engine.py.224: train: epoch 8 | batch 1440/2304 [I 2022-10-26 12:44:03] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:07:11 [I 2022-10-26 12:44:03] ...trap/engines/engine.py.229: process: 0.49393 | load: 0.00245 [I 2022-10-26 12:44:03] ...trap/engines/engine.py.230: loss: 0.22845 [I 2022-10-26 12:44:03] ...optimizers/trijoint.py.151: total_norm: 4.549230 [I 2022-10-26 12:44:08] ...trap/engines/engine.py.224: train: epoch 8 | batch 1450/2304 [I 2022-10-26 12:44:08] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:07:07 [I 2022-10-26 12:44:08] ...trap/engines/engine.py.229: process: 0.50020 | load: 0.00241 [I 2022-10-26 12:44:08] ...trap/engines/engine.py.230: loss: 0.17441 [I 2022-10-26 12:44:08] ...optimizers/trijoint.py.151: total_norm: 3.209398 [I 2022-10-26 12:44:13] ...trap/engines/engine.py.224: train: epoch 8 | batch 1460/2304 [I 2022-10-26 12:44:13] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:07:02 [I 2022-10-26 12:44:13] ...trap/engines/engine.py.229: process: 0.50197 | load: 0.00259 [I 2022-10-26 12:44:13] ...trap/engines/engine.py.230: loss: 0.36759 [I 2022-10-26 12:44:13] ...optimizers/trijoint.py.151: total_norm: 3.843175 [I 2022-10-26 12:44:18] ...trap/engines/engine.py.224: train: epoch 8 | batch 1470/2304 [I 2022-10-26 12:44:18] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:06:54 [I 2022-10-26 12:44:18] ...trap/engines/engine.py.229: process: 0.49867 | load: 0.00248 [I 2022-10-26 12:44:18] ...trap/engines/engine.py.230: loss: 0.23730 [I 2022-10-26 12:44:18] ...optimizers/trijoint.py.151: total_norm: 3.463825 [I 2022-10-26 12:44:23] ...trap/engines/engine.py.224: train: epoch 8 | batch 1480/2304 [I 2022-10-26 12:44:23] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:06:50 [I 2022-10-26 12:44:23] ...trap/engines/engine.py.229: process: 0.49899 | load: 0.00246 [I 2022-10-26 12:44:23] ...trap/engines/engine.py.230: loss: 0.21247 [I 2022-10-26 12:44:23] ...optimizers/trijoint.py.151: total_norm: 2.622660 [I 2022-10-26 12:44:28] ...trap/engines/engine.py.224: train: epoch 8 | batch 1490/2304 [I 2022-10-26 12:44:28] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:06:45 [I 2022-10-26 12:44:28] ...trap/engines/engine.py.229: process: 0.48480 | load: 0.00246 [I 2022-10-26 12:44:28] ...trap/engines/engine.py.230: loss: 0.26681 [I 2022-10-26 12:44:28] ...optimizers/trijoint.py.151: total_norm: 2.787265 [I 2022-10-26 12:44:33] ...trap/engines/engine.py.224: train: epoch 8 | batch 1500/2304 [I 2022-10-26 12:44:33] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:06:39 [I 2022-10-26 12:44:33] ...trap/engines/engine.py.229: process: 0.49764 | load: 0.00243 [I 2022-10-26 12:44:33] ...trap/engines/engine.py.230: loss: 0.26570 [I 2022-10-26 12:44:33] ...optimizers/trijoint.py.151: total_norm: 3.454155 [I 2022-10-26 12:44:37] ...trap/engines/engine.py.224: train: epoch 8 | batch 1510/2304 [I 2022-10-26 12:44:37] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:06:33 [I 2022-10-26 12:44:37] ...trap/engines/engine.py.229: process: 0.49962 | load: 0.00244 [I 2022-10-26 12:44:37] ...trap/engines/engine.py.230: loss: 0.23680 [I 2022-10-26 12:44:37] ...optimizers/trijoint.py.151: total_norm: 3.201022 [I 2022-10-26 12:44:42] ...trap/engines/engine.py.224: train: epoch 8 | batch 1520/2304 [I 2022-10-26 12:44:42] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:06:30 [I 2022-10-26 12:44:42] ...trap/engines/engine.py.229: process: 0.49611 | load: 0.00243 [I 2022-10-26 12:44:42] ...trap/engines/engine.py.230: loss: 0.23467 [I 2022-10-26 12:44:42] ...optimizers/trijoint.py.151: total_norm: 3.041315 [I 2022-10-26 12:44:47] ...trap/engines/engine.py.224: train: epoch 8 | batch 1530/2304 [I 2022-10-26 12:44:47] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:06:25 [I 2022-10-26 12:44:47] ...trap/engines/engine.py.229: process: 0.50207 | load: 0.00245 [I 2022-10-26 12:44:47] ...trap/engines/engine.py.230: loss: 0.19824 [I 2022-10-26 12:44:47] ...optimizers/trijoint.py.151: total_norm: 2.770221 [I 2022-10-26 12:44:52] ...trap/engines/engine.py.224: train: epoch 8 | batch 1540/2304 [I 2022-10-26 12:44:52] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:06:20 [I 2022-10-26 12:44:52] ...trap/engines/engine.py.229: process: 0.50499 | load: 0.00260 [I 2022-10-26 12:44:52] ...trap/engines/engine.py.230: loss: 0.22527 [I 2022-10-26 12:44:52] ...optimizers/trijoint.py.151: total_norm: 3.010829 [I 2022-10-26 12:44:57] ...trap/engines/engine.py.224: train: epoch 8 | batch 1550/2304 [I 2022-10-26 12:44:57] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:06:13 [I 2022-10-26 12:44:57] ...trap/engines/engine.py.229: process: 0.50775 | load: 0.00247 [I 2022-10-26 12:44:57] ...trap/engines/engine.py.230: loss: 0.17134 [I 2022-10-26 12:44:57] ...optimizers/trijoint.py.151: total_norm: 3.442148 [I 2022-10-26 12:45:02] ...trap/engines/engine.py.224: train: epoch 8 | batch 1560/2304 [I 2022-10-26 12:45:02] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:06:10 [I 2022-10-26 12:45:02] ...trap/engines/engine.py.229: process: 0.50033 | load: 0.00247 [I 2022-10-26 12:45:02] ...trap/engines/engine.py.230: loss: 0.25069 [I 2022-10-26 12:45:02] ...optimizers/trijoint.py.151: total_norm: 3.925452 [I 2022-10-26 12:45:07] ...trap/engines/engine.py.224: train: epoch 8 | batch 1570/2304 [I 2022-10-26 12:45:08] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:06:13 [I 2022-10-26 12:45:08] ...trap/engines/engine.py.229: process: 0.49818 | load: 0.00247 [I 2022-10-26 12:45:08] ...trap/engines/engine.py.230: loss: 0.24774 [I 2022-10-26 12:45:08] ...optimizers/trijoint.py.151: total_norm: 3.855289 [I 2022-10-26 12:45:12] ...trap/engines/engine.py.224: train: epoch 8 | batch 1580/2304 [I 2022-10-26 12:45:12] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:06:00 [I 2022-10-26 12:45:12] ...trap/engines/engine.py.229: process: 0.49833 | load: 0.00243 [I 2022-10-26 12:45:12] ...trap/engines/engine.py.230: loss: 0.29348 [I 2022-10-26 12:45:12] ...optimizers/trijoint.py.151: total_norm: 2.834601 [I 2022-10-26 12:45:18] ...trap/engines/engine.py.224: train: epoch 8 | batch 1590/2304 [I 2022-10-26 12:45:18] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:06:02 [I 2022-10-26 12:45:18] ...trap/engines/engine.py.229: process: 0.50263 | load: 0.00244 [I 2022-10-26 12:45:18] ...trap/engines/engine.py.230: loss: 0.24004 [I 2022-10-26 12:45:18] ...optimizers/trijoint.py.151: total_norm: 2.610814 [I 2022-10-26 12:45:23] ...trap/engines/engine.py.224: train: epoch 8 | batch 1600/2304 [I 2022-10-26 12:45:23] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:05:50 [I 2022-10-26 12:45:23] ...trap/engines/engine.py.229: process: 0.50313 | load: 0.00243 [I 2022-10-26 12:45:23] ...trap/engines/engine.py.230: loss: 0.23164 [I 2022-10-26 12:45:23] ...optimizers/trijoint.py.151: total_norm: 3.067813 [I 2022-10-26 12:45:28] ...trap/engines/engine.py.224: train: epoch 8 | batch 1610/2304 [I 2022-10-26 12:45:28] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:05:47 [I 2022-10-26 12:45:28] ...trap/engines/engine.py.229: process: 0.50516 | load: 0.00244 [I 2022-10-26 12:45:28] ...trap/engines/engine.py.230: loss: 0.31998 [I 2022-10-26 12:45:28] ...optimizers/trijoint.py.151: total_norm: 3.988432 [I 2022-10-26 12:45:33] ...trap/engines/engine.py.224: train: epoch 8 | batch 1620/2304 [I 2022-10-26 12:45:33] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:05:43 [I 2022-10-26 12:45:33] ...trap/engines/engine.py.229: process: 0.50475 | load: 0.00248 [I 2022-10-26 12:45:33] ...trap/engines/engine.py.230: loss: 0.22242 [I 2022-10-26 12:45:33] ...optimizers/trijoint.py.151: total_norm: 3.065864 [I 2022-10-26 12:45:38] ...trap/engines/engine.py.224: train: epoch 8 | batch 1630/2304 [I 2022-10-26 12:45:38] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:05:39 [I 2022-10-26 12:45:38] ...trap/engines/engine.py.229: process: 0.50649 | load: 0.00247 [I 2022-10-26 12:45:38] ...trap/engines/engine.py.230: loss: 0.20287 [I 2022-10-26 12:45:38] ...optimizers/trijoint.py.151: total_norm: 3.390326 [I 2022-10-26 12:45:43] ...trap/engines/engine.py.224: train: epoch 8 | batch 1640/2304 [I 2022-10-26 12:45:43] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:05:32 [I 2022-10-26 12:45:43] ...trap/engines/engine.py.229: process: 0.50673 | load: 0.00261 [I 2022-10-26 12:45:43] ...trap/engines/engine.py.230: loss: 0.32410 [I 2022-10-26 12:45:43] ...optimizers/trijoint.py.151: total_norm: 4.357001 [I 2022-10-26 12:45:48] ...trap/engines/engine.py.224: train: epoch 8 | batch 1650/2304 [I 2022-10-26 12:45:48] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:05:26 [I 2022-10-26 12:45:48] ...trap/engines/engine.py.229: process: 0.49839 | load: 0.00252 [I 2022-10-26 12:45:48] ...trap/engines/engine.py.230: loss: 0.35130 [I 2022-10-26 12:45:48] ...optimizers/trijoint.py.151: total_norm: 4.046019 [I 2022-10-26 12:45:53] ...trap/engines/engine.py.224: train: epoch 8 | batch 1660/2304 [I 2022-10-26 12:45:53] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:05:21 [I 2022-10-26 12:45:53] ...trap/engines/engine.py.229: process: 0.50480 | load: 0.00246 [I 2022-10-26 12:45:53] ...trap/engines/engine.py.230: loss: 0.33404 [I 2022-10-26 12:45:53] ...optimizers/trijoint.py.151: total_norm: 3.594696 [I 2022-10-26 12:45:58] ...trap/engines/engine.py.224: train: epoch 8 | batch 1670/2304 [I 2022-10-26 12:45:58] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:05:19 [I 2022-10-26 12:45:58] ...trap/engines/engine.py.229: process: 0.49461 | load: 0.00249 [I 2022-10-26 12:45:58] ...trap/engines/engine.py.230: loss: 0.33930 [I 2022-10-26 12:45:58] ...optimizers/trijoint.py.151: total_norm: 3.844802 [I 2022-10-26 12:46:03] ...trap/engines/engine.py.224: train: epoch 8 | batch 1680/2304 [I 2022-10-26 12:46:03] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:05:13 [I 2022-10-26 12:46:03] ...trap/engines/engine.py.229: process: 0.50952 | load: 0.00247 [I 2022-10-26 12:46:03] ...trap/engines/engine.py.230: loss: 0.20531 [I 2022-10-26 12:46:03] ...optimizers/trijoint.py.151: total_norm: 2.986370 [I 2022-10-26 12:46:08] ...trap/engines/engine.py.224: train: epoch 8 | batch 1690/2304 [I 2022-10-26 12:46:08] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:05:11 [I 2022-10-26 12:46:08] ...trap/engines/engine.py.229: process: 0.51025 | load: 0.00244 [I 2022-10-26 12:46:08] ...trap/engines/engine.py.230: loss: 0.22502 [I 2022-10-26 12:46:08] ...optimizers/trijoint.py.151: total_norm: 3.701987 [I 2022-10-26 12:46:13] ...trap/engines/engine.py.224: train: epoch 8 | batch 1700/2304 [I 2022-10-26 12:46:13] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:05:05 [I 2022-10-26 12:46:13] ...trap/engines/engine.py.229: process: 0.50036 | load: 0.00265 [I 2022-10-26 12:46:13] ...trap/engines/engine.py.230: loss: 0.23251 [I 2022-10-26 12:46:13] ...optimizers/trijoint.py.151: total_norm: 2.987211 [I 2022-10-26 12:46:18] ...trap/engines/engine.py.224: train: epoch 8 | batch 1710/2304 [I 2022-10-26 12:46:18] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:04:59 [I 2022-10-26 12:46:18] ...trap/engines/engine.py.229: process: 0.50477 | load: 0.00245 [I 2022-10-26 12:46:18] ...trap/engines/engine.py.230: loss: 0.30461 [I 2022-10-26 12:46:18] ...optimizers/trijoint.py.151: total_norm: 3.537434 [I 2022-10-26 12:46:23] ...trap/engines/engine.py.224: train: epoch 8 | batch 1720/2304 [I 2022-10-26 12:46:23] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:04:55 [I 2022-10-26 12:46:23] ...trap/engines/engine.py.229: process: 0.50931 | load: 0.00260 [I 2022-10-26 12:46:23] ...trap/engines/engine.py.230: loss: 0.16370 [I 2022-10-26 12:46:23] ...optimizers/trijoint.py.151: total_norm: 2.762575 [I 2022-10-26 12:46:28] ...trap/engines/engine.py.224: train: epoch 8 | batch 1730/2304 [I 2022-10-26 12:46:28] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:04:49 [I 2022-10-26 12:46:28] ...trap/engines/engine.py.229: process: 0.52186 | load: 0.00254 [I 2022-10-26 12:46:28] ...trap/engines/engine.py.230: loss: 0.19882 [I 2022-10-26 12:46:28] ...optimizers/trijoint.py.151: total_norm: 4.058625 [I 2022-10-26 12:46:33] ...trap/engines/engine.py.224: train: epoch 8 | batch 1740/2304 [I 2022-10-26 12:46:33] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:04:43 [I 2022-10-26 12:46:33] ...trap/engines/engine.py.229: process: 0.48587 | load: 0.00249 [I 2022-10-26 12:46:33] ...trap/engines/engine.py.230: loss: 0.24108 [I 2022-10-26 12:46:33] ...optimizers/trijoint.py.151: total_norm: 3.225231 [I 2022-10-26 12:46:38] ...trap/engines/engine.py.224: train: epoch 8 | batch 1750/2304 [I 2022-10-26 12:46:38] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:04:37 [I 2022-10-26 12:46:38] ...trap/engines/engine.py.229: process: 0.50469 | load: 0.00247 [I 2022-10-26 12:46:38] ...trap/engines/engine.py.230: loss: 0.25341 [I 2022-10-26 12:46:38] ...optimizers/trijoint.py.151: total_norm: 3.107319 [I 2022-10-26 12:46:43] ...trap/engines/engine.py.224: train: epoch 8 | batch 1760/2304 [I 2022-10-26 12:46:43] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:04:33 [I 2022-10-26 12:46:43] ...trap/engines/engine.py.229: process: 0.50171 | load: 0.00248 [I 2022-10-26 12:46:43] ...trap/engines/engine.py.230: loss: 0.29669 [I 2022-10-26 12:46:43] ...optimizers/trijoint.py.151: total_norm: 3.641955 [I 2022-10-26 12:46:48] ...trap/engines/engine.py.224: train: epoch 8 | batch 1770/2304 [I 2022-10-26 12:46:48] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:04:29 [I 2022-10-26 12:46:48] ...trap/engines/engine.py.229: process: 0.50521 | load: 0.00246 [I 2022-10-26 12:46:48] ...trap/engines/engine.py.230: loss: 0.23841 [I 2022-10-26 12:46:48] ...optimizers/trijoint.py.151: total_norm: 2.942822 [I 2022-10-26 12:46:53] ...trap/engines/engine.py.224: train: epoch 8 | batch 1780/2304 [I 2022-10-26 12:46:53] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:04:24 [I 2022-10-26 12:46:53] ...trap/engines/engine.py.229: process: 0.50756 | load: 0.00244 [I 2022-10-26 12:46:53] ...trap/engines/engine.py.230: loss: 0.21605 [I 2022-10-26 12:46:53] ...optimizers/trijoint.py.151: total_norm: 3.293754 [I 2022-10-26 12:46:59] ...trap/engines/engine.py.224: train: epoch 8 | batch 1790/2304 [I 2022-10-26 12:46:59] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:04:21 [I 2022-10-26 12:46:59] ...trap/engines/engine.py.229: process: 0.53240 | load: 0.00252 [I 2022-10-26 12:46:59] ...trap/engines/engine.py.230: loss: 0.21047 [I 2022-10-26 12:46:59] ...optimizers/trijoint.py.151: total_norm: 3.542871 [I 2022-10-26 12:47:04] ...trap/engines/engine.py.224: train: epoch 8 | batch 1800/2304 [I 2022-10-26 12:47:04] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:04:13 [I 2022-10-26 12:47:04] ...trap/engines/engine.py.229: process: 0.49262 | load: 0.00264 [I 2022-10-26 12:47:04] ...trap/engines/engine.py.230: loss: 0.26566 [I 2022-10-26 12:47:04] ...optimizers/trijoint.py.151: total_norm: 3.069692 [I 2022-10-26 12:47:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 1810/2304 [I 2022-10-26 12:47:09] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:04:10 [I 2022-10-26 12:47:09] ...trap/engines/engine.py.229: process: 0.51531 | load: 0.00244 [I 2022-10-26 12:47:09] ...trap/engines/engine.py.230: loss: 0.23484 [I 2022-10-26 12:47:09] ...optimizers/trijoint.py.151: total_norm: 3.317444 [I 2022-10-26 12:47:14] ...trap/engines/engine.py.224: train: epoch 8 | batch 1820/2304 [I 2022-10-26 12:47:14] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:04:01 [I 2022-10-26 12:47:14] ...trap/engines/engine.py.229: process: 0.48964 | load: 0.00244 [I 2022-10-26 12:47:14] ...trap/engines/engine.py.230: loss: 0.22718 [I 2022-10-26 12:47:14] ...optimizers/trijoint.py.151: total_norm: 2.705817 [I 2022-10-26 12:47:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 1830/2304 [I 2022-10-26 12:47:19] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:03:56 [I 2022-10-26 12:47:19] ...trap/engines/engine.py.229: process: 0.49125 | load: 0.00246 [I 2022-10-26 12:47:19] ...trap/engines/engine.py.230: loss: 0.19919 [I 2022-10-26 12:47:19] ...optimizers/trijoint.py.151: total_norm: 2.786231 [I 2022-10-26 12:47:24] ...trap/engines/engine.py.224: train: epoch 8 | batch 1840/2304 [I 2022-10-26 12:47:24] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:03:52 [I 2022-10-26 12:47:24] ...trap/engines/engine.py.229: process: 0.50175 | load: 0.00251 [I 2022-10-26 12:47:24] ...trap/engines/engine.py.230: loss: 0.21321 [I 2022-10-26 12:47:24] ...optimizers/trijoint.py.151: total_norm: 3.644708 [I 2022-10-26 12:47:29] ...trap/engines/engine.py.224: train: epoch 8 | batch 1850/2304 [I 2022-10-26 12:47:29] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:03:50 [I 2022-10-26 12:47:29] ...trap/engines/engine.py.229: process: 0.52034 | load: 0.00247 [I 2022-10-26 12:47:29] ...trap/engines/engine.py.230: loss: 0.21228 [I 2022-10-26 12:47:29] ...optimizers/trijoint.py.151: total_norm: 2.659760 [I 2022-10-26 12:47:34] ...trap/engines/engine.py.224: train: epoch 8 | batch 1860/2304 [I 2022-10-26 12:47:34] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:03:44 [I 2022-10-26 12:47:34] ...trap/engines/engine.py.229: process: 0.50993 | load: 0.00252 [I 2022-10-26 12:47:34] ...trap/engines/engine.py.230: loss: 0.32889 [I 2022-10-26 12:47:34] ...optimizers/trijoint.py.151: total_norm: 3.387916 [I 2022-10-26 12:47:39] ...trap/engines/engine.py.224: train: epoch 8 | batch 1870/2304 [I 2022-10-26 12:47:39] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:03:43 [I 2022-10-26 12:47:39] ...trap/engines/engine.py.229: process: 0.51180 | load: 0.00245 [I 2022-10-26 12:47:39] ...trap/engines/engine.py.230: loss: 0.24486 [I 2022-10-26 12:47:39] ...optimizers/trijoint.py.151: total_norm: 2.951344 [I 2022-10-26 12:47:44] ...trap/engines/engine.py.224: train: epoch 8 | batch 1880/2304 [I 2022-10-26 12:47:44] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:03:32 [I 2022-10-26 12:47:44] ...trap/engines/engine.py.229: process: 0.49385 | load: 0.00245 [I 2022-10-26 12:47:44] ...trap/engines/engine.py.230: loss: 0.25553 [I 2022-10-26 12:47:44] ...optimizers/trijoint.py.151: total_norm: 3.053924 [I 2022-10-26 12:47:49] ...trap/engines/engine.py.224: train: epoch 8 | batch 1890/2304 [I 2022-10-26 12:47:49] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:03:27 [I 2022-10-26 12:47:49] ...trap/engines/engine.py.229: process: 0.50703 | load: 0.00245 [I 2022-10-26 12:47:49] ...trap/engines/engine.py.230: loss: 0.27858 [I 2022-10-26 12:47:49] ...optimizers/trijoint.py.151: total_norm: 3.652449 [I 2022-10-26 12:47:54] ...trap/engines/engine.py.224: train: epoch 8 | batch 1900/2304 [I 2022-10-26 12:47:54] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:03:23 [I 2022-10-26 12:47:54] ...trap/engines/engine.py.229: process: 0.50045 | load: 0.00244 [I 2022-10-26 12:47:54] ...trap/engines/engine.py.230: loss: 0.23734 [I 2022-10-26 12:47:54] ...optimizers/trijoint.py.151: total_norm: 3.484245 [I 2022-10-26 12:47:59] ...trap/engines/engine.py.224: train: epoch 8 | batch 1910/2304 [I 2022-10-26 12:47:59] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:03:18 [I 2022-10-26 12:47:59] ...trap/engines/engine.py.229: process: 0.51011 | load: 0.00243 [I 2022-10-26 12:47:59] ...trap/engines/engine.py.230: loss: 0.23534 [I 2022-10-26 12:47:59] ...optimizers/trijoint.py.151: total_norm: 3.682557 [I 2022-10-26 12:48:04] ...trap/engines/engine.py.224: train: epoch 8 | batch 1920/2304 [I 2022-10-26 12:48:04] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:03:14 [I 2022-10-26 12:48:04] ...trap/engines/engine.py.229: process: 0.49087 | load: 0.00245 [I 2022-10-26 12:48:04] ...trap/engines/engine.py.230: loss: 0.22691 [I 2022-10-26 12:48:04] ...optimizers/trijoint.py.151: total_norm: 2.958675 [I 2022-10-26 12:48:09] ...trap/engines/engine.py.224: train: epoch 8 | batch 1930/2304 [I 2022-10-26 12:48:09] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:03:08 [I 2022-10-26 12:48:09] ...trap/engines/engine.py.229: process: 0.49784 | load: 0.00247 [I 2022-10-26 12:48:09] ...trap/engines/engine.py.230: loss: 0.18441 [I 2022-10-26 12:48:09] ...optimizers/trijoint.py.151: total_norm: 2.681265 [I 2022-10-26 12:48:14] ...trap/engines/engine.py.224: train: epoch 8 | batch 1940/2304 [I 2022-10-26 12:48:14] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:03:03 [I 2022-10-26 12:48:14] ...trap/engines/engine.py.229: process: 0.50491 | load: 0.00243 [I 2022-10-26 12:48:14] ...trap/engines/engine.py.230: loss: 0.26109 [I 2022-10-26 12:48:14] ...optimizers/trijoint.py.151: total_norm: 3.781959 [I 2022-10-26 12:48:19] ...trap/engines/engine.py.224: train: epoch 8 | batch 1950/2304 [I 2022-10-26 12:48:19] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:02:57 [I 2022-10-26 12:48:19] ...trap/engines/engine.py.229: process: 0.51444 | load: 0.00254 [I 2022-10-26 12:48:19] ...trap/engines/engine.py.230: loss: 0.23247 [I 2022-10-26 12:48:19] ...optimizers/trijoint.py.151: total_norm: 3.046219 [I 2022-10-26 12:48:24] ...trap/engines/engine.py.224: train: epoch 8 | batch 1960/2304 [I 2022-10-26 12:48:24] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:02:53 [I 2022-10-26 12:48:24] ...trap/engines/engine.py.229: process: 0.49446 | load: 0.00249 [I 2022-10-26 12:48:24] ...trap/engines/engine.py.230: loss: 0.23943 [I 2022-10-26 12:48:24] ...optimizers/trijoint.py.151: total_norm: 3.318426 [I 2022-10-26 12:48:29] ...trap/engines/engine.py.224: train: epoch 8 | batch 1970/2304 [I 2022-10-26 12:48:29] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:02:47 [I 2022-10-26 12:48:29] ...trap/engines/engine.py.229: process: 0.51011 | load: 0.00244 [I 2022-10-26 12:48:29] ...trap/engines/engine.py.230: loss: 0.47073 [I 2022-10-26 12:48:29] ...optimizers/trijoint.py.151: total_norm: 3.641179 [I 2022-10-26 12:48:35] ...trap/engines/engine.py.224: train: epoch 8 | batch 1980/2304 [I 2022-10-26 12:48:35] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:02:41 [I 2022-10-26 12:48:35] ...trap/engines/engine.py.229: process: 0.49428 | load: 0.00244 [I 2022-10-26 12:48:35] ...trap/engines/engine.py.230: loss: 0.24555 [I 2022-10-26 12:48:35] ...optimizers/trijoint.py.151: total_norm: 2.757234 [I 2022-10-26 12:48:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 1990/2304 [I 2022-10-26 12:48:40] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:02:40 [I 2022-10-26 12:48:40] ...trap/engines/engine.py.229: process: 0.56633 | load: 0.00248 [I 2022-10-26 12:48:40] ...trap/engines/engine.py.230: loss: 0.23700 [I 2022-10-26 12:48:40] ...optimizers/trijoint.py.151: total_norm: 3.056141 [I 2022-10-26 12:48:45] ...trap/engines/engine.py.224: train: epoch 8 | batch 2000/2304 [I 2022-10-26 12:48:45] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:02:35 [I 2022-10-26 12:48:45] ...trap/engines/engine.py.229: process: 0.50566 | load: 0.00243 [I 2022-10-26 12:48:45] ...trap/engines/engine.py.230: loss: 0.20836 [I 2022-10-26 12:48:45] ...optimizers/trijoint.py.151: total_norm: 2.973316 [I 2022-10-26 12:48:50] ...trap/engines/engine.py.224: train: epoch 8 | batch 2010/2304 [I 2022-10-26 12:48:50] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:02:29 [I 2022-10-26 12:48:50] ...trap/engines/engine.py.229: process: 0.52012 | load: 0.00246 [I 2022-10-26 12:48:50] ...trap/engines/engine.py.230: loss: 0.25377 [I 2022-10-26 12:48:50] ...optimizers/trijoint.py.151: total_norm: 3.837510 [I 2022-10-26 12:48:55] ...trap/engines/engine.py.224: train: epoch 8 | batch 2020/2304 [I 2022-10-26 12:48:55] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:02:22 [I 2022-10-26 12:48:55] ...trap/engines/engine.py.229: process: 0.51142 | load: 0.00252 [I 2022-10-26 12:48:55] ...trap/engines/engine.py.230: loss: 0.23399 [I 2022-10-26 12:48:55] ...optimizers/trijoint.py.151: total_norm: 2.955562 [I 2022-10-26 12:49:00] ...trap/engines/engine.py.224: train: epoch 8 | batch 2030/2304 [I 2022-10-26 12:49:00] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:02:17 [I 2022-10-26 12:49:00] ...trap/engines/engine.py.229: process: 0.50351 | load: 0.00253 [I 2022-10-26 12:49:00] ...trap/engines/engine.py.230: loss: 0.28840 [I 2022-10-26 12:49:00] ...optimizers/trijoint.py.151: total_norm: 3.100956 [I 2022-10-26 12:49:05] ...trap/engines/engine.py.224: train: epoch 8 | batch 2040/2304 [I 2022-10-26 12:49:05] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:02:12 [I 2022-10-26 12:49:05] ...trap/engines/engine.py.229: process: 0.50254 | load: 0.00247 [I 2022-10-26 12:49:05] ...trap/engines/engine.py.230: loss: 0.20880 [I 2022-10-26 12:49:05] ...optimizers/trijoint.py.151: total_norm: 2.990699 [I 2022-10-26 12:49:10] ...trap/engines/engine.py.224: train: epoch 8 | batch 2050/2304 [I 2022-10-26 12:49:10] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:02:07 [I 2022-10-26 12:49:10] ...trap/engines/engine.py.229: process: 0.51343 | load: 0.00245 [I 2022-10-26 12:49:10] ...trap/engines/engine.py.230: loss: 0.24224 [I 2022-10-26 12:49:10] ...optimizers/trijoint.py.151: total_norm: 4.063854 [I 2022-10-26 12:49:15] ...trap/engines/engine.py.224: train: epoch 8 | batch 2060/2304 [I 2022-10-26 12:49:15] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:02:03 [I 2022-10-26 12:49:15] ...trap/engines/engine.py.229: process: 0.52158 | load: 0.00253 [I 2022-10-26 12:49:15] ...trap/engines/engine.py.230: loss: 0.20972 [I 2022-10-26 12:49:15] ...optimizers/trijoint.py.151: total_norm: 3.508680 [I 2022-10-26 12:49:20] ...trap/engines/engine.py.224: train: epoch 8 | batch 2070/2304 [I 2022-10-26 12:49:20] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:01:58 [I 2022-10-26 12:49:20] ...trap/engines/engine.py.229: process: 0.50526 | load: 0.00247 [I 2022-10-26 12:49:20] ...trap/engines/engine.py.230: loss: 0.18441 [I 2022-10-26 12:49:20] ...optimizers/trijoint.py.151: total_norm: 3.362209 [I 2022-10-26 12:49:25] ...trap/engines/engine.py.224: train: epoch 8 | batch 2080/2304 [I 2022-10-26 12:49:25] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:01:52 [I 2022-10-26 12:49:25] ...trap/engines/engine.py.229: process: 0.50014 | load: 0.00242 [I 2022-10-26 12:49:25] ...trap/engines/engine.py.230: loss: 0.25544 [I 2022-10-26 12:49:25] ...optimizers/trijoint.py.151: total_norm: 3.269608 [I 2022-10-26 12:49:30] ...trap/engines/engine.py.224: train: epoch 8 | batch 2090/2304 [I 2022-10-26 12:49:30] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:01:47 [I 2022-10-26 12:49:30] ...trap/engines/engine.py.229: process: 0.51524 | load: 0.00247 [I 2022-10-26 12:49:30] ...trap/engines/engine.py.230: loss: 0.22578 [I 2022-10-26 12:49:30] ...optimizers/trijoint.py.151: total_norm: 3.960510 [I 2022-10-26 12:49:35] ...trap/engines/engine.py.224: train: epoch 8 | batch 2100/2304 [I 2022-10-26 12:49:35] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:01:41 [I 2022-10-26 12:49:35] ...trap/engines/engine.py.229: process: 0.50207 | load: 0.00254 [I 2022-10-26 12:49:35] ...trap/engines/engine.py.230: loss: 0.26901 [I 2022-10-26 12:49:35] ...optimizers/trijoint.py.151: total_norm: 3.558884 [I 2022-10-26 12:49:40] ...trap/engines/engine.py.224: train: epoch 8 | batch 2110/2304 [I 2022-10-26 12:49:40] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:01:37 [I 2022-10-26 12:49:40] ...trap/engines/engine.py.229: process: 0.50188 | load: 0.00243 [I 2022-10-26 12:49:40] ...trap/engines/engine.py.230: loss: 0.22181 [I 2022-10-26 12:49:40] ...optimizers/trijoint.py.151: total_norm: 2.428630 [I 2022-10-26 12:49:45] ...trap/engines/engine.py.224: train: epoch 8 | batch 2120/2304 [I 2022-10-26 12:49:45] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:01:34 [I 2022-10-26 12:49:45] ...trap/engines/engine.py.229: process: 0.50476 | load: 0.00248 [I 2022-10-26 12:49:45] ...trap/engines/engine.py.230: loss: 0.23519 [I 2022-10-26 12:49:45] ...optimizers/trijoint.py.151: total_norm: 2.666446 [I 2022-10-26 12:49:50] ...trap/engines/engine.py.224: train: epoch 8 | batch 2130/2304 [I 2022-10-26 12:49:50] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:01:27 [I 2022-10-26 12:49:50] ...trap/engines/engine.py.229: process: 0.50272 | load: 0.00249 [I 2022-10-26 12:49:50] ...trap/engines/engine.py.230: loss: 0.20699 [I 2022-10-26 12:49:50] ...optimizers/trijoint.py.151: total_norm: 2.694709 [I 2022-10-26 12:49:55] ...trap/engines/engine.py.224: train: epoch 8 | batch 2140/2304 [I 2022-10-26 12:49:55] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:01:21 [I 2022-10-26 12:49:55] ...trap/engines/engine.py.229: process: 0.49450 | load: 0.00265 [I 2022-10-26 12:49:55] ...trap/engines/engine.py.230: loss: 0.27432 [I 2022-10-26 12:49:55] ...optimizers/trijoint.py.151: total_norm: 3.972617 [I 2022-10-26 12:50:00] ...trap/engines/engine.py.224: train: epoch 8 | batch 2150/2304 [I 2022-10-26 12:50:00] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:01:16 [I 2022-10-26 12:50:00] ...trap/engines/engine.py.229: process: 0.49651 | load: 0.00250 [I 2022-10-26 12:50:00] ...trap/engines/engine.py.230: loss: 0.22918 [I 2022-10-26 12:50:00] ...optimizers/trijoint.py.151: total_norm: 2.690997 [I 2022-10-26 12:50:05] ...trap/engines/engine.py.224: train: epoch 8 | batch 2160/2304 [I 2022-10-26 12:50:05] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:01:11 [I 2022-10-26 12:50:05] ...trap/engines/engine.py.229: process: 0.49396 | load: 0.00258 [I 2022-10-26 12:50:05] ...trap/engines/engine.py.230: loss: 0.35849 [I 2022-10-26 12:50:05] ...optimizers/trijoint.py.151: total_norm: 4.126424 [I 2022-10-26 12:50:10] ...trap/engines/engine.py.224: train: epoch 8 | batch 2170/2304 [I 2022-10-26 12:50:10] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:01:07 [I 2022-10-26 12:50:10] ...trap/engines/engine.py.229: process: 0.49715 | load: 0.00259 [I 2022-10-26 12:50:10] ...trap/engines/engine.py.230: loss: 0.25277 [I 2022-10-26 12:50:10] ...optimizers/trijoint.py.151: total_norm: 4.836095 [I 2022-10-26 12:50:15] ...trap/engines/engine.py.224: train: epoch 8 | batch 2180/2304 [I 2022-10-26 12:50:15] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:01:01 [I 2022-10-26 12:50:15] ...trap/engines/engine.py.229: process: 0.50904 | load: 0.00256 [I 2022-10-26 12:50:15] ...trap/engines/engine.py.230: loss: 0.23236 [I 2022-10-26 12:50:15] ...optimizers/trijoint.py.151: total_norm: 3.873910 [I 2022-10-26 12:50:20] ...trap/engines/engine.py.224: train: epoch 8 | batch 2190/2304 [I 2022-10-26 12:50:20] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:00:56 [I 2022-10-26 12:50:20] ...trap/engines/engine.py.229: process: 0.49999 | load: 0.00245 [I 2022-10-26 12:50:20] ...trap/engines/engine.py.230: loss: 0.24570 [I 2022-10-26 12:50:20] ...optimizers/trijoint.py.151: total_norm: 3.893400 [I 2022-10-26 12:50:26] ...trap/engines/engine.py.224: train: epoch 8 | batch 2200/2304 [I 2022-10-26 12:50:26] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:00:52 [I 2022-10-26 12:50:26] ...trap/engines/engine.py.229: process: 0.49909 | load: 0.00246 [I 2022-10-26 12:50:26] ...trap/engines/engine.py.230: loss: 0.20328 [I 2022-10-26 12:50:26] ...optimizers/trijoint.py.151: total_norm: 3.336384 [I 2022-10-26 12:50:31] ...trap/engines/engine.py.224: train: epoch 8 | batch 2210/2304 [I 2022-10-26 12:50:31] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:00:47 [I 2022-10-26 12:50:31] ...trap/engines/engine.py.229: process: 0.50041 | load: 0.00246 [I 2022-10-26 12:50:31] ...trap/engines/engine.py.230: loss: 0.16229 [I 2022-10-26 12:50:31] ...optimizers/trijoint.py.151: total_norm: 2.362967 [I 2022-10-26 12:50:36] ...trap/engines/engine.py.224: train: epoch 8 | batch 2220/2304 [I 2022-10-26 12:50:36] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:00:41 [I 2022-10-26 12:50:36] ...trap/engines/engine.py.229: process: 0.49623 | load: 0.00245 [I 2022-10-26 12:50:36] ...trap/engines/engine.py.230: loss: 0.29102 [I 2022-10-26 12:50:36] ...optimizers/trijoint.py.151: total_norm: 3.772019 [I 2022-10-26 12:50:41] ...trap/engines/engine.py.224: train: epoch 8 | batch 2230/2304 [I 2022-10-26 12:50:41] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:00:36 [I 2022-10-26 12:50:41] ...trap/engines/engine.py.229: process: 0.49618 | load: 0.00244 [I 2022-10-26 12:50:41] ...trap/engines/engine.py.230: loss: 0.30220 [I 2022-10-26 12:50:41] ...optimizers/trijoint.py.151: total_norm: 4.145209 [I 2022-10-26 12:50:46] ...trap/engines/engine.py.224: train: epoch 8 | batch 2240/2304 [I 2022-10-26 12:50:46] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:00:32 [I 2022-10-26 12:50:46] ...trap/engines/engine.py.229: process: 0.50576 | load: 0.00245 [I 2022-10-26 12:50:46] ...trap/engines/engine.py.230: loss: 0.25308 [I 2022-10-26 12:50:46] ...optimizers/trijoint.py.151: total_norm: 3.521398 [I 2022-10-26 12:50:51] ...trap/engines/engine.py.224: train: epoch 8 | batch 2250/2304 [I 2022-10-26 12:50:51] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:00:27 [I 2022-10-26 12:50:51] ...trap/engines/engine.py.229: process: 0.49286 | load: 0.00243 [I 2022-10-26 12:50:51] ...trap/engines/engine.py.230: loss: 0.23095 [I 2022-10-26 12:50:51] ...optimizers/trijoint.py.151: total_norm: 2.387217 [I 2022-10-26 12:50:56] ...trap/engines/engine.py.224: train: epoch 8 | batch 2260/2304 [I 2022-10-26 12:50:56] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:00:21 [I 2022-10-26 12:50:56] ...trap/engines/engine.py.229: process: 0.49438 | load: 0.00248 [I 2022-10-26 12:50:56] ...trap/engines/engine.py.230: loss: 0.25305 [I 2022-10-26 12:50:56] ...optimizers/trijoint.py.151: total_norm: 3.858911 [I 2022-10-26 12:51:01] ...trap/engines/engine.py.224: train: epoch 8 | batch 2270/2304 [I 2022-10-26 12:51:01] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:00:17 [I 2022-10-26 12:51:01] ...trap/engines/engine.py.229: process: 0.50115 | load: 0.00247 [I 2022-10-26 12:51:01] ...trap/engines/engine.py.230: loss: 0.21081 [I 2022-10-26 12:51:01] ...optimizers/trijoint.py.151: total_norm: 2.997286 [I 2022-10-26 12:51:06] ...trap/engines/engine.py.224: train: epoch 8 | batch 2280/2304 [I 2022-10-26 12:51:06] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:00:11 [I 2022-10-26 12:51:06] ...trap/engines/engine.py.229: process: 0.49791 | load: 0.00244 [I 2022-10-26 12:51:06] ...trap/engines/engine.py.230: loss: 0.32692 [I 2022-10-26 12:51:06] ...optimizers/trijoint.py.151: total_norm: 4.210312 [I 2022-10-26 12:51:11] ...trap/engines/engine.py.224: train: epoch 8 | batch 2290/2304 [I 2022-10-26 12:51:11] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:00:06 [I 2022-10-26 12:51:11] ...trap/engines/engine.py.229: process: 0.49522 | load: 0.00258 [I 2022-10-26 12:51:11] ...trap/engines/engine.py.230: loss: 0.31336 [I 2022-10-26 12:51:11] ...optimizers/trijoint.py.151: total_norm: 4.060791 [I 2022-10-26 12:51:16] ...trap/engines/engine.py.224: train: epoch 8 | batch 2300/2304 [I 2022-10-26 12:51:16] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:00:02 [I 2022-10-26 12:51:16] ...trap/engines/engine.py.229: process: 0.49653 | load: 0.00014 [I 2022-10-26 12:51:16] ...trap/engines/engine.py.230: loss: 0.18858 [I 2022-10-26 12:51:16] ...optimizers/trijoint.py.151: total_norm: 2.398485 [I 2022-10-26 12:51:18] ...trap/engines/engine.py.224: train: epoch 8 | batch 2304/2304 [I 2022-10-26 12:51:18] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:00:00 [I 2022-10-26 12:51:18] ...trap/engines/engine.py.229: process: 0.48745 | load: 0.00014 [I 2022-10-26 12:51:18] ...trap/engines/engine.py.230: loss: 0.30603 [I 2022-10-26 12:51:18] ...optimizers/trijoint.py.151: total_norm: 4.968806 [S 2022-10-26 12:51:19] ...trap/engines/engine.py.129: train_epoch.epoch: 8 [S 2022-10-26 12:51:19] ...trap/engines/engine.py.129: train_epoch.loss: 0.24768193195414387 [I 2022-10-26 12:51:19] ...s/criterions/triplet.py.84: margin = 0.09500000000000003 [I 2022-10-26 12:51:19] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 8 [I 2022-10-26 12:51:19] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 12:51:21] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 12:51:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 12:51:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 12:51:21] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 12:51:22] ...trap/engines/engine.py.321: eval: epoch 8 | batch 0/510 [I 2022-10-26 12:51:22] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:37 [I 2022-10-26 12:51:22] ...trap/engines/engine.py.326: process: 2.66139 | load: 2.28858 [I 2022-10-26 12:51:25] ...trap/engines/engine.py.321: eval: epoch 8 | batch 10/510 [I 2022-10-26 12:51:25] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:53 [I 2022-10-26 12:51:25] ...trap/engines/engine.py.326: process: 0.34647 | load: 0.00021 [I 2022-10-26 12:51:28] ...trap/engines/engine.py.321: eval: epoch 8 | batch 20/510 [I 2022-10-26 12:51:28] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:58 [I 2022-10-26 12:51:28] ...trap/engines/engine.py.326: process: 0.32989 | load: 0.00020 [I 2022-10-26 12:51:32] ...trap/engines/engine.py.321: eval: epoch 8 | batch 30/510 [I 2022-10-26 12:51:32] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:44 [I 2022-10-26 12:51:32] ...trap/engines/engine.py.326: process: 0.36528 | load: 0.00016 [I 2022-10-26 12:51:35] ...trap/engines/engine.py.321: eval: epoch 8 | batch 40/510 [I 2022-10-26 12:51:35] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:36 [I 2022-10-26 12:51:35] ...trap/engines/engine.py.326: process: 0.32339 | load: 0.00018 [I 2022-10-26 12:51:39] ...trap/engines/engine.py.321: eval: epoch 8 | batch 50/510 [I 2022-10-26 12:51:39] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-26 12:51:39] ...trap/engines/engine.py.326: process: 0.33484 | load: 0.00022 [I 2022-10-26 12:51:42] ...trap/engines/engine.py.321: eval: epoch 8 | batch 60/510 [I 2022-10-26 12:51:42] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:35 [I 2022-10-26 12:51:42] ...trap/engines/engine.py.326: process: 0.37014 | load: 0.00020 [I 2022-10-26 12:51:45] ...trap/engines/engine.py.321: eval: epoch 8 | batch 70/510 [I 2022-10-26 12:51:45] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-26 12:51:45] ...trap/engines/engine.py.326: process: 0.32927 | load: 0.00018 [I 2022-10-26 12:51:49] ...trap/engines/engine.py.321: eval: epoch 8 | batch 80/510 [I 2022-10-26 12:51:49] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:23 [I 2022-10-26 12:51:49] ...trap/engines/engine.py.326: process: 0.32302 | load: 0.00018 [I 2022-10-26 12:51:52] ...trap/engines/engine.py.321: eval: epoch 8 | batch 90/510 [I 2022-10-26 12:51:52] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-26 12:51:52] ...trap/engines/engine.py.326: process: 0.35175 | load: 0.00020 [I 2022-10-26 12:51:55] ...trap/engines/engine.py.321: eval: epoch 8 | batch 100/510 [I 2022-10-26 12:51:55] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:14 [I 2022-10-26 12:51:55] ...trap/engines/engine.py.326: process: 0.28470 | load: 0.00019 [I 2022-10-26 12:51:58] ...trap/engines/engine.py.321: eval: epoch 8 | batch 110/510 [I 2022-10-26 12:51:58] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:59 [I 2022-10-26 12:51:58] ...trap/engines/engine.py.326: process: 0.29943 | load: 0.00020 [I 2022-10-26 12:52:01] ...trap/engines/engine.py.321: eval: epoch 8 | batch 120/510 [I 2022-10-26 12:52:01] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 12:52:01] ...trap/engines/engine.py.326: process: 0.28608 | load: 0.00014 [I 2022-10-26 12:52:04] ...trap/engines/engine.py.321: eval: epoch 8 | batch 130/510 [I 2022-10-26 12:52:04] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-26 12:52:04] ...trap/engines/engine.py.326: process: 0.27970 | load: 0.00020 [I 2022-10-26 12:52:07] ...trap/engines/engine.py.321: eval: epoch 8 | batch 140/510 [I 2022-10-26 12:52:07] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-26 12:52:07] ...trap/engines/engine.py.326: process: 0.28743 | load: 0.00023 [I 2022-10-26 12:52:10] ...trap/engines/engine.py.321: eval: epoch 8 | batch 150/510 [I 2022-10-26 12:52:10] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:43 [I 2022-10-26 12:52:10] ...trap/engines/engine.py.326: process: 0.29030 | load: 0.00020 [I 2022-10-26 12:52:13] ...trap/engines/engine.py.321: eval: epoch 8 | batch 160/510 [I 2022-10-26 12:52:13] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 12:52:13] ...trap/engines/engine.py.326: process: 0.28178 | load: 0.00020 [I 2022-10-26 12:52:16] ...trap/engines/engine.py.321: eval: epoch 8 | batch 170/510 [I 2022-10-26 12:52:16] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:39 [I 2022-10-26 12:52:16] ...trap/engines/engine.py.326: process: 0.28905 | load: 0.00019 [I 2022-10-26 12:52:19] ...trap/engines/engine.py.321: eval: epoch 8 | batch 180/510 [I 2022-10-26 12:52:19] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 12:52:19] ...trap/engines/engine.py.326: process: 0.29142 | load: 0.00018 [I 2022-10-26 12:52:22] ...trap/engines/engine.py.321: eval: epoch 8 | batch 190/510 [I 2022-10-26 12:52:22] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 12:52:22] ...trap/engines/engine.py.326: process: 0.28871 | load: 0.00017 [I 2022-10-26 12:52:25] ...trap/engines/engine.py.321: eval: epoch 8 | batch 200/510 [I 2022-10-26 12:52:25] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-26 12:52:25] ...trap/engines/engine.py.326: process: 0.29272 | load: 0.00020 [I 2022-10-26 12:52:27] ...trap/engines/engine.py.321: eval: epoch 8 | batch 210/510 [I 2022-10-26 12:52:27] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 12:52:27] ...trap/engines/engine.py.326: process: 0.28525 | load: 0.00016 [I 2022-10-26 12:52:30] ...trap/engines/engine.py.321: eval: epoch 8 | batch 220/510 [I 2022-10-26 12:52:30] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 12:52:30] ...trap/engines/engine.py.326: process: 0.28750 | load: 0.00015 [I 2022-10-26 12:52:33] ...trap/engines/engine.py.321: eval: epoch 8 | batch 230/510 [I 2022-10-26 12:52:33] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:19 [I 2022-10-26 12:52:33] ...trap/engines/engine.py.326: process: 0.28779 | load: 0.00015 [I 2022-10-26 12:52:36] ...trap/engines/engine.py.321: eval: epoch 8 | batch 240/510 [I 2022-10-26 12:52:36] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 12:52:36] ...trap/engines/engine.py.326: process: 0.28668 | load: 0.00014 [I 2022-10-26 12:52:39] ...trap/engines/engine.py.321: eval: epoch 8 | batch 250/510 [I 2022-10-26 12:52:39] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:14 [I 2022-10-26 12:52:39] ...trap/engines/engine.py.326: process: 0.28478 | load: 0.00015 [I 2022-10-26 12:52:42] ...trap/engines/engine.py.321: eval: epoch 8 | batch 260/510 [I 2022-10-26 12:52:42] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:11 [I 2022-10-26 12:52:42] ...trap/engines/engine.py.326: process: 0.28751 | load: 0.00017 [I 2022-10-26 12:52:45] ...trap/engines/engine.py.321: eval: epoch 8 | batch 270/510 [I 2022-10-26 12:52:45] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-26 12:52:45] ...trap/engines/engine.py.326: process: 0.28658 | load: 0.00021 [I 2022-10-26 12:52:48] ...trap/engines/engine.py.321: eval: epoch 8 | batch 280/510 [I 2022-10-26 12:52:48] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 12:52:48] ...trap/engines/engine.py.326: process: 0.29006 | load: 0.00020 [I 2022-10-26 12:52:50] ...trap/engines/engine.py.321: eval: epoch 8 | batch 290/510 [I 2022-10-26 12:52:50] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 12:52:50] ...trap/engines/engine.py.326: process: 0.28842 | load: 0.00016 [I 2022-10-26 12:52:53] ...trap/engines/engine.py.321: eval: epoch 8 | batch 300/510 [I 2022-10-26 12:52:53] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 12:52:53] ...trap/engines/engine.py.326: process: 0.28226 | load: 0.00020 [I 2022-10-26 12:52:56] ...trap/engines/engine.py.321: eval: epoch 8 | batch 310/510 [I 2022-10-26 12:52:56] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:58 [I 2022-10-26 12:52:56] ...trap/engines/engine.py.326: process: 0.29180 | load: 0.00016 [I 2022-10-26 12:52:59] ...trap/engines/engine.py.321: eval: epoch 8 | batch 320/510 [I 2022-10-26 12:52:59] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:55 [I 2022-10-26 12:52:59] ...trap/engines/engine.py.326: process: 0.28987 | load: 0.00015 [I 2022-10-26 12:53:02] ...trap/engines/engine.py.321: eval: epoch 8 | batch 330/510 [I 2022-10-26 12:53:02] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:51 [I 2022-10-26 12:53:02] ...trap/engines/engine.py.326: process: 0.28844 | load: 0.00015 [I 2022-10-26 12:53:05] ...trap/engines/engine.py.321: eval: epoch 8 | batch 340/510 [I 2022-10-26 12:53:05] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:49 [I 2022-10-26 12:53:05] ...trap/engines/engine.py.326: process: 0.30627 | load: 0.00015 [I 2022-10-26 12:53:08] ...trap/engines/engine.py.321: eval: epoch 8 | batch 350/510 [I 2022-10-26 12:53:08] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 12:53:08] ...trap/engines/engine.py.326: process: 0.29170 | load: 0.00019 [I 2022-10-26 12:53:11] ...trap/engines/engine.py.321: eval: epoch 8 | batch 360/510 [I 2022-10-26 12:53:11] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 12:53:11] ...trap/engines/engine.py.326: process: 0.29007 | load: 0.00015 [I 2022-10-26 12:53:13] ...trap/engines/engine.py.321: eval: epoch 8 | batch 370/510 [I 2022-10-26 12:53:13] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:39 [I 2022-10-26 12:53:13] ...trap/engines/engine.py.326: process: 0.28024 | load: 0.00018 [I 2022-10-26 12:53:16] ...trap/engines/engine.py.321: eval: epoch 8 | batch 380/510 [I 2022-10-26 12:53:16] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 12:53:16] ...trap/engines/engine.py.326: process: 0.28516 | load: 0.00016 [I 2022-10-26 12:53:19] ...trap/engines/engine.py.321: eval: epoch 8 | batch 390/510 [I 2022-10-26 12:53:19] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 12:53:19] ...trap/engines/engine.py.326: process: 0.28760 | load: 0.00018 [I 2022-10-26 12:53:22] ...trap/engines/engine.py.321: eval: epoch 8 | batch 400/510 [I 2022-10-26 12:53:22] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 12:53:22] ...trap/engines/engine.py.326: process: 0.27995 | load: 0.00015 [I 2022-10-26 12:53:25] ...trap/engines/engine.py.321: eval: epoch 8 | batch 410/510 [I 2022-10-26 12:53:25] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:28 [I 2022-10-26 12:53:25] ...trap/engines/engine.py.326: process: 0.29042 | load: 0.00017 [I 2022-10-26 12:53:28] ...trap/engines/engine.py.321: eval: epoch 8 | batch 420/510 [I 2022-10-26 12:53:28] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-26 12:53:28] ...trap/engines/engine.py.326: process: 0.28472 | load: 0.00016 [I 2022-10-26 12:53:31] ...trap/engines/engine.py.321: eval: epoch 8 | batch 430/510 [I 2022-10-26 12:53:31] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-26 12:53:31] ...trap/engines/engine.py.326: process: 0.29404 | load: 0.00015 [I 2022-10-26 12:53:34] ...trap/engines/engine.py.321: eval: epoch 8 | batch 440/510 [I 2022-10-26 12:53:34] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 12:53:34] ...trap/engines/engine.py.326: process: 0.28777 | load: 0.00016 [I 2022-10-26 12:53:36] ...trap/engines/engine.py.321: eval: epoch 8 | batch 450/510 [I 2022-10-26 12:53:36] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 12:53:36] ...trap/engines/engine.py.326: process: 0.28668 | load: 0.00016 [I 2022-10-26 12:53:39] ...trap/engines/engine.py.321: eval: epoch 8 | batch 460/510 [I 2022-10-26 12:53:39] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 12:53:39] ...trap/engines/engine.py.326: process: 0.28149 | load: 0.00016 [I 2022-10-26 12:53:42] ...trap/engines/engine.py.321: eval: epoch 8 | batch 470/510 [I 2022-10-26 12:53:42] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 12:53:42] ...trap/engines/engine.py.326: process: 0.28709 | load: 0.00016 [I 2022-10-26 12:53:45] ...trap/engines/engine.py.321: eval: epoch 8 | batch 480/510 [I 2022-10-26 12:53:45] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 12:53:45] ...trap/engines/engine.py.326: process: 0.28819 | load: 0.00016 [I 2022-10-26 12:53:48] ...trap/engines/engine.py.321: eval: epoch 8 | batch 490/510 [I 2022-10-26 12:53:48] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:05 [I 2022-10-26 12:53:48] ...trap/engines/engine.py.326: process: 0.28484 | load: 0.00016 [I 2022-10-26 12:53:51] ...trap/engines/engine.py.321: eval: epoch 8 | batch 500/510 [I 2022-10-26 12:53:51] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-26 12:53:51] ...trap/engines/engine.py.326: process: 0.28285 | load: 0.00016 [I 2022-10-26 12:53:54] ...trap/engines/engine.py.321: eval: epoch 8 | batch 510/510 [I 2022-10-26 12:53:54] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 12:53:54] ...trap/engines/engine.py.326: process: 0.28720 | load: 0.00012 [S 2022-10-26 12:53:54] ...trap/engines/engine.py.132: eval_epoch.epoch: 8 [I 2022-10-26 12:53:54] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.60 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.60 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.492400 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.787700 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.870200 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.502900 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.791600 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.866000 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.489898 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.489898 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017945 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011824 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.012156 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014903 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012241 [S 2022-10-26 12:53:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010139 [I 2022-10-26 12:53:59] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-26 12:53:59] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 12:53:59] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 12:53:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 12:53:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 12:53:59] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 12:54:02] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 12:54:07] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 12:54:07] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 12:54:07] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 12:54:10] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 12:54:15] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 12:54:15] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 12:54:15] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 12:54:18] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 12:54:24] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 12:54:24] ...trap/engines/engine.py.163: Training model on trainset for epoch 9 [I 2022-10-26 12:54:24] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 12:54:26] ...trap/engines/engine.py.224: train: epoch 9 | batch 0/2304 [I 2022-10-26 12:54:26] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:12 [I 2022-10-26 12:54:26] ...trap/engines/engine.py.229: process: 2.79185 | load: 2.23034 [I 2022-10-26 12:54:26] ...trap/engines/engine.py.230: loss: 0.17449 [I 2022-10-26 12:54:26] ...optimizers/trijoint.py.151: total_norm: 1.987268 [I 2022-10-26 12:54:31] ...trap/engines/engine.py.224: train: epoch 9 | batch 10/2304 [I 2022-10-26 12:54:31] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:32 [I 2022-10-26 12:54:31] ...trap/engines/engine.py.229: process: 0.50139 | load: 0.00398 [I 2022-10-26 12:54:31] ...trap/engines/engine.py.230: loss: 0.25860 [I 2022-10-26 12:54:31] ...optimizers/trijoint.py.151: total_norm: 3.541295 [I 2022-10-26 12:54:36] ...trap/engines/engine.py.224: train: epoch 9 | batch 20/2304 [I 2022-10-26 12:54:36] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:05 [I 2022-10-26 12:54:36] ...trap/engines/engine.py.229: process: 0.49612 | load: 0.00398 [I 2022-10-26 12:54:36] ...trap/engines/engine.py.230: loss: 0.22054 [I 2022-10-26 12:54:36] ...optimizers/trijoint.py.151: total_norm: 3.343592 [I 2022-10-26 12:54:41] ...trap/engines/engine.py.224: train: epoch 9 | batch 30/2304 [I 2022-10-26 12:54:41] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:01 [I 2022-10-26 12:54:41] ...trap/engines/engine.py.229: process: 0.50030 | load: 0.00361 [I 2022-10-26 12:54:41] ...trap/engines/engine.py.230: loss: 0.26801 [I 2022-10-26 12:54:41] ...optimizers/trijoint.py.151: total_norm: 3.679900 [I 2022-10-26 12:54:47] ...trap/engines/engine.py.224: train: epoch 9 | batch 40/2304 [I 2022-10-26 12:54:47] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:57 [I 2022-10-26 12:54:47] ...trap/engines/engine.py.229: process: 0.49413 | load: 0.00316 [I 2022-10-26 12:54:47] ...trap/engines/engine.py.230: loss: 0.23920 [I 2022-10-26 12:54:47] ...optimizers/trijoint.py.151: total_norm: 3.115904 [I 2022-10-26 12:54:51] ...trap/engines/engine.py.224: train: epoch 9 | batch 50/2304 [I 2022-10-26 12:54:51] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:38 [I 2022-10-26 12:54:51] ...trap/engines/engine.py.229: process: 0.49794 | load: 0.00286 [I 2022-10-26 12:54:51] ...trap/engines/engine.py.230: loss: 0.29319 [I 2022-10-26 12:54:51] ...optimizers/trijoint.py.151: total_norm: 3.042280 [I 2022-10-26 12:54:57] ...trap/engines/engine.py.224: train: epoch 9 | batch 60/2304 [I 2022-10-26 12:54:57] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:37 [I 2022-10-26 12:54:57] ...trap/engines/engine.py.229: process: 0.48966 | load: 0.00277 [I 2022-10-26 12:54:57] ...trap/engines/engine.py.230: loss: 0.22185 [I 2022-10-26 12:54:57] ...optimizers/trijoint.py.151: total_norm: 2.503116 [I 2022-10-26 12:55:02] ...trap/engines/engine.py.224: train: epoch 9 | batch 70/2304 [I 2022-10-26 12:55:02] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:35 [I 2022-10-26 12:55:02] ...trap/engines/engine.py.229: process: 0.50439 | load: 0.00279 [I 2022-10-26 12:55:02] ...trap/engines/engine.py.230: loss: 0.20609 [I 2022-10-26 12:55:02] ...optimizers/trijoint.py.151: total_norm: 2.782253 [I 2022-10-26 12:55:07] ...trap/engines/engine.py.224: train: epoch 9 | batch 80/2304 [I 2022-10-26 12:55:07] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:37 [I 2022-10-26 12:55:07] ...trap/engines/engine.py.229: process: 0.49791 | load: 0.00263 [I 2022-10-26 12:55:07] ...trap/engines/engine.py.230: loss: 0.19571 [I 2022-10-26 12:55:07] ...optimizers/trijoint.py.151: total_norm: 2.555747 [I 2022-10-26 12:55:12] ...trap/engines/engine.py.224: train: epoch 9 | batch 90/2304 [I 2022-10-26 12:55:12] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:18:43 [I 2022-10-26 12:55:12] ...trap/engines/engine.py.229: process: 0.51115 | load: 0.00284 [I 2022-10-26 12:55:12] ...trap/engines/engine.py.230: loss: 0.24335 [I 2022-10-26 12:55:12] ...optimizers/trijoint.py.151: total_norm: 3.466112 [I 2022-10-26 12:55:17] ...trap/engines/engine.py.224: train: epoch 9 | batch 100/2304 [I 2022-10-26 12:55:17] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:18:29 [I 2022-10-26 12:55:17] ...trap/engines/engine.py.229: process: 0.48919 | load: 0.00287 [I 2022-10-26 12:55:17] ...trap/engines/engine.py.230: loss: 0.21383 [I 2022-10-26 12:55:17] ...optimizers/trijoint.py.151: total_norm: 2.987099 [I 2022-10-26 12:55:22] ...trap/engines/engine.py.224: train: epoch 9 | batch 110/2304 [I 2022-10-26 12:55:22] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:18:18 [I 2022-10-26 12:55:22] ...trap/engines/engine.py.229: process: 0.51101 | load: 0.00281 [I 2022-10-26 12:55:22] ...trap/engines/engine.py.230: loss: 0.25246 [I 2022-10-26 12:55:22] ...optimizers/trijoint.py.151: total_norm: 3.072201 [I 2022-10-26 12:55:27] ...trap/engines/engine.py.224: train: epoch 9 | batch 120/2304 [I 2022-10-26 12:55:27] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:26 [I 2022-10-26 12:55:27] ...trap/engines/engine.py.229: process: 0.49880 | load: 0.00273 [I 2022-10-26 12:55:27] ...trap/engines/engine.py.230: loss: 0.26228 [I 2022-10-26 12:55:27] ...optimizers/trijoint.py.151: total_norm: 2.974204 [I 2022-10-26 12:55:32] ...trap/engines/engine.py.224: train: epoch 9 | batch 130/2304 [I 2022-10-26 12:55:32] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:11 [I 2022-10-26 12:55:32] ...trap/engines/engine.py.229: process: 0.50576 | load: 0.00276 [I 2022-10-26 12:55:32] ...trap/engines/engine.py.230: loss: 0.19663 [I 2022-10-26 12:55:32] ...optimizers/trijoint.py.151: total_norm: 3.150364 [I 2022-10-26 12:55:37] ...trap/engines/engine.py.224: train: epoch 9 | batch 140/2304 [I 2022-10-26 12:55:37] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:18:29 [I 2022-10-26 12:55:37] ...trap/engines/engine.py.229: process: 0.51205 | load: 0.00285 [I 2022-10-26 12:55:37] ...trap/engines/engine.py.230: loss: 0.21131 [I 2022-10-26 12:55:37] ...optimizers/trijoint.py.151: total_norm: 3.665510 [I 2022-10-26 12:55:42] ...trap/engines/engine.py.224: train: epoch 9 | batch 150/2304 [I 2022-10-26 12:55:42] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:17:55 [I 2022-10-26 12:55:42] ...trap/engines/engine.py.229: process: 0.49844 | load: 0.00250 [I 2022-10-26 12:55:42] ...trap/engines/engine.py.230: loss: 0.27690 [I 2022-10-26 12:55:42] ...optimizers/trijoint.py.151: total_norm: 3.847442 [I 2022-10-26 12:55:47] ...trap/engines/engine.py.224: train: epoch 9 | batch 160/2304 [I 2022-10-26 12:55:47] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:17:50 [I 2022-10-26 12:55:47] ...trap/engines/engine.py.229: process: 0.50129 | load: 0.00285 [I 2022-10-26 12:55:47] ...trap/engines/engine.py.230: loss: 0.21111 [I 2022-10-26 12:55:47] ...optimizers/trijoint.py.151: total_norm: 3.483557 [I 2022-10-26 12:55:52] ...trap/engines/engine.py.224: train: epoch 9 | batch 170/2304 [I 2022-10-26 12:55:52] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:45 [I 2022-10-26 12:55:52] ...trap/engines/engine.py.229: process: 0.49362 | load: 0.00259 [I 2022-10-26 12:55:52] ...trap/engines/engine.py.230: loss: 0.28216 [I 2022-10-26 12:55:52] ...optimizers/trijoint.py.151: total_norm: 4.972704 [I 2022-10-26 12:55:57] ...trap/engines/engine.py.224: train: epoch 9 | batch 180/2304 [I 2022-10-26 12:55:57] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:17:31 [I 2022-10-26 12:55:57] ...trap/engines/engine.py.229: process: 0.49059 | load: 0.00269 [I 2022-10-26 12:55:57] ...trap/engines/engine.py.230: loss: 0.23656 [I 2022-10-26 12:55:57] ...optimizers/trijoint.py.151: total_norm: 3.253316 [I 2022-10-26 12:56:02] ...trap/engines/engine.py.224: train: epoch 9 | batch 190/2304 [I 2022-10-26 12:56:02] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:17:46 [I 2022-10-26 12:56:02] ...trap/engines/engine.py.229: process: 0.49786 | load: 0.00262 [I 2022-10-26 12:56:02] ...trap/engines/engine.py.230: loss: 0.18108 [I 2022-10-26 12:56:02] ...optimizers/trijoint.py.151: total_norm: 3.217431 [I 2022-10-26 12:56:07] ...trap/engines/engine.py.224: train: epoch 9 | batch 200/2304 [I 2022-10-26 12:56:07] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:19 [I 2022-10-26 12:56:07] ...trap/engines/engine.py.229: process: 0.48567 | load: 0.00260 [I 2022-10-26 12:56:07] ...trap/engines/engine.py.230: loss: 0.25397 [I 2022-10-26 12:56:07] ...optimizers/trijoint.py.151: total_norm: 2.739697 [I 2022-10-26 12:56:12] ...trap/engines/engine.py.224: train: epoch 9 | batch 210/2304 [I 2022-10-26 12:56:12] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:22 [I 2022-10-26 12:56:12] ...trap/engines/engine.py.229: process: 0.49053 | load: 0.00256 [I 2022-10-26 12:56:12] ...trap/engines/engine.py.230: loss: 0.15976 [I 2022-10-26 12:56:12] ...optimizers/trijoint.py.151: total_norm: 2.352986 [I 2022-10-26 12:56:17] ...trap/engines/engine.py.224: train: epoch 9 | batch 220/2304 [I 2022-10-26 12:56:17] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:33 [I 2022-10-26 12:56:17] ...trap/engines/engine.py.229: process: 0.51846 | load: 0.00250 [I 2022-10-26 12:56:17] ...trap/engines/engine.py.230: loss: 0.23029 [I 2022-10-26 12:56:17] ...optimizers/trijoint.py.151: total_norm: 3.598024 [I 2022-10-26 12:56:22] ...trap/engines/engine.py.224: train: epoch 9 | batch 230/2304 [I 2022-10-26 12:56:22] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:25 [I 2022-10-26 12:56:22] ...trap/engines/engine.py.229: process: 0.50901 | load: 0.00258 [I 2022-10-26 12:56:22] ...trap/engines/engine.py.230: loss: 0.28936 [I 2022-10-26 12:56:22] ...optimizers/trijoint.py.151: total_norm: 3.527736 [I 2022-10-26 12:56:27] ...trap/engines/engine.py.224: train: epoch 9 | batch 240/2304 [I 2022-10-26 12:56:27] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:09 [I 2022-10-26 12:56:27] ...trap/engines/engine.py.229: process: 0.47915 | load: 0.00260 [I 2022-10-26 12:56:27] ...trap/engines/engine.py.230: loss: 0.21389 [I 2022-10-26 12:56:27] ...optimizers/trijoint.py.151: total_norm: 2.872374 [I 2022-10-26 12:56:32] ...trap/engines/engine.py.224: train: epoch 9 | batch 250/2304 [I 2022-10-26 12:56:32] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:02 [I 2022-10-26 12:56:32] ...trap/engines/engine.py.229: process: 0.50267 | load: 0.00257 [I 2022-10-26 12:56:32] ...trap/engines/engine.py.230: loss: 0.32565 [I 2022-10-26 12:56:32] ...optimizers/trijoint.py.151: total_norm: 5.976533 [I 2022-10-26 12:56:37] ...trap/engines/engine.py.224: train: epoch 9 | batch 260/2304 [I 2022-10-26 12:56:37] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:02 [I 2022-10-26 12:56:37] ...trap/engines/engine.py.229: process: 0.50239 | load: 0.00245 [I 2022-10-26 12:56:37] ...trap/engines/engine.py.230: loss: 0.24121 [I 2022-10-26 12:56:37] ...optimizers/trijoint.py.151: total_norm: 3.137709 [I 2022-10-26 12:56:42] ...trap/engines/engine.py.224: train: epoch 9 | batch 270/2304 [I 2022-10-26 12:56:42] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:17:00 [I 2022-10-26 12:56:42] ...trap/engines/engine.py.229: process: 0.50412 | load: 0.00245 [I 2022-10-26 12:56:42] ...trap/engines/engine.py.230: loss: 0.25514 [I 2022-10-26 12:56:42] ...optimizers/trijoint.py.151: total_norm: 3.183063 [I 2022-10-26 12:57:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 280/2304 [I 2022-10-26 12:57:08] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:53:53 [I 2022-10-26 12:57:08] ...trap/engines/engine.py.229: process: 0.49991 | load: 0.00245 [I 2022-10-26 12:57:08] ...trap/engines/engine.py.230: loss: 0.28510 [I 2022-10-26 12:57:08] ...optimizers/trijoint.py.151: total_norm: 3.505338 [I 2022-10-26 12:57:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 290/2304 [I 2022-10-26 12:57:13] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:20:39 [I 2022-10-26 12:57:13] ...trap/engines/engine.py.229: process: 0.49138 | load: 0.00240 [I 2022-10-26 12:57:13] ...trap/engines/engine.py.230: loss: 0.23066 [I 2022-10-26 12:57:13] ...optimizers/trijoint.py.151: total_norm: 2.760627 [I 2022-10-26 12:57:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 300/2304 [I 2022-10-26 12:57:18] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:16:59 [I 2022-10-26 12:57:18] ...trap/engines/engine.py.229: process: 0.48826 | load: 0.00245 [I 2022-10-26 12:57:18] ...trap/engines/engine.py.230: loss: 0.18863 [I 2022-10-26 12:57:18] ...optimizers/trijoint.py.151: total_norm: 3.217388 [I 2022-10-26 12:57:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 310/2304 [I 2022-10-26 12:57:23] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:16:34 [I 2022-10-26 12:57:23] ...trap/engines/engine.py.229: process: 0.49626 | load: 0.00236 [I 2022-10-26 12:57:23] ...trap/engines/engine.py.230: loss: 0.28271 [I 2022-10-26 12:57:23] ...optimizers/trijoint.py.151: total_norm: 3.757145 [I 2022-10-26 12:57:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 320/2304 [I 2022-10-26 12:57:28] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:16:34 [I 2022-10-26 12:57:28] ...trap/engines/engine.py.229: process: 0.51122 | load: 0.00236 [I 2022-10-26 12:57:28] ...trap/engines/engine.py.230: loss: 0.26435 [I 2022-10-26 12:57:28] ...optimizers/trijoint.py.151: total_norm: 3.396466 [I 2022-10-26 12:57:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 330/2304 [I 2022-10-26 12:57:33] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:16:25 [I 2022-10-26 12:57:33] ...trap/engines/engine.py.229: process: 0.50459 | load: 0.00235 [I 2022-10-26 12:57:33] ...trap/engines/engine.py.230: loss: 0.29509 [I 2022-10-26 12:57:33] ...optimizers/trijoint.py.151: total_norm: 4.025672 [I 2022-10-26 12:57:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 340/2304 [I 2022-10-26 12:57:38] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:16:10 [I 2022-10-26 12:57:38] ...trap/engines/engine.py.229: process: 0.49522 | load: 0.00237 [I 2022-10-26 12:57:38] ...trap/engines/engine.py.230: loss: 0.22184 [I 2022-10-26 12:57:38] ...optimizers/trijoint.py.151: total_norm: 3.565144 [I 2022-10-26 12:57:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 350/2304 [I 2022-10-26 12:57:43] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:16:03 [I 2022-10-26 12:57:43] ...trap/engines/engine.py.229: process: 0.49585 | load: 0.00248 [I 2022-10-26 12:57:43] ...trap/engines/engine.py.230: loss: 0.17479 [I 2022-10-26 12:57:43] ...optimizers/trijoint.py.151: total_norm: 2.719779 [I 2022-10-26 12:57:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 360/2304 [I 2022-10-26 12:57:48] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:16:49 [I 2022-10-26 12:57:48] ...trap/engines/engine.py.229: process: 0.56789 | load: 0.00266 [I 2022-10-26 12:57:48] ...trap/engines/engine.py.230: loss: 0.18840 [I 2022-10-26 12:57:48] ...optimizers/trijoint.py.151: total_norm: 2.964283 [I 2022-10-26 12:57:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 370/2304 [I 2022-10-26 12:57:53] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:16:05 [I 2022-10-26 12:57:53] ...trap/engines/engine.py.229: process: 0.49402 | load: 0.00241 [I 2022-10-26 12:57:53] ...trap/engines/engine.py.230: loss: 0.22712 [I 2022-10-26 12:57:53] ...optimizers/trijoint.py.151: total_norm: 2.769923 [I 2022-10-26 12:57:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 380/2304 [I 2022-10-26 12:57:58] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:16:03 [I 2022-10-26 12:57:58] ...trap/engines/engine.py.229: process: 0.49937 | load: 0.00241 [I 2022-10-26 12:57:58] ...trap/engines/engine.py.230: loss: 0.15808 [I 2022-10-26 12:57:58] ...optimizers/trijoint.py.151: total_norm: 2.715710 [I 2022-10-26 12:58:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 390/2304 [I 2022-10-26 12:58:03] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:15:52 [I 2022-10-26 12:58:03] ...trap/engines/engine.py.229: process: 0.49994 | load: 0.00242 [I 2022-10-26 12:58:03] ...trap/engines/engine.py.230: loss: 0.29024 [I 2022-10-26 12:58:03] ...optimizers/trijoint.py.151: total_norm: 3.455327 [I 2022-10-26 12:58:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 400/2304 [I 2022-10-26 12:58:08] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:15:53 [I 2022-10-26 12:58:08] ...trap/engines/engine.py.229: process: 0.50065 | load: 0.00244 [I 2022-10-26 12:58:08] ...trap/engines/engine.py.230: loss: 0.22000 [I 2022-10-26 12:58:08] ...optimizers/trijoint.py.151: total_norm: 3.522279 [I 2022-10-26 12:58:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 410/2304 [I 2022-10-26 12:58:13] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:15:37 [I 2022-10-26 12:58:13] ...trap/engines/engine.py.229: process: 0.49040 | load: 0.00242 [I 2022-10-26 12:58:13] ...trap/engines/engine.py.230: loss: 0.23771 [I 2022-10-26 12:58:13] ...optimizers/trijoint.py.151: total_norm: 3.278903 [I 2022-10-26 12:58:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 420/2304 [I 2022-10-26 12:58:18] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:15:40 [I 2022-10-26 12:58:18] ...trap/engines/engine.py.229: process: 0.49729 | load: 0.00241 [I 2022-10-26 12:58:18] ...trap/engines/engine.py.230: loss: 0.17719 [I 2022-10-26 12:58:18] ...optimizers/trijoint.py.151: total_norm: 3.593994 [I 2022-10-26 12:58:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 430/2304 [I 2022-10-26 12:58:23] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:15:34 [I 2022-10-26 12:58:23] ...trap/engines/engine.py.229: process: 0.49497 | load: 0.00240 [I 2022-10-26 12:58:23] ...trap/engines/engine.py.230: loss: 0.25909 [I 2022-10-26 12:58:23] ...optimizers/trijoint.py.151: total_norm: 4.349280 [I 2022-10-26 12:58:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 440/2304 [I 2022-10-26 12:58:28] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:15:26 [I 2022-10-26 12:58:28] ...trap/engines/engine.py.229: process: 0.50458 | load: 0.00245 [I 2022-10-26 12:58:28] ...trap/engines/engine.py.230: loss: 0.22078 [I 2022-10-26 12:58:28] ...optimizers/trijoint.py.151: total_norm: 3.910935 [I 2022-10-26 12:58:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 450/2304 [I 2022-10-26 12:58:33] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:15:21 [I 2022-10-26 12:58:33] ...trap/engines/engine.py.229: process: 0.49828 | load: 0.00259 [I 2022-10-26 12:58:33] ...trap/engines/engine.py.230: loss: 0.27637 [I 2022-10-26 12:58:33] ...optimizers/trijoint.py.151: total_norm: 3.977285 [I 2022-10-26 12:58:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 460/2304 [I 2022-10-26 12:58:38] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:15:16 [I 2022-10-26 12:58:38] ...trap/engines/engine.py.229: process: 0.49560 | load: 0.00248 [I 2022-10-26 12:58:38] ...trap/engines/engine.py.230: loss: 0.23509 [I 2022-10-26 12:58:38] ...optimizers/trijoint.py.151: total_norm: 3.917615 [I 2022-10-26 12:58:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 470/2304 [I 2022-10-26 12:58:43] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:15:12 [I 2022-10-26 12:58:43] ...trap/engines/engine.py.229: process: 0.50694 | load: 0.00246 [I 2022-10-26 12:58:43] ...trap/engines/engine.py.230: loss: 0.26923 [I 2022-10-26 12:58:43] ...optimizers/trijoint.py.151: total_norm: 3.601923 [I 2022-10-26 12:58:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 480/2304 [I 2022-10-26 12:58:48] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:15:33 [I 2022-10-26 12:58:48] ...trap/engines/engine.py.229: process: 0.49638 | load: 0.00242 [I 2022-10-26 12:58:48] ...trap/engines/engine.py.230: loss: 0.21234 [I 2022-10-26 12:58:48] ...optimizers/trijoint.py.151: total_norm: 3.400973 [I 2022-10-26 12:58:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 490/2304 [I 2022-10-26 12:58:53] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:15:13 [I 2022-10-26 12:58:53] ...trap/engines/engine.py.229: process: 0.51042 | load: 0.00245 [I 2022-10-26 12:58:53] ...trap/engines/engine.py.230: loss: 0.21609 [I 2022-10-26 12:58:53] ...optimizers/trijoint.py.151: total_norm: 2.866323 [I 2022-10-26 12:58:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 500/2304 [I 2022-10-26 12:58:58] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:14:59 [I 2022-10-26 12:58:58] ...trap/engines/engine.py.229: process: 0.49322 | load: 0.00246 [I 2022-10-26 12:58:58] ...trap/engines/engine.py.230: loss: 0.24011 [I 2022-10-26 12:58:58] ...optimizers/trijoint.py.151: total_norm: 2.648111 [I 2022-10-26 12:59:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 510/2304 [I 2022-10-26 12:59:03] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:14:48 [I 2022-10-26 12:59:03] ...trap/engines/engine.py.229: process: 0.49416 | load: 0.00242 [I 2022-10-26 12:59:03] ...trap/engines/engine.py.230: loss: 0.18639 [I 2022-10-26 12:59:03] ...optimizers/trijoint.py.151: total_norm: 3.285689 [I 2022-10-26 12:59:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 520/2304 [I 2022-10-26 12:59:08] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:14:43 [I 2022-10-26 12:59:08] ...trap/engines/engine.py.229: process: 0.49835 | load: 0.00242 [I 2022-10-26 12:59:08] ...trap/engines/engine.py.230: loss: 0.17464 [I 2022-10-26 12:59:08] ...optimizers/trijoint.py.151: total_norm: 3.755574 [I 2022-10-26 12:59:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 530/2304 [I 2022-10-26 12:59:13] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:14:50 [I 2022-10-26 12:59:13] ...trap/engines/engine.py.229: process: 0.50195 | load: 0.00262 [I 2022-10-26 12:59:13] ...trap/engines/engine.py.230: loss: 0.22957 [I 2022-10-26 12:59:13] ...optimizers/trijoint.py.151: total_norm: 3.963108 [I 2022-10-26 12:59:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 540/2304 [I 2022-10-26 12:59:18] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:14:43 [I 2022-10-26 12:59:18] ...trap/engines/engine.py.229: process: 0.50428 | load: 0.00247 [I 2022-10-26 12:59:18] ...trap/engines/engine.py.230: loss: 0.27175 [I 2022-10-26 12:59:18] ...optimizers/trijoint.py.151: total_norm: 3.817656 [I 2022-10-26 12:59:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 550/2304 [I 2022-10-26 12:59:23] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:14:34 [I 2022-10-26 12:59:23] ...trap/engines/engine.py.229: process: 0.49753 | load: 0.00246 [I 2022-10-26 12:59:23] ...trap/engines/engine.py.230: loss: 0.18922 [I 2022-10-26 12:59:23] ...optimizers/trijoint.py.151: total_norm: 2.724767 [I 2022-10-26 12:59:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 560/2304 [I 2022-10-26 12:59:28] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:14:30 [I 2022-10-26 12:59:28] ...trap/engines/engine.py.229: process: 0.49927 | load: 0.00246 [I 2022-10-26 12:59:28] ...trap/engines/engine.py.230: loss: 0.25633 [I 2022-10-26 12:59:28] ...optimizers/trijoint.py.151: total_norm: 2.885671 [I 2022-10-26 12:59:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 570/2304 [I 2022-10-26 12:59:33] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:14:22 [I 2022-10-26 12:59:33] ...trap/engines/engine.py.229: process: 0.49864 | load: 0.00242 [I 2022-10-26 12:59:33] ...trap/engines/engine.py.230: loss: 0.19988 [I 2022-10-26 12:59:33] ...optimizers/trijoint.py.151: total_norm: 2.322847 [I 2022-10-26 12:59:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 580/2304 [I 2022-10-26 12:59:38] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:14:13 [I 2022-10-26 12:59:38] ...trap/engines/engine.py.229: process: 0.49299 | load: 0.00242 [I 2022-10-26 12:59:38] ...trap/engines/engine.py.230: loss: 0.21100 [I 2022-10-26 12:59:38] ...optimizers/trijoint.py.151: total_norm: 3.703340 [I 2022-10-26 12:59:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 590/2304 [I 2022-10-26 12:59:43] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:14:12 [I 2022-10-26 12:59:43] ...trap/engines/engine.py.229: process: 0.49733 | load: 0.00252 [I 2022-10-26 12:59:43] ...trap/engines/engine.py.230: loss: 0.21828 [I 2022-10-26 12:59:43] ...optimizers/trijoint.py.151: total_norm: 3.851371 [I 2022-10-26 12:59:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 600/2304 [I 2022-10-26 12:59:48] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:14:07 [I 2022-10-26 12:59:48] ...trap/engines/engine.py.229: process: 0.49813 | load: 0.00254 [I 2022-10-26 12:59:48] ...trap/engines/engine.py.230: loss: 0.17409 [I 2022-10-26 12:59:48] ...optimizers/trijoint.py.151: total_norm: 3.178091 [I 2022-10-26 12:59:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 610/2304 [I 2022-10-26 12:59:53] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:14:15 [I 2022-10-26 12:59:53] ...trap/engines/engine.py.229: process: 0.48470 | load: 0.00276 [I 2022-10-26 12:59:53] ...trap/engines/engine.py.230: loss: 0.23219 [I 2022-10-26 12:59:53] ...optimizers/trijoint.py.151: total_norm: 3.363406 [I 2022-10-26 12:59:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 620/2304 [I 2022-10-26 12:59:58] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:14:01 [I 2022-10-26 12:59:58] ...trap/engines/engine.py.229: process: 0.49970 | load: 0.00245 [I 2022-10-26 12:59:58] ...trap/engines/engine.py.230: loss: 0.33111 [I 2022-10-26 12:59:58] ...optimizers/trijoint.py.151: total_norm: 4.119000 [I 2022-10-26 13:00:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 630/2304 [I 2022-10-26 13:00:03] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:13:48 [I 2022-10-26 13:00:03] ...trap/engines/engine.py.229: process: 0.48778 | load: 0.00246 [I 2022-10-26 13:00:03] ...trap/engines/engine.py.230: loss: 0.24007 [I 2022-10-26 13:00:03] ...optimizers/trijoint.py.151: total_norm: 3.179735 [I 2022-10-26 13:00:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 640/2304 [I 2022-10-26 13:00:08] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:13:42 [I 2022-10-26 13:00:08] ...trap/engines/engine.py.229: process: 0.49087 | load: 0.00243 [I 2022-10-26 13:00:08] ...trap/engines/engine.py.230: loss: 0.27222 [I 2022-10-26 13:00:08] ...optimizers/trijoint.py.151: total_norm: 3.762151 [I 2022-10-26 13:00:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 650/2304 [I 2022-10-26 13:00:13] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:13:47 [I 2022-10-26 13:00:13] ...trap/engines/engine.py.229: process: 0.50204 | load: 0.00243 [I 2022-10-26 13:00:13] ...trap/engines/engine.py.230: loss: 0.17665 [I 2022-10-26 13:00:13] ...optimizers/trijoint.py.151: total_norm: 2.491506 [I 2022-10-26 13:00:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 660/2304 [I 2022-10-26 13:00:18] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:13:40 [I 2022-10-26 13:00:18] ...trap/engines/engine.py.229: process: 0.50093 | load: 0.00252 [I 2022-10-26 13:00:18] ...trap/engines/engine.py.230: loss: 0.17774 [I 2022-10-26 13:00:18] ...optimizers/trijoint.py.151: total_norm: 2.879113 [I 2022-10-26 13:00:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 670/2304 [I 2022-10-26 13:00:23] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:13:32 [I 2022-10-26 13:00:23] ...trap/engines/engine.py.229: process: 0.49482 | load: 0.00246 [I 2022-10-26 13:00:23] ...trap/engines/engine.py.230: loss: 0.21982 [I 2022-10-26 13:00:23] ...optimizers/trijoint.py.151: total_norm: 3.537771 [I 2022-10-26 13:00:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 680/2304 [I 2022-10-26 13:00:28] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:13:24 [I 2022-10-26 13:00:28] ...trap/engines/engine.py.229: process: 0.49777 | load: 0.00242 [I 2022-10-26 13:00:28] ...trap/engines/engine.py.230: loss: 0.21779 [I 2022-10-26 13:00:28] ...optimizers/trijoint.py.151: total_norm: 3.637317 [I 2022-10-26 13:00:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 690/2304 [I 2022-10-26 13:00:33] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:13:23 [I 2022-10-26 13:00:33] ...trap/engines/engine.py.229: process: 0.48912 | load: 0.00246 [I 2022-10-26 13:00:33] ...trap/engines/engine.py.230: loss: 0.24357 [I 2022-10-26 13:00:33] ...optimizers/trijoint.py.151: total_norm: 3.020336 [I 2022-10-26 13:00:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 700/2304 [I 2022-10-26 13:00:38] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:13:17 [I 2022-10-26 13:00:38] ...trap/engines/engine.py.229: process: 0.50197 | load: 0.00245 [I 2022-10-26 13:00:38] ...trap/engines/engine.py.230: loss: 0.20692 [I 2022-10-26 13:00:38] ...optimizers/trijoint.py.151: total_norm: 3.382167 [I 2022-10-26 13:00:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 710/2304 [I 2022-10-26 13:00:43] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:13:16 [I 2022-10-26 13:00:43] ...trap/engines/engine.py.229: process: 0.50113 | load: 0.00245 [I 2022-10-26 13:00:43] ...trap/engines/engine.py.230: loss: 0.23519 [I 2022-10-26 13:00:43] ...optimizers/trijoint.py.151: total_norm: 3.471766 [I 2022-10-26 13:00:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 720/2304 [I 2022-10-26 13:00:48] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:13:07 [I 2022-10-26 13:00:48] ...trap/engines/engine.py.229: process: 0.50466 | load: 0.00242 [I 2022-10-26 13:00:48] ...trap/engines/engine.py.230: loss: 0.24464 [I 2022-10-26 13:00:48] ...optimizers/trijoint.py.151: total_norm: 3.500132 [I 2022-10-26 13:00:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 730/2304 [I 2022-10-26 13:00:53] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:13:04 [I 2022-10-26 13:00:53] ...trap/engines/engine.py.229: process: 0.50146 | load: 0.00243 [I 2022-10-26 13:00:53] ...trap/engines/engine.py.230: loss: 0.22029 [I 2022-10-26 13:00:53] ...optimizers/trijoint.py.151: total_norm: 3.432300 [I 2022-10-26 13:00:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 740/2304 [I 2022-10-26 13:00:58] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:12:54 [I 2022-10-26 13:00:58] ...trap/engines/engine.py.229: process: 0.49683 | load: 0.00243 [I 2022-10-26 13:00:58] ...trap/engines/engine.py.230: loss: 0.27058 [I 2022-10-26 13:00:58] ...optimizers/trijoint.py.151: total_norm: 3.922915 [I 2022-10-26 13:01:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 750/2304 [I 2022-10-26 13:01:03] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:12:55 [I 2022-10-26 13:01:03] ...trap/engines/engine.py.229: process: 0.49307 | load: 0.00252 [I 2022-10-26 13:01:03] ...trap/engines/engine.py.230: loss: 0.26542 [I 2022-10-26 13:01:03] ...optimizers/trijoint.py.151: total_norm: 3.511599 [I 2022-10-26 13:01:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 760/2304 [I 2022-10-26 13:01:08] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:12:53 [I 2022-10-26 13:01:08] ...trap/engines/engine.py.229: process: 0.50125 | load: 0.00248 [I 2022-10-26 13:01:08] ...trap/engines/engine.py.230: loss: 0.23668 [I 2022-10-26 13:01:08] ...optimizers/trijoint.py.151: total_norm: 3.840993 [I 2022-10-26 13:01:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 770/2304 [I 2022-10-26 13:01:13] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:12:46 [I 2022-10-26 13:01:13] ...trap/engines/engine.py.229: process: 0.49654 | load: 0.00244 [I 2022-10-26 13:01:13] ...trap/engines/engine.py.230: loss: 0.26898 [I 2022-10-26 13:01:13] ...optimizers/trijoint.py.151: total_norm: 3.715106 [I 2022-10-26 13:01:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 780/2304 [I 2022-10-26 13:01:18] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:12:39 [I 2022-10-26 13:01:18] ...trap/engines/engine.py.229: process: 0.49208 | load: 0.00262 [I 2022-10-26 13:01:18] ...trap/engines/engine.py.230: loss: 0.21051 [I 2022-10-26 13:01:18] ...optimizers/trijoint.py.151: total_norm: 2.990616 [I 2022-10-26 13:01:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 790/2304 [I 2022-10-26 13:01:23] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:12:34 [I 2022-10-26 13:01:23] ...trap/engines/engine.py.229: process: 0.50039 | load: 0.00247 [I 2022-10-26 13:01:23] ...trap/engines/engine.py.230: loss: 0.14332 [I 2022-10-26 13:01:23] ...optimizers/trijoint.py.151: total_norm: 2.317145 [I 2022-10-26 13:01:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 800/2304 [I 2022-10-26 13:01:28] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:12:32 [I 2022-10-26 13:01:28] ...trap/engines/engine.py.229: process: 0.49527 | load: 0.00252 [I 2022-10-26 13:01:28] ...trap/engines/engine.py.230: loss: 0.34720 [I 2022-10-26 13:01:28] ...optimizers/trijoint.py.151: total_norm: 4.773574 [I 2022-10-26 13:01:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 810/2304 [I 2022-10-26 13:01:33] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:12:30 [I 2022-10-26 13:01:33] ...trap/engines/engine.py.229: process: 0.50741 | load: 0.00249 [I 2022-10-26 13:01:33] ...trap/engines/engine.py.230: loss: 0.23700 [I 2022-10-26 13:01:33] ...optimizers/trijoint.py.151: total_norm: 3.325891 [I 2022-10-26 13:01:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 820/2304 [I 2022-10-26 13:01:38] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:12:17 [I 2022-10-26 13:01:38] ...trap/engines/engine.py.229: process: 0.49530 | load: 0.00245 [I 2022-10-26 13:01:38] ...trap/engines/engine.py.230: loss: 0.28132 [I 2022-10-26 13:01:38] ...optimizers/trijoint.py.151: total_norm: 3.480278 [I 2022-10-26 13:01:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 830/2304 [I 2022-10-26 13:01:43] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:12:16 [I 2022-10-26 13:01:43] ...trap/engines/engine.py.229: process: 0.50316 | load: 0.00244 [I 2022-10-26 13:01:43] ...trap/engines/engine.py.230: loss: 0.22659 [I 2022-10-26 13:01:43] ...optimizers/trijoint.py.151: total_norm: 4.704292 [I 2022-10-26 13:01:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 840/2304 [I 2022-10-26 13:01:48] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:12:11 [I 2022-10-26 13:01:48] ...trap/engines/engine.py.229: process: 0.49213 | load: 0.00242 [I 2022-10-26 13:01:48] ...trap/engines/engine.py.230: loss: 0.24182 [I 2022-10-26 13:01:48] ...optimizers/trijoint.py.151: total_norm: 2.883284 [I 2022-10-26 13:01:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 850/2304 [I 2022-10-26 13:01:53] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:12:02 [I 2022-10-26 13:01:53] ...trap/engines/engine.py.229: process: 0.50125 | load: 0.00253 [I 2022-10-26 13:01:53] ...trap/engines/engine.py.230: loss: 0.18099 [I 2022-10-26 13:01:53] ...optimizers/trijoint.py.151: total_norm: 2.763021 [I 2022-10-26 13:01:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 860/2304 [I 2022-10-26 13:01:58] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:11:58 [I 2022-10-26 13:01:58] ...trap/engines/engine.py.229: process: 0.49497 | load: 0.00247 [I 2022-10-26 13:01:58] ...trap/engines/engine.py.230: loss: 0.20072 [I 2022-10-26 13:01:58] ...optimizers/trijoint.py.151: total_norm: 2.209731 [I 2022-10-26 13:02:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 870/2304 [I 2022-10-26 13:02:03] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:11:51 [I 2022-10-26 13:02:03] ...trap/engines/engine.py.229: process: 0.48987 | load: 0.00243 [I 2022-10-26 13:02:03] ...trap/engines/engine.py.230: loss: 0.16658 [I 2022-10-26 13:02:03] ...optimizers/trijoint.py.151: total_norm: 3.408645 [I 2022-10-26 13:02:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 880/2304 [I 2022-10-26 13:02:08] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:11:41 [I 2022-10-26 13:02:08] ...trap/engines/engine.py.229: process: 0.49509 | load: 0.00246 [I 2022-10-26 13:02:08] ...trap/engines/engine.py.230: loss: 0.20138 [I 2022-10-26 13:02:08] ...optimizers/trijoint.py.151: total_norm: 3.060906 [I 2022-10-26 13:02:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 890/2304 [I 2022-10-26 13:02:13] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:11:43 [I 2022-10-26 13:02:13] ...trap/engines/engine.py.229: process: 0.49702 | load: 0.00247 [I 2022-10-26 13:02:13] ...trap/engines/engine.py.230: loss: 0.21422 [I 2022-10-26 13:02:13] ...optimizers/trijoint.py.151: total_norm: 3.218741 [I 2022-10-26 13:02:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 900/2304 [I 2022-10-26 13:02:18] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:11:41 [I 2022-10-26 13:02:18] ...trap/engines/engine.py.229: process: 0.49946 | load: 0.00243 [I 2022-10-26 13:02:18] ...trap/engines/engine.py.230: loss: 0.29657 [I 2022-10-26 13:02:18] ...optimizers/trijoint.py.151: total_norm: 4.004414 [I 2022-10-26 13:02:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 910/2304 [I 2022-10-26 13:02:23] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:11:35 [I 2022-10-26 13:02:23] ...trap/engines/engine.py.229: process: 0.50360 | load: 0.00244 [I 2022-10-26 13:02:23] ...trap/engines/engine.py.230: loss: 0.34453 [I 2022-10-26 13:02:23] ...optimizers/trijoint.py.151: total_norm: 4.024327 [I 2022-10-26 13:02:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 920/2304 [I 2022-10-26 13:02:28] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:11:25 [I 2022-10-26 13:02:28] ...trap/engines/engine.py.229: process: 0.49612 | load: 0.00245 [I 2022-10-26 13:02:28] ...trap/engines/engine.py.230: loss: 0.19366 [I 2022-10-26 13:02:28] ...optimizers/trijoint.py.151: total_norm: 2.885567 [I 2022-10-26 13:02:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 930/2304 [I 2022-10-26 13:02:33] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:11:19 [I 2022-10-26 13:02:33] ...trap/engines/engine.py.229: process: 0.49024 | load: 0.00244 [I 2022-10-26 13:02:33] ...trap/engines/engine.py.230: loss: 0.26544 [I 2022-10-26 13:02:33] ...optimizers/trijoint.py.151: total_norm: 4.048245 [I 2022-10-26 13:02:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 940/2304 [I 2022-10-26 13:02:38] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:11:16 [I 2022-10-26 13:02:38] ...trap/engines/engine.py.229: process: 0.48740 | load: 0.00243 [I 2022-10-26 13:02:38] ...trap/engines/engine.py.230: loss: 0.22393 [I 2022-10-26 13:02:38] ...optimizers/trijoint.py.151: total_norm: 3.202186 [I 2022-10-26 13:02:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 950/2304 [I 2022-10-26 13:02:43] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:11:12 [I 2022-10-26 13:02:43] ...trap/engines/engine.py.229: process: 0.49506 | load: 0.00245 [I 2022-10-26 13:02:43] ...trap/engines/engine.py.230: loss: 0.25743 [I 2022-10-26 13:02:43] ...optimizers/trijoint.py.151: total_norm: 3.146398 [I 2022-10-26 13:02:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 960/2304 [I 2022-10-26 13:02:48] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:11:12 [I 2022-10-26 13:02:48] ...trap/engines/engine.py.229: process: 0.50439 | load: 0.00245 [I 2022-10-26 13:02:48] ...trap/engines/engine.py.230: loss: 0.25210 [I 2022-10-26 13:02:48] ...optimizers/trijoint.py.151: total_norm: 3.274980 [I 2022-10-26 13:02:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 970/2304 [I 2022-10-26 13:02:53] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:11:01 [I 2022-10-26 13:02:53] ...trap/engines/engine.py.229: process: 0.49533 | load: 0.00251 [I 2022-10-26 13:02:53] ...trap/engines/engine.py.230: loss: 0.23447 [I 2022-10-26 13:02:53] ...optimizers/trijoint.py.151: total_norm: 4.041707 [I 2022-10-26 13:02:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 980/2304 [I 2022-10-26 13:02:58] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:10:57 [I 2022-10-26 13:02:58] ...trap/engines/engine.py.229: process: 0.48554 | load: 0.00245 [I 2022-10-26 13:02:58] ...trap/engines/engine.py.230: loss: 0.23267 [I 2022-10-26 13:02:58] ...optimizers/trijoint.py.151: total_norm: 3.226024 [I 2022-10-26 13:03:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 990/2304 [I 2022-10-26 13:03:03] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:10:55 [I 2022-10-26 13:03:03] ...trap/engines/engine.py.229: process: 0.50388 | load: 0.00241 [I 2022-10-26 13:03:03] ...trap/engines/engine.py.230: loss: 0.30054 [I 2022-10-26 13:03:03] ...optimizers/trijoint.py.151: total_norm: 3.719694 [I 2022-10-26 13:03:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 1000/2304 [I 2022-10-26 13:03:08] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:10:47 [I 2022-10-26 13:03:08] ...trap/engines/engine.py.229: process: 0.49236 | load: 0.00241 [I 2022-10-26 13:03:08] ...trap/engines/engine.py.230: loss: 0.26938 [I 2022-10-26 13:03:08] ...optimizers/trijoint.py.151: total_norm: 2.708393 [I 2022-10-26 13:03:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 1010/2304 [I 2022-10-26 13:03:13] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:10:45 [I 2022-10-26 13:03:13] ...trap/engines/engine.py.229: process: 0.49600 | load: 0.00240 [I 2022-10-26 13:03:13] ...trap/engines/engine.py.230: loss: 0.17055 [I 2022-10-26 13:03:13] ...optimizers/trijoint.py.151: total_norm: 2.836421 [I 2022-10-26 13:03:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 1020/2304 [I 2022-10-26 13:03:18] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:10:40 [I 2022-10-26 13:03:18] ...trap/engines/engine.py.229: process: 0.50138 | load: 0.00251 [I 2022-10-26 13:03:18] ...trap/engines/engine.py.230: loss: 0.24509 [I 2022-10-26 13:03:18] ...optimizers/trijoint.py.151: total_norm: 2.892466 [I 2022-10-26 13:03:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 1030/2304 [I 2022-10-26 13:03:23] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:10:31 [I 2022-10-26 13:03:23] ...trap/engines/engine.py.229: process: 0.49542 | load: 0.00251 [I 2022-10-26 13:03:23] ...trap/engines/engine.py.230: loss: 0.24359 [I 2022-10-26 13:03:23] ...optimizers/trijoint.py.151: total_norm: 2.736551 [I 2022-10-26 13:03:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 1040/2304 [I 2022-10-26 13:03:28] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:10:25 [I 2022-10-26 13:03:28] ...trap/engines/engine.py.229: process: 0.48208 | load: 0.00250 [I 2022-10-26 13:03:28] ...trap/engines/engine.py.230: loss: 0.27168 [I 2022-10-26 13:03:28] ...optimizers/trijoint.py.151: total_norm: 3.786453 [I 2022-10-26 13:03:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 1050/2304 [I 2022-10-26 13:03:33] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:10:23 [I 2022-10-26 13:03:33] ...trap/engines/engine.py.229: process: 0.49926 | load: 0.00246 [I 2022-10-26 13:03:33] ...trap/engines/engine.py.230: loss: 0.26093 [I 2022-10-26 13:03:33] ...optimizers/trijoint.py.151: total_norm: 3.233547 [I 2022-10-26 13:03:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 1060/2304 [I 2022-10-26 13:03:38] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:10:26 [I 2022-10-26 13:03:38] ...trap/engines/engine.py.229: process: 0.50281 | load: 0.00244 [I 2022-10-26 13:03:38] ...trap/engines/engine.py.230: loss: 0.25312 [I 2022-10-26 13:03:38] ...optimizers/trijoint.py.151: total_norm: 3.213191 [I 2022-10-26 13:03:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 1070/2304 [I 2022-10-26 13:03:43] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:10:13 [I 2022-10-26 13:03:43] ...trap/engines/engine.py.229: process: 0.48834 | load: 0.00244 [I 2022-10-26 13:03:43] ...trap/engines/engine.py.230: loss: 0.24364 [I 2022-10-26 13:03:43] ...optimizers/trijoint.py.151: total_norm: 3.594775 [I 2022-10-26 13:03:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 1080/2304 [I 2022-10-26 13:03:48] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:10:08 [I 2022-10-26 13:03:48] ...trap/engines/engine.py.229: process: 0.49435 | load: 0.00247 [I 2022-10-26 13:03:48] ...trap/engines/engine.py.230: loss: 0.32989 [I 2022-10-26 13:03:48] ...optimizers/trijoint.py.151: total_norm: 4.385678 [I 2022-10-26 13:03:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 1090/2304 [I 2022-10-26 13:03:53] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:10:07 [I 2022-10-26 13:03:53] ...trap/engines/engine.py.229: process: 0.50000 | load: 0.00244 [I 2022-10-26 13:03:53] ...trap/engines/engine.py.230: loss: 0.26812 [I 2022-10-26 13:03:53] ...optimizers/trijoint.py.151: total_norm: 3.771797 [I 2022-10-26 13:03:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 1100/2304 [I 2022-10-26 13:03:58] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:10:02 [I 2022-10-26 13:03:58] ...trap/engines/engine.py.229: process: 0.49745 | load: 0.00242 [I 2022-10-26 13:03:58] ...trap/engines/engine.py.230: loss: 0.25099 [I 2022-10-26 13:03:58] ...optimizers/trijoint.py.151: total_norm: 3.518680 [I 2022-10-26 13:04:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 1110/2304 [I 2022-10-26 13:04:03] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:10:01 [I 2022-10-26 13:04:03] ...trap/engines/engine.py.229: process: 0.50258 | load: 0.00246 [I 2022-10-26 13:04:03] ...trap/engines/engine.py.230: loss: 0.31448 [I 2022-10-26 13:04:03] ...optimizers/trijoint.py.151: total_norm: 4.348752 [I 2022-10-26 13:04:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 1120/2304 [I 2022-10-26 13:04:08] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:09:47 [I 2022-10-26 13:04:08] ...trap/engines/engine.py.229: process: 0.48896 | load: 0.00249 [I 2022-10-26 13:04:08] ...trap/engines/engine.py.230: loss: 0.20199 [I 2022-10-26 13:04:08] ...optimizers/trijoint.py.151: total_norm: 2.799358 [I 2022-10-26 13:04:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 1130/2304 [I 2022-10-26 13:04:13] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:09:42 [I 2022-10-26 13:04:13] ...trap/engines/engine.py.229: process: 0.49495 | load: 0.00246 [I 2022-10-26 13:04:13] ...trap/engines/engine.py.230: loss: 0.21972 [I 2022-10-26 13:04:13] ...optimizers/trijoint.py.151: total_norm: 2.874866 [I 2022-10-26 13:04:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 1140/2304 [I 2022-10-26 13:04:18] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:09:38 [I 2022-10-26 13:04:18] ...trap/engines/engine.py.229: process: 0.49381 | load: 0.00244 [I 2022-10-26 13:04:18] ...trap/engines/engine.py.230: loss: 0.25710 [I 2022-10-26 13:04:18] ...optimizers/trijoint.py.151: total_norm: 3.547185 [I 2022-10-26 13:04:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 1150/2304 [I 2022-10-26 13:04:23] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:09:31 [I 2022-10-26 13:04:23] ...trap/engines/engine.py.229: process: 0.49464 | load: 0.00245 [I 2022-10-26 13:04:23] ...trap/engines/engine.py.230: loss: 0.25258 [I 2022-10-26 13:04:23] ...optimizers/trijoint.py.151: total_norm: 3.456735 [I 2022-10-26 13:04:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 1160/2304 [I 2022-10-26 13:04:28] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:09:32 [I 2022-10-26 13:04:28] ...trap/engines/engine.py.229: process: 0.50302 | load: 0.00252 [I 2022-10-26 13:04:28] ...trap/engines/engine.py.230: loss: 0.24756 [I 2022-10-26 13:04:28] ...optimizers/trijoint.py.151: total_norm: 3.519839 [I 2022-10-26 13:04:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 1170/2304 [I 2022-10-26 13:04:33] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:09:25 [I 2022-10-26 13:04:33] ...trap/engines/engine.py.229: process: 0.50136 | load: 0.00246 [I 2022-10-26 13:04:33] ...trap/engines/engine.py.230: loss: 0.24339 [I 2022-10-26 13:04:33] ...optimizers/trijoint.py.151: total_norm: 3.895255 [I 2022-10-26 13:04:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 1180/2304 [I 2022-10-26 13:04:38] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:09:20 [I 2022-10-26 13:04:38] ...trap/engines/engine.py.229: process: 0.49159 | load: 0.00244 [I 2022-10-26 13:04:38] ...trap/engines/engine.py.230: loss: 0.23341 [I 2022-10-26 13:04:38] ...optimizers/trijoint.py.151: total_norm: 3.207142 [I 2022-10-26 13:04:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 1190/2304 [I 2022-10-26 13:04:43] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:09:16 [I 2022-10-26 13:04:43] ...trap/engines/engine.py.229: process: 0.49900 | load: 0.00245 [I 2022-10-26 13:04:43] ...trap/engines/engine.py.230: loss: 0.24410 [I 2022-10-26 13:04:43] ...optimizers/trijoint.py.151: total_norm: 2.927908 [I 2022-10-26 13:04:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 1200/2304 [I 2022-10-26 13:04:48] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:09:09 [I 2022-10-26 13:04:48] ...trap/engines/engine.py.229: process: 0.50181 | load: 0.00242 [I 2022-10-26 13:04:48] ...trap/engines/engine.py.230: loss: 0.22524 [I 2022-10-26 13:04:48] ...optimizers/trijoint.py.151: total_norm: 3.719249 [I 2022-10-26 13:04:53] ...trap/engines/engine.py.224: train: epoch 9 | batch 1210/2304 [I 2022-10-26 13:04:53] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:09:07 [I 2022-10-26 13:04:53] ...trap/engines/engine.py.229: process: 0.51215 | load: 0.00243 [I 2022-10-26 13:04:53] ...trap/engines/engine.py.230: loss: 0.18733 [I 2022-10-26 13:04:53] ...optimizers/trijoint.py.151: total_norm: 3.126073 [I 2022-10-26 13:04:58] ...trap/engines/engine.py.224: train: epoch 9 | batch 1220/2304 [I 2022-10-26 13:04:58] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:08:57 [I 2022-10-26 13:04:58] ...trap/engines/engine.py.229: process: 0.48696 | load: 0.00250 [I 2022-10-26 13:04:58] ...trap/engines/engine.py.230: loss: 0.21010 [I 2022-10-26 13:04:58] ...optimizers/trijoint.py.151: total_norm: 3.494936 [I 2022-10-26 13:05:03] ...trap/engines/engine.py.224: train: epoch 9 | batch 1230/2304 [I 2022-10-26 13:05:03] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:08:55 [I 2022-10-26 13:05:03] ...trap/engines/engine.py.229: process: 0.49754 | load: 0.00249 [I 2022-10-26 13:05:03] ...trap/engines/engine.py.230: loss: 0.32049 [I 2022-10-26 13:05:03] ...optimizers/trijoint.py.151: total_norm: 3.662668 [I 2022-10-26 13:05:08] ...trap/engines/engine.py.224: train: epoch 9 | batch 1240/2304 [I 2022-10-26 13:05:08] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:09:06 [I 2022-10-26 13:05:08] ...trap/engines/engine.py.229: process: 0.50723 | load: 0.00242 [I 2022-10-26 13:05:08] ...trap/engines/engine.py.230: loss: 0.17900 [I 2022-10-26 13:05:08] ...optimizers/trijoint.py.151: total_norm: 2.767656 [I 2022-10-26 13:05:13] ...trap/engines/engine.py.224: train: epoch 9 | batch 1250/2304 [I 2022-10-26 13:05:13] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:08:46 [I 2022-10-26 13:05:13] ...trap/engines/engine.py.229: process: 0.50012 | load: 0.00245 [I 2022-10-26 13:05:13] ...trap/engines/engine.py.230: loss: 0.27554 [I 2022-10-26 13:05:13] ...optimizers/trijoint.py.151: total_norm: 3.841497 [I 2022-10-26 13:05:18] ...trap/engines/engine.py.224: train: epoch 9 | batch 1260/2304 [I 2022-10-26 13:05:18] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:08:55 [I 2022-10-26 13:05:18] ...trap/engines/engine.py.229: process: 0.50244 | load: 0.00245 [I 2022-10-26 13:05:18] ...trap/engines/engine.py.230: loss: 0.26437 [I 2022-10-26 13:05:18] ...optimizers/trijoint.py.151: total_norm: 4.414226 [I 2022-10-26 13:05:23] ...trap/engines/engine.py.224: train: epoch 9 | batch 1270/2304 [I 2022-10-26 13:05:23] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:08:38 [I 2022-10-26 13:05:23] ...trap/engines/engine.py.229: process: 0.48773 | load: 0.00241 [I 2022-10-26 13:05:23] ...trap/engines/engine.py.230: loss: 0.26311 [I 2022-10-26 13:05:23] ...optimizers/trijoint.py.151: total_norm: 4.393978 [I 2022-10-26 13:05:28] ...trap/engines/engine.py.224: train: epoch 9 | batch 1280/2304 [I 2022-10-26 13:05:28] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:08:28 [I 2022-10-26 13:05:28] ...trap/engines/engine.py.229: process: 0.48891 | load: 0.00240 [I 2022-10-26 13:05:28] ...trap/engines/engine.py.230: loss: 0.35060 [I 2022-10-26 13:05:28] ...optimizers/trijoint.py.151: total_norm: 4.680524 [I 2022-10-26 13:05:33] ...trap/engines/engine.py.224: train: epoch 9 | batch 1290/2304 [I 2022-10-26 13:05:33] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:08:28 [I 2022-10-26 13:05:33] ...trap/engines/engine.py.229: process: 0.49493 | load: 0.00242 [I 2022-10-26 13:05:33] ...trap/engines/engine.py.230: loss: 0.20051 [I 2022-10-26 13:05:33] ...optimizers/trijoint.py.151: total_norm: 3.197717 [I 2022-10-26 13:05:38] ...trap/engines/engine.py.224: train: epoch 9 | batch 1300/2304 [I 2022-10-26 13:05:38] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:08:21 [I 2022-10-26 13:05:38] ...trap/engines/engine.py.229: process: 0.50516 | load: 0.00242 [I 2022-10-26 13:05:38] ...trap/engines/engine.py.230: loss: 0.24715 [I 2022-10-26 13:05:38] ...optimizers/trijoint.py.151: total_norm: 3.935833 [I 2022-10-26 13:05:43] ...trap/engines/engine.py.224: train: epoch 9 | batch 1310/2304 [I 2022-10-26 13:05:43] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:08:18 [I 2022-10-26 13:05:43] ...trap/engines/engine.py.229: process: 0.50331 | load: 0.00262 [I 2022-10-26 13:05:43] ...trap/engines/engine.py.230: loss: 0.19044 [I 2022-10-26 13:05:43] ...optimizers/trijoint.py.151: total_norm: 2.247145 [I 2022-10-26 13:05:48] ...trap/engines/engine.py.224: train: epoch 9 | batch 1320/2304 [I 2022-10-26 13:05:48] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:08:09 [I 2022-10-26 13:05:48] ...trap/engines/engine.py.229: process: 0.49149 | load: 0.00246 [I 2022-10-26 13:05:48] ...trap/engines/engine.py.230: loss: 0.22878 [I 2022-10-26 13:05:48] ...optimizers/trijoint.py.151: total_norm: 3.034210 [I 2022-10-26 13:05:54] ...trap/engines/engine.py.224: train: epoch 9 | batch 1330/2304 [I 2022-10-26 13:05:54] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:08:11 [I 2022-10-26 13:05:54] ...trap/engines/engine.py.229: process: 0.49817 | load: 0.00243 [I 2022-10-26 13:05:54] ...trap/engines/engine.py.230: loss: 0.22750 [I 2022-10-26 13:05:54] ...optimizers/trijoint.py.151: total_norm: 3.719917 [I 2022-10-26 13:05:59] ...trap/engines/engine.py.224: train: epoch 9 | batch 1340/2304 [I 2022-10-26 13:05:59] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:08:02 [I 2022-10-26 13:05:59] ...trap/engines/engine.py.229: process: 0.49226 | load: 0.00244 [I 2022-10-26 13:05:59] ...trap/engines/engine.py.230: loss: 0.29226 [I 2022-10-26 13:05:59] ...optimizers/trijoint.py.151: total_norm: 3.828795 [I 2022-10-26 13:06:04] ...trap/engines/engine.py.224: train: epoch 9 | batch 1350/2304 [I 2022-10-26 13:06:04] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:07:57 [I 2022-10-26 13:06:04] ...trap/engines/engine.py.229: process: 0.49805 | load: 0.00243 [I 2022-10-26 13:06:04] ...trap/engines/engine.py.230: loss: 0.24326 [I 2022-10-26 13:06:04] ...optimizers/trijoint.py.151: total_norm: 3.681501 [I 2022-10-26 13:06:09] ...trap/engines/engine.py.224: train: epoch 9 | batch 1360/2304 [I 2022-10-26 13:06:09] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:07:50 [I 2022-10-26 13:06:09] ...trap/engines/engine.py.229: process: 0.50083 | load: 0.00242 [I 2022-10-26 13:06:09] ...trap/engines/engine.py.230: loss: 0.22268 [I 2022-10-26 13:06:09] ...optimizers/trijoint.py.151: total_norm: 3.028378 [I 2022-10-26 13:06:14] ...trap/engines/engine.py.224: train: epoch 9 | batch 1370/2304 [I 2022-10-26 13:06:14] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:07:46 [I 2022-10-26 13:06:14] ...trap/engines/engine.py.229: process: 0.49495 | load: 0.00244 [I 2022-10-26 13:06:14] ...trap/engines/engine.py.230: loss: 0.17170 [I 2022-10-26 13:06:14] ...optimizers/trijoint.py.151: total_norm: 2.944943 [I 2022-10-26 13:06:19] ...trap/engines/engine.py.224: train: epoch 9 | batch 1380/2304 [I 2022-10-26 13:06:19] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:07:43 [I 2022-10-26 13:06:19] ...trap/engines/engine.py.229: process: 0.50912 | load: 0.00245 [I 2022-10-26 13:06:19] ...trap/engines/engine.py.230: loss: 0.25160 [I 2022-10-26 13:06:19] ...optimizers/trijoint.py.151: total_norm: 3.236267 [I 2022-10-26 13:06:24] ...trap/engines/engine.py.224: train: epoch 9 | batch 1390/2304 [I 2022-10-26 13:06:24] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:07:37 [I 2022-10-26 13:06:24] ...trap/engines/engine.py.229: process: 0.50290 | load: 0.00245 [I 2022-10-26 13:06:24] ...trap/engines/engine.py.230: loss: 0.23804 [I 2022-10-26 13:06:24] ...optimizers/trijoint.py.151: total_norm: 3.441667 [I 2022-10-26 13:06:29] ...trap/engines/engine.py.224: train: epoch 9 | batch 1400/2304 [I 2022-10-26 13:06:29] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:07:45 [I 2022-10-26 13:06:29] ...trap/engines/engine.py.229: process: 0.50200 | load: 0.00243 [I 2022-10-26 13:06:29] ...trap/engines/engine.py.230: loss: 0.20712 [I 2022-10-26 13:06:29] ...optimizers/trijoint.py.151: total_norm: 2.804565 [I 2022-10-26 13:06:34] ...trap/engines/engine.py.224: train: epoch 9 | batch 1410/2304 [I 2022-10-26 13:06:34] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:07:29 [I 2022-10-26 13:06:34] ...trap/engines/engine.py.229: process: 0.50375 | load: 0.00244 [I 2022-10-26 13:06:34] ...trap/engines/engine.py.230: loss: 0.22190 [I 2022-10-26 13:06:34] ...optimizers/trijoint.py.151: total_norm: 3.389545 [I 2022-10-26 13:06:39] ...trap/engines/engine.py.224: train: epoch 9 | batch 1420/2304 [I 2022-10-26 13:06:39] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:07:19 [I 2022-10-26 13:06:39] ...trap/engines/engine.py.229: process: 0.48814 | load: 0.00249 [I 2022-10-26 13:06:39] ...trap/engines/engine.py.230: loss: 0.25992 [I 2022-10-26 13:06:39] ...optimizers/trijoint.py.151: total_norm: 3.313945 [I 2022-10-26 13:06:44] ...trap/engines/engine.py.224: train: epoch 9 | batch 1430/2304 [I 2022-10-26 13:06:44] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:07:14 [I 2022-10-26 13:06:44] ...trap/engines/engine.py.229: process: 0.50364 | load: 0.00248 [I 2022-10-26 13:06:44] ...trap/engines/engine.py.230: loss: 0.20418 [I 2022-10-26 13:06:44] ...optimizers/trijoint.py.151: total_norm: 2.664706 [I 2022-10-26 13:06:49] ...trap/engines/engine.py.224: train: epoch 9 | batch 1440/2304 [I 2022-10-26 13:06:49] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:07:08 [I 2022-10-26 13:06:49] ...trap/engines/engine.py.229: process: 0.49606 | load: 0.00246 [I 2022-10-26 13:06:49] ...trap/engines/engine.py.230: loss: 0.26912 [I 2022-10-26 13:06:49] ...optimizers/trijoint.py.151: total_norm: 3.936422 [I 2022-10-26 13:06:54] ...trap/engines/engine.py.224: train: epoch 9 | batch 1450/2304 [I 2022-10-26 13:06:54] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:07:04 [I 2022-10-26 13:06:54] ...trap/engines/engine.py.229: process: 0.49325 | load: 0.00245 [I 2022-10-26 13:06:54] ...trap/engines/engine.py.230: loss: 0.17238 [I 2022-10-26 13:06:54] ...optimizers/trijoint.py.151: total_norm: 2.673497 [I 2022-10-26 13:06:59] ...trap/engines/engine.py.224: train: epoch 9 | batch 1460/2304 [I 2022-10-26 13:06:59] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:07:00 [I 2022-10-26 13:06:59] ...trap/engines/engine.py.229: process: 0.49201 | load: 0.00247 [I 2022-10-26 13:06:59] ...trap/engines/engine.py.230: loss: 0.22798 [I 2022-10-26 13:06:59] ...optimizers/trijoint.py.151: total_norm: 3.303919 [I 2022-10-26 13:07:04] ...trap/engines/engine.py.224: train: epoch 9 | batch 1470/2304 [I 2022-10-26 13:07:04] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:07:01 [I 2022-10-26 13:07:04] ...trap/engines/engine.py.229: process: 0.54474 | load: 0.00245 [I 2022-10-26 13:07:04] ...trap/engines/engine.py.230: loss: 0.21549 [I 2022-10-26 13:07:04] ...optimizers/trijoint.py.151: total_norm: 3.013031 [I 2022-10-26 13:07:09] ...trap/engines/engine.py.224: train: epoch 9 | batch 1480/2304 [I 2022-10-26 13:07:09] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:06:54 [I 2022-10-26 13:07:09] ...trap/engines/engine.py.229: process: 0.49806 | load: 0.00244 [I 2022-10-26 13:07:09] ...trap/engines/engine.py.230: loss: 0.19694 [I 2022-10-26 13:07:09] ...optimizers/trijoint.py.151: total_norm: 2.941337 [I 2022-10-26 13:07:14] ...trap/engines/engine.py.224: train: epoch 9 | batch 1490/2304 [I 2022-10-26 13:07:14] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:06:45 [I 2022-10-26 13:07:14] ...trap/engines/engine.py.229: process: 0.48447 | load: 0.00245 [I 2022-10-26 13:07:14] ...trap/engines/engine.py.230: loss: 0.22295 [I 2022-10-26 13:07:14] ...optimizers/trijoint.py.151: total_norm: 3.066437 [I 2022-10-26 13:07:19] ...trap/engines/engine.py.224: train: epoch 9 | batch 1500/2304 [I 2022-10-26 13:07:19] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:06:41 [I 2022-10-26 13:07:19] ...trap/engines/engine.py.229: process: 0.49577 | load: 0.00243 [I 2022-10-26 13:07:19] ...trap/engines/engine.py.230: loss: 0.18402 [I 2022-10-26 13:07:19] ...optimizers/trijoint.py.151: total_norm: 3.174351 [I 2022-10-26 13:07:24] ...trap/engines/engine.py.224: train: epoch 9 | batch 1510/2304 [I 2022-10-26 13:07:24] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:06:37 [I 2022-10-26 13:07:24] ...trap/engines/engine.py.229: process: 0.50633 | load: 0.00242 [I 2022-10-26 13:07:24] ...trap/engines/engine.py.230: loss: 0.22227 [I 2022-10-26 13:07:24] ...optimizers/trijoint.py.151: total_norm: 2.794425 [I 2022-10-26 13:07:29] ...trap/engines/engine.py.224: train: epoch 9 | batch 1520/2304 [I 2022-10-26 13:07:29] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:06:31 [I 2022-10-26 13:07:29] ...trap/engines/engine.py.229: process: 0.50278 | load: 0.00239 [I 2022-10-26 13:07:29] ...trap/engines/engine.py.230: loss: 0.29932 [I 2022-10-26 13:07:29] ...optimizers/trijoint.py.151: total_norm: 3.444268 [I 2022-10-26 13:07:34] ...trap/engines/engine.py.224: train: epoch 9 | batch 1530/2304 [I 2022-10-26 13:07:34] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:06:25 [I 2022-10-26 13:07:34] ...trap/engines/engine.py.229: process: 0.49377 | load: 0.00243 [I 2022-10-26 13:07:34] ...trap/engines/engine.py.230: loss: 0.25072 [I 2022-10-26 13:07:34] ...optimizers/trijoint.py.151: total_norm: 3.326582 [I 2022-10-26 13:07:39] ...trap/engines/engine.py.224: train: epoch 9 | batch 1540/2304 [I 2022-10-26 13:07:39] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:06:20 [I 2022-10-26 13:07:39] ...trap/engines/engine.py.229: process: 0.49750 | load: 0.00244 [I 2022-10-26 13:07:39] ...trap/engines/engine.py.230: loss: 0.24077 [I 2022-10-26 13:07:39] ...optimizers/trijoint.py.151: total_norm: 2.898436 [I 2022-10-26 13:07:44] ...trap/engines/engine.py.224: train: epoch 9 | batch 1550/2304 [I 2022-10-26 13:07:44] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:06:20 [I 2022-10-26 13:07:44] ...trap/engines/engine.py.229: process: 0.49256 | load: 0.00245 [I 2022-10-26 13:07:44] ...trap/engines/engine.py.230: loss: 0.17051 [I 2022-10-26 13:07:44] ...optimizers/trijoint.py.151: total_norm: 2.677474 [I 2022-10-26 13:07:49] ...trap/engines/engine.py.224: train: epoch 9 | batch 1560/2304 [I 2022-10-26 13:07:49] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:06:12 [I 2022-10-26 13:07:49] ...trap/engines/engine.py.229: process: 0.50426 | load: 0.00244 [I 2022-10-26 13:07:49] ...trap/engines/engine.py.230: loss: 0.24634 [I 2022-10-26 13:07:49] ...optimizers/trijoint.py.151: total_norm: 3.027722 [I 2022-10-26 13:07:54] ...trap/engines/engine.py.224: train: epoch 9 | batch 1570/2304 [I 2022-10-26 13:07:54] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:06:05 [I 2022-10-26 13:07:54] ...trap/engines/engine.py.229: process: 0.50448 | load: 0.00258 [I 2022-10-26 13:07:54] ...trap/engines/engine.py.230: loss: 0.34439 [I 2022-10-26 13:07:54] ...optimizers/trijoint.py.151: total_norm: 4.330558 [I 2022-10-26 13:07:59] ...trap/engines/engine.py.224: train: epoch 9 | batch 1580/2304 [I 2022-10-26 13:07:59] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:06:00 [I 2022-10-26 13:07:59] ...trap/engines/engine.py.229: process: 0.50303 | load: 0.00243 [I 2022-10-26 13:07:59] ...trap/engines/engine.py.230: loss: 0.24019 [I 2022-10-26 13:07:59] ...optimizers/trijoint.py.151: total_norm: 3.200802 [I 2022-10-26 13:08:04] ...trap/engines/engine.py.224: train: epoch 9 | batch 1590/2304 [I 2022-10-26 13:08:04] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:05:55 [I 2022-10-26 13:08:04] ...trap/engines/engine.py.229: process: 0.50382 | load: 0.00244 [I 2022-10-26 13:08:04] ...trap/engines/engine.py.230: loss: 0.18816 [I 2022-10-26 13:08:04] ...optimizers/trijoint.py.151: total_norm: 2.581100 [I 2022-10-26 13:08:09] ...trap/engines/engine.py.224: train: epoch 9 | batch 1600/2304 [I 2022-10-26 13:08:09] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:05:53 [I 2022-10-26 13:08:09] ...trap/engines/engine.py.229: process: 0.49945 | load: 0.00260 [I 2022-10-26 13:08:09] ...trap/engines/engine.py.230: loss: 0.26835 [I 2022-10-26 13:08:09] ...optimizers/trijoint.py.151: total_norm: 3.007450 [I 2022-10-26 13:08:14] ...trap/engines/engine.py.224: train: epoch 9 | batch 1610/2304 [I 2022-10-26 13:08:14] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:05:45 [I 2022-10-26 13:08:14] ...trap/engines/engine.py.229: process: 0.48413 | load: 0.00248 [I 2022-10-26 13:08:14] ...trap/engines/engine.py.230: loss: 0.23268 [I 2022-10-26 13:08:14] ...optimizers/trijoint.py.151: total_norm: 3.441523 [I 2022-10-26 13:08:19] ...trap/engines/engine.py.224: train: epoch 9 | batch 1620/2304 [I 2022-10-26 13:08:19] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:05:48 [I 2022-10-26 13:08:19] ...trap/engines/engine.py.229: process: 0.50710 | load: 0.00246 [I 2022-10-26 13:08:19] ...trap/engines/engine.py.230: loss: 0.32106 [I 2022-10-26 13:08:19] ...optimizers/trijoint.py.151: total_norm: 3.763054 [I 2022-10-26 13:08:24] ...trap/engines/engine.py.224: train: epoch 9 | batch 1630/2304 [I 2022-10-26 13:08:24] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:05:37 [I 2022-10-26 13:08:24] ...trap/engines/engine.py.229: process: 0.50515 | load: 0.00245 [I 2022-10-26 13:08:24] ...trap/engines/engine.py.230: loss: 0.15773 [I 2022-10-26 13:08:24] ...optimizers/trijoint.py.151: total_norm: 2.287589 [I 2022-10-26 13:08:29] ...trap/engines/engine.py.224: train: epoch 9 | batch 1640/2304 [I 2022-10-26 13:08:29] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:05:32 [I 2022-10-26 13:08:29] ...trap/engines/engine.py.229: process: 0.50081 | load: 0.00244 [I 2022-10-26 13:08:29] ...trap/engines/engine.py.230: loss: 0.22522 [I 2022-10-26 13:08:29] ...optimizers/trijoint.py.151: total_norm: 3.333584 [I 2022-10-26 13:08:34] ...trap/engines/engine.py.224: train: epoch 9 | batch 1650/2304 [I 2022-10-26 13:08:34] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:05:28 [I 2022-10-26 13:08:34] ...trap/engines/engine.py.229: process: 0.50535 | load: 0.00244 [I 2022-10-26 13:08:34] ...trap/engines/engine.py.230: loss: 0.25680 [I 2022-10-26 13:08:34] ...optimizers/trijoint.py.151: total_norm: 4.121644 [I 2022-10-26 13:08:39] ...trap/engines/engine.py.224: train: epoch 9 | batch 1660/2304 [I 2022-10-26 13:08:39] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:05:23 [I 2022-10-26 13:08:39] ...trap/engines/engine.py.229: process: 0.50381 | load: 0.00242 [I 2022-10-26 13:08:39] ...trap/engines/engine.py.230: loss: 0.24149 [I 2022-10-26 13:08:39] ...optimizers/trijoint.py.151: total_norm: 3.007919 [I 2022-10-26 13:08:44] ...trap/engines/engine.py.224: train: epoch 9 | batch 1670/2304 [I 2022-10-26 13:08:44] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:05:18 [I 2022-10-26 13:08:44] ...trap/engines/engine.py.229: process: 0.49532 | load: 0.00242 [I 2022-10-26 13:08:44] ...trap/engines/engine.py.230: loss: 0.26409 [I 2022-10-26 13:08:44] ...optimizers/trijoint.py.151: total_norm: 4.001624 [I 2022-10-26 13:08:49] ...trap/engines/engine.py.224: train: epoch 9 | batch 1680/2304 [I 2022-10-26 13:08:49] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:05:11 [I 2022-10-26 13:08:49] ...trap/engines/engine.py.229: process: 0.50489 | load: 0.00246 [I 2022-10-26 13:08:49] ...trap/engines/engine.py.230: loss: 0.27372 [I 2022-10-26 13:08:49] ...optimizers/trijoint.py.151: total_norm: 3.350356 [I 2022-10-26 13:08:54] ...trap/engines/engine.py.224: train: epoch 9 | batch 1690/2304 [I 2022-10-26 13:08:54] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:05:10 [I 2022-10-26 13:08:54] ...trap/engines/engine.py.229: process: 0.49708 | load: 0.00261 [I 2022-10-26 13:08:54] ...trap/engines/engine.py.230: loss: 0.26248 [I 2022-10-26 13:08:54] ...optimizers/trijoint.py.151: total_norm: 3.480379 [I 2022-10-26 13:08:59] ...trap/engines/engine.py.224: train: epoch 9 | batch 1700/2304 [I 2022-10-26 13:08:59] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:05:01 [I 2022-10-26 13:08:59] ...trap/engines/engine.py.229: process: 0.48809 | load: 0.00243 [I 2022-10-26 13:08:59] ...trap/engines/engine.py.230: loss: 0.20586 [I 2022-10-26 13:08:59] ...optimizers/trijoint.py.151: total_norm: 3.263880 [I 2022-10-26 13:09:04] ...trap/engines/engine.py.224: train: epoch 9 | batch 1710/2304 [I 2022-10-26 13:09:04] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:04:58 [I 2022-10-26 13:09:04] ...trap/engines/engine.py.229: process: 0.50061 | load: 0.00241 [I 2022-10-26 13:09:04] ...trap/engines/engine.py.230: loss: 0.25462 [I 2022-10-26 13:09:04] ...optimizers/trijoint.py.151: total_norm: 4.704261 [I 2022-10-26 13:09:09] ...trap/engines/engine.py.224: train: epoch 9 | batch 1720/2304 [I 2022-10-26 13:09:10] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:04:53 [I 2022-10-26 13:09:10] ...trap/engines/engine.py.229: process: 0.49261 | load: 0.00243 [I 2022-10-26 13:09:10] ...trap/engines/engine.py.230: loss: 0.25143 [I 2022-10-26 13:09:10] ...optimizers/trijoint.py.151: total_norm: 3.692281 [I 2022-10-26 13:09:15] ...trap/engines/engine.py.224: train: epoch 9 | batch 1730/2304 [I 2022-10-26 13:09:15] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:04:49 [I 2022-10-26 13:09:15] ...trap/engines/engine.py.229: process: 0.50430 | load: 0.00244 [I 2022-10-26 13:09:15] ...trap/engines/engine.py.230: loss: 0.19990 [I 2022-10-26 13:09:15] ...optimizers/trijoint.py.151: total_norm: 3.029820 [I 2022-10-26 13:09:20] ...trap/engines/engine.py.224: train: epoch 9 | batch 1740/2304 [I 2022-10-26 13:09:20] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:04:42 [I 2022-10-26 13:09:20] ...trap/engines/engine.py.229: process: 0.50627 | load: 0.00241 [I 2022-10-26 13:09:20] ...trap/engines/engine.py.230: loss: 0.19057 [I 2022-10-26 13:09:20] ...optimizers/trijoint.py.151: total_norm: 2.746300 [I 2022-10-26 13:09:25] ...trap/engines/engine.py.224: train: epoch 9 | batch 1750/2304 [I 2022-10-26 13:09:25] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:04:36 [I 2022-10-26 13:09:25] ...trap/engines/engine.py.229: process: 0.49718 | load: 0.00247 [I 2022-10-26 13:09:25] ...trap/engines/engine.py.230: loss: 0.38518 [I 2022-10-26 13:09:25] ...optimizers/trijoint.py.151: total_norm: 4.021915 [I 2022-10-26 13:09:30] ...trap/engines/engine.py.224: train: epoch 9 | batch 1760/2304 [I 2022-10-26 13:09:30] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:04:39 [I 2022-10-26 13:09:30] ...trap/engines/engine.py.229: process: 0.50344 | load: 0.00245 [I 2022-10-26 13:09:30] ...trap/engines/engine.py.230: loss: 0.19211 [I 2022-10-26 13:09:30] ...optimizers/trijoint.py.151: total_norm: 3.135834 [I 2022-10-26 13:09:35] ...trap/engines/engine.py.224: train: epoch 9 | batch 1770/2304 [I 2022-10-26 13:09:35] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:04:24 [I 2022-10-26 13:09:35] ...trap/engines/engine.py.229: process: 0.50133 | load: 0.00244 [I 2022-10-26 13:09:35] ...trap/engines/engine.py.230: loss: 0.20851 [I 2022-10-26 13:09:35] ...optimizers/trijoint.py.151: total_norm: 3.348778 [I 2022-10-26 13:09:40] ...trap/engines/engine.py.224: train: epoch 9 | batch 1780/2304 [I 2022-10-26 13:09:40] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:04:20 [I 2022-10-26 13:09:40] ...trap/engines/engine.py.229: process: 0.49414 | load: 0.00244 [I 2022-10-26 13:09:40] ...trap/engines/engine.py.230: loss: 0.20060 [I 2022-10-26 13:09:40] ...optimizers/trijoint.py.151: total_norm: 3.644459 [I 2022-10-26 13:09:45] ...trap/engines/engine.py.224: train: epoch 9 | batch 1790/2304 [I 2022-10-26 13:09:45] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:04:19 [I 2022-10-26 13:09:45] ...trap/engines/engine.py.229: process: 0.50397 | load: 0.00243 [I 2022-10-26 13:09:45] ...trap/engines/engine.py.230: loss: 0.24341 [I 2022-10-26 13:09:45] ...optimizers/trijoint.py.151: total_norm: 3.578914 [I 2022-10-26 13:09:50] ...trap/engines/engine.py.224: train: epoch 9 | batch 1800/2304 [I 2022-10-26 13:09:50] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:04:11 [I 2022-10-26 13:09:50] ...trap/engines/engine.py.229: process: 0.49850 | load: 0.00244 [I 2022-10-26 13:09:50] ...trap/engines/engine.py.230: loss: 0.25863 [I 2022-10-26 13:09:50] ...optimizers/trijoint.py.151: total_norm: 3.083304 [I 2022-10-26 13:09:55] ...trap/engines/engine.py.224: train: epoch 9 | batch 1810/2304 [I 2022-10-26 13:09:55] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:04:07 [I 2022-10-26 13:09:55] ...trap/engines/engine.py.229: process: 0.50200 | load: 0.00247 [I 2022-10-26 13:09:55] ...trap/engines/engine.py.230: loss: 0.16762 [I 2022-10-26 13:09:55] ...optimizers/trijoint.py.151: total_norm: 2.427581 [I 2022-10-26 13:10:00] ...trap/engines/engine.py.224: train: epoch 9 | batch 1820/2304 [I 2022-10-26 13:10:00] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:04:02 [I 2022-10-26 13:10:00] ...trap/engines/engine.py.229: process: 0.49300 | load: 0.00252 [I 2022-10-26 13:10:00] ...trap/engines/engine.py.230: loss: 0.28596 [I 2022-10-26 13:10:00] ...optimizers/trijoint.py.151: total_norm: 5.971678 [I 2022-10-26 13:10:05] ...trap/engines/engine.py.224: train: epoch 9 | batch 1830/2304 [I 2022-10-26 13:10:05] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:04:02 [I 2022-10-26 13:10:05] ...trap/engines/engine.py.229: process: 0.49642 | load: 0.00246 [I 2022-10-26 13:10:05] ...trap/engines/engine.py.230: loss: 0.21465 [I 2022-10-26 13:10:05] ...optimizers/trijoint.py.151: total_norm: 3.073457 [I 2022-10-26 13:10:10] ...trap/engines/engine.py.224: train: epoch 9 | batch 1840/2304 [I 2022-10-26 13:10:10] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:03:51 [I 2022-10-26 13:10:10] ...trap/engines/engine.py.229: process: 0.49874 | load: 0.00246 [I 2022-10-26 13:10:10] ...trap/engines/engine.py.230: loss: 0.15808 [I 2022-10-26 13:10:10] ...optimizers/trijoint.py.151: total_norm: 2.653267 [I 2022-10-26 13:10:15] ...trap/engines/engine.py.224: train: epoch 9 | batch 1850/2304 [I 2022-10-26 13:10:15] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:03:45 [I 2022-10-26 13:10:15] ...trap/engines/engine.py.229: process: 0.49634 | load: 0.00264 [I 2022-10-26 13:10:15] ...trap/engines/engine.py.230: loss: 0.17635 [I 2022-10-26 13:10:15] ...optimizers/trijoint.py.151: total_norm: 2.674420 [I 2022-10-26 13:10:20] ...trap/engines/engine.py.224: train: epoch 9 | batch 1860/2304 [I 2022-10-26 13:10:20] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:03:40 [I 2022-10-26 13:10:20] ...trap/engines/engine.py.229: process: 0.49124 | load: 0.00246 [I 2022-10-26 13:10:20] ...trap/engines/engine.py.230: loss: 0.18234 [I 2022-10-26 13:10:20] ...optimizers/trijoint.py.151: total_norm: 3.193059 [I 2022-10-26 13:10:25] ...trap/engines/engine.py.224: train: epoch 9 | batch 1870/2304 [I 2022-10-26 13:10:25] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:03:35 [I 2022-10-26 13:10:25] ...trap/engines/engine.py.229: process: 0.49615 | load: 0.00242 [I 2022-10-26 13:10:25] ...trap/engines/engine.py.230: loss: 0.31734 [I 2022-10-26 13:10:25] ...optimizers/trijoint.py.151: total_norm: 3.745234 [I 2022-10-26 13:10:30] ...trap/engines/engine.py.224: train: epoch 9 | batch 1880/2304 [I 2022-10-26 13:10:30] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:03:30 [I 2022-10-26 13:10:30] ...trap/engines/engine.py.229: process: 0.48309 | load: 0.00245 [I 2022-10-26 13:10:30] ...trap/engines/engine.py.230: loss: 0.17869 [I 2022-10-26 13:10:30] ...optimizers/trijoint.py.151: total_norm: 3.707102 [I 2022-10-26 13:10:35] ...trap/engines/engine.py.224: train: epoch 9 | batch 1890/2304 [I 2022-10-26 13:10:35] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:03:26 [I 2022-10-26 13:10:35] ...trap/engines/engine.py.229: process: 0.50079 | load: 0.00243 [I 2022-10-26 13:10:35] ...trap/engines/engine.py.230: loss: 0.28331 [I 2022-10-26 13:10:35] ...optimizers/trijoint.py.151: total_norm: 4.416034 [I 2022-10-26 13:10:40] ...trap/engines/engine.py.224: train: epoch 9 | batch 1900/2304 [I 2022-10-26 13:10:40] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:03:23 [I 2022-10-26 13:10:40] ...trap/engines/engine.py.229: process: 0.50334 | load: 0.00254 [I 2022-10-26 13:10:40] ...trap/engines/engine.py.230: loss: 0.23083 [I 2022-10-26 13:10:40] ...optimizers/trijoint.py.151: total_norm: 4.107139 [I 2022-10-26 13:10:45] ...trap/engines/engine.py.224: train: epoch 9 | batch 1910/2304 [I 2022-10-26 13:10:45] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:03:16 [I 2022-10-26 13:10:45] ...trap/engines/engine.py.229: process: 0.49488 | load: 0.00247 [I 2022-10-26 13:10:45] ...trap/engines/engine.py.230: loss: 0.19970 [I 2022-10-26 13:10:45] ...optimizers/trijoint.py.151: total_norm: 3.028141 [I 2022-10-26 13:10:50] ...trap/engines/engine.py.224: train: epoch 9 | batch 1920/2304 [I 2022-10-26 13:10:50] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:03:10 [I 2022-10-26 13:10:50] ...trap/engines/engine.py.229: process: 0.49585 | load: 0.00245 [I 2022-10-26 13:10:50] ...trap/engines/engine.py.230: loss: 0.26954 [I 2022-10-26 13:10:50] ...optimizers/trijoint.py.151: total_norm: 4.571723 [I 2022-10-26 13:10:55] ...trap/engines/engine.py.224: train: epoch 9 | batch 1930/2304 [I 2022-10-26 13:10:55] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:03:06 [I 2022-10-26 13:10:55] ...trap/engines/engine.py.229: process: 0.50351 | load: 0.00244 [I 2022-10-26 13:10:55] ...trap/engines/engine.py.230: loss: 0.23193 [I 2022-10-26 13:10:55] ...optimizers/trijoint.py.151: total_norm: 3.925770 [I 2022-10-26 13:11:00] ...trap/engines/engine.py.224: train: epoch 9 | batch 1940/2304 [I 2022-10-26 13:11:00] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:03:02 [I 2022-10-26 13:11:00] ...trap/engines/engine.py.229: process: 0.50122 | load: 0.00242 [I 2022-10-26 13:11:00] ...trap/engines/engine.py.230: loss: 0.21619 [I 2022-10-26 13:11:00] ...optimizers/trijoint.py.151: total_norm: 3.729991 [I 2022-10-26 13:11:05] ...trap/engines/engine.py.224: train: epoch 9 | batch 1950/2304 [I 2022-10-26 13:11:05] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:02:56 [I 2022-10-26 13:11:05] ...trap/engines/engine.py.229: process: 0.50575 | load: 0.00246 [I 2022-10-26 13:11:05] ...trap/engines/engine.py.230: loss: 0.20334 [I 2022-10-26 13:11:05] ...optimizers/trijoint.py.151: total_norm: 3.260953 [I 2022-10-26 13:11:10] ...trap/engines/engine.py.224: train: epoch 9 | batch 1960/2304 [I 2022-10-26 13:11:10] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:02:52 [I 2022-10-26 13:11:10] ...trap/engines/engine.py.229: process: 0.50734 | load: 0.00240 [I 2022-10-26 13:11:10] ...trap/engines/engine.py.230: loss: 0.28497 [I 2022-10-26 13:11:10] ...optimizers/trijoint.py.151: total_norm: 3.660311 [I 2022-10-26 13:11:15] ...trap/engines/engine.py.224: train: epoch 9 | batch 1970/2304 [I 2022-10-26 13:11:15] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:02:45 [I 2022-10-26 13:11:15] ...trap/engines/engine.py.229: process: 0.48951 | load: 0.00241 [I 2022-10-26 13:11:15] ...trap/engines/engine.py.230: loss: 0.36127 [I 2022-10-26 13:11:15] ...optimizers/trijoint.py.151: total_norm: 3.970632 [I 2022-10-26 13:11:20] ...trap/engines/engine.py.224: train: epoch 9 | batch 1980/2304 [I 2022-10-26 13:11:20] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:02:41 [I 2022-10-26 13:11:20] ...trap/engines/engine.py.229: process: 0.50207 | load: 0.00245 [I 2022-10-26 13:11:20] ...trap/engines/engine.py.230: loss: 0.22207 [I 2022-10-26 13:11:20] ...optimizers/trijoint.py.151: total_norm: 2.964668 [I 2022-10-26 13:11:25] ...trap/engines/engine.py.224: train: epoch 9 | batch 1990/2304 [I 2022-10-26 13:11:25] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:02:36 [I 2022-10-26 13:11:25] ...trap/engines/engine.py.229: process: 0.49633 | load: 0.00261 [I 2022-10-26 13:11:25] ...trap/engines/engine.py.230: loss: 0.23369 [I 2022-10-26 13:11:25] ...optimizers/trijoint.py.151: total_norm: 4.928145 [I 2022-10-26 13:11:30] ...trap/engines/engine.py.224: train: epoch 9 | batch 2000/2304 [I 2022-10-26 13:11:30] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:02:31 [I 2022-10-26 13:11:30] ...trap/engines/engine.py.229: process: 0.49980 | load: 0.00241 [I 2022-10-26 13:11:30] ...trap/engines/engine.py.230: loss: 0.33267 [I 2022-10-26 13:11:30] ...optimizers/trijoint.py.151: total_norm: 4.162671 [I 2022-10-26 13:11:35] ...trap/engines/engine.py.224: train: epoch 9 | batch 2010/2304 [I 2022-10-26 13:11:35] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:02:27 [I 2022-10-26 13:11:35] ...trap/engines/engine.py.229: process: 0.50521 | load: 0.00245 [I 2022-10-26 13:11:35] ...trap/engines/engine.py.230: loss: 0.20343 [I 2022-10-26 13:11:35] ...optimizers/trijoint.py.151: total_norm: 3.622806 [I 2022-10-26 13:11:40] ...trap/engines/engine.py.224: train: epoch 9 | batch 2020/2304 [I 2022-10-26 13:11:40] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:02:21 [I 2022-10-26 13:11:40] ...trap/engines/engine.py.229: process: 0.49698 | load: 0.00249 [I 2022-10-26 13:11:40] ...trap/engines/engine.py.230: loss: 0.23930 [I 2022-10-26 13:11:40] ...optimizers/trijoint.py.151: total_norm: 2.945594 [I 2022-10-26 13:11:45] ...trap/engines/engine.py.224: train: epoch 9 | batch 2030/2304 [I 2022-10-26 13:11:45] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:02:17 [I 2022-10-26 13:11:45] ...trap/engines/engine.py.229: process: 0.50688 | load: 0.00246 [I 2022-10-26 13:11:45] ...trap/engines/engine.py.230: loss: 0.26262 [I 2022-10-26 13:11:45] ...optimizers/trijoint.py.151: total_norm: 3.479262 [I 2022-10-26 13:11:50] ...trap/engines/engine.py.224: train: epoch 9 | batch 2040/2304 [I 2022-10-26 13:11:50] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:02:12 [I 2022-10-26 13:11:50] ...trap/engines/engine.py.229: process: 0.51051 | load: 0.00246 [I 2022-10-26 13:11:50] ...trap/engines/engine.py.230: loss: 0.31923 [I 2022-10-26 13:11:50] ...optimizers/trijoint.py.151: total_norm: 3.383793 [I 2022-10-26 13:11:55] ...trap/engines/engine.py.224: train: epoch 9 | batch 2050/2304 [I 2022-10-26 13:11:55] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:02:07 [I 2022-10-26 13:11:55] ...trap/engines/engine.py.229: process: 0.50179 | load: 0.00244 [I 2022-10-26 13:11:55] ...trap/engines/engine.py.230: loss: 0.25556 [I 2022-10-26 13:11:55] ...optimizers/trijoint.py.151: total_norm: 3.506442 [I 2022-10-26 13:12:00] ...trap/engines/engine.py.224: train: epoch 9 | batch 2060/2304 [I 2022-10-26 13:12:00] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:02:02 [I 2022-10-26 13:12:00] ...trap/engines/engine.py.229: process: 0.50699 | load: 0.00241 [I 2022-10-26 13:12:00] ...trap/engines/engine.py.230: loss: 0.21908 [I 2022-10-26 13:12:00] ...optimizers/trijoint.py.151: total_norm: 3.322509 [I 2022-10-26 13:12:05] ...trap/engines/engine.py.224: train: epoch 9 | batch 2070/2304 [I 2022-10-26 13:12:05] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:01:57 [I 2022-10-26 13:12:05] ...trap/engines/engine.py.229: process: 0.50200 | load: 0.00244 [I 2022-10-26 13:12:05] ...trap/engines/engine.py.230: loss: 0.19679 [I 2022-10-26 13:12:05] ...optimizers/trijoint.py.151: total_norm: 3.214829 [I 2022-10-26 13:12:10] ...trap/engines/engine.py.224: train: epoch 9 | batch 2080/2304 [I 2022-10-26 13:12:10] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:01:51 [I 2022-10-26 13:12:10] ...trap/engines/engine.py.229: process: 0.48769 | load: 0.00244 [I 2022-10-26 13:12:10] ...trap/engines/engine.py.230: loss: 0.23220 [I 2022-10-26 13:12:10] ...optimizers/trijoint.py.151: total_norm: 3.392987 [I 2022-10-26 13:12:15] ...trap/engines/engine.py.224: train: epoch 9 | batch 2090/2304 [I 2022-10-26 13:12:15] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:01:46 [I 2022-10-26 13:12:15] ...trap/engines/engine.py.229: process: 0.50633 | load: 0.00245 [I 2022-10-26 13:12:15] ...trap/engines/engine.py.230: loss: 0.28269 [I 2022-10-26 13:12:15] ...optimizers/trijoint.py.151: total_norm: 4.176981 [I 2022-10-26 13:12:20] ...trap/engines/engine.py.224: train: epoch 9 | batch 2100/2304 [I 2022-10-26 13:12:20] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:01:41 [I 2022-10-26 13:12:20] ...trap/engines/engine.py.229: process: 0.48356 | load: 0.00244 [I 2022-10-26 13:12:20] ...trap/engines/engine.py.230: loss: 0.23020 [I 2022-10-26 13:12:20] ...optimizers/trijoint.py.151: total_norm: 3.963262 [I 2022-10-26 13:12:25] ...trap/engines/engine.py.224: train: epoch 9 | batch 2110/2304 [I 2022-10-26 13:12:25] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:01:36 [I 2022-10-26 13:12:25] ...trap/engines/engine.py.229: process: 0.48626 | load: 0.00243 [I 2022-10-26 13:12:25] ...trap/engines/engine.py.230: loss: 0.17277 [I 2022-10-26 13:12:25] ...optimizers/trijoint.py.151: total_norm: 2.854287 [I 2022-10-26 13:12:30] ...trap/engines/engine.py.224: train: epoch 9 | batch 2120/2304 [I 2022-10-26 13:12:30] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:01:32 [I 2022-10-26 13:12:30] ...trap/engines/engine.py.229: process: 0.49570 | load: 0.00259 [I 2022-10-26 13:12:30] ...trap/engines/engine.py.230: loss: 0.26798 [I 2022-10-26 13:12:30] ...optimizers/trijoint.py.151: total_norm: 4.127482 [I 2022-10-26 13:12:35] ...trap/engines/engine.py.224: train: epoch 9 | batch 2130/2304 [I 2022-10-26 13:12:35] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:01:26 [I 2022-10-26 13:12:35] ...trap/engines/engine.py.229: process: 0.49991 | load: 0.00254 [I 2022-10-26 13:12:35] ...trap/engines/engine.py.230: loss: 0.27664 [I 2022-10-26 13:12:35] ...optimizers/trijoint.py.151: total_norm: 3.922279 [I 2022-10-26 13:12:40] ...trap/engines/engine.py.224: train: epoch 9 | batch 2140/2304 [I 2022-10-26 13:12:40] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:01:21 [I 2022-10-26 13:12:40] ...trap/engines/engine.py.229: process: 0.49931 | load: 0.00249 [I 2022-10-26 13:12:40] ...trap/engines/engine.py.230: loss: 0.34132 [I 2022-10-26 13:12:40] ...optimizers/trijoint.py.151: total_norm: 3.496494 [I 2022-10-26 13:12:45] ...trap/engines/engine.py.224: train: epoch 9 | batch 2150/2304 [I 2022-10-26 13:12:45] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:01:16 [I 2022-10-26 13:12:45] ...trap/engines/engine.py.229: process: 0.49578 | load: 0.00253 [I 2022-10-26 13:12:45] ...trap/engines/engine.py.230: loss: 0.25260 [I 2022-10-26 13:12:45] ...optimizers/trijoint.py.151: total_norm: 3.234866 [I 2022-10-26 13:12:50] ...trap/engines/engine.py.224: train: epoch 9 | batch 2160/2304 [I 2022-10-26 13:12:50] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:01:11 [I 2022-10-26 13:12:50] ...trap/engines/engine.py.229: process: 0.49108 | load: 0.00248 [I 2022-10-26 13:12:50] ...trap/engines/engine.py.230: loss: 0.18763 [I 2022-10-26 13:12:50] ...optimizers/trijoint.py.151: total_norm: 3.252116 [I 2022-10-26 13:12:55] ...trap/engines/engine.py.224: train: epoch 9 | batch 2170/2304 [I 2022-10-26 13:12:55] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:01:06 [I 2022-10-26 13:12:55] ...trap/engines/engine.py.229: process: 0.49250 | load: 0.00247 [I 2022-10-26 13:12:55] ...trap/engines/engine.py.230: loss: 0.21628 [I 2022-10-26 13:12:55] ...optimizers/trijoint.py.151: total_norm: 3.230869 [I 2022-10-26 13:13:00] ...trap/engines/engine.py.224: train: epoch 9 | batch 2180/2304 [I 2022-10-26 13:13:00] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:01:01 [I 2022-10-26 13:13:00] ...trap/engines/engine.py.229: process: 0.48360 | load: 0.00247 [I 2022-10-26 13:13:00] ...trap/engines/engine.py.230: loss: 0.22837 [I 2022-10-26 13:13:00] ...optimizers/trijoint.py.151: total_norm: 4.091807 [I 2022-10-26 13:13:05] ...trap/engines/engine.py.224: train: epoch 9 | batch 2190/2304 [I 2022-10-26 13:13:05] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:00:56 [I 2022-10-26 13:13:05] ...trap/engines/engine.py.229: process: 0.48927 | load: 0.00242 [I 2022-10-26 13:13:05] ...trap/engines/engine.py.230: loss: 0.25191 [I 2022-10-26 13:13:05] ...optimizers/trijoint.py.151: total_norm: 3.240158 [I 2022-10-26 13:13:10] ...trap/engines/engine.py.224: train: epoch 9 | batch 2200/2304 [I 2022-10-26 13:13:10] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:00:51 [I 2022-10-26 13:13:10] ...trap/engines/engine.py.229: process: 0.48721 | load: 0.00242 [I 2022-10-26 13:13:10] ...trap/engines/engine.py.230: loss: 0.32307 [I 2022-10-26 13:13:10] ...optimizers/trijoint.py.151: total_norm: 4.195980 [I 2022-10-26 13:13:15] ...trap/engines/engine.py.224: train: epoch 9 | batch 2210/2304 [I 2022-10-26 13:13:15] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:00:46 [I 2022-10-26 13:13:15] ...trap/engines/engine.py.229: process: 0.49296 | load: 0.00241 [I 2022-10-26 13:13:15] ...trap/engines/engine.py.230: loss: 0.27059 [I 2022-10-26 13:13:15] ...optimizers/trijoint.py.151: total_norm: 4.385726 [I 2022-10-26 13:13:20] ...trap/engines/engine.py.224: train: epoch 9 | batch 2220/2304 [I 2022-10-26 13:13:20] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:00:42 [I 2022-10-26 13:13:20] ...trap/engines/engine.py.229: process: 0.49381 | load: 0.00241 [I 2022-10-26 13:13:20] ...trap/engines/engine.py.230: loss: 0.21594 [I 2022-10-26 13:13:20] ...optimizers/trijoint.py.151: total_norm: 2.590240 [I 2022-10-26 13:13:44] ...trap/engines/engine.py.224: train: epoch 9 | batch 2230/2304 [I 2022-10-26 13:13:44] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:03:30 [I 2022-10-26 13:13:44] ...trap/engines/engine.py.229: process: 0.51590 | load: 0.00567 [I 2022-10-26 13:13:44] ...trap/engines/engine.py.230: loss: 0.14571 [I 2022-10-26 13:13:44] ...optimizers/trijoint.py.151: total_norm: 2.314358 [I 2022-10-26 13:13:49] ...trap/engines/engine.py.224: train: epoch 9 | batch 2240/2304 [I 2022-10-26 13:13:49] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:00:48 [I 2022-10-26 13:13:49] ...trap/engines/engine.py.229: process: 0.50215 | load: 0.00236 [I 2022-10-26 13:13:49] ...trap/engines/engine.py.230: loss: 0.35495 [I 2022-10-26 13:13:49] ...optimizers/trijoint.py.151: total_norm: 3.799490 [I 2022-10-26 13:13:54] ...trap/engines/engine.py.224: train: epoch 9 | batch 2250/2304 [I 2022-10-26 13:13:54] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:00:28 [I 2022-10-26 13:13:54] ...trap/engines/engine.py.229: process: 0.50136 | load: 0.00236 [I 2022-10-26 13:13:54] ...trap/engines/engine.py.230: loss: 0.23736 [I 2022-10-26 13:13:54] ...optimizers/trijoint.py.151: total_norm: 3.213386 [I 2022-10-26 13:13:59] ...trap/engines/engine.py.224: train: epoch 9 | batch 2260/2304 [I 2022-10-26 13:13:59] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:00:22 [I 2022-10-26 13:13:59] ...trap/engines/engine.py.229: process: 0.50733 | load: 0.00264 [I 2022-10-26 13:13:59] ...trap/engines/engine.py.230: loss: 0.27372 [I 2022-10-26 13:13:59] ...optimizers/trijoint.py.151: total_norm: 3.959859 [I 2022-10-26 13:14:04] ...trap/engines/engine.py.224: train: epoch 9 | batch 2270/2304 [I 2022-10-26 13:14:04] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:00:16 [I 2022-10-26 13:14:04] ...trap/engines/engine.py.229: process: 0.48992 | load: 0.00261 [I 2022-10-26 13:14:04] ...trap/engines/engine.py.230: loss: 0.19227 [I 2022-10-26 13:14:04] ...optimizers/trijoint.py.151: total_norm: 2.422255 [I 2022-10-26 13:14:09] ...trap/engines/engine.py.224: train: epoch 9 | batch 2280/2304 [I 2022-10-26 13:14:09] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:00:12 [I 2022-10-26 13:14:09] ...trap/engines/engine.py.229: process: 0.50696 | load: 0.00242 [I 2022-10-26 13:14:09] ...trap/engines/engine.py.230: loss: 0.20950 [I 2022-10-26 13:14:09] ...optimizers/trijoint.py.151: total_norm: 2.822426 [I 2022-10-26 13:14:14] ...trap/engines/engine.py.224: train: epoch 9 | batch 2290/2304 [I 2022-10-26 13:14:14] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:00:07 [I 2022-10-26 13:14:14] ...trap/engines/engine.py.229: process: 0.48910 | load: 0.00247 [I 2022-10-26 13:14:14] ...trap/engines/engine.py.230: loss: 0.27125 [I 2022-10-26 13:14:14] ...optimizers/trijoint.py.151: total_norm: 3.787795 [I 2022-10-26 13:14:19] ...trap/engines/engine.py.224: train: epoch 9 | batch 2300/2304 [I 2022-10-26 13:14:19] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:00:02 [I 2022-10-26 13:14:19] ...trap/engines/engine.py.229: process: 0.49672 | load: 0.00015 [I 2022-10-26 13:14:19] ...trap/engines/engine.py.230: loss: 0.26345 [I 2022-10-26 13:14:19] ...optimizers/trijoint.py.151: total_norm: 3.538761 [I 2022-10-26 13:14:21] ...trap/engines/engine.py.224: train: epoch 9 | batch 2304/2304 [I 2022-10-26 13:14:21] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:00:00 [I 2022-10-26 13:14:21] ...trap/engines/engine.py.229: process: 0.48671 | load: 0.00017 [I 2022-10-26 13:14:21] ...trap/engines/engine.py.230: loss: 0.30848 [I 2022-10-26 13:14:21] ...optimizers/trijoint.py.151: total_norm: 3.718023 [S 2022-10-26 13:14:22] ...trap/engines/engine.py.129: train_epoch.epoch: 9 [S 2022-10-26 13:14:22] ...trap/engines/engine.py.129: train_epoch.loss: 0.24063566948353857 [I 2022-10-26 13:14:22] ...s/criterions/triplet.py.84: margin = 0.10000000000000003 [I 2022-10-26 13:14:22] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 9 [I 2022-10-26 13:14:22] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 13:14:23] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 13:14:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 13:14:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 13:14:24] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 13:14:25] ...trap/engines/engine.py.321: eval: epoch 9 | batch 0/510 [I 2022-10-26 13:14:25] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:29 [I 2022-10-26 13:14:25] ...trap/engines/engine.py.326: process: 2.64662 | load: 2.25592 [I 2022-10-26 13:14:28] ...trap/engines/engine.py.321: eval: epoch 9 | batch 10/510 [I 2022-10-26 13:14:28] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:55 [I 2022-10-26 13:14:28] ...trap/engines/engine.py.326: process: 0.35465 | load: 0.00020 [I 2022-10-26 13:14:32] ...trap/engines/engine.py.321: eval: epoch 9 | batch 20/510 [I 2022-10-26 13:14:32] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:59 [I 2022-10-26 13:14:32] ...trap/engines/engine.py.326: process: 0.33097 | load: 0.00020 [I 2022-10-26 13:14:35] ...trap/engines/engine.py.321: eval: epoch 9 | batch 30/510 [I 2022-10-26 13:14:35] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:43 [I 2022-10-26 13:14:35] ...trap/engines/engine.py.326: process: 0.36307 | load: 0.00036 [I 2022-10-26 13:14:38] ...trap/engines/engine.py.321: eval: epoch 9 | batch 40/510 [I 2022-10-26 13:14:38] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:40 [I 2022-10-26 13:14:38] ...trap/engines/engine.py.326: process: 0.33972 | load: 0.00025 [I 2022-10-26 13:14:42] ...trap/engines/engine.py.321: eval: epoch 9 | batch 50/510 [I 2022-10-26 13:14:42] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-26 13:14:42] ...trap/engines/engine.py.326: process: 0.36091 | load: 0.00038 [I 2022-10-26 13:14:45] ...trap/engines/engine.py.321: eval: epoch 9 | batch 60/510 [I 2022-10-26 13:14:45] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:31 [I 2022-10-26 13:14:45] ...trap/engines/engine.py.326: process: 0.34809 | load: 0.00019 [I 2022-10-26 13:14:48] ...trap/engines/engine.py.321: eval: epoch 9 | batch 70/510 [I 2022-10-26 13:14:48] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:26 [I 2022-10-26 13:14:48] ...trap/engines/engine.py.326: process: 0.33285 | load: 0.00017 [I 2022-10-26 13:14:52] ...trap/engines/engine.py.321: eval: epoch 9 | batch 80/510 [I 2022-10-26 13:14:52] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:37 [I 2022-10-26 13:14:52] ...trap/engines/engine.py.326: process: 0.47969 | load: 0.00017 [I 2022-10-26 13:14:55] ...trap/engines/engine.py.321: eval: epoch 9 | batch 90/510 [I 2022-10-26 13:14:55] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:24 [I 2022-10-26 13:14:55] ...trap/engines/engine.py.326: process: 0.36458 | load: 0.00016 [I 2022-10-26 13:14:59] ...trap/engines/engine.py.321: eval: epoch 9 | batch 100/510 [I 2022-10-26 13:14:59] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:16 [I 2022-10-26 13:14:59] ...trap/engines/engine.py.326: process: 0.28198 | load: 0.00029 [I 2022-10-26 13:15:02] ...trap/engines/engine.py.321: eval: epoch 9 | batch 110/510 [I 2022-10-26 13:15:02] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:59 [I 2022-10-26 13:15:02] ...trap/engines/engine.py.326: process: 0.30956 | load: 0.00014 [I 2022-10-26 13:15:04] ...trap/engines/engine.py.321: eval: epoch 9 | batch 120/510 [I 2022-10-26 13:15:04] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:55 [I 2022-10-26 13:15:04] ...trap/engines/engine.py.326: process: 0.29915 | load: 0.00017 [I 2022-10-26 13:15:07] ...trap/engines/engine.py.321: eval: epoch 9 | batch 130/510 [I 2022-10-26 13:15:07] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:49 [I 2022-10-26 13:15:07] ...trap/engines/engine.py.326: process: 0.28835 | load: 0.00015 [I 2022-10-26 13:15:10] ...trap/engines/engine.py.321: eval: epoch 9 | batch 140/510 [I 2022-10-26 13:15:10] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-26 13:15:10] ...trap/engines/engine.py.326: process: 0.28425 | load: 0.00016 [I 2022-10-26 13:15:13] ...trap/engines/engine.py.321: eval: epoch 9 | batch 150/510 [I 2022-10-26 13:15:13] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 13:15:13] ...trap/engines/engine.py.326: process: 0.29230 | load: 0.00017 [I 2022-10-26 13:15:16] ...trap/engines/engine.py.321: eval: epoch 9 | batch 160/510 [I 2022-10-26 13:15:16] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-26 13:15:16] ...trap/engines/engine.py.326: process: 0.28607 | load: 0.00019 [I 2022-10-26 13:15:19] ...trap/engines/engine.py.321: eval: epoch 9 | batch 170/510 [I 2022-10-26 13:15:19] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 13:15:19] ...trap/engines/engine.py.326: process: 0.28541 | load: 0.00027 [I 2022-10-26 13:15:22] ...trap/engines/engine.py.321: eval: epoch 9 | batch 180/510 [I 2022-10-26 13:15:22] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 13:15:22] ...trap/engines/engine.py.326: process: 0.28875 | load: 0.00019 [I 2022-10-26 13:15:25] ...trap/engines/engine.py.321: eval: epoch 9 | batch 190/510 [I 2022-10-26 13:15:25] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:33 [I 2022-10-26 13:15:25] ...trap/engines/engine.py.326: process: 0.28482 | load: 0.00015 [I 2022-10-26 13:15:28] ...trap/engines/engine.py.321: eval: epoch 9 | batch 200/510 [I 2022-10-26 13:15:28] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 13:15:28] ...trap/engines/engine.py.326: process: 0.29017 | load: 0.00032 [I 2022-10-26 13:15:31] ...trap/engines/engine.py.321: eval: epoch 9 | batch 210/510 [I 2022-10-26 13:15:31] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 13:15:31] ...trap/engines/engine.py.326: process: 0.28774 | load: 0.00015 [I 2022-10-26 13:15:34] ...trap/engines/engine.py.321: eval: epoch 9 | batch 220/510 [I 2022-10-26 13:15:34] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 13:15:34] ...trap/engines/engine.py.326: process: 0.28709 | load: 0.00017 [I 2022-10-26 13:15:37] ...trap/engines/engine.py.321: eval: epoch 9 | batch 230/510 [I 2022-10-26 13:15:37] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 13:15:37] ...trap/engines/engine.py.326: process: 0.28853 | load: 0.00029 [I 2022-10-26 13:15:39] ...trap/engines/engine.py.321: eval: epoch 9 | batch 240/510 [I 2022-10-26 13:15:39] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 13:15:39] ...trap/engines/engine.py.326: process: 0.29655 | load: 0.00019 [I 2022-10-26 13:15:42] ...trap/engines/engine.py.321: eval: epoch 9 | batch 250/510 [I 2022-10-26 13:15:42] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-26 13:15:42] ...trap/engines/engine.py.326: process: 0.28558 | load: 0.00015 [I 2022-10-26 13:15:45] ...trap/engines/engine.py.321: eval: epoch 9 | batch 260/510 [I 2022-10-26 13:15:45] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-26 13:15:45] ...trap/engines/engine.py.326: process: 0.28152 | load: 0.00024 [I 2022-10-26 13:15:48] ...trap/engines/engine.py.321: eval: epoch 9 | batch 270/510 [I 2022-10-26 13:15:48] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-26 13:15:48] ...trap/engines/engine.py.326: process: 0.27796 | load: 0.00015 [I 2022-10-26 13:15:51] ...trap/engines/engine.py.321: eval: epoch 9 | batch 280/510 [I 2022-10-26 13:15:51] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 13:15:51] ...trap/engines/engine.py.326: process: 0.28392 | load: 0.00019 [I 2022-10-26 13:15:54] ...trap/engines/engine.py.321: eval: epoch 9 | batch 290/510 [I 2022-10-26 13:15:54] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 13:15:54] ...trap/engines/engine.py.326: process: 0.28529 | load: 0.00020 [I 2022-10-26 13:15:57] ...trap/engines/engine.py.321: eval: epoch 9 | batch 300/510 [I 2022-10-26 13:15:57] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 13:15:57] ...trap/engines/engine.py.326: process: 0.28600 | load: 0.00018 [I 2022-10-26 13:16:00] ...trap/engines/engine.py.321: eval: epoch 9 | batch 310/510 [I 2022-10-26 13:16:00] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 13:16:00] ...trap/engines/engine.py.326: process: 0.28743 | load: 0.00015 [I 2022-10-26 13:16:03] ...trap/engines/engine.py.321: eval: epoch 9 | batch 320/510 [I 2022-10-26 13:16:03] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:56 [I 2022-10-26 13:16:03] ...trap/engines/engine.py.326: process: 0.29499 | load: 0.00024 [I 2022-10-26 13:16:06] ...trap/engines/engine.py.321: eval: epoch 9 | batch 330/510 [I 2022-10-26 13:16:06] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 13:16:06] ...trap/engines/engine.py.326: process: 0.28779 | load: 0.00016 [I 2022-10-26 13:16:08] ...trap/engines/engine.py.321: eval: epoch 9 | batch 340/510 [I 2022-10-26 13:16:08] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 13:16:08] ...trap/engines/engine.py.326: process: 0.28265 | load: 0.00021 [I 2022-10-26 13:16:11] ...trap/engines/engine.py.321: eval: epoch 9 | batch 350/510 [I 2022-10-26 13:16:11] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 13:16:11] ...trap/engines/engine.py.326: process: 0.28768 | load: 0.00019 [I 2022-10-26 13:16:14] ...trap/engines/engine.py.321: eval: epoch 9 | batch 360/510 [I 2022-10-26 13:16:14] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 13:16:14] ...trap/engines/engine.py.326: process: 0.28589 | load: 0.00021 [I 2022-10-26 13:16:17] ...trap/engines/engine.py.321: eval: epoch 9 | batch 370/510 [I 2022-10-26 13:16:17] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-26 13:16:17] ...trap/engines/engine.py.326: process: 0.28587 | load: 0.00016 [I 2022-10-26 13:16:20] ...trap/engines/engine.py.321: eval: epoch 9 | batch 380/510 [I 2022-10-26 13:16:20] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 13:16:20] ...trap/engines/engine.py.326: process: 0.28409 | load: 0.00019 [I 2022-10-26 13:16:23] ...trap/engines/engine.py.321: eval: epoch 9 | batch 390/510 [I 2022-10-26 13:16:23] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 13:16:23] ...trap/engines/engine.py.326: process: 0.28478 | load: 0.00017 [I 2022-10-26 13:16:26] ...trap/engines/engine.py.321: eval: epoch 9 | batch 400/510 [I 2022-10-26 13:16:26] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 13:16:26] ...trap/engines/engine.py.326: process: 0.29247 | load: 0.00019 [I 2022-10-26 13:16:29] ...trap/engines/engine.py.321: eval: epoch 9 | batch 410/510 [I 2022-10-26 13:16:29] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 13:16:29] ...trap/engines/engine.py.326: process: 0.28728 | load: 0.00015 [I 2022-10-26 13:16:31] ...trap/engines/engine.py.321: eval: epoch 9 | batch 420/510 [I 2022-10-26 13:16:31] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 13:16:31] ...trap/engines/engine.py.326: process: 0.28311 | load: 0.00016 [I 2022-10-26 13:16:34] ...trap/engines/engine.py.321: eval: epoch 9 | batch 430/510 [I 2022-10-26 13:16:34] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:22 [I 2022-10-26 13:16:34] ...trap/engines/engine.py.326: process: 0.28657 | load: 0.00016 [I 2022-10-26 13:16:37] ...trap/engines/engine.py.321: eval: epoch 9 | batch 440/510 [I 2022-10-26 13:16:37] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 13:16:37] ...trap/engines/engine.py.326: process: 0.28726 | load: 0.00018 [I 2022-10-26 13:16:40] ...trap/engines/engine.py.321: eval: epoch 9 | batch 450/510 [I 2022-10-26 13:16:40] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 13:16:40] ...trap/engines/engine.py.326: process: 0.28874 | load: 0.00020 [I 2022-10-26 13:16:43] ...trap/engines/engine.py.321: eval: epoch 9 | batch 460/510 [I 2022-10-26 13:16:43] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-26 13:16:43] ...trap/engines/engine.py.326: process: 0.28093 | load: 0.00019 [I 2022-10-26 13:16:46] ...trap/engines/engine.py.321: eval: epoch 9 | batch 470/510 [I 2022-10-26 13:16:46] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 13:16:46] ...trap/engines/engine.py.326: process: 0.28528 | load: 0.00018 [I 2022-10-26 13:16:49] ...trap/engines/engine.py.321: eval: epoch 9 | batch 480/510 [I 2022-10-26 13:16:49] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 13:16:49] ...trap/engines/engine.py.326: process: 0.29334 | load: 0.00016 [I 2022-10-26 13:16:52] ...trap/engines/engine.py.321: eval: epoch 9 | batch 490/510 [I 2022-10-26 13:16:52] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 13:16:52] ...trap/engines/engine.py.326: process: 0.28886 | load: 0.00016 [I 2022-10-26 13:16:55] ...trap/engines/engine.py.321: eval: epoch 9 | batch 500/510 [I 2022-10-26 13:16:55] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 13:16:55] ...trap/engines/engine.py.326: process: 0.28753 | load: 0.00015 [I 2022-10-26 13:16:57] ...trap/engines/engine.py.321: eval: epoch 9 | batch 510/510 [I 2022-10-26 13:16:57] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 13:16:57] ...trap/engines/engine.py.326: process: 0.28737 | load: 0.00016 [S 2022-10-26 13:16:58] ...trap/engines/engine.py.132: eval_epoch.epoch: 9 [I 2022-10-26 13:16:58] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.30 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.50 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.506100 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.796500 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.870000 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.502900 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.796400 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.871000 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.458258 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.500000 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016434 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011543 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011992 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017312 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010623 [S 2022-10-26 13:17:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009716 [I 2022-10-26 13:17:03] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-26 13:17:03] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 13:17:03] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 13:17:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 13:17:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 13:17:03] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 13:17:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 13:17:11] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 13:17:11] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 13:17:11] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 13:17:14] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 13:17:19] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 13:17:19] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 13:17:19] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 13:17:22] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 13:17:28] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 13:17:28] ...trap/engines/engine.py.163: Training model on trainset for epoch 10 [I 2022-10-26 13:17:28] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 13:17:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 0/2304 [I 2022-10-26 13:17:31] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:08 [I 2022-10-26 13:17:31] ...trap/engines/engine.py.229: process: 2.76415 | load: 2.19090 [I 2022-10-26 13:17:31] ...trap/engines/engine.py.230: loss: 0.34501 [I 2022-10-26 13:17:31] ...optimizers/trijoint.py.151: total_norm: 4.047519 [I 2022-10-26 13:17:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 10/2304 [I 2022-10-26 13:17:36] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:48 [I 2022-10-26 13:17:36] ...trap/engines/engine.py.229: process: 0.50355 | load: 0.00455 [I 2022-10-26 13:17:36] ...trap/engines/engine.py.230: loss: 0.23736 [I 2022-10-26 13:17:36] ...optimizers/trijoint.py.151: total_norm: 3.528455 [I 2022-10-26 13:17:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 20/2304 [I 2022-10-26 13:17:41] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:09 [I 2022-10-26 13:17:41] ...trap/engines/engine.py.229: process: 0.50152 | load: 0.00440 [I 2022-10-26 13:17:41] ...trap/engines/engine.py.230: loss: 0.23236 [I 2022-10-26 13:17:41] ...optimizers/trijoint.py.151: total_norm: 2.876576 [I 2022-10-26 13:17:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 30/2304 [I 2022-10-26 13:17:46] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:16 [I 2022-10-26 13:17:46] ...trap/engines/engine.py.229: process: 0.50761 | load: 0.00338 [I 2022-10-26 13:17:46] ...trap/engines/engine.py.230: loss: 0.22194 [I 2022-10-26 13:17:46] ...optimizers/trijoint.py.151: total_norm: 3.398125 [I 2022-10-26 13:17:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 40/2304 [I 2022-10-26 13:17:51] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:45 [I 2022-10-26 13:17:51] ...trap/engines/engine.py.229: process: 0.49521 | load: 0.00306 [I 2022-10-26 13:17:51] ...trap/engines/engine.py.230: loss: 0.19471 [I 2022-10-26 13:17:51] ...optimizers/trijoint.py.151: total_norm: 2.838813 [I 2022-10-26 13:17:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 50/2304 [I 2022-10-26 13:17:56] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:19:01 [I 2022-10-26 13:17:56] ...trap/engines/engine.py.229: process: 0.53046 | load: 0.00300 [I 2022-10-26 13:17:56] ...trap/engines/engine.py.230: loss: 0.22818 [I 2022-10-26 13:17:56] ...optimizers/trijoint.py.151: total_norm: 3.246250 [I 2022-10-26 13:18:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 60/2304 [I 2022-10-26 13:18:01] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:36 [I 2022-10-26 13:18:01] ...trap/engines/engine.py.229: process: 0.49056 | load: 0.00286 [I 2022-10-26 13:18:01] ...trap/engines/engine.py.230: loss: 0.36307 [I 2022-10-26 13:18:01] ...optimizers/trijoint.py.151: total_norm: 4.782408 [I 2022-10-26 13:18:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 70/2304 [I 2022-10-26 13:18:06] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:30 [I 2022-10-26 13:18:06] ...trap/engines/engine.py.229: process: 0.49392 | load: 0.00291 [I 2022-10-26 13:18:06] ...trap/engines/engine.py.230: loss: 0.24515 [I 2022-10-26 13:18:06] ...optimizers/trijoint.py.151: total_norm: 3.268741 [I 2022-10-26 13:18:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 80/2304 [I 2022-10-26 13:18:11] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:22 [I 2022-10-26 13:18:11] ...trap/engines/engine.py.229: process: 0.49282 | load: 0.00274 [I 2022-10-26 13:18:11] ...trap/engines/engine.py.230: loss: 0.20963 [I 2022-10-26 13:18:11] ...optimizers/trijoint.py.151: total_norm: 3.203159 [I 2022-10-26 13:18:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 90/2304 [I 2022-10-26 13:18:16] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:18:17 [I 2022-10-26 13:18:16] ...trap/engines/engine.py.229: process: 0.50059 | load: 0.00262 [I 2022-10-26 13:18:16] ...trap/engines/engine.py.230: loss: 0.28076 [I 2022-10-26 13:18:16] ...optimizers/trijoint.py.151: total_norm: 4.649850 [I 2022-10-26 13:18:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 100/2304 [I 2022-10-26 13:18:21] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:18:19 [I 2022-10-26 13:18:21] ...trap/engines/engine.py.229: process: 0.50125 | load: 0.00253 [I 2022-10-26 13:18:21] ...trap/engines/engine.py.230: loss: 0.28646 [I 2022-10-26 13:18:21] ...optimizers/trijoint.py.151: total_norm: 3.632978 [I 2022-10-26 13:18:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 110/2304 [I 2022-10-26 13:18:26] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:18:18 [I 2022-10-26 13:18:26] ...trap/engines/engine.py.229: process: 0.48524 | load: 0.00275 [I 2022-10-26 13:18:26] ...trap/engines/engine.py.230: loss: 0.32606 [I 2022-10-26 13:18:26] ...optimizers/trijoint.py.151: total_norm: 4.355238 [I 2022-10-26 13:18:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 120/2304 [I 2022-10-26 13:18:31] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:18:08 [I 2022-10-26 13:18:31] ...trap/engines/engine.py.229: process: 0.50000 | load: 0.00265 [I 2022-10-26 13:18:31] ...trap/engines/engine.py.230: loss: 0.30333 [I 2022-10-26 13:18:31] ...optimizers/trijoint.py.151: total_norm: 4.306897 [I 2022-10-26 13:18:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 130/2304 [I 2022-10-26 13:18:36] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:20 [I 2022-10-26 13:18:36] ...trap/engines/engine.py.229: process: 0.50047 | load: 0.00259 [I 2022-10-26 13:18:36] ...trap/engines/engine.py.230: loss: 0.24009 [I 2022-10-26 13:18:36] ...optimizers/trijoint.py.151: total_norm: 3.880302 [I 2022-10-26 13:18:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 140/2304 [I 2022-10-26 13:18:41] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:17:57 [I 2022-10-26 13:18:41] ...trap/engines/engine.py.229: process: 0.49537 | load: 0.00253 [I 2022-10-26 13:18:41] ...trap/engines/engine.py.230: loss: 0.31838 [I 2022-10-26 13:18:41] ...optimizers/trijoint.py.151: total_norm: 4.918299 [I 2022-10-26 13:18:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 150/2304 [I 2022-10-26 13:18:46] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:17:51 [I 2022-10-26 13:18:46] ...trap/engines/engine.py.229: process: 0.48284 | load: 0.00263 [I 2022-10-26 13:18:46] ...trap/engines/engine.py.230: loss: 0.28354 [I 2022-10-26 13:18:46] ...optimizers/trijoint.py.151: total_norm: 3.538486 [I 2022-10-26 13:18:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 160/2304 [I 2022-10-26 13:18:51] ...trap/engines/engine.py.228: elapsed: 0:01:22 | left: 0:17:39 [I 2022-10-26 13:18:51] ...trap/engines/engine.py.229: process: 0.49320 | load: 0.00245 [I 2022-10-26 13:18:51] ...trap/engines/engine.py.230: loss: 0.27712 [I 2022-10-26 13:18:51] ...optimizers/trijoint.py.151: total_norm: 3.189368 [I 2022-10-26 13:18:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 170/2304 [I 2022-10-26 13:18:56] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:17:47 [I 2022-10-26 13:18:56] ...trap/engines/engine.py.229: process: 0.49950 | load: 0.00259 [I 2022-10-26 13:18:56] ...trap/engines/engine.py.230: loss: 0.26574 [I 2022-10-26 13:18:56] ...optimizers/trijoint.py.151: total_norm: 3.481187 [I 2022-10-26 13:19:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 180/2304 [I 2022-10-26 13:19:01] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:17:37 [I 2022-10-26 13:19:01] ...trap/engines/engine.py.229: process: 0.49544 | load: 0.00243 [I 2022-10-26 13:19:01] ...trap/engines/engine.py.230: loss: 0.20774 [I 2022-10-26 13:19:01] ...optimizers/trijoint.py.151: total_norm: 2.878603 [I 2022-10-26 13:19:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 190/2304 [I 2022-10-26 13:19:06] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:17:28 [I 2022-10-26 13:19:06] ...trap/engines/engine.py.229: process: 0.49715 | load: 0.00261 [I 2022-10-26 13:19:06] ...trap/engines/engine.py.230: loss: 0.16875 [I 2022-10-26 13:19:06] ...optimizers/trijoint.py.151: total_norm: 2.811802 [I 2022-10-26 13:19:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 200/2304 [I 2022-10-26 13:19:11] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:17:26 [I 2022-10-26 13:19:11] ...trap/engines/engine.py.229: process: 0.49955 | load: 0.00257 [I 2022-10-26 13:19:11] ...trap/engines/engine.py.230: loss: 0.21912 [I 2022-10-26 13:19:11] ...optimizers/trijoint.py.151: total_norm: 3.253865 [I 2022-10-26 13:19:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 210/2304 [I 2022-10-26 13:19:16] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:17:18 [I 2022-10-26 13:19:16] ...trap/engines/engine.py.229: process: 0.48232 | load: 0.00245 [I 2022-10-26 13:19:16] ...trap/engines/engine.py.230: loss: 0.21508 [I 2022-10-26 13:19:16] ...optimizers/trijoint.py.151: total_norm: 3.600959 [I 2022-10-26 13:19:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 220/2304 [I 2022-10-26 13:19:21] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:17:16 [I 2022-10-26 13:19:21] ...trap/engines/engine.py.229: process: 0.49440 | load: 0.00251 [I 2022-10-26 13:19:21] ...trap/engines/engine.py.230: loss: 0.19920 [I 2022-10-26 13:19:21] ...optimizers/trijoint.py.151: total_norm: 2.892225 [I 2022-10-26 13:19:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 230/2304 [I 2022-10-26 13:19:26] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:17:12 [I 2022-10-26 13:19:26] ...trap/engines/engine.py.229: process: 0.49763 | load: 0.00277 [I 2022-10-26 13:19:26] ...trap/engines/engine.py.230: loss: 0.31041 [I 2022-10-26 13:19:26] ...optimizers/trijoint.py.151: total_norm: 3.315131 [I 2022-10-26 13:19:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 240/2304 [I 2022-10-26 13:19:31] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:17:05 [I 2022-10-26 13:19:31] ...trap/engines/engine.py.229: process: 0.49770 | load: 0.00250 [I 2022-10-26 13:19:31] ...trap/engines/engine.py.230: loss: 0.18924 [I 2022-10-26 13:19:31] ...optimizers/trijoint.py.151: total_norm: 2.571244 [I 2022-10-26 13:19:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 250/2304 [I 2022-10-26 13:19:36] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:17:01 [I 2022-10-26 13:19:36] ...trap/engines/engine.py.229: process: 0.50052 | load: 0.00243 [I 2022-10-26 13:19:36] ...trap/engines/engine.py.230: loss: 0.25923 [I 2022-10-26 13:19:36] ...optimizers/trijoint.py.151: total_norm: 3.142519 [I 2022-10-26 13:19:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 260/2304 [I 2022-10-26 13:19:41] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:17:15 [I 2022-10-26 13:19:41] ...trap/engines/engine.py.229: process: 0.49594 | load: 0.00246 [I 2022-10-26 13:19:41] ...trap/engines/engine.py.230: loss: 0.33920 [I 2022-10-26 13:19:41] ...optimizers/trijoint.py.151: total_norm: 3.794648 [I 2022-10-26 13:19:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 270/2304 [I 2022-10-26 13:19:46] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:16:53 [I 2022-10-26 13:19:46] ...trap/engines/engine.py.229: process: 0.49755 | load: 0.00252 [I 2022-10-26 13:19:46] ...trap/engines/engine.py.230: loss: 0.20040 [I 2022-10-26 13:19:46] ...optimizers/trijoint.py.151: total_norm: 2.682130 [I 2022-10-26 13:19:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 280/2304 [I 2022-10-26 13:19:51] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:16:39 [I 2022-10-26 13:19:51] ...trap/engines/engine.py.229: process: 0.48960 | load: 0.00242 [I 2022-10-26 13:19:51] ...trap/engines/engine.py.230: loss: 0.21486 [I 2022-10-26 13:19:51] ...optimizers/trijoint.py.151: total_norm: 2.822602 [I 2022-10-26 13:19:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 290/2304 [I 2022-10-26 13:19:56] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:16:40 [I 2022-10-26 13:19:56] ...trap/engines/engine.py.229: process: 0.49815 | load: 0.00251 [I 2022-10-26 13:19:56] ...trap/engines/engine.py.230: loss: 0.20829 [I 2022-10-26 13:19:56] ...optimizers/trijoint.py.151: total_norm: 4.301782 [I 2022-10-26 13:20:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 300/2304 [I 2022-10-26 13:20:01] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:16:45 [I 2022-10-26 13:20:01] ...trap/engines/engine.py.229: process: 0.52997 | load: 0.00247 [I 2022-10-26 13:20:01] ...trap/engines/engine.py.230: loss: 0.21489 [I 2022-10-26 13:20:01] ...optimizers/trijoint.py.151: total_norm: 3.870196 [I 2022-10-26 13:20:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 310/2304 [I 2022-10-26 13:20:06] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:16:28 [I 2022-10-26 13:20:06] ...trap/engines/engine.py.229: process: 0.49663 | load: 0.00246 [I 2022-10-26 13:20:06] ...trap/engines/engine.py.230: loss: 0.30330 [I 2022-10-26 13:20:06] ...optimizers/trijoint.py.151: total_norm: 3.695524 [I 2022-10-26 13:20:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 320/2304 [I 2022-10-26 13:20:11] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:16:23 [I 2022-10-26 13:20:11] ...trap/engines/engine.py.229: process: 0.48993 | load: 0.00255 [I 2022-10-26 13:20:11] ...trap/engines/engine.py.230: loss: 0.30203 [I 2022-10-26 13:20:11] ...optimizers/trijoint.py.151: total_norm: 4.542988 [I 2022-10-26 13:20:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 330/2304 [I 2022-10-26 13:20:16] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:16:41 [I 2022-10-26 13:20:16] ...trap/engines/engine.py.229: process: 0.47974 | load: 0.00248 [I 2022-10-26 13:20:16] ...trap/engines/engine.py.230: loss: 0.26880 [I 2022-10-26 13:20:16] ...optimizers/trijoint.py.151: total_norm: 3.031521 [I 2022-10-26 13:20:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 340/2304 [I 2022-10-26 13:20:21] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:16:22 [I 2022-10-26 13:20:21] ...trap/engines/engine.py.229: process: 0.50280 | load: 0.00246 [I 2022-10-26 13:20:21] ...trap/engines/engine.py.230: loss: 0.18929 [I 2022-10-26 13:20:21] ...optimizers/trijoint.py.151: total_norm: 2.696656 [I 2022-10-26 13:20:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 350/2304 [I 2022-10-26 13:20:26] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:16:09 [I 2022-10-26 13:20:26] ...trap/engines/engine.py.229: process: 0.50016 | load: 0.00246 [I 2022-10-26 13:20:26] ...trap/engines/engine.py.230: loss: 0.33153 [I 2022-10-26 13:20:26] ...optimizers/trijoint.py.151: total_norm: 3.276544 [I 2022-10-26 13:20:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 360/2304 [I 2022-10-26 13:20:31] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:16:08 [I 2022-10-26 13:20:31] ...trap/engines/engine.py.229: process: 0.49321 | load: 0.00244 [I 2022-10-26 13:20:31] ...trap/engines/engine.py.230: loss: 0.16064 [I 2022-10-26 13:20:31] ...optimizers/trijoint.py.151: total_norm: 2.390548 [I 2022-10-26 13:20:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 370/2304 [I 2022-10-26 13:20:36] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:16:01 [I 2022-10-26 13:20:36] ...trap/engines/engine.py.229: process: 0.49887 | load: 0.00246 [I 2022-10-26 13:20:36] ...trap/engines/engine.py.230: loss: 0.24580 [I 2022-10-26 13:20:36] ...optimizers/trijoint.py.151: total_norm: 4.428546 [I 2022-10-26 13:20:40] ...trap/engines/engine.py.224: train: epoch 10 | batch 380/2304 [I 2022-10-26 13:20:40] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:15:46 [I 2022-10-26 13:20:40] ...trap/engines/engine.py.229: process: 0.48656 | load: 0.00253 [I 2022-10-26 13:20:40] ...trap/engines/engine.py.230: loss: 0.19343 [I 2022-10-26 13:20:40] ...optimizers/trijoint.py.151: total_norm: 2.168971 [I 2022-10-26 13:20:45] ...trap/engines/engine.py.224: train: epoch 10 | batch 390/2304 [I 2022-10-26 13:20:45] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:15:52 [I 2022-10-26 13:20:45] ...trap/engines/engine.py.229: process: 0.49364 | load: 0.00242 [I 2022-10-26 13:20:45] ...trap/engines/engine.py.230: loss: 0.28911 [I 2022-10-26 13:20:45] ...optimizers/trijoint.py.151: total_norm: 3.548931 [I 2022-10-26 13:20:50] ...trap/engines/engine.py.224: train: epoch 10 | batch 400/2304 [I 2022-10-26 13:20:50] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:15:51 [I 2022-10-26 13:20:50] ...trap/engines/engine.py.229: process: 0.49641 | load: 0.00258 [I 2022-10-26 13:20:50] ...trap/engines/engine.py.230: loss: 0.22109 [I 2022-10-26 13:20:50] ...optimizers/trijoint.py.151: total_norm: 5.192167 [I 2022-10-26 13:20:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 410/2304 [I 2022-10-26 13:20:56] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:15:45 [I 2022-10-26 13:20:56] ...trap/engines/engine.py.229: process: 0.49792 | load: 0.00259 [I 2022-10-26 13:20:56] ...trap/engines/engine.py.230: loss: 0.25652 [I 2022-10-26 13:20:56] ...optimizers/trijoint.py.151: total_norm: 3.442802 [I 2022-10-26 13:21:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 420/2304 [I 2022-10-26 13:21:01] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:15:43 [I 2022-10-26 13:21:01] ...trap/engines/engine.py.229: process: 0.49540 | load: 0.00253 [I 2022-10-26 13:21:01] ...trap/engines/engine.py.230: loss: 0.29979 [I 2022-10-26 13:21:01] ...optimizers/trijoint.py.151: total_norm: 4.068013 [I 2022-10-26 13:21:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 430/2304 [I 2022-10-26 13:21:06] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:15:36 [I 2022-10-26 13:21:06] ...trap/engines/engine.py.229: process: 0.50150 | load: 0.00251 [I 2022-10-26 13:21:06] ...trap/engines/engine.py.230: loss: 0.20838 [I 2022-10-26 13:21:06] ...optimizers/trijoint.py.151: total_norm: 2.908236 [I 2022-10-26 13:21:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 440/2304 [I 2022-10-26 13:21:11] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:15:27 [I 2022-10-26 13:21:11] ...trap/engines/engine.py.229: process: 0.49604 | load: 0.00246 [I 2022-10-26 13:21:11] ...trap/engines/engine.py.230: loss: 0.34593 [I 2022-10-26 13:21:11] ...optimizers/trijoint.py.151: total_norm: 4.277105 [I 2022-10-26 13:21:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 450/2304 [I 2022-10-26 13:21:16] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:15:28 [I 2022-10-26 13:21:16] ...trap/engines/engine.py.229: process: 0.50318 | load: 0.00254 [I 2022-10-26 13:21:16] ...trap/engines/engine.py.230: loss: 0.24365 [I 2022-10-26 13:21:16] ...optimizers/trijoint.py.151: total_norm: 3.363444 [I 2022-10-26 13:21:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 460/2304 [I 2022-10-26 13:21:21] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:15:16 [I 2022-10-26 13:21:21] ...trap/engines/engine.py.229: process: 0.50280 | load: 0.00245 [I 2022-10-26 13:21:21] ...trap/engines/engine.py.230: loss: 0.19978 [I 2022-10-26 13:21:21] ...optimizers/trijoint.py.151: total_norm: 2.205550 [I 2022-10-26 13:21:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 470/2304 [I 2022-10-26 13:21:26] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:15:25 [I 2022-10-26 13:21:26] ...trap/engines/engine.py.229: process: 0.50687 | load: 0.00249 [I 2022-10-26 13:21:26] ...trap/engines/engine.py.230: loss: 0.23413 [I 2022-10-26 13:21:26] ...optimizers/trijoint.py.151: total_norm: 3.692977 [I 2022-10-26 13:21:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 480/2304 [I 2022-10-26 13:21:31] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:15:15 [I 2022-10-26 13:21:31] ...trap/engines/engine.py.229: process: 0.50050 | load: 0.00243 [I 2022-10-26 13:21:31] ...trap/engines/engine.py.230: loss: 0.24977 [I 2022-10-26 13:21:31] ...optimizers/trijoint.py.151: total_norm: 3.618984 [I 2022-10-26 13:21:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 490/2304 [I 2022-10-26 13:21:36] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:15:01 [I 2022-10-26 13:21:36] ...trap/engines/engine.py.229: process: 0.48907 | load: 0.00252 [I 2022-10-26 13:21:36] ...trap/engines/engine.py.230: loss: 0.27288 [I 2022-10-26 13:21:36] ...optimizers/trijoint.py.151: total_norm: 3.420245 [I 2022-10-26 13:21:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 500/2304 [I 2022-10-26 13:21:41] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:15:03 [I 2022-10-26 13:21:41] ...trap/engines/engine.py.229: process: 0.49984 | load: 0.00244 [I 2022-10-26 13:21:41] ...trap/engines/engine.py.230: loss: 0.28352 [I 2022-10-26 13:21:41] ...optimizers/trijoint.py.151: total_norm: 3.195449 [I 2022-10-26 13:21:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 510/2304 [I 2022-10-26 13:21:46] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:15:01 [I 2022-10-26 13:21:46] ...trap/engines/engine.py.229: process: 0.50160 | load: 0.00251 [I 2022-10-26 13:21:46] ...trap/engines/engine.py.230: loss: 0.26994 [I 2022-10-26 13:21:46] ...optimizers/trijoint.py.151: total_norm: 3.931384 [I 2022-10-26 13:21:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 520/2304 [I 2022-10-26 13:21:51] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:14:49 [I 2022-10-26 13:21:51] ...trap/engines/engine.py.229: process: 0.49316 | load: 0.00250 [I 2022-10-26 13:21:51] ...trap/engines/engine.py.230: loss: 0.17757 [I 2022-10-26 13:21:51] ...optimizers/trijoint.py.151: total_norm: 3.859587 [I 2022-10-26 13:21:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 530/2304 [I 2022-10-26 13:21:56] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:14:47 [I 2022-10-26 13:21:56] ...trap/engines/engine.py.229: process: 0.49393 | load: 0.00250 [I 2022-10-26 13:21:56] ...trap/engines/engine.py.230: loss: 0.31700 [I 2022-10-26 13:21:56] ...optimizers/trijoint.py.151: total_norm: 5.423123 [I 2022-10-26 13:22:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 540/2304 [I 2022-10-26 13:22:01] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:14:43 [I 2022-10-26 13:22:01] ...trap/engines/engine.py.229: process: 0.50229 | load: 0.00264 [I 2022-10-26 13:22:01] ...trap/engines/engine.py.230: loss: 0.15881 [I 2022-10-26 13:22:01] ...optimizers/trijoint.py.151: total_norm: 2.474964 [I 2022-10-26 13:22:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 550/2304 [I 2022-10-26 13:22:06] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:14:46 [I 2022-10-26 13:22:06] ...trap/engines/engine.py.229: process: 0.53367 | load: 0.00248 [I 2022-10-26 13:22:06] ...trap/engines/engine.py.230: loss: 0.23676 [I 2022-10-26 13:22:06] ...optimizers/trijoint.py.151: total_norm: 3.297930 [I 2022-10-26 13:22:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 560/2304 [I 2022-10-26 13:22:11] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:14:26 [I 2022-10-26 13:22:11] ...trap/engines/engine.py.229: process: 0.48385 | load: 0.00255 [I 2022-10-26 13:22:11] ...trap/engines/engine.py.230: loss: 0.32403 [I 2022-10-26 13:22:11] ...optimizers/trijoint.py.151: total_norm: 4.043702 [I 2022-10-26 13:22:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 570/2304 [I 2022-10-26 13:22:16] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:14:22 [I 2022-10-26 13:22:16] ...trap/engines/engine.py.229: process: 0.49316 | load: 0.00247 [I 2022-10-26 13:22:16] ...trap/engines/engine.py.230: loss: 0.22646 [I 2022-10-26 13:22:16] ...optimizers/trijoint.py.151: total_norm: 3.107871 [I 2022-10-26 13:22:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 580/2304 [I 2022-10-26 13:22:21] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:14:26 [I 2022-10-26 13:22:21] ...trap/engines/engine.py.229: process: 0.49895 | load: 0.00244 [I 2022-10-26 13:22:21] ...trap/engines/engine.py.230: loss: 0.21490 [I 2022-10-26 13:22:21] ...optimizers/trijoint.py.151: total_norm: 3.718335 [I 2022-10-26 13:22:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 590/2304 [I 2022-10-26 13:22:26] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:14:09 [I 2022-10-26 13:22:26] ...trap/engines/engine.py.229: process: 0.49350 | load: 0.00247 [I 2022-10-26 13:22:26] ...trap/engines/engine.py.230: loss: 0.23523 [I 2022-10-26 13:22:26] ...optimizers/trijoint.py.151: total_norm: 2.993005 [I 2022-10-26 13:22:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 600/2304 [I 2022-10-26 13:22:31] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:14:06 [I 2022-10-26 13:22:31] ...trap/engines/engine.py.229: process: 0.50245 | load: 0.00243 [I 2022-10-26 13:22:31] ...trap/engines/engine.py.230: loss: 0.36542 [I 2022-10-26 13:22:31] ...optimizers/trijoint.py.151: total_norm: 3.927064 [I 2022-10-26 13:22:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 610/2304 [I 2022-10-26 13:22:36] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:13:58 [I 2022-10-26 13:22:36] ...trap/engines/engine.py.229: process: 0.48788 | load: 0.00245 [I 2022-10-26 13:22:36] ...trap/engines/engine.py.230: loss: 0.25447 [I 2022-10-26 13:22:36] ...optimizers/trijoint.py.151: total_norm: 3.672037 [I 2022-10-26 13:22:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 620/2304 [I 2022-10-26 13:22:41] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:13:53 [I 2022-10-26 13:22:41] ...trap/engines/engine.py.229: process: 0.49381 | load: 0.00248 [I 2022-10-26 13:22:41] ...trap/engines/engine.py.230: loss: 0.27132 [I 2022-10-26 13:22:41] ...optimizers/trijoint.py.151: total_norm: 3.488266 [I 2022-10-26 13:22:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 630/2304 [I 2022-10-26 13:22:46] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:13:58 [I 2022-10-26 13:22:46] ...trap/engines/engine.py.229: process: 0.49758 | load: 0.00246 [I 2022-10-26 13:22:46] ...trap/engines/engine.py.230: loss: 0.23101 [I 2022-10-26 13:22:46] ...optimizers/trijoint.py.151: total_norm: 3.367743 [I 2022-10-26 13:22:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 640/2304 [I 2022-10-26 13:22:51] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:13:49 [I 2022-10-26 13:22:51] ...trap/engines/engine.py.229: process: 0.49379 | load: 0.00262 [I 2022-10-26 13:22:51] ...trap/engines/engine.py.230: loss: 0.19204 [I 2022-10-26 13:22:51] ...optimizers/trijoint.py.151: total_norm: 2.418444 [I 2022-10-26 13:22:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 650/2304 [I 2022-10-26 13:22:56] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:13:44 [I 2022-10-26 13:22:56] ...trap/engines/engine.py.229: process: 0.50995 | load: 0.00247 [I 2022-10-26 13:22:56] ...trap/engines/engine.py.230: loss: 0.25085 [I 2022-10-26 13:22:56] ...optimizers/trijoint.py.151: total_norm: 3.159864 [I 2022-10-26 13:23:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 660/2304 [I 2022-10-26 13:23:01] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:13:32 [I 2022-10-26 13:23:01] ...trap/engines/engine.py.229: process: 0.49698 | load: 0.00246 [I 2022-10-26 13:23:01] ...trap/engines/engine.py.230: loss: 0.21867 [I 2022-10-26 13:23:01] ...optimizers/trijoint.py.151: total_norm: 2.381698 [I 2022-10-26 13:23:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 670/2304 [I 2022-10-26 13:23:06] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:13:37 [I 2022-10-26 13:23:06] ...trap/engines/engine.py.229: process: 0.50396 | load: 0.00245 [I 2022-10-26 13:23:06] ...trap/engines/engine.py.230: loss: 0.35293 [I 2022-10-26 13:23:06] ...optimizers/trijoint.py.151: total_norm: 4.395538 [I 2022-10-26 13:23:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 680/2304 [I 2022-10-26 13:23:11] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:13:28 [I 2022-10-26 13:23:11] ...trap/engines/engine.py.229: process: 0.49775 | load: 0.00250 [I 2022-10-26 13:23:11] ...trap/engines/engine.py.230: loss: 0.27841 [I 2022-10-26 13:23:11] ...optimizers/trijoint.py.151: total_norm: 2.972179 [I 2022-10-26 13:23:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 690/2304 [I 2022-10-26 13:23:16] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:13:27 [I 2022-10-26 13:23:16] ...trap/engines/engine.py.229: process: 0.50428 | load: 0.00246 [I 2022-10-26 13:23:16] ...trap/engines/engine.py.230: loss: 0.23480 [I 2022-10-26 13:23:16] ...optimizers/trijoint.py.151: total_norm: 3.209999 [I 2022-10-26 13:23:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 700/2304 [I 2022-10-26 13:23:21] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:13:26 [I 2022-10-26 13:23:21] ...trap/engines/engine.py.229: process: 0.50231 | load: 0.00249 [I 2022-10-26 13:23:21] ...trap/engines/engine.py.230: loss: 0.13719 [I 2022-10-26 13:23:21] ...optimizers/trijoint.py.151: total_norm: 1.956512 [I 2022-10-26 13:23:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 710/2304 [I 2022-10-26 13:23:26] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:13:15 [I 2022-10-26 13:23:26] ...trap/engines/engine.py.229: process: 0.49762 | load: 0.00250 [I 2022-10-26 13:23:26] ...trap/engines/engine.py.230: loss: 0.21084 [I 2022-10-26 13:23:26] ...optimizers/trijoint.py.151: total_norm: 2.612545 [I 2022-10-26 13:23:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 720/2304 [I 2022-10-26 13:23:31] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:13:11 [I 2022-10-26 13:23:31] ...trap/engines/engine.py.229: process: 0.50325 | load: 0.00245 [I 2022-10-26 13:23:31] ...trap/engines/engine.py.230: loss: 0.33853 [I 2022-10-26 13:23:31] ...optimizers/trijoint.py.151: total_norm: 3.512641 [I 2022-10-26 13:23:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 730/2304 [I 2022-10-26 13:23:36] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:12:56 [I 2022-10-26 13:23:36] ...trap/engines/engine.py.229: process: 0.49083 | load: 0.00247 [I 2022-10-26 13:23:36] ...trap/engines/engine.py.230: loss: 0.23076 [I 2022-10-26 13:23:36] ...optimizers/trijoint.py.151: total_norm: 3.198658 [I 2022-10-26 13:23:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 740/2304 [I 2022-10-26 13:23:41] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:12:49 [I 2022-10-26 13:23:41] ...trap/engines/engine.py.229: process: 0.48427 | load: 0.00239 [I 2022-10-26 13:23:41] ...trap/engines/engine.py.230: loss: 0.23238 [I 2022-10-26 13:23:41] ...optimizers/trijoint.py.151: total_norm: 2.966634 [I 2022-10-26 13:23:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 750/2304 [I 2022-10-26 13:23:46] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:12:51 [I 2022-10-26 13:23:46] ...trap/engines/engine.py.229: process: 0.50030 | load: 0.00242 [I 2022-10-26 13:23:46] ...trap/engines/engine.py.230: loss: 0.26222 [I 2022-10-26 13:23:46] ...optimizers/trijoint.py.151: total_norm: 3.339335 [I 2022-10-26 13:23:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 760/2304 [I 2022-10-26 13:23:51] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:12:50 [I 2022-10-26 13:23:51] ...trap/engines/engine.py.229: process: 0.49234 | load: 0.00244 [I 2022-10-26 13:23:51] ...trap/engines/engine.py.230: loss: 0.17055 [I 2022-10-26 13:23:51] ...optimizers/trijoint.py.151: total_norm: 2.475384 [I 2022-10-26 13:23:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 770/2304 [I 2022-10-26 13:23:56] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:12:40 [I 2022-10-26 13:23:56] ...trap/engines/engine.py.229: process: 0.48782 | load: 0.00242 [I 2022-10-26 13:23:56] ...trap/engines/engine.py.230: loss: 0.29217 [I 2022-10-26 13:23:56] ...optimizers/trijoint.py.151: total_norm: 3.251362 [I 2022-10-26 13:24:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 780/2304 [I 2022-10-26 13:24:01] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:12:47 [I 2022-10-26 13:24:01] ...trap/engines/engine.py.229: process: 0.50626 | load: 0.00262 [I 2022-10-26 13:24:01] ...trap/engines/engine.py.230: loss: 0.24675 [I 2022-10-26 13:24:01] ...optimizers/trijoint.py.151: total_norm: 2.946095 [I 2022-10-26 13:24:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 790/2304 [I 2022-10-26 13:24:06] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:12:36 [I 2022-10-26 13:24:06] ...trap/engines/engine.py.229: process: 0.49782 | load: 0.00241 [I 2022-10-26 13:24:06] ...trap/engines/engine.py.230: loss: 0.31092 [I 2022-10-26 13:24:06] ...optimizers/trijoint.py.151: total_norm: 3.281880 [I 2022-10-26 13:24:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 800/2304 [I 2022-10-26 13:24:11] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:12:32 [I 2022-10-26 13:24:11] ...trap/engines/engine.py.229: process: 0.50450 | load: 0.00250 [I 2022-10-26 13:24:11] ...trap/engines/engine.py.230: loss: 0.24393 [I 2022-10-26 13:24:11] ...optimizers/trijoint.py.151: total_norm: 4.554909 [I 2022-10-26 13:24:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 810/2304 [I 2022-10-26 13:24:16] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:12:27 [I 2022-10-26 13:24:16] ...trap/engines/engine.py.229: process: 0.50073 | load: 0.00248 [I 2022-10-26 13:24:16] ...trap/engines/engine.py.230: loss: 0.19665 [I 2022-10-26 13:24:16] ...optimizers/trijoint.py.151: total_norm: 2.917114 [I 2022-10-26 13:24:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 820/2304 [I 2022-10-26 13:24:21] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:12:19 [I 2022-10-26 13:24:21] ...trap/engines/engine.py.229: process: 0.49769 | load: 0.00245 [I 2022-10-26 13:24:21] ...trap/engines/engine.py.230: loss: 0.19077 [I 2022-10-26 13:24:21] ...optimizers/trijoint.py.151: total_norm: 3.143816 [I 2022-10-26 13:24:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 830/2304 [I 2022-10-26 13:24:26] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:12:10 [I 2022-10-26 13:24:26] ...trap/engines/engine.py.229: process: 0.48944 | load: 0.00252 [I 2022-10-26 13:24:26] ...trap/engines/engine.py.230: loss: 0.17749 [I 2022-10-26 13:24:26] ...optimizers/trijoint.py.151: total_norm: 2.912610 [I 2022-10-26 13:24:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 840/2304 [I 2022-10-26 13:24:31] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:12:08 [I 2022-10-26 13:24:31] ...trap/engines/engine.py.229: process: 0.49142 | load: 0.00247 [I 2022-10-26 13:24:31] ...trap/engines/engine.py.230: loss: 0.21649 [I 2022-10-26 13:24:31] ...optimizers/trijoint.py.151: total_norm: 3.271469 [I 2022-10-26 13:24:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 850/2304 [I 2022-10-26 13:24:36] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:12:14 [I 2022-10-26 13:24:36] ...trap/engines/engine.py.229: process: 0.51188 | load: 0.00250 [I 2022-10-26 13:24:36] ...trap/engines/engine.py.230: loss: 0.24896 [I 2022-10-26 13:24:36] ...optimizers/trijoint.py.151: total_norm: 3.201355 [I 2022-10-26 13:24:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 860/2304 [I 2022-10-26 13:24:41] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:12:03 [I 2022-10-26 13:24:41] ...trap/engines/engine.py.229: process: 0.49878 | load: 0.00252 [I 2022-10-26 13:24:41] ...trap/engines/engine.py.230: loss: 0.23545 [I 2022-10-26 13:24:41] ...optimizers/trijoint.py.151: total_norm: 3.864636 [I 2022-10-26 13:24:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 870/2304 [I 2022-10-26 13:24:46] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:11:53 [I 2022-10-26 13:24:46] ...trap/engines/engine.py.229: process: 0.49315 | load: 0.00247 [I 2022-10-26 13:24:46] ...trap/engines/engine.py.230: loss: 0.22864 [I 2022-10-26 13:24:46] ...optimizers/trijoint.py.151: total_norm: 3.517173 [I 2022-10-26 13:24:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 880/2304 [I 2022-10-26 13:24:51] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:11:47 [I 2022-10-26 13:24:51] ...trap/engines/engine.py.229: process: 0.48804 | load: 0.00258 [I 2022-10-26 13:24:51] ...trap/engines/engine.py.230: loss: 0.24952 [I 2022-10-26 13:24:51] ...optimizers/trijoint.py.151: total_norm: 3.475711 [I 2022-10-26 13:24:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 890/2304 [I 2022-10-26 13:24:56] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:11:46 [I 2022-10-26 13:24:56] ...trap/engines/engine.py.229: process: 0.51351 | load: 0.00255 [I 2022-10-26 13:24:56] ...trap/engines/engine.py.230: loss: 0.18876 [I 2022-10-26 13:24:56] ...optimizers/trijoint.py.151: total_norm: 2.678735 [I 2022-10-26 13:25:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 900/2304 [I 2022-10-26 13:25:01] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:11:42 [I 2022-10-26 13:25:01] ...trap/engines/engine.py.229: process: 0.50546 | load: 0.00244 [I 2022-10-26 13:25:01] ...trap/engines/engine.py.230: loss: 0.16655 [I 2022-10-26 13:25:01] ...optimizers/trijoint.py.151: total_norm: 3.585176 [I 2022-10-26 13:25:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 910/2304 [I 2022-10-26 13:25:06] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:11:38 [I 2022-10-26 13:25:06] ...trap/engines/engine.py.229: process: 0.50488 | load: 0.00244 [I 2022-10-26 13:25:06] ...trap/engines/engine.py.230: loss: 0.20633 [I 2022-10-26 13:25:06] ...optimizers/trijoint.py.151: total_norm: 2.522744 [I 2022-10-26 13:25:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 920/2304 [I 2022-10-26 13:25:11] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:11:29 [I 2022-10-26 13:25:11] ...trap/engines/engine.py.229: process: 0.50150 | load: 0.00244 [I 2022-10-26 13:25:11] ...trap/engines/engine.py.230: loss: 0.26205 [I 2022-10-26 13:25:11] ...optimizers/trijoint.py.151: total_norm: 3.163366 [I 2022-10-26 13:25:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 930/2304 [I 2022-10-26 13:25:16] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:11:24 [I 2022-10-26 13:25:16] ...trap/engines/engine.py.229: process: 0.50345 | load: 0.00243 [I 2022-10-26 13:25:16] ...trap/engines/engine.py.230: loss: 0.21126 [I 2022-10-26 13:25:16] ...optimizers/trijoint.py.151: total_norm: 3.289547 [I 2022-10-26 13:25:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 940/2304 [I 2022-10-26 13:25:21] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:11:31 [I 2022-10-26 13:25:21] ...trap/engines/engine.py.229: process: 0.50176 | load: 0.00242 [I 2022-10-26 13:25:21] ...trap/engines/engine.py.230: loss: 0.26845 [I 2022-10-26 13:25:21] ...optimizers/trijoint.py.151: total_norm: 3.368400 [I 2022-10-26 13:25:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 950/2304 [I 2022-10-26 13:25:26] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:11:13 [I 2022-10-26 13:25:26] ...trap/engines/engine.py.229: process: 0.50173 | load: 0.00246 [I 2022-10-26 13:25:26] ...trap/engines/engine.py.230: loss: 0.14961 [I 2022-10-26 13:25:26] ...optimizers/trijoint.py.151: total_norm: 2.348442 [I 2022-10-26 13:25:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 960/2304 [I 2022-10-26 13:25:31] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:11:32 [I 2022-10-26 13:25:31] ...trap/engines/engine.py.229: process: 0.49990 | load: 0.00244 [I 2022-10-26 13:25:31] ...trap/engines/engine.py.230: loss: 0.30156 [I 2022-10-26 13:25:31] ...optimizers/trijoint.py.151: total_norm: 3.039552 [I 2022-10-26 13:25:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 970/2304 [I 2022-10-26 13:25:36] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:11:03 [I 2022-10-26 13:25:36] ...trap/engines/engine.py.229: process: 0.50127 | load: 0.00260 [I 2022-10-26 13:25:36] ...trap/engines/engine.py.230: loss: 0.23025 [I 2022-10-26 13:25:36] ...optimizers/trijoint.py.151: total_norm: 3.136446 [I 2022-10-26 13:25:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 980/2304 [I 2022-10-26 13:25:41] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:10:54 [I 2022-10-26 13:25:41] ...trap/engines/engine.py.229: process: 0.49169 | load: 0.00252 [I 2022-10-26 13:25:41] ...trap/engines/engine.py.230: loss: 0.26976 [I 2022-10-26 13:25:41] ...optimizers/trijoint.py.151: total_norm: 3.358719 [I 2022-10-26 13:25:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 990/2304 [I 2022-10-26 13:25:46] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:10:51 [I 2022-10-26 13:25:46] ...trap/engines/engine.py.229: process: 0.49329 | load: 0.00249 [I 2022-10-26 13:25:46] ...trap/engines/engine.py.230: loss: 0.29464 [I 2022-10-26 13:25:46] ...optimizers/trijoint.py.151: total_norm: 3.860733 [I 2022-10-26 13:25:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 1000/2304 [I 2022-10-26 13:25:51] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:10:47 [I 2022-10-26 13:25:51] ...trap/engines/engine.py.229: process: 0.50141 | load: 0.00247 [I 2022-10-26 13:25:51] ...trap/engines/engine.py.230: loss: 0.16548 [I 2022-10-26 13:25:51] ...optimizers/trijoint.py.151: total_norm: 1.926744 [I 2022-10-26 13:25:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 1010/2304 [I 2022-10-26 13:25:56] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:10:44 [I 2022-10-26 13:25:56] ...trap/engines/engine.py.229: process: 0.50685 | load: 0.00245 [I 2022-10-26 13:25:56] ...trap/engines/engine.py.230: loss: 0.25319 [I 2022-10-26 13:25:56] ...optimizers/trijoint.py.151: total_norm: 3.778802 [I 2022-10-26 13:26:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 1020/2304 [I 2022-10-26 13:26:01] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:10:45 [I 2022-10-26 13:26:01] ...trap/engines/engine.py.229: process: 0.51705 | load: 0.00244 [I 2022-10-26 13:26:01] ...trap/engines/engine.py.230: loss: 0.18572 [I 2022-10-26 13:26:01] ...optimizers/trijoint.py.151: total_norm: 2.962126 [I 2022-10-26 13:26:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 1030/2304 [I 2022-10-26 13:26:06] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:10:35 [I 2022-10-26 13:26:06] ...trap/engines/engine.py.229: process: 0.49504 | load: 0.00242 [I 2022-10-26 13:26:06] ...trap/engines/engine.py.230: loss: 0.14231 [I 2022-10-26 13:26:06] ...optimizers/trijoint.py.151: total_norm: 2.091347 [I 2022-10-26 13:26:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 1040/2304 [I 2022-10-26 13:26:11] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:10:50 [I 2022-10-26 13:26:11] ...trap/engines/engine.py.229: process: 0.49833 | load: 0.00251 [I 2022-10-26 13:26:11] ...trap/engines/engine.py.230: loss: 0.31653 [I 2022-10-26 13:26:11] ...optimizers/trijoint.py.151: total_norm: 3.088062 [I 2022-10-26 13:26:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 1050/2304 [I 2022-10-26 13:26:16] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:10:28 [I 2022-10-26 13:26:16] ...trap/engines/engine.py.229: process: 0.49270 | load: 0.00252 [I 2022-10-26 13:26:16] ...trap/engines/engine.py.230: loss: 0.12865 [I 2022-10-26 13:26:16] ...optimizers/trijoint.py.151: total_norm: 2.108134 [I 2022-10-26 13:26:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 1060/2304 [I 2022-10-26 13:26:21] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:10:16 [I 2022-10-26 13:26:21] ...trap/engines/engine.py.229: process: 0.49504 | load: 0.00248 [I 2022-10-26 13:26:21] ...trap/engines/engine.py.230: loss: 0.26438 [I 2022-10-26 13:26:21] ...optimizers/trijoint.py.151: total_norm: 3.328834 [I 2022-10-26 13:26:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 1070/2304 [I 2022-10-26 13:26:26] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:10:17 [I 2022-10-26 13:26:26] ...trap/engines/engine.py.229: process: 0.50504 | load: 0.00245 [I 2022-10-26 13:26:26] ...trap/engines/engine.py.230: loss: 0.20470 [I 2022-10-26 13:26:26] ...optimizers/trijoint.py.151: total_norm: 3.145906 [I 2022-10-26 13:26:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 1080/2304 [I 2022-10-26 13:26:31] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:10:05 [I 2022-10-26 13:26:31] ...trap/engines/engine.py.229: process: 0.50070 | load: 0.00264 [I 2022-10-26 13:26:31] ...trap/engines/engine.py.230: loss: 0.21524 [I 2022-10-26 13:26:31] ...optimizers/trijoint.py.151: total_norm: 3.956236 [I 2022-10-26 13:26:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 1090/2304 [I 2022-10-26 13:26:36] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:10:01 [I 2022-10-26 13:26:36] ...trap/engines/engine.py.229: process: 0.49928 | load: 0.00243 [I 2022-10-26 13:26:36] ...trap/engines/engine.py.230: loss: 0.15457 [I 2022-10-26 13:26:36] ...optimizers/trijoint.py.151: total_norm: 3.659282 [I 2022-10-26 13:26:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 1100/2304 [I 2022-10-26 13:26:41] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:10:05 [I 2022-10-26 13:26:41] ...trap/engines/engine.py.229: process: 0.50219 | load: 0.00251 [I 2022-10-26 13:26:41] ...trap/engines/engine.py.230: loss: 0.22188 [I 2022-10-26 13:26:41] ...optimizers/trijoint.py.151: total_norm: 2.669343 [I 2022-10-26 13:26:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 1110/2304 [I 2022-10-26 13:26:46] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:10:09 [I 2022-10-26 13:26:46] ...trap/engines/engine.py.229: process: 0.50211 | load: 0.00248 [I 2022-10-26 13:26:46] ...trap/engines/engine.py.230: loss: 0.19668 [I 2022-10-26 13:26:46] ...optimizers/trijoint.py.151: total_norm: 3.508232 [I 2022-10-26 13:26:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 1120/2304 [I 2022-10-26 13:26:51] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:09:46 [I 2022-10-26 13:26:51] ...trap/engines/engine.py.229: process: 0.49697 | load: 0.00253 [I 2022-10-26 13:26:51] ...trap/engines/engine.py.230: loss: 0.27096 [I 2022-10-26 13:26:51] ...optimizers/trijoint.py.151: total_norm: 4.313886 [I 2022-10-26 13:26:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 1130/2304 [I 2022-10-26 13:26:56] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:09:40 [I 2022-10-26 13:26:56] ...trap/engines/engine.py.229: process: 0.50008 | load: 0.00248 [I 2022-10-26 13:26:56] ...trap/engines/engine.py.230: loss: 0.21557 [I 2022-10-26 13:26:56] ...optimizers/trijoint.py.151: total_norm: 2.409133 [I 2022-10-26 13:27:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 1140/2304 [I 2022-10-26 13:27:01] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:09:36 [I 2022-10-26 13:27:01] ...trap/engines/engine.py.229: process: 0.49955 | load: 0.00244 [I 2022-10-26 13:27:01] ...trap/engines/engine.py.230: loss: 0.20641 [I 2022-10-26 13:27:01] ...optimizers/trijoint.py.151: total_norm: 2.517459 [I 2022-10-26 13:27:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 1150/2304 [I 2022-10-26 13:27:06] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:09:31 [I 2022-10-26 13:27:06] ...trap/engines/engine.py.229: process: 0.49830 | load: 0.00246 [I 2022-10-26 13:27:06] ...trap/engines/engine.py.230: loss: 0.20209 [I 2022-10-26 13:27:06] ...optimizers/trijoint.py.151: total_norm: 3.268390 [I 2022-10-26 13:27:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 1160/2304 [I 2022-10-26 13:27:11] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:09:33 [I 2022-10-26 13:27:11] ...trap/engines/engine.py.229: process: 0.50966 | load: 0.00256 [I 2022-10-26 13:27:11] ...trap/engines/engine.py.230: loss: 0.33108 [I 2022-10-26 13:27:11] ...optimizers/trijoint.py.151: total_norm: 5.255906 [I 2022-10-26 13:27:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 1170/2304 [I 2022-10-26 13:27:16] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:09:33 [I 2022-10-26 13:27:16] ...trap/engines/engine.py.229: process: 0.50441 | load: 0.00244 [I 2022-10-26 13:27:16] ...trap/engines/engine.py.230: loss: 0.23478 [I 2022-10-26 13:27:16] ...optimizers/trijoint.py.151: total_norm: 3.658530 [I 2022-10-26 13:27:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 1180/2304 [I 2022-10-26 13:27:21] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:09:23 [I 2022-10-26 13:27:21] ...trap/engines/engine.py.229: process: 0.50486 | load: 0.00252 [I 2022-10-26 13:27:21] ...trap/engines/engine.py.230: loss: 0.18993 [I 2022-10-26 13:27:21] ...optimizers/trijoint.py.151: total_norm: 3.695719 [I 2022-10-26 13:27:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 1190/2304 [I 2022-10-26 13:27:26] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:09:18 [I 2022-10-26 13:27:26] ...trap/engines/engine.py.229: process: 0.50473 | load: 0.00248 [I 2022-10-26 13:27:26] ...trap/engines/engine.py.230: loss: 0.21793 [I 2022-10-26 13:27:26] ...optimizers/trijoint.py.151: total_norm: 2.829693 [I 2022-10-26 13:27:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 1200/2304 [I 2022-10-26 13:27:31] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:09:13 [I 2022-10-26 13:27:31] ...trap/engines/engine.py.229: process: 0.49471 | load: 0.00246 [I 2022-10-26 13:27:31] ...trap/engines/engine.py.230: loss: 0.35182 [I 2022-10-26 13:27:31] ...optimizers/trijoint.py.151: total_norm: 3.538295 [I 2022-10-26 13:27:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 1210/2304 [I 2022-10-26 13:27:36] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:09:11 [I 2022-10-26 13:27:36] ...trap/engines/engine.py.229: process: 0.50883 | load: 0.00245 [I 2022-10-26 13:27:36] ...trap/engines/engine.py.230: loss: 0.27371 [I 2022-10-26 13:27:36] ...optimizers/trijoint.py.151: total_norm: 3.943322 [I 2022-10-26 13:27:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 1220/2304 [I 2022-10-26 13:27:41] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:09:15 [I 2022-10-26 13:27:41] ...trap/engines/engine.py.229: process: 0.50351 | load: 0.00246 [I 2022-10-26 13:27:41] ...trap/engines/engine.py.230: loss: 0.20736 [I 2022-10-26 13:27:41] ...optimizers/trijoint.py.151: total_norm: 3.596761 [I 2022-10-26 13:27:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 1230/2304 [I 2022-10-26 13:27:46] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:09:05 [I 2022-10-26 13:27:46] ...trap/engines/engine.py.229: process: 0.50051 | load: 0.00244 [I 2022-10-26 13:27:46] ...trap/engines/engine.py.230: loss: 0.23565 [I 2022-10-26 13:27:46] ...optimizers/trijoint.py.151: total_norm: 3.384373 [I 2022-10-26 13:27:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 1240/2304 [I 2022-10-26 13:27:51] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:08:52 [I 2022-10-26 13:27:51] ...trap/engines/engine.py.229: process: 0.50087 | load: 0.00248 [I 2022-10-26 13:27:51] ...trap/engines/engine.py.230: loss: 0.22087 [I 2022-10-26 13:27:51] ...optimizers/trijoint.py.151: total_norm: 4.066803 [I 2022-10-26 13:27:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 1250/2304 [I 2022-10-26 13:27:56] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:08:48 [I 2022-10-26 13:27:56] ...trap/engines/engine.py.229: process: 0.50452 | load: 0.00242 [I 2022-10-26 13:27:56] ...trap/engines/engine.py.230: loss: 0.19272 [I 2022-10-26 13:27:56] ...optimizers/trijoint.py.151: total_norm: 3.670014 [I 2022-10-26 13:28:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 1260/2304 [I 2022-10-26 13:28:01] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:08:39 [I 2022-10-26 13:28:01] ...trap/engines/engine.py.229: process: 0.49530 | load: 0.00255 [I 2022-10-26 13:28:01] ...trap/engines/engine.py.230: loss: 0.20279 [I 2022-10-26 13:28:01] ...optimizers/trijoint.py.151: total_norm: 3.141145 [I 2022-10-26 13:28:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 1270/2304 [I 2022-10-26 13:28:06] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:08:32 [I 2022-10-26 13:28:06] ...trap/engines/engine.py.229: process: 0.48478 | load: 0.00248 [I 2022-10-26 13:28:06] ...trap/engines/engine.py.230: loss: 0.16469 [I 2022-10-26 13:28:06] ...optimizers/trijoint.py.151: total_norm: 2.213605 [I 2022-10-26 13:28:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 1280/2304 [I 2022-10-26 13:28:11] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:08:31 [I 2022-10-26 13:28:11] ...trap/engines/engine.py.229: process: 0.50823 | load: 0.00246 [I 2022-10-26 13:28:11] ...trap/engines/engine.py.230: loss: 0.25827 [I 2022-10-26 13:28:11] ...optimizers/trijoint.py.151: total_norm: 3.463974 [I 2022-10-26 13:28:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 1290/2304 [I 2022-10-26 13:28:16] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:08:26 [I 2022-10-26 13:28:16] ...trap/engines/engine.py.229: process: 0.48932 | load: 0.00245 [I 2022-10-26 13:28:16] ...trap/engines/engine.py.230: loss: 0.25248 [I 2022-10-26 13:28:16] ...optimizers/trijoint.py.151: total_norm: 3.365467 [I 2022-10-26 13:28:21] ...trap/engines/engine.py.224: train: epoch 10 | batch 1300/2304 [I 2022-10-26 13:28:21] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:08:18 [I 2022-10-26 13:28:21] ...trap/engines/engine.py.229: process: 0.49041 | load: 0.00248 [I 2022-10-26 13:28:21] ...trap/engines/engine.py.230: loss: 0.23374 [I 2022-10-26 13:28:21] ...optimizers/trijoint.py.151: total_norm: 3.227849 [I 2022-10-26 13:28:26] ...trap/engines/engine.py.224: train: epoch 10 | batch 1310/2304 [I 2022-10-26 13:28:26] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:08:14 [I 2022-10-26 13:28:26] ...trap/engines/engine.py.229: process: 0.49836 | load: 0.00246 [I 2022-10-26 13:28:26] ...trap/engines/engine.py.230: loss: 0.25043 [I 2022-10-26 13:28:26] ...optimizers/trijoint.py.151: total_norm: 3.002611 [I 2022-10-26 13:28:31] ...trap/engines/engine.py.224: train: epoch 10 | batch 1320/2304 [I 2022-10-26 13:28:31] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:08:07 [I 2022-10-26 13:28:31] ...trap/engines/engine.py.229: process: 0.48918 | load: 0.00249 [I 2022-10-26 13:28:31] ...trap/engines/engine.py.230: loss: 0.29381 [I 2022-10-26 13:28:31] ...optimizers/trijoint.py.151: total_norm: 4.332770 [I 2022-10-26 13:28:36] ...trap/engines/engine.py.224: train: epoch 10 | batch 1330/2304 [I 2022-10-26 13:28:36] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:08:03 [I 2022-10-26 13:28:36] ...trap/engines/engine.py.229: process: 0.50139 | load: 0.00244 [I 2022-10-26 13:28:36] ...trap/engines/engine.py.230: loss: 0.25934 [I 2022-10-26 13:28:36] ...optimizers/trijoint.py.151: total_norm: 3.730652 [I 2022-10-26 13:28:41] ...trap/engines/engine.py.224: train: epoch 10 | batch 1340/2304 [I 2022-10-26 13:28:41] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:08:01 [I 2022-10-26 13:28:41] ...trap/engines/engine.py.229: process: 0.50196 | load: 0.00245 [I 2022-10-26 13:28:41] ...trap/engines/engine.py.230: loss: 0.30225 [I 2022-10-26 13:28:41] ...optimizers/trijoint.py.151: total_norm: 3.466742 [I 2022-10-26 13:28:46] ...trap/engines/engine.py.224: train: epoch 10 | batch 1350/2304 [I 2022-10-26 13:28:46] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:08:23 [I 2022-10-26 13:28:46] ...trap/engines/engine.py.229: process: 0.58669 | load: 0.00270 [I 2022-10-26 13:28:46] ...trap/engines/engine.py.230: loss: 0.23525 [I 2022-10-26 13:28:46] ...optimizers/trijoint.py.151: total_norm: 4.700934 [I 2022-10-26 13:28:51] ...trap/engines/engine.py.224: train: epoch 10 | batch 1360/2304 [I 2022-10-26 13:28:51] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:07:57 [I 2022-10-26 13:28:51] ...trap/engines/engine.py.229: process: 0.49706 | load: 0.00245 [I 2022-10-26 13:28:51] ...trap/engines/engine.py.230: loss: 0.18234 [I 2022-10-26 13:28:51] ...optimizers/trijoint.py.151: total_norm: 2.835148 [I 2022-10-26 13:28:56] ...trap/engines/engine.py.224: train: epoch 10 | batch 1370/2304 [I 2022-10-26 13:28:56] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:07:49 [I 2022-10-26 13:28:56] ...trap/engines/engine.py.229: process: 0.49832 | load: 0.00244 [I 2022-10-26 13:28:56] ...trap/engines/engine.py.230: loss: 0.18079 [I 2022-10-26 13:28:56] ...optimizers/trijoint.py.151: total_norm: 2.679005 [I 2022-10-26 13:29:01] ...trap/engines/engine.py.224: train: epoch 10 | batch 1380/2304 [I 2022-10-26 13:29:01] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:07:41 [I 2022-10-26 13:29:01] ...trap/engines/engine.py.229: process: 0.50331 | load: 0.00250 [I 2022-10-26 13:29:01] ...trap/engines/engine.py.230: loss: 0.29039 [I 2022-10-26 13:29:01] ...optimizers/trijoint.py.151: total_norm: 3.888950 [I 2022-10-26 13:29:06] ...trap/engines/engine.py.224: train: epoch 10 | batch 1390/2304 [I 2022-10-26 13:29:06] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:07:34 [I 2022-10-26 13:29:06] ...trap/engines/engine.py.229: process: 0.49643 | load: 0.00249 [I 2022-10-26 13:29:06] ...trap/engines/engine.py.230: loss: 0.20726 [I 2022-10-26 13:29:06] ...optimizers/trijoint.py.151: total_norm: 3.292044 [I 2022-10-26 13:29:11] ...trap/engines/engine.py.224: train: epoch 10 | batch 1400/2304 [I 2022-10-26 13:29:11] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:07:33 [I 2022-10-26 13:29:11] ...trap/engines/engine.py.229: process: 0.50150 | load: 0.00268 [I 2022-10-26 13:29:11] ...trap/engines/engine.py.230: loss: 0.31923 [I 2022-10-26 13:29:11] ...optimizers/trijoint.py.151: total_norm: 5.138742 [I 2022-10-26 13:29:16] ...trap/engines/engine.py.224: train: epoch 10 | batch 1410/2304 [I 2022-10-26 13:29:16] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:07:32 [I 2022-10-26 13:29:16] ...trap/engines/engine.py.229: process: 0.50750 | load: 0.00248 [I 2022-10-26 13:29:16] ...trap/engines/engine.py.230: loss: 0.19706 [I 2022-10-26 13:29:16] ...optimizers/trijoint.py.151: total_norm: 2.801715 [I 2022-10-26 13:29:22] ...trap/engines/engine.py.224: train: epoch 10 | batch 1420/2304 [I 2022-10-26 13:29:22] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:07:27 [I 2022-10-26 13:29:22] ...trap/engines/engine.py.229: process: 0.51225 | load: 0.00250 [I 2022-10-26 13:29:22] ...trap/engines/engine.py.230: loss: 0.26583 [I 2022-10-26 13:29:22] ...optimizers/trijoint.py.151: total_norm: 3.590747 [I 2022-10-26 13:29:27] ...trap/engines/engine.py.224: train: epoch 10 | batch 1430/2304 [I 2022-10-26 13:29:27] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:07:21 [I 2022-10-26 13:29:27] ...trap/engines/engine.py.229: process: 0.52968 | load: 0.00248 [I 2022-10-26 13:29:27] ...trap/engines/engine.py.230: loss: 0.23119 [I 2022-10-26 13:29:27] ...optimizers/trijoint.py.151: total_norm: 3.656009 [I 2022-10-26 13:29:32] ...trap/engines/engine.py.224: train: epoch 10 | batch 1440/2304 [I 2022-10-26 13:29:32] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:07:16 [I 2022-10-26 13:29:32] ...trap/engines/engine.py.229: process: 0.50884 | load: 0.00253 [I 2022-10-26 13:29:32] ...trap/engines/engine.py.230: loss: 0.34115 [I 2022-10-26 13:29:32] ...optimizers/trijoint.py.151: total_norm: 3.668650 [I 2022-10-26 13:29:37] ...trap/engines/engine.py.224: train: epoch 10 | batch 1450/2304 [I 2022-10-26 13:29:37] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:07:09 [I 2022-10-26 13:29:37] ...trap/engines/engine.py.229: process: 0.49988 | load: 0.00249 [I 2022-10-26 13:29:37] ...trap/engines/engine.py.230: loss: 0.27308 [I 2022-10-26 13:29:37] ...optimizers/trijoint.py.151: total_norm: 4.020147 [I 2022-10-26 13:29:42] ...trap/engines/engine.py.224: train: epoch 10 | batch 1460/2304 [I 2022-10-26 13:29:42] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:07:06 [I 2022-10-26 13:29:42] ...trap/engines/engine.py.229: process: 0.50356 | load: 0.00248 [I 2022-10-26 13:29:42] ...trap/engines/engine.py.230: loss: 0.22366 [I 2022-10-26 13:29:42] ...optimizers/trijoint.py.151: total_norm: 3.127224 [I 2022-10-26 13:29:47] ...trap/engines/engine.py.224: train: epoch 10 | batch 1470/2304 [I 2022-10-26 13:29:47] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:06:58 [I 2022-10-26 13:29:47] ...trap/engines/engine.py.229: process: 0.49903 | load: 0.00249 [I 2022-10-26 13:29:47] ...trap/engines/engine.py.230: loss: 0.15031 [I 2022-10-26 13:29:47] ...optimizers/trijoint.py.151: total_norm: 2.719887 [I 2022-10-26 13:29:52] ...trap/engines/engine.py.224: train: epoch 10 | batch 1480/2304 [I 2022-10-26 13:29:52] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:06:54 [I 2022-10-26 13:29:52] ...trap/engines/engine.py.229: process: 0.52095 | load: 0.00250 [I 2022-10-26 13:29:52] ...trap/engines/engine.py.230: loss: 0.23885 [I 2022-10-26 13:29:52] ...optimizers/trijoint.py.151: total_norm: 2.826362 [I 2022-10-26 13:29:57] ...trap/engines/engine.py.224: train: epoch 10 | batch 1490/2304 [I 2022-10-26 13:29:57] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:06:46 [I 2022-10-26 13:29:57] ...trap/engines/engine.py.229: process: 0.50528 | load: 0.00247 [I 2022-10-26 13:29:57] ...trap/engines/engine.py.230: loss: 0.23649 [I 2022-10-26 13:29:57] ...optimizers/trijoint.py.151: total_norm: 3.306932 [I 2022-10-26 13:30:02] ...trap/engines/engine.py.224: train: epoch 10 | batch 1500/2304 [I 2022-10-26 13:30:02] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:06:39 [I 2022-10-26 13:30:02] ...trap/engines/engine.py.229: process: 0.49126 | load: 0.00250 [I 2022-10-26 13:30:02] ...trap/engines/engine.py.230: loss: 0.29522 [I 2022-10-26 13:30:02] ...optimizers/trijoint.py.151: total_norm: 3.373261 [I 2022-10-26 13:30:07] ...trap/engines/engine.py.224: train: epoch 10 | batch 1510/2304 [I 2022-10-26 13:30:07] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:06:35 [I 2022-10-26 13:30:07] ...trap/engines/engine.py.229: process: 0.50038 | load: 0.00244 [I 2022-10-26 13:30:07] ...trap/engines/engine.py.230: loss: 0.29853 [I 2022-10-26 13:30:07] ...optimizers/trijoint.py.151: total_norm: 4.532353 [I 2022-10-26 13:30:12] ...trap/engines/engine.py.224: train: epoch 10 | batch 1520/2304 [I 2022-10-26 13:30:12] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:06:31 [I 2022-10-26 13:30:12] ...trap/engines/engine.py.229: process: 0.49389 | load: 0.00246 [I 2022-10-26 13:30:12] ...trap/engines/engine.py.230: loss: 0.18290 [I 2022-10-26 13:30:12] ...optimizers/trijoint.py.151: total_norm: 4.164547 [I 2022-10-26 13:30:17] ...trap/engines/engine.py.224: train: epoch 10 | batch 1530/2304 [I 2022-10-26 13:30:17] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:06:27 [I 2022-10-26 13:30:17] ...trap/engines/engine.py.229: process: 0.50176 | load: 0.00244 [I 2022-10-26 13:30:17] ...trap/engines/engine.py.230: loss: 0.25533 [I 2022-10-26 13:30:17] ...optimizers/trijoint.py.151: total_norm: 3.218892 [I 2022-10-26 13:30:22] ...trap/engines/engine.py.224: train: epoch 10 | batch 1540/2304 [I 2022-10-26 13:30:22] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:06:23 [I 2022-10-26 13:30:22] ...trap/engines/engine.py.229: process: 0.50740 | load: 0.00248 [I 2022-10-26 13:30:22] ...trap/engines/engine.py.230: loss: 0.23526 [I 2022-10-26 13:30:22] ...optimizers/trijoint.py.151: total_norm: 3.885889 [I 2022-10-26 13:30:27] ...trap/engines/engine.py.224: train: epoch 10 | batch 1550/2304 [I 2022-10-26 13:30:27] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:06:19 [I 2022-10-26 13:30:27] ...trap/engines/engine.py.229: process: 0.50488 | load: 0.00249 [I 2022-10-26 13:30:27] ...trap/engines/engine.py.230: loss: 0.22572 [I 2022-10-26 13:30:27] ...optimizers/trijoint.py.151: total_norm: 2.671448 [I 2022-10-26 13:30:32] ...trap/engines/engine.py.224: train: epoch 10 | batch 1560/2304 [I 2022-10-26 13:30:32] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:06:14 [I 2022-10-26 13:30:32] ...trap/engines/engine.py.229: process: 0.50216 | load: 0.00268 [I 2022-10-26 13:30:32] ...trap/engines/engine.py.230: loss: 0.24107 [I 2022-10-26 13:30:32] ...optimizers/trijoint.py.151: total_norm: 2.975631 [I 2022-10-26 13:30:37] ...trap/engines/engine.py.224: train: epoch 10 | batch 1570/2304 [I 2022-10-26 13:30:37] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:06:05 [I 2022-10-26 13:30:37] ...trap/engines/engine.py.229: process: 0.49465 | load: 0.00255 [I 2022-10-26 13:30:37] ...trap/engines/engine.py.230: loss: 0.21483 [I 2022-10-26 13:30:37] ...optimizers/trijoint.py.151: total_norm: 2.467615 [I 2022-10-26 13:30:42] ...trap/engines/engine.py.224: train: epoch 10 | batch 1580/2304 [I 2022-10-26 13:30:42] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:05:59 [I 2022-10-26 13:30:42] ...trap/engines/engine.py.229: process: 0.48985 | load: 0.00248 [I 2022-10-26 13:30:42] ...trap/engines/engine.py.230: loss: 0.16459 [I 2022-10-26 13:30:42] ...optimizers/trijoint.py.151: total_norm: 3.086092 [I 2022-10-26 13:30:47] ...trap/engines/engine.py.224: train: epoch 10 | batch 1590/2304 [I 2022-10-26 13:30:47] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:05:55 [I 2022-10-26 13:30:47] ...trap/engines/engine.py.229: process: 0.50498 | load: 0.00245 [I 2022-10-26 13:30:47] ...trap/engines/engine.py.230: loss: 0.23737 [I 2022-10-26 13:30:47] ...optimizers/trijoint.py.151: total_norm: 3.836098 [I 2022-10-26 13:30:52] ...trap/engines/engine.py.224: train: epoch 10 | batch 1600/2304 [I 2022-10-26 13:30:52] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:05:53 [I 2022-10-26 13:30:52] ...trap/engines/engine.py.229: process: 0.50854 | load: 0.00263 [I 2022-10-26 13:30:52] ...trap/engines/engine.py.230: loss: 0.18692 [I 2022-10-26 13:30:52] ...optimizers/trijoint.py.151: total_norm: 3.486355 [I 2022-10-26 13:30:57] ...trap/engines/engine.py.224: train: epoch 10 | batch 1610/2304 [I 2022-10-26 13:30:57] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:05:50 [I 2022-10-26 13:30:57] ...trap/engines/engine.py.229: process: 0.52314 | load: 0.00248 [I 2022-10-26 13:30:57] ...trap/engines/engine.py.230: loss: 0.23420 [I 2022-10-26 13:30:57] ...optimizers/trijoint.py.151: total_norm: 2.787673 [I 2022-10-26 13:31:02] ...trap/engines/engine.py.224: train: epoch 10 | batch 1620/2304 [I 2022-10-26 13:31:02] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:05:43 [I 2022-10-26 13:31:02] ...trap/engines/engine.py.229: process: 0.50477 | load: 0.00249 [I 2022-10-26 13:31:02] ...trap/engines/engine.py.230: loss: 0.15881 [I 2022-10-26 13:31:02] ...optimizers/trijoint.py.151: total_norm: 2.862787 [I 2022-10-26 13:31:07] ...trap/engines/engine.py.224: train: epoch 10 | batch 1630/2304 [I 2022-10-26 13:31:07] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:05:37 [I 2022-10-26 13:31:07] ...trap/engines/engine.py.229: process: 0.50310 | load: 0.00247 [I 2022-10-26 13:31:07] ...trap/engines/engine.py.230: loss: 0.26024 [I 2022-10-26 13:31:07] ...optimizers/trijoint.py.151: total_norm: 3.092873 [I 2022-10-26 13:31:12] ...trap/engines/engine.py.224: train: epoch 10 | batch 1640/2304 [I 2022-10-26 13:31:12] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:05:31 [I 2022-10-26 13:31:12] ...trap/engines/engine.py.229: process: 0.49525 | load: 0.00253 [I 2022-10-26 13:31:12] ...trap/engines/engine.py.230: loss: 0.20364 [I 2022-10-26 13:31:12] ...optimizers/trijoint.py.151: total_norm: 3.651672 [I 2022-10-26 13:31:17] ...trap/engines/engine.py.224: train: epoch 10 | batch 1650/2304 [I 2022-10-26 13:31:17] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:05:28 [I 2022-10-26 13:31:17] ...trap/engines/engine.py.229: process: 0.51372 | load: 0.00254 [I 2022-10-26 13:31:17] ...trap/engines/engine.py.230: loss: 0.26975 [I 2022-10-26 13:31:17] ...optimizers/trijoint.py.151: total_norm: 3.406053 [I 2022-10-26 13:31:22] ...trap/engines/engine.py.224: train: epoch 10 | batch 1660/2304 [I 2022-10-26 13:31:22] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:05:21 [I 2022-10-26 13:31:22] ...trap/engines/engine.py.229: process: 0.49504 | load: 0.00249 [I 2022-10-26 13:31:22] ...trap/engines/engine.py.230: loss: 0.17671 [I 2022-10-26 13:31:22] ...optimizers/trijoint.py.151: total_norm: 2.837509 [I 2022-10-26 13:31:27] ...trap/engines/engine.py.224: train: epoch 10 | batch 1670/2304 [I 2022-10-26 13:31:27] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:05:17 [I 2022-10-26 13:31:27] ...trap/engines/engine.py.229: process: 0.50116 | load: 0.00248 [I 2022-10-26 13:31:27] ...trap/engines/engine.py.230: loss: 0.18357 [I 2022-10-26 13:31:27] ...optimizers/trijoint.py.151: total_norm: 2.823657 [I 2022-10-26 13:31:32] ...trap/engines/engine.py.224: train: epoch 10 | batch 1680/2304 [I 2022-10-26 13:31:32] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:05:11 [I 2022-10-26 13:31:32] ...trap/engines/engine.py.229: process: 0.50255 | load: 0.00245 [I 2022-10-26 13:31:32] ...trap/engines/engine.py.230: loss: 0.26069 [I 2022-10-26 13:31:32] ...optimizers/trijoint.py.151: total_norm: 3.875173 [I 2022-10-26 13:31:37] ...trap/engines/engine.py.224: train: epoch 10 | batch 1690/2304 [I 2022-10-26 13:31:37] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:05:15 [I 2022-10-26 13:31:37] ...trap/engines/engine.py.229: process: 0.50957 | load: 0.00244 [I 2022-10-26 13:31:37] ...trap/engines/engine.py.230: loss: 0.25809 [I 2022-10-26 13:31:37] ...optimizers/trijoint.py.151: total_norm: 3.891679 [I 2022-10-26 13:31:42] ...trap/engines/engine.py.224: train: epoch 10 | batch 1700/2304 [I 2022-10-26 13:31:42] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:05:12 [I 2022-10-26 13:31:42] ...trap/engines/engine.py.229: process: 0.51121 | load: 0.00247 [I 2022-10-26 13:31:42] ...trap/engines/engine.py.230: loss: 0.15521 [I 2022-10-26 13:31:42] ...optimizers/trijoint.py.151: total_norm: 2.138955 [I 2022-10-26 13:31:48] ...trap/engines/engine.py.224: train: epoch 10 | batch 1710/2304 [I 2022-10-26 13:31:48] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:05:03 [I 2022-10-26 13:31:48] ...trap/engines/engine.py.229: process: 0.50777 | load: 0.00246 [I 2022-10-26 13:31:48] ...trap/engines/engine.py.230: loss: 0.24951 [I 2022-10-26 13:31:48] ...optimizers/trijoint.py.151: total_norm: 3.436361 [I 2022-10-26 13:31:53] ...trap/engines/engine.py.224: train: epoch 10 | batch 1720/2304 [I 2022-10-26 13:31:53] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:05:00 [I 2022-10-26 13:31:53] ...trap/engines/engine.py.229: process: 0.51435 | load: 0.00259 [I 2022-10-26 13:31:53] ...trap/engines/engine.py.230: loss: 0.17918 [I 2022-10-26 13:31:53] ...optimizers/trijoint.py.151: total_norm: 3.036892 [I 2022-10-26 13:31:58] ...trap/engines/engine.py.224: train: epoch 10 | batch 1730/2304 [I 2022-10-26 13:31:58] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:04:54 [I 2022-10-26 13:31:58] ...trap/engines/engine.py.229: process: 0.51712 | load: 0.00252 [I 2022-10-26 13:31:58] ...trap/engines/engine.py.230: loss: 0.24744 [I 2022-10-26 13:31:58] ...optimizers/trijoint.py.151: total_norm: 3.529548 [I 2022-10-26 13:32:03] ...trap/engines/engine.py.224: train: epoch 10 | batch 1740/2304 [I 2022-10-26 13:32:03] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:04:47 [I 2022-10-26 13:32:03] ...trap/engines/engine.py.229: process: 0.50349 | load: 0.00245 [I 2022-10-26 13:32:03] ...trap/engines/engine.py.230: loss: 0.21113 [I 2022-10-26 13:32:03] ...optimizers/trijoint.py.151: total_norm: 3.660230 [I 2022-10-26 13:32:08] ...trap/engines/engine.py.224: train: epoch 10 | batch 1750/2304 [I 2022-10-26 13:32:08] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:04:36 [I 2022-10-26 13:32:08] ...trap/engines/engine.py.229: process: 0.50012 | load: 0.00247 [I 2022-10-26 13:32:08] ...trap/engines/engine.py.230: loss: 0.20068 [I 2022-10-26 13:32:08] ...optimizers/trijoint.py.151: total_norm: 3.416669 [I 2022-10-26 13:32:13] ...trap/engines/engine.py.224: train: epoch 10 | batch 1760/2304 [I 2022-10-26 13:32:13] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:04:30 [I 2022-10-26 13:32:13] ...trap/engines/engine.py.229: process: 0.48847 | load: 0.00243 [I 2022-10-26 13:32:13] ...trap/engines/engine.py.230: loss: 0.18383 [I 2022-10-26 13:32:13] ...optimizers/trijoint.py.151: total_norm: 3.079457 [I 2022-10-26 13:32:18] ...trap/engines/engine.py.224: train: epoch 10 | batch 1770/2304 [I 2022-10-26 13:32:18] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:04:29 [I 2022-10-26 13:32:18] ...trap/engines/engine.py.229: process: 0.52474 | load: 0.00243 [I 2022-10-26 13:32:18] ...trap/engines/engine.py.230: loss: 0.31657 [I 2022-10-26 13:32:18] ...optimizers/trijoint.py.151: total_norm: 4.452275 [I 2022-10-26 13:32:23] ...trap/engines/engine.py.224: train: epoch 10 | batch 1780/2304 [I 2022-10-26 13:32:23] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:04:19 [I 2022-10-26 13:32:23] ...trap/engines/engine.py.229: process: 0.49723 | load: 0.00255 [I 2022-10-26 13:32:23] ...trap/engines/engine.py.230: loss: 0.16964 [I 2022-10-26 13:32:23] ...optimizers/trijoint.py.151: total_norm: 2.495795 [I 2022-10-26 13:32:28] ...trap/engines/engine.py.224: train: epoch 10 | batch 1790/2304 [I 2022-10-26 13:32:28] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:04:15 [I 2022-10-26 13:32:28] ...trap/engines/engine.py.229: process: 0.48955 | load: 0.00265 [I 2022-10-26 13:32:28] ...trap/engines/engine.py.230: loss: 0.19369 [I 2022-10-26 13:32:28] ...optimizers/trijoint.py.151: total_norm: 3.689482 [I 2022-10-26 13:32:33] ...trap/engines/engine.py.224: train: epoch 10 | batch 1800/2304 [I 2022-10-26 13:32:33] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:04:11 [I 2022-10-26 13:32:33] ...trap/engines/engine.py.229: process: 0.50338 | load: 0.00241 [I 2022-10-26 13:32:33] ...trap/engines/engine.py.230: loss: 0.26522 [I 2022-10-26 13:32:33] ...optimizers/trijoint.py.151: total_norm: 3.071136 [I 2022-10-26 13:32:38] ...trap/engines/engine.py.224: train: epoch 10 | batch 1810/2304 [I 2022-10-26 13:32:38] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:04:06 [I 2022-10-26 13:32:38] ...trap/engines/engine.py.229: process: 0.50006 | load: 0.00253 [I 2022-10-26 13:32:38] ...trap/engines/engine.py.230: loss: 0.24981 [I 2022-10-26 13:32:38] ...optimizers/trijoint.py.151: total_norm: 3.719252 [I 2022-10-26 13:32:43] ...trap/engines/engine.py.224: train: epoch 10 | batch 1820/2304 [I 2022-10-26 13:32:43] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:04:00 [I 2022-10-26 13:32:43] ...trap/engines/engine.py.229: process: 0.50222 | load: 0.00250 [I 2022-10-26 13:32:43] ...trap/engines/engine.py.230: loss: 0.21831 [I 2022-10-26 13:32:43] ...optimizers/trijoint.py.151: total_norm: 3.150752 [I 2022-10-26 13:32:48] ...trap/engines/engine.py.224: train: epoch 10 | batch 1830/2304 [I 2022-10-26 13:32:48] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:03:56 [I 2022-10-26 13:32:48] ...trap/engines/engine.py.229: process: 0.50738 | load: 0.00249 [I 2022-10-26 13:32:48] ...trap/engines/engine.py.230: loss: 0.22218 [I 2022-10-26 13:32:48] ...optimizers/trijoint.py.151: total_norm: 3.606492 [I 2022-10-26 13:32:53] ...trap/engines/engine.py.224: train: epoch 10 | batch 1840/2304 [I 2022-10-26 13:32:53] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:03:51 [I 2022-10-26 13:32:53] ...trap/engines/engine.py.229: process: 0.49641 | load: 0.00246 [I 2022-10-26 13:32:53] ...trap/engines/engine.py.230: loss: 0.26981 [I 2022-10-26 13:32:53] ...optimizers/trijoint.py.151: total_norm: 3.080134 [I 2022-10-26 13:32:58] ...trap/engines/engine.py.224: train: epoch 10 | batch 1850/2304 [I 2022-10-26 13:32:58] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:03:50 [I 2022-10-26 13:32:58] ...trap/engines/engine.py.229: process: 0.52099 | load: 0.00245 [I 2022-10-26 13:32:58] ...trap/engines/engine.py.230: loss: 0.13284 [I 2022-10-26 13:32:58] ...optimizers/trijoint.py.151: total_norm: 2.632739 [I 2022-10-26 13:33:03] ...trap/engines/engine.py.224: train: epoch 10 | batch 1860/2304 [I 2022-10-26 13:33:03] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:03:43 [I 2022-10-26 13:33:03] ...trap/engines/engine.py.229: process: 0.49461 | load: 0.00246 [I 2022-10-26 13:33:03] ...trap/engines/engine.py.230: loss: 0.20692 [I 2022-10-26 13:33:03] ...optimizers/trijoint.py.151: total_norm: 3.474828 [I 2022-10-26 13:33:08] ...trap/engines/engine.py.224: train: epoch 10 | batch 1870/2304 [I 2022-10-26 13:33:08] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:03:37 [I 2022-10-26 13:33:08] ...trap/engines/engine.py.229: process: 0.50998 | load: 0.00256 [I 2022-10-26 13:33:08] ...trap/engines/engine.py.230: loss: 0.31498 [I 2022-10-26 13:33:08] ...optimizers/trijoint.py.151: total_norm: 2.735627 [I 2022-10-26 13:33:13] ...trap/engines/engine.py.224: train: epoch 10 | batch 1880/2304 [I 2022-10-26 13:33:13] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:03:34 [I 2022-10-26 13:33:13] ...trap/engines/engine.py.229: process: 0.51747 | load: 0.00247 [I 2022-10-26 13:33:13] ...trap/engines/engine.py.230: loss: 0.14885 [I 2022-10-26 13:33:13] ...optimizers/trijoint.py.151: total_norm: 2.298589 [I 2022-10-26 13:33:18] ...trap/engines/engine.py.224: train: epoch 10 | batch 1890/2304 [I 2022-10-26 13:33:18] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:03:30 [I 2022-10-26 13:33:18] ...trap/engines/engine.py.229: process: 0.50795 | load: 0.00246 [I 2022-10-26 13:33:18] ...trap/engines/engine.py.230: loss: 0.12141 [I 2022-10-26 13:33:18] ...optimizers/trijoint.py.151: total_norm: 2.034520 [I 2022-10-26 13:33:23] ...trap/engines/engine.py.224: train: epoch 10 | batch 1900/2304 [I 2022-10-26 13:33:23] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:03:21 [I 2022-10-26 13:33:23] ...trap/engines/engine.py.229: process: 0.49676 | load: 0.00245 [I 2022-10-26 13:33:23] ...trap/engines/engine.py.230: loss: 0.23495 [I 2022-10-26 13:33:23] ...optimizers/trijoint.py.151: total_norm: 2.964013 [I 2022-10-26 13:33:28] ...trap/engines/engine.py.224: train: epoch 10 | batch 1910/2304 [I 2022-10-26 13:33:28] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:03:17 [I 2022-10-26 13:33:28] ...trap/engines/engine.py.229: process: 0.50153 | load: 0.00245 [I 2022-10-26 13:33:28] ...trap/engines/engine.py.230: loss: 0.21235 [I 2022-10-26 13:33:28] ...optimizers/trijoint.py.151: total_norm: 3.448669 [I 2022-10-26 13:33:33] ...trap/engines/engine.py.224: train: epoch 10 | batch 1920/2304 [I 2022-10-26 13:33:33] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:03:12 [I 2022-10-26 13:33:33] ...trap/engines/engine.py.229: process: 0.49320 | load: 0.00246 [I 2022-10-26 13:33:33] ...trap/engines/engine.py.230: loss: 0.19824 [I 2022-10-26 13:33:33] ...optimizers/trijoint.py.151: total_norm: 3.370875 [I 2022-10-26 13:33:38] ...trap/engines/engine.py.224: train: epoch 10 | batch 1930/2304 [I 2022-10-26 13:33:38] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:03:07 [I 2022-10-26 13:33:38] ...trap/engines/engine.py.229: process: 0.50069 | load: 0.00247 [I 2022-10-26 13:33:38] ...trap/engines/engine.py.230: loss: 0.21788 [I 2022-10-26 13:33:38] ...optimizers/trijoint.py.151: total_norm: 2.820836 [I 2022-10-26 13:33:43] ...trap/engines/engine.py.224: train: epoch 10 | batch 1940/2304 [I 2022-10-26 13:33:43] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:03:02 [I 2022-10-26 13:33:43] ...trap/engines/engine.py.229: process: 0.50637 | load: 0.00256 [I 2022-10-26 13:33:43] ...trap/engines/engine.py.230: loss: 0.26754 [I 2022-10-26 13:33:43] ...optimizers/trijoint.py.151: total_norm: 3.766619 [I 2022-10-26 13:33:48] ...trap/engines/engine.py.224: train: epoch 10 | batch 1950/2304 [I 2022-10-26 13:33:48] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:02:55 [I 2022-10-26 13:33:48] ...trap/engines/engine.py.229: process: 0.49547 | load: 0.00250 [I 2022-10-26 13:33:48] ...trap/engines/engine.py.230: loss: 0.21546 [I 2022-10-26 13:33:48] ...optimizers/trijoint.py.151: total_norm: 3.782899 [I 2022-10-26 13:33:53] ...trap/engines/engine.py.224: train: epoch 10 | batch 1960/2304 [I 2022-10-26 13:33:53] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:02:51 [I 2022-10-26 13:33:53] ...trap/engines/engine.py.229: process: 0.50217 | load: 0.00247 [I 2022-10-26 13:33:53] ...trap/engines/engine.py.230: loss: 0.25911 [I 2022-10-26 13:33:53] ...optimizers/trijoint.py.151: total_norm: 3.707277 [I 2022-10-26 13:33:58] ...trap/engines/engine.py.224: train: epoch 10 | batch 1970/2304 [I 2022-10-26 13:33:58] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:02:45 [I 2022-10-26 13:33:58] ...trap/engines/engine.py.229: process: 0.50125 | load: 0.00247 [I 2022-10-26 13:33:58] ...trap/engines/engine.py.230: loss: 0.28519 [I 2022-10-26 13:33:58] ...optimizers/trijoint.py.151: total_norm: 4.079539 [I 2022-10-26 13:34:03] ...trap/engines/engine.py.224: train: epoch 10 | batch 1980/2304 [I 2022-10-26 13:34:03] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:02:41 [I 2022-10-26 13:34:03] ...trap/engines/engine.py.229: process: 0.49319 | load: 0.00248 [I 2022-10-26 13:34:03] ...trap/engines/engine.py.230: loss: 0.21573 [I 2022-10-26 13:34:03] ...optimizers/trijoint.py.151: total_norm: 3.772467 [I 2022-10-26 13:34:08] ...trap/engines/engine.py.224: train: epoch 10 | batch 1990/2304 [I 2022-10-26 13:34:08] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:02:36 [I 2022-10-26 13:34:08] ...trap/engines/engine.py.229: process: 0.49560 | load: 0.00244 [I 2022-10-26 13:34:08] ...trap/engines/engine.py.230: loss: 0.21696 [I 2022-10-26 13:34:08] ...optimizers/trijoint.py.151: total_norm: 3.100408 [I 2022-10-26 13:34:13] ...trap/engines/engine.py.224: train: epoch 10 | batch 2000/2304 [I 2022-10-26 13:34:13] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:02:34 [I 2022-10-26 13:34:13] ...trap/engines/engine.py.229: process: 0.52571 | load: 0.00244 [I 2022-10-26 13:34:13] ...trap/engines/engine.py.230: loss: 0.19462 [I 2022-10-26 13:34:13] ...optimizers/trijoint.py.151: total_norm: 4.013756 [I 2022-10-26 13:34:18] ...trap/engines/engine.py.224: train: epoch 10 | batch 2010/2304 [I 2022-10-26 13:34:18] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:02:27 [I 2022-10-26 13:34:18] ...trap/engines/engine.py.229: process: 0.50047 | load: 0.00252 [I 2022-10-26 13:34:18] ...trap/engines/engine.py.230: loss: 0.31345 [I 2022-10-26 13:34:18] ...optimizers/trijoint.py.151: total_norm: 3.407349 [I 2022-10-26 13:34:23] ...trap/engines/engine.py.224: train: epoch 10 | batch 2020/2304 [I 2022-10-26 13:34:23] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:02:21 [I 2022-10-26 13:34:23] ...trap/engines/engine.py.229: process: 0.49931 | load: 0.00243 [I 2022-10-26 13:34:23] ...trap/engines/engine.py.230: loss: 0.18467 [I 2022-10-26 13:34:23] ...optimizers/trijoint.py.151: total_norm: 2.555614 [I 2022-10-26 13:34:28] ...trap/engines/engine.py.224: train: epoch 10 | batch 2030/2304 [I 2022-10-26 13:34:28] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:02:17 [I 2022-10-26 13:34:28] ...trap/engines/engine.py.229: process: 0.49999 | load: 0.00251 [I 2022-10-26 13:34:28] ...trap/engines/engine.py.230: loss: 0.27054 [I 2022-10-26 13:34:28] ...optimizers/trijoint.py.151: total_norm: 4.366904 [I 2022-10-26 13:34:33] ...trap/engines/engine.py.224: train: epoch 10 | batch 2040/2304 [I 2022-10-26 13:34:33] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:02:11 [I 2022-10-26 13:34:33] ...trap/engines/engine.py.229: process: 0.49605 | load: 0.00248 [I 2022-10-26 13:34:33] ...trap/engines/engine.py.230: loss: 0.16723 [I 2022-10-26 13:34:33] ...optimizers/trijoint.py.151: total_norm: 2.663635 [I 2022-10-26 13:34:38] ...trap/engines/engine.py.224: train: epoch 10 | batch 2050/2304 [I 2022-10-26 13:34:38] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:02:06 [I 2022-10-26 13:34:38] ...trap/engines/engine.py.229: process: 0.50090 | load: 0.00251 [I 2022-10-26 13:34:38] ...trap/engines/engine.py.230: loss: 0.25488 [I 2022-10-26 13:34:38] ...optimizers/trijoint.py.151: total_norm: 3.672606 [I 2022-10-26 13:34:43] ...trap/engines/engine.py.224: train: epoch 10 | batch 2060/2304 [I 2022-10-26 13:34:43] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:02:01 [I 2022-10-26 13:34:43] ...trap/engines/engine.py.229: process: 0.50236 | load: 0.00249 [I 2022-10-26 13:34:43] ...trap/engines/engine.py.230: loss: 0.23048 [I 2022-10-26 13:34:43] ...optimizers/trijoint.py.151: total_norm: 3.163291 [I 2022-10-26 13:34:48] ...trap/engines/engine.py.224: train: epoch 10 | batch 2070/2304 [I 2022-10-26 13:34:48] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:01:57 [I 2022-10-26 13:34:48] ...trap/engines/engine.py.229: process: 0.50289 | load: 0.00244 [I 2022-10-26 13:34:48] ...trap/engines/engine.py.230: loss: 0.21705 [I 2022-10-26 13:34:48] ...optimizers/trijoint.py.151: total_norm: 3.043667 [I 2022-10-26 13:34:53] ...trap/engines/engine.py.224: train: epoch 10 | batch 2080/2304 [I 2022-10-26 13:34:53] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:01:52 [I 2022-10-26 13:34:53] ...trap/engines/engine.py.229: process: 0.51046 | load: 0.00248 [I 2022-10-26 13:34:53] ...trap/engines/engine.py.230: loss: 0.19794 [I 2022-10-26 13:34:53] ...optimizers/trijoint.py.151: total_norm: 3.244548 [I 2022-10-26 13:34:58] ...trap/engines/engine.py.224: train: epoch 10 | batch 2090/2304 [I 2022-10-26 13:34:58] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:01:47 [I 2022-10-26 13:34:58] ...trap/engines/engine.py.229: process: 0.50167 | load: 0.00252 [I 2022-10-26 13:34:58] ...trap/engines/engine.py.230: loss: 0.16308 [I 2022-10-26 13:34:58] ...optimizers/trijoint.py.151: total_norm: 2.771045 [I 2022-10-26 13:35:03] ...trap/engines/engine.py.224: train: epoch 10 | batch 2100/2304 [I 2022-10-26 13:35:03] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:01:42 [I 2022-10-26 13:35:03] ...trap/engines/engine.py.229: process: 0.49709 | load: 0.00243 [I 2022-10-26 13:35:03] ...trap/engines/engine.py.230: loss: 0.30601 [I 2022-10-26 13:35:03] ...optimizers/trijoint.py.151: total_norm: 3.884137 [I 2022-10-26 13:35:08] ...trap/engines/engine.py.224: train: epoch 10 | batch 2110/2304 [I 2022-10-26 13:35:08] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:01:39 [I 2022-10-26 13:35:08] ...trap/engines/engine.py.229: process: 0.49452 | load: 0.00253 [I 2022-10-26 13:35:08] ...trap/engines/engine.py.230: loss: 0.31095 [I 2022-10-26 13:35:08] ...optimizers/trijoint.py.151: total_norm: 3.489996 [I 2022-10-26 13:35:13] ...trap/engines/engine.py.224: train: epoch 10 | batch 2120/2304 [I 2022-10-26 13:35:13] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:01:31 [I 2022-10-26 13:35:13] ...trap/engines/engine.py.229: process: 0.49127 | load: 0.00266 [I 2022-10-26 13:35:13] ...trap/engines/engine.py.230: loss: 0.25720 [I 2022-10-26 13:35:13] ...optimizers/trijoint.py.151: total_norm: 2.987475 [I 2022-10-26 13:35:18] ...trap/engines/engine.py.224: train: epoch 10 | batch 2130/2304 [I 2022-10-26 13:35:18] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:01:26 [I 2022-10-26 13:35:18] ...trap/engines/engine.py.229: process: 0.50180 | load: 0.00253 [I 2022-10-26 13:35:18] ...trap/engines/engine.py.230: loss: 0.28204 [I 2022-10-26 13:35:18] ...optimizers/trijoint.py.151: total_norm: 3.473290 [I 2022-10-26 13:35:23] ...trap/engines/engine.py.224: train: epoch 10 | batch 2140/2304 [I 2022-10-26 13:35:23] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:01:22 [I 2022-10-26 13:35:23] ...trap/engines/engine.py.229: process: 0.50772 | load: 0.00248 [I 2022-10-26 13:35:23] ...trap/engines/engine.py.230: loss: 0.19408 [I 2022-10-26 13:35:23] ...optimizers/trijoint.py.151: total_norm: 2.666245 [I 2022-10-26 13:35:28] ...trap/engines/engine.py.224: train: epoch 10 | batch 2150/2304 [I 2022-10-26 13:35:28] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:01:17 [I 2022-10-26 13:35:28] ...trap/engines/engine.py.229: process: 0.51368 | load: 0.00246 [I 2022-10-26 13:35:28] ...trap/engines/engine.py.230: loss: 0.25704 [I 2022-10-26 13:35:28] ...optimizers/trijoint.py.151: total_norm: 3.130848 [I 2022-10-26 13:35:33] ...trap/engines/engine.py.224: train: epoch 10 | batch 2160/2304 [I 2022-10-26 13:35:33] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:01:11 [I 2022-10-26 13:35:33] ...trap/engines/engine.py.229: process: 0.49213 | load: 0.00249 [I 2022-10-26 13:35:33] ...trap/engines/engine.py.230: loss: 0.21890 [I 2022-10-26 13:35:33] ...optimizers/trijoint.py.151: total_norm: 2.718544 [I 2022-10-26 13:35:38] ...trap/engines/engine.py.224: train: epoch 10 | batch 2170/2304 [I 2022-10-26 13:35:38] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:01:06 [I 2022-10-26 13:35:38] ...trap/engines/engine.py.229: process: 0.48816 | load: 0.00244 [I 2022-10-26 13:35:38] ...trap/engines/engine.py.230: loss: 0.28497 [I 2022-10-26 13:35:38] ...optimizers/trijoint.py.151: total_norm: 2.930979 [I 2022-10-26 13:35:43] ...trap/engines/engine.py.224: train: epoch 10 | batch 2180/2304 [I 2022-10-26 13:35:43] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:01:01 [I 2022-10-26 13:35:43] ...trap/engines/engine.py.229: process: 0.50167 | load: 0.00248 [I 2022-10-26 13:35:43] ...trap/engines/engine.py.230: loss: 0.25259 [I 2022-10-26 13:35:43] ...optimizers/trijoint.py.151: total_norm: 3.777525 [I 2022-10-26 13:35:48] ...trap/engines/engine.py.224: train: epoch 10 | batch 2190/2304 [I 2022-10-26 13:35:48] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:00:57 [I 2022-10-26 13:35:48] ...trap/engines/engine.py.229: process: 0.50711 | load: 0.00247 [I 2022-10-26 13:35:48] ...trap/engines/engine.py.230: loss: 0.23104 [I 2022-10-26 13:35:48] ...optimizers/trijoint.py.151: total_norm: 2.727660 [I 2022-10-26 13:35:53] ...trap/engines/engine.py.224: train: epoch 10 | batch 2200/2304 [I 2022-10-26 13:35:53] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:00:52 [I 2022-10-26 13:35:53] ...trap/engines/engine.py.229: process: 0.49417 | load: 0.00245 [I 2022-10-26 13:35:53] ...trap/engines/engine.py.230: loss: 0.21377 [I 2022-10-26 13:35:53] ...optimizers/trijoint.py.151: total_norm: 3.340707 [I 2022-10-26 13:35:58] ...trap/engines/engine.py.224: train: epoch 10 | batch 2210/2304 [I 2022-10-26 13:35:58] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:00:46 [I 2022-10-26 13:35:58] ...trap/engines/engine.py.229: process: 0.49844 | load: 0.00245 [I 2022-10-26 13:35:58] ...trap/engines/engine.py.230: loss: 0.20919 [I 2022-10-26 13:35:58] ...optimizers/trijoint.py.151: total_norm: 4.500193 [I 2022-10-26 13:36:03] ...trap/engines/engine.py.224: train: epoch 10 | batch 2220/2304 [I 2022-10-26 13:36:03] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:00:41 [I 2022-10-26 13:36:03] ...trap/engines/engine.py.229: process: 0.48941 | load: 0.00246 [I 2022-10-26 13:36:03] ...trap/engines/engine.py.230: loss: 0.19142 [I 2022-10-26 13:36:03] ...optimizers/trijoint.py.151: total_norm: 2.851250 [I 2022-10-26 13:36:08] ...trap/engines/engine.py.224: train: epoch 10 | batch 2230/2304 [I 2022-10-26 13:36:08] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:00:37 [I 2022-10-26 13:36:08] ...trap/engines/engine.py.229: process: 0.51240 | load: 0.00253 [I 2022-10-26 13:36:08] ...trap/engines/engine.py.230: loss: 0.20735 [I 2022-10-26 13:36:08] ...optimizers/trijoint.py.151: total_norm: 3.157096 [I 2022-10-26 13:36:13] ...trap/engines/engine.py.224: train: epoch 10 | batch 2240/2304 [I 2022-10-26 13:36:13] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:00:32 [I 2022-10-26 13:36:13] ...trap/engines/engine.py.229: process: 0.49219 | load: 0.00264 [I 2022-10-26 13:36:13] ...trap/engines/engine.py.230: loss: 0.20554 [I 2022-10-26 13:36:13] ...optimizers/trijoint.py.151: total_norm: 2.987968 [I 2022-10-26 13:36:18] ...trap/engines/engine.py.224: train: epoch 10 | batch 2250/2304 [I 2022-10-26 13:36:18] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:00:26 [I 2022-10-26 13:36:18] ...trap/engines/engine.py.229: process: 0.49324 | load: 0.00254 [I 2022-10-26 13:36:18] ...trap/engines/engine.py.230: loss: 0.20908 [I 2022-10-26 13:36:18] ...optimizers/trijoint.py.151: total_norm: 2.777724 [I 2022-10-26 13:36:23] ...trap/engines/engine.py.224: train: epoch 10 | batch 2260/2304 [I 2022-10-26 13:36:23] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:00:21 [I 2022-10-26 13:36:23] ...trap/engines/engine.py.229: process: 0.49094 | load: 0.00247 [I 2022-10-26 13:36:23] ...trap/engines/engine.py.230: loss: 0.19919 [I 2022-10-26 13:36:23] ...optimizers/trijoint.py.151: total_norm: 3.660995 [I 2022-10-26 13:36:28] ...trap/engines/engine.py.224: train: epoch 10 | batch 2270/2304 [I 2022-10-26 13:36:28] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:00:16 [I 2022-10-26 13:36:28] ...trap/engines/engine.py.229: process: 0.48968 | load: 0.00247 [I 2022-10-26 13:36:28] ...trap/engines/engine.py.230: loss: 0.25133 [I 2022-10-26 13:36:28] ...optimizers/trijoint.py.151: total_norm: 3.231987 [I 2022-10-26 13:36:33] ...trap/engines/engine.py.224: train: epoch 10 | batch 2280/2304 [I 2022-10-26 13:36:33] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:00:11 [I 2022-10-26 13:36:33] ...trap/engines/engine.py.229: process: 0.49472 | load: 0.00247 [I 2022-10-26 13:36:33] ...trap/engines/engine.py.230: loss: 0.21798 [I 2022-10-26 13:36:33] ...optimizers/trijoint.py.151: total_norm: 2.923383 [I 2022-10-26 13:36:38] ...trap/engines/engine.py.224: train: epoch 10 | batch 2290/2304 [I 2022-10-26 13:36:38] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:00:06 [I 2022-10-26 13:36:38] ...trap/engines/engine.py.229: process: 0.50394 | load: 0.00245 [I 2022-10-26 13:36:38] ...trap/engines/engine.py.230: loss: 0.25989 [I 2022-10-26 13:36:38] ...optimizers/trijoint.py.151: total_norm: 3.708821 [I 2022-10-26 13:36:43] ...trap/engines/engine.py.224: train: epoch 10 | batch 2300/2304 [I 2022-10-26 13:36:43] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:00:02 [I 2022-10-26 13:36:43] ...trap/engines/engine.py.229: process: 0.49575 | load: 0.00014 [I 2022-10-26 13:36:43] ...trap/engines/engine.py.230: loss: 0.30123 [I 2022-10-26 13:36:43] ...optimizers/trijoint.py.151: total_norm: 4.123731 [I 2022-10-26 13:36:45] ...trap/engines/engine.py.224: train: epoch 10 | batch 2304/2304 [I 2022-10-26 13:36:45] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:00:00 [I 2022-10-26 13:36:45] ...trap/engines/engine.py.229: process: 0.49811 | load: 0.00013 [I 2022-10-26 13:36:45] ...trap/engines/engine.py.230: loss: 0.25044 [I 2022-10-26 13:36:45] ...optimizers/trijoint.py.151: total_norm: 3.603345 [S 2022-10-26 13:36:47] ...trap/engines/engine.py.129: train_epoch.epoch: 10 [S 2022-10-26 13:36:47] ...trap/engines/engine.py.129: train_epoch.loss: 0.23580997115296035 [I 2022-10-26 13:36:47] ...s/criterions/triplet.py.84: margin = 0.10500000000000004 [I 2022-10-26 13:36:47] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 10 [I 2022-10-26 13:36:47] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 13:36:49] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 13:36:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 13:36:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 13:36:49] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 13:36:50] ...trap/engines/engine.py.321: eval: epoch 10 | batch 0/510 [I 2022-10-26 13:36:50] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:37 [I 2022-10-26 13:36:50] ...trap/engines/engine.py.326: process: 2.66235 | load: 2.29055 [I 2022-10-26 13:36:53] ...trap/engines/engine.py.321: eval: epoch 10 | batch 10/510 [I 2022-10-26 13:36:53] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:53 [I 2022-10-26 13:36:53] ...trap/engines/engine.py.326: process: 0.34741 | load: 0.00017 [I 2022-10-26 13:36:56] ...trap/engines/engine.py.321: eval: epoch 10 | batch 20/510 [I 2022-10-26 13:36:56] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:57 [I 2022-10-26 13:36:56] ...trap/engines/engine.py.326: process: 0.32025 | load: 0.00018 [I 2022-10-26 13:37:00] ...trap/engines/engine.py.321: eval: epoch 10 | batch 30/510 [I 2022-10-26 13:37:00] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:47 [I 2022-10-26 13:37:00] ...trap/engines/engine.py.326: process: 0.33689 | load: 0.00019 [I 2022-10-26 13:37:03] ...trap/engines/engine.py.321: eval: epoch 10 | batch 40/510 [I 2022-10-26 13:37:03] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:42 [I 2022-10-26 13:37:03] ...trap/engines/engine.py.326: process: 0.32825 | load: 0.00024 [I 2022-10-26 13:37:07] ...trap/engines/engine.py.321: eval: epoch 10 | batch 50/510 [I 2022-10-26 13:37:07] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-26 13:37:07] ...trap/engines/engine.py.326: process: 0.31981 | load: 0.00024 [I 2022-10-26 13:37:10] ...trap/engines/engine.py.321: eval: epoch 10 | batch 60/510 [I 2022-10-26 13:37:10] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:30 [I 2022-10-26 13:37:10] ...trap/engines/engine.py.326: process: 0.33183 | load: 0.00022 [I 2022-10-26 13:37:13] ...trap/engines/engine.py.321: eval: epoch 10 | batch 70/510 [I 2022-10-26 13:37:13] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:29 [I 2022-10-26 13:37:13] ...trap/engines/engine.py.326: process: 0.33296 | load: 0.00023 [I 2022-10-26 13:37:17] ...trap/engines/engine.py.321: eval: epoch 10 | batch 80/510 [I 2022-10-26 13:37:17] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:29 [I 2022-10-26 13:37:17] ...trap/engines/engine.py.326: process: 0.37679 | load: 0.00019 [I 2022-10-26 13:37:20] ...trap/engines/engine.py.321: eval: epoch 10 | batch 90/510 [I 2022-10-26 13:37:20] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:23 [I 2022-10-26 13:37:20] ...trap/engines/engine.py.326: process: 0.34659 | load: 0.00019 [I 2022-10-26 13:37:24] ...trap/engines/engine.py.321: eval: epoch 10 | batch 100/510 [I 2022-10-26 13:37:24] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:14 [I 2022-10-26 13:37:24] ...trap/engines/engine.py.326: process: 0.29470 | load: 0.00023 [I 2022-10-26 13:37:26] ...trap/engines/engine.py.321: eval: epoch 10 | batch 110/510 [I 2022-10-26 13:37:26] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-26 13:37:26] ...trap/engines/engine.py.326: process: 0.28992 | load: 0.00014 [I 2022-10-26 13:37:29] ...trap/engines/engine.py.321: eval: epoch 10 | batch 120/510 [I 2022-10-26 13:37:29] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-26 13:37:29] ...trap/engines/engine.py.326: process: 0.28999 | load: 0.00016 [I 2022-10-26 13:37:32] ...trap/engines/engine.py.321: eval: epoch 10 | batch 130/510 [I 2022-10-26 13:37:32] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 13:37:32] ...trap/engines/engine.py.326: process: 0.28469 | load: 0.00022 [I 2022-10-26 13:37:35] ...trap/engines/engine.py.321: eval: epoch 10 | batch 140/510 [I 2022-10-26 13:37:35] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:49 [I 2022-10-26 13:37:35] ...trap/engines/engine.py.326: process: 0.30937 | load: 0.00019 [I 2022-10-26 13:37:38] ...trap/engines/engine.py.321: eval: epoch 10 | batch 150/510 [I 2022-10-26 13:37:38] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:51 [I 2022-10-26 13:37:38] ...trap/engines/engine.py.326: process: 0.34201 | load: 0.00014 [I 2022-10-26 13:37:41] ...trap/engines/engine.py.321: eval: epoch 10 | batch 160/510 [I 2022-10-26 13:37:41] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:43 [I 2022-10-26 13:37:41] ...trap/engines/engine.py.326: process: 0.29145 | load: 0.00015 [I 2022-10-26 13:37:44] ...trap/engines/engine.py.321: eval: epoch 10 | batch 170/510 [I 2022-10-26 13:37:44] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-26 13:37:44] ...trap/engines/engine.py.326: process: 0.28505 | load: 0.00015 [I 2022-10-26 13:37:47] ...trap/engines/engine.py.321: eval: epoch 10 | batch 180/510 [I 2022-10-26 13:37:47] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:34 [I 2022-10-26 13:37:47] ...trap/engines/engine.py.326: process: 0.28974 | load: 0.00016 [I 2022-10-26 13:37:50] ...trap/engines/engine.py.321: eval: epoch 10 | batch 190/510 [I 2022-10-26 13:37:50] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:31 [I 2022-10-26 13:37:50] ...trap/engines/engine.py.326: process: 0.28838 | load: 0.00016 [I 2022-10-26 13:37:53] ...trap/engines/engine.py.321: eval: epoch 10 | batch 200/510 [I 2022-10-26 13:37:53] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 13:37:53] ...trap/engines/engine.py.326: process: 0.29007 | load: 0.00018 [I 2022-10-26 13:37:56] ...trap/engines/engine.py.321: eval: epoch 10 | batch 210/510 [I 2022-10-26 13:37:56] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 13:37:56] ...trap/engines/engine.py.326: process: 0.28207 | load: 0.00017 [I 2022-10-26 13:37:58] ...trap/engines/engine.py.321: eval: epoch 10 | batch 220/510 [I 2022-10-26 13:37:58] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 13:37:58] ...trap/engines/engine.py.326: process: 0.28665 | load: 0.00022 [I 2022-10-26 13:38:01] ...trap/engines/engine.py.321: eval: epoch 10 | batch 230/510 [I 2022-10-26 13:38:01] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 13:38:01] ...trap/engines/engine.py.326: process: 0.28852 | load: 0.00016 [I 2022-10-26 13:38:04] ...trap/engines/engine.py.321: eval: epoch 10 | batch 240/510 [I 2022-10-26 13:38:04] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 13:38:04] ...trap/engines/engine.py.326: process: 0.29304 | load: 0.00016 [I 2022-10-26 13:38:07] ...trap/engines/engine.py.321: eval: epoch 10 | batch 250/510 [I 2022-10-26 13:38:07] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-26 13:38:07] ...trap/engines/engine.py.326: process: 0.28640 | load: 0.00035 [I 2022-10-26 13:38:10] ...trap/engines/engine.py.321: eval: epoch 10 | batch 260/510 [I 2022-10-26 13:38:10] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-26 13:38:10] ...trap/engines/engine.py.326: process: 0.29083 | load: 0.00020 [I 2022-10-26 13:38:13] ...trap/engines/engine.py.321: eval: epoch 10 | batch 270/510 [I 2022-10-26 13:38:13] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-26 13:38:13] ...trap/engines/engine.py.326: process: 0.28639 | load: 0.00016 [I 2022-10-26 13:38:16] ...trap/engines/engine.py.321: eval: epoch 10 | batch 280/510 [I 2022-10-26 13:38:16] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:08 [I 2022-10-26 13:38:16] ...trap/engines/engine.py.326: process: 0.30589 | load: 0.00016 [I 2022-10-26 13:38:19] ...trap/engines/engine.py.321: eval: epoch 10 | batch 290/510 [I 2022-10-26 13:38:19] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 13:38:19] ...trap/engines/engine.py.326: process: 0.28609 | load: 0.00016 [I 2022-10-26 13:38:22] ...trap/engines/engine.py.321: eval: epoch 10 | batch 300/510 [I 2022-10-26 13:38:22] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:01 [I 2022-10-26 13:38:22] ...trap/engines/engine.py.326: process: 0.28847 | load: 0.00020 [I 2022-10-26 13:38:25] ...trap/engines/engine.py.321: eval: epoch 10 | batch 310/510 [I 2022-10-26 13:38:25] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 13:38:25] ...trap/engines/engine.py.326: process: 0.29066 | load: 0.00014 [I 2022-10-26 13:38:27] ...trap/engines/engine.py.321: eval: epoch 10 | batch 320/510 [I 2022-10-26 13:38:27] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 13:38:27] ...trap/engines/engine.py.326: process: 0.28778 | load: 0.00016 [I 2022-10-26 13:38:30] ...trap/engines/engine.py.321: eval: epoch 10 | batch 330/510 [I 2022-10-26 13:38:30] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 13:38:30] ...trap/engines/engine.py.326: process: 0.28678 | load: 0.00019 [I 2022-10-26 13:38:33] ...trap/engines/engine.py.321: eval: epoch 10 | batch 340/510 [I 2022-10-26 13:38:33] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 13:38:33] ...trap/engines/engine.py.326: process: 0.28666 | load: 0.00017 [I 2022-10-26 13:38:36] ...trap/engines/engine.py.321: eval: epoch 10 | batch 350/510 [I 2022-10-26 13:38:36] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-26 13:38:36] ...trap/engines/engine.py.326: process: 0.28638 | load: 0.00019 [I 2022-10-26 13:38:39] ...trap/engines/engine.py.321: eval: epoch 10 | batch 360/510 [I 2022-10-26 13:38:39] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:42 [I 2022-10-26 13:38:39] ...trap/engines/engine.py.326: process: 0.29040 | load: 0.00017 [I 2022-10-26 13:38:42] ...trap/engines/engine.py.321: eval: epoch 10 | batch 370/510 [I 2022-10-26 13:38:42] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 13:38:42] ...trap/engines/engine.py.326: process: 0.28439 | load: 0.00015 [I 2022-10-26 13:38:45] ...trap/engines/engine.py.321: eval: epoch 10 | batch 380/510 [I 2022-10-26 13:38:45] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 13:38:45] ...trap/engines/engine.py.326: process: 0.28525 | load: 0.00016 [I 2022-10-26 13:38:48] ...trap/engines/engine.py.321: eval: epoch 10 | batch 390/510 [I 2022-10-26 13:38:48] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 13:38:48] ...trap/engines/engine.py.326: process: 0.28512 | load: 0.00019 [I 2022-10-26 13:38:50] ...trap/engines/engine.py.321: eval: epoch 10 | batch 400/510 [I 2022-10-26 13:38:50] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:32 [I 2022-10-26 13:38:50] ...trap/engines/engine.py.326: process: 0.28892 | load: 0.00019 [I 2022-10-26 13:38:53] ...trap/engines/engine.py.321: eval: epoch 10 | batch 410/510 [I 2022-10-26 13:38:53] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 13:38:53] ...trap/engines/engine.py.326: process: 0.28983 | load: 0.00014 [I 2022-10-26 13:38:56] ...trap/engines/engine.py.321: eval: epoch 10 | batch 420/510 [I 2022-10-26 13:38:56] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 13:38:56] ...trap/engines/engine.py.326: process: 0.27788 | load: 0.00015 [I 2022-10-26 13:38:59] ...trap/engines/engine.py.321: eval: epoch 10 | batch 430/510 [I 2022-10-26 13:38:59] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 13:38:59] ...trap/engines/engine.py.326: process: 0.29638 | load: 0.00015 [I 2022-10-26 13:39:02] ...trap/engines/engine.py.321: eval: epoch 10 | batch 440/510 [I 2022-10-26 13:39:02] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 13:39:02] ...trap/engines/engine.py.326: process: 0.28829 | load: 0.00016 [I 2022-10-26 13:39:05] ...trap/engines/engine.py.321: eval: epoch 10 | batch 450/510 [I 2022-10-26 13:39:05] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 13:39:05] ...trap/engines/engine.py.326: process: 0.29219 | load: 0.00015 [I 2022-10-26 13:39:08] ...trap/engines/engine.py.321: eval: epoch 10 | batch 460/510 [I 2022-10-26 13:39:08] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 13:39:08] ...trap/engines/engine.py.326: process: 0.29230 | load: 0.00019 [I 2022-10-26 13:39:11] ...trap/engines/engine.py.321: eval: epoch 10 | batch 470/510 [I 2022-10-26 13:39:11] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 13:39:11] ...trap/engines/engine.py.326: process: 0.28583 | load: 0.00034 [I 2022-10-26 13:39:13] ...trap/engines/engine.py.321: eval: epoch 10 | batch 480/510 [I 2022-10-26 13:39:13] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 13:39:13] ...trap/engines/engine.py.326: process: 0.28528 | load: 0.00021 [I 2022-10-26 13:39:16] ...trap/engines/engine.py.321: eval: epoch 10 | batch 490/510 [I 2022-10-26 13:39:16] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 13:39:16] ...trap/engines/engine.py.326: process: 0.30497 | load: 0.00016 [I 2022-10-26 13:39:19] ...trap/engines/engine.py.321: eval: epoch 10 | batch 500/510 [I 2022-10-26 13:39:19] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 13:39:19] ...trap/engines/engine.py.326: process: 0.28084 | load: 0.00015 [I 2022-10-26 13:39:22] ...trap/engines/engine.py.321: eval: epoch 10 | batch 510/510 [I 2022-10-26 13:39:22] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 13:39:22] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00011 [S 2022-10-26 13:39:23] ...trap/engines/engine.py.132: eval_epoch.epoch: 10 [I 2022-10-26 13:39:23] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.10 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.30 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.515100 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.803300 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.874400 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.511000 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.804700 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.876800 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.300000 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.458258 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011528 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009550 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008321 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015862 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008741 [S 2022-10-26 13:39:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006838 [I 2022-10-26 13:39:28] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-26 13:39:28] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 13:39:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 13:39:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 13:39:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 13:39:28] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 13:39:31] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 13:39:36] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 13:39:36] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 13:39:36] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 13:39:39] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 13:39:45] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 13:39:45] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 13:39:45] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 13:39:48] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 13:39:53] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 13:39:53] ...trap/engines/engine.py.163: Training model on trainset for epoch 11 [I 2022-10-26 13:39:53] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 13:39:55] ...trap/engines/engine.py.224: train: epoch 11 | batch 0/2304 [I 2022-10-26 13:39:55] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:25 [I 2022-10-26 13:39:55] ...trap/engines/engine.py.229: process: 2.77155 | load: 2.20010 [I 2022-10-26 13:39:55] ...trap/engines/engine.py.230: loss: 0.17201 [I 2022-10-26 13:39:55] ...optimizers/trijoint.py.151: total_norm: 3.379909 [I 2022-10-26 13:40:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 10/2304 [I 2022-10-26 13:40:01] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:38 [I 2022-10-26 13:40:01] ...trap/engines/engine.py.229: process: 0.49646 | load: 0.00473 [I 2022-10-26 13:40:01] ...trap/engines/engine.py.230: loss: 0.26330 [I 2022-10-26 13:40:01] ...optimizers/trijoint.py.151: total_norm: 3.740938 [I 2022-10-26 13:40:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 20/2304 [I 2022-10-26 13:40:06] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:20:28 [I 2022-10-26 13:40:06] ...trap/engines/engine.py.229: process: 0.50226 | load: 0.00345 [I 2022-10-26 13:40:06] ...trap/engines/engine.py.230: loss: 0.19389 [I 2022-10-26 13:40:06] ...optimizers/trijoint.py.151: total_norm: 2.411532 [I 2022-10-26 13:40:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 30/2304 [I 2022-10-26 13:40:11] ...trap/engines/engine.py.228: elapsed: 0:00:18 | left: 0:19:07 [I 2022-10-26 13:40:11] ...trap/engines/engine.py.229: process: 0.50602 | load: 0.00352 [I 2022-10-26 13:40:11] ...trap/engines/engine.py.230: loss: 0.15124 [I 2022-10-26 13:40:11] ...optimizers/trijoint.py.151: total_norm: 2.332561 [I 2022-10-26 13:40:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 40/2304 [I 2022-10-26 13:40:16] ...trap/engines/engine.py.228: elapsed: 0:00:23 | left: 0:18:45 [I 2022-10-26 13:40:16] ...trap/engines/engine.py.229: process: 0.48950 | load: 0.00288 [I 2022-10-26 13:40:16] ...trap/engines/engine.py.230: loss: 0.22505 [I 2022-10-26 13:40:16] ...optimizers/trijoint.py.151: total_norm: 2.584461 [I 2022-10-26 13:40:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 50/2304 [I 2022-10-26 13:40:21] ...trap/engines/engine.py.228: elapsed: 0:00:28 | left: 0:18:45 [I 2022-10-26 13:40:21] ...trap/engines/engine.py.229: process: 0.49654 | load: 0.00303 [I 2022-10-26 13:40:21] ...trap/engines/engine.py.230: loss: 0.19047 [I 2022-10-26 13:40:21] ...optimizers/trijoint.py.151: total_norm: 3.476126 [I 2022-10-26 13:40:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 60/2304 [I 2022-10-26 13:40:26] ...trap/engines/engine.py.228: elapsed: 0:00:33 | left: 0:18:44 [I 2022-10-26 13:40:26] ...trap/engines/engine.py.229: process: 0.49340 | load: 0.00300 [I 2022-10-26 13:40:26] ...trap/engines/engine.py.230: loss: 0.24479 [I 2022-10-26 13:40:26] ...optimizers/trijoint.py.151: total_norm: 3.062555 [I 2022-10-26 13:40:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 70/2304 [I 2022-10-26 13:40:31] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:18:25 [I 2022-10-26 13:40:31] ...trap/engines/engine.py.229: process: 0.48481 | load: 0.00288 [I 2022-10-26 13:40:31] ...trap/engines/engine.py.230: loss: 0.27836 [I 2022-10-26 13:40:31] ...optimizers/trijoint.py.151: total_norm: 2.856775 [I 2022-10-26 13:40:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 80/2304 [I 2022-10-26 13:40:36] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:18:22 [I 2022-10-26 13:40:36] ...trap/engines/engine.py.229: process: 0.48828 | load: 0.00287 [I 2022-10-26 13:40:36] ...trap/engines/engine.py.230: loss: 0.30630 [I 2022-10-26 13:40:36] ...optimizers/trijoint.py.151: total_norm: 3.927242 [I 2022-10-26 13:40:41] ...trap/engines/engine.py.224: train: epoch 11 | batch 90/2304 [I 2022-10-26 13:40:41] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:18:23 [I 2022-10-26 13:40:41] ...trap/engines/engine.py.229: process: 0.50359 | load: 0.00302 [I 2022-10-26 13:40:41] ...trap/engines/engine.py.230: loss: 0.33540 [I 2022-10-26 13:40:41] ...optimizers/trijoint.py.151: total_norm: 4.708964 [I 2022-10-26 13:40:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 100/2304 [I 2022-10-26 13:40:46] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:18:29 [I 2022-10-26 13:40:46] ...trap/engines/engine.py.229: process: 0.49691 | load: 0.00318 [I 2022-10-26 13:40:46] ...trap/engines/engine.py.230: loss: 0.26162 [I 2022-10-26 13:40:46] ...optimizers/trijoint.py.151: total_norm: 3.365010 [I 2022-10-26 13:40:51] ...trap/engines/engine.py.224: train: epoch 11 | batch 110/2304 [I 2022-10-26 13:40:51] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:18:15 [I 2022-10-26 13:40:51] ...trap/engines/engine.py.229: process: 0.50046 | load: 0.00287 [I 2022-10-26 13:40:51] ...trap/engines/engine.py.230: loss: 0.21615 [I 2022-10-26 13:40:51] ...optimizers/trijoint.py.151: total_norm: 4.038267 [I 2022-10-26 13:40:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 120/2304 [I 2022-10-26 13:40:56] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:03 [I 2022-10-26 13:40:56] ...trap/engines/engine.py.229: process: 0.50380 | load: 0.00270 [I 2022-10-26 13:40:56] ...trap/engines/engine.py.230: loss: 0.25882 [I 2022-10-26 13:40:56] ...optimizers/trijoint.py.151: total_norm: 3.835620 [I 2022-10-26 13:41:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 130/2304 [I 2022-10-26 13:41:01] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:01 [I 2022-10-26 13:41:01] ...trap/engines/engine.py.229: process: 0.50026 | load: 0.00276 [I 2022-10-26 13:41:01] ...trap/engines/engine.py.230: loss: 0.23613 [I 2022-10-26 13:41:01] ...optimizers/trijoint.py.151: total_norm: 2.729696 [I 2022-10-26 13:41:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 140/2304 [I 2022-10-26 13:41:06] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:18:05 [I 2022-10-26 13:41:06] ...trap/engines/engine.py.229: process: 0.49569 | load: 0.00284 [I 2022-10-26 13:41:06] ...trap/engines/engine.py.230: loss: 0.16196 [I 2022-10-26 13:41:06] ...optimizers/trijoint.py.151: total_norm: 3.308580 [I 2022-10-26 13:41:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 150/2304 [I 2022-10-26 13:41:11] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:17:55 [I 2022-10-26 13:41:11] ...trap/engines/engine.py.229: process: 0.50023 | load: 0.00277 [I 2022-10-26 13:41:11] ...trap/engines/engine.py.230: loss: 0.30616 [I 2022-10-26 13:41:11] ...optimizers/trijoint.py.151: total_norm: 3.393540 [I 2022-10-26 13:41:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 160/2304 [I 2022-10-26 13:41:16] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:17:38 [I 2022-10-26 13:41:16] ...trap/engines/engine.py.229: process: 0.49585 | load: 0.00289 [I 2022-10-26 13:41:16] ...trap/engines/engine.py.230: loss: 0.17551 [I 2022-10-26 13:41:16] ...optimizers/trijoint.py.151: total_norm: 2.761007 [I 2022-10-26 13:41:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 170/2304 [I 2022-10-26 13:41:21] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:41 [I 2022-10-26 13:41:21] ...trap/engines/engine.py.229: process: 0.50485 | load: 0.00274 [I 2022-10-26 13:41:21] ...trap/engines/engine.py.230: loss: 0.21644 [I 2022-10-26 13:41:21] ...optimizers/trijoint.py.151: total_norm: 4.444482 [I 2022-10-26 13:41:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 180/2304 [I 2022-10-26 13:41:26] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:17:49 [I 2022-10-26 13:41:26] ...trap/engines/engine.py.229: process: 0.49300 | load: 0.00291 [I 2022-10-26 13:41:26] ...trap/engines/engine.py.230: loss: 0.22871 [I 2022-10-26 13:41:26] ...optimizers/trijoint.py.151: total_norm: 2.581294 [I 2022-10-26 13:41:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 190/2304 [I 2022-10-26 13:41:31] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:17:44 [I 2022-10-26 13:41:31] ...trap/engines/engine.py.229: process: 0.51104 | load: 0.00268 [I 2022-10-26 13:41:31] ...trap/engines/engine.py.230: loss: 0.13297 [I 2022-10-26 13:41:31] ...optimizers/trijoint.py.151: total_norm: 2.810270 [I 2022-10-26 13:41:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 200/2304 [I 2022-10-26 13:41:36] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:28 [I 2022-10-26 13:41:36] ...trap/engines/engine.py.229: process: 0.49783 | load: 0.00263 [I 2022-10-26 13:41:36] ...trap/engines/engine.py.230: loss: 0.22205 [I 2022-10-26 13:41:36] ...optimizers/trijoint.py.151: total_norm: 2.842129 [I 2022-10-26 13:41:41] ...trap/engines/engine.py.224: train: epoch 11 | batch 210/2304 [I 2022-10-26 13:41:41] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:24 [I 2022-10-26 13:41:41] ...trap/engines/engine.py.229: process: 0.50198 | load: 0.00254 [I 2022-10-26 13:41:41] ...trap/engines/engine.py.230: loss: 0.24251 [I 2022-10-26 13:41:41] ...optimizers/trijoint.py.151: total_norm: 3.206741 [I 2022-10-26 13:41:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 220/2304 [I 2022-10-26 13:41:46] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:18 [I 2022-10-26 13:41:46] ...trap/engines/engine.py.229: process: 0.48903 | load: 0.00267 [I 2022-10-26 13:41:46] ...trap/engines/engine.py.230: loss: 0.28541 [I 2022-10-26 13:41:46] ...optimizers/trijoint.py.151: total_norm: 3.002252 [I 2022-10-26 13:41:51] ...trap/engines/engine.py.224: train: epoch 11 | batch 230/2304 [I 2022-10-26 13:41:51] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:10 [I 2022-10-26 13:41:51] ...trap/engines/engine.py.229: process: 0.50218 | load: 0.00257 [I 2022-10-26 13:41:51] ...trap/engines/engine.py.230: loss: 0.24111 [I 2022-10-26 13:41:51] ...optimizers/trijoint.py.151: total_norm: 3.090999 [I 2022-10-26 13:41:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 240/2304 [I 2022-10-26 13:41:56] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:00 [I 2022-10-26 13:41:56] ...trap/engines/engine.py.229: process: 0.48961 | load: 0.00257 [I 2022-10-26 13:41:56] ...trap/engines/engine.py.230: loss: 0.26442 [I 2022-10-26 13:41:56] ...optimizers/trijoint.py.151: total_norm: 4.037454 [I 2022-10-26 13:42:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 250/2304 [I 2022-10-26 13:42:01] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:16:59 [I 2022-10-26 13:42:01] ...trap/engines/engine.py.229: process: 0.50312 | load: 0.00256 [I 2022-10-26 13:42:01] ...trap/engines/engine.py.230: loss: 0.22983 [I 2022-10-26 13:42:01] ...optimizers/trijoint.py.151: total_norm: 2.959490 [I 2022-10-26 13:42:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 260/2304 [I 2022-10-26 13:42:06] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:09 [I 2022-10-26 13:42:06] ...trap/engines/engine.py.229: process: 0.50701 | load: 0.00245 [I 2022-10-26 13:42:06] ...trap/engines/engine.py.230: loss: 0.18934 [I 2022-10-26 13:42:06] ...optimizers/trijoint.py.151: total_norm: 2.698462 [I 2022-10-26 13:42:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 270/2304 [I 2022-10-26 13:42:11] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:16:57 [I 2022-10-26 13:42:11] ...trap/engines/engine.py.229: process: 0.49850 | load: 0.00250 [I 2022-10-26 13:42:11] ...trap/engines/engine.py.230: loss: 0.28335 [I 2022-10-26 13:42:11] ...optimizers/trijoint.py.151: total_norm: 3.660934 [I 2022-10-26 13:42:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 280/2304 [I 2022-10-26 13:42:16] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:16:41 [I 2022-10-26 13:42:16] ...trap/engines/engine.py.229: process: 0.49482 | load: 0.00245 [I 2022-10-26 13:42:16] ...trap/engines/engine.py.230: loss: 0.18170 [I 2022-10-26 13:42:16] ...optimizers/trijoint.py.151: total_norm: 3.007092 [I 2022-10-26 13:42:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 290/2304 [I 2022-10-26 13:42:21] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:16:53 [I 2022-10-26 13:42:21] ...trap/engines/engine.py.229: process: 0.51677 | load: 0.00253 [I 2022-10-26 13:42:21] ...trap/engines/engine.py.230: loss: 0.26004 [I 2022-10-26 13:42:21] ...optimizers/trijoint.py.151: total_norm: 3.534881 [I 2022-10-26 13:42:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 300/2304 [I 2022-10-26 13:42:26] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:16:41 [I 2022-10-26 13:42:26] ...trap/engines/engine.py.229: process: 0.49088 | load: 0.00263 [I 2022-10-26 13:42:26] ...trap/engines/engine.py.230: loss: 0.21962 [I 2022-10-26 13:42:26] ...optimizers/trijoint.py.151: total_norm: 2.909573 [I 2022-10-26 13:42:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 310/2304 [I 2022-10-26 13:42:31] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:16:43 [I 2022-10-26 13:42:31] ...trap/engines/engine.py.229: process: 0.49437 | load: 0.00248 [I 2022-10-26 13:42:31] ...trap/engines/engine.py.230: loss: 0.22626 [I 2022-10-26 13:42:31] ...optimizers/trijoint.py.151: total_norm: 3.602239 [I 2022-10-26 13:42:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 320/2304 [I 2022-10-26 13:42:36] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:16:39 [I 2022-10-26 13:42:36] ...trap/engines/engine.py.229: process: 0.49572 | load: 0.00262 [I 2022-10-26 13:42:36] ...trap/engines/engine.py.230: loss: 0.17003 [I 2022-10-26 13:42:36] ...optimizers/trijoint.py.151: total_norm: 3.081831 [I 2022-10-26 13:42:41] ...trap/engines/engine.py.224: train: epoch 11 | batch 330/2304 [I 2022-10-26 13:42:41] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:16:34 [I 2022-10-26 13:42:41] ...trap/engines/engine.py.229: process: 0.50399 | load: 0.00258 [I 2022-10-26 13:42:41] ...trap/engines/engine.py.230: loss: 0.37371 [I 2022-10-26 13:42:41] ...optimizers/trijoint.py.151: total_norm: 4.179139 [I 2022-10-26 13:42:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 340/2304 [I 2022-10-26 13:42:46] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:16:30 [I 2022-10-26 13:42:46] ...trap/engines/engine.py.229: process: 0.50713 | load: 0.00249 [I 2022-10-26 13:42:46] ...trap/engines/engine.py.230: loss: 0.18110 [I 2022-10-26 13:42:46] ...optimizers/trijoint.py.151: total_norm: 2.384296 [I 2022-10-26 13:42:51] ...trap/engines/engine.py.224: train: epoch 11 | batch 350/2304 [I 2022-10-26 13:42:51] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:16:21 [I 2022-10-26 13:42:51] ...trap/engines/engine.py.229: process: 0.49493 | load: 0.00281 [I 2022-10-26 13:42:51] ...trap/engines/engine.py.230: loss: 0.28987 [I 2022-10-26 13:42:51] ...optimizers/trijoint.py.151: total_norm: 4.024704 [I 2022-10-26 13:42:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 360/2304 [I 2022-10-26 13:42:56] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:16:06 [I 2022-10-26 13:42:56] ...trap/engines/engine.py.229: process: 0.48932 | load: 0.00266 [I 2022-10-26 13:42:56] ...trap/engines/engine.py.230: loss: 0.25833 [I 2022-10-26 13:42:56] ...optimizers/trijoint.py.151: total_norm: 3.573959 [I 2022-10-26 13:43:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 370/2304 [I 2022-10-26 13:43:01] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:15:57 [I 2022-10-26 13:43:01] ...trap/engines/engine.py.229: process: 0.48670 | load: 0.00246 [I 2022-10-26 13:43:01] ...trap/engines/engine.py.230: loss: 0.17647 [I 2022-10-26 13:43:01] ...optimizers/trijoint.py.151: total_norm: 2.677890 [I 2022-10-26 13:43:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 380/2304 [I 2022-10-26 13:43:06] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:15:52 [I 2022-10-26 13:43:06] ...trap/engines/engine.py.229: process: 0.50400 | load: 0.00249 [I 2022-10-26 13:43:06] ...trap/engines/engine.py.230: loss: 0.19527 [I 2022-10-26 13:43:06] ...optimizers/trijoint.py.151: total_norm: 3.007189 [I 2022-10-26 13:43:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 390/2304 [I 2022-10-26 13:43:11] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:15:45 [I 2022-10-26 13:43:11] ...trap/engines/engine.py.229: process: 0.49939 | load: 0.00250 [I 2022-10-26 13:43:11] ...trap/engines/engine.py.230: loss: 0.27621 [I 2022-10-26 13:43:11] ...optimizers/trijoint.py.151: total_norm: 4.394139 [I 2022-10-26 13:43:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 400/2304 [I 2022-10-26 13:43:16] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:15:52 [I 2022-10-26 13:43:16] ...trap/engines/engine.py.229: process: 0.50632 | load: 0.00244 [I 2022-10-26 13:43:16] ...trap/engines/engine.py.230: loss: 0.21387 [I 2022-10-26 13:43:16] ...optimizers/trijoint.py.151: total_norm: 2.917930 [I 2022-10-26 13:43:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 410/2304 [I 2022-10-26 13:43:21] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:15:45 [I 2022-10-26 13:43:21] ...trap/engines/engine.py.229: process: 0.50439 | load: 0.00246 [I 2022-10-26 13:43:21] ...trap/engines/engine.py.230: loss: 0.19014 [I 2022-10-26 13:43:21] ...optimizers/trijoint.py.151: total_norm: 3.093822 [I 2022-10-26 13:43:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 420/2304 [I 2022-10-26 13:43:26] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:15:35 [I 2022-10-26 13:43:26] ...trap/engines/engine.py.229: process: 0.49949 | load: 0.00246 [I 2022-10-26 13:43:26] ...trap/engines/engine.py.230: loss: 0.21998 [I 2022-10-26 13:43:26] ...optimizers/trijoint.py.151: total_norm: 2.754402 [I 2022-10-26 13:43:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 430/2304 [I 2022-10-26 13:43:31] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:15:32 [I 2022-10-26 13:43:31] ...trap/engines/engine.py.229: process: 0.49378 | load: 0.00259 [I 2022-10-26 13:43:31] ...trap/engines/engine.py.230: loss: 0.18782 [I 2022-10-26 13:43:31] ...optimizers/trijoint.py.151: total_norm: 2.980447 [I 2022-10-26 13:43:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 440/2304 [I 2022-10-26 13:43:36] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:15:25 [I 2022-10-26 13:43:36] ...trap/engines/engine.py.229: process: 0.50236 | load: 0.00247 [I 2022-10-26 13:43:36] ...trap/engines/engine.py.230: loss: 0.28007 [I 2022-10-26 13:43:36] ...optimizers/trijoint.py.151: total_norm: 4.120305 [I 2022-10-26 13:43:41] ...trap/engines/engine.py.224: train: epoch 11 | batch 450/2304 [I 2022-10-26 13:43:41] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:15:17 [I 2022-10-26 13:43:41] ...trap/engines/engine.py.229: process: 0.49341 | load: 0.00251 [I 2022-10-26 13:43:41] ...trap/engines/engine.py.230: loss: 0.31254 [I 2022-10-26 13:43:41] ...optimizers/trijoint.py.151: total_norm: 3.291592 [I 2022-10-26 13:43:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 460/2304 [I 2022-10-26 13:43:46] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:15:19 [I 2022-10-26 13:43:46] ...trap/engines/engine.py.229: process: 0.49103 | load: 0.00248 [I 2022-10-26 13:43:46] ...trap/engines/engine.py.230: loss: 0.26300 [I 2022-10-26 13:43:46] ...optimizers/trijoint.py.151: total_norm: 3.489289 [I 2022-10-26 13:43:51] ...trap/engines/engine.py.224: train: epoch 11 | batch 470/2304 [I 2022-10-26 13:43:51] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:15:14 [I 2022-10-26 13:43:51] ...trap/engines/engine.py.229: process: 0.49560 | load: 0.00245 [I 2022-10-26 13:43:51] ...trap/engines/engine.py.230: loss: 0.24614 [I 2022-10-26 13:43:51] ...optimizers/trijoint.py.151: total_norm: 3.084979 [I 2022-10-26 13:43:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 480/2304 [I 2022-10-26 13:43:56] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:15:12 [I 2022-10-26 13:43:56] ...trap/engines/engine.py.229: process: 0.49866 | load: 0.00250 [I 2022-10-26 13:43:56] ...trap/engines/engine.py.230: loss: 0.25345 [I 2022-10-26 13:43:56] ...optimizers/trijoint.py.151: total_norm: 3.214051 [I 2022-10-26 13:44:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 490/2304 [I 2022-10-26 13:44:01] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:15:03 [I 2022-10-26 13:44:01] ...trap/engines/engine.py.229: process: 0.49577 | load: 0.00246 [I 2022-10-26 13:44:01] ...trap/engines/engine.py.230: loss: 0.17952 [I 2022-10-26 13:44:01] ...optimizers/trijoint.py.151: total_norm: 2.871225 [I 2022-10-26 13:44:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 500/2304 [I 2022-10-26 13:44:06] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:15:04 [I 2022-10-26 13:44:06] ...trap/engines/engine.py.229: process: 0.49699 | load: 0.00263 [I 2022-10-26 13:44:06] ...trap/engines/engine.py.230: loss: 0.26014 [I 2022-10-26 13:44:06] ...optimizers/trijoint.py.151: total_norm: 3.274975 [I 2022-10-26 13:44:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 510/2304 [I 2022-10-26 13:44:11] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:14:52 [I 2022-10-26 13:44:11] ...trap/engines/engine.py.229: process: 0.49771 | load: 0.00253 [I 2022-10-26 13:44:11] ...trap/engines/engine.py.230: loss: 0.16438 [I 2022-10-26 13:44:11] ...optimizers/trijoint.py.151: total_norm: 3.636658 [I 2022-10-26 13:44:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 520/2304 [I 2022-10-26 13:44:16] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:14:45 [I 2022-10-26 13:44:16] ...trap/engines/engine.py.229: process: 0.49897 | load: 0.00258 [I 2022-10-26 13:44:16] ...trap/engines/engine.py.230: loss: 0.27188 [I 2022-10-26 13:44:16] ...optimizers/trijoint.py.151: total_norm: 3.769198 [I 2022-10-26 13:44:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 530/2304 [I 2022-10-26 13:44:21] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:14:37 [I 2022-10-26 13:44:21] ...trap/engines/engine.py.229: process: 0.49881 | load: 0.00279 [I 2022-10-26 13:44:21] ...trap/engines/engine.py.230: loss: 0.17890 [I 2022-10-26 13:44:21] ...optimizers/trijoint.py.151: total_norm: 3.502310 [I 2022-10-26 13:44:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 540/2304 [I 2022-10-26 13:44:26] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:14:27 [I 2022-10-26 13:44:26] ...trap/engines/engine.py.229: process: 0.48737 | load: 0.00243 [I 2022-10-26 13:44:26] ...trap/engines/engine.py.230: loss: 0.32256 [I 2022-10-26 13:44:26] ...optimizers/trijoint.py.151: total_norm: 3.455336 [I 2022-10-26 13:44:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 550/2304 [I 2022-10-26 13:44:31] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:14:22 [I 2022-10-26 13:44:31] ...trap/engines/engine.py.229: process: 0.48454 | load: 0.00245 [I 2022-10-26 13:44:31] ...trap/engines/engine.py.230: loss: 0.27487 [I 2022-10-26 13:44:31] ...optimizers/trijoint.py.151: total_norm: 3.946193 [I 2022-10-26 13:44:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 560/2304 [I 2022-10-26 13:44:36] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:14:28 [I 2022-10-26 13:44:36] ...trap/engines/engine.py.229: process: 0.49014 | load: 0.00262 [I 2022-10-26 13:44:36] ...trap/engines/engine.py.230: loss: 0.26213 [I 2022-10-26 13:44:36] ...optimizers/trijoint.py.151: total_norm: 3.338861 [I 2022-10-26 13:44:41] ...trap/engines/engine.py.224: train: epoch 11 | batch 570/2304 [I 2022-10-26 13:44:41] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:14:26 [I 2022-10-26 13:44:41] ...trap/engines/engine.py.229: process: 0.49235 | load: 0.00247 [I 2022-10-26 13:44:41] ...trap/engines/engine.py.230: loss: 0.17967 [I 2022-10-26 13:44:41] ...optimizers/trijoint.py.151: total_norm: 3.447333 [I 2022-10-26 13:44:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 580/2304 [I 2022-10-26 13:44:46] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:14:19 [I 2022-10-26 13:44:46] ...trap/engines/engine.py.229: process: 0.50189 | load: 0.00247 [I 2022-10-26 13:44:46] ...trap/engines/engine.py.230: loss: 0.21375 [I 2022-10-26 13:44:46] ...optimizers/trijoint.py.151: total_norm: 3.040857 [I 2022-10-26 13:44:51] ...trap/engines/engine.py.224: train: epoch 11 | batch 590/2304 [I 2022-10-26 13:44:51] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:14:08 [I 2022-10-26 13:44:51] ...trap/engines/engine.py.229: process: 0.49724 | load: 0.00253 [I 2022-10-26 13:44:51] ...trap/engines/engine.py.230: loss: 0.33953 [I 2022-10-26 13:44:51] ...optimizers/trijoint.py.151: total_norm: 4.042441 [I 2022-10-26 13:44:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 600/2304 [I 2022-10-26 13:44:56] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:14:07 [I 2022-10-26 13:44:56] ...trap/engines/engine.py.229: process: 0.50049 | load: 0.00247 [I 2022-10-26 13:44:56] ...trap/engines/engine.py.230: loss: 0.23808 [I 2022-10-26 13:44:56] ...optimizers/trijoint.py.151: total_norm: 3.620061 [I 2022-10-26 13:45:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 610/2304 [I 2022-10-26 13:45:01] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:13:59 [I 2022-10-26 13:45:01] ...trap/engines/engine.py.229: process: 0.48861 | load: 0.00247 [I 2022-10-26 13:45:01] ...trap/engines/engine.py.230: loss: 0.24687 [I 2022-10-26 13:45:01] ...optimizers/trijoint.py.151: total_norm: 3.338369 [I 2022-10-26 13:45:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 620/2304 [I 2022-10-26 13:45:06] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:13:54 [I 2022-10-26 13:45:06] ...trap/engines/engine.py.229: process: 0.49603 | load: 0.00248 [I 2022-10-26 13:45:06] ...trap/engines/engine.py.230: loss: 0.42963 [I 2022-10-26 13:45:06] ...optimizers/trijoint.py.151: total_norm: 5.107564 [I 2022-10-26 13:45:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 630/2304 [I 2022-10-26 13:45:11] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:13:52 [I 2022-10-26 13:45:11] ...trap/engines/engine.py.229: process: 0.49468 | load: 0.00246 [I 2022-10-26 13:45:11] ...trap/engines/engine.py.230: loss: 0.26874 [I 2022-10-26 13:45:11] ...optimizers/trijoint.py.151: total_norm: 4.204822 [I 2022-10-26 13:45:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 640/2304 [I 2022-10-26 13:45:16] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:14:03 [I 2022-10-26 13:45:16] ...trap/engines/engine.py.229: process: 0.51101 | load: 0.00246 [I 2022-10-26 13:45:16] ...trap/engines/engine.py.230: loss: 0.15536 [I 2022-10-26 13:45:16] ...optimizers/trijoint.py.151: total_norm: 2.128765 [I 2022-10-26 13:45:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 650/2304 [I 2022-10-26 13:45:21] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:13:53 [I 2022-10-26 13:45:21] ...trap/engines/engine.py.229: process: 0.50114 | load: 0.00244 [I 2022-10-26 13:45:21] ...trap/engines/engine.py.230: loss: 0.25718 [I 2022-10-26 13:45:21] ...optimizers/trijoint.py.151: total_norm: 3.304109 [I 2022-10-26 13:45:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 660/2304 [I 2022-10-26 13:45:26] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:14:08 [I 2022-10-26 13:45:26] ...trap/engines/engine.py.229: process: 0.49576 | load: 0.00262 [I 2022-10-26 13:45:26] ...trap/engines/engine.py.230: loss: 0.16791 [I 2022-10-26 13:45:26] ...optimizers/trijoint.py.151: total_norm: 3.361640 [I 2022-10-26 13:45:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 670/2304 [I 2022-10-26 13:45:31] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:13:35 [I 2022-10-26 13:45:31] ...trap/engines/engine.py.229: process: 0.49080 | load: 0.00247 [I 2022-10-26 13:45:31] ...trap/engines/engine.py.230: loss: 0.18965 [I 2022-10-26 13:45:31] ...optimizers/trijoint.py.151: total_norm: 2.501170 [I 2022-10-26 13:45:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 680/2304 [I 2022-10-26 13:45:36] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:13:29 [I 2022-10-26 13:45:36] ...trap/engines/engine.py.229: process: 0.49681 | load: 0.00242 [I 2022-10-26 13:45:36] ...trap/engines/engine.py.230: loss: 0.23724 [I 2022-10-26 13:45:36] ...optimizers/trijoint.py.151: total_norm: 3.314743 [I 2022-10-26 13:45:41] ...trap/engines/engine.py.224: train: epoch 11 | batch 690/2304 [I 2022-10-26 13:45:41] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:13:20 [I 2022-10-26 13:45:41] ...trap/engines/engine.py.229: process: 0.48489 | load: 0.00244 [I 2022-10-26 13:45:41] ...trap/engines/engine.py.230: loss: 0.30425 [I 2022-10-26 13:45:41] ...optimizers/trijoint.py.151: total_norm: 3.558060 [I 2022-10-26 13:45:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 700/2304 [I 2022-10-26 13:45:46] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:13:17 [I 2022-10-26 13:45:46] ...trap/engines/engine.py.229: process: 0.49618 | load: 0.00252 [I 2022-10-26 13:45:46] ...trap/engines/engine.py.230: loss: 0.25757 [I 2022-10-26 13:45:46] ...optimizers/trijoint.py.151: total_norm: 3.571049 [I 2022-10-26 13:45:51] ...trap/engines/engine.py.224: train: epoch 11 | batch 710/2304 [I 2022-10-26 13:45:51] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:13:12 [I 2022-10-26 13:45:51] ...trap/engines/engine.py.229: process: 0.50004 | load: 0.00247 [I 2022-10-26 13:45:51] ...trap/engines/engine.py.230: loss: 0.22161 [I 2022-10-26 13:45:51] ...optimizers/trijoint.py.151: total_norm: 4.154259 [I 2022-10-26 13:45:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 720/2304 [I 2022-10-26 13:45:56] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:13:11 [I 2022-10-26 13:45:56] ...trap/engines/engine.py.229: process: 0.49828 | load: 0.00244 [I 2022-10-26 13:45:56] ...trap/engines/engine.py.230: loss: 0.19385 [I 2022-10-26 13:45:56] ...optimizers/trijoint.py.151: total_norm: 3.536197 [I 2022-10-26 13:46:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 730/2304 [I 2022-10-26 13:46:01] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:13:06 [I 2022-10-26 13:46:01] ...trap/engines/engine.py.229: process: 0.50692 | load: 0.00245 [I 2022-10-26 13:46:01] ...trap/engines/engine.py.230: loss: 0.19012 [I 2022-10-26 13:46:01] ...optimizers/trijoint.py.151: total_norm: 3.245392 [I 2022-10-26 13:46:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 740/2304 [I 2022-10-26 13:46:06] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:12:56 [I 2022-10-26 13:46:06] ...trap/engines/engine.py.229: process: 0.49799 | load: 0.00242 [I 2022-10-26 13:46:06] ...trap/engines/engine.py.230: loss: 0.24626 [I 2022-10-26 13:46:06] ...optimizers/trijoint.py.151: total_norm: 3.359201 [I 2022-10-26 13:46:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 750/2304 [I 2022-10-26 13:46:11] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:13:14 [I 2022-10-26 13:46:11] ...trap/engines/engine.py.229: process: 0.52096 | load: 0.00247 [I 2022-10-26 13:46:11] ...trap/engines/engine.py.230: loss: 0.24138 [I 2022-10-26 13:46:11] ...optimizers/trijoint.py.151: total_norm: 3.333364 [I 2022-10-26 13:46:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 760/2304 [I 2022-10-26 13:46:16] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:12:55 [I 2022-10-26 13:46:16] ...trap/engines/engine.py.229: process: 0.48505 | load: 0.00250 [I 2022-10-26 13:46:16] ...trap/engines/engine.py.230: loss: 0.28029 [I 2022-10-26 13:46:16] ...optimizers/trijoint.py.151: total_norm: 4.553440 [I 2022-10-26 13:46:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 770/2304 [I 2022-10-26 13:46:21] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:13:00 [I 2022-10-26 13:46:21] ...trap/engines/engine.py.229: process: 0.50510 | load: 0.00255 [I 2022-10-26 13:46:21] ...trap/engines/engine.py.230: loss: 0.14086 [I 2022-10-26 13:46:21] ...optimizers/trijoint.py.151: total_norm: 3.112827 [I 2022-10-26 13:46:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 780/2304 [I 2022-10-26 13:46:26] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:12:52 [I 2022-10-26 13:46:26] ...trap/engines/engine.py.229: process: 0.51066 | load: 0.00266 [I 2022-10-26 13:46:26] ...trap/engines/engine.py.230: loss: 0.23773 [I 2022-10-26 13:46:26] ...optimizers/trijoint.py.151: total_norm: 2.932290 [I 2022-10-26 13:46:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 790/2304 [I 2022-10-26 13:46:31] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:12:35 [I 2022-10-26 13:46:31] ...trap/engines/engine.py.229: process: 0.50034 | load: 0.00248 [I 2022-10-26 13:46:31] ...trap/engines/engine.py.230: loss: 0.20000 [I 2022-10-26 13:46:31] ...optimizers/trijoint.py.151: total_norm: 3.312034 [I 2022-10-26 13:46:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 800/2304 [I 2022-10-26 13:46:36] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:12:28 [I 2022-10-26 13:46:36] ...trap/engines/engine.py.229: process: 0.49721 | load: 0.00247 [I 2022-10-26 13:46:36] ...trap/engines/engine.py.230: loss: 0.20451 [I 2022-10-26 13:46:36] ...optimizers/trijoint.py.151: total_norm: 3.615939 [I 2022-10-26 13:46:41] ...trap/engines/engine.py.224: train: epoch 11 | batch 810/2304 [I 2022-10-26 13:46:41] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:12:23 [I 2022-10-26 13:46:41] ...trap/engines/engine.py.229: process: 0.49800 | load: 0.00249 [I 2022-10-26 13:46:41] ...trap/engines/engine.py.230: loss: 0.18121 [I 2022-10-26 13:46:41] ...optimizers/trijoint.py.151: total_norm: 2.837978 [I 2022-10-26 13:46:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 820/2304 [I 2022-10-26 13:46:46] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:12:20 [I 2022-10-26 13:46:46] ...trap/engines/engine.py.229: process: 0.49328 | load: 0.00248 [I 2022-10-26 13:46:46] ...trap/engines/engine.py.230: loss: 0.21693 [I 2022-10-26 13:46:46] ...optimizers/trijoint.py.151: total_norm: 2.892681 [I 2022-10-26 13:46:51] ...trap/engines/engine.py.224: train: epoch 11 | batch 830/2304 [I 2022-10-26 13:46:51] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:12:12 [I 2022-10-26 13:46:51] ...trap/engines/engine.py.229: process: 0.49650 | load: 0.00244 [I 2022-10-26 13:46:51] ...trap/engines/engine.py.230: loss: 0.19221 [I 2022-10-26 13:46:51] ...optimizers/trijoint.py.151: total_norm: 2.864888 [I 2022-10-26 13:46:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 840/2304 [I 2022-10-26 13:46:56] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:12:10 [I 2022-10-26 13:46:56] ...trap/engines/engine.py.229: process: 0.50919 | load: 0.00242 [I 2022-10-26 13:46:56] ...trap/engines/engine.py.230: loss: 0.24412 [I 2022-10-26 13:46:56] ...optimizers/trijoint.py.151: total_norm: 3.115098 [I 2022-10-26 13:47:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 850/2304 [I 2022-10-26 13:47:01] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:12:00 [I 2022-10-26 13:47:01] ...trap/engines/engine.py.229: process: 0.49304 | load: 0.00250 [I 2022-10-26 13:47:01] ...trap/engines/engine.py.230: loss: 0.23961 [I 2022-10-26 13:47:01] ...optimizers/trijoint.py.151: total_norm: 3.759485 [I 2022-10-26 13:47:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 860/2304 [I 2022-10-26 13:47:06] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:11:58 [I 2022-10-26 13:47:06] ...trap/engines/engine.py.229: process: 0.50076 | load: 0.00247 [I 2022-10-26 13:47:06] ...trap/engines/engine.py.230: loss: 0.29372 [I 2022-10-26 13:47:06] ...optimizers/trijoint.py.151: total_norm: 3.658023 [I 2022-10-26 13:47:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 870/2304 [I 2022-10-26 13:47:11] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:11:51 [I 2022-10-26 13:47:11] ...trap/engines/engine.py.229: process: 0.48965 | load: 0.00244 [I 2022-10-26 13:47:11] ...trap/engines/engine.py.230: loss: 0.17125 [I 2022-10-26 13:47:11] ...optimizers/trijoint.py.151: total_norm: 2.697350 [I 2022-10-26 13:47:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 880/2304 [I 2022-10-26 13:47:16] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:11:51 [I 2022-10-26 13:47:16] ...trap/engines/engine.py.229: process: 0.50329 | load: 0.00253 [I 2022-10-26 13:47:16] ...trap/engines/engine.py.230: loss: 0.22044 [I 2022-10-26 13:47:16] ...optimizers/trijoint.py.151: total_norm: 4.114009 [I 2022-10-26 13:47:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 890/2304 [I 2022-10-26 13:47:21] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:11:40 [I 2022-10-26 13:47:21] ...trap/engines/engine.py.229: process: 0.49037 | load: 0.00247 [I 2022-10-26 13:47:21] ...trap/engines/engine.py.230: loss: 0.17177 [I 2022-10-26 13:47:21] ...optimizers/trijoint.py.151: total_norm: 2.822616 [I 2022-10-26 13:47:26] ...trap/engines/engine.py.224: train: epoch 11 | batch 900/2304 [I 2022-10-26 13:47:26] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:11:34 [I 2022-10-26 13:47:26] ...trap/engines/engine.py.229: process: 0.50305 | load: 0.00246 [I 2022-10-26 13:47:26] ...trap/engines/engine.py.230: loss: 0.26811 [I 2022-10-26 13:47:26] ...optimizers/trijoint.py.151: total_norm: 3.210681 [I 2022-10-26 13:47:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 910/2304 [I 2022-10-26 13:47:31] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:11:30 [I 2022-10-26 13:47:31] ...trap/engines/engine.py.229: process: 0.48867 | load: 0.00247 [I 2022-10-26 13:47:31] ...trap/engines/engine.py.230: loss: 0.23426 [I 2022-10-26 13:47:31] ...optimizers/trijoint.py.151: total_norm: 2.786200 [I 2022-10-26 13:47:36] ...trap/engines/engine.py.224: train: epoch 11 | batch 920/2304 [I 2022-10-26 13:47:36] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:11:30 [I 2022-10-26 13:47:36] ...trap/engines/engine.py.229: process: 0.50387 | load: 0.00243 [I 2022-10-26 13:47:36] ...trap/engines/engine.py.230: loss: 0.19053 [I 2022-10-26 13:47:36] ...optimizers/trijoint.py.151: total_norm: 2.758003 [I 2022-10-26 13:47:41] ...trap/engines/engine.py.224: train: epoch 11 | batch 930/2304 [I 2022-10-26 13:47:41] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:11:25 [I 2022-10-26 13:47:41] ...trap/engines/engine.py.229: process: 0.49861 | load: 0.00243 [I 2022-10-26 13:47:41] ...trap/engines/engine.py.230: loss: 0.18228 [I 2022-10-26 13:47:41] ...optimizers/trijoint.py.151: total_norm: 2.251020 [I 2022-10-26 13:47:46] ...trap/engines/engine.py.224: train: epoch 11 | batch 940/2304 [I 2022-10-26 13:47:46] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:11:31 [I 2022-10-26 13:47:46] ...trap/engines/engine.py.229: process: 0.51106 | load: 0.00246 [I 2022-10-26 13:47:46] ...trap/engines/engine.py.230: loss: 0.24378 [I 2022-10-26 13:47:46] ...optimizers/trijoint.py.151: total_norm: 4.808314 [I 2022-10-26 13:47:51] ...trap/engines/engine.py.224: train: epoch 11 | batch 950/2304 [I 2022-10-26 13:47:51] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:11:36 [I 2022-10-26 13:47:51] ...trap/engines/engine.py.229: process: 0.48997 | load: 0.00277 [I 2022-10-26 13:47:51] ...trap/engines/engine.py.230: loss: 0.40036 [I 2022-10-26 13:47:51] ...optimizers/trijoint.py.151: total_norm: 4.432539 [I 2022-10-26 13:47:56] ...trap/engines/engine.py.224: train: epoch 11 | batch 960/2304 [I 2022-10-26 13:47:56] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:11:10 [I 2022-10-26 13:47:56] ...trap/engines/engine.py.229: process: 0.49713 | load: 0.00255 [I 2022-10-26 13:47:56] ...trap/engines/engine.py.230: loss: 0.15750 [I 2022-10-26 13:47:56] ...optimizers/trijoint.py.151: total_norm: 3.904201 [I 2022-10-26 13:48:01] ...trap/engines/engine.py.224: train: epoch 11 | batch 970/2304 [I 2022-10-26 13:48:01] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:11:01 [I 2022-10-26 13:48:01] ...trap/engines/engine.py.229: process: 0.49598 | load: 0.00249 [I 2022-10-26 13:48:01] ...trap/engines/engine.py.230: loss: 0.22819 [I 2022-10-26 13:48:01] ...optimizers/trijoint.py.151: total_norm: 3.342210 [I 2022-10-26 13:48:06] ...trap/engines/engine.py.224: train: epoch 11 | batch 980/2304 [I 2022-10-26 13:48:06] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:10:52 [I 2022-10-26 13:48:06] ...trap/engines/engine.py.229: process: 0.50440 | load: 0.00250 [I 2022-10-26 13:48:06] ...trap/engines/engine.py.230: loss: 0.22814 [I 2022-10-26 13:48:06] ...optimizers/trijoint.py.151: total_norm: 3.311369 [I 2022-10-26 13:48:11] ...trap/engines/engine.py.224: train: epoch 11 | batch 990/2304 [I 2022-10-26 13:48:11] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:11:07 [I 2022-10-26 13:48:11] ...trap/engines/engine.py.229: process: 0.51591 | load: 0.00248 [I 2022-10-26 13:48:11] ...trap/engines/engine.py.230: loss: 0.24700 [I 2022-10-26 13:48:11] ...optimizers/trijoint.py.151: total_norm: 3.271791 [I 2022-10-26 13:48:16] ...trap/engines/engine.py.224: train: epoch 11 | batch 1000/2304 [I 2022-10-26 13:48:16] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:11:10 [I 2022-10-26 13:48:16] ...trap/engines/engine.py.229: process: 0.50961 | load: 0.00245 [I 2022-10-26 13:48:16] ...trap/engines/engine.py.230: loss: 0.29097 [I 2022-10-26 13:48:16] ...optimizers/trijoint.py.151: total_norm: 3.452050 [I 2022-10-26 13:48:21] ...trap/engines/engine.py.224: train: epoch 11 | batch 1010/2304 [I 2022-10-26 13:48:21] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:11:06 [I 2022-10-26 13:48:21] ...trap/engines/engine.py.229: process: 0.51041 | load: 0.00250 [I 2022-10-26 13:48:21] ...trap/engines/engine.py.230: loss: 0.19070 [I 2022-10-26 13:48:21] ...optimizers/trijoint.py.151: total_norm: 3.137789 [I 2022-10-26 13:48:27] ...trap/engines/engine.py.224: train: epoch 11 | batch 1020/2304 [I 2022-10-26 13:48:27] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:11:00 [I 2022-10-26 13:48:27] ...trap/engines/engine.py.229: process: 0.52576 | load: 0.00251 [I 2022-10-26 13:48:27] ...trap/engines/engine.py.230: loss: 0.25918 [I 2022-10-26 13:48:27] ...optimizers/trijoint.py.151: total_norm: 3.831658 [I 2022-10-26 13:48:32] ...trap/engines/engine.py.224: train: epoch 11 | batch 1030/2304 [I 2022-10-26 13:48:32] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:10:50 [I 2022-10-26 13:48:32] ...trap/engines/engine.py.229: process: 0.51210 | load: 0.00259 [I 2022-10-26 13:48:32] ...trap/engines/engine.py.230: loss: 0.29938 [I 2022-10-26 13:48:32] ...optimizers/trijoint.py.151: total_norm: 3.468181 [I 2022-10-26 13:48:37] ...trap/engines/engine.py.224: train: epoch 11 | batch 1040/2304 [I 2022-10-26 13:48:37] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:10:28 [I 2022-10-26 13:48:37] ...trap/engines/engine.py.229: process: 0.49591 | load: 0.00248 [I 2022-10-26 13:48:37] ...trap/engines/engine.py.230: loss: 0.20922 [I 2022-10-26 13:48:37] ...optimizers/trijoint.py.151: total_norm: 2.313092 [I 2022-10-26 13:48:42] ...trap/engines/engine.py.224: train: epoch 11 | batch 1050/2304 [I 2022-10-26 13:48:42] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:10:27 [I 2022-10-26 13:48:42] ...trap/engines/engine.py.229: process: 0.50031 | load: 0.00248 [I 2022-10-26 13:48:42] ...trap/engines/engine.py.230: loss: 0.22166 [I 2022-10-26 13:48:42] ...optimizers/trijoint.py.151: total_norm: 3.418247 [I 2022-10-26 13:48:47] ...trap/engines/engine.py.224: train: epoch 11 | batch 1060/2304 [I 2022-10-26 13:48:47] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:10:19 [I 2022-10-26 13:48:47] ...trap/engines/engine.py.229: process: 0.50036 | load: 0.00244 [I 2022-10-26 13:48:47] ...trap/engines/engine.py.230: loss: 0.21826 [I 2022-10-26 13:48:47] ...optimizers/trijoint.py.151: total_norm: 3.491417 [I 2022-10-26 13:48:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 1070/2304 [I 2022-10-26 13:48:52] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:10:35 [I 2022-10-26 13:48:52] ...trap/engines/engine.py.229: process: 0.50106 | load: 0.00245 [I 2022-10-26 13:48:52] ...trap/engines/engine.py.230: loss: 0.19497 [I 2022-10-26 13:48:52] ...optimizers/trijoint.py.151: total_norm: 2.702029 [I 2022-10-26 13:48:57] ...trap/engines/engine.py.224: train: epoch 11 | batch 1080/2304 [I 2022-10-26 13:48:57] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:10:16 [I 2022-10-26 13:48:57] ...trap/engines/engine.py.229: process: 0.50187 | load: 0.00248 [I 2022-10-26 13:48:57] ...trap/engines/engine.py.230: loss: 0.23325 [I 2022-10-26 13:48:57] ...optimizers/trijoint.py.151: total_norm: 4.121771 [I 2022-10-26 13:49:02] ...trap/engines/engine.py.224: train: epoch 11 | batch 1090/2304 [I 2022-10-26 13:49:02] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:10:10 [I 2022-10-26 13:49:02] ...trap/engines/engine.py.229: process: 0.50986 | load: 0.00244 [I 2022-10-26 13:49:02] ...trap/engines/engine.py.230: loss: 0.18751 [I 2022-10-26 13:49:02] ...optimizers/trijoint.py.151: total_norm: 2.808929 [I 2022-10-26 13:49:07] ...trap/engines/engine.py.224: train: epoch 11 | batch 1100/2304 [I 2022-10-26 13:49:07] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:09:53 [I 2022-10-26 13:49:07] ...trap/engines/engine.py.229: process: 0.48505 | load: 0.00254 [I 2022-10-26 13:49:07] ...trap/engines/engine.py.230: loss: 0.26981 [I 2022-10-26 13:49:07] ...optimizers/trijoint.py.151: total_norm: 3.049768 [I 2022-10-26 13:49:12] ...trap/engines/engine.py.224: train: epoch 11 | batch 1110/2304 [I 2022-10-26 13:49:12] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:09:51 [I 2022-10-26 13:49:12] ...trap/engines/engine.py.229: process: 0.50410 | load: 0.00250 [I 2022-10-26 13:49:12] ...trap/engines/engine.py.230: loss: 0.31242 [I 2022-10-26 13:49:12] ...optimizers/trijoint.py.151: total_norm: 3.681480 [I 2022-10-26 13:49:17] ...trap/engines/engine.py.224: train: epoch 11 | batch 1120/2304 [I 2022-10-26 13:49:17] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:09:47 [I 2022-10-26 13:49:17] ...trap/engines/engine.py.229: process: 0.49804 | load: 0.00267 [I 2022-10-26 13:49:17] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-26 13:49:17] ...optimizers/trijoint.py.151: total_norm: 2.494497 [I 2022-10-26 13:49:22] ...trap/engines/engine.py.224: train: epoch 11 | batch 1130/2304 [I 2022-10-26 13:49:22] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:09:44 [I 2022-10-26 13:49:22] ...trap/engines/engine.py.229: process: 0.49114 | load: 0.00246 [I 2022-10-26 13:49:22] ...trap/engines/engine.py.230: loss: 0.26610 [I 2022-10-26 13:49:22] ...optimizers/trijoint.py.151: total_norm: 3.094092 [I 2022-10-26 13:49:27] ...trap/engines/engine.py.224: train: epoch 11 | batch 1140/2304 [I 2022-10-26 13:49:27] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:09:43 [I 2022-10-26 13:49:27] ...trap/engines/engine.py.229: process: 0.50148 | load: 0.00245 [I 2022-10-26 13:49:27] ...trap/engines/engine.py.230: loss: 0.18875 [I 2022-10-26 13:49:27] ...optimizers/trijoint.py.151: total_norm: 3.537709 [I 2022-10-26 13:49:32] ...trap/engines/engine.py.224: train: epoch 11 | batch 1150/2304 [I 2022-10-26 13:49:32] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:09:34 [I 2022-10-26 13:49:32] ...trap/engines/engine.py.229: process: 0.48966 | load: 0.00244 [I 2022-10-26 13:49:32] ...trap/engines/engine.py.230: loss: 0.26644 [I 2022-10-26 13:49:32] ...optimizers/trijoint.py.151: total_norm: 3.127786 [I 2022-10-26 13:49:37] ...trap/engines/engine.py.224: train: epoch 11 | batch 1160/2304 [I 2022-10-26 13:49:37] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:09:29 [I 2022-10-26 13:49:37] ...trap/engines/engine.py.229: process: 0.49262 | load: 0.00243 [I 2022-10-26 13:49:37] ...trap/engines/engine.py.230: loss: 0.24506 [I 2022-10-26 13:49:37] ...optimizers/trijoint.py.151: total_norm: 3.122533 [I 2022-10-26 13:49:42] ...trap/engines/engine.py.224: train: epoch 11 | batch 1170/2304 [I 2022-10-26 13:49:42] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:09:28 [I 2022-10-26 13:49:42] ...trap/engines/engine.py.229: process: 0.51369 | load: 0.00243 [I 2022-10-26 13:49:42] ...trap/engines/engine.py.230: loss: 0.14248 [I 2022-10-26 13:49:42] ...optimizers/trijoint.py.151: total_norm: 2.420937 [I 2022-10-26 13:49:47] ...trap/engines/engine.py.224: train: epoch 11 | batch 1180/2304 [I 2022-10-26 13:49:47] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:09:25 [I 2022-10-26 13:49:47] ...trap/engines/engine.py.229: process: 0.49543 | load: 0.00248 [I 2022-10-26 13:49:47] ...trap/engines/engine.py.230: loss: 0.24039 [I 2022-10-26 13:49:47] ...optimizers/trijoint.py.151: total_norm: 3.152081 [I 2022-10-26 13:49:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 1190/2304 [I 2022-10-26 13:49:52] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:09:16 [I 2022-10-26 13:49:52] ...trap/engines/engine.py.229: process: 0.50431 | load: 0.00289 [I 2022-10-26 13:49:52] ...trap/engines/engine.py.230: loss: 0.15406 [I 2022-10-26 13:49:52] ...optimizers/trijoint.py.151: total_norm: 2.783219 [I 2022-10-26 13:49:57] ...trap/engines/engine.py.224: train: epoch 11 | batch 1200/2304 [I 2022-10-26 13:49:57] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:09:13 [I 2022-10-26 13:49:57] ...trap/engines/engine.py.229: process: 0.48621 | load: 0.00243 [I 2022-10-26 13:49:57] ...trap/engines/engine.py.230: loss: 0.22110 [I 2022-10-26 13:49:57] ...optimizers/trijoint.py.151: total_norm: 3.080196 [I 2022-10-26 13:50:02] ...trap/engines/engine.py.224: train: epoch 11 | batch 1210/2304 [I 2022-10-26 13:50:02] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:09:08 [I 2022-10-26 13:50:02] ...trap/engines/engine.py.229: process: 0.50195 | load: 0.00252 [I 2022-10-26 13:50:02] ...trap/engines/engine.py.230: loss: 0.17935 [I 2022-10-26 13:50:02] ...optimizers/trijoint.py.151: total_norm: 3.204889 [I 2022-10-26 13:50:07] ...trap/engines/engine.py.224: train: epoch 11 | batch 1220/2304 [I 2022-10-26 13:50:07] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:09:01 [I 2022-10-26 13:50:07] ...trap/engines/engine.py.229: process: 0.49878 | load: 0.00251 [I 2022-10-26 13:50:07] ...trap/engines/engine.py.230: loss: 0.21823 [I 2022-10-26 13:50:07] ...optimizers/trijoint.py.151: total_norm: 2.545116 [I 2022-10-26 13:50:12] ...trap/engines/engine.py.224: train: epoch 11 | batch 1230/2304 [I 2022-10-26 13:50:12] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:08:54 [I 2022-10-26 13:50:12] ...trap/engines/engine.py.229: process: 0.49930 | load: 0.00245 [I 2022-10-26 13:50:12] ...trap/engines/engine.py.230: loss: 0.17750 [I 2022-10-26 13:50:12] ...optimizers/trijoint.py.151: total_norm: 3.344986 [I 2022-10-26 13:50:17] ...trap/engines/engine.py.224: train: epoch 11 | batch 1240/2304 [I 2022-10-26 13:50:17] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:08:46 [I 2022-10-26 13:50:17] ...trap/engines/engine.py.229: process: 0.49268 | load: 0.00245 [I 2022-10-26 13:50:17] ...trap/engines/engine.py.230: loss: 0.21283 [I 2022-10-26 13:50:17] ...optimizers/trijoint.py.151: total_norm: 3.415759 [I 2022-10-26 13:50:22] ...trap/engines/engine.py.224: train: epoch 11 | batch 1250/2304 [I 2022-10-26 13:50:22] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:08:40 [I 2022-10-26 13:50:22] ...trap/engines/engine.py.229: process: 0.48253 | load: 0.00245 [I 2022-10-26 13:50:22] ...trap/engines/engine.py.230: loss: 0.28386 [I 2022-10-26 13:50:22] ...optimizers/trijoint.py.151: total_norm: 3.070505 [I 2022-10-26 13:50:27] ...trap/engines/engine.py.224: train: epoch 11 | batch 1260/2304 [I 2022-10-26 13:50:27] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:08:37 [I 2022-10-26 13:50:27] ...trap/engines/engine.py.229: process: 0.48669 | load: 0.00244 [I 2022-10-26 13:50:27] ...trap/engines/engine.py.230: loss: 0.19336 [I 2022-10-26 13:50:27] ...optimizers/trijoint.py.151: total_norm: 3.022744 [I 2022-10-26 13:50:32] ...trap/engines/engine.py.224: train: epoch 11 | batch 1270/2304 [I 2022-10-26 13:50:32] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:08:37 [I 2022-10-26 13:50:32] ...trap/engines/engine.py.229: process: 0.50129 | load: 0.00243 [I 2022-10-26 13:50:32] ...trap/engines/engine.py.230: loss: 0.20023 [I 2022-10-26 13:50:32] ...optimizers/trijoint.py.151: total_norm: 3.027029 [I 2022-10-26 13:50:37] ...trap/engines/engine.py.224: train: epoch 11 | batch 1280/2304 [I 2022-10-26 13:50:37] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:08:30 [I 2022-10-26 13:50:37] ...trap/engines/engine.py.229: process: 0.49806 | load: 0.00244 [I 2022-10-26 13:50:37] ...trap/engines/engine.py.230: loss: 0.15337 [I 2022-10-26 13:50:37] ...optimizers/trijoint.py.151: total_norm: 2.745563 [I 2022-10-26 13:50:42] ...trap/engines/engine.py.224: train: epoch 11 | batch 1290/2304 [I 2022-10-26 13:50:42] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:08:21 [I 2022-10-26 13:50:42] ...trap/engines/engine.py.229: process: 0.49310 | load: 0.00254 [I 2022-10-26 13:50:42] ...trap/engines/engine.py.230: loss: 0.18417 [I 2022-10-26 13:50:42] ...optimizers/trijoint.py.151: total_norm: 2.583630 [I 2022-10-26 13:50:47] ...trap/engines/engine.py.224: train: epoch 11 | batch 1300/2304 [I 2022-10-26 13:50:47] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:08:19 [I 2022-10-26 13:50:47] ...trap/engines/engine.py.229: process: 0.50138 | load: 0.00246 [I 2022-10-26 13:50:47] ...trap/engines/engine.py.230: loss: 0.26756 [I 2022-10-26 13:50:47] ...optimizers/trijoint.py.151: total_norm: 3.503503 [I 2022-10-26 13:50:52] ...trap/engines/engine.py.224: train: epoch 11 | batch 1310/2304 [I 2022-10-26 13:50:52] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:08:21 [I 2022-10-26 13:50:52] ...trap/engines/engine.py.229: process: 0.50720 | load: 0.00261 [I 2022-10-26 13:50:52] ...trap/engines/engine.py.230: loss: 0.27765 [I 2022-10-26 13:50:52] ...optimizers/trijoint.py.151: total_norm: 2.717842 [I 2022-10-26 13:50:57] ...trap/engines/engine.py.224: train: epoch 11 | batch 1320/2304 [I 2022-10-26 13:50:57] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:08:10 [I 2022-10-26 13:50:57] ...trap/engines/engine.py.229: process: 0.49645 | load: 0.00253 [I 2022-10-26 13:50:57] ...trap/engines/engine.py.230: loss: 0.15085 [I 2022-10-26 13:50:57] ...optimizers/trijoint.py.151: total_norm: 2.857709 [I 2022-10-26 13:51:02] ...trap/engines/engine.py.224: train: epoch 11 | batch 1330/2304 [I 2022-10-26 13:51:02] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:08:05 [I 2022-10-26 13:51:02] ...trap/engines/engine.py.229: process: 0.50124 | load: 0.00245 [I 2022-10-26 13:51:02] ...trap/engines/engine.py.230: loss: 0.20298 [I 2022-10-26 13:51:02] ...optimizers/trijoint.py.151: total_norm: 2.957597 [I 2022-10-26 13:51:07] ...trap/engines/engine.py.224: train: epoch 11 | batch 1340/2304 [I 2022-10-26 13:51:07] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:07:59 [I 2022-10-26 13:51:07] ...trap/engines/engine.py.229: process: 0.49660 | load: 0.00245 [I 2022-10-26 13:51:07] ...trap/engines/engine.py.230: loss: 0.26886 [I 2022-10-26 13:51:07] ...optimizers/trijoint.py.151: total_norm: 4.164863 [I 2022-10-26 13:51:12] ...trap/engines/engine.py.224: train: epoch 11 | batch 1350/2304 [I 2022-10-26 13:51:12] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:07:58 [I 2022-10-26 13:51:12] ...trap/engines/engine.py.229: process: 0.50626 | load: 0.00244 [I 2022-10-26 13:51:12] ...trap/engines/engine.py.230: loss: 0.17347 [I 2022-10-26 13:51:12] ...optimizers/trijoint.py.151: total_norm: 3.391325 [I 2022-10-26 13:51:17] ...trap/engines/engine.py.224: train: epoch 11 | batch 1360/2304 [I 2022-10-26 13:51:17] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:07:51 [I 2022-10-26 13:51:17] ...trap/engines/engine.py.229: process: 0.50503 | load: 0.00244 [I 2022-10-26 13:51:17] ...trap/engines/engine.py.230: loss: 0.28550 [I 2022-10-26 13:51:17] ...optimizers/trijoint.py.151: total_norm: 4.202212 [I 2022-10-26 13:51:22] ...trap/engines/engine.py.224: train: epoch 11 | batch 1370/2304 [I 2022-10-26 13:51:22] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:07:50 [I 2022-10-26 13:51:22] ...trap/engines/engine.py.229: process: 0.49622 | load: 0.00250 [I 2022-10-26 13:51:22] ...trap/engines/engine.py.230: loss: 0.21918 [I 2022-10-26 13:51:22] ...optimizers/trijoint.py.151: total_norm: 3.069203 [I 2022-10-26 13:51:49] ...trap/engines/engine.py.224: train: epoch 11 | batch 1380/2304 [I 2022-10-26 13:51:49] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:43:08 [I 2022-10-26 13:51:49] ...trap/engines/engine.py.229: process: 0.50534 | load: 0.00236 [I 2022-10-26 13:51:49] ...trap/engines/engine.py.230: loss: 0.20387 [I 2022-10-26 13:51:49] ...optimizers/trijoint.py.151: total_norm: 3.206012 [I 2022-10-26 13:51:54] ...trap/engines/engine.py.224: train: epoch 11 | batch 1390/2304 [I 2022-10-26 13:51:54] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:11:21 [I 2022-10-26 13:51:54] ...trap/engines/engine.py.229: process: 0.48938 | load: 0.00237 [I 2022-10-26 13:51:54] ...trap/engines/engine.py.230: loss: 0.25645 [I 2022-10-26 13:51:54] ...optimizers/trijoint.py.151: total_norm: 3.435305 [I 2022-10-26 13:51:59] ...trap/engines/engine.py.224: train: epoch 11 | batch 1400/2304 [I 2022-10-26 13:51:59] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:07:53 [I 2022-10-26 13:51:59] ...trap/engines/engine.py.229: process: 0.49323 | load: 0.00237 [I 2022-10-26 13:51:59] ...trap/engines/engine.py.230: loss: 0.29781 [I 2022-10-26 13:51:59] ...optimizers/trijoint.py.151: total_norm: 4.326341 [I 2022-10-26 13:52:04] ...trap/engines/engine.py.224: train: epoch 11 | batch 1410/2304 [I 2022-10-26 13:52:04] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:07:27 [I 2022-10-26 13:52:04] ...trap/engines/engine.py.229: process: 0.49833 | load: 0.00237 [I 2022-10-26 13:52:04] ...trap/engines/engine.py.230: loss: 0.14913 [I 2022-10-26 13:52:04] ...optimizers/trijoint.py.151: total_norm: 3.432928 [I 2022-10-26 13:52:09] ...trap/engines/engine.py.224: train: epoch 11 | batch 1420/2304 [I 2022-10-26 13:52:09] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:07:21 [I 2022-10-26 13:52:09] ...trap/engines/engine.py.229: process: 0.50221 | load: 0.00253 [I 2022-10-26 13:52:09] ...trap/engines/engine.py.230: loss: 0.31206 [I 2022-10-26 13:52:09] ...optimizers/trijoint.py.151: total_norm: 3.850266 [I 2022-10-26 13:52:14] ...trap/engines/engine.py.224: train: epoch 11 | batch 1430/2304 [I 2022-10-26 13:52:14] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:07:17 [I 2022-10-26 13:52:14] ...trap/engines/engine.py.229: process: 0.50228 | load: 0.00247 [I 2022-10-26 13:52:14] ...trap/engines/engine.py.230: loss: 0.23388 [I 2022-10-26 13:52:14] ...optimizers/trijoint.py.151: total_norm: 3.302122 [I 2022-10-26 13:52:19] ...trap/engines/engine.py.224: train: epoch 11 | batch 1440/2304 [I 2022-10-26 13:52:19] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:07:10 [I 2022-10-26 13:52:19] ...trap/engines/engine.py.229: process: 0.49835 | load: 0.00243 [I 2022-10-26 13:52:19] ...trap/engines/engine.py.230: loss: 0.33693 [I 2022-10-26 13:52:19] ...optimizers/trijoint.py.151: total_norm: 3.696395 [I 2022-10-26 13:52:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 1450/2304 [I 2022-10-26 13:52:24] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:07:05 [I 2022-10-26 13:52:24] ...trap/engines/engine.py.229: process: 0.49232 | load: 0.00240 [I 2022-10-26 13:52:24] ...trap/engines/engine.py.230: loss: 0.29111 [I 2022-10-26 13:52:24] ...optimizers/trijoint.py.151: total_norm: 3.082778 [I 2022-10-26 13:52:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 1460/2304 [I 2022-10-26 13:52:29] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:06:59 [I 2022-10-26 13:52:29] ...trap/engines/engine.py.229: process: 0.50068 | load: 0.00238 [I 2022-10-26 13:52:29] ...trap/engines/engine.py.230: loss: 0.30588 [I 2022-10-26 13:52:29] ...optimizers/trijoint.py.151: total_norm: 4.389391 [I 2022-10-26 13:52:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 1470/2304 [I 2022-10-26 13:52:34] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:06:54 [I 2022-10-26 13:52:34] ...trap/engines/engine.py.229: process: 0.50051 | load: 0.00241 [I 2022-10-26 13:52:34] ...trap/engines/engine.py.230: loss: 0.21804 [I 2022-10-26 13:52:34] ...optimizers/trijoint.py.151: total_norm: 2.860246 [I 2022-10-26 13:52:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 1480/2304 [I 2022-10-26 13:52:39] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:06:49 [I 2022-10-26 13:52:39] ...trap/engines/engine.py.229: process: 0.48942 | load: 0.00242 [I 2022-10-26 13:52:39] ...trap/engines/engine.py.230: loss: 0.27539 [I 2022-10-26 13:52:39] ...optimizers/trijoint.py.151: total_norm: 3.999511 [I 2022-10-26 13:52:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 1490/2304 [I 2022-10-26 13:52:44] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:06:47 [I 2022-10-26 13:52:44] ...trap/engines/engine.py.229: process: 0.50692 | load: 0.00240 [I 2022-10-26 13:52:44] ...trap/engines/engine.py.230: loss: 0.36340 [I 2022-10-26 13:52:44] ...optimizers/trijoint.py.151: total_norm: 3.660189 [I 2022-10-26 13:52:49] ...trap/engines/engine.py.224: train: epoch 11 | batch 1500/2304 [I 2022-10-26 13:52:49] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:06:40 [I 2022-10-26 13:52:49] ...trap/engines/engine.py.229: process: 0.49445 | load: 0.00260 [I 2022-10-26 13:52:49] ...trap/engines/engine.py.230: loss: 0.23285 [I 2022-10-26 13:52:49] ...optimizers/trijoint.py.151: total_norm: 2.883924 [I 2022-10-26 13:52:54] ...trap/engines/engine.py.224: train: epoch 11 | batch 1510/2304 [I 2022-10-26 13:52:54] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:06:36 [I 2022-10-26 13:52:54] ...trap/engines/engine.py.229: process: 0.49515 | load: 0.00239 [I 2022-10-26 13:52:54] ...trap/engines/engine.py.230: loss: 0.22522 [I 2022-10-26 13:52:54] ...optimizers/trijoint.py.151: total_norm: 2.512811 [I 2022-10-26 13:52:59] ...trap/engines/engine.py.224: train: epoch 11 | batch 1520/2304 [I 2022-10-26 13:52:59] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:06:29 [I 2022-10-26 13:52:59] ...trap/engines/engine.py.229: process: 0.49452 | load: 0.00244 [I 2022-10-26 13:52:59] ...trap/engines/engine.py.230: loss: 0.15104 [I 2022-10-26 13:52:59] ...optimizers/trijoint.py.151: total_norm: 2.393932 [I 2022-10-26 13:53:04] ...trap/engines/engine.py.224: train: epoch 11 | batch 1530/2304 [I 2022-10-26 13:53:04] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:06:26 [I 2022-10-26 13:53:04] ...trap/engines/engine.py.229: process: 0.50188 | load: 0.00242 [I 2022-10-26 13:53:04] ...trap/engines/engine.py.230: loss: 0.22390 [I 2022-10-26 13:53:04] ...optimizers/trijoint.py.151: total_norm: 3.118234 [I 2022-10-26 13:53:09] ...trap/engines/engine.py.224: train: epoch 11 | batch 1540/2304 [I 2022-10-26 13:53:09] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:06:23 [I 2022-10-26 13:53:09] ...trap/engines/engine.py.229: process: 0.50221 | load: 0.00244 [I 2022-10-26 13:53:09] ...trap/engines/engine.py.230: loss: 0.22857 [I 2022-10-26 13:53:09] ...optimizers/trijoint.py.151: total_norm: 4.447808 [I 2022-10-26 13:53:14] ...trap/engines/engine.py.224: train: epoch 11 | batch 1550/2304 [I 2022-10-26 13:53:14] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:06:16 [I 2022-10-26 13:53:14] ...trap/engines/engine.py.229: process: 0.50047 | load: 0.00241 [I 2022-10-26 13:53:14] ...trap/engines/engine.py.230: loss: 0.28519 [I 2022-10-26 13:53:14] ...optimizers/trijoint.py.151: total_norm: 3.580370 [I 2022-10-26 13:53:19] ...trap/engines/engine.py.224: train: epoch 11 | batch 1560/2304 [I 2022-10-26 13:53:19] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:06:10 [I 2022-10-26 13:53:19] ...trap/engines/engine.py.229: process: 0.49567 | load: 0.00240 [I 2022-10-26 13:53:19] ...trap/engines/engine.py.230: loss: 0.30384 [I 2022-10-26 13:53:19] ...optimizers/trijoint.py.151: total_norm: 3.543781 [I 2022-10-26 13:53:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 1570/2304 [I 2022-10-26 13:53:24] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:06:07 [I 2022-10-26 13:53:24] ...trap/engines/engine.py.229: process: 0.49970 | load: 0.00241 [I 2022-10-26 13:53:24] ...trap/engines/engine.py.230: loss: 0.25002 [I 2022-10-26 13:53:24] ...optimizers/trijoint.py.151: total_norm: 2.880025 [I 2022-10-26 13:53:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 1580/2304 [I 2022-10-26 13:53:29] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:06:01 [I 2022-10-26 13:53:29] ...trap/engines/engine.py.229: process: 0.50349 | load: 0.00239 [I 2022-10-26 13:53:29] ...trap/engines/engine.py.230: loss: 0.25504 [I 2022-10-26 13:53:29] ...optimizers/trijoint.py.151: total_norm: 4.723221 [I 2022-10-26 13:53:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 1590/2304 [I 2022-10-26 13:53:34] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:05:54 [I 2022-10-26 13:53:34] ...trap/engines/engine.py.229: process: 0.49949 | load: 0.00240 [I 2022-10-26 13:53:34] ...trap/engines/engine.py.230: loss: 0.21134 [I 2022-10-26 13:53:34] ...optimizers/trijoint.py.151: total_norm: 4.619375 [I 2022-10-26 13:53:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 1600/2304 [I 2022-10-26 13:53:39] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:05:48 [I 2022-10-26 13:53:39] ...trap/engines/engine.py.229: process: 0.49726 | load: 0.00239 [I 2022-10-26 13:53:39] ...trap/engines/engine.py.230: loss: 0.17339 [I 2022-10-26 13:53:39] ...optimizers/trijoint.py.151: total_norm: 3.134191 [I 2022-10-26 13:53:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 1610/2304 [I 2022-10-26 13:53:44] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:05:43 [I 2022-10-26 13:53:44] ...trap/engines/engine.py.229: process: 0.49449 | load: 0.00242 [I 2022-10-26 13:53:44] ...trap/engines/engine.py.230: loss: 0.30164 [I 2022-10-26 13:53:44] ...optimizers/trijoint.py.151: total_norm: 3.534268 [I 2022-10-26 13:53:49] ...trap/engines/engine.py.224: train: epoch 11 | batch 1620/2304 [I 2022-10-26 13:53:49] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:05:41 [I 2022-10-26 13:53:49] ...trap/engines/engine.py.229: process: 0.51224 | load: 0.00244 [I 2022-10-26 13:53:49] ...trap/engines/engine.py.230: loss: 0.24578 [I 2022-10-26 13:53:49] ...optimizers/trijoint.py.151: total_norm: 3.392071 [I 2022-10-26 13:53:54] ...trap/engines/engine.py.224: train: epoch 11 | batch 1630/2304 [I 2022-10-26 13:53:54] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:05:33 [I 2022-10-26 13:53:54] ...trap/engines/engine.py.229: process: 0.48846 | load: 0.00241 [I 2022-10-26 13:53:54] ...trap/engines/engine.py.230: loss: 0.21146 [I 2022-10-26 13:53:54] ...optimizers/trijoint.py.151: total_norm: 2.921642 [I 2022-10-26 13:53:59] ...trap/engines/engine.py.224: train: epoch 11 | batch 1640/2304 [I 2022-10-26 13:53:59] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:05:30 [I 2022-10-26 13:53:59] ...trap/engines/engine.py.229: process: 0.50004 | load: 0.00241 [I 2022-10-26 13:53:59] ...trap/engines/engine.py.230: loss: 0.18177 [I 2022-10-26 13:53:59] ...optimizers/trijoint.py.151: total_norm: 2.773520 [I 2022-10-26 13:54:04] ...trap/engines/engine.py.224: train: epoch 11 | batch 1650/2304 [I 2022-10-26 13:54:04] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:05:25 [I 2022-10-26 13:54:04] ...trap/engines/engine.py.229: process: 0.50247 | load: 0.00240 [I 2022-10-26 13:54:04] ...trap/engines/engine.py.230: loss: 0.15879 [I 2022-10-26 13:54:04] ...optimizers/trijoint.py.151: total_norm: 2.939001 [I 2022-10-26 13:54:09] ...trap/engines/engine.py.224: train: epoch 11 | batch 1660/2304 [I 2022-10-26 13:54:09] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:05:18 [I 2022-10-26 13:54:09] ...trap/engines/engine.py.229: process: 0.49690 | load: 0.00239 [I 2022-10-26 13:54:09] ...trap/engines/engine.py.230: loss: 0.11403 [I 2022-10-26 13:54:09] ...optimizers/trijoint.py.151: total_norm: 1.902703 [I 2022-10-26 13:54:14] ...trap/engines/engine.py.224: train: epoch 11 | batch 1670/2304 [I 2022-10-26 13:54:14] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:05:16 [I 2022-10-26 13:54:14] ...trap/engines/engine.py.229: process: 0.49911 | load: 0.00250 [I 2022-10-26 13:54:14] ...trap/engines/engine.py.230: loss: 0.20117 [I 2022-10-26 13:54:14] ...optimizers/trijoint.py.151: total_norm: 3.223830 [I 2022-10-26 13:54:19] ...trap/engines/engine.py.224: train: epoch 11 | batch 1680/2304 [I 2022-10-26 13:54:19] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:05:11 [I 2022-10-26 13:54:19] ...trap/engines/engine.py.229: process: 0.48895 | load: 0.00245 [I 2022-10-26 13:54:19] ...trap/engines/engine.py.230: loss: 0.21271 [I 2022-10-26 13:54:19] ...optimizers/trijoint.py.151: total_norm: 3.107197 [I 2022-10-26 13:54:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 1690/2304 [I 2022-10-26 13:54:24] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:05:05 [I 2022-10-26 13:54:24] ...trap/engines/engine.py.229: process: 0.49813 | load: 0.00243 [I 2022-10-26 13:54:24] ...trap/engines/engine.py.230: loss: 0.33758 [I 2022-10-26 13:54:24] ...optimizers/trijoint.py.151: total_norm: 4.675270 [I 2022-10-26 13:54:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 1700/2304 [I 2022-10-26 13:54:29] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:05:00 [I 2022-10-26 13:54:29] ...trap/engines/engine.py.229: process: 0.48969 | load: 0.00243 [I 2022-10-26 13:54:29] ...trap/engines/engine.py.230: loss: 0.15735 [I 2022-10-26 13:54:29] ...optimizers/trijoint.py.151: total_norm: 2.861409 [I 2022-10-26 13:54:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 1710/2304 [I 2022-10-26 13:54:34] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:04:54 [I 2022-10-26 13:54:34] ...trap/engines/engine.py.229: process: 0.49679 | load: 0.00242 [I 2022-10-26 13:54:34] ...trap/engines/engine.py.230: loss: 0.26698 [I 2022-10-26 13:54:34] ...optimizers/trijoint.py.151: total_norm: 2.765821 [I 2022-10-26 13:54:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 1720/2304 [I 2022-10-26 13:54:39] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:04:50 [I 2022-10-26 13:54:39] ...trap/engines/engine.py.229: process: 0.49682 | load: 0.00242 [I 2022-10-26 13:54:39] ...trap/engines/engine.py.230: loss: 0.29955 [I 2022-10-26 13:54:39] ...optimizers/trijoint.py.151: total_norm: 3.666461 [I 2022-10-26 13:54:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 1730/2304 [I 2022-10-26 13:54:44] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:04:45 [I 2022-10-26 13:54:44] ...trap/engines/engine.py.229: process: 0.50934 | load: 0.00241 [I 2022-10-26 13:54:44] ...trap/engines/engine.py.230: loss: 0.21530 [I 2022-10-26 13:54:44] ...optimizers/trijoint.py.151: total_norm: 3.000430 [I 2022-10-26 13:54:49] ...trap/engines/engine.py.224: train: epoch 11 | batch 1740/2304 [I 2022-10-26 13:54:49] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:04:39 [I 2022-10-26 13:54:49] ...trap/engines/engine.py.229: process: 0.49345 | load: 0.00240 [I 2022-10-26 13:54:49] ...trap/engines/engine.py.230: loss: 0.21672 [I 2022-10-26 13:54:49] ...optimizers/trijoint.py.151: total_norm: 3.945792 [I 2022-10-26 13:54:54] ...trap/engines/engine.py.224: train: epoch 11 | batch 1750/2304 [I 2022-10-26 13:54:54] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:04:35 [I 2022-10-26 13:54:54] ...trap/engines/engine.py.229: process: 0.48945 | load: 0.00242 [I 2022-10-26 13:54:54] ...trap/engines/engine.py.230: loss: 0.23510 [I 2022-10-26 13:54:54] ...optimizers/trijoint.py.151: total_norm: 3.859240 [I 2022-10-26 13:54:59] ...trap/engines/engine.py.224: train: epoch 11 | batch 1760/2304 [I 2022-10-26 13:54:59] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:04:31 [I 2022-10-26 13:54:59] ...trap/engines/engine.py.229: process: 0.50530 | load: 0.00243 [I 2022-10-26 13:54:59] ...trap/engines/engine.py.230: loss: 0.21872 [I 2022-10-26 13:54:59] ...optimizers/trijoint.py.151: total_norm: 2.952486 [I 2022-10-26 13:55:04] ...trap/engines/engine.py.224: train: epoch 11 | batch 1770/2304 [I 2022-10-26 13:55:04] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:04:24 [I 2022-10-26 13:55:04] ...trap/engines/engine.py.229: process: 0.48822 | load: 0.00243 [I 2022-10-26 13:55:04] ...trap/engines/engine.py.230: loss: 0.21536 [I 2022-10-26 13:55:04] ...optimizers/trijoint.py.151: total_norm: 3.920084 [I 2022-10-26 13:55:09] ...trap/engines/engine.py.224: train: epoch 11 | batch 1780/2304 [I 2022-10-26 13:55:09] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:04:28 [I 2022-10-26 13:55:09] ...trap/engines/engine.py.229: process: 0.51025 | load: 0.00238 [I 2022-10-26 13:55:09] ...trap/engines/engine.py.230: loss: 0.21817 [I 2022-10-26 13:55:09] ...optimizers/trijoint.py.151: total_norm: 3.162118 [I 2022-10-26 13:55:14] ...trap/engines/engine.py.224: train: epoch 11 | batch 1790/2304 [I 2022-10-26 13:55:14] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:04:18 [I 2022-10-26 13:55:14] ...trap/engines/engine.py.229: process: 0.50273 | load: 0.00241 [I 2022-10-26 13:55:14] ...trap/engines/engine.py.230: loss: 0.32256 [I 2022-10-26 13:55:14] ...optimizers/trijoint.py.151: total_norm: 4.245853 [I 2022-10-26 13:55:19] ...trap/engines/engine.py.224: train: epoch 11 | batch 1800/2304 [I 2022-10-26 13:55:19] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:04:15 [I 2022-10-26 13:55:19] ...trap/engines/engine.py.229: process: 0.48092 | load: 0.00239 [I 2022-10-26 13:55:19] ...trap/engines/engine.py.230: loss: 0.29885 [I 2022-10-26 13:55:19] ...optimizers/trijoint.py.151: total_norm: 3.808638 [I 2022-10-26 13:55:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 1810/2304 [I 2022-10-26 13:55:24] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:04:08 [I 2022-10-26 13:55:24] ...trap/engines/engine.py.229: process: 0.50920 | load: 0.00239 [I 2022-10-26 13:55:24] ...trap/engines/engine.py.230: loss: 0.22560 [I 2022-10-26 13:55:24] ...optimizers/trijoint.py.151: total_norm: 2.664823 [I 2022-10-26 13:55:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 1820/2304 [I 2022-10-26 13:55:29] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:04:01 [I 2022-10-26 13:55:29] ...trap/engines/engine.py.229: process: 0.49092 | load: 0.00242 [I 2022-10-26 13:55:29] ...trap/engines/engine.py.230: loss: 0.20250 [I 2022-10-26 13:55:29] ...optimizers/trijoint.py.151: total_norm: 3.195837 [I 2022-10-26 13:55:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 1830/2304 [I 2022-10-26 13:55:34] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:03:57 [I 2022-10-26 13:55:34] ...trap/engines/engine.py.229: process: 0.50315 | load: 0.00241 [I 2022-10-26 13:55:34] ...trap/engines/engine.py.230: loss: 0.25342 [I 2022-10-26 13:55:34] ...optimizers/trijoint.py.151: total_norm: 3.264430 [I 2022-10-26 13:55:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 1840/2304 [I 2022-10-26 13:55:39] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:03:51 [I 2022-10-26 13:55:39] ...trap/engines/engine.py.229: process: 0.48677 | load: 0.00243 [I 2022-10-26 13:55:39] ...trap/engines/engine.py.230: loss: 0.19954 [I 2022-10-26 13:55:39] ...optimizers/trijoint.py.151: total_norm: 3.323783 [I 2022-10-26 13:55:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 1850/2304 [I 2022-10-26 13:55:44] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:03:46 [I 2022-10-26 13:55:44] ...trap/engines/engine.py.229: process: 0.50462 | load: 0.00245 [I 2022-10-26 13:55:44] ...trap/engines/engine.py.230: loss: 0.18903 [I 2022-10-26 13:55:44] ...optimizers/trijoint.py.151: total_norm: 2.903220 [I 2022-10-26 13:55:49] ...trap/engines/engine.py.224: train: epoch 11 | batch 1860/2304 [I 2022-10-26 13:55:49] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:03:42 [I 2022-10-26 13:55:49] ...trap/engines/engine.py.229: process: 0.49630 | load: 0.00251 [I 2022-10-26 13:55:49] ...trap/engines/engine.py.230: loss: 0.19194 [I 2022-10-26 13:55:49] ...optimizers/trijoint.py.151: total_norm: 3.222817 [I 2022-10-26 13:55:54] ...trap/engines/engine.py.224: train: epoch 11 | batch 1870/2304 [I 2022-10-26 13:55:54] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:03:36 [I 2022-10-26 13:55:54] ...trap/engines/engine.py.229: process: 0.48661 | load: 0.00245 [I 2022-10-26 13:55:54] ...trap/engines/engine.py.230: loss: 0.24655 [I 2022-10-26 13:55:54] ...optimizers/trijoint.py.151: total_norm: 3.342916 [I 2022-10-26 13:56:00] ...trap/engines/engine.py.224: train: epoch 11 | batch 1880/2304 [I 2022-10-26 13:56:00] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:03:31 [I 2022-10-26 13:56:00] ...trap/engines/engine.py.229: process: 0.49209 | load: 0.00244 [I 2022-10-26 13:56:00] ...trap/engines/engine.py.230: loss: 0.25530 [I 2022-10-26 13:56:00] ...optimizers/trijoint.py.151: total_norm: 2.938110 [I 2022-10-26 13:56:04] ...trap/engines/engine.py.224: train: epoch 11 | batch 1890/2304 [I 2022-10-26 13:56:04] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:03:25 [I 2022-10-26 13:56:04] ...trap/engines/engine.py.229: process: 0.48503 | load: 0.00244 [I 2022-10-26 13:56:04] ...trap/engines/engine.py.230: loss: 0.18850 [I 2022-10-26 13:56:04] ...optimizers/trijoint.py.151: total_norm: 4.303170 [I 2022-10-26 13:56:09] ...trap/engines/engine.py.224: train: epoch 11 | batch 1900/2304 [I 2022-10-26 13:56:09] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:03:21 [I 2022-10-26 13:56:09] ...trap/engines/engine.py.229: process: 0.49344 | load: 0.00240 [I 2022-10-26 13:56:09] ...trap/engines/engine.py.230: loss: 0.23059 [I 2022-10-26 13:56:09] ...optimizers/trijoint.py.151: total_norm: 3.363728 [I 2022-10-26 13:56:14] ...trap/engines/engine.py.224: train: epoch 11 | batch 1910/2304 [I 2022-10-26 13:56:14] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:03:15 [I 2022-10-26 13:56:14] ...trap/engines/engine.py.229: process: 0.49060 | load: 0.00240 [I 2022-10-26 13:56:14] ...trap/engines/engine.py.230: loss: 0.21738 [I 2022-10-26 13:56:14] ...optimizers/trijoint.py.151: total_norm: 2.504787 [I 2022-10-26 13:56:19] ...trap/engines/engine.py.224: train: epoch 11 | batch 1920/2304 [I 2022-10-26 13:56:19] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:03:10 [I 2022-10-26 13:56:19] ...trap/engines/engine.py.229: process: 0.48936 | load: 0.00244 [I 2022-10-26 13:56:19] ...trap/engines/engine.py.230: loss: 0.24207 [I 2022-10-26 13:56:19] ...optimizers/trijoint.py.151: total_norm: 3.818960 [I 2022-10-26 13:56:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 1930/2304 [I 2022-10-26 13:56:24] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:03:05 [I 2022-10-26 13:56:24] ...trap/engines/engine.py.229: process: 0.48833 | load: 0.00249 [I 2022-10-26 13:56:24] ...trap/engines/engine.py.230: loss: 0.22822 [I 2022-10-26 13:56:24] ...optimizers/trijoint.py.151: total_norm: 3.737065 [I 2022-10-26 13:56:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 1940/2304 [I 2022-10-26 13:56:29] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:03:00 [I 2022-10-26 13:56:29] ...trap/engines/engine.py.229: process: 0.48721 | load: 0.00243 [I 2022-10-26 13:56:29] ...trap/engines/engine.py.230: loss: 0.20838 [I 2022-10-26 13:56:29] ...optimizers/trijoint.py.151: total_norm: 3.644217 [I 2022-10-26 13:56:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 1950/2304 [I 2022-10-26 13:56:34] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:03:00 [I 2022-10-26 13:56:34] ...trap/engines/engine.py.229: process: 0.51295 | load: 0.00242 [I 2022-10-26 13:56:34] ...trap/engines/engine.py.230: loss: 0.31547 [I 2022-10-26 13:56:34] ...optimizers/trijoint.py.151: total_norm: 3.485965 [I 2022-10-26 13:56:40] ...trap/engines/engine.py.224: train: epoch 11 | batch 1960/2304 [I 2022-10-26 13:56:40] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:02:52 [I 2022-10-26 13:56:40] ...trap/engines/engine.py.229: process: 0.51008 | load: 0.00245 [I 2022-10-26 13:56:40] ...trap/engines/engine.py.230: loss: 0.21215 [I 2022-10-26 13:56:40] ...optimizers/trijoint.py.151: total_norm: 3.122041 [I 2022-10-26 13:56:45] ...trap/engines/engine.py.224: train: epoch 11 | batch 1970/2304 [I 2022-10-26 13:56:45] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:02:47 [I 2022-10-26 13:56:45] ...trap/engines/engine.py.229: process: 0.49527 | load: 0.00244 [I 2022-10-26 13:56:45] ...trap/engines/engine.py.230: loss: 0.31396 [I 2022-10-26 13:56:45] ...optimizers/trijoint.py.151: total_norm: 3.713537 [I 2022-10-26 13:56:50] ...trap/engines/engine.py.224: train: epoch 11 | batch 1980/2304 [I 2022-10-26 13:56:50] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:02:42 [I 2022-10-26 13:56:50] ...trap/engines/engine.py.229: process: 0.50654 | load: 0.00246 [I 2022-10-26 13:56:50] ...trap/engines/engine.py.230: loss: 0.25773 [I 2022-10-26 13:56:50] ...optimizers/trijoint.py.151: total_norm: 2.982217 [I 2022-10-26 13:56:55] ...trap/engines/engine.py.224: train: epoch 11 | batch 1990/2304 [I 2022-10-26 13:56:55] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:02:37 [I 2022-10-26 13:56:55] ...trap/engines/engine.py.229: process: 0.50618 | load: 0.00249 [I 2022-10-26 13:56:55] ...trap/engines/engine.py.230: loss: 0.24248 [I 2022-10-26 13:56:55] ...optimizers/trijoint.py.151: total_norm: 3.675677 [I 2022-10-26 13:57:00] ...trap/engines/engine.py.224: train: epoch 11 | batch 2000/2304 [I 2022-10-26 13:57:00] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:02:31 [I 2022-10-26 13:57:00] ...trap/engines/engine.py.229: process: 0.49148 | load: 0.00247 [I 2022-10-26 13:57:00] ...trap/engines/engine.py.230: loss: 0.27380 [I 2022-10-26 13:57:00] ...optimizers/trijoint.py.151: total_norm: 3.219996 [I 2022-10-26 13:57:05] ...trap/engines/engine.py.224: train: epoch 11 | batch 2010/2304 [I 2022-10-26 13:57:05] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:02:25 [I 2022-10-26 13:57:05] ...trap/engines/engine.py.229: process: 0.50407 | load: 0.00243 [I 2022-10-26 13:57:05] ...trap/engines/engine.py.230: loss: 0.25317 [I 2022-10-26 13:57:05] ...optimizers/trijoint.py.151: total_norm: 3.199576 [I 2022-10-26 13:57:10] ...trap/engines/engine.py.224: train: epoch 11 | batch 2020/2304 [I 2022-10-26 13:57:10] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:02:21 [I 2022-10-26 13:57:10] ...trap/engines/engine.py.229: process: 0.50467 | load: 0.00241 [I 2022-10-26 13:57:10] ...trap/engines/engine.py.230: loss: 0.26776 [I 2022-10-26 13:57:10] ...optimizers/trijoint.py.151: total_norm: 2.938769 [I 2022-10-26 13:57:15] ...trap/engines/engine.py.224: train: epoch 11 | batch 2030/2304 [I 2022-10-26 13:57:15] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:02:15 [I 2022-10-26 13:57:15] ...trap/engines/engine.py.229: process: 0.49237 | load: 0.00243 [I 2022-10-26 13:57:15] ...trap/engines/engine.py.230: loss: 0.18961 [I 2022-10-26 13:57:15] ...optimizers/trijoint.py.151: total_norm: 2.432000 [I 2022-10-26 13:57:20] ...trap/engines/engine.py.224: train: epoch 11 | batch 2040/2304 [I 2022-10-26 13:57:20] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:02:11 [I 2022-10-26 13:57:20] ...trap/engines/engine.py.229: process: 0.50134 | load: 0.00242 [I 2022-10-26 13:57:20] ...trap/engines/engine.py.230: loss: 0.25556 [I 2022-10-26 13:57:20] ...optimizers/trijoint.py.151: total_norm: 3.645825 [I 2022-10-26 13:57:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 2050/2304 [I 2022-10-26 13:57:24] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:02:05 [I 2022-10-26 13:57:24] ...trap/engines/engine.py.229: process: 0.49265 | load: 0.00244 [I 2022-10-26 13:57:24] ...trap/engines/engine.py.230: loss: 0.29148 [I 2022-10-26 13:57:24] ...optimizers/trijoint.py.151: total_norm: 3.588112 [I 2022-10-26 13:57:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 2060/2304 [I 2022-10-26 13:57:29] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:02:01 [I 2022-10-26 13:57:29] ...trap/engines/engine.py.229: process: 0.49825 | load: 0.00241 [I 2022-10-26 13:57:29] ...trap/engines/engine.py.230: loss: 0.21605 [I 2022-10-26 13:57:29] ...optimizers/trijoint.py.151: total_norm: 3.589317 [I 2022-10-26 13:57:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 2070/2304 [I 2022-10-26 13:57:34] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:01:55 [I 2022-10-26 13:57:34] ...trap/engines/engine.py.229: process: 0.50104 | load: 0.00245 [I 2022-10-26 13:57:34] ...trap/engines/engine.py.230: loss: 0.37073 [I 2022-10-26 13:57:34] ...optimizers/trijoint.py.151: total_norm: 3.939028 [I 2022-10-26 13:57:40] ...trap/engines/engine.py.224: train: epoch 11 | batch 2080/2304 [I 2022-10-26 13:57:40] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:01:52 [I 2022-10-26 13:57:40] ...trap/engines/engine.py.229: process: 0.49547 | load: 0.00247 [I 2022-10-26 13:57:40] ...trap/engines/engine.py.230: loss: 0.25763 [I 2022-10-26 13:57:40] ...optimizers/trijoint.py.151: total_norm: 3.911556 [I 2022-10-26 13:57:45] ...trap/engines/engine.py.224: train: epoch 11 | batch 2090/2304 [I 2022-10-26 13:57:45] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:01:46 [I 2022-10-26 13:57:45] ...trap/engines/engine.py.229: process: 0.49164 | load: 0.00249 [I 2022-10-26 13:57:45] ...trap/engines/engine.py.230: loss: 0.15353 [I 2022-10-26 13:57:45] ...optimizers/trijoint.py.151: total_norm: 2.994761 [I 2022-10-26 13:57:50] ...trap/engines/engine.py.224: train: epoch 11 | batch 2100/2304 [I 2022-10-26 13:57:50] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:01:41 [I 2022-10-26 13:57:50] ...trap/engines/engine.py.229: process: 0.50071 | load: 0.00243 [I 2022-10-26 13:57:50] ...trap/engines/engine.py.230: loss: 0.29709 [I 2022-10-26 13:57:50] ...optimizers/trijoint.py.151: total_norm: 4.341280 [I 2022-10-26 13:57:55] ...trap/engines/engine.py.224: train: epoch 11 | batch 2110/2304 [I 2022-10-26 13:57:55] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:01:36 [I 2022-10-26 13:57:55] ...trap/engines/engine.py.229: process: 0.50491 | load: 0.00244 [I 2022-10-26 13:57:55] ...trap/engines/engine.py.230: loss: 0.26981 [I 2022-10-26 13:57:55] ...optimizers/trijoint.py.151: total_norm: 3.781480 [I 2022-10-26 13:58:00] ...trap/engines/engine.py.224: train: epoch 11 | batch 2120/2304 [I 2022-10-26 13:58:00] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:01:31 [I 2022-10-26 13:58:00] ...trap/engines/engine.py.229: process: 0.50412 | load: 0.00245 [I 2022-10-26 13:58:00] ...trap/engines/engine.py.230: loss: 0.22439 [I 2022-10-26 13:58:00] ...optimizers/trijoint.py.151: total_norm: 2.909604 [I 2022-10-26 13:58:05] ...trap/engines/engine.py.224: train: epoch 11 | batch 2130/2304 [I 2022-10-26 13:58:05] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:01:26 [I 2022-10-26 13:58:05] ...trap/engines/engine.py.229: process: 0.49517 | load: 0.00242 [I 2022-10-26 13:58:05] ...trap/engines/engine.py.230: loss: 0.18755 [I 2022-10-26 13:58:05] ...optimizers/trijoint.py.151: total_norm: 2.623069 [I 2022-10-26 13:58:09] ...trap/engines/engine.py.224: train: epoch 11 | batch 2140/2304 [I 2022-10-26 13:58:09] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:01:21 [I 2022-10-26 13:58:09] ...trap/engines/engine.py.229: process: 0.49829 | load: 0.00240 [I 2022-10-26 13:58:09] ...trap/engines/engine.py.230: loss: 0.31518 [I 2022-10-26 13:58:09] ...optimizers/trijoint.py.151: total_norm: 3.227131 [I 2022-10-26 13:58:14] ...trap/engines/engine.py.224: train: epoch 11 | batch 2150/2304 [I 2022-10-26 13:58:14] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:01:16 [I 2022-10-26 13:58:14] ...trap/engines/engine.py.229: process: 0.50020 | load: 0.00244 [I 2022-10-26 13:58:14] ...trap/engines/engine.py.230: loss: 0.32412 [I 2022-10-26 13:58:14] ...optimizers/trijoint.py.151: total_norm: 4.770483 [I 2022-10-26 13:58:19] ...trap/engines/engine.py.224: train: epoch 11 | batch 2160/2304 [I 2022-10-26 13:58:19] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:01:11 [I 2022-10-26 13:58:19] ...trap/engines/engine.py.229: process: 0.50394 | load: 0.00242 [I 2022-10-26 13:58:19] ...trap/engines/engine.py.230: loss: 0.24643 [I 2022-10-26 13:58:19] ...optimizers/trijoint.py.151: total_norm: 3.488080 [I 2022-10-26 13:58:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 2170/2304 [I 2022-10-26 13:58:24] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:01:06 [I 2022-10-26 13:58:24] ...trap/engines/engine.py.229: process: 0.49589 | load: 0.00242 [I 2022-10-26 13:58:24] ...trap/engines/engine.py.230: loss: 0.13080 [I 2022-10-26 13:58:24] ...optimizers/trijoint.py.151: total_norm: 2.560601 [I 2022-10-26 13:58:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 2180/2304 [I 2022-10-26 13:58:29] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:01:01 [I 2022-10-26 13:58:29] ...trap/engines/engine.py.229: process: 0.49360 | load: 0.00242 [I 2022-10-26 13:58:29] ...trap/engines/engine.py.230: loss: 0.23893 [I 2022-10-26 13:58:29] ...optimizers/trijoint.py.151: total_norm: 4.008029 [I 2022-10-26 13:58:34] ...trap/engines/engine.py.224: train: epoch 11 | batch 2190/2304 [I 2022-10-26 13:58:34] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:00:56 [I 2022-10-26 13:58:34] ...trap/engines/engine.py.229: process: 0.50354 | load: 0.00241 [I 2022-10-26 13:58:34] ...trap/engines/engine.py.230: loss: 0.33765 [I 2022-10-26 13:58:34] ...optimizers/trijoint.py.151: total_norm: 3.177629 [I 2022-10-26 13:58:39] ...trap/engines/engine.py.224: train: epoch 11 | batch 2200/2304 [I 2022-10-26 13:58:39] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:00:50 [I 2022-10-26 13:58:39] ...trap/engines/engine.py.229: process: 0.47988 | load: 0.00242 [I 2022-10-26 13:58:39] ...trap/engines/engine.py.230: loss: 0.20567 [I 2022-10-26 13:58:39] ...optimizers/trijoint.py.151: total_norm: 2.871121 [I 2022-10-26 13:58:44] ...trap/engines/engine.py.224: train: epoch 11 | batch 2210/2304 [I 2022-10-26 13:58:44] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:00:47 [I 2022-10-26 13:58:44] ...trap/engines/engine.py.229: process: 0.50124 | load: 0.00261 [I 2022-10-26 13:58:44] ...trap/engines/engine.py.230: loss: 0.23749 [I 2022-10-26 13:58:44] ...optimizers/trijoint.py.151: total_norm: 3.166911 [I 2022-10-26 13:58:49] ...trap/engines/engine.py.224: train: epoch 11 | batch 2220/2304 [I 2022-10-26 13:58:49] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:00:41 [I 2022-10-26 13:58:49] ...trap/engines/engine.py.229: process: 0.48526 | load: 0.00243 [I 2022-10-26 13:58:49] ...trap/engines/engine.py.230: loss: 0.20603 [I 2022-10-26 13:58:49] ...optimizers/trijoint.py.151: total_norm: 3.108382 [I 2022-10-26 13:58:54] ...trap/engines/engine.py.224: train: epoch 11 | batch 2230/2304 [I 2022-10-26 13:58:54] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:00:36 [I 2022-10-26 13:58:54] ...trap/engines/engine.py.229: process: 0.50579 | load: 0.00242 [I 2022-10-26 13:58:54] ...trap/engines/engine.py.230: loss: 0.21110 [I 2022-10-26 13:58:54] ...optimizers/trijoint.py.151: total_norm: 3.710114 [I 2022-10-26 13:58:59] ...trap/engines/engine.py.224: train: epoch 11 | batch 2240/2304 [I 2022-10-26 13:58:59] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:00:31 [I 2022-10-26 13:58:59] ...trap/engines/engine.py.229: process: 0.50181 | load: 0.00242 [I 2022-10-26 13:58:59] ...trap/engines/engine.py.230: loss: 0.27530 [I 2022-10-26 13:58:59] ...optimizers/trijoint.py.151: total_norm: 3.822156 [I 2022-10-26 13:59:04] ...trap/engines/engine.py.224: train: epoch 11 | batch 2250/2304 [I 2022-10-26 13:59:04] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:00:26 [I 2022-10-26 13:59:04] ...trap/engines/engine.py.229: process: 0.50485 | load: 0.00239 [I 2022-10-26 13:59:04] ...trap/engines/engine.py.230: loss: 0.24584 [I 2022-10-26 13:59:04] ...optimizers/trijoint.py.151: total_norm: 3.116954 [I 2022-10-26 13:59:09] ...trap/engines/engine.py.224: train: epoch 11 | batch 2260/2304 [I 2022-10-26 13:59:09] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:00:21 [I 2022-10-26 13:59:09] ...trap/engines/engine.py.229: process: 0.48668 | load: 0.00243 [I 2022-10-26 13:59:09] ...trap/engines/engine.py.230: loss: 0.29496 [I 2022-10-26 13:59:09] ...optimizers/trijoint.py.151: total_norm: 2.954512 [I 2022-10-26 13:59:14] ...trap/engines/engine.py.224: train: epoch 11 | batch 2270/2304 [I 2022-10-26 13:59:14] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:00:16 [I 2022-10-26 13:59:14] ...trap/engines/engine.py.229: process: 0.49660 | load: 0.00256 [I 2022-10-26 13:59:14] ...trap/engines/engine.py.230: loss: 0.27007 [I 2022-10-26 13:59:14] ...optimizers/trijoint.py.151: total_norm: 3.827466 [I 2022-10-26 13:59:19] ...trap/engines/engine.py.224: train: epoch 11 | batch 2280/2304 [I 2022-10-26 13:59:19] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:00:11 [I 2022-10-26 13:59:19] ...trap/engines/engine.py.229: process: 0.49678 | load: 0.00262 [I 2022-10-26 13:59:19] ...trap/engines/engine.py.230: loss: 0.16300 [I 2022-10-26 13:59:19] ...optimizers/trijoint.py.151: total_norm: 2.665082 [I 2022-10-26 13:59:24] ...trap/engines/engine.py.224: train: epoch 11 | batch 2290/2304 [I 2022-10-26 13:59:24] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:00:06 [I 2022-10-26 13:59:24] ...trap/engines/engine.py.229: process: 0.50033 | load: 0.00256 [I 2022-10-26 13:59:24] ...trap/engines/engine.py.230: loss: 0.26818 [I 2022-10-26 13:59:24] ...optimizers/trijoint.py.151: total_norm: 3.606090 [I 2022-10-26 13:59:29] ...trap/engines/engine.py.224: train: epoch 11 | batch 2300/2304 [I 2022-10-26 13:59:29] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:00:02 [I 2022-10-26 13:59:29] ...trap/engines/engine.py.229: process: 0.48652 | load: 0.00015 [I 2022-10-26 13:59:29] ...trap/engines/engine.py.230: loss: 0.31751 [I 2022-10-26 13:59:29] ...optimizers/trijoint.py.151: total_norm: 4.444272 [I 2022-10-26 13:59:31] ...trap/engines/engine.py.224: train: epoch 11 | batch 2304/2304 [I 2022-10-26 13:59:31] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:00:00 [I 2022-10-26 13:59:31] ...trap/engines/engine.py.229: process: 0.49111 | load: 0.00015 [I 2022-10-26 13:59:31] ...trap/engines/engine.py.230: loss: 0.22026 [I 2022-10-26 13:59:31] ...optimizers/trijoint.py.151: total_norm: 3.715749 [S 2022-10-26 13:59:33] ...trap/engines/engine.py.129: train_epoch.epoch: 11 [S 2022-10-26 13:59:33] ...trap/engines/engine.py.129: train_epoch.loss: 0.23183061165367444 [I 2022-10-26 13:59:33] ...s/criterions/triplet.py.84: margin = 0.11000000000000004 [I 2022-10-26 13:59:33] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 11 [I 2022-10-26 13:59:33] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 13:59:35] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 13:59:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 13:59:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 13:59:35] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 13:59:35] ...trap/engines/engine.py.321: eval: epoch 11 | batch 0/510 [I 2022-10-26 13:59:35] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:22 [I 2022-10-26 13:59:35] ...trap/engines/engine.py.326: process: 2.63241 | load: 2.24514 [I 2022-10-26 13:59:39] ...trap/engines/engine.py.321: eval: epoch 11 | batch 10/510 [I 2022-10-26 13:59:39] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:04:49 [I 2022-10-26 13:59:39] ...trap/engines/engine.py.326: process: 0.31972 | load: 0.00017 [I 2022-10-26 13:59:42] ...trap/engines/engine.py.321: eval: epoch 11 | batch 20/510 [I 2022-10-26 13:59:42] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:53 [I 2022-10-26 13:59:42] ...trap/engines/engine.py.326: process: 0.33183 | load: 0.00017 [I 2022-10-26 13:59:45] ...trap/engines/engine.py.321: eval: epoch 11 | batch 30/510 [I 2022-10-26 13:59:45] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:41 [I 2022-10-26 13:59:45] ...trap/engines/engine.py.326: process: 0.32966 | load: 0.00018 [I 2022-10-26 13:59:49] ...trap/engines/engine.py.321: eval: epoch 11 | batch 40/510 [I 2022-10-26 13:59:49] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:38 [I 2022-10-26 13:59:49] ...trap/engines/engine.py.326: process: 0.33321 | load: 0.00018 [I 2022-10-26 13:59:52] ...trap/engines/engine.py.321: eval: epoch 11 | batch 50/510 [I 2022-10-26 13:59:52] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-26 13:59:52] ...trap/engines/engine.py.326: process: 0.35514 | load: 0.00020 [I 2022-10-26 13:59:56] ...trap/engines/engine.py.321: eval: epoch 11 | batch 60/510 [I 2022-10-26 13:59:56] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:31 [I 2022-10-26 13:59:56] ...trap/engines/engine.py.326: process: 0.32682 | load: 0.00017 [I 2022-10-26 13:59:59] ...trap/engines/engine.py.321: eval: epoch 11 | batch 70/510 [I 2022-10-26 13:59:59] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-26 13:59:59] ...trap/engines/engine.py.326: process: 0.33462 | load: 0.00039 [I 2022-10-26 14:00:02] ...trap/engines/engine.py.321: eval: epoch 11 | batch 80/510 [I 2022-10-26 14:00:02] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:22 [I 2022-10-26 14:00:02] ...trap/engines/engine.py.326: process: 0.33597 | load: 0.00017 [I 2022-10-26 14:00:06] ...trap/engines/engine.py.321: eval: epoch 11 | batch 90/510 [I 2022-10-26 14:00:06] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:18 [I 2022-10-26 14:00:06] ...trap/engines/engine.py.326: process: 0.33054 | load: 0.00018 [I 2022-10-26 14:00:09] ...trap/engines/engine.py.321: eval: epoch 11 | batch 100/510 [I 2022-10-26 14:00:09] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 14:00:09] ...trap/engines/engine.py.326: process: 0.28501 | load: 0.00018 [I 2022-10-26 14:00:12] ...trap/engines/engine.py.321: eval: epoch 11 | batch 110/510 [I 2022-10-26 14:00:12] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 14:00:12] ...trap/engines/engine.py.326: process: 0.29029 | load: 0.00014 [I 2022-10-26 14:00:15] ...trap/engines/engine.py.321: eval: epoch 11 | batch 120/510 [I 2022-10-26 14:00:15] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:54 [I 2022-10-26 14:00:15] ...trap/engines/engine.py.326: process: 0.29658 | load: 0.00015 [I 2022-10-26 14:00:18] ...trap/engines/engine.py.321: eval: epoch 11 | batch 130/510 [I 2022-10-26 14:00:18] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:49 [I 2022-10-26 14:00:18] ...trap/engines/engine.py.326: process: 0.28856 | load: 0.00016 [I 2022-10-26 14:00:21] ...trap/engines/engine.py.321: eval: epoch 11 | batch 140/510 [I 2022-10-26 14:00:21] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:48 [I 2022-10-26 14:00:21] ...trap/engines/engine.py.326: process: 0.30922 | load: 0.00014 [I 2022-10-26 14:00:23] ...trap/engines/engine.py.321: eval: epoch 11 | batch 150/510 [I 2022-10-26 14:00:23] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:43 [I 2022-10-26 14:00:23] ...trap/engines/engine.py.326: process: 0.28456 | load: 0.00014 [I 2022-10-26 14:00:26] ...trap/engines/engine.py.321: eval: epoch 11 | batch 160/510 [I 2022-10-26 14:00:26] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:41 [I 2022-10-26 14:00:26] ...trap/engines/engine.py.326: process: 0.29081 | load: 0.00024 [I 2022-10-26 14:00:29] ...trap/engines/engine.py.321: eval: epoch 11 | batch 170/510 [I 2022-10-26 14:00:29] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 14:00:29] ...trap/engines/engine.py.326: process: 0.29111 | load: 0.00021 [I 2022-10-26 14:00:32] ...trap/engines/engine.py.321: eval: epoch 11 | batch 180/510 [I 2022-10-26 14:00:32] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:37 [I 2022-10-26 14:00:32] ...trap/engines/engine.py.326: process: 0.28809 | load: 0.00015 [I 2022-10-26 14:00:35] ...trap/engines/engine.py.321: eval: epoch 11 | batch 190/510 [I 2022-10-26 14:00:35] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-26 14:00:35] ...trap/engines/engine.py.326: process: 0.29085 | load: 0.00021 [I 2022-10-26 14:00:38] ...trap/engines/engine.py.321: eval: epoch 11 | batch 200/510 [I 2022-10-26 14:00:38] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 14:00:38] ...trap/engines/engine.py.326: process: 0.28731 | load: 0.00020 [I 2022-10-26 14:00:41] ...trap/engines/engine.py.321: eval: epoch 11 | batch 210/510 [I 2022-10-26 14:00:41] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 14:00:41] ...trap/engines/engine.py.326: process: 0.28410 | load: 0.00016 [I 2022-10-26 14:00:44] ...trap/engines/engine.py.321: eval: epoch 11 | batch 220/510 [I 2022-10-26 14:00:44] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 14:00:44] ...trap/engines/engine.py.326: process: 0.28893 | load: 0.00015 [I 2022-10-26 14:00:47] ...trap/engines/engine.py.321: eval: epoch 11 | batch 230/510 [I 2022-10-26 14:00:47] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:20 [I 2022-10-26 14:00:47] ...trap/engines/engine.py.326: process: 0.28553 | load: 0.00016 [I 2022-10-26 14:00:50] ...trap/engines/engine.py.321: eval: epoch 11 | batch 240/510 [I 2022-10-26 14:00:50] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:17 [I 2022-10-26 14:00:50] ...trap/engines/engine.py.326: process: 0.28476 | load: 0.00015 [I 2022-10-26 14:00:52] ...trap/engines/engine.py.321: eval: epoch 11 | batch 250/510 [I 2022-10-26 14:00:52] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:14 [I 2022-10-26 14:00:52] ...trap/engines/engine.py.326: process: 0.28695 | load: 0.00014 [I 2022-10-26 14:00:55] ...trap/engines/engine.py.321: eval: epoch 11 | batch 260/510 [I 2022-10-26 14:00:55] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:11 [I 2022-10-26 14:00:55] ...trap/engines/engine.py.326: process: 0.28404 | load: 0.00015 [I 2022-10-26 14:00:58] ...trap/engines/engine.py.321: eval: epoch 11 | batch 270/510 [I 2022-10-26 14:00:58] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-26 14:00:58] ...trap/engines/engine.py.326: process: 0.29022 | load: 0.00015 [I 2022-10-26 14:01:01] ...trap/engines/engine.py.321: eval: epoch 11 | batch 280/510 [I 2022-10-26 14:01:01] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 14:01:01] ...trap/engines/engine.py.326: process: 0.29101 | load: 0.00016 [I 2022-10-26 14:01:04] ...trap/engines/engine.py.321: eval: epoch 11 | batch 290/510 [I 2022-10-26 14:01:04] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 14:01:04] ...trap/engines/engine.py.326: process: 0.28701 | load: 0.00016 [I 2022-10-26 14:01:07] ...trap/engines/engine.py.321: eval: epoch 11 | batch 300/510 [I 2022-10-26 14:01:07] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:00 [I 2022-10-26 14:01:07] ...trap/engines/engine.py.326: process: 0.28828 | load: 0.00019 [I 2022-10-26 14:01:10] ...trap/engines/engine.py.321: eval: epoch 11 | batch 310/510 [I 2022-10-26 14:01:10] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:00:57 [I 2022-10-26 14:01:10] ...trap/engines/engine.py.326: process: 0.28341 | load: 0.00016 [I 2022-10-26 14:01:12] ...trap/engines/engine.py.321: eval: epoch 11 | batch 320/510 [I 2022-10-26 14:01:12] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:54 [I 2022-10-26 14:01:12] ...trap/engines/engine.py.326: process: 0.28562 | load: 0.00016 [I 2022-10-26 14:01:15] ...trap/engines/engine.py.321: eval: epoch 11 | batch 330/510 [I 2022-10-26 14:01:15] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:51 [I 2022-10-26 14:01:15] ...trap/engines/engine.py.326: process: 0.28634 | load: 0.00015 [I 2022-10-26 14:01:18] ...trap/engines/engine.py.321: eval: epoch 11 | batch 340/510 [I 2022-10-26 14:01:18] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:48 [I 2022-10-26 14:01:18] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00015 [I 2022-10-26 14:01:21] ...trap/engines/engine.py.321: eval: epoch 11 | batch 350/510 [I 2022-10-26 14:01:21] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:45 [I 2022-10-26 14:01:21] ...trap/engines/engine.py.326: process: 0.28527 | load: 0.00015 [I 2022-10-26 14:01:24] ...trap/engines/engine.py.321: eval: epoch 11 | batch 360/510 [I 2022-10-26 14:01:24] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 14:01:24] ...trap/engines/engine.py.326: process: 0.28754 | load: 0.00019 [I 2022-10-26 14:01:27] ...trap/engines/engine.py.321: eval: epoch 11 | batch 370/510 [I 2022-10-26 14:01:27] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 14:01:27] ...trap/engines/engine.py.326: process: 0.28862 | load: 0.00016 [I 2022-10-26 14:01:30] ...trap/engines/engine.py.321: eval: epoch 11 | batch 380/510 [I 2022-10-26 14:01:30] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:37 [I 2022-10-26 14:01:30] ...trap/engines/engine.py.326: process: 0.28288 | load: 0.00019 [I 2022-10-26 14:01:33] ...trap/engines/engine.py.321: eval: epoch 11 | batch 390/510 [I 2022-10-26 14:01:33] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:34 [I 2022-10-26 14:01:33] ...trap/engines/engine.py.326: process: 0.28956 | load: 0.00016 [I 2022-10-26 14:01:35] ...trap/engines/engine.py.321: eval: epoch 11 | batch 400/510 [I 2022-10-26 14:01:35] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:31 [I 2022-10-26 14:01:35] ...trap/engines/engine.py.326: process: 0.29195 | load: 0.00018 [I 2022-10-26 14:01:38] ...trap/engines/engine.py.321: eval: epoch 11 | batch 410/510 [I 2022-10-26 14:01:38] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:28 [I 2022-10-26 14:01:38] ...trap/engines/engine.py.326: process: 0.28970 | load: 0.00017 [I 2022-10-26 14:01:41] ...trap/engines/engine.py.321: eval: epoch 11 | batch 420/510 [I 2022-10-26 14:01:41] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:26 [I 2022-10-26 14:01:41] ...trap/engines/engine.py.326: process: 0.28902 | load: 0.00018 [I 2022-10-26 14:01:44] ...trap/engines/engine.py.321: eval: epoch 11 | batch 430/510 [I 2022-10-26 14:01:44] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-26 14:01:44] ...trap/engines/engine.py.326: process: 0.29402 | load: 0.00019 [I 2022-10-26 14:01:47] ...trap/engines/engine.py.321: eval: epoch 11 | batch 440/510 [I 2022-10-26 14:01:47] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 14:01:47] ...trap/engines/engine.py.326: process: 0.28977 | load: 0.00018 [I 2022-10-26 14:01:50] ...trap/engines/engine.py.321: eval: epoch 11 | batch 450/510 [I 2022-10-26 14:01:50] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 14:01:50] ...trap/engines/engine.py.326: process: 0.28153 | load: 0.00016 [I 2022-10-26 14:01:53] ...trap/engines/engine.py.321: eval: epoch 11 | batch 460/510 [I 2022-10-26 14:01:53] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:14 [I 2022-10-26 14:01:53] ...trap/engines/engine.py.326: process: 0.28375 | load: 0.00020 [I 2022-10-26 14:01:56] ...trap/engines/engine.py.321: eval: epoch 11 | batch 470/510 [I 2022-10-26 14:01:56] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:11 [I 2022-10-26 14:01:56] ...trap/engines/engine.py.326: process: 0.29066 | load: 0.00017 [I 2022-10-26 14:01:58] ...trap/engines/engine.py.321: eval: epoch 11 | batch 480/510 [I 2022-10-26 14:01:58] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:08 [I 2022-10-26 14:01:58] ...trap/engines/engine.py.326: process: 0.29105 | load: 0.00019 [I 2022-10-26 14:02:01] ...trap/engines/engine.py.321: eval: epoch 11 | batch 490/510 [I 2022-10-26 14:02:01] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:05 [I 2022-10-26 14:02:01] ...trap/engines/engine.py.326: process: 0.28661 | load: 0.00018 [I 2022-10-26 14:02:04] ...trap/engines/engine.py.321: eval: epoch 11 | batch 500/510 [I 2022-10-26 14:02:04] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-26 14:02:04] ...trap/engines/engine.py.326: process: 0.29018 | load: 0.00023 [I 2022-10-26 14:02:07] ...trap/engines/engine.py.321: eval: epoch 11 | batch 510/510 [I 2022-10-26 14:02:07] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 14:02:07] ...trap/engines/engine.py.326: process: 0.28699 | load: 0.00011 [S 2022-10-26 14:02:08] ...trap/engines/engine.py.132: eval_epoch.epoch: 11 [I 2022-10-26 14:02:08] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.520800 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.811200 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.879600 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.533200 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.809900 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.881500 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.007305 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008807 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007940 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010167 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009321 [S 2022-10-26 14:02:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007852 [I 2022-10-26 14:02:13] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.med_im2recipe_mean [I 2022-10-26 14:02:13] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 14:02:13] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 14:02:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 14:02:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 14:02:13] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 14:02:16] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 14:02:21] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 14:02:21] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 14:02:21] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 14:02:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 14:02:29] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 14:02:29] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 14:02:29] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 14:02:32] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 14:02:36] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 14:02:36] ...trap/engines/engine.py.163: Training model on trainset for epoch 12 [I 2022-10-26 14:02:36] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 14:02:39] ...trap/engines/engine.py.224: train: epoch 12 | batch 0/2304 [I 2022-10-26 14:02:39] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:08 [I 2022-10-26 14:02:39] ...trap/engines/engine.py.229: process: 2.76408 | load: 2.17101 [I 2022-10-26 14:02:39] ...trap/engines/engine.py.230: loss: 0.18829 [I 2022-10-26 14:02:39] ...optimizers/trijoint.py.151: total_norm: 2.589895 [I 2022-10-26 14:02:44] ...trap/engines/engine.py.224: train: epoch 12 | batch 10/2304 [I 2022-10-26 14:02:44] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:24 [I 2022-10-26 14:02:44] ...trap/engines/engine.py.229: process: 0.49960 | load: 0.00414 [I 2022-10-26 14:02:44] ...trap/engines/engine.py.230: loss: 0.20276 [I 2022-10-26 14:02:44] ...optimizers/trijoint.py.151: total_norm: 3.762921 [I 2022-10-26 14:02:49] ...trap/engines/engine.py.224: train: epoch 12 | batch 20/2304 [I 2022-10-26 14:02:49] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:13 [I 2022-10-26 14:02:49] ...trap/engines/engine.py.229: process: 0.49902 | load: 0.00490 [I 2022-10-26 14:02:49] ...trap/engines/engine.py.230: loss: 0.19093 [I 2022-10-26 14:02:49] ...optimizers/trijoint.py.151: total_norm: 2.647620 [I 2022-10-26 14:02:54] ...trap/engines/engine.py.224: train: epoch 12 | batch 30/2304 [I 2022-10-26 14:02:54] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:05 [I 2022-10-26 14:02:54] ...trap/engines/engine.py.229: process: 0.49285 | load: 0.00295 [I 2022-10-26 14:02:54] ...trap/engines/engine.py.230: loss: 0.20762 [I 2022-10-26 14:02:54] ...optimizers/trijoint.py.151: total_norm: 3.340560 [I 2022-10-26 14:02:59] ...trap/engines/engine.py.224: train: epoch 12 | batch 40/2304 [I 2022-10-26 14:02:59] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:46 [I 2022-10-26 14:02:59] ...trap/engines/engine.py.229: process: 0.48196 | load: 0.00313 [I 2022-10-26 14:02:59] ...trap/engines/engine.py.230: loss: 0.28507 [I 2022-10-26 14:02:59] ...optimizers/trijoint.py.151: total_norm: 3.887175 [I 2022-10-26 14:03:04] ...trap/engines/engine.py.224: train: epoch 12 | batch 50/2304 [I 2022-10-26 14:03:04] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:39 [I 2022-10-26 14:03:04] ...trap/engines/engine.py.229: process: 0.49777 | load: 0.00310 [I 2022-10-26 14:03:04] ...trap/engines/engine.py.230: loss: 0.22225 [I 2022-10-26 14:03:04] ...optimizers/trijoint.py.151: total_norm: 3.789170 [I 2022-10-26 14:03:09] ...trap/engines/engine.py.224: train: epoch 12 | batch 60/2304 [I 2022-10-26 14:03:09] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:35 [I 2022-10-26 14:03:09] ...trap/engines/engine.py.229: process: 0.49806 | load: 0.00275 [I 2022-10-26 14:03:09] ...trap/engines/engine.py.230: loss: 0.16049 [I 2022-10-26 14:03:09] ...optimizers/trijoint.py.151: total_norm: 3.085288 [I 2022-10-26 14:03:14] ...trap/engines/engine.py.224: train: epoch 12 | batch 70/2304 [I 2022-10-26 14:03:14] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:33 [I 2022-10-26 14:03:14] ...trap/engines/engine.py.229: process: 0.51290 | load: 0.00272 [I 2022-10-26 14:03:14] ...trap/engines/engine.py.230: loss: 0.29825 [I 2022-10-26 14:03:14] ...optimizers/trijoint.py.151: total_norm: 3.573439 [I 2022-10-26 14:03:19] ...trap/engines/engine.py.224: train: epoch 12 | batch 80/2304 [I 2022-10-26 14:03:19] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:30 [I 2022-10-26 14:03:19] ...trap/engines/engine.py.229: process: 0.50768 | load: 0.00291 [I 2022-10-26 14:03:19] ...trap/engines/engine.py.230: loss: 0.21556 [I 2022-10-26 14:03:19] ...optimizers/trijoint.py.151: total_norm: 2.645141 [I 2022-10-26 14:03:24] ...trap/engines/engine.py.224: train: epoch 12 | batch 90/2304 [I 2022-10-26 14:03:24] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:18:10 [I 2022-10-26 14:03:24] ...trap/engines/engine.py.229: process: 0.48974 | load: 0.00254 [I 2022-10-26 14:03:24] ...trap/engines/engine.py.230: loss: 0.21092 [I 2022-10-26 14:03:24] ...optimizers/trijoint.py.151: total_norm: 2.701898 [I 2022-10-26 14:03:29] ...trap/engines/engine.py.224: train: epoch 12 | batch 100/2304 [I 2022-10-26 14:03:29] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:18:15 [I 2022-10-26 14:03:29] ...trap/engines/engine.py.229: process: 0.49553 | load: 0.00276 [I 2022-10-26 14:03:29] ...trap/engines/engine.py.230: loss: 0.21840 [I 2022-10-26 14:03:29] ...optimizers/trijoint.py.151: total_norm: 2.429208 [I 2022-10-26 14:03:34] ...trap/engines/engine.py.224: train: epoch 12 | batch 110/2304 [I 2022-10-26 14:03:34] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:18:18 [I 2022-10-26 14:03:34] ...trap/engines/engine.py.229: process: 0.51118 | load: 0.00270 [I 2022-10-26 14:03:34] ...trap/engines/engine.py.230: loss: 0.17893 [I 2022-10-26 14:03:34] ...optimizers/trijoint.py.151: total_norm: 2.982622 [I 2022-10-26 14:03:39] ...trap/engines/engine.py.224: train: epoch 12 | batch 120/2304 [I 2022-10-26 14:03:39] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:18:08 [I 2022-10-26 14:03:39] ...trap/engines/engine.py.229: process: 0.49829 | load: 0.00284 [I 2022-10-26 14:03:39] ...trap/engines/engine.py.230: loss: 0.24913 [I 2022-10-26 14:03:39] ...optimizers/trijoint.py.151: total_norm: 3.616586 [I 2022-10-26 14:03:44] ...trap/engines/engine.py.224: train: epoch 12 | batch 130/2304 [I 2022-10-26 14:03:44] ...trap/engines/engine.py.228: elapsed: 0:01:07 | left: 0:18:19 [I 2022-10-26 14:03:44] ...trap/engines/engine.py.229: process: 0.52317 | load: 0.00271 [I 2022-10-26 14:03:44] ...trap/engines/engine.py.230: loss: 0.24637 [I 2022-10-26 14:03:44] ...optimizers/trijoint.py.151: total_norm: 3.319577 [I 2022-10-26 14:03:49] ...trap/engines/engine.py.224: train: epoch 12 | batch 140/2304 [I 2022-10-26 14:03:49] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:17:57 [I 2022-10-26 14:03:49] ...trap/engines/engine.py.229: process: 0.49774 | load: 0.00265 [I 2022-10-26 14:03:49] ...trap/engines/engine.py.230: loss: 0.26684 [I 2022-10-26 14:03:49] ...optimizers/trijoint.py.151: total_norm: 3.420204 [I 2022-10-26 14:03:54] ...trap/engines/engine.py.224: train: epoch 12 | batch 150/2304 [I 2022-10-26 14:03:54] ...trap/engines/engine.py.228: elapsed: 0:01:17 | left: 0:17:57 [I 2022-10-26 14:03:54] ...trap/engines/engine.py.229: process: 0.51184 | load: 0.00286 [I 2022-10-26 14:03:54] ...trap/engines/engine.py.230: loss: 0.27407 [I 2022-10-26 14:03:54] ...optimizers/trijoint.py.151: total_norm: 3.857206 [I 2022-10-26 14:03:59] ...trap/engines/engine.py.224: train: epoch 12 | batch 160/2304 [I 2022-10-26 14:03:59] ...trap/engines/engine.py.228: elapsed: 0:01:22 | left: 0:17:49 [I 2022-10-26 14:03:59] ...trap/engines/engine.py.229: process: 0.49871 | load: 0.00269 [I 2022-10-26 14:03:59] ...trap/engines/engine.py.230: loss: 0.21688 [I 2022-10-26 14:03:59] ...optimizers/trijoint.py.151: total_norm: 2.993263 [I 2022-10-26 14:04:04] ...trap/engines/engine.py.224: train: epoch 12 | batch 170/2304 [I 2022-10-26 14:04:04] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:17:49 [I 2022-10-26 14:04:04] ...trap/engines/engine.py.229: process: 0.50322 | load: 0.00303 [I 2022-10-26 14:04:04] ...trap/engines/engine.py.230: loss: 0.35137 [I 2022-10-26 14:04:04] ...optimizers/trijoint.py.151: total_norm: 3.800268 [I 2022-10-26 14:04:09] ...trap/engines/engine.py.224: train: epoch 12 | batch 180/2304 [I 2022-10-26 14:04:09] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:17:35 [I 2022-10-26 14:04:09] ...trap/engines/engine.py.229: process: 0.49330 | load: 0.00268 [I 2022-10-26 14:04:09] ...trap/engines/engine.py.230: loss: 0.17555 [I 2022-10-26 14:04:09] ...optimizers/trijoint.py.151: total_norm: 3.272386 [I 2022-10-26 14:04:14] ...trap/engines/engine.py.224: train: epoch 12 | batch 190/2304 [I 2022-10-26 14:04:14] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:17:31 [I 2022-10-26 14:04:14] ...trap/engines/engine.py.229: process: 0.48565 | load: 0.00269 [I 2022-10-26 14:04:14] ...trap/engines/engine.py.230: loss: 0.20772 [I 2022-10-26 14:04:14] ...optimizers/trijoint.py.151: total_norm: 2.723005 [I 2022-10-26 14:04:19] ...trap/engines/engine.py.224: train: epoch 12 | batch 200/2304 [I 2022-10-26 14:04:19] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:17:22 [I 2022-10-26 14:04:19] ...trap/engines/engine.py.229: process: 0.49191 | load: 0.00267 [I 2022-10-26 14:04:19] ...trap/engines/engine.py.230: loss: 0.17829 [I 2022-10-26 14:04:19] ...optimizers/trijoint.py.151: total_norm: 3.118958 [I 2022-10-26 14:04:24] ...trap/engines/engine.py.224: train: epoch 12 | batch 210/2304 [I 2022-10-26 14:04:24] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:17:11 [I 2022-10-26 14:04:24] ...trap/engines/engine.py.229: process: 0.48964 | load: 0.00262 [I 2022-10-26 14:04:24] ...trap/engines/engine.py.230: loss: 0.23679 [I 2022-10-26 14:04:24] ...optimizers/trijoint.py.151: total_norm: 3.035634 [I 2022-10-26 14:04:29] ...trap/engines/engine.py.224: train: epoch 12 | batch 220/2304 [I 2022-10-26 14:04:29] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:17:25 [I 2022-10-26 14:04:29] ...trap/engines/engine.py.229: process: 0.50675 | load: 0.00273 [I 2022-10-26 14:04:29] ...trap/engines/engine.py.230: loss: 0.24395 [I 2022-10-26 14:04:29] ...optimizers/trijoint.py.151: total_norm: 3.612702 [I 2022-10-26 14:04:34] ...trap/engines/engine.py.224: train: epoch 12 | batch 230/2304 [I 2022-10-26 14:04:34] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:17:12 [I 2022-10-26 14:04:34] ...trap/engines/engine.py.229: process: 0.49681 | load: 0.00253 [I 2022-10-26 14:04:34] ...trap/engines/engine.py.230: loss: 0.19956 [I 2022-10-26 14:04:34] ...optimizers/trijoint.py.151: total_norm: 2.954875 [I 2022-10-26 14:04:39] ...trap/engines/engine.py.224: train: epoch 12 | batch 240/2304 [I 2022-10-26 14:04:39] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:17:09 [I 2022-10-26 14:04:39] ...trap/engines/engine.py.229: process: 0.50507 | load: 0.00249 [I 2022-10-26 14:04:39] ...trap/engines/engine.py.230: loss: 0.17318 [I 2022-10-26 14:04:39] ...optimizers/trijoint.py.151: total_norm: 2.057973 [I 2022-10-26 14:04:44] ...trap/engines/engine.py.224: train: epoch 12 | batch 250/2304 [I 2022-10-26 14:04:44] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:16:49 [I 2022-10-26 14:04:44] ...trap/engines/engine.py.229: process: 0.47710 | load: 0.00257 [I 2022-10-26 14:04:44] ...trap/engines/engine.py.230: loss: 0.18070 [I 2022-10-26 14:04:44] ...optimizers/trijoint.py.151: total_norm: 2.737712 [I 2022-10-26 14:04:49] ...trap/engines/engine.py.224: train: epoch 12 | batch 260/2304 [I 2022-10-26 14:04:49] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:16:56 [I 2022-10-26 14:04:49] ...trap/engines/engine.py.229: process: 0.49728 | load: 0.00257 [I 2022-10-26 14:04:49] ...trap/engines/engine.py.230: loss: 0.21745 [I 2022-10-26 14:04:49] ...optimizers/trijoint.py.151: total_norm: 3.338746 [I 2022-10-26 14:04:54] ...trap/engines/engine.py.224: train: epoch 12 | batch 270/2304 [I 2022-10-26 14:04:54] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:16:57 [I 2022-10-26 14:04:54] ...trap/engines/engine.py.229: process: 0.49921 | load: 0.00261 [I 2022-10-26 14:04:54] ...trap/engines/engine.py.230: loss: 0.36344 [I 2022-10-26 14:04:54] ...optimizers/trijoint.py.151: total_norm: 3.886096 [I 2022-10-26 14:04:59] ...trap/engines/engine.py.224: train: epoch 12 | batch 280/2304 [I 2022-10-26 14:04:59] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:16:46 [I 2022-10-26 14:04:59] ...trap/engines/engine.py.229: process: 0.49013 | load: 0.00256 [I 2022-10-26 14:04:59] ...trap/engines/engine.py.230: loss: 0.22304 [I 2022-10-26 14:04:59] ...optimizers/trijoint.py.151: total_norm: 3.528163 [I 2022-10-26 14:05:04] ...trap/engines/engine.py.224: train: epoch 12 | batch 290/2304 [I 2022-10-26 14:05:04] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:16:45 [I 2022-10-26 14:05:04] ...trap/engines/engine.py.229: process: 0.49225 | load: 0.00259 [I 2022-10-26 14:05:04] ...trap/engines/engine.py.230: loss: 0.23757 [I 2022-10-26 14:05:04] ...optimizers/trijoint.py.151: total_norm: 3.010248 [I 2022-10-26 14:05:09] ...trap/engines/engine.py.224: train: epoch 12 | batch 300/2304 [I 2022-10-26 14:05:09] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:16:32 [I 2022-10-26 14:05:09] ...trap/engines/engine.py.229: process: 0.48825 | load: 0.00251 [I 2022-10-26 14:05:09] ...trap/engines/engine.py.230: loss: 0.27830 [I 2022-10-26 14:05:09] ...optimizers/trijoint.py.151: total_norm: 4.049676 [I 2022-10-26 14:05:14] ...trap/engines/engine.py.224: train: epoch 12 | batch 310/2304 [I 2022-10-26 14:05:14] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:16:51 [I 2022-10-26 14:05:14] ...trap/engines/engine.py.229: process: 0.50152 | load: 0.00257 [I 2022-10-26 14:05:14] ...trap/engines/engine.py.230: loss: 0.15284 [I 2022-10-26 14:05:14] ...optimizers/trijoint.py.151: total_norm: 3.589401 [I 2022-10-26 14:05:19] ...trap/engines/engine.py.224: train: epoch 12 | batch 320/2304 [I 2022-10-26 14:05:19] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:16:30 [I 2022-10-26 14:05:19] ...trap/engines/engine.py.229: process: 0.49986 | load: 0.00261 [I 2022-10-26 14:05:19] ...trap/engines/engine.py.230: loss: 0.20195 [I 2022-10-26 14:05:19] ...optimizers/trijoint.py.151: total_norm: 4.073129 [I 2022-10-26 14:05:24] ...trap/engines/engine.py.224: train: epoch 12 | batch 330/2304 [I 2022-10-26 14:05:24] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:16:23 [I 2022-10-26 14:05:24] ...trap/engines/engine.py.229: process: 0.49061 | load: 0.00261 [I 2022-10-26 14:05:24] ...trap/engines/engine.py.230: loss: 0.20939 [I 2022-10-26 14:05:24] ...optimizers/trijoint.py.151: total_norm: 3.096554 [I 2022-10-26 14:05:29] ...trap/engines/engine.py.224: train: epoch 12 | batch 340/2304 [I 2022-10-26 14:05:29] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:18:17 [I 2022-10-26 14:05:29] ...trap/engines/engine.py.229: process: 0.66651 | load: 0.00256 [I 2022-10-26 14:05:29] ...trap/engines/engine.py.230: loss: 0.21688 [I 2022-10-26 14:05:29] ...optimizers/trijoint.py.151: total_norm: 3.570762 [I 2022-10-26 14:05:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 350/2304 [I 2022-10-26 14:05:36] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:19:02 [I 2022-10-26 14:05:36] ...trap/engines/engine.py.229: process: 0.49176 | load: 0.00258 [I 2022-10-26 14:05:36] ...trap/engines/engine.py.230: loss: 0.17288 [I 2022-10-26 14:05:36] ...optimizers/trijoint.py.151: total_norm: 2.546086 [I 2022-10-26 14:05:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 360/2304 [I 2022-10-26 14:05:41] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:16:32 [I 2022-10-26 14:05:41] ...trap/engines/engine.py.229: process: 0.51045 | load: 0.00256 [I 2022-10-26 14:05:41] ...trap/engines/engine.py.230: loss: 0.20988 [I 2022-10-26 14:05:41] ...optimizers/trijoint.py.151: total_norm: 3.251400 [I 2022-10-26 14:05:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 370/2304 [I 2022-10-26 14:05:46] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:16:03 [I 2022-10-26 14:05:46] ...trap/engines/engine.py.229: process: 0.48268 | load: 0.00254 [I 2022-10-26 14:05:46] ...trap/engines/engine.py.230: loss: 0.14860 [I 2022-10-26 14:05:46] ...optimizers/trijoint.py.151: total_norm: 2.431270 [I 2022-10-26 14:05:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 380/2304 [I 2022-10-26 14:05:51] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:16:04 [I 2022-10-26 14:05:51] ...trap/engines/engine.py.229: process: 0.50491 | load: 0.00261 [I 2022-10-26 14:05:51] ...trap/engines/engine.py.230: loss: 0.24135 [I 2022-10-26 14:05:51] ...optimizers/trijoint.py.151: total_norm: 2.735409 [I 2022-10-26 14:05:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 390/2304 [I 2022-10-26 14:05:56] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:16:02 [I 2022-10-26 14:05:56] ...trap/engines/engine.py.229: process: 0.50299 | load: 0.00259 [I 2022-10-26 14:05:56] ...trap/engines/engine.py.230: loss: 0.23059 [I 2022-10-26 14:05:56] ...optimizers/trijoint.py.151: total_norm: 3.775439 [I 2022-10-26 14:06:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 400/2304 [I 2022-10-26 14:06:01] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:16:01 [I 2022-10-26 14:06:01] ...trap/engines/engine.py.229: process: 0.54050 | load: 0.00254 [I 2022-10-26 14:06:01] ...trap/engines/engine.py.230: loss: 0.21084 [I 2022-10-26 14:06:01] ...optimizers/trijoint.py.151: total_norm: 3.019800 [I 2022-10-26 14:06:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 410/2304 [I 2022-10-26 14:06:06] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:15:53 [I 2022-10-26 14:06:06] ...trap/engines/engine.py.229: process: 0.51686 | load: 0.00271 [I 2022-10-26 14:06:06] ...trap/engines/engine.py.230: loss: 0.14617 [I 2022-10-26 14:06:06] ...optimizers/trijoint.py.151: total_norm: 2.849674 [I 2022-10-26 14:06:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 420/2304 [I 2022-10-26 14:06:11] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:15:40 [I 2022-10-26 14:06:11] ...trap/engines/engine.py.229: process: 0.48591 | load: 0.00256 [I 2022-10-26 14:06:11] ...trap/engines/engine.py.230: loss: 0.29669 [I 2022-10-26 14:06:11] ...optimizers/trijoint.py.151: total_norm: 3.609996 [I 2022-10-26 14:06:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 430/2304 [I 2022-10-26 14:06:16] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:15:42 [I 2022-10-26 14:06:16] ...trap/engines/engine.py.229: process: 0.50075 | load: 0.00259 [I 2022-10-26 14:06:16] ...trap/engines/engine.py.230: loss: 0.17778 [I 2022-10-26 14:06:16] ...optimizers/trijoint.py.151: total_norm: 3.102545 [I 2022-10-26 14:06:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 440/2304 [I 2022-10-26 14:06:21] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:15:31 [I 2022-10-26 14:06:21] ...trap/engines/engine.py.229: process: 0.48950 | load: 0.00253 [I 2022-10-26 14:06:21] ...trap/engines/engine.py.230: loss: 0.15651 [I 2022-10-26 14:06:21] ...optimizers/trijoint.py.151: total_norm: 2.927732 [I 2022-10-26 14:06:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 450/2304 [I 2022-10-26 14:06:26] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:15:27 [I 2022-10-26 14:06:26] ...trap/engines/engine.py.229: process: 0.49972 | load: 0.00273 [I 2022-10-26 14:06:26] ...trap/engines/engine.py.230: loss: 0.24038 [I 2022-10-26 14:06:26] ...optimizers/trijoint.py.151: total_norm: 3.325362 [I 2022-10-26 14:06:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 460/2304 [I 2022-10-26 14:06:31] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:15:20 [I 2022-10-26 14:06:31] ...trap/engines/engine.py.229: process: 0.48821 | load: 0.00251 [I 2022-10-26 14:06:31] ...trap/engines/engine.py.230: loss: 0.14920 [I 2022-10-26 14:06:31] ...optimizers/trijoint.py.151: total_norm: 2.377117 [I 2022-10-26 14:06:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 470/2304 [I 2022-10-26 14:06:36] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:15:21 [I 2022-10-26 14:06:36] ...trap/engines/engine.py.229: process: 0.51337 | load: 0.00249 [I 2022-10-26 14:06:36] ...trap/engines/engine.py.230: loss: 0.28580 [I 2022-10-26 14:06:36] ...optimizers/trijoint.py.151: total_norm: 3.415781 [I 2022-10-26 14:06:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 480/2304 [I 2022-10-26 14:06:41] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:15:19 [I 2022-10-26 14:06:41] ...trap/engines/engine.py.229: process: 0.49394 | load: 0.00242 [I 2022-10-26 14:06:41] ...trap/engines/engine.py.230: loss: 0.30626 [I 2022-10-26 14:06:41] ...optimizers/trijoint.py.151: total_norm: 3.824604 [I 2022-10-26 14:06:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 490/2304 [I 2022-10-26 14:06:46] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:15:07 [I 2022-10-26 14:06:46] ...trap/engines/engine.py.229: process: 0.50024 | load: 0.00244 [I 2022-10-26 14:06:46] ...trap/engines/engine.py.230: loss: 0.27019 [I 2022-10-26 14:06:46] ...optimizers/trijoint.py.151: total_norm: 2.917098 [I 2022-10-26 14:06:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 500/2304 [I 2022-10-26 14:06:51] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:15:09 [I 2022-10-26 14:06:51] ...trap/engines/engine.py.229: process: 0.51477 | load: 0.00243 [I 2022-10-26 14:06:51] ...trap/engines/engine.py.230: loss: 0.23811 [I 2022-10-26 14:06:51] ...optimizers/trijoint.py.151: total_norm: 4.408955 [I 2022-10-26 14:06:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 510/2304 [I 2022-10-26 14:06:56] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:14:59 [I 2022-10-26 14:06:56] ...trap/engines/engine.py.229: process: 0.50668 | load: 0.00242 [I 2022-10-26 14:06:56] ...trap/engines/engine.py.230: loss: 0.22554 [I 2022-10-26 14:06:56] ...optimizers/trijoint.py.151: total_norm: 3.025945 [I 2022-10-26 14:07:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 520/2304 [I 2022-10-26 14:07:01] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:14:40 [I 2022-10-26 14:07:01] ...trap/engines/engine.py.229: process: 0.49940 | load: 0.00241 [I 2022-10-26 14:07:01] ...trap/engines/engine.py.230: loss: 0.25456 [I 2022-10-26 14:07:01] ...optimizers/trijoint.py.151: total_norm: 3.104963 [I 2022-10-26 14:07:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 530/2304 [I 2022-10-26 14:07:06] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:14:45 [I 2022-10-26 14:07:06] ...trap/engines/engine.py.229: process: 0.49881 | load: 0.00268 [I 2022-10-26 14:07:06] ...trap/engines/engine.py.230: loss: 0.21103 [I 2022-10-26 14:07:06] ...optimizers/trijoint.py.151: total_norm: 3.286350 [I 2022-10-26 14:07:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 540/2304 [I 2022-10-26 14:07:11] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:14:46 [I 2022-10-26 14:07:11] ...trap/engines/engine.py.229: process: 0.51205 | load: 0.00255 [I 2022-10-26 14:07:11] ...trap/engines/engine.py.230: loss: 0.19529 [I 2022-10-26 14:07:11] ...optimizers/trijoint.py.151: total_norm: 2.494720 [I 2022-10-26 14:07:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 550/2304 [I 2022-10-26 14:07:16] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:14:39 [I 2022-10-26 14:07:16] ...trap/engines/engine.py.229: process: 0.49334 | load: 0.00247 [I 2022-10-26 14:07:16] ...trap/engines/engine.py.230: loss: 0.27888 [I 2022-10-26 14:07:16] ...optimizers/trijoint.py.151: total_norm: 4.215398 [I 2022-10-26 14:07:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 560/2304 [I 2022-10-26 14:07:21] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:14:26 [I 2022-10-26 14:07:21] ...trap/engines/engine.py.229: process: 0.49719 | load: 0.00253 [I 2022-10-26 14:07:21] ...trap/engines/engine.py.230: loss: 0.23434 [I 2022-10-26 14:07:21] ...optimizers/trijoint.py.151: total_norm: 3.296366 [I 2022-10-26 14:07:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 570/2304 [I 2022-10-26 14:07:26] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:14:29 [I 2022-10-26 14:07:26] ...trap/engines/engine.py.229: process: 0.50327 | load: 0.00253 [I 2022-10-26 14:07:26] ...trap/engines/engine.py.230: loss: 0.15662 [I 2022-10-26 14:07:26] ...optimizers/trijoint.py.151: total_norm: 3.356786 [I 2022-10-26 14:07:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 580/2304 [I 2022-10-26 14:07:31] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:14:16 [I 2022-10-26 14:07:31] ...trap/engines/engine.py.229: process: 0.50276 | load: 0.00253 [I 2022-10-26 14:07:31] ...trap/engines/engine.py.230: loss: 0.13541 [I 2022-10-26 14:07:31] ...optimizers/trijoint.py.151: total_norm: 1.726680 [I 2022-10-26 14:07:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 590/2304 [I 2022-10-26 14:07:36] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:14:18 [I 2022-10-26 14:07:36] ...trap/engines/engine.py.229: process: 0.50963 | load: 0.00260 [I 2022-10-26 14:07:36] ...trap/engines/engine.py.230: loss: 0.15199 [I 2022-10-26 14:07:36] ...optimizers/trijoint.py.151: total_norm: 2.569906 [I 2022-10-26 14:07:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 600/2304 [I 2022-10-26 14:07:41] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:14:09 [I 2022-10-26 14:07:41] ...trap/engines/engine.py.229: process: 0.49079 | load: 0.00250 [I 2022-10-26 14:07:41] ...trap/engines/engine.py.230: loss: 0.29264 [I 2022-10-26 14:07:41] ...optimizers/trijoint.py.151: total_norm: 4.469952 [I 2022-10-26 14:07:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 610/2304 [I 2022-10-26 14:07:46] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:14:07 [I 2022-10-26 14:07:46] ...trap/engines/engine.py.229: process: 0.50022 | load: 0.00248 [I 2022-10-26 14:07:46] ...trap/engines/engine.py.230: loss: 0.19818 [I 2022-10-26 14:07:46] ...optimizers/trijoint.py.151: total_norm: 3.960951 [I 2022-10-26 14:07:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 620/2304 [I 2022-10-26 14:07:51] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:14:06 [I 2022-10-26 14:07:51] ...trap/engines/engine.py.229: process: 0.49781 | load: 0.00252 [I 2022-10-26 14:07:51] ...trap/engines/engine.py.230: loss: 0.23248 [I 2022-10-26 14:07:51] ...optimizers/trijoint.py.151: total_norm: 3.731217 [I 2022-10-26 14:07:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 630/2304 [I 2022-10-26 14:07:56] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:14:02 [I 2022-10-26 14:07:56] ...trap/engines/engine.py.229: process: 0.49924 | load: 0.00245 [I 2022-10-26 14:07:56] ...trap/engines/engine.py.230: loss: 0.20592 [I 2022-10-26 14:07:56] ...optimizers/trijoint.py.151: total_norm: 3.376344 [I 2022-10-26 14:08:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 640/2304 [I 2022-10-26 14:08:01] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:13:50 [I 2022-10-26 14:08:01] ...trap/engines/engine.py.229: process: 0.49522 | load: 0.00258 [I 2022-10-26 14:08:01] ...trap/engines/engine.py.230: loss: 0.24196 [I 2022-10-26 14:08:01] ...optimizers/trijoint.py.151: total_norm: 2.838634 [I 2022-10-26 14:08:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 650/2304 [I 2022-10-26 14:08:06] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:13:40 [I 2022-10-26 14:08:06] ...trap/engines/engine.py.229: process: 0.50495 | load: 0.00252 [I 2022-10-26 14:08:06] ...trap/engines/engine.py.230: loss: 0.15605 [I 2022-10-26 14:08:06] ...optimizers/trijoint.py.151: total_norm: 2.806069 [I 2022-10-26 14:08:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 660/2304 [I 2022-10-26 14:08:11] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:13:40 [I 2022-10-26 14:08:11] ...trap/engines/engine.py.229: process: 0.50244 | load: 0.00246 [I 2022-10-26 14:08:11] ...trap/engines/engine.py.230: loss: 0.21160 [I 2022-10-26 14:08:11] ...optimizers/trijoint.py.151: total_norm: 3.214751 [I 2022-10-26 14:08:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 670/2304 [I 2022-10-26 14:08:16] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:13:31 [I 2022-10-26 14:08:16] ...trap/engines/engine.py.229: process: 0.49848 | load: 0.00243 [I 2022-10-26 14:08:16] ...trap/engines/engine.py.230: loss: 0.26262 [I 2022-10-26 14:08:16] ...optimizers/trijoint.py.151: total_norm: 3.547153 [I 2022-10-26 14:08:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 680/2304 [I 2022-10-26 14:08:21] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:13:24 [I 2022-10-26 14:08:21] ...trap/engines/engine.py.229: process: 0.48722 | load: 0.00266 [I 2022-10-26 14:08:21] ...trap/engines/engine.py.230: loss: 0.19666 [I 2022-10-26 14:08:21] ...optimizers/trijoint.py.151: total_norm: 3.779985 [I 2022-10-26 14:08:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 690/2304 [I 2022-10-26 14:08:26] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:13:47 [I 2022-10-26 14:08:26] ...trap/engines/engine.py.229: process: 0.48549 | load: 0.00248 [I 2022-10-26 14:08:26] ...trap/engines/engine.py.230: loss: 0.26617 [I 2022-10-26 14:08:26] ...optimizers/trijoint.py.151: total_norm: 3.753360 [I 2022-10-26 14:08:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 700/2304 [I 2022-10-26 14:08:31] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:13:26 [I 2022-10-26 14:08:31] ...trap/engines/engine.py.229: process: 0.51129 | load: 0.00243 [I 2022-10-26 14:08:31] ...trap/engines/engine.py.230: loss: 0.23055 [I 2022-10-26 14:08:31] ...optimizers/trijoint.py.151: total_norm: 3.528388 [I 2022-10-26 14:08:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 710/2304 [I 2022-10-26 14:08:36] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:13:17 [I 2022-10-26 14:08:36] ...trap/engines/engine.py.229: process: 0.50146 | load: 0.00242 [I 2022-10-26 14:08:36] ...trap/engines/engine.py.230: loss: 0.24671 [I 2022-10-26 14:08:36] ...optimizers/trijoint.py.151: total_norm: 4.160911 [I 2022-10-26 14:08:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 720/2304 [I 2022-10-26 14:08:41] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:13:11 [I 2022-10-26 14:08:41] ...trap/engines/engine.py.229: process: 0.50438 | load: 0.00243 [I 2022-10-26 14:08:41] ...trap/engines/engine.py.230: loss: 0.15562 [I 2022-10-26 14:08:41] ...optimizers/trijoint.py.151: total_norm: 3.111781 [I 2022-10-26 14:08:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 730/2304 [I 2022-10-26 14:08:46] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:13:06 [I 2022-10-26 14:08:46] ...trap/engines/engine.py.229: process: 0.50006 | load: 0.00242 [I 2022-10-26 14:08:46] ...trap/engines/engine.py.230: loss: 0.20431 [I 2022-10-26 14:08:46] ...optimizers/trijoint.py.151: total_norm: 3.108334 [I 2022-10-26 14:08:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 740/2304 [I 2022-10-26 14:08:51] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:13:05 [I 2022-10-26 14:08:51] ...trap/engines/engine.py.229: process: 0.50703 | load: 0.00242 [I 2022-10-26 14:08:51] ...trap/engines/engine.py.230: loss: 0.21730 [I 2022-10-26 14:08:51] ...optimizers/trijoint.py.151: total_norm: 2.251031 [I 2022-10-26 14:08:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 750/2304 [I 2022-10-26 14:08:56] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:12:51 [I 2022-10-26 14:08:56] ...trap/engines/engine.py.229: process: 0.49776 | load: 0.00241 [I 2022-10-26 14:08:56] ...trap/engines/engine.py.230: loss: 0.26700 [I 2022-10-26 14:08:56] ...optimizers/trijoint.py.151: total_norm: 3.688890 [I 2022-10-26 14:09:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 760/2304 [I 2022-10-26 14:09:01] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:13:22 [I 2022-10-26 14:09:01] ...trap/engines/engine.py.229: process: 0.58378 | load: 0.00264 [I 2022-10-26 14:09:01] ...trap/engines/engine.py.230: loss: 0.28601 [I 2022-10-26 14:09:01] ...optimizers/trijoint.py.151: total_norm: 3.662340 [I 2022-10-26 14:09:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 770/2304 [I 2022-10-26 14:09:06] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:12:51 [I 2022-10-26 14:09:06] ...trap/engines/engine.py.229: process: 0.50026 | load: 0.00245 [I 2022-10-26 14:09:06] ...trap/engines/engine.py.230: loss: 0.19319 [I 2022-10-26 14:09:06] ...optimizers/trijoint.py.151: total_norm: 2.719317 [I 2022-10-26 14:09:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 780/2304 [I 2022-10-26 14:09:11] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:12:38 [I 2022-10-26 14:09:11] ...trap/engines/engine.py.229: process: 0.49225 | load: 0.00253 [I 2022-10-26 14:09:11] ...trap/engines/engine.py.230: loss: 0.23540 [I 2022-10-26 14:09:11] ...optimizers/trijoint.py.151: total_norm: 3.018101 [I 2022-10-26 14:09:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 790/2304 [I 2022-10-26 14:09:16] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:12:34 [I 2022-10-26 14:09:16] ...trap/engines/engine.py.229: process: 0.49991 | load: 0.00247 [I 2022-10-26 14:09:16] ...trap/engines/engine.py.230: loss: 0.23266 [I 2022-10-26 14:09:16] ...optimizers/trijoint.py.151: total_norm: 2.991197 [I 2022-10-26 14:09:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 800/2304 [I 2022-10-26 14:09:21] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:12:29 [I 2022-10-26 14:09:21] ...trap/engines/engine.py.229: process: 0.49964 | load: 0.00248 [I 2022-10-26 14:09:21] ...trap/engines/engine.py.230: loss: 0.23458 [I 2022-10-26 14:09:21] ...optimizers/trijoint.py.151: total_norm: 3.542912 [I 2022-10-26 14:09:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 810/2304 [I 2022-10-26 14:09:26] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:12:25 [I 2022-10-26 14:09:26] ...trap/engines/engine.py.229: process: 0.50531 | load: 0.00243 [I 2022-10-26 14:09:26] ...trap/engines/engine.py.230: loss: 0.21095 [I 2022-10-26 14:09:26] ...optimizers/trijoint.py.151: total_norm: 3.517018 [I 2022-10-26 14:09:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 820/2304 [I 2022-10-26 14:09:31] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:12:13 [I 2022-10-26 14:09:31] ...trap/engines/engine.py.229: process: 0.48762 | load: 0.00243 [I 2022-10-26 14:09:31] ...trap/engines/engine.py.230: loss: 0.16523 [I 2022-10-26 14:09:31] ...optimizers/trijoint.py.151: total_norm: 2.831336 [I 2022-10-26 14:09:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 830/2304 [I 2022-10-26 14:09:36] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:12:32 [I 2022-10-26 14:09:36] ...trap/engines/engine.py.229: process: 0.50418 | load: 0.00276 [I 2022-10-26 14:09:36] ...trap/engines/engine.py.230: loss: 0.17338 [I 2022-10-26 14:09:36] ...optimizers/trijoint.py.151: total_norm: 3.168955 [I 2022-10-26 14:09:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 840/2304 [I 2022-10-26 14:09:41] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:12:03 [I 2022-10-26 14:09:41] ...trap/engines/engine.py.229: process: 0.50015 | load: 0.00246 [I 2022-10-26 14:09:41] ...trap/engines/engine.py.230: loss: 0.21739 [I 2022-10-26 14:09:41] ...optimizers/trijoint.py.151: total_norm: 3.772100 [I 2022-10-26 14:09:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 850/2304 [I 2022-10-26 14:09:46] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:12:03 [I 2022-10-26 14:09:46] ...trap/engines/engine.py.229: process: 0.49864 | load: 0.00253 [I 2022-10-26 14:09:46] ...trap/engines/engine.py.230: loss: 0.20323 [I 2022-10-26 14:09:46] ...optimizers/trijoint.py.151: total_norm: 3.961903 [I 2022-10-26 14:09:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 860/2304 [I 2022-10-26 14:09:51] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:11:59 [I 2022-10-26 14:09:51] ...trap/engines/engine.py.229: process: 0.51451 | load: 0.00251 [I 2022-10-26 14:09:51] ...trap/engines/engine.py.230: loss: 0.22764 [I 2022-10-26 14:09:51] ...optimizers/trijoint.py.151: total_norm: 2.659730 [I 2022-10-26 14:09:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 870/2304 [I 2022-10-26 14:09:56] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:11:54 [I 2022-10-26 14:09:56] ...trap/engines/engine.py.229: process: 0.50196 | load: 0.00243 [I 2022-10-26 14:09:56] ...trap/engines/engine.py.230: loss: 0.29726 [I 2022-10-26 14:09:56] ...optimizers/trijoint.py.151: total_norm: 4.152384 [I 2022-10-26 14:10:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 880/2304 [I 2022-10-26 14:10:01] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:11:51 [I 2022-10-26 14:10:01] ...trap/engines/engine.py.229: process: 0.50165 | load: 0.00250 [I 2022-10-26 14:10:01] ...trap/engines/engine.py.230: loss: 0.30446 [I 2022-10-26 14:10:01] ...optimizers/trijoint.py.151: total_norm: 3.403700 [I 2022-10-26 14:10:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 890/2304 [I 2022-10-26 14:10:06] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:11:57 [I 2022-10-26 14:10:06] ...trap/engines/engine.py.229: process: 0.50089 | load: 0.00242 [I 2022-10-26 14:10:06] ...trap/engines/engine.py.230: loss: 0.20762 [I 2022-10-26 14:10:06] ...optimizers/trijoint.py.151: total_norm: 2.665041 [I 2022-10-26 14:10:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 900/2304 [I 2022-10-26 14:10:11] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:11:38 [I 2022-10-26 14:10:11] ...trap/engines/engine.py.229: process: 0.50066 | load: 0.00241 [I 2022-10-26 14:10:11] ...trap/engines/engine.py.230: loss: 0.20077 [I 2022-10-26 14:10:11] ...optimizers/trijoint.py.151: total_norm: 3.917104 [I 2022-10-26 14:10:17] ...trap/engines/engine.py.224: train: epoch 12 | batch 910/2304 [I 2022-10-26 14:10:17] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:11:40 [I 2022-10-26 14:10:17] ...trap/engines/engine.py.229: process: 0.49150 | load: 0.00256 [I 2022-10-26 14:10:17] ...trap/engines/engine.py.230: loss: 0.22420 [I 2022-10-26 14:10:17] ...optimizers/trijoint.py.151: total_norm: 2.939279 [I 2022-10-26 14:10:22] ...trap/engines/engine.py.224: train: epoch 12 | batch 920/2304 [I 2022-10-26 14:10:22] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:11:28 [I 2022-10-26 14:10:22] ...trap/engines/engine.py.229: process: 0.50588 | load: 0.00250 [I 2022-10-26 14:10:22] ...trap/engines/engine.py.230: loss: 0.28006 [I 2022-10-26 14:10:22] ...optimizers/trijoint.py.151: total_norm: 3.724679 [I 2022-10-26 14:10:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 930/2304 [I 2022-10-26 14:10:26] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:11:19 [I 2022-10-26 14:10:26] ...trap/engines/engine.py.229: process: 0.49149 | load: 0.00247 [I 2022-10-26 14:10:26] ...trap/engines/engine.py.230: loss: 0.28872 [I 2022-10-26 14:10:26] ...optimizers/trijoint.py.151: total_norm: 3.997232 [I 2022-10-26 14:10:32] ...trap/engines/engine.py.224: train: epoch 12 | batch 940/2304 [I 2022-10-26 14:10:32] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:11:20 [I 2022-10-26 14:10:32] ...trap/engines/engine.py.229: process: 0.48819 | load: 0.00244 [I 2022-10-26 14:10:32] ...trap/engines/engine.py.230: loss: 0.25472 [I 2022-10-26 14:10:32] ...optimizers/trijoint.py.151: total_norm: 3.412468 [I 2022-10-26 14:10:37] ...trap/engines/engine.py.224: train: epoch 12 | batch 950/2304 [I 2022-10-26 14:10:37] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:11:17 [I 2022-10-26 14:10:37] ...trap/engines/engine.py.229: process: 0.49601 | load: 0.00241 [I 2022-10-26 14:10:37] ...trap/engines/engine.py.230: loss: 0.14068 [I 2022-10-26 14:10:37] ...optimizers/trijoint.py.151: total_norm: 2.463732 [I 2022-10-26 14:10:42] ...trap/engines/engine.py.224: train: epoch 12 | batch 960/2304 [I 2022-10-26 14:10:42] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:11:07 [I 2022-10-26 14:10:42] ...trap/engines/engine.py.229: process: 0.49146 | load: 0.00244 [I 2022-10-26 14:10:42] ...trap/engines/engine.py.230: loss: 0.25408 [I 2022-10-26 14:10:42] ...optimizers/trijoint.py.151: total_norm: 3.207088 [I 2022-10-26 14:10:47] ...trap/engines/engine.py.224: train: epoch 12 | batch 970/2304 [I 2022-10-26 14:10:47] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:11:06 [I 2022-10-26 14:10:47] ...trap/engines/engine.py.229: process: 0.50394 | load: 0.00246 [I 2022-10-26 14:10:47] ...trap/engines/engine.py.230: loss: 0.28597 [I 2022-10-26 14:10:47] ...optimizers/trijoint.py.151: total_norm: 3.366965 [I 2022-10-26 14:10:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 980/2304 [I 2022-10-26 14:10:51] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:10:58 [I 2022-10-26 14:10:51] ...trap/engines/engine.py.229: process: 0.50365 | load: 0.00248 [I 2022-10-26 14:10:51] ...trap/engines/engine.py.230: loss: 0.15368 [I 2022-10-26 14:10:51] ...optimizers/trijoint.py.151: total_norm: 2.318839 [I 2022-10-26 14:10:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 990/2304 [I 2022-10-26 14:10:56] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:10:55 [I 2022-10-26 14:10:56] ...trap/engines/engine.py.229: process: 0.50489 | load: 0.00243 [I 2022-10-26 14:10:56] ...trap/engines/engine.py.230: loss: 0.27440 [I 2022-10-26 14:10:56] ...optimizers/trijoint.py.151: total_norm: 3.727490 [I 2022-10-26 14:11:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 1000/2304 [I 2022-10-26 14:11:01] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:10:46 [I 2022-10-26 14:11:01] ...trap/engines/engine.py.229: process: 0.49631 | load: 0.00243 [I 2022-10-26 14:11:01] ...trap/engines/engine.py.230: loss: 0.18754 [I 2022-10-26 14:11:01] ...optimizers/trijoint.py.151: total_norm: 3.848893 [I 2022-10-26 14:11:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1010/2304 [I 2022-10-26 14:11:06] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:10:41 [I 2022-10-26 14:11:06] ...trap/engines/engine.py.229: process: 0.48995 | load: 0.00245 [I 2022-10-26 14:11:06] ...trap/engines/engine.py.230: loss: 0.19740 [I 2022-10-26 14:11:06] ...optimizers/trijoint.py.151: total_norm: 2.599013 [I 2022-10-26 14:11:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 1020/2304 [I 2022-10-26 14:11:11] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:10:35 [I 2022-10-26 14:11:11] ...trap/engines/engine.py.229: process: 0.48391 | load: 0.00242 [I 2022-10-26 14:11:11] ...trap/engines/engine.py.230: loss: 0.25231 [I 2022-10-26 14:11:11] ...optimizers/trijoint.py.151: total_norm: 4.156709 [I 2022-10-26 14:11:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 1030/2304 [I 2022-10-26 14:11:16] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:10:33 [I 2022-10-26 14:11:16] ...trap/engines/engine.py.229: process: 0.48330 | load: 0.00246 [I 2022-10-26 14:11:16] ...trap/engines/engine.py.230: loss: 0.25895 [I 2022-10-26 14:11:16] ...optimizers/trijoint.py.151: total_norm: 3.853034 [I 2022-10-26 14:11:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 1040/2304 [I 2022-10-26 14:11:21] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:10:32 [I 2022-10-26 14:11:21] ...trap/engines/engine.py.229: process: 0.51051 | load: 0.00240 [I 2022-10-26 14:11:21] ...trap/engines/engine.py.230: loss: 0.38442 [I 2022-10-26 14:11:21] ...optimizers/trijoint.py.151: total_norm: 4.390299 [I 2022-10-26 14:11:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 1050/2304 [I 2022-10-26 14:11:26] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:10:22 [I 2022-10-26 14:11:26] ...trap/engines/engine.py.229: process: 0.49150 | load: 0.00248 [I 2022-10-26 14:11:26] ...trap/engines/engine.py.230: loss: 0.21037 [I 2022-10-26 14:11:26] ...optimizers/trijoint.py.151: total_norm: 3.783158 [I 2022-10-26 14:11:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 1060/2304 [I 2022-10-26 14:11:31] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:10:17 [I 2022-10-26 14:11:31] ...trap/engines/engine.py.229: process: 0.49130 | load: 0.00249 [I 2022-10-26 14:11:31] ...trap/engines/engine.py.230: loss: 0.26461 [I 2022-10-26 14:11:31] ...optimizers/trijoint.py.151: total_norm: 3.874287 [I 2022-10-26 14:11:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 1070/2304 [I 2022-10-26 14:11:36] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:10:22 [I 2022-10-26 14:11:36] ...trap/engines/engine.py.229: process: 0.50719 | load: 0.00241 [I 2022-10-26 14:11:36] ...trap/engines/engine.py.230: loss: 0.21775 [I 2022-10-26 14:11:36] ...optimizers/trijoint.py.151: total_norm: 4.217770 [I 2022-10-26 14:11:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 1080/2304 [I 2022-10-26 14:11:41] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:10:08 [I 2022-10-26 14:11:41] ...trap/engines/engine.py.229: process: 0.49071 | load: 0.00243 [I 2022-10-26 14:11:41] ...trap/engines/engine.py.230: loss: 0.26820 [I 2022-10-26 14:11:41] ...optimizers/trijoint.py.151: total_norm: 3.223044 [I 2022-10-26 14:11:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 1090/2304 [I 2022-10-26 14:11:46] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:10:07 [I 2022-10-26 14:11:46] ...trap/engines/engine.py.229: process: 0.50323 | load: 0.00247 [I 2022-10-26 14:11:46] ...trap/engines/engine.py.230: loss: 0.27302 [I 2022-10-26 14:11:46] ...optimizers/trijoint.py.151: total_norm: 3.186291 [I 2022-10-26 14:11:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 1100/2304 [I 2022-10-26 14:11:51] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:09:50 [I 2022-10-26 14:11:51] ...trap/engines/engine.py.229: process: 0.48534 | load: 0.00253 [I 2022-10-26 14:11:51] ...trap/engines/engine.py.230: loss: 0.28745 [I 2022-10-26 14:11:51] ...optimizers/trijoint.py.151: total_norm: 4.128971 [I 2022-10-26 14:11:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 1110/2304 [I 2022-10-26 14:11:56] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:09:53 [I 2022-10-26 14:11:56] ...trap/engines/engine.py.229: process: 0.49733 | load: 0.00255 [I 2022-10-26 14:11:56] ...trap/engines/engine.py.230: loss: 0.20062 [I 2022-10-26 14:11:56] ...optimizers/trijoint.py.151: total_norm: 3.099336 [I 2022-10-26 14:12:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 1120/2304 [I 2022-10-26 14:12:01] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:09:47 [I 2022-10-26 14:12:01] ...trap/engines/engine.py.229: process: 0.48729 | load: 0.00254 [I 2022-10-26 14:12:01] ...trap/engines/engine.py.230: loss: 0.19867 [I 2022-10-26 14:12:01] ...optimizers/trijoint.py.151: total_norm: 2.226668 [I 2022-10-26 14:12:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1130/2304 [I 2022-10-26 14:12:06] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:09:47 [I 2022-10-26 14:12:06] ...trap/engines/engine.py.229: process: 0.50409 | load: 0.00242 [I 2022-10-26 14:12:06] ...trap/engines/engine.py.230: loss: 0.21964 [I 2022-10-26 14:12:06] ...optimizers/trijoint.py.151: total_norm: 3.245501 [I 2022-10-26 14:12:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 1140/2304 [I 2022-10-26 14:12:11] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:09:43 [I 2022-10-26 14:12:11] ...trap/engines/engine.py.229: process: 0.51788 | load: 0.00250 [I 2022-10-26 14:12:11] ...trap/engines/engine.py.230: loss: 0.17015 [I 2022-10-26 14:12:11] ...optimizers/trijoint.py.151: total_norm: 3.210352 [I 2022-10-26 14:12:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 1150/2304 [I 2022-10-26 14:12:16] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:09:38 [I 2022-10-26 14:12:16] ...trap/engines/engine.py.229: process: 0.49561 | load: 0.00249 [I 2022-10-26 14:12:16] ...trap/engines/engine.py.230: loss: 0.23025 [I 2022-10-26 14:12:16] ...optimizers/trijoint.py.151: total_norm: 4.271689 [I 2022-10-26 14:12:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 1160/2304 [I 2022-10-26 14:12:21] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:09:30 [I 2022-10-26 14:12:21] ...trap/engines/engine.py.229: process: 0.49494 | load: 0.00242 [I 2022-10-26 14:12:21] ...trap/engines/engine.py.230: loss: 0.20165 [I 2022-10-26 14:12:21] ...optimizers/trijoint.py.151: total_norm: 2.606735 [I 2022-10-26 14:12:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 1170/2304 [I 2022-10-26 14:12:26] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:09:22 [I 2022-10-26 14:12:26] ...trap/engines/engine.py.229: process: 0.50065 | load: 0.00244 [I 2022-10-26 14:12:26] ...trap/engines/engine.py.230: loss: 0.25263 [I 2022-10-26 14:12:26] ...optimizers/trijoint.py.151: total_norm: 3.149131 [I 2022-10-26 14:12:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 1180/2304 [I 2022-10-26 14:12:31] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:09:13 [I 2022-10-26 14:12:31] ...trap/engines/engine.py.229: process: 0.48497 | load: 0.00244 [I 2022-10-26 14:12:31] ...trap/engines/engine.py.230: loss: 0.15777 [I 2022-10-26 14:12:31] ...optimizers/trijoint.py.151: total_norm: 2.625856 [I 2022-10-26 14:12:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 1190/2304 [I 2022-10-26 14:12:36] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:09:13 [I 2022-10-26 14:12:36] ...trap/engines/engine.py.229: process: 0.49445 | load: 0.00252 [I 2022-10-26 14:12:36] ...trap/engines/engine.py.230: loss: 0.30333 [I 2022-10-26 14:12:36] ...optimizers/trijoint.py.151: total_norm: 3.951209 [I 2022-10-26 14:12:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 1200/2304 [I 2022-10-26 14:12:41] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:09:06 [I 2022-10-26 14:12:41] ...trap/engines/engine.py.229: process: 0.48906 | load: 0.00246 [I 2022-10-26 14:12:41] ...trap/engines/engine.py.230: loss: 0.23634 [I 2022-10-26 14:12:41] ...optimizers/trijoint.py.151: total_norm: 3.306688 [I 2022-10-26 14:12:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 1210/2304 [I 2022-10-26 14:12:46] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:09:07 [I 2022-10-26 14:12:46] ...trap/engines/engine.py.229: process: 0.50395 | load: 0.00244 [I 2022-10-26 14:12:46] ...trap/engines/engine.py.230: loss: 0.16326 [I 2022-10-26 14:12:46] ...optimizers/trijoint.py.151: total_norm: 3.314121 [I 2022-10-26 14:12:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 1220/2304 [I 2022-10-26 14:12:51] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:08:57 [I 2022-10-26 14:12:51] ...trap/engines/engine.py.229: process: 0.49476 | load: 0.00248 [I 2022-10-26 14:12:51] ...trap/engines/engine.py.230: loss: 0.23952 [I 2022-10-26 14:12:51] ...optimizers/trijoint.py.151: total_norm: 3.838089 [I 2022-10-26 14:12:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 1230/2304 [I 2022-10-26 14:12:56] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:08:52 [I 2022-10-26 14:12:56] ...trap/engines/engine.py.229: process: 0.50382 | load: 0.00242 [I 2022-10-26 14:12:56] ...trap/engines/engine.py.230: loss: 0.20760 [I 2022-10-26 14:12:56] ...optimizers/trijoint.py.151: total_norm: 3.378224 [I 2022-10-26 14:13:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 1240/2304 [I 2022-10-26 14:13:01] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:08:47 [I 2022-10-26 14:13:01] ...trap/engines/engine.py.229: process: 0.49497 | load: 0.00241 [I 2022-10-26 14:13:01] ...trap/engines/engine.py.230: loss: 0.22541 [I 2022-10-26 14:13:01] ...optimizers/trijoint.py.151: total_norm: 2.955543 [I 2022-10-26 14:13:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1250/2304 [I 2022-10-26 14:13:06] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:08:46 [I 2022-10-26 14:13:06] ...trap/engines/engine.py.229: process: 0.50135 | load: 0.00244 [I 2022-10-26 14:13:06] ...trap/engines/engine.py.230: loss: 0.20044 [I 2022-10-26 14:13:06] ...optimizers/trijoint.py.151: total_norm: 3.659566 [I 2022-10-26 14:13:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 1260/2304 [I 2022-10-26 14:13:11] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:08:37 [I 2022-10-26 14:13:11] ...trap/engines/engine.py.229: process: 0.49388 | load: 0.00242 [I 2022-10-26 14:13:11] ...trap/engines/engine.py.230: loss: 0.23628 [I 2022-10-26 14:13:11] ...optimizers/trijoint.py.151: total_norm: 4.311400 [I 2022-10-26 14:13:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 1270/2304 [I 2022-10-26 14:13:16] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:08:32 [I 2022-10-26 14:13:16] ...trap/engines/engine.py.229: process: 0.48608 | load: 0.00244 [I 2022-10-26 14:13:16] ...trap/engines/engine.py.230: loss: 0.13113 [I 2022-10-26 14:13:16] ...optimizers/trijoint.py.151: total_norm: 1.987982 [I 2022-10-26 14:13:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 1280/2304 [I 2022-10-26 14:13:21] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:08:27 [I 2022-10-26 14:13:21] ...trap/engines/engine.py.229: process: 0.49493 | load: 0.00243 [I 2022-10-26 14:13:21] ...trap/engines/engine.py.230: loss: 0.20070 [I 2022-10-26 14:13:21] ...optimizers/trijoint.py.151: total_norm: 3.023957 [I 2022-10-26 14:13:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 1290/2304 [I 2022-10-26 14:13:26] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:08:24 [I 2022-10-26 14:13:26] ...trap/engines/engine.py.229: process: 0.50113 | load: 0.00241 [I 2022-10-26 14:13:26] ...trap/engines/engine.py.230: loss: 0.24001 [I 2022-10-26 14:13:26] ...optimizers/trijoint.py.151: total_norm: 3.620362 [I 2022-10-26 14:13:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 1300/2304 [I 2022-10-26 14:13:31] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:08:19 [I 2022-10-26 14:13:31] ...trap/engines/engine.py.229: process: 0.50500 | load: 0.00241 [I 2022-10-26 14:13:31] ...trap/engines/engine.py.230: loss: 0.23309 [I 2022-10-26 14:13:31] ...optimizers/trijoint.py.151: total_norm: 3.250008 [I 2022-10-26 14:13:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 1310/2304 [I 2022-10-26 14:13:36] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:08:10 [I 2022-10-26 14:13:36] ...trap/engines/engine.py.229: process: 0.49459 | load: 0.00242 [I 2022-10-26 14:13:36] ...trap/engines/engine.py.230: loss: 0.24797 [I 2022-10-26 14:13:36] ...optimizers/trijoint.py.151: total_norm: 3.216200 [I 2022-10-26 14:13:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 1320/2304 [I 2022-10-26 14:13:41] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:08:10 [I 2022-10-26 14:13:41] ...trap/engines/engine.py.229: process: 0.50247 | load: 0.00242 [I 2022-10-26 14:13:41] ...trap/engines/engine.py.230: loss: 0.21657 [I 2022-10-26 14:13:41] ...optimizers/trijoint.py.151: total_norm: 3.166987 [I 2022-10-26 14:13:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 1330/2304 [I 2022-10-26 14:13:46] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:08:01 [I 2022-10-26 14:13:46] ...trap/engines/engine.py.229: process: 0.48971 | load: 0.00240 [I 2022-10-26 14:13:46] ...trap/engines/engine.py.230: loss: 0.25567 [I 2022-10-26 14:13:46] ...optimizers/trijoint.py.151: total_norm: 3.211944 [I 2022-10-26 14:13:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 1340/2304 [I 2022-10-26 14:13:51] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:07:56 [I 2022-10-26 14:13:51] ...trap/engines/engine.py.229: process: 0.48525 | load: 0.00241 [I 2022-10-26 14:13:51] ...trap/engines/engine.py.230: loss: 0.22693 [I 2022-10-26 14:13:51] ...optimizers/trijoint.py.151: total_norm: 2.745727 [I 2022-10-26 14:13:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 1350/2304 [I 2022-10-26 14:13:56] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:07:55 [I 2022-10-26 14:13:56] ...trap/engines/engine.py.229: process: 0.50691 | load: 0.00241 [I 2022-10-26 14:13:56] ...trap/engines/engine.py.230: loss: 0.23220 [I 2022-10-26 14:13:56] ...optimizers/trijoint.py.151: total_norm: 3.301352 [I 2022-10-26 14:14:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 1360/2304 [I 2022-10-26 14:14:01] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:07:52 [I 2022-10-26 14:14:01] ...trap/engines/engine.py.229: process: 0.50286 | load: 0.00241 [I 2022-10-26 14:14:01] ...trap/engines/engine.py.230: loss: 0.12902 [I 2022-10-26 14:14:01] ...optimizers/trijoint.py.151: total_norm: 2.147871 [I 2022-10-26 14:14:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1370/2304 [I 2022-10-26 14:14:06] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:07:41 [I 2022-10-26 14:14:06] ...trap/engines/engine.py.229: process: 0.49404 | load: 0.00245 [I 2022-10-26 14:14:06] ...trap/engines/engine.py.230: loss: 0.27494 [I 2022-10-26 14:14:06] ...optimizers/trijoint.py.151: total_norm: 3.335083 [I 2022-10-26 14:14:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 1380/2304 [I 2022-10-26 14:14:11] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:07:33 [I 2022-10-26 14:14:11] ...trap/engines/engine.py.229: process: 0.48426 | load: 0.00239 [I 2022-10-26 14:14:11] ...trap/engines/engine.py.230: loss: 0.19513 [I 2022-10-26 14:14:11] ...optimizers/trijoint.py.151: total_norm: 4.007624 [I 2022-10-26 14:14:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 1390/2304 [I 2022-10-26 14:14:16] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:07:30 [I 2022-10-26 14:14:16] ...trap/engines/engine.py.229: process: 0.48641 | load: 0.00243 [I 2022-10-26 14:14:16] ...trap/engines/engine.py.230: loss: 0.18528 [I 2022-10-26 14:14:16] ...optimizers/trijoint.py.151: total_norm: 4.265919 [I 2022-10-26 14:14:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 1400/2304 [I 2022-10-26 14:14:21] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:07:32 [I 2022-10-26 14:14:21] ...trap/engines/engine.py.229: process: 0.50835 | load: 0.00259 [I 2022-10-26 14:14:21] ...trap/engines/engine.py.230: loss: 0.37859 [I 2022-10-26 14:14:21] ...optimizers/trijoint.py.151: total_norm: 5.254360 [I 2022-10-26 14:14:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 1410/2304 [I 2022-10-26 14:14:26] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:07:26 [I 2022-10-26 14:14:26] ...trap/engines/engine.py.229: process: 0.49813 | load: 0.00248 [I 2022-10-26 14:14:26] ...trap/engines/engine.py.230: loss: 0.24570 [I 2022-10-26 14:14:26] ...optimizers/trijoint.py.151: total_norm: 3.526329 [I 2022-10-26 14:14:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 1420/2304 [I 2022-10-26 14:14:31] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:07:16 [I 2022-10-26 14:14:31] ...trap/engines/engine.py.229: process: 0.49568 | load: 0.00245 [I 2022-10-26 14:14:31] ...trap/engines/engine.py.230: loss: 0.18681 [I 2022-10-26 14:14:31] ...optimizers/trijoint.py.151: total_norm: 2.830213 [I 2022-10-26 14:14:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 1430/2304 [I 2022-10-26 14:14:36] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:07:13 [I 2022-10-26 14:14:36] ...trap/engines/engine.py.229: process: 0.48214 | load: 0.00245 [I 2022-10-26 14:14:36] ...trap/engines/engine.py.230: loss: 0.22130 [I 2022-10-26 14:14:36] ...optimizers/trijoint.py.151: total_norm: 2.977607 [I 2022-10-26 14:14:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 1440/2304 [I 2022-10-26 14:14:41] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:07:09 [I 2022-10-26 14:14:41] ...trap/engines/engine.py.229: process: 0.48689 | load: 0.00254 [I 2022-10-26 14:14:41] ...trap/engines/engine.py.230: loss: 0.20856 [I 2022-10-26 14:14:41] ...optimizers/trijoint.py.151: total_norm: 4.684461 [I 2022-10-26 14:14:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 1450/2304 [I 2022-10-26 14:14:46] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:07:04 [I 2022-10-26 14:14:46] ...trap/engines/engine.py.229: process: 0.49874 | load: 0.00241 [I 2022-10-26 14:14:46] ...trap/engines/engine.py.230: loss: 0.26597 [I 2022-10-26 14:14:46] ...optimizers/trijoint.py.151: total_norm: 3.950705 [I 2022-10-26 14:14:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 1460/2304 [I 2022-10-26 14:14:51] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:06:56 [I 2022-10-26 14:14:51] ...trap/engines/engine.py.229: process: 0.49258 | load: 0.00244 [I 2022-10-26 14:14:51] ...trap/engines/engine.py.230: loss: 0.28304 [I 2022-10-26 14:14:51] ...optimizers/trijoint.py.151: total_norm: 3.122202 [I 2022-10-26 14:14:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 1470/2304 [I 2022-10-26 14:14:56] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:06:55 [I 2022-10-26 14:14:56] ...trap/engines/engine.py.229: process: 0.50753 | load: 0.00255 [I 2022-10-26 14:14:56] ...trap/engines/engine.py.230: loss: 0.25841 [I 2022-10-26 14:14:56] ...optimizers/trijoint.py.151: total_norm: 3.027546 [I 2022-10-26 14:15:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 1480/2304 [I 2022-10-26 14:15:01] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:06:52 [I 2022-10-26 14:15:01] ...trap/engines/engine.py.229: process: 0.50889 | load: 0.00247 [I 2022-10-26 14:15:01] ...trap/engines/engine.py.230: loss: 0.26220 [I 2022-10-26 14:15:01] ...optimizers/trijoint.py.151: total_norm: 3.046418 [I 2022-10-26 14:15:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1490/2304 [I 2022-10-26 14:15:06] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:06:44 [I 2022-10-26 14:15:06] ...trap/engines/engine.py.229: process: 0.49873 | load: 0.00248 [I 2022-10-26 14:15:06] ...trap/engines/engine.py.230: loss: 0.21949 [I 2022-10-26 14:15:06] ...optimizers/trijoint.py.151: total_norm: 3.538990 [I 2022-10-26 14:15:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 1500/2304 [I 2022-10-26 14:15:11] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:06:38 [I 2022-10-26 14:15:11] ...trap/engines/engine.py.229: process: 0.48986 | load: 0.00247 [I 2022-10-26 14:15:11] ...trap/engines/engine.py.230: loss: 0.16386 [I 2022-10-26 14:15:11] ...optimizers/trijoint.py.151: total_norm: 2.255863 [I 2022-10-26 14:15:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 1510/2304 [I 2022-10-26 14:15:16] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:06:33 [I 2022-10-26 14:15:16] ...trap/engines/engine.py.229: process: 0.49244 | load: 0.00251 [I 2022-10-26 14:15:16] ...trap/engines/engine.py.230: loss: 0.25735 [I 2022-10-26 14:15:16] ...optimizers/trijoint.py.151: total_norm: 3.068209 [I 2022-10-26 14:15:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 1520/2304 [I 2022-10-26 14:15:21] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:06:47 [I 2022-10-26 14:15:21] ...trap/engines/engine.py.229: process: 0.54103 | load: 0.00245 [I 2022-10-26 14:15:21] ...trap/engines/engine.py.230: loss: 0.28366 [I 2022-10-26 14:15:21] ...optimizers/trijoint.py.151: total_norm: 3.770096 [I 2022-10-26 14:15:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 1530/2304 [I 2022-10-26 14:15:26] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:06:29 [I 2022-10-26 14:15:26] ...trap/engines/engine.py.229: process: 0.50441 | load: 0.00248 [I 2022-10-26 14:15:26] ...trap/engines/engine.py.230: loss: 0.32528 [I 2022-10-26 14:15:26] ...optimizers/trijoint.py.151: total_norm: 4.159043 [I 2022-10-26 14:15:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 1540/2304 [I 2022-10-26 14:15:31] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:06:16 [I 2022-10-26 14:15:31] ...trap/engines/engine.py.229: process: 0.48579 | load: 0.00248 [I 2022-10-26 14:15:31] ...trap/engines/engine.py.230: loss: 0.17261 [I 2022-10-26 14:15:31] ...optimizers/trijoint.py.151: total_norm: 2.365309 [I 2022-10-26 14:15:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 1550/2304 [I 2022-10-26 14:15:36] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:06:22 [I 2022-10-26 14:15:36] ...trap/engines/engine.py.229: process: 0.50325 | load: 0.00245 [I 2022-10-26 14:15:36] ...trap/engines/engine.py.230: loss: 0.16070 [I 2022-10-26 14:15:36] ...optimizers/trijoint.py.151: total_norm: 2.655119 [I 2022-10-26 14:15:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 1560/2304 [I 2022-10-26 14:15:41] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:06:10 [I 2022-10-26 14:15:41] ...trap/engines/engine.py.229: process: 0.50278 | load: 0.00245 [I 2022-10-26 14:15:41] ...trap/engines/engine.py.230: loss: 0.19318 [I 2022-10-26 14:15:41] ...optimizers/trijoint.py.151: total_norm: 3.069884 [I 2022-10-26 14:15:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 1570/2304 [I 2022-10-26 14:15:46] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:06:05 [I 2022-10-26 14:15:46] ...trap/engines/engine.py.229: process: 0.50320 | load: 0.00240 [I 2022-10-26 14:15:46] ...trap/engines/engine.py.230: loss: 0.26436 [I 2022-10-26 14:15:46] ...optimizers/trijoint.py.151: total_norm: 3.403149 [I 2022-10-26 14:15:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 1580/2304 [I 2022-10-26 14:15:51] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:06:01 [I 2022-10-26 14:15:51] ...trap/engines/engine.py.229: process: 0.50339 | load: 0.00254 [I 2022-10-26 14:15:51] ...trap/engines/engine.py.230: loss: 0.29989 [I 2022-10-26 14:15:51] ...optimizers/trijoint.py.151: total_norm: 3.547876 [I 2022-10-26 14:15:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 1590/2304 [I 2022-10-26 14:15:56] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:05:56 [I 2022-10-26 14:15:56] ...trap/engines/engine.py.229: process: 0.50451 | load: 0.00250 [I 2022-10-26 14:15:56] ...trap/engines/engine.py.230: loss: 0.27372 [I 2022-10-26 14:15:56] ...optimizers/trijoint.py.151: total_norm: 3.786636 [I 2022-10-26 14:16:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 1600/2304 [I 2022-10-26 14:16:01] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:05:50 [I 2022-10-26 14:16:01] ...trap/engines/engine.py.229: process: 0.49336 | load: 0.00246 [I 2022-10-26 14:16:01] ...trap/engines/engine.py.230: loss: 0.26931 [I 2022-10-26 14:16:01] ...optimizers/trijoint.py.151: total_norm: 3.569734 [I 2022-10-26 14:16:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1610/2304 [I 2022-10-26 14:16:06] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:05:46 [I 2022-10-26 14:16:06] ...trap/engines/engine.py.229: process: 0.49455 | load: 0.00246 [I 2022-10-26 14:16:06] ...trap/engines/engine.py.230: loss: 0.24854 [I 2022-10-26 14:16:06] ...optimizers/trijoint.py.151: total_norm: 4.211622 [I 2022-10-26 14:16:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 1620/2304 [I 2022-10-26 14:16:11] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:05:42 [I 2022-10-26 14:16:11] ...trap/engines/engine.py.229: process: 0.48783 | load: 0.00245 [I 2022-10-26 14:16:11] ...trap/engines/engine.py.230: loss: 0.22232 [I 2022-10-26 14:16:11] ...optimizers/trijoint.py.151: total_norm: 3.070759 [I 2022-10-26 14:16:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 1630/2304 [I 2022-10-26 14:16:16] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:05:34 [I 2022-10-26 14:16:16] ...trap/engines/engine.py.229: process: 0.48361 | load: 0.00249 [I 2022-10-26 14:16:16] ...trap/engines/engine.py.230: loss: 0.18266 [I 2022-10-26 14:16:16] ...optimizers/trijoint.py.151: total_norm: 3.655731 [I 2022-10-26 14:16:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 1640/2304 [I 2022-10-26 14:16:21] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:05:32 [I 2022-10-26 14:16:21] ...trap/engines/engine.py.229: process: 0.50383 | load: 0.00242 [I 2022-10-26 14:16:21] ...trap/engines/engine.py.230: loss: 0.14727 [I 2022-10-26 14:16:21] ...optimizers/trijoint.py.151: total_norm: 2.643803 [I 2022-10-26 14:16:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 1650/2304 [I 2022-10-26 14:16:26] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:05:28 [I 2022-10-26 14:16:26] ...trap/engines/engine.py.229: process: 0.50518 | load: 0.00243 [I 2022-10-26 14:16:26] ...trap/engines/engine.py.230: loss: 0.17962 [I 2022-10-26 14:16:26] ...optimizers/trijoint.py.151: total_norm: 2.476342 [I 2022-10-26 14:16:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 1660/2304 [I 2022-10-26 14:16:31] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:05:23 [I 2022-10-26 14:16:31] ...trap/engines/engine.py.229: process: 0.50444 | load: 0.00241 [I 2022-10-26 14:16:31] ...trap/engines/engine.py.230: loss: 0.24934 [I 2022-10-26 14:16:31] ...optimizers/trijoint.py.151: total_norm: 3.890229 [I 2022-10-26 14:16:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 1670/2304 [I 2022-10-26 14:16:36] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:05:18 [I 2022-10-26 14:16:36] ...trap/engines/engine.py.229: process: 0.51619 | load: 0.00245 [I 2022-10-26 14:16:36] ...trap/engines/engine.py.230: loss: 0.16755 [I 2022-10-26 14:16:36] ...optimizers/trijoint.py.151: total_norm: 2.849406 [I 2022-10-26 14:16:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 1680/2304 [I 2022-10-26 14:16:41] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:05:11 [I 2022-10-26 14:16:41] ...trap/engines/engine.py.229: process: 0.49435 | load: 0.00240 [I 2022-10-26 14:16:41] ...trap/engines/engine.py.230: loss: 0.18773 [I 2022-10-26 14:16:41] ...optimizers/trijoint.py.151: total_norm: 3.018564 [I 2022-10-26 14:16:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 1690/2304 [I 2022-10-26 14:16:46] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:05:07 [I 2022-10-26 14:16:46] ...trap/engines/engine.py.229: process: 0.51004 | load: 0.00240 [I 2022-10-26 14:16:46] ...trap/engines/engine.py.230: loss: 0.21045 [I 2022-10-26 14:16:46] ...optimizers/trijoint.py.151: total_norm: 2.850670 [I 2022-10-26 14:16:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 1700/2304 [I 2022-10-26 14:16:51] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:05:01 [I 2022-10-26 14:16:51] ...trap/engines/engine.py.229: process: 0.49909 | load: 0.00247 [I 2022-10-26 14:16:51] ...trap/engines/engine.py.230: loss: 0.26614 [I 2022-10-26 14:16:51] ...optimizers/trijoint.py.151: total_norm: 3.965805 [I 2022-10-26 14:16:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 1710/2304 [I 2022-10-26 14:16:56] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:04:56 [I 2022-10-26 14:16:56] ...trap/engines/engine.py.229: process: 0.49789 | load: 0.00245 [I 2022-10-26 14:16:56] ...trap/engines/engine.py.230: loss: 0.16719 [I 2022-10-26 14:16:56] ...optimizers/trijoint.py.151: total_norm: 2.877093 [I 2022-10-26 14:17:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 1720/2304 [I 2022-10-26 14:17:01] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:04:49 [I 2022-10-26 14:17:01] ...trap/engines/engine.py.229: process: 0.49170 | load: 0.00241 [I 2022-10-26 14:17:01] ...trap/engines/engine.py.230: loss: 0.26216 [I 2022-10-26 14:17:01] ...optimizers/trijoint.py.151: total_norm: 3.961685 [I 2022-10-26 14:17:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1730/2304 [I 2022-10-26 14:17:06] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:04:45 [I 2022-10-26 14:17:06] ...trap/engines/engine.py.229: process: 0.50282 | load: 0.00241 [I 2022-10-26 14:17:06] ...trap/engines/engine.py.230: loss: 0.23904 [I 2022-10-26 14:17:06] ...optimizers/trijoint.py.151: total_norm: 3.420002 [I 2022-10-26 14:17:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 1740/2304 [I 2022-10-26 14:17:11] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:04:38 [I 2022-10-26 14:17:11] ...trap/engines/engine.py.229: process: 0.48462 | load: 0.00241 [I 2022-10-26 14:17:11] ...trap/engines/engine.py.230: loss: 0.23191 [I 2022-10-26 14:17:11] ...optimizers/trijoint.py.151: total_norm: 3.667070 [I 2022-10-26 14:17:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 1750/2304 [I 2022-10-26 14:17:16] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:04:45 [I 2022-10-26 14:17:16] ...trap/engines/engine.py.229: process: 0.58454 | load: 0.00242 [I 2022-10-26 14:17:16] ...trap/engines/engine.py.230: loss: 0.25658 [I 2022-10-26 14:17:16] ...optimizers/trijoint.py.151: total_norm: 3.376195 [I 2022-10-26 14:17:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 1760/2304 [I 2022-10-26 14:17:21] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:04:31 [I 2022-10-26 14:17:21] ...trap/engines/engine.py.229: process: 0.50356 | load: 0.00239 [I 2022-10-26 14:17:21] ...trap/engines/engine.py.230: loss: 0.17305 [I 2022-10-26 14:17:21] ...optimizers/trijoint.py.151: total_norm: 2.521343 [I 2022-10-26 14:17:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 1770/2304 [I 2022-10-26 14:17:26] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:04:25 [I 2022-10-26 14:17:26] ...trap/engines/engine.py.229: process: 0.50272 | load: 0.00243 [I 2022-10-26 14:17:26] ...trap/engines/engine.py.230: loss: 0.20692 [I 2022-10-26 14:17:26] ...optimizers/trijoint.py.151: total_norm: 3.300272 [I 2022-10-26 14:17:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 1780/2304 [I 2022-10-26 14:17:31] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:04:20 [I 2022-10-26 14:17:31] ...trap/engines/engine.py.229: process: 0.50018 | load: 0.00246 [I 2022-10-26 14:17:31] ...trap/engines/engine.py.230: loss: 0.25552 [I 2022-10-26 14:17:31] ...optimizers/trijoint.py.151: total_norm: 3.678483 [I 2022-10-26 14:17:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 1790/2304 [I 2022-10-26 14:17:36] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:04:13 [I 2022-10-26 14:17:36] ...trap/engines/engine.py.229: process: 0.50020 | load: 0.00241 [I 2022-10-26 14:17:36] ...trap/engines/engine.py.230: loss: 0.22010 [I 2022-10-26 14:17:36] ...optimizers/trijoint.py.151: total_norm: 2.979129 [I 2022-10-26 14:17:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 1800/2304 [I 2022-10-26 14:17:41] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:04:12 [I 2022-10-26 14:17:41] ...trap/engines/engine.py.229: process: 0.50768 | load: 0.00243 [I 2022-10-26 14:17:41] ...trap/engines/engine.py.230: loss: 0.21782 [I 2022-10-26 14:17:41] ...optimizers/trijoint.py.151: total_norm: 2.792933 [I 2022-10-26 14:17:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 1810/2304 [I 2022-10-26 14:17:46] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:04:04 [I 2022-10-26 14:17:46] ...trap/engines/engine.py.229: process: 0.49351 | load: 0.00241 [I 2022-10-26 14:17:46] ...trap/engines/engine.py.230: loss: 0.25303 [I 2022-10-26 14:17:46] ...optimizers/trijoint.py.151: total_norm: 3.286884 [I 2022-10-26 14:17:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 1820/2304 [I 2022-10-26 14:17:51] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:04:00 [I 2022-10-26 14:17:51] ...trap/engines/engine.py.229: process: 0.50134 | load: 0.00239 [I 2022-10-26 14:17:51] ...trap/engines/engine.py.230: loss: 0.18223 [I 2022-10-26 14:17:51] ...optimizers/trijoint.py.151: total_norm: 3.500226 [I 2022-10-26 14:17:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 1830/2304 [I 2022-10-26 14:17:56] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:03:57 [I 2022-10-26 14:17:56] ...trap/engines/engine.py.229: process: 0.48983 | load: 0.00240 [I 2022-10-26 14:17:56] ...trap/engines/engine.py.230: loss: 0.23256 [I 2022-10-26 14:17:56] ...optimizers/trijoint.py.151: total_norm: 3.040166 [I 2022-10-26 14:18:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 1840/2304 [I 2022-10-26 14:18:01] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:03:50 [I 2022-10-26 14:18:01] ...trap/engines/engine.py.229: process: 0.49145 | load: 0.00239 [I 2022-10-26 14:18:01] ...trap/engines/engine.py.230: loss: 0.25842 [I 2022-10-26 14:18:01] ...optimizers/trijoint.py.151: total_norm: 2.787241 [I 2022-10-26 14:18:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1850/2304 [I 2022-10-26 14:18:06] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:03:45 [I 2022-10-26 14:18:06] ...trap/engines/engine.py.229: process: 0.50152 | load: 0.00244 [I 2022-10-26 14:18:06] ...trap/engines/engine.py.230: loss: 0.22163 [I 2022-10-26 14:18:06] ...optimizers/trijoint.py.151: total_norm: 3.454983 [I 2022-10-26 14:18:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 1860/2304 [I 2022-10-26 14:18:11] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:03:41 [I 2022-10-26 14:18:11] ...trap/engines/engine.py.229: process: 0.50646 | load: 0.00248 [I 2022-10-26 14:18:11] ...trap/engines/engine.py.230: loss: 0.16714 [I 2022-10-26 14:18:11] ...optimizers/trijoint.py.151: total_norm: 2.843702 [I 2022-10-26 14:18:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 1870/2304 [I 2022-10-26 14:18:16] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:03:36 [I 2022-10-26 14:18:16] ...trap/engines/engine.py.229: process: 0.49261 | load: 0.00246 [I 2022-10-26 14:18:16] ...trap/engines/engine.py.230: loss: 0.25722 [I 2022-10-26 14:18:16] ...optimizers/trijoint.py.151: total_norm: 3.777006 [I 2022-10-26 14:18:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 1880/2304 [I 2022-10-26 14:18:21] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:03:30 [I 2022-10-26 14:18:21] ...trap/engines/engine.py.229: process: 0.50295 | load: 0.00244 [I 2022-10-26 14:18:21] ...trap/engines/engine.py.230: loss: 0.15857 [I 2022-10-26 14:18:21] ...optimizers/trijoint.py.151: total_norm: 3.129488 [I 2022-10-26 14:18:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 1890/2304 [I 2022-10-26 14:18:26] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:03:26 [I 2022-10-26 14:18:26] ...trap/engines/engine.py.229: process: 0.49156 | load: 0.00251 [I 2022-10-26 14:18:26] ...trap/engines/engine.py.230: loss: 0.28033 [I 2022-10-26 14:18:26] ...optimizers/trijoint.py.151: total_norm: 4.079374 [I 2022-10-26 14:18:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 1900/2304 [I 2022-10-26 14:18:31] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:03:28 [I 2022-10-26 14:18:31] ...trap/engines/engine.py.229: process: 0.48933 | load: 0.00253 [I 2022-10-26 14:18:31] ...trap/engines/engine.py.230: loss: 0.24053 [I 2022-10-26 14:18:31] ...optimizers/trijoint.py.151: total_norm: 3.521960 [I 2022-10-26 14:18:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 1910/2304 [I 2022-10-26 14:18:36] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:03:15 [I 2022-10-26 14:18:36] ...trap/engines/engine.py.229: process: 0.49291 | load: 0.00246 [I 2022-10-26 14:18:36] ...trap/engines/engine.py.230: loss: 0.27430 [I 2022-10-26 14:18:36] ...optimizers/trijoint.py.151: total_norm: 3.271588 [I 2022-10-26 14:18:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 1920/2304 [I 2022-10-26 14:18:41] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:03:11 [I 2022-10-26 14:18:41] ...trap/engines/engine.py.229: process: 0.50507 | load: 0.00260 [I 2022-10-26 14:18:41] ...trap/engines/engine.py.230: loss: 0.18109 [I 2022-10-26 14:18:41] ...optimizers/trijoint.py.151: total_norm: 2.867993 [I 2022-10-26 14:18:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 1930/2304 [I 2022-10-26 14:18:46] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:03:07 [I 2022-10-26 14:18:46] ...trap/engines/engine.py.229: process: 0.49179 | load: 0.00246 [I 2022-10-26 14:18:46] ...trap/engines/engine.py.230: loss: 0.22360 [I 2022-10-26 14:18:46] ...optimizers/trijoint.py.151: total_norm: 3.222888 [I 2022-10-26 14:18:51] ...trap/engines/engine.py.224: train: epoch 12 | batch 1940/2304 [I 2022-10-26 14:18:51] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:03:02 [I 2022-10-26 14:18:51] ...trap/engines/engine.py.229: process: 0.50052 | load: 0.00252 [I 2022-10-26 14:18:51] ...trap/engines/engine.py.230: loss: 0.23799 [I 2022-10-26 14:18:51] ...optimizers/trijoint.py.151: total_norm: 3.903836 [I 2022-10-26 14:18:56] ...trap/engines/engine.py.224: train: epoch 12 | batch 1950/2304 [I 2022-10-26 14:18:56] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:02:55 [I 2022-10-26 14:18:56] ...trap/engines/engine.py.229: process: 0.50218 | load: 0.00256 [I 2022-10-26 14:18:56] ...trap/engines/engine.py.230: loss: 0.18968 [I 2022-10-26 14:18:56] ...optimizers/trijoint.py.151: total_norm: 3.157515 [I 2022-10-26 14:19:01] ...trap/engines/engine.py.224: train: epoch 12 | batch 1960/2304 [I 2022-10-26 14:19:01] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:02:51 [I 2022-10-26 14:19:01] ...trap/engines/engine.py.229: process: 0.50397 | load: 0.00248 [I 2022-10-26 14:19:01] ...trap/engines/engine.py.230: loss: 0.25790 [I 2022-10-26 14:19:01] ...optimizers/trijoint.py.151: total_norm: 3.549143 [I 2022-10-26 14:19:06] ...trap/engines/engine.py.224: train: epoch 12 | batch 1970/2304 [I 2022-10-26 14:19:06] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:02:50 [I 2022-10-26 14:19:06] ...trap/engines/engine.py.229: process: 0.56962 | load: 0.00242 [I 2022-10-26 14:19:06] ...trap/engines/engine.py.230: loss: 0.24805 [I 2022-10-26 14:19:06] ...optimizers/trijoint.py.151: total_norm: 2.573295 [I 2022-10-26 14:19:11] ...trap/engines/engine.py.224: train: epoch 12 | batch 1980/2304 [I 2022-10-26 14:19:11] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:02:40 [I 2022-10-26 14:19:11] ...trap/engines/engine.py.229: process: 0.48977 | load: 0.00246 [I 2022-10-26 14:19:11] ...trap/engines/engine.py.230: loss: 0.23785 [I 2022-10-26 14:19:11] ...optimizers/trijoint.py.151: total_norm: 3.678688 [I 2022-10-26 14:19:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 1990/2304 [I 2022-10-26 14:19:16] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:02:36 [I 2022-10-26 14:19:16] ...trap/engines/engine.py.229: process: 0.50469 | load: 0.00244 [I 2022-10-26 14:19:16] ...trap/engines/engine.py.230: loss: 0.27214 [I 2022-10-26 14:19:16] ...optimizers/trijoint.py.151: total_norm: 3.053545 [I 2022-10-26 14:19:21] ...trap/engines/engine.py.224: train: epoch 12 | batch 2000/2304 [I 2022-10-26 14:19:21] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:02:31 [I 2022-10-26 14:19:21] ...trap/engines/engine.py.229: process: 0.49865 | load: 0.00249 [I 2022-10-26 14:19:21] ...trap/engines/engine.py.230: loss: 0.16677 [I 2022-10-26 14:19:21] ...optimizers/trijoint.py.151: total_norm: 3.119157 [I 2022-10-26 14:19:26] ...trap/engines/engine.py.224: train: epoch 12 | batch 2010/2304 [I 2022-10-26 14:19:26] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:02:26 [I 2022-10-26 14:19:26] ...trap/engines/engine.py.229: process: 0.49751 | load: 0.00255 [I 2022-10-26 14:19:26] ...trap/engines/engine.py.230: loss: 0.25632 [I 2022-10-26 14:19:26] ...optimizers/trijoint.py.151: total_norm: 2.848540 [I 2022-10-26 14:19:31] ...trap/engines/engine.py.224: train: epoch 12 | batch 2020/2304 [I 2022-10-26 14:19:31] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:02:22 [I 2022-10-26 14:19:31] ...trap/engines/engine.py.229: process: 0.50980 | load: 0.00247 [I 2022-10-26 14:19:31] ...trap/engines/engine.py.230: loss: 0.17294 [I 2022-10-26 14:19:31] ...optimizers/trijoint.py.151: total_norm: 2.787296 [I 2022-10-26 14:19:36] ...trap/engines/engine.py.224: train: epoch 12 | batch 2030/2304 [I 2022-10-26 14:19:36] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:02:16 [I 2022-10-26 14:19:36] ...trap/engines/engine.py.229: process: 0.50533 | load: 0.00249 [I 2022-10-26 14:19:36] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-26 14:19:36] ...optimizers/trijoint.py.151: total_norm: 3.004953 [I 2022-10-26 14:19:41] ...trap/engines/engine.py.224: train: epoch 12 | batch 2040/2304 [I 2022-10-26 14:19:41] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:02:13 [I 2022-10-26 14:19:41] ...trap/engines/engine.py.229: process: 0.53053 | load: 0.00252 [I 2022-10-26 14:19:41] ...trap/engines/engine.py.230: loss: 0.22475 [I 2022-10-26 14:19:41] ...optimizers/trijoint.py.151: total_norm: 2.906098 [I 2022-10-26 14:19:46] ...trap/engines/engine.py.224: train: epoch 12 | batch 2050/2304 [I 2022-10-26 14:19:46] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:02:06 [I 2022-10-26 14:19:46] ...trap/engines/engine.py.229: process: 0.49375 | load: 0.00254 [I 2022-10-26 14:19:46] ...trap/engines/engine.py.230: loss: 0.22166 [I 2022-10-26 14:19:46] ...optimizers/trijoint.py.151: total_norm: 3.581462 [I 2022-10-26 14:20:14] ...trap/engines/engine.py.224: train: epoch 12 | batch 2060/2304 [I 2022-10-26 14:20:14] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:13:50 [I 2022-10-26 14:20:14] ...trap/engines/engine.py.229: process: 0.49829 | load: 0.00238 [I 2022-10-26 14:20:14] ...trap/engines/engine.py.230: loss: 0.26132 [I 2022-10-26 14:20:14] ...optimizers/trijoint.py.151: total_norm: 3.074308 [I 2022-10-26 14:20:19] ...trap/engines/engine.py.224: train: epoch 12 | batch 2070/2304 [I 2022-10-26 14:20:19] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:03:11 [I 2022-10-26 14:20:19] ...trap/engines/engine.py.229: process: 0.48862 | load: 0.00237 [I 2022-10-26 14:20:19] ...trap/engines/engine.py.230: loss: 0.22707 [I 2022-10-26 14:20:19] ...optimizers/trijoint.py.151: total_norm: 3.243089 [I 2022-10-26 14:20:24] ...trap/engines/engine.py.224: train: epoch 12 | batch 2080/2304 [I 2022-10-26 14:20:24] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:01:58 [I 2022-10-26 14:20:24] ...trap/engines/engine.py.229: process: 0.49791 | load: 0.00245 [I 2022-10-26 14:20:24] ...trap/engines/engine.py.230: loss: 0.22387 [I 2022-10-26 14:20:24] ...optimizers/trijoint.py.151: total_norm: 3.548587 [I 2022-10-26 14:20:29] ...trap/engines/engine.py.224: train: epoch 12 | batch 2090/2304 [I 2022-10-26 14:20:29] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:01:47 [I 2022-10-26 14:20:29] ...trap/engines/engine.py.229: process: 0.49784 | load: 0.00237 [I 2022-10-26 14:20:29] ...trap/engines/engine.py.230: loss: 0.21756 [I 2022-10-26 14:20:29] ...optimizers/trijoint.py.151: total_norm: 4.546237 [I 2022-10-26 14:20:34] ...trap/engines/engine.py.224: train: epoch 12 | batch 2100/2304 [I 2022-10-26 14:20:34] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:01:41 [I 2022-10-26 14:20:34] ...trap/engines/engine.py.229: process: 0.48631 | load: 0.00237 [I 2022-10-26 14:20:34] ...trap/engines/engine.py.230: loss: 0.24915 [I 2022-10-26 14:20:34] ...optimizers/trijoint.py.151: total_norm: 3.224546 [I 2022-10-26 14:20:39] ...trap/engines/engine.py.224: train: epoch 12 | batch 2110/2304 [I 2022-10-26 14:20:39] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:01:36 [I 2022-10-26 14:20:39] ...trap/engines/engine.py.229: process: 0.50138 | load: 0.00238 [I 2022-10-26 14:20:39] ...trap/engines/engine.py.230: loss: 0.24215 [I 2022-10-26 14:20:39] ...optimizers/trijoint.py.151: total_norm: 3.855739 [I 2022-10-26 14:20:44] ...trap/engines/engine.py.224: train: epoch 12 | batch 2120/2304 [I 2022-10-26 14:20:44] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:01:31 [I 2022-10-26 14:20:44] ...trap/engines/engine.py.229: process: 0.50192 | load: 0.00247 [I 2022-10-26 14:20:44] ...trap/engines/engine.py.230: loss: 0.25313 [I 2022-10-26 14:20:44] ...optimizers/trijoint.py.151: total_norm: 3.255443 [I 2022-10-26 14:20:49] ...trap/engines/engine.py.224: train: epoch 12 | batch 2130/2304 [I 2022-10-26 14:20:49] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:01:26 [I 2022-10-26 14:20:49] ...trap/engines/engine.py.229: process: 0.50869 | load: 0.00251 [I 2022-10-26 14:20:49] ...trap/engines/engine.py.230: loss: 0.22838 [I 2022-10-26 14:20:49] ...optimizers/trijoint.py.151: total_norm: 3.069225 [I 2022-10-26 14:20:54] ...trap/engines/engine.py.224: train: epoch 12 | batch 2140/2304 [I 2022-10-26 14:20:54] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:01:21 [I 2022-10-26 14:20:54] ...trap/engines/engine.py.229: process: 0.50082 | load: 0.00251 [I 2022-10-26 14:20:54] ...trap/engines/engine.py.230: loss: 0.21585 [I 2022-10-26 14:20:54] ...optimizers/trijoint.py.151: total_norm: 3.299748 [I 2022-10-26 14:20:59] ...trap/engines/engine.py.224: train: epoch 12 | batch 2150/2304 [I 2022-10-26 14:20:59] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:01:16 [I 2022-10-26 14:20:59] ...trap/engines/engine.py.229: process: 0.49819 | load: 0.00248 [I 2022-10-26 14:20:59] ...trap/engines/engine.py.230: loss: 0.22022 [I 2022-10-26 14:20:59] ...optimizers/trijoint.py.151: total_norm: 3.533898 [I 2022-10-26 14:21:04] ...trap/engines/engine.py.224: train: epoch 12 | batch 2160/2304 [I 2022-10-26 14:21:04] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:01:11 [I 2022-10-26 14:21:04] ...trap/engines/engine.py.229: process: 0.50452 | load: 0.00245 [I 2022-10-26 14:21:04] ...trap/engines/engine.py.230: loss: 0.12217 [I 2022-10-26 14:21:04] ...optimizers/trijoint.py.151: total_norm: 1.942903 [I 2022-10-26 14:21:09] ...trap/engines/engine.py.224: train: epoch 12 | batch 2170/2304 [I 2022-10-26 14:21:09] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:01:06 [I 2022-10-26 14:21:09] ...trap/engines/engine.py.229: process: 0.49885 | load: 0.00245 [I 2022-10-26 14:21:09] ...trap/engines/engine.py.230: loss: 0.20520 [I 2022-10-26 14:21:09] ...optimizers/trijoint.py.151: total_norm: 3.493700 [I 2022-10-26 14:21:14] ...trap/engines/engine.py.224: train: epoch 12 | batch 2180/2304 [I 2022-10-26 14:21:14] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:01:01 [I 2022-10-26 14:21:14] ...trap/engines/engine.py.229: process: 0.50160 | load: 0.00245 [I 2022-10-26 14:21:14] ...trap/engines/engine.py.230: loss: 0.21603 [I 2022-10-26 14:21:14] ...optimizers/trijoint.py.151: total_norm: 2.950625 [I 2022-10-26 14:21:19] ...trap/engines/engine.py.224: train: epoch 12 | batch 2190/2304 [I 2022-10-26 14:21:19] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:00:57 [I 2022-10-26 14:21:19] ...trap/engines/engine.py.229: process: 0.50240 | load: 0.00244 [I 2022-10-26 14:21:19] ...trap/engines/engine.py.230: loss: 0.22476 [I 2022-10-26 14:21:19] ...optimizers/trijoint.py.151: total_norm: 3.463414 [I 2022-10-26 14:21:24] ...trap/engines/engine.py.224: train: epoch 12 | batch 2200/2304 [I 2022-10-26 14:21:24] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:00:52 [I 2022-10-26 14:21:24] ...trap/engines/engine.py.229: process: 0.50929 | load: 0.00243 [I 2022-10-26 14:21:24] ...trap/engines/engine.py.230: loss: 0.18476 [I 2022-10-26 14:21:24] ...optimizers/trijoint.py.151: total_norm: 3.232641 [I 2022-10-26 14:21:29] ...trap/engines/engine.py.224: train: epoch 12 | batch 2210/2304 [I 2022-10-26 14:21:29] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:00:46 [I 2022-10-26 14:21:29] ...trap/engines/engine.py.229: process: 0.50070 | load: 0.00243 [I 2022-10-26 14:21:29] ...trap/engines/engine.py.230: loss: 0.23194 [I 2022-10-26 14:21:29] ...optimizers/trijoint.py.151: total_norm: 2.781894 [I 2022-10-26 14:21:34] ...trap/engines/engine.py.224: train: epoch 12 | batch 2220/2304 [I 2022-10-26 14:21:34] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:00:41 [I 2022-10-26 14:21:34] ...trap/engines/engine.py.229: process: 0.50028 | load: 0.00246 [I 2022-10-26 14:21:34] ...trap/engines/engine.py.230: loss: 0.18255 [I 2022-10-26 14:21:34] ...optimizers/trijoint.py.151: total_norm: 2.592940 [I 2022-10-26 14:21:39] ...trap/engines/engine.py.224: train: epoch 12 | batch 2230/2304 [I 2022-10-26 14:21:39] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:00:37 [I 2022-10-26 14:21:39] ...trap/engines/engine.py.229: process: 0.50387 | load: 0.00245 [I 2022-10-26 14:21:39] ...trap/engines/engine.py.230: loss: 0.15757 [I 2022-10-26 14:21:39] ...optimizers/trijoint.py.151: total_norm: 2.895532 [I 2022-10-26 14:21:44] ...trap/engines/engine.py.224: train: epoch 12 | batch 2240/2304 [I 2022-10-26 14:21:44] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:00:31 [I 2022-10-26 14:21:44] ...trap/engines/engine.py.229: process: 0.48927 | load: 0.00244 [I 2022-10-26 14:21:44] ...trap/engines/engine.py.230: loss: 0.14384 [I 2022-10-26 14:21:44] ...optimizers/trijoint.py.151: total_norm: 2.470875 [I 2022-10-26 14:21:49] ...trap/engines/engine.py.224: train: epoch 12 | batch 2250/2304 [I 2022-10-26 14:21:49] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:00:26 [I 2022-10-26 14:21:49] ...trap/engines/engine.py.229: process: 0.49538 | load: 0.00242 [I 2022-10-26 14:21:49] ...trap/engines/engine.py.230: loss: 0.28160 [I 2022-10-26 14:21:49] ...optimizers/trijoint.py.151: total_norm: 5.372486 [I 2022-10-26 14:21:54] ...trap/engines/engine.py.224: train: epoch 12 | batch 2260/2304 [I 2022-10-26 14:21:54] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:00:21 [I 2022-10-26 14:21:54] ...trap/engines/engine.py.229: process: 0.49365 | load: 0.00243 [I 2022-10-26 14:21:54] ...trap/engines/engine.py.230: loss: 0.24250 [I 2022-10-26 14:21:54] ...optimizers/trijoint.py.151: total_norm: 3.874682 [I 2022-10-26 14:21:59] ...trap/engines/engine.py.224: train: epoch 12 | batch 2270/2304 [I 2022-10-26 14:21:59] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:00:16 [I 2022-10-26 14:21:59] ...trap/engines/engine.py.229: process: 0.49257 | load: 0.00245 [I 2022-10-26 14:21:59] ...trap/engines/engine.py.230: loss: 0.16330 [I 2022-10-26 14:21:59] ...optimizers/trijoint.py.151: total_norm: 2.334581 [I 2022-10-26 14:22:04] ...trap/engines/engine.py.224: train: epoch 12 | batch 2280/2304 [I 2022-10-26 14:22:04] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:00:11 [I 2022-10-26 14:22:04] ...trap/engines/engine.py.229: process: 0.50084 | load: 0.00242 [I 2022-10-26 14:22:04] ...trap/engines/engine.py.230: loss: 0.33107 [I 2022-10-26 14:22:04] ...optimizers/trijoint.py.151: total_norm: 3.166707 [I 2022-10-26 14:22:09] ...trap/engines/engine.py.224: train: epoch 12 | batch 2290/2304 [I 2022-10-26 14:22:09] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:00:06 [I 2022-10-26 14:22:09] ...trap/engines/engine.py.229: process: 0.50530 | load: 0.00247 [I 2022-10-26 14:22:09] ...trap/engines/engine.py.230: loss: 0.14766 [I 2022-10-26 14:22:09] ...optimizers/trijoint.py.151: total_norm: 2.684405 [I 2022-10-26 14:22:14] ...trap/engines/engine.py.224: train: epoch 12 | batch 2300/2304 [I 2022-10-26 14:22:14] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:00:02 [I 2022-10-26 14:22:14] ...trap/engines/engine.py.229: process: 0.49895 | load: 0.00016 [I 2022-10-26 14:22:14] ...trap/engines/engine.py.230: loss: 0.28704 [I 2022-10-26 14:22:14] ...optimizers/trijoint.py.151: total_norm: 3.857296 [I 2022-10-26 14:22:16] ...trap/engines/engine.py.224: train: epoch 12 | batch 2304/2304 [I 2022-10-26 14:22:16] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:00:00 [I 2022-10-26 14:22:16] ...trap/engines/engine.py.229: process: 0.49100 | load: 0.00015 [I 2022-10-26 14:22:16] ...trap/engines/engine.py.230: loss: 0.21948 [I 2022-10-26 14:22:16] ...optimizers/trijoint.py.151: total_norm: 3.157267 [S 2022-10-26 14:22:17] ...trap/engines/engine.py.129: train_epoch.epoch: 12 [S 2022-10-26 14:22:17] ...trap/engines/engine.py.129: train_epoch.loss: 0.22712900378375664 [I 2022-10-26 14:22:17] ...s/criterions/triplet.py.84: margin = 0.11500000000000005 [I 2022-10-26 14:22:17] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 12 [I 2022-10-26 14:22:17] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 14:22:18] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 14:22:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 14:22:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 14:22:19] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 14:22:20] ...trap/engines/engine.py.321: eval: epoch 12 | batch 0/510 [I 2022-10-26 14:22:20] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:13 [I 2022-10-26 14:22:20] ...trap/engines/engine.py.326: process: 2.73190 | load: 2.35675 [I 2022-10-26 14:22:23] ...trap/engines/engine.py.321: eval: epoch 12 | batch 10/510 [I 2022-10-26 14:22:23] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:55 [I 2022-10-26 14:22:23] ...trap/engines/engine.py.326: process: 0.32998 | load: 0.00018 [I 2022-10-26 14:22:26] ...trap/engines/engine.py.321: eval: epoch 12 | batch 20/510 [I 2022-10-26 14:22:26] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:03:02 [I 2022-10-26 14:22:26] ...trap/engines/engine.py.326: process: 0.37936 | load: 0.00020 [I 2022-10-26 14:22:30] ...trap/engines/engine.py.321: eval: epoch 12 | batch 30/510 [I 2022-10-26 14:22:30] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:41 [I 2022-10-26 14:22:30] ...trap/engines/engine.py.326: process: 0.32119 | load: 0.00021 [I 2022-10-26 14:22:33] ...trap/engines/engine.py.321: eval: epoch 12 | batch 40/510 [I 2022-10-26 14:22:33] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:42 [I 2022-10-26 14:22:33] ...trap/engines/engine.py.326: process: 0.36416 | load: 0.00022 [I 2022-10-26 14:22:37] ...trap/engines/engine.py.321: eval: epoch 12 | batch 50/510 [I 2022-10-26 14:22:37] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:33 [I 2022-10-26 14:22:37] ...trap/engines/engine.py.326: process: 0.32935 | load: 0.00022 [I 2022-10-26 14:22:40] ...trap/engines/engine.py.321: eval: epoch 12 | batch 60/510 [I 2022-10-26 14:22:40] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:30 [I 2022-10-26 14:22:40] ...trap/engines/engine.py.326: process: 0.32990 | load: 0.00022 [I 2022-10-26 14:22:43] ...trap/engines/engine.py.321: eval: epoch 12 | batch 70/510 [I 2022-10-26 14:22:43] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:26 [I 2022-10-26 14:22:43] ...trap/engines/engine.py.326: process: 0.32361 | load: 0.00016 [I 2022-10-26 14:22:47] ...trap/engines/engine.py.321: eval: epoch 12 | batch 80/510 [I 2022-10-26 14:22:47] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:23 [I 2022-10-26 14:22:47] ...trap/engines/engine.py.326: process: 0.33218 | load: 0.00018 [I 2022-10-26 14:22:50] ...trap/engines/engine.py.321: eval: epoch 12 | batch 90/510 [I 2022-10-26 14:22:50] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-26 14:22:50] ...trap/engines/engine.py.326: process: 0.32694 | load: 0.00022 [I 2022-10-26 14:22:53] ...trap/engines/engine.py.321: eval: epoch 12 | batch 100/510 [I 2022-10-26 14:22:53] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 14:22:53] ...trap/engines/engine.py.326: process: 0.29046 | load: 0.00023 [I 2022-10-26 14:22:56] ...trap/engines/engine.py.321: eval: epoch 12 | batch 110/510 [I 2022-10-26 14:22:56] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:59 [I 2022-10-26 14:22:56] ...trap/engines/engine.py.326: process: 0.31027 | load: 0.00015 [I 2022-10-26 14:22:59] ...trap/engines/engine.py.321: eval: epoch 12 | batch 120/510 [I 2022-10-26 14:22:59] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 14:22:59] ...trap/engines/engine.py.326: process: 0.29138 | load: 0.00022 [I 2022-10-26 14:23:02] ...trap/engines/engine.py.321: eval: epoch 12 | batch 130/510 [I 2022-10-26 14:23:02] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-26 14:23:02] ...trap/engines/engine.py.326: process: 0.29307 | load: 0.00023 [I 2022-10-26 14:23:05] ...trap/engines/engine.py.321: eval: epoch 12 | batch 140/510 [I 2022-10-26 14:23:05] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:47 [I 2022-10-26 14:23:05] ...trap/engines/engine.py.326: process: 0.28655 | load: 0.00015 [I 2022-10-26 14:23:08] ...trap/engines/engine.py.321: eval: epoch 12 | batch 150/510 [I 2022-10-26 14:23:08] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:43 [I 2022-10-26 14:23:08] ...trap/engines/engine.py.326: process: 0.29239 | load: 0.00016 [I 2022-10-26 14:23:11] ...trap/engines/engine.py.321: eval: epoch 12 | batch 160/510 [I 2022-10-26 14:23:11] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 14:23:11] ...trap/engines/engine.py.326: process: 0.28799 | load: 0.00014 [I 2022-10-26 14:23:14] ...trap/engines/engine.py.321: eval: epoch 12 | batch 170/510 [I 2022-10-26 14:23:14] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:37 [I 2022-10-26 14:23:14] ...trap/engines/engine.py.326: process: 0.28255 | load: 0.00014 [I 2022-10-26 14:23:16] ...trap/engines/engine.py.321: eval: epoch 12 | batch 180/510 [I 2022-10-26 14:23:16] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:34 [I 2022-10-26 14:23:16] ...trap/engines/engine.py.326: process: 0.28112 | load: 0.00021 [I 2022-10-26 14:23:19] ...trap/engines/engine.py.321: eval: epoch 12 | batch 190/510 [I 2022-10-26 14:23:19] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:33 [I 2022-10-26 14:23:19] ...trap/engines/engine.py.326: process: 0.29280 | load: 0.00017 [I 2022-10-26 14:23:22] ...trap/engines/engine.py.321: eval: epoch 12 | batch 200/510 [I 2022-10-26 14:23:22] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 14:23:22] ...trap/engines/engine.py.326: process: 0.28769 | load: 0.00023 [I 2022-10-26 14:23:25] ...trap/engines/engine.py.321: eval: epoch 12 | batch 210/510 [I 2022-10-26 14:23:25] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-26 14:23:25] ...trap/engines/engine.py.326: process: 0.29397 | load: 0.00016 [I 2022-10-26 14:23:28] ...trap/engines/engine.py.321: eval: epoch 12 | batch 220/510 [I 2022-10-26 14:23:28] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-26 14:23:28] ...trap/engines/engine.py.326: process: 0.30489 | load: 0.00016 [I 2022-10-26 14:23:31] ...trap/engines/engine.py.321: eval: epoch 12 | batch 230/510 [I 2022-10-26 14:23:31] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-26 14:23:31] ...trap/engines/engine.py.326: process: 0.28982 | load: 0.00019 [I 2022-10-26 14:23:34] ...trap/engines/engine.py.321: eval: epoch 12 | batch 240/510 [I 2022-10-26 14:23:34] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:19 [I 2022-10-26 14:23:34] ...trap/engines/engine.py.326: process: 0.30440 | load: 0.00018 [I 2022-10-26 14:23:37] ...trap/engines/engine.py.321: eval: epoch 12 | batch 250/510 [I 2022-10-26 14:23:37] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:15 [I 2022-10-26 14:23:37] ...trap/engines/engine.py.326: process: 0.28974 | load: 0.00017 [I 2022-10-26 14:23:40] ...trap/engines/engine.py.321: eval: epoch 12 | batch 260/510 [I 2022-10-26 14:23:40] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:11 [I 2022-10-26 14:23:40] ...trap/engines/engine.py.326: process: 0.28364 | load: 0.00017 [I 2022-10-26 14:23:43] ...trap/engines/engine.py.321: eval: epoch 12 | batch 270/510 [I 2022-10-26 14:23:43] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 14:23:43] ...trap/engines/engine.py.326: process: 0.29627 | load: 0.00017 [I 2022-10-26 14:23:45] ...trap/engines/engine.py.321: eval: epoch 12 | batch 280/510 [I 2022-10-26 14:23:45] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 14:23:45] ...trap/engines/engine.py.326: process: 0.28920 | load: 0.00021 [I 2022-10-26 14:23:48] ...trap/engines/engine.py.321: eval: epoch 12 | batch 290/510 [I 2022-10-26 14:23:48] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 14:23:48] ...trap/engines/engine.py.326: process: 0.29201 | load: 0.00021 [I 2022-10-26 14:23:51] ...trap/engines/engine.py.321: eval: epoch 12 | batch 300/510 [I 2022-10-26 14:23:51] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 14:23:51] ...trap/engines/engine.py.326: process: 0.29096 | load: 0.00017 [I 2022-10-26 14:23:54] ...trap/engines/engine.py.321: eval: epoch 12 | batch 310/510 [I 2022-10-26 14:23:54] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 14:23:54] ...trap/engines/engine.py.326: process: 0.29655 | load: 0.00017 [I 2022-10-26 14:23:57] ...trap/engines/engine.py.321: eval: epoch 12 | batch 320/510 [I 2022-10-26 14:23:57] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 14:23:57] ...trap/engines/engine.py.326: process: 0.29046 | load: 0.00018 [I 2022-10-26 14:24:00] ...trap/engines/engine.py.321: eval: epoch 12 | batch 330/510 [I 2022-10-26 14:24:00] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:52 [I 2022-10-26 14:24:00] ...trap/engines/engine.py.326: process: 0.29399 | load: 0.00017 [I 2022-10-26 14:24:03] ...trap/engines/engine.py.321: eval: epoch 12 | batch 340/510 [I 2022-10-26 14:24:03] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:48 [I 2022-10-26 14:24:03] ...trap/engines/engine.py.326: process: 0.28481 | load: 0.00021 [I 2022-10-26 14:24:06] ...trap/engines/engine.py.321: eval: epoch 12 | batch 350/510 [I 2022-10-26 14:24:06] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 14:24:06] ...trap/engines/engine.py.326: process: 0.28954 | load: 0.00020 [I 2022-10-26 14:24:09] ...trap/engines/engine.py.321: eval: epoch 12 | batch 360/510 [I 2022-10-26 14:24:09] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 14:24:09] ...trap/engines/engine.py.326: process: 0.28387 | load: 0.00019 [I 2022-10-26 14:24:11] ...trap/engines/engine.py.321: eval: epoch 12 | batch 370/510 [I 2022-10-26 14:24:11] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 14:24:11] ...trap/engines/engine.py.326: process: 0.28893 | load: 0.00021 [I 2022-10-26 14:24:14] ...trap/engines/engine.py.321: eval: epoch 12 | batch 380/510 [I 2022-10-26 14:24:14] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 14:24:14] ...trap/engines/engine.py.326: process: 0.28253 | load: 0.00024 [I 2022-10-26 14:24:17] ...trap/engines/engine.py.321: eval: epoch 12 | batch 390/510 [I 2022-10-26 14:24:17] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 14:24:17] ...trap/engines/engine.py.326: process: 0.28999 | load: 0.00019 [I 2022-10-26 14:24:20] ...trap/engines/engine.py.321: eval: epoch 12 | batch 400/510 [I 2022-10-26 14:24:20] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 14:24:20] ...trap/engines/engine.py.326: process: 0.29093 | load: 0.00022 [I 2022-10-26 14:24:23] ...trap/engines/engine.py.321: eval: epoch 12 | batch 410/510 [I 2022-10-26 14:24:23] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:28 [I 2022-10-26 14:24:23] ...trap/engines/engine.py.326: process: 0.28822 | load: 0.00026 [I 2022-10-26 14:24:26] ...trap/engines/engine.py.321: eval: epoch 12 | batch 420/510 [I 2022-10-26 14:24:26] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:26 [I 2022-10-26 14:24:26] ...trap/engines/engine.py.326: process: 0.28829 | load: 0.00016 [I 2022-10-26 14:24:29] ...trap/engines/engine.py.321: eval: epoch 12 | batch 430/510 [I 2022-10-26 14:24:29] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-26 14:24:29] ...trap/engines/engine.py.326: process: 0.29110 | load: 0.00016 [I 2022-10-26 14:24:32] ...trap/engines/engine.py.321: eval: epoch 12 | batch 440/510 [I 2022-10-26 14:24:32] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 14:24:32] ...trap/engines/engine.py.326: process: 0.28690 | load: 0.00016 [I 2022-10-26 14:24:35] ...trap/engines/engine.py.321: eval: epoch 12 | batch 450/510 [I 2022-10-26 14:24:35] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 14:24:35] ...trap/engines/engine.py.326: process: 0.28878 | load: 0.00017 [I 2022-10-26 14:24:37] ...trap/engines/engine.py.321: eval: epoch 12 | batch 460/510 [I 2022-10-26 14:24:37] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 14:24:37] ...trap/engines/engine.py.326: process: 0.30639 | load: 0.00017 [I 2022-10-26 14:24:40] ...trap/engines/engine.py.321: eval: epoch 12 | batch 470/510 [I 2022-10-26 14:24:40] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 14:24:40] ...trap/engines/engine.py.326: process: 0.29092 | load: 0.00018 [I 2022-10-26 14:24:43] ...trap/engines/engine.py.321: eval: epoch 12 | batch 480/510 [I 2022-10-26 14:24:43] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 14:24:43] ...trap/engines/engine.py.326: process: 0.28443 | load: 0.00016 [I 2022-10-26 14:24:46] ...trap/engines/engine.py.321: eval: epoch 12 | batch 490/510 [I 2022-10-26 14:24:46] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 14:24:46] ...trap/engines/engine.py.326: process: 0.28681 | load: 0.00020 [I 2022-10-26 14:24:49] ...trap/engines/engine.py.321: eval: epoch 12 | batch 500/510 [I 2022-10-26 14:24:49] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 14:24:49] ...trap/engines/engine.py.326: process: 0.27833 | load: 0.00015 [I 2022-10-26 14:24:52] ...trap/engines/engine.py.321: eval: epoch 12 | batch 510/510 [I 2022-10-26 14:24:52] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 14:24:52] ...trap/engines/engine.py.326: process: 0.28557 | load: 0.00014 [S 2022-10-26 14:24:53] ...trap/engines/engine.py.132: eval_epoch.epoch: 12 [I 2022-10-26 14:24:53] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.10 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.532400 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.808500 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.879200 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.533700 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.808000 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.880100 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.300000 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013771 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012102 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009174 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.018671 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012625 [S 2022-10-26 14:24:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008893 [I 2022-10-26 14:24:57] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 14:24:57] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 14:24:57] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 14:24:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 14:24:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 14:24:58] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 14:25:01] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 14:25:06] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 14:25:06] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 14:25:06] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 14:25:09] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 14:25:14] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 14:25:14] ...trap/engines/engine.py.163: Training model on trainset for epoch 13 [I 2022-10-26 14:25:14] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 14:25:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 0/2304 [I 2022-10-26 14:25:17] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:45:21 [I 2022-10-26 14:25:17] ...trap/engines/engine.py.229: process: 2.74353 | load: 2.16053 [I 2022-10-26 14:25:17] ...trap/engines/engine.py.230: loss: 0.20874 [I 2022-10-26 14:25:17] ...optimizers/trijoint.py.151: total_norm: 3.737329 [I 2022-10-26 14:25:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 10/2304 [I 2022-10-26 14:25:22] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:29:03 [I 2022-10-26 14:25:22] ...trap/engines/engine.py.229: process: 0.49760 | load: 0.00412 [I 2022-10-26 14:25:22] ...trap/engines/engine.py.230: loss: 0.17976 [I 2022-10-26 14:25:22] ...optimizers/trijoint.py.151: total_norm: 2.801155 [I 2022-10-26 14:25:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 20/2304 [I 2022-10-26 14:25:27] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:06 [I 2022-10-26 14:25:27] ...trap/engines/engine.py.229: process: 0.49769 | load: 0.00451 [I 2022-10-26 14:25:27] ...trap/engines/engine.py.230: loss: 0.27099 [I 2022-10-26 14:25:27] ...optimizers/trijoint.py.151: total_norm: 3.554162 [I 2022-10-26 14:25:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 30/2304 [I 2022-10-26 14:25:32] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:06 [I 2022-10-26 14:25:32] ...trap/engines/engine.py.229: process: 0.49744 | load: 0.00320 [I 2022-10-26 14:25:32] ...trap/engines/engine.py.230: loss: 0.18346 [I 2022-10-26 14:25:32] ...optimizers/trijoint.py.151: total_norm: 2.766266 [I 2022-10-26 14:25:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 40/2304 [I 2022-10-26 14:25:37] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:46 [I 2022-10-26 14:25:37] ...trap/engines/engine.py.229: process: 0.49856 | load: 0.00270 [I 2022-10-26 14:25:37] ...trap/engines/engine.py.230: loss: 0.13208 [I 2022-10-26 14:25:37] ...optimizers/trijoint.py.151: total_norm: 2.346193 [I 2022-10-26 14:25:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 50/2304 [I 2022-10-26 14:25:42] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:30 [I 2022-10-26 14:25:42] ...trap/engines/engine.py.229: process: 0.48400 | load: 0.00306 [I 2022-10-26 14:25:42] ...trap/engines/engine.py.230: loss: 0.14811 [I 2022-10-26 14:25:42] ...optimizers/trijoint.py.151: total_norm: 2.402266 [I 2022-10-26 14:25:47] ...trap/engines/engine.py.224: train: epoch 13 | batch 60/2304 [I 2022-10-26 14:25:47] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:43 [I 2022-10-26 14:25:47] ...trap/engines/engine.py.229: process: 0.49716 | load: 0.00271 [I 2022-10-26 14:25:47] ...trap/engines/engine.py.230: loss: 0.24960 [I 2022-10-26 14:25:47] ...optimizers/trijoint.py.151: total_norm: 2.907633 [I 2022-10-26 14:25:52] ...trap/engines/engine.py.224: train: epoch 13 | batch 70/2304 [I 2022-10-26 14:25:52] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:49 [I 2022-10-26 14:25:52] ...trap/engines/engine.py.229: process: 0.52111 | load: 0.00283 [I 2022-10-26 14:25:52] ...trap/engines/engine.py.230: loss: 0.19281 [I 2022-10-26 14:25:52] ...optimizers/trijoint.py.151: total_norm: 2.418900 [I 2022-10-26 14:25:57] ...trap/engines/engine.py.224: train: epoch 13 | batch 80/2304 [I 2022-10-26 14:25:57] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:19:07 [I 2022-10-26 14:25:57] ...trap/engines/engine.py.229: process: 0.48729 | load: 0.00277 [I 2022-10-26 14:25:57] ...trap/engines/engine.py.230: loss: 0.21658 [I 2022-10-26 14:25:57] ...optimizers/trijoint.py.151: total_norm: 2.474605 [I 2022-10-26 14:26:02] ...trap/engines/engine.py.224: train: epoch 13 | batch 90/2304 [I 2022-10-26 14:26:02] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:18:21 [I 2022-10-26 14:26:02] ...trap/engines/engine.py.229: process: 0.49199 | load: 0.00268 [I 2022-10-26 14:26:02] ...trap/engines/engine.py.230: loss: 0.24824 [I 2022-10-26 14:26:02] ...optimizers/trijoint.py.151: total_norm: 3.850262 [I 2022-10-26 14:26:07] ...trap/engines/engine.py.224: train: epoch 13 | batch 100/2304 [I 2022-10-26 14:26:07] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:18:15 [I 2022-10-26 14:26:07] ...trap/engines/engine.py.229: process: 0.49573 | load: 0.00253 [I 2022-10-26 14:26:07] ...trap/engines/engine.py.230: loss: 0.23269 [I 2022-10-26 14:26:07] ...optimizers/trijoint.py.151: total_norm: 4.176598 [I 2022-10-26 14:26:12] ...trap/engines/engine.py.224: train: epoch 13 | batch 110/2304 [I 2022-10-26 14:26:12] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:18:13 [I 2022-10-26 14:26:12] ...trap/engines/engine.py.229: process: 0.50558 | load: 0.00268 [I 2022-10-26 14:26:12] ...trap/engines/engine.py.230: loss: 0.16728 [I 2022-10-26 14:26:12] ...optimizers/trijoint.py.151: total_norm: 2.713744 [I 2022-10-26 14:26:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 120/2304 [I 2022-10-26 14:26:17] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:25 [I 2022-10-26 14:26:17] ...trap/engines/engine.py.229: process: 0.51561 | load: 0.00277 [I 2022-10-26 14:26:17] ...trap/engines/engine.py.230: loss: 0.23673 [I 2022-10-26 14:26:17] ...optimizers/trijoint.py.151: total_norm: 2.849239 [I 2022-10-26 14:26:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 130/2304 [I 2022-10-26 14:26:22] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:03 [I 2022-10-26 14:26:22] ...trap/engines/engine.py.229: process: 0.49095 | load: 0.00259 [I 2022-10-26 14:26:22] ...trap/engines/engine.py.230: loss: 0.21079 [I 2022-10-26 14:26:22] ...optimizers/trijoint.py.151: total_norm: 2.984063 [I 2022-10-26 14:26:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 140/2304 [I 2022-10-26 14:26:27] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:17:53 [I 2022-10-26 14:26:27] ...trap/engines/engine.py.229: process: 0.50414 | load: 0.00269 [I 2022-10-26 14:26:27] ...trap/engines/engine.py.230: loss: 0.18589 [I 2022-10-26 14:26:27] ...optimizers/trijoint.py.151: total_norm: 3.143440 [I 2022-10-26 14:26:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 150/2304 [I 2022-10-26 14:26:32] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:17:51 [I 2022-10-26 14:26:32] ...trap/engines/engine.py.229: process: 0.49886 | load: 0.00246 [I 2022-10-26 14:26:32] ...trap/engines/engine.py.230: loss: 0.26820 [I 2022-10-26 14:26:32] ...optimizers/trijoint.py.151: total_norm: 2.757908 [I 2022-10-26 14:26:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 160/2304 [I 2022-10-26 14:26:37] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:18:03 [I 2022-10-26 14:26:37] ...trap/engines/engine.py.229: process: 0.50348 | load: 0.00253 [I 2022-10-26 14:26:37] ...trap/engines/engine.py.230: loss: 0.20891 [I 2022-10-26 14:26:37] ...optimizers/trijoint.py.151: total_norm: 2.871411 [I 2022-10-26 14:26:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 170/2304 [I 2022-10-26 14:26:42] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:42 [I 2022-10-26 14:26:42] ...trap/engines/engine.py.229: process: 0.48923 | load: 0.00260 [I 2022-10-26 14:26:42] ...trap/engines/engine.py.230: loss: 0.18939 [I 2022-10-26 14:26:42] ...optimizers/trijoint.py.151: total_norm: 2.828966 [I 2022-10-26 14:26:47] ...trap/engines/engine.py.224: train: epoch 13 | batch 180/2304 [I 2022-10-26 14:26:47] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:17:50 [I 2022-10-26 14:26:47] ...trap/engines/engine.py.229: process: 0.50437 | load: 0.00268 [I 2022-10-26 14:26:47] ...trap/engines/engine.py.230: loss: 0.19108 [I 2022-10-26 14:26:47] ...optimizers/trijoint.py.151: total_norm: 2.314638 [I 2022-10-26 14:26:52] ...trap/engines/engine.py.224: train: epoch 13 | batch 190/2304 [I 2022-10-26 14:26:52] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:17:44 [I 2022-10-26 14:26:52] ...trap/engines/engine.py.229: process: 0.49518 | load: 0.00251 [I 2022-10-26 14:26:52] ...trap/engines/engine.py.230: loss: 0.28241 [I 2022-10-26 14:26:52] ...optimizers/trijoint.py.151: total_norm: 4.798911 [I 2022-10-26 14:26:57] ...trap/engines/engine.py.224: train: epoch 13 | batch 200/2304 [I 2022-10-26 14:26:57] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:29 [I 2022-10-26 14:26:57] ...trap/engines/engine.py.229: process: 0.49892 | load: 0.00257 [I 2022-10-26 14:26:57] ...trap/engines/engine.py.230: loss: 0.13822 [I 2022-10-26 14:26:57] ...optimizers/trijoint.py.151: total_norm: 2.483645 [I 2022-10-26 14:27:02] ...trap/engines/engine.py.224: train: epoch 13 | batch 210/2304 [I 2022-10-26 14:27:02] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:25 [I 2022-10-26 14:27:02] ...trap/engines/engine.py.229: process: 0.49693 | load: 0.00253 [I 2022-10-26 14:27:02] ...trap/engines/engine.py.230: loss: 0.24932 [I 2022-10-26 14:27:02] ...optimizers/trijoint.py.151: total_norm: 3.281430 [I 2022-10-26 14:27:07] ...trap/engines/engine.py.224: train: epoch 13 | batch 220/2304 [I 2022-10-26 14:27:07] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:22 [I 2022-10-26 14:27:07] ...trap/engines/engine.py.229: process: 0.50266 | load: 0.00283 [I 2022-10-26 14:27:07] ...trap/engines/engine.py.230: loss: 0.21663 [I 2022-10-26 14:27:07] ...optimizers/trijoint.py.151: total_norm: 3.027947 [I 2022-10-26 14:27:12] ...trap/engines/engine.py.224: train: epoch 13 | batch 230/2304 [I 2022-10-26 14:27:12] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:05 [I 2022-10-26 14:27:12] ...trap/engines/engine.py.229: process: 0.49967 | load: 0.00272 [I 2022-10-26 14:27:12] ...trap/engines/engine.py.230: loss: 0.13040 [I 2022-10-26 14:27:12] ...optimizers/trijoint.py.151: total_norm: 2.530509 [I 2022-10-26 14:27:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 240/2304 [I 2022-10-26 14:27:17] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:12 [I 2022-10-26 14:27:17] ...trap/engines/engine.py.229: process: 0.50729 | load: 0.00245 [I 2022-10-26 14:27:17] ...trap/engines/engine.py.230: loss: 0.23657 [I 2022-10-26 14:27:17] ...optimizers/trijoint.py.151: total_norm: 4.314175 [I 2022-10-26 14:27:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 250/2304 [I 2022-10-26 14:27:22] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:07 [I 2022-10-26 14:27:22] ...trap/engines/engine.py.229: process: 0.49901 | load: 0.00262 [I 2022-10-26 14:27:22] ...trap/engines/engine.py.230: loss: 0.20714 [I 2022-10-26 14:27:22] ...optimizers/trijoint.py.151: total_norm: 4.334547 [I 2022-10-26 14:27:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 260/2304 [I 2022-10-26 14:27:27] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:06 [I 2022-10-26 14:27:27] ...trap/engines/engine.py.229: process: 0.50262 | load: 0.00253 [I 2022-10-26 14:27:27] ...trap/engines/engine.py.230: loss: 0.17850 [I 2022-10-26 14:27:27] ...optimizers/trijoint.py.151: total_norm: 2.538000 [I 2022-10-26 14:27:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 270/2304 [I 2022-10-26 14:27:32] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:16:54 [I 2022-10-26 14:27:32] ...trap/engines/engine.py.229: process: 0.48821 | load: 0.00261 [I 2022-10-26 14:27:32] ...trap/engines/engine.py.230: loss: 0.22882 [I 2022-10-26 14:27:32] ...optimizers/trijoint.py.151: total_norm: 3.108889 [I 2022-10-26 14:27:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 280/2304 [I 2022-10-26 14:27:37] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:16:46 [I 2022-10-26 14:27:37] ...trap/engines/engine.py.229: process: 0.50232 | load: 0.00251 [I 2022-10-26 14:27:37] ...trap/engines/engine.py.230: loss: 0.24244 [I 2022-10-26 14:27:37] ...optimizers/trijoint.py.151: total_norm: 3.004300 [I 2022-10-26 14:27:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 290/2304 [I 2022-10-26 14:27:43] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:17:29 [I 2022-10-26 14:27:43] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00297 [I 2022-10-26 14:27:43] ...trap/engines/engine.py.230: loss: 0.24063 [I 2022-10-26 14:27:43] ...optimizers/trijoint.py.151: total_norm: 3.809677 [I 2022-10-26 14:27:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 300/2304 [I 2022-10-26 14:27:48] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:16:55 [I 2022-10-26 14:27:48] ...trap/engines/engine.py.229: process: 0.51981 | load: 0.00264 [I 2022-10-26 14:27:48] ...trap/engines/engine.py.230: loss: 0.25272 [I 2022-10-26 14:27:48] ...optimizers/trijoint.py.151: total_norm: 4.219978 [I 2022-10-26 14:27:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 310/2304 [I 2022-10-26 14:27:53] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:16:36 [I 2022-10-26 14:27:53] ...trap/engines/engine.py.229: process: 0.49351 | load: 0.00246 [I 2022-10-26 14:27:53] ...trap/engines/engine.py.230: loss: 0.21438 [I 2022-10-26 14:27:53] ...optimizers/trijoint.py.151: total_norm: 2.682571 [I 2022-10-26 14:27:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 320/2304 [I 2022-10-26 14:27:58] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:16:28 [I 2022-10-26 14:27:58] ...trap/engines/engine.py.229: process: 0.49100 | load: 0.00245 [I 2022-10-26 14:27:58] ...trap/engines/engine.py.230: loss: 0.21341 [I 2022-10-26 14:27:58] ...optimizers/trijoint.py.151: total_norm: 2.593029 [I 2022-10-26 14:28:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 330/2304 [I 2022-10-26 14:28:03] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:16:26 [I 2022-10-26 14:28:03] ...trap/engines/engine.py.229: process: 0.50207 | load: 0.00250 [I 2022-10-26 14:28:03] ...trap/engines/engine.py.230: loss: 0.19561 [I 2022-10-26 14:28:03] ...optimizers/trijoint.py.151: total_norm: 4.533044 [I 2022-10-26 14:28:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 340/2304 [I 2022-10-26 14:28:08] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:16:13 [I 2022-10-26 14:28:08] ...trap/engines/engine.py.229: process: 0.49647 | load: 0.00244 [I 2022-10-26 14:28:08] ...trap/engines/engine.py.230: loss: 0.21754 [I 2022-10-26 14:28:08] ...optimizers/trijoint.py.151: total_norm: 4.143823 [I 2022-10-26 14:28:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 350/2304 [I 2022-10-26 14:28:13] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:16:08 [I 2022-10-26 14:28:13] ...trap/engines/engine.py.229: process: 0.49275 | load: 0.00273 [I 2022-10-26 14:28:13] ...trap/engines/engine.py.230: loss: 0.19211 [I 2022-10-26 14:28:13] ...optimizers/trijoint.py.151: total_norm: 3.164023 [I 2022-10-26 14:28:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 360/2304 [I 2022-10-26 14:28:18] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:16:12 [I 2022-10-26 14:28:18] ...trap/engines/engine.py.229: process: 0.51102 | load: 0.00248 [I 2022-10-26 14:28:18] ...trap/engines/engine.py.230: loss: 0.26975 [I 2022-10-26 14:28:18] ...optimizers/trijoint.py.151: total_norm: 3.311960 [I 2022-10-26 14:28:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 370/2304 [I 2022-10-26 14:28:23] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:16:03 [I 2022-10-26 14:28:23] ...trap/engines/engine.py.229: process: 0.50648 | load: 0.00244 [I 2022-10-26 14:28:23] ...trap/engines/engine.py.230: loss: 0.23127 [I 2022-10-26 14:28:23] ...optimizers/trijoint.py.151: total_norm: 2.727879 [I 2022-10-26 14:28:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 380/2304 [I 2022-10-26 14:28:28] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:15:51 [I 2022-10-26 14:28:28] ...trap/engines/engine.py.229: process: 0.50008 | load: 0.00247 [I 2022-10-26 14:28:28] ...trap/engines/engine.py.230: loss: 0.21718 [I 2022-10-26 14:28:28] ...optimizers/trijoint.py.151: total_norm: 3.062717 [I 2022-10-26 14:28:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 390/2304 [I 2022-10-26 14:28:33] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:15:48 [I 2022-10-26 14:28:33] ...trap/engines/engine.py.229: process: 0.49317 | load: 0.00246 [I 2022-10-26 14:28:33] ...trap/engines/engine.py.230: loss: 0.23135 [I 2022-10-26 14:28:33] ...optimizers/trijoint.py.151: total_norm: 3.436402 [I 2022-10-26 14:28:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 400/2304 [I 2022-10-26 14:28:37] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:15:45 [I 2022-10-26 14:28:37] ...trap/engines/engine.py.229: process: 0.49291 | load: 0.00242 [I 2022-10-26 14:28:37] ...trap/engines/engine.py.230: loss: 0.20578 [I 2022-10-26 14:28:37] ...optimizers/trijoint.py.151: total_norm: 3.645364 [I 2022-10-26 14:28:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 410/2304 [I 2022-10-26 14:28:42] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:15:38 [I 2022-10-26 14:28:42] ...trap/engines/engine.py.229: process: 0.48652 | load: 0.00244 [I 2022-10-26 14:28:42] ...trap/engines/engine.py.230: loss: 0.20149 [I 2022-10-26 14:28:42] ...optimizers/trijoint.py.151: total_norm: 3.274679 [I 2022-10-26 14:28:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 420/2304 [I 2022-10-26 14:28:48] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:15:45 [I 2022-10-26 14:28:48] ...trap/engines/engine.py.229: process: 0.49345 | load: 0.00253 [I 2022-10-26 14:28:48] ...trap/engines/engine.py.230: loss: 0.20959 [I 2022-10-26 14:28:48] ...optimizers/trijoint.py.151: total_norm: 2.885413 [I 2022-10-26 14:28:52] ...trap/engines/engine.py.224: train: epoch 13 | batch 430/2304 [I 2022-10-26 14:28:52] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:15:29 [I 2022-10-26 14:28:52] ...trap/engines/engine.py.229: process: 0.49548 | load: 0.00248 [I 2022-10-26 14:28:52] ...trap/engines/engine.py.230: loss: 0.19074 [I 2022-10-26 14:28:52] ...optimizers/trijoint.py.151: total_norm: 3.163535 [I 2022-10-26 14:28:57] ...trap/engines/engine.py.224: train: epoch 13 | batch 440/2304 [I 2022-10-26 14:28:57] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:15:29 [I 2022-10-26 14:28:57] ...trap/engines/engine.py.229: process: 0.50309 | load: 0.00252 [I 2022-10-26 14:28:57] ...trap/engines/engine.py.230: loss: 0.21789 [I 2022-10-26 14:28:57] ...optimizers/trijoint.py.151: total_norm: 2.898208 [I 2022-10-26 14:29:02] ...trap/engines/engine.py.224: train: epoch 13 | batch 450/2304 [I 2022-10-26 14:29:02] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:15:17 [I 2022-10-26 14:29:02] ...trap/engines/engine.py.229: process: 0.48681 | load: 0.00246 [I 2022-10-26 14:29:02] ...trap/engines/engine.py.230: loss: 0.32064 [I 2022-10-26 14:29:02] ...optimizers/trijoint.py.151: total_norm: 4.786944 [I 2022-10-26 14:29:07] ...trap/engines/engine.py.224: train: epoch 13 | batch 460/2304 [I 2022-10-26 14:29:07] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:15:11 [I 2022-10-26 14:29:07] ...trap/engines/engine.py.229: process: 0.49271 | load: 0.00249 [I 2022-10-26 14:29:07] ...trap/engines/engine.py.230: loss: 0.22792 [I 2022-10-26 14:29:07] ...optimizers/trijoint.py.151: total_norm: 4.007804 [I 2022-10-26 14:29:12] ...trap/engines/engine.py.224: train: epoch 13 | batch 470/2304 [I 2022-10-26 14:29:12] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:15:17 [I 2022-10-26 14:29:12] ...trap/engines/engine.py.229: process: 0.51223 | load: 0.00252 [I 2022-10-26 14:29:12] ...trap/engines/engine.py.230: loss: 0.23740 [I 2022-10-26 14:29:12] ...optimizers/trijoint.py.151: total_norm: 3.210744 [I 2022-10-26 14:29:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 480/2304 [I 2022-10-26 14:29:17] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:15:12 [I 2022-10-26 14:29:17] ...trap/engines/engine.py.229: process: 0.49233 | load: 0.00263 [I 2022-10-26 14:29:17] ...trap/engines/engine.py.230: loss: 0.36148 [I 2022-10-26 14:29:17] ...optimizers/trijoint.py.151: total_norm: 3.818535 [I 2022-10-26 14:29:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 490/2304 [I 2022-10-26 14:29:22] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:15:11 [I 2022-10-26 14:29:22] ...trap/engines/engine.py.229: process: 0.50133 | load: 0.00253 [I 2022-10-26 14:29:22] ...trap/engines/engine.py.230: loss: 0.28278 [I 2022-10-26 14:29:22] ...optimizers/trijoint.py.151: total_norm: 2.717709 [I 2022-10-26 14:29:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 500/2304 [I 2022-10-26 14:29:27] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:15:03 [I 2022-10-26 14:29:27] ...trap/engines/engine.py.229: process: 0.50259 | load: 0.00252 [I 2022-10-26 14:29:27] ...trap/engines/engine.py.230: loss: 0.22017 [I 2022-10-26 14:29:27] ...optimizers/trijoint.py.151: total_norm: 3.694739 [I 2022-10-26 14:29:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 510/2304 [I 2022-10-26 14:29:32] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:14:53 [I 2022-10-26 14:29:32] ...trap/engines/engine.py.229: process: 0.49108 | load: 0.00258 [I 2022-10-26 14:29:32] ...trap/engines/engine.py.230: loss: 0.18973 [I 2022-10-26 14:29:32] ...optimizers/trijoint.py.151: total_norm: 3.705855 [I 2022-10-26 14:29:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 520/2304 [I 2022-10-26 14:29:37] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:14:54 [I 2022-10-26 14:29:37] ...trap/engines/engine.py.229: process: 0.50560 | load: 0.00249 [I 2022-10-26 14:29:37] ...trap/engines/engine.py.230: loss: 0.26914 [I 2022-10-26 14:29:37] ...optimizers/trijoint.py.151: total_norm: 3.474683 [I 2022-10-26 14:29:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 530/2304 [I 2022-10-26 14:29:42] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:14:39 [I 2022-10-26 14:29:42] ...trap/engines/engine.py.229: process: 0.50008 | load: 0.00246 [I 2022-10-26 14:29:42] ...trap/engines/engine.py.230: loss: 0.24663 [I 2022-10-26 14:29:42] ...optimizers/trijoint.py.151: total_norm: 3.728460 [I 2022-10-26 14:29:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 540/2304 [I 2022-10-26 14:29:48] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:15:03 [I 2022-10-26 14:29:48] ...trap/engines/engine.py.229: process: 0.58071 | load: 0.00257 [I 2022-10-26 14:29:48] ...trap/engines/engine.py.230: loss: 0.16346 [I 2022-10-26 14:29:48] ...optimizers/trijoint.py.151: total_norm: 2.526625 [I 2022-10-26 14:29:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 550/2304 [I 2022-10-26 14:29:53] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:14:33 [I 2022-10-26 14:29:53] ...trap/engines/engine.py.229: process: 0.48565 | load: 0.00245 [I 2022-10-26 14:29:53] ...trap/engines/engine.py.230: loss: 0.17946 [I 2022-10-26 14:29:53] ...optimizers/trijoint.py.151: total_norm: 2.939273 [I 2022-10-26 14:29:57] ...trap/engines/engine.py.224: train: epoch 13 | batch 560/2304 [I 2022-10-26 14:29:57] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:14:26 [I 2022-10-26 14:29:57] ...trap/engines/engine.py.229: process: 0.49209 | load: 0.00275 [I 2022-10-26 14:29:57] ...trap/engines/engine.py.230: loss: 0.22123 [I 2022-10-26 14:29:57] ...optimizers/trijoint.py.151: total_norm: 2.875102 [I 2022-10-26 14:30:02] ...trap/engines/engine.py.224: train: epoch 13 | batch 570/2304 [I 2022-10-26 14:30:02] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:14:18 [I 2022-10-26 14:30:02] ...trap/engines/engine.py.229: process: 0.49850 | load: 0.00256 [I 2022-10-26 14:30:02] ...trap/engines/engine.py.230: loss: 0.19671 [I 2022-10-26 14:30:02] ...optimizers/trijoint.py.151: total_norm: 3.333092 [I 2022-10-26 14:30:07] ...trap/engines/engine.py.224: train: epoch 13 | batch 580/2304 [I 2022-10-26 14:30:07] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:14:14 [I 2022-10-26 14:30:07] ...trap/engines/engine.py.229: process: 0.49896 | load: 0.00242 [I 2022-10-26 14:30:07] ...trap/engines/engine.py.230: loss: 0.25582 [I 2022-10-26 14:30:07] ...optimizers/trijoint.py.151: total_norm: 3.298248 [I 2022-10-26 14:30:12] ...trap/engines/engine.py.224: train: epoch 13 | batch 590/2304 [I 2022-10-26 14:30:12] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:14:24 [I 2022-10-26 14:30:12] ...trap/engines/engine.py.229: process: 0.51221 | load: 0.00247 [I 2022-10-26 14:30:12] ...trap/engines/engine.py.230: loss: 0.29483 [I 2022-10-26 14:30:12] ...optimizers/trijoint.py.151: total_norm: 3.878763 [I 2022-10-26 14:30:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 600/2304 [I 2022-10-26 14:30:18] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:14:23 [I 2022-10-26 14:30:18] ...trap/engines/engine.py.229: process: 0.49145 | load: 0.00253 [I 2022-10-26 14:30:18] ...trap/engines/engine.py.230: loss: 0.27980 [I 2022-10-26 14:30:18] ...optimizers/trijoint.py.151: total_norm: 3.987973 [I 2022-10-26 14:30:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 610/2304 [I 2022-10-26 14:30:23] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:14:15 [I 2022-10-26 14:30:23] ...trap/engines/engine.py.229: process: 0.50090 | load: 0.00248 [I 2022-10-26 14:30:23] ...trap/engines/engine.py.230: loss: 0.15907 [I 2022-10-26 14:30:23] ...optimizers/trijoint.py.151: total_norm: 2.196357 [I 2022-10-26 14:30:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 620/2304 [I 2022-10-26 14:30:28] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:13:59 [I 2022-10-26 14:30:28] ...trap/engines/engine.py.229: process: 0.49863 | load: 0.00247 [I 2022-10-26 14:30:28] ...trap/engines/engine.py.230: loss: 0.22409 [I 2022-10-26 14:30:28] ...optimizers/trijoint.py.151: total_norm: 4.110928 [I 2022-10-26 14:30:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 630/2304 [I 2022-10-26 14:30:33] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:14:01 [I 2022-10-26 14:30:33] ...trap/engines/engine.py.229: process: 0.51236 | load: 0.00245 [I 2022-10-26 14:30:33] ...trap/engines/engine.py.230: loss: 0.20756 [I 2022-10-26 14:30:33] ...optimizers/trijoint.py.151: total_norm: 4.320995 [I 2022-10-26 14:30:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 640/2304 [I 2022-10-26 14:30:38] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:13:46 [I 2022-10-26 14:30:38] ...trap/engines/engine.py.229: process: 0.48394 | load: 0.00242 [I 2022-10-26 14:30:38] ...trap/engines/engine.py.230: loss: 0.17176 [I 2022-10-26 14:30:38] ...optimizers/trijoint.py.151: total_norm: 2.530033 [I 2022-10-26 14:30:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 650/2304 [I 2022-10-26 14:30:43] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:13:55 [I 2022-10-26 14:30:43] ...trap/engines/engine.py.229: process: 0.50495 | load: 0.00248 [I 2022-10-26 14:30:43] ...trap/engines/engine.py.230: loss: 0.17348 [I 2022-10-26 14:30:43] ...optimizers/trijoint.py.151: total_norm: 2.345860 [I 2022-10-26 14:30:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 660/2304 [I 2022-10-26 14:30:48] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:13:44 [I 2022-10-26 14:30:48] ...trap/engines/engine.py.229: process: 0.49146 | load: 0.00255 [I 2022-10-26 14:30:48] ...trap/engines/engine.py.230: loss: 0.12270 [I 2022-10-26 14:30:48] ...optimizers/trijoint.py.151: total_norm: 2.335545 [I 2022-10-26 14:30:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 670/2304 [I 2022-10-26 14:30:53] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:13:41 [I 2022-10-26 14:30:53] ...trap/engines/engine.py.229: process: 0.50408 | load: 0.00247 [I 2022-10-26 14:30:53] ...trap/engines/engine.py.230: loss: 0.22038 [I 2022-10-26 14:30:53] ...optimizers/trijoint.py.151: total_norm: 4.508119 [I 2022-10-26 14:30:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 680/2304 [I 2022-10-26 14:30:58] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:13:40 [I 2022-10-26 14:30:58] ...trap/engines/engine.py.229: process: 0.50299 | load: 0.00248 [I 2022-10-26 14:30:58] ...trap/engines/engine.py.230: loss: 0.19936 [I 2022-10-26 14:30:58] ...optimizers/trijoint.py.151: total_norm: 3.049730 [I 2022-10-26 14:31:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 690/2304 [I 2022-10-26 14:31:03] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:13:34 [I 2022-10-26 14:31:03] ...trap/engines/engine.py.229: process: 0.50965 | load: 0.00266 [I 2022-10-26 14:31:03] ...trap/engines/engine.py.230: loss: 0.21010 [I 2022-10-26 14:31:03] ...optimizers/trijoint.py.151: total_norm: 3.722424 [I 2022-10-26 14:31:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 700/2304 [I 2022-10-26 14:31:08] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:13:26 [I 2022-10-26 14:31:08] ...trap/engines/engine.py.229: process: 0.49597 | load: 0.00252 [I 2022-10-26 14:31:08] ...trap/engines/engine.py.230: loss: 0.18111 [I 2022-10-26 14:31:08] ...optimizers/trijoint.py.151: total_norm: 3.530689 [I 2022-10-26 14:31:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 710/2304 [I 2022-10-26 14:31:13] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:13:14 [I 2022-10-26 14:31:13] ...trap/engines/engine.py.229: process: 0.50162 | load: 0.00248 [I 2022-10-26 14:31:13] ...trap/engines/engine.py.230: loss: 0.24164 [I 2022-10-26 14:31:13] ...optimizers/trijoint.py.151: total_norm: 4.354442 [I 2022-10-26 14:31:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 720/2304 [I 2022-10-26 14:31:18] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:13:04 [I 2022-10-26 14:31:18] ...trap/engines/engine.py.229: process: 0.50190 | load: 0.00245 [I 2022-10-26 14:31:18] ...trap/engines/engine.py.230: loss: 0.27872 [I 2022-10-26 14:31:18] ...optimizers/trijoint.py.151: total_norm: 4.422352 [I 2022-10-26 14:31:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 730/2304 [I 2022-10-26 14:31:23] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:13:03 [I 2022-10-26 14:31:23] ...trap/engines/engine.py.229: process: 0.49073 | load: 0.00243 [I 2022-10-26 14:31:23] ...trap/engines/engine.py.230: loss: 0.24363 [I 2022-10-26 14:31:23] ...optimizers/trijoint.py.151: total_norm: 3.529240 [I 2022-10-26 14:31:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 740/2304 [I 2022-10-26 14:31:28] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:12:55 [I 2022-10-26 14:31:28] ...trap/engines/engine.py.229: process: 0.48943 | load: 0.00276 [I 2022-10-26 14:31:28] ...trap/engines/engine.py.230: loss: 0.25530 [I 2022-10-26 14:31:28] ...optimizers/trijoint.py.151: total_norm: 3.375609 [I 2022-10-26 14:31:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 750/2304 [I 2022-10-26 14:31:33] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:12:52 [I 2022-10-26 14:31:33] ...trap/engines/engine.py.229: process: 0.49255 | load: 0.00251 [I 2022-10-26 14:31:33] ...trap/engines/engine.py.230: loss: 0.15519 [I 2022-10-26 14:31:33] ...optimizers/trijoint.py.151: total_norm: 2.540609 [I 2022-10-26 14:31:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 760/2304 [I 2022-10-26 14:31:38] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:12:49 [I 2022-10-26 14:31:38] ...trap/engines/engine.py.229: process: 0.50873 | load: 0.00257 [I 2022-10-26 14:31:38] ...trap/engines/engine.py.230: loss: 0.21581 [I 2022-10-26 14:31:38] ...optimizers/trijoint.py.151: total_norm: 3.191943 [I 2022-10-26 14:31:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 770/2304 [I 2022-10-26 14:31:43] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:12:54 [I 2022-10-26 14:31:43] ...trap/engines/engine.py.229: process: 0.50234 | load: 0.00253 [I 2022-10-26 14:31:43] ...trap/engines/engine.py.230: loss: 0.22493 [I 2022-10-26 14:31:43] ...optimizers/trijoint.py.151: total_norm: 2.619672 [I 2022-10-26 14:31:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 780/2304 [I 2022-10-26 14:31:48] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:12:31 [I 2022-10-26 14:31:48] ...trap/engines/engine.py.229: process: 0.48612 | load: 0.00255 [I 2022-10-26 14:31:48] ...trap/engines/engine.py.230: loss: 0.16996 [I 2022-10-26 14:31:48] ...optimizers/trijoint.py.151: total_norm: 3.730663 [I 2022-10-26 14:31:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 790/2304 [I 2022-10-26 14:31:53] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:12:38 [I 2022-10-26 14:31:53] ...trap/engines/engine.py.229: process: 0.50606 | load: 0.00245 [I 2022-10-26 14:31:53] ...trap/engines/engine.py.230: loss: 0.23157 [I 2022-10-26 14:31:53] ...optimizers/trijoint.py.151: total_norm: 3.492910 [I 2022-10-26 14:31:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 800/2304 [I 2022-10-26 14:31:58] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:12:32 [I 2022-10-26 14:31:58] ...trap/engines/engine.py.229: process: 0.50471 | load: 0.00243 [I 2022-10-26 14:31:58] ...trap/engines/engine.py.230: loss: 0.20233 [I 2022-10-26 14:31:58] ...optimizers/trijoint.py.151: total_norm: 3.815861 [I 2022-10-26 14:32:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 810/2304 [I 2022-10-26 14:32:03] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:12:24 [I 2022-10-26 14:32:03] ...trap/engines/engine.py.229: process: 0.49810 | load: 0.00244 [I 2022-10-26 14:32:03] ...trap/engines/engine.py.230: loss: 0.27830 [I 2022-10-26 14:32:03] ...optimizers/trijoint.py.151: total_norm: 3.964908 [I 2022-10-26 14:32:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 820/2304 [I 2022-10-26 14:32:08] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:12:14 [I 2022-10-26 14:32:08] ...trap/engines/engine.py.229: process: 0.48715 | load: 0.00255 [I 2022-10-26 14:32:08] ...trap/engines/engine.py.230: loss: 0.20150 [I 2022-10-26 14:32:08] ...optimizers/trijoint.py.151: total_norm: 3.585799 [I 2022-10-26 14:32:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 830/2304 [I 2022-10-26 14:32:13] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:12:10 [I 2022-10-26 14:32:13] ...trap/engines/engine.py.229: process: 0.49789 | load: 0.00245 [I 2022-10-26 14:32:13] ...trap/engines/engine.py.230: loss: 0.22389 [I 2022-10-26 14:32:13] ...optimizers/trijoint.py.151: total_norm: 3.598079 [I 2022-10-26 14:32:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 840/2304 [I 2022-10-26 14:32:18] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:12:04 [I 2022-10-26 14:32:18] ...trap/engines/engine.py.229: process: 0.49800 | load: 0.00244 [I 2022-10-26 14:32:18] ...trap/engines/engine.py.230: loss: 0.22866 [I 2022-10-26 14:32:18] ...optimizers/trijoint.py.151: total_norm: 3.558352 [I 2022-10-26 14:32:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 850/2304 [I 2022-10-26 14:32:23] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:11:57 [I 2022-10-26 14:32:23] ...trap/engines/engine.py.229: process: 0.49000 | load: 0.00244 [I 2022-10-26 14:32:23] ...trap/engines/engine.py.230: loss: 0.14120 [I 2022-10-26 14:32:23] ...optimizers/trijoint.py.151: total_norm: 2.390606 [I 2022-10-26 14:32:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 860/2304 [I 2022-10-26 14:32:28] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:11:53 [I 2022-10-26 14:32:28] ...trap/engines/engine.py.229: process: 0.49848 | load: 0.00266 [I 2022-10-26 14:32:28] ...trap/engines/engine.py.230: loss: 0.33869 [I 2022-10-26 14:32:28] ...optimizers/trijoint.py.151: total_norm: 4.927726 [I 2022-10-26 14:32:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 870/2304 [I 2022-10-26 14:32:33] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:11:56 [I 2022-10-26 14:32:33] ...trap/engines/engine.py.229: process: 0.49816 | load: 0.00245 [I 2022-10-26 14:32:33] ...trap/engines/engine.py.230: loss: 0.21209 [I 2022-10-26 14:32:33] ...optimizers/trijoint.py.151: total_norm: 3.790087 [I 2022-10-26 14:32:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 880/2304 [I 2022-10-26 14:32:38] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:11:47 [I 2022-10-26 14:32:38] ...trap/engines/engine.py.229: process: 0.49287 | load: 0.00246 [I 2022-10-26 14:32:38] ...trap/engines/engine.py.230: loss: 0.23493 [I 2022-10-26 14:32:38] ...optimizers/trijoint.py.151: total_norm: 2.731504 [I 2022-10-26 14:32:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 890/2304 [I 2022-10-26 14:32:43] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:11:40 [I 2022-10-26 14:32:43] ...trap/engines/engine.py.229: process: 0.49119 | load: 0.00244 [I 2022-10-26 14:32:43] ...trap/engines/engine.py.230: loss: 0.23155 [I 2022-10-26 14:32:43] ...optimizers/trijoint.py.151: total_norm: 3.291111 [I 2022-10-26 14:32:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 900/2304 [I 2022-10-26 14:32:48] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:11:32 [I 2022-10-26 14:32:48] ...trap/engines/engine.py.229: process: 0.49482 | load: 0.00244 [I 2022-10-26 14:32:48] ...trap/engines/engine.py.230: loss: 0.17850 [I 2022-10-26 14:32:48] ...optimizers/trijoint.py.151: total_norm: 4.177065 [I 2022-10-26 14:32:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 910/2304 [I 2022-10-26 14:32:53] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:11:26 [I 2022-10-26 14:32:53] ...trap/engines/engine.py.229: process: 0.48463 | load: 0.00256 [I 2022-10-26 14:32:53] ...trap/engines/engine.py.230: loss: 0.12167 [I 2022-10-26 14:32:53] ...optimizers/trijoint.py.151: total_norm: 2.216626 [I 2022-10-26 14:32:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 920/2304 [I 2022-10-26 14:32:58] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:11:26 [I 2022-10-26 14:32:58] ...trap/engines/engine.py.229: process: 0.48676 | load: 0.00247 [I 2022-10-26 14:32:58] ...trap/engines/engine.py.230: loss: 0.22578 [I 2022-10-26 14:32:58] ...optimizers/trijoint.py.151: total_norm: 3.767407 [I 2022-10-26 14:33:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 930/2304 [I 2022-10-26 14:33:03] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:11:21 [I 2022-10-26 14:33:03] ...trap/engines/engine.py.229: process: 0.50098 | load: 0.00247 [I 2022-10-26 14:33:03] ...trap/engines/engine.py.230: loss: 0.29617 [I 2022-10-26 14:33:03] ...optimizers/trijoint.py.151: total_norm: 5.787923 [I 2022-10-26 14:33:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 940/2304 [I 2022-10-26 14:33:08] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:11:20 [I 2022-10-26 14:33:08] ...trap/engines/engine.py.229: process: 0.49831 | load: 0.00246 [I 2022-10-26 14:33:08] ...trap/engines/engine.py.230: loss: 0.29004 [I 2022-10-26 14:33:08] ...optimizers/trijoint.py.151: total_norm: 4.055441 [I 2022-10-26 14:33:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 950/2304 [I 2022-10-26 14:33:13] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:11:10 [I 2022-10-26 14:33:13] ...trap/engines/engine.py.229: process: 0.48729 | load: 0.00246 [I 2022-10-26 14:33:13] ...trap/engines/engine.py.230: loss: 0.18903 [I 2022-10-26 14:33:13] ...optimizers/trijoint.py.151: total_norm: 4.774911 [I 2022-10-26 14:33:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 960/2304 [I 2022-10-26 14:33:18] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:11:07 [I 2022-10-26 14:33:18] ...trap/engines/engine.py.229: process: 0.50497 | load: 0.00244 [I 2022-10-26 14:33:18] ...trap/engines/engine.py.230: loss: 0.19953 [I 2022-10-26 14:33:18] ...optimizers/trijoint.py.151: total_norm: 2.864731 [I 2022-10-26 14:33:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 970/2304 [I 2022-10-26 14:33:23] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:11:02 [I 2022-10-26 14:33:23] ...trap/engines/engine.py.229: process: 0.49753 | load: 0.00247 [I 2022-10-26 14:33:23] ...trap/engines/engine.py.230: loss: 0.22192 [I 2022-10-26 14:33:23] ...optimizers/trijoint.py.151: total_norm: 2.812352 [I 2022-10-26 14:33:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 980/2304 [I 2022-10-26 14:33:28] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:11:01 [I 2022-10-26 14:33:28] ...trap/engines/engine.py.229: process: 0.49957 | load: 0.00249 [I 2022-10-26 14:33:28] ...trap/engines/engine.py.230: loss: 0.24828 [I 2022-10-26 14:33:28] ...optimizers/trijoint.py.151: total_norm: 4.455943 [I 2022-10-26 14:33:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 990/2304 [I 2022-10-26 14:33:33] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:10:55 [I 2022-10-26 14:33:33] ...trap/engines/engine.py.229: process: 0.48949 | load: 0.00244 [I 2022-10-26 14:33:33] ...trap/engines/engine.py.230: loss: 0.28945 [I 2022-10-26 14:33:33] ...optimizers/trijoint.py.151: total_norm: 3.346261 [I 2022-10-26 14:33:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 1000/2304 [I 2022-10-26 14:33:38] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:10:50 [I 2022-10-26 14:33:38] ...trap/engines/engine.py.229: process: 0.50090 | load: 0.00244 [I 2022-10-26 14:33:38] ...trap/engines/engine.py.230: loss: 0.21161 [I 2022-10-26 14:33:38] ...optimizers/trijoint.py.151: total_norm: 3.419868 [I 2022-10-26 14:33:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 1010/2304 [I 2022-10-26 14:33:43] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:10:46 [I 2022-10-26 14:33:43] ...trap/engines/engine.py.229: process: 0.49669 | load: 0.00243 [I 2022-10-26 14:33:43] ...trap/engines/engine.py.230: loss: 0.31006 [I 2022-10-26 14:33:43] ...optimizers/trijoint.py.151: total_norm: 3.723084 [I 2022-10-26 14:33:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1020/2304 [I 2022-10-26 14:33:48] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:10:40 [I 2022-10-26 14:33:48] ...trap/engines/engine.py.229: process: 0.50091 | load: 0.00251 [I 2022-10-26 14:33:48] ...trap/engines/engine.py.230: loss: 0.21419 [I 2022-10-26 14:33:48] ...optimizers/trijoint.py.151: total_norm: 3.121875 [I 2022-10-26 14:33:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1030/2304 [I 2022-10-26 14:33:53] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:10:30 [I 2022-10-26 14:33:53] ...trap/engines/engine.py.229: process: 0.49125 | load: 0.00246 [I 2022-10-26 14:33:53] ...trap/engines/engine.py.230: loss: 0.26959 [I 2022-10-26 14:33:53] ...optimizers/trijoint.py.151: total_norm: 3.596382 [I 2022-10-26 14:33:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1040/2304 [I 2022-10-26 14:33:58] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:10:27 [I 2022-10-26 14:33:58] ...trap/engines/engine.py.229: process: 0.49960 | load: 0.00248 [I 2022-10-26 14:33:58] ...trap/engines/engine.py.230: loss: 0.16742 [I 2022-10-26 14:33:58] ...optimizers/trijoint.py.151: total_norm: 3.057735 [I 2022-10-26 14:34:02] ...trap/engines/engine.py.224: train: epoch 13 | batch 1050/2304 [I 2022-10-26 14:34:02] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:10:19 [I 2022-10-26 14:34:02] ...trap/engines/engine.py.229: process: 0.48934 | load: 0.00245 [I 2022-10-26 14:34:02] ...trap/engines/engine.py.230: loss: 0.25922 [I 2022-10-26 14:34:02] ...optimizers/trijoint.py.151: total_norm: 3.367251 [I 2022-10-26 14:34:07] ...trap/engines/engine.py.224: train: epoch 13 | batch 1060/2304 [I 2022-10-26 14:34:07] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:10:20 [I 2022-10-26 14:34:07] ...trap/engines/engine.py.229: process: 0.50235 | load: 0.00250 [I 2022-10-26 14:34:07] ...trap/engines/engine.py.230: loss: 0.20244 [I 2022-10-26 14:34:07] ...optimizers/trijoint.py.151: total_norm: 4.074962 [I 2022-10-26 14:34:12] ...trap/engines/engine.py.224: train: epoch 13 | batch 1070/2304 [I 2022-10-26 14:34:12] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:10:13 [I 2022-10-26 14:34:12] ...trap/engines/engine.py.229: process: 0.49275 | load: 0.00240 [I 2022-10-26 14:34:12] ...trap/engines/engine.py.230: loss: 0.26729 [I 2022-10-26 14:34:12] ...optimizers/trijoint.py.151: total_norm: 3.248194 [I 2022-10-26 14:34:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 1080/2304 [I 2022-10-26 14:34:17] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:10:03 [I 2022-10-26 14:34:17] ...trap/engines/engine.py.229: process: 0.48758 | load: 0.00251 [I 2022-10-26 14:34:17] ...trap/engines/engine.py.230: loss: 0.28188 [I 2022-10-26 14:34:17] ...optimizers/trijoint.py.151: total_norm: 3.606935 [I 2022-10-26 14:34:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 1090/2304 [I 2022-10-26 14:34:22] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:10:02 [I 2022-10-26 14:34:22] ...trap/engines/engine.py.229: process: 0.49416 | load: 0.00249 [I 2022-10-26 14:34:22] ...trap/engines/engine.py.230: loss: 0.21687 [I 2022-10-26 14:34:22] ...optimizers/trijoint.py.151: total_norm: 3.341000 [I 2022-10-26 14:34:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 1100/2304 [I 2022-10-26 14:34:27] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:10:04 [I 2022-10-26 14:34:27] ...trap/engines/engine.py.229: process: 0.50615 | load: 0.00244 [I 2022-10-26 14:34:27] ...trap/engines/engine.py.230: loss: 0.22175 [I 2022-10-26 14:34:27] ...optimizers/trijoint.py.151: total_norm: 3.318209 [I 2022-10-26 14:34:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 1110/2304 [I 2022-10-26 14:34:32] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:10:00 [I 2022-10-26 14:34:32] ...trap/engines/engine.py.229: process: 0.50570 | load: 0.00244 [I 2022-10-26 14:34:32] ...trap/engines/engine.py.230: loss: 0.20135 [I 2022-10-26 14:34:32] ...optimizers/trijoint.py.151: total_norm: 2.995679 [I 2022-10-26 14:34:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 1120/2304 [I 2022-10-26 14:34:37] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:09:53 [I 2022-10-26 14:34:37] ...trap/engines/engine.py.229: process: 0.49709 | load: 0.00243 [I 2022-10-26 14:34:37] ...trap/engines/engine.py.230: loss: 0.20760 [I 2022-10-26 14:34:37] ...optimizers/trijoint.py.151: total_norm: 3.612098 [I 2022-10-26 14:34:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 1130/2304 [I 2022-10-26 14:34:42] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:09:46 [I 2022-10-26 14:34:42] ...trap/engines/engine.py.229: process: 0.50092 | load: 0.00244 [I 2022-10-26 14:34:42] ...trap/engines/engine.py.230: loss: 0.18277 [I 2022-10-26 14:34:42] ...optimizers/trijoint.py.151: total_norm: 3.245591 [I 2022-10-26 14:34:47] ...trap/engines/engine.py.224: train: epoch 13 | batch 1140/2304 [I 2022-10-26 14:34:47] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:09:39 [I 2022-10-26 14:34:47] ...trap/engines/engine.py.229: process: 0.49688 | load: 0.00254 [I 2022-10-26 14:34:47] ...trap/engines/engine.py.230: loss: 0.23714 [I 2022-10-26 14:34:47] ...optimizers/trijoint.py.151: total_norm: 3.366783 [I 2022-10-26 14:34:52] ...trap/engines/engine.py.224: train: epoch 13 | batch 1150/2304 [I 2022-10-26 14:34:52] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:09:36 [I 2022-10-26 14:34:52] ...trap/engines/engine.py.229: process: 0.49489 | load: 0.00246 [I 2022-10-26 14:34:52] ...trap/engines/engine.py.230: loss: 0.25813 [I 2022-10-26 14:34:52] ...optimizers/trijoint.py.151: total_norm: 2.926986 [I 2022-10-26 14:34:57] ...trap/engines/engine.py.224: train: epoch 13 | batch 1160/2304 [I 2022-10-26 14:34:57] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:09:28 [I 2022-10-26 14:34:57] ...trap/engines/engine.py.229: process: 0.49668 | load: 0.00252 [I 2022-10-26 14:34:57] ...trap/engines/engine.py.230: loss: 0.23456 [I 2022-10-26 14:34:57] ...optimizers/trijoint.py.151: total_norm: 3.614930 [I 2022-10-26 14:35:02] ...trap/engines/engine.py.224: train: epoch 13 | batch 1170/2304 [I 2022-10-26 14:35:02] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:09:21 [I 2022-10-26 14:35:02] ...trap/engines/engine.py.229: process: 0.48726 | load: 0.00243 [I 2022-10-26 14:35:02] ...trap/engines/engine.py.230: loss: 0.26892 [I 2022-10-26 14:35:02] ...optimizers/trijoint.py.151: total_norm: 4.381383 [I 2022-10-26 14:35:07] ...trap/engines/engine.py.224: train: epoch 13 | batch 1180/2304 [I 2022-10-26 14:35:07] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:09:20 [I 2022-10-26 14:35:07] ...trap/engines/engine.py.229: process: 0.50098 | load: 0.00245 [I 2022-10-26 14:35:07] ...trap/engines/engine.py.230: loss: 0.20383 [I 2022-10-26 14:35:07] ...optimizers/trijoint.py.151: total_norm: 3.447814 [I 2022-10-26 14:35:12] ...trap/engines/engine.py.224: train: epoch 13 | batch 1190/2304 [I 2022-10-26 14:35:12] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:09:11 [I 2022-10-26 14:35:12] ...trap/engines/engine.py.229: process: 0.48653 | load: 0.00244 [I 2022-10-26 14:35:12] ...trap/engines/engine.py.230: loss: 0.19547 [I 2022-10-26 14:35:12] ...optimizers/trijoint.py.151: total_norm: 3.735844 [I 2022-10-26 14:35:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 1200/2304 [I 2022-10-26 14:35:18] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:09:17 [I 2022-10-26 14:35:18] ...trap/engines/engine.py.229: process: 0.49034 | load: 0.00246 [I 2022-10-26 14:35:18] ...trap/engines/engine.py.230: loss: 0.13525 [I 2022-10-26 14:35:18] ...optimizers/trijoint.py.151: total_norm: 1.864099 [I 2022-10-26 14:35:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 1210/2304 [I 2022-10-26 14:35:23] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:09:10 [I 2022-10-26 14:35:23] ...trap/engines/engine.py.229: process: 0.49741 | load: 0.00259 [I 2022-10-26 14:35:23] ...trap/engines/engine.py.230: loss: 0.21304 [I 2022-10-26 14:35:23] ...optimizers/trijoint.py.151: total_norm: 3.009299 [I 2022-10-26 14:35:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 1220/2304 [I 2022-10-26 14:35:28] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:09:04 [I 2022-10-26 14:35:28] ...trap/engines/engine.py.229: process: 0.49486 | load: 0.00243 [I 2022-10-26 14:35:28] ...trap/engines/engine.py.230: loss: 0.20164 [I 2022-10-26 14:35:28] ...optimizers/trijoint.py.151: total_norm: 2.833477 [I 2022-10-26 14:35:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 1230/2304 [I 2022-10-26 14:35:33] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:08:54 [I 2022-10-26 14:35:33] ...trap/engines/engine.py.229: process: 0.49664 | load: 0.00240 [I 2022-10-26 14:35:33] ...trap/engines/engine.py.230: loss: 0.25678 [I 2022-10-26 14:35:33] ...optimizers/trijoint.py.151: total_norm: 3.195647 [I 2022-10-26 14:35:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 1240/2304 [I 2022-10-26 14:35:38] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:08:52 [I 2022-10-26 14:35:38] ...trap/engines/engine.py.229: process: 0.50450 | load: 0.00240 [I 2022-10-26 14:35:38] ...trap/engines/engine.py.230: loss: 0.29746 [I 2022-10-26 14:35:38] ...optimizers/trijoint.py.151: total_norm: 3.991264 [I 2022-10-26 14:35:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 1250/2304 [I 2022-10-26 14:35:43] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:08:45 [I 2022-10-26 14:35:43] ...trap/engines/engine.py.229: process: 0.50408 | load: 0.00249 [I 2022-10-26 14:35:43] ...trap/engines/engine.py.230: loss: 0.20580 [I 2022-10-26 14:35:43] ...optimizers/trijoint.py.151: total_norm: 3.150357 [I 2022-10-26 14:35:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1260/2304 [I 2022-10-26 14:35:48] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:08:43 [I 2022-10-26 14:35:48] ...trap/engines/engine.py.229: process: 0.50404 | load: 0.00248 [I 2022-10-26 14:35:48] ...trap/engines/engine.py.230: loss: 0.20064 [I 2022-10-26 14:35:48] ...optimizers/trijoint.py.151: total_norm: 3.226623 [I 2022-10-26 14:35:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1270/2304 [I 2022-10-26 14:35:53] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:08:35 [I 2022-10-26 14:35:53] ...trap/engines/engine.py.229: process: 0.50010 | load: 0.00250 [I 2022-10-26 14:35:53] ...trap/engines/engine.py.230: loss: 0.29797 [I 2022-10-26 14:35:53] ...optimizers/trijoint.py.151: total_norm: 3.833995 [I 2022-10-26 14:35:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1280/2304 [I 2022-10-26 14:35:58] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:08:32 [I 2022-10-26 14:35:58] ...trap/engines/engine.py.229: process: 0.50259 | load: 0.00251 [I 2022-10-26 14:35:58] ...trap/engines/engine.py.230: loss: 0.23933 [I 2022-10-26 14:35:58] ...optimizers/trijoint.py.151: total_norm: 3.980630 [I 2022-10-26 14:36:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 1290/2304 [I 2022-10-26 14:36:03] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:08:29 [I 2022-10-26 14:36:03] ...trap/engines/engine.py.229: process: 0.49688 | load: 0.00242 [I 2022-10-26 14:36:03] ...trap/engines/engine.py.230: loss: 0.22492 [I 2022-10-26 14:36:03] ...optimizers/trijoint.py.151: total_norm: 3.033978 [I 2022-10-26 14:36:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 1300/2304 [I 2022-10-26 14:36:08] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:08:29 [I 2022-10-26 14:36:08] ...trap/engines/engine.py.229: process: 0.49907 | load: 0.00245 [I 2022-10-26 14:36:08] ...trap/engines/engine.py.230: loss: 0.21504 [I 2022-10-26 14:36:08] ...optimizers/trijoint.py.151: total_norm: 3.051152 [I 2022-10-26 14:36:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 1310/2304 [I 2022-10-26 14:36:13] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:08:16 [I 2022-10-26 14:36:13] ...trap/engines/engine.py.229: process: 0.51227 | load: 0.00243 [I 2022-10-26 14:36:13] ...trap/engines/engine.py.230: loss: 0.20808 [I 2022-10-26 14:36:13] ...optimizers/trijoint.py.151: total_norm: 3.275782 [I 2022-10-26 14:36:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 1320/2304 [I 2022-10-26 14:36:18] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:08:10 [I 2022-10-26 14:36:18] ...trap/engines/engine.py.229: process: 0.49872 | load: 0.00251 [I 2022-10-26 14:36:18] ...trap/engines/engine.py.230: loss: 0.19411 [I 2022-10-26 14:36:18] ...optimizers/trijoint.py.151: total_norm: 3.061422 [I 2022-10-26 14:36:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 1330/2304 [I 2022-10-26 14:36:23] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:08:04 [I 2022-10-26 14:36:23] ...trap/engines/engine.py.229: process: 0.49960 | load: 0.00246 [I 2022-10-26 14:36:23] ...trap/engines/engine.py.230: loss: 0.17258 [I 2022-10-26 14:36:23] ...optimizers/trijoint.py.151: total_norm: 2.868595 [I 2022-10-26 14:36:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 1340/2304 [I 2022-10-26 14:36:28] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:07:58 [I 2022-10-26 14:36:28] ...trap/engines/engine.py.229: process: 0.49540 | load: 0.00242 [I 2022-10-26 14:36:28] ...trap/engines/engine.py.230: loss: 0.25034 [I 2022-10-26 14:36:28] ...optimizers/trijoint.py.151: total_norm: 2.806891 [I 2022-10-26 14:36:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 1350/2304 [I 2022-10-26 14:36:33] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:07:51 [I 2022-10-26 14:36:33] ...trap/engines/engine.py.229: process: 0.48522 | load: 0.00252 [I 2022-10-26 14:36:33] ...trap/engines/engine.py.230: loss: 0.14712 [I 2022-10-26 14:36:33] ...optimizers/trijoint.py.151: total_norm: 2.226200 [I 2022-10-26 14:36:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 1360/2304 [I 2022-10-26 14:36:38] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:07:47 [I 2022-10-26 14:36:38] ...trap/engines/engine.py.229: process: 0.49492 | load: 0.00243 [I 2022-10-26 14:36:38] ...trap/engines/engine.py.230: loss: 0.23455 [I 2022-10-26 14:36:38] ...optimizers/trijoint.py.151: total_norm: 3.304161 [I 2022-10-26 14:36:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 1370/2304 [I 2022-10-26 14:36:43] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:07:53 [I 2022-10-26 14:36:43] ...trap/engines/engine.py.229: process: 0.49478 | load: 0.00253 [I 2022-10-26 14:36:43] ...trap/engines/engine.py.230: loss: 0.26072 [I 2022-10-26 14:36:43] ...optimizers/trijoint.py.151: total_norm: 3.477439 [I 2022-10-26 14:36:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1380/2304 [I 2022-10-26 14:36:48] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:07:43 [I 2022-10-26 14:36:48] ...trap/engines/engine.py.229: process: 0.50185 | load: 0.00248 [I 2022-10-26 14:36:48] ...trap/engines/engine.py.230: loss: 0.35001 [I 2022-10-26 14:36:48] ...optimizers/trijoint.py.151: total_norm: 3.596346 [I 2022-10-26 14:36:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1390/2304 [I 2022-10-26 14:36:53] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:07:36 [I 2022-10-26 14:36:53] ...trap/engines/engine.py.229: process: 0.50369 | load: 0.00246 [I 2022-10-26 14:36:53] ...trap/engines/engine.py.230: loss: 0.21372 [I 2022-10-26 14:36:53] ...optimizers/trijoint.py.151: total_norm: 3.245628 [I 2022-10-26 14:36:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1400/2304 [I 2022-10-26 14:36:58] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:07:29 [I 2022-10-26 14:36:58] ...trap/engines/engine.py.229: process: 0.50163 | load: 0.00243 [I 2022-10-26 14:36:58] ...trap/engines/engine.py.230: loss: 0.23982 [I 2022-10-26 14:36:58] ...optimizers/trijoint.py.151: total_norm: 2.740636 [I 2022-10-26 14:37:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 1410/2304 [I 2022-10-26 14:37:03] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:07:27 [I 2022-10-26 14:37:03] ...trap/engines/engine.py.229: process: 0.50304 | load: 0.00244 [I 2022-10-26 14:37:03] ...trap/engines/engine.py.230: loss: 0.23019 [I 2022-10-26 14:37:03] ...optimizers/trijoint.py.151: total_norm: 3.258471 [I 2022-10-26 14:37:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 1420/2304 [I 2022-10-26 14:37:08] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:07:17 [I 2022-10-26 14:37:08] ...trap/engines/engine.py.229: process: 0.49021 | load: 0.00248 [I 2022-10-26 14:37:08] ...trap/engines/engine.py.230: loss: 0.17614 [I 2022-10-26 14:37:08] ...optimizers/trijoint.py.151: total_norm: 2.537576 [I 2022-10-26 14:37:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 1430/2304 [I 2022-10-26 14:37:13] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:07:16 [I 2022-10-26 14:37:13] ...trap/engines/engine.py.229: process: 0.50285 | load: 0.00247 [I 2022-10-26 14:37:13] ...trap/engines/engine.py.230: loss: 0.26534 [I 2022-10-26 14:37:13] ...optimizers/trijoint.py.151: total_norm: 3.665601 [I 2022-10-26 14:37:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 1440/2304 [I 2022-10-26 14:37:18] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:07:12 [I 2022-10-26 14:37:18] ...trap/engines/engine.py.229: process: 0.50403 | load: 0.00244 [I 2022-10-26 14:37:18] ...trap/engines/engine.py.230: loss: 0.18723 [I 2022-10-26 14:37:18] ...optimizers/trijoint.py.151: total_norm: 2.854594 [I 2022-10-26 14:37:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 1450/2304 [I 2022-10-26 14:37:23] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:07:05 [I 2022-10-26 14:37:23] ...trap/engines/engine.py.229: process: 0.49755 | load: 0.00244 [I 2022-10-26 14:37:23] ...trap/engines/engine.py.230: loss: 0.19018 [I 2022-10-26 14:37:23] ...optimizers/trijoint.py.151: total_norm: 2.789935 [I 2022-10-26 14:37:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 1460/2304 [I 2022-10-26 14:37:28] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:06:58 [I 2022-10-26 14:37:28] ...trap/engines/engine.py.229: process: 0.49436 | load: 0.00250 [I 2022-10-26 14:37:28] ...trap/engines/engine.py.230: loss: 0.14043 [I 2022-10-26 14:37:28] ...optimizers/trijoint.py.151: total_norm: 2.546698 [I 2022-10-26 14:37:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 1470/2304 [I 2022-10-26 14:37:33] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:06:55 [I 2022-10-26 14:37:33] ...trap/engines/engine.py.229: process: 0.49921 | load: 0.00261 [I 2022-10-26 14:37:33] ...trap/engines/engine.py.230: loss: 0.22225 [I 2022-10-26 14:37:33] ...optimizers/trijoint.py.151: total_norm: 3.302727 [I 2022-10-26 14:37:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 1480/2304 [I 2022-10-26 14:37:38] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:06:50 [I 2022-10-26 14:37:38] ...trap/engines/engine.py.229: process: 0.50415 | load: 0.00247 [I 2022-10-26 14:37:38] ...trap/engines/engine.py.230: loss: 0.27690 [I 2022-10-26 14:37:38] ...optimizers/trijoint.py.151: total_norm: 3.739003 [I 2022-10-26 14:37:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 1490/2304 [I 2022-10-26 14:37:43] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:06:50 [I 2022-10-26 14:37:43] ...trap/engines/engine.py.229: process: 0.49332 | load: 0.00247 [I 2022-10-26 14:37:43] ...trap/engines/engine.py.230: loss: 0.20218 [I 2022-10-26 14:37:43] ...optimizers/trijoint.py.151: total_norm: 3.000362 [I 2022-10-26 14:37:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1500/2304 [I 2022-10-26 14:37:48] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:06:41 [I 2022-10-26 14:37:48] ...trap/engines/engine.py.229: process: 0.49983 | load: 0.00246 [I 2022-10-26 14:37:48] ...trap/engines/engine.py.230: loss: 0.24541 [I 2022-10-26 14:37:48] ...optimizers/trijoint.py.151: total_norm: 3.239394 [I 2022-10-26 14:37:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1510/2304 [I 2022-10-26 14:37:53] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:06:34 [I 2022-10-26 14:37:53] ...trap/engines/engine.py.229: process: 0.49818 | load: 0.00242 [I 2022-10-26 14:37:53] ...trap/engines/engine.py.230: loss: 0.25038 [I 2022-10-26 14:37:53] ...optimizers/trijoint.py.151: total_norm: 3.304931 [I 2022-10-26 14:37:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1520/2304 [I 2022-10-26 14:37:58] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:06:32 [I 2022-10-26 14:37:58] ...trap/engines/engine.py.229: process: 0.49814 | load: 0.00259 [I 2022-10-26 14:37:58] ...trap/engines/engine.py.230: loss: 0.22578 [I 2022-10-26 14:37:58] ...optimizers/trijoint.py.151: total_norm: 3.426564 [I 2022-10-26 14:38:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 1530/2304 [I 2022-10-26 14:38:03] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:06:25 [I 2022-10-26 14:38:03] ...trap/engines/engine.py.229: process: 0.49544 | load: 0.00245 [I 2022-10-26 14:38:03] ...trap/engines/engine.py.230: loss: 0.17764 [I 2022-10-26 14:38:03] ...optimizers/trijoint.py.151: total_norm: 2.580763 [I 2022-10-26 14:38:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 1540/2304 [I 2022-10-26 14:38:08] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:06:20 [I 2022-10-26 14:38:08] ...trap/engines/engine.py.229: process: 0.49218 | load: 0.00245 [I 2022-10-26 14:38:08] ...trap/engines/engine.py.230: loss: 0.15784 [I 2022-10-26 14:38:08] ...optimizers/trijoint.py.151: total_norm: 2.895364 [I 2022-10-26 14:38:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 1550/2304 [I 2022-10-26 14:38:13] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:06:16 [I 2022-10-26 14:38:13] ...trap/engines/engine.py.229: process: 0.50614 | load: 0.00244 [I 2022-10-26 14:38:13] ...trap/engines/engine.py.230: loss: 0.22015 [I 2022-10-26 14:38:13] ...optimizers/trijoint.py.151: total_norm: 2.922175 [I 2022-10-26 14:38:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 1560/2304 [I 2022-10-26 14:38:18] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:06:09 [I 2022-10-26 14:38:18] ...trap/engines/engine.py.229: process: 0.50025 | load: 0.00243 [I 2022-10-26 14:38:18] ...trap/engines/engine.py.230: loss: 0.17267 [I 2022-10-26 14:38:18] ...optimizers/trijoint.py.151: total_norm: 3.395720 [I 2022-10-26 14:38:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 1570/2304 [I 2022-10-26 14:38:23] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:06:01 [I 2022-10-26 14:38:23] ...trap/engines/engine.py.229: process: 0.48377 | load: 0.00244 [I 2022-10-26 14:38:23] ...trap/engines/engine.py.230: loss: 0.20001 [I 2022-10-26 14:38:23] ...optimizers/trijoint.py.151: total_norm: 3.062527 [I 2022-10-26 14:38:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 1580/2304 [I 2022-10-26 14:38:28] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:05:59 [I 2022-10-26 14:38:28] ...trap/engines/engine.py.229: process: 0.50154 | load: 0.00250 [I 2022-10-26 14:38:28] ...trap/engines/engine.py.230: loss: 0.22313 [I 2022-10-26 14:38:28] ...optimizers/trijoint.py.151: total_norm: 2.514556 [I 2022-10-26 14:38:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 1590/2304 [I 2022-10-26 14:38:33] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:05:52 [I 2022-10-26 14:38:33] ...trap/engines/engine.py.229: process: 0.49753 | load: 0.00246 [I 2022-10-26 14:38:33] ...trap/engines/engine.py.230: loss: 0.15085 [I 2022-10-26 14:38:33] ...optimizers/trijoint.py.151: total_norm: 2.882316 [I 2022-10-26 14:38:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 1600/2304 [I 2022-10-26 14:38:38] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:05:48 [I 2022-10-26 14:38:38] ...trap/engines/engine.py.229: process: 0.48969 | load: 0.00244 [I 2022-10-26 14:38:38] ...trap/engines/engine.py.230: loss: 0.27372 [I 2022-10-26 14:38:38] ...optimizers/trijoint.py.151: total_norm: 2.687542 [I 2022-10-26 14:38:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 1610/2304 [I 2022-10-26 14:38:43] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:05:43 [I 2022-10-26 14:38:43] ...trap/engines/engine.py.229: process: 0.49822 | load: 0.00242 [I 2022-10-26 14:38:43] ...trap/engines/engine.py.230: loss: 0.30289 [I 2022-10-26 14:38:43] ...optimizers/trijoint.py.151: total_norm: 3.470489 [I 2022-10-26 14:38:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1620/2304 [I 2022-10-26 14:38:48] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:05:45 [I 2022-10-26 14:38:48] ...trap/engines/engine.py.229: process: 0.50054 | load: 0.00243 [I 2022-10-26 14:38:48] ...trap/engines/engine.py.230: loss: 0.19031 [I 2022-10-26 14:38:48] ...optimizers/trijoint.py.151: total_norm: 2.451853 [I 2022-10-26 14:38:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1630/2304 [I 2022-10-26 14:38:53] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:05:39 [I 2022-10-26 14:38:53] ...trap/engines/engine.py.229: process: 0.50198 | load: 0.00264 [I 2022-10-26 14:38:53] ...trap/engines/engine.py.230: loss: 0.23112 [I 2022-10-26 14:38:53] ...optimizers/trijoint.py.151: total_norm: 3.659025 [I 2022-10-26 14:38:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1640/2304 [I 2022-10-26 14:38:58] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:05:28 [I 2022-10-26 14:38:58] ...trap/engines/engine.py.229: process: 0.48144 | load: 0.00248 [I 2022-10-26 14:38:58] ...trap/engines/engine.py.230: loss: 0.22399 [I 2022-10-26 14:38:58] ...optimizers/trijoint.py.151: total_norm: 3.428479 [I 2022-10-26 14:39:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 1650/2304 [I 2022-10-26 14:39:03] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:05:25 [I 2022-10-26 14:39:03] ...trap/engines/engine.py.229: process: 0.49620 | load: 0.00246 [I 2022-10-26 14:39:03] ...trap/engines/engine.py.230: loss: 0.23826 [I 2022-10-26 14:39:03] ...optimizers/trijoint.py.151: total_norm: 4.091709 [I 2022-10-26 14:39:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 1660/2304 [I 2022-10-26 14:39:08] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:05:21 [I 2022-10-26 14:39:08] ...trap/engines/engine.py.229: process: 0.49791 | load: 0.00246 [I 2022-10-26 14:39:08] ...trap/engines/engine.py.230: loss: 0.24607 [I 2022-10-26 14:39:08] ...optimizers/trijoint.py.151: total_norm: 4.098259 [I 2022-10-26 14:39:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 1670/2304 [I 2022-10-26 14:39:13] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:05:11 [I 2022-10-26 14:39:13] ...trap/engines/engine.py.229: process: 0.47440 | load: 0.00243 [I 2022-10-26 14:39:13] ...trap/engines/engine.py.230: loss: 0.21801 [I 2022-10-26 14:39:13] ...optimizers/trijoint.py.151: total_norm: 4.206482 [I 2022-10-26 14:39:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 1680/2304 [I 2022-10-26 14:39:18] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:05:08 [I 2022-10-26 14:39:18] ...trap/engines/engine.py.229: process: 0.49580 | load: 0.00241 [I 2022-10-26 14:39:18] ...trap/engines/engine.py.230: loss: 0.20697 [I 2022-10-26 14:39:18] ...optimizers/trijoint.py.151: total_norm: 2.234921 [I 2022-10-26 14:39:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 1690/2304 [I 2022-10-26 14:39:23] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:05:05 [I 2022-10-26 14:39:23] ...trap/engines/engine.py.229: process: 0.50001 | load: 0.00249 [I 2022-10-26 14:39:23] ...trap/engines/engine.py.230: loss: 0.18599 [I 2022-10-26 14:39:23] ...optimizers/trijoint.py.151: total_norm: 2.811363 [I 2022-10-26 14:39:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 1700/2304 [I 2022-10-26 14:39:28] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:05:01 [I 2022-10-26 14:39:28] ...trap/engines/engine.py.229: process: 0.49158 | load: 0.00247 [I 2022-10-26 14:39:28] ...trap/engines/engine.py.230: loss: 0.15585 [I 2022-10-26 14:39:28] ...optimizers/trijoint.py.151: total_norm: 2.997569 [I 2022-10-26 14:39:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 1710/2304 [I 2022-10-26 14:39:33] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:04:56 [I 2022-10-26 14:39:33] ...trap/engines/engine.py.229: process: 0.50232 | load: 0.00245 [I 2022-10-26 14:39:33] ...trap/engines/engine.py.230: loss: 0.32183 [I 2022-10-26 14:39:33] ...optimizers/trijoint.py.151: total_norm: 4.693582 [I 2022-10-26 14:39:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 1720/2304 [I 2022-10-26 14:39:38] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:04:49 [I 2022-10-26 14:39:38] ...trap/engines/engine.py.229: process: 0.49132 | load: 0.00246 [I 2022-10-26 14:39:38] ...trap/engines/engine.py.230: loss: 0.17810 [I 2022-10-26 14:39:38] ...optimizers/trijoint.py.151: total_norm: 2.639626 [I 2022-10-26 14:39:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 1730/2304 [I 2022-10-26 14:39:43] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:04:48 [I 2022-10-26 14:39:43] ...trap/engines/engine.py.229: process: 0.50194 | load: 0.00245 [I 2022-10-26 14:39:43] ...trap/engines/engine.py.230: loss: 0.17344 [I 2022-10-26 14:39:43] ...optimizers/trijoint.py.151: total_norm: 2.727152 [I 2022-10-26 14:39:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1740/2304 [I 2022-10-26 14:39:48] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:04:42 [I 2022-10-26 14:39:48] ...trap/engines/engine.py.229: process: 0.49034 | load: 0.00260 [I 2022-10-26 14:39:48] ...trap/engines/engine.py.230: loss: 0.25477 [I 2022-10-26 14:39:48] ...optimizers/trijoint.py.151: total_norm: 3.919916 [I 2022-10-26 14:39:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1750/2304 [I 2022-10-26 14:39:53] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:04:46 [I 2022-10-26 14:39:53] ...trap/engines/engine.py.229: process: 0.49826 | load: 0.00246 [I 2022-10-26 14:39:53] ...trap/engines/engine.py.230: loss: 0.19962 [I 2022-10-26 14:39:53] ...optimizers/trijoint.py.151: total_norm: 3.694111 [I 2022-10-26 14:39:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1760/2304 [I 2022-10-26 14:39:58] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:04:31 [I 2022-10-26 14:39:58] ...trap/engines/engine.py.229: process: 0.49513 | load: 0.00242 [I 2022-10-26 14:39:58] ...trap/engines/engine.py.230: loss: 0.22899 [I 2022-10-26 14:39:58] ...optimizers/trijoint.py.151: total_norm: 4.600993 [I 2022-10-26 14:40:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 1770/2304 [I 2022-10-26 14:40:03] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:04:25 [I 2022-10-26 14:40:03] ...trap/engines/engine.py.229: process: 0.50058 | load: 0.00243 [I 2022-10-26 14:40:03] ...trap/engines/engine.py.230: loss: 0.21301 [I 2022-10-26 14:40:03] ...optimizers/trijoint.py.151: total_norm: 2.582607 [I 2022-10-26 14:40:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 1780/2304 [I 2022-10-26 14:40:08] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:04:19 [I 2022-10-26 14:40:08] ...trap/engines/engine.py.229: process: 0.49902 | load: 0.00242 [I 2022-10-26 14:40:08] ...trap/engines/engine.py.230: loss: 0.16380 [I 2022-10-26 14:40:08] ...optimizers/trijoint.py.151: total_norm: 3.372463 [I 2022-10-26 14:40:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 1790/2304 [I 2022-10-26 14:40:13] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:04:16 [I 2022-10-26 14:40:13] ...trap/engines/engine.py.229: process: 0.49583 | load: 0.00245 [I 2022-10-26 14:40:13] ...trap/engines/engine.py.230: loss: 0.16001 [I 2022-10-26 14:40:13] ...optimizers/trijoint.py.151: total_norm: 2.379423 [I 2022-10-26 14:40:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 1800/2304 [I 2022-10-26 14:40:18] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:04:10 [I 2022-10-26 14:40:18] ...trap/engines/engine.py.229: process: 0.49329 | load: 0.00243 [I 2022-10-26 14:40:18] ...trap/engines/engine.py.230: loss: 0.26594 [I 2022-10-26 14:40:18] ...optimizers/trijoint.py.151: total_norm: 3.267927 [I 2022-10-26 14:40:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 1810/2304 [I 2022-10-26 14:40:23] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:04:06 [I 2022-10-26 14:40:23] ...trap/engines/engine.py.229: process: 0.49955 | load: 0.00249 [I 2022-10-26 14:40:23] ...trap/engines/engine.py.230: loss: 0.19294 [I 2022-10-26 14:40:23] ...optimizers/trijoint.py.151: total_norm: 2.907169 [I 2022-10-26 14:40:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 1820/2304 [I 2022-10-26 14:40:28] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:04:00 [I 2022-10-26 14:40:28] ...trap/engines/engine.py.229: process: 0.49559 | load: 0.00245 [I 2022-10-26 14:40:28] ...trap/engines/engine.py.230: loss: 0.21464 [I 2022-10-26 14:40:28] ...optimizers/trijoint.py.151: total_norm: 4.060165 [I 2022-10-26 14:40:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 1830/2304 [I 2022-10-26 14:40:33] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:03:54 [I 2022-10-26 14:40:33] ...trap/engines/engine.py.229: process: 0.50276 | load: 0.00248 [I 2022-10-26 14:40:33] ...trap/engines/engine.py.230: loss: 0.23413 [I 2022-10-26 14:40:33] ...optimizers/trijoint.py.151: total_norm: 4.175290 [I 2022-10-26 14:40:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 1840/2304 [I 2022-10-26 14:40:38] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:03:51 [I 2022-10-26 14:40:38] ...trap/engines/engine.py.229: process: 0.49888 | load: 0.00247 [I 2022-10-26 14:40:38] ...trap/engines/engine.py.230: loss: 0.18807 [I 2022-10-26 14:40:38] ...optimizers/trijoint.py.151: total_norm: 3.287066 [I 2022-10-26 14:40:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 1850/2304 [I 2022-10-26 14:40:43] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:03:47 [I 2022-10-26 14:40:43] ...trap/engines/engine.py.229: process: 0.50375 | load: 0.00256 [I 2022-10-26 14:40:43] ...trap/engines/engine.py.230: loss: 0.16836 [I 2022-10-26 14:40:43] ...optimizers/trijoint.py.151: total_norm: 3.305234 [I 2022-10-26 14:40:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1860/2304 [I 2022-10-26 14:40:48] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:03:41 [I 2022-10-26 14:40:48] ...trap/engines/engine.py.229: process: 0.49259 | load: 0.00246 [I 2022-10-26 14:40:48] ...trap/engines/engine.py.230: loss: 0.21620 [I 2022-10-26 14:40:48] ...optimizers/trijoint.py.151: total_norm: 3.415678 [I 2022-10-26 14:40:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1870/2304 [I 2022-10-26 14:40:53] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:03:37 [I 2022-10-26 14:40:53] ...trap/engines/engine.py.229: process: 0.50361 | load: 0.00242 [I 2022-10-26 14:40:53] ...trap/engines/engine.py.230: loss: 0.20953 [I 2022-10-26 14:40:53] ...optimizers/trijoint.py.151: total_norm: 3.089125 [I 2022-10-26 14:40:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 1880/2304 [I 2022-10-26 14:40:58] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:03:31 [I 2022-10-26 14:40:58] ...trap/engines/engine.py.229: process: 0.50726 | load: 0.00243 [I 2022-10-26 14:40:58] ...trap/engines/engine.py.230: loss: 0.28239 [I 2022-10-26 14:40:58] ...optimizers/trijoint.py.151: total_norm: 4.886944 [I 2022-10-26 14:41:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 1890/2304 [I 2022-10-26 14:41:03] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:03:25 [I 2022-10-26 14:41:03] ...trap/engines/engine.py.229: process: 0.49985 | load: 0.00243 [I 2022-10-26 14:41:03] ...trap/engines/engine.py.230: loss: 0.26882 [I 2022-10-26 14:41:03] ...optimizers/trijoint.py.151: total_norm: 3.055997 [I 2022-10-26 14:41:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 1900/2304 [I 2022-10-26 14:41:08] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:03:21 [I 2022-10-26 14:41:08] ...trap/engines/engine.py.229: process: 0.50490 | load: 0.00243 [I 2022-10-26 14:41:08] ...trap/engines/engine.py.230: loss: 0.21959 [I 2022-10-26 14:41:08] ...optimizers/trijoint.py.151: total_norm: 2.777589 [I 2022-10-26 14:41:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 1910/2304 [I 2022-10-26 14:41:13] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:03:16 [I 2022-10-26 14:41:13] ...trap/engines/engine.py.229: process: 0.49896 | load: 0.00248 [I 2022-10-26 14:41:13] ...trap/engines/engine.py.230: loss: 0.17791 [I 2022-10-26 14:41:13] ...optimizers/trijoint.py.151: total_norm: 2.927560 [I 2022-10-26 14:41:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 1920/2304 [I 2022-10-26 14:41:18] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:03:11 [I 2022-10-26 14:41:18] ...trap/engines/engine.py.229: process: 0.50419 | load: 0.00251 [I 2022-10-26 14:41:18] ...trap/engines/engine.py.230: loss: 0.19809 [I 2022-10-26 14:41:18] ...optimizers/trijoint.py.151: total_norm: 2.767988 [I 2022-10-26 14:41:23] ...trap/engines/engine.py.224: train: epoch 13 | batch 1930/2304 [I 2022-10-26 14:41:23] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:03:05 [I 2022-10-26 14:41:23] ...trap/engines/engine.py.229: process: 0.49277 | load: 0.00241 [I 2022-10-26 14:41:23] ...trap/engines/engine.py.230: loss: 0.21020 [I 2022-10-26 14:41:23] ...optimizers/trijoint.py.151: total_norm: 3.577790 [I 2022-10-26 14:41:28] ...trap/engines/engine.py.224: train: epoch 13 | batch 1940/2304 [I 2022-10-26 14:41:28] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:03:01 [I 2022-10-26 14:41:28] ...trap/engines/engine.py.229: process: 0.49922 | load: 0.00244 [I 2022-10-26 14:41:28] ...trap/engines/engine.py.230: loss: 0.27537 [I 2022-10-26 14:41:28] ...optimizers/trijoint.py.151: total_norm: 3.242691 [I 2022-10-26 14:41:33] ...trap/engines/engine.py.224: train: epoch 13 | batch 1950/2304 [I 2022-10-26 14:41:33] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:02:55 [I 2022-10-26 14:41:33] ...trap/engines/engine.py.229: process: 0.48870 | load: 0.00245 [I 2022-10-26 14:41:33] ...trap/engines/engine.py.230: loss: 0.27529 [I 2022-10-26 14:41:33] ...optimizers/trijoint.py.151: total_norm: 4.021224 [I 2022-10-26 14:41:38] ...trap/engines/engine.py.224: train: epoch 13 | batch 1960/2304 [I 2022-10-26 14:41:38] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:02:51 [I 2022-10-26 14:41:38] ...trap/engines/engine.py.229: process: 0.48894 | load: 0.00244 [I 2022-10-26 14:41:38] ...trap/engines/engine.py.230: loss: 0.13316 [I 2022-10-26 14:41:38] ...optimizers/trijoint.py.151: total_norm: 2.281812 [I 2022-10-26 14:41:43] ...trap/engines/engine.py.224: train: epoch 13 | batch 1970/2304 [I 2022-10-26 14:41:43] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:02:47 [I 2022-10-26 14:41:43] ...trap/engines/engine.py.229: process: 0.49915 | load: 0.00249 [I 2022-10-26 14:41:43] ...trap/engines/engine.py.230: loss: 0.18224 [I 2022-10-26 14:41:43] ...optimizers/trijoint.py.151: total_norm: 3.429438 [I 2022-10-26 14:41:48] ...trap/engines/engine.py.224: train: epoch 13 | batch 1980/2304 [I 2022-10-26 14:41:48] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:02:41 [I 2022-10-26 14:41:48] ...trap/engines/engine.py.229: process: 0.49453 | load: 0.00243 [I 2022-10-26 14:41:48] ...trap/engines/engine.py.230: loss: 0.22293 [I 2022-10-26 14:41:48] ...optimizers/trijoint.py.151: total_norm: 2.976099 [I 2022-10-26 14:41:53] ...trap/engines/engine.py.224: train: epoch 13 | batch 1990/2304 [I 2022-10-26 14:41:53] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:02:35 [I 2022-10-26 14:41:53] ...trap/engines/engine.py.229: process: 0.50308 | load: 0.00242 [I 2022-10-26 14:41:53] ...trap/engines/engine.py.230: loss: 0.35530 [I 2022-10-26 14:41:53] ...optimizers/trijoint.py.151: total_norm: 5.052424 [I 2022-10-26 14:41:58] ...trap/engines/engine.py.224: train: epoch 13 | batch 2000/2304 [I 2022-10-26 14:41:58] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:02:31 [I 2022-10-26 14:41:58] ...trap/engines/engine.py.229: process: 0.49897 | load: 0.00245 [I 2022-10-26 14:41:58] ...trap/engines/engine.py.230: loss: 0.19920 [I 2022-10-26 14:41:58] ...optimizers/trijoint.py.151: total_norm: 3.947125 [I 2022-10-26 14:42:03] ...trap/engines/engine.py.224: train: epoch 13 | batch 2010/2304 [I 2022-10-26 14:42:03] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:02:25 [I 2022-10-26 14:42:03] ...trap/engines/engine.py.229: process: 0.49196 | load: 0.00245 [I 2022-10-26 14:42:03] ...trap/engines/engine.py.230: loss: 0.31801 [I 2022-10-26 14:42:03] ...optimizers/trijoint.py.151: total_norm: 3.353738 [I 2022-10-26 14:42:08] ...trap/engines/engine.py.224: train: epoch 13 | batch 2020/2304 [I 2022-10-26 14:42:08] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:02:21 [I 2022-10-26 14:42:08] ...trap/engines/engine.py.229: process: 0.50195 | load: 0.00243 [I 2022-10-26 14:42:08] ...trap/engines/engine.py.230: loss: 0.23746 [I 2022-10-26 14:42:08] ...optimizers/trijoint.py.151: total_norm: 4.092312 [I 2022-10-26 14:42:13] ...trap/engines/engine.py.224: train: epoch 13 | batch 2030/2304 [I 2022-10-26 14:42:13] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:02:16 [I 2022-10-26 14:42:13] ...trap/engines/engine.py.229: process: 0.49234 | load: 0.00241 [I 2022-10-26 14:42:13] ...trap/engines/engine.py.230: loss: 0.19899 [I 2022-10-26 14:42:13] ...optimizers/trijoint.py.151: total_norm: 2.956603 [I 2022-10-26 14:42:18] ...trap/engines/engine.py.224: train: epoch 13 | batch 2040/2304 [I 2022-10-26 14:42:18] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:02:11 [I 2022-10-26 14:42:18] ...trap/engines/engine.py.229: process: 0.51117 | load: 0.00249 [I 2022-10-26 14:42:18] ...trap/engines/engine.py.230: loss: 0.22731 [I 2022-10-26 14:42:18] ...optimizers/trijoint.py.151: total_norm: 4.042374 [I 2022-10-26 14:42:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 2050/2304 [I 2022-10-26 14:42:22] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:02:05 [I 2022-10-26 14:42:22] ...trap/engines/engine.py.229: process: 0.49491 | load: 0.00246 [I 2022-10-26 14:42:22] ...trap/engines/engine.py.230: loss: 0.25991 [I 2022-10-26 14:42:22] ...optimizers/trijoint.py.151: total_norm: 4.742711 [I 2022-10-26 14:42:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 2060/2304 [I 2022-10-26 14:42:27] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:02:01 [I 2022-10-26 14:42:27] ...trap/engines/engine.py.229: process: 0.50167 | load: 0.00245 [I 2022-10-26 14:42:27] ...trap/engines/engine.py.230: loss: 0.20691 [I 2022-10-26 14:42:27] ...optimizers/trijoint.py.151: total_norm: 2.807419 [I 2022-10-26 14:42:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 2070/2304 [I 2022-10-26 14:42:32] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:01:55 [I 2022-10-26 14:42:32] ...trap/engines/engine.py.229: process: 0.48993 | load: 0.00243 [I 2022-10-26 14:42:32] ...trap/engines/engine.py.230: loss: 0.20837 [I 2022-10-26 14:42:32] ...optimizers/trijoint.py.151: total_norm: 2.801884 [I 2022-10-26 14:42:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 2080/2304 [I 2022-10-26 14:42:37] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:01:51 [I 2022-10-26 14:42:37] ...trap/engines/engine.py.229: process: 0.50484 | load: 0.00242 [I 2022-10-26 14:42:37] ...trap/engines/engine.py.230: loss: 0.20645 [I 2022-10-26 14:42:37] ...optimizers/trijoint.py.151: total_norm: 2.791794 [I 2022-10-26 14:42:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 2090/2304 [I 2022-10-26 14:42:42] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:01:47 [I 2022-10-26 14:42:42] ...trap/engines/engine.py.229: process: 0.49708 | load: 0.00242 [I 2022-10-26 14:42:42] ...trap/engines/engine.py.230: loss: 0.17387 [I 2022-10-26 14:42:42] ...optimizers/trijoint.py.151: total_norm: 2.773613 [I 2022-10-26 14:42:47] ...trap/engines/engine.py.224: train: epoch 13 | batch 2100/2304 [I 2022-10-26 14:42:47] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:01:41 [I 2022-10-26 14:42:47] ...trap/engines/engine.py.229: process: 0.49658 | load: 0.00244 [I 2022-10-26 14:42:47] ...trap/engines/engine.py.230: loss: 0.25098 [I 2022-10-26 14:42:47] ...optimizers/trijoint.py.151: total_norm: 3.444160 [I 2022-10-26 14:42:52] ...trap/engines/engine.py.224: train: epoch 13 | batch 2110/2304 [I 2022-10-26 14:42:52] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:01:36 [I 2022-10-26 14:42:52] ...trap/engines/engine.py.229: process: 0.50356 | load: 0.00243 [I 2022-10-26 14:42:52] ...trap/engines/engine.py.230: loss: 0.13865 [I 2022-10-26 14:42:52] ...optimizers/trijoint.py.151: total_norm: 2.444308 [I 2022-10-26 14:42:57] ...trap/engines/engine.py.224: train: epoch 13 | batch 2120/2304 [I 2022-10-26 14:42:57] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:01:31 [I 2022-10-26 14:42:57] ...trap/engines/engine.py.229: process: 0.49453 | load: 0.00244 [I 2022-10-26 14:42:57] ...trap/engines/engine.py.230: loss: 0.31004 [I 2022-10-26 14:42:57] ...optimizers/trijoint.py.151: total_norm: 3.891369 [I 2022-10-26 14:43:02] ...trap/engines/engine.py.224: train: epoch 13 | batch 2130/2304 [I 2022-10-26 14:43:02] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:01:25 [I 2022-10-26 14:43:02] ...trap/engines/engine.py.229: process: 0.48199 | load: 0.00242 [I 2022-10-26 14:43:02] ...trap/engines/engine.py.230: loss: 0.25837 [I 2022-10-26 14:43:02] ...optimizers/trijoint.py.151: total_norm: 4.952880 [I 2022-10-26 14:43:07] ...trap/engines/engine.py.224: train: epoch 13 | batch 2140/2304 [I 2022-10-26 14:43:07] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:01:21 [I 2022-10-26 14:43:07] ...trap/engines/engine.py.229: process: 0.50375 | load: 0.00243 [I 2022-10-26 14:43:07] ...trap/engines/engine.py.230: loss: 0.11348 [I 2022-10-26 14:43:07] ...optimizers/trijoint.py.151: total_norm: 2.215730 [I 2022-10-26 14:43:12] ...trap/engines/engine.py.224: train: epoch 13 | batch 2150/2304 [I 2022-10-26 14:43:12] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:01:16 [I 2022-10-26 14:43:12] ...trap/engines/engine.py.229: process: 0.49018 | load: 0.00246 [I 2022-10-26 14:43:12] ...trap/engines/engine.py.230: loss: 0.21549 [I 2022-10-26 14:43:12] ...optimizers/trijoint.py.151: total_norm: 3.021274 [I 2022-10-26 14:43:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 2160/2304 [I 2022-10-26 14:43:17] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:01:11 [I 2022-10-26 14:43:17] ...trap/engines/engine.py.229: process: 0.50616 | load: 0.00245 [I 2022-10-26 14:43:17] ...trap/engines/engine.py.230: loss: 0.27836 [I 2022-10-26 14:43:17] ...optimizers/trijoint.py.151: total_norm: 3.693221 [I 2022-10-26 14:43:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 2170/2304 [I 2022-10-26 14:43:22] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:01:07 [I 2022-10-26 14:43:22] ...trap/engines/engine.py.229: process: 0.50285 | load: 0.00241 [I 2022-10-26 14:43:22] ...trap/engines/engine.py.230: loss: 0.31131 [I 2022-10-26 14:43:22] ...optimizers/trijoint.py.151: total_norm: 3.982931 [I 2022-10-26 14:43:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 2180/2304 [I 2022-10-26 14:43:27] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:01:01 [I 2022-10-26 14:43:27] ...trap/engines/engine.py.229: process: 0.49921 | load: 0.00242 [I 2022-10-26 14:43:27] ...trap/engines/engine.py.230: loss: 0.20710 [I 2022-10-26 14:43:27] ...optimizers/trijoint.py.151: total_norm: 4.084973 [I 2022-10-26 14:43:32] ...trap/engines/engine.py.224: train: epoch 13 | batch 2190/2304 [I 2022-10-26 14:43:32] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:00:56 [I 2022-10-26 14:43:32] ...trap/engines/engine.py.229: process: 0.48809 | load: 0.00243 [I 2022-10-26 14:43:32] ...trap/engines/engine.py.230: loss: 0.23105 [I 2022-10-26 14:43:32] ...optimizers/trijoint.py.151: total_norm: 3.114405 [I 2022-10-26 14:43:37] ...trap/engines/engine.py.224: train: epoch 13 | batch 2200/2304 [I 2022-10-26 14:43:37] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:00:51 [I 2022-10-26 14:43:37] ...trap/engines/engine.py.229: process: 0.49910 | load: 0.00239 [I 2022-10-26 14:43:37] ...trap/engines/engine.py.230: loss: 0.19817 [I 2022-10-26 14:43:37] ...optimizers/trijoint.py.151: total_norm: 2.926256 [I 2022-10-26 14:43:42] ...trap/engines/engine.py.224: train: epoch 13 | batch 2210/2304 [I 2022-10-26 14:43:42] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:00:46 [I 2022-10-26 14:43:42] ...trap/engines/engine.py.229: process: 0.50471 | load: 0.00238 [I 2022-10-26 14:43:42] ...trap/engines/engine.py.230: loss: 0.19998 [I 2022-10-26 14:43:42] ...optimizers/trijoint.py.151: total_norm: 2.822186 [I 2022-10-26 14:43:47] ...trap/engines/engine.py.224: train: epoch 13 | batch 2220/2304 [I 2022-10-26 14:43:47] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:00:42 [I 2022-10-26 14:43:47] ...trap/engines/engine.py.229: process: 0.51028 | load: 0.00240 [I 2022-10-26 14:43:47] ...trap/engines/engine.py.230: loss: 0.31005 [I 2022-10-26 14:43:47] ...optimizers/trijoint.py.151: total_norm: 3.702047 [I 2022-10-26 14:43:52] ...trap/engines/engine.py.224: train: epoch 13 | batch 2230/2304 [I 2022-10-26 14:43:52] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:00:36 [I 2022-10-26 14:43:52] ...trap/engines/engine.py.229: process: 0.49266 | load: 0.00251 [I 2022-10-26 14:43:52] ...trap/engines/engine.py.230: loss: 0.19858 [I 2022-10-26 14:43:52] ...optimizers/trijoint.py.151: total_norm: 2.116224 [I 2022-10-26 14:43:57] ...trap/engines/engine.py.224: train: epoch 13 | batch 2240/2304 [I 2022-10-26 14:43:57] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:00:31 [I 2022-10-26 14:43:57] ...trap/engines/engine.py.229: process: 0.49095 | load: 0.00245 [I 2022-10-26 14:43:57] ...trap/engines/engine.py.230: loss: 0.27670 [I 2022-10-26 14:43:57] ...optimizers/trijoint.py.151: total_norm: 3.237166 [I 2022-10-26 14:44:02] ...trap/engines/engine.py.224: train: epoch 13 | batch 2250/2304 [I 2022-10-26 14:44:02] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:00:26 [I 2022-10-26 14:44:02] ...trap/engines/engine.py.229: process: 0.48985 | load: 0.00242 [I 2022-10-26 14:44:02] ...trap/engines/engine.py.230: loss: 0.21692 [I 2022-10-26 14:44:02] ...optimizers/trijoint.py.151: total_norm: 2.328166 [I 2022-10-26 14:44:07] ...trap/engines/engine.py.224: train: epoch 13 | batch 2260/2304 [I 2022-10-26 14:44:07] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:00:21 [I 2022-10-26 14:44:07] ...trap/engines/engine.py.229: process: 0.49937 | load: 0.00242 [I 2022-10-26 14:44:07] ...trap/engines/engine.py.230: loss: 0.29912 [I 2022-10-26 14:44:07] ...optimizers/trijoint.py.151: total_norm: 3.972186 [I 2022-10-26 14:44:12] ...trap/engines/engine.py.224: train: epoch 13 | batch 2270/2304 [I 2022-10-26 14:44:12] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:00:16 [I 2022-10-26 14:44:12] ...trap/engines/engine.py.229: process: 0.49981 | load: 0.00244 [I 2022-10-26 14:44:12] ...trap/engines/engine.py.230: loss: 0.23464 [I 2022-10-26 14:44:12] ...optimizers/trijoint.py.151: total_norm: 2.778378 [I 2022-10-26 14:44:17] ...trap/engines/engine.py.224: train: epoch 13 | batch 2280/2304 [I 2022-10-26 14:44:17] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:00:11 [I 2022-10-26 14:44:17] ...trap/engines/engine.py.229: process: 0.48443 | load: 0.00241 [I 2022-10-26 14:44:17] ...trap/engines/engine.py.230: loss: 0.20041 [I 2022-10-26 14:44:17] ...optimizers/trijoint.py.151: total_norm: 2.788451 [I 2022-10-26 14:44:22] ...trap/engines/engine.py.224: train: epoch 13 | batch 2290/2304 [I 2022-10-26 14:44:22] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:00:06 [I 2022-10-26 14:44:22] ...trap/engines/engine.py.229: process: 0.49317 | load: 0.00240 [I 2022-10-26 14:44:22] ...trap/engines/engine.py.230: loss: 0.20375 [I 2022-10-26 14:44:22] ...optimizers/trijoint.py.151: total_norm: 3.161375 [I 2022-10-26 14:44:27] ...trap/engines/engine.py.224: train: epoch 13 | batch 2300/2304 [I 2022-10-26 14:44:27] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:00:02 [I 2022-10-26 14:44:27] ...trap/engines/engine.py.229: process: 0.49380 | load: 0.00013 [I 2022-10-26 14:44:27] ...trap/engines/engine.py.230: loss: 0.19688 [I 2022-10-26 14:44:27] ...optimizers/trijoint.py.151: total_norm: 2.465843 [I 2022-10-26 14:44:29] ...trap/engines/engine.py.224: train: epoch 13 | batch 2304/2304 [I 2022-10-26 14:44:29] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:00:00 [I 2022-10-26 14:44:29] ...trap/engines/engine.py.229: process: 0.48786 | load: 0.00018 [I 2022-10-26 14:44:29] ...trap/engines/engine.py.230: loss: 0.28243 [I 2022-10-26 14:44:29] ...optimizers/trijoint.py.151: total_norm: 3.347976 [S 2022-10-26 14:44:30] ...trap/engines/engine.py.129: train_epoch.epoch: 13 [S 2022-10-26 14:44:30] ...trap/engines/engine.py.129: train_epoch.loss: 0.22175385447425078 [I 2022-10-26 14:44:30] ...s/criterions/triplet.py.84: margin = 0.12000000000000005 [I 2022-10-26 14:44:31] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 13 [I 2022-10-26 14:44:31] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 14:44:32] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 14:44:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 14:44:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 14:44:33] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 14:44:33] ...trap/engines/engine.py.321: eval: epoch 13 | batch 0/510 [I 2022-10-26 14:44:33] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:06 [I 2022-10-26 14:44:33] ...trap/engines/engine.py.326: process: 2.71870 | load: 2.33909 [I 2022-10-26 14:44:37] ...trap/engines/engine.py.321: eval: epoch 13 | batch 10/510 [I 2022-10-26 14:44:37] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:54 [I 2022-10-26 14:44:37] ...trap/engines/engine.py.326: process: 0.32828 | load: 0.00016 [I 2022-10-26 14:44:40] ...trap/engines/engine.py.321: eval: epoch 13 | batch 20/510 [I 2022-10-26 14:44:40] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:03:01 [I 2022-10-26 14:44:40] ...trap/engines/engine.py.326: process: 0.36303 | load: 0.00018 [I 2022-10-26 14:44:43] ...trap/engines/engine.py.321: eval: epoch 13 | batch 30/510 [I 2022-10-26 14:44:43] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:44 [I 2022-10-26 14:44:43] ...trap/engines/engine.py.326: process: 0.35435 | load: 0.00025 [I 2022-10-26 14:44:47] ...trap/engines/engine.py.321: eval: epoch 13 | batch 40/510 [I 2022-10-26 14:44:47] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:39 [I 2022-10-26 14:44:47] ...trap/engines/engine.py.326: process: 0.32387 | load: 0.00019 [I 2022-10-26 14:44:50] ...trap/engines/engine.py.321: eval: epoch 13 | batch 50/510 [I 2022-10-26 14:44:50] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:34 [I 2022-10-26 14:44:50] ...trap/engines/engine.py.326: process: 0.32582 | load: 0.00023 [I 2022-10-26 14:44:54] ...trap/engines/engine.py.321: eval: epoch 13 | batch 60/510 [I 2022-10-26 14:44:54] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:31 [I 2022-10-26 14:44:54] ...trap/engines/engine.py.326: process: 0.32516 | load: 0.00024 [I 2022-10-26 14:44:57] ...trap/engines/engine.py.321: eval: epoch 13 | batch 70/510 [I 2022-10-26 14:44:57] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:30 [I 2022-10-26 14:44:57] ...trap/engines/engine.py.326: process: 0.35451 | load: 0.00018 [I 2022-10-26 14:45:00] ...trap/engines/engine.py.321: eval: epoch 13 | batch 80/510 [I 2022-10-26 14:45:00] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:25 [I 2022-10-26 14:45:00] ...trap/engines/engine.py.326: process: 0.32302 | load: 0.00019 [I 2022-10-26 14:45:04] ...trap/engines/engine.py.321: eval: epoch 13 | batch 90/510 [I 2022-10-26 14:45:04] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-26 14:45:04] ...trap/engines/engine.py.326: process: 0.33816 | load: 0.00019 [I 2022-10-26 14:45:07] ...trap/engines/engine.py.321: eval: epoch 13 | batch 100/510 [I 2022-10-26 14:45:07] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 14:45:07] ...trap/engines/engine.py.326: process: 0.29289 | load: 0.00019 [I 2022-10-26 14:45:10] ...trap/engines/engine.py.321: eval: epoch 13 | batch 110/510 [I 2022-10-26 14:45:10] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 14:45:10] ...trap/engines/engine.py.326: process: 0.27993 | load: 0.00024 [I 2022-10-26 14:45:13] ...trap/engines/engine.py.321: eval: epoch 13 | batch 120/510 [I 2022-10-26 14:45:13] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 14:45:13] ...trap/engines/engine.py.326: process: 0.30600 | load: 0.00021 [I 2022-10-26 14:45:16] ...trap/engines/engine.py.321: eval: epoch 13 | batch 130/510 [I 2022-10-26 14:45:16] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:52 [I 2022-10-26 14:45:16] ...trap/engines/engine.py.326: process: 0.30263 | load: 0.00021 [I 2022-10-26 14:45:19] ...trap/engines/engine.py.321: eval: epoch 13 | batch 140/510 [I 2022-10-26 14:45:19] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:49 [I 2022-10-26 14:45:19] ...trap/engines/engine.py.326: process: 0.31079 | load: 0.00015 [I 2022-10-26 14:45:22] ...trap/engines/engine.py.321: eval: epoch 13 | batch 150/510 [I 2022-10-26 14:45:22] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:46 [I 2022-10-26 14:45:22] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00015 [I 2022-10-26 14:45:25] ...trap/engines/engine.py.321: eval: epoch 13 | batch 160/510 [I 2022-10-26 14:45:25] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-26 14:45:25] ...trap/engines/engine.py.326: process: 0.28602 | load: 0.00018 [I 2022-10-26 14:45:28] ...trap/engines/engine.py.321: eval: epoch 13 | batch 170/510 [I 2022-10-26 14:45:28] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:39 [I 2022-10-26 14:45:28] ...trap/engines/engine.py.326: process: 0.30149 | load: 0.00024 [I 2022-10-26 14:45:31] ...trap/engines/engine.py.321: eval: epoch 13 | batch 180/510 [I 2022-10-26 14:45:31] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-26 14:45:31] ...trap/engines/engine.py.326: process: 0.28300 | load: 0.00018 [I 2022-10-26 14:45:33] ...trap/engines/engine.py.321: eval: epoch 13 | batch 190/510 [I 2022-10-26 14:45:33] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 14:45:33] ...trap/engines/engine.py.326: process: 0.28319 | load: 0.00024 [I 2022-10-26 14:45:36] ...trap/engines/engine.py.321: eval: epoch 13 | batch 200/510 [I 2022-10-26 14:45:36] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 14:45:36] ...trap/engines/engine.py.326: process: 0.28175 | load: 0.00021 [I 2022-10-26 14:45:39] ...trap/engines/engine.py.321: eval: epoch 13 | batch 210/510 [I 2022-10-26 14:45:39] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-26 14:45:39] ...trap/engines/engine.py.326: process: 0.28687 | load: 0.00018 [I 2022-10-26 14:45:42] ...trap/engines/engine.py.321: eval: epoch 13 | batch 220/510 [I 2022-10-26 14:45:42] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 14:45:42] ...trap/engines/engine.py.326: process: 0.28938 | load: 0.00015 [I 2022-10-26 14:45:45] ...trap/engines/engine.py.321: eval: epoch 13 | batch 230/510 [I 2022-10-26 14:45:45] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 14:45:45] ...trap/engines/engine.py.326: process: 0.28894 | load: 0.00015 [I 2022-10-26 14:45:48] ...trap/engines/engine.py.321: eval: epoch 13 | batch 240/510 [I 2022-10-26 14:45:48] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 14:45:48] ...trap/engines/engine.py.326: process: 0.28478 | load: 0.00015 [I 2022-10-26 14:45:51] ...trap/engines/engine.py.321: eval: epoch 13 | batch 250/510 [I 2022-10-26 14:45:51] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-26 14:45:51] ...trap/engines/engine.py.326: process: 0.28896 | load: 0.00015 [I 2022-10-26 14:45:54] ...trap/engines/engine.py.321: eval: epoch 13 | batch 260/510 [I 2022-10-26 14:45:54] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-26 14:45:54] ...trap/engines/engine.py.326: process: 0.30712 | load: 0.00015 [I 2022-10-26 14:45:57] ...trap/engines/engine.py.321: eval: epoch 13 | batch 270/510 [I 2022-10-26 14:45:57] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 14:45:57] ...trap/engines/engine.py.326: process: 0.28887 | load: 0.00015 [I 2022-10-26 14:45:59] ...trap/engines/engine.py.321: eval: epoch 13 | batch 280/510 [I 2022-10-26 14:45:59] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:07 [I 2022-10-26 14:45:59] ...trap/engines/engine.py.326: process: 0.29027 | load: 0.00022 [I 2022-10-26 14:46:02] ...trap/engines/engine.py.321: eval: epoch 13 | batch 290/510 [I 2022-10-26 14:46:02] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:04 [I 2022-10-26 14:46:02] ...trap/engines/engine.py.326: process: 0.28365 | load: 0.00015 [I 2022-10-26 14:46:05] ...trap/engines/engine.py.321: eval: epoch 13 | batch 300/510 [I 2022-10-26 14:46:05] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:01 [I 2022-10-26 14:46:05] ...trap/engines/engine.py.326: process: 0.29068 | load: 0.00015 [I 2022-10-26 14:46:08] ...trap/engines/engine.py.321: eval: epoch 13 | batch 310/510 [I 2022-10-26 14:46:08] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 14:46:08] ...trap/engines/engine.py.326: process: 0.28430 | load: 0.00021 [I 2022-10-26 14:46:11] ...trap/engines/engine.py.321: eval: epoch 13 | batch 320/510 [I 2022-10-26 14:46:11] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 14:46:11] ...trap/engines/engine.py.326: process: 0.29060 | load: 0.00186 [I 2022-10-26 14:46:14] ...trap/engines/engine.py.321: eval: epoch 13 | batch 330/510 [I 2022-10-26 14:46:14] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:52 [I 2022-10-26 14:46:14] ...trap/engines/engine.py.326: process: 0.28678 | load: 0.00014 [I 2022-10-26 14:46:17] ...trap/engines/engine.py.321: eval: epoch 13 | batch 340/510 [I 2022-10-26 14:46:17] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 14:46:17] ...trap/engines/engine.py.326: process: 0.28293 | load: 0.00014 [I 2022-10-26 14:46:20] ...trap/engines/engine.py.321: eval: epoch 13 | batch 350/510 [I 2022-10-26 14:46:20] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 14:46:20] ...trap/engines/engine.py.326: process: 0.29026 | load: 0.00015 [I 2022-10-26 14:46:23] ...trap/engines/engine.py.321: eval: epoch 13 | batch 360/510 [I 2022-10-26 14:46:23] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:42 [I 2022-10-26 14:46:23] ...trap/engines/engine.py.326: process: 0.28585 | load: 0.00017 [I 2022-10-26 14:46:25] ...trap/engines/engine.py.321: eval: epoch 13 | batch 370/510 [I 2022-10-26 14:46:25] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:39 [I 2022-10-26 14:46:25] ...trap/engines/engine.py.326: process: 0.28373 | load: 0.00015 [I 2022-10-26 14:46:28] ...trap/engines/engine.py.321: eval: epoch 13 | batch 380/510 [I 2022-10-26 14:46:28] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 14:46:28] ...trap/engines/engine.py.326: process: 0.29023 | load: 0.00016 [I 2022-10-26 14:46:31] ...trap/engines/engine.py.321: eval: epoch 13 | batch 390/510 [I 2022-10-26 14:46:31] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 14:46:31] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00017 [I 2022-10-26 14:46:34] ...trap/engines/engine.py.321: eval: epoch 13 | batch 400/510 [I 2022-10-26 14:46:34] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 14:46:34] ...trap/engines/engine.py.326: process: 0.28064 | load: 0.00014 [I 2022-10-26 14:46:37] ...trap/engines/engine.py.321: eval: epoch 13 | batch 410/510 [I 2022-10-26 14:46:37] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:29 [I 2022-10-26 14:46:37] ...trap/engines/engine.py.326: process: 0.29160 | load: 0.00020 [I 2022-10-26 14:46:40] ...trap/engines/engine.py.321: eval: epoch 13 | batch 420/510 [I 2022-10-26 14:46:40] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 14:46:40] ...trap/engines/engine.py.326: process: 0.28853 | load: 0.00015 [I 2022-10-26 14:46:43] ...trap/engines/engine.py.321: eval: epoch 13 | batch 430/510 [I 2022-10-26 14:46:43] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:22 [I 2022-10-26 14:46:43] ...trap/engines/engine.py.326: process: 0.28316 | load: 0.00014 [I 2022-10-26 14:46:46] ...trap/engines/engine.py.321: eval: epoch 13 | batch 440/510 [I 2022-10-26 14:46:46] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 14:46:46] ...trap/engines/engine.py.326: process: 0.28245 | load: 0.00019 [I 2022-10-26 14:46:49] ...trap/engines/engine.py.321: eval: epoch 13 | batch 450/510 [I 2022-10-26 14:46:49] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 14:46:49] ...trap/engines/engine.py.326: process: 0.29340 | load: 0.00016 [I 2022-10-26 14:46:52] ...trap/engines/engine.py.321: eval: epoch 13 | batch 460/510 [I 2022-10-26 14:46:52] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 14:46:52] ...trap/engines/engine.py.326: process: 0.28714 | load: 0.00017 [I 2022-10-26 14:46:54] ...trap/engines/engine.py.321: eval: epoch 13 | batch 470/510 [I 2022-10-26 14:46:54] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 14:46:54] ...trap/engines/engine.py.326: process: 0.28602 | load: 0.00015 [I 2022-10-26 14:46:57] ...trap/engines/engine.py.321: eval: epoch 13 | batch 480/510 [I 2022-10-26 14:46:57] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 14:46:57] ...trap/engines/engine.py.326: process: 0.28681 | load: 0.00014 [I 2022-10-26 14:47:00] ...trap/engines/engine.py.321: eval: epoch 13 | batch 490/510 [I 2022-10-26 14:47:00] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 14:47:00] ...trap/engines/engine.py.326: process: 0.29264 | load: 0.00016 [I 2022-10-26 14:47:03] ...trap/engines/engine.py.321: eval: epoch 13 | batch 500/510 [I 2022-10-26 14:47:03] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 14:47:03] ...trap/engines/engine.py.326: process: 0.28733 | load: 0.00016 [I 2022-10-26 14:47:06] ...trap/engines/engine.py.321: eval: epoch 13 | batch 510/510 [I 2022-10-26 14:47:06] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 14:47:06] ...trap/engines/engine.py.326: process: 0.28655 | load: 0.00013 [S 2022-10-26 14:47:07] ...trap/engines/engine.py.132: eval_epoch.epoch: 13 [I 2022-10-26 14:47:07] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.542500 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.822000 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.891500 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.546400 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.824100 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.888900 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.007991 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009230 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009341 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012862 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009544 [S 2022-10-26 14:47:11] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005770 [I 2022-10-26 14:47:12] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 14:47:12] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 14:47:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 14:47:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 14:47:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 14:47:12] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 14:47:15] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 14:47:20] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 14:47:20] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 14:47:20] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 14:47:23] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 14:47:28] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 14:47:28] ...trap/engines/engine.py.163: Training model on trainset for epoch 14 [I 2022-10-26 14:47:28] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 14:47:31] ...trap/engines/engine.py.224: train: epoch 14 | batch 0/2304 [I 2022-10-26 14:47:31] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:45:43 [I 2022-10-26 14:47:31] ...trap/engines/engine.py.229: process: 2.75310 | load: 2.17209 [I 2022-10-26 14:47:31] ...trap/engines/engine.py.230: loss: 0.16766 [I 2022-10-26 14:47:31] ...optimizers/trijoint.py.151: total_norm: 2.871861 [I 2022-10-26 14:47:36] ...trap/engines/engine.py.224: train: epoch 14 | batch 10/2304 [I 2022-10-26 14:47:36] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:14 [I 2022-10-26 14:47:36] ...trap/engines/engine.py.229: process: 0.49472 | load: 0.00407 [I 2022-10-26 14:47:36] ...trap/engines/engine.py.230: loss: 0.19122 [I 2022-10-26 14:47:36] ...optimizers/trijoint.py.151: total_norm: 2.235246 [I 2022-10-26 14:47:41] ...trap/engines/engine.py.224: train: epoch 14 | batch 20/2304 [I 2022-10-26 14:47:41] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:01 [I 2022-10-26 14:47:41] ...trap/engines/engine.py.229: process: 0.49750 | load: 0.00368 [I 2022-10-26 14:47:41] ...trap/engines/engine.py.230: loss: 0.25038 [I 2022-10-26 14:47:41] ...optimizers/trijoint.py.151: total_norm: 4.070304 [I 2022-10-26 14:47:46] ...trap/engines/engine.py.224: train: epoch 14 | batch 30/2304 [I 2022-10-26 14:47:46] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:18:53 [I 2022-10-26 14:47:46] ...trap/engines/engine.py.229: process: 0.50213 | load: 0.00340 [I 2022-10-26 14:47:46] ...trap/engines/engine.py.230: loss: 0.21378 [I 2022-10-26 14:47:46] ...optimizers/trijoint.py.151: total_norm: 3.013882 [I 2022-10-26 14:47:51] ...trap/engines/engine.py.224: train: epoch 14 | batch 40/2304 [I 2022-10-26 14:47:51] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:47 [I 2022-10-26 14:47:51] ...trap/engines/engine.py.229: process: 0.48921 | load: 0.00302 [I 2022-10-26 14:47:51] ...trap/engines/engine.py.230: loss: 0.27006 [I 2022-10-26 14:47:51] ...optimizers/trijoint.py.151: total_norm: 4.555095 [I 2022-10-26 14:47:56] ...trap/engines/engine.py.224: train: epoch 14 | batch 50/2304 [I 2022-10-26 14:47:56] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:19:10 [I 2022-10-26 14:47:56] ...trap/engines/engine.py.229: process: 0.49944 | load: 0.00266 [I 2022-10-26 14:47:56] ...trap/engines/engine.py.230: loss: 0.25299 [I 2022-10-26 14:47:56] ...optimizers/trijoint.py.151: total_norm: 3.394214 [I 2022-10-26 14:48:01] ...trap/engines/engine.py.224: train: epoch 14 | batch 60/2304 [I 2022-10-26 14:48:01] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:42 [I 2022-10-26 14:48:01] ...trap/engines/engine.py.229: process: 0.49554 | load: 0.00264 [I 2022-10-26 14:48:01] ...trap/engines/engine.py.230: loss: 0.23125 [I 2022-10-26 14:48:01] ...optimizers/trijoint.py.151: total_norm: 3.390397 [I 2022-10-26 14:48:06] ...trap/engines/engine.py.224: train: epoch 14 | batch 70/2304 [I 2022-10-26 14:48:06] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:47 [I 2022-10-26 14:48:06] ...trap/engines/engine.py.229: process: 0.52407 | load: 0.00288 [I 2022-10-26 14:48:06] ...trap/engines/engine.py.230: loss: 0.23966 [I 2022-10-26 14:48:06] ...optimizers/trijoint.py.151: total_norm: 3.457283 [I 2022-10-26 14:48:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 80/2304 [I 2022-10-26 14:48:11] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:29 [I 2022-10-26 14:48:11] ...trap/engines/engine.py.229: process: 0.49769 | load: 0.00263 [I 2022-10-26 14:48:11] ...trap/engines/engine.py.230: loss: 0.29164 [I 2022-10-26 14:48:11] ...optimizers/trijoint.py.151: total_norm: 3.602344 [I 2022-10-26 14:48:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 90/2304 [I 2022-10-26 14:48:16] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:18:14 [I 2022-10-26 14:48:16] ...trap/engines/engine.py.229: process: 0.48900 | load: 0.00273 [I 2022-10-26 14:48:16] ...trap/engines/engine.py.230: loss: 0.17829 [I 2022-10-26 14:48:16] ...optimizers/trijoint.py.151: total_norm: 2.682042 [I 2022-10-26 14:48:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 100/2304 [I 2022-10-26 14:48:21] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:18:33 [I 2022-10-26 14:48:21] ...trap/engines/engine.py.229: process: 0.50267 | load: 0.00264 [I 2022-10-26 14:48:21] ...trap/engines/engine.py.230: loss: 0.21812 [I 2022-10-26 14:48:21] ...optimizers/trijoint.py.151: total_norm: 3.479550 [I 2022-10-26 14:48:26] ...trap/engines/engine.py.224: train: epoch 14 | batch 110/2304 [I 2022-10-26 14:48:26] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:18:29 [I 2022-10-26 14:48:26] ...trap/engines/engine.py.229: process: 0.50928 | load: 0.00282 [I 2022-10-26 14:48:26] ...trap/engines/engine.py.230: loss: 0.22783 [I 2022-10-26 14:48:26] ...optimizers/trijoint.py.151: total_norm: 3.672367 [I 2022-10-26 14:48:31] ...trap/engines/engine.py.224: train: epoch 14 | batch 120/2304 [I 2022-10-26 14:48:31] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:08 [I 2022-10-26 14:48:31] ...trap/engines/engine.py.229: process: 0.48552 | load: 0.00278 [I 2022-10-26 14:48:31] ...trap/engines/engine.py.230: loss: 0.19914 [I 2022-10-26 14:48:31] ...optimizers/trijoint.py.151: total_norm: 2.942050 [I 2022-10-26 14:48:36] ...trap/engines/engine.py.224: train: epoch 14 | batch 130/2304 [I 2022-10-26 14:48:36] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:09 [I 2022-10-26 14:48:36] ...trap/engines/engine.py.229: process: 0.50153 | load: 0.00276 [I 2022-10-26 14:48:36] ...trap/engines/engine.py.230: loss: 0.18505 [I 2022-10-26 14:48:36] ...optimizers/trijoint.py.151: total_norm: 2.831361 [I 2022-10-26 14:48:41] ...trap/engines/engine.py.224: train: epoch 14 | batch 140/2304 [I 2022-10-26 14:48:41] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:18:04 [I 2022-10-26 14:48:41] ...trap/engines/engine.py.229: process: 0.50259 | load: 0.00248 [I 2022-10-26 14:48:41] ...trap/engines/engine.py.230: loss: 0.17414 [I 2022-10-26 14:48:41] ...optimizers/trijoint.py.151: total_norm: 3.128350 [I 2022-10-26 14:48:46] ...trap/engines/engine.py.224: train: epoch 14 | batch 150/2304 [I 2022-10-26 14:48:46] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:18:00 [I 2022-10-26 14:48:46] ...trap/engines/engine.py.229: process: 0.49623 | load: 0.00260 [I 2022-10-26 14:48:46] ...trap/engines/engine.py.230: loss: 0.18987 [I 2022-10-26 14:48:46] ...optimizers/trijoint.py.151: total_norm: 4.015473 [I 2022-10-26 14:49:15] ...trap/engines/engine.py.224: train: epoch 14 | batch 160/2304 [I 2022-10-26 14:49:15] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 1:02:25 [I 2022-10-26 14:49:15] ...trap/engines/engine.py.229: process: 0.49835 | load: 0.00245 [I 2022-10-26 14:49:15] ...trap/engines/engine.py.230: loss: 0.23147 [I 2022-10-26 14:49:15] ...optimizers/trijoint.py.151: total_norm: 3.648646 [I 2022-10-26 14:49:20] ...trap/engines/engine.py.224: train: epoch 14 | batch 170/2304 [I 2022-10-26 14:49:20] ...trap/engines/engine.py.228: elapsed: 0:01:51 | left: 0:22:37 [I 2022-10-26 14:49:20] ...trap/engines/engine.py.229: process: 0.49701 | load: 0.00240 [I 2022-10-26 14:49:20] ...trap/engines/engine.py.230: loss: 0.25753 [I 2022-10-26 14:49:20] ...optimizers/trijoint.py.151: total_norm: 4.246292 [I 2022-10-26 14:49:25] ...trap/engines/engine.py.224: train: epoch 14 | batch 180/2304 [I 2022-10-26 14:49:25] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:18:19 [I 2022-10-26 14:49:25] ...trap/engines/engine.py.229: process: 0.49855 | load: 0.00239 [I 2022-10-26 14:49:25] ...trap/engines/engine.py.230: loss: 0.17517 [I 2022-10-26 14:49:25] ...optimizers/trijoint.py.151: total_norm: 3.030758 [I 2022-10-26 14:49:30] ...trap/engines/engine.py.224: train: epoch 14 | batch 190/2304 [I 2022-10-26 14:49:30] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:17:42 [I 2022-10-26 14:49:30] ...trap/engines/engine.py.229: process: 0.50269 | load: 0.00256 [I 2022-10-26 14:49:30] ...trap/engines/engine.py.230: loss: 0.20372 [I 2022-10-26 14:49:30] ...optimizers/trijoint.py.151: total_norm: 3.131863 [I 2022-10-26 14:49:35] ...trap/engines/engine.py.224: train: epoch 14 | batch 200/2304 [I 2022-10-26 14:49:35] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:17:33 [I 2022-10-26 14:49:35] ...trap/engines/engine.py.229: process: 0.49857 | load: 0.00271 [I 2022-10-26 14:49:35] ...trap/engines/engine.py.230: loss: 0.24252 [I 2022-10-26 14:49:35] ...optimizers/trijoint.py.151: total_norm: 3.197381 [I 2022-10-26 14:49:40] ...trap/engines/engine.py.224: train: epoch 14 | batch 210/2304 [I 2022-10-26 14:49:40] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:17:33 [I 2022-10-26 14:49:40] ...trap/engines/engine.py.229: process: 0.49907 | load: 0.00249 [I 2022-10-26 14:49:40] ...trap/engines/engine.py.230: loss: 0.15419 [I 2022-10-26 14:49:40] ...optimizers/trijoint.py.151: total_norm: 3.176022 [I 2022-10-26 14:49:45] ...trap/engines/engine.py.224: train: epoch 14 | batch 220/2304 [I 2022-10-26 14:49:45] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:17:27 [I 2022-10-26 14:49:45] ...trap/engines/engine.py.229: process: 0.50440 | load: 0.00250 [I 2022-10-26 14:49:45] ...trap/engines/engine.py.230: loss: 0.22192 [I 2022-10-26 14:49:45] ...optimizers/trijoint.py.151: total_norm: 2.784461 [I 2022-10-26 14:49:50] ...trap/engines/engine.py.224: train: epoch 14 | batch 230/2304 [I 2022-10-26 14:49:50] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:17:18 [I 2022-10-26 14:49:50] ...trap/engines/engine.py.229: process: 0.49351 | load: 0.00247 [I 2022-10-26 14:49:50] ...trap/engines/engine.py.230: loss: 0.20594 [I 2022-10-26 14:49:50] ...optimizers/trijoint.py.151: total_norm: 3.407891 [I 2022-10-26 14:49:55] ...trap/engines/engine.py.224: train: epoch 14 | batch 240/2304 [I 2022-10-26 14:49:55] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:17:08 [I 2022-10-26 14:49:55] ...trap/engines/engine.py.229: process: 0.49714 | load: 0.00251 [I 2022-10-26 14:49:55] ...trap/engines/engine.py.230: loss: 0.29774 [I 2022-10-26 14:49:55] ...optimizers/trijoint.py.151: total_norm: 3.489856 [I 2022-10-26 14:50:00] ...trap/engines/engine.py.224: train: epoch 14 | batch 250/2304 [I 2022-10-26 14:50:00] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:17:06 [I 2022-10-26 14:50:00] ...trap/engines/engine.py.229: process: 0.49370 | load: 0.00247 [I 2022-10-26 14:50:00] ...trap/engines/engine.py.230: loss: 0.18771 [I 2022-10-26 14:50:00] ...optimizers/trijoint.py.151: total_norm: 3.095227 [I 2022-10-26 14:50:05] ...trap/engines/engine.py.224: train: epoch 14 | batch 260/2304 [I 2022-10-26 14:50:05] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:17:19 [I 2022-10-26 14:50:05] ...trap/engines/engine.py.229: process: 0.53091 | load: 0.00283 [I 2022-10-26 14:50:05] ...trap/engines/engine.py.230: loss: 0.18284 [I 2022-10-26 14:50:05] ...optimizers/trijoint.py.151: total_norm: 2.678795 [I 2022-10-26 14:50:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 270/2304 [I 2022-10-26 14:50:11] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:17:08 [I 2022-10-26 14:50:11] ...trap/engines/engine.py.229: process: 0.50314 | load: 0.00290 [I 2022-10-26 14:50:11] ...trap/engines/engine.py.230: loss: 0.25031 [I 2022-10-26 14:50:11] ...optimizers/trijoint.py.151: total_norm: 3.480167 [I 2022-10-26 14:50:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 280/2304 [I 2022-10-26 14:50:16] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:16:56 [I 2022-10-26 14:50:16] ...trap/engines/engine.py.229: process: 0.49884 | load: 0.00277 [I 2022-10-26 14:50:16] ...trap/engines/engine.py.230: loss: 0.18381 [I 2022-10-26 14:50:16] ...optimizers/trijoint.py.151: total_norm: 4.203258 [I 2022-10-26 14:50:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 290/2304 [I 2022-10-26 14:50:21] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:16:51 [I 2022-10-26 14:50:21] ...trap/engines/engine.py.229: process: 0.50019 | load: 0.00247 [I 2022-10-26 14:50:21] ...trap/engines/engine.py.230: loss: 0.25575 [I 2022-10-26 14:50:21] ...optimizers/trijoint.py.151: total_norm: 3.246563 [I 2022-10-26 14:50:26] ...trap/engines/engine.py.224: train: epoch 14 | batch 300/2304 [I 2022-10-26 14:50:26] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:16:34 [I 2022-10-26 14:50:26] ...trap/engines/engine.py.229: process: 0.48214 | load: 0.00243 [I 2022-10-26 14:50:26] ...trap/engines/engine.py.230: loss: 0.12952 [I 2022-10-26 14:50:26] ...optimizers/trijoint.py.151: total_norm: 2.659676 [I 2022-10-26 14:50:31] ...trap/engines/engine.py.224: train: epoch 14 | batch 310/2304 [I 2022-10-26 14:50:31] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:16:26 [I 2022-10-26 14:50:31] ...trap/engines/engine.py.229: process: 0.48488 | load: 0.00246 [I 2022-10-26 14:50:31] ...trap/engines/engine.py.230: loss: 0.31248 [I 2022-10-26 14:50:31] ...optimizers/trijoint.py.151: total_norm: 4.094754 [I 2022-10-26 14:50:36] ...trap/engines/engine.py.224: train: epoch 14 | batch 320/2304 [I 2022-10-26 14:50:36] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:16:27 [I 2022-10-26 14:50:36] ...trap/engines/engine.py.229: process: 0.49690 | load: 0.00253 [I 2022-10-26 14:50:36] ...trap/engines/engine.py.230: loss: 0.24070 [I 2022-10-26 14:50:36] ...optimizers/trijoint.py.151: total_norm: 3.812959 [I 2022-10-26 14:50:41] ...trap/engines/engine.py.224: train: epoch 14 | batch 330/2304 [I 2022-10-26 14:50:41] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:16:21 [I 2022-10-26 14:50:41] ...trap/engines/engine.py.229: process: 0.48750 | load: 0.00240 [I 2022-10-26 14:50:41] ...trap/engines/engine.py.230: loss: 0.18356 [I 2022-10-26 14:50:41] ...optimizers/trijoint.py.151: total_norm: 2.924041 [I 2022-10-26 14:50:46] ...trap/engines/engine.py.224: train: epoch 14 | batch 340/2304 [I 2022-10-26 14:50:46] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:16:13 [I 2022-10-26 14:50:46] ...trap/engines/engine.py.229: process: 0.50027 | load: 0.00241 [I 2022-10-26 14:50:46] ...trap/engines/engine.py.230: loss: 0.16980 [I 2022-10-26 14:50:46] ...optimizers/trijoint.py.151: total_norm: 2.922646 [I 2022-10-26 14:50:51] ...trap/engines/engine.py.224: train: epoch 14 | batch 350/2304 [I 2022-10-26 14:50:51] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:16:10 [I 2022-10-26 14:50:51] ...trap/engines/engine.py.229: process: 0.48628 | load: 0.00264 [I 2022-10-26 14:50:51] ...trap/engines/engine.py.230: loss: 0.35657 [I 2022-10-26 14:50:51] ...optimizers/trijoint.py.151: total_norm: 4.356773 [I 2022-10-26 14:50:56] ...trap/engines/engine.py.224: train: epoch 14 | batch 360/2304 [I 2022-10-26 14:50:56] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:16:13 [I 2022-10-26 14:50:56] ...trap/engines/engine.py.229: process: 0.49226 | load: 0.00258 [I 2022-10-26 14:50:56] ...trap/engines/engine.py.230: loss: 0.19049 [I 2022-10-26 14:50:56] ...optimizers/trijoint.py.151: total_norm: 3.848014 [I 2022-10-26 14:51:01] ...trap/engines/engine.py.224: train: epoch 14 | batch 370/2304 [I 2022-10-26 14:51:01] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:16:14 [I 2022-10-26 14:51:01] ...trap/engines/engine.py.229: process: 0.50140 | load: 0.00248 [I 2022-10-26 14:51:01] ...trap/engines/engine.py.230: loss: 0.21218 [I 2022-10-26 14:51:01] ...optimizers/trijoint.py.151: total_norm: 2.794772 [I 2022-10-26 14:51:06] ...trap/engines/engine.py.224: train: epoch 14 | batch 380/2304 [I 2022-10-26 14:51:06] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:16:02 [I 2022-10-26 14:51:06] ...trap/engines/engine.py.229: process: 0.49759 | load: 0.00244 [I 2022-10-26 14:51:06] ...trap/engines/engine.py.230: loss: 0.21497 [I 2022-10-26 14:51:06] ...optimizers/trijoint.py.151: total_norm: 2.908208 [I 2022-10-26 14:51:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 390/2304 [I 2022-10-26 14:51:11] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:15:53 [I 2022-10-26 14:51:11] ...trap/engines/engine.py.229: process: 0.49928 | load: 0.00293 [I 2022-10-26 14:51:11] ...trap/engines/engine.py.230: loss: 0.26511 [I 2022-10-26 14:51:11] ...optimizers/trijoint.py.151: total_norm: 3.067518 [I 2022-10-26 14:51:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 400/2304 [I 2022-10-26 14:51:16] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:15:50 [I 2022-10-26 14:51:16] ...trap/engines/engine.py.229: process: 0.48885 | load: 0.00251 [I 2022-10-26 14:51:16] ...trap/engines/engine.py.230: loss: 0.30584 [I 2022-10-26 14:51:16] ...optimizers/trijoint.py.151: total_norm: 4.516013 [I 2022-10-26 14:51:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 410/2304 [I 2022-10-26 14:51:21] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:15:45 [I 2022-10-26 14:51:21] ...trap/engines/engine.py.229: process: 0.49372 | load: 0.00251 [I 2022-10-26 14:51:21] ...trap/engines/engine.py.230: loss: 0.18714 [I 2022-10-26 14:51:21] ...optimizers/trijoint.py.151: total_norm: 2.409528 [I 2022-10-26 14:51:26] ...trap/engines/engine.py.224: train: epoch 14 | batch 420/2304 [I 2022-10-26 14:51:26] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:15:46 [I 2022-10-26 14:51:26] ...trap/engines/engine.py.229: process: 0.50293 | load: 0.00243 [I 2022-10-26 14:51:26] ...trap/engines/engine.py.230: loss: 0.19457 [I 2022-10-26 14:51:26] ...optimizers/trijoint.py.151: total_norm: 2.698158 [I 2022-10-26 14:51:31] ...trap/engines/engine.py.224: train: epoch 14 | batch 430/2304 [I 2022-10-26 14:51:31] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:15:46 [I 2022-10-26 14:51:31] ...trap/engines/engine.py.229: process: 0.50703 | load: 0.00252 [I 2022-10-26 14:51:31] ...trap/engines/engine.py.230: loss: 0.16912 [I 2022-10-26 14:51:31] ...optimizers/trijoint.py.151: total_norm: 2.813582 [I 2022-10-26 14:51:36] ...trap/engines/engine.py.224: train: epoch 14 | batch 440/2304 [I 2022-10-26 14:51:36] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:15:30 [I 2022-10-26 14:51:36] ...trap/engines/engine.py.229: process: 0.49159 | load: 0.00256 [I 2022-10-26 14:51:36] ...trap/engines/engine.py.230: loss: 0.21867 [I 2022-10-26 14:51:36] ...optimizers/trijoint.py.151: total_norm: 3.231745 [I 2022-10-26 14:51:41] ...trap/engines/engine.py.224: train: epoch 14 | batch 450/2304 [I 2022-10-26 14:51:41] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:15:28 [I 2022-10-26 14:51:41] ...trap/engines/engine.py.229: process: 0.50019 | load: 0.00252 [I 2022-10-26 14:51:41] ...trap/engines/engine.py.230: loss: 0.15876 [I 2022-10-26 14:51:41] ...optimizers/trijoint.py.151: total_norm: 2.746351 [I 2022-10-26 14:51:46] ...trap/engines/engine.py.224: train: epoch 14 | batch 460/2304 [I 2022-10-26 14:51:46] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:15:21 [I 2022-10-26 14:51:46] ...trap/engines/engine.py.229: process: 0.50562 | load: 0.00297 [I 2022-10-26 14:51:46] ...trap/engines/engine.py.230: loss: 0.24921 [I 2022-10-26 14:51:46] ...optimizers/trijoint.py.151: total_norm: 2.974618 [I 2022-10-26 14:51:51] ...trap/engines/engine.py.224: train: epoch 14 | batch 470/2304 [I 2022-10-26 14:51:51] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:15:09 [I 2022-10-26 14:51:51] ...trap/engines/engine.py.229: process: 0.49257 | load: 0.00270 [I 2022-10-26 14:51:51] ...trap/engines/engine.py.230: loss: 0.28709 [I 2022-10-26 14:51:51] ...optimizers/trijoint.py.151: total_norm: 3.208346 [I 2022-10-26 14:51:56] ...trap/engines/engine.py.224: train: epoch 14 | batch 480/2304 [I 2022-10-26 14:51:56] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:15:12 [I 2022-10-26 14:51:56] ...trap/engines/engine.py.229: process: 0.50085 | load: 0.00262 [I 2022-10-26 14:51:56] ...trap/engines/engine.py.230: loss: 0.14251 [I 2022-10-26 14:51:56] ...optimizers/trijoint.py.151: total_norm: 2.866512 [I 2022-10-26 14:52:01] ...trap/engines/engine.py.224: train: epoch 14 | batch 490/2304 [I 2022-10-26 14:52:01] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:15:12 [I 2022-10-26 14:52:01] ...trap/engines/engine.py.229: process: 0.49705 | load: 0.00258 [I 2022-10-26 14:52:01] ...trap/engines/engine.py.230: loss: 0.19728 [I 2022-10-26 14:52:01] ...optimizers/trijoint.py.151: total_norm: 3.113594 [I 2022-10-26 14:52:06] ...trap/engines/engine.py.224: train: epoch 14 | batch 500/2304 [I 2022-10-26 14:52:06] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:15:02 [I 2022-10-26 14:52:06] ...trap/engines/engine.py.229: process: 0.48396 | load: 0.00256 [I 2022-10-26 14:52:06] ...trap/engines/engine.py.230: loss: 0.13866 [I 2022-10-26 14:52:06] ...optimizers/trijoint.py.151: total_norm: 2.633374 [I 2022-10-26 14:52:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 510/2304 [I 2022-10-26 14:52:11] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:14:54 [I 2022-10-26 14:52:11] ...trap/engines/engine.py.229: process: 0.48565 | load: 0.00253 [I 2022-10-26 14:52:11] ...trap/engines/engine.py.230: loss: 0.17334 [I 2022-10-26 14:52:11] ...optimizers/trijoint.py.151: total_norm: 2.533962 [I 2022-10-26 14:52:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 520/2304 [I 2022-10-26 14:52:16] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:14:59 [I 2022-10-26 14:52:16] ...trap/engines/engine.py.229: process: 0.49775 | load: 0.00255 [I 2022-10-26 14:52:16] ...trap/engines/engine.py.230: loss: 0.19591 [I 2022-10-26 14:52:16] ...optimizers/trijoint.py.151: total_norm: 2.840770 [I 2022-10-26 14:52:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 530/2304 [I 2022-10-26 14:52:21] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:14:44 [I 2022-10-26 14:52:21] ...trap/engines/engine.py.229: process: 0.49494 | load: 0.00252 [I 2022-10-26 14:52:21] ...trap/engines/engine.py.230: loss: 0.26486 [I 2022-10-26 14:52:21] ...optimizers/trijoint.py.151: total_norm: 3.456153 [I 2022-10-26 14:52:26] ...trap/engines/engine.py.224: train: epoch 14 | batch 540/2304 [I 2022-10-26 14:52:26] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:14:40 [I 2022-10-26 14:52:26] ...trap/engines/engine.py.229: process: 0.50488 | load: 0.00248 [I 2022-10-26 14:52:26] ...trap/engines/engine.py.230: loss: 0.24706 [I 2022-10-26 14:52:26] ...optimizers/trijoint.py.151: total_norm: 3.152350 [I 2022-10-26 14:52:31] ...trap/engines/engine.py.224: train: epoch 14 | batch 550/2304 [I 2022-10-26 14:52:31] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:14:38 [I 2022-10-26 14:52:31] ...trap/engines/engine.py.229: process: 0.49702 | load: 0.00257 [I 2022-10-26 14:52:31] ...trap/engines/engine.py.230: loss: 0.31351 [I 2022-10-26 14:52:31] ...optimizers/trijoint.py.151: total_norm: 4.749721 [I 2022-10-26 14:52:36] ...trap/engines/engine.py.224: train: epoch 14 | batch 560/2304 [I 2022-10-26 14:52:36] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:14:24 [I 2022-10-26 14:52:36] ...trap/engines/engine.py.229: process: 0.48678 | load: 0.00253 [I 2022-10-26 14:52:36] ...trap/engines/engine.py.230: loss: 0.33438 [I 2022-10-26 14:52:36] ...optimizers/trijoint.py.151: total_norm: 4.499729 [I 2022-10-26 14:52:41] ...trap/engines/engine.py.224: train: epoch 14 | batch 570/2304 [I 2022-10-26 14:52:41] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:14:28 [I 2022-10-26 14:52:41] ...trap/engines/engine.py.229: process: 0.50013 | load: 0.00247 [I 2022-10-26 14:52:41] ...trap/engines/engine.py.230: loss: 0.20398 [I 2022-10-26 14:52:41] ...optimizers/trijoint.py.151: total_norm: 2.838152 [I 2022-10-26 14:52:46] ...trap/engines/engine.py.224: train: epoch 14 | batch 580/2304 [I 2022-10-26 14:52:46] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:14:24 [I 2022-10-26 14:52:46] ...trap/engines/engine.py.229: process: 0.50153 | load: 0.00245 [I 2022-10-26 14:52:46] ...trap/engines/engine.py.230: loss: 0.14884 [I 2022-10-26 14:52:46] ...optimizers/trijoint.py.151: total_norm: 2.994555 [I 2022-10-26 14:52:51] ...trap/engines/engine.py.224: train: epoch 14 | batch 590/2304 [I 2022-10-26 14:52:51] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:14:25 [I 2022-10-26 14:52:51] ...trap/engines/engine.py.229: process: 0.48970 | load: 0.00268 [I 2022-10-26 14:52:51] ...trap/engines/engine.py.230: loss: 0.26553 [I 2022-10-26 14:52:51] ...optimizers/trijoint.py.151: total_norm: 4.461859 [I 2022-10-26 14:52:56] ...trap/engines/engine.py.224: train: epoch 14 | batch 600/2304 [I 2022-10-26 14:52:56] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:14:18 [I 2022-10-26 14:52:56] ...trap/engines/engine.py.229: process: 0.49303 | load: 0.00250 [I 2022-10-26 14:52:56] ...trap/engines/engine.py.230: loss: 0.19881 [I 2022-10-26 14:52:56] ...optimizers/trijoint.py.151: total_norm: 2.990840 [I 2022-10-26 14:53:01] ...trap/engines/engine.py.224: train: epoch 14 | batch 610/2304 [I 2022-10-26 14:53:01] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:14:10 [I 2022-10-26 14:53:01] ...trap/engines/engine.py.229: process: 0.48964 | load: 0.00246 [I 2022-10-26 14:53:01] ...trap/engines/engine.py.230: loss: 0.22686 [I 2022-10-26 14:53:01] ...optimizers/trijoint.py.151: total_norm: 3.142774 [I 2022-10-26 14:53:06] ...trap/engines/engine.py.224: train: epoch 14 | batch 620/2304 [I 2022-10-26 14:53:06] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:14:09 [I 2022-10-26 14:53:06] ...trap/engines/engine.py.229: process: 0.50660 | load: 0.00247 [I 2022-10-26 14:53:06] ...trap/engines/engine.py.230: loss: 0.18744 [I 2022-10-26 14:53:06] ...optimizers/trijoint.py.151: total_norm: 2.546952 [I 2022-10-26 14:53:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 630/2304 [I 2022-10-26 14:53:11] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:13:54 [I 2022-10-26 14:53:11] ...trap/engines/engine.py.229: process: 0.49161 | load: 0.00242 [I 2022-10-26 14:53:11] ...trap/engines/engine.py.230: loss: 0.21147 [I 2022-10-26 14:53:11] ...optimizers/trijoint.py.151: total_norm: 2.721425 [I 2022-10-26 14:53:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 640/2304 [I 2022-10-26 14:53:16] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:14:00 [I 2022-10-26 14:53:16] ...trap/engines/engine.py.229: process: 0.50422 | load: 0.00245 [I 2022-10-26 14:53:16] ...trap/engines/engine.py.230: loss: 0.24675 [I 2022-10-26 14:53:16] ...optimizers/trijoint.py.151: total_norm: 3.942912 [I 2022-10-26 14:53:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 650/2304 [I 2022-10-26 14:53:21] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:13:50 [I 2022-10-26 14:53:21] ...trap/engines/engine.py.229: process: 0.50486 | load: 0.00258 [I 2022-10-26 14:53:21] ...trap/engines/engine.py.230: loss: 0.26101 [I 2022-10-26 14:53:21] ...optimizers/trijoint.py.151: total_norm: 4.057630 [I 2022-10-26 14:53:26] ...trap/engines/engine.py.224: train: epoch 14 | batch 660/2304 [I 2022-10-26 14:53:26] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:13:40 [I 2022-10-26 14:53:26] ...trap/engines/engine.py.229: process: 0.49487 | load: 0.00252 [I 2022-10-26 14:53:26] ...trap/engines/engine.py.230: loss: 0.16886 [I 2022-10-26 14:53:26] ...optimizers/trijoint.py.151: total_norm: 2.781680 [I 2022-10-26 14:53:31] ...trap/engines/engine.py.224: train: epoch 14 | batch 670/2304 [I 2022-10-26 14:53:31] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:13:39 [I 2022-10-26 14:53:31] ...trap/engines/engine.py.229: process: 0.49266 | load: 0.00244 [I 2022-10-26 14:53:31] ...trap/engines/engine.py.230: loss: 0.23648 [I 2022-10-26 14:53:31] ...optimizers/trijoint.py.151: total_norm: 3.453919 [I 2022-10-26 14:53:36] ...trap/engines/engine.py.224: train: epoch 14 | batch 680/2304 [I 2022-10-26 14:53:36] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:13:30 [I 2022-10-26 14:53:36] ...trap/engines/engine.py.229: process: 0.50398 | load: 0.00246 [I 2022-10-26 14:53:36] ...trap/engines/engine.py.230: loss: 0.19835 [I 2022-10-26 14:53:36] ...optimizers/trijoint.py.151: total_norm: 3.806416 [I 2022-10-26 14:53:41] ...trap/engines/engine.py.224: train: epoch 14 | batch 690/2304 [I 2022-10-26 14:53:41] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:13:23 [I 2022-10-26 14:53:41] ...trap/engines/engine.py.229: process: 0.49413 | load: 0.00247 [I 2022-10-26 14:53:41] ...trap/engines/engine.py.230: loss: 0.20059 [I 2022-10-26 14:53:41] ...optimizers/trijoint.py.151: total_norm: 3.195839 [I 2022-10-26 14:53:46] ...trap/engines/engine.py.224: train: epoch 14 | batch 700/2304 [I 2022-10-26 14:53:46] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:13:23 [I 2022-10-26 14:53:46] ...trap/engines/engine.py.229: process: 0.49466 | load: 0.00243 [I 2022-10-26 14:53:46] ...trap/engines/engine.py.230: loss: 0.24401 [I 2022-10-26 14:53:46] ...optimizers/trijoint.py.151: total_norm: 3.622427 [I 2022-10-26 14:53:51] ...trap/engines/engine.py.224: train: epoch 14 | batch 710/2304 [I 2022-10-26 14:53:51] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:13:18 [I 2022-10-26 14:53:51] ...trap/engines/engine.py.229: process: 0.50411 | load: 0.00251 [I 2022-10-26 14:53:51] ...trap/engines/engine.py.230: loss: 0.15411 [I 2022-10-26 14:53:51] ...optimizers/trijoint.py.151: total_norm: 2.113765 [I 2022-10-26 14:53:56] ...trap/engines/engine.py.224: train: epoch 14 | batch 720/2304 [I 2022-10-26 14:53:56] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:13:08 [I 2022-10-26 14:53:56] ...trap/engines/engine.py.229: process: 0.49702 | load: 0.00248 [I 2022-10-26 14:53:56] ...trap/engines/engine.py.230: loss: 0.20368 [I 2022-10-26 14:53:56] ...optimizers/trijoint.py.151: total_norm: 2.507628 [I 2022-10-26 14:54:01] ...trap/engines/engine.py.224: train: epoch 14 | batch 730/2304 [I 2022-10-26 14:54:01] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:13:00 [I 2022-10-26 14:54:01] ...trap/engines/engine.py.229: process: 0.49383 | load: 0.00250 [I 2022-10-26 14:54:01] ...trap/engines/engine.py.230: loss: 0.27291 [I 2022-10-26 14:54:01] ...optimizers/trijoint.py.151: total_norm: 4.328883 [I 2022-10-26 14:54:06] ...trap/engines/engine.py.224: train: epoch 14 | batch 740/2304 [I 2022-10-26 14:54:06] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:12:59 [I 2022-10-26 14:54:06] ...trap/engines/engine.py.229: process: 0.49814 | load: 0.00252 [I 2022-10-26 14:54:06] ...trap/engines/engine.py.230: loss: 0.22053 [I 2022-10-26 14:54:06] ...optimizers/trijoint.py.151: total_norm: 4.222203 [I 2022-10-26 14:54:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 750/2304 [I 2022-10-26 14:54:11] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:12:57 [I 2022-10-26 14:54:11] ...trap/engines/engine.py.229: process: 0.50419 | load: 0.00246 [I 2022-10-26 14:54:11] ...trap/engines/engine.py.230: loss: 0.19563 [I 2022-10-26 14:54:11] ...optimizers/trijoint.py.151: total_norm: 2.877552 [I 2022-10-26 14:54:16] ...trap/engines/engine.py.224: train: epoch 14 | batch 760/2304 [I 2022-10-26 14:54:16] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:12:47 [I 2022-10-26 14:54:16] ...trap/engines/engine.py.229: process: 0.49211 | load: 0.00243 [I 2022-10-26 14:54:16] ...trap/engines/engine.py.230: loss: 0.21992 [I 2022-10-26 14:54:16] ...optimizers/trijoint.py.151: total_norm: 3.586646 [I 2022-10-26 14:54:21] ...trap/engines/engine.py.224: train: epoch 14 | batch 770/2304 [I 2022-10-26 14:54:21] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:12:48 [I 2022-10-26 14:54:21] ...trap/engines/engine.py.229: process: 0.50723 | load: 0.00243 [I 2022-10-26 14:54:21] ...trap/engines/engine.py.230: loss: 0.25012 [I 2022-10-26 14:54:21] ...optimizers/trijoint.py.151: total_norm: 3.261099 [I 2022-10-26 14:54:27] ...trap/engines/engine.py.224: train: epoch 14 | batch 780/2304 [I 2022-10-26 14:54:27] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:12:49 [I 2022-10-26 14:54:27] ...trap/engines/engine.py.229: process: 0.50090 | load: 0.00240 [I 2022-10-26 14:54:27] ...trap/engines/engine.py.230: loss: 0.14242 [I 2022-10-26 14:54:27] ...optimizers/trijoint.py.151: total_norm: 3.073363 [I 2022-10-26 14:54:32] ...trap/engines/engine.py.224: train: epoch 14 | batch 790/2304 [I 2022-10-26 14:54:32] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:12:40 [I 2022-10-26 14:54:32] ...trap/engines/engine.py.229: process: 0.50867 | load: 0.00241 [I 2022-10-26 14:54:32] ...trap/engines/engine.py.230: loss: 0.16688 [I 2022-10-26 14:54:32] ...optimizers/trijoint.py.151: total_norm: 2.378884 [I 2022-10-26 14:54:37] ...trap/engines/engine.py.224: train: epoch 14 | batch 800/2304 [I 2022-10-26 14:54:37] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:12:29 [I 2022-10-26 14:54:37] ...trap/engines/engine.py.229: process: 0.49584 | load: 0.00257 [I 2022-10-26 14:54:37] ...trap/engines/engine.py.230: loss: 0.26453 [I 2022-10-26 14:54:37] ...optimizers/trijoint.py.151: total_norm: 2.888811 [I 2022-10-26 14:54:41] ...trap/engines/engine.py.224: train: epoch 14 | batch 810/2304 [I 2022-10-26 14:54:41] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:12:22 [I 2022-10-26 14:54:41] ...trap/engines/engine.py.229: process: 0.50583 | load: 0.00255 [I 2022-10-26 14:54:41] ...trap/engines/engine.py.230: loss: 0.18939 [I 2022-10-26 14:54:41] ...optimizers/trijoint.py.151: total_norm: 2.807237 [I 2022-10-26 14:54:46] ...trap/engines/engine.py.224: train: epoch 14 | batch 820/2304 [I 2022-10-26 14:54:46] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:12:17 [I 2022-10-26 14:54:46] ...trap/engines/engine.py.229: process: 0.48559 | load: 0.00251 [I 2022-10-26 14:54:46] ...trap/engines/engine.py.230: loss: 0.23081 [I 2022-10-26 14:54:46] ...optimizers/trijoint.py.151: total_norm: 2.906443 [I 2022-10-26 14:54:51] ...trap/engines/engine.py.224: train: epoch 14 | batch 830/2304 [I 2022-10-26 14:54:51] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:12:16 [I 2022-10-26 14:54:51] ...trap/engines/engine.py.229: process: 0.50361 | load: 0.00247 [I 2022-10-26 14:54:51] ...trap/engines/engine.py.230: loss: 0.25883 [I 2022-10-26 14:54:52] ...optimizers/trijoint.py.151: total_norm: 4.104113 [I 2022-10-26 14:54:57] ...trap/engines/engine.py.224: train: epoch 14 | batch 840/2304 [I 2022-10-26 14:54:57] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:12:15 [I 2022-10-26 14:54:57] ...trap/engines/engine.py.229: process: 0.51098 | load: 0.00250 [I 2022-10-26 14:54:57] ...trap/engines/engine.py.230: loss: 0.19290 [I 2022-10-26 14:54:57] ...optimizers/trijoint.py.151: total_norm: 3.393071 [I 2022-10-26 14:55:02] ...trap/engines/engine.py.224: train: epoch 14 | batch 850/2304 [I 2022-10-26 14:55:02] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:12:06 [I 2022-10-26 14:55:02] ...trap/engines/engine.py.229: process: 0.50327 | load: 0.00249 [I 2022-10-26 14:55:02] ...trap/engines/engine.py.230: loss: 0.21848 [I 2022-10-26 14:55:02] ...optimizers/trijoint.py.151: total_norm: 3.333686 [I 2022-10-26 14:55:07] ...trap/engines/engine.py.224: train: epoch 14 | batch 860/2304 [I 2022-10-26 14:55:07] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:12:06 [I 2022-10-26 14:55:07] ...trap/engines/engine.py.229: process: 0.51078 | load: 0.00241 [I 2022-10-26 14:55:07] ...trap/engines/engine.py.230: loss: 0.20064 [I 2022-10-26 14:55:07] ...optimizers/trijoint.py.151: total_norm: 2.704181 [I 2022-10-26 14:55:12] ...trap/engines/engine.py.224: train: epoch 14 | batch 870/2304 [I 2022-10-26 14:55:12] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:12:10 [I 2022-10-26 14:55:12] ...trap/engines/engine.py.229: process: 0.50002 | load: 0.00244 [I 2022-10-26 14:55:12] ...trap/engines/engine.py.230: loss: 0.18271 [I 2022-10-26 14:55:12] ...optimizers/trijoint.py.151: total_norm: 4.092649 [I 2022-10-26 14:55:17] ...trap/engines/engine.py.224: train: epoch 14 | batch 880/2304 [I 2022-10-26 14:55:17] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:11:52 [I 2022-10-26 14:55:17] ...trap/engines/engine.py.229: process: 0.50087 | load: 0.00243 [I 2022-10-26 14:55:17] ...trap/engines/engine.py.230: loss: 0.25195 [I 2022-10-26 14:55:17] ...optimizers/trijoint.py.151: total_norm: 3.535619 [I 2022-10-26 14:55:22] ...trap/engines/engine.py.224: train: epoch 14 | batch 890/2304 [I 2022-10-26 14:55:22] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:12:02 [I 2022-10-26 14:55:22] ...trap/engines/engine.py.229: process: 0.49855 | load: 0.00251 [I 2022-10-26 14:55:22] ...trap/engines/engine.py.230: loss: 0.18190 [I 2022-10-26 14:55:22] ...optimizers/trijoint.py.151: total_norm: 2.918223 [I 2022-10-26 14:55:27] ...trap/engines/engine.py.224: train: epoch 14 | batch 900/2304 [I 2022-10-26 14:55:27] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:11:44 [I 2022-10-26 14:55:27] ...trap/engines/engine.py.229: process: 0.50266 | load: 0.00248 [I 2022-10-26 14:55:27] ...trap/engines/engine.py.230: loss: 0.26295 [I 2022-10-26 14:55:27] ...optimizers/trijoint.py.151: total_norm: 3.525224 [I 2022-10-26 14:55:32] ...trap/engines/engine.py.224: train: epoch 14 | batch 910/2304 [I 2022-10-26 14:55:32] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:11:38 [I 2022-10-26 14:55:32] ...trap/engines/engine.py.229: process: 0.50425 | load: 0.00246 [I 2022-10-26 14:55:32] ...trap/engines/engine.py.230: loss: 0.21453 [I 2022-10-26 14:55:32] ...optimizers/trijoint.py.151: total_norm: 5.217678 [I 2022-10-26 14:55:37] ...trap/engines/engine.py.224: train: epoch 14 | batch 920/2304 [I 2022-10-26 14:55:37] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:11:33 [I 2022-10-26 14:55:37] ...trap/engines/engine.py.229: process: 0.50326 | load: 0.00249 [I 2022-10-26 14:55:37] ...trap/engines/engine.py.230: loss: 0.26056 [I 2022-10-26 14:55:37] ...optimizers/trijoint.py.151: total_norm: 5.106044 [I 2022-10-26 14:55:42] ...trap/engines/engine.py.224: train: epoch 14 | batch 930/2304 [I 2022-10-26 14:55:42] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:11:28 [I 2022-10-26 14:55:42] ...trap/engines/engine.py.229: process: 0.50790 | load: 0.00243 [I 2022-10-26 14:55:42] ...trap/engines/engine.py.230: loss: 0.26462 [I 2022-10-26 14:55:42] ...optimizers/trijoint.py.151: total_norm: 3.271278 [I 2022-10-26 14:55:47] ...trap/engines/engine.py.224: train: epoch 14 | batch 940/2304 [I 2022-10-26 14:55:47] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:11:22 [I 2022-10-26 14:55:47] ...trap/engines/engine.py.229: process: 0.49306 | load: 0.00244 [I 2022-10-26 14:55:47] ...trap/engines/engine.py.230: loss: 0.23729 [I 2022-10-26 14:55:47] ...optimizers/trijoint.py.151: total_norm: 3.251361 [I 2022-10-26 14:55:52] ...trap/engines/engine.py.224: train: epoch 14 | batch 950/2304 [I 2022-10-26 14:55:52] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:11:18 [I 2022-10-26 14:55:52] ...trap/engines/engine.py.229: process: 0.49766 | load: 0.00247 [I 2022-10-26 14:55:52] ...trap/engines/engine.py.230: loss: 0.16112 [I 2022-10-26 14:55:52] ...optimizers/trijoint.py.151: total_norm: 2.896621 [I 2022-10-26 14:55:57] ...trap/engines/engine.py.224: train: epoch 14 | batch 960/2304 [I 2022-10-26 14:55:57] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:11:10 [I 2022-10-26 14:55:57] ...trap/engines/engine.py.229: process: 0.49629 | load: 0.00241 [I 2022-10-26 14:55:57] ...trap/engines/engine.py.230: loss: 0.23370 [I 2022-10-26 14:55:57] ...optimizers/trijoint.py.151: total_norm: 3.357964 [I 2022-10-26 14:56:02] ...trap/engines/engine.py.224: train: epoch 14 | batch 970/2304 [I 2022-10-26 14:56:02] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:11:06 [I 2022-10-26 14:56:02] ...trap/engines/engine.py.229: process: 0.50282 | load: 0.00241 [I 2022-10-26 14:56:02] ...trap/engines/engine.py.230: loss: 0.19943 [I 2022-10-26 14:56:02] ...optimizers/trijoint.py.151: total_norm: 2.758249 [I 2022-10-26 14:56:07] ...trap/engines/engine.py.224: train: epoch 14 | batch 980/2304 [I 2022-10-26 14:56:07] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:11:07 [I 2022-10-26 14:56:07] ...trap/engines/engine.py.229: process: 0.50106 | load: 0.00242 [I 2022-10-26 14:56:07] ...trap/engines/engine.py.230: loss: 0.18617 [I 2022-10-26 14:56:07] ...optimizers/trijoint.py.151: total_norm: 3.393311 [I 2022-10-26 14:56:12] ...trap/engines/engine.py.224: train: epoch 14 | batch 990/2304 [I 2022-10-26 14:56:12] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:10:57 [I 2022-10-26 14:56:12] ...trap/engines/engine.py.229: process: 0.50492 | load: 0.00242 [I 2022-10-26 14:56:12] ...trap/engines/engine.py.230: loss: 0.27921 [I 2022-10-26 14:56:12] ...optimizers/trijoint.py.151: total_norm: 3.519322 [I 2022-10-26 14:56:17] ...trap/engines/engine.py.224: train: epoch 14 | batch 1000/2304 [I 2022-10-26 14:56:17] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:10:50 [I 2022-10-26 14:56:17] ...trap/engines/engine.py.229: process: 0.50107 | load: 0.00241 [I 2022-10-26 14:56:17] ...trap/engines/engine.py.230: loss: 0.24888 [I 2022-10-26 14:56:17] ...optimizers/trijoint.py.151: total_norm: 3.959223 [I 2022-10-26 14:56:22] ...trap/engines/engine.py.224: train: epoch 14 | batch 1010/2304 [I 2022-10-26 14:56:22] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:10:44 [I 2022-10-26 14:56:22] ...trap/engines/engine.py.229: process: 0.49243 | load: 0.00247 [I 2022-10-26 14:56:22] ...trap/engines/engine.py.230: loss: 0.36105 [I 2022-10-26 14:56:22] ...optimizers/trijoint.py.151: total_norm: 4.477782 [I 2022-10-26 14:56:27] ...trap/engines/engine.py.224: train: epoch 14 | batch 1020/2304 [I 2022-10-26 14:56:27] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:10:37 [I 2022-10-26 14:56:27] ...trap/engines/engine.py.229: process: 0.49105 | load: 0.00244 [I 2022-10-26 14:56:27] ...trap/engines/engine.py.230: loss: 0.17307 [I 2022-10-26 14:56:27] ...optimizers/trijoint.py.151: total_norm: 3.823593 [I 2022-10-26 14:56:32] ...trap/engines/engine.py.224: train: epoch 14 | batch 1030/2304 [I 2022-10-26 14:56:32] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:10:37 [I 2022-10-26 14:56:32] ...trap/engines/engine.py.229: process: 0.50504 | load: 0.00253 [I 2022-10-26 14:56:32] ...trap/engines/engine.py.230: loss: 0.16840 [I 2022-10-26 14:56:32] ...optimizers/trijoint.py.151: total_norm: 2.368861 [I 2022-10-26 14:56:37] ...trap/engines/engine.py.224: train: epoch 14 | batch 1040/2304 [I 2022-10-26 14:56:37] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:10:40 [I 2022-10-26 14:56:37] ...trap/engines/engine.py.229: process: 0.49686 | load: 0.00248 [I 2022-10-26 14:56:37] ...trap/engines/engine.py.230: loss: 0.19905 [I 2022-10-26 14:56:37] ...optimizers/trijoint.py.151: total_norm: 3.740509 [I 2022-10-26 14:56:42] ...trap/engines/engine.py.224: train: epoch 14 | batch 1050/2304 [I 2022-10-26 14:56:42] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:10:44 [I 2022-10-26 14:56:42] ...trap/engines/engine.py.229: process: 0.49722 | load: 0.00246 [I 2022-10-26 14:56:42] ...trap/engines/engine.py.230: loss: 0.16081 [I 2022-10-26 14:56:42] ...optimizers/trijoint.py.151: total_norm: 2.408364 [I 2022-10-26 14:56:47] ...trap/engines/engine.py.224: train: epoch 14 | batch 1060/2304 [I 2022-10-26 14:56:47] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:10:21 [I 2022-10-26 14:56:47] ...trap/engines/engine.py.229: process: 0.50156 | load: 0.00244 [I 2022-10-26 14:56:47] ...trap/engines/engine.py.230: loss: 0.17706 [I 2022-10-26 14:56:47] ...optimizers/trijoint.py.151: total_norm: 2.483654 [I 2022-10-26 14:56:52] ...trap/engines/engine.py.224: train: epoch 14 | batch 1070/2304 [I 2022-10-26 14:56:52] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:10:13 [I 2022-10-26 14:56:52] ...trap/engines/engine.py.229: process: 0.49318 | load: 0.00245 [I 2022-10-26 14:56:52] ...trap/engines/engine.py.230: loss: 0.17732 [I 2022-10-26 14:56:52] ...optimizers/trijoint.py.151: total_norm: 2.862373 [I 2022-10-26 14:56:57] ...trap/engines/engine.py.224: train: epoch 14 | batch 1080/2304 [I 2022-10-26 14:56:57] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:10:13 [I 2022-10-26 14:56:57] ...trap/engines/engine.py.229: process: 0.49300 | load: 0.00246 [I 2022-10-26 14:56:57] ...trap/engines/engine.py.230: loss: 0.19023 [I 2022-10-26 14:56:57] ...optimizers/trijoint.py.151: total_norm: 2.652745 [I 2022-10-26 14:57:02] ...trap/engines/engine.py.224: train: epoch 14 | batch 1090/2304 [I 2022-10-26 14:57:02] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:10:08 [I 2022-10-26 14:57:02] ...trap/engines/engine.py.229: process: 0.50816 | load: 0.00244 [I 2022-10-26 14:57:02] ...trap/engines/engine.py.230: loss: 0.19814 [I 2022-10-26 14:57:02] ...optimizers/trijoint.py.151: total_norm: 3.105428 [I 2022-10-26 14:57:07] ...trap/engines/engine.py.224: train: epoch 14 | batch 1100/2304 [I 2022-10-26 14:57:07] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:10:02 [I 2022-10-26 14:57:07] ...trap/engines/engine.py.229: process: 0.50078 | load: 0.00249 [I 2022-10-26 14:57:07] ...trap/engines/engine.py.230: loss: 0.21152 [I 2022-10-26 14:57:07] ...optimizers/trijoint.py.151: total_norm: 3.168659 [I 2022-10-26 14:57:12] ...trap/engines/engine.py.224: train: epoch 14 | batch 1110/2304 [I 2022-10-26 14:57:12] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:10:00 [I 2022-10-26 14:57:12] ...trap/engines/engine.py.229: process: 0.50927 | load: 0.00247 [I 2022-10-26 14:57:12] ...trap/engines/engine.py.230: loss: 0.27790 [I 2022-10-26 14:57:12] ...optimizers/trijoint.py.151: total_norm: 3.171343 [I 2022-10-26 14:57:17] ...trap/engines/engine.py.224: train: epoch 14 | batch 1120/2304 [I 2022-10-26 14:57:17] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:09:59 [I 2022-10-26 14:57:17] ...trap/engines/engine.py.229: process: 0.50559 | load: 0.00253 [I 2022-10-26 14:57:17] ...trap/engines/engine.py.230: loss: 0.20229 [I 2022-10-26 14:57:17] ...optimizers/trijoint.py.151: total_norm: 3.265309 [I 2022-10-26 14:57:22] ...trap/engines/engine.py.224: train: epoch 14 | batch 1130/2304 [I 2022-10-26 14:57:22] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:09:48 [I 2022-10-26 14:57:22] ...trap/engines/engine.py.229: process: 0.49449 | load: 0.00249 [I 2022-10-26 14:57:22] ...trap/engines/engine.py.230: loss: 0.22233 [I 2022-10-26 14:57:22] ...optimizers/trijoint.py.151: total_norm: 3.416851 [I 2022-10-26 14:57:27] ...trap/engines/engine.py.224: train: epoch 14 | batch 1140/2304 [I 2022-10-26 14:57:27] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:09:40 [I 2022-10-26 14:57:27] ...trap/engines/engine.py.229: process: 0.50026 | load: 0.00247 [I 2022-10-26 14:57:27] ...trap/engines/engine.py.230: loss: 0.28755 [I 2022-10-26 14:57:27] ...optimizers/trijoint.py.151: total_norm: 3.387179 [I 2022-10-26 14:57:32] ...trap/engines/engine.py.224: train: epoch 14 | batch 1150/2304 [I 2022-10-26 14:57:32] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:09:31 [I 2022-10-26 14:57:32] ...trap/engines/engine.py.229: process: 0.50590 | load: 0.00268 [I 2022-10-26 14:57:32] ...trap/engines/engine.py.230: loss: 0.15280 [I 2022-10-26 14:57:32] ...optimizers/trijoint.py.151: total_norm: 2.492082 [I 2022-10-26 14:57:37] ...trap/engines/engine.py.224: train: epoch 14 | batch 1160/2304 [I 2022-10-26 14:57:37] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:09:25 [I 2022-10-26 14:57:37] ...trap/engines/engine.py.229: process: 0.49188 | load: 0.00252 [I 2022-10-26 14:57:37] ...trap/engines/engine.py.230: loss: 0.17669 [I 2022-10-26 14:57:37] ...optimizers/trijoint.py.151: total_norm: 3.175552 [I 2022-10-26 14:57:42] ...trap/engines/engine.py.224: train: epoch 14 | batch 1170/2304 [I 2022-10-26 14:57:42] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:09:26 [I 2022-10-26 14:57:42] ...trap/engines/engine.py.229: process: 0.49999 | load: 0.00248 [I 2022-10-26 14:57:42] ...trap/engines/engine.py.230: loss: 0.19174 [I 2022-10-26 14:57:42] ...optimizers/trijoint.py.151: total_norm: 4.307788 [I 2022-10-26 14:57:48] ...trap/engines/engine.py.224: train: epoch 14 | batch 1180/2304 [I 2022-10-26 14:57:48] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:09:26 [I 2022-10-26 14:57:48] ...trap/engines/engine.py.229: process: 0.49318 | load: 0.00245 [I 2022-10-26 14:57:48] ...trap/engines/engine.py.230: loss: 0.18751 [I 2022-10-26 14:57:48] ...optimizers/trijoint.py.151: total_norm: 2.805197 [I 2022-10-26 14:57:53] ...trap/engines/engine.py.224: train: epoch 14 | batch 1190/2304 [I 2022-10-26 14:57:53] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:09:13 [I 2022-10-26 14:57:53] ...trap/engines/engine.py.229: process: 0.49006 | load: 0.00246 [I 2022-10-26 14:57:53] ...trap/engines/engine.py.230: loss: 0.21952 [I 2022-10-26 14:57:53] ...optimizers/trijoint.py.151: total_norm: 3.497546 [I 2022-10-26 14:57:58] ...trap/engines/engine.py.224: train: epoch 14 | batch 1200/2304 [I 2022-10-26 14:57:58] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:09:09 [I 2022-10-26 14:57:58] ...trap/engines/engine.py.229: process: 0.49882 | load: 0.00248 [I 2022-10-26 14:57:58] ...trap/engines/engine.py.230: loss: 0.17300 [I 2022-10-26 14:57:58] ...optimizers/trijoint.py.151: total_norm: 2.530318 [I 2022-10-26 14:58:03] ...trap/engines/engine.py.224: train: epoch 14 | batch 1210/2304 [I 2022-10-26 14:58:03] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:09:05 [I 2022-10-26 14:58:03] ...trap/engines/engine.py.229: process: 0.49281 | load: 0.00250 [I 2022-10-26 14:58:03] ...trap/engines/engine.py.230: loss: 0.23994 [I 2022-10-26 14:58:03] ...optimizers/trijoint.py.151: total_norm: 3.529763 [I 2022-10-26 14:58:08] ...trap/engines/engine.py.224: train: epoch 14 | batch 1220/2304 [I 2022-10-26 14:58:08] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:09:01 [I 2022-10-26 14:58:08] ...trap/engines/engine.py.229: process: 0.48943 | load: 0.00250 [I 2022-10-26 14:58:08] ...trap/engines/engine.py.230: loss: 0.31958 [I 2022-10-26 14:58:08] ...optimizers/trijoint.py.151: total_norm: 4.277060 [I 2022-10-26 14:58:13] ...trap/engines/engine.py.224: train: epoch 14 | batch 1230/2304 [I 2022-10-26 14:58:13] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:09:01 [I 2022-10-26 14:58:13] ...trap/engines/engine.py.229: process: 0.51586 | load: 0.00247 [I 2022-10-26 14:58:13] ...trap/engines/engine.py.230: loss: 0.14461 [I 2022-10-26 14:58:13] ...optimizers/trijoint.py.151: total_norm: 2.177756 [I 2022-10-26 14:58:18] ...trap/engines/engine.py.224: train: epoch 14 | batch 1240/2304 [I 2022-10-26 14:58:18] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:08:52 [I 2022-10-26 14:58:18] ...trap/engines/engine.py.229: process: 0.50665 | load: 0.00256 [I 2022-10-26 14:58:18] ...trap/engines/engine.py.230: loss: 0.24839 [I 2022-10-26 14:58:18] ...optimizers/trijoint.py.151: total_norm: 3.689590 [I 2022-10-26 14:58:23] ...trap/engines/engine.py.224: train: epoch 14 | batch 1250/2304 [I 2022-10-26 14:58:23] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:08:49 [I 2022-10-26 14:58:23] ...trap/engines/engine.py.229: process: 0.50830 | load: 0.00247 [I 2022-10-26 14:58:23] ...trap/engines/engine.py.230: loss: 0.23411 [I 2022-10-26 14:58:23] ...optimizers/trijoint.py.151: total_norm: 3.588326 [I 2022-10-26 14:58:28] ...trap/engines/engine.py.224: train: epoch 14 | batch 1260/2304 [I 2022-10-26 14:58:28] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:08:46 [I 2022-10-26 14:58:28] ...trap/engines/engine.py.229: process: 0.50718 | load: 0.00250 [I 2022-10-26 14:58:28] ...trap/engines/engine.py.230: loss: 0.17925 [I 2022-10-26 14:58:28] ...optimizers/trijoint.py.151: total_norm: 3.378994 [I 2022-10-26 14:58:33] ...trap/engines/engine.py.224: train: epoch 14 | batch 1270/2304 [I 2022-10-26 14:58:33] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:08:39 [I 2022-10-26 14:58:33] ...trap/engines/engine.py.229: process: 0.50699 | load: 0.00248 [I 2022-10-26 14:58:33] ...trap/engines/engine.py.230: loss: 0.28395 [I 2022-10-26 14:58:33] ...optimizers/trijoint.py.151: total_norm: 3.663299 [I 2022-10-26 14:58:38] ...trap/engines/engine.py.224: train: epoch 14 | batch 1280/2304 [I 2022-10-26 14:58:38] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:08:30 [I 2022-10-26 14:58:38] ...trap/engines/engine.py.229: process: 0.50114 | load: 0.00249 [I 2022-10-26 14:58:38] ...trap/engines/engine.py.230: loss: 0.20284 [I 2022-10-26 14:58:38] ...optimizers/trijoint.py.151: total_norm: 2.577852 [I 2022-10-26 14:58:43] ...trap/engines/engine.py.224: train: epoch 14 | batch 1290/2304 [I 2022-10-26 14:58:43] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:08:27 [I 2022-10-26 14:58:43] ...trap/engines/engine.py.229: process: 0.50138 | load: 0.00242 [I 2022-10-26 14:58:43] ...trap/engines/engine.py.230: loss: 0.26223 [I 2022-10-26 14:58:43] ...optimizers/trijoint.py.151: total_norm: 4.092041 [I 2022-10-26 14:58:48] ...trap/engines/engine.py.224: train: epoch 14 | batch 1300/2304 [I 2022-10-26 14:58:48] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:08:25 [I 2022-10-26 14:58:48] ...trap/engines/engine.py.229: process: 0.49226 | load: 0.00245 [I 2022-10-26 14:58:48] ...trap/engines/engine.py.230: loss: 0.22872 [I 2022-10-26 14:58:48] ...optimizers/trijoint.py.151: total_norm: 3.166895 [I 2022-10-26 14:58:53] ...trap/engines/engine.py.224: train: epoch 14 | batch 1310/2304 [I 2022-10-26 14:58:53] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:08:18 [I 2022-10-26 14:58:53] ...trap/engines/engine.py.229: process: 0.49641 | load: 0.00244 [I 2022-10-26 14:58:53] ...trap/engines/engine.py.230: loss: 0.18848 [I 2022-10-26 14:58:53] ...optimizers/trijoint.py.151: total_norm: 2.364474 [I 2022-10-26 14:58:58] ...trap/engines/engine.py.224: train: epoch 14 | batch 1320/2304 [I 2022-10-26 14:58:58] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:08:12 [I 2022-10-26 14:58:58] ...trap/engines/engine.py.229: process: 0.49587 | load: 0.00256 [I 2022-10-26 14:58:58] ...trap/engines/engine.py.230: loss: 0.12976 [I 2022-10-26 14:58:58] ...optimizers/trijoint.py.151: total_norm: 1.945653 [I 2022-10-26 14:59:03] ...trap/engines/engine.py.224: train: epoch 14 | batch 1330/2304 [I 2022-10-26 14:59:03] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:08:09 [I 2022-10-26 14:59:03] ...trap/engines/engine.py.229: process: 0.50206 | load: 0.00254 [I 2022-10-26 14:59:03] ...trap/engines/engine.py.230: loss: 0.25210 [I 2022-10-26 14:59:03] ...optimizers/trijoint.py.151: total_norm: 4.751515 [I 2022-10-26 14:59:08] ...trap/engines/engine.py.224: train: epoch 14 | batch 1340/2304 [I 2022-10-26 14:59:08] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:08:01 [I 2022-10-26 14:59:08] ...trap/engines/engine.py.229: process: 0.50616 | load: 0.00248 [I 2022-10-26 14:59:08] ...trap/engines/engine.py.230: loss: 0.18769 [I 2022-10-26 14:59:08] ...optimizers/trijoint.py.151: total_norm: 2.856565 [I 2022-10-26 14:59:13] ...trap/engines/engine.py.224: train: epoch 14 | batch 1350/2304 [I 2022-10-26 14:59:13] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:07:55 [I 2022-10-26 14:59:13] ...trap/engines/engine.py.229: process: 0.49016 | load: 0.00243 [I 2022-10-26 14:59:13] ...trap/engines/engine.py.230: loss: 0.20422 [I 2022-10-26 14:59:13] ...optimizers/trijoint.py.151: total_norm: 2.970773 [I 2022-10-26 14:59:18] ...trap/engines/engine.py.224: train: epoch 14 | batch 1360/2304 [I 2022-10-26 14:59:18] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:07:53 [I 2022-10-26 14:59:18] ...trap/engines/engine.py.229: process: 0.50103 | load: 0.00265 [I 2022-10-26 14:59:18] ...trap/engines/engine.py.230: loss: 0.16426 [I 2022-10-26 14:59:18] ...optimizers/trijoint.py.151: total_norm: 2.787517 [I 2022-10-26 14:59:23] ...trap/engines/engine.py.224: train: epoch 14 | batch 1370/2304 [I 2022-10-26 14:59:23] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:07:49 [I 2022-10-26 14:59:23] ...trap/engines/engine.py.229: process: 0.50153 | load: 0.00245 [I 2022-10-26 14:59:23] ...trap/engines/engine.py.230: loss: 0.25696 [I 2022-10-26 14:59:23] ...optimizers/trijoint.py.151: total_norm: 4.401129 [I 2022-10-26 14:59:28] ...trap/engines/engine.py.224: train: epoch 14 | batch 1380/2304 [I 2022-10-26 14:59:28] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:07:39 [I 2022-10-26 14:59:28] ...trap/engines/engine.py.229: process: 0.49060 | load: 0.00245 [I 2022-10-26 14:59:28] ...trap/engines/engine.py.230: loss: 0.26816 [I 2022-10-26 14:59:28] ...optimizers/trijoint.py.151: total_norm: 3.818787 [I 2022-10-26 14:59:33] ...trap/engines/engine.py.224: train: epoch 14 | batch 1390/2304 [I 2022-10-26 14:59:33] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:07:35 [I 2022-10-26 14:59:33] ...trap/engines/engine.py.229: process: 0.49961 | load: 0.00250 [I 2022-10-26 14:59:33] ...trap/engines/engine.py.230: loss: 0.19661 [I 2022-10-26 14:59:33] ...optimizers/trijoint.py.151: total_norm: 2.513991 [I 2022-10-26 14:59:38] ...trap/engines/engine.py.224: train: epoch 14 | batch 1400/2304 [I 2022-10-26 14:59:38] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:07:30 [I 2022-10-26 14:59:38] ...trap/engines/engine.py.229: process: 0.49655 | load: 0.00255 [I 2022-10-26 14:59:38] ...trap/engines/engine.py.230: loss: 0.16054 [I 2022-10-26 14:59:38] ...optimizers/trijoint.py.151: total_norm: 3.206221 [I 2022-10-26 14:59:43] ...trap/engines/engine.py.224: train: epoch 14 | batch 1410/2304 [I 2022-10-26 14:59:43] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:07:24 [I 2022-10-26 14:59:43] ...trap/engines/engine.py.229: process: 0.49960 | load: 0.00243 [I 2022-10-26 14:59:43] ...trap/engines/engine.py.230: loss: 0.20425 [I 2022-10-26 14:59:43] ...optimizers/trijoint.py.151: total_norm: 3.464152 [I 2022-10-26 14:59:48] ...trap/engines/engine.py.224: train: epoch 14 | batch 1420/2304 [I 2022-10-26 14:59:48] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:07:20 [I 2022-10-26 14:59:48] ...trap/engines/engine.py.229: process: 0.50362 | load: 0.00261 [I 2022-10-26 14:59:48] ...trap/engines/engine.py.230: loss: 0.26001 [I 2022-10-26 14:59:48] ...optimizers/trijoint.py.151: total_norm: 3.428963 [I 2022-10-26 14:59:53] ...trap/engines/engine.py.224: train: epoch 14 | batch 1430/2304 [I 2022-10-26 14:59:53] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:07:17 [I 2022-10-26 14:59:53] ...trap/engines/engine.py.229: process: 0.48953 | load: 0.00240 [I 2022-10-26 14:59:53] ...trap/engines/engine.py.230: loss: 0.25158 [I 2022-10-26 14:59:53] ...optimizers/trijoint.py.151: total_norm: 3.434047 [I 2022-10-26 14:59:58] ...trap/engines/engine.py.224: train: epoch 14 | batch 1440/2304 [I 2022-10-26 14:59:58] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:07:12 [I 2022-10-26 14:59:58] ...trap/engines/engine.py.229: process: 0.50090 | load: 0.00243 [I 2022-10-26 14:59:58] ...trap/engines/engine.py.230: loss: 0.15317 [I 2022-10-26 14:59:58] ...optimizers/trijoint.py.151: total_norm: 2.160817 [I 2022-10-26 15:00:03] ...trap/engines/engine.py.224: train: epoch 14 | batch 1450/2304 [I 2022-10-26 15:00:03] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:07:07 [I 2022-10-26 15:00:03] ...trap/engines/engine.py.229: process: 0.49570 | load: 0.00252 [I 2022-10-26 15:00:03] ...trap/engines/engine.py.230: loss: 0.29373 [I 2022-10-26 15:00:03] ...optimizers/trijoint.py.151: total_norm: 3.340069 [I 2022-10-26 15:00:08] ...trap/engines/engine.py.224: train: epoch 14 | batch 1460/2304 [I 2022-10-26 15:00:08] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:06:59 [I 2022-10-26 15:00:08] ...trap/engines/engine.py.229: process: 0.49160 | load: 0.00251 [I 2022-10-26 15:00:08] ...trap/engines/engine.py.230: loss: 0.20672 [I 2022-10-26 15:00:08] ...optimizers/trijoint.py.151: total_norm: 4.283548 [I 2022-10-26 15:00:13] ...trap/engines/engine.py.224: train: epoch 14 | batch 1470/2304 [I 2022-10-26 15:00:13] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:06:55 [I 2022-10-26 15:00:13] ...trap/engines/engine.py.229: process: 0.50460 | load: 0.00248 [I 2022-10-26 15:00:13] ...trap/engines/engine.py.230: loss: 0.25320 [I 2022-10-26 15:00:13] ...optimizers/trijoint.py.151: total_norm: 3.995620 [I 2022-10-26 15:00:18] ...trap/engines/engine.py.224: train: epoch 14 | batch 1480/2304 [I 2022-10-26 15:00:18] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:06:46 [I 2022-10-26 15:00:18] ...trap/engines/engine.py.229: process: 0.49161 | load: 0.00245 [I 2022-10-26 15:00:18] ...trap/engines/engine.py.230: loss: 0.21985 [I 2022-10-26 15:00:18] ...optimizers/trijoint.py.151: total_norm: 3.496400 [I 2022-10-26 15:00:23] ...trap/engines/engine.py.224: train: epoch 14 | batch 1490/2304 [I 2022-10-26 15:00:23] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:06:45 [I 2022-10-26 15:00:23] ...trap/engines/engine.py.229: process: 0.50301 | load: 0.00245 [I 2022-10-26 15:00:23] ...trap/engines/engine.py.230: loss: 0.20421 [I 2022-10-26 15:00:23] ...optimizers/trijoint.py.151: total_norm: 2.804386 [I 2022-10-26 15:00:28] ...trap/engines/engine.py.224: train: epoch 14 | batch 1500/2304 [I 2022-10-26 15:00:28] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:06:43 [I 2022-10-26 15:00:28] ...trap/engines/engine.py.229: process: 0.49259 | load: 0.00244 [I 2022-10-26 15:00:28] ...trap/engines/engine.py.230: loss: 0.15479 [I 2022-10-26 15:00:28] ...optimizers/trijoint.py.151: total_norm: 2.516012 [I 2022-10-26 15:00:33] ...trap/engines/engine.py.224: train: epoch 14 | batch 1510/2304 [I 2022-10-26 15:00:33] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:06:41 [I 2022-10-26 15:00:33] ...trap/engines/engine.py.229: process: 0.51367 | load: 0.00257 [I 2022-10-26 15:00:33] ...trap/engines/engine.py.230: loss: 0.17353 [I 2022-10-26 15:00:33] ...optimizers/trijoint.py.151: total_norm: 3.049050 [I 2022-10-26 15:00:38] ...trap/engines/engine.py.224: train: epoch 14 | batch 1520/2304 [I 2022-10-26 15:00:38] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:06:30 [I 2022-10-26 15:00:38] ...trap/engines/engine.py.229: process: 0.48953 | load: 0.00243 [I 2022-10-26 15:00:38] ...trap/engines/engine.py.230: loss: 0.15647 [I 2022-10-26 15:00:38] ...optimizers/trijoint.py.151: total_norm: 2.256059 [I 2022-10-26 15:00:43] ...trap/engines/engine.py.224: train: epoch 14 | batch 1530/2304 [I 2022-10-26 15:00:43] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:06:27 [I 2022-10-26 15:00:43] ...trap/engines/engine.py.229: process: 0.50258 | load: 0.00247 [I 2022-10-26 15:00:43] ...trap/engines/engine.py.230: loss: 0.20183 [I 2022-10-26 15:00:43] ...optimizers/trijoint.py.151: total_norm: 3.389152 [I 2022-10-26 15:00:48] ...trap/engines/engine.py.224: train: epoch 14 | batch 1540/2304 [I 2022-10-26 15:00:48] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:06:22 [I 2022-10-26 15:00:48] ...trap/engines/engine.py.229: process: 0.50018 | load: 0.00248 [I 2022-10-26 15:00:48] ...trap/engines/engine.py.230: loss: 0.17145 [I 2022-10-26 15:00:48] ...optimizers/trijoint.py.151: total_norm: 2.386786 [I 2022-10-26 15:00:53] ...trap/engines/engine.py.224: train: epoch 14 | batch 1550/2304 [I 2022-10-26 15:00:53] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:06:16 [I 2022-10-26 15:00:53] ...trap/engines/engine.py.229: process: 0.50509 | load: 0.00245 [I 2022-10-26 15:00:53] ...trap/engines/engine.py.230: loss: 0.30501 [I 2022-10-26 15:00:53] ...optimizers/trijoint.py.151: total_norm: 2.887483 [I 2022-10-26 15:00:58] ...trap/engines/engine.py.224: train: epoch 14 | batch 1560/2304 [I 2022-10-26 15:00:58] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:06:09 [I 2022-10-26 15:00:58] ...trap/engines/engine.py.229: process: 0.49090 | load: 0.00244 [I 2022-10-26 15:00:58] ...trap/engines/engine.py.230: loss: 0.26145 [I 2022-10-26 15:00:58] ...optimizers/trijoint.py.151: total_norm: 3.885050 [I 2022-10-26 15:01:03] ...trap/engines/engine.py.224: train: epoch 14 | batch 1570/2304 [I 2022-10-26 15:01:03] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:06:07 [I 2022-10-26 15:01:03] ...trap/engines/engine.py.229: process: 0.51225 | load: 0.00247 [I 2022-10-26 15:01:03] ...trap/engines/engine.py.230: loss: 0.19781 [I 2022-10-26 15:01:03] ...optimizers/trijoint.py.151: total_norm: 3.356391 [I 2022-10-26 15:01:08] ...trap/engines/engine.py.224: train: epoch 14 | batch 1580/2304 [I 2022-10-26 15:01:08] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:06:02 [I 2022-10-26 15:01:08] ...trap/engines/engine.py.229: process: 0.50195 | load: 0.00253 [I 2022-10-26 15:01:08] ...trap/engines/engine.py.230: loss: 0.17757 [I 2022-10-26 15:01:08] ...optimizers/trijoint.py.151: total_norm: 1.976162 [I 2022-10-26 15:01:13] ...trap/engines/engine.py.224: train: epoch 14 | batch 1590/2304 [I 2022-10-26 15:01:13] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:05:55 [I 2022-10-26 15:01:13] ...trap/engines/engine.py.229: process: 0.48635 | load: 0.00248 [I 2022-10-26 15:01:13] ...trap/engines/engine.py.230: loss: 0.28143 [I 2022-10-26 15:01:13] ...optimizers/trijoint.py.151: total_norm: 4.411623 [I 2022-10-26 15:01:18] ...trap/engines/engine.py.224: train: epoch 14 | batch 1600/2304 [I 2022-10-26 15:01:18] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:05:52 [I 2022-10-26 15:01:18] ...trap/engines/engine.py.229: process: 0.50471 | load: 0.00248 [I 2022-10-26 15:01:18] ...trap/engines/engine.py.230: loss: 0.23917 [I 2022-10-26 15:01:18] ...optimizers/trijoint.py.151: total_norm: 4.536066 [I 2022-10-26 15:01:23] ...trap/engines/engine.py.224: train: epoch 14 | batch 1610/2304 [I 2022-10-26 15:01:23] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:05:48 [I 2022-10-26 15:01:23] ...trap/engines/engine.py.229: process: 0.49186 | load: 0.00248 [I 2022-10-26 15:01:23] ...trap/engines/engine.py.230: loss: 0.17143 [I 2022-10-26 15:01:23] ...optimizers/trijoint.py.151: total_norm: 2.749668 [I 2022-10-26 15:01:28] ...trap/engines/engine.py.224: train: epoch 14 | batch 1620/2304 [I 2022-10-26 15:01:28] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:05:41 [I 2022-10-26 15:01:28] ...trap/engines/engine.py.229: process: 0.50134 | load: 0.00243 [I 2022-10-26 15:01:28] ...trap/engines/engine.py.230: loss: 0.17137 [I 2022-10-26 15:01:28] ...optimizers/trijoint.py.151: total_norm: 2.331541 [I 2022-10-26 15:01:33] ...trap/engines/engine.py.224: train: epoch 14 | batch 1630/2304 [I 2022-10-26 15:01:33] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:05:36 [I 2022-10-26 15:01:33] ...trap/engines/engine.py.229: process: 0.49438 | load: 0.00246 [I 2022-10-26 15:01:33] ...trap/engines/engine.py.230: loss: 0.24844 [I 2022-10-26 15:01:33] ...optimizers/trijoint.py.151: total_norm: 3.623327 [I 2022-10-26 15:01:38] ...trap/engines/engine.py.224: train: epoch 14 | batch 1640/2304 [I 2022-10-26 15:01:38] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:05:33 [I 2022-10-26 15:01:38] ...trap/engines/engine.py.229: process: 0.49342 | load: 0.00245 [I 2022-10-26 15:01:38] ...trap/engines/engine.py.230: loss: 0.30424 [I 2022-10-26 15:01:38] ...optimizers/trijoint.py.151: total_norm: 4.270638 [I 2022-10-26 15:01:43] ...trap/engines/engine.py.224: train: epoch 14 | batch 1650/2304 [I 2022-10-26 15:01:43] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:05:34 [I 2022-10-26 15:01:43] ...trap/engines/engine.py.229: process: 0.53575 | load: 0.00268 [I 2022-10-26 15:01:43] ...trap/engines/engine.py.230: loss: 0.21239 [I 2022-10-26 15:01:43] ...optimizers/trijoint.py.151: total_norm: 2.985494 [I 2022-10-26 15:01:49] ...trap/engines/engine.py.224: train: epoch 14 | batch 1660/2304 [I 2022-10-26 15:01:49] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:05:23 [I 2022-10-26 15:01:49] ...trap/engines/engine.py.229: process: 0.50433 | load: 0.00254 [I 2022-10-26 15:01:49] ...trap/engines/engine.py.230: loss: 0.14328 [I 2022-10-26 15:01:49] ...optimizers/trijoint.py.151: total_norm: 3.406245 [I 2022-10-26 15:01:54] ...trap/engines/engine.py.224: train: epoch 14 | batch 1670/2304 [I 2022-10-26 15:01:54] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:05:18 [I 2022-10-26 15:01:54] ...trap/engines/engine.py.229: process: 0.50552 | load: 0.00246 [I 2022-10-26 15:01:54] ...trap/engines/engine.py.230: loss: 0.33999 [I 2022-10-26 15:01:54] ...optimizers/trijoint.py.151: total_norm: 3.837996 [I 2022-10-26 15:01:59] ...trap/engines/engine.py.224: train: epoch 14 | batch 1680/2304 [I 2022-10-26 15:01:59] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:05:10 [I 2022-10-26 15:01:59] ...trap/engines/engine.py.229: process: 0.49110 | load: 0.00245 [I 2022-10-26 15:01:59] ...trap/engines/engine.py.230: loss: 0.21188 [I 2022-10-26 15:01:59] ...optimizers/trijoint.py.151: total_norm: 2.476331 [I 2022-10-26 15:02:04] ...trap/engines/engine.py.224: train: epoch 14 | batch 1690/2304 [I 2022-10-26 15:02:04] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:05:05 [I 2022-10-26 15:02:04] ...trap/engines/engine.py.229: process: 0.48923 | load: 0.00245 [I 2022-10-26 15:02:04] ...trap/engines/engine.py.230: loss: 0.18651 [I 2022-10-26 15:02:04] ...optimizers/trijoint.py.151: total_norm: 3.286443 [I 2022-10-26 15:02:09] ...trap/engines/engine.py.224: train: epoch 14 | batch 1700/2304 [I 2022-10-26 15:02:09] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:05:01 [I 2022-10-26 15:02:09] ...trap/engines/engine.py.229: process: 0.50132 | load: 0.00249 [I 2022-10-26 15:02:09] ...trap/engines/engine.py.230: loss: 0.26339 [I 2022-10-26 15:02:09] ...optimizers/trijoint.py.151: total_norm: 4.769628 [I 2022-10-26 15:02:14] ...trap/engines/engine.py.224: train: epoch 14 | batch 1710/2304 [I 2022-10-26 15:02:14] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:04:57 [I 2022-10-26 15:02:14] ...trap/engines/engine.py.229: process: 0.49355 | load: 0.00244 [I 2022-10-26 15:02:14] ...trap/engines/engine.py.230: loss: 0.17144 [I 2022-10-26 15:02:14] ...optimizers/trijoint.py.151: total_norm: 2.684139 [I 2022-10-26 15:02:19] ...trap/engines/engine.py.224: train: epoch 14 | batch 1720/2304 [I 2022-10-26 15:02:19] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:04:50 [I 2022-10-26 15:02:19] ...trap/engines/engine.py.229: process: 0.49707 | load: 0.00241 [I 2022-10-26 15:02:19] ...trap/engines/engine.py.230: loss: 0.24451 [I 2022-10-26 15:02:19] ...optimizers/trijoint.py.151: total_norm: 3.174353 [I 2022-10-26 15:02:24] ...trap/engines/engine.py.224: train: epoch 14 | batch 1730/2304 [I 2022-10-26 15:02:24] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:04:46 [I 2022-10-26 15:02:24] ...trap/engines/engine.py.229: process: 0.50713 | load: 0.00247 [I 2022-10-26 15:02:24] ...trap/engines/engine.py.230: loss: 0.21374 [I 2022-10-26 15:02:24] ...optimizers/trijoint.py.151: total_norm: 3.325792 [I 2022-10-26 15:02:29] ...trap/engines/engine.py.224: train: epoch 14 | batch 1740/2304 [I 2022-10-26 15:02:29] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:04:42 [I 2022-10-26 15:02:29] ...trap/engines/engine.py.229: process: 0.49533 | load: 0.00244 [I 2022-10-26 15:02:29] ...trap/engines/engine.py.230: loss: 0.20952 [I 2022-10-26 15:02:29] ...optimizers/trijoint.py.151: total_norm: 3.197750 [I 2022-10-26 15:02:34] ...trap/engines/engine.py.224: train: epoch 14 | batch 1750/2304 [I 2022-10-26 15:02:34] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:04:35 [I 2022-10-26 15:02:34] ...trap/engines/engine.py.229: process: 0.49421 | load: 0.00245 [I 2022-10-26 15:02:34] ...trap/engines/engine.py.230: loss: 0.20211 [I 2022-10-26 15:02:34] ...optimizers/trijoint.py.151: total_norm: 3.040367 [I 2022-10-26 15:02:39] ...trap/engines/engine.py.224: train: epoch 14 | batch 1760/2304 [I 2022-10-26 15:02:39] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:04:30 [I 2022-10-26 15:02:39] ...trap/engines/engine.py.229: process: 0.49737 | load: 0.00242 [I 2022-10-26 15:02:39] ...trap/engines/engine.py.230: loss: 0.21799 [I 2022-10-26 15:02:39] ...optimizers/trijoint.py.151: total_norm: 4.009597 [I 2022-10-26 15:02:44] ...trap/engines/engine.py.224: train: epoch 14 | batch 1770/2304 [I 2022-10-26 15:02:44] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:04:27 [I 2022-10-26 15:02:44] ...trap/engines/engine.py.229: process: 0.49771 | load: 0.00247 [I 2022-10-26 15:02:44] ...trap/engines/engine.py.230: loss: 0.21561 [I 2022-10-26 15:02:44] ...optimizers/trijoint.py.151: total_norm: 3.806532 [I 2022-10-26 15:02:49] ...trap/engines/engine.py.224: train: epoch 14 | batch 1780/2304 [I 2022-10-26 15:02:49] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:04:21 [I 2022-10-26 15:02:49] ...trap/engines/engine.py.229: process: 0.50210 | load: 0.00245 [I 2022-10-26 15:02:49] ...trap/engines/engine.py.230: loss: 0.27172 [I 2022-10-26 15:02:49] ...optimizers/trijoint.py.151: total_norm: 4.123433 [I 2022-10-26 15:02:54] ...trap/engines/engine.py.224: train: epoch 14 | batch 1790/2304 [I 2022-10-26 15:02:54] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:04:16 [I 2022-10-26 15:02:54] ...trap/engines/engine.py.229: process: 0.49761 | load: 0.00245 [I 2022-10-26 15:02:54] ...trap/engines/engine.py.230: loss: 0.20802 [I 2022-10-26 15:02:54] ...optimizers/trijoint.py.151: total_norm: 2.311637 [I 2022-10-26 15:02:59] ...trap/engines/engine.py.224: train: epoch 14 | batch 1800/2304 [I 2022-10-26 15:02:59] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:04:12 [I 2022-10-26 15:02:59] ...trap/engines/engine.py.229: process: 0.50418 | load: 0.00241 [I 2022-10-26 15:02:59] ...trap/engines/engine.py.230: loss: 0.19875 [I 2022-10-26 15:02:59] ...optimizers/trijoint.py.151: total_norm: 4.704826 [I 2022-10-26 15:03:04] ...trap/engines/engine.py.224: train: epoch 14 | batch 1810/2304 [I 2022-10-26 15:03:04] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:04:06 [I 2022-10-26 15:03:04] ...trap/engines/engine.py.229: process: 0.49968 | load: 0.00245 [I 2022-10-26 15:03:04] ...trap/engines/engine.py.230: loss: 0.19323 [I 2022-10-26 15:03:04] ...optimizers/trijoint.py.151: total_norm: 2.450805 [I 2022-10-26 15:03:09] ...trap/engines/engine.py.224: train: epoch 14 | batch 1820/2304 [I 2022-10-26 15:03:09] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:04:03 [I 2022-10-26 15:03:09] ...trap/engines/engine.py.229: process: 0.50837 | load: 0.00244 [I 2022-10-26 15:03:09] ...trap/engines/engine.py.230: loss: 0.28253 [I 2022-10-26 15:03:09] ...optimizers/trijoint.py.151: total_norm: 2.978706 [I 2022-10-26 15:03:14] ...trap/engines/engine.py.224: train: epoch 14 | batch 1830/2304 [I 2022-10-26 15:03:14] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:03:57 [I 2022-10-26 15:03:14] ...trap/engines/engine.py.229: process: 0.50720 | load: 0.00244 [I 2022-10-26 15:03:14] ...trap/engines/engine.py.230: loss: 0.17246 [I 2022-10-26 15:03:14] ...optimizers/trijoint.py.151: total_norm: 2.846782 [I 2022-10-26 15:03:19] ...trap/engines/engine.py.224: train: epoch 14 | batch 1840/2304 [I 2022-10-26 15:03:19] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:03:51 [I 2022-10-26 15:03:19] ...trap/engines/engine.py.229: process: 0.49917 | load: 0.00246 [I 2022-10-26 15:03:19] ...trap/engines/engine.py.230: loss: 0.24323 [I 2022-10-26 15:03:19] ...optimizers/trijoint.py.151: total_norm: 2.984599 [I 2022-10-26 15:03:24] ...trap/engines/engine.py.224: train: epoch 14 | batch 1850/2304 [I 2022-10-26 15:03:24] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:03:46 [I 2022-10-26 15:03:24] ...trap/engines/engine.py.229: process: 0.49316 | load: 0.00247 [I 2022-10-26 15:03:24] ...trap/engines/engine.py.230: loss: 0.16100 [I 2022-10-26 15:03:24] ...optimizers/trijoint.py.151: total_norm: 2.565768 [I 2022-10-26 15:03:29] ...trap/engines/engine.py.224: train: epoch 14 | batch 1860/2304 [I 2022-10-26 15:03:29] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:03:40 [I 2022-10-26 15:03:29] ...trap/engines/engine.py.229: process: 0.49182 | load: 0.00243 [I 2022-10-26 15:03:29] ...trap/engines/engine.py.230: loss: 0.15472 [I 2022-10-26 15:03:29] ...optimizers/trijoint.py.151: total_norm: 2.431950 [I 2022-10-26 15:03:34] ...trap/engines/engine.py.224: train: epoch 14 | batch 1870/2304 [I 2022-10-26 15:03:34] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:03:36 [I 2022-10-26 15:03:34] ...trap/engines/engine.py.229: process: 0.49555 | load: 0.00250 [I 2022-10-26 15:03:34] ...trap/engines/engine.py.230: loss: 0.18713 [I 2022-10-26 15:03:34] ...optimizers/trijoint.py.151: total_norm: 3.266716 [I 2022-10-26 15:03:39] ...trap/engines/engine.py.224: train: epoch 14 | batch 1880/2304 [I 2022-10-26 15:03:39] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:03:31 [I 2022-10-26 15:03:39] ...trap/engines/engine.py.229: process: 0.49277 | load: 0.00245 [I 2022-10-26 15:03:39] ...trap/engines/engine.py.230: loss: 0.22182 [I 2022-10-26 15:03:39] ...optimizers/trijoint.py.151: total_norm: 3.285223 [I 2022-10-26 15:03:44] ...trap/engines/engine.py.224: train: epoch 14 | batch 1890/2304 [I 2022-10-26 15:03:44] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:03:27 [I 2022-10-26 15:03:44] ...trap/engines/engine.py.229: process: 0.50385 | load: 0.00242 [I 2022-10-26 15:03:44] ...trap/engines/engine.py.230: loss: 0.27070 [I 2022-10-26 15:03:44] ...optimizers/trijoint.py.151: total_norm: 7.197997 [I 2022-10-26 15:03:49] ...trap/engines/engine.py.224: train: epoch 14 | batch 1900/2304 [I 2022-10-26 15:03:49] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:03:22 [I 2022-10-26 15:03:49] ...trap/engines/engine.py.229: process: 0.49998 | load: 0.00254 [I 2022-10-26 15:03:49] ...trap/engines/engine.py.230: loss: 0.19522 [I 2022-10-26 15:03:49] ...optimizers/trijoint.py.151: total_norm: 3.097537 [I 2022-10-26 15:03:54] ...trap/engines/engine.py.224: train: epoch 14 | batch 1910/2304 [I 2022-10-26 15:03:54] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:03:18 [I 2022-10-26 15:03:54] ...trap/engines/engine.py.229: process: 0.51325 | load: 0.00250 [I 2022-10-26 15:03:54] ...trap/engines/engine.py.230: loss: 0.35912 [I 2022-10-26 15:03:54] ...optimizers/trijoint.py.151: total_norm: 4.612470 [I 2022-10-26 15:03:59] ...trap/engines/engine.py.224: train: epoch 14 | batch 1920/2304 [I 2022-10-26 15:03:59] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:03:11 [I 2022-10-26 15:03:59] ...trap/engines/engine.py.229: process: 0.49591 | load: 0.00248 [I 2022-10-26 15:03:59] ...trap/engines/engine.py.230: loss: 0.29524 [I 2022-10-26 15:03:59] ...optimizers/trijoint.py.151: total_norm: 3.015992 [I 2022-10-26 15:04:04] ...trap/engines/engine.py.224: train: epoch 14 | batch 1930/2304 [I 2022-10-26 15:04:04] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:03:06 [I 2022-10-26 15:04:04] ...trap/engines/engine.py.229: process: 0.49955 | load: 0.00246 [I 2022-10-26 15:04:04] ...trap/engines/engine.py.230: loss: 0.15777 [I 2022-10-26 15:04:04] ...optimizers/trijoint.py.151: total_norm: 2.199804 [I 2022-10-26 15:04:09] ...trap/engines/engine.py.224: train: epoch 14 | batch 1940/2304 [I 2022-10-26 15:04:09] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:03:00 [I 2022-10-26 15:04:09] ...trap/engines/engine.py.229: process: 0.48891 | load: 0.00247 [I 2022-10-26 15:04:09] ...trap/engines/engine.py.230: loss: 0.30166 [I 2022-10-26 15:04:09] ...optimizers/trijoint.py.151: total_norm: 3.718768 [I 2022-10-26 15:04:14] ...trap/engines/engine.py.224: train: epoch 14 | batch 1950/2304 [I 2022-10-26 15:04:14] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:02:57 [I 2022-10-26 15:04:14] ...trap/engines/engine.py.229: process: 0.50328 | load: 0.00243 [I 2022-10-26 15:04:14] ...trap/engines/engine.py.230: loss: 0.16946 [I 2022-10-26 15:04:14] ...optimizers/trijoint.py.151: total_norm: 3.674873 [I 2022-10-26 15:04:19] ...trap/engines/engine.py.224: train: epoch 14 | batch 1960/2304 [I 2022-10-26 15:04:19] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:02:53 [I 2022-10-26 15:04:19] ...trap/engines/engine.py.229: process: 0.49417 | load: 0.00244 [I 2022-10-26 15:04:19] ...trap/engines/engine.py.230: loss: 0.23123 [I 2022-10-26 15:04:19] ...optimizers/trijoint.py.151: total_norm: 3.111033 [I 2022-10-26 15:04:24] ...trap/engines/engine.py.224: train: epoch 14 | batch 1970/2304 [I 2022-10-26 15:04:24] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:02:47 [I 2022-10-26 15:04:24] ...trap/engines/engine.py.229: process: 0.50934 | load: 0.00254 [I 2022-10-26 15:04:24] ...trap/engines/engine.py.230: loss: 0.13521 [I 2022-10-26 15:04:24] ...optimizers/trijoint.py.151: total_norm: 3.176788 [I 2022-10-26 15:04:29] ...trap/engines/engine.py.224: train: epoch 14 | batch 1980/2304 [I 2022-10-26 15:04:29] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:02:41 [I 2022-10-26 15:04:29] ...trap/engines/engine.py.229: process: 0.49081 | load: 0.00251 [I 2022-10-26 15:04:29] ...trap/engines/engine.py.230: loss: 0.22002 [I 2022-10-26 15:04:29] ...optimizers/trijoint.py.151: total_norm: 3.201783 [I 2022-10-26 15:04:34] ...trap/engines/engine.py.224: train: epoch 14 | batch 1990/2304 [I 2022-10-26 15:04:34] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:02:37 [I 2022-10-26 15:04:34] ...trap/engines/engine.py.229: process: 0.50187 | load: 0.00245 [I 2022-10-26 15:04:34] ...trap/engines/engine.py.230: loss: 0.22683 [I 2022-10-26 15:04:34] ...optimizers/trijoint.py.151: total_norm: 3.224680 [I 2022-10-26 15:04:39] ...trap/engines/engine.py.224: train: epoch 14 | batch 2000/2304 [I 2022-10-26 15:04:39] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:02:32 [I 2022-10-26 15:04:39] ...trap/engines/engine.py.229: process: 0.50447 | load: 0.00244 [I 2022-10-26 15:04:39] ...trap/engines/engine.py.230: loss: 0.22466 [I 2022-10-26 15:04:39] ...optimizers/trijoint.py.151: total_norm: 2.949885 [I 2022-10-26 15:04:44] ...trap/engines/engine.py.224: train: epoch 14 | batch 2010/2304 [I 2022-10-26 15:04:44] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:02:27 [I 2022-10-26 15:04:44] ...trap/engines/engine.py.229: process: 0.50850 | load: 0.00247 [I 2022-10-26 15:04:44] ...trap/engines/engine.py.230: loss: 0.22023 [I 2022-10-26 15:04:44] ...optimizers/trijoint.py.151: total_norm: 4.070881 [I 2022-10-26 15:04:49] ...trap/engines/engine.py.224: train: epoch 14 | batch 2020/2304 [I 2022-10-26 15:04:49] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:02:20 [I 2022-10-26 15:04:49] ...trap/engines/engine.py.229: process: 0.48760 | load: 0.00243 [I 2022-10-26 15:04:49] ...trap/engines/engine.py.230: loss: 0.15727 [I 2022-10-26 15:04:49] ...optimizers/trijoint.py.151: total_norm: 2.637555 [I 2022-10-26 15:04:54] ...trap/engines/engine.py.224: train: epoch 14 | batch 2030/2304 [I 2022-10-26 15:04:54] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:02:15 [I 2022-10-26 15:04:54] ...trap/engines/engine.py.229: process: 0.50020 | load: 0.00261 [I 2022-10-26 15:04:54] ...trap/engines/engine.py.230: loss: 0.23570 [I 2022-10-26 15:04:54] ...optimizers/trijoint.py.151: total_norm: 3.683605 [I 2022-10-26 15:04:59] ...trap/engines/engine.py.224: train: epoch 14 | batch 2040/2304 [I 2022-10-26 15:04:59] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:02:12 [I 2022-10-26 15:04:59] ...trap/engines/engine.py.229: process: 0.49762 | load: 0.00251 [I 2022-10-26 15:04:59] ...trap/engines/engine.py.230: loss: 0.24068 [I 2022-10-26 15:04:59] ...optimizers/trijoint.py.151: total_norm: 3.630507 [I 2022-10-26 15:05:04] ...trap/engines/engine.py.224: train: epoch 14 | batch 2050/2304 [I 2022-10-26 15:05:04] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:02:06 [I 2022-10-26 15:05:04] ...trap/engines/engine.py.229: process: 0.50067 | load: 0.00249 [I 2022-10-26 15:05:04] ...trap/engines/engine.py.230: loss: 0.19862 [I 2022-10-26 15:05:04] ...optimizers/trijoint.py.151: total_norm: 3.043904 [I 2022-10-26 15:05:09] ...trap/engines/engine.py.224: train: epoch 14 | batch 2060/2304 [I 2022-10-26 15:05:09] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:02:02 [I 2022-10-26 15:05:09] ...trap/engines/engine.py.229: process: 0.50237 | load: 0.00247 [I 2022-10-26 15:05:09] ...trap/engines/engine.py.230: loss: 0.22436 [I 2022-10-26 15:05:09] ...optimizers/trijoint.py.151: total_norm: 3.614876 [I 2022-10-26 15:05:14] ...trap/engines/engine.py.224: train: epoch 14 | batch 2070/2304 [I 2022-10-26 15:05:14] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:01:56 [I 2022-10-26 15:05:14] ...trap/engines/engine.py.229: process: 0.49310 | load: 0.00247 [I 2022-10-26 15:05:14] ...trap/engines/engine.py.230: loss: 0.15380 [I 2022-10-26 15:05:14] ...optimizers/trijoint.py.151: total_norm: 2.327952 [I 2022-10-26 15:05:19] ...trap/engines/engine.py.224: train: epoch 14 | batch 2080/2304 [I 2022-10-26 15:05:19] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:01:52 [I 2022-10-26 15:05:19] ...trap/engines/engine.py.229: process: 0.51426 | load: 0.00245 [I 2022-10-26 15:05:19] ...trap/engines/engine.py.230: loss: 0.17139 [I 2022-10-26 15:05:19] ...optimizers/trijoint.py.151: total_norm: 3.690351 [I 2022-10-26 15:05:24] ...trap/engines/engine.py.224: train: epoch 14 | batch 2090/2304 [I 2022-10-26 15:05:24] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:01:48 [I 2022-10-26 15:05:24] ...trap/engines/engine.py.229: process: 0.50304 | load: 0.00251 [I 2022-10-26 15:05:24] ...trap/engines/engine.py.230: loss: 0.20935 [I 2022-10-26 15:05:24] ...optimizers/trijoint.py.151: total_norm: 3.730158 [I 2022-10-26 15:05:29] ...trap/engines/engine.py.224: train: epoch 14 | batch 2100/2304 [I 2022-10-26 15:05:29] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:01:44 [I 2022-10-26 15:05:29] ...trap/engines/engine.py.229: process: 0.58145 | load: 0.00249 [I 2022-10-26 15:05:29] ...trap/engines/engine.py.230: loss: 0.29931 [I 2022-10-26 15:05:29] ...optimizers/trijoint.py.151: total_norm: 3.310102 [I 2022-10-26 15:05:34] ...trap/engines/engine.py.224: train: epoch 14 | batch 2110/2304 [I 2022-10-26 15:05:34] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:01:37 [I 2022-10-26 15:05:34] ...trap/engines/engine.py.229: process: 0.50415 | load: 0.00246 [I 2022-10-26 15:05:34] ...trap/engines/engine.py.230: loss: 0.21177 [I 2022-10-26 15:05:34] ...optimizers/trijoint.py.151: total_norm: 2.900263 [I 2022-10-26 15:05:39] ...trap/engines/engine.py.224: train: epoch 14 | batch 2120/2304 [I 2022-10-26 15:05:39] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:01:31 [I 2022-10-26 15:05:39] ...trap/engines/engine.py.229: process: 0.49156 | load: 0.00244 [I 2022-10-26 15:05:39] ...trap/engines/engine.py.230: loss: 0.17284 [I 2022-10-26 15:05:39] ...optimizers/trijoint.py.151: total_norm: 2.920816 [I 2022-10-26 15:05:44] ...trap/engines/engine.py.224: train: epoch 14 | batch 2130/2304 [I 2022-10-26 15:05:44] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:01:26 [I 2022-10-26 15:05:44] ...trap/engines/engine.py.229: process: 0.49410 | load: 0.00245 [I 2022-10-26 15:05:44] ...trap/engines/engine.py.230: loss: 0.23085 [I 2022-10-26 15:05:44] ...optimizers/trijoint.py.151: total_norm: 4.198045 [I 2022-10-26 15:05:49] ...trap/engines/engine.py.224: train: epoch 14 | batch 2140/2304 [I 2022-10-26 15:05:49] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:01:21 [I 2022-10-26 15:05:49] ...trap/engines/engine.py.229: process: 0.48915 | load: 0.00244 [I 2022-10-26 15:05:49] ...trap/engines/engine.py.230: loss: 0.24300 [I 2022-10-26 15:05:49] ...optimizers/trijoint.py.151: total_norm: 3.032300 [I 2022-10-26 15:05:54] ...trap/engines/engine.py.224: train: epoch 14 | batch 2150/2304 [I 2022-10-26 15:05:54] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:01:16 [I 2022-10-26 15:05:54] ...trap/engines/engine.py.229: process: 0.48058 | load: 0.00257 [I 2022-10-26 15:05:54] ...trap/engines/engine.py.230: loss: 0.29750 [I 2022-10-26 15:05:54] ...optimizers/trijoint.py.151: total_norm: 3.258189 [I 2022-10-26 15:05:59] ...trap/engines/engine.py.224: train: epoch 14 | batch 2160/2304 [I 2022-10-26 15:05:59] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:01:12 [I 2022-10-26 15:05:59] ...trap/engines/engine.py.229: process: 0.49981 | load: 0.00250 [I 2022-10-26 15:05:59] ...trap/engines/engine.py.230: loss: 0.22647 [I 2022-10-26 15:05:59] ...optimizers/trijoint.py.151: total_norm: 3.841201 [I 2022-10-26 15:06:04] ...trap/engines/engine.py.224: train: epoch 14 | batch 2170/2304 [I 2022-10-26 15:06:04] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:01:06 [I 2022-10-26 15:06:04] ...trap/engines/engine.py.229: process: 0.49760 | load: 0.00258 [I 2022-10-26 15:06:04] ...trap/engines/engine.py.230: loss: 0.16530 [I 2022-10-26 15:06:04] ...optimizers/trijoint.py.151: total_norm: 2.566972 [I 2022-10-26 15:06:09] ...trap/engines/engine.py.224: train: epoch 14 | batch 2180/2304 [I 2022-10-26 15:06:09] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:01:01 [I 2022-10-26 15:06:09] ...trap/engines/engine.py.229: process: 0.49610 | load: 0.00243 [I 2022-10-26 15:06:09] ...trap/engines/engine.py.230: loss: 0.16237 [I 2022-10-26 15:06:09] ...optimizers/trijoint.py.151: total_norm: 2.643180 [I 2022-10-26 15:06:14] ...trap/engines/engine.py.224: train: epoch 14 | batch 2190/2304 [I 2022-10-26 15:06:14] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:00:57 [I 2022-10-26 15:06:14] ...trap/engines/engine.py.229: process: 0.50640 | load: 0.00249 [I 2022-10-26 15:06:14] ...trap/engines/engine.py.230: loss: 0.20053 [I 2022-10-26 15:06:14] ...optimizers/trijoint.py.151: total_norm: 3.079986 [I 2022-10-26 15:06:19] ...trap/engines/engine.py.224: train: epoch 14 | batch 2200/2304 [I 2022-10-26 15:06:19] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:00:52 [I 2022-10-26 15:06:19] ...trap/engines/engine.py.229: process: 0.50649 | load: 0.00242 [I 2022-10-26 15:06:19] ...trap/engines/engine.py.230: loss: 0.24042 [I 2022-10-26 15:06:19] ...optimizers/trijoint.py.151: total_norm: 3.002069 [I 2022-10-26 15:06:24] ...trap/engines/engine.py.224: train: epoch 14 | batch 2210/2304 [I 2022-10-26 15:06:24] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:00:46 [I 2022-10-26 15:06:24] ...trap/engines/engine.py.229: process: 0.49435 | load: 0.00260 [I 2022-10-26 15:06:24] ...trap/engines/engine.py.230: loss: 0.23187 [I 2022-10-26 15:06:24] ...optimizers/trijoint.py.151: total_norm: 3.407525 [I 2022-10-26 15:06:29] ...trap/engines/engine.py.224: train: epoch 14 | batch 2220/2304 [I 2022-10-26 15:06:29] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:00:42 [I 2022-10-26 15:06:29] ...trap/engines/engine.py.229: process: 0.50082 | load: 0.00250 [I 2022-10-26 15:06:29] ...trap/engines/engine.py.230: loss: 0.24264 [I 2022-10-26 15:06:29] ...optimizers/trijoint.py.151: total_norm: 3.520920 [I 2022-10-26 15:06:34] ...trap/engines/engine.py.224: train: epoch 14 | batch 2230/2304 [I 2022-10-26 15:06:34] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:00:37 [I 2022-10-26 15:06:34] ...trap/engines/engine.py.229: process: 0.50089 | load: 0.00245 [I 2022-10-26 15:06:34] ...trap/engines/engine.py.230: loss: 0.38346 [I 2022-10-26 15:06:34] ...optimizers/trijoint.py.151: total_norm: 4.401347 [I 2022-10-26 15:06:39] ...trap/engines/engine.py.224: train: epoch 14 | batch 2240/2304 [I 2022-10-26 15:06:39] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:00:32 [I 2022-10-26 15:06:39] ...trap/engines/engine.py.229: process: 0.49925 | load: 0.00245 [I 2022-10-26 15:06:39] ...trap/engines/engine.py.230: loss: 0.24799 [I 2022-10-26 15:06:39] ...optimizers/trijoint.py.151: total_norm: 2.558186 [I 2022-10-26 15:06:44] ...trap/engines/engine.py.224: train: epoch 14 | batch 2250/2304 [I 2022-10-26 15:06:44] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:00:26 [I 2022-10-26 15:06:44] ...trap/engines/engine.py.229: process: 0.49403 | load: 0.00244 [I 2022-10-26 15:06:44] ...trap/engines/engine.py.230: loss: 0.22642 [I 2022-10-26 15:06:44] ...optimizers/trijoint.py.151: total_norm: 3.690869 [I 2022-10-26 15:06:49] ...trap/engines/engine.py.224: train: epoch 14 | batch 2260/2304 [I 2022-10-26 15:06:49] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:00:22 [I 2022-10-26 15:06:49] ...trap/engines/engine.py.229: process: 0.57707 | load: 0.00245 [I 2022-10-26 15:06:49] ...trap/engines/engine.py.230: loss: 0.18623 [I 2022-10-26 15:06:49] ...optimizers/trijoint.py.151: total_norm: 3.383888 [I 2022-10-26 15:06:54] ...trap/engines/engine.py.224: train: epoch 14 | batch 2270/2304 [I 2022-10-26 15:06:54] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:00:16 [I 2022-10-26 15:06:54] ...trap/engines/engine.py.229: process: 0.49377 | load: 0.00247 [I 2022-10-26 15:06:54] ...trap/engines/engine.py.230: loss: 0.15355 [I 2022-10-26 15:06:54] ...optimizers/trijoint.py.151: total_norm: 2.689142 [I 2022-10-26 15:06:59] ...trap/engines/engine.py.224: train: epoch 14 | batch 2280/2304 [I 2022-10-26 15:06:59] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:00:11 [I 2022-10-26 15:06:59] ...trap/engines/engine.py.229: process: 0.50481 | load: 0.00242 [I 2022-10-26 15:06:59] ...trap/engines/engine.py.230: loss: 0.22535 [I 2022-10-26 15:06:59] ...optimizers/trijoint.py.151: total_norm: 4.279567 [I 2022-10-26 15:07:04] ...trap/engines/engine.py.224: train: epoch 14 | batch 2290/2304 [I 2022-10-26 15:07:04] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:00:07 [I 2022-10-26 15:07:04] ...trap/engines/engine.py.229: process: 0.50014 | load: 0.00252 [I 2022-10-26 15:07:04] ...trap/engines/engine.py.230: loss: 0.31528 [I 2022-10-26 15:07:04] ...optimizers/trijoint.py.151: total_norm: 4.070132 [I 2022-10-26 15:07:09] ...trap/engines/engine.py.224: train: epoch 14 | batch 2300/2304 [I 2022-10-26 15:07:09] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:00:02 [I 2022-10-26 15:07:09] ...trap/engines/engine.py.229: process: 0.49558 | load: 0.00017 [I 2022-10-26 15:07:09] ...trap/engines/engine.py.230: loss: 0.19649 [I 2022-10-26 15:07:09] ...optimizers/trijoint.py.151: total_norm: 2.643064 [I 2022-10-26 15:07:11] ...trap/engines/engine.py.224: train: epoch 14 | batch 2304/2304 [I 2022-10-26 15:07:11] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:00:00 [I 2022-10-26 15:07:11] ...trap/engines/engine.py.229: process: 0.49310 | load: 0.00019 [I 2022-10-26 15:07:11] ...trap/engines/engine.py.230: loss: 0.33559 [I 2022-10-26 15:07:11] ...optimizers/trijoint.py.151: total_norm: 4.555643 [S 2022-10-26 15:07:13] ...trap/engines/engine.py.129: train_epoch.epoch: 14 [S 2022-10-26 15:07:13] ...trap/engines/engine.py.129: train_epoch.loss: 0.21862804930729876 [I 2022-10-26 15:07:13] ...s/criterions/triplet.py.84: margin = 0.12500000000000006 [I 2022-10-26 15:07:13] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 14 [I 2022-10-26 15:07:13] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 15:07:14] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 15:07:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 15:07:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 15:07:15] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 15:07:15] ...trap/engines/engine.py.321: eval: epoch 14 | batch 0/510 [I 2022-10-26 15:07:15] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:22:26 [I 2022-10-26 15:07:15] ...trap/engines/engine.py.326: process: 2.64110 | load: 2.23480 [I 2022-10-26 15:07:19] ...trap/engines/engine.py.321: eval: epoch 14 | batch 10/510 [I 2022-10-26 15:07:19] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:04:50 [I 2022-10-26 15:07:19] ...trap/engines/engine.py.326: process: 0.33809 | load: 0.00019 [I 2022-10-26 15:07:22] ...trap/engines/engine.py.321: eval: epoch 14 | batch 20/510 [I 2022-10-26 15:07:22] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:59 [I 2022-10-26 15:07:22] ...trap/engines/engine.py.326: process: 0.33619 | load: 0.00024 [I 2022-10-26 15:07:26] ...trap/engines/engine.py.321: eval: epoch 14 | batch 30/510 [I 2022-10-26 15:07:26] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:45 [I 2022-10-26 15:07:26] ...trap/engines/engine.py.326: process: 0.32880 | load: 0.00017 [I 2022-10-26 15:07:29] ...trap/engines/engine.py.321: eval: epoch 14 | batch 40/510 [I 2022-10-26 15:07:29] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:39 [I 2022-10-26 15:07:29] ...trap/engines/engine.py.326: process: 0.32799 | load: 0.00021 [I 2022-10-26 15:07:32] ...trap/engines/engine.py.321: eval: epoch 14 | batch 50/510 [I 2022-10-26 15:07:32] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-26 15:07:32] ...trap/engines/engine.py.326: process: 0.32902 | load: 0.00025 [I 2022-10-26 15:07:36] ...trap/engines/engine.py.321: eval: epoch 14 | batch 60/510 [I 2022-10-26 15:07:36] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:32 [I 2022-10-26 15:07:36] ...trap/engines/engine.py.326: process: 0.32544 | load: 0.00024 [I 2022-10-26 15:07:39] ...trap/engines/engine.py.321: eval: epoch 14 | batch 70/510 [I 2022-10-26 15:07:39] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:29 [I 2022-10-26 15:07:39] ...trap/engines/engine.py.326: process: 0.32609 | load: 0.00018 [I 2022-10-26 15:07:43] ...trap/engines/engine.py.321: eval: epoch 14 | batch 80/510 [I 2022-10-26 15:07:43] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:24 [I 2022-10-26 15:07:43] ...trap/engines/engine.py.326: process: 0.33384 | load: 0.00016 [I 2022-10-26 15:07:46] ...trap/engines/engine.py.321: eval: epoch 14 | batch 90/510 [I 2022-10-26 15:07:46] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-26 15:07:46] ...trap/engines/engine.py.326: process: 0.34304 | load: 0.00018 [I 2022-10-26 15:07:49] ...trap/engines/engine.py.321: eval: epoch 14 | batch 100/510 [I 2022-10-26 15:07:49] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:12 [I 2022-10-26 15:07:49] ...trap/engines/engine.py.326: process: 0.28331 | load: 0.00019 [I 2022-10-26 15:07:52] ...trap/engines/engine.py.321: eval: epoch 14 | batch 110/510 [I 2022-10-26 15:07:52] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:58 [I 2022-10-26 15:07:52] ...trap/engines/engine.py.326: process: 0.28717 | load: 0.00028 [I 2022-10-26 15:07:55] ...trap/engines/engine.py.321: eval: epoch 14 | batch 120/510 [I 2022-10-26 15:07:55] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 15:07:55] ...trap/engines/engine.py.326: process: 0.28746 | load: 0.00023 [I 2022-10-26 15:07:58] ...trap/engines/engine.py.321: eval: epoch 14 | batch 130/510 [I 2022-10-26 15:07:58] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 15:07:58] ...trap/engines/engine.py.326: process: 0.28944 | load: 0.00017 [I 2022-10-26 15:08:01] ...trap/engines/engine.py.321: eval: epoch 14 | batch 140/510 [I 2022-10-26 15:08:01] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:49 [I 2022-10-26 15:08:01] ...trap/engines/engine.py.326: process: 0.29018 | load: 0.00020 [I 2022-10-26 15:08:04] ...trap/engines/engine.py.321: eval: epoch 14 | batch 150/510 [I 2022-10-26 15:08:04] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 15:08:04] ...trap/engines/engine.py.326: process: 0.29245 | load: 0.00019 [I 2022-10-26 15:08:07] ...trap/engines/engine.py.321: eval: epoch 14 | batch 160/510 [I 2022-10-26 15:08:07] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 15:08:07] ...trap/engines/engine.py.326: process: 0.29057 | load: 0.00017 [I 2022-10-26 15:08:10] ...trap/engines/engine.py.321: eval: epoch 14 | batch 170/510 [I 2022-10-26 15:08:10] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 15:08:10] ...trap/engines/engine.py.326: process: 0.28659 | load: 0.00027 [I 2022-10-26 15:08:13] ...trap/engines/engine.py.321: eval: epoch 14 | batch 180/510 [I 2022-10-26 15:08:13] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 15:08:13] ...trap/engines/engine.py.326: process: 0.29062 | load: 0.00020 [I 2022-10-26 15:08:15] ...trap/engines/engine.py.321: eval: epoch 14 | batch 190/510 [I 2022-10-26 15:08:15] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 15:08:15] ...trap/engines/engine.py.326: process: 0.29193 | load: 0.00018 [I 2022-10-26 15:08:18] ...trap/engines/engine.py.321: eval: epoch 14 | batch 200/510 [I 2022-10-26 15:08:18] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 15:08:18] ...trap/engines/engine.py.326: process: 0.28424 | load: 0.00014 [I 2022-10-26 15:08:21] ...trap/engines/engine.py.321: eval: epoch 14 | batch 210/510 [I 2022-10-26 15:08:21] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-26 15:08:21] ...trap/engines/engine.py.326: process: 0.28847 | load: 0.00020 [I 2022-10-26 15:08:24] ...trap/engines/engine.py.321: eval: epoch 14 | batch 220/510 [I 2022-10-26 15:08:24] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 15:08:24] ...trap/engines/engine.py.326: process: 0.28103 | load: 0.00015 [I 2022-10-26 15:08:27] ...trap/engines/engine.py.321: eval: epoch 14 | batch 230/510 [I 2022-10-26 15:08:27] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 15:08:27] ...trap/engines/engine.py.326: process: 0.29051 | load: 0.00023 [I 2022-10-26 15:08:30] ...trap/engines/engine.py.321: eval: epoch 14 | batch 240/510 [I 2022-10-26 15:08:30] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 15:08:30] ...trap/engines/engine.py.326: process: 0.28934 | load: 0.00016 [I 2022-10-26 15:08:33] ...trap/engines/engine.py.321: eval: epoch 14 | batch 250/510 [I 2022-10-26 15:08:33] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-26 15:08:33] ...trap/engines/engine.py.326: process: 0.29128 | load: 0.00015 [I 2022-10-26 15:08:36] ...trap/engines/engine.py.321: eval: epoch 14 | batch 260/510 [I 2022-10-26 15:08:36] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:14 [I 2022-10-26 15:08:36] ...trap/engines/engine.py.326: process: 0.28886 | load: 0.00019 [I 2022-10-26 15:08:39] ...trap/engines/engine.py.321: eval: epoch 14 | batch 270/510 [I 2022-10-26 15:08:39] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 15:08:39] ...trap/engines/engine.py.326: process: 0.29189 | load: 0.00015 [I 2022-10-26 15:08:42] ...trap/engines/engine.py.321: eval: epoch 14 | batch 280/510 [I 2022-10-26 15:08:42] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:05 [I 2022-10-26 15:08:42] ...trap/engines/engine.py.326: process: 0.28248 | load: 0.00014 [I 2022-10-26 15:08:44] ...trap/engines/engine.py.321: eval: epoch 14 | batch 290/510 [I 2022-10-26 15:08:44] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 15:08:44] ...trap/engines/engine.py.326: process: 0.28899 | load: 0.00016 [I 2022-10-26 15:08:47] ...trap/engines/engine.py.321: eval: epoch 14 | batch 300/510 [I 2022-10-26 15:08:47] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 15:08:47] ...trap/engines/engine.py.326: process: 0.28636 | load: 0.00016 [I 2022-10-26 15:08:50] ...trap/engines/engine.py.321: eval: epoch 14 | batch 310/510 [I 2022-10-26 15:08:50] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 15:08:50] ...trap/engines/engine.py.326: process: 0.28665 | load: 0.00015 [I 2022-10-26 15:08:53] ...trap/engines/engine.py.321: eval: epoch 14 | batch 320/510 [I 2022-10-26 15:08:53] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 15:08:53] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00016 [I 2022-10-26 15:08:56] ...trap/engines/engine.py.321: eval: epoch 14 | batch 330/510 [I 2022-10-26 15:08:56] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 15:08:56] ...trap/engines/engine.py.326: process: 0.29142 | load: 0.00018 [I 2022-10-26 15:08:59] ...trap/engines/engine.py.321: eval: epoch 14 | batch 340/510 [I 2022-10-26 15:08:59] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 15:08:59] ...trap/engines/engine.py.326: process: 0.28692 | load: 0.00015 [I 2022-10-26 15:09:02] ...trap/engines/engine.py.321: eval: epoch 14 | batch 350/510 [I 2022-10-26 15:09:02] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 15:09:02] ...trap/engines/engine.py.326: process: 0.29166 | load: 0.00015 [I 2022-10-26 15:09:05] ...trap/engines/engine.py.321: eval: epoch 14 | batch 360/510 [I 2022-10-26 15:09:05] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:42 [I 2022-10-26 15:09:05] ...trap/engines/engine.py.326: process: 0.28512 | load: 0.00015 [I 2022-10-26 15:09:07] ...trap/engines/engine.py.321: eval: epoch 14 | batch 370/510 [I 2022-10-26 15:09:07] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 15:09:07] ...trap/engines/engine.py.326: process: 0.28562 | load: 0.00018 [I 2022-10-26 15:09:10] ...trap/engines/engine.py.321: eval: epoch 14 | batch 380/510 [I 2022-10-26 15:09:10] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 15:09:10] ...trap/engines/engine.py.326: process: 0.29252 | load: 0.00015 [I 2022-10-26 15:09:13] ...trap/engines/engine.py.321: eval: epoch 14 | batch 390/510 [I 2022-10-26 15:09:13] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:35 [I 2022-10-26 15:09:13] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00020 [I 2022-10-26 15:09:16] ...trap/engines/engine.py.321: eval: epoch 14 | batch 400/510 [I 2022-10-26 15:09:16] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 15:09:16] ...trap/engines/engine.py.326: process: 0.28558 | load: 0.00018 [I 2022-10-26 15:09:19] ...trap/engines/engine.py.321: eval: epoch 14 | batch 410/510 [I 2022-10-26 15:09:19] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 15:09:19] ...trap/engines/engine.py.326: process: 0.27734 | load: 0.00015 [I 2022-10-26 15:09:22] ...trap/engines/engine.py.321: eval: epoch 14 | batch 420/510 [I 2022-10-26 15:09:22] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 15:09:22] ...trap/engines/engine.py.326: process: 0.28874 | load: 0.00016 [I 2022-10-26 15:09:25] ...trap/engines/engine.py.321: eval: epoch 14 | batch 430/510 [I 2022-10-26 15:09:25] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-26 15:09:25] ...trap/engines/engine.py.326: process: 0.29002 | load: 0.00016 [I 2022-10-26 15:09:28] ...trap/engines/engine.py.321: eval: epoch 14 | batch 440/510 [I 2022-10-26 15:09:28] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 15:09:28] ...trap/engines/engine.py.326: process: 0.27888 | load: 0.00022 [I 2022-10-26 15:09:31] ...trap/engines/engine.py.321: eval: epoch 14 | batch 450/510 [I 2022-10-26 15:09:31] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 15:09:31] ...trap/engines/engine.py.326: process: 0.29091 | load: 0.00018 [I 2022-10-26 15:09:33] ...trap/engines/engine.py.321: eval: epoch 14 | batch 460/510 [I 2022-10-26 15:09:33] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 15:09:33] ...trap/engines/engine.py.326: process: 0.28605 | load: 0.00035 [I 2022-10-26 15:09:36] ...trap/engines/engine.py.321: eval: epoch 14 | batch 470/510 [I 2022-10-26 15:09:36] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 15:09:36] ...trap/engines/engine.py.326: process: 0.28880 | load: 0.00017 [I 2022-10-26 15:09:39] ...trap/engines/engine.py.321: eval: epoch 14 | batch 480/510 [I 2022-10-26 15:09:39] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 15:09:39] ...trap/engines/engine.py.326: process: 0.28792 | load: 0.00015 [I 2022-10-26 15:09:42] ...trap/engines/engine.py.321: eval: epoch 14 | batch 490/510 [I 2022-10-26 15:09:42] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 15:09:42] ...trap/engines/engine.py.326: process: 0.28281 | load: 0.00015 [I 2022-10-26 15:09:45] ...trap/engines/engine.py.321: eval: epoch 14 | batch 500/510 [I 2022-10-26 15:09:45] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 15:09:45] ...trap/engines/engine.py.326: process: 0.29304 | load: 0.00015 [I 2022-10-26 15:09:48] ...trap/engines/engine.py.321: eval: epoch 14 | batch 510/510 [I 2022-10-26 15:09:48] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 15:09:48] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00011 [S 2022-10-26 15:09:49] ...trap/engines/engine.py.132: eval_epoch.epoch: 14 [I 2022-10-26 15:09:49] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.548000 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.822100 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.888000 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.548200 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.826300 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.887400 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012522 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011113 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006885 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017099 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005120 [S 2022-10-26 15:09:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007378 [I 2022-10-26 15:09:54] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 15:09:54] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 15:09:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 15:09:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 15:09:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 15:09:54] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 15:09:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 15:10:02] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 15:10:02] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 15:10:02] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 15:10:05] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 15:10:09] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 15:10:09] ...trap/engines/engine.py.163: Training model on trainset for epoch 15 [I 2022-10-26 15:10:09] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 15:10:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 0/2304 [I 2022-10-26 15:10:12] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:09 [I 2022-10-26 15:10:12] ...trap/engines/engine.py.229: process: 2.71228 | load: 2.15223 [I 2022-10-26 15:10:12] ...trap/engines/engine.py.230: loss: 0.22950 [I 2022-10-26 15:10:12] ...optimizers/trijoint.py.151: total_norm: 3.291054 [I 2022-10-26 15:10:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 10/2304 [I 2022-10-26 15:10:17] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:17 [I 2022-10-26 15:10:17] ...trap/engines/engine.py.229: process: 0.50459 | load: 0.00404 [I 2022-10-26 15:10:17] ...trap/engines/engine.py.230: loss: 0.18334 [I 2022-10-26 15:10:17] ...optimizers/trijoint.py.151: total_norm: 2.983262 [I 2022-10-26 15:10:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 20/2304 [I 2022-10-26 15:10:22] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:06 [I 2022-10-26 15:10:22] ...trap/engines/engine.py.229: process: 0.50609 | load: 0.00347 [I 2022-10-26 15:10:22] ...trap/engines/engine.py.230: loss: 0.20730 [I 2022-10-26 15:10:22] ...optimizers/trijoint.py.151: total_norm: 2.725586 [I 2022-10-26 15:10:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 30/2304 [I 2022-10-26 15:10:27] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:09 [I 2022-10-26 15:10:27] ...trap/engines/engine.py.229: process: 0.51191 | load: 0.00361 [I 2022-10-26 15:10:27] ...trap/engines/engine.py.230: loss: 0.25617 [I 2022-10-26 15:10:27] ...optimizers/trijoint.py.151: total_norm: 2.730183 [I 2022-10-26 15:10:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 40/2304 [I 2022-10-26 15:10:32] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:49 [I 2022-10-26 15:10:32] ...trap/engines/engine.py.229: process: 0.49828 | load: 0.00312 [I 2022-10-26 15:10:32] ...trap/engines/engine.py.230: loss: 0.11988 [I 2022-10-26 15:10:32] ...optimizers/trijoint.py.151: total_norm: 2.766191 [I 2022-10-26 15:10:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 50/2304 [I 2022-10-26 15:10:37] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:19:02 [I 2022-10-26 15:10:37] ...trap/engines/engine.py.229: process: 0.50394 | load: 0.00341 [I 2022-10-26 15:10:37] ...trap/engines/engine.py.230: loss: 0.26856 [I 2022-10-26 15:10:37] ...optimizers/trijoint.py.151: total_norm: 4.162185 [I 2022-10-26 15:10:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 60/2304 [I 2022-10-26 15:10:42] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:42 [I 2022-10-26 15:10:42] ...trap/engines/engine.py.229: process: 0.49633 | load: 0.00292 [I 2022-10-26 15:10:42] ...trap/engines/engine.py.230: loss: 0.19259 [I 2022-10-26 15:10:42] ...optimizers/trijoint.py.151: total_norm: 2.514981 [I 2022-10-26 15:10:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 70/2304 [I 2022-10-26 15:10:47] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:36 [I 2022-10-26 15:10:47] ...trap/engines/engine.py.229: process: 0.50690 | load: 0.00310 [I 2022-10-26 15:10:47] ...trap/engines/engine.py.230: loss: 0.22946 [I 2022-10-26 15:10:47] ...optimizers/trijoint.py.151: total_norm: 3.185149 [I 2022-10-26 15:10:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 80/2304 [I 2022-10-26 15:10:52] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:39 [I 2022-10-26 15:10:52] ...trap/engines/engine.py.229: process: 0.50500 | load: 0.00329 [I 2022-10-26 15:10:52] ...trap/engines/engine.py.230: loss: 0.18050 [I 2022-10-26 15:10:52] ...optimizers/trijoint.py.151: total_norm: 3.221338 [I 2022-10-26 15:10:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 90/2304 [I 2022-10-26 15:10:57] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:18:29 [I 2022-10-26 15:10:57] ...trap/engines/engine.py.229: process: 0.49417 | load: 0.00304 [I 2022-10-26 15:10:57] ...trap/engines/engine.py.230: loss: 0.29389 [I 2022-10-26 15:10:57] ...optimizers/trijoint.py.151: total_norm: 3.560907 [I 2022-10-26 15:11:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 100/2304 [I 2022-10-26 15:11:02] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:18:16 [I 2022-10-26 15:11:02] ...trap/engines/engine.py.229: process: 0.49093 | load: 0.00319 [I 2022-10-26 15:11:02] ...trap/engines/engine.py.230: loss: 0.28319 [I 2022-10-26 15:11:02] ...optimizers/trijoint.py.151: total_norm: 3.417145 [I 2022-10-26 15:11:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 110/2304 [I 2022-10-26 15:11:07] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:18:18 [I 2022-10-26 15:11:07] ...trap/engines/engine.py.229: process: 0.50718 | load: 0.00313 [I 2022-10-26 15:11:07] ...trap/engines/engine.py.230: loss: 0.16863 [I 2022-10-26 15:11:07] ...optimizers/trijoint.py.151: total_norm: 2.645849 [I 2022-10-26 15:11:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 120/2304 [I 2022-10-26 15:11:12] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:18:08 [I 2022-10-26 15:11:12] ...trap/engines/engine.py.229: process: 0.49692 | load: 0.00281 [I 2022-10-26 15:11:12] ...trap/engines/engine.py.230: loss: 0.20696 [I 2022-10-26 15:11:12] ...optimizers/trijoint.py.151: total_norm: 4.129245 [I 2022-10-26 15:11:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 130/2304 [I 2022-10-26 15:11:17] ...trap/engines/engine.py.228: elapsed: 0:01:07 | left: 0:18:03 [I 2022-10-26 15:11:17] ...trap/engines/engine.py.229: process: 0.49705 | load: 0.00283 [I 2022-10-26 15:11:17] ...trap/engines/engine.py.230: loss: 0.16642 [I 2022-10-26 15:11:17] ...optimizers/trijoint.py.151: total_norm: 2.609097 [I 2022-10-26 15:11:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 140/2304 [I 2022-10-26 15:11:22] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:18:03 [I 2022-10-26 15:11:22] ...trap/engines/engine.py.229: process: 0.50204 | load: 0.00280 [I 2022-10-26 15:11:22] ...trap/engines/engine.py.230: loss: 0.20872 [I 2022-10-26 15:11:22] ...optimizers/trijoint.py.151: total_norm: 2.947884 [I 2022-10-26 15:11:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 150/2304 [I 2022-10-26 15:11:27] ...trap/engines/engine.py.228: elapsed: 0:01:17 | left: 0:18:01 [I 2022-10-26 15:11:27] ...trap/engines/engine.py.229: process: 0.51765 | load: 0.00265 [I 2022-10-26 15:11:27] ...trap/engines/engine.py.230: loss: 0.22140 [I 2022-10-26 15:11:27] ...optimizers/trijoint.py.151: total_norm: 3.988865 [I 2022-10-26 15:11:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 160/2304 [I 2022-10-26 15:11:32] ...trap/engines/engine.py.228: elapsed: 0:01:22 | left: 0:17:58 [I 2022-10-26 15:11:32] ...trap/engines/engine.py.229: process: 0.50682 | load: 0.00263 [I 2022-10-26 15:11:32] ...trap/engines/engine.py.230: loss: 0.28906 [I 2022-10-26 15:11:32] ...optimizers/trijoint.py.151: total_norm: 4.530830 [I 2022-10-26 15:11:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 170/2304 [I 2022-10-26 15:11:37] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:17:50 [I 2022-10-26 15:11:37] ...trap/engines/engine.py.229: process: 0.50345 | load: 0.00264 [I 2022-10-26 15:11:37] ...trap/engines/engine.py.230: loss: 0.17187 [I 2022-10-26 15:11:37] ...optimizers/trijoint.py.151: total_norm: 2.995312 [I 2022-10-26 15:11:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 180/2304 [I 2022-10-26 15:11:42] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:17:34 [I 2022-10-26 15:11:42] ...trap/engines/engine.py.229: process: 0.49067 | load: 0.00253 [I 2022-10-26 15:11:42] ...trap/engines/engine.py.230: loss: 0.23597 [I 2022-10-26 15:11:42] ...optimizers/trijoint.py.151: total_norm: 2.102649 [I 2022-10-26 15:11:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 190/2304 [I 2022-10-26 15:11:47] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:17:35 [I 2022-10-26 15:11:47] ...trap/engines/engine.py.229: process: 0.49485 | load: 0.00290 [I 2022-10-26 15:11:47] ...trap/engines/engine.py.230: loss: 0.19609 [I 2022-10-26 15:11:47] ...optimizers/trijoint.py.151: total_norm: 2.735662 [I 2022-10-26 15:11:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 200/2304 [I 2022-10-26 15:11:52] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:17:27 [I 2022-10-26 15:11:52] ...trap/engines/engine.py.229: process: 0.50213 | load: 0.00256 [I 2022-10-26 15:11:52] ...trap/engines/engine.py.230: loss: 0.14886 [I 2022-10-26 15:11:52] ...optimizers/trijoint.py.151: total_norm: 2.167123 [I 2022-10-26 15:11:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 210/2304 [I 2022-10-26 15:11:57] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:17:19 [I 2022-10-26 15:11:57] ...trap/engines/engine.py.229: process: 0.49704 | load: 0.00257 [I 2022-10-26 15:11:57] ...trap/engines/engine.py.230: loss: 0.16528 [I 2022-10-26 15:11:57] ...optimizers/trijoint.py.151: total_norm: 2.513516 [I 2022-10-26 15:12:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 220/2304 [I 2022-10-26 15:12:02] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:17:05 [I 2022-10-26 15:12:02] ...trap/engines/engine.py.229: process: 0.49420 | load: 0.00259 [I 2022-10-26 15:12:02] ...trap/engines/engine.py.230: loss: 0.22678 [I 2022-10-26 15:12:02] ...optimizers/trijoint.py.151: total_norm: 2.441813 [I 2022-10-26 15:12:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 230/2304 [I 2022-10-26 15:12:07] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:17:09 [I 2022-10-26 15:12:07] ...trap/engines/engine.py.229: process: 0.49180 | load: 0.00247 [I 2022-10-26 15:12:07] ...trap/engines/engine.py.230: loss: 0.25355 [I 2022-10-26 15:12:07] ...optimizers/trijoint.py.151: total_norm: 3.667456 [I 2022-10-26 15:12:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 240/2304 [I 2022-10-26 15:12:12] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:17:06 [I 2022-10-26 15:12:12] ...trap/engines/engine.py.229: process: 0.49874 | load: 0.00257 [I 2022-10-26 15:12:12] ...trap/engines/engine.py.230: loss: 0.17688 [I 2022-10-26 15:12:12] ...optimizers/trijoint.py.151: total_norm: 2.947253 [I 2022-10-26 15:12:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 250/2304 [I 2022-10-26 15:12:17] ...trap/engines/engine.py.228: elapsed: 0:02:07 | left: 0:17:06 [I 2022-10-26 15:12:17] ...trap/engines/engine.py.229: process: 0.50567 | load: 0.00254 [I 2022-10-26 15:12:17] ...trap/engines/engine.py.230: loss: 0.17889 [I 2022-10-26 15:12:17] ...optimizers/trijoint.py.151: total_norm: 2.836694 [I 2022-10-26 15:12:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 260/2304 [I 2022-10-26 15:12:22] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:17:02 [I 2022-10-26 15:12:22] ...trap/engines/engine.py.229: process: 0.49256 | load: 0.00253 [I 2022-10-26 15:12:22] ...trap/engines/engine.py.230: loss: 0.15463 [I 2022-10-26 15:12:22] ...optimizers/trijoint.py.151: total_norm: 2.450842 [I 2022-10-26 15:12:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 270/2304 [I 2022-10-26 15:12:27] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:16:56 [I 2022-10-26 15:12:27] ...trap/engines/engine.py.229: process: 0.48759 | load: 0.00263 [I 2022-10-26 15:12:27] ...trap/engines/engine.py.230: loss: 0.21852 [I 2022-10-26 15:12:27] ...optimizers/trijoint.py.151: total_norm: 3.189595 [I 2022-10-26 15:12:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 280/2304 [I 2022-10-26 15:12:32] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:16:53 [I 2022-10-26 15:12:32] ...trap/engines/engine.py.229: process: 0.49672 | load: 0.00257 [I 2022-10-26 15:12:32] ...trap/engines/engine.py.230: loss: 0.23614 [I 2022-10-26 15:12:32] ...optimizers/trijoint.py.151: total_norm: 4.370931 [I 2022-10-26 15:12:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 290/2304 [I 2022-10-26 15:12:37] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:16:48 [I 2022-10-26 15:12:37] ...trap/engines/engine.py.229: process: 0.51247 | load: 0.00252 [I 2022-10-26 15:12:37] ...trap/engines/engine.py.230: loss: 0.12444 [I 2022-10-26 15:12:37] ...optimizers/trijoint.py.151: total_norm: 3.700857 [I 2022-10-26 15:12:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 300/2304 [I 2022-10-26 15:12:42] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:16:34 [I 2022-10-26 15:12:42] ...trap/engines/engine.py.229: process: 0.49676 | load: 0.00248 [I 2022-10-26 15:12:42] ...trap/engines/engine.py.230: loss: 0.24658 [I 2022-10-26 15:12:42] ...optimizers/trijoint.py.151: total_norm: 4.021628 [I 2022-10-26 15:12:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 310/2304 [I 2022-10-26 15:12:47] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:16:27 [I 2022-10-26 15:12:47] ...trap/engines/engine.py.229: process: 0.48721 | load: 0.00257 [I 2022-10-26 15:12:47] ...trap/engines/engine.py.230: loss: 0.14178 [I 2022-10-26 15:12:47] ...optimizers/trijoint.py.151: total_norm: 2.649949 [I 2022-10-26 15:12:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 320/2304 [I 2022-10-26 15:12:52] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:16:33 [I 2022-10-26 15:12:52] ...trap/engines/engine.py.229: process: 0.50198 | load: 0.00260 [I 2022-10-26 15:12:52] ...trap/engines/engine.py.230: loss: 0.25492 [I 2022-10-26 15:12:52] ...optimizers/trijoint.py.151: total_norm: 2.807448 [I 2022-10-26 15:12:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 330/2304 [I 2022-10-26 15:12:57] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:16:34 [I 2022-10-26 15:12:57] ...trap/engines/engine.py.229: process: 0.51613 | load: 0.00259 [I 2022-10-26 15:12:57] ...trap/engines/engine.py.230: loss: 0.19794 [I 2022-10-26 15:12:57] ...optimizers/trijoint.py.151: total_norm: 2.903656 [I 2022-10-26 15:13:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 340/2304 [I 2022-10-26 15:13:02] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:16:23 [I 2022-10-26 15:13:02] ...trap/engines/engine.py.229: process: 0.49657 | load: 0.00257 [I 2022-10-26 15:13:02] ...trap/engines/engine.py.230: loss: 0.20134 [I 2022-10-26 15:13:02] ...optimizers/trijoint.py.151: total_norm: 3.737956 [I 2022-10-26 15:13:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 350/2304 [I 2022-10-26 15:13:07] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:16:10 [I 2022-10-26 15:13:07] ...trap/engines/engine.py.229: process: 0.49916 | load: 0.00270 [I 2022-10-26 15:13:07] ...trap/engines/engine.py.230: loss: 0.20344 [I 2022-10-26 15:13:07] ...optimizers/trijoint.py.151: total_norm: 2.966031 [I 2022-10-26 15:13:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 360/2304 [I 2022-10-26 15:13:12] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:16:09 [I 2022-10-26 15:13:12] ...trap/engines/engine.py.229: process: 0.50436 | load: 0.00254 [I 2022-10-26 15:13:12] ...trap/engines/engine.py.230: loss: 0.19958 [I 2022-10-26 15:13:12] ...optimizers/trijoint.py.151: total_norm: 3.722477 [I 2022-10-26 15:13:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 370/2304 [I 2022-10-26 15:13:17] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:16:00 [I 2022-10-26 15:13:17] ...trap/engines/engine.py.229: process: 0.49653 | load: 0.00257 [I 2022-10-26 15:13:17] ...trap/engines/engine.py.230: loss: 0.26137 [I 2022-10-26 15:13:17] ...optimizers/trijoint.py.151: total_norm: 3.196264 [I 2022-10-26 15:13:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 380/2304 [I 2022-10-26 15:13:22] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:15:53 [I 2022-10-26 15:13:22] ...trap/engines/engine.py.229: process: 0.48405 | load: 0.00265 [I 2022-10-26 15:13:22] ...trap/engines/engine.py.230: loss: 0.22541 [I 2022-10-26 15:13:22] ...optimizers/trijoint.py.151: total_norm: 3.693286 [I 2022-10-26 15:13:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 390/2304 [I 2022-10-26 15:13:27] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:15:55 [I 2022-10-26 15:13:27] ...trap/engines/engine.py.229: process: 0.49674 | load: 0.00262 [I 2022-10-26 15:13:27] ...trap/engines/engine.py.230: loss: 0.25652 [I 2022-10-26 15:13:27] ...optimizers/trijoint.py.151: total_norm: 4.100194 [I 2022-10-26 15:13:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 400/2304 [I 2022-10-26 15:13:32] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:15:49 [I 2022-10-26 15:13:32] ...trap/engines/engine.py.229: process: 0.50261 | load: 0.00255 [I 2022-10-26 15:13:32] ...trap/engines/engine.py.230: loss: 0.24485 [I 2022-10-26 15:13:32] ...optimizers/trijoint.py.151: total_norm: 3.422312 [I 2022-10-26 15:13:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 410/2304 [I 2022-10-26 15:13:37] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:15:47 [I 2022-10-26 15:13:37] ...trap/engines/engine.py.229: process: 0.49850 | load: 0.00249 [I 2022-10-26 15:13:37] ...trap/engines/engine.py.230: loss: 0.16800 [I 2022-10-26 15:13:37] ...optimizers/trijoint.py.151: total_norm: 2.422184 [I 2022-10-26 15:13:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 420/2304 [I 2022-10-26 15:13:42] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:15:38 [I 2022-10-26 15:13:42] ...trap/engines/engine.py.229: process: 0.50575 | load: 0.00270 [I 2022-10-26 15:13:42] ...trap/engines/engine.py.230: loss: 0.15036 [I 2022-10-26 15:13:42] ...optimizers/trijoint.py.151: total_norm: 3.000742 [I 2022-10-26 15:13:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 430/2304 [I 2022-10-26 15:13:47] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:15:30 [I 2022-10-26 15:13:47] ...trap/engines/engine.py.229: process: 0.47832 | load: 0.00244 [I 2022-10-26 15:13:47] ...trap/engines/engine.py.230: loss: 0.15172 [I 2022-10-26 15:13:47] ...optimizers/trijoint.py.151: total_norm: 3.118160 [I 2022-10-26 15:13:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 440/2304 [I 2022-10-26 15:13:52] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:15:26 [I 2022-10-26 15:13:52] ...trap/engines/engine.py.229: process: 0.50143 | load: 0.00252 [I 2022-10-26 15:13:52] ...trap/engines/engine.py.230: loss: 0.23605 [I 2022-10-26 15:13:52] ...optimizers/trijoint.py.151: total_norm: 3.565802 [I 2022-10-26 15:13:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 450/2304 [I 2022-10-26 15:13:57] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:15:17 [I 2022-10-26 15:13:57] ...trap/engines/engine.py.229: process: 0.49312 | load: 0.00251 [I 2022-10-26 15:13:57] ...trap/engines/engine.py.230: loss: 0.22081 [I 2022-10-26 15:13:57] ...optimizers/trijoint.py.151: total_norm: 4.450666 [I 2022-10-26 15:14:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 460/2304 [I 2022-10-26 15:14:02] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:15:16 [I 2022-10-26 15:14:02] ...trap/engines/engine.py.229: process: 0.49879 | load: 0.00250 [I 2022-10-26 15:14:02] ...trap/engines/engine.py.230: loss: 0.20693 [I 2022-10-26 15:14:02] ...optimizers/trijoint.py.151: total_norm: 4.033801 [I 2022-10-26 15:14:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 470/2304 [I 2022-10-26 15:14:07] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:15:13 [I 2022-10-26 15:14:07] ...trap/engines/engine.py.229: process: 0.49041 | load: 0.00249 [I 2022-10-26 15:14:07] ...trap/engines/engine.py.230: loss: 0.21562 [I 2022-10-26 15:14:07] ...optimizers/trijoint.py.151: total_norm: 4.029224 [I 2022-10-26 15:14:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 480/2304 [I 2022-10-26 15:14:12] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:15:09 [I 2022-10-26 15:14:12] ...trap/engines/engine.py.229: process: 0.49628 | load: 0.00247 [I 2022-10-26 15:14:12] ...trap/engines/engine.py.230: loss: 0.17188 [I 2022-10-26 15:14:12] ...optimizers/trijoint.py.151: total_norm: 3.278651 [I 2022-10-26 15:14:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 490/2304 [I 2022-10-26 15:14:17] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:15:01 [I 2022-10-26 15:14:17] ...trap/engines/engine.py.229: process: 0.50851 | load: 0.00250 [I 2022-10-26 15:14:17] ...trap/engines/engine.py.230: loss: 0.10051 [I 2022-10-26 15:14:17] ...optimizers/trijoint.py.151: total_norm: 1.855782 [I 2022-10-26 15:14:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 500/2304 [I 2022-10-26 15:14:22] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:15:02 [I 2022-10-26 15:14:22] ...trap/engines/engine.py.229: process: 0.49722 | load: 0.00246 [I 2022-10-26 15:14:22] ...trap/engines/engine.py.230: loss: 0.17499 [I 2022-10-26 15:14:22] ...optimizers/trijoint.py.151: total_norm: 2.845217 [I 2022-10-26 15:14:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 510/2304 [I 2022-10-26 15:14:27] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:15:00 [I 2022-10-26 15:14:27] ...trap/engines/engine.py.229: process: 0.50637 | load: 0.00258 [I 2022-10-26 15:14:27] ...trap/engines/engine.py.230: loss: 0.23757 [I 2022-10-26 15:14:27] ...optimizers/trijoint.py.151: total_norm: 4.436759 [I 2022-10-26 15:14:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 520/2304 [I 2022-10-26 15:14:32] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:14:52 [I 2022-10-26 15:14:32] ...trap/engines/engine.py.229: process: 0.50158 | load: 0.00245 [I 2022-10-26 15:14:32] ...trap/engines/engine.py.230: loss: 0.21645 [I 2022-10-26 15:14:32] ...optimizers/trijoint.py.151: total_norm: 2.835537 [I 2022-10-26 15:14:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 530/2304 [I 2022-10-26 15:14:37] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:14:37 [I 2022-10-26 15:14:37] ...trap/engines/engine.py.229: process: 0.48626 | load: 0.00252 [I 2022-10-26 15:14:37] ...trap/engines/engine.py.230: loss: 0.20239 [I 2022-10-26 15:14:37] ...optimizers/trijoint.py.151: total_norm: 2.459300 [I 2022-10-26 15:14:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 540/2304 [I 2022-10-26 15:14:42] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:14:35 [I 2022-10-26 15:14:42] ...trap/engines/engine.py.229: process: 0.49532 | load: 0.00252 [I 2022-10-26 15:14:42] ...trap/engines/engine.py.230: loss: 0.20588 [I 2022-10-26 15:14:42] ...optimizers/trijoint.py.151: total_norm: 3.166694 [I 2022-10-26 15:14:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 550/2304 [I 2022-10-26 15:14:47] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:14:32 [I 2022-10-26 15:14:47] ...trap/engines/engine.py.229: process: 0.50033 | load: 0.00247 [I 2022-10-26 15:14:47] ...trap/engines/engine.py.230: loss: 0.20316 [I 2022-10-26 15:14:47] ...optimizers/trijoint.py.151: total_norm: 4.366522 [I 2022-10-26 15:14:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 560/2304 [I 2022-10-26 15:14:52] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:14:24 [I 2022-10-26 15:14:52] ...trap/engines/engine.py.229: process: 0.50143 | load: 0.00249 [I 2022-10-26 15:14:52] ...trap/engines/engine.py.230: loss: 0.31750 [I 2022-10-26 15:14:52] ...optimizers/trijoint.py.151: total_norm: 4.176308 [I 2022-10-26 15:14:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 570/2304 [I 2022-10-26 15:14:57] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:14:22 [I 2022-10-26 15:14:57] ...trap/engines/engine.py.229: process: 0.49676 | load: 0.00247 [I 2022-10-26 15:14:57] ...trap/engines/engine.py.230: loss: 0.26312 [I 2022-10-26 15:14:57] ...optimizers/trijoint.py.151: total_norm: 3.180794 [I 2022-10-26 15:15:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 580/2304 [I 2022-10-26 15:15:02] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:14:19 [I 2022-10-26 15:15:02] ...trap/engines/engine.py.229: process: 0.50185 | load: 0.00242 [I 2022-10-26 15:15:02] ...trap/engines/engine.py.230: loss: 0.22044 [I 2022-10-26 15:15:02] ...optimizers/trijoint.py.151: total_norm: 2.765876 [I 2022-10-26 15:15:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 590/2304 [I 2022-10-26 15:15:07] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:14:27 [I 2022-10-26 15:15:07] ...trap/engines/engine.py.229: process: 0.51829 | load: 0.00274 [I 2022-10-26 15:15:07] ...trap/engines/engine.py.230: loss: 0.22782 [I 2022-10-26 15:15:07] ...optimizers/trijoint.py.151: total_norm: 4.634067 [I 2022-10-26 15:15:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 600/2304 [I 2022-10-26 15:15:12] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:14:12 [I 2022-10-26 15:15:12] ...trap/engines/engine.py.229: process: 0.50043 | load: 0.00249 [I 2022-10-26 15:15:12] ...trap/engines/engine.py.230: loss: 0.18521 [I 2022-10-26 15:15:12] ...optimizers/trijoint.py.151: total_norm: 3.336843 [I 2022-10-26 15:15:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 610/2304 [I 2022-10-26 15:15:17] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:14:03 [I 2022-10-26 15:15:17] ...trap/engines/engine.py.229: process: 0.49856 | load: 0.00258 [I 2022-10-26 15:15:17] ...trap/engines/engine.py.230: loss: 0.20228 [I 2022-10-26 15:15:17] ...optimizers/trijoint.py.151: total_norm: 4.038107 [I 2022-10-26 15:15:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 620/2304 [I 2022-10-26 15:15:22] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:14:09 [I 2022-10-26 15:15:22] ...trap/engines/engine.py.229: process: 0.50287 | load: 0.00257 [I 2022-10-26 15:15:22] ...trap/engines/engine.py.230: loss: 0.16906 [I 2022-10-26 15:15:22] ...optimizers/trijoint.py.151: total_norm: 3.683250 [I 2022-10-26 15:15:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 630/2304 [I 2022-10-26 15:15:27] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:14:03 [I 2022-10-26 15:15:27] ...trap/engines/engine.py.229: process: 0.50236 | load: 0.00250 [I 2022-10-26 15:15:27] ...trap/engines/engine.py.230: loss: 0.27178 [I 2022-10-26 15:15:27] ...optimizers/trijoint.py.151: total_norm: 4.322111 [I 2022-10-26 15:15:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 640/2304 [I 2022-10-26 15:15:32] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:13:54 [I 2022-10-26 15:15:32] ...trap/engines/engine.py.229: process: 0.49493 | load: 0.00283 [I 2022-10-26 15:15:32] ...trap/engines/engine.py.230: loss: 0.27273 [I 2022-10-26 15:15:32] ...optimizers/trijoint.py.151: total_norm: 3.799156 [I 2022-10-26 15:15:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 650/2304 [I 2022-10-26 15:15:37] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:13:46 [I 2022-10-26 15:15:37] ...trap/engines/engine.py.229: process: 0.49445 | load: 0.00247 [I 2022-10-26 15:15:37] ...trap/engines/engine.py.230: loss: 0.16744 [I 2022-10-26 15:15:37] ...optimizers/trijoint.py.151: total_norm: 3.115021 [I 2022-10-26 15:15:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 660/2304 [I 2022-10-26 15:15:42] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:13:38 [I 2022-10-26 15:15:42] ...trap/engines/engine.py.229: process: 0.50433 | load: 0.00251 [I 2022-10-26 15:15:42] ...trap/engines/engine.py.230: loss: 0.15878 [I 2022-10-26 15:15:42] ...optimizers/trijoint.py.151: total_norm: 2.156625 [I 2022-10-26 15:15:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 670/2304 [I 2022-10-26 15:15:47] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:13:33 [I 2022-10-26 15:15:47] ...trap/engines/engine.py.229: process: 0.50098 | load: 0.00248 [I 2022-10-26 15:15:47] ...trap/engines/engine.py.230: loss: 0.35606 [I 2022-10-26 15:15:47] ...optimizers/trijoint.py.151: total_norm: 3.765441 [I 2022-10-26 15:15:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 680/2304 [I 2022-10-26 15:15:52] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:13:24 [I 2022-10-26 15:15:52] ...trap/engines/engine.py.229: process: 0.49244 | load: 0.00247 [I 2022-10-26 15:15:52] ...trap/engines/engine.py.230: loss: 0.23601 [I 2022-10-26 15:15:52] ...optimizers/trijoint.py.151: total_norm: 3.565661 [I 2022-10-26 15:15:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 690/2304 [I 2022-10-26 15:15:57] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:13:34 [I 2022-10-26 15:15:57] ...trap/engines/engine.py.229: process: 0.50518 | load: 0.00251 [I 2022-10-26 15:15:57] ...trap/engines/engine.py.230: loss: 0.20264 [I 2022-10-26 15:15:57] ...optimizers/trijoint.py.151: total_norm: 3.423213 [I 2022-10-26 15:16:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 700/2304 [I 2022-10-26 15:16:02] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:13:19 [I 2022-10-26 15:16:02] ...trap/engines/engine.py.229: process: 0.50037 | load: 0.00254 [I 2022-10-26 15:16:02] ...trap/engines/engine.py.230: loss: 0.14845 [I 2022-10-26 15:16:02] ...optimizers/trijoint.py.151: total_norm: 2.201298 [I 2022-10-26 15:16:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 710/2304 [I 2022-10-26 15:16:07] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:13:17 [I 2022-10-26 15:16:07] ...trap/engines/engine.py.229: process: 0.49693 | load: 0.00254 [I 2022-10-26 15:16:07] ...trap/engines/engine.py.230: loss: 0.20873 [I 2022-10-26 15:16:07] ...optimizers/trijoint.py.151: total_norm: 2.994199 [I 2022-10-26 15:16:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 720/2304 [I 2022-10-26 15:16:12] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:13:14 [I 2022-10-26 15:16:12] ...trap/engines/engine.py.229: process: 0.49069 | load: 0.00261 [I 2022-10-26 15:16:12] ...trap/engines/engine.py.230: loss: 0.17957 [I 2022-10-26 15:16:12] ...optimizers/trijoint.py.151: total_norm: 2.388418 [I 2022-10-26 15:16:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 730/2304 [I 2022-10-26 15:16:17] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:13:11 [I 2022-10-26 15:16:17] ...trap/engines/engine.py.229: process: 0.50849 | load: 0.00253 [I 2022-10-26 15:16:17] ...trap/engines/engine.py.230: loss: 0.22241 [I 2022-10-26 15:16:17] ...optimizers/trijoint.py.151: total_norm: 3.128389 [I 2022-10-26 15:16:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 740/2304 [I 2022-10-26 15:16:22] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:13:05 [I 2022-10-26 15:16:22] ...trap/engines/engine.py.229: process: 0.50238 | load: 0.00249 [I 2022-10-26 15:16:22] ...trap/engines/engine.py.230: loss: 0.17790 [I 2022-10-26 15:16:22] ...optimizers/trijoint.py.151: total_norm: 3.620951 [I 2022-10-26 15:16:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 750/2304 [I 2022-10-26 15:16:27] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:12:55 [I 2022-10-26 15:16:27] ...trap/engines/engine.py.229: process: 0.49977 | load: 0.00249 [I 2022-10-26 15:16:27] ...trap/engines/engine.py.230: loss: 0.21032 [I 2022-10-26 15:16:27] ...optimizers/trijoint.py.151: total_norm: 3.739352 [I 2022-10-26 15:16:33] ...trap/engines/engine.py.224: train: epoch 15 | batch 760/2304 [I 2022-10-26 15:16:33] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:13:06 [I 2022-10-26 15:16:33] ...trap/engines/engine.py.229: process: 0.49137 | load: 0.00244 [I 2022-10-26 15:16:33] ...trap/engines/engine.py.230: loss: 0.16721 [I 2022-10-26 15:16:33] ...optimizers/trijoint.py.151: total_norm: 2.523508 [I 2022-10-26 15:16:38] ...trap/engines/engine.py.224: train: epoch 15 | batch 770/2304 [I 2022-10-26 15:16:38] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:12:41 [I 2022-10-26 15:16:38] ...trap/engines/engine.py.229: process: 0.49788 | load: 0.00248 [I 2022-10-26 15:16:38] ...trap/engines/engine.py.230: loss: 0.29717 [I 2022-10-26 15:16:38] ...optimizers/trijoint.py.151: total_norm: 3.836237 [I 2022-10-26 15:16:43] ...trap/engines/engine.py.224: train: epoch 15 | batch 780/2304 [I 2022-10-26 15:16:43] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:12:32 [I 2022-10-26 15:16:43] ...trap/engines/engine.py.229: process: 0.48828 | load: 0.00252 [I 2022-10-26 15:16:43] ...trap/engines/engine.py.230: loss: 0.17814 [I 2022-10-26 15:16:43] ...optimizers/trijoint.py.151: total_norm: 3.338116 [I 2022-10-26 15:16:48] ...trap/engines/engine.py.224: train: epoch 15 | batch 790/2304 [I 2022-10-26 15:16:48] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:12:31 [I 2022-10-26 15:16:48] ...trap/engines/engine.py.229: process: 0.49673 | load: 0.00246 [I 2022-10-26 15:16:48] ...trap/engines/engine.py.230: loss: 0.16586 [I 2022-10-26 15:16:48] ...optimizers/trijoint.py.151: total_norm: 3.602832 [I 2022-10-26 15:16:53] ...trap/engines/engine.py.224: train: epoch 15 | batch 800/2304 [I 2022-10-26 15:16:53] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:12:34 [I 2022-10-26 15:16:53] ...trap/engines/engine.py.229: process: 0.50768 | load: 0.00257 [I 2022-10-26 15:16:53] ...trap/engines/engine.py.230: loss: 0.24649 [I 2022-10-26 15:16:53] ...optimizers/trijoint.py.151: total_norm: 3.875246 [I 2022-10-26 15:16:58] ...trap/engines/engine.py.224: train: epoch 15 | batch 810/2304 [I 2022-10-26 15:16:58] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:12:21 [I 2022-10-26 15:16:58] ...trap/engines/engine.py.229: process: 0.49806 | load: 0.00251 [I 2022-10-26 15:16:58] ...trap/engines/engine.py.230: loss: 0.16281 [I 2022-10-26 15:16:58] ...optimizers/trijoint.py.151: total_norm: 3.327496 [I 2022-10-26 15:17:03] ...trap/engines/engine.py.224: train: epoch 15 | batch 820/2304 [I 2022-10-26 15:17:03] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:12:19 [I 2022-10-26 15:17:03] ...trap/engines/engine.py.229: process: 0.50129 | load: 0.00250 [I 2022-10-26 15:17:03] ...trap/engines/engine.py.230: loss: 0.19051 [I 2022-10-26 15:17:03] ...optimizers/trijoint.py.151: total_norm: 3.210941 [I 2022-10-26 15:17:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 830/2304 [I 2022-10-26 15:17:07] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:12:10 [I 2022-10-26 15:17:07] ...trap/engines/engine.py.229: process: 0.49503 | load: 0.00243 [I 2022-10-26 15:17:07] ...trap/engines/engine.py.230: loss: 0.16719 [I 2022-10-26 15:17:07] ...optimizers/trijoint.py.151: total_norm: 4.093994 [I 2022-10-26 15:17:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 840/2304 [I 2022-10-26 15:17:12] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:12:05 [I 2022-10-26 15:17:12] ...trap/engines/engine.py.229: process: 0.49616 | load: 0.00262 [I 2022-10-26 15:17:12] ...trap/engines/engine.py.230: loss: 0.25775 [I 2022-10-26 15:17:12] ...optimizers/trijoint.py.151: total_norm: 3.910483 [I 2022-10-26 15:17:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 850/2304 [I 2022-10-26 15:17:17] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:12:02 [I 2022-10-26 15:17:17] ...trap/engines/engine.py.229: process: 0.48550 | load: 0.00254 [I 2022-10-26 15:17:17] ...trap/engines/engine.py.230: loss: 0.24815 [I 2022-10-26 15:17:17] ...optimizers/trijoint.py.151: total_norm: 3.397696 [I 2022-10-26 15:17:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 860/2304 [I 2022-10-26 15:17:22] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:11:56 [I 2022-10-26 15:17:22] ...trap/engines/engine.py.229: process: 0.49903 | load: 0.00250 [I 2022-10-26 15:17:22] ...trap/engines/engine.py.230: loss: 0.23163 [I 2022-10-26 15:17:22] ...optimizers/trijoint.py.151: total_norm: 3.299139 [I 2022-10-26 15:17:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 870/2304 [I 2022-10-26 15:17:27] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:11:48 [I 2022-10-26 15:17:27] ...trap/engines/engine.py.229: process: 0.48723 | load: 0.00245 [I 2022-10-26 15:17:27] ...trap/engines/engine.py.230: loss: 0.23670 [I 2022-10-26 15:17:27] ...optimizers/trijoint.py.151: total_norm: 3.536281 [I 2022-10-26 15:17:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 880/2304 [I 2022-10-26 15:17:32] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:11:43 [I 2022-10-26 15:17:32] ...trap/engines/engine.py.229: process: 0.50258 | load: 0.00242 [I 2022-10-26 15:17:32] ...trap/engines/engine.py.230: loss: 0.21099 [I 2022-10-26 15:17:32] ...optimizers/trijoint.py.151: total_norm: 3.290306 [I 2022-10-26 15:17:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 890/2304 [I 2022-10-26 15:17:37] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:11:37 [I 2022-10-26 15:17:37] ...trap/engines/engine.py.229: process: 0.50092 | load: 0.00246 [I 2022-10-26 15:17:37] ...trap/engines/engine.py.230: loss: 0.27061 [I 2022-10-26 15:17:37] ...optimizers/trijoint.py.151: total_norm: 4.614845 [I 2022-10-26 15:17:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 900/2304 [I 2022-10-26 15:17:42] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:11:45 [I 2022-10-26 15:17:42] ...trap/engines/engine.py.229: process: 0.49740 | load: 0.00241 [I 2022-10-26 15:17:42] ...trap/engines/engine.py.230: loss: 0.25136 [I 2022-10-26 15:17:42] ...optimizers/trijoint.py.151: total_norm: 3.540146 [I 2022-10-26 15:17:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 910/2304 [I 2022-10-26 15:17:47] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:11:33 [I 2022-10-26 15:17:47] ...trap/engines/engine.py.229: process: 0.50367 | load: 0.00244 [I 2022-10-26 15:17:47] ...trap/engines/engine.py.230: loss: 0.23009 [I 2022-10-26 15:17:47] ...optimizers/trijoint.py.151: total_norm: 3.211279 [I 2022-10-26 15:17:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 920/2304 [I 2022-10-26 15:17:52] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:11:26 [I 2022-10-26 15:17:52] ...trap/engines/engine.py.229: process: 0.48948 | load: 0.00241 [I 2022-10-26 15:17:52] ...trap/engines/engine.py.230: loss: 0.23413 [I 2022-10-26 15:17:52] ...optimizers/trijoint.py.151: total_norm: 3.785805 [I 2022-10-26 15:17:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 930/2304 [I 2022-10-26 15:17:57] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:11:16 [I 2022-10-26 15:17:57] ...trap/engines/engine.py.229: process: 0.49401 | load: 0.00247 [I 2022-10-26 15:17:57] ...trap/engines/engine.py.230: loss: 0.18809 [I 2022-10-26 15:17:57] ...optimizers/trijoint.py.151: total_norm: 3.511195 [I 2022-10-26 15:18:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 940/2304 [I 2022-10-26 15:18:02] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:11:18 [I 2022-10-26 15:18:02] ...trap/engines/engine.py.229: process: 0.49698 | load: 0.00243 [I 2022-10-26 15:18:02] ...trap/engines/engine.py.230: loss: 0.16988 [I 2022-10-26 15:18:02] ...optimizers/trijoint.py.151: total_norm: 4.051531 [I 2022-10-26 15:18:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 950/2304 [I 2022-10-26 15:18:07] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:11:14 [I 2022-10-26 15:18:07] ...trap/engines/engine.py.229: process: 0.49741 | load: 0.00249 [I 2022-10-26 15:18:07] ...trap/engines/engine.py.230: loss: 0.25287 [I 2022-10-26 15:18:07] ...optimizers/trijoint.py.151: total_norm: 3.735193 [I 2022-10-26 15:18:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 960/2304 [I 2022-10-26 15:18:12] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:11:06 [I 2022-10-26 15:18:12] ...trap/engines/engine.py.229: process: 0.49489 | load: 0.00253 [I 2022-10-26 15:18:12] ...trap/engines/engine.py.230: loss: 0.20643 [I 2022-10-26 15:18:12] ...optimizers/trijoint.py.151: total_norm: 2.974377 [I 2022-10-26 15:18:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 970/2304 [I 2022-10-26 15:18:17] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:11:05 [I 2022-10-26 15:18:17] ...trap/engines/engine.py.229: process: 0.50577 | load: 0.00248 [I 2022-10-26 15:18:17] ...trap/engines/engine.py.230: loss: 0.21657 [I 2022-10-26 15:18:17] ...optimizers/trijoint.py.151: total_norm: 4.191166 [I 2022-10-26 15:18:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 980/2304 [I 2022-10-26 15:18:22] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:10:55 [I 2022-10-26 15:18:22] ...trap/engines/engine.py.229: process: 0.49822 | load: 0.00251 [I 2022-10-26 15:18:22] ...trap/engines/engine.py.230: loss: 0.28325 [I 2022-10-26 15:18:22] ...optimizers/trijoint.py.151: total_norm: 4.872654 [I 2022-10-26 15:18:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 990/2304 [I 2022-10-26 15:18:27] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:10:57 [I 2022-10-26 15:18:27] ...trap/engines/engine.py.229: process: 0.50112 | load: 0.00264 [I 2022-10-26 15:18:27] ...trap/engines/engine.py.230: loss: 0.14088 [I 2022-10-26 15:18:27] ...optimizers/trijoint.py.151: total_norm: 2.488097 [I 2022-10-26 15:18:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 1000/2304 [I 2022-10-26 15:18:32] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:10:43 [I 2022-10-26 15:18:32] ...trap/engines/engine.py.229: process: 0.48976 | load: 0.00257 [I 2022-10-26 15:18:32] ...trap/engines/engine.py.230: loss: 0.26292 [I 2022-10-26 15:18:32] ...optimizers/trijoint.py.151: total_norm: 3.401231 [I 2022-10-26 15:18:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 1010/2304 [I 2022-10-26 15:18:37] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:10:40 [I 2022-10-26 15:18:37] ...trap/engines/engine.py.229: process: 0.50353 | load: 0.00251 [I 2022-10-26 15:18:37] ...trap/engines/engine.py.230: loss: 0.17130 [I 2022-10-26 15:18:37] ...optimizers/trijoint.py.151: total_norm: 2.143352 [I 2022-10-26 15:18:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 1020/2304 [I 2022-10-26 15:18:42] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:10:43 [I 2022-10-26 15:18:42] ...trap/engines/engine.py.229: process: 0.50761 | load: 0.00247 [I 2022-10-26 15:18:42] ...trap/engines/engine.py.230: loss: 0.19744 [I 2022-10-26 15:18:42] ...optimizers/trijoint.py.151: total_norm: 3.747996 [I 2022-10-26 15:18:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 1030/2304 [I 2022-10-26 15:18:47] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:10:33 [I 2022-10-26 15:18:47] ...trap/engines/engine.py.229: process: 0.49823 | load: 0.00245 [I 2022-10-26 15:18:47] ...trap/engines/engine.py.230: loss: 0.21406 [I 2022-10-26 15:18:47] ...optimizers/trijoint.py.151: total_norm: 3.210792 [I 2022-10-26 15:18:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 1040/2304 [I 2022-10-26 15:18:52] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:10:30 [I 2022-10-26 15:18:52] ...trap/engines/engine.py.229: process: 0.50291 | load: 0.00241 [I 2022-10-26 15:18:52] ...trap/engines/engine.py.230: loss: 0.20432 [I 2022-10-26 15:18:52] ...optimizers/trijoint.py.151: total_norm: 3.085117 [I 2022-10-26 15:18:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 1050/2304 [I 2022-10-26 15:18:57] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:10:26 [I 2022-10-26 15:18:57] ...trap/engines/engine.py.229: process: 0.50213 | load: 0.00247 [I 2022-10-26 15:18:57] ...trap/engines/engine.py.230: loss: 0.18490 [I 2022-10-26 15:18:57] ...optimizers/trijoint.py.151: total_norm: 2.517439 [I 2022-10-26 15:19:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 1060/2304 [I 2022-10-26 15:19:02] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:10:16 [I 2022-10-26 15:19:02] ...trap/engines/engine.py.229: process: 0.48986 | load: 0.00242 [I 2022-10-26 15:19:02] ...trap/engines/engine.py.230: loss: 0.21176 [I 2022-10-26 15:19:02] ...optimizers/trijoint.py.151: total_norm: 4.059855 [I 2022-10-26 15:19:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 1070/2304 [I 2022-10-26 15:19:07] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:10:10 [I 2022-10-26 15:19:07] ...trap/engines/engine.py.229: process: 0.49500 | load: 0.00241 [I 2022-10-26 15:19:07] ...trap/engines/engine.py.230: loss: 0.24695 [I 2022-10-26 15:19:07] ...optimizers/trijoint.py.151: total_norm: 3.159813 [I 2022-10-26 15:19:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 1080/2304 [I 2022-10-26 15:19:12] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:10:05 [I 2022-10-26 15:19:12] ...trap/engines/engine.py.229: process: 0.48770 | load: 0.00241 [I 2022-10-26 15:19:12] ...trap/engines/engine.py.230: loss: 0.21514 [I 2022-10-26 15:19:12] ...optimizers/trijoint.py.151: total_norm: 2.795720 [I 2022-10-26 15:19:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 1090/2304 [I 2022-10-26 15:19:17] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:10:03 [I 2022-10-26 15:19:17] ...trap/engines/engine.py.229: process: 0.50051 | load: 0.00254 [I 2022-10-26 15:19:17] ...trap/engines/engine.py.230: loss: 0.29953 [I 2022-10-26 15:19:17] ...optimizers/trijoint.py.151: total_norm: 4.357807 [I 2022-10-26 15:19:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 1100/2304 [I 2022-10-26 15:19:22] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:09:56 [I 2022-10-26 15:19:22] ...trap/engines/engine.py.229: process: 0.50034 | load: 0.00245 [I 2022-10-26 15:19:22] ...trap/engines/engine.py.230: loss: 0.20852 [I 2022-10-26 15:19:22] ...optimizers/trijoint.py.151: total_norm: 2.832668 [I 2022-10-26 15:19:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 1110/2304 [I 2022-10-26 15:19:27] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:09:48 [I 2022-10-26 15:19:27] ...trap/engines/engine.py.229: process: 0.49116 | load: 0.00253 [I 2022-10-26 15:19:27] ...trap/engines/engine.py.230: loss: 0.21522 [I 2022-10-26 15:19:27] ...optimizers/trijoint.py.151: total_norm: 3.180832 [I 2022-10-26 15:19:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 1120/2304 [I 2022-10-26 15:19:32] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:09:45 [I 2022-10-26 15:19:32] ...trap/engines/engine.py.229: process: 0.49660 | load: 0.00250 [I 2022-10-26 15:19:32] ...trap/engines/engine.py.230: loss: 0.18495 [I 2022-10-26 15:19:32] ...optimizers/trijoint.py.151: total_norm: 3.675445 [I 2022-10-26 15:19:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 1130/2304 [I 2022-10-26 15:19:37] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:09:43 [I 2022-10-26 15:19:37] ...trap/engines/engine.py.229: process: 0.48898 | load: 0.00248 [I 2022-10-26 15:19:37] ...trap/engines/engine.py.230: loss: 0.25928 [I 2022-10-26 15:19:37] ...optimizers/trijoint.py.151: total_norm: 3.375073 [I 2022-10-26 15:19:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 1140/2304 [I 2022-10-26 15:19:42] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:09:46 [I 2022-10-26 15:19:42] ...trap/engines/engine.py.229: process: 0.51720 | load: 0.00252 [I 2022-10-26 15:19:42] ...trap/engines/engine.py.230: loss: 0.19540 [I 2022-10-26 15:19:42] ...optimizers/trijoint.py.151: total_norm: 3.060132 [I 2022-10-26 15:19:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 1150/2304 [I 2022-10-26 15:19:47] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:09:41 [I 2022-10-26 15:19:47] ...trap/engines/engine.py.229: process: 0.49799 | load: 0.00257 [I 2022-10-26 15:19:47] ...trap/engines/engine.py.230: loss: 0.16815 [I 2022-10-26 15:19:47] ...optimizers/trijoint.py.151: total_norm: 2.570790 [I 2022-10-26 15:19:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 1160/2304 [I 2022-10-26 15:19:52] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:09:26 [I 2022-10-26 15:19:52] ...trap/engines/engine.py.229: process: 0.49529 | load: 0.00252 [I 2022-10-26 15:19:52] ...trap/engines/engine.py.230: loss: 0.22706 [I 2022-10-26 15:19:52] ...optimizers/trijoint.py.151: total_norm: 2.978389 [I 2022-10-26 15:19:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 1170/2304 [I 2022-10-26 15:19:57] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:09:20 [I 2022-10-26 15:19:57] ...trap/engines/engine.py.229: process: 0.49689 | load: 0.00244 [I 2022-10-26 15:19:57] ...trap/engines/engine.py.230: loss: 0.16734 [I 2022-10-26 15:19:57] ...optimizers/trijoint.py.151: total_norm: 3.135989 [I 2022-10-26 15:20:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 1180/2304 [I 2022-10-26 15:20:02] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:09:20 [I 2022-10-26 15:20:02] ...trap/engines/engine.py.229: process: 0.50563 | load: 0.00245 [I 2022-10-26 15:20:02] ...trap/engines/engine.py.230: loss: 0.23450 [I 2022-10-26 15:20:02] ...optimizers/trijoint.py.151: total_norm: 3.736897 [I 2022-10-26 15:20:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 1190/2304 [I 2022-10-26 15:20:07] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:09:16 [I 2022-10-26 15:20:07] ...trap/engines/engine.py.229: process: 0.50118 | load: 0.00262 [I 2022-10-26 15:20:07] ...trap/engines/engine.py.230: loss: 0.17343 [I 2022-10-26 15:20:07] ...optimizers/trijoint.py.151: total_norm: 2.584543 [I 2022-10-26 15:20:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 1200/2304 [I 2022-10-26 15:20:12] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:09:08 [I 2022-10-26 15:20:12] ...trap/engines/engine.py.229: process: 0.48975 | load: 0.00241 [I 2022-10-26 15:20:12] ...trap/engines/engine.py.230: loss: 0.19289 [I 2022-10-26 15:20:12] ...optimizers/trijoint.py.151: total_norm: 3.059291 [I 2022-10-26 15:20:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 1210/2304 [I 2022-10-26 15:20:17] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:09:00 [I 2022-10-26 15:20:17] ...trap/engines/engine.py.229: process: 0.48259 | load: 0.00243 [I 2022-10-26 15:20:17] ...trap/engines/engine.py.230: loss: 0.15970 [I 2022-10-26 15:20:17] ...optimizers/trijoint.py.151: total_norm: 2.787935 [I 2022-10-26 15:20:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 1220/2304 [I 2022-10-26 15:20:22] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:09:02 [I 2022-10-26 15:20:22] ...trap/engines/engine.py.229: process: 0.50006 | load: 0.00264 [I 2022-10-26 15:20:22] ...trap/engines/engine.py.230: loss: 0.27119 [I 2022-10-26 15:20:22] ...optimizers/trijoint.py.151: total_norm: 3.454765 [I 2022-10-26 15:20:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 1230/2304 [I 2022-10-26 15:20:27] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:08:51 [I 2022-10-26 15:20:27] ...trap/engines/engine.py.229: process: 0.49450 | load: 0.00254 [I 2022-10-26 15:20:27] ...trap/engines/engine.py.230: loss: 0.14737 [I 2022-10-26 15:20:27] ...optimizers/trijoint.py.151: total_norm: 3.686430 [I 2022-10-26 15:20:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 1240/2304 [I 2022-10-26 15:20:32] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:08:52 [I 2022-10-26 15:20:32] ...trap/engines/engine.py.229: process: 0.49156 | load: 0.00253 [I 2022-10-26 15:20:32] ...trap/engines/engine.py.230: loss: 0.29003 [I 2022-10-26 15:20:32] ...optimizers/trijoint.py.151: total_norm: 2.817399 [I 2022-10-26 15:20:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 1250/2304 [I 2022-10-26 15:20:37] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:08:48 [I 2022-10-26 15:20:37] ...trap/engines/engine.py.229: process: 0.51166 | load: 0.00244 [I 2022-10-26 15:20:37] ...trap/engines/engine.py.230: loss: 0.26610 [I 2022-10-26 15:20:37] ...optimizers/trijoint.py.151: total_norm: 3.573744 [I 2022-10-26 15:20:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 1260/2304 [I 2022-10-26 15:20:42] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:08:39 [I 2022-10-26 15:20:42] ...trap/engines/engine.py.229: process: 0.49114 | load: 0.00254 [I 2022-10-26 15:20:42] ...trap/engines/engine.py.230: loss: 0.27700 [I 2022-10-26 15:20:42] ...optimizers/trijoint.py.151: total_norm: 4.473176 [I 2022-10-26 15:20:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 1270/2304 [I 2022-10-26 15:20:47] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:08:32 [I 2022-10-26 15:20:47] ...trap/engines/engine.py.229: process: 0.50070 | load: 0.00246 [I 2022-10-26 15:20:47] ...trap/engines/engine.py.230: loss: 0.18142 [I 2022-10-26 15:20:47] ...optimizers/trijoint.py.151: total_norm: 3.784380 [I 2022-10-26 15:20:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 1280/2304 [I 2022-10-26 15:20:52] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:08:28 [I 2022-10-26 15:20:52] ...trap/engines/engine.py.229: process: 0.50102 | load: 0.00251 [I 2022-10-26 15:20:52] ...trap/engines/engine.py.230: loss: 0.22556 [I 2022-10-26 15:20:52] ...optimizers/trijoint.py.151: total_norm: 2.932870 [I 2022-10-26 15:20:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 1290/2304 [I 2022-10-26 15:20:57] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:08:23 [I 2022-10-26 15:20:57] ...trap/engines/engine.py.229: process: 0.50691 | load: 0.00248 [I 2022-10-26 15:20:57] ...trap/engines/engine.py.230: loss: 0.21946 [I 2022-10-26 15:20:57] ...optimizers/trijoint.py.151: total_norm: 3.509651 [I 2022-10-26 15:21:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 1300/2304 [I 2022-10-26 15:21:02] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:08:17 [I 2022-10-26 15:21:02] ...trap/engines/engine.py.229: process: 0.49039 | load: 0.00243 [I 2022-10-26 15:21:02] ...trap/engines/engine.py.230: loss: 0.28316 [I 2022-10-26 15:21:02] ...optimizers/trijoint.py.151: total_norm: 4.444990 [I 2022-10-26 15:21:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 1310/2304 [I 2022-10-26 15:21:07] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:08:18 [I 2022-10-26 15:21:07] ...trap/engines/engine.py.229: process: 0.50475 | load: 0.00253 [I 2022-10-26 15:21:07] ...trap/engines/engine.py.230: loss: 0.12097 [I 2022-10-26 15:21:07] ...optimizers/trijoint.py.151: total_norm: 2.549315 [I 2022-10-26 15:21:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 1320/2304 [I 2022-10-26 15:21:12] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:08:11 [I 2022-10-26 15:21:12] ...trap/engines/engine.py.229: process: 0.50848 | load: 0.00255 [I 2022-10-26 15:21:12] ...trap/engines/engine.py.230: loss: 0.11561 [I 2022-10-26 15:21:12] ...optimizers/trijoint.py.151: total_norm: 1.847667 [I 2022-10-26 15:21:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 1330/2304 [I 2022-10-26 15:21:17] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:08:02 [I 2022-10-26 15:21:17] ...trap/engines/engine.py.229: process: 0.48815 | load: 0.00250 [I 2022-10-26 15:21:17] ...trap/engines/engine.py.230: loss: 0.20776 [I 2022-10-26 15:21:17] ...optimizers/trijoint.py.151: total_norm: 3.591153 [I 2022-10-26 15:21:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 1340/2304 [I 2022-10-26 15:21:22] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:07:59 [I 2022-10-26 15:21:22] ...trap/engines/engine.py.229: process: 0.49563 | load: 0.00264 [I 2022-10-26 15:21:22] ...trap/engines/engine.py.230: loss: 0.25235 [I 2022-10-26 15:21:22] ...optimizers/trijoint.py.151: total_norm: 3.503689 [I 2022-10-26 15:21:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 1350/2304 [I 2022-10-26 15:21:27] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:07:53 [I 2022-10-26 15:21:27] ...trap/engines/engine.py.229: process: 0.47731 | load: 0.00247 [I 2022-10-26 15:21:27] ...trap/engines/engine.py.230: loss: 0.16306 [I 2022-10-26 15:21:27] ...optimizers/trijoint.py.151: total_norm: 2.633057 [I 2022-10-26 15:21:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 1360/2304 [I 2022-10-26 15:21:32] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:07:50 [I 2022-10-26 15:21:32] ...trap/engines/engine.py.229: process: 0.48572 | load: 0.00255 [I 2022-10-26 15:21:32] ...trap/engines/engine.py.230: loss: 0.13879 [I 2022-10-26 15:21:32] ...optimizers/trijoint.py.151: total_norm: 2.284403 [I 2022-10-26 15:21:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 1370/2304 [I 2022-10-26 15:21:37] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:07:44 [I 2022-10-26 15:21:37] ...trap/engines/engine.py.229: process: 0.49465 | load: 0.00249 [I 2022-10-26 15:21:37] ...trap/engines/engine.py.230: loss: 0.16946 [I 2022-10-26 15:21:37] ...optimizers/trijoint.py.151: total_norm: 2.503716 [I 2022-10-26 15:21:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 1380/2304 [I 2022-10-26 15:21:42] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:07:40 [I 2022-10-26 15:21:42] ...trap/engines/engine.py.229: process: 0.49309 | load: 0.00247 [I 2022-10-26 15:21:42] ...trap/engines/engine.py.230: loss: 0.27173 [I 2022-10-26 15:21:42] ...optimizers/trijoint.py.151: total_norm: 3.354481 [I 2022-10-26 15:21:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 1390/2304 [I 2022-10-26 15:21:47] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:07:29 [I 2022-10-26 15:21:47] ...trap/engines/engine.py.229: process: 0.48724 | load: 0.00241 [I 2022-10-26 15:21:47] ...trap/engines/engine.py.230: loss: 0.23668 [I 2022-10-26 15:21:47] ...optimizers/trijoint.py.151: total_norm: 3.476349 [I 2022-10-26 15:21:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 1400/2304 [I 2022-10-26 15:21:52] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:07:27 [I 2022-10-26 15:21:52] ...trap/engines/engine.py.229: process: 0.49521 | load: 0.00254 [I 2022-10-26 15:21:52] ...trap/engines/engine.py.230: loss: 0.21701 [I 2022-10-26 15:21:52] ...optimizers/trijoint.py.151: total_norm: 3.589437 [I 2022-10-26 15:21:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 1410/2304 [I 2022-10-26 15:21:57] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:07:23 [I 2022-10-26 15:21:57] ...trap/engines/engine.py.229: process: 0.49454 | load: 0.00250 [I 2022-10-26 15:21:57] ...trap/engines/engine.py.230: loss: 0.19700 [I 2022-10-26 15:21:57] ...optimizers/trijoint.py.151: total_norm: 3.225265 [I 2022-10-26 15:22:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 1420/2304 [I 2022-10-26 15:22:02] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:07:18 [I 2022-10-26 15:22:02] ...trap/engines/engine.py.229: process: 0.49453 | load: 0.00257 [I 2022-10-26 15:22:02] ...trap/engines/engine.py.230: loss: 0.23092 [I 2022-10-26 15:22:02] ...optimizers/trijoint.py.151: total_norm: 3.183202 [I 2022-10-26 15:22:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 1430/2304 [I 2022-10-26 15:22:07] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:07:12 [I 2022-10-26 15:22:07] ...trap/engines/engine.py.229: process: 0.49812 | load: 0.00278 [I 2022-10-26 15:22:07] ...trap/engines/engine.py.230: loss: 0.15729 [I 2022-10-26 15:22:07] ...optimizers/trijoint.py.151: total_norm: 3.147413 [I 2022-10-26 15:22:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 1440/2304 [I 2022-10-26 15:22:12] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:07:09 [I 2022-10-26 15:22:12] ...trap/engines/engine.py.229: process: 0.50102 | load: 0.00246 [I 2022-10-26 15:22:12] ...trap/engines/engine.py.230: loss: 0.12943 [I 2022-10-26 15:22:12] ...optimizers/trijoint.py.151: total_norm: 1.882601 [I 2022-10-26 15:22:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 1450/2304 [I 2022-10-26 15:22:17] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:07:02 [I 2022-10-26 15:22:17] ...trap/engines/engine.py.229: process: 0.50475 | load: 0.00249 [I 2022-10-26 15:22:17] ...trap/engines/engine.py.230: loss: 0.18299 [I 2022-10-26 15:22:17] ...optimizers/trijoint.py.151: total_norm: 2.294367 [I 2022-10-26 15:22:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 1460/2304 [I 2022-10-26 15:22:22] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:06:59 [I 2022-10-26 15:22:22] ...trap/engines/engine.py.229: process: 0.49427 | load: 0.00256 [I 2022-10-26 15:22:22] ...trap/engines/engine.py.230: loss: 0.16931 [I 2022-10-26 15:22:22] ...optimizers/trijoint.py.151: total_norm: 2.678730 [I 2022-10-26 15:22:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 1470/2304 [I 2022-10-26 15:22:27] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:06:54 [I 2022-10-26 15:22:27] ...trap/engines/engine.py.229: process: 0.49677 | load: 0.00254 [I 2022-10-26 15:22:27] ...trap/engines/engine.py.230: loss: 0.20307 [I 2022-10-26 15:22:27] ...optimizers/trijoint.py.151: total_norm: 2.422904 [I 2022-10-26 15:22:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 1480/2304 [I 2022-10-26 15:22:32] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:06:48 [I 2022-10-26 15:22:32] ...trap/engines/engine.py.229: process: 0.49760 | load: 0.00250 [I 2022-10-26 15:22:32] ...trap/engines/engine.py.230: loss: 0.22407 [I 2022-10-26 15:22:32] ...optimizers/trijoint.py.151: total_norm: 2.592004 [I 2022-10-26 15:22:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 1490/2304 [I 2022-10-26 15:22:37] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:06:46 [I 2022-10-26 15:22:37] ...trap/engines/engine.py.229: process: 0.48800 | load: 0.00250 [I 2022-10-26 15:22:37] ...trap/engines/engine.py.230: loss: 0.27521 [I 2022-10-26 15:22:37] ...optimizers/trijoint.py.151: total_norm: 4.091332 [I 2022-10-26 15:22:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 1500/2304 [I 2022-10-26 15:22:42] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:06:38 [I 2022-10-26 15:22:42] ...trap/engines/engine.py.229: process: 0.49956 | load: 0.00247 [I 2022-10-26 15:22:42] ...trap/engines/engine.py.230: loss: 0.13918 [I 2022-10-26 15:22:42] ...optimizers/trijoint.py.151: total_norm: 2.753844 [I 2022-10-26 15:22:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 1510/2304 [I 2022-10-26 15:22:47] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:06:36 [I 2022-10-26 15:22:47] ...trap/engines/engine.py.229: process: 0.50423 | load: 0.00245 [I 2022-10-26 15:22:47] ...trap/engines/engine.py.230: loss: 0.29181 [I 2022-10-26 15:22:47] ...optimizers/trijoint.py.151: total_norm: 4.328218 [I 2022-10-26 15:22:52] ...trap/engines/engine.py.224: train: epoch 15 | batch 1520/2304 [I 2022-10-26 15:22:52] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:06:28 [I 2022-10-26 15:22:52] ...trap/engines/engine.py.229: process: 0.50486 | load: 0.00244 [I 2022-10-26 15:22:52] ...trap/engines/engine.py.230: loss: 0.18151 [I 2022-10-26 15:22:52] ...optimizers/trijoint.py.151: total_norm: 3.224737 [I 2022-10-26 15:22:57] ...trap/engines/engine.py.224: train: epoch 15 | batch 1530/2304 [I 2022-10-26 15:22:57] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:06:26 [I 2022-10-26 15:22:57] ...trap/engines/engine.py.229: process: 0.49924 | load: 0.00251 [I 2022-10-26 15:22:57] ...trap/engines/engine.py.230: loss: 0.21354 [I 2022-10-26 15:22:57] ...optimizers/trijoint.py.151: total_norm: 3.093359 [I 2022-10-26 15:23:02] ...trap/engines/engine.py.224: train: epoch 15 | batch 1540/2304 [I 2022-10-26 15:23:02] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:06:19 [I 2022-10-26 15:23:02] ...trap/engines/engine.py.229: process: 0.49057 | load: 0.00257 [I 2022-10-26 15:23:02] ...trap/engines/engine.py.230: loss: 0.19690 [I 2022-10-26 15:23:02] ...optimizers/trijoint.py.151: total_norm: 3.315552 [I 2022-10-26 15:23:07] ...trap/engines/engine.py.224: train: epoch 15 | batch 1550/2304 [I 2022-10-26 15:23:07] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:06:15 [I 2022-10-26 15:23:07] ...trap/engines/engine.py.229: process: 0.50067 | load: 0.00258 [I 2022-10-26 15:23:07] ...trap/engines/engine.py.230: loss: 0.18573 [I 2022-10-26 15:23:07] ...optimizers/trijoint.py.151: total_norm: 3.043691 [I 2022-10-26 15:23:12] ...trap/engines/engine.py.224: train: epoch 15 | batch 1560/2304 [I 2022-10-26 15:23:12] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:06:09 [I 2022-10-26 15:23:12] ...trap/engines/engine.py.229: process: 0.49278 | load: 0.00268 [I 2022-10-26 15:23:12] ...trap/engines/engine.py.230: loss: 0.28409 [I 2022-10-26 15:23:12] ...optimizers/trijoint.py.151: total_norm: 3.208826 [I 2022-10-26 15:23:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 1570/2304 [I 2022-10-26 15:23:17] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:06:08 [I 2022-10-26 15:23:17] ...trap/engines/engine.py.229: process: 0.49095 | load: 0.00260 [I 2022-10-26 15:23:17] ...trap/engines/engine.py.230: loss: 0.19653 [I 2022-10-26 15:23:17] ...optimizers/trijoint.py.151: total_norm: 2.276484 [I 2022-10-26 15:23:22] ...trap/engines/engine.py.224: train: epoch 15 | batch 1580/2304 [I 2022-10-26 15:23:22] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:05:58 [I 2022-10-26 15:23:22] ...trap/engines/engine.py.229: process: 0.49020 | load: 0.00268 [I 2022-10-26 15:23:22] ...trap/engines/engine.py.230: loss: 0.25023 [I 2022-10-26 15:23:22] ...optimizers/trijoint.py.151: total_norm: 3.890222 [I 2022-10-26 15:23:27] ...trap/engines/engine.py.224: train: epoch 15 | batch 1590/2304 [I 2022-10-26 15:23:27] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:05:56 [I 2022-10-26 15:23:27] ...trap/engines/engine.py.229: process: 0.50635 | load: 0.00256 [I 2022-10-26 15:23:27] ...trap/engines/engine.py.230: loss: 0.18814 [I 2022-10-26 15:23:27] ...optimizers/trijoint.py.151: total_norm: 3.438619 [I 2022-10-26 15:23:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 1600/2304 [I 2022-10-26 15:23:32] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:05:50 [I 2022-10-26 15:23:32] ...trap/engines/engine.py.229: process: 0.49244 | load: 0.00257 [I 2022-10-26 15:23:32] ...trap/engines/engine.py.230: loss: 0.23604 [I 2022-10-26 15:23:32] ...optimizers/trijoint.py.151: total_norm: 3.932361 [I 2022-10-26 15:23:37] ...trap/engines/engine.py.224: train: epoch 15 | batch 1610/2304 [I 2022-10-26 15:23:37] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:05:44 [I 2022-10-26 15:23:37] ...trap/engines/engine.py.229: process: 0.50652 | load: 0.00254 [I 2022-10-26 15:23:37] ...trap/engines/engine.py.230: loss: 0.16457 [I 2022-10-26 15:23:37] ...optimizers/trijoint.py.151: total_norm: 2.723280 [I 2022-10-26 15:23:41] ...trap/engines/engine.py.224: train: epoch 15 | batch 1620/2304 [I 2022-10-26 15:23:41] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:05:38 [I 2022-10-26 15:23:41] ...trap/engines/engine.py.229: process: 0.49098 | load: 0.00261 [I 2022-10-26 15:23:41] ...trap/engines/engine.py.230: loss: 0.20618 [I 2022-10-26 15:23:41] ...optimizers/trijoint.py.151: total_norm: 3.029696 [I 2022-10-26 15:23:46] ...trap/engines/engine.py.224: train: epoch 15 | batch 1630/2304 [I 2022-10-26 15:23:46] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:05:33 [I 2022-10-26 15:23:46] ...trap/engines/engine.py.229: process: 0.49329 | load: 0.00248 [I 2022-10-26 15:23:46] ...trap/engines/engine.py.230: loss: 0.24183 [I 2022-10-26 15:23:46] ...optimizers/trijoint.py.151: total_norm: 3.068888 [I 2022-10-26 15:23:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 1640/2304 [I 2022-10-26 15:23:51] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:05:28 [I 2022-10-26 15:23:51] ...trap/engines/engine.py.229: process: 0.49859 | load: 0.00259 [I 2022-10-26 15:23:51] ...trap/engines/engine.py.230: loss: 0.20073 [I 2022-10-26 15:23:51] ...optimizers/trijoint.py.151: total_norm: 3.117724 [I 2022-10-26 15:23:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 1650/2304 [I 2022-10-26 15:23:56] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:05:23 [I 2022-10-26 15:23:56] ...trap/engines/engine.py.229: process: 0.48894 | load: 0.00257 [I 2022-10-26 15:23:56] ...trap/engines/engine.py.230: loss: 0.16327 [I 2022-10-26 15:23:56] ...optimizers/trijoint.py.151: total_norm: 2.775620 [I 2022-10-26 15:24:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 1660/2304 [I 2022-10-26 15:24:01] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:05:19 [I 2022-10-26 15:24:01] ...trap/engines/engine.py.229: process: 0.49680 | load: 0.00254 [I 2022-10-26 15:24:01] ...trap/engines/engine.py.230: loss: 0.18615 [I 2022-10-26 15:24:01] ...optimizers/trijoint.py.151: total_norm: 2.793305 [I 2022-10-26 15:24:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 1670/2304 [I 2022-10-26 15:24:06] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:05:14 [I 2022-10-26 15:24:06] ...trap/engines/engine.py.229: process: 0.49748 | load: 0.00259 [I 2022-10-26 15:24:06] ...trap/engines/engine.py.230: loss: 0.16656 [I 2022-10-26 15:24:06] ...optimizers/trijoint.py.151: total_norm: 3.227928 [I 2022-10-26 15:24:11] ...trap/engines/engine.py.224: train: epoch 15 | batch 1680/2304 [I 2022-10-26 15:24:11] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:05:08 [I 2022-10-26 15:24:11] ...trap/engines/engine.py.229: process: 0.48349 | load: 0.00257 [I 2022-10-26 15:24:11] ...trap/engines/engine.py.230: loss: 0.24871 [I 2022-10-26 15:24:11] ...optimizers/trijoint.py.151: total_norm: 3.886399 [I 2022-10-26 15:24:16] ...trap/engines/engine.py.224: train: epoch 15 | batch 1690/2304 [I 2022-10-26 15:24:16] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:05:08 [I 2022-10-26 15:24:16] ...trap/engines/engine.py.229: process: 0.50270 | load: 0.00260 [I 2022-10-26 15:24:16] ...trap/engines/engine.py.230: loss: 0.25410 [I 2022-10-26 15:24:16] ...optimizers/trijoint.py.151: total_norm: 3.248450 [I 2022-10-26 15:24:21] ...trap/engines/engine.py.224: train: epoch 15 | batch 1700/2304 [I 2022-10-26 15:24:21] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:04:59 [I 2022-10-26 15:24:21] ...trap/engines/engine.py.229: process: 0.49876 | load: 0.00259 [I 2022-10-26 15:24:21] ...trap/engines/engine.py.230: loss: 0.28517 [I 2022-10-26 15:24:21] ...optimizers/trijoint.py.151: total_norm: 3.328441 [I 2022-10-26 15:24:26] ...trap/engines/engine.py.224: train: epoch 15 | batch 1710/2304 [I 2022-10-26 15:24:26] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:04:57 [I 2022-10-26 15:24:26] ...trap/engines/engine.py.229: process: 0.49921 | load: 0.00258 [I 2022-10-26 15:24:26] ...trap/engines/engine.py.230: loss: 0.20371 [I 2022-10-26 15:24:26] ...optimizers/trijoint.py.151: total_norm: 3.559508 [I 2022-10-26 15:24:31] ...trap/engines/engine.py.224: train: epoch 15 | batch 1720/2304 [I 2022-10-26 15:24:31] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:04:48 [I 2022-10-26 15:24:31] ...trap/engines/engine.py.229: process: 0.48477 | load: 0.00256 [I 2022-10-26 15:24:31] ...trap/engines/engine.py.230: loss: 0.15375 [I 2022-10-26 15:24:31] ...optimizers/trijoint.py.151: total_norm: 2.835711 [I 2022-10-26 15:24:36] ...trap/engines/engine.py.224: train: epoch 15 | batch 1730/2304 [I 2022-10-26 15:24:36] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:04:44 [I 2022-10-26 15:24:36] ...trap/engines/engine.py.229: process: 0.49370 | load: 0.00258 [I 2022-10-26 15:24:36] ...trap/engines/engine.py.230: loss: 0.23732 [I 2022-10-26 15:24:36] ...optimizers/trijoint.py.151: total_norm: 3.671028 [I 2022-10-26 15:24:41] ...trap/engines/engine.py.224: train: epoch 15 | batch 1740/2304 [I 2022-10-26 15:24:41] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:04:42 [I 2022-10-26 15:24:41] ...trap/engines/engine.py.229: process: 0.50273 | load: 0.00255 [I 2022-10-26 15:24:41] ...trap/engines/engine.py.230: loss: 0.19878 [I 2022-10-26 15:24:41] ...optimizers/trijoint.py.151: total_norm: 3.883434 [I 2022-10-26 15:24:46] ...trap/engines/engine.py.224: train: epoch 15 | batch 1750/2304 [I 2022-10-26 15:24:46] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:04:36 [I 2022-10-26 15:24:46] ...trap/engines/engine.py.229: process: 0.49735 | load: 0.00252 [I 2022-10-26 15:24:46] ...trap/engines/engine.py.230: loss: 0.23395 [I 2022-10-26 15:24:46] ...optimizers/trijoint.py.151: total_norm: 2.957943 [I 2022-10-26 15:24:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 1760/2304 [I 2022-10-26 15:24:51] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:04:30 [I 2022-10-26 15:24:51] ...trap/engines/engine.py.229: process: 0.49816 | load: 0.00251 [I 2022-10-26 15:24:51] ...trap/engines/engine.py.230: loss: 0.26960 [I 2022-10-26 15:24:51] ...optimizers/trijoint.py.151: total_norm: 3.678565 [I 2022-10-26 15:24:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 1770/2304 [I 2022-10-26 15:24:56] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:04:26 [I 2022-10-26 15:24:56] ...trap/engines/engine.py.229: process: 0.50606 | load: 0.00256 [I 2022-10-26 15:24:56] ...trap/engines/engine.py.230: loss: 0.16207 [I 2022-10-26 15:24:56] ...optimizers/trijoint.py.151: total_norm: 2.487555 [I 2022-10-26 15:25:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 1780/2304 [I 2022-10-26 15:25:01] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:04:21 [I 2022-10-26 15:25:01] ...trap/engines/engine.py.229: process: 0.50086 | load: 0.00257 [I 2022-10-26 15:25:01] ...trap/engines/engine.py.230: loss: 0.29624 [I 2022-10-26 15:25:01] ...optimizers/trijoint.py.151: total_norm: 3.822758 [I 2022-10-26 15:25:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 1790/2304 [I 2022-10-26 15:25:06] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:04:15 [I 2022-10-26 15:25:06] ...trap/engines/engine.py.229: process: 0.49622 | load: 0.00255 [I 2022-10-26 15:25:06] ...trap/engines/engine.py.230: loss: 0.17837 [I 2022-10-26 15:25:06] ...optimizers/trijoint.py.151: total_norm: 2.060188 [I 2022-10-26 15:25:11] ...trap/engines/engine.py.224: train: epoch 15 | batch 1800/2304 [I 2022-10-26 15:25:11] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:04:14 [I 2022-10-26 15:25:11] ...trap/engines/engine.py.229: process: 0.50551 | load: 0.00254 [I 2022-10-26 15:25:11] ...trap/engines/engine.py.230: loss: 0.21590 [I 2022-10-26 15:25:11] ...optimizers/trijoint.py.151: total_norm: 2.889344 [I 2022-10-26 15:25:16] ...trap/engines/engine.py.224: train: epoch 15 | batch 1810/2304 [I 2022-10-26 15:25:16] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:04:14 [I 2022-10-26 15:25:16] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00253 [I 2022-10-26 15:25:16] ...trap/engines/engine.py.230: loss: 0.17189 [I 2022-10-26 15:25:16] ...optimizers/trijoint.py.151: total_norm: 2.733364 [I 2022-10-26 15:25:21] ...trap/engines/engine.py.224: train: epoch 15 | batch 1820/2304 [I 2022-10-26 15:25:21] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:04:02 [I 2022-10-26 15:25:21] ...trap/engines/engine.py.229: process: 0.50255 | load: 0.00264 [I 2022-10-26 15:25:21] ...trap/engines/engine.py.230: loss: 0.19965 [I 2022-10-26 15:25:21] ...optimizers/trijoint.py.151: total_norm: 3.056510 [I 2022-10-26 15:25:26] ...trap/engines/engine.py.224: train: epoch 15 | batch 1830/2304 [I 2022-10-26 15:25:26] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:03:55 [I 2022-10-26 15:25:26] ...trap/engines/engine.py.229: process: 0.49836 | load: 0.00258 [I 2022-10-26 15:25:26] ...trap/engines/engine.py.230: loss: 0.23273 [I 2022-10-26 15:25:26] ...optimizers/trijoint.py.151: total_norm: 4.099492 [I 2022-10-26 15:25:31] ...trap/engines/engine.py.224: train: epoch 15 | batch 1840/2304 [I 2022-10-26 15:25:31] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:03:51 [I 2022-10-26 15:25:31] ...trap/engines/engine.py.229: process: 0.50132 | load: 0.00262 [I 2022-10-26 15:25:31] ...trap/engines/engine.py.230: loss: 0.25159 [I 2022-10-26 15:25:31] ...optimizers/trijoint.py.151: total_norm: 3.345614 [I 2022-10-26 15:25:36] ...trap/engines/engine.py.224: train: epoch 15 | batch 1850/2304 [I 2022-10-26 15:25:36] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:03:46 [I 2022-10-26 15:25:36] ...trap/engines/engine.py.229: process: 0.49888 | load: 0.00255 [I 2022-10-26 15:25:36] ...trap/engines/engine.py.230: loss: 0.21053 [I 2022-10-26 15:25:36] ...optimizers/trijoint.py.151: total_norm: 3.535078 [I 2022-10-26 15:25:41] ...trap/engines/engine.py.224: train: epoch 15 | batch 1860/2304 [I 2022-10-26 15:25:41] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:03:42 [I 2022-10-26 15:25:41] ...trap/engines/engine.py.229: process: 0.51100 | load: 0.00256 [I 2022-10-26 15:25:41] ...trap/engines/engine.py.230: loss: 0.23666 [I 2022-10-26 15:25:41] ...optimizers/trijoint.py.151: total_norm: 3.558905 [I 2022-10-26 15:25:46] ...trap/engines/engine.py.224: train: epoch 15 | batch 1870/2304 [I 2022-10-26 15:25:46] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:03:35 [I 2022-10-26 15:25:46] ...trap/engines/engine.py.229: process: 0.49832 | load: 0.00251 [I 2022-10-26 15:25:46] ...trap/engines/engine.py.230: loss: 0.19451 [I 2022-10-26 15:25:46] ...optimizers/trijoint.py.151: total_norm: 4.030540 [I 2022-10-26 15:25:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 1880/2304 [I 2022-10-26 15:25:51] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:03:32 [I 2022-10-26 15:25:51] ...trap/engines/engine.py.229: process: 0.49997 | load: 0.00273 [I 2022-10-26 15:25:51] ...trap/engines/engine.py.230: loss: 0.16414 [I 2022-10-26 15:25:51] ...optimizers/trijoint.py.151: total_norm: 2.515747 [I 2022-10-26 15:25:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 1890/2304 [I 2022-10-26 15:25:56] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:03:26 [I 2022-10-26 15:25:56] ...trap/engines/engine.py.229: process: 0.49750 | load: 0.00247 [I 2022-10-26 15:25:56] ...trap/engines/engine.py.230: loss: 0.20771 [I 2022-10-26 15:25:56] ...optimizers/trijoint.py.151: total_norm: 3.449088 [I 2022-10-26 15:26:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 1900/2304 [I 2022-10-26 15:26:01] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:03:32 [I 2022-10-26 15:26:01] ...trap/engines/engine.py.229: process: 0.58497 | load: 0.00265 [I 2022-10-26 15:26:01] ...trap/engines/engine.py.230: loss: 0.24089 [I 2022-10-26 15:26:01] ...optimizers/trijoint.py.151: total_norm: 2.824074 [I 2022-10-26 15:26:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 1910/2304 [I 2022-10-26 15:26:06] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:03:17 [I 2022-10-26 15:26:06] ...trap/engines/engine.py.229: process: 0.50092 | load: 0.00258 [I 2022-10-26 15:26:06] ...trap/engines/engine.py.230: loss: 0.34797 [I 2022-10-26 15:26:06] ...optimizers/trijoint.py.151: total_norm: 3.424312 [I 2022-10-26 15:26:11] ...trap/engines/engine.py.224: train: epoch 15 | batch 1920/2304 [I 2022-10-26 15:26:12] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:03:12 [I 2022-10-26 15:26:12] ...trap/engines/engine.py.229: process: 0.50200 | load: 0.00253 [I 2022-10-26 15:26:12] ...trap/engines/engine.py.230: loss: 0.28707 [I 2022-10-26 15:26:12] ...optimizers/trijoint.py.151: total_norm: 4.953185 [I 2022-10-26 15:26:17] ...trap/engines/engine.py.224: train: epoch 15 | batch 1930/2304 [I 2022-10-26 15:26:17] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:03:06 [I 2022-10-26 15:26:17] ...trap/engines/engine.py.229: process: 0.49627 | load: 0.00260 [I 2022-10-26 15:26:17] ...trap/engines/engine.py.230: loss: 0.24725 [I 2022-10-26 15:26:17] ...optimizers/trijoint.py.151: total_norm: 3.228624 [I 2022-10-26 15:26:21] ...trap/engines/engine.py.224: train: epoch 15 | batch 1940/2304 [I 2022-10-26 15:26:21] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:03:00 [I 2022-10-26 15:26:21] ...trap/engines/engine.py.229: process: 0.48587 | load: 0.00254 [I 2022-10-26 15:26:21] ...trap/engines/engine.py.230: loss: 0.21683 [I 2022-10-26 15:26:21] ...optimizers/trijoint.py.151: total_norm: 3.354845 [I 2022-10-26 15:26:26] ...trap/engines/engine.py.224: train: epoch 15 | batch 1950/2304 [I 2022-10-26 15:26:26] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:02:57 [I 2022-10-26 15:26:26] ...trap/engines/engine.py.229: process: 0.52283 | load: 0.00255 [I 2022-10-26 15:26:26] ...trap/engines/engine.py.230: loss: 0.19088 [I 2022-10-26 15:26:27] ...optimizers/trijoint.py.151: total_norm: 4.683494 [I 2022-10-26 15:26:32] ...trap/engines/engine.py.224: train: epoch 15 | batch 1960/2304 [I 2022-10-26 15:26:32] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:02:51 [I 2022-10-26 15:26:32] ...trap/engines/engine.py.229: process: 0.49427 | load: 0.00258 [I 2022-10-26 15:26:32] ...trap/engines/engine.py.230: loss: 0.20873 [I 2022-10-26 15:26:32] ...optimizers/trijoint.py.151: total_norm: 2.986960 [I 2022-10-26 15:26:36] ...trap/engines/engine.py.224: train: epoch 15 | batch 1970/2304 [I 2022-10-26 15:26:36] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:02:47 [I 2022-10-26 15:26:36] ...trap/engines/engine.py.229: process: 0.51201 | load: 0.00258 [I 2022-10-26 15:26:36] ...trap/engines/engine.py.230: loss: 0.19258 [I 2022-10-26 15:26:36] ...optimizers/trijoint.py.151: total_norm: 3.852921 [I 2022-10-26 15:26:42] ...trap/engines/engine.py.224: train: epoch 15 | batch 1980/2304 [I 2022-10-26 15:26:42] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:02:42 [I 2022-10-26 15:26:42] ...trap/engines/engine.py.229: process: 0.48013 | load: 0.00251 [I 2022-10-26 15:26:42] ...trap/engines/engine.py.230: loss: 0.19843 [I 2022-10-26 15:26:42] ...optimizers/trijoint.py.151: total_norm: 3.496009 [I 2022-10-26 15:26:47] ...trap/engines/engine.py.224: train: epoch 15 | batch 1990/2304 [I 2022-10-26 15:26:47] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:02:36 [I 2022-10-26 15:26:47] ...trap/engines/engine.py.229: process: 0.49753 | load: 0.00258 [I 2022-10-26 15:26:47] ...trap/engines/engine.py.230: loss: 0.18685 [I 2022-10-26 15:26:47] ...optimizers/trijoint.py.151: total_norm: 3.353120 [I 2022-10-26 15:26:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 2000/2304 [I 2022-10-26 15:26:51] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:02:30 [I 2022-10-26 15:26:51] ...trap/engines/engine.py.229: process: 0.49205 | load: 0.00256 [I 2022-10-26 15:26:51] ...trap/engines/engine.py.230: loss: 0.20836 [I 2022-10-26 15:26:51] ...optimizers/trijoint.py.151: total_norm: 3.747937 [I 2022-10-26 15:26:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 2010/2304 [I 2022-10-26 15:26:56] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:02:24 [I 2022-10-26 15:26:56] ...trap/engines/engine.py.229: process: 0.48716 | load: 0.00251 [I 2022-10-26 15:26:56] ...trap/engines/engine.py.230: loss: 0.18336 [I 2022-10-26 15:26:56] ...optimizers/trijoint.py.151: total_norm: 3.315294 [I 2022-10-26 15:27:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 2020/2304 [I 2022-10-26 15:27:01] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:02:21 [I 2022-10-26 15:27:01] ...trap/engines/engine.py.229: process: 0.49789 | load: 0.00260 [I 2022-10-26 15:27:01] ...trap/engines/engine.py.230: loss: 0.17765 [I 2022-10-26 15:27:01] ...optimizers/trijoint.py.151: total_norm: 2.834208 [I 2022-10-26 15:27:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 2030/2304 [I 2022-10-26 15:27:06] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:02:16 [I 2022-10-26 15:27:06] ...trap/engines/engine.py.229: process: 0.49452 | load: 0.00261 [I 2022-10-26 15:27:06] ...trap/engines/engine.py.230: loss: 0.30039 [I 2022-10-26 15:27:06] ...optimizers/trijoint.py.151: total_norm: 4.010159 [I 2022-10-26 15:27:11] ...trap/engines/engine.py.224: train: epoch 15 | batch 2040/2304 [I 2022-10-26 15:27:11] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:02:11 [I 2022-10-26 15:27:11] ...trap/engines/engine.py.229: process: 0.50604 | load: 0.00250 [I 2022-10-26 15:27:11] ...trap/engines/engine.py.230: loss: 0.23289 [I 2022-10-26 15:27:11] ...optimizers/trijoint.py.151: total_norm: 3.638958 [I 2022-10-26 15:27:16] ...trap/engines/engine.py.224: train: epoch 15 | batch 2050/2304 [I 2022-10-26 15:27:16] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:02:06 [I 2022-10-26 15:27:16] ...trap/engines/engine.py.229: process: 0.49991 | load: 0.00248 [I 2022-10-26 15:27:16] ...trap/engines/engine.py.230: loss: 0.19622 [I 2022-10-26 15:27:16] ...optimizers/trijoint.py.151: total_norm: 2.849100 [I 2022-10-26 15:27:21] ...trap/engines/engine.py.224: train: epoch 15 | batch 2060/2304 [I 2022-10-26 15:27:21] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:02:01 [I 2022-10-26 15:27:21] ...trap/engines/engine.py.229: process: 0.50290 | load: 0.00268 [I 2022-10-26 15:27:21] ...trap/engines/engine.py.230: loss: 0.25812 [I 2022-10-26 15:27:21] ...optimizers/trijoint.py.151: total_norm: 3.639091 [I 2022-10-26 15:27:26] ...trap/engines/engine.py.224: train: epoch 15 | batch 2070/2304 [I 2022-10-26 15:27:26] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:01:56 [I 2022-10-26 15:27:26] ...trap/engines/engine.py.229: process: 0.49299 | load: 0.00252 [I 2022-10-26 15:27:26] ...trap/engines/engine.py.230: loss: 0.18174 [I 2022-10-26 15:27:26] ...optimizers/trijoint.py.151: total_norm: 2.162502 [I 2022-10-26 15:27:31] ...trap/engines/engine.py.224: train: epoch 15 | batch 2080/2304 [I 2022-10-26 15:27:31] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:01:51 [I 2022-10-26 15:27:31] ...trap/engines/engine.py.229: process: 0.49953 | load: 0.00245 [I 2022-10-26 15:27:31] ...trap/engines/engine.py.230: loss: 0.28688 [I 2022-10-26 15:27:31] ...optimizers/trijoint.py.151: total_norm: 3.937706 [I 2022-10-26 15:27:36] ...trap/engines/engine.py.224: train: epoch 15 | batch 2090/2304 [I 2022-10-26 15:27:36] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:01:45 [I 2022-10-26 15:27:36] ...trap/engines/engine.py.229: process: 0.49497 | load: 0.00250 [I 2022-10-26 15:27:36] ...trap/engines/engine.py.230: loss: 0.21178 [I 2022-10-26 15:27:36] ...optimizers/trijoint.py.151: total_norm: 3.092003 [I 2022-10-26 15:27:41] ...trap/engines/engine.py.224: train: epoch 15 | batch 2100/2304 [I 2022-10-26 15:27:41] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:01:43 [I 2022-10-26 15:27:41] ...trap/engines/engine.py.229: process: 0.50248 | load: 0.00251 [I 2022-10-26 15:27:41] ...trap/engines/engine.py.230: loss: 0.18797 [I 2022-10-26 15:27:41] ...optimizers/trijoint.py.151: total_norm: 3.053459 [I 2022-10-26 15:27:46] ...trap/engines/engine.py.224: train: epoch 15 | batch 2110/2304 [I 2022-10-26 15:27:46] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:01:36 [I 2022-10-26 15:27:46] ...trap/engines/engine.py.229: process: 0.49575 | load: 0.00244 [I 2022-10-26 15:27:46] ...trap/engines/engine.py.230: loss: 0.23489 [I 2022-10-26 15:27:46] ...optimizers/trijoint.py.151: total_norm: 3.625450 [I 2022-10-26 15:27:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 2120/2304 [I 2022-10-26 15:27:51] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:01:31 [I 2022-10-26 15:27:51] ...trap/engines/engine.py.229: process: 0.49308 | load: 0.00246 [I 2022-10-26 15:27:51] ...trap/engines/engine.py.230: loss: 0.19916 [I 2022-10-26 15:27:51] ...optimizers/trijoint.py.151: total_norm: 3.621642 [I 2022-10-26 15:27:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 2130/2304 [I 2022-10-26 15:27:56] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:01:27 [I 2022-10-26 15:27:56] ...trap/engines/engine.py.229: process: 0.50863 | load: 0.00243 [I 2022-10-26 15:27:56] ...trap/engines/engine.py.230: loss: 0.16027 [I 2022-10-26 15:27:56] ...optimizers/trijoint.py.151: total_norm: 3.440507 [I 2022-10-26 15:28:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 2140/2304 [I 2022-10-26 15:28:01] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:01:21 [I 2022-10-26 15:28:01] ...trap/engines/engine.py.229: process: 0.49535 | load: 0.00246 [I 2022-10-26 15:28:01] ...trap/engines/engine.py.230: loss: 0.14324 [I 2022-10-26 15:28:01] ...optimizers/trijoint.py.151: total_norm: 3.092787 [I 2022-10-26 15:28:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 2150/2304 [I 2022-10-26 15:28:06] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:01:16 [I 2022-10-26 15:28:06] ...trap/engines/engine.py.229: process: 0.49670 | load: 0.00244 [I 2022-10-26 15:28:06] ...trap/engines/engine.py.230: loss: 0.16306 [I 2022-10-26 15:28:06] ...optimizers/trijoint.py.151: total_norm: 2.515512 [I 2022-10-26 15:28:11] ...trap/engines/engine.py.224: train: epoch 15 | batch 2160/2304 [I 2022-10-26 15:28:11] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:01:11 [I 2022-10-26 15:28:11] ...trap/engines/engine.py.229: process: 0.48908 | load: 0.00242 [I 2022-10-26 15:28:11] ...trap/engines/engine.py.230: loss: 0.27977 [I 2022-10-26 15:28:11] ...optimizers/trijoint.py.151: total_norm: 4.407973 [I 2022-10-26 15:28:16] ...trap/engines/engine.py.224: train: epoch 15 | batch 2170/2304 [I 2022-10-26 15:28:16] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:01:06 [I 2022-10-26 15:28:16] ...trap/engines/engine.py.229: process: 0.48667 | load: 0.00247 [I 2022-10-26 15:28:16] ...trap/engines/engine.py.230: loss: 0.21867 [I 2022-10-26 15:28:16] ...optimizers/trijoint.py.151: total_norm: 5.049984 [I 2022-10-26 15:28:21] ...trap/engines/engine.py.224: train: epoch 15 | batch 2180/2304 [I 2022-10-26 15:28:21] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:01:01 [I 2022-10-26 15:28:21] ...trap/engines/engine.py.229: process: 0.49378 | load: 0.00252 [I 2022-10-26 15:28:21] ...trap/engines/engine.py.230: loss: 0.22702 [I 2022-10-26 15:28:21] ...optimizers/trijoint.py.151: total_norm: 4.224497 [I 2022-10-26 15:28:26] ...trap/engines/engine.py.224: train: epoch 15 | batch 2190/2304 [I 2022-10-26 15:28:26] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:00:56 [I 2022-10-26 15:28:26] ...trap/engines/engine.py.229: process: 0.48240 | load: 0.00246 [I 2022-10-26 15:28:26] ...trap/engines/engine.py.230: loss: 0.32421 [I 2022-10-26 15:28:26] ...optimizers/trijoint.py.151: total_norm: 2.980797 [I 2022-10-26 15:28:31] ...trap/engines/engine.py.224: train: epoch 15 | batch 2200/2304 [I 2022-10-26 15:28:31] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:00:51 [I 2022-10-26 15:28:31] ...trap/engines/engine.py.229: process: 0.48616 | load: 0.00274 [I 2022-10-26 15:28:31] ...trap/engines/engine.py.230: loss: 0.21488 [I 2022-10-26 15:28:31] ...optimizers/trijoint.py.151: total_norm: 3.918933 [I 2022-10-26 15:28:36] ...trap/engines/engine.py.224: train: epoch 15 | batch 2210/2304 [I 2022-10-26 15:28:36] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:00:46 [I 2022-10-26 15:28:36] ...trap/engines/engine.py.229: process: 0.50297 | load: 0.00250 [I 2022-10-26 15:28:36] ...trap/engines/engine.py.230: loss: 0.25814 [I 2022-10-26 15:28:36] ...optimizers/trijoint.py.151: total_norm: 3.099762 [I 2022-10-26 15:28:41] ...trap/engines/engine.py.224: train: epoch 15 | batch 2220/2304 [I 2022-10-26 15:28:41] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:00:41 [I 2022-10-26 15:28:41] ...trap/engines/engine.py.229: process: 0.48604 | load: 0.00259 [I 2022-10-26 15:28:41] ...trap/engines/engine.py.230: loss: 0.18952 [I 2022-10-26 15:28:41] ...optimizers/trijoint.py.151: total_norm: 2.669565 [I 2022-10-26 15:28:46] ...trap/engines/engine.py.224: train: epoch 15 | batch 2230/2304 [I 2022-10-26 15:28:46] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:00:37 [I 2022-10-26 15:28:46] ...trap/engines/engine.py.229: process: 0.49923 | load: 0.00283 [I 2022-10-26 15:28:46] ...trap/engines/engine.py.230: loss: 0.15358 [I 2022-10-26 15:28:46] ...optimizers/trijoint.py.151: total_norm: 2.086886 [I 2022-10-26 15:28:51] ...trap/engines/engine.py.224: train: epoch 15 | batch 2240/2304 [I 2022-10-26 15:28:51] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:00:32 [I 2022-10-26 15:28:51] ...trap/engines/engine.py.229: process: 0.49742 | load: 0.00251 [I 2022-10-26 15:28:51] ...trap/engines/engine.py.230: loss: 0.23341 [I 2022-10-26 15:28:51] ...optimizers/trijoint.py.151: total_norm: 2.857286 [I 2022-10-26 15:28:56] ...trap/engines/engine.py.224: train: epoch 15 | batch 2250/2304 [I 2022-10-26 15:28:56] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:00:26 [I 2022-10-26 15:28:56] ...trap/engines/engine.py.229: process: 0.49743 | load: 0.00256 [I 2022-10-26 15:28:56] ...trap/engines/engine.py.230: loss: 0.27964 [I 2022-10-26 15:28:56] ...optimizers/trijoint.py.151: total_norm: 3.753581 [I 2022-10-26 15:29:01] ...trap/engines/engine.py.224: train: epoch 15 | batch 2260/2304 [I 2022-10-26 15:29:01] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:00:21 [I 2022-10-26 15:29:01] ...trap/engines/engine.py.229: process: 0.48100 | load: 0.00258 [I 2022-10-26 15:29:01] ...trap/engines/engine.py.230: loss: 0.21004 [I 2022-10-26 15:29:01] ...optimizers/trijoint.py.151: total_norm: 3.091307 [I 2022-10-26 15:29:06] ...trap/engines/engine.py.224: train: epoch 15 | batch 2270/2304 [I 2022-10-26 15:29:06] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:00:16 [I 2022-10-26 15:29:06] ...trap/engines/engine.py.229: process: 0.49181 | load: 0.00261 [I 2022-10-26 15:29:06] ...trap/engines/engine.py.230: loss: 0.31968 [I 2022-10-26 15:29:06] ...optimizers/trijoint.py.151: total_norm: 4.288542 [I 2022-10-26 15:29:11] ...trap/engines/engine.py.224: train: epoch 15 | batch 2280/2304 [I 2022-10-26 15:29:11] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:00:11 [I 2022-10-26 15:29:11] ...trap/engines/engine.py.229: process: 0.50224 | load: 0.00253 [I 2022-10-26 15:29:11] ...trap/engines/engine.py.230: loss: 0.19000 [I 2022-10-26 15:29:11] ...optimizers/trijoint.py.151: total_norm: 3.604779 [I 2022-10-26 15:29:16] ...trap/engines/engine.py.224: train: epoch 15 | batch 2290/2304 [I 2022-10-26 15:29:16] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:00:06 [I 2022-10-26 15:29:16] ...trap/engines/engine.py.229: process: 0.50172 | load: 0.00249 [I 2022-10-26 15:29:16] ...trap/engines/engine.py.230: loss: 0.23183 [I 2022-10-26 15:29:16] ...optimizers/trijoint.py.151: total_norm: 3.365701 [I 2022-10-26 15:29:21] ...trap/engines/engine.py.224: train: epoch 15 | batch 2300/2304 [I 2022-10-26 15:29:21] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:00:02 [I 2022-10-26 15:29:21] ...trap/engines/engine.py.229: process: 0.49858 | load: 0.00015 [I 2022-10-26 15:29:21] ...trap/engines/engine.py.230: loss: 0.19882 [I 2022-10-26 15:29:21] ...optimizers/trijoint.py.151: total_norm: 2.943801 [I 2022-10-26 15:29:23] ...trap/engines/engine.py.224: train: epoch 15 | batch 2304/2304 [I 2022-10-26 15:29:23] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:00:00 [I 2022-10-26 15:29:23] ...trap/engines/engine.py.229: process: 0.47787 | load: 0.00015 [I 2022-10-26 15:29:23] ...trap/engines/engine.py.230: loss: 0.26449 [I 2022-10-26 15:29:23] ...optimizers/trijoint.py.151: total_norm: 4.117587 [S 2022-10-26 15:29:24] ...trap/engines/engine.py.129: train_epoch.epoch: 15 [S 2022-10-26 15:29:24] ...trap/engines/engine.py.129: train_epoch.loss: 0.2169690011735852 [I 2022-10-26 15:29:24] ...s/criterions/triplet.py.84: margin = 0.13000000000000006 [I 2022-10-26 15:29:24] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 15 [I 2022-10-26 15:29:24] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 15:29:26] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 15:29:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 15:29:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 15:29:27] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 15:29:27] ...trap/engines/engine.py.321: eval: epoch 15 | batch 0/510 [I 2022-10-26 15:29:27] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:38 [I 2022-10-26 15:29:27] ...trap/engines/engine.py.326: process: 2.78039 | load: 2.40770 [I 2022-10-26 15:29:30] ...trap/engines/engine.py.321: eval: epoch 15 | batch 10/510 [I 2022-10-26 15:29:30] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:57 [I 2022-10-26 15:29:30] ...trap/engines/engine.py.326: process: 0.32981 | load: 0.00018 [I 2022-10-26 15:29:34] ...trap/engines/engine.py.321: eval: epoch 15 | batch 20/510 [I 2022-10-26 15:29:34] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:56 [I 2022-10-26 15:29:34] ...trap/engines/engine.py.326: process: 0.32911 | load: 0.00022 [I 2022-10-26 15:29:37] ...trap/engines/engine.py.321: eval: epoch 15 | batch 30/510 [I 2022-10-26 15:29:37] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:44 [I 2022-10-26 15:29:37] ...trap/engines/engine.py.326: process: 0.35708 | load: 0.00020 [I 2022-10-26 15:29:41] ...trap/engines/engine.py.321: eval: epoch 15 | batch 40/510 [I 2022-10-26 15:29:41] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:40 [I 2022-10-26 15:29:41] ...trap/engines/engine.py.326: process: 0.33480 | load: 0.00026 [I 2022-10-26 15:29:44] ...trap/engines/engine.py.321: eval: epoch 15 | batch 50/510 [I 2022-10-26 15:29:44] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:37 [I 2022-10-26 15:29:44] ...trap/engines/engine.py.326: process: 0.32956 | load: 0.00020 [I 2022-10-26 15:29:47] ...trap/engines/engine.py.321: eval: epoch 15 | batch 60/510 [I 2022-10-26 15:29:47] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:31 [I 2022-10-26 15:29:47] ...trap/engines/engine.py.326: process: 0.35440 | load: 0.00015 [I 2022-10-26 15:29:51] ...trap/engines/engine.py.321: eval: epoch 15 | batch 70/510 [I 2022-10-26 15:29:51] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:25 [I 2022-10-26 15:29:51] ...trap/engines/engine.py.326: process: 0.32130 | load: 0.00020 [I 2022-10-26 15:29:54] ...trap/engines/engine.py.321: eval: epoch 15 | batch 80/510 [I 2022-10-26 15:29:54] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:23 [I 2022-10-26 15:29:54] ...trap/engines/engine.py.326: process: 0.33551 | load: 0.00019 [I 2022-10-26 15:29:57] ...trap/engines/engine.py.321: eval: epoch 15 | batch 90/510 [I 2022-10-26 15:29:57] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:23 [I 2022-10-26 15:29:57] ...trap/engines/engine.py.326: process: 0.35571 | load: 0.00022 [I 2022-10-26 15:30:01] ...trap/engines/engine.py.321: eval: epoch 15 | batch 100/510 [I 2022-10-26 15:30:01] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:15 [I 2022-10-26 15:30:01] ...trap/engines/engine.py.326: process: 0.28855 | load: 0.00024 [I 2022-10-26 15:30:04] ...trap/engines/engine.py.321: eval: epoch 15 | batch 110/510 [I 2022-10-26 15:30:04] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:59 [I 2022-10-26 15:30:04] ...trap/engines/engine.py.326: process: 0.29163 | load: 0.00021 [I 2022-10-26 15:30:07] ...trap/engines/engine.py.321: eval: epoch 15 | batch 120/510 [I 2022-10-26 15:30:07] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 15:30:07] ...trap/engines/engine.py.326: process: 0.29854 | load: 0.00016 [I 2022-10-26 15:30:10] ...trap/engines/engine.py.321: eval: epoch 15 | batch 130/510 [I 2022-10-26 15:30:10] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:49 [I 2022-10-26 15:30:10] ...trap/engines/engine.py.326: process: 0.28799 | load: 0.00019 [I 2022-10-26 15:30:12] ...trap/engines/engine.py.321: eval: epoch 15 | batch 140/510 [I 2022-10-26 15:30:12] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-26 15:30:12] ...trap/engines/engine.py.326: process: 0.29433 | load: 0.00014 [I 2022-10-26 15:30:15] ...trap/engines/engine.py.321: eval: epoch 15 | batch 150/510 [I 2022-10-26 15:30:15] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:44 [I 2022-10-26 15:30:15] ...trap/engines/engine.py.326: process: 0.28965 | load: 0.00019 [I 2022-10-26 15:30:18] ...trap/engines/engine.py.321: eval: epoch 15 | batch 160/510 [I 2022-10-26 15:30:18] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 15:30:18] ...trap/engines/engine.py.326: process: 0.28343 | load: 0.00014 [I 2022-10-26 15:30:21] ...trap/engines/engine.py.321: eval: epoch 15 | batch 170/510 [I 2022-10-26 15:30:21] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 15:30:21] ...trap/engines/engine.py.326: process: 0.28314 | load: 0.00016 [I 2022-10-26 15:30:24] ...trap/engines/engine.py.321: eval: epoch 15 | batch 180/510 [I 2022-10-26 15:30:24] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:34 [I 2022-10-26 15:30:24] ...trap/engines/engine.py.326: process: 0.28180 | load: 0.00020 [I 2022-10-26 15:30:27] ...trap/engines/engine.py.321: eval: epoch 15 | batch 190/510 [I 2022-10-26 15:30:27] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 15:30:27] ...trap/engines/engine.py.326: process: 0.28600 | load: 0.00015 [I 2022-10-26 15:30:30] ...trap/engines/engine.py.321: eval: epoch 15 | batch 200/510 [I 2022-10-26 15:30:30] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 15:30:30] ...trap/engines/engine.py.326: process: 0.28789 | load: 0.00016 [I 2022-10-26 15:30:33] ...trap/engines/engine.py.321: eval: epoch 15 | batch 210/510 [I 2022-10-26 15:30:33] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 15:30:33] ...trap/engines/engine.py.326: process: 0.28802 | load: 0.00020 [I 2022-10-26 15:30:36] ...trap/engines/engine.py.321: eval: epoch 15 | batch 220/510 [I 2022-10-26 15:30:36] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-26 15:30:36] ...trap/engines/engine.py.326: process: 0.29080 | load: 0.00016 [I 2022-10-26 15:30:38] ...trap/engines/engine.py.321: eval: epoch 15 | batch 230/510 [I 2022-10-26 15:30:38] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 15:30:38] ...trap/engines/engine.py.326: process: 0.28495 | load: 0.00015 [I 2022-10-26 15:30:41] ...trap/engines/engine.py.321: eval: epoch 15 | batch 240/510 [I 2022-10-26 15:30:41] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 15:30:41] ...trap/engines/engine.py.326: process: 0.27922 | load: 0.00016 [I 2022-10-26 15:30:44] ...trap/engines/engine.py.321: eval: epoch 15 | batch 250/510 [I 2022-10-26 15:30:44] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:14 [I 2022-10-26 15:30:44] ...trap/engines/engine.py.326: process: 0.28499 | load: 0.00016 [I 2022-10-26 15:30:47] ...trap/engines/engine.py.321: eval: epoch 15 | batch 260/510 [I 2022-10-26 15:30:47] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:12 [I 2022-10-26 15:30:47] ...trap/engines/engine.py.326: process: 0.29053 | load: 0.00020 [I 2022-10-26 15:30:50] ...trap/engines/engine.py.321: eval: epoch 15 | batch 270/510 [I 2022-10-26 15:30:50] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 15:30:50] ...trap/engines/engine.py.326: process: 0.28504 | load: 0.00015 [I 2022-10-26 15:30:53] ...trap/engines/engine.py.321: eval: epoch 15 | batch 280/510 [I 2022-10-26 15:30:53] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:05 [I 2022-10-26 15:30:53] ...trap/engines/engine.py.326: process: 0.27780 | load: 0.00014 [I 2022-10-26 15:30:56] ...trap/engines/engine.py.321: eval: epoch 15 | batch 290/510 [I 2022-10-26 15:30:56] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 15:30:56] ...trap/engines/engine.py.326: process: 0.29396 | load: 0.00015 [I 2022-10-26 15:30:59] ...trap/engines/engine.py.321: eval: epoch 15 | batch 300/510 [I 2022-10-26 15:30:59] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 15:30:59] ...trap/engines/engine.py.326: process: 0.28580 | load: 0.00019 [I 2022-10-26 15:31:02] ...trap/engines/engine.py.321: eval: epoch 15 | batch 310/510 [I 2022-10-26 15:31:02] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 15:31:02] ...trap/engines/engine.py.326: process: 0.28183 | load: 0.00015 [I 2022-10-26 15:31:04] ...trap/engines/engine.py.321: eval: epoch 15 | batch 320/510 [I 2022-10-26 15:31:04] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 15:31:04] ...trap/engines/engine.py.326: process: 0.28613 | load: 0.00016 [I 2022-10-26 15:31:07] ...trap/engines/engine.py.321: eval: epoch 15 | batch 330/510 [I 2022-10-26 15:31:07] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:52 [I 2022-10-26 15:31:07] ...trap/engines/engine.py.326: process: 0.29305 | load: 0.00015 [I 2022-10-26 15:31:10] ...trap/engines/engine.py.321: eval: epoch 15 | batch 340/510 [I 2022-10-26 15:31:10] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:48 [I 2022-10-26 15:31:10] ...trap/engines/engine.py.326: process: 0.29556 | load: 0.00014 [I 2022-10-26 15:31:13] ...trap/engines/engine.py.321: eval: epoch 15 | batch 350/510 [I 2022-10-26 15:31:13] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:45 [I 2022-10-26 15:31:13] ...trap/engines/engine.py.326: process: 0.28064 | load: 0.00014 [I 2022-10-26 15:31:16] ...trap/engines/engine.py.321: eval: epoch 15 | batch 360/510 [I 2022-10-26 15:31:16] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 15:31:16] ...trap/engines/engine.py.326: process: 0.28451 | load: 0.00016 [I 2022-10-26 15:31:19] ...trap/engines/engine.py.321: eval: epoch 15 | batch 370/510 [I 2022-10-26 15:31:19] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 15:31:19] ...trap/engines/engine.py.326: process: 0.29287 | load: 0.00017 [I 2022-10-26 15:31:22] ...trap/engines/engine.py.321: eval: epoch 15 | batch 380/510 [I 2022-10-26 15:31:22] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 15:31:22] ...trap/engines/engine.py.326: process: 0.28913 | load: 0.00016 [I 2022-10-26 15:31:25] ...trap/engines/engine.py.321: eval: epoch 15 | batch 390/510 [I 2022-10-26 15:31:25] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 15:31:25] ...trap/engines/engine.py.326: process: 0.28987 | load: 0.00015 [I 2022-10-26 15:31:28] ...trap/engines/engine.py.321: eval: epoch 15 | batch 400/510 [I 2022-10-26 15:31:28] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 15:31:28] ...trap/engines/engine.py.326: process: 0.28837 | load: 0.00022 [I 2022-10-26 15:31:30] ...trap/engines/engine.py.321: eval: epoch 15 | batch 410/510 [I 2022-10-26 15:31:30] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 15:31:30] ...trap/engines/engine.py.326: process: 0.29004 | load: 0.00016 [I 2022-10-26 15:31:33] ...trap/engines/engine.py.321: eval: epoch 15 | batch 420/510 [I 2022-10-26 15:31:33] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-26 15:31:33] ...trap/engines/engine.py.326: process: 0.28975 | load: 0.00016 [I 2022-10-26 15:31:36] ...trap/engines/engine.py.321: eval: epoch 15 | batch 430/510 [I 2022-10-26 15:31:36] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-26 15:31:36] ...trap/engines/engine.py.326: process: 0.28552 | load: 0.00014 [I 2022-10-26 15:31:39] ...trap/engines/engine.py.321: eval: epoch 15 | batch 440/510 [I 2022-10-26 15:31:39] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 15:31:39] ...trap/engines/engine.py.326: process: 0.28758 | load: 0.00015 [I 2022-10-26 15:31:42] ...trap/engines/engine.py.321: eval: epoch 15 | batch 450/510 [I 2022-10-26 15:31:42] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 15:31:42] ...trap/engines/engine.py.326: process: 0.29324 | load: 0.00020 [I 2022-10-26 15:31:45] ...trap/engines/engine.py.321: eval: epoch 15 | batch 460/510 [I 2022-10-26 15:31:45] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 15:31:45] ...trap/engines/engine.py.326: process: 0.29132 | load: 0.00016 [I 2022-10-26 15:31:48] ...trap/engines/engine.py.321: eval: epoch 15 | batch 470/510 [I 2022-10-26 15:31:48] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 15:31:48] ...trap/engines/engine.py.326: process: 0.27533 | load: 0.00017 [I 2022-10-26 15:31:51] ...trap/engines/engine.py.321: eval: epoch 15 | batch 480/510 [I 2022-10-26 15:31:51] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 15:31:51] ...trap/engines/engine.py.326: process: 0.29071 | load: 0.00016 [I 2022-10-26 15:31:54] ...trap/engines/engine.py.321: eval: epoch 15 | batch 490/510 [I 2022-10-26 15:31:54] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 15:31:54] ...trap/engines/engine.py.326: process: 0.29092 | load: 0.00018 [I 2022-10-26 15:31:56] ...trap/engines/engine.py.321: eval: epoch 15 | batch 500/510 [I 2022-10-26 15:31:56] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 15:31:56] ...trap/engines/engine.py.326: process: 0.28832 | load: 0.00016 [I 2022-10-26 15:31:59] ...trap/engines/engine.py.321: eval: epoch 15 | batch 510/510 [I 2022-10-26 15:31:59] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 15:31:59] ...trap/engines/engine.py.326: process: 0.28482 | load: 0.00011 [S 2022-10-26 15:32:00] ...trap/engines/engine.py.132: eval_epoch.epoch: 15 [I 2022-10-26 15:32:00] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.533800 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.815700 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.882900 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.542500 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.823200 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.885800 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009724 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012092 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010339 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008453 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012528 [S 2022-10-26 15:32:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010088 [I 2022-10-26 15:32:05] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 15:32:05] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 15:32:05] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 15:32:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 15:32:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 15:32:05] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 15:32:08] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 15:32:13] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 15:32:13] ...trap/engines/engine.py.163: Training model on trainset for epoch 16 [I 2022-10-26 15:32:13] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 15:32:16] ...trap/engines/engine.py.224: train: epoch 16 | batch 0/2304 [I 2022-10-26 15:32:16] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:47 [I 2022-10-26 15:32:16] ...trap/engines/engine.py.229: process: 2.70271 | load: 2.11109 [I 2022-10-26 15:32:16] ...trap/engines/engine.py.230: loss: 0.30237 [I 2022-10-26 15:32:16] ...optimizers/trijoint.py.151: total_norm: 3.559809 [I 2022-10-26 15:32:21] ...trap/engines/engine.py.224: train: epoch 16 | batch 10/2304 [I 2022-10-26 15:32:21] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:01 [I 2022-10-26 15:32:21] ...trap/engines/engine.py.229: process: 0.48096 | load: 0.00451 [I 2022-10-26 15:32:21] ...trap/engines/engine.py.230: loss: 0.18334 [I 2022-10-26 15:32:21] ...optimizers/trijoint.py.151: total_norm: 2.498097 [I 2022-10-26 15:32:26] ...trap/engines/engine.py.224: train: epoch 16 | batch 20/2304 [I 2022-10-26 15:32:26] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:19:58 [I 2022-10-26 15:32:26] ...trap/engines/engine.py.229: process: 0.49873 | load: 0.00341 [I 2022-10-26 15:32:26] ...trap/engines/engine.py.230: loss: 0.17261 [I 2022-10-26 15:32:26] ...optimizers/trijoint.py.151: total_norm: 4.124676 [I 2022-10-26 15:32:31] ...trap/engines/engine.py.224: train: epoch 16 | batch 30/2304 [I 2022-10-26 15:32:31] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:04 [I 2022-10-26 15:32:31] ...trap/engines/engine.py.229: process: 0.49349 | load: 0.00273 [I 2022-10-26 15:32:31] ...trap/engines/engine.py.230: loss: 0.19658 [I 2022-10-26 15:32:31] ...optimizers/trijoint.py.151: total_norm: 3.274971 [I 2022-10-26 15:32:36] ...trap/engines/engine.py.224: train: epoch 16 | batch 40/2304 [I 2022-10-26 15:32:36] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:40 [I 2022-10-26 15:32:36] ...trap/engines/engine.py.229: process: 0.48627 | load: 0.00338 [I 2022-10-26 15:32:36] ...trap/engines/engine.py.230: loss: 0.17841 [I 2022-10-26 15:32:36] ...optimizers/trijoint.py.151: total_norm: 2.831010 [I 2022-10-26 15:32:41] ...trap/engines/engine.py.224: train: epoch 16 | batch 50/2304 [I 2022-10-26 15:32:41] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:43 [I 2022-10-26 15:32:41] ...trap/engines/engine.py.229: process: 0.49330 | load: 0.00321 [I 2022-10-26 15:32:41] ...trap/engines/engine.py.230: loss: 0.18194 [I 2022-10-26 15:32:41] ...optimizers/trijoint.py.151: total_norm: 3.600533 [I 2022-10-26 15:32:46] ...trap/engines/engine.py.224: train: epoch 16 | batch 60/2304 [I 2022-10-26 15:32:46] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:40 [I 2022-10-26 15:32:46] ...trap/engines/engine.py.229: process: 0.49252 | load: 0.00274 [I 2022-10-26 15:32:46] ...trap/engines/engine.py.230: loss: 0.15811 [I 2022-10-26 15:32:46] ...optimizers/trijoint.py.151: total_norm: 2.638546 [I 2022-10-26 15:32:51] ...trap/engines/engine.py.224: train: epoch 16 | batch 70/2304 [I 2022-10-26 15:32:51] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:36 [I 2022-10-26 15:32:51] ...trap/engines/engine.py.229: process: 0.49445 | load: 0.00290 [I 2022-10-26 15:32:51] ...trap/engines/engine.py.230: loss: 0.14102 [I 2022-10-26 15:32:51] ...optimizers/trijoint.py.151: total_norm: 2.303188 [I 2022-10-26 15:32:56] ...trap/engines/engine.py.224: train: epoch 16 | batch 80/2304 [I 2022-10-26 15:32:56] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:39 [I 2022-10-26 15:32:56] ...trap/engines/engine.py.229: process: 0.50597 | load: 0.00290 [I 2022-10-26 15:32:56] ...trap/engines/engine.py.230: loss: 0.17524 [I 2022-10-26 15:32:56] ...optimizers/trijoint.py.151: total_norm: 2.947323 [I 2022-10-26 15:33:01] ...trap/engines/engine.py.224: train: epoch 16 | batch 90/2304 [I 2022-10-26 15:33:01] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:18:42 [I 2022-10-26 15:33:01] ...trap/engines/engine.py.229: process: 0.49905 | load: 0.00268 [I 2022-10-26 15:33:01] ...trap/engines/engine.py.230: loss: 0.20969 [I 2022-10-26 15:33:01] ...optimizers/trijoint.py.151: total_norm: 2.990100 [I 2022-10-26 15:33:06] ...trap/engines/engine.py.224: train: epoch 16 | batch 100/2304 [I 2022-10-26 15:33:06] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:18:25 [I 2022-10-26 15:33:06] ...trap/engines/engine.py.229: process: 0.50664 | load: 0.00272 [I 2022-10-26 15:33:06] ...trap/engines/engine.py.230: loss: 0.23516 [I 2022-10-26 15:33:06] ...optimizers/trijoint.py.151: total_norm: 3.745560 [I 2022-10-26 15:33:11] ...trap/engines/engine.py.224: train: epoch 16 | batch 110/2304 [I 2022-10-26 15:33:11] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:18:18 [I 2022-10-26 15:33:11] ...trap/engines/engine.py.229: process: 0.50237 | load: 0.00266 [I 2022-10-26 15:33:11] ...trap/engines/engine.py.230: loss: 0.32725 [I 2022-10-26 15:33:11] ...optimizers/trijoint.py.151: total_norm: 4.349564 [I 2022-10-26 15:33:16] ...trap/engines/engine.py.224: train: epoch 16 | batch 120/2304 [I 2022-10-26 15:33:16] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:18:13 [I 2022-10-26 15:33:16] ...trap/engines/engine.py.229: process: 0.52763 | load: 0.00247 [I 2022-10-26 15:33:16] ...trap/engines/engine.py.230: loss: 0.21313 [I 2022-10-26 15:33:16] ...optimizers/trijoint.py.151: total_norm: 2.996467 [I 2022-10-26 15:33:21] ...trap/engines/engine.py.224: train: epoch 16 | batch 130/2304 [I 2022-10-26 15:33:21] ...trap/engines/engine.py.228: elapsed: 0:01:07 | left: 0:18:01 [I 2022-10-26 15:33:21] ...trap/engines/engine.py.229: process: 0.49810 | load: 0.00293 [I 2022-10-26 15:33:21] ...trap/engines/engine.py.230: loss: 0.24087 [I 2022-10-26 15:33:21] ...optimizers/trijoint.py.151: total_norm: 2.842446 [I 2022-10-26 15:33:26] ...trap/engines/engine.py.224: train: epoch 16 | batch 140/2304 [I 2022-10-26 15:33:26] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:18:03 [I 2022-10-26 15:33:26] ...trap/engines/engine.py.229: process: 0.49097 | load: 0.00268 [I 2022-10-26 15:33:26] ...trap/engines/engine.py.230: loss: 0.25075 [I 2022-10-26 15:33:26] ...optimizers/trijoint.py.151: total_norm: 3.337127 [I 2022-10-26 15:33:31] ...trap/engines/engine.py.224: train: epoch 16 | batch 150/2304 [I 2022-10-26 15:33:31] ...trap/engines/engine.py.228: elapsed: 0:01:17 | left: 0:17:57 [I 2022-10-26 15:33:31] ...trap/engines/engine.py.229: process: 0.50310 | load: 0.00268 [I 2022-10-26 15:33:31] ...trap/engines/engine.py.230: loss: 0.24316 [I 2022-10-26 15:33:31] ...optimizers/trijoint.py.151: total_norm: 3.301766 [I 2022-10-26 15:33:36] ...trap/engines/engine.py.224: train: epoch 16 | batch 160/2304 [I 2022-10-26 15:33:36] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:17:48 [I 2022-10-26 15:33:36] ...trap/engines/engine.py.229: process: 0.48662 | load: 0.00262 [I 2022-10-26 15:33:36] ...trap/engines/engine.py.230: loss: 0.21819 [I 2022-10-26 15:33:36] ...optimizers/trijoint.py.151: total_norm: 3.262315 [I 2022-10-26 15:33:41] ...trap/engines/engine.py.224: train: epoch 16 | batch 170/2304 [I 2022-10-26 15:33:41] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:45 [I 2022-10-26 15:33:41] ...trap/engines/engine.py.229: process: 0.49488 | load: 0.00258 [I 2022-10-26 15:33:41] ...trap/engines/engine.py.230: loss: 0.19917 [I 2022-10-26 15:33:41] ...optimizers/trijoint.py.151: total_norm: 4.007917 [I 2022-10-26 15:33:46] ...trap/engines/engine.py.224: train: epoch 16 | batch 180/2304 [I 2022-10-26 15:33:46] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:17:49 [I 2022-10-26 15:33:46] ...trap/engines/engine.py.229: process: 0.50291 | load: 0.00243 [I 2022-10-26 15:33:46] ...trap/engines/engine.py.230: loss: 0.19150 [I 2022-10-26 15:33:46] ...optimizers/trijoint.py.151: total_norm: 3.036996 [I 2022-10-26 15:33:51] ...trap/engines/engine.py.224: train: epoch 16 | batch 190/2304 [I 2022-10-26 15:33:51] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:18:01 [I 2022-10-26 15:33:51] ...trap/engines/engine.py.229: process: 0.50138 | load: 0.00261 [I 2022-10-26 15:33:51] ...trap/engines/engine.py.230: loss: 0.28293 [I 2022-10-26 15:33:51] ...optimizers/trijoint.py.151: total_norm: 4.120136 [I 2022-10-26 15:33:56] ...trap/engines/engine.py.224: train: epoch 16 | batch 200/2304 [I 2022-10-26 15:33:56] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:26 [I 2022-10-26 15:33:56] ...trap/engines/engine.py.229: process: 0.49210 | load: 0.00267 [I 2022-10-26 15:33:56] ...trap/engines/engine.py.230: loss: 0.20763 [I 2022-10-26 15:33:56] ...optimizers/trijoint.py.151: total_norm: 3.662117 [I 2022-10-26 15:34:01] ...trap/engines/engine.py.224: train: epoch 16 | batch 210/2304 [I 2022-10-26 15:34:01] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:29 [I 2022-10-26 15:34:01] ...trap/engines/engine.py.229: process: 0.49351 | load: 0.00259 [I 2022-10-26 15:34:01] ...trap/engines/engine.py.230: loss: 0.17157 [I 2022-10-26 15:34:01] ...optimizers/trijoint.py.151: total_norm: 3.115285 [I 2022-10-26 15:34:06] ...trap/engines/engine.py.224: train: epoch 16 | batch 220/2304 [I 2022-10-26 15:34:06] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:22 [I 2022-10-26 15:34:06] ...trap/engines/engine.py.229: process: 0.49958 | load: 0.00254 [I 2022-10-26 15:34:06] ...trap/engines/engine.py.230: loss: 0.15977 [I 2022-10-26 15:34:06] ...optimizers/trijoint.py.151: total_norm: 2.179874 [I 2022-10-26 15:34:11] ...trap/engines/engine.py.224: train: epoch 16 | batch 230/2304 [I 2022-10-26 15:34:11] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:19 [I 2022-10-26 15:34:11] ...trap/engines/engine.py.229: process: 0.50485 | load: 0.00259 [I 2022-10-26 15:34:11] ...trap/engines/engine.py.230: loss: 0.15040 [I 2022-10-26 15:34:11] ...optimizers/trijoint.py.151: total_norm: 3.884128 [I 2022-10-26 15:34:17] ...trap/engines/engine.py.224: train: epoch 16 | batch 240/2304 [I 2022-10-26 15:34:17] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:13 [I 2022-10-26 15:34:17] ...trap/engines/engine.py.229: process: 0.51186 | load: 0.00257 [I 2022-10-26 15:34:17] ...trap/engines/engine.py.230: loss: 0.15272 [I 2022-10-26 15:34:17] ...optimizers/trijoint.py.151: total_norm: 2.263353 [I 2022-10-26 15:34:22] ...trap/engines/engine.py.224: train: epoch 16 | batch 250/2304 [I 2022-10-26 15:34:22] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:12 [I 2022-10-26 15:34:22] ...trap/engines/engine.py.229: process: 0.48980 | load: 0.00253 [I 2022-10-26 15:34:22] ...trap/engines/engine.py.230: loss: 0.11144 [I 2022-10-26 15:34:22] ...optimizers/trijoint.py.151: total_norm: 2.280028 [I 2022-10-26 15:34:27] ...trap/engines/engine.py.224: train: epoch 16 | batch 260/2304 [I 2022-10-26 15:34:27] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:02 [I 2022-10-26 15:34:27] ...trap/engines/engine.py.229: process: 0.50049 | load: 0.00253 [I 2022-10-26 15:34:27] ...trap/engines/engine.py.230: loss: 0.16328 [I 2022-10-26 15:34:27] ...optimizers/trijoint.py.151: total_norm: 4.883139 [I 2022-10-26 15:34:32] ...trap/engines/engine.py.224: train: epoch 16 | batch 270/2304 [I 2022-10-26 15:34:32] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:16:50 [I 2022-10-26 15:34:32] ...trap/engines/engine.py.229: process: 0.48704 | load: 0.00254 [I 2022-10-26 15:34:32] ...trap/engines/engine.py.230: loss: 0.33102 [I 2022-10-26 15:34:32] ...optimizers/trijoint.py.151: total_norm: 4.923073 [I 2022-10-26 15:34:37] ...trap/engines/engine.py.224: train: epoch 16 | batch 280/2304 [I 2022-10-26 15:34:37] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:16:47 [I 2022-10-26 15:34:37] ...trap/engines/engine.py.229: process: 0.48725 | load: 0.00246 [I 2022-10-26 15:34:37] ...trap/engines/engine.py.230: loss: 0.22595 [I 2022-10-26 15:34:37] ...optimizers/trijoint.py.151: total_norm: 2.712677 [I 2022-10-26 15:34:42] ...trap/engines/engine.py.224: train: epoch 16 | batch 290/2304 [I 2022-10-26 15:34:42] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:16:52 [I 2022-10-26 15:34:42] ...trap/engines/engine.py.229: process: 0.50428 | load: 0.00258 [I 2022-10-26 15:34:42] ...trap/engines/engine.py.230: loss: 0.24183 [I 2022-10-26 15:34:42] ...optimizers/trijoint.py.151: total_norm: 3.555689 [I 2022-10-26 15:34:47] ...trap/engines/engine.py.224: train: epoch 16 | batch 300/2304 [I 2022-10-26 15:34:47] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:16:40 [I 2022-10-26 15:34:47] ...trap/engines/engine.py.229: process: 0.50175 | load: 0.00271 [I 2022-10-26 15:34:47] ...trap/engines/engine.py.230: loss: 0.24053 [I 2022-10-26 15:34:47] ...optimizers/trijoint.py.151: total_norm: 3.204769 [I 2022-10-26 15:34:52] ...trap/engines/engine.py.224: train: epoch 16 | batch 310/2304 [I 2022-10-26 15:34:52] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:16:46 [I 2022-10-26 15:34:52] ...trap/engines/engine.py.229: process: 0.51150 | load: 0.00253 [I 2022-10-26 15:34:52] ...trap/engines/engine.py.230: loss: 0.17966 [I 2022-10-26 15:34:52] ...optimizers/trijoint.py.151: total_norm: 2.960262 [I 2022-10-26 15:34:57] ...trap/engines/engine.py.224: train: epoch 16 | batch 320/2304 [I 2022-10-26 15:34:57] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:16:28 [I 2022-10-26 15:34:57] ...trap/engines/engine.py.229: process: 0.49710 | load: 0.00248 [I 2022-10-26 15:34:57] ...trap/engines/engine.py.230: loss: 0.13597 [I 2022-10-26 15:34:57] ...optimizers/trijoint.py.151: total_norm: 2.637825 [I 2022-10-26 15:35:02] ...trap/engines/engine.py.224: train: epoch 16 | batch 330/2304 [I 2022-10-26 15:35:02] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:16:17 [I 2022-10-26 15:35:02] ...trap/engines/engine.py.229: process: 0.48963 | load: 0.00249 [I 2022-10-26 15:35:02] ...trap/engines/engine.py.230: loss: 0.21318 [I 2022-10-26 15:35:02] ...optimizers/trijoint.py.151: total_norm: 3.025523 [I 2022-10-26 15:35:07] ...trap/engines/engine.py.224: train: epoch 16 | batch 340/2304 [I 2022-10-26 15:35:07] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:16:14 [I 2022-10-26 15:35:07] ...trap/engines/engine.py.229: process: 0.49105 | load: 0.00249 [I 2022-10-26 15:35:07] ...trap/engines/engine.py.230: loss: 0.13351 [I 2022-10-26 15:35:07] ...optimizers/trijoint.py.151: total_norm: 2.623480 [I 2022-10-26 15:35:12] ...trap/engines/engine.py.224: train: epoch 16 | batch 350/2304 [I 2022-10-26 15:35:12] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:16:30 [I 2022-10-26 15:35:12] ...trap/engines/engine.py.229: process: 0.53079 | load: 0.00251 [I 2022-10-26 15:35:12] ...trap/engines/engine.py.230: loss: 0.25196 [I 2022-10-26 15:35:12] ...optimizers/trijoint.py.151: total_norm: 4.019595 [I 2022-10-26 15:35:17] ...trap/engines/engine.py.224: train: epoch 16 | batch 360/2304 [I 2022-10-26 15:35:17] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:16:17 [I 2022-10-26 15:35:17] ...trap/engines/engine.py.229: process: 0.49951 | load: 0.00255 [I 2022-10-26 15:35:17] ...trap/engines/engine.py.230: loss: 0.18535 [I 2022-10-26 15:35:17] ...optimizers/trijoint.py.151: total_norm: 2.989482 [I 2022-10-26 15:35:22] ...trap/engines/engine.py.224: train: epoch 16 | batch 370/2304 [I 2022-10-26 15:35:22] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:16:29 [I 2022-10-26 15:35:22] ...trap/engines/engine.py.229: process: 0.49868 | load: 0.00245 [I 2022-10-26 15:35:22] ...trap/engines/engine.py.230: loss: 0.24327 [I 2022-10-26 15:35:22] ...optimizers/trijoint.py.151: total_norm: 3.480300 [I 2022-10-26 15:35:27] ...trap/engines/engine.py.224: train: epoch 16 | batch 380/2304 [I 2022-10-26 15:35:27] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:15:59 [I 2022-10-26 15:35:27] ...trap/engines/engine.py.229: process: 0.49954 | load: 0.00262 [I 2022-10-26 15:35:27] ...trap/engines/engine.py.230: loss: 0.19152 [I 2022-10-26 15:35:27] ...optimizers/trijoint.py.151: total_norm: 3.740281 [I 2022-10-26 15:35:32] ...trap/engines/engine.py.224: train: epoch 16 | batch 390/2304 [I 2022-10-26 15:35:32] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:15:56 [I 2022-10-26 15:35:32] ...trap/engines/engine.py.229: process: 0.49415 | load: 0.00248 [I 2022-10-26 15:35:32] ...trap/engines/engine.py.230: loss: 0.19345 [I 2022-10-26 15:35:32] ...optimizers/trijoint.py.151: total_norm: 3.092462 [I 2022-10-26 15:35:37] ...trap/engines/engine.py.224: train: epoch 16 | batch 400/2304 [I 2022-10-26 15:35:37] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:15:54 [I 2022-10-26 15:35:37] ...trap/engines/engine.py.229: process: 0.50970 | load: 0.00247 [I 2022-10-26 15:35:37] ...trap/engines/engine.py.230: loss: 0.22435 [I 2022-10-26 15:35:37] ...optimizers/trijoint.py.151: total_norm: 4.260972 [I 2022-10-26 15:35:42] ...trap/engines/engine.py.224: train: epoch 16 | batch 410/2304 [I 2022-10-26 15:35:42] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:15:56 [I 2022-10-26 15:35:42] ...trap/engines/engine.py.229: process: 0.49484 | load: 0.00247 [I 2022-10-26 15:35:42] ...trap/engines/engine.py.230: loss: 0.18804 [I 2022-10-26 15:35:42] ...optimizers/trijoint.py.151: total_norm: 3.440934 [I 2022-10-26 15:35:47] ...trap/engines/engine.py.224: train: epoch 16 | batch 420/2304 [I 2022-10-26 15:35:47] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:15:43 [I 2022-10-26 15:35:47] ...trap/engines/engine.py.229: process: 0.49116 | load: 0.00250 [I 2022-10-26 15:35:47] ...trap/engines/engine.py.230: loss: 0.14078 [I 2022-10-26 15:35:47] ...optimizers/trijoint.py.151: total_norm: 2.979529 [I 2022-10-26 15:35:52] ...trap/engines/engine.py.224: train: epoch 16 | batch 430/2304 [I 2022-10-26 15:35:52] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:15:36 [I 2022-10-26 15:35:52] ...trap/engines/engine.py.229: process: 0.49810 | load: 0.00259 [I 2022-10-26 15:35:52] ...trap/engines/engine.py.230: loss: 0.17423 [I 2022-10-26 15:35:52] ...optimizers/trijoint.py.151: total_norm: 2.797308 [I 2022-10-26 15:35:57] ...trap/engines/engine.py.224: train: epoch 16 | batch 440/2304 [I 2022-10-26 15:35:57] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:15:23 [I 2022-10-26 15:35:57] ...trap/engines/engine.py.229: process: 0.48661 | load: 0.00246 [I 2022-10-26 15:35:57] ...trap/engines/engine.py.230: loss: 0.20842 [I 2022-10-26 15:35:57] ...optimizers/trijoint.py.151: total_norm: 3.156699 [I 2022-10-26 15:36:02] ...trap/engines/engine.py.224: train: epoch 16 | batch 450/2304 [I 2022-10-26 15:36:02] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:15:19 [I 2022-10-26 15:36:02] ...trap/engines/engine.py.229: process: 0.48841 | load: 0.00262 [I 2022-10-26 15:36:02] ...trap/engines/engine.py.230: loss: 0.25780 [I 2022-10-26 15:36:02] ...optimizers/trijoint.py.151: total_norm: 4.225583 [I 2022-10-26 15:36:28] ...trap/engines/engine.py.224: train: epoch 16 | batch 460/2304 [I 2022-10-26 15:36:28] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:42:06 [I 2022-10-26 15:36:28] ...trap/engines/engine.py.229: process: 0.50075 | load: 0.00249 [I 2022-10-26 15:36:28] ...trap/engines/engine.py.230: loss: 0.26600 [I 2022-10-26 15:36:28] ...optimizers/trijoint.py.151: total_norm: 3.744426 [I 2022-10-26 15:36:33] ...trap/engines/engine.py.224: train: epoch 16 | batch 470/2304 [I 2022-10-26 15:36:33] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:18:00 [I 2022-10-26 15:36:33] ...trap/engines/engine.py.229: process: 0.48195 | load: 0.00240 [I 2022-10-26 15:36:33] ...trap/engines/engine.py.230: loss: 0.18226 [I 2022-10-26 15:36:33] ...optimizers/trijoint.py.151: total_norm: 2.928837 [I 2022-10-26 15:36:38] ...trap/engines/engine.py.224: train: epoch 16 | batch 480/2304 [I 2022-10-26 15:36:38] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:15:26 [I 2022-10-26 15:36:38] ...trap/engines/engine.py.229: process: 0.49404 | load: 0.00237 [I 2022-10-26 15:36:38] ...trap/engines/engine.py.230: loss: 0.29094 [I 2022-10-26 15:36:38] ...optimizers/trijoint.py.151: total_norm: 3.363555 [I 2022-10-26 15:36:43] ...trap/engines/engine.py.224: train: epoch 16 | batch 490/2304 [I 2022-10-26 15:36:43] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:15:25 [I 2022-10-26 15:36:43] ...trap/engines/engine.py.229: process: 0.49469 | load: 0.00238 [I 2022-10-26 15:36:43] ...trap/engines/engine.py.230: loss: 0.21879 [I 2022-10-26 15:36:43] ...optimizers/trijoint.py.151: total_norm: 3.466786 [I 2022-10-26 15:36:48] ...trap/engines/engine.py.224: train: epoch 16 | batch 500/2304 [I 2022-10-26 15:36:48] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:15:04 [I 2022-10-26 15:36:48] ...trap/engines/engine.py.229: process: 0.50249 | load: 0.00239 [I 2022-10-26 15:36:48] ...trap/engines/engine.py.230: loss: 0.16823 [I 2022-10-26 15:36:48] ...optimizers/trijoint.py.151: total_norm: 2.797512 [I 2022-10-26 15:36:53] ...trap/engines/engine.py.224: train: epoch 16 | batch 510/2304 [I 2022-10-26 15:36:53] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:14:55 [I 2022-10-26 15:36:53] ...trap/engines/engine.py.229: process: 0.50410 | load: 0.00239 [I 2022-10-26 15:36:53] ...trap/engines/engine.py.230: loss: 0.20829 [I 2022-10-26 15:36:53] ...optimizers/trijoint.py.151: total_norm: 2.672127 [I 2022-10-26 15:36:58] ...trap/engines/engine.py.224: train: epoch 16 | batch 520/2304 [I 2022-10-26 15:36:58] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:14:43 [I 2022-10-26 15:36:58] ...trap/engines/engine.py.229: process: 0.49122 | load: 0.00240 [I 2022-10-26 15:36:58] ...trap/engines/engine.py.230: loss: 0.21428 [I 2022-10-26 15:36:58] ...optimizers/trijoint.py.151: total_norm: 3.318245 [I 2022-10-26 15:37:03] ...trap/engines/engine.py.224: train: epoch 16 | batch 530/2304 [I 2022-10-26 15:37:03] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:14:49 [I 2022-10-26 15:37:03] ...trap/engines/engine.py.229: process: 0.49142 | load: 0.00246 [I 2022-10-26 15:37:03] ...trap/engines/engine.py.230: loss: 0.21048 [I 2022-10-26 15:37:03] ...optimizers/trijoint.py.151: total_norm: 3.100849 [I 2022-10-26 15:37:08] ...trap/engines/engine.py.224: train: epoch 16 | batch 540/2304 [I 2022-10-26 15:37:08] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:14:39 [I 2022-10-26 15:37:08] ...trap/engines/engine.py.229: process: 0.49765 | load: 0.00244 [I 2022-10-26 15:37:08] ...trap/engines/engine.py.230: loss: 0.16836 [I 2022-10-26 15:37:08] ...optimizers/trijoint.py.151: total_norm: 3.247074 [I 2022-10-26 15:37:13] ...trap/engines/engine.py.224: train: epoch 16 | batch 550/2304 [I 2022-10-26 15:37:13] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:14:37 [I 2022-10-26 15:37:13] ...trap/engines/engine.py.229: process: 0.49922 | load: 0.00241 [I 2022-10-26 15:37:13] ...trap/engines/engine.py.230: loss: 0.22972 [I 2022-10-26 15:37:13] ...optimizers/trijoint.py.151: total_norm: 4.013986 [I 2022-10-26 15:37:18] ...trap/engines/engine.py.224: train: epoch 16 | batch 560/2304 [I 2022-10-26 15:37:18] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:14:29 [I 2022-10-26 15:37:18] ...trap/engines/engine.py.229: process: 0.50738 | load: 0.00259 [I 2022-10-26 15:37:18] ...trap/engines/engine.py.230: loss: 0.27395 [I 2022-10-26 15:37:18] ...optimizers/trijoint.py.151: total_norm: 3.859357 [I 2022-10-26 15:37:23] ...trap/engines/engine.py.224: train: epoch 16 | batch 570/2304 [I 2022-10-26 15:37:23] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:14:24 [I 2022-10-26 15:37:23] ...trap/engines/engine.py.229: process: 0.49251 | load: 0.00241 [I 2022-10-26 15:37:23] ...trap/engines/engine.py.230: loss: 0.20827 [I 2022-10-26 15:37:23] ...optimizers/trijoint.py.151: total_norm: 3.779291 [I 2022-10-26 15:37:28] ...trap/engines/engine.py.224: train: epoch 16 | batch 580/2304 [I 2022-10-26 15:37:28] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:14:19 [I 2022-10-26 15:37:28] ...trap/engines/engine.py.229: process: 0.50907 | load: 0.00244 [I 2022-10-26 15:37:28] ...trap/engines/engine.py.230: loss: 0.25644 [I 2022-10-26 15:37:28] ...optimizers/trijoint.py.151: total_norm: 4.985549 [I 2022-10-26 15:37:33] ...trap/engines/engine.py.224: train: epoch 16 | batch 590/2304 [I 2022-10-26 15:37:33] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:14:23 [I 2022-10-26 15:37:33] ...trap/engines/engine.py.229: process: 0.50636 | load: 0.00245 [I 2022-10-26 15:37:33] ...trap/engines/engine.py.230: loss: 0.24981 [I 2022-10-26 15:37:33] ...optimizers/trijoint.py.151: total_norm: 3.355480 [I 2022-10-26 15:37:38] ...trap/engines/engine.py.224: train: epoch 16 | batch 600/2304 [I 2022-10-26 15:37:38] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:14:09 [I 2022-10-26 15:37:38] ...trap/engines/engine.py.229: process: 0.49565 | load: 0.00242 [I 2022-10-26 15:37:38] ...trap/engines/engine.py.230: loss: 0.32594 [I 2022-10-26 15:37:38] ...optimizers/trijoint.py.151: total_norm: 3.830582 [I 2022-10-26 15:37:43] ...trap/engines/engine.py.224: train: epoch 16 | batch 610/2304 [I 2022-10-26 15:37:43] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:14:19 [I 2022-10-26 15:37:43] ...trap/engines/engine.py.229: process: 0.49821 | load: 0.00240 [I 2022-10-26 15:37:43] ...trap/engines/engine.py.230: loss: 0.25222 [I 2022-10-26 15:37:43] ...optimizers/trijoint.py.151: total_norm: 3.459055 [I 2022-10-26 15:37:48] ...trap/engines/engine.py.224: train: epoch 16 | batch 620/2304 [I 2022-10-26 15:37:48] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:14:04 [I 2022-10-26 15:37:48] ...trap/engines/engine.py.229: process: 0.51458 | load: 0.00240 [I 2022-10-26 15:37:48] ...trap/engines/engine.py.230: loss: 0.22916 [I 2022-10-26 15:37:48] ...optimizers/trijoint.py.151: total_norm: 3.345526 [I 2022-10-26 15:37:53] ...trap/engines/engine.py.224: train: epoch 16 | batch 630/2304 [I 2022-10-26 15:37:53] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:13:58 [I 2022-10-26 15:37:53] ...trap/engines/engine.py.229: process: 0.49918 | load: 0.00238 [I 2022-10-26 15:37:53] ...trap/engines/engine.py.230: loss: 0.24420 [I 2022-10-26 15:37:53] ...optimizers/trijoint.py.151: total_norm: 4.683006 [I 2022-10-26 15:37:58] ...trap/engines/engine.py.224: train: epoch 16 | batch 640/2304 [I 2022-10-26 15:37:58] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:13:52 [I 2022-10-26 15:37:58] ...trap/engines/engine.py.229: process: 0.50226 | load: 0.00249 [I 2022-10-26 15:37:58] ...trap/engines/engine.py.230: loss: 0.16947 [I 2022-10-26 15:37:58] ...optimizers/trijoint.py.151: total_norm: 3.181618 [I 2022-10-26 15:38:03] ...trap/engines/engine.py.224: train: epoch 16 | batch 650/2304 [I 2022-10-26 15:38:03] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:13:48 [I 2022-10-26 15:38:03] ...trap/engines/engine.py.229: process: 0.50332 | load: 0.00243 [I 2022-10-26 15:38:03] ...trap/engines/engine.py.230: loss: 0.27067 [I 2022-10-26 15:38:03] ...optimizers/trijoint.py.151: total_norm: 4.965039 [I 2022-10-26 15:38:08] ...trap/engines/engine.py.224: train: epoch 16 | batch 660/2304 [I 2022-10-26 15:38:08] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:13:42 [I 2022-10-26 15:38:08] ...trap/engines/engine.py.229: process: 0.49941 | load: 0.00244 [I 2022-10-26 15:38:08] ...trap/engines/engine.py.230: loss: 0.24921 [I 2022-10-26 15:38:08] ...optimizers/trijoint.py.151: total_norm: 3.273661 [I 2022-10-26 15:38:13] ...trap/engines/engine.py.224: train: epoch 16 | batch 670/2304 [I 2022-10-26 15:38:13] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:13:38 [I 2022-10-26 15:38:13] ...trap/engines/engine.py.229: process: 0.50128 | load: 0.00241 [I 2022-10-26 15:38:13] ...trap/engines/engine.py.230: loss: 0.15983 [I 2022-10-26 15:38:13] ...optimizers/trijoint.py.151: total_norm: 2.981738 [I 2022-10-26 15:38:18] ...trap/engines/engine.py.224: train: epoch 16 | batch 680/2304 [I 2022-10-26 15:38:18] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:13:32 [I 2022-10-26 15:38:18] ...trap/engines/engine.py.229: process: 0.50037 | load: 0.00243 [I 2022-10-26 15:38:18] ...trap/engines/engine.py.230: loss: 0.20757 [I 2022-10-26 15:38:18] ...optimizers/trijoint.py.151: total_norm: 3.011181 [I 2022-10-26 15:38:23] ...trap/engines/engine.py.224: train: epoch 16 | batch 690/2304 [I 2022-10-26 15:38:23] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:13:29 [I 2022-10-26 15:38:23] ...trap/engines/engine.py.229: process: 0.49937 | load: 0.00246 [I 2022-10-26 15:38:23] ...trap/engines/engine.py.230: loss: 0.18410 [I 2022-10-26 15:38:23] ...optimizers/trijoint.py.151: total_norm: 3.660329 [I 2022-10-26 15:38:28] ...trap/engines/engine.py.224: train: epoch 16 | batch 700/2304 [I 2022-10-26 15:38:28] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:13:20 [I 2022-10-26 15:38:28] ...trap/engines/engine.py.229: process: 0.50270 | load: 0.00243 [I 2022-10-26 15:38:28] ...trap/engines/engine.py.230: loss: 0.20298 [I 2022-10-26 15:38:28] ...optimizers/trijoint.py.151: total_norm: 2.609933 [I 2022-10-26 15:38:33] ...trap/engines/engine.py.224: train: epoch 16 | batch 710/2304 [I 2022-10-26 15:38:33] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:13:15 [I 2022-10-26 15:38:33] ...trap/engines/engine.py.229: process: 0.49266 | load: 0.00241 [I 2022-10-26 15:38:33] ...trap/engines/engine.py.230: loss: 0.18616 [I 2022-10-26 15:38:33] ...optimizers/trijoint.py.151: total_norm: 2.299101 [I 2022-10-26 15:38:38] ...trap/engines/engine.py.224: train: epoch 16 | batch 720/2304 [I 2022-10-26 15:38:38] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:13:17 [I 2022-10-26 15:38:38] ...trap/engines/engine.py.229: process: 0.50922 | load: 0.00243 [I 2022-10-26 15:38:38] ...trap/engines/engine.py.230: loss: 0.23897 [I 2022-10-26 15:38:38] ...optimizers/trijoint.py.151: total_norm: 4.078743 [I 2022-10-26 15:38:43] ...trap/engines/engine.py.224: train: epoch 16 | batch 730/2304 [I 2022-10-26 15:38:43] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:13:11 [I 2022-10-26 15:38:43] ...trap/engines/engine.py.229: process: 0.50861 | load: 0.00247 [I 2022-10-26 15:38:43] ...trap/engines/engine.py.230: loss: 0.15287 [I 2022-10-26 15:38:43] ...optimizers/trijoint.py.151: total_norm: 2.366051 [I 2022-10-26 15:38:48] ...trap/engines/engine.py.224: train: epoch 16 | batch 740/2304 [I 2022-10-26 15:38:48] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:13:26 [I 2022-10-26 15:38:48] ...trap/engines/engine.py.229: process: 0.50679 | load: 0.00247 [I 2022-10-26 15:38:48] ...trap/engines/engine.py.230: loss: 0.16507 [I 2022-10-26 15:38:48] ...optimizers/trijoint.py.151: total_norm: 3.399854 [I 2022-10-26 15:38:53] ...trap/engines/engine.py.224: train: epoch 16 | batch 750/2304 [I 2022-10-26 15:38:53] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:12:55 [I 2022-10-26 15:38:53] ...trap/engines/engine.py.229: process: 0.49123 | load: 0.00245 [I 2022-10-26 15:38:53] ...trap/engines/engine.py.230: loss: 0.30632 [I 2022-10-26 15:38:53] ...optimizers/trijoint.py.151: total_norm: 3.615668 [I 2022-10-26 15:38:58] ...trap/engines/engine.py.224: train: epoch 16 | batch 760/2304 [I 2022-10-26 15:38:58] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:12:41 [I 2022-10-26 15:38:58] ...trap/engines/engine.py.229: process: 0.48872 | load: 0.00245 [I 2022-10-26 15:38:58] ...trap/engines/engine.py.230: loss: 0.19571 [I 2022-10-26 15:38:58] ...optimizers/trijoint.py.151: total_norm: 2.692567 [I 2022-10-26 15:39:03] ...trap/engines/engine.py.224: train: epoch 16 | batch 770/2304 [I 2022-10-26 15:39:03] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:12:39 [I 2022-10-26 15:39:03] ...trap/engines/engine.py.229: process: 0.49814 | load: 0.00243 [I 2022-10-26 15:39:03] ...trap/engines/engine.py.230: loss: 0.22204 [I 2022-10-26 15:39:03] ...optimizers/trijoint.py.151: total_norm: 2.747065 [I 2022-10-26 15:39:08] ...trap/engines/engine.py.224: train: epoch 16 | batch 780/2304 [I 2022-10-26 15:39:08] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:12:42 [I 2022-10-26 15:39:08] ...trap/engines/engine.py.229: process: 0.50248 | load: 0.00244 [I 2022-10-26 15:39:08] ...trap/engines/engine.py.230: loss: 0.23410 [I 2022-10-26 15:39:08] ...optimizers/trijoint.py.151: total_norm: 2.893433 [I 2022-10-26 15:39:13] ...trap/engines/engine.py.224: train: epoch 16 | batch 790/2304 [I 2022-10-26 15:39:13] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:12:41 [I 2022-10-26 15:39:13] ...trap/engines/engine.py.229: process: 0.51126 | load: 0.00244 [I 2022-10-26 15:39:13] ...trap/engines/engine.py.230: loss: 0.18552 [I 2022-10-26 15:39:13] ...optimizers/trijoint.py.151: total_norm: 2.756703 [I 2022-10-26 15:39:18] ...trap/engines/engine.py.224: train: epoch 16 | batch 800/2304 [I 2022-10-26 15:39:18] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:12:30 [I 2022-10-26 15:39:18] ...trap/engines/engine.py.229: process: 0.50518 | load: 0.00246 [I 2022-10-26 15:39:18] ...trap/engines/engine.py.230: loss: 0.22856 [I 2022-10-26 15:39:18] ...optimizers/trijoint.py.151: total_norm: 4.151983 [I 2022-10-26 15:39:23] ...trap/engines/engine.py.224: train: epoch 16 | batch 810/2304 [I 2022-10-26 15:39:23] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:12:26 [I 2022-10-26 15:39:23] ...trap/engines/engine.py.229: process: 0.50288 | load: 0.00244 [I 2022-10-26 15:39:23] ...trap/engines/engine.py.230: loss: 0.21943 [I 2022-10-26 15:39:23] ...optimizers/trijoint.py.151: total_norm: 3.193926 [I 2022-10-26 15:39:28] ...trap/engines/engine.py.224: train: epoch 16 | batch 820/2304 [I 2022-10-26 15:39:28] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:12:20 [I 2022-10-26 15:39:28] ...trap/engines/engine.py.229: process: 0.50285 | load: 0.00252 [I 2022-10-26 15:39:28] ...trap/engines/engine.py.230: loss: 0.20128 [I 2022-10-26 15:39:28] ...optimizers/trijoint.py.151: total_norm: 2.736710 [I 2022-10-26 15:39:33] ...trap/engines/engine.py.224: train: epoch 16 | batch 830/2304 [I 2022-10-26 15:39:33] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:12:15 [I 2022-10-26 15:39:33] ...trap/engines/engine.py.229: process: 0.48523 | load: 0.00267 [I 2022-10-26 15:39:33] ...trap/engines/engine.py.230: loss: 0.16752 [I 2022-10-26 15:39:33] ...optimizers/trijoint.py.151: total_norm: 2.203920 [I 2022-10-26 15:39:38] ...trap/engines/engine.py.224: train: epoch 16 | batch 840/2304 [I 2022-10-26 15:39:38] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:12:04 [I 2022-10-26 15:39:38] ...trap/engines/engine.py.229: process: 0.49172 | load: 0.00248 [I 2022-10-26 15:39:38] ...trap/engines/engine.py.230: loss: 0.29125 [I 2022-10-26 15:39:38] ...optimizers/trijoint.py.151: total_norm: 4.049167 [I 2022-10-26 15:39:43] ...trap/engines/engine.py.224: train: epoch 16 | batch 850/2304 [I 2022-10-26 15:39:43] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:12:02 [I 2022-10-26 15:39:43] ...trap/engines/engine.py.229: process: 0.50605 | load: 0.00246 [I 2022-10-26 15:39:43] ...trap/engines/engine.py.230: loss: 0.29428 [I 2022-10-26 15:39:43] ...optimizers/trijoint.py.151: total_norm: 3.256141 [I 2022-10-26 15:39:48] ...trap/engines/engine.py.224: train: epoch 16 | batch 860/2304 [I 2022-10-26 15:39:48] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:12:00 [I 2022-10-26 15:39:48] ...trap/engines/engine.py.229: process: 0.49117 | load: 0.00245 [I 2022-10-26 15:39:48] ...trap/engines/engine.py.230: loss: 0.28670 [I 2022-10-26 15:39:48] ...optimizers/trijoint.py.151: total_norm: 3.167942 [I 2022-10-26 15:39:53] ...trap/engines/engine.py.224: train: epoch 16 | batch 870/2304 [I 2022-10-26 15:39:53] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:12:08 [I 2022-10-26 15:39:53] ...trap/engines/engine.py.229: process: 0.49303 | load: 0.00242 [I 2022-10-26 15:39:53] ...trap/engines/engine.py.230: loss: 0.20184 [I 2022-10-26 15:39:53] ...optimizers/trijoint.py.151: total_norm: 3.359247 [I 2022-10-26 15:39:58] ...trap/engines/engine.py.224: train: epoch 16 | batch 880/2304 [I 2022-10-26 15:39:58] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:11:53 [I 2022-10-26 15:39:58] ...trap/engines/engine.py.229: process: 0.50252 | load: 0.00242 [I 2022-10-26 15:39:58] ...trap/engines/engine.py.230: loss: 0.18243 [I 2022-10-26 15:39:58] ...optimizers/trijoint.py.151: total_norm: 3.022567 [I 2022-10-26 15:40:03] ...trap/engines/engine.py.224: train: epoch 16 | batch 890/2304 [I 2022-10-26 15:40:03] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:11:45 [I 2022-10-26 15:40:03] ...trap/engines/engine.py.229: process: 0.49391 | load: 0.00245 [I 2022-10-26 15:40:03] ...trap/engines/engine.py.230: loss: 0.29161 [I 2022-10-26 15:40:03] ...optimizers/trijoint.py.151: total_norm: 3.654788 [I 2022-10-26 15:40:08] ...trap/engines/engine.py.224: train: epoch 16 | batch 900/2304 [I 2022-10-26 15:40:08] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:11:43 [I 2022-10-26 15:40:08] ...trap/engines/engine.py.229: process: 0.49933 | load: 0.00242 [I 2022-10-26 15:40:08] ...trap/engines/engine.py.230: loss: 0.17060 [I 2022-10-26 15:40:08] ...optimizers/trijoint.py.151: total_norm: 2.503922 [I 2022-10-26 15:40:13] ...trap/engines/engine.py.224: train: epoch 16 | batch 910/2304 [I 2022-10-26 15:40:13] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:11:38 [I 2022-10-26 15:40:13] ...trap/engines/engine.py.229: process: 0.49963 | load: 0.00243 [I 2022-10-26 15:40:13] ...trap/engines/engine.py.230: loss: 0.21271 [I 2022-10-26 15:40:13] ...optimizers/trijoint.py.151: total_norm: 3.746015 [I 2022-10-26 15:40:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 920/2304 [I 2022-10-26 15:40:19] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:11:37 [I 2022-10-26 15:40:19] ...trap/engines/engine.py.229: process: 0.51201 | load: 0.00239 [I 2022-10-26 15:40:19] ...trap/engines/engine.py.230: loss: 0.25832 [I 2022-10-26 15:40:19] ...optimizers/trijoint.py.151: total_norm: 3.919728 [I 2022-10-26 15:40:24] ...trap/engines/engine.py.224: train: epoch 16 | batch 930/2304 [I 2022-10-26 15:40:24] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:11:26 [I 2022-10-26 15:40:24] ...trap/engines/engine.py.229: process: 0.48844 | load: 0.00256 [I 2022-10-26 15:40:24] ...trap/engines/engine.py.230: loss: 0.13261 [I 2022-10-26 15:40:24] ...optimizers/trijoint.py.151: total_norm: 1.899432 [I 2022-10-26 15:40:29] ...trap/engines/engine.py.224: train: epoch 16 | batch 940/2304 [I 2022-10-26 15:40:29] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:11:22 [I 2022-10-26 15:40:29] ...trap/engines/engine.py.229: process: 0.49686 | load: 0.00248 [I 2022-10-26 15:40:29] ...trap/engines/engine.py.230: loss: 0.18537 [I 2022-10-26 15:40:29] ...optimizers/trijoint.py.151: total_norm: 2.176925 [I 2022-10-26 15:40:34] ...trap/engines/engine.py.224: train: epoch 16 | batch 950/2304 [I 2022-10-26 15:40:34] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:11:13 [I 2022-10-26 15:40:34] ...trap/engines/engine.py.229: process: 0.49051 | load: 0.00241 [I 2022-10-26 15:40:34] ...trap/engines/engine.py.230: loss: 0.19380 [I 2022-10-26 15:40:34] ...optimizers/trijoint.py.151: total_norm: 2.650878 [I 2022-10-26 15:40:39] ...trap/engines/engine.py.224: train: epoch 16 | batch 960/2304 [I 2022-10-26 15:40:39] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:11:10 [I 2022-10-26 15:40:39] ...trap/engines/engine.py.229: process: 0.50068 | load: 0.00262 [I 2022-10-26 15:40:39] ...trap/engines/engine.py.230: loss: 0.18590 [I 2022-10-26 15:40:39] ...optimizers/trijoint.py.151: total_norm: 3.021731 [I 2022-10-26 15:40:44] ...trap/engines/engine.py.224: train: epoch 16 | batch 970/2304 [I 2022-10-26 15:40:44] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:11:04 [I 2022-10-26 15:40:44] ...trap/engines/engine.py.229: process: 0.49290 | load: 0.00244 [I 2022-10-26 15:40:44] ...trap/engines/engine.py.230: loss: 0.22101 [I 2022-10-26 15:40:44] ...optimizers/trijoint.py.151: total_norm: 2.881764 [I 2022-10-26 15:40:49] ...trap/engines/engine.py.224: train: epoch 16 | batch 980/2304 [I 2022-10-26 15:40:49] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:11:01 [I 2022-10-26 15:40:49] ...trap/engines/engine.py.229: process: 0.50019 | load: 0.00241 [I 2022-10-26 15:40:49] ...trap/engines/engine.py.230: loss: 0.15954 [I 2022-10-26 15:40:49] ...optimizers/trijoint.py.151: total_norm: 2.697595 [I 2022-10-26 15:40:54] ...trap/engines/engine.py.224: train: epoch 16 | batch 990/2304 [I 2022-10-26 15:40:54] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:10:54 [I 2022-10-26 15:40:54] ...trap/engines/engine.py.229: process: 0.50150 | load: 0.00243 [I 2022-10-26 15:40:54] ...trap/engines/engine.py.230: loss: 0.16919 [I 2022-10-26 15:40:54] ...optimizers/trijoint.py.151: total_norm: 3.038031 [I 2022-10-26 15:40:59] ...trap/engines/engine.py.224: train: epoch 16 | batch 1000/2304 [I 2022-10-26 15:40:59] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:10:47 [I 2022-10-26 15:40:59] ...trap/engines/engine.py.229: process: 0.48209 | load: 0.00243 [I 2022-10-26 15:40:59] ...trap/engines/engine.py.230: loss: 0.28585 [I 2022-10-26 15:40:59] ...optimizers/trijoint.py.151: total_norm: 3.894969 [I 2022-10-26 15:41:04] ...trap/engines/engine.py.224: train: epoch 16 | batch 1010/2304 [I 2022-10-26 15:41:04] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:10:44 [I 2022-10-26 15:41:04] ...trap/engines/engine.py.229: process: 0.48386 | load: 0.00243 [I 2022-10-26 15:41:04] ...trap/engines/engine.py.230: loss: 0.19566 [I 2022-10-26 15:41:04] ...optimizers/trijoint.py.151: total_norm: 2.573919 [I 2022-10-26 15:41:09] ...trap/engines/engine.py.224: train: epoch 16 | batch 1020/2304 [I 2022-10-26 15:41:09] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:10:40 [I 2022-10-26 15:41:09] ...trap/engines/engine.py.229: process: 0.50657 | load: 0.00241 [I 2022-10-26 15:41:09] ...trap/engines/engine.py.230: loss: 0.28987 [I 2022-10-26 15:41:09] ...optimizers/trijoint.py.151: total_norm: 4.084439 [I 2022-10-26 15:41:14] ...trap/engines/engine.py.224: train: epoch 16 | batch 1030/2304 [I 2022-10-26 15:41:14] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:10:40 [I 2022-10-26 15:41:14] ...trap/engines/engine.py.229: process: 0.50179 | load: 0.00242 [I 2022-10-26 15:41:14] ...trap/engines/engine.py.230: loss: 0.17579 [I 2022-10-26 15:41:14] ...optimizers/trijoint.py.151: total_norm: 3.949823 [I 2022-10-26 15:41:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 1040/2304 [I 2022-10-26 15:41:19] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:10:33 [I 2022-10-26 15:41:19] ...trap/engines/engine.py.229: process: 0.50168 | load: 0.00247 [I 2022-10-26 15:41:19] ...trap/engines/engine.py.230: loss: 0.21638 [I 2022-10-26 15:41:19] ...optimizers/trijoint.py.151: total_norm: 4.019495 [I 2022-10-26 15:41:24] ...trap/engines/engine.py.224: train: epoch 16 | batch 1050/2304 [I 2022-10-26 15:41:24] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:10:32 [I 2022-10-26 15:41:24] ...trap/engines/engine.py.229: process: 0.50724 | load: 0.00246 [I 2022-10-26 15:41:24] ...trap/engines/engine.py.230: loss: 0.26888 [I 2022-10-26 15:41:24] ...optimizers/trijoint.py.151: total_norm: 3.828030 [I 2022-10-26 15:41:29] ...trap/engines/engine.py.224: train: epoch 16 | batch 1060/2304 [I 2022-10-26 15:41:29] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:10:27 [I 2022-10-26 15:41:29] ...trap/engines/engine.py.229: process: 0.51187 | load: 0.00243 [I 2022-10-26 15:41:29] ...trap/engines/engine.py.230: loss: 0.29369 [I 2022-10-26 15:41:29] ...optimizers/trijoint.py.151: total_norm: 4.987212 [I 2022-10-26 15:41:34] ...trap/engines/engine.py.224: train: epoch 16 | batch 1070/2304 [I 2022-10-26 15:41:34] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:10:13 [I 2022-10-26 15:41:34] ...trap/engines/engine.py.229: process: 0.49765 | load: 0.00243 [I 2022-10-26 15:41:34] ...trap/engines/engine.py.230: loss: 0.18984 [I 2022-10-26 15:41:34] ...optimizers/trijoint.py.151: total_norm: 2.240526 [I 2022-10-26 15:41:39] ...trap/engines/engine.py.224: train: epoch 16 | batch 1080/2304 [I 2022-10-26 15:41:39] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:10:14 [I 2022-10-26 15:41:39] ...trap/engines/engine.py.229: process: 0.50512 | load: 0.00245 [I 2022-10-26 15:41:39] ...trap/engines/engine.py.230: loss: 0.21114 [I 2022-10-26 15:41:39] ...optimizers/trijoint.py.151: total_norm: 3.535299 [I 2022-10-26 15:41:44] ...trap/engines/engine.py.224: train: epoch 16 | batch 1090/2304 [I 2022-10-26 15:41:44] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:10:06 [I 2022-10-26 15:41:44] ...trap/engines/engine.py.229: process: 0.49789 | load: 0.00244 [I 2022-10-26 15:41:44] ...trap/engines/engine.py.230: loss: 0.23059 [I 2022-10-26 15:41:44] ...optimizers/trijoint.py.151: total_norm: 2.724491 [I 2022-10-26 15:41:49] ...trap/engines/engine.py.224: train: epoch 16 | batch 1100/2304 [I 2022-10-26 15:41:49] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:10:02 [I 2022-10-26 15:41:49] ...trap/engines/engine.py.229: process: 0.50131 | load: 0.00246 [I 2022-10-26 15:41:49] ...trap/engines/engine.py.230: loss: 0.36264 [I 2022-10-26 15:41:49] ...optimizers/trijoint.py.151: total_norm: 4.490759 [I 2022-10-26 15:41:54] ...trap/engines/engine.py.224: train: epoch 16 | batch 1110/2304 [I 2022-10-26 15:41:54] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:09:57 [I 2022-10-26 15:41:54] ...trap/engines/engine.py.229: process: 0.50255 | load: 0.00243 [I 2022-10-26 15:41:54] ...trap/engines/engine.py.230: loss: 0.19878 [I 2022-10-26 15:41:54] ...optimizers/trijoint.py.151: total_norm: 3.374110 [I 2022-10-26 15:41:59] ...trap/engines/engine.py.224: train: epoch 16 | batch 1120/2304 [I 2022-10-26 15:41:59] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:09:51 [I 2022-10-26 15:41:59] ...trap/engines/engine.py.229: process: 0.50689 | load: 0.00265 [I 2022-10-26 15:41:59] ...trap/engines/engine.py.230: loss: 0.18101 [I 2022-10-26 15:41:59] ...optimizers/trijoint.py.151: total_norm: 2.371999 [I 2022-10-26 15:42:04] ...trap/engines/engine.py.224: train: epoch 16 | batch 1130/2304 [I 2022-10-26 15:42:04] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:09:44 [I 2022-10-26 15:42:04] ...trap/engines/engine.py.229: process: 0.48945 | load: 0.00248 [I 2022-10-26 15:42:04] ...trap/engines/engine.py.230: loss: 0.19748 [I 2022-10-26 15:42:04] ...optimizers/trijoint.py.151: total_norm: 2.697049 [I 2022-10-26 15:42:09] ...trap/engines/engine.py.224: train: epoch 16 | batch 1140/2304 [I 2022-10-26 15:42:09] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:09:39 [I 2022-10-26 15:42:09] ...trap/engines/engine.py.229: process: 0.49549 | load: 0.00245 [I 2022-10-26 15:42:09] ...trap/engines/engine.py.230: loss: 0.28211 [I 2022-10-26 15:42:09] ...optimizers/trijoint.py.151: total_norm: 3.929191 [I 2022-10-26 15:42:14] ...trap/engines/engine.py.224: train: epoch 16 | batch 1150/2304 [I 2022-10-26 15:42:14] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:09:36 [I 2022-10-26 15:42:14] ...trap/engines/engine.py.229: process: 0.50206 | load: 0.00242 [I 2022-10-26 15:42:14] ...trap/engines/engine.py.230: loss: 0.17868 [I 2022-10-26 15:42:14] ...optimizers/trijoint.py.151: total_norm: 2.902072 [I 2022-10-26 15:42:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 1160/2304 [I 2022-10-26 15:42:19] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:09:31 [I 2022-10-26 15:42:19] ...trap/engines/engine.py.229: process: 0.50012 | load: 0.00246 [I 2022-10-26 15:42:19] ...trap/engines/engine.py.230: loss: 0.21767 [I 2022-10-26 15:42:19] ...optimizers/trijoint.py.151: total_norm: 3.834043 [I 2022-10-26 15:42:24] ...trap/engines/engine.py.224: train: epoch 16 | batch 1170/2304 [I 2022-10-26 15:42:24] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:09:30 [I 2022-10-26 15:42:24] ...trap/engines/engine.py.229: process: 0.50781 | load: 0.00241 [I 2022-10-26 15:42:24] ...trap/engines/engine.py.230: loss: 0.21389 [I 2022-10-26 15:42:24] ...optimizers/trijoint.py.151: total_norm: 3.317005 [I 2022-10-26 15:42:29] ...trap/engines/engine.py.224: train: epoch 16 | batch 1180/2304 [I 2022-10-26 15:42:29] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:09:20 [I 2022-10-26 15:42:29] ...trap/engines/engine.py.229: process: 0.49402 | load: 0.00242 [I 2022-10-26 15:42:29] ...trap/engines/engine.py.230: loss: 0.24884 [I 2022-10-26 15:42:29] ...optimizers/trijoint.py.151: total_norm: 2.600052 [I 2022-10-26 15:42:34] ...trap/engines/engine.py.224: train: epoch 16 | batch 1190/2304 [I 2022-10-26 15:42:34] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:09:16 [I 2022-10-26 15:42:34] ...trap/engines/engine.py.229: process: 0.50157 | load: 0.00240 [I 2022-10-26 15:42:34] ...trap/engines/engine.py.230: loss: 0.22664 [I 2022-10-26 15:42:34] ...optimizers/trijoint.py.151: total_norm: 3.702079 [I 2022-10-26 15:42:39] ...trap/engines/engine.py.224: train: epoch 16 | batch 1200/2304 [I 2022-10-26 15:42:39] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:09:10 [I 2022-10-26 15:42:39] ...trap/engines/engine.py.229: process: 0.49423 | load: 0.00245 [I 2022-10-26 15:42:39] ...trap/engines/engine.py.230: loss: 0.16897 [I 2022-10-26 15:42:39] ...optimizers/trijoint.py.151: total_norm: 2.775104 [I 2022-10-26 15:42:44] ...trap/engines/engine.py.224: train: epoch 16 | batch 1210/2304 [I 2022-10-26 15:42:44] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:09:06 [I 2022-10-26 15:42:44] ...trap/engines/engine.py.229: process: 0.49351 | load: 0.00253 [I 2022-10-26 15:42:44] ...trap/engines/engine.py.230: loss: 0.19454 [I 2022-10-26 15:42:44] ...optimizers/trijoint.py.151: total_norm: 3.470975 [I 2022-10-26 15:42:49] ...trap/engines/engine.py.224: train: epoch 16 | batch 1220/2304 [I 2022-10-26 15:42:49] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:09:01 [I 2022-10-26 15:42:49] ...trap/engines/engine.py.229: process: 0.49716 | load: 0.00248 [I 2022-10-26 15:42:49] ...trap/engines/engine.py.230: loss: 0.15183 [I 2022-10-26 15:42:49] ...optimizers/trijoint.py.151: total_norm: 1.881631 [I 2022-10-26 15:42:54] ...trap/engines/engine.py.224: train: epoch 16 | batch 1230/2304 [I 2022-10-26 15:42:54] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:08:54 [I 2022-10-26 15:42:54] ...trap/engines/engine.py.229: process: 0.49560 | load: 0.00244 [I 2022-10-26 15:42:54] ...trap/engines/engine.py.230: loss: 0.25282 [I 2022-10-26 15:42:54] ...optimizers/trijoint.py.151: total_norm: 3.095790 [I 2022-10-26 15:42:59] ...trap/engines/engine.py.224: train: epoch 16 | batch 1240/2304 [I 2022-10-26 15:42:59] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:08:48 [I 2022-10-26 15:42:59] ...trap/engines/engine.py.229: process: 0.50322 | load: 0.00242 [I 2022-10-26 15:42:59] ...trap/engines/engine.py.230: loss: 0.21383 [I 2022-10-26 15:42:59] ...optimizers/trijoint.py.151: total_norm: 3.285996 [I 2022-10-26 15:43:04] ...trap/engines/engine.py.224: train: epoch 16 | batch 1250/2304 [I 2022-10-26 15:43:04] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:08:42 [I 2022-10-26 15:43:04] ...trap/engines/engine.py.229: process: 0.49023 | load: 0.00243 [I 2022-10-26 15:43:04] ...trap/engines/engine.py.230: loss: 0.17813 [I 2022-10-26 15:43:04] ...optimizers/trijoint.py.151: total_norm: 1.967384 [I 2022-10-26 15:43:09] ...trap/engines/engine.py.224: train: epoch 16 | batch 1260/2304 [I 2022-10-26 15:43:09] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:08:39 [I 2022-10-26 15:43:09] ...trap/engines/engine.py.229: process: 0.50307 | load: 0.00247 [I 2022-10-26 15:43:09] ...trap/engines/engine.py.230: loss: 0.19509 [I 2022-10-26 15:43:09] ...optimizers/trijoint.py.151: total_norm: 3.418182 [I 2022-10-26 15:43:14] ...trap/engines/engine.py.224: train: epoch 16 | batch 1270/2304 [I 2022-10-26 15:43:14] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:08:31 [I 2022-10-26 15:43:14] ...trap/engines/engine.py.229: process: 0.48785 | load: 0.00246 [I 2022-10-26 15:43:14] ...trap/engines/engine.py.230: loss: 0.23040 [I 2022-10-26 15:43:14] ...optimizers/trijoint.py.151: total_norm: 3.725518 [I 2022-10-26 15:43:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 1280/2304 [I 2022-10-26 15:43:19] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:08:30 [I 2022-10-26 15:43:19] ...trap/engines/engine.py.229: process: 0.49670 | load: 0.00242 [I 2022-10-26 15:43:19] ...trap/engines/engine.py.230: loss: 0.15511 [I 2022-10-26 15:43:19] ...optimizers/trijoint.py.151: total_norm: 2.498410 [I 2022-10-26 15:43:24] ...trap/engines/engine.py.224: train: epoch 16 | batch 1290/2304 [I 2022-10-26 15:43:24] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:08:28 [I 2022-10-26 15:43:24] ...trap/engines/engine.py.229: process: 0.50745 | load: 0.00241 [I 2022-10-26 15:43:24] ...trap/engines/engine.py.230: loss: 0.28462 [I 2022-10-26 15:43:24] ...optimizers/trijoint.py.151: total_norm: 3.864703 [I 2022-10-26 15:43:29] ...trap/engines/engine.py.224: train: epoch 16 | batch 1300/2304 [I 2022-10-26 15:43:29] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:08:27 [I 2022-10-26 15:43:29] ...trap/engines/engine.py.229: process: 0.51418 | load: 0.00254 [I 2022-10-26 15:43:29] ...trap/engines/engine.py.230: loss: 0.25711 [I 2022-10-26 15:43:29] ...optimizers/trijoint.py.151: total_norm: 3.998224 [I 2022-10-26 15:43:34] ...trap/engines/engine.py.224: train: epoch 16 | batch 1310/2304 [I 2022-10-26 15:43:34] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:08:19 [I 2022-10-26 15:43:34] ...trap/engines/engine.py.229: process: 0.50806 | load: 0.00248 [I 2022-10-26 15:43:34] ...trap/engines/engine.py.230: loss: 0.19847 [I 2022-10-26 15:43:34] ...optimizers/trijoint.py.151: total_norm: 3.256510 [I 2022-10-26 15:43:39] ...trap/engines/engine.py.224: train: epoch 16 | batch 1320/2304 [I 2022-10-26 15:43:39] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:08:12 [I 2022-10-26 15:43:39] ...trap/engines/engine.py.229: process: 0.50384 | load: 0.00248 [I 2022-10-26 15:43:39] ...trap/engines/engine.py.230: loss: 0.16132 [I 2022-10-26 15:43:39] ...optimizers/trijoint.py.151: total_norm: 3.788948 [I 2022-10-26 15:43:44] ...trap/engines/engine.py.224: train: epoch 16 | batch 1330/2304 [I 2022-10-26 15:43:44] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:08:07 [I 2022-10-26 15:43:44] ...trap/engines/engine.py.229: process: 0.50721 | load: 0.00241 [I 2022-10-26 15:43:44] ...trap/engines/engine.py.230: loss: 0.21257 [I 2022-10-26 15:43:44] ...optimizers/trijoint.py.151: total_norm: 2.982427 [I 2022-10-26 15:43:49] ...trap/engines/engine.py.224: train: epoch 16 | batch 1340/2304 [I 2022-10-26 15:43:49] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:08:02 [I 2022-10-26 15:43:49] ...trap/engines/engine.py.229: process: 0.48967 | load: 0.00245 [I 2022-10-26 15:43:49] ...trap/engines/engine.py.230: loss: 0.19480 [I 2022-10-26 15:43:49] ...optimizers/trijoint.py.151: total_norm: 3.997235 [I 2022-10-26 15:43:54] ...trap/engines/engine.py.224: train: epoch 16 | batch 1350/2304 [I 2022-10-26 15:43:54] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:07:53 [I 2022-10-26 15:43:54] ...trap/engines/engine.py.229: process: 0.49252 | load: 0.00245 [I 2022-10-26 15:43:54] ...trap/engines/engine.py.230: loss: 0.16755 [I 2022-10-26 15:43:54] ...optimizers/trijoint.py.151: total_norm: 3.339260 [I 2022-10-26 15:43:59] ...trap/engines/engine.py.224: train: epoch 16 | batch 1360/2304 [I 2022-10-26 15:43:59] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:07:45 [I 2022-10-26 15:43:59] ...trap/engines/engine.py.229: process: 0.48677 | load: 0.00241 [I 2022-10-26 15:43:59] ...trap/engines/engine.py.230: loss: 0.26118 [I 2022-10-26 15:43:59] ...optimizers/trijoint.py.151: total_norm: 4.252952 [I 2022-10-26 15:44:04] ...trap/engines/engine.py.224: train: epoch 16 | batch 1370/2304 [I 2022-10-26 15:44:04] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:07:39 [I 2022-10-26 15:44:04] ...trap/engines/engine.py.229: process: 0.49024 | load: 0.00244 [I 2022-10-26 15:44:04] ...trap/engines/engine.py.230: loss: 0.20198 [I 2022-10-26 15:44:04] ...optimizers/trijoint.py.151: total_norm: 3.375592 [I 2022-10-26 15:44:09] ...trap/engines/engine.py.224: train: epoch 16 | batch 1380/2304 [I 2022-10-26 15:44:09] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:07:42 [I 2022-10-26 15:44:09] ...trap/engines/engine.py.229: process: 0.49378 | load: 0.00245 [I 2022-10-26 15:44:09] ...trap/engines/engine.py.230: loss: 0.20262 [I 2022-10-26 15:44:09] ...optimizers/trijoint.py.151: total_norm: 3.426146 [I 2022-10-26 15:44:14] ...trap/engines/engine.py.224: train: epoch 16 | batch 1390/2304 [I 2022-10-26 15:44:14] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:07:36 [I 2022-10-26 15:44:14] ...trap/engines/engine.py.229: process: 0.49959 | load: 0.00244 [I 2022-10-26 15:44:14] ...trap/engines/engine.py.230: loss: 0.25834 [I 2022-10-26 15:44:14] ...optimizers/trijoint.py.151: total_norm: 3.701200 [I 2022-10-26 15:44:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 1400/2304 [I 2022-10-26 15:44:19] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:07:34 [I 2022-10-26 15:44:19] ...trap/engines/engine.py.229: process: 0.50819 | load: 0.00251 [I 2022-10-26 15:44:19] ...trap/engines/engine.py.230: loss: 0.17901 [I 2022-10-26 15:44:19] ...optimizers/trijoint.py.151: total_norm: 2.877710 [I 2022-10-26 15:44:24] ...trap/engines/engine.py.224: train: epoch 16 | batch 1410/2304 [I 2022-10-26 15:44:24] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:07:27 [I 2022-10-26 15:44:24] ...trap/engines/engine.py.229: process: 0.50009 | load: 0.00249 [I 2022-10-26 15:44:24] ...trap/engines/engine.py.230: loss: 0.18149 [I 2022-10-26 15:44:24] ...optimizers/trijoint.py.151: total_norm: 2.551778 [I 2022-10-26 15:44:29] ...trap/engines/engine.py.224: train: epoch 16 | batch 1420/2304 [I 2022-10-26 15:44:29] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:07:19 [I 2022-10-26 15:44:29] ...trap/engines/engine.py.229: process: 0.50422 | load: 0.00243 [I 2022-10-26 15:44:29] ...trap/engines/engine.py.230: loss: 0.27315 [I 2022-10-26 15:44:29] ...optimizers/trijoint.py.151: total_norm: 3.944455 [I 2022-10-26 15:44:34] ...trap/engines/engine.py.224: train: epoch 16 | batch 1430/2304 [I 2022-10-26 15:44:34] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:07:17 [I 2022-10-26 15:44:34] ...trap/engines/engine.py.229: process: 0.51276 | load: 0.00243 [I 2022-10-26 15:44:34] ...trap/engines/engine.py.230: loss: 0.17911 [I 2022-10-26 15:44:34] ...optimizers/trijoint.py.151: total_norm: 3.408612 [I 2022-10-26 15:44:39] ...trap/engines/engine.py.224: train: epoch 16 | batch 1440/2304 [I 2022-10-26 15:44:39] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:07:08 [I 2022-10-26 15:44:39] ...trap/engines/engine.py.229: process: 0.49100 | load: 0.00245 [I 2022-10-26 15:44:39] ...trap/engines/engine.py.230: loss: 0.26071 [I 2022-10-26 15:44:39] ...optimizers/trijoint.py.151: total_norm: 3.325595 [I 2022-10-26 15:44:44] ...trap/engines/engine.py.224: train: epoch 16 | batch 1450/2304 [I 2022-10-26 15:44:44] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:07:06 [I 2022-10-26 15:44:44] ...trap/engines/engine.py.229: process: 0.48611 | load: 0.00250 [I 2022-10-26 15:44:44] ...trap/engines/engine.py.230: loss: 0.18432 [I 2022-10-26 15:44:44] ...optimizers/trijoint.py.151: total_norm: 3.835566 [I 2022-10-26 15:44:49] ...trap/engines/engine.py.224: train: epoch 16 | batch 1460/2304 [I 2022-10-26 15:44:49] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:07:03 [I 2022-10-26 15:44:49] ...trap/engines/engine.py.229: process: 0.50482 | load: 0.00249 [I 2022-10-26 15:44:49] ...trap/engines/engine.py.230: loss: 0.18623 [I 2022-10-26 15:44:49] ...optimizers/trijoint.py.151: total_norm: 3.559079 [I 2022-10-26 15:44:54] ...trap/engines/engine.py.224: train: epoch 16 | batch 1470/2304 [I 2022-10-26 15:44:54] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:06:57 [I 2022-10-26 15:44:54] ...trap/engines/engine.py.229: process: 0.50229 | load: 0.00245 [I 2022-10-26 15:44:54] ...trap/engines/engine.py.230: loss: 0.23870 [I 2022-10-26 15:44:54] ...optimizers/trijoint.py.151: total_norm: 3.505733 [I 2022-10-26 15:44:59] ...trap/engines/engine.py.224: train: epoch 16 | batch 1480/2304 [I 2022-10-26 15:44:59] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:06:53 [I 2022-10-26 15:44:59] ...trap/engines/engine.py.229: process: 0.50067 | load: 0.00253 [I 2022-10-26 15:44:59] ...trap/engines/engine.py.230: loss: 0.26497 [I 2022-10-26 15:44:59] ...optimizers/trijoint.py.151: total_norm: 3.797628 [I 2022-10-26 15:45:04] ...trap/engines/engine.py.224: train: epoch 16 | batch 1490/2304 [I 2022-10-26 15:45:04] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:06:48 [I 2022-10-26 15:45:04] ...trap/engines/engine.py.229: process: 0.52179 | load: 0.00254 [I 2022-10-26 15:45:04] ...trap/engines/engine.py.230: loss: 0.18366 [I 2022-10-26 15:45:04] ...optimizers/trijoint.py.151: total_norm: 2.679665 [I 2022-10-26 15:45:09] ...trap/engines/engine.py.224: train: epoch 16 | batch 1500/2304 [I 2022-10-26 15:45:09] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:06:43 [I 2022-10-26 15:45:09] ...trap/engines/engine.py.229: process: 0.51469 | load: 0.00241 [I 2022-10-26 15:45:09] ...trap/engines/engine.py.230: loss: 0.17577 [I 2022-10-26 15:45:09] ...optimizers/trijoint.py.151: total_norm: 2.823953 [I 2022-10-26 15:45:14] ...trap/engines/engine.py.224: train: epoch 16 | batch 1510/2304 [I 2022-10-26 15:45:14] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:06:37 [I 2022-10-26 15:45:14] ...trap/engines/engine.py.229: process: 0.50225 | load: 0.00252 [I 2022-10-26 15:45:14] ...trap/engines/engine.py.230: loss: 0.26626 [I 2022-10-26 15:45:14] ...optimizers/trijoint.py.151: total_norm: 4.025174 [I 2022-10-26 15:45:19] ...trap/engines/engine.py.224: train: epoch 16 | batch 1520/2304 [I 2022-10-26 15:45:19] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:06:32 [I 2022-10-26 15:45:19] ...trap/engines/engine.py.229: process: 0.50366 | load: 0.00256 [I 2022-10-26 15:45:19] ...trap/engines/engine.py.230: loss: 0.18359 [I 2022-10-26 15:45:19] ...optimizers/trijoint.py.151: total_norm: 3.231194 [I 2022-10-26 15:45:24] ...trap/engines/engine.py.224: train: epoch 16 | batch 1530/2304 [I 2022-10-26 15:45:24] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:06:32 [I 2022-10-26 15:45:24] ...trap/engines/engine.py.229: process: 0.47922 | load: 0.00252 [I 2022-10-26 15:45:24] ...trap/engines/engine.py.230: loss: 0.31674 [I 2022-10-26 15:45:24] ...optimizers/trijoint.py.151: total_norm: 3.488530 [I 2022-10-26 15:45:29] ...trap/engines/engine.py.224: train: epoch 16 | batch 1540/2304 [I 2022-10-26 15:45:29] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:06:24 [I 2022-10-26 15:45:29] ...trap/engines/engine.py.229: process: 0.50063 | load: 0.00249 [I 2022-10-26 15:45:29] ...trap/engines/engine.py.230: loss: 0.18734 [I 2022-10-26 15:45:29] ...optimizers/trijoint.py.151: total_norm: 2.958360 [I 2022-10-26 15:45:34] ...trap/engines/engine.py.224: train: epoch 16 | batch 1550/2304 [I 2022-10-26 15:45:34] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:06:19 [I 2022-10-26 15:45:34] ...trap/engines/engine.py.229: process: 0.50188 | load: 0.00248 [I 2022-10-26 15:45:34] ...trap/engines/engine.py.230: loss: 0.27570 [I 2022-10-26 15:45:34] ...optimizers/trijoint.py.151: total_norm: 3.848377 [I 2022-10-26 15:45:39] ...trap/engines/engine.py.224: train: epoch 16 | batch 1560/2304 [I 2022-10-26 15:45:39] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:06:13 [I 2022-10-26 15:45:39] ...trap/engines/engine.py.229: process: 0.51431 | load: 0.00247 [I 2022-10-26 15:45:39] ...trap/engines/engine.py.230: loss: 0.24630 [I 2022-10-26 15:45:39] ...optimizers/trijoint.py.151: total_norm: 3.742469 [I 2022-10-26 15:45:44] ...trap/engines/engine.py.224: train: epoch 16 | batch 1570/2304 [I 2022-10-26 15:45:44] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:06:07 [I 2022-10-26 15:45:44] ...trap/engines/engine.py.229: process: 0.50073 | load: 0.00246 [I 2022-10-26 15:45:44] ...trap/engines/engine.py.230: loss: 0.14509 [I 2022-10-26 15:45:44] ...optimizers/trijoint.py.151: total_norm: 2.251779 [I 2022-10-26 15:45:49] ...trap/engines/engine.py.224: train: epoch 16 | batch 1580/2304 [I 2022-10-26 15:45:49] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:06:04 [I 2022-10-26 15:45:49] ...trap/engines/engine.py.229: process: 0.50930 | load: 0.00241 [I 2022-10-26 15:45:49] ...trap/engines/engine.py.230: loss: 0.26516 [I 2022-10-26 15:45:49] ...optimizers/trijoint.py.151: total_norm: 3.936060 [I 2022-10-26 15:45:54] ...trap/engines/engine.py.224: train: epoch 16 | batch 1590/2304 [I 2022-10-26 15:45:54] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:05:58 [I 2022-10-26 15:45:54] ...trap/engines/engine.py.229: process: 0.50359 | load: 0.00252 [I 2022-10-26 15:45:54] ...trap/engines/engine.py.230: loss: 0.27499 [I 2022-10-26 15:45:54] ...optimizers/trijoint.py.151: total_norm: 3.417089 [I 2022-10-26 15:45:59] ...trap/engines/engine.py.224: train: epoch 16 | batch 1600/2304 [I 2022-10-26 15:45:59] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:05:54 [I 2022-10-26 15:45:59] ...trap/engines/engine.py.229: process: 0.50739 | load: 0.00248 [I 2022-10-26 15:45:59] ...trap/engines/engine.py.230: loss: 0.18982 [I 2022-10-26 15:45:59] ...optimizers/trijoint.py.151: total_norm: 3.116533 [I 2022-10-26 15:46:04] ...trap/engines/engine.py.224: train: epoch 16 | batch 1610/2304 [I 2022-10-26 15:46:04] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:05:50 [I 2022-10-26 15:46:04] ...trap/engines/engine.py.229: process: 0.50853 | load: 0.00248 [I 2022-10-26 15:46:04] ...trap/engines/engine.py.230: loss: 0.18113 [I 2022-10-26 15:46:04] ...optimizers/trijoint.py.151: total_norm: 3.302365 [I 2022-10-26 15:46:09] ...trap/engines/engine.py.224: train: epoch 16 | batch 1620/2304 [I 2022-10-26 15:46:09] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:05:44 [I 2022-10-26 15:46:09] ...trap/engines/engine.py.229: process: 0.51305 | load: 0.00247 [I 2022-10-26 15:46:09] ...trap/engines/engine.py.230: loss: 0.21020 [I 2022-10-26 15:46:09] ...optimizers/trijoint.py.151: total_norm: 3.534929 [I 2022-10-26 15:46:15] ...trap/engines/engine.py.224: train: epoch 16 | batch 1630/2304 [I 2022-10-26 15:46:15] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:05:48 [I 2022-10-26 15:46:15] ...trap/engines/engine.py.229: process: 0.50460 | load: 0.00246 [I 2022-10-26 15:46:15] ...trap/engines/engine.py.230: loss: 0.25386 [I 2022-10-26 15:46:15] ...optimizers/trijoint.py.151: total_norm: 2.949908 [I 2022-10-26 15:46:20] ...trap/engines/engine.py.224: train: epoch 16 | batch 1640/2304 [I 2022-10-26 15:46:20] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:05:31 [I 2022-10-26 15:46:20] ...trap/engines/engine.py.229: process: 0.50140 | load: 0.00245 [I 2022-10-26 15:46:20] ...trap/engines/engine.py.230: loss: 0.25843 [I 2022-10-26 15:46:20] ...optimizers/trijoint.py.151: total_norm: 2.817995 [I 2022-10-26 15:46:25] ...trap/engines/engine.py.224: train: epoch 16 | batch 1650/2304 [I 2022-10-26 15:46:25] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:05:28 [I 2022-10-26 15:46:25] ...trap/engines/engine.py.229: process: 0.49968 | load: 0.00249 [I 2022-10-26 15:46:25] ...trap/engines/engine.py.230: loss: 0.26906 [I 2022-10-26 15:46:25] ...optimizers/trijoint.py.151: total_norm: 3.269919 [I 2022-10-26 15:46:30] ...trap/engines/engine.py.224: train: epoch 16 | batch 1660/2304 [I 2022-10-26 15:46:30] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:05:24 [I 2022-10-26 15:46:30] ...trap/engines/engine.py.229: process: 0.49886 | load: 0.00273 [I 2022-10-26 15:46:30] ...trap/engines/engine.py.230: loss: 0.13155 [I 2022-10-26 15:46:30] ...optimizers/trijoint.py.151: total_norm: 2.738366 [I 2022-10-26 15:46:35] ...trap/engines/engine.py.224: train: epoch 16 | batch 1670/2304 [I 2022-10-26 15:46:35] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:05:17 [I 2022-10-26 15:46:35] ...trap/engines/engine.py.229: process: 0.50308 | load: 0.00265 [I 2022-10-26 15:46:35] ...trap/engines/engine.py.230: loss: 0.14028 [I 2022-10-26 15:46:35] ...optimizers/trijoint.py.151: total_norm: 2.075164 [I 2022-10-26 15:46:40] ...trap/engines/engine.py.224: train: epoch 16 | batch 1680/2304 [I 2022-10-26 15:46:40] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:05:11 [I 2022-10-26 15:46:40] ...trap/engines/engine.py.229: process: 0.50043 | load: 0.00256 [I 2022-10-26 15:46:40] ...trap/engines/engine.py.230: loss: 0.20799 [I 2022-10-26 15:46:40] ...optimizers/trijoint.py.151: total_norm: 3.038127 [I 2022-10-26 15:46:45] ...trap/engines/engine.py.224: train: epoch 16 | batch 1690/2304 [I 2022-10-26 15:46:45] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:05:07 [I 2022-10-26 15:46:45] ...trap/engines/engine.py.229: process: 0.50507 | load: 0.00252 [I 2022-10-26 15:46:45] ...trap/engines/engine.py.230: loss: 0.15227 [I 2022-10-26 15:46:45] ...optimizers/trijoint.py.151: total_norm: 2.753041 [I 2022-10-26 15:46:50] ...trap/engines/engine.py.224: train: epoch 16 | batch 1700/2304 [I 2022-10-26 15:46:50] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:05:04 [I 2022-10-26 15:46:50] ...trap/engines/engine.py.229: process: 0.50004 | load: 0.00259 [I 2022-10-26 15:46:50] ...trap/engines/engine.py.230: loss: 0.25981 [I 2022-10-26 15:46:50] ...optimizers/trijoint.py.151: total_norm: 3.351415 [I 2022-10-26 15:46:55] ...trap/engines/engine.py.224: train: epoch 16 | batch 1710/2304 [I 2022-10-26 15:46:55] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:04:59 [I 2022-10-26 15:46:55] ...trap/engines/engine.py.229: process: 0.49662 | load: 0.00249 [I 2022-10-26 15:46:55] ...trap/engines/engine.py.230: loss: 0.17919 [I 2022-10-26 15:46:55] ...optimizers/trijoint.py.151: total_norm: 3.345989 [I 2022-10-26 15:47:00] ...trap/engines/engine.py.224: train: epoch 16 | batch 1720/2304 [I 2022-10-26 15:47:00] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:04:53 [I 2022-10-26 15:47:00] ...trap/engines/engine.py.229: process: 0.50571 | load: 0.00248 [I 2022-10-26 15:47:00] ...trap/engines/engine.py.230: loss: 0.21654 [I 2022-10-26 15:47:00] ...optimizers/trijoint.py.151: total_norm: 2.809107 [I 2022-10-26 15:47:05] ...trap/engines/engine.py.224: train: epoch 16 | batch 1730/2304 [I 2022-10-26 15:47:05] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:04:49 [I 2022-10-26 15:47:05] ...trap/engines/engine.py.229: process: 0.50682 | load: 0.00244 [I 2022-10-26 15:47:05] ...trap/engines/engine.py.230: loss: 0.22574 [I 2022-10-26 15:47:05] ...optimizers/trijoint.py.151: total_norm: 3.448037 [I 2022-10-26 15:47:10] ...trap/engines/engine.py.224: train: epoch 16 | batch 1740/2304 [I 2022-10-26 15:47:10] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:04:42 [I 2022-10-26 15:47:10] ...trap/engines/engine.py.229: process: 0.50836 | load: 0.00249 [I 2022-10-26 15:47:10] ...trap/engines/engine.py.230: loss: 0.19584 [I 2022-10-26 15:47:10] ...optimizers/trijoint.py.151: total_norm: 3.326532 [I 2022-10-26 15:47:15] ...trap/engines/engine.py.224: train: epoch 16 | batch 1750/2304 [I 2022-10-26 15:47:15] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:04:35 [I 2022-10-26 15:47:15] ...trap/engines/engine.py.229: process: 0.50575 | load: 0.00263 [I 2022-10-26 15:47:15] ...trap/engines/engine.py.230: loss: 0.25616 [I 2022-10-26 15:47:15] ...optimizers/trijoint.py.151: total_norm: 4.132553 [I 2022-10-26 15:47:20] ...trap/engines/engine.py.224: train: epoch 16 | batch 1760/2304 [I 2022-10-26 15:47:20] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:04:31 [I 2022-10-26 15:47:20] ...trap/engines/engine.py.229: process: 0.50302 | load: 0.00260 [I 2022-10-26 15:47:20] ...trap/engines/engine.py.230: loss: 0.15208 [I 2022-10-26 15:47:20] ...optimizers/trijoint.py.151: total_norm: 2.951091 [I 2022-10-26 15:47:25] ...trap/engines/engine.py.224: train: epoch 16 | batch 1770/2304 [I 2022-10-26 15:47:25] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:04:28 [I 2022-10-26 15:47:25] ...trap/engines/engine.py.229: process: 0.50575 | load: 0.00251 [I 2022-10-26 15:47:25] ...trap/engines/engine.py.230: loss: 0.16329 [I 2022-10-26 15:47:25] ...optimizers/trijoint.py.151: total_norm: 2.745230 [I 2022-10-26 15:47:30] ...trap/engines/engine.py.224: train: epoch 16 | batch 1780/2304 [I 2022-10-26 15:47:30] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:04:23 [I 2022-10-26 15:47:30] ...trap/engines/engine.py.229: process: 0.49564 | load: 0.00244 [I 2022-10-26 15:47:30] ...trap/engines/engine.py.230: loss: 0.27136 [I 2022-10-26 15:47:30] ...optimizers/trijoint.py.151: total_norm: 3.686661 [I 2022-10-26 15:47:35] ...trap/engines/engine.py.224: train: epoch 16 | batch 1790/2304 [I 2022-10-26 15:47:35] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:04:18 [I 2022-10-26 15:47:35] ...trap/engines/engine.py.229: process: 0.51027 | load: 0.00258 [I 2022-10-26 15:47:35] ...trap/engines/engine.py.230: loss: 0.24578 [I 2022-10-26 15:47:35] ...optimizers/trijoint.py.151: total_norm: 3.303337 [I 2022-10-26 15:47:40] ...trap/engines/engine.py.224: train: epoch 16 | batch 1800/2304 [I 2022-10-26 15:47:40] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:04:11 [I 2022-10-26 15:47:40] ...trap/engines/engine.py.229: process: 0.50196 | load: 0.00254 [I 2022-10-26 15:47:40] ...trap/engines/engine.py.230: loss: 0.24781 [I 2022-10-26 15:47:40] ...optimizers/trijoint.py.151: total_norm: 3.266610 [I 2022-10-26 15:47:45] ...trap/engines/engine.py.224: train: epoch 16 | batch 1810/2304 [I 2022-10-26 15:47:45] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:04:05 [I 2022-10-26 15:47:45] ...trap/engines/engine.py.229: process: 0.49895 | load: 0.00262 [I 2022-10-26 15:47:45] ...trap/engines/engine.py.230: loss: 0.21266 [I 2022-10-26 15:47:45] ...optimizers/trijoint.py.151: total_norm: 2.958463 [I 2022-10-26 15:47:50] ...trap/engines/engine.py.224: train: epoch 16 | batch 1820/2304 [I 2022-10-26 15:47:50] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:04:05 [I 2022-10-26 15:47:50] ...trap/engines/engine.py.229: process: 0.50689 | load: 0.00262 [I 2022-10-26 15:47:50] ...trap/engines/engine.py.230: loss: 0.26351 [I 2022-10-26 15:47:50] ...optimizers/trijoint.py.151: total_norm: 3.350536 [I 2022-10-26 15:47:55] ...trap/engines/engine.py.224: train: epoch 16 | batch 1830/2304 [I 2022-10-26 15:47:55] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:03:56 [I 2022-10-26 15:47:55] ...trap/engines/engine.py.229: process: 0.50048 | load: 0.00251 [I 2022-10-26 15:47:55] ...trap/engines/engine.py.230: loss: 0.18786 [I 2022-10-26 15:47:55] ...optimizers/trijoint.py.151: total_norm: 3.019905 [I 2022-10-26 15:48:00] ...trap/engines/engine.py.224: train: epoch 16 | batch 1840/2304 [I 2022-10-26 15:48:00] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:03:53 [I 2022-10-26 15:48:00] ...trap/engines/engine.py.229: process: 0.50556 | load: 0.00260 [I 2022-10-26 15:48:00] ...trap/engines/engine.py.230: loss: 0.21845 [I 2022-10-26 15:48:00] ...optimizers/trijoint.py.151: total_norm: 4.339684 [I 2022-10-26 15:48:05] ...trap/engines/engine.py.224: train: epoch 16 | batch 1850/2304 [I 2022-10-26 15:48:05] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:03:48 [I 2022-10-26 15:48:05] ...trap/engines/engine.py.229: process: 0.50641 | load: 0.00249 [I 2022-10-26 15:48:05] ...trap/engines/engine.py.230: loss: 0.21821 [I 2022-10-26 15:48:05] ...optimizers/trijoint.py.151: total_norm: 3.462319 [I 2022-10-26 15:48:10] ...trap/engines/engine.py.224: train: epoch 16 | batch 1860/2304 [I 2022-10-26 15:48:10] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:03:42 [I 2022-10-26 15:48:10] ...trap/engines/engine.py.229: process: 0.50625 | load: 0.00250 [I 2022-10-26 15:48:10] ...trap/engines/engine.py.230: loss: 0.19121 [I 2022-10-26 15:48:10] ...optimizers/trijoint.py.151: total_norm: 3.367248 [I 2022-10-26 15:48:15] ...trap/engines/engine.py.224: train: epoch 16 | batch 1870/2304 [I 2022-10-26 15:48:15] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:03:34 [I 2022-10-26 15:48:15] ...trap/engines/engine.py.229: process: 0.49083 | load: 0.00246 [I 2022-10-26 15:48:15] ...trap/engines/engine.py.230: loss: 0.30839 [I 2022-10-26 15:48:15] ...optimizers/trijoint.py.151: total_norm: 4.581789 [I 2022-10-26 15:48:20] ...trap/engines/engine.py.224: train: epoch 16 | batch 1880/2304 [I 2022-10-26 15:48:20] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:03:31 [I 2022-10-26 15:48:20] ...trap/engines/engine.py.229: process: 0.50213 | load: 0.00252 [I 2022-10-26 15:48:20] ...trap/engines/engine.py.230: loss: 0.19134 [I 2022-10-26 15:48:20] ...optimizers/trijoint.py.151: total_norm: 1.929913 [I 2022-10-26 15:48:25] ...trap/engines/engine.py.224: train: epoch 16 | batch 1890/2304 [I 2022-10-26 15:48:25] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:03:27 [I 2022-10-26 15:48:25] ...trap/engines/engine.py.229: process: 0.49988 | load: 0.00308 [I 2022-10-26 15:48:25] ...trap/engines/engine.py.230: loss: 0.16899 [I 2022-10-26 15:48:25] ...optimizers/trijoint.py.151: total_norm: 3.202424 [I 2022-10-26 15:48:30] ...trap/engines/engine.py.224: train: epoch 16 | batch 1900/2304 [I 2022-10-26 15:48:30] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:03:20 [I 2022-10-26 15:48:30] ...trap/engines/engine.py.229: process: 0.49697 | load: 0.00283 [I 2022-10-26 15:48:30] ...trap/engines/engine.py.230: loss: 0.19913 [I 2022-10-26 15:48:30] ...optimizers/trijoint.py.151: total_norm: 2.654516 [I 2022-10-26 15:48:35] ...trap/engines/engine.py.224: train: epoch 16 | batch 1910/2304 [I 2022-10-26 15:48:35] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:03:17 [I 2022-10-26 15:48:35] ...trap/engines/engine.py.229: process: 0.50408 | load: 0.00247 [I 2022-10-26 15:48:35] ...trap/engines/engine.py.230: loss: 0.19682 [I 2022-10-26 15:48:35] ...optimizers/trijoint.py.151: total_norm: 2.132852 [I 2022-10-26 15:48:40] ...trap/engines/engine.py.224: train: epoch 16 | batch 1920/2304 [I 2022-10-26 15:48:40] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:03:12 [I 2022-10-26 15:48:40] ...trap/engines/engine.py.229: process: 0.49516 | load: 0.00250 [I 2022-10-26 15:48:40] ...trap/engines/engine.py.230: loss: 0.25144 [I 2022-10-26 15:48:40] ...optimizers/trijoint.py.151: total_norm: 4.123692 [I 2022-10-26 15:48:45] ...trap/engines/engine.py.224: train: epoch 16 | batch 1930/2304 [I 2022-10-26 15:48:45] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:03:06 [I 2022-10-26 15:48:45] ...trap/engines/engine.py.229: process: 0.48947 | load: 0.00247 [I 2022-10-26 15:48:45] ...trap/engines/engine.py.230: loss: 0.36864 [I 2022-10-26 15:48:45] ...optimizers/trijoint.py.151: total_norm: 4.607047 [I 2022-10-26 15:48:50] ...trap/engines/engine.py.224: train: epoch 16 | batch 1940/2304 [I 2022-10-26 15:48:50] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:03:06 [I 2022-10-26 15:48:50] ...trap/engines/engine.py.229: process: 0.50908 | load: 0.00249 [I 2022-10-26 15:48:50] ...trap/engines/engine.py.230: loss: 0.21265 [I 2022-10-26 15:48:50] ...optimizers/trijoint.py.151: total_norm: 3.797510 [I 2022-10-26 15:48:55] ...trap/engines/engine.py.224: train: epoch 16 | batch 1950/2304 [I 2022-10-26 15:48:55] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:02:59 [I 2022-10-26 15:48:55] ...trap/engines/engine.py.229: process: 0.52410 | load: 0.00247 [I 2022-10-26 15:48:55] ...trap/engines/engine.py.230: loss: 0.15152 [I 2022-10-26 15:48:55] ...optimizers/trijoint.py.151: total_norm: 2.841169 [I 2022-10-26 15:49:00] ...trap/engines/engine.py.224: train: epoch 16 | batch 1960/2304 [I 2022-10-26 15:49:00] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:02:53 [I 2022-10-26 15:49:00] ...trap/engines/engine.py.229: process: 0.51095 | load: 0.00252 [I 2022-10-26 15:49:00] ...trap/engines/engine.py.230: loss: 0.27741 [I 2022-10-26 15:49:00] ...optimizers/trijoint.py.151: total_norm: 3.319867 [I 2022-10-26 15:49:05] ...trap/engines/engine.py.224: train: epoch 16 | batch 1970/2304 [I 2022-10-26 15:49:05] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:02:47 [I 2022-10-26 15:49:05] ...trap/engines/engine.py.229: process: 0.49310 | load: 0.00242 [I 2022-10-26 15:49:05] ...trap/engines/engine.py.230: loss: 0.19016 [I 2022-10-26 15:49:05] ...optimizers/trijoint.py.151: total_norm: 2.440233 [I 2022-10-26 15:49:10] ...trap/engines/engine.py.224: train: epoch 16 | batch 1980/2304 [I 2022-10-26 15:49:10] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:02:41 [I 2022-10-26 15:49:10] ...trap/engines/engine.py.229: process: 0.48443 | load: 0.00249 [I 2022-10-26 15:49:10] ...trap/engines/engine.py.230: loss: 0.30203 [I 2022-10-26 15:49:10] ...optimizers/trijoint.py.151: total_norm: 3.722648 [I 2022-10-26 15:49:15] ...trap/engines/engine.py.224: train: epoch 16 | batch 1990/2304 [I 2022-10-26 15:49:15] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:02:37 [I 2022-10-26 15:49:15] ...trap/engines/engine.py.229: process: 0.51248 | load: 0.00247 [I 2022-10-26 15:49:15] ...trap/engines/engine.py.230: loss: 0.22408 [I 2022-10-26 15:49:15] ...optimizers/trijoint.py.151: total_norm: 3.883739 [I 2022-10-26 15:49:20] ...trap/engines/engine.py.224: train: epoch 16 | batch 2000/2304 [I 2022-10-26 15:49:20] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:02:30 [I 2022-10-26 15:49:20] ...trap/engines/engine.py.229: process: 0.49561 | load: 0.00243 [I 2022-10-26 15:49:20] ...trap/engines/engine.py.230: loss: 0.28662 [I 2022-10-26 15:49:20] ...optimizers/trijoint.py.151: total_norm: 5.093930 [I 2022-10-26 15:49:25] ...trap/engines/engine.py.224: train: epoch 16 | batch 2010/2304 [I 2022-10-26 15:49:25] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:02:26 [I 2022-10-26 15:49:25] ...trap/engines/engine.py.229: process: 0.49843 | load: 0.00253 [I 2022-10-26 15:49:25] ...trap/engines/engine.py.230: loss: 0.21385 [I 2022-10-26 15:49:25] ...optimizers/trijoint.py.151: total_norm: 3.584905 [I 2022-10-26 15:49:30] ...trap/engines/engine.py.224: train: epoch 16 | batch 2020/2304 [I 2022-10-26 15:49:30] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:02:21 [I 2022-10-26 15:49:30] ...trap/engines/engine.py.229: process: 0.50444 | load: 0.00247 [I 2022-10-26 15:49:30] ...trap/engines/engine.py.230: loss: 0.18392 [I 2022-10-26 15:49:30] ...optimizers/trijoint.py.151: total_norm: 2.617708 [I 2022-10-26 15:49:35] ...trap/engines/engine.py.224: train: epoch 16 | batch 2030/2304 [I 2022-10-26 15:49:35] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:02:17 [I 2022-10-26 15:49:35] ...trap/engines/engine.py.229: process: 0.49954 | load: 0.00248 [I 2022-10-26 15:49:35] ...trap/engines/engine.py.230: loss: 0.21232 [I 2022-10-26 15:49:35] ...optimizers/trijoint.py.151: total_norm: 3.554651 [I 2022-10-26 15:49:40] ...trap/engines/engine.py.224: train: epoch 16 | batch 2040/2304 [I 2022-10-26 15:49:40] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:02:12 [I 2022-10-26 15:49:40] ...trap/engines/engine.py.229: process: 0.50508 | load: 0.00249 [I 2022-10-26 15:49:40] ...trap/engines/engine.py.230: loss: 0.27730 [I 2022-10-26 15:49:40] ...optimizers/trijoint.py.151: total_norm: 3.559394 [I 2022-10-26 15:49:46] ...trap/engines/engine.py.224: train: epoch 16 | batch 2050/2304 [I 2022-10-26 15:49:46] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:02:07 [I 2022-10-26 15:49:46] ...trap/engines/engine.py.229: process: 0.50456 | load: 0.00244 [I 2022-10-26 15:49:46] ...trap/engines/engine.py.230: loss: 0.21845 [I 2022-10-26 15:49:46] ...optimizers/trijoint.py.151: total_norm: 3.029972 [I 2022-10-26 15:49:51] ...trap/engines/engine.py.224: train: epoch 16 | batch 2060/2304 [I 2022-10-26 15:49:51] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:02:01 [I 2022-10-26 15:49:51] ...trap/engines/engine.py.229: process: 0.48863 | load: 0.00243 [I 2022-10-26 15:49:51] ...trap/engines/engine.py.230: loss: 0.20526 [I 2022-10-26 15:49:51] ...optimizers/trijoint.py.151: total_norm: 3.117316 [I 2022-10-26 15:49:56] ...trap/engines/engine.py.224: train: epoch 16 | batch 2070/2304 [I 2022-10-26 15:49:56] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:01:59 [I 2022-10-26 15:49:56] ...trap/engines/engine.py.229: process: 0.50285 | load: 0.00252 [I 2022-10-26 15:49:56] ...trap/engines/engine.py.230: loss: 0.17235 [I 2022-10-26 15:49:56] ...optimizers/trijoint.py.151: total_norm: 3.171948 [I 2022-10-26 15:50:01] ...trap/engines/engine.py.224: train: epoch 16 | batch 2080/2304 [I 2022-10-26 15:50:01] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:01:51 [I 2022-10-26 15:50:01] ...trap/engines/engine.py.229: process: 0.50925 | load: 0.00265 [I 2022-10-26 15:50:01] ...trap/engines/engine.py.230: loss: 0.20410 [I 2022-10-26 15:50:01] ...optimizers/trijoint.py.151: total_norm: 3.410755 [I 2022-10-26 15:50:06] ...trap/engines/engine.py.224: train: epoch 16 | batch 2090/2304 [I 2022-10-26 15:50:06] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:01:47 [I 2022-10-26 15:50:06] ...trap/engines/engine.py.229: process: 0.50883 | load: 0.00245 [I 2022-10-26 15:50:06] ...trap/engines/engine.py.230: loss: 0.28336 [I 2022-10-26 15:50:06] ...optimizers/trijoint.py.151: total_norm: 5.434287 [I 2022-10-26 15:50:11] ...trap/engines/engine.py.224: train: epoch 16 | batch 2100/2304 [I 2022-10-26 15:50:11] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:01:42 [I 2022-10-26 15:50:11] ...trap/engines/engine.py.229: process: 0.50233 | load: 0.00244 [I 2022-10-26 15:50:11] ...trap/engines/engine.py.230: loss: 0.12941 [I 2022-10-26 15:50:11] ...optimizers/trijoint.py.151: total_norm: 2.434258 [I 2022-10-26 15:50:16] ...trap/engines/engine.py.224: train: epoch 16 | batch 2110/2304 [I 2022-10-26 15:50:16] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:01:36 [I 2022-10-26 15:50:16] ...trap/engines/engine.py.229: process: 0.49041 | load: 0.00244 [I 2022-10-26 15:50:16] ...trap/engines/engine.py.230: loss: 0.20627 [I 2022-10-26 15:50:16] ...optimizers/trijoint.py.151: total_norm: 3.692013 [I 2022-10-26 15:50:21] ...trap/engines/engine.py.224: train: epoch 16 | batch 2120/2304 [I 2022-10-26 15:50:21] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:01:31 [I 2022-10-26 15:50:21] ...trap/engines/engine.py.229: process: 0.50116 | load: 0.00264 [I 2022-10-26 15:50:21] ...trap/engines/engine.py.230: loss: 0.24267 [I 2022-10-26 15:50:21] ...optimizers/trijoint.py.151: total_norm: 3.189324 [I 2022-10-26 15:50:26] ...trap/engines/engine.py.224: train: epoch 16 | batch 2130/2304 [I 2022-10-26 15:50:26] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:01:27 [I 2022-10-26 15:50:26] ...trap/engines/engine.py.229: process: 0.49530 | load: 0.00261 [I 2022-10-26 15:50:26] ...trap/engines/engine.py.230: loss: 0.22002 [I 2022-10-26 15:50:26] ...optimizers/trijoint.py.151: total_norm: 3.101646 [I 2022-10-26 15:50:31] ...trap/engines/engine.py.224: train: epoch 16 | batch 2140/2304 [I 2022-10-26 15:50:31] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:01:22 [I 2022-10-26 15:50:31] ...trap/engines/engine.py.229: process: 0.51255 | load: 0.00250 [I 2022-10-26 15:50:31] ...trap/engines/engine.py.230: loss: 0.19930 [I 2022-10-26 15:50:31] ...optimizers/trijoint.py.151: total_norm: 2.530209 [I 2022-10-26 15:50:36] ...trap/engines/engine.py.224: train: epoch 16 | batch 2150/2304 [I 2022-10-26 15:50:36] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:01:17 [I 2022-10-26 15:50:36] ...trap/engines/engine.py.229: process: 0.49367 | load: 0.00249 [I 2022-10-26 15:50:36] ...trap/engines/engine.py.230: loss: 0.16420 [I 2022-10-26 15:50:36] ...optimizers/trijoint.py.151: total_norm: 3.007688 [I 2022-10-26 15:50:41] ...trap/engines/engine.py.224: train: epoch 16 | batch 2160/2304 [I 2022-10-26 15:50:41] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:01:12 [I 2022-10-26 15:50:41] ...trap/engines/engine.py.229: process: 0.49924 | load: 0.00248 [I 2022-10-26 15:50:41] ...trap/engines/engine.py.230: loss: 0.30382 [I 2022-10-26 15:50:41] ...optimizers/trijoint.py.151: total_norm: 3.568453 [I 2022-10-26 15:50:46] ...trap/engines/engine.py.224: train: epoch 16 | batch 2170/2304 [I 2022-10-26 15:50:46] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:01:07 [I 2022-10-26 15:50:46] ...trap/engines/engine.py.229: process: 0.52177 | load: 0.00251 [I 2022-10-26 15:50:46] ...trap/engines/engine.py.230: loss: 0.24897 [I 2022-10-26 15:50:46] ...optimizers/trijoint.py.151: total_norm: 4.198788 [I 2022-10-26 15:50:51] ...trap/engines/engine.py.224: train: epoch 16 | batch 2180/2304 [I 2022-10-26 15:50:51] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:01:02 [I 2022-10-26 15:50:51] ...trap/engines/engine.py.229: process: 0.51181 | load: 0.00262 [I 2022-10-26 15:50:51] ...trap/engines/engine.py.230: loss: 0.20419 [I 2022-10-26 15:50:51] ...optimizers/trijoint.py.151: total_norm: 2.739351 [I 2022-10-26 15:50:56] ...trap/engines/engine.py.224: train: epoch 16 | batch 2190/2304 [I 2022-10-26 15:50:56] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:00:56 [I 2022-10-26 15:50:56] ...trap/engines/engine.py.229: process: 0.49307 | load: 0.00248 [I 2022-10-26 15:50:56] ...trap/engines/engine.py.230: loss: 0.19104 [I 2022-10-26 15:50:56] ...optimizers/trijoint.py.151: total_norm: 3.787664 [I 2022-10-26 15:51:01] ...trap/engines/engine.py.224: train: epoch 16 | batch 2200/2304 [I 2022-10-26 15:51:01] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:00:51 [I 2022-10-26 15:51:01] ...trap/engines/engine.py.229: process: 0.50347 | load: 0.00244 [I 2022-10-26 15:51:01] ...trap/engines/engine.py.230: loss: 0.15940 [I 2022-10-26 15:51:01] ...optimizers/trijoint.py.151: total_norm: 2.773979 [I 2022-10-26 15:51:06] ...trap/engines/engine.py.224: train: epoch 16 | batch 2210/2304 [I 2022-10-26 15:51:06] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:00:47 [I 2022-10-26 15:51:06] ...trap/engines/engine.py.229: process: 0.51217 | load: 0.00249 [I 2022-10-26 15:51:06] ...trap/engines/engine.py.230: loss: 0.25992 [I 2022-10-26 15:51:06] ...optimizers/trijoint.py.151: total_norm: 3.165023 [I 2022-10-26 15:51:11] ...trap/engines/engine.py.224: train: epoch 16 | batch 2220/2304 [I 2022-10-26 15:51:11] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:00:41 [I 2022-10-26 15:51:11] ...trap/engines/engine.py.229: process: 0.48976 | load: 0.00250 [I 2022-10-26 15:51:11] ...trap/engines/engine.py.230: loss: 0.17172 [I 2022-10-26 15:51:11] ...optimizers/trijoint.py.151: total_norm: 2.875257 [I 2022-10-26 15:51:16] ...trap/engines/engine.py.224: train: epoch 16 | batch 2230/2304 [I 2022-10-26 15:51:16] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:00:37 [I 2022-10-26 15:51:16] ...trap/engines/engine.py.229: process: 0.50900 | load: 0.00244 [I 2022-10-26 15:51:16] ...trap/engines/engine.py.230: loss: 0.20410 [I 2022-10-26 15:51:16] ...optimizers/trijoint.py.151: total_norm: 3.049399 [I 2022-10-26 15:51:21] ...trap/engines/engine.py.224: train: epoch 16 | batch 2240/2304 [I 2022-10-26 15:51:21] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:00:31 [I 2022-10-26 15:51:21] ...trap/engines/engine.py.229: process: 0.48957 | load: 0.00248 [I 2022-10-26 15:51:21] ...trap/engines/engine.py.230: loss: 0.31520 [I 2022-10-26 15:51:21] ...optimizers/trijoint.py.151: total_norm: 4.243788 [I 2022-10-26 15:51:26] ...trap/engines/engine.py.224: train: epoch 16 | batch 2250/2304 [I 2022-10-26 15:51:26] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:00:27 [I 2022-10-26 15:51:26] ...trap/engines/engine.py.229: process: 0.50762 | load: 0.00269 [I 2022-10-26 15:51:26] ...trap/engines/engine.py.230: loss: 0.11381 [I 2022-10-26 15:51:26] ...optimizers/trijoint.py.151: total_norm: 2.066841 [I 2022-10-26 15:51:31] ...trap/engines/engine.py.224: train: epoch 16 | batch 2260/2304 [I 2022-10-26 15:51:31] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:00:22 [I 2022-10-26 15:51:31] ...trap/engines/engine.py.229: process: 0.50516 | load: 0.00258 [I 2022-10-26 15:51:31] ...trap/engines/engine.py.230: loss: 0.24007 [I 2022-10-26 15:51:31] ...optimizers/trijoint.py.151: total_norm: 3.158437 [I 2022-10-26 15:51:36] ...trap/engines/engine.py.224: train: epoch 16 | batch 2270/2304 [I 2022-10-26 15:51:36] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:00:17 [I 2022-10-26 15:51:36] ...trap/engines/engine.py.229: process: 0.50633 | load: 0.00247 [I 2022-10-26 15:51:36] ...trap/engines/engine.py.230: loss: 0.24702 [I 2022-10-26 15:51:36] ...optimizers/trijoint.py.151: total_norm: 3.887743 [I 2022-10-26 15:51:41] ...trap/engines/engine.py.224: train: epoch 16 | batch 2280/2304 [I 2022-10-26 15:51:41] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:00:12 [I 2022-10-26 15:51:41] ...trap/engines/engine.py.229: process: 0.51006 | load: 0.00246 [I 2022-10-26 15:51:41] ...trap/engines/engine.py.230: loss: 0.25086 [I 2022-10-26 15:51:41] ...optimizers/trijoint.py.151: total_norm: 3.926192 [I 2022-10-26 15:51:46] ...trap/engines/engine.py.224: train: epoch 16 | batch 2290/2304 [I 2022-10-26 15:51:46] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:00:06 [I 2022-10-26 15:51:46] ...trap/engines/engine.py.229: process: 0.49852 | load: 0.00247 [I 2022-10-26 15:51:46] ...trap/engines/engine.py.230: loss: 0.18941 [I 2022-10-26 15:51:46] ...optimizers/trijoint.py.151: total_norm: 2.837397 [I 2022-10-26 15:51:51] ...trap/engines/engine.py.224: train: epoch 16 | batch 2300/2304 [I 2022-10-26 15:51:51] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:00:02 [I 2022-10-26 15:51:51] ...trap/engines/engine.py.229: process: 0.50142 | load: 0.00014 [I 2022-10-26 15:51:51] ...trap/engines/engine.py.230: loss: 0.24649 [I 2022-10-26 15:51:51] ...optimizers/trijoint.py.151: total_norm: 4.826309 [I 2022-10-26 15:51:53] ...trap/engines/engine.py.224: train: epoch 16 | batch 2304/2304 [I 2022-10-26 15:51:53] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:00:00 [I 2022-10-26 15:51:53] ...trap/engines/engine.py.229: process: 0.49993 | load: 0.00016 [I 2022-10-26 15:51:53] ...trap/engines/engine.py.230: loss: 0.30774 [I 2022-10-26 15:51:53] ...optimizers/trijoint.py.151: total_norm: 4.411787 [S 2022-10-26 15:51:54] ...trap/engines/engine.py.129: train_epoch.epoch: 16 [S 2022-10-26 15:51:54] ...trap/engines/engine.py.129: train_epoch.loss: 0.2108820944895196 [I 2022-10-26 15:51:54] ...s/criterions/triplet.py.84: margin = 0.13500000000000006 [I 2022-10-26 15:51:55] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 16 [I 2022-10-26 15:51:55] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 15:51:56] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 15:51:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 15:51:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 15:51:57] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 15:51:57] ...trap/engines/engine.py.321: eval: epoch 16 | batch 0/510 [I 2022-10-26 15:51:57] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:40 [I 2022-10-26 15:51:57] ...trap/engines/engine.py.326: process: 2.78432 | load: 2.41713 [I 2022-10-26 15:52:01] ...trap/engines/engine.py.321: eval: epoch 16 | batch 10/510 [I 2022-10-26 15:52:01] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:58 [I 2022-10-26 15:52:01] ...trap/engines/engine.py.326: process: 0.32500 | load: 0.00046 [I 2022-10-26 15:52:04] ...trap/engines/engine.py.321: eval: epoch 16 | batch 20/510 [I 2022-10-26 15:52:04] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:58 [I 2022-10-26 15:52:04] ...trap/engines/engine.py.326: process: 0.32992 | load: 0.00023 [I 2022-10-26 15:52:07] ...trap/engines/engine.py.321: eval: epoch 16 | batch 30/510 [I 2022-10-26 15:52:07] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:44 [I 2022-10-26 15:52:07] ...trap/engines/engine.py.326: process: 0.33953 | load: 0.00026 [I 2022-10-26 15:52:11] ...trap/engines/engine.py.321: eval: epoch 16 | batch 40/510 [I 2022-10-26 15:52:11] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:41 [I 2022-10-26 15:52:11] ...trap/engines/engine.py.326: process: 0.35649 | load: 0.00032 [I 2022-10-26 15:52:14] ...trap/engines/engine.py.321: eval: epoch 16 | batch 50/510 [I 2022-10-26 15:52:14] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-26 15:52:14] ...trap/engines/engine.py.326: process: 0.32637 | load: 0.00021 [I 2022-10-26 15:52:18] ...trap/engines/engine.py.321: eval: epoch 16 | batch 60/510 [I 2022-10-26 15:52:18] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:30 [I 2022-10-26 15:52:18] ...trap/engines/engine.py.326: process: 0.32164 | load: 0.00024 [I 2022-10-26 15:52:21] ...trap/engines/engine.py.321: eval: epoch 16 | batch 70/510 [I 2022-10-26 15:52:21] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:27 [I 2022-10-26 15:52:21] ...trap/engines/engine.py.326: process: 0.33485 | load: 0.00021 [I 2022-10-26 15:52:24] ...trap/engines/engine.py.321: eval: epoch 16 | batch 80/510 [I 2022-10-26 15:52:24] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:24 [I 2022-10-26 15:52:24] ...trap/engines/engine.py.326: process: 0.34005 | load: 0.00017 [I 2022-10-26 15:52:28] ...trap/engines/engine.py.321: eval: epoch 16 | batch 90/510 [I 2022-10-26 15:52:28] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:21 [I 2022-10-26 15:52:28] ...trap/engines/engine.py.326: process: 0.33821 | load: 0.00020 [I 2022-10-26 15:52:31] ...trap/engines/engine.py.321: eval: epoch 16 | batch 100/510 [I 2022-10-26 15:52:31] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:16 [I 2022-10-26 15:52:31] ...trap/engines/engine.py.326: process: 0.30641 | load: 0.00020 [I 2022-10-26 15:52:34] ...trap/engines/engine.py.321: eval: epoch 16 | batch 110/510 [I 2022-10-26 15:52:34] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:58 [I 2022-10-26 15:52:34] ...trap/engines/engine.py.326: process: 0.27853 | load: 0.00017 [I 2022-10-26 15:52:37] ...trap/engines/engine.py.321: eval: epoch 16 | batch 120/510 [I 2022-10-26 15:52:37] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 15:52:37] ...trap/engines/engine.py.326: process: 0.30074 | load: 0.00017 [I 2022-10-26 15:52:40] ...trap/engines/engine.py.321: eval: epoch 16 | batch 130/510 [I 2022-10-26 15:52:40] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 15:52:40] ...trap/engines/engine.py.326: process: 0.28896 | load: 0.00016 [I 2022-10-26 15:52:43] ...trap/engines/engine.py.321: eval: epoch 16 | batch 140/510 [I 2022-10-26 15:52:43] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:45 [I 2022-10-26 15:52:43] ...trap/engines/engine.py.326: process: 0.28166 | load: 0.00014 [I 2022-10-26 15:52:46] ...trap/engines/engine.py.321: eval: epoch 16 | batch 150/510 [I 2022-10-26 15:52:46] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:42 [I 2022-10-26 15:52:46] ...trap/engines/engine.py.326: process: 0.28061 | load: 0.00019 [I 2022-10-26 15:52:48] ...trap/engines/engine.py.321: eval: epoch 16 | batch 160/510 [I 2022-10-26 15:52:48] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 15:52:48] ...trap/engines/engine.py.326: process: 0.29269 | load: 0.00016 [I 2022-10-26 15:52:51] ...trap/engines/engine.py.321: eval: epoch 16 | batch 170/510 [I 2022-10-26 15:52:51] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 15:52:51] ...trap/engines/engine.py.326: process: 0.29316 | load: 0.00016 [I 2022-10-26 15:52:54] ...trap/engines/engine.py.321: eval: epoch 16 | batch 180/510 [I 2022-10-26 15:52:54] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 15:52:54] ...trap/engines/engine.py.326: process: 0.29594 | load: 0.00015 [I 2022-10-26 15:52:57] ...trap/engines/engine.py.321: eval: epoch 16 | batch 190/510 [I 2022-10-26 15:52:57] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 15:52:57] ...trap/engines/engine.py.326: process: 0.28590 | load: 0.00014 [I 2022-10-26 15:53:00] ...trap/engines/engine.py.321: eval: epoch 16 | batch 200/510 [I 2022-10-26 15:53:00] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 15:53:00] ...trap/engines/engine.py.326: process: 0.28441 | load: 0.00022 [I 2022-10-26 15:53:03] ...trap/engines/engine.py.321: eval: epoch 16 | batch 210/510 [I 2022-10-26 15:53:03] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 15:53:03] ...trap/engines/engine.py.326: process: 0.29268 | load: 0.00014 [I 2022-10-26 15:53:06] ...trap/engines/engine.py.321: eval: epoch 16 | batch 220/510 [I 2022-10-26 15:53:06] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 15:53:06] ...trap/engines/engine.py.326: process: 0.28992 | load: 0.00015 [I 2022-10-26 15:53:09] ...trap/engines/engine.py.321: eval: epoch 16 | batch 230/510 [I 2022-10-26 15:53:09] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 15:53:09] ...trap/engines/engine.py.326: process: 0.28585 | load: 0.00020 [I 2022-10-26 15:53:12] ...trap/engines/engine.py.321: eval: epoch 16 | batch 240/510 [I 2022-10-26 15:53:12] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 15:53:12] ...trap/engines/engine.py.326: process: 0.29368 | load: 0.00016 [I 2022-10-26 15:53:15] ...trap/engines/engine.py.321: eval: epoch 16 | batch 250/510 [I 2022-10-26 15:53:15] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:16 [I 2022-10-26 15:53:15] ...trap/engines/engine.py.326: process: 0.28874 | load: 0.00024 [I 2022-10-26 15:53:17] ...trap/engines/engine.py.321: eval: epoch 16 | batch 260/510 [I 2022-10-26 15:53:17] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:13 [I 2022-10-26 15:53:17] ...trap/engines/engine.py.326: process: 0.29233 | load: 0.00017 [I 2022-10-26 15:53:20] ...trap/engines/engine.py.321: eval: epoch 16 | batch 270/510 [I 2022-10-26 15:53:20] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 15:53:20] ...trap/engines/engine.py.326: process: 0.28935 | load: 0.00016 [I 2022-10-26 15:53:23] ...trap/engines/engine.py.321: eval: epoch 16 | batch 280/510 [I 2022-10-26 15:53:23] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 15:53:23] ...trap/engines/engine.py.326: process: 0.29449 | load: 0.00015 [I 2022-10-26 15:53:26] ...trap/engines/engine.py.321: eval: epoch 16 | batch 290/510 [I 2022-10-26 15:53:26] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 15:53:26] ...trap/engines/engine.py.326: process: 0.28615 | load: 0.00015 [I 2022-10-26 15:53:29] ...trap/engines/engine.py.321: eval: epoch 16 | batch 300/510 [I 2022-10-26 15:53:29] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 15:53:29] ...trap/engines/engine.py.326: process: 0.28752 | load: 0.00019 [I 2022-10-26 15:53:32] ...trap/engines/engine.py.321: eval: epoch 16 | batch 310/510 [I 2022-10-26 15:53:32] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 15:53:32] ...trap/engines/engine.py.326: process: 0.29136 | load: 0.00016 [I 2022-10-26 15:53:35] ...trap/engines/engine.py.321: eval: epoch 16 | batch 320/510 [I 2022-10-26 15:53:35] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 15:53:35] ...trap/engines/engine.py.326: process: 0.29070 | load: 0.00016 [I 2022-10-26 15:53:38] ...trap/engines/engine.py.321: eval: epoch 16 | batch 330/510 [I 2022-10-26 15:53:38] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 15:53:38] ...trap/engines/engine.py.326: process: 0.29038 | load: 0.00016 [I 2022-10-26 15:53:41] ...trap/engines/engine.py.321: eval: epoch 16 | batch 340/510 [I 2022-10-26 15:53:41] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:49 [I 2022-10-26 15:53:41] ...trap/engines/engine.py.326: process: 0.29516 | load: 0.00016 [I 2022-10-26 15:53:43] ...trap/engines/engine.py.321: eval: epoch 16 | batch 350/510 [I 2022-10-26 15:53:43] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 15:53:43] ...trap/engines/engine.py.326: process: 0.29281 | load: 0.00016 [I 2022-10-26 15:53:46] ...trap/engines/engine.py.321: eval: epoch 16 | batch 360/510 [I 2022-10-26 15:53:46] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 15:53:46] ...trap/engines/engine.py.326: process: 0.29062 | load: 0.00019 [I 2022-10-26 15:53:49] ...trap/engines/engine.py.321: eval: epoch 16 | batch 370/510 [I 2022-10-26 15:53:49] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 15:53:49] ...trap/engines/engine.py.326: process: 0.28121 | load: 0.00023 [I 2022-10-26 15:53:52] ...trap/engines/engine.py.321: eval: epoch 16 | batch 380/510 [I 2022-10-26 15:53:52] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 15:53:52] ...trap/engines/engine.py.326: process: 0.29201 | load: 0.00015 [I 2022-10-26 15:53:55] ...trap/engines/engine.py.321: eval: epoch 16 | batch 390/510 [I 2022-10-26 15:53:55] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 15:53:55] ...trap/engines/engine.py.326: process: 0.28871 | load: 0.00015 [I 2022-10-26 15:53:58] ...trap/engines/engine.py.321: eval: epoch 16 | batch 400/510 [I 2022-10-26 15:53:58] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 15:53:58] ...trap/engines/engine.py.326: process: 0.29038 | load: 0.00020 [I 2022-10-26 15:54:01] ...trap/engines/engine.py.321: eval: epoch 16 | batch 410/510 [I 2022-10-26 15:54:01] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 15:54:01] ...trap/engines/engine.py.326: process: 0.28798 | load: 0.00015 [I 2022-10-26 15:54:04] ...trap/engines/engine.py.321: eval: epoch 16 | batch 420/510 [I 2022-10-26 15:54:04] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 15:54:04] ...trap/engines/engine.py.326: process: 0.29330 | load: 0.00016 [I 2022-10-26 15:54:06] ...trap/engines/engine.py.321: eval: epoch 16 | batch 430/510 [I 2022-10-26 15:54:06] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:22 [I 2022-10-26 15:54:06] ...trap/engines/engine.py.326: process: 0.28368 | load: 0.00016 [I 2022-10-26 15:54:09] ...trap/engines/engine.py.321: eval: epoch 16 | batch 440/510 [I 2022-10-26 15:54:09] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 15:54:09] ...trap/engines/engine.py.326: process: 0.29240 | load: 0.00021 [I 2022-10-26 15:54:12] ...trap/engines/engine.py.321: eval: epoch 16 | batch 450/510 [I 2022-10-26 15:54:12] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 15:54:12] ...trap/engines/engine.py.326: process: 0.29108 | load: 0.00016 [I 2022-10-26 15:54:15] ...trap/engines/engine.py.321: eval: epoch 16 | batch 460/510 [I 2022-10-26 15:54:15] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 15:54:15] ...trap/engines/engine.py.326: process: 0.28301 | load: 0.00016 [I 2022-10-26 15:54:18] ...trap/engines/engine.py.321: eval: epoch 16 | batch 470/510 [I 2022-10-26 15:54:18] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 15:54:18] ...trap/engines/engine.py.326: process: 0.28423 | load: 0.00020 [I 2022-10-26 15:54:21] ...trap/engines/engine.py.321: eval: epoch 16 | batch 480/510 [I 2022-10-26 15:54:21] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 15:54:21] ...trap/engines/engine.py.326: process: 0.28543 | load: 0.00020 [I 2022-10-26 15:54:24] ...trap/engines/engine.py.321: eval: epoch 16 | batch 490/510 [I 2022-10-26 15:54:24] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 15:54:24] ...trap/engines/engine.py.326: process: 0.31043 | load: 0.00015 [I 2022-10-26 15:54:27] ...trap/engines/engine.py.321: eval: epoch 16 | batch 500/510 [I 2022-10-26 15:54:27] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 15:54:27] ...trap/engines/engine.py.326: process: 0.28375 | load: 0.00015 [I 2022-10-26 15:54:30] ...trap/engines/engine.py.321: eval: epoch 16 | batch 510/510 [I 2022-10-26 15:54:30] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 15:54:30] ...trap/engines/engine.py.326: process: 0.28211 | load: 0.00011 [S 2022-10-26 15:54:30] ...trap/engines/engine.py.132: eval_epoch.epoch: 16 [I 2022-10-26 15:54:30] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.556900 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.827700 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.889800 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.557000 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.831100 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.894600 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012864 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010296 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008542 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.019870 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009016 [S 2022-10-26 15:54:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006232 [I 2022-10-26 15:54:35] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 15:54:35] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 15:54:35] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 15:54:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 15:54:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 15:54:36] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 15:54:39] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 15:54:44] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 15:54:44] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 15:54:44] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 15:54:47] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 15:54:52] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 15:54:52] ...trap/engines/engine.py.163: Training model on trainset for epoch 17 [I 2022-10-26 15:54:52] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 15:54:55] ...trap/engines/engine.py.224: train: epoch 17 | batch 0/2304 [I 2022-10-26 15:54:55] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:45:17 [I 2022-10-26 15:54:55] ...trap/engines/engine.py.229: process: 2.74198 | load: 2.15555 [I 2022-10-26 15:54:55] ...trap/engines/engine.py.230: loss: 0.17240 [I 2022-10-26 15:54:55] ...optimizers/trijoint.py.151: total_norm: 2.785170 [I 2022-10-26 15:55:00] ...trap/engines/engine.py.224: train: epoch 17 | batch 10/2304 [I 2022-10-26 15:55:00] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:08 [I 2022-10-26 15:55:00] ...trap/engines/engine.py.229: process: 0.48889 | load: 0.00498 [I 2022-10-26 15:55:00] ...trap/engines/engine.py.230: loss: 0.19361 [I 2022-10-26 15:55:00] ...optimizers/trijoint.py.151: total_norm: 3.119538 [I 2022-10-26 15:55:05] ...trap/engines/engine.py.224: train: epoch 17 | batch 20/2304 [I 2022-10-26 15:55:05] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:09 [I 2022-10-26 15:55:05] ...trap/engines/engine.py.229: process: 0.52212 | load: 0.00360 [I 2022-10-26 15:55:05] ...trap/engines/engine.py.230: loss: 0.09400 [I 2022-10-26 15:55:05] ...optimizers/trijoint.py.151: total_norm: 1.524560 [I 2022-10-26 15:55:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 30/2304 [I 2022-10-26 15:55:10] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:00 [I 2022-10-26 15:55:10] ...trap/engines/engine.py.229: process: 0.49903 | load: 0.00346 [I 2022-10-26 15:55:10] ...trap/engines/engine.py.230: loss: 0.24642 [I 2022-10-26 15:55:10] ...optimizers/trijoint.py.151: total_norm: 3.214465 [I 2022-10-26 15:55:15] ...trap/engines/engine.py.224: train: epoch 17 | batch 40/2304 [I 2022-10-26 15:55:15] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:50 [I 2022-10-26 15:55:15] ...trap/engines/engine.py.229: process: 0.49804 | load: 0.00302 [I 2022-10-26 15:55:15] ...trap/engines/engine.py.230: loss: 0.22053 [I 2022-10-26 15:55:15] ...optimizers/trijoint.py.151: total_norm: 3.422644 [I 2022-10-26 15:55:20] ...trap/engines/engine.py.224: train: epoch 17 | batch 50/2304 [I 2022-10-26 15:55:20] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:49 [I 2022-10-26 15:55:20] ...trap/engines/engine.py.229: process: 0.49372 | load: 0.00288 [I 2022-10-26 15:55:20] ...trap/engines/engine.py.230: loss: 0.26325 [I 2022-10-26 15:55:20] ...optimizers/trijoint.py.151: total_norm: 4.049510 [I 2022-10-26 15:55:25] ...trap/engines/engine.py.224: train: epoch 17 | batch 60/2304 [I 2022-10-26 15:55:25] ...trap/engines/engine.py.228: elapsed: 0:00:33 | left: 0:19:07 [I 2022-10-26 15:55:25] ...trap/engines/engine.py.229: process: 0.51695 | load: 0.00274 [I 2022-10-26 15:55:25] ...trap/engines/engine.py.230: loss: 0.17491 [I 2022-10-26 15:55:25] ...optimizers/trijoint.py.151: total_norm: 2.782386 [I 2022-10-26 15:55:30] ...trap/engines/engine.py.224: train: epoch 17 | batch 70/2304 [I 2022-10-26 15:55:30] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:18:42 [I 2022-10-26 15:55:30] ...trap/engines/engine.py.229: process: 0.49615 | load: 0.00263 [I 2022-10-26 15:55:30] ...trap/engines/engine.py.230: loss: 0.21804 [I 2022-10-26 15:55:30] ...optimizers/trijoint.py.151: total_norm: 3.565978 [I 2022-10-26 15:55:35] ...trap/engines/engine.py.224: train: epoch 17 | batch 80/2304 [I 2022-10-26 15:55:35] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:18:45 [I 2022-10-26 15:55:35] ...trap/engines/engine.py.229: process: 0.49643 | load: 0.00270 [I 2022-10-26 15:55:35] ...trap/engines/engine.py.230: loss: 0.21265 [I 2022-10-26 15:55:35] ...optimizers/trijoint.py.151: total_norm: 4.076757 [I 2022-10-26 15:55:40] ...trap/engines/engine.py.224: train: epoch 17 | batch 90/2304 [I 2022-10-26 15:55:40] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:18:34 [I 2022-10-26 15:55:40] ...trap/engines/engine.py.229: process: 0.49952 | load: 0.00272 [I 2022-10-26 15:55:40] ...trap/engines/engine.py.230: loss: 0.17001 [I 2022-10-26 15:55:40] ...optimizers/trijoint.py.151: total_norm: 3.064906 [I 2022-10-26 15:55:45] ...trap/engines/engine.py.224: train: epoch 17 | batch 100/2304 [I 2022-10-26 15:55:45] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:18:27 [I 2022-10-26 15:55:45] ...trap/engines/engine.py.229: process: 0.51054 | load: 0.00281 [I 2022-10-26 15:55:45] ...trap/engines/engine.py.230: loss: 0.26632 [I 2022-10-26 15:55:45] ...optimizers/trijoint.py.151: total_norm: 4.696852 [I 2022-10-26 15:55:50] ...trap/engines/engine.py.224: train: epoch 17 | batch 110/2304 [I 2022-10-26 15:55:50] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:18:16 [I 2022-10-26 15:55:50] ...trap/engines/engine.py.229: process: 0.50308 | load: 0.00268 [I 2022-10-26 15:55:50] ...trap/engines/engine.py.230: loss: 0.18576 [I 2022-10-26 15:55:50] ...optimizers/trijoint.py.151: total_norm: 2.687971 [I 2022-10-26 15:55:55] ...trap/engines/engine.py.224: train: epoch 17 | batch 120/2304 [I 2022-10-26 15:55:55] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:21 [I 2022-10-26 15:55:55] ...trap/engines/engine.py.229: process: 0.50106 | load: 0.00304 [I 2022-10-26 15:55:55] ...trap/engines/engine.py.230: loss: 0.16211 [I 2022-10-26 15:55:55] ...optimizers/trijoint.py.151: total_norm: 2.901064 [I 2022-10-26 15:56:00] ...trap/engines/engine.py.224: train: epoch 17 | batch 130/2304 [I 2022-10-26 15:56:00] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:13 [I 2022-10-26 15:56:00] ...trap/engines/engine.py.229: process: 0.49312 | load: 0.00292 [I 2022-10-26 15:56:00] ...trap/engines/engine.py.230: loss: 0.12293 [I 2022-10-26 15:56:00] ...optimizers/trijoint.py.151: total_norm: 2.379900 [I 2022-10-26 15:56:05] ...trap/engines/engine.py.224: train: epoch 17 | batch 140/2304 [I 2022-10-26 15:56:05] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:18:00 [I 2022-10-26 15:56:05] ...trap/engines/engine.py.229: process: 0.50110 | load: 0.00264 [I 2022-10-26 15:56:05] ...trap/engines/engine.py.230: loss: 0.19563 [I 2022-10-26 15:56:05] ...optimizers/trijoint.py.151: total_norm: 3.116289 [I 2022-10-26 15:56:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 150/2304 [I 2022-10-26 15:56:10] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:18:24 [I 2022-10-26 15:56:10] ...trap/engines/engine.py.229: process: 0.52786 | load: 0.00253 [I 2022-10-26 15:56:10] ...trap/engines/engine.py.230: loss: 0.23787 [I 2022-10-26 15:56:10] ...optimizers/trijoint.py.151: total_norm: 4.030029 [I 2022-10-26 15:56:15] ...trap/engines/engine.py.224: train: epoch 17 | batch 160/2304 [I 2022-10-26 15:56:15] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:18:04 [I 2022-10-26 15:56:15] ...trap/engines/engine.py.229: process: 0.50176 | load: 0.00273 [I 2022-10-26 15:56:15] ...trap/engines/engine.py.230: loss: 0.19413 [I 2022-10-26 15:56:15] ...optimizers/trijoint.py.151: total_norm: 3.554090 [I 2022-10-26 15:56:21] ...trap/engines/engine.py.224: train: epoch 17 | batch 170/2304 [I 2022-10-26 15:56:21] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:58 [I 2022-10-26 15:56:21] ...trap/engines/engine.py.229: process: 0.50259 | load: 0.00275 [I 2022-10-26 15:56:21] ...trap/engines/engine.py.230: loss: 0.21364 [I 2022-10-26 15:56:21] ...optimizers/trijoint.py.151: total_norm: 3.127082 [I 2022-10-26 15:56:26] ...trap/engines/engine.py.224: train: epoch 17 | batch 180/2304 [I 2022-10-26 15:56:26] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:17:48 [I 2022-10-26 15:56:26] ...trap/engines/engine.py.229: process: 0.51394 | load: 0.00272 [I 2022-10-26 15:56:26] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-26 15:56:26] ...optimizers/trijoint.py.151: total_norm: 2.502836 [I 2022-10-26 15:56:31] ...trap/engines/engine.py.224: train: epoch 17 | batch 190/2304 [I 2022-10-26 15:56:31] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:17:44 [I 2022-10-26 15:56:31] ...trap/engines/engine.py.229: process: 0.50326 | load: 0.00290 [I 2022-10-26 15:56:31] ...trap/engines/engine.py.230: loss: 0.21766 [I 2022-10-26 15:56:31] ...optimizers/trijoint.py.151: total_norm: 3.637382 [I 2022-10-26 15:56:36] ...trap/engines/engine.py.224: train: epoch 17 | batch 200/2304 [I 2022-10-26 15:56:36] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:26 [I 2022-10-26 15:56:36] ...trap/engines/engine.py.229: process: 0.49019 | load: 0.00249 [I 2022-10-26 15:56:36] ...trap/engines/engine.py.230: loss: 0.22820 [I 2022-10-26 15:56:36] ...optimizers/trijoint.py.151: total_norm: 3.674168 [I 2022-10-26 15:56:41] ...trap/engines/engine.py.224: train: epoch 17 | batch 210/2304 [I 2022-10-26 15:56:41] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:41 [I 2022-10-26 15:56:41] ...trap/engines/engine.py.229: process: 0.49934 | load: 0.00252 [I 2022-10-26 15:56:41] ...trap/engines/engine.py.230: loss: 0.28294 [I 2022-10-26 15:56:41] ...optimizers/trijoint.py.151: total_norm: 3.289673 [I 2022-10-26 15:56:46] ...trap/engines/engine.py.224: train: epoch 17 | batch 220/2304 [I 2022-10-26 15:56:46] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:22 [I 2022-10-26 15:56:46] ...trap/engines/engine.py.229: process: 0.49571 | load: 0.00256 [I 2022-10-26 15:56:46] ...trap/engines/engine.py.230: loss: 0.20165 [I 2022-10-26 15:56:46] ...optimizers/trijoint.py.151: total_norm: 3.319000 [I 2022-10-26 15:56:51] ...trap/engines/engine.py.224: train: epoch 17 | batch 230/2304 [I 2022-10-26 15:56:51] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:24 [I 2022-10-26 15:56:51] ...trap/engines/engine.py.229: process: 0.49939 | load: 0.00273 [I 2022-10-26 15:56:51] ...trap/engines/engine.py.230: loss: 0.22595 [I 2022-10-26 15:56:51] ...optimizers/trijoint.py.151: total_norm: 2.678127 [I 2022-10-26 15:56:56] ...trap/engines/engine.py.224: train: epoch 17 | batch 240/2304 [I 2022-10-26 15:56:56] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:08 [I 2022-10-26 15:56:56] ...trap/engines/engine.py.229: process: 0.49828 | load: 0.00256 [I 2022-10-26 15:56:56] ...trap/engines/engine.py.230: loss: 0.30150 [I 2022-10-26 15:56:56] ...optimizers/trijoint.py.151: total_norm: 3.321618 [I 2022-10-26 15:57:01] ...trap/engines/engine.py.224: train: epoch 17 | batch 250/2304 [I 2022-10-26 15:57:01] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:03 [I 2022-10-26 15:57:01] ...trap/engines/engine.py.229: process: 0.50010 | load: 0.00251 [I 2022-10-26 15:57:01] ...trap/engines/engine.py.230: loss: 0.15143 [I 2022-10-26 15:57:01] ...optimizers/trijoint.py.151: total_norm: 3.185400 [I 2022-10-26 15:57:06] ...trap/engines/engine.py.224: train: epoch 17 | batch 260/2304 [I 2022-10-26 15:57:06] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:02 [I 2022-10-26 15:57:06] ...trap/engines/engine.py.229: process: 0.50745 | load: 0.00253 [I 2022-10-26 15:57:06] ...trap/engines/engine.py.230: loss: 0.21956 [I 2022-10-26 15:57:06] ...optimizers/trijoint.py.151: total_norm: 3.825469 [I 2022-10-26 15:57:11] ...trap/engines/engine.py.224: train: epoch 17 | batch 270/2304 [I 2022-10-26 15:57:11] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:17:07 [I 2022-10-26 15:57:11] ...trap/engines/engine.py.229: process: 0.50727 | load: 0.00258 [I 2022-10-26 15:57:11] ...trap/engines/engine.py.230: loss: 0.14112 [I 2022-10-26 15:57:11] ...optimizers/trijoint.py.151: total_norm: 2.801687 [I 2022-10-26 15:57:16] ...trap/engines/engine.py.224: train: epoch 17 | batch 280/2304 [I 2022-10-26 15:57:16] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:16:53 [I 2022-10-26 15:57:16] ...trap/engines/engine.py.229: process: 0.50061 | load: 0.00245 [I 2022-10-26 15:57:16] ...trap/engines/engine.py.230: loss: 0.22540 [I 2022-10-26 15:57:16] ...optimizers/trijoint.py.151: total_norm: 3.327358 [I 2022-10-26 15:57:21] ...trap/engines/engine.py.224: train: epoch 17 | batch 290/2304 [I 2022-10-26 15:57:21] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:16:42 [I 2022-10-26 15:57:21] ...trap/engines/engine.py.229: process: 0.48722 | load: 0.00249 [I 2022-10-26 15:57:21] ...trap/engines/engine.py.230: loss: 0.19101 [I 2022-10-26 15:57:21] ...optimizers/trijoint.py.151: total_norm: 3.687521 [I 2022-10-26 15:57:26] ...trap/engines/engine.py.224: train: epoch 17 | batch 300/2304 [I 2022-10-26 15:57:26] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:16:44 [I 2022-10-26 15:57:26] ...trap/engines/engine.py.229: process: 0.49753 | load: 0.00259 [I 2022-10-26 15:57:26] ...trap/engines/engine.py.230: loss: 0.19458 [I 2022-10-26 15:57:26] ...optimizers/trijoint.py.151: total_norm: 3.097197 [I 2022-10-26 15:57:31] ...trap/engines/engine.py.224: train: epoch 17 | batch 310/2304 [I 2022-10-26 15:57:31] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:16:28 [I 2022-10-26 15:57:31] ...trap/engines/engine.py.229: process: 0.49324 | load: 0.00243 [I 2022-10-26 15:57:31] ...trap/engines/engine.py.230: loss: 0.23204 [I 2022-10-26 15:57:31] ...optimizers/trijoint.py.151: total_norm: 2.906482 [I 2022-10-26 15:57:36] ...trap/engines/engine.py.224: train: epoch 17 | batch 320/2304 [I 2022-10-26 15:57:36] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:16:34 [I 2022-10-26 15:57:36] ...trap/engines/engine.py.229: process: 0.50645 | load: 0.00247 [I 2022-10-26 15:57:36] ...trap/engines/engine.py.230: loss: 0.23097 [I 2022-10-26 15:57:36] ...optimizers/trijoint.py.151: total_norm: 2.920130 [I 2022-10-26 15:57:41] ...trap/engines/engine.py.224: train: epoch 17 | batch 330/2304 [I 2022-10-26 15:57:41] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:16:13 [I 2022-10-26 15:57:41] ...trap/engines/engine.py.229: process: 0.47747 | load: 0.00245 [I 2022-10-26 15:57:41] ...trap/engines/engine.py.230: loss: 0.26158 [I 2022-10-26 15:57:41] ...optimizers/trijoint.py.151: total_norm: 4.450828 [I 2022-10-26 15:57:46] ...trap/engines/engine.py.224: train: epoch 17 | batch 340/2304 [I 2022-10-26 15:57:46] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:16:16 [I 2022-10-26 15:57:46] ...trap/engines/engine.py.229: process: 0.49791 | load: 0.00244 [I 2022-10-26 15:57:46] ...trap/engines/engine.py.230: loss: 0.22882 [I 2022-10-26 15:57:46] ...optimizers/trijoint.py.151: total_norm: 3.906549 [I 2022-10-26 15:57:51] ...trap/engines/engine.py.224: train: epoch 17 | batch 350/2304 [I 2022-10-26 15:57:51] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:16:19 [I 2022-10-26 15:57:51] ...trap/engines/engine.py.229: process: 0.50565 | load: 0.00241 [I 2022-10-26 15:57:51] ...trap/engines/engine.py.230: loss: 0.14244 [I 2022-10-26 15:57:51] ...optimizers/trijoint.py.151: total_norm: 3.347278 [I 2022-10-26 15:57:56] ...trap/engines/engine.py.224: train: epoch 17 | batch 360/2304 [I 2022-10-26 15:57:56] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:16:06 [I 2022-10-26 15:57:56] ...trap/engines/engine.py.229: process: 0.50337 | load: 0.00244 [I 2022-10-26 15:57:56] ...trap/engines/engine.py.230: loss: 0.27680 [I 2022-10-26 15:57:56] ...optimizers/trijoint.py.151: total_norm: 3.502156 [I 2022-10-26 15:58:01] ...trap/engines/engine.py.224: train: epoch 17 | batch 370/2304 [I 2022-10-26 15:58:01] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:16:16 [I 2022-10-26 15:58:01] ...trap/engines/engine.py.229: process: 0.49777 | load: 0.00241 [I 2022-10-26 15:58:01] ...trap/engines/engine.py.230: loss: 0.13752 [I 2022-10-26 15:58:01] ...optimizers/trijoint.py.151: total_norm: 2.795087 [I 2022-10-26 15:58:06] ...trap/engines/engine.py.224: train: epoch 17 | batch 380/2304 [I 2022-10-26 15:58:06] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:16:23 [I 2022-10-26 15:58:06] ...trap/engines/engine.py.229: process: 0.51661 | load: 0.00250 [I 2022-10-26 15:58:06] ...trap/engines/engine.py.230: loss: 0.27913 [I 2022-10-26 15:58:06] ...optimizers/trijoint.py.151: total_norm: 4.162392 [I 2022-10-26 15:58:11] ...trap/engines/engine.py.224: train: epoch 17 | batch 390/2304 [I 2022-10-26 15:58:11] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:15:51 [I 2022-10-26 15:58:11] ...trap/engines/engine.py.229: process: 0.49395 | load: 0.00260 [I 2022-10-26 15:58:11] ...trap/engines/engine.py.230: loss: 0.21636 [I 2022-10-26 15:58:11] ...optimizers/trijoint.py.151: total_norm: 3.346726 [I 2022-10-26 15:58:16] ...trap/engines/engine.py.224: train: epoch 17 | batch 400/2304 [I 2022-10-26 15:58:16] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:15:51 [I 2022-10-26 15:58:16] ...trap/engines/engine.py.229: process: 0.49228 | load: 0.00250 [I 2022-10-26 15:58:16] ...trap/engines/engine.py.230: loss: 0.12831 [I 2022-10-26 15:58:16] ...optimizers/trijoint.py.151: total_norm: 2.528859 [I 2022-10-26 15:58:21] ...trap/engines/engine.py.224: train: epoch 17 | batch 410/2304 [I 2022-10-26 15:58:21] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:15:44 [I 2022-10-26 15:58:21] ...trap/engines/engine.py.229: process: 0.49693 | load: 0.00251 [I 2022-10-26 15:58:21] ...trap/engines/engine.py.230: loss: 0.21359 [I 2022-10-26 15:58:21] ...optimizers/trijoint.py.151: total_norm: 3.172073 [I 2022-10-26 15:58:26] ...trap/engines/engine.py.224: train: epoch 17 | batch 420/2304 [I 2022-10-26 15:58:26] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:15:48 [I 2022-10-26 15:58:26] ...trap/engines/engine.py.229: process: 0.51731 | load: 0.00245 [I 2022-10-26 15:58:26] ...trap/engines/engine.py.230: loss: 0.21253 [I 2022-10-26 15:58:26] ...optimizers/trijoint.py.151: total_norm: 4.630341 [I 2022-10-26 15:58:31] ...trap/engines/engine.py.224: train: epoch 17 | batch 430/2304 [I 2022-10-26 15:58:31] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:15:35 [I 2022-10-26 15:58:31] ...trap/engines/engine.py.229: process: 0.49960 | load: 0.00247 [I 2022-10-26 15:58:31] ...trap/engines/engine.py.230: loss: 0.14190 [I 2022-10-26 15:58:31] ...optimizers/trijoint.py.151: total_norm: 3.392946 [I 2022-10-26 15:58:36] ...trap/engines/engine.py.224: train: epoch 17 | batch 440/2304 [I 2022-10-26 15:58:36] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:15:27 [I 2022-10-26 15:58:36] ...trap/engines/engine.py.229: process: 0.49177 | load: 0.00249 [I 2022-10-26 15:58:36] ...trap/engines/engine.py.230: loss: 0.19354 [I 2022-10-26 15:58:36] ...optimizers/trijoint.py.151: total_norm: 2.706084 [I 2022-10-26 15:58:41] ...trap/engines/engine.py.224: train: epoch 17 | batch 450/2304 [I 2022-10-26 15:58:41] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:15:38 [I 2022-10-26 15:58:41] ...trap/engines/engine.py.229: process: 0.53067 | load: 0.00253 [I 2022-10-26 15:58:41] ...trap/engines/engine.py.230: loss: 0.16050 [I 2022-10-26 15:58:41] ...optimizers/trijoint.py.151: total_norm: 3.436800 [I 2022-10-26 15:58:46] ...trap/engines/engine.py.224: train: epoch 17 | batch 460/2304 [I 2022-10-26 15:58:46] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:15:23 [I 2022-10-26 15:58:46] ...trap/engines/engine.py.229: process: 0.50103 | load: 0.00252 [I 2022-10-26 15:58:46] ...trap/engines/engine.py.230: loss: 0.15871 [I 2022-10-26 15:58:46] ...optimizers/trijoint.py.151: total_norm: 2.536910 [I 2022-10-26 15:58:51] ...trap/engines/engine.py.224: train: epoch 17 | batch 470/2304 [I 2022-10-26 15:58:51] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:15:08 [I 2022-10-26 15:58:51] ...trap/engines/engine.py.229: process: 0.49363 | load: 0.00251 [I 2022-10-26 15:58:51] ...trap/engines/engine.py.230: loss: 0.19710 [I 2022-10-26 15:58:51] ...optimizers/trijoint.py.151: total_norm: 3.459529 [I 2022-10-26 15:58:56] ...trap/engines/engine.py.224: train: epoch 17 | batch 480/2304 [I 2022-10-26 15:58:56] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:15:16 [I 2022-10-26 15:58:56] ...trap/engines/engine.py.229: process: 0.51152 | load: 0.00251 [I 2022-10-26 15:58:56] ...trap/engines/engine.py.230: loss: 0.20415 [I 2022-10-26 15:58:56] ...optimizers/trijoint.py.151: total_norm: 3.249846 [I 2022-10-26 15:59:01] ...trap/engines/engine.py.224: train: epoch 17 | batch 490/2304 [I 2022-10-26 15:59:01] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:15:14 [I 2022-10-26 15:59:01] ...trap/engines/engine.py.229: process: 0.49121 | load: 0.00279 [I 2022-10-26 15:59:01] ...trap/engines/engine.py.230: loss: 0.21752 [I 2022-10-26 15:59:01] ...optimizers/trijoint.py.151: total_norm: 3.489053 [I 2022-10-26 15:59:06] ...trap/engines/engine.py.224: train: epoch 17 | batch 500/2304 [I 2022-10-26 15:59:06] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:15:10 [I 2022-10-26 15:59:06] ...trap/engines/engine.py.229: process: 0.49538 | load: 0.00254 [I 2022-10-26 15:59:06] ...trap/engines/engine.py.230: loss: 0.19385 [I 2022-10-26 15:59:06] ...optimizers/trijoint.py.151: total_norm: 2.929355 [I 2022-10-26 15:59:11] ...trap/engines/engine.py.224: train: epoch 17 | batch 510/2304 [I 2022-10-26 15:59:11] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:14:59 [I 2022-10-26 15:59:11] ...trap/engines/engine.py.229: process: 0.50037 | load: 0.00250 [I 2022-10-26 15:59:11] ...trap/engines/engine.py.230: loss: 0.18924 [I 2022-10-26 15:59:11] ...optimizers/trijoint.py.151: total_norm: 2.117283 [I 2022-10-26 15:59:16] ...trap/engines/engine.py.224: train: epoch 17 | batch 520/2304 [I 2022-10-26 15:59:16] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:14:44 [I 2022-10-26 15:59:16] ...trap/engines/engine.py.229: process: 0.48014 | load: 0.00244 [I 2022-10-26 15:59:16] ...trap/engines/engine.py.230: loss: 0.18905 [I 2022-10-26 15:59:16] ...optimizers/trijoint.py.151: total_norm: 3.130018 [I 2022-10-26 15:59:21] ...trap/engines/engine.py.224: train: epoch 17 | batch 530/2304 [I 2022-10-26 15:59:21] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:14:45 [I 2022-10-26 15:59:21] ...trap/engines/engine.py.229: process: 0.49988 | load: 0.00267 [I 2022-10-26 15:59:21] ...trap/engines/engine.py.230: loss: 0.22173 [I 2022-10-26 15:59:21] ...optimizers/trijoint.py.151: total_norm: 3.563385 [I 2022-10-26 15:59:26] ...trap/engines/engine.py.224: train: epoch 17 | batch 540/2304 [I 2022-10-26 15:59:26] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:14:37 [I 2022-10-26 15:59:26] ...trap/engines/engine.py.229: process: 0.49299 | load: 0.00255 [I 2022-10-26 15:59:26] ...trap/engines/engine.py.230: loss: 0.18831 [I 2022-10-26 15:59:26] ...optimizers/trijoint.py.151: total_norm: 3.782124 [I 2022-10-26 15:59:31] ...trap/engines/engine.py.224: train: epoch 17 | batch 550/2304 [I 2022-10-26 15:59:31] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:14:34 [I 2022-10-26 15:59:31] ...trap/engines/engine.py.229: process: 0.49123 | load: 0.00246 [I 2022-10-26 15:59:31] ...trap/engines/engine.py.230: loss: 0.19895 [I 2022-10-26 15:59:31] ...optimizers/trijoint.py.151: total_norm: 2.717378 [I 2022-10-26 15:59:36] ...trap/engines/engine.py.224: train: epoch 17 | batch 560/2304 [I 2022-10-26 15:59:36] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:14:28 [I 2022-10-26 15:59:36] ...trap/engines/engine.py.229: process: 0.49802 | load: 0.00246 [I 2022-10-26 15:59:36] ...trap/engines/engine.py.230: loss: 0.23201 [I 2022-10-26 15:59:36] ...optimizers/trijoint.py.151: total_norm: 3.965390 [I 2022-10-26 15:59:41] ...trap/engines/engine.py.224: train: epoch 17 | batch 570/2304 [I 2022-10-26 15:59:41] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:14:22 [I 2022-10-26 15:59:41] ...trap/engines/engine.py.229: process: 0.49398 | load: 0.00244 [I 2022-10-26 15:59:41] ...trap/engines/engine.py.230: loss: 0.14644 [I 2022-10-26 15:59:41] ...optimizers/trijoint.py.151: total_norm: 2.687974 [I 2022-10-26 15:59:46] ...trap/engines/engine.py.224: train: epoch 17 | batch 580/2304 [I 2022-10-26 15:59:46] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:14:28 [I 2022-10-26 15:59:46] ...trap/engines/engine.py.229: process: 0.52370 | load: 0.00247 [I 2022-10-26 15:59:46] ...trap/engines/engine.py.230: loss: 0.25453 [I 2022-10-26 15:59:46] ...optimizers/trijoint.py.151: total_norm: 3.744097 [I 2022-10-26 15:59:51] ...trap/engines/engine.py.224: train: epoch 17 | batch 590/2304 [I 2022-10-26 15:59:51] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:14:15 [I 2022-10-26 15:59:51] ...trap/engines/engine.py.229: process: 0.49093 | load: 0.00260 [I 2022-10-26 15:59:51] ...trap/engines/engine.py.230: loss: 0.24180 [I 2022-10-26 15:59:51] ...optimizers/trijoint.py.151: total_norm: 2.778032 [I 2022-10-26 15:59:56] ...trap/engines/engine.py.224: train: epoch 17 | batch 600/2304 [I 2022-10-26 15:59:56] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:14:06 [I 2022-10-26 15:59:56] ...trap/engines/engine.py.229: process: 0.49845 | load: 0.00246 [I 2022-10-26 15:59:56] ...trap/engines/engine.py.230: loss: 0.25251 [I 2022-10-26 15:59:56] ...optimizers/trijoint.py.151: total_norm: 3.768925 [I 2022-10-26 16:00:01] ...trap/engines/engine.py.224: train: epoch 17 | batch 610/2304 [I 2022-10-26 16:00:01] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:14:05 [I 2022-10-26 16:00:01] ...trap/engines/engine.py.229: process: 0.50277 | load: 0.00246 [I 2022-10-26 16:00:01] ...trap/engines/engine.py.230: loss: 0.18174 [I 2022-10-26 16:00:01] ...optimizers/trijoint.py.151: total_norm: 2.814657 [I 2022-10-26 16:00:06] ...trap/engines/engine.py.224: train: epoch 17 | batch 620/2304 [I 2022-10-26 16:00:06] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:13:57 [I 2022-10-26 16:00:06] ...trap/engines/engine.py.229: process: 0.49863 | load: 0.00244 [I 2022-10-26 16:00:06] ...trap/engines/engine.py.230: loss: 0.20257 [I 2022-10-26 16:00:06] ...optimizers/trijoint.py.151: total_norm: 4.024572 [I 2022-10-26 16:00:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 630/2304 [I 2022-10-26 16:00:12] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:14:06 [I 2022-10-26 16:00:12] ...trap/engines/engine.py.229: process: 0.50212 | load: 0.00243 [I 2022-10-26 16:00:12] ...trap/engines/engine.py.230: loss: 0.26276 [I 2022-10-26 16:00:12] ...optimizers/trijoint.py.151: total_norm: 3.313118 [I 2022-10-26 16:00:17] ...trap/engines/engine.py.224: train: epoch 17 | batch 640/2304 [I 2022-10-26 16:00:17] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:13:53 [I 2022-10-26 16:00:17] ...trap/engines/engine.py.229: process: 0.49686 | load: 0.00241 [I 2022-10-26 16:00:17] ...trap/engines/engine.py.230: loss: 0.28958 [I 2022-10-26 16:00:17] ...optimizers/trijoint.py.151: total_norm: 3.257226 [I 2022-10-26 16:00:22] ...trap/engines/engine.py.224: train: epoch 17 | batch 650/2304 [I 2022-10-26 16:00:22] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:13:45 [I 2022-10-26 16:00:22] ...trap/engines/engine.py.229: process: 0.49335 | load: 0.00254 [I 2022-10-26 16:00:22] ...trap/engines/engine.py.230: loss: 0.25125 [I 2022-10-26 16:00:22] ...optimizers/trijoint.py.151: total_norm: 2.933677 [I 2022-10-26 16:00:27] ...trap/engines/engine.py.224: train: epoch 17 | batch 660/2304 [I 2022-10-26 16:00:27] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:13:43 [I 2022-10-26 16:00:27] ...trap/engines/engine.py.229: process: 0.50615 | load: 0.00251 [I 2022-10-26 16:00:27] ...trap/engines/engine.py.230: loss: 0.16364 [I 2022-10-26 16:00:27] ...optimizers/trijoint.py.151: total_norm: 2.024332 [I 2022-10-26 16:00:32] ...trap/engines/engine.py.224: train: epoch 17 | batch 670/2304 [I 2022-10-26 16:00:32] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:13:39 [I 2022-10-26 16:00:32] ...trap/engines/engine.py.229: process: 0.50258 | load: 0.00246 [I 2022-10-26 16:00:32] ...trap/engines/engine.py.230: loss: 0.28154 [I 2022-10-26 16:00:32] ...optimizers/trijoint.py.151: total_norm: 4.505479 [I 2022-10-26 16:00:37] ...trap/engines/engine.py.224: train: epoch 17 | batch 680/2304 [I 2022-10-26 16:00:37] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:13:29 [I 2022-10-26 16:00:37] ...trap/engines/engine.py.229: process: 0.49524 | load: 0.00246 [I 2022-10-26 16:00:37] ...trap/engines/engine.py.230: loss: 0.22136 [I 2022-10-26 16:00:37] ...optimizers/trijoint.py.151: total_norm: 2.746926 [I 2022-10-26 16:00:42] ...trap/engines/engine.py.224: train: epoch 17 | batch 690/2304 [I 2022-10-26 16:00:42] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:13:22 [I 2022-10-26 16:00:42] ...trap/engines/engine.py.229: process: 0.49619 | load: 0.00245 [I 2022-10-26 16:00:42] ...trap/engines/engine.py.230: loss: 0.29545 [I 2022-10-26 16:00:42] ...optimizers/trijoint.py.151: total_norm: 3.627144 [I 2022-10-26 16:00:47] ...trap/engines/engine.py.224: train: epoch 17 | batch 700/2304 [I 2022-10-26 16:00:47] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:13:29 [I 2022-10-26 16:00:47] ...trap/engines/engine.py.229: process: 0.50241 | load: 0.00265 [I 2022-10-26 16:00:47] ...trap/engines/engine.py.230: loss: 0.16929 [I 2022-10-26 16:00:47] ...optimizers/trijoint.py.151: total_norm: 3.512729 [I 2022-10-26 16:00:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 710/2304 [I 2022-10-26 16:00:52] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:13:07 [I 2022-10-26 16:00:52] ...trap/engines/engine.py.229: process: 0.48610 | load: 0.00251 [I 2022-10-26 16:00:52] ...trap/engines/engine.py.230: loss: 0.20382 [I 2022-10-26 16:00:52] ...optimizers/trijoint.py.151: total_norm: 4.810426 [I 2022-10-26 16:00:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 720/2304 [I 2022-10-26 16:00:57] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:13:10 [I 2022-10-26 16:00:57] ...trap/engines/engine.py.229: process: 0.49917 | load: 0.00251 [I 2022-10-26 16:00:57] ...trap/engines/engine.py.230: loss: 0.25426 [I 2022-10-26 16:00:57] ...optimizers/trijoint.py.151: total_norm: 4.462232 [I 2022-10-26 16:01:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 730/2304 [I 2022-10-26 16:01:02] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:13:06 [I 2022-10-26 16:01:02] ...trap/engines/engine.py.229: process: 0.51109 | load: 0.00248 [I 2022-10-26 16:01:02] ...trap/engines/engine.py.230: loss: 0.17759 [I 2022-10-26 16:01:02] ...optimizers/trijoint.py.151: total_norm: 3.190959 [I 2022-10-26 16:01:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 740/2304 [I 2022-10-26 16:01:07] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:12:59 [I 2022-10-26 16:01:07] ...trap/engines/engine.py.229: process: 0.50471 | load: 0.00246 [I 2022-10-26 16:01:07] ...trap/engines/engine.py.230: loss: 0.24541 [I 2022-10-26 16:01:07] ...optimizers/trijoint.py.151: total_norm: 3.704233 [I 2022-10-26 16:01:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 750/2304 [I 2022-10-26 16:01:12] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:12:55 [I 2022-10-26 16:01:12] ...trap/engines/engine.py.229: process: 0.49205 | load: 0.00242 [I 2022-10-26 16:01:12] ...trap/engines/engine.py.230: loss: 0.25669 [I 2022-10-26 16:01:12] ...optimizers/trijoint.py.151: total_norm: 3.409548 [I 2022-10-26 16:01:17] ...trap/engines/engine.py.224: train: epoch 17 | batch 760/2304 [I 2022-10-26 16:01:17] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:12:56 [I 2022-10-26 16:01:17] ...trap/engines/engine.py.229: process: 0.49976 | load: 0.00246 [I 2022-10-26 16:01:17] ...trap/engines/engine.py.230: loss: 0.26181 [I 2022-10-26 16:01:17] ...optimizers/trijoint.py.151: total_norm: 3.616639 [I 2022-10-26 16:01:22] ...trap/engines/engine.py.224: train: epoch 17 | batch 770/2304 [I 2022-10-26 16:01:22] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:12:47 [I 2022-10-26 16:01:22] ...trap/engines/engine.py.229: process: 0.50170 | load: 0.00261 [I 2022-10-26 16:01:22] ...trap/engines/engine.py.230: loss: 0.20684 [I 2022-10-26 16:01:22] ...optimizers/trijoint.py.151: total_norm: 2.720386 [I 2022-10-26 16:01:27] ...trap/engines/engine.py.224: train: epoch 17 | batch 780/2304 [I 2022-10-26 16:01:27] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:12:41 [I 2022-10-26 16:01:27] ...trap/engines/engine.py.229: process: 0.49266 | load: 0.00254 [I 2022-10-26 16:01:27] ...trap/engines/engine.py.230: loss: 0.20681 [I 2022-10-26 16:01:27] ...optimizers/trijoint.py.151: total_norm: 3.133386 [I 2022-10-26 16:01:32] ...trap/engines/engine.py.224: train: epoch 17 | batch 790/2304 [I 2022-10-26 16:01:32] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:12:40 [I 2022-10-26 16:01:32] ...trap/engines/engine.py.229: process: 0.50418 | load: 0.00248 [I 2022-10-26 16:01:32] ...trap/engines/engine.py.230: loss: 0.10830 [I 2022-10-26 16:01:32] ...optimizers/trijoint.py.151: total_norm: 2.033939 [I 2022-10-26 16:01:37] ...trap/engines/engine.py.224: train: epoch 17 | batch 800/2304 [I 2022-10-26 16:01:37] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:12:44 [I 2022-10-26 16:01:37] ...trap/engines/engine.py.229: process: 0.52512 | load: 0.00252 [I 2022-10-26 16:01:37] ...trap/engines/engine.py.230: loss: 0.32091 [I 2022-10-26 16:01:37] ...optimizers/trijoint.py.151: total_norm: 3.982290 [I 2022-10-26 16:01:42] ...trap/engines/engine.py.224: train: epoch 17 | batch 810/2304 [I 2022-10-26 16:01:42] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:12:36 [I 2022-10-26 16:01:42] ...trap/engines/engine.py.229: process: 0.50721 | load: 0.00247 [I 2022-10-26 16:01:42] ...trap/engines/engine.py.230: loss: 0.21994 [I 2022-10-26 16:01:42] ...optimizers/trijoint.py.151: total_norm: 3.905591 [I 2022-10-26 16:01:47] ...trap/engines/engine.py.224: train: epoch 17 | batch 820/2304 [I 2022-10-26 16:01:47] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:12:18 [I 2022-10-26 16:01:47] ...trap/engines/engine.py.229: process: 0.50099 | load: 0.00246 [I 2022-10-26 16:01:47] ...trap/engines/engine.py.230: loss: 0.24759 [I 2022-10-26 16:01:47] ...optimizers/trijoint.py.151: total_norm: 3.633507 [I 2022-10-26 16:01:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 830/2304 [I 2022-10-26 16:01:52] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:12:18 [I 2022-10-26 16:01:52] ...trap/engines/engine.py.229: process: 0.50129 | load: 0.00245 [I 2022-10-26 16:01:52] ...trap/engines/engine.py.230: loss: 0.15370 [I 2022-10-26 16:01:52] ...optimizers/trijoint.py.151: total_norm: 2.433504 [I 2022-10-26 16:01:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 840/2304 [I 2022-10-26 16:01:57] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:12:10 [I 2022-10-26 16:01:57] ...trap/engines/engine.py.229: process: 0.51181 | load: 0.00249 [I 2022-10-26 16:01:57] ...trap/engines/engine.py.230: loss: 0.21359 [I 2022-10-26 16:01:57] ...optimizers/trijoint.py.151: total_norm: 3.787895 [I 2022-10-26 16:02:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 850/2304 [I 2022-10-26 16:02:02] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:12:04 [I 2022-10-26 16:02:02] ...trap/engines/engine.py.229: process: 0.50346 | load: 0.00242 [I 2022-10-26 16:02:02] ...trap/engines/engine.py.230: loss: 0.20496 [I 2022-10-26 16:02:02] ...optimizers/trijoint.py.151: total_norm: 3.187997 [I 2022-10-26 16:02:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 860/2304 [I 2022-10-26 16:02:07] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:11:51 [I 2022-10-26 16:02:07] ...trap/engines/engine.py.229: process: 0.48679 | load: 0.00245 [I 2022-10-26 16:02:07] ...trap/engines/engine.py.230: loss: 0.23360 [I 2022-10-26 16:02:07] ...optimizers/trijoint.py.151: total_norm: 4.087836 [I 2022-10-26 16:02:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 870/2304 [I 2022-10-26 16:02:12] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:11:55 [I 2022-10-26 16:02:12] ...trap/engines/engine.py.229: process: 0.49493 | load: 0.00243 [I 2022-10-26 16:02:12] ...trap/engines/engine.py.230: loss: 0.22703 [I 2022-10-26 16:02:12] ...optimizers/trijoint.py.151: total_norm: 3.570455 [I 2022-10-26 16:02:17] ...trap/engines/engine.py.224: train: epoch 17 | batch 880/2304 [I 2022-10-26 16:02:17] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:11:52 [I 2022-10-26 16:02:17] ...trap/engines/engine.py.229: process: 0.49345 | load: 0.00250 [I 2022-10-26 16:02:17] ...trap/engines/engine.py.230: loss: 0.24308 [I 2022-10-26 16:02:17] ...optimizers/trijoint.py.151: total_norm: 3.882575 [I 2022-10-26 16:02:22] ...trap/engines/engine.py.224: train: epoch 17 | batch 890/2304 [I 2022-10-26 16:02:22] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:11:45 [I 2022-10-26 16:02:22] ...trap/engines/engine.py.229: process: 0.48906 | load: 0.00245 [I 2022-10-26 16:02:22] ...trap/engines/engine.py.230: loss: 0.17133 [I 2022-10-26 16:02:22] ...optimizers/trijoint.py.151: total_norm: 3.117380 [I 2022-10-26 16:02:27] ...trap/engines/engine.py.224: train: epoch 17 | batch 900/2304 [I 2022-10-26 16:02:27] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:11:42 [I 2022-10-26 16:02:27] ...trap/engines/engine.py.229: process: 0.49900 | load: 0.00245 [I 2022-10-26 16:02:27] ...trap/engines/engine.py.230: loss: 0.15359 [I 2022-10-26 16:02:27] ...optimizers/trijoint.py.151: total_norm: 2.316892 [I 2022-10-26 16:02:32] ...trap/engines/engine.py.224: train: epoch 17 | batch 910/2304 [I 2022-10-26 16:02:32] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:11:36 [I 2022-10-26 16:02:32] ...trap/engines/engine.py.229: process: 0.49310 | load: 0.00246 [I 2022-10-26 16:02:32] ...trap/engines/engine.py.230: loss: 0.18165 [I 2022-10-26 16:02:32] ...optimizers/trijoint.py.151: total_norm: 3.015966 [I 2022-10-26 16:02:37] ...trap/engines/engine.py.224: train: epoch 17 | batch 920/2304 [I 2022-10-26 16:02:37] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:11:30 [I 2022-10-26 16:02:37] ...trap/engines/engine.py.229: process: 0.50223 | load: 0.00250 [I 2022-10-26 16:02:37] ...trap/engines/engine.py.230: loss: 0.23953 [I 2022-10-26 16:02:37] ...optimizers/trijoint.py.151: total_norm: 3.746533 [I 2022-10-26 16:02:42] ...trap/engines/engine.py.224: train: epoch 17 | batch 930/2304 [I 2022-10-26 16:02:42] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:11:21 [I 2022-10-26 16:02:42] ...trap/engines/engine.py.229: process: 0.50048 | load: 0.00248 [I 2022-10-26 16:02:42] ...trap/engines/engine.py.230: loss: 0.26265 [I 2022-10-26 16:02:42] ...optimizers/trijoint.py.151: total_norm: 3.632292 [I 2022-10-26 16:02:47] ...trap/engines/engine.py.224: train: epoch 17 | batch 940/2304 [I 2022-10-26 16:02:47] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:11:21 [I 2022-10-26 16:02:47] ...trap/engines/engine.py.229: process: 0.50400 | load: 0.00247 [I 2022-10-26 16:02:47] ...trap/engines/engine.py.230: loss: 0.18928 [I 2022-10-26 16:02:47] ...optimizers/trijoint.py.151: total_norm: 2.682217 [I 2022-10-26 16:02:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 950/2304 [I 2022-10-26 16:02:52] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:11:17 [I 2022-10-26 16:02:52] ...trap/engines/engine.py.229: process: 0.49133 | load: 0.00244 [I 2022-10-26 16:02:52] ...trap/engines/engine.py.230: loss: 0.25798 [I 2022-10-26 16:02:52] ...optimizers/trijoint.py.151: total_norm: 3.479666 [I 2022-10-26 16:02:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 960/2304 [I 2022-10-26 16:02:57] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:11:11 [I 2022-10-26 16:02:57] ...trap/engines/engine.py.229: process: 0.48762 | load: 0.00242 [I 2022-10-26 16:02:57] ...trap/engines/engine.py.230: loss: 0.14840 [I 2022-10-26 16:02:57] ...optimizers/trijoint.py.151: total_norm: 2.272459 [I 2022-10-26 16:03:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 970/2304 [I 2022-10-26 16:03:02] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:11:03 [I 2022-10-26 16:03:02] ...trap/engines/engine.py.229: process: 0.49877 | load: 0.00247 [I 2022-10-26 16:03:02] ...trap/engines/engine.py.230: loss: 0.15173 [I 2022-10-26 16:03:02] ...optimizers/trijoint.py.151: total_norm: 3.178499 [I 2022-10-26 16:03:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 980/2304 [I 2022-10-26 16:03:07] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:10:57 [I 2022-10-26 16:03:07] ...trap/engines/engine.py.229: process: 0.49280 | load: 0.00248 [I 2022-10-26 16:03:07] ...trap/engines/engine.py.230: loss: 0.19452 [I 2022-10-26 16:03:07] ...optimizers/trijoint.py.151: total_norm: 3.655788 [I 2022-10-26 16:03:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 990/2304 [I 2022-10-26 16:03:12] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:10:57 [I 2022-10-26 16:03:12] ...trap/engines/engine.py.229: process: 0.49765 | load: 0.00248 [I 2022-10-26 16:03:12] ...trap/engines/engine.py.230: loss: 0.21569 [I 2022-10-26 16:03:12] ...optimizers/trijoint.py.151: total_norm: 3.319271 [I 2022-10-26 16:03:17] ...trap/engines/engine.py.224: train: epoch 17 | batch 1000/2304 [I 2022-10-26 16:03:17] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:10:58 [I 2022-10-26 16:03:17] ...trap/engines/engine.py.229: process: 0.50028 | load: 0.00248 [I 2022-10-26 16:03:17] ...trap/engines/engine.py.230: loss: 0.19527 [I 2022-10-26 16:03:17] ...optimizers/trijoint.py.151: total_norm: 4.060163 [I 2022-10-26 16:03:22] ...trap/engines/engine.py.224: train: epoch 17 | batch 1010/2304 [I 2022-10-26 16:03:22] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:10:40 [I 2022-10-26 16:03:22] ...trap/engines/engine.py.229: process: 0.48563 | load: 0.00253 [I 2022-10-26 16:03:22] ...trap/engines/engine.py.230: loss: 0.20242 [I 2022-10-26 16:03:22] ...optimizers/trijoint.py.151: total_norm: 4.361646 [I 2022-10-26 16:03:27] ...trap/engines/engine.py.224: train: epoch 17 | batch 1020/2304 [I 2022-10-26 16:03:27] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:10:38 [I 2022-10-26 16:03:27] ...trap/engines/engine.py.229: process: 0.49462 | load: 0.00244 [I 2022-10-26 16:03:27] ...trap/engines/engine.py.230: loss: 0.25131 [I 2022-10-26 16:03:27] ...optimizers/trijoint.py.151: total_norm: 3.498124 [I 2022-10-26 16:03:32] ...trap/engines/engine.py.224: train: epoch 17 | batch 1030/2304 [I 2022-10-26 16:03:32] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:10:34 [I 2022-10-26 16:03:32] ...trap/engines/engine.py.229: process: 0.50293 | load: 0.00244 [I 2022-10-26 16:03:32] ...trap/engines/engine.py.230: loss: 0.15184 [I 2022-10-26 16:03:32] ...optimizers/trijoint.py.151: total_norm: 2.684117 [I 2022-10-26 16:03:37] ...trap/engines/engine.py.224: train: epoch 17 | batch 1040/2304 [I 2022-10-26 16:03:37] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:10:26 [I 2022-10-26 16:03:37] ...trap/engines/engine.py.229: process: 0.48661 | load: 0.00243 [I 2022-10-26 16:03:37] ...trap/engines/engine.py.230: loss: 0.17218 [I 2022-10-26 16:03:37] ...optimizers/trijoint.py.151: total_norm: 3.837386 [I 2022-10-26 16:03:42] ...trap/engines/engine.py.224: train: epoch 17 | batch 1050/2304 [I 2022-10-26 16:03:42] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:10:26 [I 2022-10-26 16:03:42] ...trap/engines/engine.py.229: process: 0.50071 | load: 0.00243 [I 2022-10-26 16:03:42] ...trap/engines/engine.py.230: loss: 0.33484 [I 2022-10-26 16:03:42] ...optimizers/trijoint.py.151: total_norm: 4.618863 [I 2022-10-26 16:03:47] ...trap/engines/engine.py.224: train: epoch 17 | batch 1060/2304 [I 2022-10-26 16:03:47] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:10:22 [I 2022-10-26 16:03:47] ...trap/engines/engine.py.229: process: 0.50943 | load: 0.00241 [I 2022-10-26 16:03:47] ...trap/engines/engine.py.230: loss: 0.10920 [I 2022-10-26 16:03:47] ...optimizers/trijoint.py.151: total_norm: 1.686552 [I 2022-10-26 16:03:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 1070/2304 [I 2022-10-26 16:03:52] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:10:19 [I 2022-10-26 16:03:52] ...trap/engines/engine.py.229: process: 0.51526 | load: 0.00251 [I 2022-10-26 16:03:52] ...trap/engines/engine.py.230: loss: 0.22674 [I 2022-10-26 16:03:52] ...optimizers/trijoint.py.151: total_norm: 3.782704 [I 2022-10-26 16:03:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 1080/2304 [I 2022-10-26 16:03:57] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:10:15 [I 2022-10-26 16:03:57] ...trap/engines/engine.py.229: process: 0.50693 | load: 0.00249 [I 2022-10-26 16:03:57] ...trap/engines/engine.py.230: loss: 0.23236 [I 2022-10-26 16:03:57] ...optimizers/trijoint.py.151: total_norm: 4.258496 [I 2022-10-26 16:04:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 1090/2304 [I 2022-10-26 16:04:02] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:10:07 [I 2022-10-26 16:04:02] ...trap/engines/engine.py.229: process: 0.50929 | load: 0.00307 [I 2022-10-26 16:04:02] ...trap/engines/engine.py.230: loss: 0.20163 [I 2022-10-26 16:04:02] ...optimizers/trijoint.py.151: total_norm: 3.244618 [I 2022-10-26 16:04:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 1100/2304 [I 2022-10-26 16:04:07] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:09:58 [I 2022-10-26 16:04:07] ...trap/engines/engine.py.229: process: 0.48855 | load: 0.00246 [I 2022-10-26 16:04:07] ...trap/engines/engine.py.230: loss: 0.22780 [I 2022-10-26 16:04:07] ...optimizers/trijoint.py.151: total_norm: 3.637546 [I 2022-10-26 16:04:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 1110/2304 [I 2022-10-26 16:04:12] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:09:58 [I 2022-10-26 16:04:12] ...trap/engines/engine.py.229: process: 0.50307 | load: 0.00247 [I 2022-10-26 16:04:12] ...trap/engines/engine.py.230: loss: 0.25328 [I 2022-10-26 16:04:12] ...optimizers/trijoint.py.151: total_norm: 4.747326 [I 2022-10-26 16:04:17] ...trap/engines/engine.py.224: train: epoch 17 | batch 1120/2304 [I 2022-10-26 16:04:17] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:09:49 [I 2022-10-26 16:04:17] ...trap/engines/engine.py.229: process: 0.50210 | load: 0.00252 [I 2022-10-26 16:04:17] ...trap/engines/engine.py.230: loss: 0.17350 [I 2022-10-26 16:04:17] ...optimizers/trijoint.py.151: total_norm: 2.753095 [I 2022-10-26 16:04:22] ...trap/engines/engine.py.224: train: epoch 17 | batch 1130/2304 [I 2022-10-26 16:04:22] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:09:46 [I 2022-10-26 16:04:22] ...trap/engines/engine.py.229: process: 0.49869 | load: 0.00252 [I 2022-10-26 16:04:22] ...trap/engines/engine.py.230: loss: 0.18546 [I 2022-10-26 16:04:22] ...optimizers/trijoint.py.151: total_norm: 2.297493 [I 2022-10-26 16:04:27] ...trap/engines/engine.py.224: train: epoch 17 | batch 1140/2304 [I 2022-10-26 16:04:27] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:09:36 [I 2022-10-26 16:04:27] ...trap/engines/engine.py.229: process: 0.49173 | load: 0.00244 [I 2022-10-26 16:04:27] ...trap/engines/engine.py.230: loss: 0.16472 [I 2022-10-26 16:04:27] ...optimizers/trijoint.py.151: total_norm: 4.495492 [I 2022-10-26 16:04:32] ...trap/engines/engine.py.224: train: epoch 17 | batch 1150/2304 [I 2022-10-26 16:04:32] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:09:36 [I 2022-10-26 16:04:32] ...trap/engines/engine.py.229: process: 0.49985 | load: 0.00305 [I 2022-10-26 16:04:32] ...trap/engines/engine.py.230: loss: 0.20683 [I 2022-10-26 16:04:32] ...optimizers/trijoint.py.151: total_norm: 2.619233 [I 2022-10-26 16:04:37] ...trap/engines/engine.py.224: train: epoch 17 | batch 1160/2304 [I 2022-10-26 16:04:37] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:09:31 [I 2022-10-26 16:04:37] ...trap/engines/engine.py.229: process: 0.50153 | load: 0.00270 [I 2022-10-26 16:04:37] ...trap/engines/engine.py.230: loss: 0.36927 [I 2022-10-26 16:04:37] ...optimizers/trijoint.py.151: total_norm: 5.008091 [I 2022-10-26 16:04:42] ...trap/engines/engine.py.224: train: epoch 17 | batch 1170/2304 [I 2022-10-26 16:04:42] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:09:27 [I 2022-10-26 16:04:42] ...trap/engines/engine.py.229: process: 0.50808 | load: 0.00247 [I 2022-10-26 16:04:42] ...trap/engines/engine.py.230: loss: 0.24324 [I 2022-10-26 16:04:42] ...optimizers/trijoint.py.151: total_norm: 3.786413 [I 2022-10-26 16:04:47] ...trap/engines/engine.py.224: train: epoch 17 | batch 1180/2304 [I 2022-10-26 16:04:47] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:09:23 [I 2022-10-26 16:04:47] ...trap/engines/engine.py.229: process: 0.50288 | load: 0.00245 [I 2022-10-26 16:04:47] ...trap/engines/engine.py.230: loss: 0.19374 [I 2022-10-26 16:04:47] ...optimizers/trijoint.py.151: total_norm: 3.540221 [I 2022-10-26 16:04:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 1190/2304 [I 2022-10-26 16:04:52] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:09:23 [I 2022-10-26 16:04:52] ...trap/engines/engine.py.229: process: 0.50688 | load: 0.00245 [I 2022-10-26 16:04:52] ...trap/engines/engine.py.230: loss: 0.17209 [I 2022-10-26 16:04:52] ...optimizers/trijoint.py.151: total_norm: 3.372293 [I 2022-10-26 16:04:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 1200/2304 [I 2022-10-26 16:04:57] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:09:11 [I 2022-10-26 16:04:57] ...trap/engines/engine.py.229: process: 0.48799 | load: 0.00247 [I 2022-10-26 16:04:57] ...trap/engines/engine.py.230: loss: 0.23865 [I 2022-10-26 16:04:57] ...optimizers/trijoint.py.151: total_norm: 3.912205 [I 2022-10-26 16:05:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 1210/2304 [I 2022-10-26 16:05:02] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:09:03 [I 2022-10-26 16:05:02] ...trap/engines/engine.py.229: process: 0.48592 | load: 0.00246 [I 2022-10-26 16:05:02] ...trap/engines/engine.py.230: loss: 0.20394 [I 2022-10-26 16:05:02] ...optimizers/trijoint.py.151: total_norm: 2.850099 [I 2022-10-26 16:05:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 1220/2304 [I 2022-10-26 16:05:07] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:09:00 [I 2022-10-26 16:05:07] ...trap/engines/engine.py.229: process: 0.49353 | load: 0.00248 [I 2022-10-26 16:05:07] ...trap/engines/engine.py.230: loss: 0.29469 [I 2022-10-26 16:05:07] ...optimizers/trijoint.py.151: total_norm: 4.450003 [I 2022-10-26 16:05:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 1230/2304 [I 2022-10-26 16:05:12] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:08:59 [I 2022-10-26 16:05:12] ...trap/engines/engine.py.229: process: 0.51305 | load: 0.00240 [I 2022-10-26 16:05:12] ...trap/engines/engine.py.230: loss: 0.20344 [I 2022-10-26 16:05:12] ...optimizers/trijoint.py.151: total_norm: 2.942254 [I 2022-10-26 16:05:17] ...trap/engines/engine.py.224: train: epoch 17 | batch 1240/2304 [I 2022-10-26 16:05:17] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:08:47 [I 2022-10-26 16:05:17] ...trap/engines/engine.py.229: process: 0.50020 | load: 0.00245 [I 2022-10-26 16:05:17] ...trap/engines/engine.py.230: loss: 0.19896 [I 2022-10-26 16:05:17] ...optimizers/trijoint.py.151: total_norm: 3.958261 [I 2022-10-26 16:05:22] ...trap/engines/engine.py.224: train: epoch 17 | batch 1250/2304 [I 2022-10-26 16:05:22] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:08:55 [I 2022-10-26 16:05:22] ...trap/engines/engine.py.229: process: 0.50379 | load: 0.00259 [I 2022-10-26 16:05:22] ...trap/engines/engine.py.230: loss: 0.19289 [I 2022-10-26 16:05:22] ...optimizers/trijoint.py.151: total_norm: 2.453888 [I 2022-10-26 16:05:27] ...trap/engines/engine.py.224: train: epoch 17 | batch 1260/2304 [I 2022-10-26 16:05:27] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:08:44 [I 2022-10-26 16:05:27] ...trap/engines/engine.py.229: process: 0.50194 | load: 0.00252 [I 2022-10-26 16:05:27] ...trap/engines/engine.py.230: loss: 0.23159 [I 2022-10-26 16:05:27] ...optimizers/trijoint.py.151: total_norm: 4.073104 [I 2022-10-26 16:05:32] ...trap/engines/engine.py.224: train: epoch 17 | batch 1270/2304 [I 2022-10-26 16:05:32] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:08:47 [I 2022-10-26 16:05:32] ...trap/engines/engine.py.229: process: 0.50734 | load: 0.00245 [I 2022-10-26 16:05:32] ...trap/engines/engine.py.230: loss: 0.16822 [I 2022-10-26 16:05:32] ...optimizers/trijoint.py.151: total_norm: 2.992004 [I 2022-10-26 16:05:37] ...trap/engines/engine.py.224: train: epoch 17 | batch 1280/2304 [I 2022-10-26 16:05:37] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:08:33 [I 2022-10-26 16:05:37] ...trap/engines/engine.py.229: process: 0.50375 | load: 0.00257 [I 2022-10-26 16:05:37] ...trap/engines/engine.py.230: loss: 0.13322 [I 2022-10-26 16:05:37] ...optimizers/trijoint.py.151: total_norm: 2.365847 [I 2022-10-26 16:05:42] ...trap/engines/engine.py.224: train: epoch 17 | batch 1290/2304 [I 2022-10-26 16:05:42] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:08:22 [I 2022-10-26 16:05:42] ...trap/engines/engine.py.229: process: 0.49314 | load: 0.00245 [I 2022-10-26 16:05:42] ...trap/engines/engine.py.230: loss: 0.17368 [I 2022-10-26 16:05:42] ...optimizers/trijoint.py.151: total_norm: 3.003600 [I 2022-10-26 16:05:47] ...trap/engines/engine.py.224: train: epoch 17 | batch 1300/2304 [I 2022-10-26 16:05:47] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:08:24 [I 2022-10-26 16:05:47] ...trap/engines/engine.py.229: process: 0.50207 | load: 0.00246 [I 2022-10-26 16:05:47] ...trap/engines/engine.py.230: loss: 0.22915 [I 2022-10-26 16:05:47] ...optimizers/trijoint.py.151: total_norm: 4.361561 [I 2022-10-26 16:05:52] ...trap/engines/engine.py.224: train: epoch 17 | batch 1310/2304 [I 2022-10-26 16:05:52] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:08:22 [I 2022-10-26 16:05:52] ...trap/engines/engine.py.229: process: 0.52075 | load: 0.00243 [I 2022-10-26 16:05:52] ...trap/engines/engine.py.230: loss: 0.22500 [I 2022-10-26 16:05:52] ...optimizers/trijoint.py.151: total_norm: 3.806079 [I 2022-10-26 16:05:57] ...trap/engines/engine.py.224: train: epoch 17 | batch 1320/2304 [I 2022-10-26 16:05:57] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:08:10 [I 2022-10-26 16:05:57] ...trap/engines/engine.py.229: process: 0.49854 | load: 0.00251 [I 2022-10-26 16:05:57] ...trap/engines/engine.py.230: loss: 0.25587 [I 2022-10-26 16:05:57] ...optimizers/trijoint.py.151: total_norm: 3.882967 [I 2022-10-26 16:06:02] ...trap/engines/engine.py.224: train: epoch 17 | batch 1330/2304 [I 2022-10-26 16:06:02] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:08:06 [I 2022-10-26 16:06:02] ...trap/engines/engine.py.229: process: 0.49253 | load: 0.00248 [I 2022-10-26 16:06:02] ...trap/engines/engine.py.230: loss: 0.20164 [I 2022-10-26 16:06:02] ...optimizers/trijoint.py.151: total_norm: 2.872103 [I 2022-10-26 16:06:07] ...trap/engines/engine.py.224: train: epoch 17 | batch 1340/2304 [I 2022-10-26 16:06:07] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:08:03 [I 2022-10-26 16:06:07] ...trap/engines/engine.py.229: process: 0.50286 | load: 0.00245 [I 2022-10-26 16:06:07] ...trap/engines/engine.py.230: loss: 0.23267 [I 2022-10-26 16:06:07] ...optimizers/trijoint.py.151: total_norm: 3.664528 [I 2022-10-26 16:06:13] ...trap/engines/engine.py.224: train: epoch 17 | batch 1350/2304 [I 2022-10-26 16:06:13] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:08:05 [I 2022-10-26 16:06:13] ...trap/engines/engine.py.229: process: 0.50429 | load: 0.00245 [I 2022-10-26 16:06:13] ...trap/engines/engine.py.230: loss: 0.20165 [I 2022-10-26 16:06:13] ...optimizers/trijoint.py.151: total_norm: 2.986827 [I 2022-10-26 16:06:18] ...trap/engines/engine.py.224: train: epoch 17 | batch 1360/2304 [I 2022-10-26 16:06:18] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:07:54 [I 2022-10-26 16:06:18] ...trap/engines/engine.py.229: process: 0.49498 | load: 0.00243 [I 2022-10-26 16:06:18] ...trap/engines/engine.py.230: loss: 0.14779 [I 2022-10-26 16:06:18] ...optimizers/trijoint.py.151: total_norm: 3.144368 [I 2022-10-26 16:06:23] ...trap/engines/engine.py.224: train: epoch 17 | batch 1370/2304 [I 2022-10-26 16:06:23] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:07:42 [I 2022-10-26 16:06:23] ...trap/engines/engine.py.229: process: 0.47972 | load: 0.00242 [I 2022-10-26 16:06:23] ...trap/engines/engine.py.230: loss: 0.16673 [I 2022-10-26 16:06:23] ...optimizers/trijoint.py.151: total_norm: 2.514955 [I 2022-10-26 16:06:28] ...trap/engines/engine.py.224: train: epoch 17 | batch 1380/2304 [I 2022-10-26 16:06:28] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:07:42 [I 2022-10-26 16:06:28] ...trap/engines/engine.py.229: process: 0.49419 | load: 0.00241 [I 2022-10-26 16:06:28] ...trap/engines/engine.py.230: loss: 0.16222 [I 2022-10-26 16:06:28] ...optimizers/trijoint.py.151: total_norm: 2.043816 [I 2022-10-26 16:06:33] ...trap/engines/engine.py.224: train: epoch 17 | batch 1390/2304 [I 2022-10-26 16:06:33] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:07:34 [I 2022-10-26 16:06:33] ...trap/engines/engine.py.229: process: 0.50149 | load: 0.00265 [I 2022-10-26 16:06:33] ...trap/engines/engine.py.230: loss: 0.26417 [I 2022-10-26 16:06:33] ...optimizers/trijoint.py.151: total_norm: 5.420565 [I 2022-10-26 16:06:38] ...trap/engines/engine.py.224: train: epoch 17 | batch 1400/2304 [I 2022-10-26 16:06:38] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:07:32 [I 2022-10-26 16:06:38] ...trap/engines/engine.py.229: process: 0.49283 | load: 0.00262 [I 2022-10-26 16:06:38] ...trap/engines/engine.py.230: loss: 0.21028 [I 2022-10-26 16:06:38] ...optimizers/trijoint.py.151: total_norm: 3.054325 [I 2022-10-26 16:06:43] ...trap/engines/engine.py.224: train: epoch 17 | batch 1410/2304 [I 2022-10-26 16:06:43] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:07:26 [I 2022-10-26 16:06:43] ...trap/engines/engine.py.229: process: 0.50212 | load: 0.00252 [I 2022-10-26 16:06:43] ...trap/engines/engine.py.230: loss: 0.17745 [I 2022-10-26 16:06:43] ...optimizers/trijoint.py.151: total_norm: 2.944103 [I 2022-10-26 16:06:48] ...trap/engines/engine.py.224: train: epoch 17 | batch 1420/2304 [I 2022-10-26 16:06:48] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:07:29 [I 2022-10-26 16:06:48] ...trap/engines/engine.py.229: process: 0.48456 | load: 0.00250 [I 2022-10-26 16:06:48] ...trap/engines/engine.py.230: loss: 0.23288 [I 2022-10-26 16:06:48] ...optimizers/trijoint.py.151: total_norm: 3.232267 [I 2022-10-26 16:06:53] ...trap/engines/engine.py.224: train: epoch 17 | batch 1430/2304 [I 2022-10-26 16:06:53] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:07:25 [I 2022-10-26 16:06:53] ...trap/engines/engine.py.229: process: 0.52134 | load: 0.00250 [I 2022-10-26 16:06:53] ...trap/engines/engine.py.230: loss: 0.20054 [I 2022-10-26 16:06:53] ...optimizers/trijoint.py.151: total_norm: 2.610116 [I 2022-10-26 16:06:58] ...trap/engines/engine.py.224: train: epoch 17 | batch 1440/2304 [I 2022-10-26 16:06:58] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:07:12 [I 2022-10-26 16:06:58] ...trap/engines/engine.py.229: process: 0.50082 | load: 0.00244 [I 2022-10-26 16:06:58] ...trap/engines/engine.py.230: loss: 0.29642 [I 2022-10-26 16:06:58] ...optimizers/trijoint.py.151: total_norm: 3.798926 [I 2022-10-26 16:07:03] ...trap/engines/engine.py.224: train: epoch 17 | batch 1450/2304 [I 2022-10-26 16:07:03] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:07:05 [I 2022-10-26 16:07:03] ...trap/engines/engine.py.229: process: 0.50805 | load: 0.00243 [I 2022-10-26 16:07:03] ...trap/engines/engine.py.230: loss: 0.23660 [I 2022-10-26 16:07:03] ...optimizers/trijoint.py.151: total_norm: 3.578507 [I 2022-10-26 16:07:08] ...trap/engines/engine.py.224: train: epoch 17 | batch 1460/2304 [I 2022-10-26 16:07:08] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:07:01 [I 2022-10-26 16:07:08] ...trap/engines/engine.py.229: process: 0.49995 | load: 0.00248 [I 2022-10-26 16:07:08] ...trap/engines/engine.py.230: loss: 0.29771 [I 2022-10-26 16:07:08] ...optimizers/trijoint.py.151: total_norm: 2.987895 [I 2022-10-26 16:07:13] ...trap/engines/engine.py.224: train: epoch 17 | batch 1470/2304 [I 2022-10-26 16:07:13] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:06:58 [I 2022-10-26 16:07:13] ...trap/engines/engine.py.229: process: 0.51295 | load: 0.00253 [I 2022-10-26 16:07:13] ...trap/engines/engine.py.230: loss: 0.25714 [I 2022-10-26 16:07:13] ...optimizers/trijoint.py.151: total_norm: 3.852336 [I 2022-10-26 16:07:18] ...trap/engines/engine.py.224: train: epoch 17 | batch 1480/2304 [I 2022-10-26 16:07:18] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:06:52 [I 2022-10-26 16:07:18] ...trap/engines/engine.py.229: process: 0.50354 | load: 0.00244 [I 2022-10-26 16:07:18] ...trap/engines/engine.py.230: loss: 0.26534 [I 2022-10-26 16:07:18] ...optimizers/trijoint.py.151: total_norm: 4.130642 [I 2022-10-26 16:07:23] ...trap/engines/engine.py.224: train: epoch 17 | batch 1490/2304 [I 2022-10-26 16:07:23] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:06:52 [I 2022-10-26 16:07:23] ...trap/engines/engine.py.229: process: 0.51437 | load: 0.00257 [I 2022-10-26 16:07:23] ...trap/engines/engine.py.230: loss: 0.25646 [I 2022-10-26 16:07:23] ...optimizers/trijoint.py.151: total_norm: 2.970636 [I 2022-10-26 16:07:28] ...trap/engines/engine.py.224: train: epoch 17 | batch 1500/2304 [I 2022-10-26 16:07:28] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:06:42 [I 2022-10-26 16:07:28] ...trap/engines/engine.py.229: process: 0.49452 | load: 0.00248 [I 2022-10-26 16:07:28] ...trap/engines/engine.py.230: loss: 0.21526 [I 2022-10-26 16:07:28] ...optimizers/trijoint.py.151: total_norm: 2.886340 [I 2022-10-26 16:07:33] ...trap/engines/engine.py.224: train: epoch 17 | batch 1510/2304 [I 2022-10-26 16:07:33] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:06:40 [I 2022-10-26 16:07:33] ...trap/engines/engine.py.229: process: 0.49394 | load: 0.00244 [I 2022-10-26 16:07:33] ...trap/engines/engine.py.230: loss: 0.18276 [I 2022-10-26 16:07:33] ...optimizers/trijoint.py.151: total_norm: 3.560707 [I 2022-10-26 16:07:38] ...trap/engines/engine.py.224: train: epoch 17 | batch 1520/2304 [I 2022-10-26 16:07:38] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:06:35 [I 2022-10-26 16:07:38] ...trap/engines/engine.py.229: process: 0.49999 | load: 0.00250 [I 2022-10-26 16:07:38] ...trap/engines/engine.py.230: loss: 0.23367 [I 2022-10-26 16:07:38] ...optimizers/trijoint.py.151: total_norm: 4.464883 [I 2022-10-26 16:07:43] ...trap/engines/engine.py.224: train: epoch 17 | batch 1530/2304 [I 2022-10-26 16:07:43] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:06:31 [I 2022-10-26 16:07:43] ...trap/engines/engine.py.229: process: 0.49932 | load: 0.00252 [I 2022-10-26 16:07:43] ...trap/engines/engine.py.230: loss: 0.25877 [I 2022-10-26 16:07:43] ...optimizers/trijoint.py.151: total_norm: 4.032400 [I 2022-10-26 16:07:48] ...trap/engines/engine.py.224: train: epoch 17 | batch 1540/2304 [I 2022-10-26 16:07:48] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:06:24 [I 2022-10-26 16:07:48] ...trap/engines/engine.py.229: process: 0.49595 | load: 0.00243 [I 2022-10-26 16:07:48] ...trap/engines/engine.py.230: loss: 0.27714 [I 2022-10-26 16:07:48] ...optimizers/trijoint.py.151: total_norm: 3.647208 [I 2022-10-26 16:07:53] ...trap/engines/engine.py.224: train: epoch 17 | batch 1550/2304 [I 2022-10-26 16:07:53] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:06:30 [I 2022-10-26 16:07:53] ...trap/engines/engine.py.229: process: 0.49406 | load: 0.00243 [I 2022-10-26 16:07:53] ...trap/engines/engine.py.230: loss: 0.18142 [I 2022-10-26 16:07:53] ...optimizers/trijoint.py.151: total_norm: 3.321306 [I 2022-10-26 16:07:58] ...trap/engines/engine.py.224: train: epoch 17 | batch 1560/2304 [I 2022-10-26 16:07:58] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:06:14 [I 2022-10-26 16:07:58] ...trap/engines/engine.py.229: process: 0.50137 | load: 0.00248 [I 2022-10-26 16:07:58] ...trap/engines/engine.py.230: loss: 0.13299 [I 2022-10-26 16:07:58] ...optimizers/trijoint.py.151: total_norm: 2.115539 [I 2022-10-26 16:08:03] ...trap/engines/engine.py.224: train: epoch 17 | batch 1570/2304 [I 2022-10-26 16:08:03] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:06:06 [I 2022-10-26 16:08:03] ...trap/engines/engine.py.229: process: 0.49612 | load: 0.00244 [I 2022-10-26 16:08:03] ...trap/engines/engine.py.230: loss: 0.18363 [I 2022-10-26 16:08:03] ...optimizers/trijoint.py.151: total_norm: 2.910826 [I 2022-10-26 16:08:08] ...trap/engines/engine.py.224: train: epoch 17 | batch 1580/2304 [I 2022-10-26 16:08:08] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:06:00 [I 2022-10-26 16:08:08] ...trap/engines/engine.py.229: process: 0.49408 | load: 0.00252 [I 2022-10-26 16:08:08] ...trap/engines/engine.py.230: loss: 0.20916 [I 2022-10-26 16:08:08] ...optimizers/trijoint.py.151: total_norm: 4.418550 [I 2022-10-26 16:08:13] ...trap/engines/engine.py.224: train: epoch 17 | batch 1590/2304 [I 2022-10-26 16:08:13] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:05:56 [I 2022-10-26 16:08:13] ...trap/engines/engine.py.229: process: 0.50817 | load: 0.00252 [I 2022-10-26 16:08:13] ...trap/engines/engine.py.230: loss: 0.17664 [I 2022-10-26 16:08:13] ...optimizers/trijoint.py.151: total_norm: 2.490123 [I 2022-10-26 16:08:18] ...trap/engines/engine.py.224: train: epoch 17 | batch 1600/2304 [I 2022-10-26 16:08:18] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:05:51 [I 2022-10-26 16:08:18] ...trap/engines/engine.py.229: process: 0.50062 | load: 0.00255 [I 2022-10-26 16:08:18] ...trap/engines/engine.py.230: loss: 0.21207 [I 2022-10-26 16:08:18] ...optimizers/trijoint.py.151: total_norm: 2.713245 [I 2022-10-26 16:08:23] ...trap/engines/engine.py.224: train: epoch 17 | batch 1610/2304 [I 2022-10-26 16:08:23] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:05:44 [I 2022-10-26 16:08:23] ...trap/engines/engine.py.229: process: 0.49289 | load: 0.00258 [I 2022-10-26 16:08:23] ...trap/engines/engine.py.230: loss: 0.28959 [I 2022-10-26 16:08:23] ...optimizers/trijoint.py.151: total_norm: 3.274987 [I 2022-10-26 16:08:28] ...trap/engines/engine.py.224: train: epoch 17 | batch 1620/2304 [I 2022-10-26 16:08:28] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:05:44 [I 2022-10-26 16:08:28] ...trap/engines/engine.py.229: process: 0.50296 | load: 0.00264 [I 2022-10-26 16:08:28] ...trap/engines/engine.py.230: loss: 0.20415 [I 2022-10-26 16:08:28] ...optimizers/trijoint.py.151: total_norm: 3.108649 [I 2022-10-26 16:08:33] ...trap/engines/engine.py.224: train: epoch 17 | batch 1630/2304 [I 2022-10-26 16:08:33] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:05:36 [I 2022-10-26 16:08:33] ...trap/engines/engine.py.229: process: 0.50167 | load: 0.00249 [I 2022-10-26 16:08:33] ...trap/engines/engine.py.230: loss: 0.21166 [I 2022-10-26 16:08:33] ...optimizers/trijoint.py.151: total_norm: 3.559103 [I 2022-10-26 16:08:38] ...trap/engines/engine.py.224: train: epoch 17 | batch 1640/2304 [I 2022-10-26 16:08:38] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:05:31 [I 2022-10-26 16:08:38] ...trap/engines/engine.py.229: process: 0.50482 | load: 0.00251 [I 2022-10-26 16:08:38] ...trap/engines/engine.py.230: loss: 0.25296 [I 2022-10-26 16:08:38] ...optimizers/trijoint.py.151: total_norm: 3.722964 [I 2022-10-26 16:08:43] ...trap/engines/engine.py.224: train: epoch 17 | batch 1650/2304 [I 2022-10-26 16:08:43] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:05:25 [I 2022-10-26 16:08:43] ...trap/engines/engine.py.229: process: 0.50403 | load: 0.00244 [I 2022-10-26 16:08:43] ...trap/engines/engine.py.230: loss: 0.25914 [I 2022-10-26 16:08:43] ...optimizers/trijoint.py.151: total_norm: 3.482138 [I 2022-10-26 16:08:48] ...trap/engines/engine.py.224: train: epoch 17 | batch 1660/2304 [I 2022-10-26 16:08:48] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:05:21 [I 2022-10-26 16:08:48] ...trap/engines/engine.py.229: process: 0.50777 | load: 0.00249 [I 2022-10-26 16:08:48] ...trap/engines/engine.py.230: loss: 0.18264 [I 2022-10-26 16:08:48] ...optimizers/trijoint.py.151: total_norm: 3.606469 [I 2022-10-26 16:08:54] ...trap/engines/engine.py.224: train: epoch 17 | batch 1670/2304 [I 2022-10-26 16:08:54] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:05:29 [I 2022-10-26 16:08:54] ...trap/engines/engine.py.229: process: 0.59629 | load: 0.00263 [I 2022-10-26 16:08:54] ...trap/engines/engine.py.230: loss: 0.30000 [I 2022-10-26 16:08:54] ...optimizers/trijoint.py.151: total_norm: 3.335650 [I 2022-10-26 16:08:59] ...trap/engines/engine.py.224: train: epoch 17 | batch 1680/2304 [I 2022-10-26 16:08:59] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:05:15 [I 2022-10-26 16:08:59] ...trap/engines/engine.py.229: process: 0.51113 | load: 0.00249 [I 2022-10-26 16:08:59] ...trap/engines/engine.py.230: loss: 0.24014 [I 2022-10-26 16:08:59] ...optimizers/trijoint.py.151: total_norm: 3.057550 [I 2022-10-26 16:09:04] ...trap/engines/engine.py.224: train: epoch 17 | batch 1690/2304 [I 2022-10-26 16:09:04] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:05:09 [I 2022-10-26 16:09:04] ...trap/engines/engine.py.229: process: 0.49406 | load: 0.00291 [I 2022-10-26 16:09:04] ...trap/engines/engine.py.230: loss: 0.25739 [I 2022-10-26 16:09:04] ...optimizers/trijoint.py.151: total_norm: 3.445437 [I 2022-10-26 16:09:09] ...trap/engines/engine.py.224: train: epoch 17 | batch 1700/2304 [I 2022-10-26 16:09:09] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:05:04 [I 2022-10-26 16:09:09] ...trap/engines/engine.py.229: process: 0.50066 | load: 0.00244 [I 2022-10-26 16:09:09] ...trap/engines/engine.py.230: loss: 0.34750 [I 2022-10-26 16:09:09] ...optimizers/trijoint.py.151: total_norm: 4.854546 [I 2022-10-26 16:09:14] ...trap/engines/engine.py.224: train: epoch 17 | batch 1710/2304 [I 2022-10-26 16:09:14] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:04:58 [I 2022-10-26 16:09:14] ...trap/engines/engine.py.229: process: 0.49869 | load: 0.00247 [I 2022-10-26 16:09:14] ...trap/engines/engine.py.230: loss: 0.14843 [I 2022-10-26 16:09:14] ...optimizers/trijoint.py.151: total_norm: 2.459400 [I 2022-10-26 16:09:19] ...trap/engines/engine.py.224: train: epoch 17 | batch 1720/2304 [I 2022-10-26 16:09:19] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:04:49 [I 2022-10-26 16:09:19] ...trap/engines/engine.py.229: process: 0.49152 | load: 0.00245 [I 2022-10-26 16:09:19] ...trap/engines/engine.py.230: loss: 0.25310 [I 2022-10-26 16:09:19] ...optimizers/trijoint.py.151: total_norm: 3.751558 [I 2022-10-26 16:09:24] ...trap/engines/engine.py.224: train: epoch 17 | batch 1730/2304 [I 2022-10-26 16:09:24] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:04:48 [I 2022-10-26 16:09:24] ...trap/engines/engine.py.229: process: 0.50400 | load: 0.00245 [I 2022-10-26 16:09:24] ...trap/engines/engine.py.230: loss: 0.21551 [I 2022-10-26 16:09:24] ...optimizers/trijoint.py.151: total_norm: 3.165255 [I 2022-10-26 16:09:29] ...trap/engines/engine.py.224: train: epoch 17 | batch 1740/2304 [I 2022-10-26 16:09:29] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:04:40 [I 2022-10-26 16:09:29] ...trap/engines/engine.py.229: process: 0.49572 | load: 0.00245 [I 2022-10-26 16:09:29] ...trap/engines/engine.py.230: loss: 0.20810 [I 2022-10-26 16:09:29] ...optimizers/trijoint.py.151: total_norm: 2.824475 [I 2022-10-26 16:09:34] ...trap/engines/engine.py.224: train: epoch 17 | batch 1750/2304 [I 2022-10-26 16:09:34] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:04:38 [I 2022-10-26 16:09:34] ...trap/engines/engine.py.229: process: 0.50598 | load: 0.00244 [I 2022-10-26 16:09:34] ...trap/engines/engine.py.230: loss: 0.24896 [I 2022-10-26 16:09:34] ...optimizers/trijoint.py.151: total_norm: 3.950473 [I 2022-10-26 16:09:39] ...trap/engines/engine.py.224: train: epoch 17 | batch 1760/2304 [I 2022-10-26 16:09:39] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:04:32 [I 2022-10-26 16:09:39] ...trap/engines/engine.py.229: process: 0.50138 | load: 0.00242 [I 2022-10-26 16:09:39] ...trap/engines/engine.py.230: loss: 0.28232 [I 2022-10-26 16:09:39] ...optimizers/trijoint.py.151: total_norm: 3.666654 [I 2022-10-26 16:09:44] ...trap/engines/engine.py.224: train: epoch 17 | batch 1770/2304 [I 2022-10-26 16:09:44] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:04:28 [I 2022-10-26 16:09:44] ...trap/engines/engine.py.229: process: 0.49267 | load: 0.00252 [I 2022-10-26 16:09:44] ...trap/engines/engine.py.230: loss: 0.16782 [I 2022-10-26 16:09:44] ...optimizers/trijoint.py.151: total_norm: 2.844699 [I 2022-10-26 16:09:49] ...trap/engines/engine.py.224: train: epoch 17 | batch 1780/2304 [I 2022-10-26 16:09:49] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:04:22 [I 2022-10-26 16:09:49] ...trap/engines/engine.py.229: process: 0.50098 | load: 0.00248 [I 2022-10-26 16:09:49] ...trap/engines/engine.py.230: loss: 0.19621 [I 2022-10-26 16:09:49] ...optimizers/trijoint.py.151: total_norm: 2.913397 [I 2022-10-26 16:09:54] ...trap/engines/engine.py.224: train: epoch 17 | batch 1790/2304 [I 2022-10-26 16:09:54] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:04:17 [I 2022-10-26 16:09:54] ...trap/engines/engine.py.229: process: 0.49691 | load: 0.00246 [I 2022-10-26 16:09:54] ...trap/engines/engine.py.230: loss: 0.23517 [I 2022-10-26 16:09:54] ...optimizers/trijoint.py.151: total_norm: 3.331584 [I 2022-10-26 16:09:59] ...trap/engines/engine.py.224: train: epoch 17 | batch 1800/2304 [I 2022-10-26 16:09:59] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:04:15 [I 2022-10-26 16:09:59] ...trap/engines/engine.py.229: process: 0.50167 | load: 0.00247 [I 2022-10-26 16:09:59] ...trap/engines/engine.py.230: loss: 0.21460 [I 2022-10-26 16:09:59] ...optimizers/trijoint.py.151: total_norm: 3.396042 [I 2022-10-26 16:10:04] ...trap/engines/engine.py.224: train: epoch 17 | batch 1810/2304 [I 2022-10-26 16:10:04] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:04:08 [I 2022-10-26 16:10:04] ...trap/engines/engine.py.229: process: 0.50339 | load: 0.00243 [I 2022-10-26 16:10:04] ...trap/engines/engine.py.230: loss: 0.30625 [I 2022-10-26 16:10:04] ...optimizers/trijoint.py.151: total_norm: 4.065442 [I 2022-10-26 16:10:09] ...trap/engines/engine.py.224: train: epoch 17 | batch 1820/2304 [I 2022-10-26 16:10:09] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:04:06 [I 2022-10-26 16:10:09] ...trap/engines/engine.py.229: process: 0.51281 | load: 0.00246 [I 2022-10-26 16:10:09] ...trap/engines/engine.py.230: loss: 0.18570 [I 2022-10-26 16:10:09] ...optimizers/trijoint.py.151: total_norm: 2.799366 [I 2022-10-26 16:10:14] ...trap/engines/engine.py.224: train: epoch 17 | batch 1830/2304 [I 2022-10-26 16:10:14] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:03:57 [I 2022-10-26 16:10:14] ...trap/engines/engine.py.229: process: 0.49571 | load: 0.00245 [I 2022-10-26 16:10:14] ...trap/engines/engine.py.230: loss: 0.21537 [I 2022-10-26 16:10:14] ...optimizers/trijoint.py.151: total_norm: 3.930667 [I 2022-10-26 16:10:19] ...trap/engines/engine.py.224: train: epoch 17 | batch 1840/2304 [I 2022-10-26 16:10:19] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:03:53 [I 2022-10-26 16:10:19] ...trap/engines/engine.py.229: process: 0.49488 | load: 0.00244 [I 2022-10-26 16:10:19] ...trap/engines/engine.py.230: loss: 0.14560 [I 2022-10-26 16:10:19] ...optimizers/trijoint.py.151: total_norm: 2.600266 [I 2022-10-26 16:10:24] ...trap/engines/engine.py.224: train: epoch 17 | batch 1850/2304 [I 2022-10-26 16:10:24] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:03:48 [I 2022-10-26 16:10:24] ...trap/engines/engine.py.229: process: 0.50981 | load: 0.01039 [I 2022-10-26 16:10:24] ...trap/engines/engine.py.230: loss: 0.15586 [I 2022-10-26 16:10:24] ...optimizers/trijoint.py.151: total_norm: 2.046293 [I 2022-10-26 16:10:29] ...trap/engines/engine.py.224: train: epoch 17 | batch 1860/2304 [I 2022-10-26 16:10:29] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:03:42 [I 2022-10-26 16:10:29] ...trap/engines/engine.py.229: process: 0.49570 | load: 0.00243 [I 2022-10-26 16:10:29] ...trap/engines/engine.py.230: loss: 0.28179 [I 2022-10-26 16:10:29] ...optimizers/trijoint.py.151: total_norm: 3.535916 [I 2022-10-26 16:10:34] ...trap/engines/engine.py.224: train: epoch 17 | batch 1870/2304 [I 2022-10-26 16:10:34] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:03:39 [I 2022-10-26 16:10:34] ...trap/engines/engine.py.229: process: 0.49752 | load: 0.00245 [I 2022-10-26 16:10:34] ...trap/engines/engine.py.230: loss: 0.18528 [I 2022-10-26 16:10:34] ...optimizers/trijoint.py.151: total_norm: 3.319349 [I 2022-10-26 16:10:39] ...trap/engines/engine.py.224: train: epoch 17 | batch 1880/2304 [I 2022-10-26 16:10:39] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:03:30 [I 2022-10-26 16:10:39] ...trap/engines/engine.py.229: process: 0.49956 | load: 0.00245 [I 2022-10-26 16:10:39] ...trap/engines/engine.py.230: loss: 0.19687 [I 2022-10-26 16:10:39] ...optimizers/trijoint.py.151: total_norm: 2.798238 [I 2022-10-26 16:10:44] ...trap/engines/engine.py.224: train: epoch 17 | batch 1890/2304 [I 2022-10-26 16:10:44] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:03:27 [I 2022-10-26 16:10:44] ...trap/engines/engine.py.229: process: 0.50363 | load: 0.00253 [I 2022-10-26 16:10:44] ...trap/engines/engine.py.230: loss: 0.22551 [I 2022-10-26 16:10:44] ...optimizers/trijoint.py.151: total_norm: 3.569115 [I 2022-10-26 16:10:49] ...trap/engines/engine.py.224: train: epoch 17 | batch 1900/2304 [I 2022-10-26 16:10:49] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:03:22 [I 2022-10-26 16:10:49] ...trap/engines/engine.py.229: process: 0.50015 | load: 0.00252 [I 2022-10-26 16:10:49] ...trap/engines/engine.py.230: loss: 0.23166 [I 2022-10-26 16:10:49] ...optimizers/trijoint.py.151: total_norm: 3.331383 [I 2022-10-26 16:10:54] ...trap/engines/engine.py.224: train: epoch 17 | batch 1910/2304 [I 2022-10-26 16:10:54] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:03:18 [I 2022-10-26 16:10:54] ...trap/engines/engine.py.229: process: 0.50322 | load: 0.00249 [I 2022-10-26 16:10:54] ...trap/engines/engine.py.230: loss: 0.25755 [I 2022-10-26 16:10:54] ...optimizers/trijoint.py.151: total_norm: 3.529283 [I 2022-10-26 16:10:59] ...trap/engines/engine.py.224: train: epoch 17 | batch 1920/2304 [I 2022-10-26 16:10:59] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:03:11 [I 2022-10-26 16:10:59] ...trap/engines/engine.py.229: process: 0.49588 | load: 0.00245 [I 2022-10-26 16:10:59] ...trap/engines/engine.py.230: loss: 0.23755 [I 2022-10-26 16:10:59] ...optimizers/trijoint.py.151: total_norm: 3.784813 [I 2022-10-26 16:11:04] ...trap/engines/engine.py.224: train: epoch 17 | batch 1930/2304 [I 2022-10-26 16:11:04] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:03:07 [I 2022-10-26 16:11:04] ...trap/engines/engine.py.229: process: 0.49971 | load: 0.00245 [I 2022-10-26 16:11:04] ...trap/engines/engine.py.230: loss: 0.27251 [I 2022-10-26 16:11:04] ...optimizers/trijoint.py.151: total_norm: 4.015092 [I 2022-10-26 16:11:09] ...trap/engines/engine.py.224: train: epoch 17 | batch 1940/2304 [I 2022-10-26 16:11:09] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:03:02 [I 2022-10-26 16:11:09] ...trap/engines/engine.py.229: process: 0.49471 | load: 0.00252 [I 2022-10-26 16:11:09] ...trap/engines/engine.py.230: loss: 0.22848 [I 2022-10-26 16:11:09] ...optimizers/trijoint.py.151: total_norm: 2.731654 [I 2022-10-26 16:11:14] ...trap/engines/engine.py.224: train: epoch 17 | batch 1950/2304 [I 2022-10-26 16:11:14] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:02:55 [I 2022-10-26 16:11:14] ...trap/engines/engine.py.229: process: 0.48279 | load: 0.00246 [I 2022-10-26 16:11:14] ...trap/engines/engine.py.230: loss: 0.25828 [I 2022-10-26 16:11:14] ...optimizers/trijoint.py.151: total_norm: 4.063176 [I 2022-10-26 16:11:19] ...trap/engines/engine.py.224: train: epoch 17 | batch 1960/2304 [I 2022-10-26 16:11:19] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:02:51 [I 2022-10-26 16:11:19] ...trap/engines/engine.py.229: process: 0.50018 | load: 0.00245 [I 2022-10-26 16:11:19] ...trap/engines/engine.py.230: loss: 0.19996 [I 2022-10-26 16:11:19] ...optimizers/trijoint.py.151: total_norm: 3.884712 [I 2022-10-26 16:11:25] ...trap/engines/engine.py.224: train: epoch 17 | batch 1970/2304 [I 2022-10-26 16:11:25] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:02:48 [I 2022-10-26 16:11:25] ...trap/engines/engine.py.229: process: 0.50005 | load: 0.00243 [I 2022-10-26 16:11:25] ...trap/engines/engine.py.230: loss: 0.23802 [I 2022-10-26 16:11:25] ...optimizers/trijoint.py.151: total_norm: 3.999519 [I 2022-10-26 16:11:30] ...trap/engines/engine.py.224: train: epoch 17 | batch 1980/2304 [I 2022-10-26 16:11:30] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:02:42 [I 2022-10-26 16:11:30] ...trap/engines/engine.py.229: process: 0.50482 | load: 0.00242 [I 2022-10-26 16:11:30] ...trap/engines/engine.py.230: loss: 0.18929 [I 2022-10-26 16:11:30] ...optimizers/trijoint.py.151: total_norm: 3.853662 [I 2022-10-26 16:11:35] ...trap/engines/engine.py.224: train: epoch 17 | batch 1990/2304 [I 2022-10-26 16:11:35] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:02:38 [I 2022-10-26 16:11:35] ...trap/engines/engine.py.229: process: 0.50551 | load: 0.00244 [I 2022-10-26 16:11:35] ...trap/engines/engine.py.230: loss: 0.28584 [I 2022-10-26 16:11:35] ...optimizers/trijoint.py.151: total_norm: 3.145826 [I 2022-10-26 16:11:40] ...trap/engines/engine.py.224: train: epoch 17 | batch 2000/2304 [I 2022-10-26 16:11:40] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:02:32 [I 2022-10-26 16:11:40] ...trap/engines/engine.py.229: process: 0.49539 | load: 0.00252 [I 2022-10-26 16:11:40] ...trap/engines/engine.py.230: loss: 0.23190 [I 2022-10-26 16:11:40] ...optimizers/trijoint.py.151: total_norm: 3.797804 [I 2022-10-26 16:11:45] ...trap/engines/engine.py.224: train: epoch 17 | batch 2010/2304 [I 2022-10-26 16:11:45] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:02:28 [I 2022-10-26 16:11:45] ...trap/engines/engine.py.229: process: 0.50428 | load: 0.00254 [I 2022-10-26 16:11:45] ...trap/engines/engine.py.230: loss: 0.25313 [I 2022-10-26 16:11:45] ...optimizers/trijoint.py.151: total_norm: 3.953083 [I 2022-10-26 16:11:50] ...trap/engines/engine.py.224: train: epoch 17 | batch 2020/2304 [I 2022-10-26 16:11:50] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:02:22 [I 2022-10-26 16:11:50] ...trap/engines/engine.py.229: process: 0.50645 | load: 0.00245 [I 2022-10-26 16:11:50] ...trap/engines/engine.py.230: loss: 0.21339 [I 2022-10-26 16:11:50] ...optimizers/trijoint.py.151: total_norm: 2.735548 [I 2022-10-26 16:11:55] ...trap/engines/engine.py.224: train: epoch 17 | batch 2030/2304 [I 2022-10-26 16:11:55] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:02:16 [I 2022-10-26 16:11:55] ...trap/engines/engine.py.229: process: 0.48722 | load: 0.00258 [I 2022-10-26 16:11:55] ...trap/engines/engine.py.230: loss: 0.23139 [I 2022-10-26 16:11:55] ...optimizers/trijoint.py.151: total_norm: 4.650702 [I 2022-10-26 16:12:00] ...trap/engines/engine.py.224: train: epoch 17 | batch 2040/2304 [I 2022-10-26 16:12:00] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:02:12 [I 2022-10-26 16:12:00] ...trap/engines/engine.py.229: process: 0.48969 | load: 0.00245 [I 2022-10-26 16:12:00] ...trap/engines/engine.py.230: loss: 0.25581 [I 2022-10-26 16:12:00] ...optimizers/trijoint.py.151: total_norm: 3.664580 [I 2022-10-26 16:12:05] ...trap/engines/engine.py.224: train: epoch 17 | batch 2050/2304 [I 2022-10-26 16:12:05] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:02:06 [I 2022-10-26 16:12:05] ...trap/engines/engine.py.229: process: 0.49405 | load: 0.00249 [I 2022-10-26 16:12:05] ...trap/engines/engine.py.230: loss: 0.26655 [I 2022-10-26 16:12:05] ...optimizers/trijoint.py.151: total_norm: 3.977420 [I 2022-10-26 16:12:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 2060/2304 [I 2022-10-26 16:12:10] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:02:03 [I 2022-10-26 16:12:10] ...trap/engines/engine.py.229: process: 0.51210 | load: 0.00247 [I 2022-10-26 16:12:10] ...trap/engines/engine.py.230: loss: 0.17398 [I 2022-10-26 16:12:10] ...optimizers/trijoint.py.151: total_norm: 2.758843 [I 2022-10-26 16:12:15] ...trap/engines/engine.py.224: train: epoch 17 | batch 2070/2304 [I 2022-10-26 16:12:15] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:01:57 [I 2022-10-26 16:12:15] ...trap/engines/engine.py.229: process: 0.49607 | load: 0.00243 [I 2022-10-26 16:12:15] ...trap/engines/engine.py.230: loss: 0.22715 [I 2022-10-26 16:12:15] ...optimizers/trijoint.py.151: total_norm: 3.656232 [I 2022-10-26 16:12:20] ...trap/engines/engine.py.224: train: epoch 17 | batch 2080/2304 [I 2022-10-26 16:12:20] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:01:52 [I 2022-10-26 16:12:20] ...trap/engines/engine.py.229: process: 0.51136 | load: 0.00247 [I 2022-10-26 16:12:20] ...trap/engines/engine.py.230: loss: 0.22526 [I 2022-10-26 16:12:20] ...optimizers/trijoint.py.151: total_norm: 3.432351 [I 2022-10-26 16:12:25] ...trap/engines/engine.py.224: train: epoch 17 | batch 2090/2304 [I 2022-10-26 16:12:25] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:01:46 [I 2022-10-26 16:12:25] ...trap/engines/engine.py.229: process: 0.50267 | load: 0.00246 [I 2022-10-26 16:12:25] ...trap/engines/engine.py.230: loss: 0.17972 [I 2022-10-26 16:12:25] ...optimizers/trijoint.py.151: total_norm: 2.717467 [I 2022-10-26 16:12:30] ...trap/engines/engine.py.224: train: epoch 17 | batch 2100/2304 [I 2022-10-26 16:12:30] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:01:42 [I 2022-10-26 16:12:30] ...trap/engines/engine.py.229: process: 0.50143 | load: 0.00252 [I 2022-10-26 16:12:30] ...trap/engines/engine.py.230: loss: 0.19656 [I 2022-10-26 16:12:30] ...optimizers/trijoint.py.151: total_norm: 3.500821 [I 2022-10-26 16:12:35] ...trap/engines/engine.py.224: train: epoch 17 | batch 2110/2304 [I 2022-10-26 16:12:35] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:01:37 [I 2022-10-26 16:12:35] ...trap/engines/engine.py.229: process: 0.50923 | load: 0.00250 [I 2022-10-26 16:12:35] ...trap/engines/engine.py.230: loss: 0.19036 [I 2022-10-26 16:12:35] ...optimizers/trijoint.py.151: total_norm: 3.048877 [I 2022-10-26 16:12:40] ...trap/engines/engine.py.224: train: epoch 17 | batch 2120/2304 [I 2022-10-26 16:12:40] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:01:31 [I 2022-10-26 16:12:40] ...trap/engines/engine.py.229: process: 0.49918 | load: 0.00248 [I 2022-10-26 16:12:40] ...trap/engines/engine.py.230: loss: 0.22742 [I 2022-10-26 16:12:40] ...optimizers/trijoint.py.151: total_norm: 3.394677 [I 2022-10-26 16:12:45] ...trap/engines/engine.py.224: train: epoch 17 | batch 2130/2304 [I 2022-10-26 16:12:45] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:01:27 [I 2022-10-26 16:12:45] ...trap/engines/engine.py.229: process: 0.50946 | load: 0.00242 [I 2022-10-26 16:12:45] ...trap/engines/engine.py.230: loss: 0.20158 [I 2022-10-26 16:12:45] ...optimizers/trijoint.py.151: total_norm: 2.733890 [I 2022-10-26 16:12:50] ...trap/engines/engine.py.224: train: epoch 17 | batch 2140/2304 [I 2022-10-26 16:12:50] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:01:22 [I 2022-10-26 16:12:50] ...trap/engines/engine.py.229: process: 0.50297 | load: 0.00245 [I 2022-10-26 16:12:50] ...trap/engines/engine.py.230: loss: 0.15652 [I 2022-10-26 16:12:50] ...optimizers/trijoint.py.151: total_norm: 2.890338 [I 2022-10-26 16:12:55] ...trap/engines/engine.py.224: train: epoch 17 | batch 2150/2304 [I 2022-10-26 16:12:55] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:01:17 [I 2022-10-26 16:12:55] ...trap/engines/engine.py.229: process: 0.49576 | load: 0.00241 [I 2022-10-26 16:12:55] ...trap/engines/engine.py.230: loss: 0.23238 [I 2022-10-26 16:12:55] ...optimizers/trijoint.py.151: total_norm: 3.532719 [I 2022-10-26 16:13:00] ...trap/engines/engine.py.224: train: epoch 17 | batch 2160/2304 [I 2022-10-26 16:13:00] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:01:11 [I 2022-10-26 16:13:00] ...trap/engines/engine.py.229: process: 0.48436 | load: 0.00247 [I 2022-10-26 16:13:00] ...trap/engines/engine.py.230: loss: 0.30962 [I 2022-10-26 16:13:00] ...optimizers/trijoint.py.151: total_norm: 3.045599 [I 2022-10-26 16:13:05] ...trap/engines/engine.py.224: train: epoch 17 | batch 2170/2304 [I 2022-10-26 16:13:05] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:01:06 [I 2022-10-26 16:13:05] ...trap/engines/engine.py.229: process: 0.49518 | load: 0.00245 [I 2022-10-26 16:13:05] ...trap/engines/engine.py.230: loss: 0.14311 [I 2022-10-26 16:13:05] ...optimizers/trijoint.py.151: total_norm: 2.340833 [I 2022-10-26 16:13:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 2180/2304 [I 2022-10-26 16:13:10] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:01:01 [I 2022-10-26 16:13:10] ...trap/engines/engine.py.229: process: 0.48231 | load: 0.00244 [I 2022-10-26 16:13:10] ...trap/engines/engine.py.230: loss: 0.24193 [I 2022-10-26 16:13:10] ...optimizers/trijoint.py.151: total_norm: 2.264432 [I 2022-10-26 16:13:15] ...trap/engines/engine.py.224: train: epoch 17 | batch 2190/2304 [I 2022-10-26 16:13:15] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:00:57 [I 2022-10-26 16:13:15] ...trap/engines/engine.py.229: process: 0.50563 | load: 0.00244 [I 2022-10-26 16:13:15] ...trap/engines/engine.py.230: loss: 0.22027 [I 2022-10-26 16:13:15] ...optimizers/trijoint.py.151: total_norm: 3.612902 [I 2022-10-26 16:13:20] ...trap/engines/engine.py.224: train: epoch 17 | batch 2200/2304 [I 2022-10-26 16:13:20] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:00:51 [I 2022-10-26 16:13:20] ...trap/engines/engine.py.229: process: 0.49223 | load: 0.00242 [I 2022-10-26 16:13:20] ...trap/engines/engine.py.230: loss: 0.19190 [I 2022-10-26 16:13:20] ...optimizers/trijoint.py.151: total_norm: 3.407456 [I 2022-10-26 16:13:25] ...trap/engines/engine.py.224: train: epoch 17 | batch 2210/2304 [I 2022-10-26 16:13:25] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:00:46 [I 2022-10-26 16:13:25] ...trap/engines/engine.py.229: process: 0.49599 | load: 0.00242 [I 2022-10-26 16:13:25] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-26 16:13:25] ...optimizers/trijoint.py.151: total_norm: 2.123128 [I 2022-10-26 16:13:30] ...trap/engines/engine.py.224: train: epoch 17 | batch 2220/2304 [I 2022-10-26 16:13:30] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:00:42 [I 2022-10-26 16:13:30] ...trap/engines/engine.py.229: process: 0.50537 | load: 0.00244 [I 2022-10-26 16:13:30] ...trap/engines/engine.py.230: loss: 0.14812 [I 2022-10-26 16:13:30] ...optimizers/trijoint.py.151: total_norm: 2.007011 [I 2022-10-26 16:13:35] ...trap/engines/engine.py.224: train: epoch 17 | batch 2230/2304 [I 2022-10-26 16:13:35] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:00:36 [I 2022-10-26 16:13:35] ...trap/engines/engine.py.229: process: 0.48921 | load: 0.00249 [I 2022-10-26 16:13:35] ...trap/engines/engine.py.230: loss: 0.12827 [I 2022-10-26 16:13:35] ...optimizers/trijoint.py.151: total_norm: 2.569808 [I 2022-10-26 16:13:40] ...trap/engines/engine.py.224: train: epoch 17 | batch 2240/2304 [I 2022-10-26 16:13:40] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:00:32 [I 2022-10-26 16:13:40] ...trap/engines/engine.py.229: process: 0.50593 | load: 0.00245 [I 2022-10-26 16:13:40] ...trap/engines/engine.py.230: loss: 0.18561 [I 2022-10-26 16:13:40] ...optimizers/trijoint.py.151: total_norm: 4.003687 [I 2022-10-26 16:13:45] ...trap/engines/engine.py.224: train: epoch 17 | batch 2250/2304 [I 2022-10-26 16:13:45] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:00:26 [I 2022-10-26 16:13:45] ...trap/engines/engine.py.229: process: 0.48973 | load: 0.00246 [I 2022-10-26 16:13:45] ...trap/engines/engine.py.230: loss: 0.16481 [I 2022-10-26 16:13:45] ...optimizers/trijoint.py.151: total_norm: 2.747999 [I 2022-10-26 16:13:50] ...trap/engines/engine.py.224: train: epoch 17 | batch 2260/2304 [I 2022-10-26 16:13:50] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:00:21 [I 2022-10-26 16:13:50] ...trap/engines/engine.py.229: process: 0.49547 | load: 0.00247 [I 2022-10-26 16:13:50] ...trap/engines/engine.py.230: loss: 0.14257 [I 2022-10-26 16:13:50] ...optimizers/trijoint.py.151: total_norm: 2.463023 [I 2022-10-26 16:13:55] ...trap/engines/engine.py.224: train: epoch 17 | batch 2270/2304 [I 2022-10-26 16:13:55] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:00:16 [I 2022-10-26 16:13:55] ...trap/engines/engine.py.229: process: 0.49431 | load: 0.00244 [I 2022-10-26 16:13:55] ...trap/engines/engine.py.230: loss: 0.25743 [I 2022-10-26 16:13:55] ...optimizers/trijoint.py.151: total_norm: 2.492618 [I 2022-10-26 16:14:00] ...trap/engines/engine.py.224: train: epoch 17 | batch 2280/2304 [I 2022-10-26 16:14:00] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:00:12 [I 2022-10-26 16:14:00] ...trap/engines/engine.py.229: process: 0.50627 | load: 0.00250 [I 2022-10-26 16:14:00] ...trap/engines/engine.py.230: loss: 0.22005 [I 2022-10-26 16:14:00] ...optimizers/trijoint.py.151: total_norm: 3.195600 [I 2022-10-26 16:14:05] ...trap/engines/engine.py.224: train: epoch 17 | batch 2290/2304 [I 2022-10-26 16:14:05] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:00:07 [I 2022-10-26 16:14:05] ...trap/engines/engine.py.229: process: 0.50415 | load: 0.00245 [I 2022-10-26 16:14:05] ...trap/engines/engine.py.230: loss: 0.16380 [I 2022-10-26 16:14:05] ...optimizers/trijoint.py.151: total_norm: 3.775677 [I 2022-10-26 16:14:10] ...trap/engines/engine.py.224: train: epoch 17 | batch 2300/2304 [I 2022-10-26 16:14:10] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:00:02 [I 2022-10-26 16:14:10] ...trap/engines/engine.py.229: process: 0.49965 | load: 0.00014 [I 2022-10-26 16:14:10] ...trap/engines/engine.py.230: loss: 0.26054 [I 2022-10-26 16:14:10] ...optimizers/trijoint.py.151: total_norm: 4.343473 [I 2022-10-26 16:14:12] ...trap/engines/engine.py.224: train: epoch 17 | batch 2304/2304 [I 2022-10-26 16:14:12] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:00:00 [I 2022-10-26 16:14:12] ...trap/engines/engine.py.229: process: 0.50492 | load: 0.00014 [I 2022-10-26 16:14:12] ...trap/engines/engine.py.230: loss: 0.26468 [I 2022-10-26 16:14:12] ...optimizers/trijoint.py.151: total_norm: 2.710115 [S 2022-10-26 16:14:13] ...trap/engines/engine.py.129: train_epoch.epoch: 17 [S 2022-10-26 16:14:13] ...trap/engines/engine.py.129: train_epoch.loss: 0.20881184509868478 [I 2022-10-26 16:14:13] ...s/criterions/triplet.py.84: margin = 0.14000000000000007 [I 2022-10-26 16:14:14] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 17 [I 2022-10-26 16:14:14] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 16:14:15] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 16:14:15] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 16:14:15] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 16:14:16] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 16:14:16] ...trap/engines/engine.py.321: eval: epoch 17 | batch 0/510 [I 2022-10-26 16:14:16] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:37 [I 2022-10-26 16:14:16] ...trap/engines/engine.py.326: process: 2.77972 | load: 2.41626 [I 2022-10-26 16:14:20] ...trap/engines/engine.py.321: eval: epoch 17 | batch 10/510 [I 2022-10-26 16:14:20] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:58 [I 2022-10-26 16:14:20] ...trap/engines/engine.py.326: process: 0.32202 | load: 0.00020 [I 2022-10-26 16:14:23] ...trap/engines/engine.py.321: eval: epoch 17 | batch 20/510 [I 2022-10-26 16:14:23] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:57 [I 2022-10-26 16:14:23] ...trap/engines/engine.py.326: process: 0.32919 | load: 0.00032 [I 2022-10-26 16:14:27] ...trap/engines/engine.py.321: eval: epoch 17 | batch 30/510 [I 2022-10-26 16:14:27] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:45 [I 2022-10-26 16:14:27] ...trap/engines/engine.py.326: process: 0.35863 | load: 0.00035 [I 2022-10-26 16:14:30] ...trap/engines/engine.py.321: eval: epoch 17 | batch 40/510 [I 2022-10-26 16:14:30] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:40 [I 2022-10-26 16:14:30] ...trap/engines/engine.py.326: process: 0.32628 | load: 0.00020 [I 2022-10-26 16:14:33] ...trap/engines/engine.py.321: eval: epoch 17 | batch 50/510 [I 2022-10-26 16:14:33] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-26 16:14:33] ...trap/engines/engine.py.326: process: 0.33525 | load: 0.00019 [I 2022-10-26 16:14:37] ...trap/engines/engine.py.321: eval: epoch 17 | batch 60/510 [I 2022-10-26 16:14:37] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:29 [I 2022-10-26 16:14:37] ...trap/engines/engine.py.326: process: 0.32228 | load: 0.00019 [I 2022-10-26 16:14:40] ...trap/engines/engine.py.321: eval: epoch 17 | batch 70/510 [I 2022-10-26 16:14:40] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:31 [I 2022-10-26 16:14:40] ...trap/engines/engine.py.326: process: 0.36156 | load: 0.00030 [I 2022-10-26 16:14:44] ...trap/engines/engine.py.321: eval: epoch 17 | batch 80/510 [I 2022-10-26 16:14:44] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:29 [I 2022-10-26 16:14:44] ...trap/engines/engine.py.326: process: 0.35408 | load: 0.00017 [I 2022-10-26 16:14:47] ...trap/engines/engine.py.321: eval: epoch 17 | batch 90/510 [I 2022-10-26 16:14:47] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-26 16:14:47] ...trap/engines/engine.py.326: process: 0.33068 | load: 0.00019 [I 2022-10-26 16:14:50] ...trap/engines/engine.py.321: eval: epoch 17 | batch 100/510 [I 2022-10-26 16:14:50] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:15 [I 2022-10-26 16:14:50] ...trap/engines/engine.py.326: process: 0.30178 | load: 0.00028 [I 2022-10-26 16:14:53] ...trap/engines/engine.py.321: eval: epoch 17 | batch 110/510 [I 2022-10-26 16:14:53] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:58 [I 2022-10-26 16:14:53] ...trap/engines/engine.py.326: process: 0.29228 | load: 0.00015 [I 2022-10-26 16:14:56] ...trap/engines/engine.py.321: eval: epoch 17 | batch 120/510 [I 2022-10-26 16:14:56] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 16:14:56] ...trap/engines/engine.py.326: process: 0.29166 | load: 0.00019 [I 2022-10-26 16:14:59] ...trap/engines/engine.py.321: eval: epoch 17 | batch 130/510 [I 2022-10-26 16:14:59] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-26 16:14:59] ...trap/engines/engine.py.326: process: 0.29095 | load: 0.00017 [I 2022-10-26 16:15:02] ...trap/engines/engine.py.321: eval: epoch 17 | batch 140/510 [I 2022-10-26 16:15:02] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-26 16:15:02] ...trap/engines/engine.py.326: process: 0.27874 | load: 0.00015 [I 2022-10-26 16:15:05] ...trap/engines/engine.py.321: eval: epoch 17 | batch 150/510 [I 2022-10-26 16:15:05] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-26 16:15:05] ...trap/engines/engine.py.326: process: 0.28542 | load: 0.00018 [I 2022-10-26 16:15:08] ...trap/engines/engine.py.321: eval: epoch 17 | batch 160/510 [I 2022-10-26 16:15:08] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-26 16:15:08] ...trap/engines/engine.py.326: process: 0.29475 | load: 0.00016 [I 2022-10-26 16:15:11] ...trap/engines/engine.py.321: eval: epoch 17 | batch 170/510 [I 2022-10-26 16:15:11] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 16:15:11] ...trap/engines/engine.py.326: process: 0.28103 | load: 0.00020 [I 2022-10-26 16:15:14] ...trap/engines/engine.py.321: eval: epoch 17 | batch 180/510 [I 2022-10-26 16:15:14] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 16:15:14] ...trap/engines/engine.py.326: process: 0.27964 | load: 0.00022 [I 2022-10-26 16:15:16] ...trap/engines/engine.py.321: eval: epoch 17 | batch 190/510 [I 2022-10-26 16:15:16] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:33 [I 2022-10-26 16:15:16] ...trap/engines/engine.py.326: process: 0.29077 | load: 0.00015 [I 2022-10-26 16:15:19] ...trap/engines/engine.py.321: eval: epoch 17 | batch 200/510 [I 2022-10-26 16:15:19] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 16:15:19] ...trap/engines/engine.py.326: process: 0.28390 | load: 0.00015 [I 2022-10-26 16:15:22] ...trap/engines/engine.py.321: eval: epoch 17 | batch 210/510 [I 2022-10-26 16:15:22] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:30 [I 2022-10-26 16:15:22] ...trap/engines/engine.py.326: process: 0.32636 | load: 0.00034 [I 2022-10-26 16:15:25] ...trap/engines/engine.py.321: eval: epoch 17 | batch 220/510 [I 2022-10-26 16:15:25] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-26 16:15:25] ...trap/engines/engine.py.326: process: 0.29709 | load: 0.00020 [I 2022-10-26 16:15:28] ...trap/engines/engine.py.321: eval: epoch 17 | batch 230/510 [I 2022-10-26 16:15:28] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-26 16:15:28] ...trap/engines/engine.py.326: process: 0.29475 | load: 0.00019 [I 2022-10-26 16:15:31] ...trap/engines/engine.py.321: eval: epoch 17 | batch 240/510 [I 2022-10-26 16:15:31] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:20 [I 2022-10-26 16:15:31] ...trap/engines/engine.py.326: process: 0.32888 | load: 0.00022 [I 2022-10-26 16:15:34] ...trap/engines/engine.py.321: eval: epoch 17 | batch 250/510 [I 2022-10-26 16:15:34] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-26 16:15:34] ...trap/engines/engine.py.326: process: 0.28442 | load: 0.00022 [I 2022-10-26 16:15:37] ...trap/engines/engine.py.321: eval: epoch 17 | batch 260/510 [I 2022-10-26 16:15:37] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-26 16:15:37] ...trap/engines/engine.py.326: process: 0.29327 | load: 0.00016 [I 2022-10-26 16:15:40] ...trap/engines/engine.py.321: eval: epoch 17 | batch 270/510 [I 2022-10-26 16:15:40] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-26 16:15:40] ...trap/engines/engine.py.326: process: 0.30758 | load: 0.00015 [I 2022-10-26 16:15:43] ...trap/engines/engine.py.321: eval: epoch 17 | batch 280/510 [I 2022-10-26 16:15:43] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:05 [I 2022-10-26 16:15:43] ...trap/engines/engine.py.326: process: 0.28308 | load: 0.00016 [I 2022-10-26 16:15:46] ...trap/engines/engine.py.321: eval: epoch 17 | batch 290/510 [I 2022-10-26 16:15:46] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 16:15:46] ...trap/engines/engine.py.326: process: 0.28361 | load: 0.00021 [I 2022-10-26 16:15:48] ...trap/engines/engine.py.321: eval: epoch 17 | batch 300/510 [I 2022-10-26 16:15:48] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 16:15:48] ...trap/engines/engine.py.326: process: 0.28336 | load: 0.00015 [I 2022-10-26 16:15:51] ...trap/engines/engine.py.321: eval: epoch 17 | batch 310/510 [I 2022-10-26 16:15:51] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 16:15:51] ...trap/engines/engine.py.326: process: 0.28929 | load: 0.00015 [I 2022-10-26 16:15:54] ...trap/engines/engine.py.321: eval: epoch 17 | batch 320/510 [I 2022-10-26 16:15:54] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:55 [I 2022-10-26 16:15:54] ...trap/engines/engine.py.326: process: 0.28300 | load: 0.00015 [I 2022-10-26 16:15:57] ...trap/engines/engine.py.321: eval: epoch 17 | batch 330/510 [I 2022-10-26 16:15:57] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:52 [I 2022-10-26 16:15:57] ...trap/engines/engine.py.326: process: 0.28791 | load: 0.00015 [I 2022-10-26 16:16:00] ...trap/engines/engine.py.321: eval: epoch 17 | batch 340/510 [I 2022-10-26 16:16:00] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 16:16:00] ...trap/engines/engine.py.326: process: 0.28586 | load: 0.00015 [I 2022-10-26 16:16:03] ...trap/engines/engine.py.321: eval: epoch 17 | batch 350/510 [I 2022-10-26 16:16:03] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 16:16:03] ...trap/engines/engine.py.326: process: 0.29115 | load: 0.00014 [I 2022-10-26 16:16:06] ...trap/engines/engine.py.321: eval: epoch 17 | batch 360/510 [I 2022-10-26 16:16:06] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 16:16:06] ...trap/engines/engine.py.326: process: 0.28738 | load: 0.00015 [I 2022-10-26 16:16:09] ...trap/engines/engine.py.321: eval: epoch 17 | batch 370/510 [I 2022-10-26 16:16:09] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-26 16:16:09] ...trap/engines/engine.py.326: process: 0.28316 | load: 0.00021 [I 2022-10-26 16:16:12] ...trap/engines/engine.py.321: eval: epoch 17 | batch 380/510 [I 2022-10-26 16:16:12] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 16:16:12] ...trap/engines/engine.py.326: process: 0.28809 | load: 0.00015 [I 2022-10-26 16:16:14] ...trap/engines/engine.py.321: eval: epoch 17 | batch 390/510 [I 2022-10-26 16:16:14] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:35 [I 2022-10-26 16:16:14] ...trap/engines/engine.py.326: process: 0.31193 | load: 0.00020 [I 2022-10-26 16:16:17] ...trap/engines/engine.py.321: eval: epoch 17 | batch 400/510 [I 2022-10-26 16:16:17] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 16:16:17] ...trap/engines/engine.py.326: process: 0.28977 | load: 0.00018 [I 2022-10-26 16:16:20] ...trap/engines/engine.py.321: eval: epoch 17 | batch 410/510 [I 2022-10-26 16:16:20] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 16:16:20] ...trap/engines/engine.py.326: process: 0.28385 | load: 0.00019 [I 2022-10-26 16:16:23] ...trap/engines/engine.py.321: eval: epoch 17 | batch 420/510 [I 2022-10-26 16:16:23] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 16:16:23] ...trap/engines/engine.py.326: process: 0.28921 | load: 0.00022 [I 2022-10-26 16:16:26] ...trap/engines/engine.py.321: eval: epoch 17 | batch 430/510 [I 2022-10-26 16:16:26] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:22 [I 2022-10-26 16:16:26] ...trap/engines/engine.py.326: process: 0.28897 | load: 0.00016 [I 2022-10-26 16:16:29] ...trap/engines/engine.py.321: eval: epoch 17 | batch 440/510 [I 2022-10-26 16:16:29] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 16:16:29] ...trap/engines/engine.py.326: process: 0.28657 | load: 0.00015 [I 2022-10-26 16:16:32] ...trap/engines/engine.py.321: eval: epoch 17 | batch 450/510 [I 2022-10-26 16:16:32] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 16:16:32] ...trap/engines/engine.py.326: process: 0.29129 | load: 0.00015 [I 2022-10-26 16:16:35] ...trap/engines/engine.py.321: eval: epoch 17 | batch 460/510 [I 2022-10-26 16:16:35] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-26 16:16:35] ...trap/engines/engine.py.326: process: 0.28167 | load: 0.00021 [I 2022-10-26 16:16:38] ...trap/engines/engine.py.321: eval: epoch 17 | batch 470/510 [I 2022-10-26 16:16:38] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 16:16:38] ...trap/engines/engine.py.326: process: 0.28601 | load: 0.00016 [I 2022-10-26 16:16:40] ...trap/engines/engine.py.321: eval: epoch 17 | batch 480/510 [I 2022-10-26 16:16:40] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 16:16:40] ...trap/engines/engine.py.326: process: 0.28921 | load: 0.00017 [I 2022-10-26 16:16:43] ...trap/engines/engine.py.321: eval: epoch 17 | batch 490/510 [I 2022-10-26 16:16:43] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 16:16:43] ...trap/engines/engine.py.326: process: 0.28865 | load: 0.00020 [I 2022-10-26 16:16:46] ...trap/engines/engine.py.321: eval: epoch 17 | batch 500/510 [I 2022-10-26 16:16:46] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 16:16:46] ...trap/engines/engine.py.326: process: 0.28435 | load: 0.00025 [I 2022-10-26 16:16:49] ...trap/engines/engine.py.321: eval: epoch 17 | batch 510/510 [I 2022-10-26 16:16:49] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 16:16:49] ...trap/engines/engine.py.326: process: 0.30307 | load: 0.00010 [S 2022-10-26 16:16:50] ...trap/engines/engine.py.132: eval_epoch.epoch: 17 [I 2022-10-26 16:16:50] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.569000 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.837600 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.896200 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.564400 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.834100 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.894800 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010431 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009922 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010600 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013858 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012128 [S 2022-10-26 16:16:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007613 [I 2022-10-26 16:16:55] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 16:16:55] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 16:16:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 16:16:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 16:16:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 16:16:56] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 16:16:59] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 16:17:04] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 16:17:04] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 16:17:04] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 16:17:07] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 16:17:12] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 16:17:12] ...trap/engines/engine.py.163: Training model on trainset for epoch 18 [I 2022-10-26 16:17:12] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 16:17:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 0/2304 [I 2022-10-26 16:17:15] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:58 [I 2022-10-26 16:17:15] ...trap/engines/engine.py.229: process: 2.73388 | load: 2.13415 [I 2022-10-26 16:17:15] ...trap/engines/engine.py.230: loss: 0.17609 [I 2022-10-26 16:17:15] ...optimizers/trijoint.py.151: total_norm: 3.349991 [I 2022-10-26 16:17:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 10/2304 [I 2022-10-26 16:17:20] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:18 [I 2022-10-26 16:17:20] ...trap/engines/engine.py.229: process: 0.49788 | load: 0.00389 [I 2022-10-26 16:17:20] ...trap/engines/engine.py.230: loss: 0.19213 [I 2022-10-26 16:17:20] ...optimizers/trijoint.py.151: total_norm: 3.063379 [I 2022-10-26 16:17:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 20/2304 [I 2022-10-26 16:17:25] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:15 [I 2022-10-26 16:17:25] ...trap/engines/engine.py.229: process: 0.51258 | load: 0.00407 [I 2022-10-26 16:17:25] ...trap/engines/engine.py.230: loss: 0.19998 [I 2022-10-26 16:17:25] ...optimizers/trijoint.py.151: total_norm: 2.758836 [I 2022-10-26 16:17:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 30/2304 [I 2022-10-26 16:17:30] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:04 [I 2022-10-26 16:17:30] ...trap/engines/engine.py.229: process: 0.50022 | load: 0.00328 [I 2022-10-26 16:17:30] ...trap/engines/engine.py.230: loss: 0.19692 [I 2022-10-26 16:17:30] ...optimizers/trijoint.py.151: total_norm: 3.896684 [I 2022-10-26 16:17:35] ...trap/engines/engine.py.224: train: epoch 18 | batch 40/2304 [I 2022-10-26 16:17:35] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:18:52 [I 2022-10-26 16:17:35] ...trap/engines/engine.py.229: process: 0.49395 | load: 0.00315 [I 2022-10-26 16:17:35] ...trap/engines/engine.py.230: loss: 0.25937 [I 2022-10-26 16:17:35] ...optimizers/trijoint.py.151: total_norm: 3.459194 [I 2022-10-26 16:17:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 50/2304 [I 2022-10-26 16:17:40] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:39 [I 2022-10-26 16:17:40] ...trap/engines/engine.py.229: process: 0.48692 | load: 0.00312 [I 2022-10-26 16:17:40] ...trap/engines/engine.py.230: loss: 0.21619 [I 2022-10-26 16:17:40] ...optimizers/trijoint.py.151: total_norm: 2.773014 [I 2022-10-26 16:17:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 60/2304 [I 2022-10-26 16:17:45] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:32 [I 2022-10-26 16:17:45] ...trap/engines/engine.py.229: process: 0.48949 | load: 0.00298 [I 2022-10-26 16:17:45] ...trap/engines/engine.py.230: loss: 0.19956 [I 2022-10-26 16:17:45] ...optimizers/trijoint.py.151: total_norm: 3.431169 [I 2022-10-26 16:17:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 70/2304 [I 2022-10-26 16:17:50] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:38 [I 2022-10-26 16:17:50] ...trap/engines/engine.py.229: process: 0.48936 | load: 0.00273 [I 2022-10-26 16:17:50] ...trap/engines/engine.py.230: loss: 0.27240 [I 2022-10-26 16:17:50] ...optimizers/trijoint.py.151: total_norm: 2.810135 [I 2022-10-26 16:17:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 80/2304 [I 2022-10-26 16:17:55] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:33 [I 2022-10-26 16:17:55] ...trap/engines/engine.py.229: process: 0.50234 | load: 0.00308 [I 2022-10-26 16:17:55] ...trap/engines/engine.py.230: loss: 0.25085 [I 2022-10-26 16:17:55] ...optimizers/trijoint.py.151: total_norm: 3.101481 [I 2022-10-26 16:18:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 90/2304 [I 2022-10-26 16:18:00] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:18:32 [I 2022-10-26 16:18:00] ...trap/engines/engine.py.229: process: 0.49414 | load: 0.00273 [I 2022-10-26 16:18:00] ...trap/engines/engine.py.230: loss: 0.20893 [I 2022-10-26 16:18:00] ...optimizers/trijoint.py.151: total_norm: 3.224809 [I 2022-10-26 16:18:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 100/2304 [I 2022-10-26 16:18:05] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:18:20 [I 2022-10-26 16:18:05] ...trap/engines/engine.py.229: process: 0.50317 | load: 0.00258 [I 2022-10-26 16:18:05] ...trap/engines/engine.py.230: loss: 0.13089 [I 2022-10-26 16:18:05] ...optimizers/trijoint.py.151: total_norm: 2.168137 [I 2022-10-26 16:18:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 110/2304 [I 2022-10-26 16:18:10] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:18:24 [I 2022-10-26 16:18:10] ...trap/engines/engine.py.229: process: 0.50346 | load: 0.00285 [I 2022-10-26 16:18:10] ...trap/engines/engine.py.230: loss: 0.22044 [I 2022-10-26 16:18:10] ...optimizers/trijoint.py.151: total_norm: 3.677339 [I 2022-10-26 16:18:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 120/2304 [I 2022-10-26 16:18:15] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:07 [I 2022-10-26 16:18:15] ...trap/engines/engine.py.229: process: 0.49145 | load: 0.00268 [I 2022-10-26 16:18:15] ...trap/engines/engine.py.230: loss: 0.14790 [I 2022-10-26 16:18:15] ...optimizers/trijoint.py.151: total_norm: 2.314964 [I 2022-10-26 16:18:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 130/2304 [I 2022-10-26 16:18:20] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:17:55 [I 2022-10-26 16:18:20] ...trap/engines/engine.py.229: process: 0.49438 | load: 0.00247 [I 2022-10-26 16:18:20] ...trap/engines/engine.py.230: loss: 0.16723 [I 2022-10-26 16:18:20] ...optimizers/trijoint.py.151: total_norm: 2.935045 [I 2022-10-26 16:18:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 140/2304 [I 2022-10-26 16:18:25] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:18:04 [I 2022-10-26 16:18:25] ...trap/engines/engine.py.229: process: 0.51229 | load: 0.00269 [I 2022-10-26 16:18:25] ...trap/engines/engine.py.230: loss: 0.30694 [I 2022-10-26 16:18:25] ...optimizers/trijoint.py.151: total_norm: 5.183419 [I 2022-10-26 16:18:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 150/2304 [I 2022-10-26 16:18:30] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:18:08 [I 2022-10-26 16:18:30] ...trap/engines/engine.py.229: process: 0.52357 | load: 0.00286 [I 2022-10-26 16:18:30] ...trap/engines/engine.py.230: loss: 0.19767 [I 2022-10-26 16:18:30] ...optimizers/trijoint.py.151: total_norm: 2.686402 [I 2022-10-26 16:18:35] ...trap/engines/engine.py.224: train: epoch 18 | batch 160/2304 [I 2022-10-26 16:18:35] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:18:03 [I 2022-10-26 16:18:35] ...trap/engines/engine.py.229: process: 0.50040 | load: 0.00279 [I 2022-10-26 16:18:35] ...trap/engines/engine.py.230: loss: 0.20242 [I 2022-10-26 16:18:35] ...optimizers/trijoint.py.151: total_norm: 3.737471 [I 2022-10-26 16:18:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 170/2304 [I 2022-10-26 16:18:40] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:41 [I 2022-10-26 16:18:40] ...trap/engines/engine.py.229: process: 0.49682 | load: 0.00269 [I 2022-10-26 16:18:40] ...trap/engines/engine.py.230: loss: 0.14526 [I 2022-10-26 16:18:40] ...optimizers/trijoint.py.151: total_norm: 2.382644 [I 2022-10-26 16:18:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 180/2304 [I 2022-10-26 16:18:45] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:17:56 [I 2022-10-26 16:18:45] ...trap/engines/engine.py.229: process: 0.50907 | load: 0.00257 [I 2022-10-26 16:18:45] ...trap/engines/engine.py.230: loss: 0.18239 [I 2022-10-26 16:18:45] ...optimizers/trijoint.py.151: total_norm: 3.554488 [I 2022-10-26 16:18:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 190/2304 [I 2022-10-26 16:18:50] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:18:00 [I 2022-10-26 16:18:50] ...trap/engines/engine.py.229: process: 0.50440 | load: 0.00263 [I 2022-10-26 16:18:50] ...trap/engines/engine.py.230: loss: 0.15729 [I 2022-10-26 16:18:50] ...optimizers/trijoint.py.151: total_norm: 2.016112 [I 2022-10-26 16:18:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 200/2304 [I 2022-10-26 16:18:55] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:40 [I 2022-10-26 16:18:55] ...trap/engines/engine.py.229: process: 0.50041 | load: 0.00258 [I 2022-10-26 16:18:55] ...trap/engines/engine.py.230: loss: 0.34294 [I 2022-10-26 16:18:55] ...optimizers/trijoint.py.151: total_norm: 3.641135 [I 2022-10-26 16:19:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 210/2304 [I 2022-10-26 16:19:00] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:39 [I 2022-10-26 16:19:00] ...trap/engines/engine.py.229: process: 0.53377 | load: 0.00265 [I 2022-10-26 16:19:00] ...trap/engines/engine.py.230: loss: 0.23802 [I 2022-10-26 16:19:00] ...optimizers/trijoint.py.151: total_norm: 3.401205 [I 2022-10-26 16:19:06] ...trap/engines/engine.py.224: train: epoch 18 | batch 220/2304 [I 2022-10-26 16:19:06] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:40 [I 2022-10-26 16:19:06] ...trap/engines/engine.py.229: process: 0.49366 | load: 0.00260 [I 2022-10-26 16:19:06] ...trap/engines/engine.py.230: loss: 0.24141 [I 2022-10-26 16:19:06] ...optimizers/trijoint.py.151: total_norm: 3.220511 [I 2022-10-26 16:19:11] ...trap/engines/engine.py.224: train: epoch 18 | batch 230/2304 [I 2022-10-26 16:19:11] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:28 [I 2022-10-26 16:19:11] ...trap/engines/engine.py.229: process: 0.50114 | load: 0.00280 [I 2022-10-26 16:19:11] ...trap/engines/engine.py.230: loss: 0.19259 [I 2022-10-26 16:19:11] ...optimizers/trijoint.py.151: total_norm: 2.777827 [I 2022-10-26 16:19:16] ...trap/engines/engine.py.224: train: epoch 18 | batch 240/2304 [I 2022-10-26 16:19:16] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:15 [I 2022-10-26 16:19:16] ...trap/engines/engine.py.229: process: 0.49908 | load: 0.00252 [I 2022-10-26 16:19:16] ...trap/engines/engine.py.230: loss: 0.24653 [I 2022-10-26 16:19:16] ...optimizers/trijoint.py.151: total_norm: 4.112204 [I 2022-10-26 16:19:21] ...trap/engines/engine.py.224: train: epoch 18 | batch 250/2304 [I 2022-10-26 16:19:21] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:06 [I 2022-10-26 16:19:21] ...trap/engines/engine.py.229: process: 0.50271 | load: 0.00248 [I 2022-10-26 16:19:21] ...trap/engines/engine.py.230: loss: 0.23404 [I 2022-10-26 16:19:21] ...optimizers/trijoint.py.151: total_norm: 3.043530 [I 2022-10-26 16:19:26] ...trap/engines/engine.py.224: train: epoch 18 | batch 260/2304 [I 2022-10-26 16:19:26] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:17:02 [I 2022-10-26 16:19:26] ...trap/engines/engine.py.229: process: 0.50318 | load: 0.00247 [I 2022-10-26 16:19:26] ...trap/engines/engine.py.230: loss: 0.20314 [I 2022-10-26 16:19:26] ...optimizers/trijoint.py.151: total_norm: 2.953007 [I 2022-10-26 16:19:31] ...trap/engines/engine.py.224: train: epoch 18 | batch 270/2304 [I 2022-10-26 16:19:31] ...trap/engines/engine.py.228: elapsed: 0:02:18 | left: 0:16:58 [I 2022-10-26 16:19:31] ...trap/engines/engine.py.229: process: 0.49818 | load: 0.00251 [I 2022-10-26 16:19:31] ...trap/engines/engine.py.230: loss: 0.15157 [I 2022-10-26 16:19:31] ...optimizers/trijoint.py.151: total_norm: 3.178198 [I 2022-10-26 16:19:36] ...trap/engines/engine.py.224: train: epoch 18 | batch 280/2304 [I 2022-10-26 16:19:36] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:16:50 [I 2022-10-26 16:19:36] ...trap/engines/engine.py.229: process: 0.49797 | load: 0.00255 [I 2022-10-26 16:19:36] ...trap/engines/engine.py.230: loss: 0.23778 [I 2022-10-26 16:19:36] ...optimizers/trijoint.py.151: total_norm: 3.520402 [I 2022-10-26 16:19:41] ...trap/engines/engine.py.224: train: epoch 18 | batch 290/2304 [I 2022-10-26 16:19:41] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:16:46 [I 2022-10-26 16:19:41] ...trap/engines/engine.py.229: process: 0.50353 | load: 0.00251 [I 2022-10-26 16:19:41] ...trap/engines/engine.py.230: loss: 0.18454 [I 2022-10-26 16:19:41] ...optimizers/trijoint.py.151: total_norm: 3.135095 [I 2022-10-26 16:19:46] ...trap/engines/engine.py.224: train: epoch 18 | batch 300/2304 [I 2022-10-26 16:19:46] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:16:29 [I 2022-10-26 16:19:46] ...trap/engines/engine.py.229: process: 0.49198 | load: 0.00248 [I 2022-10-26 16:19:46] ...trap/engines/engine.py.230: loss: 0.12675 [I 2022-10-26 16:19:46] ...optimizers/trijoint.py.151: total_norm: 1.763114 [I 2022-10-26 16:19:51] ...trap/engines/engine.py.224: train: epoch 18 | batch 310/2304 [I 2022-10-26 16:19:51] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:16:56 [I 2022-10-26 16:19:51] ...trap/engines/engine.py.229: process: 0.52745 | load: 0.00246 [I 2022-10-26 16:19:51] ...trap/engines/engine.py.230: loss: 0.18580 [I 2022-10-26 16:19:51] ...optimizers/trijoint.py.151: total_norm: 2.809653 [I 2022-10-26 16:20:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 320/2304 [I 2022-10-26 16:20:18] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 1:15:37 [I 2022-10-26 16:20:18] ...trap/engines/engine.py.229: process: 0.48784 | load: 0.00239 [I 2022-10-26 16:20:18] ...trap/engines/engine.py.230: loss: 0.25275 [I 2022-10-26 16:20:18] ...optimizers/trijoint.py.151: total_norm: 3.706275 [I 2022-10-26 16:20:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 330/2304 [I 2022-10-26 16:20:23] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:22:39 [I 2022-10-26 16:20:23] ...trap/engines/engine.py.229: process: 0.49290 | load: 0.00240 [I 2022-10-26 16:20:23] ...trap/engines/engine.py.230: loss: 0.18987 [I 2022-10-26 16:20:23] ...optimizers/trijoint.py.151: total_norm: 3.526511 [I 2022-10-26 16:20:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 340/2304 [I 2022-10-26 16:20:28] ...trap/engines/engine.py.228: elapsed: 0:03:15 | left: 0:17:03 [I 2022-10-26 16:20:28] ...trap/engines/engine.py.229: process: 0.49421 | load: 0.00250 [I 2022-10-26 16:20:28] ...trap/engines/engine.py.230: loss: 0.16042 [I 2022-10-26 16:20:28] ...optimizers/trijoint.py.151: total_norm: 2.307211 [I 2022-10-26 16:20:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 350/2304 [I 2022-10-26 16:20:33] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:16:18 [I 2022-10-26 16:20:33] ...trap/engines/engine.py.229: process: 0.50212 | load: 0.00247 [I 2022-10-26 16:20:33] ...trap/engines/engine.py.230: loss: 0.18289 [I 2022-10-26 16:20:33] ...optimizers/trijoint.py.151: total_norm: 3.511415 [I 2022-10-26 16:20:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 360/2304 [I 2022-10-26 16:20:38] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:16:14 [I 2022-10-26 16:20:38] ...trap/engines/engine.py.229: process: 0.50763 | load: 0.00248 [I 2022-10-26 16:20:38] ...trap/engines/engine.py.230: loss: 0.21340 [I 2022-10-26 16:20:38] ...optimizers/trijoint.py.151: total_norm: 3.625436 [I 2022-10-26 16:20:43] ...trap/engines/engine.py.224: train: epoch 18 | batch 370/2304 [I 2022-10-26 16:20:43] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:16:11 [I 2022-10-26 16:20:43] ...trap/engines/engine.py.229: process: 0.50658 | load: 0.00236 [I 2022-10-26 16:20:43] ...trap/engines/engine.py.230: loss: 0.28279 [I 2022-10-26 16:20:43] ...optimizers/trijoint.py.151: total_norm: 3.481060 [I 2022-10-26 16:20:48] ...trap/engines/engine.py.224: train: epoch 18 | batch 380/2304 [I 2022-10-26 16:20:48] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:16:00 [I 2022-10-26 16:20:48] ...trap/engines/engine.py.229: process: 0.49974 | load: 0.00245 [I 2022-10-26 16:20:48] ...trap/engines/engine.py.230: loss: 0.23016 [I 2022-10-26 16:20:48] ...optimizers/trijoint.py.151: total_norm: 3.743501 [I 2022-10-26 16:20:53] ...trap/engines/engine.py.224: train: epoch 18 | batch 390/2304 [I 2022-10-26 16:20:53] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:16:02 [I 2022-10-26 16:20:53] ...trap/engines/engine.py.229: process: 0.50933 | load: 0.00252 [I 2022-10-26 16:20:53] ...trap/engines/engine.py.230: loss: 0.19971 [I 2022-10-26 16:20:53] ...optimizers/trijoint.py.151: total_norm: 3.612877 [I 2022-10-26 16:20:58] ...trap/engines/engine.py.224: train: epoch 18 | batch 400/2304 [I 2022-10-26 16:20:58] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:15:46 [I 2022-10-26 16:20:58] ...trap/engines/engine.py.229: process: 0.49274 | load: 0.00242 [I 2022-10-26 16:20:58] ...trap/engines/engine.py.230: loss: 0.21594 [I 2022-10-26 16:20:58] ...optimizers/trijoint.py.151: total_norm: 3.875583 [I 2022-10-26 16:21:03] ...trap/engines/engine.py.224: train: epoch 18 | batch 410/2304 [I 2022-10-26 16:21:03] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:15:58 [I 2022-10-26 16:21:03] ...trap/engines/engine.py.229: process: 0.51494 | load: 0.00246 [I 2022-10-26 16:21:03] ...trap/engines/engine.py.230: loss: 0.21225 [I 2022-10-26 16:21:03] ...optimizers/trijoint.py.151: total_norm: 4.481201 [I 2022-10-26 16:21:08] ...trap/engines/engine.py.224: train: epoch 18 | batch 420/2304 [I 2022-10-26 16:21:08] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:15:43 [I 2022-10-26 16:21:08] ...trap/engines/engine.py.229: process: 0.48864 | load: 0.00243 [I 2022-10-26 16:21:08] ...trap/engines/engine.py.230: loss: 0.15986 [I 2022-10-26 16:21:08] ...optimizers/trijoint.py.151: total_norm: 3.052601 [I 2022-10-26 16:21:13] ...trap/engines/engine.py.224: train: epoch 18 | batch 430/2304 [I 2022-10-26 16:21:13] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:15:36 [I 2022-10-26 16:21:13] ...trap/engines/engine.py.229: process: 0.50690 | load: 0.00240 [I 2022-10-26 16:21:13] ...trap/engines/engine.py.230: loss: 0.20010 [I 2022-10-26 16:21:13] ...optimizers/trijoint.py.151: total_norm: 2.125717 [I 2022-10-26 16:21:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 440/2304 [I 2022-10-26 16:21:18] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:15:34 [I 2022-10-26 16:21:18] ...trap/engines/engine.py.229: process: 0.50134 | load: 0.00249 [I 2022-10-26 16:21:18] ...trap/engines/engine.py.230: loss: 0.12540 [I 2022-10-26 16:21:18] ...optimizers/trijoint.py.151: total_norm: 1.973313 [I 2022-10-26 16:21:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 450/2304 [I 2022-10-26 16:21:23] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:15:32 [I 2022-10-26 16:21:23] ...trap/engines/engine.py.229: process: 0.50069 | load: 0.00241 [I 2022-10-26 16:21:23] ...trap/engines/engine.py.230: loss: 0.33551 [I 2022-10-26 16:21:23] ...optimizers/trijoint.py.151: total_norm: 4.547215 [I 2022-10-26 16:21:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 460/2304 [I 2022-10-26 16:21:28] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:15:23 [I 2022-10-26 16:21:28] ...trap/engines/engine.py.229: process: 0.49687 | load: 0.00248 [I 2022-10-26 16:21:28] ...trap/engines/engine.py.230: loss: 0.15121 [I 2022-10-26 16:21:28] ...optimizers/trijoint.py.151: total_norm: 2.073803 [I 2022-10-26 16:21:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 470/2304 [I 2022-10-26 16:21:33] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:15:15 [I 2022-10-26 16:21:33] ...trap/engines/engine.py.229: process: 0.49107 | load: 0.00246 [I 2022-10-26 16:21:33] ...trap/engines/engine.py.230: loss: 0.15468 [I 2022-10-26 16:21:33] ...optimizers/trijoint.py.151: total_norm: 3.172391 [I 2022-10-26 16:21:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 480/2304 [I 2022-10-26 16:21:38] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:15:13 [I 2022-10-26 16:21:38] ...trap/engines/engine.py.229: process: 0.51004 | load: 0.00243 [I 2022-10-26 16:21:38] ...trap/engines/engine.py.230: loss: 0.21050 [I 2022-10-26 16:21:38] ...optimizers/trijoint.py.151: total_norm: 2.940150 [I 2022-10-26 16:21:43] ...trap/engines/engine.py.224: train: epoch 18 | batch 490/2304 [I 2022-10-26 16:21:43] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:15:07 [I 2022-10-26 16:21:43] ...trap/engines/engine.py.229: process: 0.49540 | load: 0.00248 [I 2022-10-26 16:21:43] ...trap/engines/engine.py.230: loss: 0.15356 [I 2022-10-26 16:21:43] ...optimizers/trijoint.py.151: total_norm: 2.795631 [I 2022-10-26 16:21:48] ...trap/engines/engine.py.224: train: epoch 18 | batch 500/2304 [I 2022-10-26 16:21:48] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:15:12 [I 2022-10-26 16:21:48] ...trap/engines/engine.py.229: process: 0.50713 | load: 0.00307 [I 2022-10-26 16:21:48] ...trap/engines/engine.py.230: loss: 0.25380 [I 2022-10-26 16:21:48] ...optimizers/trijoint.py.151: total_norm: 4.163446 [I 2022-10-26 16:21:53] ...trap/engines/engine.py.224: train: epoch 18 | batch 510/2304 [I 2022-10-26 16:21:53] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:14:59 [I 2022-10-26 16:21:53] ...trap/engines/engine.py.229: process: 0.49711 | load: 0.00241 [I 2022-10-26 16:21:53] ...trap/engines/engine.py.230: loss: 0.13491 [I 2022-10-26 16:21:53] ...optimizers/trijoint.py.151: total_norm: 3.081034 [I 2022-10-26 16:21:58] ...trap/engines/engine.py.224: train: epoch 18 | batch 520/2304 [I 2022-10-26 16:21:58] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:14:53 [I 2022-10-26 16:21:58] ...trap/engines/engine.py.229: process: 0.50145 | load: 0.00240 [I 2022-10-26 16:21:58] ...trap/engines/engine.py.230: loss: 0.15887 [I 2022-10-26 16:21:58] ...optimizers/trijoint.py.151: total_norm: 2.133233 [I 2022-10-26 16:22:03] ...trap/engines/engine.py.224: train: epoch 18 | batch 530/2304 [I 2022-10-26 16:22:03] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:14:45 [I 2022-10-26 16:22:03] ...trap/engines/engine.py.229: process: 0.49729 | load: 0.00243 [I 2022-10-26 16:22:03] ...trap/engines/engine.py.230: loss: 0.14895 [I 2022-10-26 16:22:03] ...optimizers/trijoint.py.151: total_norm: 2.155060 [I 2022-10-26 16:22:08] ...trap/engines/engine.py.224: train: epoch 18 | batch 540/2304 [I 2022-10-26 16:22:08] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:14:36 [I 2022-10-26 16:22:08] ...trap/engines/engine.py.229: process: 0.49522 | load: 0.00245 [I 2022-10-26 16:22:08] ...trap/engines/engine.py.230: loss: 0.19177 [I 2022-10-26 16:22:08] ...optimizers/trijoint.py.151: total_norm: 3.402667 [I 2022-10-26 16:22:13] ...trap/engines/engine.py.224: train: epoch 18 | batch 550/2304 [I 2022-10-26 16:22:13] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:14:32 [I 2022-10-26 16:22:13] ...trap/engines/engine.py.229: process: 0.50825 | load: 0.00246 [I 2022-10-26 16:22:13] ...trap/engines/engine.py.230: loss: 0.19924 [I 2022-10-26 16:22:13] ...optimizers/trijoint.py.151: total_norm: 3.305279 [I 2022-10-26 16:22:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 560/2304 [I 2022-10-26 16:22:18] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:14:36 [I 2022-10-26 16:22:18] ...trap/engines/engine.py.229: process: 0.50187 | load: 0.00241 [I 2022-10-26 16:22:18] ...trap/engines/engine.py.230: loss: 0.16463 [I 2022-10-26 16:22:18] ...optimizers/trijoint.py.151: total_norm: 2.252099 [I 2022-10-26 16:22:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 570/2304 [I 2022-10-26 16:22:23] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:14:24 [I 2022-10-26 16:22:23] ...trap/engines/engine.py.229: process: 0.49962 | load: 0.00243 [I 2022-10-26 16:22:23] ...trap/engines/engine.py.230: loss: 0.11988 [I 2022-10-26 16:22:23] ...optimizers/trijoint.py.151: total_norm: 2.038985 [I 2022-10-26 16:22:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 580/2304 [I 2022-10-26 16:22:28] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:14:22 [I 2022-10-26 16:22:28] ...trap/engines/engine.py.229: process: 0.49574 | load: 0.00252 [I 2022-10-26 16:22:28] ...trap/engines/engine.py.230: loss: 0.20155 [I 2022-10-26 16:22:28] ...optimizers/trijoint.py.151: total_norm: 2.899834 [I 2022-10-26 16:22:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 590/2304 [I 2022-10-26 16:22:33] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:14:10 [I 2022-10-26 16:22:33] ...trap/engines/engine.py.229: process: 0.48454 | load: 0.00243 [I 2022-10-26 16:22:33] ...trap/engines/engine.py.230: loss: 0.22793 [I 2022-10-26 16:22:33] ...optimizers/trijoint.py.151: total_norm: 3.446134 [I 2022-10-26 16:22:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 600/2304 [I 2022-10-26 16:22:38] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:14:05 [I 2022-10-26 16:22:38] ...trap/engines/engine.py.229: process: 0.49106 | load: 0.00245 [I 2022-10-26 16:22:38] ...trap/engines/engine.py.230: loss: 0.19318 [I 2022-10-26 16:22:38] ...optimizers/trijoint.py.151: total_norm: 2.919905 [I 2022-10-26 16:22:43] ...trap/engines/engine.py.224: train: epoch 18 | batch 610/2304 [I 2022-10-26 16:22:43] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:14:16 [I 2022-10-26 16:22:43] ...trap/engines/engine.py.229: process: 0.52773 | load: 0.00243 [I 2022-10-26 16:22:43] ...trap/engines/engine.py.230: loss: 0.21594 [I 2022-10-26 16:22:43] ...optimizers/trijoint.py.151: total_norm: 2.860227 [I 2022-10-26 16:22:48] ...trap/engines/engine.py.224: train: epoch 18 | batch 620/2304 [I 2022-10-26 16:22:48] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:13:58 [I 2022-10-26 16:22:48] ...trap/engines/engine.py.229: process: 0.49303 | load: 0.00269 [I 2022-10-26 16:22:48] ...trap/engines/engine.py.230: loss: 0.20531 [I 2022-10-26 16:22:48] ...optimizers/trijoint.py.151: total_norm: 4.000355 [I 2022-10-26 16:22:53] ...trap/engines/engine.py.224: train: epoch 18 | batch 630/2304 [I 2022-10-26 16:22:53] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:13:49 [I 2022-10-26 16:22:53] ...trap/engines/engine.py.229: process: 0.48742 | load: 0.00252 [I 2022-10-26 16:22:53] ...trap/engines/engine.py.230: loss: 0.19507 [I 2022-10-26 16:22:53] ...optimizers/trijoint.py.151: total_norm: 3.039433 [I 2022-10-26 16:22:58] ...trap/engines/engine.py.224: train: epoch 18 | batch 640/2304 [I 2022-10-26 16:22:58] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:13:51 [I 2022-10-26 16:22:58] ...trap/engines/engine.py.229: process: 0.50667 | load: 0.00255 [I 2022-10-26 16:22:58] ...trap/engines/engine.py.230: loss: 0.23686 [I 2022-10-26 16:22:58] ...optimizers/trijoint.py.151: total_norm: 4.119932 [I 2022-10-26 16:23:03] ...trap/engines/engine.py.224: train: epoch 18 | batch 650/2304 [I 2022-10-26 16:23:03] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:13:49 [I 2022-10-26 16:23:03] ...trap/engines/engine.py.229: process: 0.49993 | load: 0.00250 [I 2022-10-26 16:23:03] ...trap/engines/engine.py.230: loss: 0.19494 [I 2022-10-26 16:23:03] ...optimizers/trijoint.py.151: total_norm: 2.927743 [I 2022-10-26 16:23:08] ...trap/engines/engine.py.224: train: epoch 18 | batch 660/2304 [I 2022-10-26 16:23:08] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:13:42 [I 2022-10-26 16:23:08] ...trap/engines/engine.py.229: process: 0.50054 | load: 0.00556 [I 2022-10-26 16:23:08] ...trap/engines/engine.py.230: loss: 0.19244 [I 2022-10-26 16:23:08] ...optimizers/trijoint.py.151: total_norm: 3.100436 [I 2022-10-26 16:23:13] ...trap/engines/engine.py.224: train: epoch 18 | batch 670/2304 [I 2022-10-26 16:23:13] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:13:33 [I 2022-10-26 16:23:13] ...trap/engines/engine.py.229: process: 0.49554 | load: 0.00253 [I 2022-10-26 16:23:13] ...trap/engines/engine.py.230: loss: 0.16090 [I 2022-10-26 16:23:13] ...optimizers/trijoint.py.151: total_norm: 3.994391 [I 2022-10-26 16:23:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 680/2304 [I 2022-10-26 16:23:18] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:13:32 [I 2022-10-26 16:23:18] ...trap/engines/engine.py.229: process: 0.49600 | load: 0.00247 [I 2022-10-26 16:23:18] ...trap/engines/engine.py.230: loss: 0.23279 [I 2022-10-26 16:23:18] ...optimizers/trijoint.py.151: total_norm: 3.663318 [I 2022-10-26 16:23:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 690/2304 [I 2022-10-26 16:23:23] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:13:23 [I 2022-10-26 16:23:23] ...trap/engines/engine.py.229: process: 0.50406 | load: 0.00247 [I 2022-10-26 16:23:23] ...trap/engines/engine.py.230: loss: 0.16284 [I 2022-10-26 16:23:23] ...optimizers/trijoint.py.151: total_norm: 3.479568 [I 2022-10-26 16:23:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 700/2304 [I 2022-10-26 16:23:28] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:13:27 [I 2022-10-26 16:23:28] ...trap/engines/engine.py.229: process: 0.51559 | load: 0.00250 [I 2022-10-26 16:23:28] ...trap/engines/engine.py.230: loss: 0.22797 [I 2022-10-26 16:23:28] ...optimizers/trijoint.py.151: total_norm: 3.304517 [I 2022-10-26 16:23:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 710/2304 [I 2022-10-26 16:23:33] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:13:23 [I 2022-10-26 16:23:33] ...trap/engines/engine.py.229: process: 0.50188 | load: 0.00330 [I 2022-10-26 16:23:33] ...trap/engines/engine.py.230: loss: 0.14454 [I 2022-10-26 16:23:33] ...optimizers/trijoint.py.151: total_norm: 2.252166 [I 2022-10-26 16:23:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 720/2304 [I 2022-10-26 16:23:38] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:13:07 [I 2022-10-26 16:23:38] ...trap/engines/engine.py.229: process: 0.48850 | load: 0.00279 [I 2022-10-26 16:23:38] ...trap/engines/engine.py.230: loss: 0.17358 [I 2022-10-26 16:23:38] ...optimizers/trijoint.py.151: total_norm: 2.595605 [I 2022-10-26 16:23:43] ...trap/engines/engine.py.224: train: epoch 18 | batch 730/2304 [I 2022-10-26 16:23:43] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:13:07 [I 2022-10-26 16:23:43] ...trap/engines/engine.py.229: process: 0.50484 | load: 0.00265 [I 2022-10-26 16:23:43] ...trap/engines/engine.py.230: loss: 0.20560 [I 2022-10-26 16:23:43] ...optimizers/trijoint.py.151: total_norm: 4.255344 [I 2022-10-26 16:23:48] ...trap/engines/engine.py.224: train: epoch 18 | batch 740/2304 [I 2022-10-26 16:23:48] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:12:55 [I 2022-10-26 16:23:48] ...trap/engines/engine.py.229: process: 0.49511 | load: 0.00246 [I 2022-10-26 16:23:48] ...trap/engines/engine.py.230: loss: 0.26534 [I 2022-10-26 16:23:48] ...optimizers/trijoint.py.151: total_norm: 3.312714 [I 2022-10-26 16:23:53] ...trap/engines/engine.py.224: train: epoch 18 | batch 750/2304 [I 2022-10-26 16:23:53] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:12:56 [I 2022-10-26 16:23:53] ...trap/engines/engine.py.229: process: 0.50397 | load: 0.00245 [I 2022-10-26 16:23:53] ...trap/engines/engine.py.230: loss: 0.19134 [I 2022-10-26 16:23:53] ...optimizers/trijoint.py.151: total_norm: 3.128810 [I 2022-10-26 16:23:58] ...trap/engines/engine.py.224: train: epoch 18 | batch 760/2304 [I 2022-10-26 16:23:58] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:12:54 [I 2022-10-26 16:23:58] ...trap/engines/engine.py.229: process: 0.50549 | load: 0.00243 [I 2022-10-26 16:23:58] ...trap/engines/engine.py.230: loss: 0.19318 [I 2022-10-26 16:23:58] ...optimizers/trijoint.py.151: total_norm: 2.429086 [I 2022-10-26 16:24:03] ...trap/engines/engine.py.224: train: epoch 18 | batch 770/2304 [I 2022-10-26 16:24:03] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:12:47 [I 2022-10-26 16:24:03] ...trap/engines/engine.py.229: process: 0.50701 | load: 0.00250 [I 2022-10-26 16:24:03] ...trap/engines/engine.py.230: loss: 0.14110 [I 2022-10-26 16:24:03] ...optimizers/trijoint.py.151: total_norm: 2.199957 [I 2022-10-26 16:24:08] ...trap/engines/engine.py.224: train: epoch 18 | batch 780/2304 [I 2022-10-26 16:24:08] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:12:47 [I 2022-10-26 16:24:08] ...trap/engines/engine.py.229: process: 0.52107 | load: 0.00254 [I 2022-10-26 16:24:08] ...trap/engines/engine.py.230: loss: 0.20531 [I 2022-10-26 16:24:08] ...optimizers/trijoint.py.151: total_norm: 3.484347 [I 2022-10-26 16:24:13] ...trap/engines/engine.py.224: train: epoch 18 | batch 790/2304 [I 2022-10-26 16:24:13] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:12:34 [I 2022-10-26 16:24:13] ...trap/engines/engine.py.229: process: 0.50379 | load: 0.00249 [I 2022-10-26 16:24:13] ...trap/engines/engine.py.230: loss: 0.31056 [I 2022-10-26 16:24:13] ...optimizers/trijoint.py.151: total_norm: 4.333751 [I 2022-10-26 16:24:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 800/2304 [I 2022-10-26 16:24:18] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:12:33 [I 2022-10-26 16:24:18] ...trap/engines/engine.py.229: process: 0.50042 | load: 0.00245 [I 2022-10-26 16:24:18] ...trap/engines/engine.py.230: loss: 0.23359 [I 2022-10-26 16:24:18] ...optimizers/trijoint.py.151: total_norm: 3.995788 [I 2022-10-26 16:24:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 810/2304 [I 2022-10-26 16:24:23] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:12:27 [I 2022-10-26 16:24:23] ...trap/engines/engine.py.229: process: 0.49398 | load: 0.00250 [I 2022-10-26 16:24:23] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-26 16:24:23] ...optimizers/trijoint.py.151: total_norm: 2.659251 [I 2022-10-26 16:24:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 820/2304 [I 2022-10-26 16:24:28] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:12:19 [I 2022-10-26 16:24:28] ...trap/engines/engine.py.229: process: 0.50034 | load: 0.00242 [I 2022-10-26 16:24:28] ...trap/engines/engine.py.230: loss: 0.19811 [I 2022-10-26 16:24:28] ...optimizers/trijoint.py.151: total_norm: 4.043811 [I 2022-10-26 16:24:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 830/2304 [I 2022-10-26 16:24:33] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:12:13 [I 2022-10-26 16:24:33] ...trap/engines/engine.py.229: process: 0.50365 | load: 0.00243 [I 2022-10-26 16:24:33] ...trap/engines/engine.py.230: loss: 0.17091 [I 2022-10-26 16:24:33] ...optimizers/trijoint.py.151: total_norm: 3.772716 [I 2022-10-26 16:24:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 840/2304 [I 2022-10-26 16:24:38] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:12:11 [I 2022-10-26 16:24:38] ...trap/engines/engine.py.229: process: 0.50057 | load: 0.00264 [I 2022-10-26 16:24:38] ...trap/engines/engine.py.230: loss: 0.18220 [I 2022-10-26 16:24:38] ...optimizers/trijoint.py.151: total_norm: 2.590772 [I 2022-10-26 16:24:43] ...trap/engines/engine.py.224: train: epoch 18 | batch 850/2304 [I 2022-10-26 16:24:43] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:12:10 [I 2022-10-26 16:24:43] ...trap/engines/engine.py.229: process: 0.50407 | load: 0.00253 [I 2022-10-26 16:24:43] ...trap/engines/engine.py.230: loss: 0.26602 [I 2022-10-26 16:24:43] ...optimizers/trijoint.py.151: total_norm: 4.374533 [I 2022-10-26 16:24:48] ...trap/engines/engine.py.224: train: epoch 18 | batch 860/2304 [I 2022-10-26 16:24:48] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:11:57 [I 2022-10-26 16:24:48] ...trap/engines/engine.py.229: process: 0.49264 | load: 0.00248 [I 2022-10-26 16:24:48] ...trap/engines/engine.py.230: loss: 0.13730 [I 2022-10-26 16:24:48] ...optimizers/trijoint.py.151: total_norm: 2.029318 [I 2022-10-26 16:24:53] ...trap/engines/engine.py.224: train: epoch 18 | batch 870/2304 [I 2022-10-26 16:24:53] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:11:59 [I 2022-10-26 16:24:53] ...trap/engines/engine.py.229: process: 0.50832 | load: 0.00247 [I 2022-10-26 16:24:53] ...trap/engines/engine.py.230: loss: 0.22403 [I 2022-10-26 16:24:53] ...optimizers/trijoint.py.151: total_norm: 3.973187 [I 2022-10-26 16:24:58] ...trap/engines/engine.py.224: train: epoch 18 | batch 880/2304 [I 2022-10-26 16:24:58] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:11:46 [I 2022-10-26 16:24:58] ...trap/engines/engine.py.229: process: 0.48831 | load: 0.00248 [I 2022-10-26 16:24:58] ...trap/engines/engine.py.230: loss: 0.17327 [I 2022-10-26 16:24:58] ...optimizers/trijoint.py.151: total_norm: 3.706205 [I 2022-10-26 16:25:03] ...trap/engines/engine.py.224: train: epoch 18 | batch 890/2304 [I 2022-10-26 16:25:03] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:11:49 [I 2022-10-26 16:25:03] ...trap/engines/engine.py.229: process: 0.50393 | load: 0.00245 [I 2022-10-26 16:25:03] ...trap/engines/engine.py.230: loss: 0.25416 [I 2022-10-26 16:25:03] ...optimizers/trijoint.py.151: total_norm: 3.425396 [I 2022-10-26 16:25:08] ...trap/engines/engine.py.224: train: epoch 18 | batch 900/2304 [I 2022-10-26 16:25:08] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:11:38 [I 2022-10-26 16:25:08] ...trap/engines/engine.py.229: process: 0.49634 | load: 0.00245 [I 2022-10-26 16:25:08] ...trap/engines/engine.py.230: loss: 0.21036 [I 2022-10-26 16:25:08] ...optimizers/trijoint.py.151: total_norm: 4.194667 [I 2022-10-26 16:25:13] ...trap/engines/engine.py.224: train: epoch 18 | batch 910/2304 [I 2022-10-26 16:25:13] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:11:36 [I 2022-10-26 16:25:13] ...trap/engines/engine.py.229: process: 0.50759 | load: 0.00245 [I 2022-10-26 16:25:13] ...trap/engines/engine.py.230: loss: 0.17983 [I 2022-10-26 16:25:13] ...optimizers/trijoint.py.151: total_norm: 2.625435 [I 2022-10-26 16:25:18] ...trap/engines/engine.py.224: train: epoch 18 | batch 920/2304 [I 2022-10-26 16:25:18] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:11:30 [I 2022-10-26 16:25:18] ...trap/engines/engine.py.229: process: 0.50050 | load: 0.00245 [I 2022-10-26 16:25:18] ...trap/engines/engine.py.230: loss: 0.20821 [I 2022-10-26 16:25:18] ...optimizers/trijoint.py.151: total_norm: 2.789619 [I 2022-10-26 16:25:23] ...trap/engines/engine.py.224: train: epoch 18 | batch 930/2304 [I 2022-10-26 16:25:23] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:11:34 [I 2022-10-26 16:25:23] ...trap/engines/engine.py.229: process: 0.51037 | load: 0.00244 [I 2022-10-26 16:25:23] ...trap/engines/engine.py.230: loss: 0.17069 [I 2022-10-26 16:25:23] ...optimizers/trijoint.py.151: total_norm: 3.099156 [I 2022-10-26 16:25:28] ...trap/engines/engine.py.224: train: epoch 18 | batch 940/2304 [I 2022-10-26 16:25:28] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:11:24 [I 2022-10-26 16:25:28] ...trap/engines/engine.py.229: process: 0.50324 | load: 0.00244 [I 2022-10-26 16:25:28] ...trap/engines/engine.py.230: loss: 0.16981 [I 2022-10-26 16:25:28] ...optimizers/trijoint.py.151: total_norm: 2.865047 [I 2022-10-26 16:25:33] ...trap/engines/engine.py.224: train: epoch 18 | batch 950/2304 [I 2022-10-26 16:25:33] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:11:26 [I 2022-10-26 16:25:33] ...trap/engines/engine.py.229: process: 0.50320 | load: 0.00246 [I 2022-10-26 16:25:33] ...trap/engines/engine.py.230: loss: 0.19230 [I 2022-10-26 16:25:33] ...optimizers/trijoint.py.151: total_norm: 3.531689 [I 2022-10-26 16:25:38] ...trap/engines/engine.py.224: train: epoch 18 | batch 960/2304 [I 2022-10-26 16:25:38] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:11:10 [I 2022-10-26 16:25:38] ...trap/engines/engine.py.229: process: 0.50383 | load: 0.00244 [I 2022-10-26 16:25:38] ...trap/engines/engine.py.230: loss: 0.29791 [I 2022-10-26 16:25:38] ...optimizers/trijoint.py.151: total_norm: 4.749566 [I 2022-10-26 16:25:43] ...trap/engines/engine.py.224: train: epoch 18 | batch 970/2304 [I 2022-10-26 16:25:43] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:11:04 [I 2022-10-26 16:25:43] ...trap/engines/engine.py.229: process: 0.49731 | load: 0.00243 [I 2022-10-26 16:25:43] ...trap/engines/engine.py.230: loss: 0.21807 [I 2022-10-26 16:25:43] ...optimizers/trijoint.py.151: total_norm: 4.389547 [I 2022-10-26 16:25:48] ...trap/engines/engine.py.224: train: epoch 18 | batch 980/2304 [I 2022-10-26 16:25:48] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:10:58 [I 2022-10-26 16:25:48] ...trap/engines/engine.py.229: process: 0.49312 | load: 0.00243 [I 2022-10-26 16:25:48] ...trap/engines/engine.py.230: loss: 0.15281 [I 2022-10-26 16:25:48] ...optimizers/trijoint.py.151: total_norm: 2.185939 [I 2022-10-26 16:25:53] ...trap/engines/engine.py.224: train: epoch 18 | batch 990/2304 [I 2022-10-26 16:25:53] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:10:55 [I 2022-10-26 16:25:53] ...trap/engines/engine.py.229: process: 0.50160 | load: 0.00243 [I 2022-10-26 16:25:53] ...trap/engines/engine.py.230: loss: 0.19435 [I 2022-10-26 16:25:53] ...optimizers/trijoint.py.151: total_norm: 3.388911 [I 2022-10-26 16:25:58] ...trap/engines/engine.py.224: train: epoch 18 | batch 1000/2304 [I 2022-10-26 16:25:58] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:10:56 [I 2022-10-26 16:25:58] ...trap/engines/engine.py.229: process: 0.50354 | load: 0.00254 [I 2022-10-26 16:25:58] ...trap/engines/engine.py.230: loss: 0.15553 [I 2022-10-26 16:25:58] ...optimizers/trijoint.py.151: total_norm: 2.810374 [I 2022-10-26 16:26:03] ...trap/engines/engine.py.224: train: epoch 18 | batch 1010/2304 [I 2022-10-26 16:26:03] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:10:43 [I 2022-10-26 16:26:03] ...trap/engines/engine.py.229: process: 0.49406 | load: 0.00251 [I 2022-10-26 16:26:03] ...trap/engines/engine.py.230: loss: 0.20666 [I 2022-10-26 16:26:03] ...optimizers/trijoint.py.151: total_norm: 3.141094 [I 2022-10-26 16:26:08] ...trap/engines/engine.py.224: train: epoch 18 | batch 1020/2304 [I 2022-10-26 16:26:08] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:10:43 [I 2022-10-26 16:26:08] ...trap/engines/engine.py.229: process: 0.49905 | load: 0.00252 [I 2022-10-26 16:26:08] ...trap/engines/engine.py.230: loss: 0.22224 [I 2022-10-26 16:26:08] ...optimizers/trijoint.py.151: total_norm: 4.217423 [I 2022-10-26 16:26:14] ...trap/engines/engine.py.224: train: epoch 18 | batch 1030/2304 [I 2022-10-26 16:26:14] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:10:42 [I 2022-10-26 16:26:14] ...trap/engines/engine.py.229: process: 0.49063 | load: 0.00249 [I 2022-10-26 16:26:14] ...trap/engines/engine.py.230: loss: 0.16759 [I 2022-10-26 16:26:14] ...optimizers/trijoint.py.151: total_norm: 3.008516 [I 2022-10-26 16:26:19] ...trap/engines/engine.py.224: train: epoch 18 | batch 1040/2304 [I 2022-10-26 16:26:19] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:10:31 [I 2022-10-26 16:26:19] ...trap/engines/engine.py.229: process: 0.49384 | load: 0.00247 [I 2022-10-26 16:26:19] ...trap/engines/engine.py.230: loss: 0.17049 [I 2022-10-26 16:26:19] ...optimizers/trijoint.py.151: total_norm: 2.862951 [I 2022-10-26 16:26:24] ...trap/engines/engine.py.224: train: epoch 18 | batch 1050/2304 [I 2022-10-26 16:26:24] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:10:36 [I 2022-10-26 16:26:24] ...trap/engines/engine.py.229: process: 0.52434 | load: 0.00246 [I 2022-10-26 16:26:24] ...trap/engines/engine.py.230: loss: 0.17497 [I 2022-10-26 16:26:24] ...optimizers/trijoint.py.151: total_norm: 2.839125 [I 2022-10-26 16:26:29] ...trap/engines/engine.py.224: train: epoch 18 | batch 1060/2304 [I 2022-10-26 16:26:29] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:10:27 [I 2022-10-26 16:26:29] ...trap/engines/engine.py.229: process: 0.50072 | load: 0.00246 [I 2022-10-26 16:26:29] ...trap/engines/engine.py.230: loss: 0.10625 [I 2022-10-26 16:26:29] ...optimizers/trijoint.py.151: total_norm: 2.636039 [I 2022-10-26 16:26:34] ...trap/engines/engine.py.224: train: epoch 18 | batch 1070/2304 [I 2022-10-26 16:26:34] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:10:18 [I 2022-10-26 16:26:34] ...trap/engines/engine.py.229: process: 0.51067 | load: 0.00244 [I 2022-10-26 16:26:34] ...trap/engines/engine.py.230: loss: 0.23433 [I 2022-10-26 16:26:34] ...optimizers/trijoint.py.151: total_norm: 3.608120 [I 2022-10-26 16:26:39] ...trap/engines/engine.py.224: train: epoch 18 | batch 1080/2304 [I 2022-10-26 16:26:39] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:10:12 [I 2022-10-26 16:26:39] ...trap/engines/engine.py.229: process: 0.49644 | load: 0.00242 [I 2022-10-26 16:26:39] ...trap/engines/engine.py.230: loss: 0.20167 [I 2022-10-26 16:26:39] ...optimizers/trijoint.py.151: total_norm: 3.060419 [I 2022-10-26 16:26:44] ...trap/engines/engine.py.224: train: epoch 18 | batch 1090/2304 [I 2022-10-26 16:26:44] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:10:05 [I 2022-10-26 16:26:44] ...trap/engines/engine.py.229: process: 0.49930 | load: 0.00260 [I 2022-10-26 16:26:44] ...trap/engines/engine.py.230: loss: 0.16138 [I 2022-10-26 16:26:44] ...optimizers/trijoint.py.151: total_norm: 2.902334 [I 2022-10-26 16:26:49] ...trap/engines/engine.py.224: train: epoch 18 | batch 1100/2304 [I 2022-10-26 16:26:49] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:10:06 [I 2022-10-26 16:26:49] ...trap/engines/engine.py.229: process: 0.50553 | load: 0.00242 [I 2022-10-26 16:26:49] ...trap/engines/engine.py.230: loss: 0.14561 [I 2022-10-26 16:26:49] ...optimizers/trijoint.py.151: total_norm: 2.659591 [I 2022-10-26 16:26:54] ...trap/engines/engine.py.224: train: epoch 18 | batch 1110/2304 [I 2022-10-26 16:26:54] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:09:56 [I 2022-10-26 16:26:54] ...trap/engines/engine.py.229: process: 0.50626 | load: 0.00243 [I 2022-10-26 16:26:54] ...trap/engines/engine.py.230: loss: 0.20599 [I 2022-10-26 16:26:54] ...optimizers/trijoint.py.151: total_norm: 3.157385 [I 2022-10-26 16:26:59] ...trap/engines/engine.py.224: train: epoch 18 | batch 1120/2304 [I 2022-10-26 16:26:59] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:09:49 [I 2022-10-26 16:26:59] ...trap/engines/engine.py.229: process: 0.49685 | load: 0.00238 [I 2022-10-26 16:26:59] ...trap/engines/engine.py.230: loss: 0.18236 [I 2022-10-26 16:26:59] ...optimizers/trijoint.py.151: total_norm: 3.041982 [I 2022-10-26 16:27:04] ...trap/engines/engine.py.224: train: epoch 18 | batch 1130/2304 [I 2022-10-26 16:27:04] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:09:47 [I 2022-10-26 16:27:04] ...trap/engines/engine.py.229: process: 0.50287 | load: 0.00255 [I 2022-10-26 16:27:04] ...trap/engines/engine.py.230: loss: 0.13916 [I 2022-10-26 16:27:04] ...optimizers/trijoint.py.151: total_norm: 2.778631 [I 2022-10-26 16:27:09] ...trap/engines/engine.py.224: train: epoch 18 | batch 1140/2304 [I 2022-10-26 16:27:09] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:09:40 [I 2022-10-26 16:27:09] ...trap/engines/engine.py.229: process: 0.49089 | load: 0.00249 [I 2022-10-26 16:27:09] ...trap/engines/engine.py.230: loss: 0.18020 [I 2022-10-26 16:27:09] ...optimizers/trijoint.py.151: total_norm: 4.070226 [I 2022-10-26 16:27:14] ...trap/engines/engine.py.224: train: epoch 18 | batch 1150/2304 [I 2022-10-26 16:27:14] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:09:34 [I 2022-10-26 16:27:14] ...trap/engines/engine.py.229: process: 0.50282 | load: 0.00247 [I 2022-10-26 16:27:14] ...trap/engines/engine.py.230: loss: 0.25404 [I 2022-10-26 16:27:14] ...optimizers/trijoint.py.151: total_norm: 3.215773 [I 2022-10-26 16:27:19] ...trap/engines/engine.py.224: train: epoch 18 | batch 1160/2304 [I 2022-10-26 16:27:19] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:09:25 [I 2022-10-26 16:27:19] ...trap/engines/engine.py.229: process: 0.48533 | load: 0.00243 [I 2022-10-26 16:27:19] ...trap/engines/engine.py.230: loss: 0.16662 [I 2022-10-26 16:27:19] ...optimizers/trijoint.py.151: total_norm: 3.039975 [I 2022-10-26 16:27:24] ...trap/engines/engine.py.224: train: epoch 18 | batch 1170/2304 [I 2022-10-26 16:27:24] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:09:28 [I 2022-10-26 16:27:24] ...trap/engines/engine.py.229: process: 0.50683 | load: 0.00247 [I 2022-10-26 16:27:24] ...trap/engines/engine.py.230: loss: 0.27053 [I 2022-10-26 16:27:24] ...optimizers/trijoint.py.151: total_norm: 3.993073 [I 2022-10-26 16:27:29] ...trap/engines/engine.py.224: train: epoch 18 | batch 1180/2304 [I 2022-10-26 16:27:29] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:09:26 [I 2022-10-26 16:27:29] ...trap/engines/engine.py.229: process: 0.51398 | load: 0.00248 [I 2022-10-26 16:27:29] ...trap/engines/engine.py.230: loss: 0.19527 [I 2022-10-26 16:27:29] ...optimizers/trijoint.py.151: total_norm: 3.266909 [I 2022-10-26 16:27:34] ...trap/engines/engine.py.224: train: epoch 18 | batch 1190/2304 [I 2022-10-26 16:27:34] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:09:22 [I 2022-10-26 16:27:34] ...trap/engines/engine.py.229: process: 0.49517 | load: 0.00254 [I 2022-10-26 16:27:34] ...trap/engines/engine.py.230: loss: 0.23999 [I 2022-10-26 16:27:34] ...optimizers/trijoint.py.151: total_norm: 3.172972 [I 2022-10-26 16:27:39] ...trap/engines/engine.py.224: train: epoch 18 | batch 1200/2304 [I 2022-10-26 16:27:39] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:09:18 [I 2022-10-26 16:27:39] ...trap/engines/engine.py.229: process: 0.51027 | load: 0.00248 [I 2022-10-26 16:27:39] ...trap/engines/engine.py.230: loss: 0.24715 [I 2022-10-26 16:27:39] ...optimizers/trijoint.py.151: total_norm: 3.255562 [I 2022-10-26 16:27:44] ...trap/engines/engine.py.224: train: epoch 18 | batch 1210/2304 [I 2022-10-26 16:27:44] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:09:12 [I 2022-10-26 16:27:44] ...trap/engines/engine.py.229: process: 0.50768 | load: 0.00248 [I 2022-10-26 16:27:44] ...trap/engines/engine.py.230: loss: 0.22796 [I 2022-10-26 16:27:44] ...optimizers/trijoint.py.151: total_norm: 3.504894 [I 2022-10-26 16:27:49] ...trap/engines/engine.py.224: train: epoch 18 | batch 1220/2304 [I 2022-10-26 16:27:49] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:09:03 [I 2022-10-26 16:27:49] ...trap/engines/engine.py.229: process: 0.49894 | load: 0.00244 [I 2022-10-26 16:27:49] ...trap/engines/engine.py.230: loss: 0.20537 [I 2022-10-26 16:27:49] ...optimizers/trijoint.py.151: total_norm: 2.982097 [I 2022-10-26 16:27:54] ...trap/engines/engine.py.224: train: epoch 18 | batch 1230/2304 [I 2022-10-26 16:27:54] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:09:03 [I 2022-10-26 16:27:54] ...trap/engines/engine.py.229: process: 0.50779 | load: 0.00244 [I 2022-10-26 16:27:54] ...trap/engines/engine.py.230: loss: 0.17515 [I 2022-10-26 16:27:54] ...optimizers/trijoint.py.151: total_norm: 2.539371 [I 2022-10-26 16:27:59] ...trap/engines/engine.py.224: train: epoch 18 | batch 1240/2304 [I 2022-10-26 16:27:59] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:08:50 [I 2022-10-26 16:27:59] ...trap/engines/engine.py.229: process: 0.49524 | load: 0.00241 [I 2022-10-26 16:27:59] ...trap/engines/engine.py.230: loss: 0.15130 [I 2022-10-26 16:27:59] ...optimizers/trijoint.py.151: total_norm: 1.827815 [I 2022-10-26 16:28:04] ...trap/engines/engine.py.224: train: epoch 18 | batch 1250/2304 [I 2022-10-26 16:28:04] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:08:43 [I 2022-10-26 16:28:04] ...trap/engines/engine.py.229: process: 0.48627 | load: 0.00245 [I 2022-10-26 16:28:04] ...trap/engines/engine.py.230: loss: 0.14391 [I 2022-10-26 16:28:04] ...optimizers/trijoint.py.151: total_norm: 3.182212 [I 2022-10-26 16:28:09] ...trap/engines/engine.py.224: train: epoch 18 | batch 1260/2304 [I 2022-10-26 16:28:09] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:08:43 [I 2022-10-26 16:28:09] ...trap/engines/engine.py.229: process: 0.50050 | load: 0.00250 [I 2022-10-26 16:28:09] ...trap/engines/engine.py.230: loss: 0.15687 [I 2022-10-26 16:28:09] ...optimizers/trijoint.py.151: total_norm: 3.534996 [I 2022-10-26 16:28:14] ...trap/engines/engine.py.224: train: epoch 18 | batch 1270/2304 [I 2022-10-26 16:28:14] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:08:36 [I 2022-10-26 16:28:14] ...trap/engines/engine.py.229: process: 0.49097 | load: 0.00248 [I 2022-10-26 16:28:14] ...trap/engines/engine.py.230: loss: 0.18297 [I 2022-10-26 16:28:14] ...optimizers/trijoint.py.151: total_norm: 2.256154 [I 2022-10-26 16:28:19] ...trap/engines/engine.py.224: train: epoch 18 | batch 1280/2304 [I 2022-10-26 16:28:19] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:08:30 [I 2022-10-26 16:28:19] ...trap/engines/engine.py.229: process: 0.50214 | load: 0.00246 [I 2022-10-26 16:28:19] ...trap/engines/engine.py.230: loss: 0.22349 [I 2022-10-26 16:28:19] ...optimizers/trijoint.py.151: total_norm: 3.780578 [I 2022-10-26 16:28:24] ...trap/engines/engine.py.224: train: epoch 18 | batch 1290/2304 [I 2022-10-26 16:28:24] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:08:27 [I 2022-10-26 16:28:24] ...trap/engines/engine.py.229: process: 0.49613 | load: 0.00247 [I 2022-10-26 16:28:24] ...trap/engines/engine.py.230: loss: 0.26497 [I 2022-10-26 16:28:24] ...optimizers/trijoint.py.151: total_norm: 4.147749 [I 2022-10-26 16:28:29] ...trap/engines/engine.py.224: train: epoch 18 | batch 1300/2304 [I 2022-10-26 16:28:29] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:08:23 [I 2022-10-26 16:28:29] ...trap/engines/engine.py.229: process: 0.51456 | load: 0.00247 [I 2022-10-26 16:28:29] ...trap/engines/engine.py.230: loss: 0.18739 [I 2022-10-26 16:28:29] ...optimizers/trijoint.py.151: total_norm: 2.918376 [I 2022-10-26 16:28:34] ...trap/engines/engine.py.224: train: epoch 18 | batch 1310/2304 [I 2022-10-26 16:28:34] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:08:18 [I 2022-10-26 16:28:34] ...trap/engines/engine.py.229: process: 0.50737 | load: 0.00251 [I 2022-10-26 16:28:34] ...trap/engines/engine.py.230: loss: 0.14589 [I 2022-10-26 16:28:34] ...optimizers/trijoint.py.151: total_norm: 3.348229 [I 2022-10-26 16:28:39] ...trap/engines/engine.py.224: train: epoch 18 | batch 1320/2304 [I 2022-10-26 16:28:39] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:08:20 [I 2022-10-26 16:28:39] ...trap/engines/engine.py.229: process: 0.51587 | load: 0.00247 [I 2022-10-26 16:28:39] ...trap/engines/engine.py.230: loss: 0.18966 [I 2022-10-26 16:28:39] ...optimizers/trijoint.py.151: total_norm: 2.351139 [I 2022-10-26 16:28:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 1330/2304 [I 2022-10-26 16:28:45] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:08:09 [I 2022-10-26 16:28:45] ...trap/engines/engine.py.229: process: 0.50736 | load: 0.00244 [I 2022-10-26 16:28:45] ...trap/engines/engine.py.230: loss: 0.18208 [I 2022-10-26 16:28:45] ...optimizers/trijoint.py.151: total_norm: 3.000938 [I 2022-10-26 16:28:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 1340/2304 [I 2022-10-26 16:28:50] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:08:04 [I 2022-10-26 16:28:50] ...trap/engines/engine.py.229: process: 0.51937 | load: 0.00244 [I 2022-10-26 16:28:50] ...trap/engines/engine.py.230: loss: 0.23969 [I 2022-10-26 16:28:50] ...optimizers/trijoint.py.151: total_norm: 3.382037 [I 2022-10-26 16:28:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 1350/2304 [I 2022-10-26 16:28:55] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:08:01 [I 2022-10-26 16:28:55] ...trap/engines/engine.py.229: process: 0.50171 | load: 0.00243 [I 2022-10-26 16:28:55] ...trap/engines/engine.py.230: loss: 0.19372 [I 2022-10-26 16:28:55] ...optimizers/trijoint.py.151: total_norm: 3.478370 [I 2022-10-26 16:29:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 1360/2304 [I 2022-10-26 16:29:00] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:07:50 [I 2022-10-26 16:29:00] ...trap/engines/engine.py.229: process: 0.49814 | load: 0.00242 [I 2022-10-26 16:29:00] ...trap/engines/engine.py.230: loss: 0.18783 [I 2022-10-26 16:29:00] ...optimizers/trijoint.py.151: total_norm: 3.513523 [I 2022-10-26 16:29:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 1370/2304 [I 2022-10-26 16:29:05] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:07:47 [I 2022-10-26 16:29:05] ...trap/engines/engine.py.229: process: 0.50463 | load: 0.00255 [I 2022-10-26 16:29:05] ...trap/engines/engine.py.230: loss: 0.26062 [I 2022-10-26 16:29:05] ...optimizers/trijoint.py.151: total_norm: 3.231254 [I 2022-10-26 16:29:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 1380/2304 [I 2022-10-26 16:29:10] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:07:40 [I 2022-10-26 16:29:10] ...trap/engines/engine.py.229: process: 0.49630 | load: 0.00248 [I 2022-10-26 16:29:10] ...trap/engines/engine.py.230: loss: 0.11664 [I 2022-10-26 16:29:10] ...optimizers/trijoint.py.151: total_norm: 2.443622 [I 2022-10-26 16:29:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 1390/2304 [I 2022-10-26 16:29:15] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:07:32 [I 2022-10-26 16:29:15] ...trap/engines/engine.py.229: process: 0.49213 | load: 0.00243 [I 2022-10-26 16:29:15] ...trap/engines/engine.py.230: loss: 0.23428 [I 2022-10-26 16:29:15] ...optimizers/trijoint.py.151: total_norm: 4.721219 [I 2022-10-26 16:29:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 1400/2304 [I 2022-10-26 16:29:20] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:07:30 [I 2022-10-26 16:29:20] ...trap/engines/engine.py.229: process: 0.49729 | load: 0.00244 [I 2022-10-26 16:29:20] ...trap/engines/engine.py.230: loss: 0.19520 [I 2022-10-26 16:29:20] ...optimizers/trijoint.py.151: total_norm: 3.556536 [I 2022-10-26 16:29:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 1410/2304 [I 2022-10-26 16:29:25] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:07:26 [I 2022-10-26 16:29:25] ...trap/engines/engine.py.229: process: 0.49360 | load: 0.00250 [I 2022-10-26 16:29:25] ...trap/engines/engine.py.230: loss: 0.13044 [I 2022-10-26 16:29:25] ...optimizers/trijoint.py.151: total_norm: 3.016032 [I 2022-10-26 16:29:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 1420/2304 [I 2022-10-26 16:29:30] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:07:24 [I 2022-10-26 16:29:30] ...trap/engines/engine.py.229: process: 0.50721 | load: 0.00250 [I 2022-10-26 16:29:30] ...trap/engines/engine.py.230: loss: 0.16140 [I 2022-10-26 16:29:30] ...optimizers/trijoint.py.151: total_norm: 2.930214 [I 2022-10-26 16:29:35] ...trap/engines/engine.py.224: train: epoch 18 | batch 1430/2304 [I 2022-10-26 16:29:35] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:07:12 [I 2022-10-26 16:29:35] ...trap/engines/engine.py.229: process: 0.48223 | load: 0.00264 [I 2022-10-26 16:29:35] ...trap/engines/engine.py.230: loss: 0.18557 [I 2022-10-26 16:29:35] ...optimizers/trijoint.py.151: total_norm: 3.389058 [I 2022-10-26 16:29:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 1440/2304 [I 2022-10-26 16:29:40] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:07:09 [I 2022-10-26 16:29:40] ...trap/engines/engine.py.229: process: 0.49086 | load: 0.00248 [I 2022-10-26 16:29:40] ...trap/engines/engine.py.230: loss: 0.18384 [I 2022-10-26 16:29:40] ...optimizers/trijoint.py.151: total_norm: 3.193845 [I 2022-10-26 16:29:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 1450/2304 [I 2022-10-26 16:29:45] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:07:00 [I 2022-10-26 16:29:45] ...trap/engines/engine.py.229: process: 0.48938 | load: 0.00246 [I 2022-10-26 16:29:45] ...trap/engines/engine.py.230: loss: 0.31531 [I 2022-10-26 16:29:45] ...optimizers/trijoint.py.151: total_norm: 4.743621 [I 2022-10-26 16:29:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 1460/2304 [I 2022-10-26 16:29:50] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:07:02 [I 2022-10-26 16:29:50] ...trap/engines/engine.py.229: process: 0.49838 | load: 0.00249 [I 2022-10-26 16:29:50] ...trap/engines/engine.py.230: loss: 0.26511 [I 2022-10-26 16:29:50] ...optimizers/trijoint.py.151: total_norm: 5.466624 [I 2022-10-26 16:29:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 1470/2304 [I 2022-10-26 16:29:55] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:06:55 [I 2022-10-26 16:29:55] ...trap/engines/engine.py.229: process: 0.48400 | load: 0.00242 [I 2022-10-26 16:29:55] ...trap/engines/engine.py.230: loss: 0.32057 [I 2022-10-26 16:29:55] ...optimizers/trijoint.py.151: total_norm: 4.139761 [I 2022-10-26 16:30:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 1480/2304 [I 2022-10-26 16:30:00] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:07:00 [I 2022-10-26 16:30:00] ...trap/engines/engine.py.229: process: 0.49900 | load: 0.00269 [I 2022-10-26 16:30:00] ...trap/engines/engine.py.230: loss: 0.20292 [I 2022-10-26 16:30:00] ...optimizers/trijoint.py.151: total_norm: 3.471722 [I 2022-10-26 16:30:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 1490/2304 [I 2022-10-26 16:30:05] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:06:47 [I 2022-10-26 16:30:05] ...trap/engines/engine.py.229: process: 0.50368 | load: 0.00244 [I 2022-10-26 16:30:05] ...trap/engines/engine.py.230: loss: 0.19412 [I 2022-10-26 16:30:05] ...optimizers/trijoint.py.151: total_norm: 2.939114 [I 2022-10-26 16:30:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 1500/2304 [I 2022-10-26 16:30:10] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:06:43 [I 2022-10-26 16:30:10] ...trap/engines/engine.py.229: process: 0.50467 | load: 0.00263 [I 2022-10-26 16:30:10] ...trap/engines/engine.py.230: loss: 0.15669 [I 2022-10-26 16:30:10] ...optimizers/trijoint.py.151: total_norm: 1.747284 [I 2022-10-26 16:30:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 1510/2304 [I 2022-10-26 16:30:15] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:06:35 [I 2022-10-26 16:30:15] ...trap/engines/engine.py.229: process: 0.50140 | load: 0.00253 [I 2022-10-26 16:30:15] ...trap/engines/engine.py.230: loss: 0.20109 [I 2022-10-26 16:30:15] ...optimizers/trijoint.py.151: total_norm: 2.304833 [I 2022-10-26 16:30:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 1520/2304 [I 2022-10-26 16:30:20] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:06:36 [I 2022-10-26 16:30:20] ...trap/engines/engine.py.229: process: 0.51119 | load: 0.00250 [I 2022-10-26 16:30:20] ...trap/engines/engine.py.230: loss: 0.18798 [I 2022-10-26 16:30:20] ...optimizers/trijoint.py.151: total_norm: 4.060843 [I 2022-10-26 16:30:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 1530/2304 [I 2022-10-26 16:30:25] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:06:31 [I 2022-10-26 16:30:25] ...trap/engines/engine.py.229: process: 0.52067 | load: 0.00248 [I 2022-10-26 16:30:25] ...trap/engines/engine.py.230: loss: 0.14365 [I 2022-10-26 16:30:25] ...optimizers/trijoint.py.151: total_norm: 2.758193 [I 2022-10-26 16:30:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 1540/2304 [I 2022-10-26 16:30:30] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:06:25 [I 2022-10-26 16:30:30] ...trap/engines/engine.py.229: process: 0.50765 | load: 0.00247 [I 2022-10-26 16:30:30] ...trap/engines/engine.py.230: loss: 0.28104 [I 2022-10-26 16:30:30] ...optimizers/trijoint.py.151: total_norm: 4.809915 [I 2022-10-26 16:30:35] ...trap/engines/engine.py.224: train: epoch 18 | batch 1550/2304 [I 2022-10-26 16:30:35] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:06:20 [I 2022-10-26 16:30:35] ...trap/engines/engine.py.229: process: 0.51336 | load: 0.00245 [I 2022-10-26 16:30:35] ...trap/engines/engine.py.230: loss: 0.24419 [I 2022-10-26 16:30:35] ...optimizers/trijoint.py.151: total_norm: 3.636708 [I 2022-10-26 16:30:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 1560/2304 [I 2022-10-26 16:30:40] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:06:13 [I 2022-10-26 16:30:40] ...trap/engines/engine.py.229: process: 0.49081 | load: 0.00246 [I 2022-10-26 16:30:40] ...trap/engines/engine.py.230: loss: 0.22276 [I 2022-10-26 16:30:40] ...optimizers/trijoint.py.151: total_norm: 4.047351 [I 2022-10-26 16:30:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 1570/2304 [I 2022-10-26 16:30:45] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:06:09 [I 2022-10-26 16:30:45] ...trap/engines/engine.py.229: process: 0.49233 | load: 0.00267 [I 2022-10-26 16:30:45] ...trap/engines/engine.py.230: loss: 0.23655 [I 2022-10-26 16:30:45] ...optimizers/trijoint.py.151: total_norm: 3.805447 [I 2022-10-26 16:30:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 1580/2304 [I 2022-10-26 16:30:50] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:06:05 [I 2022-10-26 16:30:50] ...trap/engines/engine.py.229: process: 0.51339 | load: 0.00252 [I 2022-10-26 16:30:50] ...trap/engines/engine.py.230: loss: 0.17158 [I 2022-10-26 16:30:50] ...optimizers/trijoint.py.151: total_norm: 3.751592 [I 2022-10-26 16:30:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 1590/2304 [I 2022-10-26 16:30:55] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:05:56 [I 2022-10-26 16:30:55] ...trap/engines/engine.py.229: process: 0.49995 | load: 0.00248 [I 2022-10-26 16:30:55] ...trap/engines/engine.py.230: loss: 0.17832 [I 2022-10-26 16:30:55] ...optimizers/trijoint.py.151: total_norm: 3.605021 [I 2022-10-26 16:31:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 1600/2304 [I 2022-10-26 16:31:00] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:05:49 [I 2022-10-26 16:31:00] ...trap/engines/engine.py.229: process: 0.49171 | load: 0.00243 [I 2022-10-26 16:31:00] ...trap/engines/engine.py.230: loss: 0.24941 [I 2022-10-26 16:31:00] ...optimizers/trijoint.py.151: total_norm: 3.625364 [I 2022-10-26 16:31:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 1610/2304 [I 2022-10-26 16:31:05] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:05:44 [I 2022-10-26 16:31:05] ...trap/engines/engine.py.229: process: 0.49489 | load: 0.00240 [I 2022-10-26 16:31:05] ...trap/engines/engine.py.230: loss: 0.19349 [I 2022-10-26 16:31:05] ...optimizers/trijoint.py.151: total_norm: 3.557762 [I 2022-10-26 16:31:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 1620/2304 [I 2022-10-26 16:31:10] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:05:39 [I 2022-10-26 16:31:10] ...trap/engines/engine.py.229: process: 0.50165 | load: 0.00243 [I 2022-10-26 16:31:10] ...trap/engines/engine.py.230: loss: 0.13214 [I 2022-10-26 16:31:10] ...optimizers/trijoint.py.151: total_norm: 2.135517 [I 2022-10-26 16:31:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 1630/2304 [I 2022-10-26 16:31:15] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:05:35 [I 2022-10-26 16:31:15] ...trap/engines/engine.py.229: process: 0.49845 | load: 0.00248 [I 2022-10-26 16:31:15] ...trap/engines/engine.py.230: loss: 0.18432 [I 2022-10-26 16:31:15] ...optimizers/trijoint.py.151: total_norm: 3.295929 [I 2022-10-26 16:31:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 1640/2304 [I 2022-10-26 16:31:20] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:05:28 [I 2022-10-26 16:31:20] ...trap/engines/engine.py.229: process: 0.48512 | load: 0.00243 [I 2022-10-26 16:31:20] ...trap/engines/engine.py.230: loss: 0.22270 [I 2022-10-26 16:31:20] ...optimizers/trijoint.py.151: total_norm: 3.170680 [I 2022-10-26 16:31:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 1650/2304 [I 2022-10-26 16:31:25] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:05:27 [I 2022-10-26 16:31:25] ...trap/engines/engine.py.229: process: 0.49764 | load: 0.00243 [I 2022-10-26 16:31:25] ...trap/engines/engine.py.230: loss: 0.18806 [I 2022-10-26 16:31:25] ...optimizers/trijoint.py.151: total_norm: 2.301770 [I 2022-10-26 16:31:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 1660/2304 [I 2022-10-26 16:31:30] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:05:20 [I 2022-10-26 16:31:30] ...trap/engines/engine.py.229: process: 0.48855 | load: 0.00266 [I 2022-10-26 16:31:30] ...trap/engines/engine.py.230: loss: 0.18813 [I 2022-10-26 16:31:30] ...optimizers/trijoint.py.151: total_norm: 3.220237 [I 2022-10-26 16:31:35] ...trap/engines/engine.py.224: train: epoch 18 | batch 1670/2304 [I 2022-10-26 16:31:35] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:05:17 [I 2022-10-26 16:31:35] ...trap/engines/engine.py.229: process: 0.50679 | load: 0.00247 [I 2022-10-26 16:31:35] ...trap/engines/engine.py.230: loss: 0.22072 [I 2022-10-26 16:31:35] ...optimizers/trijoint.py.151: total_norm: 2.529762 [I 2022-10-26 16:31:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 1680/2304 [I 2022-10-26 16:31:40] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:05:12 [I 2022-10-26 16:31:40] ...trap/engines/engine.py.229: process: 0.49938 | load: 0.00248 [I 2022-10-26 16:31:40] ...trap/engines/engine.py.230: loss: 0.21059 [I 2022-10-26 16:31:40] ...optimizers/trijoint.py.151: total_norm: 3.374127 [I 2022-10-26 16:31:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 1690/2304 [I 2022-10-26 16:31:45] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:05:05 [I 2022-10-26 16:31:45] ...trap/engines/engine.py.229: process: 0.49294 | load: 0.00264 [I 2022-10-26 16:31:45] ...trap/engines/engine.py.230: loss: 0.20849 [I 2022-10-26 16:31:45] ...optimizers/trijoint.py.151: total_norm: 3.202263 [I 2022-10-26 16:31:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 1700/2304 [I 2022-10-26 16:31:50] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:05:01 [I 2022-10-26 16:31:50] ...trap/engines/engine.py.229: process: 0.50150 | load: 0.00248 [I 2022-10-26 16:31:50] ...trap/engines/engine.py.230: loss: 0.25629 [I 2022-10-26 16:31:50] ...optimizers/trijoint.py.151: total_norm: 3.967538 [I 2022-10-26 16:31:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 1710/2304 [I 2022-10-26 16:31:55] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:04:54 [I 2022-10-26 16:31:55] ...trap/engines/engine.py.229: process: 0.48457 | load: 0.00244 [I 2022-10-26 16:31:55] ...trap/engines/engine.py.230: loss: 0.22057 [I 2022-10-26 16:31:55] ...optimizers/trijoint.py.151: total_norm: 4.126706 [I 2022-10-26 16:32:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 1720/2304 [I 2022-10-26 16:32:00] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:04:53 [I 2022-10-26 16:32:00] ...trap/engines/engine.py.229: process: 0.49184 | load: 0.00248 [I 2022-10-26 16:32:00] ...trap/engines/engine.py.230: loss: 0.19642 [I 2022-10-26 16:32:00] ...optimizers/trijoint.py.151: total_norm: 3.765504 [I 2022-10-26 16:32:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 1730/2304 [I 2022-10-26 16:32:05] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:04:47 [I 2022-10-26 16:32:05] ...trap/engines/engine.py.229: process: 0.50464 | load: 0.00245 [I 2022-10-26 16:32:05] ...trap/engines/engine.py.230: loss: 0.13938 [I 2022-10-26 16:32:05] ...optimizers/trijoint.py.151: total_norm: 2.592392 [I 2022-10-26 16:32:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 1740/2304 [I 2022-10-26 16:32:10] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:04:43 [I 2022-10-26 16:32:10] ...trap/engines/engine.py.229: process: 0.49279 | load: 0.00262 [I 2022-10-26 16:32:10] ...trap/engines/engine.py.230: loss: 0.16555 [I 2022-10-26 16:32:10] ...optimizers/trijoint.py.151: total_norm: 2.253637 [I 2022-10-26 16:32:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 1750/2304 [I 2022-10-26 16:32:15] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:04:38 [I 2022-10-26 16:32:15] ...trap/engines/engine.py.229: process: 0.50055 | load: 0.00244 [I 2022-10-26 16:32:15] ...trap/engines/engine.py.230: loss: 0.13430 [I 2022-10-26 16:32:15] ...optimizers/trijoint.py.151: total_norm: 3.177980 [I 2022-10-26 16:32:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 1760/2304 [I 2022-10-26 16:32:20] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:04:31 [I 2022-10-26 16:32:20] ...trap/engines/engine.py.229: process: 0.50143 | load: 0.00253 [I 2022-10-26 16:32:20] ...trap/engines/engine.py.230: loss: 0.28433 [I 2022-10-26 16:32:20] ...optimizers/trijoint.py.151: total_norm: 4.077606 [I 2022-10-26 16:32:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 1770/2304 [I 2022-10-26 16:32:25] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:04:27 [I 2022-10-26 16:32:25] ...trap/engines/engine.py.229: process: 0.49963 | load: 0.00249 [I 2022-10-26 16:32:25] ...trap/engines/engine.py.230: loss: 0.22007 [I 2022-10-26 16:32:25] ...optimizers/trijoint.py.151: total_norm: 3.352266 [I 2022-10-26 16:32:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 1780/2304 [I 2022-10-26 16:32:30] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:04:20 [I 2022-10-26 16:32:30] ...trap/engines/engine.py.229: process: 0.49578 | load: 0.00247 [I 2022-10-26 16:32:30] ...trap/engines/engine.py.230: loss: 0.23011 [I 2022-10-26 16:32:30] ...optimizers/trijoint.py.151: total_norm: 3.475778 [I 2022-10-26 16:32:35] ...trap/engines/engine.py.224: train: epoch 18 | batch 1790/2304 [I 2022-10-26 16:32:35] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:04:15 [I 2022-10-26 16:32:35] ...trap/engines/engine.py.229: process: 0.50212 | load: 0.00246 [I 2022-10-26 16:32:35] ...trap/engines/engine.py.230: loss: 0.19732 [I 2022-10-26 16:32:35] ...optimizers/trijoint.py.151: total_norm: 4.064528 [I 2022-10-26 16:32:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 1800/2304 [I 2022-10-26 16:32:40] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:04:11 [I 2022-10-26 16:32:40] ...trap/engines/engine.py.229: process: 0.49055 | load: 0.00261 [I 2022-10-26 16:32:40] ...trap/engines/engine.py.230: loss: 0.29567 [I 2022-10-26 16:32:40] ...optimizers/trijoint.py.151: total_norm: 4.304273 [I 2022-10-26 16:32:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 1810/2304 [I 2022-10-26 16:32:45] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:04:05 [I 2022-10-26 16:32:45] ...trap/engines/engine.py.229: process: 0.48638 | load: 0.00251 [I 2022-10-26 16:32:45] ...trap/engines/engine.py.230: loss: 0.24568 [I 2022-10-26 16:32:45] ...optimizers/trijoint.py.151: total_norm: 2.760560 [I 2022-10-26 16:32:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 1820/2304 [I 2022-10-26 16:32:50] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:04:02 [I 2022-10-26 16:32:50] ...trap/engines/engine.py.229: process: 0.50182 | load: 0.00246 [I 2022-10-26 16:32:50] ...trap/engines/engine.py.230: loss: 0.27103 [I 2022-10-26 16:32:50] ...optimizers/trijoint.py.151: total_norm: 4.099100 [I 2022-10-26 16:32:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 1830/2304 [I 2022-10-26 16:32:55] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:03:57 [I 2022-10-26 16:32:55] ...trap/engines/engine.py.229: process: 0.50016 | load: 0.00244 [I 2022-10-26 16:32:55] ...trap/engines/engine.py.230: loss: 0.20358 [I 2022-10-26 16:32:55] ...optimizers/trijoint.py.151: total_norm: 3.353823 [I 2022-10-26 16:33:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 1840/2304 [I 2022-10-26 16:33:00] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:03:50 [I 2022-10-26 16:33:00] ...trap/engines/engine.py.229: process: 0.49001 | load: 0.00244 [I 2022-10-26 16:33:00] ...trap/engines/engine.py.230: loss: 0.21537 [I 2022-10-26 16:33:00] ...optimizers/trijoint.py.151: total_norm: 3.015212 [I 2022-10-26 16:33:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 1850/2304 [I 2022-10-26 16:33:05] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:03:45 [I 2022-10-26 16:33:05] ...trap/engines/engine.py.229: process: 0.49654 | load: 0.00252 [I 2022-10-26 16:33:05] ...trap/engines/engine.py.230: loss: 0.25824 [I 2022-10-26 16:33:05] ...optimizers/trijoint.py.151: total_norm: 3.710289 [I 2022-10-26 16:33:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 1860/2304 [I 2022-10-26 16:33:10] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:03:40 [I 2022-10-26 16:33:10] ...trap/engines/engine.py.229: process: 0.50029 | load: 0.00247 [I 2022-10-26 16:33:10] ...trap/engines/engine.py.230: loss: 0.15260 [I 2022-10-26 16:33:10] ...optimizers/trijoint.py.151: total_norm: 3.063331 [I 2022-10-26 16:33:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 1870/2304 [I 2022-10-26 16:33:15] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:03:35 [I 2022-10-26 16:33:15] ...trap/engines/engine.py.229: process: 0.49429 | load: 0.00248 [I 2022-10-26 16:33:15] ...trap/engines/engine.py.230: loss: 0.27640 [I 2022-10-26 16:33:15] ...optimizers/trijoint.py.151: total_norm: 3.757271 [I 2022-10-26 16:33:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 1880/2304 [I 2022-10-26 16:33:20] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:03:30 [I 2022-10-26 16:33:20] ...trap/engines/engine.py.229: process: 0.49630 | load: 0.00261 [I 2022-10-26 16:33:20] ...trap/engines/engine.py.230: loss: 0.21935 [I 2022-10-26 16:33:20] ...optimizers/trijoint.py.151: total_norm: 3.151863 [I 2022-10-26 16:33:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 1890/2304 [I 2022-10-26 16:33:25] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:03:27 [I 2022-10-26 16:33:25] ...trap/engines/engine.py.229: process: 0.49890 | load: 0.00245 [I 2022-10-26 16:33:25] ...trap/engines/engine.py.230: loss: 0.21069 [I 2022-10-26 16:33:25] ...optimizers/trijoint.py.151: total_norm: 2.721422 [I 2022-10-26 16:33:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 1900/2304 [I 2022-10-26 16:33:30] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:03:20 [I 2022-10-26 16:33:30] ...trap/engines/engine.py.229: process: 0.49522 | load: 0.00254 [I 2022-10-26 16:33:30] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-26 16:33:30] ...optimizers/trijoint.py.151: total_norm: 3.013262 [I 2022-10-26 16:33:35] ...trap/engines/engine.py.224: train: epoch 18 | batch 1910/2304 [I 2022-10-26 16:33:35] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:03:16 [I 2022-10-26 16:33:35] ...trap/engines/engine.py.229: process: 0.50247 | load: 0.00245 [I 2022-10-26 16:33:35] ...trap/engines/engine.py.230: loss: 0.22010 [I 2022-10-26 16:33:35] ...optimizers/trijoint.py.151: total_norm: 2.956614 [I 2022-10-26 16:33:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 1920/2304 [I 2022-10-26 16:33:40] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:03:10 [I 2022-10-26 16:33:40] ...trap/engines/engine.py.229: process: 0.49304 | load: 0.00246 [I 2022-10-26 16:33:40] ...trap/engines/engine.py.230: loss: 0.22748 [I 2022-10-26 16:33:40] ...optimizers/trijoint.py.151: total_norm: 4.099755 [I 2022-10-26 16:33:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 1930/2304 [I 2022-10-26 16:33:45] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:03:06 [I 2022-10-26 16:33:45] ...trap/engines/engine.py.229: process: 0.49993 | load: 0.00246 [I 2022-10-26 16:33:45] ...trap/engines/engine.py.230: loss: 0.14027 [I 2022-10-26 16:33:45] ...optimizers/trijoint.py.151: total_norm: 3.006178 [I 2022-10-26 16:33:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 1940/2304 [I 2022-10-26 16:33:50] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:03:00 [I 2022-10-26 16:33:50] ...trap/engines/engine.py.229: process: 0.49161 | load: 0.00241 [I 2022-10-26 16:33:50] ...trap/engines/engine.py.230: loss: 0.27265 [I 2022-10-26 16:33:50] ...optimizers/trijoint.py.151: total_norm: 4.440738 [I 2022-10-26 16:33:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 1950/2304 [I 2022-10-26 16:33:55] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:02:57 [I 2022-10-26 16:33:55] ...trap/engines/engine.py.229: process: 0.49019 | load: 0.00255 [I 2022-10-26 16:33:55] ...trap/engines/engine.py.230: loss: 0.26132 [I 2022-10-26 16:33:55] ...optimizers/trijoint.py.151: total_norm: 2.996965 [I 2022-10-26 16:34:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 1960/2304 [I 2022-10-26 16:34:00] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:02:52 [I 2022-10-26 16:34:00] ...trap/engines/engine.py.229: process: 0.49784 | load: 0.00251 [I 2022-10-26 16:34:00] ...trap/engines/engine.py.230: loss: 0.29121 [I 2022-10-26 16:34:00] ...optimizers/trijoint.py.151: total_norm: 4.565717 [I 2022-10-26 16:34:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 1970/2304 [I 2022-10-26 16:34:05] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:02:47 [I 2022-10-26 16:34:05] ...trap/engines/engine.py.229: process: 0.50475 | load: 0.00245 [I 2022-10-26 16:34:05] ...trap/engines/engine.py.230: loss: 0.16569 [I 2022-10-26 16:34:05] ...optimizers/trijoint.py.151: total_norm: 2.902052 [I 2022-10-26 16:34:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 1980/2304 [I 2022-10-26 16:34:10] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:02:41 [I 2022-10-26 16:34:10] ...trap/engines/engine.py.229: process: 0.49835 | load: 0.00248 [I 2022-10-26 16:34:10] ...trap/engines/engine.py.230: loss: 0.24312 [I 2022-10-26 16:34:10] ...optimizers/trijoint.py.151: total_norm: 3.848792 [I 2022-10-26 16:34:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 1990/2304 [I 2022-10-26 16:34:15] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:02:38 [I 2022-10-26 16:34:15] ...trap/engines/engine.py.229: process: 0.50265 | load: 0.00249 [I 2022-10-26 16:34:15] ...trap/engines/engine.py.230: loss: 0.27413 [I 2022-10-26 16:34:15] ...optimizers/trijoint.py.151: total_norm: 3.897959 [I 2022-10-26 16:34:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 2000/2304 [I 2022-10-26 16:34:20] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:02:30 [I 2022-10-26 16:34:20] ...trap/engines/engine.py.229: process: 0.48463 | load: 0.00243 [I 2022-10-26 16:34:20] ...trap/engines/engine.py.230: loss: 0.21914 [I 2022-10-26 16:34:20] ...optimizers/trijoint.py.151: total_norm: 3.113442 [I 2022-10-26 16:34:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 2010/2304 [I 2022-10-26 16:34:25] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:02:27 [I 2022-10-26 16:34:25] ...trap/engines/engine.py.229: process: 0.50260 | load: 0.00260 [I 2022-10-26 16:34:25] ...trap/engines/engine.py.230: loss: 0.17173 [I 2022-10-26 16:34:25] ...optimizers/trijoint.py.151: total_norm: 2.274615 [I 2022-10-26 16:34:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 2020/2304 [I 2022-10-26 16:34:30] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:02:21 [I 2022-10-26 16:34:30] ...trap/engines/engine.py.229: process: 0.49279 | load: 0.00251 [I 2022-10-26 16:34:30] ...trap/engines/engine.py.230: loss: 0.23912 [I 2022-10-26 16:34:30] ...optimizers/trijoint.py.151: total_norm: 3.096947 [I 2022-10-26 16:34:35] ...trap/engines/engine.py.224: train: epoch 18 | batch 2030/2304 [I 2022-10-26 16:34:35] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:02:17 [I 2022-10-26 16:34:35] ...trap/engines/engine.py.229: process: 0.50682 | load: 0.00243 [I 2022-10-26 16:34:35] ...trap/engines/engine.py.230: loss: 0.20311 [I 2022-10-26 16:34:35] ...optimizers/trijoint.py.151: total_norm: 2.319470 [I 2022-10-26 16:34:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 2040/2304 [I 2022-10-26 16:34:40] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:02:12 [I 2022-10-26 16:34:40] ...trap/engines/engine.py.229: process: 0.50303 | load: 0.00246 [I 2022-10-26 16:34:40] ...trap/engines/engine.py.230: loss: 0.37602 [I 2022-10-26 16:34:40] ...optimizers/trijoint.py.151: total_norm: 5.967668 [I 2022-10-26 16:34:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 2050/2304 [I 2022-10-26 16:34:45] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:02:07 [I 2022-10-26 16:34:45] ...trap/engines/engine.py.229: process: 0.49668 | load: 0.00244 [I 2022-10-26 16:34:45] ...trap/engines/engine.py.230: loss: 0.21060 [I 2022-10-26 16:34:45] ...optimizers/trijoint.py.151: total_norm: 3.509639 [I 2022-10-26 16:34:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 2060/2304 [I 2022-10-26 16:34:50] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:02:00 [I 2022-10-26 16:34:50] ...trap/engines/engine.py.229: process: 0.48759 | load: 0.00245 [I 2022-10-26 16:34:50] ...trap/engines/engine.py.230: loss: 0.26538 [I 2022-10-26 16:34:50] ...optimizers/trijoint.py.151: total_norm: 3.914874 [I 2022-10-26 16:34:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 2070/2304 [I 2022-10-26 16:34:55] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:01:56 [I 2022-10-26 16:34:55] ...trap/engines/engine.py.229: process: 0.50599 | load: 0.00241 [I 2022-10-26 16:34:55] ...trap/engines/engine.py.230: loss: 0.16461 [I 2022-10-26 16:34:55] ...optimizers/trijoint.py.151: total_norm: 2.764444 [I 2022-10-26 16:35:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 2080/2304 [I 2022-10-26 16:35:00] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:01:50 [I 2022-10-26 16:35:00] ...trap/engines/engine.py.229: process: 0.49225 | load: 0.00239 [I 2022-10-26 16:35:00] ...trap/engines/engine.py.230: loss: 0.21483 [I 2022-10-26 16:35:00] ...optimizers/trijoint.py.151: total_norm: 3.880045 [I 2022-10-26 16:35:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 2090/2304 [I 2022-10-26 16:35:05] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:01:46 [I 2022-10-26 16:35:05] ...trap/engines/engine.py.229: process: 0.49723 | load: 0.00244 [I 2022-10-26 16:35:05] ...trap/engines/engine.py.230: loss: 0.17570 [I 2022-10-26 16:35:05] ...optimizers/trijoint.py.151: total_norm: 3.626484 [I 2022-10-26 16:35:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 2100/2304 [I 2022-10-26 16:35:10] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:01:40 [I 2022-10-26 16:35:10] ...trap/engines/engine.py.229: process: 0.49849 | load: 0.00261 [I 2022-10-26 16:35:10] ...trap/engines/engine.py.230: loss: 0.19813 [I 2022-10-26 16:35:10] ...optimizers/trijoint.py.151: total_norm: 2.558973 [I 2022-10-26 16:35:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 2110/2304 [I 2022-10-26 16:35:15] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:01:36 [I 2022-10-26 16:35:15] ...trap/engines/engine.py.229: process: 0.50715 | load: 0.00249 [I 2022-10-26 16:35:15] ...trap/engines/engine.py.230: loss: 0.15822 [I 2022-10-26 16:35:15] ...optimizers/trijoint.py.151: total_norm: 3.247715 [I 2022-10-26 16:35:20] ...trap/engines/engine.py.224: train: epoch 18 | batch 2120/2304 [I 2022-10-26 16:35:20] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:01:31 [I 2022-10-26 16:35:20] ...trap/engines/engine.py.229: process: 0.48863 | load: 0.00280 [I 2022-10-26 16:35:20] ...trap/engines/engine.py.230: loss: 0.21579 [I 2022-10-26 16:35:20] ...optimizers/trijoint.py.151: total_norm: 3.614318 [I 2022-10-26 16:35:25] ...trap/engines/engine.py.224: train: epoch 18 | batch 2130/2304 [I 2022-10-26 16:35:25] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:01:29 [I 2022-10-26 16:35:25] ...trap/engines/engine.py.229: process: 0.51557 | load: 0.00245 [I 2022-10-26 16:35:25] ...trap/engines/engine.py.230: loss: 0.21267 [I 2022-10-26 16:35:25] ...optimizers/trijoint.py.151: total_norm: 3.653322 [I 2022-10-26 16:35:30] ...trap/engines/engine.py.224: train: epoch 18 | batch 2140/2304 [I 2022-10-26 16:35:30] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:01:22 [I 2022-10-26 16:35:30] ...trap/engines/engine.py.229: process: 0.49741 | load: 0.00246 [I 2022-10-26 16:35:30] ...trap/engines/engine.py.230: loss: 0.15894 [I 2022-10-26 16:35:30] ...optimizers/trijoint.py.151: total_norm: 2.870468 [I 2022-10-26 16:35:35] ...trap/engines/engine.py.224: train: epoch 18 | batch 2150/2304 [I 2022-10-26 16:35:35] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:01:20 [I 2022-10-26 16:35:35] ...trap/engines/engine.py.229: process: 0.49873 | load: 0.00244 [I 2022-10-26 16:35:35] ...trap/engines/engine.py.230: loss: 0.20485 [I 2022-10-26 16:35:35] ...optimizers/trijoint.py.151: total_norm: 2.692216 [I 2022-10-26 16:35:40] ...trap/engines/engine.py.224: train: epoch 18 | batch 2160/2304 [I 2022-10-26 16:35:40] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:01:12 [I 2022-10-26 16:35:40] ...trap/engines/engine.py.229: process: 0.49900 | load: 0.00247 [I 2022-10-26 16:35:40] ...trap/engines/engine.py.230: loss: 0.21982 [I 2022-10-26 16:35:40] ...optimizers/trijoint.py.151: total_norm: 3.256790 [I 2022-10-26 16:35:45] ...trap/engines/engine.py.224: train: epoch 18 | batch 2170/2304 [I 2022-10-26 16:35:45] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:01:06 [I 2022-10-26 16:35:45] ...trap/engines/engine.py.229: process: 0.49443 | load: 0.00249 [I 2022-10-26 16:35:45] ...trap/engines/engine.py.230: loss: 0.14735 [I 2022-10-26 16:35:45] ...optimizers/trijoint.py.151: total_norm: 2.318680 [I 2022-10-26 16:35:50] ...trap/engines/engine.py.224: train: epoch 18 | batch 2180/2304 [I 2022-10-26 16:35:50] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:01:01 [I 2022-10-26 16:35:50] ...trap/engines/engine.py.229: process: 0.50932 | load: 0.00244 [I 2022-10-26 16:35:50] ...trap/engines/engine.py.230: loss: 0.23665 [I 2022-10-26 16:35:50] ...optimizers/trijoint.py.151: total_norm: 2.992726 [I 2022-10-26 16:35:55] ...trap/engines/engine.py.224: train: epoch 18 | batch 2190/2304 [I 2022-10-26 16:35:55] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:00:56 [I 2022-10-26 16:35:55] ...trap/engines/engine.py.229: process: 0.48699 | load: 0.00246 [I 2022-10-26 16:35:55] ...trap/engines/engine.py.230: loss: 0.19293 [I 2022-10-26 16:35:55] ...optimizers/trijoint.py.151: total_norm: 3.302843 [I 2022-10-26 16:36:00] ...trap/engines/engine.py.224: train: epoch 18 | batch 2200/2304 [I 2022-10-26 16:36:00] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:00:51 [I 2022-10-26 16:36:00] ...trap/engines/engine.py.229: process: 0.49758 | load: 0.00245 [I 2022-10-26 16:36:00] ...trap/engines/engine.py.230: loss: 0.26129 [I 2022-10-26 16:36:00] ...optimizers/trijoint.py.151: total_norm: 3.375665 [I 2022-10-26 16:36:05] ...trap/engines/engine.py.224: train: epoch 18 | batch 2210/2304 [I 2022-10-26 16:36:05] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:00:47 [I 2022-10-26 16:36:05] ...trap/engines/engine.py.229: process: 0.49788 | load: 0.00243 [I 2022-10-26 16:36:05] ...trap/engines/engine.py.230: loss: 0.21616 [I 2022-10-26 16:36:05] ...optimizers/trijoint.py.151: total_norm: 3.049497 [I 2022-10-26 16:36:10] ...trap/engines/engine.py.224: train: epoch 18 | batch 2220/2304 [I 2022-10-26 16:36:10] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:00:42 [I 2022-10-26 16:36:10] ...trap/engines/engine.py.229: process: 0.50754 | load: 0.00254 [I 2022-10-26 16:36:10] ...trap/engines/engine.py.230: loss: 0.23364 [I 2022-10-26 16:36:10] ...optimizers/trijoint.py.151: total_norm: 3.803716 [I 2022-10-26 16:36:15] ...trap/engines/engine.py.224: train: epoch 18 | batch 2230/2304 [I 2022-10-26 16:36:15] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:00:37 [I 2022-10-26 16:36:15] ...trap/engines/engine.py.229: process: 0.51420 | load: 0.00247 [I 2022-10-26 16:36:15] ...trap/engines/engine.py.230: loss: 0.20429 [I 2022-10-26 16:36:15] ...optimizers/trijoint.py.151: total_norm: 3.219064 [I 2022-10-26 16:36:21] ...trap/engines/engine.py.224: train: epoch 18 | batch 2240/2304 [I 2022-10-26 16:36:21] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:00:32 [I 2022-10-26 16:36:21] ...trap/engines/engine.py.229: process: 0.49799 | load: 0.00247 [I 2022-10-26 16:36:21] ...trap/engines/engine.py.230: loss: 0.32368 [I 2022-10-26 16:36:21] ...optimizers/trijoint.py.151: total_norm: 4.193486 [I 2022-10-26 16:36:26] ...trap/engines/engine.py.224: train: epoch 18 | batch 2250/2304 [I 2022-10-26 16:36:26] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:00:26 [I 2022-10-26 16:36:26] ...trap/engines/engine.py.229: process: 0.50101 | load: 0.00245 [I 2022-10-26 16:36:26] ...trap/engines/engine.py.230: loss: 0.23503 [I 2022-10-26 16:36:26] ...optimizers/trijoint.py.151: total_norm: 2.630028 [I 2022-10-26 16:36:31] ...trap/engines/engine.py.224: train: epoch 18 | batch 2260/2304 [I 2022-10-26 16:36:31] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:00:21 [I 2022-10-26 16:36:31] ...trap/engines/engine.py.229: process: 0.49563 | load: 0.00259 [I 2022-10-26 16:36:31] ...trap/engines/engine.py.230: loss: 0.19654 [I 2022-10-26 16:36:31] ...optimizers/trijoint.py.151: total_norm: 3.169173 [I 2022-10-26 16:36:36] ...trap/engines/engine.py.224: train: epoch 18 | batch 2270/2304 [I 2022-10-26 16:36:36] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:00:16 [I 2022-10-26 16:36:36] ...trap/engines/engine.py.229: process: 0.49322 | load: 0.00247 [I 2022-10-26 16:36:36] ...trap/engines/engine.py.230: loss: 0.22058 [I 2022-10-26 16:36:36] ...optimizers/trijoint.py.151: total_norm: 3.676458 [I 2022-10-26 16:36:41] ...trap/engines/engine.py.224: train: epoch 18 | batch 2280/2304 [I 2022-10-26 16:36:41] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:00:11 [I 2022-10-26 16:36:41] ...trap/engines/engine.py.229: process: 0.49378 | load: 0.00242 [I 2022-10-26 16:36:41] ...trap/engines/engine.py.230: loss: 0.16581 [I 2022-10-26 16:36:41] ...optimizers/trijoint.py.151: total_norm: 3.655800 [I 2022-10-26 16:36:46] ...trap/engines/engine.py.224: train: epoch 18 | batch 2290/2304 [I 2022-10-26 16:36:46] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:00:07 [I 2022-10-26 16:36:46] ...trap/engines/engine.py.229: process: 0.51540 | load: 0.00246 [I 2022-10-26 16:36:46] ...trap/engines/engine.py.230: loss: 0.27708 [I 2022-10-26 16:36:46] ...optimizers/trijoint.py.151: total_norm: 2.379461 [I 2022-10-26 16:36:51] ...trap/engines/engine.py.224: train: epoch 18 | batch 2300/2304 [I 2022-10-26 16:36:51] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:00:02 [I 2022-10-26 16:36:51] ...trap/engines/engine.py.229: process: 0.49182 | load: 0.00017 [I 2022-10-26 16:36:51] ...trap/engines/engine.py.230: loss: 0.19585 [I 2022-10-26 16:36:51] ...optimizers/trijoint.py.151: total_norm: 3.332126 [I 2022-10-26 16:36:53] ...trap/engines/engine.py.224: train: epoch 18 | batch 2304/2304 [I 2022-10-26 16:36:53] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:00:00 [I 2022-10-26 16:36:53] ...trap/engines/engine.py.229: process: 0.48974 | load: 0.00015 [I 2022-10-26 16:36:53] ...trap/engines/engine.py.230: loss: 0.26234 [I 2022-10-26 16:36:53] ...optimizers/trijoint.py.151: total_norm: 3.555587 [S 2022-10-26 16:36:54] ...trap/engines/engine.py.129: train_epoch.epoch: 18 [S 2022-10-26 16:36:54] ...trap/engines/engine.py.129: train_epoch.loss: 0.20546009429041578 [I 2022-10-26 16:36:54] ...s/criterions/triplet.py.84: margin = 0.14500000000000007 [I 2022-10-26 16:36:54] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 18 [I 2022-10-26 16:36:54] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 16:36:56] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 16:36:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 16:36:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 16:36:57] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 16:36:57] ...trap/engines/engine.py.321: eval: epoch 18 | batch 0/510 [I 2022-10-26 16:36:57] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:21 [I 2022-10-26 16:36:57] ...trap/engines/engine.py.326: process: 2.74746 | load: 2.35083 [I 2022-10-26 16:37:00] ...trap/engines/engine.py.321: eval: epoch 18 | batch 10/510 [I 2022-10-26 16:37:00] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:56 [I 2022-10-26 16:37:00] ...trap/engines/engine.py.326: process: 0.33288 | load: 0.00025 [I 2022-10-26 16:37:04] ...trap/engines/engine.py.321: eval: epoch 18 | batch 20/510 [I 2022-10-26 16:37:04] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:58 [I 2022-10-26 16:37:04] ...trap/engines/engine.py.326: process: 0.31929 | load: 0.00027 [I 2022-10-26 16:37:07] ...trap/engines/engine.py.321: eval: epoch 18 | batch 30/510 [I 2022-10-26 16:37:07] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:45 [I 2022-10-26 16:37:07] ...trap/engines/engine.py.326: process: 0.35758 | load: 0.00020 [I 2022-10-26 16:37:11] ...trap/engines/engine.py.321: eval: epoch 18 | batch 40/510 [I 2022-10-26 16:37:11] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:37 [I 2022-10-26 16:37:11] ...trap/engines/engine.py.326: process: 0.33737 | load: 0.00024 [I 2022-10-26 16:37:14] ...trap/engines/engine.py.321: eval: epoch 18 | batch 50/510 [I 2022-10-26 16:37:14] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-26 16:37:14] ...trap/engines/engine.py.326: process: 0.33171 | load: 0.00020 [I 2022-10-26 16:37:17] ...trap/engines/engine.py.321: eval: epoch 18 | batch 60/510 [I 2022-10-26 16:37:17] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:35 [I 2022-10-26 16:37:17] ...trap/engines/engine.py.326: process: 0.37386 | load: 0.00021 [I 2022-10-26 16:37:21] ...trap/engines/engine.py.321: eval: epoch 18 | batch 70/510 [I 2022-10-26 16:37:21] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:29 [I 2022-10-26 16:37:21] ...trap/engines/engine.py.326: process: 0.32638 | load: 0.00020 [I 2022-10-26 16:37:24] ...trap/engines/engine.py.321: eval: epoch 18 | batch 80/510 [I 2022-10-26 16:37:24] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:24 [I 2022-10-26 16:37:24] ...trap/engines/engine.py.326: process: 0.32125 | load: 0.00019 [I 2022-10-26 16:37:28] ...trap/engines/engine.py.321: eval: epoch 18 | batch 90/510 [I 2022-10-26 16:37:28] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:30 [I 2022-10-26 16:37:28] ...trap/engines/engine.py.326: process: 0.45058 | load: 0.00017 [I 2022-10-26 16:37:31] ...trap/engines/engine.py.321: eval: epoch 18 | batch 100/510 [I 2022-10-26 16:37:31] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:14 [I 2022-10-26 16:37:31] ...trap/engines/engine.py.326: process: 0.28661 | load: 0.00020 [I 2022-10-26 16:37:34] ...trap/engines/engine.py.321: eval: epoch 18 | batch 110/510 [I 2022-10-26 16:37:34] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:58 [I 2022-10-26 16:37:34] ...trap/engines/engine.py.326: process: 0.29254 | load: 0.00020 [I 2022-10-26 16:37:37] ...trap/engines/engine.py.321: eval: epoch 18 | batch 120/510 [I 2022-10-26 16:37:37] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-26 16:37:37] ...trap/engines/engine.py.326: process: 0.28749 | load: 0.00022 [I 2022-10-26 16:37:40] ...trap/engines/engine.py.321: eval: epoch 18 | batch 130/510 [I 2022-10-26 16:37:40] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:52 [I 2022-10-26 16:37:40] ...trap/engines/engine.py.326: process: 0.31049 | load: 0.00014 [I 2022-10-26 16:37:43] ...trap/engines/engine.py.321: eval: epoch 18 | batch 140/510 [I 2022-10-26 16:37:43] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-26 16:37:43] ...trap/engines/engine.py.326: process: 0.28584 | load: 0.00020 [I 2022-10-26 16:37:45] ...trap/engines/engine.py.321: eval: epoch 18 | batch 150/510 [I 2022-10-26 16:37:45] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-26 16:37:45] ...trap/engines/engine.py.326: process: 0.28706 | load: 0.00020 [I 2022-10-26 16:37:48] ...trap/engines/engine.py.321: eval: epoch 18 | batch 160/510 [I 2022-10-26 16:37:48] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 16:37:48] ...trap/engines/engine.py.326: process: 0.29155 | load: 0.00015 [I 2022-10-26 16:37:51] ...trap/engines/engine.py.321: eval: epoch 18 | batch 170/510 [I 2022-10-26 16:37:51] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 16:37:51] ...trap/engines/engine.py.326: process: 0.28954 | load: 0.00017 [I 2022-10-26 16:37:54] ...trap/engines/engine.py.321: eval: epoch 18 | batch 180/510 [I 2022-10-26 16:37:54] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 16:37:54] ...trap/engines/engine.py.326: process: 0.29215 | load: 0.00031 [I 2022-10-26 16:37:57] ...trap/engines/engine.py.321: eval: epoch 18 | batch 190/510 [I 2022-10-26 16:37:57] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:31 [I 2022-10-26 16:37:57] ...trap/engines/engine.py.326: process: 0.28204 | load: 0.00015 [I 2022-10-26 16:38:00] ...trap/engines/engine.py.321: eval: epoch 18 | batch 200/510 [I 2022-10-26 16:38:00] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 16:38:00] ...trap/engines/engine.py.326: process: 0.28403 | load: 0.00029 [I 2022-10-26 16:38:03] ...trap/engines/engine.py.321: eval: epoch 18 | batch 210/510 [I 2022-10-26 16:38:03] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 16:38:03] ...trap/engines/engine.py.326: process: 0.28363 | load: 0.00021 [I 2022-10-26 16:38:06] ...trap/engines/engine.py.321: eval: epoch 18 | batch 220/510 [I 2022-10-26 16:38:06] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-26 16:38:06] ...trap/engines/engine.py.326: process: 0.28769 | load: 0.00015 [I 2022-10-26 16:38:09] ...trap/engines/engine.py.321: eval: epoch 18 | batch 230/510 [I 2022-10-26 16:38:09] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-26 16:38:09] ...trap/engines/engine.py.326: process: 0.28750 | load: 0.00021 [I 2022-10-26 16:38:11] ...trap/engines/engine.py.321: eval: epoch 18 | batch 240/510 [I 2022-10-26 16:38:11] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 16:38:11] ...trap/engines/engine.py.326: process: 0.29094 | load: 0.00020 [I 2022-10-26 16:38:14] ...trap/engines/engine.py.321: eval: epoch 18 | batch 250/510 [I 2022-10-26 16:38:14] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-26 16:38:14] ...trap/engines/engine.py.326: process: 0.29064 | load: 0.00016 [I 2022-10-26 16:38:17] ...trap/engines/engine.py.321: eval: epoch 18 | batch 260/510 [I 2022-10-26 16:38:17] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:11 [I 2022-10-26 16:38:17] ...trap/engines/engine.py.326: process: 0.27916 | load: 0.00015 [I 2022-10-26 16:38:20] ...trap/engines/engine.py.321: eval: epoch 18 | batch 270/510 [I 2022-10-26 16:38:20] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-26 16:38:20] ...trap/engines/engine.py.326: process: 0.28200 | load: 0.00015 [I 2022-10-26 16:38:23] ...trap/engines/engine.py.321: eval: epoch 18 | batch 280/510 [I 2022-10-26 16:38:23] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:05 [I 2022-10-26 16:38:23] ...trap/engines/engine.py.326: process: 0.28818 | load: 0.00015 [I 2022-10-26 16:38:26] ...trap/engines/engine.py.321: eval: epoch 18 | batch 290/510 [I 2022-10-26 16:38:26] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 16:38:26] ...trap/engines/engine.py.326: process: 0.28717 | load: 0.00014 [I 2022-10-26 16:38:29] ...trap/engines/engine.py.321: eval: epoch 18 | batch 300/510 [I 2022-10-26 16:38:29] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 16:38:29] ...trap/engines/engine.py.326: process: 0.27997 | load: 0.00016 [I 2022-10-26 16:38:32] ...trap/engines/engine.py.321: eval: epoch 18 | batch 310/510 [I 2022-10-26 16:38:32] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 16:38:32] ...trap/engines/engine.py.326: process: 0.28335 | load: 0.00015 [I 2022-10-26 16:38:34] ...trap/engines/engine.py.321: eval: epoch 18 | batch 320/510 [I 2022-10-26 16:38:34] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 16:38:34] ...trap/engines/engine.py.326: process: 0.28238 | load: 0.00021 [I 2022-10-26 16:38:37] ...trap/engines/engine.py.321: eval: epoch 18 | batch 330/510 [I 2022-10-26 16:38:37] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 16:38:37] ...trap/engines/engine.py.326: process: 0.28837 | load: 0.00016 [I 2022-10-26 16:38:40] ...trap/engines/engine.py.321: eval: epoch 18 | batch 340/510 [I 2022-10-26 16:38:40] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:48 [I 2022-10-26 16:38:40] ...trap/engines/engine.py.326: process: 0.28977 | load: 0.00015 [I 2022-10-26 16:38:43] ...trap/engines/engine.py.321: eval: epoch 18 | batch 350/510 [I 2022-10-26 16:38:43] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-26 16:38:43] ...trap/engines/engine.py.326: process: 0.36902 | load: 0.00019 [I 2022-10-26 16:38:46] ...trap/engines/engine.py.321: eval: epoch 18 | batch 360/510 [I 2022-10-26 16:38:46] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 16:38:46] ...trap/engines/engine.py.326: process: 0.29036 | load: 0.00015 [I 2022-10-26 16:38:49] ...trap/engines/engine.py.321: eval: epoch 18 | batch 370/510 [I 2022-10-26 16:38:49] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 16:38:49] ...trap/engines/engine.py.326: process: 0.29040 | load: 0.00020 [I 2022-10-26 16:38:52] ...trap/engines/engine.py.321: eval: epoch 18 | batch 380/510 [I 2022-10-26 16:38:52] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 16:38:52] ...trap/engines/engine.py.326: process: 0.28412 | load: 0.00014 [I 2022-10-26 16:38:55] ...trap/engines/engine.py.321: eval: epoch 18 | batch 390/510 [I 2022-10-26 16:38:55] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 16:38:55] ...trap/engines/engine.py.326: process: 0.29179 | load: 0.00015 [I 2022-10-26 16:38:58] ...trap/engines/engine.py.321: eval: epoch 18 | batch 400/510 [I 2022-10-26 16:38:58] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 16:38:58] ...trap/engines/engine.py.326: process: 0.28872 | load: 0.00016 [I 2022-10-26 16:39:00] ...trap/engines/engine.py.321: eval: epoch 18 | batch 410/510 [I 2022-10-26 16:39:00] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 16:39:00] ...trap/engines/engine.py.326: process: 0.28819 | load: 0.00016 [I 2022-10-26 16:39:03] ...trap/engines/engine.py.321: eval: epoch 18 | batch 420/510 [I 2022-10-26 16:39:03] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-26 16:39:03] ...trap/engines/engine.py.326: process: 0.28132 | load: 0.00020 [I 2022-10-26 16:39:06] ...trap/engines/engine.py.321: eval: epoch 18 | batch 430/510 [I 2022-10-26 16:39:06] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:22 [I 2022-10-26 16:39:06] ...trap/engines/engine.py.326: process: 0.28901 | load: 0.00016 [I 2022-10-26 16:39:09] ...trap/engines/engine.py.321: eval: epoch 18 | batch 440/510 [I 2022-10-26 16:39:09] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 16:39:09] ...trap/engines/engine.py.326: process: 0.28956 | load: 0.00015 [I 2022-10-26 16:39:12] ...trap/engines/engine.py.321: eval: epoch 18 | batch 450/510 [I 2022-10-26 16:39:12] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 16:39:12] ...trap/engines/engine.py.326: process: 0.28266 | load: 0.00016 [I 2022-10-26 16:39:15] ...trap/engines/engine.py.321: eval: epoch 18 | batch 460/510 [I 2022-10-26 16:39:15] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 16:39:15] ...trap/engines/engine.py.326: process: 0.28506 | load: 0.00016 [I 2022-10-26 16:39:18] ...trap/engines/engine.py.321: eval: epoch 18 | batch 470/510 [I 2022-10-26 16:39:18] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 16:39:18] ...trap/engines/engine.py.326: process: 0.28149 | load: 0.00015 [I 2022-10-26 16:39:21] ...trap/engines/engine.py.321: eval: epoch 18 | batch 480/510 [I 2022-10-26 16:39:21] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 16:39:21] ...trap/engines/engine.py.326: process: 0.28976 | load: 0.00016 [I 2022-10-26 16:39:23] ...trap/engines/engine.py.321: eval: epoch 18 | batch 490/510 [I 2022-10-26 16:39:23] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 16:39:23] ...trap/engines/engine.py.326: process: 0.28040 | load: 0.00017 [I 2022-10-26 16:39:26] ...trap/engines/engine.py.321: eval: epoch 18 | batch 500/510 [I 2022-10-26 16:39:26] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 16:39:26] ...trap/engines/engine.py.326: process: 0.28514 | load: 0.00015 [I 2022-10-26 16:39:29] ...trap/engines/engine.py.321: eval: epoch 18 | batch 510/510 [I 2022-10-26 16:39:29] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 16:39:29] ...trap/engines/engine.py.326: process: 0.27618 | load: 0.00013 [S 2022-10-26 16:39:30] ...trap/engines/engine.py.132: eval_epoch.epoch: 18 [I 2022-10-26 16:39:30] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.568600 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.836800 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.895100 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.567200 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.829800 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.891000 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011817 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009998 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008654 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011487 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008727 [S 2022-10-26 16:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007389 [I 2022-10-26 16:39:35] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 16:39:35] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 16:39:35] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 16:39:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 16:39:35] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 16:39:36] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 16:39:39] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 16:39:44] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 16:39:44] ...trap/engines/engine.py.163: Training model on trainset for epoch 19 [I 2022-10-26 16:39:44] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 16:39:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 0/2304 [I 2022-10-26 16:39:46] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:54 [I 2022-10-26 16:39:46] ...trap/engines/engine.py.229: process: 2.73216 | load: 2.14767 [I 2022-10-26 16:39:46] ...trap/engines/engine.py.230: loss: 0.21051 [I 2022-10-26 16:39:46] ...optimizers/trijoint.py.151: total_norm: 3.448306 [I 2022-10-26 16:39:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 10/2304 [I 2022-10-26 16:39:51] ...trap/engines/engine.py.228: elapsed: 0:00:07 | left: 0:28:22 [I 2022-10-26 16:39:51] ...trap/engines/engine.py.229: process: 0.49092 | load: 0.00447 [I 2022-10-26 16:39:51] ...trap/engines/engine.py.230: loss: 0.17014 [I 2022-10-26 16:39:51] ...optimizers/trijoint.py.151: total_norm: 3.586469 [I 2022-10-26 16:39:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 20/2304 [I 2022-10-26 16:39:56] ...trap/engines/engine.py.228: elapsed: 0:00:12 | left: 0:20:10 [I 2022-10-26 16:39:56] ...trap/engines/engine.py.229: process: 0.49905 | load: 0.00372 [I 2022-10-26 16:39:56] ...trap/engines/engine.py.230: loss: 0.24499 [I 2022-10-26 16:39:56] ...optimizers/trijoint.py.151: total_norm: 3.252246 [I 2022-10-26 16:40:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 30/2304 [I 2022-10-26 16:40:01] ...trap/engines/engine.py.228: elapsed: 0:00:17 | left: 0:19:03 [I 2022-10-26 16:40:01] ...trap/engines/engine.py.229: process: 0.50212 | load: 0.00293 [I 2022-10-26 16:40:01] ...trap/engines/engine.py.230: loss: 0.22930 [I 2022-10-26 16:40:01] ...optimizers/trijoint.py.151: total_norm: 3.946953 [I 2022-10-26 16:40:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 40/2304 [I 2022-10-26 16:40:06] ...trap/engines/engine.py.228: elapsed: 0:00:22 | left: 0:19:03 [I 2022-10-26 16:40:06] ...trap/engines/engine.py.229: process: 0.50659 | load: 0.00315 [I 2022-10-26 16:40:06] ...trap/engines/engine.py.230: loss: 0.28532 [I 2022-10-26 16:40:06] ...optimizers/trijoint.py.151: total_norm: 4.073167 [I 2022-10-26 16:40:12] ...trap/engines/engine.py.224: train: epoch 19 | batch 50/2304 [I 2022-10-26 16:40:12] ...trap/engines/engine.py.228: elapsed: 0:00:27 | left: 0:18:54 [I 2022-10-26 16:40:12] ...trap/engines/engine.py.229: process: 0.51074 | load: 0.00283 [I 2022-10-26 16:40:12] ...trap/engines/engine.py.230: loss: 0.22923 [I 2022-10-26 16:40:12] ...optimizers/trijoint.py.151: total_norm: 3.286539 [I 2022-10-26 16:40:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 60/2304 [I 2022-10-26 16:40:16] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:18:40 [I 2022-10-26 16:40:16] ...trap/engines/engine.py.229: process: 0.50266 | load: 0.00281 [I 2022-10-26 16:40:16] ...trap/engines/engine.py.230: loss: 0.19372 [I 2022-10-26 16:40:16] ...optimizers/trijoint.py.151: total_norm: 3.737656 [I 2022-10-26 16:40:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 70/2304 [I 2022-10-26 16:40:21] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:18:32 [I 2022-10-26 16:40:21] ...trap/engines/engine.py.229: process: 0.49758 | load: 0.00273 [I 2022-10-26 16:40:21] ...trap/engines/engine.py.230: loss: 0.17745 [I 2022-10-26 16:40:21] ...optimizers/trijoint.py.151: total_norm: 3.532346 [I 2022-10-26 16:40:27] ...trap/engines/engine.py.224: train: epoch 19 | batch 80/2304 [I 2022-10-26 16:40:27] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:18:38 [I 2022-10-26 16:40:27] ...trap/engines/engine.py.229: process: 0.50240 | load: 0.00303 [I 2022-10-26 16:40:27] ...trap/engines/engine.py.230: loss: 0.23525 [I 2022-10-26 16:40:27] ...optimizers/trijoint.py.151: total_norm: 3.877794 [I 2022-10-26 16:40:32] ...trap/engines/engine.py.224: train: epoch 19 | batch 90/2304 [I 2022-10-26 16:40:32] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:18:19 [I 2022-10-26 16:40:32] ...trap/engines/engine.py.229: process: 0.48566 | load: 0.00288 [I 2022-10-26 16:40:32] ...trap/engines/engine.py.230: loss: 0.16354 [I 2022-10-26 16:40:32] ...optimizers/trijoint.py.151: total_norm: 2.501001 [I 2022-10-26 16:40:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 100/2304 [I 2022-10-26 16:40:36] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:18:16 [I 2022-10-26 16:40:36] ...trap/engines/engine.py.229: process: 0.49297 | load: 0.00289 [I 2022-10-26 16:40:36] ...trap/engines/engine.py.230: loss: 0.19626 [I 2022-10-26 16:40:37] ...optimizers/trijoint.py.151: total_norm: 3.233000 [I 2022-10-26 16:40:42] ...trap/engines/engine.py.224: train: epoch 19 | batch 110/2304 [I 2022-10-26 16:40:42] ...trap/engines/engine.py.228: elapsed: 0:00:57 | left: 0:18:18 [I 2022-10-26 16:40:42] ...trap/engines/engine.py.229: process: 0.50585 | load: 0.00247 [I 2022-10-26 16:40:42] ...trap/engines/engine.py.230: loss: 0.20715 [I 2022-10-26 16:40:42] ...optimizers/trijoint.py.151: total_norm: 2.777351 [I 2022-10-26 16:40:47] ...trap/engines/engine.py.224: train: epoch 19 | batch 120/2304 [I 2022-10-26 16:40:47] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:18:27 [I 2022-10-26 16:40:47] ...trap/engines/engine.py.229: process: 0.51298 | load: 0.00268 [I 2022-10-26 16:40:47] ...trap/engines/engine.py.230: loss: 0.23777 [I 2022-10-26 16:40:47] ...optimizers/trijoint.py.151: total_norm: 3.858219 [I 2022-10-26 16:40:52] ...trap/engines/engine.py.224: train: epoch 19 | batch 130/2304 [I 2022-10-26 16:40:52] ...trap/engines/engine.py.228: elapsed: 0:01:08 | left: 0:18:17 [I 2022-10-26 16:40:52] ...trap/engines/engine.py.229: process: 0.49054 | load: 0.00254 [I 2022-10-26 16:40:52] ...trap/engines/engine.py.230: loss: 0.22583 [I 2022-10-26 16:40:52] ...optimizers/trijoint.py.151: total_norm: 5.733987 [I 2022-10-26 16:40:57] ...trap/engines/engine.py.224: train: epoch 19 | batch 140/2304 [I 2022-10-26 16:40:57] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:17:59 [I 2022-10-26 16:40:57] ...trap/engines/engine.py.229: process: 0.48522 | load: 0.00263 [I 2022-10-26 16:40:57] ...trap/engines/engine.py.230: loss: 0.22307 [I 2022-10-26 16:40:57] ...optimizers/trijoint.py.151: total_norm: 3.259701 [I 2022-10-26 16:41:02] ...trap/engines/engine.py.224: train: epoch 19 | batch 150/2304 [I 2022-10-26 16:41:02] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:17:43 [I 2022-10-26 16:41:02] ...trap/engines/engine.py.229: process: 0.48438 | load: 0.00247 [I 2022-10-26 16:41:02] ...trap/engines/engine.py.230: loss: 0.18907 [I 2022-10-26 16:41:02] ...optimizers/trijoint.py.151: total_norm: 3.484322 [I 2022-10-26 16:41:07] ...trap/engines/engine.py.224: train: epoch 19 | batch 160/2304 [I 2022-10-26 16:41:07] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:17:46 [I 2022-10-26 16:41:07] ...trap/engines/engine.py.229: process: 0.49983 | load: 0.00255 [I 2022-10-26 16:41:07] ...trap/engines/engine.py.230: loss: 0.28726 [I 2022-10-26 16:41:07] ...optimizers/trijoint.py.151: total_norm: 3.741069 [I 2022-10-26 16:41:12] ...trap/engines/engine.py.224: train: epoch 19 | batch 170/2304 [I 2022-10-26 16:41:12] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:17:44 [I 2022-10-26 16:41:12] ...trap/engines/engine.py.229: process: 0.51431 | load: 0.00245 [I 2022-10-26 16:41:12] ...trap/engines/engine.py.230: loss: 0.19368 [I 2022-10-26 16:41:12] ...optimizers/trijoint.py.151: total_norm: 3.434743 [I 2022-10-26 16:41:17] ...trap/engines/engine.py.224: train: epoch 19 | batch 180/2304 [I 2022-10-26 16:41:17] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:17:32 [I 2022-10-26 16:41:17] ...trap/engines/engine.py.229: process: 0.50271 | load: 0.00263 [I 2022-10-26 16:41:17] ...trap/engines/engine.py.230: loss: 0.19261 [I 2022-10-26 16:41:17] ...optimizers/trijoint.py.151: total_norm: 2.459171 [I 2022-10-26 16:41:22] ...trap/engines/engine.py.224: train: epoch 19 | batch 190/2304 [I 2022-10-26 16:41:22] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:17:42 [I 2022-10-26 16:41:22] ...trap/engines/engine.py.229: process: 0.50132 | load: 0.00265 [I 2022-10-26 16:41:22] ...trap/engines/engine.py.230: loss: 0.18526 [I 2022-10-26 16:41:22] ...optimizers/trijoint.py.151: total_norm: 2.668646 [I 2022-10-26 16:41:27] ...trap/engines/engine.py.224: train: epoch 19 | batch 200/2304 [I 2022-10-26 16:41:27] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:17:29 [I 2022-10-26 16:41:27] ...trap/engines/engine.py.229: process: 0.49765 | load: 0.00247 [I 2022-10-26 16:41:27] ...trap/engines/engine.py.230: loss: 0.13003 [I 2022-10-26 16:41:27] ...optimizers/trijoint.py.151: total_norm: 2.590129 [I 2022-10-26 16:41:32] ...trap/engines/engine.py.224: train: epoch 19 | batch 210/2304 [I 2022-10-26 16:41:32] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:17:31 [I 2022-10-26 16:41:32] ...trap/engines/engine.py.229: process: 0.51835 | load: 0.01731 [I 2022-10-26 16:41:32] ...trap/engines/engine.py.230: loss: 0.15765 [I 2022-10-26 16:41:32] ...optimizers/trijoint.py.151: total_norm: 2.402686 [I 2022-10-26 16:41:37] ...trap/engines/engine.py.224: train: epoch 19 | batch 220/2304 [I 2022-10-26 16:41:37] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:17:27 [I 2022-10-26 16:41:37] ...trap/engines/engine.py.229: process: 0.49414 | load: 0.00249 [I 2022-10-26 16:41:37] ...trap/engines/engine.py.230: loss: 0.21596 [I 2022-10-26 16:41:37] ...optimizers/trijoint.py.151: total_norm: 3.230396 [I 2022-10-26 16:41:42] ...trap/engines/engine.py.224: train: epoch 19 | batch 230/2304 [I 2022-10-26 16:41:42] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:17:12 [I 2022-10-26 16:41:42] ...trap/engines/engine.py.229: process: 0.49786 | load: 0.00255 [I 2022-10-26 16:41:42] ...trap/engines/engine.py.230: loss: 0.20584 [I 2022-10-26 16:41:42] ...optimizers/trijoint.py.151: total_norm: 3.006423 [I 2022-10-26 16:41:47] ...trap/engines/engine.py.224: train: epoch 19 | batch 240/2304 [I 2022-10-26 16:41:47] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:17:06 [I 2022-10-26 16:41:47] ...trap/engines/engine.py.229: process: 0.49773 | load: 0.00245 [I 2022-10-26 16:41:47] ...trap/engines/engine.py.230: loss: 0.18287 [I 2022-10-26 16:41:47] ...optimizers/trijoint.py.151: total_norm: 2.637162 [I 2022-10-26 16:41:52] ...trap/engines/engine.py.224: train: epoch 19 | batch 250/2304 [I 2022-10-26 16:41:52] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:17:06 [I 2022-10-26 16:41:52] ...trap/engines/engine.py.229: process: 0.49895 | load: 0.00240 [I 2022-10-26 16:41:52] ...trap/engines/engine.py.230: loss: 0.18181 [I 2022-10-26 16:41:52] ...optimizers/trijoint.py.151: total_norm: 3.199098 [I 2022-10-26 16:41:57] ...trap/engines/engine.py.224: train: epoch 19 | batch 260/2304 [I 2022-10-26 16:41:57] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:16:51 [I 2022-10-26 16:41:57] ...trap/engines/engine.py.229: process: 0.48205 | load: 0.00253 [I 2022-10-26 16:41:57] ...trap/engines/engine.py.230: loss: 0.17988 [I 2022-10-26 16:41:57] ...optimizers/trijoint.py.151: total_norm: 3.818325 [I 2022-10-26 16:42:02] ...trap/engines/engine.py.224: train: epoch 19 | batch 270/2304 [I 2022-10-26 16:42:02] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:16:48 [I 2022-10-26 16:42:02] ...trap/engines/engine.py.229: process: 0.50216 | load: 0.00245 [I 2022-10-26 16:42:02] ...trap/engines/engine.py.230: loss: 0.17701 [I 2022-10-26 16:42:02] ...optimizers/trijoint.py.151: total_norm: 2.603216 [I 2022-10-26 16:42:07] ...trap/engines/engine.py.224: train: epoch 19 | batch 280/2304 [I 2022-10-26 16:42:07] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:16:54 [I 2022-10-26 16:42:07] ...trap/engines/engine.py.229: process: 0.50584 | load: 0.00245 [I 2022-10-26 16:42:07] ...trap/engines/engine.py.230: loss: 0.18360 [I 2022-10-26 16:42:07] ...optimizers/trijoint.py.151: total_norm: 3.129610 [I 2022-10-26 16:42:12] ...trap/engines/engine.py.224: train: epoch 19 | batch 290/2304 [I 2022-10-26 16:42:12] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:16:46 [I 2022-10-26 16:42:12] ...trap/engines/engine.py.229: process: 0.50677 | load: 0.00249 [I 2022-10-26 16:42:12] ...trap/engines/engine.py.230: loss: 0.16881 [I 2022-10-26 16:42:12] ...optimizers/trijoint.py.151: total_norm: 3.630074 [I 2022-10-26 16:42:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 300/2304 [I 2022-10-26 16:42:16] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:16:39 [I 2022-10-26 16:42:16] ...trap/engines/engine.py.229: process: 0.50179 | load: 0.00240 [I 2022-10-26 16:42:16] ...trap/engines/engine.py.230: loss: 0.20082 [I 2022-10-26 16:42:16] ...optimizers/trijoint.py.151: total_norm: 3.200513 [I 2022-10-26 16:42:22] ...trap/engines/engine.py.224: train: epoch 19 | batch 310/2304 [I 2022-10-26 16:42:22] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:16:36 [I 2022-10-26 16:42:22] ...trap/engines/engine.py.229: process: 0.49493 | load: 0.00246 [I 2022-10-26 16:42:22] ...trap/engines/engine.py.230: loss: 0.26345 [I 2022-10-26 16:42:22] ...optimizers/trijoint.py.151: total_norm: 3.049358 [I 2022-10-26 16:42:27] ...trap/engines/engine.py.224: train: epoch 19 | batch 320/2304 [I 2022-10-26 16:42:27] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:16:31 [I 2022-10-26 16:42:27] ...trap/engines/engine.py.229: process: 0.50065 | load: 0.00243 [I 2022-10-26 16:42:27] ...trap/engines/engine.py.230: loss: 0.21310 [I 2022-10-26 16:42:27] ...optimizers/trijoint.py.151: total_norm: 2.750708 [I 2022-10-26 16:42:32] ...trap/engines/engine.py.224: train: epoch 19 | batch 330/2304 [I 2022-10-26 16:42:32] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:16:30 [I 2022-10-26 16:42:32] ...trap/engines/engine.py.229: process: 0.50307 | load: 0.00250 [I 2022-10-26 16:42:32] ...trap/engines/engine.py.230: loss: 0.23014 [I 2022-10-26 16:42:32] ...optimizers/trijoint.py.151: total_norm: 2.870088 [I 2022-10-26 16:42:37] ...trap/engines/engine.py.224: train: epoch 19 | batch 340/2304 [I 2022-10-26 16:42:37] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:16:14 [I 2022-10-26 16:42:37] ...trap/engines/engine.py.229: process: 0.47925 | load: 0.00240 [I 2022-10-26 16:42:37] ...trap/engines/engine.py.230: loss: 0.23025 [I 2022-10-26 16:42:37] ...optimizers/trijoint.py.151: total_norm: 4.063045 [I 2022-10-26 16:42:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 350/2304 [I 2022-10-26 16:42:41] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:16:05 [I 2022-10-26 16:42:41] ...trap/engines/engine.py.229: process: 0.49852 | load: 0.00241 [I 2022-10-26 16:42:41] ...trap/engines/engine.py.230: loss: 0.19510 [I 2022-10-26 16:42:41] ...optimizers/trijoint.py.151: total_norm: 3.468554 [I 2022-10-26 16:42:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 360/2304 [I 2022-10-26 16:42:46] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:16:06 [I 2022-10-26 16:42:46] ...trap/engines/engine.py.229: process: 0.49657 | load: 0.00246 [I 2022-10-26 16:42:46] ...trap/engines/engine.py.230: loss: 0.21794 [I 2022-10-26 16:42:46] ...optimizers/trijoint.py.151: total_norm: 2.728705 [I 2022-10-26 16:42:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 370/2304 [I 2022-10-26 16:42:51] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:15:59 [I 2022-10-26 16:42:51] ...trap/engines/engine.py.229: process: 0.50162 | load: 0.00240 [I 2022-10-26 16:42:51] ...trap/engines/engine.py.230: loss: 0.16579 [I 2022-10-26 16:42:51] ...optimizers/trijoint.py.151: total_norm: 3.819862 [I 2022-10-26 16:42:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 380/2304 [I 2022-10-26 16:42:56] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:15:57 [I 2022-10-26 16:42:56] ...trap/engines/engine.py.229: process: 0.50001 | load: 0.00262 [I 2022-10-26 16:42:56] ...trap/engines/engine.py.230: loss: 0.21973 [I 2022-10-26 16:42:56] ...optimizers/trijoint.py.151: total_norm: 3.766490 [I 2022-10-26 16:43:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 390/2304 [I 2022-10-26 16:43:01] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:15:52 [I 2022-10-26 16:43:01] ...trap/engines/engine.py.229: process: 0.49812 | load: 0.00249 [I 2022-10-26 16:43:01] ...trap/engines/engine.py.230: loss: 0.13552 [I 2022-10-26 16:43:01] ...optimizers/trijoint.py.151: total_norm: 2.914227 [I 2022-10-26 16:43:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 400/2304 [I 2022-10-26 16:43:06] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:15:52 [I 2022-10-26 16:43:06] ...trap/engines/engine.py.229: process: 0.50265 | load: 0.00256 [I 2022-10-26 16:43:06] ...trap/engines/engine.py.230: loss: 0.20861 [I 2022-10-26 16:43:06] ...optimizers/trijoint.py.151: total_norm: 3.357980 [I 2022-10-26 16:43:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 410/2304 [I 2022-10-26 16:43:11] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:15:40 [I 2022-10-26 16:43:11] ...trap/engines/engine.py.229: process: 0.49529 | load: 0.00242 [I 2022-10-26 16:43:11] ...trap/engines/engine.py.230: loss: 0.23618 [I 2022-10-26 16:43:11] ...optimizers/trijoint.py.151: total_norm: 3.251711 [I 2022-10-26 16:43:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 420/2304 [I 2022-10-26 16:43:16] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:15:38 [I 2022-10-26 16:43:16] ...trap/engines/engine.py.229: process: 0.49835 | load: 0.00238 [I 2022-10-26 16:43:16] ...trap/engines/engine.py.230: loss: 0.19119 [I 2022-10-26 16:43:16] ...optimizers/trijoint.py.151: total_norm: 2.972702 [I 2022-10-26 16:43:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 430/2304 [I 2022-10-26 16:43:21] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:15:29 [I 2022-10-26 16:43:21] ...trap/engines/engine.py.229: process: 0.49283 | load: 0.00240 [I 2022-10-26 16:43:21] ...trap/engines/engine.py.230: loss: 0.16798 [I 2022-10-26 16:43:21] ...optimizers/trijoint.py.151: total_norm: 2.077320 [I 2022-10-26 16:43:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 440/2304 [I 2022-10-26 16:43:26] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:15:23 [I 2022-10-26 16:43:26] ...trap/engines/engine.py.229: process: 0.49989 | load: 0.00244 [I 2022-10-26 16:43:26] ...trap/engines/engine.py.230: loss: 0.14871 [I 2022-10-26 16:43:26] ...optimizers/trijoint.py.151: total_norm: 3.566561 [I 2022-10-26 16:43:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 450/2304 [I 2022-10-26 16:43:31] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:15:17 [I 2022-10-26 16:43:31] ...trap/engines/engine.py.229: process: 0.48214 | load: 0.00242 [I 2022-10-26 16:43:31] ...trap/engines/engine.py.230: loss: 0.18893 [I 2022-10-26 16:43:31] ...optimizers/trijoint.py.151: total_norm: 3.268299 [I 2022-10-26 16:43:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 460/2304 [I 2022-10-26 16:43:36] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:15:10 [I 2022-10-26 16:43:36] ...trap/engines/engine.py.229: process: 0.49835 | load: 0.00240 [I 2022-10-26 16:43:36] ...trap/engines/engine.py.230: loss: 0.14217 [I 2022-10-26 16:43:36] ...optimizers/trijoint.py.151: total_norm: 3.239607 [I 2022-10-26 16:43:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 470/2304 [I 2022-10-26 16:43:41] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:15:11 [I 2022-10-26 16:43:41] ...trap/engines/engine.py.229: process: 0.50882 | load: 0.00249 [I 2022-10-26 16:43:41] ...trap/engines/engine.py.230: loss: 0.24576 [I 2022-10-26 16:43:41] ...optimizers/trijoint.py.151: total_norm: 3.672060 [I 2022-10-26 16:43:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 480/2304 [I 2022-10-26 16:43:46] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:15:05 [I 2022-10-26 16:43:46] ...trap/engines/engine.py.229: process: 0.48774 | load: 0.00246 [I 2022-10-26 16:43:46] ...trap/engines/engine.py.230: loss: 0.23930 [I 2022-10-26 16:43:46] ...optimizers/trijoint.py.151: total_norm: 3.218968 [I 2022-10-26 16:43:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 490/2304 [I 2022-10-26 16:43:51] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:15:02 [I 2022-10-26 16:43:51] ...trap/engines/engine.py.229: process: 0.49480 | load: 0.00243 [I 2022-10-26 16:43:51] ...trap/engines/engine.py.230: loss: 0.20649 [I 2022-10-26 16:43:51] ...optimizers/trijoint.py.151: total_norm: 3.527478 [I 2022-10-26 16:43:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 500/2304 [I 2022-10-26 16:43:56] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:14:54 [I 2022-10-26 16:43:56] ...trap/engines/engine.py.229: process: 0.49211 | load: 0.00247 [I 2022-10-26 16:43:56] ...trap/engines/engine.py.230: loss: 0.21183 [I 2022-10-26 16:43:56] ...optimizers/trijoint.py.151: total_norm: 4.159524 [I 2022-10-26 16:44:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 510/2304 [I 2022-10-26 16:44:01] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:14:54 [I 2022-10-26 16:44:01] ...trap/engines/engine.py.229: process: 0.50831 | load: 0.00244 [I 2022-10-26 16:44:01] ...trap/engines/engine.py.230: loss: 0.21671 [I 2022-10-26 16:44:01] ...optimizers/trijoint.py.151: total_norm: 3.580795 [I 2022-10-26 16:44:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 520/2304 [I 2022-10-26 16:44:06] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:14:44 [I 2022-10-26 16:44:06] ...trap/engines/engine.py.229: process: 0.50461 | load: 0.00245 [I 2022-10-26 16:44:06] ...trap/engines/engine.py.230: loss: 0.19781 [I 2022-10-26 16:44:06] ...optimizers/trijoint.py.151: total_norm: 4.112985 [I 2022-10-26 16:44:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 530/2304 [I 2022-10-26 16:44:11] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:14:43 [I 2022-10-26 16:44:11] ...trap/engines/engine.py.229: process: 0.50033 | load: 0.00253 [I 2022-10-26 16:44:11] ...trap/engines/engine.py.230: loss: 0.29643 [I 2022-10-26 16:44:11] ...optimizers/trijoint.py.151: total_norm: 3.149018 [I 2022-10-26 16:44:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 540/2304 [I 2022-10-26 16:44:16] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:14:40 [I 2022-10-26 16:44:16] ...trap/engines/engine.py.229: process: 0.49746 | load: 0.00242 [I 2022-10-26 16:44:16] ...trap/engines/engine.py.230: loss: 0.26566 [I 2022-10-26 16:44:16] ...optimizers/trijoint.py.151: total_norm: 3.797312 [I 2022-10-26 16:44:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 550/2304 [I 2022-10-26 16:44:21] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:14:28 [I 2022-10-26 16:44:21] ...trap/engines/engine.py.229: process: 0.49252 | load: 0.00240 [I 2022-10-26 16:44:21] ...trap/engines/engine.py.230: loss: 0.20296 [I 2022-10-26 16:44:21] ...optimizers/trijoint.py.151: total_norm: 2.660651 [I 2022-10-26 16:44:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 560/2304 [I 2022-10-26 16:44:26] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:14:28 [I 2022-10-26 16:44:26] ...trap/engines/engine.py.229: process: 0.50051 | load: 0.00242 [I 2022-10-26 16:44:26] ...trap/engines/engine.py.230: loss: 0.17850 [I 2022-10-26 16:44:26] ...optimizers/trijoint.py.151: total_norm: 3.182047 [I 2022-10-26 16:44:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 570/2304 [I 2022-10-26 16:44:31] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:14:25 [I 2022-10-26 16:44:31] ...trap/engines/engine.py.229: process: 0.49357 | load: 0.00242 [I 2022-10-26 16:44:31] ...trap/engines/engine.py.230: loss: 0.17164 [I 2022-10-26 16:44:31] ...optimizers/trijoint.py.151: total_norm: 2.825376 [I 2022-10-26 16:44:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 580/2304 [I 2022-10-26 16:44:36] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:14:15 [I 2022-10-26 16:44:36] ...trap/engines/engine.py.229: process: 0.49247 | load: 0.00245 [I 2022-10-26 16:44:36] ...trap/engines/engine.py.230: loss: 0.15950 [I 2022-10-26 16:44:36] ...optimizers/trijoint.py.151: total_norm: 2.322668 [I 2022-10-26 16:44:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 590/2304 [I 2022-10-26 16:44:41] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:14:06 [I 2022-10-26 16:44:41] ...trap/engines/engine.py.229: process: 0.49857 | load: 0.00242 [I 2022-10-26 16:44:41] ...trap/engines/engine.py.230: loss: 0.13398 [I 2022-10-26 16:44:41] ...optimizers/trijoint.py.151: total_norm: 2.466438 [I 2022-10-26 16:44:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 600/2304 [I 2022-10-26 16:44:46] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:14:10 [I 2022-10-26 16:44:46] ...trap/engines/engine.py.229: process: 0.50053 | load: 0.00245 [I 2022-10-26 16:44:46] ...trap/engines/engine.py.230: loss: 0.19004 [I 2022-10-26 16:44:46] ...optimizers/trijoint.py.151: total_norm: 4.781446 [I 2022-10-26 16:44:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 610/2304 [I 2022-10-26 16:44:51] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:14:06 [I 2022-10-26 16:44:51] ...trap/engines/engine.py.229: process: 0.49902 | load: 0.00246 [I 2022-10-26 16:44:51] ...trap/engines/engine.py.230: loss: 0.15922 [I 2022-10-26 16:44:51] ...optimizers/trijoint.py.151: total_norm: 2.814884 [I 2022-10-26 16:44:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 620/2304 [I 2022-10-26 16:44:56] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:14:01 [I 2022-10-26 16:44:56] ...trap/engines/engine.py.229: process: 0.49918 | load: 0.00238 [I 2022-10-26 16:44:56] ...trap/engines/engine.py.230: loss: 0.17480 [I 2022-10-26 16:44:56] ...optimizers/trijoint.py.151: total_norm: 3.421126 [I 2022-10-26 16:45:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 630/2304 [I 2022-10-26 16:45:01] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:13:53 [I 2022-10-26 16:45:01] ...trap/engines/engine.py.229: process: 0.50036 | load: 0.00246 [I 2022-10-26 16:45:01] ...trap/engines/engine.py.230: loss: 0.24673 [I 2022-10-26 16:45:01] ...optimizers/trijoint.py.151: total_norm: 3.140441 [I 2022-10-26 16:45:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 640/2304 [I 2022-10-26 16:45:06] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:13:49 [I 2022-10-26 16:45:06] ...trap/engines/engine.py.229: process: 0.49681 | load: 0.00245 [I 2022-10-26 16:45:06] ...trap/engines/engine.py.230: loss: 0.19449 [I 2022-10-26 16:45:06] ...optimizers/trijoint.py.151: total_norm: 3.170426 [I 2022-10-26 16:45:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 650/2304 [I 2022-10-26 16:45:11] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:13:39 [I 2022-10-26 16:45:11] ...trap/engines/engine.py.229: process: 0.49123 | load: 0.00241 [I 2022-10-26 16:45:11] ...trap/engines/engine.py.230: loss: 0.11795 [I 2022-10-26 16:45:11] ...optimizers/trijoint.py.151: total_norm: 1.479800 [I 2022-10-26 16:45:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 660/2304 [I 2022-10-26 16:45:16] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:13:47 [I 2022-10-26 16:45:16] ...trap/engines/engine.py.229: process: 0.50777 | load: 0.00242 [I 2022-10-26 16:45:16] ...trap/engines/engine.py.230: loss: 0.19699 [I 2022-10-26 16:45:16] ...optimizers/trijoint.py.151: total_norm: 3.416483 [I 2022-10-26 16:45:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 670/2304 [I 2022-10-26 16:45:21] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:14:04 [I 2022-10-26 16:45:21] ...trap/engines/engine.py.229: process: 0.56867 | load: 0.00246 [I 2022-10-26 16:45:21] ...trap/engines/engine.py.230: loss: 0.17392 [I 2022-10-26 16:45:21] ...optimizers/trijoint.py.151: total_norm: 2.557832 [I 2022-10-26 16:45:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 680/2304 [I 2022-10-26 16:45:26] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:13:34 [I 2022-10-26 16:45:26] ...trap/engines/engine.py.229: process: 0.49731 | load: 0.00241 [I 2022-10-26 16:45:26] ...trap/engines/engine.py.230: loss: 0.20357 [I 2022-10-26 16:45:26] ...optimizers/trijoint.py.151: total_norm: 3.228324 [I 2022-10-26 16:45:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 690/2304 [I 2022-10-26 16:45:31] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:13:24 [I 2022-10-26 16:45:31] ...trap/engines/engine.py.229: process: 0.49908 | load: 0.00241 [I 2022-10-26 16:45:31] ...trap/engines/engine.py.230: loss: 0.17650 [I 2022-10-26 16:45:31] ...optimizers/trijoint.py.151: total_norm: 2.982664 [I 2022-10-26 16:45:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 700/2304 [I 2022-10-26 16:45:36] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:13:20 [I 2022-10-26 16:45:36] ...trap/engines/engine.py.229: process: 0.50314 | load: 0.00243 [I 2022-10-26 16:45:36] ...trap/engines/engine.py.230: loss: 0.22276 [I 2022-10-26 16:45:36] ...optimizers/trijoint.py.151: total_norm: 3.909069 [I 2022-10-26 16:45:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 710/2304 [I 2022-10-26 16:45:41] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:13:15 [I 2022-10-26 16:45:41] ...trap/engines/engine.py.229: process: 0.50151 | load: 0.00243 [I 2022-10-26 16:45:41] ...trap/engines/engine.py.230: loss: 0.21675 [I 2022-10-26 16:45:41] ...optimizers/trijoint.py.151: total_norm: 3.130239 [I 2022-10-26 16:45:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 720/2304 [I 2022-10-26 16:45:46] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:13:12 [I 2022-10-26 16:45:46] ...trap/engines/engine.py.229: process: 0.50147 | load: 0.00242 [I 2022-10-26 16:45:46] ...trap/engines/engine.py.230: loss: 0.15695 [I 2022-10-26 16:45:46] ...optimizers/trijoint.py.151: total_norm: 2.589374 [I 2022-10-26 16:45:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 730/2304 [I 2022-10-26 16:45:51] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:13:06 [I 2022-10-26 16:45:51] ...trap/engines/engine.py.229: process: 0.50860 | load: 0.00241 [I 2022-10-26 16:45:51] ...trap/engines/engine.py.230: loss: 0.16134 [I 2022-10-26 16:45:51] ...optimizers/trijoint.py.151: total_norm: 2.563369 [I 2022-10-26 16:45:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 740/2304 [I 2022-10-26 16:45:56] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:13:00 [I 2022-10-26 16:45:56] ...trap/engines/engine.py.229: process: 0.49899 | load: 0.00240 [I 2022-10-26 16:45:56] ...trap/engines/engine.py.230: loss: 0.22528 [I 2022-10-26 16:45:56] ...optimizers/trijoint.py.151: total_norm: 3.850317 [I 2022-10-26 16:46:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 750/2304 [I 2022-10-26 16:46:01] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:12:57 [I 2022-10-26 16:46:01] ...trap/engines/engine.py.229: process: 0.50024 | load: 0.00257 [I 2022-10-26 16:46:01] ...trap/engines/engine.py.230: loss: 0.16315 [I 2022-10-26 16:46:01] ...optimizers/trijoint.py.151: total_norm: 3.243274 [I 2022-10-26 16:46:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 760/2304 [I 2022-10-26 16:46:06] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:13:14 [I 2022-10-26 16:46:06] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00257 [I 2022-10-26 16:46:06] ...trap/engines/engine.py.230: loss: 0.22708 [I 2022-10-26 16:46:06] ...optimizers/trijoint.py.151: total_norm: 3.142706 [I 2022-10-26 16:46:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 770/2304 [I 2022-10-26 16:46:11] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:12:46 [I 2022-10-26 16:46:11] ...trap/engines/engine.py.229: process: 0.49934 | load: 0.00242 [I 2022-10-26 16:46:11] ...trap/engines/engine.py.230: loss: 0.19492 [I 2022-10-26 16:46:11] ...optimizers/trijoint.py.151: total_norm: 3.069044 [I 2022-10-26 16:46:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 780/2304 [I 2022-10-26 16:46:16] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:12:44 [I 2022-10-26 16:46:16] ...trap/engines/engine.py.229: process: 0.50923 | load: 0.00243 [I 2022-10-26 16:46:16] ...trap/engines/engine.py.230: loss: 0.21737 [I 2022-10-26 16:46:16] ...optimizers/trijoint.py.151: total_norm: 3.168896 [I 2022-10-26 16:46:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 790/2304 [I 2022-10-26 16:46:21] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:12:36 [I 2022-10-26 16:46:21] ...trap/engines/engine.py.229: process: 0.49710 | load: 0.00243 [I 2022-10-26 16:46:21] ...trap/engines/engine.py.230: loss: 0.19358 [I 2022-10-26 16:46:21] ...optimizers/trijoint.py.151: total_norm: 2.611242 [I 2022-10-26 16:46:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 800/2304 [I 2022-10-26 16:46:26] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:12:29 [I 2022-10-26 16:46:26] ...trap/engines/engine.py.229: process: 0.49862 | load: 0.00242 [I 2022-10-26 16:46:26] ...trap/engines/engine.py.230: loss: 0.32991 [I 2022-10-26 16:46:26] ...optimizers/trijoint.py.151: total_norm: 3.786440 [I 2022-10-26 16:46:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 810/2304 [I 2022-10-26 16:46:31] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:12:19 [I 2022-10-26 16:46:31] ...trap/engines/engine.py.229: process: 0.48340 | load: 0.00243 [I 2022-10-26 16:46:31] ...trap/engines/engine.py.230: loss: 0.20399 [I 2022-10-26 16:46:31] ...optimizers/trijoint.py.151: total_norm: 3.780287 [I 2022-10-26 16:46:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 820/2304 [I 2022-10-26 16:46:36] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:12:14 [I 2022-10-26 16:46:36] ...trap/engines/engine.py.229: process: 0.49964 | load: 0.00241 [I 2022-10-26 16:46:36] ...trap/engines/engine.py.230: loss: 0.21782 [I 2022-10-26 16:46:36] ...optimizers/trijoint.py.151: total_norm: 3.853012 [I 2022-10-26 16:46:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 830/2304 [I 2022-10-26 16:46:41] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:12:31 [I 2022-10-26 16:46:41] ...trap/engines/engine.py.229: process: 0.49163 | load: 0.00239 [I 2022-10-26 16:46:41] ...trap/engines/engine.py.230: loss: 0.18970 [I 2022-10-26 16:46:41] ...optimizers/trijoint.py.151: total_norm: 3.783427 [I 2022-10-26 16:46:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 840/2304 [I 2022-10-26 16:46:46] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:12:07 [I 2022-10-26 16:46:46] ...trap/engines/engine.py.229: process: 0.49951 | load: 0.00244 [I 2022-10-26 16:46:46] ...trap/engines/engine.py.230: loss: 0.23715 [I 2022-10-26 16:46:46] ...optimizers/trijoint.py.151: total_norm: 3.320996 [I 2022-10-26 16:46:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 850/2304 [I 2022-10-26 16:46:51] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:11:59 [I 2022-10-26 16:46:51] ...trap/engines/engine.py.229: process: 0.48364 | load: 0.00242 [I 2022-10-26 16:46:51] ...trap/engines/engine.py.230: loss: 0.13656 [I 2022-10-26 16:46:51] ...optimizers/trijoint.py.151: total_norm: 2.273461 [I 2022-10-26 16:46:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 860/2304 [I 2022-10-26 16:46:56] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:11:59 [I 2022-10-26 16:46:56] ...trap/engines/engine.py.229: process: 0.50235 | load: 0.00244 [I 2022-10-26 16:46:56] ...trap/engines/engine.py.230: loss: 0.29499 [I 2022-10-26 16:46:56] ...optimizers/trijoint.py.151: total_norm: 3.872250 [I 2022-10-26 16:47:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 870/2304 [I 2022-10-26 16:47:01] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:11:52 [I 2022-10-26 16:47:01] ...trap/engines/engine.py.229: process: 0.50381 | load: 0.00244 [I 2022-10-26 16:47:01] ...trap/engines/engine.py.230: loss: 0.15085 [I 2022-10-26 16:47:01] ...optimizers/trijoint.py.151: total_norm: 3.598943 [I 2022-10-26 16:47:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 880/2304 [I 2022-10-26 16:47:06] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:11:44 [I 2022-10-26 16:47:06] ...trap/engines/engine.py.229: process: 0.49662 | load: 0.00242 [I 2022-10-26 16:47:06] ...trap/engines/engine.py.230: loss: 0.17526 [I 2022-10-26 16:47:06] ...optimizers/trijoint.py.151: total_norm: 3.810653 [I 2022-10-26 16:47:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 890/2304 [I 2022-10-26 16:47:11] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:11:42 [I 2022-10-26 16:47:11] ...trap/engines/engine.py.229: process: 0.49354 | load: 0.00244 [I 2022-10-26 16:47:11] ...trap/engines/engine.py.230: loss: 0.23787 [I 2022-10-26 16:47:11] ...optimizers/trijoint.py.151: total_norm: 4.201059 [I 2022-10-26 16:47:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 900/2304 [I 2022-10-26 16:47:16] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:11:39 [I 2022-10-26 16:47:16] ...trap/engines/engine.py.229: process: 0.50607 | load: 0.00246 [I 2022-10-26 16:47:16] ...trap/engines/engine.py.230: loss: 0.25267 [I 2022-10-26 16:47:16] ...optimizers/trijoint.py.151: total_norm: 4.191470 [I 2022-10-26 16:47:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 910/2304 [I 2022-10-26 16:47:21] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:11:28 [I 2022-10-26 16:47:21] ...trap/engines/engine.py.229: process: 0.49175 | load: 0.00250 [I 2022-10-26 16:47:21] ...trap/engines/engine.py.230: loss: 0.25951 [I 2022-10-26 16:47:21] ...optimizers/trijoint.py.151: total_norm: 3.540589 [I 2022-10-26 16:47:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 920/2304 [I 2022-10-26 16:47:26] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:11:30 [I 2022-10-26 16:47:26] ...trap/engines/engine.py.229: process: 0.50316 | load: 0.00246 [I 2022-10-26 16:47:26] ...trap/engines/engine.py.230: loss: 0.15015 [I 2022-10-26 16:47:26] ...optimizers/trijoint.py.151: total_norm: 3.417903 [I 2022-10-26 16:47:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 930/2304 [I 2022-10-26 16:47:31] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:11:18 [I 2022-10-26 16:47:31] ...trap/engines/engine.py.229: process: 0.49516 | load: 0.00243 [I 2022-10-26 16:47:31] ...trap/engines/engine.py.230: loss: 0.19238 [I 2022-10-26 16:47:31] ...optimizers/trijoint.py.151: total_norm: 2.939824 [I 2022-10-26 16:47:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 940/2304 [I 2022-10-26 16:47:36] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:11:23 [I 2022-10-26 16:47:36] ...trap/engines/engine.py.229: process: 0.50434 | load: 0.00239 [I 2022-10-26 16:47:36] ...trap/engines/engine.py.230: loss: 0.23154 [I 2022-10-26 16:47:36] ...optimizers/trijoint.py.151: total_norm: 2.386021 [I 2022-10-26 16:47:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 950/2304 [I 2022-10-26 16:47:41] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:11:12 [I 2022-10-26 16:47:41] ...trap/engines/engine.py.229: process: 0.49008 | load: 0.00242 [I 2022-10-26 16:47:41] ...trap/engines/engine.py.230: loss: 0.19797 [I 2022-10-26 16:47:41] ...optimizers/trijoint.py.151: total_norm: 3.933858 [I 2022-10-26 16:47:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 960/2304 [I 2022-10-26 16:47:46] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:11:08 [I 2022-10-26 16:47:46] ...trap/engines/engine.py.229: process: 0.48866 | load: 0.00241 [I 2022-10-26 16:47:46] ...trap/engines/engine.py.230: loss: 0.23107 [I 2022-10-26 16:47:46] ...optimizers/trijoint.py.151: total_norm: 3.496360 [I 2022-10-26 16:47:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 970/2304 [I 2022-10-26 16:47:51] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:11:08 [I 2022-10-26 16:47:51] ...trap/engines/engine.py.229: process: 0.49975 | load: 0.00241 [I 2022-10-26 16:47:51] ...trap/engines/engine.py.230: loss: 0.17303 [I 2022-10-26 16:47:51] ...optimizers/trijoint.py.151: total_norm: 2.377515 [I 2022-10-26 16:47:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 980/2304 [I 2022-10-26 16:47:56] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:10:59 [I 2022-10-26 16:47:56] ...trap/engines/engine.py.229: process: 0.49117 | load: 0.00242 [I 2022-10-26 16:47:56] ...trap/engines/engine.py.230: loss: 0.22342 [I 2022-10-26 16:47:56] ...optimizers/trijoint.py.151: total_norm: 3.996564 [I 2022-10-26 16:48:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 990/2304 [I 2022-10-26 16:48:01] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:10:55 [I 2022-10-26 16:48:01] ...trap/engines/engine.py.229: process: 0.50002 | load: 0.00243 [I 2022-10-26 16:48:01] ...trap/engines/engine.py.230: loss: 0.37609 [I 2022-10-26 16:48:01] ...optimizers/trijoint.py.151: total_norm: 5.547615 [I 2022-10-26 16:48:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1000/2304 [I 2022-10-26 16:48:06] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:10:48 [I 2022-10-26 16:48:06] ...trap/engines/engine.py.229: process: 0.50702 | load: 0.00260 [I 2022-10-26 16:48:06] ...trap/engines/engine.py.230: loss: 0.21289 [I 2022-10-26 16:48:06] ...optimizers/trijoint.py.151: total_norm: 2.897021 [I 2022-10-26 16:48:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1010/2304 [I 2022-10-26 16:48:11] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:10:48 [I 2022-10-26 16:48:11] ...trap/engines/engine.py.229: process: 0.51790 | load: 0.00242 [I 2022-10-26 16:48:11] ...trap/engines/engine.py.230: loss: 0.16454 [I 2022-10-26 16:48:11] ...optimizers/trijoint.py.151: total_norm: 2.755995 [I 2022-10-26 16:48:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1020/2304 [I 2022-10-26 16:48:16] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:10:43 [I 2022-10-26 16:48:16] ...trap/engines/engine.py.229: process: 0.50018 | load: 0.00245 [I 2022-10-26 16:48:16] ...trap/engines/engine.py.230: loss: 0.21560 [I 2022-10-26 16:48:16] ...optimizers/trijoint.py.151: total_norm: 3.394994 [I 2022-10-26 16:48:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 1030/2304 [I 2022-10-26 16:48:21] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:10:31 [I 2022-10-26 16:48:21] ...trap/engines/engine.py.229: process: 0.49320 | load: 0.00247 [I 2022-10-26 16:48:21] ...trap/engines/engine.py.230: loss: 0.18491 [I 2022-10-26 16:48:21] ...optimizers/trijoint.py.151: total_norm: 3.755931 [I 2022-10-26 16:48:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 1040/2304 [I 2022-10-26 16:48:26] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:10:28 [I 2022-10-26 16:48:26] ...trap/engines/engine.py.229: process: 0.49330 | load: 0.00241 [I 2022-10-26 16:48:26] ...trap/engines/engine.py.230: loss: 0.22461 [I 2022-10-26 16:48:26] ...optimizers/trijoint.py.151: total_norm: 2.879048 [I 2022-10-26 16:48:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 1050/2304 [I 2022-10-26 16:48:31] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:10:23 [I 2022-10-26 16:48:31] ...trap/engines/engine.py.229: process: 0.50605 | load: 0.00238 [I 2022-10-26 16:48:31] ...trap/engines/engine.py.230: loss: 0.19623 [I 2022-10-26 16:48:31] ...optimizers/trijoint.py.151: total_norm: 2.891865 [I 2022-10-26 16:48:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 1060/2304 [I 2022-10-26 16:48:36] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:10:15 [I 2022-10-26 16:48:36] ...trap/engines/engine.py.229: process: 0.49843 | load: 0.00241 [I 2022-10-26 16:48:36] ...trap/engines/engine.py.230: loss: 0.15099 [I 2022-10-26 16:48:36] ...optimizers/trijoint.py.151: total_norm: 2.955904 [I 2022-10-26 16:48:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 1070/2304 [I 2022-10-26 16:48:41] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:10:13 [I 2022-10-26 16:48:41] ...trap/engines/engine.py.229: process: 0.49306 | load: 0.00242 [I 2022-10-26 16:48:41] ...trap/engines/engine.py.230: loss: 0.21017 [I 2022-10-26 16:48:41] ...optimizers/trijoint.py.151: total_norm: 3.161836 [I 2022-10-26 16:48:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 1080/2304 [I 2022-10-26 16:48:46] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:10:11 [I 2022-10-26 16:48:46] ...trap/engines/engine.py.229: process: 0.49705 | load: 0.00243 [I 2022-10-26 16:48:46] ...trap/engines/engine.py.230: loss: 0.13356 [I 2022-10-26 16:48:46] ...optimizers/trijoint.py.151: total_norm: 2.971507 [I 2022-10-26 16:48:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 1090/2304 [I 2022-10-26 16:48:51] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:10:04 [I 2022-10-26 16:48:51] ...trap/engines/engine.py.229: process: 0.50292 | load: 0.00240 [I 2022-10-26 16:48:51] ...trap/engines/engine.py.230: loss: 0.18396 [I 2022-10-26 16:48:51] ...optimizers/trijoint.py.151: total_norm: 2.479537 [I 2022-10-26 16:48:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1100/2304 [I 2022-10-26 16:48:56] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:10:04 [I 2022-10-26 16:48:56] ...trap/engines/engine.py.229: process: 0.49204 | load: 0.00239 [I 2022-10-26 16:48:56] ...trap/engines/engine.py.230: loss: 0.22755 [I 2022-10-26 16:48:56] ...optimizers/trijoint.py.151: total_norm: 3.305209 [I 2022-10-26 16:49:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1110/2304 [I 2022-10-26 16:49:01] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:09:55 [I 2022-10-26 16:49:01] ...trap/engines/engine.py.229: process: 0.50241 | load: 0.00240 [I 2022-10-26 16:49:01] ...trap/engines/engine.py.230: loss: 0.19838 [I 2022-10-26 16:49:01] ...optimizers/trijoint.py.151: total_norm: 4.279033 [I 2022-10-26 16:49:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1120/2304 [I 2022-10-26 16:49:06] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:09:51 [I 2022-10-26 16:49:06] ...trap/engines/engine.py.229: process: 0.50486 | load: 0.00246 [I 2022-10-26 16:49:06] ...trap/engines/engine.py.230: loss: 0.18122 [I 2022-10-26 16:49:06] ...optimizers/trijoint.py.151: total_norm: 2.789137 [I 2022-10-26 16:49:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1130/2304 [I 2022-10-26 16:49:11] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:09:44 [I 2022-10-26 16:49:11] ...trap/engines/engine.py.229: process: 0.49958 | load: 0.00242 [I 2022-10-26 16:49:11] ...trap/engines/engine.py.230: loss: 0.20305 [I 2022-10-26 16:49:11] ...optimizers/trijoint.py.151: total_norm: 3.316754 [I 2022-10-26 16:49:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1140/2304 [I 2022-10-26 16:49:16] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:09:42 [I 2022-10-26 16:49:16] ...trap/engines/engine.py.229: process: 0.50143 | load: 0.00240 [I 2022-10-26 16:49:16] ...trap/engines/engine.py.230: loss: 0.15346 [I 2022-10-26 16:49:16] ...optimizers/trijoint.py.151: total_norm: 2.907149 [I 2022-10-26 16:49:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 1150/2304 [I 2022-10-26 16:49:21] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:09:32 [I 2022-10-26 16:49:21] ...trap/engines/engine.py.229: process: 0.48451 | load: 0.00242 [I 2022-10-26 16:49:21] ...trap/engines/engine.py.230: loss: 0.20715 [I 2022-10-26 16:49:21] ...optimizers/trijoint.py.151: total_norm: 3.349943 [I 2022-10-26 16:49:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 1160/2304 [I 2022-10-26 16:49:26] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:09:23 [I 2022-10-26 16:49:26] ...trap/engines/engine.py.229: process: 0.48328 | load: 0.00245 [I 2022-10-26 16:49:26] ...trap/engines/engine.py.230: loss: 0.23012 [I 2022-10-26 16:49:26] ...optimizers/trijoint.py.151: total_norm: 3.148493 [I 2022-10-26 16:49:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 1170/2304 [I 2022-10-26 16:49:31] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:09:20 [I 2022-10-26 16:49:31] ...trap/engines/engine.py.229: process: 0.49273 | load: 0.00244 [I 2022-10-26 16:49:31] ...trap/engines/engine.py.230: loss: 0.18468 [I 2022-10-26 16:49:31] ...optimizers/trijoint.py.151: total_norm: 3.072000 [I 2022-10-26 16:49:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 1180/2304 [I 2022-10-26 16:49:36] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:09:17 [I 2022-10-26 16:49:36] ...trap/engines/engine.py.229: process: 0.49507 | load: 0.00241 [I 2022-10-26 16:49:36] ...trap/engines/engine.py.230: loss: 0.23046 [I 2022-10-26 16:49:36] ...optimizers/trijoint.py.151: total_norm: 3.949185 [I 2022-10-26 16:49:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 1190/2304 [I 2022-10-26 16:49:41] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:09:11 [I 2022-10-26 16:49:41] ...trap/engines/engine.py.229: process: 0.49436 | load: 0.00252 [I 2022-10-26 16:49:41] ...trap/engines/engine.py.230: loss: 0.15755 [I 2022-10-26 16:49:41] ...optimizers/trijoint.py.151: total_norm: 2.299028 [I 2022-10-26 16:49:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 1200/2304 [I 2022-10-26 16:49:46] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:09:09 [I 2022-10-26 16:49:46] ...trap/engines/engine.py.229: process: 0.50648 | load: 0.00241 [I 2022-10-26 16:49:46] ...trap/engines/engine.py.230: loss: 0.15363 [I 2022-10-26 16:49:46] ...optimizers/trijoint.py.151: total_norm: 3.519425 [I 2022-10-26 16:49:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 1210/2304 [I 2022-10-26 16:49:51] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:09:05 [I 2022-10-26 16:49:51] ...trap/engines/engine.py.229: process: 0.50096 | load: 0.00263 [I 2022-10-26 16:49:51] ...trap/engines/engine.py.230: loss: 0.16690 [I 2022-10-26 16:49:51] ...optimizers/trijoint.py.151: total_norm: 3.133545 [I 2022-10-26 16:49:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1220/2304 [I 2022-10-26 16:49:56] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:09:18 [I 2022-10-26 16:49:56] ...trap/engines/engine.py.229: process: 0.50433 | load: 0.00241 [I 2022-10-26 16:49:56] ...trap/engines/engine.py.230: loss: 0.20192 [I 2022-10-26 16:49:56] ...optimizers/trijoint.py.151: total_norm: 2.716934 [I 2022-10-26 16:50:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1230/2304 [I 2022-10-26 16:50:01] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:08:57 [I 2022-10-26 16:50:01] ...trap/engines/engine.py.229: process: 0.48773 | load: 0.00241 [I 2022-10-26 16:50:01] ...trap/engines/engine.py.230: loss: 0.20754 [I 2022-10-26 16:50:01] ...optimizers/trijoint.py.151: total_norm: 2.774139 [I 2022-10-26 16:50:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1240/2304 [I 2022-10-26 16:50:06] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:08:50 [I 2022-10-26 16:50:06] ...trap/engines/engine.py.229: process: 0.49629 | load: 0.00249 [I 2022-10-26 16:50:06] ...trap/engines/engine.py.230: loss: 0.20043 [I 2022-10-26 16:50:06] ...optimizers/trijoint.py.151: total_norm: 2.926764 [I 2022-10-26 16:50:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1250/2304 [I 2022-10-26 16:50:11] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:08:45 [I 2022-10-26 16:50:11] ...trap/engines/engine.py.229: process: 0.50360 | load: 0.00248 [I 2022-10-26 16:50:11] ...trap/engines/engine.py.230: loss: 0.31387 [I 2022-10-26 16:50:11] ...optimizers/trijoint.py.151: total_norm: 4.032782 [I 2022-10-26 16:50:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1260/2304 [I 2022-10-26 16:50:16] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:08:40 [I 2022-10-26 16:50:16] ...trap/engines/engine.py.229: process: 0.51015 | load: 0.00264 [I 2022-10-26 16:50:16] ...trap/engines/engine.py.230: loss: 0.16628 [I 2022-10-26 16:50:16] ...optimizers/trijoint.py.151: total_norm: 2.673544 [I 2022-10-26 16:50:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 1270/2304 [I 2022-10-26 16:50:21] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:08:34 [I 2022-10-26 16:50:21] ...trap/engines/engine.py.229: process: 0.49685 | load: 0.00239 [I 2022-10-26 16:50:21] ...trap/engines/engine.py.230: loss: 0.19513 [I 2022-10-26 16:50:21] ...optimizers/trijoint.py.151: total_norm: 3.353286 [I 2022-10-26 16:50:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 1280/2304 [I 2022-10-26 16:50:26] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:08:33 [I 2022-10-26 16:50:26] ...trap/engines/engine.py.229: process: 0.49146 | load: 0.00241 [I 2022-10-26 16:50:26] ...trap/engines/engine.py.230: loss: 0.20993 [I 2022-10-26 16:50:26] ...optimizers/trijoint.py.151: total_norm: 2.534854 [I 2022-10-26 16:50:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 1290/2304 [I 2022-10-26 16:50:31] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:08:26 [I 2022-10-26 16:50:31] ...trap/engines/engine.py.229: process: 0.49901 | load: 0.00253 [I 2022-10-26 16:50:31] ...trap/engines/engine.py.230: loss: 0.14030 [I 2022-10-26 16:50:31] ...optimizers/trijoint.py.151: total_norm: 2.721395 [I 2022-10-26 16:50:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 1300/2304 [I 2022-10-26 16:50:36] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:08:22 [I 2022-10-26 16:50:36] ...trap/engines/engine.py.229: process: 0.50302 | load: 0.00265 [I 2022-10-26 16:50:36] ...trap/engines/engine.py.230: loss: 0.22453 [I 2022-10-26 16:50:36] ...optimizers/trijoint.py.151: total_norm: 3.410826 [I 2022-10-26 16:50:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 1310/2304 [I 2022-10-26 16:50:41] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:08:11 [I 2022-10-26 16:50:41] ...trap/engines/engine.py.229: process: 0.49454 | load: 0.00250 [I 2022-10-26 16:50:41] ...trap/engines/engine.py.230: loss: 0.20267 [I 2022-10-26 16:50:41] ...optimizers/trijoint.py.151: total_norm: 2.716912 [I 2022-10-26 16:50:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 1320/2304 [I 2022-10-26 16:50:46] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:08:09 [I 2022-10-26 16:50:46] ...trap/engines/engine.py.229: process: 0.50793 | load: 0.00247 [I 2022-10-26 16:50:46] ...trap/engines/engine.py.230: loss: 0.15056 [I 2022-10-26 16:50:46] ...optimizers/trijoint.py.151: total_norm: 2.990015 [I 2022-10-26 16:50:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 1330/2304 [I 2022-10-26 16:50:51] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:08:04 [I 2022-10-26 16:50:51] ...trap/engines/engine.py.229: process: 0.49181 | load: 0.00252 [I 2022-10-26 16:50:51] ...trap/engines/engine.py.230: loss: 0.17975 [I 2022-10-26 16:50:51] ...optimizers/trijoint.py.151: total_norm: 2.004395 [I 2022-10-26 16:50:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1340/2304 [I 2022-10-26 16:50:56] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:08:00 [I 2022-10-26 16:50:56] ...trap/engines/engine.py.229: process: 0.50242 | load: 0.00247 [I 2022-10-26 16:50:56] ...trap/engines/engine.py.230: loss: 0.15342 [I 2022-10-26 16:50:56] ...optimizers/trijoint.py.151: total_norm: 2.678207 [I 2022-10-26 16:51:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1350/2304 [I 2022-10-26 16:51:01] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:07:54 [I 2022-10-26 16:51:01] ...trap/engines/engine.py.229: process: 0.48766 | load: 0.00248 [I 2022-10-26 16:51:01] ...trap/engines/engine.py.230: loss: 0.20945 [I 2022-10-26 16:51:01] ...optimizers/trijoint.py.151: total_norm: 3.060419 [I 2022-10-26 16:51:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1360/2304 [I 2022-10-26 16:51:06] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:07:49 [I 2022-10-26 16:51:06] ...trap/engines/engine.py.229: process: 0.50142 | load: 0.00245 [I 2022-10-26 16:51:06] ...trap/engines/engine.py.230: loss: 0.19495 [I 2022-10-26 16:51:06] ...optimizers/trijoint.py.151: total_norm: 3.337306 [I 2022-10-26 16:51:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1370/2304 [I 2022-10-26 16:51:11] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:07:45 [I 2022-10-26 16:51:11] ...trap/engines/engine.py.229: process: 0.49418 | load: 0.00246 [I 2022-10-26 16:51:11] ...trap/engines/engine.py.230: loss: 0.14859 [I 2022-10-26 16:51:11] ...optimizers/trijoint.py.151: total_norm: 2.823878 [I 2022-10-26 16:51:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1380/2304 [I 2022-10-26 16:51:16] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:07:41 [I 2022-10-26 16:51:16] ...trap/engines/engine.py.229: process: 0.50147 | load: 0.00242 [I 2022-10-26 16:51:16] ...trap/engines/engine.py.230: loss: 0.17912 [I 2022-10-26 16:51:16] ...optimizers/trijoint.py.151: total_norm: 2.274846 [I 2022-10-26 16:51:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 1390/2304 [I 2022-10-26 16:51:21] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:07:35 [I 2022-10-26 16:51:21] ...trap/engines/engine.py.229: process: 0.49969 | load: 0.00254 [I 2022-10-26 16:51:21] ...trap/engines/engine.py.230: loss: 0.13724 [I 2022-10-26 16:51:21] ...optimizers/trijoint.py.151: total_norm: 2.160953 [I 2022-10-26 16:51:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 1400/2304 [I 2022-10-26 16:51:26] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:07:31 [I 2022-10-26 16:51:26] ...trap/engines/engine.py.229: process: 0.49058 | load: 0.00249 [I 2022-10-26 16:51:26] ...trap/engines/engine.py.230: loss: 0.27170 [I 2022-10-26 16:51:26] ...optimizers/trijoint.py.151: total_norm: 3.772790 [I 2022-10-26 16:51:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 1410/2304 [I 2022-10-26 16:51:31] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:07:23 [I 2022-10-26 16:51:31] ...trap/engines/engine.py.229: process: 0.49942 | load: 0.00244 [I 2022-10-26 16:51:31] ...trap/engines/engine.py.230: loss: 0.13637 [I 2022-10-26 16:51:31] ...optimizers/trijoint.py.151: total_norm: 1.873909 [I 2022-10-26 16:51:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 1420/2304 [I 2022-10-26 16:51:36] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:07:22 [I 2022-10-26 16:51:36] ...trap/engines/engine.py.229: process: 0.49977 | load: 0.00255 [I 2022-10-26 16:51:36] ...trap/engines/engine.py.230: loss: 0.17269 [I 2022-10-26 16:51:36] ...optimizers/trijoint.py.151: total_norm: 2.245533 [I 2022-10-26 16:51:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 1430/2304 [I 2022-10-26 16:51:41] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:07:20 [I 2022-10-26 16:51:41] ...trap/engines/engine.py.229: process: 0.51993 | load: 0.00241 [I 2022-10-26 16:51:41] ...trap/engines/engine.py.230: loss: 0.11137 [I 2022-10-26 16:51:41] ...optimizers/trijoint.py.151: total_norm: 2.588144 [I 2022-10-26 16:51:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 1440/2304 [I 2022-10-26 16:51:46] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:07:16 [I 2022-10-26 16:51:46] ...trap/engines/engine.py.229: process: 0.50282 | load: 0.00240 [I 2022-10-26 16:51:46] ...trap/engines/engine.py.230: loss: 0.21360 [I 2022-10-26 16:51:46] ...optimizers/trijoint.py.151: total_norm: 2.805378 [I 2022-10-26 16:51:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 1450/2304 [I 2022-10-26 16:51:51] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:07:07 [I 2022-10-26 16:51:51] ...trap/engines/engine.py.229: process: 0.48657 | load: 0.00249 [I 2022-10-26 16:51:51] ...trap/engines/engine.py.230: loss: 0.19479 [I 2022-10-26 16:51:51] ...optimizers/trijoint.py.151: total_norm: 2.976495 [I 2022-10-26 16:51:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1460/2304 [I 2022-10-26 16:51:56] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:07:01 [I 2022-10-26 16:51:56] ...trap/engines/engine.py.229: process: 0.50203 | load: 0.00245 [I 2022-10-26 16:51:56] ...trap/engines/engine.py.230: loss: 0.25736 [I 2022-10-26 16:51:56] ...optimizers/trijoint.py.151: total_norm: 2.951468 [I 2022-10-26 16:52:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1470/2304 [I 2022-10-26 16:52:01] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:06:55 [I 2022-10-26 16:52:01] ...trap/engines/engine.py.229: process: 0.48988 | load: 0.00243 [I 2022-10-26 16:52:01] ...trap/engines/engine.py.230: loss: 0.16333 [I 2022-10-26 16:52:01] ...optimizers/trijoint.py.151: total_norm: 2.977559 [I 2022-10-26 16:52:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1480/2304 [I 2022-10-26 16:52:06] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:06:54 [I 2022-10-26 16:52:06] ...trap/engines/engine.py.229: process: 0.50268 | load: 0.00243 [I 2022-10-26 16:52:06] ...trap/engines/engine.py.230: loss: 0.15786 [I 2022-10-26 16:52:06] ...optimizers/trijoint.py.151: total_norm: 2.686281 [I 2022-10-26 16:52:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1490/2304 [I 2022-10-26 16:52:11] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:06:43 [I 2022-10-26 16:52:11] ...trap/engines/engine.py.229: process: 0.49695 | load: 0.00241 [I 2022-10-26 16:52:11] ...trap/engines/engine.py.230: loss: 0.19067 [I 2022-10-26 16:52:11] ...optimizers/trijoint.py.151: total_norm: 2.610802 [I 2022-10-26 16:52:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1500/2304 [I 2022-10-26 16:52:16] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:06:41 [I 2022-10-26 16:52:16] ...trap/engines/engine.py.229: process: 0.49205 | load: 0.00247 [I 2022-10-26 16:52:16] ...trap/engines/engine.py.230: loss: 0.22212 [I 2022-10-26 16:52:16] ...optimizers/trijoint.py.151: total_norm: 3.690850 [I 2022-10-26 16:52:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 1510/2304 [I 2022-10-26 16:52:21] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:06:38 [I 2022-10-26 16:52:21] ...trap/engines/engine.py.229: process: 0.49653 | load: 0.00240 [I 2022-10-26 16:52:21] ...trap/engines/engine.py.230: loss: 0.16436 [I 2022-10-26 16:52:21] ...optimizers/trijoint.py.151: total_norm: 2.364083 [I 2022-10-26 16:52:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 1520/2304 [I 2022-10-26 16:52:26] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:06:32 [I 2022-10-26 16:52:26] ...trap/engines/engine.py.229: process: 0.50379 | load: 0.00247 [I 2022-10-26 16:52:26] ...trap/engines/engine.py.230: loss: 0.30322 [I 2022-10-26 16:52:26] ...optimizers/trijoint.py.151: total_norm: 3.451431 [I 2022-10-26 16:52:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 1530/2304 [I 2022-10-26 16:52:31] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:06:25 [I 2022-10-26 16:52:31] ...trap/engines/engine.py.229: process: 0.49264 | load: 0.00242 [I 2022-10-26 16:52:31] ...trap/engines/engine.py.230: loss: 0.16116 [I 2022-10-26 16:52:31] ...optimizers/trijoint.py.151: total_norm: 2.895604 [I 2022-10-26 16:52:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 1540/2304 [I 2022-10-26 16:52:36] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:06:19 [I 2022-10-26 16:52:36] ...trap/engines/engine.py.229: process: 0.49706 | load: 0.00250 [I 2022-10-26 16:52:36] ...trap/engines/engine.py.230: loss: 0.25816 [I 2022-10-26 16:52:36] ...optimizers/trijoint.py.151: total_norm: 3.817955 [I 2022-10-26 16:52:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 1550/2304 [I 2022-10-26 16:52:41] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:06:15 [I 2022-10-26 16:52:41] ...trap/engines/engine.py.229: process: 0.50226 | load: 0.00248 [I 2022-10-26 16:52:41] ...trap/engines/engine.py.230: loss: 0.12727 [I 2022-10-26 16:52:41] ...optimizers/trijoint.py.151: total_norm: 2.366564 [I 2022-10-26 16:52:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 1560/2304 [I 2022-10-26 16:52:46] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:06:11 [I 2022-10-26 16:52:46] ...trap/engines/engine.py.229: process: 0.50355 | load: 0.00241 [I 2022-10-26 16:52:46] ...trap/engines/engine.py.230: loss: 0.12973 [I 2022-10-26 16:52:46] ...optimizers/trijoint.py.151: total_norm: 2.937787 [I 2022-10-26 16:52:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 1570/2304 [I 2022-10-26 16:52:51] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:06:08 [I 2022-10-26 16:52:51] ...trap/engines/engine.py.229: process: 0.49711 | load: 0.00246 [I 2022-10-26 16:52:51] ...trap/engines/engine.py.230: loss: 0.22616 [I 2022-10-26 16:52:51] ...optimizers/trijoint.py.151: total_norm: 3.138369 [I 2022-10-26 16:52:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1580/2304 [I 2022-10-26 16:52:56] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:06:00 [I 2022-10-26 16:52:56] ...trap/engines/engine.py.229: process: 0.49515 | load: 0.00245 [I 2022-10-26 16:52:56] ...trap/engines/engine.py.230: loss: 0.21127 [I 2022-10-26 16:52:56] ...optimizers/trijoint.py.151: total_norm: 3.972171 [I 2022-10-26 16:53:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1590/2304 [I 2022-10-26 16:53:01] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:05:55 [I 2022-10-26 16:53:01] ...trap/engines/engine.py.229: process: 0.49978 | load: 0.00242 [I 2022-10-26 16:53:01] ...trap/engines/engine.py.230: loss: 0.27254 [I 2022-10-26 16:53:01] ...optimizers/trijoint.py.151: total_norm: 4.070036 [I 2022-10-26 16:53:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1600/2304 [I 2022-10-26 16:53:06] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:05:53 [I 2022-10-26 16:53:06] ...trap/engines/engine.py.229: process: 0.51698 | load: 0.00241 [I 2022-10-26 16:53:06] ...trap/engines/engine.py.230: loss: 0.25408 [I 2022-10-26 16:53:06] ...optimizers/trijoint.py.151: total_norm: 4.002151 [I 2022-10-26 16:53:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1610/2304 [I 2022-10-26 16:53:11] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:05:46 [I 2022-10-26 16:53:11] ...trap/engines/engine.py.229: process: 0.50025 | load: 0.00241 [I 2022-10-26 16:53:11] ...trap/engines/engine.py.230: loss: 0.21126 [I 2022-10-26 16:53:11] ...optimizers/trijoint.py.151: total_norm: 3.415708 [I 2022-10-26 16:53:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1620/2304 [I 2022-10-26 16:53:16] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:05:43 [I 2022-10-26 16:53:16] ...trap/engines/engine.py.229: process: 0.50706 | load: 0.00249 [I 2022-10-26 16:53:16] ...trap/engines/engine.py.230: loss: 0.22229 [I 2022-10-26 16:53:16] ...optimizers/trijoint.py.151: total_norm: 2.910297 [I 2022-10-26 16:53:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 1630/2304 [I 2022-10-26 16:53:21] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:05:32 [I 2022-10-26 16:53:21] ...trap/engines/engine.py.229: process: 0.48997 | load: 0.00248 [I 2022-10-26 16:53:21] ...trap/engines/engine.py.230: loss: 0.32178 [I 2022-10-26 16:53:21] ...optimizers/trijoint.py.151: total_norm: 3.934040 [I 2022-10-26 16:53:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 1640/2304 [I 2022-10-26 16:53:26] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:05:32 [I 2022-10-26 16:53:26] ...trap/engines/engine.py.229: process: 0.50373 | load: 0.00244 [I 2022-10-26 16:53:26] ...trap/engines/engine.py.230: loss: 0.22852 [I 2022-10-26 16:53:26] ...optimizers/trijoint.py.151: total_norm: 3.389839 [I 2022-10-26 16:53:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 1650/2304 [I 2022-10-26 16:53:31] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:05:26 [I 2022-10-26 16:53:31] ...trap/engines/engine.py.229: process: 0.49887 | load: 0.00245 [I 2022-10-26 16:53:31] ...trap/engines/engine.py.230: loss: 0.17160 [I 2022-10-26 16:53:31] ...optimizers/trijoint.py.151: total_norm: 2.814807 [I 2022-10-26 16:53:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 1660/2304 [I 2022-10-26 16:53:36] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:05:21 [I 2022-10-26 16:53:36] ...trap/engines/engine.py.229: process: 0.50519 | load: 0.00241 [I 2022-10-26 16:53:36] ...trap/engines/engine.py.230: loss: 0.19772 [I 2022-10-26 16:53:36] ...optimizers/trijoint.py.151: total_norm: 3.318899 [I 2022-10-26 16:53:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 1670/2304 [I 2022-10-26 16:53:41] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:05:14 [I 2022-10-26 16:53:41] ...trap/engines/engine.py.229: process: 0.50154 | load: 0.00243 [I 2022-10-26 16:53:41] ...trap/engines/engine.py.230: loss: 0.17861 [I 2022-10-26 16:53:41] ...optimizers/trijoint.py.151: total_norm: 2.713017 [I 2022-10-26 16:53:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 1680/2304 [I 2022-10-26 16:53:46] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:05:11 [I 2022-10-26 16:53:46] ...trap/engines/engine.py.229: process: 0.50018 | load: 0.00245 [I 2022-10-26 16:53:46] ...trap/engines/engine.py.230: loss: 0.24030 [I 2022-10-26 16:53:46] ...optimizers/trijoint.py.151: total_norm: 3.549753 [I 2022-10-26 16:53:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 1690/2304 [I 2022-10-26 16:53:51] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:05:04 [I 2022-10-26 16:53:51] ...trap/engines/engine.py.229: process: 0.49661 | load: 0.00240 [I 2022-10-26 16:53:51] ...trap/engines/engine.py.230: loss: 0.18882 [I 2022-10-26 16:53:51] ...optimizers/trijoint.py.151: total_norm: 3.093189 [I 2022-10-26 16:53:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1700/2304 [I 2022-10-26 16:53:56] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:05:00 [I 2022-10-26 16:53:56] ...trap/engines/engine.py.229: process: 0.50072 | load: 0.00244 [I 2022-10-26 16:53:56] ...trap/engines/engine.py.230: loss: 0.12671 [I 2022-10-26 16:53:56] ...optimizers/trijoint.py.151: total_norm: 1.859524 [I 2022-10-26 16:54:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1710/2304 [I 2022-10-26 16:54:01] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:04:54 [I 2022-10-26 16:54:01] ...trap/engines/engine.py.229: process: 0.49627 | load: 0.00250 [I 2022-10-26 16:54:01] ...trap/engines/engine.py.230: loss: 0.16747 [I 2022-10-26 16:54:01] ...optimizers/trijoint.py.151: total_norm: 2.622135 [I 2022-10-26 16:54:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1720/2304 [I 2022-10-26 16:54:06] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:04:50 [I 2022-10-26 16:54:06] ...trap/engines/engine.py.229: process: 0.50043 | load: 0.00247 [I 2022-10-26 16:54:06] ...trap/engines/engine.py.230: loss: 0.25806 [I 2022-10-26 16:54:06] ...optimizers/trijoint.py.151: total_norm: 4.655401 [I 2022-10-26 16:54:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1730/2304 [I 2022-10-26 16:54:11] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:04:45 [I 2022-10-26 16:54:11] ...trap/engines/engine.py.229: process: 0.49903 | load: 0.00248 [I 2022-10-26 16:54:11] ...trap/engines/engine.py.230: loss: 0.26208 [I 2022-10-26 16:54:11] ...optimizers/trijoint.py.151: total_norm: 4.249459 [I 2022-10-26 16:54:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1740/2304 [I 2022-10-26 16:54:16] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:04:39 [I 2022-10-26 16:54:16] ...trap/engines/engine.py.229: process: 0.48189 | load: 0.00245 [I 2022-10-26 16:54:16] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-26 16:54:16] ...optimizers/trijoint.py.151: total_norm: 1.735476 [I 2022-10-26 16:54:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 1750/2304 [I 2022-10-26 16:54:21] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:04:36 [I 2022-10-26 16:54:21] ...trap/engines/engine.py.229: process: 0.49467 | load: 0.00245 [I 2022-10-26 16:54:21] ...trap/engines/engine.py.230: loss: 0.19809 [I 2022-10-26 16:54:21] ...optimizers/trijoint.py.151: total_norm: 3.291406 [I 2022-10-26 16:54:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 1760/2304 [I 2022-10-26 16:54:26] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:04:32 [I 2022-10-26 16:54:26] ...trap/engines/engine.py.229: process: 0.50408 | load: 0.00252 [I 2022-10-26 16:54:26] ...trap/engines/engine.py.230: loss: 0.15350 [I 2022-10-26 16:54:26] ...optimizers/trijoint.py.151: total_norm: 3.336215 [I 2022-10-26 16:54:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 1770/2304 [I 2022-10-26 16:54:31] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:04:27 [I 2022-10-26 16:54:31] ...trap/engines/engine.py.229: process: 0.50052 | load: 0.00244 [I 2022-10-26 16:54:31] ...trap/engines/engine.py.230: loss: 0.17826 [I 2022-10-26 16:54:31] ...optimizers/trijoint.py.151: total_norm: 2.630206 [I 2022-10-26 16:54:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 1780/2304 [I 2022-10-26 16:54:36] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:04:20 [I 2022-10-26 16:54:36] ...trap/engines/engine.py.229: process: 0.48519 | load: 0.00245 [I 2022-10-26 16:54:36] ...trap/engines/engine.py.230: loss: 0.14182 [I 2022-10-26 16:54:36] ...optimizers/trijoint.py.151: total_norm: 2.803517 [I 2022-10-26 16:54:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 1790/2304 [I 2022-10-26 16:54:41] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:04:16 [I 2022-10-26 16:54:41] ...trap/engines/engine.py.229: process: 0.49077 | load: 0.00242 [I 2022-10-26 16:54:41] ...trap/engines/engine.py.230: loss: 0.18827 [I 2022-10-26 16:54:41] ...optimizers/trijoint.py.151: total_norm: 3.371628 [I 2022-10-26 16:54:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 1800/2304 [I 2022-10-26 16:54:46] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:04:10 [I 2022-10-26 16:54:46] ...trap/engines/engine.py.229: process: 0.48825 | load: 0.00242 [I 2022-10-26 16:54:46] ...trap/engines/engine.py.230: loss: 0.26319 [I 2022-10-26 16:54:46] ...optimizers/trijoint.py.151: total_norm: 4.512298 [I 2022-10-26 16:54:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 1810/2304 [I 2022-10-26 16:54:51] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:04:04 [I 2022-10-26 16:54:51] ...trap/engines/engine.py.229: process: 0.48892 | load: 0.00244 [I 2022-10-26 16:54:51] ...trap/engines/engine.py.230: loss: 0.29670 [I 2022-10-26 16:54:51] ...optimizers/trijoint.py.151: total_norm: 3.564594 [I 2022-10-26 16:54:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1820/2304 [I 2022-10-26 16:54:56] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:04:01 [I 2022-10-26 16:54:56] ...trap/engines/engine.py.229: process: 0.49697 | load: 0.00243 [I 2022-10-26 16:54:56] ...trap/engines/engine.py.230: loss: 0.24189 [I 2022-10-26 16:54:56] ...optimizers/trijoint.py.151: total_norm: 3.414878 [I 2022-10-26 16:55:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1830/2304 [I 2022-10-26 16:55:01] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:03:55 [I 2022-10-26 16:55:01] ...trap/engines/engine.py.229: process: 0.50121 | load: 0.00242 [I 2022-10-26 16:55:01] ...trap/engines/engine.py.230: loss: 0.25217 [I 2022-10-26 16:55:01] ...optimizers/trijoint.py.151: total_norm: 3.466854 [I 2022-10-26 16:55:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1840/2304 [I 2022-10-26 16:55:06] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:03:52 [I 2022-10-26 16:55:06] ...trap/engines/engine.py.229: process: 0.50405 | load: 0.00240 [I 2022-10-26 16:55:06] ...trap/engines/engine.py.230: loss: 0.27677 [I 2022-10-26 16:55:06] ...optimizers/trijoint.py.151: total_norm: 3.997501 [I 2022-10-26 16:55:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1850/2304 [I 2022-10-26 16:55:11] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:03:47 [I 2022-10-26 16:55:11] ...trap/engines/engine.py.229: process: 0.51461 | load: 0.00241 [I 2022-10-26 16:55:11] ...trap/engines/engine.py.230: loss: 0.27914 [I 2022-10-26 16:55:11] ...optimizers/trijoint.py.151: total_norm: 3.586936 [I 2022-10-26 16:55:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1860/2304 [I 2022-10-26 16:55:16] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:03:40 [I 2022-10-26 16:55:16] ...trap/engines/engine.py.229: process: 0.50098 | load: 0.00244 [I 2022-10-26 16:55:16] ...trap/engines/engine.py.230: loss: 0.18512 [I 2022-10-26 16:55:16] ...optimizers/trijoint.py.151: total_norm: 2.640248 [I 2022-10-26 16:55:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 1870/2304 [I 2022-10-26 16:55:21] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:03:39 [I 2022-10-26 16:55:21] ...trap/engines/engine.py.229: process: 0.50046 | load: 0.00241 [I 2022-10-26 16:55:21] ...trap/engines/engine.py.230: loss: 0.18723 [I 2022-10-26 16:55:21] ...optimizers/trijoint.py.151: total_norm: 2.912823 [I 2022-10-26 16:55:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 1880/2304 [I 2022-10-26 16:55:26] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:03:31 [I 2022-10-26 16:55:26] ...trap/engines/engine.py.229: process: 0.49627 | load: 0.00240 [I 2022-10-26 16:55:26] ...trap/engines/engine.py.230: loss: 0.22555 [I 2022-10-26 16:55:26] ...optimizers/trijoint.py.151: total_norm: 3.281707 [I 2022-10-26 16:55:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 1890/2304 [I 2022-10-26 16:55:31] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:03:29 [I 2022-10-26 16:55:31] ...trap/engines/engine.py.229: process: 0.50275 | load: 0.00260 [I 2022-10-26 16:55:31] ...trap/engines/engine.py.230: loss: 0.25169 [I 2022-10-26 16:55:31] ...optimizers/trijoint.py.151: total_norm: 3.629344 [I 2022-10-26 16:55:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 1900/2304 [I 2022-10-26 16:55:36] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:03:21 [I 2022-10-26 16:55:36] ...trap/engines/engine.py.229: process: 0.49249 | load: 0.00249 [I 2022-10-26 16:55:36] ...trap/engines/engine.py.230: loss: 0.13473 [I 2022-10-26 16:55:36] ...optimizers/trijoint.py.151: total_norm: 2.591171 [I 2022-10-26 16:55:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 1910/2304 [I 2022-10-26 16:55:41] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:03:16 [I 2022-10-26 16:55:41] ...trap/engines/engine.py.229: process: 0.49676 | load: 0.00240 [I 2022-10-26 16:55:41] ...trap/engines/engine.py.230: loss: 0.17926 [I 2022-10-26 16:55:41] ...optimizers/trijoint.py.151: total_norm: 2.875439 [I 2022-10-26 16:55:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 1920/2304 [I 2022-10-26 16:55:46] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:03:10 [I 2022-10-26 16:55:46] ...trap/engines/engine.py.229: process: 0.50460 | load: 0.00242 [I 2022-10-26 16:55:46] ...trap/engines/engine.py.230: loss: 0.24045 [I 2022-10-26 16:55:46] ...optimizers/trijoint.py.151: total_norm: 3.813216 [I 2022-10-26 16:55:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 1930/2304 [I 2022-10-26 16:55:51] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:03:05 [I 2022-10-26 16:55:51] ...trap/engines/engine.py.229: process: 0.49978 | load: 0.00249 [I 2022-10-26 16:55:51] ...trap/engines/engine.py.230: loss: 0.21459 [I 2022-10-26 16:55:51] ...optimizers/trijoint.py.151: total_norm: 4.172851 [I 2022-10-26 16:55:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 1940/2304 [I 2022-10-26 16:55:56] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:03:01 [I 2022-10-26 16:55:56] ...trap/engines/engine.py.229: process: 0.49566 | load: 0.00247 [I 2022-10-26 16:55:56] ...trap/engines/engine.py.230: loss: 0.20144 [I 2022-10-26 16:55:56] ...optimizers/trijoint.py.151: total_norm: 3.407038 [I 2022-10-26 16:56:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 1950/2304 [I 2022-10-26 16:56:01] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:02:57 [I 2022-10-26 16:56:01] ...trap/engines/engine.py.229: process: 0.49631 | load: 0.00243 [I 2022-10-26 16:56:01] ...trap/engines/engine.py.230: loss: 0.21420 [I 2022-10-26 16:56:01] ...optimizers/trijoint.py.151: total_norm: 3.435265 [I 2022-10-26 16:56:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 1960/2304 [I 2022-10-26 16:56:06] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:02:53 [I 2022-10-26 16:56:06] ...trap/engines/engine.py.229: process: 0.50019 | load: 0.00248 [I 2022-10-26 16:56:06] ...trap/engines/engine.py.230: loss: 0.11220 [I 2022-10-26 16:56:06] ...optimizers/trijoint.py.151: total_norm: 2.377592 [I 2022-10-26 16:56:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 1970/2304 [I 2022-10-26 16:56:11] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:02:47 [I 2022-10-26 16:56:11] ...trap/engines/engine.py.229: process: 0.50141 | load: 0.00239 [I 2022-10-26 16:56:11] ...trap/engines/engine.py.230: loss: 0.23140 [I 2022-10-26 16:56:11] ...optimizers/trijoint.py.151: total_norm: 2.853997 [I 2022-10-26 16:56:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 1980/2304 [I 2022-10-26 16:56:16] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:02:40 [I 2022-10-26 16:56:16] ...trap/engines/engine.py.229: process: 0.48469 | load: 0.00245 [I 2022-10-26 16:56:16] ...trap/engines/engine.py.230: loss: 0.19350 [I 2022-10-26 16:56:16] ...optimizers/trijoint.py.151: total_norm: 3.230179 [I 2022-10-26 16:56:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 1990/2304 [I 2022-10-26 16:56:21] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:02:36 [I 2022-10-26 16:56:21] ...trap/engines/engine.py.229: process: 0.49897 | load: 0.00241 [I 2022-10-26 16:56:21] ...trap/engines/engine.py.230: loss: 0.18229 [I 2022-10-26 16:56:21] ...optimizers/trijoint.py.151: total_norm: 3.145446 [I 2022-10-26 16:56:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 2000/2304 [I 2022-10-26 16:56:26] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:02:31 [I 2022-10-26 16:56:26] ...trap/engines/engine.py.229: process: 0.50407 | load: 0.00244 [I 2022-10-26 16:56:26] ...trap/engines/engine.py.230: loss: 0.27233 [I 2022-10-26 16:56:26] ...optimizers/trijoint.py.151: total_norm: 3.942873 [I 2022-10-26 16:56:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 2010/2304 [I 2022-10-26 16:56:31] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:02:27 [I 2022-10-26 16:56:31] ...trap/engines/engine.py.229: process: 0.50606 | load: 0.00243 [I 2022-10-26 16:56:31] ...trap/engines/engine.py.230: loss: 0.15356 [I 2022-10-26 16:56:31] ...optimizers/trijoint.py.151: total_norm: 3.543702 [I 2022-10-26 16:56:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 2020/2304 [I 2022-10-26 16:56:36] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:02:21 [I 2022-10-26 16:56:36] ...trap/engines/engine.py.229: process: 0.49982 | load: 0.00248 [I 2022-10-26 16:56:36] ...trap/engines/engine.py.230: loss: 0.16397 [I 2022-10-26 16:56:36] ...optimizers/trijoint.py.151: total_norm: 3.945082 [I 2022-10-26 16:56:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 2030/2304 [I 2022-10-26 16:56:41] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:02:19 [I 2022-10-26 16:56:41] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00244 [I 2022-10-26 16:56:41] ...trap/engines/engine.py.230: loss: 0.18619 [I 2022-10-26 16:56:41] ...optimizers/trijoint.py.151: total_norm: 2.852032 [I 2022-10-26 16:56:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 2040/2304 [I 2022-10-26 16:56:46] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:02:12 [I 2022-10-26 16:56:46] ...trap/engines/engine.py.229: process: 0.49925 | load: 0.00242 [I 2022-10-26 16:56:46] ...trap/engines/engine.py.230: loss: 0.26098 [I 2022-10-26 16:56:46] ...optimizers/trijoint.py.151: total_norm: 3.913185 [I 2022-10-26 16:56:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 2050/2304 [I 2022-10-26 16:56:51] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:02:06 [I 2022-10-26 16:56:51] ...trap/engines/engine.py.229: process: 0.49418 | load: 0.00244 [I 2022-10-26 16:56:51] ...trap/engines/engine.py.230: loss: 0.19779 [I 2022-10-26 16:56:51] ...optimizers/trijoint.py.151: total_norm: 3.740304 [I 2022-10-26 16:56:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 2060/2304 [I 2022-10-26 16:56:56] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:02:01 [I 2022-10-26 16:56:56] ...trap/engines/engine.py.229: process: 0.48601 | load: 0.00243 [I 2022-10-26 16:56:56] ...trap/engines/engine.py.230: loss: 0.16820 [I 2022-10-26 16:56:56] ...optimizers/trijoint.py.151: total_norm: 3.517972 [I 2022-10-26 16:57:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 2070/2304 [I 2022-10-26 16:57:01] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:01:56 [I 2022-10-26 16:57:01] ...trap/engines/engine.py.229: process: 0.50413 | load: 0.00242 [I 2022-10-26 16:57:01] ...trap/engines/engine.py.230: loss: 0.25228 [I 2022-10-26 16:57:01] ...optimizers/trijoint.py.151: total_norm: 3.925447 [I 2022-10-26 16:57:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 2080/2304 [I 2022-10-26 16:57:06] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:01:51 [I 2022-10-26 16:57:06] ...trap/engines/engine.py.229: process: 0.49048 | load: 0.00240 [I 2022-10-26 16:57:06] ...trap/engines/engine.py.230: loss: 0.20941 [I 2022-10-26 16:57:06] ...optimizers/trijoint.py.151: total_norm: 2.942691 [I 2022-10-26 16:57:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 2090/2304 [I 2022-10-26 16:57:11] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:01:46 [I 2022-10-26 16:57:11] ...trap/engines/engine.py.229: process: 0.51260 | load: 0.00259 [I 2022-10-26 16:57:11] ...trap/engines/engine.py.230: loss: 0.21205 [I 2022-10-26 16:57:11] ...optimizers/trijoint.py.151: total_norm: 3.335663 [I 2022-10-26 16:57:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 2100/2304 [I 2022-10-26 16:57:16] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:01:44 [I 2022-10-26 16:57:16] ...trap/engines/engine.py.229: process: 0.58885 | load: 0.00241 [I 2022-10-26 16:57:16] ...trap/engines/engine.py.230: loss: 0.24014 [I 2022-10-26 16:57:16] ...optimizers/trijoint.py.151: total_norm: 3.471183 [I 2022-10-26 16:57:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 2110/2304 [I 2022-10-26 16:57:21] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:01:37 [I 2022-10-26 16:57:21] ...trap/engines/engine.py.229: process: 0.49674 | load: 0.00250 [I 2022-10-26 16:57:21] ...trap/engines/engine.py.230: loss: 0.21750 [I 2022-10-26 16:57:21] ...optimizers/trijoint.py.151: total_norm: 2.946150 [I 2022-10-26 16:57:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 2120/2304 [I 2022-10-26 16:57:26] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:01:31 [I 2022-10-26 16:57:26] ...trap/engines/engine.py.229: process: 0.49901 | load: 0.00244 [I 2022-10-26 16:57:26] ...trap/engines/engine.py.230: loss: 0.18617 [I 2022-10-26 16:57:26] ...optimizers/trijoint.py.151: total_norm: 3.043845 [I 2022-10-26 16:57:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 2130/2304 [I 2022-10-26 16:57:31] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:01:26 [I 2022-10-26 16:57:31] ...trap/engines/engine.py.229: process: 0.49950 | load: 0.00243 [I 2022-10-26 16:57:31] ...trap/engines/engine.py.230: loss: 0.15207 [I 2022-10-26 16:57:31] ...optimizers/trijoint.py.151: total_norm: 2.603472 [I 2022-10-26 16:57:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 2140/2304 [I 2022-10-26 16:57:36] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:01:21 [I 2022-10-26 16:57:36] ...trap/engines/engine.py.229: process: 0.49666 | load: 0.00245 [I 2022-10-26 16:57:36] ...trap/engines/engine.py.230: loss: 0.14906 [I 2022-10-26 16:57:36] ...optimizers/trijoint.py.151: total_norm: 2.854475 [I 2022-10-26 16:57:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 2150/2304 [I 2022-10-26 16:57:41] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:01:16 [I 2022-10-26 16:57:41] ...trap/engines/engine.py.229: process: 0.49653 | load: 0.00248 [I 2022-10-26 16:57:41] ...trap/engines/engine.py.230: loss: 0.27661 [I 2022-10-26 16:57:41] ...optimizers/trijoint.py.151: total_norm: 3.263793 [I 2022-10-26 16:57:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 2160/2304 [I 2022-10-26 16:57:46] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:01:11 [I 2022-10-26 16:57:46] ...trap/engines/engine.py.229: process: 0.50689 | load: 0.00246 [I 2022-10-26 16:57:46] ...trap/engines/engine.py.230: loss: 0.17125 [I 2022-10-26 16:57:46] ...optimizers/trijoint.py.151: total_norm: 2.190676 [I 2022-10-26 16:57:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 2170/2304 [I 2022-10-26 16:57:51] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:01:06 [I 2022-10-26 16:57:51] ...trap/engines/engine.py.229: process: 0.49447 | load: 0.00245 [I 2022-10-26 16:57:51] ...trap/engines/engine.py.230: loss: 0.18536 [I 2022-10-26 16:57:51] ...optimizers/trijoint.py.151: total_norm: 3.519717 [I 2022-10-26 16:57:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 2180/2304 [I 2022-10-26 16:57:56] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:01:02 [I 2022-10-26 16:57:56] ...trap/engines/engine.py.229: process: 0.48658 | load: 0.00244 [I 2022-10-26 16:57:56] ...trap/engines/engine.py.230: loss: 0.22800 [I 2022-10-26 16:57:56] ...optimizers/trijoint.py.151: total_norm: 3.224315 [I 2022-10-26 16:58:01] ...trap/engines/engine.py.224: train: epoch 19 | batch 2190/2304 [I 2022-10-26 16:58:01] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:00:56 [I 2022-10-26 16:58:01] ...trap/engines/engine.py.229: process: 0.49783 | load: 0.00239 [I 2022-10-26 16:58:01] ...trap/engines/engine.py.230: loss: 0.17369 [I 2022-10-26 16:58:01] ...optimizers/trijoint.py.151: total_norm: 2.567888 [I 2022-10-26 16:58:06] ...trap/engines/engine.py.224: train: epoch 19 | batch 2200/2304 [I 2022-10-26 16:58:06] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:00:52 [I 2022-10-26 16:58:06] ...trap/engines/engine.py.229: process: 0.51477 | load: 0.00239 [I 2022-10-26 16:58:06] ...trap/engines/engine.py.230: loss: 0.18760 [I 2022-10-26 16:58:06] ...optimizers/trijoint.py.151: total_norm: 3.070507 [I 2022-10-26 16:58:11] ...trap/engines/engine.py.224: train: epoch 19 | batch 2210/2304 [I 2022-10-26 16:58:11] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:00:46 [I 2022-10-26 16:58:11] ...trap/engines/engine.py.229: process: 0.50052 | load: 0.00251 [I 2022-10-26 16:58:11] ...trap/engines/engine.py.230: loss: 0.20194 [I 2022-10-26 16:58:11] ...optimizers/trijoint.py.151: total_norm: 3.015250 [I 2022-10-26 16:58:16] ...trap/engines/engine.py.224: train: epoch 19 | batch 2220/2304 [I 2022-10-26 16:58:16] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:00:41 [I 2022-10-26 16:58:16] ...trap/engines/engine.py.229: process: 0.49988 | load: 0.00244 [I 2022-10-26 16:58:16] ...trap/engines/engine.py.230: loss: 0.22530 [I 2022-10-26 16:58:16] ...optimizers/trijoint.py.151: total_norm: 3.761553 [I 2022-10-26 16:58:21] ...trap/engines/engine.py.224: train: epoch 19 | batch 2230/2304 [I 2022-10-26 16:58:21] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:00:36 [I 2022-10-26 16:58:21] ...trap/engines/engine.py.229: process: 0.49424 | load: 0.00243 [I 2022-10-26 16:58:21] ...trap/engines/engine.py.230: loss: 0.27818 [I 2022-10-26 16:58:21] ...optimizers/trijoint.py.151: total_norm: 3.880800 [I 2022-10-26 16:58:26] ...trap/engines/engine.py.224: train: epoch 19 | batch 2240/2304 [I 2022-10-26 16:58:26] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:00:31 [I 2022-10-26 16:58:26] ...trap/engines/engine.py.229: process: 0.49990 | load: 0.00262 [I 2022-10-26 16:58:26] ...trap/engines/engine.py.230: loss: 0.20869 [I 2022-10-26 16:58:26] ...optimizers/trijoint.py.151: total_norm: 3.616962 [I 2022-10-26 16:58:31] ...trap/engines/engine.py.224: train: epoch 19 | batch 2250/2304 [I 2022-10-26 16:58:31] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:00:27 [I 2022-10-26 16:58:31] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00265 [I 2022-10-26 16:58:31] ...trap/engines/engine.py.230: loss: 0.19373 [I 2022-10-26 16:58:31] ...optimizers/trijoint.py.151: total_norm: 3.585206 [I 2022-10-26 16:58:36] ...trap/engines/engine.py.224: train: epoch 19 | batch 2260/2304 [I 2022-10-26 16:58:36] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:00:22 [I 2022-10-26 16:58:36] ...trap/engines/engine.py.229: process: 0.49420 | load: 0.00242 [I 2022-10-26 16:58:36] ...trap/engines/engine.py.230: loss: 0.32054 [I 2022-10-26 16:58:36] ...optimizers/trijoint.py.151: total_norm: 3.792511 [I 2022-10-26 16:58:41] ...trap/engines/engine.py.224: train: epoch 19 | batch 2270/2304 [I 2022-10-26 16:58:41] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:00:16 [I 2022-10-26 16:58:41] ...trap/engines/engine.py.229: process: 0.48436 | load: 0.00242 [I 2022-10-26 16:58:41] ...trap/engines/engine.py.230: loss: 0.19855 [I 2022-10-26 16:58:41] ...optimizers/trijoint.py.151: total_norm: 3.295885 [I 2022-10-26 16:58:46] ...trap/engines/engine.py.224: train: epoch 19 | batch 2280/2304 [I 2022-10-26 16:58:46] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:00:11 [I 2022-10-26 16:58:46] ...trap/engines/engine.py.229: process: 0.49529 | load: 0.00240 [I 2022-10-26 16:58:46] ...trap/engines/engine.py.230: loss: 0.12665 [I 2022-10-26 16:58:46] ...optimizers/trijoint.py.151: total_norm: 2.099314 [I 2022-10-26 16:58:51] ...trap/engines/engine.py.224: train: epoch 19 | batch 2290/2304 [I 2022-10-26 16:58:51] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:00:06 [I 2022-10-26 16:58:51] ...trap/engines/engine.py.229: process: 0.49482 | load: 0.00260 [I 2022-10-26 16:58:51] ...trap/engines/engine.py.230: loss: 0.13391 [I 2022-10-26 16:58:51] ...optimizers/trijoint.py.151: total_norm: 2.125346 [I 2022-10-26 16:58:56] ...trap/engines/engine.py.224: train: epoch 19 | batch 2300/2304 [I 2022-10-26 16:58:56] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:00:02 [I 2022-10-26 16:58:56] ...trap/engines/engine.py.229: process: 0.49596 | load: 0.00015 [I 2022-10-26 16:58:56] ...trap/engines/engine.py.230: loss: 0.27013 [I 2022-10-26 16:58:56] ...optimizers/trijoint.py.151: total_norm: 3.233510 [I 2022-10-26 16:58:58] ...trap/engines/engine.py.224: train: epoch 19 | batch 2304/2304 [I 2022-10-26 16:58:58] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:00:00 [I 2022-10-26 16:58:58] ...trap/engines/engine.py.229: process: 0.48632 | load: 0.00015 [I 2022-10-26 16:58:58] ...trap/engines/engine.py.230: loss: 0.14498 [I 2022-10-26 16:58:58] ...optimizers/trijoint.py.151: total_norm: 2.575110 [S 2022-10-26 16:58:59] ...trap/engines/engine.py.129: train_epoch.epoch: 19 [S 2022-10-26 16:58:59] ...trap/engines/engine.py.129: train_epoch.loss: 0.20389264103453486 [I 2022-10-26 16:58:59] ...s/criterions/triplet.py.84: margin = 0.15000000000000008 [I 2022-10-26 16:59:00] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 19 [I 2022-10-26 16:59:00] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 16:59:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 16:59:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 16:59:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 16:59:02] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 16:59:02] ...trap/engines/engine.py.321: eval: epoch 19 | batch 0/510 [I 2022-10-26 16:59:02] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:23:46 [I 2022-10-26 16:59:02] ...trap/engines/engine.py.326: process: 2.79674 | load: 2.42535 [I 2022-10-26 16:59:06] ...trap/engines/engine.py.321: eval: epoch 19 | batch 10/510 [I 2022-10-26 16:59:06] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:07 [I 2022-10-26 16:59:06] ...trap/engines/engine.py.326: process: 0.32400 | load: 0.00024 [I 2022-10-26 16:59:09] ...trap/engines/engine.py.321: eval: epoch 19 | batch 20/510 [I 2022-10-26 16:59:09] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:56 [I 2022-10-26 16:59:09] ...trap/engines/engine.py.326: process: 0.32291 | load: 0.00019 [I 2022-10-26 16:59:12] ...trap/engines/engine.py.321: eval: epoch 19 | batch 30/510 [I 2022-10-26 16:59:12] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:40 [I 2022-10-26 16:59:12] ...trap/engines/engine.py.326: process: 0.32203 | load: 0.00017 [I 2022-10-26 16:59:16] ...trap/engines/engine.py.321: eval: epoch 19 | batch 40/510 [I 2022-10-26 16:59:16] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:37 [I 2022-10-26 16:59:16] ...trap/engines/engine.py.326: process: 0.35733 | load: 0.00023 [I 2022-10-26 16:59:19] ...trap/engines/engine.py.321: eval: epoch 19 | batch 50/510 [I 2022-10-26 16:59:19] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-26 16:59:19] ...trap/engines/engine.py.326: process: 0.35980 | load: 0.00017 [I 2022-10-26 16:59:22] ...trap/engines/engine.py.321: eval: epoch 19 | batch 60/510 [I 2022-10-26 16:59:22] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:30 [I 2022-10-26 16:59:22] ...trap/engines/engine.py.326: process: 0.32655 | load: 0.00020 [I 2022-10-26 16:59:26] ...trap/engines/engine.py.321: eval: epoch 19 | batch 70/510 [I 2022-10-26 16:59:26] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:30 [I 2022-10-26 16:59:26] ...trap/engines/engine.py.326: process: 0.37225 | load: 0.00017 [I 2022-10-26 16:59:29] ...trap/engines/engine.py.321: eval: epoch 19 | batch 80/510 [I 2022-10-26 16:59:29] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:24 [I 2022-10-26 16:59:29] ...trap/engines/engine.py.326: process: 0.32050 | load: 0.00017 [I 2022-10-26 16:59:33] ...trap/engines/engine.py.321: eval: epoch 19 | batch 90/510 [I 2022-10-26 16:59:33] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:21 [I 2022-10-26 16:59:33] ...trap/engines/engine.py.326: process: 0.33173 | load: 0.00018 [I 2022-10-26 16:59:36] ...trap/engines/engine.py.321: eval: epoch 19 | batch 100/510 [I 2022-10-26 16:59:36] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:12 [I 2022-10-26 16:59:36] ...trap/engines/engine.py.326: process: 0.29111 | load: 0.00017 [I 2022-10-26 16:59:39] ...trap/engines/engine.py.321: eval: epoch 19 | batch 110/510 [I 2022-10-26 16:59:39] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 16:59:39] ...trap/engines/engine.py.326: process: 0.29070 | load: 0.00016 [I 2022-10-26 16:59:42] ...trap/engines/engine.py.321: eval: epoch 19 | batch 120/510 [I 2022-10-26 16:59:42] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-26 16:59:42] ...trap/engines/engine.py.326: process: 0.28379 | load: 0.00019 [I 2022-10-26 16:59:45] ...trap/engines/engine.py.321: eval: epoch 19 | batch 130/510 [I 2022-10-26 16:59:45] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-26 16:59:45] ...trap/engines/engine.py.326: process: 0.30517 | load: 0.00016 [I 2022-10-26 16:59:47] ...trap/engines/engine.py.321: eval: epoch 19 | batch 140/510 [I 2022-10-26 16:59:47] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:46 [I 2022-10-26 16:59:47] ...trap/engines/engine.py.326: process: 0.28121 | load: 0.00016 [I 2022-10-26 16:59:50] ...trap/engines/engine.py.321: eval: epoch 19 | batch 150/510 [I 2022-10-26 16:59:50] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:45 [I 2022-10-26 16:59:50] ...trap/engines/engine.py.326: process: 0.29030 | load: 0.00020 [I 2022-10-26 16:59:53] ...trap/engines/engine.py.321: eval: epoch 19 | batch 160/510 [I 2022-10-26 16:59:53] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:41 [I 2022-10-26 16:59:53] ...trap/engines/engine.py.326: process: 0.28580 | load: 0.00015 [I 2022-10-26 16:59:56] ...trap/engines/engine.py.321: eval: epoch 19 | batch 170/510 [I 2022-10-26 16:59:56] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:39 [I 2022-10-26 16:59:56] ...trap/engines/engine.py.326: process: 0.30966 | load: 0.00015 [I 2022-10-26 16:59:59] ...trap/engines/engine.py.321: eval: epoch 19 | batch 180/510 [I 2022-10-26 16:59:59] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 16:59:59] ...trap/engines/engine.py.326: process: 0.29267 | load: 0.00014 [I 2022-10-26 17:00:02] ...trap/engines/engine.py.321: eval: epoch 19 | batch 190/510 [I 2022-10-26 17:00:02] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:31 [I 2022-10-26 17:00:02] ...trap/engines/engine.py.326: process: 0.29004 | load: 0.00014 [I 2022-10-26 17:00:05] ...trap/engines/engine.py.321: eval: epoch 19 | batch 200/510 [I 2022-10-26 17:00:05] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:28 [I 2022-10-26 17:00:05] ...trap/engines/engine.py.326: process: 0.28150 | load: 0.00018 [I 2022-10-26 17:00:08] ...trap/engines/engine.py.321: eval: epoch 19 | batch 210/510 [I 2022-10-26 17:00:08] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 17:00:08] ...trap/engines/engine.py.326: process: 0.28605 | load: 0.00015 [I 2022-10-26 17:00:11] ...trap/engines/engine.py.321: eval: epoch 19 | batch 220/510 [I 2022-10-26 17:00:11] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:23 [I 2022-10-26 17:00:11] ...trap/engines/engine.py.326: process: 0.28685 | load: 0.00016 [I 2022-10-26 17:00:13] ...trap/engines/engine.py.321: eval: epoch 19 | batch 230/510 [I 2022-10-26 17:00:13] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:21 [I 2022-10-26 17:00:13] ...trap/engines/engine.py.326: process: 0.28861 | load: 0.00015 [I 2022-10-26 17:00:16] ...trap/engines/engine.py.321: eval: epoch 19 | batch 240/510 [I 2022-10-26 17:00:16] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:17 [I 2022-10-26 17:00:16] ...trap/engines/engine.py.326: process: 0.28782 | load: 0.00015 [I 2022-10-26 17:00:19] ...trap/engines/engine.py.321: eval: epoch 19 | batch 250/510 [I 2022-10-26 17:00:19] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:15 [I 2022-10-26 17:00:19] ...trap/engines/engine.py.326: process: 0.28290 | load: 0.00018 [I 2022-10-26 17:00:22] ...trap/engines/engine.py.321: eval: epoch 19 | batch 260/510 [I 2022-10-26 17:00:22] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:13 [I 2022-10-26 17:00:22] ...trap/engines/engine.py.326: process: 0.30480 | load: 0.00015 [I 2022-10-26 17:00:25] ...trap/engines/engine.py.321: eval: epoch 19 | batch 270/510 [I 2022-10-26 17:00:25] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 17:00:25] ...trap/engines/engine.py.326: process: 0.30651 | load: 0.00015 [I 2022-10-26 17:00:28] ...trap/engines/engine.py.321: eval: epoch 19 | batch 280/510 [I 2022-10-26 17:00:28] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 17:00:28] ...trap/engines/engine.py.326: process: 0.29460 | load: 0.00014 [I 2022-10-26 17:00:31] ...trap/engines/engine.py.321: eval: epoch 19 | batch 290/510 [I 2022-10-26 17:00:31] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 17:00:31] ...trap/engines/engine.py.326: process: 0.28573 | load: 0.00019 [I 2022-10-26 17:00:34] ...trap/engines/engine.py.321: eval: epoch 19 | batch 300/510 [I 2022-10-26 17:00:34] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 17:00:34] ...trap/engines/engine.py.326: process: 0.28717 | load: 0.00015 [I 2022-10-26 17:00:37] ...trap/engines/engine.py.321: eval: epoch 19 | batch 310/510 [I 2022-10-26 17:00:37] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 17:00:37] ...trap/engines/engine.py.326: process: 0.29030 | load: 0.00016 [I 2022-10-26 17:00:39] ...trap/engines/engine.py.321: eval: epoch 19 | batch 320/510 [I 2022-10-26 17:00:39] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:54 [I 2022-10-26 17:00:39] ...trap/engines/engine.py.326: process: 0.28742 | load: 0.00015 [I 2022-10-26 17:00:42] ...trap/engines/engine.py.321: eval: epoch 19 | batch 330/510 [I 2022-10-26 17:00:42] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:52 [I 2022-10-26 17:00:42] ...trap/engines/engine.py.326: process: 0.29089 | load: 0.00014 [I 2022-10-26 17:00:45] ...trap/engines/engine.py.321: eval: epoch 19 | batch 340/510 [I 2022-10-26 17:00:45] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:48 [I 2022-10-26 17:00:45] ...trap/engines/engine.py.326: process: 0.28218 | load: 0.00015 [I 2022-10-26 17:00:48] ...trap/engines/engine.py.321: eval: epoch 19 | batch 350/510 [I 2022-10-26 17:00:48] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:45 [I 2022-10-26 17:00:48] ...trap/engines/engine.py.326: process: 0.29018 | load: 0.00015 [I 2022-10-26 17:00:51] ...trap/engines/engine.py.321: eval: epoch 19 | batch 360/510 [I 2022-10-26 17:00:51] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 17:00:51] ...trap/engines/engine.py.326: process: 0.28906 | load: 0.00032 [I 2022-10-26 17:00:54] ...trap/engines/engine.py.321: eval: epoch 19 | batch 370/510 [I 2022-10-26 17:00:54] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 17:00:54] ...trap/engines/engine.py.326: process: 0.29235 | load: 0.00015 [I 2022-10-26 17:00:57] ...trap/engines/engine.py.321: eval: epoch 19 | batch 380/510 [I 2022-10-26 17:00:57] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 17:00:57] ...trap/engines/engine.py.326: process: 0.28606 | load: 0.00015 [I 2022-10-26 17:01:00] ...trap/engines/engine.py.321: eval: epoch 19 | batch 390/510 [I 2022-10-26 17:01:00] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 17:01:00] ...trap/engines/engine.py.326: process: 0.28706 | load: 0.00014 [I 2022-10-26 17:01:03] ...trap/engines/engine.py.321: eval: epoch 19 | batch 400/510 [I 2022-10-26 17:01:03] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 17:01:03] ...trap/engines/engine.py.326: process: 0.28783 | load: 0.00016 [I 2022-10-26 17:01:05] ...trap/engines/engine.py.321: eval: epoch 19 | batch 410/510 [I 2022-10-26 17:01:05] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:28 [I 2022-10-26 17:01:05] ...trap/engines/engine.py.326: process: 0.28991 | load: 0.00015 [I 2022-10-26 17:01:08] ...trap/engines/engine.py.321: eval: epoch 19 | batch 420/510 [I 2022-10-26 17:01:08] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:26 [I 2022-10-26 17:01:08] ...trap/engines/engine.py.326: process: 0.28654 | load: 0.00015 [I 2022-10-26 17:01:11] ...trap/engines/engine.py.321: eval: epoch 19 | batch 430/510 [I 2022-10-26 17:01:11] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-26 17:01:11] ...trap/engines/engine.py.326: process: 0.29149 | load: 0.00015 [I 2022-10-26 17:01:14] ...trap/engines/engine.py.321: eval: epoch 19 | batch 440/510 [I 2022-10-26 17:01:14] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 17:01:14] ...trap/engines/engine.py.326: process: 0.28716 | load: 0.00015 [I 2022-10-26 17:01:17] ...trap/engines/engine.py.321: eval: epoch 19 | batch 450/510 [I 2022-10-26 17:01:17] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 17:01:17] ...trap/engines/engine.py.326: process: 0.29177 | load: 0.00025 [I 2022-10-26 17:01:20] ...trap/engines/engine.py.321: eval: epoch 19 | batch 460/510 [I 2022-10-26 17:01:20] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 17:01:20] ...trap/engines/engine.py.326: process: 0.28603 | load: 0.00015 [I 2022-10-26 17:01:23] ...trap/engines/engine.py.321: eval: epoch 19 | batch 470/510 [I 2022-10-26 17:01:23] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 17:01:23] ...trap/engines/engine.py.326: process: 0.28789 | load: 0.00020 [I 2022-10-26 17:01:26] ...trap/engines/engine.py.321: eval: epoch 19 | batch 480/510 [I 2022-10-26 17:01:26] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 17:01:26] ...trap/engines/engine.py.326: process: 0.28096 | load: 0.00015 [I 2022-10-26 17:01:29] ...trap/engines/engine.py.321: eval: epoch 19 | batch 490/510 [I 2022-10-26 17:01:29] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:05 [I 2022-10-26 17:01:29] ...trap/engines/engine.py.326: process: 0.28555 | load: 0.00014 [I 2022-10-26 17:01:31] ...trap/engines/engine.py.321: eval: epoch 19 | batch 500/510 [I 2022-10-26 17:01:31] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-26 17:01:31] ...trap/engines/engine.py.326: process: 0.28471 | load: 0.00021 [I 2022-10-26 17:01:34] ...trap/engines/engine.py.321: eval: epoch 19 | batch 510/510 [I 2022-10-26 17:01:34] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 17:01:34] ...trap/engines/engine.py.326: process: 0.27918 | load: 0.00012 [S 2022-10-26 17:01:35] ...trap/engines/engine.py.132: eval_epoch.epoch: 19 [I 2022-10-26 17:01:35] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.574800 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.834000 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.894700 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.568300 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.832400 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.890800 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016792 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.016715 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011455 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015550 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.013358 [S 2022-10-26 17:01:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.012147 [I 2022-10-26 17:01:40] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 17:01:40] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 17:01:40] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 17:01:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 17:01:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 17:01:41] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 17:01:43] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 17:01:48] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 17:01:48] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 17:01:48] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 17:01:51] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 17:01:56] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 17:01:56] ...trap/engines/engine.py.163: Training model on trainset for epoch 20 [I 2022-10-26 17:01:56] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 17:01:56] ...etworks/tfood_vicha.py.184: training aux kw modules [I 2022-10-26 17:01:56] ...optimizers/trijoint.py.189: Switched to optimizer recipe&image [I 2022-10-26 17:01:59] ...trap/engines/engine.py.224: train: epoch 20 | batch 0/2304 [I 2022-10-26 17:01:59] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:49:12 [I 2022-10-26 17:01:59] ...trap/engines/engine.py.229: process: 2.84415 | load: 2.02210 [I 2022-10-26 17:01:59] ...trap/engines/engine.py.230: loss: 0.21798 [I 2022-10-26 17:01:59] ...optimizers/trijoint.py.151: total_norm: 27.612190 [I 2022-10-26 17:02:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 10/2304 [I 2022-10-26 17:02:05] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:31:21 [I 2022-10-26 17:02:05] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00343 [I 2022-10-26 17:02:05] ...trap/engines/engine.py.230: loss: 0.25032 [I 2022-10-26 17:02:05] ...optimizers/trijoint.py.151: total_norm: 30.318779 [I 2022-10-26 17:02:10] ...trap/engines/engine.py.224: train: epoch 20 | batch 20/2304 [I 2022-10-26 17:02:10] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:27 [I 2022-10-26 17:02:10] ...trap/engines/engine.py.229: process: 0.53248 | load: 0.00392 [I 2022-10-26 17:02:10] ...trap/engines/engine.py.230: loss: 0.16157 [I 2022-10-26 17:02:10] ...optimizers/trijoint.py.151: total_norm: 20.955750 [I 2022-10-26 17:02:16] ...trap/engines/engine.py.224: train: epoch 20 | batch 30/2304 [I 2022-10-26 17:02:16] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 0:21:33 [I 2022-10-26 17:02:16] ...trap/engines/engine.py.229: process: 0.57975 | load: 0.00284 [I 2022-10-26 17:02:16] ...trap/engines/engine.py.230: loss: 0.20931 [I 2022-10-26 17:02:16] ...optimizers/trijoint.py.151: total_norm: 21.657152 [I 2022-10-26 17:02:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 40/2304 [I 2022-10-26 17:02:22] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:19 [I 2022-10-26 17:02:22] ...trap/engines/engine.py.229: process: 0.56542 | load: 0.00318 [I 2022-10-26 17:02:22] ...trap/engines/engine.py.230: loss: 0.18606 [I 2022-10-26 17:02:22] ...optimizers/trijoint.py.151: total_norm: 23.444191 [I 2022-10-26 17:02:27] ...trap/engines/engine.py.224: train: epoch 20 | batch 50/2304 [I 2022-10-26 17:02:27] ...trap/engines/engine.py.228: elapsed: 0:00:31 | left: 0:20:56 [I 2022-10-26 17:02:27] ...trap/engines/engine.py.229: process: 0.55378 | load: 0.00349 [I 2022-10-26 17:02:27] ...trap/engines/engine.py.230: loss: 0.22166 [I 2022-10-26 17:02:27] ...optimizers/trijoint.py.151: total_norm: 23.861784 [I 2022-10-26 17:02:33] ...trap/engines/engine.py.224: train: epoch 20 | batch 60/2304 [I 2022-10-26 17:02:33] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:20:59 [I 2022-10-26 17:02:33] ...trap/engines/engine.py.229: process: 0.56893 | load: 0.00268 [I 2022-10-26 17:02:33] ...trap/engines/engine.py.230: loss: 0.12969 [I 2022-10-26 17:02:33] ...optimizers/trijoint.py.151: total_norm: 21.282108 [I 2022-10-26 17:02:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 70/2304 [I 2022-10-26 17:02:39] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:55 [I 2022-10-26 17:02:39] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00280 [I 2022-10-26 17:02:39] ...trap/engines/engine.py.230: loss: 0.19494 [I 2022-10-26 17:02:39] ...optimizers/trijoint.py.151: total_norm: 25.409515 [I 2022-10-26 17:02:44] ...trap/engines/engine.py.224: train: epoch 20 | batch 80/2304 [I 2022-10-26 17:02:44] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:20:39 [I 2022-10-26 17:02:44] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00260 [I 2022-10-26 17:02:44] ...trap/engines/engine.py.230: loss: 0.18773 [I 2022-10-26 17:02:44] ...optimizers/trijoint.py.151: total_norm: 20.697802 [I 2022-10-26 17:02:50] ...trap/engines/engine.py.224: train: epoch 20 | batch 90/2304 [I 2022-10-26 17:02:50] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:34 [I 2022-10-26 17:02:50] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00284 [I 2022-10-26 17:02:50] ...trap/engines/engine.py.230: loss: 0.14798 [I 2022-10-26 17:02:50] ...optimizers/trijoint.py.151: total_norm: 25.667614 [I 2022-10-26 17:02:55] ...trap/engines/engine.py.224: train: epoch 20 | batch 100/2304 [I 2022-10-26 17:02:55] ...trap/engines/engine.py.228: elapsed: 0:00:59 | left: 0:20:42 [I 2022-10-26 17:02:55] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00284 [I 2022-10-26 17:02:55] ...trap/engines/engine.py.230: loss: 0.22490 [I 2022-10-26 17:02:55] ...optimizers/trijoint.py.151: total_norm: 56.645489 [I 2022-10-26 17:03:01] ...trap/engines/engine.py.224: train: epoch 20 | batch 110/2304 [I 2022-10-26 17:03:01] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:20:44 [I 2022-10-26 17:03:01] ...trap/engines/engine.py.229: process: 0.56606 | load: 0.00286 [I 2022-10-26 17:03:01] ...trap/engines/engine.py.230: loss: 0.19666 [I 2022-10-26 17:03:01] ...optimizers/trijoint.py.151: total_norm: 28.927353 [I 2022-10-26 17:03:07] ...trap/engines/engine.py.224: train: epoch 20 | batch 120/2304 [I 2022-10-26 17:03:07] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:20 [I 2022-10-26 17:03:07] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00276 [I 2022-10-26 17:03:07] ...trap/engines/engine.py.230: loss: 0.24231 [I 2022-10-26 17:03:07] ...optimizers/trijoint.py.151: total_norm: 23.641043 [I 2022-10-26 17:03:12] ...trap/engines/engine.py.224: train: epoch 20 | batch 130/2304 [I 2022-10-26 17:03:12] ...trap/engines/engine.py.228: elapsed: 0:01:16 | left: 0:20:27 [I 2022-10-26 17:03:12] ...trap/engines/engine.py.229: process: 0.56114 | load: 0.00259 [I 2022-10-26 17:03:12] ...trap/engines/engine.py.230: loss: 0.17187 [I 2022-10-26 17:03:12] ...optimizers/trijoint.py.151: total_norm: 22.856606 [I 2022-10-26 17:03:18] ...trap/engines/engine.py.224: train: epoch 20 | batch 140/2304 [I 2022-10-26 17:03:18] ...trap/engines/engine.py.228: elapsed: 0:01:22 | left: 0:20:12 [I 2022-10-26 17:03:18] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00270 [I 2022-10-26 17:03:18] ...trap/engines/engine.py.230: loss: 0.16260 [I 2022-10-26 17:03:18] ...optimizers/trijoint.py.151: total_norm: 18.748497 [I 2022-10-26 17:03:23] ...trap/engines/engine.py.224: train: epoch 20 | batch 150/2304 [I 2022-10-26 17:03:23] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:19:55 [I 2022-10-26 17:03:23] ...trap/engines/engine.py.229: process: 0.55169 | load: 0.00265 [I 2022-10-26 17:03:23] ...trap/engines/engine.py.230: loss: 0.18774 [I 2022-10-26 17:03:23] ...optimizers/trijoint.py.151: total_norm: 23.388439 [I 2022-10-26 17:03:49] ...trap/engines/engine.py.224: train: epoch 20 | batch 160/2304 [I 2022-10-26 17:03:49] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 1:51:23 [I 2022-10-26 17:03:49] ...trap/engines/engine.py.229: process: 0.56527 | load: 0.00269 [I 2022-10-26 17:03:49] ...trap/engines/engine.py.230: loss: 0.22944 [I 2022-10-26 17:03:49] ...optimizers/trijoint.py.151: total_norm: 19.608778 [I 2022-10-26 17:03:55] ...trap/engines/engine.py.224: train: epoch 20 | batch 170/2304 [I 2022-10-26 17:03:55] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:29:34 [I 2022-10-26 17:03:55] ...trap/engines/engine.py.229: process: 0.56390 | load: 0.00242 [I 2022-10-26 17:03:55] ...trap/engines/engine.py.230: loss: 0.19848 [I 2022-10-26 17:03:55] ...optimizers/trijoint.py.151: total_norm: 20.795059 [I 2022-10-26 17:04:00] ...trap/engines/engine.py.224: train: epoch 20 | batch 180/2304 [I 2022-10-26 17:04:00] ...trap/engines/engine.py.228: elapsed: 0:02:04 | left: 0:20:41 [I 2022-10-26 17:04:00] ...trap/engines/engine.py.229: process: 0.55271 | load: 0.00261 [I 2022-10-26 17:04:00] ...trap/engines/engine.py.230: loss: 0.19985 [I 2022-10-26 17:04:00] ...optimizers/trijoint.py.151: total_norm: 24.519722 [I 2022-10-26 17:04:06] ...trap/engines/engine.py.224: train: epoch 20 | batch 190/2304 [I 2022-10-26 17:04:06] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:19:45 [I 2022-10-26 17:04:06] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00255 [I 2022-10-26 17:04:06] ...trap/engines/engine.py.230: loss: 0.13469 [I 2022-10-26 17:04:06] ...optimizers/trijoint.py.151: total_norm: 15.656422 [I 2022-10-26 17:04:11] ...trap/engines/engine.py.224: train: epoch 20 | batch 200/2304 [I 2022-10-26 17:04:11] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:19:27 [I 2022-10-26 17:04:11] ...trap/engines/engine.py.229: process: 0.54894 | load: 0.00250 [I 2022-10-26 17:04:11] ...trap/engines/engine.py.230: loss: 0.18363 [I 2022-10-26 17:04:11] ...optimizers/trijoint.py.151: total_norm: 30.942369 [I 2022-10-26 17:04:17] ...trap/engines/engine.py.224: train: epoch 20 | batch 210/2304 [I 2022-10-26 17:04:17] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:19:21 [I 2022-10-26 17:04:17] ...trap/engines/engine.py.229: process: 0.55942 | load: 0.00241 [I 2022-10-26 17:04:17] ...trap/engines/engine.py.230: loss: 0.25082 [I 2022-10-26 17:04:17] ...optimizers/trijoint.py.151: total_norm: 39.433044 [I 2022-10-26 17:04:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 220/2304 [I 2022-10-26 17:04:22] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:19:25 [I 2022-10-26 17:04:22] ...trap/engines/engine.py.229: process: 0.56776 | load: 0.00257 [I 2022-10-26 17:04:22] ...trap/engines/engine.py.230: loss: 0.27539 [I 2022-10-26 17:04:22] ...optimizers/trijoint.py.151: total_norm: 25.808474 [I 2022-10-26 17:04:28] ...trap/engines/engine.py.224: train: epoch 20 | batch 230/2304 [I 2022-10-26 17:04:28] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:19:19 [I 2022-10-26 17:04:28] ...trap/engines/engine.py.229: process: 0.56757 | load: 0.00261 [I 2022-10-26 17:04:28] ...trap/engines/engine.py.230: loss: 0.20407 [I 2022-10-26 17:04:28] ...optimizers/trijoint.py.151: total_norm: 73.254463 [I 2022-10-26 17:04:34] ...trap/engines/engine.py.224: train: epoch 20 | batch 240/2304 [I 2022-10-26 17:04:34] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:19:18 [I 2022-10-26 17:04:34] ...trap/engines/engine.py.229: process: 0.55458 | load: 0.00250 [I 2022-10-26 17:04:34] ...trap/engines/engine.py.230: loss: 0.18268 [I 2022-10-26 17:04:34] ...optimizers/trijoint.py.151: total_norm: 24.394983 [I 2022-10-26 17:04:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 250/2304 [I 2022-10-26 17:04:39] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:19:17 [I 2022-10-26 17:04:39] ...trap/engines/engine.py.229: process: 0.56847 | load: 0.00257 [I 2022-10-26 17:04:39] ...trap/engines/engine.py.230: loss: 0.19522 [I 2022-10-26 17:04:39] ...optimizers/trijoint.py.151: total_norm: 17.359690 [I 2022-10-26 17:04:45] ...trap/engines/engine.py.224: train: epoch 20 | batch 260/2304 [I 2022-10-26 17:04:45] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:19:02 [I 2022-10-26 17:04:45] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00315 [I 2022-10-26 17:04:45] ...trap/engines/engine.py.230: loss: 0.24043 [I 2022-10-26 17:04:45] ...optimizers/trijoint.py.151: total_norm: 33.485298 [I 2022-10-26 17:04:51] ...trap/engines/engine.py.224: train: epoch 20 | batch 270/2304 [I 2022-10-26 17:04:51] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:19:05 [I 2022-10-26 17:04:51] ...trap/engines/engine.py.229: process: 0.57235 | load: 0.00245 [I 2022-10-26 17:04:51] ...trap/engines/engine.py.230: loss: 0.17913 [I 2022-10-26 17:04:51] ...optimizers/trijoint.py.151: total_norm: 29.689556 [I 2022-10-26 17:04:56] ...trap/engines/engine.py.224: train: epoch 20 | batch 280/2304 [I 2022-10-26 17:04:56] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:19:07 [I 2022-10-26 17:04:56] ...trap/engines/engine.py.229: process: 0.57139 | load: 0.00268 [I 2022-10-26 17:04:56] ...trap/engines/engine.py.230: loss: 0.20370 [I 2022-10-26 17:04:56] ...optimizers/trijoint.py.151: total_norm: 19.865515 [I 2022-10-26 17:05:02] ...trap/engines/engine.py.224: train: epoch 20 | batch 290/2304 [I 2022-10-26 17:05:02] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:45 [I 2022-10-26 17:05:02] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00255 [I 2022-10-26 17:05:02] ...trap/engines/engine.py.230: loss: 0.16957 [I 2022-10-26 17:05:02] ...optimizers/trijoint.py.151: total_norm: 17.651506 [I 2022-10-26 17:05:07] ...trap/engines/engine.py.224: train: epoch 20 | batch 300/2304 [I 2022-10-26 17:05:07] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:18:37 [I 2022-10-26 17:05:07] ...trap/engines/engine.py.229: process: 0.57090 | load: 0.00253 [I 2022-10-26 17:05:07] ...trap/engines/engine.py.230: loss: 0.22065 [I 2022-10-26 17:05:07] ...optimizers/trijoint.py.151: total_norm: 24.641819 [I 2022-10-26 17:05:13] ...trap/engines/engine.py.224: train: epoch 20 | batch 310/2304 [I 2022-10-26 17:05:13] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:18:38 [I 2022-10-26 17:05:13] ...trap/engines/engine.py.229: process: 0.55360 | load: 0.00290 [I 2022-10-26 17:05:13] ...trap/engines/engine.py.230: loss: 0.12703 [I 2022-10-26 17:05:13] ...optimizers/trijoint.py.151: total_norm: 32.570217 [I 2022-10-26 17:05:19] ...trap/engines/engine.py.224: train: epoch 20 | batch 320/2304 [I 2022-10-26 17:05:19] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:18:30 [I 2022-10-26 17:05:19] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00256 [I 2022-10-26 17:05:19] ...trap/engines/engine.py.230: loss: 0.22199 [I 2022-10-26 17:05:19] ...optimizers/trijoint.py.151: total_norm: 25.808079 [I 2022-10-26 17:05:24] ...trap/engines/engine.py.224: train: epoch 20 | batch 330/2304 [I 2022-10-26 17:05:24] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:18:32 [I 2022-10-26 17:05:24] ...trap/engines/engine.py.229: process: 0.56940 | load: 0.00278 [I 2022-10-26 17:05:24] ...trap/engines/engine.py.230: loss: 0.20632 [I 2022-10-26 17:05:24] ...optimizers/trijoint.py.151: total_norm: 110.584991 [I 2022-10-26 17:05:30] ...trap/engines/engine.py.224: train: epoch 20 | batch 340/2304 [I 2022-10-26 17:05:30] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:18:41 [I 2022-10-26 17:05:30] ...trap/engines/engine.py.229: process: 0.61696 | load: 0.00266 [I 2022-10-26 17:05:30] ...trap/engines/engine.py.230: loss: 0.22883 [I 2022-10-26 17:05:30] ...optimizers/trijoint.py.151: total_norm: 20.509020 [I 2022-10-26 17:05:36] ...trap/engines/engine.py.224: train: epoch 20 | batch 350/2304 [I 2022-10-26 17:05:36] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:18:20 [I 2022-10-26 17:05:36] ...trap/engines/engine.py.229: process: 0.56641 | load: 0.00286 [I 2022-10-26 17:05:36] ...trap/engines/engine.py.230: loss: 0.19382 [I 2022-10-26 17:05:36] ...optimizers/trijoint.py.151: total_norm: 35.606331 [I 2022-10-26 17:05:41] ...trap/engines/engine.py.224: train: epoch 20 | batch 360/2304 [I 2022-10-26 17:05:41] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:18:00 [I 2022-10-26 17:05:41] ...trap/engines/engine.py.229: process: 0.54562 | load: 0.00296 [I 2022-10-26 17:05:41] ...trap/engines/engine.py.230: loss: 0.13280 [I 2022-10-26 17:05:41] ...optimizers/trijoint.py.151: total_norm: 16.698383 [I 2022-10-26 17:05:47] ...trap/engines/engine.py.224: train: epoch 20 | batch 370/2304 [I 2022-10-26 17:05:47] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:18:04 [I 2022-10-26 17:05:47] ...trap/engines/engine.py.229: process: 0.56039 | load: 0.00285 [I 2022-10-26 17:05:47] ...trap/engines/engine.py.230: loss: 0.15459 [I 2022-10-26 17:05:47] ...optimizers/trijoint.py.151: total_norm: 15.359537 [I 2022-10-26 17:05:52] ...trap/engines/engine.py.224: train: epoch 20 | batch 380/2304 [I 2022-10-26 17:05:52] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:17:55 [I 2022-10-26 17:05:52] ...trap/engines/engine.py.229: process: 0.58284 | load: 0.00270 [I 2022-10-26 17:05:52] ...trap/engines/engine.py.230: loss: 0.19041 [I 2022-10-26 17:05:52] ...optimizers/trijoint.py.151: total_norm: 24.863922 [I 2022-10-26 17:05:58] ...trap/engines/engine.py.224: train: epoch 20 | batch 390/2304 [I 2022-10-26 17:05:58] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:44 [I 2022-10-26 17:05:58] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00279 [I 2022-10-26 17:05:58] ...trap/engines/engine.py.230: loss: 0.16982 [I 2022-10-26 17:05:58] ...optimizers/trijoint.py.151: total_norm: 17.466076 [I 2022-10-26 17:06:04] ...trap/engines/engine.py.224: train: epoch 20 | batch 400/2304 [I 2022-10-26 17:06:04] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:17:46 [I 2022-10-26 17:06:04] ...trap/engines/engine.py.229: process: 0.56367 | load: 0.00269 [I 2022-10-26 17:06:04] ...trap/engines/engine.py.230: loss: 0.13560 [I 2022-10-26 17:06:04] ...optimizers/trijoint.py.151: total_norm: 19.420271 [I 2022-10-26 17:06:09] ...trap/engines/engine.py.224: train: epoch 20 | batch 410/2304 [I 2022-10-26 17:06:09] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:45 [I 2022-10-26 17:06:09] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00253 [I 2022-10-26 17:06:09] ...trap/engines/engine.py.230: loss: 0.12450 [I 2022-10-26 17:06:09] ...optimizers/trijoint.py.151: total_norm: 15.398959 [I 2022-10-26 17:06:15] ...trap/engines/engine.py.224: train: epoch 20 | batch 420/2304 [I 2022-10-26 17:06:15] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:41 [I 2022-10-26 17:06:15] ...trap/engines/engine.py.229: process: 0.55609 | load: 0.00528 [I 2022-10-26 17:06:15] ...trap/engines/engine.py.230: loss: 0.15158 [I 2022-10-26 17:06:15] ...optimizers/trijoint.py.151: total_norm: 16.838346 [I 2022-10-26 17:06:21] ...trap/engines/engine.py.224: train: epoch 20 | batch 430/2304 [I 2022-10-26 17:06:21] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:17:30 [I 2022-10-26 17:06:21] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00268 [I 2022-10-26 17:06:21] ...trap/engines/engine.py.230: loss: 0.15261 [I 2022-10-26 17:06:21] ...optimizers/trijoint.py.151: total_norm: 19.411547 [I 2022-10-26 17:06:26] ...trap/engines/engine.py.224: train: epoch 20 | batch 440/2304 [I 2022-10-26 17:06:26] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:17:23 [I 2022-10-26 17:06:26] ...trap/engines/engine.py.229: process: 0.55265 | load: 0.00265 [I 2022-10-26 17:06:26] ...trap/engines/engine.py.230: loss: 0.31762 [I 2022-10-26 17:06:26] ...optimizers/trijoint.py.151: total_norm: 34.041706 [I 2022-10-26 17:06:32] ...trap/engines/engine.py.224: train: epoch 20 | batch 450/2304 [I 2022-10-26 17:06:32] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:17:10 [I 2022-10-26 17:06:32] ...trap/engines/engine.py.229: process: 0.54681 | load: 0.00254 [I 2022-10-26 17:06:32] ...trap/engines/engine.py.230: loss: 0.20981 [I 2022-10-26 17:06:32] ...optimizers/trijoint.py.151: total_norm: 27.122614 [I 2022-10-26 17:06:37] ...trap/engines/engine.py.224: train: epoch 20 | batch 460/2304 [I 2022-10-26 17:06:37] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:17:06 [I 2022-10-26 17:06:37] ...trap/engines/engine.py.229: process: 0.55461 | load: 0.00251 [I 2022-10-26 17:06:37] ...trap/engines/engine.py.230: loss: 0.20950 [I 2022-10-26 17:06:37] ...optimizers/trijoint.py.151: total_norm: 27.950417 [I 2022-10-26 17:06:43] ...trap/engines/engine.py.224: train: epoch 20 | batch 470/2304 [I 2022-10-26 17:06:43] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:17:06 [I 2022-10-26 17:06:43] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00270 [I 2022-10-26 17:06:43] ...trap/engines/engine.py.230: loss: 0.23517 [I 2022-10-26 17:06:43] ...optimizers/trijoint.py.151: total_norm: 23.443184 [I 2022-10-26 17:06:49] ...trap/engines/engine.py.224: train: epoch 20 | batch 480/2304 [I 2022-10-26 17:06:49] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:17:05 [I 2022-10-26 17:06:49] ...trap/engines/engine.py.229: process: 0.54511 | load: 0.00252 [I 2022-10-26 17:06:49] ...trap/engines/engine.py.230: loss: 0.21137 [I 2022-10-26 17:06:49] ...optimizers/trijoint.py.151: total_norm: 32.827785 [I 2022-10-26 17:06:54] ...trap/engines/engine.py.224: train: epoch 20 | batch 490/2304 [I 2022-10-26 17:06:54] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:58 [I 2022-10-26 17:06:54] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00262 [I 2022-10-26 17:06:54] ...trap/engines/engine.py.230: loss: 0.22959 [I 2022-10-26 17:06:54] ...optimizers/trijoint.py.151: total_norm: 25.838408 [I 2022-10-26 17:07:00] ...trap/engines/engine.py.224: train: epoch 20 | batch 500/2304 [I 2022-10-26 17:07:00] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:16:46 [I 2022-10-26 17:07:00] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00250 [I 2022-10-26 17:07:00] ...trap/engines/engine.py.230: loss: 0.32787 [I 2022-10-26 17:07:00] ...optimizers/trijoint.py.151: total_norm: 23.281216 [I 2022-10-26 17:07:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 510/2304 [I 2022-10-26 17:07:05] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:45 [I 2022-10-26 17:07:05] ...trap/engines/engine.py.229: process: 0.56603 | load: 0.00258 [I 2022-10-26 17:07:05] ...trap/engines/engine.py.230: loss: 0.17439 [I 2022-10-26 17:07:05] ...optimizers/trijoint.py.151: total_norm: 23.773439 [I 2022-10-26 17:07:11] ...trap/engines/engine.py.224: train: epoch 20 | batch 520/2304 [I 2022-10-26 17:07:11] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:29 [I 2022-10-26 17:07:11] ...trap/engines/engine.py.229: process: 0.54490 | load: 0.00253 [I 2022-10-26 17:07:11] ...trap/engines/engine.py.230: loss: 0.19283 [I 2022-10-26 17:07:11] ...optimizers/trijoint.py.151: total_norm: 20.700834 [I 2022-10-26 17:07:17] ...trap/engines/engine.py.224: train: epoch 20 | batch 530/2304 [I 2022-10-26 17:07:17] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:16:23 [I 2022-10-26 17:07:17] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00251 [I 2022-10-26 17:07:17] ...trap/engines/engine.py.230: loss: 0.16188 [I 2022-10-26 17:07:17] ...optimizers/trijoint.py.151: total_norm: 14.477235 [I 2022-10-26 17:07:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 540/2304 [I 2022-10-26 17:07:22] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:16 [I 2022-10-26 17:07:22] ...trap/engines/engine.py.229: process: 0.54394 | load: 0.00244 [I 2022-10-26 17:07:22] ...trap/engines/engine.py.230: loss: 0.25029 [I 2022-10-26 17:07:22] ...optimizers/trijoint.py.151: total_norm: 16.012083 [I 2022-10-26 17:07:28] ...trap/engines/engine.py.224: train: epoch 20 | batch 550/2304 [I 2022-10-26 17:07:28] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:16:34 [I 2022-10-26 17:07:28] ...trap/engines/engine.py.229: process: 0.57744 | load: 0.00264 [I 2022-10-26 17:07:28] ...trap/engines/engine.py.230: loss: 0.12754 [I 2022-10-26 17:07:28] ...optimizers/trijoint.py.151: total_norm: 16.683575 [I 2022-10-26 17:07:33] ...trap/engines/engine.py.224: train: epoch 20 | batch 560/2304 [I 2022-10-26 17:07:33] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:16:12 [I 2022-10-26 17:07:33] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00251 [I 2022-10-26 17:07:33] ...trap/engines/engine.py.230: loss: 0.17112 [I 2022-10-26 17:07:33] ...optimizers/trijoint.py.151: total_norm: 19.746338 [I 2022-10-26 17:07:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 570/2304 [I 2022-10-26 17:07:39] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:16:10 [I 2022-10-26 17:07:39] ...trap/engines/engine.py.229: process: 0.56570 | load: 0.00261 [I 2022-10-26 17:07:39] ...trap/engines/engine.py.230: loss: 0.18429 [I 2022-10-26 17:07:39] ...optimizers/trijoint.py.151: total_norm: 29.215256 [I 2022-10-26 17:07:45] ...trap/engines/engine.py.224: train: epoch 20 | batch 580/2304 [I 2022-10-26 17:07:45] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:16:03 [I 2022-10-26 17:07:45] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00271 [I 2022-10-26 17:07:45] ...trap/engines/engine.py.230: loss: 0.21708 [I 2022-10-26 17:07:45] ...optimizers/trijoint.py.151: total_norm: 17.619629 [I 2022-10-26 17:07:50] ...trap/engines/engine.py.224: train: epoch 20 | batch 590/2304 [I 2022-10-26 17:07:50] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:16:11 [I 2022-10-26 17:07:50] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00256 [I 2022-10-26 17:07:50] ...trap/engines/engine.py.230: loss: 0.19406 [I 2022-10-26 17:07:50] ...optimizers/trijoint.py.151: total_norm: 19.698677 [I 2022-10-26 17:07:56] ...trap/engines/engine.py.224: train: epoch 20 | batch 600/2304 [I 2022-10-26 17:07:56] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:55 [I 2022-10-26 17:07:56] ...trap/engines/engine.py.229: process: 0.56050 | load: 0.00262 [I 2022-10-26 17:07:56] ...trap/engines/engine.py.230: loss: 0.15788 [I 2022-10-26 17:07:56] ...optimizers/trijoint.py.151: total_norm: 14.122639 [I 2022-10-26 17:08:01] ...trap/engines/engine.py.224: train: epoch 20 | batch 610/2304 [I 2022-10-26 17:08:01] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:44 [I 2022-10-26 17:08:01] ...trap/engines/engine.py.229: process: 0.55453 | load: 0.00262 [I 2022-10-26 17:08:01] ...trap/engines/engine.py.230: loss: 0.22712 [I 2022-10-26 17:08:01] ...optimizers/trijoint.py.151: total_norm: 33.540581 [I 2022-10-26 17:08:07] ...trap/engines/engine.py.224: train: epoch 20 | batch 620/2304 [I 2022-10-26 17:08:07] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:39 [I 2022-10-26 17:08:07] ...trap/engines/engine.py.229: process: 0.55089 | load: 0.00253 [I 2022-10-26 17:08:07] ...trap/engines/engine.py.230: loss: 0.14991 [I 2022-10-26 17:08:07] ...optimizers/trijoint.py.151: total_norm: 13.020392 [I 2022-10-26 17:08:13] ...trap/engines/engine.py.224: train: epoch 20 | batch 630/2304 [I 2022-10-26 17:08:13] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:26 [I 2022-10-26 17:08:13] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00250 [I 2022-10-26 17:08:13] ...trap/engines/engine.py.230: loss: 0.14554 [I 2022-10-26 17:08:13] ...optimizers/trijoint.py.151: total_norm: 20.253586 [I 2022-10-26 17:08:18] ...trap/engines/engine.py.224: train: epoch 20 | batch 640/2304 [I 2022-10-26 17:08:18] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:23 [I 2022-10-26 17:08:18] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00247 [I 2022-10-26 17:08:18] ...trap/engines/engine.py.230: loss: 0.16551 [I 2022-10-26 17:08:18] ...optimizers/trijoint.py.151: total_norm: 23.508343 [I 2022-10-26 17:08:24] ...trap/engines/engine.py.224: train: epoch 20 | batch 650/2304 [I 2022-10-26 17:08:24] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:15:20 [I 2022-10-26 17:08:24] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00241 [I 2022-10-26 17:08:24] ...trap/engines/engine.py.230: loss: 0.17219 [I 2022-10-26 17:08:24] ...optimizers/trijoint.py.151: total_norm: 20.580906 [I 2022-10-26 17:08:29] ...trap/engines/engine.py.224: train: epoch 20 | batch 660/2304 [I 2022-10-26 17:08:29] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:15:09 [I 2022-10-26 17:08:29] ...trap/engines/engine.py.229: process: 0.55070 | load: 0.00243 [I 2022-10-26 17:08:29] ...trap/engines/engine.py.230: loss: 0.20843 [I 2022-10-26 17:08:29] ...optimizers/trijoint.py.151: total_norm: 16.179539 [I 2022-10-26 17:08:35] ...trap/engines/engine.py.224: train: epoch 20 | batch 670/2304 [I 2022-10-26 17:08:35] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:15:04 [I 2022-10-26 17:08:35] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00258 [I 2022-10-26 17:08:35] ...trap/engines/engine.py.230: loss: 0.12561 [I 2022-10-26 17:08:35] ...optimizers/trijoint.py.151: total_norm: 23.802637 [I 2022-10-26 17:08:40] ...trap/engines/engine.py.224: train: epoch 20 | batch 680/2304 [I 2022-10-26 17:08:40] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:15:01 [I 2022-10-26 17:08:40] ...trap/engines/engine.py.229: process: 0.54719 | load: 0.00254 [I 2022-10-26 17:08:40] ...trap/engines/engine.py.230: loss: 0.19443 [I 2022-10-26 17:08:40] ...optimizers/trijoint.py.151: total_norm: 18.958071 [I 2022-10-26 17:08:46] ...trap/engines/engine.py.224: train: epoch 20 | batch 690/2304 [I 2022-10-26 17:08:46] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:49 [I 2022-10-26 17:08:46] ...trap/engines/engine.py.229: process: 0.54827 | load: 0.00249 [I 2022-10-26 17:08:46] ...trap/engines/engine.py.230: loss: 0.10914 [I 2022-10-26 17:08:46] ...optimizers/trijoint.py.151: total_norm: 9.993428 [I 2022-10-26 17:08:52] ...trap/engines/engine.py.224: train: epoch 20 | batch 700/2304 [I 2022-10-26 17:08:52] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:15:24 [I 2022-10-26 17:08:52] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00254 [I 2022-10-26 17:08:52] ...trap/engines/engine.py.230: loss: 0.17971 [I 2022-10-26 17:08:52] ...optimizers/trijoint.py.151: total_norm: 30.817390 [I 2022-10-26 17:08:57] ...trap/engines/engine.py.224: train: epoch 20 | batch 710/2304 [I 2022-10-26 17:08:57] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:46 [I 2022-10-26 17:08:57] ...trap/engines/engine.py.229: process: 0.54431 | load: 0.00256 [I 2022-10-26 17:08:57] ...trap/engines/engine.py.230: loss: 0.19354 [I 2022-10-26 17:08:57] ...optimizers/trijoint.py.151: total_norm: 35.583458 [I 2022-10-26 17:09:03] ...trap/engines/engine.py.224: train: epoch 20 | batch 720/2304 [I 2022-10-26 17:09:03] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:53 [I 2022-10-26 17:09:03] ...trap/engines/engine.py.229: process: 0.57870 | load: 0.00268 [I 2022-10-26 17:09:03] ...trap/engines/engine.py.230: loss: 0.19238 [I 2022-10-26 17:09:03] ...optimizers/trijoint.py.151: total_norm: 19.283852 [I 2022-10-26 17:09:08] ...trap/engines/engine.py.224: train: epoch 20 | batch 730/2304 [I 2022-10-26 17:09:08] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:39 [I 2022-10-26 17:09:08] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00255 [I 2022-10-26 17:09:08] ...trap/engines/engine.py.230: loss: 0.17037 [I 2022-10-26 17:09:08] ...optimizers/trijoint.py.151: total_norm: 17.513529 [I 2022-10-26 17:09:14] ...trap/engines/engine.py.224: train: epoch 20 | batch 740/2304 [I 2022-10-26 17:09:14] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:28 [I 2022-10-26 17:09:14] ...trap/engines/engine.py.229: process: 0.56580 | load: 0.00252 [I 2022-10-26 17:09:14] ...trap/engines/engine.py.230: loss: 0.13581 [I 2022-10-26 17:09:14] ...optimizers/trijoint.py.151: total_norm: 16.485605 [I 2022-10-26 17:09:19] ...trap/engines/engine.py.224: train: epoch 20 | batch 750/2304 [I 2022-10-26 17:09:19] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:20 [I 2022-10-26 17:09:19] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00244 [I 2022-10-26 17:09:19] ...trap/engines/engine.py.230: loss: 0.10945 [I 2022-10-26 17:09:19] ...optimizers/trijoint.py.151: total_norm: 20.117155 [I 2022-10-26 17:09:25] ...trap/engines/engine.py.224: train: epoch 20 | batch 760/2304 [I 2022-10-26 17:09:25] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:14:21 [I 2022-10-26 17:09:25] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00248 [I 2022-10-26 17:09:25] ...trap/engines/engine.py.230: loss: 0.27905 [I 2022-10-26 17:09:25] ...optimizers/trijoint.py.151: total_norm: 22.961792 [I 2022-10-26 17:09:31] ...trap/engines/engine.py.224: train: epoch 20 | batch 770/2304 [I 2022-10-26 17:09:31] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:14:12 [I 2022-10-26 17:09:31] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00254 [I 2022-10-26 17:09:31] ...trap/engines/engine.py.230: loss: 0.19171 [I 2022-10-26 17:09:31] ...optimizers/trijoint.py.151: total_norm: 21.892172 [I 2022-10-26 17:09:36] ...trap/engines/engine.py.224: train: epoch 20 | batch 780/2304 [I 2022-10-26 17:09:36] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:14:05 [I 2022-10-26 17:09:36] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00246 [I 2022-10-26 17:09:36] ...trap/engines/engine.py.230: loss: 0.22958 [I 2022-10-26 17:09:36] ...optimizers/trijoint.py.151: total_norm: 27.779119 [I 2022-10-26 17:09:42] ...trap/engines/engine.py.224: train: epoch 20 | batch 790/2304 [I 2022-10-26 17:09:42] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:14:00 [I 2022-10-26 17:09:42] ...trap/engines/engine.py.229: process: 0.55422 | load: 0.00255 [I 2022-10-26 17:09:42] ...trap/engines/engine.py.230: loss: 0.21166 [I 2022-10-26 17:09:42] ...optimizers/trijoint.py.151: total_norm: 29.847637 [I 2022-10-26 17:09:47] ...trap/engines/engine.py.224: train: epoch 20 | batch 800/2304 [I 2022-10-26 17:09:47] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:55 [I 2022-10-26 17:09:47] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00264 [I 2022-10-26 17:09:47] ...trap/engines/engine.py.230: loss: 0.15304 [I 2022-10-26 17:09:47] ...optimizers/trijoint.py.151: total_norm: 20.107767 [I 2022-10-26 17:09:53] ...trap/engines/engine.py.224: train: epoch 20 | batch 810/2304 [I 2022-10-26 17:09:53] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:51 [I 2022-10-26 17:09:53] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00265 [I 2022-10-26 17:09:53] ...trap/engines/engine.py.230: loss: 0.29631 [I 2022-10-26 17:09:53] ...optimizers/trijoint.py.151: total_norm: 30.653662 [I 2022-10-26 17:09:59] ...trap/engines/engine.py.224: train: epoch 20 | batch 820/2304 [I 2022-10-26 17:09:59] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:14:19 [I 2022-10-26 17:09:59] ...trap/engines/engine.py.229: process: 0.64903 | load: 0.00278 [I 2022-10-26 17:09:59] ...trap/engines/engine.py.230: loss: 0.24278 [I 2022-10-26 17:09:59] ...optimizers/trijoint.py.151: total_norm: 24.774811 [I 2022-10-26 17:10:04] ...trap/engines/engine.py.224: train: epoch 20 | batch 830/2304 [I 2022-10-26 17:10:04] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:43 [I 2022-10-26 17:10:04] ...trap/engines/engine.py.229: process: 0.56222 | load: 0.00256 [I 2022-10-26 17:10:04] ...trap/engines/engine.py.230: loss: 0.10124 [I 2022-10-26 17:10:04] ...optimizers/trijoint.py.151: total_norm: 14.388336 [I 2022-10-26 17:10:10] ...trap/engines/engine.py.224: train: epoch 20 | batch 840/2304 [I 2022-10-26 17:10:10] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:38 [I 2022-10-26 17:10:10] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00253 [I 2022-10-26 17:10:10] ...trap/engines/engine.py.230: loss: 0.17667 [I 2022-10-26 17:10:10] ...optimizers/trijoint.py.151: total_norm: 20.925159 [I 2022-10-26 17:10:15] ...trap/engines/engine.py.224: train: epoch 20 | batch 850/2304 [I 2022-10-26 17:10:15] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:25 [I 2022-10-26 17:10:15] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00248 [I 2022-10-26 17:10:15] ...trap/engines/engine.py.230: loss: 0.17941 [I 2022-10-26 17:10:15] ...optimizers/trijoint.py.151: total_norm: 18.886337 [I 2022-10-26 17:10:21] ...trap/engines/engine.py.224: train: epoch 20 | batch 860/2304 [I 2022-10-26 17:10:21] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:19 [I 2022-10-26 17:10:21] ...trap/engines/engine.py.229: process: 0.55094 | load: 0.00248 [I 2022-10-26 17:10:21] ...trap/engines/engine.py.230: loss: 0.16170 [I 2022-10-26 17:10:21] ...optimizers/trijoint.py.151: total_norm: 17.514305 [I 2022-10-26 17:10:26] ...trap/engines/engine.py.224: train: epoch 20 | batch 870/2304 [I 2022-10-26 17:10:26] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:13:17 [I 2022-10-26 17:10:26] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00287 [I 2022-10-26 17:10:26] ...trap/engines/engine.py.230: loss: 0.16143 [I 2022-10-26 17:10:26] ...optimizers/trijoint.py.151: total_norm: 19.859459 [I 2022-10-26 17:10:32] ...trap/engines/engine.py.224: train: epoch 20 | batch 880/2304 [I 2022-10-26 17:10:32] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:13:16 [I 2022-10-26 17:10:32] ...trap/engines/engine.py.229: process: 0.56722 | load: 0.00249 [I 2022-10-26 17:10:32] ...trap/engines/engine.py.230: loss: 0.22526 [I 2022-10-26 17:10:32] ...optimizers/trijoint.py.151: total_norm: 28.534174 [I 2022-10-26 17:10:38] ...trap/engines/engine.py.224: train: epoch 20 | batch 890/2304 [I 2022-10-26 17:10:38] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:13:04 [I 2022-10-26 17:10:38] ...trap/engines/engine.py.229: process: 0.54710 | load: 0.00257 [I 2022-10-26 17:10:38] ...trap/engines/engine.py.230: loss: 0.24464 [I 2022-10-26 17:10:38] ...optimizers/trijoint.py.151: total_norm: 69.175606 [I 2022-10-26 17:10:43] ...trap/engines/engine.py.224: train: epoch 20 | batch 900/2304 [I 2022-10-26 17:10:43] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:55 [I 2022-10-26 17:10:43] ...trap/engines/engine.py.229: process: 0.55372 | load: 0.00251 [I 2022-10-26 17:10:43] ...trap/engines/engine.py.230: loss: 0.17066 [I 2022-10-26 17:10:43] ...optimizers/trijoint.py.151: total_norm: 24.314808 [I 2022-10-26 17:10:49] ...trap/engines/engine.py.224: train: epoch 20 | batch 910/2304 [I 2022-10-26 17:10:49] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:53 [I 2022-10-26 17:10:49] ...trap/engines/engine.py.229: process: 0.55350 | load: 0.00287 [I 2022-10-26 17:10:49] ...trap/engines/engine.py.230: loss: 0.20957 [I 2022-10-26 17:10:49] ...optimizers/trijoint.py.151: total_norm: 24.395803 [I 2022-10-26 17:10:54] ...trap/engines/engine.py.224: train: epoch 20 | batch 920/2304 [I 2022-10-26 17:10:54] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:46 [I 2022-10-26 17:10:54] ...trap/engines/engine.py.229: process: 0.54423 | load: 0.00249 [I 2022-10-26 17:10:54] ...trap/engines/engine.py.230: loss: 0.13414 [I 2022-10-26 17:10:54] ...optimizers/trijoint.py.151: total_norm: 23.104950 [I 2022-10-26 17:11:00] ...trap/engines/engine.py.224: train: epoch 20 | batch 930/2304 [I 2022-10-26 17:11:00] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:45 [I 2022-10-26 17:11:00] ...trap/engines/engine.py.229: process: 0.55461 | load: 0.00258 [I 2022-10-26 17:11:00] ...trap/engines/engine.py.230: loss: 0.18404 [I 2022-10-26 17:11:00] ...optimizers/trijoint.py.151: total_norm: 19.650171 [I 2022-10-26 17:11:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 940/2304 [I 2022-10-26 17:11:05] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:35 [I 2022-10-26 17:11:05] ...trap/engines/engine.py.229: process: 0.54917 | load: 0.00249 [I 2022-10-26 17:11:05] ...trap/engines/engine.py.230: loss: 0.21053 [I 2022-10-26 17:11:05] ...optimizers/trijoint.py.151: total_norm: 25.342390 [I 2022-10-26 17:11:11] ...trap/engines/engine.py.224: train: epoch 20 | batch 950/2304 [I 2022-10-26 17:11:11] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:31 [I 2022-10-26 17:11:11] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00248 [I 2022-10-26 17:11:11] ...trap/engines/engine.py.230: loss: 0.23400 [I 2022-10-26 17:11:11] ...optimizers/trijoint.py.151: total_norm: 39.573978 [I 2022-10-26 17:11:17] ...trap/engines/engine.py.224: train: epoch 20 | batch 960/2304 [I 2022-10-26 17:11:17] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:14:22 [I 2022-10-26 17:11:17] ...trap/engines/engine.py.229: process: 0.62479 | load: 0.00266 [I 2022-10-26 17:11:17] ...trap/engines/engine.py.230: loss: 0.14374 [I 2022-10-26 17:11:17] ...optimizers/trijoint.py.151: total_norm: 15.851482 [I 2022-10-26 17:11:23] ...trap/engines/engine.py.224: train: epoch 20 | batch 970/2304 [I 2022-10-26 17:11:23] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:37 [I 2022-10-26 17:11:23] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00249 [I 2022-10-26 17:11:23] ...trap/engines/engine.py.230: loss: 0.17485 [I 2022-10-26 17:11:23] ...optimizers/trijoint.py.151: total_norm: 25.395498 [I 2022-10-26 17:11:28] ...trap/engines/engine.py.224: train: epoch 20 | batch 980/2304 [I 2022-10-26 17:11:28] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:12:14 [I 2022-10-26 17:11:28] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00261 [I 2022-10-26 17:11:28] ...trap/engines/engine.py.230: loss: 0.12669 [I 2022-10-26 17:11:28] ...optimizers/trijoint.py.151: total_norm: 17.778790 [I 2022-10-26 17:11:34] ...trap/engines/engine.py.224: train: epoch 20 | batch 990/2304 [I 2022-10-26 17:11:34] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:12:14 [I 2022-10-26 17:11:34] ...trap/engines/engine.py.229: process: 0.56292 | load: 0.00248 [I 2022-10-26 17:11:34] ...trap/engines/engine.py.230: loss: 0.14769 [I 2022-10-26 17:11:34] ...optimizers/trijoint.py.151: total_norm: 25.099684 [I 2022-10-26 17:11:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 1000/2304 [I 2022-10-26 17:11:39] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:12:09 [I 2022-10-26 17:11:39] ...trap/engines/engine.py.229: process: 0.56634 | load: 0.00239 [I 2022-10-26 17:11:39] ...trap/engines/engine.py.230: loss: 0.23444 [I 2022-10-26 17:11:39] ...optimizers/trijoint.py.151: total_norm: 44.498909 [I 2022-10-26 17:11:45] ...trap/engines/engine.py.224: train: epoch 20 | batch 1010/2304 [I 2022-10-26 17:11:45] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:12:03 [I 2022-10-26 17:11:45] ...trap/engines/engine.py.229: process: 0.54488 | load: 0.00247 [I 2022-10-26 17:11:45] ...trap/engines/engine.py.230: loss: 0.16994 [I 2022-10-26 17:11:45] ...optimizers/trijoint.py.151: total_norm: 15.514374 [I 2022-10-26 17:11:51] ...trap/engines/engine.py.224: train: epoch 20 | batch 1020/2304 [I 2022-10-26 17:11:51] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:53 [I 2022-10-26 17:11:51] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00249 [I 2022-10-26 17:11:51] ...trap/engines/engine.py.230: loss: 0.13395 [I 2022-10-26 17:11:51] ...optimizers/trijoint.py.151: total_norm: 17.352747 [I 2022-10-26 17:11:56] ...trap/engines/engine.py.224: train: epoch 20 | batch 1030/2304 [I 2022-10-26 17:11:56] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:47 [I 2022-10-26 17:11:56] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00247 [I 2022-10-26 17:11:56] ...trap/engines/engine.py.230: loss: 0.27737 [I 2022-10-26 17:11:56] ...optimizers/trijoint.py.151: total_norm: 35.549591 [I 2022-10-26 17:12:02] ...trap/engines/engine.py.224: train: epoch 20 | batch 1040/2304 [I 2022-10-26 17:12:02] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:40 [I 2022-10-26 17:12:02] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00253 [I 2022-10-26 17:12:02] ...trap/engines/engine.py.230: loss: 0.21162 [I 2022-10-26 17:12:02] ...optimizers/trijoint.py.151: total_norm: 15.058803 [I 2022-10-26 17:12:07] ...trap/engines/engine.py.224: train: epoch 20 | batch 1050/2304 [I 2022-10-26 17:12:07] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:40 [I 2022-10-26 17:12:07] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00261 [I 2022-10-26 17:12:07] ...trap/engines/engine.py.230: loss: 0.21681 [I 2022-10-26 17:12:07] ...optimizers/trijoint.py.151: total_norm: 38.852299 [I 2022-10-26 17:12:13] ...trap/engines/engine.py.224: train: epoch 20 | batch 1060/2304 [I 2022-10-26 17:12:13] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:34 [I 2022-10-26 17:12:13] ...trap/engines/engine.py.229: process: 0.55361 | load: 0.00255 [I 2022-10-26 17:12:13] ...trap/engines/engine.py.230: loss: 0.20850 [I 2022-10-26 17:12:13] ...optimizers/trijoint.py.151: total_norm: 36.598438 [I 2022-10-26 17:12:19] ...trap/engines/engine.py.224: train: epoch 20 | batch 1070/2304 [I 2022-10-26 17:12:19] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:33 [I 2022-10-26 17:12:19] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00256 [I 2022-10-26 17:12:19] ...trap/engines/engine.py.230: loss: 0.26499 [I 2022-10-26 17:12:19] ...optimizers/trijoint.py.151: total_norm: 26.676300 [I 2022-10-26 17:12:24] ...trap/engines/engine.py.224: train: epoch 20 | batch 1080/2304 [I 2022-10-26 17:12:24] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:24 [I 2022-10-26 17:12:24] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00253 [I 2022-10-26 17:12:24] ...trap/engines/engine.py.230: loss: 0.19529 [I 2022-10-26 17:12:24] ...optimizers/trijoint.py.151: total_norm: 25.293716 [I 2022-10-26 17:12:30] ...trap/engines/engine.py.224: train: epoch 20 | batch 1090/2304 [I 2022-10-26 17:12:30] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:11:17 [I 2022-10-26 17:12:30] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00253 [I 2022-10-26 17:12:30] ...trap/engines/engine.py.230: loss: 0.21737 [I 2022-10-26 17:12:30] ...optimizers/trijoint.py.151: total_norm: 46.579945 [I 2022-10-26 17:12:35] ...trap/engines/engine.py.224: train: epoch 20 | batch 1100/2304 [I 2022-10-26 17:12:35] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:11:18 [I 2022-10-26 17:12:35] ...trap/engines/engine.py.229: process: 0.57651 | load: 0.00247 [I 2022-10-26 17:12:35] ...trap/engines/engine.py.230: loss: 0.16629 [I 2022-10-26 17:12:35] ...optimizers/trijoint.py.151: total_norm: 22.614223 [I 2022-10-26 17:12:41] ...trap/engines/engine.py.224: train: epoch 20 | batch 1110/2304 [I 2022-10-26 17:12:41] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:11:06 [I 2022-10-26 17:12:41] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00258 [I 2022-10-26 17:12:41] ...trap/engines/engine.py.230: loss: 0.15026 [I 2022-10-26 17:12:41] ...optimizers/trijoint.py.151: total_norm: 20.154539 [I 2022-10-26 17:12:47] ...trap/engines/engine.py.224: train: epoch 20 | batch 1120/2304 [I 2022-10-26 17:12:47] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:11:02 [I 2022-10-26 17:12:47] ...trap/engines/engine.py.229: process: 0.55428 | load: 0.00272 [I 2022-10-26 17:12:47] ...trap/engines/engine.py.230: loss: 0.19329 [I 2022-10-26 17:12:47] ...optimizers/trijoint.py.151: total_norm: 24.964767 [I 2022-10-26 17:12:52] ...trap/engines/engine.py.224: train: epoch 20 | batch 1130/2304 [I 2022-10-26 17:12:52] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:54 [I 2022-10-26 17:12:52] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00253 [I 2022-10-26 17:12:52] ...trap/engines/engine.py.230: loss: 0.22848 [I 2022-10-26 17:12:52] ...optimizers/trijoint.py.151: total_norm: 24.233944 [I 2022-10-26 17:12:58] ...trap/engines/engine.py.224: train: epoch 20 | batch 1140/2304 [I 2022-10-26 17:12:58] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:55 [I 2022-10-26 17:12:58] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00257 [I 2022-10-26 17:12:58] ...trap/engines/engine.py.230: loss: 0.20010 [I 2022-10-26 17:12:58] ...optimizers/trijoint.py.151: total_norm: 29.638172 [I 2022-10-26 17:13:03] ...trap/engines/engine.py.224: train: epoch 20 | batch 1150/2304 [I 2022-10-26 17:13:03] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:46 [I 2022-10-26 17:13:03] ...trap/engines/engine.py.229: process: 0.56383 | load: 0.00256 [I 2022-10-26 17:13:03] ...trap/engines/engine.py.230: loss: 0.13870 [I 2022-10-26 17:13:03] ...optimizers/trijoint.py.151: total_norm: 20.997360 [I 2022-10-26 17:13:09] ...trap/engines/engine.py.224: train: epoch 20 | batch 1160/2304 [I 2022-10-26 17:13:09] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:34 [I 2022-10-26 17:13:09] ...trap/engines/engine.py.229: process: 0.54947 | load: 0.00251 [I 2022-10-26 17:13:09] ...trap/engines/engine.py.230: loss: 0.22790 [I 2022-10-26 17:13:09] ...optimizers/trijoint.py.151: total_norm: 28.883493 [I 2022-10-26 17:13:14] ...trap/engines/engine.py.224: train: epoch 20 | batch 1170/2304 [I 2022-10-26 17:13:14] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:28 [I 2022-10-26 17:13:14] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00254 [I 2022-10-26 17:13:14] ...trap/engines/engine.py.230: loss: 0.18514 [I 2022-10-26 17:13:14] ...optimizers/trijoint.py.151: total_norm: 18.717995 [I 2022-10-26 17:13:20] ...trap/engines/engine.py.224: train: epoch 20 | batch 1180/2304 [I 2022-10-26 17:13:20] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:21 [I 2022-10-26 17:13:20] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00250 [I 2022-10-26 17:13:20] ...trap/engines/engine.py.230: loss: 0.26266 [I 2022-10-26 17:13:20] ...optimizers/trijoint.py.151: total_norm: 27.070175 [I 2022-10-26 17:13:26] ...trap/engines/engine.py.224: train: epoch 20 | batch 1190/2304 [I 2022-10-26 17:13:26] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:10:19 [I 2022-10-26 17:13:26] ...trap/engines/engine.py.229: process: 0.55336 | load: 0.00245 [I 2022-10-26 17:13:26] ...trap/engines/engine.py.230: loss: 0.18656 [I 2022-10-26 17:13:26] ...optimizers/trijoint.py.151: total_norm: 24.487061 [I 2022-10-26 17:13:31] ...trap/engines/engine.py.224: train: epoch 20 | batch 1200/2304 [I 2022-10-26 17:13:31] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:10:16 [I 2022-10-26 17:13:31] ...trap/engines/engine.py.229: process: 0.55526 | load: 0.00250 [I 2022-10-26 17:13:31] ...trap/engines/engine.py.230: loss: 0.23915 [I 2022-10-26 17:13:31] ...optimizers/trijoint.py.151: total_norm: 24.122530 [I 2022-10-26 17:13:37] ...trap/engines/engine.py.224: train: epoch 20 | batch 1210/2304 [I 2022-10-26 17:13:37] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:10:09 [I 2022-10-26 17:13:37] ...trap/engines/engine.py.229: process: 0.56550 | load: 0.00249 [I 2022-10-26 17:13:37] ...trap/engines/engine.py.230: loss: 0.28248 [I 2022-10-26 17:13:37] ...optimizers/trijoint.py.151: total_norm: 24.672743 [I 2022-10-26 17:13:42] ...trap/engines/engine.py.224: train: epoch 20 | batch 1220/2304 [I 2022-10-26 17:13:42] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:10:05 [I 2022-10-26 17:13:42] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00255 [I 2022-10-26 17:13:42] ...trap/engines/engine.py.230: loss: 0.23261 [I 2022-10-26 17:13:42] ...optimizers/trijoint.py.151: total_norm: 27.162466 [I 2022-10-26 17:13:48] ...trap/engines/engine.py.224: train: epoch 20 | batch 1230/2304 [I 2022-10-26 17:13:48] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:10:03 [I 2022-10-26 17:13:48] ...trap/engines/engine.py.229: process: 0.55561 | load: 0.00255 [I 2022-10-26 17:13:48] ...trap/engines/engine.py.230: loss: 0.21158 [I 2022-10-26 17:13:48] ...optimizers/trijoint.py.151: total_norm: 21.055031 [I 2022-10-26 17:13:54] ...trap/engines/engine.py.224: train: epoch 20 | batch 1240/2304 [I 2022-10-26 17:13:54] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:52 [I 2022-10-26 17:13:54] ...trap/engines/engine.py.229: process: 0.56465 | load: 0.00254 [I 2022-10-26 17:13:54] ...trap/engines/engine.py.230: loss: 0.19139 [I 2022-10-26 17:13:54] ...optimizers/trijoint.py.151: total_norm: 23.819347 [I 2022-10-26 17:13:59] ...trap/engines/engine.py.224: train: epoch 20 | batch 1250/2304 [I 2022-10-26 17:13:59] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:47 [I 2022-10-26 17:13:59] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00252 [I 2022-10-26 17:13:59] ...trap/engines/engine.py.230: loss: 0.22159 [I 2022-10-26 17:13:59] ...optimizers/trijoint.py.151: total_norm: 29.162483 [I 2022-10-26 17:14:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 1260/2304 [I 2022-10-26 17:14:05] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:36 [I 2022-10-26 17:14:05] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00251 [I 2022-10-26 17:14:05] ...trap/engines/engine.py.230: loss: 0.10455 [I 2022-10-26 17:14:05] ...optimizers/trijoint.py.151: total_norm: 20.555571 [I 2022-10-26 17:14:10] ...trap/engines/engine.py.224: train: epoch 20 | batch 1270/2304 [I 2022-10-26 17:14:10] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:36 [I 2022-10-26 17:14:10] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00243 [I 2022-10-26 17:14:10] ...trap/engines/engine.py.230: loss: 0.15488 [I 2022-10-26 17:14:10] ...optimizers/trijoint.py.151: total_norm: 22.430059 [I 2022-10-26 17:14:16] ...trap/engines/engine.py.224: train: epoch 20 | batch 1280/2304 [I 2022-10-26 17:14:16] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:32 [I 2022-10-26 17:14:16] ...trap/engines/engine.py.229: process: 0.56318 | load: 0.00329 [I 2022-10-26 17:14:16] ...trap/engines/engine.py.230: loss: 0.16056 [I 2022-10-26 17:14:16] ...optimizers/trijoint.py.151: total_norm: 26.307693 [I 2022-10-26 17:14:21] ...trap/engines/engine.py.224: train: epoch 20 | batch 1290/2304 [I 2022-10-26 17:14:21] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:24 [I 2022-10-26 17:14:21] ...trap/engines/engine.py.229: process: 0.55434 | load: 0.00263 [I 2022-10-26 17:14:21] ...trap/engines/engine.py.230: loss: 0.19950 [I 2022-10-26 17:14:21] ...optimizers/trijoint.py.151: total_norm: 18.916119 [I 2022-10-26 17:14:27] ...trap/engines/engine.py.224: train: epoch 20 | batch 1300/2304 [I 2022-10-26 17:14:27] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:09:23 [I 2022-10-26 17:14:27] ...trap/engines/engine.py.229: process: 0.56764 | load: 0.00258 [I 2022-10-26 17:14:27] ...trap/engines/engine.py.230: loss: 0.20758 [I 2022-10-26 17:14:27] ...optimizers/trijoint.py.151: total_norm: 36.585663 [I 2022-10-26 17:14:33] ...trap/engines/engine.py.224: train: epoch 20 | batch 1310/2304 [I 2022-10-26 17:14:33] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:09:10 [I 2022-10-26 17:14:33] ...trap/engines/engine.py.229: process: 0.54762 | load: 0.00258 [I 2022-10-26 17:14:33] ...trap/engines/engine.py.230: loss: 0.22256 [I 2022-10-26 17:14:33] ...optimizers/trijoint.py.151: total_norm: 29.957375 [I 2022-10-26 17:14:38] ...trap/engines/engine.py.224: train: epoch 20 | batch 1320/2304 [I 2022-10-26 17:14:38] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:09:10 [I 2022-10-26 17:14:38] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00271 [I 2022-10-26 17:14:38] ...trap/engines/engine.py.230: loss: 0.16277 [I 2022-10-26 17:14:38] ...optimizers/trijoint.py.151: total_norm: 19.483021 [I 2022-10-26 17:14:44] ...trap/engines/engine.py.224: train: epoch 20 | batch 1330/2304 [I 2022-10-26 17:14:44] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:59 [I 2022-10-26 17:14:44] ...trap/engines/engine.py.229: process: 0.55419 | load: 0.00254 [I 2022-10-26 17:14:44] ...trap/engines/engine.py.230: loss: 0.16047 [I 2022-10-26 17:14:44] ...optimizers/trijoint.py.151: total_norm: 14.364242 [I 2022-10-26 17:14:49] ...trap/engines/engine.py.224: train: epoch 20 | batch 1340/2304 [I 2022-10-26 17:14:49] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:58 [I 2022-10-26 17:14:49] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00254 [I 2022-10-26 17:14:49] ...trap/engines/engine.py.230: loss: 0.20477 [I 2022-10-26 17:14:49] ...optimizers/trijoint.py.151: total_norm: 21.367224 [I 2022-10-26 17:14:55] ...trap/engines/engine.py.224: train: epoch 20 | batch 1350/2304 [I 2022-10-26 17:14:55] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:53 [I 2022-10-26 17:14:55] ...trap/engines/engine.py.229: process: 0.56417 | load: 0.00246 [I 2022-10-26 17:14:55] ...trap/engines/engine.py.230: loss: 0.28318 [I 2022-10-26 17:14:55] ...optimizers/trijoint.py.151: total_norm: 32.132355 [I 2022-10-26 17:15:00] ...trap/engines/engine.py.224: train: epoch 20 | batch 1360/2304 [I 2022-10-26 17:15:00] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:40 [I 2022-10-26 17:15:00] ...trap/engines/engine.py.229: process: 0.54541 | load: 0.00248 [I 2022-10-26 17:15:00] ...trap/engines/engine.py.230: loss: 0.22432 [I 2022-10-26 17:15:00] ...optimizers/trijoint.py.151: total_norm: 22.383650 [I 2022-10-26 17:15:06] ...trap/engines/engine.py.224: train: epoch 20 | batch 1370/2304 [I 2022-10-26 17:15:06] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:44 [I 2022-10-26 17:15:06] ...trap/engines/engine.py.229: process: 0.56584 | load: 0.00253 [I 2022-10-26 17:15:06] ...trap/engines/engine.py.230: loss: 0.31014 [I 2022-10-26 17:15:06] ...optimizers/trijoint.py.151: total_norm: 39.427868 [I 2022-10-26 17:15:12] ...trap/engines/engine.py.224: train: epoch 20 | batch 1380/2304 [I 2022-10-26 17:15:12] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:37 [I 2022-10-26 17:15:12] ...trap/engines/engine.py.229: process: 0.56500 | load: 0.00255 [I 2022-10-26 17:15:12] ...trap/engines/engine.py.230: loss: 0.27104 [I 2022-10-26 17:15:12] ...optimizers/trijoint.py.151: total_norm: 29.584438 [I 2022-10-26 17:15:17] ...trap/engines/engine.py.224: train: epoch 20 | batch 1390/2304 [I 2022-10-26 17:15:17] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:42 [I 2022-10-26 17:15:17] ...trap/engines/engine.py.229: process: 0.56783 | load: 0.00268 [I 2022-10-26 17:15:17] ...trap/engines/engine.py.230: loss: 0.16057 [I 2022-10-26 17:15:17] ...optimizers/trijoint.py.151: total_norm: 15.948243 [I 2022-10-26 17:15:23] ...trap/engines/engine.py.224: train: epoch 20 | batch 1400/2304 [I 2022-10-26 17:15:23] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:25 [I 2022-10-26 17:15:23] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00268 [I 2022-10-26 17:15:23] ...trap/engines/engine.py.230: loss: 0.21997 [I 2022-10-26 17:15:23] ...optimizers/trijoint.py.151: total_norm: 20.063353 [I 2022-10-26 17:15:29] ...trap/engines/engine.py.224: train: epoch 20 | batch 1410/2304 [I 2022-10-26 17:15:29] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:08:20 [I 2022-10-26 17:15:29] ...trap/engines/engine.py.229: process: 0.54736 | load: 0.00252 [I 2022-10-26 17:15:29] ...trap/engines/engine.py.230: loss: 0.19107 [I 2022-10-26 17:15:29] ...optimizers/trijoint.py.151: total_norm: 19.770084 [I 2022-10-26 17:15:34] ...trap/engines/engine.py.224: train: epoch 20 | batch 1420/2304 [I 2022-10-26 17:15:34] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:08:11 [I 2022-10-26 17:15:34] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00252 [I 2022-10-26 17:15:34] ...trap/engines/engine.py.230: loss: 0.24651 [I 2022-10-26 17:15:34] ...optimizers/trijoint.py.151: total_norm: 23.554554 [I 2022-10-26 17:15:40] ...trap/engines/engine.py.224: train: epoch 20 | batch 1430/2304 [I 2022-10-26 17:15:40] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:08:05 [I 2022-10-26 17:15:40] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00258 [I 2022-10-26 17:15:40] ...trap/engines/engine.py.230: loss: 0.23717 [I 2022-10-26 17:15:40] ...optimizers/trijoint.py.151: total_norm: 27.269991 [I 2022-10-26 17:15:45] ...trap/engines/engine.py.224: train: epoch 20 | batch 1440/2304 [I 2022-10-26 17:15:45] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:57 [I 2022-10-26 17:15:45] ...trap/engines/engine.py.229: process: 0.54300 | load: 0.00250 [I 2022-10-26 17:15:45] ...trap/engines/engine.py.230: loss: 0.20308 [I 2022-10-26 17:15:45] ...optimizers/trijoint.py.151: total_norm: 17.409048 [I 2022-10-26 17:15:51] ...trap/engines/engine.py.224: train: epoch 20 | batch 1450/2304 [I 2022-10-26 17:15:51] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:52 [I 2022-10-26 17:15:51] ...trap/engines/engine.py.229: process: 0.55016 | load: 0.00247 [I 2022-10-26 17:15:51] ...trap/engines/engine.py.230: loss: 0.14866 [I 2022-10-26 17:15:51] ...optimizers/trijoint.py.151: total_norm: 16.585281 [I 2022-10-26 17:15:56] ...trap/engines/engine.py.224: train: epoch 20 | batch 1460/2304 [I 2022-10-26 17:15:56] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:51 [I 2022-10-26 17:15:56] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00262 [I 2022-10-26 17:15:56] ...trap/engines/engine.py.230: loss: 0.19511 [I 2022-10-26 17:15:56] ...optimizers/trijoint.py.151: total_norm: 23.143486 [I 2022-10-26 17:16:02] ...trap/engines/engine.py.224: train: epoch 20 | batch 1470/2304 [I 2022-10-26 17:16:02] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:45 [I 2022-10-26 17:16:02] ...trap/engines/engine.py.229: process: 0.55471 | load: 0.00246 [I 2022-10-26 17:16:02] ...trap/engines/engine.py.230: loss: 0.16348 [I 2022-10-26 17:16:02] ...optimizers/trijoint.py.151: total_norm: 25.065502 [I 2022-10-26 17:16:08] ...trap/engines/engine.py.224: train: epoch 20 | batch 1480/2304 [I 2022-10-26 17:16:08] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:50 [I 2022-10-26 17:16:08] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00255 [I 2022-10-26 17:16:08] ...trap/engines/engine.py.230: loss: 0.18640 [I 2022-10-26 17:16:08] ...optimizers/trijoint.py.151: total_norm: 17.885374 [I 2022-10-26 17:16:13] ...trap/engines/engine.py.224: train: epoch 20 | batch 1490/2304 [I 2022-10-26 17:16:13] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:34 [I 2022-10-26 17:16:13] ...trap/engines/engine.py.229: process: 0.55169 | load: 0.00250 [I 2022-10-26 17:16:13] ...trap/engines/engine.py.230: loss: 0.16051 [I 2022-10-26 17:16:13] ...optimizers/trijoint.py.151: total_norm: 14.394986 [I 2022-10-26 17:16:19] ...trap/engines/engine.py.224: train: epoch 20 | batch 1500/2304 [I 2022-10-26 17:16:19] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:28 [I 2022-10-26 17:16:19] ...trap/engines/engine.py.229: process: 0.55830 | load: 0.00248 [I 2022-10-26 17:16:19] ...trap/engines/engine.py.230: loss: 0.24635 [I 2022-10-26 17:16:19] ...optimizers/trijoint.py.151: total_norm: 36.755241 [I 2022-10-26 17:16:24] ...trap/engines/engine.py.224: train: epoch 20 | batch 1510/2304 [I 2022-10-26 17:16:24] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:22 [I 2022-10-26 17:16:24] ...trap/engines/engine.py.229: process: 0.55155 | load: 0.00246 [I 2022-10-26 17:16:24] ...trap/engines/engine.py.230: loss: 0.17976 [I 2022-10-26 17:16:24] ...optimizers/trijoint.py.151: total_norm: 18.139450 [I 2022-10-26 17:16:30] ...trap/engines/engine.py.224: train: epoch 20 | batch 1520/2304 [I 2022-10-26 17:16:30] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:07:21 [I 2022-10-26 17:16:30] ...trap/engines/engine.py.229: process: 0.56642 | load: 0.00263 [I 2022-10-26 17:16:30] ...trap/engines/engine.py.230: loss: 0.22594 [I 2022-10-26 17:16:30] ...optimizers/trijoint.py.151: total_norm: 41.072205 [I 2022-10-26 17:16:36] ...trap/engines/engine.py.224: train: epoch 20 | batch 1530/2304 [I 2022-10-26 17:16:36] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:07:15 [I 2022-10-26 17:16:36] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00253 [I 2022-10-26 17:16:36] ...trap/engines/engine.py.230: loss: 0.14836 [I 2022-10-26 17:16:36] ...optimizers/trijoint.py.151: total_norm: 22.154060 [I 2022-10-26 17:16:41] ...trap/engines/engine.py.224: train: epoch 20 | batch 1540/2304 [I 2022-10-26 17:16:41] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:07:06 [I 2022-10-26 17:16:41] ...trap/engines/engine.py.229: process: 0.54298 | load: 0.00254 [I 2022-10-26 17:16:41] ...trap/engines/engine.py.230: loss: 0.17000 [I 2022-10-26 17:16:41] ...optimizers/trijoint.py.151: total_norm: 16.802513 [I 2022-10-26 17:16:47] ...trap/engines/engine.py.224: train: epoch 20 | batch 1550/2304 [I 2022-10-26 17:16:47] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:07:05 [I 2022-10-26 17:16:47] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00325 [I 2022-10-26 17:16:47] ...trap/engines/engine.py.230: loss: 0.17415 [I 2022-10-26 17:16:47] ...optimizers/trijoint.py.151: total_norm: 17.236645 [I 2022-10-26 17:16:53] ...trap/engines/engine.py.224: train: epoch 20 | batch 1560/2304 [I 2022-10-26 17:16:53] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:57 [I 2022-10-26 17:16:53] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00253 [I 2022-10-26 17:16:53] ...trap/engines/engine.py.230: loss: 0.22173 [I 2022-10-26 17:16:53] ...optimizers/trijoint.py.151: total_norm: 37.399662 [I 2022-10-26 17:16:58] ...trap/engines/engine.py.224: train: epoch 20 | batch 1570/2304 [I 2022-10-26 17:16:58] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:52 [I 2022-10-26 17:16:58] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00248 [I 2022-10-26 17:16:58] ...trap/engines/engine.py.230: loss: 0.22713 [I 2022-10-26 17:16:58] ...optimizers/trijoint.py.151: total_norm: 20.784014 [I 2022-10-26 17:17:04] ...trap/engines/engine.py.224: train: epoch 20 | batch 1580/2304 [I 2022-10-26 17:17:04] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:45 [I 2022-10-26 17:17:04] ...trap/engines/engine.py.229: process: 0.55611 | load: 0.00254 [I 2022-10-26 17:17:04] ...trap/engines/engine.py.230: loss: 0.29082 [I 2022-10-26 17:17:04] ...optimizers/trijoint.py.151: total_norm: 22.339119 [I 2022-10-26 17:17:09] ...trap/engines/engine.py.224: train: epoch 20 | batch 1590/2304 [I 2022-10-26 17:17:09] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:38 [I 2022-10-26 17:17:09] ...trap/engines/engine.py.229: process: 0.54673 | load: 0.00250 [I 2022-10-26 17:17:09] ...trap/engines/engine.py.230: loss: 0.21934 [I 2022-10-26 17:17:09] ...optimizers/trijoint.py.151: total_norm: 26.865627 [I 2022-10-26 17:17:15] ...trap/engines/engine.py.224: train: epoch 20 | batch 1600/2304 [I 2022-10-26 17:17:15] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:32 [I 2022-10-26 17:17:15] ...trap/engines/engine.py.229: process: 0.55053 | load: 0.00255 [I 2022-10-26 17:17:15] ...trap/engines/engine.py.230: loss: 0.18371 [I 2022-10-26 17:17:15] ...optimizers/trijoint.py.151: total_norm: 24.170652 [I 2022-10-26 17:17:21] ...trap/engines/engine.py.224: train: epoch 20 | batch 1610/2304 [I 2022-10-26 17:17:21] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:25 [I 2022-10-26 17:17:21] ...trap/engines/engine.py.229: process: 0.56223 | load: 0.00254 [I 2022-10-26 17:17:21] ...trap/engines/engine.py.230: loss: 0.21498 [I 2022-10-26 17:17:21] ...optimizers/trijoint.py.151: total_norm: 26.712606 [I 2022-10-26 17:17:26] ...trap/engines/engine.py.224: train: epoch 20 | batch 1620/2304 [I 2022-10-26 17:17:26] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:19 [I 2022-10-26 17:17:26] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00251 [I 2022-10-26 17:17:26] ...trap/engines/engine.py.230: loss: 0.14737 [I 2022-10-26 17:17:26] ...optimizers/trijoint.py.151: total_norm: 16.768858 [I 2022-10-26 17:17:32] ...trap/engines/engine.py.224: train: epoch 20 | batch 1630/2304 [I 2022-10-26 17:17:32] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:06:17 [I 2022-10-26 17:17:32] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00265 [I 2022-10-26 17:17:32] ...trap/engines/engine.py.230: loss: 0.17667 [I 2022-10-26 17:17:32] ...optimizers/trijoint.py.151: total_norm: 17.038311 [I 2022-10-26 17:17:37] ...trap/engines/engine.py.224: train: epoch 20 | batch 1640/2304 [I 2022-10-26 17:17:37] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:06:10 [I 2022-10-26 17:17:37] ...trap/engines/engine.py.229: process: 0.56490 | load: 0.00251 [I 2022-10-26 17:17:37] ...trap/engines/engine.py.230: loss: 0.24177 [I 2022-10-26 17:17:37] ...optimizers/trijoint.py.151: total_norm: 27.856987 [I 2022-10-26 17:17:43] ...trap/engines/engine.py.224: train: epoch 20 | batch 1650/2304 [I 2022-10-26 17:17:43] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:06:05 [I 2022-10-26 17:17:43] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00257 [I 2022-10-26 17:17:43] ...trap/engines/engine.py.230: loss: 0.17443 [I 2022-10-26 17:17:43] ...optimizers/trijoint.py.151: total_norm: 23.402763 [I 2022-10-26 17:17:49] ...trap/engines/engine.py.224: train: epoch 20 | batch 1660/2304 [I 2022-10-26 17:17:49] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:06:10 [I 2022-10-26 17:17:49] ...trap/engines/engine.py.229: process: 0.55048 | load: 0.00242 [I 2022-10-26 17:17:49] ...trap/engines/engine.py.230: loss: 0.14118 [I 2022-10-26 17:17:49] ...optimizers/trijoint.py.151: total_norm: 16.280569 [I 2022-10-26 17:17:54] ...trap/engines/engine.py.224: train: epoch 20 | batch 1670/2304 [I 2022-10-26 17:17:54] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:52 [I 2022-10-26 17:17:54] ...trap/engines/engine.py.229: process: 0.55943 | load: 0.00248 [I 2022-10-26 17:17:54] ...trap/engines/engine.py.230: loss: 0.16279 [I 2022-10-26 17:17:54] ...optimizers/trijoint.py.151: total_norm: 19.657650 [I 2022-10-26 17:18:00] ...trap/engines/engine.py.224: train: epoch 20 | batch 1680/2304 [I 2022-10-26 17:18:00] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:46 [I 2022-10-26 17:18:00] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00266 [I 2022-10-26 17:18:00] ...trap/engines/engine.py.230: loss: 0.24537 [I 2022-10-26 17:18:00] ...optimizers/trijoint.py.151: total_norm: 21.816597 [I 2022-10-26 17:18:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 1690/2304 [I 2022-10-26 17:18:05] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:38 [I 2022-10-26 17:18:05] ...trap/engines/engine.py.229: process: 0.53429 | load: 0.00249 [I 2022-10-26 17:18:05] ...trap/engines/engine.py.230: loss: 0.24432 [I 2022-10-26 17:18:05] ...optimizers/trijoint.py.151: total_norm: 23.289848 [I 2022-10-26 17:18:11] ...trap/engines/engine.py.224: train: epoch 20 | batch 1700/2304 [I 2022-10-26 17:18:11] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:35 [I 2022-10-26 17:18:11] ...trap/engines/engine.py.229: process: 0.55215 | load: 0.00255 [I 2022-10-26 17:18:11] ...trap/engines/engine.py.230: loss: 0.15364 [I 2022-10-26 17:18:11] ...optimizers/trijoint.py.151: total_norm: 10.243987 [I 2022-10-26 17:18:17] ...trap/engines/engine.py.224: train: epoch 20 | batch 1710/2304 [I 2022-10-26 17:18:17] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:30 [I 2022-10-26 17:18:17] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00249 [I 2022-10-26 17:18:17] ...trap/engines/engine.py.230: loss: 0.17075 [I 2022-10-26 17:18:17] ...optimizers/trijoint.py.151: total_norm: 24.920034 [I 2022-10-26 17:18:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 1720/2304 [I 2022-10-26 17:18:22] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:24 [I 2022-10-26 17:18:22] ...trap/engines/engine.py.229: process: 0.54509 | load: 0.00256 [I 2022-10-26 17:18:22] ...trap/engines/engine.py.230: loss: 0.26235 [I 2022-10-26 17:18:22] ...optimizers/trijoint.py.151: total_norm: 26.258587 [I 2022-10-26 17:18:28] ...trap/engines/engine.py.224: train: epoch 20 | batch 1730/2304 [I 2022-10-26 17:18:28] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:05:18 [I 2022-10-26 17:18:28] ...trap/engines/engine.py.229: process: 0.56250 | load: 0.00249 [I 2022-10-26 17:18:28] ...trap/engines/engine.py.230: loss: 0.21532 [I 2022-10-26 17:18:28] ...optimizers/trijoint.py.151: total_norm: 39.537033 [I 2022-10-26 17:18:33] ...trap/engines/engine.py.224: train: epoch 20 | batch 1740/2304 [I 2022-10-26 17:18:33] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:05:13 [I 2022-10-26 17:18:33] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00255 [I 2022-10-26 17:18:33] ...trap/engines/engine.py.230: loss: 0.21123 [I 2022-10-26 17:18:33] ...optimizers/trijoint.py.151: total_norm: 24.624588 [I 2022-10-26 17:18:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 1750/2304 [I 2022-10-26 17:18:39] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:10 [I 2022-10-26 17:18:39] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00256 [I 2022-10-26 17:18:39] ...trap/engines/engine.py.230: loss: 0.14901 [I 2022-10-26 17:18:39] ...optimizers/trijoint.py.151: total_norm: 15.483381 [I 2022-10-26 17:18:44] ...trap/engines/engine.py.224: train: epoch 20 | batch 1760/2304 [I 2022-10-26 17:18:44] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:05:01 [I 2022-10-26 17:18:44] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00249 [I 2022-10-26 17:18:44] ...trap/engines/engine.py.230: loss: 0.17057 [I 2022-10-26 17:18:44] ...optimizers/trijoint.py.151: total_norm: 13.175382 [I 2022-10-26 17:18:50] ...trap/engines/engine.py.224: train: epoch 20 | batch 1770/2304 [I 2022-10-26 17:18:50] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:57 [I 2022-10-26 17:18:50] ...trap/engines/engine.py.229: process: 0.56449 | load: 0.00245 [I 2022-10-26 17:18:50] ...trap/engines/engine.py.230: loss: 0.18652 [I 2022-10-26 17:18:50] ...optimizers/trijoint.py.151: total_norm: 14.542083 [I 2022-10-26 17:18:56] ...trap/engines/engine.py.224: train: epoch 20 | batch 1780/2304 [I 2022-10-26 17:18:56] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:54 [I 2022-10-26 17:18:56] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00245 [I 2022-10-26 17:18:56] ...trap/engines/engine.py.230: loss: 0.19766 [I 2022-10-26 17:18:56] ...optimizers/trijoint.py.151: total_norm: 16.507010 [I 2022-10-26 17:19:01] ...trap/engines/engine.py.224: train: epoch 20 | batch 1790/2304 [I 2022-10-26 17:19:01] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:42 [I 2022-10-26 17:19:01] ...trap/engines/engine.py.229: process: 0.54650 | load: 0.00252 [I 2022-10-26 17:19:01] ...trap/engines/engine.py.230: loss: 0.25845 [I 2022-10-26 17:19:01] ...optimizers/trijoint.py.151: total_norm: 50.353603 [I 2022-10-26 17:19:07] ...trap/engines/engine.py.224: train: epoch 20 | batch 1800/2304 [I 2022-10-26 17:19:07] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:41 [I 2022-10-26 17:19:07] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00252 [I 2022-10-26 17:19:07] ...trap/engines/engine.py.230: loss: 0.19441 [I 2022-10-26 17:19:07] ...optimizers/trijoint.py.151: total_norm: 16.890659 [I 2022-10-26 17:19:12] ...trap/engines/engine.py.224: train: epoch 20 | batch 1810/2304 [I 2022-10-26 17:19:12] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:35 [I 2022-10-26 17:19:12] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00245 [I 2022-10-26 17:19:12] ...trap/engines/engine.py.230: loss: 0.10965 [I 2022-10-26 17:19:12] ...optimizers/trijoint.py.151: total_norm: 21.090569 [I 2022-10-26 17:19:18] ...trap/engines/engine.py.224: train: epoch 20 | batch 1820/2304 [I 2022-10-26 17:19:18] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:30 [I 2022-10-26 17:19:18] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00245 [I 2022-10-26 17:19:18] ...trap/engines/engine.py.230: loss: 0.18195 [I 2022-10-26 17:19:18] ...optimizers/trijoint.py.151: total_norm: 26.349417 [I 2022-10-26 17:19:24] ...trap/engines/engine.py.224: train: epoch 20 | batch 1830/2304 [I 2022-10-26 17:19:24] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:25 [I 2022-10-26 17:19:24] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00256 [I 2022-10-26 17:19:24] ...trap/engines/engine.py.230: loss: 0.19024 [I 2022-10-26 17:19:24] ...optimizers/trijoint.py.151: total_norm: 16.822838 [I 2022-10-26 17:19:29] ...trap/engines/engine.py.224: train: epoch 20 | batch 1840/2304 [I 2022-10-26 17:19:29] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:04:18 [I 2022-10-26 17:19:29] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00249 [I 2022-10-26 17:19:29] ...trap/engines/engine.py.230: loss: 0.17621 [I 2022-10-26 17:19:29] ...optimizers/trijoint.py.151: total_norm: 22.878454 [I 2022-10-26 17:19:35] ...trap/engines/engine.py.224: train: epoch 20 | batch 1850/2304 [I 2022-10-26 17:19:35] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:04:10 [I 2022-10-26 17:19:35] ...trap/engines/engine.py.229: process: 0.55228 | load: 0.00249 [I 2022-10-26 17:19:35] ...trap/engines/engine.py.230: loss: 0.16491 [I 2022-10-26 17:19:35] ...optimizers/trijoint.py.151: total_norm: 19.668083 [I 2022-10-26 17:19:40] ...trap/engines/engine.py.224: train: epoch 20 | batch 1860/2304 [I 2022-10-26 17:19:40] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:04:06 [I 2022-10-26 17:19:40] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00259 [I 2022-10-26 17:19:40] ...trap/engines/engine.py.230: loss: 0.21188 [I 2022-10-26 17:19:40] ...optimizers/trijoint.py.151: total_norm: 27.143532 [I 2022-10-26 17:19:46] ...trap/engines/engine.py.224: train: epoch 20 | batch 1870/2304 [I 2022-10-26 17:19:46] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:04:00 [I 2022-10-26 17:19:46] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00254 [I 2022-10-26 17:19:46] ...trap/engines/engine.py.230: loss: 0.20968 [I 2022-10-26 17:19:46] ...optimizers/trijoint.py.151: total_norm: 26.110554 [I 2022-10-26 17:19:51] ...trap/engines/engine.py.224: train: epoch 20 | batch 1880/2304 [I 2022-10-26 17:19:51] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:57 [I 2022-10-26 17:19:51] ...trap/engines/engine.py.229: process: 0.56547 | load: 0.00245 [I 2022-10-26 17:19:51] ...trap/engines/engine.py.230: loss: 0.27727 [I 2022-10-26 17:19:51] ...optimizers/trijoint.py.151: total_norm: 32.847267 [I 2022-10-26 17:19:57] ...trap/engines/engine.py.224: train: epoch 20 | batch 1890/2304 [I 2022-10-26 17:19:57] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:53 [I 2022-10-26 17:19:57] ...trap/engines/engine.py.229: process: 0.55372 | load: 0.00270 [I 2022-10-26 17:19:57] ...trap/engines/engine.py.230: loss: 0.27702 [I 2022-10-26 17:19:57] ...optimizers/trijoint.py.151: total_norm: 26.324184 [I 2022-10-26 17:20:03] ...trap/engines/engine.py.224: train: epoch 20 | batch 1900/2304 [I 2022-10-26 17:20:03] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:45 [I 2022-10-26 17:20:03] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00252 [I 2022-10-26 17:20:03] ...trap/engines/engine.py.230: loss: 0.27616 [I 2022-10-26 17:20:03] ...optimizers/trijoint.py.151: total_norm: 24.440269 [I 2022-10-26 17:20:08] ...trap/engines/engine.py.224: train: epoch 20 | batch 1910/2304 [I 2022-10-26 17:20:08] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:37 [I 2022-10-26 17:20:08] ...trap/engines/engine.py.229: process: 0.55242 | load: 0.00247 [I 2022-10-26 17:20:08] ...trap/engines/engine.py.230: loss: 0.19330 [I 2022-10-26 17:20:08] ...optimizers/trijoint.py.151: total_norm: 27.317520 [I 2022-10-26 17:20:14] ...trap/engines/engine.py.224: train: epoch 20 | batch 1920/2304 [I 2022-10-26 17:20:14] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:33 [I 2022-10-26 17:20:14] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00249 [I 2022-10-26 17:20:14] ...trap/engines/engine.py.230: loss: 0.25941 [I 2022-10-26 17:20:14] ...optimizers/trijoint.py.151: total_norm: 29.895323 [I 2022-10-26 17:20:19] ...trap/engines/engine.py.224: train: epoch 20 | batch 1930/2304 [I 2022-10-26 17:20:19] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:27 [I 2022-10-26 17:20:19] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00243 [I 2022-10-26 17:20:19] ...trap/engines/engine.py.230: loss: 0.19599 [I 2022-10-26 17:20:19] ...optimizers/trijoint.py.151: total_norm: 19.102806 [I 2022-10-26 17:20:25] ...trap/engines/engine.py.224: train: epoch 20 | batch 1940/2304 [I 2022-10-26 17:20:25] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:23 [I 2022-10-26 17:20:25] ...trap/engines/engine.py.229: process: 0.56156 | load: 0.00241 [I 2022-10-26 17:20:25] ...trap/engines/engine.py.230: loss: 0.21354 [I 2022-10-26 17:20:25] ...optimizers/trijoint.py.151: total_norm: 30.748144 [I 2022-10-26 17:20:31] ...trap/engines/engine.py.224: train: epoch 20 | batch 1950/2304 [I 2022-10-26 17:20:31] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:03:17 [I 2022-10-26 17:20:31] ...trap/engines/engine.py.229: process: 0.55088 | load: 0.00244 [I 2022-10-26 17:20:31] ...trap/engines/engine.py.230: loss: 0.17756 [I 2022-10-26 17:20:31] ...optimizers/trijoint.py.151: total_norm: 25.385683 [I 2022-10-26 17:20:36] ...trap/engines/engine.py.224: train: epoch 20 | batch 1960/2304 [I 2022-10-26 17:20:36] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:03:11 [I 2022-10-26 17:20:36] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00250 [I 2022-10-26 17:20:36] ...trap/engines/engine.py.230: loss: 0.26810 [I 2022-10-26 17:20:36] ...optimizers/trijoint.py.151: total_norm: 20.983540 [I 2022-10-26 17:20:42] ...trap/engines/engine.py.224: train: epoch 20 | batch 1970/2304 [I 2022-10-26 17:20:42] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:03:05 [I 2022-10-26 17:20:42] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00253 [I 2022-10-26 17:20:42] ...trap/engines/engine.py.230: loss: 0.15648 [I 2022-10-26 17:20:42] ...optimizers/trijoint.py.151: total_norm: 16.657877 [I 2022-10-26 17:20:47] ...trap/engines/engine.py.224: train: epoch 20 | batch 1980/2304 [I 2022-10-26 17:20:47] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:03:00 [I 2022-10-26 17:20:47] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00262 [I 2022-10-26 17:20:47] ...trap/engines/engine.py.230: loss: 0.20293 [I 2022-10-26 17:20:47] ...optimizers/trijoint.py.151: total_norm: 30.344582 [I 2022-10-26 17:20:53] ...trap/engines/engine.py.224: train: epoch 20 | batch 1990/2304 [I 2022-10-26 17:20:53] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:54 [I 2022-10-26 17:20:53] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00263 [I 2022-10-26 17:20:53] ...trap/engines/engine.py.230: loss: 0.14711 [I 2022-10-26 17:20:53] ...optimizers/trijoint.py.151: total_norm: 16.289326 [I 2022-10-26 17:20:58] ...trap/engines/engine.py.224: train: epoch 20 | batch 2000/2304 [I 2022-10-26 17:20:58] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:49 [I 2022-10-26 17:20:58] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00259 [I 2022-10-26 17:20:58] ...trap/engines/engine.py.230: loss: 0.20197 [I 2022-10-26 17:20:58] ...optimizers/trijoint.py.151: total_norm: 24.487797 [I 2022-10-26 17:21:04] ...trap/engines/engine.py.224: train: epoch 20 | batch 2010/2304 [I 2022-10-26 17:21:04] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:44 [I 2022-10-26 17:21:04] ...trap/engines/engine.py.229: process: 0.54557 | load: 0.00250 [I 2022-10-26 17:21:04] ...trap/engines/engine.py.230: loss: 0.19409 [I 2022-10-26 17:21:04] ...optimizers/trijoint.py.151: total_norm: 19.993578 [I 2022-10-26 17:21:10] ...trap/engines/engine.py.224: train: epoch 20 | batch 2020/2304 [I 2022-10-26 17:21:10] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:38 [I 2022-10-26 17:21:10] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00258 [I 2022-10-26 17:21:10] ...trap/engines/engine.py.230: loss: 0.19618 [I 2022-10-26 17:21:10] ...optimizers/trijoint.py.151: total_norm: 34.593395 [I 2022-10-26 17:21:15] ...trap/engines/engine.py.224: train: epoch 20 | batch 2030/2304 [I 2022-10-26 17:21:15] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:32 [I 2022-10-26 17:21:15] ...trap/engines/engine.py.229: process: 0.54528 | load: 0.00252 [I 2022-10-26 17:21:15] ...trap/engines/engine.py.230: loss: 0.25804 [I 2022-10-26 17:21:15] ...optimizers/trijoint.py.151: total_norm: 26.327864 [I 2022-10-26 17:21:21] ...trap/engines/engine.py.224: train: epoch 20 | batch 2040/2304 [I 2022-10-26 17:21:21] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:27 [I 2022-10-26 17:21:21] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00257 [I 2022-10-26 17:21:21] ...trap/engines/engine.py.230: loss: 0.15839 [I 2022-10-26 17:21:21] ...optimizers/trijoint.py.151: total_norm: 15.339240 [I 2022-10-26 17:21:26] ...trap/engines/engine.py.224: train: epoch 20 | batch 2050/2304 [I 2022-10-26 17:21:26] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:02:22 [I 2022-10-26 17:21:26] ...trap/engines/engine.py.229: process: 0.57148 | load: 0.00256 [I 2022-10-26 17:21:26] ...trap/engines/engine.py.230: loss: 0.16613 [I 2022-10-26 17:21:26] ...optimizers/trijoint.py.151: total_norm: 12.404603 [I 2022-10-26 17:21:32] ...trap/engines/engine.py.224: train: epoch 20 | batch 2060/2304 [I 2022-10-26 17:21:32] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:02:15 [I 2022-10-26 17:21:32] ...trap/engines/engine.py.229: process: 0.54651 | load: 0.00252 [I 2022-10-26 17:21:32] ...trap/engines/engine.py.230: loss: 0.27607 [I 2022-10-26 17:21:32] ...optimizers/trijoint.py.151: total_norm: 21.129248 [I 2022-10-26 17:21:37] ...trap/engines/engine.py.224: train: epoch 20 | batch 2070/2304 [I 2022-10-26 17:21:37] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:02:09 [I 2022-10-26 17:21:37] ...trap/engines/engine.py.229: process: 0.54992 | load: 0.00249 [I 2022-10-26 17:21:37] ...trap/engines/engine.py.230: loss: 0.20970 [I 2022-10-26 17:21:37] ...optimizers/trijoint.py.151: total_norm: 33.592030 [I 2022-10-26 17:21:43] ...trap/engines/engine.py.224: train: epoch 20 | batch 2080/2304 [I 2022-10-26 17:21:43] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:02:04 [I 2022-10-26 17:21:43] ...trap/engines/engine.py.229: process: 0.54994 | load: 0.00249 [I 2022-10-26 17:21:43] ...trap/engines/engine.py.230: loss: 0.22407 [I 2022-10-26 17:21:43] ...optimizers/trijoint.py.151: total_norm: 27.306696 [I 2022-10-26 17:21:49] ...trap/engines/engine.py.224: train: epoch 20 | batch 2090/2304 [I 2022-10-26 17:21:49] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:58 [I 2022-10-26 17:21:49] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00255 [I 2022-10-26 17:21:49] ...trap/engines/engine.py.230: loss: 0.25000 [I 2022-10-26 17:21:49] ...optimizers/trijoint.py.151: total_norm: 28.719290 [I 2022-10-26 17:21:54] ...trap/engines/engine.py.224: train: epoch 20 | batch 2100/2304 [I 2022-10-26 17:21:54] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:52 [I 2022-10-26 17:21:54] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00261 [I 2022-10-26 17:21:54] ...trap/engines/engine.py.230: loss: 0.33693 [I 2022-10-26 17:21:54] ...optimizers/trijoint.py.151: total_norm: 30.751915 [I 2022-10-26 17:22:00] ...trap/engines/engine.py.224: train: epoch 20 | batch 2110/2304 [I 2022-10-26 17:22:00] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:47 [I 2022-10-26 17:22:00] ...trap/engines/engine.py.229: process: 0.55387 | load: 0.00253 [I 2022-10-26 17:22:00] ...trap/engines/engine.py.230: loss: 0.22506 [I 2022-10-26 17:22:00] ...optimizers/trijoint.py.151: total_norm: 26.074245 [I 2022-10-26 17:22:05] ...trap/engines/engine.py.224: train: epoch 20 | batch 2120/2304 [I 2022-10-26 17:22:05] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:42 [I 2022-10-26 17:22:05] ...trap/engines/engine.py.229: process: 0.56672 | load: 0.00259 [I 2022-10-26 17:22:05] ...trap/engines/engine.py.230: loss: 0.20167 [I 2022-10-26 17:22:05] ...optimizers/trijoint.py.151: total_norm: 23.100964 [I 2022-10-26 17:22:11] ...trap/engines/engine.py.224: train: epoch 20 | batch 2130/2304 [I 2022-10-26 17:22:11] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:36 [I 2022-10-26 17:22:11] ...trap/engines/engine.py.229: process: 0.54389 | load: 0.00250 [I 2022-10-26 17:22:11] ...trap/engines/engine.py.230: loss: 0.21842 [I 2022-10-26 17:22:11] ...optimizers/trijoint.py.151: total_norm: 29.409189 [I 2022-10-26 17:22:16] ...trap/engines/engine.py.224: train: epoch 20 | batch 2140/2304 [I 2022-10-26 17:22:16] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:30 [I 2022-10-26 17:22:16] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00258 [I 2022-10-26 17:22:16] ...trap/engines/engine.py.230: loss: 0.20817 [I 2022-10-26 17:22:16] ...optimizers/trijoint.py.151: total_norm: 16.269133 [I 2022-10-26 17:22:22] ...trap/engines/engine.py.224: train: epoch 20 | batch 2150/2304 [I 2022-10-26 17:22:22] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:25 [I 2022-10-26 17:22:22] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00249 [I 2022-10-26 17:22:22] ...trap/engines/engine.py.230: loss: 0.28043 [I 2022-10-26 17:22:22] ...optimizers/trijoint.py.151: total_norm: 24.626238 [I 2022-10-26 17:22:27] ...trap/engines/engine.py.224: train: epoch 20 | batch 2160/2304 [I 2022-10-26 17:22:27] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:19 [I 2022-10-26 17:22:27] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00267 [I 2022-10-26 17:22:27] ...trap/engines/engine.py.230: loss: 0.13390 [I 2022-10-26 17:22:27] ...optimizers/trijoint.py.151: total_norm: 18.357372 [I 2022-10-26 17:22:33] ...trap/engines/engine.py.224: train: epoch 20 | batch 2170/2304 [I 2022-10-26 17:22:33] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:01:14 [I 2022-10-26 17:22:33] ...trap/engines/engine.py.229: process: 0.54724 | load: 0.00250 [I 2022-10-26 17:22:33] ...trap/engines/engine.py.230: loss: 0.14772 [I 2022-10-26 17:22:33] ...optimizers/trijoint.py.151: total_norm: 15.979664 [I 2022-10-26 17:22:39] ...trap/engines/engine.py.224: train: epoch 20 | batch 2180/2304 [I 2022-10-26 17:22:39] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:08 [I 2022-10-26 17:22:39] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00244 [I 2022-10-26 17:22:39] ...trap/engines/engine.py.230: loss: 0.26430 [I 2022-10-26 17:22:39] ...optimizers/trijoint.py.151: total_norm: 34.986675 [I 2022-10-26 17:22:44] ...trap/engines/engine.py.224: train: epoch 20 | batch 2190/2304 [I 2022-10-26 17:22:44] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:01:02 [I 2022-10-26 17:22:44] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00250 [I 2022-10-26 17:22:44] ...trap/engines/engine.py.230: loss: 0.16226 [I 2022-10-26 17:22:44] ...optimizers/trijoint.py.151: total_norm: 22.289408 [I 2022-10-26 17:22:50] ...trap/engines/engine.py.224: train: epoch 20 | batch 2200/2304 [I 2022-10-26 17:22:50] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:57 [I 2022-10-26 17:22:50] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00247 [I 2022-10-26 17:22:50] ...trap/engines/engine.py.230: loss: 0.17718 [I 2022-10-26 17:22:50] ...optimizers/trijoint.py.151: total_norm: 23.835651 [I 2022-10-26 17:22:55] ...trap/engines/engine.py.224: train: epoch 20 | batch 2210/2304 [I 2022-10-26 17:22:55] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:52 [I 2022-10-26 17:22:55] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00244 [I 2022-10-26 17:22:55] ...trap/engines/engine.py.230: loss: 0.13056 [I 2022-10-26 17:22:55] ...optimizers/trijoint.py.151: total_norm: 12.221087 [I 2022-10-26 17:23:01] ...trap/engines/engine.py.224: train: epoch 20 | batch 2220/2304 [I 2022-10-26 17:23:01] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:46 [I 2022-10-26 17:23:01] ...trap/engines/engine.py.229: process: 0.55583 | load: 0.00249 [I 2022-10-26 17:23:01] ...trap/engines/engine.py.230: loss: 0.28010 [I 2022-10-26 17:23:01] ...optimizers/trijoint.py.151: total_norm: 40.020393 [I 2022-10-26 17:23:06] ...trap/engines/engine.py.224: train: epoch 20 | batch 2230/2304 [I 2022-10-26 17:23:06] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:40 [I 2022-10-26 17:23:06] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00255 [I 2022-10-26 17:23:06] ...trap/engines/engine.py.230: loss: 0.23679 [I 2022-10-26 17:23:06] ...optimizers/trijoint.py.151: total_norm: 24.026155 [I 2022-10-26 17:23:12] ...trap/engines/engine.py.224: train: epoch 20 | batch 2240/2304 [I 2022-10-26 17:23:12] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:35 [I 2022-10-26 17:23:12] ...trap/engines/engine.py.229: process: 0.55421 | load: 0.00251 [I 2022-10-26 17:23:12] ...trap/engines/engine.py.230: loss: 0.22635 [I 2022-10-26 17:23:12] ...optimizers/trijoint.py.151: total_norm: 33.569912 [I 2022-10-26 17:23:17] ...trap/engines/engine.py.224: train: epoch 20 | batch 2250/2304 [I 2022-10-26 17:23:17] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:29 [I 2022-10-26 17:23:17] ...trap/engines/engine.py.229: process: 0.54880 | load: 0.00246 [I 2022-10-26 17:23:17] ...trap/engines/engine.py.230: loss: 0.20718 [I 2022-10-26 17:23:17] ...optimizers/trijoint.py.151: total_norm: 28.190332 [I 2022-10-26 17:23:23] ...trap/engines/engine.py.224: train: epoch 20 | batch 2260/2304 [I 2022-10-26 17:23:23] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:24 [I 2022-10-26 17:23:23] ...trap/engines/engine.py.229: process: 0.55337 | load: 0.00248 [I 2022-10-26 17:23:23] ...trap/engines/engine.py.230: loss: 0.18567 [I 2022-10-26 17:23:23] ...optimizers/trijoint.py.151: total_norm: 18.543669 [I 2022-10-26 17:23:29] ...trap/engines/engine.py.224: train: epoch 20 | batch 2270/2304 [I 2022-10-26 17:23:29] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:18 [I 2022-10-26 17:23:29] ...trap/engines/engine.py.229: process: 0.54624 | load: 0.00254 [I 2022-10-26 17:23:29] ...trap/engines/engine.py.230: loss: 0.25206 [I 2022-10-26 17:23:29] ...optimizers/trijoint.py.151: total_norm: 16.644787 [I 2022-10-26 17:23:34] ...trap/engines/engine.py.224: train: epoch 20 | batch 2280/2304 [I 2022-10-26 17:23:34] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:13 [I 2022-10-26 17:23:34] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00256 [I 2022-10-26 17:23:34] ...trap/engines/engine.py.230: loss: 0.21366 [I 2022-10-26 17:23:34] ...optimizers/trijoint.py.151: total_norm: 21.705252 [I 2022-10-26 17:23:40] ...trap/engines/engine.py.224: train: epoch 20 | batch 2290/2304 [I 2022-10-26 17:23:40] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:00:07 [I 2022-10-26 17:23:40] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00248 [I 2022-10-26 17:23:40] ...trap/engines/engine.py.230: loss: 0.16839 [I 2022-10-26 17:23:40] ...optimizers/trijoint.py.151: total_norm: 15.636817 [I 2022-10-26 17:23:45] ...trap/engines/engine.py.224: train: epoch 20 | batch 2300/2304 [I 2022-10-26 17:23:45] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:02 [I 2022-10-26 17:23:45] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00025 [I 2022-10-26 17:23:45] ...trap/engines/engine.py.230: loss: 0.20891 [I 2022-10-26 17:23:45] ...optimizers/trijoint.py.151: total_norm: 25.571590 [I 2022-10-26 17:23:48] ...trap/engines/engine.py.224: train: epoch 20 | batch 2304/2304 [I 2022-10-26 17:23:48] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:00 [I 2022-10-26 17:23:48] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00016 [I 2022-10-26 17:23:48] ...trap/engines/engine.py.230: loss: 0.18340 [I 2022-10-26 17:23:48] ...optimizers/trijoint.py.151: total_norm: 33.070900 [S 2022-10-26 17:23:49] ...trap/engines/engine.py.129: train_epoch.epoch: 20 [S 2022-10-26 17:23:49] ...trap/engines/engine.py.129: train_epoch.loss: 0.19729220183836924 [I 2022-10-26 17:23:49] ...s/criterions/triplet.py.84: margin = 0.15500000000000008 [I 2022-10-26 17:23:49] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 20 [I 2022-10-26 17:23:49] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 17:23:51] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 17:23:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 17:23:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 17:23:51] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 17:23:52] ...trap/engines/engine.py.321: eval: epoch 20 | batch 0/510 [I 2022-10-26 17:23:52] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:25:57 [I 2022-10-26 17:23:52] ...trap/engines/engine.py.326: process: 3.05376 | load: 2.41987 [I 2022-10-26 17:23:55] ...trap/engines/engine.py.321: eval: epoch 20 | batch 10/510 [I 2022-10-26 17:23:55] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:13 [I 2022-10-26 17:23:55] ...trap/engines/engine.py.326: process: 0.32372 | load: 0.00017 [I 2022-10-26 17:23:59] ...trap/engines/engine.py.321: eval: epoch 20 | batch 20/510 [I 2022-10-26 17:23:59] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:03:01 [I 2022-10-26 17:23:59] ...trap/engines/engine.py.326: process: 0.36675 | load: 0.00017 [I 2022-10-26 17:24:02] ...trap/engines/engine.py.321: eval: epoch 20 | batch 30/510 [I 2022-10-26 17:24:02] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:40 [I 2022-10-26 17:24:02] ...trap/engines/engine.py.326: process: 0.32661 | load: 0.00016 [I 2022-10-26 17:24:05] ...trap/engines/engine.py.321: eval: epoch 20 | batch 40/510 [I 2022-10-26 17:24:05] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:35 [I 2022-10-26 17:24:05] ...trap/engines/engine.py.326: process: 0.32740 | load: 0.00017 [I 2022-10-26 17:24:09] ...trap/engines/engine.py.321: eval: epoch 20 | batch 50/510 [I 2022-10-26 17:24:09] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-26 17:24:09] ...trap/engines/engine.py.326: process: 0.36268 | load: 0.00016 [I 2022-10-26 17:24:12] ...trap/engines/engine.py.321: eval: epoch 20 | batch 60/510 [I 2022-10-26 17:24:12] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:29 [I 2022-10-26 17:24:12] ...trap/engines/engine.py.326: process: 0.34165 | load: 0.00017 [I 2022-10-26 17:24:15] ...trap/engines/engine.py.321: eval: epoch 20 | batch 70/510 [I 2022-10-26 17:24:15] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:26 [I 2022-10-26 17:24:15] ...trap/engines/engine.py.326: process: 0.33680 | load: 0.00037 [I 2022-10-26 17:24:19] ...trap/engines/engine.py.321: eval: epoch 20 | batch 80/510 [I 2022-10-26 17:24:19] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:23 [I 2022-10-26 17:24:19] ...trap/engines/engine.py.326: process: 0.35136 | load: 0.00017 [I 2022-10-26 17:24:22] ...trap/engines/engine.py.321: eval: epoch 20 | batch 90/510 [I 2022-10-26 17:24:22] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:21 [I 2022-10-26 17:24:22] ...trap/engines/engine.py.326: process: 0.35373 | load: 0.00017 [I 2022-10-26 17:24:25] ...trap/engines/engine.py.321: eval: epoch 20 | batch 100/510 [I 2022-10-26 17:24:25] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 17:24:25] ...trap/engines/engine.py.326: process: 0.29150 | load: 0.00016 [I 2022-10-26 17:24:28] ...trap/engines/engine.py.321: eval: epoch 20 | batch 110/510 [I 2022-10-26 17:24:28] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 17:24:28] ...trap/engines/engine.py.326: process: 0.28228 | load: 0.00018 [I 2022-10-26 17:24:31] ...trap/engines/engine.py.321: eval: epoch 20 | batch 120/510 [I 2022-10-26 17:24:31] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-26 17:24:31] ...trap/engines/engine.py.326: process: 0.28959 | load: 0.00014 [I 2022-10-26 17:24:34] ...trap/engines/engine.py.321: eval: epoch 20 | batch 130/510 [I 2022-10-26 17:24:34] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 17:24:34] ...trap/engines/engine.py.326: process: 0.29232 | load: 0.00017 [I 2022-10-26 17:24:37] ...trap/engines/engine.py.321: eval: epoch 20 | batch 140/510 [I 2022-10-26 17:24:37] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:48 [I 2022-10-26 17:24:37] ...trap/engines/engine.py.326: process: 0.30473 | load: 0.00014 [I 2022-10-26 17:24:40] ...trap/engines/engine.py.321: eval: epoch 20 | batch 150/510 [I 2022-10-26 17:24:40] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:43 [I 2022-10-26 17:24:40] ...trap/engines/engine.py.326: process: 0.28522 | load: 0.00014 [I 2022-10-26 17:24:43] ...trap/engines/engine.py.321: eval: epoch 20 | batch 160/510 [I 2022-10-26 17:24:43] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 17:24:43] ...trap/engines/engine.py.326: process: 0.28704 | load: 0.00018 [I 2022-10-26 17:24:46] ...trap/engines/engine.py.321: eval: epoch 20 | batch 170/510 [I 2022-10-26 17:24:46] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 17:24:46] ...trap/engines/engine.py.326: process: 0.29047 | load: 0.00014 [I 2022-10-26 17:24:49] ...trap/engines/engine.py.321: eval: epoch 20 | batch 180/510 [I 2022-10-26 17:24:49] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:34 [I 2022-10-26 17:24:49] ...trap/engines/engine.py.326: process: 0.28682 | load: 0.00015 [I 2022-10-26 17:24:51] ...trap/engines/engine.py.321: eval: epoch 20 | batch 190/510 [I 2022-10-26 17:24:51] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 17:24:51] ...trap/engines/engine.py.326: process: 0.27886 | load: 0.00015 [I 2022-10-26 17:24:54] ...trap/engines/engine.py.321: eval: epoch 20 | batch 200/510 [I 2022-10-26 17:24:54] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 17:24:54] ...trap/engines/engine.py.326: process: 0.28769 | load: 0.00015 [I 2022-10-26 17:24:57] ...trap/engines/engine.py.321: eval: epoch 20 | batch 210/510 [I 2022-10-26 17:24:57] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-26 17:24:57] ...trap/engines/engine.py.326: process: 0.30591 | load: 0.00014 [I 2022-10-26 17:25:00] ...trap/engines/engine.py.321: eval: epoch 20 | batch 220/510 [I 2022-10-26 17:25:00] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:25 [I 2022-10-26 17:25:00] ...trap/engines/engine.py.326: process: 0.30825 | load: 0.00014 [I 2022-10-26 17:25:03] ...trap/engines/engine.py.321: eval: epoch 20 | batch 230/510 [I 2022-10-26 17:25:03] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:21 [I 2022-10-26 17:25:03] ...trap/engines/engine.py.326: process: 0.28021 | load: 0.00020 [I 2022-10-26 17:25:06] ...trap/engines/engine.py.321: eval: epoch 20 | batch 240/510 [I 2022-10-26 17:25:06] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:17 [I 2022-10-26 17:25:06] ...trap/engines/engine.py.326: process: 0.29020 | load: 0.00015 [I 2022-10-26 17:25:09] ...trap/engines/engine.py.321: eval: epoch 20 | batch 250/510 [I 2022-10-26 17:25:09] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:15 [I 2022-10-26 17:25:09] ...trap/engines/engine.py.326: process: 0.28783 | load: 0.00015 [I 2022-10-26 17:25:12] ...trap/engines/engine.py.321: eval: epoch 20 | batch 260/510 [I 2022-10-26 17:25:12] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:11 [I 2022-10-26 17:25:12] ...trap/engines/engine.py.326: process: 0.28201 | load: 0.00015 [I 2022-10-26 17:25:15] ...trap/engines/engine.py.321: eval: epoch 20 | batch 270/510 [I 2022-10-26 17:25:15] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-26 17:25:15] ...trap/engines/engine.py.326: process: 0.28590 | load: 0.00015 [I 2022-10-26 17:25:17] ...trap/engines/engine.py.321: eval: epoch 20 | batch 280/510 [I 2022-10-26 17:25:17] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 17:25:17] ...trap/engines/engine.py.326: process: 0.29015 | load: 0.00015 [I 2022-10-26 17:25:20] ...trap/engines/engine.py.321: eval: epoch 20 | batch 290/510 [I 2022-10-26 17:25:20] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:05 [I 2022-10-26 17:25:20] ...trap/engines/engine.py.326: process: 0.30616 | load: 0.00017 [I 2022-10-26 17:25:23] ...trap/engines/engine.py.321: eval: epoch 20 | batch 300/510 [I 2022-10-26 17:25:23] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 17:25:23] ...trap/engines/engine.py.326: process: 0.29379 | load: 0.00014 [I 2022-10-26 17:25:26] ...trap/engines/engine.py.321: eval: epoch 20 | batch 310/510 [I 2022-10-26 17:25:26] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 17:25:26] ...trap/engines/engine.py.326: process: 0.28733 | load: 0.00017 [I 2022-10-26 17:25:29] ...trap/engines/engine.py.321: eval: epoch 20 | batch 320/510 [I 2022-10-26 17:25:29] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:55 [I 2022-10-26 17:25:29] ...trap/engines/engine.py.326: process: 0.29267 | load: 0.00015 [I 2022-10-26 17:25:32] ...trap/engines/engine.py.321: eval: epoch 20 | batch 330/510 [I 2022-10-26 17:25:32] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:52 [I 2022-10-26 17:25:32] ...trap/engines/engine.py.326: process: 0.29076 | load: 0.00021 [I 2022-10-26 17:25:35] ...trap/engines/engine.py.321: eval: epoch 20 | batch 340/510 [I 2022-10-26 17:25:35] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:49 [I 2022-10-26 17:25:35] ...trap/engines/engine.py.326: process: 0.29199 | load: 0.00022 [I 2022-10-26 17:25:38] ...trap/engines/engine.py.321: eval: epoch 20 | batch 350/510 [I 2022-10-26 17:25:38] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:47 [I 2022-10-26 17:25:38] ...trap/engines/engine.py.326: process: 0.31045 | load: 0.00015 [I 2022-10-26 17:25:41] ...trap/engines/engine.py.321: eval: epoch 20 | batch 360/510 [I 2022-10-26 17:25:41] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 17:25:41] ...trap/engines/engine.py.326: process: 0.28743 | load: 0.00019 [I 2022-10-26 17:25:44] ...trap/engines/engine.py.321: eval: epoch 20 | batch 370/510 [I 2022-10-26 17:25:44] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 17:25:44] ...trap/engines/engine.py.326: process: 0.28712 | load: 0.00016 [I 2022-10-26 17:25:47] ...trap/engines/engine.py.321: eval: epoch 20 | batch 380/510 [I 2022-10-26 17:25:47] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 17:25:47] ...trap/engines/engine.py.326: process: 0.29002 | load: 0.00015 [I 2022-10-26 17:25:49] ...trap/engines/engine.py.321: eval: epoch 20 | batch 390/510 [I 2022-10-26 17:25:49] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 17:25:49] ...trap/engines/engine.py.326: process: 0.28793 | load: 0.00015 [I 2022-10-26 17:25:52] ...trap/engines/engine.py.321: eval: epoch 20 | batch 400/510 [I 2022-10-26 17:25:52] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 17:25:52] ...trap/engines/engine.py.326: process: 0.28943 | load: 0.00021 [I 2022-10-26 17:25:55] ...trap/engines/engine.py.321: eval: epoch 20 | batch 410/510 [I 2022-10-26 17:25:55] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 17:25:55] ...trap/engines/engine.py.326: process: 0.28800 | load: 0.00020 [I 2022-10-26 17:25:58] ...trap/engines/engine.py.321: eval: epoch 20 | batch 420/510 [I 2022-10-26 17:25:58] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-26 17:25:58] ...trap/engines/engine.py.326: process: 0.28959 | load: 0.00019 [I 2022-10-26 17:26:01] ...trap/engines/engine.py.321: eval: epoch 20 | batch 430/510 [I 2022-10-26 17:26:01] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:22 [I 2022-10-26 17:26:01] ...trap/engines/engine.py.326: process: 0.28500 | load: 0.00040 [I 2022-10-26 17:26:04] ...trap/engines/engine.py.321: eval: epoch 20 | batch 440/510 [I 2022-10-26 17:26:04] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 17:26:04] ...trap/engines/engine.py.326: process: 0.29105 | load: 0.00024 [I 2022-10-26 17:26:07] ...trap/engines/engine.py.321: eval: epoch 20 | batch 450/510 [I 2022-10-26 17:26:07] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 17:26:07] ...trap/engines/engine.py.326: process: 0.28709 | load: 0.00016 [I 2022-10-26 17:26:10] ...trap/engines/engine.py.321: eval: epoch 20 | batch 460/510 [I 2022-10-26 17:26:10] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 17:26:10] ...trap/engines/engine.py.326: process: 0.29304 | load: 0.00024 [I 2022-10-26 17:26:13] ...trap/engines/engine.py.321: eval: epoch 20 | batch 470/510 [I 2022-10-26 17:26:13] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 17:26:13] ...trap/engines/engine.py.326: process: 0.28069 | load: 0.00019 [I 2022-10-26 17:26:15] ...trap/engines/engine.py.321: eval: epoch 20 | batch 480/510 [I 2022-10-26 17:26:15] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 17:26:15] ...trap/engines/engine.py.326: process: 0.28729 | load: 0.00023 [I 2022-10-26 17:26:18] ...trap/engines/engine.py.321: eval: epoch 20 | batch 490/510 [I 2022-10-26 17:26:18] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 17:26:18] ...trap/engines/engine.py.326: process: 0.28837 | load: 0.00018 [I 2022-10-26 17:26:21] ...trap/engines/engine.py.321: eval: epoch 20 | batch 500/510 [I 2022-10-26 17:26:21] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 17:26:21] ...trap/engines/engine.py.326: process: 0.28717 | load: 0.00018 [I 2022-10-26 17:26:24] ...trap/engines/engine.py.321: eval: epoch 20 | batch 510/510 [I 2022-10-26 17:26:24] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 17:26:24] ...trap/engines/engine.py.326: process: 0.28509 | load: 0.00017 [S 2022-10-26 17:26:25] ...trap/engines/engine.py.132: eval_epoch.epoch: 20 [I 2022-10-26 17:26:25] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.603400 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.847600 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.905200 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.591700 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.843300 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.900600 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016572 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012068 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007960 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013799 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.013454 [S 2022-10-26 17:26:29] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007889 [I 2022-10-26 17:26:30] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 17:26:30] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 17:26:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 17:26:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 17:26:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 17:26:31] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 17:26:33] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 17:26:40] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 17:26:40] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 17:26:40] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 17:26:43] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 17:26:49] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 17:26:49] ...trap/engines/engine.py.163: Training model on trainset for epoch 21 [I 2022-10-26 17:26:49] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 17:26:52] ...trap/engines/engine.py.224: train: epoch 21 | batch 0/2304 [I 2022-10-26 17:26:52] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:50:13 [I 2022-10-26 17:26:52] ...trap/engines/engine.py.229: process: 2.87034 | load: 1.99275 [I 2022-10-26 17:26:52] ...trap/engines/engine.py.230: loss: 0.22541 [I 2022-10-26 17:26:52] ...optimizers/trijoint.py.151: total_norm: 21.874960 [I 2022-10-26 17:26:57] ...trap/engines/engine.py.224: train: epoch 21 | batch 10/2304 [I 2022-10-26 17:26:57] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:40 [I 2022-10-26 17:26:57] ...trap/engines/engine.py.229: process: 0.54992 | load: 0.00427 [I 2022-10-26 17:26:57] ...trap/engines/engine.py.230: loss: 0.20749 [I 2022-10-26 17:26:57] ...optimizers/trijoint.py.151: total_norm: 26.448986 [I 2022-10-26 17:27:03] ...trap/engines/engine.py.224: train: epoch 21 | batch 20/2304 [I 2022-10-26 17:27:03] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:10 [I 2022-10-26 17:27:03] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00402 [I 2022-10-26 17:27:03] ...trap/engines/engine.py.230: loss: 0.14122 [I 2022-10-26 17:27:03] ...optimizers/trijoint.py.151: total_norm: 19.762249 [I 2022-10-26 17:27:08] ...trap/engines/engine.py.224: train: epoch 21 | batch 30/2304 [I 2022-10-26 17:27:08] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:03 [I 2022-10-26 17:27:08] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00343 [I 2022-10-26 17:27:08] ...trap/engines/engine.py.230: loss: 0.14390 [I 2022-10-26 17:27:08] ...optimizers/trijoint.py.151: total_norm: 18.789949 [I 2022-10-26 17:27:14] ...trap/engines/engine.py.224: train: epoch 21 | batch 40/2304 [I 2022-10-26 17:27:14] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:05 [I 2022-10-26 17:27:14] ...trap/engines/engine.py.229: process: 0.56982 | load: 0.00260 [I 2022-10-26 17:27:14] ...trap/engines/engine.py.230: loss: 0.23593 [I 2022-10-26 17:27:14] ...optimizers/trijoint.py.151: total_norm: 36.124546 [I 2022-10-26 17:27:20] ...trap/engines/engine.py.224: train: epoch 21 | batch 50/2304 [I 2022-10-26 17:27:20] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:55 [I 2022-10-26 17:27:20] ...trap/engines/engine.py.229: process: 0.56714 | load: 0.00287 [I 2022-10-26 17:27:20] ...trap/engines/engine.py.230: loss: 0.18411 [I 2022-10-26 17:27:20] ...optimizers/trijoint.py.151: total_norm: 21.469799 [I 2022-10-26 17:27:25] ...trap/engines/engine.py.224: train: epoch 21 | batch 60/2304 [I 2022-10-26 17:27:25] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:43 [I 2022-10-26 17:27:25] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00281 [I 2022-10-26 17:27:25] ...trap/engines/engine.py.230: loss: 0.17713 [I 2022-10-26 17:27:25] ...optimizers/trijoint.py.151: total_norm: 25.247597 [I 2022-10-26 17:27:31] ...trap/engines/engine.py.224: train: epoch 21 | batch 70/2304 [I 2022-10-26 17:27:31] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:30 [I 2022-10-26 17:27:31] ...trap/engines/engine.py.229: process: 0.55552 | load: 0.00312 [I 2022-10-26 17:27:31] ...trap/engines/engine.py.230: loss: 0.22767 [I 2022-10-26 17:27:31] ...optimizers/trijoint.py.151: total_norm: 23.810854 [I 2022-10-26 17:27:36] ...trap/engines/engine.py.224: train: epoch 21 | batch 80/2304 [I 2022-10-26 17:27:36] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:48 [I 2022-10-26 17:27:36] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00269 [I 2022-10-26 17:27:36] ...trap/engines/engine.py.230: loss: 0.16197 [I 2022-10-26 17:27:36] ...optimizers/trijoint.py.151: total_norm: 21.320005 [I 2022-10-26 17:27:42] ...trap/engines/engine.py.224: train: epoch 21 | batch 90/2304 [I 2022-10-26 17:27:42] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:52 [I 2022-10-26 17:27:42] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00263 [I 2022-10-26 17:27:42] ...trap/engines/engine.py.230: loss: 0.23048 [I 2022-10-26 17:27:42] ...optimizers/trijoint.py.151: total_norm: 34.630829 [I 2022-10-26 17:27:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 100/2304 [I 2022-10-26 17:27:47] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:25 [I 2022-10-26 17:27:47] ...trap/engines/engine.py.229: process: 0.55656 | load: 0.00269 [I 2022-10-26 17:27:47] ...trap/engines/engine.py.230: loss: 0.17863 [I 2022-10-26 17:27:47] ...optimizers/trijoint.py.151: total_norm: 29.542362 [I 2022-10-26 17:27:53] ...trap/engines/engine.py.224: train: epoch 21 | batch 110/2304 [I 2022-10-26 17:27:53] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:21 [I 2022-10-26 17:27:53] ...trap/engines/engine.py.229: process: 0.54902 | load: 0.00277 [I 2022-10-26 17:27:53] ...trap/engines/engine.py.230: loss: 0.13675 [I 2022-10-26 17:27:53] ...optimizers/trijoint.py.151: total_norm: 16.315786 [I 2022-10-26 17:27:59] ...trap/engines/engine.py.224: train: epoch 21 | batch 120/2304 [I 2022-10-26 17:27:59] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:15 [I 2022-10-26 17:27:59] ...trap/engines/engine.py.229: process: 0.55013 | load: 0.00270 [I 2022-10-26 17:27:59] ...trap/engines/engine.py.230: loss: 0.19085 [I 2022-10-26 17:27:59] ...optimizers/trijoint.py.151: total_norm: 17.378563 [I 2022-10-26 17:28:04] ...trap/engines/engine.py.224: train: epoch 21 | batch 130/2304 [I 2022-10-26 17:28:04] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:12 [I 2022-10-26 17:28:04] ...trap/engines/engine.py.229: process: 0.54877 | load: 0.00259 [I 2022-10-26 17:28:04] ...trap/engines/engine.py.230: loss: 0.15864 [I 2022-10-26 17:28:04] ...optimizers/trijoint.py.151: total_norm: 26.612549 [I 2022-10-26 17:28:10] ...trap/engines/engine.py.224: train: epoch 21 | batch 140/2304 [I 2022-10-26 17:28:10] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:35 [I 2022-10-26 17:28:10] ...trap/engines/engine.py.229: process: 0.59071 | load: 0.00275 [I 2022-10-26 17:28:10] ...trap/engines/engine.py.230: loss: 0.27043 [I 2022-10-26 17:28:10] ...optimizers/trijoint.py.151: total_norm: 32.970020 [I 2022-10-26 17:28:15] ...trap/engines/engine.py.224: train: epoch 21 | batch 150/2304 [I 2022-10-26 17:28:15] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:04 [I 2022-10-26 17:28:15] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00263 [I 2022-10-26 17:28:15] ...trap/engines/engine.py.230: loss: 0.12096 [I 2022-10-26 17:28:15] ...optimizers/trijoint.py.151: total_norm: 16.995386 [I 2022-10-26 17:28:21] ...trap/engines/engine.py.224: train: epoch 21 | batch 160/2304 [I 2022-10-26 17:28:21] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:02 [I 2022-10-26 17:28:21] ...trap/engines/engine.py.229: process: 0.53969 | load: 0.00245 [I 2022-10-26 17:28:21] ...trap/engines/engine.py.230: loss: 0.31418 [I 2022-10-26 17:28:21] ...optimizers/trijoint.py.151: total_norm: 32.146370 [I 2022-10-26 17:28:27] ...trap/engines/engine.py.224: train: epoch 21 | batch 170/2304 [I 2022-10-26 17:28:27] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:50 [I 2022-10-26 17:28:27] ...trap/engines/engine.py.229: process: 0.56156 | load: 0.00260 [I 2022-10-26 17:28:27] ...trap/engines/engine.py.230: loss: 0.17252 [I 2022-10-26 17:28:27] ...optimizers/trijoint.py.151: total_norm: 24.141253 [I 2022-10-26 17:28:32] ...trap/engines/engine.py.224: train: epoch 21 | batch 180/2304 [I 2022-10-26 17:28:32] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:49 [I 2022-10-26 17:28:32] ...trap/engines/engine.py.229: process: 0.55521 | load: 0.00265 [I 2022-10-26 17:28:32] ...trap/engines/engine.py.230: loss: 0.22567 [I 2022-10-26 17:28:32] ...optimizers/trijoint.py.151: total_norm: 23.687702 [I 2022-10-26 17:28:38] ...trap/engines/engine.py.224: train: epoch 21 | batch 190/2304 [I 2022-10-26 17:28:38] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:31 [I 2022-10-26 17:28:38] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00257 [I 2022-10-26 17:28:38] ...trap/engines/engine.py.230: loss: 0.22634 [I 2022-10-26 17:28:38] ...optimizers/trijoint.py.151: total_norm: 23.086384 [I 2022-10-26 17:28:43] ...trap/engines/engine.py.224: train: epoch 21 | batch 200/2304 [I 2022-10-26 17:28:43] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:29 [I 2022-10-26 17:28:43] ...trap/engines/engine.py.229: process: 0.54866 | load: 0.00244 [I 2022-10-26 17:28:43] ...trap/engines/engine.py.230: loss: 0.18121 [I 2022-10-26 17:28:43] ...optimizers/trijoint.py.151: total_norm: 28.883097 [I 2022-10-26 17:28:49] ...trap/engines/engine.py.224: train: epoch 21 | batch 210/2304 [I 2022-10-26 17:28:49] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:37 [I 2022-10-26 17:28:49] ...trap/engines/engine.py.229: process: 0.55411 | load: 0.00257 [I 2022-10-26 17:28:49] ...trap/engines/engine.py.230: loss: 0.19999 [I 2022-10-26 17:28:49] ...optimizers/trijoint.py.151: total_norm: 22.439465 [I 2022-10-26 17:28:55] ...trap/engines/engine.py.224: train: epoch 21 | batch 220/2304 [I 2022-10-26 17:28:55] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:27 [I 2022-10-26 17:28:55] ...trap/engines/engine.py.229: process: 0.55724 | load: 0.00244 [I 2022-10-26 17:28:55] ...trap/engines/engine.py.230: loss: 0.15685 [I 2022-10-26 17:28:55] ...optimizers/trijoint.py.151: total_norm: 27.621338 [I 2022-10-26 17:29:00] ...trap/engines/engine.py.224: train: epoch 21 | batch 230/2304 [I 2022-10-26 17:29:00] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:16 [I 2022-10-26 17:29:00] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00249 [I 2022-10-26 17:29:00] ...trap/engines/engine.py.230: loss: 0.25874 [I 2022-10-26 17:29:00] ...optimizers/trijoint.py.151: total_norm: 32.991051 [I 2022-10-26 17:29:06] ...trap/engines/engine.py.224: train: epoch 21 | batch 240/2304 [I 2022-10-26 17:29:06] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:13 [I 2022-10-26 17:29:06] ...trap/engines/engine.py.229: process: 0.54623 | load: 0.00269 [I 2022-10-26 17:29:06] ...trap/engines/engine.py.230: loss: 0.18099 [I 2022-10-26 17:29:06] ...optimizers/trijoint.py.151: total_norm: 20.729641 [I 2022-10-26 17:29:12] ...trap/engines/engine.py.224: train: epoch 21 | batch 250/2304 [I 2022-10-26 17:29:12] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:03 [I 2022-10-26 17:29:12] ...trap/engines/engine.py.229: process: 0.56193 | load: 0.00245 [I 2022-10-26 17:29:12] ...trap/engines/engine.py.230: loss: 0.14005 [I 2022-10-26 17:29:12] ...optimizers/trijoint.py.151: total_norm: 18.745399 [I 2022-10-26 17:29:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 260/2304 [I 2022-10-26 17:29:17] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:00 [I 2022-10-26 17:29:17] ...trap/engines/engine.py.229: process: 0.55153 | load: 0.00245 [I 2022-10-26 17:29:17] ...trap/engines/engine.py.230: loss: 0.22540 [I 2022-10-26 17:29:17] ...optimizers/trijoint.py.151: total_norm: 22.237173 [I 2022-10-26 17:29:23] ...trap/engines/engine.py.224: train: epoch 21 | batch 270/2304 [I 2022-10-26 17:29:23] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:05 [I 2022-10-26 17:29:23] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00253 [I 2022-10-26 17:29:23] ...trap/engines/engine.py.230: loss: 0.18921 [I 2022-10-26 17:29:23] ...optimizers/trijoint.py.151: total_norm: 21.549171 [I 2022-10-26 17:29:28] ...trap/engines/engine.py.224: train: epoch 21 | batch 280/2304 [I 2022-10-26 17:29:28] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:53 [I 2022-10-26 17:29:28] ...trap/engines/engine.py.229: process: 0.54931 | load: 0.00249 [I 2022-10-26 17:29:28] ...trap/engines/engine.py.230: loss: 0.16399 [I 2022-10-26 17:29:28] ...optimizers/trijoint.py.151: total_norm: 21.884771 [I 2022-10-26 17:29:34] ...trap/engines/engine.py.224: train: epoch 21 | batch 290/2304 [I 2022-10-26 17:29:34] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:39 [I 2022-10-26 17:29:34] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00254 [I 2022-10-26 17:29:34] ...trap/engines/engine.py.230: loss: 0.22631 [I 2022-10-26 17:29:34] ...optimizers/trijoint.py.151: total_norm: 21.737110 [I 2022-10-26 17:29:40] ...trap/engines/engine.py.224: train: epoch 21 | batch 300/2304 [I 2022-10-26 17:29:40] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:34 [I 2022-10-26 17:29:40] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00245 [I 2022-10-26 17:29:40] ...trap/engines/engine.py.230: loss: 0.18461 [I 2022-10-26 17:29:40] ...optimizers/trijoint.py.151: total_norm: 22.459801 [I 2022-10-26 17:29:45] ...trap/engines/engine.py.224: train: epoch 21 | batch 310/2304 [I 2022-10-26 17:29:45] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:29 [I 2022-10-26 17:29:45] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00253 [I 2022-10-26 17:29:45] ...trap/engines/engine.py.230: loss: 0.18109 [I 2022-10-26 17:29:45] ...optimizers/trijoint.py.151: total_norm: 16.836010 [I 2022-10-26 17:29:51] ...trap/engines/engine.py.224: train: epoch 21 | batch 320/2304 [I 2022-10-26 17:29:51] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:20 [I 2022-10-26 17:29:51] ...trap/engines/engine.py.229: process: 0.54365 | load: 0.00251 [I 2022-10-26 17:29:51] ...trap/engines/engine.py.230: loss: 0.14043 [I 2022-10-26 17:29:51] ...optimizers/trijoint.py.151: total_norm: 18.436655 [I 2022-10-26 17:29:56] ...trap/engines/engine.py.224: train: epoch 21 | batch 330/2304 [I 2022-10-26 17:29:56] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:16 [I 2022-10-26 17:29:56] ...trap/engines/engine.py.229: process: 0.54853 | load: 0.00245 [I 2022-10-26 17:29:56] ...trap/engines/engine.py.230: loss: 0.15681 [I 2022-10-26 17:29:56] ...optimizers/trijoint.py.151: total_norm: 16.327471 [I 2022-10-26 17:30:02] ...trap/engines/engine.py.224: train: epoch 21 | batch 340/2304 [I 2022-10-26 17:30:02] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:34 [I 2022-10-26 17:30:02] ...trap/engines/engine.py.229: process: 0.56433 | load: 0.00246 [I 2022-10-26 17:30:02] ...trap/engines/engine.py.230: loss: 0.25990 [I 2022-10-26 17:30:02] ...optimizers/trijoint.py.151: total_norm: 31.506212 [I 2022-10-26 17:30:08] ...trap/engines/engine.py.224: train: epoch 21 | batch 350/2304 [I 2022-10-26 17:30:08] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:06 [I 2022-10-26 17:30:08] ...trap/engines/engine.py.229: process: 0.55227 | load: 0.00245 [I 2022-10-26 17:30:08] ...trap/engines/engine.py.230: loss: 0.32999 [I 2022-10-26 17:30:08] ...optimizers/trijoint.py.151: total_norm: 39.135254 [I 2022-10-26 17:30:13] ...trap/engines/engine.py.224: train: epoch 21 | batch 360/2304 [I 2022-10-26 17:30:13] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:03 [I 2022-10-26 17:30:13] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00255 [I 2022-10-26 17:30:13] ...trap/engines/engine.py.230: loss: 0.21573 [I 2022-10-26 17:30:13] ...optimizers/trijoint.py.151: total_norm: 22.131987 [I 2022-10-26 17:30:19] ...trap/engines/engine.py.224: train: epoch 21 | batch 370/2304 [I 2022-10-26 17:30:19] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:57 [I 2022-10-26 17:30:19] ...trap/engines/engine.py.229: process: 0.54560 | load: 0.00258 [I 2022-10-26 17:30:19] ...trap/engines/engine.py.230: loss: 0.16957 [I 2022-10-26 17:30:19] ...optimizers/trijoint.py.151: total_norm: 26.944988 [I 2022-10-26 17:30:24] ...trap/engines/engine.py.224: train: epoch 21 | batch 380/2304 [I 2022-10-26 17:30:24] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:56 [I 2022-10-26 17:30:24] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00250 [I 2022-10-26 17:30:24] ...trap/engines/engine.py.230: loss: 0.23636 [I 2022-10-26 17:30:24] ...optimizers/trijoint.py.151: total_norm: 32.262207 [I 2022-10-26 17:30:30] ...trap/engines/engine.py.224: train: epoch 21 | batch 390/2304 [I 2022-10-26 17:30:30] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:42 [I 2022-10-26 17:30:30] ...trap/engines/engine.py.229: process: 0.55000 | load: 0.00244 [I 2022-10-26 17:30:30] ...trap/engines/engine.py.230: loss: 0.17660 [I 2022-10-26 17:30:30] ...optimizers/trijoint.py.151: total_norm: 15.348095 [I 2022-10-26 17:30:35] ...trap/engines/engine.py.224: train: epoch 21 | batch 400/2304 [I 2022-10-26 17:30:35] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:37 [I 2022-10-26 17:30:35] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00244 [I 2022-10-26 17:30:35] ...trap/engines/engine.py.230: loss: 0.16866 [I 2022-10-26 17:30:35] ...optimizers/trijoint.py.151: total_norm: 29.212301 [I 2022-10-26 17:30:41] ...trap/engines/engine.py.224: train: epoch 21 | batch 410/2304 [I 2022-10-26 17:30:41] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:31 [I 2022-10-26 17:30:41] ...trap/engines/engine.py.229: process: 0.54377 | load: 0.00242 [I 2022-10-26 17:30:41] ...trap/engines/engine.py.230: loss: 0.13174 [I 2022-10-26 17:30:41] ...optimizers/trijoint.py.151: total_norm: 43.789043 [I 2022-10-26 17:30:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 420/2304 [I 2022-10-26 17:30:47] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:28 [I 2022-10-26 17:30:47] ...trap/engines/engine.py.229: process: 0.55059 | load: 0.00248 [I 2022-10-26 17:30:47] ...trap/engines/engine.py.230: loss: 0.18487 [I 2022-10-26 17:30:47] ...optimizers/trijoint.py.151: total_norm: 24.272728 [I 2022-10-26 17:30:52] ...trap/engines/engine.py.224: train: epoch 21 | batch 430/2304 [I 2022-10-26 17:30:52] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:28 [I 2022-10-26 17:30:52] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00245 [I 2022-10-26 17:30:52] ...trap/engines/engine.py.230: loss: 0.15040 [I 2022-10-26 17:30:52] ...optimizers/trijoint.py.151: total_norm: 15.305525 [I 2022-10-26 17:30:58] ...trap/engines/engine.py.224: train: epoch 21 | batch 440/2304 [I 2022-10-26 17:30:58] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:14 [I 2022-10-26 17:30:58] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00252 [I 2022-10-26 17:30:58] ...trap/engines/engine.py.230: loss: 0.28893 [I 2022-10-26 17:30:58] ...optimizers/trijoint.py.151: total_norm: 23.746695 [I 2022-10-26 17:31:03] ...trap/engines/engine.py.224: train: epoch 21 | batch 450/2304 [I 2022-10-26 17:31:03] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:09 [I 2022-10-26 17:31:03] ...trap/engines/engine.py.229: process: 0.55212 | load: 0.00250 [I 2022-10-26 17:31:03] ...trap/engines/engine.py.230: loss: 0.16509 [I 2022-10-26 17:31:03] ...optimizers/trijoint.py.151: total_norm: 21.362446 [I 2022-10-26 17:31:09] ...trap/engines/engine.py.224: train: epoch 21 | batch 460/2304 [I 2022-10-26 17:31:09] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:03 [I 2022-10-26 17:31:09] ...trap/engines/engine.py.229: process: 0.55080 | load: 0.00255 [I 2022-10-26 17:31:09] ...trap/engines/engine.py.230: loss: 0.16133 [I 2022-10-26 17:31:09] ...optimizers/trijoint.py.151: total_norm: 12.679651 [I 2022-10-26 17:31:14] ...trap/engines/engine.py.224: train: epoch 21 | batch 470/2304 [I 2022-10-26 17:31:14] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:53 [I 2022-10-26 17:31:14] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00242 [I 2022-10-26 17:31:14] ...trap/engines/engine.py.230: loss: 0.13969 [I 2022-10-26 17:31:14] ...optimizers/trijoint.py.151: total_norm: 34.501534 [I 2022-10-26 17:31:20] ...trap/engines/engine.py.224: train: epoch 21 | batch 480/2304 [I 2022-10-26 17:31:20] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:50 [I 2022-10-26 17:31:20] ...trap/engines/engine.py.229: process: 0.54720 | load: 0.00247 [I 2022-10-26 17:31:20] ...trap/engines/engine.py.230: loss: 0.17864 [I 2022-10-26 17:31:20] ...optimizers/trijoint.py.151: total_norm: 18.494127 [I 2022-10-26 17:31:25] ...trap/engines/engine.py.224: train: epoch 21 | batch 490/2304 [I 2022-10-26 17:31:25] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:43 [I 2022-10-26 17:31:25] ...trap/engines/engine.py.229: process: 0.55033 | load: 0.00242 [I 2022-10-26 17:31:25] ...trap/engines/engine.py.230: loss: 0.19461 [I 2022-10-26 17:31:25] ...optimizers/trijoint.py.151: total_norm: 37.049904 [I 2022-10-26 17:31:31] ...trap/engines/engine.py.224: train: epoch 21 | batch 500/2304 [I 2022-10-26 17:31:31] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:39 [I 2022-10-26 17:31:31] ...trap/engines/engine.py.229: process: 0.55027 | load: 0.00240 [I 2022-10-26 17:31:31] ...trap/engines/engine.py.230: loss: 0.13832 [I 2022-10-26 17:31:31] ...optimizers/trijoint.py.151: total_norm: 14.526060 [I 2022-10-26 17:31:37] ...trap/engines/engine.py.224: train: epoch 21 | batch 510/2304 [I 2022-10-26 17:31:37] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:34 [I 2022-10-26 17:31:37] ...trap/engines/engine.py.229: process: 0.55197 | load: 0.00242 [I 2022-10-26 17:31:37] ...trap/engines/engine.py.230: loss: 0.22702 [I 2022-10-26 17:31:37] ...optimizers/trijoint.py.151: total_norm: 18.650053 [I 2022-10-26 17:31:42] ...trap/engines/engine.py.224: train: epoch 21 | batch 520/2304 [I 2022-10-26 17:31:42] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:29 [I 2022-10-26 17:31:42] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00253 [I 2022-10-26 17:31:42] ...trap/engines/engine.py.230: loss: 0.13077 [I 2022-10-26 17:31:42] ...optimizers/trijoint.py.151: total_norm: 17.216434 [I 2022-10-26 17:31:48] ...trap/engines/engine.py.224: train: epoch 21 | batch 530/2304 [I 2022-10-26 17:31:48] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:21 [I 2022-10-26 17:31:48] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00262 [I 2022-10-26 17:31:48] ...trap/engines/engine.py.230: loss: 0.25798 [I 2022-10-26 17:31:48] ...optimizers/trijoint.py.151: total_norm: 27.813982 [I 2022-10-26 17:31:53] ...trap/engines/engine.py.224: train: epoch 21 | batch 540/2304 [I 2022-10-26 17:31:53] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:16 [I 2022-10-26 17:31:53] ...trap/engines/engine.py.229: process: 0.55638 | load: 0.00239 [I 2022-10-26 17:31:53] ...trap/engines/engine.py.230: loss: 0.13817 [I 2022-10-26 17:31:53] ...optimizers/trijoint.py.151: total_norm: 14.604492 [I 2022-10-26 17:31:59] ...trap/engines/engine.py.224: train: epoch 21 | batch 550/2304 [I 2022-10-26 17:31:59] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:10 [I 2022-10-26 17:31:59] ...trap/engines/engine.py.229: process: 0.55501 | load: 0.00240 [I 2022-10-26 17:31:59] ...trap/engines/engine.py.230: loss: 0.18386 [I 2022-10-26 17:31:59] ...optimizers/trijoint.py.151: total_norm: 30.150633 [I 2022-10-26 17:32:04] ...trap/engines/engine.py.224: train: epoch 21 | batch 560/2304 [I 2022-10-26 17:32:04] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:09 [I 2022-10-26 17:32:04] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00250 [I 2022-10-26 17:32:04] ...trap/engines/engine.py.230: loss: 0.21165 [I 2022-10-26 17:32:04] ...optimizers/trijoint.py.151: total_norm: 20.212511 [I 2022-10-26 17:32:10] ...trap/engines/engine.py.224: train: epoch 21 | batch 570/2304 [I 2022-10-26 17:32:10] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:02 [I 2022-10-26 17:32:10] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00245 [I 2022-10-26 17:32:10] ...trap/engines/engine.py.230: loss: 0.24923 [I 2022-10-26 17:32:10] ...optimizers/trijoint.py.151: total_norm: 22.061510 [I 2022-10-26 17:32:15] ...trap/engines/engine.py.224: train: epoch 21 | batch 580/2304 [I 2022-10-26 17:32:15] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:15:57 [I 2022-10-26 17:32:15] ...trap/engines/engine.py.229: process: 0.55693 | load: 0.00251 [I 2022-10-26 17:32:15] ...trap/engines/engine.py.230: loss: 0.25431 [I 2022-10-26 17:32:15] ...optimizers/trijoint.py.151: total_norm: 26.834967 [I 2022-10-26 17:32:21] ...trap/engines/engine.py.224: train: epoch 21 | batch 590/2304 [I 2022-10-26 17:32:21] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:51 [I 2022-10-26 17:32:21] ...trap/engines/engine.py.229: process: 0.56087 | load: 0.00243 [I 2022-10-26 17:32:21] ...trap/engines/engine.py.230: loss: 0.24255 [I 2022-10-26 17:32:21] ...optimizers/trijoint.py.151: total_norm: 18.645910 [I 2022-10-26 17:32:27] ...trap/engines/engine.py.224: train: epoch 21 | batch 600/2304 [I 2022-10-26 17:32:27] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:44 [I 2022-10-26 17:32:27] ...trap/engines/engine.py.229: process: 0.56312 | load: 0.00238 [I 2022-10-26 17:32:27] ...trap/engines/engine.py.230: loss: 0.19250 [I 2022-10-26 17:32:27] ...optimizers/trijoint.py.151: total_norm: 31.247911 [I 2022-10-26 17:32:32] ...trap/engines/engine.py.224: train: epoch 21 | batch 610/2304 [I 2022-10-26 17:32:32] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:48 [I 2022-10-26 17:32:32] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00242 [I 2022-10-26 17:32:32] ...trap/engines/engine.py.230: loss: 0.19684 [I 2022-10-26 17:32:32] ...optimizers/trijoint.py.151: total_norm: 12.402732 [I 2022-10-26 17:32:38] ...trap/engines/engine.py.224: train: epoch 21 | batch 620/2304 [I 2022-10-26 17:32:38] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:30 [I 2022-10-26 17:32:38] ...trap/engines/engine.py.229: process: 0.54455 | load: 0.00246 [I 2022-10-26 17:32:38] ...trap/engines/engine.py.230: loss: 0.30553 [I 2022-10-26 17:32:38] ...optimizers/trijoint.py.151: total_norm: 24.732918 [I 2022-10-26 17:32:43] ...trap/engines/engine.py.224: train: epoch 21 | batch 630/2304 [I 2022-10-26 17:32:43] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:19 [I 2022-10-26 17:32:43] ...trap/engines/engine.py.229: process: 0.54804 | load: 0.00243 [I 2022-10-26 17:32:43] ...trap/engines/engine.py.230: loss: 0.22283 [I 2022-10-26 17:32:43] ...optimizers/trijoint.py.151: total_norm: 38.876472 [I 2022-10-26 17:32:49] ...trap/engines/engine.py.224: train: epoch 21 | batch 640/2304 [I 2022-10-26 17:32:49] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:20 [I 2022-10-26 17:32:49] ...trap/engines/engine.py.229: process: 0.55568 | load: 0.00242 [I 2022-10-26 17:32:49] ...trap/engines/engine.py.230: loss: 0.19541 [I 2022-10-26 17:32:49] ...optimizers/trijoint.py.151: total_norm: 17.324875 [I 2022-10-26 17:32:54] ...trap/engines/engine.py.224: train: epoch 21 | batch 650/2304 [I 2022-10-26 17:32:54] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:24 [I 2022-10-26 17:32:54] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00241 [I 2022-10-26 17:32:54] ...trap/engines/engine.py.230: loss: 0.22742 [I 2022-10-26 17:32:54] ...optimizers/trijoint.py.151: total_norm: 20.267080 [I 2022-10-26 17:33:00] ...trap/engines/engine.py.224: train: epoch 21 | batch 660/2304 [I 2022-10-26 17:33:00] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:07 [I 2022-10-26 17:33:00] ...trap/engines/engine.py.229: process: 0.55024 | load: 0.00243 [I 2022-10-26 17:33:00] ...trap/engines/engine.py.230: loss: 0.20957 [I 2022-10-26 17:33:00] ...optimizers/trijoint.py.151: total_norm: 15.061384 [I 2022-10-26 17:33:06] ...trap/engines/engine.py.224: train: epoch 21 | batch 670/2304 [I 2022-10-26 17:33:06] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:15 [I 2022-10-26 17:33:06] ...trap/engines/engine.py.229: process: 0.56249 | load: 0.00244 [I 2022-10-26 17:33:06] ...trap/engines/engine.py.230: loss: 0.16968 [I 2022-10-26 17:33:06] ...optimizers/trijoint.py.151: total_norm: 29.916780 [I 2022-10-26 17:33:11] ...trap/engines/engine.py.224: train: epoch 21 | batch 680/2304 [I 2022-10-26 17:33:11] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:05 [I 2022-10-26 17:33:11] ...trap/engines/engine.py.229: process: 0.56843 | load: 0.00245 [I 2022-10-26 17:33:11] ...trap/engines/engine.py.230: loss: 0.14772 [I 2022-10-26 17:33:11] ...optimizers/trijoint.py.151: total_norm: 21.248775 [I 2022-10-26 17:33:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 690/2304 [I 2022-10-26 17:33:17] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:14:58 [I 2022-10-26 17:33:17] ...trap/engines/engine.py.229: process: 0.56611 | load: 0.00243 [I 2022-10-26 17:33:17] ...trap/engines/engine.py.230: loss: 0.19086 [I 2022-10-26 17:33:17] ...optimizers/trijoint.py.151: total_norm: 24.630804 [I 2022-10-26 17:33:22] ...trap/engines/engine.py.224: train: epoch 21 | batch 700/2304 [I 2022-10-26 17:33:22] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:48 [I 2022-10-26 17:33:22] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00245 [I 2022-10-26 17:33:22] ...trap/engines/engine.py.230: loss: 0.10017 [I 2022-10-26 17:33:22] ...optimizers/trijoint.py.151: total_norm: 12.467751 [I 2022-10-26 17:33:28] ...trap/engines/engine.py.224: train: epoch 21 | batch 710/2304 [I 2022-10-26 17:33:28] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:14:46 [I 2022-10-26 17:33:28] ...trap/engines/engine.py.229: process: 0.55395 | load: 0.00241 [I 2022-10-26 17:33:28] ...trap/engines/engine.py.230: loss: 0.13358 [I 2022-10-26 17:33:28] ...optimizers/trijoint.py.151: total_norm: 23.192875 [I 2022-10-26 17:33:33] ...trap/engines/engine.py.224: train: epoch 21 | batch 720/2304 [I 2022-10-26 17:33:33] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:48 [I 2022-10-26 17:33:33] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00244 [I 2022-10-26 17:33:33] ...trap/engines/engine.py.230: loss: 0.13863 [I 2022-10-26 17:33:33] ...optimizers/trijoint.py.151: total_norm: 30.210682 [I 2022-10-26 17:33:39] ...trap/engines/engine.py.224: train: epoch 21 | batch 730/2304 [I 2022-10-26 17:33:39] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:31 [I 2022-10-26 17:33:39] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00244 [I 2022-10-26 17:33:39] ...trap/engines/engine.py.230: loss: 0.24501 [I 2022-10-26 17:33:39] ...optimizers/trijoint.py.151: total_norm: 18.243942 [I 2022-10-26 17:33:44] ...trap/engines/engine.py.224: train: epoch 21 | batch 740/2304 [I 2022-10-26 17:33:44] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:20 [I 2022-10-26 17:33:44] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00238 [I 2022-10-26 17:33:44] ...trap/engines/engine.py.230: loss: 0.18507 [I 2022-10-26 17:33:44] ...optimizers/trijoint.py.151: total_norm: 19.793365 [I 2022-10-26 17:33:50] ...trap/engines/engine.py.224: train: epoch 21 | batch 750/2304 [I 2022-10-26 17:33:50] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:26 [I 2022-10-26 17:33:50] ...trap/engines/engine.py.229: process: 0.55494 | load: 0.00241 [I 2022-10-26 17:33:50] ...trap/engines/engine.py.230: loss: 0.17166 [I 2022-10-26 17:33:50] ...optimizers/trijoint.py.151: total_norm: 17.515579 [I 2022-10-26 17:33:56] ...trap/engines/engine.py.224: train: epoch 21 | batch 760/2304 [I 2022-10-26 17:33:56] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:09 [I 2022-10-26 17:33:56] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00242 [I 2022-10-26 17:33:56] ...trap/engines/engine.py.230: loss: 0.18757 [I 2022-10-26 17:33:56] ...optimizers/trijoint.py.151: total_norm: 33.778553 [I 2022-10-26 17:34:01] ...trap/engines/engine.py.224: train: epoch 21 | batch 770/2304 [I 2022-10-26 17:34:01] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:19 [I 2022-10-26 17:34:01] ...trap/engines/engine.py.229: process: 0.56802 | load: 0.00243 [I 2022-10-26 17:34:01] ...trap/engines/engine.py.230: loss: 0.23126 [I 2022-10-26 17:34:01] ...optimizers/trijoint.py.151: total_norm: 20.563486 [I 2022-10-26 17:34:07] ...trap/engines/engine.py.224: train: epoch 21 | batch 780/2304 [I 2022-10-26 17:34:07] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:14:06 [I 2022-10-26 17:34:07] ...trap/engines/engine.py.229: process: 0.56220 | load: 0.00241 [I 2022-10-26 17:34:07] ...trap/engines/engine.py.230: loss: 0.11737 [I 2022-10-26 17:34:07] ...optimizers/trijoint.py.151: total_norm: 23.901239 [I 2022-10-26 17:34:12] ...trap/engines/engine.py.224: train: epoch 21 | batch 790/2304 [I 2022-10-26 17:34:12] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:06 [I 2022-10-26 17:34:12] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00241 [I 2022-10-26 17:34:12] ...trap/engines/engine.py.230: loss: 0.26910 [I 2022-10-26 17:34:12] ...optimizers/trijoint.py.151: total_norm: 27.179836 [I 2022-10-26 17:34:18] ...trap/engines/engine.py.224: train: epoch 21 | batch 800/2304 [I 2022-10-26 17:34:18] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:59 [I 2022-10-26 17:34:18] ...trap/engines/engine.py.229: process: 0.55925 | load: 0.00240 [I 2022-10-26 17:34:18] ...trap/engines/engine.py.230: loss: 0.13781 [I 2022-10-26 17:34:18] ...optimizers/trijoint.py.151: total_norm: 18.222521 [I 2022-10-26 17:34:23] ...trap/engines/engine.py.224: train: epoch 21 | batch 810/2304 [I 2022-10-26 17:34:24] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:13:57 [I 2022-10-26 17:34:24] ...trap/engines/engine.py.229: process: 0.58255 | load: 0.00240 [I 2022-10-26 17:34:24] ...trap/engines/engine.py.230: loss: 0.21091 [I 2022-10-26 17:34:24] ...optimizers/trijoint.py.151: total_norm: 46.373875 [I 2022-10-26 17:34:29] ...trap/engines/engine.py.224: train: epoch 21 | batch 820/2304 [I 2022-10-26 17:34:29] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:47 [I 2022-10-26 17:34:29] ...trap/engines/engine.py.229: process: 0.56354 | load: 0.00245 [I 2022-10-26 17:34:29] ...trap/engines/engine.py.230: loss: 0.16893 [I 2022-10-26 17:34:29] ...optimizers/trijoint.py.151: total_norm: 21.583412 [I 2022-10-26 17:34:35] ...trap/engines/engine.py.224: train: epoch 21 | batch 830/2304 [I 2022-10-26 17:34:35] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:13:36 [I 2022-10-26 17:34:35] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00267 [I 2022-10-26 17:34:35] ...trap/engines/engine.py.230: loss: 0.16132 [I 2022-10-26 17:34:35] ...optimizers/trijoint.py.151: total_norm: 18.058004 [I 2022-10-26 17:34:40] ...trap/engines/engine.py.224: train: epoch 21 | batch 840/2304 [I 2022-10-26 17:34:40] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:33 [I 2022-10-26 17:34:40] ...trap/engines/engine.py.229: process: 0.54499 | load: 0.00252 [I 2022-10-26 17:34:40] ...trap/engines/engine.py.230: loss: 0.10270 [I 2022-10-26 17:34:40] ...optimizers/trijoint.py.151: total_norm: 11.481338 [I 2022-10-26 17:34:46] ...trap/engines/engine.py.224: train: epoch 21 | batch 850/2304 [I 2022-10-26 17:34:46] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:26 [I 2022-10-26 17:34:46] ...trap/engines/engine.py.229: process: 0.56522 | load: 0.00252 [I 2022-10-26 17:34:46] ...trap/engines/engine.py.230: loss: 0.20707 [I 2022-10-26 17:34:46] ...optimizers/trijoint.py.151: total_norm: 21.427950 [I 2022-10-26 17:34:51] ...trap/engines/engine.py.224: train: epoch 21 | batch 860/2304 [I 2022-10-26 17:34:51] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:24 [I 2022-10-26 17:34:51] ...trap/engines/engine.py.229: process: 0.54854 | load: 0.00248 [I 2022-10-26 17:34:51] ...trap/engines/engine.py.230: loss: 0.19898 [I 2022-10-26 17:34:51] ...optimizers/trijoint.py.151: total_norm: 21.847908 [I 2022-10-26 17:34:57] ...trap/engines/engine.py.224: train: epoch 21 | batch 870/2304 [I 2022-10-26 17:34:57] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:25 [I 2022-10-26 17:34:57] ...trap/engines/engine.py.229: process: 0.57062 | load: 0.00242 [I 2022-10-26 17:34:57] ...trap/engines/engine.py.230: loss: 0.19461 [I 2022-10-26 17:34:57] ...optimizers/trijoint.py.151: total_norm: 11.935036 [I 2022-10-26 17:35:03] ...trap/engines/engine.py.224: train: epoch 21 | batch 880/2304 [I 2022-10-26 17:35:03] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:14 [I 2022-10-26 17:35:03] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00241 [I 2022-10-26 17:35:03] ...trap/engines/engine.py.230: loss: 0.20833 [I 2022-10-26 17:35:03] ...optimizers/trijoint.py.151: total_norm: 19.992865 [I 2022-10-26 17:35:08] ...trap/engines/engine.py.224: train: epoch 21 | batch 890/2304 [I 2022-10-26 17:35:08] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:05 [I 2022-10-26 17:35:08] ...trap/engines/engine.py.229: process: 0.55897 | load: 0.00242 [I 2022-10-26 17:35:08] ...trap/engines/engine.py.230: loss: 0.15535 [I 2022-10-26 17:35:08] ...optimizers/trijoint.py.151: total_norm: 27.057360 [I 2022-10-26 17:35:14] ...trap/engines/engine.py.224: train: epoch 21 | batch 900/2304 [I 2022-10-26 17:35:14] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:00 [I 2022-10-26 17:35:14] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00243 [I 2022-10-26 17:35:14] ...trap/engines/engine.py.230: loss: 0.23388 [I 2022-10-26 17:35:14] ...optimizers/trijoint.py.151: total_norm: 19.825825 [I 2022-10-26 17:35:19] ...trap/engines/engine.py.224: train: epoch 21 | batch 910/2304 [I 2022-10-26 17:35:19] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:56 [I 2022-10-26 17:35:19] ...trap/engines/engine.py.229: process: 0.55597 | load: 0.00238 [I 2022-10-26 17:35:19] ...trap/engines/engine.py.230: loss: 0.17664 [I 2022-10-26 17:35:19] ...optimizers/trijoint.py.151: total_norm: 14.557508 [I 2022-10-26 17:35:25] ...trap/engines/engine.py.224: train: epoch 21 | batch 920/2304 [I 2022-10-26 17:35:25] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:12:42 [I 2022-10-26 17:35:25] ...trap/engines/engine.py.229: process: 0.54655 | load: 0.00243 [I 2022-10-26 17:35:25] ...trap/engines/engine.py.230: loss: 0.21749 [I 2022-10-26 17:35:25] ...optimizers/trijoint.py.151: total_norm: 27.017565 [I 2022-10-26 17:35:30] ...trap/engines/engine.py.224: train: epoch 21 | batch 930/2304 [I 2022-10-26 17:35:30] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:12:56 [I 2022-10-26 17:35:30] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00243 [I 2022-10-26 17:35:30] ...trap/engines/engine.py.230: loss: 0.19182 [I 2022-10-26 17:35:30] ...optimizers/trijoint.py.151: total_norm: 43.039410 [I 2022-10-26 17:35:36] ...trap/engines/engine.py.224: train: epoch 21 | batch 940/2304 [I 2022-10-26 17:35:36] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:45 [I 2022-10-26 17:35:36] ...trap/engines/engine.py.229: process: 0.57748 | load: 0.00242 [I 2022-10-26 17:35:36] ...trap/engines/engine.py.230: loss: 0.15657 [I 2022-10-26 17:35:36] ...optimizers/trijoint.py.151: total_norm: 37.093967 [I 2022-10-26 17:35:42] ...trap/engines/engine.py.224: train: epoch 21 | batch 950/2304 [I 2022-10-26 17:35:42] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:50 [I 2022-10-26 17:35:42] ...trap/engines/engine.py.229: process: 0.57785 | load: 0.00244 [I 2022-10-26 17:35:42] ...trap/engines/engine.py.230: loss: 0.23534 [I 2022-10-26 17:35:42] ...optimizers/trijoint.py.151: total_norm: 21.079424 [I 2022-10-26 17:35:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 960/2304 [I 2022-10-26 17:35:47] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:32 [I 2022-10-26 17:35:47] ...trap/engines/engine.py.229: process: 0.56160 | load: 0.00247 [I 2022-10-26 17:35:47] ...trap/engines/engine.py.230: loss: 0.18902 [I 2022-10-26 17:35:47] ...optimizers/trijoint.py.151: total_norm: 27.561384 [I 2022-10-26 17:35:53] ...trap/engines/engine.py.224: train: epoch 21 | batch 970/2304 [I 2022-10-26 17:35:53] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:27 [I 2022-10-26 17:35:53] ...trap/engines/engine.py.229: process: 0.56407 | load: 0.00244 [I 2022-10-26 17:35:53] ...trap/engines/engine.py.230: loss: 0.19968 [I 2022-10-26 17:35:53] ...optimizers/trijoint.py.151: total_norm: 20.528391 [I 2022-10-26 17:35:59] ...trap/engines/engine.py.224: train: epoch 21 | batch 980/2304 [I 2022-10-26 17:35:59] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:25 [I 2022-10-26 17:35:59] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00243 [I 2022-10-26 17:35:59] ...trap/engines/engine.py.230: loss: 0.19151 [I 2022-10-26 17:35:59] ...optimizers/trijoint.py.151: total_norm: 18.940165 [I 2022-10-26 17:36:04] ...trap/engines/engine.py.224: train: epoch 21 | batch 990/2304 [I 2022-10-26 17:36:04] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:15 [I 2022-10-26 17:36:04] ...trap/engines/engine.py.229: process: 0.55431 | load: 0.00240 [I 2022-10-26 17:36:04] ...trap/engines/engine.py.230: loss: 0.24735 [I 2022-10-26 17:36:04] ...optimizers/trijoint.py.151: total_norm: 33.341965 [I 2022-10-26 17:36:10] ...trap/engines/engine.py.224: train: epoch 21 | batch 1000/2304 [I 2022-10-26 17:36:10] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:06 [I 2022-10-26 17:36:10] ...trap/engines/engine.py.229: process: 0.56316 | load: 0.00246 [I 2022-10-26 17:36:10] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-26 17:36:10] ...optimizers/trijoint.py.151: total_norm: 15.447271 [I 2022-10-26 17:36:15] ...trap/engines/engine.py.224: train: epoch 21 | batch 1010/2304 [I 2022-10-26 17:36:15] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:11:52 [I 2022-10-26 17:36:15] ...trap/engines/engine.py.229: process: 0.54545 | load: 0.00246 [I 2022-10-26 17:36:15] ...trap/engines/engine.py.230: loss: 0.26799 [I 2022-10-26 17:36:15] ...optimizers/trijoint.py.151: total_norm: 19.964716 [I 2022-10-26 17:36:21] ...trap/engines/engine.py.224: train: epoch 21 | batch 1020/2304 [I 2022-10-26 17:36:21] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:12:00 [I 2022-10-26 17:36:21] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00246 [I 2022-10-26 17:36:21] ...trap/engines/engine.py.230: loss: 0.23361 [I 2022-10-26 17:36:21] ...optimizers/trijoint.py.151: total_norm: 24.182858 [I 2022-10-26 17:36:26] ...trap/engines/engine.py.224: train: epoch 21 | batch 1030/2304 [I 2022-10-26 17:36:26] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:56 [I 2022-10-26 17:36:26] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00241 [I 2022-10-26 17:36:26] ...trap/engines/engine.py.230: loss: 0.19247 [I 2022-10-26 17:36:26] ...optimizers/trijoint.py.151: total_norm: 27.543734 [I 2022-10-26 17:36:32] ...trap/engines/engine.py.224: train: epoch 21 | batch 1040/2304 [I 2022-10-26 17:36:32] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:53 [I 2022-10-26 17:36:32] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00249 [I 2022-10-26 17:36:32] ...trap/engines/engine.py.230: loss: 0.27257 [I 2022-10-26 17:36:32] ...optimizers/trijoint.py.151: total_norm: 24.099354 [I 2022-10-26 17:36:38] ...trap/engines/engine.py.224: train: epoch 21 | batch 1050/2304 [I 2022-10-26 17:36:38] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:44 [I 2022-10-26 17:36:38] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00249 [I 2022-10-26 17:36:38] ...trap/engines/engine.py.230: loss: 0.24984 [I 2022-10-26 17:36:38] ...optimizers/trijoint.py.151: total_norm: 22.978867 [I 2022-10-26 17:36:43] ...trap/engines/engine.py.224: train: epoch 21 | batch 1060/2304 [I 2022-10-26 17:36:43] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:35 [I 2022-10-26 17:36:43] ...trap/engines/engine.py.229: process: 0.54562 | load: 0.00243 [I 2022-10-26 17:36:43] ...trap/engines/engine.py.230: loss: 0.21626 [I 2022-10-26 17:36:43] ...optimizers/trijoint.py.151: total_norm: 28.864552 [I 2022-10-26 17:36:49] ...trap/engines/engine.py.224: train: epoch 21 | batch 1070/2304 [I 2022-10-26 17:36:49] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:31 [I 2022-10-26 17:36:49] ...trap/engines/engine.py.229: process: 0.56284 | load: 0.00244 [I 2022-10-26 17:36:49] ...trap/engines/engine.py.230: loss: 0.15543 [I 2022-10-26 17:36:49] ...optimizers/trijoint.py.151: total_norm: 9.901376 [I 2022-10-26 17:36:55] ...trap/engines/engine.py.224: train: epoch 21 | batch 1080/2304 [I 2022-10-26 17:36:55] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:48 [I 2022-10-26 17:36:55] ...trap/engines/engine.py.229: process: 0.66618 | load: 0.00244 [I 2022-10-26 17:36:55] ...trap/engines/engine.py.230: loss: 0.21934 [I 2022-10-26 17:36:55] ...optimizers/trijoint.py.151: total_norm: 20.834044 [I 2022-10-26 17:37:00] ...trap/engines/engine.py.224: train: epoch 21 | batch 1090/2304 [I 2022-10-26 17:37:00] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:23 [I 2022-10-26 17:37:00] ...trap/engines/engine.py.229: process: 0.54625 | load: 0.00242 [I 2022-10-26 17:37:00] ...trap/engines/engine.py.230: loss: 0.29610 [I 2022-10-26 17:37:00] ...optimizers/trijoint.py.151: total_norm: 38.579937 [I 2022-10-26 17:37:06] ...trap/engines/engine.py.224: train: epoch 21 | batch 1100/2304 [I 2022-10-26 17:37:06] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:13 [I 2022-10-26 17:37:06] ...trap/engines/engine.py.229: process: 0.57065 | load: 0.00247 [I 2022-10-26 17:37:06] ...trap/engines/engine.py.230: loss: 0.25431 [I 2022-10-26 17:37:06] ...optimizers/trijoint.py.151: total_norm: 20.621517 [I 2022-10-26 17:37:12] ...trap/engines/engine.py.224: train: epoch 21 | batch 1110/2304 [I 2022-10-26 17:37:12] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:08 [I 2022-10-26 17:37:12] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00249 [I 2022-10-26 17:37:12] ...trap/engines/engine.py.230: loss: 0.18428 [I 2022-10-26 17:37:12] ...optimizers/trijoint.py.151: total_norm: 20.612120 [I 2022-10-26 17:37:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 1120/2304 [I 2022-10-26 17:37:17] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:10:59 [I 2022-10-26 17:37:17] ...trap/engines/engine.py.229: process: 0.55006 | load: 0.00245 [I 2022-10-26 17:37:17] ...trap/engines/engine.py.230: loss: 0.25353 [I 2022-10-26 17:37:17] ...optimizers/trijoint.py.151: total_norm: 29.137629 [I 2022-10-26 17:37:23] ...trap/engines/engine.py.224: train: epoch 21 | batch 1130/2304 [I 2022-10-26 17:37:23] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:53 [I 2022-10-26 17:37:23] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00240 [I 2022-10-26 17:37:23] ...trap/engines/engine.py.230: loss: 0.17810 [I 2022-10-26 17:37:23] ...optimizers/trijoint.py.151: total_norm: 21.242956 [I 2022-10-26 17:37:28] ...trap/engines/engine.py.224: train: epoch 21 | batch 1140/2304 [I 2022-10-26 17:37:28] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:48 [I 2022-10-26 17:37:28] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00242 [I 2022-10-26 17:37:28] ...trap/engines/engine.py.230: loss: 0.22681 [I 2022-10-26 17:37:28] ...optimizers/trijoint.py.151: total_norm: 23.290071 [I 2022-10-26 17:37:34] ...trap/engines/engine.py.224: train: epoch 21 | batch 1150/2304 [I 2022-10-26 17:37:34] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:48 [I 2022-10-26 17:37:34] ...trap/engines/engine.py.229: process: 0.54772 | load: 0.00272 [I 2022-10-26 17:37:34] ...trap/engines/engine.py.230: loss: 0.19502 [I 2022-10-26 17:37:34] ...optimizers/trijoint.py.151: total_norm: 13.232736 [I 2022-10-26 17:37:40] ...trap/engines/engine.py.224: train: epoch 21 | batch 1160/2304 [I 2022-10-26 17:37:40] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:39 [I 2022-10-26 17:37:40] ...trap/engines/engine.py.229: process: 0.55727 | load: 0.00245 [I 2022-10-26 17:37:40] ...trap/engines/engine.py.230: loss: 0.18825 [I 2022-10-26 17:37:40] ...optimizers/trijoint.py.151: total_norm: 18.897211 [I 2022-10-26 17:37:45] ...trap/engines/engine.py.224: train: epoch 21 | batch 1170/2304 [I 2022-10-26 17:37:45] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:33 [I 2022-10-26 17:37:45] ...trap/engines/engine.py.229: process: 0.54875 | load: 0.00245 [I 2022-10-26 17:37:45] ...trap/engines/engine.py.230: loss: 0.22307 [I 2022-10-26 17:37:45] ...optimizers/trijoint.py.151: total_norm: 21.400164 [I 2022-10-26 17:37:51] ...trap/engines/engine.py.224: train: epoch 21 | batch 1180/2304 [I 2022-10-26 17:37:51] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:30 [I 2022-10-26 17:37:51] ...trap/engines/engine.py.229: process: 0.56384 | load: 0.00252 [I 2022-10-26 17:37:51] ...trap/engines/engine.py.230: loss: 0.20134 [I 2022-10-26 17:37:51] ...optimizers/trijoint.py.151: total_norm: 18.768736 [I 2022-10-26 17:37:56] ...trap/engines/engine.py.224: train: epoch 21 | batch 1190/2304 [I 2022-10-26 17:37:56] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:25 [I 2022-10-26 17:37:56] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00262 [I 2022-10-26 17:37:56] ...trap/engines/engine.py.230: loss: 0.16703 [I 2022-10-26 17:37:56] ...optimizers/trijoint.py.151: total_norm: 15.815821 [I 2022-10-26 17:38:02] ...trap/engines/engine.py.224: train: epoch 21 | batch 1200/2304 [I 2022-10-26 17:38:02] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:17 [I 2022-10-26 17:38:02] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00283 [I 2022-10-26 17:38:02] ...trap/engines/engine.py.230: loss: 0.16762 [I 2022-10-26 17:38:02] ...optimizers/trijoint.py.151: total_norm: 20.282928 [I 2022-10-26 17:38:08] ...trap/engines/engine.py.224: train: epoch 21 | batch 1210/2304 [I 2022-10-26 17:38:08] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:13 [I 2022-10-26 17:38:08] ...trap/engines/engine.py.229: process: 0.54758 | load: 0.00286 [I 2022-10-26 17:38:08] ...trap/engines/engine.py.230: loss: 0.19411 [I 2022-10-26 17:38:08] ...optimizers/trijoint.py.151: total_norm: 15.768199 [I 2022-10-26 17:38:13] ...trap/engines/engine.py.224: train: epoch 21 | batch 1220/2304 [I 2022-10-26 17:38:13] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:09 [I 2022-10-26 17:38:13] ...trap/engines/engine.py.229: process: 0.56915 | load: 0.00513 [I 2022-10-26 17:38:13] ...trap/engines/engine.py.230: loss: 0.17447 [I 2022-10-26 17:38:13] ...optimizers/trijoint.py.151: total_norm: 23.497272 [I 2022-10-26 17:38:19] ...trap/engines/engine.py.224: train: epoch 21 | batch 1230/2304 [I 2022-10-26 17:38:19] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:57 [I 2022-10-26 17:38:19] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00251 [I 2022-10-26 17:38:19] ...trap/engines/engine.py.230: loss: 0.27042 [I 2022-10-26 17:38:19] ...optimizers/trijoint.py.151: total_norm: 40.964729 [I 2022-10-26 17:38:25] ...trap/engines/engine.py.224: train: epoch 21 | batch 1240/2304 [I 2022-10-26 17:38:25] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:54 [I 2022-10-26 17:38:25] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00254 [I 2022-10-26 17:38:25] ...trap/engines/engine.py.230: loss: 0.23302 [I 2022-10-26 17:38:25] ...optimizers/trijoint.py.151: total_norm: 32.976631 [I 2022-10-26 17:38:30] ...trap/engines/engine.py.224: train: epoch 21 | batch 1250/2304 [I 2022-10-26 17:38:30] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:50 [I 2022-10-26 17:38:30] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00250 [I 2022-10-26 17:38:30] ...trap/engines/engine.py.230: loss: 0.16051 [I 2022-10-26 17:38:30] ...optimizers/trijoint.py.151: total_norm: 25.083296 [I 2022-10-26 17:38:36] ...trap/engines/engine.py.224: train: epoch 21 | batch 1260/2304 [I 2022-10-26 17:38:36] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:47 [I 2022-10-26 17:38:36] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00244 [I 2022-10-26 17:38:36] ...trap/engines/engine.py.230: loss: 0.22369 [I 2022-10-26 17:38:36] ...optimizers/trijoint.py.151: total_norm: 25.544409 [I 2022-10-26 17:38:41] ...trap/engines/engine.py.224: train: epoch 21 | batch 1270/2304 [I 2022-10-26 17:38:41] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:53 [I 2022-10-26 17:38:41] ...trap/engines/engine.py.229: process: 0.63902 | load: 0.00289 [I 2022-10-26 17:38:41] ...trap/engines/engine.py.230: loss: 0.14210 [I 2022-10-26 17:38:41] ...optimizers/trijoint.py.151: total_norm: 14.605657 [I 2022-10-26 17:38:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 1280/2304 [I 2022-10-26 17:38:47] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:40 [I 2022-10-26 17:38:47] ...trap/engines/engine.py.229: process: 0.56934 | load: 0.00249 [I 2022-10-26 17:38:47] ...trap/engines/engine.py.230: loss: 0.18363 [I 2022-10-26 17:38:47] ...optimizers/trijoint.py.151: total_norm: 27.088894 [I 2022-10-26 17:38:53] ...trap/engines/engine.py.224: train: epoch 21 | batch 1290/2304 [I 2022-10-26 17:38:53] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:31 [I 2022-10-26 17:38:53] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00252 [I 2022-10-26 17:38:53] ...trap/engines/engine.py.230: loss: 0.22425 [I 2022-10-26 17:38:53] ...optimizers/trijoint.py.151: total_norm: 18.628521 [I 2022-10-26 17:38:58] ...trap/engines/engine.py.224: train: epoch 21 | batch 1300/2304 [I 2022-10-26 17:38:58] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:25 [I 2022-10-26 17:38:58] ...trap/engines/engine.py.229: process: 0.55587 | load: 0.00250 [I 2022-10-26 17:38:58] ...trap/engines/engine.py.230: loss: 0.18162 [I 2022-10-26 17:38:58] ...optimizers/trijoint.py.151: total_norm: 13.547138 [I 2022-10-26 17:39:04] ...trap/engines/engine.py.224: train: epoch 21 | batch 1310/2304 [I 2022-10-26 17:39:04] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:19 [I 2022-10-26 17:39:04] ...trap/engines/engine.py.229: process: 0.56724 | load: 0.00242 [I 2022-10-26 17:39:04] ...trap/engines/engine.py.230: loss: 0.15263 [I 2022-10-26 17:39:04] ...optimizers/trijoint.py.151: total_norm: 22.995081 [I 2022-10-26 17:39:10] ...trap/engines/engine.py.224: train: epoch 21 | batch 1320/2304 [I 2022-10-26 17:39:10] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:06 [I 2022-10-26 17:39:10] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00262 [I 2022-10-26 17:39:10] ...trap/engines/engine.py.230: loss: 0.17520 [I 2022-10-26 17:39:10] ...optimizers/trijoint.py.151: total_norm: 18.574837 [I 2022-10-26 17:39:15] ...trap/engines/engine.py.224: train: epoch 21 | batch 1330/2304 [I 2022-10-26 17:39:15] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:08:57 [I 2022-10-26 17:39:15] ...trap/engines/engine.py.229: process: 0.55071 | load: 0.00244 [I 2022-10-26 17:39:15] ...trap/engines/engine.py.230: loss: 0.15350 [I 2022-10-26 17:39:15] ...optimizers/trijoint.py.151: total_norm: 32.294281 [I 2022-10-26 17:39:21] ...trap/engines/engine.py.224: train: epoch 21 | batch 1340/2304 [I 2022-10-26 17:39:21] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:09:04 [I 2022-10-26 17:39:21] ...trap/engines/engine.py.229: process: 0.54876 | load: 0.00253 [I 2022-10-26 17:39:21] ...trap/engines/engine.py.230: loss: 0.19561 [I 2022-10-26 17:39:21] ...optimizers/trijoint.py.151: total_norm: 20.385666 [I 2022-10-26 17:39:26] ...trap/engines/engine.py.224: train: epoch 21 | batch 1350/2304 [I 2022-10-26 17:39:26] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:54 [I 2022-10-26 17:39:26] ...trap/engines/engine.py.229: process: 0.55575 | load: 0.00246 [I 2022-10-26 17:39:26] ...trap/engines/engine.py.230: loss: 0.14983 [I 2022-10-26 17:39:26] ...optimizers/trijoint.py.151: total_norm: 20.248999 [I 2022-10-26 17:39:32] ...trap/engines/engine.py.224: train: epoch 21 | batch 1360/2304 [I 2022-10-26 17:39:32] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:52 [I 2022-10-26 17:39:32] ...trap/engines/engine.py.229: process: 0.59093 | load: 0.00245 [I 2022-10-26 17:39:32] ...trap/engines/engine.py.230: loss: 0.15694 [I 2022-10-26 17:39:32] ...optimizers/trijoint.py.151: total_norm: 29.847113 [I 2022-10-26 17:39:38] ...trap/engines/engine.py.224: train: epoch 21 | batch 1370/2304 [I 2022-10-26 17:39:38] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:38 [I 2022-10-26 17:39:38] ...trap/engines/engine.py.229: process: 0.55005 | load: 0.00241 [I 2022-10-26 17:39:38] ...trap/engines/engine.py.230: loss: 0.20650 [I 2022-10-26 17:39:38] ...optimizers/trijoint.py.151: total_norm: 20.020115 [I 2022-10-26 17:39:43] ...trap/engines/engine.py.224: train: epoch 21 | batch 1380/2304 [I 2022-10-26 17:39:43] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:36 [I 2022-10-26 17:39:43] ...trap/engines/engine.py.229: process: 0.57053 | load: 0.00244 [I 2022-10-26 17:39:43] ...trap/engines/engine.py.230: loss: 0.16182 [I 2022-10-26 17:39:43] ...optimizers/trijoint.py.151: total_norm: 19.265583 [I 2022-10-26 17:39:49] ...trap/engines/engine.py.224: train: epoch 21 | batch 1390/2304 [I 2022-10-26 17:39:49] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:24 [I 2022-10-26 17:39:49] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00260 [I 2022-10-26 17:39:49] ...trap/engines/engine.py.230: loss: 0.16954 [I 2022-10-26 17:39:49] ...optimizers/trijoint.py.151: total_norm: 23.973112 [I 2022-10-26 17:39:54] ...trap/engines/engine.py.224: train: epoch 21 | batch 1400/2304 [I 2022-10-26 17:39:54] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:32 [I 2022-10-26 17:39:54] ...trap/engines/engine.py.229: process: 0.56351 | load: 0.00245 [I 2022-10-26 17:39:54] ...trap/engines/engine.py.230: loss: 0.14451 [I 2022-10-26 17:39:54] ...optimizers/trijoint.py.151: total_norm: 16.365326 [I 2022-10-26 17:40:00] ...trap/engines/engine.py.224: train: epoch 21 | batch 1410/2304 [I 2022-10-26 17:40:00] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:21 [I 2022-10-26 17:40:00] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00243 [I 2022-10-26 17:40:00] ...trap/engines/engine.py.230: loss: 0.28192 [I 2022-10-26 17:40:00] ...optimizers/trijoint.py.151: total_norm: 33.862797 [I 2022-10-26 17:40:06] ...trap/engines/engine.py.224: train: epoch 21 | batch 1420/2304 [I 2022-10-26 17:40:06] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:16 [I 2022-10-26 17:40:06] ...trap/engines/engine.py.229: process: 0.55758 | load: 0.00246 [I 2022-10-26 17:40:06] ...trap/engines/engine.py.230: loss: 0.22437 [I 2022-10-26 17:40:06] ...optimizers/trijoint.py.151: total_norm: 17.619349 [I 2022-10-26 17:40:11] ...trap/engines/engine.py.224: train: epoch 21 | batch 1430/2304 [I 2022-10-26 17:40:11] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:07 [I 2022-10-26 17:40:11] ...trap/engines/engine.py.229: process: 0.54551 | load: 0.00243 [I 2022-10-26 17:40:11] ...trap/engines/engine.py.230: loss: 0.24124 [I 2022-10-26 17:40:11] ...optimizers/trijoint.py.151: total_norm: 31.974512 [I 2022-10-26 17:40:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 1440/2304 [I 2022-10-26 17:40:17] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:05 [I 2022-10-26 17:40:17] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00261 [I 2022-10-26 17:40:17] ...trap/engines/engine.py.230: loss: 0.16922 [I 2022-10-26 17:40:17] ...optimizers/trijoint.py.151: total_norm: 15.535353 [I 2022-10-26 17:40:23] ...trap/engines/engine.py.224: train: epoch 21 | batch 1450/2304 [I 2022-10-26 17:40:23] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:57 [I 2022-10-26 17:40:23] ...trap/engines/engine.py.229: process: 0.56292 | load: 0.00251 [I 2022-10-26 17:40:23] ...trap/engines/engine.py.230: loss: 0.15068 [I 2022-10-26 17:40:23] ...optimizers/trijoint.py.151: total_norm: 16.494085 [I 2022-10-26 17:40:28] ...trap/engines/engine.py.224: train: epoch 21 | batch 1460/2304 [I 2022-10-26 17:40:28] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:08:01 [I 2022-10-26 17:40:28] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00246 [I 2022-10-26 17:40:28] ...trap/engines/engine.py.230: loss: 0.24037 [I 2022-10-26 17:40:28] ...optimizers/trijoint.py.151: total_norm: 30.181498 [I 2022-10-26 17:40:34] ...trap/engines/engine.py.224: train: epoch 21 | batch 1470/2304 [I 2022-10-26 17:40:34] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:48 [I 2022-10-26 17:40:34] ...trap/engines/engine.py.229: process: 0.55246 | load: 0.00251 [I 2022-10-26 17:40:34] ...trap/engines/engine.py.230: loss: 0.14857 [I 2022-10-26 17:40:34] ...optimizers/trijoint.py.151: total_norm: 13.920751 [I 2022-10-26 17:41:00] ...trap/engines/engine.py.224: train: epoch 21 | batch 1480/2304 [I 2022-10-26 17:41:00] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 1:04:22 [I 2022-10-26 17:41:00] ...trap/engines/engine.py.229: process: 21.18948 | load: 0.00243 [I 2022-10-26 17:41:00] ...trap/engines/engine.py.230: loss: 0.19241 [I 2022-10-26 17:41:00] ...optimizers/trijoint.py.151: total_norm: 22.340473 [I 2022-10-26 17:41:06] ...trap/engines/engine.py.224: train: epoch 21 | batch 1490/2304 [I 2022-10-26 17:41:06] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:13:37 [I 2022-10-26 17:41:06] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00240 [I 2022-10-26 17:41:06] ...trap/engines/engine.py.230: loss: 0.18721 [I 2022-10-26 17:41:06] ...optimizers/trijoint.py.151: total_norm: 29.346889 [I 2022-10-26 17:41:11] ...trap/engines/engine.py.224: train: epoch 21 | batch 1500/2304 [I 2022-10-26 17:41:11] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:08:06 [I 2022-10-26 17:41:11] ...trap/engines/engine.py.229: process: 0.54969 | load: 0.00237 [I 2022-10-26 17:41:11] ...trap/engines/engine.py.230: loss: 0.20087 [I 2022-10-26 17:41:11] ...optimizers/trijoint.py.151: total_norm: 19.557768 [I 2022-10-26 17:41:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 1510/2304 [I 2022-10-26 17:41:17] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:27 [I 2022-10-26 17:41:17] ...trap/engines/engine.py.229: process: 0.54577 | load: 0.00236 [I 2022-10-26 17:41:17] ...trap/engines/engine.py.230: loss: 0.26832 [I 2022-10-26 17:41:17] ...optimizers/trijoint.py.151: total_norm: 21.520987 [I 2022-10-26 17:41:23] ...trap/engines/engine.py.224: train: epoch 21 | batch 1520/2304 [I 2022-10-26 17:41:23] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:19 [I 2022-10-26 17:41:23] ...trap/engines/engine.py.229: process: 0.55544 | load: 0.00238 [I 2022-10-26 17:41:23] ...trap/engines/engine.py.230: loss: 0.13735 [I 2022-10-26 17:41:23] ...optimizers/trijoint.py.151: total_norm: 19.472849 [I 2022-10-26 17:41:28] ...trap/engines/engine.py.224: train: epoch 21 | batch 1530/2304 [I 2022-10-26 17:41:28] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:07:14 [I 2022-10-26 17:41:28] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00254 [I 2022-10-26 17:41:28] ...trap/engines/engine.py.230: loss: 0.19341 [I 2022-10-26 17:41:28] ...optimizers/trijoint.py.151: total_norm: 66.926674 [I 2022-10-26 17:41:34] ...trap/engines/engine.py.224: train: epoch 21 | batch 1540/2304 [I 2022-10-26 17:41:34] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:07:07 [I 2022-10-26 17:41:34] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00242 [I 2022-10-26 17:41:34] ...trap/engines/engine.py.230: loss: 0.14291 [I 2022-10-26 17:41:34] ...optimizers/trijoint.py.151: total_norm: 15.534886 [I 2022-10-26 17:41:39] ...trap/engines/engine.py.224: train: epoch 21 | batch 1550/2304 [I 2022-10-26 17:41:39] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:57 [I 2022-10-26 17:41:39] ...trap/engines/engine.py.229: process: 0.54411 | load: 0.00242 [I 2022-10-26 17:41:39] ...trap/engines/engine.py.230: loss: 0.11508 [I 2022-10-26 17:41:39] ...optimizers/trijoint.py.151: total_norm: 18.053280 [I 2022-10-26 17:41:45] ...trap/engines/engine.py.224: train: epoch 21 | batch 1560/2304 [I 2022-10-26 17:41:45] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:56 [I 2022-10-26 17:41:45] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00243 [I 2022-10-26 17:41:45] ...trap/engines/engine.py.230: loss: 0.24395 [I 2022-10-26 17:41:45] ...optimizers/trijoint.py.151: total_norm: 28.731064 [I 2022-10-26 17:41:51] ...trap/engines/engine.py.224: train: epoch 21 | batch 1570/2304 [I 2022-10-26 17:41:51] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:51 [I 2022-10-26 17:41:51] ...trap/engines/engine.py.229: process: 0.55044 | load: 0.00239 [I 2022-10-26 17:41:51] ...trap/engines/engine.py.230: loss: 0.16387 [I 2022-10-26 17:41:51] ...optimizers/trijoint.py.151: total_norm: 21.772348 [I 2022-10-26 17:41:56] ...trap/engines/engine.py.224: train: epoch 21 | batch 1580/2304 [I 2022-10-26 17:41:56] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:47 [I 2022-10-26 17:41:56] ...trap/engines/engine.py.229: process: 0.54926 | load: 0.00240 [I 2022-10-26 17:41:56] ...trap/engines/engine.py.230: loss: 0.13055 [I 2022-10-26 17:41:56] ...optimizers/trijoint.py.151: total_norm: 30.198050 [I 2022-10-26 17:42:02] ...trap/engines/engine.py.224: train: epoch 21 | batch 1590/2304 [I 2022-10-26 17:42:02] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:41 [I 2022-10-26 17:42:02] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00239 [I 2022-10-26 17:42:02] ...trap/engines/engine.py.230: loss: 0.16045 [I 2022-10-26 17:42:02] ...optimizers/trijoint.py.151: total_norm: 24.271547 [I 2022-10-26 17:42:08] ...trap/engines/engine.py.224: train: epoch 21 | batch 1600/2304 [I 2022-10-26 17:42:08] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:34 [I 2022-10-26 17:42:08] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00240 [I 2022-10-26 17:42:08] ...trap/engines/engine.py.230: loss: 0.19058 [I 2022-10-26 17:42:08] ...optimizers/trijoint.py.151: total_norm: 23.476292 [I 2022-10-26 17:42:13] ...trap/engines/engine.py.224: train: epoch 21 | batch 1610/2304 [I 2022-10-26 17:42:13] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:28 [I 2022-10-26 17:42:13] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00252 [I 2022-10-26 17:42:13] ...trap/engines/engine.py.230: loss: 0.19711 [I 2022-10-26 17:42:13] ...optimizers/trijoint.py.151: total_norm: 18.334112 [I 2022-10-26 17:42:19] ...trap/engines/engine.py.224: train: epoch 21 | batch 1620/2304 [I 2022-10-26 17:42:19] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:21 [I 2022-10-26 17:42:19] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00249 [I 2022-10-26 17:42:19] ...trap/engines/engine.py.230: loss: 0.19785 [I 2022-10-26 17:42:19] ...optimizers/trijoint.py.151: total_norm: 17.927233 [I 2022-10-26 17:42:24] ...trap/engines/engine.py.224: train: epoch 21 | batch 1630/2304 [I 2022-10-26 17:42:24] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:06:20 [I 2022-10-26 17:42:24] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00248 [I 2022-10-26 17:42:24] ...trap/engines/engine.py.230: loss: 0.17027 [I 2022-10-26 17:42:24] ...optimizers/trijoint.py.151: total_norm: 17.136873 [I 2022-10-26 17:42:30] ...trap/engines/engine.py.224: train: epoch 21 | batch 1640/2304 [I 2022-10-26 17:42:30] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:06:10 [I 2022-10-26 17:42:30] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00249 [I 2022-10-26 17:42:30] ...trap/engines/engine.py.230: loss: 0.24801 [I 2022-10-26 17:42:30] ...optimizers/trijoint.py.151: total_norm: 20.626001 [I 2022-10-26 17:42:36] ...trap/engines/engine.py.224: train: epoch 21 | batch 1650/2304 [I 2022-10-26 17:42:36] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:06:02 [I 2022-10-26 17:42:36] ...trap/engines/engine.py.229: process: 0.54477 | load: 0.00241 [I 2022-10-26 17:42:36] ...trap/engines/engine.py.230: loss: 0.22380 [I 2022-10-26 17:42:36] ...optimizers/trijoint.py.151: total_norm: 20.151964 [I 2022-10-26 17:42:41] ...trap/engines/engine.py.224: train: epoch 21 | batch 1660/2304 [I 2022-10-26 17:42:41] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:06:00 [I 2022-10-26 17:42:41] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00259 [I 2022-10-26 17:42:41] ...trap/engines/engine.py.230: loss: 0.19114 [I 2022-10-26 17:42:41] ...optimizers/trijoint.py.151: total_norm: 23.627663 [I 2022-10-26 17:42:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 1670/2304 [I 2022-10-26 17:42:47] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:55 [I 2022-10-26 17:42:47] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00249 [I 2022-10-26 17:42:47] ...trap/engines/engine.py.230: loss: 0.24354 [I 2022-10-26 17:42:47] ...optimizers/trijoint.py.151: total_norm: 32.874603 [I 2022-10-26 17:42:52] ...trap/engines/engine.py.224: train: epoch 21 | batch 1680/2304 [I 2022-10-26 17:42:52] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:47 [I 2022-10-26 17:42:52] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00245 [I 2022-10-26 17:42:52] ...trap/engines/engine.py.230: loss: 0.22409 [I 2022-10-26 17:42:52] ...optimizers/trijoint.py.151: total_norm: 18.294525 [I 2022-10-26 17:42:58] ...trap/engines/engine.py.224: train: epoch 21 | batch 1690/2304 [I 2022-10-26 17:42:58] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:42 [I 2022-10-26 17:42:58] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00245 [I 2022-10-26 17:42:58] ...trap/engines/engine.py.230: loss: 0.18965 [I 2022-10-26 17:42:58] ...optimizers/trijoint.py.151: total_norm: 23.646322 [I 2022-10-26 17:43:04] ...trap/engines/engine.py.224: train: epoch 21 | batch 1700/2304 [I 2022-10-26 17:43:04] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:37 [I 2022-10-26 17:43:04] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00243 [I 2022-10-26 17:43:04] ...trap/engines/engine.py.230: loss: 0.16760 [I 2022-10-26 17:43:04] ...optimizers/trijoint.py.151: total_norm: 11.600327 [I 2022-10-26 17:43:09] ...trap/engines/engine.py.224: train: epoch 21 | batch 1710/2304 [I 2022-10-26 17:43:09] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:32 [I 2022-10-26 17:43:09] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00255 [I 2022-10-26 17:43:09] ...trap/engines/engine.py.230: loss: 0.21546 [I 2022-10-26 17:43:09] ...optimizers/trijoint.py.151: total_norm: 42.172012 [I 2022-10-26 17:43:15] ...trap/engines/engine.py.224: train: epoch 21 | batch 1720/2304 [I 2022-10-26 17:43:15] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:23 [I 2022-10-26 17:43:15] ...trap/engines/engine.py.229: process: 0.54536 | load: 0.00245 [I 2022-10-26 17:43:15] ...trap/engines/engine.py.230: loss: 0.39011 [I 2022-10-26 17:43:15] ...optimizers/trijoint.py.151: total_norm: 25.613102 [I 2022-10-26 17:43:20] ...trap/engines/engine.py.224: train: epoch 21 | batch 1730/2304 [I 2022-10-26 17:43:20] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:05:21 [I 2022-10-26 17:43:20] ...trap/engines/engine.py.229: process: 0.56611 | load: 0.00248 [I 2022-10-26 17:43:20] ...trap/engines/engine.py.230: loss: 0.15474 [I 2022-10-26 17:43:20] ...optimizers/trijoint.py.151: total_norm: 35.897804 [I 2022-10-26 17:43:26] ...trap/engines/engine.py.224: train: epoch 21 | batch 1740/2304 [I 2022-10-26 17:43:26] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:05:16 [I 2022-10-26 17:43:26] ...trap/engines/engine.py.229: process: 0.56664 | load: 0.00247 [I 2022-10-26 17:43:26] ...trap/engines/engine.py.230: loss: 0.15590 [I 2022-10-26 17:43:26] ...optimizers/trijoint.py.151: total_norm: 28.896954 [I 2022-10-26 17:43:32] ...trap/engines/engine.py.224: train: epoch 21 | batch 1750/2304 [I 2022-10-26 17:43:32] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:05:09 [I 2022-10-26 17:43:32] ...trap/engines/engine.py.229: process: 0.54988 | load: 0.00252 [I 2022-10-26 17:43:32] ...trap/engines/engine.py.230: loss: 0.18491 [I 2022-10-26 17:43:32] ...optimizers/trijoint.py.151: total_norm: 20.586098 [I 2022-10-26 17:43:37] ...trap/engines/engine.py.224: train: epoch 21 | batch 1760/2304 [I 2022-10-26 17:43:37] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:05:04 [I 2022-10-26 17:43:37] ...trap/engines/engine.py.229: process: 0.55137 | load: 0.00245 [I 2022-10-26 17:43:37] ...trap/engines/engine.py.230: loss: 0.21624 [I 2022-10-26 17:43:37] ...optimizers/trijoint.py.151: total_norm: 25.799364 [I 2022-10-26 17:43:43] ...trap/engines/engine.py.224: train: epoch 21 | batch 1770/2304 [I 2022-10-26 17:43:43] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:57 [I 2022-10-26 17:43:43] ...trap/engines/engine.py.229: process: 0.56358 | load: 0.00244 [I 2022-10-26 17:43:43] ...trap/engines/engine.py.230: loss: 0.22623 [I 2022-10-26 17:43:43] ...optimizers/trijoint.py.151: total_norm: 26.503927 [I 2022-10-26 17:43:48] ...trap/engines/engine.py.224: train: epoch 21 | batch 1780/2304 [I 2022-10-26 17:43:48] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:49 [I 2022-10-26 17:43:48] ...trap/engines/engine.py.229: process: 0.54955 | load: 0.00244 [I 2022-10-26 17:43:48] ...trap/engines/engine.py.230: loss: 0.20900 [I 2022-10-26 17:43:48] ...optimizers/trijoint.py.151: total_norm: 17.837641 [I 2022-10-26 17:43:54] ...trap/engines/engine.py.224: train: epoch 21 | batch 1790/2304 [I 2022-10-26 17:43:54] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:45 [I 2022-10-26 17:43:54] ...trap/engines/engine.py.229: process: 0.56308 | load: 0.00259 [I 2022-10-26 17:43:54] ...trap/engines/engine.py.230: loss: 0.20987 [I 2022-10-26 17:43:54] ...optimizers/trijoint.py.151: total_norm: 13.960217 [I 2022-10-26 17:43:59] ...trap/engines/engine.py.224: train: epoch 21 | batch 1800/2304 [I 2022-10-26 17:43:59] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:38 [I 2022-10-26 17:43:59] ...trap/engines/engine.py.229: process: 0.55272 | load: 0.00251 [I 2022-10-26 17:43:59] ...trap/engines/engine.py.230: loss: 0.19461 [I 2022-10-26 17:43:59] ...optimizers/trijoint.py.151: total_norm: 26.120831 [I 2022-10-26 17:44:05] ...trap/engines/engine.py.224: train: epoch 21 | batch 1810/2304 [I 2022-10-26 17:44:05] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:36 [I 2022-10-26 17:44:05] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00247 [I 2022-10-26 17:44:05] ...trap/engines/engine.py.230: loss: 0.18944 [I 2022-10-26 17:44:05] ...optimizers/trijoint.py.151: total_norm: 23.218182 [I 2022-10-26 17:44:11] ...trap/engines/engine.py.224: train: epoch 21 | batch 1820/2304 [I 2022-10-26 17:44:11] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:28 [I 2022-10-26 17:44:11] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00245 [I 2022-10-26 17:44:11] ...trap/engines/engine.py.230: loss: 0.19707 [I 2022-10-26 17:44:11] ...optimizers/trijoint.py.151: total_norm: 28.527998 [I 2022-10-26 17:44:16] ...trap/engines/engine.py.224: train: epoch 21 | batch 1830/2304 [I 2022-10-26 17:44:16] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:22 [I 2022-10-26 17:44:16] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00242 [I 2022-10-26 17:44:16] ...trap/engines/engine.py.230: loss: 0.27488 [I 2022-10-26 17:44:16] ...optimizers/trijoint.py.151: total_norm: 26.619759 [I 2022-10-26 17:44:22] ...trap/engines/engine.py.224: train: epoch 21 | batch 1840/2304 [I 2022-10-26 17:44:22] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:17 [I 2022-10-26 17:44:22] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00248 [I 2022-10-26 17:44:22] ...trap/engines/engine.py.230: loss: 0.26201 [I 2022-10-26 17:44:22] ...optimizers/trijoint.py.151: total_norm: 37.360909 [I 2022-10-26 17:44:27] ...trap/engines/engine.py.224: train: epoch 21 | batch 1850/2304 [I 2022-10-26 17:44:27] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:04:10 [I 2022-10-26 17:44:27] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00246 [I 2022-10-26 17:44:27] ...trap/engines/engine.py.230: loss: 0.26441 [I 2022-10-26 17:44:27] ...optimizers/trijoint.py.151: total_norm: 22.447035 [I 2022-10-26 17:44:33] ...trap/engines/engine.py.224: train: epoch 21 | batch 1860/2304 [I 2022-10-26 17:44:33] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:04:05 [I 2022-10-26 17:44:33] ...trap/engines/engine.py.229: process: 0.54780 | load: 0.00245 [I 2022-10-26 17:44:33] ...trap/engines/engine.py.230: loss: 0.14766 [I 2022-10-26 17:44:33] ...optimizers/trijoint.py.151: total_norm: 11.081491 [I 2022-10-26 17:44:38] ...trap/engines/engine.py.224: train: epoch 21 | batch 1870/2304 [I 2022-10-26 17:44:38] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:04:02 [I 2022-10-26 17:44:38] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00244 [I 2022-10-26 17:44:38] ...trap/engines/engine.py.230: loss: 0.32019 [I 2022-10-26 17:44:38] ...optimizers/trijoint.py.151: total_norm: 40.676891 [I 2022-10-26 17:44:44] ...trap/engines/engine.py.224: train: epoch 21 | batch 1880/2304 [I 2022-10-26 17:44:44] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:55 [I 2022-10-26 17:44:44] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00241 [I 2022-10-26 17:44:44] ...trap/engines/engine.py.230: loss: 0.12458 [I 2022-10-26 17:44:44] ...optimizers/trijoint.py.151: total_norm: 18.275995 [I 2022-10-26 17:44:49] ...trap/engines/engine.py.224: train: epoch 21 | batch 1890/2304 [I 2022-10-26 17:44:49] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:48 [I 2022-10-26 17:44:49] ...trap/engines/engine.py.229: process: 0.54010 | load: 0.00253 [I 2022-10-26 17:44:49] ...trap/engines/engine.py.230: loss: 0.18959 [I 2022-10-26 17:44:49] ...optimizers/trijoint.py.151: total_norm: 19.201229 [I 2022-10-26 17:44:55] ...trap/engines/engine.py.224: train: epoch 21 | batch 1900/2304 [I 2022-10-26 17:44:55] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:44 [I 2022-10-26 17:44:55] ...trap/engines/engine.py.229: process: 0.54774 | load: 0.00246 [I 2022-10-26 17:44:55] ...trap/engines/engine.py.230: loss: 0.19978 [I 2022-10-26 17:44:55] ...optimizers/trijoint.py.151: total_norm: 13.831758 [I 2022-10-26 17:45:01] ...trap/engines/engine.py.224: train: epoch 21 | batch 1910/2304 [I 2022-10-26 17:45:01] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:38 [I 2022-10-26 17:45:01] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00245 [I 2022-10-26 17:45:01] ...trap/engines/engine.py.230: loss: 0.25687 [I 2022-10-26 17:45:01] ...optimizers/trijoint.py.151: total_norm: 31.544558 [I 2022-10-26 17:45:06] ...trap/engines/engine.py.224: train: epoch 21 | batch 1920/2304 [I 2022-10-26 17:45:06] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:34 [I 2022-10-26 17:45:06] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00247 [I 2022-10-26 17:45:06] ...trap/engines/engine.py.230: loss: 0.17582 [I 2022-10-26 17:45:06] ...optimizers/trijoint.py.151: total_norm: 35.612556 [I 2022-10-26 17:45:12] ...trap/engines/engine.py.224: train: epoch 21 | batch 1930/2304 [I 2022-10-26 17:45:12] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:28 [I 2022-10-26 17:45:12] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00244 [I 2022-10-26 17:45:12] ...trap/engines/engine.py.230: loss: 0.25107 [I 2022-10-26 17:45:12] ...optimizers/trijoint.py.151: total_norm: 24.274071 [I 2022-10-26 17:45:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 1940/2304 [I 2022-10-26 17:45:17] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:23 [I 2022-10-26 17:45:17] ...trap/engines/engine.py.229: process: 0.56878 | load: 0.00250 [I 2022-10-26 17:45:17] ...trap/engines/engine.py.230: loss: 0.21460 [I 2022-10-26 17:45:17] ...optimizers/trijoint.py.151: total_norm: 33.956455 [I 2022-10-26 17:45:23] ...trap/engines/engine.py.224: train: epoch 21 | batch 1950/2304 [I 2022-10-26 17:45:23] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:03:17 [I 2022-10-26 17:45:23] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00246 [I 2022-10-26 17:45:23] ...trap/engines/engine.py.230: loss: 0.20865 [I 2022-10-26 17:45:23] ...optimizers/trijoint.py.151: total_norm: 20.306330 [I 2022-10-26 17:45:29] ...trap/engines/engine.py.224: train: epoch 21 | batch 1960/2304 [I 2022-10-26 17:45:29] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:03:11 [I 2022-10-26 17:45:29] ...trap/engines/engine.py.229: process: 0.54893 | load: 0.00244 [I 2022-10-26 17:45:29] ...trap/engines/engine.py.230: loss: 0.18061 [I 2022-10-26 17:45:29] ...optimizers/trijoint.py.151: total_norm: 22.178205 [I 2022-10-26 17:45:34] ...trap/engines/engine.py.224: train: epoch 21 | batch 1970/2304 [I 2022-10-26 17:45:34] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:03:11 [I 2022-10-26 17:45:34] ...trap/engines/engine.py.229: process: 0.64551 | load: 0.00243 [I 2022-10-26 17:45:34] ...trap/engines/engine.py.230: loss: 0.21862 [I 2022-10-26 17:45:34] ...optimizers/trijoint.py.151: total_norm: 21.219717 [I 2022-10-26 17:45:40] ...trap/engines/engine.py.224: train: epoch 21 | batch 1980/2304 [I 2022-10-26 17:45:40] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:03:00 [I 2022-10-26 17:45:40] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00242 [I 2022-10-26 17:45:40] ...trap/engines/engine.py.230: loss: 0.15108 [I 2022-10-26 17:45:40] ...optimizers/trijoint.py.151: total_norm: 23.499556 [I 2022-10-26 17:45:45] ...trap/engines/engine.py.224: train: epoch 21 | batch 1990/2304 [I 2022-10-26 17:45:45] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:55 [I 2022-10-26 17:45:45] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00254 [I 2022-10-26 17:45:45] ...trap/engines/engine.py.230: loss: 0.29898 [I 2022-10-26 17:45:45] ...optimizers/trijoint.py.151: total_norm: 20.965223 [I 2022-10-26 17:45:51] ...trap/engines/engine.py.224: train: epoch 21 | batch 2000/2304 [I 2022-10-26 17:45:51] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:48 [I 2022-10-26 17:45:51] ...trap/engines/engine.py.229: process: 0.54803 | load: 0.00261 [I 2022-10-26 17:45:51] ...trap/engines/engine.py.230: loss: 0.21598 [I 2022-10-26 17:45:51] ...optimizers/trijoint.py.151: total_norm: 33.389080 [I 2022-10-26 17:45:56] ...trap/engines/engine.py.224: train: epoch 21 | batch 2010/2304 [I 2022-10-26 17:45:56] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:43 [I 2022-10-26 17:45:56] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00247 [I 2022-10-26 17:45:56] ...trap/engines/engine.py.230: loss: 0.24551 [I 2022-10-26 17:45:56] ...optimizers/trijoint.py.151: total_norm: 23.341745 [I 2022-10-26 17:46:02] ...trap/engines/engine.py.224: train: epoch 21 | batch 2020/2304 [I 2022-10-26 17:46:02] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:38 [I 2022-10-26 17:46:02] ...trap/engines/engine.py.229: process: 0.55157 | load: 0.00245 [I 2022-10-26 17:46:02] ...trap/engines/engine.py.230: loss: 0.16515 [I 2022-10-26 17:46:02] ...optimizers/trijoint.py.151: total_norm: 21.915462 [I 2022-10-26 17:46:08] ...trap/engines/engine.py.224: train: epoch 21 | batch 2030/2304 [I 2022-10-26 17:46:08] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:32 [I 2022-10-26 17:46:08] ...trap/engines/engine.py.229: process: 0.56636 | load: 0.00245 [I 2022-10-26 17:46:08] ...trap/engines/engine.py.230: loss: 0.19412 [I 2022-10-26 17:46:08] ...optimizers/trijoint.py.151: total_norm: 33.972073 [I 2022-10-26 17:46:13] ...trap/engines/engine.py.224: train: epoch 21 | batch 2040/2304 [I 2022-10-26 17:46:13] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:27 [I 2022-10-26 17:46:13] ...trap/engines/engine.py.229: process: 0.56627 | load: 0.00245 [I 2022-10-26 17:46:13] ...trap/engines/engine.py.230: loss: 0.18525 [I 2022-10-26 17:46:13] ...optimizers/trijoint.py.151: total_norm: 23.081612 [I 2022-10-26 17:46:19] ...trap/engines/engine.py.224: train: epoch 21 | batch 2050/2304 [I 2022-10-26 17:46:19] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:02:21 [I 2022-10-26 17:46:19] ...trap/engines/engine.py.229: process: 0.55049 | load: 0.00258 [I 2022-10-26 17:46:19] ...trap/engines/engine.py.230: loss: 0.23123 [I 2022-10-26 17:46:19] ...optimizers/trijoint.py.151: total_norm: 24.053946 [I 2022-10-26 17:46:24] ...trap/engines/engine.py.224: train: epoch 21 | batch 2060/2304 [I 2022-10-26 17:46:24] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:02:15 [I 2022-10-26 17:46:24] ...trap/engines/engine.py.229: process: 0.54923 | load: 0.00244 [I 2022-10-26 17:46:24] ...trap/engines/engine.py.230: loss: 0.26739 [I 2022-10-26 17:46:24] ...optimizers/trijoint.py.151: total_norm: 21.169964 [I 2022-10-26 17:46:30] ...trap/engines/engine.py.224: train: epoch 21 | batch 2070/2304 [I 2022-10-26 17:46:30] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:02:10 [I 2022-10-26 17:46:30] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00252 [I 2022-10-26 17:46:30] ...trap/engines/engine.py.230: loss: 0.17919 [I 2022-10-26 17:46:30] ...optimizers/trijoint.py.151: total_norm: 26.909592 [I 2022-10-26 17:46:36] ...trap/engines/engine.py.224: train: epoch 21 | batch 2080/2304 [I 2022-10-26 17:46:36] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:02:04 [I 2022-10-26 17:46:36] ...trap/engines/engine.py.229: process: 0.56137 | load: 0.00245 [I 2022-10-26 17:46:36] ...trap/engines/engine.py.230: loss: 0.37766 [I 2022-10-26 17:46:36] ...optimizers/trijoint.py.151: total_norm: 26.161325 [I 2022-10-26 17:46:41] ...trap/engines/engine.py.224: train: epoch 21 | batch 2090/2304 [I 2022-10-26 17:46:41] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:59 [I 2022-10-26 17:46:41] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00244 [I 2022-10-26 17:46:41] ...trap/engines/engine.py.230: loss: 0.17777 [I 2022-10-26 17:46:41] ...optimizers/trijoint.py.151: total_norm: 11.648269 [I 2022-10-26 17:46:47] ...trap/engines/engine.py.224: train: epoch 21 | batch 2100/2304 [I 2022-10-26 17:46:47] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:53 [I 2022-10-26 17:46:47] ...trap/engines/engine.py.229: process: 0.55532 | load: 0.00241 [I 2022-10-26 17:46:47] ...trap/engines/engine.py.230: loss: 0.21507 [I 2022-10-26 17:46:47] ...optimizers/trijoint.py.151: total_norm: 69.389214 [I 2022-10-26 17:46:52] ...trap/engines/engine.py.224: train: epoch 21 | batch 2110/2304 [I 2022-10-26 17:46:52] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:47 [I 2022-10-26 17:46:52] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00243 [I 2022-10-26 17:46:52] ...trap/engines/engine.py.230: loss: 0.22744 [I 2022-10-26 17:46:52] ...optimizers/trijoint.py.151: total_norm: 42.465267 [I 2022-10-26 17:46:58] ...trap/engines/engine.py.224: train: epoch 21 | batch 2120/2304 [I 2022-10-26 17:46:58] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:43 [I 2022-10-26 17:46:58] ...trap/engines/engine.py.229: process: 0.54606 | load: 0.00241 [I 2022-10-26 17:46:58] ...trap/engines/engine.py.230: loss: 0.14313 [I 2022-10-26 17:46:58] ...optimizers/trijoint.py.151: total_norm: 16.135094 [I 2022-10-26 17:47:04] ...trap/engines/engine.py.224: train: epoch 21 | batch 2130/2304 [I 2022-10-26 17:47:04] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:37 [I 2022-10-26 17:47:04] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00243 [I 2022-10-26 17:47:04] ...trap/engines/engine.py.230: loss: 0.16569 [I 2022-10-26 17:47:04] ...optimizers/trijoint.py.151: total_norm: 16.813753 [I 2022-10-26 17:47:09] ...trap/engines/engine.py.224: train: epoch 21 | batch 2140/2304 [I 2022-10-26 17:47:09] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:31 [I 2022-10-26 17:47:09] ...trap/engines/engine.py.229: process: 0.56893 | load: 0.00254 [I 2022-10-26 17:47:09] ...trap/engines/engine.py.230: loss: 0.17762 [I 2022-10-26 17:47:09] ...optimizers/trijoint.py.151: total_norm: 18.964548 [I 2022-10-26 17:47:15] ...trap/engines/engine.py.224: train: epoch 21 | batch 2150/2304 [I 2022-10-26 17:47:15] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:25 [I 2022-10-26 17:47:15] ...trap/engines/engine.py.229: process: 0.56590 | load: 0.00247 [I 2022-10-26 17:47:15] ...trap/engines/engine.py.230: loss: 0.19466 [I 2022-10-26 17:47:15] ...optimizers/trijoint.py.151: total_norm: 22.967447 [I 2022-10-26 17:47:20] ...trap/engines/engine.py.224: train: epoch 21 | batch 2160/2304 [I 2022-10-26 17:47:20] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:20 [I 2022-10-26 17:47:20] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00247 [I 2022-10-26 17:47:20] ...trap/engines/engine.py.230: loss: 0.20757 [I 2022-10-26 17:47:20] ...optimizers/trijoint.py.151: total_norm: 30.870655 [I 2022-10-26 17:47:26] ...trap/engines/engine.py.224: train: epoch 21 | batch 2170/2304 [I 2022-10-26 17:47:26] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:01:14 [I 2022-10-26 17:47:26] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00244 [I 2022-10-26 17:47:26] ...trap/engines/engine.py.230: loss: 0.18057 [I 2022-10-26 17:47:26] ...optimizers/trijoint.py.151: total_norm: 19.416971 [I 2022-10-26 17:47:32] ...trap/engines/engine.py.224: train: epoch 21 | batch 2180/2304 [I 2022-10-26 17:47:32] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:09 [I 2022-10-26 17:47:32] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00246 [I 2022-10-26 17:47:32] ...trap/engines/engine.py.230: loss: 0.23729 [I 2022-10-26 17:47:32] ...optimizers/trijoint.py.151: total_norm: 33.667557 [I 2022-10-26 17:47:37] ...trap/engines/engine.py.224: train: epoch 21 | batch 2190/2304 [I 2022-10-26 17:47:37] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:01:03 [I 2022-10-26 17:47:37] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00245 [I 2022-10-26 17:47:37] ...trap/engines/engine.py.230: loss: 0.24129 [I 2022-10-26 17:47:37] ...optimizers/trijoint.py.151: total_norm: 40.456028 [I 2022-10-26 17:47:43] ...trap/engines/engine.py.224: train: epoch 21 | batch 2200/2304 [I 2022-10-26 17:47:43] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:58 [I 2022-10-26 17:47:43] ...trap/engines/engine.py.229: process: 0.57070 | load: 0.00255 [I 2022-10-26 17:47:43] ...trap/engines/engine.py.230: loss: 0.17364 [I 2022-10-26 17:47:43] ...optimizers/trijoint.py.151: total_norm: 14.687533 [I 2022-10-26 17:47:48] ...trap/engines/engine.py.224: train: epoch 21 | batch 2210/2304 [I 2022-10-26 17:47:48] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:52 [I 2022-10-26 17:47:48] ...trap/engines/engine.py.229: process: 0.56579 | load: 0.00291 [I 2022-10-26 17:47:48] ...trap/engines/engine.py.230: loss: 0.16025 [I 2022-10-26 17:47:48] ...optimizers/trijoint.py.151: total_norm: 13.191855 [I 2022-10-26 17:47:54] ...trap/engines/engine.py.224: train: epoch 21 | batch 2220/2304 [I 2022-10-26 17:47:54] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:48 [I 2022-10-26 17:47:54] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00541 [I 2022-10-26 17:47:54] ...trap/engines/engine.py.230: loss: 0.23408 [I 2022-10-26 17:47:54] ...optimizers/trijoint.py.151: total_norm: 18.755564 [I 2022-10-26 17:48:00] ...trap/engines/engine.py.224: train: epoch 21 | batch 2230/2304 [I 2022-10-26 17:48:00] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:41 [I 2022-10-26 17:48:00] ...trap/engines/engine.py.229: process: 0.54560 | load: 0.00243 [I 2022-10-26 17:48:00] ...trap/engines/engine.py.230: loss: 0.23432 [I 2022-10-26 17:48:00] ...optimizers/trijoint.py.151: total_norm: 23.460155 [I 2022-10-26 17:48:05] ...trap/engines/engine.py.224: train: epoch 21 | batch 2240/2304 [I 2022-10-26 17:48:05] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:36 [I 2022-10-26 17:48:05] ...trap/engines/engine.py.229: process: 0.57122 | load: 0.00239 [I 2022-10-26 17:48:05] ...trap/engines/engine.py.230: loss: 0.17000 [I 2022-10-26 17:48:05] ...optimizers/trijoint.py.151: total_norm: 26.317055 [I 2022-10-26 17:48:11] ...trap/engines/engine.py.224: train: epoch 21 | batch 2250/2304 [I 2022-10-26 17:48:11] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:30 [I 2022-10-26 17:48:11] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00258 [I 2022-10-26 17:48:11] ...trap/engines/engine.py.230: loss: 0.15175 [I 2022-10-26 17:48:11] ...optimizers/trijoint.py.151: total_norm: 23.833632 [I 2022-10-26 17:48:17] ...trap/engines/engine.py.224: train: epoch 21 | batch 2260/2304 [I 2022-10-26 17:48:17] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:24 [I 2022-10-26 17:48:17] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00271 [I 2022-10-26 17:48:17] ...trap/engines/engine.py.230: loss: 0.21029 [I 2022-10-26 17:48:17] ...optimizers/trijoint.py.151: total_norm: 23.313921 [I 2022-10-26 17:48:22] ...trap/engines/engine.py.224: train: epoch 21 | batch 2270/2304 [I 2022-10-26 17:48:22] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:19 [I 2022-10-26 17:48:22] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00248 [I 2022-10-26 17:48:22] ...trap/engines/engine.py.230: loss: 0.19563 [I 2022-10-26 17:48:22] ...optimizers/trijoint.py.151: total_norm: 29.279713 [I 2022-10-26 17:48:28] ...trap/engines/engine.py.224: train: epoch 21 | batch 2280/2304 [I 2022-10-26 17:48:28] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:13 [I 2022-10-26 17:48:28] ...trap/engines/engine.py.229: process: 0.56480 | load: 0.00248 [I 2022-10-26 17:48:28] ...trap/engines/engine.py.230: loss: 0.19730 [I 2022-10-26 17:48:28] ...optimizers/trijoint.py.151: total_norm: 26.539446 [I 2022-10-26 17:48:33] ...trap/engines/engine.py.224: train: epoch 21 | batch 2290/2304 [I 2022-10-26 17:48:33] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:00:07 [I 2022-10-26 17:48:33] ...trap/engines/engine.py.229: process: 0.57418 | load: 0.00248 [I 2022-10-26 17:48:33] ...trap/engines/engine.py.230: loss: 0.21906 [I 2022-10-26 17:48:33] ...optimizers/trijoint.py.151: total_norm: 19.345480 [I 2022-10-26 17:48:39] ...trap/engines/engine.py.224: train: epoch 21 | batch 2300/2304 [I 2022-10-26 17:48:39] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:02 [I 2022-10-26 17:48:39] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00015 [I 2022-10-26 17:48:39] ...trap/engines/engine.py.230: loss: 0.20744 [I 2022-10-26 17:48:39] ...optimizers/trijoint.py.151: total_norm: 19.565907 [I 2022-10-26 17:48:41] ...trap/engines/engine.py.224: train: epoch 21 | batch 2304/2304 [I 2022-10-26 17:48:41] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:00:00 [I 2022-10-26 17:48:41] ...trap/engines/engine.py.229: process: 0.54233 | load: 0.00018 [I 2022-10-26 17:48:41] ...trap/engines/engine.py.230: loss: 0.16749 [I 2022-10-26 17:48:41] ...optimizers/trijoint.py.151: total_norm: 14.987989 [S 2022-10-26 17:48:42] ...trap/engines/engine.py.129: train_epoch.epoch: 21 [S 2022-10-26 17:48:42] ...trap/engines/engine.py.129: train_epoch.loss: 0.19954594268974668 [I 2022-10-26 17:48:42] ...s/criterions/triplet.py.84: margin = 0.1600000000000001 [I 2022-10-26 17:48:43] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 21 [I 2022-10-26 17:48:43] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 17:48:45] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 17:48:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 17:48:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 17:48:45] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 17:48:46] ...trap/engines/engine.py.321: eval: epoch 21 | batch 0/510 [I 2022-10-26 17:48:46] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:25:23 [I 2022-10-26 17:48:46] ...trap/engines/engine.py.326: process: 2.98689 | load: 2.58213 [I 2022-10-26 17:48:49] ...trap/engines/engine.py.321: eval: epoch 21 | batch 10/510 [I 2022-10-26 17:48:49] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:11 [I 2022-10-26 17:48:49] ...trap/engines/engine.py.326: process: 0.34610 | load: 0.00020 [I 2022-10-26 17:48:53] ...trap/engines/engine.py.321: eval: epoch 21 | batch 20/510 [I 2022-10-26 17:48:53] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:03:00 [I 2022-10-26 17:48:53] ...trap/engines/engine.py.326: process: 0.32090 | load: 0.00024 [I 2022-10-26 17:48:56] ...trap/engines/engine.py.321: eval: epoch 21 | batch 30/510 [I 2022-10-26 17:48:56] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:47 [I 2022-10-26 17:48:56] ...trap/engines/engine.py.326: process: 0.31678 | load: 0.00021 [I 2022-10-26 17:48:59] ...trap/engines/engine.py.321: eval: epoch 21 | batch 40/510 [I 2022-10-26 17:48:59] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:41 [I 2022-10-26 17:48:59] ...trap/engines/engine.py.326: process: 0.33712 | load: 0.00018 [I 2022-10-26 17:49:03] ...trap/engines/engine.py.321: eval: epoch 21 | batch 50/510 [I 2022-10-26 17:49:03] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:35 [I 2022-10-26 17:49:03] ...trap/engines/engine.py.326: process: 0.32962 | load: 0.00018 [I 2022-10-26 17:49:06] ...trap/engines/engine.py.321: eval: epoch 21 | batch 60/510 [I 2022-10-26 17:49:06] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:32 [I 2022-10-26 17:49:06] ...trap/engines/engine.py.326: process: 0.34689 | load: 0.00018 [I 2022-10-26 17:49:10] ...trap/engines/engine.py.321: eval: epoch 21 | batch 70/510 [I 2022-10-26 17:49:10] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-26 17:49:10] ...trap/engines/engine.py.326: process: 0.32390 | load: 0.00024 [I 2022-10-26 17:49:13] ...trap/engines/engine.py.321: eval: epoch 21 | batch 80/510 [I 2022-10-26 17:49:13] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:26 [I 2022-10-26 17:49:13] ...trap/engines/engine.py.326: process: 0.32288 | load: 0.00020 [I 2022-10-26 17:49:16] ...trap/engines/engine.py.321: eval: epoch 21 | batch 90/510 [I 2022-10-26 17:49:16] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:22 [I 2022-10-26 17:49:16] ...trap/engines/engine.py.326: process: 0.34262 | load: 0.00025 [I 2022-10-26 17:49:20] ...trap/engines/engine.py.321: eval: epoch 21 | batch 100/510 [I 2022-10-26 17:49:20] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:14 [I 2022-10-26 17:49:20] ...trap/engines/engine.py.326: process: 0.28377 | load: 0.00021 [I 2022-10-26 17:49:23] ...trap/engines/engine.py.321: eval: epoch 21 | batch 110/510 [I 2022-10-26 17:49:23] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:59 [I 2022-10-26 17:49:23] ...trap/engines/engine.py.326: process: 0.28804 | load: 0.00026 [I 2022-10-26 17:49:26] ...trap/engines/engine.py.321: eval: epoch 21 | batch 120/510 [I 2022-10-26 17:49:26] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 17:49:26] ...trap/engines/engine.py.326: process: 0.28588 | load: 0.00019 [I 2022-10-26 17:49:29] ...trap/engines/engine.py.321: eval: epoch 21 | batch 130/510 [I 2022-10-26 17:49:29] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-26 17:49:29] ...trap/engines/engine.py.326: process: 0.29044 | load: 0.00015 [I 2022-10-26 17:49:32] ...trap/engines/engine.py.321: eval: epoch 21 | batch 140/510 [I 2022-10-26 17:49:32] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-26 17:49:32] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00019 [I 2022-10-26 17:49:34] ...trap/engines/engine.py.321: eval: epoch 21 | batch 150/510 [I 2022-10-26 17:49:34] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 17:49:34] ...trap/engines/engine.py.326: process: 0.29701 | load: 0.00016 [I 2022-10-26 17:49:37] ...trap/engines/engine.py.321: eval: epoch 21 | batch 160/510 [I 2022-10-26 17:49:37] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:42 [I 2022-10-26 17:49:37] ...trap/engines/engine.py.326: process: 0.28940 | load: 0.00025 [I 2022-10-26 17:49:40] ...trap/engines/engine.py.321: eval: epoch 21 | batch 170/510 [I 2022-10-26 17:49:40] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:40 [I 2022-10-26 17:49:40] ...trap/engines/engine.py.326: process: 0.30762 | load: 0.00018 [I 2022-10-26 17:49:43] ...trap/engines/engine.py.321: eval: epoch 21 | batch 180/510 [I 2022-10-26 17:49:43] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:36 [I 2022-10-26 17:49:43] ...trap/engines/engine.py.326: process: 0.28501 | load: 0.00016 [I 2022-10-26 17:49:46] ...trap/engines/engine.py.321: eval: epoch 21 | batch 190/510 [I 2022-10-26 17:49:46] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:33 [I 2022-10-26 17:49:46] ...trap/engines/engine.py.326: process: 0.28674 | load: 0.00023 [I 2022-10-26 17:49:49] ...trap/engines/engine.py.321: eval: epoch 21 | batch 200/510 [I 2022-10-26 17:49:49] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-26 17:49:49] ...trap/engines/engine.py.326: process: 0.28884 | load: 0.00020 [I 2022-10-26 17:49:52] ...trap/engines/engine.py.321: eval: epoch 21 | batch 210/510 [I 2022-10-26 17:49:52] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:27 [I 2022-10-26 17:49:52] ...trap/engines/engine.py.326: process: 0.30544 | load: 0.00022 [I 2022-10-26 17:49:55] ...trap/engines/engine.py.321: eval: epoch 21 | batch 220/510 [I 2022-10-26 17:49:55] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:24 [I 2022-10-26 17:49:55] ...trap/engines/engine.py.326: process: 0.29206 | load: 0.00017 [I 2022-10-26 17:49:58] ...trap/engines/engine.py.321: eval: epoch 21 | batch 230/510 [I 2022-10-26 17:49:58] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:22 [I 2022-10-26 17:49:58] ...trap/engines/engine.py.326: process: 0.31504 | load: 0.00016 [I 2022-10-26 17:50:01] ...trap/engines/engine.py.321: eval: epoch 21 | batch 240/510 [I 2022-10-26 17:50:01] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:19 [I 2022-10-26 17:50:01] ...trap/engines/engine.py.326: process: 0.28993 | load: 0.00025 [I 2022-10-26 17:50:04] ...trap/engines/engine.py.321: eval: epoch 21 | batch 250/510 [I 2022-10-26 17:50:04] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-26 17:50:04] ...trap/engines/engine.py.326: process: 0.27926 | load: 0.00018 [I 2022-10-26 17:50:07] ...trap/engines/engine.py.321: eval: epoch 21 | batch 260/510 [I 2022-10-26 17:50:07] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-26 17:50:07] ...trap/engines/engine.py.326: process: 0.28723 | load: 0.00022 [I 2022-10-26 17:50:10] ...trap/engines/engine.py.321: eval: epoch 21 | batch 270/510 [I 2022-10-26 17:50:10] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-26 17:50:10] ...trap/engines/engine.py.326: process: 0.28632 | load: 0.00018 [I 2022-10-26 17:50:12] ...trap/engines/engine.py.321: eval: epoch 21 | batch 280/510 [I 2022-10-26 17:50:12] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-26 17:50:12] ...trap/engines/engine.py.326: process: 0.29315 | load: 0.00020 [I 2022-10-26 17:50:15] ...trap/engines/engine.py.321: eval: epoch 21 | batch 290/510 [I 2022-10-26 17:50:15] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-26 17:50:15] ...trap/engines/engine.py.326: process: 0.29229 | load: 0.00017 [I 2022-10-26 17:50:18] ...trap/engines/engine.py.321: eval: epoch 21 | batch 300/510 [I 2022-10-26 17:50:18] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:02 [I 2022-10-26 17:50:18] ...trap/engines/engine.py.326: process: 0.30668 | load: 0.00019 [I 2022-10-26 17:50:21] ...trap/engines/engine.py.321: eval: epoch 21 | batch 310/510 [I 2022-10-26 17:50:21] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:58 [I 2022-10-26 17:50:21] ...trap/engines/engine.py.326: process: 0.29796 | load: 0.00018 [I 2022-10-26 17:50:24] ...trap/engines/engine.py.321: eval: epoch 21 | batch 320/510 [I 2022-10-26 17:50:24] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:55 [I 2022-10-26 17:50:24] ...trap/engines/engine.py.326: process: 0.27745 | load: 0.00016 [I 2022-10-26 17:50:27] ...trap/engines/engine.py.321: eval: epoch 21 | batch 330/510 [I 2022-10-26 17:50:27] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-26 17:50:27] ...trap/engines/engine.py.326: process: 0.28628 | load: 0.00018 [I 2022-10-26 17:50:30] ...trap/engines/engine.py.321: eval: epoch 21 | batch 340/510 [I 2022-10-26 17:50:30] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-26 17:50:30] ...trap/engines/engine.py.326: process: 0.28581 | load: 0.00019 [I 2022-10-26 17:50:33] ...trap/engines/engine.py.321: eval: epoch 21 | batch 350/510 [I 2022-10-26 17:50:33] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-26 17:50:33] ...trap/engines/engine.py.326: process: 0.28587 | load: 0.00025 [I 2022-10-26 17:50:36] ...trap/engines/engine.py.321: eval: epoch 21 | batch 360/510 [I 2022-10-26 17:50:36] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 17:50:36] ...trap/engines/engine.py.326: process: 0.28203 | load: 0.00022 [I 2022-10-26 17:50:39] ...trap/engines/engine.py.321: eval: epoch 21 | batch 370/510 [I 2022-10-26 17:50:39] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-26 17:50:39] ...trap/engines/engine.py.326: process: 0.29117 | load: 0.00023 [I 2022-10-26 17:50:41] ...trap/engines/engine.py.321: eval: epoch 21 | batch 380/510 [I 2022-10-26 17:50:41] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-26 17:50:41] ...trap/engines/engine.py.326: process: 0.28880 | load: 0.00024 [I 2022-10-26 17:50:44] ...trap/engines/engine.py.321: eval: epoch 21 | batch 390/510 [I 2022-10-26 17:50:44] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:35 [I 2022-10-26 17:50:44] ...trap/engines/engine.py.326: process: 0.30792 | load: 0.00017 [I 2022-10-26 17:50:47] ...trap/engines/engine.py.321: eval: epoch 21 | batch 400/510 [I 2022-10-26 17:50:47] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:32 [I 2022-10-26 17:50:47] ...trap/engines/engine.py.326: process: 0.30308 | load: 0.00018 [I 2022-10-26 17:50:50] ...trap/engines/engine.py.321: eval: epoch 21 | batch 410/510 [I 2022-10-26 17:50:50] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-26 17:50:50] ...trap/engines/engine.py.326: process: 0.28465 | load: 0.00015 [I 2022-10-26 17:50:53] ...trap/engines/engine.py.321: eval: epoch 21 | batch 420/510 [I 2022-10-26 17:50:53] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-26 17:50:53] ...trap/engines/engine.py.326: process: 0.28257 | load: 0.00015 [I 2022-10-26 17:50:56] ...trap/engines/engine.py.321: eval: epoch 21 | batch 430/510 [I 2022-10-26 17:50:56] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-26 17:50:56] ...trap/engines/engine.py.326: process: 0.28504 | load: 0.00022 [I 2022-10-26 17:50:59] ...trap/engines/engine.py.321: eval: epoch 21 | batch 440/510 [I 2022-10-26 17:50:59] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 17:50:59] ...trap/engines/engine.py.326: process: 0.28663 | load: 0.00029 [I 2022-10-26 17:51:02] ...trap/engines/engine.py.321: eval: epoch 21 | batch 450/510 [I 2022-10-26 17:51:02] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 17:51:02] ...trap/engines/engine.py.326: process: 0.28518 | load: 0.00022 [I 2022-10-26 17:51:05] ...trap/engines/engine.py.321: eval: epoch 21 | batch 460/510 [I 2022-10-26 17:51:05] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-26 17:51:05] ...trap/engines/engine.py.326: process: 0.28498 | load: 0.00019 [I 2022-10-26 17:51:07] ...trap/engines/engine.py.321: eval: epoch 21 | batch 470/510 [I 2022-10-26 17:51:07] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-26 17:51:07] ...trap/engines/engine.py.326: process: 0.28827 | load: 0.00032 [I 2022-10-26 17:51:10] ...trap/engines/engine.py.321: eval: epoch 21 | batch 480/510 [I 2022-10-26 17:51:10] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-26 17:51:10] ...trap/engines/engine.py.326: process: 0.28335 | load: 0.00015 [I 2022-10-26 17:51:13] ...trap/engines/engine.py.321: eval: epoch 21 | batch 490/510 [I 2022-10-26 17:51:13] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-26 17:51:13] ...trap/engines/engine.py.326: process: 0.29080 | load: 0.00023 [I 2022-10-26 17:51:16] ...trap/engines/engine.py.321: eval: epoch 21 | batch 500/510 [I 2022-10-26 17:51:16] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-26 17:51:16] ...trap/engines/engine.py.326: process: 0.28700 | load: 0.00020 [I 2022-10-26 17:51:19] ...trap/engines/engine.py.321: eval: epoch 21 | batch 510/510 [I 2022-10-26 17:51:19] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-26 17:51:19] ...trap/engines/engine.py.326: process: 0.28758 | load: 0.00010 [S 2022-10-26 17:51:20] ...trap/engines/engine.py.132: eval_epoch.epoch: 21 [I 2022-10-26 17:51:20] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.612000 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.855900 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.914100 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.599500 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.852300 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.907800 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014360 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008700 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007176 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012339 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009613 [S 2022-10-26 17:51:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008635 [I 2022-10-26 17:51:25] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 17:51:25] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 17:51:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 17:51:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 17:51:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 17:51:26] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 17:51:29] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 17:51:35] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 17:51:35] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 17:51:35] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 17:51:38] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 17:51:44] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 17:51:44] ...trap/engines/engine.py.163: Training model on trainset for epoch 22 [I 2022-10-26 17:51:44] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 17:51:47] ...trap/engines/engine.py.224: train: epoch 22 | batch 0/2304 [I 2022-10-26 17:51:47] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:50:02 [I 2022-10-26 17:51:47] ...trap/engines/engine.py.229: process: 2.86559 | load: 2.22342 [I 2022-10-26 17:51:47] ...trap/engines/engine.py.230: loss: 0.19079 [I 2022-10-26 17:51:47] ...optimizers/trijoint.py.151: total_norm: 17.653992 [I 2022-10-26 17:51:53] ...trap/engines/engine.py.224: train: epoch 22 | batch 10/2304 [I 2022-10-26 17:51:53] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:31:01 [I 2022-10-26 17:51:53] ...trap/engines/engine.py.229: process: 0.56535 | load: 0.00453 [I 2022-10-26 17:51:53] ...trap/engines/engine.py.230: loss: 0.22212 [I 2022-10-26 17:51:53] ...optimizers/trijoint.py.151: total_norm: 14.732703 [I 2022-10-26 17:51:58] ...trap/engines/engine.py.224: train: epoch 22 | batch 20/2304 [I 2022-10-26 17:51:58] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:16 [I 2022-10-26 17:51:58] ...trap/engines/engine.py.229: process: 0.54472 | load: 0.00421 [I 2022-10-26 17:51:58] ...trap/engines/engine.py.230: loss: 0.14782 [I 2022-10-26 17:51:58] ...optimizers/trijoint.py.151: total_norm: 22.030401 [I 2022-10-26 17:52:04] ...trap/engines/engine.py.224: train: epoch 22 | batch 30/2304 [I 2022-10-26 17:52:04] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:05 [I 2022-10-26 17:52:04] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00397 [I 2022-10-26 17:52:04] ...trap/engines/engine.py.230: loss: 0.17748 [I 2022-10-26 17:52:04] ...optimizers/trijoint.py.151: total_norm: 18.549175 [I 2022-10-26 17:52:09] ...trap/engines/engine.py.224: train: epoch 22 | batch 40/2304 [I 2022-10-26 17:52:09] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:10 [I 2022-10-26 17:52:09] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00315 [I 2022-10-26 17:52:09] ...trap/engines/engine.py.230: loss: 0.17234 [I 2022-10-26 17:52:09] ...optimizers/trijoint.py.151: total_norm: 16.000507 [I 2022-10-26 17:52:15] ...trap/engines/engine.py.224: train: epoch 22 | batch 50/2304 [I 2022-10-26 17:52:15] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:09 [I 2022-10-26 17:52:15] ...trap/engines/engine.py.229: process: 0.57147 | load: 0.00277 [I 2022-10-26 17:52:15] ...trap/engines/engine.py.230: loss: 0.18852 [I 2022-10-26 17:52:15] ...optimizers/trijoint.py.151: total_norm: 21.400148 [I 2022-10-26 17:52:21] ...trap/engines/engine.py.224: train: epoch 22 | batch 60/2304 [I 2022-10-26 17:52:21] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:48 [I 2022-10-26 17:52:21] ...trap/engines/engine.py.229: process: 0.54504 | load: 0.00273 [I 2022-10-26 17:52:21] ...trap/engines/engine.py.230: loss: 0.26876 [I 2022-10-26 17:52:21] ...optimizers/trijoint.py.151: total_norm: 37.502274 [I 2022-10-26 17:52:26] ...trap/engines/engine.py.224: train: epoch 22 | batch 70/2304 [I 2022-10-26 17:52:26] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:44 [I 2022-10-26 17:52:26] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00301 [I 2022-10-26 17:52:26] ...trap/engines/engine.py.230: loss: 0.18214 [I 2022-10-26 17:52:26] ...optimizers/trijoint.py.151: total_norm: 17.878241 [I 2022-10-26 17:52:32] ...trap/engines/engine.py.224: train: epoch 22 | batch 80/2304 [I 2022-10-26 17:52:32] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:38 [I 2022-10-26 17:52:32] ...trap/engines/engine.py.229: process: 0.54819 | load: 0.00264 [I 2022-10-26 17:52:32] ...trap/engines/engine.py.230: loss: 0.22394 [I 2022-10-26 17:52:32] ...optimizers/trijoint.py.151: total_norm: 26.385859 [I 2022-10-26 17:52:37] ...trap/engines/engine.py.224: train: epoch 22 | batch 90/2304 [I 2022-10-26 17:52:37] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:36 [I 2022-10-26 17:52:37] ...trap/engines/engine.py.229: process: 0.57100 | load: 0.00282 [I 2022-10-26 17:52:37] ...trap/engines/engine.py.230: loss: 0.17902 [I 2022-10-26 17:52:37] ...optimizers/trijoint.py.151: total_norm: 18.271845 [I 2022-10-26 17:52:43] ...trap/engines/engine.py.224: train: epoch 22 | batch 100/2304 [I 2022-10-26 17:52:43] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:23 [I 2022-10-26 17:52:43] ...trap/engines/engine.py.229: process: 0.55089 | load: 0.00286 [I 2022-10-26 17:52:43] ...trap/engines/engine.py.230: loss: 0.14733 [I 2022-10-26 17:52:43] ...optimizers/trijoint.py.151: total_norm: 14.042653 [I 2022-10-26 17:52:49] ...trap/engines/engine.py.224: train: epoch 22 | batch 110/2304 [I 2022-10-26 17:52:49] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:23 [I 2022-10-26 17:52:49] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00258 [I 2022-10-26 17:52:49] ...trap/engines/engine.py.230: loss: 0.23544 [I 2022-10-26 17:52:49] ...optimizers/trijoint.py.151: total_norm: 17.113455 [I 2022-10-26 17:52:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 120/2304 [I 2022-10-26 17:52:54] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:25 [I 2022-10-26 17:52:54] ...trap/engines/engine.py.229: process: 0.57752 | load: 0.00265 [I 2022-10-26 17:52:54] ...trap/engines/engine.py.230: loss: 0.18232 [I 2022-10-26 17:52:54] ...optimizers/trijoint.py.151: total_norm: 17.938581 [I 2022-10-26 17:53:00] ...trap/engines/engine.py.224: train: epoch 22 | batch 130/2304 [I 2022-10-26 17:53:00] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:21 [I 2022-10-26 17:53:00] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00257 [I 2022-10-26 17:53:00] ...trap/engines/engine.py.230: loss: 0.16901 [I 2022-10-26 17:53:00] ...optimizers/trijoint.py.151: total_norm: 27.246225 [I 2022-10-26 17:53:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 140/2304 [I 2022-10-26 17:53:05] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:02 [I 2022-10-26 17:53:05] ...trap/engines/engine.py.229: process: 0.55938 | load: 0.00282 [I 2022-10-26 17:53:05] ...trap/engines/engine.py.230: loss: 0.19330 [I 2022-10-26 17:53:05] ...optimizers/trijoint.py.151: total_norm: 29.395449 [I 2022-10-26 17:53:11] ...trap/engines/engine.py.224: train: epoch 22 | batch 150/2304 [I 2022-10-26 17:53:11] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:03 [I 2022-10-26 17:53:11] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00265 [I 2022-10-26 17:53:11] ...trap/engines/engine.py.230: loss: 0.16700 [I 2022-10-26 17:53:11] ...optimizers/trijoint.py.151: total_norm: 21.865427 [I 2022-10-26 17:53:17] ...trap/engines/engine.py.224: train: epoch 22 | batch 160/2304 [I 2022-10-26 17:53:17] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:50 [I 2022-10-26 17:53:17] ...trap/engines/engine.py.229: process: 0.55180 | load: 0.00274 [I 2022-10-26 17:53:17] ...trap/engines/engine.py.230: loss: 0.18309 [I 2022-10-26 17:53:17] ...optimizers/trijoint.py.151: total_norm: 21.174953 [I 2022-10-26 17:53:22] ...trap/engines/engine.py.224: train: epoch 22 | batch 170/2304 [I 2022-10-26 17:53:22] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:48 [I 2022-10-26 17:53:22] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00252 [I 2022-10-26 17:53:22] ...trap/engines/engine.py.230: loss: 0.13655 [I 2022-10-26 17:53:22] ...optimizers/trijoint.py.151: total_norm: 19.247759 [I 2022-10-26 17:53:28] ...trap/engines/engine.py.224: train: epoch 22 | batch 180/2304 [I 2022-10-26 17:53:28] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:43 [I 2022-10-26 17:53:28] ...trap/engines/engine.py.229: process: 0.55183 | load: 0.00276 [I 2022-10-26 17:53:28] ...trap/engines/engine.py.230: loss: 0.23751 [I 2022-10-26 17:53:28] ...optimizers/trijoint.py.151: total_norm: 15.894041 [I 2022-10-26 17:53:33] ...trap/engines/engine.py.224: train: epoch 22 | batch 190/2304 [I 2022-10-26 17:53:33] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:34 [I 2022-10-26 17:53:33] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00262 [I 2022-10-26 17:53:33] ...trap/engines/engine.py.230: loss: 0.18553 [I 2022-10-26 17:53:33] ...optimizers/trijoint.py.151: total_norm: 32.872223 [I 2022-10-26 17:53:39] ...trap/engines/engine.py.224: train: epoch 22 | batch 200/2304 [I 2022-10-26 17:53:39] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:28 [I 2022-10-26 17:53:39] ...trap/engines/engine.py.229: process: 0.54369 | load: 0.00256 [I 2022-10-26 17:53:39] ...trap/engines/engine.py.230: loss: 0.23295 [I 2022-10-26 17:53:39] ...optimizers/trijoint.py.151: total_norm: 20.434319 [I 2022-10-26 17:53:44] ...trap/engines/engine.py.224: train: epoch 22 | batch 210/2304 [I 2022-10-26 17:53:44] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:29 [I 2022-10-26 17:53:44] ...trap/engines/engine.py.229: process: 0.55347 | load: 0.00264 [I 2022-10-26 17:53:44] ...trap/engines/engine.py.230: loss: 0.18128 [I 2022-10-26 17:53:44] ...optimizers/trijoint.py.151: total_norm: 49.623955 [I 2022-10-26 17:53:50] ...trap/engines/engine.py.224: train: epoch 22 | batch 220/2304 [I 2022-10-26 17:53:50] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:12 [I 2022-10-26 17:53:50] ...trap/engines/engine.py.229: process: 0.55214 | load: 0.00253 [I 2022-10-26 17:53:50] ...trap/engines/engine.py.230: loss: 0.20300 [I 2022-10-26 17:53:50] ...optimizers/trijoint.py.151: total_norm: 33.073803 [I 2022-10-26 17:53:55] ...trap/engines/engine.py.224: train: epoch 22 | batch 230/2304 [I 2022-10-26 17:53:55] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:08 [I 2022-10-26 17:53:55] ...trap/engines/engine.py.229: process: 0.55764 | load: 0.00243 [I 2022-10-26 17:53:55] ...trap/engines/engine.py.230: loss: 0.24590 [I 2022-10-26 17:53:55] ...optimizers/trijoint.py.151: total_norm: 34.478886 [I 2022-10-26 17:54:01] ...trap/engines/engine.py.224: train: epoch 22 | batch 240/2304 [I 2022-10-26 17:54:01] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:03 [I 2022-10-26 17:54:01] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00247 [I 2022-10-26 17:54:01] ...trap/engines/engine.py.230: loss: 0.16425 [I 2022-10-26 17:54:01] ...optimizers/trijoint.py.151: total_norm: 24.712547 [I 2022-10-26 17:54:07] ...trap/engines/engine.py.224: train: epoch 22 | batch 250/2304 [I 2022-10-26 17:54:07] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:58 [I 2022-10-26 17:54:07] ...trap/engines/engine.py.229: process: 0.55499 | load: 0.00245 [I 2022-10-26 17:54:07] ...trap/engines/engine.py.230: loss: 0.15649 [I 2022-10-26 17:54:07] ...optimizers/trijoint.py.151: total_norm: 19.102079 [I 2022-10-26 17:54:12] ...trap/engines/engine.py.224: train: epoch 22 | batch 260/2304 [I 2022-10-26 17:54:12] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:03 [I 2022-10-26 17:54:12] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00255 [I 2022-10-26 17:54:12] ...trap/engines/engine.py.230: loss: 0.20234 [I 2022-10-26 17:54:12] ...optimizers/trijoint.py.151: total_norm: 14.274170 [I 2022-10-26 17:54:18] ...trap/engines/engine.py.224: train: epoch 22 | batch 270/2304 [I 2022-10-26 17:54:18] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:46 [I 2022-10-26 17:54:18] ...trap/engines/engine.py.229: process: 0.54858 | load: 0.00243 [I 2022-10-26 17:54:18] ...trap/engines/engine.py.230: loss: 0.24656 [I 2022-10-26 17:54:18] ...optimizers/trijoint.py.151: total_norm: 40.430431 [I 2022-10-26 17:54:23] ...trap/engines/engine.py.224: train: epoch 22 | batch 280/2304 [I 2022-10-26 17:54:23] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:44 [I 2022-10-26 17:54:23] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00245 [I 2022-10-26 17:54:23] ...trap/engines/engine.py.230: loss: 0.24486 [I 2022-10-26 17:54:23] ...optimizers/trijoint.py.151: total_norm: 20.283743 [I 2022-10-26 17:54:29] ...trap/engines/engine.py.224: train: epoch 22 | batch 290/2304 [I 2022-10-26 17:54:29] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:37 [I 2022-10-26 17:54:29] ...trap/engines/engine.py.229: process: 0.54303 | load: 0.00260 [I 2022-10-26 17:54:29] ...trap/engines/engine.py.230: loss: 0.20774 [I 2022-10-26 17:54:29] ...optimizers/trijoint.py.151: total_norm: 22.178303 [I 2022-10-26 17:54:34] ...trap/engines/engine.py.224: train: epoch 22 | batch 300/2304 [I 2022-10-26 17:54:34] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:25 [I 2022-10-26 17:54:34] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00253 [I 2022-10-26 17:54:34] ...trap/engines/engine.py.230: loss: 0.17600 [I 2022-10-26 17:54:34] ...optimizers/trijoint.py.151: total_norm: 22.732733 [I 2022-10-26 17:54:40] ...trap/engines/engine.py.224: train: epoch 22 | batch 310/2304 [I 2022-10-26 17:54:40] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:26 [I 2022-10-26 17:54:40] ...trap/engines/engine.py.229: process: 0.54882 | load: 0.00247 [I 2022-10-26 17:54:40] ...trap/engines/engine.py.230: loss: 0.20870 [I 2022-10-26 17:54:40] ...optimizers/trijoint.py.151: total_norm: 24.544634 [I 2022-10-26 17:54:46] ...trap/engines/engine.py.224: train: epoch 22 | batch 320/2304 [I 2022-10-26 17:54:46] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:24 [I 2022-10-26 17:54:46] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00252 [I 2022-10-26 17:54:46] ...trap/engines/engine.py.230: loss: 0.17133 [I 2022-10-26 17:54:46] ...optimizers/trijoint.py.151: total_norm: 17.930016 [I 2022-10-26 17:54:51] ...trap/engines/engine.py.224: train: epoch 22 | batch 330/2304 [I 2022-10-26 17:54:51] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:11 [I 2022-10-26 17:54:51] ...trap/engines/engine.py.229: process: 0.54310 | load: 0.00246 [I 2022-10-26 17:54:51] ...trap/engines/engine.py.230: loss: 0.31180 [I 2022-10-26 17:54:51] ...optimizers/trijoint.py.151: total_norm: 34.435799 [I 2022-10-26 17:54:57] ...trap/engines/engine.py.224: train: epoch 22 | batch 340/2304 [I 2022-10-26 17:54:57] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:14 [I 2022-10-26 17:54:57] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00256 [I 2022-10-26 17:54:57] ...trap/engines/engine.py.230: loss: 0.17256 [I 2022-10-26 17:54:57] ...optimizers/trijoint.py.151: total_norm: 18.852034 [I 2022-10-26 17:55:02] ...trap/engines/engine.py.224: train: epoch 22 | batch 350/2304 [I 2022-10-26 17:55:02] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:07 [I 2022-10-26 17:55:02] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00248 [I 2022-10-26 17:55:02] ...trap/engines/engine.py.230: loss: 0.21233 [I 2022-10-26 17:55:02] ...optimizers/trijoint.py.151: total_norm: 44.669041 [I 2022-10-26 17:55:08] ...trap/engines/engine.py.224: train: epoch 22 | batch 360/2304 [I 2022-10-26 17:55:08] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:04 [I 2022-10-26 17:55:08] ...trap/engines/engine.py.229: process: 0.55096 | load: 0.00263 [I 2022-10-26 17:55:08] ...trap/engines/engine.py.230: loss: 0.14526 [I 2022-10-26 17:55:08] ...optimizers/trijoint.py.151: total_norm: 26.096605 [I 2022-10-26 17:55:13] ...trap/engines/engine.py.224: train: epoch 22 | batch 370/2304 [I 2022-10-26 17:55:13] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:53 [I 2022-10-26 17:55:13] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00246 [I 2022-10-26 17:55:13] ...trap/engines/engine.py.230: loss: 0.26781 [I 2022-10-26 17:55:13] ...optimizers/trijoint.py.151: total_norm: 20.799585 [I 2022-10-26 17:55:19] ...trap/engines/engine.py.224: train: epoch 22 | batch 380/2304 [I 2022-10-26 17:55:19] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:42 [I 2022-10-26 17:55:19] ...trap/engines/engine.py.229: process: 0.54158 | load: 0.00241 [I 2022-10-26 17:55:19] ...trap/engines/engine.py.230: loss: 0.24396 [I 2022-10-26 17:55:19] ...optimizers/trijoint.py.151: total_norm: 27.817373 [I 2022-10-26 17:55:25] ...trap/engines/engine.py.224: train: epoch 22 | batch 390/2304 [I 2022-10-26 17:55:25] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:46 [I 2022-10-26 17:55:25] ...trap/engines/engine.py.229: process: 0.57506 | load: 0.00241 [I 2022-10-26 17:55:25] ...trap/engines/engine.py.230: loss: 0.15279 [I 2022-10-26 17:55:25] ...optimizers/trijoint.py.151: total_norm: 16.408718 [I 2022-10-26 17:55:30] ...trap/engines/engine.py.224: train: epoch 22 | batch 400/2304 [I 2022-10-26 17:55:30] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:46 [I 2022-10-26 17:55:30] ...trap/engines/engine.py.229: process: 0.57299 | load: 0.00244 [I 2022-10-26 17:55:30] ...trap/engines/engine.py.230: loss: 0.20599 [I 2022-10-26 17:55:30] ...optimizers/trijoint.py.151: total_norm: 20.211847 [I 2022-10-26 17:55:36] ...trap/engines/engine.py.224: train: epoch 22 | batch 410/2304 [I 2022-10-26 17:55:36] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:44 [I 2022-10-26 17:55:36] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00247 [I 2022-10-26 17:55:36] ...trap/engines/engine.py.230: loss: 0.19116 [I 2022-10-26 17:55:36] ...optimizers/trijoint.py.151: total_norm: 22.459999 [I 2022-10-26 17:55:42] ...trap/engines/engine.py.224: train: epoch 22 | batch 420/2304 [I 2022-10-26 17:55:42] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:18:19 [I 2022-10-26 17:55:42] ...trap/engines/engine.py.229: process: 0.69459 | load: 0.00251 [I 2022-10-26 17:55:42] ...trap/engines/engine.py.230: loss: 0.16574 [I 2022-10-26 17:55:42] ...optimizers/trijoint.py.151: total_norm: 15.412081 [I 2022-10-26 17:55:47] ...trap/engines/engine.py.224: train: epoch 22 | batch 430/2304 [I 2022-10-26 17:55:47] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:28 [I 2022-10-26 17:55:47] ...trap/engines/engine.py.229: process: 0.57090 | load: 0.00253 [I 2022-10-26 17:55:47] ...trap/engines/engine.py.230: loss: 0.22690 [I 2022-10-26 17:55:47] ...optimizers/trijoint.py.151: total_norm: 22.376953 [I 2022-10-26 17:55:53] ...trap/engines/engine.py.224: train: epoch 22 | batch 440/2304 [I 2022-10-26 17:55:53] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:08 [I 2022-10-26 17:55:53] ...trap/engines/engine.py.229: process: 0.54531 | load: 0.00244 [I 2022-10-26 17:55:53] ...trap/engines/engine.py.230: loss: 0.12881 [I 2022-10-26 17:55:53] ...optimizers/trijoint.py.151: total_norm: 26.228306 [I 2022-10-26 17:55:58] ...trap/engines/engine.py.224: train: epoch 22 | batch 450/2304 [I 2022-10-26 17:55:58] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:04 [I 2022-10-26 17:55:58] ...trap/engines/engine.py.229: process: 0.54998 | load: 0.00252 [I 2022-10-26 17:55:58] ...trap/engines/engine.py.230: loss: 0.22874 [I 2022-10-26 17:55:58] ...optimizers/trijoint.py.151: total_norm: 20.267591 [I 2022-10-26 17:56:04] ...trap/engines/engine.py.224: train: epoch 22 | batch 460/2304 [I 2022-10-26 17:56:04] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:10 [I 2022-10-26 17:56:04] ...trap/engines/engine.py.229: process: 0.57199 | load: 0.00302 [I 2022-10-26 17:56:04] ...trap/engines/engine.py.230: loss: 0.17931 [I 2022-10-26 17:56:04] ...optimizers/trijoint.py.151: total_norm: 33.188332 [I 2022-10-26 17:56:09] ...trap/engines/engine.py.224: train: epoch 22 | batch 470/2304 [I 2022-10-26 17:56:09] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:01 [I 2022-10-26 17:56:09] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00250 [I 2022-10-26 17:56:09] ...trap/engines/engine.py.230: loss: 0.32382 [I 2022-10-26 17:56:09] ...optimizers/trijoint.py.151: total_norm: 27.713625 [I 2022-10-26 17:56:15] ...trap/engines/engine.py.224: train: epoch 22 | batch 480/2304 [I 2022-10-26 17:56:15] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:49 [I 2022-10-26 17:56:15] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00243 [I 2022-10-26 17:56:15] ...trap/engines/engine.py.230: loss: 0.24062 [I 2022-10-26 17:56:15] ...optimizers/trijoint.py.151: total_norm: 20.205755 [I 2022-10-26 17:56:21] ...trap/engines/engine.py.224: train: epoch 22 | batch 490/2304 [I 2022-10-26 17:56:21] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:17:08 [I 2022-10-26 17:56:21] ...trap/engines/engine.py.229: process: 0.62427 | load: 0.00244 [I 2022-10-26 17:56:21] ...trap/engines/engine.py.230: loss: 0.21287 [I 2022-10-26 17:56:21] ...optimizers/trijoint.py.151: total_norm: 31.157166 [I 2022-10-26 17:56:26] ...trap/engines/engine.py.224: train: epoch 22 | batch 500/2304 [I 2022-10-26 17:56:26] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:49 [I 2022-10-26 17:56:26] ...trap/engines/engine.py.229: process: 0.55063 | load: 0.00244 [I 2022-10-26 17:56:26] ...trap/engines/engine.py.230: loss: 0.23021 [I 2022-10-26 17:56:26] ...optimizers/trijoint.py.151: total_norm: 30.569935 [I 2022-10-26 17:56:32] ...trap/engines/engine.py.224: train: epoch 22 | batch 510/2304 [I 2022-10-26 17:56:32] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:35 [I 2022-10-26 17:56:32] ...trap/engines/engine.py.229: process: 0.55162 | load: 0.00244 [I 2022-10-26 17:56:32] ...trap/engines/engine.py.230: loss: 0.20196 [I 2022-10-26 17:56:32] ...optimizers/trijoint.py.151: total_norm: 21.974264 [I 2022-10-26 17:56:37] ...trap/engines/engine.py.224: train: epoch 22 | batch 520/2304 [I 2022-10-26 17:56:37] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:32 [I 2022-10-26 17:56:37] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00249 [I 2022-10-26 17:56:37] ...trap/engines/engine.py.230: loss: 0.15752 [I 2022-10-26 17:56:37] ...optimizers/trijoint.py.151: total_norm: 17.825577 [I 2022-10-26 17:56:43] ...trap/engines/engine.py.224: train: epoch 22 | batch 530/2304 [I 2022-10-26 17:56:43] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:21 [I 2022-10-26 17:56:43] ...trap/engines/engine.py.229: process: 0.55499 | load: 0.00248 [I 2022-10-26 17:56:43] ...trap/engines/engine.py.230: loss: 0.17864 [I 2022-10-26 17:56:43] ...optimizers/trijoint.py.151: total_norm: 9.594367 [I 2022-10-26 17:56:48] ...trap/engines/engine.py.224: train: epoch 22 | batch 540/2304 [I 2022-10-26 17:56:48] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:19 [I 2022-10-26 17:56:48] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00250 [I 2022-10-26 17:56:48] ...trap/engines/engine.py.230: loss: 0.25188 [I 2022-10-26 17:56:48] ...optimizers/trijoint.py.151: total_norm: 27.872416 [I 2022-10-26 17:56:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 550/2304 [I 2022-10-26 17:56:54] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:15 [I 2022-10-26 17:56:54] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00244 [I 2022-10-26 17:56:54] ...trap/engines/engine.py.230: loss: 0.17030 [I 2022-10-26 17:56:54] ...optimizers/trijoint.py.151: total_norm: 19.648680 [I 2022-10-26 17:57:00] ...trap/engines/engine.py.224: train: epoch 22 | batch 560/2304 [I 2022-10-26 17:57:00] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:24 [I 2022-10-26 17:57:00] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00251 [I 2022-10-26 17:57:00] ...trap/engines/engine.py.230: loss: 0.15345 [I 2022-10-26 17:57:00] ...optimizers/trijoint.py.151: total_norm: 17.555824 [I 2022-10-26 17:57:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 570/2304 [I 2022-10-26 17:57:05] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:11 [I 2022-10-26 17:57:05] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00250 [I 2022-10-26 17:57:05] ...trap/engines/engine.py.230: loss: 0.20869 [I 2022-10-26 17:57:05] ...optimizers/trijoint.py.151: total_norm: 22.290701 [I 2022-10-26 17:57:11] ...trap/engines/engine.py.224: train: epoch 22 | batch 580/2304 [I 2022-10-26 17:57:11] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:15:58 [I 2022-10-26 17:57:11] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00249 [I 2022-10-26 17:57:11] ...trap/engines/engine.py.230: loss: 0.14815 [I 2022-10-26 17:57:11] ...optimizers/trijoint.py.151: total_norm: 17.044439 [I 2022-10-26 17:57:16] ...trap/engines/engine.py.224: train: epoch 22 | batch 590/2304 [I 2022-10-26 17:57:16] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:53 [I 2022-10-26 17:57:16] ...trap/engines/engine.py.229: process: 0.55453 | load: 0.00245 [I 2022-10-26 17:57:16] ...trap/engines/engine.py.230: loss: 0.26761 [I 2022-10-26 17:57:16] ...optimizers/trijoint.py.151: total_norm: 41.306870 [I 2022-10-26 17:57:22] ...trap/engines/engine.py.224: train: epoch 22 | batch 600/2304 [I 2022-10-26 17:57:22] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:46 [I 2022-10-26 17:57:22] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00243 [I 2022-10-26 17:57:22] ...trap/engines/engine.py.230: loss: 0.13919 [I 2022-10-26 17:57:22] ...optimizers/trijoint.py.151: total_norm: 26.942266 [I 2022-10-26 17:57:28] ...trap/engines/engine.py.224: train: epoch 22 | batch 610/2304 [I 2022-10-26 17:57:28] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:42 [I 2022-10-26 17:57:28] ...trap/engines/engine.py.229: process: 0.54510 | load: 0.00249 [I 2022-10-26 17:57:28] ...trap/engines/engine.py.230: loss: 0.15252 [I 2022-10-26 17:57:28] ...optimizers/trijoint.py.151: total_norm: 9.970533 [I 2022-10-26 17:57:33] ...trap/engines/engine.py.224: train: epoch 22 | batch 620/2304 [I 2022-10-26 17:57:33] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:35 [I 2022-10-26 17:57:33] ...trap/engines/engine.py.229: process: 0.55466 | load: 0.00271 [I 2022-10-26 17:57:33] ...trap/engines/engine.py.230: loss: 0.22658 [I 2022-10-26 17:57:33] ...optimizers/trijoint.py.151: total_norm: 25.160736 [I 2022-10-26 17:57:39] ...trap/engines/engine.py.224: train: epoch 22 | batch 630/2304 [I 2022-10-26 17:57:39] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:32 [I 2022-10-26 17:57:39] ...trap/engines/engine.py.229: process: 0.54807 | load: 0.00250 [I 2022-10-26 17:57:39] ...trap/engines/engine.py.230: loss: 0.22155 [I 2022-10-26 17:57:39] ...optimizers/trijoint.py.151: total_norm: 23.988230 [I 2022-10-26 17:57:44] ...trap/engines/engine.py.224: train: epoch 22 | batch 640/2304 [I 2022-10-26 17:57:44] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:22 [I 2022-10-26 17:57:44] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00250 [I 2022-10-26 17:57:44] ...trap/engines/engine.py.230: loss: 0.22034 [I 2022-10-26 17:57:44] ...optimizers/trijoint.py.151: total_norm: 39.045826 [I 2022-10-26 17:57:50] ...trap/engines/engine.py.224: train: epoch 22 | batch 650/2304 [I 2022-10-26 17:57:50] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:23 [I 2022-10-26 17:57:50] ...trap/engines/engine.py.229: process: 0.57054 | load: 0.00247 [I 2022-10-26 17:57:50] ...trap/engines/engine.py.230: loss: 0.20498 [I 2022-10-26 17:57:50] ...optimizers/trijoint.py.151: total_norm: 19.041344 [I 2022-10-26 17:57:55] ...trap/engines/engine.py.224: train: epoch 22 | batch 660/2304 [I 2022-10-26 17:57:55] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:24 [I 2022-10-26 17:57:55] ...trap/engines/engine.py.229: process: 0.54950 | load: 0.00248 [I 2022-10-26 17:57:55] ...trap/engines/engine.py.230: loss: 0.14767 [I 2022-10-26 17:57:55] ...optimizers/trijoint.py.151: total_norm: 21.575050 [I 2022-10-26 17:58:01] ...trap/engines/engine.py.224: train: epoch 22 | batch 670/2304 [I 2022-10-26 17:58:01] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:12 [I 2022-10-26 17:58:01] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00248 [I 2022-10-26 17:58:01] ...trap/engines/engine.py.230: loss: 0.20279 [I 2022-10-26 17:58:01] ...optimizers/trijoint.py.151: total_norm: 24.845684 [I 2022-10-26 17:58:07] ...trap/engines/engine.py.224: train: epoch 22 | batch 680/2304 [I 2022-10-26 17:58:07] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:13 [I 2022-10-26 17:58:07] ...trap/engines/engine.py.229: process: 0.57947 | load: 0.00246 [I 2022-10-26 17:58:07] ...trap/engines/engine.py.230: loss: 0.21144 [I 2022-10-26 17:58:07] ...optimizers/trijoint.py.151: total_norm: 22.139168 [I 2022-10-26 17:58:12] ...trap/engines/engine.py.224: train: epoch 22 | batch 690/2304 [I 2022-10-26 17:58:12] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:02 [I 2022-10-26 17:58:12] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00250 [I 2022-10-26 17:58:12] ...trap/engines/engine.py.230: loss: 0.17451 [I 2022-10-26 17:58:12] ...optimizers/trijoint.py.151: total_norm: 20.693140 [I 2022-10-26 17:58:18] ...trap/engines/engine.py.224: train: epoch 22 | batch 700/2304 [I 2022-10-26 17:58:18] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:49 [I 2022-10-26 17:58:18] ...trap/engines/engine.py.229: process: 0.56005 | load: 0.00249 [I 2022-10-26 17:58:18] ...trap/engines/engine.py.230: loss: 0.18192 [I 2022-10-26 17:58:18] ...optimizers/trijoint.py.151: total_norm: 26.937998 [I 2022-10-26 17:58:23] ...trap/engines/engine.py.224: train: epoch 22 | batch 710/2304 [I 2022-10-26 17:58:23] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:45 [I 2022-10-26 17:58:23] ...trap/engines/engine.py.229: process: 0.54613 | load: 0.00241 [I 2022-10-26 17:58:23] ...trap/engines/engine.py.230: loss: 0.16885 [I 2022-10-26 17:58:23] ...optimizers/trijoint.py.151: total_norm: 16.678444 [I 2022-10-26 17:58:29] ...trap/engines/engine.py.224: train: epoch 22 | batch 720/2304 [I 2022-10-26 17:58:29] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:31 [I 2022-10-26 17:58:29] ...trap/engines/engine.py.229: process: 0.53746 | load: 0.00251 [I 2022-10-26 17:58:29] ...trap/engines/engine.py.230: loss: 0.14838 [I 2022-10-26 17:58:29] ...optimizers/trijoint.py.151: total_norm: 28.484474 [I 2022-10-26 17:58:35] ...trap/engines/engine.py.224: train: epoch 22 | batch 730/2304 [I 2022-10-26 17:58:35] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:28 [I 2022-10-26 17:58:35] ...trap/engines/engine.py.229: process: 0.54678 | load: 0.00240 [I 2022-10-26 17:58:35] ...trap/engines/engine.py.230: loss: 0.16211 [I 2022-10-26 17:58:35] ...optimizers/trijoint.py.151: total_norm: 27.611483 [I 2022-10-26 17:58:40] ...trap/engines/engine.py.224: train: epoch 22 | batch 740/2304 [I 2022-10-26 17:58:40] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:33 [I 2022-10-26 17:58:40] ...trap/engines/engine.py.229: process: 0.57351 | load: 0.00245 [I 2022-10-26 17:58:40] ...trap/engines/engine.py.230: loss: 0.22166 [I 2022-10-26 17:58:40] ...optimizers/trijoint.py.151: total_norm: 18.004368 [I 2022-10-26 17:58:46] ...trap/engines/engine.py.224: train: epoch 22 | batch 750/2304 [I 2022-10-26 17:58:46] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:22 [I 2022-10-26 17:58:46] ...trap/engines/engine.py.229: process: 0.55437 | load: 0.00274 [I 2022-10-26 17:58:46] ...trap/engines/engine.py.230: loss: 0.25355 [I 2022-10-26 17:58:46] ...optimizers/trijoint.py.151: total_norm: 24.951784 [I 2022-10-26 17:58:51] ...trap/engines/engine.py.224: train: epoch 22 | batch 760/2304 [I 2022-10-26 17:58:51] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:16 [I 2022-10-26 17:58:51] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00245 [I 2022-10-26 17:58:51] ...trap/engines/engine.py.230: loss: 0.23188 [I 2022-10-26 17:58:51] ...optimizers/trijoint.py.151: total_norm: 42.611797 [I 2022-10-26 17:58:57] ...trap/engines/engine.py.224: train: epoch 22 | batch 770/2304 [I 2022-10-26 17:58:57] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:12 [I 2022-10-26 17:58:57] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00249 [I 2022-10-26 17:58:57] ...trap/engines/engine.py.230: loss: 0.19819 [I 2022-10-26 17:58:57] ...optimizers/trijoint.py.151: total_norm: 15.680009 [I 2022-10-26 17:59:02] ...trap/engines/engine.py.224: train: epoch 22 | batch 780/2304 [I 2022-10-26 17:59:02] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:10 [I 2022-10-26 17:59:02] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00251 [I 2022-10-26 17:59:02] ...trap/engines/engine.py.230: loss: 0.16357 [I 2022-10-26 17:59:02] ...optimizers/trijoint.py.151: total_norm: 18.836691 [I 2022-10-26 17:59:08] ...trap/engines/engine.py.224: train: epoch 22 | batch 790/2304 [I 2022-10-26 17:59:08] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:14 [I 2022-10-26 17:59:08] ...trap/engines/engine.py.229: process: 0.59840 | load: 0.00268 [I 2022-10-26 17:59:08] ...trap/engines/engine.py.230: loss: 0.21466 [I 2022-10-26 17:59:08] ...optimizers/trijoint.py.151: total_norm: 28.629593 [I 2022-10-26 17:59:14] ...trap/engines/engine.py.224: train: epoch 22 | batch 800/2304 [I 2022-10-26 17:59:14] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:14:03 [I 2022-10-26 17:59:14] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00251 [I 2022-10-26 17:59:14] ...trap/engines/engine.py.230: loss: 0.15356 [I 2022-10-26 17:59:14] ...optimizers/trijoint.py.151: total_norm: 17.808498 [I 2022-10-26 17:59:19] ...trap/engines/engine.py.224: train: epoch 22 | batch 810/2304 [I 2022-10-26 17:59:19] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:54 [I 2022-10-26 17:59:19] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00240 [I 2022-10-26 17:59:19] ...trap/engines/engine.py.230: loss: 0.14757 [I 2022-10-26 17:59:19] ...optimizers/trijoint.py.151: total_norm: 18.820242 [I 2022-10-26 17:59:25] ...trap/engines/engine.py.224: train: epoch 22 | batch 820/2304 [I 2022-10-26 17:59:25] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:47 [I 2022-10-26 17:59:25] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00248 [I 2022-10-26 17:59:25] ...trap/engines/engine.py.230: loss: 0.16322 [I 2022-10-26 17:59:25] ...optimizers/trijoint.py.151: total_norm: 24.166838 [I 2022-10-26 17:59:30] ...trap/engines/engine.py.224: train: epoch 22 | batch 830/2304 [I 2022-10-26 17:59:30] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:44 [I 2022-10-26 17:59:30] ...trap/engines/engine.py.229: process: 0.56336 | load: 0.00251 [I 2022-10-26 17:59:30] ...trap/engines/engine.py.230: loss: 0.14538 [I 2022-10-26 17:59:30] ...optimizers/trijoint.py.151: total_norm: 27.367184 [I 2022-10-26 17:59:36] ...trap/engines/engine.py.224: train: epoch 22 | batch 840/2304 [I 2022-10-26 17:59:36] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:31 [I 2022-10-26 17:59:36] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00256 [I 2022-10-26 17:59:36] ...trap/engines/engine.py.230: loss: 0.15323 [I 2022-10-26 17:59:36] ...optimizers/trijoint.py.151: total_norm: 35.612835 [I 2022-10-26 17:59:42] ...trap/engines/engine.py.224: train: epoch 22 | batch 850/2304 [I 2022-10-26 17:59:42] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:28 [I 2022-10-26 17:59:42] ...trap/engines/engine.py.229: process: 0.55268 | load: 0.00255 [I 2022-10-26 17:59:42] ...trap/engines/engine.py.230: loss: 0.30257 [I 2022-10-26 17:59:42] ...optimizers/trijoint.py.151: total_norm: 35.087242 [I 2022-10-26 17:59:47] ...trap/engines/engine.py.224: train: epoch 22 | batch 860/2304 [I 2022-10-26 17:59:47] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:26 [I 2022-10-26 17:59:47] ...trap/engines/engine.py.229: process: 0.56914 | load: 0.00249 [I 2022-10-26 17:59:47] ...trap/engines/engine.py.230: loss: 0.19383 [I 2022-10-26 17:59:47] ...optimizers/trijoint.py.151: total_norm: 31.143118 [I 2022-10-26 17:59:53] ...trap/engines/engine.py.224: train: epoch 22 | batch 870/2304 [I 2022-10-26 17:59:53] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:18 [I 2022-10-26 17:59:53] ...trap/engines/engine.py.229: process: 0.55936 | load: 0.00250 [I 2022-10-26 17:59:53] ...trap/engines/engine.py.230: loss: 0.21250 [I 2022-10-26 17:59:53] ...optimizers/trijoint.py.151: total_norm: 19.294857 [I 2022-10-26 17:59:58] ...trap/engines/engine.py.224: train: epoch 22 | batch 880/2304 [I 2022-10-26 17:59:58] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:07 [I 2022-10-26 17:59:58] ...trap/engines/engine.py.229: process: 0.54789 | load: 0.00251 [I 2022-10-26 17:59:58] ...trap/engines/engine.py.230: loss: 0.20321 [I 2022-10-26 17:59:58] ...optimizers/trijoint.py.151: total_norm: 19.042830 [I 2022-10-26 18:00:04] ...trap/engines/engine.py.224: train: epoch 22 | batch 890/2304 [I 2022-10-26 18:00:04] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:07 [I 2022-10-26 18:00:04] ...trap/engines/engine.py.229: process: 0.55986 | load: 0.00246 [I 2022-10-26 18:00:04] ...trap/engines/engine.py.230: loss: 0.18892 [I 2022-10-26 18:00:04] ...optimizers/trijoint.py.151: total_norm: 23.990623 [I 2022-10-26 18:00:10] ...trap/engines/engine.py.224: train: epoch 22 | batch 900/2304 [I 2022-10-26 18:00:10] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:15 [I 2022-10-26 18:00:10] ...trap/engines/engine.py.229: process: 0.55911 | load: 0.00248 [I 2022-10-26 18:00:10] ...trap/engines/engine.py.230: loss: 0.29755 [I 2022-10-26 18:00:10] ...optimizers/trijoint.py.151: total_norm: 26.518696 [I 2022-10-26 18:00:15] ...trap/engines/engine.py.224: train: epoch 22 | batch 910/2304 [I 2022-10-26 18:00:15] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:57 [I 2022-10-26 18:00:15] ...trap/engines/engine.py.229: process: 0.55284 | load: 0.00247 [I 2022-10-26 18:00:15] ...trap/engines/engine.py.230: loss: 0.11797 [I 2022-10-26 18:00:15] ...optimizers/trijoint.py.151: total_norm: 22.679899 [I 2022-10-26 18:00:21] ...trap/engines/engine.py.224: train: epoch 22 | batch 920/2304 [I 2022-10-26 18:00:21] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:54 [I 2022-10-26 18:00:21] ...trap/engines/engine.py.229: process: 0.56631 | load: 0.00248 [I 2022-10-26 18:00:21] ...trap/engines/engine.py.230: loss: 0.15687 [I 2022-10-26 18:00:21] ...optimizers/trijoint.py.151: total_norm: 22.832312 [I 2022-10-26 18:00:26] ...trap/engines/engine.py.224: train: epoch 22 | batch 930/2304 [I 2022-10-26 18:00:26] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:46 [I 2022-10-26 18:00:26] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00242 [I 2022-10-26 18:00:26] ...trap/engines/engine.py.230: loss: 0.24177 [I 2022-10-26 18:00:26] ...optimizers/trijoint.py.151: total_norm: 17.854404 [I 2022-10-26 18:00:32] ...trap/engines/engine.py.224: train: epoch 22 | batch 940/2304 [I 2022-10-26 18:00:32] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:41 [I 2022-10-26 18:00:32] ...trap/engines/engine.py.229: process: 0.56454 | load: 0.00273 [I 2022-10-26 18:00:32] ...trap/engines/engine.py.230: loss: 0.19075 [I 2022-10-26 18:00:32] ...optimizers/trijoint.py.151: total_norm: 20.525766 [I 2022-10-26 18:00:37] ...trap/engines/engine.py.224: train: epoch 22 | batch 950/2304 [I 2022-10-26 18:00:37] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:35 [I 2022-10-26 18:00:37] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00246 [I 2022-10-26 18:00:37] ...trap/engines/engine.py.230: loss: 0.22173 [I 2022-10-26 18:00:37] ...optimizers/trijoint.py.151: total_norm: 31.982363 [I 2022-10-26 18:00:43] ...trap/engines/engine.py.224: train: epoch 22 | batch 960/2304 [I 2022-10-26 18:00:43] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:35 [I 2022-10-26 18:00:43] ...trap/engines/engine.py.229: process: 0.57201 | load: 0.00244 [I 2022-10-26 18:00:43] ...trap/engines/engine.py.230: loss: 0.20831 [I 2022-10-26 18:00:43] ...optimizers/trijoint.py.151: total_norm: 19.347731 [I 2022-10-26 18:00:49] ...trap/engines/engine.py.224: train: epoch 22 | batch 970/2304 [I 2022-10-26 18:00:49] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:19 [I 2022-10-26 18:00:49] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00292 [I 2022-10-26 18:00:49] ...trap/engines/engine.py.230: loss: 0.19432 [I 2022-10-26 18:00:49] ...optimizers/trijoint.py.151: total_norm: 25.450592 [I 2022-10-26 18:00:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 980/2304 [I 2022-10-26 18:00:54] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:17 [I 2022-10-26 18:00:54] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00259 [I 2022-10-26 18:00:54] ...trap/engines/engine.py.230: loss: 0.22790 [I 2022-10-26 18:00:54] ...optimizers/trijoint.py.151: total_norm: 19.874914 [I 2022-10-26 18:01:00] ...trap/engines/engine.py.224: train: epoch 22 | batch 990/2304 [I 2022-10-26 18:01:00] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:10 [I 2022-10-26 18:01:00] ...trap/engines/engine.py.229: process: 0.56433 | load: 0.00249 [I 2022-10-26 18:01:00] ...trap/engines/engine.py.230: loss: 0.17855 [I 2022-10-26 18:01:00] ...optimizers/trijoint.py.151: total_norm: 34.099373 [I 2022-10-26 18:01:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 1000/2304 [I 2022-10-26 18:01:05] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:11 [I 2022-10-26 18:01:05] ...trap/engines/engine.py.229: process: 0.56214 | load: 0.00249 [I 2022-10-26 18:01:05] ...trap/engines/engine.py.230: loss: 0.15061 [I 2022-10-26 18:01:05] ...optimizers/trijoint.py.151: total_norm: 19.154921 [I 2022-10-26 18:01:11] ...trap/engines/engine.py.224: train: epoch 22 | batch 1010/2304 [I 2022-10-26 18:01:11] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:02 [I 2022-10-26 18:01:11] ...trap/engines/engine.py.229: process: 0.55191 | load: 0.00250 [I 2022-10-26 18:01:11] ...trap/engines/engine.py.230: loss: 0.18001 [I 2022-10-26 18:01:11] ...optimizers/trijoint.py.151: total_norm: 14.791212 [I 2022-10-26 18:01:17] ...trap/engines/engine.py.224: train: epoch 22 | batch 1020/2304 [I 2022-10-26 18:01:17] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:56 [I 2022-10-26 18:01:17] ...trap/engines/engine.py.229: process: 0.55485 | load: 0.00251 [I 2022-10-26 18:01:17] ...trap/engines/engine.py.230: loss: 0.26138 [I 2022-10-26 18:01:17] ...optimizers/trijoint.py.151: total_norm: 23.674864 [I 2022-10-26 18:01:22] ...trap/engines/engine.py.224: train: epoch 22 | batch 1030/2304 [I 2022-10-26 18:01:22] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:45 [I 2022-10-26 18:01:22] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00249 [I 2022-10-26 18:01:22] ...trap/engines/engine.py.230: loss: 0.21674 [I 2022-10-26 18:01:22] ...optimizers/trijoint.py.151: total_norm: 23.289495 [I 2022-10-26 18:01:28] ...trap/engines/engine.py.224: train: epoch 22 | batch 1040/2304 [I 2022-10-26 18:01:28] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:45 [I 2022-10-26 18:01:28] ...trap/engines/engine.py.229: process: 0.56511 | load: 0.00245 [I 2022-10-26 18:01:28] ...trap/engines/engine.py.230: loss: 0.20469 [I 2022-10-26 18:01:28] ...optimizers/trijoint.py.151: total_norm: 15.865573 [I 2022-10-26 18:01:33] ...trap/engines/engine.py.224: train: epoch 22 | batch 1050/2304 [I 2022-10-26 18:01:33] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:46 [I 2022-10-26 18:01:33] ...trap/engines/engine.py.229: process: 0.56990 | load: 0.00252 [I 2022-10-26 18:01:33] ...trap/engines/engine.py.230: loss: 0.25166 [I 2022-10-26 18:01:33] ...optimizers/trijoint.py.151: total_norm: 27.113155 [I 2022-10-26 18:01:39] ...trap/engines/engine.py.224: train: epoch 22 | batch 1060/2304 [I 2022-10-26 18:01:39] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:33 [I 2022-10-26 18:01:39] ...trap/engines/engine.py.229: process: 0.55509 | load: 0.00256 [I 2022-10-26 18:01:39] ...trap/engines/engine.py.230: loss: 0.16711 [I 2022-10-26 18:01:39] ...optimizers/trijoint.py.151: total_norm: 14.803027 [I 2022-10-26 18:01:45] ...trap/engines/engine.py.224: train: epoch 22 | batch 1070/2304 [I 2022-10-26 18:01:45] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:30 [I 2022-10-26 18:01:45] ...trap/engines/engine.py.229: process: 0.55893 | load: 0.00245 [I 2022-10-26 18:01:45] ...trap/engines/engine.py.230: loss: 0.18339 [I 2022-10-26 18:01:45] ...optimizers/trijoint.py.151: total_norm: 19.925207 [I 2022-10-26 18:01:50] ...trap/engines/engine.py.224: train: epoch 22 | batch 1080/2304 [I 2022-10-26 18:01:50] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:24 [I 2022-10-26 18:01:50] ...trap/engines/engine.py.229: process: 0.55336 | load: 0.00242 [I 2022-10-26 18:01:50] ...trap/engines/engine.py.230: loss: 0.20550 [I 2022-10-26 18:01:50] ...optimizers/trijoint.py.151: total_norm: 30.982801 [I 2022-10-26 18:01:56] ...trap/engines/engine.py.224: train: epoch 22 | batch 1090/2304 [I 2022-10-26 18:01:56] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:18 [I 2022-10-26 18:01:56] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00248 [I 2022-10-26 18:01:56] ...trap/engines/engine.py.230: loss: 0.20501 [I 2022-10-26 18:01:56] ...optimizers/trijoint.py.151: total_norm: 21.597361 [I 2022-10-26 18:02:01] ...trap/engines/engine.py.224: train: epoch 22 | batch 1100/2304 [I 2022-10-26 18:02:01] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:10 [I 2022-10-26 18:02:01] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00249 [I 2022-10-26 18:02:01] ...trap/engines/engine.py.230: loss: 0.27285 [I 2022-10-26 18:02:01] ...optimizers/trijoint.py.151: total_norm: 42.004627 [I 2022-10-26 18:02:07] ...trap/engines/engine.py.224: train: epoch 22 | batch 1110/2304 [I 2022-10-26 18:02:07] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:00 [I 2022-10-26 18:02:07] ...trap/engines/engine.py.229: process: 0.54995 | load: 0.00248 [I 2022-10-26 18:02:07] ...trap/engines/engine.py.230: loss: 0.15639 [I 2022-10-26 18:02:07] ...optimizers/trijoint.py.151: total_norm: 21.592876 [I 2022-10-26 18:02:13] ...trap/engines/engine.py.224: train: epoch 22 | batch 1120/2304 [I 2022-10-26 18:02:13] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:03 [I 2022-10-26 18:02:13] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00247 [I 2022-10-26 18:02:13] ...trap/engines/engine.py.230: loss: 0.15750 [I 2022-10-26 18:02:13] ...optimizers/trijoint.py.151: total_norm: 36.304127 [I 2022-10-26 18:02:18] ...trap/engines/engine.py.224: train: epoch 22 | batch 1130/2304 [I 2022-10-26 18:02:18] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:55 [I 2022-10-26 18:02:18] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00245 [I 2022-10-26 18:02:18] ...trap/engines/engine.py.230: loss: 0.22483 [I 2022-10-26 18:02:18] ...optimizers/trijoint.py.151: total_norm: 13.290175 [I 2022-10-26 18:02:24] ...trap/engines/engine.py.224: train: epoch 22 | batch 1140/2304 [I 2022-10-26 18:02:24] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:49 [I 2022-10-26 18:02:24] ...trap/engines/engine.py.229: process: 0.55842 | load: 0.00246 [I 2022-10-26 18:02:24] ...trap/engines/engine.py.230: loss: 0.18705 [I 2022-10-26 18:02:24] ...optimizers/trijoint.py.151: total_norm: 24.565670 [I 2022-10-26 18:02:29] ...trap/engines/engine.py.224: train: epoch 22 | batch 1150/2304 [I 2022-10-26 18:02:29] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:44 [I 2022-10-26 18:02:29] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00249 [I 2022-10-26 18:02:29] ...trap/engines/engine.py.230: loss: 0.14463 [I 2022-10-26 18:02:29] ...optimizers/trijoint.py.151: total_norm: 18.899046 [I 2022-10-26 18:02:35] ...trap/engines/engine.py.224: train: epoch 22 | batch 1160/2304 [I 2022-10-26 18:02:35] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:36 [I 2022-10-26 18:02:35] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00251 [I 2022-10-26 18:02:35] ...trap/engines/engine.py.230: loss: 0.15065 [I 2022-10-26 18:02:35] ...optimizers/trijoint.py.151: total_norm: 12.369171 [I 2022-10-26 18:02:41] ...trap/engines/engine.py.224: train: epoch 22 | batch 1170/2304 [I 2022-10-26 18:02:41] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:32 [I 2022-10-26 18:02:41] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00241 [I 2022-10-26 18:02:41] ...trap/engines/engine.py.230: loss: 0.26206 [I 2022-10-26 18:02:41] ...optimizers/trijoint.py.151: total_norm: 27.252710 [I 2022-10-26 18:02:46] ...trap/engines/engine.py.224: train: epoch 22 | batch 1180/2304 [I 2022-10-26 18:02:46] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:25 [I 2022-10-26 18:02:46] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00250 [I 2022-10-26 18:02:46] ...trap/engines/engine.py.230: loss: 0.21878 [I 2022-10-26 18:02:46] ...optimizers/trijoint.py.151: total_norm: 18.436680 [I 2022-10-26 18:02:52] ...trap/engines/engine.py.224: train: epoch 22 | batch 1190/2304 [I 2022-10-26 18:02:52] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:21 [I 2022-10-26 18:02:52] ...trap/engines/engine.py.229: process: 0.54830 | load: 0.00246 [I 2022-10-26 18:02:52] ...trap/engines/engine.py.230: loss: 0.16245 [I 2022-10-26 18:02:52] ...optimizers/trijoint.py.151: total_norm: 25.464975 [I 2022-10-26 18:02:57] ...trap/engines/engine.py.224: train: epoch 22 | batch 1200/2304 [I 2022-10-26 18:02:57] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:13 [I 2022-10-26 18:02:57] ...trap/engines/engine.py.229: process: 0.54773 | load: 0.00255 [I 2022-10-26 18:02:57] ...trap/engines/engine.py.230: loss: 0.19990 [I 2022-10-26 18:02:57] ...optimizers/trijoint.py.151: total_norm: 24.169863 [I 2022-10-26 18:03:03] ...trap/engines/engine.py.224: train: epoch 22 | batch 1210/2304 [I 2022-10-26 18:03:03] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:10 [I 2022-10-26 18:03:03] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00250 [I 2022-10-26 18:03:03] ...trap/engines/engine.py.230: loss: 0.11092 [I 2022-10-26 18:03:03] ...optimizers/trijoint.py.151: total_norm: 19.728449 [I 2022-10-26 18:03:08] ...trap/engines/engine.py.224: train: epoch 22 | batch 1220/2304 [I 2022-10-26 18:03:08] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:04 [I 2022-10-26 18:03:08] ...trap/engines/engine.py.229: process: 0.55289 | load: 0.00245 [I 2022-10-26 18:03:08] ...trap/engines/engine.py.230: loss: 0.15276 [I 2022-10-26 18:03:08] ...optimizers/trijoint.py.151: total_norm: 14.073915 [I 2022-10-26 18:03:14] ...trap/engines/engine.py.224: train: epoch 22 | batch 1230/2304 [I 2022-10-26 18:03:14] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:10:01 [I 2022-10-26 18:03:14] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00247 [I 2022-10-26 18:03:14] ...trap/engines/engine.py.230: loss: 0.21111 [I 2022-10-26 18:03:14] ...optimizers/trijoint.py.151: total_norm: 17.384850 [I 2022-10-26 18:03:20] ...trap/engines/engine.py.224: train: epoch 22 | batch 1240/2304 [I 2022-10-26 18:03:20] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:48 [I 2022-10-26 18:03:20] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00244 [I 2022-10-26 18:03:20] ...trap/engines/engine.py.230: loss: 0.23733 [I 2022-10-26 18:03:20] ...optimizers/trijoint.py.151: total_norm: 39.331062 [I 2022-10-26 18:03:25] ...trap/engines/engine.py.224: train: epoch 22 | batch 1250/2304 [I 2022-10-26 18:03:25] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:50 [I 2022-10-26 18:03:25] ...trap/engines/engine.py.229: process: 0.56225 | load: 0.00254 [I 2022-10-26 18:03:25] ...trap/engines/engine.py.230: loss: 0.22032 [I 2022-10-26 18:03:25] ...optimizers/trijoint.py.151: total_norm: 22.098820 [I 2022-10-26 18:03:31] ...trap/engines/engine.py.224: train: epoch 22 | batch 1260/2304 [I 2022-10-26 18:03:31] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:40 [I 2022-10-26 18:03:31] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00254 [I 2022-10-26 18:03:31] ...trap/engines/engine.py.230: loss: 0.18669 [I 2022-10-26 18:03:31] ...optimizers/trijoint.py.151: total_norm: 17.097589 [I 2022-10-26 18:03:36] ...trap/engines/engine.py.224: train: epoch 22 | batch 1270/2304 [I 2022-10-26 18:03:36] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:35 [I 2022-10-26 18:03:36] ...trap/engines/engine.py.229: process: 0.55376 | load: 0.00251 [I 2022-10-26 18:03:36] ...trap/engines/engine.py.230: loss: 0.20970 [I 2022-10-26 18:03:36] ...optimizers/trijoint.py.151: total_norm: 25.829977 [I 2022-10-26 18:03:42] ...trap/engines/engine.py.224: train: epoch 22 | batch 1280/2304 [I 2022-10-26 18:03:42] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:35 [I 2022-10-26 18:03:42] ...trap/engines/engine.py.229: process: 0.56452 | load: 0.00250 [I 2022-10-26 18:03:42] ...trap/engines/engine.py.230: loss: 0.20938 [I 2022-10-26 18:03:42] ...optimizers/trijoint.py.151: total_norm: 17.822599 [I 2022-10-26 18:03:48] ...trap/engines/engine.py.224: train: epoch 22 | batch 1290/2304 [I 2022-10-26 18:03:48] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:26 [I 2022-10-26 18:03:48] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00246 [I 2022-10-26 18:03:48] ...trap/engines/engine.py.230: loss: 0.19646 [I 2022-10-26 18:03:48] ...optimizers/trijoint.py.151: total_norm: 25.377708 [I 2022-10-26 18:03:53] ...trap/engines/engine.py.224: train: epoch 22 | batch 1300/2304 [I 2022-10-26 18:03:53] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:20 [I 2022-10-26 18:03:53] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00243 [I 2022-10-26 18:03:53] ...trap/engines/engine.py.230: loss: 0.24011 [I 2022-10-26 18:03:53] ...optimizers/trijoint.py.151: total_norm: 16.756159 [I 2022-10-26 18:03:59] ...trap/engines/engine.py.224: train: epoch 22 | batch 1310/2304 [I 2022-10-26 18:03:59] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:16 [I 2022-10-26 18:03:59] ...trap/engines/engine.py.229: process: 0.55925 | load: 0.00249 [I 2022-10-26 18:03:59] ...trap/engines/engine.py.230: loss: 0.14663 [I 2022-10-26 18:03:59] ...optimizers/trijoint.py.151: total_norm: 16.433088 [I 2022-10-26 18:04:04] ...trap/engines/engine.py.224: train: epoch 22 | batch 1320/2304 [I 2022-10-26 18:04:04] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:09 [I 2022-10-26 18:04:04] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00245 [I 2022-10-26 18:04:04] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-26 18:04:04] ...optimizers/trijoint.py.151: total_norm: 24.160942 [I 2022-10-26 18:04:10] ...trap/engines/engine.py.224: train: epoch 22 | batch 1330/2304 [I 2022-10-26 18:04:10] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:08:57 [I 2022-10-26 18:04:10] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00249 [I 2022-10-26 18:04:10] ...trap/engines/engine.py.230: loss: 0.19655 [I 2022-10-26 18:04:10] ...optimizers/trijoint.py.151: total_norm: 15.212646 [I 2022-10-26 18:04:15] ...trap/engines/engine.py.224: train: epoch 22 | batch 1340/2304 [I 2022-10-26 18:04:15] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:53 [I 2022-10-26 18:04:15] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00247 [I 2022-10-26 18:04:15] ...trap/engines/engine.py.230: loss: 0.23021 [I 2022-10-26 18:04:15] ...optimizers/trijoint.py.151: total_norm: 19.445948 [I 2022-10-26 18:04:21] ...trap/engines/engine.py.224: train: epoch 22 | batch 1350/2304 [I 2022-10-26 18:04:21] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:50 [I 2022-10-26 18:04:21] ...trap/engines/engine.py.229: process: 0.55893 | load: 0.00242 [I 2022-10-26 18:04:21] ...trap/engines/engine.py.230: loss: 0.13634 [I 2022-10-26 18:04:21] ...optimizers/trijoint.py.151: total_norm: 44.254971 [I 2022-10-26 18:04:27] ...trap/engines/engine.py.224: train: epoch 22 | batch 1360/2304 [I 2022-10-26 18:04:27] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:49 [I 2022-10-26 18:04:27] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00247 [I 2022-10-26 18:04:27] ...trap/engines/engine.py.230: loss: 0.19615 [I 2022-10-26 18:04:27] ...optimizers/trijoint.py.151: total_norm: 21.746435 [I 2022-10-26 18:04:32] ...trap/engines/engine.py.224: train: epoch 22 | batch 1370/2304 [I 2022-10-26 18:04:32] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:40 [I 2022-10-26 18:04:32] ...trap/engines/engine.py.229: process: 0.54175 | load: 0.00255 [I 2022-10-26 18:04:32] ...trap/engines/engine.py.230: loss: 0.20632 [I 2022-10-26 18:04:32] ...optimizers/trijoint.py.151: total_norm: 21.921566 [I 2022-10-26 18:04:38] ...trap/engines/engine.py.224: train: epoch 22 | batch 1380/2304 [I 2022-10-26 18:04:38] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:34 [I 2022-10-26 18:04:38] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00244 [I 2022-10-26 18:04:38] ...trap/engines/engine.py.230: loss: 0.26913 [I 2022-10-26 18:04:38] ...optimizers/trijoint.py.151: total_norm: 27.978355 [I 2022-10-26 18:04:43] ...trap/engines/engine.py.224: train: epoch 22 | batch 1390/2304 [I 2022-10-26 18:04:43] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:31 [I 2022-10-26 18:04:43] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00244 [I 2022-10-26 18:04:43] ...trap/engines/engine.py.230: loss: 0.14893 [I 2022-10-26 18:04:43] ...optimizers/trijoint.py.151: total_norm: 19.399170 [I 2022-10-26 18:04:49] ...trap/engines/engine.py.224: train: epoch 22 | batch 1400/2304 [I 2022-10-26 18:04:49] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:23 [I 2022-10-26 18:04:49] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00244 [I 2022-10-26 18:04:49] ...trap/engines/engine.py.230: loss: 0.16057 [I 2022-10-26 18:04:49] ...optimizers/trijoint.py.151: total_norm: 20.521646 [I 2022-10-26 18:04:55] ...trap/engines/engine.py.224: train: epoch 22 | batch 1410/2304 [I 2022-10-26 18:04:55] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:22 [I 2022-10-26 18:04:55] ...trap/engines/engine.py.229: process: 0.56164 | load: 0.00249 [I 2022-10-26 18:04:55] ...trap/engines/engine.py.230: loss: 0.20583 [I 2022-10-26 18:04:55] ...optimizers/trijoint.py.151: total_norm: 21.108437 [I 2022-10-26 18:05:00] ...trap/engines/engine.py.224: train: epoch 22 | batch 1420/2304 [I 2022-10-26 18:05:00] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:13 [I 2022-10-26 18:05:00] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00243 [I 2022-10-26 18:05:00] ...trap/engines/engine.py.230: loss: 0.18859 [I 2022-10-26 18:05:00] ...optimizers/trijoint.py.151: total_norm: 15.809452 [I 2022-10-26 18:05:06] ...trap/engines/engine.py.224: train: epoch 22 | batch 1430/2304 [I 2022-10-26 18:05:06] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:07 [I 2022-10-26 18:05:06] ...trap/engines/engine.py.229: process: 0.57101 | load: 0.00257 [I 2022-10-26 18:05:06] ...trap/engines/engine.py.230: loss: 0.16567 [I 2022-10-26 18:05:06] ...optimizers/trijoint.py.151: total_norm: 33.561413 [I 2022-10-26 18:05:11] ...trap/engines/engine.py.224: train: epoch 22 | batch 1440/2304 [I 2022-10-26 18:05:11] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:01 [I 2022-10-26 18:05:11] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00255 [I 2022-10-26 18:05:11] ...trap/engines/engine.py.230: loss: 0.18750 [I 2022-10-26 18:05:11] ...optimizers/trijoint.py.151: total_norm: 39.833160 [I 2022-10-26 18:05:17] ...trap/engines/engine.py.224: train: epoch 22 | batch 1450/2304 [I 2022-10-26 18:05:17] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:07:57 [I 2022-10-26 18:05:17] ...trap/engines/engine.py.229: process: 0.56413 | load: 0.00250 [I 2022-10-26 18:05:17] ...trap/engines/engine.py.230: loss: 0.12108 [I 2022-10-26 18:05:17] ...optimizers/trijoint.py.151: total_norm: 16.785637 [I 2022-10-26 18:05:23] ...trap/engines/engine.py.224: train: epoch 22 | batch 1460/2304 [I 2022-10-26 18:05:23] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:54 [I 2022-10-26 18:05:23] ...trap/engines/engine.py.229: process: 0.59181 | load: 0.00244 [I 2022-10-26 18:05:23] ...trap/engines/engine.py.230: loss: 0.19375 [I 2022-10-26 18:05:23] ...optimizers/trijoint.py.151: total_norm: 23.667574 [I 2022-10-26 18:05:28] ...trap/engines/engine.py.224: train: epoch 22 | batch 1470/2304 [I 2022-10-26 18:05:28] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:51 [I 2022-10-26 18:05:28] ...trap/engines/engine.py.229: process: 0.57070 | load: 0.00245 [I 2022-10-26 18:05:28] ...trap/engines/engine.py.230: loss: 0.29173 [I 2022-10-26 18:05:28] ...optimizers/trijoint.py.151: total_norm: 25.299231 [I 2022-10-26 18:05:34] ...trap/engines/engine.py.224: train: epoch 22 | batch 1480/2304 [I 2022-10-26 18:05:34] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:54 [I 2022-10-26 18:05:34] ...trap/engines/engine.py.229: process: 0.66035 | load: 0.00250 [I 2022-10-26 18:05:34] ...trap/engines/engine.py.230: loss: 0.17741 [I 2022-10-26 18:05:34] ...optimizers/trijoint.py.151: total_norm: 19.877560 [I 2022-10-26 18:05:40] ...trap/engines/engine.py.224: train: epoch 22 | batch 1490/2304 [I 2022-10-26 18:05:40] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:33 [I 2022-10-26 18:05:40] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00243 [I 2022-10-26 18:05:40] ...trap/engines/engine.py.230: loss: 0.12764 [I 2022-10-26 18:05:40] ...optimizers/trijoint.py.151: total_norm: 24.495275 [I 2022-10-26 18:05:45] ...trap/engines/engine.py.224: train: epoch 22 | batch 1500/2304 [I 2022-10-26 18:05:45] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:28 [I 2022-10-26 18:05:45] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00241 [I 2022-10-26 18:05:45] ...trap/engines/engine.py.230: loss: 0.21715 [I 2022-10-26 18:05:45] ...optimizers/trijoint.py.151: total_norm: 31.345951 [I 2022-10-26 18:05:51] ...trap/engines/engine.py.224: train: epoch 22 | batch 1510/2304 [I 2022-10-26 18:05:51] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:21 [I 2022-10-26 18:05:51] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00264 [I 2022-10-26 18:05:51] ...trap/engines/engine.py.230: loss: 0.13897 [I 2022-10-26 18:05:51] ...optimizers/trijoint.py.151: total_norm: 20.455931 [I 2022-10-26 18:05:56] ...trap/engines/engine.py.224: train: epoch 22 | batch 1520/2304 [I 2022-10-26 18:05:56] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:15 [I 2022-10-26 18:05:56] ...trap/engines/engine.py.229: process: 0.54936 | load: 0.00243 [I 2022-10-26 18:05:56] ...trap/engines/engine.py.230: loss: 0.25603 [I 2022-10-26 18:05:56] ...optimizers/trijoint.py.151: total_norm: 19.856861 [I 2022-10-26 18:06:02] ...trap/engines/engine.py.224: train: epoch 22 | batch 1530/2304 [I 2022-10-26 18:06:02] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:09 [I 2022-10-26 18:06:02] ...trap/engines/engine.py.229: process: 0.54703 | load: 0.00251 [I 2022-10-26 18:06:02] ...trap/engines/engine.py.230: loss: 0.15629 [I 2022-10-26 18:06:02] ...optimizers/trijoint.py.151: total_norm: 14.688451 [I 2022-10-26 18:06:07] ...trap/engines/engine.py.224: train: epoch 22 | batch 1540/2304 [I 2022-10-26 18:06:07] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:04 [I 2022-10-26 18:06:07] ...trap/engines/engine.py.229: process: 0.54099 | load: 0.00250 [I 2022-10-26 18:06:07] ...trap/engines/engine.py.230: loss: 0.18688 [I 2022-10-26 18:06:07] ...optimizers/trijoint.py.151: total_norm: 21.789854 [I 2022-10-26 18:06:13] ...trap/engines/engine.py.224: train: epoch 22 | batch 1550/2304 [I 2022-10-26 18:06:13] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:11 [I 2022-10-26 18:06:13] ...trap/engines/engine.py.229: process: 0.54325 | load: 0.00251 [I 2022-10-26 18:06:13] ...trap/engines/engine.py.230: loss: 0.18027 [I 2022-10-26 18:06:13] ...optimizers/trijoint.py.151: total_norm: 15.159116 [I 2022-10-26 18:06:19] ...trap/engines/engine.py.224: train: epoch 22 | batch 1560/2304 [I 2022-10-26 18:06:19] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:06:56 [I 2022-10-26 18:06:19] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00250 [I 2022-10-26 18:06:19] ...trap/engines/engine.py.230: loss: 0.21110 [I 2022-10-26 18:06:19] ...optimizers/trijoint.py.151: total_norm: 18.381184 [I 2022-10-26 18:06:24] ...trap/engines/engine.py.224: train: epoch 22 | batch 1570/2304 [I 2022-10-26 18:06:24] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:49 [I 2022-10-26 18:06:24] ...trap/engines/engine.py.229: process: 0.56783 | load: 0.00249 [I 2022-10-26 18:06:24] ...trap/engines/engine.py.230: loss: 0.17807 [I 2022-10-26 18:06:24] ...optimizers/trijoint.py.151: total_norm: 26.839647 [I 2022-10-26 18:06:30] ...trap/engines/engine.py.224: train: epoch 22 | batch 1580/2304 [I 2022-10-26 18:06:30] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:44 [I 2022-10-26 18:06:30] ...trap/engines/engine.py.229: process: 0.55197 | load: 0.00251 [I 2022-10-26 18:06:30] ...trap/engines/engine.py.230: loss: 0.26458 [I 2022-10-26 18:06:30] ...optimizers/trijoint.py.151: total_norm: 32.499065 [I 2022-10-26 18:06:35] ...trap/engines/engine.py.224: train: epoch 22 | batch 1590/2304 [I 2022-10-26 18:06:35] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:37 [I 2022-10-26 18:06:35] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00267 [I 2022-10-26 18:06:35] ...trap/engines/engine.py.230: loss: 0.23064 [I 2022-10-26 18:06:35] ...optimizers/trijoint.py.151: total_norm: 23.835766 [I 2022-10-26 18:06:41] ...trap/engines/engine.py.224: train: epoch 22 | batch 1600/2304 [I 2022-10-26 18:06:41] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:31 [I 2022-10-26 18:06:41] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00243 [I 2022-10-26 18:06:41] ...trap/engines/engine.py.230: loss: 0.29832 [I 2022-10-26 18:06:41] ...optimizers/trijoint.py.151: total_norm: 23.034658 [I 2022-10-26 18:06:47] ...trap/engines/engine.py.224: train: epoch 22 | batch 1610/2304 [I 2022-10-26 18:06:47] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:32 [I 2022-10-26 18:06:47] ...trap/engines/engine.py.229: process: 0.62121 | load: 0.00249 [I 2022-10-26 18:06:47] ...trap/engines/engine.py.230: loss: 0.17734 [I 2022-10-26 18:06:47] ...optimizers/trijoint.py.151: total_norm: 17.829163 [I 2022-10-26 18:06:52] ...trap/engines/engine.py.224: train: epoch 22 | batch 1620/2304 [I 2022-10-26 18:06:52] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:17 [I 2022-10-26 18:06:52] ...trap/engines/engine.py.229: process: 0.53476 | load: 0.00245 [I 2022-10-26 18:06:52] ...trap/engines/engine.py.230: loss: 0.21124 [I 2022-10-26 18:06:52] ...optimizers/trijoint.py.151: total_norm: 21.555019 [I 2022-10-26 18:06:58] ...trap/engines/engine.py.224: train: epoch 22 | batch 1630/2304 [I 2022-10-26 18:06:58] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:13 [I 2022-10-26 18:06:58] ...trap/engines/engine.py.229: process: 0.54542 | load: 0.00247 [I 2022-10-26 18:06:58] ...trap/engines/engine.py.230: loss: 0.13603 [I 2022-10-26 18:06:58] ...optimizers/trijoint.py.151: total_norm: 16.751965 [I 2022-10-26 18:07:03] ...trap/engines/engine.py.224: train: epoch 22 | batch 1640/2304 [I 2022-10-26 18:07:03] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:07 [I 2022-10-26 18:07:03] ...trap/engines/engine.py.229: process: 0.55345 | load: 0.00244 [I 2022-10-26 18:07:03] ...trap/engines/engine.py.230: loss: 0.18985 [I 2022-10-26 18:07:03] ...optimizers/trijoint.py.151: total_norm: 14.136499 [I 2022-10-26 18:07:09] ...trap/engines/engine.py.224: train: epoch 22 | batch 1650/2304 [I 2022-10-26 18:07:09] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:02 [I 2022-10-26 18:07:09] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00250 [I 2022-10-26 18:07:09] ...trap/engines/engine.py.230: loss: 0.22733 [I 2022-10-26 18:07:09] ...optimizers/trijoint.py.151: total_norm: 31.054998 [I 2022-10-26 18:07:14] ...trap/engines/engine.py.224: train: epoch 22 | batch 1660/2304 [I 2022-10-26 18:07:14] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:01 [I 2022-10-26 18:07:14] ...trap/engines/engine.py.229: process: 0.57822 | load: 0.00245 [I 2022-10-26 18:07:14] ...trap/engines/engine.py.230: loss: 0.15413 [I 2022-10-26 18:07:14] ...optimizers/trijoint.py.151: total_norm: 20.473600 [I 2022-10-26 18:07:20] ...trap/engines/engine.py.224: train: epoch 22 | batch 1670/2304 [I 2022-10-26 18:07:20] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:05:53 [I 2022-10-26 18:07:20] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00255 [I 2022-10-26 18:07:20] ...trap/engines/engine.py.230: loss: 0.20785 [I 2022-10-26 18:07:20] ...optimizers/trijoint.py.151: total_norm: 18.081932 [I 2022-10-26 18:07:26] ...trap/engines/engine.py.224: train: epoch 22 | batch 1680/2304 [I 2022-10-26 18:07:26] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:49 [I 2022-10-26 18:07:26] ...trap/engines/engine.py.229: process: 0.56942 | load: 0.00247 [I 2022-10-26 18:07:26] ...trap/engines/engine.py.230: loss: 0.26731 [I 2022-10-26 18:07:26] ...optimizers/trijoint.py.151: total_norm: 35.560936 [I 2022-10-26 18:07:31] ...trap/engines/engine.py.224: train: epoch 22 | batch 1690/2304 [I 2022-10-26 18:07:31] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:42 [I 2022-10-26 18:07:31] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00253 [I 2022-10-26 18:07:31] ...trap/engines/engine.py.230: loss: 0.22105 [I 2022-10-26 18:07:31] ...optimizers/trijoint.py.151: total_norm: 19.925550 [I 2022-10-26 18:07:37] ...trap/engines/engine.py.224: train: epoch 22 | batch 1700/2304 [I 2022-10-26 18:07:37] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:33 [I 2022-10-26 18:07:37] ...trap/engines/engine.py.229: process: 0.54802 | load: 0.00269 [I 2022-10-26 18:07:37] ...trap/engines/engine.py.230: loss: 0.17820 [I 2022-10-26 18:07:37] ...optimizers/trijoint.py.151: total_norm: 25.924364 [I 2022-10-26 18:07:42] ...trap/engines/engine.py.224: train: epoch 22 | batch 1710/2304 [I 2022-10-26 18:07:42] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:30 [I 2022-10-26 18:07:42] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00248 [I 2022-10-26 18:07:42] ...trap/engines/engine.py.230: loss: 0.18523 [I 2022-10-26 18:07:42] ...optimizers/trijoint.py.151: total_norm: 21.344448 [I 2022-10-26 18:07:48] ...trap/engines/engine.py.224: train: epoch 22 | batch 1720/2304 [I 2022-10-26 18:07:48] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:27 [I 2022-10-26 18:07:48] ...trap/engines/engine.py.229: process: 0.56829 | load: 0.00259 [I 2022-10-26 18:07:48] ...trap/engines/engine.py.230: loss: 0.23858 [I 2022-10-26 18:07:48] ...optimizers/trijoint.py.151: total_norm: 18.721729 [I 2022-10-26 18:07:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 1730/2304 [I 2022-10-26 18:07:54] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:19 [I 2022-10-26 18:07:54] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00252 [I 2022-10-26 18:07:54] ...trap/engines/engine.py.230: loss: 0.17997 [I 2022-10-26 18:07:54] ...optimizers/trijoint.py.151: total_norm: 17.475010 [I 2022-10-26 18:07:59] ...trap/engines/engine.py.224: train: epoch 22 | batch 1740/2304 [I 2022-10-26 18:07:59] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:23 [I 2022-10-26 18:07:59] ...trap/engines/engine.py.229: process: 0.55688 | load: 0.00245 [I 2022-10-26 18:07:59] ...trap/engines/engine.py.230: loss: 0.13946 [I 2022-10-26 18:07:59] ...optimizers/trijoint.py.151: total_norm: 14.938904 [I 2022-10-26 18:08:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 1750/2304 [I 2022-10-26 18:08:05] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:10 [I 2022-10-26 18:08:05] ...trap/engines/engine.py.229: process: 0.56573 | load: 0.00252 [I 2022-10-26 18:08:05] ...trap/engines/engine.py.230: loss: 0.24179 [I 2022-10-26 18:08:05] ...optimizers/trijoint.py.151: total_norm: 30.279181 [I 2022-10-26 18:08:10] ...trap/engines/engine.py.224: train: epoch 22 | batch 1760/2304 [I 2022-10-26 18:08:10] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:04 [I 2022-10-26 18:08:10] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00261 [I 2022-10-26 18:08:10] ...trap/engines/engine.py.230: loss: 0.17803 [I 2022-10-26 18:08:10] ...optimizers/trijoint.py.151: total_norm: 14.925784 [I 2022-10-26 18:08:16] ...trap/engines/engine.py.224: train: epoch 22 | batch 1770/2304 [I 2022-10-26 18:08:16] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:04:56 [I 2022-10-26 18:08:16] ...trap/engines/engine.py.229: process: 0.54910 | load: 0.00248 [I 2022-10-26 18:08:16] ...trap/engines/engine.py.230: loss: 0.24530 [I 2022-10-26 18:08:16] ...optimizers/trijoint.py.151: total_norm: 38.563370 [I 2022-10-26 18:08:43] ...trap/engines/engine.py.224: train: epoch 22 | batch 1780/2304 [I 2022-10-26 18:08:43] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:28:12 [I 2022-10-26 18:08:43] ...trap/engines/engine.py.229: process: 0.56555 | load: 0.00239 [I 2022-10-26 18:08:43] ...trap/engines/engine.py.230: loss: 0.20088 [I 2022-10-26 18:08:43] ...optimizers/trijoint.py.151: total_norm: 17.917688 [I 2022-10-26 18:08:48] ...trap/engines/engine.py.224: train: epoch 22 | batch 1790/2304 [I 2022-10-26 18:08:48] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:07:13 [I 2022-10-26 18:08:48] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00238 [I 2022-10-26 18:08:48] ...trap/engines/engine.py.230: loss: 0.23208 [I 2022-10-26 18:08:48] ...optimizers/trijoint.py.151: total_norm: 19.294170 [I 2022-10-26 18:08:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 1800/2304 [I 2022-10-26 18:08:54] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:57 [I 2022-10-26 18:08:54] ...trap/engines/engine.py.229: process: 0.57438 | load: 0.00237 [I 2022-10-26 18:08:54] ...trap/engines/engine.py.230: loss: 0.18110 [I 2022-10-26 18:08:54] ...optimizers/trijoint.py.151: total_norm: 17.690653 [I 2022-10-26 18:08:59] ...trap/engines/engine.py.224: train: epoch 22 | batch 1810/2304 [I 2022-10-26 18:08:59] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:35 [I 2022-10-26 18:08:59] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00236 [I 2022-10-26 18:08:59] ...trap/engines/engine.py.230: loss: 0.22538 [I 2022-10-26 18:08:59] ...optimizers/trijoint.py.151: total_norm: 34.001984 [I 2022-10-26 18:09:05] ...trap/engines/engine.py.224: train: epoch 22 | batch 1820/2304 [I 2022-10-26 18:09:05] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:29 [I 2022-10-26 18:09:05] ...trap/engines/engine.py.229: process: 0.56264 | load: 0.00245 [I 2022-10-26 18:09:05] ...trap/engines/engine.py.230: loss: 0.18047 [I 2022-10-26 18:09:05] ...optimizers/trijoint.py.151: total_norm: 29.095709 [I 2022-10-26 18:09:10] ...trap/engines/engine.py.224: train: epoch 22 | batch 1830/2304 [I 2022-10-26 18:09:10] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:25 [I 2022-10-26 18:09:10] ...trap/engines/engine.py.229: process: 0.56502 | load: 0.00241 [I 2022-10-26 18:09:10] ...trap/engines/engine.py.230: loss: 0.18673 [I 2022-10-26 18:09:10] ...optimizers/trijoint.py.151: total_norm: 18.717274 [I 2022-10-26 18:09:16] ...trap/engines/engine.py.224: train: epoch 22 | batch 1840/2304 [I 2022-10-26 18:09:16] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:04:20 [I 2022-10-26 18:09:16] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00246 [I 2022-10-26 18:09:16] ...trap/engines/engine.py.230: loss: 0.13465 [I 2022-10-26 18:09:16] ...optimizers/trijoint.py.151: total_norm: 14.711226 [I 2022-10-26 18:09:22] ...trap/engines/engine.py.224: train: epoch 22 | batch 1850/2304 [I 2022-10-26 18:09:22] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:04:14 [I 2022-10-26 18:09:22] ...trap/engines/engine.py.229: process: 0.56428 | load: 0.00244 [I 2022-10-26 18:09:22] ...trap/engines/engine.py.230: loss: 0.20227 [I 2022-10-26 18:09:22] ...optimizers/trijoint.py.151: total_norm: 20.256166 [I 2022-10-26 18:09:27] ...trap/engines/engine.py.224: train: epoch 22 | batch 1860/2304 [I 2022-10-26 18:09:27] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:04:07 [I 2022-10-26 18:09:27] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00245 [I 2022-10-26 18:09:27] ...trap/engines/engine.py.230: loss: 0.19928 [I 2022-10-26 18:09:27] ...optimizers/trijoint.py.151: total_norm: 23.264292 [I 2022-10-26 18:09:33] ...trap/engines/engine.py.224: train: epoch 22 | batch 1870/2304 [I 2022-10-26 18:09:33] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:04:03 [I 2022-10-26 18:09:33] ...trap/engines/engine.py.229: process: 0.57036 | load: 0.00248 [I 2022-10-26 18:09:33] ...trap/engines/engine.py.230: loss: 0.19389 [I 2022-10-26 18:09:33] ...optimizers/trijoint.py.151: total_norm: 20.127897 [I 2022-10-26 18:09:38] ...trap/engines/engine.py.224: train: epoch 22 | batch 1880/2304 [I 2022-10-26 18:09:38] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:57 [I 2022-10-26 18:09:38] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00253 [I 2022-10-26 18:09:38] ...trap/engines/engine.py.230: loss: 0.15549 [I 2022-10-26 18:09:38] ...optimizers/trijoint.py.151: total_norm: 23.393217 [I 2022-10-26 18:09:44] ...trap/engines/engine.py.224: train: epoch 22 | batch 1890/2304 [I 2022-10-26 18:09:44] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:52 [I 2022-10-26 18:09:44] ...trap/engines/engine.py.229: process: 0.56215 | load: 0.00249 [I 2022-10-26 18:09:44] ...trap/engines/engine.py.230: loss: 0.18757 [I 2022-10-26 18:09:44] ...optimizers/trijoint.py.151: total_norm: 23.026392 [I 2022-10-26 18:09:50] ...trap/engines/engine.py.224: train: epoch 22 | batch 1900/2304 [I 2022-10-26 18:09:50] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:44 [I 2022-10-26 18:09:50] ...trap/engines/engine.py.229: process: 0.55848 | load: 0.00288 [I 2022-10-26 18:09:50] ...trap/engines/engine.py.230: loss: 0.18547 [I 2022-10-26 18:09:50] ...optimizers/trijoint.py.151: total_norm: 15.219846 [I 2022-10-26 18:09:55] ...trap/engines/engine.py.224: train: epoch 22 | batch 1910/2304 [I 2022-10-26 18:09:55] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:41 [I 2022-10-26 18:09:55] ...trap/engines/engine.py.229: process: 0.56372 | load: 0.00268 [I 2022-10-26 18:09:55] ...trap/engines/engine.py.230: loss: 0.14927 [I 2022-10-26 18:09:55] ...optimizers/trijoint.py.151: total_norm: 22.914299 [I 2022-10-26 18:10:01] ...trap/engines/engine.py.224: train: epoch 22 | batch 1920/2304 [I 2022-10-26 18:10:01] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:35 [I 2022-10-26 18:10:01] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00268 [I 2022-10-26 18:10:01] ...trap/engines/engine.py.230: loss: 0.21316 [I 2022-10-26 18:10:01] ...optimizers/trijoint.py.151: total_norm: 34.971661 [I 2022-10-26 18:10:06] ...trap/engines/engine.py.224: train: epoch 22 | batch 1930/2304 [I 2022-10-26 18:10:06] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:27 [I 2022-10-26 18:10:06] ...trap/engines/engine.py.229: process: 0.55102 | load: 0.00278 [I 2022-10-26 18:10:06] ...trap/engines/engine.py.230: loss: 0.24243 [I 2022-10-26 18:10:06] ...optimizers/trijoint.py.151: total_norm: 19.701885 [I 2022-10-26 18:10:12] ...trap/engines/engine.py.224: train: epoch 22 | batch 1940/2304 [I 2022-10-26 18:10:12] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:22 [I 2022-10-26 18:10:12] ...trap/engines/engine.py.229: process: 0.54653 | load: 0.00269 [I 2022-10-26 18:10:12] ...trap/engines/engine.py.230: loss: 0.21325 [I 2022-10-26 18:10:12] ...optimizers/trijoint.py.151: total_norm: 18.367031 [I 2022-10-26 18:10:18] ...trap/engines/engine.py.224: train: epoch 22 | batch 1950/2304 [I 2022-10-26 18:10:18] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:18 [I 2022-10-26 18:10:18] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00341 [I 2022-10-26 18:10:18] ...trap/engines/engine.py.230: loss: 0.20059 [I 2022-10-26 18:10:18] ...optimizers/trijoint.py.151: total_norm: 17.167543 [I 2022-10-26 18:10:23] ...trap/engines/engine.py.224: train: epoch 22 | batch 1960/2304 [I 2022-10-26 18:10:23] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:03:12 [I 2022-10-26 18:10:23] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00281 [I 2022-10-26 18:10:23] ...trap/engines/engine.py.230: loss: 0.25479 [I 2022-10-26 18:10:23] ...optimizers/trijoint.py.151: total_norm: 19.810366 [I 2022-10-26 18:10:29] ...trap/engines/engine.py.224: train: epoch 22 | batch 1970/2304 [I 2022-10-26 18:10:29] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:03:08 [I 2022-10-26 18:10:29] ...trap/engines/engine.py.229: process: 0.56566 | load: 0.00288 [I 2022-10-26 18:10:29] ...trap/engines/engine.py.230: loss: 0.20195 [I 2022-10-26 18:10:29] ...optimizers/trijoint.py.151: total_norm: 21.609653 [I 2022-10-26 18:10:35] ...trap/engines/engine.py.224: train: epoch 22 | batch 1980/2304 [I 2022-10-26 18:10:35] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:03:01 [I 2022-10-26 18:10:35] ...trap/engines/engine.py.229: process: 0.56696 | load: 0.00275 [I 2022-10-26 18:10:35] ...trap/engines/engine.py.230: loss: 0.22435 [I 2022-10-26 18:10:35] ...optimizers/trijoint.py.151: total_norm: 23.666430 [I 2022-10-26 18:10:40] ...trap/engines/engine.py.224: train: epoch 22 | batch 1990/2304 [I 2022-10-26 18:10:40] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:55 [I 2022-10-26 18:10:40] ...trap/engines/engine.py.229: process: 0.56153 | load: 0.00262 [I 2022-10-26 18:10:40] ...trap/engines/engine.py.230: loss: 0.19046 [I 2022-10-26 18:10:40] ...optimizers/trijoint.py.151: total_norm: 17.486383 [I 2022-10-26 18:10:46] ...trap/engines/engine.py.224: train: epoch 22 | batch 2000/2304 [I 2022-10-26 18:10:46] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:49 [I 2022-10-26 18:10:46] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00272 [I 2022-10-26 18:10:46] ...trap/engines/engine.py.230: loss: 0.14988 [I 2022-10-26 18:10:46] ...optimizers/trijoint.py.151: total_norm: 20.965347 [I 2022-10-26 18:10:51] ...trap/engines/engine.py.224: train: epoch 22 | batch 2010/2304 [I 2022-10-26 18:10:51] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:42 [I 2022-10-26 18:10:51] ...trap/engines/engine.py.229: process: 0.55093 | load: 0.00264 [I 2022-10-26 18:10:51] ...trap/engines/engine.py.230: loss: 0.16805 [I 2022-10-26 18:10:51] ...optimizers/trijoint.py.151: total_norm: 16.490959 [I 2022-10-26 18:10:57] ...trap/engines/engine.py.224: train: epoch 22 | batch 2020/2304 [I 2022-10-26 18:10:57] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:39 [I 2022-10-26 18:10:57] ...trap/engines/engine.py.229: process: 0.58648 | load: 0.00254 [I 2022-10-26 18:10:57] ...trap/engines/engine.py.230: loss: 0.30648 [I 2022-10-26 18:10:57] ...optimizers/trijoint.py.151: total_norm: 31.906866 [I 2022-10-26 18:11:03] ...trap/engines/engine.py.224: train: epoch 22 | batch 2030/2304 [I 2022-10-26 18:11:03] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:32 [I 2022-10-26 18:11:03] ...trap/engines/engine.py.229: process: 0.56422 | load: 0.00279 [I 2022-10-26 18:11:03] ...trap/engines/engine.py.230: loss: 0.21104 [I 2022-10-26 18:11:03] ...optimizers/trijoint.py.151: total_norm: 18.537588 [I 2022-10-26 18:11:08] ...trap/engines/engine.py.224: train: epoch 22 | batch 2040/2304 [I 2022-10-26 18:11:08] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:27 [I 2022-10-26 18:11:08] ...trap/engines/engine.py.229: process: 0.57154 | load: 0.00252 [I 2022-10-26 18:11:08] ...trap/engines/engine.py.230: loss: 0.24391 [I 2022-10-26 18:11:08] ...optimizers/trijoint.py.151: total_norm: 26.151361 [I 2022-10-26 18:11:14] ...trap/engines/engine.py.224: train: epoch 22 | batch 2050/2304 [I 2022-10-26 18:11:14] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:21 [I 2022-10-26 18:11:14] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00249 [I 2022-10-26 18:11:14] ...trap/engines/engine.py.230: loss: 0.15812 [I 2022-10-26 18:11:14] ...optimizers/trijoint.py.151: total_norm: 24.604744 [I 2022-10-26 18:11:19] ...trap/engines/engine.py.224: train: epoch 22 | batch 2060/2304 [I 2022-10-26 18:11:19] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:02:16 [I 2022-10-26 18:11:19] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00257 [I 2022-10-26 18:11:19] ...trap/engines/engine.py.230: loss: 0.16137 [I 2022-10-26 18:11:19] ...optimizers/trijoint.py.151: total_norm: 18.252935 [I 2022-10-26 18:11:25] ...trap/engines/engine.py.224: train: epoch 22 | batch 2070/2304 [I 2022-10-26 18:11:25] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:02:10 [I 2022-10-26 18:11:25] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00261 [I 2022-10-26 18:11:25] ...trap/engines/engine.py.230: loss: 0.24699 [I 2022-10-26 18:11:25] ...optimizers/trijoint.py.151: total_norm: 20.241808 [I 2022-10-26 18:11:30] ...trap/engines/engine.py.224: train: epoch 22 | batch 2080/2304 [I 2022-10-26 18:11:30] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:02:05 [I 2022-10-26 18:11:30] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00266 [I 2022-10-26 18:11:31] ...trap/engines/engine.py.230: loss: 0.26491 [I 2022-10-26 18:11:31] ...optimizers/trijoint.py.151: total_norm: 26.138111 [I 2022-10-26 18:11:36] ...trap/engines/engine.py.224: train: epoch 22 | batch 2090/2304 [I 2022-10-26 18:11:36] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:58 [I 2022-10-26 18:11:36] ...trap/engines/engine.py.229: process: 0.55594 | load: 0.00250 [I 2022-10-26 18:11:36] ...trap/engines/engine.py.230: loss: 0.20564 [I 2022-10-26 18:11:36] ...optimizers/trijoint.py.151: total_norm: 16.772165 [I 2022-10-26 18:11:42] ...trap/engines/engine.py.224: train: epoch 22 | batch 2100/2304 [I 2022-10-26 18:11:42] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:54 [I 2022-10-26 18:11:42] ...trap/engines/engine.py.229: process: 0.58760 | load: 0.00258 [I 2022-10-26 18:11:42] ...trap/engines/engine.py.230: loss: 0.22340 [I 2022-10-26 18:11:42] ...optimizers/trijoint.py.151: total_norm: 17.882160 [I 2022-10-26 18:11:47] ...trap/engines/engine.py.224: train: epoch 22 | batch 2110/2304 [I 2022-10-26 18:11:47] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:47 [I 2022-10-26 18:11:47] ...trap/engines/engine.py.229: process: 0.54562 | load: 0.00252 [I 2022-10-26 18:11:47] ...trap/engines/engine.py.230: loss: 0.25411 [I 2022-10-26 18:11:47] ...optimizers/trijoint.py.151: total_norm: 23.925278 [I 2022-10-26 18:11:53] ...trap/engines/engine.py.224: train: epoch 22 | batch 2120/2304 [I 2022-10-26 18:11:53] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:43 [I 2022-10-26 18:11:53] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00260 [I 2022-10-26 18:11:53] ...trap/engines/engine.py.230: loss: 0.20632 [I 2022-10-26 18:11:53] ...optimizers/trijoint.py.151: total_norm: 22.333601 [I 2022-10-26 18:11:58] ...trap/engines/engine.py.224: train: epoch 22 | batch 2130/2304 [I 2022-10-26 18:11:58] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:37 [I 2022-10-26 18:11:58] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00268 [I 2022-10-26 18:11:58] ...trap/engines/engine.py.230: loss: 0.17216 [I 2022-10-26 18:11:58] ...optimizers/trijoint.py.151: total_norm: 16.450811 [I 2022-10-26 18:12:04] ...trap/engines/engine.py.224: train: epoch 22 | batch 2140/2304 [I 2022-10-26 18:12:04] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:32 [I 2022-10-26 18:12:04] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00261 [I 2022-10-26 18:12:04] ...trap/engines/engine.py.230: loss: 0.20046 [I 2022-10-26 18:12:04] ...optimizers/trijoint.py.151: total_norm: 17.770746 [I 2022-10-26 18:12:10] ...trap/engines/engine.py.224: train: epoch 22 | batch 2150/2304 [I 2022-10-26 18:12:10] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:26 [I 2022-10-26 18:12:10] ...trap/engines/engine.py.229: process: 0.55101 | load: 0.00247 [I 2022-10-26 18:12:10] ...trap/engines/engine.py.230: loss: 0.18676 [I 2022-10-26 18:12:10] ...optimizers/trijoint.py.151: total_norm: 20.607134 [I 2022-10-26 18:12:15] ...trap/engines/engine.py.224: train: epoch 22 | batch 2160/2304 [I 2022-10-26 18:12:15] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:21 [I 2022-10-26 18:12:15] ...trap/engines/engine.py.229: process: 0.55482 | load: 0.00261 [I 2022-10-26 18:12:15] ...trap/engines/engine.py.230: loss: 0.18333 [I 2022-10-26 18:12:15] ...optimizers/trijoint.py.151: total_norm: 17.630674 [I 2022-10-26 18:12:21] ...trap/engines/engine.py.224: train: epoch 22 | batch 2170/2304 [I 2022-10-26 18:12:21] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:14 [I 2022-10-26 18:12:21] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00253 [I 2022-10-26 18:12:21] ...trap/engines/engine.py.230: loss: 0.20298 [I 2022-10-26 18:12:21] ...optimizers/trijoint.py.151: total_norm: 26.416138 [I 2022-10-26 18:12:27] ...trap/engines/engine.py.224: train: epoch 22 | batch 2180/2304 [I 2022-10-26 18:12:27] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:09 [I 2022-10-26 18:12:27] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00255 [I 2022-10-26 18:12:27] ...trap/engines/engine.py.230: loss: 0.13870 [I 2022-10-26 18:12:27] ...optimizers/trijoint.py.151: total_norm: 38.207806 [I 2022-10-26 18:12:32] ...trap/engines/engine.py.224: train: epoch 22 | batch 2190/2304 [I 2022-10-26 18:12:32] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:01:03 [I 2022-10-26 18:12:32] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00268 [I 2022-10-26 18:12:32] ...trap/engines/engine.py.230: loss: 0.20117 [I 2022-10-26 18:12:32] ...optimizers/trijoint.py.151: total_norm: 21.204494 [I 2022-10-26 18:12:38] ...trap/engines/engine.py.224: train: epoch 22 | batch 2200/2304 [I 2022-10-26 18:12:38] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:57 [I 2022-10-26 18:12:38] ...trap/engines/engine.py.229: process: 0.54446 | load: 0.00256 [I 2022-10-26 18:12:38] ...trap/engines/engine.py.230: loss: 0.20997 [I 2022-10-26 18:12:38] ...optimizers/trijoint.py.151: total_norm: 20.120377 [I 2022-10-26 18:12:43] ...trap/engines/engine.py.224: train: epoch 22 | batch 2210/2304 [I 2022-10-26 18:12:43] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:52 [I 2022-10-26 18:12:43] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00254 [I 2022-10-26 18:12:43] ...trap/engines/engine.py.230: loss: 0.20971 [I 2022-10-26 18:12:43] ...optimizers/trijoint.py.151: total_norm: 24.195570 [I 2022-10-26 18:12:49] ...trap/engines/engine.py.224: train: epoch 22 | batch 2220/2304 [I 2022-10-26 18:12:49] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:46 [I 2022-10-26 18:12:49] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00263 [I 2022-10-26 18:12:49] ...trap/engines/engine.py.230: loss: 0.19355 [I 2022-10-26 18:12:49] ...optimizers/trijoint.py.151: total_norm: 24.589794 [I 2022-10-26 18:12:54] ...trap/engines/engine.py.224: train: epoch 22 | batch 2230/2304 [I 2022-10-26 18:12:54] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:41 [I 2022-10-26 18:12:54] ...trap/engines/engine.py.229: process: 0.54480 | load: 0.00258 [I 2022-10-26 18:12:54] ...trap/engines/engine.py.230: loss: 0.21837 [I 2022-10-26 18:12:54] ...optimizers/trijoint.py.151: total_norm: 29.016571 [I 2022-10-26 18:13:00] ...trap/engines/engine.py.224: train: epoch 22 | batch 2240/2304 [I 2022-10-26 18:13:00] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:35 [I 2022-10-26 18:13:00] ...trap/engines/engine.py.229: process: 0.55044 | load: 0.00259 [I 2022-10-26 18:13:00] ...trap/engines/engine.py.230: loss: 0.23112 [I 2022-10-26 18:13:00] ...optimizers/trijoint.py.151: total_norm: 20.842115 [I 2022-10-26 18:13:06] ...trap/engines/engine.py.224: train: epoch 22 | batch 2250/2304 [I 2022-10-26 18:13:06] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:30 [I 2022-10-26 18:13:06] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00261 [I 2022-10-26 18:13:06] ...trap/engines/engine.py.230: loss: 0.16968 [I 2022-10-26 18:13:06] ...optimizers/trijoint.py.151: total_norm: 17.483429 [I 2022-10-26 18:13:11] ...trap/engines/engine.py.224: train: epoch 22 | batch 2260/2304 [I 2022-10-26 18:13:11] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:24 [I 2022-10-26 18:13:11] ...trap/engines/engine.py.229: process: 0.54060 | load: 0.00252 [I 2022-10-26 18:13:11] ...trap/engines/engine.py.230: loss: 0.19563 [I 2022-10-26 18:13:11] ...optimizers/trijoint.py.151: total_norm: 26.079039 [I 2022-10-26 18:13:17] ...trap/engines/engine.py.224: train: epoch 22 | batch 2270/2304 [I 2022-10-26 18:13:17] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:18 [I 2022-10-26 18:13:17] ...trap/engines/engine.py.229: process: 0.55490 | load: 0.00258 [I 2022-10-26 18:13:17] ...trap/engines/engine.py.230: loss: 0.21619 [I 2022-10-26 18:13:17] ...optimizers/trijoint.py.151: total_norm: 14.657534 [I 2022-10-26 18:13:22] ...trap/engines/engine.py.224: train: epoch 22 | batch 2280/2304 [I 2022-10-26 18:13:22] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:13 [I 2022-10-26 18:13:22] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00249 [I 2022-10-26 18:13:22] ...trap/engines/engine.py.230: loss: 0.25601 [I 2022-10-26 18:13:22] ...optimizers/trijoint.py.151: total_norm: 18.390532 [I 2022-10-26 18:13:28] ...trap/engines/engine.py.224: train: epoch 22 | batch 2290/2304 [I 2022-10-26 18:13:28] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:00:07 [I 2022-10-26 18:13:28] ...trap/engines/engine.py.229: process: 0.55799 | load: 0.00248 [I 2022-10-26 18:13:28] ...trap/engines/engine.py.230: loss: 0.25011 [I 2022-10-26 18:13:28] ...optimizers/trijoint.py.151: total_norm: 21.893114 [I 2022-10-26 18:13:33] ...trap/engines/engine.py.224: train: epoch 22 | batch 2300/2304 [I 2022-10-26 18:13:33] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:02 [I 2022-10-26 18:13:33] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00014 [I 2022-10-26 18:13:33] ...trap/engines/engine.py.230: loss: 0.26346 [I 2022-10-26 18:13:33] ...optimizers/trijoint.py.151: total_norm: 28.654043 [I 2022-10-26 18:13:36] ...trap/engines/engine.py.224: train: epoch 22 | batch 2304/2304 [I 2022-10-26 18:13:36] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:00 [I 2022-10-26 18:13:36] ...trap/engines/engine.py.229: process: 0.54917 | load: 0.00015 [I 2022-10-26 18:13:36] ...trap/engines/engine.py.230: loss: 0.21205 [I 2022-10-26 18:13:36] ...optimizers/trijoint.py.151: total_norm: 23.607416 [S 2022-10-26 18:13:37] ...trap/engines/engine.py.129: train_epoch.epoch: 22 [S 2022-10-26 18:13:37] ...trap/engines/engine.py.129: train_epoch.loss: 0.1990387242101537 [I 2022-10-26 18:13:37] ...s/criterions/triplet.py.84: margin = 0.1650000000000001 [I 2022-10-26 18:13:37] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 22 [I 2022-10-26 18:13:37] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 18:13:39] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 18:13:39] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 18:13:39] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 18:13:40] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 18:13:40] ...trap/engines/engine.py.321: eval: epoch 22 | batch 0/510 [I 2022-10-26 18:13:40] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:24:04 [I 2022-10-26 18:13:40] ...trap/engines/engine.py.326: process: 2.83195 | load: 2.44230 [I 2022-10-26 18:13:43] ...trap/engines/engine.py.321: eval: epoch 22 | batch 10/510 [I 2022-10-26 18:13:43] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:04:58 [I 2022-10-26 18:13:43] ...trap/engines/engine.py.326: process: 0.32428 | load: 0.00016 [I 2022-10-26 18:13:47] ...trap/engines/engine.py.321: eval: epoch 22 | batch 20/510 [I 2022-10-26 18:13:47] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:57 [I 2022-10-26 18:13:47] ...trap/engines/engine.py.326: process: 0.33320 | load: 0.00017 [I 2022-10-26 18:13:50] ...trap/engines/engine.py.321: eval: epoch 22 | batch 30/510 [I 2022-10-26 18:13:50] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:02:44 [I 2022-10-26 18:13:50] ...trap/engines/engine.py.326: process: 0.32164 | load: 0.00019 [I 2022-10-26 18:13:53] ...trap/engines/engine.py.321: eval: epoch 22 | batch 40/510 [I 2022-10-26 18:13:53] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:38 [I 2022-10-26 18:13:53] ...trap/engines/engine.py.326: process: 0.31814 | load: 0.00019 [I 2022-10-26 18:13:57] ...trap/engines/engine.py.321: eval: epoch 22 | batch 50/510 [I 2022-10-26 18:13:57] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-26 18:13:57] ...trap/engines/engine.py.326: process: 0.34487 | load: 0.00016 [I 2022-10-26 18:14:00] ...trap/engines/engine.py.321: eval: epoch 22 | batch 60/510 [I 2022-10-26 18:14:00] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:31 [I 2022-10-26 18:14:00] ...trap/engines/engine.py.326: process: 0.32920 | load: 0.00018 [I 2022-10-26 18:14:04] ...trap/engines/engine.py.321: eval: epoch 22 | batch 70/510 [I 2022-10-26 18:14:04] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:29 [I 2022-10-26 18:14:04] ...trap/engines/engine.py.326: process: 0.35719 | load: 0.00017 [I 2022-10-26 18:14:07] ...trap/engines/engine.py.321: eval: epoch 22 | batch 80/510 [I 2022-10-26 18:14:07] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:26 [I 2022-10-26 18:14:07] ...trap/engines/engine.py.326: process: 0.36111 | load: 0.00020 [I 2022-10-26 18:14:10] ...trap/engines/engine.py.321: eval: epoch 22 | batch 90/510 [I 2022-10-26 18:14:10] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:24 [I 2022-10-26 18:14:10] ...trap/engines/engine.py.326: process: 0.35439 | load: 0.00016 [I 2022-10-26 18:14:14] ...trap/engines/engine.py.321: eval: epoch 22 | batch 100/510 [I 2022-10-26 18:14:14] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 18:14:14] ...trap/engines/engine.py.326: process: 0.29753 | load: 0.00036 [I 2022-10-26 18:14:17] ...trap/engines/engine.py.321: eval: epoch 22 | batch 110/510 [I 2022-10-26 18:14:17] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 18:14:17] ...trap/engines/engine.py.326: process: 0.29195 | load: 0.00017 [I 2022-10-26 18:14:19] ...trap/engines/engine.py.321: eval: epoch 22 | batch 120/510 [I 2022-10-26 18:14:19] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-26 18:14:19] ...trap/engines/engine.py.326: process: 0.28675 | load: 0.00023 [I 2022-10-26 18:14:22] ...trap/engines/engine.py.321: eval: epoch 22 | batch 130/510 [I 2022-10-26 18:14:22] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-26 18:14:22] ...trap/engines/engine.py.326: process: 0.30818 | load: 0.00016 [I 2022-10-26 18:14:25] ...trap/engines/engine.py.321: eval: epoch 22 | batch 140/510 [I 2022-10-26 18:14:25] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:48 [I 2022-10-26 18:14:25] ...trap/engines/engine.py.326: process: 0.28470 | load: 0.00015 [I 2022-10-26 18:14:28] ...trap/engines/engine.py.321: eval: epoch 22 | batch 150/510 [I 2022-10-26 18:14:28] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 18:14:28] ...trap/engines/engine.py.326: process: 0.28585 | load: 0.00014 [I 2022-10-26 18:14:31] ...trap/engines/engine.py.321: eval: epoch 22 | batch 160/510 [I 2022-10-26 18:14:31] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 18:14:31] ...trap/engines/engine.py.326: process: 0.29157 | load: 0.00017 [I 2022-10-26 18:14:34] ...trap/engines/engine.py.321: eval: epoch 22 | batch 170/510 [I 2022-10-26 18:14:34] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 18:14:34] ...trap/engines/engine.py.326: process: 0.29270 | load: 0.00017 [I 2022-10-26 18:14:37] ...trap/engines/engine.py.321: eval: epoch 22 | batch 180/510 [I 2022-10-26 18:14:37] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:36 [I 2022-10-26 18:14:37] ...trap/engines/engine.py.326: process: 0.29030 | load: 0.00015 [I 2022-10-26 18:14:40] ...trap/engines/engine.py.321: eval: epoch 22 | batch 190/510 [I 2022-10-26 18:14:40] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:33 [I 2022-10-26 18:14:40] ...trap/engines/engine.py.326: process: 0.28744 | load: 0.00016 [I 2022-10-26 18:14:43] ...trap/engines/engine.py.321: eval: epoch 22 | batch 200/510 [I 2022-10-26 18:14:43] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-26 18:14:43] ...trap/engines/engine.py.326: process: 0.30725 | load: 0.00015 [I 2022-10-26 18:14:46] ...trap/engines/engine.py.321: eval: epoch 22 | batch 210/510 [I 2022-10-26 18:14:46] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-26 18:14:46] ...trap/engines/engine.py.326: process: 0.28335 | load: 0.00018 [I 2022-10-26 18:14:49] ...trap/engines/engine.py.321: eval: epoch 22 | batch 220/510 [I 2022-10-26 18:14:49] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 18:14:49] ...trap/engines/engine.py.326: process: 0.28677 | load: 0.00016 [I 2022-10-26 18:14:51] ...trap/engines/engine.py.321: eval: epoch 22 | batch 230/510 [I 2022-10-26 18:14:51] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 18:14:51] ...trap/engines/engine.py.326: process: 0.28488 | load: 0.00019 [I 2022-10-26 18:14:54] ...trap/engines/engine.py.321: eval: epoch 22 | batch 240/510 [I 2022-10-26 18:14:54] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 18:14:54] ...trap/engines/engine.py.326: process: 0.29461 | load: 0.00022 [I 2022-10-26 18:14:57] ...trap/engines/engine.py.321: eval: epoch 22 | batch 250/510 [I 2022-10-26 18:14:57] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-26 18:14:57] ...trap/engines/engine.py.326: process: 0.28503 | load: 0.00015 [I 2022-10-26 18:15:00] ...trap/engines/engine.py.321: eval: epoch 22 | batch 260/510 [I 2022-10-26 18:15:00] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-26 18:15:00] ...trap/engines/engine.py.326: process: 0.28612 | load: 0.00018 [I 2022-10-26 18:15:03] ...trap/engines/engine.py.321: eval: epoch 22 | batch 270/510 [I 2022-10-26 18:15:03] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:10 [I 2022-10-26 18:15:03] ...trap/engines/engine.py.326: process: 0.31378 | load: 0.00015 [I 2022-10-26 18:15:06] ...trap/engines/engine.py.321: eval: epoch 22 | batch 280/510 [I 2022-10-26 18:15:06] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 18:15:06] ...trap/engines/engine.py.326: process: 0.28369 | load: 0.00041 [I 2022-10-26 18:15:09] ...trap/engines/engine.py.321: eval: epoch 22 | batch 290/510 [I 2022-10-26 18:15:09] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 18:15:09] ...trap/engines/engine.py.326: process: 0.29821 | load: 0.00014 [I 2022-10-26 18:15:12] ...trap/engines/engine.py.321: eval: epoch 22 | batch 300/510 [I 2022-10-26 18:15:12] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 18:15:12] ...trap/engines/engine.py.326: process: 0.28621 | load: 0.00018 [I 2022-10-26 18:15:15] ...trap/engines/engine.py.321: eval: epoch 22 | batch 310/510 [I 2022-10-26 18:15:15] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 18:15:15] ...trap/engines/engine.py.326: process: 0.28536 | load: 0.00019 [I 2022-10-26 18:15:18] ...trap/engines/engine.py.321: eval: epoch 22 | batch 320/510 [I 2022-10-26 18:15:18] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 18:15:18] ...trap/engines/engine.py.326: process: 0.29102 | load: 0.00017 [I 2022-10-26 18:15:20] ...trap/engines/engine.py.321: eval: epoch 22 | batch 330/510 [I 2022-10-26 18:15:20] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:53 [I 2022-10-26 18:15:20] ...trap/engines/engine.py.326: process: 0.29560 | load: 0.00018 [I 2022-10-26 18:15:24] ...trap/engines/engine.py.321: eval: epoch 22 | batch 340/510 [I 2022-10-26 18:15:24] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:58 [I 2022-10-26 18:15:24] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00020 [I 2022-10-26 18:15:28] ...trap/engines/engine.py.321: eval: epoch 22 | batch 350/510 [I 2022-10-26 18:15:28] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:58 [I 2022-10-26 18:15:28] ...trap/engines/engine.py.326: process: 0.43935 | load: 0.00020 [I 2022-10-26 18:15:31] ...trap/engines/engine.py.321: eval: epoch 22 | batch 360/510 [I 2022-10-26 18:15:31] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:47 [I 2022-10-26 18:15:31] ...trap/engines/engine.py.326: process: 0.38031 | load: 0.00023 [I 2022-10-26 18:15:34] ...trap/engines/engine.py.321: eval: epoch 22 | batch 370/510 [I 2022-10-26 18:15:34] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:41 [I 2022-10-26 18:15:34] ...trap/engines/engine.py.326: process: 0.30027 | load: 0.00014 [I 2022-10-26 18:15:36] ...trap/engines/engine.py.321: eval: epoch 22 | batch 380/510 [I 2022-10-26 18:15:36] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-26 18:15:36] ...trap/engines/engine.py.326: process: 0.29064 | load: 0.00016 [I 2022-10-26 18:15:39] ...trap/engines/engine.py.321: eval: epoch 22 | batch 390/510 [I 2022-10-26 18:15:39] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-26 18:15:39] ...trap/engines/engine.py.326: process: 0.28798 | load: 0.00016 [I 2022-10-26 18:15:42] ...trap/engines/engine.py.321: eval: epoch 22 | batch 400/510 [I 2022-10-26 18:15:42] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-26 18:15:42] ...trap/engines/engine.py.326: process: 0.28586 | load: 0.00020 [I 2022-10-26 18:15:45] ...trap/engines/engine.py.321: eval: epoch 22 | batch 410/510 [I 2022-10-26 18:15:45] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-26 18:15:45] ...trap/engines/engine.py.326: process: 0.28161 | load: 0.00021 [I 2022-10-26 18:15:48] ...trap/engines/engine.py.321: eval: epoch 22 | batch 420/510 [I 2022-10-26 18:15:48] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-26 18:15:48] ...trap/engines/engine.py.326: process: 0.28656 | load: 0.00020 [I 2022-10-26 18:15:51] ...trap/engines/engine.py.321: eval: epoch 22 | batch 430/510 [I 2022-10-26 18:15:51] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-26 18:15:51] ...trap/engines/engine.py.326: process: 0.28740 | load: 0.00020 [I 2022-10-26 18:15:54] ...trap/engines/engine.py.321: eval: epoch 22 | batch 440/510 [I 2022-10-26 18:15:54] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-26 18:15:54] ...trap/engines/engine.py.326: process: 0.29544 | load: 0.00018 [I 2022-10-26 18:15:57] ...trap/engines/engine.py.321: eval: epoch 22 | batch 450/510 [I 2022-10-26 18:15:57] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-26 18:15:57] ...trap/engines/engine.py.326: process: 0.28730 | load: 0.00022 [I 2022-10-26 18:16:00] ...trap/engines/engine.py.321: eval: epoch 22 | batch 460/510 [I 2022-10-26 18:16:00] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-26 18:16:00] ...trap/engines/engine.py.326: process: 0.29075 | load: 0.00016 [I 2022-10-26 18:16:03] ...trap/engines/engine.py.321: eval: epoch 22 | batch 470/510 [I 2022-10-26 18:16:03] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-26 18:16:03] ...trap/engines/engine.py.326: process: 0.30661 | load: 0.00017 [I 2022-10-26 18:16:05] ...trap/engines/engine.py.321: eval: epoch 22 | batch 480/510 [I 2022-10-26 18:16:05] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-26 18:16:05] ...trap/engines/engine.py.326: process: 0.28591 | load: 0.00016 [I 2022-10-26 18:16:08] ...trap/engines/engine.py.321: eval: epoch 22 | batch 490/510 [I 2022-10-26 18:16:08] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-26 18:16:08] ...trap/engines/engine.py.326: process: 0.28731 | load: 0.00019 [I 2022-10-26 18:16:11] ...trap/engines/engine.py.321: eval: epoch 22 | batch 500/510 [I 2022-10-26 18:16:11] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-26 18:16:11] ...trap/engines/engine.py.326: process: 0.28717 | load: 0.00018 [I 2022-10-26 18:16:14] ...trap/engines/engine.py.321: eval: epoch 22 | batch 510/510 [I 2022-10-26 18:16:14] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-26 18:16:14] ...trap/engines/engine.py.326: process: 0.28334 | load: 0.00022 [S 2022-10-26 18:16:15] ...trap/engines/engine.py.132: eval_epoch.epoch: 22 [I 2022-10-26 18:16:15] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.613600 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.861800 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.911600 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.607800 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.856500 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.906700 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008297 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010980 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009035 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013703 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010614 [S 2022-10-26 18:16:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008474 [I 2022-10-26 18:16:20] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 18:16:20] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 18:16:20] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 18:16:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 18:16:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 18:16:21] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 18:16:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 18:16:30] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 18:16:30] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 18:16:30] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 18:16:33] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 18:16:39] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 18:16:39] ...trap/engines/engine.py.163: Training model on trainset for epoch 23 [I 2022-10-26 18:16:39] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 18:16:42] ...trap/engines/engine.py.224: train: epoch 23 | batch 0/2304 [I 2022-10-26 18:16:42] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:44 [I 2022-10-26 18:16:42] ...trap/engines/engine.py.229: process: 2.67559 | load: 2.03590 [I 2022-10-26 18:16:42] ...trap/engines/engine.py.230: loss: 0.18585 [I 2022-10-26 18:16:42] ...optimizers/trijoint.py.151: total_norm: 24.296370 [I 2022-10-26 18:16:48] ...trap/engines/engine.py.224: train: epoch 23 | batch 10/2304 [I 2022-10-26 18:16:48] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:35 [I 2022-10-26 18:16:48] ...trap/engines/engine.py.229: process: 0.63862 | load: 0.00391 [I 2022-10-26 18:16:48] ...trap/engines/engine.py.230: loss: 0.25582 [I 2022-10-26 18:16:48] ...optimizers/trijoint.py.151: total_norm: 23.198708 [I 2022-10-26 18:16:53] ...trap/engines/engine.py.224: train: epoch 23 | batch 20/2304 [I 2022-10-26 18:16:53] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:06 [I 2022-10-26 18:16:53] ...trap/engines/engine.py.229: process: 0.55209 | load: 0.00366 [I 2022-10-26 18:16:53] ...trap/engines/engine.py.230: loss: 0.15036 [I 2022-10-26 18:16:53] ...optimizers/trijoint.py.151: total_norm: 22.194696 [I 2022-10-26 18:16:59] ...trap/engines/engine.py.224: train: epoch 23 | batch 30/2304 [I 2022-10-26 18:16:59] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:00 [I 2022-10-26 18:16:59] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00343 [I 2022-10-26 18:16:59] ...trap/engines/engine.py.230: loss: 0.22420 [I 2022-10-26 18:16:59] ...optimizers/trijoint.py.151: total_norm: 38.790112 [I 2022-10-26 18:17:04] ...trap/engines/engine.py.224: train: epoch 23 | batch 40/2304 [I 2022-10-26 18:17:04] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:58 [I 2022-10-26 18:17:04] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00305 [I 2022-10-26 18:17:04] ...trap/engines/engine.py.230: loss: 0.28171 [I 2022-10-26 18:17:04] ...optimizers/trijoint.py.151: total_norm: 15.145608 [I 2022-10-26 18:17:10] ...trap/engines/engine.py.224: train: epoch 23 | batch 50/2304 [I 2022-10-26 18:17:10] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:06 [I 2022-10-26 18:17:10] ...trap/engines/engine.py.229: process: 0.58267 | load: 0.00274 [I 2022-10-26 18:17:10] ...trap/engines/engine.py.230: loss: 0.12285 [I 2022-10-26 18:17:10] ...optimizers/trijoint.py.151: total_norm: 22.708134 [I 2022-10-26 18:17:15] ...trap/engines/engine.py.224: train: epoch 23 | batch 60/2304 [I 2022-10-26 18:17:15] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:54 [I 2022-10-26 18:17:15] ...trap/engines/engine.py.229: process: 0.56585 | load: 0.00299 [I 2022-10-26 18:17:15] ...trap/engines/engine.py.230: loss: 0.19147 [I 2022-10-26 18:17:15] ...optimizers/trijoint.py.151: total_norm: 26.450684 [I 2022-10-26 18:17:21] ...trap/engines/engine.py.224: train: epoch 23 | batch 70/2304 [I 2022-10-26 18:17:21] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:39 [I 2022-10-26 18:17:21] ...trap/engines/engine.py.229: process: 0.56555 | load: 0.00274 [I 2022-10-26 18:17:21] ...trap/engines/engine.py.230: loss: 0.23252 [I 2022-10-26 18:17:21] ...optimizers/trijoint.py.151: total_norm: 31.567543 [I 2022-10-26 18:17:27] ...trap/engines/engine.py.224: train: epoch 23 | batch 80/2304 [I 2022-10-26 18:17:27] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:58 [I 2022-10-26 18:17:27] ...trap/engines/engine.py.229: process: 0.55858 | load: 0.00294 [I 2022-10-26 18:17:27] ...trap/engines/engine.py.230: loss: 0.22364 [I 2022-10-26 18:17:27] ...optimizers/trijoint.py.151: total_norm: 20.587593 [I 2022-10-26 18:17:32] ...trap/engines/engine.py.224: train: epoch 23 | batch 90/2304 [I 2022-10-26 18:17:32] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:38 [I 2022-10-26 18:17:32] ...trap/engines/engine.py.229: process: 0.57387 | load: 0.00297 [I 2022-10-26 18:17:32] ...trap/engines/engine.py.230: loss: 0.15381 [I 2022-10-26 18:17:32] ...optimizers/trijoint.py.151: total_norm: 20.738092 [I 2022-10-26 18:17:38] ...trap/engines/engine.py.224: train: epoch 23 | batch 100/2304 [I 2022-10-26 18:17:38] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:43 [I 2022-10-26 18:17:38] ...trap/engines/engine.py.229: process: 0.58408 | load: 0.00263 [I 2022-10-26 18:17:38] ...trap/engines/engine.py.230: loss: 0.20016 [I 2022-10-26 18:17:38] ...optimizers/trijoint.py.151: total_norm: 15.700265 [I 2022-10-26 18:17:44] ...trap/engines/engine.py.224: train: epoch 23 | batch 110/2304 [I 2022-10-26 18:17:44] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:24 [I 2022-10-26 18:17:44] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00275 [I 2022-10-26 18:17:44] ...trap/engines/engine.py.230: loss: 0.13218 [I 2022-10-26 18:17:44] ...optimizers/trijoint.py.151: total_norm: 15.116138 [I 2022-10-26 18:17:49] ...trap/engines/engine.py.224: train: epoch 23 | batch 120/2304 [I 2022-10-26 18:17:49] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:17 [I 2022-10-26 18:17:49] ...trap/engines/engine.py.229: process: 0.54367 | load: 0.00264 [I 2022-10-26 18:17:49] ...trap/engines/engine.py.230: loss: 0.12493 [I 2022-10-26 18:17:49] ...optimizers/trijoint.py.151: total_norm: 17.775131 [I 2022-10-26 18:17:55] ...trap/engines/engine.py.224: train: epoch 23 | batch 130/2304 [I 2022-10-26 18:17:55] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:04 [I 2022-10-26 18:17:55] ...trap/engines/engine.py.229: process: 0.53509 | load: 0.00270 [I 2022-10-26 18:17:55] ...trap/engines/engine.py.230: loss: 0.15556 [I 2022-10-26 18:17:55] ...optimizers/trijoint.py.151: total_norm: 30.131742 [I 2022-10-26 18:18:00] ...trap/engines/engine.py.224: train: epoch 23 | batch 140/2304 [I 2022-10-26 18:18:00] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:17 [I 2022-10-26 18:18:00] ...trap/engines/engine.py.229: process: 0.54841 | load: 0.00267 [I 2022-10-26 18:18:00] ...trap/engines/engine.py.230: loss: 0.20367 [I 2022-10-26 18:18:00] ...optimizers/trijoint.py.151: total_norm: 23.063715 [I 2022-10-26 18:18:06] ...trap/engines/engine.py.224: train: epoch 23 | batch 150/2304 [I 2022-10-26 18:18:06] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:08 [I 2022-10-26 18:18:06] ...trap/engines/engine.py.229: process: 0.58052 | load: 0.00264 [I 2022-10-26 18:18:06] ...trap/engines/engine.py.230: loss: 0.19192 [I 2022-10-26 18:18:06] ...optimizers/trijoint.py.151: total_norm: 54.838058 [I 2022-10-26 18:18:11] ...trap/engines/engine.py.224: train: epoch 23 | batch 160/2304 [I 2022-10-26 18:18:11] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:40 [I 2022-10-26 18:18:11] ...trap/engines/engine.py.229: process: 0.54635 | load: 0.00279 [I 2022-10-26 18:18:11] ...trap/engines/engine.py.230: loss: 0.15926 [I 2022-10-26 18:18:11] ...optimizers/trijoint.py.151: total_norm: 25.618845 [I 2022-10-26 18:18:17] ...trap/engines/engine.py.224: train: epoch 23 | batch 170/2304 [I 2022-10-26 18:18:17] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:47 [I 2022-10-26 18:18:17] ...trap/engines/engine.py.229: process: 0.54814 | load: 0.00264 [I 2022-10-26 18:18:17] ...trap/engines/engine.py.230: loss: 0.25780 [I 2022-10-26 18:18:17] ...optimizers/trijoint.py.151: total_norm: 19.348043 [I 2022-10-26 18:18:23] ...trap/engines/engine.py.224: train: epoch 23 | batch 180/2304 [I 2022-10-26 18:18:23] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:41 [I 2022-10-26 18:18:23] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00244 [I 2022-10-26 18:18:23] ...trap/engines/engine.py.230: loss: 0.19337 [I 2022-10-26 18:18:23] ...optimizers/trijoint.py.151: total_norm: 23.874901 [I 2022-10-26 18:18:28] ...trap/engines/engine.py.224: train: epoch 23 | batch 190/2304 [I 2022-10-26 18:18:28] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:39 [I 2022-10-26 18:18:28] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00243 [I 2022-10-26 18:18:28] ...trap/engines/engine.py.230: loss: 0.27791 [I 2022-10-26 18:18:28] ...optimizers/trijoint.py.151: total_norm: 23.084625 [I 2022-10-26 18:18:34] ...trap/engines/engine.py.224: train: epoch 23 | batch 200/2304 [I 2022-10-26 18:18:34] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:20:01 [I 2022-10-26 18:18:34] ...trap/engines/engine.py.229: process: 0.61593 | load: 0.00246 [I 2022-10-26 18:18:34] ...trap/engines/engine.py.230: loss: 0.17901 [I 2022-10-26 18:18:34] ...optimizers/trijoint.py.151: total_norm: 34.638474 [I 2022-10-26 18:18:40] ...trap/engines/engine.py.224: train: epoch 23 | batch 210/2304 [I 2022-10-26 18:18:40] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:40 [I 2022-10-26 18:18:40] ...trap/engines/engine.py.229: process: 0.56391 | load: 0.00276 [I 2022-10-26 18:18:40] ...trap/engines/engine.py.230: loss: 0.25000 [I 2022-10-26 18:18:40] ...optimizers/trijoint.py.151: total_norm: 20.053490 [I 2022-10-26 18:18:45] ...trap/engines/engine.py.224: train: epoch 23 | batch 220/2304 [I 2022-10-26 18:18:45] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:12 [I 2022-10-26 18:18:45] ...trap/engines/engine.py.229: process: 0.54665 | load: 0.00258 [I 2022-10-26 18:18:45] ...trap/engines/engine.py.230: loss: 0.20276 [I 2022-10-26 18:18:45] ...optimizers/trijoint.py.151: total_norm: 23.683149 [I 2022-10-26 18:18:51] ...trap/engines/engine.py.224: train: epoch 23 | batch 230/2304 [I 2022-10-26 18:18:51] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:15 [I 2022-10-26 18:18:51] ...trap/engines/engine.py.229: process: 0.55567 | load: 0.00245 [I 2022-10-26 18:18:51] ...trap/engines/engine.py.230: loss: 0.25382 [I 2022-10-26 18:18:51] ...optimizers/trijoint.py.151: total_norm: 22.143005 [I 2022-10-26 18:18:56] ...trap/engines/engine.py.224: train: epoch 23 | batch 240/2304 [I 2022-10-26 18:18:56] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:04 [I 2022-10-26 18:18:56] ...trap/engines/engine.py.229: process: 0.54698 | load: 0.00258 [I 2022-10-26 18:18:56] ...trap/engines/engine.py.230: loss: 0.14171 [I 2022-10-26 18:18:56] ...optimizers/trijoint.py.151: total_norm: 14.705622 [I 2022-10-26 18:19:02] ...trap/engines/engine.py.224: train: epoch 23 | batch 250/2304 [I 2022-10-26 18:19:02] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:02 [I 2022-10-26 18:19:02] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00243 [I 2022-10-26 18:19:02] ...trap/engines/engine.py.230: loss: 0.27424 [I 2022-10-26 18:19:02] ...optimizers/trijoint.py.151: total_norm: 27.767393 [I 2022-10-26 18:19:07] ...trap/engines/engine.py.224: train: epoch 23 | batch 260/2304 [I 2022-10-26 18:19:07] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:04 [I 2022-10-26 18:19:07] ...trap/engines/engine.py.229: process: 0.56192 | load: 0.00253 [I 2022-10-26 18:19:07] ...trap/engines/engine.py.230: loss: 0.22731 [I 2022-10-26 18:19:07] ...optimizers/trijoint.py.151: total_norm: 26.095474 [I 2022-10-26 18:19:13] ...trap/engines/engine.py.224: train: epoch 23 | batch 270/2304 [I 2022-10-26 18:19:13] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:57 [I 2022-10-26 18:19:13] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00243 [I 2022-10-26 18:19:13] ...trap/engines/engine.py.230: loss: 0.14547 [I 2022-10-26 18:19:13] ...optimizers/trijoint.py.151: total_norm: 19.013681 [I 2022-10-26 18:19:19] ...trap/engines/engine.py.224: train: epoch 23 | batch 280/2304 [I 2022-10-26 18:19:19] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:56 [I 2022-10-26 18:19:19] ...trap/engines/engine.py.229: process: 0.56930 | load: 0.00244 [I 2022-10-26 18:19:19] ...trap/engines/engine.py.230: loss: 0.14561 [I 2022-10-26 18:19:19] ...optimizers/trijoint.py.151: total_norm: 11.855614 [I 2022-10-26 18:19:24] ...trap/engines/engine.py.224: train: epoch 23 | batch 290/2304 [I 2022-10-26 18:19:24] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:38 [I 2022-10-26 18:19:24] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00242 [I 2022-10-26 18:19:24] ...trap/engines/engine.py.230: loss: 0.22517 [I 2022-10-26 18:19:24] ...optimizers/trijoint.py.151: total_norm: 24.701275 [I 2022-10-26 18:19:30] ...trap/engines/engine.py.224: train: epoch 23 | batch 300/2304 [I 2022-10-26 18:19:30] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:31 [I 2022-10-26 18:19:30] ...trap/engines/engine.py.229: process: 0.54917 | load: 0.00242 [I 2022-10-26 18:19:30] ...trap/engines/engine.py.230: loss: 0.13165 [I 2022-10-26 18:19:30] ...optimizers/trijoint.py.151: total_norm: 20.210947 [I 2022-10-26 18:19:35] ...trap/engines/engine.py.224: train: epoch 23 | batch 310/2304 [I 2022-10-26 18:19:35] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:36 [I 2022-10-26 18:19:35] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00248 [I 2022-10-26 18:19:35] ...trap/engines/engine.py.230: loss: 0.21525 [I 2022-10-26 18:19:35] ...optimizers/trijoint.py.151: total_norm: 29.320915 [I 2022-10-26 18:19:41] ...trap/engines/engine.py.224: train: epoch 23 | batch 320/2304 [I 2022-10-26 18:19:41] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:28 [I 2022-10-26 18:19:41] ...trap/engines/engine.py.229: process: 0.56153 | load: 0.00272 [I 2022-10-26 18:19:41] ...trap/engines/engine.py.230: loss: 0.15417 [I 2022-10-26 18:19:41] ...optimizers/trijoint.py.151: total_norm: 21.000010 [I 2022-10-26 18:19:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 330/2304 [I 2022-10-26 18:19:47] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:59 [I 2022-10-26 18:19:47] ...trap/engines/engine.py.229: process: 0.55117 | load: 0.00246 [I 2022-10-26 18:19:47] ...trap/engines/engine.py.230: loss: 0.18580 [I 2022-10-26 18:19:47] ...optimizers/trijoint.py.151: total_norm: 19.362122 [I 2022-10-26 18:19:52] ...trap/engines/engine.py.224: train: epoch 23 | batch 340/2304 [I 2022-10-26 18:19:52] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:16 [I 2022-10-26 18:19:52] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00242 [I 2022-10-26 18:19:52] ...trap/engines/engine.py.230: loss: 0.18586 [I 2022-10-26 18:19:52] ...optimizers/trijoint.py.151: total_norm: 25.265623 [I 2022-10-26 18:19:58] ...trap/engines/engine.py.224: train: epoch 23 | batch 350/2304 [I 2022-10-26 18:19:58] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:06 [I 2022-10-26 18:19:58] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00247 [I 2022-10-26 18:19:58] ...trap/engines/engine.py.230: loss: 0.20142 [I 2022-10-26 18:19:58] ...optimizers/trijoint.py.151: total_norm: 15.880574 [I 2022-10-26 18:20:03] ...trap/engines/engine.py.224: train: epoch 23 | batch 360/2304 [I 2022-10-26 18:20:03] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:00 [I 2022-10-26 18:20:03] ...trap/engines/engine.py.229: process: 0.54575 | load: 0.00247 [I 2022-10-26 18:20:03] ...trap/engines/engine.py.230: loss: 0.13251 [I 2022-10-26 18:20:03] ...optimizers/trijoint.py.151: total_norm: 13.852565 [I 2022-10-26 18:20:09] ...trap/engines/engine.py.224: train: epoch 23 | batch 370/2304 [I 2022-10-26 18:20:09] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:46 [I 2022-10-26 18:20:09] ...trap/engines/engine.py.229: process: 0.53454 | load: 0.00247 [I 2022-10-26 18:20:09] ...trap/engines/engine.py.230: loss: 0.20768 [I 2022-10-26 18:20:09] ...optimizers/trijoint.py.151: total_norm: 15.760133 [I 2022-10-26 18:20:15] ...trap/engines/engine.py.224: train: epoch 23 | batch 380/2304 [I 2022-10-26 18:20:15] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:44 [I 2022-10-26 18:20:15] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00240 [I 2022-10-26 18:20:15] ...trap/engines/engine.py.230: loss: 0.16641 [I 2022-10-26 18:20:15] ...optimizers/trijoint.py.151: total_norm: 17.977594 [I 2022-10-26 18:20:20] ...trap/engines/engine.py.224: train: epoch 23 | batch 390/2304 [I 2022-10-26 18:20:20] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:44 [I 2022-10-26 18:20:20] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00251 [I 2022-10-26 18:20:20] ...trap/engines/engine.py.230: loss: 0.23490 [I 2022-10-26 18:20:20] ...optimizers/trijoint.py.151: total_norm: 23.903959 [I 2022-10-26 18:20:26] ...trap/engines/engine.py.224: train: epoch 23 | batch 400/2304 [I 2022-10-26 18:20:26] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:54 [I 2022-10-26 18:20:26] ...trap/engines/engine.py.229: process: 0.56109 | load: 0.00241 [I 2022-10-26 18:20:26] ...trap/engines/engine.py.230: loss: 0.15685 [I 2022-10-26 18:20:26] ...optimizers/trijoint.py.151: total_norm: 17.005291 [I 2022-10-26 18:20:31] ...trap/engines/engine.py.224: train: epoch 23 | batch 410/2304 [I 2022-10-26 18:20:31] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:31 [I 2022-10-26 18:20:31] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00241 [I 2022-10-26 18:20:31] ...trap/engines/engine.py.230: loss: 0.16695 [I 2022-10-26 18:20:31] ...optimizers/trijoint.py.151: total_norm: 13.688840 [I 2022-10-26 18:20:37] ...trap/engines/engine.py.224: train: epoch 23 | batch 420/2304 [I 2022-10-26 18:20:37] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:34 [I 2022-10-26 18:20:37] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00250 [I 2022-10-26 18:20:37] ...trap/engines/engine.py.230: loss: 0.19152 [I 2022-10-26 18:20:37] ...optimizers/trijoint.py.151: total_norm: 14.188662 [I 2022-10-26 18:20:43] ...trap/engines/engine.py.224: train: epoch 23 | batch 430/2304 [I 2022-10-26 18:20:43] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:21 [I 2022-10-26 18:20:43] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00244 [I 2022-10-26 18:20:43] ...trap/engines/engine.py.230: loss: 0.15570 [I 2022-10-26 18:20:43] ...optimizers/trijoint.py.151: total_norm: 12.459158 [I 2022-10-26 18:20:48] ...trap/engines/engine.py.224: train: epoch 23 | batch 440/2304 [I 2022-10-26 18:20:48] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:14 [I 2022-10-26 18:20:48] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00247 [I 2022-10-26 18:20:48] ...trap/engines/engine.py.230: loss: 0.22964 [I 2022-10-26 18:20:48] ...optimizers/trijoint.py.151: total_norm: 20.373472 [I 2022-10-26 18:20:54] ...trap/engines/engine.py.224: train: epoch 23 | batch 450/2304 [I 2022-10-26 18:20:54] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:07 [I 2022-10-26 18:20:54] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00244 [I 2022-10-26 18:20:54] ...trap/engines/engine.py.230: loss: 0.19000 [I 2022-10-26 18:20:54] ...optimizers/trijoint.py.151: total_norm: 21.469145 [I 2022-10-26 18:20:59] ...trap/engines/engine.py.224: train: epoch 23 | batch 460/2304 [I 2022-10-26 18:20:59] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:04 [I 2022-10-26 18:20:59] ...trap/engines/engine.py.229: process: 0.56546 | load: 0.00242 [I 2022-10-26 18:20:59] ...trap/engines/engine.py.230: loss: 0.16811 [I 2022-10-26 18:20:59] ...optimizers/trijoint.py.151: total_norm: 15.840029 [I 2022-10-26 18:21:05] ...trap/engines/engine.py.224: train: epoch 23 | batch 470/2304 [I 2022-10-26 18:21:05] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:00 [I 2022-10-26 18:21:05] ...trap/engines/engine.py.229: process: 0.56392 | load: 0.00246 [I 2022-10-26 18:21:05] ...trap/engines/engine.py.230: loss: 0.25958 [I 2022-10-26 18:21:05] ...optimizers/trijoint.py.151: total_norm: 23.291996 [I 2022-10-26 18:21:10] ...trap/engines/engine.py.224: train: epoch 23 | batch 480/2304 [I 2022-10-26 18:21:10] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:50 [I 2022-10-26 18:21:10] ...trap/engines/engine.py.229: process: 0.54387 | load: 0.00241 [I 2022-10-26 18:21:10] ...trap/engines/engine.py.230: loss: 0.18487 [I 2022-10-26 18:21:10] ...optimizers/trijoint.py.151: total_norm: 27.363892 [I 2022-10-26 18:21:16] ...trap/engines/engine.py.224: train: epoch 23 | batch 490/2304 [I 2022-10-26 18:21:16] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:47 [I 2022-10-26 18:21:16] ...trap/engines/engine.py.229: process: 0.56369 | load: 0.00245 [I 2022-10-26 18:21:16] ...trap/engines/engine.py.230: loss: 0.25905 [I 2022-10-26 18:21:16] ...optimizers/trijoint.py.151: total_norm: 29.151651 [I 2022-10-26 18:21:22] ...trap/engines/engine.py.224: train: epoch 23 | batch 500/2304 [I 2022-10-26 18:21:22] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:43 [I 2022-10-26 18:21:22] ...trap/engines/engine.py.229: process: 0.56679 | load: 0.00245 [I 2022-10-26 18:21:22] ...trap/engines/engine.py.230: loss: 0.21050 [I 2022-10-26 18:21:22] ...optimizers/trijoint.py.151: total_norm: 23.518454 [I 2022-10-26 18:21:27] ...trap/engines/engine.py.224: train: epoch 23 | batch 510/2304 [I 2022-10-26 18:21:27] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:39 [I 2022-10-26 18:21:27] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00260 [I 2022-10-26 18:21:27] ...trap/engines/engine.py.230: loss: 0.24086 [I 2022-10-26 18:21:27] ...optimizers/trijoint.py.151: total_norm: 20.092354 [I 2022-10-26 18:21:33] ...trap/engines/engine.py.224: train: epoch 23 | batch 520/2304 [I 2022-10-26 18:21:33] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:35 [I 2022-10-26 18:21:33] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00242 [I 2022-10-26 18:21:33] ...trap/engines/engine.py.230: loss: 0.21218 [I 2022-10-26 18:21:33] ...optimizers/trijoint.py.151: total_norm: 27.944565 [I 2022-10-26 18:21:38] ...trap/engines/engine.py.224: train: epoch 23 | batch 530/2304 [I 2022-10-26 18:21:38] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:23 [I 2022-10-26 18:21:38] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00243 [I 2022-10-26 18:21:38] ...trap/engines/engine.py.230: loss: 0.20510 [I 2022-10-26 18:21:38] ...optimizers/trijoint.py.151: total_norm: 21.939400 [I 2022-10-26 18:21:44] ...trap/engines/engine.py.224: train: epoch 23 | batch 540/2304 [I 2022-10-26 18:21:44] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:31 [I 2022-10-26 18:21:44] ...trap/engines/engine.py.229: process: 0.58266 | load: 0.00242 [I 2022-10-26 18:21:44] ...trap/engines/engine.py.230: loss: 0.19418 [I 2022-10-26 18:21:44] ...optimizers/trijoint.py.151: total_norm: 34.643967 [I 2022-10-26 18:21:49] ...trap/engines/engine.py.224: train: epoch 23 | batch 550/2304 [I 2022-10-26 18:21:49] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:18 [I 2022-10-26 18:21:49] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00259 [I 2022-10-26 18:21:49] ...trap/engines/engine.py.230: loss: 0.27580 [I 2022-10-26 18:21:49] ...optimizers/trijoint.py.151: total_norm: 31.351168 [I 2022-10-26 18:21:55] ...trap/engines/engine.py.224: train: epoch 23 | batch 560/2304 [I 2022-10-26 18:21:55] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:20 [I 2022-10-26 18:21:55] ...trap/engines/engine.py.229: process: 0.57315 | load: 0.00241 [I 2022-10-26 18:21:55] ...trap/engines/engine.py.230: loss: 0.20681 [I 2022-10-26 18:21:55] ...optimizers/trijoint.py.151: total_norm: 16.773520 [I 2022-10-26 18:22:01] ...trap/engines/engine.py.224: train: epoch 23 | batch 570/2304 [I 2022-10-26 18:22:01] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:10 [I 2022-10-26 18:22:01] ...trap/engines/engine.py.229: process: 0.54732 | load: 0.00260 [I 2022-10-26 18:22:01] ...trap/engines/engine.py.230: loss: 0.19892 [I 2022-10-26 18:22:01] ...optimizers/trijoint.py.151: total_norm: 19.729364 [I 2022-10-26 18:22:06] ...trap/engines/engine.py.224: train: epoch 23 | batch 580/2304 [I 2022-10-26 18:22:06] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:01 [I 2022-10-26 18:22:06] ...trap/engines/engine.py.229: process: 0.54398 | load: 0.00248 [I 2022-10-26 18:22:06] ...trap/engines/engine.py.230: loss: 0.15767 [I 2022-10-26 18:22:06] ...optimizers/trijoint.py.151: total_norm: 12.531941 [I 2022-10-26 18:22:12] ...trap/engines/engine.py.224: train: epoch 23 | batch 590/2304 [I 2022-10-26 18:22:12] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:53 [I 2022-10-26 18:22:12] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00292 [I 2022-10-26 18:22:12] ...trap/engines/engine.py.230: loss: 0.15212 [I 2022-10-26 18:22:12] ...optimizers/trijoint.py.151: total_norm: 27.912157 [I 2022-10-26 18:22:17] ...trap/engines/engine.py.224: train: epoch 23 | batch 600/2304 [I 2022-10-26 18:22:17] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:48 [I 2022-10-26 18:22:17] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00246 [I 2022-10-26 18:22:17] ...trap/engines/engine.py.230: loss: 0.14692 [I 2022-10-26 18:22:17] ...optimizers/trijoint.py.151: total_norm: 22.697802 [I 2022-10-26 18:22:23] ...trap/engines/engine.py.224: train: epoch 23 | batch 610/2304 [I 2022-10-26 18:22:23] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:48 [I 2022-10-26 18:22:23] ...trap/engines/engine.py.229: process: 0.56856 | load: 0.00247 [I 2022-10-26 18:22:23] ...trap/engines/engine.py.230: loss: 0.31315 [I 2022-10-26 18:22:23] ...optimizers/trijoint.py.151: total_norm: 38.688709 [I 2022-10-26 18:22:29] ...trap/engines/engine.py.224: train: epoch 23 | batch 620/2304 [I 2022-10-26 18:22:29] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:37 [I 2022-10-26 18:22:29] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00247 [I 2022-10-26 18:22:29] ...trap/engines/engine.py.230: loss: 0.30065 [I 2022-10-26 18:22:29] ...optimizers/trijoint.py.151: total_norm: 42.894588 [I 2022-10-26 18:22:34] ...trap/engines/engine.py.224: train: epoch 23 | batch 630/2304 [I 2022-10-26 18:22:34] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:35 [I 2022-10-26 18:22:34] ...trap/engines/engine.py.229: process: 0.56757 | load: 0.00256 [I 2022-10-26 18:22:34] ...trap/engines/engine.py.230: loss: 0.19860 [I 2022-10-26 18:22:34] ...optimizers/trijoint.py.151: total_norm: 24.387518 [I 2022-10-26 18:22:40] ...trap/engines/engine.py.224: train: epoch 23 | batch 640/2304 [I 2022-10-26 18:22:40] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:33 [I 2022-10-26 18:22:40] ...trap/engines/engine.py.229: process: 0.55985 | load: 0.00245 [I 2022-10-26 18:22:40] ...trap/engines/engine.py.230: loss: 0.20352 [I 2022-10-26 18:22:40] ...optimizers/trijoint.py.151: total_norm: 20.018805 [I 2022-10-26 18:22:45] ...trap/engines/engine.py.224: train: epoch 23 | batch 650/2304 [I 2022-10-26 18:22:45] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:27 [I 2022-10-26 18:22:45] ...trap/engines/engine.py.229: process: 0.56758 | load: 0.00242 [I 2022-10-26 18:22:45] ...trap/engines/engine.py.230: loss: 0.19691 [I 2022-10-26 18:22:45] ...optimizers/trijoint.py.151: total_norm: 21.256002 [I 2022-10-26 18:22:51] ...trap/engines/engine.py.224: train: epoch 23 | batch 660/2304 [I 2022-10-26 18:22:51] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:08 [I 2022-10-26 18:22:51] ...trap/engines/engine.py.229: process: 0.54440 | load: 0.00260 [I 2022-10-26 18:22:51] ...trap/engines/engine.py.230: loss: 0.27830 [I 2022-10-26 18:22:51] ...optimizers/trijoint.py.151: total_norm: 24.084408 [I 2022-10-26 18:22:57] ...trap/engines/engine.py.224: train: epoch 23 | batch 670/2304 [I 2022-10-26 18:22:57] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:15 [I 2022-10-26 18:22:57] ...trap/engines/engine.py.229: process: 0.56741 | load: 0.00246 [I 2022-10-26 18:22:57] ...trap/engines/engine.py.230: loss: 0.26890 [I 2022-10-26 18:22:57] ...optimizers/trijoint.py.151: total_norm: 16.640089 [I 2022-10-26 18:23:02] ...trap/engines/engine.py.224: train: epoch 23 | batch 680/2304 [I 2022-10-26 18:23:02] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:05 [I 2022-10-26 18:23:02] ...trap/engines/engine.py.229: process: 0.55141 | load: 0.00244 [I 2022-10-26 18:23:02] ...trap/engines/engine.py.230: loss: 0.14625 [I 2022-10-26 18:23:02] ...optimizers/trijoint.py.151: total_norm: 18.808128 [I 2022-10-26 18:23:08] ...trap/engines/engine.py.224: train: epoch 23 | batch 690/2304 [I 2022-10-26 18:23:08] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:58 [I 2022-10-26 18:23:08] ...trap/engines/engine.py.229: process: 0.55711 | load: 0.00243 [I 2022-10-26 18:23:08] ...trap/engines/engine.py.230: loss: 0.21294 [I 2022-10-26 18:23:08] ...optimizers/trijoint.py.151: total_norm: 24.060522 [I 2022-10-26 18:23:13] ...trap/engines/engine.py.224: train: epoch 23 | batch 700/2304 [I 2022-10-26 18:23:13] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:15:11 [I 2022-10-26 18:23:13] ...trap/engines/engine.py.229: process: 0.59108 | load: 0.00250 [I 2022-10-26 18:23:13] ...trap/engines/engine.py.230: loss: 0.19809 [I 2022-10-26 18:23:13] ...optimizers/trijoint.py.151: total_norm: 23.171328 [I 2022-10-26 18:23:19] ...trap/engines/engine.py.224: train: epoch 23 | batch 710/2304 [I 2022-10-26 18:23:19] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:54 [I 2022-10-26 18:23:19] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00249 [I 2022-10-26 18:23:19] ...trap/engines/engine.py.230: loss: 0.24971 [I 2022-10-26 18:23:19] ...optimizers/trijoint.py.151: total_norm: 23.475422 [I 2022-10-26 18:23:25] ...trap/engines/engine.py.224: train: epoch 23 | batch 720/2304 [I 2022-10-26 18:23:25] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:43 [I 2022-10-26 18:23:25] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00248 [I 2022-10-26 18:23:25] ...trap/engines/engine.py.230: loss: 0.13664 [I 2022-10-26 18:23:25] ...optimizers/trijoint.py.151: total_norm: 22.803066 [I 2022-10-26 18:23:30] ...trap/engines/engine.py.224: train: epoch 23 | batch 730/2304 [I 2022-10-26 18:23:30] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:37 [I 2022-10-26 18:23:30] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00245 [I 2022-10-26 18:23:30] ...trap/engines/engine.py.230: loss: 0.21162 [I 2022-10-26 18:23:30] ...optimizers/trijoint.py.151: total_norm: 20.002029 [I 2022-10-26 18:23:36] ...trap/engines/engine.py.224: train: epoch 23 | batch 740/2304 [I 2022-10-26 18:23:36] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:41 [I 2022-10-26 18:23:36] ...trap/engines/engine.py.229: process: 0.57100 | load: 0.00249 [I 2022-10-26 18:23:36] ...trap/engines/engine.py.230: loss: 0.17963 [I 2022-10-26 18:23:36] ...optimizers/trijoint.py.151: total_norm: 31.150925 [I 2022-10-26 18:23:41] ...trap/engines/engine.py.224: train: epoch 23 | batch 750/2304 [I 2022-10-26 18:23:41] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:19 [I 2022-10-26 18:23:41] ...trap/engines/engine.py.229: process: 0.53740 | load: 0.00250 [I 2022-10-26 18:23:41] ...trap/engines/engine.py.230: loss: 0.17151 [I 2022-10-26 18:23:41] ...optimizers/trijoint.py.151: total_norm: 19.264273 [I 2022-10-26 18:23:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 760/2304 [I 2022-10-26 18:23:47] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:26 [I 2022-10-26 18:23:47] ...trap/engines/engine.py.229: process: 0.55847 | load: 0.00249 [I 2022-10-26 18:23:47] ...trap/engines/engine.py.230: loss: 0.19907 [I 2022-10-26 18:23:47] ...optimizers/trijoint.py.151: total_norm: 35.295059 [I 2022-10-26 18:23:53] ...trap/engines/engine.py.224: train: epoch 23 | batch 770/2304 [I 2022-10-26 18:23:53] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:20 [I 2022-10-26 18:23:53] ...trap/engines/engine.py.229: process: 0.57036 | load: 0.00246 [I 2022-10-26 18:23:53] ...trap/engines/engine.py.230: loss: 0.23266 [I 2022-10-26 18:23:53] ...optimizers/trijoint.py.151: total_norm: 19.395241 [I 2022-10-26 18:23:58] ...trap/engines/engine.py.224: train: epoch 23 | batch 780/2304 [I 2022-10-26 18:23:58] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:05 [I 2022-10-26 18:23:58] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00242 [I 2022-10-26 18:23:58] ...trap/engines/engine.py.230: loss: 0.14097 [I 2022-10-26 18:23:58] ...optimizers/trijoint.py.151: total_norm: 18.867298 [I 2022-10-26 18:24:04] ...trap/engines/engine.py.224: train: epoch 23 | batch 790/2304 [I 2022-10-26 18:24:04] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:09 [I 2022-10-26 18:24:04] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00247 [I 2022-10-26 18:24:04] ...trap/engines/engine.py.230: loss: 0.12684 [I 2022-10-26 18:24:04] ...optimizers/trijoint.py.151: total_norm: 18.542036 [I 2022-10-26 18:24:09] ...trap/engines/engine.py.224: train: epoch 23 | batch 800/2304 [I 2022-10-26 18:24:09] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:54 [I 2022-10-26 18:24:09] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00246 [I 2022-10-26 18:24:09] ...trap/engines/engine.py.230: loss: 0.24942 [I 2022-10-26 18:24:09] ...optimizers/trijoint.py.151: total_norm: 31.631878 [I 2022-10-26 18:24:15] ...trap/engines/engine.py.224: train: epoch 23 | batch 810/2304 [I 2022-10-26 18:24:15] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:48 [I 2022-10-26 18:24:15] ...trap/engines/engine.py.229: process: 0.56164 | load: 0.00248 [I 2022-10-26 18:24:15] ...trap/engines/engine.py.230: loss: 0.26800 [I 2022-10-26 18:24:15] ...optimizers/trijoint.py.151: total_norm: 23.322968 [I 2022-10-26 18:24:20] ...trap/engines/engine.py.224: train: epoch 23 | batch 820/2304 [I 2022-10-26 18:24:20] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:48 [I 2022-10-26 18:24:20] ...trap/engines/engine.py.229: process: 0.56320 | load: 0.00251 [I 2022-10-26 18:24:20] ...trap/engines/engine.py.230: loss: 0.16719 [I 2022-10-26 18:24:20] ...optimizers/trijoint.py.151: total_norm: 23.631794 [I 2022-10-26 18:24:26] ...trap/engines/engine.py.224: train: epoch 23 | batch 830/2304 [I 2022-10-26 18:24:26] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:41 [I 2022-10-26 18:24:26] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00373 [I 2022-10-26 18:24:26] ...trap/engines/engine.py.230: loss: 0.17724 [I 2022-10-26 18:24:26] ...optimizers/trijoint.py.151: total_norm: 18.999235 [I 2022-10-26 18:24:32] ...trap/engines/engine.py.224: train: epoch 23 | batch 840/2304 [I 2022-10-26 18:24:32] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:43 [I 2022-10-26 18:24:32] ...trap/engines/engine.py.229: process: 0.56591 | load: 0.00263 [I 2022-10-26 18:24:32] ...trap/engines/engine.py.230: loss: 0.31300 [I 2022-10-26 18:24:32] ...optimizers/trijoint.py.151: total_norm: 49.762352 [I 2022-10-26 18:24:37] ...trap/engines/engine.py.224: train: epoch 23 | batch 850/2304 [I 2022-10-26 18:24:37] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:36 [I 2022-10-26 18:24:37] ...trap/engines/engine.py.229: process: 0.57730 | load: 0.00254 [I 2022-10-26 18:24:37] ...trap/engines/engine.py.230: loss: 0.25358 [I 2022-10-26 18:24:37] ...optimizers/trijoint.py.151: total_norm: 33.784924 [I 2022-10-26 18:24:43] ...trap/engines/engine.py.224: train: epoch 23 | batch 860/2304 [I 2022-10-26 18:24:43] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:27 [I 2022-10-26 18:24:43] ...trap/engines/engine.py.229: process: 0.55605 | load: 0.00250 [I 2022-10-26 18:24:43] ...trap/engines/engine.py.230: loss: 0.23459 [I 2022-10-26 18:24:43] ...optimizers/trijoint.py.151: total_norm: 13.822113 [I 2022-10-26 18:24:48] ...trap/engines/engine.py.224: train: epoch 23 | batch 870/2304 [I 2022-10-26 18:24:48] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:18 [I 2022-10-26 18:24:48] ...trap/engines/engine.py.229: process: 0.54858 | load: 0.00246 [I 2022-10-26 18:24:48] ...trap/engines/engine.py.230: loss: 0.20060 [I 2022-10-26 18:24:48] ...optimizers/trijoint.py.151: total_norm: 16.455372 [I 2022-10-26 18:24:54] ...trap/engines/engine.py.224: train: epoch 23 | batch 880/2304 [I 2022-10-26 18:24:54] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:21 [I 2022-10-26 18:24:54] ...trap/engines/engine.py.229: process: 0.56998 | load: 0.00248 [I 2022-10-26 18:24:54] ...trap/engines/engine.py.230: loss: 0.21960 [I 2022-10-26 18:24:54] ...optimizers/trijoint.py.151: total_norm: 22.267979 [I 2022-10-26 18:25:00] ...trap/engines/engine.py.224: train: epoch 23 | batch 890/2304 [I 2022-10-26 18:25:00] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:10 [I 2022-10-26 18:25:00] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00254 [I 2022-10-26 18:25:00] ...trap/engines/engine.py.230: loss: 0.43219 [I 2022-10-26 18:25:00] ...optimizers/trijoint.py.151: total_norm: 29.576777 [I 2022-10-26 18:25:05] ...trap/engines/engine.py.224: train: epoch 23 | batch 900/2304 [I 2022-10-26 18:25:05] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:04 [I 2022-10-26 18:25:05] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00247 [I 2022-10-26 18:25:05] ...trap/engines/engine.py.230: loss: 0.24432 [I 2022-10-26 18:25:05] ...optimizers/trijoint.py.151: total_norm: 36.236187 [I 2022-10-26 18:25:11] ...trap/engines/engine.py.224: train: epoch 23 | batch 910/2304 [I 2022-10-26 18:25:11] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:57 [I 2022-10-26 18:25:11] ...trap/engines/engine.py.229: process: 0.56512 | load: 0.00244 [I 2022-10-26 18:25:11] ...trap/engines/engine.py.230: loss: 0.26497 [I 2022-10-26 18:25:11] ...optimizers/trijoint.py.151: total_norm: 18.664749 [I 2022-10-26 18:25:16] ...trap/engines/engine.py.224: train: epoch 23 | batch 920/2304 [I 2022-10-26 18:25:16] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:52 [I 2022-10-26 18:25:16] ...trap/engines/engine.py.229: process: 0.56073 | load: 0.00243 [I 2022-10-26 18:25:16] ...trap/engines/engine.py.230: loss: 0.14365 [I 2022-10-26 18:25:16] ...optimizers/trijoint.py.151: total_norm: 11.751422 [I 2022-10-26 18:25:22] ...trap/engines/engine.py.224: train: epoch 23 | batch 930/2304 [I 2022-10-26 18:25:22] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:13:03 [I 2022-10-26 18:25:22] ...trap/engines/engine.py.229: process: 0.58481 | load: 0.00240 [I 2022-10-26 18:25:22] ...trap/engines/engine.py.230: loss: 0.17734 [I 2022-10-26 18:25:22] ...optimizers/trijoint.py.151: total_norm: 18.201273 [I 2022-10-26 18:25:28] ...trap/engines/engine.py.224: train: epoch 23 | batch 940/2304 [I 2022-10-26 18:25:28] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:39 [I 2022-10-26 18:25:28] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00243 [I 2022-10-26 18:25:28] ...trap/engines/engine.py.230: loss: 0.19657 [I 2022-10-26 18:25:28] ...optimizers/trijoint.py.151: total_norm: 16.088808 [I 2022-10-26 18:25:33] ...trap/engines/engine.py.224: train: epoch 23 | batch 950/2304 [I 2022-10-26 18:25:33] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:57 [I 2022-10-26 18:25:33] ...trap/engines/engine.py.229: process: 0.56542 | load: 0.00255 [I 2022-10-26 18:25:33] ...trap/engines/engine.py.230: loss: 0.25864 [I 2022-10-26 18:25:33] ...optimizers/trijoint.py.151: total_norm: 22.971123 [I 2022-10-26 18:25:39] ...trap/engines/engine.py.224: train: epoch 23 | batch 960/2304 [I 2022-10-26 18:25:39] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:33 [I 2022-10-26 18:25:39] ...trap/engines/engine.py.229: process: 0.56564 | load: 0.00248 [I 2022-10-26 18:25:39] ...trap/engines/engine.py.230: loss: 0.15885 [I 2022-10-26 18:25:39] ...optimizers/trijoint.py.151: total_norm: 15.595563 [I 2022-10-26 18:25:45] ...trap/engines/engine.py.224: train: epoch 23 | batch 970/2304 [I 2022-10-26 18:25:45] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:18 [I 2022-10-26 18:25:45] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00243 [I 2022-10-26 18:25:45] ...trap/engines/engine.py.230: loss: 0.17082 [I 2022-10-26 18:25:45] ...optimizers/trijoint.py.151: total_norm: 22.398798 [I 2022-10-26 18:25:50] ...trap/engines/engine.py.224: train: epoch 23 | batch 980/2304 [I 2022-10-26 18:25:50] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:14 [I 2022-10-26 18:25:50] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00243 [I 2022-10-26 18:25:50] ...trap/engines/engine.py.230: loss: 0.17809 [I 2022-10-26 18:25:50] ...optimizers/trijoint.py.151: total_norm: 18.977383 [I 2022-10-26 18:25:56] ...trap/engines/engine.py.224: train: epoch 23 | batch 990/2304 [I 2022-10-26 18:25:56] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:14 [I 2022-10-26 18:25:56] ...trap/engines/engine.py.229: process: 0.54932 | load: 0.00243 [I 2022-10-26 18:25:56] ...trap/engines/engine.py.230: loss: 0.18762 [I 2022-10-26 18:25:56] ...optimizers/trijoint.py.151: total_norm: 19.129910 [I 2022-10-26 18:26:01] ...trap/engines/engine.py.224: train: epoch 23 | batch 1000/2304 [I 2022-10-26 18:26:01] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:10 [I 2022-10-26 18:26:01] ...trap/engines/engine.py.229: process: 0.54812 | load: 0.00245 [I 2022-10-26 18:26:01] ...trap/engines/engine.py.230: loss: 0.18329 [I 2022-10-26 18:26:01] ...optimizers/trijoint.py.151: total_norm: 32.308609 [I 2022-10-26 18:26:07] ...trap/engines/engine.py.224: train: epoch 23 | batch 1010/2304 [I 2022-10-26 18:26:07] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:11:58 [I 2022-10-26 18:26:07] ...trap/engines/engine.py.229: process: 0.55809 | load: 0.00248 [I 2022-10-26 18:26:07] ...trap/engines/engine.py.230: loss: 0.17189 [I 2022-10-26 18:26:07] ...optimizers/trijoint.py.151: total_norm: 15.333111 [I 2022-10-26 18:26:13] ...trap/engines/engine.py.224: train: epoch 23 | batch 1020/2304 [I 2022-10-26 18:26:13] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:12:11 [I 2022-10-26 18:26:13] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00247 [I 2022-10-26 18:26:13] ...trap/engines/engine.py.230: loss: 0.23983 [I 2022-10-26 18:26:13] ...optimizers/trijoint.py.151: total_norm: 30.942947 [I 2022-10-26 18:26:18] ...trap/engines/engine.py.224: train: epoch 23 | batch 1030/2304 [I 2022-10-26 18:26:18] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:48 [I 2022-10-26 18:26:18] ...trap/engines/engine.py.229: process: 0.55552 | load: 0.00242 [I 2022-10-26 18:26:18] ...trap/engines/engine.py.230: loss: 0.17174 [I 2022-10-26 18:26:18] ...optimizers/trijoint.py.151: total_norm: 22.279121 [I 2022-10-26 18:26:24] ...trap/engines/engine.py.224: train: epoch 23 | batch 1040/2304 [I 2022-10-26 18:26:24] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:47 [I 2022-10-26 18:26:24] ...trap/engines/engine.py.229: process: 0.56460 | load: 0.00241 [I 2022-10-26 18:26:24] ...trap/engines/engine.py.230: loss: 0.24628 [I 2022-10-26 18:26:24] ...optimizers/trijoint.py.151: total_norm: 33.159573 [I 2022-10-26 18:26:29] ...trap/engines/engine.py.224: train: epoch 23 | batch 1050/2304 [I 2022-10-26 18:26:29] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:40 [I 2022-10-26 18:26:29] ...trap/engines/engine.py.229: process: 0.56353 | load: 0.00242 [I 2022-10-26 18:26:29] ...trap/engines/engine.py.230: loss: 0.20353 [I 2022-10-26 18:26:29] ...optimizers/trijoint.py.151: total_norm: 26.023312 [I 2022-10-26 18:26:35] ...trap/engines/engine.py.224: train: epoch 23 | batch 1060/2304 [I 2022-10-26 18:26:35] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:30 [I 2022-10-26 18:26:35] ...trap/engines/engine.py.229: process: 0.54346 | load: 0.00250 [I 2022-10-26 18:26:35] ...trap/engines/engine.py.230: loss: 0.21954 [I 2022-10-26 18:26:35] ...optimizers/trijoint.py.151: total_norm: 31.176825 [I 2022-10-26 18:26:40] ...trap/engines/engine.py.224: train: epoch 23 | batch 1070/2304 [I 2022-10-26 18:26:40] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:22 [I 2022-10-26 18:26:40] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00247 [I 2022-10-26 18:26:40] ...trap/engines/engine.py.230: loss: 0.17189 [I 2022-10-26 18:26:40] ...optimizers/trijoint.py.151: total_norm: 12.700293 [I 2022-10-26 18:26:46] ...trap/engines/engine.py.224: train: epoch 23 | batch 1080/2304 [I 2022-10-26 18:26:46] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:34 [I 2022-10-26 18:26:46] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00245 [I 2022-10-26 18:26:46] ...trap/engines/engine.py.230: loss: 0.16840 [I 2022-10-26 18:26:46] ...optimizers/trijoint.py.151: total_norm: 17.164286 [I 2022-10-26 18:26:52] ...trap/engines/engine.py.224: train: epoch 23 | batch 1090/2304 [I 2022-10-26 18:26:52] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:19 [I 2022-10-26 18:26:52] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00248 [I 2022-10-26 18:26:52] ...trap/engines/engine.py.230: loss: 0.17430 [I 2022-10-26 18:26:52] ...optimizers/trijoint.py.151: total_norm: 18.304615 [I 2022-10-26 18:26:57] ...trap/engines/engine.py.224: train: epoch 23 | batch 1100/2304 [I 2022-10-26 18:26:57] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:10 [I 2022-10-26 18:26:57] ...trap/engines/engine.py.229: process: 0.56791 | load: 0.00250 [I 2022-10-26 18:26:57] ...trap/engines/engine.py.230: loss: 0.18655 [I 2022-10-26 18:26:57] ...optimizers/trijoint.py.151: total_norm: 15.655538 [I 2022-10-26 18:27:03] ...trap/engines/engine.py.224: train: epoch 23 | batch 1110/2304 [I 2022-10-26 18:27:03] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:04 [I 2022-10-26 18:27:03] ...trap/engines/engine.py.229: process: 0.56044 | load: 0.00244 [I 2022-10-26 18:27:03] ...trap/engines/engine.py.230: loss: 0.21090 [I 2022-10-26 18:27:03] ...optimizers/trijoint.py.151: total_norm: 24.610275 [I 2022-10-26 18:27:08] ...trap/engines/engine.py.224: train: epoch 23 | batch 1120/2304 [I 2022-10-26 18:27:08] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:10:58 [I 2022-10-26 18:27:08] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00248 [I 2022-10-26 18:27:08] ...trap/engines/engine.py.230: loss: 0.16883 [I 2022-10-26 18:27:08] ...optimizers/trijoint.py.151: total_norm: 18.920771 [I 2022-10-26 18:27:14] ...trap/engines/engine.py.224: train: epoch 23 | batch 1130/2304 [I 2022-10-26 18:27:14] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:52 [I 2022-10-26 18:27:14] ...trap/engines/engine.py.229: process: 0.54571 | load: 0.00249 [I 2022-10-26 18:27:14] ...trap/engines/engine.py.230: loss: 0.19536 [I 2022-10-26 18:27:14] ...optimizers/trijoint.py.151: total_norm: 19.386616 [I 2022-10-26 18:27:20] ...trap/engines/engine.py.224: train: epoch 23 | batch 1140/2304 [I 2022-10-26 18:27:20] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:47 [I 2022-10-26 18:27:20] ...trap/engines/engine.py.229: process: 0.56384 | load: 0.00246 [I 2022-10-26 18:27:20] ...trap/engines/engine.py.230: loss: 0.19421 [I 2022-10-26 18:27:20] ...optimizers/trijoint.py.151: total_norm: 14.061430 [I 2022-10-26 18:27:25] ...trap/engines/engine.py.224: train: epoch 23 | batch 1150/2304 [I 2022-10-26 18:27:25] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:44 [I 2022-10-26 18:27:25] ...trap/engines/engine.py.229: process: 0.56194 | load: 0.00243 [I 2022-10-26 18:27:25] ...trap/engines/engine.py.230: loss: 0.18645 [I 2022-10-26 18:27:25] ...optimizers/trijoint.py.151: total_norm: 17.742655 [I 2022-10-26 18:27:31] ...trap/engines/engine.py.224: train: epoch 23 | batch 1160/2304 [I 2022-10-26 18:27:31] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:34 [I 2022-10-26 18:27:31] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00247 [I 2022-10-26 18:27:31] ...trap/engines/engine.py.230: loss: 0.15168 [I 2022-10-26 18:27:31] ...optimizers/trijoint.py.151: total_norm: 18.800053 [I 2022-10-26 18:27:36] ...trap/engines/engine.py.224: train: epoch 23 | batch 1170/2304 [I 2022-10-26 18:27:36] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:28 [I 2022-10-26 18:27:36] ...trap/engines/engine.py.229: process: 0.54423 | load: 0.00248 [I 2022-10-26 18:27:36] ...trap/engines/engine.py.230: loss: 0.29301 [I 2022-10-26 18:27:36] ...optimizers/trijoint.py.151: total_norm: 38.668343 [I 2022-10-26 18:27:42] ...trap/engines/engine.py.224: train: epoch 23 | batch 1180/2304 [I 2022-10-26 18:27:42] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:21 [I 2022-10-26 18:27:42] ...trap/engines/engine.py.229: process: 0.54976 | load: 0.00248 [I 2022-10-26 18:27:42] ...trap/engines/engine.py.230: loss: 0.17233 [I 2022-10-26 18:27:42] ...optimizers/trijoint.py.151: total_norm: 43.538559 [I 2022-10-26 18:27:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 1190/2304 [I 2022-10-26 18:27:47] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:16 [I 2022-10-26 18:27:47] ...trap/engines/engine.py.229: process: 0.55851 | load: 0.00247 [I 2022-10-26 18:27:47] ...trap/engines/engine.py.230: loss: 0.16937 [I 2022-10-26 18:27:47] ...optimizers/trijoint.py.151: total_norm: 15.839051 [I 2022-10-26 18:27:53] ...trap/engines/engine.py.224: train: epoch 23 | batch 1200/2304 [I 2022-10-26 18:27:53] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:26 [I 2022-10-26 18:27:53] ...trap/engines/engine.py.229: process: 0.57394 | load: 0.00283 [I 2022-10-26 18:27:53] ...trap/engines/engine.py.230: loss: 0.12754 [I 2022-10-26 18:27:53] ...optimizers/trijoint.py.151: total_norm: 17.739946 [I 2022-10-26 18:27:59] ...trap/engines/engine.py.224: train: epoch 23 | batch 1210/2304 [I 2022-10-26 18:27:59] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:10 [I 2022-10-26 18:27:59] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00249 [I 2022-10-26 18:27:59] ...trap/engines/engine.py.230: loss: 0.19006 [I 2022-10-26 18:27:59] ...optimizers/trijoint.py.151: total_norm: 13.647096 [I 2022-10-26 18:28:04] ...trap/engines/engine.py.224: train: epoch 23 | batch 1220/2304 [I 2022-10-26 18:28:04] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:04 [I 2022-10-26 18:28:04] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00246 [I 2022-10-26 18:28:04] ...trap/engines/engine.py.230: loss: 0.22578 [I 2022-10-26 18:28:04] ...optimizers/trijoint.py.151: total_norm: 19.001181 [I 2022-10-26 18:28:10] ...trap/engines/engine.py.224: train: epoch 23 | batch 1230/2304 [I 2022-10-26 18:28:10] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:58 [I 2022-10-26 18:28:10] ...trap/engines/engine.py.229: process: 0.54978 | load: 0.00252 [I 2022-10-26 18:28:10] ...trap/engines/engine.py.230: loss: 0.20262 [I 2022-10-26 18:28:10] ...optimizers/trijoint.py.151: total_norm: 18.928789 [I 2022-10-26 18:28:15] ...trap/engines/engine.py.224: train: epoch 23 | batch 1240/2304 [I 2022-10-26 18:28:15] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:55 [I 2022-10-26 18:28:15] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00242 [I 2022-10-26 18:28:15] ...trap/engines/engine.py.230: loss: 0.20315 [I 2022-10-26 18:28:15] ...optimizers/trijoint.py.151: total_norm: 22.489227 [I 2022-10-26 18:28:21] ...trap/engines/engine.py.224: train: epoch 23 | batch 1250/2304 [I 2022-10-26 18:28:21] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:49 [I 2022-10-26 18:28:21] ...trap/engines/engine.py.229: process: 0.57166 | load: 0.00247 [I 2022-10-26 18:28:21] ...trap/engines/engine.py.230: loss: 0.18616 [I 2022-10-26 18:28:21] ...optimizers/trijoint.py.151: total_norm: 20.456015 [I 2022-10-26 18:28:27] ...trap/engines/engine.py.224: train: epoch 23 | batch 1260/2304 [I 2022-10-26 18:28:27] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:43 [I 2022-10-26 18:28:27] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00254 [I 2022-10-26 18:28:27] ...trap/engines/engine.py.230: loss: 0.18856 [I 2022-10-26 18:28:27] ...optimizers/trijoint.py.151: total_norm: 18.249075 [I 2022-10-26 18:28:32] ...trap/engines/engine.py.224: train: epoch 23 | batch 1270/2304 [I 2022-10-26 18:28:32] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:36 [I 2022-10-26 18:28:32] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00249 [I 2022-10-26 18:28:32] ...trap/engines/engine.py.230: loss: 0.26877 [I 2022-10-26 18:28:32] ...optimizers/trijoint.py.151: total_norm: 35.844784 [I 2022-10-26 18:28:38] ...trap/engines/engine.py.224: train: epoch 23 | batch 1280/2304 [I 2022-10-26 18:28:38] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:30 [I 2022-10-26 18:28:38] ...trap/engines/engine.py.229: process: 0.56775 | load: 0.00248 [I 2022-10-26 18:28:38] ...trap/engines/engine.py.230: loss: 0.14168 [I 2022-10-26 18:28:38] ...optimizers/trijoint.py.151: total_norm: 11.265038 [I 2022-10-26 18:28:43] ...trap/engines/engine.py.224: train: epoch 23 | batch 1290/2304 [I 2022-10-26 18:28:43] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:29 [I 2022-10-26 18:28:43] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00245 [I 2022-10-26 18:28:43] ...trap/engines/engine.py.230: loss: 0.15922 [I 2022-10-26 18:28:43] ...optimizers/trijoint.py.151: total_norm: 23.916424 [I 2022-10-26 18:28:49] ...trap/engines/engine.py.224: train: epoch 23 | batch 1300/2304 [I 2022-10-26 18:28:49] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:34 [I 2022-10-26 18:28:49] ...trap/engines/engine.py.229: process: 0.58083 | load: 0.00254 [I 2022-10-26 18:28:49] ...trap/engines/engine.py.230: loss: 0.23324 [I 2022-10-26 18:28:49] ...optimizers/trijoint.py.151: total_norm: 18.605719 [I 2022-10-26 18:28:55] ...trap/engines/engine.py.224: train: epoch 23 | batch 1310/2304 [I 2022-10-26 18:28:55] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:18 [I 2022-10-26 18:28:55] ...trap/engines/engine.py.229: process: 0.56344 | load: 0.00248 [I 2022-10-26 18:28:55] ...trap/engines/engine.py.230: loss: 0.23325 [I 2022-10-26 18:28:55] ...optimizers/trijoint.py.151: total_norm: 20.095928 [I 2022-10-26 18:29:00] ...trap/engines/engine.py.224: train: epoch 23 | batch 1320/2304 [I 2022-10-26 18:29:00] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:14 [I 2022-10-26 18:29:00] ...trap/engines/engine.py.229: process: 0.57239 | load: 0.00245 [I 2022-10-26 18:29:00] ...trap/engines/engine.py.230: loss: 0.19057 [I 2022-10-26 18:29:00] ...optimizers/trijoint.py.151: total_norm: 31.198292 [I 2022-10-26 18:29:06] ...trap/engines/engine.py.224: train: epoch 23 | batch 1330/2304 [I 2022-10-26 18:29:06] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:07 [I 2022-10-26 18:29:06] ...trap/engines/engine.py.229: process: 0.54634 | load: 0.00245 [I 2022-10-26 18:29:06] ...trap/engines/engine.py.230: loss: 0.29023 [I 2022-10-26 18:29:06] ...optimizers/trijoint.py.151: total_norm: 15.835285 [I 2022-10-26 18:29:12] ...trap/engines/engine.py.224: train: epoch 23 | batch 1340/2304 [I 2022-10-26 18:29:12] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:09:01 [I 2022-10-26 18:29:12] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00244 [I 2022-10-26 18:29:12] ...trap/engines/engine.py.230: loss: 0.19520 [I 2022-10-26 18:29:12] ...optimizers/trijoint.py.151: total_norm: 18.399824 [I 2022-10-26 18:29:17] ...trap/engines/engine.py.224: train: epoch 23 | batch 1350/2304 [I 2022-10-26 18:29:17] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:55 [I 2022-10-26 18:29:17] ...trap/engines/engine.py.229: process: 0.54690 | load: 0.00254 [I 2022-10-26 18:29:17] ...trap/engines/engine.py.230: loss: 0.15903 [I 2022-10-26 18:29:17] ...optimizers/trijoint.py.151: total_norm: 20.273376 [I 2022-10-26 18:29:23] ...trap/engines/engine.py.224: train: epoch 23 | batch 1360/2304 [I 2022-10-26 18:29:23] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:47 [I 2022-10-26 18:29:23] ...trap/engines/engine.py.229: process: 0.54483 | load: 0.00258 [I 2022-10-26 18:29:23] ...trap/engines/engine.py.230: loss: 0.17533 [I 2022-10-26 18:29:23] ...optimizers/trijoint.py.151: total_norm: 21.226236 [I 2022-10-26 18:29:29] ...trap/engines/engine.py.224: train: epoch 23 | batch 1370/2304 [I 2022-10-26 18:29:29] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:43 [I 2022-10-26 18:29:29] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00243 [I 2022-10-26 18:29:29] ...trap/engines/engine.py.230: loss: 0.20529 [I 2022-10-26 18:29:29] ...optimizers/trijoint.py.151: total_norm: 25.065477 [I 2022-10-26 18:29:34] ...trap/engines/engine.py.224: train: epoch 23 | batch 1380/2304 [I 2022-10-26 18:29:34] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:39 [I 2022-10-26 18:29:34] ...trap/engines/engine.py.229: process: 0.57186 | load: 0.00246 [I 2022-10-26 18:29:34] ...trap/engines/engine.py.230: loss: 0.26164 [I 2022-10-26 18:29:34] ...optimizers/trijoint.py.151: total_norm: 30.538107 [I 2022-10-26 18:29:40] ...trap/engines/engine.py.224: train: epoch 23 | batch 1390/2304 [I 2022-10-26 18:29:40] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:28 [I 2022-10-26 18:29:40] ...trap/engines/engine.py.229: process: 0.55104 | load: 0.00245 [I 2022-10-26 18:29:40] ...trap/engines/engine.py.230: loss: 0.18641 [I 2022-10-26 18:29:40] ...optimizers/trijoint.py.151: total_norm: 21.406853 [I 2022-10-26 18:29:45] ...trap/engines/engine.py.224: train: epoch 23 | batch 1400/2304 [I 2022-10-26 18:29:45] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:28 [I 2022-10-26 18:29:45] ...trap/engines/engine.py.229: process: 0.57647 | load: 0.00245 [I 2022-10-26 18:29:45] ...trap/engines/engine.py.230: loss: 0.20987 [I 2022-10-26 18:29:45] ...optimizers/trijoint.py.151: total_norm: 17.700468 [I 2022-10-26 18:29:51] ...trap/engines/engine.py.224: train: epoch 23 | batch 1410/2304 [I 2022-10-26 18:29:51] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:22 [I 2022-10-26 18:29:51] ...trap/engines/engine.py.229: process: 0.54641 | load: 0.00261 [I 2022-10-26 18:29:51] ...trap/engines/engine.py.230: loss: 0.22056 [I 2022-10-26 18:29:51] ...optimizers/trijoint.py.151: total_norm: 16.775604 [I 2022-10-26 18:29:57] ...trap/engines/engine.py.224: train: epoch 23 | batch 1420/2304 [I 2022-10-26 18:29:57] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:20 [I 2022-10-26 18:29:57] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00248 [I 2022-10-26 18:29:57] ...trap/engines/engine.py.230: loss: 0.13371 [I 2022-10-26 18:29:57] ...optimizers/trijoint.py.151: total_norm: 13.444831 [I 2022-10-26 18:30:02] ...trap/engines/engine.py.224: train: epoch 23 | batch 1430/2304 [I 2022-10-26 18:30:02] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:07 [I 2022-10-26 18:30:02] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00245 [I 2022-10-26 18:30:02] ...trap/engines/engine.py.230: loss: 0.18770 [I 2022-10-26 18:30:02] ...optimizers/trijoint.py.151: total_norm: 18.090277 [I 2022-10-26 18:30:08] ...trap/engines/engine.py.224: train: epoch 23 | batch 1440/2304 [I 2022-10-26 18:30:08] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:05 [I 2022-10-26 18:30:08] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00243 [I 2022-10-26 18:30:08] ...trap/engines/engine.py.230: loss: 0.24196 [I 2022-10-26 18:30:08] ...optimizers/trijoint.py.151: total_norm: 34.504921 [I 2022-10-26 18:30:14] ...trap/engines/engine.py.224: train: epoch 23 | batch 1450/2304 [I 2022-10-26 18:30:14] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:07:57 [I 2022-10-26 18:30:14] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00242 [I 2022-10-26 18:30:14] ...trap/engines/engine.py.230: loss: 0.16916 [I 2022-10-26 18:30:14] ...optimizers/trijoint.py.151: total_norm: 22.686663 [I 2022-10-26 18:30:19] ...trap/engines/engine.py.224: train: epoch 23 | batch 1460/2304 [I 2022-10-26 18:30:19] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:48 [I 2022-10-26 18:30:19] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00242 [I 2022-10-26 18:30:19] ...trap/engines/engine.py.230: loss: 0.20401 [I 2022-10-26 18:30:19] ...optimizers/trijoint.py.151: total_norm: 15.917340 [I 2022-10-26 18:30:25] ...trap/engines/engine.py.224: train: epoch 23 | batch 1470/2304 [I 2022-10-26 18:30:25] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:43 [I 2022-10-26 18:30:25] ...trap/engines/engine.py.229: process: 0.54604 | load: 0.00244 [I 2022-10-26 18:30:25] ...trap/engines/engine.py.230: loss: 0.20433 [I 2022-10-26 18:30:25] ...optimizers/trijoint.py.151: total_norm: 18.523733 [I 2022-10-26 18:30:30] ...trap/engines/engine.py.224: train: epoch 23 | batch 1480/2304 [I 2022-10-26 18:30:30] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:50 [I 2022-10-26 18:30:30] ...trap/engines/engine.py.229: process: 0.55105 | load: 0.00241 [I 2022-10-26 18:30:30] ...trap/engines/engine.py.230: loss: 0.13523 [I 2022-10-26 18:30:30] ...optimizers/trijoint.py.151: total_norm: 15.781074 [I 2022-10-26 18:30:36] ...trap/engines/engine.py.224: train: epoch 23 | batch 1490/2304 [I 2022-10-26 18:30:36] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:35 [I 2022-10-26 18:30:36] ...trap/engines/engine.py.229: process: 0.56881 | load: 0.00239 [I 2022-10-26 18:30:36] ...trap/engines/engine.py.230: loss: 0.18895 [I 2022-10-26 18:30:36] ...optimizers/trijoint.py.151: total_norm: 21.341141 [I 2022-10-26 18:30:42] ...trap/engines/engine.py.224: train: epoch 23 | batch 1500/2304 [I 2022-10-26 18:30:42] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:29 [I 2022-10-26 18:30:42] ...trap/engines/engine.py.229: process: 0.57076 | load: 0.00250 [I 2022-10-26 18:30:42] ...trap/engines/engine.py.230: loss: 0.19976 [I 2022-10-26 18:30:42] ...optimizers/trijoint.py.151: total_norm: 16.221844 [I 2022-10-26 18:30:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 1510/2304 [I 2022-10-26 18:30:47] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:21 [I 2022-10-26 18:30:47] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00244 [I 2022-10-26 18:30:47] ...trap/engines/engine.py.230: loss: 0.19350 [I 2022-10-26 18:30:47] ...optimizers/trijoint.py.151: total_norm: 22.799656 [I 2022-10-26 18:30:53] ...trap/engines/engine.py.224: train: epoch 23 | batch 1520/2304 [I 2022-10-26 18:30:53] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:28 [I 2022-10-26 18:30:53] ...trap/engines/engine.py.229: process: 0.56318 | load: 0.00244 [I 2022-10-26 18:30:53] ...trap/engines/engine.py.230: loss: 0.21870 [I 2022-10-26 18:30:53] ...optimizers/trijoint.py.151: total_norm: 20.975321 [I 2022-10-26 18:30:59] ...trap/engines/engine.py.224: train: epoch 23 | batch 1530/2304 [I 2022-10-26 18:30:59] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:12 [I 2022-10-26 18:30:59] ...trap/engines/engine.py.229: process: 0.55905 | load: 0.00247 [I 2022-10-26 18:30:59] ...trap/engines/engine.py.230: loss: 0.18210 [I 2022-10-26 18:30:59] ...optimizers/trijoint.py.151: total_norm: 18.801600 [I 2022-10-26 18:31:04] ...trap/engines/engine.py.224: train: epoch 23 | batch 1540/2304 [I 2022-10-26 18:31:04] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:07 [I 2022-10-26 18:31:04] ...trap/engines/engine.py.229: process: 0.56833 | load: 0.01645 [I 2022-10-26 18:31:04] ...trap/engines/engine.py.230: loss: 0.14810 [I 2022-10-26 18:31:04] ...optimizers/trijoint.py.151: total_norm: 17.272614 [I 2022-10-26 18:31:10] ...trap/engines/engine.py.224: train: epoch 23 | batch 1550/2304 [I 2022-10-26 18:31:10] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:06:58 [I 2022-10-26 18:31:10] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00245 [I 2022-10-26 18:31:10] ...trap/engines/engine.py.230: loss: 0.24185 [I 2022-10-26 18:31:10] ...optimizers/trijoint.py.151: total_norm: 19.028570 [I 2022-10-26 18:31:15] ...trap/engines/engine.py.224: train: epoch 23 | batch 1560/2304 [I 2022-10-26 18:31:15] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:54 [I 2022-10-26 18:31:15] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00242 [I 2022-10-26 18:31:15] ...trap/engines/engine.py.230: loss: 0.12968 [I 2022-10-26 18:31:15] ...optimizers/trijoint.py.151: total_norm: 8.648486 [I 2022-10-26 18:31:21] ...trap/engines/engine.py.224: train: epoch 23 | batch 1570/2304 [I 2022-10-26 18:31:21] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:47 [I 2022-10-26 18:31:21] ...trap/engines/engine.py.229: process: 0.56314 | load: 0.00244 [I 2022-10-26 18:31:21] ...trap/engines/engine.py.230: loss: 0.19076 [I 2022-10-26 18:31:21] ...optimizers/trijoint.py.151: total_norm: 25.534081 [I 2022-10-26 18:31:26] ...trap/engines/engine.py.224: train: epoch 23 | batch 1580/2304 [I 2022-10-26 18:31:26] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:42 [I 2022-10-26 18:31:26] ...trap/engines/engine.py.229: process: 0.54928 | load: 0.00242 [I 2022-10-26 18:31:26] ...trap/engines/engine.py.230: loss: 0.28096 [I 2022-10-26 18:31:26] ...optimizers/trijoint.py.151: total_norm: 26.321913 [I 2022-10-26 18:31:32] ...trap/engines/engine.py.224: train: epoch 23 | batch 1590/2304 [I 2022-10-26 18:31:32] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:35 [I 2022-10-26 18:31:32] ...trap/engines/engine.py.229: process: 0.55061 | load: 0.00253 [I 2022-10-26 18:31:32] ...trap/engines/engine.py.230: loss: 0.19459 [I 2022-10-26 18:31:32] ...optimizers/trijoint.py.151: total_norm: 25.134825 [I 2022-10-26 18:31:38] ...trap/engines/engine.py.224: train: epoch 23 | batch 1600/2304 [I 2022-10-26 18:31:38] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:30 [I 2022-10-26 18:31:38] ...trap/engines/engine.py.229: process: 0.54635 | load: 0.00248 [I 2022-10-26 18:31:38] ...trap/engines/engine.py.230: loss: 0.19822 [I 2022-10-26 18:31:38] ...optimizers/trijoint.py.151: total_norm: 17.243313 [I 2022-10-26 18:31:43] ...trap/engines/engine.py.224: train: epoch 23 | batch 1610/2304 [I 2022-10-26 18:31:43] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:28 [I 2022-10-26 18:31:43] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00243 [I 2022-10-26 18:31:43] ...trap/engines/engine.py.230: loss: 0.26189 [I 2022-10-26 18:31:43] ...optimizers/trijoint.py.151: total_norm: 28.774721 [I 2022-10-26 18:31:49] ...trap/engines/engine.py.224: train: epoch 23 | batch 1620/2304 [I 2022-10-26 18:31:49] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:21 [I 2022-10-26 18:31:49] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00244 [I 2022-10-26 18:31:49] ...trap/engines/engine.py.230: loss: 0.18867 [I 2022-10-26 18:31:49] ...optimizers/trijoint.py.151: total_norm: 21.843945 [I 2022-10-26 18:31:54] ...trap/engines/engine.py.224: train: epoch 23 | batch 1630/2304 [I 2022-10-26 18:31:54] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:16 [I 2022-10-26 18:31:54] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00243 [I 2022-10-26 18:31:54] ...trap/engines/engine.py.230: loss: 0.16501 [I 2022-10-26 18:31:54] ...optimizers/trijoint.py.151: total_norm: 19.059967 [I 2022-10-26 18:32:00] ...trap/engines/engine.py.224: train: epoch 23 | batch 1640/2304 [I 2022-10-26 18:32:00] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:09 [I 2022-10-26 18:32:00] ...trap/engines/engine.py.229: process: 0.55975 | load: 0.00241 [I 2022-10-26 18:32:00] ...trap/engines/engine.py.230: loss: 0.27968 [I 2022-10-26 18:32:00] ...optimizers/trijoint.py.151: total_norm: 24.567993 [I 2022-10-26 18:32:05] ...trap/engines/engine.py.224: train: epoch 23 | batch 1650/2304 [I 2022-10-26 18:32:05] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:05 [I 2022-10-26 18:32:05] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00245 [I 2022-10-26 18:32:05] ...trap/engines/engine.py.230: loss: 0.14001 [I 2022-10-26 18:32:05] ...optimizers/trijoint.py.151: total_norm: 13.796986 [I 2022-10-26 18:32:11] ...trap/engines/engine.py.224: train: epoch 23 | batch 1660/2304 [I 2022-10-26 18:32:11] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:57 [I 2022-10-26 18:32:11] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00243 [I 2022-10-26 18:32:11] ...trap/engines/engine.py.230: loss: 0.18139 [I 2022-10-26 18:32:11] ...optimizers/trijoint.py.151: total_norm: 20.136875 [I 2022-10-26 18:32:17] ...trap/engines/engine.py.224: train: epoch 23 | batch 1670/2304 [I 2022-10-26 18:32:17] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:05:54 [I 2022-10-26 18:32:17] ...trap/engines/engine.py.229: process: 0.55851 | load: 0.00241 [I 2022-10-26 18:32:17] ...trap/engines/engine.py.230: loss: 0.23146 [I 2022-10-26 18:32:17] ...optimizers/trijoint.py.151: total_norm: 22.676983 [I 2022-10-26 18:32:22] ...trap/engines/engine.py.224: train: epoch 23 | batch 1680/2304 [I 2022-10-26 18:32:22] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:45 [I 2022-10-26 18:32:22] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00242 [I 2022-10-26 18:32:22] ...trap/engines/engine.py.230: loss: 0.21004 [I 2022-10-26 18:32:22] ...optimizers/trijoint.py.151: total_norm: 23.619993 [I 2022-10-26 18:32:28] ...trap/engines/engine.py.224: train: epoch 23 | batch 1690/2304 [I 2022-10-26 18:32:28] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:39 [I 2022-10-26 18:32:28] ...trap/engines/engine.py.229: process: 0.54817 | load: 0.00243 [I 2022-10-26 18:32:28] ...trap/engines/engine.py.230: loss: 0.19013 [I 2022-10-26 18:32:28] ...optimizers/trijoint.py.151: total_norm: 20.268303 [I 2022-10-26 18:32:33] ...trap/engines/engine.py.224: train: epoch 23 | batch 1700/2304 [I 2022-10-26 18:32:33] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:36 [I 2022-10-26 18:32:33] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00242 [I 2022-10-26 18:32:33] ...trap/engines/engine.py.230: loss: 0.10105 [I 2022-10-26 18:32:33] ...optimizers/trijoint.py.151: total_norm: 10.002832 [I 2022-10-26 18:32:39] ...trap/engines/engine.py.224: train: epoch 23 | batch 1710/2304 [I 2022-10-26 18:32:39] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:29 [I 2022-10-26 18:32:39] ...trap/engines/engine.py.229: process: 0.54718 | load: 0.00251 [I 2022-10-26 18:32:39] ...trap/engines/engine.py.230: loss: 0.19835 [I 2022-10-26 18:32:39] ...optimizers/trijoint.py.151: total_norm: 21.920753 [I 2022-10-26 18:32:44] ...trap/engines/engine.py.224: train: epoch 23 | batch 1720/2304 [I 2022-10-26 18:32:44] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:23 [I 2022-10-26 18:32:44] ...trap/engines/engine.py.229: process: 0.56062 | load: 0.00269 [I 2022-10-26 18:32:44] ...trap/engines/engine.py.230: loss: 0.16332 [I 2022-10-26 18:32:44] ...optimizers/trijoint.py.151: total_norm: 17.986675 [I 2022-10-26 18:32:50] ...trap/engines/engine.py.224: train: epoch 23 | batch 1730/2304 [I 2022-10-26 18:32:50] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:20 [I 2022-10-26 18:32:50] ...trap/engines/engine.py.229: process: 0.56316 | load: 0.00245 [I 2022-10-26 18:32:50] ...trap/engines/engine.py.230: loss: 0.21311 [I 2022-10-26 18:32:50] ...optimizers/trijoint.py.151: total_norm: 29.978752 [I 2022-10-26 18:32:56] ...trap/engines/engine.py.224: train: epoch 23 | batch 1740/2304 [I 2022-10-26 18:32:56] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:14 [I 2022-10-26 18:32:56] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00247 [I 2022-10-26 18:32:56] ...trap/engines/engine.py.230: loss: 0.17873 [I 2022-10-26 18:32:56] ...optimizers/trijoint.py.151: total_norm: 14.123841 [I 2022-10-26 18:33:01] ...trap/engines/engine.py.224: train: epoch 23 | batch 1750/2304 [I 2022-10-26 18:33:01] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:07 [I 2022-10-26 18:33:01] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00246 [I 2022-10-26 18:33:01] ...trap/engines/engine.py.230: loss: 0.18479 [I 2022-10-26 18:33:01] ...optimizers/trijoint.py.151: total_norm: 13.069842 [I 2022-10-26 18:33:07] ...trap/engines/engine.py.224: train: epoch 23 | batch 1760/2304 [I 2022-10-26 18:33:07] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:01 [I 2022-10-26 18:33:07] ...trap/engines/engine.py.229: process: 0.54761 | load: 0.00253 [I 2022-10-26 18:33:07] ...trap/engines/engine.py.230: loss: 0.14289 [I 2022-10-26 18:33:07] ...optimizers/trijoint.py.151: total_norm: 23.252048 [I 2022-10-26 18:33:12] ...trap/engines/engine.py.224: train: epoch 23 | batch 1770/2304 [I 2022-10-26 18:33:12] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:04:59 [I 2022-10-26 18:33:12] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00250 [I 2022-10-26 18:33:12] ...trap/engines/engine.py.230: loss: 0.19562 [I 2022-10-26 18:33:12] ...optimizers/trijoint.py.151: total_norm: 25.871147 [I 2022-10-26 18:33:18] ...trap/engines/engine.py.224: train: epoch 23 | batch 1780/2304 [I 2022-10-26 18:33:18] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:50 [I 2022-10-26 18:33:18] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00253 [I 2022-10-26 18:33:18] ...trap/engines/engine.py.230: loss: 0.17192 [I 2022-10-26 18:33:18] ...optimizers/trijoint.py.151: total_norm: 32.108730 [I 2022-10-26 18:33:24] ...trap/engines/engine.py.224: train: epoch 23 | batch 1790/2304 [I 2022-10-26 18:33:24] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:48 [I 2022-10-26 18:33:24] ...trap/engines/engine.py.229: process: 0.56584 | load: 0.00244 [I 2022-10-26 18:33:24] ...trap/engines/engine.py.230: loss: 0.15392 [I 2022-10-26 18:33:24] ...optimizers/trijoint.py.151: total_norm: 15.666316 [I 2022-10-26 18:33:29] ...trap/engines/engine.py.224: train: epoch 23 | batch 1800/2304 [I 2022-10-26 18:33:29] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:39 [I 2022-10-26 18:33:29] ...trap/engines/engine.py.229: process: 0.54562 | load: 0.00263 [I 2022-10-26 18:33:29] ...trap/engines/engine.py.230: loss: 0.17738 [I 2022-10-26 18:33:29] ...optimizers/trijoint.py.151: total_norm: 17.555328 [I 2022-10-26 18:33:35] ...trap/engines/engine.py.224: train: epoch 23 | batch 1810/2304 [I 2022-10-26 18:33:35] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:36 [I 2022-10-26 18:33:35] ...trap/engines/engine.py.229: process: 0.55145 | load: 0.00249 [I 2022-10-26 18:33:35] ...trap/engines/engine.py.230: loss: 0.15775 [I 2022-10-26 18:33:35] ...optimizers/trijoint.py.151: total_norm: 18.287834 [I 2022-10-26 18:33:40] ...trap/engines/engine.py.224: train: epoch 23 | batch 1820/2304 [I 2022-10-26 18:33:40] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:28 [I 2022-10-26 18:33:40] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00256 [I 2022-10-26 18:33:40] ...trap/engines/engine.py.230: loss: 0.17680 [I 2022-10-26 18:33:40] ...optimizers/trijoint.py.151: total_norm: 14.570484 [I 2022-10-26 18:33:46] ...trap/engines/engine.py.224: train: epoch 23 | batch 1830/2304 [I 2022-10-26 18:33:46] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:25 [I 2022-10-26 18:33:46] ...trap/engines/engine.py.229: process: 0.56112 | load: 0.00244 [I 2022-10-26 18:33:46] ...trap/engines/engine.py.230: loss: 0.22151 [I 2022-10-26 18:33:46] ...optimizers/trijoint.py.151: total_norm: 18.624056 [I 2022-10-26 18:33:51] ...trap/engines/engine.py.224: train: epoch 23 | batch 1840/2304 [I 2022-10-26 18:33:51] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:18 [I 2022-10-26 18:33:51] ...trap/engines/engine.py.229: process: 0.54408 | load: 0.00244 [I 2022-10-26 18:33:51] ...trap/engines/engine.py.230: loss: 0.24213 [I 2022-10-26 18:33:51] ...optimizers/trijoint.py.151: total_norm: 18.949566 [I 2022-10-26 18:33:57] ...trap/engines/engine.py.224: train: epoch 23 | batch 1850/2304 [I 2022-10-26 18:33:57] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:11 [I 2022-10-26 18:33:57] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00246 [I 2022-10-26 18:33:57] ...trap/engines/engine.py.230: loss: 0.18761 [I 2022-10-26 18:33:57] ...optimizers/trijoint.py.151: total_norm: 21.117306 [I 2022-10-26 18:34:03] ...trap/engines/engine.py.224: train: epoch 23 | batch 1860/2304 [I 2022-10-26 18:34:03] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:06 [I 2022-10-26 18:34:03] ...trap/engines/engine.py.229: process: 0.55858 | load: 0.00251 [I 2022-10-26 18:34:03] ...trap/engines/engine.py.230: loss: 0.29272 [I 2022-10-26 18:34:03] ...optimizers/trijoint.py.151: total_norm: 23.930365 [I 2022-10-26 18:34:08] ...trap/engines/engine.py.224: train: epoch 23 | batch 1870/2304 [I 2022-10-26 18:34:08] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:03:59 [I 2022-10-26 18:34:08] ...trap/engines/engine.py.229: process: 0.54816 | load: 0.00243 [I 2022-10-26 18:34:08] ...trap/engines/engine.py.230: loss: 0.14722 [I 2022-10-26 18:34:08] ...optimizers/trijoint.py.151: total_norm: 16.788448 [I 2022-10-26 18:34:14] ...trap/engines/engine.py.224: train: epoch 23 | batch 1880/2304 [I 2022-10-26 18:34:14] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:55 [I 2022-10-26 18:34:14] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00248 [I 2022-10-26 18:34:14] ...trap/engines/engine.py.230: loss: 0.20184 [I 2022-10-26 18:34:14] ...optimizers/trijoint.py.151: total_norm: 18.673883 [I 2022-10-26 18:34:19] ...trap/engines/engine.py.224: train: epoch 23 | batch 1890/2304 [I 2022-10-26 18:34:19] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:03:51 [I 2022-10-26 18:34:19] ...trap/engines/engine.py.229: process: 0.56573 | load: 0.00245 [I 2022-10-26 18:34:19] ...trap/engines/engine.py.230: loss: 0.19360 [I 2022-10-26 18:34:19] ...optimizers/trijoint.py.151: total_norm: 20.850956 [I 2022-10-26 18:34:25] ...trap/engines/engine.py.224: train: epoch 23 | batch 1900/2304 [I 2022-10-26 18:34:25] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:44 [I 2022-10-26 18:34:25] ...trap/engines/engine.py.229: process: 0.55945 | load: 0.00241 [I 2022-10-26 18:34:25] ...trap/engines/engine.py.230: loss: 0.19498 [I 2022-10-26 18:34:25] ...optimizers/trijoint.py.151: total_norm: 20.584675 [I 2022-10-26 18:34:30] ...trap/engines/engine.py.224: train: epoch 23 | batch 1910/2304 [I 2022-10-26 18:34:30] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:39 [I 2022-10-26 18:34:30] ...trap/engines/engine.py.229: process: 0.56166 | load: 0.00242 [I 2022-10-26 18:34:30] ...trap/engines/engine.py.230: loss: 0.26357 [I 2022-10-26 18:34:30] ...optimizers/trijoint.py.151: total_norm: 29.969648 [I 2022-10-26 18:34:36] ...trap/engines/engine.py.224: train: epoch 23 | batch 1920/2304 [I 2022-10-26 18:34:36] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:34 [I 2022-10-26 18:34:36] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00247 [I 2022-10-26 18:34:36] ...trap/engines/engine.py.230: loss: 0.14928 [I 2022-10-26 18:34:36] ...optimizers/trijoint.py.151: total_norm: 14.869328 [I 2022-10-26 18:34:42] ...trap/engines/engine.py.224: train: epoch 23 | batch 1930/2304 [I 2022-10-26 18:34:42] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:28 [I 2022-10-26 18:34:42] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00246 [I 2022-10-26 18:34:42] ...trap/engines/engine.py.230: loss: 0.18394 [I 2022-10-26 18:34:42] ...optimizers/trijoint.py.151: total_norm: 12.188481 [I 2022-10-26 18:34:47] ...trap/engines/engine.py.224: train: epoch 23 | batch 1940/2304 [I 2022-10-26 18:34:47] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:22 [I 2022-10-26 18:34:47] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00243 [I 2022-10-26 18:34:47] ...trap/engines/engine.py.230: loss: 0.23010 [I 2022-10-26 18:34:47] ...optimizers/trijoint.py.151: total_norm: 22.618242 [I 2022-10-26 18:34:53] ...trap/engines/engine.py.224: train: epoch 23 | batch 1950/2304 [I 2022-10-26 18:34:53] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:17 [I 2022-10-26 18:34:53] ...trap/engines/engine.py.229: process: 0.55055 | load: 0.00247 [I 2022-10-26 18:34:53] ...trap/engines/engine.py.230: loss: 0.14088 [I 2022-10-26 18:34:53] ...optimizers/trijoint.py.151: total_norm: 17.739008 [I 2022-10-26 18:34:58] ...trap/engines/engine.py.224: train: epoch 23 | batch 1960/2304 [I 2022-10-26 18:34:58] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:11 [I 2022-10-26 18:34:58] ...trap/engines/engine.py.229: process: 0.54896 | load: 0.00245 [I 2022-10-26 18:34:58] ...trap/engines/engine.py.230: loss: 0.30403 [I 2022-10-26 18:34:58] ...optimizers/trijoint.py.151: total_norm: 21.701244 [I 2022-10-26 18:35:04] ...trap/engines/engine.py.224: train: epoch 23 | batch 1970/2304 [I 2022-10-26 18:35:04] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:06 [I 2022-10-26 18:35:04] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00248 [I 2022-10-26 18:35:04] ...trap/engines/engine.py.230: loss: 0.38449 [I 2022-10-26 18:35:04] ...optimizers/trijoint.py.151: total_norm: 24.898701 [I 2022-10-26 18:35:10] ...trap/engines/engine.py.224: train: epoch 23 | batch 1980/2304 [I 2022-10-26 18:35:10] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:02:59 [I 2022-10-26 18:35:10] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00245 [I 2022-10-26 18:35:10] ...trap/engines/engine.py.230: loss: 0.20711 [I 2022-10-26 18:35:10] ...optimizers/trijoint.py.151: total_norm: 30.481277 [I 2022-10-26 18:35:15] ...trap/engines/engine.py.224: train: epoch 23 | batch 1990/2304 [I 2022-10-26 18:35:15] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:54 [I 2022-10-26 18:35:15] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00245 [I 2022-10-26 18:35:15] ...trap/engines/engine.py.230: loss: 0.29828 [I 2022-10-26 18:35:15] ...optimizers/trijoint.py.151: total_norm: 34.502693 [I 2022-10-26 18:35:21] ...trap/engines/engine.py.224: train: epoch 23 | batch 2000/2304 [I 2022-10-26 18:35:21] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:02:49 [I 2022-10-26 18:35:21] ...trap/engines/engine.py.229: process: 0.56751 | load: 0.00241 [I 2022-10-26 18:35:21] ...trap/engines/engine.py.230: loss: 0.14338 [I 2022-10-26 18:35:21] ...optimizers/trijoint.py.151: total_norm: 10.332967 [I 2022-10-26 18:35:26] ...trap/engines/engine.py.224: train: epoch 23 | batch 2010/2304 [I 2022-10-26 18:35:26] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:02:43 [I 2022-10-26 18:35:26] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00244 [I 2022-10-26 18:35:26] ...trap/engines/engine.py.230: loss: 0.22004 [I 2022-10-26 18:35:26] ...optimizers/trijoint.py.151: total_norm: 16.310349 [I 2022-10-26 18:35:32] ...trap/engines/engine.py.224: train: epoch 23 | batch 2020/2304 [I 2022-10-26 18:35:32] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:38 [I 2022-10-26 18:35:32] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00242 [I 2022-10-26 18:35:32] ...trap/engines/engine.py.230: loss: 0.16875 [I 2022-10-26 18:35:32] ...optimizers/trijoint.py.151: total_norm: 19.367842 [I 2022-10-26 18:35:37] ...trap/engines/engine.py.224: train: epoch 23 | batch 2030/2304 [I 2022-10-26 18:35:37] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:33 [I 2022-10-26 18:35:37] ...trap/engines/engine.py.229: process: 0.58160 | load: 0.00253 [I 2022-10-26 18:35:37] ...trap/engines/engine.py.230: loss: 0.20196 [I 2022-10-26 18:35:37] ...optimizers/trijoint.py.151: total_norm: 17.873108 [I 2022-10-26 18:35:43] ...trap/engines/engine.py.224: train: epoch 23 | batch 2040/2304 [I 2022-10-26 18:35:43] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:27 [I 2022-10-26 18:35:43] ...trap/engines/engine.py.229: process: 0.55298 | load: 0.00245 [I 2022-10-26 18:35:43] ...trap/engines/engine.py.230: loss: 0.16129 [I 2022-10-26 18:35:43] ...optimizers/trijoint.py.151: total_norm: 14.873060 [I 2022-10-26 18:35:49] ...trap/engines/engine.py.224: train: epoch 23 | batch 2050/2304 [I 2022-10-26 18:35:49] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:20 [I 2022-10-26 18:35:49] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00242 [I 2022-10-26 18:35:49] ...trap/engines/engine.py.230: loss: 0.22532 [I 2022-10-26 18:35:49] ...optimizers/trijoint.py.151: total_norm: 64.067299 [I 2022-10-26 18:35:54] ...trap/engines/engine.py.224: train: epoch 23 | batch 2060/2304 [I 2022-10-26 18:35:54] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:15 [I 2022-10-26 18:35:54] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00244 [I 2022-10-26 18:35:54] ...trap/engines/engine.py.230: loss: 0.13629 [I 2022-10-26 18:35:54] ...optimizers/trijoint.py.151: total_norm: 10.348735 [I 2022-10-26 18:36:00] ...trap/engines/engine.py.224: train: epoch 23 | batch 2070/2304 [I 2022-10-26 18:36:00] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:10 [I 2022-10-26 18:36:00] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00242 [I 2022-10-26 18:36:00] ...trap/engines/engine.py.230: loss: 0.15021 [I 2022-10-26 18:36:00] ...optimizers/trijoint.py.151: total_norm: 19.943419 [I 2022-10-26 18:36:05] ...trap/engines/engine.py.224: train: epoch 23 | batch 2080/2304 [I 2022-10-26 18:36:05] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:04 [I 2022-10-26 18:36:05] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00250 [I 2022-10-26 18:36:05] ...trap/engines/engine.py.230: loss: 0.14559 [I 2022-10-26 18:36:05] ...optimizers/trijoint.py.151: total_norm: 18.549923 [I 2022-10-26 18:36:11] ...trap/engines/engine.py.224: train: epoch 23 | batch 2090/2304 [I 2022-10-26 18:36:11] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:01:58 [I 2022-10-26 18:36:11] ...trap/engines/engine.py.229: process: 0.54576 | load: 0.00241 [I 2022-10-26 18:36:11] ...trap/engines/engine.py.230: loss: 0.23826 [I 2022-10-26 18:36:11] ...optimizers/trijoint.py.151: total_norm: 25.417093 [I 2022-10-26 18:36:17] ...trap/engines/engine.py.224: train: epoch 23 | batch 2100/2304 [I 2022-10-26 18:36:17] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:53 [I 2022-10-26 18:36:17] ...trap/engines/engine.py.229: process: 0.56681 | load: 0.00238 [I 2022-10-26 18:36:17] ...trap/engines/engine.py.230: loss: 0.17553 [I 2022-10-26 18:36:17] ...optimizers/trijoint.py.151: total_norm: 24.788004 [I 2022-10-26 18:36:22] ...trap/engines/engine.py.224: train: epoch 23 | batch 2110/2304 [I 2022-10-26 18:36:22] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:01:47 [I 2022-10-26 18:36:22] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00267 [I 2022-10-26 18:36:22] ...trap/engines/engine.py.230: loss: 0.20704 [I 2022-10-26 18:36:22] ...optimizers/trijoint.py.151: total_norm: 31.275055 [I 2022-10-26 18:36:28] ...trap/engines/engine.py.224: train: epoch 23 | batch 2120/2304 [I 2022-10-26 18:36:28] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:43 [I 2022-10-26 18:36:28] ...trap/engines/engine.py.229: process: 0.55005 | load: 0.00242 [I 2022-10-26 18:36:28] ...trap/engines/engine.py.230: loss: 0.28119 [I 2022-10-26 18:36:28] ...optimizers/trijoint.py.151: total_norm: 21.125620 [I 2022-10-26 18:36:33] ...trap/engines/engine.py.224: train: epoch 23 | batch 2130/2304 [I 2022-10-26 18:36:33] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:37 [I 2022-10-26 18:36:33] ...trap/engines/engine.py.229: process: 0.56499 | load: 0.00254 [I 2022-10-26 18:36:33] ...trap/engines/engine.py.230: loss: 0.27120 [I 2022-10-26 18:36:33] ...optimizers/trijoint.py.151: total_norm: 28.614634 [I 2022-10-26 18:36:39] ...trap/engines/engine.py.224: train: epoch 23 | batch 2140/2304 [I 2022-10-26 18:36:39] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:30 [I 2022-10-26 18:36:39] ...trap/engines/engine.py.229: process: 0.54768 | load: 0.00248 [I 2022-10-26 18:36:39] ...trap/engines/engine.py.230: loss: 0.20950 [I 2022-10-26 18:36:39] ...optimizers/trijoint.py.151: total_norm: 19.103857 [I 2022-10-26 18:36:45] ...trap/engines/engine.py.224: train: epoch 23 | batch 2150/2304 [I 2022-10-26 18:36:45] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:25 [I 2022-10-26 18:36:45] ...trap/engines/engine.py.229: process: 0.55014 | load: 0.00243 [I 2022-10-26 18:36:45] ...trap/engines/engine.py.230: loss: 0.24998 [I 2022-10-26 18:36:45] ...optimizers/trijoint.py.151: total_norm: 26.972034 [I 2022-10-26 18:36:50] ...trap/engines/engine.py.224: train: epoch 23 | batch 2160/2304 [I 2022-10-26 18:36:50] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:20 [I 2022-10-26 18:36:50] ...trap/engines/engine.py.229: process: 0.56887 | load: 0.00246 [I 2022-10-26 18:36:50] ...trap/engines/engine.py.230: loss: 0.19401 [I 2022-10-26 18:36:50] ...optimizers/trijoint.py.151: total_norm: 11.868476 [I 2022-10-26 18:36:56] ...trap/engines/engine.py.224: train: epoch 23 | batch 2170/2304 [I 2022-10-26 18:36:56] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:14 [I 2022-10-26 18:36:56] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00244 [I 2022-10-26 18:36:56] ...trap/engines/engine.py.230: loss: 0.16315 [I 2022-10-26 18:36:56] ...optimizers/trijoint.py.151: total_norm: 20.590801 [I 2022-10-26 18:37:02] ...trap/engines/engine.py.224: train: epoch 23 | batch 2180/2304 [I 2022-10-26 18:37:02] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:09 [I 2022-10-26 18:37:02] ...trap/engines/engine.py.229: process: 0.56366 | load: 0.00247 [I 2022-10-26 18:37:02] ...trap/engines/engine.py.230: loss: 0.18486 [I 2022-10-26 18:37:02] ...optimizers/trijoint.py.151: total_norm: 18.464823 [I 2022-10-26 18:37:07] ...trap/engines/engine.py.224: train: epoch 23 | batch 2190/2304 [I 2022-10-26 18:37:07] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:03 [I 2022-10-26 18:37:07] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00246 [I 2022-10-26 18:37:07] ...trap/engines/engine.py.230: loss: 0.14528 [I 2022-10-26 18:37:07] ...optimizers/trijoint.py.151: total_norm: 15.202085 [I 2022-10-26 18:37:13] ...trap/engines/engine.py.224: train: epoch 23 | batch 2200/2304 [I 2022-10-26 18:37:13] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:57 [I 2022-10-26 18:37:13] ...trap/engines/engine.py.229: process: 0.56166 | load: 0.00242 [I 2022-10-26 18:37:13] ...trap/engines/engine.py.230: loss: 0.15549 [I 2022-10-26 18:37:13] ...optimizers/trijoint.py.151: total_norm: 16.031796 [I 2022-10-26 18:37:18] ...trap/engines/engine.py.224: train: epoch 23 | batch 2210/2304 [I 2022-10-26 18:37:18] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:00:52 [I 2022-10-26 18:37:18] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00245 [I 2022-10-26 18:37:18] ...trap/engines/engine.py.230: loss: 0.26789 [I 2022-10-26 18:37:18] ...optimizers/trijoint.py.151: total_norm: 21.369556 [I 2022-10-26 18:37:24] ...trap/engines/engine.py.224: train: epoch 23 | batch 2220/2304 [I 2022-10-26 18:37:24] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:00:46 [I 2022-10-26 18:37:24] ...trap/engines/engine.py.229: process: 0.56361 | load: 0.00244 [I 2022-10-26 18:37:24] ...trap/engines/engine.py.230: loss: 0.17789 [I 2022-10-26 18:37:24] ...optimizers/trijoint.py.151: total_norm: 19.975697 [I 2022-10-26 18:37:29] ...trap/engines/engine.py.224: train: epoch 23 | batch 2230/2304 [I 2022-10-26 18:37:29] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:41 [I 2022-10-26 18:37:29] ...trap/engines/engine.py.229: process: 0.55121 | load: 0.00254 [I 2022-10-26 18:37:29] ...trap/engines/engine.py.230: loss: 0.17466 [I 2022-10-26 18:37:29] ...optimizers/trijoint.py.151: total_norm: 31.401070 [I 2022-10-26 18:37:35] ...trap/engines/engine.py.224: train: epoch 23 | batch 2240/2304 [I 2022-10-26 18:37:35] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:35 [I 2022-10-26 18:37:35] ...trap/engines/engine.py.229: process: 0.56524 | load: 0.00249 [I 2022-10-26 18:37:35] ...trap/engines/engine.py.230: loss: 0.30017 [I 2022-10-26 18:37:35] ...optimizers/trijoint.py.151: total_norm: 33.008797 [I 2022-10-26 18:37:41] ...trap/engines/engine.py.224: train: epoch 23 | batch 2250/2304 [I 2022-10-26 18:37:41] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:30 [I 2022-10-26 18:37:41] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00251 [I 2022-10-26 18:37:41] ...trap/engines/engine.py.230: loss: 0.16699 [I 2022-10-26 18:37:41] ...optimizers/trijoint.py.151: total_norm: 18.683420 [I 2022-10-26 18:37:46] ...trap/engines/engine.py.224: train: epoch 23 | batch 2260/2304 [I 2022-10-26 18:37:46] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:24 [I 2022-10-26 18:37:46] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00245 [I 2022-10-26 18:37:46] ...trap/engines/engine.py.230: loss: 0.13955 [I 2022-10-26 18:37:46] ...optimizers/trijoint.py.151: total_norm: 16.449579 [I 2022-10-26 18:37:52] ...trap/engines/engine.py.224: train: epoch 23 | batch 2270/2304 [I 2022-10-26 18:37:52] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:18 [I 2022-10-26 18:37:52] ...trap/engines/engine.py.229: process: 0.56475 | load: 0.00253 [I 2022-10-26 18:37:52] ...trap/engines/engine.py.230: loss: 0.11617 [I 2022-10-26 18:37:52] ...optimizers/trijoint.py.151: total_norm: 11.061344 [I 2022-10-26 18:37:57] ...trap/engines/engine.py.224: train: epoch 23 | batch 2280/2304 [I 2022-10-26 18:37:57] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:13 [I 2022-10-26 18:37:57] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00245 [I 2022-10-26 18:37:57] ...trap/engines/engine.py.230: loss: 0.21750 [I 2022-10-26 18:37:57] ...optimizers/trijoint.py.151: total_norm: 35.388226 [I 2022-10-26 18:38:03] ...trap/engines/engine.py.224: train: epoch 23 | batch 2290/2304 [I 2022-10-26 18:38:03] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:07 [I 2022-10-26 18:38:03] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00244 [I 2022-10-26 18:38:03] ...trap/engines/engine.py.230: loss: 0.20348 [I 2022-10-26 18:38:03] ...optimizers/trijoint.py.151: total_norm: 16.952671 [I 2022-10-26 18:38:09] ...trap/engines/engine.py.224: train: epoch 23 | batch 2300/2304 [I 2022-10-26 18:38:09] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:02 [I 2022-10-26 18:38:09] ...trap/engines/engine.py.229: process: 0.55268 | load: 0.00014 [I 2022-10-26 18:38:09] ...trap/engines/engine.py.230: loss: 0.12061 [I 2022-10-26 18:38:09] ...optimizers/trijoint.py.151: total_norm: 14.039838 [I 2022-10-26 18:38:11] ...trap/engines/engine.py.224: train: epoch 23 | batch 2304/2304 [I 2022-10-26 18:38:11] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:00 [I 2022-10-26 18:38:11] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00015 [I 2022-10-26 18:38:11] ...trap/engines/engine.py.230: loss: 0.13906 [I 2022-10-26 18:38:11] ...optimizers/trijoint.py.151: total_norm: 20.789667 [S 2022-10-26 18:38:12] ...trap/engines/engine.py.129: train_epoch.epoch: 23 [S 2022-10-26 18:38:12] ...trap/engines/engine.py.129: train_epoch.loss: 0.19532408390089126 [I 2022-10-26 18:38:12] ...s/criterions/triplet.py.84: margin = 0.1700000000000001 [I 2022-10-26 18:38:12] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 23 [I 2022-10-26 18:38:12] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 18:38:14] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 18:38:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 18:38:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 18:38:15] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 18:38:15] ...trap/engines/engine.py.321: eval: epoch 23 | batch 0/510 [I 2022-10-26 18:38:15] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:25:12 [I 2022-10-26 18:38:15] ...trap/engines/engine.py.326: process: 2.96638 | load: 2.56304 [I 2022-10-26 18:38:19] ...trap/engines/engine.py.321: eval: epoch 23 | batch 10/510 [I 2022-10-26 18:38:19] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:07 [I 2022-10-26 18:38:19] ...trap/engines/engine.py.326: process: 0.32781 | load: 0.00019 [I 2022-10-26 18:38:22] ...trap/engines/engine.py.321: eval: epoch 23 | batch 20/510 [I 2022-10-26 18:38:22] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:57 [I 2022-10-26 18:38:22] ...trap/engines/engine.py.326: process: 0.32947 | load: 0.00020 [I 2022-10-26 18:38:25] ...trap/engines/engine.py.321: eval: epoch 23 | batch 30/510 [I 2022-10-26 18:38:25] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:43 [I 2022-10-26 18:38:25] ...trap/engines/engine.py.326: process: 0.33747 | load: 0.00021 [I 2022-10-26 18:38:29] ...trap/engines/engine.py.321: eval: epoch 23 | batch 40/510 [I 2022-10-26 18:38:29] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:40 [I 2022-10-26 18:38:29] ...trap/engines/engine.py.326: process: 0.32464 | load: 0.00020 [I 2022-10-26 18:38:32] ...trap/engines/engine.py.321: eval: epoch 23 | batch 50/510 [I 2022-10-26 18:38:32] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:34 [I 2022-10-26 18:38:32] ...trap/engines/engine.py.326: process: 0.32392 | load: 0.00021 [I 2022-10-26 18:38:35] ...trap/engines/engine.py.321: eval: epoch 23 | batch 60/510 [I 2022-10-26 18:38:35] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:31 [I 2022-10-26 18:38:35] ...trap/engines/engine.py.326: process: 0.33511 | load: 0.00018 [I 2022-10-26 18:38:39] ...trap/engines/engine.py.321: eval: epoch 23 | batch 70/510 [I 2022-10-26 18:38:39] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:27 [I 2022-10-26 18:38:39] ...trap/engines/engine.py.326: process: 0.33184 | load: 0.00019 [I 2022-10-26 18:38:42] ...trap/engines/engine.py.321: eval: epoch 23 | batch 80/510 [I 2022-10-26 18:38:42] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:23 [I 2022-10-26 18:38:42] ...trap/engines/engine.py.326: process: 0.32689 | load: 0.00022 [I 2022-10-26 18:38:46] ...trap/engines/engine.py.321: eval: epoch 23 | batch 90/510 [I 2022-10-26 18:38:46] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-26 18:38:46] ...trap/engines/engine.py.326: process: 0.33196 | load: 0.00021 [I 2022-10-26 18:38:49] ...trap/engines/engine.py.321: eval: epoch 23 | batch 100/510 [I 2022-10-26 18:38:49] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:15 [I 2022-10-26 18:38:49] ...trap/engines/engine.py.326: process: 0.28792 | load: 0.00024 [I 2022-10-26 18:38:52] ...trap/engines/engine.py.321: eval: epoch 23 | batch 110/510 [I 2022-10-26 18:38:52] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-26 18:38:52] ...trap/engines/engine.py.326: process: 0.28111 | load: 0.00015 [I 2022-10-26 18:38:55] ...trap/engines/engine.py.321: eval: epoch 23 | batch 120/510 [I 2022-10-26 18:38:55] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-26 18:38:55] ...trap/engines/engine.py.326: process: 0.29180 | load: 0.00023 [I 2022-10-26 18:38:58] ...trap/engines/engine.py.321: eval: epoch 23 | batch 130/510 [I 2022-10-26 18:38:58] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:49 [I 2022-10-26 18:38:58] ...trap/engines/engine.py.326: process: 0.28166 | load: 0.00022 [I 2022-10-26 18:39:01] ...trap/engines/engine.py.321: eval: epoch 23 | batch 140/510 [I 2022-10-26 18:39:01] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-26 18:39:01] ...trap/engines/engine.py.326: process: 0.28691 | load: 0.00015 [I 2022-10-26 18:39:03] ...trap/engines/engine.py.321: eval: epoch 23 | batch 150/510 [I 2022-10-26 18:39:03] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-26 18:39:03] ...trap/engines/engine.py.326: process: 0.29023 | load: 0.00020 [I 2022-10-26 18:39:06] ...trap/engines/engine.py.321: eval: epoch 23 | batch 160/510 [I 2022-10-26 18:39:06] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 18:39:06] ...trap/engines/engine.py.326: process: 0.29003 | load: 0.00027 [I 2022-10-26 18:39:09] ...trap/engines/engine.py.321: eval: epoch 23 | batch 170/510 [I 2022-10-26 18:39:09] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-26 18:39:09] ...trap/engines/engine.py.326: process: 0.29201 | load: 0.00027 [I 2022-10-26 18:39:12] ...trap/engines/engine.py.321: eval: epoch 23 | batch 180/510 [I 2022-10-26 18:39:12] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 18:39:12] ...trap/engines/engine.py.326: process: 0.28597 | load: 0.00023 [I 2022-10-26 18:39:15] ...trap/engines/engine.py.321: eval: epoch 23 | batch 190/510 [I 2022-10-26 18:39:15] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 18:39:15] ...trap/engines/engine.py.326: process: 0.29036 | load: 0.00015 [I 2022-10-26 18:39:18] ...trap/engines/engine.py.321: eval: epoch 23 | batch 200/510 [I 2022-10-26 18:39:18] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 18:39:18] ...trap/engines/engine.py.326: process: 0.28504 | load: 0.00020 [I 2022-10-26 18:39:21] ...trap/engines/engine.py.321: eval: epoch 23 | batch 210/510 [I 2022-10-26 18:39:21] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 18:39:21] ...trap/engines/engine.py.326: process: 0.28672 | load: 0.00015 [I 2022-10-26 18:39:24] ...trap/engines/engine.py.321: eval: epoch 23 | batch 220/510 [I 2022-10-26 18:39:24] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-26 18:39:24] ...trap/engines/engine.py.326: process: 0.29576 | load: 0.00017 [I 2022-10-26 18:39:27] ...trap/engines/engine.py.321: eval: epoch 23 | batch 230/510 [I 2022-10-26 18:39:27] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 18:39:27] ...trap/engines/engine.py.326: process: 0.29667 | load: 0.00014 [I 2022-10-26 18:39:30] ...trap/engines/engine.py.321: eval: epoch 23 | batch 240/510 [I 2022-10-26 18:39:30] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 18:39:30] ...trap/engines/engine.py.326: process: 0.29105 | load: 0.00017 [I 2022-10-26 18:39:32] ...trap/engines/engine.py.321: eval: epoch 23 | batch 250/510 [I 2022-10-26 18:39:32] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-26 18:39:32] ...trap/engines/engine.py.326: process: 0.28759 | load: 0.00018 [I 2022-10-26 18:39:35] ...trap/engines/engine.py.321: eval: epoch 23 | batch 260/510 [I 2022-10-26 18:39:35] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:12 [I 2022-10-26 18:39:35] ...trap/engines/engine.py.326: process: 0.28834 | load: 0.00021 [I 2022-10-26 18:39:38] ...trap/engines/engine.py.321: eval: epoch 23 | batch 270/510 [I 2022-10-26 18:39:38] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-26 18:39:38] ...trap/engines/engine.py.326: process: 0.28636 | load: 0.00018 [I 2022-10-26 18:39:41] ...trap/engines/engine.py.321: eval: epoch 23 | batch 280/510 [I 2022-10-26 18:39:41] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 18:39:41] ...trap/engines/engine.py.326: process: 0.28675 | load: 0.00016 [I 2022-10-26 18:39:44] ...trap/engines/engine.py.321: eval: epoch 23 | batch 290/510 [I 2022-10-26 18:39:44] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 18:39:44] ...trap/engines/engine.py.326: process: 0.28727 | load: 0.00023 [I 2022-10-26 18:39:47] ...trap/engines/engine.py.321: eval: epoch 23 | batch 300/510 [I 2022-10-26 18:39:47] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 18:39:47] ...trap/engines/engine.py.326: process: 0.28542 | load: 0.00020 [I 2022-10-26 18:39:50] ...trap/engines/engine.py.321: eval: epoch 23 | batch 310/510 [I 2022-10-26 18:39:50] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 18:39:50] ...trap/engines/engine.py.326: process: 0.28460 | load: 0.00022 [I 2022-10-26 18:39:53] ...trap/engines/engine.py.321: eval: epoch 23 | batch 320/510 [I 2022-10-26 18:39:53] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 18:39:53] ...trap/engines/engine.py.326: process: 0.29295 | load: 0.00015 [I 2022-10-26 18:39:55] ...trap/engines/engine.py.321: eval: epoch 23 | batch 330/510 [I 2022-10-26 18:39:55] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 18:39:55] ...trap/engines/engine.py.326: process: 0.28567 | load: 0.00014 [I 2022-10-26 18:39:58] ...trap/engines/engine.py.321: eval: epoch 23 | batch 340/510 [I 2022-10-26 18:39:58] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:49 [I 2022-10-26 18:39:58] ...trap/engines/engine.py.326: process: 0.28849 | load: 0.00021 [I 2022-10-26 18:40:01] ...trap/engines/engine.py.321: eval: epoch 23 | batch 350/510 [I 2022-10-26 18:40:01] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 18:40:01] ...trap/engines/engine.py.326: process: 0.29039 | load: 0.00018 [I 2022-10-26 18:40:04] ...trap/engines/engine.py.321: eval: epoch 23 | batch 360/510 [I 2022-10-26 18:40:04] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 18:40:04] ...trap/engines/engine.py.326: process: 0.28850 | load: 0.00017 [I 2022-10-26 18:40:07] ...trap/engines/engine.py.321: eval: epoch 23 | batch 370/510 [I 2022-10-26 18:40:07] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:41 [I 2022-10-26 18:40:07] ...trap/engines/engine.py.326: process: 0.28043 | load: 0.00018 [I 2022-10-26 18:40:10] ...trap/engines/engine.py.321: eval: epoch 23 | batch 380/510 [I 2022-10-26 18:40:10] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 18:40:10] ...trap/engines/engine.py.326: process: 0.28670 | load: 0.00016 [I 2022-10-26 18:40:13] ...trap/engines/engine.py.321: eval: epoch 23 | batch 390/510 [I 2022-10-26 18:40:13] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 18:40:13] ...trap/engines/engine.py.326: process: 0.28571 | load: 0.00016 [I 2022-10-26 18:40:16] ...trap/engines/engine.py.321: eval: epoch 23 | batch 400/510 [I 2022-10-26 18:40:16] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 18:40:16] ...trap/engines/engine.py.326: process: 0.28091 | load: 0.00016 [I 2022-10-26 18:40:19] ...trap/engines/engine.py.321: eval: epoch 23 | batch 410/510 [I 2022-10-26 18:40:19] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 18:40:19] ...trap/engines/engine.py.326: process: 0.28499 | load: 0.00016 [I 2022-10-26 18:40:21] ...trap/engines/engine.py.321: eval: epoch 23 | batch 420/510 [I 2022-10-26 18:40:21] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 18:40:21] ...trap/engines/engine.py.326: process: 0.28910 | load: 0.00014 [I 2022-10-26 18:40:24] ...trap/engines/engine.py.321: eval: epoch 23 | batch 430/510 [I 2022-10-26 18:40:24] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 18:40:24] ...trap/engines/engine.py.326: process: 0.28824 | load: 0.00016 [I 2022-10-26 18:40:27] ...trap/engines/engine.py.321: eval: epoch 23 | batch 440/510 [I 2022-10-26 18:40:27] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 18:40:27] ...trap/engines/engine.py.326: process: 0.30830 | load: 0.00017 [I 2022-10-26 18:40:30] ...trap/engines/engine.py.321: eval: epoch 23 | batch 450/510 [I 2022-10-26 18:40:30] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 18:40:30] ...trap/engines/engine.py.326: process: 0.28434 | load: 0.00020 [I 2022-10-26 18:40:33] ...trap/engines/engine.py.321: eval: epoch 23 | batch 460/510 [I 2022-10-26 18:40:33] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 18:40:33] ...trap/engines/engine.py.326: process: 0.29098 | load: 0.00019 [I 2022-10-26 18:40:36] ...trap/engines/engine.py.321: eval: epoch 23 | batch 470/510 [I 2022-10-26 18:40:36] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 18:40:36] ...trap/engines/engine.py.326: process: 0.28565 | load: 0.00018 [I 2022-10-26 18:40:39] ...trap/engines/engine.py.321: eval: epoch 23 | batch 480/510 [I 2022-10-26 18:40:39] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 18:40:39] ...trap/engines/engine.py.326: process: 0.28386 | load: 0.00022 [I 2022-10-26 18:40:42] ...trap/engines/engine.py.321: eval: epoch 23 | batch 490/510 [I 2022-10-26 18:40:42] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 18:40:42] ...trap/engines/engine.py.326: process: 0.29037 | load: 0.00016 [I 2022-10-26 18:40:45] ...trap/engines/engine.py.321: eval: epoch 23 | batch 500/510 [I 2022-10-26 18:40:45] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 18:40:45] ...trap/engines/engine.py.326: process: 0.29267 | load: 0.00018 [I 2022-10-26 18:40:47] ...trap/engines/engine.py.321: eval: epoch 23 | batch 510/510 [I 2022-10-26 18:40:47] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 18:40:47] ...trap/engines/engine.py.326: process: 0.27908 | load: 0.00017 [S 2022-10-26 18:40:48] ...trap/engines/engine.py.132: eval_epoch.epoch: 23 [I 2022-10-26 18:40:48] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.630600 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.871000 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.919200 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.626100 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.867000 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.911700 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012769 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011679 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010127 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012841 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009737 [S 2022-10-26 18:40:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010863 [I 2022-10-26 18:40:53] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 18:40:53] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 18:40:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 18:40:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 18:40:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 18:40:54] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 18:40:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 18:41:03] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 18:41:03] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 18:41:03] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 18:41:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 18:41:12] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 18:41:12] ...trap/engines/engine.py.163: Training model on trainset for epoch 24 [I 2022-10-26 18:41:12] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 18:41:15] ...trap/engines/engine.py.224: train: epoch 24 | batch 0/2304 [I 2022-10-26 18:41:15] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:14 [I 2022-10-26 18:41:15] ...trap/engines/engine.py.229: process: 2.68860 | load: 2.03809 [I 2022-10-26 18:41:15] ...trap/engines/engine.py.230: loss: 0.15736 [I 2022-10-26 18:41:15] ...optimizers/trijoint.py.151: total_norm: 19.446974 [I 2022-10-26 18:41:20] ...trap/engines/engine.py.224: train: epoch 24 | batch 10/2304 [I 2022-10-26 18:41:20] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:58 [I 2022-10-26 18:41:20] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00354 [I 2022-10-26 18:41:20] ...trap/engines/engine.py.230: loss: 0.25799 [I 2022-10-26 18:41:20] ...optimizers/trijoint.py.151: total_norm: 14.512928 [I 2022-10-26 18:41:26] ...trap/engines/engine.py.224: train: epoch 24 | batch 20/2304 [I 2022-10-26 18:41:26] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:01 [I 2022-10-26 18:41:26] ...trap/engines/engine.py.229: process: 0.54332 | load: 0.00316 [I 2022-10-26 18:41:26] ...trap/engines/engine.py.230: loss: 0.22435 [I 2022-10-26 18:41:26] ...optimizers/trijoint.py.151: total_norm: 12.860924 [I 2022-10-26 18:41:31] ...trap/engines/engine.py.224: train: epoch 24 | batch 30/2304 [I 2022-10-26 18:41:31] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:10 [I 2022-10-26 18:41:31] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00356 [I 2022-10-26 18:41:31] ...trap/engines/engine.py.230: loss: 0.22976 [I 2022-10-26 18:41:31] ...optimizers/trijoint.py.151: total_norm: 19.867823 [I 2022-10-26 18:41:37] ...trap/engines/engine.py.224: train: epoch 24 | batch 40/2304 [I 2022-10-26 18:41:37] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:11 [I 2022-10-26 18:41:37] ...trap/engines/engine.py.229: process: 0.56044 | load: 0.00294 [I 2022-10-26 18:41:37] ...trap/engines/engine.py.230: loss: 0.23332 [I 2022-10-26 18:41:37] ...optimizers/trijoint.py.151: total_norm: 24.150963 [I 2022-10-26 18:41:43] ...trap/engines/engine.py.224: train: epoch 24 | batch 50/2304 [I 2022-10-26 18:41:43] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:53 [I 2022-10-26 18:41:43] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00279 [I 2022-10-26 18:41:43] ...trap/engines/engine.py.230: loss: 0.11774 [I 2022-10-26 18:41:43] ...optimizers/trijoint.py.151: total_norm: 10.834758 [I 2022-10-26 18:41:48] ...trap/engines/engine.py.224: train: epoch 24 | batch 60/2304 [I 2022-10-26 18:41:48] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:53 [I 2022-10-26 18:41:48] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00262 [I 2022-10-26 18:41:48] ...trap/engines/engine.py.230: loss: 0.18616 [I 2022-10-26 18:41:48] ...optimizers/trijoint.py.151: total_norm: 17.351370 [I 2022-10-26 18:41:54] ...trap/engines/engine.py.224: train: epoch 24 | batch 70/2304 [I 2022-10-26 18:41:54] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:51 [I 2022-10-26 18:41:54] ...trap/engines/engine.py.229: process: 0.57043 | load: 0.00303 [I 2022-10-26 18:41:54] ...trap/engines/engine.py.230: loss: 0.23109 [I 2022-10-26 18:41:54] ...optimizers/trijoint.py.151: total_norm: 14.010427 [I 2022-10-26 18:41:59] ...trap/engines/engine.py.224: train: epoch 24 | batch 80/2304 [I 2022-10-26 18:41:59] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:43 [I 2022-10-26 18:41:59] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00284 [I 2022-10-26 18:41:59] ...trap/engines/engine.py.230: loss: 0.19413 [I 2022-10-26 18:41:59] ...optimizers/trijoint.py.151: total_norm: 17.394947 [I 2022-10-26 18:42:05] ...trap/engines/engine.py.224: train: epoch 24 | batch 90/2304 [I 2022-10-26 18:42:05] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:35 [I 2022-10-26 18:42:05] ...trap/engines/engine.py.229: process: 0.54821 | load: 0.00272 [I 2022-10-26 18:42:05] ...trap/engines/engine.py.230: loss: 0.20966 [I 2022-10-26 18:42:05] ...optimizers/trijoint.py.151: total_norm: 44.591599 [I 2022-10-26 18:42:11] ...trap/engines/engine.py.224: train: epoch 24 | batch 100/2304 [I 2022-10-26 18:42:11] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:26 [I 2022-10-26 18:42:11] ...trap/engines/engine.py.229: process: 0.54983 | load: 0.00265 [I 2022-10-26 18:42:11] ...trap/engines/engine.py.230: loss: 0.17784 [I 2022-10-26 18:42:11] ...optimizers/trijoint.py.151: total_norm: 13.770495 [I 2022-10-26 18:42:16] ...trap/engines/engine.py.224: train: epoch 24 | batch 110/2304 [I 2022-10-26 18:42:16] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:23 [I 2022-10-26 18:42:16] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00310 [I 2022-10-26 18:42:16] ...trap/engines/engine.py.230: loss: 0.22325 [I 2022-10-26 18:42:16] ...optimizers/trijoint.py.151: total_norm: 18.048655 [I 2022-10-26 18:42:22] ...trap/engines/engine.py.224: train: epoch 24 | batch 120/2304 [I 2022-10-26 18:42:22] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:32 [I 2022-10-26 18:42:22] ...trap/engines/engine.py.229: process: 0.55905 | load: 0.00274 [I 2022-10-26 18:42:22] ...trap/engines/engine.py.230: loss: 0.24038 [I 2022-10-26 18:42:22] ...optimizers/trijoint.py.151: total_norm: 28.912828 [I 2022-10-26 18:42:27] ...trap/engines/engine.py.224: train: epoch 24 | batch 130/2304 [I 2022-10-26 18:42:27] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:20 [I 2022-10-26 18:42:27] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00270 [I 2022-10-26 18:42:27] ...trap/engines/engine.py.230: loss: 0.19898 [I 2022-10-26 18:42:27] ...optimizers/trijoint.py.151: total_norm: 29.476139 [I 2022-10-26 18:42:33] ...trap/engines/engine.py.224: train: epoch 24 | batch 140/2304 [I 2022-10-26 18:42:33] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:09 [I 2022-10-26 18:42:33] ...trap/engines/engine.py.229: process: 0.55656 | load: 0.00271 [I 2022-10-26 18:42:33] ...trap/engines/engine.py.230: loss: 0.20841 [I 2022-10-26 18:42:33] ...optimizers/trijoint.py.151: total_norm: 24.910160 [I 2022-10-26 18:42:39] ...trap/engines/engine.py.224: train: epoch 24 | batch 150/2304 [I 2022-10-26 18:42:39] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:02 [I 2022-10-26 18:42:39] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00270 [I 2022-10-26 18:42:39] ...trap/engines/engine.py.230: loss: 0.19644 [I 2022-10-26 18:42:39] ...optimizers/trijoint.py.151: total_norm: 9.360881 [I 2022-10-26 18:42:44] ...trap/engines/engine.py.224: train: epoch 24 | batch 160/2304 [I 2022-10-26 18:42:44] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:50 [I 2022-10-26 18:42:44] ...trap/engines/engine.py.229: process: 0.54410 | load: 0.00254 [I 2022-10-26 18:42:44] ...trap/engines/engine.py.230: loss: 0.18935 [I 2022-10-26 18:42:44] ...optimizers/trijoint.py.151: total_norm: 16.671721 [I 2022-10-26 18:42:50] ...trap/engines/engine.py.224: train: epoch 24 | batch 170/2304 [I 2022-10-26 18:42:50] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:47 [I 2022-10-26 18:42:50] ...trap/engines/engine.py.229: process: 0.57017 | load: 0.00239 [I 2022-10-26 18:42:50] ...trap/engines/engine.py.230: loss: 0.12790 [I 2022-10-26 18:42:50] ...optimizers/trijoint.py.151: total_norm: 11.854102 [I 2022-10-26 18:42:55] ...trap/engines/engine.py.224: train: epoch 24 | batch 180/2304 [I 2022-10-26 18:42:55] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:39 [I 2022-10-26 18:42:55] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00262 [I 2022-10-26 18:42:55] ...trap/engines/engine.py.230: loss: 0.24158 [I 2022-10-26 18:42:55] ...optimizers/trijoint.py.151: total_norm: 24.755384 [I 2022-10-26 18:43:01] ...trap/engines/engine.py.224: train: epoch 24 | batch 190/2304 [I 2022-10-26 18:43:01] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:26 [I 2022-10-26 18:43:01] ...trap/engines/engine.py.229: process: 0.55243 | load: 0.00246 [I 2022-10-26 18:43:01] ...trap/engines/engine.py.230: loss: 0.20297 [I 2022-10-26 18:43:01] ...optimizers/trijoint.py.151: total_norm: 27.333704 [I 2022-10-26 18:43:06] ...trap/engines/engine.py.224: train: epoch 24 | batch 200/2304 [I 2022-10-26 18:43:06] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:31 [I 2022-10-26 18:43:06] ...trap/engines/engine.py.229: process: 0.55433 | load: 0.00253 [I 2022-10-26 18:43:06] ...trap/engines/engine.py.230: loss: 0.18471 [I 2022-10-26 18:43:06] ...optimizers/trijoint.py.151: total_norm: 18.546190 [I 2022-10-26 18:43:12] ...trap/engines/engine.py.224: train: epoch 24 | batch 210/2304 [I 2022-10-26 18:43:12] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:27 [I 2022-10-26 18:43:12] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00242 [I 2022-10-26 18:43:12] ...trap/engines/engine.py.230: loss: 0.21874 [I 2022-10-26 18:43:12] ...optimizers/trijoint.py.151: total_norm: 22.228765 [I 2022-10-26 18:43:18] ...trap/engines/engine.py.224: train: epoch 24 | batch 220/2304 [I 2022-10-26 18:43:18] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:17 [I 2022-10-26 18:43:18] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00265 [I 2022-10-26 18:43:18] ...trap/engines/engine.py.230: loss: 0.16145 [I 2022-10-26 18:43:18] ...optimizers/trijoint.py.151: total_norm: 14.794138 [I 2022-10-26 18:43:23] ...trap/engines/engine.py.224: train: epoch 24 | batch 230/2304 [I 2022-10-26 18:43:23] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:08 [I 2022-10-26 18:43:23] ...trap/engines/engine.py.229: process: 0.54923 | load: 0.00267 [I 2022-10-26 18:43:23] ...trap/engines/engine.py.230: loss: 0.15406 [I 2022-10-26 18:43:23] ...optimizers/trijoint.py.151: total_norm: 19.210497 [I 2022-10-26 18:43:29] ...trap/engines/engine.py.224: train: epoch 24 | batch 240/2304 [I 2022-10-26 18:43:29] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:11 [I 2022-10-26 18:43:29] ...trap/engines/engine.py.229: process: 0.56599 | load: 0.00251 [I 2022-10-26 18:43:29] ...trap/engines/engine.py.230: loss: 0.14156 [I 2022-10-26 18:43:29] ...optimizers/trijoint.py.151: total_norm: 14.514498 [I 2022-10-26 18:43:34] ...trap/engines/engine.py.224: train: epoch 24 | batch 250/2304 [I 2022-10-26 18:43:34] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:07 [I 2022-10-26 18:43:34] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00246 [I 2022-10-26 18:43:34] ...trap/engines/engine.py.230: loss: 0.18463 [I 2022-10-26 18:43:34] ...optimizers/trijoint.py.151: total_norm: 25.615990 [I 2022-10-26 18:43:40] ...trap/engines/engine.py.224: train: epoch 24 | batch 260/2304 [I 2022-10-26 18:43:40] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:57 [I 2022-10-26 18:43:40] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00243 [I 2022-10-26 18:43:40] ...trap/engines/engine.py.230: loss: 0.21726 [I 2022-10-26 18:43:40] ...optimizers/trijoint.py.151: total_norm: 20.609440 [I 2022-10-26 18:43:45] ...trap/engines/engine.py.224: train: epoch 24 | batch 270/2304 [I 2022-10-26 18:43:45] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:50 [I 2022-10-26 18:43:45] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00243 [I 2022-10-26 18:43:45] ...trap/engines/engine.py.230: loss: 0.26583 [I 2022-10-26 18:43:45] ...optimizers/trijoint.py.151: total_norm: 21.576021 [I 2022-10-26 18:43:51] ...trap/engines/engine.py.224: train: epoch 24 | batch 280/2304 [I 2022-10-26 18:43:51] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:48 [I 2022-10-26 18:43:51] ...trap/engines/engine.py.229: process: 0.56781 | load: 0.00243 [I 2022-10-26 18:43:51] ...trap/engines/engine.py.230: loss: 0.24224 [I 2022-10-26 18:43:51] ...optimizers/trijoint.py.151: total_norm: 22.409586 [I 2022-10-26 18:43:57] ...trap/engines/engine.py.224: train: epoch 24 | batch 290/2304 [I 2022-10-26 18:43:57] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:41 [I 2022-10-26 18:43:57] ...trap/engines/engine.py.229: process: 0.56394 | load: 0.00248 [I 2022-10-26 18:43:57] ...trap/engines/engine.py.230: loss: 0.20687 [I 2022-10-26 18:43:57] ...optimizers/trijoint.py.151: total_norm: 31.072355 [I 2022-10-26 18:44:02] ...trap/engines/engine.py.224: train: epoch 24 | batch 300/2304 [I 2022-10-26 18:44:02] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:35 [I 2022-10-26 18:44:02] ...trap/engines/engine.py.229: process: 0.54304 | load: 0.00246 [I 2022-10-26 18:44:02] ...trap/engines/engine.py.230: loss: 0.20269 [I 2022-10-26 18:44:02] ...optimizers/trijoint.py.151: total_norm: 19.503517 [I 2022-10-26 18:44:08] ...trap/engines/engine.py.224: train: epoch 24 | batch 310/2304 [I 2022-10-26 18:44:08] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:39 [I 2022-10-26 18:44:08] ...trap/engines/engine.py.229: process: 0.56463 | load: 0.00247 [I 2022-10-26 18:44:08] ...trap/engines/engine.py.230: loss: 0.19373 [I 2022-10-26 18:44:08] ...optimizers/trijoint.py.151: total_norm: 19.450029 [I 2022-10-26 18:44:13] ...trap/engines/engine.py.224: train: epoch 24 | batch 320/2304 [I 2022-10-26 18:44:13] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:13 [I 2022-10-26 18:44:13] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00245 [I 2022-10-26 18:44:13] ...trap/engines/engine.py.230: loss: 0.17993 [I 2022-10-26 18:44:13] ...optimizers/trijoint.py.151: total_norm: 16.351768 [I 2022-10-26 18:44:19] ...trap/engines/engine.py.224: train: epoch 24 | batch 330/2304 [I 2022-10-26 18:44:19] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:23 [I 2022-10-26 18:44:19] ...trap/engines/engine.py.229: process: 0.55888 | load: 0.00260 [I 2022-10-26 18:44:19] ...trap/engines/engine.py.230: loss: 0.28485 [I 2022-10-26 18:44:19] ...optimizers/trijoint.py.151: total_norm: 31.801098 [I 2022-10-26 18:44:25] ...trap/engines/engine.py.224: train: epoch 24 | batch 340/2304 [I 2022-10-26 18:44:25] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:20 [I 2022-10-26 18:44:25] ...trap/engines/engine.py.229: process: 0.54780 | load: 0.00257 [I 2022-10-26 18:44:25] ...trap/engines/engine.py.230: loss: 0.17525 [I 2022-10-26 18:44:25] ...optimizers/trijoint.py.151: total_norm: 37.241558 [I 2022-10-26 18:44:30] ...trap/engines/engine.py.224: train: epoch 24 | batch 350/2304 [I 2022-10-26 18:44:30] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:12 [I 2022-10-26 18:44:30] ...trap/engines/engine.py.229: process: 0.55832 | load: 0.00247 [I 2022-10-26 18:44:30] ...trap/engines/engine.py.230: loss: 0.18947 [I 2022-10-26 18:44:30] ...optimizers/trijoint.py.151: total_norm: 37.663422 [I 2022-10-26 18:44:36] ...trap/engines/engine.py.224: train: epoch 24 | batch 360/2304 [I 2022-10-26 18:44:36] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:00 [I 2022-10-26 18:44:36] ...trap/engines/engine.py.229: process: 0.55364 | load: 0.00244 [I 2022-10-26 18:44:36] ...trap/engines/engine.py.230: loss: 0.17300 [I 2022-10-26 18:44:36] ...optimizers/trijoint.py.151: total_norm: 13.605070 [I 2022-10-26 18:44:41] ...trap/engines/engine.py.224: train: epoch 24 | batch 370/2304 [I 2022-10-26 18:44:41] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:56 [I 2022-10-26 18:44:41] ...trap/engines/engine.py.229: process: 0.55513 | load: 0.00264 [I 2022-10-26 18:44:41] ...trap/engines/engine.py.230: loss: 0.17734 [I 2022-10-26 18:44:41] ...optimizers/trijoint.py.151: total_norm: 27.019175 [I 2022-10-26 18:44:47] ...trap/engines/engine.py.224: train: epoch 24 | batch 380/2304 [I 2022-10-26 18:44:47] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:50 [I 2022-10-26 18:44:47] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00244 [I 2022-10-26 18:44:47] ...trap/engines/engine.py.230: loss: 0.15837 [I 2022-10-26 18:44:47] ...optimizers/trijoint.py.151: total_norm: 16.300308 [I 2022-10-26 18:44:53] ...trap/engines/engine.py.224: train: epoch 24 | batch 390/2304 [I 2022-10-26 18:44:53] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:59 [I 2022-10-26 18:44:53] ...trap/engines/engine.py.229: process: 0.57367 | load: 0.00257 [I 2022-10-26 18:44:53] ...trap/engines/engine.py.230: loss: 0.18920 [I 2022-10-26 18:44:53] ...optimizers/trijoint.py.151: total_norm: 14.197417 [I 2022-10-26 18:44:58] ...trap/engines/engine.py.224: train: epoch 24 | batch 400/2304 [I 2022-10-26 18:44:58] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:45 [I 2022-10-26 18:44:58] ...trap/engines/engine.py.229: process: 0.55814 | load: 0.00250 [I 2022-10-26 18:44:58] ...trap/engines/engine.py.230: loss: 0.24021 [I 2022-10-26 18:44:58] ...optimizers/trijoint.py.151: total_norm: 28.978878 [I 2022-10-26 18:45:04] ...trap/engines/engine.py.224: train: epoch 24 | batch 410/2304 [I 2022-10-26 18:45:04] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:36 [I 2022-10-26 18:45:04] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00244 [I 2022-10-26 18:45:04] ...trap/engines/engine.py.230: loss: 0.16849 [I 2022-10-26 18:45:04] ...optimizers/trijoint.py.151: total_norm: 13.862084 [I 2022-10-26 18:45:09] ...trap/engines/engine.py.224: train: epoch 24 | batch 420/2304 [I 2022-10-26 18:45:09] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:26 [I 2022-10-26 18:45:09] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00245 [I 2022-10-26 18:45:09] ...trap/engines/engine.py.230: loss: 0.14319 [I 2022-10-26 18:45:09] ...optimizers/trijoint.py.151: total_norm: 11.218024 [I 2022-10-26 18:45:15] ...trap/engines/engine.py.224: train: epoch 24 | batch 430/2304 [I 2022-10-26 18:45:15] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:18 [I 2022-10-26 18:45:15] ...trap/engines/engine.py.229: process: 0.54875 | load: 0.00246 [I 2022-10-26 18:45:15] ...trap/engines/engine.py.230: loss: 0.16957 [I 2022-10-26 18:45:15] ...optimizers/trijoint.py.151: total_norm: 16.250786 [I 2022-10-26 18:45:21] ...trap/engines/engine.py.224: train: epoch 24 | batch 440/2304 [I 2022-10-26 18:45:21] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:21 [I 2022-10-26 18:45:21] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00250 [I 2022-10-26 18:45:21] ...trap/engines/engine.py.230: loss: 0.15099 [I 2022-10-26 18:45:21] ...optimizers/trijoint.py.151: total_norm: 14.151441 [I 2022-10-26 18:45:26] ...trap/engines/engine.py.224: train: epoch 24 | batch 450/2304 [I 2022-10-26 18:45:26] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:01 [I 2022-10-26 18:45:26] ...trap/engines/engine.py.229: process: 0.54622 | load: 0.00246 [I 2022-10-26 18:45:26] ...trap/engines/engine.py.230: loss: 0.26694 [I 2022-10-26 18:45:26] ...optimizers/trijoint.py.151: total_norm: 26.247383 [I 2022-10-26 18:45:32] ...trap/engines/engine.py.224: train: epoch 24 | batch 460/2304 [I 2022-10-26 18:45:32] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:09 [I 2022-10-26 18:45:32] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00246 [I 2022-10-26 18:45:32] ...trap/engines/engine.py.230: loss: 0.17423 [I 2022-10-26 18:45:32] ...optimizers/trijoint.py.151: total_norm: 22.832523 [I 2022-10-26 18:45:37] ...trap/engines/engine.py.224: train: epoch 24 | batch 470/2304 [I 2022-10-26 18:45:37] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:19 [I 2022-10-26 18:45:37] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00251 [I 2022-10-26 18:45:37] ...trap/engines/engine.py.230: loss: 0.18660 [I 2022-10-26 18:45:37] ...optimizers/trijoint.py.151: total_norm: 23.940336 [I 2022-10-26 18:45:43] ...trap/engines/engine.py.224: train: epoch 24 | batch 480/2304 [I 2022-10-26 18:45:43] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:05 [I 2022-10-26 18:45:43] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00250 [I 2022-10-26 18:45:43] ...trap/engines/engine.py.230: loss: 0.21776 [I 2022-10-26 18:45:43] ...optimizers/trijoint.py.151: total_norm: 23.181461 [I 2022-10-26 18:45:49] ...trap/engines/engine.py.224: train: epoch 24 | batch 490/2304 [I 2022-10-26 18:45:49] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:59 [I 2022-10-26 18:45:49] ...trap/engines/engine.py.229: process: 0.54850 | load: 0.00254 [I 2022-10-26 18:45:49] ...trap/engines/engine.py.230: loss: 0.16279 [I 2022-10-26 18:45:49] ...optimizers/trijoint.py.151: total_norm: 15.690915 [I 2022-10-26 18:45:54] ...trap/engines/engine.py.224: train: epoch 24 | batch 500/2304 [I 2022-10-26 18:45:54] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:46 [I 2022-10-26 18:45:54] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00255 [I 2022-10-26 18:45:54] ...trap/engines/engine.py.230: loss: 0.22759 [I 2022-10-26 18:45:54] ...optimizers/trijoint.py.151: total_norm: 14.605112 [I 2022-10-26 18:46:00] ...trap/engines/engine.py.224: train: epoch 24 | batch 510/2304 [I 2022-10-26 18:46:00] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:40 [I 2022-10-26 18:46:00] ...trap/engines/engine.py.229: process: 0.55737 | load: 0.00246 [I 2022-10-26 18:46:00] ...trap/engines/engine.py.230: loss: 0.22123 [I 2022-10-26 18:46:00] ...optimizers/trijoint.py.151: total_norm: 41.040871 [I 2022-10-26 18:46:05] ...trap/engines/engine.py.224: train: epoch 24 | batch 520/2304 [I 2022-10-26 18:46:05] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:38 [I 2022-10-26 18:46:05] ...trap/engines/engine.py.229: process: 0.56785 | load: 0.00245 [I 2022-10-26 18:46:05] ...trap/engines/engine.py.230: loss: 0.21388 [I 2022-10-26 18:46:05] ...optimizers/trijoint.py.151: total_norm: 25.428667 [I 2022-10-26 18:46:11] ...trap/engines/engine.py.224: train: epoch 24 | batch 530/2304 [I 2022-10-26 18:46:11] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:26 [I 2022-10-26 18:46:11] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00247 [I 2022-10-26 18:46:11] ...trap/engines/engine.py.230: loss: 0.21683 [I 2022-10-26 18:46:11] ...optimizers/trijoint.py.151: total_norm: 33.360512 [I 2022-10-26 18:46:17] ...trap/engines/engine.py.224: train: epoch 24 | batch 540/2304 [I 2022-10-26 18:46:17] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:28 [I 2022-10-26 18:46:17] ...trap/engines/engine.py.229: process: 0.56692 | load: 0.00258 [I 2022-10-26 18:46:17] ...trap/engines/engine.py.230: loss: 0.17895 [I 2022-10-26 18:46:17] ...optimizers/trijoint.py.151: total_norm: 19.915001 [I 2022-10-26 18:46:22] ...trap/engines/engine.py.224: train: epoch 24 | batch 550/2304 [I 2022-10-26 18:46:22] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:04 [I 2022-10-26 18:46:22] ...trap/engines/engine.py.229: process: 0.54259 | load: 0.00243 [I 2022-10-26 18:46:22] ...trap/engines/engine.py.230: loss: 0.27086 [I 2022-10-26 18:46:22] ...optimizers/trijoint.py.151: total_norm: 21.867682 [I 2022-10-26 18:46:28] ...trap/engines/engine.py.224: train: epoch 24 | batch 560/2304 [I 2022-10-26 18:46:28] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:13 [I 2022-10-26 18:46:28] ...trap/engines/engine.py.229: process: 0.54660 | load: 0.00252 [I 2022-10-26 18:46:28] ...trap/engines/engine.py.230: loss: 0.17288 [I 2022-10-26 18:46:28] ...optimizers/trijoint.py.151: total_norm: 30.026939 [I 2022-10-26 18:46:33] ...trap/engines/engine.py.224: train: epoch 24 | batch 570/2304 [I 2022-10-26 18:46:33] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:05 [I 2022-10-26 18:46:33] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00245 [I 2022-10-26 18:46:33] ...trap/engines/engine.py.230: loss: 0.17943 [I 2022-10-26 18:46:33] ...optimizers/trijoint.py.151: total_norm: 19.786215 [I 2022-10-26 18:46:39] ...trap/engines/engine.py.224: train: epoch 24 | batch 580/2304 [I 2022-10-26 18:46:39] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:57 [I 2022-10-26 18:46:39] ...trap/engines/engine.py.229: process: 0.54269 | load: 0.00246 [I 2022-10-26 18:46:39] ...trap/engines/engine.py.230: loss: 0.18727 [I 2022-10-26 18:46:39] ...optimizers/trijoint.py.151: total_norm: 16.305271 [I 2022-10-26 18:46:45] ...trap/engines/engine.py.224: train: epoch 24 | batch 590/2304 [I 2022-10-26 18:46:45] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:59 [I 2022-10-26 18:46:45] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00245 [I 2022-10-26 18:46:45] ...trap/engines/engine.py.230: loss: 0.22106 [I 2022-10-26 18:46:45] ...optimizers/trijoint.py.151: total_norm: 26.913357 [I 2022-10-26 18:46:50] ...trap/engines/engine.py.224: train: epoch 24 | batch 600/2304 [I 2022-10-26 18:46:50] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:16:01 [I 2022-10-26 18:46:50] ...trap/engines/engine.py.229: process: 0.55499 | load: 0.00244 [I 2022-10-26 18:46:50] ...trap/engines/engine.py.230: loss: 0.17001 [I 2022-10-26 18:46:50] ...optimizers/trijoint.py.151: total_norm: 17.276316 [I 2022-10-26 18:46:56] ...trap/engines/engine.py.224: train: epoch 24 | batch 610/2304 [I 2022-10-26 18:46:56] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:45 [I 2022-10-26 18:46:56] ...trap/engines/engine.py.229: process: 0.56274 | load: 0.00254 [I 2022-10-26 18:46:56] ...trap/engines/engine.py.230: loss: 0.17007 [I 2022-10-26 18:46:56] ...optimizers/trijoint.py.151: total_norm: 16.585575 [I 2022-10-26 18:47:02] ...trap/engines/engine.py.224: train: epoch 24 | batch 620/2304 [I 2022-10-26 18:47:02] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:16:08 [I 2022-10-26 18:47:02] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00242 [I 2022-10-26 18:47:02] ...trap/engines/engine.py.230: loss: 0.09901 [I 2022-10-26 18:47:02] ...optimizers/trijoint.py.151: total_norm: 8.353960 [I 2022-10-26 18:47:07] ...trap/engines/engine.py.224: train: epoch 24 | batch 630/2304 [I 2022-10-26 18:47:07] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:38 [I 2022-10-26 18:47:07] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00241 [I 2022-10-26 18:47:07] ...trap/engines/engine.py.230: loss: 0.20983 [I 2022-10-26 18:47:07] ...optimizers/trijoint.py.151: total_norm: 24.192657 [I 2022-10-26 18:47:13] ...trap/engines/engine.py.224: train: epoch 24 | batch 640/2304 [I 2022-10-26 18:47:13] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:25 [I 2022-10-26 18:47:13] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00251 [I 2022-10-26 18:47:13] ...trap/engines/engine.py.230: loss: 0.21769 [I 2022-10-26 18:47:13] ...optimizers/trijoint.py.151: total_norm: 20.101475 [I 2022-10-26 18:47:18] ...trap/engines/engine.py.224: train: epoch 24 | batch 650/2304 [I 2022-10-26 18:47:18] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:21 [I 2022-10-26 18:47:18] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00242 [I 2022-10-26 18:47:18] ...trap/engines/engine.py.230: loss: 0.16131 [I 2022-10-26 18:47:18] ...optimizers/trijoint.py.151: total_norm: 20.938400 [I 2022-10-26 18:47:24] ...trap/engines/engine.py.224: train: epoch 24 | batch 660/2304 [I 2022-10-26 18:47:24] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:14 [I 2022-10-26 18:47:24] ...trap/engines/engine.py.229: process: 0.56141 | load: 0.00250 [I 2022-10-26 18:47:24] ...trap/engines/engine.py.230: loss: 0.14002 [I 2022-10-26 18:47:24] ...optimizers/trijoint.py.151: total_norm: 20.585505 [I 2022-10-26 18:47:29] ...trap/engines/engine.py.224: train: epoch 24 | batch 670/2304 [I 2022-10-26 18:47:29] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:10 [I 2022-10-26 18:47:29] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00247 [I 2022-10-26 18:47:29] ...trap/engines/engine.py.230: loss: 0.13652 [I 2022-10-26 18:47:29] ...optimizers/trijoint.py.151: total_norm: 11.788212 [I 2022-10-26 18:47:35] ...trap/engines/engine.py.224: train: epoch 24 | batch 680/2304 [I 2022-10-26 18:47:35] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:02 [I 2022-10-26 18:47:35] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00247 [I 2022-10-26 18:47:35] ...trap/engines/engine.py.230: loss: 0.15759 [I 2022-10-26 18:47:35] ...optimizers/trijoint.py.151: total_norm: 18.456028 [I 2022-10-26 18:47:41] ...trap/engines/engine.py.224: train: epoch 24 | batch 690/2304 [I 2022-10-26 18:47:41] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:55 [I 2022-10-26 18:47:41] ...trap/engines/engine.py.229: process: 0.55076 | load: 0.00252 [I 2022-10-26 18:47:41] ...trap/engines/engine.py.230: loss: 0.16383 [I 2022-10-26 18:47:41] ...optimizers/trijoint.py.151: total_norm: 20.570074 [I 2022-10-26 18:47:46] ...trap/engines/engine.py.224: train: epoch 24 | batch 700/2304 [I 2022-10-26 18:47:46] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:53 [I 2022-10-26 18:47:46] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00251 [I 2022-10-26 18:47:46] ...trap/engines/engine.py.230: loss: 0.23843 [I 2022-10-26 18:47:46] ...optimizers/trijoint.py.151: total_norm: 24.331545 [I 2022-10-26 18:47:52] ...trap/engines/engine.py.224: train: epoch 24 | batch 710/2304 [I 2022-10-26 18:47:52] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:50 [I 2022-10-26 18:47:52] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00265 [I 2022-10-26 18:47:52] ...trap/engines/engine.py.230: loss: 0.22681 [I 2022-10-26 18:47:52] ...optimizers/trijoint.py.151: total_norm: 25.322878 [I 2022-10-26 18:47:57] ...trap/engines/engine.py.224: train: epoch 24 | batch 720/2304 [I 2022-10-26 18:47:57] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:42 [I 2022-10-26 18:47:57] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00248 [I 2022-10-26 18:47:57] ...trap/engines/engine.py.230: loss: 0.21854 [I 2022-10-26 18:47:57] ...optimizers/trijoint.py.151: total_norm: 32.378826 [I 2022-10-26 18:48:03] ...trap/engines/engine.py.224: train: epoch 24 | batch 730/2304 [I 2022-10-26 18:48:03] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:15:04 [I 2022-10-26 18:48:03] ...trap/engines/engine.py.229: process: 0.55431 | load: 0.00252 [I 2022-10-26 18:48:03] ...trap/engines/engine.py.230: loss: 0.19632 [I 2022-10-26 18:48:03] ...optimizers/trijoint.py.151: total_norm: 16.459824 [I 2022-10-26 18:48:09] ...trap/engines/engine.py.224: train: epoch 24 | batch 740/2304 [I 2022-10-26 18:48:09] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:29 [I 2022-10-26 18:48:09] ...trap/engines/engine.py.229: process: 0.56389 | load: 0.00248 [I 2022-10-26 18:48:09] ...trap/engines/engine.py.230: loss: 0.17305 [I 2022-10-26 18:48:09] ...optimizers/trijoint.py.151: total_norm: 21.170004 [I 2022-10-26 18:48:14] ...trap/engines/engine.py.224: train: epoch 24 | batch 750/2304 [I 2022-10-26 18:48:14] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:24 [I 2022-10-26 18:48:14] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00246 [I 2022-10-26 18:48:14] ...trap/engines/engine.py.230: loss: 0.15232 [I 2022-10-26 18:48:14] ...optimizers/trijoint.py.151: total_norm: 15.797130 [I 2022-10-26 18:48:20] ...trap/engines/engine.py.224: train: epoch 24 | batch 760/2304 [I 2022-10-26 18:48:20] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:23 [I 2022-10-26 18:48:20] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00246 [I 2022-10-26 18:48:20] ...trap/engines/engine.py.230: loss: 0.24725 [I 2022-10-26 18:48:20] ...optimizers/trijoint.py.151: total_norm: 15.996531 [I 2022-10-26 18:48:25] ...trap/engines/engine.py.224: train: epoch 24 | batch 770/2304 [I 2022-10-26 18:48:25] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:15 [I 2022-10-26 18:48:25] ...trap/engines/engine.py.229: process: 0.55362 | load: 0.00251 [I 2022-10-26 18:48:25] ...trap/engines/engine.py.230: loss: 0.22984 [I 2022-10-26 18:48:25] ...optimizers/trijoint.py.151: total_norm: 38.602062 [I 2022-10-26 18:48:31] ...trap/engines/engine.py.224: train: epoch 24 | batch 780/2304 [I 2022-10-26 18:48:31] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:07 [I 2022-10-26 18:48:31] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00245 [I 2022-10-26 18:48:31] ...trap/engines/engine.py.230: loss: 0.17903 [I 2022-10-26 18:48:31] ...optimizers/trijoint.py.151: total_norm: 17.834858 [I 2022-10-26 18:48:37] ...trap/engines/engine.py.224: train: epoch 24 | batch 790/2304 [I 2022-10-26 18:48:37] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:13:58 [I 2022-10-26 18:48:37] ...trap/engines/engine.py.229: process: 0.54997 | load: 0.00240 [I 2022-10-26 18:48:37] ...trap/engines/engine.py.230: loss: 0.19252 [I 2022-10-26 18:48:37] ...optimizers/trijoint.py.151: total_norm: 21.509504 [I 2022-10-26 18:48:42] ...trap/engines/engine.py.224: train: epoch 24 | batch 800/2304 [I 2022-10-26 18:48:42] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:57 [I 2022-10-26 18:48:42] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00247 [I 2022-10-26 18:48:42] ...trap/engines/engine.py.230: loss: 0.19521 [I 2022-10-26 18:48:42] ...optimizers/trijoint.py.151: total_norm: 21.163021 [I 2022-10-26 18:48:48] ...trap/engines/engine.py.224: train: epoch 24 | batch 810/2304 [I 2022-10-26 18:48:48] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:48 [I 2022-10-26 18:48:48] ...trap/engines/engine.py.229: process: 0.54577 | load: 0.00250 [I 2022-10-26 18:48:48] ...trap/engines/engine.py.230: loss: 0.16921 [I 2022-10-26 18:48:48] ...optimizers/trijoint.py.151: total_norm: 19.706026 [I 2022-10-26 18:48:53] ...trap/engines/engine.py.224: train: epoch 24 | batch 820/2304 [I 2022-10-26 18:48:53] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:43 [I 2022-10-26 18:48:53] ...trap/engines/engine.py.229: process: 0.56091 | load: 0.00246 [I 2022-10-26 18:48:53] ...trap/engines/engine.py.230: loss: 0.22971 [I 2022-10-26 18:48:53] ...optimizers/trijoint.py.151: total_norm: 29.894611 [I 2022-10-26 18:48:59] ...trap/engines/engine.py.224: train: epoch 24 | batch 830/2304 [I 2022-10-26 18:48:59] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:42 [I 2022-10-26 18:48:59] ...trap/engines/engine.py.229: process: 0.55834 | load: 0.00245 [I 2022-10-26 18:48:59] ...trap/engines/engine.py.230: loss: 0.19541 [I 2022-10-26 18:48:59] ...optimizers/trijoint.py.151: total_norm: 15.580366 [I 2022-10-26 18:49:04] ...trap/engines/engine.py.224: train: epoch 24 | batch 840/2304 [I 2022-10-26 18:49:04] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:39 [I 2022-10-26 18:49:04] ...trap/engines/engine.py.229: process: 0.54229 | load: 0.00243 [I 2022-10-26 18:49:04] ...trap/engines/engine.py.230: loss: 0.19407 [I 2022-10-26 18:49:04] ...optimizers/trijoint.py.151: total_norm: 14.738849 [I 2022-10-26 18:49:10] ...trap/engines/engine.py.224: train: epoch 24 | batch 850/2304 [I 2022-10-26 18:49:10] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:33 [I 2022-10-26 18:49:10] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00247 [I 2022-10-26 18:49:10] ...trap/engines/engine.py.230: loss: 0.19244 [I 2022-10-26 18:49:10] ...optimizers/trijoint.py.151: total_norm: 21.538074 [I 2022-10-26 18:49:16] ...trap/engines/engine.py.224: train: epoch 24 | batch 860/2304 [I 2022-10-26 18:49:16] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:24 [I 2022-10-26 18:49:16] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00244 [I 2022-10-26 18:49:16] ...trap/engines/engine.py.230: loss: 0.17065 [I 2022-10-26 18:49:16] ...optimizers/trijoint.py.151: total_norm: 16.687847 [I 2022-10-26 18:49:21] ...trap/engines/engine.py.224: train: epoch 24 | batch 870/2304 [I 2022-10-26 18:49:21] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:19 [I 2022-10-26 18:49:21] ...trap/engines/engine.py.229: process: 0.56343 | load: 0.00246 [I 2022-10-26 18:49:21] ...trap/engines/engine.py.230: loss: 0.17509 [I 2022-10-26 18:49:21] ...optimizers/trijoint.py.151: total_norm: 21.168549 [I 2022-10-26 18:49:27] ...trap/engines/engine.py.224: train: epoch 24 | batch 880/2304 [I 2022-10-26 18:49:27] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:16 [I 2022-10-26 18:49:27] ...trap/engines/engine.py.229: process: 0.56094 | load: 0.00242 [I 2022-10-26 18:49:27] ...trap/engines/engine.py.230: loss: 0.18336 [I 2022-10-26 18:49:27] ...optimizers/trijoint.py.151: total_norm: 20.691393 [I 2022-10-26 18:49:32] ...trap/engines/engine.py.224: train: epoch 24 | batch 890/2304 [I 2022-10-26 18:49:32] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:14 [I 2022-10-26 18:49:32] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00244 [I 2022-10-26 18:49:32] ...trap/engines/engine.py.230: loss: 0.15947 [I 2022-10-26 18:49:32] ...optimizers/trijoint.py.151: total_norm: 18.226885 [I 2022-10-26 18:49:38] ...trap/engines/engine.py.224: train: epoch 24 | batch 900/2304 [I 2022-10-26 18:49:38] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:06 [I 2022-10-26 18:49:38] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00254 [I 2022-10-26 18:49:38] ...trap/engines/engine.py.230: loss: 0.18352 [I 2022-10-26 18:49:38] ...optimizers/trijoint.py.151: total_norm: 19.710709 [I 2022-10-26 18:49:44] ...trap/engines/engine.py.224: train: epoch 24 | batch 910/2304 [I 2022-10-26 18:49:44] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:57 [I 2022-10-26 18:49:44] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00250 [I 2022-10-26 18:49:44] ...trap/engines/engine.py.230: loss: 0.25221 [I 2022-10-26 18:49:44] ...optimizers/trijoint.py.151: total_norm: 24.010284 [I 2022-10-26 18:49:49] ...trap/engines/engine.py.224: train: epoch 24 | batch 920/2304 [I 2022-10-26 18:49:49] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:54 [I 2022-10-26 18:49:49] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00244 [I 2022-10-26 18:49:49] ...trap/engines/engine.py.230: loss: 0.15608 [I 2022-10-26 18:49:49] ...optimizers/trijoint.py.151: total_norm: 17.122658 [I 2022-10-26 18:49:55] ...trap/engines/engine.py.224: train: epoch 24 | batch 930/2304 [I 2022-10-26 18:49:55] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:46 [I 2022-10-26 18:49:55] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00253 [I 2022-10-26 18:49:55] ...trap/engines/engine.py.230: loss: 0.19804 [I 2022-10-26 18:49:55] ...optimizers/trijoint.py.151: total_norm: 15.831488 [I 2022-10-26 18:50:00] ...trap/engines/engine.py.224: train: epoch 24 | batch 940/2304 [I 2022-10-26 18:50:00] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:43 [I 2022-10-26 18:50:00] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00245 [I 2022-10-26 18:50:00] ...trap/engines/engine.py.230: loss: 0.21695 [I 2022-10-26 18:50:00] ...optimizers/trijoint.py.151: total_norm: 13.711074 [I 2022-10-26 18:50:06] ...trap/engines/engine.py.224: train: epoch 24 | batch 950/2304 [I 2022-10-26 18:50:06] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:31 [I 2022-10-26 18:50:06] ...trap/engines/engine.py.229: process: 0.54311 | load: 0.00250 [I 2022-10-26 18:50:06] ...trap/engines/engine.py.230: loss: 0.22919 [I 2022-10-26 18:50:06] ...optimizers/trijoint.py.151: total_norm: 20.796618 [I 2022-10-26 18:50:12] ...trap/engines/engine.py.224: train: epoch 24 | batch 960/2304 [I 2022-10-26 18:50:12] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:42 [I 2022-10-26 18:50:12] ...trap/engines/engine.py.229: process: 0.60294 | load: 0.00246 [I 2022-10-26 18:50:12] ...trap/engines/engine.py.230: loss: 0.21235 [I 2022-10-26 18:50:12] ...optimizers/trijoint.py.151: total_norm: 34.641750 [I 2022-10-26 18:50:17] ...trap/engines/engine.py.224: train: epoch 24 | batch 970/2304 [I 2022-10-26 18:50:17] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:18 [I 2022-10-26 18:50:17] ...trap/engines/engine.py.229: process: 0.54671 | load: 0.00245 [I 2022-10-26 18:50:17] ...trap/engines/engine.py.230: loss: 0.13955 [I 2022-10-26 18:50:17] ...optimizers/trijoint.py.151: total_norm: 15.609303 [I 2022-10-26 18:50:23] ...trap/engines/engine.py.224: train: epoch 24 | batch 980/2304 [I 2022-10-26 18:50:23] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:19 [I 2022-10-26 18:50:23] ...trap/engines/engine.py.229: process: 0.56700 | load: 0.00242 [I 2022-10-26 18:50:23] ...trap/engines/engine.py.230: loss: 0.22162 [I 2022-10-26 18:50:23] ...optimizers/trijoint.py.151: total_norm: 14.763114 [I 2022-10-26 18:50:28] ...trap/engines/engine.py.224: train: epoch 24 | batch 990/2304 [I 2022-10-26 18:50:28] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:10 [I 2022-10-26 18:50:28] ...trap/engines/engine.py.229: process: 0.55819 | load: 0.00242 [I 2022-10-26 18:50:28] ...trap/engines/engine.py.230: loss: 0.19202 [I 2022-10-26 18:50:28] ...optimizers/trijoint.py.151: total_norm: 29.742579 [I 2022-10-26 18:50:34] ...trap/engines/engine.py.224: train: epoch 24 | batch 1000/2304 [I 2022-10-26 18:50:34] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:04 [I 2022-10-26 18:50:34] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00249 [I 2022-10-26 18:50:34] ...trap/engines/engine.py.230: loss: 0.20604 [I 2022-10-26 18:50:34] ...optimizers/trijoint.py.151: total_norm: 32.294849 [I 2022-10-26 18:50:40] ...trap/engines/engine.py.224: train: epoch 24 | batch 1010/2304 [I 2022-10-26 18:50:40] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:00 [I 2022-10-26 18:50:40] ...trap/engines/engine.py.229: process: 0.55725 | load: 0.00246 [I 2022-10-26 18:50:40] ...trap/engines/engine.py.230: loss: 0.28346 [I 2022-10-26 18:50:40] ...optimizers/trijoint.py.151: total_norm: 16.353722 [I 2022-10-26 18:50:45] ...trap/engines/engine.py.224: train: epoch 24 | batch 1020/2304 [I 2022-10-26 18:50:45] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:53 [I 2022-10-26 18:50:45] ...trap/engines/engine.py.229: process: 0.55657 | load: 0.00245 [I 2022-10-26 18:50:45] ...trap/engines/engine.py.230: loss: 0.27090 [I 2022-10-26 18:50:45] ...optimizers/trijoint.py.151: total_norm: 19.468706 [I 2022-10-26 18:50:51] ...trap/engines/engine.py.224: train: epoch 24 | batch 1030/2304 [I 2022-10-26 18:50:51] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:54 [I 2022-10-26 18:50:51] ...trap/engines/engine.py.229: process: 0.56822 | load: 0.00248 [I 2022-10-26 18:50:51] ...trap/engines/engine.py.230: loss: 0.25048 [I 2022-10-26 18:50:51] ...optimizers/trijoint.py.151: total_norm: 35.531872 [I 2022-10-26 18:50:56] ...trap/engines/engine.py.224: train: epoch 24 | batch 1040/2304 [I 2022-10-26 18:50:56] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:46 [I 2022-10-26 18:50:56] ...trap/engines/engine.py.229: process: 0.56119 | load: 0.00251 [I 2022-10-26 18:50:56] ...trap/engines/engine.py.230: loss: 0.17077 [I 2022-10-26 18:50:56] ...optimizers/trijoint.py.151: total_norm: 13.191723 [I 2022-10-26 18:51:02] ...trap/engines/engine.py.224: train: epoch 24 | batch 1050/2304 [I 2022-10-26 18:51:02] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:43 [I 2022-10-26 18:51:02] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00262 [I 2022-10-26 18:51:02] ...trap/engines/engine.py.230: loss: 0.16536 [I 2022-10-26 18:51:02] ...optimizers/trijoint.py.151: total_norm: 19.154102 [I 2022-10-26 18:51:07] ...trap/engines/engine.py.224: train: epoch 24 | batch 1060/2304 [I 2022-10-26 18:51:07] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:27 [I 2022-10-26 18:51:07] ...trap/engines/engine.py.229: process: 0.54470 | load: 0.00245 [I 2022-10-26 18:51:07] ...trap/engines/engine.py.230: loss: 0.28838 [I 2022-10-26 18:51:07] ...optimizers/trijoint.py.151: total_norm: 29.791557 [I 2022-10-26 18:51:13] ...trap/engines/engine.py.224: train: epoch 24 | batch 1070/2304 [I 2022-10-26 18:51:13] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:29 [I 2022-10-26 18:51:13] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00240 [I 2022-10-26 18:51:13] ...trap/engines/engine.py.230: loss: 0.23010 [I 2022-10-26 18:51:13] ...optimizers/trijoint.py.151: total_norm: 37.219181 [I 2022-10-26 18:51:19] ...trap/engines/engine.py.224: train: epoch 24 | batch 1080/2304 [I 2022-10-26 18:51:19] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:23 [I 2022-10-26 18:51:19] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00245 [I 2022-10-26 18:51:19] ...trap/engines/engine.py.230: loss: 0.17001 [I 2022-10-26 18:51:19] ...optimizers/trijoint.py.151: total_norm: 22.619680 [I 2022-10-26 18:51:24] ...trap/engines/engine.py.224: train: epoch 24 | batch 1090/2304 [I 2022-10-26 18:51:24] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:14 [I 2022-10-26 18:51:24] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00245 [I 2022-10-26 18:51:24] ...trap/engines/engine.py.230: loss: 0.16616 [I 2022-10-26 18:51:24] ...optimizers/trijoint.py.151: total_norm: 24.954729 [I 2022-10-26 18:51:30] ...trap/engines/engine.py.224: train: epoch 24 | batch 1100/2304 [I 2022-10-26 18:51:30] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:09 [I 2022-10-26 18:51:30] ...trap/engines/engine.py.229: process: 0.54769 | load: 0.00241 [I 2022-10-26 18:51:30] ...trap/engines/engine.py.230: loss: 0.13091 [I 2022-10-26 18:51:30] ...optimizers/trijoint.py.151: total_norm: 16.055334 [I 2022-10-26 18:51:35] ...trap/engines/engine.py.224: train: epoch 24 | batch 1110/2304 [I 2022-10-26 18:51:35] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:06 [I 2022-10-26 18:51:35] ...trap/engines/engine.py.229: process: 0.55924 | load: 0.00247 [I 2022-10-26 18:51:35] ...trap/engines/engine.py.230: loss: 0.15219 [I 2022-10-26 18:51:35] ...optimizers/trijoint.py.151: total_norm: 17.201328 [I 2022-10-26 18:51:41] ...trap/engines/engine.py.224: train: epoch 24 | batch 1120/2304 [I 2022-10-26 18:51:41] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:02 [I 2022-10-26 18:51:41] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00242 [I 2022-10-26 18:51:41] ...trap/engines/engine.py.230: loss: 0.17684 [I 2022-10-26 18:51:41] ...optimizers/trijoint.py.151: total_norm: 23.317129 [I 2022-10-26 18:51:47] ...trap/engines/engine.py.224: train: epoch 24 | batch 1130/2304 [I 2022-10-26 18:51:47] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:53 [I 2022-10-26 18:51:47] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00251 [I 2022-10-26 18:51:47] ...trap/engines/engine.py.230: loss: 0.16060 [I 2022-10-26 18:51:47] ...optimizers/trijoint.py.151: total_norm: 25.132534 [I 2022-10-26 18:51:52] ...trap/engines/engine.py.224: train: epoch 24 | batch 1140/2304 [I 2022-10-26 18:51:52] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:47 [I 2022-10-26 18:51:52] ...trap/engines/engine.py.229: process: 0.55719 | load: 0.00245 [I 2022-10-26 18:51:52] ...trap/engines/engine.py.230: loss: 0.26096 [I 2022-10-26 18:51:52] ...optimizers/trijoint.py.151: total_norm: 26.719072 [I 2022-10-26 18:51:58] ...trap/engines/engine.py.224: train: epoch 24 | batch 1150/2304 [I 2022-10-26 18:51:58] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:42 [I 2022-10-26 18:51:58] ...trap/engines/engine.py.229: process: 0.55262 | load: 0.00256 [I 2022-10-26 18:51:58] ...trap/engines/engine.py.230: loss: 0.24762 [I 2022-10-26 18:51:58] ...optimizers/trijoint.py.151: total_norm: 20.289938 [I 2022-10-26 18:52:03] ...trap/engines/engine.py.224: train: epoch 24 | batch 1160/2304 [I 2022-10-26 18:52:03] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:36 [I 2022-10-26 18:52:03] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00246 [I 2022-10-26 18:52:03] ...trap/engines/engine.py.230: loss: 0.10210 [I 2022-10-26 18:52:03] ...optimizers/trijoint.py.151: total_norm: 8.838051 [I 2022-10-26 18:52:09] ...trap/engines/engine.py.224: train: epoch 24 | batch 1170/2304 [I 2022-10-26 18:52:09] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:27 [I 2022-10-26 18:52:09] ...trap/engines/engine.py.229: process: 0.54888 | load: 0.00253 [I 2022-10-26 18:52:09] ...trap/engines/engine.py.230: loss: 0.16118 [I 2022-10-26 18:52:09] ...optimizers/trijoint.py.151: total_norm: 10.004497 [I 2022-10-26 18:52:15] ...trap/engines/engine.py.224: train: epoch 24 | batch 1180/2304 [I 2022-10-26 18:52:15] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:27 [I 2022-10-26 18:52:15] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00253 [I 2022-10-26 18:52:15] ...trap/engines/engine.py.230: loss: 0.27021 [I 2022-10-26 18:52:15] ...optimizers/trijoint.py.151: total_norm: 21.788027 [I 2022-10-26 18:52:20] ...trap/engines/engine.py.224: train: epoch 24 | batch 1190/2304 [I 2022-10-26 18:52:20] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:22 [I 2022-10-26 18:52:20] ...trap/engines/engine.py.229: process: 0.56332 | load: 0.00246 [I 2022-10-26 18:52:20] ...trap/engines/engine.py.230: loss: 0.11653 [I 2022-10-26 18:52:20] ...optimizers/trijoint.py.151: total_norm: 13.845441 [I 2022-10-26 18:52:26] ...trap/engines/engine.py.224: train: epoch 24 | batch 1200/2304 [I 2022-10-26 18:52:26] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:12 [I 2022-10-26 18:52:26] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00245 [I 2022-10-26 18:52:26] ...trap/engines/engine.py.230: loss: 0.17366 [I 2022-10-26 18:52:26] ...optimizers/trijoint.py.151: total_norm: 25.375002 [I 2022-10-26 18:52:31] ...trap/engines/engine.py.224: train: epoch 24 | batch 1210/2304 [I 2022-10-26 18:52:31] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:05 [I 2022-10-26 18:52:31] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00242 [I 2022-10-26 18:52:31] ...trap/engines/engine.py.230: loss: 0.18781 [I 2022-10-26 18:52:31] ...optimizers/trijoint.py.151: total_norm: 29.643049 [I 2022-10-26 18:52:37] ...trap/engines/engine.py.224: train: epoch 24 | batch 1220/2304 [I 2022-10-26 18:52:37] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:06 [I 2022-10-26 18:52:37] ...trap/engines/engine.py.229: process: 0.55557 | load: 0.00242 [I 2022-10-26 18:52:37] ...trap/engines/engine.py.230: loss: 0.36413 [I 2022-10-26 18:52:37] ...optimizers/trijoint.py.151: total_norm: 22.931311 [I 2022-10-26 18:52:42] ...trap/engines/engine.py.224: train: epoch 24 | batch 1230/2304 [I 2022-10-26 18:52:42] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:01 [I 2022-10-26 18:52:42] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00244 [I 2022-10-26 18:52:42] ...trap/engines/engine.py.230: loss: 0.16547 [I 2022-10-26 18:52:42] ...optimizers/trijoint.py.151: total_norm: 19.170925 [I 2022-10-26 18:52:48] ...trap/engines/engine.py.224: train: epoch 24 | batch 1240/2304 [I 2022-10-26 18:52:48] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:56 [I 2022-10-26 18:52:48] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00249 [I 2022-10-26 18:52:48] ...trap/engines/engine.py.230: loss: 0.18830 [I 2022-10-26 18:52:48] ...optimizers/trijoint.py.151: total_norm: 41.727615 [I 2022-10-26 18:52:54] ...trap/engines/engine.py.224: train: epoch 24 | batch 1250/2304 [I 2022-10-26 18:52:54] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:46 [I 2022-10-26 18:52:54] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00246 [I 2022-10-26 18:52:54] ...trap/engines/engine.py.230: loss: 0.17829 [I 2022-10-26 18:52:54] ...optimizers/trijoint.py.151: total_norm: 50.082615 [I 2022-10-26 18:52:59] ...trap/engines/engine.py.224: train: epoch 24 | batch 1260/2304 [I 2022-10-26 18:52:59] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:46 [I 2022-10-26 18:52:59] ...trap/engines/engine.py.229: process: 0.56383 | load: 0.00247 [I 2022-10-26 18:52:59] ...trap/engines/engine.py.230: loss: 0.13792 [I 2022-10-26 18:52:59] ...optimizers/trijoint.py.151: total_norm: 14.567048 [I 2022-10-26 18:53:05] ...trap/engines/engine.py.224: train: epoch 24 | batch 1270/2304 [I 2022-10-26 18:53:05] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:39 [I 2022-10-26 18:53:05] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00248 [I 2022-10-26 18:53:05] ...trap/engines/engine.py.230: loss: 0.23946 [I 2022-10-26 18:53:05] ...optimizers/trijoint.py.151: total_norm: 32.901836 [I 2022-10-26 18:53:10] ...trap/engines/engine.py.224: train: epoch 24 | batch 1280/2304 [I 2022-10-26 18:53:10] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:31 [I 2022-10-26 18:53:10] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00245 [I 2022-10-26 18:53:10] ...trap/engines/engine.py.230: loss: 0.19369 [I 2022-10-26 18:53:10] ...optimizers/trijoint.py.151: total_norm: 20.863382 [I 2022-10-26 18:53:16] ...trap/engines/engine.py.224: train: epoch 24 | batch 1290/2304 [I 2022-10-26 18:53:16] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:25 [I 2022-10-26 18:53:16] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00244 [I 2022-10-26 18:53:16] ...trap/engines/engine.py.230: loss: 0.17977 [I 2022-10-26 18:53:16] ...optimizers/trijoint.py.151: total_norm: 13.541714 [I 2022-10-26 18:53:22] ...trap/engines/engine.py.224: train: epoch 24 | batch 1300/2304 [I 2022-10-26 18:53:22] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:20 [I 2022-10-26 18:53:22] ...trap/engines/engine.py.229: process: 0.55881 | load: 0.00252 [I 2022-10-26 18:53:22] ...trap/engines/engine.py.230: loss: 0.26194 [I 2022-10-26 18:53:22] ...optimizers/trijoint.py.151: total_norm: 24.926065 [I 2022-10-26 18:53:27] ...trap/engines/engine.py.224: train: epoch 24 | batch 1310/2304 [I 2022-10-26 18:53:27] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:13 [I 2022-10-26 18:53:27] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00250 [I 2022-10-26 18:53:27] ...trap/engines/engine.py.230: loss: 0.21919 [I 2022-10-26 18:53:27] ...optimizers/trijoint.py.151: total_norm: 33.553886 [I 2022-10-26 18:53:33] ...trap/engines/engine.py.224: train: epoch 24 | batch 1320/2304 [I 2022-10-26 18:53:33] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:11 [I 2022-10-26 18:53:33] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00242 [I 2022-10-26 18:53:33] ...trap/engines/engine.py.230: loss: 0.16881 [I 2022-10-26 18:53:33] ...optimizers/trijoint.py.151: total_norm: 29.526533 [I 2022-10-26 18:53:38] ...trap/engines/engine.py.224: train: epoch 24 | batch 1330/2304 [I 2022-10-26 18:53:38] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:02 [I 2022-10-26 18:53:38] ...trap/engines/engine.py.229: process: 0.56417 | load: 0.00242 [I 2022-10-26 18:53:38] ...trap/engines/engine.py.230: loss: 0.18147 [I 2022-10-26 18:53:38] ...optimizers/trijoint.py.151: total_norm: 14.982662 [I 2022-10-26 18:53:44] ...trap/engines/engine.py.224: train: epoch 24 | batch 1340/2304 [I 2022-10-26 18:53:44] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:55 [I 2022-10-26 18:53:44] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00250 [I 2022-10-26 18:53:44] ...trap/engines/engine.py.230: loss: 0.14677 [I 2022-10-26 18:53:44] ...optimizers/trijoint.py.151: total_norm: 11.847019 [I 2022-10-26 18:53:50] ...trap/engines/engine.py.224: train: epoch 24 | batch 1350/2304 [I 2022-10-26 18:53:50] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:53 [I 2022-10-26 18:53:50] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00257 [I 2022-10-26 18:53:50] ...trap/engines/engine.py.230: loss: 0.15926 [I 2022-10-26 18:53:50] ...optimizers/trijoint.py.151: total_norm: 11.797050 [I 2022-10-26 18:53:55] ...trap/engines/engine.py.224: train: epoch 24 | batch 1360/2304 [I 2022-10-26 18:53:55] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:44 [I 2022-10-26 18:53:55] ...trap/engines/engine.py.229: process: 0.55124 | load: 0.00249 [I 2022-10-26 18:53:55] ...trap/engines/engine.py.230: loss: 0.14450 [I 2022-10-26 18:53:55] ...optimizers/trijoint.py.151: total_norm: 28.872614 [I 2022-10-26 18:54:01] ...trap/engines/engine.py.224: train: epoch 24 | batch 1370/2304 [I 2022-10-26 18:54:01] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:42 [I 2022-10-26 18:54:01] ...trap/engines/engine.py.229: process: 0.56288 | load: 0.00247 [I 2022-10-26 18:54:01] ...trap/engines/engine.py.230: loss: 0.15003 [I 2022-10-26 18:54:01] ...optimizers/trijoint.py.151: total_norm: 21.530058 [I 2022-10-26 18:54:06] ...trap/engines/engine.py.224: train: epoch 24 | batch 1380/2304 [I 2022-10-26 18:54:06] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:37 [I 2022-10-26 18:54:06] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00245 [I 2022-10-26 18:54:06] ...trap/engines/engine.py.230: loss: 0.28234 [I 2022-10-26 18:54:06] ...optimizers/trijoint.py.151: total_norm: 26.571642 [I 2022-10-26 18:54:12] ...trap/engines/engine.py.224: train: epoch 24 | batch 1390/2304 [I 2022-10-26 18:54:12] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:31 [I 2022-10-26 18:54:12] ...trap/engines/engine.py.229: process: 0.55567 | load: 0.00258 [I 2022-10-26 18:54:12] ...trap/engines/engine.py.230: loss: 0.21446 [I 2022-10-26 18:54:12] ...optimizers/trijoint.py.151: total_norm: 17.372787 [I 2022-10-26 18:54:18] ...trap/engines/engine.py.224: train: epoch 24 | batch 1400/2304 [I 2022-10-26 18:54:18] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:27 [I 2022-10-26 18:54:18] ...trap/engines/engine.py.229: process: 0.55090 | load: 0.00250 [I 2022-10-26 18:54:18] ...trap/engines/engine.py.230: loss: 0.14231 [I 2022-10-26 18:54:18] ...optimizers/trijoint.py.151: total_norm: 27.569275 [I 2022-10-26 18:54:23] ...trap/engines/engine.py.224: train: epoch 24 | batch 1410/2304 [I 2022-10-26 18:54:23] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:17 [I 2022-10-26 18:54:23] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00247 [I 2022-10-26 18:54:23] ...trap/engines/engine.py.230: loss: 0.20213 [I 2022-10-26 18:54:23] ...optimizers/trijoint.py.151: total_norm: 25.655334 [I 2022-10-26 18:54:29] ...trap/engines/engine.py.224: train: epoch 24 | batch 1420/2304 [I 2022-10-26 18:54:29] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:11 [I 2022-10-26 18:54:29] ...trap/engines/engine.py.229: process: 0.56227 | load: 0.00253 [I 2022-10-26 18:54:29] ...trap/engines/engine.py.230: loss: 0.14383 [I 2022-10-26 18:54:29] ...optimizers/trijoint.py.151: total_norm: 23.040955 [I 2022-10-26 18:54:34] ...trap/engines/engine.py.224: train: epoch 24 | batch 1430/2304 [I 2022-10-26 18:54:34] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:05 [I 2022-10-26 18:54:34] ...trap/engines/engine.py.229: process: 0.55578 | load: 0.00246 [I 2022-10-26 18:54:34] ...trap/engines/engine.py.230: loss: 0.16718 [I 2022-10-26 18:54:34] ...optimizers/trijoint.py.151: total_norm: 12.433911 [I 2022-10-26 18:54:40] ...trap/engines/engine.py.224: train: epoch 24 | batch 1440/2304 [I 2022-10-26 18:54:40] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:01 [I 2022-10-26 18:54:40] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00241 [I 2022-10-26 18:54:40] ...trap/engines/engine.py.230: loss: 0.23549 [I 2022-10-26 18:54:40] ...optimizers/trijoint.py.151: total_norm: 67.030563 [I 2022-10-26 18:54:45] ...trap/engines/engine.py.224: train: epoch 24 | batch 1450/2304 [I 2022-10-26 18:54:45] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:52 [I 2022-10-26 18:54:45] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00254 [I 2022-10-26 18:54:45] ...trap/engines/engine.py.230: loss: 0.21528 [I 2022-10-26 18:54:45] ...optimizers/trijoint.py.151: total_norm: 19.725216 [I 2022-10-26 18:54:51] ...trap/engines/engine.py.224: train: epoch 24 | batch 1460/2304 [I 2022-10-26 18:54:51] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:50 [I 2022-10-26 18:54:51] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00252 [I 2022-10-26 18:54:51] ...trap/engines/engine.py.230: loss: 0.29156 [I 2022-10-26 18:54:51] ...optimizers/trijoint.py.151: total_norm: 31.963148 [I 2022-10-26 18:54:57] ...trap/engines/engine.py.224: train: epoch 24 | batch 1470/2304 [I 2022-10-26 18:54:57] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:46 [I 2022-10-26 18:54:57] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00248 [I 2022-10-26 18:54:57] ...trap/engines/engine.py.230: loss: 0.19734 [I 2022-10-26 18:54:57] ...optimizers/trijoint.py.151: total_norm: 16.783365 [I 2022-10-26 18:55:02] ...trap/engines/engine.py.224: train: epoch 24 | batch 1480/2304 [I 2022-10-26 18:55:02] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:39 [I 2022-10-26 18:55:02] ...trap/engines/engine.py.229: process: 0.56521 | load: 0.00242 [I 2022-10-26 18:55:02] ...trap/engines/engine.py.230: loss: 0.18282 [I 2022-10-26 18:55:02] ...optimizers/trijoint.py.151: total_norm: 18.588299 [I 2022-10-26 18:55:08] ...trap/engines/engine.py.224: train: epoch 24 | batch 1490/2304 [I 2022-10-26 18:55:08] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:35 [I 2022-10-26 18:55:08] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00252 [I 2022-10-26 18:55:08] ...trap/engines/engine.py.230: loss: 0.21762 [I 2022-10-26 18:55:08] ...optimizers/trijoint.py.151: total_norm: 21.860939 [I 2022-10-26 18:55:13] ...trap/engines/engine.py.224: train: epoch 24 | batch 1500/2304 [I 2022-10-26 18:55:13] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:27 [I 2022-10-26 18:55:13] ...trap/engines/engine.py.229: process: 0.55301 | load: 0.00247 [I 2022-10-26 18:55:13] ...trap/engines/engine.py.230: loss: 0.18265 [I 2022-10-26 18:55:13] ...optimizers/trijoint.py.151: total_norm: 13.976470 [I 2022-10-26 18:55:19] ...trap/engines/engine.py.224: train: epoch 24 | batch 1510/2304 [I 2022-10-26 18:55:19] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:22 [I 2022-10-26 18:55:19] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00246 [I 2022-10-26 18:55:19] ...trap/engines/engine.py.230: loss: 0.17142 [I 2022-10-26 18:55:19] ...optimizers/trijoint.py.151: total_norm: 13.947188 [I 2022-10-26 18:55:24] ...trap/engines/engine.py.224: train: epoch 24 | batch 1520/2304 [I 2022-10-26 18:55:24] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:19 [I 2022-10-26 18:55:24] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00247 [I 2022-10-26 18:55:24] ...trap/engines/engine.py.230: loss: 0.13615 [I 2022-10-26 18:55:24] ...optimizers/trijoint.py.151: total_norm: 13.807296 [I 2022-10-26 18:55:30] ...trap/engines/engine.py.224: train: epoch 24 | batch 1530/2304 [I 2022-10-26 18:55:30] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:14 [I 2022-10-26 18:55:30] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00245 [I 2022-10-26 18:55:30] ...trap/engines/engine.py.230: loss: 0.19092 [I 2022-10-26 18:55:30] ...optimizers/trijoint.py.151: total_norm: 19.458529 [I 2022-10-26 18:55:36] ...trap/engines/engine.py.224: train: epoch 24 | batch 1540/2304 [I 2022-10-26 18:55:36] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:10 [I 2022-10-26 18:55:36] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00243 [I 2022-10-26 18:55:36] ...trap/engines/engine.py.230: loss: 0.13329 [I 2022-10-26 18:55:36] ...optimizers/trijoint.py.151: total_norm: 17.871517 [I 2022-10-26 18:55:41] ...trap/engines/engine.py.224: train: epoch 24 | batch 1550/2304 [I 2022-10-26 18:55:41] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:01 [I 2022-10-26 18:55:41] ...trap/engines/engine.py.229: process: 0.54691 | load: 0.00247 [I 2022-10-26 18:55:41] ...trap/engines/engine.py.230: loss: 0.24779 [I 2022-10-26 18:55:41] ...optimizers/trijoint.py.151: total_norm: 24.774824 [I 2022-10-26 18:55:47] ...trap/engines/engine.py.224: train: epoch 24 | batch 1560/2304 [I 2022-10-26 18:55:47] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:06:52 [I 2022-10-26 18:55:47] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00252 [I 2022-10-26 18:55:47] ...trap/engines/engine.py.230: loss: 0.16996 [I 2022-10-26 18:55:47] ...optimizers/trijoint.py.151: total_norm: 22.731764 [I 2022-10-26 18:55:53] ...trap/engines/engine.py.224: train: epoch 24 | batch 1570/2304 [I 2022-10-26 18:55:53] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:49 [I 2022-10-26 18:55:53] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00250 [I 2022-10-26 18:55:53] ...trap/engines/engine.py.230: loss: 0.22678 [I 2022-10-26 18:55:53] ...optimizers/trijoint.py.151: total_norm: 31.299553 [I 2022-10-26 18:55:58] ...trap/engines/engine.py.224: train: epoch 24 | batch 1580/2304 [I 2022-10-26 18:55:58] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:41 [I 2022-10-26 18:55:58] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00254 [I 2022-10-26 18:55:58] ...trap/engines/engine.py.230: loss: 0.23292 [I 2022-10-26 18:55:58] ...optimizers/trijoint.py.151: total_norm: 24.182295 [I 2022-10-26 18:56:04] ...trap/engines/engine.py.224: train: epoch 24 | batch 1590/2304 [I 2022-10-26 18:56:04] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:38 [I 2022-10-26 18:56:04] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00244 [I 2022-10-26 18:56:04] ...trap/engines/engine.py.230: loss: 0.15610 [I 2022-10-26 18:56:04] ...optimizers/trijoint.py.151: total_norm: 15.436226 [I 2022-10-26 18:56:09] ...trap/engines/engine.py.224: train: epoch 24 | batch 1600/2304 [I 2022-10-26 18:56:09] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:34 [I 2022-10-26 18:56:09] ...trap/engines/engine.py.229: process: 0.56402 | load: 0.00256 [I 2022-10-26 18:56:09] ...trap/engines/engine.py.230: loss: 0.18024 [I 2022-10-26 18:56:09] ...optimizers/trijoint.py.151: total_norm: 24.109554 [I 2022-10-26 18:56:15] ...trap/engines/engine.py.224: train: epoch 24 | batch 1610/2304 [I 2022-10-26 18:56:15] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:28 [I 2022-10-26 18:56:15] ...trap/engines/engine.py.229: process: 0.56326 | load: 0.00247 [I 2022-10-26 18:56:15] ...trap/engines/engine.py.230: loss: 0.21691 [I 2022-10-26 18:56:15] ...optimizers/trijoint.py.151: total_norm: 20.437082 [I 2022-10-26 18:56:21] ...trap/engines/engine.py.224: train: epoch 24 | batch 1620/2304 [I 2022-10-26 18:56:21] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:27 [I 2022-10-26 18:56:21] ...trap/engines/engine.py.229: process: 0.56242 | load: 0.00244 [I 2022-10-26 18:56:21] ...trap/engines/engine.py.230: loss: 0.15385 [I 2022-10-26 18:56:21] ...optimizers/trijoint.py.151: total_norm: 30.286829 [I 2022-10-26 18:56:26] ...trap/engines/engine.py.224: train: epoch 24 | batch 1630/2304 [I 2022-10-26 18:56:26] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:15 [I 2022-10-26 18:56:26] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00244 [I 2022-10-26 18:56:26] ...trap/engines/engine.py.230: loss: 0.23807 [I 2022-10-26 18:56:26] ...optimizers/trijoint.py.151: total_norm: 15.144879 [I 2022-10-26 18:56:32] ...trap/engines/engine.py.224: train: epoch 24 | batch 1640/2304 [I 2022-10-26 18:56:32] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:09 [I 2022-10-26 18:56:32] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00244 [I 2022-10-26 18:56:32] ...trap/engines/engine.py.230: loss: 0.16435 [I 2022-10-26 18:56:32] ...optimizers/trijoint.py.151: total_norm: 19.194998 [I 2022-10-26 18:56:37] ...trap/engines/engine.py.224: train: epoch 24 | batch 1650/2304 [I 2022-10-26 18:56:37] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:00 [I 2022-10-26 18:56:37] ...trap/engines/engine.py.229: process: 0.54431 | load: 0.00247 [I 2022-10-26 18:56:37] ...trap/engines/engine.py.230: loss: 0.17217 [I 2022-10-26 18:56:37] ...optimizers/trijoint.py.151: total_norm: 21.240393 [I 2022-10-26 18:56:43] ...trap/engines/engine.py.224: train: epoch 24 | batch 1660/2304 [I 2022-10-26 18:56:43] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:05:58 [I 2022-10-26 18:56:43] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00247 [I 2022-10-26 18:56:43] ...trap/engines/engine.py.230: loss: 0.23881 [I 2022-10-26 18:56:43] ...optimizers/trijoint.py.151: total_norm: 27.117090 [I 2022-10-26 18:56:48] ...trap/engines/engine.py.224: train: epoch 24 | batch 1670/2304 [I 2022-10-26 18:56:48] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:52 [I 2022-10-26 18:56:48] ...trap/engines/engine.py.229: process: 0.55295 | load: 0.00302 [I 2022-10-26 18:56:48] ...trap/engines/engine.py.230: loss: 0.21791 [I 2022-10-26 18:56:48] ...optimizers/trijoint.py.151: total_norm: 14.074788 [I 2022-10-26 18:56:54] ...trap/engines/engine.py.224: train: epoch 24 | batch 1680/2304 [I 2022-10-26 18:56:54] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:59 [I 2022-10-26 18:56:54] ...trap/engines/engine.py.229: process: 0.54792 | load: 0.00262 [I 2022-10-26 18:56:54] ...trap/engines/engine.py.230: loss: 0.18940 [I 2022-10-26 18:56:54] ...optimizers/trijoint.py.151: total_norm: 23.863104 [I 2022-10-26 18:57:00] ...trap/engines/engine.py.224: train: epoch 24 | batch 1690/2304 [I 2022-10-26 18:57:00] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:44 [I 2022-10-26 18:57:00] ...trap/engines/engine.py.229: process: 0.55611 | load: 0.00248 [I 2022-10-26 18:57:00] ...trap/engines/engine.py.230: loss: 0.22504 [I 2022-10-26 18:57:00] ...optimizers/trijoint.py.151: total_norm: 19.889927 [I 2022-10-26 18:57:05] ...trap/engines/engine.py.224: train: epoch 24 | batch 1700/2304 [I 2022-10-26 18:57:05] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:37 [I 2022-10-26 18:57:05] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00246 [I 2022-10-26 18:57:05] ...trap/engines/engine.py.230: loss: 0.28288 [I 2022-10-26 18:57:05] ...optimizers/trijoint.py.151: total_norm: 14.813966 [I 2022-10-26 18:57:11] ...trap/engines/engine.py.224: train: epoch 24 | batch 1710/2304 [I 2022-10-26 18:57:11] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:32 [I 2022-10-26 18:57:11] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00245 [I 2022-10-26 18:57:11] ...trap/engines/engine.py.230: loss: 0.20498 [I 2022-10-26 18:57:11] ...optimizers/trijoint.py.151: total_norm: 18.641809 [I 2022-10-26 18:57:17] ...trap/engines/engine.py.224: train: epoch 24 | batch 1720/2304 [I 2022-10-26 18:57:17] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:28 [I 2022-10-26 18:57:17] ...trap/engines/engine.py.229: process: 0.57538 | load: 0.00249 [I 2022-10-26 18:57:17] ...trap/engines/engine.py.230: loss: 0.15823 [I 2022-10-26 18:57:17] ...optimizers/trijoint.py.151: total_norm: 19.684990 [I 2022-10-26 18:57:22] ...trap/engines/engine.py.224: train: epoch 24 | batch 1730/2304 [I 2022-10-26 18:57:22] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:18 [I 2022-10-26 18:57:22] ...trap/engines/engine.py.229: process: 0.57093 | load: 0.00250 [I 2022-10-26 18:57:22] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-26 18:57:22] ...optimizers/trijoint.py.151: total_norm: 13.616418 [I 2022-10-26 18:57:28] ...trap/engines/engine.py.224: train: epoch 24 | batch 1740/2304 [I 2022-10-26 18:57:28] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:18 [I 2022-10-26 18:57:28] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00247 [I 2022-10-26 18:57:28] ...trap/engines/engine.py.230: loss: 0.24813 [I 2022-10-26 18:57:28] ...optimizers/trijoint.py.151: total_norm: 41.866051 [I 2022-10-26 18:57:33] ...trap/engines/engine.py.224: train: epoch 24 | batch 1750/2304 [I 2022-10-26 18:57:33] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:09 [I 2022-10-26 18:57:33] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00243 [I 2022-10-26 18:57:33] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-26 18:57:33] ...optimizers/trijoint.py.151: total_norm: 12.787686 [I 2022-10-26 18:57:39] ...trap/engines/engine.py.224: train: epoch 24 | batch 1760/2304 [I 2022-10-26 18:57:39] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:02 [I 2022-10-26 18:57:39] ...trap/engines/engine.py.229: process: 0.54500 | load: 0.00243 [I 2022-10-26 18:57:39] ...trap/engines/engine.py.230: loss: 0.16533 [I 2022-10-26 18:57:39] ...optimizers/trijoint.py.151: total_norm: 19.261585 [I 2022-10-26 18:57:44] ...trap/engines/engine.py.224: train: epoch 24 | batch 1770/2304 [I 2022-10-26 18:57:44] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:04:57 [I 2022-10-26 18:57:44] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00240 [I 2022-10-26 18:57:44] ...trap/engines/engine.py.230: loss: 0.20701 [I 2022-10-26 18:57:44] ...optimizers/trijoint.py.151: total_norm: 23.378691 [I 2022-10-26 18:57:50] ...trap/engines/engine.py.224: train: epoch 24 | batch 1780/2304 [I 2022-10-26 18:57:50] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:52 [I 2022-10-26 18:57:50] ...trap/engines/engine.py.229: process: 0.55525 | load: 0.00243 [I 2022-10-26 18:57:50] ...trap/engines/engine.py.230: loss: 0.15680 [I 2022-10-26 18:57:50] ...optimizers/trijoint.py.151: total_norm: 16.652260 [I 2022-10-26 18:57:56] ...trap/engines/engine.py.224: train: epoch 24 | batch 1790/2304 [I 2022-10-26 18:57:56] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:47 [I 2022-10-26 18:57:56] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00242 [I 2022-10-26 18:57:56] ...trap/engines/engine.py.230: loss: 0.25217 [I 2022-10-26 18:57:56] ...optimizers/trijoint.py.151: total_norm: 31.947248 [I 2022-10-26 18:58:01] ...trap/engines/engine.py.224: train: epoch 24 | batch 1800/2304 [I 2022-10-26 18:58:01] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:40 [I 2022-10-26 18:58:01] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00250 [I 2022-10-26 18:58:01] ...trap/engines/engine.py.230: loss: 0.24489 [I 2022-10-26 18:58:01] ...optimizers/trijoint.py.151: total_norm: 22.607073 [I 2022-10-26 18:58:07] ...trap/engines/engine.py.224: train: epoch 24 | batch 1810/2304 [I 2022-10-26 18:58:07] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:36 [I 2022-10-26 18:58:07] ...trap/engines/engine.py.229: process: 0.55006 | load: 0.00245 [I 2022-10-26 18:58:07] ...trap/engines/engine.py.230: loss: 0.15191 [I 2022-10-26 18:58:07] ...optimizers/trijoint.py.151: total_norm: 12.248823 [I 2022-10-26 18:58:12] ...trap/engines/engine.py.224: train: epoch 24 | batch 1820/2304 [I 2022-10-26 18:58:12] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:29 [I 2022-10-26 18:58:12] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00243 [I 2022-10-26 18:58:12] ...trap/engines/engine.py.230: loss: 0.17648 [I 2022-10-26 18:58:12] ...optimizers/trijoint.py.151: total_norm: 20.864752 [I 2022-10-26 18:58:18] ...trap/engines/engine.py.224: train: epoch 24 | batch 1830/2304 [I 2022-10-26 18:58:18] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:24 [I 2022-10-26 18:58:18] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00247 [I 2022-10-26 18:58:18] ...trap/engines/engine.py.230: loss: 0.31964 [I 2022-10-26 18:58:18] ...optimizers/trijoint.py.151: total_norm: 39.249588 [I 2022-10-26 18:58:24] ...trap/engines/engine.py.224: train: epoch 24 | batch 1840/2304 [I 2022-10-26 18:58:24] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:17 [I 2022-10-26 18:58:24] ...trap/engines/engine.py.229: process: 0.55478 | load: 0.00245 [I 2022-10-26 18:58:24] ...trap/engines/engine.py.230: loss: 0.33698 [I 2022-10-26 18:58:24] ...optimizers/trijoint.py.151: total_norm: 28.878256 [I 2022-10-26 18:58:29] ...trap/engines/engine.py.224: train: epoch 24 | batch 1850/2304 [I 2022-10-26 18:58:29] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:13 [I 2022-10-26 18:58:29] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00260 [I 2022-10-26 18:58:29] ...trap/engines/engine.py.230: loss: 0.21304 [I 2022-10-26 18:58:29] ...optimizers/trijoint.py.151: total_norm: 37.857788 [I 2022-10-26 18:58:35] ...trap/engines/engine.py.224: train: epoch 24 | batch 1860/2304 [I 2022-10-26 18:58:35] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:07 [I 2022-10-26 18:58:35] ...trap/engines/engine.py.229: process: 0.55949 | load: 0.00265 [I 2022-10-26 18:58:35] ...trap/engines/engine.py.230: loss: 0.16125 [I 2022-10-26 18:58:35] ...optimizers/trijoint.py.151: total_norm: 14.512586 [I 2022-10-26 18:58:41] ...trap/engines/engine.py.224: train: epoch 24 | batch 1870/2304 [I 2022-10-26 18:58:41] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:06 [I 2022-10-26 18:58:41] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00244 [I 2022-10-26 18:58:41] ...trap/engines/engine.py.230: loss: 0.16001 [I 2022-10-26 18:58:41] ...optimizers/trijoint.py.151: total_norm: 22.178446 [I 2022-10-26 18:58:46] ...trap/engines/engine.py.224: train: epoch 24 | batch 1880/2304 [I 2022-10-26 18:58:46] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:56 [I 2022-10-26 18:58:46] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00246 [I 2022-10-26 18:58:46] ...trap/engines/engine.py.230: loss: 0.16245 [I 2022-10-26 18:58:46] ...optimizers/trijoint.py.151: total_norm: 14.208336 [I 2022-10-26 18:58:52] ...trap/engines/engine.py.224: train: epoch 24 | batch 1890/2304 [I 2022-10-26 18:58:52] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:03:50 [I 2022-10-26 18:58:52] ...trap/engines/engine.py.229: process: 0.55543 | load: 0.00244 [I 2022-10-26 18:58:52] ...trap/engines/engine.py.230: loss: 0.14084 [I 2022-10-26 18:58:52] ...optimizers/trijoint.py.151: total_norm: 26.489178 [I 2022-10-26 18:58:57] ...trap/engines/engine.py.224: train: epoch 24 | batch 1900/2304 [I 2022-10-26 18:58:57] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:45 [I 2022-10-26 18:58:57] ...trap/engines/engine.py.229: process: 0.55857 | load: 0.00254 [I 2022-10-26 18:58:57] ...trap/engines/engine.py.230: loss: 0.19395 [I 2022-10-26 18:58:57] ...optimizers/trijoint.py.151: total_norm: 25.377672 [I 2022-10-26 18:59:03] ...trap/engines/engine.py.224: train: epoch 24 | batch 1910/2304 [I 2022-10-26 18:59:03] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:39 [I 2022-10-26 18:59:03] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00247 [I 2022-10-26 18:59:03] ...trap/engines/engine.py.230: loss: 0.23133 [I 2022-10-26 18:59:03] ...optimizers/trijoint.py.151: total_norm: 25.605164 [I 2022-10-26 18:59:08] ...trap/engines/engine.py.224: train: epoch 24 | batch 1920/2304 [I 2022-10-26 18:59:08] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:33 [I 2022-10-26 18:59:08] ...trap/engines/engine.py.229: process: 0.54713 | load: 0.00248 [I 2022-10-26 18:59:08] ...trap/engines/engine.py.230: loss: 0.20997 [I 2022-10-26 18:59:08] ...optimizers/trijoint.py.151: total_norm: 26.665466 [I 2022-10-26 18:59:14] ...trap/engines/engine.py.224: train: epoch 24 | batch 1930/2304 [I 2022-10-26 18:59:14] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:28 [I 2022-10-26 18:59:14] ...trap/engines/engine.py.229: process: 0.54497 | load: 0.00248 [I 2022-10-26 18:59:14] ...trap/engines/engine.py.230: loss: 0.17036 [I 2022-10-26 18:59:14] ...optimizers/trijoint.py.151: total_norm: 25.343292 [I 2022-10-26 18:59:20] ...trap/engines/engine.py.224: train: epoch 24 | batch 1940/2304 [I 2022-10-26 18:59:20] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:25 [I 2022-10-26 18:59:20] ...trap/engines/engine.py.229: process: 0.56513 | load: 0.00254 [I 2022-10-26 18:59:20] ...trap/engines/engine.py.230: loss: 0.25569 [I 2022-10-26 18:59:20] ...optimizers/trijoint.py.151: total_norm: 24.366777 [I 2022-10-26 18:59:25] ...trap/engines/engine.py.224: train: epoch 24 | batch 1950/2304 [I 2022-10-26 18:59:25] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:17 [I 2022-10-26 18:59:25] ...trap/engines/engine.py.229: process: 0.56117 | load: 0.00247 [I 2022-10-26 18:59:25] ...trap/engines/engine.py.230: loss: 0.23849 [I 2022-10-26 18:59:25] ...optimizers/trijoint.py.151: total_norm: 30.059223 [I 2022-10-26 18:59:31] ...trap/engines/engine.py.224: train: epoch 24 | batch 1960/2304 [I 2022-10-26 18:59:31] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:12 [I 2022-10-26 18:59:31] ...trap/engines/engine.py.229: process: 0.56183 | load: 0.00245 [I 2022-10-26 18:59:31] ...trap/engines/engine.py.230: loss: 0.15561 [I 2022-10-26 18:59:31] ...optimizers/trijoint.py.151: total_norm: 17.070765 [I 2022-10-26 18:59:36] ...trap/engines/engine.py.224: train: epoch 24 | batch 1970/2304 [I 2022-10-26 18:59:36] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:04 [I 2022-10-26 18:59:36] ...trap/engines/engine.py.229: process: 0.54627 | load: 0.00244 [I 2022-10-26 18:59:36] ...trap/engines/engine.py.230: loss: 0.19259 [I 2022-10-26 18:59:36] ...optimizers/trijoint.py.151: total_norm: 17.918394 [I 2022-10-26 18:59:42] ...trap/engines/engine.py.224: train: epoch 24 | batch 1980/2304 [I 2022-10-26 18:59:42] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:00 [I 2022-10-26 18:59:42] ...trap/engines/engine.py.229: process: 0.56658 | load: 0.00253 [I 2022-10-26 18:59:42] ...trap/engines/engine.py.230: loss: 0.30721 [I 2022-10-26 18:59:42] ...optimizers/trijoint.py.151: total_norm: 28.353706 [I 2022-10-26 18:59:48] ...trap/engines/engine.py.224: train: epoch 24 | batch 1990/2304 [I 2022-10-26 18:59:48] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:54 [I 2022-10-26 18:59:48] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00251 [I 2022-10-26 18:59:48] ...trap/engines/engine.py.230: loss: 0.21727 [I 2022-10-26 18:59:48] ...optimizers/trijoint.py.151: total_norm: 23.441830 [I 2022-10-26 18:59:53] ...trap/engines/engine.py.224: train: epoch 24 | batch 2000/2304 [I 2022-10-26 18:59:53] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:02:52 [I 2022-10-26 18:59:53] ...trap/engines/engine.py.229: process: 0.56759 | load: 0.00248 [I 2022-10-26 18:59:53] ...trap/engines/engine.py.230: loss: 0.18217 [I 2022-10-26 18:59:53] ...optimizers/trijoint.py.151: total_norm: 25.022623 [I 2022-10-26 18:59:59] ...trap/engines/engine.py.224: train: epoch 24 | batch 2010/2304 [I 2022-10-26 18:59:59] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:44 [I 2022-10-26 18:59:59] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00246 [I 2022-10-26 18:59:59] ...trap/engines/engine.py.230: loss: 0.21761 [I 2022-10-26 18:59:59] ...optimizers/trijoint.py.151: total_norm: 20.078302 [I 2022-10-26 19:00:05] ...trap/engines/engine.py.224: train: epoch 24 | batch 2020/2304 [I 2022-10-26 19:00:05] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:39 [I 2022-10-26 19:00:05] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00250 [I 2022-10-26 19:00:05] ...trap/engines/engine.py.230: loss: 0.15373 [I 2022-10-26 19:00:05] ...optimizers/trijoint.py.151: total_norm: 10.716215 [I 2022-10-26 19:00:10] ...trap/engines/engine.py.224: train: epoch 24 | batch 2030/2304 [I 2022-10-26 19:00:10] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:34 [I 2022-10-26 19:00:10] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00248 [I 2022-10-26 19:00:10] ...trap/engines/engine.py.230: loss: 0.20494 [I 2022-10-26 19:00:10] ...optimizers/trijoint.py.151: total_norm: 19.181540 [I 2022-10-26 19:00:16] ...trap/engines/engine.py.224: train: epoch 24 | batch 2040/2304 [I 2022-10-26 19:00:16] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:27 [I 2022-10-26 19:00:16] ...trap/engines/engine.py.229: process: 0.55171 | load: 0.00244 [I 2022-10-26 19:00:16] ...trap/engines/engine.py.230: loss: 0.16211 [I 2022-10-26 19:00:16] ...optimizers/trijoint.py.151: total_norm: 24.940458 [I 2022-10-26 19:00:21] ...trap/engines/engine.py.224: train: epoch 24 | batch 2050/2304 [I 2022-10-26 19:00:21] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:21 [I 2022-10-26 19:00:21] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00243 [I 2022-10-26 19:00:21] ...trap/engines/engine.py.230: loss: 0.21063 [I 2022-10-26 19:00:21] ...optimizers/trijoint.py.151: total_norm: 20.952000 [I 2022-10-26 19:00:27] ...trap/engines/engine.py.224: train: epoch 24 | batch 2060/2304 [I 2022-10-26 19:00:27] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:18 [I 2022-10-26 19:00:27] ...trap/engines/engine.py.229: process: 0.57607 | load: 0.00254 [I 2022-10-26 19:00:27] ...trap/engines/engine.py.230: loss: 0.13986 [I 2022-10-26 19:00:27] ...optimizers/trijoint.py.151: total_norm: 12.506983 [I 2022-10-26 19:00:33] ...trap/engines/engine.py.224: train: epoch 24 | batch 2070/2304 [I 2022-10-26 19:00:33] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:10 [I 2022-10-26 19:00:33] ...trap/engines/engine.py.229: process: 0.56258 | load: 0.00249 [I 2022-10-26 19:00:33] ...trap/engines/engine.py.230: loss: 0.38449 [I 2022-10-26 19:00:33] ...optimizers/trijoint.py.151: total_norm: 31.775764 [I 2022-10-26 19:00:38] ...trap/engines/engine.py.224: train: epoch 24 | batch 2080/2304 [I 2022-10-26 19:00:38] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:05 [I 2022-10-26 19:00:38] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00248 [I 2022-10-26 19:00:38] ...trap/engines/engine.py.230: loss: 0.15030 [I 2022-10-26 19:00:38] ...optimizers/trijoint.py.151: total_norm: 19.490509 [I 2022-10-26 19:00:44] ...trap/engines/engine.py.224: train: epoch 24 | batch 2090/2304 [I 2022-10-26 19:00:44] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:01:58 [I 2022-10-26 19:00:44] ...trap/engines/engine.py.229: process: 0.54910 | load: 0.00246 [I 2022-10-26 19:00:44] ...trap/engines/engine.py.230: loss: 0.20868 [I 2022-10-26 19:00:44] ...optimizers/trijoint.py.151: total_norm: 26.461039 [I 2022-10-26 19:00:49] ...trap/engines/engine.py.224: train: epoch 24 | batch 2100/2304 [I 2022-10-26 19:00:49] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:53 [I 2022-10-26 19:00:49] ...trap/engines/engine.py.229: process: 0.54732 | load: 0.00242 [I 2022-10-26 19:00:49] ...trap/engines/engine.py.230: loss: 0.25145 [I 2022-10-26 19:00:49] ...optimizers/trijoint.py.151: total_norm: 27.118994 [I 2022-10-26 19:00:55] ...trap/engines/engine.py.224: train: epoch 24 | batch 2110/2304 [I 2022-10-26 19:00:55] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:01:48 [I 2022-10-26 19:00:55] ...trap/engines/engine.py.229: process: 0.56930 | load: 0.00246 [I 2022-10-26 19:00:55] ...trap/engines/engine.py.230: loss: 0.23297 [I 2022-10-26 19:00:55] ...optimizers/trijoint.py.151: total_norm: 20.127266 [I 2022-10-26 19:01:01] ...trap/engines/engine.py.224: train: epoch 24 | batch 2120/2304 [I 2022-10-26 19:01:01] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:42 [I 2022-10-26 19:01:01] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00251 [I 2022-10-26 19:01:01] ...trap/engines/engine.py.230: loss: 0.26039 [I 2022-10-26 19:01:01] ...optimizers/trijoint.py.151: total_norm: 30.709312 [I 2022-10-26 19:01:06] ...trap/engines/engine.py.224: train: epoch 24 | batch 2130/2304 [I 2022-10-26 19:01:06] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:37 [I 2022-10-26 19:01:06] ...trap/engines/engine.py.229: process: 0.55865 | load: 0.00246 [I 2022-10-26 19:01:06] ...trap/engines/engine.py.230: loss: 0.20635 [I 2022-10-26 19:01:06] ...optimizers/trijoint.py.151: total_norm: 37.237717 [I 2022-10-26 19:01:12] ...trap/engines/engine.py.224: train: epoch 24 | batch 2140/2304 [I 2022-10-26 19:01:12] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:31 [I 2022-10-26 19:01:12] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00242 [I 2022-10-26 19:01:12] ...trap/engines/engine.py.230: loss: 0.14184 [I 2022-10-26 19:01:12] ...optimizers/trijoint.py.151: total_norm: 12.851035 [I 2022-10-26 19:01:17] ...trap/engines/engine.py.224: train: epoch 24 | batch 2150/2304 [I 2022-10-26 19:01:17] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:25 [I 2022-10-26 19:01:17] ...trap/engines/engine.py.229: process: 0.56265 | load: 0.00245 [I 2022-10-26 19:01:17] ...trap/engines/engine.py.230: loss: 0.23378 [I 2022-10-26 19:01:17] ...optimizers/trijoint.py.151: total_norm: 23.857166 [I 2022-10-26 19:01:23] ...trap/engines/engine.py.224: train: epoch 24 | batch 2160/2304 [I 2022-10-26 19:01:23] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:19 [I 2022-10-26 19:01:23] ...trap/engines/engine.py.229: process: 0.55829 | load: 0.00243 [I 2022-10-26 19:01:23] ...trap/engines/engine.py.230: loss: 0.20049 [I 2022-10-26 19:01:23] ...optimizers/trijoint.py.151: total_norm: 22.959820 [I 2022-10-26 19:01:28] ...trap/engines/engine.py.224: train: epoch 24 | batch 2170/2304 [I 2022-10-26 19:01:28] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:14 [I 2022-10-26 19:01:28] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00242 [I 2022-10-26 19:01:28] ...trap/engines/engine.py.230: loss: 0.15907 [I 2022-10-26 19:01:28] ...optimizers/trijoint.py.151: total_norm: 22.254290 [I 2022-10-26 19:01:34] ...trap/engines/engine.py.224: train: epoch 24 | batch 2180/2304 [I 2022-10-26 19:01:34] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:08 [I 2022-10-26 19:01:34] ...trap/engines/engine.py.229: process: 0.54410 | load: 0.00246 [I 2022-10-26 19:01:34] ...trap/engines/engine.py.230: loss: 0.22167 [I 2022-10-26 19:01:34] ...optimizers/trijoint.py.151: total_norm: 15.659992 [I 2022-10-26 19:01:39] ...trap/engines/engine.py.224: train: epoch 24 | batch 2190/2304 [I 2022-10-26 19:01:39] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:03 [I 2022-10-26 19:01:39] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00250 [I 2022-10-26 19:01:39] ...trap/engines/engine.py.230: loss: 0.24270 [I 2022-10-26 19:01:39] ...optimizers/trijoint.py.151: total_norm: 31.937193 [I 2022-10-26 19:01:45] ...trap/engines/engine.py.224: train: epoch 24 | batch 2200/2304 [I 2022-10-26 19:01:45] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:57 [I 2022-10-26 19:01:45] ...trap/engines/engine.py.229: process: 0.54922 | load: 0.00251 [I 2022-10-26 19:01:45] ...trap/engines/engine.py.230: loss: 0.18227 [I 2022-10-26 19:01:45] ...optimizers/trijoint.py.151: total_norm: 28.380056 [I 2022-10-26 19:01:51] ...trap/engines/engine.py.224: train: epoch 24 | batch 2210/2304 [I 2022-10-26 19:01:51] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:00:52 [I 2022-10-26 19:01:51] ...trap/engines/engine.py.229: process: 0.56024 | load: 0.00245 [I 2022-10-26 19:01:51] ...trap/engines/engine.py.230: loss: 0.18872 [I 2022-10-26 19:01:51] ...optimizers/trijoint.py.151: total_norm: 34.666237 [I 2022-10-26 19:01:56] ...trap/engines/engine.py.224: train: epoch 24 | batch 2220/2304 [I 2022-10-26 19:01:56] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:00:46 [I 2022-10-26 19:01:56] ...trap/engines/engine.py.229: process: 0.55543 | load: 0.00246 [I 2022-10-26 19:01:56] ...trap/engines/engine.py.230: loss: 0.20341 [I 2022-10-26 19:01:56] ...optimizers/trijoint.py.151: total_norm: 28.789560 [I 2022-10-26 19:02:02] ...trap/engines/engine.py.224: train: epoch 24 | batch 2230/2304 [I 2022-10-26 19:02:02] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:41 [I 2022-10-26 19:02:02] ...trap/engines/engine.py.229: process: 0.56284 | load: 0.00252 [I 2022-10-26 19:02:02] ...trap/engines/engine.py.230: loss: 0.15205 [I 2022-10-26 19:02:02] ...optimizers/trijoint.py.151: total_norm: 20.081161 [I 2022-10-26 19:02:07] ...trap/engines/engine.py.224: train: epoch 24 | batch 2240/2304 [I 2022-10-26 19:02:07] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:35 [I 2022-10-26 19:02:07] ...trap/engines/engine.py.229: process: 0.55399 | load: 0.00246 [I 2022-10-26 19:02:07] ...trap/engines/engine.py.230: loss: 0.19619 [I 2022-10-26 19:02:07] ...optimizers/trijoint.py.151: total_norm: 16.604469 [I 2022-10-26 19:02:13] ...trap/engines/engine.py.224: train: epoch 24 | batch 2250/2304 [I 2022-10-26 19:02:13] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:29 [I 2022-10-26 19:02:13] ...trap/engines/engine.py.229: process: 0.54985 | load: 0.00247 [I 2022-10-26 19:02:13] ...trap/engines/engine.py.230: loss: 0.16922 [I 2022-10-26 19:02:13] ...optimizers/trijoint.py.151: total_norm: 15.225271 [I 2022-10-26 19:02:19] ...trap/engines/engine.py.224: train: epoch 24 | batch 2260/2304 [I 2022-10-26 19:02:19] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:24 [I 2022-10-26 19:02:19] ...trap/engines/engine.py.229: process: 0.55980 | load: 0.00244 [I 2022-10-26 19:02:19] ...trap/engines/engine.py.230: loss: 0.24321 [I 2022-10-26 19:02:19] ...optimizers/trijoint.py.151: total_norm: 24.169554 [I 2022-10-26 19:02:24] ...trap/engines/engine.py.224: train: epoch 24 | batch 2270/2304 [I 2022-10-26 19:02:24] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:18 [I 2022-10-26 19:02:24] ...trap/engines/engine.py.229: process: 0.53469 | load: 0.00242 [I 2022-10-26 19:02:24] ...trap/engines/engine.py.230: loss: 0.15988 [I 2022-10-26 19:02:24] ...optimizers/trijoint.py.151: total_norm: 14.952703 [I 2022-10-26 19:02:30] ...trap/engines/engine.py.224: train: epoch 24 | batch 2280/2304 [I 2022-10-26 19:02:30] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:13 [I 2022-10-26 19:02:30] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00253 [I 2022-10-26 19:02:30] ...trap/engines/engine.py.230: loss: 0.16992 [I 2022-10-26 19:02:30] ...optimizers/trijoint.py.151: total_norm: 17.775354 [I 2022-10-26 19:02:35] ...trap/engines/engine.py.224: train: epoch 24 | batch 2290/2304 [I 2022-10-26 19:02:35] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:07 [I 2022-10-26 19:02:35] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00249 [I 2022-10-26 19:02:35] ...trap/engines/engine.py.230: loss: 0.19218 [I 2022-10-26 19:02:35] ...optimizers/trijoint.py.151: total_norm: 19.686424 [I 2022-10-26 19:02:41] ...trap/engines/engine.py.224: train: epoch 24 | batch 2300/2304 [I 2022-10-26 19:02:41] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:02 [I 2022-10-26 19:02:41] ...trap/engines/engine.py.229: process: 0.54249 | load: 0.00015 [I 2022-10-26 19:02:41] ...trap/engines/engine.py.230: loss: 0.23293 [I 2022-10-26 19:02:41] ...optimizers/trijoint.py.151: total_norm: 20.307020 [I 2022-10-26 19:02:43] ...trap/engines/engine.py.224: train: epoch 24 | batch 2304/2304 [I 2022-10-26 19:02:43] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:00 [I 2022-10-26 19:02:43] ...trap/engines/engine.py.229: process: 0.55309 | load: 0.00027 [I 2022-10-26 19:02:43] ...trap/engines/engine.py.230: loss: 0.20205 [I 2022-10-26 19:02:43] ...optimizers/trijoint.py.151: total_norm: 13.539179 [S 2022-10-26 19:02:44] ...trap/engines/engine.py.129: train_epoch.epoch: 24 [S 2022-10-26 19:02:44] ...trap/engines/engine.py.129: train_epoch.loss: 0.1924101143622864 [I 2022-10-26 19:02:44] ...s/criterions/triplet.py.84: margin = 0.1750000000000001 [I 2022-10-26 19:02:44] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 24 [I 2022-10-26 19:02:44] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 19:02:46] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 19:02:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 19:02:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 19:02:47] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 19:02:47] ...trap/engines/engine.py.321: eval: epoch 24 | batch 0/510 [I 2022-10-26 19:02:47] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:24:45 [I 2022-10-26 19:02:47] ...trap/engines/engine.py.326: process: 2.91210 | load: 2.53441 [I 2022-10-26 19:02:51] ...trap/engines/engine.py.321: eval: epoch 24 | batch 10/510 [I 2022-10-26 19:02:51] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:06 [I 2022-10-26 19:02:51] ...trap/engines/engine.py.326: process: 0.32181 | load: 0.00020 [I 2022-10-26 19:02:54] ...trap/engines/engine.py.321: eval: epoch 24 | batch 20/510 [I 2022-10-26 19:02:54] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:03:02 [I 2022-10-26 19:02:54] ...trap/engines/engine.py.326: process: 0.36368 | load: 0.00021 [I 2022-10-26 19:02:58] ...trap/engines/engine.py.321: eval: epoch 24 | batch 30/510 [I 2022-10-26 19:02:58] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:45 [I 2022-10-26 19:02:58] ...trap/engines/engine.py.326: process: 0.32494 | load: 0.00018 [I 2022-10-26 19:03:01] ...trap/engines/engine.py.321: eval: epoch 24 | batch 40/510 [I 2022-10-26 19:03:01] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:37 [I 2022-10-26 19:03:01] ...trap/engines/engine.py.326: process: 0.33439 | load: 0.00023 [I 2022-10-26 19:03:04] ...trap/engines/engine.py.321: eval: epoch 24 | batch 50/510 [I 2022-10-26 19:03:04] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:34 [I 2022-10-26 19:03:04] ...trap/engines/engine.py.326: process: 0.33424 | load: 0.00021 [I 2022-10-26 19:03:08] ...trap/engines/engine.py.321: eval: epoch 24 | batch 60/510 [I 2022-10-26 19:03:08] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:34 [I 2022-10-26 19:03:08] ...trap/engines/engine.py.326: process: 0.32840 | load: 0.00023 [I 2022-10-26 19:03:11] ...trap/engines/engine.py.321: eval: epoch 24 | batch 70/510 [I 2022-10-26 19:03:11] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:29 [I 2022-10-26 19:03:11] ...trap/engines/engine.py.326: process: 0.33016 | load: 0.00031 [I 2022-10-26 19:03:14] ...trap/engines/engine.py.321: eval: epoch 24 | batch 80/510 [I 2022-10-26 19:03:14] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:25 [I 2022-10-26 19:03:14] ...trap/engines/engine.py.326: process: 0.33823 | load: 0.00016 [I 2022-10-26 19:03:18] ...trap/engines/engine.py.321: eval: epoch 24 | batch 90/510 [I 2022-10-26 19:03:18] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:21 [I 2022-10-26 19:03:18] ...trap/engines/engine.py.326: process: 0.32028 | load: 0.00018 [I 2022-10-26 19:03:21] ...trap/engines/engine.py.321: eval: epoch 24 | batch 100/510 [I 2022-10-26 19:03:21] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:12 [I 2022-10-26 19:03:21] ...trap/engines/engine.py.326: process: 0.28673 | load: 0.00018 [I 2022-10-26 19:03:24] ...trap/engines/engine.py.321: eval: epoch 24 | batch 110/510 [I 2022-10-26 19:03:24] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 19:03:24] ...trap/engines/engine.py.326: process: 0.28707 | load: 0.00014 [I 2022-10-26 19:03:27] ...trap/engines/engine.py.321: eval: epoch 24 | batch 120/510 [I 2022-10-26 19:03:27] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:51 [I 2022-10-26 19:03:27] ...trap/engines/engine.py.326: process: 0.28263 | load: 0.00014 [I 2022-10-26 19:03:30] ...trap/engines/engine.py.321: eval: epoch 24 | batch 130/510 [I 2022-10-26 19:03:30] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 19:03:30] ...trap/engines/engine.py.326: process: 0.28519 | load: 0.00025 [I 2022-10-26 19:03:33] ...trap/engines/engine.py.321: eval: epoch 24 | batch 140/510 [I 2022-10-26 19:03:33] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-26 19:03:33] ...trap/engines/engine.py.326: process: 0.28690 | load: 0.00017 [I 2022-10-26 19:03:36] ...trap/engines/engine.py.321: eval: epoch 24 | batch 150/510 [I 2022-10-26 19:03:36] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:45 [I 2022-10-26 19:03:36] ...trap/engines/engine.py.326: process: 0.29143 | load: 0.00022 [I 2022-10-26 19:03:38] ...trap/engines/engine.py.321: eval: epoch 24 | batch 160/510 [I 2022-10-26 19:03:38] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-26 19:03:38] ...trap/engines/engine.py.326: process: 0.28302 | load: 0.00015 [I 2022-10-26 19:03:41] ...trap/engines/engine.py.321: eval: epoch 24 | batch 170/510 [I 2022-10-26 19:03:41] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:37 [I 2022-10-26 19:03:41] ...trap/engines/engine.py.326: process: 0.28440 | load: 0.00015 [I 2022-10-26 19:03:44] ...trap/engines/engine.py.321: eval: epoch 24 | batch 180/510 [I 2022-10-26 19:03:44] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 19:03:44] ...trap/engines/engine.py.326: process: 0.28969 | load: 0.00015 [I 2022-10-26 19:03:47] ...trap/engines/engine.py.321: eval: epoch 24 | batch 190/510 [I 2022-10-26 19:03:47] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:31 [I 2022-10-26 19:03:47] ...trap/engines/engine.py.326: process: 0.28377 | load: 0.00016 [I 2022-10-26 19:03:50] ...trap/engines/engine.py.321: eval: epoch 24 | batch 200/510 [I 2022-10-26 19:03:50] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 19:03:50] ...trap/engines/engine.py.326: process: 0.29097 | load: 0.00015 [I 2022-10-26 19:03:53] ...trap/engines/engine.py.321: eval: epoch 24 | batch 210/510 [I 2022-10-26 19:03:53] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 19:03:53] ...trap/engines/engine.py.326: process: 0.28549 | load: 0.00015 [I 2022-10-26 19:03:56] ...trap/engines/engine.py.321: eval: epoch 24 | batch 220/510 [I 2022-10-26 19:03:56] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 19:03:56] ...trap/engines/engine.py.326: process: 0.29071 | load: 0.00017 [I 2022-10-26 19:03:59] ...trap/engines/engine.py.321: eval: epoch 24 | batch 230/510 [I 2022-10-26 19:03:59] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 19:03:59] ...trap/engines/engine.py.326: process: 0.28875 | load: 0.00015 [I 2022-10-26 19:04:02] ...trap/engines/engine.py.321: eval: epoch 24 | batch 240/510 [I 2022-10-26 19:04:02] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 19:04:02] ...trap/engines/engine.py.326: process: 0.28731 | load: 0.00015 [I 2022-10-26 19:04:04] ...trap/engines/engine.py.321: eval: epoch 24 | batch 250/510 [I 2022-10-26 19:04:04] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:14 [I 2022-10-26 19:04:04] ...trap/engines/engine.py.326: process: 0.28774 | load: 0.00019 [I 2022-10-26 19:04:07] ...trap/engines/engine.py.321: eval: epoch 24 | batch 260/510 [I 2022-10-26 19:04:07] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:11 [I 2022-10-26 19:04:07] ...trap/engines/engine.py.326: process: 0.28454 | load: 0.00021 [I 2022-10-26 19:04:10] ...trap/engines/engine.py.321: eval: epoch 24 | batch 270/510 [I 2022-10-26 19:04:10] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 19:04:10] ...trap/engines/engine.py.326: process: 0.29235 | load: 0.00016 [I 2022-10-26 19:04:13] ...trap/engines/engine.py.321: eval: epoch 24 | batch 280/510 [I 2022-10-26 19:04:13] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 19:04:13] ...trap/engines/engine.py.326: process: 0.28694 | load: 0.00015 [I 2022-10-26 19:04:16] ...trap/engines/engine.py.321: eval: epoch 24 | batch 290/510 [I 2022-10-26 19:04:16] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 19:04:16] ...trap/engines/engine.py.326: process: 0.29316 | load: 0.00022 [I 2022-10-26 19:04:19] ...trap/engines/engine.py.321: eval: epoch 24 | batch 300/510 [I 2022-10-26 19:04:19] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 19:04:19] ...trap/engines/engine.py.326: process: 0.28870 | load: 0.00015 [I 2022-10-26 19:04:22] ...trap/engines/engine.py.321: eval: epoch 24 | batch 310/510 [I 2022-10-26 19:04:22] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:58 [I 2022-10-26 19:04:22] ...trap/engines/engine.py.326: process: 0.28676 | load: 0.00015 [I 2022-10-26 19:04:25] ...trap/engines/engine.py.321: eval: epoch 24 | batch 320/510 [I 2022-10-26 19:04:25] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:55 [I 2022-10-26 19:04:25] ...trap/engines/engine.py.326: process: 0.28286 | load: 0.00014 [I 2022-10-26 19:04:28] ...trap/engines/engine.py.321: eval: epoch 24 | batch 330/510 [I 2022-10-26 19:04:28] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 19:04:28] ...trap/engines/engine.py.326: process: 0.28819 | load: 0.00020 [I 2022-10-26 19:04:30] ...trap/engines/engine.py.321: eval: epoch 24 | batch 340/510 [I 2022-10-26 19:04:30] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:49 [I 2022-10-26 19:04:30] ...trap/engines/engine.py.326: process: 0.29911 | load: 0.00015 [I 2022-10-26 19:04:33] ...trap/engines/engine.py.321: eval: epoch 24 | batch 350/510 [I 2022-10-26 19:04:33] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 19:04:33] ...trap/engines/engine.py.326: process: 0.28803 | load: 0.00023 [I 2022-10-26 19:04:36] ...trap/engines/engine.py.321: eval: epoch 24 | batch 360/510 [I 2022-10-26 19:04:36] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 19:04:36] ...trap/engines/engine.py.326: process: 0.28959 | load: 0.00015 [I 2022-10-26 19:04:39] ...trap/engines/engine.py.321: eval: epoch 24 | batch 370/510 [I 2022-10-26 19:04:39] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 19:04:39] ...trap/engines/engine.py.326: process: 0.28538 | load: 0.00028 [I 2022-10-26 19:04:42] ...trap/engines/engine.py.321: eval: epoch 24 | batch 380/510 [I 2022-10-26 19:04:42] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 19:04:42] ...trap/engines/engine.py.326: process: 0.28899 | load: 0.00016 [I 2022-10-26 19:04:45] ...trap/engines/engine.py.321: eval: epoch 24 | batch 390/510 [I 2022-10-26 19:04:45] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 19:04:45] ...trap/engines/engine.py.326: process: 0.28440 | load: 0.00015 [I 2022-10-26 19:04:48] ...trap/engines/engine.py.321: eval: epoch 24 | batch 400/510 [I 2022-10-26 19:04:48] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 19:04:48] ...trap/engines/engine.py.326: process: 0.28561 | load: 0.00015 [I 2022-10-26 19:04:51] ...trap/engines/engine.py.321: eval: epoch 24 | batch 410/510 [I 2022-10-26 19:04:51] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 19:04:51] ...trap/engines/engine.py.326: process: 0.28274 | load: 0.00017 [I 2022-10-26 19:04:53] ...trap/engines/engine.py.321: eval: epoch 24 | batch 420/510 [I 2022-10-26 19:04:53] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-26 19:04:53] ...trap/engines/engine.py.326: process: 0.28180 | load: 0.00018 [I 2022-10-26 19:04:56] ...trap/engines/engine.py.321: eval: epoch 24 | batch 430/510 [I 2022-10-26 19:04:56] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-26 19:04:56] ...trap/engines/engine.py.326: process: 0.28642 | load: 0.00016 [I 2022-10-26 19:04:59] ...trap/engines/engine.py.321: eval: epoch 24 | batch 440/510 [I 2022-10-26 19:04:59] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 19:04:59] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00021 [I 2022-10-26 19:05:02] ...trap/engines/engine.py.321: eval: epoch 24 | batch 450/510 [I 2022-10-26 19:05:02] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 19:05:02] ...trap/engines/engine.py.326: process: 0.29188 | load: 0.00016 [I 2022-10-26 19:05:05] ...trap/engines/engine.py.321: eval: epoch 24 | batch 460/510 [I 2022-10-26 19:05:05] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 19:05:05] ...trap/engines/engine.py.326: process: 0.28423 | load: 0.00016 [I 2022-10-26 19:05:08] ...trap/engines/engine.py.321: eval: epoch 24 | batch 470/510 [I 2022-10-26 19:05:08] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 19:05:08] ...trap/engines/engine.py.326: process: 0.28650 | load: 0.00016 [I 2022-10-26 19:05:11] ...trap/engines/engine.py.321: eval: epoch 24 | batch 480/510 [I 2022-10-26 19:05:11] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 19:05:11] ...trap/engines/engine.py.326: process: 0.28458 | load: 0.00016 [I 2022-10-26 19:05:14] ...trap/engines/engine.py.321: eval: epoch 24 | batch 490/510 [I 2022-10-26 19:05:14] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 19:05:14] ...trap/engines/engine.py.326: process: 0.28902 | load: 0.00016 [I 2022-10-26 19:05:16] ...trap/engines/engine.py.321: eval: epoch 24 | batch 500/510 [I 2022-10-26 19:05:16] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-26 19:05:16] ...trap/engines/engine.py.326: process: 0.28480 | load: 0.00014 [I 2022-10-26 19:05:19] ...trap/engines/engine.py.321: eval: epoch 24 | batch 510/510 [I 2022-10-26 19:05:19] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 19:05:19] ...trap/engines/engine.py.326: process: 0.28649 | load: 0.00012 [S 2022-10-26 19:05:20] ...trap/engines/engine.py.132: eval_epoch.epoch: 24 [I 2022-10-26 19:05:20] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.625200 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.867600 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.918100 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.610200 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.864100 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.910900 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.018605 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012823 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010397 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012632 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.014748 [S 2022-10-26 19:05:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010793 [I 2022-10-26 19:05:25] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 19:05:25] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 19:05:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 19:05:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 19:05:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 19:05:26] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 19:05:29] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 19:05:35] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 19:05:35] ...trap/engines/engine.py.163: Training model on trainset for epoch 25 [I 2022-10-26 19:05:35] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 19:05:38] ...trap/engines/engine.py.224: train: epoch 25 | batch 0/2304 [I 2022-10-26 19:05:38] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:49:57 [I 2022-10-26 19:05:38] ...trap/engines/engine.py.229: process: 2.86363 | load: 2.20593 [I 2022-10-26 19:05:38] ...trap/engines/engine.py.230: loss: 0.17608 [I 2022-10-26 19:05:38] ...optimizers/trijoint.py.151: total_norm: 17.890142 [I 2022-10-26 19:05:44] ...trap/engines/engine.py.224: train: epoch 25 | batch 10/2304 [I 2022-10-26 19:05:44] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:54 [I 2022-10-26 19:05:44] ...trap/engines/engine.py.229: process: 0.55524 | load: 0.00446 [I 2022-10-26 19:05:44] ...trap/engines/engine.py.230: loss: 0.20435 [I 2022-10-26 19:05:44] ...optimizers/trijoint.py.151: total_norm: 38.026657 [I 2022-10-26 19:05:50] ...trap/engines/engine.py.224: train: epoch 25 | batch 20/2304 [I 2022-10-26 19:05:50] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:13 [I 2022-10-26 19:05:50] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00304 [I 2022-10-26 19:05:50] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-26 19:05:50] ...optimizers/trijoint.py.151: total_norm: 30.056059 [I 2022-10-26 19:05:55] ...trap/engines/engine.py.224: train: epoch 25 | batch 30/2304 [I 2022-10-26 19:05:55] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:23 [I 2022-10-26 19:05:55] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00305 [I 2022-10-26 19:05:55] ...trap/engines/engine.py.230: loss: 0.20494 [I 2022-10-26 19:05:55] ...optimizers/trijoint.py.151: total_norm: 19.920656 [I 2022-10-26 19:06:01] ...trap/engines/engine.py.224: train: epoch 25 | batch 40/2304 [I 2022-10-26 19:06:01] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:04 [I 2022-10-26 19:06:01] ...trap/engines/engine.py.229: process: 0.54806 | load: 0.00319 [I 2022-10-26 19:06:01] ...trap/engines/engine.py.230: loss: 0.13087 [I 2022-10-26 19:06:01] ...optimizers/trijoint.py.151: total_norm: 10.945505 [I 2022-10-26 19:06:06] ...trap/engines/engine.py.224: train: epoch 25 | batch 50/2304 [I 2022-10-26 19:06:06] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:56 [I 2022-10-26 19:06:06] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00278 [I 2022-10-26 19:06:06] ...trap/engines/engine.py.230: loss: 0.26403 [I 2022-10-26 19:06:06] ...optimizers/trijoint.py.151: total_norm: 31.384113 [I 2022-10-26 19:06:12] ...trap/engines/engine.py.224: train: epoch 25 | batch 60/2304 [I 2022-10-26 19:06:12] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:43 [I 2022-10-26 19:06:12] ...trap/engines/engine.py.229: process: 0.54079 | load: 0.00292 [I 2022-10-26 19:06:12] ...trap/engines/engine.py.230: loss: 0.19565 [I 2022-10-26 19:06:12] ...optimizers/trijoint.py.151: total_norm: 18.538624 [I 2022-10-26 19:06:18] ...trap/engines/engine.py.224: train: epoch 25 | batch 70/2304 [I 2022-10-26 19:06:18] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:21:21 [I 2022-10-26 19:06:18] ...trap/engines/engine.py.229: process: 0.57612 | load: 0.00317 [I 2022-10-26 19:06:18] ...trap/engines/engine.py.230: loss: 0.21406 [I 2022-10-26 19:06:18] ...optimizers/trijoint.py.151: total_norm: 27.819679 [I 2022-10-26 19:06:23] ...trap/engines/engine.py.224: train: epoch 25 | batch 80/2304 [I 2022-10-26 19:06:23] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:39 [I 2022-10-26 19:06:23] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00273 [I 2022-10-26 19:06:23] ...trap/engines/engine.py.230: loss: 0.19204 [I 2022-10-26 19:06:23] ...optimizers/trijoint.py.151: total_norm: 21.098293 [I 2022-10-26 19:06:29] ...trap/engines/engine.py.224: train: epoch 25 | batch 90/2304 [I 2022-10-26 19:06:29] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:29 [I 2022-10-26 19:06:29] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00281 [I 2022-10-26 19:06:29] ...trap/engines/engine.py.230: loss: 0.15314 [I 2022-10-26 19:06:29] ...optimizers/trijoint.py.151: total_norm: 17.690357 [I 2022-10-26 19:06:34] ...trap/engines/engine.py.224: train: epoch 25 | batch 100/2304 [I 2022-10-26 19:06:34] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:27 [I 2022-10-26 19:06:34] ...trap/engines/engine.py.229: process: 0.56256 | load: 0.00295 [I 2022-10-26 19:06:34] ...trap/engines/engine.py.230: loss: 0.27309 [I 2022-10-26 19:06:34] ...optimizers/trijoint.py.151: total_norm: 15.143047 [I 2022-10-26 19:06:40] ...trap/engines/engine.py.224: train: epoch 25 | batch 110/2304 [I 2022-10-26 19:06:40] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:16 [I 2022-10-26 19:06:40] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00255 [I 2022-10-26 19:06:40] ...trap/engines/engine.py.230: loss: 0.16350 [I 2022-10-26 19:06:40] ...optimizers/trijoint.py.151: total_norm: 20.686285 [I 2022-10-26 19:06:45] ...trap/engines/engine.py.224: train: epoch 25 | batch 120/2304 [I 2022-10-26 19:06:45] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:18 [I 2022-10-26 19:06:45] ...trap/engines/engine.py.229: process: 0.54220 | load: 0.00259 [I 2022-10-26 19:06:45] ...trap/engines/engine.py.230: loss: 0.23084 [I 2022-10-26 19:06:45] ...optimizers/trijoint.py.151: total_norm: 34.916550 [I 2022-10-26 19:06:51] ...trap/engines/engine.py.224: train: epoch 25 | batch 130/2304 [I 2022-10-26 19:06:51] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:23 [I 2022-10-26 19:06:51] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00258 [I 2022-10-26 19:06:51] ...trap/engines/engine.py.230: loss: 0.15199 [I 2022-10-26 19:06:51] ...optimizers/trijoint.py.151: total_norm: 13.325104 [I 2022-10-26 19:06:57] ...trap/engines/engine.py.224: train: epoch 25 | batch 140/2304 [I 2022-10-26 19:06:57] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:24 [I 2022-10-26 19:06:57] ...trap/engines/engine.py.229: process: 0.54962 | load: 0.00265 [I 2022-10-26 19:06:57] ...trap/engines/engine.py.230: loss: 0.13843 [I 2022-10-26 19:06:57] ...optimizers/trijoint.py.151: total_norm: 27.302193 [I 2022-10-26 19:07:02] ...trap/engines/engine.py.224: train: epoch 25 | batch 150/2304 [I 2022-10-26 19:07:02] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:56 [I 2022-10-26 19:07:02] ...trap/engines/engine.py.229: process: 0.55814 | load: 0.00261 [I 2022-10-26 19:07:02] ...trap/engines/engine.py.230: loss: 0.17245 [I 2022-10-26 19:07:02] ...optimizers/trijoint.py.151: total_norm: 28.270094 [I 2022-10-26 19:07:08] ...trap/engines/engine.py.224: train: epoch 25 | batch 160/2304 [I 2022-10-26 19:07:08] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:54 [I 2022-10-26 19:07:08] ...trap/engines/engine.py.229: process: 0.56823 | load: 0.00268 [I 2022-10-26 19:07:08] ...trap/engines/engine.py.230: loss: 0.24153 [I 2022-10-26 19:07:08] ...optimizers/trijoint.py.151: total_norm: 14.769986 [I 2022-10-26 19:07:14] ...trap/engines/engine.py.224: train: epoch 25 | batch 170/2304 [I 2022-10-26 19:07:14] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:56 [I 2022-10-26 19:07:14] ...trap/engines/engine.py.229: process: 0.56594 | load: 0.00257 [I 2022-10-26 19:07:14] ...trap/engines/engine.py.230: loss: 0.17684 [I 2022-10-26 19:07:14] ...optimizers/trijoint.py.151: total_norm: 42.073986 [I 2022-10-26 19:07:19] ...trap/engines/engine.py.224: train: epoch 25 | batch 180/2304 [I 2022-10-26 19:07:19] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:49 [I 2022-10-26 19:07:19] ...trap/engines/engine.py.229: process: 0.53302 | load: 0.00255 [I 2022-10-26 19:07:19] ...trap/engines/engine.py.230: loss: 0.15093 [I 2022-10-26 19:07:19] ...optimizers/trijoint.py.151: total_norm: 14.492219 [I 2022-10-26 19:07:25] ...trap/engines/engine.py.224: train: epoch 25 | batch 190/2304 [I 2022-10-26 19:07:25] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:40 [I 2022-10-26 19:07:25] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00260 [I 2022-10-26 19:07:25] ...trap/engines/engine.py.230: loss: 0.25206 [I 2022-10-26 19:07:25] ...optimizers/trijoint.py.151: total_norm: 19.239302 [I 2022-10-26 19:07:30] ...trap/engines/engine.py.224: train: epoch 25 | batch 200/2304 [I 2022-10-26 19:07:30] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:30 [I 2022-10-26 19:07:30] ...trap/engines/engine.py.229: process: 0.54957 | load: 0.00250 [I 2022-10-26 19:07:30] ...trap/engines/engine.py.230: loss: 0.24202 [I 2022-10-26 19:07:30] ...optimizers/trijoint.py.151: total_norm: 29.203259 [I 2022-10-26 19:07:36] ...trap/engines/engine.py.224: train: epoch 25 | batch 210/2304 [I 2022-10-26 19:07:36] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:33 [I 2022-10-26 19:07:36] ...trap/engines/engine.py.229: process: 0.56535 | load: 0.00257 [I 2022-10-26 19:07:36] ...trap/engines/engine.py.230: loss: 0.16350 [I 2022-10-26 19:07:36] ...optimizers/trijoint.py.151: total_norm: 15.912317 [I 2022-10-26 19:07:42] ...trap/engines/engine.py.224: train: epoch 25 | batch 220/2304 [I 2022-10-26 19:07:42] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:30 [I 2022-10-26 19:07:42] ...trap/engines/engine.py.229: process: 0.57075 | load: 0.00267 [I 2022-10-26 19:07:42] ...trap/engines/engine.py.230: loss: 0.10786 [I 2022-10-26 19:07:42] ...optimizers/trijoint.py.151: total_norm: 11.348917 [I 2022-10-26 19:07:47] ...trap/engines/engine.py.224: train: epoch 25 | batch 230/2304 [I 2022-10-26 19:07:47] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:19 [I 2022-10-26 19:07:47] ...trap/engines/engine.py.229: process: 0.55980 | load: 0.00254 [I 2022-10-26 19:07:47] ...trap/engines/engine.py.230: loss: 0.17688 [I 2022-10-26 19:07:47] ...optimizers/trijoint.py.151: total_norm: 19.331059 [I 2022-10-26 19:07:53] ...trap/engines/engine.py.224: train: epoch 25 | batch 240/2304 [I 2022-10-26 19:07:53] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:05 [I 2022-10-26 19:07:53] ...trap/engines/engine.py.229: process: 0.54292 | load: 0.00257 [I 2022-10-26 19:07:53] ...trap/engines/engine.py.230: loss: 0.16269 [I 2022-10-26 19:07:53] ...optimizers/trijoint.py.151: total_norm: 44.008224 [I 2022-10-26 19:07:58] ...trap/engines/engine.py.224: train: epoch 25 | batch 250/2304 [I 2022-10-26 19:07:58] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:08 [I 2022-10-26 19:07:58] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00254 [I 2022-10-26 19:07:58] ...trap/engines/engine.py.230: loss: 0.17197 [I 2022-10-26 19:07:58] ...optimizers/trijoint.py.151: total_norm: 21.854198 [I 2022-10-26 19:08:04] ...trap/engines/engine.py.224: train: epoch 25 | batch 260/2304 [I 2022-10-26 19:08:04] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:02 [I 2022-10-26 19:08:04] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00255 [I 2022-10-26 19:08:04] ...trap/engines/engine.py.230: loss: 0.16505 [I 2022-10-26 19:08:04] ...optimizers/trijoint.py.151: total_norm: 24.419403 [I 2022-10-26 19:08:10] ...trap/engines/engine.py.224: train: epoch 25 | batch 270/2304 [I 2022-10-26 19:08:10] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:19:02 [I 2022-10-26 19:08:10] ...trap/engines/engine.py.229: process: 0.56087 | load: 0.00256 [I 2022-10-26 19:08:10] ...trap/engines/engine.py.230: loss: 0.14729 [I 2022-10-26 19:08:10] ...optimizers/trijoint.py.151: total_norm: 15.289309 [I 2022-10-26 19:08:15] ...trap/engines/engine.py.224: train: epoch 25 | batch 280/2304 [I 2022-10-26 19:08:15] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:58 [I 2022-10-26 19:08:15] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00254 [I 2022-10-26 19:08:15] ...trap/engines/engine.py.230: loss: 0.15254 [I 2022-10-26 19:08:15] ...optimizers/trijoint.py.151: total_norm: 14.764027 [I 2022-10-26 19:08:21] ...trap/engines/engine.py.224: train: epoch 25 | batch 290/2304 [I 2022-10-26 19:08:21] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:46 [I 2022-10-26 19:08:21] ...trap/engines/engine.py.229: process: 0.56402 | load: 0.00248 [I 2022-10-26 19:08:21] ...trap/engines/engine.py.230: loss: 0.18653 [I 2022-10-26 19:08:21] ...optimizers/trijoint.py.151: total_norm: 23.820381 [I 2022-10-26 19:08:26] ...trap/engines/engine.py.224: train: epoch 25 | batch 300/2304 [I 2022-10-26 19:08:26] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:30 [I 2022-10-26 19:08:26] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00253 [I 2022-10-26 19:08:26] ...trap/engines/engine.py.230: loss: 0.14346 [I 2022-10-26 19:08:26] ...optimizers/trijoint.py.151: total_norm: 14.897789 [I 2022-10-26 19:08:32] ...trap/engines/engine.py.224: train: epoch 25 | batch 310/2304 [I 2022-10-26 19:08:32] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:21 [I 2022-10-26 19:08:32] ...trap/engines/engine.py.229: process: 0.54624 | load: 0.00247 [I 2022-10-26 19:08:32] ...trap/engines/engine.py.230: loss: 0.15908 [I 2022-10-26 19:08:32] ...optimizers/trijoint.py.151: total_norm: 13.019592 [I 2022-10-26 19:08:37] ...trap/engines/engine.py.224: train: epoch 25 | batch 320/2304 [I 2022-10-26 19:08:37] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:21 [I 2022-10-26 19:08:37] ...trap/engines/engine.py.229: process: 0.54803 | load: 0.00254 [I 2022-10-26 19:08:38] ...trap/engines/engine.py.230: loss: 0.14319 [I 2022-10-26 19:08:38] ...optimizers/trijoint.py.151: total_norm: 26.181002 [I 2022-10-26 19:08:43] ...trap/engines/engine.py.224: train: epoch 25 | batch 330/2304 [I 2022-10-26 19:08:43] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:08 [I 2022-10-26 19:08:43] ...trap/engines/engine.py.229: process: 0.55119 | load: 0.00248 [I 2022-10-26 19:08:43] ...trap/engines/engine.py.230: loss: 0.19213 [I 2022-10-26 19:08:43] ...optimizers/trijoint.py.151: total_norm: 20.598429 [I 2022-10-26 19:08:49] ...trap/engines/engine.py.224: train: epoch 25 | batch 340/2304 [I 2022-10-26 19:08:49] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:11 [I 2022-10-26 19:08:49] ...trap/engines/engine.py.229: process: 0.54688 | load: 0.00248 [I 2022-10-26 19:08:49] ...trap/engines/engine.py.230: loss: 0.16185 [I 2022-10-26 19:08:49] ...optimizers/trijoint.py.151: total_norm: 25.537449 [I 2022-10-26 19:08:54] ...trap/engines/engine.py.224: train: epoch 25 | batch 350/2304 [I 2022-10-26 19:08:54] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:17:56 [I 2022-10-26 19:08:54] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00248 [I 2022-10-26 19:08:54] ...trap/engines/engine.py.230: loss: 0.16215 [I 2022-10-26 19:08:54] ...optimizers/trijoint.py.151: total_norm: 19.756084 [I 2022-10-26 19:09:00] ...trap/engines/engine.py.224: train: epoch 25 | batch 360/2304 [I 2022-10-26 19:09:00] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:58 [I 2022-10-26 19:09:00] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00243 [I 2022-10-26 19:09:00] ...trap/engines/engine.py.230: loss: 0.28966 [I 2022-10-26 19:09:00] ...optimizers/trijoint.py.151: total_norm: 26.637428 [I 2022-10-26 19:09:05] ...trap/engines/engine.py.224: train: epoch 25 | batch 370/2304 [I 2022-10-26 19:09:05] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:03 [I 2022-10-26 19:09:05] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00244 [I 2022-10-26 19:09:05] ...trap/engines/engine.py.230: loss: 0.22819 [I 2022-10-26 19:09:05] ...optimizers/trijoint.py.151: total_norm: 29.170145 [I 2022-10-26 19:09:11] ...trap/engines/engine.py.224: train: epoch 25 | batch 380/2304 [I 2022-10-26 19:09:11] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:46 [I 2022-10-26 19:09:11] ...trap/engines/engine.py.229: process: 0.54202 | load: 0.00248 [I 2022-10-26 19:09:11] ...trap/engines/engine.py.230: loss: 0.22690 [I 2022-10-26 19:09:11] ...optimizers/trijoint.py.151: total_norm: 21.225460 [I 2022-10-26 19:09:17] ...trap/engines/engine.py.224: train: epoch 25 | batch 390/2304 [I 2022-10-26 19:09:17] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:46 [I 2022-10-26 19:09:17] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00252 [I 2022-10-26 19:09:17] ...trap/engines/engine.py.230: loss: 0.15837 [I 2022-10-26 19:09:17] ...optimizers/trijoint.py.151: total_norm: 17.293797 [I 2022-10-26 19:09:22] ...trap/engines/engine.py.224: train: epoch 25 | batch 400/2304 [I 2022-10-26 19:09:22] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:37 [I 2022-10-26 19:09:22] ...trap/engines/engine.py.229: process: 0.55520 | load: 0.00246 [I 2022-10-26 19:09:22] ...trap/engines/engine.py.230: loss: 0.16642 [I 2022-10-26 19:09:22] ...optimizers/trijoint.py.151: total_norm: 65.986801 [I 2022-10-26 19:09:28] ...trap/engines/engine.py.224: train: epoch 25 | batch 410/2304 [I 2022-10-26 19:09:28] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:37 [I 2022-10-26 19:09:28] ...trap/engines/engine.py.229: process: 0.56697 | load: 0.00243 [I 2022-10-26 19:09:28] ...trap/engines/engine.py.230: loss: 0.16838 [I 2022-10-26 19:09:28] ...optimizers/trijoint.py.151: total_norm: 15.596095 [I 2022-10-26 19:09:33] ...trap/engines/engine.py.224: train: epoch 25 | batch 420/2304 [I 2022-10-26 19:09:33] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:31 [I 2022-10-26 19:09:33] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00249 [I 2022-10-26 19:09:33] ...trap/engines/engine.py.230: loss: 0.17294 [I 2022-10-26 19:09:33] ...optimizers/trijoint.py.151: total_norm: 14.141855 [I 2022-10-26 19:09:39] ...trap/engines/engine.py.224: train: epoch 25 | batch 430/2304 [I 2022-10-26 19:09:39] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:27 [I 2022-10-26 19:09:39] ...trap/engines/engine.py.229: process: 0.56215 | load: 0.00243 [I 2022-10-26 19:09:39] ...trap/engines/engine.py.230: loss: 0.17451 [I 2022-10-26 19:09:39] ...optimizers/trijoint.py.151: total_norm: 15.150064 [I 2022-10-26 19:09:44] ...trap/engines/engine.py.224: train: epoch 25 | batch 440/2304 [I 2022-10-26 19:09:44] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:10 [I 2022-10-26 19:09:44] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00253 [I 2022-10-26 19:09:44] ...trap/engines/engine.py.230: loss: 0.15120 [I 2022-10-26 19:09:44] ...optimizers/trijoint.py.151: total_norm: 18.492720 [I 2022-10-26 19:09:50] ...trap/engines/engine.py.224: train: epoch 25 | batch 450/2304 [I 2022-10-26 19:09:50] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:13 [I 2022-10-26 19:09:50] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00246 [I 2022-10-26 19:09:50] ...trap/engines/engine.py.230: loss: 0.16211 [I 2022-10-26 19:09:50] ...optimizers/trijoint.py.151: total_norm: 16.788960 [I 2022-10-26 19:09:55] ...trap/engines/engine.py.224: train: epoch 25 | batch 460/2304 [I 2022-10-26 19:09:55] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:16:59 [I 2022-10-26 19:09:55] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00244 [I 2022-10-26 19:09:55] ...trap/engines/engine.py.230: loss: 0.21158 [I 2022-10-26 19:09:55] ...optimizers/trijoint.py.151: total_norm: 38.171692 [I 2022-10-26 19:10:01] ...trap/engines/engine.py.224: train: epoch 25 | batch 470/2304 [I 2022-10-26 19:10:01] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:54 [I 2022-10-26 19:10:01] ...trap/engines/engine.py.229: process: 0.54799 | load: 0.00242 [I 2022-10-26 19:10:01] ...trap/engines/engine.py.230: loss: 0.22523 [I 2022-10-26 19:10:01] ...optimizers/trijoint.py.151: total_norm: 17.870895 [I 2022-10-26 19:10:07] ...trap/engines/engine.py.224: train: epoch 25 | batch 480/2304 [I 2022-10-26 19:10:07] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:48 [I 2022-10-26 19:10:07] ...trap/engines/engine.py.229: process: 0.64642 | load: 0.00263 [I 2022-10-26 19:10:07] ...trap/engines/engine.py.230: loss: 0.14317 [I 2022-10-26 19:10:07] ...optimizers/trijoint.py.151: total_norm: 25.418463 [I 2022-10-26 19:10:12] ...trap/engines/engine.py.224: train: epoch 25 | batch 490/2304 [I 2022-10-26 19:10:12] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:45 [I 2022-10-26 19:10:12] ...trap/engines/engine.py.229: process: 0.55015 | load: 0.00243 [I 2022-10-26 19:10:12] ...trap/engines/engine.py.230: loss: 0.18471 [I 2022-10-26 19:10:12] ...optimizers/trijoint.py.151: total_norm: 13.650989 [I 2022-10-26 19:10:18] ...trap/engines/engine.py.224: train: epoch 25 | batch 500/2304 [I 2022-10-26 19:10:18] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:40 [I 2022-10-26 19:10:18] ...trap/engines/engine.py.229: process: 0.55112 | load: 0.00240 [I 2022-10-26 19:10:18] ...trap/engines/engine.py.230: loss: 0.15279 [I 2022-10-26 19:10:18] ...optimizers/trijoint.py.151: total_norm: 20.072351 [I 2022-10-26 19:10:23] ...trap/engines/engine.py.224: train: epoch 25 | batch 510/2304 [I 2022-10-26 19:10:23] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:39 [I 2022-10-26 19:10:23] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00243 [I 2022-10-26 19:10:23] ...trap/engines/engine.py.230: loss: 0.15102 [I 2022-10-26 19:10:23] ...optimizers/trijoint.py.151: total_norm: 22.893391 [I 2022-10-26 19:10:29] ...trap/engines/engine.py.224: train: epoch 25 | batch 520/2304 [I 2022-10-26 19:10:29] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:38 [I 2022-10-26 19:10:29] ...trap/engines/engine.py.229: process: 0.54998 | load: 0.00241 [I 2022-10-26 19:10:29] ...trap/engines/engine.py.230: loss: 0.33689 [I 2022-10-26 19:10:29] ...optimizers/trijoint.py.151: total_norm: 29.373095 [I 2022-10-26 19:10:35] ...trap/engines/engine.py.224: train: epoch 25 | batch 530/2304 [I 2022-10-26 19:10:35] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:28 [I 2022-10-26 19:10:35] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00250 [I 2022-10-26 19:10:35] ...trap/engines/engine.py.230: loss: 0.16354 [I 2022-10-26 19:10:35] ...optimizers/trijoint.py.151: total_norm: 25.507805 [I 2022-10-26 19:10:40] ...trap/engines/engine.py.224: train: epoch 25 | batch 540/2304 [I 2022-10-26 19:10:40] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:23 [I 2022-10-26 19:10:40] ...trap/engines/engine.py.229: process: 0.55442 | load: 0.00240 [I 2022-10-26 19:10:40] ...trap/engines/engine.py.230: loss: 0.26345 [I 2022-10-26 19:10:40] ...optimizers/trijoint.py.151: total_norm: 29.524052 [I 2022-10-26 19:10:46] ...trap/engines/engine.py.224: train: epoch 25 | batch 550/2304 [I 2022-10-26 19:10:46] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:12 [I 2022-10-26 19:10:46] ...trap/engines/engine.py.229: process: 0.55029 | load: 0.00247 [I 2022-10-26 19:10:46] ...trap/engines/engine.py.230: loss: 0.17559 [I 2022-10-26 19:10:46] ...optimizers/trijoint.py.151: total_norm: 13.940657 [I 2022-10-26 19:10:51] ...trap/engines/engine.py.224: train: epoch 25 | batch 560/2304 [I 2022-10-26 19:10:51] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:14 [I 2022-10-26 19:10:51] ...trap/engines/engine.py.229: process: 0.54206 | load: 0.00244 [I 2022-10-26 19:10:51] ...trap/engines/engine.py.230: loss: 0.14027 [I 2022-10-26 19:10:51] ...optimizers/trijoint.py.151: total_norm: 12.953164 [I 2022-10-26 19:10:57] ...trap/engines/engine.py.224: train: epoch 25 | batch 570/2304 [I 2022-10-26 19:10:57] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:03 [I 2022-10-26 19:10:57] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00242 [I 2022-10-26 19:10:57] ...trap/engines/engine.py.230: loss: 0.11360 [I 2022-10-26 19:10:57] ...optimizers/trijoint.py.151: total_norm: 11.709032 [I 2022-10-26 19:11:02] ...trap/engines/engine.py.224: train: epoch 25 | batch 580/2304 [I 2022-10-26 19:11:02] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:56 [I 2022-10-26 19:11:02] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00240 [I 2022-10-26 19:11:02] ...trap/engines/engine.py.230: loss: 0.12701 [I 2022-10-26 19:11:02] ...optimizers/trijoint.py.151: total_norm: 27.709476 [I 2022-10-26 19:11:08] ...trap/engines/engine.py.224: train: epoch 25 | batch 590/2304 [I 2022-10-26 19:11:08] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:49 [I 2022-10-26 19:11:08] ...trap/engines/engine.py.229: process: 0.55578 | load: 0.00242 [I 2022-10-26 19:11:08] ...trap/engines/engine.py.230: loss: 0.24778 [I 2022-10-26 19:11:08] ...optimizers/trijoint.py.151: total_norm: 26.007677 [I 2022-10-26 19:11:14] ...trap/engines/engine.py.224: train: epoch 25 | batch 600/2304 [I 2022-10-26 19:11:14] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:44 [I 2022-10-26 19:11:14] ...trap/engines/engine.py.229: process: 0.54432 | load: 0.00252 [I 2022-10-26 19:11:14] ...trap/engines/engine.py.230: loss: 0.20021 [I 2022-10-26 19:11:14] ...optimizers/trijoint.py.151: total_norm: 19.697605 [I 2022-10-26 19:11:19] ...trap/engines/engine.py.224: train: epoch 25 | batch 610/2304 [I 2022-10-26 19:11:19] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:36 [I 2022-10-26 19:11:19] ...trap/engines/engine.py.229: process: 0.54868 | load: 0.00248 [I 2022-10-26 19:11:19] ...trap/engines/engine.py.230: loss: 0.17097 [I 2022-10-26 19:11:19] ...optimizers/trijoint.py.151: total_norm: 20.366163 [I 2022-10-26 19:11:25] ...trap/engines/engine.py.224: train: epoch 25 | batch 620/2304 [I 2022-10-26 19:11:25] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:38 [I 2022-10-26 19:11:25] ...trap/engines/engine.py.229: process: 0.55868 | load: 0.00240 [I 2022-10-26 19:11:25] ...trap/engines/engine.py.230: loss: 0.21431 [I 2022-10-26 19:11:25] ...optimizers/trijoint.py.151: total_norm: 25.565660 [I 2022-10-26 19:11:30] ...trap/engines/engine.py.224: train: epoch 25 | batch 630/2304 [I 2022-10-26 19:11:30] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:32 [I 2022-10-26 19:11:30] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00246 [I 2022-10-26 19:11:30] ...trap/engines/engine.py.230: loss: 0.26291 [I 2022-10-26 19:11:30] ...optimizers/trijoint.py.151: total_norm: 27.380970 [I 2022-10-26 19:11:36] ...trap/engines/engine.py.224: train: epoch 25 | batch 640/2304 [I 2022-10-26 19:11:36] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:26 [I 2022-10-26 19:11:36] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00243 [I 2022-10-26 19:11:36] ...trap/engines/engine.py.230: loss: 0.22039 [I 2022-10-26 19:11:36] ...optimizers/trijoint.py.151: total_norm: 27.213999 [I 2022-10-26 19:11:41] ...trap/engines/engine.py.224: train: epoch 25 | batch 650/2304 [I 2022-10-26 19:11:41] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:18 [I 2022-10-26 19:11:41] ...trap/engines/engine.py.229: process: 0.55408 | load: 0.00247 [I 2022-10-26 19:11:41] ...trap/engines/engine.py.230: loss: 0.17792 [I 2022-10-26 19:11:41] ...optimizers/trijoint.py.151: total_norm: 32.873737 [I 2022-10-26 19:11:47] ...trap/engines/engine.py.224: train: epoch 25 | batch 660/2304 [I 2022-10-26 19:11:47] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:20 [I 2022-10-26 19:11:47] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00245 [I 2022-10-26 19:11:47] ...trap/engines/engine.py.230: loss: 0.16258 [I 2022-10-26 19:11:47] ...optimizers/trijoint.py.151: total_norm: 24.782356 [I 2022-10-26 19:11:53] ...trap/engines/engine.py.224: train: epoch 25 | batch 670/2304 [I 2022-10-26 19:11:53] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:09 [I 2022-10-26 19:11:53] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00243 [I 2022-10-26 19:11:53] ...trap/engines/engine.py.230: loss: 0.13124 [I 2022-10-26 19:11:53] ...optimizers/trijoint.py.151: total_norm: 16.314905 [I 2022-10-26 19:11:58] ...trap/engines/engine.py.224: train: epoch 25 | batch 680/2304 [I 2022-10-26 19:11:58] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:05 [I 2022-10-26 19:11:58] ...trap/engines/engine.py.229: process: 0.55450 | load: 0.00246 [I 2022-10-26 19:11:58] ...trap/engines/engine.py.230: loss: 0.12587 [I 2022-10-26 19:11:58] ...optimizers/trijoint.py.151: total_norm: 17.975218 [I 2022-10-26 19:12:04] ...trap/engines/engine.py.224: train: epoch 25 | batch 690/2304 [I 2022-10-26 19:12:04] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:56 [I 2022-10-26 19:12:04] ...trap/engines/engine.py.229: process: 0.55088 | load: 0.00246 [I 2022-10-26 19:12:04] ...trap/engines/engine.py.230: loss: 0.19147 [I 2022-10-26 19:12:04] ...optimizers/trijoint.py.151: total_norm: 20.877005 [I 2022-10-26 19:12:09] ...trap/engines/engine.py.224: train: epoch 25 | batch 700/2304 [I 2022-10-26 19:12:09] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:52 [I 2022-10-26 19:12:09] ...trap/engines/engine.py.229: process: 0.56491 | load: 0.00245 [I 2022-10-26 19:12:09] ...trap/engines/engine.py.230: loss: 0.17514 [I 2022-10-26 19:12:09] ...optimizers/trijoint.py.151: total_norm: 21.863676 [I 2022-10-26 19:12:15] ...trap/engines/engine.py.224: train: epoch 25 | batch 710/2304 [I 2022-10-26 19:12:15] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:44 [I 2022-10-26 19:12:15] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00243 [I 2022-10-26 19:12:15] ...trap/engines/engine.py.230: loss: 0.21540 [I 2022-10-26 19:12:15] ...optimizers/trijoint.py.151: total_norm: 16.907537 [I 2022-10-26 19:12:20] ...trap/engines/engine.py.224: train: epoch 25 | batch 720/2304 [I 2022-10-26 19:12:20] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:38 [I 2022-10-26 19:12:20] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00243 [I 2022-10-26 19:12:20] ...trap/engines/engine.py.230: loss: 0.15385 [I 2022-10-26 19:12:20] ...optimizers/trijoint.py.151: total_norm: 23.708319 [I 2022-10-26 19:12:26] ...trap/engines/engine.py.224: train: epoch 25 | batch 730/2304 [I 2022-10-26 19:12:26] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:37 [I 2022-10-26 19:12:26] ...trap/engines/engine.py.229: process: 0.56182 | load: 0.00254 [I 2022-10-26 19:12:26] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-26 19:12:26] ...optimizers/trijoint.py.151: total_norm: 12.523021 [I 2022-10-26 19:12:32] ...trap/engines/engine.py.224: train: epoch 25 | batch 740/2304 [I 2022-10-26 19:12:32] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:31 [I 2022-10-26 19:12:32] ...trap/engines/engine.py.229: process: 0.54983 | load: 0.00245 [I 2022-10-26 19:12:32] ...trap/engines/engine.py.230: loss: 0.16502 [I 2022-10-26 19:12:32] ...optimizers/trijoint.py.151: total_norm: 12.824120 [I 2022-10-26 19:12:37] ...trap/engines/engine.py.224: train: epoch 25 | batch 750/2304 [I 2022-10-26 19:12:37] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:30 [I 2022-10-26 19:12:37] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00242 [I 2022-10-26 19:12:37] ...trap/engines/engine.py.230: loss: 0.18461 [I 2022-10-26 19:12:37] ...optimizers/trijoint.py.151: total_norm: 28.895329 [I 2022-10-26 19:12:43] ...trap/engines/engine.py.224: train: epoch 25 | batch 760/2304 [I 2022-10-26 19:12:43] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:18 [I 2022-10-26 19:12:43] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00244 [I 2022-10-26 19:12:43] ...trap/engines/engine.py.230: loss: 0.14218 [I 2022-10-26 19:12:43] ...optimizers/trijoint.py.151: total_norm: 16.616875 [I 2022-10-26 19:12:48] ...trap/engines/engine.py.224: train: epoch 25 | batch 770/2304 [I 2022-10-26 19:12:48] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:13 [I 2022-10-26 19:12:48] ...trap/engines/engine.py.229: process: 0.54642 | load: 0.00258 [I 2022-10-26 19:12:48] ...trap/engines/engine.py.230: loss: 0.18997 [I 2022-10-26 19:12:48] ...optimizers/trijoint.py.151: total_norm: 19.044962 [I 2022-10-26 19:12:54] ...trap/engines/engine.py.224: train: epoch 25 | batch 780/2304 [I 2022-10-26 19:12:54] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:10 [I 2022-10-26 19:12:54] ...trap/engines/engine.py.229: process: 0.56375 | load: 0.00241 [I 2022-10-26 19:12:54] ...trap/engines/engine.py.230: loss: 0.21863 [I 2022-10-26 19:12:54] ...optimizers/trijoint.py.151: total_norm: 21.157436 [I 2022-10-26 19:13:00] ...trap/engines/engine.py.224: train: epoch 25 | batch 790/2304 [I 2022-10-26 19:13:00] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:02 [I 2022-10-26 19:13:00] ...trap/engines/engine.py.229: process: 0.54998 | load: 0.00250 [I 2022-10-26 19:13:00] ...trap/engines/engine.py.230: loss: 0.24833 [I 2022-10-26 19:13:00] ...optimizers/trijoint.py.151: total_norm: 25.601637 [I 2022-10-26 19:13:05] ...trap/engines/engine.py.224: train: epoch 25 | batch 800/2304 [I 2022-10-26 19:13:05] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:58 [I 2022-10-26 19:13:05] ...trap/engines/engine.py.229: process: 0.54800 | load: 0.00249 [I 2022-10-26 19:13:05] ...trap/engines/engine.py.230: loss: 0.20324 [I 2022-10-26 19:13:05] ...optimizers/trijoint.py.151: total_norm: 26.318565 [I 2022-10-26 19:13:11] ...trap/engines/engine.py.224: train: epoch 25 | batch 810/2304 [I 2022-10-26 19:13:11] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:50 [I 2022-10-26 19:13:11] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00246 [I 2022-10-26 19:13:11] ...trap/engines/engine.py.230: loss: 0.31978 [I 2022-10-26 19:13:11] ...optimizers/trijoint.py.151: total_norm: 33.784481 [I 2022-10-26 19:13:16] ...trap/engines/engine.py.224: train: epoch 25 | batch 820/2304 [I 2022-10-26 19:13:16] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:47 [I 2022-10-26 19:13:16] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00245 [I 2022-10-26 19:13:16] ...trap/engines/engine.py.230: loss: 0.20169 [I 2022-10-26 19:13:16] ...optimizers/trijoint.py.151: total_norm: 28.381155 [I 2022-10-26 19:13:22] ...trap/engines/engine.py.224: train: epoch 25 | batch 830/2304 [I 2022-10-26 19:13:22] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:37 [I 2022-10-26 19:13:22] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00242 [I 2022-10-26 19:13:22] ...trap/engines/engine.py.230: loss: 0.16300 [I 2022-10-26 19:13:22] ...optimizers/trijoint.py.151: total_norm: 19.199669 [I 2022-10-26 19:13:27] ...trap/engines/engine.py.224: train: epoch 25 | batch 840/2304 [I 2022-10-26 19:13:27] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:35 [I 2022-10-26 19:13:27] ...trap/engines/engine.py.229: process: 0.55316 | load: 0.00241 [I 2022-10-26 19:13:27] ...trap/engines/engine.py.230: loss: 0.17526 [I 2022-10-26 19:13:27] ...optimizers/trijoint.py.151: total_norm: 24.632889 [I 2022-10-26 19:13:33] ...trap/engines/engine.py.224: train: epoch 25 | batch 850/2304 [I 2022-10-26 19:13:33] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:33 [I 2022-10-26 19:13:33] ...trap/engines/engine.py.229: process: 0.56924 | load: 0.00251 [I 2022-10-26 19:13:33] ...trap/engines/engine.py.230: loss: 0.25902 [I 2022-10-26 19:13:33] ...optimizers/trijoint.py.151: total_norm: 21.142490 [I 2022-10-26 19:13:39] ...trap/engines/engine.py.224: train: epoch 25 | batch 860/2304 [I 2022-10-26 19:13:39] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:24 [I 2022-10-26 19:13:39] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00249 [I 2022-10-26 19:13:39] ...trap/engines/engine.py.230: loss: 0.28004 [I 2022-10-26 19:13:39] ...optimizers/trijoint.py.151: total_norm: 27.247669 [I 2022-10-26 19:13:44] ...trap/engines/engine.py.224: train: epoch 25 | batch 870/2304 [I 2022-10-26 19:13:44] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:16 [I 2022-10-26 19:13:44] ...trap/engines/engine.py.229: process: 0.54794 | load: 0.00248 [I 2022-10-26 19:13:44] ...trap/engines/engine.py.230: loss: 0.20396 [I 2022-10-26 19:13:44] ...optimizers/trijoint.py.151: total_norm: 27.363861 [I 2022-10-26 19:13:50] ...trap/engines/engine.py.224: train: epoch 25 | batch 880/2304 [I 2022-10-26 19:13:50] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:11 [I 2022-10-26 19:13:50] ...trap/engines/engine.py.229: process: 0.55046 | load: 0.00245 [I 2022-10-26 19:13:50] ...trap/engines/engine.py.230: loss: 0.16772 [I 2022-10-26 19:13:50] ...optimizers/trijoint.py.151: total_norm: 20.087431 [I 2022-10-26 19:13:55] ...trap/engines/engine.py.224: train: epoch 25 | batch 890/2304 [I 2022-10-26 19:13:55] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:12:59 [I 2022-10-26 19:13:55] ...trap/engines/engine.py.229: process: 0.54298 | load: 0.00241 [I 2022-10-26 19:13:55] ...trap/engines/engine.py.230: loss: 0.16692 [I 2022-10-26 19:13:55] ...optimizers/trijoint.py.151: total_norm: 17.240629 [I 2022-10-26 19:14:01] ...trap/engines/engine.py.224: train: epoch 25 | batch 900/2304 [I 2022-10-26 19:14:01] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:02 [I 2022-10-26 19:14:01] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00244 [I 2022-10-26 19:14:01] ...trap/engines/engine.py.230: loss: 0.12229 [I 2022-10-26 19:14:01] ...optimizers/trijoint.py.151: total_norm: 15.687780 [I 2022-10-26 19:14:07] ...trap/engines/engine.py.224: train: epoch 25 | batch 910/2304 [I 2022-10-26 19:14:07] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:13:03 [I 2022-10-26 19:14:07] ...trap/engines/engine.py.229: process: 0.58739 | load: 0.00244 [I 2022-10-26 19:14:07] ...trap/engines/engine.py.230: loss: 0.23208 [I 2022-10-26 19:14:07] ...optimizers/trijoint.py.151: total_norm: 43.206635 [I 2022-10-26 19:14:12] ...trap/engines/engine.py.224: train: epoch 25 | batch 920/2304 [I 2022-10-26 19:14:12] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:51 [I 2022-10-26 19:14:12] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00250 [I 2022-10-26 19:14:12] ...trap/engines/engine.py.230: loss: 0.20718 [I 2022-10-26 19:14:12] ...optimizers/trijoint.py.151: total_norm: 21.740826 [I 2022-10-26 19:14:18] ...trap/engines/engine.py.224: train: epoch 25 | batch 930/2304 [I 2022-10-26 19:14:18] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:48 [I 2022-10-26 19:14:18] ...trap/engines/engine.py.229: process: 0.55897 | load: 0.00269 [I 2022-10-26 19:14:18] ...trap/engines/engine.py.230: loss: 0.16444 [I 2022-10-26 19:14:18] ...optimizers/trijoint.py.151: total_norm: 18.261969 [I 2022-10-26 19:14:23] ...trap/engines/engine.py.224: train: epoch 25 | batch 940/2304 [I 2022-10-26 19:14:23] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:41 [I 2022-10-26 19:14:23] ...trap/engines/engine.py.229: process: 0.55809 | load: 0.00246 [I 2022-10-26 19:14:23] ...trap/engines/engine.py.230: loss: 0.23822 [I 2022-10-26 19:14:23] ...optimizers/trijoint.py.151: total_norm: 19.672033 [I 2022-10-26 19:14:29] ...trap/engines/engine.py.224: train: epoch 25 | batch 950/2304 [I 2022-10-26 19:14:29] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:38 [I 2022-10-26 19:14:29] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00243 [I 2022-10-26 19:14:29] ...trap/engines/engine.py.230: loss: 0.15590 [I 2022-10-26 19:14:29] ...optimizers/trijoint.py.151: total_norm: 16.414335 [I 2022-10-26 19:14:35] ...trap/engines/engine.py.224: train: epoch 25 | batch 960/2304 [I 2022-10-26 19:14:35] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:28 [I 2022-10-26 19:14:35] ...trap/engines/engine.py.229: process: 0.54981 | load: 0.00252 [I 2022-10-26 19:14:35] ...trap/engines/engine.py.230: loss: 0.15102 [I 2022-10-26 19:14:35] ...optimizers/trijoint.py.151: total_norm: 15.046787 [I 2022-10-26 19:14:40] ...trap/engines/engine.py.224: train: epoch 25 | batch 970/2304 [I 2022-10-26 19:14:40] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:20 [I 2022-10-26 19:14:40] ...trap/engines/engine.py.229: process: 0.53695 | load: 0.00251 [I 2022-10-26 19:14:40] ...trap/engines/engine.py.230: loss: 0.16388 [I 2022-10-26 19:14:40] ...optimizers/trijoint.py.151: total_norm: 27.457602 [I 2022-10-26 19:14:46] ...trap/engines/engine.py.224: train: epoch 25 | batch 980/2304 [I 2022-10-26 19:14:46] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:21 [I 2022-10-26 19:14:46] ...trap/engines/engine.py.229: process: 0.55111 | load: 0.00246 [I 2022-10-26 19:14:46] ...trap/engines/engine.py.230: loss: 0.18721 [I 2022-10-26 19:14:46] ...optimizers/trijoint.py.151: total_norm: 20.194201 [I 2022-10-26 19:14:51] ...trap/engines/engine.py.224: train: epoch 25 | batch 990/2304 [I 2022-10-26 19:14:51] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:14 [I 2022-10-26 19:14:51] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00244 [I 2022-10-26 19:14:51] ...trap/engines/engine.py.230: loss: 0.17415 [I 2022-10-26 19:14:51] ...optimizers/trijoint.py.151: total_norm: 16.582853 [I 2022-10-26 19:14:57] ...trap/engines/engine.py.224: train: epoch 25 | batch 1000/2304 [I 2022-10-26 19:14:57] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:08 [I 2022-10-26 19:14:57] ...trap/engines/engine.py.229: process: 0.56551 | load: 0.00244 [I 2022-10-26 19:14:57] ...trap/engines/engine.py.230: loss: 0.15077 [I 2022-10-26 19:14:57] ...optimizers/trijoint.py.151: total_norm: 23.015984 [I 2022-10-26 19:15:03] ...trap/engines/engine.py.224: train: epoch 25 | batch 1010/2304 [I 2022-10-26 19:15:03] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:03 [I 2022-10-26 19:15:03] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00245 [I 2022-10-26 19:15:03] ...trap/engines/engine.py.230: loss: 0.23716 [I 2022-10-26 19:15:03] ...optimizers/trijoint.py.151: total_norm: 46.222282 [I 2022-10-26 19:15:08] ...trap/engines/engine.py.224: train: epoch 25 | batch 1020/2304 [I 2022-10-26 19:15:08] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:12:01 [I 2022-10-26 19:15:08] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00246 [I 2022-10-26 19:15:08] ...trap/engines/engine.py.230: loss: 0.20135 [I 2022-10-26 19:15:08] ...optimizers/trijoint.py.151: total_norm: 15.769116 [I 2022-10-26 19:15:14] ...trap/engines/engine.py.224: train: epoch 25 | batch 1030/2304 [I 2022-10-26 19:15:14] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:44 [I 2022-10-26 19:15:14] ...trap/engines/engine.py.229: process: 0.55062 | load: 0.00245 [I 2022-10-26 19:15:14] ...trap/engines/engine.py.230: loss: 0.16117 [I 2022-10-26 19:15:14] ...optimizers/trijoint.py.151: total_norm: 30.156425 [I 2022-10-26 19:15:19] ...trap/engines/engine.py.224: train: epoch 25 | batch 1040/2304 [I 2022-10-26 19:15:19] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:12:09 [I 2022-10-26 19:15:19] ...trap/engines/engine.py.229: process: 0.59474 | load: 0.00244 [I 2022-10-26 19:15:19] ...trap/engines/engine.py.230: loss: 0.17502 [I 2022-10-26 19:15:19] ...optimizers/trijoint.py.151: total_norm: 11.765426 [I 2022-10-26 19:15:25] ...trap/engines/engine.py.224: train: epoch 25 | batch 1050/2304 [I 2022-10-26 19:15:25] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:43 [I 2022-10-26 19:15:25] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00245 [I 2022-10-26 19:15:25] ...trap/engines/engine.py.230: loss: 0.17357 [I 2022-10-26 19:15:25] ...optimizers/trijoint.py.151: total_norm: 23.889236 [I 2022-10-26 19:15:31] ...trap/engines/engine.py.224: train: epoch 25 | batch 1060/2304 [I 2022-10-26 19:15:31] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:52 [I 2022-10-26 19:15:31] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00244 [I 2022-10-26 19:15:31] ...trap/engines/engine.py.230: loss: 0.21436 [I 2022-10-26 19:15:31] ...optimizers/trijoint.py.151: total_norm: 28.478380 [I 2022-10-26 19:15:36] ...trap/engines/engine.py.224: train: epoch 25 | batch 1070/2304 [I 2022-10-26 19:15:36] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:30 [I 2022-10-26 19:15:36] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00244 [I 2022-10-26 19:15:36] ...trap/engines/engine.py.230: loss: 0.15023 [I 2022-10-26 19:15:36] ...optimizers/trijoint.py.151: total_norm: 13.843633 [I 2022-10-26 19:15:42] ...trap/engines/engine.py.224: train: epoch 25 | batch 1080/2304 [I 2022-10-26 19:15:42] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:22 [I 2022-10-26 19:15:42] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00242 [I 2022-10-26 19:15:42] ...trap/engines/engine.py.230: loss: 0.16723 [I 2022-10-26 19:15:42] ...optimizers/trijoint.py.151: total_norm: 17.385338 [I 2022-10-26 19:15:47] ...trap/engines/engine.py.224: train: epoch 25 | batch 1090/2304 [I 2022-10-26 19:15:47] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:10 [I 2022-10-26 19:15:47] ...trap/engines/engine.py.229: process: 0.55041 | load: 0.00516 [I 2022-10-26 19:15:47] ...trap/engines/engine.py.230: loss: 0.23615 [I 2022-10-26 19:15:47] ...optimizers/trijoint.py.151: total_norm: 26.204348 [I 2022-10-26 19:15:53] ...trap/engines/engine.py.224: train: epoch 25 | batch 1100/2304 [I 2022-10-26 19:15:53] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:11 [I 2022-10-26 19:15:53] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00251 [I 2022-10-26 19:15:53] ...trap/engines/engine.py.230: loss: 0.15131 [I 2022-10-26 19:15:53] ...optimizers/trijoint.py.151: total_norm: 17.773129 [I 2022-10-26 19:15:59] ...trap/engines/engine.py.224: train: epoch 25 | batch 1110/2304 [I 2022-10-26 19:15:59] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:05 [I 2022-10-26 19:15:59] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00245 [I 2022-10-26 19:15:59] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-26 19:15:59] ...optimizers/trijoint.py.151: total_norm: 9.645249 [I 2022-10-26 19:16:04] ...trap/engines/engine.py.224: train: epoch 25 | batch 1120/2304 [I 2022-10-26 19:16:04] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:10:58 [I 2022-10-26 19:16:04] ...trap/engines/engine.py.229: process: 0.55366 | load: 0.00248 [I 2022-10-26 19:16:04] ...trap/engines/engine.py.230: loss: 0.14897 [I 2022-10-26 19:16:04] ...optimizers/trijoint.py.151: total_norm: 19.119507 [I 2022-10-26 19:16:10] ...trap/engines/engine.py.224: train: epoch 25 | batch 1130/2304 [I 2022-10-26 19:16:10] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:11:09 [I 2022-10-26 19:16:10] ...trap/engines/engine.py.229: process: 0.56670 | load: 0.00244 [I 2022-10-26 19:16:10] ...trap/engines/engine.py.230: loss: 0.14808 [I 2022-10-26 19:16:10] ...optimizers/trijoint.py.151: total_norm: 34.514351 [I 2022-10-26 19:16:16] ...trap/engines/engine.py.224: train: epoch 25 | batch 1140/2304 [I 2022-10-26 19:16:16] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:45 [I 2022-10-26 19:16:16] ...trap/engines/engine.py.229: process: 0.55532 | load: 0.00245 [I 2022-10-26 19:16:16] ...trap/engines/engine.py.230: loss: 0.11926 [I 2022-10-26 19:16:16] ...optimizers/trijoint.py.151: total_norm: 23.588055 [I 2022-10-26 19:16:21] ...trap/engines/engine.py.224: train: epoch 25 | batch 1150/2304 [I 2022-10-26 19:16:21] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:45 [I 2022-10-26 19:16:21] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00250 [I 2022-10-26 19:16:21] ...trap/engines/engine.py.230: loss: 0.15820 [I 2022-10-26 19:16:21] ...optimizers/trijoint.py.151: total_norm: 13.638227 [I 2022-10-26 19:16:27] ...trap/engines/engine.py.224: train: epoch 25 | batch 1160/2304 [I 2022-10-26 19:16:27] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:38 [I 2022-10-26 19:16:27] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00244 [I 2022-10-26 19:16:27] ...trap/engines/engine.py.230: loss: 0.16975 [I 2022-10-26 19:16:27] ...optimizers/trijoint.py.151: total_norm: 15.972151 [I 2022-10-26 19:16:32] ...trap/engines/engine.py.224: train: epoch 25 | batch 1170/2304 [I 2022-10-26 19:16:32] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:27 [I 2022-10-26 19:16:32] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00245 [I 2022-10-26 19:16:32] ...trap/engines/engine.py.230: loss: 0.21164 [I 2022-10-26 19:16:32] ...optimizers/trijoint.py.151: total_norm: 19.755239 [I 2022-10-26 19:16:38] ...trap/engines/engine.py.224: train: epoch 25 | batch 1180/2304 [I 2022-10-26 19:16:38] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:25 [I 2022-10-26 19:16:38] ...trap/engines/engine.py.229: process: 0.54643 | load: 0.00240 [I 2022-10-26 19:16:38] ...trap/engines/engine.py.230: loss: 0.17039 [I 2022-10-26 19:16:38] ...optimizers/trijoint.py.151: total_norm: 10.748952 [I 2022-10-26 19:16:43] ...trap/engines/engine.py.224: train: epoch 25 | batch 1190/2304 [I 2022-10-26 19:16:43] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:14 [I 2022-10-26 19:16:43] ...trap/engines/engine.py.229: process: 0.54342 | load: 0.00243 [I 2022-10-26 19:16:43] ...trap/engines/engine.py.230: loss: 0.18078 [I 2022-10-26 19:16:43] ...optimizers/trijoint.py.151: total_norm: 34.000587 [I 2022-10-26 19:16:49] ...trap/engines/engine.py.224: train: epoch 25 | batch 1200/2304 [I 2022-10-26 19:16:49] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:22 [I 2022-10-26 19:16:49] ...trap/engines/engine.py.229: process: 0.56426 | load: 0.00253 [I 2022-10-26 19:16:49] ...trap/engines/engine.py.230: loss: 0.32458 [I 2022-10-26 19:16:49] ...optimizers/trijoint.py.151: total_norm: 16.634367 [I 2022-10-26 19:16:55] ...trap/engines/engine.py.224: train: epoch 25 | batch 1210/2304 [I 2022-10-26 19:16:55] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:09 [I 2022-10-26 19:16:55] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00250 [I 2022-10-26 19:16:55] ...trap/engines/engine.py.230: loss: 0.15090 [I 2022-10-26 19:16:55] ...optimizers/trijoint.py.151: total_norm: 16.048868 [I 2022-10-26 19:17:00] ...trap/engines/engine.py.224: train: epoch 25 | batch 1220/2304 [I 2022-10-26 19:17:00] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:07 [I 2022-10-26 19:17:00] ...trap/engines/engine.py.229: process: 0.57211 | load: 0.00250 [I 2022-10-26 19:17:00] ...trap/engines/engine.py.230: loss: 0.25433 [I 2022-10-26 19:17:00] ...optimizers/trijoint.py.151: total_norm: 16.572714 [I 2022-10-26 19:17:06] ...trap/engines/engine.py.224: train: epoch 25 | batch 1230/2304 [I 2022-10-26 19:17:06] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:58 [I 2022-10-26 19:17:06] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00245 [I 2022-10-26 19:17:06] ...trap/engines/engine.py.230: loss: 0.12389 [I 2022-10-26 19:17:06] ...optimizers/trijoint.py.151: total_norm: 15.602641 [I 2022-10-26 19:17:11] ...trap/engines/engine.py.224: train: epoch 25 | batch 1240/2304 [I 2022-10-26 19:17:11] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:50 [I 2022-10-26 19:17:11] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00247 [I 2022-10-26 19:17:11] ...trap/engines/engine.py.230: loss: 0.18917 [I 2022-10-26 19:17:11] ...optimizers/trijoint.py.151: total_norm: 24.742064 [I 2022-10-26 19:17:17] ...trap/engines/engine.py.224: train: epoch 25 | batch 1250/2304 [I 2022-10-26 19:17:17] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:45 [I 2022-10-26 19:17:17] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00267 [I 2022-10-26 19:17:17] ...trap/engines/engine.py.230: loss: 0.19896 [I 2022-10-26 19:17:17] ...optimizers/trijoint.py.151: total_norm: 21.418224 [I 2022-10-26 19:17:23] ...trap/engines/engine.py.224: train: epoch 25 | batch 1260/2304 [I 2022-10-26 19:17:23] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:38 [I 2022-10-26 19:17:23] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00251 [I 2022-10-26 19:17:23] ...trap/engines/engine.py.230: loss: 0.15717 [I 2022-10-26 19:17:23] ...optimizers/trijoint.py.151: total_norm: 24.084768 [I 2022-10-26 19:17:28] ...trap/engines/engine.py.224: train: epoch 25 | batch 1270/2304 [I 2022-10-26 19:17:28] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:33 [I 2022-10-26 19:17:28] ...trap/engines/engine.py.229: process: 0.54828 | load: 0.00246 [I 2022-10-26 19:17:28] ...trap/engines/engine.py.230: loss: 0.16848 [I 2022-10-26 19:17:28] ...optimizers/trijoint.py.151: total_norm: 27.039513 [I 2022-10-26 19:17:34] ...trap/engines/engine.py.224: train: epoch 25 | batch 1280/2304 [I 2022-10-26 19:17:34] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:33 [I 2022-10-26 19:17:34] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00257 [I 2022-10-26 19:17:34] ...trap/engines/engine.py.230: loss: 0.27322 [I 2022-10-26 19:17:34] ...optimizers/trijoint.py.151: total_norm: 16.633268 [I 2022-10-26 19:17:39] ...trap/engines/engine.py.224: train: epoch 25 | batch 1290/2304 [I 2022-10-26 19:17:39] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:26 [I 2022-10-26 19:17:39] ...trap/engines/engine.py.229: process: 0.56351 | load: 0.00249 [I 2022-10-26 19:17:39] ...trap/engines/engine.py.230: loss: 0.16253 [I 2022-10-26 19:17:39] ...optimizers/trijoint.py.151: total_norm: 11.474718 [I 2022-10-26 19:17:45] ...trap/engines/engine.py.224: train: epoch 25 | batch 1300/2304 [I 2022-10-26 19:17:45] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:18 [I 2022-10-26 19:17:45] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00250 [I 2022-10-26 19:17:45] ...trap/engines/engine.py.230: loss: 0.25703 [I 2022-10-26 19:17:45] ...optimizers/trijoint.py.151: total_norm: 15.695759 [I 2022-10-26 19:17:51] ...trap/engines/engine.py.224: train: epoch 25 | batch 1310/2304 [I 2022-10-26 19:17:51] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:25 [I 2022-10-26 19:17:51] ...trap/engines/engine.py.229: process: 0.56173 | load: 0.00244 [I 2022-10-26 19:17:51] ...trap/engines/engine.py.230: loss: 0.14981 [I 2022-10-26 19:17:51] ...optimizers/trijoint.py.151: total_norm: 14.037787 [I 2022-10-26 19:17:56] ...trap/engines/engine.py.224: train: epoch 25 | batch 1320/2304 [I 2022-10-26 19:17:56] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:08 [I 2022-10-26 19:17:56] ...trap/engines/engine.py.229: process: 0.54841 | load: 0.00252 [I 2022-10-26 19:17:56] ...trap/engines/engine.py.230: loss: 0.18616 [I 2022-10-26 19:17:56] ...optimizers/trijoint.py.151: total_norm: 29.043253 [I 2022-10-26 19:18:02] ...trap/engines/engine.py.224: train: epoch 25 | batch 1330/2304 [I 2022-10-26 19:18:02] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:04 [I 2022-10-26 19:18:02] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00254 [I 2022-10-26 19:18:02] ...trap/engines/engine.py.230: loss: 0.20540 [I 2022-10-26 19:18:02] ...optimizers/trijoint.py.151: total_norm: 27.603626 [I 2022-10-26 19:18:07] ...trap/engines/engine.py.224: train: epoch 25 | batch 1340/2304 [I 2022-10-26 19:18:07] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:55 [I 2022-10-26 19:18:07] ...trap/engines/engine.py.229: process: 0.54791 | load: 0.00250 [I 2022-10-26 19:18:07] ...trap/engines/engine.py.230: loss: 0.17721 [I 2022-10-26 19:18:07] ...optimizers/trijoint.py.151: total_norm: 17.797571 [I 2022-10-26 19:18:13] ...trap/engines/engine.py.224: train: epoch 25 | batch 1350/2304 [I 2022-10-26 19:18:13] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:49 [I 2022-10-26 19:18:13] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00249 [I 2022-10-26 19:18:13] ...trap/engines/engine.py.230: loss: 0.11166 [I 2022-10-26 19:18:13] ...optimizers/trijoint.py.151: total_norm: 15.993858 [I 2022-10-26 19:18:18] ...trap/engines/engine.py.224: train: epoch 25 | batch 1360/2304 [I 2022-10-26 19:18:18] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:48 [I 2022-10-26 19:18:18] ...trap/engines/engine.py.229: process: 0.56495 | load: 0.00241 [I 2022-10-26 19:18:18] ...trap/engines/engine.py.230: loss: 0.20533 [I 2022-10-26 19:18:18] ...optimizers/trijoint.py.151: total_norm: 15.235376 [I 2022-10-26 19:18:24] ...trap/engines/engine.py.224: train: epoch 25 | batch 1370/2304 [I 2022-10-26 19:18:24] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:40 [I 2022-10-26 19:18:24] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00269 [I 2022-10-26 19:18:24] ...trap/engines/engine.py.230: loss: 0.26032 [I 2022-10-26 19:18:24] ...optimizers/trijoint.py.151: total_norm: 22.784143 [I 2022-10-26 19:18:30] ...trap/engines/engine.py.224: train: epoch 25 | batch 1380/2304 [I 2022-10-26 19:18:30] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:37 [I 2022-10-26 19:18:30] ...trap/engines/engine.py.229: process: 0.56157 | load: 0.00260 [I 2022-10-26 19:18:30] ...trap/engines/engine.py.230: loss: 0.17156 [I 2022-10-26 19:18:30] ...optimizers/trijoint.py.151: total_norm: 37.220257 [I 2022-10-26 19:18:35] ...trap/engines/engine.py.224: train: epoch 25 | batch 1390/2304 [I 2022-10-26 19:18:35] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:26 [I 2022-10-26 19:18:35] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00245 [I 2022-10-26 19:18:35] ...trap/engines/engine.py.230: loss: 0.11619 [I 2022-10-26 19:18:35] ...optimizers/trijoint.py.151: total_norm: 20.016840 [I 2022-10-26 19:18:41] ...trap/engines/engine.py.224: train: epoch 25 | batch 1400/2304 [I 2022-10-26 19:18:41] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:21 [I 2022-10-26 19:18:41] ...trap/engines/engine.py.229: process: 0.54343 | load: 0.00244 [I 2022-10-26 19:18:41] ...trap/engines/engine.py.230: loss: 0.12255 [I 2022-10-26 19:18:41] ...optimizers/trijoint.py.151: total_norm: 11.579435 [I 2022-10-26 19:18:46] ...trap/engines/engine.py.224: train: epoch 25 | batch 1410/2304 [I 2022-10-26 19:18:46] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:21 [I 2022-10-26 19:18:46] ...trap/engines/engine.py.229: process: 0.55144 | load: 0.00268 [I 2022-10-26 19:18:46] ...trap/engines/engine.py.230: loss: 0.16138 [I 2022-10-26 19:18:46] ...optimizers/trijoint.py.151: total_norm: 12.934302 [I 2022-10-26 19:18:52] ...trap/engines/engine.py.224: train: epoch 25 | batch 1420/2304 [I 2022-10-26 19:18:52] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:16 [I 2022-10-26 19:18:52] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00275 [I 2022-10-26 19:18:52] ...trap/engines/engine.py.230: loss: 0.16964 [I 2022-10-26 19:18:52] ...optimizers/trijoint.py.151: total_norm: 19.334078 [I 2022-10-26 19:18:58] ...trap/engines/engine.py.224: train: epoch 25 | batch 1430/2304 [I 2022-10-26 19:18:58] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:08 [I 2022-10-26 19:18:58] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00257 [I 2022-10-26 19:18:58] ...trap/engines/engine.py.230: loss: 0.19278 [I 2022-10-26 19:18:58] ...optimizers/trijoint.py.151: total_norm: 27.106092 [I 2022-10-26 19:19:03] ...trap/engines/engine.py.224: train: epoch 25 | batch 1440/2304 [I 2022-10-26 19:19:03] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:01 [I 2022-10-26 19:19:03] ...trap/engines/engine.py.229: process: 0.56312 | load: 0.00256 [I 2022-10-26 19:19:03] ...trap/engines/engine.py.230: loss: 0.19195 [I 2022-10-26 19:19:03] ...optimizers/trijoint.py.151: total_norm: 22.501778 [I 2022-10-26 19:19:09] ...trap/engines/engine.py.224: train: epoch 25 | batch 1450/2304 [I 2022-10-26 19:19:09] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:57 [I 2022-10-26 19:19:09] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00297 [I 2022-10-26 19:19:09] ...trap/engines/engine.py.230: loss: 0.13922 [I 2022-10-26 19:19:09] ...optimizers/trijoint.py.151: total_norm: 22.803301 [I 2022-10-26 19:19:14] ...trap/engines/engine.py.224: train: epoch 25 | batch 1460/2304 [I 2022-10-26 19:19:14] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:48 [I 2022-10-26 19:19:14] ...trap/engines/engine.py.229: process: 0.55539 | load: 0.00273 [I 2022-10-26 19:19:14] ...trap/engines/engine.py.230: loss: 0.14091 [I 2022-10-26 19:19:14] ...optimizers/trijoint.py.151: total_norm: 16.098909 [I 2022-10-26 19:19:20] ...trap/engines/engine.py.224: train: epoch 25 | batch 1470/2304 [I 2022-10-26 19:19:20] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:46 [I 2022-10-26 19:19:20] ...trap/engines/engine.py.229: process: 0.55217 | load: 0.00265 [I 2022-10-26 19:19:20] ...trap/engines/engine.py.230: loss: 0.26698 [I 2022-10-26 19:19:20] ...optimizers/trijoint.py.151: total_norm: 13.617640 [I 2022-10-26 19:19:26] ...trap/engines/engine.py.224: train: epoch 25 | batch 1480/2304 [I 2022-10-26 19:19:26] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:40 [I 2022-10-26 19:19:26] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00261 [I 2022-10-26 19:19:26] ...trap/engines/engine.py.230: loss: 0.15323 [I 2022-10-26 19:19:26] ...optimizers/trijoint.py.151: total_norm: 21.726463 [I 2022-10-26 19:19:31] ...trap/engines/engine.py.224: train: epoch 25 | batch 1490/2304 [I 2022-10-26 19:19:31] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:30 [I 2022-10-26 19:19:31] ...trap/engines/engine.py.229: process: 0.55319 | load: 0.00253 [I 2022-10-26 19:19:31] ...trap/engines/engine.py.230: loss: 0.26790 [I 2022-10-26 19:19:31] ...optimizers/trijoint.py.151: total_norm: 25.878242 [I 2022-10-26 19:19:37] ...trap/engines/engine.py.224: train: epoch 25 | batch 1500/2304 [I 2022-10-26 19:19:37] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:24 [I 2022-10-26 19:19:37] ...trap/engines/engine.py.229: process: 0.54610 | load: 0.00255 [I 2022-10-26 19:19:37] ...trap/engines/engine.py.230: loss: 0.13754 [I 2022-10-26 19:19:37] ...optimizers/trijoint.py.151: total_norm: 14.908392 [I 2022-10-26 19:19:42] ...trap/engines/engine.py.224: train: epoch 25 | batch 1510/2304 [I 2022-10-26 19:19:42] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:21 [I 2022-10-26 19:19:42] ...trap/engines/engine.py.229: process: 0.54029 | load: 0.00271 [I 2022-10-26 19:19:42] ...trap/engines/engine.py.230: loss: 0.24598 [I 2022-10-26 19:19:42] ...optimizers/trijoint.py.151: total_norm: 18.640516 [I 2022-10-26 19:19:48] ...trap/engines/engine.py.224: train: epoch 25 | batch 1520/2304 [I 2022-10-26 19:19:48] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:16 [I 2022-10-26 19:19:48] ...trap/engines/engine.py.229: process: 0.56635 | load: 0.00257 [I 2022-10-26 19:19:48] ...trap/engines/engine.py.230: loss: 0.15879 [I 2022-10-26 19:19:48] ...optimizers/trijoint.py.151: total_norm: 14.197707 [I 2022-10-26 19:19:54] ...trap/engines/engine.py.224: train: epoch 25 | batch 1530/2304 [I 2022-10-26 19:19:54] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:24 [I 2022-10-26 19:19:54] ...trap/engines/engine.py.229: process: 0.55667 | load: 0.00246 [I 2022-10-26 19:19:54] ...trap/engines/engine.py.230: loss: 0.22574 [I 2022-10-26 19:19:54] ...optimizers/trijoint.py.151: total_norm: 16.504950 [I 2022-10-26 19:19:59] ...trap/engines/engine.py.224: train: epoch 25 | batch 1540/2304 [I 2022-10-26 19:19:59] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:06 [I 2022-10-26 19:19:59] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00252 [I 2022-10-26 19:19:59] ...trap/engines/engine.py.230: loss: 0.20666 [I 2022-10-26 19:19:59] ...optimizers/trijoint.py.151: total_norm: 25.236231 [I 2022-10-26 19:20:05] ...trap/engines/engine.py.224: train: epoch 25 | batch 1550/2304 [I 2022-10-26 19:20:05] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:00 [I 2022-10-26 19:20:05] ...trap/engines/engine.py.229: process: 0.55301 | load: 0.00252 [I 2022-10-26 19:20:05] ...trap/engines/engine.py.230: loss: 0.31801 [I 2022-10-26 19:20:05] ...optimizers/trijoint.py.151: total_norm: 23.481030 [I 2022-10-26 19:20:10] ...trap/engines/engine.py.224: train: epoch 25 | batch 1560/2304 [I 2022-10-26 19:20:10] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:06:54 [I 2022-10-26 19:20:10] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00242 [I 2022-10-26 19:20:10] ...trap/engines/engine.py.230: loss: 0.12741 [I 2022-10-26 19:20:10] ...optimizers/trijoint.py.151: total_norm: 20.401011 [I 2022-10-26 19:20:16] ...trap/engines/engine.py.224: train: epoch 25 | batch 1570/2304 [I 2022-10-26 19:20:16] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:47 [I 2022-10-26 19:20:16] ...trap/engines/engine.py.229: process: 0.54725 | load: 0.00247 [I 2022-10-26 19:20:16] ...trap/engines/engine.py.230: loss: 0.12779 [I 2022-10-26 19:20:16] ...optimizers/trijoint.py.151: total_norm: 14.177024 [I 2022-10-26 19:20:22] ...trap/engines/engine.py.224: train: epoch 25 | batch 1580/2304 [I 2022-10-26 19:20:22] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:42 [I 2022-10-26 19:20:22] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00273 [I 2022-10-26 19:20:22] ...trap/engines/engine.py.230: loss: 0.16288 [I 2022-10-26 19:20:22] ...optimizers/trijoint.py.151: total_norm: 18.669416 [I 2022-10-26 19:20:27] ...trap/engines/engine.py.224: train: epoch 25 | batch 1590/2304 [I 2022-10-26 19:20:27] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:44 [I 2022-10-26 19:20:27] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00266 [I 2022-10-26 19:20:27] ...trap/engines/engine.py.230: loss: 0.16325 [I 2022-10-26 19:20:27] ...optimizers/trijoint.py.151: total_norm: 15.725684 [I 2022-10-26 19:20:33] ...trap/engines/engine.py.224: train: epoch 25 | batch 1600/2304 [I 2022-10-26 19:20:33] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:34 [I 2022-10-26 19:20:33] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00249 [I 2022-10-26 19:20:33] ...trap/engines/engine.py.230: loss: 0.10699 [I 2022-10-26 19:20:33] ...optimizers/trijoint.py.151: total_norm: 11.596040 [I 2022-10-26 19:20:38] ...trap/engines/engine.py.224: train: epoch 25 | batch 1610/2304 [I 2022-10-26 19:20:38] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:26 [I 2022-10-26 19:20:38] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00246 [I 2022-10-26 19:20:38] ...trap/engines/engine.py.230: loss: 0.15382 [I 2022-10-26 19:20:38] ...optimizers/trijoint.py.151: total_norm: 31.813797 [I 2022-10-26 19:20:44] ...trap/engines/engine.py.224: train: epoch 25 | batch 1620/2304 [I 2022-10-26 19:20:44] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:23 [I 2022-10-26 19:20:44] ...trap/engines/engine.py.229: process: 0.57097 | load: 0.00256 [I 2022-10-26 19:20:44] ...trap/engines/engine.py.230: loss: 0.20850 [I 2022-10-26 19:20:44] ...optimizers/trijoint.py.151: total_norm: 28.056980 [I 2022-10-26 19:20:50] ...trap/engines/engine.py.224: train: epoch 25 | batch 1630/2304 [I 2022-10-26 19:20:50] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:18 [I 2022-10-26 19:20:50] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00246 [I 2022-10-26 19:20:50] ...trap/engines/engine.py.230: loss: 0.16519 [I 2022-10-26 19:20:50] ...optimizers/trijoint.py.151: total_norm: 14.217437 [I 2022-10-26 19:20:55] ...trap/engines/engine.py.224: train: epoch 25 | batch 1640/2304 [I 2022-10-26 19:20:55] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:07 [I 2022-10-26 19:20:55] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00243 [I 2022-10-26 19:20:55] ...trap/engines/engine.py.230: loss: 0.17259 [I 2022-10-26 19:20:55] ...optimizers/trijoint.py.151: total_norm: 15.160622 [I 2022-10-26 19:21:01] ...trap/engines/engine.py.224: train: epoch 25 | batch 1650/2304 [I 2022-10-26 19:21:01] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:07 [I 2022-10-26 19:21:01] ...trap/engines/engine.py.229: process: 0.56704 | load: 0.00245 [I 2022-10-26 19:21:01] ...trap/engines/engine.py.230: loss: 0.18636 [I 2022-10-26 19:21:01] ...optimizers/trijoint.py.151: total_norm: 19.752438 [I 2022-10-26 19:21:07] ...trap/engines/engine.py.224: train: epoch 25 | batch 1660/2304 [I 2022-10-26 19:21:07] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:00 [I 2022-10-26 19:21:07] ...trap/engines/engine.py.229: process: 0.56314 | load: 0.00251 [I 2022-10-26 19:21:07] ...trap/engines/engine.py.230: loss: 0.20448 [I 2022-10-26 19:21:07] ...optimizers/trijoint.py.151: total_norm: 22.152760 [I 2022-10-26 19:21:12] ...trap/engines/engine.py.224: train: epoch 25 | batch 1670/2304 [I 2022-10-26 19:21:12] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:53 [I 2022-10-26 19:21:12] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00249 [I 2022-10-26 19:21:12] ...trap/engines/engine.py.230: loss: 0.18936 [I 2022-10-26 19:21:12] ...optimizers/trijoint.py.151: total_norm: 20.769566 [I 2022-10-26 19:21:18] ...trap/engines/engine.py.224: train: epoch 25 | batch 1680/2304 [I 2022-10-26 19:21:18] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:46 [I 2022-10-26 19:21:18] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00245 [I 2022-10-26 19:21:18] ...trap/engines/engine.py.230: loss: 0.20045 [I 2022-10-26 19:21:18] ...optimizers/trijoint.py.151: total_norm: 15.410327 [I 2022-10-26 19:21:23] ...trap/engines/engine.py.224: train: epoch 25 | batch 1690/2304 [I 2022-10-26 19:21:23] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:42 [I 2022-10-26 19:21:23] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00242 [I 2022-10-26 19:21:23] ...trap/engines/engine.py.230: loss: 0.19256 [I 2022-10-26 19:21:23] ...optimizers/trijoint.py.151: total_norm: 24.726303 [I 2022-10-26 19:21:29] ...trap/engines/engine.py.224: train: epoch 25 | batch 1700/2304 [I 2022-10-26 19:21:29] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:34 [I 2022-10-26 19:21:29] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00244 [I 2022-10-26 19:21:29] ...trap/engines/engine.py.230: loss: 0.19426 [I 2022-10-26 19:21:29] ...optimizers/trijoint.py.151: total_norm: 13.064757 [I 2022-10-26 19:21:34] ...trap/engines/engine.py.224: train: epoch 25 | batch 1710/2304 [I 2022-10-26 19:21:34] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:30 [I 2022-10-26 19:21:34] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00251 [I 2022-10-26 19:21:34] ...trap/engines/engine.py.230: loss: 0.21034 [I 2022-10-26 19:21:34] ...optimizers/trijoint.py.151: total_norm: 23.886641 [I 2022-10-26 19:21:40] ...trap/engines/engine.py.224: train: epoch 25 | batch 1720/2304 [I 2022-10-26 19:21:40] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:22 [I 2022-10-26 19:21:40] ...trap/engines/engine.py.229: process: 0.54376 | load: 0.00244 [I 2022-10-26 19:21:40] ...trap/engines/engine.py.230: loss: 0.31801 [I 2022-10-26 19:21:40] ...optimizers/trijoint.py.151: total_norm: 31.680857 [I 2022-10-26 19:21:45] ...trap/engines/engine.py.224: train: epoch 25 | batch 1730/2304 [I 2022-10-26 19:21:45] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:18 [I 2022-10-26 19:21:45] ...trap/engines/engine.py.229: process: 0.54834 | load: 0.00249 [I 2022-10-26 19:21:45] ...trap/engines/engine.py.230: loss: 0.24335 [I 2022-10-26 19:21:45] ...optimizers/trijoint.py.151: total_norm: 14.357522 [I 2022-10-26 19:21:51] ...trap/engines/engine.py.224: train: epoch 25 | batch 1740/2304 [I 2022-10-26 19:21:51] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:13 [I 2022-10-26 19:21:51] ...trap/engines/engine.py.229: process: 0.56554 | load: 0.00247 [I 2022-10-26 19:21:51] ...trap/engines/engine.py.230: loss: 0.13647 [I 2022-10-26 19:21:51] ...optimizers/trijoint.py.151: total_norm: 11.658106 [I 2022-10-26 19:21:57] ...trap/engines/engine.py.224: train: epoch 25 | batch 1750/2304 [I 2022-10-26 19:21:57] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:08 [I 2022-10-26 19:21:57] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00484 [I 2022-10-26 19:21:57] ...trap/engines/engine.py.230: loss: 0.15919 [I 2022-10-26 19:21:57] ...optimizers/trijoint.py.151: total_norm: 21.751217 [I 2022-10-26 19:22:02] ...trap/engines/engine.py.224: train: epoch 25 | batch 1760/2304 [I 2022-10-26 19:22:02] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:02 [I 2022-10-26 19:22:02] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00240 [I 2022-10-26 19:22:02] ...trap/engines/engine.py.230: loss: 0.15724 [I 2022-10-26 19:22:02] ...optimizers/trijoint.py.151: total_norm: 11.444945 [I 2022-10-26 19:22:08] ...trap/engines/engine.py.224: train: epoch 25 | batch 1770/2304 [I 2022-10-26 19:22:08] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:04:59 [I 2022-10-26 19:22:08] ...trap/engines/engine.py.229: process: 0.57527 | load: 0.00245 [I 2022-10-26 19:22:08] ...trap/engines/engine.py.230: loss: 0.22624 [I 2022-10-26 19:22:08] ...optimizers/trijoint.py.151: total_norm: 20.214594 [I 2022-10-26 19:22:13] ...trap/engines/engine.py.224: train: epoch 25 | batch 1780/2304 [I 2022-10-26 19:22:13] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:04:49 [I 2022-10-26 19:22:13] ...trap/engines/engine.py.229: process: 0.54122 | load: 0.00243 [I 2022-10-26 19:22:13] ...trap/engines/engine.py.230: loss: 0.16977 [I 2022-10-26 19:22:13] ...optimizers/trijoint.py.151: total_norm: 20.282259 [I 2022-10-26 19:22:19] ...trap/engines/engine.py.224: train: epoch 25 | batch 1790/2304 [I 2022-10-26 19:22:19] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:47 [I 2022-10-26 19:22:19] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00250 [I 2022-10-26 19:22:19] ...trap/engines/engine.py.230: loss: 0.13904 [I 2022-10-26 19:22:19] ...optimizers/trijoint.py.151: total_norm: 18.215687 [I 2022-10-26 19:22:25] ...trap/engines/engine.py.224: train: epoch 25 | batch 1800/2304 [I 2022-10-26 19:22:25] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:39 [I 2022-10-26 19:22:25] ...trap/engines/engine.py.229: process: 0.55321 | load: 0.00246 [I 2022-10-26 19:22:25] ...trap/engines/engine.py.230: loss: 0.13856 [I 2022-10-26 19:22:25] ...optimizers/trijoint.py.151: total_norm: 15.302763 [I 2022-10-26 19:22:30] ...trap/engines/engine.py.224: train: epoch 25 | batch 1810/2304 [I 2022-10-26 19:22:30] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:36 [I 2022-10-26 19:22:30] ...trap/engines/engine.py.229: process: 0.56725 | load: 0.00244 [I 2022-10-26 19:22:30] ...trap/engines/engine.py.230: loss: 0.17160 [I 2022-10-26 19:22:30] ...optimizers/trijoint.py.151: total_norm: 19.303762 [I 2022-10-26 19:22:36] ...trap/engines/engine.py.224: train: epoch 25 | batch 1820/2304 [I 2022-10-26 19:22:36] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:30 [I 2022-10-26 19:22:36] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00244 [I 2022-10-26 19:22:36] ...trap/engines/engine.py.230: loss: 0.26925 [I 2022-10-26 19:22:36] ...optimizers/trijoint.py.151: total_norm: 19.376690 [I 2022-10-26 19:22:41] ...trap/engines/engine.py.224: train: epoch 25 | batch 1830/2304 [I 2022-10-26 19:22:41] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:21 [I 2022-10-26 19:22:41] ...trap/engines/engine.py.229: process: 0.53849 | load: 0.00246 [I 2022-10-26 19:22:41] ...trap/engines/engine.py.230: loss: 0.14653 [I 2022-10-26 19:22:41] ...optimizers/trijoint.py.151: total_norm: 14.278845 [I 2022-10-26 19:22:47] ...trap/engines/engine.py.224: train: epoch 25 | batch 1840/2304 [I 2022-10-26 19:22:47] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:16 [I 2022-10-26 19:22:47] ...trap/engines/engine.py.229: process: 0.54844 | load: 0.00247 [I 2022-10-26 19:22:47] ...trap/engines/engine.py.230: loss: 0.12842 [I 2022-10-26 19:22:47] ...optimizers/trijoint.py.151: total_norm: 14.630891 [I 2022-10-26 19:22:52] ...trap/engines/engine.py.224: train: epoch 25 | batch 1850/2304 [I 2022-10-26 19:22:52] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:12 [I 2022-10-26 19:22:52] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00249 [I 2022-10-26 19:22:52] ...trap/engines/engine.py.230: loss: 0.19118 [I 2022-10-26 19:22:52] ...optimizers/trijoint.py.151: total_norm: 18.269766 [I 2022-10-26 19:22:58] ...trap/engines/engine.py.224: train: epoch 25 | batch 1860/2304 [I 2022-10-26 19:22:58] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:07 [I 2022-10-26 19:22:58] ...trap/engines/engine.py.229: process: 0.56475 | load: 0.00247 [I 2022-10-26 19:22:58] ...trap/engines/engine.py.230: loss: 0.25289 [I 2022-10-26 19:22:58] ...optimizers/trijoint.py.151: total_norm: 25.456842 [I 2022-10-26 19:23:04] ...trap/engines/engine.py.224: train: epoch 25 | batch 1870/2304 [I 2022-10-26 19:23:04] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:02 [I 2022-10-26 19:23:04] ...trap/engines/engine.py.229: process: 0.56501 | load: 0.00246 [I 2022-10-26 19:23:04] ...trap/engines/engine.py.230: loss: 0.16498 [I 2022-10-26 19:23:04] ...optimizers/trijoint.py.151: total_norm: 30.166727 [I 2022-10-26 19:23:09] ...trap/engines/engine.py.224: train: epoch 25 | batch 1880/2304 [I 2022-10-26 19:23:09] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:03:56 [I 2022-10-26 19:23:09] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00244 [I 2022-10-26 19:23:09] ...trap/engines/engine.py.230: loss: 0.21619 [I 2022-10-26 19:23:09] ...optimizers/trijoint.py.151: total_norm: 43.981312 [I 2022-10-26 19:23:15] ...trap/engines/engine.py.224: train: epoch 25 | batch 1890/2304 [I 2022-10-26 19:23:15] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:03:49 [I 2022-10-26 19:23:15] ...trap/engines/engine.py.229: process: 0.54855 | load: 0.00244 [I 2022-10-26 19:23:15] ...trap/engines/engine.py.230: loss: 0.17614 [I 2022-10-26 19:23:15] ...optimizers/trijoint.py.151: total_norm: 21.055710 [I 2022-10-26 19:23:20] ...trap/engines/engine.py.224: train: epoch 25 | batch 1900/2304 [I 2022-10-26 19:23:20] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:03:45 [I 2022-10-26 19:23:20] ...trap/engines/engine.py.229: process: 0.56610 | load: 0.00246 [I 2022-10-26 19:23:20] ...trap/engines/engine.py.230: loss: 0.21846 [I 2022-10-26 19:23:20] ...optimizers/trijoint.py.151: total_norm: 24.528147 [I 2022-10-26 19:23:26] ...trap/engines/engine.py.224: train: epoch 25 | batch 1910/2304 [I 2022-10-26 19:23:26] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:38 [I 2022-10-26 19:23:26] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00250 [I 2022-10-26 19:23:26] ...trap/engines/engine.py.230: loss: 0.21642 [I 2022-10-26 19:23:26] ...optimizers/trijoint.py.151: total_norm: 16.172735 [I 2022-10-26 19:23:31] ...trap/engines/engine.py.224: train: epoch 25 | batch 1920/2304 [I 2022-10-26 19:23:31] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:33 [I 2022-10-26 19:23:31] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00248 [I 2022-10-26 19:23:31] ...trap/engines/engine.py.230: loss: 0.20608 [I 2022-10-26 19:23:31] ...optimizers/trijoint.py.151: total_norm: 19.100283 [I 2022-10-26 19:23:37] ...trap/engines/engine.py.224: train: epoch 25 | batch 1930/2304 [I 2022-10-26 19:23:37] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:28 [I 2022-10-26 19:23:37] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00247 [I 2022-10-26 19:23:37] ...trap/engines/engine.py.230: loss: 0.14961 [I 2022-10-26 19:23:37] ...optimizers/trijoint.py.151: total_norm: 15.518927 [I 2022-10-26 19:23:43] ...trap/engines/engine.py.224: train: epoch 25 | batch 1940/2304 [I 2022-10-26 19:23:43] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:23 [I 2022-10-26 19:23:43] ...trap/engines/engine.py.229: process: 0.56432 | load: 0.00242 [I 2022-10-26 19:23:43] ...trap/engines/engine.py.230: loss: 0.10586 [I 2022-10-26 19:23:43] ...optimizers/trijoint.py.151: total_norm: 12.932769 [I 2022-10-26 19:23:48] ...trap/engines/engine.py.224: train: epoch 25 | batch 1950/2304 [I 2022-10-26 19:23:48] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:16 [I 2022-10-26 19:23:48] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00248 [I 2022-10-26 19:23:48] ...trap/engines/engine.py.230: loss: 0.18659 [I 2022-10-26 19:23:48] ...optimizers/trijoint.py.151: total_norm: 28.046200 [I 2022-10-26 19:23:54] ...trap/engines/engine.py.224: train: epoch 25 | batch 1960/2304 [I 2022-10-26 19:23:54] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:11 [I 2022-10-26 19:23:54] ...trap/engines/engine.py.229: process: 0.57108 | load: 0.00248 [I 2022-10-26 19:23:54] ...trap/engines/engine.py.230: loss: 0.32049 [I 2022-10-26 19:23:54] ...optimizers/trijoint.py.151: total_norm: 16.263172 [I 2022-10-26 19:23:59] ...trap/engines/engine.py.224: train: epoch 25 | batch 1970/2304 [I 2022-10-26 19:23:59] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:05 [I 2022-10-26 19:23:59] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00249 [I 2022-10-26 19:23:59] ...trap/engines/engine.py.230: loss: 0.17990 [I 2022-10-26 19:23:59] ...optimizers/trijoint.py.151: total_norm: 23.254934 [I 2022-10-26 19:24:05] ...trap/engines/engine.py.224: train: epoch 25 | batch 1980/2304 [I 2022-10-26 19:24:05] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:01 [I 2022-10-26 19:24:05] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00250 [I 2022-10-26 19:24:05] ...trap/engines/engine.py.230: loss: 0.17808 [I 2022-10-26 19:24:05] ...optimizers/trijoint.py.151: total_norm: 16.917313 [I 2022-10-26 19:24:10] ...trap/engines/engine.py.224: train: epoch 25 | batch 1990/2304 [I 2022-10-26 19:24:10] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:55 [I 2022-10-26 19:24:10] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00251 [I 2022-10-26 19:24:10] ...trap/engines/engine.py.230: loss: 0.18960 [I 2022-10-26 19:24:10] ...optimizers/trijoint.py.151: total_norm: 13.521099 [I 2022-10-26 19:24:16] ...trap/engines/engine.py.224: train: epoch 25 | batch 2000/2304 [I 2022-10-26 19:24:16] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:02:50 [I 2022-10-26 19:24:16] ...trap/engines/engine.py.229: process: 0.56168 | load: 0.00249 [I 2022-10-26 19:24:16] ...trap/engines/engine.py.230: loss: 0.19153 [I 2022-10-26 19:24:16] ...optimizers/trijoint.py.151: total_norm: 26.127594 [I 2022-10-26 19:24:22] ...trap/engines/engine.py.224: train: epoch 25 | batch 2010/2304 [I 2022-10-26 19:24:22] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:44 [I 2022-10-26 19:24:22] ...trap/engines/engine.py.229: process: 0.56464 | load: 0.00247 [I 2022-10-26 19:24:22] ...trap/engines/engine.py.230: loss: 0.13326 [I 2022-10-26 19:24:22] ...optimizers/trijoint.py.151: total_norm: 17.420261 [I 2022-10-26 19:24:27] ...trap/engines/engine.py.224: train: epoch 25 | batch 2020/2304 [I 2022-10-26 19:24:27] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:39 [I 2022-10-26 19:24:27] ...trap/engines/engine.py.229: process: 0.57179 | load: 0.00247 [I 2022-10-26 19:24:27] ...trap/engines/engine.py.230: loss: 0.16062 [I 2022-10-26 19:24:27] ...optimizers/trijoint.py.151: total_norm: 23.553778 [I 2022-10-26 19:24:33] ...trap/engines/engine.py.224: train: epoch 25 | batch 2030/2304 [I 2022-10-26 19:24:33] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:32 [I 2022-10-26 19:24:33] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00248 [I 2022-10-26 19:24:33] ...trap/engines/engine.py.230: loss: 0.16643 [I 2022-10-26 19:24:33] ...optimizers/trijoint.py.151: total_norm: 17.068127 [I 2022-10-26 19:24:38] ...trap/engines/engine.py.224: train: epoch 25 | batch 2040/2304 [I 2022-10-26 19:24:38] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:26 [I 2022-10-26 19:24:38] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00252 [I 2022-10-26 19:24:38] ...trap/engines/engine.py.230: loss: 0.16386 [I 2022-10-26 19:24:38] ...optimizers/trijoint.py.151: total_norm: 17.632381 [I 2022-10-26 19:24:44] ...trap/engines/engine.py.224: train: epoch 25 | batch 2050/2304 [I 2022-10-26 19:24:44] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:20 [I 2022-10-26 19:24:44] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00253 [I 2022-10-26 19:24:44] ...trap/engines/engine.py.230: loss: 0.22757 [I 2022-10-26 19:24:44] ...optimizers/trijoint.py.151: total_norm: 28.961472 [I 2022-10-26 19:24:50] ...trap/engines/engine.py.224: train: epoch 25 | batch 2060/2304 [I 2022-10-26 19:24:50] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:14 [I 2022-10-26 19:24:50] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00244 [I 2022-10-26 19:24:50] ...trap/engines/engine.py.230: loss: 0.19106 [I 2022-10-26 19:24:50] ...optimizers/trijoint.py.151: total_norm: 17.115021 [I 2022-10-26 19:24:55] ...trap/engines/engine.py.224: train: epoch 25 | batch 2070/2304 [I 2022-10-26 19:24:55] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:10 [I 2022-10-26 19:24:55] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00245 [I 2022-10-26 19:24:55] ...trap/engines/engine.py.230: loss: 0.13031 [I 2022-10-26 19:24:55] ...optimizers/trijoint.py.151: total_norm: 15.336044 [I 2022-10-26 19:25:01] ...trap/engines/engine.py.224: train: epoch 25 | batch 2080/2304 [I 2022-10-26 19:25:01] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:04 [I 2022-10-26 19:25:01] ...trap/engines/engine.py.229: process: 0.56298 | load: 0.00244 [I 2022-10-26 19:25:01] ...trap/engines/engine.py.230: loss: 0.15700 [I 2022-10-26 19:25:01] ...optimizers/trijoint.py.151: total_norm: 14.983087 [I 2022-10-26 19:25:06] ...trap/engines/engine.py.224: train: epoch 25 | batch 2090/2304 [I 2022-10-26 19:25:06] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:01:58 [I 2022-10-26 19:25:06] ...trap/engines/engine.py.229: process: 0.56358 | load: 0.00243 [I 2022-10-26 19:25:06] ...trap/engines/engine.py.230: loss: 0.18752 [I 2022-10-26 19:25:06] ...optimizers/trijoint.py.151: total_norm: 15.311178 [I 2022-10-26 19:25:12] ...trap/engines/engine.py.224: train: epoch 25 | batch 2100/2304 [I 2022-10-26 19:25:12] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:01:53 [I 2022-10-26 19:25:12] ...trap/engines/engine.py.229: process: 0.56309 | load: 0.00251 [I 2022-10-26 19:25:12] ...trap/engines/engine.py.230: loss: 0.18957 [I 2022-10-26 19:25:12] ...optimizers/trijoint.py.151: total_norm: 20.747557 [I 2022-10-26 19:25:17] ...trap/engines/engine.py.224: train: epoch 25 | batch 2110/2304 [I 2022-10-26 19:25:17] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:01:47 [I 2022-10-26 19:25:17] ...trap/engines/engine.py.229: process: 0.54669 | load: 0.00246 [I 2022-10-26 19:25:17] ...trap/engines/engine.py.230: loss: 0.17652 [I 2022-10-26 19:25:17] ...optimizers/trijoint.py.151: total_norm: 19.518133 [I 2022-10-26 19:25:23] ...trap/engines/engine.py.224: train: epoch 25 | batch 2120/2304 [I 2022-10-26 19:25:23] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:01:46 [I 2022-10-26 19:25:23] ...trap/engines/engine.py.229: process: 0.59899 | load: 0.00255 [I 2022-10-26 19:25:23] ...trap/engines/engine.py.230: loss: 0.20159 [I 2022-10-26 19:25:23] ...optimizers/trijoint.py.151: total_norm: 26.865534 [I 2022-10-26 19:25:29] ...trap/engines/engine.py.224: train: epoch 25 | batch 2130/2304 [I 2022-10-26 19:25:29] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:37 [I 2022-10-26 19:25:29] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00244 [I 2022-10-26 19:25:29] ...trap/engines/engine.py.230: loss: 0.15791 [I 2022-10-26 19:25:29] ...optimizers/trijoint.py.151: total_norm: 18.327894 [I 2022-10-26 19:25:35] ...trap/engines/engine.py.224: train: epoch 25 | batch 2140/2304 [I 2022-10-26 19:25:35] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:32 [I 2022-10-26 19:25:35] ...trap/engines/engine.py.229: process: 0.54997 | load: 0.00244 [I 2022-10-26 19:25:35] ...trap/engines/engine.py.230: loss: 0.14159 [I 2022-10-26 19:25:35] ...optimizers/trijoint.py.151: total_norm: 16.012745 [I 2022-10-26 19:25:40] ...trap/engines/engine.py.224: train: epoch 25 | batch 2150/2304 [I 2022-10-26 19:25:40] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:25 [I 2022-10-26 19:25:40] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00245 [I 2022-10-26 19:25:40] ...trap/engines/engine.py.230: loss: 0.18884 [I 2022-10-26 19:25:40] ...optimizers/trijoint.py.151: total_norm: 17.911089 [I 2022-10-26 19:25:46] ...trap/engines/engine.py.224: train: epoch 25 | batch 2160/2304 [I 2022-10-26 19:25:46] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:20 [I 2022-10-26 19:25:46] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00255 [I 2022-10-26 19:25:46] ...trap/engines/engine.py.230: loss: 0.23599 [I 2022-10-26 19:25:46] ...optimizers/trijoint.py.151: total_norm: 18.440895 [I 2022-10-26 19:25:51] ...trap/engines/engine.py.224: train: epoch 25 | batch 2170/2304 [I 2022-10-26 19:25:51] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:14 [I 2022-10-26 19:25:51] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00248 [I 2022-10-26 19:25:51] ...trap/engines/engine.py.230: loss: 0.23212 [I 2022-10-26 19:25:51] ...optimizers/trijoint.py.151: total_norm: 20.735632 [I 2022-10-26 19:25:57] ...trap/engines/engine.py.224: train: epoch 25 | batch 2180/2304 [I 2022-10-26 19:25:57] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:08 [I 2022-10-26 19:25:57] ...trap/engines/engine.py.229: process: 0.54704 | load: 0.00247 [I 2022-10-26 19:25:57] ...trap/engines/engine.py.230: loss: 0.12305 [I 2022-10-26 19:25:57] ...optimizers/trijoint.py.151: total_norm: 13.385143 [I 2022-10-26 19:26:02] ...trap/engines/engine.py.224: train: epoch 25 | batch 2190/2304 [I 2022-10-26 19:26:02] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:03 [I 2022-10-26 19:26:02] ...trap/engines/engine.py.229: process: 0.54646 | load: 0.00242 [I 2022-10-26 19:26:02] ...trap/engines/engine.py.230: loss: 0.12286 [I 2022-10-26 19:26:02] ...optimizers/trijoint.py.151: total_norm: 13.266269 [I 2022-10-26 19:26:08] ...trap/engines/engine.py.224: train: epoch 25 | batch 2200/2304 [I 2022-10-26 19:26:08] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:00:57 [I 2022-10-26 19:26:08] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00267 [I 2022-10-26 19:26:08] ...trap/engines/engine.py.230: loss: 0.22999 [I 2022-10-26 19:26:08] ...optimizers/trijoint.py.151: total_norm: 24.877350 [I 2022-10-26 19:26:14] ...trap/engines/engine.py.224: train: epoch 25 | batch 2210/2304 [I 2022-10-26 19:26:14] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:00:52 [I 2022-10-26 19:26:14] ...trap/engines/engine.py.229: process: 0.55362 | load: 0.00244 [I 2022-10-26 19:26:14] ...trap/engines/engine.py.230: loss: 0.33143 [I 2022-10-26 19:26:14] ...optimizers/trijoint.py.151: total_norm: 27.813812 [I 2022-10-26 19:26:19] ...trap/engines/engine.py.224: train: epoch 25 | batch 2220/2304 [I 2022-10-26 19:26:19] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:00:47 [I 2022-10-26 19:26:19] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00241 [I 2022-10-26 19:26:19] ...trap/engines/engine.py.230: loss: 0.22613 [I 2022-10-26 19:26:19] ...optimizers/trijoint.py.151: total_norm: 19.472109 [I 2022-10-26 19:26:25] ...trap/engines/engine.py.224: train: epoch 25 | batch 2230/2304 [I 2022-10-26 19:26:25] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:41 [I 2022-10-26 19:26:25] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00254 [I 2022-10-26 19:26:25] ...trap/engines/engine.py.230: loss: 0.15559 [I 2022-10-26 19:26:25] ...optimizers/trijoint.py.151: total_norm: 27.060841 [I 2022-10-26 19:26:30] ...trap/engines/engine.py.224: train: epoch 25 | batch 2240/2304 [I 2022-10-26 19:26:30] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:35 [I 2022-10-26 19:26:30] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00248 [I 2022-10-26 19:26:30] ...trap/engines/engine.py.230: loss: 0.20946 [I 2022-10-26 19:26:30] ...optimizers/trijoint.py.151: total_norm: 37.719158 [I 2022-10-26 19:26:36] ...trap/engines/engine.py.224: train: epoch 25 | batch 2250/2304 [I 2022-10-26 19:26:36] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:30 [I 2022-10-26 19:26:36] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00244 [I 2022-10-26 19:26:36] ...trap/engines/engine.py.230: loss: 0.19285 [I 2022-10-26 19:26:36] ...optimizers/trijoint.py.151: total_norm: 17.742371 [I 2022-10-26 19:26:41] ...trap/engines/engine.py.224: train: epoch 25 | batch 2260/2304 [I 2022-10-26 19:26:41] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:24 [I 2022-10-26 19:26:41] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00245 [I 2022-10-26 19:26:41] ...trap/engines/engine.py.230: loss: 0.25487 [I 2022-10-26 19:26:41] ...optimizers/trijoint.py.151: total_norm: 17.022902 [I 2022-10-26 19:26:47] ...trap/engines/engine.py.224: train: epoch 25 | batch 2270/2304 [I 2022-10-26 19:26:47] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:18 [I 2022-10-26 19:26:47] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00245 [I 2022-10-26 19:26:47] ...trap/engines/engine.py.230: loss: 0.18724 [I 2022-10-26 19:26:47] ...optimizers/trijoint.py.151: total_norm: 22.644016 [I 2022-10-26 19:26:53] ...trap/engines/engine.py.224: train: epoch 25 | batch 2280/2304 [I 2022-10-26 19:26:53] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:13 [I 2022-10-26 19:26:53] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00243 [I 2022-10-26 19:26:53] ...trap/engines/engine.py.230: loss: 0.26063 [I 2022-10-26 19:26:53] ...optimizers/trijoint.py.151: total_norm: 27.067987 [I 2022-10-26 19:26:58] ...trap/engines/engine.py.224: train: epoch 25 | batch 2290/2304 [I 2022-10-26 19:26:58] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:07 [I 2022-10-26 19:26:58] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00249 [I 2022-10-26 19:26:58] ...trap/engines/engine.py.230: loss: 0.15629 [I 2022-10-26 19:26:58] ...optimizers/trijoint.py.151: total_norm: 12.271449 [I 2022-10-26 19:27:04] ...trap/engines/engine.py.224: train: epoch 25 | batch 2300/2304 [I 2022-10-26 19:27:04] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:02 [I 2022-10-26 19:27:04] ...trap/engines/engine.py.229: process: 0.56389 | load: 0.00015 [I 2022-10-26 19:27:04] ...trap/engines/engine.py.230: loss: 0.17130 [I 2022-10-26 19:27:04] ...optimizers/trijoint.py.151: total_norm: 13.785565 [I 2022-10-26 19:27:06] ...trap/engines/engine.py.224: train: epoch 25 | batch 2304/2304 [I 2022-10-26 19:27:06] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:00 [I 2022-10-26 19:27:06] ...trap/engines/engine.py.229: process: 0.54025 | load: 0.00028 [I 2022-10-26 19:27:06] ...trap/engines/engine.py.230: loss: 0.22768 [I 2022-10-26 19:27:06] ...optimizers/trijoint.py.151: total_norm: 27.303190 [S 2022-10-26 19:27:07] ...trap/engines/engine.py.129: train_epoch.epoch: 25 [S 2022-10-26 19:27:07] ...trap/engines/engine.py.129: train_epoch.loss: 0.19064906287671687 [I 2022-10-26 19:27:07] ...s/criterions/triplet.py.84: margin = 0.1800000000000001 [I 2022-10-26 19:27:08] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 25 [I 2022-10-26 19:27:08] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 19:27:09] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 19:27:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 19:27:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 19:27:10] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 19:27:11] ...trap/engines/engine.py.321: eval: epoch 25 | batch 0/510 [I 2022-10-26 19:27:11] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:25:29 [I 2022-10-26 19:27:11] ...trap/engines/engine.py.326: process: 2.99916 | load: 2.59459 [I 2022-10-26 19:27:14] ...trap/engines/engine.py.321: eval: epoch 25 | batch 10/510 [I 2022-10-26 19:27:14] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:11 [I 2022-10-26 19:27:14] ...trap/engines/engine.py.326: process: 0.33216 | load: 0.00020 [I 2022-10-26 19:27:17] ...trap/engines/engine.py.321: eval: epoch 25 | batch 20/510 [I 2022-10-26 19:27:17] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:59 [I 2022-10-26 19:27:17] ...trap/engines/engine.py.326: process: 0.33830 | load: 0.00018 [I 2022-10-26 19:27:21] ...trap/engines/engine.py.321: eval: epoch 25 | batch 30/510 [I 2022-10-26 19:27:21] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:43 [I 2022-10-26 19:27:21] ...trap/engines/engine.py.326: process: 0.34782 | load: 0.00019 [I 2022-10-26 19:27:24] ...trap/engines/engine.py.321: eval: epoch 25 | batch 40/510 [I 2022-10-26 19:27:24] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:39 [I 2022-10-26 19:27:24] ...trap/engines/engine.py.326: process: 0.32777 | load: 0.00021 [I 2022-10-26 19:27:28] ...trap/engines/engine.py.321: eval: epoch 25 | batch 50/510 [I 2022-10-26 19:27:28] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:38 [I 2022-10-26 19:27:28] ...trap/engines/engine.py.326: process: 0.35145 | load: 0.00017 [I 2022-10-26 19:27:31] ...trap/engines/engine.py.321: eval: epoch 25 | batch 60/510 [I 2022-10-26 19:27:31] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:30 [I 2022-10-26 19:27:31] ...trap/engines/engine.py.326: process: 0.33582 | load: 0.00018 [I 2022-10-26 19:27:34] ...trap/engines/engine.py.321: eval: epoch 25 | batch 70/510 [I 2022-10-26 19:27:34] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:27 [I 2022-10-26 19:27:34] ...trap/engines/engine.py.326: process: 0.34041 | load: 0.00020 [I 2022-10-26 19:27:38] ...trap/engines/engine.py.321: eval: epoch 25 | batch 80/510 [I 2022-10-26 19:27:38] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:29 [I 2022-10-26 19:27:38] ...trap/engines/engine.py.326: process: 0.35636 | load: 0.00021 [I 2022-10-26 19:27:41] ...trap/engines/engine.py.321: eval: epoch 25 | batch 90/510 [I 2022-10-26 19:27:41] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:24 [I 2022-10-26 19:27:41] ...trap/engines/engine.py.326: process: 0.36079 | load: 0.00021 [I 2022-10-26 19:27:44] ...trap/engines/engine.py.321: eval: epoch 25 | batch 100/510 [I 2022-10-26 19:27:44] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:14 [I 2022-10-26 19:27:44] ...trap/engines/engine.py.326: process: 0.28837 | load: 0.00018 [I 2022-10-26 19:27:47] ...trap/engines/engine.py.321: eval: epoch 25 | batch 110/510 [I 2022-10-26 19:27:47] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-26 19:27:47] ...trap/engines/engine.py.326: process: 0.28852 | load: 0.00023 [I 2022-10-26 19:27:50] ...trap/engines/engine.py.321: eval: epoch 25 | batch 120/510 [I 2022-10-26 19:27:50] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:55 [I 2022-10-26 19:27:50] ...trap/engines/engine.py.326: process: 0.30601 | load: 0.00016 [I 2022-10-26 19:27:53] ...trap/engines/engine.py.321: eval: epoch 25 | batch 130/510 [I 2022-10-26 19:27:53] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:53 [I 2022-10-26 19:27:53] ...trap/engines/engine.py.326: process: 0.30868 | load: 0.00015 [I 2022-10-26 19:27:56] ...trap/engines/engine.py.321: eval: epoch 25 | batch 140/510 [I 2022-10-26 19:27:56] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:55 [I 2022-10-26 19:27:56] ...trap/engines/engine.py.326: process: 0.28893 | load: 0.00020 [I 2022-10-26 19:27:59] ...trap/engines/engine.py.321: eval: epoch 25 | batch 150/510 [I 2022-10-26 19:27:59] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 19:27:59] ...trap/engines/engine.py.326: process: 0.28099 | load: 0.00015 [I 2022-10-26 19:28:02] ...trap/engines/engine.py.321: eval: epoch 25 | batch 160/510 [I 2022-10-26 19:28:02] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-26 19:28:02] ...trap/engines/engine.py.326: process: 0.28803 | load: 0.00019 [I 2022-10-26 19:28:05] ...trap/engines/engine.py.321: eval: epoch 25 | batch 170/510 [I 2022-10-26 19:28:05] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-26 19:28:05] ...trap/engines/engine.py.326: process: 0.28758 | load: 0.00016 [I 2022-10-26 19:28:08] ...trap/engines/engine.py.321: eval: epoch 25 | batch 180/510 [I 2022-10-26 19:28:08] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:36 [I 2022-10-26 19:28:08] ...trap/engines/engine.py.326: process: 0.28814 | load: 0.00015 [I 2022-10-26 19:28:11] ...trap/engines/engine.py.321: eval: epoch 25 | batch 190/510 [I 2022-10-26 19:28:11] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:34 [I 2022-10-26 19:28:11] ...trap/engines/engine.py.326: process: 0.30319 | load: 0.00015 [I 2022-10-26 19:28:14] ...trap/engines/engine.py.321: eval: epoch 25 | batch 200/510 [I 2022-10-26 19:28:14] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:30 [I 2022-10-26 19:28:14] ...trap/engines/engine.py.326: process: 0.29641 | load: 0.00015 [I 2022-10-26 19:28:17] ...trap/engines/engine.py.321: eval: epoch 25 | batch 210/510 [I 2022-10-26 19:28:17] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 19:28:17] ...trap/engines/engine.py.326: process: 0.28059 | load: 0.00016 [I 2022-10-26 19:28:20] ...trap/engines/engine.py.321: eval: epoch 25 | batch 220/510 [I 2022-10-26 19:28:20] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 19:28:20] ...trap/engines/engine.py.326: process: 0.28726 | load: 0.00015 [I 2022-10-26 19:28:22] ...trap/engines/engine.py.321: eval: epoch 25 | batch 230/510 [I 2022-10-26 19:28:22] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 19:28:22] ...trap/engines/engine.py.326: process: 0.27914 | load: 0.00020 [I 2022-10-26 19:28:25] ...trap/engines/engine.py.321: eval: epoch 25 | batch 240/510 [I 2022-10-26 19:28:25] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 19:28:25] ...trap/engines/engine.py.326: process: 0.28806 | load: 0.00015 [I 2022-10-26 19:28:28] ...trap/engines/engine.py.321: eval: epoch 25 | batch 250/510 [I 2022-10-26 19:28:28] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-26 19:28:28] ...trap/engines/engine.py.326: process: 0.28120 | load: 0.00021 [I 2022-10-26 19:28:31] ...trap/engines/engine.py.321: eval: epoch 25 | batch 260/510 [I 2022-10-26 19:28:31] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-26 19:28:31] ...trap/engines/engine.py.326: process: 0.28997 | load: 0.00021 [I 2022-10-26 19:28:34] ...trap/engines/engine.py.321: eval: epoch 25 | batch 270/510 [I 2022-10-26 19:28:34] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-26 19:28:34] ...trap/engines/engine.py.326: process: 0.28479 | load: 0.00016 [I 2022-10-26 19:28:37] ...trap/engines/engine.py.321: eval: epoch 25 | batch 280/510 [I 2022-10-26 19:28:37] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-26 19:28:37] ...trap/engines/engine.py.326: process: 0.29017 | load: 0.00017 [I 2022-10-26 19:28:40] ...trap/engines/engine.py.321: eval: epoch 25 | batch 290/510 [I 2022-10-26 19:28:40] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-26 19:28:40] ...trap/engines/engine.py.326: process: 0.28670 | load: 0.00018 [I 2022-10-26 19:28:43] ...trap/engines/engine.py.321: eval: epoch 25 | batch 300/510 [I 2022-10-26 19:28:43] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:01 [I 2022-10-26 19:28:43] ...trap/engines/engine.py.326: process: 0.29233 | load: 0.00014 [I 2022-10-26 19:28:46] ...trap/engines/engine.py.321: eval: epoch 25 | batch 310/510 [I 2022-10-26 19:28:46] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:59 [I 2022-10-26 19:28:46] ...trap/engines/engine.py.326: process: 0.30956 | load: 0.00020 [I 2022-10-26 19:28:48] ...trap/engines/engine.py.321: eval: epoch 25 | batch 320/510 [I 2022-10-26 19:28:48] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 19:28:48] ...trap/engines/engine.py.326: process: 0.28787 | load: 0.00022 [I 2022-10-26 19:28:51] ...trap/engines/engine.py.321: eval: epoch 25 | batch 330/510 [I 2022-10-26 19:28:51] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:52 [I 2022-10-26 19:28:51] ...trap/engines/engine.py.326: process: 0.28020 | load: 0.00055 [I 2022-10-26 19:28:54] ...trap/engines/engine.py.321: eval: epoch 25 | batch 340/510 [I 2022-10-26 19:28:54] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:49 [I 2022-10-26 19:28:54] ...trap/engines/engine.py.326: process: 0.29093 | load: 0.00015 [I 2022-10-26 19:28:57] ...trap/engines/engine.py.321: eval: epoch 25 | batch 350/510 [I 2022-10-26 19:28:57] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 19:28:57] ...trap/engines/engine.py.326: process: 0.28461 | load: 0.00020 [I 2022-10-26 19:29:00] ...trap/engines/engine.py.321: eval: epoch 25 | batch 360/510 [I 2022-10-26 19:29:00] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 19:29:00] ...trap/engines/engine.py.326: process: 0.28992 | load: 0.00016 [I 2022-10-26 19:29:03] ...trap/engines/engine.py.321: eval: epoch 25 | batch 370/510 [I 2022-10-26 19:29:03] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-26 19:29:03] ...trap/engines/engine.py.326: process: 0.28538 | load: 0.00021 [I 2022-10-26 19:29:06] ...trap/engines/engine.py.321: eval: epoch 25 | batch 380/510 [I 2022-10-26 19:29:06] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:39 [I 2022-10-26 19:29:06] ...trap/engines/engine.py.326: process: 0.28823 | load: 0.00022 [I 2022-10-26 19:29:09] ...trap/engines/engine.py.321: eval: epoch 25 | batch 390/510 [I 2022-10-26 19:29:09] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-26 19:29:09] ...trap/engines/engine.py.326: process: 0.28695 | load: 0.00039 [I 2022-10-26 19:29:12] ...trap/engines/engine.py.321: eval: epoch 25 | batch 400/510 [I 2022-10-26 19:29:12] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 19:29:12] ...trap/engines/engine.py.326: process: 0.28904 | load: 0.00015 [I 2022-10-26 19:29:15] ...trap/engines/engine.py.321: eval: epoch 25 | batch 410/510 [I 2022-10-26 19:29:15] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 19:29:15] ...trap/engines/engine.py.326: process: 0.29361 | load: 0.00015 [I 2022-10-26 19:29:17] ...trap/engines/engine.py.321: eval: epoch 25 | batch 420/510 [I 2022-10-26 19:29:17] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 19:29:17] ...trap/engines/engine.py.326: process: 0.29051 | load: 0.00020 [I 2022-10-26 19:29:20] ...trap/engines/engine.py.321: eval: epoch 25 | batch 430/510 [I 2022-10-26 19:29:20] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 19:29:20] ...trap/engines/engine.py.326: process: 0.29019 | load: 0.00016 [I 2022-10-26 19:29:23] ...trap/engines/engine.py.321: eval: epoch 25 | batch 440/510 [I 2022-10-26 19:29:23] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 19:29:23] ...trap/engines/engine.py.326: process: 0.28615 | load: 0.00016 [I 2022-10-26 19:29:26] ...trap/engines/engine.py.321: eval: epoch 25 | batch 450/510 [I 2022-10-26 19:29:26] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 19:29:26] ...trap/engines/engine.py.326: process: 0.28689 | load: 0.00022 [I 2022-10-26 19:29:29] ...trap/engines/engine.py.321: eval: epoch 25 | batch 460/510 [I 2022-10-26 19:29:29] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-26 19:29:29] ...trap/engines/engine.py.326: process: 0.29682 | load: 0.00017 [I 2022-10-26 19:29:32] ...trap/engines/engine.py.321: eval: epoch 25 | batch 470/510 [I 2022-10-26 19:29:32] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-26 19:29:32] ...trap/engines/engine.py.326: process: 0.29213 | load: 0.00016 [I 2022-10-26 19:29:35] ...trap/engines/engine.py.321: eval: epoch 25 | batch 480/510 [I 2022-10-26 19:29:35] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-26 19:29:35] ...trap/engines/engine.py.326: process: 0.29227 | load: 0.00015 [I 2022-10-26 19:29:38] ...trap/engines/engine.py.321: eval: epoch 25 | batch 490/510 [I 2022-10-26 19:29:38] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 19:29:38] ...trap/engines/engine.py.326: process: 0.28824 | load: 0.00014 [I 2022-10-26 19:29:40] ...trap/engines/engine.py.321: eval: epoch 25 | batch 500/510 [I 2022-10-26 19:29:40] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 19:29:40] ...trap/engines/engine.py.326: process: 0.28354 | load: 0.00016 [I 2022-10-26 19:29:43] ...trap/engines/engine.py.321: eval: epoch 25 | batch 510/510 [I 2022-10-26 19:29:43] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 19:29:43] ...trap/engines/engine.py.326: process: 0.27824 | load: 0.00017 [S 2022-10-26 19:29:44] ...trap/engines/engine.py.132: eval_epoch.epoch: 25 [I 2022-10-26 19:29:44] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.629800 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.867100 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.918100 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.614500 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.862500 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.913700 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016253 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007489 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008166 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012761 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009952 [S 2022-10-26 19:29:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009317 [I 2022-10-26 19:29:49] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 19:29:49] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 19:29:49] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 19:29:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 19:29:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 19:29:50] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 19:29:53] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 19:29:59] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 19:29:59] ...trap/engines/engine.py.163: Training model on trainset for epoch 26 [I 2022-10-26 19:29:59] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 19:30:01] ...trap/engines/engine.py.224: train: epoch 26 | batch 0/2304 [I 2022-10-26 19:30:01] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:04 [I 2022-10-26 19:30:01] ...trap/engines/engine.py.229: process: 2.68429 | load: 2.04105 [I 2022-10-26 19:30:01] ...trap/engines/engine.py.230: loss: 0.17557 [I 2022-10-26 19:30:01] ...optimizers/trijoint.py.151: total_norm: 21.343182 [I 2022-10-26 19:30:07] ...trap/engines/engine.py.224: train: epoch 26 | batch 10/2304 [I 2022-10-26 19:30:07] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:03 [I 2022-10-26 19:30:07] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00404 [I 2022-10-26 19:30:07] ...trap/engines/engine.py.230: loss: 0.20366 [I 2022-10-26 19:30:07] ...optimizers/trijoint.py.151: total_norm: 13.134379 [I 2022-10-26 19:30:13] ...trap/engines/engine.py.224: train: epoch 26 | batch 20/2304 [I 2022-10-26 19:30:13] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:33 [I 2022-10-26 19:30:13] ...trap/engines/engine.py.229: process: 0.56349 | load: 0.00378 [I 2022-10-26 19:30:13] ...trap/engines/engine.py.230: loss: 0.16663 [I 2022-10-26 19:30:13] ...optimizers/trijoint.py.151: total_norm: 17.263084 [I 2022-10-26 19:30:18] ...trap/engines/engine.py.224: train: epoch 26 | batch 30/2304 [I 2022-10-26 19:30:18] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:08 [I 2022-10-26 19:30:18] ...trap/engines/engine.py.229: process: 0.54807 | load: 0.00294 [I 2022-10-26 19:30:18] ...trap/engines/engine.py.230: loss: 0.16118 [I 2022-10-26 19:30:18] ...optimizers/trijoint.py.151: total_norm: 23.396492 [I 2022-10-26 19:30:24] ...trap/engines/engine.py.224: train: epoch 26 | batch 40/2304 [I 2022-10-26 19:30:24] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:06 [I 2022-10-26 19:30:24] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00307 [I 2022-10-26 19:30:24] ...trap/engines/engine.py.230: loss: 0.14638 [I 2022-10-26 19:30:24] ...optimizers/trijoint.py.151: total_norm: 21.200203 [I 2022-10-26 19:30:29] ...trap/engines/engine.py.224: train: epoch 26 | batch 50/2304 [I 2022-10-26 19:30:29] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:44 [I 2022-10-26 19:30:29] ...trap/engines/engine.py.229: process: 0.54193 | load: 0.00263 [I 2022-10-26 19:30:29] ...trap/engines/engine.py.230: loss: 0.14659 [I 2022-10-26 19:30:29] ...optimizers/trijoint.py.151: total_norm: 16.560501 [I 2022-10-26 19:30:35] ...trap/engines/engine.py.224: train: epoch 26 | batch 60/2304 [I 2022-10-26 19:30:35] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:57 [I 2022-10-26 19:30:35] ...trap/engines/engine.py.229: process: 0.54408 | load: 0.00301 [I 2022-10-26 19:30:35] ...trap/engines/engine.py.230: loss: 0.20199 [I 2022-10-26 19:30:35] ...optimizers/trijoint.py.151: total_norm: 21.520067 [I 2022-10-26 19:30:41] ...trap/engines/engine.py.224: train: epoch 26 | batch 70/2304 [I 2022-10-26 19:30:41] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:45 [I 2022-10-26 19:30:41] ...trap/engines/engine.py.229: process: 0.54750 | load: 0.00285 [I 2022-10-26 19:30:41] ...trap/engines/engine.py.230: loss: 0.13830 [I 2022-10-26 19:30:41] ...optimizers/trijoint.py.151: total_norm: 18.364117 [I 2022-10-26 19:30:46] ...trap/engines/engine.py.224: train: epoch 26 | batch 80/2304 [I 2022-10-26 19:30:46] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:42 [I 2022-10-26 19:30:46] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00278 [I 2022-10-26 19:30:46] ...trap/engines/engine.py.230: loss: 0.11096 [I 2022-10-26 19:30:46] ...optimizers/trijoint.py.151: total_norm: 16.624619 [I 2022-10-26 19:31:15] ...trap/engines/engine.py.224: train: epoch 26 | batch 90/2304 [I 2022-10-26 19:31:15] ...trap/engines/engine.py.228: elapsed: 0:01:16 | left: 0:49:16 [I 2022-10-26 19:31:15] ...trap/engines/engine.py.229: process: 0.54553 | load: 0.00257 [I 2022-10-26 19:31:15] ...trap/engines/engine.py.230: loss: 0.21184 [I 2022-10-26 19:31:15] ...optimizers/trijoint.py.151: total_norm: 17.853430 [I 2022-10-26 19:31:21] ...trap/engines/engine.py.224: train: epoch 26 | batch 100/2304 [I 2022-10-26 19:31:21] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:23:34 [I 2022-10-26 19:31:21] ...trap/engines/engine.py.229: process: 0.55993 | load: 0.00238 [I 2022-10-26 19:31:21] ...trap/engines/engine.py.230: loss: 0.14209 [I 2022-10-26 19:31:21] ...optimizers/trijoint.py.151: total_norm: 15.533364 [I 2022-10-26 19:31:26] ...trap/engines/engine.py.224: train: epoch 26 | batch 110/2304 [I 2022-10-26 19:31:26] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:20:43 [I 2022-10-26 19:31:26] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00258 [I 2022-10-26 19:31:26] ...trap/engines/engine.py.230: loss: 0.15969 [I 2022-10-26 19:31:26] ...optimizers/trijoint.py.151: total_norm: 20.767696 [I 2022-10-26 19:31:32] ...trap/engines/engine.py.224: train: epoch 26 | batch 120/2304 [I 2022-10-26 19:31:32] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:17 [I 2022-10-26 19:31:32] ...trap/engines/engine.py.229: process: 0.54898 | load: 0.00253 [I 2022-10-26 19:31:32] ...trap/engines/engine.py.230: loss: 0.15427 [I 2022-10-26 19:31:32] ...optimizers/trijoint.py.151: total_norm: 15.540203 [I 2022-10-26 19:31:37] ...trap/engines/engine.py.224: train: epoch 26 | batch 130/2304 [I 2022-10-26 19:31:37] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:20:04 [I 2022-10-26 19:31:37] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00250 [I 2022-10-26 19:31:37] ...trap/engines/engine.py.230: loss: 0.21786 [I 2022-10-26 19:31:37] ...optimizers/trijoint.py.151: total_norm: 18.752222 [I 2022-10-26 19:31:43] ...trap/engines/engine.py.224: train: epoch 26 | batch 140/2304 [I 2022-10-26 19:31:43] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:20:00 [I 2022-10-26 19:31:43] ...trap/engines/engine.py.229: process: 0.54121 | load: 0.00236 [I 2022-10-26 19:31:43] ...trap/engines/engine.py.230: loss: 0.12457 [I 2022-10-26 19:31:43] ...optimizers/trijoint.py.151: total_norm: 10.103661 [I 2022-10-26 19:31:48] ...trap/engines/engine.py.224: train: epoch 26 | batch 150/2304 [I 2022-10-26 19:31:48] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:50 [I 2022-10-26 19:31:48] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00257 [I 2022-10-26 19:31:48] ...trap/engines/engine.py.230: loss: 0.21280 [I 2022-10-26 19:31:48] ...optimizers/trijoint.py.151: total_norm: 19.227875 [I 2022-10-26 19:31:54] ...trap/engines/engine.py.224: train: epoch 26 | batch 160/2304 [I 2022-10-26 19:31:54] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:48 [I 2022-10-26 19:31:54] ...trap/engines/engine.py.229: process: 0.54886 | load: 0.00246 [I 2022-10-26 19:31:54] ...trap/engines/engine.py.230: loss: 0.12706 [I 2022-10-26 19:31:54] ...optimizers/trijoint.py.151: total_norm: 12.127367 [I 2022-10-26 19:31:59] ...trap/engines/engine.py.224: train: epoch 26 | batch 170/2304 [I 2022-10-26 19:31:59] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:52 [I 2022-10-26 19:31:59] ...trap/engines/engine.py.229: process: 0.56604 | load: 0.00238 [I 2022-10-26 19:31:59] ...trap/engines/engine.py.230: loss: 0.20935 [I 2022-10-26 19:31:59] ...optimizers/trijoint.py.151: total_norm: 26.027359 [I 2022-10-26 19:32:05] ...trap/engines/engine.py.224: train: epoch 26 | batch 180/2304 [I 2022-10-26 19:32:05] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:41 [I 2022-10-26 19:32:05] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00250 [I 2022-10-26 19:32:05] ...trap/engines/engine.py.230: loss: 0.19157 [I 2022-10-26 19:32:05] ...optimizers/trijoint.py.151: total_norm: 21.789318 [I 2022-10-26 19:32:11] ...trap/engines/engine.py.224: train: epoch 26 | batch 190/2304 [I 2022-10-26 19:32:11] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:33 [I 2022-10-26 19:32:11] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00237 [I 2022-10-26 19:32:11] ...trap/engines/engine.py.230: loss: 0.15104 [I 2022-10-26 19:32:11] ...optimizers/trijoint.py.151: total_norm: 12.486805 [I 2022-10-26 19:32:16] ...trap/engines/engine.py.224: train: epoch 26 | batch 200/2304 [I 2022-10-26 19:32:16] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:30 [I 2022-10-26 19:32:16] ...trap/engines/engine.py.229: process: 0.55131 | load: 0.00251 [I 2022-10-26 19:32:16] ...trap/engines/engine.py.230: loss: 0.14794 [I 2022-10-26 19:32:16] ...optimizers/trijoint.py.151: total_norm: 18.946249 [I 2022-10-26 19:32:22] ...trap/engines/engine.py.224: train: epoch 26 | batch 210/2304 [I 2022-10-26 19:32:22] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:24 [I 2022-10-26 19:32:22] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00273 [I 2022-10-26 19:32:22] ...trap/engines/engine.py.230: loss: 0.20854 [I 2022-10-26 19:32:22] ...optimizers/trijoint.py.151: total_norm: 21.288078 [I 2022-10-26 19:32:27] ...trap/engines/engine.py.224: train: epoch 26 | batch 220/2304 [I 2022-10-26 19:32:27] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:26 [I 2022-10-26 19:32:27] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00246 [I 2022-10-26 19:32:27] ...trap/engines/engine.py.230: loss: 0.22464 [I 2022-10-26 19:32:27] ...optimizers/trijoint.py.151: total_norm: 20.372536 [I 2022-10-26 19:32:33] ...trap/engines/engine.py.224: train: epoch 26 | batch 230/2304 [I 2022-10-26 19:32:33] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:19:12 [I 2022-10-26 19:32:33] ...trap/engines/engine.py.229: process: 0.54853 | load: 0.00250 [I 2022-10-26 19:32:33] ...trap/engines/engine.py.230: loss: 0.14058 [I 2022-10-26 19:32:33] ...optimizers/trijoint.py.151: total_norm: 11.553954 [I 2022-10-26 19:32:39] ...trap/engines/engine.py.224: train: epoch 26 | batch 240/2304 [I 2022-10-26 19:32:39] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:19:01 [I 2022-10-26 19:32:39] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00249 [I 2022-10-26 19:32:39] ...trap/engines/engine.py.230: loss: 0.24699 [I 2022-10-26 19:32:39] ...optimizers/trijoint.py.151: total_norm: 12.317717 [I 2022-10-26 19:32:44] ...trap/engines/engine.py.224: train: epoch 26 | batch 250/2304 [I 2022-10-26 19:32:44] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:19:06 [I 2022-10-26 19:32:44] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00246 [I 2022-10-26 19:32:44] ...trap/engines/engine.py.230: loss: 0.29729 [I 2022-10-26 19:32:44] ...optimizers/trijoint.py.151: total_norm: 25.035919 [I 2022-10-26 19:32:50] ...trap/engines/engine.py.224: train: epoch 26 | batch 260/2304 [I 2022-10-26 19:32:50] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:55 [I 2022-10-26 19:32:50] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00263 [I 2022-10-26 19:32:50] ...trap/engines/engine.py.230: loss: 0.15624 [I 2022-10-26 19:32:50] ...optimizers/trijoint.py.151: total_norm: 14.939512 [I 2022-10-26 19:32:55] ...trap/engines/engine.py.224: train: epoch 26 | batch 270/2304 [I 2022-10-26 19:32:55] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:58 [I 2022-10-26 19:32:55] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00240 [I 2022-10-26 19:32:55] ...trap/engines/engine.py.230: loss: 0.15557 [I 2022-10-26 19:32:55] ...optimizers/trijoint.py.151: total_norm: 57.937389 [I 2022-10-26 19:33:01] ...trap/engines/engine.py.224: train: epoch 26 | batch 280/2304 [I 2022-10-26 19:33:01] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:49 [I 2022-10-26 19:33:01] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00251 [I 2022-10-26 19:33:01] ...trap/engines/engine.py.230: loss: 0.14856 [I 2022-10-26 19:33:01] ...optimizers/trijoint.py.151: total_norm: 10.942831 [I 2022-10-26 19:33:06] ...trap/engines/engine.py.224: train: epoch 26 | batch 290/2304 [I 2022-10-26 19:33:06] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:43 [I 2022-10-26 19:33:06] ...trap/engines/engine.py.229: process: 0.57053 | load: 0.00252 [I 2022-10-26 19:33:06] ...trap/engines/engine.py.230: loss: 0.13552 [I 2022-10-26 19:33:06] ...optimizers/trijoint.py.151: total_norm: 14.257574 [I 2022-10-26 19:33:12] ...trap/engines/engine.py.224: train: epoch 26 | batch 300/2304 [I 2022-10-26 19:33:12] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:33 [I 2022-10-26 19:33:12] ...trap/engines/engine.py.229: process: 0.54394 | load: 0.00256 [I 2022-10-26 19:33:12] ...trap/engines/engine.py.230: loss: 0.12590 [I 2022-10-26 19:33:12] ...optimizers/trijoint.py.151: total_norm: 11.048695 [I 2022-10-26 19:33:18] ...trap/engines/engine.py.224: train: epoch 26 | batch 310/2304 [I 2022-10-26 19:33:18] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:32 [I 2022-10-26 19:33:18] ...trap/engines/engine.py.229: process: 0.56396 | load: 0.00242 [I 2022-10-26 19:33:18] ...trap/engines/engine.py.230: loss: 0.21177 [I 2022-10-26 19:33:18] ...optimizers/trijoint.py.151: total_norm: 36.072277 [I 2022-10-26 19:33:23] ...trap/engines/engine.py.224: train: epoch 26 | batch 320/2304 [I 2022-10-26 19:33:23] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:20 [I 2022-10-26 19:33:23] ...trap/engines/engine.py.229: process: 0.54700 | load: 0.00261 [I 2022-10-26 19:33:23] ...trap/engines/engine.py.230: loss: 0.30688 [I 2022-10-26 19:33:23] ...optimizers/trijoint.py.151: total_norm: 22.564508 [I 2022-10-26 19:33:29] ...trap/engines/engine.py.224: train: epoch 26 | batch 330/2304 [I 2022-10-26 19:33:29] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:18 [I 2022-10-26 19:33:29] ...trap/engines/engine.py.229: process: 0.54866 | load: 0.00248 [I 2022-10-26 19:33:29] ...trap/engines/engine.py.230: loss: 0.14006 [I 2022-10-26 19:33:29] ...optimizers/trijoint.py.151: total_norm: 27.428026 [I 2022-10-26 19:33:34] ...trap/engines/engine.py.224: train: epoch 26 | batch 340/2304 [I 2022-10-26 19:33:34] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:18:16 [I 2022-10-26 19:33:34] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00250 [I 2022-10-26 19:33:34] ...trap/engines/engine.py.230: loss: 0.23557 [I 2022-10-26 19:33:34] ...optimizers/trijoint.py.151: total_norm: 28.309193 [I 2022-10-26 19:33:40] ...trap/engines/engine.py.224: train: epoch 26 | batch 350/2304 [I 2022-10-26 19:33:40] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:18:11 [I 2022-10-26 19:33:40] ...trap/engines/engine.py.229: process: 0.56533 | load: 0.00243 [I 2022-10-26 19:33:40] ...trap/engines/engine.py.230: loss: 0.17807 [I 2022-10-26 19:33:40] ...optimizers/trijoint.py.151: total_norm: 15.064343 [I 2022-10-26 19:33:45] ...trap/engines/engine.py.224: train: epoch 26 | batch 360/2304 [I 2022-10-26 19:33:45] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:18:00 [I 2022-10-26 19:33:45] ...trap/engines/engine.py.229: process: 0.54913 | load: 0.00250 [I 2022-10-26 19:33:45] ...trap/engines/engine.py.230: loss: 0.31271 [I 2022-10-26 19:33:45] ...optimizers/trijoint.py.151: total_norm: 21.445059 [I 2022-10-26 19:33:51] ...trap/engines/engine.py.224: train: epoch 26 | batch 370/2304 [I 2022-10-26 19:33:51] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:59 [I 2022-10-26 19:33:51] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00251 [I 2022-10-26 19:33:51] ...trap/engines/engine.py.230: loss: 0.14598 [I 2022-10-26 19:33:51] ...optimizers/trijoint.py.151: total_norm: 31.441040 [I 2022-10-26 19:33:57] ...trap/engines/engine.py.224: train: epoch 26 | batch 380/2304 [I 2022-10-26 19:33:57] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:55 [I 2022-10-26 19:33:57] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00242 [I 2022-10-26 19:33:57] ...trap/engines/engine.py.230: loss: 0.19585 [I 2022-10-26 19:33:57] ...optimizers/trijoint.py.151: total_norm: 14.477598 [I 2022-10-26 19:34:02] ...trap/engines/engine.py.224: train: epoch 26 | batch 390/2304 [I 2022-10-26 19:34:02] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:40 [I 2022-10-26 19:34:02] ...trap/engines/engine.py.229: process: 0.55119 | load: 0.00242 [I 2022-10-26 19:34:02] ...trap/engines/engine.py.230: loss: 0.20315 [I 2022-10-26 19:34:02] ...optimizers/trijoint.py.151: total_norm: 20.683252 [I 2022-10-26 19:34:08] ...trap/engines/engine.py.224: train: epoch 26 | batch 400/2304 [I 2022-10-26 19:34:08] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:41 [I 2022-10-26 19:34:08] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00266 [I 2022-10-26 19:34:08] ...trap/engines/engine.py.230: loss: 0.21383 [I 2022-10-26 19:34:08] ...optimizers/trijoint.py.151: total_norm: 26.040365 [I 2022-10-26 19:34:13] ...trap/engines/engine.py.224: train: epoch 26 | batch 410/2304 [I 2022-10-26 19:34:13] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:27 [I 2022-10-26 19:34:13] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00245 [I 2022-10-26 19:34:13] ...trap/engines/engine.py.230: loss: 0.18896 [I 2022-10-26 19:34:13] ...optimizers/trijoint.py.151: total_norm: 20.209980 [I 2022-10-26 19:34:19] ...trap/engines/engine.py.224: train: epoch 26 | batch 420/2304 [I 2022-10-26 19:34:19] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:25 [I 2022-10-26 19:34:19] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00251 [I 2022-10-26 19:34:19] ...trap/engines/engine.py.230: loss: 0.13920 [I 2022-10-26 19:34:19] ...optimizers/trijoint.py.151: total_norm: 8.901936 [I 2022-10-26 19:34:25] ...trap/engines/engine.py.224: train: epoch 26 | batch 430/2304 [I 2022-10-26 19:34:25] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:23 [I 2022-10-26 19:34:25] ...trap/engines/engine.py.229: process: 0.55030 | load: 0.00240 [I 2022-10-26 19:34:25] ...trap/engines/engine.py.230: loss: 0.15460 [I 2022-10-26 19:34:25] ...optimizers/trijoint.py.151: total_norm: 15.571859 [I 2022-10-26 19:34:30] ...trap/engines/engine.py.224: train: epoch 26 | batch 440/2304 [I 2022-10-26 19:34:30] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:12 [I 2022-10-26 19:34:30] ...trap/engines/engine.py.229: process: 0.55227 | load: 0.00249 [I 2022-10-26 19:34:30] ...trap/engines/engine.py.230: loss: 0.24562 [I 2022-10-26 19:34:30] ...optimizers/trijoint.py.151: total_norm: 13.905139 [I 2022-10-26 19:34:36] ...trap/engines/engine.py.224: train: epoch 26 | batch 450/2304 [I 2022-10-26 19:34:36] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:17:07 [I 2022-10-26 19:34:36] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00253 [I 2022-10-26 19:34:36] ...trap/engines/engine.py.230: loss: 0.13874 [I 2022-10-26 19:34:36] ...optimizers/trijoint.py.151: total_norm: 19.423006 [I 2022-10-26 19:34:41] ...trap/engines/engine.py.224: train: epoch 26 | batch 460/2304 [I 2022-10-26 19:34:41] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:17:04 [I 2022-10-26 19:34:41] ...trap/engines/engine.py.229: process: 0.54357 | load: 0.00250 [I 2022-10-26 19:34:41] ...trap/engines/engine.py.230: loss: 0.14352 [I 2022-10-26 19:34:41] ...optimizers/trijoint.py.151: total_norm: 11.103769 [I 2022-10-26 19:34:47] ...trap/engines/engine.py.224: train: epoch 26 | batch 470/2304 [I 2022-10-26 19:34:47] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:17:01 [I 2022-10-26 19:34:47] ...trap/engines/engine.py.229: process: 0.56467 | load: 0.00247 [I 2022-10-26 19:34:47] ...trap/engines/engine.py.230: loss: 0.19951 [I 2022-10-26 19:34:47] ...optimizers/trijoint.py.151: total_norm: 21.800123 [I 2022-10-26 19:34:52] ...trap/engines/engine.py.224: train: epoch 26 | batch 480/2304 [I 2022-10-26 19:34:52] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:51 [I 2022-10-26 19:34:52] ...trap/engines/engine.py.229: process: 0.56160 | load: 0.00242 [I 2022-10-26 19:34:52] ...trap/engines/engine.py.230: loss: 0.15499 [I 2022-10-26 19:34:52] ...optimizers/trijoint.py.151: total_norm: 16.209625 [I 2022-10-26 19:34:58] ...trap/engines/engine.py.224: train: epoch 26 | batch 490/2304 [I 2022-10-26 19:34:58] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:40 [I 2022-10-26 19:34:58] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00242 [I 2022-10-26 19:34:58] ...trap/engines/engine.py.230: loss: 0.24147 [I 2022-10-26 19:34:58] ...optimizers/trijoint.py.151: total_norm: 57.907238 [I 2022-10-26 19:35:03] ...trap/engines/engine.py.224: train: epoch 26 | batch 500/2304 [I 2022-10-26 19:35:03] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:40 [I 2022-10-26 19:35:03] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00254 [I 2022-10-26 19:35:03] ...trap/engines/engine.py.230: loss: 0.13390 [I 2022-10-26 19:35:03] ...optimizers/trijoint.py.151: total_norm: 14.896472 [I 2022-10-26 19:35:09] ...trap/engines/engine.py.224: train: epoch 26 | batch 510/2304 [I 2022-10-26 19:35:09] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:45 [I 2022-10-26 19:35:09] ...trap/engines/engine.py.229: process: 0.56604 | load: 0.00249 [I 2022-10-26 19:35:09] ...trap/engines/engine.py.230: loss: 0.23242 [I 2022-10-26 19:35:09] ...optimizers/trijoint.py.151: total_norm: 25.392082 [I 2022-10-26 19:35:15] ...trap/engines/engine.py.224: train: epoch 26 | batch 520/2304 [I 2022-10-26 19:35:15] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:35 [I 2022-10-26 19:35:15] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00249 [I 2022-10-26 19:35:15] ...trap/engines/engine.py.230: loss: 0.12471 [I 2022-10-26 19:35:15] ...optimizers/trijoint.py.151: total_norm: 14.638210 [I 2022-10-26 19:35:20] ...trap/engines/engine.py.224: train: epoch 26 | batch 530/2304 [I 2022-10-26 19:35:20] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:24 [I 2022-10-26 19:35:20] ...trap/engines/engine.py.229: process: 0.54750 | load: 0.00246 [I 2022-10-26 19:35:20] ...trap/engines/engine.py.230: loss: 0.16271 [I 2022-10-26 19:35:20] ...optimizers/trijoint.py.151: total_norm: 13.750777 [I 2022-10-26 19:35:26] ...trap/engines/engine.py.224: train: epoch 26 | batch 540/2304 [I 2022-10-26 19:35:26] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:19 [I 2022-10-26 19:35:26] ...trap/engines/engine.py.229: process: 0.56184 | load: 0.00244 [I 2022-10-26 19:35:26] ...trap/engines/engine.py.230: loss: 0.15923 [I 2022-10-26 19:35:26] ...optimizers/trijoint.py.151: total_norm: 17.122812 [I 2022-10-26 19:35:31] ...trap/engines/engine.py.224: train: epoch 26 | batch 550/2304 [I 2022-10-26 19:35:31] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:16:15 [I 2022-10-26 19:35:31] ...trap/engines/engine.py.229: process: 0.55335 | load: 0.00244 [I 2022-10-26 19:35:31] ...trap/engines/engine.py.230: loss: 0.18272 [I 2022-10-26 19:35:31] ...optimizers/trijoint.py.151: total_norm: 25.371832 [I 2022-10-26 19:35:37] ...trap/engines/engine.py.224: train: epoch 26 | batch 560/2304 [I 2022-10-26 19:35:37] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:16:20 [I 2022-10-26 19:35:37] ...trap/engines/engine.py.229: process: 0.56109 | load: 0.00241 [I 2022-10-26 19:35:37] ...trap/engines/engine.py.230: loss: 0.16734 [I 2022-10-26 19:35:37] ...optimizers/trijoint.py.151: total_norm: 19.300810 [I 2022-10-26 19:35:43] ...trap/engines/engine.py.224: train: epoch 26 | batch 570/2304 [I 2022-10-26 19:35:43] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:58 [I 2022-10-26 19:35:43] ...trap/engines/engine.py.229: process: 0.54997 | load: 0.00241 [I 2022-10-26 19:35:43] ...trap/engines/engine.py.230: loss: 0.21527 [I 2022-10-26 19:35:43] ...optimizers/trijoint.py.151: total_norm: 24.985033 [I 2022-10-26 19:35:48] ...trap/engines/engine.py.224: train: epoch 26 | batch 580/2304 [I 2022-10-26 19:35:48] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:16:00 [I 2022-10-26 19:35:48] ...trap/engines/engine.py.229: process: 0.57484 | load: 0.00240 [I 2022-10-26 19:35:48] ...trap/engines/engine.py.230: loss: 0.22292 [I 2022-10-26 19:35:48] ...optimizers/trijoint.py.151: total_norm: 17.294226 [I 2022-10-26 19:35:54] ...trap/engines/engine.py.224: train: epoch 26 | batch 590/2304 [I 2022-10-26 19:35:54] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:49 [I 2022-10-26 19:35:54] ...trap/engines/engine.py.229: process: 0.54731 | load: 0.00243 [I 2022-10-26 19:35:54] ...trap/engines/engine.py.230: loss: 0.16553 [I 2022-10-26 19:35:54] ...optimizers/trijoint.py.151: total_norm: 18.305771 [I 2022-10-26 19:35:59] ...trap/engines/engine.py.224: train: epoch 26 | batch 600/2304 [I 2022-10-26 19:35:59] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:46 [I 2022-10-26 19:35:59] ...trap/engines/engine.py.229: process: 0.54790 | load: 0.00240 [I 2022-10-26 19:35:59] ...trap/engines/engine.py.230: loss: 0.12389 [I 2022-10-26 19:35:59] ...optimizers/trijoint.py.151: total_norm: 15.058007 [I 2022-10-26 19:36:05] ...trap/engines/engine.py.224: train: epoch 26 | batch 610/2304 [I 2022-10-26 19:36:05] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:42 [I 2022-10-26 19:36:05] ...trap/engines/engine.py.229: process: 0.55975 | load: 0.00243 [I 2022-10-26 19:36:05] ...trap/engines/engine.py.230: loss: 0.23017 [I 2022-10-26 19:36:05] ...optimizers/trijoint.py.151: total_norm: 18.323696 [I 2022-10-26 19:36:11] ...trap/engines/engine.py.224: train: epoch 26 | batch 620/2304 [I 2022-10-26 19:36:11] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:49 [I 2022-10-26 19:36:11] ...trap/engines/engine.py.229: process: 0.58261 | load: 0.00274 [I 2022-10-26 19:36:11] ...trap/engines/engine.py.230: loss: 0.22203 [I 2022-10-26 19:36:11] ...optimizers/trijoint.py.151: total_norm: 23.340389 [I 2022-10-26 19:36:16] ...trap/engines/engine.py.224: train: epoch 26 | batch 630/2304 [I 2022-10-26 19:36:16] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:29 [I 2022-10-26 19:36:16] ...trap/engines/engine.py.229: process: 0.55025 | load: 0.00249 [I 2022-10-26 19:36:16] ...trap/engines/engine.py.230: loss: 0.26211 [I 2022-10-26 19:36:16] ...optimizers/trijoint.py.151: total_norm: 18.850916 [I 2022-10-26 19:36:22] ...trap/engines/engine.py.224: train: epoch 26 | batch 640/2304 [I 2022-10-26 19:36:22] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:16:00 [I 2022-10-26 19:36:22] ...trap/engines/engine.py.229: process: 0.56990 | load: 0.00266 [I 2022-10-26 19:36:22] ...trap/engines/engine.py.230: loss: 0.24233 [I 2022-10-26 19:36:22] ...optimizers/trijoint.py.151: total_norm: 20.029385 [I 2022-10-26 19:36:27] ...trap/engines/engine.py.224: train: epoch 26 | batch 650/2304 [I 2022-10-26 19:36:27] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:24 [I 2022-10-26 19:36:27] ...trap/engines/engine.py.229: process: 0.54985 | load: 0.00246 [I 2022-10-26 19:36:27] ...trap/engines/engine.py.230: loss: 0.18837 [I 2022-10-26 19:36:27] ...optimizers/trijoint.py.151: total_norm: 14.310778 [I 2022-10-26 19:36:33] ...trap/engines/engine.py.224: train: epoch 26 | batch 660/2304 [I 2022-10-26 19:36:33] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:15:18 [I 2022-10-26 19:36:33] ...trap/engines/engine.py.229: process: 0.55309 | load: 0.00243 [I 2022-10-26 19:36:33] ...trap/engines/engine.py.230: loss: 0.19068 [I 2022-10-26 19:36:33] ...optimizers/trijoint.py.151: total_norm: 25.491419 [I 2022-10-26 19:36:39] ...trap/engines/engine.py.224: train: epoch 26 | batch 670/2304 [I 2022-10-26 19:36:39] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:15:12 [I 2022-10-26 19:36:39] ...trap/engines/engine.py.229: process: 0.55847 | load: 0.00242 [I 2022-10-26 19:36:39] ...trap/engines/engine.py.230: loss: 0.13714 [I 2022-10-26 19:36:39] ...optimizers/trijoint.py.151: total_norm: 12.806604 [I 2022-10-26 19:36:44] ...trap/engines/engine.py.224: train: epoch 26 | batch 680/2304 [I 2022-10-26 19:36:44] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:15:02 [I 2022-10-26 19:36:44] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00243 [I 2022-10-26 19:36:44] ...trap/engines/engine.py.230: loss: 0.20732 [I 2022-10-26 19:36:44] ...optimizers/trijoint.py.151: total_norm: 23.772890 [I 2022-10-26 19:36:50] ...trap/engines/engine.py.224: train: epoch 26 | batch 690/2304 [I 2022-10-26 19:36:50] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:58 [I 2022-10-26 19:36:50] ...trap/engines/engine.py.229: process: 0.55834 | load: 0.00250 [I 2022-10-26 19:36:50] ...trap/engines/engine.py.230: loss: 0.13098 [I 2022-10-26 19:36:50] ...optimizers/trijoint.py.151: total_norm: 11.338479 [I 2022-10-26 19:36:55] ...trap/engines/engine.py.224: train: epoch 26 | batch 700/2304 [I 2022-10-26 19:36:55] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:52 [I 2022-10-26 19:36:55] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00239 [I 2022-10-26 19:36:55] ...trap/engines/engine.py.230: loss: 0.17573 [I 2022-10-26 19:36:55] ...optimizers/trijoint.py.151: total_norm: 20.582472 [I 2022-10-26 19:37:01] ...trap/engines/engine.py.224: train: epoch 26 | batch 710/2304 [I 2022-10-26 19:37:01] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:56 [I 2022-10-26 19:37:01] ...trap/engines/engine.py.229: process: 0.55320 | load: 0.00246 [I 2022-10-26 19:37:01] ...trap/engines/engine.py.230: loss: 0.15161 [I 2022-10-26 19:37:01] ...optimizers/trijoint.py.151: total_norm: 22.013071 [I 2022-10-26 19:37:07] ...trap/engines/engine.py.224: train: epoch 26 | batch 720/2304 [I 2022-10-26 19:37:07] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:41 [I 2022-10-26 19:37:07] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00245 [I 2022-10-26 19:37:07] ...trap/engines/engine.py.230: loss: 0.21495 [I 2022-10-26 19:37:07] ...optimizers/trijoint.py.151: total_norm: 15.832088 [I 2022-10-26 19:37:12] ...trap/engines/engine.py.224: train: epoch 26 | batch 730/2304 [I 2022-10-26 19:37:12] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:39 [I 2022-10-26 19:37:12] ...trap/engines/engine.py.229: process: 0.56261 | load: 0.00244 [I 2022-10-26 19:37:12] ...trap/engines/engine.py.230: loss: 0.15046 [I 2022-10-26 19:37:12] ...optimizers/trijoint.py.151: total_norm: 18.358864 [I 2022-10-26 19:37:18] ...trap/engines/engine.py.224: train: epoch 26 | batch 740/2304 [I 2022-10-26 19:37:18] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:31 [I 2022-10-26 19:37:18] ...trap/engines/engine.py.229: process: 0.55280 | load: 0.00261 [I 2022-10-26 19:37:18] ...trap/engines/engine.py.230: loss: 0.15377 [I 2022-10-26 19:37:18] ...optimizers/trijoint.py.151: total_norm: 19.763979 [I 2022-10-26 19:37:23] ...trap/engines/engine.py.224: train: epoch 26 | batch 750/2304 [I 2022-10-26 19:37:23] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:24 [I 2022-10-26 19:37:23] ...trap/engines/engine.py.229: process: 0.56761 | load: 0.00242 [I 2022-10-26 19:37:23] ...trap/engines/engine.py.230: loss: 0.20501 [I 2022-10-26 19:37:23] ...optimizers/trijoint.py.151: total_norm: 12.439609 [I 2022-10-26 19:37:29] ...trap/engines/engine.py.224: train: epoch 26 | batch 760/2304 [I 2022-10-26 19:37:29] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:20 [I 2022-10-26 19:37:29] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00244 [I 2022-10-26 19:37:29] ...trap/engines/engine.py.230: loss: 0.29364 [I 2022-10-26 19:37:29] ...optimizers/trijoint.py.151: total_norm: 37.416107 [I 2022-10-26 19:37:35] ...trap/engines/engine.py.224: train: epoch 26 | batch 770/2304 [I 2022-10-26 19:37:35] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:14:40 [I 2022-10-26 19:37:35] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00242 [I 2022-10-26 19:37:35] ...trap/engines/engine.py.230: loss: 0.15979 [I 2022-10-26 19:37:35] ...optimizers/trijoint.py.151: total_norm: 18.275366 [I 2022-10-26 19:37:40] ...trap/engines/engine.py.224: train: epoch 26 | batch 780/2304 [I 2022-10-26 19:37:40] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:14:12 [I 2022-10-26 19:37:40] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00239 [I 2022-10-26 19:37:40] ...trap/engines/engine.py.230: loss: 0.13945 [I 2022-10-26 19:37:40] ...optimizers/trijoint.py.151: total_norm: 19.947903 [I 2022-10-26 19:37:46] ...trap/engines/engine.py.224: train: epoch 26 | batch 790/2304 [I 2022-10-26 19:37:46] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:14:01 [I 2022-10-26 19:37:46] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00241 [I 2022-10-26 19:37:46] ...trap/engines/engine.py.230: loss: 0.16442 [I 2022-10-26 19:37:46] ...optimizers/trijoint.py.151: total_norm: 37.020153 [I 2022-10-26 19:37:51] ...trap/engines/engine.py.224: train: epoch 26 | batch 800/2304 [I 2022-10-26 19:37:51] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:52 [I 2022-10-26 19:37:51] ...trap/engines/engine.py.229: process: 0.54740 | load: 0.00240 [I 2022-10-26 19:37:51] ...trap/engines/engine.py.230: loss: 0.15433 [I 2022-10-26 19:37:51] ...optimizers/trijoint.py.151: total_norm: 21.511955 [I 2022-10-26 19:37:57] ...trap/engines/engine.py.224: train: epoch 26 | batch 810/2304 [I 2022-10-26 19:37:57] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:46 [I 2022-10-26 19:37:57] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00260 [I 2022-10-26 19:37:57] ...trap/engines/engine.py.230: loss: 0.14880 [I 2022-10-26 19:37:57] ...optimizers/trijoint.py.151: total_norm: 9.465802 [I 2022-10-26 19:38:02] ...trap/engines/engine.py.224: train: epoch 26 | batch 820/2304 [I 2022-10-26 19:38:02] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:45 [I 2022-10-26 19:38:02] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00247 [I 2022-10-26 19:38:02] ...trap/engines/engine.py.230: loss: 0.20467 [I 2022-10-26 19:38:02] ...optimizers/trijoint.py.151: total_norm: 38.519299 [I 2022-10-26 19:38:08] ...trap/engines/engine.py.224: train: epoch 26 | batch 830/2304 [I 2022-10-26 19:38:08] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:39 [I 2022-10-26 19:38:08] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00248 [I 2022-10-26 19:38:08] ...trap/engines/engine.py.230: loss: 0.14963 [I 2022-10-26 19:38:08] ...optimizers/trijoint.py.151: total_norm: 15.413465 [I 2022-10-26 19:38:14] ...trap/engines/engine.py.224: train: epoch 26 | batch 840/2304 [I 2022-10-26 19:38:14] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:38 [I 2022-10-26 19:38:14] ...trap/engines/engine.py.229: process: 0.55211 | load: 0.00247 [I 2022-10-26 19:38:14] ...trap/engines/engine.py.230: loss: 0.25037 [I 2022-10-26 19:38:14] ...optimizers/trijoint.py.151: total_norm: 23.313425 [I 2022-10-26 19:38:19] ...trap/engines/engine.py.224: train: epoch 26 | batch 850/2304 [I 2022-10-26 19:38:19] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:20 [I 2022-10-26 19:38:19] ...trap/engines/engine.py.229: process: 0.54713 | load: 0.00247 [I 2022-10-26 19:38:19] ...trap/engines/engine.py.230: loss: 0.15790 [I 2022-10-26 19:38:19] ...optimizers/trijoint.py.151: total_norm: 17.079969 [I 2022-10-26 19:38:25] ...trap/engines/engine.py.224: train: epoch 26 | batch 860/2304 [I 2022-10-26 19:38:25] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:26 [I 2022-10-26 19:38:25] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00248 [I 2022-10-26 19:38:25] ...trap/engines/engine.py.230: loss: 0.20547 [I 2022-10-26 19:38:25] ...optimizers/trijoint.py.151: total_norm: 21.684324 [I 2022-10-26 19:38:30] ...trap/engines/engine.py.224: train: epoch 26 | batch 870/2304 [I 2022-10-26 19:38:30] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:13:20 [I 2022-10-26 19:38:30] ...trap/engines/engine.py.229: process: 0.54698 | load: 0.00242 [I 2022-10-26 19:38:30] ...trap/engines/engine.py.230: loss: 0.14510 [I 2022-10-26 19:38:30] ...optimizers/trijoint.py.151: total_norm: 31.003578 [I 2022-10-26 19:38:36] ...trap/engines/engine.py.224: train: epoch 26 | batch 880/2304 [I 2022-10-26 19:38:36] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:13:08 [I 2022-10-26 19:38:36] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00242 [I 2022-10-26 19:38:36] ...trap/engines/engine.py.230: loss: 0.14742 [I 2022-10-26 19:38:36] ...optimizers/trijoint.py.151: total_norm: 12.809201 [I 2022-10-26 19:38:41] ...trap/engines/engine.py.224: train: epoch 26 | batch 890/2304 [I 2022-10-26 19:38:41] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:13:05 [I 2022-10-26 19:38:41] ...trap/engines/engine.py.229: process: 0.54573 | load: 0.00248 [I 2022-10-26 19:38:41] ...trap/engines/engine.py.230: loss: 0.27190 [I 2022-10-26 19:38:41] ...optimizers/trijoint.py.151: total_norm: 17.412214 [I 2022-10-26 19:38:47] ...trap/engines/engine.py.224: train: epoch 26 | batch 900/2304 [I 2022-10-26 19:38:47] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:13:24 [I 2022-10-26 19:38:47] ...trap/engines/engine.py.229: process: 0.57040 | load: 0.00274 [I 2022-10-26 19:38:47] ...trap/engines/engine.py.230: loss: 0.24518 [I 2022-10-26 19:38:47] ...optimizers/trijoint.py.151: total_norm: 21.705107 [I 2022-10-26 19:38:53] ...trap/engines/engine.py.224: train: epoch 26 | batch 910/2304 [I 2022-10-26 19:38:53] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:58 [I 2022-10-26 19:38:53] ...trap/engines/engine.py.229: process: 0.56134 | load: 0.00244 [I 2022-10-26 19:38:53] ...trap/engines/engine.py.230: loss: 0.24305 [I 2022-10-26 19:38:53] ...optimizers/trijoint.py.151: total_norm: 30.648935 [I 2022-10-26 19:38:58] ...trap/engines/engine.py.224: train: epoch 26 | batch 920/2304 [I 2022-10-26 19:38:58] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:51 [I 2022-10-26 19:38:58] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00248 [I 2022-10-26 19:38:58] ...trap/engines/engine.py.230: loss: 0.25120 [I 2022-10-26 19:38:58] ...optimizers/trijoint.py.151: total_norm: 25.086218 [I 2022-10-26 19:39:04] ...trap/engines/engine.py.224: train: epoch 26 | batch 930/2304 [I 2022-10-26 19:39:04] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:40 [I 2022-10-26 19:39:04] ...trap/engines/engine.py.229: process: 0.55858 | load: 0.00246 [I 2022-10-26 19:39:04] ...trap/engines/engine.py.230: loss: 0.17150 [I 2022-10-26 19:39:04] ...optimizers/trijoint.py.151: total_norm: 15.572052 [I 2022-10-26 19:39:09] ...trap/engines/engine.py.224: train: epoch 26 | batch 940/2304 [I 2022-10-26 19:39:09] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:36 [I 2022-10-26 19:39:09] ...trap/engines/engine.py.229: process: 0.56720 | load: 0.00243 [I 2022-10-26 19:39:09] ...trap/engines/engine.py.230: loss: 0.16925 [I 2022-10-26 19:39:09] ...optimizers/trijoint.py.151: total_norm: 20.584913 [I 2022-10-26 19:39:15] ...trap/engines/engine.py.224: train: epoch 26 | batch 950/2304 [I 2022-10-26 19:39:15] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:35 [I 2022-10-26 19:39:15] ...trap/engines/engine.py.229: process: 0.56159 | load: 0.00243 [I 2022-10-26 19:39:15] ...trap/engines/engine.py.230: loss: 0.21632 [I 2022-10-26 19:39:15] ...optimizers/trijoint.py.151: total_norm: 17.050406 [I 2022-10-26 19:39:20] ...trap/engines/engine.py.224: train: epoch 26 | batch 960/2304 [I 2022-10-26 19:39:20] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:25 [I 2022-10-26 19:39:20] ...trap/engines/engine.py.229: process: 0.54952 | load: 0.00241 [I 2022-10-26 19:39:20] ...trap/engines/engine.py.230: loss: 0.17493 [I 2022-10-26 19:39:20] ...optimizers/trijoint.py.151: total_norm: 18.724617 [I 2022-10-26 19:39:26] ...trap/engines/engine.py.224: train: epoch 26 | batch 970/2304 [I 2022-10-26 19:39:26] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:31 [I 2022-10-26 19:39:26] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00242 [I 2022-10-26 19:39:26] ...trap/engines/engine.py.230: loss: 0.21741 [I 2022-10-26 19:39:26] ...optimizers/trijoint.py.151: total_norm: 11.116795 [I 2022-10-26 19:39:32] ...trap/engines/engine.py.224: train: epoch 26 | batch 980/2304 [I 2022-10-26 19:39:32] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:12:17 [I 2022-10-26 19:39:32] ...trap/engines/engine.py.229: process: 0.54849 | load: 0.00243 [I 2022-10-26 19:39:32] ...trap/engines/engine.py.230: loss: 0.16182 [I 2022-10-26 19:39:32] ...optimizers/trijoint.py.151: total_norm: 16.545883 [I 2022-10-26 19:39:37] ...trap/engines/engine.py.224: train: epoch 26 | batch 990/2304 [I 2022-10-26 19:39:37] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:12:08 [I 2022-10-26 19:39:37] ...trap/engines/engine.py.229: process: 0.55426 | load: 0.00245 [I 2022-10-26 19:39:37] ...trap/engines/engine.py.230: loss: 0.14970 [I 2022-10-26 19:39:37] ...optimizers/trijoint.py.151: total_norm: 15.056638 [I 2022-10-26 19:39:43] ...trap/engines/engine.py.224: train: epoch 26 | batch 1000/2304 [I 2022-10-26 19:39:43] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:12:06 [I 2022-10-26 19:39:43] ...trap/engines/engine.py.229: process: 0.55880 | load: 0.00249 [I 2022-10-26 19:39:43] ...trap/engines/engine.py.230: loss: 0.15123 [I 2022-10-26 19:39:43] ...optimizers/trijoint.py.151: total_norm: 20.892841 [I 2022-10-26 19:39:49] ...trap/engines/engine.py.224: train: epoch 26 | batch 1010/2304 [I 2022-10-26 19:39:49] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:12:01 [I 2022-10-26 19:39:49] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00248 [I 2022-10-26 19:39:49] ...trap/engines/engine.py.230: loss: 0.22970 [I 2022-10-26 19:39:49] ...optimizers/trijoint.py.151: total_norm: 17.830215 [I 2022-10-26 19:39:54] ...trap/engines/engine.py.224: train: epoch 26 | batch 1020/2304 [I 2022-10-26 19:39:54] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:57 [I 2022-10-26 19:39:54] ...trap/engines/engine.py.229: process: 0.55893 | load: 0.00243 [I 2022-10-26 19:39:54] ...trap/engines/engine.py.230: loss: 0.20974 [I 2022-10-26 19:39:54] ...optimizers/trijoint.py.151: total_norm: 24.364683 [I 2022-10-26 19:40:00] ...trap/engines/engine.py.224: train: epoch 26 | batch 1030/2304 [I 2022-10-26 19:40:00] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:12:00 [I 2022-10-26 19:40:00] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00245 [I 2022-10-26 19:40:00] ...trap/engines/engine.py.230: loss: 0.17978 [I 2022-10-26 19:40:00] ...optimizers/trijoint.py.151: total_norm: 18.801054 [I 2022-10-26 19:40:05] ...trap/engines/engine.py.224: train: epoch 26 | batch 1040/2304 [I 2022-10-26 19:40:05] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:42 [I 2022-10-26 19:40:05] ...trap/engines/engine.py.229: process: 0.55567 | load: 0.00243 [I 2022-10-26 19:40:05] ...trap/engines/engine.py.230: loss: 0.14097 [I 2022-10-26 19:40:05] ...optimizers/trijoint.py.151: total_norm: 15.104679 [I 2022-10-26 19:40:11] ...trap/engines/engine.py.224: train: epoch 26 | batch 1050/2304 [I 2022-10-26 19:40:11] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:33 [I 2022-10-26 19:40:11] ...trap/engines/engine.py.229: process: 0.54835 | load: 0.00244 [I 2022-10-26 19:40:11] ...trap/engines/engine.py.230: loss: 0.16959 [I 2022-10-26 19:40:11] ...optimizers/trijoint.py.151: total_norm: 17.976725 [I 2022-10-26 19:40:16] ...trap/engines/engine.py.224: train: epoch 26 | batch 1060/2304 [I 2022-10-26 19:40:16] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:27 [I 2022-10-26 19:40:16] ...trap/engines/engine.py.229: process: 0.54536 | load: 0.00246 [I 2022-10-26 19:40:16] ...trap/engines/engine.py.230: loss: 0.15884 [I 2022-10-26 19:40:16] ...optimizers/trijoint.py.151: total_norm: 26.680866 [I 2022-10-26 19:40:22] ...trap/engines/engine.py.224: train: epoch 26 | batch 1070/2304 [I 2022-10-26 19:40:22] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:26 [I 2022-10-26 19:40:22] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00291 [I 2022-10-26 19:40:22] ...trap/engines/engine.py.230: loss: 0.18443 [I 2022-10-26 19:40:22] ...optimizers/trijoint.py.151: total_norm: 21.138321 [I 2022-10-26 19:40:28] ...trap/engines/engine.py.224: train: epoch 26 | batch 1080/2304 [I 2022-10-26 19:40:28] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:19 [I 2022-10-26 19:40:28] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00247 [I 2022-10-26 19:40:28] ...trap/engines/engine.py.230: loss: 0.17281 [I 2022-10-26 19:40:28] ...optimizers/trijoint.py.151: total_norm: 15.769908 [I 2022-10-26 19:40:33] ...trap/engines/engine.py.224: train: epoch 26 | batch 1090/2304 [I 2022-10-26 19:40:33] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:11:15 [I 2022-10-26 19:40:33] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00246 [I 2022-10-26 19:40:33] ...trap/engines/engine.py.230: loss: 0.16113 [I 2022-10-26 19:40:33] ...optimizers/trijoint.py.151: total_norm: 29.486382 [I 2022-10-26 19:40:39] ...trap/engines/engine.py.224: train: epoch 26 | batch 1100/2304 [I 2022-10-26 19:40:39] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:11:10 [I 2022-10-26 19:40:39] ...trap/engines/engine.py.229: process: 0.55558 | load: 0.00250 [I 2022-10-26 19:40:39] ...trap/engines/engine.py.230: loss: 0.15765 [I 2022-10-26 19:40:39] ...optimizers/trijoint.py.151: total_norm: 17.568316 [I 2022-10-26 19:40:44] ...trap/engines/engine.py.224: train: epoch 26 | batch 1110/2304 [I 2022-10-26 19:40:44] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:11:06 [I 2022-10-26 19:40:44] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00246 [I 2022-10-26 19:40:44] ...trap/engines/engine.py.230: loss: 0.13339 [I 2022-10-26 19:40:44] ...optimizers/trijoint.py.151: total_norm: 12.786646 [I 2022-10-26 19:40:50] ...trap/engines/engine.py.224: train: epoch 26 | batch 1120/2304 [I 2022-10-26 19:40:50] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:11:00 [I 2022-10-26 19:40:50] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00247 [I 2022-10-26 19:40:50] ...trap/engines/engine.py.230: loss: 0.20845 [I 2022-10-26 19:40:50] ...optimizers/trijoint.py.151: total_norm: 13.953249 [I 2022-10-26 19:40:56] ...trap/engines/engine.py.224: train: epoch 26 | batch 1130/2304 [I 2022-10-26 19:40:56] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:52 [I 2022-10-26 19:40:56] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00246 [I 2022-10-26 19:40:56] ...trap/engines/engine.py.230: loss: 0.14196 [I 2022-10-26 19:40:56] ...optimizers/trijoint.py.151: total_norm: 10.774552 [I 2022-10-26 19:41:01] ...trap/engines/engine.py.224: train: epoch 26 | batch 1140/2304 [I 2022-10-26 19:41:01] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:49 [I 2022-10-26 19:41:01] ...trap/engines/engine.py.229: process: 0.56226 | load: 0.00241 [I 2022-10-26 19:41:01] ...trap/engines/engine.py.230: loss: 0.23646 [I 2022-10-26 19:41:01] ...optimizers/trijoint.py.151: total_norm: 19.422321 [I 2022-10-26 19:41:07] ...trap/engines/engine.py.224: train: epoch 26 | batch 1150/2304 [I 2022-10-26 19:41:07] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:43 [I 2022-10-26 19:41:07] ...trap/engines/engine.py.229: process: 0.54585 | load: 0.00260 [I 2022-10-26 19:41:07] ...trap/engines/engine.py.230: loss: 0.19243 [I 2022-10-26 19:41:07] ...optimizers/trijoint.py.151: total_norm: 14.808278 [I 2022-10-26 19:41:12] ...trap/engines/engine.py.224: train: epoch 26 | batch 1160/2304 [I 2022-10-26 19:41:12] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:34 [I 2022-10-26 19:41:12] ...trap/engines/engine.py.229: process: 0.54444 | load: 0.00242 [I 2022-10-26 19:41:12] ...trap/engines/engine.py.230: loss: 0.13083 [I 2022-10-26 19:41:12] ...optimizers/trijoint.py.151: total_norm: 16.515318 [I 2022-10-26 19:41:18] ...trap/engines/engine.py.224: train: epoch 26 | batch 1170/2304 [I 2022-10-26 19:41:18] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:28 [I 2022-10-26 19:41:18] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00244 [I 2022-10-26 19:41:18] ...trap/engines/engine.py.230: loss: 0.31850 [I 2022-10-26 19:41:18] ...optimizers/trijoint.py.151: total_norm: 33.496780 [I 2022-10-26 19:41:23] ...trap/engines/engine.py.224: train: epoch 26 | batch 1180/2304 [I 2022-10-26 19:41:23] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:23 [I 2022-10-26 19:41:23] ...trap/engines/engine.py.229: process: 0.54446 | load: 0.00241 [I 2022-10-26 19:41:23] ...trap/engines/engine.py.230: loss: 0.15683 [I 2022-10-26 19:41:23] ...optimizers/trijoint.py.151: total_norm: 16.171909 [I 2022-10-26 19:41:29] ...trap/engines/engine.py.224: train: epoch 26 | batch 1190/2304 [I 2022-10-26 19:41:29] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:18 [I 2022-10-26 19:41:29] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00263 [I 2022-10-26 19:41:29] ...trap/engines/engine.py.230: loss: 0.20023 [I 2022-10-26 19:41:29] ...optimizers/trijoint.py.151: total_norm: 22.493530 [I 2022-10-26 19:41:35] ...trap/engines/engine.py.224: train: epoch 26 | batch 1200/2304 [I 2022-10-26 19:41:35] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:10:16 [I 2022-10-26 19:41:35] ...trap/engines/engine.py.229: process: 0.56962 | load: 0.00252 [I 2022-10-26 19:41:35] ...trap/engines/engine.py.230: loss: 0.17965 [I 2022-10-26 19:41:35] ...optimizers/trijoint.py.151: total_norm: 24.349691 [I 2022-10-26 19:41:40] ...trap/engines/engine.py.224: train: epoch 26 | batch 1210/2304 [I 2022-10-26 19:41:40] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:10:07 [I 2022-10-26 19:41:40] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00248 [I 2022-10-26 19:41:40] ...trap/engines/engine.py.230: loss: 0.21649 [I 2022-10-26 19:41:40] ...optimizers/trijoint.py.151: total_norm: 17.299488 [I 2022-10-26 19:41:46] ...trap/engines/engine.py.224: train: epoch 26 | batch 1220/2304 [I 2022-10-26 19:41:46] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:10:05 [I 2022-10-26 19:41:46] ...trap/engines/engine.py.229: process: 0.56119 | load: 0.00242 [I 2022-10-26 19:41:46] ...trap/engines/engine.py.230: loss: 0.20859 [I 2022-10-26 19:41:46] ...optimizers/trijoint.py.151: total_norm: 28.887423 [I 2022-10-26 19:41:51] ...trap/engines/engine.py.224: train: epoch 26 | batch 1230/2304 [I 2022-10-26 19:41:51] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:10:01 [I 2022-10-26 19:41:51] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00243 [I 2022-10-26 19:41:51] ...trap/engines/engine.py.230: loss: 0.21946 [I 2022-10-26 19:41:51] ...optimizers/trijoint.py.151: total_norm: 12.457106 [I 2022-10-26 19:41:57] ...trap/engines/engine.py.224: train: epoch 26 | batch 1240/2304 [I 2022-10-26 19:41:57] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:53 [I 2022-10-26 19:41:57] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00244 [I 2022-10-26 19:41:57] ...trap/engines/engine.py.230: loss: 0.17734 [I 2022-10-26 19:41:57] ...optimizers/trijoint.py.151: total_norm: 18.687712 [I 2022-10-26 19:42:03] ...trap/engines/engine.py.224: train: epoch 26 | batch 1250/2304 [I 2022-10-26 19:42:03] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:44 [I 2022-10-26 19:42:03] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00245 [I 2022-10-26 19:42:03] ...trap/engines/engine.py.230: loss: 0.13588 [I 2022-10-26 19:42:03] ...optimizers/trijoint.py.151: total_norm: 15.595485 [I 2022-10-26 19:42:08] ...trap/engines/engine.py.224: train: epoch 26 | batch 1260/2304 [I 2022-10-26 19:42:08] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:41 [I 2022-10-26 19:42:08] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00245 [I 2022-10-26 19:42:08] ...trap/engines/engine.py.230: loss: 0.18763 [I 2022-10-26 19:42:08] ...optimizers/trijoint.py.151: total_norm: 20.306252 [I 2022-10-26 19:42:14] ...trap/engines/engine.py.224: train: epoch 26 | batch 1270/2304 [I 2022-10-26 19:42:14] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:36 [I 2022-10-26 19:42:14] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00250 [I 2022-10-26 19:42:14] ...trap/engines/engine.py.230: loss: 0.19445 [I 2022-10-26 19:42:14] ...optimizers/trijoint.py.151: total_norm: 44.364582 [I 2022-10-26 19:42:19] ...trap/engines/engine.py.224: train: epoch 26 | batch 1280/2304 [I 2022-10-26 19:42:19] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:33 [I 2022-10-26 19:42:19] ...trap/engines/engine.py.229: process: 0.56863 | load: 0.00286 [I 2022-10-26 19:42:19] ...trap/engines/engine.py.230: loss: 0.15744 [I 2022-10-26 19:42:19] ...optimizers/trijoint.py.151: total_norm: 13.986426 [I 2022-10-26 19:42:25] ...trap/engines/engine.py.224: train: epoch 26 | batch 1290/2304 [I 2022-10-26 19:42:25] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:25 [I 2022-10-26 19:42:25] ...trap/engines/engine.py.229: process: 0.56226 | load: 0.00325 [I 2022-10-26 19:42:25] ...trap/engines/engine.py.230: loss: 0.23416 [I 2022-10-26 19:42:25] ...optimizers/trijoint.py.151: total_norm: 20.225624 [I 2022-10-26 19:42:30] ...trap/engines/engine.py.224: train: epoch 26 | batch 1300/2304 [I 2022-10-26 19:42:30] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:09:18 [I 2022-10-26 19:42:30] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00247 [I 2022-10-26 19:42:30] ...trap/engines/engine.py.230: loss: 0.13739 [I 2022-10-26 19:42:30] ...optimizers/trijoint.py.151: total_norm: 12.785431 [I 2022-10-26 19:42:36] ...trap/engines/engine.py.224: train: epoch 26 | batch 1310/2304 [I 2022-10-26 19:42:36] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:09:11 [I 2022-10-26 19:42:36] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00243 [I 2022-10-26 19:42:36] ...trap/engines/engine.py.230: loss: 0.21432 [I 2022-10-26 19:42:36] ...optimizers/trijoint.py.151: total_norm: 14.121473 [I 2022-10-26 19:42:42] ...trap/engines/engine.py.224: train: epoch 26 | batch 1320/2304 [I 2022-10-26 19:42:42] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:09:09 [I 2022-10-26 19:42:42] ...trap/engines/engine.py.229: process: 0.56036 | load: 0.00245 [I 2022-10-26 19:42:42] ...trap/engines/engine.py.230: loss: 0.16775 [I 2022-10-26 19:42:42] ...optimizers/trijoint.py.151: total_norm: 13.770103 [I 2022-10-26 19:42:47] ...trap/engines/engine.py.224: train: epoch 26 | batch 1330/2304 [I 2022-10-26 19:42:47] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:09:05 [I 2022-10-26 19:42:47] ...trap/engines/engine.py.229: process: 0.56162 | load: 0.00243 [I 2022-10-26 19:42:47] ...trap/engines/engine.py.230: loss: 0.29502 [I 2022-10-26 19:42:47] ...optimizers/trijoint.py.151: total_norm: 13.593475 [I 2022-10-26 19:42:53] ...trap/engines/engine.py.224: train: epoch 26 | batch 1340/2304 [I 2022-10-26 19:42:53] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:55 [I 2022-10-26 19:42:53] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00243 [I 2022-10-26 19:42:53] ...trap/engines/engine.py.230: loss: 0.19719 [I 2022-10-26 19:42:53] ...optimizers/trijoint.py.151: total_norm: 30.104565 [I 2022-10-26 19:42:58] ...trap/engines/engine.py.224: train: epoch 26 | batch 1350/2304 [I 2022-10-26 19:42:58] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:50 [I 2022-10-26 19:42:58] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00240 [I 2022-10-26 19:42:58] ...trap/engines/engine.py.230: loss: 0.17918 [I 2022-10-26 19:42:58] ...optimizers/trijoint.py.151: total_norm: 27.244646 [I 2022-10-26 19:43:04] ...trap/engines/engine.py.224: train: epoch 26 | batch 1360/2304 [I 2022-10-26 19:43:04] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:46 [I 2022-10-26 19:43:04] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00244 [I 2022-10-26 19:43:04] ...trap/engines/engine.py.230: loss: 0.13895 [I 2022-10-26 19:43:04] ...optimizers/trijoint.py.151: total_norm: 15.195827 [I 2022-10-26 19:43:09] ...trap/engines/engine.py.224: train: epoch 26 | batch 1370/2304 [I 2022-10-26 19:43:09] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:34 [I 2022-10-26 19:43:09] ...trap/engines/engine.py.229: process: 0.54517 | load: 0.00251 [I 2022-10-26 19:43:09] ...trap/engines/engine.py.230: loss: 0.21535 [I 2022-10-26 19:43:09] ...optimizers/trijoint.py.151: total_norm: 20.633492 [I 2022-10-26 19:43:15] ...trap/engines/engine.py.224: train: epoch 26 | batch 1380/2304 [I 2022-10-26 19:43:15] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:36 [I 2022-10-26 19:43:15] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00243 [I 2022-10-26 19:43:15] ...trap/engines/engine.py.230: loss: 0.31451 [I 2022-10-26 19:43:15] ...optimizers/trijoint.py.151: total_norm: 21.751926 [I 2022-10-26 19:43:21] ...trap/engines/engine.py.224: train: epoch 26 | batch 1390/2304 [I 2022-10-26 19:43:21] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:27 [I 2022-10-26 19:43:21] ...trap/engines/engine.py.229: process: 0.55942 | load: 0.00244 [I 2022-10-26 19:43:21] ...trap/engines/engine.py.230: loss: 0.15000 [I 2022-10-26 19:43:21] ...optimizers/trijoint.py.151: total_norm: 13.036744 [I 2022-10-26 19:43:26] ...trap/engines/engine.py.224: train: epoch 26 | batch 1400/2304 [I 2022-10-26 19:43:26] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:23 [I 2022-10-26 19:43:26] ...trap/engines/engine.py.229: process: 0.56116 | load: 0.00242 [I 2022-10-26 19:43:26] ...trap/engines/engine.py.230: loss: 0.21253 [I 2022-10-26 19:43:26] ...optimizers/trijoint.py.151: total_norm: 16.823412 [I 2022-10-26 19:43:32] ...trap/engines/engine.py.224: train: epoch 26 | batch 1410/2304 [I 2022-10-26 19:43:32] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:08:18 [I 2022-10-26 19:43:32] ...trap/engines/engine.py.229: process: 0.56393 | load: 0.00241 [I 2022-10-26 19:43:32] ...trap/engines/engine.py.230: loss: 0.18565 [I 2022-10-26 19:43:32] ...optimizers/trijoint.py.151: total_norm: 15.364152 [I 2022-10-26 19:43:37] ...trap/engines/engine.py.224: train: epoch 26 | batch 1420/2304 [I 2022-10-26 19:43:37] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:08:12 [I 2022-10-26 19:43:37] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00244 [I 2022-10-26 19:43:37] ...trap/engines/engine.py.230: loss: 0.17473 [I 2022-10-26 19:43:37] ...optimizers/trijoint.py.151: total_norm: 16.734428 [I 2022-10-26 19:43:43] ...trap/engines/engine.py.224: train: epoch 26 | batch 1430/2304 [I 2022-10-26 19:43:43] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:08:06 [I 2022-10-26 19:43:43] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00244 [I 2022-10-26 19:43:43] ...trap/engines/engine.py.230: loss: 0.21392 [I 2022-10-26 19:43:43] ...optimizers/trijoint.py.151: total_norm: 17.576778 [I 2022-10-26 19:43:49] ...trap/engines/engine.py.224: train: epoch 26 | batch 1440/2304 [I 2022-10-26 19:43:49] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:08:01 [I 2022-10-26 19:43:49] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00246 [I 2022-10-26 19:43:49] ...trap/engines/engine.py.230: loss: 0.19384 [I 2022-10-26 19:43:49] ...optimizers/trijoint.py.151: total_norm: 18.584604 [I 2022-10-26 19:43:54] ...trap/engines/engine.py.224: train: epoch 26 | batch 1450/2304 [I 2022-10-26 19:43:54] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:08:00 [I 2022-10-26 19:43:54] ...trap/engines/engine.py.229: process: 0.57096 | load: 0.00249 [I 2022-10-26 19:43:54] ...trap/engines/engine.py.230: loss: 0.15830 [I 2022-10-26 19:43:54] ...optimizers/trijoint.py.151: total_norm: 20.625572 [I 2022-10-26 19:44:00] ...trap/engines/engine.py.224: train: epoch 26 | batch 1460/2304 [I 2022-10-26 19:44:00] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:51 [I 2022-10-26 19:44:00] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00248 [I 2022-10-26 19:44:00] ...trap/engines/engine.py.230: loss: 0.25917 [I 2022-10-26 19:44:00] ...optimizers/trijoint.py.151: total_norm: 30.754450 [I 2022-10-26 19:44:05] ...trap/engines/engine.py.224: train: epoch 26 | batch 1470/2304 [I 2022-10-26 19:44:05] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:43 [I 2022-10-26 19:44:05] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00246 [I 2022-10-26 19:44:05] ...trap/engines/engine.py.230: loss: 0.17261 [I 2022-10-26 19:44:05] ...optimizers/trijoint.py.151: total_norm: 19.067036 [I 2022-10-26 19:44:11] ...trap/engines/engine.py.224: train: epoch 26 | batch 1480/2304 [I 2022-10-26 19:44:11] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:40 [I 2022-10-26 19:44:11] ...trap/engines/engine.py.229: process: 0.56300 | load: 0.00246 [I 2022-10-26 19:44:11] ...trap/engines/engine.py.230: loss: 0.13709 [I 2022-10-26 19:44:11] ...optimizers/trijoint.py.151: total_norm: 21.440048 [I 2022-10-26 19:44:17] ...trap/engines/engine.py.224: train: epoch 26 | batch 1490/2304 [I 2022-10-26 19:44:17] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:33 [I 2022-10-26 19:44:17] ...trap/engines/engine.py.229: process: 0.56318 | load: 0.00245 [I 2022-10-26 19:44:17] ...trap/engines/engine.py.230: loss: 0.19212 [I 2022-10-26 19:44:17] ...optimizers/trijoint.py.151: total_norm: 17.221626 [I 2022-10-26 19:44:22] ...trap/engines/engine.py.224: train: epoch 26 | batch 1500/2304 [I 2022-10-26 19:44:22] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:27 [I 2022-10-26 19:44:22] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00244 [I 2022-10-26 19:44:22] ...trap/engines/engine.py.230: loss: 0.21021 [I 2022-10-26 19:44:22] ...optimizers/trijoint.py.151: total_norm: 29.958576 [I 2022-10-26 19:44:28] ...trap/engines/engine.py.224: train: epoch 26 | batch 1510/2304 [I 2022-10-26 19:44:28] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:21 [I 2022-10-26 19:44:28] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00243 [I 2022-10-26 19:44:28] ...trap/engines/engine.py.230: loss: 0.16152 [I 2022-10-26 19:44:28] ...optimizers/trijoint.py.151: total_norm: 21.380276 [I 2022-10-26 19:44:33] ...trap/engines/engine.py.224: train: epoch 26 | batch 1520/2304 [I 2022-10-26 19:44:33] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:07:15 [I 2022-10-26 19:44:33] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00241 [I 2022-10-26 19:44:33] ...trap/engines/engine.py.230: loss: 0.14538 [I 2022-10-26 19:44:33] ...optimizers/trijoint.py.151: total_norm: 16.414110 [I 2022-10-26 19:44:39] ...trap/engines/engine.py.224: train: epoch 26 | batch 1530/2304 [I 2022-10-26 19:44:39] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:07:09 [I 2022-10-26 19:44:39] ...trap/engines/engine.py.229: process: 0.53718 | load: 0.00241 [I 2022-10-26 19:44:39] ...trap/engines/engine.py.230: loss: 0.19056 [I 2022-10-26 19:44:39] ...optimizers/trijoint.py.151: total_norm: 22.668825 [I 2022-10-26 19:44:44] ...trap/engines/engine.py.224: train: epoch 26 | batch 1540/2304 [I 2022-10-26 19:44:44] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:07:05 [I 2022-10-26 19:44:44] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00243 [I 2022-10-26 19:44:44] ...trap/engines/engine.py.230: loss: 0.16386 [I 2022-10-26 19:44:44] ...optimizers/trijoint.py.151: total_norm: 27.484262 [I 2022-10-26 19:44:50] ...trap/engines/engine.py.224: train: epoch 26 | batch 1550/2304 [I 2022-10-26 19:44:50] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:56 [I 2022-10-26 19:44:50] ...trap/engines/engine.py.229: process: 0.54854 | load: 0.00250 [I 2022-10-26 19:44:50] ...trap/engines/engine.py.230: loss: 0.26577 [I 2022-10-26 19:44:50] ...optimizers/trijoint.py.151: total_norm: 23.306002 [I 2022-10-26 19:44:56] ...trap/engines/engine.py.224: train: epoch 26 | batch 1560/2304 [I 2022-10-26 19:44:56] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:48 [I 2022-10-26 19:44:56] ...trap/engines/engine.py.229: process: 0.54125 | load: 0.00246 [I 2022-10-26 19:44:56] ...trap/engines/engine.py.230: loss: 0.23280 [I 2022-10-26 19:44:56] ...optimizers/trijoint.py.151: total_norm: 16.375895 [I 2022-10-26 19:45:01] ...trap/engines/engine.py.224: train: epoch 26 | batch 1570/2304 [I 2022-10-26 19:45:01] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:46 [I 2022-10-26 19:45:01] ...trap/engines/engine.py.229: process: 0.54366 | load: 0.00245 [I 2022-10-26 19:45:01] ...trap/engines/engine.py.230: loss: 0.24149 [I 2022-10-26 19:45:01] ...optimizers/trijoint.py.151: total_norm: 24.807922 [I 2022-10-26 19:45:07] ...trap/engines/engine.py.224: train: epoch 26 | batch 1580/2304 [I 2022-10-26 19:45:07] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:45 [I 2022-10-26 19:45:07] ...trap/engines/engine.py.229: process: 0.56847 | load: 0.00243 [I 2022-10-26 19:45:07] ...trap/engines/engine.py.230: loss: 0.21221 [I 2022-10-26 19:45:07] ...optimizers/trijoint.py.151: total_norm: 35.702591 [I 2022-10-26 19:45:12] ...trap/engines/engine.py.224: train: epoch 26 | batch 1590/2304 [I 2022-10-26 19:45:12] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:36 [I 2022-10-26 19:45:12] ...trap/engines/engine.py.229: process: 0.54007 | load: 0.00241 [I 2022-10-26 19:45:12] ...trap/engines/engine.py.230: loss: 0.25262 [I 2022-10-26 19:45:12] ...optimizers/trijoint.py.151: total_norm: 17.658142 [I 2022-10-26 19:45:18] ...trap/engines/engine.py.224: train: epoch 26 | batch 1600/2304 [I 2022-10-26 19:45:18] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:32 [I 2022-10-26 19:45:18] ...trap/engines/engine.py.229: process: 0.56263 | load: 0.00245 [I 2022-10-26 19:45:18] ...trap/engines/engine.py.230: loss: 0.22610 [I 2022-10-26 19:45:18] ...optimizers/trijoint.py.151: total_norm: 15.175347 [I 2022-10-26 19:45:24] ...trap/engines/engine.py.224: train: epoch 26 | batch 1610/2304 [I 2022-10-26 19:45:24] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:34 [I 2022-10-26 19:45:24] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00240 [I 2022-10-26 19:45:24] ...trap/engines/engine.py.230: loss: 0.17112 [I 2022-10-26 19:45:24] ...optimizers/trijoint.py.151: total_norm: 19.704493 [I 2022-10-26 19:45:29] ...trap/engines/engine.py.224: train: epoch 26 | batch 1620/2304 [I 2022-10-26 19:45:29] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:23 [I 2022-10-26 19:45:29] ...trap/engines/engine.py.229: process: 0.55847 | load: 0.00243 [I 2022-10-26 19:45:29] ...trap/engines/engine.py.230: loss: 0.15105 [I 2022-10-26 19:45:29] ...optimizers/trijoint.py.151: total_norm: 16.140385 [I 2022-10-26 19:45:35] ...trap/engines/engine.py.224: train: epoch 26 | batch 1630/2304 [I 2022-10-26 19:45:35] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:22 [I 2022-10-26 19:45:35] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00245 [I 2022-10-26 19:45:35] ...trap/engines/engine.py.230: loss: 0.13842 [I 2022-10-26 19:45:35] ...optimizers/trijoint.py.151: total_norm: 16.875174 [I 2022-10-26 19:45:40] ...trap/engines/engine.py.224: train: epoch 26 | batch 1640/2304 [I 2022-10-26 19:45:40] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:06:09 [I 2022-10-26 19:45:40] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00245 [I 2022-10-26 19:45:40] ...trap/engines/engine.py.230: loss: 0.20469 [I 2022-10-26 19:45:40] ...optimizers/trijoint.py.151: total_norm: 17.862526 [I 2022-10-26 19:45:46] ...trap/engines/engine.py.224: train: epoch 26 | batch 1650/2304 [I 2022-10-26 19:45:46] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:06:04 [I 2022-10-26 19:45:46] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00245 [I 2022-10-26 19:45:46] ...trap/engines/engine.py.230: loss: 0.21425 [I 2022-10-26 19:45:46] ...optimizers/trijoint.py.151: total_norm: 13.465076 [I 2022-10-26 19:45:52] ...trap/engines/engine.py.224: train: epoch 26 | batch 1660/2304 [I 2022-10-26 19:45:52] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:57 [I 2022-10-26 19:45:52] ...trap/engines/engine.py.229: process: 0.55097 | load: 0.00247 [I 2022-10-26 19:45:52] ...trap/engines/engine.py.230: loss: 0.14867 [I 2022-10-26 19:45:52] ...optimizers/trijoint.py.151: total_norm: 19.174608 [I 2022-10-26 19:45:57] ...trap/engines/engine.py.224: train: epoch 26 | batch 1670/2304 [I 2022-10-26 19:45:57] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:51 [I 2022-10-26 19:45:57] ...trap/engines/engine.py.229: process: 0.54623 | load: 0.00312 [I 2022-10-26 19:45:57] ...trap/engines/engine.py.230: loss: 0.11879 [I 2022-10-26 19:45:57] ...optimizers/trijoint.py.151: total_norm: 13.739184 [I 2022-10-26 19:46:03] ...trap/engines/engine.py.224: train: epoch 26 | batch 1680/2304 [I 2022-10-26 19:46:03] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:44 [I 2022-10-26 19:46:03] ...trap/engines/engine.py.229: process: 0.54986 | load: 0.00245 [I 2022-10-26 19:46:03] ...trap/engines/engine.py.230: loss: 0.26922 [I 2022-10-26 19:46:03] ...optimizers/trijoint.py.151: total_norm: 20.857040 [I 2022-10-26 19:46:08] ...trap/engines/engine.py.224: train: epoch 26 | batch 1690/2304 [I 2022-10-26 19:46:08] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:44 [I 2022-10-26 19:46:08] ...trap/engines/engine.py.229: process: 0.56278 | load: 0.00242 [I 2022-10-26 19:46:08] ...trap/engines/engine.py.230: loss: 0.20986 [I 2022-10-26 19:46:08] ...optimizers/trijoint.py.151: total_norm: 21.314323 [I 2022-10-26 19:46:14] ...trap/engines/engine.py.224: train: epoch 26 | batch 1700/2304 [I 2022-10-26 19:46:14] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:43 [I 2022-10-26 19:46:14] ...trap/engines/engine.py.229: process: 0.56496 | load: 0.00241 [I 2022-10-26 19:46:14] ...trap/engines/engine.py.230: loss: 0.18609 [I 2022-10-26 19:46:14] ...optimizers/trijoint.py.151: total_norm: 22.760897 [I 2022-10-26 19:46:20] ...trap/engines/engine.py.224: train: epoch 26 | batch 1710/2304 [I 2022-10-26 19:46:20] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:34 [I 2022-10-26 19:46:20] ...trap/engines/engine.py.229: process: 0.56462 | load: 0.00246 [I 2022-10-26 19:46:20] ...trap/engines/engine.py.230: loss: 0.19940 [I 2022-10-26 19:46:20] ...optimizers/trijoint.py.151: total_norm: 24.748470 [I 2022-10-26 19:46:25] ...trap/engines/engine.py.224: train: epoch 26 | batch 1720/2304 [I 2022-10-26 19:46:25] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:26 [I 2022-10-26 19:46:25] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00242 [I 2022-10-26 19:46:25] ...trap/engines/engine.py.230: loss: 0.15552 [I 2022-10-26 19:46:25] ...optimizers/trijoint.py.151: total_norm: 45.774357 [I 2022-10-26 19:46:31] ...trap/engines/engine.py.224: train: epoch 26 | batch 1730/2304 [I 2022-10-26 19:46:31] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:18 [I 2022-10-26 19:46:31] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00246 [I 2022-10-26 19:46:31] ...trap/engines/engine.py.230: loss: 0.23723 [I 2022-10-26 19:46:31] ...optimizers/trijoint.py.151: total_norm: 19.640827 [I 2022-10-26 19:46:36] ...trap/engines/engine.py.224: train: epoch 26 | batch 1740/2304 [I 2022-10-26 19:46:36] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:05:15 [I 2022-10-26 19:46:36] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00243 [I 2022-10-26 19:46:36] ...trap/engines/engine.py.230: loss: 0.22867 [I 2022-10-26 19:46:36] ...optimizers/trijoint.py.151: total_norm: 17.662630 [I 2022-10-26 19:46:42] ...trap/engines/engine.py.224: train: epoch 26 | batch 1750/2304 [I 2022-10-26 19:46:42] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:10 [I 2022-10-26 19:46:42] ...trap/engines/engine.py.229: process: 0.56787 | load: 0.00242 [I 2022-10-26 19:46:42] ...trap/engines/engine.py.230: loss: 0.23191 [I 2022-10-26 19:46:42] ...optimizers/trijoint.py.151: total_norm: 13.986584 [I 2022-10-26 19:46:48] ...trap/engines/engine.py.224: train: epoch 26 | batch 1760/2304 [I 2022-10-26 19:46:48] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:05:11 [I 2022-10-26 19:46:48] ...trap/engines/engine.py.229: process: 0.63793 | load: 0.00251 [I 2022-10-26 19:46:48] ...trap/engines/engine.py.230: loss: 0.18313 [I 2022-10-26 19:46:48] ...optimizers/trijoint.py.151: total_norm: 19.981749 [I 2022-10-26 19:46:53] ...trap/engines/engine.py.224: train: epoch 26 | batch 1770/2304 [I 2022-10-26 19:46:53] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:57 [I 2022-10-26 19:46:53] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00245 [I 2022-10-26 19:46:53] ...trap/engines/engine.py.230: loss: 0.21902 [I 2022-10-26 19:46:53] ...optimizers/trijoint.py.151: total_norm: 28.892305 [I 2022-10-26 19:46:59] ...trap/engines/engine.py.224: train: epoch 26 | batch 1780/2304 [I 2022-10-26 19:46:59] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:53 [I 2022-10-26 19:46:59] ...trap/engines/engine.py.229: process: 0.55363 | load: 0.00243 [I 2022-10-26 19:46:59] ...trap/engines/engine.py.230: loss: 0.13600 [I 2022-10-26 19:46:59] ...optimizers/trijoint.py.151: total_norm: 26.605196 [I 2022-10-26 19:47:05] ...trap/engines/engine.py.224: train: epoch 26 | batch 1790/2304 [I 2022-10-26 19:47:05] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:47 [I 2022-10-26 19:47:05] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00243 [I 2022-10-26 19:47:05] ...trap/engines/engine.py.230: loss: 0.21681 [I 2022-10-26 19:47:05] ...optimizers/trijoint.py.151: total_norm: 21.683243 [I 2022-10-26 19:47:10] ...trap/engines/engine.py.224: train: epoch 26 | batch 1800/2304 [I 2022-10-26 19:47:10] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:40 [I 2022-10-26 19:47:10] ...trap/engines/engine.py.229: process: 0.55444 | load: 0.00244 [I 2022-10-26 19:47:10] ...trap/engines/engine.py.230: loss: 0.15252 [I 2022-10-26 19:47:10] ...optimizers/trijoint.py.151: total_norm: 17.691086 [I 2022-10-26 19:47:16] ...trap/engines/engine.py.224: train: epoch 26 | batch 1810/2304 [I 2022-10-26 19:47:16] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:33 [I 2022-10-26 19:47:16] ...trap/engines/engine.py.229: process: 0.55058 | load: 0.00243 [I 2022-10-26 19:47:16] ...trap/engines/engine.py.230: loss: 0.22162 [I 2022-10-26 19:47:16] ...optimizers/trijoint.py.151: total_norm: 31.564226 [I 2022-10-26 19:47:21] ...trap/engines/engine.py.224: train: epoch 26 | batch 1820/2304 [I 2022-10-26 19:47:21] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:27 [I 2022-10-26 19:47:21] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00243 [I 2022-10-26 19:47:21] ...trap/engines/engine.py.230: loss: 0.18548 [I 2022-10-26 19:47:21] ...optimizers/trijoint.py.151: total_norm: 20.416046 [I 2022-10-26 19:47:27] ...trap/engines/engine.py.224: train: epoch 26 | batch 1830/2304 [I 2022-10-26 19:47:27] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:23 [I 2022-10-26 19:47:27] ...trap/engines/engine.py.229: process: 0.55044 | load: 0.00243 [I 2022-10-26 19:47:27] ...trap/engines/engine.py.230: loss: 0.21457 [I 2022-10-26 19:47:27] ...optimizers/trijoint.py.151: total_norm: 21.633747 [I 2022-10-26 19:47:32] ...trap/engines/engine.py.224: train: epoch 26 | batch 1840/2304 [I 2022-10-26 19:47:32] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:04:18 [I 2022-10-26 19:47:32] ...trap/engines/engine.py.229: process: 0.55822 | load: 0.00240 [I 2022-10-26 19:47:32] ...trap/engines/engine.py.230: loss: 0.21719 [I 2022-10-26 19:47:32] ...optimizers/trijoint.py.151: total_norm: 17.537775 [I 2022-10-26 19:47:38] ...trap/engines/engine.py.224: train: epoch 26 | batch 1850/2304 [I 2022-10-26 19:47:38] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:04:13 [I 2022-10-26 19:47:38] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00253 [I 2022-10-26 19:47:38] ...trap/engines/engine.py.230: loss: 0.15228 [I 2022-10-26 19:47:38] ...optimizers/trijoint.py.151: total_norm: 14.895889 [I 2022-10-26 19:47:44] ...trap/engines/engine.py.224: train: epoch 26 | batch 1860/2304 [I 2022-10-26 19:47:44] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:04:13 [I 2022-10-26 19:47:44] ...trap/engines/engine.py.229: process: 0.62341 | load: 0.00246 [I 2022-10-26 19:47:44] ...trap/engines/engine.py.230: loss: 0.27364 [I 2022-10-26 19:47:44] ...optimizers/trijoint.py.151: total_norm: 20.980820 [I 2022-10-26 19:47:49] ...trap/engines/engine.py.224: train: epoch 26 | batch 1870/2304 [I 2022-10-26 19:47:49] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:04:02 [I 2022-10-26 19:47:49] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00245 [I 2022-10-26 19:47:49] ...trap/engines/engine.py.230: loss: 0.22799 [I 2022-10-26 19:47:49] ...optimizers/trijoint.py.151: total_norm: 19.333328 [I 2022-10-26 19:47:55] ...trap/engines/engine.py.224: train: epoch 26 | batch 1880/2304 [I 2022-10-26 19:47:55] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:56 [I 2022-10-26 19:47:55] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00285 [I 2022-10-26 19:47:55] ...trap/engines/engine.py.230: loss: 0.16123 [I 2022-10-26 19:47:55] ...optimizers/trijoint.py.151: total_norm: 28.387339 [I 2022-10-26 19:48:00] ...trap/engines/engine.py.224: train: epoch 26 | batch 1890/2304 [I 2022-10-26 19:48:00] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:51 [I 2022-10-26 19:48:00] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00241 [I 2022-10-26 19:48:00] ...trap/engines/engine.py.230: loss: 0.16916 [I 2022-10-26 19:48:00] ...optimizers/trijoint.py.151: total_norm: 19.284771 [I 2022-10-26 19:48:23] ...trap/engines/engine.py.224: train: epoch 26 | batch 1900/2304 [I 2022-10-26 19:48:23] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:07:38 [I 2022-10-26 19:48:23] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00237 [I 2022-10-26 19:48:23] ...trap/engines/engine.py.230: loss: 0.20501 [I 2022-10-26 19:48:23] ...optimizers/trijoint.py.151: total_norm: 41.788677 [I 2022-10-26 19:48:29] ...trap/engines/engine.py.224: train: epoch 26 | batch 1910/2304 [I 2022-10-26 19:48:29] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:04:03 [I 2022-10-26 19:48:29] ...trap/engines/engine.py.229: process: 0.54905 | load: 0.00238 [I 2022-10-26 19:48:29] ...trap/engines/engine.py.230: loss: 0.17017 [I 2022-10-26 19:48:29] ...optimizers/trijoint.py.151: total_norm: 40.324951 [I 2022-10-26 19:48:34] ...trap/engines/engine.py.224: train: epoch 26 | batch 1920/2304 [I 2022-10-26 19:48:34] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:37 [I 2022-10-26 19:48:34] ...trap/engines/engine.py.229: process: 0.55257 | load: 0.00248 [I 2022-10-26 19:48:34] ...trap/engines/engine.py.230: loss: 0.13930 [I 2022-10-26 19:48:34] ...optimizers/trijoint.py.151: total_norm: 13.322232 [I 2022-10-26 19:48:40] ...trap/engines/engine.py.224: train: epoch 26 | batch 1930/2304 [I 2022-10-26 19:48:40] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:03:28 [I 2022-10-26 19:48:40] ...trap/engines/engine.py.229: process: 0.55421 | load: 0.00243 [I 2022-10-26 19:48:40] ...trap/engines/engine.py.230: loss: 0.23120 [I 2022-10-26 19:48:40] ...optimizers/trijoint.py.151: total_norm: 23.011272 [I 2022-10-26 19:48:46] ...trap/engines/engine.py.224: train: epoch 26 | batch 1940/2304 [I 2022-10-26 19:48:46] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:03:23 [I 2022-10-26 19:48:46] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00242 [I 2022-10-26 19:48:46] ...trap/engines/engine.py.230: loss: 0.19574 [I 2022-10-26 19:48:46] ...optimizers/trijoint.py.151: total_norm: 16.155970 [I 2022-10-26 19:48:51] ...trap/engines/engine.py.224: train: epoch 26 | batch 1950/2304 [I 2022-10-26 19:48:51] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:03:19 [I 2022-10-26 19:48:51] ...trap/engines/engine.py.229: process: 0.54749 | load: 0.00240 [I 2022-10-26 19:48:51] ...trap/engines/engine.py.230: loss: 0.15228 [I 2022-10-26 19:48:51] ...optimizers/trijoint.py.151: total_norm: 14.677400 [I 2022-10-26 19:48:57] ...trap/engines/engine.py.224: train: epoch 26 | batch 1960/2304 [I 2022-10-26 19:48:57] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:03:12 [I 2022-10-26 19:48:57] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00237 [I 2022-10-26 19:48:57] ...trap/engines/engine.py.230: loss: 0.22927 [I 2022-10-26 19:48:57] ...optimizers/trijoint.py.151: total_norm: 34.738892 [I 2022-10-26 19:49:02] ...trap/engines/engine.py.224: train: epoch 26 | batch 1970/2304 [I 2022-10-26 19:49:02] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:03:05 [I 2022-10-26 19:49:02] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00242 [I 2022-10-26 19:49:02] ...trap/engines/engine.py.230: loss: 0.19198 [I 2022-10-26 19:49:02] ...optimizers/trijoint.py.151: total_norm: 18.244215 [I 2022-10-26 19:49:08] ...trap/engines/engine.py.224: train: epoch 26 | batch 1980/2304 [I 2022-10-26 19:49:08] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:03:00 [I 2022-10-26 19:49:08] ...trap/engines/engine.py.229: process: 0.53667 | load: 0.00244 [I 2022-10-26 19:49:08] ...trap/engines/engine.py.230: loss: 0.13605 [I 2022-10-26 19:49:08] ...optimizers/trijoint.py.151: total_norm: 7.554904 [I 2022-10-26 19:49:14] ...trap/engines/engine.py.224: train: epoch 26 | batch 1990/2304 [I 2022-10-26 19:49:14] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:54 [I 2022-10-26 19:49:14] ...trap/engines/engine.py.229: process: 0.55160 | load: 0.00244 [I 2022-10-26 19:49:14] ...trap/engines/engine.py.230: loss: 0.11688 [I 2022-10-26 19:49:14] ...optimizers/trijoint.py.151: total_norm: 20.895174 [I 2022-10-26 19:49:19] ...trap/engines/engine.py.224: train: epoch 26 | batch 2000/2304 [I 2022-10-26 19:49:19] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:48 [I 2022-10-26 19:49:19] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00243 [I 2022-10-26 19:49:19] ...trap/engines/engine.py.230: loss: 0.18339 [I 2022-10-26 19:49:19] ...optimizers/trijoint.py.151: total_norm: 23.196571 [I 2022-10-26 19:49:25] ...trap/engines/engine.py.224: train: epoch 26 | batch 2010/2304 [I 2022-10-26 19:49:25] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:44 [I 2022-10-26 19:49:25] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00243 [I 2022-10-26 19:49:25] ...trap/engines/engine.py.230: loss: 0.19959 [I 2022-10-26 19:49:25] ...optimizers/trijoint.py.151: total_norm: 19.605904 [I 2022-10-26 19:49:30] ...trap/engines/engine.py.224: train: epoch 26 | batch 2020/2304 [I 2022-10-26 19:49:30] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:37 [I 2022-10-26 19:49:30] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00240 [I 2022-10-26 19:49:30] ...trap/engines/engine.py.230: loss: 0.12407 [I 2022-10-26 19:49:30] ...optimizers/trijoint.py.151: total_norm: 20.281973 [I 2022-10-26 19:49:36] ...trap/engines/engine.py.224: train: epoch 26 | batch 2030/2304 [I 2022-10-26 19:49:36] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:31 [I 2022-10-26 19:49:36] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00242 [I 2022-10-26 19:49:36] ...trap/engines/engine.py.230: loss: 0.15986 [I 2022-10-26 19:49:36] ...optimizers/trijoint.py.151: total_norm: 15.272659 [I 2022-10-26 19:49:41] ...trap/engines/engine.py.224: train: epoch 26 | batch 2040/2304 [I 2022-10-26 19:49:41] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:27 [I 2022-10-26 19:49:41] ...trap/engines/engine.py.229: process: 0.56279 | load: 0.00242 [I 2022-10-26 19:49:41] ...trap/engines/engine.py.230: loss: 0.20046 [I 2022-10-26 19:49:41] ...optimizers/trijoint.py.151: total_norm: 25.596531 [I 2022-10-26 19:49:47] ...trap/engines/engine.py.224: train: epoch 26 | batch 2050/2304 [I 2022-10-26 19:49:47] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:02:20 [I 2022-10-26 19:49:47] ...trap/engines/engine.py.229: process: 0.53753 | load: 0.00245 [I 2022-10-26 19:49:47] ...trap/engines/engine.py.230: loss: 0.17170 [I 2022-10-26 19:49:47] ...optimizers/trijoint.py.151: total_norm: 19.612370 [I 2022-10-26 19:49:53] ...trap/engines/engine.py.224: train: epoch 26 | batch 2060/2304 [I 2022-10-26 19:49:53] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:02:15 [I 2022-10-26 19:49:53] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00241 [I 2022-10-26 19:49:53] ...trap/engines/engine.py.230: loss: 0.14216 [I 2022-10-26 19:49:53] ...optimizers/trijoint.py.151: total_norm: 16.698959 [I 2022-10-26 19:49:58] ...trap/engines/engine.py.224: train: epoch 26 | batch 2070/2304 [I 2022-10-26 19:49:58] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:02:12 [I 2022-10-26 19:49:58] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00247 [I 2022-10-26 19:49:58] ...trap/engines/engine.py.230: loss: 0.14441 [I 2022-10-26 19:49:58] ...optimizers/trijoint.py.151: total_norm: 14.678904 [I 2022-10-26 19:50:04] ...trap/engines/engine.py.224: train: epoch 26 | batch 2080/2304 [I 2022-10-26 19:50:04] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:02:05 [I 2022-10-26 19:50:04] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00265 [I 2022-10-26 19:50:04] ...trap/engines/engine.py.230: loss: 0.12080 [I 2022-10-26 19:50:04] ...optimizers/trijoint.py.151: total_norm: 15.893468 [I 2022-10-26 19:50:09] ...trap/engines/engine.py.224: train: epoch 26 | batch 2090/2304 [I 2022-10-26 19:50:09] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:58 [I 2022-10-26 19:50:09] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00245 [I 2022-10-26 19:50:09] ...trap/engines/engine.py.230: loss: 0.17470 [I 2022-10-26 19:50:09] ...optimizers/trijoint.py.151: total_norm: 24.683672 [I 2022-10-26 19:50:15] ...trap/engines/engine.py.224: train: epoch 26 | batch 2100/2304 [I 2022-10-26 19:50:15] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:52 [I 2022-10-26 19:50:15] ...trap/engines/engine.py.229: process: 0.53617 | load: 0.00241 [I 2022-10-26 19:50:15] ...trap/engines/engine.py.230: loss: 0.16613 [I 2022-10-26 19:50:15] ...optimizers/trijoint.py.151: total_norm: 12.124446 [I 2022-10-26 19:50:20] ...trap/engines/engine.py.224: train: epoch 26 | batch 2110/2304 [I 2022-10-26 19:50:20] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:46 [I 2022-10-26 19:50:20] ...trap/engines/engine.py.229: process: 0.54562 | load: 0.00240 [I 2022-10-26 19:50:20] ...trap/engines/engine.py.230: loss: 0.15993 [I 2022-10-26 19:50:20] ...optimizers/trijoint.py.151: total_norm: 18.994596 [I 2022-10-26 19:50:26] ...trap/engines/engine.py.224: train: epoch 26 | batch 2120/2304 [I 2022-10-26 19:50:26] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:42 [I 2022-10-26 19:50:26] ...trap/engines/engine.py.229: process: 0.55687 | load: 0.00241 [I 2022-10-26 19:50:26] ...trap/engines/engine.py.230: loss: 0.18227 [I 2022-10-26 19:50:26] ...optimizers/trijoint.py.151: total_norm: 16.254503 [I 2022-10-26 19:50:32] ...trap/engines/engine.py.224: train: epoch 26 | batch 2130/2304 [I 2022-10-26 19:50:32] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:39 [I 2022-10-26 19:50:32] ...trap/engines/engine.py.229: process: 0.56553 | load: 0.00251 [I 2022-10-26 19:50:32] ...trap/engines/engine.py.230: loss: 0.16552 [I 2022-10-26 19:50:32] ...optimizers/trijoint.py.151: total_norm: 18.136467 [I 2022-10-26 19:50:37] ...trap/engines/engine.py.224: train: epoch 26 | batch 2140/2304 [I 2022-10-26 19:50:37] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:31 [I 2022-10-26 19:50:37] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00243 [I 2022-10-26 19:50:37] ...trap/engines/engine.py.230: loss: 0.25033 [I 2022-10-26 19:50:37] ...optimizers/trijoint.py.151: total_norm: 18.431774 [I 2022-10-26 19:50:43] ...trap/engines/engine.py.224: train: epoch 26 | batch 2150/2304 [I 2022-10-26 19:50:43] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:25 [I 2022-10-26 19:50:43] ...trap/engines/engine.py.229: process: 0.55045 | load: 0.00243 [I 2022-10-26 19:50:43] ...trap/engines/engine.py.230: loss: 0.22186 [I 2022-10-26 19:50:43] ...optimizers/trijoint.py.151: total_norm: 25.613825 [I 2022-10-26 19:50:48] ...trap/engines/engine.py.224: train: epoch 26 | batch 2160/2304 [I 2022-10-26 19:50:48] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:01:19 [I 2022-10-26 19:50:48] ...trap/engines/engine.py.229: process: 0.55432 | load: 0.00241 [I 2022-10-26 19:50:48] ...trap/engines/engine.py.230: loss: 0.17680 [I 2022-10-26 19:50:48] ...optimizers/trijoint.py.151: total_norm: 22.177374 [I 2022-10-26 19:50:54] ...trap/engines/engine.py.224: train: epoch 26 | batch 2170/2304 [I 2022-10-26 19:50:54] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:01:14 [I 2022-10-26 19:50:54] ...trap/engines/engine.py.229: process: 0.55248 | load: 0.00239 [I 2022-10-26 19:50:54] ...trap/engines/engine.py.230: loss: 0.21269 [I 2022-10-26 19:50:54] ...optimizers/trijoint.py.151: total_norm: 16.696577 [I 2022-10-26 19:51:00] ...trap/engines/engine.py.224: train: epoch 26 | batch 2180/2304 [I 2022-10-26 19:51:00] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:01:08 [I 2022-10-26 19:51:00] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00246 [I 2022-10-26 19:51:00] ...trap/engines/engine.py.230: loss: 0.17514 [I 2022-10-26 19:51:00] ...optimizers/trijoint.py.151: total_norm: 23.558546 [I 2022-10-26 19:51:05] ...trap/engines/engine.py.224: train: epoch 26 | batch 2190/2304 [I 2022-10-26 19:51:05] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:01:03 [I 2022-10-26 19:51:05] ...trap/engines/engine.py.229: process: 0.54831 | load: 0.00239 [I 2022-10-26 19:51:05] ...trap/engines/engine.py.230: loss: 0.19301 [I 2022-10-26 19:51:05] ...optimizers/trijoint.py.151: total_norm: 24.491194 [I 2022-10-26 19:51:11] ...trap/engines/engine.py.224: train: epoch 26 | batch 2200/2304 [I 2022-10-26 19:51:11] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:57 [I 2022-10-26 19:51:11] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00240 [I 2022-10-26 19:51:11] ...trap/engines/engine.py.230: loss: 0.19389 [I 2022-10-26 19:51:11] ...optimizers/trijoint.py.151: total_norm: 16.973728 [I 2022-10-26 19:51:16] ...trap/engines/engine.py.224: train: epoch 26 | batch 2210/2304 [I 2022-10-26 19:51:16] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:52 [I 2022-10-26 19:51:16] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00243 [I 2022-10-26 19:51:16] ...trap/engines/engine.py.230: loss: 0.13640 [I 2022-10-26 19:51:16] ...optimizers/trijoint.py.151: total_norm: 14.733065 [I 2022-10-26 19:51:22] ...trap/engines/engine.py.224: train: epoch 26 | batch 2220/2304 [I 2022-10-26 19:51:22] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:47 [I 2022-10-26 19:51:22] ...trap/engines/engine.py.229: process: 0.57319 | load: 0.00247 [I 2022-10-26 19:51:22] ...trap/engines/engine.py.230: loss: 0.16725 [I 2022-10-26 19:51:22] ...optimizers/trijoint.py.151: total_norm: 23.922596 [I 2022-10-26 19:51:27] ...trap/engines/engine.py.224: train: epoch 26 | batch 2230/2304 [I 2022-10-26 19:51:27] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:40 [I 2022-10-26 19:51:27] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00240 [I 2022-10-26 19:51:27] ...trap/engines/engine.py.230: loss: 0.14665 [I 2022-10-26 19:51:27] ...optimizers/trijoint.py.151: total_norm: 16.923281 [I 2022-10-26 19:51:33] ...trap/engines/engine.py.224: train: epoch 26 | batch 2240/2304 [I 2022-10-26 19:51:33] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:35 [I 2022-10-26 19:51:33] ...trap/engines/engine.py.229: process: 0.55193 | load: 0.00252 [I 2022-10-26 19:51:33] ...trap/engines/engine.py.230: loss: 0.19118 [I 2022-10-26 19:51:33] ...optimizers/trijoint.py.151: total_norm: 15.574248 [I 2022-10-26 19:51:39] ...trap/engines/engine.py.224: train: epoch 26 | batch 2250/2304 [I 2022-10-26 19:51:39] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:30 [I 2022-10-26 19:51:39] ...trap/engines/engine.py.229: process: 0.55130 | load: 0.00256 [I 2022-10-26 19:51:39] ...trap/engines/engine.py.230: loss: 0.19939 [I 2022-10-26 19:51:39] ...optimizers/trijoint.py.151: total_norm: 26.878647 [I 2022-10-26 19:51:44] ...trap/engines/engine.py.224: train: epoch 26 | batch 2260/2304 [I 2022-10-26 19:51:44] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:24 [I 2022-10-26 19:51:44] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00243 [I 2022-10-26 19:51:44] ...trap/engines/engine.py.230: loss: 0.17634 [I 2022-10-26 19:51:44] ...optimizers/trijoint.py.151: total_norm: 22.628782 [I 2022-10-26 19:51:50] ...trap/engines/engine.py.224: train: epoch 26 | batch 2270/2304 [I 2022-10-26 19:51:50] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:18 [I 2022-10-26 19:51:50] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00244 [I 2022-10-26 19:51:50] ...trap/engines/engine.py.230: loss: 0.16414 [I 2022-10-26 19:51:50] ...optimizers/trijoint.py.151: total_norm: 24.047544 [I 2022-10-26 19:51:55] ...trap/engines/engine.py.224: train: epoch 26 | batch 2280/2304 [I 2022-10-26 19:51:55] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:00:13 [I 2022-10-26 19:51:55] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00241 [I 2022-10-26 19:51:55] ...trap/engines/engine.py.230: loss: 0.13543 [I 2022-10-26 19:51:55] ...optimizers/trijoint.py.151: total_norm: 18.468473 [I 2022-10-26 19:52:01] ...trap/engines/engine.py.224: train: epoch 26 | batch 2290/2304 [I 2022-10-26 19:52:01] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:00:07 [I 2022-10-26 19:52:01] ...trap/engines/engine.py.229: process: 0.54528 | load: 0.00262 [I 2022-10-26 19:52:01] ...trap/engines/engine.py.230: loss: 0.16759 [I 2022-10-26 19:52:01] ...optimizers/trijoint.py.151: total_norm: 15.371984 [I 2022-10-26 19:52:06] ...trap/engines/engine.py.224: train: epoch 26 | batch 2300/2304 [I 2022-10-26 19:52:06] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:00:02 [I 2022-10-26 19:52:06] ...trap/engines/engine.py.229: process: 0.54232 | load: 0.00015 [I 2022-10-26 19:52:06] ...trap/engines/engine.py.230: loss: 0.22447 [I 2022-10-26 19:52:06] ...optimizers/trijoint.py.151: total_norm: 19.230125 [I 2022-10-26 19:52:09] ...trap/engines/engine.py.224: train: epoch 26 | batch 2304/2304 [I 2022-10-26 19:52:09] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:00:00 [I 2022-10-26 19:52:09] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00016 [I 2022-10-26 19:52:09] ...trap/engines/engine.py.230: loss: 0.20509 [I 2022-10-26 19:52:09] ...optimizers/trijoint.py.151: total_norm: 32.087078 [S 2022-10-26 19:52:10] ...trap/engines/engine.py.129: train_epoch.epoch: 26 [S 2022-10-26 19:52:10] ...trap/engines/engine.py.129: train_epoch.loss: 0.18808439544638428 [I 2022-10-26 19:52:10] ...s/criterions/triplet.py.84: margin = 0.1850000000000001 [I 2022-10-26 19:52:10] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 26 [I 2022-10-26 19:52:10] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 19:52:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 19:52:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 19:52:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 19:52:13] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 19:52:13] ...trap/engines/engine.py.321: eval: epoch 26 | batch 0/510 [I 2022-10-26 19:52:13] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:25:05 [I 2022-10-26 19:52:13] ...trap/engines/engine.py.326: process: 2.95247 | load: 2.58517 [I 2022-10-26 19:52:16] ...trap/engines/engine.py.321: eval: epoch 26 | batch 10/510 [I 2022-10-26 19:52:16] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:07 [I 2022-10-26 19:52:16] ...trap/engines/engine.py.326: process: 0.34067 | load: 0.00020 [I 2022-10-26 19:52:20] ...trap/engines/engine.py.321: eval: epoch 26 | batch 20/510 [I 2022-10-26 19:52:20] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:59 [I 2022-10-26 19:52:20] ...trap/engines/engine.py.326: process: 0.32482 | load: 0.00019 [I 2022-10-26 19:52:23] ...trap/engines/engine.py.321: eval: epoch 26 | batch 30/510 [I 2022-10-26 19:52:23] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:41 [I 2022-10-26 19:52:23] ...trap/engines/engine.py.326: process: 0.32425 | load: 0.00020 [I 2022-10-26 19:52:27] ...trap/engines/engine.py.321: eval: epoch 26 | batch 40/510 [I 2022-10-26 19:52:27] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:37 [I 2022-10-26 19:52:27] ...trap/engines/engine.py.326: process: 0.32358 | load: 0.00017 [I 2022-10-26 19:52:30] ...trap/engines/engine.py.321: eval: epoch 26 | batch 50/510 [I 2022-10-26 19:52:30] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:33 [I 2022-10-26 19:52:30] ...trap/engines/engine.py.326: process: 0.31885 | load: 0.00016 [I 2022-10-26 19:52:33] ...trap/engines/engine.py.321: eval: epoch 26 | batch 60/510 [I 2022-10-26 19:52:33] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:32 [I 2022-10-26 19:52:33] ...trap/engines/engine.py.326: process: 0.33364 | load: 0.00018 [I 2022-10-26 19:52:37] ...trap/engines/engine.py.321: eval: epoch 26 | batch 70/510 [I 2022-10-26 19:52:37] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:27 [I 2022-10-26 19:52:37] ...trap/engines/engine.py.326: process: 0.33306 | load: 0.00016 [I 2022-10-26 19:52:40] ...trap/engines/engine.py.321: eval: epoch 26 | batch 80/510 [I 2022-10-26 19:52:40] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:23 [I 2022-10-26 19:52:40] ...trap/engines/engine.py.326: process: 0.33440 | load: 0.00021 [I 2022-10-26 19:52:43] ...trap/engines/engine.py.321: eval: epoch 26 | batch 90/510 [I 2022-10-26 19:52:43] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:19 [I 2022-10-26 19:52:43] ...trap/engines/engine.py.326: process: 0.34350 | load: 0.00017 [I 2022-10-26 19:52:47] ...trap/engines/engine.py.321: eval: epoch 26 | batch 100/510 [I 2022-10-26 19:52:47] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 19:52:47] ...trap/engines/engine.py.326: process: 0.28466 | load: 0.00016 [I 2022-10-26 19:52:49] ...trap/engines/engine.py.321: eval: epoch 26 | batch 110/510 [I 2022-10-26 19:52:49] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 19:52:49] ...trap/engines/engine.py.326: process: 0.28164 | load: 0.00018 [I 2022-10-26 19:52:52] ...trap/engines/engine.py.321: eval: epoch 26 | batch 120/510 [I 2022-10-26 19:52:52] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-26 19:52:52] ...trap/engines/engine.py.326: process: 0.28884 | load: 0.00019 [I 2022-10-26 19:52:55] ...trap/engines/engine.py.321: eval: epoch 26 | batch 130/510 [I 2022-10-26 19:52:55] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 19:52:55] ...trap/engines/engine.py.326: process: 0.29454 | load: 0.00015 [I 2022-10-26 19:52:58] ...trap/engines/engine.py.321: eval: epoch 26 | batch 140/510 [I 2022-10-26 19:52:58] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:47 [I 2022-10-26 19:52:58] ...trap/engines/engine.py.326: process: 0.29231 | load: 0.00021 [I 2022-10-26 19:53:01] ...trap/engines/engine.py.321: eval: epoch 26 | batch 150/510 [I 2022-10-26 19:53:01] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:44 [I 2022-10-26 19:53:01] ...trap/engines/engine.py.326: process: 0.28995 | load: 0.00046 [I 2022-10-26 19:53:04] ...trap/engines/engine.py.321: eval: epoch 26 | batch 160/510 [I 2022-10-26 19:53:04] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:42 [I 2022-10-26 19:53:04] ...trap/engines/engine.py.326: process: 0.28677 | load: 0.00018 [I 2022-10-26 19:53:07] ...trap/engines/engine.py.321: eval: epoch 26 | batch 170/510 [I 2022-10-26 19:53:07] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 19:53:07] ...trap/engines/engine.py.326: process: 0.28871 | load: 0.00024 [I 2022-10-26 19:53:10] ...trap/engines/engine.py.321: eval: epoch 26 | batch 180/510 [I 2022-10-26 19:53:10] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:36 [I 2022-10-26 19:53:10] ...trap/engines/engine.py.326: process: 0.28672 | load: 0.00014 [I 2022-10-26 19:53:13] ...trap/engines/engine.py.321: eval: epoch 26 | batch 190/510 [I 2022-10-26 19:53:13] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 19:53:13] ...trap/engines/engine.py.326: process: 0.28648 | load: 0.00021 [I 2022-10-26 19:53:16] ...trap/engines/engine.py.321: eval: epoch 26 | batch 200/510 [I 2022-10-26 19:53:16] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 19:53:16] ...trap/engines/engine.py.326: process: 0.28432 | load: 0.00014 [I 2022-10-26 19:53:19] ...trap/engines/engine.py.321: eval: epoch 26 | batch 210/510 [I 2022-10-26 19:53:19] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 19:53:19] ...trap/engines/engine.py.326: process: 0.28897 | load: 0.00029 [I 2022-10-26 19:53:21] ...trap/engines/engine.py.321: eval: epoch 26 | batch 220/510 [I 2022-10-26 19:53:21] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 19:53:21] ...trap/engines/engine.py.326: process: 0.29692 | load: 0.00016 [I 2022-10-26 19:53:24] ...trap/engines/engine.py.321: eval: epoch 26 | batch 230/510 [I 2022-10-26 19:53:24] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 19:53:24] ...trap/engines/engine.py.326: process: 0.28348 | load: 0.00026 [I 2022-10-26 19:53:27] ...trap/engines/engine.py.321: eval: epoch 26 | batch 240/510 [I 2022-10-26 19:53:27] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:18 [I 2022-10-26 19:53:27] ...trap/engines/engine.py.326: process: 0.28840 | load: 0.00020 [I 2022-10-26 19:53:30] ...trap/engines/engine.py.321: eval: epoch 26 | batch 250/510 [I 2022-10-26 19:53:30] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:15 [I 2022-10-26 19:53:30] ...trap/engines/engine.py.326: process: 0.29157 | load: 0.00028 [I 2022-10-26 19:53:33] ...trap/engines/engine.py.321: eval: epoch 26 | batch 260/510 [I 2022-10-26 19:53:33] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:13 [I 2022-10-26 19:53:33] ...trap/engines/engine.py.326: process: 0.29266 | load: 0.00047 [I 2022-10-26 19:53:36] ...trap/engines/engine.py.321: eval: epoch 26 | batch 270/510 [I 2022-10-26 19:53:36] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 19:53:36] ...trap/engines/engine.py.326: process: 0.29013 | load: 0.00018 [I 2022-10-26 19:53:39] ...trap/engines/engine.py.321: eval: epoch 26 | batch 280/510 [I 2022-10-26 19:53:39] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:07 [I 2022-10-26 19:53:39] ...trap/engines/engine.py.326: process: 0.29415 | load: 0.00018 [I 2022-10-26 19:53:42] ...trap/engines/engine.py.321: eval: epoch 26 | batch 290/510 [I 2022-10-26 19:53:42] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:04 [I 2022-10-26 19:53:42] ...trap/engines/engine.py.326: process: 0.29279 | load: 0.00023 [I 2022-10-26 19:53:45] ...trap/engines/engine.py.321: eval: epoch 26 | batch 300/510 [I 2022-10-26 19:53:45] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:01 [I 2022-10-26 19:53:45] ...trap/engines/engine.py.326: process: 0.29188 | load: 0.00020 [I 2022-10-26 19:53:48] ...trap/engines/engine.py.321: eval: epoch 26 | batch 310/510 [I 2022-10-26 19:53:48] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 19:53:48] ...trap/engines/engine.py.326: process: 0.28833 | load: 0.00016 [I 2022-10-26 19:53:50] ...trap/engines/engine.py.321: eval: epoch 26 | batch 320/510 [I 2022-10-26 19:53:50] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 19:53:50] ...trap/engines/engine.py.326: process: 0.28502 | load: 0.00016 [I 2022-10-26 19:53:53] ...trap/engines/engine.py.321: eval: epoch 26 | batch 330/510 [I 2022-10-26 19:53:53] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 19:53:53] ...trap/engines/engine.py.326: process: 0.28917 | load: 0.00024 [I 2022-10-26 19:53:56] ...trap/engines/engine.py.321: eval: epoch 26 | batch 340/510 [I 2022-10-26 19:53:56] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:49 [I 2022-10-26 19:53:56] ...trap/engines/engine.py.326: process: 0.29283 | load: 0.00017 [I 2022-10-26 19:53:59] ...trap/engines/engine.py.321: eval: epoch 26 | batch 350/510 [I 2022-10-26 19:53:59] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 19:53:59] ...trap/engines/engine.py.326: process: 0.27959 | load: 0.00025 [I 2022-10-26 19:54:02] ...trap/engines/engine.py.321: eval: epoch 26 | batch 360/510 [I 2022-10-26 19:54:02] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 19:54:02] ...trap/engines/engine.py.326: process: 0.28519 | load: 0.00019 [I 2022-10-26 19:54:05] ...trap/engines/engine.py.321: eval: epoch 26 | batch 370/510 [I 2022-10-26 19:54:05] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 19:54:05] ...trap/engines/engine.py.326: process: 0.28887 | load: 0.00025 [I 2022-10-26 19:54:08] ...trap/engines/engine.py.321: eval: epoch 26 | batch 380/510 [I 2022-10-26 19:54:08] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 19:54:08] ...trap/engines/engine.py.326: process: 0.29263 | load: 0.00022 [I 2022-10-26 19:54:11] ...trap/engines/engine.py.321: eval: epoch 26 | batch 390/510 [I 2022-10-26 19:54:11] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 19:54:11] ...trap/engines/engine.py.326: process: 0.29005 | load: 0.00025 [I 2022-10-26 19:54:14] ...trap/engines/engine.py.321: eval: epoch 26 | batch 400/510 [I 2022-10-26 19:54:14] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 19:54:14] ...trap/engines/engine.py.326: process: 0.28767 | load: 0.00021 [I 2022-10-26 19:54:16] ...trap/engines/engine.py.321: eval: epoch 26 | batch 410/510 [I 2022-10-26 19:54:16] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:29 [I 2022-10-26 19:54:16] ...trap/engines/engine.py.326: process: 0.29027 | load: 0.00024 [I 2022-10-26 19:54:19] ...trap/engines/engine.py.321: eval: epoch 26 | batch 420/510 [I 2022-10-26 19:54:19] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:26 [I 2022-10-26 19:54:19] ...trap/engines/engine.py.326: process: 0.28913 | load: 0.00020 [I 2022-10-26 19:54:22] ...trap/engines/engine.py.321: eval: epoch 26 | batch 430/510 [I 2022-10-26 19:54:22] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 19:54:22] ...trap/engines/engine.py.326: process: 0.29491 | load: 0.00016 [I 2022-10-26 19:54:25] ...trap/engines/engine.py.321: eval: epoch 26 | batch 440/510 [I 2022-10-26 19:54:25] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 19:54:25] ...trap/engines/engine.py.326: process: 0.28956 | load: 0.00021 [I 2022-10-26 19:54:28] ...trap/engines/engine.py.321: eval: epoch 26 | batch 450/510 [I 2022-10-26 19:54:28] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 19:54:28] ...trap/engines/engine.py.326: process: 0.29295 | load: 0.00026 [I 2022-10-26 19:54:31] ...trap/engines/engine.py.321: eval: epoch 26 | batch 460/510 [I 2022-10-26 19:54:31] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 19:54:31] ...trap/engines/engine.py.326: process: 0.29356 | load: 0.00022 [I 2022-10-26 19:54:34] ...trap/engines/engine.py.321: eval: epoch 26 | batch 470/510 [I 2022-10-26 19:54:34] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 19:54:34] ...trap/engines/engine.py.326: process: 0.28950 | load: 0.00024 [I 2022-10-26 19:54:37] ...trap/engines/engine.py.321: eval: epoch 26 | batch 480/510 [I 2022-10-26 19:54:37] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 19:54:37] ...trap/engines/engine.py.326: process: 0.28431 | load: 0.00020 [I 2022-10-26 19:54:40] ...trap/engines/engine.py.321: eval: epoch 26 | batch 490/510 [I 2022-10-26 19:54:40] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 19:54:40] ...trap/engines/engine.py.326: process: 0.29354 | load: 0.00020 [I 2022-10-26 19:54:43] ...trap/engines/engine.py.321: eval: epoch 26 | batch 500/510 [I 2022-10-26 19:54:43] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 19:54:43] ...trap/engines/engine.py.326: process: 0.28737 | load: 0.00027 [I 2022-10-26 19:54:45] ...trap/engines/engine.py.321: eval: epoch 26 | batch 510/510 [I 2022-10-26 19:54:45] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 19:54:45] ...trap/engines/engine.py.326: process: 0.28324 | load: 0.00017 [S 2022-10-26 19:54:46] ...trap/engines/engine.py.132: eval_epoch.epoch: 26 [I 2022-10-26 19:54:46] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.653000 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.883300 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.926600 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.639000 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.878400 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.921700 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014478 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007100 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005748 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012696 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005535 [S 2022-10-26 19:54:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005216 [I 2022-10-26 19:54:51] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 19:54:51] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 19:54:51] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 19:54:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 19:54:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 19:54:52] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 19:54:55] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 19:55:01] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 19:55:01] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 19:55:01] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 19:55:04] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 19:55:10] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 19:55:10] ...trap/engines/engine.py.163: Training model on trainset for epoch 27 [I 2022-10-26 19:55:10] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 19:55:12] ...trap/engines/engine.py.224: train: epoch 27 | batch 0/2304 [I 2022-10-26 19:55:12] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:40:41 [I 2022-10-26 19:55:12] ...trap/engines/engine.py.229: process: 2.62238 | load: 1.97849 [I 2022-10-26 19:55:12] ...trap/engines/engine.py.230: loss: 0.21401 [I 2022-10-26 19:55:12] ...optimizers/trijoint.py.151: total_norm: 14.379587 [I 2022-10-26 19:55:18] ...trap/engines/engine.py.224: train: epoch 27 | batch 10/2304 [I 2022-10-26 19:55:18] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:44 [I 2022-10-26 19:55:18] ...trap/engines/engine.py.229: process: 0.55169 | load: 0.00434 [I 2022-10-26 19:55:18] ...trap/engines/engine.py.230: loss: 0.30089 [I 2022-10-26 19:55:18] ...optimizers/trijoint.py.151: total_norm: 21.180519 [I 2022-10-26 19:55:24] ...trap/engines/engine.py.224: train: epoch 27 | batch 20/2304 [I 2022-10-26 19:55:24] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:05 [I 2022-10-26 19:55:24] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00392 [I 2022-10-26 19:55:24] ...trap/engines/engine.py.230: loss: 0.14359 [I 2022-10-26 19:55:24] ...optimizers/trijoint.py.151: total_norm: 13.982145 [I 2022-10-26 19:55:29] ...trap/engines/engine.py.224: train: epoch 27 | batch 30/2304 [I 2022-10-26 19:55:29] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:20:58 [I 2022-10-26 19:55:29] ...trap/engines/engine.py.229: process: 0.53038 | load: 0.00292 [I 2022-10-26 19:55:29] ...trap/engines/engine.py.230: loss: 0.15673 [I 2022-10-26 19:55:29] ...optimizers/trijoint.py.151: total_norm: 15.784789 [I 2022-10-26 19:55:35] ...trap/engines/engine.py.224: train: epoch 27 | batch 40/2304 [I 2022-10-26 19:55:35] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:58 [I 2022-10-26 19:55:35] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00331 [I 2022-10-26 19:55:35] ...trap/engines/engine.py.230: loss: 0.13158 [I 2022-10-26 19:55:35] ...optimizers/trijoint.py.151: total_norm: 18.103966 [I 2022-10-26 19:55:40] ...trap/engines/engine.py.224: train: epoch 27 | batch 50/2304 [I 2022-10-26 19:55:40] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:24 [I 2022-10-26 19:55:40] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00316 [I 2022-10-26 19:55:40] ...trap/engines/engine.py.230: loss: 0.20350 [I 2022-10-26 19:55:40] ...optimizers/trijoint.py.151: total_norm: 16.223976 [I 2022-10-26 19:55:46] ...trap/engines/engine.py.224: train: epoch 27 | batch 60/2304 [I 2022-10-26 19:55:46] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:50 [I 2022-10-26 19:55:46] ...trap/engines/engine.py.229: process: 0.54888 | load: 0.00299 [I 2022-10-26 19:55:46] ...trap/engines/engine.py.230: loss: 0.17022 [I 2022-10-26 19:55:46] ...optimizers/trijoint.py.151: total_norm: 25.410204 [I 2022-10-26 19:55:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 70/2304 [I 2022-10-26 19:55:52] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:54 [I 2022-10-26 19:55:52] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00315 [I 2022-10-26 19:55:52] ...trap/engines/engine.py.230: loss: 0.12046 [I 2022-10-26 19:55:52] ...optimizers/trijoint.py.151: total_norm: 14.519430 [I 2022-10-26 19:55:57] ...trap/engines/engine.py.224: train: epoch 27 | batch 80/2304 [I 2022-10-26 19:55:57] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:44 [I 2022-10-26 19:55:57] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00298 [I 2022-10-26 19:55:57] ...trap/engines/engine.py.230: loss: 0.23155 [I 2022-10-26 19:55:57] ...optimizers/trijoint.py.151: total_norm: 23.859280 [I 2022-10-26 19:56:03] ...trap/engines/engine.py.224: train: epoch 27 | batch 90/2304 [I 2022-10-26 19:56:03] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:36 [I 2022-10-26 19:56:03] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00260 [I 2022-10-26 19:56:03] ...trap/engines/engine.py.230: loss: 0.21518 [I 2022-10-26 19:56:03] ...optimizers/trijoint.py.151: total_norm: 20.184656 [I 2022-10-26 19:56:08] ...trap/engines/engine.py.224: train: epoch 27 | batch 100/2304 [I 2022-10-26 19:56:08] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:29 [I 2022-10-26 19:56:08] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00299 [I 2022-10-26 19:56:08] ...trap/engines/engine.py.230: loss: 0.24118 [I 2022-10-26 19:56:08] ...optimizers/trijoint.py.151: total_norm: 27.349051 [I 2022-10-26 19:56:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 110/2304 [I 2022-10-26 19:56:14] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:21 [I 2022-10-26 19:56:14] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00266 [I 2022-10-26 19:56:14] ...trap/engines/engine.py.230: loss: 0.16120 [I 2022-10-26 19:56:14] ...optimizers/trijoint.py.151: total_norm: 18.659964 [I 2022-10-26 19:56:20] ...trap/engines/engine.py.224: train: epoch 27 | batch 120/2304 [I 2022-10-26 19:56:20] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:15 [I 2022-10-26 19:56:20] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00252 [I 2022-10-26 19:56:20] ...trap/engines/engine.py.230: loss: 0.20955 [I 2022-10-26 19:56:20] ...optimizers/trijoint.py.151: total_norm: 40.373455 [I 2022-10-26 19:56:25] ...trap/engines/engine.py.224: train: epoch 27 | batch 130/2304 [I 2022-10-26 19:56:25] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:21 [I 2022-10-26 19:56:25] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00264 [I 2022-10-26 19:56:25] ...trap/engines/engine.py.230: loss: 0.26403 [I 2022-10-26 19:56:25] ...optimizers/trijoint.py.151: total_norm: 21.981644 [I 2022-10-26 19:56:31] ...trap/engines/engine.py.224: train: epoch 27 | batch 140/2304 [I 2022-10-26 19:56:31] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:25 [I 2022-10-26 19:56:31] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00257 [I 2022-10-26 19:56:31] ...trap/engines/engine.py.230: loss: 0.18892 [I 2022-10-26 19:56:31] ...optimizers/trijoint.py.151: total_norm: 25.942177 [I 2022-10-26 19:56:36] ...trap/engines/engine.py.224: train: epoch 27 | batch 150/2304 [I 2022-10-26 19:56:36] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:02 [I 2022-10-26 19:56:36] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00255 [I 2022-10-26 19:56:36] ...trap/engines/engine.py.230: loss: 0.16364 [I 2022-10-26 19:56:36] ...optimizers/trijoint.py.151: total_norm: 15.971976 [I 2022-10-26 19:56:42] ...trap/engines/engine.py.224: train: epoch 27 | batch 160/2304 [I 2022-10-26 19:56:42] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:48 [I 2022-10-26 19:56:42] ...trap/engines/engine.py.229: process: 0.55042 | load: 0.00252 [I 2022-10-26 19:56:42] ...trap/engines/engine.py.230: loss: 0.20578 [I 2022-10-26 19:56:42] ...optimizers/trijoint.py.151: total_norm: 24.638245 [I 2022-10-26 19:56:48] ...trap/engines/engine.py.224: train: epoch 27 | batch 170/2304 [I 2022-10-26 19:56:48] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:43 [I 2022-10-26 19:56:48] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00254 [I 2022-10-26 19:56:48] ...trap/engines/engine.py.230: loss: 0.12899 [I 2022-10-26 19:56:48] ...optimizers/trijoint.py.151: total_norm: 16.378130 [I 2022-10-26 19:56:53] ...trap/engines/engine.py.224: train: epoch 27 | batch 180/2304 [I 2022-10-26 19:56:53] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:32 [I 2022-10-26 19:56:53] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00255 [I 2022-10-26 19:56:53] ...trap/engines/engine.py.230: loss: 0.21986 [I 2022-10-26 19:56:53] ...optimizers/trijoint.py.151: total_norm: 15.678844 [I 2022-10-26 19:56:59] ...trap/engines/engine.py.224: train: epoch 27 | batch 190/2304 [I 2022-10-26 19:56:59] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:38 [I 2022-10-26 19:56:59] ...trap/engines/engine.py.229: process: 0.55623 | load: 0.00243 [I 2022-10-26 19:56:59] ...trap/engines/engine.py.230: loss: 0.19053 [I 2022-10-26 19:56:59] ...optimizers/trijoint.py.151: total_norm: 9.061298 [I 2022-10-26 19:57:04] ...trap/engines/engine.py.224: train: epoch 27 | batch 200/2304 [I 2022-10-26 19:57:04] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:25 [I 2022-10-26 19:57:04] ...trap/engines/engine.py.229: process: 0.55275 | load: 0.00271 [I 2022-10-26 19:57:04] ...trap/engines/engine.py.230: loss: 0.20238 [I 2022-10-26 19:57:04] ...optimizers/trijoint.py.151: total_norm: 15.618038 [I 2022-10-26 19:57:10] ...trap/engines/engine.py.224: train: epoch 27 | batch 210/2304 [I 2022-10-26 19:57:10] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:31 [I 2022-10-26 19:57:10] ...trap/engines/engine.py.229: process: 0.55571 | load: 0.00240 [I 2022-10-26 19:57:10] ...trap/engines/engine.py.230: loss: 0.19033 [I 2022-10-26 19:57:10] ...optimizers/trijoint.py.151: total_norm: 34.830090 [I 2022-10-26 19:57:16] ...trap/engines/engine.py.224: train: epoch 27 | batch 220/2304 [I 2022-10-26 19:57:16] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:25 [I 2022-10-26 19:57:16] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00240 [I 2022-10-26 19:57:16] ...trap/engines/engine.py.230: loss: 0.21548 [I 2022-10-26 19:57:16] ...optimizers/trijoint.py.151: total_norm: 34.789162 [I 2022-10-26 19:57:21] ...trap/engines/engine.py.224: train: epoch 27 | batch 230/2304 [I 2022-10-26 19:57:21] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:22 [I 2022-10-26 19:57:21] ...trap/engines/engine.py.229: process: 0.56498 | load: 0.00246 [I 2022-10-26 19:57:21] ...trap/engines/engine.py.230: loss: 0.16116 [I 2022-10-26 19:57:21] ...optimizers/trijoint.py.151: total_norm: 20.759748 [I 2022-10-26 19:57:27] ...trap/engines/engine.py.224: train: epoch 27 | batch 240/2304 [I 2022-10-26 19:57:27] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:22 [I 2022-10-26 19:57:27] ...trap/engines/engine.py.229: process: 0.57751 | load: 0.00274 [I 2022-10-26 19:57:27] ...trap/engines/engine.py.230: loss: 0.24794 [I 2022-10-26 19:57:27] ...optimizers/trijoint.py.151: total_norm: 22.062540 [I 2022-10-26 19:57:32] ...trap/engines/engine.py.224: train: epoch 27 | batch 250/2304 [I 2022-10-26 19:57:32] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:02 [I 2022-10-26 19:57:32] ...trap/engines/engine.py.229: process: 0.54572 | load: 0.00244 [I 2022-10-26 19:57:32] ...trap/engines/engine.py.230: loss: 0.19905 [I 2022-10-26 19:57:32] ...optimizers/trijoint.py.151: total_norm: 32.158844 [I 2022-10-26 19:57:38] ...trap/engines/engine.py.224: train: epoch 27 | batch 260/2304 [I 2022-10-26 19:57:38] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:52 [I 2022-10-26 19:57:38] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00249 [I 2022-10-26 19:57:38] ...trap/engines/engine.py.230: loss: 0.15446 [I 2022-10-26 19:57:38] ...optimizers/trijoint.py.151: total_norm: 31.293707 [I 2022-10-26 19:57:44] ...trap/engines/engine.py.224: train: epoch 27 | batch 270/2304 [I 2022-10-26 19:57:44] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:50 [I 2022-10-26 19:57:44] ...trap/engines/engine.py.229: process: 0.54955 | load: 0.00249 [I 2022-10-26 19:57:44] ...trap/engines/engine.py.230: loss: 0.18717 [I 2022-10-26 19:57:44] ...optimizers/trijoint.py.151: total_norm: 20.625454 [I 2022-10-26 19:57:49] ...trap/engines/engine.py.224: train: epoch 27 | batch 280/2304 [I 2022-10-26 19:57:49] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:54 [I 2022-10-26 19:57:49] ...trap/engines/engine.py.229: process: 0.56599 | load: 0.00246 [I 2022-10-26 19:57:49] ...trap/engines/engine.py.230: loss: 0.17082 [I 2022-10-26 19:57:49] ...optimizers/trijoint.py.151: total_norm: 19.401440 [I 2022-10-26 19:57:55] ...trap/engines/engine.py.224: train: epoch 27 | batch 290/2304 [I 2022-10-26 19:57:55] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:45 [I 2022-10-26 19:57:55] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00253 [I 2022-10-26 19:57:55] ...trap/engines/engine.py.230: loss: 0.21020 [I 2022-10-26 19:57:55] ...optimizers/trijoint.py.151: total_norm: 20.971874 [I 2022-10-26 19:58:00] ...trap/engines/engine.py.224: train: epoch 27 | batch 300/2304 [I 2022-10-26 19:58:00] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:40 [I 2022-10-26 19:58:00] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00270 [I 2022-10-26 19:58:00] ...trap/engines/engine.py.230: loss: 0.18009 [I 2022-10-26 19:58:00] ...optimizers/trijoint.py.151: total_norm: 19.233908 [I 2022-10-26 19:58:06] ...trap/engines/engine.py.224: train: epoch 27 | batch 310/2304 [I 2022-10-26 19:58:06] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:19:40 [I 2022-10-26 19:58:06] ...trap/engines/engine.py.229: process: 0.71996 | load: 0.00259 [I 2022-10-26 19:58:06] ...trap/engines/engine.py.230: loss: 0.20283 [I 2022-10-26 19:58:06] ...optimizers/trijoint.py.151: total_norm: 14.089961 [I 2022-10-26 19:58:12] ...trap/engines/engine.py.224: train: epoch 27 | batch 320/2304 [I 2022-10-26 19:58:12] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:29 [I 2022-10-26 19:58:12] ...trap/engines/engine.py.229: process: 0.54900 | load: 0.00259 [I 2022-10-26 19:58:12] ...trap/engines/engine.py.230: loss: 0.21947 [I 2022-10-26 19:58:12] ...optimizers/trijoint.py.151: total_norm: 23.840744 [I 2022-10-26 19:58:17] ...trap/engines/engine.py.224: train: epoch 27 | batch 330/2304 [I 2022-10-26 19:58:17] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:14 [I 2022-10-26 19:58:17] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00271 [I 2022-10-26 19:58:17] ...trap/engines/engine.py.230: loss: 0.20360 [I 2022-10-26 19:58:17] ...optimizers/trijoint.py.151: total_norm: 16.451195 [I 2022-10-26 19:58:23] ...trap/engines/engine.py.224: train: epoch 27 | batch 340/2304 [I 2022-10-26 19:58:23] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:21 [I 2022-10-26 19:58:23] ...trap/engines/engine.py.229: process: 0.56297 | load: 0.00248 [I 2022-10-26 19:58:23] ...trap/engines/engine.py.230: loss: 0.17502 [I 2022-10-26 19:58:23] ...optimizers/trijoint.py.151: total_norm: 16.534937 [I 2022-10-26 19:58:28] ...trap/engines/engine.py.224: train: epoch 27 | batch 350/2304 [I 2022-10-26 19:58:28] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:07 [I 2022-10-26 19:58:28] ...trap/engines/engine.py.229: process: 0.56471 | load: 0.00245 [I 2022-10-26 19:58:28] ...trap/engines/engine.py.230: loss: 0.22236 [I 2022-10-26 19:58:28] ...optimizers/trijoint.py.151: total_norm: 24.499756 [I 2022-10-26 19:58:34] ...trap/engines/engine.py.224: train: epoch 27 | batch 360/2304 [I 2022-10-26 19:58:34] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:59 [I 2022-10-26 19:58:34] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00254 [I 2022-10-26 19:58:34] ...trap/engines/engine.py.230: loss: 0.16063 [I 2022-10-26 19:58:34] ...optimizers/trijoint.py.151: total_norm: 18.748030 [I 2022-10-26 19:58:40] ...trap/engines/engine.py.224: train: epoch 27 | batch 370/2304 [I 2022-10-26 19:58:40] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:56 [I 2022-10-26 19:58:40] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00246 [I 2022-10-26 19:58:40] ...trap/engines/engine.py.230: loss: 0.18636 [I 2022-10-26 19:58:40] ...optimizers/trijoint.py.151: total_norm: 17.231335 [I 2022-10-26 19:58:45] ...trap/engines/engine.py.224: train: epoch 27 | batch 380/2304 [I 2022-10-26 19:58:45] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:54 [I 2022-10-26 19:58:45] ...trap/engines/engine.py.229: process: 0.57618 | load: 0.00243 [I 2022-10-26 19:58:45] ...trap/engines/engine.py.230: loss: 0.19332 [I 2022-10-26 19:58:45] ...optimizers/trijoint.py.151: total_norm: 20.748337 [I 2022-10-26 19:58:51] ...trap/engines/engine.py.224: train: epoch 27 | batch 390/2304 [I 2022-10-26 19:58:51] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:42 [I 2022-10-26 19:58:51] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00268 [I 2022-10-26 19:58:51] ...trap/engines/engine.py.230: loss: 0.24306 [I 2022-10-26 19:58:51] ...optimizers/trijoint.py.151: total_norm: 37.089783 [I 2022-10-26 19:58:56] ...trap/engines/engine.py.224: train: epoch 27 | batch 400/2304 [I 2022-10-26 19:58:56] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:40 [I 2022-10-26 19:58:56] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00247 [I 2022-10-26 19:58:56] ...trap/engines/engine.py.230: loss: 0.18415 [I 2022-10-26 19:58:56] ...optimizers/trijoint.py.151: total_norm: 21.480356 [I 2022-10-26 19:59:02] ...trap/engines/engine.py.224: train: epoch 27 | batch 410/2304 [I 2022-10-26 19:59:02] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:35 [I 2022-10-26 19:59:02] ...trap/engines/engine.py.229: process: 0.56510 | load: 0.00246 [I 2022-10-26 19:59:02] ...trap/engines/engine.py.230: loss: 0.13344 [I 2022-10-26 19:59:02] ...optimizers/trijoint.py.151: total_norm: 25.562660 [I 2022-10-26 19:59:08] ...trap/engines/engine.py.224: train: epoch 27 | batch 420/2304 [I 2022-10-26 19:59:08] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:31 [I 2022-10-26 19:59:08] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00246 [I 2022-10-26 19:59:08] ...trap/engines/engine.py.230: loss: 0.16031 [I 2022-10-26 19:59:08] ...optimizers/trijoint.py.151: total_norm: 14.861814 [I 2022-10-26 19:59:13] ...trap/engines/engine.py.224: train: epoch 27 | batch 430/2304 [I 2022-10-26 19:59:13] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:21 [I 2022-10-26 19:59:13] ...trap/engines/engine.py.229: process: 0.54718 | load: 0.00248 [I 2022-10-26 19:59:13] ...trap/engines/engine.py.230: loss: 0.20740 [I 2022-10-26 19:59:13] ...optimizers/trijoint.py.151: total_norm: 21.677950 [I 2022-10-26 19:59:19] ...trap/engines/engine.py.224: train: epoch 27 | batch 440/2304 [I 2022-10-26 19:59:19] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:16 [I 2022-10-26 19:59:19] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00253 [I 2022-10-26 19:59:19] ...trap/engines/engine.py.230: loss: 0.16524 [I 2022-10-26 19:59:19] ...optimizers/trijoint.py.151: total_norm: 20.000959 [I 2022-10-26 19:59:24] ...trap/engines/engine.py.224: train: epoch 27 | batch 450/2304 [I 2022-10-26 19:59:24] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:06 [I 2022-10-26 19:59:24] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00244 [I 2022-10-26 19:59:24] ...trap/engines/engine.py.230: loss: 0.18923 [I 2022-10-26 19:59:24] ...optimizers/trijoint.py.151: total_norm: 12.868046 [I 2022-10-26 19:59:30] ...trap/engines/engine.py.224: train: epoch 27 | batch 460/2304 [I 2022-10-26 19:59:30] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:06 [I 2022-10-26 19:59:30] ...trap/engines/engine.py.229: process: 0.55316 | load: 0.00248 [I 2022-10-26 19:59:30] ...trap/engines/engine.py.230: loss: 0.20384 [I 2022-10-26 19:59:30] ...optimizers/trijoint.py.151: total_norm: 27.752598 [I 2022-10-26 19:59:35] ...trap/engines/engine.py.224: train: epoch 27 | batch 470/2304 [I 2022-10-26 19:59:35] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:01 [I 2022-10-26 19:59:35] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00244 [I 2022-10-26 19:59:35] ...trap/engines/engine.py.230: loss: 0.19066 [I 2022-10-26 19:59:35] ...optimizers/trijoint.py.151: total_norm: 28.430933 [I 2022-10-26 19:59:41] ...trap/engines/engine.py.224: train: epoch 27 | batch 480/2304 [I 2022-10-26 19:59:41] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:49 [I 2022-10-26 19:59:41] ...trap/engines/engine.py.229: process: 0.54518 | load: 0.00249 [I 2022-10-26 19:59:41] ...trap/engines/engine.py.230: loss: 0.22728 [I 2022-10-26 19:59:41] ...optimizers/trijoint.py.151: total_norm: 11.391326 [I 2022-10-26 19:59:47] ...trap/engines/engine.py.224: train: epoch 27 | batch 490/2304 [I 2022-10-26 19:59:47] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:50 [I 2022-10-26 19:59:47] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00250 [I 2022-10-26 19:59:47] ...trap/engines/engine.py.230: loss: 0.19465 [I 2022-10-26 19:59:47] ...optimizers/trijoint.py.151: total_norm: 19.760670 [I 2022-10-26 19:59:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 500/2304 [I 2022-10-26 19:59:52] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:34 [I 2022-10-26 19:59:52] ...trap/engines/engine.py.229: process: 0.55224 | load: 0.00260 [I 2022-10-26 19:59:52] ...trap/engines/engine.py.230: loss: 0.17121 [I 2022-10-26 19:59:52] ...optimizers/trijoint.py.151: total_norm: 15.944633 [I 2022-10-26 19:59:58] ...trap/engines/engine.py.224: train: epoch 27 | batch 510/2304 [I 2022-10-26 19:59:58] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:39 [I 2022-10-26 19:59:58] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00246 [I 2022-10-26 19:59:58] ...trap/engines/engine.py.230: loss: 0.18580 [I 2022-10-26 19:59:58] ...optimizers/trijoint.py.151: total_norm: 19.413248 [I 2022-10-26 20:00:03] ...trap/engines/engine.py.224: train: epoch 27 | batch 520/2304 [I 2022-10-26 20:00:03] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:32 [I 2022-10-26 20:00:03] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00243 [I 2022-10-26 20:00:03] ...trap/engines/engine.py.230: loss: 0.22366 [I 2022-10-26 20:00:03] ...optimizers/trijoint.py.151: total_norm: 14.773626 [I 2022-10-26 20:00:09] ...trap/engines/engine.py.224: train: epoch 27 | batch 530/2304 [I 2022-10-26 20:00:09] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:20 [I 2022-10-26 20:00:09] ...trap/engines/engine.py.229: process: 0.54379 | load: 0.00245 [I 2022-10-26 20:00:09] ...trap/engines/engine.py.230: loss: 0.15572 [I 2022-10-26 20:00:09] ...optimizers/trijoint.py.151: total_norm: 17.645542 [I 2022-10-26 20:00:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 540/2304 [I 2022-10-26 20:00:14] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:22 [I 2022-10-26 20:00:14] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00251 [I 2022-10-26 20:00:14] ...trap/engines/engine.py.230: loss: 0.20409 [I 2022-10-26 20:00:14] ...optimizers/trijoint.py.151: total_norm: 20.774849 [I 2022-10-26 20:00:20] ...trap/engines/engine.py.224: train: epoch 27 | batch 550/2304 [I 2022-10-26 20:00:20] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:11 [I 2022-10-26 20:00:20] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00245 [I 2022-10-26 20:00:20] ...trap/engines/engine.py.230: loss: 0.18632 [I 2022-10-26 20:00:20] ...optimizers/trijoint.py.151: total_norm: 23.075535 [I 2022-10-26 20:00:26] ...trap/engines/engine.py.224: train: epoch 27 | batch 560/2304 [I 2022-10-26 20:00:26] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:14 [I 2022-10-26 20:00:26] ...trap/engines/engine.py.229: process: 0.55284 | load: 0.00246 [I 2022-10-26 20:00:26] ...trap/engines/engine.py.230: loss: 0.19292 [I 2022-10-26 20:00:26] ...optimizers/trijoint.py.151: total_norm: 26.502365 [I 2022-10-26 20:00:31] ...trap/engines/engine.py.224: train: epoch 27 | batch 570/2304 [I 2022-10-26 20:00:31] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:07 [I 2022-10-26 20:00:31] ...trap/engines/engine.py.229: process: 0.54862 | load: 0.00244 [I 2022-10-26 20:00:31] ...trap/engines/engine.py.230: loss: 0.14538 [I 2022-10-26 20:00:31] ...optimizers/trijoint.py.151: total_norm: 10.915564 [I 2022-10-26 20:00:37] ...trap/engines/engine.py.224: train: epoch 27 | batch 580/2304 [I 2022-10-26 20:00:37] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:56 [I 2022-10-26 20:00:37] ...trap/engines/engine.py.229: process: 0.55035 | load: 0.00246 [I 2022-10-26 20:00:37] ...trap/engines/engine.py.230: loss: 0.24791 [I 2022-10-26 20:00:37] ...optimizers/trijoint.py.151: total_norm: 26.548298 [I 2022-10-26 20:00:42] ...trap/engines/engine.py.224: train: epoch 27 | batch 590/2304 [I 2022-10-26 20:00:42] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:54 [I 2022-10-26 20:00:42] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00245 [I 2022-10-26 20:00:42] ...trap/engines/engine.py.230: loss: 0.16179 [I 2022-10-26 20:00:42] ...optimizers/trijoint.py.151: total_norm: 18.623171 [I 2022-10-26 20:00:48] ...trap/engines/engine.py.224: train: epoch 27 | batch 600/2304 [I 2022-10-26 20:00:48] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:47 [I 2022-10-26 20:00:48] ...trap/engines/engine.py.229: process: 0.55230 | load: 0.00246 [I 2022-10-26 20:00:48] ...trap/engines/engine.py.230: loss: 0.17516 [I 2022-10-26 20:00:48] ...optimizers/trijoint.py.151: total_norm: 40.612461 [I 2022-10-26 20:00:54] ...trap/engines/engine.py.224: train: epoch 27 | batch 610/2304 [I 2022-10-26 20:00:54] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:46 [I 2022-10-26 20:00:54] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00249 [I 2022-10-26 20:00:54] ...trap/engines/engine.py.230: loss: 0.21889 [I 2022-10-26 20:00:54] ...optimizers/trijoint.py.151: total_norm: 24.635826 [I 2022-10-26 20:00:59] ...trap/engines/engine.py.224: train: epoch 27 | batch 620/2304 [I 2022-10-26 20:00:59] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:38 [I 2022-10-26 20:00:59] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00244 [I 2022-10-26 20:00:59] ...trap/engines/engine.py.230: loss: 0.24638 [I 2022-10-26 20:00:59] ...optimizers/trijoint.py.151: total_norm: 19.468254 [I 2022-10-26 20:01:05] ...trap/engines/engine.py.224: train: epoch 27 | batch 630/2304 [I 2022-10-26 20:01:05] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:28 [I 2022-10-26 20:01:05] ...trap/engines/engine.py.229: process: 0.55993 | load: 0.00244 [I 2022-10-26 20:01:05] ...trap/engines/engine.py.230: loss: 0.13008 [I 2022-10-26 20:01:05] ...optimizers/trijoint.py.151: total_norm: 14.025708 [I 2022-10-26 20:01:10] ...trap/engines/engine.py.224: train: epoch 27 | batch 640/2304 [I 2022-10-26 20:01:10] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:22 [I 2022-10-26 20:01:10] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00242 [I 2022-10-26 20:01:10] ...trap/engines/engine.py.230: loss: 0.11567 [I 2022-10-26 20:01:10] ...optimizers/trijoint.py.151: total_norm: 13.958385 [I 2022-10-26 20:01:16] ...trap/engines/engine.py.224: train: epoch 27 | batch 650/2304 [I 2022-10-26 20:01:16] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:26 [I 2022-10-26 20:01:16] ...trap/engines/engine.py.229: process: 0.56417 | load: 0.00242 [I 2022-10-26 20:01:16] ...trap/engines/engine.py.230: loss: 0.13963 [I 2022-10-26 20:01:16] ...optimizers/trijoint.py.151: total_norm: 21.848581 [I 2022-10-26 20:01:21] ...trap/engines/engine.py.224: train: epoch 27 | batch 660/2304 [I 2022-10-26 20:01:21] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:12 [I 2022-10-26 20:01:21] ...trap/engines/engine.py.229: process: 0.55895 | load: 0.00249 [I 2022-10-26 20:01:21] ...trap/engines/engine.py.230: loss: 0.13147 [I 2022-10-26 20:01:21] ...optimizers/trijoint.py.151: total_norm: 38.250732 [I 2022-10-26 20:01:27] ...trap/engines/engine.py.224: train: epoch 27 | batch 670/2304 [I 2022-10-26 20:01:27] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:13 [I 2022-10-26 20:01:27] ...trap/engines/engine.py.229: process: 0.56389 | load: 0.00247 [I 2022-10-26 20:01:27] ...trap/engines/engine.py.230: loss: 0.21154 [I 2022-10-26 20:01:27] ...optimizers/trijoint.py.151: total_norm: 19.834435 [I 2022-10-26 20:01:33] ...trap/engines/engine.py.224: train: epoch 27 | batch 680/2304 [I 2022-10-26 20:01:33] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:14:56 [I 2022-10-26 20:01:33] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00251 [I 2022-10-26 20:01:33] ...trap/engines/engine.py.230: loss: 0.14396 [I 2022-10-26 20:01:33] ...optimizers/trijoint.py.151: total_norm: 17.978563 [I 2022-10-26 20:01:38] ...trap/engines/engine.py.224: train: epoch 27 | batch 690/2304 [I 2022-10-26 20:01:38] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:56 [I 2022-10-26 20:01:38] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00243 [I 2022-10-26 20:01:38] ...trap/engines/engine.py.230: loss: 0.28633 [I 2022-10-26 20:01:38] ...optimizers/trijoint.py.151: total_norm: 30.608034 [I 2022-10-26 20:01:44] ...trap/engines/engine.py.224: train: epoch 27 | batch 700/2304 [I 2022-10-26 20:01:44] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:52 [I 2022-10-26 20:01:44] ...trap/engines/engine.py.229: process: 0.56816 | load: 0.00242 [I 2022-10-26 20:01:44] ...trap/engines/engine.py.230: loss: 0.22261 [I 2022-10-26 20:01:44] ...optimizers/trijoint.py.151: total_norm: 20.287825 [I 2022-10-26 20:01:49] ...trap/engines/engine.py.224: train: epoch 27 | batch 710/2304 [I 2022-10-26 20:01:49] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:45 [I 2022-10-26 20:01:49] ...trap/engines/engine.py.229: process: 0.55231 | load: 0.00245 [I 2022-10-26 20:01:49] ...trap/engines/engine.py.230: loss: 0.19505 [I 2022-10-26 20:01:49] ...optimizers/trijoint.py.151: total_norm: 24.078007 [I 2022-10-26 20:01:55] ...trap/engines/engine.py.224: train: epoch 27 | batch 720/2304 [I 2022-10-26 20:01:55] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:36 [I 2022-10-26 20:01:55] ...trap/engines/engine.py.229: process: 0.54248 | load: 0.00244 [I 2022-10-26 20:01:55] ...trap/engines/engine.py.230: loss: 0.14648 [I 2022-10-26 20:01:55] ...optimizers/trijoint.py.151: total_norm: 16.272882 [I 2022-10-26 20:02:00] ...trap/engines/engine.py.224: train: epoch 27 | batch 730/2304 [I 2022-10-26 20:02:00] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:33 [I 2022-10-26 20:02:00] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00241 [I 2022-10-26 20:02:00] ...trap/engines/engine.py.230: loss: 0.14922 [I 2022-10-26 20:02:00] ...optimizers/trijoint.py.151: total_norm: 18.775803 [I 2022-10-26 20:02:06] ...trap/engines/engine.py.224: train: epoch 27 | batch 740/2304 [I 2022-10-26 20:02:06] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:33 [I 2022-10-26 20:02:06] ...trap/engines/engine.py.229: process: 0.56414 | load: 0.00241 [I 2022-10-26 20:02:06] ...trap/engines/engine.py.230: loss: 0.14430 [I 2022-10-26 20:02:06] ...optimizers/trijoint.py.151: total_norm: 16.260588 [I 2022-10-26 20:02:11] ...trap/engines/engine.py.224: train: epoch 27 | batch 750/2304 [I 2022-10-26 20:02:11] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:19 [I 2022-10-26 20:02:11] ...trap/engines/engine.py.229: process: 0.54608 | load: 0.00241 [I 2022-10-26 20:02:12] ...trap/engines/engine.py.230: loss: 0.19451 [I 2022-10-26 20:02:12] ...optimizers/trijoint.py.151: total_norm: 18.662970 [I 2022-10-26 20:02:17] ...trap/engines/engine.py.224: train: epoch 27 | batch 760/2304 [I 2022-10-26 20:02:17] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:20 [I 2022-10-26 20:02:17] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00240 [I 2022-10-26 20:02:17] ...trap/engines/engine.py.230: loss: 0.21655 [I 2022-10-26 20:02:17] ...optimizers/trijoint.py.151: total_norm: 33.971775 [I 2022-10-26 20:02:23] ...trap/engines/engine.py.224: train: epoch 27 | batch 770/2304 [I 2022-10-26 20:02:23] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:10 [I 2022-10-26 20:02:23] ...trap/engines/engine.py.229: process: 0.53677 | load: 0.00245 [I 2022-10-26 20:02:23] ...trap/engines/engine.py.230: loss: 0.20512 [I 2022-10-26 20:02:23] ...optimizers/trijoint.py.151: total_norm: 17.335127 [I 2022-10-26 20:02:28] ...trap/engines/engine.py.224: train: epoch 27 | batch 780/2304 [I 2022-10-26 20:02:28] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:06 [I 2022-10-26 20:02:28] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00250 [I 2022-10-26 20:02:28] ...trap/engines/engine.py.230: loss: 0.17982 [I 2022-10-26 20:02:28] ...optimizers/trijoint.py.151: total_norm: 21.592390 [I 2022-10-26 20:02:34] ...trap/engines/engine.py.224: train: epoch 27 | batch 790/2304 [I 2022-10-26 20:02:34] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:01 [I 2022-10-26 20:02:34] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00248 [I 2022-10-26 20:02:34] ...trap/engines/engine.py.230: loss: 0.17456 [I 2022-10-26 20:02:34] ...optimizers/trijoint.py.151: total_norm: 11.675961 [I 2022-10-26 20:02:39] ...trap/engines/engine.py.224: train: epoch 27 | batch 800/2304 [I 2022-10-26 20:02:39] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:54 [I 2022-10-26 20:02:39] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00244 [I 2022-10-26 20:02:39] ...trap/engines/engine.py.230: loss: 0.20835 [I 2022-10-26 20:02:39] ...optimizers/trijoint.py.151: total_norm: 24.260725 [I 2022-10-26 20:02:45] ...trap/engines/engine.py.224: train: epoch 27 | batch 810/2304 [I 2022-10-26 20:02:45] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:45 [I 2022-10-26 20:02:45] ...trap/engines/engine.py.229: process: 0.54854 | load: 0.00245 [I 2022-10-26 20:02:45] ...trap/engines/engine.py.230: loss: 0.19033 [I 2022-10-26 20:02:45] ...optimizers/trijoint.py.151: total_norm: 18.701689 [I 2022-10-26 20:02:50] ...trap/engines/engine.py.224: train: epoch 27 | batch 820/2304 [I 2022-10-26 20:02:50] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:43 [I 2022-10-26 20:02:50] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00246 [I 2022-10-26 20:02:50] ...trap/engines/engine.py.230: loss: 0.19945 [I 2022-10-26 20:02:50] ...optimizers/trijoint.py.151: total_norm: 17.836380 [I 2022-10-26 20:02:56] ...trap/engines/engine.py.224: train: epoch 27 | batch 830/2304 [I 2022-10-26 20:02:56] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:43 [I 2022-10-26 20:02:56] ...trap/engines/engine.py.229: process: 0.56311 | load: 0.00253 [I 2022-10-26 20:02:56] ...trap/engines/engine.py.230: loss: 0.22471 [I 2022-10-26 20:02:56] ...optimizers/trijoint.py.151: total_norm: 26.775686 [I 2022-10-26 20:03:02] ...trap/engines/engine.py.224: train: epoch 27 | batch 840/2304 [I 2022-10-26 20:03:02] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:38 [I 2022-10-26 20:03:02] ...trap/engines/engine.py.229: process: 0.56245 | load: 0.00245 [I 2022-10-26 20:03:02] ...trap/engines/engine.py.230: loss: 0.21991 [I 2022-10-26 20:03:02] ...optimizers/trijoint.py.151: total_norm: 16.473568 [I 2022-10-26 20:03:07] ...trap/engines/engine.py.224: train: epoch 27 | batch 850/2304 [I 2022-10-26 20:03:07] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:21 [I 2022-10-26 20:03:07] ...trap/engines/engine.py.229: process: 0.54720 | load: 0.00249 [I 2022-10-26 20:03:07] ...trap/engines/engine.py.230: loss: 0.15070 [I 2022-10-26 20:03:07] ...optimizers/trijoint.py.151: total_norm: 14.952615 [I 2022-10-26 20:03:13] ...trap/engines/engine.py.224: train: epoch 27 | batch 860/2304 [I 2022-10-26 20:03:13] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:21 [I 2022-10-26 20:03:13] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00243 [I 2022-10-26 20:03:13] ...trap/engines/engine.py.230: loss: 0.24365 [I 2022-10-26 20:03:13] ...optimizers/trijoint.py.151: total_norm: 24.976828 [I 2022-10-26 20:03:18] ...trap/engines/engine.py.224: train: epoch 27 | batch 870/2304 [I 2022-10-26 20:03:18] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:16 [I 2022-10-26 20:03:18] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00240 [I 2022-10-26 20:03:18] ...trap/engines/engine.py.230: loss: 0.25237 [I 2022-10-26 20:03:18] ...optimizers/trijoint.py.151: total_norm: 35.472530 [I 2022-10-26 20:03:24] ...trap/engines/engine.py.224: train: epoch 27 | batch 880/2304 [I 2022-10-26 20:03:24] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:04 [I 2022-10-26 20:03:24] ...trap/engines/engine.py.229: process: 0.54338 | load: 0.00267 [I 2022-10-26 20:03:24] ...trap/engines/engine.py.230: loss: 0.16495 [I 2022-10-26 20:03:24] ...optimizers/trijoint.py.151: total_norm: 18.225584 [I 2022-10-26 20:03:29] ...trap/engines/engine.py.224: train: epoch 27 | batch 890/2304 [I 2022-10-26 20:03:29] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:10 [I 2022-10-26 20:03:29] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00246 [I 2022-10-26 20:03:29] ...trap/engines/engine.py.230: loss: 0.13961 [I 2022-10-26 20:03:29] ...optimizers/trijoint.py.151: total_norm: 11.355566 [I 2022-10-26 20:03:35] ...trap/engines/engine.py.224: train: epoch 27 | batch 900/2304 [I 2022-10-26 20:03:35] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:03 [I 2022-10-26 20:03:35] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00242 [I 2022-10-26 20:03:35] ...trap/engines/engine.py.230: loss: 0.24111 [I 2022-10-26 20:03:35] ...optimizers/trijoint.py.151: total_norm: 20.599621 [I 2022-10-26 20:03:41] ...trap/engines/engine.py.224: train: epoch 27 | batch 910/2304 [I 2022-10-26 20:03:41] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:57 [I 2022-10-26 20:03:41] ...trap/engines/engine.py.229: process: 0.56603 | load: 0.00242 [I 2022-10-26 20:03:41] ...trap/engines/engine.py.230: loss: 0.17775 [I 2022-10-26 20:03:41] ...optimizers/trijoint.py.151: total_norm: 27.245245 [I 2022-10-26 20:03:46] ...trap/engines/engine.py.224: train: epoch 27 | batch 920/2304 [I 2022-10-26 20:03:46] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:42 [I 2022-10-26 20:03:46] ...trap/engines/engine.py.229: process: 0.54658 | load: 0.00242 [I 2022-10-26 20:03:46] ...trap/engines/engine.py.230: loss: 0.12900 [I 2022-10-26 20:03:46] ...optimizers/trijoint.py.151: total_norm: 16.115917 [I 2022-10-26 20:03:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 930/2304 [I 2022-10-26 20:03:52] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:12:42 [I 2022-10-26 20:03:52] ...trap/engines/engine.py.229: process: 0.54974 | load: 0.00259 [I 2022-10-26 20:03:52] ...trap/engines/engine.py.230: loss: 0.23737 [I 2022-10-26 20:03:52] ...optimizers/trijoint.py.151: total_norm: 16.093847 [I 2022-10-26 20:03:57] ...trap/engines/engine.py.224: train: epoch 27 | batch 940/2304 [I 2022-10-26 20:03:57] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:34 [I 2022-10-26 20:03:57] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00252 [I 2022-10-26 20:03:57] ...trap/engines/engine.py.230: loss: 0.17760 [I 2022-10-26 20:03:57] ...optimizers/trijoint.py.151: total_norm: 17.021084 [I 2022-10-26 20:04:03] ...trap/engines/engine.py.224: train: epoch 27 | batch 950/2304 [I 2022-10-26 20:04:03] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:35 [I 2022-10-26 20:04:03] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00248 [I 2022-10-26 20:04:03] ...trap/engines/engine.py.230: loss: 0.12956 [I 2022-10-26 20:04:03] ...optimizers/trijoint.py.151: total_norm: 15.792336 [I 2022-10-26 20:04:08] ...trap/engines/engine.py.224: train: epoch 27 | batch 960/2304 [I 2022-10-26 20:04:08] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:24 [I 2022-10-26 20:04:08] ...trap/engines/engine.py.229: process: 0.54966 | load: 0.00243 [I 2022-10-26 20:04:08] ...trap/engines/engine.py.230: loss: 0.20008 [I 2022-10-26 20:04:08] ...optimizers/trijoint.py.151: total_norm: 20.833246 [I 2022-10-26 20:04:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 970/2304 [I 2022-10-26 20:04:14] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:20 [I 2022-10-26 20:04:14] ...trap/engines/engine.py.229: process: 0.55618 | load: 0.00241 [I 2022-10-26 20:04:14] ...trap/engines/engine.py.230: loss: 0.16361 [I 2022-10-26 20:04:14] ...optimizers/trijoint.py.151: total_norm: 15.031323 [I 2022-10-26 20:04:20] ...trap/engines/engine.py.224: train: epoch 27 | batch 980/2304 [I 2022-10-26 20:04:20] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:16 [I 2022-10-26 20:04:20] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00244 [I 2022-10-26 20:04:20] ...trap/engines/engine.py.230: loss: 0.18818 [I 2022-10-26 20:04:20] ...optimizers/trijoint.py.151: total_norm: 20.801147 [I 2022-10-26 20:04:25] ...trap/engines/engine.py.224: train: epoch 27 | batch 990/2304 [I 2022-10-26 20:04:25] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:12 [I 2022-10-26 20:04:25] ...trap/engines/engine.py.229: process: 0.55043 | load: 0.00251 [I 2022-10-26 20:04:25] ...trap/engines/engine.py.230: loss: 0.19522 [I 2022-10-26 20:04:25] ...optimizers/trijoint.py.151: total_norm: 27.038044 [I 2022-10-26 20:04:31] ...trap/engines/engine.py.224: train: epoch 27 | batch 1000/2304 [I 2022-10-26 20:04:31] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:05 [I 2022-10-26 20:04:31] ...trap/engines/engine.py.229: process: 0.54958 | load: 0.00251 [I 2022-10-26 20:04:31] ...trap/engines/engine.py.230: loss: 0.21798 [I 2022-10-26 20:04:31] ...optimizers/trijoint.py.151: total_norm: 21.635038 [I 2022-10-26 20:04:36] ...trap/engines/engine.py.224: train: epoch 27 | batch 1010/2304 [I 2022-10-26 20:04:36] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:01 [I 2022-10-26 20:04:36] ...trap/engines/engine.py.229: process: 0.56817 | load: 0.00247 [I 2022-10-26 20:04:36] ...trap/engines/engine.py.230: loss: 0.23305 [I 2022-10-26 20:04:36] ...optimizers/trijoint.py.151: total_norm: 17.062578 [I 2022-10-26 20:04:42] ...trap/engines/engine.py.224: train: epoch 27 | batch 1020/2304 [I 2022-10-26 20:04:42] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:53 [I 2022-10-26 20:04:42] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00246 [I 2022-10-26 20:04:42] ...trap/engines/engine.py.230: loss: 0.18712 [I 2022-10-26 20:04:42] ...optimizers/trijoint.py.151: total_norm: 24.198717 [I 2022-10-26 20:04:47] ...trap/engines/engine.py.224: train: epoch 27 | batch 1030/2304 [I 2022-10-26 20:04:47] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:45 [I 2022-10-26 20:04:47] ...trap/engines/engine.py.229: process: 0.55006 | load: 0.00246 [I 2022-10-26 20:04:47] ...trap/engines/engine.py.230: loss: 0.19167 [I 2022-10-26 20:04:47] ...optimizers/trijoint.py.151: total_norm: 15.192510 [I 2022-10-26 20:04:53] ...trap/engines/engine.py.224: train: epoch 27 | batch 1040/2304 [I 2022-10-26 20:04:53] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:43 [I 2022-10-26 20:04:53] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00246 [I 2022-10-26 20:04:53] ...trap/engines/engine.py.230: loss: 0.21073 [I 2022-10-26 20:04:53] ...optimizers/trijoint.py.151: total_norm: 44.199543 [I 2022-10-26 20:04:59] ...trap/engines/engine.py.224: train: epoch 27 | batch 1050/2304 [I 2022-10-26 20:04:59] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:37 [I 2022-10-26 20:04:59] ...trap/engines/engine.py.229: process: 0.55986 | load: 0.00242 [I 2022-10-26 20:04:59] ...trap/engines/engine.py.230: loss: 0.13554 [I 2022-10-26 20:04:59] ...optimizers/trijoint.py.151: total_norm: 19.457748 [I 2022-10-26 20:05:04] ...trap/engines/engine.py.224: train: epoch 27 | batch 1060/2304 [I 2022-10-26 20:05:04] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:31 [I 2022-10-26 20:05:04] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00244 [I 2022-10-26 20:05:04] ...trap/engines/engine.py.230: loss: 0.12159 [I 2022-10-26 20:05:04] ...optimizers/trijoint.py.151: total_norm: 15.662962 [I 2022-10-26 20:05:10] ...trap/engines/engine.py.224: train: epoch 27 | batch 1070/2304 [I 2022-10-26 20:05:10] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:31 [I 2022-10-26 20:05:10] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00252 [I 2022-10-26 20:05:10] ...trap/engines/engine.py.230: loss: 0.16268 [I 2022-10-26 20:05:10] ...optimizers/trijoint.py.151: total_norm: 12.512037 [I 2022-10-26 20:05:15] ...trap/engines/engine.py.224: train: epoch 27 | batch 1080/2304 [I 2022-10-26 20:05:15] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:21 [I 2022-10-26 20:05:15] ...trap/engines/engine.py.229: process: 0.55101 | load: 0.00247 [I 2022-10-26 20:05:15] ...trap/engines/engine.py.230: loss: 0.20688 [I 2022-10-26 20:05:15] ...optimizers/trijoint.py.151: total_norm: 20.315529 [I 2022-10-26 20:05:21] ...trap/engines/engine.py.224: train: epoch 27 | batch 1090/2304 [I 2022-10-26 20:05:21] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:15 [I 2022-10-26 20:05:21] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00245 [I 2022-10-26 20:05:21] ...trap/engines/engine.py.230: loss: 0.19112 [I 2022-10-26 20:05:21] ...optimizers/trijoint.py.151: total_norm: 29.181217 [I 2022-10-26 20:05:27] ...trap/engines/engine.py.224: train: epoch 27 | batch 1100/2304 [I 2022-10-26 20:05:27] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:14 [I 2022-10-26 20:05:27] ...trap/engines/engine.py.229: process: 0.56497 | load: 0.00243 [I 2022-10-26 20:05:27] ...trap/engines/engine.py.230: loss: 0.17806 [I 2022-10-26 20:05:27] ...optimizers/trijoint.py.151: total_norm: 18.712811 [I 2022-10-26 20:05:32] ...trap/engines/engine.py.224: train: epoch 27 | batch 1110/2304 [I 2022-10-26 20:05:32] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:18 [I 2022-10-26 20:05:32] ...trap/engines/engine.py.229: process: 0.55756 | load: 0.00244 [I 2022-10-26 20:05:32] ...trap/engines/engine.py.230: loss: 0.13933 [I 2022-10-26 20:05:32] ...optimizers/trijoint.py.151: total_norm: 27.765987 [I 2022-10-26 20:05:38] ...trap/engines/engine.py.224: train: epoch 27 | batch 1120/2304 [I 2022-10-26 20:05:38] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:10:58 [I 2022-10-26 20:05:38] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00254 [I 2022-10-26 20:05:38] ...trap/engines/engine.py.230: loss: 0.15348 [I 2022-10-26 20:05:38] ...optimizers/trijoint.py.151: total_norm: 9.701394 [I 2022-10-26 20:05:43] ...trap/engines/engine.py.224: train: epoch 27 | batch 1130/2304 [I 2022-10-26 20:05:43] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:52 [I 2022-10-26 20:05:43] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00252 [I 2022-10-26 20:05:43] ...trap/engines/engine.py.230: loss: 0.20041 [I 2022-10-26 20:05:43] ...optimizers/trijoint.py.151: total_norm: 21.767466 [I 2022-10-26 20:05:49] ...trap/engines/engine.py.224: train: epoch 27 | batch 1140/2304 [I 2022-10-26 20:05:49] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:43 [I 2022-10-26 20:05:49] ...trap/engines/engine.py.229: process: 0.54139 | load: 0.00247 [I 2022-10-26 20:05:49] ...trap/engines/engine.py.230: loss: 0.16356 [I 2022-10-26 20:05:49] ...optimizers/trijoint.py.151: total_norm: 17.236979 [I 2022-10-26 20:05:55] ...trap/engines/engine.py.224: train: epoch 27 | batch 1150/2304 [I 2022-10-26 20:05:55] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:10:36 [I 2022-10-26 20:05:55] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00245 [I 2022-10-26 20:05:55] ...trap/engines/engine.py.230: loss: 0.11299 [I 2022-10-26 20:05:55] ...optimizers/trijoint.py.151: total_norm: 18.059134 [I 2022-10-26 20:06:00] ...trap/engines/engine.py.224: train: epoch 27 | batch 1160/2304 [I 2022-10-26 20:06:00] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:35 [I 2022-10-26 20:06:00] ...trap/engines/engine.py.229: process: 0.54851 | load: 0.00265 [I 2022-10-26 20:06:00] ...trap/engines/engine.py.230: loss: 0.15435 [I 2022-10-26 20:06:00] ...optimizers/trijoint.py.151: total_norm: 16.643999 [I 2022-10-26 20:06:06] ...trap/engines/engine.py.224: train: epoch 27 | batch 1170/2304 [I 2022-10-26 20:06:06] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:32 [I 2022-10-26 20:06:06] ...trap/engines/engine.py.229: process: 0.56856 | load: 0.00243 [I 2022-10-26 20:06:06] ...trap/engines/engine.py.230: loss: 0.16908 [I 2022-10-26 20:06:06] ...optimizers/trijoint.py.151: total_norm: 23.646597 [I 2022-10-26 20:06:11] ...trap/engines/engine.py.224: train: epoch 27 | batch 1180/2304 [I 2022-10-26 20:06:11] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:22 [I 2022-10-26 20:06:11] ...trap/engines/engine.py.229: process: 0.54981 | load: 0.00244 [I 2022-10-26 20:06:11] ...trap/engines/engine.py.230: loss: 0.14071 [I 2022-10-26 20:06:11] ...optimizers/trijoint.py.151: total_norm: 17.089886 [I 2022-10-26 20:06:17] ...trap/engines/engine.py.224: train: epoch 27 | batch 1190/2304 [I 2022-10-26 20:06:17] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:29 [I 2022-10-26 20:06:17] ...trap/engines/engine.py.229: process: 0.56683 | load: 0.00244 [I 2022-10-26 20:06:17] ...trap/engines/engine.py.230: loss: 0.24986 [I 2022-10-26 20:06:17] ...optimizers/trijoint.py.151: total_norm: 18.577381 [I 2022-10-26 20:06:22] ...trap/engines/engine.py.224: train: epoch 27 | batch 1200/2304 [I 2022-10-26 20:06:22] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:14 [I 2022-10-26 20:06:22] ...trap/engines/engine.py.229: process: 0.57158 | load: 0.00244 [I 2022-10-26 20:06:22] ...trap/engines/engine.py.230: loss: 0.13818 [I 2022-10-26 20:06:22] ...optimizers/trijoint.py.151: total_norm: 21.475752 [I 2022-10-26 20:06:28] ...trap/engines/engine.py.224: train: epoch 27 | batch 1210/2304 [I 2022-10-26 20:06:28] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:07 [I 2022-10-26 20:06:28] ...trap/engines/engine.py.229: process: 0.54475 | load: 0.00250 [I 2022-10-26 20:06:28] ...trap/engines/engine.py.230: loss: 0.12504 [I 2022-10-26 20:06:28] ...optimizers/trijoint.py.151: total_norm: 16.710297 [I 2022-10-26 20:06:34] ...trap/engines/engine.py.224: train: epoch 27 | batch 1220/2304 [I 2022-10-26 20:06:34] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:07 [I 2022-10-26 20:06:34] ...trap/engines/engine.py.229: process: 0.55428 | load: 0.00268 [I 2022-10-26 20:06:34] ...trap/engines/engine.py.230: loss: 0.21656 [I 2022-10-26 20:06:34] ...optimizers/trijoint.py.151: total_norm: 30.084703 [I 2022-10-26 20:06:39] ...trap/engines/engine.py.224: train: epoch 27 | batch 1230/2304 [I 2022-10-26 20:06:39] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:09:57 [I 2022-10-26 20:06:39] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00247 [I 2022-10-26 20:06:39] ...trap/engines/engine.py.230: loss: 0.20171 [I 2022-10-26 20:06:39] ...optimizers/trijoint.py.151: total_norm: 14.727210 [I 2022-10-26 20:06:45] ...trap/engines/engine.py.224: train: epoch 27 | batch 1240/2304 [I 2022-10-26 20:06:45] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:52 [I 2022-10-26 20:06:45] ...trap/engines/engine.py.229: process: 0.57011 | load: 0.00244 [I 2022-10-26 20:06:45] ...trap/engines/engine.py.230: loss: 0.11520 [I 2022-10-26 20:06:45] ...optimizers/trijoint.py.151: total_norm: 16.747688 [I 2022-10-26 20:06:50] ...trap/engines/engine.py.224: train: epoch 27 | batch 1250/2304 [I 2022-10-26 20:06:50] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:10:00 [I 2022-10-26 20:06:50] ...trap/engines/engine.py.229: process: 0.62027 | load: 0.00247 [I 2022-10-26 20:06:50] ...trap/engines/engine.py.230: loss: 0.21197 [I 2022-10-26 20:06:50] ...optimizers/trijoint.py.151: total_norm: 31.168030 [I 2022-10-26 20:06:56] ...trap/engines/engine.py.224: train: epoch 27 | batch 1260/2304 [I 2022-10-26 20:06:56] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:43 [I 2022-10-26 20:06:56] ...trap/engines/engine.py.229: process: 0.56101 | load: 0.00253 [I 2022-10-26 20:06:56] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-26 20:06:56] ...optimizers/trijoint.py.151: total_norm: 18.673653 [I 2022-10-26 20:07:02] ...trap/engines/engine.py.224: train: epoch 27 | batch 1270/2304 [I 2022-10-26 20:07:02] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:28 [I 2022-10-26 20:07:02] ...trap/engines/engine.py.229: process: 0.53583 | load: 0.00252 [I 2022-10-26 20:07:02] ...trap/engines/engine.py.230: loss: 0.22711 [I 2022-10-26 20:07:02] ...optimizers/trijoint.py.151: total_norm: 37.018806 [I 2022-10-26 20:07:07] ...trap/engines/engine.py.224: train: epoch 27 | batch 1280/2304 [I 2022-10-26 20:07:07] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:26 [I 2022-10-26 20:07:07] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00245 [I 2022-10-26 20:07:07] ...trap/engines/engine.py.230: loss: 0.14735 [I 2022-10-26 20:07:07] ...optimizers/trijoint.py.151: total_norm: 22.722452 [I 2022-10-26 20:07:13] ...trap/engines/engine.py.224: train: epoch 27 | batch 1290/2304 [I 2022-10-26 20:07:13] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:20 [I 2022-10-26 20:07:13] ...trap/engines/engine.py.229: process: 0.55217 | load: 0.00246 [I 2022-10-26 20:07:13] ...trap/engines/engine.py.230: loss: 0.15982 [I 2022-10-26 20:07:13] ...optimizers/trijoint.py.151: total_norm: 14.294710 [I 2022-10-26 20:07:18] ...trap/engines/engine.py.224: train: epoch 27 | batch 1300/2304 [I 2022-10-26 20:07:18] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:17 [I 2022-10-26 20:07:18] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00243 [I 2022-10-26 20:07:18] ...trap/engines/engine.py.230: loss: 0.21084 [I 2022-10-26 20:07:18] ...optimizers/trijoint.py.151: total_norm: 41.886623 [I 2022-10-26 20:07:24] ...trap/engines/engine.py.224: train: epoch 27 | batch 1310/2304 [I 2022-10-26 20:07:24] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:16 [I 2022-10-26 20:07:24] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00248 [I 2022-10-26 20:07:24] ...trap/engines/engine.py.230: loss: 0.12236 [I 2022-10-26 20:07:24] ...optimizers/trijoint.py.151: total_norm: 14.593122 [I 2022-10-26 20:07:30] ...trap/engines/engine.py.224: train: epoch 27 | batch 1320/2304 [I 2022-10-26 20:07:30] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:17 [I 2022-10-26 20:07:30] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00243 [I 2022-10-26 20:07:30] ...trap/engines/engine.py.230: loss: 0.20396 [I 2022-10-26 20:07:30] ...optimizers/trijoint.py.151: total_norm: 21.065853 [I 2022-10-26 20:07:35] ...trap/engines/engine.py.224: train: epoch 27 | batch 1330/2304 [I 2022-10-26 20:07:35] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:00 [I 2022-10-26 20:07:35] ...trap/engines/engine.py.229: process: 0.55679 | load: 0.00247 [I 2022-10-26 20:07:35] ...trap/engines/engine.py.230: loss: 0.35923 [I 2022-10-26 20:07:35] ...optimizers/trijoint.py.151: total_norm: 33.514343 [I 2022-10-26 20:07:41] ...trap/engines/engine.py.224: train: epoch 27 | batch 1340/2304 [I 2022-10-26 20:07:41] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:08:51 [I 2022-10-26 20:07:41] ...trap/engines/engine.py.229: process: 0.54897 | load: 0.00245 [I 2022-10-26 20:07:41] ...trap/engines/engine.py.230: loss: 0.19936 [I 2022-10-26 20:07:41] ...optimizers/trijoint.py.151: total_norm: 33.675156 [I 2022-10-26 20:07:46] ...trap/engines/engine.py.224: train: epoch 27 | batch 1350/2304 [I 2022-10-26 20:07:46] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:50 [I 2022-10-26 20:07:46] ...trap/engines/engine.py.229: process: 0.54349 | load: 0.00250 [I 2022-10-26 20:07:46] ...trap/engines/engine.py.230: loss: 0.19131 [I 2022-10-26 20:07:46] ...optimizers/trijoint.py.151: total_norm: 21.341022 [I 2022-10-26 20:07:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 1360/2304 [I 2022-10-26 20:07:52] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:45 [I 2022-10-26 20:07:52] ...trap/engines/engine.py.229: process: 0.55320 | load: 0.00248 [I 2022-10-26 20:07:52] ...trap/engines/engine.py.230: loss: 0.15142 [I 2022-10-26 20:07:52] ...optimizers/trijoint.py.151: total_norm: 18.213171 [I 2022-10-26 20:07:57] ...trap/engines/engine.py.224: train: epoch 27 | batch 1370/2304 [I 2022-10-26 20:07:57] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:36 [I 2022-10-26 20:07:57] ...trap/engines/engine.py.229: process: 0.54641 | load: 0.00268 [I 2022-10-26 20:07:57] ...trap/engines/engine.py.230: loss: 0.12414 [I 2022-10-26 20:07:57] ...optimizers/trijoint.py.151: total_norm: 17.138615 [I 2022-10-26 20:08:03] ...trap/engines/engine.py.224: train: epoch 27 | batch 1380/2304 [I 2022-10-26 20:08:03] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:37 [I 2022-10-26 20:08:03] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00257 [I 2022-10-26 20:08:03] ...trap/engines/engine.py.230: loss: 0.13380 [I 2022-10-26 20:08:03] ...optimizers/trijoint.py.151: total_norm: 33.494797 [I 2022-10-26 20:08:09] ...trap/engines/engine.py.224: train: epoch 27 | batch 1390/2304 [I 2022-10-26 20:08:09] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:29 [I 2022-10-26 20:08:09] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00249 [I 2022-10-26 20:08:09] ...trap/engines/engine.py.230: loss: 0.16200 [I 2022-10-26 20:08:09] ...optimizers/trijoint.py.151: total_norm: 23.406231 [I 2022-10-26 20:08:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 1400/2304 [I 2022-10-26 20:08:14] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:25 [I 2022-10-26 20:08:14] ...trap/engines/engine.py.229: process: 0.56911 | load: 0.00254 [I 2022-10-26 20:08:14] ...trap/engines/engine.py.230: loss: 0.17752 [I 2022-10-26 20:08:14] ...optimizers/trijoint.py.151: total_norm: 14.290291 [I 2022-10-26 20:08:20] ...trap/engines/engine.py.224: train: epoch 27 | batch 1410/2304 [I 2022-10-26 20:08:20] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:16 [I 2022-10-26 20:08:20] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00245 [I 2022-10-26 20:08:20] ...trap/engines/engine.py.230: loss: 0.18875 [I 2022-10-26 20:08:20] ...optimizers/trijoint.py.151: total_norm: 17.199749 [I 2022-10-26 20:08:25] ...trap/engines/engine.py.224: train: epoch 27 | batch 1420/2304 [I 2022-10-26 20:08:25] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:10 [I 2022-10-26 20:08:25] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00243 [I 2022-10-26 20:08:25] ...trap/engines/engine.py.230: loss: 0.15518 [I 2022-10-26 20:08:25] ...optimizers/trijoint.py.151: total_norm: 16.724211 [I 2022-10-26 20:08:31] ...trap/engines/engine.py.224: train: epoch 27 | batch 1430/2304 [I 2022-10-26 20:08:31] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:01 [I 2022-10-26 20:08:31] ...trap/engines/engine.py.229: process: 0.54145 | load: 0.00244 [I 2022-10-26 20:08:31] ...trap/engines/engine.py.230: loss: 0.18253 [I 2022-10-26 20:08:31] ...optimizers/trijoint.py.151: total_norm: 12.777902 [I 2022-10-26 20:08:36] ...trap/engines/engine.py.224: train: epoch 27 | batch 1440/2304 [I 2022-10-26 20:08:36] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:00 [I 2022-10-26 20:08:36] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00246 [I 2022-10-26 20:08:36] ...trap/engines/engine.py.230: loss: 0.23885 [I 2022-10-26 20:08:36] ...optimizers/trijoint.py.151: total_norm: 22.170525 [I 2022-10-26 20:08:42] ...trap/engines/engine.py.224: train: epoch 27 | batch 1450/2304 [I 2022-10-26 20:08:42] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:08:04 [I 2022-10-26 20:08:42] ...trap/engines/engine.py.229: process: 0.54673 | load: 0.00245 [I 2022-10-26 20:08:42] ...trap/engines/engine.py.230: loss: 0.18548 [I 2022-10-26 20:08:42] ...optimizers/trijoint.py.151: total_norm: 28.289812 [I 2022-10-26 20:08:48] ...trap/engines/engine.py.224: train: epoch 27 | batch 1460/2304 [I 2022-10-26 20:08:48] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:07:49 [I 2022-10-26 20:08:48] ...trap/engines/engine.py.229: process: 0.55262 | load: 0.00253 [I 2022-10-26 20:08:48] ...trap/engines/engine.py.230: loss: 0.22628 [I 2022-10-26 20:08:48] ...optimizers/trijoint.py.151: total_norm: 20.434772 [I 2022-10-26 20:08:53] ...trap/engines/engine.py.224: train: epoch 27 | batch 1470/2304 [I 2022-10-26 20:08:53] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:07:45 [I 2022-10-26 20:08:53] ...trap/engines/engine.py.229: process: 0.55679 | load: 0.00249 [I 2022-10-26 20:08:53] ...trap/engines/engine.py.230: loss: 0.20544 [I 2022-10-26 20:08:53] ...optimizers/trijoint.py.151: total_norm: 47.903362 [I 2022-10-26 20:08:59] ...trap/engines/engine.py.224: train: epoch 27 | batch 1480/2304 [I 2022-10-26 20:08:59] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:38 [I 2022-10-26 20:08:59] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00244 [I 2022-10-26 20:08:59] ...trap/engines/engine.py.230: loss: 0.18494 [I 2022-10-26 20:08:59] ...optimizers/trijoint.py.151: total_norm: 20.173172 [I 2022-10-26 20:09:04] ...trap/engines/engine.py.224: train: epoch 27 | batch 1490/2304 [I 2022-10-26 20:09:04] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:31 [I 2022-10-26 20:09:04] ...trap/engines/engine.py.229: process: 0.56157 | load: 0.00244 [I 2022-10-26 20:09:04] ...trap/engines/engine.py.230: loss: 0.15310 [I 2022-10-26 20:09:04] ...optimizers/trijoint.py.151: total_norm: 18.626095 [I 2022-10-26 20:09:10] ...trap/engines/engine.py.224: train: epoch 27 | batch 1500/2304 [I 2022-10-26 20:09:10] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:25 [I 2022-10-26 20:09:10] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00243 [I 2022-10-26 20:09:10] ...trap/engines/engine.py.230: loss: 0.21382 [I 2022-10-26 20:09:10] ...optimizers/trijoint.py.151: total_norm: 33.765762 [I 2022-10-26 20:09:16] ...trap/engines/engine.py.224: train: epoch 27 | batch 1510/2304 [I 2022-10-26 20:09:16] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:36 [I 2022-10-26 20:09:16] ...trap/engines/engine.py.229: process: 0.66046 | load: 0.00248 [I 2022-10-26 20:09:16] ...trap/engines/engine.py.230: loss: 0.12004 [I 2022-10-26 20:09:16] ...optimizers/trijoint.py.151: total_norm: 14.575962 [I 2022-10-26 20:09:21] ...trap/engines/engine.py.224: train: epoch 27 | batch 1520/2304 [I 2022-10-26 20:09:21] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:16 [I 2022-10-26 20:09:21] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00246 [I 2022-10-26 20:09:21] ...trap/engines/engine.py.230: loss: 0.19841 [I 2022-10-26 20:09:21] ...optimizers/trijoint.py.151: total_norm: 24.485928 [I 2022-10-26 20:09:27] ...trap/engines/engine.py.224: train: epoch 27 | batch 1530/2304 [I 2022-10-26 20:09:27] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:08 [I 2022-10-26 20:09:27] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00245 [I 2022-10-26 20:09:27] ...trap/engines/engine.py.230: loss: 0.20051 [I 2022-10-26 20:09:27] ...optimizers/trijoint.py.151: total_norm: 20.382627 [I 2022-10-26 20:09:32] ...trap/engines/engine.py.224: train: epoch 27 | batch 1540/2304 [I 2022-10-26 20:09:32] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:06:59 [I 2022-10-26 20:09:32] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00247 [I 2022-10-26 20:09:32] ...trap/engines/engine.py.230: loss: 0.20507 [I 2022-10-26 20:09:32] ...optimizers/trijoint.py.151: total_norm: 25.430628 [I 2022-10-26 20:09:38] ...trap/engines/engine.py.224: train: epoch 27 | batch 1550/2304 [I 2022-10-26 20:09:38] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:06:58 [I 2022-10-26 20:09:38] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00244 [I 2022-10-26 20:09:38] ...trap/engines/engine.py.230: loss: 0.22268 [I 2022-10-26 20:09:38] ...optimizers/trijoint.py.151: total_norm: 17.201880 [I 2022-10-26 20:09:43] ...trap/engines/engine.py.224: train: epoch 27 | batch 1560/2304 [I 2022-10-26 20:09:43] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:06:51 [I 2022-10-26 20:09:43] ...trap/engines/engine.py.229: process: 0.55035 | load: 0.00245 [I 2022-10-26 20:09:43] ...trap/engines/engine.py.230: loss: 0.16757 [I 2022-10-26 20:09:43] ...optimizers/trijoint.py.151: total_norm: 17.883905 [I 2022-10-26 20:09:49] ...trap/engines/engine.py.224: train: epoch 27 | batch 1570/2304 [I 2022-10-26 20:09:49] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:06:49 [I 2022-10-26 20:09:49] ...trap/engines/engine.py.229: process: 0.56548 | load: 0.00253 [I 2022-10-26 20:09:49] ...trap/engines/engine.py.230: loss: 0.17613 [I 2022-10-26 20:09:49] ...optimizers/trijoint.py.151: total_norm: 22.009329 [I 2022-10-26 20:09:55] ...trap/engines/engine.py.224: train: epoch 27 | batch 1580/2304 [I 2022-10-26 20:09:55] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:06:44 [I 2022-10-26 20:09:55] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00244 [I 2022-10-26 20:09:55] ...trap/engines/engine.py.230: loss: 0.20370 [I 2022-10-26 20:09:55] ...optimizers/trijoint.py.151: total_norm: 28.833035 [I 2022-10-26 20:10:00] ...trap/engines/engine.py.224: train: epoch 27 | batch 1590/2304 [I 2022-10-26 20:10:00] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:38 [I 2022-10-26 20:10:00] ...trap/engines/engine.py.229: process: 0.54975 | load: 0.00247 [I 2022-10-26 20:10:00] ...trap/engines/engine.py.230: loss: 0.23837 [I 2022-10-26 20:10:00] ...optimizers/trijoint.py.151: total_norm: 14.945315 [I 2022-10-26 20:10:06] ...trap/engines/engine.py.224: train: epoch 27 | batch 1600/2304 [I 2022-10-26 20:10:06] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:31 [I 2022-10-26 20:10:06] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00244 [I 2022-10-26 20:10:06] ...trap/engines/engine.py.230: loss: 0.24403 [I 2022-10-26 20:10:06] ...optimizers/trijoint.py.151: total_norm: 34.406200 [I 2022-10-26 20:10:11] ...trap/engines/engine.py.224: train: epoch 27 | batch 1610/2304 [I 2022-10-26 20:10:11] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:26 [I 2022-10-26 20:10:11] ...trap/engines/engine.py.229: process: 0.55160 | load: 0.00247 [I 2022-10-26 20:10:11] ...trap/engines/engine.py.230: loss: 0.16436 [I 2022-10-26 20:10:11] ...optimizers/trijoint.py.151: total_norm: 19.193193 [I 2022-10-26 20:10:17] ...trap/engines/engine.py.224: train: epoch 27 | batch 1620/2304 [I 2022-10-26 20:10:17] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:20 [I 2022-10-26 20:10:17] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00243 [I 2022-10-26 20:10:17] ...trap/engines/engine.py.230: loss: 0.20042 [I 2022-10-26 20:10:17] ...optimizers/trijoint.py.151: total_norm: 28.594980 [I 2022-10-26 20:10:22] ...trap/engines/engine.py.224: train: epoch 27 | batch 1630/2304 [I 2022-10-26 20:10:22] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:15 [I 2022-10-26 20:10:22] ...trap/engines/engine.py.229: process: 0.56306 | load: 0.00254 [I 2022-10-26 20:10:22] ...trap/engines/engine.py.230: loss: 0.26332 [I 2022-10-26 20:10:22] ...optimizers/trijoint.py.151: total_norm: 20.988388 [I 2022-10-26 20:10:28] ...trap/engines/engine.py.224: train: epoch 27 | batch 1640/2304 [I 2022-10-26 20:10:28] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:15 [I 2022-10-26 20:10:28] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00250 [I 2022-10-26 20:10:28] ...trap/engines/engine.py.230: loss: 0.16956 [I 2022-10-26 20:10:28] ...optimizers/trijoint.py.151: total_norm: 11.353736 [I 2022-10-26 20:10:34] ...trap/engines/engine.py.224: train: epoch 27 | batch 1650/2304 [I 2022-10-26 20:10:34] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:04 [I 2022-10-26 20:10:34] ...trap/engines/engine.py.229: process: 0.54985 | load: 0.00245 [I 2022-10-26 20:10:34] ...trap/engines/engine.py.230: loss: 0.19991 [I 2022-10-26 20:10:34] ...optimizers/trijoint.py.151: total_norm: 22.321449 [I 2022-10-26 20:10:39] ...trap/engines/engine.py.224: train: epoch 27 | batch 1660/2304 [I 2022-10-26 20:10:39] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:05:54 [I 2022-10-26 20:10:39] ...trap/engines/engine.py.229: process: 0.54687 | load: 0.00244 [I 2022-10-26 20:10:39] ...trap/engines/engine.py.230: loss: 0.19618 [I 2022-10-26 20:10:39] ...optimizers/trijoint.py.151: total_norm: 14.845589 [I 2022-10-26 20:10:45] ...trap/engines/engine.py.224: train: epoch 27 | batch 1670/2304 [I 2022-10-26 20:10:45] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:05:52 [I 2022-10-26 20:10:45] ...trap/engines/engine.py.229: process: 0.56685 | load: 0.00246 [I 2022-10-26 20:10:45] ...trap/engines/engine.py.230: loss: 0.13763 [I 2022-10-26 20:10:45] ...optimizers/trijoint.py.151: total_norm: 15.549742 [I 2022-10-26 20:10:50] ...trap/engines/engine.py.224: train: epoch 27 | batch 1680/2304 [I 2022-10-26 20:10:50] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:05:47 [I 2022-10-26 20:10:50] ...trap/engines/engine.py.229: process: 0.55764 | load: 0.00243 [I 2022-10-26 20:10:50] ...trap/engines/engine.py.230: loss: 0.15820 [I 2022-10-26 20:10:50] ...optimizers/trijoint.py.151: total_norm: 9.251349 [I 2022-10-26 20:10:56] ...trap/engines/engine.py.224: train: epoch 27 | batch 1690/2304 [I 2022-10-26 20:10:56] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:05:38 [I 2022-10-26 20:10:56] ...trap/engines/engine.py.229: process: 0.55277 | load: 0.00241 [I 2022-10-26 20:10:56] ...trap/engines/engine.py.230: loss: 0.18034 [I 2022-10-26 20:10:56] ...optimizers/trijoint.py.151: total_norm: 21.912668 [I 2022-10-26 20:11:02] ...trap/engines/engine.py.224: train: epoch 27 | batch 1700/2304 [I 2022-10-26 20:11:02] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:36 [I 2022-10-26 20:11:02] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00248 [I 2022-10-26 20:11:02] ...trap/engines/engine.py.230: loss: 0.25583 [I 2022-10-26 20:11:02] ...optimizers/trijoint.py.151: total_norm: 26.698172 [I 2022-10-26 20:11:07] ...trap/engines/engine.py.224: train: epoch 27 | batch 1710/2304 [I 2022-10-26 20:11:07] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:31 [I 2022-10-26 20:11:07] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00254 [I 2022-10-26 20:11:07] ...trap/engines/engine.py.230: loss: 0.30366 [I 2022-10-26 20:11:07] ...optimizers/trijoint.py.151: total_norm: 24.841324 [I 2022-10-26 20:11:13] ...trap/engines/engine.py.224: train: epoch 27 | batch 1720/2304 [I 2022-10-26 20:11:13] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:24 [I 2022-10-26 20:11:13] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00259 [I 2022-10-26 20:11:13] ...trap/engines/engine.py.230: loss: 0.20208 [I 2022-10-26 20:11:13] ...optimizers/trijoint.py.151: total_norm: 15.717000 [I 2022-10-26 20:11:18] ...trap/engines/engine.py.224: train: epoch 27 | batch 1730/2304 [I 2022-10-26 20:11:18] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:17 [I 2022-10-26 20:11:18] ...trap/engines/engine.py.229: process: 0.54563 | load: 0.00252 [I 2022-10-26 20:11:18] ...trap/engines/engine.py.230: loss: 0.11602 [I 2022-10-26 20:11:18] ...optimizers/trijoint.py.151: total_norm: 20.057423 [I 2022-10-26 20:11:24] ...trap/engines/engine.py.224: train: epoch 27 | batch 1740/2304 [I 2022-10-26 20:11:24] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:13 [I 2022-10-26 20:11:24] ...trap/engines/engine.py.229: process: 0.54918 | load: 0.00255 [I 2022-10-26 20:11:24] ...trap/engines/engine.py.230: loss: 0.17221 [I 2022-10-26 20:11:24] ...optimizers/trijoint.py.151: total_norm: 14.006556 [I 2022-10-26 20:11:29] ...trap/engines/engine.py.224: train: epoch 27 | batch 1750/2304 [I 2022-10-26 20:11:29] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:06 [I 2022-10-26 20:11:29] ...trap/engines/engine.py.229: process: 0.54430 | load: 0.00241 [I 2022-10-26 20:11:29] ...trap/engines/engine.py.230: loss: 0.20767 [I 2022-10-26 20:11:29] ...optimizers/trijoint.py.151: total_norm: 16.105040 [I 2022-10-26 20:11:35] ...trap/engines/engine.py.224: train: epoch 27 | batch 1760/2304 [I 2022-10-26 20:11:35] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:01 [I 2022-10-26 20:11:35] ...trap/engines/engine.py.229: process: 0.54651 | load: 0.00251 [I 2022-10-26 20:11:35] ...trap/engines/engine.py.230: loss: 0.20185 [I 2022-10-26 20:11:35] ...optimizers/trijoint.py.151: total_norm: 31.129244 [I 2022-10-26 20:11:40] ...trap/engines/engine.py.224: train: epoch 27 | batch 1770/2304 [I 2022-10-26 20:11:40] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:04:57 [I 2022-10-26 20:11:40] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00520 [I 2022-10-26 20:11:40] ...trap/engines/engine.py.230: loss: 0.21710 [I 2022-10-26 20:11:40] ...optimizers/trijoint.py.151: total_norm: 20.074375 [I 2022-10-26 20:11:46] ...trap/engines/engine.py.224: train: epoch 27 | batch 1780/2304 [I 2022-10-26 20:11:46] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:04:51 [I 2022-10-26 20:11:46] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00251 [I 2022-10-26 20:11:46] ...trap/engines/engine.py.230: loss: 0.19367 [I 2022-10-26 20:11:46] ...optimizers/trijoint.py.151: total_norm: 24.905584 [I 2022-10-26 20:11:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 1790/2304 [I 2022-10-26 20:11:52] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:04:47 [I 2022-10-26 20:11:52] ...trap/engines/engine.py.229: process: 0.56380 | load: 0.00248 [I 2022-10-26 20:11:52] ...trap/engines/engine.py.230: loss: 0.21325 [I 2022-10-26 20:11:52] ...optimizers/trijoint.py.151: total_norm: 12.778533 [I 2022-10-26 20:11:57] ...trap/engines/engine.py.224: train: epoch 27 | batch 1800/2304 [I 2022-10-26 20:11:57] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:40 [I 2022-10-26 20:11:57] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00243 [I 2022-10-26 20:11:57] ...trap/engines/engine.py.230: loss: 0.16038 [I 2022-10-26 20:11:57] ...optimizers/trijoint.py.151: total_norm: 19.844572 [I 2022-10-26 20:12:03] ...trap/engines/engine.py.224: train: epoch 27 | batch 1810/2304 [I 2022-10-26 20:12:03] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:32 [I 2022-10-26 20:12:03] ...trap/engines/engine.py.229: process: 0.54891 | load: 0.00246 [I 2022-10-26 20:12:03] ...trap/engines/engine.py.230: loss: 0.15559 [I 2022-10-26 20:12:03] ...optimizers/trijoint.py.151: total_norm: 22.821108 [I 2022-10-26 20:12:08] ...trap/engines/engine.py.224: train: epoch 27 | batch 1820/2304 [I 2022-10-26 20:12:08] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:29 [I 2022-10-26 20:12:08] ...trap/engines/engine.py.229: process: 0.56332 | load: 0.00243 [I 2022-10-26 20:12:08] ...trap/engines/engine.py.230: loss: 0.20357 [I 2022-10-26 20:12:08] ...optimizers/trijoint.py.151: total_norm: 21.555992 [I 2022-10-26 20:12:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 1830/2304 [I 2022-10-26 20:12:14] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:22 [I 2022-10-26 20:12:14] ...trap/engines/engine.py.229: process: 0.55936 | load: 0.00243 [I 2022-10-26 20:12:14] ...trap/engines/engine.py.230: loss: 0.20175 [I 2022-10-26 20:12:14] ...optimizers/trijoint.py.151: total_norm: 35.486210 [I 2022-10-26 20:12:19] ...trap/engines/engine.py.224: train: epoch 27 | batch 1840/2304 [I 2022-10-26 20:12:19] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:15 [I 2022-10-26 20:12:19] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00246 [I 2022-10-26 20:12:19] ...trap/engines/engine.py.230: loss: 0.14324 [I 2022-10-26 20:12:19] ...optimizers/trijoint.py.151: total_norm: 17.159000 [I 2022-10-26 20:12:25] ...trap/engines/engine.py.224: train: epoch 27 | batch 1850/2304 [I 2022-10-26 20:12:25] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:13 [I 2022-10-26 20:12:25] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00247 [I 2022-10-26 20:12:25] ...trap/engines/engine.py.230: loss: 0.14141 [I 2022-10-26 20:12:25] ...optimizers/trijoint.py.151: total_norm: 15.525118 [I 2022-10-26 20:12:31] ...trap/engines/engine.py.224: train: epoch 27 | batch 1860/2304 [I 2022-10-26 20:12:31] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:06 [I 2022-10-26 20:12:31] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00245 [I 2022-10-26 20:12:31] ...trap/engines/engine.py.230: loss: 0.19688 [I 2022-10-26 20:12:31] ...optimizers/trijoint.py.151: total_norm: 31.279114 [I 2022-10-26 20:12:36] ...trap/engines/engine.py.224: train: epoch 27 | batch 1870/2304 [I 2022-10-26 20:12:36] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:02 [I 2022-10-26 20:12:36] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00246 [I 2022-10-26 20:12:36] ...trap/engines/engine.py.230: loss: 0.14904 [I 2022-10-26 20:12:36] ...optimizers/trijoint.py.151: total_norm: 20.010689 [I 2022-10-26 20:12:42] ...trap/engines/engine.py.224: train: epoch 27 | batch 1880/2304 [I 2022-10-26 20:12:42] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:03:55 [I 2022-10-26 20:12:42] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00244 [I 2022-10-26 20:12:42] ...trap/engines/engine.py.230: loss: 0.17596 [I 2022-10-26 20:12:42] ...optimizers/trijoint.py.151: total_norm: 27.591852 [I 2022-10-26 20:12:47] ...trap/engines/engine.py.224: train: epoch 27 | batch 1890/2304 [I 2022-10-26 20:12:47] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:03:49 [I 2022-10-26 20:12:47] ...trap/engines/engine.py.229: process: 0.54386 | load: 0.00248 [I 2022-10-26 20:12:47] ...trap/engines/engine.py.230: loss: 0.15529 [I 2022-10-26 20:12:47] ...optimizers/trijoint.py.151: total_norm: 24.581385 [I 2022-10-26 20:12:53] ...trap/engines/engine.py.224: train: epoch 27 | batch 1900/2304 [I 2022-10-26 20:12:53] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:03:43 [I 2022-10-26 20:12:53] ...trap/engines/engine.py.229: process: 0.55185 | load: 0.00242 [I 2022-10-26 20:12:53] ...trap/engines/engine.py.230: loss: 0.20086 [I 2022-10-26 20:12:53] ...optimizers/trijoint.py.151: total_norm: 27.279896 [I 2022-10-26 20:12:58] ...trap/engines/engine.py.224: train: epoch 27 | batch 1910/2304 [I 2022-10-26 20:12:58] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:03:40 [I 2022-10-26 20:12:58] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00240 [I 2022-10-26 20:12:58] ...trap/engines/engine.py.230: loss: 0.11626 [I 2022-10-26 20:12:58] ...optimizers/trijoint.py.151: total_norm: 15.930270 [I 2022-10-26 20:13:04] ...trap/engines/engine.py.224: train: epoch 27 | batch 1920/2304 [I 2022-10-26 20:13:04] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:34 [I 2022-10-26 20:13:04] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00241 [I 2022-10-26 20:13:04] ...trap/engines/engine.py.230: loss: 0.16135 [I 2022-10-26 20:13:04] ...optimizers/trijoint.py.151: total_norm: 16.539547 [I 2022-10-26 20:13:10] ...trap/engines/engine.py.224: train: epoch 27 | batch 1930/2304 [I 2022-10-26 20:13:10] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:29 [I 2022-10-26 20:13:10] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00240 [I 2022-10-26 20:13:10] ...trap/engines/engine.py.230: loss: 0.23410 [I 2022-10-26 20:13:10] ...optimizers/trijoint.py.151: total_norm: 25.464203 [I 2022-10-26 20:13:15] ...trap/engines/engine.py.224: train: epoch 27 | batch 1940/2304 [I 2022-10-26 20:13:15] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:22 [I 2022-10-26 20:13:15] ...trap/engines/engine.py.229: process: 0.56186 | load: 0.00246 [I 2022-10-26 20:13:15] ...trap/engines/engine.py.230: loss: 0.12382 [I 2022-10-26 20:13:15] ...optimizers/trijoint.py.151: total_norm: 24.082943 [I 2022-10-26 20:13:21] ...trap/engines/engine.py.224: train: epoch 27 | batch 1950/2304 [I 2022-10-26 20:13:21] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:17 [I 2022-10-26 20:13:21] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00252 [I 2022-10-26 20:13:21] ...trap/engines/engine.py.230: loss: 0.34800 [I 2022-10-26 20:13:21] ...optimizers/trijoint.py.151: total_norm: 25.208736 [I 2022-10-26 20:13:26] ...trap/engines/engine.py.224: train: epoch 27 | batch 1960/2304 [I 2022-10-26 20:13:26] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:11 [I 2022-10-26 20:13:26] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00251 [I 2022-10-26 20:13:26] ...trap/engines/engine.py.230: loss: 0.24524 [I 2022-10-26 20:13:26] ...optimizers/trijoint.py.151: total_norm: 19.686764 [I 2022-10-26 20:13:32] ...trap/engines/engine.py.224: train: epoch 27 | batch 1970/2304 [I 2022-10-26 20:13:32] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:05 [I 2022-10-26 20:13:32] ...trap/engines/engine.py.229: process: 0.55614 | load: 0.00263 [I 2022-10-26 20:13:32] ...trap/engines/engine.py.230: loss: 0.15001 [I 2022-10-26 20:13:32] ...optimizers/trijoint.py.151: total_norm: 18.951157 [I 2022-10-26 20:13:37] ...trap/engines/engine.py.224: train: epoch 27 | batch 1980/2304 [I 2022-10-26 20:13:37] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:00 [I 2022-10-26 20:13:37] ...trap/engines/engine.py.229: process: 0.55284 | load: 0.00248 [I 2022-10-26 20:13:37] ...trap/engines/engine.py.230: loss: 0.16431 [I 2022-10-26 20:13:37] ...optimizers/trijoint.py.151: total_norm: 12.202101 [I 2022-10-26 20:13:43] ...trap/engines/engine.py.224: train: epoch 27 | batch 1990/2304 [I 2022-10-26 20:13:43] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:02:54 [I 2022-10-26 20:13:43] ...trap/engines/engine.py.229: process: 0.56214 | load: 0.00246 [I 2022-10-26 20:13:43] ...trap/engines/engine.py.230: loss: 0.13111 [I 2022-10-26 20:13:43] ...optimizers/trijoint.py.151: total_norm: 17.307547 [I 2022-10-26 20:13:49] ...trap/engines/engine.py.224: train: epoch 27 | batch 2000/2304 [I 2022-10-26 20:13:49] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:02:49 [I 2022-10-26 20:13:49] ...trap/engines/engine.py.229: process: 0.56274 | load: 0.00248 [I 2022-10-26 20:13:49] ...trap/engines/engine.py.230: loss: 0.16415 [I 2022-10-26 20:13:49] ...optimizers/trijoint.py.151: total_norm: 19.845407 [I 2022-10-26 20:13:54] ...trap/engines/engine.py.224: train: epoch 27 | batch 2010/2304 [I 2022-10-26 20:13:54] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:02:42 [I 2022-10-26 20:13:54] ...trap/engines/engine.py.229: process: 0.56445 | load: 0.00246 [I 2022-10-26 20:13:54] ...trap/engines/engine.py.230: loss: 0.21706 [I 2022-10-26 20:13:54] ...optimizers/trijoint.py.151: total_norm: 28.857018 [I 2022-10-26 20:14:00] ...trap/engines/engine.py.224: train: epoch 27 | batch 2020/2304 [I 2022-10-26 20:14:00] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:38 [I 2022-10-26 20:14:00] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00247 [I 2022-10-26 20:14:00] ...trap/engines/engine.py.230: loss: 0.15770 [I 2022-10-26 20:14:00] ...optimizers/trijoint.py.151: total_norm: 14.976319 [I 2022-10-26 20:14:05] ...trap/engines/engine.py.224: train: epoch 27 | batch 2030/2304 [I 2022-10-26 20:14:05] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:33 [I 2022-10-26 20:14:05] ...trap/engines/engine.py.229: process: 0.57434 | load: 0.00243 [I 2022-10-26 20:14:05] ...trap/engines/engine.py.230: loss: 0.12342 [I 2022-10-26 20:14:05] ...optimizers/trijoint.py.151: total_norm: 23.657452 [I 2022-10-26 20:14:11] ...trap/engines/engine.py.224: train: epoch 27 | batch 2040/2304 [I 2022-10-26 20:14:11] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:27 [I 2022-10-26 20:14:11] ...trap/engines/engine.py.229: process: 0.56210 | load: 0.00243 [I 2022-10-26 20:14:11] ...trap/engines/engine.py.230: loss: 0.19309 [I 2022-10-26 20:14:11] ...optimizers/trijoint.py.151: total_norm: 17.029913 [I 2022-10-26 20:14:17] ...trap/engines/engine.py.224: train: epoch 27 | batch 2050/2304 [I 2022-10-26 20:14:17] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:02:20 [I 2022-10-26 20:14:17] ...trap/engines/engine.py.229: process: 0.54215 | load: 0.00251 [I 2022-10-26 20:14:17] ...trap/engines/engine.py.230: loss: 0.20173 [I 2022-10-26 20:14:17] ...optimizers/trijoint.py.151: total_norm: 23.131626 [I 2022-10-26 20:14:22] ...trap/engines/engine.py.224: train: epoch 27 | batch 2060/2304 [I 2022-10-26 20:14:22] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:15 [I 2022-10-26 20:14:22] ...trap/engines/engine.py.229: process: 0.54420 | load: 0.00246 [I 2022-10-26 20:14:22] ...trap/engines/engine.py.230: loss: 0.26120 [I 2022-10-26 20:14:22] ...optimizers/trijoint.py.151: total_norm: 12.733523 [I 2022-10-26 20:14:28] ...trap/engines/engine.py.224: train: epoch 27 | batch 2070/2304 [I 2022-10-26 20:14:28] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:10 [I 2022-10-26 20:14:28] ...trap/engines/engine.py.229: process: 0.55444 | load: 0.00246 [I 2022-10-26 20:14:28] ...trap/engines/engine.py.230: loss: 0.25711 [I 2022-10-26 20:14:28] ...optimizers/trijoint.py.151: total_norm: 14.917065 [I 2022-10-26 20:14:33] ...trap/engines/engine.py.224: train: epoch 27 | batch 2080/2304 [I 2022-10-26 20:14:33] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:04 [I 2022-10-26 20:14:33] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00245 [I 2022-10-26 20:14:33] ...trap/engines/engine.py.230: loss: 0.17178 [I 2022-10-26 20:14:33] ...optimizers/trijoint.py.151: total_norm: 13.237990 [I 2022-10-26 20:14:39] ...trap/engines/engine.py.224: train: epoch 27 | batch 2090/2304 [I 2022-10-26 20:14:39] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:01:58 [I 2022-10-26 20:14:39] ...trap/engines/engine.py.229: process: 0.55597 | load: 0.00243 [I 2022-10-26 20:14:39] ...trap/engines/engine.py.230: loss: 0.15309 [I 2022-10-26 20:14:39] ...optimizers/trijoint.py.151: total_norm: 15.208286 [I 2022-10-26 20:14:44] ...trap/engines/engine.py.224: train: epoch 27 | batch 2100/2304 [I 2022-10-26 20:14:44] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:01:53 [I 2022-10-26 20:14:44] ...trap/engines/engine.py.229: process: 0.55257 | load: 0.00243 [I 2022-10-26 20:14:44] ...trap/engines/engine.py.230: loss: 0.17742 [I 2022-10-26 20:14:44] ...optimizers/trijoint.py.151: total_norm: 25.974062 [I 2022-10-26 20:14:50] ...trap/engines/engine.py.224: train: epoch 27 | batch 2110/2304 [I 2022-10-26 20:14:50] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:01:48 [I 2022-10-26 20:14:50] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00244 [I 2022-10-26 20:14:50] ...trap/engines/engine.py.230: loss: 0.19485 [I 2022-10-26 20:14:50] ...optimizers/trijoint.py.151: total_norm: 23.740595 [I 2022-10-26 20:14:56] ...trap/engines/engine.py.224: train: epoch 27 | batch 2120/2304 [I 2022-10-26 20:14:56] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:01:42 [I 2022-10-26 20:14:56] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00260 [I 2022-10-26 20:14:56] ...trap/engines/engine.py.230: loss: 0.15101 [I 2022-10-26 20:14:56] ...optimizers/trijoint.py.151: total_norm: 30.955179 [I 2022-10-26 20:15:01] ...trap/engines/engine.py.224: train: epoch 27 | batch 2130/2304 [I 2022-10-26 20:15:01] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:36 [I 2022-10-26 20:15:01] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00256 [I 2022-10-26 20:15:01] ...trap/engines/engine.py.230: loss: 0.22632 [I 2022-10-26 20:15:01] ...optimizers/trijoint.py.151: total_norm: 27.394424 [I 2022-10-26 20:15:07] ...trap/engines/engine.py.224: train: epoch 27 | batch 2140/2304 [I 2022-10-26 20:15:07] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:30 [I 2022-10-26 20:15:07] ...trap/engines/engine.py.229: process: 0.54985 | load: 0.00248 [I 2022-10-26 20:15:07] ...trap/engines/engine.py.230: loss: 0.15299 [I 2022-10-26 20:15:07] ...optimizers/trijoint.py.151: total_norm: 14.601599 [I 2022-10-26 20:15:12] ...trap/engines/engine.py.224: train: epoch 27 | batch 2150/2304 [I 2022-10-26 20:15:12] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:25 [I 2022-10-26 20:15:12] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00248 [I 2022-10-26 20:15:12] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-26 20:15:12] ...optimizers/trijoint.py.151: total_norm: 15.475871 [I 2022-10-26 20:15:18] ...trap/engines/engine.py.224: train: epoch 27 | batch 2160/2304 [I 2022-10-26 20:15:18] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:20 [I 2022-10-26 20:15:18] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00244 [I 2022-10-26 20:15:18] ...trap/engines/engine.py.230: loss: 0.10759 [I 2022-10-26 20:15:18] ...optimizers/trijoint.py.151: total_norm: 19.664911 [I 2022-10-26 20:15:24] ...trap/engines/engine.py.224: train: epoch 27 | batch 2170/2304 [I 2022-10-26 20:15:24] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:16 [I 2022-10-26 20:15:24] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00245 [I 2022-10-26 20:15:24] ...trap/engines/engine.py.230: loss: 0.11831 [I 2022-10-26 20:15:24] ...optimizers/trijoint.py.151: total_norm: 10.737086 [I 2022-10-26 20:15:29] ...trap/engines/engine.py.224: train: epoch 27 | batch 2180/2304 [I 2022-10-26 20:15:29] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:08 [I 2022-10-26 20:15:29] ...trap/engines/engine.py.229: process: 0.55191 | load: 0.00244 [I 2022-10-26 20:15:29] ...trap/engines/engine.py.230: loss: 0.17252 [I 2022-10-26 20:15:29] ...optimizers/trijoint.py.151: total_norm: 29.391661 [I 2022-10-26 20:15:35] ...trap/engines/engine.py.224: train: epoch 27 | batch 2190/2304 [I 2022-10-26 20:15:35] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:03 [I 2022-10-26 20:15:35] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00243 [I 2022-10-26 20:15:35] ...trap/engines/engine.py.230: loss: 0.14348 [I 2022-10-26 20:15:35] ...optimizers/trijoint.py.151: total_norm: 12.635819 [I 2022-10-26 20:15:40] ...trap/engines/engine.py.224: train: epoch 27 | batch 2200/2304 [I 2022-10-26 20:15:40] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:00:57 [I 2022-10-26 20:15:40] ...trap/engines/engine.py.229: process: 0.54585 | load: 0.00250 [I 2022-10-26 20:15:40] ...trap/engines/engine.py.230: loss: 0.12864 [I 2022-10-26 20:15:40] ...optimizers/trijoint.py.151: total_norm: 10.476405 [I 2022-10-26 20:15:46] ...trap/engines/engine.py.224: train: epoch 27 | batch 2210/2304 [I 2022-10-26 20:15:46] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:00:52 [I 2022-10-26 20:15:46] ...trap/engines/engine.py.229: process: 0.54813 | load: 0.00246 [I 2022-10-26 20:15:46] ...trap/engines/engine.py.230: loss: 0.18892 [I 2022-10-26 20:15:46] ...optimizers/trijoint.py.151: total_norm: 13.048473 [I 2022-10-26 20:15:52] ...trap/engines/engine.py.224: train: epoch 27 | batch 2220/2304 [I 2022-10-26 20:15:52] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:00:46 [I 2022-10-26 20:15:52] ...trap/engines/engine.py.229: process: 0.53131 | load: 0.00250 [I 2022-10-26 20:15:52] ...trap/engines/engine.py.230: loss: 0.19266 [I 2022-10-26 20:15:52] ...optimizers/trijoint.py.151: total_norm: 31.809694 [I 2022-10-26 20:15:57] ...trap/engines/engine.py.224: train: epoch 27 | batch 2230/2304 [I 2022-10-26 20:15:57] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:00:41 [I 2022-10-26 20:15:57] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00246 [I 2022-10-26 20:15:57] ...trap/engines/engine.py.230: loss: 0.18176 [I 2022-10-26 20:15:57] ...optimizers/trijoint.py.151: total_norm: 14.764893 [I 2022-10-26 20:16:03] ...trap/engines/engine.py.224: train: epoch 27 | batch 2240/2304 [I 2022-10-26 20:16:03] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:35 [I 2022-10-26 20:16:03] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00244 [I 2022-10-26 20:16:03] ...trap/engines/engine.py.230: loss: 0.20331 [I 2022-10-26 20:16:03] ...optimizers/trijoint.py.151: total_norm: 12.805905 [I 2022-10-26 20:16:08] ...trap/engines/engine.py.224: train: epoch 27 | batch 2250/2304 [I 2022-10-26 20:16:08] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:29 [I 2022-10-26 20:16:08] ...trap/engines/engine.py.229: process: 0.54827 | load: 0.00240 [I 2022-10-26 20:16:08] ...trap/engines/engine.py.230: loss: 0.19863 [I 2022-10-26 20:16:08] ...optimizers/trijoint.py.151: total_norm: 16.149860 [I 2022-10-26 20:16:14] ...trap/engines/engine.py.224: train: epoch 27 | batch 2260/2304 [I 2022-10-26 20:16:14] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:25 [I 2022-10-26 20:16:14] ...trap/engines/engine.py.229: process: 0.54818 | load: 0.00264 [I 2022-10-26 20:16:14] ...trap/engines/engine.py.230: loss: 0.18694 [I 2022-10-26 20:16:14] ...optimizers/trijoint.py.151: total_norm: 23.017620 [I 2022-10-26 20:16:19] ...trap/engines/engine.py.224: train: epoch 27 | batch 2270/2304 [I 2022-10-26 20:16:19] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:18 [I 2022-10-26 20:16:19] ...trap/engines/engine.py.229: process: 0.54405 | load: 0.00240 [I 2022-10-26 20:16:19] ...trap/engines/engine.py.230: loss: 0.17182 [I 2022-10-26 20:16:19] ...optimizers/trijoint.py.151: total_norm: 22.031971 [I 2022-10-26 20:16:25] ...trap/engines/engine.py.224: train: epoch 27 | batch 2280/2304 [I 2022-10-26 20:16:25] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:13 [I 2022-10-26 20:16:25] ...trap/engines/engine.py.229: process: 0.54680 | load: 0.00244 [I 2022-10-26 20:16:25] ...trap/engines/engine.py.230: loss: 0.18678 [I 2022-10-26 20:16:25] ...optimizers/trijoint.py.151: total_norm: 20.961470 [I 2022-10-26 20:16:31] ...trap/engines/engine.py.224: train: epoch 27 | batch 2290/2304 [I 2022-10-26 20:16:31] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:07 [I 2022-10-26 20:16:31] ...trap/engines/engine.py.229: process: 0.54716 | load: 0.00240 [I 2022-10-26 20:16:31] ...trap/engines/engine.py.230: loss: 0.19251 [I 2022-10-26 20:16:31] ...optimizers/trijoint.py.151: total_norm: 13.928943 [I 2022-10-26 20:16:36] ...trap/engines/engine.py.224: train: epoch 27 | batch 2300/2304 [I 2022-10-26 20:16:36] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:02 [I 2022-10-26 20:16:36] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00014 [I 2022-10-26 20:16:36] ...trap/engines/engine.py.230: loss: 0.14674 [I 2022-10-26 20:16:36] ...optimizers/trijoint.py.151: total_norm: 17.804630 [I 2022-10-26 20:16:38] ...trap/engines/engine.py.224: train: epoch 27 | batch 2304/2304 [I 2022-10-26 20:16:38] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:00 [I 2022-10-26 20:16:38] ...trap/engines/engine.py.229: process: 0.53635 | load: 0.00015 [I 2022-10-26 20:16:38] ...trap/engines/engine.py.230: loss: 0.22976 [I 2022-10-26 20:16:38] ...optimizers/trijoint.py.151: total_norm: 26.775209 [S 2022-10-26 20:16:39] ...trap/engines/engine.py.129: train_epoch.epoch: 27 [S 2022-10-26 20:16:39] ...trap/engines/engine.py.129: train_epoch.loss: 0.1864664892086238 [I 2022-10-26 20:16:39] ...s/criterions/triplet.py.84: margin = 0.1900000000000001 [I 2022-10-26 20:16:40] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 27 [I 2022-10-26 20:16:40] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 20:16:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 20:16:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 20:16:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 20:16:42] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 20:16:43] ...trap/engines/engine.py.321: eval: epoch 27 | batch 0/510 [I 2022-10-26 20:16:43] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:25:39 [I 2022-10-26 20:16:43] ...trap/engines/engine.py.326: process: 3.01934 | load: 2.62852 [I 2022-10-26 20:16:46] ...trap/engines/engine.py.321: eval: epoch 27 | batch 10/510 [I 2022-10-26 20:16:46] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:10 [I 2022-10-26 20:16:46] ...trap/engines/engine.py.326: process: 0.34212 | load: 0.00017 [I 2022-10-26 20:16:50] ...trap/engines/engine.py.321: eval: epoch 27 | batch 20/510 [I 2022-10-26 20:16:50] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:58 [I 2022-10-26 20:16:50] ...trap/engines/engine.py.326: process: 0.33203 | load: 0.00016 [I 2022-10-26 20:16:53] ...trap/engines/engine.py.321: eval: epoch 27 | batch 30/510 [I 2022-10-26 20:16:53] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:41 [I 2022-10-26 20:16:53] ...trap/engines/engine.py.326: process: 0.32832 | load: 0.00016 [I 2022-10-26 20:16:56] ...trap/engines/engine.py.321: eval: epoch 27 | batch 40/510 [I 2022-10-26 20:16:56] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:35 [I 2022-10-26 20:16:56] ...trap/engines/engine.py.326: process: 0.32025 | load: 0.00019 [I 2022-10-26 20:17:00] ...trap/engines/engine.py.321: eval: epoch 27 | batch 50/510 [I 2022-10-26 20:17:00] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:32 [I 2022-10-26 20:17:00] ...trap/engines/engine.py.326: process: 0.34154 | load: 0.00021 [I 2022-10-26 20:17:03] ...trap/engines/engine.py.321: eval: epoch 27 | batch 60/510 [I 2022-10-26 20:17:03] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:28 [I 2022-10-26 20:17:03] ...trap/engines/engine.py.326: process: 0.32282 | load: 0.00016 [I 2022-10-26 20:17:06] ...trap/engines/engine.py.321: eval: epoch 27 | batch 70/510 [I 2022-10-26 20:17:06] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:26 [I 2022-10-26 20:17:06] ...trap/engines/engine.py.326: process: 0.31867 | load: 0.00020 [I 2022-10-26 20:17:10] ...trap/engines/engine.py.321: eval: epoch 27 | batch 80/510 [I 2022-10-26 20:17:10] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:22 [I 2022-10-26 20:17:10] ...trap/engines/engine.py.326: process: 0.33076 | load: 0.00017 [I 2022-10-26 20:17:13] ...trap/engines/engine.py.321: eval: epoch 27 | batch 90/510 [I 2022-10-26 20:17:13] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:19 [I 2022-10-26 20:17:13] ...trap/engines/engine.py.326: process: 0.33306 | load: 0.00021 [I 2022-10-26 20:17:16] ...trap/engines/engine.py.321: eval: epoch 27 | batch 100/510 [I 2022-10-26 20:17:16] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 20:17:16] ...trap/engines/engine.py.326: process: 0.28577 | load: 0.00016 [I 2022-10-26 20:17:19] ...trap/engines/engine.py.321: eval: epoch 27 | batch 110/510 [I 2022-10-26 20:17:19] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-26 20:17:19] ...trap/engines/engine.py.326: process: 0.29371 | load: 0.00019 [I 2022-10-26 20:17:22] ...trap/engines/engine.py.321: eval: epoch 27 | batch 120/510 [I 2022-10-26 20:17:22] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-26 20:17:22] ...trap/engines/engine.py.326: process: 0.28721 | load: 0.00023 [I 2022-10-26 20:17:25] ...trap/engines/engine.py.321: eval: epoch 27 | batch 130/510 [I 2022-10-26 20:17:25] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-26 20:17:25] ...trap/engines/engine.py.326: process: 0.29135 | load: 0.00031 [I 2022-10-26 20:17:28] ...trap/engines/engine.py.321: eval: epoch 27 | batch 140/510 [I 2022-10-26 20:17:28] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:47 [I 2022-10-26 20:17:28] ...trap/engines/engine.py.326: process: 0.28750 | load: 0.00019 [I 2022-10-26 20:17:31] ...trap/engines/engine.py.321: eval: epoch 27 | batch 150/510 [I 2022-10-26 20:17:31] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:43 [I 2022-10-26 20:17:31] ...trap/engines/engine.py.326: process: 0.28637 | load: 0.00014 [I 2022-10-26 20:17:34] ...trap/engines/engine.py.321: eval: epoch 27 | batch 160/510 [I 2022-10-26 20:17:34] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 20:17:34] ...trap/engines/engine.py.326: process: 0.28655 | load: 0.00017 [I 2022-10-26 20:17:36] ...trap/engines/engine.py.321: eval: epoch 27 | batch 170/510 [I 2022-10-26 20:17:36] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 20:17:36] ...trap/engines/engine.py.326: process: 0.29168 | load: 0.00018 [I 2022-10-26 20:17:39] ...trap/engines/engine.py.321: eval: epoch 27 | batch 180/510 [I 2022-10-26 20:17:39] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 20:17:39] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00015 [I 2022-10-26 20:17:42] ...trap/engines/engine.py.321: eval: epoch 27 | batch 190/510 [I 2022-10-26 20:17:42] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 20:17:42] ...trap/engines/engine.py.326: process: 0.28878 | load: 0.00018 [I 2022-10-26 20:17:45] ...trap/engines/engine.py.321: eval: epoch 27 | batch 200/510 [I 2022-10-26 20:17:45] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 20:17:45] ...trap/engines/engine.py.326: process: 0.28864 | load: 0.00019 [I 2022-10-26 20:17:48] ...trap/engines/engine.py.321: eval: epoch 27 | batch 210/510 [I 2022-10-26 20:17:48] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-26 20:17:48] ...trap/engines/engine.py.326: process: 0.29009 | load: 0.00022 [I 2022-10-26 20:17:51] ...trap/engines/engine.py.321: eval: epoch 27 | batch 220/510 [I 2022-10-26 20:17:51] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-26 20:17:51] ...trap/engines/engine.py.326: process: 0.28486 | load: 0.00024 [I 2022-10-26 20:17:54] ...trap/engines/engine.py.321: eval: epoch 27 | batch 230/510 [I 2022-10-26 20:17:54] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-26 20:17:54] ...trap/engines/engine.py.326: process: 0.28878 | load: 0.00015 [I 2022-10-26 20:17:57] ...trap/engines/engine.py.321: eval: epoch 27 | batch 240/510 [I 2022-10-26 20:17:57] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:18 [I 2022-10-26 20:17:57] ...trap/engines/engine.py.326: process: 0.30084 | load: 0.00017 [I 2022-10-26 20:18:00] ...trap/engines/engine.py.321: eval: epoch 27 | batch 250/510 [I 2022-10-26 20:18:00] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:16 [I 2022-10-26 20:18:00] ...trap/engines/engine.py.326: process: 0.28749 | load: 0.00024 [I 2022-10-26 20:18:03] ...trap/engines/engine.py.321: eval: epoch 27 | batch 260/510 [I 2022-10-26 20:18:03] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:12 [I 2022-10-26 20:18:03] ...trap/engines/engine.py.326: process: 0.29038 | load: 0.00016 [I 2022-10-26 20:18:06] ...trap/engines/engine.py.321: eval: epoch 27 | batch 270/510 [I 2022-10-26 20:18:06] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-26 20:18:06] ...trap/engines/engine.py.326: process: 0.28251 | load: 0.00023 [I 2022-10-26 20:18:08] ...trap/engines/engine.py.321: eval: epoch 27 | batch 280/510 [I 2022-10-26 20:18:08] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 20:18:08] ...trap/engines/engine.py.326: process: 0.28245 | load: 0.00021 [I 2022-10-26 20:18:11] ...trap/engines/engine.py.321: eval: epoch 27 | batch 290/510 [I 2022-10-26 20:18:11] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 20:18:11] ...trap/engines/engine.py.326: process: 0.30070 | load: 0.00016 [I 2022-10-26 20:18:14] ...trap/engines/engine.py.321: eval: epoch 27 | batch 300/510 [I 2022-10-26 20:18:14] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 20:18:14] ...trap/engines/engine.py.326: process: 0.29645 | load: 0.00022 [I 2022-10-26 20:18:17] ...trap/engines/engine.py.321: eval: epoch 27 | batch 310/510 [I 2022-10-26 20:18:17] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 20:18:17] ...trap/engines/engine.py.326: process: 0.28792 | load: 0.00015 [I 2022-10-26 20:18:20] ...trap/engines/engine.py.321: eval: epoch 27 | batch 320/510 [I 2022-10-26 20:18:20] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 20:18:20] ...trap/engines/engine.py.326: process: 0.28509 | load: 0.00021 [I 2022-10-26 20:18:23] ...trap/engines/engine.py.321: eval: epoch 27 | batch 330/510 [I 2022-10-26 20:18:23] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 20:18:23] ...trap/engines/engine.py.326: process: 0.29420 | load: 0.00016 [I 2022-10-26 20:18:26] ...trap/engines/engine.py.321: eval: epoch 27 | batch 340/510 [I 2022-10-26 20:18:26] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:49 [I 2022-10-26 20:18:26] ...trap/engines/engine.py.326: process: 0.28832 | load: 0.00022 [I 2022-10-26 20:18:29] ...trap/engines/engine.py.321: eval: epoch 27 | batch 350/510 [I 2022-10-26 20:18:29] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 20:18:29] ...trap/engines/engine.py.326: process: 0.28625 | load: 0.00020 [I 2022-10-26 20:18:32] ...trap/engines/engine.py.321: eval: epoch 27 | batch 360/510 [I 2022-10-26 20:18:32] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:44 [I 2022-10-26 20:18:32] ...trap/engines/engine.py.326: process: 0.30685 | load: 0.00016 [I 2022-10-26 20:18:35] ...trap/engines/engine.py.321: eval: epoch 27 | batch 370/510 [I 2022-10-26 20:18:35] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 20:18:35] ...trap/engines/engine.py.326: process: 0.28350 | load: 0.00015 [I 2022-10-26 20:18:37] ...trap/engines/engine.py.321: eval: epoch 27 | batch 380/510 [I 2022-10-26 20:18:37] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:38 [I 2022-10-26 20:18:37] ...trap/engines/engine.py.326: process: 0.28693 | load: 0.00023 [I 2022-10-26 20:18:40] ...trap/engines/engine.py.321: eval: epoch 27 | batch 390/510 [I 2022-10-26 20:18:40] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 20:18:40] ...trap/engines/engine.py.326: process: 0.28850 | load: 0.00018 [I 2022-10-26 20:18:43] ...trap/engines/engine.py.321: eval: epoch 27 | batch 400/510 [I 2022-10-26 20:18:43] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 20:18:43] ...trap/engines/engine.py.326: process: 0.28662 | load: 0.00016 [I 2022-10-26 20:18:46] ...trap/engines/engine.py.321: eval: epoch 27 | batch 410/510 [I 2022-10-26 20:18:46] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 20:18:46] ...trap/engines/engine.py.326: process: 0.29238 | load: 0.00016 [I 2022-10-26 20:18:49] ...trap/engines/engine.py.321: eval: epoch 27 | batch 420/510 [I 2022-10-26 20:18:49] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 20:18:49] ...trap/engines/engine.py.326: process: 0.27692 | load: 0.00020 [I 2022-10-26 20:18:52] ...trap/engines/engine.py.321: eval: epoch 27 | batch 430/510 [I 2022-10-26 20:18:52] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:22 [I 2022-10-26 20:18:52] ...trap/engines/engine.py.326: process: 0.28368 | load: 0.00021 [I 2022-10-26 20:18:55] ...trap/engines/engine.py.321: eval: epoch 27 | batch 440/510 [I 2022-10-26 20:18:55] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 20:18:55] ...trap/engines/engine.py.326: process: 0.28286 | load: 0.00019 [I 2022-10-26 20:18:58] ...trap/engines/engine.py.321: eval: epoch 27 | batch 450/510 [I 2022-10-26 20:18:58] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 20:18:58] ...trap/engines/engine.py.326: process: 0.29286 | load: 0.00023 [I 2022-10-26 20:19:00] ...trap/engines/engine.py.321: eval: epoch 27 | batch 460/510 [I 2022-10-26 20:19:00] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 20:19:00] ...trap/engines/engine.py.326: process: 0.28543 | load: 0.00015 [I 2022-10-26 20:19:03] ...trap/engines/engine.py.321: eval: epoch 27 | batch 470/510 [I 2022-10-26 20:19:03] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 20:19:03] ...trap/engines/engine.py.326: process: 0.28818 | load: 0.00016 [I 2022-10-26 20:19:06] ...trap/engines/engine.py.321: eval: epoch 27 | batch 480/510 [I 2022-10-26 20:19:06] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 20:19:06] ...trap/engines/engine.py.326: process: 0.28945 | load: 0.00024 [I 2022-10-26 20:19:09] ...trap/engines/engine.py.321: eval: epoch 27 | batch 490/510 [I 2022-10-26 20:19:09] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 20:19:09] ...trap/engines/engine.py.326: process: 0.29042 | load: 0.00018 [I 2022-10-26 20:19:12] ...trap/engines/engine.py.321: eval: epoch 27 | batch 500/510 [I 2022-10-26 20:19:12] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 20:19:12] ...trap/engines/engine.py.326: process: 0.28354 | load: 0.00024 [I 2022-10-26 20:19:15] ...trap/engines/engine.py.321: eval: epoch 27 | batch 510/510 [I 2022-10-26 20:19:15] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 20:19:15] ...trap/engines/engine.py.326: process: 0.28812 | load: 0.00015 [S 2022-10-26 20:19:16] ...trap/engines/engine.py.132: eval_epoch.epoch: 27 [I 2022-10-26 20:19:16] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.645200 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.872300 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.918400 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.638700 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.867700 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.913800 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014559 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010149 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009942 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013950 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009571 [S 2022-10-26 20:19:20] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008085 [I 2022-10-26 20:19:21] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 20:19:21] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 20:19:21] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 20:19:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 20:19:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 20:19:22] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 20:19:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 20:19:30] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 20:19:30] ...trap/engines/engine.py.163: Training model on trainset for epoch 28 [I 2022-10-26 20:19:30] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 20:19:33] ...trap/engines/engine.py.224: train: epoch 28 | batch 0/2304 [I 2022-10-26 20:19:33] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:40:34 [I 2022-10-26 20:19:33] ...trap/engines/engine.py.229: process: 2.61920 | load: 1.97935 [I 2022-10-26 20:19:33] ...trap/engines/engine.py.230: loss: 0.33334 [I 2022-10-26 20:19:33] ...optimizers/trijoint.py.151: total_norm: 26.222244 [I 2022-10-26 20:19:38] ...trap/engines/engine.py.224: train: epoch 28 | batch 10/2304 [I 2022-10-26 20:19:38] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:39 [I 2022-10-26 20:19:38] ...trap/engines/engine.py.229: process: 0.55320 | load: 0.00463 [I 2022-10-26 20:19:38] ...trap/engines/engine.py.230: loss: 0.14555 [I 2022-10-26 20:19:38] ...optimizers/trijoint.py.151: total_norm: 22.773115 [I 2022-10-26 20:19:44] ...trap/engines/engine.py.224: train: epoch 28 | batch 20/2304 [I 2022-10-26 20:19:44] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:06 [I 2022-10-26 20:19:44] ...trap/engines/engine.py.229: process: 0.54674 | load: 0.00342 [I 2022-10-26 20:19:44] ...trap/engines/engine.py.230: loss: 0.12378 [I 2022-10-26 20:19:44] ...optimizers/trijoint.py.151: total_norm: 18.692034 [I 2022-10-26 20:19:50] ...trap/engines/engine.py.224: train: epoch 28 | batch 30/2304 [I 2022-10-26 20:19:50] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:04 [I 2022-10-26 20:19:50] ...trap/engines/engine.py.229: process: 0.54355 | load: 0.00292 [I 2022-10-26 20:19:50] ...trap/engines/engine.py.230: loss: 0.16121 [I 2022-10-26 20:19:50] ...optimizers/trijoint.py.151: total_norm: 13.431403 [I 2022-10-26 20:19:55] ...trap/engines/engine.py.224: train: epoch 28 | batch 40/2304 [I 2022-10-26 20:19:55] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:54 [I 2022-10-26 20:19:55] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00300 [I 2022-10-26 20:19:55] ...trap/engines/engine.py.230: loss: 0.15640 [I 2022-10-26 20:19:55] ...optimizers/trijoint.py.151: total_norm: 19.237110 [I 2022-10-26 20:20:01] ...trap/engines/engine.py.224: train: epoch 28 | batch 50/2304 [I 2022-10-26 20:20:01] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:44 [I 2022-10-26 20:20:01] ...trap/engines/engine.py.229: process: 0.55095 | load: 0.00259 [I 2022-10-26 20:20:01] ...trap/engines/engine.py.230: loss: 0.12441 [I 2022-10-26 20:20:01] ...optimizers/trijoint.py.151: total_norm: 16.541271 [I 2022-10-26 20:20:06] ...trap/engines/engine.py.224: train: epoch 28 | batch 60/2304 [I 2022-10-26 20:20:06] ...trap/engines/engine.py.228: elapsed: 0:00:35 | left: 0:20:38 [I 2022-10-26 20:20:06] ...trap/engines/engine.py.229: process: 0.56260 | load: 0.00307 [I 2022-10-26 20:20:06] ...trap/engines/engine.py.230: loss: 0.26566 [I 2022-10-26 20:20:06] ...optimizers/trijoint.py.151: total_norm: 22.326990 [I 2022-10-26 20:20:12] ...trap/engines/engine.py.224: train: epoch 28 | batch 70/2304 [I 2022-10-26 20:20:12] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:21:06 [I 2022-10-26 20:20:12] ...trap/engines/engine.py.229: process: 0.56734 | load: 0.00291 [I 2022-10-26 20:20:12] ...trap/engines/engine.py.230: loss: 0.25834 [I 2022-10-26 20:20:12] ...optimizers/trijoint.py.151: total_norm: 26.591192 [I 2022-10-26 20:20:17] ...trap/engines/engine.py.224: train: epoch 28 | batch 80/2304 [I 2022-10-26 20:20:17] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:38 [I 2022-10-26 20:20:17] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00262 [I 2022-10-26 20:20:17] ...trap/engines/engine.py.230: loss: 0.19720 [I 2022-10-26 20:20:17] ...optimizers/trijoint.py.151: total_norm: 23.713539 [I 2022-10-26 20:20:23] ...trap/engines/engine.py.224: train: epoch 28 | batch 90/2304 [I 2022-10-26 20:20:23] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:33 [I 2022-10-26 20:20:23] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00254 [I 2022-10-26 20:20:23] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-26 20:20:23] ...optimizers/trijoint.py.151: total_norm: 13.530799 [I 2022-10-26 20:20:28] ...trap/engines/engine.py.224: train: epoch 28 | batch 100/2304 [I 2022-10-26 20:20:28] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:19 [I 2022-10-26 20:20:28] ...trap/engines/engine.py.229: process: 0.55364 | load: 0.00251 [I 2022-10-26 20:20:28] ...trap/engines/engine.py.230: loss: 0.14934 [I 2022-10-26 20:20:28] ...optimizers/trijoint.py.151: total_norm: 13.889965 [I 2022-10-26 20:20:34] ...trap/engines/engine.py.224: train: epoch 28 | batch 110/2304 [I 2022-10-26 20:20:34] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:20:21 [I 2022-10-26 20:20:34] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00244 [I 2022-10-26 20:20:34] ...trap/engines/engine.py.230: loss: 0.17292 [I 2022-10-26 20:20:34] ...optimizers/trijoint.py.151: total_norm: 20.762850 [I 2022-10-26 20:20:40] ...trap/engines/engine.py.224: train: epoch 28 | batch 120/2304 [I 2022-10-26 20:20:40] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:07 [I 2022-10-26 20:20:40] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00247 [I 2022-10-26 20:20:40] ...trap/engines/engine.py.230: loss: 0.23612 [I 2022-10-26 20:20:40] ...optimizers/trijoint.py.151: total_norm: 16.163111 [I 2022-10-26 20:20:45] ...trap/engines/engine.py.224: train: epoch 28 | batch 130/2304 [I 2022-10-26 20:20:45] ...trap/engines/engine.py.228: elapsed: 0:01:14 | left: 0:20:10 [I 2022-10-26 20:20:45] ...trap/engines/engine.py.229: process: 0.54736 | load: 0.00263 [I 2022-10-26 20:20:45] ...trap/engines/engine.py.230: loss: 0.24599 [I 2022-10-26 20:20:45] ...optimizers/trijoint.py.151: total_norm: 19.422411 [I 2022-10-26 20:20:51] ...trap/engines/engine.py.224: train: epoch 28 | batch 140/2304 [I 2022-10-26 20:20:51] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:59 [I 2022-10-26 20:20:51] ...trap/engines/engine.py.229: process: 0.54714 | load: 0.00248 [I 2022-10-26 20:20:51] ...trap/engines/engine.py.230: loss: 0.17172 [I 2022-10-26 20:20:51] ...optimizers/trijoint.py.151: total_norm: 20.020065 [I 2022-10-26 20:20:56] ...trap/engines/engine.py.224: train: epoch 28 | batch 150/2304 [I 2022-10-26 20:20:56] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:14 [I 2022-10-26 20:20:56] ...trap/engines/engine.py.229: process: 0.56327 | load: 0.00266 [I 2022-10-26 20:20:56] ...trap/engines/engine.py.230: loss: 0.24684 [I 2022-10-26 20:20:56] ...optimizers/trijoint.py.151: total_norm: 21.895382 [I 2022-10-26 20:21:02] ...trap/engines/engine.py.224: train: epoch 28 | batch 160/2304 [I 2022-10-26 20:21:02] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:20:06 [I 2022-10-26 20:21:02] ...trap/engines/engine.py.229: process: 0.55993 | load: 0.00259 [I 2022-10-26 20:21:02] ...trap/engines/engine.py.230: loss: 0.16890 [I 2022-10-26 20:21:02] ...optimizers/trijoint.py.151: total_norm: 18.751720 [I 2022-10-26 20:21:08] ...trap/engines/engine.py.224: train: epoch 28 | batch 170/2304 [I 2022-10-26 20:21:08] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:43 [I 2022-10-26 20:21:08] ...trap/engines/engine.py.229: process: 0.54823 | load: 0.00252 [I 2022-10-26 20:21:08] ...trap/engines/engine.py.230: loss: 0.10548 [I 2022-10-26 20:21:08] ...optimizers/trijoint.py.151: total_norm: 14.218851 [I 2022-10-26 20:21:13] ...trap/engines/engine.py.224: train: epoch 28 | batch 180/2304 [I 2022-10-26 20:21:13] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:19:34 [I 2022-10-26 20:21:13] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00250 [I 2022-10-26 20:21:13] ...trap/engines/engine.py.230: loss: 0.18052 [I 2022-10-26 20:21:13] ...optimizers/trijoint.py.151: total_norm: 33.528946 [I 2022-10-26 20:21:19] ...trap/engines/engine.py.224: train: epoch 28 | batch 190/2304 [I 2022-10-26 20:21:19] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:38 [I 2022-10-26 20:21:19] ...trap/engines/engine.py.229: process: 0.56785 | load: 0.00261 [I 2022-10-26 20:21:19] ...trap/engines/engine.py.230: loss: 0.14302 [I 2022-10-26 20:21:19] ...optimizers/trijoint.py.151: total_norm: 11.236877 [I 2022-10-26 20:21:24] ...trap/engines/engine.py.224: train: epoch 28 | batch 200/2304 [I 2022-10-26 20:21:24] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:30 [I 2022-10-26 20:21:24] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00265 [I 2022-10-26 20:21:24] ...trap/engines/engine.py.230: loss: 0.16502 [I 2022-10-26 20:21:24] ...optimizers/trijoint.py.151: total_norm: 20.048607 [I 2022-10-26 20:21:30] ...trap/engines/engine.py.224: train: epoch 28 | batch 210/2304 [I 2022-10-26 20:21:30] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:25 [I 2022-10-26 20:21:30] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00249 [I 2022-10-26 20:21:30] ...trap/engines/engine.py.230: loss: 0.14044 [I 2022-10-26 20:21:30] ...optimizers/trijoint.py.151: total_norm: 20.527262 [I 2022-10-26 20:21:35] ...trap/engines/engine.py.224: train: epoch 28 | batch 220/2304 [I 2022-10-26 20:21:35] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:23 [I 2022-10-26 20:21:35] ...trap/engines/engine.py.229: process: 0.55046 | load: 0.00264 [I 2022-10-26 20:21:35] ...trap/engines/engine.py.230: loss: 0.18854 [I 2022-10-26 20:21:35] ...optimizers/trijoint.py.151: total_norm: 18.122126 [I 2022-10-26 20:21:41] ...trap/engines/engine.py.224: train: epoch 28 | batch 230/2304 [I 2022-10-26 20:21:41] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:14 [I 2022-10-26 20:21:41] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00254 [I 2022-10-26 20:21:41] ...trap/engines/engine.py.230: loss: 0.17121 [I 2022-10-26 20:21:41] ...optimizers/trijoint.py.151: total_norm: 17.481592 [I 2022-10-26 20:21:47] ...trap/engines/engine.py.224: train: epoch 28 | batch 240/2304 [I 2022-10-26 20:21:47] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:01 [I 2022-10-26 20:21:47] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00266 [I 2022-10-26 20:21:47] ...trap/engines/engine.py.230: loss: 0.20726 [I 2022-10-26 20:21:47] ...optimizers/trijoint.py.151: total_norm: 17.924080 [I 2022-10-26 20:21:52] ...trap/engines/engine.py.224: train: epoch 28 | batch 250/2304 [I 2022-10-26 20:21:52] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:18:58 [I 2022-10-26 20:21:52] ...trap/engines/engine.py.229: process: 0.55819 | load: 0.00263 [I 2022-10-26 20:21:52] ...trap/engines/engine.py.230: loss: 0.18811 [I 2022-10-26 20:21:52] ...optimizers/trijoint.py.151: total_norm: 26.846096 [I 2022-10-26 20:21:58] ...trap/engines/engine.py.224: train: epoch 28 | batch 260/2304 [I 2022-10-26 20:21:58] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:00 [I 2022-10-26 20:21:58] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00255 [I 2022-10-26 20:21:58] ...trap/engines/engine.py.230: loss: 0.15574 [I 2022-10-26 20:21:58] ...optimizers/trijoint.py.151: total_norm: 20.275759 [I 2022-10-26 20:22:03] ...trap/engines/engine.py.224: train: epoch 28 | batch 270/2304 [I 2022-10-26 20:22:03] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:47 [I 2022-10-26 20:22:03] ...trap/engines/engine.py.229: process: 0.54780 | load: 0.00245 [I 2022-10-26 20:22:03] ...trap/engines/engine.py.230: loss: 0.22314 [I 2022-10-26 20:22:03] ...optimizers/trijoint.py.151: total_norm: 25.074892 [I 2022-10-26 20:22:09] ...trap/engines/engine.py.224: train: epoch 28 | batch 280/2304 [I 2022-10-26 20:22:09] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:50 [I 2022-10-26 20:22:09] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00245 [I 2022-10-26 20:22:09] ...trap/engines/engine.py.230: loss: 0.19898 [I 2022-10-26 20:22:09] ...optimizers/trijoint.py.151: total_norm: 26.321264 [I 2022-10-26 20:22:14] ...trap/engines/engine.py.224: train: epoch 28 | batch 290/2304 [I 2022-10-26 20:22:14] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:36 [I 2022-10-26 20:22:14] ...trap/engines/engine.py.229: process: 0.55614 | load: 0.00252 [I 2022-10-26 20:22:14] ...trap/engines/engine.py.230: loss: 0.15207 [I 2022-10-26 20:22:14] ...optimizers/trijoint.py.151: total_norm: 18.255438 [I 2022-10-26 20:22:20] ...trap/engines/engine.py.224: train: epoch 28 | batch 300/2304 [I 2022-10-26 20:22:20] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:18:34 [I 2022-10-26 20:22:20] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00243 [I 2022-10-26 20:22:20] ...trap/engines/engine.py.230: loss: 0.18211 [I 2022-10-26 20:22:20] ...optimizers/trijoint.py.151: total_norm: 18.548649 [I 2022-10-26 20:22:25] ...trap/engines/engine.py.224: train: epoch 28 | batch 310/2304 [I 2022-10-26 20:22:25] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:22 [I 2022-10-26 20:22:25] ...trap/engines/engine.py.229: process: 0.55366 | load: 0.00262 [I 2022-10-26 20:22:25] ...trap/engines/engine.py.230: loss: 0.17180 [I 2022-10-26 20:22:25] ...optimizers/trijoint.py.151: total_norm: 17.363663 [I 2022-10-26 20:22:31] ...trap/engines/engine.py.224: train: epoch 28 | batch 320/2304 [I 2022-10-26 20:22:31] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:18:19 [I 2022-10-26 20:22:31] ...trap/engines/engine.py.229: process: 0.54216 | load: 0.00246 [I 2022-10-26 20:22:31] ...trap/engines/engine.py.230: loss: 0.16678 [I 2022-10-26 20:22:31] ...optimizers/trijoint.py.151: total_norm: 26.145479 [I 2022-10-26 20:22:37] ...trap/engines/engine.py.224: train: epoch 28 | batch 330/2304 [I 2022-10-26 20:22:37] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:21 [I 2022-10-26 20:22:37] ...trap/engines/engine.py.229: process: 0.56455 | load: 0.00243 [I 2022-10-26 20:22:37] ...trap/engines/engine.py.230: loss: 0.15000 [I 2022-10-26 20:22:37] ...optimizers/trijoint.py.151: total_norm: 20.364059 [I 2022-10-26 20:22:42] ...trap/engines/engine.py.224: train: epoch 28 | batch 340/2304 [I 2022-10-26 20:22:42] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:05 [I 2022-10-26 20:22:42] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00262 [I 2022-10-26 20:22:42] ...trap/engines/engine.py.230: loss: 0.15894 [I 2022-10-26 20:22:42] ...optimizers/trijoint.py.151: total_norm: 17.285803 [I 2022-10-26 20:22:48] ...trap/engines/engine.py.224: train: epoch 28 | batch 350/2304 [I 2022-10-26 20:22:48] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:18:03 [I 2022-10-26 20:22:48] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00248 [I 2022-10-26 20:22:48] ...trap/engines/engine.py.230: loss: 0.17419 [I 2022-10-26 20:22:48] ...optimizers/trijoint.py.151: total_norm: 18.494148 [I 2022-10-26 20:22:53] ...trap/engines/engine.py.224: train: epoch 28 | batch 360/2304 [I 2022-10-26 20:22:53] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:59 [I 2022-10-26 20:22:53] ...trap/engines/engine.py.229: process: 0.54930 | load: 0.00252 [I 2022-10-26 20:22:53] ...trap/engines/engine.py.230: loss: 0.15690 [I 2022-10-26 20:22:53] ...optimizers/trijoint.py.151: total_norm: 10.190279 [I 2022-10-26 20:22:59] ...trap/engines/engine.py.224: train: epoch 28 | batch 370/2304 [I 2022-10-26 20:22:59] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:17:53 [I 2022-10-26 20:22:59] ...trap/engines/engine.py.229: process: 0.56576 | load: 0.00246 [I 2022-10-26 20:22:59] ...trap/engines/engine.py.230: loss: 0.17106 [I 2022-10-26 20:22:59] ...optimizers/trijoint.py.151: total_norm: 15.245710 [I 2022-10-26 20:23:04] ...trap/engines/engine.py.224: train: epoch 28 | batch 380/2304 [I 2022-10-26 20:23:04] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:42 [I 2022-10-26 20:23:04] ...trap/engines/engine.py.229: process: 0.54425 | load: 0.00242 [I 2022-10-26 20:23:04] ...trap/engines/engine.py.230: loss: 0.11453 [I 2022-10-26 20:23:04] ...optimizers/trijoint.py.151: total_norm: 15.647587 [I 2022-10-26 20:23:10] ...trap/engines/engine.py.224: train: epoch 28 | batch 390/2304 [I 2022-10-26 20:23:10] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:17:41 [I 2022-10-26 20:23:10] ...trap/engines/engine.py.229: process: 0.54986 | load: 0.00250 [I 2022-10-26 20:23:10] ...trap/engines/engine.py.230: loss: 0.17564 [I 2022-10-26 20:23:10] ...optimizers/trijoint.py.151: total_norm: 13.025212 [I 2022-10-26 20:23:16] ...trap/engines/engine.py.224: train: epoch 28 | batch 400/2304 [I 2022-10-26 20:23:16] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:37 [I 2022-10-26 20:23:16] ...trap/engines/engine.py.229: process: 0.56480 | load: 0.00245 [I 2022-10-26 20:23:16] ...trap/engines/engine.py.230: loss: 0.24272 [I 2022-10-26 20:23:16] ...optimizers/trijoint.py.151: total_norm: 28.928131 [I 2022-10-26 20:23:21] ...trap/engines/engine.py.224: train: epoch 28 | batch 410/2304 [I 2022-10-26 20:23:21] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:17:28 [I 2022-10-26 20:23:21] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00245 [I 2022-10-26 20:23:21] ...trap/engines/engine.py.230: loss: 0.22899 [I 2022-10-26 20:23:21] ...optimizers/trijoint.py.151: total_norm: 17.054337 [I 2022-10-26 20:23:27] ...trap/engines/engine.py.224: train: epoch 28 | batch 420/2304 [I 2022-10-26 20:23:27] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:17:25 [I 2022-10-26 20:23:27] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00246 [I 2022-10-26 20:23:27] ...trap/engines/engine.py.230: loss: 0.17595 [I 2022-10-26 20:23:27] ...optimizers/trijoint.py.151: total_norm: 29.264362 [I 2022-10-26 20:23:32] ...trap/engines/engine.py.224: train: epoch 28 | batch 430/2304 [I 2022-10-26 20:23:32] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:18 [I 2022-10-26 20:23:32] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00247 [I 2022-10-26 20:23:32] ...trap/engines/engine.py.230: loss: 0.22581 [I 2022-10-26 20:23:32] ...optimizers/trijoint.py.151: total_norm: 19.024591 [I 2022-10-26 20:23:38] ...trap/engines/engine.py.224: train: epoch 28 | batch 440/2304 [I 2022-10-26 20:23:38] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:17:16 [I 2022-10-26 20:23:38] ...trap/engines/engine.py.229: process: 0.55335 | load: 0.00242 [I 2022-10-26 20:23:38] ...trap/engines/engine.py.230: loss: 0.15037 [I 2022-10-26 20:23:38] ...optimizers/trijoint.py.151: total_norm: 28.179605 [I 2022-10-26 20:23:43] ...trap/engines/engine.py.224: train: epoch 28 | batch 450/2304 [I 2022-10-26 20:23:43] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:09 [I 2022-10-26 20:23:43] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00242 [I 2022-10-26 20:23:43] ...trap/engines/engine.py.230: loss: 0.14316 [I 2022-10-26 20:23:43] ...optimizers/trijoint.py.151: total_norm: 15.703801 [I 2022-10-26 20:23:49] ...trap/engines/engine.py.224: train: epoch 28 | batch 460/2304 [I 2022-10-26 20:23:49] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:17:08 [I 2022-10-26 20:23:49] ...trap/engines/engine.py.229: process: 0.56557 | load: 0.00254 [I 2022-10-26 20:23:49] ...trap/engines/engine.py.230: loss: 0.24372 [I 2022-10-26 20:23:49] ...optimizers/trijoint.py.151: total_norm: 21.641291 [I 2022-10-26 20:23:54] ...trap/engines/engine.py.224: train: epoch 28 | batch 470/2304 [I 2022-10-26 20:23:54] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:17:03 [I 2022-10-26 20:23:54] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00251 [I 2022-10-26 20:23:54] ...trap/engines/engine.py.230: loss: 0.09159 [I 2022-10-26 20:23:54] ...optimizers/trijoint.py.151: total_norm: 8.819069 [I 2022-10-26 20:24:00] ...trap/engines/engine.py.224: train: epoch 28 | batch 480/2304 [I 2022-10-26 20:24:00] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:16:43 [I 2022-10-26 20:24:00] ...trap/engines/engine.py.229: process: 0.54427 | load: 0.00248 [I 2022-10-26 20:24:00] ...trap/engines/engine.py.230: loss: 0.18220 [I 2022-10-26 20:24:00] ...optimizers/trijoint.py.151: total_norm: 14.543406 [I 2022-10-26 20:24:06] ...trap/engines/engine.py.224: train: epoch 28 | batch 490/2304 [I 2022-10-26 20:24:06] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:16:47 [I 2022-10-26 20:24:06] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00242 [I 2022-10-26 20:24:06] ...trap/engines/engine.py.230: loss: 0.23402 [I 2022-10-26 20:24:06] ...optimizers/trijoint.py.151: total_norm: 27.392260 [I 2022-10-26 20:24:11] ...trap/engines/engine.py.224: train: epoch 28 | batch 500/2304 [I 2022-10-26 20:24:11] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:16:39 [I 2022-10-26 20:24:11] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00243 [I 2022-10-26 20:24:11] ...trap/engines/engine.py.230: loss: 0.15660 [I 2022-10-26 20:24:11] ...optimizers/trijoint.py.151: total_norm: 22.244211 [I 2022-10-26 20:24:17] ...trap/engines/engine.py.224: train: epoch 28 | batch 510/2304 [I 2022-10-26 20:24:17] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:16:36 [I 2022-10-26 20:24:17] ...trap/engines/engine.py.229: process: 0.56529 | load: 0.00244 [I 2022-10-26 20:24:17] ...trap/engines/engine.py.230: loss: 0.18089 [I 2022-10-26 20:24:17] ...optimizers/trijoint.py.151: total_norm: 23.933552 [I 2022-10-26 20:24:22] ...trap/engines/engine.py.224: train: epoch 28 | batch 520/2304 [I 2022-10-26 20:24:22] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:16:22 [I 2022-10-26 20:24:22] ...trap/engines/engine.py.229: process: 0.55408 | load: 0.00244 [I 2022-10-26 20:24:22] ...trap/engines/engine.py.230: loss: 0.26503 [I 2022-10-26 20:24:22] ...optimizers/trijoint.py.151: total_norm: 25.439409 [I 2022-10-26 20:24:28] ...trap/engines/engine.py.224: train: epoch 28 | batch 530/2304 [I 2022-10-26 20:24:28] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:16:10 [I 2022-10-26 20:24:28] ...trap/engines/engine.py.229: process: 0.53818 | load: 0.00241 [I 2022-10-26 20:24:28] ...trap/engines/engine.py.230: loss: 0.19575 [I 2022-10-26 20:24:28] ...optimizers/trijoint.py.151: total_norm: 19.743912 [I 2022-10-26 20:24:33] ...trap/engines/engine.py.224: train: epoch 28 | batch 540/2304 [I 2022-10-26 20:24:33] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:16:14 [I 2022-10-26 20:24:33] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00243 [I 2022-10-26 20:24:33] ...trap/engines/engine.py.230: loss: 0.15205 [I 2022-10-26 20:24:33] ...optimizers/trijoint.py.151: total_norm: 20.797092 [I 2022-10-26 20:24:39] ...trap/engines/engine.py.224: train: epoch 28 | batch 550/2304 [I 2022-10-26 20:24:39] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:16:11 [I 2022-10-26 20:24:39] ...trap/engines/engine.py.229: process: 0.56410 | load: 0.00240 [I 2022-10-26 20:24:39] ...trap/engines/engine.py.230: loss: 0.12175 [I 2022-10-26 20:24:39] ...optimizers/trijoint.py.151: total_norm: 42.044697 [I 2022-10-26 20:24:44] ...trap/engines/engine.py.224: train: epoch 28 | batch 560/2304 [I 2022-10-26 20:24:44] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:16:07 [I 2022-10-26 20:24:44] ...trap/engines/engine.py.229: process: 0.55193 | load: 0.00242 [I 2022-10-26 20:24:44] ...trap/engines/engine.py.230: loss: 0.21021 [I 2022-10-26 20:24:44] ...optimizers/trijoint.py.151: total_norm: 17.622557 [I 2022-10-26 20:24:50] ...trap/engines/engine.py.224: train: epoch 28 | batch 570/2304 [I 2022-10-26 20:24:50] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:15:58 [I 2022-10-26 20:24:50] ...trap/engines/engine.py.229: process: 0.54385 | load: 0.00246 [I 2022-10-26 20:24:50] ...trap/engines/engine.py.230: loss: 0.21586 [I 2022-10-26 20:24:50] ...optimizers/trijoint.py.151: total_norm: 27.558935 [I 2022-10-26 20:24:55] ...trap/engines/engine.py.224: train: epoch 28 | batch 580/2304 [I 2022-10-26 20:24:55] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:15:55 [I 2022-10-26 20:24:55] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00268 [I 2022-10-26 20:24:55] ...trap/engines/engine.py.230: loss: 0.13661 [I 2022-10-26 20:24:55] ...optimizers/trijoint.py.151: total_norm: 12.654383 [I 2022-10-26 20:25:01] ...trap/engines/engine.py.224: train: epoch 28 | batch 590/2304 [I 2022-10-26 20:25:01] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:15:51 [I 2022-10-26 20:25:01] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00245 [I 2022-10-26 20:25:01] ...trap/engines/engine.py.230: loss: 0.20252 [I 2022-10-26 20:25:01] ...optimizers/trijoint.py.151: total_norm: 19.545004 [I 2022-10-26 20:25:07] ...trap/engines/engine.py.224: train: epoch 28 | batch 600/2304 [I 2022-10-26 20:25:07] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:15:42 [I 2022-10-26 20:25:07] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00243 [I 2022-10-26 20:25:07] ...trap/engines/engine.py.230: loss: 0.13028 [I 2022-10-26 20:25:07] ...optimizers/trijoint.py.151: total_norm: 8.550417 [I 2022-10-26 20:25:12] ...trap/engines/engine.py.224: train: epoch 28 | batch 610/2304 [I 2022-10-26 20:25:12] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:15:42 [I 2022-10-26 20:25:12] ...trap/engines/engine.py.229: process: 0.56610 | load: 0.00244 [I 2022-10-26 20:25:12] ...trap/engines/engine.py.230: loss: 0.12347 [I 2022-10-26 20:25:12] ...optimizers/trijoint.py.151: total_norm: 13.600180 [I 2022-10-26 20:25:18] ...trap/engines/engine.py.224: train: epoch 28 | batch 620/2304 [I 2022-10-26 20:25:18] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:15:33 [I 2022-10-26 20:25:18] ...trap/engines/engine.py.229: process: 0.54771 | load: 0.00244 [I 2022-10-26 20:25:18] ...trap/engines/engine.py.230: loss: 0.18848 [I 2022-10-26 20:25:18] ...optimizers/trijoint.py.151: total_norm: 12.692379 [I 2022-10-26 20:25:23] ...trap/engines/engine.py.224: train: epoch 28 | batch 630/2304 [I 2022-10-26 20:25:23] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:15:29 [I 2022-10-26 20:25:23] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00240 [I 2022-10-26 20:25:23] ...trap/engines/engine.py.230: loss: 0.14645 [I 2022-10-26 20:25:23] ...optimizers/trijoint.py.151: total_norm: 19.892771 [I 2022-10-26 20:25:29] ...trap/engines/engine.py.224: train: epoch 28 | batch 640/2304 [I 2022-10-26 20:25:29] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:44 [I 2022-10-26 20:25:29] ...trap/engines/engine.py.229: process: 0.59173 | load: 0.00246 [I 2022-10-26 20:25:29] ...trap/engines/engine.py.230: loss: 0.17930 [I 2022-10-26 20:25:29] ...optimizers/trijoint.py.151: total_norm: 33.118237 [I 2022-10-26 20:25:34] ...trap/engines/engine.py.224: train: epoch 28 | batch 650/2304 [I 2022-10-26 20:25:34] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:22 [I 2022-10-26 20:25:34] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00253 [I 2022-10-26 20:25:34] ...trap/engines/engine.py.230: loss: 0.16883 [I 2022-10-26 20:25:34] ...optimizers/trijoint.py.151: total_norm: 21.493904 [I 2022-10-26 20:25:40] ...trap/engines/engine.py.224: train: epoch 28 | batch 660/2304 [I 2022-10-26 20:25:40] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:15:48 [I 2022-10-26 20:25:40] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00246 [I 2022-10-26 20:25:40] ...trap/engines/engine.py.230: loss: 0.18317 [I 2022-10-26 20:25:40] ...optimizers/trijoint.py.151: total_norm: 16.308491 [I 2022-10-26 20:25:46] ...trap/engines/engine.py.224: train: epoch 28 | batch 670/2304 [I 2022-10-26 20:25:46] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:15 [I 2022-10-26 20:25:46] ...trap/engines/engine.py.229: process: 0.56062 | load: 0.00242 [I 2022-10-26 20:25:46] ...trap/engines/engine.py.230: loss: 0.16561 [I 2022-10-26 20:25:46] ...optimizers/trijoint.py.151: total_norm: 20.211248 [I 2022-10-26 20:25:51] ...trap/engines/engine.py.224: train: epoch 28 | batch 680/2304 [I 2022-10-26 20:25:51] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:15:03 [I 2022-10-26 20:25:51] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00262 [I 2022-10-26 20:25:51] ...trap/engines/engine.py.230: loss: 0.20833 [I 2022-10-26 20:25:51] ...optimizers/trijoint.py.151: total_norm: 15.658875 [I 2022-10-26 20:25:57] ...trap/engines/engine.py.224: train: epoch 28 | batch 690/2304 [I 2022-10-26 20:25:57] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:14:57 [I 2022-10-26 20:25:57] ...trap/engines/engine.py.229: process: 0.56440 | load: 0.00244 [I 2022-10-26 20:25:57] ...trap/engines/engine.py.230: loss: 0.09518 [I 2022-10-26 20:25:57] ...optimizers/trijoint.py.151: total_norm: 17.027908 [I 2022-10-26 20:26:03] ...trap/engines/engine.py.224: train: epoch 28 | batch 700/2304 [I 2022-10-26 20:26:03] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:14:53 [I 2022-10-26 20:26:03] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00246 [I 2022-10-26 20:26:03] ...trap/engines/engine.py.230: loss: 0.22377 [I 2022-10-26 20:26:03] ...optimizers/trijoint.py.151: total_norm: 14.749143 [I 2022-10-26 20:26:08] ...trap/engines/engine.py.224: train: epoch 28 | batch 710/2304 [I 2022-10-26 20:26:08] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:14:39 [I 2022-10-26 20:26:08] ...trap/engines/engine.py.229: process: 0.55628 | load: 0.00243 [I 2022-10-26 20:26:08] ...trap/engines/engine.py.230: loss: 0.16943 [I 2022-10-26 20:26:08] ...optimizers/trijoint.py.151: total_norm: 17.351278 [I 2022-10-26 20:26:14] ...trap/engines/engine.py.224: train: epoch 28 | batch 720/2304 [I 2022-10-26 20:26:14] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:14:41 [I 2022-10-26 20:26:14] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00242 [I 2022-10-26 20:26:14] ...trap/engines/engine.py.230: loss: 0.12912 [I 2022-10-26 20:26:14] ...optimizers/trijoint.py.151: total_norm: 17.570326 [I 2022-10-26 20:26:19] ...trap/engines/engine.py.224: train: epoch 28 | batch 730/2304 [I 2022-10-26 20:26:19] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:14:40 [I 2022-10-26 20:26:19] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00251 [I 2022-10-26 20:26:19] ...trap/engines/engine.py.230: loss: 0.17082 [I 2022-10-26 20:26:19] ...optimizers/trijoint.py.151: total_norm: 22.766939 [I 2022-10-26 20:26:25] ...trap/engines/engine.py.224: train: epoch 28 | batch 740/2304 [I 2022-10-26 20:26:25] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:14:24 [I 2022-10-26 20:26:25] ...trap/engines/engine.py.229: process: 0.54162 | load: 0.00243 [I 2022-10-26 20:26:25] ...trap/engines/engine.py.230: loss: 0.17469 [I 2022-10-26 20:26:25] ...optimizers/trijoint.py.151: total_norm: 20.527628 [I 2022-10-26 20:26:30] ...trap/engines/engine.py.224: train: epoch 28 | batch 750/2304 [I 2022-10-26 20:26:30] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:27 [I 2022-10-26 20:26:30] ...trap/engines/engine.py.229: process: 0.55243 | load: 0.00244 [I 2022-10-26 20:26:30] ...trap/engines/engine.py.230: loss: 0.14903 [I 2022-10-26 20:26:30] ...optimizers/trijoint.py.151: total_norm: 12.643691 [I 2022-10-26 20:26:36] ...trap/engines/engine.py.224: train: epoch 28 | batch 760/2304 [I 2022-10-26 20:26:36] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:15 [I 2022-10-26 20:26:36] ...trap/engines/engine.py.229: process: 0.55486 | load: 0.00245 [I 2022-10-26 20:26:36] ...trap/engines/engine.py.230: loss: 0.17505 [I 2022-10-26 20:26:36] ...optimizers/trijoint.py.151: total_norm: 15.053998 [I 2022-10-26 20:26:42] ...trap/engines/engine.py.224: train: epoch 28 | batch 770/2304 [I 2022-10-26 20:26:42] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:15 [I 2022-10-26 20:26:42] ...trap/engines/engine.py.229: process: 0.56710 | load: 0.00251 [I 2022-10-26 20:26:42] ...trap/engines/engine.py.230: loss: 0.12109 [I 2022-10-26 20:26:42] ...optimizers/trijoint.py.151: total_norm: 13.234187 [I 2022-10-26 20:26:47] ...trap/engines/engine.py.224: train: epoch 28 | batch 780/2304 [I 2022-10-26 20:26:47] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:08 [I 2022-10-26 20:26:47] ...trap/engines/engine.py.229: process: 0.56227 | load: 0.00246 [I 2022-10-26 20:26:47] ...trap/engines/engine.py.230: loss: 0.17365 [I 2022-10-26 20:26:47] ...optimizers/trijoint.py.151: total_norm: 23.117193 [I 2022-10-26 20:26:53] ...trap/engines/engine.py.224: train: epoch 28 | batch 790/2304 [I 2022-10-26 20:26:53] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:00 [I 2022-10-26 20:26:53] ...trap/engines/engine.py.229: process: 0.55301 | load: 0.00240 [I 2022-10-26 20:26:53] ...trap/engines/engine.py.230: loss: 0.16932 [I 2022-10-26 20:26:53] ...optimizers/trijoint.py.151: total_norm: 38.060005 [I 2022-10-26 20:26:58] ...trap/engines/engine.py.224: train: epoch 28 | batch 800/2304 [I 2022-10-26 20:26:58] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:29 [I 2022-10-26 20:26:58] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00242 [I 2022-10-26 20:26:58] ...trap/engines/engine.py.230: loss: 0.29357 [I 2022-10-26 20:26:58] ...optimizers/trijoint.py.151: total_norm: 14.841183 [I 2022-10-26 20:27:04] ...trap/engines/engine.py.224: train: epoch 28 | batch 810/2304 [I 2022-10-26 20:27:04] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:13:52 [I 2022-10-26 20:27:04] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00244 [I 2022-10-26 20:27:04] ...trap/engines/engine.py.230: loss: 0.17688 [I 2022-10-26 20:27:04] ...optimizers/trijoint.py.151: total_norm: 17.111364 [I 2022-10-26 20:27:10] ...trap/engines/engine.py.224: train: epoch 28 | batch 820/2304 [I 2022-10-26 20:27:10] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:17 [I 2022-10-26 20:27:10] ...trap/engines/engine.py.229: process: 0.54529 | load: 0.00255 [I 2022-10-26 20:27:10] ...trap/engines/engine.py.230: loss: 0.13876 [I 2022-10-26 20:27:10] ...optimizers/trijoint.py.151: total_norm: 13.250764 [I 2022-10-26 20:27:15] ...trap/engines/engine.py.224: train: epoch 28 | batch 830/2304 [I 2022-10-26 20:27:15] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:13:49 [I 2022-10-26 20:27:15] ...trap/engines/engine.py.229: process: 0.55858 | load: 0.00250 [I 2022-10-26 20:27:15] ...trap/engines/engine.py.230: loss: 0.15502 [I 2022-10-26 20:27:15] ...optimizers/trijoint.py.151: total_norm: 43.902618 [I 2022-10-26 20:27:21] ...trap/engines/engine.py.224: train: epoch 28 | batch 840/2304 [I 2022-10-26 20:27:21] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:13:34 [I 2022-10-26 20:27:21] ...trap/engines/engine.py.229: process: 0.54209 | load: 0.00248 [I 2022-10-26 20:27:21] ...trap/engines/engine.py.230: loss: 0.14125 [I 2022-10-26 20:27:21] ...optimizers/trijoint.py.151: total_norm: 15.923613 [I 2022-10-26 20:27:26] ...trap/engines/engine.py.224: train: epoch 28 | batch 850/2304 [I 2022-10-26 20:27:26] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:28 [I 2022-10-26 20:27:26] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00245 [I 2022-10-26 20:27:26] ...trap/engines/engine.py.230: loss: 0.15518 [I 2022-10-26 20:27:26] ...optimizers/trijoint.py.151: total_norm: 19.375217 [I 2022-10-26 20:27:32] ...trap/engines/engine.py.224: train: epoch 28 | batch 860/2304 [I 2022-10-26 20:27:32] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:13:17 [I 2022-10-26 20:27:32] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00245 [I 2022-10-26 20:27:32] ...trap/engines/engine.py.230: loss: 0.22654 [I 2022-10-26 20:27:32] ...optimizers/trijoint.py.151: total_norm: 20.970423 [I 2022-10-26 20:27:37] ...trap/engines/engine.py.224: train: epoch 28 | batch 870/2304 [I 2022-10-26 20:27:37] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:15 [I 2022-10-26 20:27:37] ...trap/engines/engine.py.229: process: 0.55098 | load: 0.00244 [I 2022-10-26 20:27:37] ...trap/engines/engine.py.230: loss: 0.24415 [I 2022-10-26 20:27:37] ...optimizers/trijoint.py.151: total_norm: 18.221212 [I 2022-10-26 20:27:43] ...trap/engines/engine.py.224: train: epoch 28 | batch 880/2304 [I 2022-10-26 20:27:43] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:13:08 [I 2022-10-26 20:27:43] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00243 [I 2022-10-26 20:27:43] ...trap/engines/engine.py.230: loss: 0.16134 [I 2022-10-26 20:27:43] ...optimizers/trijoint.py.151: total_norm: 25.640705 [I 2022-10-26 20:27:49] ...trap/engines/engine.py.224: train: epoch 28 | batch 890/2304 [I 2022-10-26 20:27:49] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:00 [I 2022-10-26 20:27:49] ...trap/engines/engine.py.229: process: 0.54219 | load: 0.00243 [I 2022-10-26 20:27:49] ...trap/engines/engine.py.230: loss: 0.26412 [I 2022-10-26 20:27:49] ...optimizers/trijoint.py.151: total_norm: 25.460581 [I 2022-10-26 20:27:54] ...trap/engines/engine.py.224: train: epoch 28 | batch 900/2304 [I 2022-10-26 20:27:54] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:12:56 [I 2022-10-26 20:27:54] ...trap/engines/engine.py.229: process: 0.54079 | load: 0.00245 [I 2022-10-26 20:27:54] ...trap/engines/engine.py.230: loss: 0.17886 [I 2022-10-26 20:27:54] ...optimizers/trijoint.py.151: total_norm: 14.706965 [I 2022-10-26 20:28:00] ...trap/engines/engine.py.224: train: epoch 28 | batch 910/2304 [I 2022-10-26 20:28:00] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:12:53 [I 2022-10-26 20:28:00] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00267 [I 2022-10-26 20:28:00] ...trap/engines/engine.py.230: loss: 0.12750 [I 2022-10-26 20:28:00] ...optimizers/trijoint.py.151: total_norm: 12.622348 [I 2022-10-26 20:28:05] ...trap/engines/engine.py.224: train: epoch 28 | batch 920/2304 [I 2022-10-26 20:28:05] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:12:53 [I 2022-10-26 20:28:05] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00243 [I 2022-10-26 20:28:05] ...trap/engines/engine.py.230: loss: 0.12693 [I 2022-10-26 20:28:05] ...optimizers/trijoint.py.151: total_norm: 14.639838 [I 2022-10-26 20:28:11] ...trap/engines/engine.py.224: train: epoch 28 | batch 930/2304 [I 2022-10-26 20:28:11] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:12:40 [I 2022-10-26 20:28:11] ...trap/engines/engine.py.229: process: 0.55892 | load: 0.00242 [I 2022-10-26 20:28:11] ...trap/engines/engine.py.230: loss: 0.16005 [I 2022-10-26 20:28:11] ...optimizers/trijoint.py.151: total_norm: 23.601841 [I 2022-10-26 20:28:16] ...trap/engines/engine.py.224: train: epoch 28 | batch 940/2304 [I 2022-10-26 20:28:16] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:12:32 [I 2022-10-26 20:28:16] ...trap/engines/engine.py.229: process: 0.54386 | load: 0.00244 [I 2022-10-26 20:28:16] ...trap/engines/engine.py.230: loss: 0.23774 [I 2022-10-26 20:28:16] ...optimizers/trijoint.py.151: total_norm: 20.366402 [I 2022-10-26 20:28:22] ...trap/engines/engine.py.224: train: epoch 28 | batch 950/2304 [I 2022-10-26 20:28:22] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:12:32 [I 2022-10-26 20:28:22] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00242 [I 2022-10-26 20:28:22] ...trap/engines/engine.py.230: loss: 0.19686 [I 2022-10-26 20:28:22] ...optimizers/trijoint.py.151: total_norm: 15.171567 [I 2022-10-26 20:28:27] ...trap/engines/engine.py.224: train: epoch 28 | batch 960/2304 [I 2022-10-26 20:28:27] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:24 [I 2022-10-26 20:28:27] ...trap/engines/engine.py.229: process: 0.54719 | load: 0.00247 [I 2022-10-26 20:28:27] ...trap/engines/engine.py.230: loss: 0.16667 [I 2022-10-26 20:28:27] ...optimizers/trijoint.py.151: total_norm: 16.741888 [I 2022-10-26 20:28:33] ...trap/engines/engine.py.224: train: epoch 28 | batch 970/2304 [I 2022-10-26 20:28:33] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:12:19 [I 2022-10-26 20:28:33] ...trap/engines/engine.py.229: process: 0.55632 | load: 0.00247 [I 2022-10-26 20:28:33] ...trap/engines/engine.py.230: loss: 0.17644 [I 2022-10-26 20:28:33] ...optimizers/trijoint.py.151: total_norm: 19.332951 [I 2022-10-26 20:28:39] ...trap/engines/engine.py.224: train: epoch 28 | batch 980/2304 [I 2022-10-26 20:28:39] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:12:10 [I 2022-10-26 20:28:39] ...trap/engines/engine.py.229: process: 0.54495 | load: 0.00251 [I 2022-10-26 20:28:39] ...trap/engines/engine.py.230: loss: 0.22571 [I 2022-10-26 20:28:39] ...optimizers/trijoint.py.151: total_norm: 19.667597 [I 2022-10-26 20:28:44] ...trap/engines/engine.py.224: train: epoch 28 | batch 990/2304 [I 2022-10-26 20:28:44] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:10 [I 2022-10-26 20:28:44] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00247 [I 2022-10-26 20:28:44] ...trap/engines/engine.py.230: loss: 0.16483 [I 2022-10-26 20:28:44] ...optimizers/trijoint.py.151: total_norm: 15.151729 [I 2022-10-26 20:28:50] ...trap/engines/engine.py.224: train: epoch 28 | batch 1000/2304 [I 2022-10-26 20:28:50] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:12:04 [I 2022-10-26 20:28:50] ...trap/engines/engine.py.229: process: 0.55614 | load: 0.00246 [I 2022-10-26 20:28:50] ...trap/engines/engine.py.230: loss: 0.13521 [I 2022-10-26 20:28:50] ...optimizers/trijoint.py.151: total_norm: 15.910837 [I 2022-10-26 20:28:55] ...trap/engines/engine.py.224: train: epoch 28 | batch 1010/2304 [I 2022-10-26 20:28:55] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:11:51 [I 2022-10-26 20:28:55] ...trap/engines/engine.py.229: process: 0.55408 | load: 0.00243 [I 2022-10-26 20:28:55] ...trap/engines/engine.py.230: loss: 0.15745 [I 2022-10-26 20:28:55] ...optimizers/trijoint.py.151: total_norm: 18.379251 [I 2022-10-26 20:29:01] ...trap/engines/engine.py.224: train: epoch 28 | batch 1020/2304 [I 2022-10-26 20:29:01] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:11:52 [I 2022-10-26 20:29:01] ...trap/engines/engine.py.229: process: 0.56215 | load: 0.00244 [I 2022-10-26 20:29:01] ...trap/engines/engine.py.230: loss: 0.17772 [I 2022-10-26 20:29:01] ...optimizers/trijoint.py.151: total_norm: 11.841476 [I 2022-10-26 20:29:06] ...trap/engines/engine.py.224: train: epoch 28 | batch 1030/2304 [I 2022-10-26 20:29:06] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:11:51 [I 2022-10-26 20:29:06] ...trap/engines/engine.py.229: process: 0.54947 | load: 0.00244 [I 2022-10-26 20:29:06] ...trap/engines/engine.py.230: loss: 0.19025 [I 2022-10-26 20:29:06] ...optimizers/trijoint.py.151: total_norm: 28.649502 [I 2022-10-26 20:29:12] ...trap/engines/engine.py.224: train: epoch 28 | batch 1040/2304 [I 2022-10-26 20:29:12] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:11:41 [I 2022-10-26 20:29:12] ...trap/engines/engine.py.229: process: 0.56119 | load: 0.00243 [I 2022-10-26 20:29:12] ...trap/engines/engine.py.230: loss: 0.22442 [I 2022-10-26 20:29:12] ...optimizers/trijoint.py.151: total_norm: 32.088345 [I 2022-10-26 20:29:17] ...trap/engines/engine.py.224: train: epoch 28 | batch 1050/2304 [I 2022-10-26 20:29:17] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:11:30 [I 2022-10-26 20:29:17] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00242 [I 2022-10-26 20:29:17] ...trap/engines/engine.py.230: loss: 0.19583 [I 2022-10-26 20:29:17] ...optimizers/trijoint.py.151: total_norm: 17.780050 [I 2022-10-26 20:29:23] ...trap/engines/engine.py.224: train: epoch 28 | batch 1060/2304 [I 2022-10-26 20:29:23] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:28 [I 2022-10-26 20:29:23] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00246 [I 2022-10-26 20:29:23] ...trap/engines/engine.py.230: loss: 0.14239 [I 2022-10-26 20:29:23] ...optimizers/trijoint.py.151: total_norm: 20.752821 [I 2022-10-26 20:29:28] ...trap/engines/engine.py.224: train: epoch 28 | batch 1070/2304 [I 2022-10-26 20:29:28] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:17 [I 2022-10-26 20:29:28] ...trap/engines/engine.py.229: process: 0.53833 | load: 0.00241 [I 2022-10-26 20:29:28] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-26 20:29:28] ...optimizers/trijoint.py.151: total_norm: 13.315490 [I 2022-10-26 20:29:34] ...trap/engines/engine.py.224: train: epoch 28 | batch 1080/2304 [I 2022-10-26 20:29:34] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:19 [I 2022-10-26 20:29:34] ...trap/engines/engine.py.229: process: 0.56414 | load: 0.00243 [I 2022-10-26 20:29:34] ...trap/engines/engine.py.230: loss: 0.18020 [I 2022-10-26 20:29:34] ...optimizers/trijoint.py.151: total_norm: 18.882143 [I 2022-10-26 20:29:39] ...trap/engines/engine.py.224: train: epoch 28 | batch 1090/2304 [I 2022-10-26 20:29:39] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:09 [I 2022-10-26 20:29:39] ...trap/engines/engine.py.229: process: 0.54160 | load: 0.00243 [I 2022-10-26 20:29:39] ...trap/engines/engine.py.230: loss: 0.18351 [I 2022-10-26 20:29:39] ...optimizers/trijoint.py.151: total_norm: 33.432434 [I 2022-10-26 20:29:45] ...trap/engines/engine.py.224: train: epoch 28 | batch 1100/2304 [I 2022-10-26 20:29:45] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:05 [I 2022-10-26 20:29:45] ...trap/engines/engine.py.229: process: 0.54831 | load: 0.00241 [I 2022-10-26 20:29:45] ...trap/engines/engine.py.230: loss: 0.20306 [I 2022-10-26 20:29:45] ...optimizers/trijoint.py.151: total_norm: 12.308595 [I 2022-10-26 20:29:51] ...trap/engines/engine.py.224: train: epoch 28 | batch 1110/2304 [I 2022-10-26 20:29:51] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:01 [I 2022-10-26 20:29:51] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00244 [I 2022-10-26 20:29:51] ...trap/engines/engine.py.230: loss: 0.20912 [I 2022-10-26 20:29:51] ...optimizers/trijoint.py.151: total_norm: 19.300013 [I 2022-10-26 20:29:56] ...trap/engines/engine.py.224: train: epoch 28 | batch 1120/2304 [I 2022-10-26 20:29:56] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:10:51 [I 2022-10-26 20:29:56] ...trap/engines/engine.py.229: process: 0.55145 | load: 0.00241 [I 2022-10-26 20:29:56] ...trap/engines/engine.py.230: loss: 0.16613 [I 2022-10-26 20:29:56] ...optimizers/trijoint.py.151: total_norm: 21.343636 [I 2022-10-26 20:30:02] ...trap/engines/engine.py.224: train: epoch 28 | batch 1130/2304 [I 2022-10-26 20:30:02] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:04 [I 2022-10-26 20:30:02] ...trap/engines/engine.py.229: process: 0.55799 | load: 0.00259 [I 2022-10-26 20:30:02] ...trap/engines/engine.py.230: loss: 0.15640 [I 2022-10-26 20:30:02] ...optimizers/trijoint.py.151: total_norm: 27.358858 [I 2022-10-26 20:30:07] ...trap/engines/engine.py.224: train: epoch 28 | batch 1140/2304 [I 2022-10-26 20:30:07] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:10:47 [I 2022-10-26 20:30:07] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00244 [I 2022-10-26 20:30:07] ...trap/engines/engine.py.230: loss: 0.16459 [I 2022-10-26 20:30:07] ...optimizers/trijoint.py.151: total_norm: 13.706308 [I 2022-10-26 20:30:13] ...trap/engines/engine.py.224: train: epoch 28 | batch 1150/2304 [I 2022-10-26 20:30:13] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:41 [I 2022-10-26 20:30:13] ...trap/engines/engine.py.229: process: 0.55106 | load: 0.00243 [I 2022-10-26 20:30:13] ...trap/engines/engine.py.230: loss: 0.15273 [I 2022-10-26 20:30:13] ...optimizers/trijoint.py.151: total_norm: 13.162225 [I 2022-10-26 20:30:18] ...trap/engines/engine.py.224: train: epoch 28 | batch 1160/2304 [I 2022-10-26 20:30:18] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:39 [I 2022-10-26 20:30:18] ...trap/engines/engine.py.229: process: 0.56243 | load: 0.00265 [I 2022-10-26 20:30:18] ...trap/engines/engine.py.230: loss: 0.20313 [I 2022-10-26 20:30:18] ...optimizers/trijoint.py.151: total_norm: 23.160095 [I 2022-10-26 20:30:24] ...trap/engines/engine.py.224: train: epoch 28 | batch 1170/2304 [I 2022-10-26 20:30:24] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:33 [I 2022-10-26 20:30:24] ...trap/engines/engine.py.229: process: 0.56004 | load: 0.00250 [I 2022-10-26 20:30:24] ...trap/engines/engine.py.230: loss: 0.28373 [I 2022-10-26 20:30:24] ...optimizers/trijoint.py.151: total_norm: 17.615465 [I 2022-10-26 20:30:30] ...trap/engines/engine.py.224: train: epoch 28 | batch 1180/2304 [I 2022-10-26 20:30:30] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:22 [I 2022-10-26 20:30:30] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00247 [I 2022-10-26 20:30:30] ...trap/engines/engine.py.230: loss: 0.14728 [I 2022-10-26 20:30:30] ...optimizers/trijoint.py.151: total_norm: 30.966433 [I 2022-10-26 20:30:35] ...trap/engines/engine.py.224: train: epoch 28 | batch 1190/2304 [I 2022-10-26 20:30:35] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:19 [I 2022-10-26 20:30:35] ...trap/engines/engine.py.229: process: 0.55113 | load: 0.00244 [I 2022-10-26 20:30:35] ...trap/engines/engine.py.230: loss: 0.17200 [I 2022-10-26 20:30:35] ...optimizers/trijoint.py.151: total_norm: 13.682405 [I 2022-10-26 20:30:41] ...trap/engines/engine.py.224: train: epoch 28 | batch 1200/2304 [I 2022-10-26 20:30:41] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:12 [I 2022-10-26 20:30:41] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00245 [I 2022-10-26 20:30:41] ...trap/engines/engine.py.230: loss: 0.16649 [I 2022-10-26 20:30:41] ...optimizers/trijoint.py.151: total_norm: 16.897261 [I 2022-10-26 20:30:46] ...trap/engines/engine.py.224: train: epoch 28 | batch 1210/2304 [I 2022-10-26 20:30:46] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:07 [I 2022-10-26 20:30:46] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00246 [I 2022-10-26 20:30:46] ...trap/engines/engine.py.230: loss: 0.20877 [I 2022-10-26 20:30:46] ...optimizers/trijoint.py.151: total_norm: 21.836006 [I 2022-10-26 20:30:52] ...trap/engines/engine.py.224: train: epoch 28 | batch 1220/2304 [I 2022-10-26 20:30:52] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:01 [I 2022-10-26 20:30:52] ...trap/engines/engine.py.229: process: 0.54404 | load: 0.00244 [I 2022-10-26 20:30:52] ...trap/engines/engine.py.230: loss: 0.14908 [I 2022-10-26 20:30:52] ...optimizers/trijoint.py.151: total_norm: 22.922823 [I 2022-10-26 20:30:57] ...trap/engines/engine.py.224: train: epoch 28 | batch 1230/2304 [I 2022-10-26 20:30:57] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:09:48 [I 2022-10-26 20:30:57] ...trap/engines/engine.py.229: process: 0.54336 | load: 0.00248 [I 2022-10-26 20:30:57] ...trap/engines/engine.py.230: loss: 0.15973 [I 2022-10-26 20:30:57] ...optimizers/trijoint.py.151: total_norm: 22.133913 [I 2022-10-26 20:31:03] ...trap/engines/engine.py.224: train: epoch 28 | batch 1240/2304 [I 2022-10-26 20:31:03] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:09:52 [I 2022-10-26 20:31:03] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00251 [I 2022-10-26 20:31:03] ...trap/engines/engine.py.230: loss: 0.17175 [I 2022-10-26 20:31:03] ...optimizers/trijoint.py.151: total_norm: 77.017174 [I 2022-10-26 20:31:09] ...trap/engines/engine.py.224: train: epoch 28 | batch 1250/2304 [I 2022-10-26 20:31:09] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:09:49 [I 2022-10-26 20:31:09] ...trap/engines/engine.py.229: process: 0.56283 | load: 0.00251 [I 2022-10-26 20:31:09] ...trap/engines/engine.py.230: loss: 0.19999 [I 2022-10-26 20:31:09] ...optimizers/trijoint.py.151: total_norm: 19.334057 [I 2022-10-26 20:31:14] ...trap/engines/engine.py.224: train: epoch 28 | batch 1260/2304 [I 2022-10-26 20:31:14] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:40 [I 2022-10-26 20:31:14] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00247 [I 2022-10-26 20:31:14] ...trap/engines/engine.py.230: loss: 0.28289 [I 2022-10-26 20:31:14] ...optimizers/trijoint.py.151: total_norm: 21.858892 [I 2022-10-26 20:31:20] ...trap/engines/engine.py.224: train: epoch 28 | batch 1270/2304 [I 2022-10-26 20:31:20] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:09:33 [I 2022-10-26 20:31:20] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00245 [I 2022-10-26 20:31:20] ...trap/engines/engine.py.230: loss: 0.15256 [I 2022-10-26 20:31:20] ...optimizers/trijoint.py.151: total_norm: 19.801405 [I 2022-10-26 20:31:25] ...trap/engines/engine.py.224: train: epoch 28 | batch 1280/2304 [I 2022-10-26 20:31:25] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:28 [I 2022-10-26 20:31:25] ...trap/engines/engine.py.229: process: 0.55703 | load: 0.00244 [I 2022-10-26 20:31:25] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-26 20:31:25] ...optimizers/trijoint.py.151: total_norm: 27.113405 [I 2022-10-26 20:31:31] ...trap/engines/engine.py.224: train: epoch 28 | batch 1290/2304 [I 2022-10-26 20:31:31] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:09:24 [I 2022-10-26 20:31:31] ...trap/engines/engine.py.229: process: 0.55695 | load: 0.00247 [I 2022-10-26 20:31:31] ...trap/engines/engine.py.230: loss: 0.18506 [I 2022-10-26 20:31:31] ...optimizers/trijoint.py.151: total_norm: 22.410969 [I 2022-10-26 20:31:36] ...trap/engines/engine.py.224: train: epoch 28 | batch 1300/2304 [I 2022-10-26 20:31:36] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:16 [I 2022-10-26 20:31:36] ...trap/engines/engine.py.229: process: 0.55399 | load: 0.00243 [I 2022-10-26 20:31:36] ...trap/engines/engine.py.230: loss: 0.14892 [I 2022-10-26 20:31:36] ...optimizers/trijoint.py.151: total_norm: 16.440092 [I 2022-10-26 20:31:42] ...trap/engines/engine.py.224: train: epoch 28 | batch 1310/2304 [I 2022-10-26 20:31:42] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:06 [I 2022-10-26 20:31:42] ...trap/engines/engine.py.229: process: 0.53558 | load: 0.00246 [I 2022-10-26 20:31:42] ...trap/engines/engine.py.230: loss: 0.13999 [I 2022-10-26 20:31:42] ...optimizers/trijoint.py.151: total_norm: 12.845255 [I 2022-10-26 20:31:47] ...trap/engines/engine.py.224: train: epoch 28 | batch 1320/2304 [I 2022-10-26 20:31:47] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:05 [I 2022-10-26 20:31:47] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00250 [I 2022-10-26 20:31:47] ...trap/engines/engine.py.230: loss: 0.15211 [I 2022-10-26 20:31:47] ...optimizers/trijoint.py.151: total_norm: 12.481266 [I 2022-10-26 20:31:53] ...trap/engines/engine.py.224: train: epoch 28 | batch 1330/2304 [I 2022-10-26 20:31:53] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:00 [I 2022-10-26 20:31:53] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00248 [I 2022-10-26 20:31:53] ...trap/engines/engine.py.230: loss: 0.21616 [I 2022-10-26 20:31:53] ...optimizers/trijoint.py.151: total_norm: 17.436413 [I 2022-10-26 20:31:58] ...trap/engines/engine.py.224: train: epoch 28 | batch 1340/2304 [I 2022-10-26 20:31:58] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:08:53 [I 2022-10-26 20:31:58] ...trap/engines/engine.py.229: process: 0.55542 | load: 0.00246 [I 2022-10-26 20:31:58] ...trap/engines/engine.py.230: loss: 0.13111 [I 2022-10-26 20:31:58] ...optimizers/trijoint.py.151: total_norm: 15.345168 [I 2022-10-26 20:32:04] ...trap/engines/engine.py.224: train: epoch 28 | batch 1350/2304 [I 2022-10-26 20:32:04] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:08:45 [I 2022-10-26 20:32:04] ...trap/engines/engine.py.229: process: 0.54421 | load: 0.00244 [I 2022-10-26 20:32:04] ...trap/engines/engine.py.230: loss: 0.19790 [I 2022-10-26 20:32:04] ...optimizers/trijoint.py.151: total_norm: 23.082550 [I 2022-10-26 20:32:10] ...trap/engines/engine.py.224: train: epoch 28 | batch 1360/2304 [I 2022-10-26 20:32:10] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:08:42 [I 2022-10-26 20:32:10] ...trap/engines/engine.py.229: process: 0.55150 | load: 0.00242 [I 2022-10-26 20:32:10] ...trap/engines/engine.py.230: loss: 0.17088 [I 2022-10-26 20:32:10] ...optimizers/trijoint.py.151: total_norm: 22.538425 [I 2022-10-26 20:32:15] ...trap/engines/engine.py.224: train: epoch 28 | batch 1370/2304 [I 2022-10-26 20:32:15] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:38 [I 2022-10-26 20:32:15] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00244 [I 2022-10-26 20:32:15] ...trap/engines/engine.py.230: loss: 0.22015 [I 2022-10-26 20:32:15] ...optimizers/trijoint.py.151: total_norm: 20.120621 [I 2022-10-26 20:32:21] ...trap/engines/engine.py.224: train: epoch 28 | batch 1380/2304 [I 2022-10-26 20:32:21] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:33 [I 2022-10-26 20:32:21] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00248 [I 2022-10-26 20:32:21] ...trap/engines/engine.py.230: loss: 0.18790 [I 2022-10-26 20:32:21] ...optimizers/trijoint.py.151: total_norm: 18.453653 [I 2022-10-26 20:32:26] ...trap/engines/engine.py.224: train: epoch 28 | batch 1390/2304 [I 2022-10-26 20:32:26] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:28 [I 2022-10-26 20:32:26] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00240 [I 2022-10-26 20:32:26] ...trap/engines/engine.py.230: loss: 0.20202 [I 2022-10-26 20:32:26] ...optimizers/trijoint.py.151: total_norm: 16.011513 [I 2022-10-26 20:32:32] ...trap/engines/engine.py.224: train: epoch 28 | batch 1400/2304 [I 2022-10-26 20:32:32] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:20 [I 2022-10-26 20:32:32] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00245 [I 2022-10-26 20:32:32] ...trap/engines/engine.py.230: loss: 0.16520 [I 2022-10-26 20:32:32] ...optimizers/trijoint.py.151: total_norm: 15.674753 [I 2022-10-26 20:32:37] ...trap/engines/engine.py.224: train: epoch 28 | batch 1410/2304 [I 2022-10-26 20:32:37] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:08:17 [I 2022-10-26 20:32:37] ...trap/engines/engine.py.229: process: 0.55356 | load: 0.00252 [I 2022-10-26 20:32:37] ...trap/engines/engine.py.230: loss: 0.15807 [I 2022-10-26 20:32:37] ...optimizers/trijoint.py.151: total_norm: 18.259701 [I 2022-10-26 20:32:43] ...trap/engines/engine.py.224: train: epoch 28 | batch 1420/2304 [I 2022-10-26 20:32:43] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:11 [I 2022-10-26 20:32:43] ...trap/engines/engine.py.229: process: 0.55079 | load: 0.00264 [I 2022-10-26 20:32:43] ...trap/engines/engine.py.230: loss: 0.14180 [I 2022-10-26 20:32:43] ...optimizers/trijoint.py.151: total_norm: 13.515737 [I 2022-10-26 20:32:49] ...trap/engines/engine.py.224: train: epoch 28 | batch 1430/2304 [I 2022-10-26 20:32:49] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:06 [I 2022-10-26 20:32:49] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00251 [I 2022-10-26 20:32:49] ...trap/engines/engine.py.230: loss: 0.23369 [I 2022-10-26 20:32:49] ...optimizers/trijoint.py.151: total_norm: 23.822683 [I 2022-10-26 20:32:54] ...trap/engines/engine.py.224: train: epoch 28 | batch 1440/2304 [I 2022-10-26 20:32:54] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:02 [I 2022-10-26 20:32:54] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00255 [I 2022-10-26 20:32:54] ...trap/engines/engine.py.230: loss: 0.15369 [I 2022-10-26 20:32:54] ...optimizers/trijoint.py.151: total_norm: 13.548723 [I 2022-10-26 20:33:00] ...trap/engines/engine.py.224: train: epoch 28 | batch 1450/2304 [I 2022-10-26 20:33:00] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:07:55 [I 2022-10-26 20:33:00] ...trap/engines/engine.py.229: process: 0.55229 | load: 0.00259 [I 2022-10-26 20:33:00] ...trap/engines/engine.py.230: loss: 0.13518 [I 2022-10-26 20:33:00] ...optimizers/trijoint.py.151: total_norm: 14.308746 [I 2022-10-26 20:33:05] ...trap/engines/engine.py.224: train: epoch 28 | batch 1460/2304 [I 2022-10-26 20:33:05] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:07:49 [I 2022-10-26 20:33:05] ...trap/engines/engine.py.229: process: 0.55628 | load: 0.00243 [I 2022-10-26 20:33:05] ...trap/engines/engine.py.230: loss: 0.27568 [I 2022-10-26 20:33:05] ...optimizers/trijoint.py.151: total_norm: 23.212446 [I 2022-10-26 20:33:11] ...trap/engines/engine.py.224: train: epoch 28 | batch 1470/2304 [I 2022-10-26 20:33:11] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:42 [I 2022-10-26 20:33:11] ...trap/engines/engine.py.229: process: 0.55098 | load: 0.00270 [I 2022-10-26 20:33:11] ...trap/engines/engine.py.230: loss: 0.20903 [I 2022-10-26 20:33:11] ...optimizers/trijoint.py.151: total_norm: 23.540646 [I 2022-10-26 20:33:16] ...trap/engines/engine.py.224: train: epoch 28 | batch 1480/2304 [I 2022-10-26 20:33:16] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:37 [I 2022-10-26 20:33:16] ...trap/engines/engine.py.229: process: 0.55821 | load: 0.00251 [I 2022-10-26 20:33:16] ...trap/engines/engine.py.230: loss: 0.16906 [I 2022-10-26 20:33:16] ...optimizers/trijoint.py.151: total_norm: 17.984970 [I 2022-10-26 20:33:22] ...trap/engines/engine.py.224: train: epoch 28 | batch 1490/2304 [I 2022-10-26 20:33:22] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:32 [I 2022-10-26 20:33:22] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00281 [I 2022-10-26 20:33:22] ...trap/engines/engine.py.230: loss: 0.19555 [I 2022-10-26 20:33:22] ...optimizers/trijoint.py.151: total_norm: 28.880262 [I 2022-10-26 20:33:27] ...trap/engines/engine.py.224: train: epoch 28 | batch 1500/2304 [I 2022-10-26 20:33:27] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:24 [I 2022-10-26 20:33:27] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00246 [I 2022-10-26 20:33:27] ...trap/engines/engine.py.230: loss: 0.38322 [I 2022-10-26 20:33:27] ...optimizers/trijoint.py.151: total_norm: 22.555490 [I 2022-10-26 20:33:33] ...trap/engines/engine.py.224: train: epoch 28 | batch 1510/2304 [I 2022-10-26 20:33:33] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:15 [I 2022-10-26 20:33:33] ...trap/engines/engine.py.229: process: 0.54338 | load: 0.00243 [I 2022-10-26 20:33:33] ...trap/engines/engine.py.230: loss: 0.16703 [I 2022-10-26 20:33:33] ...optimizers/trijoint.py.151: total_norm: 16.802187 [I 2022-10-26 20:33:39] ...trap/engines/engine.py.224: train: epoch 28 | batch 1520/2304 [I 2022-10-26 20:33:39] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:14 [I 2022-10-26 20:33:39] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00246 [I 2022-10-26 20:33:39] ...trap/engines/engine.py.230: loss: 0.14546 [I 2022-10-26 20:33:39] ...optimizers/trijoint.py.151: total_norm: 12.395679 [I 2022-10-26 20:33:44] ...trap/engines/engine.py.224: train: epoch 28 | batch 1530/2304 [I 2022-10-26 20:33:44] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:06 [I 2022-10-26 20:33:44] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00248 [I 2022-10-26 20:33:44] ...trap/engines/engine.py.230: loss: 0.19881 [I 2022-10-26 20:33:44] ...optimizers/trijoint.py.151: total_norm: 23.044842 [I 2022-10-26 20:33:50] ...trap/engines/engine.py.224: train: epoch 28 | batch 1540/2304 [I 2022-10-26 20:33:50] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:00 [I 2022-10-26 20:33:50] ...trap/engines/engine.py.229: process: 0.54084 | load: 0.00244 [I 2022-10-26 20:33:50] ...trap/engines/engine.py.230: loss: 0.22978 [I 2022-10-26 20:33:50] ...optimizers/trijoint.py.151: total_norm: 32.590347 [I 2022-10-26 20:33:55] ...trap/engines/engine.py.224: train: epoch 28 | batch 1550/2304 [I 2022-10-26 20:33:55] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:06:57 [I 2022-10-26 20:33:55] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00242 [I 2022-10-26 20:33:55] ...trap/engines/engine.py.230: loss: 0.16564 [I 2022-10-26 20:33:55] ...optimizers/trijoint.py.151: total_norm: 51.343388 [I 2022-10-26 20:34:01] ...trap/engines/engine.py.224: train: epoch 28 | batch 1560/2304 [I 2022-10-26 20:34:01] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:06:51 [I 2022-10-26 20:34:01] ...trap/engines/engine.py.229: process: 0.54629 | load: 0.00240 [I 2022-10-26 20:34:01] ...trap/engines/engine.py.230: loss: 0.14967 [I 2022-10-26 20:34:01] ...optimizers/trijoint.py.151: total_norm: 12.184773 [I 2022-10-26 20:34:06] ...trap/engines/engine.py.224: train: epoch 28 | batch 1570/2304 [I 2022-10-26 20:34:06] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:47 [I 2022-10-26 20:34:06] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00272 [I 2022-10-26 20:34:06] ...trap/engines/engine.py.230: loss: 0.18886 [I 2022-10-26 20:34:06] ...optimizers/trijoint.py.151: total_norm: 21.758617 [I 2022-10-26 20:34:12] ...trap/engines/engine.py.224: train: epoch 28 | batch 1580/2304 [I 2022-10-26 20:34:12] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:38 [I 2022-10-26 20:34:12] ...trap/engines/engine.py.229: process: 0.54572 | load: 0.00255 [I 2022-10-26 20:34:12] ...trap/engines/engine.py.230: loss: 0.20774 [I 2022-10-26 20:34:12] ...optimizers/trijoint.py.151: total_norm: 20.415621 [I 2022-10-26 20:34:17] ...trap/engines/engine.py.224: train: epoch 28 | batch 1590/2304 [I 2022-10-26 20:34:17] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:36 [I 2022-10-26 20:34:17] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00254 [I 2022-10-26 20:34:17] ...trap/engines/engine.py.230: loss: 0.14095 [I 2022-10-26 20:34:17] ...optimizers/trijoint.py.151: total_norm: 14.638960 [I 2022-10-26 20:34:23] ...trap/engines/engine.py.224: train: epoch 28 | batch 1600/2304 [I 2022-10-26 20:34:23] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:30 [I 2022-10-26 20:34:23] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00250 [I 2022-10-26 20:34:23] ...trap/engines/engine.py.230: loss: 0.15528 [I 2022-10-26 20:34:23] ...optimizers/trijoint.py.151: total_norm: 25.888664 [I 2022-10-26 20:34:28] ...trap/engines/engine.py.224: train: epoch 28 | batch 1610/2304 [I 2022-10-26 20:34:28] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:25 [I 2022-10-26 20:34:28] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00246 [I 2022-10-26 20:34:28] ...trap/engines/engine.py.230: loss: 0.27755 [I 2022-10-26 20:34:28] ...optimizers/trijoint.py.151: total_norm: 32.455898 [I 2022-10-26 20:34:34] ...trap/engines/engine.py.224: train: epoch 28 | batch 1620/2304 [I 2022-10-26 20:34:34] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:19 [I 2022-10-26 20:34:34] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00244 [I 2022-10-26 20:34:34] ...trap/engines/engine.py.230: loss: 0.17803 [I 2022-10-26 20:34:34] ...optimizers/trijoint.py.151: total_norm: 16.073891 [I 2022-10-26 20:34:40] ...trap/engines/engine.py.224: train: epoch 28 | batch 1630/2304 [I 2022-10-26 20:34:40] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:15 [I 2022-10-26 20:34:40] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00240 [I 2022-10-26 20:34:40] ...trap/engines/engine.py.230: loss: 0.15228 [I 2022-10-26 20:34:40] ...optimizers/trijoint.py.151: total_norm: 17.889946 [I 2022-10-26 20:34:45] ...trap/engines/engine.py.224: train: epoch 28 | batch 1640/2304 [I 2022-10-26 20:34:45] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:10 [I 2022-10-26 20:34:45] ...trap/engines/engine.py.229: process: 0.56357 | load: 0.00245 [I 2022-10-26 20:34:45] ...trap/engines/engine.py.230: loss: 0.14874 [I 2022-10-26 20:34:45] ...optimizers/trijoint.py.151: total_norm: 15.274994 [I 2022-10-26 20:34:51] ...trap/engines/engine.py.224: train: epoch 28 | batch 1650/2304 [I 2022-10-26 20:34:51] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:05 [I 2022-10-26 20:34:51] ...trap/engines/engine.py.229: process: 0.55787 | load: 0.00282 [I 2022-10-26 20:34:51] ...trap/engines/engine.py.230: loss: 0.12097 [I 2022-10-26 20:34:51] ...optimizers/trijoint.py.151: total_norm: 15.243402 [I 2022-10-26 20:34:56] ...trap/engines/engine.py.224: train: epoch 28 | batch 1660/2304 [I 2022-10-26 20:34:56] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:05:57 [I 2022-10-26 20:34:56] ...trap/engines/engine.py.229: process: 0.55040 | load: 0.00246 [I 2022-10-26 20:34:56] ...trap/engines/engine.py.230: loss: 0.21088 [I 2022-10-26 20:34:56] ...optimizers/trijoint.py.151: total_norm: 15.280867 [I 2022-10-26 20:35:02] ...trap/engines/engine.py.224: train: epoch 28 | batch 1670/2304 [I 2022-10-26 20:35:02] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:49 [I 2022-10-26 20:35:02] ...trap/engines/engine.py.229: process: 0.54898 | load: 0.00246 [I 2022-10-26 20:35:02] ...trap/engines/engine.py.230: loss: 0.19732 [I 2022-10-26 20:35:02] ...optimizers/trijoint.py.151: total_norm: 23.135595 [I 2022-10-26 20:35:07] ...trap/engines/engine.py.224: train: epoch 28 | batch 1680/2304 [I 2022-10-26 20:35:07] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:05:45 [I 2022-10-26 20:35:07] ...trap/engines/engine.py.229: process: 0.55521 | load: 0.00247 [I 2022-10-26 20:35:07] ...trap/engines/engine.py.230: loss: 0.14634 [I 2022-10-26 20:35:07] ...optimizers/trijoint.py.151: total_norm: 8.662115 [I 2022-10-26 20:35:13] ...trap/engines/engine.py.224: train: epoch 28 | batch 1690/2304 [I 2022-10-26 20:35:13] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:40 [I 2022-10-26 20:35:13] ...trap/engines/engine.py.229: process: 0.57106 | load: 0.00248 [I 2022-10-26 20:35:13] ...trap/engines/engine.py.230: loss: 0.13436 [I 2022-10-26 20:35:13] ...optimizers/trijoint.py.151: total_norm: 14.593719 [I 2022-10-26 20:35:19] ...trap/engines/engine.py.224: train: epoch 28 | batch 1700/2304 [I 2022-10-26 20:35:19] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:35 [I 2022-10-26 20:35:19] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00242 [I 2022-10-26 20:35:19] ...trap/engines/engine.py.230: loss: 0.18670 [I 2022-10-26 20:35:19] ...optimizers/trijoint.py.151: total_norm: 16.371134 [I 2022-10-26 20:35:24] ...trap/engines/engine.py.224: train: epoch 28 | batch 1710/2304 [I 2022-10-26 20:35:24] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:30 [I 2022-10-26 20:35:24] ...trap/engines/engine.py.229: process: 0.57286 | load: 0.00249 [I 2022-10-26 20:35:24] ...trap/engines/engine.py.230: loss: 0.17446 [I 2022-10-26 20:35:24] ...optimizers/trijoint.py.151: total_norm: 16.657087 [I 2022-10-26 20:35:30] ...trap/engines/engine.py.224: train: epoch 28 | batch 1720/2304 [I 2022-10-26 20:35:30] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:27 [I 2022-10-26 20:35:30] ...trap/engines/engine.py.229: process: 0.58352 | load: 0.00249 [I 2022-10-26 20:35:30] ...trap/engines/engine.py.230: loss: 0.11785 [I 2022-10-26 20:35:30] ...optimizers/trijoint.py.151: total_norm: 13.801964 [I 2022-10-26 20:35:35] ...trap/engines/engine.py.224: train: epoch 28 | batch 1730/2304 [I 2022-10-26 20:35:35] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:20 [I 2022-10-26 20:35:35] ...trap/engines/engine.py.229: process: 0.55006 | load: 0.00245 [I 2022-10-26 20:35:35] ...trap/engines/engine.py.230: loss: 0.17846 [I 2022-10-26 20:35:35] ...optimizers/trijoint.py.151: total_norm: 11.469177 [I 2022-10-26 20:35:41] ...trap/engines/engine.py.224: train: epoch 28 | batch 1740/2304 [I 2022-10-26 20:35:41] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:17 [I 2022-10-26 20:35:41] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00246 [I 2022-10-26 20:35:41] ...trap/engines/engine.py.230: loss: 0.21260 [I 2022-10-26 20:35:41] ...optimizers/trijoint.py.151: total_norm: 18.484568 [I 2022-10-26 20:35:47] ...trap/engines/engine.py.224: train: epoch 28 | batch 1750/2304 [I 2022-10-26 20:35:47] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:08 [I 2022-10-26 20:35:47] ...trap/engines/engine.py.229: process: 0.56173 | load: 0.00241 [I 2022-10-26 20:35:47] ...trap/engines/engine.py.230: loss: 0.19927 [I 2022-10-26 20:35:47] ...optimizers/trijoint.py.151: total_norm: 17.606165 [I 2022-10-26 20:35:52] ...trap/engines/engine.py.224: train: epoch 28 | batch 1760/2304 [I 2022-10-26 20:35:52] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:01 [I 2022-10-26 20:35:52] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00247 [I 2022-10-26 20:35:52] ...trap/engines/engine.py.230: loss: 0.26192 [I 2022-10-26 20:35:52] ...optimizers/trijoint.py.151: total_norm: 31.231649 [I 2022-10-26 20:35:58] ...trap/engines/engine.py.224: train: epoch 28 | batch 1770/2304 [I 2022-10-26 20:35:58] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:04:57 [I 2022-10-26 20:35:58] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00238 [I 2022-10-26 20:35:58] ...trap/engines/engine.py.230: loss: 0.13151 [I 2022-10-26 20:35:58] ...optimizers/trijoint.py.151: total_norm: 13.492401 [I 2022-10-26 20:36:03] ...trap/engines/engine.py.224: train: epoch 28 | batch 1780/2304 [I 2022-10-26 20:36:03] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:04:51 [I 2022-10-26 20:36:03] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00242 [I 2022-10-26 20:36:03] ...trap/engines/engine.py.230: loss: 0.16530 [I 2022-10-26 20:36:03] ...optimizers/trijoint.py.151: total_norm: 12.474308 [I 2022-10-26 20:36:09] ...trap/engines/engine.py.224: train: epoch 28 | batch 1790/2304 [I 2022-10-26 20:36:09] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:47 [I 2022-10-26 20:36:09] ...trap/engines/engine.py.229: process: 0.56610 | load: 0.00239 [I 2022-10-26 20:36:09] ...trap/engines/engine.py.230: loss: 0.28580 [I 2022-10-26 20:36:09] ...optimizers/trijoint.py.151: total_norm: 17.884413 [I 2022-10-26 20:36:14] ...trap/engines/engine.py.224: train: epoch 28 | batch 1800/2304 [I 2022-10-26 20:36:14] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:40 [I 2022-10-26 20:36:14] ...trap/engines/engine.py.229: process: 0.55234 | load: 0.00239 [I 2022-10-26 20:36:14] ...trap/engines/engine.py.230: loss: 0.20787 [I 2022-10-26 20:36:14] ...optimizers/trijoint.py.151: total_norm: 23.618286 [I 2022-10-26 20:36:20] ...trap/engines/engine.py.224: train: epoch 28 | batch 1810/2304 [I 2022-10-26 20:36:20] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:32 [I 2022-10-26 20:36:20] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00252 [I 2022-10-26 20:36:20] ...trap/engines/engine.py.230: loss: 0.19961 [I 2022-10-26 20:36:20] ...optimizers/trijoint.py.151: total_norm: 13.340566 [I 2022-10-26 20:36:26] ...trap/engines/engine.py.224: train: epoch 28 | batch 1820/2304 [I 2022-10-26 20:36:26] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:30 [I 2022-10-26 20:36:26] ...trap/engines/engine.py.229: process: 0.54380 | load: 0.00246 [I 2022-10-26 20:36:26] ...trap/engines/engine.py.230: loss: 0.15891 [I 2022-10-26 20:36:26] ...optimizers/trijoint.py.151: total_norm: 15.038077 [I 2022-10-26 20:36:31] ...trap/engines/engine.py.224: train: epoch 28 | batch 1830/2304 [I 2022-10-26 20:36:31] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:24 [I 2022-10-26 20:36:31] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00245 [I 2022-10-26 20:36:31] ...trap/engines/engine.py.230: loss: 0.18020 [I 2022-10-26 20:36:31] ...optimizers/trijoint.py.151: total_norm: 32.455742 [I 2022-10-26 20:36:37] ...trap/engines/engine.py.224: train: epoch 28 | batch 1840/2304 [I 2022-10-26 20:36:37] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:15 [I 2022-10-26 20:36:37] ...trap/engines/engine.py.229: process: 0.53970 | load: 0.00244 [I 2022-10-26 20:36:37] ...trap/engines/engine.py.230: loss: 0.26555 [I 2022-10-26 20:36:37] ...optimizers/trijoint.py.151: total_norm: 27.454355 [I 2022-10-26 20:36:42] ...trap/engines/engine.py.224: train: epoch 28 | batch 1850/2304 [I 2022-10-26 20:36:42] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:12 [I 2022-10-26 20:36:42] ...trap/engines/engine.py.229: process: 0.54726 | load: 0.00245 [I 2022-10-26 20:36:42] ...trap/engines/engine.py.230: loss: 0.24095 [I 2022-10-26 20:36:42] ...optimizers/trijoint.py.151: total_norm: 23.203226 [I 2022-10-26 20:36:48] ...trap/engines/engine.py.224: train: epoch 28 | batch 1860/2304 [I 2022-10-26 20:36:48] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:04 [I 2022-10-26 20:36:48] ...trap/engines/engine.py.229: process: 0.55006 | load: 0.00243 [I 2022-10-26 20:36:48] ...trap/engines/engine.py.230: loss: 0.21417 [I 2022-10-26 20:36:48] ...optimizers/trijoint.py.151: total_norm: 24.576441 [I 2022-10-26 20:36:53] ...trap/engines/engine.py.224: train: epoch 28 | batch 1870/2304 [I 2022-10-26 20:36:53] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:01 [I 2022-10-26 20:36:53] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00251 [I 2022-10-26 20:36:53] ...trap/engines/engine.py.230: loss: 0.13183 [I 2022-10-26 20:36:53] ...optimizers/trijoint.py.151: total_norm: 26.957756 [I 2022-10-26 20:36:59] ...trap/engines/engine.py.224: train: epoch 28 | batch 1880/2304 [I 2022-10-26 20:36:59] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:03:59 [I 2022-10-26 20:36:59] ...trap/engines/engine.py.229: process: 0.56596 | load: 0.00246 [I 2022-10-26 20:36:59] ...trap/engines/engine.py.230: loss: 0.23508 [I 2022-10-26 20:36:59] ...optimizers/trijoint.py.151: total_norm: 21.342054 [I 2022-10-26 20:37:04] ...trap/engines/engine.py.224: train: epoch 28 | batch 1890/2304 [I 2022-10-26 20:37:04] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:49 [I 2022-10-26 20:37:04] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00250 [I 2022-10-26 20:37:04] ...trap/engines/engine.py.230: loss: 0.22845 [I 2022-10-26 20:37:04] ...optimizers/trijoint.py.151: total_norm: 17.873152 [I 2022-10-26 20:37:10] ...trap/engines/engine.py.224: train: epoch 28 | batch 1900/2304 [I 2022-10-26 20:37:10] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:03:44 [I 2022-10-26 20:37:10] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00251 [I 2022-10-26 20:37:10] ...trap/engines/engine.py.230: loss: 0.12186 [I 2022-10-26 20:37:10] ...optimizers/trijoint.py.151: total_norm: 15.767417 [I 2022-10-26 20:37:16] ...trap/engines/engine.py.224: train: epoch 28 | batch 1910/2304 [I 2022-10-26 20:37:16] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:38 [I 2022-10-26 20:37:16] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00250 [I 2022-10-26 20:37:16] ...trap/engines/engine.py.230: loss: 0.18365 [I 2022-10-26 20:37:16] ...optimizers/trijoint.py.151: total_norm: 23.357079 [I 2022-10-26 20:37:21] ...trap/engines/engine.py.224: train: epoch 28 | batch 1920/2304 [I 2022-10-26 20:37:21] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:31 [I 2022-10-26 20:37:21] ...trap/engines/engine.py.229: process: 0.54656 | load: 0.00248 [I 2022-10-26 20:37:21] ...trap/engines/engine.py.230: loss: 0.12200 [I 2022-10-26 20:37:21] ...optimizers/trijoint.py.151: total_norm: 9.446080 [I 2022-10-26 20:37:27] ...trap/engines/engine.py.224: train: epoch 28 | batch 1930/2304 [I 2022-10-26 20:37:27] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:27 [I 2022-10-26 20:37:27] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00243 [I 2022-10-26 20:37:27] ...trap/engines/engine.py.230: loss: 0.18652 [I 2022-10-26 20:37:27] ...optimizers/trijoint.py.151: total_norm: 21.529617 [I 2022-10-26 20:37:32] ...trap/engines/engine.py.224: train: epoch 28 | batch 1940/2304 [I 2022-10-26 20:37:32] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:21 [I 2022-10-26 20:37:32] ...trap/engines/engine.py.229: process: 0.54887 | load: 0.00247 [I 2022-10-26 20:37:32] ...trap/engines/engine.py.230: loss: 0.24277 [I 2022-10-26 20:37:32] ...optimizers/trijoint.py.151: total_norm: 19.516157 [I 2022-10-26 20:37:38] ...trap/engines/engine.py.224: train: epoch 28 | batch 1950/2304 [I 2022-10-26 20:37:38] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:15 [I 2022-10-26 20:37:38] ...trap/engines/engine.py.229: process: 0.54705 | load: 0.00244 [I 2022-10-26 20:37:38] ...trap/engines/engine.py.230: loss: 0.18511 [I 2022-10-26 20:37:38] ...optimizers/trijoint.py.151: total_norm: 12.000555 [I 2022-10-26 20:37:43] ...trap/engines/engine.py.224: train: epoch 28 | batch 1960/2304 [I 2022-10-26 20:37:43] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:09 [I 2022-10-26 20:37:43] ...trap/engines/engine.py.229: process: 0.56181 | load: 0.00244 [I 2022-10-26 20:37:43] ...trap/engines/engine.py.230: loss: 0.15320 [I 2022-10-26 20:37:43] ...optimizers/trijoint.py.151: total_norm: 21.593241 [I 2022-10-26 20:37:49] ...trap/engines/engine.py.224: train: epoch 28 | batch 1970/2304 [I 2022-10-26 20:37:49] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:05 [I 2022-10-26 20:37:49] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00243 [I 2022-10-26 20:37:49] ...trap/engines/engine.py.230: loss: 0.17652 [I 2022-10-26 20:37:49] ...optimizers/trijoint.py.151: total_norm: 19.210020 [I 2022-10-26 20:37:54] ...trap/engines/engine.py.224: train: epoch 28 | batch 1980/2304 [I 2022-10-26 20:37:54] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:02:59 [I 2022-10-26 20:37:54] ...trap/engines/engine.py.229: process: 0.54990 | load: 0.00246 [I 2022-10-26 20:37:54] ...trap/engines/engine.py.230: loss: 0.19173 [I 2022-10-26 20:37:54] ...optimizers/trijoint.py.151: total_norm: 29.304350 [I 2022-10-26 20:38:00] ...trap/engines/engine.py.224: train: epoch 28 | batch 1990/2304 [I 2022-10-26 20:38:00] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:02:54 [I 2022-10-26 20:38:00] ...trap/engines/engine.py.229: process: 0.56062 | load: 0.00242 [I 2022-10-26 20:38:00] ...trap/engines/engine.py.230: loss: 0.18744 [I 2022-10-26 20:38:00] ...optimizers/trijoint.py.151: total_norm: 14.275672 [I 2022-10-26 20:38:06] ...trap/engines/engine.py.224: train: epoch 28 | batch 2000/2304 [I 2022-10-26 20:38:06] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:49 [I 2022-10-26 20:38:06] ...trap/engines/engine.py.229: process: 0.54974 | load: 0.00241 [I 2022-10-26 20:38:06] ...trap/engines/engine.py.230: loss: 0.18762 [I 2022-10-26 20:38:06] ...optimizers/trijoint.py.151: total_norm: 18.443731 [I 2022-10-26 20:38:11] ...trap/engines/engine.py.224: train: epoch 28 | batch 2010/2304 [I 2022-10-26 20:38:11] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:02:43 [I 2022-10-26 20:38:11] ...trap/engines/engine.py.229: process: 0.56137 | load: 0.00262 [I 2022-10-26 20:38:11] ...trap/engines/engine.py.230: loss: 0.16075 [I 2022-10-26 20:38:11] ...optimizers/trijoint.py.151: total_norm: 15.987351 [I 2022-10-26 20:38:17] ...trap/engines/engine.py.224: train: epoch 28 | batch 2020/2304 [I 2022-10-26 20:38:17] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:36 [I 2022-10-26 20:38:17] ...trap/engines/engine.py.229: process: 0.54382 | load: 0.00241 [I 2022-10-26 20:38:17] ...trap/engines/engine.py.230: loss: 0.14295 [I 2022-10-26 20:38:17] ...optimizers/trijoint.py.151: total_norm: 18.936237 [I 2022-10-26 20:38:22] ...trap/engines/engine.py.224: train: epoch 28 | batch 2030/2304 [I 2022-10-26 20:38:22] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:31 [I 2022-10-26 20:38:22] ...trap/engines/engine.py.229: process: 0.55931 | load: 0.00242 [I 2022-10-26 20:38:22] ...trap/engines/engine.py.230: loss: 0.22191 [I 2022-10-26 20:38:22] ...optimizers/trijoint.py.151: total_norm: 15.711718 [I 2022-10-26 20:38:28] ...trap/engines/engine.py.224: train: epoch 28 | batch 2040/2304 [I 2022-10-26 20:38:28] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:25 [I 2022-10-26 20:38:28] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00242 [I 2022-10-26 20:38:28] ...trap/engines/engine.py.230: loss: 0.24995 [I 2022-10-26 20:38:28] ...optimizers/trijoint.py.151: total_norm: 25.388968 [I 2022-10-26 20:38:33] ...trap/engines/engine.py.224: train: epoch 28 | batch 2050/2304 [I 2022-10-26 20:38:33] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:20 [I 2022-10-26 20:38:33] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00240 [I 2022-10-26 20:38:33] ...trap/engines/engine.py.230: loss: 0.14362 [I 2022-10-26 20:38:33] ...optimizers/trijoint.py.151: total_norm: 22.804737 [I 2022-10-26 20:38:39] ...trap/engines/engine.py.224: train: epoch 28 | batch 2060/2304 [I 2022-10-26 20:38:39] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:16 [I 2022-10-26 20:38:39] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00240 [I 2022-10-26 20:38:39] ...trap/engines/engine.py.230: loss: 0.20372 [I 2022-10-26 20:38:39] ...optimizers/trijoint.py.151: total_norm: 22.030352 [I 2022-10-26 20:38:44] ...trap/engines/engine.py.224: train: epoch 28 | batch 2070/2304 [I 2022-10-26 20:38:44] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:10 [I 2022-10-26 20:38:44] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00241 [I 2022-10-26 20:38:44] ...trap/engines/engine.py.230: loss: 0.20769 [I 2022-10-26 20:38:44] ...optimizers/trijoint.py.151: total_norm: 22.594278 [I 2022-10-26 20:38:50] ...trap/engines/engine.py.224: train: epoch 28 | batch 2080/2304 [I 2022-10-26 20:38:50] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:04 [I 2022-10-26 20:38:50] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00254 [I 2022-10-26 20:38:50] ...trap/engines/engine.py.230: loss: 0.22469 [I 2022-10-26 20:38:50] ...optimizers/trijoint.py.151: total_norm: 16.019970 [I 2022-10-26 20:38:56] ...trap/engines/engine.py.224: train: epoch 28 | batch 2090/2304 [I 2022-10-26 20:38:56] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:01:58 [I 2022-10-26 20:38:56] ...trap/engines/engine.py.229: process: 0.54682 | load: 0.00245 [I 2022-10-26 20:38:56] ...trap/engines/engine.py.230: loss: 0.15188 [I 2022-10-26 20:38:56] ...optimizers/trijoint.py.151: total_norm: 18.472145 [I 2022-10-26 20:39:01] ...trap/engines/engine.py.224: train: epoch 28 | batch 2100/2304 [I 2022-10-26 20:39:01] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:01:52 [I 2022-10-26 20:39:01] ...trap/engines/engine.py.229: process: 0.53837 | load: 0.00246 [I 2022-10-26 20:39:01] ...trap/engines/engine.py.230: loss: 0.16396 [I 2022-10-26 20:39:01] ...optimizers/trijoint.py.151: total_norm: 14.891210 [I 2022-10-26 20:39:07] ...trap/engines/engine.py.224: train: epoch 28 | batch 2110/2304 [I 2022-10-26 20:39:07] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:01:47 [I 2022-10-26 20:39:07] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00243 [I 2022-10-26 20:39:07] ...trap/engines/engine.py.230: loss: 0.20140 [I 2022-10-26 20:39:07] ...optimizers/trijoint.py.151: total_norm: 17.519323 [I 2022-10-26 20:39:12] ...trap/engines/engine.py.224: train: epoch 28 | batch 2120/2304 [I 2022-10-26 20:39:12] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:01:43 [I 2022-10-26 20:39:12] ...trap/engines/engine.py.229: process: 0.55095 | load: 0.00244 [I 2022-10-26 20:39:12] ...trap/engines/engine.py.230: loss: 0.16383 [I 2022-10-26 20:39:12] ...optimizers/trijoint.py.151: total_norm: 21.260742 [I 2022-10-26 20:39:18] ...trap/engines/engine.py.224: train: epoch 28 | batch 2130/2304 [I 2022-10-26 20:39:18] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:01:35 [I 2022-10-26 20:39:18] ...trap/engines/engine.py.229: process: 0.54486 | load: 0.00243 [I 2022-10-26 20:39:18] ...trap/engines/engine.py.230: loss: 0.18906 [I 2022-10-26 20:39:18] ...optimizers/trijoint.py.151: total_norm: 10.851676 [I 2022-10-26 20:39:23] ...trap/engines/engine.py.224: train: epoch 28 | batch 2140/2304 [I 2022-10-26 20:39:23] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:29 [I 2022-10-26 20:39:23] ...trap/engines/engine.py.229: process: 0.53802 | load: 0.00259 [I 2022-10-26 20:39:23] ...trap/engines/engine.py.230: loss: 0.17975 [I 2022-10-26 20:39:23] ...optimizers/trijoint.py.151: total_norm: 17.362198 [I 2022-10-26 20:39:29] ...trap/engines/engine.py.224: train: epoch 28 | batch 2150/2304 [I 2022-10-26 20:39:29] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:25 [I 2022-10-26 20:39:29] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00246 [I 2022-10-26 20:39:29] ...trap/engines/engine.py.230: loss: 0.15991 [I 2022-10-26 20:39:29] ...optimizers/trijoint.py.151: total_norm: 12.762603 [I 2022-10-26 20:39:35] ...trap/engines/engine.py.224: train: epoch 28 | batch 2160/2304 [I 2022-10-26 20:39:35] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:19 [I 2022-10-26 20:39:35] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00242 [I 2022-10-26 20:39:35] ...trap/engines/engine.py.230: loss: 0.16122 [I 2022-10-26 20:39:35] ...optimizers/trijoint.py.151: total_norm: 12.866489 [I 2022-10-26 20:39:40] ...trap/engines/engine.py.224: train: epoch 28 | batch 2170/2304 [I 2022-10-26 20:39:40] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:14 [I 2022-10-26 20:39:40] ...trap/engines/engine.py.229: process: 0.54124 | load: 0.00246 [I 2022-10-26 20:39:40] ...trap/engines/engine.py.230: loss: 0.13923 [I 2022-10-26 20:39:40] ...optimizers/trijoint.py.151: total_norm: 11.186868 [I 2022-10-26 20:39:46] ...trap/engines/engine.py.224: train: epoch 28 | batch 2180/2304 [I 2022-10-26 20:39:46] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:08 [I 2022-10-26 20:39:46] ...trap/engines/engine.py.229: process: 0.54769 | load: 0.00248 [I 2022-10-26 20:39:46] ...trap/engines/engine.py.230: loss: 0.15997 [I 2022-10-26 20:39:46] ...optimizers/trijoint.py.151: total_norm: 12.450141 [I 2022-10-26 20:39:51] ...trap/engines/engine.py.224: train: epoch 28 | batch 2190/2304 [I 2022-10-26 20:39:51] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:03 [I 2022-10-26 20:39:51] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00243 [I 2022-10-26 20:39:51] ...trap/engines/engine.py.230: loss: 0.11449 [I 2022-10-26 20:39:51] ...optimizers/trijoint.py.151: total_norm: 38.167374 [I 2022-10-26 20:39:57] ...trap/engines/engine.py.224: train: epoch 28 | batch 2200/2304 [I 2022-10-26 20:39:57] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:00:58 [I 2022-10-26 20:39:57] ...trap/engines/engine.py.229: process: 0.56375 | load: 0.00242 [I 2022-10-26 20:39:57] ...trap/engines/engine.py.230: loss: 0.18860 [I 2022-10-26 20:39:57] ...optimizers/trijoint.py.151: total_norm: 22.349730 [I 2022-10-26 20:40:02] ...trap/engines/engine.py.224: train: epoch 28 | batch 2210/2304 [I 2022-10-26 20:40:02] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:00:52 [I 2022-10-26 20:40:02] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00245 [I 2022-10-26 20:40:02] ...trap/engines/engine.py.230: loss: 0.21808 [I 2022-10-26 20:40:02] ...optimizers/trijoint.py.151: total_norm: 37.962807 [I 2022-10-26 20:40:08] ...trap/engines/engine.py.224: train: epoch 28 | batch 2220/2304 [I 2022-10-26 20:40:08] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:00:46 [I 2022-10-26 20:40:08] ...trap/engines/engine.py.229: process: 0.53783 | load: 0.00243 [I 2022-10-26 20:40:08] ...trap/engines/engine.py.230: loss: 0.19670 [I 2022-10-26 20:40:08] ...optimizers/trijoint.py.151: total_norm: 28.829733 [I 2022-10-26 20:40:14] ...trap/engines/engine.py.224: train: epoch 28 | batch 2230/2304 [I 2022-10-26 20:40:14] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:00:42 [I 2022-10-26 20:40:14] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00240 [I 2022-10-26 20:40:14] ...trap/engines/engine.py.230: loss: 0.14909 [I 2022-10-26 20:40:14] ...optimizers/trijoint.py.151: total_norm: 14.110770 [I 2022-10-26 20:40:19] ...trap/engines/engine.py.224: train: epoch 28 | batch 2240/2304 [I 2022-10-26 20:40:19] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:35 [I 2022-10-26 20:40:19] ...trap/engines/engine.py.229: process: 0.54225 | load: 0.00242 [I 2022-10-26 20:40:19] ...trap/engines/engine.py.230: loss: 0.17806 [I 2022-10-26 20:40:19] ...optimizers/trijoint.py.151: total_norm: 18.809965 [I 2022-10-26 20:40:25] ...trap/engines/engine.py.224: train: epoch 28 | batch 2250/2304 [I 2022-10-26 20:40:25] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:29 [I 2022-10-26 20:40:25] ...trap/engines/engine.py.229: process: 0.55936 | load: 0.00245 [I 2022-10-26 20:40:25] ...trap/engines/engine.py.230: loss: 0.21425 [I 2022-10-26 20:40:25] ...optimizers/trijoint.py.151: total_norm: 37.659702 [I 2022-10-26 20:40:30] ...trap/engines/engine.py.224: train: epoch 28 | batch 2260/2304 [I 2022-10-26 20:40:30] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:24 [I 2022-10-26 20:40:30] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00246 [I 2022-10-26 20:40:30] ...trap/engines/engine.py.230: loss: 0.16141 [I 2022-10-26 20:40:30] ...optimizers/trijoint.py.151: total_norm: 22.578243 [I 2022-10-26 20:40:36] ...trap/engines/engine.py.224: train: epoch 28 | batch 2270/2304 [I 2022-10-26 20:40:36] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:18 [I 2022-10-26 20:40:36] ...trap/engines/engine.py.229: process: 0.56185 | load: 0.00248 [I 2022-10-26 20:40:36] ...trap/engines/engine.py.230: loss: 0.13947 [I 2022-10-26 20:40:36] ...optimizers/trijoint.py.151: total_norm: 24.952642 [I 2022-10-26 20:40:41] ...trap/engines/engine.py.224: train: epoch 28 | batch 2280/2304 [I 2022-10-26 20:40:41] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:13 [I 2022-10-26 20:40:41] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00244 [I 2022-10-26 20:40:41] ...trap/engines/engine.py.230: loss: 0.21325 [I 2022-10-26 20:40:41] ...optimizers/trijoint.py.151: total_norm: 19.025158 [I 2022-10-26 20:40:47] ...trap/engines/engine.py.224: train: epoch 28 | batch 2290/2304 [I 2022-10-26 20:40:47] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:07 [I 2022-10-26 20:40:47] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00241 [I 2022-10-26 20:40:47] ...trap/engines/engine.py.230: loss: 0.17688 [I 2022-10-26 20:40:47] ...optimizers/trijoint.py.151: total_norm: 23.356527 [I 2022-10-26 20:40:52] ...trap/engines/engine.py.224: train: epoch 28 | batch 2300/2304 [I 2022-10-26 20:40:52] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:02 [I 2022-10-26 20:40:52] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00015 [I 2022-10-26 20:40:52] ...trap/engines/engine.py.230: loss: 0.14137 [I 2022-10-26 20:40:52] ...optimizers/trijoint.py.151: total_norm: 14.583924 [I 2022-10-26 20:40:55] ...trap/engines/engine.py.224: train: epoch 28 | batch 2304/2304 [I 2022-10-26 20:40:55] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:00 [I 2022-10-26 20:40:55] ...trap/engines/engine.py.229: process: 0.54759 | load: 0.00016 [I 2022-10-26 20:40:55] ...trap/engines/engine.py.230: loss: 0.18281 [I 2022-10-26 20:40:55] ...optimizers/trijoint.py.151: total_norm: 21.774693 [S 2022-10-26 20:40:56] ...trap/engines/engine.py.129: train_epoch.epoch: 28 [S 2022-10-26 20:40:56] ...trap/engines/engine.py.129: train_epoch.loss: 0.18392878828040948 [I 2022-10-26 20:40:56] ...s/criterions/triplet.py.84: margin = 0.19500000000000012 [I 2022-10-26 20:40:56] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 28 [I 2022-10-26 20:40:56] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 20:40:58] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 20:40:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 20:40:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 20:40:59] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 20:40:59] ...trap/engines/engine.py.321: eval: epoch 28 | batch 0/510 [I 2022-10-26 20:40:59] ...trap/engines/engine.py.325: elapsed: 0:00:02 | left: 0:25:24 [I 2022-10-26 20:40:59] ...trap/engines/engine.py.326: process: 2.98990 | load: 2.60207 [I 2022-10-26 20:41:02] ...trap/engines/engine.py.321: eval: epoch 28 | batch 10/510 [I 2022-10-26 20:41:02] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:08 [I 2022-10-26 20:41:02] ...trap/engines/engine.py.326: process: 0.32770 | load: 0.00018 [I 2022-10-26 20:41:06] ...trap/engines/engine.py.321: eval: epoch 28 | batch 20/510 [I 2022-10-26 20:41:06] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:03:02 [I 2022-10-26 20:41:06] ...trap/engines/engine.py.326: process: 0.38791 | load: 0.00017 [I 2022-10-26 20:41:09] ...trap/engines/engine.py.321: eval: epoch 28 | batch 30/510 [I 2022-10-26 20:41:09] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:42 [I 2022-10-26 20:41:09] ...trap/engines/engine.py.326: process: 0.34834 | load: 0.00016 [I 2022-10-26 20:41:13] ...trap/engines/engine.py.321: eval: epoch 28 | batch 40/510 [I 2022-10-26 20:41:13] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:37 [I 2022-10-26 20:41:13] ...trap/engines/engine.py.326: process: 0.34454 | load: 0.00016 [I 2022-10-26 20:41:16] ...trap/engines/engine.py.321: eval: epoch 28 | batch 50/510 [I 2022-10-26 20:41:16] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-26 20:41:16] ...trap/engines/engine.py.326: process: 0.35100 | load: 0.00019 [I 2022-10-26 20:41:19] ...trap/engines/engine.py.321: eval: epoch 28 | batch 60/510 [I 2022-10-26 20:41:19] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:29 [I 2022-10-26 20:41:19] ...trap/engines/engine.py.326: process: 0.32874 | load: 0.00022 [I 2022-10-26 20:41:23] ...trap/engines/engine.py.321: eval: epoch 28 | batch 70/510 [I 2022-10-26 20:41:23] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:26 [I 2022-10-26 20:41:23] ...trap/engines/engine.py.326: process: 0.33383 | load: 0.00016 [I 2022-10-26 20:41:26] ...trap/engines/engine.py.321: eval: epoch 28 | batch 80/510 [I 2022-10-26 20:41:26] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:26 [I 2022-10-26 20:41:26] ...trap/engines/engine.py.326: process: 0.37367 | load: 0.00017 [I 2022-10-26 20:41:29] ...trap/engines/engine.py.321: eval: epoch 28 | batch 90/510 [I 2022-10-26 20:41:29] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-26 20:41:29] ...trap/engines/engine.py.326: process: 0.32788 | load: 0.00016 [I 2022-10-26 20:41:33] ...trap/engines/engine.py.321: eval: epoch 28 | batch 100/510 [I 2022-10-26 20:41:33] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:14 [I 2022-10-26 20:41:33] ...trap/engines/engine.py.326: process: 0.29724 | load: 0.00017 [I 2022-10-26 20:41:35] ...trap/engines/engine.py.321: eval: epoch 28 | batch 110/510 [I 2022-10-26 20:41:35] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 20:41:35] ...trap/engines/engine.py.326: process: 0.28228 | load: 0.00019 [I 2022-10-26 20:41:38] ...trap/engines/engine.py.321: eval: epoch 28 | batch 120/510 [I 2022-10-26 20:41:38] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-26 20:41:38] ...trap/engines/engine.py.326: process: 0.29009 | load: 0.00015 [I 2022-10-26 20:41:41] ...trap/engines/engine.py.321: eval: epoch 28 | batch 130/510 [I 2022-10-26 20:41:41] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:48 [I 2022-10-26 20:41:41] ...trap/engines/engine.py.326: process: 0.28054 | load: 0.00015 [I 2022-10-26 20:41:44] ...trap/engines/engine.py.321: eval: epoch 28 | batch 140/510 [I 2022-10-26 20:41:44] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:46 [I 2022-10-26 20:41:44] ...trap/engines/engine.py.326: process: 0.28662 | load: 0.00019 [I 2022-10-26 20:41:47] ...trap/engines/engine.py.321: eval: epoch 28 | batch 150/510 [I 2022-10-26 20:41:47] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:44 [I 2022-10-26 20:41:47] ...trap/engines/engine.py.326: process: 0.29057 | load: 0.00017 [I 2022-10-26 20:41:50] ...trap/engines/engine.py.321: eval: epoch 28 | batch 160/510 [I 2022-10-26 20:41:50] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:41 [I 2022-10-26 20:41:50] ...trap/engines/engine.py.326: process: 0.28693 | load: 0.00018 [I 2022-10-26 20:41:53] ...trap/engines/engine.py.321: eval: epoch 28 | batch 170/510 [I 2022-10-26 20:41:53] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 20:41:53] ...trap/engines/engine.py.326: process: 0.28995 | load: 0.00017 [I 2022-10-26 20:41:56] ...trap/engines/engine.py.321: eval: epoch 28 | batch 180/510 [I 2022-10-26 20:41:56] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 20:41:56] ...trap/engines/engine.py.326: process: 0.28968 | load: 0.00016 [I 2022-10-26 20:41:59] ...trap/engines/engine.py.321: eval: epoch 28 | batch 190/510 [I 2022-10-26 20:41:59] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 20:41:59] ...trap/engines/engine.py.326: process: 0.28929 | load: 0.00033 [I 2022-10-26 20:42:01] ...trap/engines/engine.py.321: eval: epoch 28 | batch 200/510 [I 2022-10-26 20:42:01] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 20:42:01] ...trap/engines/engine.py.326: process: 0.30315 | load: 0.00018 [I 2022-10-26 20:42:04] ...trap/engines/engine.py.321: eval: epoch 28 | batch 210/510 [I 2022-10-26 20:42:04] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 20:42:04] ...trap/engines/engine.py.326: process: 0.28855 | load: 0.00020 [I 2022-10-26 20:42:07] ...trap/engines/engine.py.321: eval: epoch 28 | batch 220/510 [I 2022-10-26 20:42:07] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 20:42:07] ...trap/engines/engine.py.326: process: 0.28757 | load: 0.00018 [I 2022-10-26 20:42:10] ...trap/engines/engine.py.321: eval: epoch 28 | batch 230/510 [I 2022-10-26 20:42:10] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:20 [I 2022-10-26 20:42:10] ...trap/engines/engine.py.326: process: 0.28719 | load: 0.00020 [I 2022-10-26 20:42:13] ...trap/engines/engine.py.321: eval: epoch 28 | batch 240/510 [I 2022-10-26 20:42:13] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:18 [I 2022-10-26 20:42:13] ...trap/engines/engine.py.326: process: 0.28982 | load: 0.00016 [I 2022-10-26 20:42:16] ...trap/engines/engine.py.321: eval: epoch 28 | batch 250/510 [I 2022-10-26 20:42:16] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:15 [I 2022-10-26 20:42:16] ...trap/engines/engine.py.326: process: 0.28514 | load: 0.00016 [I 2022-10-26 20:42:19] ...trap/engines/engine.py.321: eval: epoch 28 | batch 260/510 [I 2022-10-26 20:42:19] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:11 [I 2022-10-26 20:42:19] ...trap/engines/engine.py.326: process: 0.28216 | load: 0.00016 [I 2022-10-26 20:42:22] ...trap/engines/engine.py.321: eval: epoch 28 | batch 270/510 [I 2022-10-26 20:42:22] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 20:42:22] ...trap/engines/engine.py.326: process: 0.29104 | load: 0.00016 [I 2022-10-26 20:42:25] ...trap/engines/engine.py.321: eval: epoch 28 | batch 280/510 [I 2022-10-26 20:42:25] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 20:42:25] ...trap/engines/engine.py.326: process: 0.29053 | load: 0.00018 [I 2022-10-26 20:42:27] ...trap/engines/engine.py.321: eval: epoch 28 | batch 290/510 [I 2022-10-26 20:42:27] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 20:42:27] ...trap/engines/engine.py.326: process: 0.28819 | load: 0.00017 [I 2022-10-26 20:42:30] ...trap/engines/engine.py.321: eval: epoch 28 | batch 300/510 [I 2022-10-26 20:42:30] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 20:42:30] ...trap/engines/engine.py.326: process: 0.27992 | load: 0.00019 [I 2022-10-26 20:42:33] ...trap/engines/engine.py.321: eval: epoch 28 | batch 310/510 [I 2022-10-26 20:42:33] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 20:42:33] ...trap/engines/engine.py.326: process: 0.28958 | load: 0.00018 [I 2022-10-26 20:42:36] ...trap/engines/engine.py.321: eval: epoch 28 | batch 320/510 [I 2022-10-26 20:42:36] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:54 [I 2022-10-26 20:42:36] ...trap/engines/engine.py.326: process: 0.28820 | load: 0.00019 [I 2022-10-26 20:42:39] ...trap/engines/engine.py.321: eval: epoch 28 | batch 330/510 [I 2022-10-26 20:42:39] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:51 [I 2022-10-26 20:42:39] ...trap/engines/engine.py.326: process: 0.29056 | load: 0.00016 [I 2022-10-26 20:42:42] ...trap/engines/engine.py.321: eval: epoch 28 | batch 340/510 [I 2022-10-26 20:42:42] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:48 [I 2022-10-26 20:42:42] ...trap/engines/engine.py.326: process: 0.28451 | load: 0.00016 [I 2022-10-26 20:42:45] ...trap/engines/engine.py.321: eval: epoch 28 | batch 350/510 [I 2022-10-26 20:42:45] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:46 [I 2022-10-26 20:42:45] ...trap/engines/engine.py.326: process: 0.28977 | load: 0.00015 [I 2022-10-26 20:42:48] ...trap/engines/engine.py.321: eval: epoch 28 | batch 360/510 [I 2022-10-26 20:42:48] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 20:42:48] ...trap/engines/engine.py.326: process: 0.28643 | load: 0.00016 [I 2022-10-26 20:42:50] ...trap/engines/engine.py.321: eval: epoch 28 | batch 370/510 [I 2022-10-26 20:42:50] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 20:42:50] ...trap/engines/engine.py.326: process: 0.28908 | load: 0.00018 [I 2022-10-26 20:42:53] ...trap/engines/engine.py.321: eval: epoch 28 | batch 380/510 [I 2022-10-26 20:42:53] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 20:42:53] ...trap/engines/engine.py.326: process: 0.27732 | load: 0.00020 [I 2022-10-26 20:42:56] ...trap/engines/engine.py.321: eval: epoch 28 | batch 390/510 [I 2022-10-26 20:42:56] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 20:42:56] ...trap/engines/engine.py.326: process: 0.29291 | load: 0.00015 [I 2022-10-26 20:42:59] ...trap/engines/engine.py.321: eval: epoch 28 | batch 400/510 [I 2022-10-26 20:42:59] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:31 [I 2022-10-26 20:42:59] ...trap/engines/engine.py.326: process: 0.28726 | load: 0.00020 [I 2022-10-26 20:43:02] ...trap/engines/engine.py.321: eval: epoch 28 | batch 410/510 [I 2022-10-26 20:43:02] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:28 [I 2022-10-26 20:43:02] ...trap/engines/engine.py.326: process: 0.28457 | load: 0.00016 [I 2022-10-26 20:43:05] ...trap/engines/engine.py.321: eval: epoch 28 | batch 420/510 [I 2022-10-26 20:43:05] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-26 20:43:05] ...trap/engines/engine.py.326: process: 0.28433 | load: 0.00016 [I 2022-10-26 20:43:08] ...trap/engines/engine.py.321: eval: epoch 28 | batch 430/510 [I 2022-10-26 20:43:08] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-26 20:43:08] ...trap/engines/engine.py.326: process: 0.29199 | load: 0.00016 [I 2022-10-26 20:43:11] ...trap/engines/engine.py.321: eval: epoch 28 | batch 440/510 [I 2022-10-26 20:43:11] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-26 20:43:11] ...trap/engines/engine.py.326: process: 0.27923 | load: 0.00018 [I 2022-10-26 20:43:14] ...trap/engines/engine.py.321: eval: epoch 28 | batch 450/510 [I 2022-10-26 20:43:14] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 20:43:14] ...trap/engines/engine.py.326: process: 0.28780 | load: 0.00020 [I 2022-10-26 20:43:16] ...trap/engines/engine.py.321: eval: epoch 28 | batch 460/510 [I 2022-10-26 20:43:16] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 20:43:16] ...trap/engines/engine.py.326: process: 0.28532 | load: 0.00021 [I 2022-10-26 20:43:19] ...trap/engines/engine.py.321: eval: epoch 28 | batch 470/510 [I 2022-10-26 20:43:19] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 20:43:19] ...trap/engines/engine.py.326: process: 0.28006 | load: 0.00018 [I 2022-10-26 20:43:22] ...trap/engines/engine.py.321: eval: epoch 28 | batch 480/510 [I 2022-10-26 20:43:22] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:08 [I 2022-10-26 20:43:22] ...trap/engines/engine.py.326: process: 0.28388 | load: 0.00021 [I 2022-10-26 20:43:25] ...trap/engines/engine.py.321: eval: epoch 28 | batch 490/510 [I 2022-10-26 20:43:25] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:05 [I 2022-10-26 20:43:25] ...trap/engines/engine.py.326: process: 0.28852 | load: 0.00018 [I 2022-10-26 20:43:28] ...trap/engines/engine.py.321: eval: epoch 28 | batch 500/510 [I 2022-10-26 20:43:28] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-26 20:43:28] ...trap/engines/engine.py.326: process: 0.29065 | load: 0.00018 [I 2022-10-26 20:43:31] ...trap/engines/engine.py.321: eval: epoch 28 | batch 510/510 [I 2022-10-26 20:43:31] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-26 20:43:31] ...trap/engines/engine.py.326: process: 0.28502 | load: 0.00010 [S 2022-10-26 20:43:32] ...trap/engines/engine.py.132: eval_epoch.epoch: 28 [I 2022-10-26 20:43:32] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.655300 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.879000 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.927400 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.645600 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.877000 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.919600 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014206 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007746 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006888 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014249 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006229 [S 2022-10-26 20:43:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004432 [I 2022-10-26 20:43:37] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 20:43:37] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 20:43:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 20:43:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 20:43:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 20:43:38] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 20:43:40] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 20:43:46] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 20:43:46] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 20:43:46] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 20:43:49] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 20:43:55] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 20:43:55] ...trap/engines/engine.py.163: Training model on trainset for epoch 29 [I 2022-10-26 20:43:55] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 20:43:58] ...trap/engines/engine.py.224: train: epoch 29 | batch 0/2304 [I 2022-10-26 20:43:58] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:40:10 [I 2022-10-26 20:43:58] ...trap/engines/engine.py.229: process: 2.60865 | load: 1.96793 [I 2022-10-26 20:43:58] ...trap/engines/engine.py.230: loss: 0.17266 [I 2022-10-26 20:43:58] ...optimizers/trijoint.py.151: total_norm: 13.873875 [I 2022-10-26 20:44:03] ...trap/engines/engine.py.224: train: epoch 29 | batch 10/2304 [I 2022-10-26 20:44:03] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:41 [I 2022-10-26 20:44:03] ...trap/engines/engine.py.229: process: 0.54955 | load: 0.00460 [I 2022-10-26 20:44:03] ...trap/engines/engine.py.230: loss: 0.21640 [I 2022-10-26 20:44:03] ...optimizers/trijoint.py.151: total_norm: 16.248199 [I 2022-10-26 20:44:09] ...trap/engines/engine.py.224: train: epoch 29 | batch 20/2304 [I 2022-10-26 20:44:09] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:05 [I 2022-10-26 20:44:09] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00456 [I 2022-10-26 20:44:09] ...trap/engines/engine.py.230: loss: 0.15249 [I 2022-10-26 20:44:09] ...optimizers/trijoint.py.151: total_norm: 20.376568 [I 2022-10-26 20:44:14] ...trap/engines/engine.py.224: train: epoch 29 | batch 30/2304 [I 2022-10-26 20:44:14] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:07 [I 2022-10-26 20:44:14] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00333 [I 2022-10-26 20:44:14] ...trap/engines/engine.py.230: loss: 0.27104 [I 2022-10-26 20:44:14] ...optimizers/trijoint.py.151: total_norm: 21.407448 [I 2022-10-26 20:44:20] ...trap/engines/engine.py.224: train: epoch 29 | batch 40/2304 [I 2022-10-26 20:44:20] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:59 [I 2022-10-26 20:44:20] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00304 [I 2022-10-26 20:44:20] ...trap/engines/engine.py.230: loss: 0.16567 [I 2022-10-26 20:44:20] ...optimizers/trijoint.py.151: total_norm: 22.567337 [I 2022-10-26 20:44:26] ...trap/engines/engine.py.224: train: epoch 29 | batch 50/2304 [I 2022-10-26 20:44:26] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:48 [I 2022-10-26 20:44:26] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00282 [I 2022-10-26 20:44:26] ...trap/engines/engine.py.230: loss: 0.22667 [I 2022-10-26 20:44:26] ...optimizers/trijoint.py.151: total_norm: 22.538288 [I 2022-10-26 20:44:31] ...trap/engines/engine.py.224: train: epoch 29 | batch 60/2304 [I 2022-10-26 20:44:31] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:52 [I 2022-10-26 20:44:31] ...trap/engines/engine.py.229: process: 0.56396 | load: 0.00269 [I 2022-10-26 20:44:31] ...trap/engines/engine.py.230: loss: 0.10775 [I 2022-10-26 20:44:31] ...optimizers/trijoint.py.151: total_norm: 12.241325 [I 2022-10-26 20:44:37] ...trap/engines/engine.py.224: train: epoch 29 | batch 70/2304 [I 2022-10-26 20:44:37] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:51 [I 2022-10-26 20:44:37] ...trap/engines/engine.py.229: process: 0.55980 | load: 0.00289 [I 2022-10-26 20:44:37] ...trap/engines/engine.py.230: loss: 0.11995 [I 2022-10-26 20:44:37] ...optimizers/trijoint.py.151: total_norm: 9.067126 [I 2022-10-26 20:44:42] ...trap/engines/engine.py.224: train: epoch 29 | batch 80/2304 [I 2022-10-26 20:44:42] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:31 [I 2022-10-26 20:44:42] ...trap/engines/engine.py.229: process: 0.54625 | load: 0.00244 [I 2022-10-26 20:44:42] ...trap/engines/engine.py.230: loss: 0.19179 [I 2022-10-26 20:44:42] ...optimizers/trijoint.py.151: total_norm: 16.524050 [I 2022-10-26 20:44:48] ...trap/engines/engine.py.224: train: epoch 29 | batch 90/2304 [I 2022-10-26 20:44:48] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:30 [I 2022-10-26 20:44:48] ...trap/engines/engine.py.229: process: 0.56387 | load: 0.00257 [I 2022-10-26 20:44:48] ...trap/engines/engine.py.230: loss: 0.14118 [I 2022-10-26 20:44:48] ...optimizers/trijoint.py.151: total_norm: 15.378263 [I 2022-10-26 20:44:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 100/2304 [I 2022-10-26 20:44:53] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:23 [I 2022-10-26 20:44:53] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00281 [I 2022-10-26 20:44:53] ...trap/engines/engine.py.230: loss: 0.23288 [I 2022-10-26 20:44:53] ...optimizers/trijoint.py.151: total_norm: 27.078934 [I 2022-10-26 20:44:59] ...trap/engines/engine.py.224: train: epoch 29 | batch 110/2304 [I 2022-10-26 20:44:59] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:20:19 [I 2022-10-26 20:44:59] ...trap/engines/engine.py.229: process: 0.56842 | load: 0.00299 [I 2022-10-26 20:44:59] ...trap/engines/engine.py.230: loss: 0.18823 [I 2022-10-26 20:44:59] ...optimizers/trijoint.py.151: total_norm: 21.532888 [I 2022-10-26 20:45:05] ...trap/engines/engine.py.224: train: epoch 29 | batch 120/2304 [I 2022-10-26 20:45:05] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:22 [I 2022-10-26 20:45:05] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00262 [I 2022-10-26 20:45:05] ...trap/engines/engine.py.230: loss: 0.16049 [I 2022-10-26 20:45:05] ...optimizers/trijoint.py.151: total_norm: 17.352245 [I 2022-10-26 20:45:10] ...trap/engines/engine.py.224: train: epoch 29 | batch 130/2304 [I 2022-10-26 20:45:10] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:12 [I 2022-10-26 20:45:10] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00263 [I 2022-10-26 20:45:10] ...trap/engines/engine.py.230: loss: 0.18086 [I 2022-10-26 20:45:10] ...optimizers/trijoint.py.151: total_norm: 15.704250 [I 2022-10-26 20:45:16] ...trap/engines/engine.py.224: train: epoch 29 | batch 140/2304 [I 2022-10-26 20:45:16] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:01 [I 2022-10-26 20:45:16] ...trap/engines/engine.py.229: process: 0.55194 | load: 0.00285 [I 2022-10-26 20:45:16] ...trap/engines/engine.py.230: loss: 0.16469 [I 2022-10-26 20:45:16] ...optimizers/trijoint.py.151: total_norm: 24.819838 [I 2022-10-26 20:45:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 150/2304 [I 2022-10-26 20:45:21] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:02 [I 2022-10-26 20:45:21] ...trap/engines/engine.py.229: process: 0.55360 | load: 0.00300 [I 2022-10-26 20:45:21] ...trap/engines/engine.py.230: loss: 0.21919 [I 2022-10-26 20:45:21] ...optimizers/trijoint.py.151: total_norm: 32.038864 [I 2022-10-26 20:45:27] ...trap/engines/engine.py.224: train: epoch 29 | batch 160/2304 [I 2022-10-26 20:45:27] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:53 [I 2022-10-26 20:45:27] ...trap/engines/engine.py.229: process: 0.55320 | load: 0.00264 [I 2022-10-26 20:45:27] ...trap/engines/engine.py.230: loss: 0.20181 [I 2022-10-26 20:45:27] ...optimizers/trijoint.py.151: total_norm: 21.389048 [I 2022-10-26 20:45:33] ...trap/engines/engine.py.224: train: epoch 29 | batch 170/2304 [I 2022-10-26 20:45:33] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:59 [I 2022-10-26 20:45:33] ...trap/engines/engine.py.229: process: 0.56347 | load: 0.00258 [I 2022-10-26 20:45:33] ...trap/engines/engine.py.230: loss: 0.17490 [I 2022-10-26 20:45:33] ...optimizers/trijoint.py.151: total_norm: 17.100647 [I 2022-10-26 20:45:38] ...trap/engines/engine.py.224: train: epoch 29 | batch 180/2304 [I 2022-10-26 20:45:38] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:20:12 [I 2022-10-26 20:45:38] ...trap/engines/engine.py.229: process: 0.55638 | load: 0.00242 [I 2022-10-26 20:45:38] ...trap/engines/engine.py.230: loss: 0.16002 [I 2022-10-26 20:45:38] ...optimizers/trijoint.py.151: total_norm: 15.583776 [I 2022-10-26 20:45:44] ...trap/engines/engine.py.224: train: epoch 29 | batch 190/2304 [I 2022-10-26 20:45:44] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:46 [I 2022-10-26 20:45:44] ...trap/engines/engine.py.229: process: 0.56268 | load: 0.00252 [I 2022-10-26 20:45:44] ...trap/engines/engine.py.230: loss: 0.14629 [I 2022-10-26 20:45:44] ...optimizers/trijoint.py.151: total_norm: 16.420374 [I 2022-10-26 20:45:50] ...trap/engines/engine.py.224: train: epoch 29 | batch 200/2304 [I 2022-10-26 20:45:50] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:35 [I 2022-10-26 20:45:50] ...trap/engines/engine.py.229: process: 0.54174 | load: 0.00267 [I 2022-10-26 20:45:50] ...trap/engines/engine.py.230: loss: 0.23053 [I 2022-10-26 20:45:50] ...optimizers/trijoint.py.151: total_norm: 18.117245 [I 2022-10-26 20:45:55] ...trap/engines/engine.py.224: train: epoch 29 | batch 210/2304 [I 2022-10-26 20:45:55] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:30 [I 2022-10-26 20:45:55] ...trap/engines/engine.py.229: process: 0.56102 | load: 0.00256 [I 2022-10-26 20:45:55] ...trap/engines/engine.py.230: loss: 0.20821 [I 2022-10-26 20:45:55] ...optimizers/trijoint.py.151: total_norm: 14.772013 [I 2022-10-26 20:46:01] ...trap/engines/engine.py.224: train: epoch 29 | batch 220/2304 [I 2022-10-26 20:46:01] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:22 [I 2022-10-26 20:46:01] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00252 [I 2022-10-26 20:46:01] ...trap/engines/engine.py.230: loss: 0.19782 [I 2022-10-26 20:46:01] ...optimizers/trijoint.py.151: total_norm: 15.540347 [I 2022-10-26 20:46:06] ...trap/engines/engine.py.224: train: epoch 29 | batch 230/2304 [I 2022-10-26 20:46:06] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:17 [I 2022-10-26 20:46:06] ...trap/engines/engine.py.229: process: 0.54592 | load: 0.00257 [I 2022-10-26 20:46:06] ...trap/engines/engine.py.230: loss: 0.11660 [I 2022-10-26 20:46:06] ...optimizers/trijoint.py.151: total_norm: 12.155389 [I 2022-10-26 20:46:12] ...trap/engines/engine.py.224: train: epoch 29 | batch 240/2304 [I 2022-10-26 20:46:12] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:05 [I 2022-10-26 20:46:12] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00261 [I 2022-10-26 20:46:12] ...trap/engines/engine.py.230: loss: 0.20667 [I 2022-10-26 20:46:12] ...optimizers/trijoint.py.151: total_norm: 15.163535 [I 2022-10-26 20:46:18] ...trap/engines/engine.py.224: train: epoch 29 | batch 250/2304 [I 2022-10-26 20:46:18] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:04 [I 2022-10-26 20:46:18] ...trap/engines/engine.py.229: process: 0.55824 | load: 0.00246 [I 2022-10-26 20:46:18] ...trap/engines/engine.py.230: loss: 0.22793 [I 2022-10-26 20:46:18] ...optimizers/trijoint.py.151: total_norm: 19.247290 [I 2022-10-26 20:46:23] ...trap/engines/engine.py.224: train: epoch 29 | batch 260/2304 [I 2022-10-26 20:46:23] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:59 [I 2022-10-26 20:46:23] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00244 [I 2022-10-26 20:46:23] ...trap/engines/engine.py.230: loss: 0.14223 [I 2022-10-26 20:46:23] ...optimizers/trijoint.py.151: total_norm: 17.591326 [I 2022-10-26 20:46:29] ...trap/engines/engine.py.224: train: epoch 29 | batch 270/2304 [I 2022-10-26 20:46:29] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:07 [I 2022-10-26 20:46:29] ...trap/engines/engine.py.229: process: 0.55096 | load: 0.00259 [I 2022-10-26 20:46:29] ...trap/engines/engine.py.230: loss: 0.29803 [I 2022-10-26 20:46:29] ...optimizers/trijoint.py.151: total_norm: 22.965096 [I 2022-10-26 20:46:34] ...trap/engines/engine.py.224: train: epoch 29 | batch 280/2304 [I 2022-10-26 20:46:34] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:51 [I 2022-10-26 20:46:34] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00250 [I 2022-10-26 20:46:34] ...trap/engines/engine.py.230: loss: 0.21274 [I 2022-10-26 20:46:34] ...optimizers/trijoint.py.151: total_norm: 22.896368 [I 2022-10-26 20:46:40] ...trap/engines/engine.py.224: train: epoch 29 | batch 290/2304 [I 2022-10-26 20:46:40] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:49 [I 2022-10-26 20:46:40] ...trap/engines/engine.py.229: process: 0.56127 | load: 0.00254 [I 2022-10-26 20:46:40] ...trap/engines/engine.py.230: loss: 0.16318 [I 2022-10-26 20:46:40] ...optimizers/trijoint.py.151: total_norm: 14.394887 [I 2022-10-26 20:46:46] ...trap/engines/engine.py.224: train: epoch 29 | batch 300/2304 [I 2022-10-26 20:46:46] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:34 [I 2022-10-26 20:46:46] ...trap/engines/engine.py.229: process: 0.54486 | load: 0.00245 [I 2022-10-26 20:46:46] ...trap/engines/engine.py.230: loss: 0.18471 [I 2022-10-26 20:46:46] ...optimizers/trijoint.py.151: total_norm: 24.485020 [I 2022-10-26 20:46:51] ...trap/engines/engine.py.224: train: epoch 29 | batch 310/2304 [I 2022-10-26 20:46:51] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:32 [I 2022-10-26 20:46:51] ...trap/engines/engine.py.229: process: 0.56426 | load: 0.00244 [I 2022-10-26 20:46:51] ...trap/engines/engine.py.230: loss: 0.19273 [I 2022-10-26 20:46:51] ...optimizers/trijoint.py.151: total_norm: 28.141443 [I 2022-10-26 20:46:57] ...trap/engines/engine.py.224: train: epoch 29 | batch 320/2304 [I 2022-10-26 20:46:57] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:19 [I 2022-10-26 20:46:57] ...trap/engines/engine.py.229: process: 0.55070 | load: 0.00252 [I 2022-10-26 20:46:57] ...trap/engines/engine.py.230: loss: 0.17198 [I 2022-10-26 20:46:57] ...optimizers/trijoint.py.151: total_norm: 15.015105 [I 2022-10-26 20:47:03] ...trap/engines/engine.py.224: train: epoch 29 | batch 330/2304 [I 2022-10-26 20:47:03] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:56 [I 2022-10-26 20:47:03] ...trap/engines/engine.py.229: process: 0.56976 | load: 0.00243 [I 2022-10-26 20:47:03] ...trap/engines/engine.py.230: loss: 0.14724 [I 2022-10-26 20:47:03] ...optimizers/trijoint.py.151: total_norm: 15.958414 [I 2022-10-26 20:47:08] ...trap/engines/engine.py.224: train: epoch 29 | batch 340/2304 [I 2022-10-26 20:47:08] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:21 [I 2022-10-26 20:47:08] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00243 [I 2022-10-26 20:47:08] ...trap/engines/engine.py.230: loss: 0.16495 [I 2022-10-26 20:47:08] ...optimizers/trijoint.py.151: total_norm: 16.667681 [I 2022-10-26 20:47:14] ...trap/engines/engine.py.224: train: epoch 29 | batch 350/2304 [I 2022-10-26 20:47:14] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:07 [I 2022-10-26 20:47:14] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00246 [I 2022-10-26 20:47:14] ...trap/engines/engine.py.230: loss: 0.23083 [I 2022-10-26 20:47:14] ...optimizers/trijoint.py.151: total_norm: 17.321955 [I 2022-10-26 20:47:19] ...trap/engines/engine.py.224: train: epoch 29 | batch 360/2304 [I 2022-10-26 20:47:19] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:05 [I 2022-10-26 20:47:19] ...trap/engines/engine.py.229: process: 0.54807 | load: 0.00245 [I 2022-10-26 20:47:19] ...trap/engines/engine.py.230: loss: 0.15274 [I 2022-10-26 20:47:19] ...optimizers/trijoint.py.151: total_norm: 15.561730 [I 2022-10-26 20:47:25] ...trap/engines/engine.py.224: train: epoch 29 | batch 370/2304 [I 2022-10-26 20:47:25] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:05 [I 2022-10-26 20:47:25] ...trap/engines/engine.py.229: process: 0.57143 | load: 0.00242 [I 2022-10-26 20:47:25] ...trap/engines/engine.py.230: loss: 0.13216 [I 2022-10-26 20:47:25] ...optimizers/trijoint.py.151: total_norm: 14.989370 [I 2022-10-26 20:47:30] ...trap/engines/engine.py.224: train: epoch 29 | batch 380/2304 [I 2022-10-26 20:47:30] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:50 [I 2022-10-26 20:47:30] ...trap/engines/engine.py.229: process: 0.55235 | load: 0.00256 [I 2022-10-26 20:47:30] ...trap/engines/engine.py.230: loss: 0.16328 [I 2022-10-26 20:47:30] ...optimizers/trijoint.py.151: total_norm: 15.521315 [I 2022-10-26 20:47:36] ...trap/engines/engine.py.224: train: epoch 29 | batch 390/2304 [I 2022-10-26 20:47:36] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:52 [I 2022-10-26 20:47:36] ...trap/engines/engine.py.229: process: 0.56657 | load: 0.00251 [I 2022-10-26 20:47:36] ...trap/engines/engine.py.230: loss: 0.20573 [I 2022-10-26 20:47:36] ...optimizers/trijoint.py.151: total_norm: 13.489052 [I 2022-10-26 20:47:42] ...trap/engines/engine.py.224: train: epoch 29 | batch 400/2304 [I 2022-10-26 20:47:42] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:42 [I 2022-10-26 20:47:42] ...trap/engines/engine.py.229: process: 0.55474 | load: 0.00243 [I 2022-10-26 20:47:42] ...trap/engines/engine.py.230: loss: 0.11880 [I 2022-10-26 20:47:42] ...optimizers/trijoint.py.151: total_norm: 8.077933 [I 2022-10-26 20:47:47] ...trap/engines/engine.py.224: train: epoch 29 | batch 410/2304 [I 2022-10-26 20:47:47] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:38 [I 2022-10-26 20:47:47] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00247 [I 2022-10-26 20:47:47] ...trap/engines/engine.py.230: loss: 0.14481 [I 2022-10-26 20:47:47] ...optimizers/trijoint.py.151: total_norm: 17.070499 [I 2022-10-26 20:47:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 420/2304 [I 2022-10-26 20:47:53] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:30 [I 2022-10-26 20:47:53] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00245 [I 2022-10-26 20:47:53] ...trap/engines/engine.py.230: loss: 0.12175 [I 2022-10-26 20:47:53] ...optimizers/trijoint.py.151: total_norm: 19.259638 [I 2022-10-26 20:47:59] ...trap/engines/engine.py.224: train: epoch 29 | batch 430/2304 [I 2022-10-26 20:47:59] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:45 [I 2022-10-26 20:47:59] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00244 [I 2022-10-26 20:47:59] ...trap/engines/engine.py.230: loss: 0.19610 [I 2022-10-26 20:47:59] ...optimizers/trijoint.py.151: total_norm: 21.146351 [I 2022-10-26 20:48:04] ...trap/engines/engine.py.224: train: epoch 29 | batch 440/2304 [I 2022-10-26 20:48:04] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:18 [I 2022-10-26 20:48:04] ...trap/engines/engine.py.229: process: 0.54296 | load: 0.00240 [I 2022-10-26 20:48:04] ...trap/engines/engine.py.230: loss: 0.16368 [I 2022-10-26 20:48:04] ...optimizers/trijoint.py.151: total_norm: 18.208412 [I 2022-10-26 20:48:10] ...trap/engines/engine.py.224: train: epoch 29 | batch 450/2304 [I 2022-10-26 20:48:10] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:08 [I 2022-10-26 20:48:10] ...trap/engines/engine.py.229: process: 0.54907 | load: 0.00247 [I 2022-10-26 20:48:10] ...trap/engines/engine.py.230: loss: 0.18107 [I 2022-10-26 20:48:10] ...optimizers/trijoint.py.151: total_norm: 18.351955 [I 2022-10-26 20:48:15] ...trap/engines/engine.py.224: train: epoch 29 | batch 460/2304 [I 2022-10-26 20:48:15] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:05 [I 2022-10-26 20:48:15] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00249 [I 2022-10-26 20:48:15] ...trap/engines/engine.py.230: loss: 0.16720 [I 2022-10-26 20:48:15] ...optimizers/trijoint.py.151: total_norm: 13.911505 [I 2022-10-26 20:48:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 470/2304 [I 2022-10-26 20:48:21] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:56 [I 2022-10-26 20:48:21] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00257 [I 2022-10-26 20:48:21] ...trap/engines/engine.py.230: loss: 0.13292 [I 2022-10-26 20:48:21] ...optimizers/trijoint.py.151: total_norm: 13.545319 [I 2022-10-26 20:48:26] ...trap/engines/engine.py.224: train: epoch 29 | batch 480/2304 [I 2022-10-26 20:48:26] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:01 [I 2022-10-26 20:48:26] ...trap/engines/engine.py.229: process: 0.55099 | load: 0.00257 [I 2022-10-26 20:48:26] ...trap/engines/engine.py.230: loss: 0.17255 [I 2022-10-26 20:48:26] ...optimizers/trijoint.py.151: total_norm: 21.708372 [I 2022-10-26 20:48:32] ...trap/engines/engine.py.224: train: epoch 29 | batch 490/2304 [I 2022-10-26 20:48:32] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:46 [I 2022-10-26 20:48:32] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00268 [I 2022-10-26 20:48:32] ...trap/engines/engine.py.230: loss: 0.12419 [I 2022-10-26 20:48:32] ...optimizers/trijoint.py.151: total_norm: 19.362705 [I 2022-10-26 20:48:38] ...trap/engines/engine.py.224: train: epoch 29 | batch 500/2304 [I 2022-10-26 20:48:38] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:44 [I 2022-10-26 20:48:38] ...trap/engines/engine.py.229: process: 0.55868 | load: 0.00243 [I 2022-10-26 20:48:38] ...trap/engines/engine.py.230: loss: 0.23641 [I 2022-10-26 20:48:38] ...optimizers/trijoint.py.151: total_norm: 28.782787 [I 2022-10-26 20:48:43] ...trap/engines/engine.py.224: train: epoch 29 | batch 510/2304 [I 2022-10-26 20:48:43] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:36 [I 2022-10-26 20:48:43] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00245 [I 2022-10-26 20:48:43] ...trap/engines/engine.py.230: loss: 0.15169 [I 2022-10-26 20:48:43] ...optimizers/trijoint.py.151: total_norm: 25.089592 [I 2022-10-26 20:48:49] ...trap/engines/engine.py.224: train: epoch 29 | batch 520/2304 [I 2022-10-26 20:48:49] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:29 [I 2022-10-26 20:48:49] ...trap/engines/engine.py.229: process: 0.56082 | load: 0.00245 [I 2022-10-26 20:48:49] ...trap/engines/engine.py.230: loss: 0.13403 [I 2022-10-26 20:48:49] ...optimizers/trijoint.py.151: total_norm: 44.500767 [I 2022-10-26 20:48:54] ...trap/engines/engine.py.224: train: epoch 29 | batch 530/2304 [I 2022-10-26 20:48:54] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:25 [I 2022-10-26 20:48:54] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00252 [I 2022-10-26 20:48:54] ...trap/engines/engine.py.230: loss: 0.23945 [I 2022-10-26 20:48:54] ...optimizers/trijoint.py.151: total_norm: 35.046967 [I 2022-10-26 20:49:00] ...trap/engines/engine.py.224: train: epoch 29 | batch 540/2304 [I 2022-10-26 20:49:00] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:22 [I 2022-10-26 20:49:00] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00243 [I 2022-10-26 20:49:00] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-26 20:49:00] ...optimizers/trijoint.py.151: total_norm: 13.843025 [I 2022-10-26 20:49:05] ...trap/engines/engine.py.224: train: epoch 29 | batch 550/2304 [I 2022-10-26 20:49:05] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:09 [I 2022-10-26 20:49:05] ...trap/engines/engine.py.229: process: 0.54767 | load: 0.00243 [I 2022-10-26 20:49:05] ...trap/engines/engine.py.230: loss: 0.13399 [I 2022-10-26 20:49:05] ...optimizers/trijoint.py.151: total_norm: 6.969630 [I 2022-10-26 20:49:11] ...trap/engines/engine.py.224: train: epoch 29 | batch 560/2304 [I 2022-10-26 20:49:11] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:33 [I 2022-10-26 20:49:11] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00250 [I 2022-10-26 20:49:11] ...trap/engines/engine.py.230: loss: 0.19282 [I 2022-10-26 20:49:11] ...optimizers/trijoint.py.151: total_norm: 15.540966 [I 2022-10-26 20:49:17] ...trap/engines/engine.py.224: train: epoch 29 | batch 570/2304 [I 2022-10-26 20:49:17] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:08 [I 2022-10-26 20:49:17] ...trap/engines/engine.py.229: process: 0.54569 | load: 0.00247 [I 2022-10-26 20:49:17] ...trap/engines/engine.py.230: loss: 0.14645 [I 2022-10-26 20:49:17] ...optimizers/trijoint.py.151: total_norm: 20.125956 [I 2022-10-26 20:49:22] ...trap/engines/engine.py.224: train: epoch 29 | batch 580/2304 [I 2022-10-26 20:49:22] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:10 [I 2022-10-26 20:49:22] ...trap/engines/engine.py.229: process: 0.56082 | load: 0.00242 [I 2022-10-26 20:49:22] ...trap/engines/engine.py.230: loss: 0.19470 [I 2022-10-26 20:49:22] ...optimizers/trijoint.py.151: total_norm: 15.332913 [I 2022-10-26 20:49:28] ...trap/engines/engine.py.224: train: epoch 29 | batch 590/2304 [I 2022-10-26 20:49:28] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:16:03 [I 2022-10-26 20:49:28] ...trap/engines/engine.py.229: process: 0.56185 | load: 0.00245 [I 2022-10-26 20:49:28] ...trap/engines/engine.py.230: loss: 0.18404 [I 2022-10-26 20:49:28] ...optimizers/trijoint.py.151: total_norm: 26.833923 [I 2022-10-26 20:49:33] ...trap/engines/engine.py.224: train: epoch 29 | batch 600/2304 [I 2022-10-26 20:49:33] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:47 [I 2022-10-26 20:49:33] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00247 [I 2022-10-26 20:49:33] ...trap/engines/engine.py.230: loss: 0.22005 [I 2022-10-26 20:49:33] ...optimizers/trijoint.py.151: total_norm: 27.129131 [I 2022-10-26 20:49:39] ...trap/engines/engine.py.224: train: epoch 29 | batch 610/2304 [I 2022-10-26 20:49:39] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:47 [I 2022-10-26 20:49:39] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00260 [I 2022-10-26 20:49:39] ...trap/engines/engine.py.230: loss: 0.19205 [I 2022-10-26 20:49:39] ...optimizers/trijoint.py.151: total_norm: 16.408421 [I 2022-10-26 20:49:45] ...trap/engines/engine.py.224: train: epoch 29 | batch 620/2304 [I 2022-10-26 20:49:45] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:36 [I 2022-10-26 20:49:45] ...trap/engines/engine.py.229: process: 0.55480 | load: 0.00247 [I 2022-10-26 20:49:45] ...trap/engines/engine.py.230: loss: 0.16434 [I 2022-10-26 20:49:45] ...optimizers/trijoint.py.151: total_norm: 17.556585 [I 2022-10-26 20:49:50] ...trap/engines/engine.py.224: train: epoch 29 | batch 630/2304 [I 2022-10-26 20:49:50] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:32 [I 2022-10-26 20:49:50] ...trap/engines/engine.py.229: process: 0.54766 | load: 0.00244 [I 2022-10-26 20:49:50] ...trap/engines/engine.py.230: loss: 0.14300 [I 2022-10-26 20:49:50] ...optimizers/trijoint.py.151: total_norm: 20.896803 [I 2022-10-26 20:49:56] ...trap/engines/engine.py.224: train: epoch 29 | batch 640/2304 [I 2022-10-26 20:49:56] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:27 [I 2022-10-26 20:49:56] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00243 [I 2022-10-26 20:49:56] ...trap/engines/engine.py.230: loss: 0.28340 [I 2022-10-26 20:49:56] ...optimizers/trijoint.py.151: total_norm: 19.348610 [I 2022-10-26 20:50:01] ...trap/engines/engine.py.224: train: epoch 29 | batch 650/2304 [I 2022-10-26 20:50:01] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:29 [I 2022-10-26 20:50:01] ...trap/engines/engine.py.229: process: 0.56884 | load: 0.00243 [I 2022-10-26 20:50:01] ...trap/engines/engine.py.230: loss: 0.22499 [I 2022-10-26 20:50:01] ...optimizers/trijoint.py.151: total_norm: 18.361738 [I 2022-10-26 20:50:07] ...trap/engines/engine.py.224: train: epoch 29 | batch 660/2304 [I 2022-10-26 20:50:07] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:20 [I 2022-10-26 20:50:07] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00240 [I 2022-10-26 20:50:07] ...trap/engines/engine.py.230: loss: 0.25016 [I 2022-10-26 20:50:07] ...optimizers/trijoint.py.151: total_norm: 23.622986 [I 2022-10-26 20:50:13] ...trap/engines/engine.py.224: train: epoch 29 | batch 670/2304 [I 2022-10-26 20:50:13] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:09 [I 2022-10-26 20:50:13] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00243 [I 2022-10-26 20:50:13] ...trap/engines/engine.py.230: loss: 0.14204 [I 2022-10-26 20:50:13] ...optimizers/trijoint.py.151: total_norm: 18.796791 [I 2022-10-26 20:50:18] ...trap/engines/engine.py.224: train: epoch 29 | batch 680/2304 [I 2022-10-26 20:50:18] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:03 [I 2022-10-26 20:50:18] ...trap/engines/engine.py.229: process: 0.55968 | load: 0.00248 [I 2022-10-26 20:50:18] ...trap/engines/engine.py.230: loss: 0.12172 [I 2022-10-26 20:50:18] ...optimizers/trijoint.py.151: total_norm: 14.846280 [I 2022-10-26 20:50:24] ...trap/engines/engine.py.224: train: epoch 29 | batch 690/2304 [I 2022-10-26 20:50:24] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:10 [I 2022-10-26 20:50:24] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00247 [I 2022-10-26 20:50:24] ...trap/engines/engine.py.230: loss: 0.16141 [I 2022-10-26 20:50:24] ...optimizers/trijoint.py.151: total_norm: 9.727560 [I 2022-10-26 20:50:29] ...trap/engines/engine.py.224: train: epoch 29 | batch 700/2304 [I 2022-10-26 20:50:29] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:55 [I 2022-10-26 20:50:29] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00256 [I 2022-10-26 20:50:29] ...trap/engines/engine.py.230: loss: 0.19837 [I 2022-10-26 20:50:29] ...optimizers/trijoint.py.151: total_norm: 21.931314 [I 2022-10-26 20:50:35] ...trap/engines/engine.py.224: train: epoch 29 | batch 710/2304 [I 2022-10-26 20:50:35] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:49 [I 2022-10-26 20:50:35] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00249 [I 2022-10-26 20:50:35] ...trap/engines/engine.py.230: loss: 0.13317 [I 2022-10-26 20:50:35] ...optimizers/trijoint.py.151: total_norm: 12.254764 [I 2022-10-26 20:50:41] ...trap/engines/engine.py.224: train: epoch 29 | batch 720/2304 [I 2022-10-26 20:50:41] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:41 [I 2022-10-26 20:50:41] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00249 [I 2022-10-26 20:50:41] ...trap/engines/engine.py.230: loss: 0.19036 [I 2022-10-26 20:50:41] ...optimizers/trijoint.py.151: total_norm: 29.709162 [I 2022-10-26 20:50:46] ...trap/engines/engine.py.224: train: epoch 29 | batch 730/2304 [I 2022-10-26 20:50:46] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:37 [I 2022-10-26 20:50:46] ...trap/engines/engine.py.229: process: 0.55147 | load: 0.00245 [I 2022-10-26 20:50:46] ...trap/engines/engine.py.230: loss: 0.17938 [I 2022-10-26 20:50:46] ...optimizers/trijoint.py.151: total_norm: 12.466025 [I 2022-10-26 20:50:52] ...trap/engines/engine.py.224: train: epoch 29 | batch 740/2304 [I 2022-10-26 20:50:52] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:29 [I 2022-10-26 20:50:52] ...trap/engines/engine.py.229: process: 0.55154 | load: 0.00244 [I 2022-10-26 20:50:52] ...trap/engines/engine.py.230: loss: 0.21421 [I 2022-10-26 20:50:52] ...optimizers/trijoint.py.151: total_norm: 15.733682 [I 2022-10-26 20:50:57] ...trap/engines/engine.py.224: train: epoch 29 | batch 750/2304 [I 2022-10-26 20:50:57] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:26 [I 2022-10-26 20:50:57] ...trap/engines/engine.py.229: process: 0.55756 | load: 0.00242 [I 2022-10-26 20:50:57] ...trap/engines/engine.py.230: loss: 0.16599 [I 2022-10-26 20:50:57] ...optimizers/trijoint.py.151: total_norm: 20.749058 [I 2022-10-26 20:51:03] ...trap/engines/engine.py.224: train: epoch 29 | batch 760/2304 [I 2022-10-26 20:51:03] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:15 [I 2022-10-26 20:51:03] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00240 [I 2022-10-26 20:51:03] ...trap/engines/engine.py.230: loss: 0.20819 [I 2022-10-26 20:51:03] ...optimizers/trijoint.py.151: total_norm: 14.131040 [I 2022-10-26 20:51:08] ...trap/engines/engine.py.224: train: epoch 29 | batch 770/2304 [I 2022-10-26 20:51:08] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:12 [I 2022-10-26 20:51:08] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00248 [I 2022-10-26 20:51:08] ...trap/engines/engine.py.230: loss: 0.16938 [I 2022-10-26 20:51:08] ...optimizers/trijoint.py.151: total_norm: 24.639740 [I 2022-10-26 20:51:14] ...trap/engines/engine.py.224: train: epoch 29 | batch 780/2304 [I 2022-10-26 20:51:14] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:10 [I 2022-10-26 20:51:14] ...trap/engines/engine.py.229: process: 0.54710 | load: 0.00244 [I 2022-10-26 20:51:14] ...trap/engines/engine.py.230: loss: 0.25753 [I 2022-10-26 20:51:14] ...optimizers/trijoint.py.151: total_norm: 28.316624 [I 2022-10-26 20:51:20] ...trap/engines/engine.py.224: train: epoch 29 | batch 790/2304 [I 2022-10-26 20:51:20] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:05 [I 2022-10-26 20:51:20] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00243 [I 2022-10-26 20:51:20] ...trap/engines/engine.py.230: loss: 0.22398 [I 2022-10-26 20:51:20] ...optimizers/trijoint.py.151: total_norm: 23.233582 [I 2022-10-26 20:51:25] ...trap/engines/engine.py.224: train: epoch 29 | batch 800/2304 [I 2022-10-26 20:51:25] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:07 [I 2022-10-26 20:51:25] ...trap/engines/engine.py.229: process: 0.56450 | load: 0.00242 [I 2022-10-26 20:51:25] ...trap/engines/engine.py.230: loss: 0.13385 [I 2022-10-26 20:51:25] ...optimizers/trijoint.py.151: total_norm: 12.123596 [I 2022-10-26 20:51:31] ...trap/engines/engine.py.224: train: epoch 29 | batch 810/2304 [I 2022-10-26 20:51:31] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:51 [I 2022-10-26 20:51:31] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00255 [I 2022-10-26 20:51:31] ...trap/engines/engine.py.230: loss: 0.18558 [I 2022-10-26 20:51:31] ...optimizers/trijoint.py.151: total_norm: 26.077322 [I 2022-10-26 20:51:36] ...trap/engines/engine.py.224: train: epoch 29 | batch 820/2304 [I 2022-10-26 20:51:36] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:39 [I 2022-10-26 20:51:36] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00250 [I 2022-10-26 20:51:36] ...trap/engines/engine.py.230: loss: 0.16225 [I 2022-10-26 20:51:36] ...optimizers/trijoint.py.151: total_norm: 15.363524 [I 2022-10-26 20:51:42] ...trap/engines/engine.py.224: train: epoch 29 | batch 830/2304 [I 2022-10-26 20:51:42] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:39 [I 2022-10-26 20:51:42] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00247 [I 2022-10-26 20:51:42] ...trap/engines/engine.py.230: loss: 0.13255 [I 2022-10-26 20:51:42] ...optimizers/trijoint.py.151: total_norm: 11.735779 [I 2022-10-26 20:51:48] ...trap/engines/engine.py.224: train: epoch 29 | batch 840/2304 [I 2022-10-26 20:51:48] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:30 [I 2022-10-26 20:51:48] ...trap/engines/engine.py.229: process: 0.54970 | load: 0.00241 [I 2022-10-26 20:51:48] ...trap/engines/engine.py.230: loss: 0.18799 [I 2022-10-26 20:51:48] ...optimizers/trijoint.py.151: total_norm: 19.650126 [I 2022-10-26 20:51:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 850/2304 [I 2022-10-26 20:51:53] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:25 [I 2022-10-26 20:51:53] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00241 [I 2022-10-26 20:51:53] ...trap/engines/engine.py.230: loss: 0.14304 [I 2022-10-26 20:51:53] ...optimizers/trijoint.py.151: total_norm: 15.192640 [I 2022-10-26 20:51:59] ...trap/engines/engine.py.224: train: epoch 29 | batch 860/2304 [I 2022-10-26 20:51:59] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:18 [I 2022-10-26 20:51:59] ...trap/engines/engine.py.229: process: 0.55557 | load: 0.00244 [I 2022-10-26 20:51:59] ...trap/engines/engine.py.230: loss: 0.17807 [I 2022-10-26 20:51:59] ...optimizers/trijoint.py.151: total_norm: 21.799814 [I 2022-10-26 20:52:04] ...trap/engines/engine.py.224: train: epoch 29 | batch 870/2304 [I 2022-10-26 20:52:04] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:16 [I 2022-10-26 20:52:04] ...trap/engines/engine.py.229: process: 0.55793 | load: 0.00243 [I 2022-10-26 20:52:04] ...trap/engines/engine.py.230: loss: 0.27012 [I 2022-10-26 20:52:04] ...optimizers/trijoint.py.151: total_norm: 34.198467 [I 2022-10-26 20:52:10] ...trap/engines/engine.py.224: train: epoch 29 | batch 880/2304 [I 2022-10-26 20:52:10] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:16 [I 2022-10-26 20:52:10] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00241 [I 2022-10-26 20:52:10] ...trap/engines/engine.py.230: loss: 0.15702 [I 2022-10-26 20:52:10] ...optimizers/trijoint.py.151: total_norm: 14.440018 [I 2022-10-26 20:52:15] ...trap/engines/engine.py.224: train: epoch 29 | batch 890/2304 [I 2022-10-26 20:52:15] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:12 [I 2022-10-26 20:52:15] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00247 [I 2022-10-26 20:52:15] ...trap/engines/engine.py.230: loss: 0.17982 [I 2022-10-26 20:52:15] ...optimizers/trijoint.py.151: total_norm: 23.059317 [I 2022-10-26 20:52:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 900/2304 [I 2022-10-26 20:52:21] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:12:57 [I 2022-10-26 20:52:21] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00250 [I 2022-10-26 20:52:21] ...trap/engines/engine.py.230: loss: 0.17488 [I 2022-10-26 20:52:21] ...optimizers/trijoint.py.151: total_norm: 16.797873 [I 2022-10-26 20:52:27] ...trap/engines/engine.py.224: train: epoch 29 | batch 910/2304 [I 2022-10-26 20:52:27] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:57 [I 2022-10-26 20:52:27] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00246 [I 2022-10-26 20:52:27] ...trap/engines/engine.py.230: loss: 0.15977 [I 2022-10-26 20:52:27] ...optimizers/trijoint.py.151: total_norm: 13.595709 [I 2022-10-26 20:52:32] ...trap/engines/engine.py.224: train: epoch 29 | batch 920/2304 [I 2022-10-26 20:52:32] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:48 [I 2022-10-26 20:52:32] ...trap/engines/engine.py.229: process: 0.55076 | load: 0.00244 [I 2022-10-26 20:52:32] ...trap/engines/engine.py.230: loss: 0.10864 [I 2022-10-26 20:52:32] ...optimizers/trijoint.py.151: total_norm: 11.755482 [I 2022-10-26 20:52:38] ...trap/engines/engine.py.224: train: epoch 29 | batch 930/2304 [I 2022-10-26 20:52:38] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:41 [I 2022-10-26 20:52:38] ...trap/engines/engine.py.229: process: 0.54705 | load: 0.00250 [I 2022-10-26 20:52:38] ...trap/engines/engine.py.230: loss: 0.13543 [I 2022-10-26 20:52:38] ...optimizers/trijoint.py.151: total_norm: 12.325326 [I 2022-10-26 20:52:43] ...trap/engines/engine.py.224: train: epoch 29 | batch 940/2304 [I 2022-10-26 20:52:43] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:46 [I 2022-10-26 20:52:43] ...trap/engines/engine.py.229: process: 0.55895 | load: 0.00242 [I 2022-10-26 20:52:43] ...trap/engines/engine.py.230: loss: 0.16981 [I 2022-10-26 20:52:43] ...optimizers/trijoint.py.151: total_norm: 16.547510 [I 2022-10-26 20:52:49] ...trap/engines/engine.py.224: train: epoch 29 | batch 950/2304 [I 2022-10-26 20:52:49] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:36 [I 2022-10-26 20:52:49] ...trap/engines/engine.py.229: process: 0.56462 | load: 0.00245 [I 2022-10-26 20:52:49] ...trap/engines/engine.py.230: loss: 0.19374 [I 2022-10-26 20:52:49] ...optimizers/trijoint.py.151: total_norm: 20.825447 [I 2022-10-26 20:52:55] ...trap/engines/engine.py.224: train: epoch 29 | batch 960/2304 [I 2022-10-26 20:52:55] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:29 [I 2022-10-26 20:52:55] ...trap/engines/engine.py.229: process: 0.54984 | load: 0.00242 [I 2022-10-26 20:52:55] ...trap/engines/engine.py.230: loss: 0.21438 [I 2022-10-26 20:52:55] ...optimizers/trijoint.py.151: total_norm: 28.289274 [I 2022-10-26 20:53:00] ...trap/engines/engine.py.224: train: epoch 29 | batch 970/2304 [I 2022-10-26 20:53:00] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:25 [I 2022-10-26 20:53:00] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00244 [I 2022-10-26 20:53:00] ...trap/engines/engine.py.230: loss: 0.22563 [I 2022-10-26 20:53:00] ...optimizers/trijoint.py.151: total_norm: 12.336151 [I 2022-10-26 20:53:06] ...trap/engines/engine.py.224: train: epoch 29 | batch 980/2304 [I 2022-10-26 20:53:06] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:13 [I 2022-10-26 20:53:06] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00246 [I 2022-10-26 20:53:06] ...trap/engines/engine.py.230: loss: 0.22320 [I 2022-10-26 20:53:06] ...optimizers/trijoint.py.151: total_norm: 21.127525 [I 2022-10-26 20:53:32] ...trap/engines/engine.py.224: train: epoch 29 | batch 990/2304 [I 2022-10-26 20:53:32] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:48:53 [I 2022-10-26 20:53:32] ...trap/engines/engine.py.229: process: 0.55122 | load: 0.00241 [I 2022-10-26 20:53:32] ...trap/engines/engine.py.230: loss: 0.21474 [I 2022-10-26 20:53:32] ...optimizers/trijoint.py.151: total_norm: 20.933935 [I 2022-10-26 20:53:37] ...trap/engines/engine.py.224: train: epoch 29 | batch 1000/2304 [I 2022-10-26 20:53:37] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:15:57 [I 2022-10-26 20:53:37] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00241 [I 2022-10-26 20:53:37] ...trap/engines/engine.py.230: loss: 0.20393 [I 2022-10-26 20:53:37] ...optimizers/trijoint.py.151: total_norm: 19.648844 [I 2022-10-26 20:53:43] ...trap/engines/engine.py.224: train: epoch 29 | batch 1010/2304 [I 2022-10-26 20:53:43] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:12:21 [I 2022-10-26 20:53:43] ...trap/engines/engine.py.229: process: 0.54667 | load: 0.00240 [I 2022-10-26 20:53:43] ...trap/engines/engine.py.230: loss: 0.17773 [I 2022-10-26 20:53:43] ...optimizers/trijoint.py.151: total_norm: 15.332475 [I 2022-10-26 20:53:48] ...trap/engines/engine.py.224: train: epoch 29 | batch 1020/2304 [I 2022-10-26 20:53:48] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:55 [I 2022-10-26 20:53:48] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00241 [I 2022-10-26 20:53:48] ...trap/engines/engine.py.230: loss: 0.21618 [I 2022-10-26 20:53:48] ...optimizers/trijoint.py.151: total_norm: 13.435577 [I 2022-10-26 20:53:54] ...trap/engines/engine.py.224: train: epoch 29 | batch 1030/2304 [I 2022-10-26 20:53:54] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:51 [I 2022-10-26 20:53:54] ...trap/engines/engine.py.229: process: 0.56575 | load: 0.00240 [I 2022-10-26 20:53:54] ...trap/engines/engine.py.230: loss: 0.25856 [I 2022-10-26 20:53:54] ...optimizers/trijoint.py.151: total_norm: 22.616741 [I 2022-10-26 20:54:00] ...trap/engines/engine.py.224: train: epoch 29 | batch 1040/2304 [I 2022-10-26 20:54:00] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:43 [I 2022-10-26 20:54:00] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00239 [I 2022-10-26 20:54:00] ...trap/engines/engine.py.230: loss: 0.12823 [I 2022-10-26 20:54:00] ...optimizers/trijoint.py.151: total_norm: 15.544879 [I 2022-10-26 20:54:05] ...trap/engines/engine.py.224: train: epoch 29 | batch 1050/2304 [I 2022-10-26 20:54:05] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:37 [I 2022-10-26 20:54:05] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00242 [I 2022-10-26 20:54:05] ...trap/engines/engine.py.230: loss: 0.15459 [I 2022-10-26 20:54:05] ...optimizers/trijoint.py.151: total_norm: 9.264886 [I 2022-10-26 20:54:11] ...trap/engines/engine.py.224: train: epoch 29 | batch 1060/2304 [I 2022-10-26 20:54:11] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:28 [I 2022-10-26 20:54:11] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00241 [I 2022-10-26 20:54:11] ...trap/engines/engine.py.230: loss: 0.16822 [I 2022-10-26 20:54:11] ...optimizers/trijoint.py.151: total_norm: 16.317614 [I 2022-10-26 20:54:16] ...trap/engines/engine.py.224: train: epoch 29 | batch 1070/2304 [I 2022-10-26 20:54:16] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:26 [I 2022-10-26 20:54:16] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00241 [I 2022-10-26 20:54:16] ...trap/engines/engine.py.230: loss: 0.16732 [I 2022-10-26 20:54:16] ...optimizers/trijoint.py.151: total_norm: 11.902567 [I 2022-10-26 20:54:22] ...trap/engines/engine.py.224: train: epoch 29 | batch 1080/2304 [I 2022-10-26 20:54:22] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:23 [I 2022-10-26 20:54:22] ...trap/engines/engine.py.229: process: 0.56258 | load: 0.00242 [I 2022-10-26 20:54:22] ...trap/engines/engine.py.230: loss: 0.13514 [I 2022-10-26 20:54:22] ...optimizers/trijoint.py.151: total_norm: 29.044451 [I 2022-10-26 20:54:27] ...trap/engines/engine.py.224: train: epoch 29 | batch 1090/2304 [I 2022-10-26 20:54:27] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:18 [I 2022-10-26 20:54:27] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00243 [I 2022-10-26 20:54:27] ...trap/engines/engine.py.230: loss: 0.20492 [I 2022-10-26 20:54:27] ...optimizers/trijoint.py.151: total_norm: 20.240957 [I 2022-10-26 20:54:33] ...trap/engines/engine.py.224: train: epoch 29 | batch 1100/2304 [I 2022-10-26 20:54:33] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:13 [I 2022-10-26 20:54:33] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00238 [I 2022-10-26 20:54:33] ...trap/engines/engine.py.230: loss: 0.26068 [I 2022-10-26 20:54:33] ...optimizers/trijoint.py.151: total_norm: 36.644402 [I 2022-10-26 20:54:39] ...trap/engines/engine.py.224: train: epoch 29 | batch 1110/2304 [I 2022-10-26 20:54:39] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:11:03 [I 2022-10-26 20:54:39] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00243 [I 2022-10-26 20:54:39] ...trap/engines/engine.py.230: loss: 0.22928 [I 2022-10-26 20:54:39] ...optimizers/trijoint.py.151: total_norm: 17.653923 [I 2022-10-26 20:54:44] ...trap/engines/engine.py.224: train: epoch 29 | batch 1120/2304 [I 2022-10-26 20:54:44] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:11:02 [I 2022-10-26 20:54:44] ...trap/engines/engine.py.229: process: 0.56464 | load: 0.00241 [I 2022-10-26 20:54:44] ...trap/engines/engine.py.230: loss: 0.24090 [I 2022-10-26 20:54:44] ...optimizers/trijoint.py.151: total_norm: 33.901505 [I 2022-10-26 20:54:50] ...trap/engines/engine.py.224: train: epoch 29 | batch 1130/2304 [I 2022-10-26 20:54:50] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:50 [I 2022-10-26 20:54:50] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00243 [I 2022-10-26 20:54:50] ...trap/engines/engine.py.230: loss: 0.12856 [I 2022-10-26 20:54:50] ...optimizers/trijoint.py.151: total_norm: 9.915666 [I 2022-10-26 20:54:55] ...trap/engines/engine.py.224: train: epoch 29 | batch 1140/2304 [I 2022-10-26 20:54:55] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:44 [I 2022-10-26 20:54:55] ...trap/engines/engine.py.229: process: 0.54834 | load: 0.00242 [I 2022-10-26 20:54:55] ...trap/engines/engine.py.230: loss: 0.20904 [I 2022-10-26 20:54:55] ...optimizers/trijoint.py.151: total_norm: 18.521900 [I 2022-10-26 20:55:01] ...trap/engines/engine.py.224: train: epoch 29 | batch 1150/2304 [I 2022-10-26 20:55:01] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:41 [I 2022-10-26 20:55:01] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00245 [I 2022-10-26 20:55:01] ...trap/engines/engine.py.230: loss: 0.18751 [I 2022-10-26 20:55:01] ...optimizers/trijoint.py.151: total_norm: 16.338146 [I 2022-10-26 20:55:06] ...trap/engines/engine.py.224: train: epoch 29 | batch 1160/2304 [I 2022-10-26 20:55:06] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:32 [I 2022-10-26 20:55:06] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00240 [I 2022-10-26 20:55:06] ...trap/engines/engine.py.230: loss: 0.23300 [I 2022-10-26 20:55:06] ...optimizers/trijoint.py.151: total_norm: 17.359045 [I 2022-10-26 20:55:12] ...trap/engines/engine.py.224: train: epoch 29 | batch 1170/2304 [I 2022-10-26 20:55:12] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:30 [I 2022-10-26 20:55:12] ...trap/engines/engine.py.229: process: 0.54656 | load: 0.00245 [I 2022-10-26 20:55:12] ...trap/engines/engine.py.230: loss: 0.18109 [I 2022-10-26 20:55:12] ...optimizers/trijoint.py.151: total_norm: 15.809679 [I 2022-10-26 20:55:18] ...trap/engines/engine.py.224: train: epoch 29 | batch 1180/2304 [I 2022-10-26 20:55:18] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:27 [I 2022-10-26 20:55:18] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00241 [I 2022-10-26 20:55:18] ...trap/engines/engine.py.230: loss: 0.15393 [I 2022-10-26 20:55:18] ...optimizers/trijoint.py.151: total_norm: 17.077730 [I 2022-10-26 20:55:23] ...trap/engines/engine.py.224: train: epoch 29 | batch 1190/2304 [I 2022-10-26 20:55:23] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:24 [I 2022-10-26 20:55:23] ...trap/engines/engine.py.229: process: 0.56792 | load: 0.00244 [I 2022-10-26 20:55:23] ...trap/engines/engine.py.230: loss: 0.15279 [I 2022-10-26 20:55:23] ...optimizers/trijoint.py.151: total_norm: 22.076149 [I 2022-10-26 20:55:29] ...trap/engines/engine.py.224: train: epoch 29 | batch 1200/2304 [I 2022-10-26 20:55:29] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:11 [I 2022-10-26 20:55:29] ...trap/engines/engine.py.229: process: 0.55147 | load: 0.00243 [I 2022-10-26 20:55:29] ...trap/engines/engine.py.230: loss: 0.18076 [I 2022-10-26 20:55:29] ...optimizers/trijoint.py.151: total_norm: 41.559643 [I 2022-10-26 20:55:34] ...trap/engines/engine.py.224: train: epoch 29 | batch 1210/2304 [I 2022-10-26 20:55:34] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:10:15 [I 2022-10-26 20:55:34] ...trap/engines/engine.py.229: process: 0.56412 | load: 0.00242 [I 2022-10-26 20:55:34] ...trap/engines/engine.py.230: loss: 0.17651 [I 2022-10-26 20:55:34] ...optimizers/trijoint.py.151: total_norm: 18.755833 [I 2022-10-26 20:55:40] ...trap/engines/engine.py.224: train: epoch 29 | batch 1220/2304 [I 2022-10-26 20:55:40] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:10:03 [I 2022-10-26 20:55:40] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00242 [I 2022-10-26 20:55:40] ...trap/engines/engine.py.230: loss: 0.24934 [I 2022-10-26 20:55:40] ...optimizers/trijoint.py.151: total_norm: 19.342979 [I 2022-10-26 20:55:46] ...trap/engines/engine.py.224: train: epoch 29 | batch 1230/2304 [I 2022-10-26 20:55:46] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:10:06 [I 2022-10-26 20:55:46] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00239 [I 2022-10-26 20:55:46] ...trap/engines/engine.py.230: loss: 0.26414 [I 2022-10-26 20:55:46] ...optimizers/trijoint.py.151: total_norm: 20.109404 [I 2022-10-26 20:55:51] ...trap/engines/engine.py.224: train: epoch 29 | batch 1240/2304 [I 2022-10-26 20:55:51] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:53 [I 2022-10-26 20:55:51] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00243 [I 2022-10-26 20:55:51] ...trap/engines/engine.py.230: loss: 0.14724 [I 2022-10-26 20:55:51] ...optimizers/trijoint.py.151: total_norm: 18.911703 [I 2022-10-26 20:55:57] ...trap/engines/engine.py.224: train: epoch 29 | batch 1250/2304 [I 2022-10-26 20:55:57] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:49 [I 2022-10-26 20:55:57] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00243 [I 2022-10-26 20:55:57] ...trap/engines/engine.py.230: loss: 0.17438 [I 2022-10-26 20:55:57] ...optimizers/trijoint.py.151: total_norm: 19.785997 [I 2022-10-26 20:56:02] ...trap/engines/engine.py.224: train: epoch 29 | batch 1260/2304 [I 2022-10-26 20:56:02] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:35 [I 2022-10-26 20:56:02] ...trap/engines/engine.py.229: process: 0.55065 | load: 0.00242 [I 2022-10-26 20:56:02] ...trap/engines/engine.py.230: loss: 0.22001 [I 2022-10-26 20:56:02] ...optimizers/trijoint.py.151: total_norm: 50.820202 [I 2022-10-26 20:56:08] ...trap/engines/engine.py.224: train: epoch 29 | batch 1270/2304 [I 2022-10-26 20:56:08] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:35 [I 2022-10-26 20:56:08] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00248 [I 2022-10-26 20:56:08] ...trap/engines/engine.py.230: loss: 0.21808 [I 2022-10-26 20:56:08] ...optimizers/trijoint.py.151: total_norm: 13.183395 [I 2022-10-26 20:56:14] ...trap/engines/engine.py.224: train: epoch 29 | batch 1280/2304 [I 2022-10-26 20:56:14] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:31 [I 2022-10-26 20:56:14] ...trap/engines/engine.py.229: process: 0.56611 | load: 0.00248 [I 2022-10-26 20:56:14] ...trap/engines/engine.py.230: loss: 0.21745 [I 2022-10-26 20:56:14] ...optimizers/trijoint.py.151: total_norm: 18.659172 [I 2022-10-26 20:56:19] ...trap/engines/engine.py.224: train: epoch 29 | batch 1290/2304 [I 2022-10-26 20:56:19] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:53 [I 2022-10-26 20:56:19] ...trap/engines/engine.py.229: process: 0.69927 | load: 0.00246 [I 2022-10-26 20:56:19] ...trap/engines/engine.py.230: loss: 0.16586 [I 2022-10-26 20:56:19] ...optimizers/trijoint.py.151: total_norm: 9.479332 [I 2022-10-26 20:56:25] ...trap/engines/engine.py.224: train: epoch 29 | batch 1300/2304 [I 2022-10-26 20:56:25] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:20 [I 2022-10-26 20:56:25] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00241 [I 2022-10-26 20:56:25] ...trap/engines/engine.py.230: loss: 0.17451 [I 2022-10-26 20:56:25] ...optimizers/trijoint.py.151: total_norm: 15.960744 [I 2022-10-26 20:56:30] ...trap/engines/engine.py.224: train: epoch 29 | batch 1310/2304 [I 2022-10-26 20:56:30] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:16 [I 2022-10-26 20:56:30] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00243 [I 2022-10-26 20:56:30] ...trap/engines/engine.py.230: loss: 0.18421 [I 2022-10-26 20:56:30] ...optimizers/trijoint.py.151: total_norm: 16.027504 [I 2022-10-26 20:56:36] ...trap/engines/engine.py.224: train: epoch 29 | batch 1320/2304 [I 2022-10-26 20:56:36] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:08 [I 2022-10-26 20:56:36] ...trap/engines/engine.py.229: process: 0.56972 | load: 0.00245 [I 2022-10-26 20:56:36] ...trap/engines/engine.py.230: loss: 0.20949 [I 2022-10-26 20:56:36] ...optimizers/trijoint.py.151: total_norm: 15.033589 [I 2022-10-26 20:56:42] ...trap/engines/engine.py.224: train: epoch 29 | batch 1330/2304 [I 2022-10-26 20:56:42] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:09:02 [I 2022-10-26 20:56:42] ...trap/engines/engine.py.229: process: 0.55140 | load: 0.00241 [I 2022-10-26 20:56:42] ...trap/engines/engine.py.230: loss: 0.16089 [I 2022-10-26 20:56:42] ...optimizers/trijoint.py.151: total_norm: 11.654738 [I 2022-10-26 20:56:47] ...trap/engines/engine.py.224: train: epoch 29 | batch 1340/2304 [I 2022-10-26 20:56:47] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:57 [I 2022-10-26 20:56:47] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00242 [I 2022-10-26 20:56:47] ...trap/engines/engine.py.230: loss: 0.21450 [I 2022-10-26 20:56:47] ...optimizers/trijoint.py.151: total_norm: 10.798818 [I 2022-10-26 20:56:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 1350/2304 [I 2022-10-26 20:56:53] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:53 [I 2022-10-26 20:56:53] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00243 [I 2022-10-26 20:56:53] ...trap/engines/engine.py.230: loss: 0.26133 [I 2022-10-26 20:56:53] ...optimizers/trijoint.py.151: total_norm: 108.020340 [I 2022-10-26 20:56:59] ...trap/engines/engine.py.224: train: epoch 29 | batch 1360/2304 [I 2022-10-26 20:56:59] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:09:02 [I 2022-10-26 20:56:59] ...trap/engines/engine.py.229: process: 0.56055 | load: 0.00259 [I 2022-10-26 20:56:59] ...trap/engines/engine.py.230: loss: 0.24199 [I 2022-10-26 20:56:59] ...optimizers/trijoint.py.151: total_norm: 11.872797 [I 2022-10-26 20:57:04] ...trap/engines/engine.py.224: train: epoch 29 | batch 1370/2304 [I 2022-10-26 20:57:04] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:41 [I 2022-10-26 20:57:04] ...trap/engines/engine.py.229: process: 0.55942 | load: 0.00244 [I 2022-10-26 20:57:04] ...trap/engines/engine.py.230: loss: 0.16806 [I 2022-10-26 20:57:04] ...optimizers/trijoint.py.151: total_norm: 15.478642 [I 2022-10-26 20:57:10] ...trap/engines/engine.py.224: train: epoch 29 | batch 1380/2304 [I 2022-10-26 20:57:10] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:34 [I 2022-10-26 20:57:10] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00244 [I 2022-10-26 20:57:10] ...trap/engines/engine.py.230: loss: 0.13508 [I 2022-10-26 20:57:10] ...optimizers/trijoint.py.151: total_norm: 16.913780 [I 2022-10-26 20:57:15] ...trap/engines/engine.py.224: train: epoch 29 | batch 1390/2304 [I 2022-10-26 20:57:15] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:27 [I 2022-10-26 20:57:15] ...trap/engines/engine.py.229: process: 0.55171 | load: 0.00242 [I 2022-10-26 20:57:15] ...trap/engines/engine.py.230: loss: 0.16694 [I 2022-10-26 20:57:15] ...optimizers/trijoint.py.151: total_norm: 11.966119 [I 2022-10-26 20:57:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 1400/2304 [I 2022-10-26 20:57:21] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:19 [I 2022-10-26 20:57:21] ...trap/engines/engine.py.229: process: 0.54741 | load: 0.00242 [I 2022-10-26 20:57:21] ...trap/engines/engine.py.230: loss: 0.19517 [I 2022-10-26 20:57:21] ...optimizers/trijoint.py.151: total_norm: 16.516775 [I 2022-10-26 20:57:26] ...trap/engines/engine.py.224: train: epoch 29 | batch 1410/2304 [I 2022-10-26 20:57:26] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:15 [I 2022-10-26 20:57:26] ...trap/engines/engine.py.229: process: 0.55641 | load: 0.00243 [I 2022-10-26 20:57:26] ...trap/engines/engine.py.230: loss: 0.17309 [I 2022-10-26 20:57:26] ...optimizers/trijoint.py.151: total_norm: 23.282080 [I 2022-10-26 20:57:32] ...trap/engines/engine.py.224: train: epoch 29 | batch 1420/2304 [I 2022-10-26 20:57:32] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:12 [I 2022-10-26 20:57:32] ...trap/engines/engine.py.229: process: 0.56616 | load: 0.00241 [I 2022-10-26 20:57:32] ...trap/engines/engine.py.230: loss: 0.25602 [I 2022-10-26 20:57:32] ...optimizers/trijoint.py.151: total_norm: 24.438625 [I 2022-10-26 20:57:37] ...trap/engines/engine.py.224: train: epoch 29 | batch 1430/2304 [I 2022-10-26 20:57:37] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:08:07 [I 2022-10-26 20:57:37] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00242 [I 2022-10-26 20:57:37] ...trap/engines/engine.py.230: loss: 0.27466 [I 2022-10-26 20:57:37] ...optimizers/trijoint.py.151: total_norm: 20.848974 [I 2022-10-26 20:57:43] ...trap/engines/engine.py.224: train: epoch 29 | batch 1440/2304 [I 2022-10-26 20:57:43] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:08:00 [I 2022-10-26 20:57:43] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00242 [I 2022-10-26 20:57:43] ...trap/engines/engine.py.230: loss: 0.17985 [I 2022-10-26 20:57:43] ...optimizers/trijoint.py.151: total_norm: 15.897954 [I 2022-10-26 20:57:49] ...trap/engines/engine.py.224: train: epoch 29 | batch 1450/2304 [I 2022-10-26 20:57:49] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:53 [I 2022-10-26 20:57:49] ...trap/engines/engine.py.229: process: 0.55260 | load: 0.00239 [I 2022-10-26 20:57:49] ...trap/engines/engine.py.230: loss: 0.17352 [I 2022-10-26 20:57:49] ...optimizers/trijoint.py.151: total_norm: 20.786829 [I 2022-10-26 20:57:54] ...trap/engines/engine.py.224: train: epoch 29 | batch 1460/2304 [I 2022-10-26 20:57:54] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:46 [I 2022-10-26 20:57:54] ...trap/engines/engine.py.229: process: 0.55185 | load: 0.00242 [I 2022-10-26 20:57:54] ...trap/engines/engine.py.230: loss: 0.19703 [I 2022-10-26 20:57:54] ...optimizers/trijoint.py.151: total_norm: 39.946213 [I 2022-10-26 20:58:00] ...trap/engines/engine.py.224: train: epoch 29 | batch 1470/2304 [I 2022-10-26 20:58:00] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:42 [I 2022-10-26 20:58:00] ...trap/engines/engine.py.229: process: 0.55301 | load: 0.00242 [I 2022-10-26 20:58:00] ...trap/engines/engine.py.230: loss: 0.27172 [I 2022-10-26 20:58:00] ...optimizers/trijoint.py.151: total_norm: 21.587151 [I 2022-10-26 20:58:05] ...trap/engines/engine.py.224: train: epoch 29 | batch 1480/2304 [I 2022-10-26 20:58:05] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:43 [I 2022-10-26 20:58:05] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00242 [I 2022-10-26 20:58:05] ...trap/engines/engine.py.230: loss: 0.14475 [I 2022-10-26 20:58:05] ...optimizers/trijoint.py.151: total_norm: 20.820995 [I 2022-10-26 20:58:11] ...trap/engines/engine.py.224: train: epoch 29 | batch 1490/2304 [I 2022-10-26 20:58:11] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:34 [I 2022-10-26 20:58:11] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00519 [I 2022-10-26 20:58:11] ...trap/engines/engine.py.230: loss: 0.18883 [I 2022-10-26 20:58:11] ...optimizers/trijoint.py.151: total_norm: 23.810541 [I 2022-10-26 20:58:16] ...trap/engines/engine.py.224: train: epoch 29 | batch 1500/2304 [I 2022-10-26 20:58:16] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:26 [I 2022-10-26 20:58:16] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00244 [I 2022-10-26 20:58:16] ...trap/engines/engine.py.230: loss: 0.16888 [I 2022-10-26 20:58:16] ...optimizers/trijoint.py.151: total_norm: 18.301960 [I 2022-10-26 20:58:22] ...trap/engines/engine.py.224: train: epoch 29 | batch 1510/2304 [I 2022-10-26 20:58:22] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:23 [I 2022-10-26 20:58:22] ...trap/engines/engine.py.229: process: 0.56522 | load: 0.00242 [I 2022-10-26 20:58:22] ...trap/engines/engine.py.230: loss: 0.19202 [I 2022-10-26 20:58:22] ...optimizers/trijoint.py.151: total_norm: 36.031479 [I 2022-10-26 20:58:28] ...trap/engines/engine.py.224: train: epoch 29 | batch 1520/2304 [I 2022-10-26 20:58:28] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:16 [I 2022-10-26 20:58:28] ...trap/engines/engine.py.229: process: 0.54976 | load: 0.00239 [I 2022-10-26 20:58:28] ...trap/engines/engine.py.230: loss: 0.14301 [I 2022-10-26 20:58:28] ...optimizers/trijoint.py.151: total_norm: 19.249786 [I 2022-10-26 20:58:33] ...trap/engines/engine.py.224: train: epoch 29 | batch 1530/2304 [I 2022-10-26 20:58:33] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:07:09 [I 2022-10-26 20:58:33] ...trap/engines/engine.py.229: process: 0.55168 | load: 0.00252 [I 2022-10-26 20:58:33] ...trap/engines/engine.py.230: loss: 0.18104 [I 2022-10-26 20:58:33] ...optimizers/trijoint.py.151: total_norm: 25.245220 [I 2022-10-26 20:58:39] ...trap/engines/engine.py.224: train: epoch 29 | batch 1540/2304 [I 2022-10-26 20:58:39] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:07:05 [I 2022-10-26 20:58:39] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00245 [I 2022-10-26 20:58:39] ...trap/engines/engine.py.230: loss: 0.12671 [I 2022-10-26 20:58:39] ...optimizers/trijoint.py.151: total_norm: 31.895741 [I 2022-10-26 20:58:44] ...trap/engines/engine.py.224: train: epoch 29 | batch 1550/2304 [I 2022-10-26 20:58:44] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:07:00 [I 2022-10-26 20:58:44] ...trap/engines/engine.py.229: process: 0.56164 | load: 0.00245 [I 2022-10-26 20:58:44] ...trap/engines/engine.py.230: loss: 0.18518 [I 2022-10-26 20:58:44] ...optimizers/trijoint.py.151: total_norm: 20.527714 [I 2022-10-26 20:58:50] ...trap/engines/engine.py.224: train: epoch 29 | batch 1560/2304 [I 2022-10-26 20:58:50] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:51 [I 2022-10-26 20:58:50] ...trap/engines/engine.py.229: process: 0.54731 | load: 0.00244 [I 2022-10-26 20:58:50] ...trap/engines/engine.py.230: loss: 0.15838 [I 2022-10-26 20:58:50] ...optimizers/trijoint.py.151: total_norm: 24.849220 [I 2022-10-26 20:58:56] ...trap/engines/engine.py.224: train: epoch 29 | batch 1570/2304 [I 2022-10-26 20:58:56] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:50 [I 2022-10-26 20:58:56] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00241 [I 2022-10-26 20:58:56] ...trap/engines/engine.py.230: loss: 0.16955 [I 2022-10-26 20:58:56] ...optimizers/trijoint.py.151: total_norm: 31.569521 [I 2022-10-26 20:59:01] ...trap/engines/engine.py.224: train: epoch 29 | batch 1580/2304 [I 2022-10-26 20:59:01] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:39 [I 2022-10-26 20:59:01] ...trap/engines/engine.py.229: process: 0.54186 | load: 0.00243 [I 2022-10-26 20:59:01] ...trap/engines/engine.py.230: loss: 0.15871 [I 2022-10-26 20:59:01] ...optimizers/trijoint.py.151: total_norm: 11.720922 [I 2022-10-26 20:59:07] ...trap/engines/engine.py.224: train: epoch 29 | batch 1590/2304 [I 2022-10-26 20:59:07] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:35 [I 2022-10-26 20:59:07] ...trap/engines/engine.py.229: process: 0.55302 | load: 0.00242 [I 2022-10-26 20:59:07] ...trap/engines/engine.py.230: loss: 0.18263 [I 2022-10-26 20:59:07] ...optimizers/trijoint.py.151: total_norm: 14.292596 [I 2022-10-26 20:59:12] ...trap/engines/engine.py.224: train: epoch 29 | batch 1600/2304 [I 2022-10-26 20:59:12] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:45 [I 2022-10-26 20:59:12] ...trap/engines/engine.py.229: process: 0.56159 | load: 0.00240 [I 2022-10-26 20:59:12] ...trap/engines/engine.py.230: loss: 0.11095 [I 2022-10-26 20:59:12] ...optimizers/trijoint.py.151: total_norm: 51.128040 [I 2022-10-26 20:59:18] ...trap/engines/engine.py.224: train: epoch 29 | batch 1610/2304 [I 2022-10-26 20:59:18] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:26 [I 2022-10-26 20:59:18] ...trap/engines/engine.py.229: process: 0.54747 | load: 0.00246 [I 2022-10-26 20:59:18] ...trap/engines/engine.py.230: loss: 0.17167 [I 2022-10-26 20:59:18] ...optimizers/trijoint.py.151: total_norm: 18.940626 [I 2022-10-26 20:59:23] ...trap/engines/engine.py.224: train: epoch 29 | batch 1620/2304 [I 2022-10-26 20:59:23] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:20 [I 2022-10-26 20:59:23] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00241 [I 2022-10-26 20:59:23] ...trap/engines/engine.py.230: loss: 0.15951 [I 2022-10-26 20:59:23] ...optimizers/trijoint.py.151: total_norm: 10.416408 [I 2022-10-26 20:59:29] ...trap/engines/engine.py.224: train: epoch 29 | batch 1630/2304 [I 2022-10-26 20:59:29] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:16 [I 2022-10-26 20:59:29] ...trap/engines/engine.py.229: process: 0.56639 | load: 0.00246 [I 2022-10-26 20:59:29] ...trap/engines/engine.py.230: loss: 0.18891 [I 2022-10-26 20:59:29] ...optimizers/trijoint.py.151: total_norm: 15.359642 [I 2022-10-26 20:59:35] ...trap/engines/engine.py.224: train: epoch 29 | batch 1640/2304 [I 2022-10-26 20:59:35] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:06:10 [I 2022-10-26 20:59:35] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00243 [I 2022-10-26 20:59:35] ...trap/engines/engine.py.230: loss: 0.23485 [I 2022-10-26 20:59:35] ...optimizers/trijoint.py.151: total_norm: 21.862076 [I 2022-10-26 20:59:40] ...trap/engines/engine.py.224: train: epoch 29 | batch 1650/2304 [I 2022-10-26 20:59:40] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:06:05 [I 2022-10-26 20:59:40] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00241 [I 2022-10-26 20:59:40] ...trap/engines/engine.py.230: loss: 0.23643 [I 2022-10-26 20:59:40] ...optimizers/trijoint.py.151: total_norm: 24.618422 [I 2022-10-26 20:59:46] ...trap/engines/engine.py.224: train: epoch 29 | batch 1660/2304 [I 2022-10-26 20:59:46] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:05:58 [I 2022-10-26 20:59:46] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00242 [I 2022-10-26 20:59:46] ...trap/engines/engine.py.230: loss: 0.09910 [I 2022-10-26 20:59:46] ...optimizers/trijoint.py.151: total_norm: 9.493435 [I 2022-10-26 20:59:51] ...trap/engines/engine.py.224: train: epoch 29 | batch 1670/2304 [I 2022-10-26 20:59:51] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:52 [I 2022-10-26 20:59:51] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00241 [I 2022-10-26 20:59:51] ...trap/engines/engine.py.230: loss: 0.16603 [I 2022-10-26 20:59:51] ...optimizers/trijoint.py.151: total_norm: 16.175245 [I 2022-10-26 20:59:57] ...trap/engines/engine.py.224: train: epoch 29 | batch 1680/2304 [I 2022-10-26 20:59:57] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:49 [I 2022-10-26 20:59:57] ...trap/engines/engine.py.229: process: 0.56667 | load: 0.00297 [I 2022-10-26 20:59:57] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-26 20:59:57] ...optimizers/trijoint.py.151: total_norm: 25.189249 [I 2022-10-26 21:00:03] ...trap/engines/engine.py.224: train: epoch 29 | batch 1690/2304 [I 2022-10-26 21:00:03] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:43 [I 2022-10-26 21:00:03] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00246 [I 2022-10-26 21:00:03] ...trap/engines/engine.py.230: loss: 0.15708 [I 2022-10-26 21:00:03] ...optimizers/trijoint.py.151: total_norm: 14.839118 [I 2022-10-26 21:00:08] ...trap/engines/engine.py.224: train: epoch 29 | batch 1700/2304 [I 2022-10-26 21:00:08] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:36 [I 2022-10-26 21:00:08] ...trap/engines/engine.py.229: process: 0.54874 | load: 0.00253 [I 2022-10-26 21:00:08] ...trap/engines/engine.py.230: loss: 0.18620 [I 2022-10-26 21:00:08] ...optimizers/trijoint.py.151: total_norm: 28.850857 [I 2022-10-26 21:00:14] ...trap/engines/engine.py.224: train: epoch 29 | batch 1710/2304 [I 2022-10-26 21:00:14] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:31 [I 2022-10-26 21:00:14] ...trap/engines/engine.py.229: process: 0.55999 | load: 0.00249 [I 2022-10-26 21:00:14] ...trap/engines/engine.py.230: loss: 0.14067 [I 2022-10-26 21:00:14] ...optimizers/trijoint.py.151: total_norm: 34.548134 [I 2022-10-26 21:00:19] ...trap/engines/engine.py.224: train: epoch 29 | batch 1720/2304 [I 2022-10-26 21:00:19] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:34 [I 2022-10-26 21:00:19] ...trap/engines/engine.py.229: process: 0.56426 | load: 0.00246 [I 2022-10-26 21:00:19] ...trap/engines/engine.py.230: loss: 0.15132 [I 2022-10-26 21:00:19] ...optimizers/trijoint.py.151: total_norm: 13.653970 [I 2022-10-26 21:00:25] ...trap/engines/engine.py.224: train: epoch 29 | batch 1730/2304 [I 2022-10-26 21:00:25] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:21 [I 2022-10-26 21:00:25] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00243 [I 2022-10-26 21:00:25] ...trap/engines/engine.py.230: loss: 0.18828 [I 2022-10-26 21:00:25] ...optimizers/trijoint.py.151: total_norm: 23.477304 [I 2022-10-26 21:00:31] ...trap/engines/engine.py.224: train: epoch 29 | batch 1740/2304 [I 2022-10-26 21:00:31] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:05:16 [I 2022-10-26 21:00:31] ...trap/engines/engine.py.229: process: 0.56766 | load: 0.00242 [I 2022-10-26 21:00:31] ...trap/engines/engine.py.230: loss: 0.15700 [I 2022-10-26 21:00:31] ...optimizers/trijoint.py.151: total_norm: 26.289356 [I 2022-10-26 21:00:36] ...trap/engines/engine.py.224: train: epoch 29 | batch 1750/2304 [I 2022-10-26 21:00:36] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:09 [I 2022-10-26 21:00:36] ...trap/engines/engine.py.229: process: 0.55786 | load: 0.00244 [I 2022-10-26 21:00:36] ...trap/engines/engine.py.230: loss: 0.14988 [I 2022-10-26 21:00:36] ...optimizers/trijoint.py.151: total_norm: 10.420969 [I 2022-10-26 21:00:42] ...trap/engines/engine.py.224: train: epoch 29 | batch 1760/2304 [I 2022-10-26 21:00:42] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:05:02 [I 2022-10-26 21:00:42] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00244 [I 2022-10-26 21:00:42] ...trap/engines/engine.py.230: loss: 0.14870 [I 2022-10-26 21:00:42] ...optimizers/trijoint.py.151: total_norm: 10.618742 [I 2022-10-26 21:00:47] ...trap/engines/engine.py.224: train: epoch 29 | batch 1770/2304 [I 2022-10-26 21:00:47] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:58 [I 2022-10-26 21:00:47] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00244 [I 2022-10-26 21:00:47] ...trap/engines/engine.py.230: loss: 0.21188 [I 2022-10-26 21:00:47] ...optimizers/trijoint.py.151: total_norm: 34.661999 [I 2022-10-26 21:00:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 1780/2304 [I 2022-10-26 21:00:53] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:51 [I 2022-10-26 21:00:53] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00242 [I 2022-10-26 21:00:53] ...trap/engines/engine.py.230: loss: 0.21843 [I 2022-10-26 21:00:53] ...optimizers/trijoint.py.151: total_norm: 22.149563 [I 2022-10-26 21:00:59] ...trap/engines/engine.py.224: train: epoch 29 | batch 1790/2304 [I 2022-10-26 21:00:59] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:04:45 [I 2022-10-26 21:00:59] ...trap/engines/engine.py.229: process: 0.55480 | load: 0.00242 [I 2022-10-26 21:00:59] ...trap/engines/engine.py.230: loss: 0.23622 [I 2022-10-26 21:00:59] ...optimizers/trijoint.py.151: total_norm: 15.811375 [I 2022-10-26 21:01:04] ...trap/engines/engine.py.224: train: epoch 29 | batch 1800/2304 [I 2022-10-26 21:01:04] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:38 [I 2022-10-26 21:01:04] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00243 [I 2022-10-26 21:01:04] ...trap/engines/engine.py.230: loss: 0.21116 [I 2022-10-26 21:01:04] ...optimizers/trijoint.py.151: total_norm: 14.365095 [I 2022-10-26 21:01:10] ...trap/engines/engine.py.224: train: epoch 29 | batch 1810/2304 [I 2022-10-26 21:01:10] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:35 [I 2022-10-26 21:01:10] ...trap/engines/engine.py.229: process: 0.56188 | load: 0.00247 [I 2022-10-26 21:01:10] ...trap/engines/engine.py.230: loss: 0.24553 [I 2022-10-26 21:01:10] ...optimizers/trijoint.py.151: total_norm: 17.269621 [I 2022-10-26 21:01:15] ...trap/engines/engine.py.224: train: epoch 29 | batch 1820/2304 [I 2022-10-26 21:01:15] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:28 [I 2022-10-26 21:01:15] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00244 [I 2022-10-26 21:01:15] ...trap/engines/engine.py.230: loss: 0.26904 [I 2022-10-26 21:01:15] ...optimizers/trijoint.py.151: total_norm: 12.941249 [I 2022-10-26 21:01:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 1830/2304 [I 2022-10-26 21:01:21] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:22 [I 2022-10-26 21:01:21] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00244 [I 2022-10-26 21:01:21] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-26 21:01:21] ...optimizers/trijoint.py.151: total_norm: 16.956068 [I 2022-10-26 21:01:26] ...trap/engines/engine.py.224: train: epoch 29 | batch 1840/2304 [I 2022-10-26 21:01:26] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:04:18 [I 2022-10-26 21:01:26] ...trap/engines/engine.py.229: process: 0.55496 | load: 0.00246 [I 2022-10-26 21:01:26] ...trap/engines/engine.py.230: loss: 0.17156 [I 2022-10-26 21:01:26] ...optimizers/trijoint.py.151: total_norm: 14.686405 [I 2022-10-26 21:01:32] ...trap/engines/engine.py.224: train: epoch 29 | batch 1850/2304 [I 2022-10-26 21:01:32] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:04:12 [I 2022-10-26 21:01:32] ...trap/engines/engine.py.229: process: 0.55305 | load: 0.00243 [I 2022-10-26 21:01:32] ...trap/engines/engine.py.230: loss: 0.12787 [I 2022-10-26 21:01:32] ...optimizers/trijoint.py.151: total_norm: 19.513643 [I 2022-10-26 21:01:38] ...trap/engines/engine.py.224: train: epoch 29 | batch 1860/2304 [I 2022-10-26 21:01:38] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:04:09 [I 2022-10-26 21:01:38] ...trap/engines/engine.py.229: process: 0.57216 | load: 0.00247 [I 2022-10-26 21:01:38] ...trap/engines/engine.py.230: loss: 0.12312 [I 2022-10-26 21:01:38] ...optimizers/trijoint.py.151: total_norm: 10.148393 [I 2022-10-26 21:01:43] ...trap/engines/engine.py.224: train: epoch 29 | batch 1870/2304 [I 2022-10-26 21:01:43] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:04:02 [I 2022-10-26 21:01:43] ...trap/engines/engine.py.229: process: 0.57125 | load: 0.00244 [I 2022-10-26 21:01:43] ...trap/engines/engine.py.230: loss: 0.19004 [I 2022-10-26 21:01:43] ...optimizers/trijoint.py.151: total_norm: 22.553356 [I 2022-10-26 21:01:49] ...trap/engines/engine.py.224: train: epoch 29 | batch 1880/2304 [I 2022-10-26 21:01:49] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:57 [I 2022-10-26 21:01:49] ...trap/engines/engine.py.229: process: 0.55162 | load: 0.00240 [I 2022-10-26 21:01:49] ...trap/engines/engine.py.230: loss: 0.15589 [I 2022-10-26 21:01:49] ...optimizers/trijoint.py.151: total_norm: 11.362788 [I 2022-10-26 21:01:54] ...trap/engines/engine.py.224: train: epoch 29 | batch 1890/2304 [I 2022-10-26 21:01:54] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:50 [I 2022-10-26 21:01:54] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00246 [I 2022-10-26 21:01:54] ...trap/engines/engine.py.230: loss: 0.12854 [I 2022-10-26 21:01:54] ...optimizers/trijoint.py.151: total_norm: 14.630114 [I 2022-10-26 21:02:00] ...trap/engines/engine.py.224: train: epoch 29 | batch 1900/2304 [I 2022-10-26 21:02:00] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:44 [I 2022-10-26 21:02:00] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00242 [I 2022-10-26 21:02:00] ...trap/engines/engine.py.230: loss: 0.16458 [I 2022-10-26 21:02:00] ...optimizers/trijoint.py.151: total_norm: 17.226505 [I 2022-10-26 21:02:06] ...trap/engines/engine.py.224: train: epoch 29 | batch 1910/2304 [I 2022-10-26 21:02:06] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:03:39 [I 2022-10-26 21:02:06] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00260 [I 2022-10-26 21:02:06] ...trap/engines/engine.py.230: loss: 0.18700 [I 2022-10-26 21:02:06] ...optimizers/trijoint.py.151: total_norm: 21.326241 [I 2022-10-26 21:02:11] ...trap/engines/engine.py.224: train: epoch 29 | batch 1920/2304 [I 2022-10-26 21:02:11] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:32 [I 2022-10-26 21:02:11] ...trap/engines/engine.py.229: process: 0.56168 | load: 0.00257 [I 2022-10-26 21:02:11] ...trap/engines/engine.py.230: loss: 0.21652 [I 2022-10-26 21:02:11] ...optimizers/trijoint.py.151: total_norm: 16.439400 [I 2022-10-26 21:02:17] ...trap/engines/engine.py.224: train: epoch 29 | batch 1930/2304 [I 2022-10-26 21:02:17] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:27 [I 2022-10-26 21:02:17] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00255 [I 2022-10-26 21:02:17] ...trap/engines/engine.py.230: loss: 0.21736 [I 2022-10-26 21:02:17] ...optimizers/trijoint.py.151: total_norm: 23.400888 [I 2022-10-26 21:02:22] ...trap/engines/engine.py.224: train: epoch 29 | batch 1940/2304 [I 2022-10-26 21:02:22] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:22 [I 2022-10-26 21:02:22] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00241 [I 2022-10-26 21:02:22] ...trap/engines/engine.py.230: loss: 0.16005 [I 2022-10-26 21:02:22] ...optimizers/trijoint.py.151: total_norm: 11.178885 [I 2022-10-26 21:02:28] ...trap/engines/engine.py.224: train: epoch 29 | batch 1950/2304 [I 2022-10-26 21:02:28] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:16 [I 2022-10-26 21:02:28] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00242 [I 2022-10-26 21:02:28] ...trap/engines/engine.py.230: loss: 0.16318 [I 2022-10-26 21:02:28] ...optimizers/trijoint.py.151: total_norm: 14.949751 [I 2022-10-26 21:02:33] ...trap/engines/engine.py.224: train: epoch 29 | batch 1960/2304 [I 2022-10-26 21:02:33] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:03:11 [I 2022-10-26 21:02:33] ...trap/engines/engine.py.229: process: 0.54839 | load: 0.00240 [I 2022-10-26 21:02:33] ...trap/engines/engine.py.230: loss: 0.22743 [I 2022-10-26 21:02:33] ...optimizers/trijoint.py.151: total_norm: 15.166565 [I 2022-10-26 21:02:39] ...trap/engines/engine.py.224: train: epoch 29 | batch 1970/2304 [I 2022-10-26 21:02:39] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:03:04 [I 2022-10-26 21:02:39] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00245 [I 2022-10-26 21:02:39] ...trap/engines/engine.py.230: loss: 0.26356 [I 2022-10-26 21:02:39] ...optimizers/trijoint.py.151: total_norm: 20.494528 [I 2022-10-26 21:02:45] ...trap/engines/engine.py.224: train: epoch 29 | batch 1980/2304 [I 2022-10-26 21:02:45] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:03:00 [I 2022-10-26 21:02:45] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00244 [I 2022-10-26 21:02:45] ...trap/engines/engine.py.230: loss: 0.13445 [I 2022-10-26 21:02:45] ...optimizers/trijoint.py.151: total_norm: 21.821434 [I 2022-10-26 21:02:50] ...trap/engines/engine.py.224: train: epoch 29 | batch 1990/2304 [I 2022-10-26 21:02:50] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:54 [I 2022-10-26 21:02:50] ...trap/engines/engine.py.229: process: 0.56328 | load: 0.00245 [I 2022-10-26 21:02:50] ...trap/engines/engine.py.230: loss: 0.16735 [I 2022-10-26 21:02:50] ...optimizers/trijoint.py.151: total_norm: 18.386728 [I 2022-10-26 21:02:56] ...trap/engines/engine.py.224: train: epoch 29 | batch 2000/2304 [I 2022-10-26 21:02:56] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:49 [I 2022-10-26 21:02:56] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00243 [I 2022-10-26 21:02:56] ...trap/engines/engine.py.230: loss: 0.21905 [I 2022-10-26 21:02:56] ...optimizers/trijoint.py.151: total_norm: 17.490622 [I 2022-10-26 21:03:01] ...trap/engines/engine.py.224: train: epoch 29 | batch 2010/2304 [I 2022-10-26 21:03:01] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:02:43 [I 2022-10-26 21:03:01] ...trap/engines/engine.py.229: process: 0.55034 | load: 0.00241 [I 2022-10-26 21:03:01] ...trap/engines/engine.py.230: loss: 0.24678 [I 2022-10-26 21:03:01] ...optimizers/trijoint.py.151: total_norm: 21.951279 [I 2022-10-26 21:03:07] ...trap/engines/engine.py.224: train: epoch 29 | batch 2020/2304 [I 2022-10-26 21:03:07] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:38 [I 2022-10-26 21:03:07] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00240 [I 2022-10-26 21:03:07] ...trap/engines/engine.py.230: loss: 0.17970 [I 2022-10-26 21:03:07] ...optimizers/trijoint.py.151: total_norm: 27.488924 [I 2022-10-26 21:03:12] ...trap/engines/engine.py.224: train: epoch 29 | batch 2030/2304 [I 2022-10-26 21:03:12] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:31 [I 2022-10-26 21:03:12] ...trap/engines/engine.py.229: process: 0.54893 | load: 0.00243 [I 2022-10-26 21:03:12] ...trap/engines/engine.py.230: loss: 0.14137 [I 2022-10-26 21:03:12] ...optimizers/trijoint.py.151: total_norm: 17.654831 [I 2022-10-26 21:03:18] ...trap/engines/engine.py.224: train: epoch 29 | batch 2040/2304 [I 2022-10-26 21:03:18] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:28 [I 2022-10-26 21:03:18] ...trap/engines/engine.py.229: process: 0.56911 | load: 0.00238 [I 2022-10-26 21:03:18] ...trap/engines/engine.py.230: loss: 0.21835 [I 2022-10-26 21:03:18] ...optimizers/trijoint.py.151: total_norm: 12.890643 [I 2022-10-26 21:03:24] ...trap/engines/engine.py.224: train: epoch 29 | batch 2050/2304 [I 2022-10-26 21:03:24] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:21 [I 2022-10-26 21:03:24] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00241 [I 2022-10-26 21:03:24] ...trap/engines/engine.py.230: loss: 0.21646 [I 2022-10-26 21:03:24] ...optimizers/trijoint.py.151: total_norm: 15.278624 [I 2022-10-26 21:03:29] ...trap/engines/engine.py.224: train: epoch 29 | batch 2060/2304 [I 2022-10-26 21:03:29] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:14 [I 2022-10-26 21:03:29] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00242 [I 2022-10-26 21:03:29] ...trap/engines/engine.py.230: loss: 0.20997 [I 2022-10-26 21:03:29] ...optimizers/trijoint.py.151: total_norm: 20.876690 [I 2022-10-26 21:03:35] ...trap/engines/engine.py.224: train: epoch 29 | batch 2070/2304 [I 2022-10-26 21:03:35] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:02:11 [I 2022-10-26 21:03:35] ...trap/engines/engine.py.229: process: 0.56344 | load: 0.00256 [I 2022-10-26 21:03:35] ...trap/engines/engine.py.230: loss: 0.18211 [I 2022-10-26 21:03:35] ...optimizers/trijoint.py.151: total_norm: 19.915497 [I 2022-10-26 21:03:40] ...trap/engines/engine.py.224: train: epoch 29 | batch 2080/2304 [I 2022-10-26 21:03:40] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:02:06 [I 2022-10-26 21:03:40] ...trap/engines/engine.py.229: process: 0.56624 | load: 0.00260 [I 2022-10-26 21:03:40] ...trap/engines/engine.py.230: loss: 0.17581 [I 2022-10-26 21:03:40] ...optimizers/trijoint.py.151: total_norm: 15.145433 [I 2022-10-26 21:03:46] ...trap/engines/engine.py.224: train: epoch 29 | batch 2090/2304 [I 2022-10-26 21:03:46] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:59 [I 2022-10-26 21:03:46] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00251 [I 2022-10-26 21:03:46] ...trap/engines/engine.py.230: loss: 0.28165 [I 2022-10-26 21:03:46] ...optimizers/trijoint.py.151: total_norm: 32.549515 [I 2022-10-26 21:03:51] ...trap/engines/engine.py.224: train: epoch 29 | batch 2100/2304 [I 2022-10-26 21:03:51] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:53 [I 2022-10-26 21:03:51] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00252 [I 2022-10-26 21:03:51] ...trap/engines/engine.py.230: loss: 0.19319 [I 2022-10-26 21:03:51] ...optimizers/trijoint.py.151: total_norm: 19.682207 [I 2022-10-26 21:03:57] ...trap/engines/engine.py.224: train: epoch 29 | batch 2110/2304 [I 2022-10-26 21:03:57] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:47 [I 2022-10-26 21:03:57] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00255 [I 2022-10-26 21:03:57] ...trap/engines/engine.py.230: loss: 0.14480 [I 2022-10-26 21:03:57] ...optimizers/trijoint.py.151: total_norm: 11.808886 [I 2022-10-26 21:04:03] ...trap/engines/engine.py.224: train: epoch 29 | batch 2120/2304 [I 2022-10-26 21:04:03] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:42 [I 2022-10-26 21:04:03] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00253 [I 2022-10-26 21:04:03] ...trap/engines/engine.py.230: loss: 0.13698 [I 2022-10-26 21:04:03] ...optimizers/trijoint.py.151: total_norm: 13.699746 [I 2022-10-26 21:04:08] ...trap/engines/engine.py.224: train: epoch 29 | batch 2130/2304 [I 2022-10-26 21:04:08] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:36 [I 2022-10-26 21:04:08] ...trap/engines/engine.py.229: process: 0.54488 | load: 0.00249 [I 2022-10-26 21:04:08] ...trap/engines/engine.py.230: loss: 0.14836 [I 2022-10-26 21:04:08] ...optimizers/trijoint.py.151: total_norm: 18.874573 [I 2022-10-26 21:04:14] ...trap/engines/engine.py.224: train: epoch 29 | batch 2140/2304 [I 2022-10-26 21:04:14] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:31 [I 2022-10-26 21:04:14] ...trap/engines/engine.py.229: process: 0.56865 | load: 0.00249 [I 2022-10-26 21:04:14] ...trap/engines/engine.py.230: loss: 0.21710 [I 2022-10-26 21:04:14] ...optimizers/trijoint.py.151: total_norm: 11.842000 [I 2022-10-26 21:04:19] ...trap/engines/engine.py.224: train: epoch 29 | batch 2150/2304 [I 2022-10-26 21:04:19] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:25 [I 2022-10-26 21:04:19] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00243 [I 2022-10-26 21:04:19] ...trap/engines/engine.py.230: loss: 0.13825 [I 2022-10-26 21:04:19] ...optimizers/trijoint.py.151: total_norm: 12.963518 [I 2022-10-26 21:04:25] ...trap/engines/engine.py.224: train: epoch 29 | batch 2160/2304 [I 2022-10-26 21:04:25] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:20 [I 2022-10-26 21:04:25] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00248 [I 2022-10-26 21:04:25] ...trap/engines/engine.py.230: loss: 0.13805 [I 2022-10-26 21:04:25] ...optimizers/trijoint.py.151: total_norm: 15.092016 [I 2022-10-26 21:04:31] ...trap/engines/engine.py.224: train: epoch 29 | batch 2170/2304 [I 2022-10-26 21:04:31] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:01:15 [I 2022-10-26 21:04:31] ...trap/engines/engine.py.229: process: 0.54872 | load: 0.00246 [I 2022-10-26 21:04:31] ...trap/engines/engine.py.230: loss: 0.16577 [I 2022-10-26 21:04:31] ...optimizers/trijoint.py.151: total_norm: 18.409630 [I 2022-10-26 21:04:36] ...trap/engines/engine.py.224: train: epoch 29 | batch 2180/2304 [I 2022-10-26 21:04:36] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:01:08 [I 2022-10-26 21:04:36] ...trap/engines/engine.py.229: process: 0.54919 | load: 0.00247 [I 2022-10-26 21:04:36] ...trap/engines/engine.py.230: loss: 0.19679 [I 2022-10-26 21:04:36] ...optimizers/trijoint.py.151: total_norm: 19.247446 [I 2022-10-26 21:04:42] ...trap/engines/engine.py.224: train: epoch 29 | batch 2190/2304 [I 2022-10-26 21:04:42] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:01:03 [I 2022-10-26 21:04:42] ...trap/engines/engine.py.229: process: 0.56265 | load: 0.00240 [I 2022-10-26 21:04:42] ...trap/engines/engine.py.230: loss: 0.20383 [I 2022-10-26 21:04:42] ...optimizers/trijoint.py.151: total_norm: 21.011759 [I 2022-10-26 21:04:47] ...trap/engines/engine.py.224: train: epoch 29 | batch 2200/2304 [I 2022-10-26 21:04:47] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:58 [I 2022-10-26 21:04:47] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00240 [I 2022-10-26 21:04:47] ...trap/engines/engine.py.230: loss: 0.18073 [I 2022-10-26 21:04:47] ...optimizers/trijoint.py.151: total_norm: 18.951950 [I 2022-10-26 21:04:53] ...trap/engines/engine.py.224: train: epoch 29 | batch 2210/2304 [I 2022-10-26 21:04:53] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:52 [I 2022-10-26 21:04:53] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00242 [I 2022-10-26 21:04:53] ...trap/engines/engine.py.230: loss: 0.16249 [I 2022-10-26 21:04:53] ...optimizers/trijoint.py.151: total_norm: 13.667379 [I 2022-10-26 21:04:59] ...trap/engines/engine.py.224: train: epoch 29 | batch 2220/2304 [I 2022-10-26 21:04:59] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:46 [I 2022-10-26 21:04:59] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00240 [I 2022-10-26 21:04:59] ...trap/engines/engine.py.230: loss: 0.26289 [I 2022-10-26 21:04:59] ...optimizers/trijoint.py.151: total_norm: 14.924859 [I 2022-10-26 21:05:04] ...trap/engines/engine.py.224: train: epoch 29 | batch 2230/2304 [I 2022-10-26 21:05:04] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:40 [I 2022-10-26 21:05:04] ...trap/engines/engine.py.229: process: 0.54908 | load: 0.00262 [I 2022-10-26 21:05:04] ...trap/engines/engine.py.230: loss: 0.18823 [I 2022-10-26 21:05:04] ...optimizers/trijoint.py.151: total_norm: 15.227892 [I 2022-10-26 21:05:10] ...trap/engines/engine.py.224: train: epoch 29 | batch 2240/2304 [I 2022-10-26 21:05:10] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:35 [I 2022-10-26 21:05:10] ...trap/engines/engine.py.229: process: 0.55352 | load: 0.00240 [I 2022-10-26 21:05:10] ...trap/engines/engine.py.230: loss: 0.14578 [I 2022-10-26 21:05:10] ...optimizers/trijoint.py.151: total_norm: 15.599116 [I 2022-10-26 21:05:15] ...trap/engines/engine.py.224: train: epoch 29 | batch 2250/2304 [I 2022-10-26 21:05:15] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:30 [I 2022-10-26 21:05:15] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00253 [I 2022-10-26 21:05:15] ...trap/engines/engine.py.230: loss: 0.22239 [I 2022-10-26 21:05:15] ...optimizers/trijoint.py.151: total_norm: 23.789480 [I 2022-10-26 21:05:21] ...trap/engines/engine.py.224: train: epoch 29 | batch 2260/2304 [I 2022-10-26 21:05:21] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:24 [I 2022-10-26 21:05:21] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00246 [I 2022-10-26 21:05:21] ...trap/engines/engine.py.230: loss: 0.15754 [I 2022-10-26 21:05:21] ...optimizers/trijoint.py.151: total_norm: 12.535212 [I 2022-10-26 21:05:26] ...trap/engines/engine.py.224: train: epoch 29 | batch 2270/2304 [I 2022-10-26 21:05:26] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:18 [I 2022-10-26 21:05:26] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00243 [I 2022-10-26 21:05:26] ...trap/engines/engine.py.230: loss: 0.19966 [I 2022-10-26 21:05:26] ...optimizers/trijoint.py.151: total_norm: 15.417843 [I 2022-10-26 21:05:32] ...trap/engines/engine.py.224: train: epoch 29 | batch 2280/2304 [I 2022-10-26 21:05:32] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:13 [I 2022-10-26 21:05:32] ...trap/engines/engine.py.229: process: 0.54653 | load: 0.00240 [I 2022-10-26 21:05:32] ...trap/engines/engine.py.230: loss: 0.24912 [I 2022-10-26 21:05:32] ...optimizers/trijoint.py.151: total_norm: 26.294062 [I 2022-10-26 21:05:38] ...trap/engines/engine.py.224: train: epoch 29 | batch 2290/2304 [I 2022-10-26 21:05:38] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:00:07 [I 2022-10-26 21:05:38] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00241 [I 2022-10-26 21:05:38] ...trap/engines/engine.py.230: loss: 0.24744 [I 2022-10-26 21:05:38] ...optimizers/trijoint.py.151: total_norm: 17.216194 [I 2022-10-26 21:05:43] ...trap/engines/engine.py.224: train: epoch 29 | batch 2300/2304 [I 2022-10-26 21:05:43] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:02 [I 2022-10-26 21:05:43] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00015 [I 2022-10-26 21:05:43] ...trap/engines/engine.py.230: loss: 0.13432 [I 2022-10-26 21:05:43] ...optimizers/trijoint.py.151: total_norm: 21.802876 [I 2022-10-26 21:05:46] ...trap/engines/engine.py.224: train: epoch 29 | batch 2304/2304 [I 2022-10-26 21:05:46] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:00 [I 2022-10-26 21:05:46] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00015 [I 2022-10-26 21:05:46] ...trap/engines/engine.py.230: loss: 0.33808 [I 2022-10-26 21:05:46] ...optimizers/trijoint.py.151: total_norm: 21.657598 [S 2022-10-26 21:05:47] ...trap/engines/engine.py.129: train_epoch.epoch: 29 [S 2022-10-26 21:05:47] ...trap/engines/engine.py.129: train_epoch.loss: 0.18241162918459566 [I 2022-10-26 21:05:47] ...s/criterions/triplet.py.84: margin = 0.20000000000000012 [I 2022-10-26 21:05:47] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 29 [I 2022-10-26 21:05:47] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 21:05:49] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 21:05:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 21:05:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 21:05:50] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 21:05:50] ...trap/engines/engine.py.321: eval: epoch 29 | batch 0/510 [I 2022-10-26 21:05:50] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:01 [I 2022-10-26 21:05:50] ...trap/engines/engine.py.326: process: 3.17935 | load: 2.80264 [I 2022-10-26 21:05:54] ...trap/engines/engine.py.321: eval: epoch 29 | batch 10/510 [I 2022-10-26 21:05:54] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:19 [I 2022-10-26 21:05:54] ...trap/engines/engine.py.326: process: 0.34543 | load: 0.00020 [I 2022-10-26 21:05:57] ...trap/engines/engine.py.321: eval: epoch 29 | batch 20/510 [I 2022-10-26 21:05:57] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:59 [I 2022-10-26 21:05:57] ...trap/engines/engine.py.326: process: 0.33062 | load: 0.00021 [I 2022-10-26 21:06:00] ...trap/engines/engine.py.321: eval: epoch 29 | batch 30/510 [I 2022-10-26 21:06:00] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:44 [I 2022-10-26 21:06:00] ...trap/engines/engine.py.326: process: 0.36077 | load: 0.00018 [I 2022-10-26 21:06:04] ...trap/engines/engine.py.321: eval: epoch 29 | batch 40/510 [I 2022-10-26 21:06:04] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:38 [I 2022-10-26 21:06:04] ...trap/engines/engine.py.326: process: 0.34499 | load: 0.00020 [I 2022-10-26 21:06:07] ...trap/engines/engine.py.321: eval: epoch 29 | batch 50/510 [I 2022-10-26 21:06:07] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-26 21:06:07] ...trap/engines/engine.py.326: process: 0.35459 | load: 0.00030 [I 2022-10-26 21:06:11] ...trap/engines/engine.py.321: eval: epoch 29 | batch 60/510 [I 2022-10-26 21:06:11] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:30 [I 2022-10-26 21:06:11] ...trap/engines/engine.py.326: process: 0.34620 | load: 0.00026 [I 2022-10-26 21:06:14] ...trap/engines/engine.py.321: eval: epoch 29 | batch 70/510 [I 2022-10-26 21:06:14] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:26 [I 2022-10-26 21:06:14] ...trap/engines/engine.py.326: process: 0.33293 | load: 0.00031 [I 2022-10-26 21:06:17] ...trap/engines/engine.py.321: eval: epoch 29 | batch 80/510 [I 2022-10-26 21:06:17] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:24 [I 2022-10-26 21:06:17] ...trap/engines/engine.py.326: process: 0.32919 | load: 0.00020 [I 2022-10-26 21:06:21] ...trap/engines/engine.py.321: eval: epoch 29 | batch 90/510 [I 2022-10-26 21:06:21] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:24 [I 2022-10-26 21:06:21] ...trap/engines/engine.py.326: process: 0.35665 | load: 0.00020 [I 2022-10-26 21:06:24] ...trap/engines/engine.py.321: eval: epoch 29 | batch 100/510 [I 2022-10-26 21:06:24] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:16 [I 2022-10-26 21:06:24] ...trap/engines/engine.py.326: process: 0.28489 | load: 0.00023 [I 2022-10-26 21:06:27] ...trap/engines/engine.py.321: eval: epoch 29 | batch 110/510 [I 2022-10-26 21:06:27] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:58 [I 2022-10-26 21:06:27] ...trap/engines/engine.py.326: process: 0.28740 | load: 0.00016 [I 2022-10-26 21:06:30] ...trap/engines/engine.py.321: eval: epoch 29 | batch 120/510 [I 2022-10-26 21:06:30] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-26 21:06:30] ...trap/engines/engine.py.326: process: 0.28590 | load: 0.00017 [I 2022-10-26 21:06:33] ...trap/engines/engine.py.321: eval: epoch 29 | batch 130/510 [I 2022-10-26 21:06:33] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 21:06:33] ...trap/engines/engine.py.326: process: 0.29415 | load: 0.00018 [I 2022-10-26 21:06:36] ...trap/engines/engine.py.321: eval: epoch 29 | batch 140/510 [I 2022-10-26 21:06:36] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-26 21:06:36] ...trap/engines/engine.py.326: process: 0.29041 | load: 0.00014 [I 2022-10-26 21:06:39] ...trap/engines/engine.py.321: eval: epoch 29 | batch 150/510 [I 2022-10-26 21:06:39] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 21:06:39] ...trap/engines/engine.py.326: process: 0.29303 | load: 0.00014 [I 2022-10-26 21:06:41] ...trap/engines/engine.py.321: eval: epoch 29 | batch 160/510 [I 2022-10-26 21:06:41] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-26 21:06:41] ...trap/engines/engine.py.326: process: 0.28902 | load: 0.00017 [I 2022-10-26 21:06:44] ...trap/engines/engine.py.321: eval: epoch 29 | batch 170/510 [I 2022-10-26 21:06:44] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-26 21:06:44] ...trap/engines/engine.py.326: process: 0.29319 | load: 0.00016 [I 2022-10-26 21:06:47] ...trap/engines/engine.py.321: eval: epoch 29 | batch 180/510 [I 2022-10-26 21:06:47] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:34 [I 2022-10-26 21:06:47] ...trap/engines/engine.py.326: process: 0.28683 | load: 0.00017 [I 2022-10-26 21:06:50] ...trap/engines/engine.py.321: eval: epoch 29 | batch 190/510 [I 2022-10-26 21:06:50] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:31 [I 2022-10-26 21:06:50] ...trap/engines/engine.py.326: process: 0.28810 | load: 0.00017 [I 2022-10-26 21:06:53] ...trap/engines/engine.py.321: eval: epoch 29 | batch 200/510 [I 2022-10-26 21:06:53] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 21:06:53] ...trap/engines/engine.py.326: process: 0.29075 | load: 0.00017 [I 2022-10-26 21:06:56] ...trap/engines/engine.py.321: eval: epoch 29 | batch 210/510 [I 2022-10-26 21:06:56] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 21:06:56] ...trap/engines/engine.py.326: process: 0.28953 | load: 0.00022 [I 2022-10-26 21:06:59] ...trap/engines/engine.py.321: eval: epoch 29 | batch 220/510 [I 2022-10-26 21:06:59] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:28 [I 2022-10-26 21:06:59] ...trap/engines/engine.py.326: process: 0.28465 | load: 0.00023 [I 2022-10-26 21:07:02] ...trap/engines/engine.py.321: eval: epoch 29 | batch 230/510 [I 2022-10-26 21:07:02] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 21:07:02] ...trap/engines/engine.py.326: process: 0.28887 | load: 0.00015 [I 2022-10-26 21:07:05] ...trap/engines/engine.py.321: eval: epoch 29 | batch 240/510 [I 2022-10-26 21:07:05] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 21:07:05] ...trap/engines/engine.py.326: process: 0.30662 | load: 0.00018 [I 2022-10-26 21:07:07] ...trap/engines/engine.py.321: eval: epoch 29 | batch 250/510 [I 2022-10-26 21:07:07] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-26 21:07:07] ...trap/engines/engine.py.326: process: 0.28491 | load: 0.00024 [I 2022-10-26 21:07:10] ...trap/engines/engine.py.321: eval: epoch 29 | batch 260/510 [I 2022-10-26 21:07:10] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-26 21:07:10] ...trap/engines/engine.py.326: process: 0.28858 | load: 0.00027 [I 2022-10-26 21:07:13] ...trap/engines/engine.py.321: eval: epoch 29 | batch 270/510 [I 2022-10-26 21:07:13] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:09 [I 2022-10-26 21:07:13] ...trap/engines/engine.py.326: process: 0.28904 | load: 0.00022 [I 2022-10-26 21:07:16] ...trap/engines/engine.py.321: eval: epoch 29 | batch 280/510 [I 2022-10-26 21:07:16] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 21:07:16] ...trap/engines/engine.py.326: process: 0.28890 | load: 0.00017 [I 2022-10-26 21:07:19] ...trap/engines/engine.py.321: eval: epoch 29 | batch 290/510 [I 2022-10-26 21:07:19] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 21:07:19] ...trap/engines/engine.py.326: process: 0.29021 | load: 0.00016 [I 2022-10-26 21:07:22] ...trap/engines/engine.py.321: eval: epoch 29 | batch 300/510 [I 2022-10-26 21:07:22] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 21:07:22] ...trap/engines/engine.py.326: process: 0.28998 | load: 0.00022 [I 2022-10-26 21:07:25] ...trap/engines/engine.py.321: eval: epoch 29 | batch 310/510 [I 2022-10-26 21:07:25] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 21:07:25] ...trap/engines/engine.py.326: process: 0.29068 | load: 0.00016 [I 2022-10-26 21:07:28] ...trap/engines/engine.py.321: eval: epoch 29 | batch 320/510 [I 2022-10-26 21:07:28] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 21:07:28] ...trap/engines/engine.py.326: process: 0.29103 | load: 0.00017 [I 2022-10-26 21:07:31] ...trap/engines/engine.py.321: eval: epoch 29 | batch 330/510 [I 2022-10-26 21:07:31] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 21:07:31] ...trap/engines/engine.py.326: process: 0.28786 | load: 0.00014 [I 2022-10-26 21:07:33] ...trap/engines/engine.py.321: eval: epoch 29 | batch 340/510 [I 2022-10-26 21:07:33] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:49 [I 2022-10-26 21:07:33] ...trap/engines/engine.py.326: process: 0.28620 | load: 0.00015 [I 2022-10-26 21:07:36] ...trap/engines/engine.py.321: eval: epoch 29 | batch 350/510 [I 2022-10-26 21:07:36] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 21:07:36] ...trap/engines/engine.py.326: process: 0.29433 | load: 0.00017 [I 2022-10-26 21:07:39] ...trap/engines/engine.py.321: eval: epoch 29 | batch 360/510 [I 2022-10-26 21:07:39] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 21:07:39] ...trap/engines/engine.py.326: process: 0.29047 | load: 0.00017 [I 2022-10-26 21:07:42] ...trap/engines/engine.py.321: eval: epoch 29 | batch 370/510 [I 2022-10-26 21:07:42] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 21:07:42] ...trap/engines/engine.py.326: process: 0.29441 | load: 0.00016 [I 2022-10-26 21:07:45] ...trap/engines/engine.py.321: eval: epoch 29 | batch 380/510 [I 2022-10-26 21:07:45] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 21:07:45] ...trap/engines/engine.py.326: process: 0.29055 | load: 0.00017 [I 2022-10-26 21:07:48] ...trap/engines/engine.py.321: eval: epoch 29 | batch 390/510 [I 2022-10-26 21:07:48] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 21:07:48] ...trap/engines/engine.py.326: process: 0.28400 | load: 0.00018 [I 2022-10-26 21:07:51] ...trap/engines/engine.py.321: eval: epoch 29 | batch 400/510 [I 2022-10-26 21:07:51] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 21:07:51] ...trap/engines/engine.py.326: process: 0.28815 | load: 0.00017 [I 2022-10-26 21:07:54] ...trap/engines/engine.py.321: eval: epoch 29 | batch 410/510 [I 2022-10-26 21:07:54] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:29 [I 2022-10-26 21:07:54] ...trap/engines/engine.py.326: process: 0.28583 | load: 0.00022 [I 2022-10-26 21:07:56] ...trap/engines/engine.py.321: eval: epoch 29 | batch 420/510 [I 2022-10-26 21:07:56] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:26 [I 2022-10-26 21:07:56] ...trap/engines/engine.py.326: process: 0.30119 | load: 0.00015 [I 2022-10-26 21:07:59] ...trap/engines/engine.py.321: eval: epoch 29 | batch 430/510 [I 2022-10-26 21:07:59] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 21:07:59] ...trap/engines/engine.py.326: process: 0.30801 | load: 0.00015 [I 2022-10-26 21:08:02] ...trap/engines/engine.py.321: eval: epoch 29 | batch 440/510 [I 2022-10-26 21:08:02] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 21:08:02] ...trap/engines/engine.py.326: process: 0.28598 | load: 0.00015 [I 2022-10-26 21:08:05] ...trap/engines/engine.py.321: eval: epoch 29 | batch 450/510 [I 2022-10-26 21:08:05] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 21:08:05] ...trap/engines/engine.py.326: process: 0.29755 | load: 0.00018 [I 2022-10-26 21:08:08] ...trap/engines/engine.py.321: eval: epoch 29 | batch 460/510 [I 2022-10-26 21:08:08] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 21:08:08] ...trap/engines/engine.py.326: process: 0.28187 | load: 0.00019 [I 2022-10-26 21:08:11] ...trap/engines/engine.py.321: eval: epoch 29 | batch 470/510 [I 2022-10-26 21:08:11] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 21:08:11] ...trap/engines/engine.py.326: process: 0.28571 | load: 0.00015 [I 2022-10-26 21:08:14] ...trap/engines/engine.py.321: eval: epoch 29 | batch 480/510 [I 2022-10-26 21:08:14] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 21:08:14] ...trap/engines/engine.py.326: process: 0.27951 | load: 0.00016 [I 2022-10-26 21:08:17] ...trap/engines/engine.py.321: eval: epoch 29 | batch 490/510 [I 2022-10-26 21:08:17] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 21:08:17] ...trap/engines/engine.py.326: process: 0.28913 | load: 0.00015 [I 2022-10-26 21:08:20] ...trap/engines/engine.py.321: eval: epoch 29 | batch 500/510 [I 2022-10-26 21:08:20] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 21:08:20] ...trap/engines/engine.py.326: process: 0.28230 | load: 0.00015 [I 2022-10-26 21:08:22] ...trap/engines/engine.py.321: eval: epoch 29 | batch 510/510 [I 2022-10-26 21:08:22] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 21:08:22] ...trap/engines/engine.py.326: process: 0.28378 | load: 0.00012 [S 2022-10-26 21:08:23] ...trap/engines/engine.py.132: eval_epoch.epoch: 29 [I 2022-10-26 21:08:23] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.650900 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.879000 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.923200 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.645200 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.879100 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.919600 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015129 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006885 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007808 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.007947 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006252 [S 2022-10-26 21:08:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006184 [I 2022-10-26 21:08:28] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 21:08:28] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 21:08:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 21:08:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 21:08:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 21:08:30] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 21:08:32] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 21:08:38] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 21:08:38] ...trap/engines/engine.py.163: Training model on trainset for epoch 30 [I 2022-10-26 21:08:38] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 21:08:41] ...trap/engines/engine.py.224: train: epoch 30 | batch 0/2304 [I 2022-10-26 21:08:41] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:53 [I 2022-10-26 21:08:41] ...trap/engines/engine.py.229: process: 2.70557 | load: 2.03962 [I 2022-10-26 21:08:41] ...trap/engines/engine.py.230: loss: 0.17028 [I 2022-10-26 21:08:41] ...optimizers/trijoint.py.151: total_norm: 40.601528 [I 2022-10-26 21:08:46] ...trap/engines/engine.py.224: train: epoch 30 | batch 10/2304 [I 2022-10-26 21:08:46] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:06 [I 2022-10-26 21:08:46] ...trap/engines/engine.py.229: process: 0.55362 | load: 0.00418 [I 2022-10-26 21:08:46] ...trap/engines/engine.py.230: loss: 0.20544 [I 2022-10-26 21:08:46] ...optimizers/trijoint.py.151: total_norm: 22.384214 [I 2022-10-26 21:08:52] ...trap/engines/engine.py.224: train: epoch 30 | batch 20/2304 [I 2022-10-26 21:08:52] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:06 [I 2022-10-26 21:08:52] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00359 [I 2022-10-26 21:08:52] ...trap/engines/engine.py.230: loss: 0.15015 [I 2022-10-26 21:08:52] ...optimizers/trijoint.py.151: total_norm: 18.665564 [I 2022-10-26 21:08:57] ...trap/engines/engine.py.224: train: epoch 30 | batch 30/2304 [I 2022-10-26 21:08:57] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:29 [I 2022-10-26 21:08:57] ...trap/engines/engine.py.229: process: 0.57764 | load: 0.00413 [I 2022-10-26 21:08:57] ...trap/engines/engine.py.230: loss: 0.12530 [I 2022-10-26 21:08:57] ...optimizers/trijoint.py.151: total_norm: 10.465444 [I 2022-10-26 21:09:03] ...trap/engines/engine.py.224: train: epoch 30 | batch 40/2304 [I 2022-10-26 21:09:03] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:25 [I 2022-10-26 21:09:03] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00283 [I 2022-10-26 21:09:03] ...trap/engines/engine.py.230: loss: 0.22452 [I 2022-10-26 21:09:03] ...optimizers/trijoint.py.151: total_norm: 11.540290 [I 2022-10-26 21:09:09] ...trap/engines/engine.py.224: train: epoch 30 | batch 50/2304 [I 2022-10-26 21:09:09] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:57 [I 2022-10-26 21:09:09] ...trap/engines/engine.py.229: process: 0.55283 | load: 0.00272 [I 2022-10-26 21:09:09] ...trap/engines/engine.py.230: loss: 0.15879 [I 2022-10-26 21:09:09] ...optimizers/trijoint.py.151: total_norm: 17.363132 [I 2022-10-26 21:09:14] ...trap/engines/engine.py.224: train: epoch 30 | batch 60/2304 [I 2022-10-26 21:09:14] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:55 [I 2022-10-26 21:09:14] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00252 [I 2022-10-26 21:09:14] ...trap/engines/engine.py.230: loss: 0.15572 [I 2022-10-26 21:09:14] ...optimizers/trijoint.py.151: total_norm: 14.330623 [I 2022-10-26 21:09:20] ...trap/engines/engine.py.224: train: epoch 30 | batch 70/2304 [I 2022-10-26 21:09:20] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:46 [I 2022-10-26 21:09:20] ...trap/engines/engine.py.229: process: 0.56499 | load: 0.00284 [I 2022-10-26 21:09:20] ...trap/engines/engine.py.230: loss: 0.17154 [I 2022-10-26 21:09:20] ...optimizers/trijoint.py.151: total_norm: 75.801605 [I 2022-10-26 21:09:26] ...trap/engines/engine.py.224: train: epoch 30 | batch 80/2304 [I 2022-10-26 21:09:26] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:37 [I 2022-10-26 21:09:26] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00268 [I 2022-10-26 21:09:26] ...trap/engines/engine.py.230: loss: 0.24204 [I 2022-10-26 21:09:26] ...optimizers/trijoint.py.151: total_norm: 22.406439 [I 2022-10-26 21:09:31] ...trap/engines/engine.py.224: train: epoch 30 | batch 90/2304 [I 2022-10-26 21:09:31] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:27 [I 2022-10-26 21:09:31] ...trap/engines/engine.py.229: process: 0.54996 | load: 0.00252 [I 2022-10-26 21:09:31] ...trap/engines/engine.py.230: loss: 0.21654 [I 2022-10-26 21:09:31] ...optimizers/trijoint.py.151: total_norm: 25.382795 [I 2022-10-26 21:09:37] ...trap/engines/engine.py.224: train: epoch 30 | batch 100/2304 [I 2022-10-26 21:09:37] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:24 [I 2022-10-26 21:09:37] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00244 [I 2022-10-26 21:09:37] ...trap/engines/engine.py.230: loss: 0.15825 [I 2022-10-26 21:09:37] ...optimizers/trijoint.py.151: total_norm: 18.832529 [I 2022-10-26 21:09:42] ...trap/engines/engine.py.224: train: epoch 30 | batch 110/2304 [I 2022-10-26 21:09:42] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:18 [I 2022-10-26 21:09:42] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00263 [I 2022-10-26 21:09:42] ...trap/engines/engine.py.230: loss: 0.21273 [I 2022-10-26 21:09:42] ...optimizers/trijoint.py.151: total_norm: 29.886526 [I 2022-10-26 21:09:48] ...trap/engines/engine.py.224: train: epoch 30 | batch 120/2304 [I 2022-10-26 21:09:48] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:13 [I 2022-10-26 21:09:48] ...trap/engines/engine.py.229: process: 0.54823 | load: 0.00254 [I 2022-10-26 21:09:48] ...trap/engines/engine.py.230: loss: 0.15989 [I 2022-10-26 21:09:48] ...optimizers/trijoint.py.151: total_norm: 12.483283 [I 2022-10-26 21:09:53] ...trap/engines/engine.py.224: train: epoch 30 | batch 130/2304 [I 2022-10-26 21:09:53] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:14 [I 2022-10-26 21:09:53] ...trap/engines/engine.py.229: process: 0.58020 | load: 0.00283 [I 2022-10-26 21:09:53] ...trap/engines/engine.py.230: loss: 0.18857 [I 2022-10-26 21:09:53] ...optimizers/trijoint.py.151: total_norm: 21.166134 [I 2022-10-26 21:09:59] ...trap/engines/engine.py.224: train: epoch 30 | batch 140/2304 [I 2022-10-26 21:09:59] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:01 [I 2022-10-26 21:09:59] ...trap/engines/engine.py.229: process: 0.53829 | load: 0.00254 [I 2022-10-26 21:09:59] ...trap/engines/engine.py.230: loss: 0.27982 [I 2022-10-26 21:09:59] ...optimizers/trijoint.py.151: total_norm: 17.328613 [I 2022-10-26 21:10:05] ...trap/engines/engine.py.224: train: epoch 30 | batch 150/2304 [I 2022-10-26 21:10:05] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:53 [I 2022-10-26 21:10:05] ...trap/engines/engine.py.229: process: 0.54665 | load: 0.00253 [I 2022-10-26 21:10:05] ...trap/engines/engine.py.230: loss: 0.15534 [I 2022-10-26 21:10:05] ...optimizers/trijoint.py.151: total_norm: 14.632544 [I 2022-10-26 21:10:10] ...trap/engines/engine.py.224: train: epoch 30 | batch 160/2304 [I 2022-10-26 21:10:10] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:15 [I 2022-10-26 21:10:10] ...trap/engines/engine.py.229: process: 0.56115 | load: 0.00256 [I 2022-10-26 21:10:10] ...trap/engines/engine.py.230: loss: 0.19582 [I 2022-10-26 21:10:10] ...optimizers/trijoint.py.151: total_norm: 34.837627 [I 2022-10-26 21:10:16] ...trap/engines/engine.py.224: train: epoch 30 | batch 170/2304 [I 2022-10-26 21:10:16] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:49 [I 2022-10-26 21:10:16] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00246 [I 2022-10-26 21:10:16] ...trap/engines/engine.py.230: loss: 0.16747 [I 2022-10-26 21:10:16] ...optimizers/trijoint.py.151: total_norm: 16.756725 [I 2022-10-26 21:10:21] ...trap/engines/engine.py.224: train: epoch 30 | batch 180/2304 [I 2022-10-26 21:10:21] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:48 [I 2022-10-26 21:10:21] ...trap/engines/engine.py.229: process: 0.55219 | load: 0.00239 [I 2022-10-26 21:10:21] ...trap/engines/engine.py.230: loss: 0.16513 [I 2022-10-26 21:10:21] ...optimizers/trijoint.py.151: total_norm: 34.093216 [I 2022-10-26 21:10:27] ...trap/engines/engine.py.224: train: epoch 30 | batch 190/2304 [I 2022-10-26 21:10:27] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:42 [I 2022-10-26 21:10:27] ...trap/engines/engine.py.229: process: 0.57408 | load: 0.00262 [I 2022-10-26 21:10:27] ...trap/engines/engine.py.230: loss: 0.25028 [I 2022-10-26 21:10:27] ...optimizers/trijoint.py.151: total_norm: 18.411320 [I 2022-10-26 21:10:33] ...trap/engines/engine.py.224: train: epoch 30 | batch 200/2304 [I 2022-10-26 21:10:33] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:34 [I 2022-10-26 21:10:33] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00246 [I 2022-10-26 21:10:33] ...trap/engines/engine.py.230: loss: 0.16313 [I 2022-10-26 21:10:33] ...optimizers/trijoint.py.151: total_norm: 20.096796 [I 2022-10-26 21:10:38] ...trap/engines/engine.py.224: train: epoch 30 | batch 210/2304 [I 2022-10-26 21:10:38] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:34 [I 2022-10-26 21:10:38] ...trap/engines/engine.py.229: process: 0.56986 | load: 0.00256 [I 2022-10-26 21:10:38] ...trap/engines/engine.py.230: loss: 0.15366 [I 2022-10-26 21:10:38] ...optimizers/trijoint.py.151: total_norm: 13.110685 [I 2022-10-26 21:10:44] ...trap/engines/engine.py.224: train: epoch 30 | batch 220/2304 [I 2022-10-26 21:10:44] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:16 [I 2022-10-26 21:10:44] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00251 [I 2022-10-26 21:10:44] ...trap/engines/engine.py.230: loss: 0.18117 [I 2022-10-26 21:10:44] ...optimizers/trijoint.py.151: total_norm: 16.542252 [I 2022-10-26 21:10:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 230/2304 [I 2022-10-26 21:10:49] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:17 [I 2022-10-26 21:10:49] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00243 [I 2022-10-26 21:10:49] ...trap/engines/engine.py.230: loss: 0.14701 [I 2022-10-26 21:10:49] ...optimizers/trijoint.py.151: total_norm: 10.815507 [I 2022-10-26 21:10:55] ...trap/engines/engine.py.224: train: epoch 30 | batch 240/2304 [I 2022-10-26 21:10:55] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:13 [I 2022-10-26 21:10:55] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00251 [I 2022-10-26 21:10:55] ...trap/engines/engine.py.230: loss: 0.21291 [I 2022-10-26 21:10:55] ...optimizers/trijoint.py.151: total_norm: 16.791821 [I 2022-10-26 21:11:01] ...trap/engines/engine.py.224: train: epoch 30 | batch 250/2304 [I 2022-10-26 21:11:01] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:05 [I 2022-10-26 21:11:01] ...trap/engines/engine.py.229: process: 0.54903 | load: 0.00263 [I 2022-10-26 21:11:01] ...trap/engines/engine.py.230: loss: 0.12799 [I 2022-10-26 21:11:01] ...optimizers/trijoint.py.151: total_norm: 15.775377 [I 2022-10-26 21:11:06] ...trap/engines/engine.py.224: train: epoch 30 | batch 260/2304 [I 2022-10-26 21:11:06] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:02 [I 2022-10-26 21:11:06] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00253 [I 2022-10-26 21:11:06] ...trap/engines/engine.py.230: loss: 0.29128 [I 2022-10-26 21:11:06] ...optimizers/trijoint.py.151: total_norm: 34.872093 [I 2022-10-26 21:11:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 270/2304 [I 2022-10-26 21:11:12] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:56 [I 2022-10-26 21:11:12] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00244 [I 2022-10-26 21:11:12] ...trap/engines/engine.py.230: loss: 0.14744 [I 2022-10-26 21:11:12] ...optimizers/trijoint.py.151: total_norm: 11.951769 [I 2022-10-26 21:11:17] ...trap/engines/engine.py.224: train: epoch 30 | batch 280/2304 [I 2022-10-26 21:11:17] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:44 [I 2022-10-26 21:11:17] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00249 [I 2022-10-26 21:11:17] ...trap/engines/engine.py.230: loss: 0.16769 [I 2022-10-26 21:11:17] ...optimizers/trijoint.py.151: total_norm: 18.775013 [I 2022-10-26 21:11:23] ...trap/engines/engine.py.224: train: epoch 30 | batch 290/2304 [I 2022-10-26 21:11:23] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:41 [I 2022-10-26 21:11:23] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00250 [I 2022-10-26 21:11:23] ...trap/engines/engine.py.230: loss: 0.19269 [I 2022-10-26 21:11:23] ...optimizers/trijoint.py.151: total_norm: 19.271578 [I 2022-10-26 21:11:28] ...trap/engines/engine.py.224: train: epoch 30 | batch 300/2304 [I 2022-10-26 21:11:28] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:34 [I 2022-10-26 21:11:28] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00242 [I 2022-10-26 21:11:28] ...trap/engines/engine.py.230: loss: 0.20083 [I 2022-10-26 21:11:28] ...optimizers/trijoint.py.151: total_norm: 25.310991 [I 2022-10-26 21:11:34] ...trap/engines/engine.py.224: train: epoch 30 | batch 310/2304 [I 2022-10-26 21:11:34] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:34 [I 2022-10-26 21:11:34] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00243 [I 2022-10-26 21:11:34] ...trap/engines/engine.py.230: loss: 0.17545 [I 2022-10-26 21:11:34] ...optimizers/trijoint.py.151: total_norm: 26.107262 [I 2022-10-26 21:11:40] ...trap/engines/engine.py.224: train: epoch 30 | batch 320/2304 [I 2022-10-26 21:11:40] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:22 [I 2022-10-26 21:11:40] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00241 [I 2022-10-26 21:11:40] ...trap/engines/engine.py.230: loss: 0.24150 [I 2022-10-26 21:11:40] ...optimizers/trijoint.py.151: total_norm: 22.412262 [I 2022-10-26 21:11:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 330/2304 [I 2022-10-26 21:11:45] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:25 [I 2022-10-26 21:11:45] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00251 [I 2022-10-26 21:11:45] ...trap/engines/engine.py.230: loss: 0.13470 [I 2022-10-26 21:11:45] ...optimizers/trijoint.py.151: total_norm: 11.249839 [I 2022-10-26 21:11:51] ...trap/engines/engine.py.224: train: epoch 30 | batch 340/2304 [I 2022-10-26 21:11:51] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:09 [I 2022-10-26 21:11:51] ...trap/engines/engine.py.229: process: 0.56351 | load: 0.00243 [I 2022-10-26 21:11:51] ...trap/engines/engine.py.230: loss: 0.22367 [I 2022-10-26 21:11:51] ...optimizers/trijoint.py.151: total_norm: 19.726332 [I 2022-10-26 21:11:56] ...trap/engines/engine.py.224: train: epoch 30 | batch 350/2304 [I 2022-10-26 21:11:56] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:16 [I 2022-10-26 21:11:56] ...trap/engines/engine.py.229: process: 0.56587 | load: 0.00243 [I 2022-10-26 21:11:56] ...trap/engines/engine.py.230: loss: 0.12439 [I 2022-10-26 21:11:56] ...optimizers/trijoint.py.151: total_norm: 20.301489 [I 2022-10-26 21:12:02] ...trap/engines/engine.py.224: train: epoch 30 | batch 360/2304 [I 2022-10-26 21:12:02] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:01 [I 2022-10-26 21:12:02] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00238 [I 2022-10-26 21:12:02] ...trap/engines/engine.py.230: loss: 0.15398 [I 2022-10-26 21:12:02] ...optimizers/trijoint.py.151: total_norm: 19.131668 [I 2022-10-26 21:12:08] ...trap/engines/engine.py.224: train: epoch 30 | batch 370/2304 [I 2022-10-26 21:12:08] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:50 [I 2022-10-26 21:12:08] ...trap/engines/engine.py.229: process: 0.54747 | load: 0.00241 [I 2022-10-26 21:12:08] ...trap/engines/engine.py.230: loss: 0.11057 [I 2022-10-26 21:12:08] ...optimizers/trijoint.py.151: total_norm: 10.919892 [I 2022-10-26 21:12:13] ...trap/engines/engine.py.224: train: epoch 30 | batch 380/2304 [I 2022-10-26 21:12:13] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:49 [I 2022-10-26 21:12:13] ...trap/engines/engine.py.229: process: 0.55937 | load: 0.00239 [I 2022-10-26 21:12:13] ...trap/engines/engine.py.230: loss: 0.19382 [I 2022-10-26 21:12:13] ...optimizers/trijoint.py.151: total_norm: 27.866873 [I 2022-10-26 21:12:19] ...trap/engines/engine.py.224: train: epoch 30 | batch 390/2304 [I 2022-10-26 21:12:19] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:48 [I 2022-10-26 21:12:19] ...trap/engines/engine.py.229: process: 0.56504 | load: 0.00240 [I 2022-10-26 21:12:19] ...trap/engines/engine.py.230: loss: 0.16111 [I 2022-10-26 21:12:19] ...optimizers/trijoint.py.151: total_norm: 25.645294 [I 2022-10-26 21:12:24] ...trap/engines/engine.py.224: train: epoch 30 | batch 400/2304 [I 2022-10-26 21:12:24] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:36 [I 2022-10-26 21:12:24] ...trap/engines/engine.py.229: process: 0.54619 | load: 0.00249 [I 2022-10-26 21:12:24] ...trap/engines/engine.py.230: loss: 0.20044 [I 2022-10-26 21:12:24] ...optimizers/trijoint.py.151: total_norm: 23.161621 [I 2022-10-26 21:12:30] ...trap/engines/engine.py.224: train: epoch 30 | batch 410/2304 [I 2022-10-26 21:12:30] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:30 [I 2022-10-26 21:12:30] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00244 [I 2022-10-26 21:12:30] ...trap/engines/engine.py.230: loss: 0.13216 [I 2022-10-26 21:12:30] ...optimizers/trijoint.py.151: total_norm: 15.283131 [I 2022-10-26 21:12:35] ...trap/engines/engine.py.224: train: epoch 30 | batch 420/2304 [I 2022-10-26 21:12:35] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:25 [I 2022-10-26 21:12:35] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00241 [I 2022-10-26 21:12:35] ...trap/engines/engine.py.230: loss: 0.16824 [I 2022-10-26 21:12:35] ...optimizers/trijoint.py.151: total_norm: 16.782869 [I 2022-10-26 21:12:41] ...trap/engines/engine.py.224: train: epoch 30 | batch 430/2304 [I 2022-10-26 21:12:41] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:25 [I 2022-10-26 21:12:41] ...trap/engines/engine.py.229: process: 0.54824 | load: 0.00243 [I 2022-10-26 21:12:41] ...trap/engines/engine.py.230: loss: 0.20915 [I 2022-10-26 21:12:41] ...optimizers/trijoint.py.151: total_norm: 31.292694 [I 2022-10-26 21:12:47] ...trap/engines/engine.py.224: train: epoch 30 | batch 440/2304 [I 2022-10-26 21:12:47] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:14 [I 2022-10-26 21:12:47] ...trap/engines/engine.py.229: process: 0.55263 | load: 0.00248 [I 2022-10-26 21:12:47] ...trap/engines/engine.py.230: loss: 0.15348 [I 2022-10-26 21:12:47] ...optimizers/trijoint.py.151: total_norm: 24.068993 [I 2022-10-26 21:12:52] ...trap/engines/engine.py.224: train: epoch 30 | batch 450/2304 [I 2022-10-26 21:12:52] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:06 [I 2022-10-26 21:12:52] ...trap/engines/engine.py.229: process: 0.56657 | load: 0.00243 [I 2022-10-26 21:12:52] ...trap/engines/engine.py.230: loss: 0.16954 [I 2022-10-26 21:12:52] ...optimizers/trijoint.py.151: total_norm: 26.809696 [I 2022-10-26 21:12:58] ...trap/engines/engine.py.224: train: epoch 30 | batch 460/2304 [I 2022-10-26 21:12:58] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:17 [I 2022-10-26 21:12:58] ...trap/engines/engine.py.229: process: 0.58387 | load: 0.00254 [I 2022-10-26 21:12:58] ...trap/engines/engine.py.230: loss: 0.16543 [I 2022-10-26 21:12:58] ...optimizers/trijoint.py.151: total_norm: 16.191803 [I 2022-10-26 21:13:03] ...trap/engines/engine.py.224: train: epoch 30 | batch 470/2304 [I 2022-10-26 21:13:03] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:02 [I 2022-10-26 21:13:03] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00248 [I 2022-10-26 21:13:03] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-26 21:13:03] ...optimizers/trijoint.py.151: total_norm: 22.025967 [I 2022-10-26 21:13:09] ...trap/engines/engine.py.224: train: epoch 30 | batch 480/2304 [I 2022-10-26 21:13:09] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:47 [I 2022-10-26 21:13:09] ...trap/engines/engine.py.229: process: 0.54995 | load: 0.00243 [I 2022-10-26 21:13:09] ...trap/engines/engine.py.230: loss: 0.21177 [I 2022-10-26 21:13:09] ...optimizers/trijoint.py.151: total_norm: 13.789966 [I 2022-10-26 21:13:14] ...trap/engines/engine.py.224: train: epoch 30 | batch 490/2304 [I 2022-10-26 21:13:14] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:44 [I 2022-10-26 21:13:14] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00246 [I 2022-10-26 21:13:14] ...trap/engines/engine.py.230: loss: 0.12701 [I 2022-10-26 21:13:14] ...optimizers/trijoint.py.151: total_norm: 11.813972 [I 2022-10-26 21:13:20] ...trap/engines/engine.py.224: train: epoch 30 | batch 500/2304 [I 2022-10-26 21:13:20] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:43 [I 2022-10-26 21:13:20] ...trap/engines/engine.py.229: process: 0.54818 | load: 0.00242 [I 2022-10-26 21:13:20] ...trap/engines/engine.py.230: loss: 0.19302 [I 2022-10-26 21:13:20] ...optimizers/trijoint.py.151: total_norm: 29.667658 [I 2022-10-26 21:13:26] ...trap/engines/engine.py.224: train: epoch 30 | batch 510/2304 [I 2022-10-26 21:13:26] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:32 [I 2022-10-26 21:13:26] ...trap/engines/engine.py.229: process: 0.55186 | load: 0.00240 [I 2022-10-26 21:13:26] ...trap/engines/engine.py.230: loss: 0.19723 [I 2022-10-26 21:13:26] ...optimizers/trijoint.py.151: total_norm: 22.838768 [I 2022-10-26 21:13:31] ...trap/engines/engine.py.224: train: epoch 30 | batch 520/2304 [I 2022-10-26 21:13:31] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:31 [I 2022-10-26 21:13:31] ...trap/engines/engine.py.229: process: 0.55088 | load: 0.00241 [I 2022-10-26 21:13:31] ...trap/engines/engine.py.230: loss: 0.18512 [I 2022-10-26 21:13:31] ...optimizers/trijoint.py.151: total_norm: 18.443892 [I 2022-10-26 21:13:37] ...trap/engines/engine.py.224: train: epoch 30 | batch 530/2304 [I 2022-10-26 21:13:37] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:24 [I 2022-10-26 21:13:37] ...trap/engines/engine.py.229: process: 0.55073 | load: 0.00259 [I 2022-10-26 21:13:37] ...trap/engines/engine.py.230: loss: 0.21570 [I 2022-10-26 21:13:37] ...optimizers/trijoint.py.151: total_norm: 17.955444 [I 2022-10-26 21:13:42] ...trap/engines/engine.py.224: train: epoch 30 | batch 540/2304 [I 2022-10-26 21:13:42] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:25 [I 2022-10-26 21:13:42] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00248 [I 2022-10-26 21:13:42] ...trap/engines/engine.py.230: loss: 0.16844 [I 2022-10-26 21:13:42] ...optimizers/trijoint.py.151: total_norm: 17.756760 [I 2022-10-26 21:13:48] ...trap/engines/engine.py.224: train: epoch 30 | batch 550/2304 [I 2022-10-26 21:13:48] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:10 [I 2022-10-26 21:13:48] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00244 [I 2022-10-26 21:13:48] ...trap/engines/engine.py.230: loss: 0.17763 [I 2022-10-26 21:13:48] ...optimizers/trijoint.py.151: total_norm: 14.804648 [I 2022-10-26 21:13:54] ...trap/engines/engine.py.224: train: epoch 30 | batch 560/2304 [I 2022-10-26 21:13:54] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:15 [I 2022-10-26 21:13:54] ...trap/engines/engine.py.229: process: 0.56486 | load: 0.00252 [I 2022-10-26 21:13:54] ...trap/engines/engine.py.230: loss: 0.19294 [I 2022-10-26 21:13:54] ...optimizers/trijoint.py.151: total_norm: 16.054379 [I 2022-10-26 21:13:59] ...trap/engines/engine.py.224: train: epoch 30 | batch 570/2304 [I 2022-10-26 21:13:59] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:10 [I 2022-10-26 21:13:59] ...trap/engines/engine.py.229: process: 0.56432 | load: 0.00245 [I 2022-10-26 21:13:59] ...trap/engines/engine.py.230: loss: 0.15758 [I 2022-10-26 21:13:59] ...optimizers/trijoint.py.151: total_norm: 11.992910 [I 2022-10-26 21:14:05] ...trap/engines/engine.py.224: train: epoch 30 | batch 580/2304 [I 2022-10-26 21:14:05] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:03 [I 2022-10-26 21:14:05] ...trap/engines/engine.py.229: process: 0.55063 | load: 0.00248 [I 2022-10-26 21:14:05] ...trap/engines/engine.py.230: loss: 0.14854 [I 2022-10-26 21:14:05] ...optimizers/trijoint.py.151: total_norm: 12.165541 [I 2022-10-26 21:14:10] ...trap/engines/engine.py.224: train: epoch 30 | batch 590/2304 [I 2022-10-26 21:14:10] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:16:02 [I 2022-10-26 21:14:10] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00258 [I 2022-10-26 21:14:10] ...trap/engines/engine.py.230: loss: 0.17150 [I 2022-10-26 21:14:10] ...optimizers/trijoint.py.151: total_norm: 18.000313 [I 2022-10-26 21:14:16] ...trap/engines/engine.py.224: train: epoch 30 | batch 600/2304 [I 2022-10-26 21:14:16] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:55 [I 2022-10-26 21:14:16] ...trap/engines/engine.py.229: process: 0.56503 | load: 0.00247 [I 2022-10-26 21:14:16] ...trap/engines/engine.py.230: loss: 0.22793 [I 2022-10-26 21:14:16] ...optimizers/trijoint.py.151: total_norm: 25.753052 [I 2022-10-26 21:14:22] ...trap/engines/engine.py.224: train: epoch 30 | batch 610/2304 [I 2022-10-26 21:14:22] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:48 [I 2022-10-26 21:14:22] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00245 [I 2022-10-26 21:14:22] ...trap/engines/engine.py.230: loss: 0.20926 [I 2022-10-26 21:14:22] ...optimizers/trijoint.py.151: total_norm: 17.354725 [I 2022-10-26 21:14:27] ...trap/engines/engine.py.224: train: epoch 30 | batch 620/2304 [I 2022-10-26 21:14:27] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:37 [I 2022-10-26 21:14:27] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00241 [I 2022-10-26 21:14:27] ...trap/engines/engine.py.230: loss: 0.12997 [I 2022-10-26 21:14:27] ...optimizers/trijoint.py.151: total_norm: 10.600829 [I 2022-10-26 21:14:33] ...trap/engines/engine.py.224: train: epoch 30 | batch 630/2304 [I 2022-10-26 21:14:33] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:29 [I 2022-10-26 21:14:33] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00242 [I 2022-10-26 21:14:33] ...trap/engines/engine.py.230: loss: 0.21112 [I 2022-10-26 21:14:33] ...optimizers/trijoint.py.151: total_norm: 35.513729 [I 2022-10-26 21:14:38] ...trap/engines/engine.py.224: train: epoch 30 | batch 640/2304 [I 2022-10-26 21:14:38] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:24 [I 2022-10-26 21:14:38] ...trap/engines/engine.py.229: process: 0.56282 | load: 0.00241 [I 2022-10-26 21:14:38] ...trap/engines/engine.py.230: loss: 0.18763 [I 2022-10-26 21:14:38] ...optimizers/trijoint.py.151: total_norm: 19.508266 [I 2022-10-26 21:14:44] ...trap/engines/engine.py.224: train: epoch 30 | batch 650/2304 [I 2022-10-26 21:14:44] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:24 [I 2022-10-26 21:14:44] ...trap/engines/engine.py.229: process: 0.56669 | load: 0.00241 [I 2022-10-26 21:14:44] ...trap/engines/engine.py.230: loss: 0.17792 [I 2022-10-26 21:14:44] ...optimizers/trijoint.py.151: total_norm: 13.832808 [I 2022-10-26 21:14:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 660/2304 [I 2022-10-26 21:14:49] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:14 [I 2022-10-26 21:14:49] ...trap/engines/engine.py.229: process: 0.54576 | load: 0.00249 [I 2022-10-26 21:14:49] ...trap/engines/engine.py.230: loss: 0.25827 [I 2022-10-26 21:14:49] ...optimizers/trijoint.py.151: total_norm: 26.063944 [I 2022-10-26 21:14:55] ...trap/engines/engine.py.224: train: epoch 30 | batch 670/2304 [I 2022-10-26 21:14:55] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:09 [I 2022-10-26 21:14:55] ...trap/engines/engine.py.229: process: 0.55954 | load: 0.00247 [I 2022-10-26 21:14:55] ...trap/engines/engine.py.230: loss: 0.18240 [I 2022-10-26 21:14:55] ...optimizers/trijoint.py.151: total_norm: 11.467960 [I 2022-10-26 21:15:01] ...trap/engines/engine.py.224: train: epoch 30 | batch 680/2304 [I 2022-10-26 21:15:01] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:02 [I 2022-10-26 21:15:01] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00244 [I 2022-10-26 21:15:01] ...trap/engines/engine.py.230: loss: 0.14977 [I 2022-10-26 21:15:01] ...optimizers/trijoint.py.151: total_norm: 17.025127 [I 2022-10-26 21:15:06] ...trap/engines/engine.py.224: train: epoch 30 | batch 690/2304 [I 2022-10-26 21:15:06] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:55 [I 2022-10-26 21:15:06] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00245 [I 2022-10-26 21:15:06] ...trap/engines/engine.py.230: loss: 0.17107 [I 2022-10-26 21:15:06] ...optimizers/trijoint.py.151: total_norm: 15.771001 [I 2022-10-26 21:15:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 700/2304 [I 2022-10-26 21:15:12] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:49 [I 2022-10-26 21:15:12] ...trap/engines/engine.py.229: process: 0.55048 | load: 0.00243 [I 2022-10-26 21:15:12] ...trap/engines/engine.py.230: loss: 0.12612 [I 2022-10-26 21:15:12] ...optimizers/trijoint.py.151: total_norm: 20.332844 [I 2022-10-26 21:15:17] ...trap/engines/engine.py.224: train: epoch 30 | batch 710/2304 [I 2022-10-26 21:15:17] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:48 [I 2022-10-26 21:15:17] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00249 [I 2022-10-26 21:15:17] ...trap/engines/engine.py.230: loss: 0.24686 [I 2022-10-26 21:15:17] ...optimizers/trijoint.py.151: total_norm: 34.655579 [I 2022-10-26 21:15:23] ...trap/engines/engine.py.224: train: epoch 30 | batch 720/2304 [I 2022-10-26 21:15:23] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:44 [I 2022-10-26 21:15:23] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00258 [I 2022-10-26 21:15:23] ...trap/engines/engine.py.230: loss: 0.14993 [I 2022-10-26 21:15:23] ...optimizers/trijoint.py.151: total_norm: 14.755816 [I 2022-10-26 21:15:28] ...trap/engines/engine.py.224: train: epoch 30 | batch 730/2304 [I 2022-10-26 21:15:28] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:35 [I 2022-10-26 21:15:28] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00250 [I 2022-10-26 21:15:28] ...trap/engines/engine.py.230: loss: 0.25283 [I 2022-10-26 21:15:28] ...optimizers/trijoint.py.151: total_norm: 19.236677 [I 2022-10-26 21:15:34] ...trap/engines/engine.py.224: train: epoch 30 | batch 740/2304 [I 2022-10-26 21:15:34] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:34 [I 2022-10-26 21:15:34] ...trap/engines/engine.py.229: process: 0.56451 | load: 0.00248 [I 2022-10-26 21:15:34] ...trap/engines/engine.py.230: loss: 0.13696 [I 2022-10-26 21:15:34] ...optimizers/trijoint.py.151: total_norm: 21.052250 [I 2022-10-26 21:15:40] ...trap/engines/engine.py.224: train: epoch 30 | batch 750/2304 [I 2022-10-26 21:15:40] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:22 [I 2022-10-26 21:15:40] ...trap/engines/engine.py.229: process: 0.54923 | load: 0.00243 [I 2022-10-26 21:15:40] ...trap/engines/engine.py.230: loss: 0.09745 [I 2022-10-26 21:15:40] ...optimizers/trijoint.py.151: total_norm: 7.473349 [I 2022-10-26 21:15:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 760/2304 [I 2022-10-26 21:15:45] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:34 [I 2022-10-26 21:15:45] ...trap/engines/engine.py.229: process: 0.55101 | load: 0.00243 [I 2022-10-26 21:15:45] ...trap/engines/engine.py.230: loss: 0.17334 [I 2022-10-26 21:15:45] ...optimizers/trijoint.py.151: total_norm: 15.760751 [I 2022-10-26 21:15:51] ...trap/engines/engine.py.224: train: epoch 30 | batch 770/2304 [I 2022-10-26 21:15:51] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:15 [I 2022-10-26 21:15:51] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00240 [I 2022-10-26 21:15:51] ...trap/engines/engine.py.230: loss: 0.10534 [I 2022-10-26 21:15:51] ...optimizers/trijoint.py.151: total_norm: 11.986190 [I 2022-10-26 21:15:57] ...trap/engines/engine.py.224: train: epoch 30 | batch 780/2304 [I 2022-10-26 21:15:57] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:09 [I 2022-10-26 21:15:57] ...trap/engines/engine.py.229: process: 0.54869 | load: 0.00238 [I 2022-10-26 21:15:57] ...trap/engines/engine.py.230: loss: 0.13362 [I 2022-10-26 21:15:57] ...optimizers/trijoint.py.151: total_norm: 17.370646 [I 2022-10-26 21:16:02] ...trap/engines/engine.py.224: train: epoch 30 | batch 790/2304 [I 2022-10-26 21:16:02] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:03 [I 2022-10-26 21:16:02] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00242 [I 2022-10-26 21:16:02] ...trap/engines/engine.py.230: loss: 0.14219 [I 2022-10-26 21:16:02] ...optimizers/trijoint.py.151: total_norm: 33.221947 [I 2022-10-26 21:16:08] ...trap/engines/engine.py.224: train: epoch 30 | batch 800/2304 [I 2022-10-26 21:16:08] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:54 [I 2022-10-26 21:16:08] ...trap/engines/engine.py.229: process: 0.56155 | load: 0.00239 [I 2022-10-26 21:16:08] ...trap/engines/engine.py.230: loss: 0.12072 [I 2022-10-26 21:16:08] ...optimizers/trijoint.py.151: total_norm: 17.988623 [I 2022-10-26 21:16:33] ...trap/engines/engine.py.224: train: epoch 30 | batch 810/2304 [I 2022-10-26 21:16:33] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:30:05 [I 2022-10-26 21:16:33] ...trap/engines/engine.py.229: process: 0.54745 | load: 0.00244 [I 2022-10-26 21:16:33] ...trap/engines/engine.py.230: loss: 0.16722 [I 2022-10-26 21:16:33] ...optimizers/trijoint.py.151: total_norm: 34.381268 [I 2022-10-26 21:16:38] ...trap/engines/engine.py.224: train: epoch 30 | batch 820/2304 [I 2022-10-26 21:16:38] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:15:29 [I 2022-10-26 21:16:38] ...trap/engines/engine.py.229: process: 0.55432 | load: 0.00240 [I 2022-10-26 21:16:38] ...trap/engines/engine.py.230: loss: 0.17392 [I 2022-10-26 21:16:38] ...optimizers/trijoint.py.151: total_norm: 20.150249 [I 2022-10-26 21:16:44] ...trap/engines/engine.py.224: train: epoch 30 | batch 830/2304 [I 2022-10-26 21:16:44] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:13:56 [I 2022-10-26 21:16:44] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00240 [I 2022-10-26 21:16:44] ...trap/engines/engine.py.230: loss: 0.22735 [I 2022-10-26 21:16:44] ...optimizers/trijoint.py.151: total_norm: 33.359837 [I 2022-10-26 21:16:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 840/2304 [I 2022-10-26 21:16:49] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:35 [I 2022-10-26 21:16:49] ...trap/engines/engine.py.229: process: 0.55260 | load: 0.00242 [I 2022-10-26 21:16:49] ...trap/engines/engine.py.230: loss: 0.20233 [I 2022-10-26 21:16:49] ...optimizers/trijoint.py.151: total_norm: 28.935371 [I 2022-10-26 21:16:55] ...trap/engines/engine.py.224: train: epoch 30 | batch 850/2304 [I 2022-10-26 21:16:55] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:13:26 [I 2022-10-26 21:16:55] ...trap/engines/engine.py.229: process: 0.54537 | load: 0.00243 [I 2022-10-26 21:16:55] ...trap/engines/engine.py.230: loss: 0.12826 [I 2022-10-26 21:16:55] ...optimizers/trijoint.py.151: total_norm: 17.987946 [I 2022-10-26 21:17:01] ...trap/engines/engine.py.224: train: epoch 30 | batch 860/2304 [I 2022-10-26 21:17:01] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:13:45 [I 2022-10-26 21:17:01] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00244 [I 2022-10-26 21:17:01] ...trap/engines/engine.py.230: loss: 0.20231 [I 2022-10-26 21:17:01] ...optimizers/trijoint.py.151: total_norm: 17.275137 [I 2022-10-26 21:17:06] ...trap/engines/engine.py.224: train: epoch 30 | batch 870/2304 [I 2022-10-26 21:17:06] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:17 [I 2022-10-26 21:17:06] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00244 [I 2022-10-26 21:17:06] ...trap/engines/engine.py.230: loss: 0.13581 [I 2022-10-26 21:17:06] ...optimizers/trijoint.py.151: total_norm: 12.262909 [I 2022-10-26 21:17:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 880/2304 [I 2022-10-26 21:17:12] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:13:14 [I 2022-10-26 21:17:12] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00244 [I 2022-10-26 21:17:12] ...trap/engines/engine.py.230: loss: 0.16003 [I 2022-10-26 21:17:12] ...optimizers/trijoint.py.151: total_norm: 11.373472 [I 2022-10-26 21:17:17] ...trap/engines/engine.py.224: train: epoch 30 | batch 890/2304 [I 2022-10-26 21:17:17] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:13:03 [I 2022-10-26 21:17:17] ...trap/engines/engine.py.229: process: 0.54782 | load: 0.00266 [I 2022-10-26 21:17:17] ...trap/engines/engine.py.230: loss: 0.17582 [I 2022-10-26 21:17:17] ...optimizers/trijoint.py.151: total_norm: 23.617310 [I 2022-10-26 21:17:23] ...trap/engines/engine.py.224: train: epoch 30 | batch 900/2304 [I 2022-10-26 21:17:23] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:13:01 [I 2022-10-26 21:17:23] ...trap/engines/engine.py.229: process: 0.54876 | load: 0.00245 [I 2022-10-26 21:17:23] ...trap/engines/engine.py.230: loss: 0.28119 [I 2022-10-26 21:17:23] ...optimizers/trijoint.py.151: total_norm: 26.912275 [I 2022-10-26 21:17:29] ...trap/engines/engine.py.224: train: epoch 30 | batch 910/2304 [I 2022-10-26 21:17:29] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:12:57 [I 2022-10-26 21:17:29] ...trap/engines/engine.py.229: process: 0.56207 | load: 0.00252 [I 2022-10-26 21:17:29] ...trap/engines/engine.py.230: loss: 0.16036 [I 2022-10-26 21:17:29] ...optimizers/trijoint.py.151: total_norm: 18.662991 [I 2022-10-26 21:17:34] ...trap/engines/engine.py.224: train: epoch 30 | batch 920/2304 [I 2022-10-26 21:17:34] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:12:50 [I 2022-10-26 21:17:34] ...trap/engines/engine.py.229: process: 0.55485 | load: 0.00250 [I 2022-10-26 21:17:34] ...trap/engines/engine.py.230: loss: 0.23248 [I 2022-10-26 21:17:34] ...optimizers/trijoint.py.151: total_norm: 18.571550 [I 2022-10-26 21:17:40] ...trap/engines/engine.py.224: train: epoch 30 | batch 930/2304 [I 2022-10-26 21:17:40] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:47 [I 2022-10-26 21:17:40] ...trap/engines/engine.py.229: process: 0.56989 | load: 0.00341 [I 2022-10-26 21:17:40] ...trap/engines/engine.py.230: loss: 0.17574 [I 2022-10-26 21:17:40] ...optimizers/trijoint.py.151: total_norm: 16.773941 [I 2022-10-26 21:17:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 940/2304 [I 2022-10-26 21:17:45] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:41 [I 2022-10-26 21:17:45] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00257 [I 2022-10-26 21:17:45] ...trap/engines/engine.py.230: loss: 0.20286 [I 2022-10-26 21:17:45] ...optimizers/trijoint.py.151: total_norm: 24.933533 [I 2022-10-26 21:17:51] ...trap/engines/engine.py.224: train: epoch 30 | batch 950/2304 [I 2022-10-26 21:17:51] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:37 [I 2022-10-26 21:17:51] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00253 [I 2022-10-26 21:17:51] ...trap/engines/engine.py.230: loss: 0.11592 [I 2022-10-26 21:17:51] ...optimizers/trijoint.py.151: total_norm: 13.046515 [I 2022-10-26 21:17:57] ...trap/engines/engine.py.224: train: epoch 30 | batch 960/2304 [I 2022-10-26 21:17:57] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:35 [I 2022-10-26 21:17:57] ...trap/engines/engine.py.229: process: 0.57225 | load: 0.00251 [I 2022-10-26 21:17:57] ...trap/engines/engine.py.230: loss: 0.23466 [I 2022-10-26 21:17:57] ...optimizers/trijoint.py.151: total_norm: 31.300064 [I 2022-10-26 21:18:02] ...trap/engines/engine.py.224: train: epoch 30 | batch 970/2304 [I 2022-10-26 21:18:02] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:33 [I 2022-10-26 21:18:02] ...trap/engines/engine.py.229: process: 0.56288 | load: 0.00267 [I 2022-10-26 21:18:02] ...trap/engines/engine.py.230: loss: 0.11343 [I 2022-10-26 21:18:02] ...optimizers/trijoint.py.151: total_norm: 20.750319 [I 2022-10-26 21:18:08] ...trap/engines/engine.py.224: train: epoch 30 | batch 980/2304 [I 2022-10-26 21:18:08] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:12:32 [I 2022-10-26 21:18:08] ...trap/engines/engine.py.229: process: 0.57856 | load: 0.00286 [I 2022-10-26 21:18:08] ...trap/engines/engine.py.230: loss: 0.17635 [I 2022-10-26 21:18:08] ...optimizers/trijoint.py.151: total_norm: 18.634583 [I 2022-10-26 21:18:14] ...trap/engines/engine.py.224: train: epoch 30 | batch 990/2304 [I 2022-10-26 21:18:14] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:12:14 [I 2022-10-26 21:18:14] ...trap/engines/engine.py.229: process: 0.55693 | load: 0.00284 [I 2022-10-26 21:18:14] ...trap/engines/engine.py.230: loss: 0.17631 [I 2022-10-26 21:18:14] ...optimizers/trijoint.py.151: total_norm: 26.493212 [I 2022-10-26 21:18:19] ...trap/engines/engine.py.224: train: epoch 30 | batch 1000/2304 [I 2022-10-26 21:18:19] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:12:06 [I 2022-10-26 21:18:19] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00269 [I 2022-10-26 21:18:19] ...trap/engines/engine.py.230: loss: 0.20762 [I 2022-10-26 21:18:19] ...optimizers/trijoint.py.151: total_norm: 38.282204 [I 2022-10-26 21:18:25] ...trap/engines/engine.py.224: train: epoch 30 | batch 1010/2304 [I 2022-10-26 21:18:25] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:12:02 [I 2022-10-26 21:18:25] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00261 [I 2022-10-26 21:18:25] ...trap/engines/engine.py.230: loss: 0.24472 [I 2022-10-26 21:18:25] ...optimizers/trijoint.py.151: total_norm: 53.907333 [I 2022-10-26 21:18:30] ...trap/engines/engine.py.224: train: epoch 30 | batch 1020/2304 [I 2022-10-26 21:18:30] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:53 [I 2022-10-26 21:18:30] ...trap/engines/engine.py.229: process: 0.54574 | load: 0.00259 [I 2022-10-26 21:18:30] ...trap/engines/engine.py.230: loss: 0.13462 [I 2022-10-26 21:18:30] ...optimizers/trijoint.py.151: total_norm: 6.890058 [I 2022-10-26 21:18:36] ...trap/engines/engine.py.224: train: epoch 30 | batch 1030/2304 [I 2022-10-26 21:18:36] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:50 [I 2022-10-26 21:18:36] ...trap/engines/engine.py.229: process: 0.56024 | load: 0.00302 [I 2022-10-26 21:18:36] ...trap/engines/engine.py.230: loss: 0.25430 [I 2022-10-26 21:18:36] ...optimizers/trijoint.py.151: total_norm: 23.396141 [I 2022-10-26 21:18:41] ...trap/engines/engine.py.224: train: epoch 30 | batch 1040/2304 [I 2022-10-26 21:18:41] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:43 [I 2022-10-26 21:18:41] ...trap/engines/engine.py.229: process: 0.56811 | load: 0.00281 [I 2022-10-26 21:18:41] ...trap/engines/engine.py.230: loss: 0.27625 [I 2022-10-26 21:18:41] ...optimizers/trijoint.py.151: total_norm: 14.849508 [I 2022-10-26 21:18:47] ...trap/engines/engine.py.224: train: epoch 30 | batch 1050/2304 [I 2022-10-26 21:18:47] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:39 [I 2022-10-26 21:18:47] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00279 [I 2022-10-26 21:18:47] ...trap/engines/engine.py.230: loss: 0.12494 [I 2022-10-26 21:18:47] ...optimizers/trijoint.py.151: total_norm: 13.198256 [I 2022-10-26 21:18:53] ...trap/engines/engine.py.224: train: epoch 30 | batch 1060/2304 [I 2022-10-26 21:18:53] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:37 [I 2022-10-26 21:18:53] ...trap/engines/engine.py.229: process: 0.57021 | load: 0.00268 [I 2022-10-26 21:18:53] ...trap/engines/engine.py.230: loss: 0.14559 [I 2022-10-26 21:18:53] ...optimizers/trijoint.py.151: total_norm: 22.275412 [I 2022-10-26 21:18:58] ...trap/engines/engine.py.224: train: epoch 30 | batch 1070/2304 [I 2022-10-26 21:18:58] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:30 [I 2022-10-26 21:18:58] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00269 [I 2022-10-26 21:18:58] ...trap/engines/engine.py.230: loss: 0.15754 [I 2022-10-26 21:18:58] ...optimizers/trijoint.py.151: total_norm: 18.928455 [I 2022-10-26 21:19:04] ...trap/engines/engine.py.224: train: epoch 30 | batch 1080/2304 [I 2022-10-26 21:19:04] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:18 [I 2022-10-26 21:19:04] ...trap/engines/engine.py.229: process: 0.54133 | load: 0.00258 [I 2022-10-26 21:19:04] ...trap/engines/engine.py.230: loss: 0.15737 [I 2022-10-26 21:19:04] ...optimizers/trijoint.py.151: total_norm: 14.639998 [I 2022-10-26 21:19:09] ...trap/engines/engine.py.224: train: epoch 30 | batch 1090/2304 [I 2022-10-26 21:19:09] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:18 [I 2022-10-26 21:19:09] ...trap/engines/engine.py.229: process: 0.55829 | load: 0.00255 [I 2022-10-26 21:19:09] ...trap/engines/engine.py.230: loss: 0.17288 [I 2022-10-26 21:19:09] ...optimizers/trijoint.py.151: total_norm: 21.791872 [I 2022-10-26 21:19:15] ...trap/engines/engine.py.224: train: epoch 30 | batch 1100/2304 [I 2022-10-26 21:19:15] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:16 [I 2022-10-26 21:19:15] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00254 [I 2022-10-26 21:19:15] ...trap/engines/engine.py.230: loss: 0.12895 [I 2022-10-26 21:19:15] ...optimizers/trijoint.py.151: total_norm: 10.839601 [I 2022-10-26 21:19:21] ...trap/engines/engine.py.224: train: epoch 30 | batch 1110/2304 [I 2022-10-26 21:19:21] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:11:15 [I 2022-10-26 21:19:21] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00263 [I 2022-10-26 21:19:21] ...trap/engines/engine.py.230: loss: 0.26356 [I 2022-10-26 21:19:21] ...optimizers/trijoint.py.151: total_norm: 22.433897 [I 2022-10-26 21:19:26] ...trap/engines/engine.py.224: train: epoch 30 | batch 1120/2304 [I 2022-10-26 21:19:26] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:11:06 [I 2022-10-26 21:19:26] ...trap/engines/engine.py.229: process: 0.57114 | load: 0.00258 [I 2022-10-26 21:19:26] ...trap/engines/engine.py.230: loss: 0.18901 [I 2022-10-26 21:19:26] ...optimizers/trijoint.py.151: total_norm: 17.806614 [I 2022-10-26 21:19:32] ...trap/engines/engine.py.224: train: epoch 30 | batch 1130/2304 [I 2022-10-26 21:19:32] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:58 [I 2022-10-26 21:19:32] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00268 [I 2022-10-26 21:19:32] ...trap/engines/engine.py.230: loss: 0.13777 [I 2022-10-26 21:19:32] ...optimizers/trijoint.py.151: total_norm: 17.111103 [I 2022-10-26 21:19:38] ...trap/engines/engine.py.224: train: epoch 30 | batch 1140/2304 [I 2022-10-26 21:19:38] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:47 [I 2022-10-26 21:19:38] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00265 [I 2022-10-26 21:19:38] ...trap/engines/engine.py.230: loss: 0.22545 [I 2022-10-26 21:19:38] ...optimizers/trijoint.py.151: total_norm: 32.050503 [I 2022-10-26 21:19:43] ...trap/engines/engine.py.224: train: epoch 30 | batch 1150/2304 [I 2022-10-26 21:19:43] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:43 [I 2022-10-26 21:19:43] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00255 [I 2022-10-26 21:19:43] ...trap/engines/engine.py.230: loss: 0.16318 [I 2022-10-26 21:19:43] ...optimizers/trijoint.py.151: total_norm: 19.464306 [I 2022-10-26 21:19:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 1160/2304 [I 2022-10-26 21:19:49] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:44 [I 2022-10-26 21:19:49] ...trap/engines/engine.py.229: process: 0.54749 | load: 0.00269 [I 2022-10-26 21:19:49] ...trap/engines/engine.py.230: loss: 0.20567 [I 2022-10-26 21:19:49] ...optimizers/trijoint.py.151: total_norm: 15.776990 [I 2022-10-26 21:19:54] ...trap/engines/engine.py.224: train: epoch 30 | batch 1170/2304 [I 2022-10-26 21:19:54] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:37 [I 2022-10-26 21:19:54] ...trap/engines/engine.py.229: process: 0.56382 | load: 0.00332 [I 2022-10-26 21:19:54] ...trap/engines/engine.py.230: loss: 0.22768 [I 2022-10-26 21:19:54] ...optimizers/trijoint.py.151: total_norm: 23.504141 [I 2022-10-26 21:20:00] ...trap/engines/engine.py.224: train: epoch 30 | batch 1180/2304 [I 2022-10-26 21:20:00] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:26 [I 2022-10-26 21:20:00] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00276 [I 2022-10-26 21:20:00] ...trap/engines/engine.py.230: loss: 0.16256 [I 2022-10-26 21:20:00] ...optimizers/trijoint.py.151: total_norm: 25.447523 [I 2022-10-26 21:20:06] ...trap/engines/engine.py.224: train: epoch 30 | batch 1190/2304 [I 2022-10-26 21:20:06] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:22 [I 2022-10-26 21:20:06] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00251 [I 2022-10-26 21:20:06] ...trap/engines/engine.py.230: loss: 0.17391 [I 2022-10-26 21:20:06] ...optimizers/trijoint.py.151: total_norm: 15.503190 [I 2022-10-26 21:20:11] ...trap/engines/engine.py.224: train: epoch 30 | batch 1200/2304 [I 2022-10-26 21:20:11] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:18 [I 2022-10-26 21:20:11] ...trap/engines/engine.py.229: process: 0.56901 | load: 0.00245 [I 2022-10-26 21:20:11] ...trap/engines/engine.py.230: loss: 0.22021 [I 2022-10-26 21:20:11] ...optimizers/trijoint.py.151: total_norm: 25.174938 [I 2022-10-26 21:20:17] ...trap/engines/engine.py.224: train: epoch 30 | batch 1210/2304 [I 2022-10-26 21:20:17] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:10:27 [I 2022-10-26 21:20:17] ...trap/engines/engine.py.229: process: 0.53903 | load: 0.00274 [I 2022-10-26 21:20:17] ...trap/engines/engine.py.230: loss: 0.14425 [I 2022-10-26 21:20:17] ...optimizers/trijoint.py.151: total_norm: 11.685926 [I 2022-10-26 21:20:22] ...trap/engines/engine.py.224: train: epoch 30 | batch 1220/2304 [I 2022-10-26 21:20:22] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:10:09 [I 2022-10-26 21:20:22] ...trap/engines/engine.py.229: process: 0.57387 | load: 0.00260 [I 2022-10-26 21:20:22] ...trap/engines/engine.py.230: loss: 0.18454 [I 2022-10-26 21:20:22] ...optimizers/trijoint.py.151: total_norm: 16.502722 [I 2022-10-26 21:20:28] ...trap/engines/engine.py.224: train: epoch 30 | batch 1230/2304 [I 2022-10-26 21:20:28] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:57 [I 2022-10-26 21:20:28] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00252 [I 2022-10-26 21:20:28] ...trap/engines/engine.py.230: loss: 0.14302 [I 2022-10-26 21:20:28] ...optimizers/trijoint.py.151: total_norm: 67.682686 [I 2022-10-26 21:20:34] ...trap/engines/engine.py.224: train: epoch 30 | batch 1240/2304 [I 2022-10-26 21:20:34] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:51 [I 2022-10-26 21:20:34] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00258 [I 2022-10-26 21:20:34] ...trap/engines/engine.py.230: loss: 0.12383 [I 2022-10-26 21:20:34] ...optimizers/trijoint.py.151: total_norm: 19.030930 [I 2022-10-26 21:20:39] ...trap/engines/engine.py.224: train: epoch 30 | batch 1250/2304 [I 2022-10-26 21:20:39] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:44 [I 2022-10-26 21:20:39] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00274 [I 2022-10-26 21:20:39] ...trap/engines/engine.py.230: loss: 0.16039 [I 2022-10-26 21:20:39] ...optimizers/trijoint.py.151: total_norm: 10.974999 [I 2022-10-26 21:20:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 1260/2304 [I 2022-10-26 21:20:45] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:39 [I 2022-10-26 21:20:45] ...trap/engines/engine.py.229: process: 0.54853 | load: 0.00265 [I 2022-10-26 21:20:45] ...trap/engines/engine.py.230: loss: 0.15863 [I 2022-10-26 21:20:45] ...optimizers/trijoint.py.151: total_norm: 10.677864 [I 2022-10-26 21:20:50] ...trap/engines/engine.py.224: train: epoch 30 | batch 1270/2304 [I 2022-10-26 21:20:50] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:34 [I 2022-10-26 21:20:50] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00271 [I 2022-10-26 21:20:50] ...trap/engines/engine.py.230: loss: 0.17550 [I 2022-10-26 21:20:50] ...optimizers/trijoint.py.151: total_norm: 41.130249 [I 2022-10-26 21:20:56] ...trap/engines/engine.py.224: train: epoch 30 | batch 1280/2304 [I 2022-10-26 21:20:56] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:25 [I 2022-10-26 21:20:56] ...trap/engines/engine.py.229: process: 0.54336 | load: 0.00250 [I 2022-10-26 21:20:56] ...trap/engines/engine.py.230: loss: 0.15567 [I 2022-10-26 21:20:56] ...optimizers/trijoint.py.151: total_norm: 10.229079 [I 2022-10-26 21:21:01] ...trap/engines/engine.py.224: train: epoch 30 | batch 1290/2304 [I 2022-10-26 21:21:01] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:22 [I 2022-10-26 21:21:01] ...trap/engines/engine.py.229: process: 0.54878 | load: 0.00254 [I 2022-10-26 21:21:01] ...trap/engines/engine.py.230: loss: 0.37639 [I 2022-10-26 21:21:01] ...optimizers/trijoint.py.151: total_norm: 20.323704 [I 2022-10-26 21:21:07] ...trap/engines/engine.py.224: train: epoch 30 | batch 1300/2304 [I 2022-10-26 21:21:07] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:17 [I 2022-10-26 21:21:07] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00249 [I 2022-10-26 21:21:07] ...trap/engines/engine.py.230: loss: 0.15917 [I 2022-10-26 21:21:07] ...optimizers/trijoint.py.151: total_norm: 18.318846 [I 2022-10-26 21:21:13] ...trap/engines/engine.py.224: train: epoch 30 | batch 1310/2304 [I 2022-10-26 21:21:13] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:09:09 [I 2022-10-26 21:21:13] ...trap/engines/engine.py.229: process: 0.54172 | load: 0.00247 [I 2022-10-26 21:21:13] ...trap/engines/engine.py.230: loss: 0.18842 [I 2022-10-26 21:21:13] ...optimizers/trijoint.py.151: total_norm: 47.127953 [I 2022-10-26 21:21:18] ...trap/engines/engine.py.224: train: epoch 30 | batch 1320/2304 [I 2022-10-26 21:21:18] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:01 [I 2022-10-26 21:21:18] ...trap/engines/engine.py.229: process: 0.54371 | load: 0.00246 [I 2022-10-26 21:21:18] ...trap/engines/engine.py.230: loss: 0.25792 [I 2022-10-26 21:21:18] ...optimizers/trijoint.py.151: total_norm: 32.395985 [I 2022-10-26 21:21:24] ...trap/engines/engine.py.224: train: epoch 30 | batch 1330/2304 [I 2022-10-26 21:21:24] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:09:01 [I 2022-10-26 21:21:24] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00259 [I 2022-10-26 21:21:24] ...trap/engines/engine.py.230: loss: 0.18759 [I 2022-10-26 21:21:24] ...optimizers/trijoint.py.151: total_norm: 19.007423 [I 2022-10-26 21:21:29] ...trap/engines/engine.py.224: train: epoch 30 | batch 1340/2304 [I 2022-10-26 21:21:29] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:52 [I 2022-10-26 21:21:29] ...trap/engines/engine.py.229: process: 0.54271 | load: 0.00250 [I 2022-10-26 21:21:29] ...trap/engines/engine.py.230: loss: 0.14481 [I 2022-10-26 21:21:29] ...optimizers/trijoint.py.151: total_norm: 13.853249 [I 2022-10-26 21:21:35] ...trap/engines/engine.py.224: train: epoch 30 | batch 1350/2304 [I 2022-10-26 21:21:35] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:52 [I 2022-10-26 21:21:35] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00254 [I 2022-10-26 21:21:35] ...trap/engines/engine.py.230: loss: 0.12449 [I 2022-10-26 21:21:35] ...optimizers/trijoint.py.151: total_norm: 9.009048 [I 2022-10-26 21:21:40] ...trap/engines/engine.py.224: train: epoch 30 | batch 1360/2304 [I 2022-10-26 21:21:40] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:44 [I 2022-10-26 21:21:40] ...trap/engines/engine.py.229: process: 0.55034 | load: 0.00251 [I 2022-10-26 21:21:40] ...trap/engines/engine.py.230: loss: 0.18913 [I 2022-10-26 21:21:40] ...optimizers/trijoint.py.151: total_norm: 16.828735 [I 2022-10-26 21:21:46] ...trap/engines/engine.py.224: train: epoch 30 | batch 1370/2304 [I 2022-10-26 21:21:46] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:41 [I 2022-10-26 21:21:46] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00254 [I 2022-10-26 21:21:46] ...trap/engines/engine.py.230: loss: 0.16013 [I 2022-10-26 21:21:46] ...optimizers/trijoint.py.151: total_norm: 35.773365 [I 2022-10-26 21:21:52] ...trap/engines/engine.py.224: train: epoch 30 | batch 1380/2304 [I 2022-10-26 21:21:52] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:38 [I 2022-10-26 21:21:52] ...trap/engines/engine.py.229: process: 0.55450 | load: 0.00251 [I 2022-10-26 21:21:52] ...trap/engines/engine.py.230: loss: 0.17671 [I 2022-10-26 21:21:52] ...optimizers/trijoint.py.151: total_norm: 20.893394 [I 2022-10-26 21:21:57] ...trap/engines/engine.py.224: train: epoch 30 | batch 1390/2304 [I 2022-10-26 21:21:57] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:33 [I 2022-10-26 21:21:57] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00273 [I 2022-10-26 21:21:57] ...trap/engines/engine.py.230: loss: 0.17694 [I 2022-10-26 21:21:57] ...optimizers/trijoint.py.151: total_norm: 15.251678 [I 2022-10-26 21:22:03] ...trap/engines/engine.py.224: train: epoch 30 | batch 1400/2304 [I 2022-10-26 21:22:03] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:23 [I 2022-10-26 21:22:03] ...trap/engines/engine.py.229: process: 0.55695 | load: 0.00250 [I 2022-10-26 21:22:03] ...trap/engines/engine.py.230: loss: 0.23604 [I 2022-10-26 21:22:03] ...optimizers/trijoint.py.151: total_norm: 24.851337 [I 2022-10-26 21:22:08] ...trap/engines/engine.py.224: train: epoch 30 | batch 1410/2304 [I 2022-10-26 21:22:08] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:18 [I 2022-10-26 21:22:08] ...trap/engines/engine.py.229: process: 0.55206 | load: 0.00248 [I 2022-10-26 21:22:08] ...trap/engines/engine.py.230: loss: 0.15314 [I 2022-10-26 21:22:08] ...optimizers/trijoint.py.151: total_norm: 34.066692 [I 2022-10-26 21:22:14] ...trap/engines/engine.py.224: train: epoch 30 | batch 1420/2304 [I 2022-10-26 21:22:14] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:12 [I 2022-10-26 21:22:14] ...trap/engines/engine.py.229: process: 0.55215 | load: 0.00247 [I 2022-10-26 21:22:14] ...trap/engines/engine.py.230: loss: 0.17724 [I 2022-10-26 21:22:14] ...optimizers/trijoint.py.151: total_norm: 17.280460 [I 2022-10-26 21:22:20] ...trap/engines/engine.py.224: train: epoch 30 | batch 1430/2304 [I 2022-10-26 21:22:20] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:08:10 [I 2022-10-26 21:22:20] ...trap/engines/engine.py.229: process: 0.56080 | load: 0.00251 [I 2022-10-26 21:22:20] ...trap/engines/engine.py.230: loss: 0.16868 [I 2022-10-26 21:22:20] ...optimizers/trijoint.py.151: total_norm: 11.386314 [I 2022-10-26 21:22:25] ...trap/engines/engine.py.224: train: epoch 30 | batch 1440/2304 [I 2022-10-26 21:22:25] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:07:56 [I 2022-10-26 21:22:25] ...trap/engines/engine.py.229: process: 0.54599 | load: 0.00246 [I 2022-10-26 21:22:25] ...trap/engines/engine.py.230: loss: 0.17311 [I 2022-10-26 21:22:25] ...optimizers/trijoint.py.151: total_norm: 21.699326 [I 2022-10-26 21:22:31] ...trap/engines/engine.py.224: train: epoch 30 | batch 1450/2304 [I 2022-10-26 21:22:31] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:53 [I 2022-10-26 21:22:31] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00261 [I 2022-10-26 21:22:31] ...trap/engines/engine.py.230: loss: 0.14428 [I 2022-10-26 21:22:31] ...optimizers/trijoint.py.151: total_norm: 23.872198 [I 2022-10-26 21:22:36] ...trap/engines/engine.py.224: train: epoch 30 | batch 1460/2304 [I 2022-10-26 21:22:36] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:50 [I 2022-10-26 21:22:36] ...trap/engines/engine.py.229: process: 0.56023 | load: 0.00250 [I 2022-10-26 21:22:36] ...trap/engines/engine.py.230: loss: 0.25127 [I 2022-10-26 21:22:36] ...optimizers/trijoint.py.151: total_norm: 20.636398 [I 2022-10-26 21:22:42] ...trap/engines/engine.py.224: train: epoch 30 | batch 1470/2304 [I 2022-10-26 21:22:42] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:07:42 [I 2022-10-26 21:22:42] ...trap/engines/engine.py.229: process: 0.55017 | load: 0.00254 [I 2022-10-26 21:22:42] ...trap/engines/engine.py.230: loss: 0.17260 [I 2022-10-26 21:22:42] ...optimizers/trijoint.py.151: total_norm: 20.341179 [I 2022-10-26 21:22:47] ...trap/engines/engine.py.224: train: epoch 30 | batch 1480/2304 [I 2022-10-26 21:22:47] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:39 [I 2022-10-26 21:22:47] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00262 [I 2022-10-26 21:22:47] ...trap/engines/engine.py.230: loss: 0.23234 [I 2022-10-26 21:22:47] ...optimizers/trijoint.py.151: total_norm: 20.748262 [I 2022-10-26 21:22:53] ...trap/engines/engine.py.224: train: epoch 30 | batch 1490/2304 [I 2022-10-26 21:22:53] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:34 [I 2022-10-26 21:22:53] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00268 [I 2022-10-26 21:22:53] ...trap/engines/engine.py.230: loss: 0.23678 [I 2022-10-26 21:22:53] ...optimizers/trijoint.py.151: total_norm: 20.403490 [I 2022-10-26 21:22:59] ...trap/engines/engine.py.224: train: epoch 30 | batch 1500/2304 [I 2022-10-26 21:22:59] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:30 [I 2022-10-26 21:22:59] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00248 [I 2022-10-26 21:22:59] ...trap/engines/engine.py.230: loss: 0.22527 [I 2022-10-26 21:22:59] ...optimizers/trijoint.py.151: total_norm: 27.962498 [I 2022-10-26 21:23:04] ...trap/engines/engine.py.224: train: epoch 30 | batch 1510/2304 [I 2022-10-26 21:23:04] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:23 [I 2022-10-26 21:23:04] ...trap/engines/engine.py.229: process: 0.56493 | load: 0.00262 [I 2022-10-26 21:23:04] ...trap/engines/engine.py.230: loss: 0.13872 [I 2022-10-26 21:23:04] ...optimizers/trijoint.py.151: total_norm: 18.205803 [I 2022-10-26 21:23:10] ...trap/engines/engine.py.224: train: epoch 30 | batch 1520/2304 [I 2022-10-26 21:23:10] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:07:17 [I 2022-10-26 21:23:10] ...trap/engines/engine.py.229: process: 0.56326 | load: 0.00258 [I 2022-10-26 21:23:10] ...trap/engines/engine.py.230: loss: 0.17290 [I 2022-10-26 21:23:10] ...optimizers/trijoint.py.151: total_norm: 21.821819 [I 2022-10-26 21:23:15] ...trap/engines/engine.py.224: train: epoch 30 | batch 1530/2304 [I 2022-10-26 21:23:15] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:07:11 [I 2022-10-26 21:23:15] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00255 [I 2022-10-26 21:23:15] ...trap/engines/engine.py.230: loss: 0.16389 [I 2022-10-26 21:23:15] ...optimizers/trijoint.py.151: total_norm: 16.241697 [I 2022-10-26 21:23:21] ...trap/engines/engine.py.224: train: epoch 30 | batch 1540/2304 [I 2022-10-26 21:23:21] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:07:09 [I 2022-10-26 21:23:21] ...trap/engines/engine.py.229: process: 0.57020 | load: 0.00251 [I 2022-10-26 21:23:21] ...trap/engines/engine.py.230: loss: 0.18800 [I 2022-10-26 21:23:21] ...optimizers/trijoint.py.151: total_norm: 28.103857 [I 2022-10-26 21:23:27] ...trap/engines/engine.py.224: train: epoch 30 | batch 1550/2304 [I 2022-10-26 21:23:27] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:07:02 [I 2022-10-26 21:23:27] ...trap/engines/engine.py.229: process: 0.56422 | load: 0.00253 [I 2022-10-26 21:23:27] ...trap/engines/engine.py.230: loss: 0.25924 [I 2022-10-26 21:23:27] ...optimizers/trijoint.py.151: total_norm: 18.498390 [I 2022-10-26 21:23:32] ...trap/engines/engine.py.224: train: epoch 30 | batch 1560/2304 [I 2022-10-26 21:23:32] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:56 [I 2022-10-26 21:23:32] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00266 [I 2022-10-26 21:23:32] ...trap/engines/engine.py.230: loss: 0.17439 [I 2022-10-26 21:23:32] ...optimizers/trijoint.py.151: total_norm: 13.432210 [I 2022-10-26 21:23:38] ...trap/engines/engine.py.224: train: epoch 30 | batch 1570/2304 [I 2022-10-26 21:23:38] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:44 [I 2022-10-26 21:23:38] ...trap/engines/engine.py.229: process: 0.54801 | load: 0.00255 [I 2022-10-26 21:23:38] ...trap/engines/engine.py.230: loss: 0.23146 [I 2022-10-26 21:23:38] ...optimizers/trijoint.py.151: total_norm: 21.132814 [I 2022-10-26 21:23:43] ...trap/engines/engine.py.224: train: epoch 30 | batch 1580/2304 [I 2022-10-26 21:23:43] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:41 [I 2022-10-26 21:23:43] ...trap/engines/engine.py.229: process: 0.54665 | load: 0.00279 [I 2022-10-26 21:23:43] ...trap/engines/engine.py.230: loss: 0.19150 [I 2022-10-26 21:23:43] ...optimizers/trijoint.py.151: total_norm: 18.537500 [I 2022-10-26 21:23:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 1590/2304 [I 2022-10-26 21:23:49] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:34 [I 2022-10-26 21:23:49] ...trap/engines/engine.py.229: process: 0.54993 | load: 0.00275 [I 2022-10-26 21:23:49] ...trap/engines/engine.py.230: loss: 0.22630 [I 2022-10-26 21:23:49] ...optimizers/trijoint.py.151: total_norm: 15.950989 [I 2022-10-26 21:23:54] ...trap/engines/engine.py.224: train: epoch 30 | batch 1600/2304 [I 2022-10-26 21:23:54] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:32 [I 2022-10-26 21:23:54] ...trap/engines/engine.py.229: process: 0.55466 | load: 0.00261 [I 2022-10-26 21:23:54] ...trap/engines/engine.py.230: loss: 0.17575 [I 2022-10-26 21:23:54] ...optimizers/trijoint.py.151: total_norm: 13.019113 [I 2022-10-26 21:24:00] ...trap/engines/engine.py.224: train: epoch 30 | batch 1610/2304 [I 2022-10-26 21:24:00] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:26 [I 2022-10-26 21:24:00] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00252 [I 2022-10-26 21:24:00] ...trap/engines/engine.py.230: loss: 0.18633 [I 2022-10-26 21:24:00] ...optimizers/trijoint.py.151: total_norm: 22.106689 [I 2022-10-26 21:24:06] ...trap/engines/engine.py.224: train: epoch 30 | batch 1620/2304 [I 2022-10-26 21:24:06] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:24 [I 2022-10-26 21:24:06] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00294 [I 2022-10-26 21:24:06] ...trap/engines/engine.py.230: loss: 0.09883 [I 2022-10-26 21:24:06] ...optimizers/trijoint.py.151: total_norm: 9.290478 [I 2022-10-26 21:24:11] ...trap/engines/engine.py.224: train: epoch 30 | batch 1630/2304 [I 2022-10-26 21:24:11] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:15 [I 2022-10-26 21:24:11] ...trap/engines/engine.py.229: process: 0.56071 | load: 0.00265 [I 2022-10-26 21:24:11] ...trap/engines/engine.py.230: loss: 0.15543 [I 2022-10-26 21:24:11] ...optimizers/trijoint.py.151: total_norm: 17.518812 [I 2022-10-26 21:24:17] ...trap/engines/engine.py.224: train: epoch 30 | batch 1640/2304 [I 2022-10-26 21:24:17] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:06:11 [I 2022-10-26 21:24:17] ...trap/engines/engine.py.229: process: 0.54291 | load: 0.00254 [I 2022-10-26 21:24:17] ...trap/engines/engine.py.230: loss: 0.19948 [I 2022-10-26 21:24:17] ...optimizers/trijoint.py.151: total_norm: 15.423150 [I 2022-10-26 21:24:22] ...trap/engines/engine.py.224: train: epoch 30 | batch 1650/2304 [I 2022-10-26 21:24:22] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:06:03 [I 2022-10-26 21:24:22] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00255 [I 2022-10-26 21:24:22] ...trap/engines/engine.py.230: loss: 0.16128 [I 2022-10-26 21:24:22] ...optimizers/trijoint.py.151: total_norm: 16.018528 [I 2022-10-26 21:24:28] ...trap/engines/engine.py.224: train: epoch 30 | batch 1660/2304 [I 2022-10-26 21:24:28] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:59 [I 2022-10-26 21:24:28] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00252 [I 2022-10-26 21:24:28] ...trap/engines/engine.py.230: loss: 0.15212 [I 2022-10-26 21:24:28] ...optimizers/trijoint.py.151: total_norm: 29.274181 [I 2022-10-26 21:24:34] ...trap/engines/engine.py.224: train: epoch 30 | batch 1670/2304 [I 2022-10-26 21:24:34] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:53 [I 2022-10-26 21:24:34] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00252 [I 2022-10-26 21:24:34] ...trap/engines/engine.py.230: loss: 0.17687 [I 2022-10-26 21:24:34] ...optimizers/trijoint.py.151: total_norm: 18.467161 [I 2022-10-26 21:24:39] ...trap/engines/engine.py.224: train: epoch 30 | batch 1680/2304 [I 2022-10-26 21:24:39] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:46 [I 2022-10-26 21:24:39] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00246 [I 2022-10-26 21:24:39] ...trap/engines/engine.py.230: loss: 0.24184 [I 2022-10-26 21:24:39] ...optimizers/trijoint.py.151: total_norm: 18.759418 [I 2022-10-26 21:24:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 1690/2304 [I 2022-10-26 21:24:45] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:42 [I 2022-10-26 21:24:45] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00246 [I 2022-10-26 21:24:45] ...trap/engines/engine.py.230: loss: 0.20370 [I 2022-10-26 21:24:45] ...optimizers/trijoint.py.151: total_norm: 48.518223 [I 2022-10-26 21:24:50] ...trap/engines/engine.py.224: train: epoch 30 | batch 1700/2304 [I 2022-10-26 21:24:50] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:37 [I 2022-10-26 21:24:50] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00254 [I 2022-10-26 21:24:50] ...trap/engines/engine.py.230: loss: 0.17335 [I 2022-10-26 21:24:50] ...optimizers/trijoint.py.151: total_norm: 15.218003 [I 2022-10-26 21:24:56] ...trap/engines/engine.py.224: train: epoch 30 | batch 1710/2304 [I 2022-10-26 21:24:56] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:30 [I 2022-10-26 21:24:56] ...trap/engines/engine.py.229: process: 0.56519 | load: 0.00248 [I 2022-10-26 21:24:56] ...trap/engines/engine.py.230: loss: 0.19928 [I 2022-10-26 21:24:56] ...optimizers/trijoint.py.151: total_norm: 28.019354 [I 2022-10-26 21:25:02] ...trap/engines/engine.py.224: train: epoch 30 | batch 1720/2304 [I 2022-10-26 21:25:02] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:22 [I 2022-10-26 21:25:02] ...trap/engines/engine.py.229: process: 0.54084 | load: 0.00247 [I 2022-10-26 21:25:02] ...trap/engines/engine.py.230: loss: 0.16657 [I 2022-10-26 21:25:02] ...optimizers/trijoint.py.151: total_norm: 19.521694 [I 2022-10-26 21:25:07] ...trap/engines/engine.py.224: train: epoch 30 | batch 1730/2304 [I 2022-10-26 21:25:07] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:18 [I 2022-10-26 21:25:07] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00257 [I 2022-10-26 21:25:07] ...trap/engines/engine.py.230: loss: 0.19007 [I 2022-10-26 21:25:07] ...optimizers/trijoint.py.151: total_norm: 36.945675 [I 2022-10-26 21:25:13] ...trap/engines/engine.py.224: train: epoch 30 | batch 1740/2304 [I 2022-10-26 21:25:13] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:05:11 [I 2022-10-26 21:25:13] ...trap/engines/engine.py.229: process: 0.55034 | load: 0.00254 [I 2022-10-26 21:25:13] ...trap/engines/engine.py.230: loss: 0.16608 [I 2022-10-26 21:25:13] ...optimizers/trijoint.py.151: total_norm: 16.728571 [I 2022-10-26 21:25:18] ...trap/engines/engine.py.224: train: epoch 30 | batch 1750/2304 [I 2022-10-26 21:25:18] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:05:08 [I 2022-10-26 21:25:18] ...trap/engines/engine.py.229: process: 0.56478 | load: 0.00257 [I 2022-10-26 21:25:18] ...trap/engines/engine.py.230: loss: 0.15655 [I 2022-10-26 21:25:18] ...optimizers/trijoint.py.151: total_norm: 14.730908 [I 2022-10-26 21:25:24] ...trap/engines/engine.py.224: train: epoch 30 | batch 1760/2304 [I 2022-10-26 21:25:24] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:05:04 [I 2022-10-26 21:25:24] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00261 [I 2022-10-26 21:25:24] ...trap/engines/engine.py.230: loss: 0.18636 [I 2022-10-26 21:25:24] ...optimizers/trijoint.py.151: total_norm: 32.360451 [I 2022-10-26 21:25:29] ...trap/engines/engine.py.224: train: epoch 30 | batch 1770/2304 [I 2022-10-26 21:25:29] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:05:00 [I 2022-10-26 21:25:29] ...trap/engines/engine.py.229: process: 0.53594 | load: 0.00256 [I 2022-10-26 21:25:29] ...trap/engines/engine.py.230: loss: 0.12456 [I 2022-10-26 21:25:29] ...optimizers/trijoint.py.151: total_norm: 30.459412 [I 2022-10-26 21:25:35] ...trap/engines/engine.py.224: train: epoch 30 | batch 1780/2304 [I 2022-10-26 21:25:35] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:55 [I 2022-10-26 21:25:35] ...trap/engines/engine.py.229: process: 0.56611 | load: 0.00242 [I 2022-10-26 21:25:35] ...trap/engines/engine.py.230: loss: 0.21408 [I 2022-10-26 21:25:35] ...optimizers/trijoint.py.151: total_norm: 28.171337 [I 2022-10-26 21:25:41] ...trap/engines/engine.py.224: train: epoch 30 | batch 1790/2304 [I 2022-10-26 21:25:41] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:46 [I 2022-10-26 21:25:41] ...trap/engines/engine.py.229: process: 0.54735 | load: 0.00245 [I 2022-10-26 21:25:41] ...trap/engines/engine.py.230: loss: 0.14105 [I 2022-10-26 21:25:41] ...optimizers/trijoint.py.151: total_norm: 20.262238 [I 2022-10-26 21:25:46] ...trap/engines/engine.py.224: train: epoch 30 | batch 1800/2304 [I 2022-10-26 21:25:46] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:42 [I 2022-10-26 21:25:46] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00256 [I 2022-10-26 21:25:46] ...trap/engines/engine.py.230: loss: 0.18181 [I 2022-10-26 21:25:46] ...optimizers/trijoint.py.151: total_norm: 22.297132 [I 2022-10-26 21:25:52] ...trap/engines/engine.py.224: train: epoch 30 | batch 1810/2304 [I 2022-10-26 21:25:52] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:34 [I 2022-10-26 21:25:52] ...trap/engines/engine.py.229: process: 0.54572 | load: 0.00249 [I 2022-10-26 21:25:52] ...trap/engines/engine.py.230: loss: 0.22515 [I 2022-10-26 21:25:52] ...optimizers/trijoint.py.151: total_norm: 25.448622 [I 2022-10-26 21:25:57] ...trap/engines/engine.py.224: train: epoch 30 | batch 1820/2304 [I 2022-10-26 21:25:57] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:29 [I 2022-10-26 21:25:57] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00255 [I 2022-10-26 21:25:57] ...trap/engines/engine.py.230: loss: 0.16718 [I 2022-10-26 21:25:57] ...optimizers/trijoint.py.151: total_norm: 20.402704 [I 2022-10-26 21:26:03] ...trap/engines/engine.py.224: train: epoch 30 | batch 1830/2304 [I 2022-10-26 21:26:03] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:24 [I 2022-10-26 21:26:03] ...trap/engines/engine.py.229: process: 0.56025 | load: 0.00244 [I 2022-10-26 21:26:03] ...trap/engines/engine.py.230: loss: 0.15633 [I 2022-10-26 21:26:03] ...optimizers/trijoint.py.151: total_norm: 12.765570 [I 2022-10-26 21:26:09] ...trap/engines/engine.py.224: train: epoch 30 | batch 1840/2304 [I 2022-10-26 21:26:09] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:18 [I 2022-10-26 21:26:09] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00242 [I 2022-10-26 21:26:09] ...trap/engines/engine.py.230: loss: 0.18391 [I 2022-10-26 21:26:09] ...optimizers/trijoint.py.151: total_norm: 14.361485 [I 2022-10-26 21:26:14] ...trap/engines/engine.py.224: train: epoch 30 | batch 1850/2304 [I 2022-10-26 21:26:14] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:04:13 [I 2022-10-26 21:26:14] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00259 [I 2022-10-26 21:26:14] ...trap/engines/engine.py.230: loss: 0.18130 [I 2022-10-26 21:26:14] ...optimizers/trijoint.py.151: total_norm: 17.488773 [I 2022-10-26 21:26:20] ...trap/engines/engine.py.224: train: epoch 30 | batch 1860/2304 [I 2022-10-26 21:26:20] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:04:13 [I 2022-10-26 21:26:20] ...trap/engines/engine.py.229: process: 0.56917 | load: 0.00250 [I 2022-10-26 21:26:20] ...trap/engines/engine.py.230: loss: 0.12668 [I 2022-10-26 21:26:20] ...optimizers/trijoint.py.151: total_norm: 15.651745 [I 2022-10-26 21:26:25] ...trap/engines/engine.py.224: train: epoch 30 | batch 1870/2304 [I 2022-10-26 21:26:25] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:04:00 [I 2022-10-26 21:26:25] ...trap/engines/engine.py.229: process: 0.55177 | load: 0.00248 [I 2022-10-26 21:26:25] ...trap/engines/engine.py.230: loss: 0.30367 [I 2022-10-26 21:26:25] ...optimizers/trijoint.py.151: total_norm: 30.723890 [I 2022-10-26 21:26:31] ...trap/engines/engine.py.224: train: epoch 30 | batch 1880/2304 [I 2022-10-26 21:26:31] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:57 [I 2022-10-26 21:26:31] ...trap/engines/engine.py.229: process: 0.57427 | load: 0.00249 [I 2022-10-26 21:26:31] ...trap/engines/engine.py.230: loss: 0.16889 [I 2022-10-26 21:26:31] ...optimizers/trijoint.py.151: total_norm: 15.290981 [I 2022-10-26 21:26:37] ...trap/engines/engine.py.224: train: epoch 30 | batch 1890/2304 [I 2022-10-26 21:26:37] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:50 [I 2022-10-26 21:26:37] ...trap/engines/engine.py.229: process: 0.56392 | load: 0.00264 [I 2022-10-26 21:26:37] ...trap/engines/engine.py.230: loss: 0.11673 [I 2022-10-26 21:26:37] ...optimizers/trijoint.py.151: total_norm: 22.876156 [I 2022-10-26 21:26:42] ...trap/engines/engine.py.224: train: epoch 30 | batch 1900/2304 [I 2022-10-26 21:26:42] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:46 [I 2022-10-26 21:26:42] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00246 [I 2022-10-26 21:26:42] ...trap/engines/engine.py.230: loss: 0.20296 [I 2022-10-26 21:26:42] ...optimizers/trijoint.py.151: total_norm: 27.029089 [I 2022-10-26 21:26:48] ...trap/engines/engine.py.224: train: epoch 30 | batch 1910/2304 [I 2022-10-26 21:26:48] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:41 [I 2022-10-26 21:26:48] ...trap/engines/engine.py.229: process: 0.55938 | load: 0.00255 [I 2022-10-26 21:26:48] ...trap/engines/engine.py.230: loss: 0.13547 [I 2022-10-26 21:26:48] ...optimizers/trijoint.py.151: total_norm: 14.386610 [I 2022-10-26 21:26:53] ...trap/engines/engine.py.224: train: epoch 30 | batch 1920/2304 [I 2022-10-26 21:26:53] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:32 [I 2022-10-26 21:26:53] ...trap/engines/engine.py.229: process: 0.55242 | load: 0.00254 [I 2022-10-26 21:26:53] ...trap/engines/engine.py.230: loss: 0.16486 [I 2022-10-26 21:26:53] ...optimizers/trijoint.py.151: total_norm: 20.871403 [I 2022-10-26 21:26:59] ...trap/engines/engine.py.224: train: epoch 30 | batch 1930/2304 [I 2022-10-26 21:26:59] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:31 [I 2022-10-26 21:26:59] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00258 [I 2022-10-26 21:26:59] ...trap/engines/engine.py.230: loss: 0.15207 [I 2022-10-26 21:26:59] ...optimizers/trijoint.py.151: total_norm: 17.116261 [I 2022-10-26 21:27:05] ...trap/engines/engine.py.224: train: epoch 30 | batch 1940/2304 [I 2022-10-26 21:27:05] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:22 [I 2022-10-26 21:27:05] ...trap/engines/engine.py.229: process: 0.53889 | load: 0.00250 [I 2022-10-26 21:27:05] ...trap/engines/engine.py.230: loss: 0.10122 [I 2022-10-26 21:27:05] ...optimizers/trijoint.py.151: total_norm: 8.711331 [I 2022-10-26 21:27:10] ...trap/engines/engine.py.224: train: epoch 30 | batch 1950/2304 [I 2022-10-26 21:27:10] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:17 [I 2022-10-26 21:27:10] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00251 [I 2022-10-26 21:27:10] ...trap/engines/engine.py.230: loss: 0.19111 [I 2022-10-26 21:27:10] ...optimizers/trijoint.py.151: total_norm: 31.920607 [I 2022-10-26 21:27:16] ...trap/engines/engine.py.224: train: epoch 30 | batch 1960/2304 [I 2022-10-26 21:27:16] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:03:11 [I 2022-10-26 21:27:16] ...trap/engines/engine.py.229: process: 0.55008 | load: 0.00244 [I 2022-10-26 21:27:16] ...trap/engines/engine.py.230: loss: 0.20928 [I 2022-10-26 21:27:16] ...optimizers/trijoint.py.151: total_norm: 22.814625 [I 2022-10-26 21:27:21] ...trap/engines/engine.py.224: train: epoch 30 | batch 1970/2304 [I 2022-10-26 21:27:21] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:03:04 [I 2022-10-26 21:27:21] ...trap/engines/engine.py.229: process: 0.54389 | load: 0.00242 [I 2022-10-26 21:27:21] ...trap/engines/engine.py.230: loss: 0.20369 [I 2022-10-26 21:27:21] ...optimizers/trijoint.py.151: total_norm: 18.101131 [I 2022-10-26 21:27:27] ...trap/engines/engine.py.224: train: epoch 30 | batch 1980/2304 [I 2022-10-26 21:27:27] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:03:00 [I 2022-10-26 21:27:27] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00249 [I 2022-10-26 21:27:27] ...trap/engines/engine.py.230: loss: 0.20851 [I 2022-10-26 21:27:27] ...optimizers/trijoint.py.151: total_norm: 20.826048 [I 2022-10-26 21:27:33] ...trap/engines/engine.py.224: train: epoch 30 | batch 1990/2304 [I 2022-10-26 21:27:33] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:55 [I 2022-10-26 21:27:33] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00253 [I 2022-10-26 21:27:33] ...trap/engines/engine.py.230: loss: 0.23041 [I 2022-10-26 21:27:33] ...optimizers/trijoint.py.151: total_norm: 19.849527 [I 2022-10-26 21:27:38] ...trap/engines/engine.py.224: train: epoch 30 | batch 2000/2304 [I 2022-10-26 21:27:38] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:50 [I 2022-10-26 21:27:38] ...trap/engines/engine.py.229: process: 0.56280 | load: 0.00244 [I 2022-10-26 21:27:38] ...trap/engines/engine.py.230: loss: 0.13046 [I 2022-10-26 21:27:38] ...optimizers/trijoint.py.151: total_norm: 17.068676 [I 2022-10-26 21:27:44] ...trap/engines/engine.py.224: train: epoch 30 | batch 2010/2304 [I 2022-10-26 21:27:44] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:44 [I 2022-10-26 21:27:44] ...trap/engines/engine.py.229: process: 0.56519 | load: 0.00239 [I 2022-10-26 21:27:44] ...trap/engines/engine.py.230: loss: 0.19090 [I 2022-10-26 21:27:44] ...optimizers/trijoint.py.151: total_norm: 18.569948 [I 2022-10-26 21:27:49] ...trap/engines/engine.py.224: train: epoch 30 | batch 2020/2304 [I 2022-10-26 21:27:49] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:38 [I 2022-10-26 21:27:49] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00259 [I 2022-10-26 21:27:49] ...trap/engines/engine.py.230: loss: 0.18668 [I 2022-10-26 21:27:49] ...optimizers/trijoint.py.151: total_norm: 16.015144 [I 2022-10-26 21:27:55] ...trap/engines/engine.py.224: train: epoch 30 | batch 2030/2304 [I 2022-10-26 21:27:55] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:32 [I 2022-10-26 21:27:55] ...trap/engines/engine.py.229: process: 0.54813 | load: 0.00249 [I 2022-10-26 21:27:55] ...trap/engines/engine.py.230: loss: 0.25696 [I 2022-10-26 21:27:55] ...optimizers/trijoint.py.151: total_norm: 32.632290 [I 2022-10-26 21:28:01] ...trap/engines/engine.py.224: train: epoch 30 | batch 2040/2304 [I 2022-10-26 21:28:01] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:28 [I 2022-10-26 21:28:01] ...trap/engines/engine.py.229: process: 0.56087 | load: 0.00244 [I 2022-10-26 21:28:01] ...trap/engines/engine.py.230: loss: 0.18563 [I 2022-10-26 21:28:01] ...optimizers/trijoint.py.151: total_norm: 14.940872 [I 2022-10-26 21:28:06] ...trap/engines/engine.py.224: train: epoch 30 | batch 2050/2304 [I 2022-10-26 21:28:06] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:20 [I 2022-10-26 21:28:06] ...trap/engines/engine.py.229: process: 0.54963 | load: 0.00244 [I 2022-10-26 21:28:06] ...trap/engines/engine.py.230: loss: 0.18881 [I 2022-10-26 21:28:06] ...optimizers/trijoint.py.151: total_norm: 16.527853 [I 2022-10-26 21:28:12] ...trap/engines/engine.py.224: train: epoch 30 | batch 2060/2304 [I 2022-10-26 21:28:12] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:15 [I 2022-10-26 21:28:12] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00256 [I 2022-10-26 21:28:12] ...trap/engines/engine.py.230: loss: 0.14210 [I 2022-10-26 21:28:12] ...optimizers/trijoint.py.151: total_norm: 15.529559 [I 2022-10-26 21:28:17] ...trap/engines/engine.py.224: train: epoch 30 | batch 2070/2304 [I 2022-10-26 21:28:17] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:02:11 [I 2022-10-26 21:28:17] ...trap/engines/engine.py.229: process: 0.56852 | load: 0.00254 [I 2022-10-26 21:28:17] ...trap/engines/engine.py.230: loss: 0.12496 [I 2022-10-26 21:28:17] ...optimizers/trijoint.py.151: total_norm: 14.446624 [I 2022-10-26 21:28:23] ...trap/engines/engine.py.224: train: epoch 30 | batch 2080/2304 [I 2022-10-26 21:28:23] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:02:05 [I 2022-10-26 21:28:23] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00254 [I 2022-10-26 21:28:23] ...trap/engines/engine.py.230: loss: 0.20438 [I 2022-10-26 21:28:23] ...optimizers/trijoint.py.151: total_norm: 16.292923 [I 2022-10-26 21:28:29] ...trap/engines/engine.py.224: train: epoch 30 | batch 2090/2304 [I 2022-10-26 21:28:29] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:02:00 [I 2022-10-26 21:28:29] ...trap/engines/engine.py.229: process: 0.56312 | load: 0.00258 [I 2022-10-26 21:28:29] ...trap/engines/engine.py.230: loss: 0.19365 [I 2022-10-26 21:28:29] ...optimizers/trijoint.py.151: total_norm: 21.556299 [I 2022-10-26 21:28:34] ...trap/engines/engine.py.224: train: epoch 30 | batch 2100/2304 [I 2022-10-26 21:28:34] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:54 [I 2022-10-26 21:28:34] ...trap/engines/engine.py.229: process: 0.57107 | load: 0.00257 [I 2022-10-26 21:28:34] ...trap/engines/engine.py.230: loss: 0.24734 [I 2022-10-26 21:28:34] ...optimizers/trijoint.py.151: total_norm: 14.570508 [I 2022-10-26 21:28:40] ...trap/engines/engine.py.224: train: epoch 30 | batch 2110/2304 [I 2022-10-26 21:28:40] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:48 [I 2022-10-26 21:28:40] ...trap/engines/engine.py.229: process: 0.56395 | load: 0.00250 [I 2022-10-26 21:28:40] ...trap/engines/engine.py.230: loss: 0.17618 [I 2022-10-26 21:28:40] ...optimizers/trijoint.py.151: total_norm: 25.312544 [I 2022-10-26 21:28:45] ...trap/engines/engine.py.224: train: epoch 30 | batch 2120/2304 [I 2022-10-26 21:28:45] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:42 [I 2022-10-26 21:28:45] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00247 [I 2022-10-26 21:28:45] ...trap/engines/engine.py.230: loss: 0.16232 [I 2022-10-26 21:28:45] ...optimizers/trijoint.py.151: total_norm: 17.200928 [I 2022-10-26 21:28:51] ...trap/engines/engine.py.224: train: epoch 30 | batch 2130/2304 [I 2022-10-26 21:28:51] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:37 [I 2022-10-26 21:28:51] ...trap/engines/engine.py.229: process: 0.56434 | load: 0.00249 [I 2022-10-26 21:28:51] ...trap/engines/engine.py.230: loss: 0.14589 [I 2022-10-26 21:28:51] ...optimizers/trijoint.py.151: total_norm: 24.636084 [I 2022-10-26 21:28:57] ...trap/engines/engine.py.224: train: epoch 30 | batch 2140/2304 [I 2022-10-26 21:28:57] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:33 [I 2022-10-26 21:28:57] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00271 [I 2022-10-26 21:28:57] ...trap/engines/engine.py.230: loss: 0.21149 [I 2022-10-26 21:28:57] ...optimizers/trijoint.py.151: total_norm: 24.057121 [I 2022-10-26 21:29:02] ...trap/engines/engine.py.224: train: epoch 30 | batch 2150/2304 [I 2022-10-26 21:29:02] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:26 [I 2022-10-26 21:29:02] ...trap/engines/engine.py.229: process: 0.55317 | load: 0.00246 [I 2022-10-26 21:29:02] ...trap/engines/engine.py.230: loss: 0.20264 [I 2022-10-26 21:29:02] ...optimizers/trijoint.py.151: total_norm: 29.939219 [I 2022-10-26 21:29:08] ...trap/engines/engine.py.224: train: epoch 30 | batch 2160/2304 [I 2022-10-26 21:29:08] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:01:20 [I 2022-10-26 21:29:08] ...trap/engines/engine.py.229: process: 0.55436 | load: 0.00241 [I 2022-10-26 21:29:08] ...trap/engines/engine.py.230: loss: 0.15673 [I 2022-10-26 21:29:08] ...optimizers/trijoint.py.151: total_norm: 15.534788 [I 2022-10-26 21:29:14] ...trap/engines/engine.py.224: train: epoch 30 | batch 2170/2304 [I 2022-10-26 21:29:14] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:01:14 [I 2022-10-26 21:29:14] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00251 [I 2022-10-26 21:29:14] ...trap/engines/engine.py.230: loss: 0.14160 [I 2022-10-26 21:29:14] ...optimizers/trijoint.py.151: total_norm: 18.754414 [I 2022-10-26 21:29:19] ...trap/engines/engine.py.224: train: epoch 30 | batch 2180/2304 [I 2022-10-26 21:29:19] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:01:09 [I 2022-10-26 21:29:19] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00247 [I 2022-10-26 21:29:19] ...trap/engines/engine.py.230: loss: 0.20053 [I 2022-10-26 21:29:19] ...optimizers/trijoint.py.151: total_norm: 12.981024 [I 2022-10-26 21:29:25] ...trap/engines/engine.py.224: train: epoch 30 | batch 2190/2304 [I 2022-10-26 21:29:25] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:01:03 [I 2022-10-26 21:29:25] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00251 [I 2022-10-26 21:29:25] ...trap/engines/engine.py.230: loss: 0.18003 [I 2022-10-26 21:29:25] ...optimizers/trijoint.py.151: total_norm: 25.086157 [I 2022-10-26 21:29:30] ...trap/engines/engine.py.224: train: epoch 30 | batch 2200/2304 [I 2022-10-26 21:29:30] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:57 [I 2022-10-26 21:29:30] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00250 [I 2022-10-26 21:29:30] ...trap/engines/engine.py.230: loss: 0.17779 [I 2022-10-26 21:29:30] ...optimizers/trijoint.py.151: total_norm: 13.397985 [I 2022-10-26 21:29:36] ...trap/engines/engine.py.224: train: epoch 30 | batch 2210/2304 [I 2022-10-26 21:29:36] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:52 [I 2022-10-26 21:29:36] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00242 [I 2022-10-26 21:29:36] ...trap/engines/engine.py.230: loss: 0.17887 [I 2022-10-26 21:29:36] ...optimizers/trijoint.py.151: total_norm: 12.752500 [I 2022-10-26 21:29:42] ...trap/engines/engine.py.224: train: epoch 30 | batch 2220/2304 [I 2022-10-26 21:29:42] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:47 [I 2022-10-26 21:29:42] ...trap/engines/engine.py.229: process: 0.56165 | load: 0.00245 [I 2022-10-26 21:29:42] ...trap/engines/engine.py.230: loss: 0.21034 [I 2022-10-26 21:29:42] ...optimizers/trijoint.py.151: total_norm: 12.493448 [I 2022-10-26 21:29:47] ...trap/engines/engine.py.224: train: epoch 30 | batch 2230/2304 [I 2022-10-26 21:29:47] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:41 [I 2022-10-26 21:29:47] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00246 [I 2022-10-26 21:29:47] ...trap/engines/engine.py.230: loss: 0.14605 [I 2022-10-26 21:29:47] ...optimizers/trijoint.py.151: total_norm: 15.496719 [I 2022-10-26 21:29:53] ...trap/engines/engine.py.224: train: epoch 30 | batch 2240/2304 [I 2022-10-26 21:29:53] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:35 [I 2022-10-26 21:29:53] ...trap/engines/engine.py.229: process: 0.55180 | load: 0.00250 [I 2022-10-26 21:29:53] ...trap/engines/engine.py.230: loss: 0.17059 [I 2022-10-26 21:29:53] ...optimizers/trijoint.py.151: total_norm: 15.260702 [I 2022-10-26 21:29:58] ...trap/engines/engine.py.224: train: epoch 30 | batch 2250/2304 [I 2022-10-26 21:29:58] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:30 [I 2022-10-26 21:29:58] ...trap/engines/engine.py.229: process: 0.54848 | load: 0.00248 [I 2022-10-26 21:29:58] ...trap/engines/engine.py.230: loss: 0.15297 [I 2022-10-26 21:29:58] ...optimizers/trijoint.py.151: total_norm: 32.741985 [I 2022-10-26 21:30:04] ...trap/engines/engine.py.224: train: epoch 30 | batch 2260/2304 [I 2022-10-26 21:30:04] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:24 [I 2022-10-26 21:30:04] ...trap/engines/engine.py.229: process: 0.56445 | load: 0.00245 [I 2022-10-26 21:30:04] ...trap/engines/engine.py.230: loss: 0.22041 [I 2022-10-26 21:30:04] ...optimizers/trijoint.py.151: total_norm: 25.118113 [I 2022-10-26 21:30:10] ...trap/engines/engine.py.224: train: epoch 30 | batch 2270/2304 [I 2022-10-26 21:30:10] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:19 [I 2022-10-26 21:30:10] ...trap/engines/engine.py.229: process: 0.57509 | load: 0.00243 [I 2022-10-26 21:30:10] ...trap/engines/engine.py.230: loss: 0.17647 [I 2022-10-26 21:30:10] ...optimizers/trijoint.py.151: total_norm: 22.408533 [I 2022-10-26 21:30:15] ...trap/engines/engine.py.224: train: epoch 30 | batch 2280/2304 [I 2022-10-26 21:30:15] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:00:13 [I 2022-10-26 21:30:15] ...trap/engines/engine.py.229: process: 0.56591 | load: 0.00242 [I 2022-10-26 21:30:15] ...trap/engines/engine.py.230: loss: 0.20317 [I 2022-10-26 21:30:15] ...optimizers/trijoint.py.151: total_norm: 19.126139 [I 2022-10-26 21:30:21] ...trap/engines/engine.py.224: train: epoch 30 | batch 2290/2304 [I 2022-10-26 21:30:21] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:00:07 [I 2022-10-26 21:30:21] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00243 [I 2022-10-26 21:30:21] ...trap/engines/engine.py.230: loss: 0.15532 [I 2022-10-26 21:30:21] ...optimizers/trijoint.py.151: total_norm: 13.832586 [I 2022-10-26 21:30:26] ...trap/engines/engine.py.224: train: epoch 30 | batch 2300/2304 [I 2022-10-26 21:30:26] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:02 [I 2022-10-26 21:30:26] ...trap/engines/engine.py.229: process: 0.53786 | load: 0.00014 [I 2022-10-26 21:30:26] ...trap/engines/engine.py.230: loss: 0.20675 [I 2022-10-26 21:30:26] ...optimizers/trijoint.py.151: total_norm: 24.152975 [I 2022-10-26 21:30:29] ...trap/engines/engine.py.224: train: epoch 30 | batch 2304/2304 [I 2022-10-26 21:30:29] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:00 [I 2022-10-26 21:30:29] ...trap/engines/engine.py.229: process: 0.54527 | load: 0.00014 [I 2022-10-26 21:30:29] ...trap/engines/engine.py.230: loss: 0.20767 [I 2022-10-26 21:30:29] ...optimizers/trijoint.py.151: total_norm: 16.881041 [S 2022-10-26 21:30:30] ...trap/engines/engine.py.129: train_epoch.epoch: 30 [S 2022-10-26 21:30:30] ...trap/engines/engine.py.129: train_epoch.loss: 0.18023270331403957 [I 2022-10-26 21:30:30] ...s/criterions/triplet.py.84: margin = 0.20500000000000013 [I 2022-10-26 21:30:30] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 30 [I 2022-10-26 21:30:30] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 21:30:32] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 21:30:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 21:30:32] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 21:30:33] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 21:30:33] ...trap/engines/engine.py.321: eval: epoch 30 | batch 0/510 [I 2022-10-26 21:30:33] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:40 [I 2022-10-26 21:30:33] ...trap/engines/engine.py.326: process: 3.25620 | load: 2.89571 [I 2022-10-26 21:30:37] ...trap/engines/engine.py.321: eval: epoch 30 | batch 10/510 [I 2022-10-26 21:30:37] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:24 [I 2022-10-26 21:30:37] ...trap/engines/engine.py.326: process: 0.33015 | load: 0.00017 [I 2022-10-26 21:30:40] ...trap/engines/engine.py.321: eval: epoch 30 | batch 20/510 [I 2022-10-26 21:30:40] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:58 [I 2022-10-26 21:30:40] ...trap/engines/engine.py.326: process: 0.33932 | load: 0.00018 [I 2022-10-26 21:30:43] ...trap/engines/engine.py.321: eval: epoch 30 | batch 30/510 [I 2022-10-26 21:30:43] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:40 [I 2022-10-26 21:30:43] ...trap/engines/engine.py.326: process: 0.32597 | load: 0.00018 [I 2022-10-26 21:30:47] ...trap/engines/engine.py.321: eval: epoch 30 | batch 40/510 [I 2022-10-26 21:30:47] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:37 [I 2022-10-26 21:30:47] ...trap/engines/engine.py.326: process: 0.32634 | load: 0.00017 [I 2022-10-26 21:30:50] ...trap/engines/engine.py.321: eval: epoch 30 | batch 50/510 [I 2022-10-26 21:30:50] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:32 [I 2022-10-26 21:30:50] ...trap/engines/engine.py.326: process: 0.32934 | load: 0.00019 [I 2022-10-26 21:30:54] ...trap/engines/engine.py.321: eval: epoch 30 | batch 60/510 [I 2022-10-26 21:30:54] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:32 [I 2022-10-26 21:30:54] ...trap/engines/engine.py.326: process: 0.35240 | load: 0.00017 [I 2022-10-26 21:30:57] ...trap/engines/engine.py.321: eval: epoch 30 | batch 70/510 [I 2022-10-26 21:30:57] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:27 [I 2022-10-26 21:30:57] ...trap/engines/engine.py.326: process: 0.33119 | load: 0.00018 [I 2022-10-26 21:31:00] ...trap/engines/engine.py.321: eval: epoch 30 | batch 80/510 [I 2022-10-26 21:31:00] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-26 21:31:00] ...trap/engines/engine.py.326: process: 0.32897 | load: 0.00019 [I 2022-10-26 21:31:04] ...trap/engines/engine.py.321: eval: epoch 30 | batch 90/510 [I 2022-10-26 21:31:04] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:23 [I 2022-10-26 21:31:04] ...trap/engines/engine.py.326: process: 0.35296 | load: 0.00018 [I 2022-10-26 21:31:07] ...trap/engines/engine.py.321: eval: epoch 30 | batch 100/510 [I 2022-10-26 21:31:07] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:14 [I 2022-10-26 21:31:07] ...trap/engines/engine.py.326: process: 0.28717 | load: 0.00016 [I 2022-10-26 21:31:10] ...trap/engines/engine.py.321: eval: epoch 30 | batch 110/510 [I 2022-10-26 21:31:10] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-26 21:31:10] ...trap/engines/engine.py.326: process: 0.29222 | load: 0.00017 [I 2022-10-26 21:31:13] ...trap/engines/engine.py.321: eval: epoch 30 | batch 120/510 [I 2022-10-26 21:31:13] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-26 21:31:13] ...trap/engines/engine.py.326: process: 0.28784 | load: 0.00014 [I 2022-10-26 21:31:16] ...trap/engines/engine.py.321: eval: epoch 30 | batch 130/510 [I 2022-10-26 21:31:16] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:49 [I 2022-10-26 21:31:16] ...trap/engines/engine.py.326: process: 0.28648 | load: 0.00016 [I 2022-10-26 21:31:19] ...trap/engines/engine.py.321: eval: epoch 30 | batch 140/510 [I 2022-10-26 21:31:19] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-26 21:31:19] ...trap/engines/engine.py.326: process: 0.29055 | load: 0.00015 [I 2022-10-26 21:31:21] ...trap/engines/engine.py.321: eval: epoch 30 | batch 150/510 [I 2022-10-26 21:31:21] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 21:31:21] ...trap/engines/engine.py.326: process: 0.29086 | load: 0.00017 [I 2022-10-26 21:31:24] ...trap/engines/engine.py.321: eval: epoch 30 | batch 160/510 [I 2022-10-26 21:31:24] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-26 21:31:24] ...trap/engines/engine.py.326: process: 0.28923 | load: 0.00019 [I 2022-10-26 21:31:27] ...trap/engines/engine.py.321: eval: epoch 30 | batch 170/510 [I 2022-10-26 21:31:27] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-26 21:31:27] ...trap/engines/engine.py.326: process: 0.28236 | load: 0.00016 [I 2022-10-26 21:31:30] ...trap/engines/engine.py.321: eval: epoch 30 | batch 180/510 [I 2022-10-26 21:31:30] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-26 21:31:30] ...trap/engines/engine.py.326: process: 0.29211 | load: 0.00016 [I 2022-10-26 21:31:33] ...trap/engines/engine.py.321: eval: epoch 30 | batch 190/510 [I 2022-10-26 21:31:33] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 21:31:33] ...trap/engines/engine.py.326: process: 0.28893 | load: 0.00022 [I 2022-10-26 21:31:36] ...trap/engines/engine.py.321: eval: epoch 30 | batch 200/510 [I 2022-10-26 21:31:36] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 21:31:36] ...trap/engines/engine.py.326: process: 0.28571 | load: 0.00020 [I 2022-10-26 21:31:39] ...trap/engines/engine.py.321: eval: epoch 30 | batch 210/510 [I 2022-10-26 21:31:39] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 21:31:39] ...trap/engines/engine.py.326: process: 0.28303 | load: 0.00021 [I 2022-10-26 21:31:42] ...trap/engines/engine.py.321: eval: epoch 30 | batch 220/510 [I 2022-10-26 21:31:42] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 21:31:42] ...trap/engines/engine.py.326: process: 0.28773 | load: 0.00015 [I 2022-10-26 21:31:45] ...trap/engines/engine.py.321: eval: epoch 30 | batch 230/510 [I 2022-10-26 21:31:45] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-26 21:31:45] ...trap/engines/engine.py.326: process: 0.29399 | load: 0.00016 [I 2022-10-26 21:31:47] ...trap/engines/engine.py.321: eval: epoch 30 | batch 240/510 [I 2022-10-26 21:31:47] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 21:31:47] ...trap/engines/engine.py.326: process: 0.28638 | load: 0.00015 [I 2022-10-26 21:31:50] ...trap/engines/engine.py.321: eval: epoch 30 | batch 250/510 [I 2022-10-26 21:31:50] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-26 21:31:50] ...trap/engines/engine.py.326: process: 0.28958 | load: 0.00021 [I 2022-10-26 21:31:53] ...trap/engines/engine.py.321: eval: epoch 30 | batch 260/510 [I 2022-10-26 21:31:53] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-26 21:31:53] ...trap/engines/engine.py.326: process: 0.29527 | load: 0.00019 [I 2022-10-26 21:31:56] ...trap/engines/engine.py.321: eval: epoch 30 | batch 270/510 [I 2022-10-26 21:31:56] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-26 21:31:56] ...trap/engines/engine.py.326: process: 0.27865 | load: 0.00016 [I 2022-10-26 21:31:59] ...trap/engines/engine.py.321: eval: epoch 30 | batch 280/510 [I 2022-10-26 21:31:59] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-26 21:31:59] ...trap/engines/engine.py.326: process: 0.28799 | load: 0.00015 [I 2022-10-26 21:32:02] ...trap/engines/engine.py.321: eval: epoch 30 | batch 290/510 [I 2022-10-26 21:32:02] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 21:32:02] ...trap/engines/engine.py.326: process: 0.28705 | load: 0.00015 [I 2022-10-26 21:32:05] ...trap/engines/engine.py.321: eval: epoch 30 | batch 300/510 [I 2022-10-26 21:32:05] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:01 [I 2022-10-26 21:32:05] ...trap/engines/engine.py.326: process: 0.29370 | load: 0.00016 [I 2022-10-26 21:32:08] ...trap/engines/engine.py.321: eval: epoch 30 | batch 310/510 [I 2022-10-26 21:32:08] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 21:32:08] ...trap/engines/engine.py.326: process: 0.28808 | load: 0.00016 [I 2022-10-26 21:32:11] ...trap/engines/engine.py.321: eval: epoch 30 | batch 320/510 [I 2022-10-26 21:32:11] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:55 [I 2022-10-26 21:32:11] ...trap/engines/engine.py.326: process: 0.29096 | load: 0.00021 [I 2022-10-26 21:32:14] ...trap/engines/engine.py.321: eval: epoch 30 | batch 330/510 [I 2022-10-26 21:32:14] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:52 [I 2022-10-26 21:32:14] ...trap/engines/engine.py.326: process: 0.29919 | load: 0.00018 [I 2022-10-26 21:32:16] ...trap/engines/engine.py.321: eval: epoch 30 | batch 340/510 [I 2022-10-26 21:32:16] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:49 [I 2022-10-26 21:32:16] ...trap/engines/engine.py.326: process: 0.29116 | load: 0.00019 [I 2022-10-26 21:32:19] ...trap/engines/engine.py.321: eval: epoch 30 | batch 350/510 [I 2022-10-26 21:32:19] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 21:32:19] ...trap/engines/engine.py.326: process: 0.29321 | load: 0.00020 [I 2022-10-26 21:32:22] ...trap/engines/engine.py.321: eval: epoch 30 | batch 360/510 [I 2022-10-26 21:32:22] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 21:32:22] ...trap/engines/engine.py.326: process: 0.28748 | load: 0.00015 [I 2022-10-26 21:32:25] ...trap/engines/engine.py.321: eval: epoch 30 | batch 370/510 [I 2022-10-26 21:32:25] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 21:32:25] ...trap/engines/engine.py.326: process: 0.28811 | load: 0.00015 [I 2022-10-26 21:32:28] ...trap/engines/engine.py.321: eval: epoch 30 | batch 380/510 [I 2022-10-26 21:32:28] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 21:32:28] ...trap/engines/engine.py.326: process: 0.29194 | load: 0.00015 [I 2022-10-26 21:32:31] ...trap/engines/engine.py.321: eval: epoch 30 | batch 390/510 [I 2022-10-26 21:32:31] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 21:32:31] ...trap/engines/engine.py.326: process: 0.29104 | load: 0.00016 [I 2022-10-26 21:32:34] ...trap/engines/engine.py.321: eval: epoch 30 | batch 400/510 [I 2022-10-26 21:32:34] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 21:32:34] ...trap/engines/engine.py.326: process: 0.29167 | load: 0.00018 [I 2022-10-26 21:32:37] ...trap/engines/engine.py.321: eval: epoch 30 | batch 410/510 [I 2022-10-26 21:32:37] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 21:32:37] ...trap/engines/engine.py.326: process: 0.28459 | load: 0.00016 [I 2022-10-26 21:32:40] ...trap/engines/engine.py.321: eval: epoch 30 | batch 420/510 [I 2022-10-26 21:32:40] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 21:32:40] ...trap/engines/engine.py.326: process: 0.29411 | load: 0.00015 [I 2022-10-26 21:32:42] ...trap/engines/engine.py.321: eval: epoch 30 | batch 430/510 [I 2022-10-26 21:32:42] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 21:32:42] ...trap/engines/engine.py.326: process: 0.29163 | load: 0.00019 [I 2022-10-26 21:32:45] ...trap/engines/engine.py.321: eval: epoch 30 | batch 440/510 [I 2022-10-26 21:32:45] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 21:32:45] ...trap/engines/engine.py.326: process: 0.29363 | load: 0.00015 [I 2022-10-26 21:32:48] ...trap/engines/engine.py.321: eval: epoch 30 | batch 450/510 [I 2022-10-26 21:32:48] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 21:32:48] ...trap/engines/engine.py.326: process: 0.28917 | load: 0.00024 [I 2022-10-26 21:32:51] ...trap/engines/engine.py.321: eval: epoch 30 | batch 460/510 [I 2022-10-26 21:32:51] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 21:32:51] ...trap/engines/engine.py.326: process: 0.28510 | load: 0.00016 [I 2022-10-26 21:32:54] ...trap/engines/engine.py.321: eval: epoch 30 | batch 470/510 [I 2022-10-26 21:32:54] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 21:32:54] ...trap/engines/engine.py.326: process: 0.28827 | load: 0.00020 [I 2022-10-26 21:32:57] ...trap/engines/engine.py.321: eval: epoch 30 | batch 480/510 [I 2022-10-26 21:32:57] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 21:32:57] ...trap/engines/engine.py.326: process: 0.28749 | load: 0.00017 [I 2022-10-26 21:33:00] ...trap/engines/engine.py.321: eval: epoch 30 | batch 490/510 [I 2022-10-26 21:33:00] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 21:33:00] ...trap/engines/engine.py.326: process: 0.28726 | load: 0.00019 [I 2022-10-26 21:33:03] ...trap/engines/engine.py.321: eval: epoch 30 | batch 500/510 [I 2022-10-26 21:33:03] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 21:33:03] ...trap/engines/engine.py.326: process: 0.28564 | load: 0.00015 [I 2022-10-26 21:33:05] ...trap/engines/engine.py.321: eval: epoch 30 | batch 510/510 [I 2022-10-26 21:33:05] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 21:33:05] ...trap/engines/engine.py.326: process: 0.28588 | load: 0.00016 [S 2022-10-26 21:33:06] ...trap/engines/engine.py.132: eval_epoch.epoch: 30 [I 2022-10-26 21:33:06] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.660200 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.880400 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.923600 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.653600 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.874900 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.917800 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012221 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010528 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009531 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013887 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009513 [S 2022-10-26 21:33:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005896 [I 2022-10-26 21:33:11] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 21:33:11] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 21:33:11] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 21:33:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 21:33:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 21:33:12] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 21:33:15] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 21:33:21] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 21:33:21] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 21:33:21] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 21:33:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 21:33:30] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 21:33:30] ...trap/engines/engine.py.163: Training model on trainset for epoch 31 [I 2022-10-26 21:33:30] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 21:33:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 0/2304 [I 2022-10-26 21:33:33] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:28 [I 2022-10-26 21:33:33] ...trap/engines/engine.py.229: process: 2.72067 | load: 2.07717 [I 2022-10-26 21:33:33] ...trap/engines/engine.py.230: loss: 0.12134 [I 2022-10-26 21:33:33] ...optimizers/trijoint.py.151: total_norm: 19.885532 [I 2022-10-26 21:33:38] ...trap/engines/engine.py.224: train: epoch 31 | batch 10/2304 [I 2022-10-26 21:33:38] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:15 [I 2022-10-26 21:33:38] ...trap/engines/engine.py.229: process: 0.55848 | load: 0.00438 [I 2022-10-26 21:33:38] ...trap/engines/engine.py.230: loss: 0.17926 [I 2022-10-26 21:33:38] ...optimizers/trijoint.py.151: total_norm: 29.391468 [I 2022-10-26 21:33:44] ...trap/engines/engine.py.224: train: epoch 31 | batch 20/2304 [I 2022-10-26 21:33:44] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:15 [I 2022-10-26 21:33:44] ...trap/engines/engine.py.229: process: 0.57318 | load: 0.00354 [I 2022-10-26 21:33:44] ...trap/engines/engine.py.230: loss: 0.16799 [I 2022-10-26 21:33:44] ...optimizers/trijoint.py.151: total_norm: 30.388721 [I 2022-10-26 21:33:49] ...trap/engines/engine.py.224: train: epoch 31 | batch 30/2304 [I 2022-10-26 21:33:49] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:09 [I 2022-10-26 21:33:49] ...trap/engines/engine.py.229: process: 0.55026 | load: 0.00322 [I 2022-10-26 21:33:49] ...trap/engines/engine.py.230: loss: 0.17857 [I 2022-10-26 21:33:49] ...optimizers/trijoint.py.151: total_norm: 15.608587 [I 2022-10-26 21:33:55] ...trap/engines/engine.py.224: train: epoch 31 | batch 40/2304 [I 2022-10-26 21:33:55] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:01 [I 2022-10-26 21:33:55] ...trap/engines/engine.py.229: process: 0.56192 | load: 0.00310 [I 2022-10-26 21:33:55] ...trap/engines/engine.py.230: loss: 0.28440 [I 2022-10-26 21:33:55] ...optimizers/trijoint.py.151: total_norm: 16.615166 [I 2022-10-26 21:34:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 50/2304 [I 2022-10-26 21:34:01] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:58 [I 2022-10-26 21:34:01] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00292 [I 2022-10-26 21:34:01] ...trap/engines/engine.py.230: loss: 0.16653 [I 2022-10-26 21:34:01] ...optimizers/trijoint.py.151: total_norm: 11.502463 [I 2022-10-26 21:34:06] ...trap/engines/engine.py.224: train: epoch 31 | batch 60/2304 [I 2022-10-26 21:34:06] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:02 [I 2022-10-26 21:34:06] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00264 [I 2022-10-26 21:34:06] ...trap/engines/engine.py.230: loss: 0.14555 [I 2022-10-26 21:34:06] ...optimizers/trijoint.py.151: total_norm: 12.882729 [I 2022-10-26 21:34:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 70/2304 [I 2022-10-26 21:34:12] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:45 [I 2022-10-26 21:34:12] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00267 [I 2022-10-26 21:34:12] ...trap/engines/engine.py.230: loss: 0.20350 [I 2022-10-26 21:34:12] ...optimizers/trijoint.py.151: total_norm: 19.416952 [I 2022-10-26 21:34:17] ...trap/engines/engine.py.224: train: epoch 31 | batch 80/2304 [I 2022-10-26 21:34:17] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:48 [I 2022-10-26 21:34:17] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00288 [I 2022-10-26 21:34:17] ...trap/engines/engine.py.230: loss: 0.19183 [I 2022-10-26 21:34:17] ...optimizers/trijoint.py.151: total_norm: 25.842543 [I 2022-10-26 21:34:23] ...trap/engines/engine.py.224: train: epoch 31 | batch 90/2304 [I 2022-10-26 21:34:23] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:32 [I 2022-10-26 21:34:23] ...trap/engines/engine.py.229: process: 0.55066 | load: 0.00276 [I 2022-10-26 21:34:23] ...trap/engines/engine.py.230: loss: 0.13816 [I 2022-10-26 21:34:23] ...optimizers/trijoint.py.151: total_norm: 15.775819 [I 2022-10-26 21:34:29] ...trap/engines/engine.py.224: train: epoch 31 | batch 100/2304 [I 2022-10-26 21:34:29] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:29 [I 2022-10-26 21:34:29] ...trap/engines/engine.py.229: process: 0.56438 | load: 0.00263 [I 2022-10-26 21:34:29] ...trap/engines/engine.py.230: loss: 0.25433 [I 2022-10-26 21:34:29] ...optimizers/trijoint.py.151: total_norm: 26.491812 [I 2022-10-26 21:34:34] ...trap/engines/engine.py.224: train: epoch 31 | batch 110/2304 [I 2022-10-26 21:34:34] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:14 [I 2022-10-26 21:34:34] ...trap/engines/engine.py.229: process: 0.54533 | load: 0.00253 [I 2022-10-26 21:34:34] ...trap/engines/engine.py.230: loss: 0.24343 [I 2022-10-26 21:34:34] ...optimizers/trijoint.py.151: total_norm: 30.231079 [I 2022-10-26 21:34:40] ...trap/engines/engine.py.224: train: epoch 31 | batch 120/2304 [I 2022-10-26 21:34:40] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:15 [I 2022-10-26 21:34:40] ...trap/engines/engine.py.229: process: 0.56497 | load: 0.00263 [I 2022-10-26 21:34:40] ...trap/engines/engine.py.230: loss: 0.14532 [I 2022-10-26 21:34:40] ...optimizers/trijoint.py.151: total_norm: 11.508591 [I 2022-10-26 21:34:45] ...trap/engines/engine.py.224: train: epoch 31 | batch 130/2304 [I 2022-10-26 21:34:45] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:03 [I 2022-10-26 21:34:45] ...trap/engines/engine.py.229: process: 0.55474 | load: 0.00257 [I 2022-10-26 21:34:45] ...trap/engines/engine.py.230: loss: 0.25726 [I 2022-10-26 21:34:45] ...optimizers/trijoint.py.151: total_norm: 29.271296 [I 2022-10-26 21:34:51] ...trap/engines/engine.py.224: train: epoch 31 | batch 140/2304 [I 2022-10-26 21:34:51] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:08 [I 2022-10-26 21:34:51] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00240 [I 2022-10-26 21:34:51] ...trap/engines/engine.py.230: loss: 0.13129 [I 2022-10-26 21:34:51] ...optimizers/trijoint.py.151: total_norm: 18.554438 [I 2022-10-26 21:34:57] ...trap/engines/engine.py.224: train: epoch 31 | batch 150/2304 [I 2022-10-26 21:34:57] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:08 [I 2022-10-26 21:34:57] ...trap/engines/engine.py.229: process: 0.56487 | load: 0.00250 [I 2022-10-26 21:34:57] ...trap/engines/engine.py.230: loss: 0.13156 [I 2022-10-26 21:34:57] ...optimizers/trijoint.py.151: total_norm: 11.495503 [I 2022-10-26 21:35:02] ...trap/engines/engine.py.224: train: epoch 31 | batch 160/2304 [I 2022-10-26 21:35:02] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:08 [I 2022-10-26 21:35:02] ...trap/engines/engine.py.229: process: 0.56647 | load: 0.00258 [I 2022-10-26 21:35:02] ...trap/engines/engine.py.230: loss: 0.17621 [I 2022-10-26 21:35:02] ...optimizers/trijoint.py.151: total_norm: 20.994030 [I 2022-10-26 21:35:08] ...trap/engines/engine.py.224: train: epoch 31 | batch 170/2304 [I 2022-10-26 21:35:08] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:20:02 [I 2022-10-26 21:35:08] ...trap/engines/engine.py.229: process: 0.56682 | load: 0.00264 [I 2022-10-26 21:35:08] ...trap/engines/engine.py.230: loss: 0.18257 [I 2022-10-26 21:35:08] ...optimizers/trijoint.py.151: total_norm: 25.818745 [I 2022-10-26 21:35:13] ...trap/engines/engine.py.224: train: epoch 31 | batch 180/2304 [I 2022-10-26 21:35:13] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:43 [I 2022-10-26 21:35:13] ...trap/engines/engine.py.229: process: 0.54610 | load: 0.00246 [I 2022-10-26 21:35:13] ...trap/engines/engine.py.230: loss: 0.32719 [I 2022-10-26 21:35:13] ...optimizers/trijoint.py.151: total_norm: 25.527271 [I 2022-10-26 21:35:19] ...trap/engines/engine.py.224: train: epoch 31 | batch 190/2304 [I 2022-10-26 21:35:19] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:48 [I 2022-10-26 21:35:19] ...trap/engines/engine.py.229: process: 0.56756 | load: 0.00263 [I 2022-10-26 21:35:19] ...trap/engines/engine.py.230: loss: 0.10536 [I 2022-10-26 21:35:19] ...optimizers/trijoint.py.151: total_norm: 11.942551 [I 2022-10-26 21:35:25] ...trap/engines/engine.py.224: train: epoch 31 | batch 200/2304 [I 2022-10-26 21:35:25] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:31 [I 2022-10-26 21:35:25] ...trap/engines/engine.py.229: process: 0.55010 | load: 0.00255 [I 2022-10-26 21:35:25] ...trap/engines/engine.py.230: loss: 0.16934 [I 2022-10-26 21:35:25] ...optimizers/trijoint.py.151: total_norm: 14.458067 [I 2022-10-26 21:35:30] ...trap/engines/engine.py.224: train: epoch 31 | batch 210/2304 [I 2022-10-26 21:35:30] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:25 [I 2022-10-26 21:35:30] ...trap/engines/engine.py.229: process: 0.54358 | load: 0.00247 [I 2022-10-26 21:35:30] ...trap/engines/engine.py.230: loss: 0.14538 [I 2022-10-26 21:35:30] ...optimizers/trijoint.py.151: total_norm: 19.514574 [I 2022-10-26 21:35:36] ...trap/engines/engine.py.224: train: epoch 31 | batch 220/2304 [I 2022-10-26 21:35:36] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:24 [I 2022-10-26 21:35:36] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00247 [I 2022-10-26 21:35:36] ...trap/engines/engine.py.230: loss: 0.23464 [I 2022-10-26 21:35:36] ...optimizers/trijoint.py.151: total_norm: 25.817038 [I 2022-10-26 21:35:41] ...trap/engines/engine.py.224: train: epoch 31 | batch 230/2304 [I 2022-10-26 21:35:41] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:19 [I 2022-10-26 21:35:41] ...trap/engines/engine.py.229: process: 0.57136 | load: 0.00267 [I 2022-10-26 21:35:41] ...trap/engines/engine.py.230: loss: 0.28482 [I 2022-10-26 21:35:41] ...optimizers/trijoint.py.151: total_norm: 29.580971 [I 2022-10-26 21:35:47] ...trap/engines/engine.py.224: train: epoch 31 | batch 240/2304 [I 2022-10-26 21:35:47] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:22 [I 2022-10-26 21:35:47] ...trap/engines/engine.py.229: process: 0.54767 | load: 0.00248 [I 2022-10-26 21:35:47] ...trap/engines/engine.py.230: loss: 0.19291 [I 2022-10-26 21:35:47] ...optimizers/trijoint.py.151: total_norm: 16.053040 [I 2022-10-26 21:35:53] ...trap/engines/engine.py.224: train: epoch 31 | batch 250/2304 [I 2022-10-26 21:35:53] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:14 [I 2022-10-26 21:35:53] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00260 [I 2022-10-26 21:35:53] ...trap/engines/engine.py.230: loss: 0.10600 [I 2022-10-26 21:35:53] ...optimizers/trijoint.py.151: total_norm: 17.295862 [I 2022-10-26 21:35:58] ...trap/engines/engine.py.224: train: epoch 31 | batch 260/2304 [I 2022-10-26 21:35:58] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:00 [I 2022-10-26 21:35:58] ...trap/engines/engine.py.229: process: 0.56894 | load: 0.00259 [I 2022-10-26 21:35:58] ...trap/engines/engine.py.230: loss: 0.18133 [I 2022-10-26 21:35:58] ...optimizers/trijoint.py.151: total_norm: 25.268782 [I 2022-10-26 21:36:04] ...trap/engines/engine.py.224: train: epoch 31 | batch 270/2304 [I 2022-10-26 21:36:04] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:53 [I 2022-10-26 21:36:04] ...trap/engines/engine.py.229: process: 0.55494 | load: 0.00251 [I 2022-10-26 21:36:04] ...trap/engines/engine.py.230: loss: 0.28000 [I 2022-10-26 21:36:04] ...optimizers/trijoint.py.151: total_norm: 17.198664 [I 2022-10-26 21:36:09] ...trap/engines/engine.py.224: train: epoch 31 | batch 280/2304 [I 2022-10-26 21:36:09] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:40 [I 2022-10-26 21:36:09] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00270 [I 2022-10-26 21:36:09] ...trap/engines/engine.py.230: loss: 0.21111 [I 2022-10-26 21:36:09] ...optimizers/trijoint.py.151: total_norm: 28.559914 [I 2022-10-26 21:36:15] ...trap/engines/engine.py.224: train: epoch 31 | batch 290/2304 [I 2022-10-26 21:36:15] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:42 [I 2022-10-26 21:36:15] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00243 [I 2022-10-26 21:36:15] ...trap/engines/engine.py.230: loss: 0.15943 [I 2022-10-26 21:36:15] ...optimizers/trijoint.py.151: total_norm: 20.478722 [I 2022-10-26 21:36:21] ...trap/engines/engine.py.224: train: epoch 31 | batch 300/2304 [I 2022-10-26 21:36:21] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:36 [I 2022-10-26 21:36:21] ...trap/engines/engine.py.229: process: 0.56734 | load: 0.00261 [I 2022-10-26 21:36:21] ...trap/engines/engine.py.230: loss: 0.27151 [I 2022-10-26 21:36:21] ...optimizers/trijoint.py.151: total_norm: 25.157349 [I 2022-10-26 21:36:26] ...trap/engines/engine.py.224: train: epoch 31 | batch 310/2304 [I 2022-10-26 21:36:26] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:49 [I 2022-10-26 21:36:26] ...trap/engines/engine.py.229: process: 0.56157 | load: 0.00253 [I 2022-10-26 21:36:26] ...trap/engines/engine.py.230: loss: 0.16877 [I 2022-10-26 21:36:26] ...optimizers/trijoint.py.151: total_norm: 23.969254 [I 2022-10-26 21:36:32] ...trap/engines/engine.py.224: train: epoch 31 | batch 320/2304 [I 2022-10-26 21:36:32] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:29 [I 2022-10-26 21:36:32] ...trap/engines/engine.py.229: process: 0.55072 | load: 0.00246 [I 2022-10-26 21:36:32] ...trap/engines/engine.py.230: loss: 0.14228 [I 2022-10-26 21:36:32] ...optimizers/trijoint.py.151: total_norm: 33.347309 [I 2022-10-26 21:36:37] ...trap/engines/engine.py.224: train: epoch 31 | batch 330/2304 [I 2022-10-26 21:36:37] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:17 [I 2022-10-26 21:36:37] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00249 [I 2022-10-26 21:36:37] ...trap/engines/engine.py.230: loss: 0.12223 [I 2022-10-26 21:36:37] ...optimizers/trijoint.py.151: total_norm: 10.111341 [I 2022-10-26 21:36:43] ...trap/engines/engine.py.224: train: epoch 31 | batch 340/2304 [I 2022-10-26 21:36:43] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:04 [I 2022-10-26 21:36:43] ...trap/engines/engine.py.229: process: 0.55026 | load: 0.00253 [I 2022-10-26 21:36:43] ...trap/engines/engine.py.230: loss: 0.20865 [I 2022-10-26 21:36:43] ...optimizers/trijoint.py.151: total_norm: 19.568953 [I 2022-10-26 21:36:49] ...trap/engines/engine.py.224: train: epoch 31 | batch 350/2304 [I 2022-10-26 21:36:49] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:17:58 [I 2022-10-26 21:36:49] ...trap/engines/engine.py.229: process: 0.55790 | load: 0.00260 [I 2022-10-26 21:36:49] ...trap/engines/engine.py.230: loss: 0.17692 [I 2022-10-26 21:36:49] ...optimizers/trijoint.py.151: total_norm: 14.424627 [I 2022-10-26 21:36:54] ...trap/engines/engine.py.224: train: epoch 31 | batch 360/2304 [I 2022-10-26 21:36:54] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:51 [I 2022-10-26 21:36:54] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00240 [I 2022-10-26 21:36:54] ...trap/engines/engine.py.230: loss: 0.16437 [I 2022-10-26 21:36:54] ...optimizers/trijoint.py.151: total_norm: 19.452316 [I 2022-10-26 21:37:00] ...trap/engines/engine.py.224: train: epoch 31 | batch 370/2304 [I 2022-10-26 21:37:00] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:48 [I 2022-10-26 21:37:00] ...trap/engines/engine.py.229: process: 0.54016 | load: 0.00254 [I 2022-10-26 21:37:00] ...trap/engines/engine.py.230: loss: 0.17021 [I 2022-10-26 21:37:00] ...optimizers/trijoint.py.151: total_norm: 15.914997 [I 2022-10-26 21:37:05] ...trap/engines/engine.py.224: train: epoch 31 | batch 380/2304 [I 2022-10-26 21:37:05] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:54 [I 2022-10-26 21:37:05] ...trap/engines/engine.py.229: process: 0.54986 | load: 0.00253 [I 2022-10-26 21:37:05] ...trap/engines/engine.py.230: loss: 0.18633 [I 2022-10-26 21:37:05] ...optimizers/trijoint.py.151: total_norm: 20.348669 [I 2022-10-26 21:37:11] ...trap/engines/engine.py.224: train: epoch 31 | batch 390/2304 [I 2022-10-26 21:37:11] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:47 [I 2022-10-26 21:37:11] ...trap/engines/engine.py.229: process: 0.54349 | load: 0.00249 [I 2022-10-26 21:37:11] ...trap/engines/engine.py.230: loss: 0.20187 [I 2022-10-26 21:37:11] ...optimizers/trijoint.py.151: total_norm: 29.578632 [I 2022-10-26 21:37:16] ...trap/engines/engine.py.224: train: epoch 31 | batch 400/2304 [I 2022-10-26 21:37:16] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:35 [I 2022-10-26 21:37:16] ...trap/engines/engine.py.229: process: 0.54889 | load: 0.00244 [I 2022-10-26 21:37:16] ...trap/engines/engine.py.230: loss: 0.15540 [I 2022-10-26 21:37:16] ...optimizers/trijoint.py.151: total_norm: 26.705116 [I 2022-10-26 21:37:22] ...trap/engines/engine.py.224: train: epoch 31 | batch 410/2304 [I 2022-10-26 21:37:22] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:31 [I 2022-10-26 21:37:22] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00242 [I 2022-10-26 21:37:22] ...trap/engines/engine.py.230: loss: 0.22707 [I 2022-10-26 21:37:22] ...optimizers/trijoint.py.151: total_norm: 27.591782 [I 2022-10-26 21:37:28] ...trap/engines/engine.py.224: train: epoch 31 | batch 420/2304 [I 2022-10-26 21:37:28] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:23 [I 2022-10-26 21:37:28] ...trap/engines/engine.py.229: process: 0.55417 | load: 0.00243 [I 2022-10-26 21:37:28] ...trap/engines/engine.py.230: loss: 0.26363 [I 2022-10-26 21:37:28] ...optimizers/trijoint.py.151: total_norm: 41.679310 [I 2022-10-26 21:37:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 430/2304 [I 2022-10-26 21:37:33] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:19 [I 2022-10-26 21:37:33] ...trap/engines/engine.py.229: process: 0.55364 | load: 0.00242 [I 2022-10-26 21:37:33] ...trap/engines/engine.py.230: loss: 0.22204 [I 2022-10-26 21:37:33] ...optimizers/trijoint.py.151: total_norm: 25.939362 [I 2022-10-26 21:37:39] ...trap/engines/engine.py.224: train: epoch 31 | batch 440/2304 [I 2022-10-26 21:37:39] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:20 [I 2022-10-26 21:37:39] ...trap/engines/engine.py.229: process: 0.57040 | load: 0.00241 [I 2022-10-26 21:37:39] ...trap/engines/engine.py.230: loss: 0.21333 [I 2022-10-26 21:37:39] ...optimizers/trijoint.py.151: total_norm: 17.504749 [I 2022-10-26 21:37:44] ...trap/engines/engine.py.224: train: epoch 31 | batch 450/2304 [I 2022-10-26 21:37:44] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:16 [I 2022-10-26 21:37:44] ...trap/engines/engine.py.229: process: 0.55926 | load: 0.00269 [I 2022-10-26 21:37:44] ...trap/engines/engine.py.230: loss: 0.09562 [I 2022-10-26 21:37:44] ...optimizers/trijoint.py.151: total_norm: 11.246456 [I 2022-10-26 21:37:50] ...trap/engines/engine.py.224: train: epoch 31 | batch 460/2304 [I 2022-10-26 21:37:50] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:13 [I 2022-10-26 21:37:50] ...trap/engines/engine.py.229: process: 0.56324 | load: 0.00241 [I 2022-10-26 21:37:50] ...trap/engines/engine.py.230: loss: 0.17704 [I 2022-10-26 21:37:50] ...optimizers/trijoint.py.151: total_norm: 16.018730 [I 2022-10-26 21:37:56] ...trap/engines/engine.py.224: train: epoch 31 | batch 470/2304 [I 2022-10-26 21:37:56] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:57 [I 2022-10-26 21:37:56] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00240 [I 2022-10-26 21:37:56] ...trap/engines/engine.py.230: loss: 0.26745 [I 2022-10-26 21:37:56] ...optimizers/trijoint.py.151: total_norm: 26.364946 [I 2022-10-26 21:38:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 480/2304 [I 2022-10-26 21:38:01] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:59 [I 2022-10-26 21:38:01] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00247 [I 2022-10-26 21:38:01] ...trap/engines/engine.py.230: loss: 0.17461 [I 2022-10-26 21:38:01] ...optimizers/trijoint.py.151: total_norm: 16.655176 [I 2022-10-26 21:38:07] ...trap/engines/engine.py.224: train: epoch 31 | batch 490/2304 [I 2022-10-26 21:38:07] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:53 [I 2022-10-26 21:38:07] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00246 [I 2022-10-26 21:38:07] ...trap/engines/engine.py.230: loss: 0.20821 [I 2022-10-26 21:38:07] ...optimizers/trijoint.py.151: total_norm: 20.287043 [I 2022-10-26 21:38:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 500/2304 [I 2022-10-26 21:38:12] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:38 [I 2022-10-26 21:38:12] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00243 [I 2022-10-26 21:38:12] ...trap/engines/engine.py.230: loss: 0.27773 [I 2022-10-26 21:38:12] ...optimizers/trijoint.py.151: total_norm: 37.865150 [I 2022-10-26 21:38:38] ...trap/engines/engine.py.224: train: epoch 31 | batch 510/2304 [I 2022-10-26 21:38:38] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:37:06 [I 2022-10-26 21:38:38] ...trap/engines/engine.py.229: process: 0.57051 | load: 0.00236 [I 2022-10-26 21:38:38] ...trap/engines/engine.py.230: loss: 0.17108 [I 2022-10-26 21:38:38] ...optimizers/trijoint.py.151: total_norm: 19.352856 [I 2022-10-26 21:38:44] ...trap/engines/engine.py.224: train: epoch 31 | batch 520/2304 [I 2022-10-26 21:38:44] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:18:41 [I 2022-10-26 21:38:44] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00241 [I 2022-10-26 21:38:44] ...trap/engines/engine.py.230: loss: 0.22168 [I 2022-10-26 21:38:44] ...optimizers/trijoint.py.151: total_norm: 20.225443 [I 2022-10-26 21:38:49] ...trap/engines/engine.py.224: train: epoch 31 | batch 530/2304 [I 2022-10-26 21:38:49] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:16:52 [I 2022-10-26 21:38:49] ...trap/engines/engine.py.229: process: 0.56336 | load: 0.00239 [I 2022-10-26 21:38:49] ...trap/engines/engine.py.230: loss: 0.16472 [I 2022-10-26 21:38:49] ...optimizers/trijoint.py.151: total_norm: 13.697180 [I 2022-10-26 21:38:55] ...trap/engines/engine.py.224: train: epoch 31 | batch 540/2304 [I 2022-10-26 21:38:55] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:16:27 [I 2022-10-26 21:38:55] ...trap/engines/engine.py.229: process: 0.55925 | load: 0.00237 [I 2022-10-26 21:38:55] ...trap/engines/engine.py.230: loss: 0.24371 [I 2022-10-26 21:38:55] ...optimizers/trijoint.py.151: total_norm: 17.502632 [I 2022-10-26 21:39:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 550/2304 [I 2022-10-26 21:39:01] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:16:18 [I 2022-10-26 21:39:01] ...trap/engines/engine.py.229: process: 0.55088 | load: 0.00238 [I 2022-10-26 21:39:01] ...trap/engines/engine.py.230: loss: 0.18317 [I 2022-10-26 21:39:01] ...optimizers/trijoint.py.151: total_norm: 13.415915 [I 2022-10-26 21:39:06] ...trap/engines/engine.py.224: train: epoch 31 | batch 560/2304 [I 2022-10-26 21:39:06] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:16:31 [I 2022-10-26 21:39:06] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00238 [I 2022-10-26 21:39:06] ...trap/engines/engine.py.230: loss: 0.23624 [I 2022-10-26 21:39:06] ...optimizers/trijoint.py.151: total_norm: 16.377081 [I 2022-10-26 21:39:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 570/2304 [I 2022-10-26 21:39:12] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:16:10 [I 2022-10-26 21:39:12] ...trap/engines/engine.py.229: process: 0.57036 | load: 0.00260 [I 2022-10-26 21:39:12] ...trap/engines/engine.py.230: loss: 0.12088 [I 2022-10-26 21:39:12] ...optimizers/trijoint.py.151: total_norm: 11.598496 [I 2022-10-26 21:39:18] ...trap/engines/engine.py.224: train: epoch 31 | batch 580/2304 [I 2022-10-26 21:39:18] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:16:09 [I 2022-10-26 21:39:18] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00249 [I 2022-10-26 21:39:18] ...trap/engines/engine.py.230: loss: 0.20222 [I 2022-10-26 21:39:18] ...optimizers/trijoint.py.151: total_norm: 23.747498 [I 2022-10-26 21:39:23] ...trap/engines/engine.py.224: train: epoch 31 | batch 590/2304 [I 2022-10-26 21:39:23] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:15:58 [I 2022-10-26 21:39:23] ...trap/engines/engine.py.229: process: 0.55472 | load: 0.00260 [I 2022-10-26 21:39:23] ...trap/engines/engine.py.230: loss: 0.21029 [I 2022-10-26 21:39:23] ...optimizers/trijoint.py.151: total_norm: 18.600302 [I 2022-10-26 21:39:29] ...trap/engines/engine.py.224: train: epoch 31 | batch 600/2304 [I 2022-10-26 21:39:29] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:51 [I 2022-10-26 21:39:29] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00240 [I 2022-10-26 21:39:29] ...trap/engines/engine.py.230: loss: 0.15468 [I 2022-10-26 21:39:29] ...optimizers/trijoint.py.151: total_norm: 20.579014 [I 2022-10-26 21:39:34] ...trap/engines/engine.py.224: train: epoch 31 | batch 610/2304 [I 2022-10-26 21:39:34] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:47 [I 2022-10-26 21:39:34] ...trap/engines/engine.py.229: process: 0.55821 | load: 0.00242 [I 2022-10-26 21:39:34] ...trap/engines/engine.py.230: loss: 0.30617 [I 2022-10-26 21:39:34] ...optimizers/trijoint.py.151: total_norm: 17.793377 [I 2022-10-26 21:39:40] ...trap/engines/engine.py.224: train: epoch 31 | batch 620/2304 [I 2022-10-26 21:39:40] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:36 [I 2022-10-26 21:39:40] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00241 [I 2022-10-26 21:39:40] ...trap/engines/engine.py.230: loss: 0.20879 [I 2022-10-26 21:39:40] ...optimizers/trijoint.py.151: total_norm: 14.911475 [I 2022-10-26 21:39:45] ...trap/engines/engine.py.224: train: epoch 31 | batch 630/2304 [I 2022-10-26 21:39:45] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:32 [I 2022-10-26 21:39:45] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00242 [I 2022-10-26 21:39:45] ...trap/engines/engine.py.230: loss: 0.12091 [I 2022-10-26 21:39:45] ...optimizers/trijoint.py.151: total_norm: 10.694206 [I 2022-10-26 21:39:51] ...trap/engines/engine.py.224: train: epoch 31 | batch 640/2304 [I 2022-10-26 21:39:51] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:15:26 [I 2022-10-26 21:39:51] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00242 [I 2022-10-26 21:39:51] ...trap/engines/engine.py.230: loss: 0.14365 [I 2022-10-26 21:39:51] ...optimizers/trijoint.py.151: total_norm: 25.865614 [I 2022-10-26 21:39:57] ...trap/engines/engine.py.224: train: epoch 31 | batch 650/2304 [I 2022-10-26 21:39:57] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:17 [I 2022-10-26 21:39:57] ...trap/engines/engine.py.229: process: 0.54852 | load: 0.00248 [I 2022-10-26 21:39:57] ...trap/engines/engine.py.230: loss: 0.14081 [I 2022-10-26 21:39:57] ...optimizers/trijoint.py.151: total_norm: 19.794294 [I 2022-10-26 21:40:02] ...trap/engines/engine.py.224: train: epoch 31 | batch 660/2304 [I 2022-10-26 21:40:02] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:15:21 [I 2022-10-26 21:40:02] ...trap/engines/engine.py.229: process: 0.55980 | load: 0.00257 [I 2022-10-26 21:40:02] ...trap/engines/engine.py.230: loss: 0.20804 [I 2022-10-26 21:40:02] ...optimizers/trijoint.py.151: total_norm: 22.068285 [I 2022-10-26 21:40:08] ...trap/engines/engine.py.224: train: epoch 31 | batch 670/2304 [I 2022-10-26 21:40:08] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:33 [I 2022-10-26 21:40:08] ...trap/engines/engine.py.229: process: 0.55408 | load: 0.00243 [I 2022-10-26 21:40:08] ...trap/engines/engine.py.230: loss: 0.14501 [I 2022-10-26 21:40:08] ...optimizers/trijoint.py.151: total_norm: 13.203524 [I 2022-10-26 21:40:14] ...trap/engines/engine.py.224: train: epoch 31 | batch 680/2304 [I 2022-10-26 21:40:14] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:11 [I 2022-10-26 21:40:14] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00252 [I 2022-10-26 21:40:14] ...trap/engines/engine.py.230: loss: 0.17271 [I 2022-10-26 21:40:14] ...optimizers/trijoint.py.151: total_norm: 25.210949 [I 2022-10-26 21:40:19] ...trap/engines/engine.py.224: train: epoch 31 | batch 690/2304 [I 2022-10-26 21:40:19] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:14:57 [I 2022-10-26 21:40:19] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00245 [I 2022-10-26 21:40:19] ...trap/engines/engine.py.230: loss: 0.22226 [I 2022-10-26 21:40:19] ...optimizers/trijoint.py.151: total_norm: 26.098227 [I 2022-10-26 21:40:25] ...trap/engines/engine.py.224: train: epoch 31 | batch 700/2304 [I 2022-10-26 21:40:25] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:14:40 [I 2022-10-26 21:40:25] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00245 [I 2022-10-26 21:40:25] ...trap/engines/engine.py.230: loss: 0.23126 [I 2022-10-26 21:40:25] ...optimizers/trijoint.py.151: total_norm: 25.336941 [I 2022-10-26 21:40:30] ...trap/engines/engine.py.224: train: epoch 31 | batch 710/2304 [I 2022-10-26 21:40:30] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:44 [I 2022-10-26 21:40:30] ...trap/engines/engine.py.229: process: 0.54689 | load: 0.00242 [I 2022-10-26 21:40:30] ...trap/engines/engine.py.230: loss: 0.19743 [I 2022-10-26 21:40:30] ...optimizers/trijoint.py.151: total_norm: 12.783457 [I 2022-10-26 21:40:36] ...trap/engines/engine.py.224: train: epoch 31 | batch 720/2304 [I 2022-10-26 21:40:36] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:33 [I 2022-10-26 21:40:36] ...trap/engines/engine.py.229: process: 0.55641 | load: 0.00247 [I 2022-10-26 21:40:36] ...trap/engines/engine.py.230: loss: 0.17412 [I 2022-10-26 21:40:36] ...optimizers/trijoint.py.151: total_norm: 12.512415 [I 2022-10-26 21:40:41] ...trap/engines/engine.py.224: train: epoch 31 | batch 730/2304 [I 2022-10-26 21:40:41] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:42 [I 2022-10-26 21:40:41] ...trap/engines/engine.py.229: process: 0.56737 | load: 0.00248 [I 2022-10-26 21:40:41] ...trap/engines/engine.py.230: loss: 0.26154 [I 2022-10-26 21:40:41] ...optimizers/trijoint.py.151: total_norm: 33.801579 [I 2022-10-26 21:40:47] ...trap/engines/engine.py.224: train: epoch 31 | batch 740/2304 [I 2022-10-26 21:40:47] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:14:32 [I 2022-10-26 21:40:47] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00242 [I 2022-10-26 21:40:47] ...trap/engines/engine.py.230: loss: 0.20557 [I 2022-10-26 21:40:47] ...optimizers/trijoint.py.151: total_norm: 24.539240 [I 2022-10-26 21:40:53] ...trap/engines/engine.py.224: train: epoch 31 | batch 750/2304 [I 2022-10-26 21:40:53] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:28 [I 2022-10-26 21:40:53] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00269 [I 2022-10-26 21:40:53] ...trap/engines/engine.py.230: loss: 0.18205 [I 2022-10-26 21:40:53] ...optimizers/trijoint.py.151: total_norm: 11.292908 [I 2022-10-26 21:40:58] ...trap/engines/engine.py.224: train: epoch 31 | batch 760/2304 [I 2022-10-26 21:40:58] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:22 [I 2022-10-26 21:40:58] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00254 [I 2022-10-26 21:40:58] ...trap/engines/engine.py.230: loss: 0.13904 [I 2022-10-26 21:40:58] ...optimizers/trijoint.py.151: total_norm: 9.823493 [I 2022-10-26 21:41:04] ...trap/engines/engine.py.224: train: epoch 31 | batch 770/2304 [I 2022-10-26 21:41:04] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:14:18 [I 2022-10-26 21:41:04] ...trap/engines/engine.py.229: process: 0.56226 | load: 0.00246 [I 2022-10-26 21:41:04] ...trap/engines/engine.py.230: loss: 0.29355 [I 2022-10-26 21:41:04] ...optimizers/trijoint.py.151: total_norm: 27.580957 [I 2022-10-26 21:41:09] ...trap/engines/engine.py.224: train: epoch 31 | batch 780/2304 [I 2022-10-26 21:41:09] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:07 [I 2022-10-26 21:41:09] ...trap/engines/engine.py.229: process: 0.56490 | load: 0.00246 [I 2022-10-26 21:41:09] ...trap/engines/engine.py.230: loss: 0.26871 [I 2022-10-26 21:41:09] ...optimizers/trijoint.py.151: total_norm: 25.079330 [I 2022-10-26 21:41:15] ...trap/engines/engine.py.224: train: epoch 31 | batch 790/2304 [I 2022-10-26 21:41:15] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:14:06 [I 2022-10-26 21:41:15] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00244 [I 2022-10-26 21:41:15] ...trap/engines/engine.py.230: loss: 0.19009 [I 2022-10-26 21:41:15] ...optimizers/trijoint.py.151: total_norm: 60.566628 [I 2022-10-26 21:41:21] ...trap/engines/engine.py.224: train: epoch 31 | batch 800/2304 [I 2022-10-26 21:41:21] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:14:01 [I 2022-10-26 21:41:21] ...trap/engines/engine.py.229: process: 0.55159 | load: 0.00255 [I 2022-10-26 21:41:21] ...trap/engines/engine.py.230: loss: 0.12577 [I 2022-10-26 21:41:21] ...optimizers/trijoint.py.151: total_norm: 11.218123 [I 2022-10-26 21:41:26] ...trap/engines/engine.py.224: train: epoch 31 | batch 810/2304 [I 2022-10-26 21:41:26] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:49 [I 2022-10-26 21:41:26] ...trap/engines/engine.py.229: process: 0.55140 | load: 0.00246 [I 2022-10-26 21:41:26] ...trap/engines/engine.py.230: loss: 0.10555 [I 2022-10-26 21:41:26] ...optimizers/trijoint.py.151: total_norm: 9.448968 [I 2022-10-26 21:41:32] ...trap/engines/engine.py.224: train: epoch 31 | batch 820/2304 [I 2022-10-26 21:41:32] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:13:50 [I 2022-10-26 21:41:32] ...trap/engines/engine.py.229: process: 0.57964 | load: 0.00245 [I 2022-10-26 21:41:32] ...trap/engines/engine.py.230: loss: 0.14863 [I 2022-10-26 21:41:32] ...optimizers/trijoint.py.151: total_norm: 10.926923 [I 2022-10-26 21:41:37] ...trap/engines/engine.py.224: train: epoch 31 | batch 830/2304 [I 2022-10-26 21:41:37] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:46 [I 2022-10-26 21:41:37] ...trap/engines/engine.py.229: process: 0.56716 | load: 0.00257 [I 2022-10-26 21:41:37] ...trap/engines/engine.py.230: loss: 0.25023 [I 2022-10-26 21:41:37] ...optimizers/trijoint.py.151: total_norm: 57.478294 [I 2022-10-26 21:41:43] ...trap/engines/engine.py.224: train: epoch 31 | batch 840/2304 [I 2022-10-26 21:41:43] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:13:34 [I 2022-10-26 21:41:43] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00251 [I 2022-10-26 21:41:43] ...trap/engines/engine.py.230: loss: 0.18987 [I 2022-10-26 21:41:43] ...optimizers/trijoint.py.151: total_norm: 20.734713 [I 2022-10-26 21:41:49] ...trap/engines/engine.py.224: train: epoch 31 | batch 850/2304 [I 2022-10-26 21:41:49] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:29 [I 2022-10-26 21:41:49] ...trap/engines/engine.py.229: process: 0.54722 | load: 0.00251 [I 2022-10-26 21:41:49] ...trap/engines/engine.py.230: loss: 0.11060 [I 2022-10-26 21:41:49] ...optimizers/trijoint.py.151: total_norm: 27.294329 [I 2022-10-26 21:41:54] ...trap/engines/engine.py.224: train: epoch 31 | batch 860/2304 [I 2022-10-26 21:41:54] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:21 [I 2022-10-26 21:41:54] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00246 [I 2022-10-26 21:41:54] ...trap/engines/engine.py.230: loss: 0.12431 [I 2022-10-26 21:41:54] ...optimizers/trijoint.py.151: total_norm: 19.787542 [I 2022-10-26 21:42:00] ...trap/engines/engine.py.224: train: epoch 31 | batch 870/2304 [I 2022-10-26 21:42:00] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:13:15 [I 2022-10-26 21:42:00] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00247 [I 2022-10-26 21:42:00] ...trap/engines/engine.py.230: loss: 0.23495 [I 2022-10-26 21:42:00] ...optimizers/trijoint.py.151: total_norm: 19.261166 [I 2022-10-26 21:42:05] ...trap/engines/engine.py.224: train: epoch 31 | batch 880/2304 [I 2022-10-26 21:42:05] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:10 [I 2022-10-26 21:42:05] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00247 [I 2022-10-26 21:42:05] ...trap/engines/engine.py.230: loss: 0.16262 [I 2022-10-26 21:42:05] ...optimizers/trijoint.py.151: total_norm: 31.699074 [I 2022-10-26 21:42:11] ...trap/engines/engine.py.224: train: epoch 31 | batch 890/2304 [I 2022-10-26 21:42:11] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:13:04 [I 2022-10-26 21:42:11] ...trap/engines/engine.py.229: process: 0.56137 | load: 0.00244 [I 2022-10-26 21:42:11] ...trap/engines/engine.py.230: loss: 0.21862 [I 2022-10-26 21:42:11] ...optimizers/trijoint.py.151: total_norm: 21.109499 [I 2022-10-26 21:42:16] ...trap/engines/engine.py.224: train: epoch 31 | batch 900/2304 [I 2022-10-26 21:42:16] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:13:03 [I 2022-10-26 21:42:16] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00250 [I 2022-10-26 21:42:16] ...trap/engines/engine.py.230: loss: 0.14633 [I 2022-10-26 21:42:16] ...optimizers/trijoint.py.151: total_norm: 19.186707 [I 2022-10-26 21:42:22] ...trap/engines/engine.py.224: train: epoch 31 | batch 910/2304 [I 2022-10-26 21:42:22] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:13:00 [I 2022-10-26 21:42:22] ...trap/engines/engine.py.229: process: 0.56594 | load: 0.00244 [I 2022-10-26 21:42:22] ...trap/engines/engine.py.230: loss: 0.13558 [I 2022-10-26 21:42:22] ...optimizers/trijoint.py.151: total_norm: 18.142048 [I 2022-10-26 21:42:27] ...trap/engines/engine.py.224: train: epoch 31 | batch 920/2304 [I 2022-10-26 21:42:27] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:48 [I 2022-10-26 21:42:27] ...trap/engines/engine.py.229: process: 0.56877 | load: 0.00246 [I 2022-10-26 21:42:27] ...trap/engines/engine.py.230: loss: 0.11071 [I 2022-10-26 21:42:27] ...optimizers/trijoint.py.151: total_norm: 14.604738 [I 2022-10-26 21:42:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 930/2304 [I 2022-10-26 21:42:33] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:47 [I 2022-10-26 21:42:33] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00241 [I 2022-10-26 21:42:33] ...trap/engines/engine.py.230: loss: 0.25012 [I 2022-10-26 21:42:33] ...optimizers/trijoint.py.151: total_norm: 23.514250 [I 2022-10-26 21:42:39] ...trap/engines/engine.py.224: train: epoch 31 | batch 940/2304 [I 2022-10-26 21:42:39] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:12:42 [I 2022-10-26 21:42:39] ...trap/engines/engine.py.229: process: 0.56242 | load: 0.00243 [I 2022-10-26 21:42:39] ...trap/engines/engine.py.230: loss: 0.19718 [I 2022-10-26 21:42:39] ...optimizers/trijoint.py.151: total_norm: 17.609079 [I 2022-10-26 21:42:44] ...trap/engines/engine.py.224: train: epoch 31 | batch 950/2304 [I 2022-10-26 21:42:44] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:31 [I 2022-10-26 21:42:44] ...trap/engines/engine.py.229: process: 0.54891 | load: 0.00246 [I 2022-10-26 21:42:44] ...trap/engines/engine.py.230: loss: 0.15899 [I 2022-10-26 21:42:44] ...optimizers/trijoint.py.151: total_norm: 9.150640 [I 2022-10-26 21:42:50] ...trap/engines/engine.py.224: train: epoch 31 | batch 960/2304 [I 2022-10-26 21:42:50] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:12:28 [I 2022-10-26 21:42:50] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00246 [I 2022-10-26 21:42:50] ...trap/engines/engine.py.230: loss: 0.19514 [I 2022-10-26 21:42:50] ...optimizers/trijoint.py.151: total_norm: 25.520483 [I 2022-10-26 21:42:55] ...trap/engines/engine.py.224: train: epoch 31 | batch 970/2304 [I 2022-10-26 21:42:55] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:12:20 [I 2022-10-26 21:42:55] ...trap/engines/engine.py.229: process: 0.55001 | load: 0.00245 [I 2022-10-26 21:42:55] ...trap/engines/engine.py.230: loss: 0.16118 [I 2022-10-26 21:42:55] ...optimizers/trijoint.py.151: total_norm: 19.958363 [I 2022-10-26 21:43:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 980/2304 [I 2022-10-26 21:43:01] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:18 [I 2022-10-26 21:43:01] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00246 [I 2022-10-26 21:43:01] ...trap/engines/engine.py.230: loss: 0.17995 [I 2022-10-26 21:43:01] ...optimizers/trijoint.py.151: total_norm: 20.372065 [I 2022-10-26 21:43:07] ...trap/engines/engine.py.224: train: epoch 31 | batch 990/2304 [I 2022-10-26 21:43:07] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:12:07 [I 2022-10-26 21:43:07] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00246 [I 2022-10-26 21:43:07] ...trap/engines/engine.py.230: loss: 0.22782 [I 2022-10-26 21:43:07] ...optimizers/trijoint.py.151: total_norm: 23.569138 [I 2022-10-26 21:43:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 1000/2304 [I 2022-10-26 21:43:12] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:08 [I 2022-10-26 21:43:12] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00248 [I 2022-10-26 21:43:12] ...trap/engines/engine.py.230: loss: 0.17790 [I 2022-10-26 21:43:12] ...optimizers/trijoint.py.151: total_norm: 16.044292 [I 2022-10-26 21:43:18] ...trap/engines/engine.py.224: train: epoch 31 | batch 1010/2304 [I 2022-10-26 21:43:18] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:12:02 [I 2022-10-26 21:43:18] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00247 [I 2022-10-26 21:43:18] ...trap/engines/engine.py.230: loss: 0.16444 [I 2022-10-26 21:43:18] ...optimizers/trijoint.py.151: total_norm: 13.722089 [I 2022-10-26 21:43:23] ...trap/engines/engine.py.224: train: epoch 31 | batch 1020/2304 [I 2022-10-26 21:43:23] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:58 [I 2022-10-26 21:43:23] ...trap/engines/engine.py.229: process: 0.55073 | load: 0.00243 [I 2022-10-26 21:43:23] ...trap/engines/engine.py.230: loss: 0.19622 [I 2022-10-26 21:43:23] ...optimizers/trijoint.py.151: total_norm: 19.147928 [I 2022-10-26 21:43:29] ...trap/engines/engine.py.224: train: epoch 31 | batch 1030/2304 [I 2022-10-26 21:43:29] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:52 [I 2022-10-26 21:43:29] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00247 [I 2022-10-26 21:43:29] ...trap/engines/engine.py.230: loss: 0.16722 [I 2022-10-26 21:43:29] ...optimizers/trijoint.py.151: total_norm: 24.537437 [I 2022-10-26 21:43:34] ...trap/engines/engine.py.224: train: epoch 31 | batch 1040/2304 [I 2022-10-26 21:43:34] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:45 [I 2022-10-26 21:43:34] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00243 [I 2022-10-26 21:43:34] ...trap/engines/engine.py.230: loss: 0.18089 [I 2022-10-26 21:43:34] ...optimizers/trijoint.py.151: total_norm: 22.175755 [I 2022-10-26 21:43:40] ...trap/engines/engine.py.224: train: epoch 31 | batch 1050/2304 [I 2022-10-26 21:43:40] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:38 [I 2022-10-26 21:43:40] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00248 [I 2022-10-26 21:43:40] ...trap/engines/engine.py.230: loss: 0.19057 [I 2022-10-26 21:43:40] ...optimizers/trijoint.py.151: total_norm: 18.696711 [I 2022-10-26 21:43:46] ...trap/engines/engine.py.224: train: epoch 31 | batch 1060/2304 [I 2022-10-26 21:43:46] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:31 [I 2022-10-26 21:43:46] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00247 [I 2022-10-26 21:43:46] ...trap/engines/engine.py.230: loss: 0.19875 [I 2022-10-26 21:43:46] ...optimizers/trijoint.py.151: total_norm: 10.829439 [I 2022-10-26 21:43:51] ...trap/engines/engine.py.224: train: epoch 31 | batch 1070/2304 [I 2022-10-26 21:43:51] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:27 [I 2022-10-26 21:43:51] ...trap/engines/engine.py.229: process: 0.56535 | load: 0.00248 [I 2022-10-26 21:43:51] ...trap/engines/engine.py.230: loss: 0.21081 [I 2022-10-26 21:43:51] ...optimizers/trijoint.py.151: total_norm: 15.673727 [I 2022-10-26 21:43:57] ...trap/engines/engine.py.224: train: epoch 31 | batch 1080/2304 [I 2022-10-26 21:43:57] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:18 [I 2022-10-26 21:43:57] ...trap/engines/engine.py.229: process: 0.54609 | load: 0.00242 [I 2022-10-26 21:43:57] ...trap/engines/engine.py.230: loss: 0.15778 [I 2022-10-26 21:43:57] ...optimizers/trijoint.py.151: total_norm: 8.129889 [I 2022-10-26 21:44:02] ...trap/engines/engine.py.224: train: epoch 31 | batch 1090/2304 [I 2022-10-26 21:44:02] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:13 [I 2022-10-26 21:44:02] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00264 [I 2022-10-26 21:44:02] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-26 21:44:02] ...optimizers/trijoint.py.151: total_norm: 18.949369 [I 2022-10-26 21:44:08] ...trap/engines/engine.py.224: train: epoch 31 | batch 1100/2304 [I 2022-10-26 21:44:08] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:09 [I 2022-10-26 21:44:08] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00243 [I 2022-10-26 21:44:08] ...trap/engines/engine.py.230: loss: 0.18198 [I 2022-10-26 21:44:08] ...optimizers/trijoint.py.151: total_norm: 15.587755 [I 2022-10-26 21:44:13] ...trap/engines/engine.py.224: train: epoch 31 | batch 1110/2304 [I 2022-10-26 21:44:13] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:11:08 [I 2022-10-26 21:44:13] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00245 [I 2022-10-26 21:44:13] ...trap/engines/engine.py.230: loss: 0.12750 [I 2022-10-26 21:44:13] ...optimizers/trijoint.py.151: total_norm: 20.307491 [I 2022-10-26 21:44:19] ...trap/engines/engine.py.224: train: epoch 31 | batch 1120/2304 [I 2022-10-26 21:44:19] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:10:57 [I 2022-10-26 21:44:19] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00251 [I 2022-10-26 21:44:19] ...trap/engines/engine.py.230: loss: 0.17675 [I 2022-10-26 21:44:19] ...optimizers/trijoint.py.151: total_norm: 11.899961 [I 2022-10-26 21:44:25] ...trap/engines/engine.py.224: train: epoch 31 | batch 1130/2304 [I 2022-10-26 21:44:25] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:53 [I 2022-10-26 21:44:25] ...trap/engines/engine.py.229: process: 0.55225 | load: 0.00248 [I 2022-10-26 21:44:25] ...trap/engines/engine.py.230: loss: 0.18789 [I 2022-10-26 21:44:25] ...optimizers/trijoint.py.151: total_norm: 18.187143 [I 2022-10-26 21:44:30] ...trap/engines/engine.py.224: train: epoch 31 | batch 1140/2304 [I 2022-10-26 21:44:30] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:45 [I 2022-10-26 21:44:30] ...trap/engines/engine.py.229: process: 0.56037 | load: 0.00245 [I 2022-10-26 21:44:30] ...trap/engines/engine.py.230: loss: 0.15343 [I 2022-10-26 21:44:30] ...optimizers/trijoint.py.151: total_norm: 21.215799 [I 2022-10-26 21:44:36] ...trap/engines/engine.py.224: train: epoch 31 | batch 1150/2304 [I 2022-10-26 21:44:36] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:41 [I 2022-10-26 21:44:36] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00242 [I 2022-10-26 21:44:36] ...trap/engines/engine.py.230: loss: 0.20711 [I 2022-10-26 21:44:36] ...optimizers/trijoint.py.151: total_norm: 23.505989 [I 2022-10-26 21:44:41] ...trap/engines/engine.py.224: train: epoch 31 | batch 1160/2304 [I 2022-10-26 21:44:41] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:37 [I 2022-10-26 21:44:41] ...trap/engines/engine.py.229: process: 0.55605 | load: 0.00242 [I 2022-10-26 21:44:41] ...trap/engines/engine.py.230: loss: 0.21394 [I 2022-10-26 21:44:41] ...optimizers/trijoint.py.151: total_norm: 22.947779 [I 2022-10-26 21:44:47] ...trap/engines/engine.py.224: train: epoch 31 | batch 1170/2304 [I 2022-10-26 21:44:47] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:29 [I 2022-10-26 21:44:47] ...trap/engines/engine.py.229: process: 0.55181 | load: 0.00240 [I 2022-10-26 21:44:47] ...trap/engines/engine.py.230: loss: 0.13141 [I 2022-10-26 21:44:47] ...optimizers/trijoint.py.151: total_norm: 14.689972 [I 2022-10-26 21:44:52] ...trap/engines/engine.py.224: train: epoch 31 | batch 1180/2304 [I 2022-10-26 21:44:52] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:22 [I 2022-10-26 21:44:52] ...trap/engines/engine.py.229: process: 0.55264 | load: 0.00241 [I 2022-10-26 21:44:52] ...trap/engines/engine.py.230: loss: 0.20174 [I 2022-10-26 21:44:52] ...optimizers/trijoint.py.151: total_norm: 24.044456 [I 2022-10-26 21:44:58] ...trap/engines/engine.py.224: train: epoch 31 | batch 1190/2304 [I 2022-10-26 21:44:58] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:17 [I 2022-10-26 21:44:58] ...trap/engines/engine.py.229: process: 0.56056 | load: 0.00240 [I 2022-10-26 21:44:58] ...trap/engines/engine.py.230: loss: 0.17880 [I 2022-10-26 21:44:58] ...optimizers/trijoint.py.151: total_norm: 12.855133 [I 2022-10-26 21:45:04] ...trap/engines/engine.py.224: train: epoch 31 | batch 1200/2304 [I 2022-10-26 21:45:04] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:18 [I 2022-10-26 21:45:04] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00271 [I 2022-10-26 21:45:04] ...trap/engines/engine.py.230: loss: 0.16135 [I 2022-10-26 21:45:04] ...optimizers/trijoint.py.151: total_norm: 16.620979 [I 2022-10-26 21:45:09] ...trap/engines/engine.py.224: train: epoch 31 | batch 1210/2304 [I 2022-10-26 21:45:09] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:10:09 [I 2022-10-26 21:45:09] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00246 [I 2022-10-26 21:45:09] ...trap/engines/engine.py.230: loss: 0.26095 [I 2022-10-26 21:45:09] ...optimizers/trijoint.py.151: total_norm: 16.743137 [I 2022-10-26 21:45:15] ...trap/engines/engine.py.224: train: epoch 31 | batch 1220/2304 [I 2022-10-26 21:45:15] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:10:04 [I 2022-10-26 21:45:15] ...trap/engines/engine.py.229: process: 0.56549 | load: 0.00246 [I 2022-10-26 21:45:15] ...trap/engines/engine.py.230: loss: 0.16053 [I 2022-10-26 21:45:15] ...optimizers/trijoint.py.151: total_norm: 13.379622 [I 2022-10-26 21:45:20] ...trap/engines/engine.py.224: train: epoch 31 | batch 1230/2304 [I 2022-10-26 21:45:20] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:55 [I 2022-10-26 21:45:20] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00246 [I 2022-10-26 21:45:20] ...trap/engines/engine.py.230: loss: 0.17883 [I 2022-10-26 21:45:20] ...optimizers/trijoint.py.151: total_norm: 26.319456 [I 2022-10-26 21:45:26] ...trap/engines/engine.py.224: train: epoch 31 | batch 1240/2304 [I 2022-10-26 21:45:26] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:10:02 [I 2022-10-26 21:45:26] ...trap/engines/engine.py.229: process: 0.56112 | load: 0.00245 [I 2022-10-26 21:45:26] ...trap/engines/engine.py.230: loss: 0.19601 [I 2022-10-26 21:45:26] ...optimizers/trijoint.py.151: total_norm: 16.035112 [I 2022-10-26 21:45:32] ...trap/engines/engine.py.224: train: epoch 31 | batch 1250/2304 [I 2022-10-26 21:45:32] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:51 [I 2022-10-26 21:45:32] ...trap/engines/engine.py.229: process: 0.55375 | load: 0.00252 [I 2022-10-26 21:45:32] ...trap/engines/engine.py.230: loss: 0.19386 [I 2022-10-26 21:45:32] ...optimizers/trijoint.py.151: total_norm: 19.125776 [I 2022-10-26 21:45:37] ...trap/engines/engine.py.224: train: epoch 31 | batch 1260/2304 [I 2022-10-26 21:45:37] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:59 [I 2022-10-26 21:45:37] ...trap/engines/engine.py.229: process: 0.57118 | load: 0.00270 [I 2022-10-26 21:45:37] ...trap/engines/engine.py.230: loss: 0.10292 [I 2022-10-26 21:45:37] ...optimizers/trijoint.py.151: total_norm: 12.340667 [I 2022-10-26 21:45:43] ...trap/engines/engine.py.224: train: epoch 31 | batch 1270/2304 [I 2022-10-26 21:45:43] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:39 [I 2022-10-26 21:45:43] ...trap/engines/engine.py.229: process: 0.56193 | load: 0.00244 [I 2022-10-26 21:45:43] ...trap/engines/engine.py.230: loss: 0.18738 [I 2022-10-26 21:45:43] ...optimizers/trijoint.py.151: total_norm: 36.842453 [I 2022-10-26 21:45:48] ...trap/engines/engine.py.224: train: epoch 31 | batch 1280/2304 [I 2022-10-26 21:45:48] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:30 [I 2022-10-26 21:45:48] ...trap/engines/engine.py.229: process: 0.56094 | load: 0.00242 [I 2022-10-26 21:45:48] ...trap/engines/engine.py.230: loss: 0.15543 [I 2022-10-26 21:45:48] ...optimizers/trijoint.py.151: total_norm: 15.047821 [I 2022-10-26 21:45:54] ...trap/engines/engine.py.224: train: epoch 31 | batch 1290/2304 [I 2022-10-26 21:45:54] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:26 [I 2022-10-26 21:45:54] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00243 [I 2022-10-26 21:45:54] ...trap/engines/engine.py.230: loss: 0.17461 [I 2022-10-26 21:45:54] ...optimizers/trijoint.py.151: total_norm: 17.062307 [I 2022-10-26 21:46:00] ...trap/engines/engine.py.224: train: epoch 31 | batch 1300/2304 [I 2022-10-26 21:46:00] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:18 [I 2022-10-26 21:46:00] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00251 [I 2022-10-26 21:46:00] ...trap/engines/engine.py.230: loss: 0.16545 [I 2022-10-26 21:46:00] ...optimizers/trijoint.py.151: total_norm: 13.003192 [I 2022-10-26 21:46:05] ...trap/engines/engine.py.224: train: epoch 31 | batch 1310/2304 [I 2022-10-26 21:46:05] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:10 [I 2022-10-26 21:46:05] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00245 [I 2022-10-26 21:46:05] ...trap/engines/engine.py.230: loss: 0.12952 [I 2022-10-26 21:46:05] ...optimizers/trijoint.py.151: total_norm: 11.249217 [I 2022-10-26 21:46:11] ...trap/engines/engine.py.224: train: epoch 31 | batch 1320/2304 [I 2022-10-26 21:46:11] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:07 [I 2022-10-26 21:46:11] ...trap/engines/engine.py.229: process: 0.56297 | load: 0.00243 [I 2022-10-26 21:46:11] ...trap/engines/engine.py.230: loss: 0.18215 [I 2022-10-26 21:46:11] ...optimizers/trijoint.py.151: total_norm: 26.474667 [I 2022-10-26 21:46:16] ...trap/engines/engine.py.224: train: epoch 31 | batch 1330/2304 [I 2022-10-26 21:46:16] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:09:03 [I 2022-10-26 21:46:16] ...trap/engines/engine.py.229: process: 0.55555 | load: 0.00251 [I 2022-10-26 21:46:16] ...trap/engines/engine.py.230: loss: 0.13199 [I 2022-10-26 21:46:16] ...optimizers/trijoint.py.151: total_norm: 18.370520 [I 2022-10-26 21:46:22] ...trap/engines/engine.py.224: train: epoch 31 | batch 1340/2304 [I 2022-10-26 21:46:22] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:09:02 [I 2022-10-26 21:46:22] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00259 [I 2022-10-26 21:46:22] ...trap/engines/engine.py.230: loss: 0.26370 [I 2022-10-26 21:46:22] ...optimizers/trijoint.py.151: total_norm: 17.239561 [I 2022-10-26 21:46:28] ...trap/engines/engine.py.224: train: epoch 31 | batch 1350/2304 [I 2022-10-26 21:46:28] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:52 [I 2022-10-26 21:46:28] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00246 [I 2022-10-26 21:46:28] ...trap/engines/engine.py.230: loss: 0.17875 [I 2022-10-26 21:46:28] ...optimizers/trijoint.py.151: total_norm: 22.030933 [I 2022-10-26 21:46:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 1360/2304 [I 2022-10-26 21:46:33] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:43 [I 2022-10-26 21:46:33] ...trap/engines/engine.py.229: process: 0.55937 | load: 0.00249 [I 2022-10-26 21:46:33] ...trap/engines/engine.py.230: loss: 0.13054 [I 2022-10-26 21:46:33] ...optimizers/trijoint.py.151: total_norm: 14.109177 [I 2022-10-26 21:46:39] ...trap/engines/engine.py.224: train: epoch 31 | batch 1370/2304 [I 2022-10-26 21:46:39] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:38 [I 2022-10-26 21:46:39] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00245 [I 2022-10-26 21:46:39] ...trap/engines/engine.py.230: loss: 0.20584 [I 2022-10-26 21:46:39] ...optimizers/trijoint.py.151: total_norm: 27.763706 [I 2022-10-26 21:46:44] ...trap/engines/engine.py.224: train: epoch 31 | batch 1380/2304 [I 2022-10-26 21:46:44] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:37 [I 2022-10-26 21:46:44] ...trap/engines/engine.py.229: process: 0.56833 | load: 0.00244 [I 2022-10-26 21:46:44] ...trap/engines/engine.py.230: loss: 0.18297 [I 2022-10-26 21:46:44] ...optimizers/trijoint.py.151: total_norm: 16.798491 [I 2022-10-26 21:46:50] ...trap/engines/engine.py.224: train: epoch 31 | batch 1390/2304 [I 2022-10-26 21:46:50] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:30 [I 2022-10-26 21:46:50] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00243 [I 2022-10-26 21:46:50] ...trap/engines/engine.py.230: loss: 0.22384 [I 2022-10-26 21:46:50] ...optimizers/trijoint.py.151: total_norm: 19.164997 [I 2022-10-26 21:46:56] ...trap/engines/engine.py.224: train: epoch 31 | batch 1400/2304 [I 2022-10-26 21:46:56] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:30 [I 2022-10-26 21:46:56] ...trap/engines/engine.py.229: process: 0.56785 | load: 0.00245 [I 2022-10-26 21:46:56] ...trap/engines/engine.py.230: loss: 0.20950 [I 2022-10-26 21:46:56] ...optimizers/trijoint.py.151: total_norm: 24.824018 [I 2022-10-26 21:47:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 1410/2304 [I 2022-10-26 21:47:01] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:21 [I 2022-10-26 21:47:01] ...trap/engines/engine.py.229: process: 0.56384 | load: 0.00242 [I 2022-10-26 21:47:01] ...trap/engines/engine.py.230: loss: 0.20486 [I 2022-10-26 21:47:01] ...optimizers/trijoint.py.151: total_norm: 18.321146 [I 2022-10-26 21:47:07] ...trap/engines/engine.py.224: train: epoch 31 | batch 1420/2304 [I 2022-10-26 21:47:07] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:12 [I 2022-10-26 21:47:07] ...trap/engines/engine.py.229: process: 0.56778 | load: 0.00250 [I 2022-10-26 21:47:07] ...trap/engines/engine.py.230: loss: 0.11801 [I 2022-10-26 21:47:07] ...optimizers/trijoint.py.151: total_norm: 16.345558 [I 2022-10-26 21:47:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 1430/2304 [I 2022-10-26 21:47:12] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:08:07 [I 2022-10-26 21:47:12] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00252 [I 2022-10-26 21:47:12] ...trap/engines/engine.py.230: loss: 0.18184 [I 2022-10-26 21:47:12] ...optimizers/trijoint.py.151: total_norm: 18.640793 [I 2022-10-26 21:47:18] ...trap/engines/engine.py.224: train: epoch 31 | batch 1440/2304 [I 2022-10-26 21:47:18] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:08:02 [I 2022-10-26 21:47:18] ...trap/engines/engine.py.229: process: 0.56565 | load: 0.00245 [I 2022-10-26 21:47:18] ...trap/engines/engine.py.230: loss: 0.14982 [I 2022-10-26 21:47:18] ...optimizers/trijoint.py.151: total_norm: 13.614580 [I 2022-10-26 21:47:24] ...trap/engines/engine.py.224: train: epoch 31 | batch 1450/2304 [I 2022-10-26 21:47:24] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:51 [I 2022-10-26 21:47:24] ...trap/engines/engine.py.229: process: 0.55512 | load: 0.00241 [I 2022-10-26 21:47:24] ...trap/engines/engine.py.230: loss: 0.20383 [I 2022-10-26 21:47:24] ...optimizers/trijoint.py.151: total_norm: 15.666916 [I 2022-10-26 21:47:29] ...trap/engines/engine.py.224: train: epoch 31 | batch 1460/2304 [I 2022-10-26 21:47:29] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:08:02 [I 2022-10-26 21:47:29] ...trap/engines/engine.py.229: process: 0.58048 | load: 0.00272 [I 2022-10-26 21:47:29] ...trap/engines/engine.py.230: loss: 0.09564 [I 2022-10-26 21:47:29] ...optimizers/trijoint.py.151: total_norm: 8.974153 [I 2022-10-26 21:47:35] ...trap/engines/engine.py.224: train: epoch 31 | batch 1470/2304 [I 2022-10-26 21:47:35] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:45 [I 2022-10-26 21:47:35] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00244 [I 2022-10-26 21:47:35] ...trap/engines/engine.py.230: loss: 0.17721 [I 2022-10-26 21:47:35] ...optimizers/trijoint.py.151: total_norm: 14.066704 [I 2022-10-26 21:47:40] ...trap/engines/engine.py.224: train: epoch 31 | batch 1480/2304 [I 2022-10-26 21:47:40] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:34 [I 2022-10-26 21:47:40] ...trap/engines/engine.py.229: process: 0.55225 | load: 0.00243 [I 2022-10-26 21:47:40] ...trap/engines/engine.py.230: loss: 0.18333 [I 2022-10-26 21:47:40] ...optimizers/trijoint.py.151: total_norm: 19.629627 [I 2022-10-26 21:47:46] ...trap/engines/engine.py.224: train: epoch 31 | batch 1490/2304 [I 2022-10-26 21:47:46] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:30 [I 2022-10-26 21:47:46] ...trap/engines/engine.py.229: process: 0.54783 | load: 0.00249 [I 2022-10-26 21:47:46] ...trap/engines/engine.py.230: loss: 0.18464 [I 2022-10-26 21:47:46] ...optimizers/trijoint.py.151: total_norm: 26.988878 [I 2022-10-26 21:47:51] ...trap/engines/engine.py.224: train: epoch 31 | batch 1500/2304 [I 2022-10-26 21:47:51] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:28 [I 2022-10-26 21:47:51] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00246 [I 2022-10-26 21:47:51] ...trap/engines/engine.py.230: loss: 0.13530 [I 2022-10-26 21:47:51] ...optimizers/trijoint.py.151: total_norm: 12.376595 [I 2022-10-26 21:47:57] ...trap/engines/engine.py.224: train: epoch 31 | batch 1510/2304 [I 2022-10-26 21:47:57] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:19 [I 2022-10-26 21:47:57] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00243 [I 2022-10-26 21:47:57] ...trap/engines/engine.py.230: loss: 0.24016 [I 2022-10-26 21:47:57] ...optimizers/trijoint.py.151: total_norm: 16.887381 [I 2022-10-26 21:48:03] ...trap/engines/engine.py.224: train: epoch 31 | batch 1520/2304 [I 2022-10-26 21:48:03] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:12 [I 2022-10-26 21:48:03] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00247 [I 2022-10-26 21:48:03] ...trap/engines/engine.py.230: loss: 0.15423 [I 2022-10-26 21:48:03] ...optimizers/trijoint.py.151: total_norm: 22.375252 [I 2022-10-26 21:48:08] ...trap/engines/engine.py.224: train: epoch 31 | batch 1530/2304 [I 2022-10-26 21:48:08] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:07:11 [I 2022-10-26 21:48:08] ...trap/engines/engine.py.229: process: 0.54439 | load: 0.00245 [I 2022-10-26 21:48:08] ...trap/engines/engine.py.230: loss: 0.16522 [I 2022-10-26 21:48:08] ...optimizers/trijoint.py.151: total_norm: 20.169336 [I 2022-10-26 21:48:14] ...trap/engines/engine.py.224: train: epoch 31 | batch 1540/2304 [I 2022-10-26 21:48:14] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:07:03 [I 2022-10-26 21:48:14] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00240 [I 2022-10-26 21:48:14] ...trap/engines/engine.py.230: loss: 0.14942 [I 2022-10-26 21:48:14] ...optimizers/trijoint.py.151: total_norm: 17.909018 [I 2022-10-26 21:48:19] ...trap/engines/engine.py.224: train: epoch 31 | batch 1550/2304 [I 2022-10-26 21:48:19] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:06:57 [I 2022-10-26 21:48:19] ...trap/engines/engine.py.229: process: 0.55284 | load: 0.00245 [I 2022-10-26 21:48:19] ...trap/engines/engine.py.230: loss: 0.19308 [I 2022-10-26 21:48:19] ...optimizers/trijoint.py.151: total_norm: 18.151287 [I 2022-10-26 21:48:25] ...trap/engines/engine.py.224: train: epoch 31 | batch 1560/2304 [I 2022-10-26 21:48:25] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:54 [I 2022-10-26 21:48:25] ...trap/engines/engine.py.229: process: 0.56212 | load: 0.00243 [I 2022-10-26 21:48:25] ...trap/engines/engine.py.230: loss: 0.14640 [I 2022-10-26 21:48:25] ...optimizers/trijoint.py.151: total_norm: 15.935380 [I 2022-10-26 21:48:30] ...trap/engines/engine.py.224: train: epoch 31 | batch 1570/2304 [I 2022-10-26 21:48:30] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:48 [I 2022-10-26 21:48:30] ...trap/engines/engine.py.229: process: 0.55672 | load: 0.00243 [I 2022-10-26 21:48:30] ...trap/engines/engine.py.230: loss: 0.14280 [I 2022-10-26 21:48:30] ...optimizers/trijoint.py.151: total_norm: 10.618247 [I 2022-10-26 21:48:36] ...trap/engines/engine.py.224: train: epoch 31 | batch 1580/2304 [I 2022-10-26 21:48:36] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:39 [I 2022-10-26 21:48:36] ...trap/engines/engine.py.229: process: 0.54835 | load: 0.00248 [I 2022-10-26 21:48:36] ...trap/engines/engine.py.230: loss: 0.17975 [I 2022-10-26 21:48:36] ...optimizers/trijoint.py.151: total_norm: 21.941410 [I 2022-10-26 21:48:42] ...trap/engines/engine.py.224: train: epoch 31 | batch 1590/2304 [I 2022-10-26 21:48:42] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:38 [I 2022-10-26 21:48:42] ...trap/engines/engine.py.229: process: 0.54443 | load: 0.00244 [I 2022-10-26 21:48:42] ...trap/engines/engine.py.230: loss: 0.13004 [I 2022-10-26 21:48:42] ...optimizers/trijoint.py.151: total_norm: 8.005712 [I 2022-10-26 21:48:47] ...trap/engines/engine.py.224: train: epoch 31 | batch 1600/2304 [I 2022-10-26 21:48:47] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:28 [I 2022-10-26 21:48:47] ...trap/engines/engine.py.229: process: 0.54023 | load: 0.00240 [I 2022-10-26 21:48:47] ...trap/engines/engine.py.230: loss: 0.18832 [I 2022-10-26 21:48:47] ...optimizers/trijoint.py.151: total_norm: 18.382542 [I 2022-10-26 21:48:53] ...trap/engines/engine.py.224: train: epoch 31 | batch 1610/2304 [I 2022-10-26 21:48:53] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:24 [I 2022-10-26 21:48:53] ...trap/engines/engine.py.229: process: 0.55317 | load: 0.00245 [I 2022-10-26 21:48:53] ...trap/engines/engine.py.230: loss: 0.16978 [I 2022-10-26 21:48:53] ...optimizers/trijoint.py.151: total_norm: 15.094083 [I 2022-10-26 21:48:58] ...trap/engines/engine.py.224: train: epoch 31 | batch 1620/2304 [I 2022-10-26 21:48:58] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:19 [I 2022-10-26 21:48:58] ...trap/engines/engine.py.229: process: 0.55863 | load: 0.00244 [I 2022-10-26 21:48:58] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-26 21:48:58] ...optimizers/trijoint.py.151: total_norm: 9.560602 [I 2022-10-26 21:49:04] ...trap/engines/engine.py.224: train: epoch 31 | batch 1630/2304 [I 2022-10-26 21:49:04] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:14 [I 2022-10-26 21:49:04] ...trap/engines/engine.py.229: process: 0.56233 | load: 0.00245 [I 2022-10-26 21:49:04] ...trap/engines/engine.py.230: loss: 0.26078 [I 2022-10-26 21:49:04] ...optimizers/trijoint.py.151: total_norm: 22.691370 [I 2022-10-26 21:49:10] ...trap/engines/engine.py.224: train: epoch 31 | batch 1640/2304 [I 2022-10-26 21:49:10] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:06:10 [I 2022-10-26 21:49:10] ...trap/engines/engine.py.229: process: 0.56724 | load: 0.00242 [I 2022-10-26 21:49:10] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-26 21:49:10] ...optimizers/trijoint.py.151: total_norm: 9.845712 [I 2022-10-26 21:49:15] ...trap/engines/engine.py.224: train: epoch 31 | batch 1650/2304 [I 2022-10-26 21:49:15] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:06:05 [I 2022-10-26 21:49:15] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00245 [I 2022-10-26 21:49:15] ...trap/engines/engine.py.230: loss: 0.12845 [I 2022-10-26 21:49:15] ...optimizers/trijoint.py.151: total_norm: 15.145282 [I 2022-10-26 21:49:21] ...trap/engines/engine.py.224: train: epoch 31 | batch 1660/2304 [I 2022-10-26 21:49:21] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:06:00 [I 2022-10-26 21:49:21] ...trap/engines/engine.py.229: process: 0.55350 | load: 0.00261 [I 2022-10-26 21:49:21] ...trap/engines/engine.py.230: loss: 0.12289 [I 2022-10-26 21:49:21] ...optimizers/trijoint.py.151: total_norm: 14.023066 [I 2022-10-26 21:49:26] ...trap/engines/engine.py.224: train: epoch 31 | batch 1670/2304 [I 2022-10-26 21:49:26] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:55 [I 2022-10-26 21:49:26] ...trap/engines/engine.py.229: process: 0.56070 | load: 0.00268 [I 2022-10-26 21:49:26] ...trap/engines/engine.py.230: loss: 0.23381 [I 2022-10-26 21:49:26] ...optimizers/trijoint.py.151: total_norm: 25.737335 [I 2022-10-26 21:49:32] ...trap/engines/engine.py.224: train: epoch 31 | batch 1680/2304 [I 2022-10-26 21:49:32] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:45 [I 2022-10-26 21:49:32] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00252 [I 2022-10-26 21:49:32] ...trap/engines/engine.py.230: loss: 0.19172 [I 2022-10-26 21:49:32] ...optimizers/trijoint.py.151: total_norm: 25.712975 [I 2022-10-26 21:49:38] ...trap/engines/engine.py.224: train: epoch 31 | batch 1690/2304 [I 2022-10-26 21:49:38] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:42 [I 2022-10-26 21:49:38] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00250 [I 2022-10-26 21:49:38] ...trap/engines/engine.py.230: loss: 0.16504 [I 2022-10-26 21:49:38] ...optimizers/trijoint.py.151: total_norm: 22.218819 [I 2022-10-26 21:49:43] ...trap/engines/engine.py.224: train: epoch 31 | batch 1700/2304 [I 2022-10-26 21:49:43] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:35 [I 2022-10-26 21:49:43] ...trap/engines/engine.py.229: process: 0.55557 | load: 0.00244 [I 2022-10-26 21:49:43] ...trap/engines/engine.py.230: loss: 0.18376 [I 2022-10-26 21:49:43] ...optimizers/trijoint.py.151: total_norm: 16.984205 [I 2022-10-26 21:49:49] ...trap/engines/engine.py.224: train: epoch 31 | batch 1710/2304 [I 2022-10-26 21:49:49] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:30 [I 2022-10-26 21:49:49] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00248 [I 2022-10-26 21:49:49] ...trap/engines/engine.py.230: loss: 0.16115 [I 2022-10-26 21:49:49] ...optimizers/trijoint.py.151: total_norm: 13.774782 [I 2022-10-26 21:49:54] ...trap/engines/engine.py.224: train: epoch 31 | batch 1720/2304 [I 2022-10-26 21:49:54] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:27 [I 2022-10-26 21:49:54] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00245 [I 2022-10-26 21:49:54] ...trap/engines/engine.py.230: loss: 0.13300 [I 2022-10-26 21:49:54] ...optimizers/trijoint.py.151: total_norm: 14.128826 [I 2022-10-26 21:50:00] ...trap/engines/engine.py.224: train: epoch 31 | batch 1730/2304 [I 2022-10-26 21:50:00] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:15 [I 2022-10-26 21:50:00] ...trap/engines/engine.py.229: process: 0.53886 | load: 0.00243 [I 2022-10-26 21:50:00] ...trap/engines/engine.py.230: loss: 0.15950 [I 2022-10-26 21:50:00] ...optimizers/trijoint.py.151: total_norm: 18.119839 [I 2022-10-26 21:50:05] ...trap/engines/engine.py.224: train: epoch 31 | batch 1740/2304 [I 2022-10-26 21:50:05] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:05:13 [I 2022-10-26 21:50:05] ...trap/engines/engine.py.229: process: 0.56794 | load: 0.00243 [I 2022-10-26 21:50:05] ...trap/engines/engine.py.230: loss: 0.14653 [I 2022-10-26 21:50:05] ...optimizers/trijoint.py.151: total_norm: 14.044780 [I 2022-10-26 21:50:11] ...trap/engines/engine.py.224: train: epoch 31 | batch 1750/2304 [I 2022-10-26 21:50:11] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:08 [I 2022-10-26 21:50:11] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00240 [I 2022-10-26 21:50:11] ...trap/engines/engine.py.230: loss: 0.16507 [I 2022-10-26 21:50:11] ...optimizers/trijoint.py.151: total_norm: 14.980294 [I 2022-10-26 21:50:17] ...trap/engines/engine.py.224: train: epoch 31 | batch 1760/2304 [I 2022-10-26 21:50:17] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:05:02 [I 2022-10-26 21:50:17] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00242 [I 2022-10-26 21:50:17] ...trap/engines/engine.py.230: loss: 0.15401 [I 2022-10-26 21:50:17] ...optimizers/trijoint.py.151: total_norm: 17.692369 [I 2022-10-26 21:50:22] ...trap/engines/engine.py.224: train: epoch 31 | batch 1770/2304 [I 2022-10-26 21:50:22] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:56 [I 2022-10-26 21:50:22] ...trap/engines/engine.py.229: process: 0.54501 | load: 0.00241 [I 2022-10-26 21:50:22] ...trap/engines/engine.py.230: loss: 0.15420 [I 2022-10-26 21:50:22] ...optimizers/trijoint.py.151: total_norm: 14.499181 [I 2022-10-26 21:50:28] ...trap/engines/engine.py.224: train: epoch 31 | batch 1780/2304 [I 2022-10-26 21:50:28] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:59 [I 2022-10-26 21:50:28] ...trap/engines/engine.py.229: process: 0.55835 | load: 0.00253 [I 2022-10-26 21:50:28] ...trap/engines/engine.py.230: loss: 0.19928 [I 2022-10-26 21:50:28] ...optimizers/trijoint.py.151: total_norm: 22.223835 [I 2022-10-26 21:50:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 1790/2304 [I 2022-10-26 21:50:33] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:04:47 [I 2022-10-26 21:50:33] ...trap/engines/engine.py.229: process: 0.55667 | load: 0.00246 [I 2022-10-26 21:50:33] ...trap/engines/engine.py.230: loss: 0.12776 [I 2022-10-26 21:50:33] ...optimizers/trijoint.py.151: total_norm: 10.410643 [I 2022-10-26 21:50:39] ...trap/engines/engine.py.224: train: epoch 31 | batch 1800/2304 [I 2022-10-26 21:50:39] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:40 [I 2022-10-26 21:50:39] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00241 [I 2022-10-26 21:50:39] ...trap/engines/engine.py.230: loss: 0.19281 [I 2022-10-26 21:50:39] ...optimizers/trijoint.py.151: total_norm: 27.312040 [I 2022-10-26 21:50:45] ...trap/engines/engine.py.224: train: epoch 31 | batch 1810/2304 [I 2022-10-26 21:50:45] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:34 [I 2022-10-26 21:50:45] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00245 [I 2022-10-26 21:50:45] ...trap/engines/engine.py.230: loss: 0.11787 [I 2022-10-26 21:50:45] ...optimizers/trijoint.py.151: total_norm: 26.397364 [I 2022-10-26 21:50:50] ...trap/engines/engine.py.224: train: epoch 31 | batch 1820/2304 [I 2022-10-26 21:50:50] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:29 [I 2022-10-26 21:50:50] ...trap/engines/engine.py.229: process: 0.55477 | load: 0.00242 [I 2022-10-26 21:50:50] ...trap/engines/engine.py.230: loss: 0.13713 [I 2022-10-26 21:50:50] ...optimizers/trijoint.py.151: total_norm: 15.183008 [I 2022-10-26 21:50:56] ...trap/engines/engine.py.224: train: epoch 31 | batch 1830/2304 [I 2022-10-26 21:50:56] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:23 [I 2022-10-26 21:50:56] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00242 [I 2022-10-26 21:50:56] ...trap/engines/engine.py.230: loss: 0.20069 [I 2022-10-26 21:50:56] ...optimizers/trijoint.py.151: total_norm: 37.104984 [I 2022-10-26 21:51:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 1840/2304 [I 2022-10-26 21:51:01] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:04:17 [I 2022-10-26 21:51:01] ...trap/engines/engine.py.229: process: 0.55114 | load: 0.00245 [I 2022-10-26 21:51:01] ...trap/engines/engine.py.230: loss: 0.21027 [I 2022-10-26 21:51:01] ...optimizers/trijoint.py.151: total_norm: 22.923346 [I 2022-10-26 21:51:07] ...trap/engines/engine.py.224: train: epoch 31 | batch 1850/2304 [I 2022-10-26 21:51:07] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:04:11 [I 2022-10-26 21:51:07] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00260 [I 2022-10-26 21:51:07] ...trap/engines/engine.py.230: loss: 0.21790 [I 2022-10-26 21:51:07] ...optimizers/trijoint.py.151: total_norm: 17.913006 [I 2022-10-26 21:51:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 1860/2304 [I 2022-10-26 21:51:12] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:04:05 [I 2022-10-26 21:51:12] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00247 [I 2022-10-26 21:51:12] ...trap/engines/engine.py.230: loss: 0.16675 [I 2022-10-26 21:51:12] ...optimizers/trijoint.py.151: total_norm: 26.142927 [I 2022-10-26 21:51:18] ...trap/engines/engine.py.224: train: epoch 31 | batch 1870/2304 [I 2022-10-26 21:51:18] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:04:00 [I 2022-10-26 21:51:18] ...trap/engines/engine.py.229: process: 0.55154 | load: 0.00248 [I 2022-10-26 21:51:18] ...trap/engines/engine.py.230: loss: 0.25497 [I 2022-10-26 21:51:18] ...optimizers/trijoint.py.151: total_norm: 33.955990 [I 2022-10-26 21:51:23] ...trap/engines/engine.py.224: train: epoch 31 | batch 1880/2304 [I 2022-10-26 21:51:23] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:56 [I 2022-10-26 21:51:23] ...trap/engines/engine.py.229: process: 0.56446 | load: 0.00252 [I 2022-10-26 21:51:23] ...trap/engines/engine.py.230: loss: 0.20884 [I 2022-10-26 21:51:23] ...optimizers/trijoint.py.151: total_norm: 16.389442 [I 2022-10-26 21:51:29] ...trap/engines/engine.py.224: train: epoch 31 | batch 1890/2304 [I 2022-10-26 21:51:29] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:49 [I 2022-10-26 21:51:29] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00243 [I 2022-10-26 21:51:29] ...trap/engines/engine.py.230: loss: 0.18776 [I 2022-10-26 21:51:29] ...optimizers/trijoint.py.151: total_norm: 18.370840 [I 2022-10-26 21:51:35] ...trap/engines/engine.py.224: train: epoch 31 | batch 1900/2304 [I 2022-10-26 21:51:35] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:46 [I 2022-10-26 21:51:35] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00244 [I 2022-10-26 21:51:35] ...trap/engines/engine.py.230: loss: 0.12593 [I 2022-10-26 21:51:35] ...optimizers/trijoint.py.151: total_norm: 10.120969 [I 2022-10-26 21:51:40] ...trap/engines/engine.py.224: train: epoch 31 | batch 1910/2304 [I 2022-10-26 21:51:40] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:03:39 [I 2022-10-26 21:51:40] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00244 [I 2022-10-26 21:51:40] ...trap/engines/engine.py.230: loss: 0.18271 [I 2022-10-26 21:51:40] ...optimizers/trijoint.py.151: total_norm: 21.291248 [I 2022-10-26 21:51:46] ...trap/engines/engine.py.224: train: epoch 31 | batch 1920/2304 [I 2022-10-26 21:51:46] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:33 [I 2022-10-26 21:51:46] ...trap/engines/engine.py.229: process: 0.54992 | load: 0.00241 [I 2022-10-26 21:51:46] ...trap/engines/engine.py.230: loss: 0.20037 [I 2022-10-26 21:51:46] ...optimizers/trijoint.py.151: total_norm: 14.527695 [I 2022-10-26 21:51:51] ...trap/engines/engine.py.224: train: epoch 31 | batch 1930/2304 [I 2022-10-26 21:51:51] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:27 [I 2022-10-26 21:51:51] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00244 [I 2022-10-26 21:51:51] ...trap/engines/engine.py.230: loss: 0.22164 [I 2022-10-26 21:51:51] ...optimizers/trijoint.py.151: total_norm: 27.881914 [I 2022-10-26 21:51:57] ...trap/engines/engine.py.224: train: epoch 31 | batch 1940/2304 [I 2022-10-26 21:51:57] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:24 [I 2022-10-26 21:51:57] ...trap/engines/engine.py.229: process: 0.56913 | load: 0.00244 [I 2022-10-26 21:51:57] ...trap/engines/engine.py.230: loss: 0.16115 [I 2022-10-26 21:51:57] ...optimizers/trijoint.py.151: total_norm: 24.197758 [I 2022-10-26 21:52:03] ...trap/engines/engine.py.224: train: epoch 31 | batch 1950/2304 [I 2022-10-26 21:52:03] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:17 [I 2022-10-26 21:52:03] ...trap/engines/engine.py.229: process: 0.56045 | load: 0.00246 [I 2022-10-26 21:52:03] ...trap/engines/engine.py.230: loss: 0.23538 [I 2022-10-26 21:52:03] ...optimizers/trijoint.py.151: total_norm: 17.778606 [I 2022-10-26 21:52:08] ...trap/engines/engine.py.224: train: epoch 31 | batch 1960/2304 [I 2022-10-26 21:52:08] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:03:09 [I 2022-10-26 21:52:08] ...trap/engines/engine.py.229: process: 0.54593 | load: 0.00255 [I 2022-10-26 21:52:08] ...trap/engines/engine.py.230: loss: 0.15793 [I 2022-10-26 21:52:08] ...optimizers/trijoint.py.151: total_norm: 38.140392 [I 2022-10-26 21:52:14] ...trap/engines/engine.py.224: train: epoch 31 | batch 1970/2304 [I 2022-10-26 21:52:14] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:03:04 [I 2022-10-26 21:52:14] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00248 [I 2022-10-26 21:52:14] ...trap/engines/engine.py.230: loss: 0.21405 [I 2022-10-26 21:52:14] ...optimizers/trijoint.py.151: total_norm: 22.535589 [I 2022-10-26 21:52:19] ...trap/engines/engine.py.224: train: epoch 31 | batch 1980/2304 [I 2022-10-26 21:52:19] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:02:58 [I 2022-10-26 21:52:19] ...trap/engines/engine.py.229: process: 0.54331 | load: 0.00246 [I 2022-10-26 21:52:19] ...trap/engines/engine.py.230: loss: 0.14858 [I 2022-10-26 21:52:19] ...optimizers/trijoint.py.151: total_norm: 16.181234 [I 2022-10-26 21:52:25] ...trap/engines/engine.py.224: train: epoch 31 | batch 1990/2304 [I 2022-10-26 21:52:25] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:52 [I 2022-10-26 21:52:25] ...trap/engines/engine.py.229: process: 0.53990 | load: 0.00248 [I 2022-10-26 21:52:25] ...trap/engines/engine.py.230: loss: 0.13762 [I 2022-10-26 21:52:25] ...optimizers/trijoint.py.151: total_norm: 9.473424 [I 2022-10-26 21:52:30] ...trap/engines/engine.py.224: train: epoch 31 | batch 2000/2304 [I 2022-10-26 21:52:30] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:49 [I 2022-10-26 21:52:30] ...trap/engines/engine.py.229: process: 0.55436 | load: 0.00243 [I 2022-10-26 21:52:30] ...trap/engines/engine.py.230: loss: 0.29496 [I 2022-10-26 21:52:30] ...optimizers/trijoint.py.151: total_norm: 28.409384 [I 2022-10-26 21:52:36] ...trap/engines/engine.py.224: train: epoch 31 | batch 2010/2304 [I 2022-10-26 21:52:36] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:43 [I 2022-10-26 21:52:36] ...trap/engines/engine.py.229: process: 0.54527 | load: 0.00243 [I 2022-10-26 21:52:36] ...trap/engines/engine.py.230: loss: 0.11669 [I 2022-10-26 21:52:36] ...optimizers/trijoint.py.151: total_norm: 12.560711 [I 2022-10-26 21:52:41] ...trap/engines/engine.py.224: train: epoch 31 | batch 2020/2304 [I 2022-10-26 21:52:41] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:37 [I 2022-10-26 21:52:41] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00245 [I 2022-10-26 21:52:41] ...trap/engines/engine.py.230: loss: 0.22788 [I 2022-10-26 21:52:41] ...optimizers/trijoint.py.151: total_norm: 17.035128 [I 2022-10-26 21:52:47] ...trap/engines/engine.py.224: train: epoch 31 | batch 2030/2304 [I 2022-10-26 21:52:47] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:33 [I 2022-10-26 21:52:47] ...trap/engines/engine.py.229: process: 0.56309 | load: 0.00245 [I 2022-10-26 21:52:47] ...trap/engines/engine.py.230: loss: 0.15509 [I 2022-10-26 21:52:47] ...optimizers/trijoint.py.151: total_norm: 9.961014 [I 2022-10-26 21:52:53] ...trap/engines/engine.py.224: train: epoch 31 | batch 2040/2304 [I 2022-10-26 21:52:53] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:26 [I 2022-10-26 21:52:53] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00243 [I 2022-10-26 21:52:53] ...trap/engines/engine.py.230: loss: 0.22374 [I 2022-10-26 21:52:53] ...optimizers/trijoint.py.151: total_norm: 14.414254 [I 2022-10-26 21:52:58] ...trap/engines/engine.py.224: train: epoch 31 | batch 2050/2304 [I 2022-10-26 21:52:58] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:21 [I 2022-10-26 21:52:58] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00243 [I 2022-10-26 21:52:58] ...trap/engines/engine.py.230: loss: 0.19357 [I 2022-10-26 21:52:58] ...optimizers/trijoint.py.151: total_norm: 19.364599 [I 2022-10-26 21:53:04] ...trap/engines/engine.py.224: train: epoch 31 | batch 2060/2304 [I 2022-10-26 21:53:04] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:15 [I 2022-10-26 21:53:04] ...trap/engines/engine.py.229: process: 0.56115 | load: 0.00242 [I 2022-10-26 21:53:04] ...trap/engines/engine.py.230: loss: 0.20748 [I 2022-10-26 21:53:04] ...optimizers/trijoint.py.151: total_norm: 28.519123 [I 2022-10-26 21:53:09] ...trap/engines/engine.py.224: train: epoch 31 | batch 2070/2304 [I 2022-10-26 21:53:09] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:02:10 [I 2022-10-26 21:53:09] ...trap/engines/engine.py.229: process: 0.56348 | load: 0.00244 [I 2022-10-26 21:53:09] ...trap/engines/engine.py.230: loss: 0.17629 [I 2022-10-26 21:53:09] ...optimizers/trijoint.py.151: total_norm: 29.463156 [I 2022-10-26 21:53:15] ...trap/engines/engine.py.224: train: epoch 31 | batch 2080/2304 [I 2022-10-26 21:53:15] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:02:05 [I 2022-10-26 21:53:15] ...trap/engines/engine.py.229: process: 0.56387 | load: 0.00248 [I 2022-10-26 21:53:15] ...trap/engines/engine.py.230: loss: 0.19557 [I 2022-10-26 21:53:15] ...optimizers/trijoint.py.151: total_norm: 20.954077 [I 2022-10-26 21:53:20] ...trap/engines/engine.py.224: train: epoch 31 | batch 2090/2304 [I 2022-10-26 21:53:20] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:59 [I 2022-10-26 21:53:20] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00250 [I 2022-10-26 21:53:20] ...trap/engines/engine.py.230: loss: 0.12333 [I 2022-10-26 21:53:20] ...optimizers/trijoint.py.151: total_norm: 11.025066 [I 2022-10-26 21:53:26] ...trap/engines/engine.py.224: train: epoch 31 | batch 2100/2304 [I 2022-10-26 21:53:26] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:54 [I 2022-10-26 21:53:26] ...trap/engines/engine.py.229: process: 0.56764 | load: 0.00244 [I 2022-10-26 21:53:26] ...trap/engines/engine.py.230: loss: 0.15150 [I 2022-10-26 21:53:26] ...optimizers/trijoint.py.151: total_norm: 24.842607 [I 2022-10-26 21:53:32] ...trap/engines/engine.py.224: train: epoch 31 | batch 2110/2304 [I 2022-10-26 21:53:32] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:47 [I 2022-10-26 21:53:32] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00244 [I 2022-10-26 21:53:32] ...trap/engines/engine.py.230: loss: 0.22143 [I 2022-10-26 21:53:32] ...optimizers/trijoint.py.151: total_norm: 17.915016 [I 2022-10-26 21:53:37] ...trap/engines/engine.py.224: train: epoch 31 | batch 2120/2304 [I 2022-10-26 21:53:37] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:42 [I 2022-10-26 21:53:37] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00243 [I 2022-10-26 21:53:37] ...trap/engines/engine.py.230: loss: 0.13303 [I 2022-10-26 21:53:37] ...optimizers/trijoint.py.151: total_norm: 16.870962 [I 2022-10-26 21:53:43] ...trap/engines/engine.py.224: train: epoch 31 | batch 2130/2304 [I 2022-10-26 21:53:43] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:37 [I 2022-10-26 21:53:43] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00243 [I 2022-10-26 21:53:43] ...trap/engines/engine.py.230: loss: 0.22304 [I 2022-10-26 21:53:43] ...optimizers/trijoint.py.151: total_norm: 18.014769 [I 2022-10-26 21:53:48] ...trap/engines/engine.py.224: train: epoch 31 | batch 2140/2304 [I 2022-10-26 21:53:48] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:31 [I 2022-10-26 21:53:48] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00241 [I 2022-10-26 21:53:48] ...trap/engines/engine.py.230: loss: 0.16512 [I 2022-10-26 21:53:48] ...optimizers/trijoint.py.151: total_norm: 18.305590 [I 2022-10-26 21:53:54] ...trap/engines/engine.py.224: train: epoch 31 | batch 2150/2304 [I 2022-10-26 21:53:54] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:25 [I 2022-10-26 21:53:54] ...trap/engines/engine.py.229: process: 0.55383 | load: 0.00243 [I 2022-10-26 21:53:54] ...trap/engines/engine.py.230: loss: 0.16125 [I 2022-10-26 21:53:54] ...optimizers/trijoint.py.151: total_norm: 20.339598 [I 2022-10-26 21:54:00] ...trap/engines/engine.py.224: train: epoch 31 | batch 2160/2304 [I 2022-10-26 21:54:00] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:20 [I 2022-10-26 21:54:00] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00244 [I 2022-10-26 21:54:00] ...trap/engines/engine.py.230: loss: 0.17508 [I 2022-10-26 21:54:00] ...optimizers/trijoint.py.151: total_norm: 13.612314 [I 2022-10-26 21:54:05] ...trap/engines/engine.py.224: train: epoch 31 | batch 2170/2304 [I 2022-10-26 21:54:05] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:01:15 [I 2022-10-26 21:54:05] ...trap/engines/engine.py.229: process: 0.56226 | load: 0.00244 [I 2022-10-26 21:54:05] ...trap/engines/engine.py.230: loss: 0.15176 [I 2022-10-26 21:54:05] ...optimizers/trijoint.py.151: total_norm: 18.007198 [I 2022-10-26 21:54:11] ...trap/engines/engine.py.224: train: epoch 31 | batch 2180/2304 [I 2022-10-26 21:54:11] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:09 [I 2022-10-26 21:54:11] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00241 [I 2022-10-26 21:54:11] ...trap/engines/engine.py.230: loss: 0.17054 [I 2022-10-26 21:54:11] ...optimizers/trijoint.py.151: total_norm: 13.632014 [I 2022-10-26 21:54:16] ...trap/engines/engine.py.224: train: epoch 31 | batch 2190/2304 [I 2022-10-26 21:54:16] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:01:02 [I 2022-10-26 21:54:16] ...trap/engines/engine.py.229: process: 0.53862 | load: 0.00251 [I 2022-10-26 21:54:16] ...trap/engines/engine.py.230: loss: 0.18674 [I 2022-10-26 21:54:16] ...optimizers/trijoint.py.151: total_norm: 15.014912 [I 2022-10-26 21:54:22] ...trap/engines/engine.py.224: train: epoch 31 | batch 2200/2304 [I 2022-10-26 21:54:22] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:57 [I 2022-10-26 21:54:22] ...trap/engines/engine.py.229: process: 0.54611 | load: 0.00246 [I 2022-10-26 21:54:22] ...trap/engines/engine.py.230: loss: 0.17776 [I 2022-10-26 21:54:22] ...optimizers/trijoint.py.151: total_norm: 32.500515 [I 2022-10-26 21:54:27] ...trap/engines/engine.py.224: train: epoch 31 | batch 2210/2304 [I 2022-10-26 21:54:27] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:52 [I 2022-10-26 21:54:27] ...trap/engines/engine.py.229: process: 0.54980 | load: 0.00246 [I 2022-10-26 21:54:27] ...trap/engines/engine.py.230: loss: 0.26157 [I 2022-10-26 21:54:27] ...optimizers/trijoint.py.151: total_norm: 25.551069 [I 2022-10-26 21:54:33] ...trap/engines/engine.py.224: train: epoch 31 | batch 2220/2304 [I 2022-10-26 21:54:33] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:46 [I 2022-10-26 21:54:33] ...trap/engines/engine.py.229: process: 0.55383 | load: 0.00244 [I 2022-10-26 21:54:33] ...trap/engines/engine.py.230: loss: 0.19587 [I 2022-10-26 21:54:33] ...optimizers/trijoint.py.151: total_norm: 22.907112 [I 2022-10-26 21:54:38] ...trap/engines/engine.py.224: train: epoch 31 | batch 2230/2304 [I 2022-10-26 21:54:38] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:40 [I 2022-10-26 21:54:38] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00245 [I 2022-10-26 21:54:38] ...trap/engines/engine.py.230: loss: 0.19767 [I 2022-10-26 21:54:38] ...optimizers/trijoint.py.151: total_norm: 24.459240 [I 2022-10-26 21:54:44] ...trap/engines/engine.py.224: train: epoch 31 | batch 2240/2304 [I 2022-10-26 21:54:44] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:35 [I 2022-10-26 21:54:44] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00243 [I 2022-10-26 21:54:44] ...trap/engines/engine.py.230: loss: 0.17328 [I 2022-10-26 21:54:44] ...optimizers/trijoint.py.151: total_norm: 14.923514 [I 2022-10-26 21:54:50] ...trap/engines/engine.py.224: train: epoch 31 | batch 2250/2304 [I 2022-10-26 21:54:50] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:29 [I 2022-10-26 21:54:50] ...trap/engines/engine.py.229: process: 0.54267 | load: 0.00243 [I 2022-10-26 21:54:50] ...trap/engines/engine.py.230: loss: 0.24343 [I 2022-10-26 21:54:50] ...optimizers/trijoint.py.151: total_norm: 30.549984 [I 2022-10-26 21:54:55] ...trap/engines/engine.py.224: train: epoch 31 | batch 2260/2304 [I 2022-10-26 21:54:55] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:24 [I 2022-10-26 21:54:55] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00244 [I 2022-10-26 21:54:55] ...trap/engines/engine.py.230: loss: 0.14212 [I 2022-10-26 21:54:55] ...optimizers/trijoint.py.151: total_norm: 25.708197 [I 2022-10-26 21:55:01] ...trap/engines/engine.py.224: train: epoch 31 | batch 2270/2304 [I 2022-10-26 21:55:01] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:18 [I 2022-10-26 21:55:01] ...trap/engines/engine.py.229: process: 0.54720 | load: 0.00241 [I 2022-10-26 21:55:01] ...trap/engines/engine.py.230: loss: 0.17161 [I 2022-10-26 21:55:01] ...optimizers/trijoint.py.151: total_norm: 8.051403 [I 2022-10-26 21:55:06] ...trap/engines/engine.py.224: train: epoch 31 | batch 2280/2304 [I 2022-10-26 21:55:06] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:13 [I 2022-10-26 21:55:06] ...trap/engines/engine.py.229: process: 0.56541 | load: 0.00241 [I 2022-10-26 21:55:06] ...trap/engines/engine.py.230: loss: 0.23622 [I 2022-10-26 21:55:06] ...optimizers/trijoint.py.151: total_norm: 12.773583 [I 2022-10-26 21:55:12] ...trap/engines/engine.py.224: train: epoch 31 | batch 2290/2304 [I 2022-10-26 21:55:12] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:00:07 [I 2022-10-26 21:55:12] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00246 [I 2022-10-26 21:55:12] ...trap/engines/engine.py.230: loss: 0.18053 [I 2022-10-26 21:55:12] ...optimizers/trijoint.py.151: total_norm: 19.716536 [I 2022-10-26 21:55:17] ...trap/engines/engine.py.224: train: epoch 31 | batch 2300/2304 [I 2022-10-26 21:55:17] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:02 [I 2022-10-26 21:55:17] ...trap/engines/engine.py.229: process: 0.54083 | load: 0.00016 [I 2022-10-26 21:55:17] ...trap/engines/engine.py.230: loss: 0.18301 [I 2022-10-26 21:55:17] ...optimizers/trijoint.py.151: total_norm: 32.659245 [I 2022-10-26 21:55:20] ...trap/engines/engine.py.224: train: epoch 31 | batch 2304/2304 [I 2022-10-26 21:55:20] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:00 [I 2022-10-26 21:55:20] ...trap/engines/engine.py.229: process: 0.54272 | load: 0.00015 [I 2022-10-26 21:55:20] ...trap/engines/engine.py.230: loss: 0.17410 [I 2022-10-26 21:55:20] ...optimizers/trijoint.py.151: total_norm: 18.990582 [S 2022-10-26 21:55:21] ...trap/engines/engine.py.129: train_epoch.epoch: 31 [S 2022-10-26 21:55:21] ...trap/engines/engine.py.129: train_epoch.loss: 0.17850956248034625 [I 2022-10-26 21:55:21] ...s/criterions/triplet.py.84: margin = 0.21000000000000013 [I 2022-10-26 21:55:21] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 31 [I 2022-10-26 21:55:21] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 21:55:23] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 21:55:23] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 21:55:23] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 21:55:24] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 21:55:25] ...trap/engines/engine.py.321: eval: epoch 31 | batch 0/510 [I 2022-10-26 21:55:25] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:28:42 [I 2022-10-26 21:55:25] ...trap/engines/engine.py.326: process: 3.37784 | load: 2.94535 [I 2022-10-26 21:55:28] ...trap/engines/engine.py.321: eval: epoch 31 | batch 10/510 [I 2022-10-26 21:55:28] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:30 [I 2022-10-26 21:55:28] ...trap/engines/engine.py.326: process: 0.33176 | load: 0.00019 [I 2022-10-26 21:55:31] ...trap/engines/engine.py.321: eval: epoch 31 | batch 20/510 [I 2022-10-26 21:55:31] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:00 [I 2022-10-26 21:55:31] ...trap/engines/engine.py.326: process: 0.32071 | load: 0.00018 [I 2022-10-26 21:55:35] ...trap/engines/engine.py.321: eval: epoch 31 | batch 30/510 [I 2022-10-26 21:55:35] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:47 [I 2022-10-26 21:55:35] ...trap/engines/engine.py.326: process: 0.36515 | load: 0.00017 [I 2022-10-26 21:55:38] ...trap/engines/engine.py.321: eval: epoch 31 | batch 40/510 [I 2022-10-26 21:55:38] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:35 [I 2022-10-26 21:55:38] ...trap/engines/engine.py.326: process: 0.31967 | load: 0.00018 [I 2022-10-26 21:55:42] ...trap/engines/engine.py.321: eval: epoch 31 | batch 50/510 [I 2022-10-26 21:55:42] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:33 [I 2022-10-26 21:55:42] ...trap/engines/engine.py.326: process: 0.32331 | load: 0.00019 [I 2022-10-26 21:55:45] ...trap/engines/engine.py.321: eval: epoch 31 | batch 60/510 [I 2022-10-26 21:55:45] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:31 [I 2022-10-26 21:55:45] ...trap/engines/engine.py.326: process: 0.36308 | load: 0.00016 [I 2022-10-26 21:55:48] ...trap/engines/engine.py.321: eval: epoch 31 | batch 70/510 [I 2022-10-26 21:55:48] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:25 [I 2022-10-26 21:55:48] ...trap/engines/engine.py.326: process: 0.32652 | load: 0.00020 [I 2022-10-26 21:55:52] ...trap/engines/engine.py.321: eval: epoch 31 | batch 80/510 [I 2022-10-26 21:55:52] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-26 21:55:52] ...trap/engines/engine.py.326: process: 0.31955 | load: 0.00018 [I 2022-10-26 21:55:55] ...trap/engines/engine.py.321: eval: epoch 31 | batch 90/510 [I 2022-10-26 21:55:55] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:22 [I 2022-10-26 21:55:55] ...trap/engines/engine.py.326: process: 0.36638 | load: 0.00021 [I 2022-10-26 21:55:58] ...trap/engines/engine.py.321: eval: epoch 31 | batch 100/510 [I 2022-10-26 21:55:58] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 21:55:58] ...trap/engines/engine.py.326: process: 0.28426 | load: 0.00019 [I 2022-10-26 21:56:01] ...trap/engines/engine.py.321: eval: epoch 31 | batch 110/510 [I 2022-10-26 21:56:01] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 21:56:01] ...trap/engines/engine.py.326: process: 0.28053 | load: 0.00017 [I 2022-10-26 21:56:04] ...trap/engines/engine.py.321: eval: epoch 31 | batch 120/510 [I 2022-10-26 21:56:04] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-26 21:56:04] ...trap/engines/engine.py.326: process: 0.30871 | load: 0.00015 [I 2022-10-26 21:56:07] ...trap/engines/engine.py.321: eval: epoch 31 | batch 130/510 [I 2022-10-26 21:56:07] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:49 [I 2022-10-26 21:56:07] ...trap/engines/engine.py.326: process: 0.28949 | load: 0.00016 [I 2022-10-26 21:56:10] ...trap/engines/engine.py.321: eval: epoch 31 | batch 140/510 [I 2022-10-26 21:56:10] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:48 [I 2022-10-26 21:56:10] ...trap/engines/engine.py.326: process: 0.28402 | load: 0.00025 [I 2022-10-26 21:56:13] ...trap/engines/engine.py.321: eval: epoch 31 | batch 150/510 [I 2022-10-26 21:56:13] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:45 [I 2022-10-26 21:56:13] ...trap/engines/engine.py.326: process: 0.31326 | load: 0.00015 [I 2022-10-26 21:56:16] ...trap/engines/engine.py.321: eval: epoch 31 | batch 160/510 [I 2022-10-26 21:56:16] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:43 [I 2022-10-26 21:56:16] ...trap/engines/engine.py.326: process: 0.29964 | load: 0.00015 [I 2022-10-26 21:56:19] ...trap/engines/engine.py.321: eval: epoch 31 | batch 170/510 [I 2022-10-26 21:56:19] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-26 21:56:19] ...trap/engines/engine.py.326: process: 0.28227 | load: 0.00013 [I 2022-10-26 21:56:21] ...trap/engines/engine.py.321: eval: epoch 31 | batch 180/510 [I 2022-10-26 21:56:21] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:34 [I 2022-10-26 21:56:21] ...trap/engines/engine.py.326: process: 0.28234 | load: 0.00016 [I 2022-10-26 21:56:24] ...trap/engines/engine.py.321: eval: epoch 31 | batch 190/510 [I 2022-10-26 21:56:24] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 21:56:24] ...trap/engines/engine.py.326: process: 0.29084 | load: 0.00016 [I 2022-10-26 21:56:27] ...trap/engines/engine.py.321: eval: epoch 31 | batch 200/510 [I 2022-10-26 21:56:27] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 21:56:27] ...trap/engines/engine.py.326: process: 0.28849 | load: 0.00015 [I 2022-10-26 21:56:30] ...trap/engines/engine.py.321: eval: epoch 31 | batch 210/510 [I 2022-10-26 21:56:30] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 21:56:30] ...trap/engines/engine.py.326: process: 0.28225 | load: 0.00019 [I 2022-10-26 21:56:33] ...trap/engines/engine.py.321: eval: epoch 31 | batch 220/510 [I 2022-10-26 21:56:33] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:22 [I 2022-10-26 21:56:33] ...trap/engines/engine.py.326: process: 0.28035 | load: 0.00016 [I 2022-10-26 21:56:36] ...trap/engines/engine.py.321: eval: epoch 31 | batch 230/510 [I 2022-10-26 21:56:36] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 21:56:36] ...trap/engines/engine.py.326: process: 0.28865 | load: 0.00017 [I 2022-10-26 21:56:39] ...trap/engines/engine.py.321: eval: epoch 31 | batch 240/510 [I 2022-10-26 21:56:39] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 21:56:39] ...trap/engines/engine.py.326: process: 0.29049 | load: 0.00025 [I 2022-10-26 21:56:42] ...trap/engines/engine.py.321: eval: epoch 31 | batch 250/510 [I 2022-10-26 21:56:42] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-26 21:56:42] ...trap/engines/engine.py.326: process: 0.28084 | load: 0.00025 [I 2022-10-26 21:56:45] ...trap/engines/engine.py.321: eval: epoch 31 | batch 260/510 [I 2022-10-26 21:56:45] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-26 21:56:45] ...trap/engines/engine.py.326: process: 0.29426 | load: 0.00022 [I 2022-10-26 21:56:47] ...trap/engines/engine.py.321: eval: epoch 31 | batch 270/510 [I 2022-10-26 21:56:47] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-26 21:56:47] ...trap/engines/engine.py.326: process: 0.28451 | load: 0.00019 [I 2022-10-26 21:56:50] ...trap/engines/engine.py.321: eval: epoch 31 | batch 280/510 [I 2022-10-26 21:56:50] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:05 [I 2022-10-26 21:56:50] ...trap/engines/engine.py.326: process: 0.28805 | load: 0.00023 [I 2022-10-26 21:56:53] ...trap/engines/engine.py.321: eval: epoch 31 | batch 290/510 [I 2022-10-26 21:56:53] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 21:56:53] ...trap/engines/engine.py.326: process: 0.28338 | load: 0.00016 [I 2022-10-26 21:56:56] ...trap/engines/engine.py.321: eval: epoch 31 | batch 300/510 [I 2022-10-26 21:56:56] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 21:56:56] ...trap/engines/engine.py.326: process: 0.28100 | load: 0.00023 [I 2022-10-26 21:56:59] ...trap/engines/engine.py.321: eval: epoch 31 | batch 310/510 [I 2022-10-26 21:56:59] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 21:56:59] ...trap/engines/engine.py.326: process: 0.28660 | load: 0.00016 [I 2022-10-26 21:57:02] ...trap/engines/engine.py.321: eval: epoch 31 | batch 320/510 [I 2022-10-26 21:57:02] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 21:57:02] ...trap/engines/engine.py.326: process: 0.28950 | load: 0.00017 [I 2022-10-26 21:57:05] ...trap/engines/engine.py.321: eval: epoch 31 | batch 330/510 [I 2022-10-26 21:57:05] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 21:57:05] ...trap/engines/engine.py.326: process: 0.28853 | load: 0.00021 [I 2022-10-26 21:57:08] ...trap/engines/engine.py.321: eval: epoch 31 | batch 340/510 [I 2022-10-26 21:57:08] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 21:57:08] ...trap/engines/engine.py.326: process: 0.28388 | load: 0.00016 [I 2022-10-26 21:57:11] ...trap/engines/engine.py.321: eval: epoch 31 | batch 350/510 [I 2022-10-26 21:57:11] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-26 21:57:11] ...trap/engines/engine.py.326: process: 0.28412 | load: 0.00014 [I 2022-10-26 21:57:13] ...trap/engines/engine.py.321: eval: epoch 31 | batch 360/510 [I 2022-10-26 21:57:13] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-26 21:57:13] ...trap/engines/engine.py.326: process: 0.28696 | load: 0.00015 [I 2022-10-26 21:57:16] ...trap/engines/engine.py.321: eval: epoch 31 | batch 370/510 [I 2022-10-26 21:57:16] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 21:57:16] ...trap/engines/engine.py.326: process: 0.28514 | load: 0.00021 [I 2022-10-26 21:57:19] ...trap/engines/engine.py.321: eval: epoch 31 | batch 380/510 [I 2022-10-26 21:57:19] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 21:57:19] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00022 [I 2022-10-26 21:57:22] ...trap/engines/engine.py.321: eval: epoch 31 | batch 390/510 [I 2022-10-26 21:57:22] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 21:57:22] ...trap/engines/engine.py.326: process: 0.28824 | load: 0.00018 [I 2022-10-26 21:57:25] ...trap/engines/engine.py.321: eval: epoch 31 | batch 400/510 [I 2022-10-26 21:57:25] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 21:57:25] ...trap/engines/engine.py.326: process: 0.28230 | load: 0.00017 [I 2022-10-26 21:57:28] ...trap/engines/engine.py.321: eval: epoch 31 | batch 410/510 [I 2022-10-26 21:57:28] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 21:57:28] ...trap/engines/engine.py.326: process: 0.29187 | load: 0.00017 [I 2022-10-26 21:57:31] ...trap/engines/engine.py.321: eval: epoch 31 | batch 420/510 [I 2022-10-26 21:57:31] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:26 [I 2022-10-26 21:57:31] ...trap/engines/engine.py.326: process: 0.29113 | load: 0.00016 [I 2022-10-26 21:57:34] ...trap/engines/engine.py.321: eval: epoch 31 | batch 430/510 [I 2022-10-26 21:57:34] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 21:57:34] ...trap/engines/engine.py.326: process: 0.29008 | load: 0.00027 [I 2022-10-26 21:57:36] ...trap/engines/engine.py.321: eval: epoch 31 | batch 440/510 [I 2022-10-26 21:57:36] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 21:57:36] ...trap/engines/engine.py.326: process: 0.28706 | load: 0.00017 [I 2022-10-26 21:57:39] ...trap/engines/engine.py.321: eval: epoch 31 | batch 450/510 [I 2022-10-26 21:57:39] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 21:57:39] ...trap/engines/engine.py.326: process: 0.28949 | load: 0.00018 [I 2022-10-26 21:57:42] ...trap/engines/engine.py.321: eval: epoch 31 | batch 460/510 [I 2022-10-26 21:57:42] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 21:57:42] ...trap/engines/engine.py.326: process: 0.29876 | load: 0.00017 [I 2022-10-26 21:57:45] ...trap/engines/engine.py.321: eval: epoch 31 | batch 470/510 [I 2022-10-26 21:57:45] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 21:57:45] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00019 [I 2022-10-26 21:57:48] ...trap/engines/engine.py.321: eval: epoch 31 | batch 480/510 [I 2022-10-26 21:57:48] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 21:57:48] ...trap/engines/engine.py.326: process: 0.28719 | load: 0.00024 [I 2022-10-26 21:57:51] ...trap/engines/engine.py.321: eval: epoch 31 | batch 490/510 [I 2022-10-26 21:57:51] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 21:57:51] ...trap/engines/engine.py.326: process: 0.29136 | load: 0.00016 [I 2022-10-26 21:57:54] ...trap/engines/engine.py.321: eval: epoch 31 | batch 500/510 [I 2022-10-26 21:57:54] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 21:57:54] ...trap/engines/engine.py.326: process: 0.28817 | load: 0.00016 [I 2022-10-26 21:57:57] ...trap/engines/engine.py.321: eval: epoch 31 | batch 510/510 [I 2022-10-26 21:57:57] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 21:57:57] ...trap/engines/engine.py.326: process: 0.29590 | load: 0.00010 [S 2022-10-26 21:57:57] ...trap/engines/engine.py.132: eval_epoch.epoch: 31 [I 2022-10-26 21:57:57] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.660500 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.884900 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.930100 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.648100 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.878800 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.922400 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010847 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007648 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009924 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010578 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010796 [S 2022-10-26 21:58:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008834 [I 2022-10-26 21:58:03] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 21:58:03] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 21:58:03] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 21:58:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 21:58:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 21:58:04] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 21:58:07] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 21:58:12] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 21:58:12] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 21:58:12] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 21:58:15] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 21:58:22] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 21:58:22] ...trap/engines/engine.py.163: Training model on trainset for epoch 32 [I 2022-10-26 21:58:22] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 21:58:24] ...trap/engines/engine.py.224: train: epoch 32 | batch 0/2304 [I 2022-10-26 21:58:24] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:51 [I 2022-10-26 21:58:24] ...trap/engines/engine.py.229: process: 2.70476 | load: 2.07758 [I 2022-10-26 21:58:24] ...trap/engines/engine.py.230: loss: 0.17012 [I 2022-10-26 21:58:24] ...optimizers/trijoint.py.151: total_norm: 12.777413 [I 2022-10-26 21:58:30] ...trap/engines/engine.py.224: train: epoch 32 | batch 10/2304 [I 2022-10-26 21:58:30] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:05 [I 2022-10-26 21:58:30] ...trap/engines/engine.py.229: process: 0.55229 | load: 0.00503 [I 2022-10-26 21:58:30] ...trap/engines/engine.py.230: loss: 0.18952 [I 2022-10-26 21:58:30] ...optimizers/trijoint.py.151: total_norm: 17.355246 [I 2022-10-26 21:58:35] ...trap/engines/engine.py.224: train: epoch 32 | batch 20/2304 [I 2022-10-26 21:58:35] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:05 [I 2022-10-26 21:58:35] ...trap/engines/engine.py.229: process: 0.55098 | load: 0.00350 [I 2022-10-26 21:58:35] ...trap/engines/engine.py.230: loss: 0.14562 [I 2022-10-26 21:58:35] ...optimizers/trijoint.py.151: total_norm: 23.687653 [I 2022-10-26 21:58:41] ...trap/engines/engine.py.224: train: epoch 32 | batch 30/2304 [I 2022-10-26 21:58:41] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:12 [I 2022-10-26 21:58:41] ...trap/engines/engine.py.229: process: 0.54769 | load: 0.00357 [I 2022-10-26 21:58:41] ...trap/engines/engine.py.230: loss: 0.18383 [I 2022-10-26 21:58:41] ...optimizers/trijoint.py.151: total_norm: 16.139786 [I 2022-10-26 21:58:47] ...trap/engines/engine.py.224: train: epoch 32 | batch 40/2304 [I 2022-10-26 21:58:47] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:58 [I 2022-10-26 21:58:47] ...trap/engines/engine.py.229: process: 0.54551 | load: 0.00303 [I 2022-10-26 21:58:47] ...trap/engines/engine.py.230: loss: 0.18401 [I 2022-10-26 21:58:47] ...optimizers/trijoint.py.151: total_norm: 16.645569 [I 2022-10-26 21:58:52] ...trap/engines/engine.py.224: train: epoch 32 | batch 50/2304 [I 2022-10-26 21:58:52] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:04 [I 2022-10-26 21:58:52] ...trap/engines/engine.py.229: process: 0.56901 | load: 0.00309 [I 2022-10-26 21:58:52] ...trap/engines/engine.py.230: loss: 0.13804 [I 2022-10-26 21:58:52] ...optimizers/trijoint.py.151: total_norm: 18.613430 [I 2022-10-26 21:58:58] ...trap/engines/engine.py.224: train: epoch 32 | batch 60/2304 [I 2022-10-26 21:58:58] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:52 [I 2022-10-26 21:58:58] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00285 [I 2022-10-26 21:58:58] ...trap/engines/engine.py.230: loss: 0.15843 [I 2022-10-26 21:58:58] ...optimizers/trijoint.py.151: total_norm: 15.577988 [I 2022-10-26 21:59:03] ...trap/engines/engine.py.224: train: epoch 32 | batch 70/2304 [I 2022-10-26 21:59:03] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:48 [I 2022-10-26 21:59:03] ...trap/engines/engine.py.229: process: 0.54722 | load: 0.00288 [I 2022-10-26 21:59:03] ...trap/engines/engine.py.230: loss: 0.20552 [I 2022-10-26 21:59:03] ...optimizers/trijoint.py.151: total_norm: 21.556589 [I 2022-10-26 21:59:09] ...trap/engines/engine.py.224: train: epoch 32 | batch 80/2304 [I 2022-10-26 21:59:09] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:36 [I 2022-10-26 21:59:09] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00291 [I 2022-10-26 21:59:09] ...trap/engines/engine.py.230: loss: 0.16812 [I 2022-10-26 21:59:09] ...optimizers/trijoint.py.151: total_norm: 14.740024 [I 2022-10-26 21:59:14] ...trap/engines/engine.py.224: train: epoch 32 | batch 90/2304 [I 2022-10-26 21:59:14] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:20 [I 2022-10-26 21:59:14] ...trap/engines/engine.py.229: process: 0.54743 | load: 0.00253 [I 2022-10-26 21:59:14] ...trap/engines/engine.py.230: loss: 0.17756 [I 2022-10-26 21:59:14] ...optimizers/trijoint.py.151: total_norm: 22.492552 [I 2022-10-26 21:59:20] ...trap/engines/engine.py.224: train: epoch 32 | batch 100/2304 [I 2022-10-26 21:59:20] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:22 [I 2022-10-26 21:59:20] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00260 [I 2022-10-26 21:59:20] ...trap/engines/engine.py.230: loss: 0.21854 [I 2022-10-26 21:59:20] ...optimizers/trijoint.py.151: total_norm: 23.279551 [I 2022-10-26 21:59:26] ...trap/engines/engine.py.224: train: epoch 32 | batch 110/2304 [I 2022-10-26 21:59:26] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:22 [I 2022-10-26 21:59:26] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00272 [I 2022-10-26 21:59:26] ...trap/engines/engine.py.230: loss: 0.15068 [I 2022-10-26 21:59:26] ...optimizers/trijoint.py.151: total_norm: 16.181982 [I 2022-10-26 21:59:31] ...trap/engines/engine.py.224: train: epoch 32 | batch 120/2304 [I 2022-10-26 21:59:31] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:10 [I 2022-10-26 21:59:31] ...trap/engines/engine.py.229: process: 0.55431 | load: 0.00264 [I 2022-10-26 21:59:31] ...trap/engines/engine.py.230: loss: 0.13984 [I 2022-10-26 21:59:31] ...optimizers/trijoint.py.151: total_norm: 11.724177 [I 2022-10-26 21:59:37] ...trap/engines/engine.py.224: train: epoch 32 | batch 130/2304 [I 2022-10-26 21:59:37] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:05 [I 2022-10-26 21:59:37] ...trap/engines/engine.py.229: process: 0.56279 | load: 0.00261 [I 2022-10-26 21:59:37] ...trap/engines/engine.py.230: loss: 0.22856 [I 2022-10-26 21:59:37] ...optimizers/trijoint.py.151: total_norm: 28.894903 [I 2022-10-26 21:59:42] ...trap/engines/engine.py.224: train: epoch 32 | batch 140/2304 [I 2022-10-26 21:59:42] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:54 [I 2022-10-26 21:59:42] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00244 [I 2022-10-26 21:59:42] ...trap/engines/engine.py.230: loss: 0.20354 [I 2022-10-26 21:59:42] ...optimizers/trijoint.py.151: total_norm: 10.054131 [I 2022-10-26 21:59:48] ...trap/engines/engine.py.224: train: epoch 32 | batch 150/2304 [I 2022-10-26 21:59:48] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:00 [I 2022-10-26 21:59:48] ...trap/engines/engine.py.229: process: 0.54474 | load: 0.00250 [I 2022-10-26 21:59:48] ...trap/engines/engine.py.230: loss: 0.15811 [I 2022-10-26 21:59:48] ...optimizers/trijoint.py.151: total_norm: 17.899311 [I 2022-10-26 21:59:53] ...trap/engines/engine.py.224: train: epoch 32 | batch 160/2304 [I 2022-10-26 21:59:53] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:20:02 [I 2022-10-26 21:59:53] ...trap/engines/engine.py.229: process: 0.58296 | load: 0.00248 [I 2022-10-26 21:59:53] ...trap/engines/engine.py.230: loss: 0.14724 [I 2022-10-26 21:59:53] ...optimizers/trijoint.py.151: total_norm: 14.762141 [I 2022-10-26 21:59:59] ...trap/engines/engine.py.224: train: epoch 32 | batch 170/2304 [I 2022-10-26 21:59:59] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:20:14 [I 2022-10-26 21:59:59] ...trap/engines/engine.py.229: process: 0.56705 | load: 0.00258 [I 2022-10-26 21:59:59] ...trap/engines/engine.py.230: loss: 0.18509 [I 2022-10-26 21:59:59] ...optimizers/trijoint.py.151: total_norm: 15.417142 [I 2022-10-26 22:00:05] ...trap/engines/engine.py.224: train: epoch 32 | batch 180/2304 [I 2022-10-26 22:00:05] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:42 [I 2022-10-26 22:00:05] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00260 [I 2022-10-26 22:00:05] ...trap/engines/engine.py.230: loss: 0.20193 [I 2022-10-26 22:00:05] ...optimizers/trijoint.py.151: total_norm: 31.159801 [I 2022-10-26 22:00:10] ...trap/engines/engine.py.224: train: epoch 32 | batch 190/2304 [I 2022-10-26 22:00:10] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:23 [I 2022-10-26 22:00:10] ...trap/engines/engine.py.229: process: 0.54235 | load: 0.00246 [I 2022-10-26 22:00:10] ...trap/engines/engine.py.230: loss: 0.19041 [I 2022-10-26 22:00:10] ...optimizers/trijoint.py.151: total_norm: 20.366283 [I 2022-10-26 22:00:16] ...trap/engines/engine.py.224: train: epoch 32 | batch 200/2304 [I 2022-10-26 22:00:16] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:30 [I 2022-10-26 22:00:16] ...trap/engines/engine.py.229: process: 0.55350 | load: 0.00244 [I 2022-10-26 22:00:16] ...trap/engines/engine.py.230: loss: 0.20215 [I 2022-10-26 22:00:16] ...optimizers/trijoint.py.151: total_norm: 22.920719 [I 2022-10-26 22:00:21] ...trap/engines/engine.py.224: train: epoch 32 | batch 210/2304 [I 2022-10-26 22:00:21] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:19 [I 2022-10-26 22:00:21] ...trap/engines/engine.py.229: process: 0.54401 | load: 0.00285 [I 2022-10-26 22:00:21] ...trap/engines/engine.py.230: loss: 0.19494 [I 2022-10-26 22:00:21] ...optimizers/trijoint.py.151: total_norm: 14.252053 [I 2022-10-26 22:00:27] ...trap/engines/engine.py.224: train: epoch 32 | batch 220/2304 [I 2022-10-26 22:00:27] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:15 [I 2022-10-26 22:00:27] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00258 [I 2022-10-26 22:00:27] ...trap/engines/engine.py.230: loss: 0.18334 [I 2022-10-26 22:00:27] ...optimizers/trijoint.py.151: total_norm: 7.813793 [I 2022-10-26 22:00:33] ...trap/engines/engine.py.224: train: epoch 32 | batch 230/2304 [I 2022-10-26 22:00:33] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:34 [I 2022-10-26 22:00:33] ...trap/engines/engine.py.229: process: 0.54384 | load: 0.00262 [I 2022-10-26 22:00:33] ...trap/engines/engine.py.230: loss: 0.18563 [I 2022-10-26 22:00:33] ...optimizers/trijoint.py.151: total_norm: 19.506729 [I 2022-10-26 22:00:38] ...trap/engines/engine.py.224: train: epoch 32 | batch 240/2304 [I 2022-10-26 22:00:38] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:01 [I 2022-10-26 22:00:38] ...trap/engines/engine.py.229: process: 0.55075 | load: 0.00245 [I 2022-10-26 22:00:38] ...trap/engines/engine.py.230: loss: 0.20236 [I 2022-10-26 22:00:38] ...optimizers/trijoint.py.151: total_norm: 17.893936 [I 2022-10-26 22:00:44] ...trap/engines/engine.py.224: train: epoch 32 | batch 250/2304 [I 2022-10-26 22:00:44] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:05 [I 2022-10-26 22:00:44] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00259 [I 2022-10-26 22:00:44] ...trap/engines/engine.py.230: loss: 0.20562 [I 2022-10-26 22:00:44] ...optimizers/trijoint.py.151: total_norm: 23.677095 [I 2022-10-26 22:00:49] ...trap/engines/engine.py.224: train: epoch 32 | batch 260/2304 [I 2022-10-26 22:00:49] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:02 [I 2022-10-26 22:00:49] ...trap/engines/engine.py.229: process: 0.56328 | load: 0.00248 [I 2022-10-26 22:00:49] ...trap/engines/engine.py.230: loss: 0.21427 [I 2022-10-26 22:00:49] ...optimizers/trijoint.py.151: total_norm: 29.304239 [I 2022-10-26 22:00:55] ...trap/engines/engine.py.224: train: epoch 32 | batch 270/2304 [I 2022-10-26 22:00:55] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:50 [I 2022-10-26 22:00:55] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00248 [I 2022-10-26 22:00:55] ...trap/engines/engine.py.230: loss: 0.14340 [I 2022-10-26 22:00:55] ...optimizers/trijoint.py.151: total_norm: 15.429770 [I 2022-10-26 22:01:01] ...trap/engines/engine.py.224: train: epoch 32 | batch 280/2304 [I 2022-10-26 22:01:01] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:45 [I 2022-10-26 22:01:01] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00255 [I 2022-10-26 22:01:01] ...trap/engines/engine.py.230: loss: 0.16624 [I 2022-10-26 22:01:01] ...optimizers/trijoint.py.151: total_norm: 19.282715 [I 2022-10-26 22:01:06] ...trap/engines/engine.py.224: train: epoch 32 | batch 290/2304 [I 2022-10-26 22:01:06] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:50 [I 2022-10-26 22:01:06] ...trap/engines/engine.py.229: process: 0.56568 | load: 0.00260 [I 2022-10-26 22:01:06] ...trap/engines/engine.py.230: loss: 0.19331 [I 2022-10-26 22:01:06] ...optimizers/trijoint.py.151: total_norm: 23.115385 [I 2022-10-26 22:01:12] ...trap/engines/engine.py.224: train: epoch 32 | batch 300/2304 [I 2022-10-26 22:01:12] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:43 [I 2022-10-26 22:01:12] ...trap/engines/engine.py.229: process: 0.56309 | load: 0.00244 [I 2022-10-26 22:01:12] ...trap/engines/engine.py.230: loss: 0.16396 [I 2022-10-26 22:01:12] ...optimizers/trijoint.py.151: total_norm: 15.862678 [I 2022-10-26 22:01:17] ...trap/engines/engine.py.224: train: epoch 32 | batch 310/2304 [I 2022-10-26 22:01:17] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:32 [I 2022-10-26 22:01:17] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00242 [I 2022-10-26 22:01:17] ...trap/engines/engine.py.230: loss: 0.13267 [I 2022-10-26 22:01:17] ...optimizers/trijoint.py.151: total_norm: 16.419992 [I 2022-10-26 22:01:23] ...trap/engines/engine.py.224: train: epoch 32 | batch 320/2304 [I 2022-10-26 22:01:23] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:29 [I 2022-10-26 22:01:23] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00245 [I 2022-10-26 22:01:23] ...trap/engines/engine.py.230: loss: 0.15805 [I 2022-10-26 22:01:23] ...optimizers/trijoint.py.151: total_norm: 18.846222 [I 2022-10-26 22:01:29] ...trap/engines/engine.py.224: train: epoch 32 | batch 330/2304 [I 2022-10-26 22:01:29] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:25 [I 2022-10-26 22:01:29] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00246 [I 2022-10-26 22:01:29] ...trap/engines/engine.py.230: loss: 0.16118 [I 2022-10-26 22:01:29] ...optimizers/trijoint.py.151: total_norm: 14.755707 [I 2022-10-26 22:01:34] ...trap/engines/engine.py.224: train: epoch 32 | batch 340/2304 [I 2022-10-26 22:01:34] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:17 [I 2022-10-26 22:01:34] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00246 [I 2022-10-26 22:01:34] ...trap/engines/engine.py.230: loss: 0.25306 [I 2022-10-26 22:01:34] ...optimizers/trijoint.py.151: total_norm: 24.656458 [I 2022-10-26 22:01:40] ...trap/engines/engine.py.224: train: epoch 32 | batch 350/2304 [I 2022-10-26 22:01:40] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:14 [I 2022-10-26 22:01:40] ...trap/engines/engine.py.229: process: 0.54822 | load: 0.00241 [I 2022-10-26 22:01:40] ...trap/engines/engine.py.230: loss: 0.19761 [I 2022-10-26 22:01:40] ...optimizers/trijoint.py.151: total_norm: 14.343068 [I 2022-10-26 22:01:45] ...trap/engines/engine.py.224: train: epoch 32 | batch 360/2304 [I 2022-10-26 22:01:45] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:05 [I 2022-10-26 22:01:45] ...trap/engines/engine.py.229: process: 0.54973 | load: 0.00249 [I 2022-10-26 22:01:45] ...trap/engines/engine.py.230: loss: 0.14045 [I 2022-10-26 22:01:45] ...optimizers/trijoint.py.151: total_norm: 18.062662 [I 2022-10-26 22:01:51] ...trap/engines/engine.py.224: train: epoch 32 | batch 370/2304 [I 2022-10-26 22:01:51] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:50 [I 2022-10-26 22:01:51] ...trap/engines/engine.py.229: process: 0.55162 | load: 0.00248 [I 2022-10-26 22:01:51] ...trap/engines/engine.py.230: loss: 0.19609 [I 2022-10-26 22:01:51] ...optimizers/trijoint.py.151: total_norm: 16.997053 [I 2022-10-26 22:01:57] ...trap/engines/engine.py.224: train: epoch 32 | batch 380/2304 [I 2022-10-26 22:01:57] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:48 [I 2022-10-26 22:01:57] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00243 [I 2022-10-26 22:01:57] ...trap/engines/engine.py.230: loss: 0.13826 [I 2022-10-26 22:01:57] ...optimizers/trijoint.py.151: total_norm: 19.073484 [I 2022-10-26 22:02:02] ...trap/engines/engine.py.224: train: epoch 32 | batch 390/2304 [I 2022-10-26 22:02:02] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:51 [I 2022-10-26 22:02:02] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00244 [I 2022-10-26 22:02:02] ...trap/engines/engine.py.230: loss: 0.24328 [I 2022-10-26 22:02:02] ...optimizers/trijoint.py.151: total_norm: 22.861010 [I 2022-10-26 22:02:08] ...trap/engines/engine.py.224: train: epoch 32 | batch 400/2304 [I 2022-10-26 22:02:08] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:50 [I 2022-10-26 22:02:08] ...trap/engines/engine.py.229: process: 0.58130 | load: 0.00241 [I 2022-10-26 22:02:08] ...trap/engines/engine.py.230: loss: 0.11968 [I 2022-10-26 22:02:08] ...optimizers/trijoint.py.151: total_norm: 15.551667 [I 2022-10-26 22:02:13] ...trap/engines/engine.py.224: train: epoch 32 | batch 410/2304 [I 2022-10-26 22:02:13] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:49 [I 2022-10-26 22:02:13] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00258 [I 2022-10-26 22:02:13] ...trap/engines/engine.py.230: loss: 0.24851 [I 2022-10-26 22:02:13] ...optimizers/trijoint.py.151: total_norm: 25.734245 [I 2022-10-26 22:02:19] ...trap/engines/engine.py.224: train: epoch 32 | batch 420/2304 [I 2022-10-26 22:02:19] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:31 [I 2022-10-26 22:02:19] ...trap/engines/engine.py.229: process: 0.56223 | load: 0.00243 [I 2022-10-26 22:02:19] ...trap/engines/engine.py.230: loss: 0.17585 [I 2022-10-26 22:02:19] ...optimizers/trijoint.py.151: total_norm: 17.791275 [I 2022-10-26 22:02:25] ...trap/engines/engine.py.224: train: epoch 32 | batch 430/2304 [I 2022-10-26 22:02:25] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:22 [I 2022-10-26 22:02:25] ...trap/engines/engine.py.229: process: 0.55205 | load: 0.00248 [I 2022-10-26 22:02:25] ...trap/engines/engine.py.230: loss: 0.19616 [I 2022-10-26 22:02:25] ...optimizers/trijoint.py.151: total_norm: 15.825790 [I 2022-10-26 22:02:30] ...trap/engines/engine.py.224: train: epoch 32 | batch 440/2304 [I 2022-10-26 22:02:30] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:09 [I 2022-10-26 22:02:30] ...trap/engines/engine.py.229: process: 0.54698 | load: 0.00242 [I 2022-10-26 22:02:30] ...trap/engines/engine.py.230: loss: 0.25562 [I 2022-10-26 22:02:30] ...optimizers/trijoint.py.151: total_norm: 20.528116 [I 2022-10-26 22:02:36] ...trap/engines/engine.py.224: train: epoch 32 | batch 450/2304 [I 2022-10-26 22:02:36] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:18 [I 2022-10-26 22:02:36] ...trap/engines/engine.py.229: process: 0.56166 | load: 0.00245 [I 2022-10-26 22:02:36] ...trap/engines/engine.py.230: loss: 0.18756 [I 2022-10-26 22:02:36] ...optimizers/trijoint.py.151: total_norm: 20.018135 [I 2022-10-26 22:02:41] ...trap/engines/engine.py.224: train: epoch 32 | batch 460/2304 [I 2022-10-26 22:02:41] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:03 [I 2022-10-26 22:02:41] ...trap/engines/engine.py.229: process: 0.54346 | load: 0.00241 [I 2022-10-26 22:02:41] ...trap/engines/engine.py.230: loss: 0.12610 [I 2022-10-26 22:02:41] ...optimizers/trijoint.py.151: total_norm: 11.313458 [I 2022-10-26 22:02:47] ...trap/engines/engine.py.224: train: epoch 32 | batch 470/2304 [I 2022-10-26 22:02:47] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:55 [I 2022-10-26 22:02:47] ...trap/engines/engine.py.229: process: 0.55485 | load: 0.00255 [I 2022-10-26 22:02:47] ...trap/engines/engine.py.230: loss: 0.16211 [I 2022-10-26 22:02:47] ...optimizers/trijoint.py.151: total_norm: 13.689604 [I 2022-10-26 22:02:52] ...trap/engines/engine.py.224: train: epoch 32 | batch 480/2304 [I 2022-10-26 22:02:52] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:57 [I 2022-10-26 22:02:52] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00249 [I 2022-10-26 22:02:52] ...trap/engines/engine.py.230: loss: 0.17614 [I 2022-10-26 22:02:52] ...optimizers/trijoint.py.151: total_norm: 15.869502 [I 2022-10-26 22:02:58] ...trap/engines/engine.py.224: train: epoch 32 | batch 490/2304 [I 2022-10-26 22:02:58] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:46 [I 2022-10-26 22:02:58] ...trap/engines/engine.py.229: process: 0.55319 | load: 0.00247 [I 2022-10-26 22:02:58] ...trap/engines/engine.py.230: loss: 0.10553 [I 2022-10-26 22:02:58] ...optimizers/trijoint.py.151: total_norm: 11.416778 [I 2022-10-26 22:03:04] ...trap/engines/engine.py.224: train: epoch 32 | batch 500/2304 [I 2022-10-26 22:03:04] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:41 [I 2022-10-26 22:03:04] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00245 [I 2022-10-26 22:03:04] ...trap/engines/engine.py.230: loss: 0.15927 [I 2022-10-26 22:03:04] ...optimizers/trijoint.py.151: total_norm: 13.055731 [I 2022-10-26 22:03:09] ...trap/engines/engine.py.224: train: epoch 32 | batch 510/2304 [I 2022-10-26 22:03:09] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:38 [I 2022-10-26 22:03:09] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00243 [I 2022-10-26 22:03:09] ...trap/engines/engine.py.230: loss: 0.20007 [I 2022-10-26 22:03:09] ...optimizers/trijoint.py.151: total_norm: 21.817497 [I 2022-10-26 22:03:15] ...trap/engines/engine.py.224: train: epoch 32 | batch 520/2304 [I 2022-10-26 22:03:15] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:38 [I 2022-10-26 22:03:15] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00244 [I 2022-10-26 22:03:15] ...trap/engines/engine.py.230: loss: 0.21619 [I 2022-10-26 22:03:15] ...optimizers/trijoint.py.151: total_norm: 20.587187 [I 2022-10-26 22:03:20] ...trap/engines/engine.py.224: train: epoch 32 | batch 530/2304 [I 2022-10-26 22:03:20] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:28 [I 2022-10-26 22:03:20] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00248 [I 2022-10-26 22:03:20] ...trap/engines/engine.py.230: loss: 0.16701 [I 2022-10-26 22:03:20] ...optimizers/trijoint.py.151: total_norm: 12.928335 [I 2022-10-26 22:03:26] ...trap/engines/engine.py.224: train: epoch 32 | batch 540/2304 [I 2022-10-26 22:03:26] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:18 [I 2022-10-26 22:03:26] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00244 [I 2022-10-26 22:03:26] ...trap/engines/engine.py.230: loss: 0.21263 [I 2022-10-26 22:03:26] ...optimizers/trijoint.py.151: total_norm: 17.180349 [I 2022-10-26 22:03:31] ...trap/engines/engine.py.224: train: epoch 32 | batch 550/2304 [I 2022-10-26 22:03:31] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:07 [I 2022-10-26 22:03:31] ...trap/engines/engine.py.229: process: 0.55041 | load: 0.00251 [I 2022-10-26 22:03:31] ...trap/engines/engine.py.230: loss: 0.13602 [I 2022-10-26 22:03:31] ...optimizers/trijoint.py.151: total_norm: 13.645424 [I 2022-10-26 22:03:37] ...trap/engines/engine.py.224: train: epoch 32 | batch 560/2304 [I 2022-10-26 22:03:37] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:09 [I 2022-10-26 22:03:37] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00242 [I 2022-10-26 22:03:37] ...trap/engines/engine.py.230: loss: 0.15945 [I 2022-10-26 22:03:37] ...optimizers/trijoint.py.151: total_norm: 16.875137 [I 2022-10-26 22:03:43] ...trap/engines/engine.py.224: train: epoch 32 | batch 570/2304 [I 2022-10-26 22:03:43] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:15:58 [I 2022-10-26 22:03:43] ...trap/engines/engine.py.229: process: 0.55662 | load: 0.00242 [I 2022-10-26 22:03:43] ...trap/engines/engine.py.230: loss: 0.09474 [I 2022-10-26 22:03:43] ...optimizers/trijoint.py.151: total_norm: 18.973358 [I 2022-10-26 22:03:48] ...trap/engines/engine.py.224: train: epoch 32 | batch 580/2304 [I 2022-10-26 22:03:48] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:04 [I 2022-10-26 22:03:48] ...trap/engines/engine.py.229: process: 0.56666 | load: 0.00238 [I 2022-10-26 22:03:48] ...trap/engines/engine.py.230: loss: 0.17592 [I 2022-10-26 22:03:48] ...optimizers/trijoint.py.151: total_norm: 24.099133 [I 2022-10-26 22:03:54] ...trap/engines/engine.py.224: train: epoch 32 | batch 590/2304 [I 2022-10-26 22:03:54] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:47 [I 2022-10-26 22:03:54] ...trap/engines/engine.py.229: process: 0.54138 | load: 0.00243 [I 2022-10-26 22:03:54] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-26 22:03:54] ...optimizers/trijoint.py.151: total_norm: 13.480073 [I 2022-10-26 22:03:59] ...trap/engines/engine.py.224: train: epoch 32 | batch 600/2304 [I 2022-10-26 22:03:59] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:41 [I 2022-10-26 22:03:59] ...trap/engines/engine.py.229: process: 0.54759 | load: 0.00251 [I 2022-10-26 22:03:59] ...trap/engines/engine.py.230: loss: 0.19930 [I 2022-10-26 22:03:59] ...optimizers/trijoint.py.151: total_norm: 22.616268 [I 2022-10-26 22:04:05] ...trap/engines/engine.py.224: train: epoch 32 | batch 610/2304 [I 2022-10-26 22:04:05] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:37 [I 2022-10-26 22:04:05] ...trap/engines/engine.py.229: process: 0.54479 | load: 0.00257 [I 2022-10-26 22:04:05] ...trap/engines/engine.py.230: loss: 0.13798 [I 2022-10-26 22:04:05] ...optimizers/trijoint.py.151: total_norm: 13.417066 [I 2022-10-26 22:04:10] ...trap/engines/engine.py.224: train: epoch 32 | batch 620/2304 [I 2022-10-26 22:04:10] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:36 [I 2022-10-26 22:04:10] ...trap/engines/engine.py.229: process: 0.56724 | load: 0.00245 [I 2022-10-26 22:04:10] ...trap/engines/engine.py.230: loss: 0.19664 [I 2022-10-26 22:04:10] ...optimizers/trijoint.py.151: total_norm: 23.936317 [I 2022-10-26 22:04:16] ...trap/engines/engine.py.224: train: epoch 32 | batch 630/2304 [I 2022-10-26 22:04:16] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:30 [I 2022-10-26 22:04:16] ...trap/engines/engine.py.229: process: 0.54995 | load: 0.00245 [I 2022-10-26 22:04:16] ...trap/engines/engine.py.230: loss: 0.17490 [I 2022-10-26 22:04:16] ...optimizers/trijoint.py.151: total_norm: 31.646494 [I 2022-10-26 22:04:21] ...trap/engines/engine.py.224: train: epoch 32 | batch 640/2304 [I 2022-10-26 22:04:21] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:32 [I 2022-10-26 22:04:21] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00243 [I 2022-10-26 22:04:21] ...trap/engines/engine.py.230: loss: 0.13411 [I 2022-10-26 22:04:21] ...optimizers/trijoint.py.151: total_norm: 10.649995 [I 2022-10-26 22:04:27] ...trap/engines/engine.py.224: train: epoch 32 | batch 650/2304 [I 2022-10-26 22:04:27] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:25 [I 2022-10-26 22:04:27] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00250 [I 2022-10-26 22:04:27] ...trap/engines/engine.py.230: loss: 0.12660 [I 2022-10-26 22:04:27] ...optimizers/trijoint.py.151: total_norm: 22.460316 [I 2022-10-26 22:04:33] ...trap/engines/engine.py.224: train: epoch 32 | batch 660/2304 [I 2022-10-26 22:04:33] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:16 [I 2022-10-26 22:04:33] ...trap/engines/engine.py.229: process: 0.55338 | load: 0.00269 [I 2022-10-26 22:04:33] ...trap/engines/engine.py.230: loss: 0.13416 [I 2022-10-26 22:04:33] ...optimizers/trijoint.py.151: total_norm: 11.560069 [I 2022-10-26 22:04:38] ...trap/engines/engine.py.224: train: epoch 32 | batch 670/2304 [I 2022-10-26 22:04:38] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:10 [I 2022-10-26 22:04:38] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00243 [I 2022-10-26 22:04:38] ...trap/engines/engine.py.230: loss: 0.15822 [I 2022-10-26 22:04:38] ...optimizers/trijoint.py.151: total_norm: 20.006395 [I 2022-10-26 22:04:44] ...trap/engines/engine.py.224: train: epoch 32 | batch 680/2304 [I 2022-10-26 22:04:44] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:02 [I 2022-10-26 22:04:44] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00242 [I 2022-10-26 22:04:44] ...trap/engines/engine.py.230: loss: 0.27173 [I 2022-10-26 22:04:44] ...optimizers/trijoint.py.151: total_norm: 27.747555 [I 2022-10-26 22:04:49] ...trap/engines/engine.py.224: train: epoch 32 | batch 690/2304 [I 2022-10-26 22:04:49] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:14:58 [I 2022-10-26 22:04:49] ...trap/engines/engine.py.229: process: 0.56415 | load: 0.00242 [I 2022-10-26 22:04:49] ...trap/engines/engine.py.230: loss: 0.15087 [I 2022-10-26 22:04:49] ...optimizers/trijoint.py.151: total_norm: 21.862984 [I 2022-10-26 22:04:55] ...trap/engines/engine.py.224: train: epoch 32 | batch 700/2304 [I 2022-10-26 22:04:55] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:51 [I 2022-10-26 22:04:55] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00242 [I 2022-10-26 22:04:55] ...trap/engines/engine.py.230: loss: 0.28201 [I 2022-10-26 22:04:55] ...optimizers/trijoint.py.151: total_norm: 20.293438 [I 2022-10-26 22:05:00] ...trap/engines/engine.py.224: train: epoch 32 | batch 710/2304 [I 2022-10-26 22:05:00] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:14:42 [I 2022-10-26 22:05:00] ...trap/engines/engine.py.229: process: 0.54637 | load: 0.00241 [I 2022-10-26 22:05:00] ...trap/engines/engine.py.230: loss: 0.14018 [I 2022-10-26 22:05:00] ...optimizers/trijoint.py.151: total_norm: 10.237939 [I 2022-10-26 22:05:06] ...trap/engines/engine.py.224: train: epoch 32 | batch 720/2304 [I 2022-10-26 22:05:06] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:43 [I 2022-10-26 22:05:06] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00241 [I 2022-10-26 22:05:06] ...trap/engines/engine.py.230: loss: 0.21469 [I 2022-10-26 22:05:06] ...optimizers/trijoint.py.151: total_norm: 21.453609 [I 2022-10-26 22:05:12] ...trap/engines/engine.py.224: train: epoch 32 | batch 730/2304 [I 2022-10-26 22:05:12] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:36 [I 2022-10-26 22:05:12] ...trap/engines/engine.py.229: process: 0.54718 | load: 0.00241 [I 2022-10-26 22:05:12] ...trap/engines/engine.py.230: loss: 0.15877 [I 2022-10-26 22:05:12] ...optimizers/trijoint.py.151: total_norm: 12.606649 [I 2022-10-26 22:05:17] ...trap/engines/engine.py.224: train: epoch 32 | batch 740/2304 [I 2022-10-26 22:05:17] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:31 [I 2022-10-26 22:05:17] ...trap/engines/engine.py.229: process: 0.56080 | load: 0.00240 [I 2022-10-26 22:05:17] ...trap/engines/engine.py.230: loss: 0.11050 [I 2022-10-26 22:05:17] ...optimizers/trijoint.py.151: total_norm: 8.658569 [I 2022-10-26 22:05:23] ...trap/engines/engine.py.224: train: epoch 32 | batch 750/2304 [I 2022-10-26 22:05:23] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:34 [I 2022-10-26 22:05:23] ...trap/engines/engine.py.229: process: 0.54978 | load: 0.00242 [I 2022-10-26 22:05:23] ...trap/engines/engine.py.230: loss: 0.27921 [I 2022-10-26 22:05:23] ...optimizers/trijoint.py.151: total_norm: 33.319813 [I 2022-10-26 22:05:28] ...trap/engines/engine.py.224: train: epoch 32 | batch 760/2304 [I 2022-10-26 22:05:28] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:20 [I 2022-10-26 22:05:28] ...trap/engines/engine.py.229: process: 0.55562 | load: 0.00249 [I 2022-10-26 22:05:28] ...trap/engines/engine.py.230: loss: 0.14945 [I 2022-10-26 22:05:28] ...optimizers/trijoint.py.151: total_norm: 16.384159 [I 2022-10-26 22:05:34] ...trap/engines/engine.py.224: train: epoch 32 | batch 770/2304 [I 2022-10-26 22:05:34] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:29 [I 2022-10-26 22:05:34] ...trap/engines/engine.py.229: process: 0.56071 | load: 0.00245 [I 2022-10-26 22:05:34] ...trap/engines/engine.py.230: loss: 0.23176 [I 2022-10-26 22:05:34] ...optimizers/trijoint.py.151: total_norm: 21.523333 [I 2022-10-26 22:05:40] ...trap/engines/engine.py.224: train: epoch 32 | batch 780/2304 [I 2022-10-26 22:05:40] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:09 [I 2022-10-26 22:05:40] ...trap/engines/engine.py.229: process: 0.55543 | load: 0.00247 [I 2022-10-26 22:05:40] ...trap/engines/engine.py.230: loss: 0.16580 [I 2022-10-26 22:05:40] ...optimizers/trijoint.py.151: total_norm: 20.428226 [I 2022-10-26 22:05:45] ...trap/engines/engine.py.224: train: epoch 32 | batch 790/2304 [I 2022-10-26 22:05:45] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:13:59 [I 2022-10-26 22:05:45] ...trap/engines/engine.py.229: process: 0.54678 | load: 0.00241 [I 2022-10-26 22:05:45] ...trap/engines/engine.py.230: loss: 0.22841 [I 2022-10-26 22:05:45] ...optimizers/trijoint.py.151: total_norm: 17.164242 [I 2022-10-26 22:05:51] ...trap/engines/engine.py.224: train: epoch 32 | batch 800/2304 [I 2022-10-26 22:05:51] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:44 [I 2022-10-26 22:05:51] ...trap/engines/engine.py.229: process: 0.54191 | load: 0.00241 [I 2022-10-26 22:05:51] ...trap/engines/engine.py.230: loss: 0.16482 [I 2022-10-26 22:05:51] ...optimizers/trijoint.py.151: total_norm: 12.277890 [I 2022-10-26 22:05:56] ...trap/engines/engine.py.224: train: epoch 32 | batch 810/2304 [I 2022-10-26 22:05:56] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:13:48 [I 2022-10-26 22:05:56] ...trap/engines/engine.py.229: process: 0.55209 | load: 0.00248 [I 2022-10-26 22:05:56] ...trap/engines/engine.py.230: loss: 0.14635 [I 2022-10-26 22:05:56] ...optimizers/trijoint.py.151: total_norm: 24.235388 [I 2022-10-26 22:06:02] ...trap/engines/engine.py.224: train: epoch 32 | batch 820/2304 [I 2022-10-26 22:06:02] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:44 [I 2022-10-26 22:06:02] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00240 [I 2022-10-26 22:06:02] ...trap/engines/engine.py.230: loss: 0.14935 [I 2022-10-26 22:06:02] ...optimizers/trijoint.py.151: total_norm: 15.075244 [I 2022-10-26 22:06:07] ...trap/engines/engine.py.224: train: epoch 32 | batch 830/2304 [I 2022-10-26 22:06:07] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:13:44 [I 2022-10-26 22:06:07] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00247 [I 2022-10-26 22:06:07] ...trap/engines/engine.py.230: loss: 0.16065 [I 2022-10-26 22:06:07] ...optimizers/trijoint.py.151: total_norm: 18.364368 [I 2022-10-26 22:06:13] ...trap/engines/engine.py.224: train: epoch 32 | batch 840/2304 [I 2022-10-26 22:06:13] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:44 [I 2022-10-26 22:06:13] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00259 [I 2022-10-26 22:06:13] ...trap/engines/engine.py.230: loss: 0.13139 [I 2022-10-26 22:06:13] ...optimizers/trijoint.py.151: total_norm: 15.278192 [I 2022-10-26 22:06:19] ...trap/engines/engine.py.224: train: epoch 32 | batch 850/2304 [I 2022-10-26 22:06:19] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:27 [I 2022-10-26 22:06:19] ...trap/engines/engine.py.229: process: 0.54820 | load: 0.00244 [I 2022-10-26 22:06:19] ...trap/engines/engine.py.230: loss: 0.11366 [I 2022-10-26 22:06:19] ...optimizers/trijoint.py.151: total_norm: 18.387384 [I 2022-10-26 22:06:24] ...trap/engines/engine.py.224: train: epoch 32 | batch 860/2304 [I 2022-10-26 22:06:24] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:25 [I 2022-10-26 22:06:24] ...trap/engines/engine.py.229: process: 0.56504 | load: 0.00244 [I 2022-10-26 22:06:24] ...trap/engines/engine.py.230: loss: 0.20496 [I 2022-10-26 22:06:24] ...optimizers/trijoint.py.151: total_norm: 23.311443 [I 2022-10-26 22:06:30] ...trap/engines/engine.py.224: train: epoch 32 | batch 870/2304 [I 2022-10-26 22:06:30] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:14 [I 2022-10-26 22:06:30] ...trap/engines/engine.py.229: process: 0.54728 | load: 0.00242 [I 2022-10-26 22:06:30] ...trap/engines/engine.py.230: loss: 0.21831 [I 2022-10-26 22:06:30] ...optimizers/trijoint.py.151: total_norm: 24.394848 [I 2022-10-26 22:06:35] ...trap/engines/engine.py.224: train: epoch 32 | batch 880/2304 [I 2022-10-26 22:06:35] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:07 [I 2022-10-26 22:06:35] ...trap/engines/engine.py.229: process: 0.54572 | load: 0.00239 [I 2022-10-26 22:06:35] ...trap/engines/engine.py.230: loss: 0.13703 [I 2022-10-26 22:06:35] ...optimizers/trijoint.py.151: total_norm: 15.116579 [I 2022-10-26 22:06:41] ...trap/engines/engine.py.224: train: epoch 32 | batch 890/2304 [I 2022-10-26 22:06:41] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:02 [I 2022-10-26 22:06:41] ...trap/engines/engine.py.229: process: 0.55842 | load: 0.00240 [I 2022-10-26 22:06:41] ...trap/engines/engine.py.230: loss: 0.18978 [I 2022-10-26 22:06:41] ...optimizers/trijoint.py.151: total_norm: 9.208443 [I 2022-10-26 22:06:47] ...trap/engines/engine.py.224: train: epoch 32 | batch 900/2304 [I 2022-10-26 22:06:47] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:09 [I 2022-10-26 22:06:47] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00249 [I 2022-10-26 22:06:47] ...trap/engines/engine.py.230: loss: 0.16789 [I 2022-10-26 22:06:47] ...optimizers/trijoint.py.151: total_norm: 14.208886 [I 2022-10-26 22:06:52] ...trap/engines/engine.py.224: train: epoch 32 | batch 910/2304 [I 2022-10-26 22:06:52] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:56 [I 2022-10-26 22:06:52] ...trap/engines/engine.py.229: process: 0.56268 | load: 0.00242 [I 2022-10-26 22:06:52] ...trap/engines/engine.py.230: loss: 0.18124 [I 2022-10-26 22:06:52] ...optimizers/trijoint.py.151: total_norm: 13.529238 [I 2022-10-26 22:06:58] ...trap/engines/engine.py.224: train: epoch 32 | batch 920/2304 [I 2022-10-26 22:06:58] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:50 [I 2022-10-26 22:06:58] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00242 [I 2022-10-26 22:06:58] ...trap/engines/engine.py.230: loss: 0.20163 [I 2022-10-26 22:06:58] ...optimizers/trijoint.py.151: total_norm: 19.153955 [I 2022-10-26 22:07:03] ...trap/engines/engine.py.224: train: epoch 32 | batch 930/2304 [I 2022-10-26 22:07:03] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:12:42 [I 2022-10-26 22:07:03] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00240 [I 2022-10-26 22:07:03] ...trap/engines/engine.py.230: loss: 0.16213 [I 2022-10-26 22:07:03] ...optimizers/trijoint.py.151: total_norm: 16.521755 [I 2022-10-26 22:07:09] ...trap/engines/engine.py.224: train: epoch 32 | batch 940/2304 [I 2022-10-26 22:07:09] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:39 [I 2022-10-26 22:07:09] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00239 [I 2022-10-26 22:07:09] ...trap/engines/engine.py.230: loss: 0.12621 [I 2022-10-26 22:07:09] ...optimizers/trijoint.py.151: total_norm: 9.743692 [I 2022-10-26 22:07:14] ...trap/engines/engine.py.224: train: epoch 32 | batch 950/2304 [I 2022-10-26 22:07:14] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:32 [I 2022-10-26 22:07:14] ...trap/engines/engine.py.229: process: 0.55178 | load: 0.00239 [I 2022-10-26 22:07:14] ...trap/engines/engine.py.230: loss: 0.21328 [I 2022-10-26 22:07:14] ...optimizers/trijoint.py.151: total_norm: 42.014553 [I 2022-10-26 22:07:20] ...trap/engines/engine.py.224: train: epoch 32 | batch 960/2304 [I 2022-10-26 22:07:20] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:27 [I 2022-10-26 22:07:20] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00239 [I 2022-10-26 22:07:20] ...trap/engines/engine.py.230: loss: 0.18011 [I 2022-10-26 22:07:20] ...optimizers/trijoint.py.151: total_norm: 9.131221 [I 2022-10-26 22:07:26] ...trap/engines/engine.py.224: train: epoch 32 | batch 970/2304 [I 2022-10-26 22:07:26] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:27 [I 2022-10-26 22:07:26] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00246 [I 2022-10-26 22:07:26] ...trap/engines/engine.py.230: loss: 0.17669 [I 2022-10-26 22:07:26] ...optimizers/trijoint.py.151: total_norm: 15.445203 [I 2022-10-26 22:07:31] ...trap/engines/engine.py.224: train: epoch 32 | batch 980/2304 [I 2022-10-26 22:07:31] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:13 [I 2022-10-26 22:07:31] ...trap/engines/engine.py.229: process: 0.55724 | load: 0.00246 [I 2022-10-26 22:07:31] ...trap/engines/engine.py.230: loss: 0.16966 [I 2022-10-26 22:07:31] ...optimizers/trijoint.py.151: total_norm: 14.443147 [I 2022-10-26 22:07:37] ...trap/engines/engine.py.224: train: epoch 32 | batch 990/2304 [I 2022-10-26 22:07:37] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:13 [I 2022-10-26 22:07:37] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00244 [I 2022-10-26 22:07:37] ...trap/engines/engine.py.230: loss: 0.18496 [I 2022-10-26 22:07:37] ...optimizers/trijoint.py.151: total_norm: 23.012611 [I 2022-10-26 22:07:42] ...trap/engines/engine.py.224: train: epoch 32 | batch 1000/2304 [I 2022-10-26 22:07:42] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:00 [I 2022-10-26 22:07:42] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00242 [I 2022-10-26 22:07:42] ...trap/engines/engine.py.230: loss: 0.15618 [I 2022-10-26 22:07:42] ...optimizers/trijoint.py.151: total_norm: 20.617962 [I 2022-10-26 22:07:48] ...trap/engines/engine.py.224: train: epoch 32 | batch 1010/2304 [I 2022-10-26 22:07:48] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:11:59 [I 2022-10-26 22:07:48] ...trap/engines/engine.py.229: process: 0.55154 | load: 0.00241 [I 2022-10-26 22:07:48] ...trap/engines/engine.py.230: loss: 0.13792 [I 2022-10-26 22:07:48] ...optimizers/trijoint.py.151: total_norm: 26.710114 [I 2022-10-26 22:07:54] ...trap/engines/engine.py.224: train: epoch 32 | batch 1020/2304 [I 2022-10-26 22:07:54] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:56 [I 2022-10-26 22:07:54] ...trap/engines/engine.py.229: process: 0.55695 | load: 0.00239 [I 2022-10-26 22:07:54] ...trap/engines/engine.py.230: loss: 0.15728 [I 2022-10-26 22:07:54] ...optimizers/trijoint.py.151: total_norm: 18.986904 [I 2022-10-26 22:07:59] ...trap/engines/engine.py.224: train: epoch 32 | batch 1030/2304 [I 2022-10-26 22:07:59] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:49 [I 2022-10-26 22:07:59] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00240 [I 2022-10-26 22:07:59] ...trap/engines/engine.py.230: loss: 0.10373 [I 2022-10-26 22:07:59] ...optimizers/trijoint.py.151: total_norm: 11.992645 [I 2022-10-26 22:08:05] ...trap/engines/engine.py.224: train: epoch 32 | batch 1040/2304 [I 2022-10-26 22:08:05] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:36 [I 2022-10-26 22:08:05] ...trap/engines/engine.py.229: process: 0.54576 | load: 0.00239 [I 2022-10-26 22:08:05] ...trap/engines/engine.py.230: loss: 0.20534 [I 2022-10-26 22:08:05] ...optimizers/trijoint.py.151: total_norm: 23.350241 [I 2022-10-26 22:08:10] ...trap/engines/engine.py.224: train: epoch 32 | batch 1050/2304 [I 2022-10-26 22:08:10] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:40 [I 2022-10-26 22:08:10] ...trap/engines/engine.py.229: process: 0.56297 | load: 0.00244 [I 2022-10-26 22:08:10] ...trap/engines/engine.py.230: loss: 0.18203 [I 2022-10-26 22:08:10] ...optimizers/trijoint.py.151: total_norm: 25.925144 [I 2022-10-26 22:08:16] ...trap/engines/engine.py.224: train: epoch 32 | batch 1060/2304 [I 2022-10-26 22:08:16] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:32 [I 2022-10-26 22:08:16] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00242 [I 2022-10-26 22:08:16] ...trap/engines/engine.py.230: loss: 0.12570 [I 2022-10-26 22:08:16] ...optimizers/trijoint.py.151: total_norm: 14.237062 [I 2022-10-26 22:08:21] ...trap/engines/engine.py.224: train: epoch 32 | batch 1070/2304 [I 2022-10-26 22:08:21] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:21 [I 2022-10-26 22:08:21] ...trap/engines/engine.py.229: process: 0.55075 | load: 0.00240 [I 2022-10-26 22:08:21] ...trap/engines/engine.py.230: loss: 0.17564 [I 2022-10-26 22:08:21] ...optimizers/trijoint.py.151: total_norm: 20.179834 [I 2022-10-26 22:08:27] ...trap/engines/engine.py.224: train: epoch 32 | batch 1080/2304 [I 2022-10-26 22:08:27] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:16 [I 2022-10-26 22:08:27] ...trap/engines/engine.py.229: process: 0.54640 | load: 0.00241 [I 2022-10-26 22:08:27] ...trap/engines/engine.py.230: loss: 0.13816 [I 2022-10-26 22:08:27] ...optimizers/trijoint.py.151: total_norm: 15.282094 [I 2022-10-26 22:08:32] ...trap/engines/engine.py.224: train: epoch 32 | batch 1090/2304 [I 2022-10-26 22:08:32] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:10 [I 2022-10-26 22:08:32] ...trap/engines/engine.py.229: process: 0.54487 | load: 0.00246 [I 2022-10-26 22:08:32] ...trap/engines/engine.py.230: loss: 0.11194 [I 2022-10-26 22:08:32] ...optimizers/trijoint.py.151: total_norm: 23.501751 [I 2022-10-26 22:08:38] ...trap/engines/engine.py.224: train: epoch 32 | batch 1100/2304 [I 2022-10-26 22:08:38] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:07 [I 2022-10-26 22:08:38] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00242 [I 2022-10-26 22:08:38] ...trap/engines/engine.py.230: loss: 0.24031 [I 2022-10-26 22:08:38] ...optimizers/trijoint.py.151: total_norm: 23.561836 [I 2022-10-26 22:08:44] ...trap/engines/engine.py.224: train: epoch 32 | batch 1110/2304 [I 2022-10-26 22:08:44] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:10:59 [I 2022-10-26 22:08:44] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00240 [I 2022-10-26 22:08:44] ...trap/engines/engine.py.230: loss: 0.23210 [I 2022-10-26 22:08:44] ...optimizers/trijoint.py.151: total_norm: 19.227520 [I 2022-10-26 22:08:49] ...trap/engines/engine.py.224: train: epoch 32 | batch 1120/2304 [I 2022-10-26 22:08:49] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:01 [I 2022-10-26 22:08:49] ...trap/engines/engine.py.229: process: 0.54587 | load: 0.00240 [I 2022-10-26 22:08:49] ...trap/engines/engine.py.230: loss: 0.19145 [I 2022-10-26 22:08:49] ...optimizers/trijoint.py.151: total_norm: 33.506302 [I 2022-10-26 22:08:55] ...trap/engines/engine.py.224: train: epoch 32 | batch 1130/2304 [I 2022-10-26 22:08:55] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:54 [I 2022-10-26 22:08:55] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00242 [I 2022-10-26 22:08:55] ...trap/engines/engine.py.230: loss: 0.15035 [I 2022-10-26 22:08:55] ...optimizers/trijoint.py.151: total_norm: 12.854078 [I 2022-10-26 22:09:00] ...trap/engines/engine.py.224: train: epoch 32 | batch 1140/2304 [I 2022-10-26 22:09:00] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:10:48 [I 2022-10-26 22:09:00] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00241 [I 2022-10-26 22:09:00] ...trap/engines/engine.py.230: loss: 0.15244 [I 2022-10-26 22:09:00] ...optimizers/trijoint.py.151: total_norm: 35.611797 [I 2022-10-26 22:09:26] ...trap/engines/engine.py.224: train: epoch 32 | batch 1150/2304 [I 2022-10-26 22:09:26] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:26:40 [I 2022-10-26 22:09:26] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00234 [I 2022-10-26 22:09:26] ...trap/engines/engine.py.230: loss: 0.22333 [I 2022-10-26 22:09:26] ...optimizers/trijoint.py.151: total_norm: 21.161900 [I 2022-10-26 22:09:31] ...trap/engines/engine.py.224: train: epoch 32 | batch 1160/2304 [I 2022-10-26 22:09:31] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:12:16 [I 2022-10-26 22:09:31] ...trap/engines/engine.py.229: process: 0.54849 | load: 0.00235 [I 2022-10-26 22:09:31] ...trap/engines/engine.py.230: loss: 0.18196 [I 2022-10-26 22:09:31] ...optimizers/trijoint.py.151: total_norm: 26.720041 [I 2022-10-26 22:09:37] ...trap/engines/engine.py.224: train: epoch 32 | batch 1170/2304 [I 2022-10-26 22:09:37] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:44 [I 2022-10-26 22:09:37] ...trap/engines/engine.py.229: process: 0.56842 | load: 0.00234 [I 2022-10-26 22:09:37] ...trap/engines/engine.py.230: loss: 0.17584 [I 2022-10-26 22:09:37] ...optimizers/trijoint.py.151: total_norm: 23.069408 [I 2022-10-26 22:09:42] ...trap/engines/engine.py.224: train: epoch 32 | batch 1180/2304 [I 2022-10-26 22:09:42] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:23 [I 2022-10-26 22:09:42] ...trap/engines/engine.py.229: process: 0.54684 | load: 0.00236 [I 2022-10-26 22:09:42] ...trap/engines/engine.py.230: loss: 0.22260 [I 2022-10-26 22:09:42] ...optimizers/trijoint.py.151: total_norm: 16.512556 [I 2022-10-26 22:09:48] ...trap/engines/engine.py.224: train: epoch 32 | batch 1190/2304 [I 2022-10-26 22:09:48] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:21 [I 2022-10-26 22:09:48] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00237 [I 2022-10-26 22:09:48] ...trap/engines/engine.py.230: loss: 0.23025 [I 2022-10-26 22:09:48] ...optimizers/trijoint.py.151: total_norm: 24.035866 [I 2022-10-26 22:09:54] ...trap/engines/engine.py.224: train: epoch 32 | batch 1200/2304 [I 2022-10-26 22:09:54] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:10:18 [I 2022-10-26 22:09:54] ...trap/engines/engine.py.229: process: 0.56130 | load: 0.00236 [I 2022-10-26 22:09:54] ...trap/engines/engine.py.230: loss: 0.15348 [I 2022-10-26 22:09:54] ...optimizers/trijoint.py.151: total_norm: 17.038204 [I 2022-10-26 22:09:59] ...trap/engines/engine.py.224: train: epoch 32 | batch 1210/2304 [I 2022-10-26 22:09:59] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:10:10 [I 2022-10-26 22:09:59] ...trap/engines/engine.py.229: process: 0.55190 | load: 0.00234 [I 2022-10-26 22:09:59] ...trap/engines/engine.py.230: loss: 0.21057 [I 2022-10-26 22:09:59] ...optimizers/trijoint.py.151: total_norm: 35.197945 [I 2022-10-26 22:10:05] ...trap/engines/engine.py.224: train: epoch 32 | batch 1220/2304 [I 2022-10-26 22:10:05] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:10:01 [I 2022-10-26 22:10:05] ...trap/engines/engine.py.229: process: 0.54686 | load: 0.00241 [I 2022-10-26 22:10:05] ...trap/engines/engine.py.230: loss: 0.21496 [I 2022-10-26 22:10:05] ...optimizers/trijoint.py.151: total_norm: 32.039085 [I 2022-10-26 22:10:10] ...trap/engines/engine.py.224: train: epoch 32 | batch 1230/2304 [I 2022-10-26 22:10:10] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:56 [I 2022-10-26 22:10:10] ...trap/engines/engine.py.229: process: 0.54368 | load: 0.00237 [I 2022-10-26 22:10:10] ...trap/engines/engine.py.230: loss: 0.15002 [I 2022-10-26 22:10:10] ...optimizers/trijoint.py.151: total_norm: 23.441542 [I 2022-10-26 22:10:16] ...trap/engines/engine.py.224: train: epoch 32 | batch 1240/2304 [I 2022-10-26 22:10:16] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:50 [I 2022-10-26 22:10:16] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00239 [I 2022-10-26 22:10:16] ...trap/engines/engine.py.230: loss: 0.13062 [I 2022-10-26 22:10:16] ...optimizers/trijoint.py.151: total_norm: 15.127831 [I 2022-10-26 22:10:21] ...trap/engines/engine.py.224: train: epoch 32 | batch 1250/2304 [I 2022-10-26 22:10:21] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:45 [I 2022-10-26 22:10:21] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00241 [I 2022-10-26 22:10:21] ...trap/engines/engine.py.230: loss: 0.21599 [I 2022-10-26 22:10:21] ...optimizers/trijoint.py.151: total_norm: 20.593063 [I 2022-10-26 22:10:27] ...trap/engines/engine.py.224: train: epoch 32 | batch 1260/2304 [I 2022-10-26 22:10:27] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:57 [I 2022-10-26 22:10:27] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00240 [I 2022-10-26 22:10:27] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-26 22:10:27] ...optimizers/trijoint.py.151: total_norm: 14.473266 [I 2022-10-26 22:10:33] ...trap/engines/engine.py.224: train: epoch 32 | batch 1270/2304 [I 2022-10-26 22:10:33] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:32 [I 2022-10-26 22:10:33] ...trap/engines/engine.py.229: process: 0.55227 | load: 0.00238 [I 2022-10-26 22:10:33] ...trap/engines/engine.py.230: loss: 0.21337 [I 2022-10-26 22:10:33] ...optimizers/trijoint.py.151: total_norm: 24.851139 [I 2022-10-26 22:10:38] ...trap/engines/engine.py.224: train: epoch 32 | batch 1280/2304 [I 2022-10-26 22:10:38] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:26 [I 2022-10-26 22:10:38] ...trap/engines/engine.py.229: process: 0.56224 | load: 0.00243 [I 2022-10-26 22:10:38] ...trap/engines/engine.py.230: loss: 0.19283 [I 2022-10-26 22:10:38] ...optimizers/trijoint.py.151: total_norm: 20.201544 [I 2022-10-26 22:10:44] ...trap/engines/engine.py.224: train: epoch 32 | batch 1290/2304 [I 2022-10-26 22:10:44] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:21 [I 2022-10-26 22:10:44] ...trap/engines/engine.py.229: process: 0.55518 | load: 0.00240 [I 2022-10-26 22:10:44] ...trap/engines/engine.py.230: loss: 0.16831 [I 2022-10-26 22:10:44] ...optimizers/trijoint.py.151: total_norm: 20.015125 [I 2022-10-26 22:10:49] ...trap/engines/engine.py.224: train: epoch 32 | batch 1300/2304 [I 2022-10-26 22:10:49] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:16 [I 2022-10-26 22:10:49] ...trap/engines/engine.py.229: process: 0.54903 | load: 0.00243 [I 2022-10-26 22:10:49] ...trap/engines/engine.py.230: loss: 0.22665 [I 2022-10-26 22:10:49] ...optimizers/trijoint.py.151: total_norm: 21.110300 [I 2022-10-26 22:10:55] ...trap/engines/engine.py.224: train: epoch 32 | batch 1310/2304 [I 2022-10-26 22:10:55] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:09:09 [I 2022-10-26 22:10:55] ...trap/engines/engine.py.229: process: 0.54589 | load: 0.00239 [I 2022-10-26 22:10:55] ...trap/engines/engine.py.230: loss: 0.13117 [I 2022-10-26 22:10:55] ...optimizers/trijoint.py.151: total_norm: 22.406477 [I 2022-10-26 22:11:00] ...trap/engines/engine.py.224: train: epoch 32 | batch 1320/2304 [I 2022-10-26 22:11:00] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:09:05 [I 2022-10-26 22:11:00] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00240 [I 2022-10-26 22:11:00] ...trap/engines/engine.py.230: loss: 0.17923 [I 2022-10-26 22:11:00] ...optimizers/trijoint.py.151: total_norm: 23.350481 [I 2022-10-26 22:11:06] ...trap/engines/engine.py.224: train: epoch 32 | batch 1330/2304 [I 2022-10-26 22:11:06] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:09:00 [I 2022-10-26 22:11:06] ...trap/engines/engine.py.229: process: 0.55525 | load: 0.00239 [I 2022-10-26 22:11:06] ...trap/engines/engine.py.230: loss: 0.23080 [I 2022-10-26 22:11:06] ...optimizers/trijoint.py.151: total_norm: 19.210829 [I 2022-10-26 22:11:12] ...trap/engines/engine.py.224: train: epoch 32 | batch 1340/2304 [I 2022-10-26 22:11:12] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:52 [I 2022-10-26 22:11:12] ...trap/engines/engine.py.229: process: 0.54452 | load: 0.00239 [I 2022-10-26 22:11:12] ...trap/engines/engine.py.230: loss: 0.23403 [I 2022-10-26 22:11:12] ...optimizers/trijoint.py.151: total_norm: 13.793468 [I 2022-10-26 22:11:17] ...trap/engines/engine.py.224: train: epoch 32 | batch 1350/2304 [I 2022-10-26 22:11:17] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:46 [I 2022-10-26 22:11:17] ...trap/engines/engine.py.229: process: 0.55391 | load: 0.00237 [I 2022-10-26 22:11:17] ...trap/engines/engine.py.230: loss: 0.12477 [I 2022-10-26 22:11:17] ...optimizers/trijoint.py.151: total_norm: 10.272202 [I 2022-10-26 22:11:23] ...trap/engines/engine.py.224: train: epoch 32 | batch 1360/2304 [I 2022-10-26 22:11:23] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:43 [I 2022-10-26 22:11:23] ...trap/engines/engine.py.229: process: 0.54980 | load: 0.00238 [I 2022-10-26 22:11:23] ...trap/engines/engine.py.230: loss: 0.23511 [I 2022-10-26 22:11:23] ...optimizers/trijoint.py.151: total_norm: 24.111616 [I 2022-10-26 22:11:28] ...trap/engines/engine.py.224: train: epoch 32 | batch 1370/2304 [I 2022-10-26 22:11:28] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:38 [I 2022-10-26 22:11:28] ...trap/engines/engine.py.229: process: 0.56422 | load: 0.00239 [I 2022-10-26 22:11:28] ...trap/engines/engine.py.230: loss: 0.13291 [I 2022-10-26 22:11:28] ...optimizers/trijoint.py.151: total_norm: 13.094130 [I 2022-10-26 22:11:34] ...trap/engines/engine.py.224: train: epoch 32 | batch 1380/2304 [I 2022-10-26 22:11:34] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:30 [I 2022-10-26 22:11:34] ...trap/engines/engine.py.229: process: 0.55521 | load: 0.00238 [I 2022-10-26 22:11:34] ...trap/engines/engine.py.230: loss: 0.16750 [I 2022-10-26 22:11:34] ...optimizers/trijoint.py.151: total_norm: 18.917616 [I 2022-10-26 22:11:39] ...trap/engines/engine.py.224: train: epoch 32 | batch 1390/2304 [I 2022-10-26 22:11:39] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:22 [I 2022-10-26 22:11:39] ...trap/engines/engine.py.229: process: 0.54487 | load: 0.00238 [I 2022-10-26 22:11:39] ...trap/engines/engine.py.230: loss: 0.16161 [I 2022-10-26 22:11:39] ...optimizers/trijoint.py.151: total_norm: 12.810156 [I 2022-10-26 22:11:45] ...trap/engines/engine.py.224: train: epoch 32 | batch 1400/2304 [I 2022-10-26 22:11:45] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:20 [I 2022-10-26 22:11:45] ...trap/engines/engine.py.229: process: 0.55532 | load: 0.00240 [I 2022-10-26 22:11:45] ...trap/engines/engine.py.230: loss: 0.11795 [I 2022-10-26 22:11:45] ...optimizers/trijoint.py.151: total_norm: 13.944751 [I 2022-10-26 22:11:50] ...trap/engines/engine.py.224: train: epoch 32 | batch 1410/2304 [I 2022-10-26 22:11:50] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:14 [I 2022-10-26 22:11:50] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00238 [I 2022-10-26 22:11:50] ...trap/engines/engine.py.230: loss: 0.16929 [I 2022-10-26 22:11:50] ...optimizers/trijoint.py.151: total_norm: 33.788017 [I 2022-10-26 22:11:56] ...trap/engines/engine.py.224: train: epoch 32 | batch 1420/2304 [I 2022-10-26 22:11:56] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:08:08 [I 2022-10-26 22:11:56] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00236 [I 2022-10-26 22:11:56] ...trap/engines/engine.py.230: loss: 0.17362 [I 2022-10-26 22:11:56] ...optimizers/trijoint.py.151: total_norm: 16.718910 [I 2022-10-26 22:12:01] ...trap/engines/engine.py.224: train: epoch 32 | batch 1430/2304 [I 2022-10-26 22:12:01] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:08:06 [I 2022-10-26 22:12:01] ...trap/engines/engine.py.229: process: 0.56473 | load: 0.00239 [I 2022-10-26 22:12:01] ...trap/engines/engine.py.230: loss: 0.16114 [I 2022-10-26 22:12:01] ...optimizers/trijoint.py.151: total_norm: 15.023539 [I 2022-10-26 22:12:07] ...trap/engines/engine.py.224: train: epoch 32 | batch 1440/2304 [I 2022-10-26 22:12:07] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:57 [I 2022-10-26 22:12:07] ...trap/engines/engine.py.229: process: 0.55135 | load: 0.00243 [I 2022-10-26 22:12:07] ...trap/engines/engine.py.230: loss: 0.16809 [I 2022-10-26 22:12:07] ...optimizers/trijoint.py.151: total_norm: 31.979935 [I 2022-10-26 22:12:13] ...trap/engines/engine.py.224: train: epoch 32 | batch 1450/2304 [I 2022-10-26 22:12:13] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:53 [I 2022-10-26 22:12:13] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00244 [I 2022-10-26 22:12:13] ...trap/engines/engine.py.230: loss: 0.14563 [I 2022-10-26 22:12:13] ...optimizers/trijoint.py.151: total_norm: 17.763588 [I 2022-10-26 22:12:18] ...trap/engines/engine.py.224: train: epoch 32 | batch 1460/2304 [I 2022-10-26 22:12:18] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:48 [I 2022-10-26 22:12:18] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00242 [I 2022-10-26 22:12:18] ...trap/engines/engine.py.230: loss: 0.22515 [I 2022-10-26 22:12:18] ...optimizers/trijoint.py.151: total_norm: 23.007694 [I 2022-10-26 22:12:24] ...trap/engines/engine.py.224: train: epoch 32 | batch 1470/2304 [I 2022-10-26 22:12:24] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:48 [I 2022-10-26 22:12:24] ...trap/engines/engine.py.229: process: 0.56349 | load: 0.00241 [I 2022-10-26 22:12:24] ...trap/engines/engine.py.230: loss: 0.24560 [I 2022-10-26 22:12:24] ...optimizers/trijoint.py.151: total_norm: 17.015099 [I 2022-10-26 22:12:29] ...trap/engines/engine.py.224: train: epoch 32 | batch 1480/2304 [I 2022-10-26 22:12:29] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:41 [I 2022-10-26 22:12:29] ...trap/engines/engine.py.229: process: 0.56976 | load: 0.00242 [I 2022-10-26 22:12:29] ...trap/engines/engine.py.230: loss: 0.18326 [I 2022-10-26 22:12:29] ...optimizers/trijoint.py.151: total_norm: 20.705681 [I 2022-10-26 22:12:35] ...trap/engines/engine.py.224: train: epoch 32 | batch 1490/2304 [I 2022-10-26 22:12:35] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:34 [I 2022-10-26 22:12:35] ...trap/engines/engine.py.229: process: 0.56580 | load: 0.00242 [I 2022-10-26 22:12:35] ...trap/engines/engine.py.230: loss: 0.18238 [I 2022-10-26 22:12:35] ...optimizers/trijoint.py.151: total_norm: 20.121204 [I 2022-10-26 22:12:41] ...trap/engines/engine.py.224: train: epoch 32 | batch 1500/2304 [I 2022-10-26 22:12:41] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:30 [I 2022-10-26 22:12:41] ...trap/engines/engine.py.229: process: 0.56477 | load: 0.00247 [I 2022-10-26 22:12:41] ...trap/engines/engine.py.230: loss: 0.23981 [I 2022-10-26 22:12:41] ...optimizers/trijoint.py.151: total_norm: 17.096802 [I 2022-10-26 22:12:46] ...trap/engines/engine.py.224: train: epoch 32 | batch 1510/2304 [I 2022-10-26 22:12:46] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:21 [I 2022-10-26 22:12:46] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00245 [I 2022-10-26 22:12:46] ...trap/engines/engine.py.230: loss: 0.16050 [I 2022-10-26 22:12:46] ...optimizers/trijoint.py.151: total_norm: 18.305473 [I 2022-10-26 22:12:52] ...trap/engines/engine.py.224: train: epoch 32 | batch 1520/2304 [I 2022-10-26 22:12:52] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:14 [I 2022-10-26 22:12:52] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00243 [I 2022-10-26 22:12:52] ...trap/engines/engine.py.230: loss: 0.21298 [I 2022-10-26 22:12:52] ...optimizers/trijoint.py.151: total_norm: 23.423752 [I 2022-10-26 22:12:57] ...trap/engines/engine.py.224: train: epoch 32 | batch 1530/2304 [I 2022-10-26 22:12:57] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:07:16 [I 2022-10-26 22:12:57] ...trap/engines/engine.py.229: process: 0.57568 | load: 0.00250 [I 2022-10-26 22:12:57] ...trap/engines/engine.py.230: loss: 0.19480 [I 2022-10-26 22:12:57] ...optimizers/trijoint.py.151: total_norm: 14.982207 [I 2022-10-26 22:13:03] ...trap/engines/engine.py.224: train: epoch 32 | batch 1540/2304 [I 2022-10-26 22:13:03] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:07:08 [I 2022-10-26 22:13:03] ...trap/engines/engine.py.229: process: 0.56646 | load: 0.00253 [I 2022-10-26 22:13:03] ...trap/engines/engine.py.230: loss: 0.20760 [I 2022-10-26 22:13:03] ...optimizers/trijoint.py.151: total_norm: 24.488279 [I 2022-10-26 22:13:09] ...trap/engines/engine.py.224: train: epoch 32 | batch 1550/2304 [I 2022-10-26 22:13:09] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:07:00 [I 2022-10-26 22:13:09] ...trap/engines/engine.py.229: process: 0.55280 | load: 0.00250 [I 2022-10-26 22:13:09] ...trap/engines/engine.py.230: loss: 0.18553 [I 2022-10-26 22:13:09] ...optimizers/trijoint.py.151: total_norm: 24.418379 [I 2022-10-26 22:13:14] ...trap/engines/engine.py.224: train: epoch 32 | batch 1560/2304 [I 2022-10-26 22:13:14] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:53 [I 2022-10-26 22:13:14] ...trap/engines/engine.py.229: process: 0.54900 | load: 0.00244 [I 2022-10-26 22:13:14] ...trap/engines/engine.py.230: loss: 0.15722 [I 2022-10-26 22:13:14] ...optimizers/trijoint.py.151: total_norm: 21.379616 [I 2022-10-26 22:13:20] ...trap/engines/engine.py.224: train: epoch 32 | batch 1570/2304 [I 2022-10-26 22:13:20] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:47 [I 2022-10-26 22:13:20] ...trap/engines/engine.py.229: process: 0.55578 | load: 0.00244 [I 2022-10-26 22:13:20] ...trap/engines/engine.py.230: loss: 0.18121 [I 2022-10-26 22:13:20] ...optimizers/trijoint.py.151: total_norm: 12.713792 [I 2022-10-26 22:13:25] ...trap/engines/engine.py.224: train: epoch 32 | batch 1580/2304 [I 2022-10-26 22:13:25] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:41 [I 2022-10-26 22:13:25] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00245 [I 2022-10-26 22:13:25] ...trap/engines/engine.py.230: loss: 0.13555 [I 2022-10-26 22:13:25] ...optimizers/trijoint.py.151: total_norm: 10.758406 [I 2022-10-26 22:13:31] ...trap/engines/engine.py.224: train: epoch 32 | batch 1590/2304 [I 2022-10-26 22:13:31] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:35 [I 2022-10-26 22:13:31] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00243 [I 2022-10-26 22:13:31] ...trap/engines/engine.py.230: loss: 0.19376 [I 2022-10-26 22:13:31] ...optimizers/trijoint.py.151: total_norm: 20.408926 [I 2022-10-26 22:13:36] ...trap/engines/engine.py.224: train: epoch 32 | batch 1600/2304 [I 2022-10-26 22:13:36] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:31 [I 2022-10-26 22:13:36] ...trap/engines/engine.py.229: process: 0.55062 | load: 0.00242 [I 2022-10-26 22:13:36] ...trap/engines/engine.py.230: loss: 0.16672 [I 2022-10-26 22:13:36] ...optimizers/trijoint.py.151: total_norm: 24.372700 [I 2022-10-26 22:13:42] ...trap/engines/engine.py.224: train: epoch 32 | batch 1610/2304 [I 2022-10-26 22:13:42] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:26 [I 2022-10-26 22:13:42] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00241 [I 2022-10-26 22:13:42] ...trap/engines/engine.py.230: loss: 0.17989 [I 2022-10-26 22:13:42] ...optimizers/trijoint.py.151: total_norm: 12.876432 [I 2022-10-26 22:13:47] ...trap/engines/engine.py.224: train: epoch 32 | batch 1620/2304 [I 2022-10-26 22:13:47] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:20 [I 2022-10-26 22:13:47] ...trap/engines/engine.py.229: process: 0.55070 | load: 0.00251 [I 2022-10-26 22:13:47] ...trap/engines/engine.py.230: loss: 0.22806 [I 2022-10-26 22:13:47] ...optimizers/trijoint.py.151: total_norm: 28.505238 [I 2022-10-26 22:13:53] ...trap/engines/engine.py.224: train: epoch 32 | batch 1630/2304 [I 2022-10-26 22:13:53] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:16 [I 2022-10-26 22:13:53] ...trap/engines/engine.py.229: process: 0.56240 | load: 0.00243 [I 2022-10-26 22:13:53] ...trap/engines/engine.py.230: loss: 0.16199 [I 2022-10-26 22:13:53] ...optimizers/trijoint.py.151: total_norm: 44.960392 [I 2022-10-26 22:13:59] ...trap/engines/engine.py.224: train: epoch 32 | batch 1640/2304 [I 2022-10-26 22:13:59] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:06:11 [I 2022-10-26 22:13:59] ...trap/engines/engine.py.229: process: 0.55347 | load: 0.00268 [I 2022-10-26 22:13:59] ...trap/engines/engine.py.230: loss: 0.18704 [I 2022-10-26 22:13:59] ...optimizers/trijoint.py.151: total_norm: 25.004223 [I 2022-10-26 22:14:04] ...trap/engines/engine.py.224: train: epoch 32 | batch 1650/2304 [I 2022-10-26 22:14:04] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:06:03 [I 2022-10-26 22:14:04] ...trap/engines/engine.py.229: process: 0.55161 | load: 0.00261 [I 2022-10-26 22:14:04] ...trap/engines/engine.py.230: loss: 0.25862 [I 2022-10-26 22:14:04] ...optimizers/trijoint.py.151: total_norm: 14.545080 [I 2022-10-26 22:14:10] ...trap/engines/engine.py.224: train: epoch 32 | batch 1660/2304 [I 2022-10-26 22:14:10] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:57 [I 2022-10-26 22:14:10] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00250 [I 2022-10-26 22:14:10] ...trap/engines/engine.py.230: loss: 0.16772 [I 2022-10-26 22:14:10] ...optimizers/trijoint.py.151: total_norm: 11.449852 [I 2022-10-26 22:14:15] ...trap/engines/engine.py.224: train: epoch 32 | batch 1670/2304 [I 2022-10-26 22:14:15] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:56 [I 2022-10-26 22:14:15] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00250 [I 2022-10-26 22:14:15] ...trap/engines/engine.py.230: loss: 0.17972 [I 2022-10-26 22:14:15] ...optimizers/trijoint.py.151: total_norm: 9.566087 [I 2022-10-26 22:14:21] ...trap/engines/engine.py.224: train: epoch 32 | batch 1680/2304 [I 2022-10-26 22:14:21] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:49 [I 2022-10-26 22:14:21] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00260 [I 2022-10-26 22:14:21] ...trap/engines/engine.py.230: loss: 0.19884 [I 2022-10-26 22:14:21] ...optimizers/trijoint.py.151: total_norm: 16.086132 [I 2022-10-26 22:14:27] ...trap/engines/engine.py.224: train: epoch 32 | batch 1690/2304 [I 2022-10-26 22:14:27] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:42 [I 2022-10-26 22:14:27] ...trap/engines/engine.py.229: process: 0.55395 | load: 0.00252 [I 2022-10-26 22:14:27] ...trap/engines/engine.py.230: loss: 0.11405 [I 2022-10-26 22:14:27] ...optimizers/trijoint.py.151: total_norm: 11.428994 [I 2022-10-26 22:14:32] ...trap/engines/engine.py.224: train: epoch 32 | batch 1700/2304 [I 2022-10-26 22:14:32] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:37 [I 2022-10-26 22:14:32] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00258 [I 2022-10-26 22:14:32] ...trap/engines/engine.py.230: loss: 0.15955 [I 2022-10-26 22:14:32] ...optimizers/trijoint.py.151: total_norm: 18.449232 [I 2022-10-26 22:14:38] ...trap/engines/engine.py.224: train: epoch 32 | batch 1710/2304 [I 2022-10-26 22:14:38] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:32 [I 2022-10-26 22:14:38] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00284 [I 2022-10-26 22:14:38] ...trap/engines/engine.py.230: loss: 0.24333 [I 2022-10-26 22:14:38] ...optimizers/trijoint.py.151: total_norm: 18.061590 [I 2022-10-26 22:14:43] ...trap/engines/engine.py.224: train: epoch 32 | batch 1720/2304 [I 2022-10-26 22:14:43] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:26 [I 2022-10-26 22:14:43] ...trap/engines/engine.py.229: process: 0.54847 | load: 0.00268 [I 2022-10-26 22:14:43] ...trap/engines/engine.py.230: loss: 0.23718 [I 2022-10-26 22:14:43] ...optimizers/trijoint.py.151: total_norm: 14.668228 [I 2022-10-26 22:14:49] ...trap/engines/engine.py.224: train: epoch 32 | batch 1730/2304 [I 2022-10-26 22:14:49] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:18 [I 2022-10-26 22:14:49] ...trap/engines/engine.py.229: process: 0.54601 | load: 0.00265 [I 2022-10-26 22:14:49] ...trap/engines/engine.py.230: loss: 0.23187 [I 2022-10-26 22:14:49] ...optimizers/trijoint.py.151: total_norm: 14.593027 [I 2022-10-26 22:14:55] ...trap/engines/engine.py.224: train: epoch 32 | batch 1740/2304 [I 2022-10-26 22:14:55] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:05:12 [I 2022-10-26 22:14:55] ...trap/engines/engine.py.229: process: 0.54963 | load: 0.00256 [I 2022-10-26 22:14:55] ...trap/engines/engine.py.230: loss: 0.18015 [I 2022-10-26 22:14:55] ...optimizers/trijoint.py.151: total_norm: 21.081358 [I 2022-10-26 22:15:00] ...trap/engines/engine.py.224: train: epoch 32 | batch 1750/2304 [I 2022-10-26 22:15:00] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:05:06 [I 2022-10-26 22:15:00] ...trap/engines/engine.py.229: process: 0.56309 | load: 0.00256 [I 2022-10-26 22:15:00] ...trap/engines/engine.py.230: loss: 0.21034 [I 2022-10-26 22:15:00] ...optimizers/trijoint.py.151: total_norm: 23.767214 [I 2022-10-26 22:15:06] ...trap/engines/engine.py.224: train: epoch 32 | batch 1760/2304 [I 2022-10-26 22:15:06] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:59 [I 2022-10-26 22:15:06] ...trap/engines/engine.py.229: process: 0.54663 | load: 0.00261 [I 2022-10-26 22:15:06] ...trap/engines/engine.py.230: loss: 0.12298 [I 2022-10-26 22:15:06] ...optimizers/trijoint.py.151: total_norm: 10.099318 [I 2022-10-26 22:15:11] ...trap/engines/engine.py.224: train: epoch 32 | batch 1770/2304 [I 2022-10-26 22:15:11] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:55 [I 2022-10-26 22:15:11] ...trap/engines/engine.py.229: process: 0.55804 | load: 0.00253 [I 2022-10-26 22:15:11] ...trap/engines/engine.py.230: loss: 0.21491 [I 2022-10-26 22:15:11] ...optimizers/trijoint.py.151: total_norm: 28.846474 [I 2022-10-26 22:15:17] ...trap/engines/engine.py.224: train: epoch 32 | batch 1780/2304 [I 2022-10-26 22:15:17] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:51 [I 2022-10-26 22:15:17] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00249 [I 2022-10-26 22:15:17] ...trap/engines/engine.py.230: loss: 0.13581 [I 2022-10-26 22:15:17] ...optimizers/trijoint.py.151: total_norm: 9.742918 [I 2022-10-26 22:15:22] ...trap/engines/engine.py.224: train: epoch 32 | batch 1790/2304 [I 2022-10-26 22:15:22] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:44 [I 2022-10-26 22:15:22] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00253 [I 2022-10-26 22:15:22] ...trap/engines/engine.py.230: loss: 0.16655 [I 2022-10-26 22:15:22] ...optimizers/trijoint.py.151: total_norm: 17.348083 [I 2022-10-26 22:15:28] ...trap/engines/engine.py.224: train: epoch 32 | batch 1800/2304 [I 2022-10-26 22:15:28] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:47 [I 2022-10-26 22:15:28] ...trap/engines/engine.py.229: process: 0.57158 | load: 0.00312 [I 2022-10-26 22:15:28] ...trap/engines/engine.py.230: loss: 0.20887 [I 2022-10-26 22:15:28] ...optimizers/trijoint.py.151: total_norm: 16.898312 [I 2022-10-26 22:15:34] ...trap/engines/engine.py.224: train: epoch 32 | batch 1810/2304 [I 2022-10-26 22:15:34] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:34 [I 2022-10-26 22:15:34] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00276 [I 2022-10-26 22:15:34] ...trap/engines/engine.py.230: loss: 0.13369 [I 2022-10-26 22:15:34] ...optimizers/trijoint.py.151: total_norm: 14.459564 [I 2022-10-26 22:15:39] ...trap/engines/engine.py.224: train: epoch 32 | batch 1820/2304 [I 2022-10-26 22:15:39] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:33 [I 2022-10-26 22:15:39] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00269 [I 2022-10-26 22:15:39] ...trap/engines/engine.py.230: loss: 0.15357 [I 2022-10-26 22:15:39] ...optimizers/trijoint.py.151: total_norm: 20.872465 [I 2022-10-26 22:15:45] ...trap/engines/engine.py.224: train: epoch 32 | batch 1830/2304 [I 2022-10-26 22:15:45] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:23 [I 2022-10-26 22:15:45] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00257 [I 2022-10-26 22:15:45] ...trap/engines/engine.py.230: loss: 0.19721 [I 2022-10-26 22:15:45] ...optimizers/trijoint.py.151: total_norm: 17.083733 [I 2022-10-26 22:15:50] ...trap/engines/engine.py.224: train: epoch 32 | batch 1840/2304 [I 2022-10-26 22:15:50] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:16 [I 2022-10-26 22:15:50] ...trap/engines/engine.py.229: process: 0.55431 | load: 0.00258 [I 2022-10-26 22:15:50] ...trap/engines/engine.py.230: loss: 0.25643 [I 2022-10-26 22:15:50] ...optimizers/trijoint.py.151: total_norm: 25.857832 [I 2022-10-26 22:15:56] ...trap/engines/engine.py.224: train: epoch 32 | batch 1850/2304 [I 2022-10-26 22:15:56] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:04:13 [I 2022-10-26 22:15:56] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00280 [I 2022-10-26 22:15:56] ...trap/engines/engine.py.230: loss: 0.18720 [I 2022-10-26 22:15:56] ...optimizers/trijoint.py.151: total_norm: 26.629992 [I 2022-10-26 22:16:02] ...trap/engines/engine.py.224: train: epoch 32 | batch 1860/2304 [I 2022-10-26 22:16:02] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:04:08 [I 2022-10-26 22:16:02] ...trap/engines/engine.py.229: process: 0.55414 | load: 0.00246 [I 2022-10-26 22:16:02] ...trap/engines/engine.py.230: loss: 0.15528 [I 2022-10-26 22:16:02] ...optimizers/trijoint.py.151: total_norm: 11.122389 [I 2022-10-26 22:16:07] ...trap/engines/engine.py.224: train: epoch 32 | batch 1870/2304 [I 2022-10-26 22:16:07] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:04:03 [I 2022-10-26 22:16:07] ...trap/engines/engine.py.229: process: 0.54935 | load: 0.00258 [I 2022-10-26 22:16:07] ...trap/engines/engine.py.230: loss: 0.12809 [I 2022-10-26 22:16:07] ...optimizers/trijoint.py.151: total_norm: 16.338924 [I 2022-10-26 22:16:13] ...trap/engines/engine.py.224: train: epoch 32 | batch 1880/2304 [I 2022-10-26 22:16:13] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:58 [I 2022-10-26 22:16:13] ...trap/engines/engine.py.229: process: 0.57151 | load: 0.00269 [I 2022-10-26 22:16:13] ...trap/engines/engine.py.230: loss: 0.16285 [I 2022-10-26 22:16:13] ...optimizers/trijoint.py.151: total_norm: 17.085693 [I 2022-10-26 22:16:18] ...trap/engines/engine.py.224: train: epoch 32 | batch 1890/2304 [I 2022-10-26 22:16:18] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:49 [I 2022-10-26 22:16:18] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00249 [I 2022-10-26 22:16:18] ...trap/engines/engine.py.230: loss: 0.20318 [I 2022-10-26 22:16:18] ...optimizers/trijoint.py.151: total_norm: 13.840519 [I 2022-10-26 22:16:24] ...trap/engines/engine.py.224: train: epoch 32 | batch 1900/2304 [I 2022-10-26 22:16:24] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:47 [I 2022-10-26 22:16:24] ...trap/engines/engine.py.229: process: 0.54653 | load: 0.00256 [I 2022-10-26 22:16:24] ...trap/engines/engine.py.230: loss: 0.21461 [I 2022-10-26 22:16:24] ...optimizers/trijoint.py.151: total_norm: 33.026283 [I 2022-10-26 22:16:30] ...trap/engines/engine.py.224: train: epoch 32 | batch 1910/2304 [I 2022-10-26 22:16:30] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:40 [I 2022-10-26 22:16:30] ...trap/engines/engine.py.229: process: 0.55824 | load: 0.00244 [I 2022-10-26 22:16:30] ...trap/engines/engine.py.230: loss: 0.17161 [I 2022-10-26 22:16:30] ...optimizers/trijoint.py.151: total_norm: 15.258205 [I 2022-10-26 22:16:35] ...trap/engines/engine.py.224: train: epoch 32 | batch 1920/2304 [I 2022-10-26 22:16:35] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:34 [I 2022-10-26 22:16:35] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00247 [I 2022-10-26 22:16:35] ...trap/engines/engine.py.230: loss: 0.16865 [I 2022-10-26 22:16:35] ...optimizers/trijoint.py.151: total_norm: 11.604537 [I 2022-10-26 22:16:41] ...trap/engines/engine.py.224: train: epoch 32 | batch 1930/2304 [I 2022-10-26 22:16:41] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:28 [I 2022-10-26 22:16:41] ...trap/engines/engine.py.229: process: 0.56481 | load: 0.00241 [I 2022-10-26 22:16:41] ...trap/engines/engine.py.230: loss: 0.14019 [I 2022-10-26 22:16:41] ...optimizers/trijoint.py.151: total_norm: 13.727272 [I 2022-10-26 22:16:46] ...trap/engines/engine.py.224: train: epoch 32 | batch 1940/2304 [I 2022-10-26 22:16:46] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:23 [I 2022-10-26 22:16:46] ...trap/engines/engine.py.229: process: 0.56313 | load: 0.00243 [I 2022-10-26 22:16:46] ...trap/engines/engine.py.230: loss: 0.13602 [I 2022-10-26 22:16:46] ...optimizers/trijoint.py.151: total_norm: 16.516756 [I 2022-10-26 22:16:52] ...trap/engines/engine.py.224: train: epoch 32 | batch 1950/2304 [I 2022-10-26 22:16:52] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:16 [I 2022-10-26 22:16:52] ...trap/engines/engine.py.229: process: 0.55038 | load: 0.00248 [I 2022-10-26 22:16:52] ...trap/engines/engine.py.230: loss: 0.18373 [I 2022-10-26 22:16:52] ...optimizers/trijoint.py.151: total_norm: 48.526733 [I 2022-10-26 22:16:58] ...trap/engines/engine.py.224: train: epoch 32 | batch 1960/2304 [I 2022-10-26 22:16:58] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:03:17 [I 2022-10-26 22:16:58] ...trap/engines/engine.py.229: process: 0.54913 | load: 0.00246 [I 2022-10-26 22:16:58] ...trap/engines/engine.py.230: loss: 0.19246 [I 2022-10-26 22:16:58] ...optimizers/trijoint.py.151: total_norm: 21.664642 [I 2022-10-26 22:17:03] ...trap/engines/engine.py.224: train: epoch 32 | batch 1970/2304 [I 2022-10-26 22:17:03] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:03:07 [I 2022-10-26 22:17:03] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00256 [I 2022-10-26 22:17:03] ...trap/engines/engine.py.230: loss: 0.15350 [I 2022-10-26 22:17:03] ...optimizers/trijoint.py.151: total_norm: 11.786414 [I 2022-10-26 22:17:09] ...trap/engines/engine.py.224: train: epoch 32 | batch 1980/2304 [I 2022-10-26 22:17:09] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:03:00 [I 2022-10-26 22:17:09] ...trap/engines/engine.py.229: process: 0.55400 | load: 0.00251 [I 2022-10-26 22:17:09] ...trap/engines/engine.py.230: loss: 0.20469 [I 2022-10-26 22:17:09] ...optimizers/trijoint.py.151: total_norm: 47.906368 [I 2022-10-26 22:17:14] ...trap/engines/engine.py.224: train: epoch 32 | batch 1990/2304 [I 2022-10-26 22:17:14] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:54 [I 2022-10-26 22:17:14] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00248 [I 2022-10-26 22:17:14] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-26 22:17:14] ...optimizers/trijoint.py.151: total_norm: 16.535213 [I 2022-10-26 22:17:20] ...trap/engines/engine.py.224: train: epoch 32 | batch 2000/2304 [I 2022-10-26 22:17:20] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:50 [I 2022-10-26 22:17:20] ...trap/engines/engine.py.229: process: 0.56492 | load: 0.00244 [I 2022-10-26 22:17:20] ...trap/engines/engine.py.230: loss: 0.19538 [I 2022-10-26 22:17:20] ...optimizers/trijoint.py.151: total_norm: 20.876160 [I 2022-10-26 22:17:25] ...trap/engines/engine.py.224: train: epoch 32 | batch 2010/2304 [I 2022-10-26 22:17:25] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:43 [I 2022-10-26 22:17:25] ...trap/engines/engine.py.229: process: 0.55280 | load: 0.00309 [I 2022-10-26 22:17:25] ...trap/engines/engine.py.230: loss: 0.12099 [I 2022-10-26 22:17:25] ...optimizers/trijoint.py.151: total_norm: 10.229914 [I 2022-10-26 22:17:31] ...trap/engines/engine.py.224: train: epoch 32 | batch 2020/2304 [I 2022-10-26 22:17:31] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:36 [I 2022-10-26 22:17:31] ...trap/engines/engine.py.229: process: 0.54062 | load: 0.00253 [I 2022-10-26 22:17:31] ...trap/engines/engine.py.230: loss: 0.17252 [I 2022-10-26 22:17:31] ...optimizers/trijoint.py.151: total_norm: 18.255466 [I 2022-10-26 22:17:37] ...trap/engines/engine.py.224: train: epoch 32 | batch 2030/2304 [I 2022-10-26 22:17:37] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:31 [I 2022-10-26 22:17:37] ...trap/engines/engine.py.229: process: 0.55317 | load: 0.00257 [I 2022-10-26 22:17:37] ...trap/engines/engine.py.230: loss: 0.16522 [I 2022-10-26 22:17:37] ...optimizers/trijoint.py.151: total_norm: 26.704945 [I 2022-10-26 22:17:42] ...trap/engines/engine.py.224: train: epoch 32 | batch 2040/2304 [I 2022-10-26 22:17:42] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:26 [I 2022-10-26 22:17:42] ...trap/engines/engine.py.229: process: 0.56344 | load: 0.00244 [I 2022-10-26 22:17:42] ...trap/engines/engine.py.230: loss: 0.15762 [I 2022-10-26 22:17:42] ...optimizers/trijoint.py.151: total_norm: 16.637733 [I 2022-10-26 22:17:48] ...trap/engines/engine.py.224: train: epoch 32 | batch 2050/2304 [I 2022-10-26 22:17:48] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:22 [I 2022-10-26 22:17:48] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00264 [I 2022-10-26 22:17:48] ...trap/engines/engine.py.230: loss: 0.16791 [I 2022-10-26 22:17:48] ...optimizers/trijoint.py.151: total_norm: 19.177061 [I 2022-10-26 22:17:53] ...trap/engines/engine.py.224: train: epoch 32 | batch 2060/2304 [I 2022-10-26 22:17:53] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:16 [I 2022-10-26 22:17:53] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00254 [I 2022-10-26 22:17:53] ...trap/engines/engine.py.230: loss: 0.18639 [I 2022-10-26 22:17:53] ...optimizers/trijoint.py.151: total_norm: 13.142663 [I 2022-10-26 22:17:59] ...trap/engines/engine.py.224: train: epoch 32 | batch 2070/2304 [I 2022-10-26 22:17:59] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:12 [I 2022-10-26 22:17:59] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00250 [I 2022-10-26 22:17:59] ...trap/engines/engine.py.230: loss: 0.20048 [I 2022-10-26 22:17:59] ...optimizers/trijoint.py.151: total_norm: 22.163765 [I 2022-10-26 22:18:05] ...trap/engines/engine.py.224: train: epoch 32 | batch 2080/2304 [I 2022-10-26 22:18:05] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:02:05 [I 2022-10-26 22:18:05] ...trap/engines/engine.py.229: process: 0.55993 | load: 0.00251 [I 2022-10-26 22:18:05] ...trap/engines/engine.py.230: loss: 0.17849 [I 2022-10-26 22:18:05] ...optimizers/trijoint.py.151: total_norm: 14.824883 [I 2022-10-26 22:18:10] ...trap/engines/engine.py.224: train: epoch 32 | batch 2090/2304 [I 2022-10-26 22:18:10] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:59 [I 2022-10-26 22:18:10] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00269 [I 2022-10-26 22:18:10] ...trap/engines/engine.py.230: loss: 0.17297 [I 2022-10-26 22:18:10] ...optimizers/trijoint.py.151: total_norm: 16.264502 [I 2022-10-26 22:18:16] ...trap/engines/engine.py.224: train: epoch 32 | batch 2100/2304 [I 2022-10-26 22:18:16] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:53 [I 2022-10-26 22:18:16] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00254 [I 2022-10-26 22:18:16] ...trap/engines/engine.py.230: loss: 0.18816 [I 2022-10-26 22:18:16] ...optimizers/trijoint.py.151: total_norm: 17.946434 [I 2022-10-26 22:18:21] ...trap/engines/engine.py.224: train: epoch 32 | batch 2110/2304 [I 2022-10-26 22:18:21] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:48 [I 2022-10-26 22:18:21] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00247 [I 2022-10-26 22:18:21] ...trap/engines/engine.py.230: loss: 0.17676 [I 2022-10-26 22:18:21] ...optimizers/trijoint.py.151: total_norm: 20.587515 [I 2022-10-26 22:18:27] ...trap/engines/engine.py.224: train: epoch 32 | batch 2120/2304 [I 2022-10-26 22:18:27] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:41 [I 2022-10-26 22:18:27] ...trap/engines/engine.py.229: process: 0.53772 | load: 0.00247 [I 2022-10-26 22:18:27] ...trap/engines/engine.py.230: loss: 0.19783 [I 2022-10-26 22:18:27] ...optimizers/trijoint.py.151: total_norm: 19.551004 [I 2022-10-26 22:18:32] ...trap/engines/engine.py.224: train: epoch 32 | batch 2130/2304 [I 2022-10-26 22:18:32] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:35 [I 2022-10-26 22:18:32] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00245 [I 2022-10-26 22:18:32] ...trap/engines/engine.py.230: loss: 0.16707 [I 2022-10-26 22:18:32] ...optimizers/trijoint.py.151: total_norm: 16.611122 [I 2022-10-26 22:18:38] ...trap/engines/engine.py.224: train: epoch 32 | batch 2140/2304 [I 2022-10-26 22:18:38] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:31 [I 2022-10-26 22:18:38] ...trap/engines/engine.py.229: process: 0.56483 | load: 0.00250 [I 2022-10-26 22:18:38] ...trap/engines/engine.py.230: loss: 0.10116 [I 2022-10-26 22:18:38] ...optimizers/trijoint.py.151: total_norm: 13.980105 [I 2022-10-26 22:18:44] ...trap/engines/engine.py.224: train: epoch 32 | batch 2150/2304 [I 2022-10-26 22:18:44] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:25 [I 2022-10-26 22:18:44] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00243 [I 2022-10-26 22:18:44] ...trap/engines/engine.py.230: loss: 0.19592 [I 2022-10-26 22:18:44] ...optimizers/trijoint.py.151: total_norm: 19.594839 [I 2022-10-26 22:18:49] ...trap/engines/engine.py.224: train: epoch 32 | batch 2160/2304 [I 2022-10-26 22:18:49] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:20 [I 2022-10-26 22:18:49] ...trap/engines/engine.py.229: process: 0.55531 | load: 0.00258 [I 2022-10-26 22:18:49] ...trap/engines/engine.py.230: loss: 0.19349 [I 2022-10-26 22:18:49] ...optimizers/trijoint.py.151: total_norm: 13.269390 [I 2022-10-26 22:18:55] ...trap/engines/engine.py.224: train: epoch 32 | batch 2170/2304 [I 2022-10-26 22:18:55] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:17 [I 2022-10-26 22:18:55] ...trap/engines/engine.py.229: process: 0.65362 | load: 0.00253 [I 2022-10-26 22:18:55] ...trap/engines/engine.py.230: loss: 0.13614 [I 2022-10-26 22:18:55] ...optimizers/trijoint.py.151: total_norm: 13.445544 [I 2022-10-26 22:19:00] ...trap/engines/engine.py.224: train: epoch 32 | batch 2180/2304 [I 2022-10-26 22:19:00] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:08 [I 2022-10-26 22:19:00] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00262 [I 2022-10-26 22:19:00] ...trap/engines/engine.py.230: loss: 0.24644 [I 2022-10-26 22:19:00] ...optimizers/trijoint.py.151: total_norm: 25.821867 [I 2022-10-26 22:19:06] ...trap/engines/engine.py.224: train: epoch 32 | batch 2190/2304 [I 2022-10-26 22:19:06] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:03 [I 2022-10-26 22:19:06] ...trap/engines/engine.py.229: process: 0.56309 | load: 0.00267 [I 2022-10-26 22:19:06] ...trap/engines/engine.py.230: loss: 0.18159 [I 2022-10-26 22:19:06] ...optimizers/trijoint.py.151: total_norm: 29.672733 [I 2022-10-26 22:19:12] ...trap/engines/engine.py.224: train: epoch 32 | batch 2200/2304 [I 2022-10-26 22:19:12] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:57 [I 2022-10-26 22:19:12] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00249 [I 2022-10-26 22:19:12] ...trap/engines/engine.py.230: loss: 0.10095 [I 2022-10-26 22:19:12] ...optimizers/trijoint.py.151: total_norm: 17.228500 [I 2022-10-26 22:19:17] ...trap/engines/engine.py.224: train: epoch 32 | batch 2210/2304 [I 2022-10-26 22:19:17] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:51 [I 2022-10-26 22:19:17] ...trap/engines/engine.py.229: process: 0.53856 | load: 0.00253 [I 2022-10-26 22:19:17] ...trap/engines/engine.py.230: loss: 0.16598 [I 2022-10-26 22:19:17] ...optimizers/trijoint.py.151: total_norm: 16.949331 [I 2022-10-26 22:19:23] ...trap/engines/engine.py.224: train: epoch 32 | batch 2220/2304 [I 2022-10-26 22:19:23] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:46 [I 2022-10-26 22:19:23] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00251 [I 2022-10-26 22:19:23] ...trap/engines/engine.py.230: loss: 0.22608 [I 2022-10-26 22:19:23] ...optimizers/trijoint.py.151: total_norm: 24.339375 [I 2022-10-26 22:19:28] ...trap/engines/engine.py.224: train: epoch 32 | batch 2230/2304 [I 2022-10-26 22:19:28] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:41 [I 2022-10-26 22:19:28] ...trap/engines/engine.py.229: process: 0.54636 | load: 0.00254 [I 2022-10-26 22:19:28] ...trap/engines/engine.py.230: loss: 0.12610 [I 2022-10-26 22:19:28] ...optimizers/trijoint.py.151: total_norm: 12.269269 [I 2022-10-26 22:19:34] ...trap/engines/engine.py.224: train: epoch 32 | batch 2240/2304 [I 2022-10-26 22:19:34] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:35 [I 2022-10-26 22:19:34] ...trap/engines/engine.py.229: process: 0.54479 | load: 0.00259 [I 2022-10-26 22:19:34] ...trap/engines/engine.py.230: loss: 0.23501 [I 2022-10-26 22:19:34] ...optimizers/trijoint.py.151: total_norm: 21.584875 [I 2022-10-26 22:19:39] ...trap/engines/engine.py.224: train: epoch 32 | batch 2250/2304 [I 2022-10-26 22:19:39] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:30 [I 2022-10-26 22:19:39] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00254 [I 2022-10-26 22:19:39] ...trap/engines/engine.py.230: loss: 0.20545 [I 2022-10-26 22:19:39] ...optimizers/trijoint.py.151: total_norm: 30.827801 [I 2022-10-26 22:19:45] ...trap/engines/engine.py.224: train: epoch 32 | batch 2260/2304 [I 2022-10-26 22:19:45] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:24 [I 2022-10-26 22:19:45] ...trap/engines/engine.py.229: process: 0.55152 | load: 0.00250 [I 2022-10-26 22:19:45] ...trap/engines/engine.py.230: loss: 0.14529 [I 2022-10-26 22:19:45] ...optimizers/trijoint.py.151: total_norm: 21.005936 [I 2022-10-26 22:19:51] ...trap/engines/engine.py.224: train: epoch 32 | batch 2270/2304 [I 2022-10-26 22:19:51] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:18 [I 2022-10-26 22:19:51] ...trap/engines/engine.py.229: process: 0.54970 | load: 0.00248 [I 2022-10-26 22:19:51] ...trap/engines/engine.py.230: loss: 0.14924 [I 2022-10-26 22:19:51] ...optimizers/trijoint.py.151: total_norm: 12.692682 [I 2022-10-26 22:19:56] ...trap/engines/engine.py.224: train: epoch 32 | batch 2280/2304 [I 2022-10-26 22:19:56] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:14 [I 2022-10-26 22:19:56] ...trap/engines/engine.py.229: process: 0.63491 | load: 0.00270 [I 2022-10-26 22:19:56] ...trap/engines/engine.py.230: loss: 0.10901 [I 2022-10-26 22:19:56] ...optimizers/trijoint.py.151: total_norm: 12.852916 [I 2022-10-26 22:20:02] ...trap/engines/engine.py.224: train: epoch 32 | batch 2290/2304 [I 2022-10-26 22:20:02] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:07 [I 2022-10-26 22:20:02] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00245 [I 2022-10-26 22:20:02] ...trap/engines/engine.py.230: loss: 0.21814 [I 2022-10-26 22:20:02] ...optimizers/trijoint.py.151: total_norm: 19.123034 [I 2022-10-26 22:20:08] ...trap/engines/engine.py.224: train: epoch 32 | batch 2300/2304 [I 2022-10-26 22:20:08] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:02 [I 2022-10-26 22:20:08] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00016 [I 2022-10-26 22:20:08] ...trap/engines/engine.py.230: loss: 0.17686 [I 2022-10-26 22:20:08] ...optimizers/trijoint.py.151: total_norm: 15.137646 [I 2022-10-26 22:20:10] ...trap/engines/engine.py.224: train: epoch 32 | batch 2304/2304 [I 2022-10-26 22:20:10] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:00 [I 2022-10-26 22:20:10] ...trap/engines/engine.py.229: process: 0.53989 | load: 0.00015 [I 2022-10-26 22:20:10] ...trap/engines/engine.py.230: loss: 0.17274 [I 2022-10-26 22:20:10] ...optimizers/trijoint.py.151: total_norm: 11.864291 [S 2022-10-26 22:20:11] ...trap/engines/engine.py.129: train_epoch.epoch: 32 [S 2022-10-26 22:20:11] ...trap/engines/engine.py.129: train_epoch.loss: 0.17759769542744774 [I 2022-10-26 22:20:11] ...s/criterions/triplet.py.84: margin = 0.21500000000000014 [I 2022-10-26 22:20:12] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 32 [I 2022-10-26 22:20:12] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 22:20:13] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 22:20:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 22:20:13] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 22:20:14] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 22:20:15] ...trap/engines/engine.py.321: eval: epoch 32 | batch 0/510 [I 2022-10-26 22:20:15] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:02 [I 2022-10-26 22:20:15] ...trap/engines/engine.py.326: process: 3.18223 | load: 2.77751 [I 2022-10-26 22:20:18] ...trap/engines/engine.py.321: eval: epoch 32 | batch 10/510 [I 2022-10-26 22:20:18] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:17 [I 2022-10-26 22:20:18] ...trap/engines/engine.py.326: process: 0.32685 | load: 0.00016 [I 2022-10-26 22:20:21] ...trap/engines/engine.py.321: eval: epoch 32 | batch 20/510 [I 2022-10-26 22:20:21] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:58 [I 2022-10-26 22:20:21] ...trap/engines/engine.py.326: process: 0.32250 | load: 0.00017 [I 2022-10-26 22:20:25] ...trap/engines/engine.py.321: eval: epoch 32 | batch 30/510 [I 2022-10-26 22:20:25] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:43 [I 2022-10-26 22:20:25] ...trap/engines/engine.py.326: process: 0.35431 | load: 0.00018 [I 2022-10-26 22:20:28] ...trap/engines/engine.py.321: eval: epoch 32 | batch 40/510 [I 2022-10-26 22:20:28] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:36 [I 2022-10-26 22:20:28] ...trap/engines/engine.py.326: process: 0.33578 | load: 0.00019 [I 2022-10-26 22:20:31] ...trap/engines/engine.py.321: eval: epoch 32 | batch 50/510 [I 2022-10-26 22:20:31] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:38 [I 2022-10-26 22:20:31] ...trap/engines/engine.py.326: process: 0.34452 | load: 0.00017 [I 2022-10-26 22:20:35] ...trap/engines/engine.py.321: eval: epoch 32 | batch 60/510 [I 2022-10-26 22:20:35] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:34 [I 2022-10-26 22:20:35] ...trap/engines/engine.py.326: process: 0.36724 | load: 0.00026 [I 2022-10-26 22:20:38] ...trap/engines/engine.py.321: eval: epoch 32 | batch 70/510 [I 2022-10-26 22:20:38] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:27 [I 2022-10-26 22:20:38] ...trap/engines/engine.py.326: process: 0.33332 | load: 0.00016 [I 2022-10-26 22:20:42] ...trap/engines/engine.py.321: eval: epoch 32 | batch 80/510 [I 2022-10-26 22:20:42] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:24 [I 2022-10-26 22:20:42] ...trap/engines/engine.py.326: process: 0.33382 | load: 0.00020 [I 2022-10-26 22:20:45] ...trap/engines/engine.py.321: eval: epoch 32 | batch 90/510 [I 2022-10-26 22:20:45] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:23 [I 2022-10-26 22:20:45] ...trap/engines/engine.py.326: process: 0.35819 | load: 0.00019 [I 2022-10-26 22:20:48] ...trap/engines/engine.py.321: eval: epoch 32 | batch 100/510 [I 2022-10-26 22:20:48] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 22:20:48] ...trap/engines/engine.py.326: process: 0.29184 | load: 0.00018 [I 2022-10-26 22:20:51] ...trap/engines/engine.py.321: eval: epoch 32 | batch 110/510 [I 2022-10-26 22:20:51] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-26 22:20:51] ...trap/engines/engine.py.326: process: 0.28985 | load: 0.00018 [I 2022-10-26 22:20:54] ...trap/engines/engine.py.321: eval: epoch 32 | batch 120/510 [I 2022-10-26 22:20:54] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-26 22:20:54] ...trap/engines/engine.py.326: process: 0.28695 | load: 0.00018 [I 2022-10-26 22:20:57] ...trap/engines/engine.py.321: eval: epoch 32 | batch 130/510 [I 2022-10-26 22:20:57] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-26 22:20:57] ...trap/engines/engine.py.326: process: 0.28718 | load: 0.00016 [I 2022-10-26 22:21:00] ...trap/engines/engine.py.321: eval: epoch 32 | batch 140/510 [I 2022-10-26 22:21:00] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:48 [I 2022-10-26 22:21:00] ...trap/engines/engine.py.326: process: 0.29673 | load: 0.00019 [I 2022-10-26 22:21:03] ...trap/engines/engine.py.321: eval: epoch 32 | batch 150/510 [I 2022-10-26 22:21:03] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:46 [I 2022-10-26 22:21:03] ...trap/engines/engine.py.326: process: 0.30821 | load: 0.00018 [I 2022-10-26 22:21:06] ...trap/engines/engine.py.321: eval: epoch 32 | batch 160/510 [I 2022-10-26 22:21:06] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:42 [I 2022-10-26 22:21:06] ...trap/engines/engine.py.326: process: 0.29185 | load: 0.00016 [I 2022-10-26 22:21:09] ...trap/engines/engine.py.321: eval: epoch 32 | batch 170/510 [I 2022-10-26 22:21:09] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-26 22:21:09] ...trap/engines/engine.py.326: process: 0.29045 | load: 0.00015 [I 2022-10-26 22:21:12] ...trap/engines/engine.py.321: eval: epoch 32 | batch 180/510 [I 2022-10-26 22:21:12] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-26 22:21:12] ...trap/engines/engine.py.326: process: 0.30065 | load: 0.00014 [I 2022-10-26 22:21:14] ...trap/engines/engine.py.321: eval: epoch 32 | batch 190/510 [I 2022-10-26 22:21:14] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-26 22:21:14] ...trap/engines/engine.py.326: process: 0.29030 | load: 0.00017 [I 2022-10-26 22:21:17] ...trap/engines/engine.py.321: eval: epoch 32 | batch 200/510 [I 2022-10-26 22:21:17] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 22:21:17] ...trap/engines/engine.py.326: process: 0.28270 | load: 0.00015 [I 2022-10-26 22:21:20] ...trap/engines/engine.py.321: eval: epoch 32 | batch 210/510 [I 2022-10-26 22:21:20] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 22:21:20] ...trap/engines/engine.py.326: process: 0.29053 | load: 0.00019 [I 2022-10-26 22:21:23] ...trap/engines/engine.py.321: eval: epoch 32 | batch 220/510 [I 2022-10-26 22:21:23] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 22:21:23] ...trap/engines/engine.py.326: process: 0.28490 | load: 0.00015 [I 2022-10-26 22:21:26] ...trap/engines/engine.py.321: eval: epoch 32 | batch 230/510 [I 2022-10-26 22:21:26] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-26 22:21:26] ...trap/engines/engine.py.326: process: 0.28962 | load: 0.00015 [I 2022-10-26 22:21:29] ...trap/engines/engine.py.321: eval: epoch 32 | batch 240/510 [I 2022-10-26 22:21:29] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 22:21:29] ...trap/engines/engine.py.326: process: 0.29120 | load: 0.00023 [I 2022-10-26 22:21:32] ...trap/engines/engine.py.321: eval: epoch 32 | batch 250/510 [I 2022-10-26 22:21:32] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:16 [I 2022-10-26 22:21:32] ...trap/engines/engine.py.326: process: 0.28865 | load: 0.00019 [I 2022-10-26 22:21:35] ...trap/engines/engine.py.321: eval: epoch 32 | batch 260/510 [I 2022-10-26 22:21:35] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:13 [I 2022-10-26 22:21:35] ...trap/engines/engine.py.326: process: 0.29171 | load: 0.00021 [I 2022-10-26 22:21:38] ...trap/engines/engine.py.321: eval: epoch 32 | batch 270/510 [I 2022-10-26 22:21:38] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-26 22:21:38] ...trap/engines/engine.py.326: process: 0.29151 | load: 0.00018 [I 2022-10-26 22:21:41] ...trap/engines/engine.py.321: eval: epoch 32 | batch 280/510 [I 2022-10-26 22:21:41] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-26 22:21:41] ...trap/engines/engine.py.326: process: 0.28664 | load: 0.00014 [I 2022-10-26 22:21:43] ...trap/engines/engine.py.321: eval: epoch 32 | batch 290/510 [I 2022-10-26 22:21:43] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 22:21:43] ...trap/engines/engine.py.326: process: 0.28615 | load: 0.00019 [I 2022-10-26 22:21:46] ...trap/engines/engine.py.321: eval: epoch 32 | batch 300/510 [I 2022-10-26 22:21:46] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:00:59 [I 2022-10-26 22:21:46] ...trap/engines/engine.py.326: process: 0.28114 | load: 0.00020 [I 2022-10-26 22:21:49] ...trap/engines/engine.py.321: eval: epoch 32 | batch 310/510 [I 2022-10-26 22:21:49] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 22:21:49] ...trap/engines/engine.py.326: process: 0.28900 | load: 0.00019 [I 2022-10-26 22:21:52] ...trap/engines/engine.py.321: eval: epoch 32 | batch 320/510 [I 2022-10-26 22:21:52] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:55 [I 2022-10-26 22:21:52] ...trap/engines/engine.py.326: process: 0.28135 | load: 0.00019 [I 2022-10-26 22:21:55] ...trap/engines/engine.py.321: eval: epoch 32 | batch 330/510 [I 2022-10-26 22:21:55] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:52 [I 2022-10-26 22:21:55] ...trap/engines/engine.py.326: process: 0.28475 | load: 0.00023 [I 2022-10-26 22:21:58] ...trap/engines/engine.py.321: eval: epoch 32 | batch 340/510 [I 2022-10-26 22:21:58] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 22:21:58] ...trap/engines/engine.py.326: process: 0.28957 | load: 0.00021 [I 2022-10-26 22:22:01] ...trap/engines/engine.py.321: eval: epoch 32 | batch 350/510 [I 2022-10-26 22:22:01] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 22:22:01] ...trap/engines/engine.py.326: process: 0.28200 | load: 0.00018 [I 2022-10-26 22:22:04] ...trap/engines/engine.py.321: eval: epoch 32 | batch 360/510 [I 2022-10-26 22:22:04] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 22:22:04] ...trap/engines/engine.py.326: process: 0.28777 | load: 0.00019 [I 2022-10-26 22:22:07] ...trap/engines/engine.py.321: eval: epoch 32 | batch 370/510 [I 2022-10-26 22:22:07] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-26 22:22:07] ...trap/engines/engine.py.326: process: 0.28633 | load: 0.00019 [I 2022-10-26 22:22:10] ...trap/engines/engine.py.321: eval: epoch 32 | batch 380/510 [I 2022-10-26 22:22:10] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-26 22:22:10] ...trap/engines/engine.py.326: process: 0.28433 | load: 0.00016 [I 2022-10-26 22:22:12] ...trap/engines/engine.py.321: eval: epoch 32 | batch 390/510 [I 2022-10-26 22:22:12] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 22:22:12] ...trap/engines/engine.py.326: process: 0.29185 | load: 0.00038 [I 2022-10-26 22:22:15] ...trap/engines/engine.py.321: eval: epoch 32 | batch 400/510 [I 2022-10-26 22:22:15] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 22:22:15] ...trap/engines/engine.py.326: process: 0.28904 | load: 0.00016 [I 2022-10-26 22:22:18] ...trap/engines/engine.py.321: eval: epoch 32 | batch 410/510 [I 2022-10-26 22:22:18] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:29 [I 2022-10-26 22:22:18] ...trap/engines/engine.py.326: process: 0.28813 | load: 0.00020 [I 2022-10-26 22:22:21] ...trap/engines/engine.py.321: eval: epoch 32 | batch 420/510 [I 2022-10-26 22:22:21] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 22:22:21] ...trap/engines/engine.py.326: process: 0.28068 | load: 0.00020 [I 2022-10-26 22:22:24] ...trap/engines/engine.py.321: eval: epoch 32 | batch 430/510 [I 2022-10-26 22:22:24] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 22:22:24] ...trap/engines/engine.py.326: process: 0.28948 | load: 0.00018 [I 2022-10-26 22:22:27] ...trap/engines/engine.py.321: eval: epoch 32 | batch 440/510 [I 2022-10-26 22:22:27] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 22:22:27] ...trap/engines/engine.py.326: process: 0.28939 | load: 0.00017 [I 2022-10-26 22:22:30] ...trap/engines/engine.py.321: eval: epoch 32 | batch 450/510 [I 2022-10-26 22:22:30] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 22:22:30] ...trap/engines/engine.py.326: process: 0.28482 | load: 0.00016 [I 2022-10-26 22:22:33] ...trap/engines/engine.py.321: eval: epoch 32 | batch 460/510 [I 2022-10-26 22:22:33] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-26 22:22:33] ...trap/engines/engine.py.326: process: 0.28347 | load: 0.00020 [I 2022-10-26 22:22:36] ...trap/engines/engine.py.321: eval: epoch 32 | batch 470/510 [I 2022-10-26 22:22:36] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-26 22:22:36] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00020 [I 2022-10-26 22:22:39] ...trap/engines/engine.py.321: eval: epoch 32 | batch 480/510 [I 2022-10-26 22:22:39] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-26 22:22:39] ...trap/engines/engine.py.326: process: 0.29246 | load: 0.00016 [I 2022-10-26 22:22:41] ...trap/engines/engine.py.321: eval: epoch 32 | batch 490/510 [I 2022-10-26 22:22:41] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 22:22:41] ...trap/engines/engine.py.326: process: 0.28770 | load: 0.00020 [I 2022-10-26 22:22:44] ...trap/engines/engine.py.321: eval: epoch 32 | batch 500/510 [I 2022-10-26 22:22:44] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 22:22:44] ...trap/engines/engine.py.326: process: 0.29122 | load: 0.00020 [I 2022-10-26 22:22:47] ...trap/engines/engine.py.321: eval: epoch 32 | batch 510/510 [I 2022-10-26 22:22:47] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 22:22:47] ...trap/engines/engine.py.326: process: 0.28667 | load: 0.00011 [S 2022-10-26 22:22:48] ...trap/engines/engine.py.132: eval_epoch.epoch: 32 [I 2022-10-26 22:22:48] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.663500 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.887300 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.929200 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.647700 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.883300 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.922700 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014243 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008439 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005671 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011437 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008684 [S 2022-10-26 22:22:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006619 [I 2022-10-26 22:22:53] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 22:22:53] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 22:22:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 22:22:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 22:22:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 22:22:54] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 22:22:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 22:23:03] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 22:23:03] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 22:23:03] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 22:23:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 22:23:12] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 22:23:12] ...trap/engines/engine.py.163: Training model on trainset for epoch 33 [I 2022-10-26 22:23:12] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 22:23:15] ...trap/engines/engine.py.224: train: epoch 33 | batch 0/2304 [I 2022-10-26 22:23:15] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:39 [I 2022-10-26 22:23:15] ...trap/engines/engine.py.229: process: 2.64743 | load: 2.01171 [I 2022-10-26 22:23:15] ...trap/engines/engine.py.230: loss: 0.15439 [I 2022-10-26 22:23:15] ...optimizers/trijoint.py.151: total_norm: 12.939202 [I 2022-10-26 22:23:20] ...trap/engines/engine.py.224: train: epoch 33 | batch 10/2304 [I 2022-10-26 22:23:20] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:56 [I 2022-10-26 22:23:20] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00412 [I 2022-10-26 22:23:20] ...trap/engines/engine.py.230: loss: 0.19585 [I 2022-10-26 22:23:20] ...optimizers/trijoint.py.151: total_norm: 37.275146 [I 2022-10-26 22:23:26] ...trap/engines/engine.py.224: train: epoch 33 | batch 20/2304 [I 2022-10-26 22:23:26] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:17 [I 2022-10-26 22:23:26] ...trap/engines/engine.py.229: process: 0.57804 | load: 0.00342 [I 2022-10-26 22:23:26] ...trap/engines/engine.py.230: loss: 0.25071 [I 2022-10-26 22:23:26] ...optimizers/trijoint.py.151: total_norm: 28.628548 [I 2022-10-26 22:23:31] ...trap/engines/engine.py.224: train: epoch 33 | batch 30/2304 [I 2022-10-26 22:23:31] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-26 22:23:31] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00313 [I 2022-10-26 22:23:31] ...trap/engines/engine.py.230: loss: 0.19742 [I 2022-10-26 22:23:31] ...optimizers/trijoint.py.151: total_norm: 18.267710 [I 2022-10-26 22:23:37] ...trap/engines/engine.py.224: train: epoch 33 | batch 40/2304 [I 2022-10-26 22:23:37] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:57 [I 2022-10-26 22:23:37] ...trap/engines/engine.py.229: process: 0.54578 | load: 0.00297 [I 2022-10-26 22:23:37] ...trap/engines/engine.py.230: loss: 0.20631 [I 2022-10-26 22:23:37] ...optimizers/trijoint.py.151: total_norm: 17.362213 [I 2022-10-26 22:23:43] ...trap/engines/engine.py.224: train: epoch 33 | batch 50/2304 [I 2022-10-26 22:23:43] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:52 [I 2022-10-26 22:23:43] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00288 [I 2022-10-26 22:23:43] ...trap/engines/engine.py.230: loss: 0.21997 [I 2022-10-26 22:23:43] ...optimizers/trijoint.py.151: total_norm: 15.732555 [I 2022-10-26 22:23:48] ...trap/engines/engine.py.224: train: epoch 33 | batch 60/2304 [I 2022-10-26 22:23:48] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:40 [I 2022-10-26 22:23:48] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00282 [I 2022-10-26 22:23:48] ...trap/engines/engine.py.230: loss: 0.21341 [I 2022-10-26 22:23:48] ...optimizers/trijoint.py.151: total_norm: 18.347425 [I 2022-10-26 22:23:54] ...trap/engines/engine.py.224: train: epoch 33 | batch 70/2304 [I 2022-10-26 22:23:54] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:29 [I 2022-10-26 22:23:54] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00274 [I 2022-10-26 22:23:54] ...trap/engines/engine.py.230: loss: 0.17547 [I 2022-10-26 22:23:54] ...optimizers/trijoint.py.151: total_norm: 22.793581 [I 2022-10-26 22:23:59] ...trap/engines/engine.py.224: train: epoch 33 | batch 80/2304 [I 2022-10-26 22:23:59] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:32 [I 2022-10-26 22:23:59] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00248 [I 2022-10-26 22:23:59] ...trap/engines/engine.py.230: loss: 0.22556 [I 2022-10-26 22:23:59] ...optimizers/trijoint.py.151: total_norm: 26.404247 [I 2022-10-26 22:24:05] ...trap/engines/engine.py.224: train: epoch 33 | batch 90/2304 [I 2022-10-26 22:24:05] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:25 [I 2022-10-26 22:24:05] ...trap/engines/engine.py.229: process: 0.54514 | load: 0.00306 [I 2022-10-26 22:24:05] ...trap/engines/engine.py.230: loss: 0.11912 [I 2022-10-26 22:24:05] ...optimizers/trijoint.py.151: total_norm: 13.463299 [I 2022-10-26 22:24:10] ...trap/engines/engine.py.224: train: epoch 33 | batch 100/2304 [I 2022-10-26 22:24:10] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:21 [I 2022-10-26 22:24:10] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00301 [I 2022-10-26 22:24:10] ...trap/engines/engine.py.230: loss: 0.21262 [I 2022-10-26 22:24:10] ...optimizers/trijoint.py.151: total_norm: 33.324310 [I 2022-10-26 22:24:16] ...trap/engines/engine.py.224: train: epoch 33 | batch 110/2304 [I 2022-10-26 22:24:16] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:20:23 [I 2022-10-26 22:24:16] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00253 [I 2022-10-26 22:24:16] ...trap/engines/engine.py.230: loss: 0.16717 [I 2022-10-26 22:24:16] ...optimizers/trijoint.py.151: total_norm: 16.532152 [I 2022-10-26 22:24:21] ...trap/engines/engine.py.224: train: epoch 33 | batch 120/2304 [I 2022-10-26 22:24:21] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:16 [I 2022-10-26 22:24:21] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00250 [I 2022-10-26 22:24:21] ...trap/engines/engine.py.230: loss: 0.10770 [I 2022-10-26 22:24:21] ...optimizers/trijoint.py.151: total_norm: 9.785592 [I 2022-10-26 22:24:27] ...trap/engines/engine.py.224: train: epoch 33 | batch 130/2304 [I 2022-10-26 22:24:27] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:10 [I 2022-10-26 22:24:27] ...trap/engines/engine.py.229: process: 0.55507 | load: 0.00266 [I 2022-10-26 22:24:27] ...trap/engines/engine.py.230: loss: 0.15552 [I 2022-10-26 22:24:27] ...optimizers/trijoint.py.151: total_norm: 13.478288 [I 2022-10-26 22:24:33] ...trap/engines/engine.py.224: train: epoch 33 | batch 140/2304 [I 2022-10-26 22:24:33] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:01 [I 2022-10-26 22:24:33] ...trap/engines/engine.py.229: process: 0.55558 | load: 0.00249 [I 2022-10-26 22:24:33] ...trap/engines/engine.py.230: loss: 0.17267 [I 2022-10-26 22:24:33] ...optimizers/trijoint.py.151: total_norm: 19.778711 [I 2022-10-26 22:24:38] ...trap/engines/engine.py.224: train: epoch 33 | batch 150/2304 [I 2022-10-26 22:24:38] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:54 [I 2022-10-26 22:24:38] ...trap/engines/engine.py.229: process: 0.54173 | load: 0.00256 [I 2022-10-26 22:24:38] ...trap/engines/engine.py.230: loss: 0.17734 [I 2022-10-26 22:24:38] ...optimizers/trijoint.py.151: total_norm: 7.432902 [I 2022-10-26 22:24:44] ...trap/engines/engine.py.224: train: epoch 33 | batch 160/2304 [I 2022-10-26 22:24:44] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:56 [I 2022-10-26 22:24:44] ...trap/engines/engine.py.229: process: 0.54886 | load: 0.00267 [I 2022-10-26 22:24:44] ...trap/engines/engine.py.230: loss: 0.14371 [I 2022-10-26 22:24:44] ...optimizers/trijoint.py.151: total_norm: 14.361157 [I 2022-10-26 22:24:49] ...trap/engines/engine.py.224: train: epoch 33 | batch 170/2304 [I 2022-10-26 22:24:49] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:46 [I 2022-10-26 22:24:49] ...trap/engines/engine.py.229: process: 0.55020 | load: 0.00244 [I 2022-10-26 22:24:49] ...trap/engines/engine.py.230: loss: 0.19848 [I 2022-10-26 22:24:49] ...optimizers/trijoint.py.151: total_norm: 28.859430 [I 2022-10-26 22:24:55] ...trap/engines/engine.py.224: train: epoch 33 | batch 180/2304 [I 2022-10-26 22:24:55] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:19:34 [I 2022-10-26 22:24:55] ...trap/engines/engine.py.229: process: 0.55408 | load: 0.00242 [I 2022-10-26 22:24:55] ...trap/engines/engine.py.230: loss: 0.18720 [I 2022-10-26 22:24:55] ...optimizers/trijoint.py.151: total_norm: 16.921894 [I 2022-10-26 22:25:00] ...trap/engines/engine.py.224: train: epoch 33 | batch 190/2304 [I 2022-10-26 22:25:00] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:48 [I 2022-10-26 22:25:00] ...trap/engines/engine.py.229: process: 0.56418 | load: 0.00259 [I 2022-10-26 22:25:00] ...trap/engines/engine.py.230: loss: 0.15913 [I 2022-10-26 22:25:00] ...optimizers/trijoint.py.151: total_norm: 13.360049 [I 2022-10-26 22:25:06] ...trap/engines/engine.py.224: train: epoch 33 | batch 200/2304 [I 2022-10-26 22:25:06] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:31 [I 2022-10-26 22:25:06] ...trap/engines/engine.py.229: process: 0.55793 | load: 0.00240 [I 2022-10-26 22:25:06] ...trap/engines/engine.py.230: loss: 0.16797 [I 2022-10-26 22:25:06] ...optimizers/trijoint.py.151: total_norm: 19.334812 [I 2022-10-26 22:25:12] ...trap/engines/engine.py.224: train: epoch 33 | batch 210/2304 [I 2022-10-26 22:25:12] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:28 [I 2022-10-26 22:25:12] ...trap/engines/engine.py.229: process: 0.56695 | load: 0.00242 [I 2022-10-26 22:25:12] ...trap/engines/engine.py.230: loss: 0.13504 [I 2022-10-26 22:25:12] ...optimizers/trijoint.py.151: total_norm: 13.894092 [I 2022-10-26 22:25:17] ...trap/engines/engine.py.224: train: epoch 33 | batch 220/2304 [I 2022-10-26 22:25:17] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:15 [I 2022-10-26 22:25:17] ...trap/engines/engine.py.229: process: 0.56198 | load: 0.00268 [I 2022-10-26 22:25:17] ...trap/engines/engine.py.230: loss: 0.14943 [I 2022-10-26 22:25:17] ...optimizers/trijoint.py.151: total_norm: 14.598802 [I 2022-10-26 22:25:23] ...trap/engines/engine.py.224: train: epoch 33 | batch 230/2304 [I 2022-10-26 22:25:23] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:02 [I 2022-10-26 22:25:23] ...trap/engines/engine.py.229: process: 0.54469 | load: 0.00260 [I 2022-10-26 22:25:23] ...trap/engines/engine.py.230: loss: 0.29498 [I 2022-10-26 22:25:23] ...optimizers/trijoint.py.151: total_norm: 20.318472 [I 2022-10-26 22:25:28] ...trap/engines/engine.py.224: train: epoch 33 | batch 240/2304 [I 2022-10-26 22:25:28] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:09 [I 2022-10-26 22:25:28] ...trap/engines/engine.py.229: process: 0.56665 | load: 0.00296 [I 2022-10-26 22:25:28] ...trap/engines/engine.py.230: loss: 0.17640 [I 2022-10-26 22:25:28] ...optimizers/trijoint.py.151: total_norm: 19.305861 [I 2022-10-26 22:25:34] ...trap/engines/engine.py.224: train: epoch 33 | batch 250/2304 [I 2022-10-26 22:25:34] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:19:08 [I 2022-10-26 22:25:34] ...trap/engines/engine.py.229: process: 0.56738 | load: 0.00252 [I 2022-10-26 22:25:34] ...trap/engines/engine.py.230: loss: 0.11325 [I 2022-10-26 22:25:34] ...optimizers/trijoint.py.151: total_norm: 13.090734 [I 2022-10-26 22:25:40] ...trap/engines/engine.py.224: train: epoch 33 | batch 260/2304 [I 2022-10-26 22:25:40] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:07 [I 2022-10-26 22:25:40] ...trap/engines/engine.py.229: process: 0.55822 | load: 0.00249 [I 2022-10-26 22:25:40] ...trap/engines/engine.py.230: loss: 0.14710 [I 2022-10-26 22:25:40] ...optimizers/trijoint.py.151: total_norm: 22.393761 [I 2022-10-26 22:25:45] ...trap/engines/engine.py.224: train: epoch 33 | batch 270/2304 [I 2022-10-26 22:25:45] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:29 [I 2022-10-26 22:25:45] ...trap/engines/engine.py.229: process: 0.63041 | load: 0.00250 [I 2022-10-26 22:25:45] ...trap/engines/engine.py.230: loss: 0.20468 [I 2022-10-26 22:25:45] ...optimizers/trijoint.py.151: total_norm: 15.085017 [I 2022-10-26 22:25:51] ...trap/engines/engine.py.224: train: epoch 33 | batch 280/2304 [I 2022-10-26 22:25:51] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:57 [I 2022-10-26 22:25:51] ...trap/engines/engine.py.229: process: 0.55509 | load: 0.00245 [I 2022-10-26 22:25:51] ...trap/engines/engine.py.230: loss: 0.21680 [I 2022-10-26 22:25:51] ...optimizers/trijoint.py.151: total_norm: 32.240257 [I 2022-10-26 22:25:57] ...trap/engines/engine.py.224: train: epoch 33 | batch 290/2304 [I 2022-10-26 22:25:57] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:51 [I 2022-10-26 22:25:57] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00251 [I 2022-10-26 22:25:57] ...trap/engines/engine.py.230: loss: 0.13709 [I 2022-10-26 22:25:57] ...optimizers/trijoint.py.151: total_norm: 15.883561 [I 2022-10-26 22:26:02] ...trap/engines/engine.py.224: train: epoch 33 | batch 300/2304 [I 2022-10-26 22:26:02] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:40 [I 2022-10-26 22:26:02] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00253 [I 2022-10-26 22:26:02] ...trap/engines/engine.py.230: loss: 0.24640 [I 2022-10-26 22:26:02] ...optimizers/trijoint.py.151: total_norm: 24.146494 [I 2022-10-26 22:26:08] ...trap/engines/engine.py.224: train: epoch 33 | batch 310/2304 [I 2022-10-26 22:26:08] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:34 [I 2022-10-26 22:26:08] ...trap/engines/engine.py.229: process: 0.55770 | load: 0.00245 [I 2022-10-26 22:26:08] ...trap/engines/engine.py.230: loss: 0.17971 [I 2022-10-26 22:26:08] ...optimizers/trijoint.py.151: total_norm: 27.058310 [I 2022-10-26 22:26:13] ...trap/engines/engine.py.224: train: epoch 33 | batch 320/2304 [I 2022-10-26 22:26:13] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:30 [I 2022-10-26 22:26:13] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00252 [I 2022-10-26 22:26:13] ...trap/engines/engine.py.230: loss: 0.19766 [I 2022-10-26 22:26:13] ...optimizers/trijoint.py.151: total_norm: 10.393871 [I 2022-10-26 22:26:19] ...trap/engines/engine.py.224: train: epoch 33 | batch 330/2304 [I 2022-10-26 22:26:19] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:24 [I 2022-10-26 22:26:19] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00246 [I 2022-10-26 22:26:19] ...trap/engines/engine.py.230: loss: 0.11946 [I 2022-10-26 22:26:19] ...optimizers/trijoint.py.151: total_norm: 18.223783 [I 2022-10-26 22:26:25] ...trap/engines/engine.py.224: train: epoch 33 | batch 340/2304 [I 2022-10-26 22:26:25] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:31 [I 2022-10-26 22:26:25] ...trap/engines/engine.py.229: process: 0.55224 | load: 0.00246 [I 2022-10-26 22:26:25] ...trap/engines/engine.py.230: loss: 0.11756 [I 2022-10-26 22:26:25] ...optimizers/trijoint.py.151: total_norm: 9.926324 [I 2022-10-26 22:26:30] ...trap/engines/engine.py.224: train: epoch 33 | batch 350/2304 [I 2022-10-26 22:26:30] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:17:59 [I 2022-10-26 22:26:30] ...trap/engines/engine.py.229: process: 0.54174 | load: 0.00247 [I 2022-10-26 22:26:30] ...trap/engines/engine.py.230: loss: 0.13746 [I 2022-10-26 22:26:30] ...optimizers/trijoint.py.151: total_norm: 8.684766 [I 2022-10-26 22:26:36] ...trap/engines/engine.py.224: train: epoch 33 | batch 360/2304 [I 2022-10-26 22:26:36] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:08 [I 2022-10-26 22:26:36] ...trap/engines/engine.py.229: process: 0.54568 | load: 0.00248 [I 2022-10-26 22:26:36] ...trap/engines/engine.py.230: loss: 0.15041 [I 2022-10-26 22:26:36] ...optimizers/trijoint.py.151: total_norm: 16.722103 [I 2022-10-26 22:26:41] ...trap/engines/engine.py.224: train: epoch 33 | batch 370/2304 [I 2022-10-26 22:26:41] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:57 [I 2022-10-26 22:26:41] ...trap/engines/engine.py.229: process: 0.55378 | load: 0.00280 [I 2022-10-26 22:26:41] ...trap/engines/engine.py.230: loss: 0.17568 [I 2022-10-26 22:26:41] ...optimizers/trijoint.py.151: total_norm: 30.236502 [I 2022-10-26 22:26:47] ...trap/engines/engine.py.224: train: epoch 33 | batch 380/2304 [I 2022-10-26 22:26:47] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:51 [I 2022-10-26 22:26:47] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00255 [I 2022-10-26 22:26:47] ...trap/engines/engine.py.230: loss: 0.15098 [I 2022-10-26 22:26:47] ...optimizers/trijoint.py.151: total_norm: 28.545883 [I 2022-10-26 22:26:53] ...trap/engines/engine.py.224: train: epoch 33 | batch 390/2304 [I 2022-10-26 22:26:53] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:43 [I 2022-10-26 22:26:53] ...trap/engines/engine.py.229: process: 0.56497 | load: 0.00250 [I 2022-10-26 22:26:53] ...trap/engines/engine.py.230: loss: 0.18423 [I 2022-10-26 22:26:53] ...optimizers/trijoint.py.151: total_norm: 18.656141 [I 2022-10-26 22:26:58] ...trap/engines/engine.py.224: train: epoch 33 | batch 400/2304 [I 2022-10-26 22:26:58] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:47 [I 2022-10-26 22:26:58] ...trap/engines/engine.py.229: process: 0.56465 | load: 0.00242 [I 2022-10-26 22:26:58] ...trap/engines/engine.py.230: loss: 0.12941 [I 2022-10-26 22:26:58] ...optimizers/trijoint.py.151: total_norm: 23.128916 [I 2022-10-26 22:27:04] ...trap/engines/engine.py.224: train: epoch 33 | batch 410/2304 [I 2022-10-26 22:27:04] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:47 [I 2022-10-26 22:27:04] ...trap/engines/engine.py.229: process: 0.54608 | load: 0.00248 [I 2022-10-26 22:27:04] ...trap/engines/engine.py.230: loss: 0.17551 [I 2022-10-26 22:27:04] ...optimizers/trijoint.py.151: total_norm: 14.567990 [I 2022-10-26 22:27:09] ...trap/engines/engine.py.224: train: epoch 33 | batch 420/2304 [I 2022-10-26 22:27:09] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:30 [I 2022-10-26 22:27:09] ...trap/engines/engine.py.229: process: 0.54508 | load: 0.00243 [I 2022-10-26 22:27:09] ...trap/engines/engine.py.230: loss: 0.25721 [I 2022-10-26 22:27:09] ...optimizers/trijoint.py.151: total_norm: 35.292297 [I 2022-10-26 22:27:15] ...trap/engines/engine.py.224: train: epoch 33 | batch 430/2304 [I 2022-10-26 22:27:15] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:24 [I 2022-10-26 22:27:15] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00241 [I 2022-10-26 22:27:15] ...trap/engines/engine.py.230: loss: 0.14952 [I 2022-10-26 22:27:15] ...optimizers/trijoint.py.151: total_norm: 14.631784 [I 2022-10-26 22:27:21] ...trap/engines/engine.py.224: train: epoch 33 | batch 440/2304 [I 2022-10-26 22:27:21] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:15 [I 2022-10-26 22:27:21] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00243 [I 2022-10-26 22:27:21] ...trap/engines/engine.py.230: loss: 0.23600 [I 2022-10-26 22:27:21] ...optimizers/trijoint.py.151: total_norm: 18.053549 [I 2022-10-26 22:27:26] ...trap/engines/engine.py.224: train: epoch 33 | batch 450/2304 [I 2022-10-26 22:27:26] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:02 [I 2022-10-26 22:27:26] ...trap/engines/engine.py.229: process: 0.54903 | load: 0.00251 [I 2022-10-26 22:27:26] ...trap/engines/engine.py.230: loss: 0.20093 [I 2022-10-26 22:27:26] ...optimizers/trijoint.py.151: total_norm: 24.672482 [I 2022-10-26 22:27:32] ...trap/engines/engine.py.224: train: epoch 33 | batch 460/2304 [I 2022-10-26 22:27:32] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:06 [I 2022-10-26 22:27:32] ...trap/engines/engine.py.229: process: 0.55410 | load: 0.00247 [I 2022-10-26 22:27:32] ...trap/engines/engine.py.230: loss: 0.17231 [I 2022-10-26 22:27:32] ...optimizers/trijoint.py.151: total_norm: 17.313744 [I 2022-10-26 22:27:37] ...trap/engines/engine.py.224: train: epoch 33 | batch 470/2304 [I 2022-10-26 22:27:37] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:04 [I 2022-10-26 22:27:37] ...trap/engines/engine.py.229: process: 0.57060 | load: 0.00243 [I 2022-10-26 22:27:37] ...trap/engines/engine.py.230: loss: 0.18455 [I 2022-10-26 22:27:37] ...optimizers/trijoint.py.151: total_norm: 12.143372 [I 2022-10-26 22:27:43] ...trap/engines/engine.py.224: train: epoch 33 | batch 480/2304 [I 2022-10-26 22:27:43] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:57 [I 2022-10-26 22:27:43] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00243 [I 2022-10-26 22:27:43] ...trap/engines/engine.py.230: loss: 0.16761 [I 2022-10-26 22:27:43] ...optimizers/trijoint.py.151: total_norm: 12.337947 [I 2022-10-26 22:27:49] ...trap/engines/engine.py.224: train: epoch 33 | batch 490/2304 [I 2022-10-26 22:27:49] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:49 [I 2022-10-26 22:27:49] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00605 [I 2022-10-26 22:27:49] ...trap/engines/engine.py.230: loss: 0.21081 [I 2022-10-26 22:27:49] ...optimizers/trijoint.py.151: total_norm: 13.515888 [I 2022-10-26 22:27:54] ...trap/engines/engine.py.224: train: epoch 33 | batch 500/2304 [I 2022-10-26 22:27:54] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:36 [I 2022-10-26 22:27:54] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00244 [I 2022-10-26 22:27:54] ...trap/engines/engine.py.230: loss: 0.20747 [I 2022-10-26 22:27:54] ...optimizers/trijoint.py.151: total_norm: 18.838024 [I 2022-10-26 22:28:00] ...trap/engines/engine.py.224: train: epoch 33 | batch 510/2304 [I 2022-10-26 22:28:00] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:55 [I 2022-10-26 22:28:00] ...trap/engines/engine.py.229: process: 0.55059 | load: 0.00243 [I 2022-10-26 22:28:00] ...trap/engines/engine.py.230: loss: 0.23050 [I 2022-10-26 22:28:00] ...optimizers/trijoint.py.151: total_norm: 17.545530 [I 2022-10-26 22:28:05] ...trap/engines/engine.py.224: train: epoch 33 | batch 520/2304 [I 2022-10-26 22:28:05] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:29 [I 2022-10-26 22:28:05] ...trap/engines/engine.py.229: process: 0.55129 | load: 0.00246 [I 2022-10-26 22:28:05] ...trap/engines/engine.py.230: loss: 0.15324 [I 2022-10-26 22:28:05] ...optimizers/trijoint.py.151: total_norm: 17.821157 [I 2022-10-26 22:28:11] ...trap/engines/engine.py.224: train: epoch 33 | batch 530/2304 [I 2022-10-26 22:28:11] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:28 [I 2022-10-26 22:28:11] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00244 [I 2022-10-26 22:28:11] ...trap/engines/engine.py.230: loss: 0.27601 [I 2022-10-26 22:28:11] ...optimizers/trijoint.py.151: total_norm: 23.037331 [I 2022-10-26 22:28:16] ...trap/engines/engine.py.224: train: epoch 33 | batch 540/2304 [I 2022-10-26 22:28:16] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:25 [I 2022-10-26 22:28:16] ...trap/engines/engine.py.229: process: 0.56166 | load: 0.00243 [I 2022-10-26 22:28:16] ...trap/engines/engine.py.230: loss: 0.18441 [I 2022-10-26 22:28:16] ...optimizers/trijoint.py.151: total_norm: 16.568806 [I 2022-10-26 22:28:22] ...trap/engines/engine.py.224: train: epoch 33 | batch 550/2304 [I 2022-10-26 22:28:22] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:19 [I 2022-10-26 22:28:22] ...trap/engines/engine.py.229: process: 0.55481 | load: 0.00247 [I 2022-10-26 22:28:22] ...trap/engines/engine.py.230: loss: 0.16365 [I 2022-10-26 22:28:22] ...optimizers/trijoint.py.151: total_norm: 13.912758 [I 2022-10-26 22:28:28] ...trap/engines/engine.py.224: train: epoch 33 | batch 560/2304 [I 2022-10-26 22:28:28] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:15 [I 2022-10-26 22:28:28] ...trap/engines/engine.py.229: process: 0.54976 | load: 0.00250 [I 2022-10-26 22:28:28] ...trap/engines/engine.py.230: loss: 0.21832 [I 2022-10-26 22:28:28] ...optimizers/trijoint.py.151: total_norm: 20.484276 [I 2022-10-26 22:28:33] ...trap/engines/engine.py.224: train: epoch 33 | batch 570/2304 [I 2022-10-26 22:28:33] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:09 [I 2022-10-26 22:28:33] ...trap/engines/engine.py.229: process: 0.56325 | load: 0.00249 [I 2022-10-26 22:28:33] ...trap/engines/engine.py.230: loss: 0.16130 [I 2022-10-26 22:28:33] ...optimizers/trijoint.py.151: total_norm: 18.460192 [I 2022-10-26 22:28:39] ...trap/engines/engine.py.224: train: epoch 33 | batch 580/2304 [I 2022-10-26 22:28:39] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:01 [I 2022-10-26 22:28:39] ...trap/engines/engine.py.229: process: 0.55259 | load: 0.00241 [I 2022-10-26 22:28:39] ...trap/engines/engine.py.230: loss: 0.11671 [I 2022-10-26 22:28:39] ...optimizers/trijoint.py.151: total_norm: 9.135262 [I 2022-10-26 22:28:44] ...trap/engines/engine.py.224: train: epoch 33 | batch 590/2304 [I 2022-10-26 22:28:44] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:16:01 [I 2022-10-26 22:28:44] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00246 [I 2022-10-26 22:28:44] ...trap/engines/engine.py.230: loss: 0.15148 [I 2022-10-26 22:28:44] ...optimizers/trijoint.py.151: total_norm: 14.887302 [I 2022-10-26 22:28:50] ...trap/engines/engine.py.224: train: epoch 33 | batch 600/2304 [I 2022-10-26 22:28:50] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:51 [I 2022-10-26 22:28:50] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00245 [I 2022-10-26 22:28:50] ...trap/engines/engine.py.230: loss: 0.29911 [I 2022-10-26 22:28:50] ...optimizers/trijoint.py.151: total_norm: 18.959917 [I 2022-10-26 22:28:56] ...trap/engines/engine.py.224: train: epoch 33 | batch 610/2304 [I 2022-10-26 22:28:56] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:37 [I 2022-10-26 22:28:56] ...trap/engines/engine.py.229: process: 0.54776 | load: 0.00245 [I 2022-10-26 22:28:56] ...trap/engines/engine.py.230: loss: 0.17531 [I 2022-10-26 22:28:56] ...optimizers/trijoint.py.151: total_norm: 14.701682 [I 2022-10-26 22:29:01] ...trap/engines/engine.py.224: train: epoch 33 | batch 620/2304 [I 2022-10-26 22:29:01] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:43 [I 2022-10-26 22:29:01] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00242 [I 2022-10-26 22:29:01] ...trap/engines/engine.py.230: loss: 0.25301 [I 2022-10-26 22:29:01] ...optimizers/trijoint.py.151: total_norm: 18.398836 [I 2022-10-26 22:29:07] ...trap/engines/engine.py.224: train: epoch 33 | batch 630/2304 [I 2022-10-26 22:29:07] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:37 [I 2022-10-26 22:29:07] ...trap/engines/engine.py.229: process: 0.56538 | load: 0.00244 [I 2022-10-26 22:29:07] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-26 22:29:07] ...optimizers/trijoint.py.151: total_norm: 28.970678 [I 2022-10-26 22:29:12] ...trap/engines/engine.py.224: train: epoch 33 | batch 640/2304 [I 2022-10-26 22:29:12] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:29 [I 2022-10-26 22:29:12] ...trap/engines/engine.py.229: process: 0.56554 | load: 0.00241 [I 2022-10-26 22:29:12] ...trap/engines/engine.py.230: loss: 0.20826 [I 2022-10-26 22:29:12] ...optimizers/trijoint.py.151: total_norm: 24.644392 [I 2022-10-26 22:29:18] ...trap/engines/engine.py.224: train: epoch 33 | batch 650/2304 [I 2022-10-26 22:29:18] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:28 [I 2022-10-26 22:29:18] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00240 [I 2022-10-26 22:29:18] ...trap/engines/engine.py.230: loss: 0.26633 [I 2022-10-26 22:29:18] ...optimizers/trijoint.py.151: total_norm: 16.316147 [I 2022-10-26 22:29:24] ...trap/engines/engine.py.224: train: epoch 33 | batch 660/2304 [I 2022-10-26 22:29:24] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:15 [I 2022-10-26 22:29:24] ...trap/engines/engine.py.229: process: 0.54501 | load: 0.00253 [I 2022-10-26 22:29:24] ...trap/engines/engine.py.230: loss: 0.23950 [I 2022-10-26 22:29:24] ...optimizers/trijoint.py.151: total_norm: 37.337269 [I 2022-10-26 22:29:29] ...trap/engines/engine.py.224: train: epoch 33 | batch 670/2304 [I 2022-10-26 22:29:29] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:13 [I 2022-10-26 22:29:29] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00248 [I 2022-10-26 22:29:29] ...trap/engines/engine.py.230: loss: 0.17999 [I 2022-10-26 22:29:29] ...optimizers/trijoint.py.151: total_norm: 22.330095 [I 2022-10-26 22:29:35] ...trap/engines/engine.py.224: train: epoch 33 | batch 680/2304 [I 2022-10-26 22:29:35] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:09 [I 2022-10-26 22:29:35] ...trap/engines/engine.py.229: process: 0.56538 | load: 0.00246 [I 2022-10-26 22:29:35] ...trap/engines/engine.py.230: loss: 0.14622 [I 2022-10-26 22:29:35] ...optimizers/trijoint.py.151: total_norm: 8.632361 [I 2022-10-26 22:29:40] ...trap/engines/engine.py.224: train: epoch 33 | batch 690/2304 [I 2022-10-26 22:29:40] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:59 [I 2022-10-26 22:29:40] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00246 [I 2022-10-26 22:29:40] ...trap/engines/engine.py.230: loss: 0.16406 [I 2022-10-26 22:29:40] ...optimizers/trijoint.py.151: total_norm: 17.853647 [I 2022-10-26 22:29:46] ...trap/engines/engine.py.224: train: epoch 33 | batch 700/2304 [I 2022-10-26 22:29:46] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:56 [I 2022-10-26 22:29:46] ...trap/engines/engine.py.229: process: 0.54570 | load: 0.00248 [I 2022-10-26 22:29:46] ...trap/engines/engine.py.230: loss: 0.18398 [I 2022-10-26 22:29:46] ...optimizers/trijoint.py.151: total_norm: 15.934665 [I 2022-10-26 22:29:52] ...trap/engines/engine.py.224: train: epoch 33 | batch 710/2304 [I 2022-10-26 22:29:52] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:45 [I 2022-10-26 22:29:52] ...trap/engines/engine.py.229: process: 0.54982 | load: 0.00245 [I 2022-10-26 22:29:52] ...trap/engines/engine.py.230: loss: 0.24441 [I 2022-10-26 22:29:52] ...optimizers/trijoint.py.151: total_norm: 19.491989 [I 2022-10-26 22:29:57] ...trap/engines/engine.py.224: train: epoch 33 | batch 720/2304 [I 2022-10-26 22:29:57] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:41 [I 2022-10-26 22:29:57] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00245 [I 2022-10-26 22:29:57] ...trap/engines/engine.py.230: loss: 0.15771 [I 2022-10-26 22:29:57] ...optimizers/trijoint.py.151: total_norm: 9.256037 [I 2022-10-26 22:30:03] ...trap/engines/engine.py.224: train: epoch 33 | batch 730/2304 [I 2022-10-26 22:30:03] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:41 [I 2022-10-26 22:30:03] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00244 [I 2022-10-26 22:30:03] ...trap/engines/engine.py.230: loss: 0.26323 [I 2022-10-26 22:30:03] ...optimizers/trijoint.py.151: total_norm: 34.226528 [I 2022-10-26 22:30:08] ...trap/engines/engine.py.224: train: epoch 33 | batch 740/2304 [I 2022-10-26 22:30:08] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:30 [I 2022-10-26 22:30:08] ...trap/engines/engine.py.229: process: 0.56207 | load: 0.00244 [I 2022-10-26 22:30:08] ...trap/engines/engine.py.230: loss: 0.16189 [I 2022-10-26 22:30:08] ...optimizers/trijoint.py.151: total_norm: 19.014536 [I 2022-10-26 22:30:14] ...trap/engines/engine.py.224: train: epoch 33 | batch 750/2304 [I 2022-10-26 22:30:14] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:24 [I 2022-10-26 22:30:14] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00245 [I 2022-10-26 22:30:14] ...trap/engines/engine.py.230: loss: 0.17107 [I 2022-10-26 22:30:14] ...optimizers/trijoint.py.151: total_norm: 21.758955 [I 2022-10-26 22:30:20] ...trap/engines/engine.py.224: train: epoch 33 | batch 760/2304 [I 2022-10-26 22:30:20] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:46 [I 2022-10-26 22:30:20] ...trap/engines/engine.py.229: process: 0.54624 | load: 0.00245 [I 2022-10-26 22:30:20] ...trap/engines/engine.py.230: loss: 0.15353 [I 2022-10-26 22:30:20] ...optimizers/trijoint.py.151: total_norm: 21.449806 [I 2022-10-26 22:30:25] ...trap/engines/engine.py.224: train: epoch 33 | batch 770/2304 [I 2022-10-26 22:30:25] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:13 [I 2022-10-26 22:30:25] ...trap/engines/engine.py.229: process: 0.54444 | load: 0.00244 [I 2022-10-26 22:30:25] ...trap/engines/engine.py.230: loss: 0.15989 [I 2022-10-26 22:30:25] ...optimizers/trijoint.py.151: total_norm: 15.527745 [I 2022-10-26 22:30:31] ...trap/engines/engine.py.224: train: epoch 33 | batch 780/2304 [I 2022-10-26 22:30:31] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:09 [I 2022-10-26 22:30:31] ...trap/engines/engine.py.229: process: 0.56476 | load: 0.00252 [I 2022-10-26 22:30:31] ...trap/engines/engine.py.230: loss: 0.15480 [I 2022-10-26 22:30:31] ...optimizers/trijoint.py.151: total_norm: 15.544721 [I 2022-10-26 22:30:36] ...trap/engines/engine.py.224: train: epoch 33 | batch 790/2304 [I 2022-10-26 22:30:36] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:06 [I 2022-10-26 22:30:36] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00243 [I 2022-10-26 22:30:36] ...trap/engines/engine.py.230: loss: 0.15093 [I 2022-10-26 22:30:36] ...optimizers/trijoint.py.151: total_norm: 13.111409 [I 2022-10-26 22:30:42] ...trap/engines/engine.py.224: train: epoch 33 | batch 800/2304 [I 2022-10-26 22:30:42] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:02 [I 2022-10-26 22:30:42] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00245 [I 2022-10-26 22:30:42] ...trap/engines/engine.py.230: loss: 0.18995 [I 2022-10-26 22:30:42] ...optimizers/trijoint.py.151: total_norm: 15.612616 [I 2022-10-26 22:30:48] ...trap/engines/engine.py.224: train: epoch 33 | batch 810/2304 [I 2022-10-26 22:30:48] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:50 [I 2022-10-26 22:30:48] ...trap/engines/engine.py.229: process: 0.55315 | load: 0.00240 [I 2022-10-26 22:30:48] ...trap/engines/engine.py.230: loss: 0.15967 [I 2022-10-26 22:30:48] ...optimizers/trijoint.py.151: total_norm: 41.719467 [I 2022-10-26 22:30:53] ...trap/engines/engine.py.224: train: epoch 33 | batch 820/2304 [I 2022-10-26 22:30:53] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:46 [I 2022-10-26 22:30:53] ...trap/engines/engine.py.229: process: 0.56042 | load: 0.00241 [I 2022-10-26 22:30:53] ...trap/engines/engine.py.230: loss: 0.21570 [I 2022-10-26 22:30:53] ...optimizers/trijoint.py.151: total_norm: 19.219290 [I 2022-10-26 22:30:59] ...trap/engines/engine.py.224: train: epoch 33 | batch 830/2304 [I 2022-10-26 22:30:59] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:38 [I 2022-10-26 22:30:59] ...trap/engines/engine.py.229: process: 0.54996 | load: 0.00242 [I 2022-10-26 22:30:59] ...trap/engines/engine.py.230: loss: 0.19562 [I 2022-10-26 22:30:59] ...optimizers/trijoint.py.151: total_norm: 13.812294 [I 2022-10-26 22:31:04] ...trap/engines/engine.py.224: train: epoch 33 | batch 840/2304 [I 2022-10-26 22:31:04] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:33 [I 2022-10-26 22:31:04] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00244 [I 2022-10-26 22:31:04] ...trap/engines/engine.py.230: loss: 0.21334 [I 2022-10-26 22:31:04] ...optimizers/trijoint.py.151: total_norm: 20.893164 [I 2022-10-26 22:31:10] ...trap/engines/engine.py.224: train: epoch 33 | batch 850/2304 [I 2022-10-26 22:31:10] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:23 [I 2022-10-26 22:31:10] ...trap/engines/engine.py.229: process: 0.54689 | load: 0.00246 [I 2022-10-26 22:31:10] ...trap/engines/engine.py.230: loss: 0.21542 [I 2022-10-26 22:31:10] ...optimizers/trijoint.py.151: total_norm: 15.386591 [I 2022-10-26 22:31:15] ...trap/engines/engine.py.224: train: epoch 33 | batch 860/2304 [I 2022-10-26 22:31:15] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:16 [I 2022-10-26 22:31:15] ...trap/engines/engine.py.229: process: 0.55049 | load: 0.00242 [I 2022-10-26 22:31:15] ...trap/engines/engine.py.230: loss: 0.15236 [I 2022-10-26 22:31:15] ...optimizers/trijoint.py.151: total_norm: 13.347700 [I 2022-10-26 22:31:21] ...trap/engines/engine.py.224: train: epoch 33 | batch 870/2304 [I 2022-10-26 22:31:21] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:16 [I 2022-10-26 22:31:21] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00251 [I 2022-10-26 22:31:21] ...trap/engines/engine.py.230: loss: 0.15517 [I 2022-10-26 22:31:21] ...optimizers/trijoint.py.151: total_norm: 25.233498 [I 2022-10-26 22:31:27] ...trap/engines/engine.py.224: train: epoch 33 | batch 880/2304 [I 2022-10-26 22:31:27] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:10 [I 2022-10-26 22:31:27] ...trap/engines/engine.py.229: process: 0.55968 | load: 0.00244 [I 2022-10-26 22:31:27] ...trap/engines/engine.py.230: loss: 0.14195 [I 2022-10-26 22:31:27] ...optimizers/trijoint.py.151: total_norm: 16.193960 [I 2022-10-26 22:31:32] ...trap/engines/engine.py.224: train: epoch 33 | batch 890/2304 [I 2022-10-26 22:31:32] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:10 [I 2022-10-26 22:31:32] ...trap/engines/engine.py.229: process: 0.55414 | load: 0.00244 [I 2022-10-26 22:31:32] ...trap/engines/engine.py.230: loss: 0.18897 [I 2022-10-26 22:31:32] ...optimizers/trijoint.py.151: total_norm: 30.792738 [I 2022-10-26 22:31:38] ...trap/engines/engine.py.224: train: epoch 33 | batch 900/2304 [I 2022-10-26 22:31:38] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:12:59 [I 2022-10-26 22:31:38] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00247 [I 2022-10-26 22:31:38] ...trap/engines/engine.py.230: loss: 0.16996 [I 2022-10-26 22:31:38] ...optimizers/trijoint.py.151: total_norm: 31.251183 [I 2022-10-26 22:31:43] ...trap/engines/engine.py.224: train: epoch 33 | batch 910/2304 [I 2022-10-26 22:31:43] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:54 [I 2022-10-26 22:31:43] ...trap/engines/engine.py.229: process: 0.54364 | load: 0.00249 [I 2022-10-26 22:31:43] ...trap/engines/engine.py.230: loss: 0.15101 [I 2022-10-26 22:31:43] ...optimizers/trijoint.py.151: total_norm: 18.039474 [I 2022-10-26 22:31:49] ...trap/engines/engine.py.224: train: epoch 33 | batch 920/2304 [I 2022-10-26 22:31:49] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:48 [I 2022-10-26 22:31:49] ...trap/engines/engine.py.229: process: 0.55186 | load: 0.00243 [I 2022-10-26 22:31:49] ...trap/engines/engine.py.230: loss: 0.18540 [I 2022-10-26 22:31:49] ...optimizers/trijoint.py.151: total_norm: 13.224014 [I 2022-10-26 22:31:54] ...trap/engines/engine.py.224: train: epoch 33 | batch 930/2304 [I 2022-10-26 22:31:54] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:47 [I 2022-10-26 22:31:54] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00243 [I 2022-10-26 22:31:54] ...trap/engines/engine.py.230: loss: 0.16138 [I 2022-10-26 22:31:54] ...optimizers/trijoint.py.151: total_norm: 20.098938 [I 2022-10-26 22:32:00] ...trap/engines/engine.py.224: train: epoch 33 | batch 940/2304 [I 2022-10-26 22:32:00] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:39 [I 2022-10-26 22:32:00] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00246 [I 2022-10-26 22:32:00] ...trap/engines/engine.py.230: loss: 0.14486 [I 2022-10-26 22:32:00] ...optimizers/trijoint.py.151: total_norm: 14.295346 [I 2022-10-26 22:32:06] ...trap/engines/engine.py.224: train: epoch 33 | batch 950/2304 [I 2022-10-26 22:32:06] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:36 [I 2022-10-26 22:32:06] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00242 [I 2022-10-26 22:32:06] ...trap/engines/engine.py.230: loss: 0.16048 [I 2022-10-26 22:32:06] ...optimizers/trijoint.py.151: total_norm: 14.380836 [I 2022-10-26 22:32:11] ...trap/engines/engine.py.224: train: epoch 33 | batch 960/2304 [I 2022-10-26 22:32:11] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:26 [I 2022-10-26 22:32:11] ...trap/engines/engine.py.229: process: 0.54857 | load: 0.00242 [I 2022-10-26 22:32:11] ...trap/engines/engine.py.230: loss: 0.21340 [I 2022-10-26 22:32:11] ...optimizers/trijoint.py.151: total_norm: 13.215734 [I 2022-10-26 22:32:17] ...trap/engines/engine.py.224: train: epoch 33 | batch 970/2304 [I 2022-10-26 22:32:17] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:15 [I 2022-10-26 22:32:17] ...trap/engines/engine.py.229: process: 0.53730 | load: 0.00246 [I 2022-10-26 22:32:17] ...trap/engines/engine.py.230: loss: 0.23892 [I 2022-10-26 22:32:17] ...optimizers/trijoint.py.151: total_norm: 44.465801 [I 2022-10-26 22:32:22] ...trap/engines/engine.py.224: train: epoch 33 | batch 980/2304 [I 2022-10-26 22:32:22] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:18 [I 2022-10-26 22:32:22] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00248 [I 2022-10-26 22:32:22] ...trap/engines/engine.py.230: loss: 0.13109 [I 2022-10-26 22:32:22] ...optimizers/trijoint.py.151: total_norm: 26.600843 [I 2022-10-26 22:32:28] ...trap/engines/engine.py.224: train: epoch 33 | batch 990/2304 [I 2022-10-26 22:32:28] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:05 [I 2022-10-26 22:32:28] ...trap/engines/engine.py.229: process: 0.54145 | load: 0.00245 [I 2022-10-26 22:32:28] ...trap/engines/engine.py.230: loss: 0.15392 [I 2022-10-26 22:32:28] ...optimizers/trijoint.py.151: total_norm: 18.127001 [I 2022-10-26 22:32:33] ...trap/engines/engine.py.224: train: epoch 33 | batch 1000/2304 [I 2022-10-26 22:32:33] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:06 [I 2022-10-26 22:32:33] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00242 [I 2022-10-26 22:32:33] ...trap/engines/engine.py.230: loss: 0.13371 [I 2022-10-26 22:32:33] ...optimizers/trijoint.py.151: total_norm: 10.360285 [I 2022-10-26 22:32:39] ...trap/engines/engine.py.224: train: epoch 33 | batch 1010/2304 [I 2022-10-26 22:32:39] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:11:57 [I 2022-10-26 22:32:39] ...trap/engines/engine.py.229: process: 0.54312 | load: 0.00244 [I 2022-10-26 22:32:39] ...trap/engines/engine.py.230: loss: 0.13632 [I 2022-10-26 22:32:39] ...optimizers/trijoint.py.151: total_norm: 20.879456 [I 2022-10-26 22:32:45] ...trap/engines/engine.py.224: train: epoch 33 | batch 1020/2304 [I 2022-10-26 22:32:45] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:57 [I 2022-10-26 22:32:45] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00242 [I 2022-10-26 22:32:45] ...trap/engines/engine.py.230: loss: 0.22433 [I 2022-10-26 22:32:45] ...optimizers/trijoint.py.151: total_norm: 18.245041 [I 2022-10-26 22:32:50] ...trap/engines/engine.py.224: train: epoch 33 | batch 1030/2304 [I 2022-10-26 22:32:50] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:47 [I 2022-10-26 22:32:50] ...trap/engines/engine.py.229: process: 0.54785 | load: 0.00241 [I 2022-10-26 22:32:50] ...trap/engines/engine.py.230: loss: 0.19027 [I 2022-10-26 22:32:50] ...optimizers/trijoint.py.151: total_norm: 15.736460 [I 2022-10-26 22:32:56] ...trap/engines/engine.py.224: train: epoch 33 | batch 1040/2304 [I 2022-10-26 22:32:56] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:44 [I 2022-10-26 22:32:56] ...trap/engines/engine.py.229: process: 0.56403 | load: 0.00251 [I 2022-10-26 22:32:56] ...trap/engines/engine.py.230: loss: 0.20968 [I 2022-10-26 22:32:56] ...optimizers/trijoint.py.151: total_norm: 17.843100 [I 2022-10-26 22:33:01] ...trap/engines/engine.py.224: train: epoch 33 | batch 1050/2304 [I 2022-10-26 22:33:01] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:37 [I 2022-10-26 22:33:01] ...trap/engines/engine.py.229: process: 0.54834 | load: 0.00245 [I 2022-10-26 22:33:01] ...trap/engines/engine.py.230: loss: 0.23229 [I 2022-10-26 22:33:01] ...optimizers/trijoint.py.151: total_norm: 21.649046 [I 2022-10-26 22:33:07] ...trap/engines/engine.py.224: train: epoch 33 | batch 1060/2304 [I 2022-10-26 22:33:07] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:26 [I 2022-10-26 22:33:07] ...trap/engines/engine.py.229: process: 0.54430 | load: 0.00243 [I 2022-10-26 22:33:07] ...trap/engines/engine.py.230: loss: 0.22317 [I 2022-10-26 22:33:07] ...optimizers/trijoint.py.151: total_norm: 33.444485 [I 2022-10-26 22:33:12] ...trap/engines/engine.py.224: train: epoch 33 | batch 1070/2304 [I 2022-10-26 22:33:12] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:24 [I 2022-10-26 22:33:12] ...trap/engines/engine.py.229: process: 0.54519 | load: 0.00239 [I 2022-10-26 22:33:12] ...trap/engines/engine.py.230: loss: 0.16244 [I 2022-10-26 22:33:12] ...optimizers/trijoint.py.151: total_norm: 19.685209 [I 2022-10-26 22:33:18] ...trap/engines/engine.py.224: train: epoch 33 | batch 1080/2304 [I 2022-10-26 22:33:18] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:19 [I 2022-10-26 22:33:18] ...trap/engines/engine.py.229: process: 0.56410 | load: 0.00240 [I 2022-10-26 22:33:18] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-26 22:33:18] ...optimizers/trijoint.py.151: total_norm: 9.288623 [I 2022-10-26 22:33:24] ...trap/engines/engine.py.224: train: epoch 33 | batch 1090/2304 [I 2022-10-26 22:33:24] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:17 [I 2022-10-26 22:33:24] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00243 [I 2022-10-26 22:33:24] ...trap/engines/engine.py.230: loss: 0.12934 [I 2022-10-26 22:33:24] ...optimizers/trijoint.py.151: total_norm: 10.687470 [I 2022-10-26 22:33:29] ...trap/engines/engine.py.224: train: epoch 33 | batch 1100/2304 [I 2022-10-26 22:33:29] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:13 [I 2022-10-26 22:33:29] ...trap/engines/engine.py.229: process: 0.56314 | load: 0.00261 [I 2022-10-26 22:33:29] ...trap/engines/engine.py.230: loss: 0.17619 [I 2022-10-26 22:33:29] ...optimizers/trijoint.py.151: total_norm: 17.767193 [I 2022-10-26 22:33:35] ...trap/engines/engine.py.224: train: epoch 33 | batch 1110/2304 [I 2022-10-26 22:33:35] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:06 [I 2022-10-26 22:33:35] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00249 [I 2022-10-26 22:33:35] ...trap/engines/engine.py.230: loss: 0.17449 [I 2022-10-26 22:33:35] ...optimizers/trijoint.py.151: total_norm: 20.073542 [I 2022-10-26 22:33:40] ...trap/engines/engine.py.224: train: epoch 33 | batch 1120/2304 [I 2022-10-26 22:33:40] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:10:55 [I 2022-10-26 22:33:40] ...trap/engines/engine.py.229: process: 0.54849 | load: 0.00247 [I 2022-10-26 22:33:40] ...trap/engines/engine.py.230: loss: 0.26943 [I 2022-10-26 22:33:40] ...optimizers/trijoint.py.151: total_norm: 25.052835 [I 2022-10-26 22:33:46] ...trap/engines/engine.py.224: train: epoch 33 | batch 1130/2304 [I 2022-10-26 22:33:46] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:55 [I 2022-10-26 22:33:46] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00244 [I 2022-10-26 22:33:46] ...trap/engines/engine.py.230: loss: 0.25306 [I 2022-10-26 22:33:46] ...optimizers/trijoint.py.151: total_norm: 30.278419 [I 2022-10-26 22:33:51] ...trap/engines/engine.py.224: train: epoch 33 | batch 1140/2304 [I 2022-10-26 22:33:51] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:47 [I 2022-10-26 22:33:51] ...trap/engines/engine.py.229: process: 0.54983 | load: 0.00241 [I 2022-10-26 22:33:51] ...trap/engines/engine.py.230: loss: 0.17743 [I 2022-10-26 22:33:51] ...optimizers/trijoint.py.151: total_norm: 22.081161 [I 2022-10-26 22:33:57] ...trap/engines/engine.py.224: train: epoch 33 | batch 1150/2304 [I 2022-10-26 22:33:57] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:45 [I 2022-10-26 22:33:57] ...trap/engines/engine.py.229: process: 0.56521 | load: 0.00247 [I 2022-10-26 22:33:57] ...trap/engines/engine.py.230: loss: 0.16401 [I 2022-10-26 22:33:57] ...optimizers/trijoint.py.151: total_norm: 15.869220 [I 2022-10-26 22:34:03] ...trap/engines/engine.py.224: train: epoch 33 | batch 1160/2304 [I 2022-10-26 22:34:03] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:40 [I 2022-10-26 22:34:03] ...trap/engines/engine.py.229: process: 0.55635 | load: 0.00241 [I 2022-10-26 22:34:03] ...trap/engines/engine.py.230: loss: 0.17876 [I 2022-10-26 22:34:03] ...optimizers/trijoint.py.151: total_norm: 17.532475 [I 2022-10-26 22:34:08] ...trap/engines/engine.py.224: train: epoch 33 | batch 1170/2304 [I 2022-10-26 22:34:08] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:32 [I 2022-10-26 22:34:08] ...trap/engines/engine.py.229: process: 0.56964 | load: 0.00245 [I 2022-10-26 22:34:08] ...trap/engines/engine.py.230: loss: 0.16908 [I 2022-10-26 22:34:08] ...optimizers/trijoint.py.151: total_norm: 18.193762 [I 2022-10-26 22:34:14] ...trap/engines/engine.py.224: train: epoch 33 | batch 1180/2304 [I 2022-10-26 22:34:14] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:26 [I 2022-10-26 22:34:14] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00241 [I 2022-10-26 22:34:14] ...trap/engines/engine.py.230: loss: 0.12131 [I 2022-10-26 22:34:14] ...optimizers/trijoint.py.151: total_norm: 15.912908 [I 2022-10-26 22:34:19] ...trap/engines/engine.py.224: train: epoch 33 | batch 1190/2304 [I 2022-10-26 22:34:19] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:24 [I 2022-10-26 22:34:19] ...trap/engines/engine.py.229: process: 0.56865 | load: 0.00250 [I 2022-10-26 22:34:19] ...trap/engines/engine.py.230: loss: 0.16166 [I 2022-10-26 22:34:19] ...optimizers/trijoint.py.151: total_norm: 12.133397 [I 2022-10-26 22:34:25] ...trap/engines/engine.py.224: train: epoch 33 | batch 1200/2304 [I 2022-10-26 22:34:25] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:14 [I 2022-10-26 22:34:25] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00253 [I 2022-10-26 22:34:25] ...trap/engines/engine.py.230: loss: 0.13214 [I 2022-10-26 22:34:25] ...optimizers/trijoint.py.151: total_norm: 12.392641 [I 2022-10-26 22:34:31] ...trap/engines/engine.py.224: train: epoch 33 | batch 1210/2304 [I 2022-10-26 22:34:31] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:07 [I 2022-10-26 22:34:31] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00248 [I 2022-10-26 22:34:31] ...trap/engines/engine.py.230: loss: 0.21338 [I 2022-10-26 22:34:31] ...optimizers/trijoint.py.151: total_norm: 19.039724 [I 2022-10-26 22:34:36] ...trap/engines/engine.py.224: train: epoch 33 | batch 1220/2304 [I 2022-10-26 22:34:36] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:02 [I 2022-10-26 22:34:36] ...trap/engines/engine.py.229: process: 0.54795 | load: 0.00250 [I 2022-10-26 22:34:36] ...trap/engines/engine.py.230: loss: 0.16320 [I 2022-10-26 22:34:36] ...optimizers/trijoint.py.151: total_norm: 12.244837 [I 2022-10-26 22:34:42] ...trap/engines/engine.py.224: train: epoch 33 | batch 1230/2304 [I 2022-10-26 22:34:42] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:09:53 [I 2022-10-26 22:34:42] ...trap/engines/engine.py.229: process: 0.54071 | load: 0.00242 [I 2022-10-26 22:34:42] ...trap/engines/engine.py.230: loss: 0.10805 [I 2022-10-26 22:34:42] ...optimizers/trijoint.py.151: total_norm: 13.511226 [I 2022-10-26 22:34:47] ...trap/engines/engine.py.224: train: epoch 33 | batch 1240/2304 [I 2022-10-26 22:34:47] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:52 [I 2022-10-26 22:34:47] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00244 [I 2022-10-26 22:34:47] ...trap/engines/engine.py.230: loss: 0.22466 [I 2022-10-26 22:34:47] ...optimizers/trijoint.py.151: total_norm: 24.165602 [I 2022-10-26 22:34:53] ...trap/engines/engine.py.224: train: epoch 33 | batch 1250/2304 [I 2022-10-26 22:34:53] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:09:45 [I 2022-10-26 22:34:53] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00242 [I 2022-10-26 22:34:53] ...trap/engines/engine.py.230: loss: 0.23124 [I 2022-10-26 22:34:53] ...optimizers/trijoint.py.151: total_norm: 19.755932 [I 2022-10-26 22:34:58] ...trap/engines/engine.py.224: train: epoch 33 | batch 1260/2304 [I 2022-10-26 22:34:58] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:40 [I 2022-10-26 22:34:58] ...trap/engines/engine.py.229: process: 0.54969 | load: 0.00244 [I 2022-10-26 22:34:58] ...trap/engines/engine.py.230: loss: 0.19127 [I 2022-10-26 22:34:58] ...optimizers/trijoint.py.151: total_norm: 20.200754 [I 2022-10-26 22:35:04] ...trap/engines/engine.py.224: train: epoch 33 | batch 1270/2304 [I 2022-10-26 22:35:04] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:34 [I 2022-10-26 22:35:04] ...trap/engines/engine.py.229: process: 0.55246 | load: 0.00242 [I 2022-10-26 22:35:04] ...trap/engines/engine.py.230: loss: 0.22529 [I 2022-10-26 22:35:04] ...optimizers/trijoint.py.151: total_norm: 20.206490 [I 2022-10-26 22:35:10] ...trap/engines/engine.py.224: train: epoch 33 | batch 1280/2304 [I 2022-10-26 22:35:10] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:27 [I 2022-10-26 22:35:10] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00254 [I 2022-10-26 22:35:10] ...trap/engines/engine.py.230: loss: 0.20465 [I 2022-10-26 22:35:10] ...optimizers/trijoint.py.151: total_norm: 35.865509 [I 2022-10-26 22:35:15] ...trap/engines/engine.py.224: train: epoch 33 | batch 1290/2304 [I 2022-10-26 22:35:15] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:21 [I 2022-10-26 22:35:15] ...trap/engines/engine.py.229: process: 0.54387 | load: 0.00251 [I 2022-10-26 22:35:15] ...trap/engines/engine.py.230: loss: 0.25469 [I 2022-10-26 22:35:15] ...optimizers/trijoint.py.151: total_norm: 47.955090 [I 2022-10-26 22:35:21] ...trap/engines/engine.py.224: train: epoch 33 | batch 1300/2304 [I 2022-10-26 22:35:21] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:18 [I 2022-10-26 22:35:21] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00249 [I 2022-10-26 22:35:21] ...trap/engines/engine.py.230: loss: 0.19742 [I 2022-10-26 22:35:21] ...optimizers/trijoint.py.151: total_norm: 22.584984 [I 2022-10-26 22:35:26] ...trap/engines/engine.py.224: train: epoch 33 | batch 1310/2304 [I 2022-10-26 22:35:26] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:23 [I 2022-10-26 22:35:26] ...trap/engines/engine.py.229: process: 0.57493 | load: 0.00245 [I 2022-10-26 22:35:26] ...trap/engines/engine.py.230: loss: 0.13841 [I 2022-10-26 22:35:26] ...optimizers/trijoint.py.151: total_norm: 8.098928 [I 2022-10-26 22:35:32] ...trap/engines/engine.py.224: train: epoch 33 | batch 1320/2304 [I 2022-10-26 22:35:32] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:08 [I 2022-10-26 22:35:32] ...trap/engines/engine.py.229: process: 0.55949 | load: 0.00242 [I 2022-10-26 22:35:32] ...trap/engines/engine.py.230: loss: 0.14923 [I 2022-10-26 22:35:32] ...optimizers/trijoint.py.151: total_norm: 13.455656 [I 2022-10-26 22:35:38] ...trap/engines/engine.py.224: train: epoch 33 | batch 1330/2304 [I 2022-10-26 22:35:38] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:06 [I 2022-10-26 22:35:38] ...trap/engines/engine.py.229: process: 0.56585 | load: 0.00245 [I 2022-10-26 22:35:38] ...trap/engines/engine.py.230: loss: 0.11650 [I 2022-10-26 22:35:38] ...optimizers/trijoint.py.151: total_norm: 8.188851 [I 2022-10-26 22:35:43] ...trap/engines/engine.py.224: train: epoch 33 | batch 1340/2304 [I 2022-10-26 22:35:43] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:51 [I 2022-10-26 22:35:43] ...trap/engines/engine.py.229: process: 0.55064 | load: 0.00243 [I 2022-10-26 22:35:43] ...trap/engines/engine.py.230: loss: 0.20019 [I 2022-10-26 22:35:43] ...optimizers/trijoint.py.151: total_norm: 34.017860 [I 2022-10-26 22:35:49] ...trap/engines/engine.py.224: train: epoch 33 | batch 1350/2304 [I 2022-10-26 22:35:49] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:53 [I 2022-10-26 22:35:49] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00249 [I 2022-10-26 22:35:49] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-26 22:35:49] ...optimizers/trijoint.py.151: total_norm: 8.767780 [I 2022-10-26 22:35:54] ...trap/engines/engine.py.224: train: epoch 33 | batch 1360/2304 [I 2022-10-26 22:35:54] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:44 [I 2022-10-26 22:35:54] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00246 [I 2022-10-26 22:35:54] ...trap/engines/engine.py.230: loss: 0.18662 [I 2022-10-26 22:35:54] ...optimizers/trijoint.py.151: total_norm: 17.966322 [I 2022-10-26 22:36:00] ...trap/engines/engine.py.224: train: epoch 33 | batch 1370/2304 [I 2022-10-26 22:36:00] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:37 [I 2022-10-26 22:36:00] ...trap/engines/engine.py.229: process: 0.54892 | load: 0.00239 [I 2022-10-26 22:36:00] ...trap/engines/engine.py.230: loss: 0.23476 [I 2022-10-26 22:36:00] ...optimizers/trijoint.py.151: total_norm: 21.377798 [I 2022-10-26 22:36:05] ...trap/engines/engine.py.224: train: epoch 33 | batch 1380/2304 [I 2022-10-26 22:36:05] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:31 [I 2022-10-26 22:36:05] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00244 [I 2022-10-26 22:36:05] ...trap/engines/engine.py.230: loss: 0.13197 [I 2022-10-26 22:36:05] ...optimizers/trijoint.py.151: total_norm: 15.182528 [I 2022-10-26 22:36:11] ...trap/engines/engine.py.224: train: epoch 33 | batch 1390/2304 [I 2022-10-26 22:36:11] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:30 [I 2022-10-26 22:36:11] ...trap/engines/engine.py.229: process: 0.55140 | load: 0.00253 [I 2022-10-26 22:36:11] ...trap/engines/engine.py.230: loss: 0.16708 [I 2022-10-26 22:36:11] ...optimizers/trijoint.py.151: total_norm: 24.966812 [I 2022-10-26 22:36:17] ...trap/engines/engine.py.224: train: epoch 33 | batch 1400/2304 [I 2022-10-26 22:36:17] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:45 [I 2022-10-26 22:36:17] ...trap/engines/engine.py.229: process: 0.66112 | load: 0.00260 [I 2022-10-26 22:36:17] ...trap/engines/engine.py.230: loss: 0.10904 [I 2022-10-26 22:36:17] ...optimizers/trijoint.py.151: total_norm: 9.866548 [I 2022-10-26 22:36:22] ...trap/engines/engine.py.224: train: epoch 33 | batch 1410/2304 [I 2022-10-26 22:36:22] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:20 [I 2022-10-26 22:36:22] ...trap/engines/engine.py.229: process: 0.55667 | load: 0.00268 [I 2022-10-26 22:36:22] ...trap/engines/engine.py.230: loss: 0.14629 [I 2022-10-26 22:36:22] ...optimizers/trijoint.py.151: total_norm: 14.642662 [I 2022-10-26 22:36:28] ...trap/engines/engine.py.224: train: epoch 33 | batch 1420/2304 [I 2022-10-26 22:36:28] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:11 [I 2022-10-26 22:36:28] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00267 [I 2022-10-26 22:36:28] ...trap/engines/engine.py.230: loss: 0.21946 [I 2022-10-26 22:36:28] ...optimizers/trijoint.py.151: total_norm: 18.740858 [I 2022-10-26 22:36:33] ...trap/engines/engine.py.224: train: epoch 33 | batch 1430/2304 [I 2022-10-26 22:36:33] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:07 [I 2022-10-26 22:36:33] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00244 [I 2022-10-26 22:36:33] ...trap/engines/engine.py.230: loss: 0.12370 [I 2022-10-26 22:36:33] ...optimizers/trijoint.py.151: total_norm: 9.083858 [I 2022-10-26 22:36:39] ...trap/engines/engine.py.224: train: epoch 33 | batch 1440/2304 [I 2022-10-26 22:36:39] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:07:59 [I 2022-10-26 22:36:39] ...trap/engines/engine.py.229: process: 0.55316 | load: 0.00245 [I 2022-10-26 22:36:39] ...trap/engines/engine.py.230: loss: 0.14198 [I 2022-10-26 22:36:39] ...optimizers/trijoint.py.151: total_norm: 14.384971 [I 2022-10-26 22:36:45] ...trap/engines/engine.py.224: train: epoch 33 | batch 1450/2304 [I 2022-10-26 22:36:45] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:07:56 [I 2022-10-26 22:36:45] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00244 [I 2022-10-26 22:36:45] ...trap/engines/engine.py.230: loss: 0.13836 [I 2022-10-26 22:36:45] ...optimizers/trijoint.py.151: total_norm: 10.942375 [I 2022-10-26 22:36:50] ...trap/engines/engine.py.224: train: epoch 33 | batch 1460/2304 [I 2022-10-26 22:36:50] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:47 [I 2022-10-26 22:36:50] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00245 [I 2022-10-26 22:36:50] ...trap/engines/engine.py.230: loss: 0.24727 [I 2022-10-26 22:36:50] ...optimizers/trijoint.py.151: total_norm: 38.826168 [I 2022-10-26 22:36:56] ...trap/engines/engine.py.224: train: epoch 33 | batch 1470/2304 [I 2022-10-26 22:36:56] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:07:48 [I 2022-10-26 22:36:56] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00242 [I 2022-10-26 22:36:56] ...trap/engines/engine.py.230: loss: 0.14681 [I 2022-10-26 22:36:56] ...optimizers/trijoint.py.151: total_norm: 16.774549 [I 2022-10-26 22:37:01] ...trap/engines/engine.py.224: train: epoch 33 | batch 1480/2304 [I 2022-10-26 22:37:01] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:39 [I 2022-10-26 22:37:01] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00243 [I 2022-10-26 22:37:01] ...trap/engines/engine.py.230: loss: 0.14475 [I 2022-10-26 22:37:01] ...optimizers/trijoint.py.151: total_norm: 21.209656 [I 2022-10-26 22:37:07] ...trap/engines/engine.py.224: train: epoch 33 | batch 1490/2304 [I 2022-10-26 22:37:07] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:27 [I 2022-10-26 22:37:07] ...trap/engines/engine.py.229: process: 0.54712 | load: 0.00240 [I 2022-10-26 22:37:07] ...trap/engines/engine.py.230: loss: 0.19646 [I 2022-10-26 22:37:07] ...optimizers/trijoint.py.151: total_norm: 21.644873 [I 2022-10-26 22:37:12] ...trap/engines/engine.py.224: train: epoch 33 | batch 1500/2304 [I 2022-10-26 22:37:12] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:25 [I 2022-10-26 22:37:12] ...trap/engines/engine.py.229: process: 0.55212 | load: 0.00241 [I 2022-10-26 22:37:12] ...trap/engines/engine.py.230: loss: 0.16852 [I 2022-10-26 22:37:12] ...optimizers/trijoint.py.151: total_norm: 36.553673 [I 2022-10-26 22:37:18] ...trap/engines/engine.py.224: train: epoch 33 | batch 1510/2304 [I 2022-10-26 22:37:18] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:21 [I 2022-10-26 22:37:18] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00243 [I 2022-10-26 22:37:18] ...trap/engines/engine.py.230: loss: 0.13807 [I 2022-10-26 22:37:18] ...optimizers/trijoint.py.151: total_norm: 12.622195 [I 2022-10-26 22:37:23] ...trap/engines/engine.py.224: train: epoch 33 | batch 1520/2304 [I 2022-10-26 22:37:23] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:14 [I 2022-10-26 22:37:23] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00242 [I 2022-10-26 22:37:23] ...trap/engines/engine.py.230: loss: 0.15067 [I 2022-10-26 22:37:23] ...optimizers/trijoint.py.151: total_norm: 22.306290 [I 2022-10-26 22:37:29] ...trap/engines/engine.py.224: train: epoch 33 | batch 1530/2304 [I 2022-10-26 22:37:29] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:11 [I 2022-10-26 22:37:29] ...trap/engines/engine.py.229: process: 0.56710 | load: 0.00240 [I 2022-10-26 22:37:29] ...trap/engines/engine.py.230: loss: 0.12066 [I 2022-10-26 22:37:29] ...optimizers/trijoint.py.151: total_norm: 13.224697 [I 2022-10-26 22:37:35] ...trap/engines/engine.py.224: train: epoch 33 | batch 1540/2304 [I 2022-10-26 22:37:35] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:01 [I 2022-10-26 22:37:35] ...trap/engines/engine.py.229: process: 0.54142 | load: 0.00251 [I 2022-10-26 22:37:35] ...trap/engines/engine.py.230: loss: 0.17193 [I 2022-10-26 22:37:35] ...optimizers/trijoint.py.151: total_norm: 12.851504 [I 2022-10-26 22:37:40] ...trap/engines/engine.py.224: train: epoch 33 | batch 1550/2304 [I 2022-10-26 22:37:40] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:06:58 [I 2022-10-26 22:37:40] ...trap/engines/engine.py.229: process: 0.54630 | load: 0.00247 [I 2022-10-26 22:37:40] ...trap/engines/engine.py.230: loss: 0.13190 [I 2022-10-26 22:37:40] ...optimizers/trijoint.py.151: total_norm: 19.426033 [I 2022-10-26 22:37:46] ...trap/engines/engine.py.224: train: epoch 33 | batch 1560/2304 [I 2022-10-26 22:37:46] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:06:53 [I 2022-10-26 22:37:46] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00245 [I 2022-10-26 22:37:46] ...trap/engines/engine.py.230: loss: 0.12754 [I 2022-10-26 22:37:46] ...optimizers/trijoint.py.151: total_norm: 15.445129 [I 2022-10-26 22:37:51] ...trap/engines/engine.py.224: train: epoch 33 | batch 1570/2304 [I 2022-10-26 22:37:51] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:06:50 [I 2022-10-26 22:37:51] ...trap/engines/engine.py.229: process: 0.56238 | load: 0.00246 [I 2022-10-26 22:37:51] ...trap/engines/engine.py.230: loss: 0.15126 [I 2022-10-26 22:37:51] ...optimizers/trijoint.py.151: total_norm: 12.452665 [I 2022-10-26 22:37:57] ...trap/engines/engine.py.224: train: epoch 33 | batch 1580/2304 [I 2022-10-26 22:37:57] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:06:40 [I 2022-10-26 22:37:57] ...trap/engines/engine.py.229: process: 0.54913 | load: 0.00243 [I 2022-10-26 22:37:57] ...trap/engines/engine.py.230: loss: 0.14239 [I 2022-10-26 22:37:57] ...optimizers/trijoint.py.151: total_norm: 17.998837 [I 2022-10-26 22:38:03] ...trap/engines/engine.py.224: train: epoch 33 | batch 1590/2304 [I 2022-10-26 22:38:03] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:40 [I 2022-10-26 22:38:03] ...trap/engines/engine.py.229: process: 0.57370 | load: 0.00244 [I 2022-10-26 22:38:03] ...trap/engines/engine.py.230: loss: 0.16162 [I 2022-10-26 22:38:03] ...optimizers/trijoint.py.151: total_norm: 24.289549 [I 2022-10-26 22:38:08] ...trap/engines/engine.py.224: train: epoch 33 | batch 1600/2304 [I 2022-10-26 22:38:08] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:31 [I 2022-10-26 22:38:08] ...trap/engines/engine.py.229: process: 0.55359 | load: 0.00245 [I 2022-10-26 22:38:08] ...trap/engines/engine.py.230: loss: 0.22634 [I 2022-10-26 22:38:08] ...optimizers/trijoint.py.151: total_norm: 34.783661 [I 2022-10-26 22:38:14] ...trap/engines/engine.py.224: train: epoch 33 | batch 1610/2304 [I 2022-10-26 22:38:14] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:23 [I 2022-10-26 22:38:14] ...trap/engines/engine.py.229: process: 0.54684 | load: 0.00243 [I 2022-10-26 22:38:14] ...trap/engines/engine.py.230: loss: 0.17253 [I 2022-10-26 22:38:14] ...optimizers/trijoint.py.151: total_norm: 31.085630 [I 2022-10-26 22:38:19] ...trap/engines/engine.py.224: train: epoch 33 | batch 1620/2304 [I 2022-10-26 22:38:19] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:20 [I 2022-10-26 22:38:19] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00243 [I 2022-10-26 22:38:19] ...trap/engines/engine.py.230: loss: 0.13145 [I 2022-10-26 22:38:19] ...optimizers/trijoint.py.151: total_norm: 15.223066 [I 2022-10-26 22:38:25] ...trap/engines/engine.py.224: train: epoch 33 | batch 1630/2304 [I 2022-10-26 22:38:25] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:16 [I 2022-10-26 22:38:25] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00243 [I 2022-10-26 22:38:25] ...trap/engines/engine.py.230: loss: 0.19424 [I 2022-10-26 22:38:25] ...optimizers/trijoint.py.151: total_norm: 24.597878 [I 2022-10-26 22:38:30] ...trap/engines/engine.py.224: train: epoch 33 | batch 1640/2304 [I 2022-10-26 22:38:30] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:10 [I 2022-10-26 22:38:30] ...trap/engines/engine.py.229: process: 0.54821 | load: 0.00252 [I 2022-10-26 22:38:30] ...trap/engines/engine.py.230: loss: 0.13781 [I 2022-10-26 22:38:30] ...optimizers/trijoint.py.151: total_norm: 19.759758 [I 2022-10-26 22:38:36] ...trap/engines/engine.py.224: train: epoch 33 | batch 1650/2304 [I 2022-10-26 22:38:36] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:03 [I 2022-10-26 22:38:36] ...trap/engines/engine.py.229: process: 0.55274 | load: 0.00265 [I 2022-10-26 22:38:36] ...trap/engines/engine.py.230: loss: 0.15854 [I 2022-10-26 22:38:36] ...optimizers/trijoint.py.151: total_norm: 15.156915 [I 2022-10-26 22:38:42] ...trap/engines/engine.py.224: train: epoch 33 | batch 1660/2304 [I 2022-10-26 22:38:42] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:05:56 [I 2022-10-26 22:38:42] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00241 [I 2022-10-26 22:38:42] ...trap/engines/engine.py.230: loss: 0.17296 [I 2022-10-26 22:38:42] ...optimizers/trijoint.py.151: total_norm: 12.985428 [I 2022-10-26 22:38:47] ...trap/engines/engine.py.224: train: epoch 33 | batch 1670/2304 [I 2022-10-26 22:38:47] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:05:53 [I 2022-10-26 22:38:47] ...trap/engines/engine.py.229: process: 0.57054 | load: 0.00247 [I 2022-10-26 22:38:47] ...trap/engines/engine.py.230: loss: 0.16786 [I 2022-10-26 22:38:47] ...optimizers/trijoint.py.151: total_norm: 32.007572 [I 2022-10-26 22:38:53] ...trap/engines/engine.py.224: train: epoch 33 | batch 1680/2304 [I 2022-10-26 22:38:53] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:05:47 [I 2022-10-26 22:38:53] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00244 [I 2022-10-26 22:38:53] ...trap/engines/engine.py.230: loss: 0.15273 [I 2022-10-26 22:38:53] ...optimizers/trijoint.py.151: total_norm: 22.575748 [I 2022-10-26 22:38:58] ...trap/engines/engine.py.224: train: epoch 33 | batch 1690/2304 [I 2022-10-26 22:38:58] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:05:44 [I 2022-10-26 22:38:58] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00248 [I 2022-10-26 22:38:58] ...trap/engines/engine.py.230: loss: 0.17195 [I 2022-10-26 22:38:58] ...optimizers/trijoint.py.151: total_norm: 15.710896 [I 2022-10-26 22:39:04] ...trap/engines/engine.py.224: train: epoch 33 | batch 1700/2304 [I 2022-10-26 22:39:04] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:42 [I 2022-10-26 22:39:04] ...trap/engines/engine.py.229: process: 0.54300 | load: 0.00255 [I 2022-10-26 22:39:04] ...trap/engines/engine.py.230: loss: 0.25819 [I 2022-10-26 22:39:04] ...optimizers/trijoint.py.151: total_norm: 27.983772 [I 2022-10-26 22:39:09] ...trap/engines/engine.py.224: train: epoch 33 | batch 1710/2304 [I 2022-10-26 22:39:09] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:29 [I 2022-10-26 22:39:09] ...trap/engines/engine.py.229: process: 0.55364 | load: 0.00244 [I 2022-10-26 22:39:09] ...trap/engines/engine.py.230: loss: 0.18936 [I 2022-10-26 22:39:09] ...optimizers/trijoint.py.151: total_norm: 19.983858 [I 2022-10-26 22:39:15] ...trap/engines/engine.py.224: train: epoch 33 | batch 1720/2304 [I 2022-10-26 22:39:15] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:23 [I 2022-10-26 22:39:15] ...trap/engines/engine.py.229: process: 0.55309 | load: 0.00316 [I 2022-10-26 22:39:15] ...trap/engines/engine.py.230: loss: 0.24004 [I 2022-10-26 22:39:15] ...optimizers/trijoint.py.151: total_norm: 18.935360 [I 2022-10-26 22:39:21] ...trap/engines/engine.py.224: train: epoch 33 | batch 1730/2304 [I 2022-10-26 22:39:21] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:16 [I 2022-10-26 22:39:21] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00246 [I 2022-10-26 22:39:21] ...trap/engines/engine.py.230: loss: 0.21065 [I 2022-10-26 22:39:21] ...optimizers/trijoint.py.151: total_norm: 25.636536 [I 2022-10-26 22:39:26] ...trap/engines/engine.py.224: train: epoch 33 | batch 1740/2304 [I 2022-10-26 22:39:26] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:13 [I 2022-10-26 22:39:26] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00241 [I 2022-10-26 22:39:26] ...trap/engines/engine.py.230: loss: 0.19854 [I 2022-10-26 22:39:26] ...optimizers/trijoint.py.151: total_norm: 18.108221 [I 2022-10-26 22:39:32] ...trap/engines/engine.py.224: train: epoch 33 | batch 1750/2304 [I 2022-10-26 22:39:32] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:08 [I 2022-10-26 22:39:32] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00250 [I 2022-10-26 22:39:32] ...trap/engines/engine.py.230: loss: 0.16852 [I 2022-10-26 22:39:32] ...optimizers/trijoint.py.151: total_norm: 28.242132 [I 2022-10-26 22:39:37] ...trap/engines/engine.py.224: train: epoch 33 | batch 1760/2304 [I 2022-10-26 22:39:37] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:02 [I 2022-10-26 22:39:37] ...trap/engines/engine.py.229: process: 0.56327 | load: 0.00245 [I 2022-10-26 22:39:37] ...trap/engines/engine.py.230: loss: 0.17955 [I 2022-10-26 22:39:37] ...optimizers/trijoint.py.151: total_norm: 16.300783 [I 2022-10-26 22:39:43] ...trap/engines/engine.py.224: train: epoch 33 | batch 1770/2304 [I 2022-10-26 22:39:43] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:04:58 [I 2022-10-26 22:39:43] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00244 [I 2022-10-26 22:39:43] ...trap/engines/engine.py.230: loss: 0.17833 [I 2022-10-26 22:39:43] ...optimizers/trijoint.py.151: total_norm: 12.077703 [I 2022-10-26 22:39:48] ...trap/engines/engine.py.224: train: epoch 33 | batch 1780/2304 [I 2022-10-26 22:39:48] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:04:52 [I 2022-10-26 22:39:48] ...trap/engines/engine.py.229: process: 0.54854 | load: 0.00246 [I 2022-10-26 22:39:48] ...trap/engines/engine.py.230: loss: 0.22162 [I 2022-10-26 22:39:48] ...optimizers/trijoint.py.151: total_norm: 27.241686 [I 2022-10-26 22:39:54] ...trap/engines/engine.py.224: train: epoch 33 | batch 1790/2304 [I 2022-10-26 22:39:54] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:04:47 [I 2022-10-26 22:39:54] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00245 [I 2022-10-26 22:39:54] ...trap/engines/engine.py.230: loss: 0.17077 [I 2022-10-26 22:39:54] ...optimizers/trijoint.py.151: total_norm: 20.519018 [I 2022-10-26 22:40:00] ...trap/engines/engine.py.224: train: epoch 33 | batch 1800/2304 [I 2022-10-26 22:40:00] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:40 [I 2022-10-26 22:40:00] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00243 [I 2022-10-26 22:40:00] ...trap/engines/engine.py.230: loss: 0.12703 [I 2022-10-26 22:40:00] ...optimizers/trijoint.py.151: total_norm: 16.724569 [I 2022-10-26 22:40:05] ...trap/engines/engine.py.224: train: epoch 33 | batch 1810/2304 [I 2022-10-26 22:40:05] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:34 [I 2022-10-26 22:40:05] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00241 [I 2022-10-26 22:40:05] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-26 22:40:05] ...optimizers/trijoint.py.151: total_norm: 15.140098 [I 2022-10-26 22:40:11] ...trap/engines/engine.py.224: train: epoch 33 | batch 1820/2304 [I 2022-10-26 22:40:11] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:31 [I 2022-10-26 22:40:11] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00244 [I 2022-10-26 22:40:11] ...trap/engines/engine.py.230: loss: 0.29419 [I 2022-10-26 22:40:11] ...optimizers/trijoint.py.151: total_norm: 19.788311 [I 2022-10-26 22:40:16] ...trap/engines/engine.py.224: train: epoch 33 | batch 1830/2304 [I 2022-10-26 22:40:16] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:23 [I 2022-10-26 22:40:16] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00245 [I 2022-10-26 22:40:16] ...trap/engines/engine.py.230: loss: 0.21617 [I 2022-10-26 22:40:16] ...optimizers/trijoint.py.151: total_norm: 34.229374 [I 2022-10-26 22:40:22] ...trap/engines/engine.py.224: train: epoch 33 | batch 1840/2304 [I 2022-10-26 22:40:22] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:17 [I 2022-10-26 22:40:22] ...trap/engines/engine.py.229: process: 0.54108 | load: 0.00243 [I 2022-10-26 22:40:22] ...trap/engines/engine.py.230: loss: 0.12322 [I 2022-10-26 22:40:22] ...optimizers/trijoint.py.151: total_norm: 13.679843 [I 2022-10-26 22:40:28] ...trap/engines/engine.py.224: train: epoch 33 | batch 1850/2304 [I 2022-10-26 22:40:28] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:12 [I 2022-10-26 22:40:28] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00246 [I 2022-10-26 22:40:28] ...trap/engines/engine.py.230: loss: 0.18585 [I 2022-10-26 22:40:28] ...optimizers/trijoint.py.151: total_norm: 22.471664 [I 2022-10-26 22:40:33] ...trap/engines/engine.py.224: train: epoch 33 | batch 1860/2304 [I 2022-10-26 22:40:33] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:07 [I 2022-10-26 22:40:33] ...trap/engines/engine.py.229: process: 0.54684 | load: 0.00250 [I 2022-10-26 22:40:33] ...trap/engines/engine.py.230: loss: 0.21232 [I 2022-10-26 22:40:33] ...optimizers/trijoint.py.151: total_norm: 25.095129 [I 2022-10-26 22:40:39] ...trap/engines/engine.py.224: train: epoch 33 | batch 1870/2304 [I 2022-10-26 22:40:39] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:00 [I 2022-10-26 22:40:39] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00243 [I 2022-10-26 22:40:39] ...trap/engines/engine.py.230: loss: 0.18265 [I 2022-10-26 22:40:39] ...optimizers/trijoint.py.151: total_norm: 23.518450 [I 2022-10-26 22:40:44] ...trap/engines/engine.py.224: train: epoch 33 | batch 1880/2304 [I 2022-10-26 22:40:44] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:03:54 [I 2022-10-26 22:40:44] ...trap/engines/engine.py.229: process: 0.54815 | load: 0.00244 [I 2022-10-26 22:40:44] ...trap/engines/engine.py.230: loss: 0.12762 [I 2022-10-26 22:40:44] ...optimizers/trijoint.py.151: total_norm: 13.732456 [I 2022-10-26 22:40:50] ...trap/engines/engine.py.224: train: epoch 33 | batch 1890/2304 [I 2022-10-26 22:40:50] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:03:49 [I 2022-10-26 22:40:50] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00245 [I 2022-10-26 22:40:50] ...trap/engines/engine.py.230: loss: 0.16575 [I 2022-10-26 22:40:50] ...optimizers/trijoint.py.151: total_norm: 11.486076 [I 2022-10-26 22:40:55] ...trap/engines/engine.py.224: train: epoch 33 | batch 1900/2304 [I 2022-10-26 22:40:55] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:03:44 [I 2022-10-26 22:40:55] ...trap/engines/engine.py.229: process: 0.55975 | load: 0.00244 [I 2022-10-26 22:40:55] ...trap/engines/engine.py.230: loss: 0.19167 [I 2022-10-26 22:40:55] ...optimizers/trijoint.py.151: total_norm: 19.151646 [I 2022-10-26 22:41:01] ...trap/engines/engine.py.224: train: epoch 33 | batch 1910/2304 [I 2022-10-26 22:41:01] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:03:37 [I 2022-10-26 22:41:01] ...trap/engines/engine.py.229: process: 0.53525 | load: 0.00244 [I 2022-10-26 22:41:01] ...trap/engines/engine.py.230: loss: 0.12954 [I 2022-10-26 22:41:01] ...optimizers/trijoint.py.151: total_norm: 21.652342 [I 2022-10-26 22:41:06] ...trap/engines/engine.py.224: train: epoch 33 | batch 1920/2304 [I 2022-10-26 22:41:06] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:33 [I 2022-10-26 22:41:06] ...trap/engines/engine.py.229: process: 0.56218 | load: 0.00244 [I 2022-10-26 22:41:07] ...trap/engines/engine.py.230: loss: 0.19612 [I 2022-10-26 22:41:07] ...optimizers/trijoint.py.151: total_norm: 21.019611 [I 2022-10-26 22:41:12] ...trap/engines/engine.py.224: train: epoch 33 | batch 1930/2304 [I 2022-10-26 22:41:12] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:27 [I 2022-10-26 22:41:12] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00243 [I 2022-10-26 22:41:12] ...trap/engines/engine.py.230: loss: 0.15571 [I 2022-10-26 22:41:12] ...optimizers/trijoint.py.151: total_norm: 11.616441 [I 2022-10-26 22:41:18] ...trap/engines/engine.py.224: train: epoch 33 | batch 1940/2304 [I 2022-10-26 22:41:18] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:23 [I 2022-10-26 22:41:18] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00242 [I 2022-10-26 22:41:18] ...trap/engines/engine.py.230: loss: 0.22555 [I 2022-10-26 22:41:18] ...optimizers/trijoint.py.151: total_norm: 21.352339 [I 2022-10-26 22:41:23] ...trap/engines/engine.py.224: train: epoch 33 | batch 1950/2304 [I 2022-10-26 22:41:23] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:16 [I 2022-10-26 22:41:23] ...trap/engines/engine.py.229: process: 0.53602 | load: 0.00247 [I 2022-10-26 22:41:23] ...trap/engines/engine.py.230: loss: 0.14156 [I 2022-10-26 22:41:23] ...optimizers/trijoint.py.151: total_norm: 15.586843 [I 2022-10-26 22:41:29] ...trap/engines/engine.py.224: train: epoch 33 | batch 1960/2304 [I 2022-10-26 22:41:29] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:10 [I 2022-10-26 22:41:29] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00242 [I 2022-10-26 22:41:29] ...trap/engines/engine.py.230: loss: 0.16454 [I 2022-10-26 22:41:29] ...optimizers/trijoint.py.151: total_norm: 14.436137 [I 2022-10-26 22:41:34] ...trap/engines/engine.py.224: train: epoch 33 | batch 1970/2304 [I 2022-10-26 22:41:34] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:06 [I 2022-10-26 22:41:34] ...trap/engines/engine.py.229: process: 0.55767 | load: 0.00243 [I 2022-10-26 22:41:34] ...trap/engines/engine.py.230: loss: 0.27389 [I 2022-10-26 22:41:34] ...optimizers/trijoint.py.151: total_norm: 20.365135 [I 2022-10-26 22:41:40] ...trap/engines/engine.py.224: train: epoch 33 | batch 1980/2304 [I 2022-10-26 22:41:40] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:01 [I 2022-10-26 22:41:40] ...trap/engines/engine.py.229: process: 0.56673 | load: 0.00244 [I 2022-10-26 22:41:40] ...trap/engines/engine.py.230: loss: 0.24999 [I 2022-10-26 22:41:40] ...optimizers/trijoint.py.151: total_norm: 19.282270 [I 2022-10-26 22:41:46] ...trap/engines/engine.py.224: train: epoch 33 | batch 1990/2304 [I 2022-10-26 22:41:46] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:02:53 [I 2022-10-26 22:41:46] ...trap/engines/engine.py.229: process: 0.54273 | load: 0.00249 [I 2022-10-26 22:41:46] ...trap/engines/engine.py.230: loss: 0.19043 [I 2022-10-26 22:41:46] ...optimizers/trijoint.py.151: total_norm: 12.633675 [I 2022-10-26 22:41:51] ...trap/engines/engine.py.224: train: epoch 33 | batch 2000/2304 [I 2022-10-26 22:41:51] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:02:49 [I 2022-10-26 22:41:51] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00250 [I 2022-10-26 22:41:51] ...trap/engines/engine.py.230: loss: 0.15253 [I 2022-10-26 22:41:51] ...optimizers/trijoint.py.151: total_norm: 8.991092 [I 2022-10-26 22:41:57] ...trap/engines/engine.py.224: train: epoch 33 | batch 2010/2304 [I 2022-10-26 22:41:57] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:02:43 [I 2022-10-26 22:41:57] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00244 [I 2022-10-26 22:41:57] ...trap/engines/engine.py.230: loss: 0.17838 [I 2022-10-26 22:41:57] ...optimizers/trijoint.py.151: total_norm: 37.180157 [I 2022-10-26 22:42:02] ...trap/engines/engine.py.224: train: epoch 33 | batch 2020/2304 [I 2022-10-26 22:42:02] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:37 [I 2022-10-26 22:42:02] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00241 [I 2022-10-26 22:42:02] ...trap/engines/engine.py.230: loss: 0.21210 [I 2022-10-26 22:42:02] ...optimizers/trijoint.py.151: total_norm: 22.507313 [I 2022-10-26 22:42:08] ...trap/engines/engine.py.224: train: epoch 33 | batch 2030/2304 [I 2022-10-26 22:42:08] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:31 [I 2022-10-26 22:42:08] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00247 [I 2022-10-26 22:42:08] ...trap/engines/engine.py.230: loss: 0.13301 [I 2022-10-26 22:42:08] ...optimizers/trijoint.py.151: total_norm: 25.276974 [I 2022-10-26 22:42:13] ...trap/engines/engine.py.224: train: epoch 33 | batch 2040/2304 [I 2022-10-26 22:42:13] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:25 [I 2022-10-26 22:42:13] ...trap/engines/engine.py.229: process: 0.55873 | load: 0.00244 [I 2022-10-26 22:42:13] ...trap/engines/engine.py.230: loss: 0.19218 [I 2022-10-26 22:42:13] ...optimizers/trijoint.py.151: total_norm: 23.784485 [I 2022-10-26 22:42:19] ...trap/engines/engine.py.224: train: epoch 33 | batch 2050/2304 [I 2022-10-26 22:42:19] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:02:21 [I 2022-10-26 22:42:19] ...trap/engines/engine.py.229: process: 0.56509 | load: 0.00245 [I 2022-10-26 22:42:19] ...trap/engines/engine.py.230: loss: 0.11519 [I 2022-10-26 22:42:19] ...optimizers/trijoint.py.151: total_norm: 17.688643 [I 2022-10-26 22:42:24] ...trap/engines/engine.py.224: train: epoch 33 | batch 2060/2304 [I 2022-10-26 22:42:24] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:15 [I 2022-10-26 22:42:24] ...trap/engines/engine.py.229: process: 0.54981 | load: 0.00244 [I 2022-10-26 22:42:24] ...trap/engines/engine.py.230: loss: 0.23091 [I 2022-10-26 22:42:24] ...optimizers/trijoint.py.151: total_norm: 16.897102 [I 2022-10-26 22:42:30] ...trap/engines/engine.py.224: train: epoch 33 | batch 2070/2304 [I 2022-10-26 22:42:30] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:09 [I 2022-10-26 22:42:30] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00244 [I 2022-10-26 22:42:30] ...trap/engines/engine.py.230: loss: 0.18055 [I 2022-10-26 22:42:30] ...optimizers/trijoint.py.151: total_norm: 14.746401 [I 2022-10-26 22:42:36] ...trap/engines/engine.py.224: train: epoch 33 | batch 2080/2304 [I 2022-10-26 22:42:36] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:05 [I 2022-10-26 22:42:36] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00246 [I 2022-10-26 22:42:36] ...trap/engines/engine.py.230: loss: 0.18372 [I 2022-10-26 22:42:36] ...optimizers/trijoint.py.151: total_norm: 15.970066 [I 2022-10-26 22:42:41] ...trap/engines/engine.py.224: train: epoch 33 | batch 2090/2304 [I 2022-10-26 22:42:41] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:01:58 [I 2022-10-26 22:42:41] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00245 [I 2022-10-26 22:42:41] ...trap/engines/engine.py.230: loss: 0.19410 [I 2022-10-26 22:42:41] ...optimizers/trijoint.py.151: total_norm: 27.967422 [I 2022-10-26 22:42:47] ...trap/engines/engine.py.224: train: epoch 33 | batch 2100/2304 [I 2022-10-26 22:42:47] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:01:52 [I 2022-10-26 22:42:47] ...trap/engines/engine.py.229: process: 0.55345 | load: 0.00246 [I 2022-10-26 22:42:47] ...trap/engines/engine.py.230: loss: 0.17332 [I 2022-10-26 22:42:47] ...optimizers/trijoint.py.151: total_norm: 9.737944 [I 2022-10-26 22:42:52] ...trap/engines/engine.py.224: train: epoch 33 | batch 2110/2304 [I 2022-10-26 22:42:52] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:01:48 [I 2022-10-26 22:42:52] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00248 [I 2022-10-26 22:42:52] ...trap/engines/engine.py.230: loss: 0.14486 [I 2022-10-26 22:42:52] ...optimizers/trijoint.py.151: total_norm: 11.328066 [I 2022-10-26 22:42:58] ...trap/engines/engine.py.224: train: epoch 33 | batch 2120/2304 [I 2022-10-26 22:42:58] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:01:41 [I 2022-10-26 22:42:58] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00246 [I 2022-10-26 22:42:58] ...trap/engines/engine.py.230: loss: 0.15995 [I 2022-10-26 22:42:58] ...optimizers/trijoint.py.151: total_norm: 18.134932 [I 2022-10-26 22:43:03] ...trap/engines/engine.py.224: train: epoch 33 | batch 2130/2304 [I 2022-10-26 22:43:03] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:35 [I 2022-10-26 22:43:03] ...trap/engines/engine.py.229: process: 0.54331 | load: 0.00242 [I 2022-10-26 22:43:03] ...trap/engines/engine.py.230: loss: 0.20176 [I 2022-10-26 22:43:03] ...optimizers/trijoint.py.151: total_norm: 18.330511 [I 2022-10-26 22:43:09] ...trap/engines/engine.py.224: train: epoch 33 | batch 2140/2304 [I 2022-10-26 22:43:09] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:30 [I 2022-10-26 22:43:09] ...trap/engines/engine.py.229: process: 0.54581 | load: 0.00263 [I 2022-10-26 22:43:09] ...trap/engines/engine.py.230: loss: 0.22138 [I 2022-10-26 22:43:09] ...optimizers/trijoint.py.151: total_norm: 31.071388 [I 2022-10-26 22:43:14] ...trap/engines/engine.py.224: train: epoch 33 | batch 2150/2304 [I 2022-10-26 22:43:14] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:25 [I 2022-10-26 22:43:14] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00245 [I 2022-10-26 22:43:14] ...trap/engines/engine.py.230: loss: 0.22578 [I 2022-10-26 22:43:14] ...optimizers/trijoint.py.151: total_norm: 23.736937 [I 2022-10-26 22:43:20] ...trap/engines/engine.py.224: train: epoch 33 | batch 2160/2304 [I 2022-10-26 22:43:20] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:19 [I 2022-10-26 22:43:20] ...trap/engines/engine.py.229: process: 0.54610 | load: 0.00242 [I 2022-10-26 22:43:20] ...trap/engines/engine.py.230: loss: 0.14045 [I 2022-10-26 22:43:20] ...optimizers/trijoint.py.151: total_norm: 14.166081 [I 2022-10-26 22:43:26] ...trap/engines/engine.py.224: train: epoch 33 | batch 2170/2304 [I 2022-10-26 22:43:26] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:14 [I 2022-10-26 22:43:26] ...trap/engines/engine.py.229: process: 0.55638 | load: 0.00258 [I 2022-10-26 22:43:26] ...trap/engines/engine.py.230: loss: 0.15490 [I 2022-10-26 22:43:26] ...optimizers/trijoint.py.151: total_norm: 14.275721 [I 2022-10-26 22:43:31] ...trap/engines/engine.py.224: train: epoch 33 | batch 2180/2304 [I 2022-10-26 22:43:31] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:09 [I 2022-10-26 22:43:31] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00258 [I 2022-10-26 22:43:31] ...trap/engines/engine.py.230: loss: 0.22216 [I 2022-10-26 22:43:31] ...optimizers/trijoint.py.151: total_norm: 20.159544 [I 2022-10-26 22:43:37] ...trap/engines/engine.py.224: train: epoch 33 | batch 2190/2304 [I 2022-10-26 22:43:37] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:03 [I 2022-10-26 22:43:37] ...trap/engines/engine.py.229: process: 0.54363 | load: 0.00251 [I 2022-10-26 22:43:37] ...trap/engines/engine.py.230: loss: 0.20346 [I 2022-10-26 22:43:37] ...optimizers/trijoint.py.151: total_norm: 28.888330 [I 2022-10-26 22:43:42] ...trap/engines/engine.py.224: train: epoch 33 | batch 2200/2304 [I 2022-10-26 22:43:42] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:00:57 [I 2022-10-26 22:43:42] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00245 [I 2022-10-26 22:43:42] ...trap/engines/engine.py.230: loss: 0.13657 [I 2022-10-26 22:43:42] ...optimizers/trijoint.py.151: total_norm: 13.462256 [I 2022-10-26 22:43:48] ...trap/engines/engine.py.224: train: epoch 33 | batch 2210/2304 [I 2022-10-26 22:43:48] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:00:52 [I 2022-10-26 22:43:48] ...trap/engines/engine.py.229: process: 0.54595 | load: 0.00261 [I 2022-10-26 22:43:48] ...trap/engines/engine.py.230: loss: 0.23408 [I 2022-10-26 22:43:48] ...optimizers/trijoint.py.151: total_norm: 34.279263 [I 2022-10-26 22:43:53] ...trap/engines/engine.py.224: train: epoch 33 | batch 2220/2304 [I 2022-10-26 22:43:53] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:00:46 [I 2022-10-26 22:43:53] ...trap/engines/engine.py.229: process: 0.56207 | load: 0.00242 [I 2022-10-26 22:43:53] ...trap/engines/engine.py.230: loss: 0.19356 [I 2022-10-26 22:43:53] ...optimizers/trijoint.py.151: total_norm: 23.013403 [I 2022-10-26 22:43:59] ...trap/engines/engine.py.224: train: epoch 33 | batch 2230/2304 [I 2022-10-26 22:43:59] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:00:40 [I 2022-10-26 22:43:59] ...trap/engines/engine.py.229: process: 0.54825 | load: 0.00244 [I 2022-10-26 22:43:59] ...trap/engines/engine.py.230: loss: 0.19853 [I 2022-10-26 22:43:59] ...optimizers/trijoint.py.151: total_norm: 20.026611 [I 2022-10-26 22:44:05] ...trap/engines/engine.py.224: train: epoch 33 | batch 2240/2304 [I 2022-10-26 22:44:05] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:35 [I 2022-10-26 22:44:05] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00244 [I 2022-10-26 22:44:05] ...trap/engines/engine.py.230: loss: 0.14892 [I 2022-10-26 22:44:05] ...optimizers/trijoint.py.151: total_norm: 16.537329 [I 2022-10-26 22:44:10] ...trap/engines/engine.py.224: train: epoch 33 | batch 2250/2304 [I 2022-10-26 22:44:10] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:29 [I 2022-10-26 22:44:10] ...trap/engines/engine.py.229: process: 0.55462 | load: 0.00241 [I 2022-10-26 22:44:10] ...trap/engines/engine.py.230: loss: 0.19015 [I 2022-10-26 22:44:10] ...optimizers/trijoint.py.151: total_norm: 17.508654 [I 2022-10-26 22:44:16] ...trap/engines/engine.py.224: train: epoch 33 | batch 2260/2304 [I 2022-10-26 22:44:16] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:24 [I 2022-10-26 22:44:16] ...trap/engines/engine.py.229: process: 0.54700 | load: 0.00240 [I 2022-10-26 22:44:16] ...trap/engines/engine.py.230: loss: 0.16716 [I 2022-10-26 22:44:16] ...optimizers/trijoint.py.151: total_norm: 17.431910 [I 2022-10-26 22:44:21] ...trap/engines/engine.py.224: train: epoch 33 | batch 2270/2304 [I 2022-10-26 22:44:21] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:18 [I 2022-10-26 22:44:21] ...trap/engines/engine.py.229: process: 0.56470 | load: 0.00243 [I 2022-10-26 22:44:21] ...trap/engines/engine.py.230: loss: 0.17274 [I 2022-10-26 22:44:21] ...optimizers/trijoint.py.151: total_norm: 22.087715 [I 2022-10-26 22:44:27] ...trap/engines/engine.py.224: train: epoch 33 | batch 2280/2304 [I 2022-10-26 22:44:27] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:13 [I 2022-10-26 22:44:27] ...trap/engines/engine.py.229: process: 0.55206 | load: 0.00244 [I 2022-10-26 22:44:27] ...trap/engines/engine.py.230: loss: 0.14823 [I 2022-10-26 22:44:27] ...optimizers/trijoint.py.151: total_norm: 11.218430 [I 2022-10-26 22:44:32] ...trap/engines/engine.py.224: train: epoch 33 | batch 2290/2304 [I 2022-10-26 22:44:32] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:07 [I 2022-10-26 22:44:32] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00248 [I 2022-10-26 22:44:32] ...trap/engines/engine.py.230: loss: 0.19948 [I 2022-10-26 22:44:32] ...optimizers/trijoint.py.151: total_norm: 23.235468 [I 2022-10-26 22:44:38] ...trap/engines/engine.py.224: train: epoch 33 | batch 2300/2304 [I 2022-10-26 22:44:38] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:02 [I 2022-10-26 22:44:38] ...trap/engines/engine.py.229: process: 0.55472 | load: 0.00016 [I 2022-10-26 22:44:38] ...trap/engines/engine.py.230: loss: 0.19294 [I 2022-10-26 22:44:38] ...optimizers/trijoint.py.151: total_norm: 17.259499 [I 2022-10-26 22:44:40] ...trap/engines/engine.py.224: train: epoch 33 | batch 2304/2304 [I 2022-10-26 22:44:40] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:00 [I 2022-10-26 22:44:40] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00016 [I 2022-10-26 22:44:40] ...trap/engines/engine.py.230: loss: 0.15381 [I 2022-10-26 22:44:40] ...optimizers/trijoint.py.151: total_norm: 16.293062 [S 2022-10-26 22:44:41] ...trap/engines/engine.py.129: train_epoch.epoch: 33 [S 2022-10-26 22:44:41] ...trap/engines/engine.py.129: train_epoch.loss: 0.17747416731592372 [I 2022-10-26 22:44:41] ...s/criterions/triplet.py.84: margin = 0.22000000000000014 [I 2022-10-26 22:44:42] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 33 [I 2022-10-26 22:44:42] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 22:44:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 22:44:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 22:44:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 22:44:45] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 22:44:45] ...trap/engines/engine.py.321: eval: epoch 33 | batch 0/510 [I 2022-10-26 22:44:45] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:44 [I 2022-10-26 22:44:45] ...trap/engines/engine.py.326: process: 3.26402 | load: 2.89665 [I 2022-10-26 22:44:49] ...trap/engines/engine.py.321: eval: epoch 33 | batch 10/510 [I 2022-10-26 22:44:49] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:25 [I 2022-10-26 22:44:49] ...trap/engines/engine.py.326: process: 0.32443 | load: 0.00019 [I 2022-10-26 22:44:52] ...trap/engines/engine.py.321: eval: epoch 33 | batch 20/510 [I 2022-10-26 22:44:52] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:03:01 [I 2022-10-26 22:44:52] ...trap/engines/engine.py.326: process: 0.32535 | load: 0.00018 [I 2022-10-26 22:44:55] ...trap/engines/engine.py.321: eval: epoch 33 | batch 30/510 [I 2022-10-26 22:44:55] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:41 [I 2022-10-26 22:44:55] ...trap/engines/engine.py.326: process: 0.32790 | load: 0.00016 [I 2022-10-26 22:44:59] ...trap/engines/engine.py.321: eval: epoch 33 | batch 40/510 [I 2022-10-26 22:44:59] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:37 [I 2022-10-26 22:44:59] ...trap/engines/engine.py.326: process: 0.33238 | load: 0.00016 [I 2022-10-26 22:45:02] ...trap/engines/engine.py.321: eval: epoch 33 | batch 50/510 [I 2022-10-26 22:45:02] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:32 [I 2022-10-26 22:45:02] ...trap/engines/engine.py.326: process: 0.33040 | load: 0.00017 [I 2022-10-26 22:45:05] ...trap/engines/engine.py.321: eval: epoch 33 | batch 60/510 [I 2022-10-26 22:45:05] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:29 [I 2022-10-26 22:45:05] ...trap/engines/engine.py.326: process: 0.33317 | load: 0.00017 [I 2022-10-26 22:45:09] ...trap/engines/engine.py.321: eval: epoch 33 | batch 70/510 [I 2022-10-26 22:45:09] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:29 [I 2022-10-26 22:45:09] ...trap/engines/engine.py.326: process: 0.34637 | load: 0.00026 [I 2022-10-26 22:45:12] ...trap/engines/engine.py.321: eval: epoch 33 | batch 80/510 [I 2022-10-26 22:45:12] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:25 [I 2022-10-26 22:45:12] ...trap/engines/engine.py.326: process: 0.33492 | load: 0.00024 [I 2022-10-26 22:45:15] ...trap/engines/engine.py.321: eval: epoch 33 | batch 90/510 [I 2022-10-26 22:45:15] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:21 [I 2022-10-26 22:45:15] ...trap/engines/engine.py.326: process: 0.34464 | load: 0.00016 [I 2022-10-26 22:45:19] ...trap/engines/engine.py.321: eval: epoch 33 | batch 100/510 [I 2022-10-26 22:45:19] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 22:45:19] ...trap/engines/engine.py.326: process: 0.28312 | load: 0.00018 [I 2022-10-26 22:45:22] ...trap/engines/engine.py.321: eval: epoch 33 | batch 110/510 [I 2022-10-26 22:45:22] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:00 [I 2022-10-26 22:45:22] ...trap/engines/engine.py.326: process: 0.29017 | load: 0.00021 [I 2022-10-26 22:45:25] ...trap/engines/engine.py.321: eval: epoch 33 | batch 120/510 [I 2022-10-26 22:45:25] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-26 22:45:25] ...trap/engines/engine.py.326: process: 0.28530 | load: 0.00019 [I 2022-10-26 22:45:27] ...trap/engines/engine.py.321: eval: epoch 33 | batch 130/510 [I 2022-10-26 22:45:27] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:49 [I 2022-10-26 22:45:27] ...trap/engines/engine.py.326: process: 0.28526 | load: 0.00020 [I 2022-10-26 22:45:30] ...trap/engines/engine.py.321: eval: epoch 33 | batch 140/510 [I 2022-10-26 22:45:30] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:48 [I 2022-10-26 22:45:30] ...trap/engines/engine.py.326: process: 0.29125 | load: 0.00024 [I 2022-10-26 22:45:33] ...trap/engines/engine.py.321: eval: epoch 33 | batch 150/510 [I 2022-10-26 22:45:33] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:45 [I 2022-10-26 22:45:33] ...trap/engines/engine.py.326: process: 0.28906 | load: 0.00017 [I 2022-10-26 22:45:36] ...trap/engines/engine.py.321: eval: epoch 33 | batch 160/510 [I 2022-10-26 22:45:36] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-26 22:45:36] ...trap/engines/engine.py.326: process: 0.28898 | load: 0.00014 [I 2022-10-26 22:45:39] ...trap/engines/engine.py.321: eval: epoch 33 | batch 170/510 [I 2022-10-26 22:45:39] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-26 22:45:39] ...trap/engines/engine.py.326: process: 0.28810 | load: 0.00014 [I 2022-10-26 22:45:42] ...trap/engines/engine.py.321: eval: epoch 33 | batch 180/510 [I 2022-10-26 22:45:42] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:37 [I 2022-10-26 22:45:42] ...trap/engines/engine.py.326: process: 0.31046 | load: 0.00015 [I 2022-10-26 22:45:45] ...trap/engines/engine.py.321: eval: epoch 33 | batch 190/510 [I 2022-10-26 22:45:45] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:33 [I 2022-10-26 22:45:45] ...trap/engines/engine.py.326: process: 0.28818 | load: 0.00020 [I 2022-10-26 22:45:48] ...trap/engines/engine.py.321: eval: epoch 33 | batch 200/510 [I 2022-10-26 22:45:48] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:30 [I 2022-10-26 22:45:48] ...trap/engines/engine.py.326: process: 0.29165 | load: 0.00015 [I 2022-10-26 22:45:51] ...trap/engines/engine.py.321: eval: epoch 33 | batch 210/510 [I 2022-10-26 22:45:51] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:25 [I 2022-10-26 22:45:51] ...trap/engines/engine.py.326: process: 0.27576 | load: 0.00015 [I 2022-10-26 22:45:54] ...trap/engines/engine.py.321: eval: epoch 33 | batch 220/510 [I 2022-10-26 22:45:54] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 22:45:54] ...trap/engines/engine.py.326: process: 0.29372 | load: 0.00014 [I 2022-10-26 22:45:57] ...trap/engines/engine.py.321: eval: epoch 33 | batch 230/510 [I 2022-10-26 22:45:57] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 22:45:57] ...trap/engines/engine.py.326: process: 0.28666 | load: 0.00015 [I 2022-10-26 22:45:59] ...trap/engines/engine.py.321: eval: epoch 33 | batch 240/510 [I 2022-10-26 22:45:59] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 22:45:59] ...trap/engines/engine.py.326: process: 0.28765 | load: 0.00014 [I 2022-10-26 22:46:02] ...trap/engines/engine.py.321: eval: epoch 33 | batch 250/510 [I 2022-10-26 22:46:02] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-26 22:46:02] ...trap/engines/engine.py.326: process: 0.28588 | load: 0.00015 [I 2022-10-26 22:46:05] ...trap/engines/engine.py.321: eval: epoch 33 | batch 260/510 [I 2022-10-26 22:46:05] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-26 22:46:05] ...trap/engines/engine.py.326: process: 0.29227 | load: 0.00016 [I 2022-10-26 22:46:08] ...trap/engines/engine.py.321: eval: epoch 33 | batch 270/510 [I 2022-10-26 22:46:08] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-26 22:46:08] ...trap/engines/engine.py.326: process: 0.28862 | load: 0.00021 [I 2022-10-26 22:46:11] ...trap/engines/engine.py.321: eval: epoch 33 | batch 280/510 [I 2022-10-26 22:46:11] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:05 [I 2022-10-26 22:46:11] ...trap/engines/engine.py.326: process: 0.28637 | load: 0.00015 [I 2022-10-26 22:46:14] ...trap/engines/engine.py.321: eval: epoch 33 | batch 290/510 [I 2022-10-26 22:46:14] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-26 22:46:14] ...trap/engines/engine.py.326: process: 0.28558 | load: 0.00016 [I 2022-10-26 22:46:17] ...trap/engines/engine.py.321: eval: epoch 33 | batch 300/510 [I 2022-10-26 22:46:17] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 22:46:17] ...trap/engines/engine.py.326: process: 0.28044 | load: 0.00016 [I 2022-10-26 22:46:20] ...trap/engines/engine.py.321: eval: epoch 33 | batch 310/510 [I 2022-10-26 22:46:20] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 22:46:20] ...trap/engines/engine.py.326: process: 0.28739 | load: 0.00015 [I 2022-10-26 22:46:23] ...trap/engines/engine.py.321: eval: epoch 33 | batch 320/510 [I 2022-10-26 22:46:23] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 22:46:23] ...trap/engines/engine.py.326: process: 0.29158 | load: 0.00020 [I 2022-10-26 22:46:25] ...trap/engines/engine.py.321: eval: epoch 33 | batch 330/510 [I 2022-10-26 22:46:25] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 22:46:25] ...trap/engines/engine.py.326: process: 0.28706 | load: 0.00015 [I 2022-10-26 22:46:28] ...trap/engines/engine.py.321: eval: epoch 33 | batch 340/510 [I 2022-10-26 22:46:28] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:49 [I 2022-10-26 22:46:28] ...trap/engines/engine.py.326: process: 0.28559 | load: 0.00036 [I 2022-10-26 22:46:31] ...trap/engines/engine.py.321: eval: epoch 33 | batch 350/510 [I 2022-10-26 22:46:31] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 22:46:31] ...trap/engines/engine.py.326: process: 0.28687 | load: 0.00015 [I 2022-10-26 22:46:34] ...trap/engines/engine.py.321: eval: epoch 33 | batch 360/510 [I 2022-10-26 22:46:34] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 22:46:34] ...trap/engines/engine.py.326: process: 0.28908 | load: 0.00015 [I 2022-10-26 22:46:37] ...trap/engines/engine.py.321: eval: epoch 33 | batch 370/510 [I 2022-10-26 22:46:37] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-26 22:46:37] ...trap/engines/engine.py.326: process: 0.29023 | load: 0.00017 [I 2022-10-26 22:46:40] ...trap/engines/engine.py.321: eval: epoch 33 | batch 380/510 [I 2022-10-26 22:46:40] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-26 22:46:40] ...trap/engines/engine.py.326: process: 0.28364 | load: 0.00016 [I 2022-10-26 22:46:43] ...trap/engines/engine.py.321: eval: epoch 33 | batch 390/510 [I 2022-10-26 22:46:43] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 22:46:43] ...trap/engines/engine.py.326: process: 0.30177 | load: 0.00015 [I 2022-10-26 22:46:46] ...trap/engines/engine.py.321: eval: epoch 33 | batch 400/510 [I 2022-10-26 22:46:46] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 22:46:46] ...trap/engines/engine.py.326: process: 0.28616 | load: 0.00015 [I 2022-10-26 22:46:49] ...trap/engines/engine.py.321: eval: epoch 33 | batch 410/510 [I 2022-10-26 22:46:49] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:29 [I 2022-10-26 22:46:49] ...trap/engines/engine.py.326: process: 0.29269 | load: 0.00016 [I 2022-10-26 22:46:52] ...trap/engines/engine.py.321: eval: epoch 33 | batch 420/510 [I 2022-10-26 22:46:52] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:26 [I 2022-10-26 22:46:52] ...trap/engines/engine.py.326: process: 0.28492 | load: 0.00017 [I 2022-10-26 22:46:54] ...trap/engines/engine.py.321: eval: epoch 33 | batch 430/510 [I 2022-10-26 22:46:54] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 22:46:54] ...trap/engines/engine.py.326: process: 0.29650 | load: 0.00037 [I 2022-10-26 22:46:57] ...trap/engines/engine.py.321: eval: epoch 33 | batch 440/510 [I 2022-10-26 22:46:57] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 22:46:57] ...trap/engines/engine.py.326: process: 0.28641 | load: 0.00014 [I 2022-10-26 22:47:00] ...trap/engines/engine.py.321: eval: epoch 33 | batch 450/510 [I 2022-10-26 22:47:00] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 22:47:00] ...trap/engines/engine.py.326: process: 0.30032 | load: 0.00016 [I 2022-10-26 22:47:03] ...trap/engines/engine.py.321: eval: epoch 33 | batch 460/510 [I 2022-10-26 22:47:03] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-26 22:47:03] ...trap/engines/engine.py.326: process: 0.28875 | load: 0.00015 [I 2022-10-26 22:47:06] ...trap/engines/engine.py.321: eval: epoch 33 | batch 470/510 [I 2022-10-26 22:47:06] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-26 22:47:06] ...trap/engines/engine.py.326: process: 0.29017 | load: 0.00014 [I 2022-10-26 22:47:09] ...trap/engines/engine.py.321: eval: epoch 33 | batch 480/510 [I 2022-10-26 22:47:09] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-26 22:47:09] ...trap/engines/engine.py.326: process: 0.28970 | load: 0.00016 [I 2022-10-26 22:47:12] ...trap/engines/engine.py.321: eval: epoch 33 | batch 490/510 [I 2022-10-26 22:47:12] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 22:47:12] ...trap/engines/engine.py.326: process: 0.29554 | load: 0.00015 [I 2022-10-26 22:47:15] ...trap/engines/engine.py.321: eval: epoch 33 | batch 500/510 [I 2022-10-26 22:47:15] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 22:47:15] ...trap/engines/engine.py.326: process: 0.28767 | load: 0.00015 [I 2022-10-26 22:47:18] ...trap/engines/engine.py.321: eval: epoch 33 | batch 510/510 [I 2022-10-26 22:47:18] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 22:47:18] ...trap/engines/engine.py.326: process: 0.28234 | load: 0.00010 [S 2022-10-26 22:47:18] ...trap/engines/engine.py.132: eval_epoch.epoch: 33 [I 2022-10-26 22:47:18] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.679100 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.890100 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.929300 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.665900 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.885400 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.924600 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017213 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006123 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007734 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008596 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007723 [S 2022-10-26 22:47:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006651 [I 2022-10-26 22:47:23] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-26 22:47:23] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 22:47:24] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 22:47:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 22:47:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 22:47:25] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 22:47:27] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 22:47:33] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 22:47:33] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 22:47:33] ...trap/engines/engine.py.470: Saving model... [I 2022-10-26 22:47:36] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 22:47:42] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 22:47:42] ...trap/engines/engine.py.163: Training model on trainset for epoch 34 [I 2022-10-26 22:47:42] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 22:47:45] ...trap/engines/engine.py.224: train: epoch 34 | batch 0/2304 [I 2022-10-26 22:47:45] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:38 [I 2022-10-26 22:47:45] ...trap/engines/engine.py.229: process: 2.67280 | load: 2.03111 [I 2022-10-26 22:47:45] ...trap/engines/engine.py.230: loss: 0.13458 [I 2022-10-26 22:47:45] ...optimizers/trijoint.py.151: total_norm: 10.335021 [I 2022-10-26 22:47:51] ...trap/engines/engine.py.224: train: epoch 34 | batch 10/2304 [I 2022-10-26 22:47:51] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:17 [I 2022-10-26 22:47:51] ...trap/engines/engine.py.229: process: 0.54987 | load: 0.00434 [I 2022-10-26 22:47:51] ...trap/engines/engine.py.230: loss: 0.16988 [I 2022-10-26 22:47:51] ...optimizers/trijoint.py.151: total_norm: 17.713840 [I 2022-10-26 22:47:56] ...trap/engines/engine.py.224: train: epoch 34 | batch 20/2304 [I 2022-10-26 22:47:56] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:14 [I 2022-10-26 22:47:56] ...trap/engines/engine.py.229: process: 0.54686 | load: 0.00297 [I 2022-10-26 22:47:56] ...trap/engines/engine.py.230: loss: 0.14415 [I 2022-10-26 22:47:56] ...optimizers/trijoint.py.151: total_norm: 10.883500 [I 2022-10-26 22:48:02] ...trap/engines/engine.py.224: train: epoch 34 | batch 30/2304 [I 2022-10-26 22:48:02] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-26 22:48:02] ...trap/engines/engine.py.229: process: 0.56368 | load: 0.00279 [I 2022-10-26 22:48:02] ...trap/engines/engine.py.230: loss: 0.15192 [I 2022-10-26 22:48:02] ...optimizers/trijoint.py.151: total_norm: 20.242199 [I 2022-10-26 22:48:07] ...trap/engines/engine.py.224: train: epoch 34 | batch 40/2304 [I 2022-10-26 22:48:07] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:03 [I 2022-10-26 22:48:07] ...trap/engines/engine.py.229: process: 0.54973 | load: 0.00336 [I 2022-10-26 22:48:07] ...trap/engines/engine.py.230: loss: 0.13574 [I 2022-10-26 22:48:07] ...optimizers/trijoint.py.151: total_norm: 8.802055 [I 2022-10-26 22:48:13] ...trap/engines/engine.py.224: train: epoch 34 | batch 50/2304 [I 2022-10-26 22:48:13] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:58 [I 2022-10-26 22:48:13] ...trap/engines/engine.py.229: process: 0.56963 | load: 0.00314 [I 2022-10-26 22:48:13] ...trap/engines/engine.py.230: loss: 0.10942 [I 2022-10-26 22:48:13] ...optimizers/trijoint.py.151: total_norm: 8.289357 [I 2022-10-26 22:48:19] ...trap/engines/engine.py.224: train: epoch 34 | batch 60/2304 [I 2022-10-26 22:48:19] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:51 [I 2022-10-26 22:48:19] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00279 [I 2022-10-26 22:48:19] ...trap/engines/engine.py.230: loss: 0.13402 [I 2022-10-26 22:48:19] ...optimizers/trijoint.py.151: total_norm: 9.172881 [I 2022-10-26 22:48:24] ...trap/engines/engine.py.224: train: epoch 34 | batch 70/2304 [I 2022-10-26 22:48:24] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:44 [I 2022-10-26 22:48:24] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00281 [I 2022-10-26 22:48:24] ...trap/engines/engine.py.230: loss: 0.21792 [I 2022-10-26 22:48:24] ...optimizers/trijoint.py.151: total_norm: 25.094784 [I 2022-10-26 22:48:30] ...trap/engines/engine.py.224: train: epoch 34 | batch 80/2304 [I 2022-10-26 22:48:30] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:39 [I 2022-10-26 22:48:30] ...trap/engines/engine.py.229: process: 0.55419 | load: 0.00273 [I 2022-10-26 22:48:30] ...trap/engines/engine.py.230: loss: 0.25121 [I 2022-10-26 22:48:30] ...optimizers/trijoint.py.151: total_norm: 23.751032 [I 2022-10-26 22:48:35] ...trap/engines/engine.py.224: train: epoch 34 | batch 90/2304 [I 2022-10-26 22:48:35] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:37 [I 2022-10-26 22:48:35] ...trap/engines/engine.py.229: process: 0.54791 | load: 0.00271 [I 2022-10-26 22:48:35] ...trap/engines/engine.py.230: loss: 0.15487 [I 2022-10-26 22:48:35] ...optimizers/trijoint.py.151: total_norm: 17.538200 [I 2022-10-26 22:48:41] ...trap/engines/engine.py.224: train: epoch 34 | batch 100/2304 [I 2022-10-26 22:48:41] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:26 [I 2022-10-26 22:48:41] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00252 [I 2022-10-26 22:48:41] ...trap/engines/engine.py.230: loss: 0.18043 [I 2022-10-26 22:48:41] ...optimizers/trijoint.py.151: total_norm: 16.437769 [I 2022-10-26 22:48:47] ...trap/engines/engine.py.224: train: epoch 34 | batch 110/2304 [I 2022-10-26 22:48:47] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:32 [I 2022-10-26 22:48:47] ...trap/engines/engine.py.229: process: 0.55024 | load: 0.00260 [I 2022-10-26 22:48:47] ...trap/engines/engine.py.230: loss: 0.16558 [I 2022-10-26 22:48:47] ...optimizers/trijoint.py.151: total_norm: 20.690855 [I 2022-10-26 22:48:52] ...trap/engines/engine.py.224: train: epoch 34 | batch 120/2304 [I 2022-10-26 22:48:52] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:37 [I 2022-10-26 22:48:52] ...trap/engines/engine.py.229: process: 0.58076 | load: 0.00253 [I 2022-10-26 22:48:52] ...trap/engines/engine.py.230: loss: 0.13050 [I 2022-10-26 22:48:52] ...optimizers/trijoint.py.151: total_norm: 10.921978 [I 2022-10-26 22:48:58] ...trap/engines/engine.py.224: train: epoch 34 | batch 130/2304 [I 2022-10-26 22:48:58] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:13 [I 2022-10-26 22:48:58] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00255 [I 2022-10-26 22:48:58] ...trap/engines/engine.py.230: loss: 0.17474 [I 2022-10-26 22:48:58] ...optimizers/trijoint.py.151: total_norm: 24.565899 [I 2022-10-26 22:49:03] ...trap/engines/engine.py.224: train: epoch 34 | batch 140/2304 [I 2022-10-26 22:49:03] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:23 [I 2022-10-26 22:49:03] ...trap/engines/engine.py.229: process: 0.57816 | load: 0.00271 [I 2022-10-26 22:49:03] ...trap/engines/engine.py.230: loss: 0.16507 [I 2022-10-26 22:49:03] ...optimizers/trijoint.py.151: total_norm: 15.048988 [I 2022-10-26 22:49:09] ...trap/engines/engine.py.224: train: epoch 34 | batch 150/2304 [I 2022-10-26 22:49:09] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:05 [I 2022-10-26 22:49:09] ...trap/engines/engine.py.229: process: 0.56950 | load: 0.00264 [I 2022-10-26 22:49:09] ...trap/engines/engine.py.230: loss: 0.24917 [I 2022-10-26 22:49:09] ...optimizers/trijoint.py.151: total_norm: 25.724760 [I 2022-10-26 22:49:15] ...trap/engines/engine.py.224: train: epoch 34 | batch 160/2304 [I 2022-10-26 22:49:15] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:58 [I 2022-10-26 22:49:15] ...trap/engines/engine.py.229: process: 0.56401 | load: 0.00261 [I 2022-10-26 22:49:15] ...trap/engines/engine.py.230: loss: 0.20659 [I 2022-10-26 22:49:15] ...optimizers/trijoint.py.151: total_norm: 19.443581 [I 2022-10-26 22:49:20] ...trap/engines/engine.py.224: train: epoch 34 | batch 170/2304 [I 2022-10-26 22:49:20] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:20:03 [I 2022-10-26 22:49:20] ...trap/engines/engine.py.229: process: 0.58789 | load: 0.00264 [I 2022-10-26 22:49:20] ...trap/engines/engine.py.230: loss: 0.18573 [I 2022-10-26 22:49:20] ...optimizers/trijoint.py.151: total_norm: 13.564481 [I 2022-10-26 22:49:26] ...trap/engines/engine.py.224: train: epoch 34 | batch 180/2304 [I 2022-10-26 22:49:26] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:48 [I 2022-10-26 22:49:26] ...trap/engines/engine.py.229: process: 0.56658 | load: 0.00262 [I 2022-10-26 22:49:26] ...trap/engines/engine.py.230: loss: 0.14727 [I 2022-10-26 22:49:26] ...optimizers/trijoint.py.151: total_norm: 16.252914 [I 2022-10-26 22:49:31] ...trap/engines/engine.py.224: train: epoch 34 | batch 190/2304 [I 2022-10-26 22:49:31] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:37 [I 2022-10-26 22:49:31] ...trap/engines/engine.py.229: process: 0.55720 | load: 0.00247 [I 2022-10-26 22:49:31] ...trap/engines/engine.py.230: loss: 0.10697 [I 2022-10-26 22:49:31] ...optimizers/trijoint.py.151: total_norm: 12.942740 [I 2022-10-26 22:49:37] ...trap/engines/engine.py.224: train: epoch 34 | batch 200/2304 [I 2022-10-26 22:49:37] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:32 [I 2022-10-26 22:49:37] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00264 [I 2022-10-26 22:49:37] ...trap/engines/engine.py.230: loss: 0.23889 [I 2022-10-26 22:49:37] ...optimizers/trijoint.py.151: total_norm: 11.774834 [I 2022-10-26 22:49:43] ...trap/engines/engine.py.224: train: epoch 34 | batch 210/2304 [I 2022-10-26 22:49:43] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:24 [I 2022-10-26 22:49:43] ...trap/engines/engine.py.229: process: 0.54382 | load: 0.00255 [I 2022-10-26 22:49:43] ...trap/engines/engine.py.230: loss: 0.16003 [I 2022-10-26 22:49:43] ...optimizers/trijoint.py.151: total_norm: 11.746143 [I 2022-10-26 22:49:48] ...trap/engines/engine.py.224: train: epoch 34 | batch 220/2304 [I 2022-10-26 22:49:48] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:20:16 [I 2022-10-26 22:49:48] ...trap/engines/engine.py.229: process: 0.56689 | load: 0.00282 [I 2022-10-26 22:49:48] ...trap/engines/engine.py.230: loss: 0.16552 [I 2022-10-26 22:49:48] ...optimizers/trijoint.py.151: total_norm: 13.933674 [I 2022-10-26 22:49:54] ...trap/engines/engine.py.224: train: epoch 34 | batch 230/2304 [I 2022-10-26 22:49:54] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:17 [I 2022-10-26 22:49:54] ...trap/engines/engine.py.229: process: 0.55492 | load: 0.00256 [I 2022-10-26 22:49:54] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-26 22:49:54] ...optimizers/trijoint.py.151: total_norm: 12.731455 [I 2022-10-26 22:50:00] ...trap/engines/engine.py.224: train: epoch 34 | batch 240/2304 [I 2022-10-26 22:50:00] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:07 [I 2022-10-26 22:50:00] ...trap/engines/engine.py.229: process: 0.54426 | load: 0.00260 [I 2022-10-26 22:50:00] ...trap/engines/engine.py.230: loss: 0.15986 [I 2022-10-26 22:50:00] ...optimizers/trijoint.py.151: total_norm: 17.017649 [I 2022-10-26 22:50:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 250/2304 [I 2022-10-26 22:50:05] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:07 [I 2022-10-26 22:50:05] ...trap/engines/engine.py.229: process: 0.57233 | load: 0.00244 [I 2022-10-26 22:50:05] ...trap/engines/engine.py.230: loss: 0.15281 [I 2022-10-26 22:50:05] ...optimizers/trijoint.py.151: total_norm: 20.791264 [I 2022-10-26 22:50:11] ...trap/engines/engine.py.224: train: epoch 34 | batch 260/2304 [I 2022-10-26 22:50:11] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:58 [I 2022-10-26 22:50:11] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00256 [I 2022-10-26 22:50:11] ...trap/engines/engine.py.230: loss: 0.19119 [I 2022-10-26 22:50:11] ...optimizers/trijoint.py.151: total_norm: 25.628662 [I 2022-10-26 22:50:16] ...trap/engines/engine.py.224: train: epoch 34 | batch 270/2304 [I 2022-10-26 22:50:16] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:45 [I 2022-10-26 22:50:16] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00249 [I 2022-10-26 22:50:16] ...trap/engines/engine.py.230: loss: 0.19447 [I 2022-10-26 22:50:16] ...optimizers/trijoint.py.151: total_norm: 22.594028 [I 2022-10-26 22:50:22] ...trap/engines/engine.py.224: train: epoch 34 | batch 280/2304 [I 2022-10-26 22:50:22] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:39 [I 2022-10-26 22:50:22] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00257 [I 2022-10-26 22:50:22] ...trap/engines/engine.py.230: loss: 0.17093 [I 2022-10-26 22:50:22] ...optimizers/trijoint.py.151: total_norm: 15.953282 [I 2022-10-26 22:50:28] ...trap/engines/engine.py.224: train: epoch 34 | batch 290/2304 [I 2022-10-26 22:50:28] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:57 [I 2022-10-26 22:50:28] ...trap/engines/engine.py.229: process: 0.56554 | load: 0.00268 [I 2022-10-26 22:50:28] ...trap/engines/engine.py.230: loss: 0.16395 [I 2022-10-26 22:50:28] ...optimizers/trijoint.py.151: total_norm: 20.364008 [I 2022-10-26 22:50:33] ...trap/engines/engine.py.224: train: epoch 34 | batch 300/2304 [I 2022-10-26 22:50:33] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:42 [I 2022-10-26 22:50:33] ...trap/engines/engine.py.229: process: 0.56792 | load: 0.00258 [I 2022-10-26 22:50:33] ...trap/engines/engine.py.230: loss: 0.13413 [I 2022-10-26 22:50:33] ...optimizers/trijoint.py.151: total_norm: 15.831867 [I 2022-10-26 22:50:39] ...trap/engines/engine.py.224: train: epoch 34 | batch 310/2304 [I 2022-10-26 22:50:39] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:41 [I 2022-10-26 22:50:39] ...trap/engines/engine.py.229: process: 0.56539 | load: 0.00277 [I 2022-10-26 22:50:39] ...trap/engines/engine.py.230: loss: 0.16766 [I 2022-10-26 22:50:39] ...optimizers/trijoint.py.151: total_norm: 22.029793 [I 2022-10-26 22:50:44] ...trap/engines/engine.py.224: train: epoch 34 | batch 320/2304 [I 2022-10-26 22:50:44] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:24 [I 2022-10-26 22:50:44] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00243 [I 2022-10-26 22:50:44] ...trap/engines/engine.py.230: loss: 0.10189 [I 2022-10-26 22:50:44] ...optimizers/trijoint.py.151: total_norm: 16.368296 [I 2022-10-26 22:50:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 330/2304 [I 2022-10-26 22:50:50] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:24 [I 2022-10-26 22:50:50] ...trap/engines/engine.py.229: process: 0.55832 | load: 0.00252 [I 2022-10-26 22:50:50] ...trap/engines/engine.py.230: loss: 0.14006 [I 2022-10-26 22:50:50] ...optimizers/trijoint.py.151: total_norm: 14.982148 [I 2022-10-26 22:50:56] ...trap/engines/engine.py.224: train: epoch 34 | batch 340/2304 [I 2022-10-26 22:50:56] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:09 [I 2022-10-26 22:50:56] ...trap/engines/engine.py.229: process: 0.54804 | load: 0.00250 [I 2022-10-26 22:50:56] ...trap/engines/engine.py.230: loss: 0.23016 [I 2022-10-26 22:50:56] ...optimizers/trijoint.py.151: total_norm: 24.172516 [I 2022-10-26 22:51:01] ...trap/engines/engine.py.224: train: epoch 34 | batch 350/2304 [I 2022-10-26 22:51:01] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:09 [I 2022-10-26 22:51:01] ...trap/engines/engine.py.229: process: 0.57586 | load: 0.00247 [I 2022-10-26 22:51:01] ...trap/engines/engine.py.230: loss: 0.14227 [I 2022-10-26 22:51:01] ...optimizers/trijoint.py.151: total_norm: 17.509918 [I 2022-10-26 22:51:07] ...trap/engines/engine.py.224: train: epoch 34 | batch 360/2304 [I 2022-10-26 22:51:07] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:04 [I 2022-10-26 22:51:07] ...trap/engines/engine.py.229: process: 0.56182 | load: 0.00247 [I 2022-10-26 22:51:07] ...trap/engines/engine.py.230: loss: 0.11606 [I 2022-10-26 22:51:07] ...optimizers/trijoint.py.151: total_norm: 11.323629 [I 2022-10-26 22:51:12] ...trap/engines/engine.py.224: train: epoch 34 | batch 370/2304 [I 2022-10-26 22:51:12] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:00 [I 2022-10-26 22:51:12] ...trap/engines/engine.py.229: process: 0.55814 | load: 0.00247 [I 2022-10-26 22:51:12] ...trap/engines/engine.py.230: loss: 0.23234 [I 2022-10-26 22:51:12] ...optimizers/trijoint.py.151: total_norm: 17.006693 [I 2022-10-26 22:51:18] ...trap/engines/engine.py.224: train: epoch 34 | batch 380/2304 [I 2022-10-26 22:51:18] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:49 [I 2022-10-26 22:51:18] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00253 [I 2022-10-26 22:51:18] ...trap/engines/engine.py.230: loss: 0.21880 [I 2022-10-26 22:51:18] ...optimizers/trijoint.py.151: total_norm: 38.051205 [I 2022-10-26 22:51:23] ...trap/engines/engine.py.224: train: epoch 34 | batch 390/2304 [I 2022-10-26 22:51:23] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:56 [I 2022-10-26 22:51:23] ...trap/engines/engine.py.229: process: 0.56246 | load: 0.00256 [I 2022-10-26 22:51:23] ...trap/engines/engine.py.230: loss: 0.18375 [I 2022-10-26 22:51:23] ...optimizers/trijoint.py.151: total_norm: 21.705364 [I 2022-10-26 22:51:29] ...trap/engines/engine.py.224: train: epoch 34 | batch 400/2304 [I 2022-10-26 22:51:29] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:33 [I 2022-10-26 22:51:29] ...trap/engines/engine.py.229: process: 0.54983 | load: 0.00244 [I 2022-10-26 22:51:29] ...trap/engines/engine.py.230: loss: 0.16733 [I 2022-10-26 22:51:29] ...optimizers/trijoint.py.151: total_norm: 25.196508 [I 2022-10-26 22:51:35] ...trap/engines/engine.py.224: train: epoch 34 | batch 410/2304 [I 2022-10-26 22:51:35] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:34 [I 2022-10-26 22:51:35] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00245 [I 2022-10-26 22:51:35] ...trap/engines/engine.py.230: loss: 0.12258 [I 2022-10-26 22:51:35] ...optimizers/trijoint.py.151: total_norm: 17.703558 [I 2022-10-26 22:51:40] ...trap/engines/engine.py.224: train: epoch 34 | batch 420/2304 [I 2022-10-26 22:51:40] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:24 [I 2022-10-26 22:51:40] ...trap/engines/engine.py.229: process: 0.56467 | load: 0.00246 [I 2022-10-26 22:51:40] ...trap/engines/engine.py.230: loss: 0.12305 [I 2022-10-26 22:51:40] ...optimizers/trijoint.py.151: total_norm: 8.200634 [I 2022-10-26 22:51:46] ...trap/engines/engine.py.224: train: epoch 34 | batch 430/2304 [I 2022-10-26 22:51:46] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:24 [I 2022-10-26 22:51:46] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00243 [I 2022-10-26 22:51:46] ...trap/engines/engine.py.230: loss: 0.21778 [I 2022-10-26 22:51:46] ...optimizers/trijoint.py.151: total_norm: 26.309801 [I 2022-10-26 22:51:51] ...trap/engines/engine.py.224: train: epoch 34 | batch 440/2304 [I 2022-10-26 22:51:51] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:23 [I 2022-10-26 22:51:51] ...trap/engines/engine.py.229: process: 0.57511 | load: 0.00247 [I 2022-10-26 22:51:51] ...trap/engines/engine.py.230: loss: 0.18129 [I 2022-10-26 22:51:51] ...optimizers/trijoint.py.151: total_norm: 23.691170 [I 2022-10-26 22:51:57] ...trap/engines/engine.py.224: train: epoch 34 | batch 450/2304 [I 2022-10-26 22:51:57] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:15 [I 2022-10-26 22:51:57] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00247 [I 2022-10-26 22:51:57] ...trap/engines/engine.py.230: loss: 0.19261 [I 2022-10-26 22:51:57] ...optimizers/trijoint.py.151: total_norm: 15.563924 [I 2022-10-26 22:52:03] ...trap/engines/engine.py.224: train: epoch 34 | batch 460/2304 [I 2022-10-26 22:52:03] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:05 [I 2022-10-26 22:52:03] ...trap/engines/engine.py.229: process: 0.56478 | load: 0.00248 [I 2022-10-26 22:52:03] ...trap/engines/engine.py.230: loss: 0.16448 [I 2022-10-26 22:52:03] ...optimizers/trijoint.py.151: total_norm: 21.086901 [I 2022-10-26 22:52:08] ...trap/engines/engine.py.224: train: epoch 34 | batch 470/2304 [I 2022-10-26 22:52:08] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:55 [I 2022-10-26 22:52:08] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00245 [I 2022-10-26 22:52:08] ...trap/engines/engine.py.230: loss: 0.14529 [I 2022-10-26 22:52:08] ...optimizers/trijoint.py.151: total_norm: 22.103228 [I 2022-10-26 22:52:14] ...trap/engines/engine.py.224: train: epoch 34 | batch 480/2304 [I 2022-10-26 22:52:14] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:46 [I 2022-10-26 22:52:14] ...trap/engines/engine.py.229: process: 0.54081 | load: 0.00240 [I 2022-10-26 22:52:14] ...trap/engines/engine.py.230: loss: 0.17837 [I 2022-10-26 22:52:14] ...optimizers/trijoint.py.151: total_norm: 11.624549 [I 2022-10-26 22:52:19] ...trap/engines/engine.py.224: train: epoch 34 | batch 490/2304 [I 2022-10-26 22:52:19] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:58 [I 2022-10-26 22:52:19] ...trap/engines/engine.py.229: process: 0.56748 | load: 0.00248 [I 2022-10-26 22:52:19] ...trap/engines/engine.py.230: loss: 0.13622 [I 2022-10-26 22:52:19] ...optimizers/trijoint.py.151: total_norm: 19.946230 [I 2022-10-26 22:52:25] ...trap/engines/engine.py.224: train: epoch 34 | batch 500/2304 [I 2022-10-26 22:52:25] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:44 [I 2022-10-26 22:52:25] ...trap/engines/engine.py.229: process: 0.54203 | load: 0.00252 [I 2022-10-26 22:52:25] ...trap/engines/engine.py.230: loss: 0.16358 [I 2022-10-26 22:52:25] ...optimizers/trijoint.py.151: total_norm: 15.527283 [I 2022-10-26 22:52:30] ...trap/engines/engine.py.224: train: epoch 34 | batch 510/2304 [I 2022-10-26 22:52:30] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:41 [I 2022-10-26 22:52:30] ...trap/engines/engine.py.229: process: 0.56939 | load: 0.00246 [I 2022-10-26 22:52:30] ...trap/engines/engine.py.230: loss: 0.17684 [I 2022-10-26 22:52:30] ...optimizers/trijoint.py.151: total_norm: 19.254719 [I 2022-10-26 22:52:36] ...trap/engines/engine.py.224: train: epoch 34 | batch 520/2304 [I 2022-10-26 22:52:36] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:32 [I 2022-10-26 22:52:36] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00245 [I 2022-10-26 22:52:36] ...trap/engines/engine.py.230: loss: 0.10279 [I 2022-10-26 22:52:36] ...optimizers/trijoint.py.151: total_norm: 13.067716 [I 2022-10-26 22:52:42] ...trap/engines/engine.py.224: train: epoch 34 | batch 530/2304 [I 2022-10-26 22:52:42] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:33 [I 2022-10-26 22:52:42] ...trap/engines/engine.py.229: process: 0.55926 | load: 0.00245 [I 2022-10-26 22:52:42] ...trap/engines/engine.py.230: loss: 0.20735 [I 2022-10-26 22:52:42] ...optimizers/trijoint.py.151: total_norm: 19.016455 [I 2022-10-26 22:52:47] ...trap/engines/engine.py.224: train: epoch 34 | batch 540/2304 [I 2022-10-26 22:52:47] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:23 [I 2022-10-26 22:52:47] ...trap/engines/engine.py.229: process: 0.56684 | load: 0.00250 [I 2022-10-26 22:52:47] ...trap/engines/engine.py.230: loss: 0.11144 [I 2022-10-26 22:52:47] ...optimizers/trijoint.py.151: total_norm: 13.150180 [I 2022-10-26 22:52:53] ...trap/engines/engine.py.224: train: epoch 34 | batch 550/2304 [I 2022-10-26 22:52:53] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:19 [I 2022-10-26 22:52:53] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00241 [I 2022-10-26 22:52:53] ...trap/engines/engine.py.230: loss: 0.19769 [I 2022-10-26 22:52:53] ...optimizers/trijoint.py.151: total_norm: 20.356012 [I 2022-10-26 22:52:58] ...trap/engines/engine.py.224: train: epoch 34 | batch 560/2304 [I 2022-10-26 22:52:58] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:08 [I 2022-10-26 22:52:58] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00268 [I 2022-10-26 22:52:58] ...trap/engines/engine.py.230: loss: 0.19902 [I 2022-10-26 22:52:58] ...optimizers/trijoint.py.151: total_norm: 15.593070 [I 2022-10-26 22:53:04] ...trap/engines/engine.py.224: train: epoch 34 | batch 570/2304 [I 2022-10-26 22:53:04] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:06 [I 2022-10-26 22:53:04] ...trap/engines/engine.py.229: process: 0.55174 | load: 0.00248 [I 2022-10-26 22:53:04] ...trap/engines/engine.py.230: loss: 0.15862 [I 2022-10-26 22:53:04] ...optimizers/trijoint.py.151: total_norm: 19.698441 [I 2022-10-26 22:53:09] ...trap/engines/engine.py.224: train: epoch 34 | batch 580/2304 [I 2022-10-26 22:53:09] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:57 [I 2022-10-26 22:53:09] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00248 [I 2022-10-26 22:53:09] ...trap/engines/engine.py.230: loss: 0.19499 [I 2022-10-26 22:53:09] ...optimizers/trijoint.py.151: total_norm: 24.751970 [I 2022-10-26 22:53:15] ...trap/engines/engine.py.224: train: epoch 34 | batch 590/2304 [I 2022-10-26 22:53:15] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:53 [I 2022-10-26 22:53:15] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00311 [I 2022-10-26 22:53:15] ...trap/engines/engine.py.230: loss: 0.14143 [I 2022-10-26 22:53:15] ...optimizers/trijoint.py.151: total_norm: 21.441233 [I 2022-10-26 22:53:21] ...trap/engines/engine.py.224: train: epoch 34 | batch 600/2304 [I 2022-10-26 22:53:21] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:51 [I 2022-10-26 22:53:21] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00246 [I 2022-10-26 22:53:21] ...trap/engines/engine.py.230: loss: 0.13556 [I 2022-10-26 22:53:21] ...optimizers/trijoint.py.151: total_norm: 16.605249 [I 2022-10-26 22:53:26] ...trap/engines/engine.py.224: train: epoch 34 | batch 610/2304 [I 2022-10-26 22:53:26] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:42 [I 2022-10-26 22:53:26] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00248 [I 2022-10-26 22:53:26] ...trap/engines/engine.py.230: loss: 0.17011 [I 2022-10-26 22:53:26] ...optimizers/trijoint.py.151: total_norm: 15.547028 [I 2022-10-26 22:53:32] ...trap/engines/engine.py.224: train: epoch 34 | batch 620/2304 [I 2022-10-26 22:53:32] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:37 [I 2022-10-26 22:53:32] ...trap/engines/engine.py.229: process: 0.56238 | load: 0.00503 [I 2022-10-26 22:53:32] ...trap/engines/engine.py.230: loss: 0.19606 [I 2022-10-26 22:53:32] ...optimizers/trijoint.py.151: total_norm: 18.300795 [I 2022-10-26 22:53:37] ...trap/engines/engine.py.224: train: epoch 34 | batch 630/2304 [I 2022-10-26 22:53:37] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:29 [I 2022-10-26 22:53:37] ...trap/engines/engine.py.229: process: 0.54738 | load: 0.00247 [I 2022-10-26 22:53:37] ...trap/engines/engine.py.230: loss: 0.19280 [I 2022-10-26 22:53:37] ...optimizers/trijoint.py.151: total_norm: 15.701107 [I 2022-10-26 22:53:43] ...trap/engines/engine.py.224: train: epoch 34 | batch 640/2304 [I 2022-10-26 22:53:43] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:28 [I 2022-10-26 22:53:43] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00246 [I 2022-10-26 22:53:43] ...trap/engines/engine.py.230: loss: 0.20585 [I 2022-10-26 22:53:43] ...optimizers/trijoint.py.151: total_norm: 38.056450 [I 2022-10-26 22:53:49] ...trap/engines/engine.py.224: train: epoch 34 | batch 650/2304 [I 2022-10-26 22:53:49] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:24 [I 2022-10-26 22:53:49] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00261 [I 2022-10-26 22:53:49] ...trap/engines/engine.py.230: loss: 0.15272 [I 2022-10-26 22:53:49] ...optimizers/trijoint.py.151: total_norm: 21.702614 [I 2022-10-26 22:53:54] ...trap/engines/engine.py.224: train: epoch 34 | batch 660/2304 [I 2022-10-26 22:53:54] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:19 [I 2022-10-26 22:53:54] ...trap/engines/engine.py.229: process: 0.56193 | load: 0.00248 [I 2022-10-26 22:53:54] ...trap/engines/engine.py.230: loss: 0.23820 [I 2022-10-26 22:53:54] ...optimizers/trijoint.py.151: total_norm: 15.376219 [I 2022-10-26 22:54:00] ...trap/engines/engine.py.224: train: epoch 34 | batch 670/2304 [I 2022-10-26 22:54:00] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:12 [I 2022-10-26 22:54:00] ...trap/engines/engine.py.229: process: 0.56283 | load: 0.00267 [I 2022-10-26 22:54:00] ...trap/engines/engine.py.230: loss: 0.24223 [I 2022-10-26 22:54:00] ...optimizers/trijoint.py.151: total_norm: 17.622122 [I 2022-10-26 22:54:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 680/2304 [I 2022-10-26 22:54:05] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:04 [I 2022-10-26 22:54:05] ...trap/engines/engine.py.229: process: 0.55986 | load: 0.00244 [I 2022-10-26 22:54:05] ...trap/engines/engine.py.230: loss: 0.18390 [I 2022-10-26 22:54:05] ...optimizers/trijoint.py.151: total_norm: 20.818432 [I 2022-10-26 22:54:11] ...trap/engines/engine.py.224: train: epoch 34 | batch 690/2304 [I 2022-10-26 22:54:11] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:52 [I 2022-10-26 22:54:11] ...trap/engines/engine.py.229: process: 0.54611 | load: 0.00243 [I 2022-10-26 22:54:11] ...trap/engines/engine.py.230: loss: 0.13959 [I 2022-10-26 22:54:11] ...optimizers/trijoint.py.151: total_norm: 11.145799 [I 2022-10-26 22:54:16] ...trap/engines/engine.py.224: train: epoch 34 | batch 700/2304 [I 2022-10-26 22:54:16] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:49 [I 2022-10-26 22:54:16] ...trap/engines/engine.py.229: process: 0.53720 | load: 0.00240 [I 2022-10-26 22:54:16] ...trap/engines/engine.py.230: loss: 0.16653 [I 2022-10-26 22:54:16] ...optimizers/trijoint.py.151: total_norm: 15.345387 [I 2022-10-26 22:54:22] ...trap/engines/engine.py.224: train: epoch 34 | batch 710/2304 [I 2022-10-26 22:54:22] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:44 [I 2022-10-26 22:54:22] ...trap/engines/engine.py.229: process: 0.54439 | load: 0.00244 [I 2022-10-26 22:54:22] ...trap/engines/engine.py.230: loss: 0.20087 [I 2022-10-26 22:54:22] ...optimizers/trijoint.py.151: total_norm: 20.714350 [I 2022-10-26 22:54:28] ...trap/engines/engine.py.224: train: epoch 34 | batch 720/2304 [I 2022-10-26 22:54:28] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:40 [I 2022-10-26 22:54:28] ...trap/engines/engine.py.229: process: 0.56457 | load: 0.00240 [I 2022-10-26 22:54:28] ...trap/engines/engine.py.230: loss: 0.16706 [I 2022-10-26 22:54:28] ...optimizers/trijoint.py.151: total_norm: 15.663140 [I 2022-10-26 22:54:33] ...trap/engines/engine.py.224: train: epoch 34 | batch 730/2304 [I 2022-10-26 22:54:33] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:36 [I 2022-10-26 22:54:33] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00240 [I 2022-10-26 22:54:33] ...trap/engines/engine.py.230: loss: 0.17816 [I 2022-10-26 22:54:33] ...optimizers/trijoint.py.151: total_norm: 17.124056 [I 2022-10-26 22:54:39] ...trap/engines/engine.py.224: train: epoch 34 | batch 740/2304 [I 2022-10-26 22:54:39] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:24 [I 2022-10-26 22:54:39] ...trap/engines/engine.py.229: process: 0.54699 | load: 0.00246 [I 2022-10-26 22:54:39] ...trap/engines/engine.py.230: loss: 0.19731 [I 2022-10-26 22:54:39] ...optimizers/trijoint.py.151: total_norm: 22.105299 [I 2022-10-26 22:54:44] ...trap/engines/engine.py.224: train: epoch 34 | batch 750/2304 [I 2022-10-26 22:54:44] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:24 [I 2022-10-26 22:54:44] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00246 [I 2022-10-26 22:54:44] ...trap/engines/engine.py.230: loss: 0.23949 [I 2022-10-26 22:54:44] ...optimizers/trijoint.py.151: total_norm: 46.975063 [I 2022-10-26 22:54:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 760/2304 [I 2022-10-26 22:54:50] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:09 [I 2022-10-26 22:54:50] ...trap/engines/engine.py.229: process: 0.54918 | load: 0.00244 [I 2022-10-26 22:54:50] ...trap/engines/engine.py.230: loss: 0.21409 [I 2022-10-26 22:54:50] ...optimizers/trijoint.py.151: total_norm: 35.379108 [I 2022-10-26 22:54:55] ...trap/engines/engine.py.224: train: epoch 34 | batch 770/2304 [I 2022-10-26 22:54:55] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:10 [I 2022-10-26 22:54:55] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00243 [I 2022-10-26 22:54:55] ...trap/engines/engine.py.230: loss: 0.17849 [I 2022-10-26 22:54:55] ...optimizers/trijoint.py.151: total_norm: 40.322281 [I 2022-10-26 22:55:01] ...trap/engines/engine.py.224: train: epoch 34 | batch 780/2304 [I 2022-10-26 22:55:01] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:14 [I 2022-10-26 22:55:01] ...trap/engines/engine.py.229: process: 0.56988 | load: 0.00239 [I 2022-10-26 22:55:01] ...trap/engines/engine.py.230: loss: 0.20559 [I 2022-10-26 22:55:01] ...optimizers/trijoint.py.151: total_norm: 23.218723 [I 2022-10-26 22:55:06] ...trap/engines/engine.py.224: train: epoch 34 | batch 790/2304 [I 2022-10-26 22:55:06] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:13:56 [I 2022-10-26 22:55:06] ...trap/engines/engine.py.229: process: 0.54255 | load: 0.00241 [I 2022-10-26 22:55:06] ...trap/engines/engine.py.230: loss: 0.19145 [I 2022-10-26 22:55:06] ...optimizers/trijoint.py.151: total_norm: 13.554687 [I 2022-10-26 22:55:12] ...trap/engines/engine.py.224: train: epoch 34 | batch 800/2304 [I 2022-10-26 22:55:12] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:56 [I 2022-10-26 22:55:12] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00242 [I 2022-10-26 22:55:12] ...trap/engines/engine.py.230: loss: 0.18287 [I 2022-10-26 22:55:12] ...optimizers/trijoint.py.151: total_norm: 14.589545 [I 2022-10-26 22:55:18] ...trap/engines/engine.py.224: train: epoch 34 | batch 810/2304 [I 2022-10-26 22:55:18] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:45 [I 2022-10-26 22:55:18] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00241 [I 2022-10-26 22:55:18] ...trap/engines/engine.py.230: loss: 0.16136 [I 2022-10-26 22:55:18] ...optimizers/trijoint.py.151: total_norm: 14.497915 [I 2022-10-26 22:55:23] ...trap/engines/engine.py.224: train: epoch 34 | batch 820/2304 [I 2022-10-26 22:55:23] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:44 [I 2022-10-26 22:55:23] ...trap/engines/engine.py.229: process: 0.54974 | load: 0.00245 [I 2022-10-26 22:55:23] ...trap/engines/engine.py.230: loss: 0.19829 [I 2022-10-26 22:55:23] ...optimizers/trijoint.py.151: total_norm: 19.853657 [I 2022-10-26 22:55:29] ...trap/engines/engine.py.224: train: epoch 34 | batch 830/2304 [I 2022-10-26 22:55:29] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:38 [I 2022-10-26 22:55:29] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00255 [I 2022-10-26 22:55:29] ...trap/engines/engine.py.230: loss: 0.16675 [I 2022-10-26 22:55:29] ...optimizers/trijoint.py.151: total_norm: 17.372736 [I 2022-10-26 22:55:34] ...trap/engines/engine.py.224: train: epoch 34 | batch 840/2304 [I 2022-10-26 22:55:34] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:50 [I 2022-10-26 22:55:34] ...trap/engines/engine.py.229: process: 0.57232 | load: 0.00248 [I 2022-10-26 22:55:34] ...trap/engines/engine.py.230: loss: 0.17069 [I 2022-10-26 22:55:34] ...optimizers/trijoint.py.151: total_norm: 12.205595 [I 2022-10-26 22:55:40] ...trap/engines/engine.py.224: train: epoch 34 | batch 850/2304 [I 2022-10-26 22:55:40] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:28 [I 2022-10-26 22:55:40] ...trap/engines/engine.py.229: process: 0.53955 | load: 0.00252 [I 2022-10-26 22:55:40] ...trap/engines/engine.py.230: loss: 0.15976 [I 2022-10-26 22:55:40] ...optimizers/trijoint.py.151: total_norm: 27.223789 [I 2022-10-26 22:55:46] ...trap/engines/engine.py.224: train: epoch 34 | batch 860/2304 [I 2022-10-26 22:55:46] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:26 [I 2022-10-26 22:55:46] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00244 [I 2022-10-26 22:55:46] ...trap/engines/engine.py.230: loss: 0.16800 [I 2022-10-26 22:55:46] ...optimizers/trijoint.py.151: total_norm: 16.989193 [I 2022-10-26 22:55:51] ...trap/engines/engine.py.224: train: epoch 34 | batch 870/2304 [I 2022-10-26 22:55:51] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:19 [I 2022-10-26 22:55:51] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00242 [I 2022-10-26 22:55:51] ...trap/engines/engine.py.230: loss: 0.20025 [I 2022-10-26 22:55:51] ...optimizers/trijoint.py.151: total_norm: 25.647032 [I 2022-10-26 22:55:57] ...trap/engines/engine.py.224: train: epoch 34 | batch 880/2304 [I 2022-10-26 22:55:57] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:14 [I 2022-10-26 22:55:57] ...trap/engines/engine.py.229: process: 0.56220 | load: 0.00243 [I 2022-10-26 22:55:57] ...trap/engines/engine.py.230: loss: 0.22739 [I 2022-10-26 22:55:57] ...optimizers/trijoint.py.151: total_norm: 19.307280 [I 2022-10-26 22:56:02] ...trap/engines/engine.py.224: train: epoch 34 | batch 890/2304 [I 2022-10-26 22:56:02] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:01 [I 2022-10-26 22:56:02] ...trap/engines/engine.py.229: process: 0.55337 | load: 0.00246 [I 2022-10-26 22:56:02] ...trap/engines/engine.py.230: loss: 0.14040 [I 2022-10-26 22:56:02] ...optimizers/trijoint.py.151: total_norm: 9.096680 [I 2022-10-26 22:56:08] ...trap/engines/engine.py.224: train: epoch 34 | batch 900/2304 [I 2022-10-26 22:56:08] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:12:51 [I 2022-10-26 22:56:08] ...trap/engines/engine.py.229: process: 0.54616 | load: 0.00243 [I 2022-10-26 22:56:08] ...trap/engines/engine.py.230: loss: 0.15303 [I 2022-10-26 22:56:08] ...optimizers/trijoint.py.151: total_norm: 11.213012 [I 2022-10-26 22:56:13] ...trap/engines/engine.py.224: train: epoch 34 | batch 910/2304 [I 2022-10-26 22:56:13] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:50 [I 2022-10-26 22:56:13] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00250 [I 2022-10-26 22:56:13] ...trap/engines/engine.py.230: loss: 0.11617 [I 2022-10-26 22:56:13] ...optimizers/trijoint.py.151: total_norm: 11.394989 [I 2022-10-26 22:56:19] ...trap/engines/engine.py.224: train: epoch 34 | batch 920/2304 [I 2022-10-26 22:56:19] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:45 [I 2022-10-26 22:56:19] ...trap/engines/engine.py.229: process: 0.54316 | load: 0.00244 [I 2022-10-26 22:56:19] ...trap/engines/engine.py.230: loss: 0.12302 [I 2022-10-26 22:56:19] ...optimizers/trijoint.py.151: total_norm: 12.531281 [I 2022-10-26 22:56:25] ...trap/engines/engine.py.224: train: epoch 34 | batch 930/2304 [I 2022-10-26 22:56:25] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:13:03 [I 2022-10-26 22:56:25] ...trap/engines/engine.py.229: process: 0.54901 | load: 0.00246 [I 2022-10-26 22:56:25] ...trap/engines/engine.py.230: loss: 0.12382 [I 2022-10-26 22:56:25] ...optimizers/trijoint.py.151: total_norm: 21.486481 [I 2022-10-26 22:56:30] ...trap/engines/engine.py.224: train: epoch 34 | batch 940/2304 [I 2022-10-26 22:56:30] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:40 [I 2022-10-26 22:56:30] ...trap/engines/engine.py.229: process: 0.55931 | load: 0.00243 [I 2022-10-26 22:56:30] ...trap/engines/engine.py.230: loss: 0.21591 [I 2022-10-26 22:56:30] ...optimizers/trijoint.py.151: total_norm: 21.750185 [I 2022-10-26 22:56:36] ...trap/engines/engine.py.224: train: epoch 34 | batch 950/2304 [I 2022-10-26 22:56:36] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:33 [I 2022-10-26 22:56:36] ...trap/engines/engine.py.229: process: 0.55571 | load: 0.00242 [I 2022-10-26 22:56:36] ...trap/engines/engine.py.230: loss: 0.12708 [I 2022-10-26 22:56:36] ...optimizers/trijoint.py.151: total_norm: 18.986328 [I 2022-10-26 22:56:41] ...trap/engines/engine.py.224: train: epoch 34 | batch 960/2304 [I 2022-10-26 22:56:41] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:27 [I 2022-10-26 22:56:41] ...trap/engines/engine.py.229: process: 0.55758 | load: 0.00243 [I 2022-10-26 22:56:41] ...trap/engines/engine.py.230: loss: 0.14040 [I 2022-10-26 22:56:41] ...optimizers/trijoint.py.151: total_norm: 15.023322 [I 2022-10-26 22:56:47] ...trap/engines/engine.py.224: train: epoch 34 | batch 970/2304 [I 2022-10-26 22:56:47] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:18 [I 2022-10-26 22:56:47] ...trap/engines/engine.py.229: process: 0.54938 | load: 0.00259 [I 2022-10-26 22:56:47] ...trap/engines/engine.py.230: loss: 0.23285 [I 2022-10-26 22:56:47] ...optimizers/trijoint.py.151: total_norm: 24.909601 [I 2022-10-26 22:56:52] ...trap/engines/engine.py.224: train: epoch 34 | batch 980/2304 [I 2022-10-26 22:56:52] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:12 [I 2022-10-26 22:56:52] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00243 [I 2022-10-26 22:56:52] ...trap/engines/engine.py.230: loss: 0.20468 [I 2022-10-26 22:56:52] ...optimizers/trijoint.py.151: total_norm: 21.793678 [I 2022-10-26 22:56:58] ...trap/engines/engine.py.224: train: epoch 34 | batch 990/2304 [I 2022-10-26 22:56:58] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:06 [I 2022-10-26 22:56:58] ...trap/engines/engine.py.229: process: 0.55711 | load: 0.00239 [I 2022-10-26 22:56:58] ...trap/engines/engine.py.230: loss: 0.11602 [I 2022-10-26 22:56:58] ...optimizers/trijoint.py.151: total_norm: 18.422760 [I 2022-10-26 22:57:04] ...trap/engines/engine.py.224: train: epoch 34 | batch 1000/2304 [I 2022-10-26 22:57:04] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:11 [I 2022-10-26 22:57:04] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00242 [I 2022-10-26 22:57:04] ...trap/engines/engine.py.230: loss: 0.30236 [I 2022-10-26 22:57:04] ...optimizers/trijoint.py.151: total_norm: 22.131641 [I 2022-10-26 22:57:09] ...trap/engines/engine.py.224: train: epoch 34 | batch 1010/2304 [I 2022-10-26 22:57:09] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:03 [I 2022-10-26 22:57:09] ...trap/engines/engine.py.229: process: 0.55059 | load: 0.00242 [I 2022-10-26 22:57:09] ...trap/engines/engine.py.230: loss: 0.11383 [I 2022-10-26 22:57:09] ...optimizers/trijoint.py.151: total_norm: 15.119800 [I 2022-10-26 22:57:15] ...trap/engines/engine.py.224: train: epoch 34 | batch 1020/2304 [I 2022-10-26 22:57:15] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:52 [I 2022-10-26 22:57:15] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00248 [I 2022-10-26 22:57:15] ...trap/engines/engine.py.230: loss: 0.17409 [I 2022-10-26 22:57:15] ...optimizers/trijoint.py.151: total_norm: 28.488123 [I 2022-10-26 22:57:20] ...trap/engines/engine.py.224: train: epoch 34 | batch 1030/2304 [I 2022-10-26 22:57:20] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:46 [I 2022-10-26 22:57:20] ...trap/engines/engine.py.229: process: 0.55265 | load: 0.00245 [I 2022-10-26 22:57:20] ...trap/engines/engine.py.230: loss: 0.16414 [I 2022-10-26 22:57:20] ...optimizers/trijoint.py.151: total_norm: 9.713311 [I 2022-10-26 22:57:26] ...trap/engines/engine.py.224: train: epoch 34 | batch 1040/2304 [I 2022-10-26 22:57:26] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:44 [I 2022-10-26 22:57:26] ...trap/engines/engine.py.229: process: 0.55061 | load: 0.00246 [I 2022-10-26 22:57:26] ...trap/engines/engine.py.230: loss: 0.18043 [I 2022-10-26 22:57:26] ...optimizers/trijoint.py.151: total_norm: 21.192434 [I 2022-10-26 22:57:31] ...trap/engines/engine.py.224: train: epoch 34 | batch 1050/2304 [I 2022-10-26 22:57:31] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:39 [I 2022-10-26 22:57:31] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00243 [I 2022-10-26 22:57:31] ...trap/engines/engine.py.230: loss: 0.25502 [I 2022-10-26 22:57:31] ...optimizers/trijoint.py.151: total_norm: 26.295176 [I 2022-10-26 22:57:37] ...trap/engines/engine.py.224: train: epoch 34 | batch 1060/2304 [I 2022-10-26 22:57:37] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:33 [I 2022-10-26 22:57:37] ...trap/engines/engine.py.229: process: 0.56489 | load: 0.00239 [I 2022-10-26 22:57:37] ...trap/engines/engine.py.230: loss: 0.15755 [I 2022-10-26 22:57:37] ...optimizers/trijoint.py.151: total_norm: 18.138552 [I 2022-10-26 22:57:43] ...trap/engines/engine.py.224: train: epoch 34 | batch 1070/2304 [I 2022-10-26 22:57:43] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:24 [I 2022-10-26 22:57:43] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00242 [I 2022-10-26 22:57:43] ...trap/engines/engine.py.230: loss: 0.17722 [I 2022-10-26 22:57:43] ...optimizers/trijoint.py.151: total_norm: 19.920622 [I 2022-10-26 22:57:48] ...trap/engines/engine.py.224: train: epoch 34 | batch 1080/2304 [I 2022-10-26 22:57:48] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:19 [I 2022-10-26 22:57:48] ...trap/engines/engine.py.229: process: 0.56843 | load: 0.00263 [I 2022-10-26 22:57:48] ...trap/engines/engine.py.230: loss: 0.20938 [I 2022-10-26 22:57:48] ...optimizers/trijoint.py.151: total_norm: 17.267252 [I 2022-10-26 22:57:54] ...trap/engines/engine.py.224: train: epoch 34 | batch 1090/2304 [I 2022-10-26 22:57:54] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:15 [I 2022-10-26 22:57:54] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00261 [I 2022-10-26 22:57:54] ...trap/engines/engine.py.230: loss: 0.14953 [I 2022-10-26 22:57:54] ...optimizers/trijoint.py.151: total_norm: 13.644461 [I 2022-10-26 22:57:59] ...trap/engines/engine.py.224: train: epoch 34 | batch 1100/2304 [I 2022-10-26 22:57:59] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:11 [I 2022-10-26 22:57:59] ...trap/engines/engine.py.229: process: 0.54878 | load: 0.00240 [I 2022-10-26 22:57:59] ...trap/engines/engine.py.230: loss: 0.15816 [I 2022-10-26 22:57:59] ...optimizers/trijoint.py.151: total_norm: 22.722801 [I 2022-10-26 22:58:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 1110/2304 [I 2022-10-26 22:58:05] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:23 [I 2022-10-26 22:58:05] ...trap/engines/engine.py.229: process: 0.54762 | load: 0.00243 [I 2022-10-26 22:58:05] ...trap/engines/engine.py.230: loss: 0.22802 [I 2022-10-26 22:58:05] ...optimizers/trijoint.py.151: total_norm: 15.297462 [I 2022-10-26 22:58:11] ...trap/engines/engine.py.224: train: epoch 34 | batch 1120/2304 [I 2022-10-26 22:58:11] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:10:59 [I 2022-10-26 22:58:11] ...trap/engines/engine.py.229: process: 0.54329 | load: 0.00248 [I 2022-10-26 22:58:11] ...trap/engines/engine.py.230: loss: 0.19409 [I 2022-10-26 22:58:11] ...optimizers/trijoint.py.151: total_norm: 16.695938 [I 2022-10-26 22:58:16] ...trap/engines/engine.py.224: train: epoch 34 | batch 1130/2304 [I 2022-10-26 22:58:16] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:52 [I 2022-10-26 22:58:16] ...trap/engines/engine.py.229: process: 0.55486 | load: 0.00242 [I 2022-10-26 22:58:16] ...trap/engines/engine.py.230: loss: 0.15540 [I 2022-10-26 22:58:16] ...optimizers/trijoint.py.151: total_norm: 16.344131 [I 2022-10-26 22:58:22] ...trap/engines/engine.py.224: train: epoch 34 | batch 1140/2304 [I 2022-10-26 22:58:22] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:46 [I 2022-10-26 22:58:22] ...trap/engines/engine.py.229: process: 0.55102 | load: 0.00244 [I 2022-10-26 22:58:22] ...trap/engines/engine.py.230: loss: 0.25199 [I 2022-10-26 22:58:22] ...optimizers/trijoint.py.151: total_norm: 38.216698 [I 2022-10-26 22:58:27] ...trap/engines/engine.py.224: train: epoch 34 | batch 1150/2304 [I 2022-10-26 22:58:27] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:41 [I 2022-10-26 22:58:27] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00243 [I 2022-10-26 22:58:27] ...trap/engines/engine.py.230: loss: 0.20592 [I 2022-10-26 22:58:27] ...optimizers/trijoint.py.151: total_norm: 24.386129 [I 2022-10-26 22:58:33] ...trap/engines/engine.py.224: train: epoch 34 | batch 1160/2304 [I 2022-10-26 22:58:33] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:35 [I 2022-10-26 22:58:33] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00241 [I 2022-10-26 22:58:33] ...trap/engines/engine.py.230: loss: 0.20271 [I 2022-10-26 22:58:33] ...optimizers/trijoint.py.151: total_norm: 17.472725 [I 2022-10-26 22:58:38] ...trap/engines/engine.py.224: train: epoch 34 | batch 1170/2304 [I 2022-10-26 22:58:38] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:31 [I 2022-10-26 22:58:38] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00241 [I 2022-10-26 22:58:38] ...trap/engines/engine.py.230: loss: 0.20182 [I 2022-10-26 22:58:38] ...optimizers/trijoint.py.151: total_norm: 17.353724 [I 2022-10-26 22:58:44] ...trap/engines/engine.py.224: train: epoch 34 | batch 1180/2304 [I 2022-10-26 22:58:44] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:20 [I 2022-10-26 22:58:44] ...trap/engines/engine.py.229: process: 0.54264 | load: 0.00241 [I 2022-10-26 22:58:44] ...trap/engines/engine.py.230: loss: 0.17871 [I 2022-10-26 22:58:44] ...optimizers/trijoint.py.151: total_norm: 21.112568 [I 2022-10-26 22:58:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 1190/2304 [I 2022-10-26 22:58:50] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:18 [I 2022-10-26 22:58:50] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00243 [I 2022-10-26 22:58:50] ...trap/engines/engine.py.230: loss: 0.12291 [I 2022-10-26 22:58:50] ...optimizers/trijoint.py.151: total_norm: 13.742639 [I 2022-10-26 22:58:55] ...trap/engines/engine.py.224: train: epoch 34 | batch 1200/2304 [I 2022-10-26 22:58:55] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:12 [I 2022-10-26 22:58:55] ...trap/engines/engine.py.229: process: 0.55263 | load: 0.00239 [I 2022-10-26 22:58:55] ...trap/engines/engine.py.230: loss: 0.15437 [I 2022-10-26 22:58:55] ...optimizers/trijoint.py.151: total_norm: 14.665823 [I 2022-10-26 22:59:01] ...trap/engines/engine.py.224: train: epoch 34 | batch 1210/2304 [I 2022-10-26 22:59:01] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:06 [I 2022-10-26 22:59:01] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00242 [I 2022-10-26 22:59:01] ...trap/engines/engine.py.230: loss: 0.17321 [I 2022-10-26 22:59:01] ...optimizers/trijoint.py.151: total_norm: 33.768478 [I 2022-10-26 22:59:06] ...trap/engines/engine.py.224: train: epoch 34 | batch 1220/2304 [I 2022-10-26 22:59:06] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:02 [I 2022-10-26 22:59:06] ...trap/engines/engine.py.229: process: 0.55033 | load: 0.00241 [I 2022-10-26 22:59:06] ...trap/engines/engine.py.230: loss: 0.23538 [I 2022-10-26 22:59:06] ...optimizers/trijoint.py.151: total_norm: 32.834606 [I 2022-10-26 22:59:12] ...trap/engines/engine.py.224: train: epoch 34 | batch 1230/2304 [I 2022-10-26 22:59:12] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:09:52 [I 2022-10-26 22:59:12] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00250 [I 2022-10-26 22:59:12] ...trap/engines/engine.py.230: loss: 0.13444 [I 2022-10-26 22:59:12] ...optimizers/trijoint.py.151: total_norm: 9.198229 [I 2022-10-26 22:59:17] ...trap/engines/engine.py.224: train: epoch 34 | batch 1240/2304 [I 2022-10-26 22:59:17] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:51 [I 2022-10-26 22:59:17] ...trap/engines/engine.py.229: process: 0.56461 | load: 0.00246 [I 2022-10-26 22:59:17] ...trap/engines/engine.py.230: loss: 0.17300 [I 2022-10-26 22:59:17] ...optimizers/trijoint.py.151: total_norm: 11.413651 [I 2022-10-26 22:59:23] ...trap/engines/engine.py.224: train: epoch 34 | batch 1250/2304 [I 2022-10-26 22:59:23] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:09:42 [I 2022-10-26 22:59:23] ...trap/engines/engine.py.229: process: 0.53925 | load: 0.00246 [I 2022-10-26 22:59:23] ...trap/engines/engine.py.230: loss: 0.15418 [I 2022-10-26 22:59:23] ...optimizers/trijoint.py.151: total_norm: 15.506448 [I 2022-10-26 22:59:29] ...trap/engines/engine.py.224: train: epoch 34 | batch 1260/2304 [I 2022-10-26 22:59:29] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:38 [I 2022-10-26 22:59:29] ...trap/engines/engine.py.229: process: 0.54818 | load: 0.00246 [I 2022-10-26 22:59:29] ...trap/engines/engine.py.230: loss: 0.16872 [I 2022-10-26 22:59:29] ...optimizers/trijoint.py.151: total_norm: 15.288338 [I 2022-10-26 22:59:34] ...trap/engines/engine.py.224: train: epoch 34 | batch 1270/2304 [I 2022-10-26 22:59:34] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:31 [I 2022-10-26 22:59:34] ...trap/engines/engine.py.229: process: 0.54648 | load: 0.00240 [I 2022-10-26 22:59:34] ...trap/engines/engine.py.230: loss: 0.15238 [I 2022-10-26 22:59:34] ...optimizers/trijoint.py.151: total_norm: 15.778132 [I 2022-10-26 22:59:40] ...trap/engines/engine.py.224: train: epoch 34 | batch 1280/2304 [I 2022-10-26 22:59:40] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:30 [I 2022-10-26 22:59:40] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00245 [I 2022-10-26 22:59:40] ...trap/engines/engine.py.230: loss: 0.18017 [I 2022-10-26 22:59:40] ...optimizers/trijoint.py.151: total_norm: 14.193326 [I 2022-10-26 22:59:45] ...trap/engines/engine.py.224: train: epoch 34 | batch 1290/2304 [I 2022-10-26 22:59:45] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:20 [I 2022-10-26 22:59:45] ...trap/engines/engine.py.229: process: 0.55302 | load: 0.00244 [I 2022-10-26 22:59:45] ...trap/engines/engine.py.230: loss: 0.17184 [I 2022-10-26 22:59:45] ...optimizers/trijoint.py.151: total_norm: 17.006529 [I 2022-10-26 22:59:51] ...trap/engines/engine.py.224: train: epoch 34 | batch 1300/2304 [I 2022-10-26 22:59:51] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:16 [I 2022-10-26 22:59:51] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00241 [I 2022-10-26 22:59:51] ...trap/engines/engine.py.230: loss: 0.23536 [I 2022-10-26 22:59:51] ...optimizers/trijoint.py.151: total_norm: 21.995560 [I 2022-10-26 22:59:56] ...trap/engines/engine.py.224: train: epoch 34 | batch 1310/2304 [I 2022-10-26 22:59:56] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:12 [I 2022-10-26 22:59:56] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00243 [I 2022-10-26 22:59:56] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-26 22:59:56] ...optimizers/trijoint.py.151: total_norm: 13.445116 [I 2022-10-26 23:00:02] ...trap/engines/engine.py.224: train: epoch 34 | batch 1320/2304 [I 2022-10-26 23:00:02] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:07 [I 2022-10-26 23:00:02] ...trap/engines/engine.py.229: process: 0.55417 | load: 0.00252 [I 2022-10-26 23:00:02] ...trap/engines/engine.py.230: loss: 0.09602 [I 2022-10-26 23:00:02] ...optimizers/trijoint.py.151: total_norm: 15.375857 [I 2022-10-26 23:00:07] ...trap/engines/engine.py.224: train: epoch 34 | batch 1330/2304 [I 2022-10-26 23:00:07] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:08:59 [I 2022-10-26 23:00:07] ...trap/engines/engine.py.229: process: 0.56194 | load: 0.00245 [I 2022-10-26 23:00:07] ...trap/engines/engine.py.230: loss: 0.15287 [I 2022-10-26 23:00:07] ...optimizers/trijoint.py.151: total_norm: 13.709395 [I 2022-10-26 23:00:13] ...trap/engines/engine.py.224: train: epoch 34 | batch 1340/2304 [I 2022-10-26 23:00:13] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:09:09 [I 2022-10-26 23:00:13] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00246 [I 2022-10-26 23:00:13] ...trap/engines/engine.py.230: loss: 0.16040 [I 2022-10-26 23:00:13] ...optimizers/trijoint.py.151: total_norm: 11.008547 [I 2022-10-26 23:00:19] ...trap/engines/engine.py.224: train: epoch 34 | batch 1350/2304 [I 2022-10-26 23:00:19] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:53 [I 2022-10-26 23:00:19] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00242 [I 2022-10-26 23:00:19] ...trap/engines/engine.py.230: loss: 0.14334 [I 2022-10-26 23:00:19] ...optimizers/trijoint.py.151: total_norm: 16.500795 [I 2022-10-26 23:00:24] ...trap/engines/engine.py.224: train: epoch 34 | batch 1360/2304 [I 2022-10-26 23:00:24] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:43 [I 2022-10-26 23:00:24] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00241 [I 2022-10-26 23:00:24] ...trap/engines/engine.py.230: loss: 0.13898 [I 2022-10-26 23:00:24] ...optimizers/trijoint.py.151: total_norm: 15.146384 [I 2022-10-26 23:00:30] ...trap/engines/engine.py.224: train: epoch 34 | batch 1370/2304 [I 2022-10-26 23:00:30] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:39 [I 2022-10-26 23:00:30] ...trap/engines/engine.py.229: process: 0.54481 | load: 0.00245 [I 2022-10-26 23:00:30] ...trap/engines/engine.py.230: loss: 0.21930 [I 2022-10-26 23:00:30] ...optimizers/trijoint.py.151: total_norm: 19.249475 [I 2022-10-26 23:00:36] ...trap/engines/engine.py.224: train: epoch 34 | batch 1380/2304 [I 2022-10-26 23:00:36] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:33 [I 2022-10-26 23:00:36] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00244 [I 2022-10-26 23:00:36] ...trap/engines/engine.py.230: loss: 0.13051 [I 2022-10-26 23:00:36] ...optimizers/trijoint.py.151: total_norm: 14.294146 [I 2022-10-26 23:00:41] ...trap/engines/engine.py.224: train: epoch 34 | batch 1390/2304 [I 2022-10-26 23:00:41] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:28 [I 2022-10-26 23:00:41] ...trap/engines/engine.py.229: process: 0.54386 | load: 0.00250 [I 2022-10-26 23:00:41] ...trap/engines/engine.py.230: loss: 0.20998 [I 2022-10-26 23:00:41] ...optimizers/trijoint.py.151: total_norm: 14.304782 [I 2022-10-26 23:00:47] ...trap/engines/engine.py.224: train: epoch 34 | batch 1400/2304 [I 2022-10-26 23:00:47] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:22 [I 2022-10-26 23:00:47] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00245 [I 2022-10-26 23:00:47] ...trap/engines/engine.py.230: loss: 0.16462 [I 2022-10-26 23:00:47] ...optimizers/trijoint.py.151: total_norm: 10.033987 [I 2022-10-26 23:00:52] ...trap/engines/engine.py.224: train: epoch 34 | batch 1410/2304 [I 2022-10-26 23:00:52] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:14 [I 2022-10-26 23:00:52] ...trap/engines/engine.py.229: process: 0.54807 | load: 0.00242 [I 2022-10-26 23:00:52] ...trap/engines/engine.py.230: loss: 0.18468 [I 2022-10-26 23:00:52] ...optimizers/trijoint.py.151: total_norm: 14.705821 [I 2022-10-26 23:00:58] ...trap/engines/engine.py.224: train: epoch 34 | batch 1420/2304 [I 2022-10-26 23:00:58] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:09 [I 2022-10-26 23:00:58] ...trap/engines/engine.py.229: process: 0.54506 | load: 0.00251 [I 2022-10-26 23:00:58] ...trap/engines/engine.py.230: loss: 0.15588 [I 2022-10-26 23:00:58] ...optimizers/trijoint.py.151: total_norm: 19.744329 [I 2022-10-26 23:01:03] ...trap/engines/engine.py.224: train: epoch 34 | batch 1430/2304 [I 2022-10-26 23:01:03] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:04 [I 2022-10-26 23:01:03] ...trap/engines/engine.py.229: process: 0.54902 | load: 0.00248 [I 2022-10-26 23:01:03] ...trap/engines/engine.py.230: loss: 0.18907 [I 2022-10-26 23:01:03] ...optimizers/trijoint.py.151: total_norm: 22.412203 [I 2022-10-26 23:01:09] ...trap/engines/engine.py.224: train: epoch 34 | batch 1440/2304 [I 2022-10-26 23:01:09] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:07:58 [I 2022-10-26 23:01:09] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00246 [I 2022-10-26 23:01:09] ...trap/engines/engine.py.230: loss: 0.18843 [I 2022-10-26 23:01:09] ...optimizers/trijoint.py.151: total_norm: 20.195423 [I 2022-10-26 23:01:14] ...trap/engines/engine.py.224: train: epoch 34 | batch 1450/2304 [I 2022-10-26 23:01:14] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:07:51 [I 2022-10-26 23:01:14] ...trap/engines/engine.py.229: process: 0.55371 | load: 0.00245 [I 2022-10-26 23:01:14] ...trap/engines/engine.py.230: loss: 0.10705 [I 2022-10-26 23:01:14] ...optimizers/trijoint.py.151: total_norm: 8.390666 [I 2022-10-26 23:01:20] ...trap/engines/engine.py.224: train: epoch 34 | batch 1460/2304 [I 2022-10-26 23:01:20] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:07:49 [I 2022-10-26 23:01:20] ...trap/engines/engine.py.229: process: 0.56420 | load: 0.00241 [I 2022-10-26 23:01:20] ...trap/engines/engine.py.230: loss: 0.23367 [I 2022-10-26 23:01:20] ...optimizers/trijoint.py.151: total_norm: 27.870005 [I 2022-10-26 23:01:26] ...trap/engines/engine.py.224: train: epoch 34 | batch 1470/2304 [I 2022-10-26 23:01:26] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:07:43 [I 2022-10-26 23:01:26] ...trap/engines/engine.py.229: process: 0.55063 | load: 0.00245 [I 2022-10-26 23:01:26] ...trap/engines/engine.py.230: loss: 0.21016 [I 2022-10-26 23:01:26] ...optimizers/trijoint.py.151: total_norm: 18.200678 [I 2022-10-26 23:01:31] ...trap/engines/engine.py.224: train: epoch 34 | batch 1480/2304 [I 2022-10-26 23:01:31] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:33 [I 2022-10-26 23:01:31] ...trap/engines/engine.py.229: process: 0.54669 | load: 0.00245 [I 2022-10-26 23:01:31] ...trap/engines/engine.py.230: loss: 0.12623 [I 2022-10-26 23:01:31] ...optimizers/trijoint.py.151: total_norm: 10.020185 [I 2022-10-26 23:01:37] ...trap/engines/engine.py.224: train: epoch 34 | batch 1490/2304 [I 2022-10-26 23:01:37] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:33 [I 2022-10-26 23:01:37] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00244 [I 2022-10-26 23:01:37] ...trap/engines/engine.py.230: loss: 0.18039 [I 2022-10-26 23:01:37] ...optimizers/trijoint.py.151: total_norm: 15.213825 [I 2022-10-26 23:01:42] ...trap/engines/engine.py.224: train: epoch 34 | batch 1500/2304 [I 2022-10-26 23:01:42] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:26 [I 2022-10-26 23:01:42] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00251 [I 2022-10-26 23:01:42] ...trap/engines/engine.py.230: loss: 0.14804 [I 2022-10-26 23:01:42] ...optimizers/trijoint.py.151: total_norm: 29.129284 [I 2022-10-26 23:01:48] ...trap/engines/engine.py.224: train: epoch 34 | batch 1510/2304 [I 2022-10-26 23:01:48] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:26 [I 2022-10-26 23:01:48] ...trap/engines/engine.py.229: process: 0.57549 | load: 0.00247 [I 2022-10-26 23:01:48] ...trap/engines/engine.py.230: loss: 0.17745 [I 2022-10-26 23:01:48] ...optimizers/trijoint.py.151: total_norm: 14.628733 [I 2022-10-26 23:01:53] ...trap/engines/engine.py.224: train: epoch 34 | batch 1520/2304 [I 2022-10-26 23:01:53] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:14 [I 2022-10-26 23:01:54] ...trap/engines/engine.py.229: process: 0.54485 | load: 0.00246 [I 2022-10-26 23:01:54] ...trap/engines/engine.py.230: loss: 0.12047 [I 2022-10-26 23:01:54] ...optimizers/trijoint.py.151: total_norm: 19.754007 [I 2022-10-26 23:01:59] ...trap/engines/engine.py.224: train: epoch 34 | batch 1530/2304 [I 2022-10-26 23:01:59] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:10 [I 2022-10-26 23:01:59] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00242 [I 2022-10-26 23:01:59] ...trap/engines/engine.py.230: loss: 0.28210 [I 2022-10-26 23:01:59] ...optimizers/trijoint.py.151: total_norm: 30.213207 [I 2022-10-26 23:02:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 1540/2304 [I 2022-10-26 23:02:05] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:03 [I 2022-10-26 23:02:05] ...trap/engines/engine.py.229: process: 0.55462 | load: 0.00243 [I 2022-10-26 23:02:05] ...trap/engines/engine.py.230: loss: 0.24083 [I 2022-10-26 23:02:05] ...optimizers/trijoint.py.151: total_norm: 24.103447 [I 2022-10-26 23:02:10] ...trap/engines/engine.py.224: train: epoch 34 | batch 1550/2304 [I 2022-10-26 23:02:10] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:06:57 [I 2022-10-26 23:02:10] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00246 [I 2022-10-26 23:02:10] ...trap/engines/engine.py.230: loss: 0.18770 [I 2022-10-26 23:02:10] ...optimizers/trijoint.py.151: total_norm: 17.222908 [I 2022-10-26 23:02:16] ...trap/engines/engine.py.224: train: epoch 34 | batch 1560/2304 [I 2022-10-26 23:02:16] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:06:49 [I 2022-10-26 23:02:16] ...trap/engines/engine.py.229: process: 0.54518 | load: 0.00243 [I 2022-10-26 23:02:16] ...trap/engines/engine.py.230: loss: 0.13290 [I 2022-10-26 23:02:16] ...optimizers/trijoint.py.151: total_norm: 15.068955 [I 2022-10-26 23:02:21] ...trap/engines/engine.py.224: train: epoch 34 | batch 1570/2304 [I 2022-10-26 23:02:21] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:06:45 [I 2022-10-26 23:02:21] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00243 [I 2022-10-26 23:02:21] ...trap/engines/engine.py.230: loss: 0.11726 [I 2022-10-26 23:02:21] ...optimizers/trijoint.py.151: total_norm: 17.054306 [I 2022-10-26 23:02:27] ...trap/engines/engine.py.224: train: epoch 34 | batch 1580/2304 [I 2022-10-26 23:02:27] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:06:40 [I 2022-10-26 23:02:27] ...trap/engines/engine.py.229: process: 0.55453 | load: 0.00241 [I 2022-10-26 23:02:27] ...trap/engines/engine.py.230: loss: 0.15933 [I 2022-10-26 23:02:27] ...optimizers/trijoint.py.151: total_norm: 16.872181 [I 2022-10-26 23:02:32] ...trap/engines/engine.py.224: train: epoch 34 | batch 1590/2304 [I 2022-10-26 23:02:32] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:36 [I 2022-10-26 23:02:32] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00245 [I 2022-10-26 23:02:32] ...trap/engines/engine.py.230: loss: 0.16842 [I 2022-10-26 23:02:32] ...optimizers/trijoint.py.151: total_norm: 16.072721 [I 2022-10-26 23:02:38] ...trap/engines/engine.py.224: train: epoch 34 | batch 1600/2304 [I 2022-10-26 23:02:38] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:30 [I 2022-10-26 23:02:38] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00240 [I 2022-10-26 23:02:38] ...trap/engines/engine.py.230: loss: 0.13857 [I 2022-10-26 23:02:38] ...optimizers/trijoint.py.151: total_norm: 16.915426 [I 2022-10-26 23:02:43] ...trap/engines/engine.py.224: train: epoch 34 | batch 1610/2304 [I 2022-10-26 23:02:43] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:27 [I 2022-10-26 23:02:43] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00244 [I 2022-10-26 23:02:43] ...trap/engines/engine.py.230: loss: 0.16776 [I 2022-10-26 23:02:43] ...optimizers/trijoint.py.151: total_norm: 16.627483 [I 2022-10-26 23:02:49] ...trap/engines/engine.py.224: train: epoch 34 | batch 1620/2304 [I 2022-10-26 23:02:49] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:20 [I 2022-10-26 23:02:49] ...trap/engines/engine.py.229: process: 0.55925 | load: 0.00249 [I 2022-10-26 23:02:49] ...trap/engines/engine.py.230: loss: 0.13272 [I 2022-10-26 23:02:49] ...optimizers/trijoint.py.151: total_norm: 17.727348 [I 2022-10-26 23:02:55] ...trap/engines/engine.py.224: train: epoch 34 | batch 1630/2304 [I 2022-10-26 23:02:55] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:16 [I 2022-10-26 23:02:55] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00259 [I 2022-10-26 23:02:55] ...trap/engines/engine.py.230: loss: 0.18512 [I 2022-10-26 23:02:55] ...optimizers/trijoint.py.151: total_norm: 25.409378 [I 2022-10-26 23:03:00] ...trap/engines/engine.py.224: train: epoch 34 | batch 1640/2304 [I 2022-10-26 23:03:00] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:11 [I 2022-10-26 23:03:00] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00245 [I 2022-10-26 23:03:00] ...trap/engines/engine.py.230: loss: 0.17721 [I 2022-10-26 23:03:00] ...optimizers/trijoint.py.151: total_norm: 18.520533 [I 2022-10-26 23:03:06] ...trap/engines/engine.py.224: train: epoch 34 | batch 1650/2304 [I 2022-10-26 23:03:06] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:02 [I 2022-10-26 23:03:06] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00242 [I 2022-10-26 23:03:06] ...trap/engines/engine.py.230: loss: 0.22718 [I 2022-10-26 23:03:06] ...optimizers/trijoint.py.151: total_norm: 30.888784 [I 2022-10-26 23:03:11] ...trap/engines/engine.py.224: train: epoch 34 | batch 1660/2304 [I 2022-10-26 23:03:11] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:05:57 [I 2022-10-26 23:03:11] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00246 [I 2022-10-26 23:03:11] ...trap/engines/engine.py.230: loss: 0.24135 [I 2022-10-26 23:03:11] ...optimizers/trijoint.py.151: total_norm: 25.592323 [I 2022-10-26 23:03:17] ...trap/engines/engine.py.224: train: epoch 34 | batch 1670/2304 [I 2022-10-26 23:03:17] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:05:53 [I 2022-10-26 23:03:17] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00242 [I 2022-10-26 23:03:17] ...trap/engines/engine.py.230: loss: 0.13054 [I 2022-10-26 23:03:17] ...optimizers/trijoint.py.151: total_norm: 10.491739 [I 2022-10-26 23:03:23] ...trap/engines/engine.py.224: train: epoch 34 | batch 1680/2304 [I 2022-10-26 23:03:23] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:05:47 [I 2022-10-26 23:03:23] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00248 [I 2022-10-26 23:03:23] ...trap/engines/engine.py.230: loss: 0.16253 [I 2022-10-26 23:03:23] ...optimizers/trijoint.py.151: total_norm: 15.427495 [I 2022-10-26 23:03:28] ...trap/engines/engine.py.224: train: epoch 34 | batch 1690/2304 [I 2022-10-26 23:03:28] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:05:42 [I 2022-10-26 23:03:28] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00250 [I 2022-10-26 23:03:28] ...trap/engines/engine.py.230: loss: 0.17594 [I 2022-10-26 23:03:28] ...optimizers/trijoint.py.151: total_norm: 13.108566 [I 2022-10-26 23:03:34] ...trap/engines/engine.py.224: train: epoch 34 | batch 1700/2304 [I 2022-10-26 23:03:34] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:34 [I 2022-10-26 23:03:34] ...trap/engines/engine.py.229: process: 0.54703 | load: 0.00247 [I 2022-10-26 23:03:34] ...trap/engines/engine.py.230: loss: 0.22549 [I 2022-10-26 23:03:34] ...optimizers/trijoint.py.151: total_norm: 18.919697 [I 2022-10-26 23:03:39] ...trap/engines/engine.py.224: train: epoch 34 | batch 1710/2304 [I 2022-10-26 23:03:39] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:30 [I 2022-10-26 23:03:39] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00248 [I 2022-10-26 23:03:39] ...trap/engines/engine.py.230: loss: 0.19831 [I 2022-10-26 23:03:39] ...optimizers/trijoint.py.151: total_norm: 22.624973 [I 2022-10-26 23:03:45] ...trap/engines/engine.py.224: train: epoch 34 | batch 1720/2304 [I 2022-10-26 23:03:45] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:24 [I 2022-10-26 23:03:45] ...trap/engines/engine.py.229: process: 0.55320 | load: 0.00244 [I 2022-10-26 23:03:45] ...trap/engines/engine.py.230: loss: 0.17508 [I 2022-10-26 23:03:45] ...optimizers/trijoint.py.151: total_norm: 14.044406 [I 2022-10-26 23:03:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 1730/2304 [I 2022-10-26 23:03:50] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:21 [I 2022-10-26 23:03:50] ...trap/engines/engine.py.229: process: 0.56692 | load: 0.00245 [I 2022-10-26 23:03:50] ...trap/engines/engine.py.230: loss: 0.15959 [I 2022-10-26 23:03:50] ...optimizers/trijoint.py.151: total_norm: 36.376194 [I 2022-10-26 23:03:56] ...trap/engines/engine.py.224: train: epoch 34 | batch 1740/2304 [I 2022-10-26 23:03:56] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:14 [I 2022-10-26 23:03:56] ...trap/engines/engine.py.229: process: 0.56681 | load: 0.00243 [I 2022-10-26 23:03:56] ...trap/engines/engine.py.230: loss: 0.17524 [I 2022-10-26 23:03:56] ...optimizers/trijoint.py.151: total_norm: 32.706722 [I 2022-10-26 23:04:02] ...trap/engines/engine.py.224: train: epoch 34 | batch 1750/2304 [I 2022-10-26 23:04:02] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:07 [I 2022-10-26 23:04:02] ...trap/engines/engine.py.229: process: 0.55274 | load: 0.00241 [I 2022-10-26 23:04:02] ...trap/engines/engine.py.230: loss: 0.11623 [I 2022-10-26 23:04:02] ...optimizers/trijoint.py.151: total_norm: 9.743573 [I 2022-10-26 23:04:07] ...trap/engines/engine.py.224: train: epoch 34 | batch 1760/2304 [I 2022-10-26 23:04:07] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:01 [I 2022-10-26 23:04:07] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00261 [I 2022-10-26 23:04:07] ...trap/engines/engine.py.230: loss: 0.15125 [I 2022-10-26 23:04:07] ...optimizers/trijoint.py.151: total_norm: 22.974072 [I 2022-10-26 23:04:13] ...trap/engines/engine.py.224: train: epoch 34 | batch 1770/2304 [I 2022-10-26 23:04:13] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:04:56 [I 2022-10-26 23:04:13] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00253 [I 2022-10-26 23:04:13] ...trap/engines/engine.py.230: loss: 0.13959 [I 2022-10-26 23:04:13] ...optimizers/trijoint.py.151: total_norm: 21.696539 [I 2022-10-26 23:04:18] ...trap/engines/engine.py.224: train: epoch 34 | batch 1780/2304 [I 2022-10-26 23:04:18] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:04:51 [I 2022-10-26 23:04:18] ...trap/engines/engine.py.229: process: 0.55120 | load: 0.00247 [I 2022-10-26 23:04:18] ...trap/engines/engine.py.230: loss: 0.17342 [I 2022-10-26 23:04:18] ...optimizers/trijoint.py.151: total_norm: 14.464845 [I 2022-10-26 23:04:24] ...trap/engines/engine.py.224: train: epoch 34 | batch 1790/2304 [I 2022-10-26 23:04:24] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:04:45 [I 2022-10-26 23:04:24] ...trap/engines/engine.py.229: process: 0.56273 | load: 0.00246 [I 2022-10-26 23:04:24] ...trap/engines/engine.py.230: loss: 0.10557 [I 2022-10-26 23:04:24] ...optimizers/trijoint.py.151: total_norm: 14.706908 [I 2022-10-26 23:04:29] ...trap/engines/engine.py.224: train: epoch 34 | batch 1800/2304 [I 2022-10-26 23:04:29] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:40 [I 2022-10-26 23:04:29] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00244 [I 2022-10-26 23:04:29] ...trap/engines/engine.py.230: loss: 0.17403 [I 2022-10-26 23:04:29] ...optimizers/trijoint.py.151: total_norm: 23.280083 [I 2022-10-26 23:04:35] ...trap/engines/engine.py.224: train: epoch 34 | batch 1810/2304 [I 2022-10-26 23:04:35] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:34 [I 2022-10-26 23:04:35] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00244 [I 2022-10-26 23:04:35] ...trap/engines/engine.py.230: loss: 0.13360 [I 2022-10-26 23:04:35] ...optimizers/trijoint.py.151: total_norm: 10.669396 [I 2022-10-26 23:04:40] ...trap/engines/engine.py.224: train: epoch 34 | batch 1820/2304 [I 2022-10-26 23:04:40] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:28 [I 2022-10-26 23:04:40] ...trap/engines/engine.py.229: process: 0.55036 | load: 0.00248 [I 2022-10-26 23:04:40] ...trap/engines/engine.py.230: loss: 0.23484 [I 2022-10-26 23:04:40] ...optimizers/trijoint.py.151: total_norm: 20.625526 [I 2022-10-26 23:04:46] ...trap/engines/engine.py.224: train: epoch 34 | batch 1830/2304 [I 2022-10-26 23:04:46] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:04:21 [I 2022-10-26 23:04:46] ...trap/engines/engine.py.229: process: 0.55701 | load: 0.00251 [I 2022-10-26 23:04:46] ...trap/engines/engine.py.230: loss: 0.19760 [I 2022-10-26 23:04:46] ...optimizers/trijoint.py.151: total_norm: 52.141102 [I 2022-10-26 23:04:52] ...trap/engines/engine.py.224: train: epoch 34 | batch 1840/2304 [I 2022-10-26 23:04:52] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:18 [I 2022-10-26 23:04:52] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00243 [I 2022-10-26 23:04:52] ...trap/engines/engine.py.230: loss: 0.10921 [I 2022-10-26 23:04:52] ...optimizers/trijoint.py.151: total_norm: 11.853625 [I 2022-10-26 23:04:57] ...trap/engines/engine.py.224: train: epoch 34 | batch 1850/2304 [I 2022-10-26 23:04:57] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:11 [I 2022-10-26 23:04:57] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00246 [I 2022-10-26 23:04:57] ...trap/engines/engine.py.230: loss: 0.21322 [I 2022-10-26 23:04:57] ...optimizers/trijoint.py.151: total_norm: 33.192020 [I 2022-10-26 23:05:03] ...trap/engines/engine.py.224: train: epoch 34 | batch 1860/2304 [I 2022-10-26 23:05:03] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:04 [I 2022-10-26 23:05:03] ...trap/engines/engine.py.229: process: 0.54422 | load: 0.00246 [I 2022-10-26 23:05:03] ...trap/engines/engine.py.230: loss: 0.24481 [I 2022-10-26 23:05:03] ...optimizers/trijoint.py.151: total_norm: 21.015242 [I 2022-10-26 23:05:08] ...trap/engines/engine.py.224: train: epoch 34 | batch 1870/2304 [I 2022-10-26 23:05:08] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:00 [I 2022-10-26 23:05:08] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00244 [I 2022-10-26 23:05:08] ...trap/engines/engine.py.230: loss: 0.28249 [I 2022-10-26 23:05:08] ...optimizers/trijoint.py.151: total_norm: 18.253326 [I 2022-10-26 23:05:14] ...trap/engines/engine.py.224: train: epoch 34 | batch 1880/2304 [I 2022-10-26 23:05:14] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:03:53 [I 2022-10-26 23:05:14] ...trap/engines/engine.py.229: process: 0.54830 | load: 0.00244 [I 2022-10-26 23:05:14] ...trap/engines/engine.py.230: loss: 0.21129 [I 2022-10-26 23:05:14] ...optimizers/trijoint.py.151: total_norm: 29.043940 [I 2022-10-26 23:05:19] ...trap/engines/engine.py.224: train: epoch 34 | batch 1890/2304 [I 2022-10-26 23:05:19] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:03:50 [I 2022-10-26 23:05:19] ...trap/engines/engine.py.229: process: 0.54794 | load: 0.00267 [I 2022-10-26 23:05:19] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-26 23:05:19] ...optimizers/trijoint.py.151: total_norm: 11.840891 [I 2022-10-26 23:05:25] ...trap/engines/engine.py.224: train: epoch 34 | batch 1900/2304 [I 2022-10-26 23:05:25] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:03:49 [I 2022-10-26 23:05:25] ...trap/engines/engine.py.229: process: 0.55082 | load: 0.00244 [I 2022-10-26 23:05:25] ...trap/engines/engine.py.230: loss: 0.17089 [I 2022-10-26 23:05:25] ...optimizers/trijoint.py.151: total_norm: 12.950574 [I 2022-10-26 23:05:31] ...trap/engines/engine.py.224: train: epoch 34 | batch 1910/2304 [I 2022-10-26 23:05:31] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:03:39 [I 2022-10-26 23:05:31] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00243 [I 2022-10-26 23:05:31] ...trap/engines/engine.py.230: loss: 0.16476 [I 2022-10-26 23:05:31] ...optimizers/trijoint.py.151: total_norm: 13.869241 [I 2022-10-26 23:05:36] ...trap/engines/engine.py.224: train: epoch 34 | batch 1920/2304 [I 2022-10-26 23:05:36] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:34 [I 2022-10-26 23:05:36] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00246 [I 2022-10-26 23:05:36] ...trap/engines/engine.py.230: loss: 0.20268 [I 2022-10-26 23:05:36] ...optimizers/trijoint.py.151: total_norm: 34.829807 [I 2022-10-26 23:05:42] ...trap/engines/engine.py.224: train: epoch 34 | batch 1930/2304 [I 2022-10-26 23:05:42] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:27 [I 2022-10-26 23:05:42] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00254 [I 2022-10-26 23:05:42] ...trap/engines/engine.py.230: loss: 0.16367 [I 2022-10-26 23:05:42] ...optimizers/trijoint.py.151: total_norm: 14.912121 [I 2022-10-26 23:05:47] ...trap/engines/engine.py.224: train: epoch 34 | batch 1940/2304 [I 2022-10-26 23:05:47] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:20 [I 2022-10-26 23:05:47] ...trap/engines/engine.py.229: process: 0.54835 | load: 0.00247 [I 2022-10-26 23:05:47] ...trap/engines/engine.py.230: loss: 0.22965 [I 2022-10-26 23:05:47] ...optimizers/trijoint.py.151: total_norm: 26.039610 [I 2022-10-26 23:05:53] ...trap/engines/engine.py.224: train: epoch 34 | batch 1950/2304 [I 2022-10-26 23:05:53] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:03:17 [I 2022-10-26 23:05:53] ...trap/engines/engine.py.229: process: 0.56582 | load: 0.00244 [I 2022-10-26 23:05:53] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-26 23:05:53] ...optimizers/trijoint.py.151: total_norm: 24.626282 [I 2022-10-26 23:05:58] ...trap/engines/engine.py.224: train: epoch 34 | batch 1960/2304 [I 2022-10-26 23:05:58] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:10 [I 2022-10-26 23:05:58] ...trap/engines/engine.py.229: process: 0.54988 | load: 0.00242 [I 2022-10-26 23:05:58] ...trap/engines/engine.py.230: loss: 0.18633 [I 2022-10-26 23:05:58] ...optimizers/trijoint.py.151: total_norm: 10.490503 [I 2022-10-26 23:06:04] ...trap/engines/engine.py.224: train: epoch 34 | batch 1970/2304 [I 2022-10-26 23:06:04] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:05 [I 2022-10-26 23:06:04] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00243 [I 2022-10-26 23:06:04] ...trap/engines/engine.py.230: loss: 0.15427 [I 2022-10-26 23:06:04] ...optimizers/trijoint.py.151: total_norm: 15.399470 [I 2022-10-26 23:06:10] ...trap/engines/engine.py.224: train: epoch 34 | batch 1980/2304 [I 2022-10-26 23:06:10] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:00 [I 2022-10-26 23:06:10] ...trap/engines/engine.py.229: process: 0.56695 | load: 0.00241 [I 2022-10-26 23:06:10] ...trap/engines/engine.py.230: loss: 0.12659 [I 2022-10-26 23:06:10] ...optimizers/trijoint.py.151: total_norm: 14.604552 [I 2022-10-26 23:06:15] ...trap/engines/engine.py.224: train: epoch 34 | batch 1990/2304 [I 2022-10-26 23:06:15] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:02:58 [I 2022-10-26 23:06:15] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00246 [I 2022-10-26 23:06:15] ...trap/engines/engine.py.230: loss: 0.20939 [I 2022-10-26 23:06:15] ...optimizers/trijoint.py.151: total_norm: 17.783213 [I 2022-10-26 23:06:21] ...trap/engines/engine.py.224: train: epoch 34 | batch 2000/2304 [I 2022-10-26 23:06:21] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:02:49 [I 2022-10-26 23:06:21] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00247 [I 2022-10-26 23:06:21] ...trap/engines/engine.py.230: loss: 0.23060 [I 2022-10-26 23:06:21] ...optimizers/trijoint.py.151: total_norm: 16.137829 [I 2022-10-26 23:06:26] ...trap/engines/engine.py.224: train: epoch 34 | batch 2010/2304 [I 2022-10-26 23:06:26] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:02:42 [I 2022-10-26 23:06:26] ...trap/engines/engine.py.229: process: 0.54434 | load: 0.00250 [I 2022-10-26 23:06:26] ...trap/engines/engine.py.230: loss: 0.13591 [I 2022-10-26 23:06:26] ...optimizers/trijoint.py.151: total_norm: 14.783334 [I 2022-10-26 23:06:32] ...trap/engines/engine.py.224: train: epoch 34 | batch 2020/2304 [I 2022-10-26 23:06:32] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:02:37 [I 2022-10-26 23:06:32] ...trap/engines/engine.py.229: process: 0.56049 | load: 0.00249 [I 2022-10-26 23:06:32] ...trap/engines/engine.py.230: loss: 0.11518 [I 2022-10-26 23:06:32] ...optimizers/trijoint.py.151: total_norm: 13.974430 [I 2022-10-26 23:06:37] ...trap/engines/engine.py.224: train: epoch 34 | batch 2030/2304 [I 2022-10-26 23:06:37] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:31 [I 2022-10-26 23:06:37] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00244 [I 2022-10-26 23:06:37] ...trap/engines/engine.py.230: loss: 0.20349 [I 2022-10-26 23:06:37] ...optimizers/trijoint.py.151: total_norm: 15.297552 [I 2022-10-26 23:06:43] ...trap/engines/engine.py.224: train: epoch 34 | batch 2040/2304 [I 2022-10-26 23:06:43] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:26 [I 2022-10-26 23:06:43] ...trap/engines/engine.py.229: process: 0.55641 | load: 0.00243 [I 2022-10-26 23:06:43] ...trap/engines/engine.py.230: loss: 0.13139 [I 2022-10-26 23:06:43] ...optimizers/trijoint.py.151: total_norm: 25.246567 [I 2022-10-26 23:06:49] ...trap/engines/engine.py.224: train: epoch 34 | batch 2050/2304 [I 2022-10-26 23:06:49] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:02:20 [I 2022-10-26 23:06:49] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00240 [I 2022-10-26 23:06:49] ...trap/engines/engine.py.230: loss: 0.13015 [I 2022-10-26 23:06:49] ...optimizers/trijoint.py.151: total_norm: 14.451646 [I 2022-10-26 23:06:54] ...trap/engines/engine.py.224: train: epoch 34 | batch 2060/2304 [I 2022-10-26 23:06:54] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:19 [I 2022-10-26 23:06:54] ...trap/engines/engine.py.229: process: 0.55986 | load: 0.00242 [I 2022-10-26 23:06:54] ...trap/engines/engine.py.230: loss: 0.22897 [I 2022-10-26 23:06:54] ...optimizers/trijoint.py.151: total_norm: 23.114796 [I 2022-10-26 23:07:00] ...trap/engines/engine.py.224: train: epoch 34 | batch 2070/2304 [I 2022-10-26 23:07:00] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:09 [I 2022-10-26 23:07:00] ...trap/engines/engine.py.229: process: 0.54900 | load: 0.00244 [I 2022-10-26 23:07:00] ...trap/engines/engine.py.230: loss: 0.18555 [I 2022-10-26 23:07:00] ...optimizers/trijoint.py.151: total_norm: 18.588871 [I 2022-10-26 23:07:05] ...trap/engines/engine.py.224: train: epoch 34 | batch 2080/2304 [I 2022-10-26 23:07:05] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:04 [I 2022-10-26 23:07:05] ...trap/engines/engine.py.229: process: 0.55444 | load: 0.00242 [I 2022-10-26 23:07:05] ...trap/engines/engine.py.230: loss: 0.19382 [I 2022-10-26 23:07:05] ...optimizers/trijoint.py.151: total_norm: 14.726028 [I 2022-10-26 23:07:11] ...trap/engines/engine.py.224: train: epoch 34 | batch 2090/2304 [I 2022-10-26 23:07:11] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:01:58 [I 2022-10-26 23:07:11] ...trap/engines/engine.py.229: process: 0.54411 | load: 0.00243 [I 2022-10-26 23:07:11] ...trap/engines/engine.py.230: loss: 0.15763 [I 2022-10-26 23:07:11] ...optimizers/trijoint.py.151: total_norm: 12.418963 [I 2022-10-26 23:07:17] ...trap/engines/engine.py.224: train: epoch 34 | batch 2100/2304 [I 2022-10-26 23:07:17] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:01:53 [I 2022-10-26 23:07:17] ...trap/engines/engine.py.229: process: 0.54151 | load: 0.00288 [I 2022-10-26 23:07:17] ...trap/engines/engine.py.230: loss: 0.13851 [I 2022-10-26 23:07:17] ...optimizers/trijoint.py.151: total_norm: 17.398525 [I 2022-10-26 23:07:22] ...trap/engines/engine.py.224: train: epoch 34 | batch 2110/2304 [I 2022-10-26 23:07:22] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:01:47 [I 2022-10-26 23:07:22] ...trap/engines/engine.py.229: process: 0.55462 | load: 0.00260 [I 2022-10-26 23:07:22] ...trap/engines/engine.py.230: loss: 0.17521 [I 2022-10-26 23:07:22] ...optimizers/trijoint.py.151: total_norm: 14.769876 [I 2022-10-26 23:07:28] ...trap/engines/engine.py.224: train: epoch 34 | batch 2120/2304 [I 2022-10-26 23:07:28] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:01:41 [I 2022-10-26 23:07:28] ...trap/engines/engine.py.229: process: 0.54615 | load: 0.00246 [I 2022-10-26 23:07:28] ...trap/engines/engine.py.230: loss: 0.18175 [I 2022-10-26 23:07:28] ...optimizers/trijoint.py.151: total_norm: 25.489761 [I 2022-10-26 23:07:33] ...trap/engines/engine.py.224: train: epoch 34 | batch 2130/2304 [I 2022-10-26 23:07:33] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:36 [I 2022-10-26 23:07:33] ...trap/engines/engine.py.229: process: 0.55482 | load: 0.00246 [I 2022-10-26 23:07:33] ...trap/engines/engine.py.230: loss: 0.17542 [I 2022-10-26 23:07:33] ...optimizers/trijoint.py.151: total_norm: 12.408135 [I 2022-10-26 23:07:39] ...trap/engines/engine.py.224: train: epoch 34 | batch 2140/2304 [I 2022-10-26 23:07:39] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:31 [I 2022-10-26 23:07:39] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00246 [I 2022-10-26 23:07:39] ...trap/engines/engine.py.230: loss: 0.13601 [I 2022-10-26 23:07:39] ...optimizers/trijoint.py.151: total_norm: 20.655609 [I 2022-10-26 23:07:44] ...trap/engines/engine.py.224: train: epoch 34 | batch 2150/2304 [I 2022-10-26 23:07:44] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:26 [I 2022-10-26 23:07:44] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00245 [I 2022-10-26 23:07:44] ...trap/engines/engine.py.230: loss: 0.12235 [I 2022-10-26 23:07:44] ...optimizers/trijoint.py.151: total_norm: 12.914248 [I 2022-10-26 23:07:50] ...trap/engines/engine.py.224: train: epoch 34 | batch 2160/2304 [I 2022-10-26 23:07:50] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:19 [I 2022-10-26 23:07:50] ...trap/engines/engine.py.229: process: 0.55155 | load: 0.00241 [I 2022-10-26 23:07:50] ...trap/engines/engine.py.230: loss: 0.24441 [I 2022-10-26 23:07:50] ...optimizers/trijoint.py.151: total_norm: 20.144947 [I 2022-10-26 23:07:56] ...trap/engines/engine.py.224: train: epoch 34 | batch 2170/2304 [I 2022-10-26 23:07:56] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:15 [I 2022-10-26 23:07:56] ...trap/engines/engine.py.229: process: 0.55892 | load: 0.00242 [I 2022-10-26 23:07:56] ...trap/engines/engine.py.230: loss: 0.18982 [I 2022-10-26 23:07:56] ...optimizers/trijoint.py.151: total_norm: 11.631554 [I 2022-10-26 23:08:01] ...trap/engines/engine.py.224: train: epoch 34 | batch 2180/2304 [I 2022-10-26 23:08:01] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:08 [I 2022-10-26 23:08:01] ...trap/engines/engine.py.229: process: 0.55131 | load: 0.00246 [I 2022-10-26 23:08:01] ...trap/engines/engine.py.230: loss: 0.15768 [I 2022-10-26 23:08:01] ...optimizers/trijoint.py.151: total_norm: 16.958052 [I 2022-10-26 23:08:07] ...trap/engines/engine.py.224: train: epoch 34 | batch 2190/2304 [I 2022-10-26 23:08:07] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:03 [I 2022-10-26 23:08:07] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00261 [I 2022-10-26 23:08:07] ...trap/engines/engine.py.230: loss: 0.12484 [I 2022-10-26 23:08:07] ...optimizers/trijoint.py.151: total_norm: 15.103168 [I 2022-10-26 23:08:12] ...trap/engines/engine.py.224: train: epoch 34 | batch 2200/2304 [I 2022-10-26 23:08:12] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:00:57 [I 2022-10-26 23:08:12] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00278 [I 2022-10-26 23:08:12] ...trap/engines/engine.py.230: loss: 0.22236 [I 2022-10-26 23:08:12] ...optimizers/trijoint.py.151: total_norm: 21.380777 [I 2022-10-26 23:08:18] ...trap/engines/engine.py.224: train: epoch 34 | batch 2210/2304 [I 2022-10-26 23:08:18] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:00:52 [I 2022-10-26 23:08:18] ...trap/engines/engine.py.229: process: 0.56833 | load: 0.00243 [I 2022-10-26 23:08:18] ...trap/engines/engine.py.230: loss: 0.17094 [I 2022-10-26 23:08:18] ...optimizers/trijoint.py.151: total_norm: 14.782646 [I 2022-10-26 23:08:24] ...trap/engines/engine.py.224: train: epoch 34 | batch 2220/2304 [I 2022-10-26 23:08:24] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:00:46 [I 2022-10-26 23:08:24] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00246 [I 2022-10-26 23:08:24] ...trap/engines/engine.py.230: loss: 0.22417 [I 2022-10-26 23:08:24] ...optimizers/trijoint.py.151: total_norm: 19.049524 [I 2022-10-26 23:08:29] ...trap/engines/engine.py.224: train: epoch 34 | batch 2230/2304 [I 2022-10-26 23:08:29] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:00:41 [I 2022-10-26 23:08:29] ...trap/engines/engine.py.229: process: 0.57055 | load: 0.00243 [I 2022-10-26 23:08:29] ...trap/engines/engine.py.230: loss: 0.16863 [I 2022-10-26 23:08:29] ...optimizers/trijoint.py.151: total_norm: 9.752599 [I 2022-10-26 23:08:35] ...trap/engines/engine.py.224: train: epoch 34 | batch 2240/2304 [I 2022-10-26 23:08:35] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:35 [I 2022-10-26 23:08:35] ...trap/engines/engine.py.229: process: 0.56261 | load: 0.00241 [I 2022-10-26 23:08:35] ...trap/engines/engine.py.230: loss: 0.17662 [I 2022-10-26 23:08:35] ...optimizers/trijoint.py.151: total_norm: 18.841780 [I 2022-10-26 23:08:40] ...trap/engines/engine.py.224: train: epoch 34 | batch 2250/2304 [I 2022-10-26 23:08:40] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:29 [I 2022-10-26 23:08:40] ...trap/engines/engine.py.229: process: 0.55069 | load: 0.00242 [I 2022-10-26 23:08:40] ...trap/engines/engine.py.230: loss: 0.19798 [I 2022-10-26 23:08:40] ...optimizers/trijoint.py.151: total_norm: 19.935442 [I 2022-10-26 23:08:46] ...trap/engines/engine.py.224: train: epoch 34 | batch 2260/2304 [I 2022-10-26 23:08:46] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:24 [I 2022-10-26 23:08:46] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00244 [I 2022-10-26 23:08:46] ...trap/engines/engine.py.230: loss: 0.11573 [I 2022-10-26 23:08:46] ...optimizers/trijoint.py.151: total_norm: 14.087989 [I 2022-10-26 23:08:51] ...trap/engines/engine.py.224: train: epoch 34 | batch 2270/2304 [I 2022-10-26 23:08:51] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:18 [I 2022-10-26 23:08:51] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00242 [I 2022-10-26 23:08:51] ...trap/engines/engine.py.230: loss: 0.14434 [I 2022-10-26 23:08:51] ...optimizers/trijoint.py.151: total_norm: 14.511127 [I 2022-10-26 23:08:57] ...trap/engines/engine.py.224: train: epoch 34 | batch 2280/2304 [I 2022-10-26 23:08:57] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:13 [I 2022-10-26 23:08:57] ...trap/engines/engine.py.229: process: 0.57960 | load: 0.00243 [I 2022-10-26 23:08:57] ...trap/engines/engine.py.230: loss: 0.17858 [I 2022-10-26 23:08:57] ...optimizers/trijoint.py.151: total_norm: 12.399057 [I 2022-10-26 23:09:03] ...trap/engines/engine.py.224: train: epoch 34 | batch 2290/2304 [I 2022-10-26 23:09:03] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:07 [I 2022-10-26 23:09:03] ...trap/engines/engine.py.229: process: 0.56294 | load: 0.00250 [I 2022-10-26 23:09:03] ...trap/engines/engine.py.230: loss: 0.18211 [I 2022-10-26 23:09:03] ...optimizers/trijoint.py.151: total_norm: 20.786114 [I 2022-10-26 23:09:08] ...trap/engines/engine.py.224: train: epoch 34 | batch 2300/2304 [I 2022-10-26 23:09:08] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:02 [I 2022-10-26 23:09:08] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00014 [I 2022-10-26 23:09:08] ...trap/engines/engine.py.230: loss: 0.12587 [I 2022-10-26 23:09:08] ...optimizers/trijoint.py.151: total_norm: 16.044708 [I 2022-10-26 23:09:11] ...trap/engines/engine.py.224: train: epoch 34 | batch 2304/2304 [I 2022-10-26 23:09:11] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:00 [I 2022-10-26 23:09:11] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00020 [I 2022-10-26 23:09:11] ...trap/engines/engine.py.230: loss: 0.17756 [I 2022-10-26 23:09:11] ...optimizers/trijoint.py.151: total_norm: 16.225014 [S 2022-10-26 23:09:12] ...trap/engines/engine.py.129: train_epoch.epoch: 34 [S 2022-10-26 23:09:12] ...trap/engines/engine.py.129: train_epoch.loss: 0.17614076714195037 [I 2022-10-26 23:09:12] ...s/criterions/triplet.py.84: margin = 0.22500000000000014 [I 2022-10-26 23:09:12] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 34 [I 2022-10-26 23:09:12] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 23:09:14] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 23:09:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 23:09:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 23:09:15] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 23:09:15] ...trap/engines/engine.py.321: eval: epoch 34 | batch 0/510 [I 2022-10-26 23:09:15] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:27:31 [I 2022-10-26 23:09:15] ...trap/engines/engine.py.326: process: 3.23756 | load: 2.85921 [I 2022-10-26 23:09:19] ...trap/engines/engine.py.321: eval: epoch 34 | batch 10/510 [I 2022-10-26 23:09:19] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:21 [I 2022-10-26 23:09:19] ...trap/engines/engine.py.326: process: 0.33251 | load: 0.00017 [I 2022-10-26 23:09:22] ...trap/engines/engine.py.321: eval: epoch 34 | batch 20/510 [I 2022-10-26 23:09:22] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:02:59 [I 2022-10-26 23:09:22] ...trap/engines/engine.py.326: process: 0.33174 | load: 0.00018 [I 2022-10-26 23:09:25] ...trap/engines/engine.py.321: eval: epoch 34 | batch 30/510 [I 2022-10-26 23:09:25] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:45 [I 2022-10-26 23:09:25] ...trap/engines/engine.py.326: process: 0.35493 | load: 0.00030 [I 2022-10-26 23:09:29] ...trap/engines/engine.py.321: eval: epoch 34 | batch 40/510 [I 2022-10-26 23:09:29] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:39 [I 2022-10-26 23:09:29] ...trap/engines/engine.py.326: process: 0.33222 | load: 0.00027 [I 2022-10-26 23:09:32] ...trap/engines/engine.py.321: eval: epoch 34 | batch 50/510 [I 2022-10-26 23:09:32] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:36 [I 2022-10-26 23:09:32] ...trap/engines/engine.py.326: process: 0.33727 | load: 0.00024 [I 2022-10-26 23:09:36] ...trap/engines/engine.py.321: eval: epoch 34 | batch 60/510 [I 2022-10-26 23:09:36] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:31 [I 2022-10-26 23:09:36] ...trap/engines/engine.py.326: process: 0.32085 | load: 0.00022 [I 2022-10-26 23:09:39] ...trap/engines/engine.py.321: eval: epoch 34 | batch 70/510 [I 2022-10-26 23:09:39] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-26 23:09:39] ...trap/engines/engine.py.326: process: 0.34253 | load: 0.00017 [I 2022-10-26 23:09:42] ...trap/engines/engine.py.321: eval: epoch 34 | batch 80/510 [I 2022-10-26 23:09:42] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-26 23:09:42] ...trap/engines/engine.py.326: process: 0.33928 | load: 0.00022 [I 2022-10-26 23:09:46] ...trap/engines/engine.py.321: eval: epoch 34 | batch 90/510 [I 2022-10-26 23:09:46] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-26 23:09:46] ...trap/engines/engine.py.326: process: 0.32761 | load: 0.00026 [I 2022-10-26 23:09:49] ...trap/engines/engine.py.321: eval: epoch 34 | batch 100/510 [I 2022-10-26 23:09:49] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 23:09:49] ...trap/engines/engine.py.326: process: 0.28406 | load: 0.00022 [I 2022-10-26 23:09:52] ...trap/engines/engine.py.321: eval: epoch 34 | batch 110/510 [I 2022-10-26 23:09:52] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:59 [I 2022-10-26 23:09:52] ...trap/engines/engine.py.326: process: 0.29348 | load: 0.00014 [I 2022-10-26 23:09:55] ...trap/engines/engine.py.321: eval: epoch 34 | batch 120/510 [I 2022-10-26 23:09:55] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:55 [I 2022-10-26 23:09:55] ...trap/engines/engine.py.326: process: 0.30489 | load: 0.00028 [I 2022-10-26 23:09:58] ...trap/engines/engine.py.321: eval: epoch 34 | batch 130/510 [I 2022-10-26 23:09:58] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:49 [I 2022-10-26 23:09:58] ...trap/engines/engine.py.326: process: 0.28588 | load: 0.00016 [I 2022-10-26 23:10:01] ...trap/engines/engine.py.321: eval: epoch 34 | batch 140/510 [I 2022-10-26 23:10:01] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-26 23:10:01] ...trap/engines/engine.py.326: process: 0.28625 | load: 0.00023 [I 2022-10-26 23:10:04] ...trap/engines/engine.py.321: eval: epoch 34 | batch 150/510 [I 2022-10-26 23:10:04] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:46 [I 2022-10-26 23:10:04] ...trap/engines/engine.py.326: process: 0.28874 | load: 0.00019 [I 2022-10-26 23:10:06] ...trap/engines/engine.py.321: eval: epoch 34 | batch 160/510 [I 2022-10-26 23:10:06] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-26 23:10:06] ...trap/engines/engine.py.326: process: 0.28100 | load: 0.00017 [I 2022-10-26 23:10:09] ...trap/engines/engine.py.321: eval: epoch 34 | batch 170/510 [I 2022-10-26 23:10:09] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-26 23:10:09] ...trap/engines/engine.py.326: process: 0.27996 | load: 0.00017 [I 2022-10-26 23:10:12] ...trap/engines/engine.py.321: eval: epoch 34 | batch 180/510 [I 2022-10-26 23:10:12] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:36 [I 2022-10-26 23:10:12] ...trap/engines/engine.py.326: process: 0.28890 | load: 0.00021 [I 2022-10-26 23:10:15] ...trap/engines/engine.py.321: eval: epoch 34 | batch 190/510 [I 2022-10-26 23:10:15] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-26 23:10:15] ...trap/engines/engine.py.326: process: 0.28819 | load: 0.00022 [I 2022-10-26 23:10:18] ...trap/engines/engine.py.321: eval: epoch 34 | batch 200/510 [I 2022-10-26 23:10:18] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 23:10:18] ...trap/engines/engine.py.326: process: 0.29227 | load: 0.00022 [I 2022-10-26 23:10:21] ...trap/engines/engine.py.321: eval: epoch 34 | batch 210/510 [I 2022-10-26 23:10:21] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-26 23:10:21] ...trap/engines/engine.py.326: process: 0.28783 | load: 0.00021 [I 2022-10-26 23:10:24] ...trap/engines/engine.py.321: eval: epoch 34 | batch 220/510 [I 2022-10-26 23:10:24] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 23:10:24] ...trap/engines/engine.py.326: process: 0.28518 | load: 0.00014 [I 2022-10-26 23:10:27] ...trap/engines/engine.py.321: eval: epoch 34 | batch 230/510 [I 2022-10-26 23:10:27] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-26 23:10:27] ...trap/engines/engine.py.326: process: 0.29282 | load: 0.00015 [I 2022-10-26 23:10:30] ...trap/engines/engine.py.321: eval: epoch 34 | batch 240/510 [I 2022-10-26 23:10:30] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 23:10:30] ...trap/engines/engine.py.326: process: 0.28264 | load: 0.00016 [I 2022-10-26 23:10:32] ...trap/engines/engine.py.321: eval: epoch 34 | batch 250/510 [I 2022-10-26 23:10:32] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:13 [I 2022-10-26 23:10:32] ...trap/engines/engine.py.326: process: 0.28570 | load: 0.00015 [I 2022-10-26 23:10:35] ...trap/engines/engine.py.321: eval: epoch 34 | batch 260/510 [I 2022-10-26 23:10:35] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-26 23:10:35] ...trap/engines/engine.py.326: process: 0.28900 | load: 0.00015 [I 2022-10-26 23:10:38] ...trap/engines/engine.py.321: eval: epoch 34 | batch 270/510 [I 2022-10-26 23:10:38] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-26 23:10:38] ...trap/engines/engine.py.326: process: 0.29065 | load: 0.00014 [I 2022-10-26 23:10:41] ...trap/engines/engine.py.321: eval: epoch 34 | batch 280/510 [I 2022-10-26 23:10:41] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:08 [I 2022-10-26 23:10:41] ...trap/engines/engine.py.326: process: 0.28803 | load: 0.00017 [I 2022-10-26 23:10:44] ...trap/engines/engine.py.321: eval: epoch 34 | batch 290/510 [I 2022-10-26 23:10:44] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 23:10:44] ...trap/engines/engine.py.326: process: 0.28946 | load: 0.00017 [I 2022-10-26 23:10:47] ...trap/engines/engine.py.321: eval: epoch 34 | batch 300/510 [I 2022-10-26 23:10:47] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 23:10:47] ...trap/engines/engine.py.326: process: 0.28497 | load: 0.00016 [I 2022-10-26 23:10:50] ...trap/engines/engine.py.321: eval: epoch 34 | batch 310/510 [I 2022-10-26 23:10:50] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 23:10:50] ...trap/engines/engine.py.326: process: 0.28769 | load: 0.00020 [I 2022-10-26 23:10:53] ...trap/engines/engine.py.321: eval: epoch 34 | batch 320/510 [I 2022-10-26 23:10:53] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 23:10:53] ...trap/engines/engine.py.326: process: 0.28410 | load: 0.00015 [I 2022-10-26 23:10:55] ...trap/engines/engine.py.321: eval: epoch 34 | batch 330/510 [I 2022-10-26 23:10:55] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-26 23:10:55] ...trap/engines/engine.py.326: process: 0.29198 | load: 0.00014 [I 2022-10-26 23:10:58] ...trap/engines/engine.py.321: eval: epoch 34 | batch 340/510 [I 2022-10-26 23:10:58] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-26 23:10:58] ...trap/engines/engine.py.326: process: 0.28068 | load: 0.00014 [I 2022-10-26 23:11:01] ...trap/engines/engine.py.321: eval: epoch 34 | batch 350/510 [I 2022-10-26 23:11:01] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 23:11:01] ...trap/engines/engine.py.326: process: 0.28366 | load: 0.00015 [I 2022-10-26 23:11:04] ...trap/engines/engine.py.321: eval: epoch 34 | batch 360/510 [I 2022-10-26 23:11:04] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 23:11:04] ...trap/engines/engine.py.326: process: 0.28076 | load: 0.00016 [I 2022-10-26 23:11:07] ...trap/engines/engine.py.321: eval: epoch 34 | batch 370/510 [I 2022-10-26 23:11:07] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-26 23:11:07] ...trap/engines/engine.py.326: process: 0.29607 | load: 0.00015 [I 2022-10-26 23:11:10] ...trap/engines/engine.py.321: eval: epoch 34 | batch 380/510 [I 2022-10-26 23:11:10] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 23:11:10] ...trap/engines/engine.py.326: process: 0.28445 | load: 0.00014 [I 2022-10-26 23:11:13] ...trap/engines/engine.py.321: eval: epoch 34 | batch 390/510 [I 2022-10-26 23:11:13] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 23:11:13] ...trap/engines/engine.py.326: process: 0.28864 | load: 0.00014 [I 2022-10-26 23:11:16] ...trap/engines/engine.py.321: eval: epoch 34 | batch 400/510 [I 2022-10-26 23:11:16] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-26 23:11:16] ...trap/engines/engine.py.326: process: 0.28440 | load: 0.00015 [I 2022-10-26 23:11:19] ...trap/engines/engine.py.321: eval: epoch 34 | batch 410/510 [I 2022-10-26 23:11:19] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 23:11:19] ...trap/engines/engine.py.326: process: 0.28808 | load: 0.00015 [I 2022-10-26 23:11:21] ...trap/engines/engine.py.321: eval: epoch 34 | batch 420/510 [I 2022-10-26 23:11:21] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 23:11:21] ...trap/engines/engine.py.326: process: 0.28312 | load: 0.00015 [I 2022-10-26 23:11:24] ...trap/engines/engine.py.321: eval: epoch 34 | batch 430/510 [I 2022-10-26 23:11:24] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 23:11:24] ...trap/engines/engine.py.326: process: 0.29158 | load: 0.00018 [I 2022-10-26 23:11:27] ...trap/engines/engine.py.321: eval: epoch 34 | batch 440/510 [I 2022-10-26 23:11:27] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 23:11:27] ...trap/engines/engine.py.326: process: 0.28598 | load: 0.00015 [I 2022-10-26 23:11:30] ...trap/engines/engine.py.321: eval: epoch 34 | batch 450/510 [I 2022-10-26 23:11:30] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-26 23:11:30] ...trap/engines/engine.py.326: process: 0.28788 | load: 0.00016 [I 2022-10-26 23:11:33] ...trap/engines/engine.py.321: eval: epoch 34 | batch 460/510 [I 2022-10-26 23:11:33] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-26 23:11:33] ...trap/engines/engine.py.326: process: 0.29081 | load: 0.00015 [I 2022-10-26 23:11:36] ...trap/engines/engine.py.321: eval: epoch 34 | batch 470/510 [I 2022-10-26 23:11:36] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 23:11:36] ...trap/engines/engine.py.326: process: 0.28964 | load: 0.00015 [I 2022-10-26 23:11:39] ...trap/engines/engine.py.321: eval: epoch 34 | batch 480/510 [I 2022-10-26 23:11:39] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 23:11:39] ...trap/engines/engine.py.326: process: 0.28845 | load: 0.00015 [I 2022-10-26 23:11:42] ...trap/engines/engine.py.321: eval: epoch 34 | batch 490/510 [I 2022-10-26 23:11:42] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 23:11:42] ...trap/engines/engine.py.326: process: 0.28838 | load: 0.00015 [I 2022-10-26 23:11:44] ...trap/engines/engine.py.321: eval: epoch 34 | batch 500/510 [I 2022-10-26 23:11:44] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 23:11:44] ...trap/engines/engine.py.326: process: 0.28818 | load: 0.00018 [I 2022-10-26 23:11:47] ...trap/engines/engine.py.321: eval: epoch 34 | batch 510/510 [I 2022-10-26 23:11:47] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 23:11:47] ...trap/engines/engine.py.326: process: 0.28509 | load: 0.00011 [S 2022-10-26 23:11:48] ...trap/engines/engine.py.132: eval_epoch.epoch: 34 [I 2022-10-26 23:11:48] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.672400 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.882100 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.923000 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.658200 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.881400 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.919500 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011011 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009257 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010752 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016606 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009718 [S 2022-10-26 23:11:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007966 [I 2022-10-26 23:11:53] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 23:11:53] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 23:11:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 23:11:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 23:11:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 23:11:55] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 23:11:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 23:12:03] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 23:12:03] ...trap/engines/engine.py.163: Training model on trainset for epoch 35 [I 2022-10-26 23:12:03] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 23:12:06] ...trap/engines/engine.py.224: train: epoch 35 | batch 0/2304 [I 2022-10-26 23:12:06] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:45:26 [I 2022-10-26 23:12:06] ...trap/engines/engine.py.229: process: 2.74604 | load: 2.06199 [I 2022-10-26 23:12:06] ...trap/engines/engine.py.230: loss: 0.17576 [I 2022-10-26 23:12:06] ...optimizers/trijoint.py.151: total_norm: 8.181561 [I 2022-10-26 23:12:12] ...trap/engines/engine.py.224: train: epoch 35 | batch 10/2304 [I 2022-10-26 23:12:12] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:19 [I 2022-10-26 23:12:12] ...trap/engines/engine.py.229: process: 0.54859 | load: 0.00327 [I 2022-10-26 23:12:12] ...trap/engines/engine.py.230: loss: 0.13798 [I 2022-10-26 23:12:12] ...optimizers/trijoint.py.151: total_norm: 14.965702 [I 2022-10-26 23:12:17] ...trap/engines/engine.py.224: train: epoch 35 | batch 20/2304 [I 2022-10-26 23:12:17] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:07 [I 2022-10-26 23:12:17] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00346 [I 2022-10-26 23:12:17] ...trap/engines/engine.py.230: loss: 0.18956 [I 2022-10-26 23:12:17] ...optimizers/trijoint.py.151: total_norm: 15.715353 [I 2022-10-26 23:12:23] ...trap/engines/engine.py.224: train: epoch 35 | batch 30/2304 [I 2022-10-26 23:12:23] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:11 [I 2022-10-26 23:12:23] ...trap/engines/engine.py.229: process: 0.54981 | load: 0.00290 [I 2022-10-26 23:12:23] ...trap/engines/engine.py.230: loss: 0.12992 [I 2022-10-26 23:12:23] ...optimizers/trijoint.py.151: total_norm: 20.125713 [I 2022-10-26 23:12:28] ...trap/engines/engine.py.224: train: epoch 35 | batch 40/2304 [I 2022-10-26 23:12:28] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:03 [I 2022-10-26 23:12:28] ...trap/engines/engine.py.229: process: 0.56319 | load: 0.00287 [I 2022-10-26 23:12:28] ...trap/engines/engine.py.230: loss: 0.14279 [I 2022-10-26 23:12:28] ...optimizers/trijoint.py.151: total_norm: 34.858639 [I 2022-10-26 23:12:34] ...trap/engines/engine.py.224: train: epoch 35 | batch 50/2304 [I 2022-10-26 23:12:34] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:54 [I 2022-10-26 23:12:34] ...trap/engines/engine.py.229: process: 0.56401 | load: 0.00277 [I 2022-10-26 23:12:34] ...trap/engines/engine.py.230: loss: 0.12303 [I 2022-10-26 23:12:34] ...optimizers/trijoint.py.151: total_norm: 13.076808 [I 2022-10-26 23:12:40] ...trap/engines/engine.py.224: train: epoch 35 | batch 60/2304 [I 2022-10-26 23:12:40] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:50 [I 2022-10-26 23:12:40] ...trap/engines/engine.py.229: process: 0.56717 | load: 0.00337 [I 2022-10-26 23:12:40] ...trap/engines/engine.py.230: loss: 0.20141 [I 2022-10-26 23:12:40] ...optimizers/trijoint.py.151: total_norm: 18.918877 [I 2022-10-26 23:12:45] ...trap/engines/engine.py.224: train: epoch 35 | batch 70/2304 [I 2022-10-26 23:12:45] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:39 [I 2022-10-26 23:12:45] ...trap/engines/engine.py.229: process: 0.55171 | load: 0.00266 [I 2022-10-26 23:12:45] ...trap/engines/engine.py.230: loss: 0.18097 [I 2022-10-26 23:12:45] ...optimizers/trijoint.py.151: total_norm: 17.074894 [I 2022-10-26 23:12:51] ...trap/engines/engine.py.224: train: epoch 35 | batch 80/2304 [I 2022-10-26 23:12:51] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:30 [I 2022-10-26 23:12:51] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00295 [I 2022-10-26 23:12:51] ...trap/engines/engine.py.230: loss: 0.23832 [I 2022-10-26 23:12:51] ...optimizers/trijoint.py.151: total_norm: 29.500423 [I 2022-10-26 23:12:56] ...trap/engines/engine.py.224: train: epoch 35 | batch 90/2304 [I 2022-10-26 23:12:56] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:39 [I 2022-10-26 23:12:56] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00289 [I 2022-10-26 23:12:56] ...trap/engines/engine.py.230: loss: 0.15993 [I 2022-10-26 23:12:56] ...optimizers/trijoint.py.151: total_norm: 50.590805 [I 2022-10-26 23:13:02] ...trap/engines/engine.py.224: train: epoch 35 | batch 100/2304 [I 2022-10-26 23:13:02] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:17 [I 2022-10-26 23:13:02] ...trap/engines/engine.py.229: process: 0.55309 | load: 0.00255 [I 2022-10-26 23:13:02] ...trap/engines/engine.py.230: loss: 0.15548 [I 2022-10-26 23:13:02] ...optimizers/trijoint.py.151: total_norm: 24.953035 [I 2022-10-26 23:13:07] ...trap/engines/engine.py.224: train: epoch 35 | batch 110/2304 [I 2022-10-26 23:13:07] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:27 [I 2022-10-26 23:13:07] ...trap/engines/engine.py.229: process: 0.57054 | load: 0.00308 [I 2022-10-26 23:13:07] ...trap/engines/engine.py.230: loss: 0.17187 [I 2022-10-26 23:13:07] ...optimizers/trijoint.py.151: total_norm: 19.328157 [I 2022-10-26 23:13:13] ...trap/engines/engine.py.224: train: epoch 35 | batch 120/2304 [I 2022-10-26 23:13:13] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:15 [I 2022-10-26 23:13:13] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00259 [I 2022-10-26 23:13:13] ...trap/engines/engine.py.230: loss: 0.16716 [I 2022-10-26 23:13:13] ...optimizers/trijoint.py.151: total_norm: 18.417175 [I 2022-10-26 23:13:19] ...trap/engines/engine.py.224: train: epoch 35 | batch 130/2304 [I 2022-10-26 23:13:19] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:12 [I 2022-10-26 23:13:19] ...trap/engines/engine.py.229: process: 0.54938 | load: 0.00286 [I 2022-10-26 23:13:19] ...trap/engines/engine.py.230: loss: 0.19400 [I 2022-10-26 23:13:19] ...optimizers/trijoint.py.151: total_norm: 24.148275 [I 2022-10-26 23:13:24] ...trap/engines/engine.py.224: train: epoch 35 | batch 140/2304 [I 2022-10-26 23:13:24] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:05 [I 2022-10-26 23:13:24] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00268 [I 2022-10-26 23:13:24] ...trap/engines/engine.py.230: loss: 0.20196 [I 2022-10-26 23:13:24] ...optimizers/trijoint.py.151: total_norm: 16.423130 [I 2022-10-26 23:13:30] ...trap/engines/engine.py.224: train: epoch 35 | batch 150/2304 [I 2022-10-26 23:13:30] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:57 [I 2022-10-26 23:13:30] ...trap/engines/engine.py.229: process: 0.54762 | load: 0.00255 [I 2022-10-26 23:13:30] ...trap/engines/engine.py.230: loss: 0.19887 [I 2022-10-26 23:13:30] ...optimizers/trijoint.py.151: total_norm: 33.040867 [I 2022-10-26 23:13:56] ...trap/engines/engine.py.224: train: epoch 35 | batch 160/2304 [I 2022-10-26 23:13:56] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 1:36:03 [I 2022-10-26 23:13:56] ...trap/engines/engine.py.229: process: 0.56728 | load: 0.00245 [I 2022-10-26 23:13:56] ...trap/engines/engine.py.230: loss: 0.17653 [I 2022-10-26 23:13:56] ...optimizers/trijoint.py.151: total_norm: 20.712839 [I 2022-10-26 23:14:02] ...trap/engines/engine.py.224: train: epoch 35 | batch 170/2304 [I 2022-10-26 23:14:02] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:27:49 [I 2022-10-26 23:14:02] ...trap/engines/engine.py.229: process: 0.54435 | load: 0.00248 [I 2022-10-26 23:14:02] ...trap/engines/engine.py.230: loss: 0.16079 [I 2022-10-26 23:14:02] ...optimizers/trijoint.py.151: total_norm: 24.570267 [I 2022-10-26 23:14:07] ...trap/engines/engine.py.224: train: epoch 35 | batch 180/2304 [I 2022-10-26 23:14:07] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:20:28 [I 2022-10-26 23:14:07] ...trap/engines/engine.py.229: process: 0.54861 | load: 0.00235 [I 2022-10-26 23:14:07] ...trap/engines/engine.py.230: loss: 0.14884 [I 2022-10-26 23:14:07] ...optimizers/trijoint.py.151: total_norm: 12.740126 [I 2022-10-26 23:14:13] ...trap/engines/engine.py.224: train: epoch 35 | batch 190/2304 [I 2022-10-26 23:14:13] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:19:36 [I 2022-10-26 23:14:13] ...trap/engines/engine.py.229: process: 0.54689 | load: 0.00244 [I 2022-10-26 23:14:13] ...trap/engines/engine.py.230: loss: 0.16507 [I 2022-10-26 23:14:13] ...optimizers/trijoint.py.151: total_norm: 16.244608 [I 2022-10-26 23:14:18] ...trap/engines/engine.py.224: train: epoch 35 | batch 200/2304 [I 2022-10-26 23:14:18] ...trap/engines/engine.py.228: elapsed: 0:02:14 | left: 0:19:21 [I 2022-10-26 23:14:18] ...trap/engines/engine.py.229: process: 0.55112 | load: 0.00271 [I 2022-10-26 23:14:18] ...trap/engines/engine.py.230: loss: 0.17407 [I 2022-10-26 23:14:18] ...optimizers/trijoint.py.151: total_norm: 11.931386 [I 2022-10-26 23:14:24] ...trap/engines/engine.py.224: train: epoch 35 | batch 210/2304 [I 2022-10-26 23:14:24] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:19:17 [I 2022-10-26 23:14:24] ...trap/engines/engine.py.229: process: 0.54462 | load: 0.00262 [I 2022-10-26 23:14:24] ...trap/engines/engine.py.230: loss: 0.19227 [I 2022-10-26 23:14:24] ...optimizers/trijoint.py.151: total_norm: 24.779234 [I 2022-10-26 23:14:29] ...trap/engines/engine.py.224: train: epoch 35 | batch 220/2304 [I 2022-10-26 23:14:29] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:19:24 [I 2022-10-26 23:14:29] ...trap/engines/engine.py.229: process: 0.56479 | load: 0.00247 [I 2022-10-26 23:14:29] ...trap/engines/engine.py.230: loss: 0.14458 [I 2022-10-26 23:14:29] ...optimizers/trijoint.py.151: total_norm: 17.607069 [I 2022-10-26 23:14:35] ...trap/engines/engine.py.224: train: epoch 35 | batch 230/2304 [I 2022-10-26 23:14:35] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:19:18 [I 2022-10-26 23:14:35] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00252 [I 2022-10-26 23:14:35] ...trap/engines/engine.py.230: loss: 0.16059 [I 2022-10-26 23:14:35] ...optimizers/trijoint.py.151: total_norm: 29.080318 [I 2022-10-26 23:14:41] ...trap/engines/engine.py.224: train: epoch 35 | batch 240/2304 [I 2022-10-26 23:14:41] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:19:09 [I 2022-10-26 23:14:41] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00241 [I 2022-10-26 23:14:41] ...trap/engines/engine.py.230: loss: 0.16366 [I 2022-10-26 23:14:41] ...optimizers/trijoint.py.151: total_norm: 12.884063 [I 2022-10-26 23:14:46] ...trap/engines/engine.py.224: train: epoch 35 | batch 250/2304 [I 2022-10-26 23:14:46] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:18:51 [I 2022-10-26 23:14:46] ...trap/engines/engine.py.229: process: 0.53977 | load: 0.00244 [I 2022-10-26 23:14:46] ...trap/engines/engine.py.230: loss: 0.09686 [I 2022-10-26 23:14:46] ...optimizers/trijoint.py.151: total_norm: 7.900097 [I 2022-10-26 23:14:52] ...trap/engines/engine.py.224: train: epoch 35 | batch 260/2304 [I 2022-10-26 23:14:52] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:18:57 [I 2022-10-26 23:14:52] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00239 [I 2022-10-26 23:14:52] ...trap/engines/engine.py.230: loss: 0.14598 [I 2022-10-26 23:14:52] ...optimizers/trijoint.py.151: total_norm: 11.163629 [I 2022-10-26 23:14:57] ...trap/engines/engine.py.224: train: epoch 35 | batch 270/2304 [I 2022-10-26 23:14:57] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:18:53 [I 2022-10-26 23:14:57] ...trap/engines/engine.py.229: process: 0.56704 | load: 0.00259 [I 2022-10-26 23:14:57] ...trap/engines/engine.py.230: loss: 0.17535 [I 2022-10-26 23:14:57] ...optimizers/trijoint.py.151: total_norm: 60.675514 [I 2022-10-26 23:15:03] ...trap/engines/engine.py.224: train: epoch 35 | batch 280/2304 [I 2022-10-26 23:15:03] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:18:45 [I 2022-10-26 23:15:03] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00256 [I 2022-10-26 23:15:03] ...trap/engines/engine.py.230: loss: 0.19031 [I 2022-10-26 23:15:03] ...optimizers/trijoint.py.151: total_norm: 13.863438 [I 2022-10-26 23:15:08] ...trap/engines/engine.py.224: train: epoch 35 | batch 290/2304 [I 2022-10-26 23:15:08] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:18:30 [I 2022-10-26 23:15:08] ...trap/engines/engine.py.229: process: 0.54304 | load: 0.00242 [I 2022-10-26 23:15:08] ...trap/engines/engine.py.230: loss: 0.15889 [I 2022-10-26 23:15:08] ...optimizers/trijoint.py.151: total_norm: 11.532813 [I 2022-10-26 23:15:14] ...trap/engines/engine.py.224: train: epoch 35 | batch 300/2304 [I 2022-10-26 23:15:14] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:18:32 [I 2022-10-26 23:15:14] ...trap/engines/engine.py.229: process: 0.56916 | load: 0.00245 [I 2022-10-26 23:15:14] ...trap/engines/engine.py.230: loss: 0.18118 [I 2022-10-26 23:15:14] ...optimizers/trijoint.py.151: total_norm: 36.767563 [I 2022-10-26 23:15:19] ...trap/engines/engine.py.224: train: epoch 35 | batch 310/2304 [I 2022-10-26 23:15:19] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:18:27 [I 2022-10-26 23:15:19] ...trap/engines/engine.py.229: process: 0.54243 | load: 0.00249 [I 2022-10-26 23:15:19] ...trap/engines/engine.py.230: loss: 0.14445 [I 2022-10-26 23:15:19] ...optimizers/trijoint.py.151: total_norm: 22.034452 [I 2022-10-26 23:15:25] ...trap/engines/engine.py.224: train: epoch 35 | batch 320/2304 [I 2022-10-26 23:15:25] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:18:25 [I 2022-10-26 23:15:25] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00242 [I 2022-10-26 23:15:25] ...trap/engines/engine.py.230: loss: 0.20430 [I 2022-10-26 23:15:25] ...optimizers/trijoint.py.151: total_norm: 22.382030 [I 2022-10-26 23:15:31] ...trap/engines/engine.py.224: train: epoch 35 | batch 330/2304 [I 2022-10-26 23:15:31] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:18:35 [I 2022-10-26 23:15:31] ...trap/engines/engine.py.229: process: 0.60326 | load: 0.00261 [I 2022-10-26 23:15:31] ...trap/engines/engine.py.230: loss: 0.14258 [I 2022-10-26 23:15:31] ...optimizers/trijoint.py.151: total_norm: 17.858892 [I 2022-10-26 23:15:36] ...trap/engines/engine.py.224: train: epoch 35 | batch 340/2304 [I 2022-10-26 23:15:36] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:18:30 [I 2022-10-26 23:15:36] ...trap/engines/engine.py.229: process: 0.57667 | load: 0.00246 [I 2022-10-26 23:15:36] ...trap/engines/engine.py.230: loss: 0.15825 [I 2022-10-26 23:15:36] ...optimizers/trijoint.py.151: total_norm: 29.052891 [I 2022-10-26 23:15:42] ...trap/engines/engine.py.224: train: epoch 35 | batch 350/2304 [I 2022-10-26 23:15:42] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:18:02 [I 2022-10-26 23:15:42] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00241 [I 2022-10-26 23:15:42] ...trap/engines/engine.py.230: loss: 0.15543 [I 2022-10-26 23:15:42] ...optimizers/trijoint.py.151: total_norm: 21.050871 [I 2022-10-26 23:15:48] ...trap/engines/engine.py.224: train: epoch 35 | batch 360/2304 [I 2022-10-26 23:15:48] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:18:06 [I 2022-10-26 23:15:48] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00240 [I 2022-10-26 23:15:48] ...trap/engines/engine.py.230: loss: 0.17043 [I 2022-10-26 23:15:48] ...optimizers/trijoint.py.151: total_norm: 17.832970 [I 2022-10-26 23:15:53] ...trap/engines/engine.py.224: train: epoch 35 | batch 370/2304 [I 2022-10-26 23:15:53] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:17:53 [I 2022-10-26 23:15:53] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00254 [I 2022-10-26 23:15:53] ...trap/engines/engine.py.230: loss: 0.18178 [I 2022-10-26 23:15:53] ...optimizers/trijoint.py.151: total_norm: 17.978834 [I 2022-10-26 23:15:59] ...trap/engines/engine.py.224: train: epoch 35 | batch 380/2304 [I 2022-10-26 23:15:59] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:17:51 [I 2022-10-26 23:15:59] ...trap/engines/engine.py.229: process: 0.55462 | load: 0.00245 [I 2022-10-26 23:15:59] ...trap/engines/engine.py.230: loss: 0.11999 [I 2022-10-26 23:15:59] ...optimizers/trijoint.py.151: total_norm: 19.023172 [I 2022-10-26 23:16:04] ...trap/engines/engine.py.224: train: epoch 35 | batch 390/2304 [I 2022-10-26 23:16:04] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:17:40 [I 2022-10-26 23:16:04] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00248 [I 2022-10-26 23:16:04] ...trap/engines/engine.py.230: loss: 0.13826 [I 2022-10-26 23:16:04] ...optimizers/trijoint.py.151: total_norm: 18.490587 [I 2022-10-26 23:16:10] ...trap/engines/engine.py.224: train: epoch 35 | batch 400/2304 [I 2022-10-26 23:16:10] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:17:38 [I 2022-10-26 23:16:10] ...trap/engines/engine.py.229: process: 0.55426 | load: 0.00246 [I 2022-10-26 23:16:10] ...trap/engines/engine.py.230: loss: 0.24833 [I 2022-10-26 23:16:10] ...optimizers/trijoint.py.151: total_norm: 39.322994 [I 2022-10-26 23:16:15] ...trap/engines/engine.py.224: train: epoch 35 | batch 410/2304 [I 2022-10-26 23:16:15] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:17:31 [I 2022-10-26 23:16:15] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00240 [I 2022-10-26 23:16:15] ...trap/engines/engine.py.230: loss: 0.20258 [I 2022-10-26 23:16:15] ...optimizers/trijoint.py.151: total_norm: 26.507259 [I 2022-10-26 23:16:21] ...trap/engines/engine.py.224: train: epoch 35 | batch 420/2304 [I 2022-10-26 23:16:21] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:18:19 [I 2022-10-26 23:16:21] ...trap/engines/engine.py.229: process: 0.68686 | load: 0.00263 [I 2022-10-26 23:16:21] ...trap/engines/engine.py.230: loss: 0.12543 [I 2022-10-26 23:16:21] ...optimizers/trijoint.py.151: total_norm: 12.835597 [I 2022-10-26 23:16:27] ...trap/engines/engine.py.224: train: epoch 35 | batch 430/2304 [I 2022-10-26 23:16:27] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:17:19 [I 2022-10-26 23:16:27] ...trap/engines/engine.py.229: process: 0.53989 | load: 0.00246 [I 2022-10-26 23:16:27] ...trap/engines/engine.py.230: loss: 0.16951 [I 2022-10-26 23:16:27] ...optimizers/trijoint.py.151: total_norm: 16.333487 [I 2022-10-26 23:16:32] ...trap/engines/engine.py.224: train: epoch 35 | batch 440/2304 [I 2022-10-26 23:16:32] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:17:20 [I 2022-10-26 23:16:32] ...trap/engines/engine.py.229: process: 0.54740 | load: 0.00255 [I 2022-10-26 23:16:32] ...trap/engines/engine.py.230: loss: 0.11018 [I 2022-10-26 23:16:32] ...optimizers/trijoint.py.151: total_norm: 46.345764 [I 2022-10-26 23:16:38] ...trap/engines/engine.py.224: train: epoch 35 | batch 450/2304 [I 2022-10-26 23:16:38] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:17:22 [I 2022-10-26 23:16:38] ...trap/engines/engine.py.229: process: 0.57910 | load: 0.00243 [I 2022-10-26 23:16:38] ...trap/engines/engine.py.230: loss: 0.22350 [I 2022-10-26 23:16:38] ...optimizers/trijoint.py.151: total_norm: 19.744606 [I 2022-10-26 23:16:43] ...trap/engines/engine.py.224: train: epoch 35 | batch 460/2304 [I 2022-10-26 23:16:43] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:17:10 [I 2022-10-26 23:16:43] ...trap/engines/engine.py.229: process: 0.56484 | load: 0.00248 [I 2022-10-26 23:16:43] ...trap/engines/engine.py.230: loss: 0.12024 [I 2022-10-26 23:16:43] ...optimizers/trijoint.py.151: total_norm: 15.828240 [I 2022-10-26 23:16:49] ...trap/engines/engine.py.224: train: epoch 35 | batch 470/2304 [I 2022-10-26 23:16:49] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:17:02 [I 2022-10-26 23:16:49] ...trap/engines/engine.py.229: process: 0.55809 | load: 0.00246 [I 2022-10-26 23:16:49] ...trap/engines/engine.py.230: loss: 0.15034 [I 2022-10-26 23:16:49] ...optimizers/trijoint.py.151: total_norm: 23.590738 [I 2022-10-26 23:16:55] ...trap/engines/engine.py.224: train: epoch 35 | batch 480/2304 [I 2022-10-26 23:16:55] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:16:53 [I 2022-10-26 23:16:55] ...trap/engines/engine.py.229: process: 0.54828 | load: 0.00249 [I 2022-10-26 23:16:55] ...trap/engines/engine.py.230: loss: 0.19665 [I 2022-10-26 23:16:55] ...optimizers/trijoint.py.151: total_norm: 22.938583 [I 2022-10-26 23:17:00] ...trap/engines/engine.py.224: train: epoch 35 | batch 490/2304 [I 2022-10-26 23:17:00] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:17:05 [I 2022-10-26 23:17:00] ...trap/engines/engine.py.229: process: 0.54875 | load: 0.00246 [I 2022-10-26 23:17:00] ...trap/engines/engine.py.230: loss: 0.17790 [I 2022-10-26 23:17:00] ...optimizers/trijoint.py.151: total_norm: 13.148652 [I 2022-10-26 23:17:06] ...trap/engines/engine.py.224: train: epoch 35 | batch 500/2304 [I 2022-10-26 23:17:06] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:16:41 [I 2022-10-26 23:17:06] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00243 [I 2022-10-26 23:17:06] ...trap/engines/engine.py.230: loss: 0.14612 [I 2022-10-26 23:17:06] ...optimizers/trijoint.py.151: total_norm: 67.064270 [I 2022-10-26 23:17:11] ...trap/engines/engine.py.224: train: epoch 35 | batch 510/2304 [I 2022-10-26 23:17:11] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:16:35 [I 2022-10-26 23:17:11] ...trap/engines/engine.py.229: process: 0.55028 | load: 0.00246 [I 2022-10-26 23:17:11] ...trap/engines/engine.py.230: loss: 0.22387 [I 2022-10-26 23:17:11] ...optimizers/trijoint.py.151: total_norm: 21.802340 [I 2022-10-26 23:17:17] ...trap/engines/engine.py.224: train: epoch 35 | batch 520/2304 [I 2022-10-26 23:17:17] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:16:27 [I 2022-10-26 23:17:17] ...trap/engines/engine.py.229: process: 0.55208 | load: 0.00246 [I 2022-10-26 23:17:17] ...trap/engines/engine.py.230: loss: 0.14932 [I 2022-10-26 23:17:17] ...optimizers/trijoint.py.151: total_norm: 11.811302 [I 2022-10-26 23:17:23] ...trap/engines/engine.py.224: train: epoch 35 | batch 530/2304 [I 2022-10-26 23:17:23] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:16:20 [I 2022-10-26 23:17:23] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00242 [I 2022-10-26 23:17:23] ...trap/engines/engine.py.230: loss: 0.15849 [I 2022-10-26 23:17:23] ...optimizers/trijoint.py.151: total_norm: 26.788767 [I 2022-10-26 23:17:28] ...trap/engines/engine.py.224: train: epoch 35 | batch 540/2304 [I 2022-10-26 23:17:28] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:16:23 [I 2022-10-26 23:17:28] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00243 [I 2022-10-26 23:17:28] ...trap/engines/engine.py.230: loss: 0.23016 [I 2022-10-26 23:17:28] ...optimizers/trijoint.py.151: total_norm: 24.470551 [I 2022-10-26 23:17:34] ...trap/engines/engine.py.224: train: epoch 35 | batch 550/2304 [I 2022-10-26 23:17:34] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:16:09 [I 2022-10-26 23:17:34] ...trap/engines/engine.py.229: process: 0.54329 | load: 0.00244 [I 2022-10-26 23:17:34] ...trap/engines/engine.py.230: loss: 0.15244 [I 2022-10-26 23:17:34] ...optimizers/trijoint.py.151: total_norm: 18.478506 [I 2022-10-26 23:17:39] ...trap/engines/engine.py.224: train: epoch 35 | batch 560/2304 [I 2022-10-26 23:17:39] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:16:07 [I 2022-10-26 23:17:39] ...trap/engines/engine.py.229: process: 0.55103 | load: 0.00244 [I 2022-10-26 23:17:39] ...trap/engines/engine.py.230: loss: 0.14871 [I 2022-10-26 23:17:39] ...optimizers/trijoint.py.151: total_norm: 21.609947 [I 2022-10-26 23:17:45] ...trap/engines/engine.py.224: train: epoch 35 | batch 570/2304 [I 2022-10-26 23:17:45] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:15:56 [I 2022-10-26 23:17:45] ...trap/engines/engine.py.229: process: 0.54028 | load: 0.00266 [I 2022-10-26 23:17:45] ...trap/engines/engine.py.230: loss: 0.20822 [I 2022-10-26 23:17:45] ...optimizers/trijoint.py.151: total_norm: 14.878435 [I 2022-10-26 23:17:50] ...trap/engines/engine.py.224: train: epoch 35 | batch 580/2304 [I 2022-10-26 23:17:50] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:15:55 [I 2022-10-26 23:17:50] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00249 [I 2022-10-26 23:17:50] ...trap/engines/engine.py.230: loss: 0.13472 [I 2022-10-26 23:17:50] ...optimizers/trijoint.py.151: total_norm: 17.245747 [I 2022-10-26 23:17:56] ...trap/engines/engine.py.224: train: epoch 35 | batch 590/2304 [I 2022-10-26 23:17:56] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:15:45 [I 2022-10-26 23:17:56] ...trap/engines/engine.py.229: process: 0.54923 | load: 0.00251 [I 2022-10-26 23:17:56] ...trap/engines/engine.py.230: loss: 0.11908 [I 2022-10-26 23:17:56] ...optimizers/trijoint.py.151: total_norm: 23.250664 [I 2022-10-26 23:18:01] ...trap/engines/engine.py.224: train: epoch 35 | batch 600/2304 [I 2022-10-26 23:18:01] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:55 [I 2022-10-26 23:18:01] ...trap/engines/engine.py.229: process: 0.57149 | load: 0.00246 [I 2022-10-26 23:18:01] ...trap/engines/engine.py.230: loss: 0.16422 [I 2022-10-26 23:18:01] ...optimizers/trijoint.py.151: total_norm: 12.893618 [I 2022-10-26 23:18:07] ...trap/engines/engine.py.224: train: epoch 35 | batch 610/2304 [I 2022-10-26 23:18:07] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:15:50 [I 2022-10-26 23:18:07] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00240 [I 2022-10-26 23:18:07] ...trap/engines/engine.py.230: loss: 0.16362 [I 2022-10-26 23:18:07] ...optimizers/trijoint.py.151: total_norm: 15.527905 [I 2022-10-26 23:18:13] ...trap/engines/engine.py.224: train: epoch 35 | batch 620/2304 [I 2022-10-26 23:18:13] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:51 [I 2022-10-26 23:18:13] ...trap/engines/engine.py.229: process: 0.57816 | load: 0.00249 [I 2022-10-26 23:18:13] ...trap/engines/engine.py.230: loss: 0.27898 [I 2022-10-26 23:18:13] ...optimizers/trijoint.py.151: total_norm: 30.308853 [I 2022-10-26 23:18:18] ...trap/engines/engine.py.224: train: epoch 35 | batch 630/2304 [I 2022-10-26 23:18:18] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:39 [I 2022-10-26 23:18:18] ...trap/engines/engine.py.229: process: 0.53808 | load: 0.00244 [I 2022-10-26 23:18:18] ...trap/engines/engine.py.230: loss: 0.13072 [I 2022-10-26 23:18:18] ...optimizers/trijoint.py.151: total_norm: 14.746413 [I 2022-10-26 23:18:24] ...trap/engines/engine.py.224: train: epoch 35 | batch 640/2304 [I 2022-10-26 23:18:24] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:26 [I 2022-10-26 23:18:24] ...trap/engines/engine.py.229: process: 0.54997 | load: 0.00246 [I 2022-10-26 23:18:24] ...trap/engines/engine.py.230: loss: 0.11256 [I 2022-10-26 23:18:24] ...optimizers/trijoint.py.151: total_norm: 7.429399 [I 2022-10-26 23:18:30] ...trap/engines/engine.py.224: train: epoch 35 | batch 650/2304 [I 2022-10-26 23:18:30] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:26 [I 2022-10-26 23:18:30] ...trap/engines/engine.py.229: process: 0.54694 | load: 0.00242 [I 2022-10-26 23:18:30] ...trap/engines/engine.py.230: loss: 0.20645 [I 2022-10-26 23:18:30] ...optimizers/trijoint.py.151: total_norm: 21.207207 [I 2022-10-26 23:18:35] ...trap/engines/engine.py.224: train: epoch 35 | batch 660/2304 [I 2022-10-26 23:18:35] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:15:25 [I 2022-10-26 23:18:35] ...trap/engines/engine.py.229: process: 0.56889 | load: 0.00243 [I 2022-10-26 23:18:35] ...trap/engines/engine.py.230: loss: 0.14292 [I 2022-10-26 23:18:35] ...optimizers/trijoint.py.151: total_norm: 15.524372 [I 2022-10-26 23:18:41] ...trap/engines/engine.py.224: train: epoch 35 | batch 670/2304 [I 2022-10-26 23:18:41] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:21 [I 2022-10-26 23:18:41] ...trap/engines/engine.py.229: process: 0.55213 | load: 0.00243 [I 2022-10-26 23:18:41] ...trap/engines/engine.py.230: loss: 0.30850 [I 2022-10-26 23:18:41] ...optimizers/trijoint.py.151: total_norm: 21.401361 [I 2022-10-26 23:18:47] ...trap/engines/engine.py.224: train: epoch 35 | batch 680/2304 [I 2022-10-26 23:18:47] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:17 [I 2022-10-26 23:18:47] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00247 [I 2022-10-26 23:18:47] ...trap/engines/engine.py.230: loss: 0.15955 [I 2022-10-26 23:18:47] ...optimizers/trijoint.py.151: total_norm: 16.128504 [I 2022-10-26 23:18:52] ...trap/engines/engine.py.224: train: epoch 35 | batch 690/2304 [I 2022-10-26 23:18:52] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:15:05 [I 2022-10-26 23:18:52] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00248 [I 2022-10-26 23:18:52] ...trap/engines/engine.py.230: loss: 0.15937 [I 2022-10-26 23:18:52] ...optimizers/trijoint.py.151: total_norm: 18.686293 [I 2022-10-26 23:18:58] ...trap/engines/engine.py.224: train: epoch 35 | batch 700/2304 [I 2022-10-26 23:18:58] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:15:02 [I 2022-10-26 23:18:58] ...trap/engines/engine.py.229: process: 0.58589 | load: 0.00254 [I 2022-10-26 23:18:58] ...trap/engines/engine.py.230: loss: 0.15464 [I 2022-10-26 23:18:58] ...optimizers/trijoint.py.151: total_norm: 14.752529 [I 2022-10-26 23:19:04] ...trap/engines/engine.py.224: train: epoch 35 | batch 710/2304 [I 2022-10-26 23:19:04] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:15:23 [I 2022-10-26 23:19:04] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00247 [I 2022-10-26 23:19:04] ...trap/engines/engine.py.230: loss: 0.15366 [I 2022-10-26 23:19:04] ...optimizers/trijoint.py.151: total_norm: 13.915894 [I 2022-10-26 23:19:09] ...trap/engines/engine.py.224: train: epoch 35 | batch 720/2304 [I 2022-10-26 23:19:09] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:59 [I 2022-10-26 23:19:09] ...trap/engines/engine.py.229: process: 0.58230 | load: 0.00244 [I 2022-10-26 23:19:09] ...trap/engines/engine.py.230: loss: 0.19397 [I 2022-10-26 23:19:09] ...optimizers/trijoint.py.151: total_norm: 12.931673 [I 2022-10-26 23:19:15] ...trap/engines/engine.py.224: train: epoch 35 | batch 730/2304 [I 2022-10-26 23:19:15] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:46 [I 2022-10-26 23:19:15] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00244 [I 2022-10-26 23:19:15] ...trap/engines/engine.py.230: loss: 0.24019 [I 2022-10-26 23:19:15] ...optimizers/trijoint.py.151: total_norm: 41.350716 [I 2022-10-26 23:19:21] ...trap/engines/engine.py.224: train: epoch 35 | batch 740/2304 [I 2022-10-26 23:19:21] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:14:37 [I 2022-10-26 23:19:21] ...trap/engines/engine.py.229: process: 0.57092 | load: 0.00245 [I 2022-10-26 23:19:21] ...trap/engines/engine.py.230: loss: 0.24456 [I 2022-10-26 23:19:21] ...optimizers/trijoint.py.151: total_norm: 16.102161 [I 2022-10-26 23:19:26] ...trap/engines/engine.py.224: train: epoch 35 | batch 750/2304 [I 2022-10-26 23:19:26] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:35 [I 2022-10-26 23:19:26] ...trap/engines/engine.py.229: process: 0.55507 | load: 0.00242 [I 2022-10-26 23:19:26] ...trap/engines/engine.py.230: loss: 0.18709 [I 2022-10-26 23:19:26] ...optimizers/trijoint.py.151: total_norm: 11.874795 [I 2022-10-26 23:19:32] ...trap/engines/engine.py.224: train: epoch 35 | batch 760/2304 [I 2022-10-26 23:19:32] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:27 [I 2022-10-26 23:19:32] ...trap/engines/engine.py.229: process: 0.56475 | load: 0.00240 [I 2022-10-26 23:19:32] ...trap/engines/engine.py.230: loss: 0.15215 [I 2022-10-26 23:19:32] ...optimizers/trijoint.py.151: total_norm: 28.604050 [I 2022-10-26 23:19:37] ...trap/engines/engine.py.224: train: epoch 35 | batch 770/2304 [I 2022-10-26 23:19:37] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:14:17 [I 2022-10-26 23:19:37] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00251 [I 2022-10-26 23:19:37] ...trap/engines/engine.py.230: loss: 0.14134 [I 2022-10-26 23:19:37] ...optimizers/trijoint.py.151: total_norm: 10.694569 [I 2022-10-26 23:19:43] ...trap/engines/engine.py.224: train: epoch 35 | batch 780/2304 [I 2022-10-26 23:19:43] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:17 [I 2022-10-26 23:19:43] ...trap/engines/engine.py.229: process: 0.56684 | load: 0.00245 [I 2022-10-26 23:19:43] ...trap/engines/engine.py.230: loss: 0.27875 [I 2022-10-26 23:19:43] ...optimizers/trijoint.py.151: total_norm: 41.542038 [I 2022-10-26 23:19:49] ...trap/engines/engine.py.224: train: epoch 35 | batch 790/2304 [I 2022-10-26 23:19:49] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:14:13 [I 2022-10-26 23:19:49] ...trap/engines/engine.py.229: process: 0.55297 | load: 0.00245 [I 2022-10-26 23:19:49] ...trap/engines/engine.py.230: loss: 0.20297 [I 2022-10-26 23:19:49] ...optimizers/trijoint.py.151: total_norm: 14.226091 [I 2022-10-26 23:19:54] ...trap/engines/engine.py.224: train: epoch 35 | batch 800/2304 [I 2022-10-26 23:19:54] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:14:07 [I 2022-10-26 23:19:54] ...trap/engines/engine.py.229: process: 0.58019 | load: 0.00241 [I 2022-10-26 23:19:54] ...trap/engines/engine.py.230: loss: 0.16753 [I 2022-10-26 23:19:54] ...optimizers/trijoint.py.151: total_norm: 31.172792 [I 2022-10-26 23:20:00] ...trap/engines/engine.py.224: train: epoch 35 | batch 810/2304 [I 2022-10-26 23:20:00] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:53 [I 2022-10-26 23:20:00] ...trap/engines/engine.py.229: process: 0.54614 | load: 0.00254 [I 2022-10-26 23:20:00] ...trap/engines/engine.py.230: loss: 0.24352 [I 2022-10-26 23:20:00] ...optimizers/trijoint.py.151: total_norm: 26.626436 [I 2022-10-26 23:20:06] ...trap/engines/engine.py.224: train: epoch 35 | batch 820/2304 [I 2022-10-26 23:20:06] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:14:15 [I 2022-10-26 23:20:06] ...trap/engines/engine.py.229: process: 0.63095 | load: 0.00252 [I 2022-10-26 23:20:06] ...trap/engines/engine.py.230: loss: 0.17531 [I 2022-10-26 23:20:06] ...optimizers/trijoint.py.151: total_norm: 19.949841 [I 2022-10-26 23:20:11] ...trap/engines/engine.py.224: train: epoch 35 | batch 830/2304 [I 2022-10-26 23:20:11] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:50 [I 2022-10-26 23:20:11] ...trap/engines/engine.py.229: process: 0.55152 | load: 0.00296 [I 2022-10-26 23:20:11] ...trap/engines/engine.py.230: loss: 0.19929 [I 2022-10-26 23:20:11] ...optimizers/trijoint.py.151: total_norm: 16.322815 [I 2022-10-26 23:20:17] ...trap/engines/engine.py.224: train: epoch 35 | batch 840/2304 [I 2022-10-26 23:20:17] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:40 [I 2022-10-26 23:20:17] ...trap/engines/engine.py.229: process: 0.56732 | load: 0.00250 [I 2022-10-26 23:20:17] ...trap/engines/engine.py.230: loss: 0.18674 [I 2022-10-26 23:20:17] ...optimizers/trijoint.py.151: total_norm: 22.940655 [I 2022-10-26 23:20:23] ...trap/engines/engine.py.224: train: epoch 35 | batch 850/2304 [I 2022-10-26 23:20:23] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:37 [I 2022-10-26 23:20:23] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00248 [I 2022-10-26 23:20:23] ...trap/engines/engine.py.230: loss: 0.17160 [I 2022-10-26 23:20:23] ...optimizers/trijoint.py.151: total_norm: 19.949223 [I 2022-10-26 23:20:28] ...trap/engines/engine.py.224: train: epoch 35 | batch 860/2304 [I 2022-10-26 23:20:28] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:27 [I 2022-10-26 23:20:28] ...trap/engines/engine.py.229: process: 0.54775 | load: 0.00250 [I 2022-10-26 23:20:28] ...trap/engines/engine.py.230: loss: 0.18082 [I 2022-10-26 23:20:28] ...optimizers/trijoint.py.151: total_norm: 29.355816 [I 2022-10-26 23:20:34] ...trap/engines/engine.py.224: train: epoch 35 | batch 870/2304 [I 2022-10-26 23:20:34] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:13:14 [I 2022-10-26 23:20:34] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00243 [I 2022-10-26 23:20:34] ...trap/engines/engine.py.230: loss: 0.12688 [I 2022-10-26 23:20:34] ...optimizers/trijoint.py.151: total_norm: 21.455978 [I 2022-10-26 23:20:39] ...trap/engines/engine.py.224: train: epoch 35 | batch 880/2304 [I 2022-10-26 23:20:39] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:17 [I 2022-10-26 23:20:39] ...trap/engines/engine.py.229: process: 0.58901 | load: 0.00245 [I 2022-10-26 23:20:39] ...trap/engines/engine.py.230: loss: 0.13110 [I 2022-10-26 23:20:39] ...optimizers/trijoint.py.151: total_norm: 21.881041 [I 2022-10-26 23:20:45] ...trap/engines/engine.py.224: train: epoch 35 | batch 890/2304 [I 2022-10-26 23:20:45] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:13:17 [I 2022-10-26 23:20:45] ...trap/engines/engine.py.229: process: 0.56686 | load: 0.00241 [I 2022-10-26 23:20:45] ...trap/engines/engine.py.230: loss: 0.14152 [I 2022-10-26 23:20:45] ...optimizers/trijoint.py.151: total_norm: 15.449404 [I 2022-10-26 23:20:51] ...trap/engines/engine.py.224: train: epoch 35 | batch 900/2304 [I 2022-10-26 23:20:51] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:13:03 [I 2022-10-26 23:20:51] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00242 [I 2022-10-26 23:20:51] ...trap/engines/engine.py.230: loss: 0.20328 [I 2022-10-26 23:20:51] ...optimizers/trijoint.py.151: total_norm: 22.599461 [I 2022-10-26 23:20:56] ...trap/engines/engine.py.224: train: epoch 35 | batch 910/2304 [I 2022-10-26 23:20:56] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:55 [I 2022-10-26 23:20:56] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00247 [I 2022-10-26 23:20:56] ...trap/engines/engine.py.230: loss: 0.13928 [I 2022-10-26 23:20:56] ...optimizers/trijoint.py.151: total_norm: 14.342891 [I 2022-10-26 23:21:02] ...trap/engines/engine.py.224: train: epoch 35 | batch 920/2304 [I 2022-10-26 23:21:02] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:51 [I 2022-10-26 23:21:02] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00256 [I 2022-10-26 23:21:02] ...trap/engines/engine.py.230: loss: 0.15916 [I 2022-10-26 23:21:02] ...optimizers/trijoint.py.151: total_norm: 13.815304 [I 2022-10-26 23:21:07] ...trap/engines/engine.py.224: train: epoch 35 | batch 930/2304 [I 2022-10-26 23:21:07] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:42 [I 2022-10-26 23:21:07] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00247 [I 2022-10-26 23:21:07] ...trap/engines/engine.py.230: loss: 0.14606 [I 2022-10-26 23:21:07] ...optimizers/trijoint.py.151: total_norm: 28.806303 [I 2022-10-26 23:21:13] ...trap/engines/engine.py.224: train: epoch 35 | batch 940/2304 [I 2022-10-26 23:21:13] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:39 [I 2022-10-26 23:21:13] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00271 [I 2022-10-26 23:21:13] ...trap/engines/engine.py.230: loss: 0.12241 [I 2022-10-26 23:21:13] ...optimizers/trijoint.py.151: total_norm: 11.453979 [I 2022-10-26 23:21:18] ...trap/engines/engine.py.224: train: epoch 35 | batch 950/2304 [I 2022-10-26 23:21:18] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:26 [I 2022-10-26 23:21:18] ...trap/engines/engine.py.229: process: 0.55632 | load: 0.00250 [I 2022-10-26 23:21:18] ...trap/engines/engine.py.230: loss: 0.24118 [I 2022-10-26 23:21:18] ...optimizers/trijoint.py.151: total_norm: 16.710064 [I 2022-10-26 23:21:24] ...trap/engines/engine.py.224: train: epoch 35 | batch 960/2304 [I 2022-10-26 23:21:24] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:24 [I 2022-10-26 23:21:24] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00244 [I 2022-10-26 23:21:24] ...trap/engines/engine.py.230: loss: 0.17334 [I 2022-10-26 23:21:24] ...optimizers/trijoint.py.151: total_norm: 18.823833 [I 2022-10-26 23:21:30] ...trap/engines/engine.py.224: train: epoch 35 | batch 970/2304 [I 2022-10-26 23:21:30] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:24 [I 2022-10-26 23:21:30] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00248 [I 2022-10-26 23:21:30] ...trap/engines/engine.py.230: loss: 0.13504 [I 2022-10-26 23:21:30] ...optimizers/trijoint.py.151: total_norm: 15.946180 [I 2022-10-26 23:21:35] ...trap/engines/engine.py.224: train: epoch 35 | batch 980/2304 [I 2022-10-26 23:21:35] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:26 [I 2022-10-26 23:21:35] ...trap/engines/engine.py.229: process: 0.57929 | load: 0.00245 [I 2022-10-26 23:21:35] ...trap/engines/engine.py.230: loss: 0.17595 [I 2022-10-26 23:21:35] ...optimizers/trijoint.py.151: total_norm: 12.582415 [I 2022-10-26 23:21:41] ...trap/engines/engine.py.224: train: epoch 35 | batch 990/2304 [I 2022-10-26 23:21:41] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:12:11 [I 2022-10-26 23:21:41] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00242 [I 2022-10-26 23:21:41] ...trap/engines/engine.py.230: loss: 0.22694 [I 2022-10-26 23:21:41] ...optimizers/trijoint.py.151: total_norm: 24.788532 [I 2022-10-26 23:21:46] ...trap/engines/engine.py.224: train: epoch 35 | batch 1000/2304 [I 2022-10-26 23:21:46] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:04 [I 2022-10-26 23:21:46] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00247 [I 2022-10-26 23:21:46] ...trap/engines/engine.py.230: loss: 0.11997 [I 2022-10-26 23:21:46] ...optimizers/trijoint.py.151: total_norm: 24.508478 [I 2022-10-26 23:21:52] ...trap/engines/engine.py.224: train: epoch 35 | batch 1010/2304 [I 2022-10-26 23:21:52] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:12:00 [I 2022-10-26 23:21:52] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00249 [I 2022-10-26 23:21:52] ...trap/engines/engine.py.230: loss: 0.17421 [I 2022-10-26 23:21:52] ...optimizers/trijoint.py.151: total_norm: 21.449770 [I 2022-10-26 23:21:57] ...trap/engines/engine.py.224: train: epoch 35 | batch 1020/2304 [I 2022-10-26 23:21:57] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:51 [I 2022-10-26 23:21:57] ...trap/engines/engine.py.229: process: 0.54795 | load: 0.00243 [I 2022-10-26 23:21:57] ...trap/engines/engine.py.230: loss: 0.13203 [I 2022-10-26 23:21:57] ...optimizers/trijoint.py.151: total_norm: 15.124069 [I 2022-10-26 23:22:03] ...trap/engines/engine.py.224: train: epoch 35 | batch 1030/2304 [I 2022-10-26 23:22:03] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:47 [I 2022-10-26 23:22:03] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00240 [I 2022-10-26 23:22:03] ...trap/engines/engine.py.230: loss: 0.12618 [I 2022-10-26 23:22:03] ...optimizers/trijoint.py.151: total_norm: 9.469392 [I 2022-10-26 23:22:09] ...trap/engines/engine.py.224: train: epoch 35 | batch 1040/2304 [I 2022-10-26 23:22:09] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:43 [I 2022-10-26 23:22:09] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00243 [I 2022-10-26 23:22:09] ...trap/engines/engine.py.230: loss: 0.15285 [I 2022-10-26 23:22:09] ...optimizers/trijoint.py.151: total_norm: 25.249277 [I 2022-10-26 23:22:14] ...trap/engines/engine.py.224: train: epoch 35 | batch 1050/2304 [I 2022-10-26 23:22:14] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:40 [I 2022-10-26 23:22:14] ...trap/engines/engine.py.229: process: 0.56222 | load: 0.00239 [I 2022-10-26 23:22:14] ...trap/engines/engine.py.230: loss: 0.18272 [I 2022-10-26 23:22:14] ...optimizers/trijoint.py.151: total_norm: 19.137512 [I 2022-10-26 23:22:20] ...trap/engines/engine.py.224: train: epoch 35 | batch 1060/2304 [I 2022-10-26 23:22:20] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:33 [I 2022-10-26 23:22:20] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00241 [I 2022-10-26 23:22:20] ...trap/engines/engine.py.230: loss: 0.25866 [I 2022-10-26 23:22:20] ...optimizers/trijoint.py.151: total_norm: 30.774837 [I 2022-10-26 23:22:25] ...trap/engines/engine.py.224: train: epoch 35 | batch 1070/2304 [I 2022-10-26 23:22:25] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:29 [I 2022-10-26 23:22:25] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00243 [I 2022-10-26 23:22:25] ...trap/engines/engine.py.230: loss: 0.22747 [I 2022-10-26 23:22:25] ...optimizers/trijoint.py.151: total_norm: 36.294460 [I 2022-10-26 23:22:31] ...trap/engines/engine.py.224: train: epoch 35 | batch 1080/2304 [I 2022-10-26 23:22:31] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:20 [I 2022-10-26 23:22:31] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00240 [I 2022-10-26 23:22:31] ...trap/engines/engine.py.230: loss: 0.19702 [I 2022-10-26 23:22:31] ...optimizers/trijoint.py.151: total_norm: 20.819559 [I 2022-10-26 23:22:36] ...trap/engines/engine.py.224: train: epoch 35 | batch 1090/2304 [I 2022-10-26 23:22:36] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:11:12 [I 2022-10-26 23:22:36] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00242 [I 2022-10-26 23:22:36] ...trap/engines/engine.py.230: loss: 0.18765 [I 2022-10-26 23:22:36] ...optimizers/trijoint.py.151: total_norm: 24.711094 [I 2022-10-26 23:22:42] ...trap/engines/engine.py.224: train: epoch 35 | batch 1100/2304 [I 2022-10-26 23:22:42] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:11:08 [I 2022-10-26 23:22:42] ...trap/engines/engine.py.229: process: 0.55190 | load: 0.00240 [I 2022-10-26 23:22:42] ...trap/engines/engine.py.230: loss: 0.14941 [I 2022-10-26 23:22:42] ...optimizers/trijoint.py.151: total_norm: 11.288938 [I 2022-10-26 23:22:48] ...trap/engines/engine.py.224: train: epoch 35 | batch 1110/2304 [I 2022-10-26 23:22:48] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:11:03 [I 2022-10-26 23:22:48] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00247 [I 2022-10-26 23:22:48] ...trap/engines/engine.py.230: loss: 0.15842 [I 2022-10-26 23:22:48] ...optimizers/trijoint.py.151: total_norm: 13.180889 [I 2022-10-26 23:22:53] ...trap/engines/engine.py.224: train: epoch 35 | batch 1120/2304 [I 2022-10-26 23:22:53] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:10:58 [I 2022-10-26 23:22:53] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00280 [I 2022-10-26 23:22:53] ...trap/engines/engine.py.230: loss: 0.21653 [I 2022-10-26 23:22:53] ...optimizers/trijoint.py.151: total_norm: 29.261297 [I 2022-10-26 23:22:59] ...trap/engines/engine.py.224: train: epoch 35 | batch 1130/2304 [I 2022-10-26 23:22:59] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:51 [I 2022-10-26 23:22:59] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00269 [I 2022-10-26 23:22:59] ...trap/engines/engine.py.230: loss: 0.11676 [I 2022-10-26 23:22:59] ...optimizers/trijoint.py.151: total_norm: 17.926525 [I 2022-10-26 23:23:04] ...trap/engines/engine.py.224: train: epoch 35 | batch 1140/2304 [I 2022-10-26 23:23:04] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:48 [I 2022-10-26 23:23:04] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00258 [I 2022-10-26 23:23:04] ...trap/engines/engine.py.230: loss: 0.14294 [I 2022-10-26 23:23:04] ...optimizers/trijoint.py.151: total_norm: 11.162227 [I 2022-10-26 23:23:10] ...trap/engines/engine.py.224: train: epoch 35 | batch 1150/2304 [I 2022-10-26 23:23:10] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:41 [I 2022-10-26 23:23:10] ...trap/engines/engine.py.229: process: 0.54630 | load: 0.00251 [I 2022-10-26 23:23:10] ...trap/engines/engine.py.230: loss: 0.22334 [I 2022-10-26 23:23:10] ...optimizers/trijoint.py.151: total_norm: 16.546043 [I 2022-10-26 23:23:15] ...trap/engines/engine.py.224: train: epoch 35 | batch 1160/2304 [I 2022-10-26 23:23:15] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:31 [I 2022-10-26 23:23:15] ...trap/engines/engine.py.229: process: 0.54295 | load: 0.00250 [I 2022-10-26 23:23:15] ...trap/engines/engine.py.230: loss: 0.15464 [I 2022-10-26 23:23:15] ...optimizers/trijoint.py.151: total_norm: 17.994738 [I 2022-10-26 23:23:21] ...trap/engines/engine.py.224: train: epoch 35 | batch 1170/2304 [I 2022-10-26 23:23:21] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:30 [I 2022-10-26 23:23:21] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00247 [I 2022-10-26 23:23:21] ...trap/engines/engine.py.230: loss: 0.15959 [I 2022-10-26 23:23:21] ...optimizers/trijoint.py.151: total_norm: 23.812433 [I 2022-10-26 23:23:26] ...trap/engines/engine.py.224: train: epoch 35 | batch 1180/2304 [I 2022-10-26 23:23:26] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:23 [I 2022-10-26 23:23:26] ...trap/engines/engine.py.229: process: 0.55346 | load: 0.00245 [I 2022-10-26 23:23:26] ...trap/engines/engine.py.230: loss: 0.13071 [I 2022-10-26 23:23:26] ...optimizers/trijoint.py.151: total_norm: 13.013100 [I 2022-10-26 23:23:32] ...trap/engines/engine.py.224: train: epoch 35 | batch 1190/2304 [I 2022-10-26 23:23:32] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:21 [I 2022-10-26 23:23:32] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00241 [I 2022-10-26 23:23:32] ...trap/engines/engine.py.230: loss: 0.20090 [I 2022-10-26 23:23:32] ...optimizers/trijoint.py.151: total_norm: 21.915905 [I 2022-10-26 23:23:38] ...trap/engines/engine.py.224: train: epoch 35 | batch 1200/2304 [I 2022-10-26 23:23:38] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:10:12 [I 2022-10-26 23:23:38] ...trap/engines/engine.py.229: process: 0.54973 | load: 0.00261 [I 2022-10-26 23:23:38] ...trap/engines/engine.py.230: loss: 0.11101 [I 2022-10-26 23:23:38] ...optimizers/trijoint.py.151: total_norm: 7.270982 [I 2022-10-26 23:23:43] ...trap/engines/engine.py.224: train: epoch 35 | batch 1210/2304 [I 2022-10-26 23:23:43] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:10:13 [I 2022-10-26 23:23:43] ...trap/engines/engine.py.229: process: 0.56445 | load: 0.00247 [I 2022-10-26 23:23:43] ...trap/engines/engine.py.230: loss: 0.14433 [I 2022-10-26 23:23:43] ...optimizers/trijoint.py.151: total_norm: 16.727388 [I 2022-10-26 23:23:49] ...trap/engines/engine.py.224: train: epoch 35 | batch 1220/2304 [I 2022-10-26 23:23:49] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:10:04 [I 2022-10-26 23:23:49] ...trap/engines/engine.py.229: process: 0.56447 | load: 0.00245 [I 2022-10-26 23:23:49] ...trap/engines/engine.py.230: loss: 0.23088 [I 2022-10-26 23:23:49] ...optimizers/trijoint.py.151: total_norm: 17.336178 [I 2022-10-26 23:23:54] ...trap/engines/engine.py.224: train: epoch 35 | batch 1230/2304 [I 2022-10-26 23:23:54] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:56 [I 2022-10-26 23:23:54] ...trap/engines/engine.py.229: process: 0.54417 | load: 0.00249 [I 2022-10-26 23:23:54] ...trap/engines/engine.py.230: loss: 0.12625 [I 2022-10-26 23:23:54] ...optimizers/trijoint.py.151: total_norm: 14.647552 [I 2022-10-26 23:24:00] ...trap/engines/engine.py.224: train: epoch 35 | batch 1240/2304 [I 2022-10-26 23:24:00] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:51 [I 2022-10-26 23:24:00] ...trap/engines/engine.py.229: process: 0.57794 | load: 0.00252 [I 2022-10-26 23:24:00] ...trap/engines/engine.py.230: loss: 0.19186 [I 2022-10-26 23:24:00] ...optimizers/trijoint.py.151: total_norm: 29.537413 [I 2022-10-26 23:24:06] ...trap/engines/engine.py.224: train: epoch 35 | batch 1250/2304 [I 2022-10-26 23:24:06] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:48 [I 2022-10-26 23:24:06] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00244 [I 2022-10-26 23:24:06] ...trap/engines/engine.py.230: loss: 0.12390 [I 2022-10-26 23:24:06] ...optimizers/trijoint.py.151: total_norm: 11.947347 [I 2022-10-26 23:24:11] ...trap/engines/engine.py.224: train: epoch 35 | batch 1260/2304 [I 2022-10-26 23:24:11] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:40 [I 2022-10-26 23:24:11] ...trap/engines/engine.py.229: process: 0.56393 | load: 0.00243 [I 2022-10-26 23:24:11] ...trap/engines/engine.py.230: loss: 0.15026 [I 2022-10-26 23:24:11] ...optimizers/trijoint.py.151: total_norm: 28.340691 [I 2022-10-26 23:24:17] ...trap/engines/engine.py.224: train: epoch 35 | batch 1270/2304 [I 2022-10-26 23:24:17] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:35 [I 2022-10-26 23:24:17] ...trap/engines/engine.py.229: process: 0.54987 | load: 0.00242 [I 2022-10-26 23:24:17] ...trap/engines/engine.py.230: loss: 0.22382 [I 2022-10-26 23:24:17] ...optimizers/trijoint.py.151: total_norm: 21.152729 [I 2022-10-26 23:24:22] ...trap/engines/engine.py.224: train: epoch 35 | batch 1280/2304 [I 2022-10-26 23:24:22] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:31 [I 2022-10-26 23:24:22] ...trap/engines/engine.py.229: process: 0.56273 | load: 0.00242 [I 2022-10-26 23:24:22] ...trap/engines/engine.py.230: loss: 0.11208 [I 2022-10-26 23:24:22] ...optimizers/trijoint.py.151: total_norm: 10.446206 [I 2022-10-26 23:24:28] ...trap/engines/engine.py.224: train: epoch 35 | batch 1290/2304 [I 2022-10-26 23:24:28] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:21 [I 2022-10-26 23:24:28] ...trap/engines/engine.py.229: process: 0.55943 | load: 0.00245 [I 2022-10-26 23:24:28] ...trap/engines/engine.py.230: loss: 0.19745 [I 2022-10-26 23:24:28] ...optimizers/trijoint.py.151: total_norm: 23.803537 [I 2022-10-26 23:24:33] ...trap/engines/engine.py.224: train: epoch 35 | batch 1300/2304 [I 2022-10-26 23:24:33] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:09:20 [I 2022-10-26 23:24:33] ...trap/engines/engine.py.229: process: 0.56579 | load: 0.00246 [I 2022-10-26 23:24:33] ...trap/engines/engine.py.230: loss: 0.20039 [I 2022-10-26 23:24:33] ...optimizers/trijoint.py.151: total_norm: 22.346912 [I 2022-10-26 23:24:39] ...trap/engines/engine.py.224: train: epoch 35 | batch 1310/2304 [I 2022-10-26 23:24:39] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:17 [I 2022-10-26 23:24:39] ...trap/engines/engine.py.229: process: 0.56624 | load: 0.00239 [I 2022-10-26 23:24:39] ...trap/engines/engine.py.230: loss: 0.13099 [I 2022-10-26 23:24:39] ...optimizers/trijoint.py.151: total_norm: 12.492270 [I 2022-10-26 23:24:45] ...trap/engines/engine.py.224: train: epoch 35 | batch 1320/2304 [I 2022-10-26 23:24:45] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:09:09 [I 2022-10-26 23:24:45] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00238 [I 2022-10-26 23:24:45] ...trap/engines/engine.py.230: loss: 0.17904 [I 2022-10-26 23:24:45] ...optimizers/trijoint.py.151: total_norm: 17.732868 [I 2022-10-26 23:24:50] ...trap/engines/engine.py.224: train: epoch 35 | batch 1330/2304 [I 2022-10-26 23:24:50] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:09:02 [I 2022-10-26 23:24:50] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00250 [I 2022-10-26 23:24:50] ...trap/engines/engine.py.230: loss: 0.16762 [I 2022-10-26 23:24:50] ...optimizers/trijoint.py.151: total_norm: 19.246550 [I 2022-10-26 23:24:56] ...trap/engines/engine.py.224: train: epoch 35 | batch 1340/2304 [I 2022-10-26 23:24:56] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:09:00 [I 2022-10-26 23:24:56] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00247 [I 2022-10-26 23:24:56] ...trap/engines/engine.py.230: loss: 0.11461 [I 2022-10-26 23:24:56] ...optimizers/trijoint.py.151: total_norm: 11.510565 [I 2022-10-26 23:25:01] ...trap/engines/engine.py.224: train: epoch 35 | batch 1350/2304 [I 2022-10-26 23:25:01] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:53 [I 2022-10-26 23:25:01] ...trap/engines/engine.py.229: process: 0.54873 | load: 0.00244 [I 2022-10-26 23:25:01] ...trap/engines/engine.py.230: loss: 0.23277 [I 2022-10-26 23:25:01] ...optimizers/trijoint.py.151: total_norm: 23.774746 [I 2022-10-26 23:25:07] ...trap/engines/engine.py.224: train: epoch 35 | batch 1360/2304 [I 2022-10-26 23:25:07] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:49 [I 2022-10-26 23:25:07] ...trap/engines/engine.py.229: process: 0.56983 | load: 0.00247 [I 2022-10-26 23:25:07] ...trap/engines/engine.py.230: loss: 0.18626 [I 2022-10-26 23:25:07] ...optimizers/trijoint.py.151: total_norm: 11.591755 [I 2022-10-26 23:25:13] ...trap/engines/engine.py.224: train: epoch 35 | batch 1370/2304 [I 2022-10-26 23:25:13] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:46 [I 2022-10-26 23:25:13] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00245 [I 2022-10-26 23:25:13] ...trap/engines/engine.py.230: loss: 0.12371 [I 2022-10-26 23:25:13] ...optimizers/trijoint.py.151: total_norm: 12.875627 [I 2022-10-26 23:25:18] ...trap/engines/engine.py.224: train: epoch 35 | batch 1380/2304 [I 2022-10-26 23:25:18] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:32 [I 2022-10-26 23:25:18] ...trap/engines/engine.py.229: process: 0.54745 | load: 0.00251 [I 2022-10-26 23:25:18] ...trap/engines/engine.py.230: loss: 0.15699 [I 2022-10-26 23:25:18] ...optimizers/trijoint.py.151: total_norm: 21.468401 [I 2022-10-26 23:25:24] ...trap/engines/engine.py.224: train: epoch 35 | batch 1390/2304 [I 2022-10-26 23:25:24] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:27 [I 2022-10-26 23:25:24] ...trap/engines/engine.py.229: process: 0.54812 | load: 0.00255 [I 2022-10-26 23:25:24] ...trap/engines/engine.py.230: loss: 0.18562 [I 2022-10-26 23:25:24] ...optimizers/trijoint.py.151: total_norm: 15.327124 [I 2022-10-26 23:25:29] ...trap/engines/engine.py.224: train: epoch 35 | batch 1400/2304 [I 2022-10-26 23:25:29] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:23 [I 2022-10-26 23:25:29] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00246 [I 2022-10-26 23:25:29] ...trap/engines/engine.py.230: loss: 0.14293 [I 2022-10-26 23:25:29] ...optimizers/trijoint.py.151: total_norm: 15.083688 [I 2022-10-26 23:25:35] ...trap/engines/engine.py.224: train: epoch 35 | batch 1410/2304 [I 2022-10-26 23:25:35] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:32 [I 2022-10-26 23:25:35] ...trap/engines/engine.py.229: process: 0.58265 | load: 0.00253 [I 2022-10-26 23:25:35] ...trap/engines/engine.py.230: loss: 0.20429 [I 2022-10-26 23:25:35] ...optimizers/trijoint.py.151: total_norm: 28.399416 [I 2022-10-26 23:25:41] ...trap/engines/engine.py.224: train: epoch 35 | batch 1420/2304 [I 2022-10-26 23:25:41] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:08:16 [I 2022-10-26 23:25:41] ...trap/engines/engine.py.229: process: 0.57983 | load: 0.00253 [I 2022-10-26 23:25:41] ...trap/engines/engine.py.230: loss: 0.23589 [I 2022-10-26 23:25:41] ...optimizers/trijoint.py.151: total_norm: 64.697289 [I 2022-10-26 23:25:47] ...trap/engines/engine.py.224: train: epoch 35 | batch 1430/2304 [I 2022-10-26 23:25:47] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:08:21 [I 2022-10-26 23:25:47] ...trap/engines/engine.py.229: process: 0.56856 | load: 0.00262 [I 2022-10-26 23:25:47] ...trap/engines/engine.py.230: loss: 0.14743 [I 2022-10-26 23:25:47] ...optimizers/trijoint.py.151: total_norm: 13.609234 [I 2022-10-26 23:25:52] ...trap/engines/engine.py.224: train: epoch 35 | batch 1440/2304 [I 2022-10-26 23:25:52] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:58 [I 2022-10-26 23:25:52] ...trap/engines/engine.py.229: process: 0.54644 | load: 0.00259 [I 2022-10-26 23:25:52] ...trap/engines/engine.py.230: loss: 0.23532 [I 2022-10-26 23:25:52] ...optimizers/trijoint.py.151: total_norm: 45.403442 [I 2022-10-26 23:25:58] ...trap/engines/engine.py.224: train: epoch 35 | batch 1450/2304 [I 2022-10-26 23:25:58] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:54 [I 2022-10-26 23:25:58] ...trap/engines/engine.py.229: process: 0.56414 | load: 0.00258 [I 2022-10-26 23:25:58] ...trap/engines/engine.py.230: loss: 0.20850 [I 2022-10-26 23:25:58] ...optimizers/trijoint.py.151: total_norm: 13.847814 [I 2022-10-26 23:26:03] ...trap/engines/engine.py.224: train: epoch 35 | batch 1460/2304 [I 2022-10-26 23:26:03] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:49 [I 2022-10-26 23:26:03] ...trap/engines/engine.py.229: process: 0.56407 | load: 0.00258 [I 2022-10-26 23:26:03] ...trap/engines/engine.py.230: loss: 0.18151 [I 2022-10-26 23:26:03] ...optimizers/trijoint.py.151: total_norm: 18.317587 [I 2022-10-26 23:26:09] ...trap/engines/engine.py.224: train: epoch 35 | batch 1470/2304 [I 2022-10-26 23:26:09] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:43 [I 2022-10-26 23:26:09] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00249 [I 2022-10-26 23:26:09] ...trap/engines/engine.py.230: loss: 0.18766 [I 2022-10-26 23:26:09] ...optimizers/trijoint.py.151: total_norm: 23.622940 [I 2022-10-26 23:26:14] ...trap/engines/engine.py.224: train: epoch 35 | batch 1480/2304 [I 2022-10-26 23:26:14] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:36 [I 2022-10-26 23:26:14] ...trap/engines/engine.py.229: process: 0.55395 | load: 0.00258 [I 2022-10-26 23:26:14] ...trap/engines/engine.py.230: loss: 0.16883 [I 2022-10-26 23:26:14] ...optimizers/trijoint.py.151: total_norm: 22.438717 [I 2022-10-26 23:26:20] ...trap/engines/engine.py.224: train: epoch 35 | batch 1490/2304 [I 2022-10-26 23:26:20] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:32 [I 2022-10-26 23:26:20] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00251 [I 2022-10-26 23:26:20] ...trap/engines/engine.py.230: loss: 0.17952 [I 2022-10-26 23:26:20] ...optimizers/trijoint.py.151: total_norm: 51.906567 [I 2022-10-26 23:26:26] ...trap/engines/engine.py.224: train: epoch 35 | batch 1500/2304 [I 2022-10-26 23:26:26] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:30 [I 2022-10-26 23:26:26] ...trap/engines/engine.py.229: process: 0.56331 | load: 0.00249 [I 2022-10-26 23:26:26] ...trap/engines/engine.py.230: loss: 0.16281 [I 2022-10-26 23:26:26] ...optimizers/trijoint.py.151: total_norm: 20.785130 [I 2022-10-26 23:26:31] ...trap/engines/engine.py.224: train: epoch 35 | batch 1510/2304 [I 2022-10-26 23:26:31] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:23 [I 2022-10-26 23:26:31] ...trap/engines/engine.py.229: process: 0.56219 | load: 0.00249 [I 2022-10-26 23:26:31] ...trap/engines/engine.py.230: loss: 0.21048 [I 2022-10-26 23:26:31] ...optimizers/trijoint.py.151: total_norm: 15.448550 [I 2022-10-26 23:26:37] ...trap/engines/engine.py.224: train: epoch 35 | batch 1520/2304 [I 2022-10-26 23:26:37] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:15 [I 2022-10-26 23:26:37] ...trap/engines/engine.py.229: process: 0.54896 | load: 0.00248 [I 2022-10-26 23:26:37] ...trap/engines/engine.py.230: loss: 0.12637 [I 2022-10-26 23:26:37] ...optimizers/trijoint.py.151: total_norm: 15.772615 [I 2022-10-26 23:26:42] ...trap/engines/engine.py.224: train: epoch 35 | batch 1530/2304 [I 2022-10-26 23:26:42] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:07:07 [I 2022-10-26 23:26:42] ...trap/engines/engine.py.229: process: 0.54823 | load: 0.00240 [I 2022-10-26 23:26:42] ...trap/engines/engine.py.230: loss: 0.17225 [I 2022-10-26 23:26:42] ...optimizers/trijoint.py.151: total_norm: 14.087206 [I 2022-10-26 23:26:48] ...trap/engines/engine.py.224: train: epoch 35 | batch 1540/2304 [I 2022-10-26 23:26:48] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:07:03 [I 2022-10-26 23:26:48] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00243 [I 2022-10-26 23:26:48] ...trap/engines/engine.py.230: loss: 0.15824 [I 2022-10-26 23:26:48] ...optimizers/trijoint.py.151: total_norm: 20.920496 [I 2022-10-26 23:26:53] ...trap/engines/engine.py.224: train: epoch 35 | batch 1550/2304 [I 2022-10-26 23:26:53] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:07:00 [I 2022-10-26 23:26:53] ...trap/engines/engine.py.229: process: 0.54766 | load: 0.00254 [I 2022-10-26 23:26:53] ...trap/engines/engine.py.230: loss: 0.14079 [I 2022-10-26 23:26:53] ...optimizers/trijoint.py.151: total_norm: 13.216947 [I 2022-10-26 23:26:59] ...trap/engines/engine.py.224: train: epoch 35 | batch 1560/2304 [I 2022-10-26 23:26:59] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:07:05 [I 2022-10-26 23:26:59] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00265 [I 2022-10-26 23:26:59] ...trap/engines/engine.py.230: loss: 0.16557 [I 2022-10-26 23:26:59] ...optimizers/trijoint.py.151: total_norm: 17.138817 [I 2022-10-26 23:27:05] ...trap/engines/engine.py.224: train: epoch 35 | batch 1570/2304 [I 2022-10-26 23:27:05] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:48 [I 2022-10-26 23:27:05] ...trap/engines/engine.py.229: process: 0.54689 | load: 0.00252 [I 2022-10-26 23:27:05] ...trap/engines/engine.py.230: loss: 0.16010 [I 2022-10-26 23:27:05] ...optimizers/trijoint.py.151: total_norm: 12.576152 [I 2022-10-26 23:27:10] ...trap/engines/engine.py.224: train: epoch 35 | batch 1580/2304 [I 2022-10-26 23:27:10] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:44 [I 2022-10-26 23:27:10] ...trap/engines/engine.py.229: process: 0.56738 | load: 0.00247 [I 2022-10-26 23:27:10] ...trap/engines/engine.py.230: loss: 0.29032 [I 2022-10-26 23:27:10] ...optimizers/trijoint.py.151: total_norm: 28.292227 [I 2022-10-26 23:27:16] ...trap/engines/engine.py.224: train: epoch 35 | batch 1590/2304 [I 2022-10-26 23:27:16] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:36 [I 2022-10-26 23:27:16] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00250 [I 2022-10-26 23:27:16] ...trap/engines/engine.py.230: loss: 0.26268 [I 2022-10-26 23:27:16] ...optimizers/trijoint.py.151: total_norm: 17.591772 [I 2022-10-26 23:27:21] ...trap/engines/engine.py.224: train: epoch 35 | batch 1600/2304 [I 2022-10-26 23:27:21] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:31 [I 2022-10-26 23:27:21] ...trap/engines/engine.py.229: process: 0.55594 | load: 0.00247 [I 2022-10-26 23:27:21] ...trap/engines/engine.py.230: loss: 0.17765 [I 2022-10-26 23:27:21] ...optimizers/trijoint.py.151: total_norm: 17.681438 [I 2022-10-26 23:27:27] ...trap/engines/engine.py.224: train: epoch 35 | batch 1610/2304 [I 2022-10-26 23:27:27] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:24 [I 2022-10-26 23:27:27] ...trap/engines/engine.py.229: process: 0.54398 | load: 0.00248 [I 2022-10-26 23:27:27] ...trap/engines/engine.py.230: loss: 0.13656 [I 2022-10-26 23:27:27] ...optimizers/trijoint.py.151: total_norm: 30.283087 [I 2022-10-26 23:27:33] ...trap/engines/engine.py.224: train: epoch 35 | batch 1620/2304 [I 2022-10-26 23:27:33] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:21 [I 2022-10-26 23:27:33] ...trap/engines/engine.py.229: process: 0.56126 | load: 0.00241 [I 2022-10-26 23:27:33] ...trap/engines/engine.py.230: loss: 0.14872 [I 2022-10-26 23:27:33] ...optimizers/trijoint.py.151: total_norm: 12.208647 [I 2022-10-26 23:27:38] ...trap/engines/engine.py.224: train: epoch 35 | batch 1630/2304 [I 2022-10-26 23:27:38] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:06:15 [I 2022-10-26 23:27:38] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00245 [I 2022-10-26 23:27:38] ...trap/engines/engine.py.230: loss: 0.14803 [I 2022-10-26 23:27:38] ...optimizers/trijoint.py.151: total_norm: 17.989456 [I 2022-10-26 23:27:44] ...trap/engines/engine.py.224: train: epoch 35 | batch 1640/2304 [I 2022-10-26 23:27:44] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:06:08 [I 2022-10-26 23:27:44] ...trap/engines/engine.py.229: process: 0.54865 | load: 0.00241 [I 2022-10-26 23:27:44] ...trap/engines/engine.py.230: loss: 0.20030 [I 2022-10-26 23:27:44] ...optimizers/trijoint.py.151: total_norm: 10.517904 [I 2022-10-26 23:27:49] ...trap/engines/engine.py.224: train: epoch 35 | batch 1650/2304 [I 2022-10-26 23:27:49] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:06:03 [I 2022-10-26 23:27:49] ...trap/engines/engine.py.229: process: 0.56624 | load: 0.00243 [I 2022-10-26 23:27:49] ...trap/engines/engine.py.230: loss: 0.16827 [I 2022-10-26 23:27:49] ...optimizers/trijoint.py.151: total_norm: 15.117279 [I 2022-10-26 23:27:55] ...trap/engines/engine.py.224: train: epoch 35 | batch 1660/2304 [I 2022-10-26 23:27:55] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:06:01 [I 2022-10-26 23:27:55] ...trap/engines/engine.py.229: process: 0.57579 | load: 0.00243 [I 2022-10-26 23:27:55] ...trap/engines/engine.py.230: loss: 0.19771 [I 2022-10-26 23:27:55] ...optimizers/trijoint.py.151: total_norm: 15.803347 [I 2022-10-26 23:28:01] ...trap/engines/engine.py.224: train: epoch 35 | batch 1670/2304 [I 2022-10-26 23:28:01] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:55 [I 2022-10-26 23:28:01] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00250 [I 2022-10-26 23:28:01] ...trap/engines/engine.py.230: loss: 0.23002 [I 2022-10-26 23:28:01] ...optimizers/trijoint.py.151: total_norm: 18.984104 [I 2022-10-26 23:28:06] ...trap/engines/engine.py.224: train: epoch 35 | batch 1680/2304 [I 2022-10-26 23:28:06] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:50 [I 2022-10-26 23:28:06] ...trap/engines/engine.py.229: process: 0.54834 | load: 0.00248 [I 2022-10-26 23:28:06] ...trap/engines/engine.py.230: loss: 0.18572 [I 2022-10-26 23:28:06] ...optimizers/trijoint.py.151: total_norm: 23.483141 [I 2022-10-26 23:28:12] ...trap/engines/engine.py.224: train: epoch 35 | batch 1690/2304 [I 2022-10-26 23:28:12] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:40 [I 2022-10-26 23:28:12] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00248 [I 2022-10-26 23:28:12] ...trap/engines/engine.py.230: loss: 0.14942 [I 2022-10-26 23:28:12] ...optimizers/trijoint.py.151: total_norm: 36.530285 [I 2022-10-26 23:28:17] ...trap/engines/engine.py.224: train: epoch 35 | batch 1700/2304 [I 2022-10-26 23:28:17] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:36 [I 2022-10-26 23:28:17] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00249 [I 2022-10-26 23:28:17] ...trap/engines/engine.py.230: loss: 0.22802 [I 2022-10-26 23:28:17] ...optimizers/trijoint.py.151: total_norm: 28.452255 [I 2022-10-26 23:28:23] ...trap/engines/engine.py.224: train: epoch 35 | batch 1710/2304 [I 2022-10-26 23:28:23] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:32 [I 2022-10-26 23:28:23] ...trap/engines/engine.py.229: process: 0.56805 | load: 0.00254 [I 2022-10-26 23:28:23] ...trap/engines/engine.py.230: loss: 0.13613 [I 2022-10-26 23:28:23] ...optimizers/trijoint.py.151: total_norm: 17.465443 [I 2022-10-26 23:28:29] ...trap/engines/engine.py.224: train: epoch 35 | batch 1720/2304 [I 2022-10-26 23:28:29] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:29 [I 2022-10-26 23:28:29] ...trap/engines/engine.py.229: process: 0.56959 | load: 0.00240 [I 2022-10-26 23:28:29] ...trap/engines/engine.py.230: loss: 0.15035 [I 2022-10-26 23:28:29] ...optimizers/trijoint.py.151: total_norm: 12.724325 [I 2022-10-26 23:28:34] ...trap/engines/engine.py.224: train: epoch 35 | batch 1730/2304 [I 2022-10-26 23:28:34] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:20 [I 2022-10-26 23:28:34] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00241 [I 2022-10-26 23:28:34] ...trap/engines/engine.py.230: loss: 0.22282 [I 2022-10-26 23:28:34] ...optimizers/trijoint.py.151: total_norm: 23.856522 [I 2022-10-26 23:28:40] ...trap/engines/engine.py.224: train: epoch 35 | batch 1740/2304 [I 2022-10-26 23:28:40] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:05:12 [I 2022-10-26 23:28:40] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00243 [I 2022-10-26 23:28:40] ...trap/engines/engine.py.230: loss: 0.19680 [I 2022-10-26 23:28:40] ...optimizers/trijoint.py.151: total_norm: 17.742189 [I 2022-10-26 23:28:45] ...trap/engines/engine.py.224: train: epoch 35 | batch 1750/2304 [I 2022-10-26 23:28:45] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:05:07 [I 2022-10-26 23:28:45] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00249 [I 2022-10-26 23:28:45] ...trap/engines/engine.py.230: loss: 0.16973 [I 2022-10-26 23:28:45] ...optimizers/trijoint.py.151: total_norm: 13.571661 [I 2022-10-26 23:28:51] ...trap/engines/engine.py.224: train: epoch 35 | batch 1760/2304 [I 2022-10-26 23:28:51] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:05:03 [I 2022-10-26 23:28:51] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00242 [I 2022-10-26 23:28:51] ...trap/engines/engine.py.230: loss: 0.10970 [I 2022-10-26 23:28:51] ...optimizers/trijoint.py.151: total_norm: 10.767769 [I 2022-10-26 23:28:57] ...trap/engines/engine.py.224: train: epoch 35 | batch 1770/2304 [I 2022-10-26 23:28:57] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:56 [I 2022-10-26 23:28:57] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00243 [I 2022-10-26 23:28:57] ...trap/engines/engine.py.230: loss: 0.21051 [I 2022-10-26 23:28:57] ...optimizers/trijoint.py.151: total_norm: 23.557102 [I 2022-10-26 23:29:02] ...trap/engines/engine.py.224: train: epoch 35 | batch 1780/2304 [I 2022-10-26 23:29:02] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:05:01 [I 2022-10-26 23:29:02] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00242 [I 2022-10-26 23:29:02] ...trap/engines/engine.py.230: loss: 0.22804 [I 2022-10-26 23:29:02] ...optimizers/trijoint.py.151: total_norm: 24.996708 [I 2022-10-26 23:29:08] ...trap/engines/engine.py.224: train: epoch 35 | batch 1790/2304 [I 2022-10-26 23:29:08] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:45 [I 2022-10-26 23:29:08] ...trap/engines/engine.py.229: process: 0.54948 | load: 0.00240 [I 2022-10-26 23:29:08] ...trap/engines/engine.py.230: loss: 0.23055 [I 2022-10-26 23:29:08] ...optimizers/trijoint.py.151: total_norm: 15.794863 [I 2022-10-26 23:29:13] ...trap/engines/engine.py.224: train: epoch 35 | batch 1800/2304 [I 2022-10-26 23:29:13] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:39 [I 2022-10-26 23:29:13] ...trap/engines/engine.py.229: process: 0.53798 | load: 0.00240 [I 2022-10-26 23:29:13] ...trap/engines/engine.py.230: loss: 0.14250 [I 2022-10-26 23:29:13] ...optimizers/trijoint.py.151: total_norm: 13.048799 [I 2022-10-26 23:29:19] ...trap/engines/engine.py.224: train: epoch 35 | batch 1810/2304 [I 2022-10-26 23:29:19] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:36 [I 2022-10-26 23:29:19] ...trap/engines/engine.py.229: process: 0.55571 | load: 0.00243 [I 2022-10-26 23:29:19] ...trap/engines/engine.py.230: loss: 0.17721 [I 2022-10-26 23:29:19] ...optimizers/trijoint.py.151: total_norm: 18.763639 [I 2022-10-26 23:29:25] ...trap/engines/engine.py.224: train: epoch 35 | batch 1820/2304 [I 2022-10-26 23:29:25] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:31 [I 2022-10-26 23:29:25] ...trap/engines/engine.py.229: process: 0.56689 | load: 0.00255 [I 2022-10-26 23:29:25] ...trap/engines/engine.py.230: loss: 0.16124 [I 2022-10-26 23:29:25] ...optimizers/trijoint.py.151: total_norm: 14.808255 [I 2022-10-26 23:29:30] ...trap/engines/engine.py.224: train: epoch 35 | batch 1830/2304 [I 2022-10-26 23:29:30] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:25 [I 2022-10-26 23:29:30] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00252 [I 2022-10-26 23:29:30] ...trap/engines/engine.py.230: loss: 0.14061 [I 2022-10-26 23:29:30] ...optimizers/trijoint.py.151: total_norm: 12.032261 [I 2022-10-26 23:29:36] ...trap/engines/engine.py.224: train: epoch 35 | batch 1840/2304 [I 2022-10-26 23:29:36] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:19 [I 2022-10-26 23:29:36] ...trap/engines/engine.py.229: process: 0.56746 | load: 0.00247 [I 2022-10-26 23:29:36] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-26 23:29:36] ...optimizers/trijoint.py.151: total_norm: 15.643808 [I 2022-10-26 23:29:41] ...trap/engines/engine.py.224: train: epoch 35 | batch 1850/2304 [I 2022-10-26 23:29:41] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:04:10 [I 2022-10-26 23:29:41] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00242 [I 2022-10-26 23:29:41] ...trap/engines/engine.py.230: loss: 0.16608 [I 2022-10-26 23:29:41] ...optimizers/trijoint.py.151: total_norm: 15.821142 [I 2022-10-26 23:29:47] ...trap/engines/engine.py.224: train: epoch 35 | batch 1860/2304 [I 2022-10-26 23:29:47] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:04:08 [I 2022-10-26 23:29:47] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00242 [I 2022-10-26 23:29:47] ...trap/engines/engine.py.230: loss: 0.09708 [I 2022-10-26 23:29:47] ...optimizers/trijoint.py.151: total_norm: 15.282271 [I 2022-10-26 23:29:53] ...trap/engines/engine.py.224: train: epoch 35 | batch 1870/2304 [I 2022-10-26 23:29:53] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:04:02 [I 2022-10-26 23:29:53] ...trap/engines/engine.py.229: process: 0.56572 | load: 0.00244 [I 2022-10-26 23:29:53] ...trap/engines/engine.py.230: loss: 0.14989 [I 2022-10-26 23:29:53] ...optimizers/trijoint.py.151: total_norm: 17.548960 [I 2022-10-26 23:29:58] ...trap/engines/engine.py.224: train: epoch 35 | batch 1880/2304 [I 2022-10-26 23:29:58] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:55 [I 2022-10-26 23:29:58] ...trap/engines/engine.py.229: process: 0.54770 | load: 0.00240 [I 2022-10-26 23:29:58] ...trap/engines/engine.py.230: loss: 0.12658 [I 2022-10-26 23:29:58] ...optimizers/trijoint.py.151: total_norm: 29.945414 [I 2022-10-26 23:30:04] ...trap/engines/engine.py.224: train: epoch 35 | batch 1890/2304 [I 2022-10-26 23:30:04] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:50 [I 2022-10-26 23:30:04] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00246 [I 2022-10-26 23:30:04] ...trap/engines/engine.py.230: loss: 0.28845 [I 2022-10-26 23:30:04] ...optimizers/trijoint.py.151: total_norm: 16.957602 [I 2022-10-26 23:30:09] ...trap/engines/engine.py.224: train: epoch 35 | batch 1900/2304 [I 2022-10-26 23:30:09] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:45 [I 2022-10-26 23:30:09] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00245 [I 2022-10-26 23:30:09] ...trap/engines/engine.py.230: loss: 0.29908 [I 2022-10-26 23:30:09] ...optimizers/trijoint.py.151: total_norm: 27.237381 [I 2022-10-26 23:30:15] ...trap/engines/engine.py.224: train: epoch 35 | batch 1910/2304 [I 2022-10-26 23:30:15] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:44 [I 2022-10-26 23:30:15] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00249 [I 2022-10-26 23:30:15] ...trap/engines/engine.py.230: loss: 0.19732 [I 2022-10-26 23:30:15] ...optimizers/trijoint.py.151: total_norm: 19.184078 [I 2022-10-26 23:30:21] ...trap/engines/engine.py.224: train: epoch 35 | batch 1920/2304 [I 2022-10-26 23:30:21] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:35 [I 2022-10-26 23:30:21] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00247 [I 2022-10-26 23:30:21] ...trap/engines/engine.py.230: loss: 0.20854 [I 2022-10-26 23:30:21] ...optimizers/trijoint.py.151: total_norm: 14.037815 [I 2022-10-26 23:30:26] ...trap/engines/engine.py.224: train: epoch 35 | batch 1930/2304 [I 2022-10-26 23:30:26] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:29 [I 2022-10-26 23:30:26] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00244 [I 2022-10-26 23:30:26] ...trap/engines/engine.py.230: loss: 0.17360 [I 2022-10-26 23:30:26] ...optimizers/trijoint.py.151: total_norm: 33.119827 [I 2022-10-26 23:30:32] ...trap/engines/engine.py.224: train: epoch 35 | batch 1940/2304 [I 2022-10-26 23:30:32] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:23 [I 2022-10-26 23:30:32] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00280 [I 2022-10-26 23:30:32] ...trap/engines/engine.py.230: loss: 0.16000 [I 2022-10-26 23:30:32] ...optimizers/trijoint.py.151: total_norm: 11.842197 [I 2022-10-26 23:30:37] ...trap/engines/engine.py.224: train: epoch 35 | batch 1950/2304 [I 2022-10-26 23:30:37] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:03:18 [I 2022-10-26 23:30:37] ...trap/engines/engine.py.229: process: 0.57267 | load: 0.00243 [I 2022-10-26 23:30:37] ...trap/engines/engine.py.230: loss: 0.27087 [I 2022-10-26 23:30:37] ...optimizers/trijoint.py.151: total_norm: 29.481565 [I 2022-10-26 23:30:43] ...trap/engines/engine.py.224: train: epoch 35 | batch 1960/2304 [I 2022-10-26 23:30:43] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:03:13 [I 2022-10-26 23:30:43] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00247 [I 2022-10-26 23:30:43] ...trap/engines/engine.py.230: loss: 0.16336 [I 2022-10-26 23:30:43] ...optimizers/trijoint.py.151: total_norm: 13.572823 [I 2022-10-26 23:30:49] ...trap/engines/engine.py.224: train: epoch 35 | batch 1970/2304 [I 2022-10-26 23:30:49] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:03:07 [I 2022-10-26 23:30:49] ...trap/engines/engine.py.229: process: 0.56420 | load: 0.00247 [I 2022-10-26 23:30:49] ...trap/engines/engine.py.230: loss: 0.19213 [I 2022-10-26 23:30:49] ...optimizers/trijoint.py.151: total_norm: 19.782604 [I 2022-10-26 23:30:54] ...trap/engines/engine.py.224: train: epoch 35 | batch 1980/2304 [I 2022-10-26 23:30:54] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:03:02 [I 2022-10-26 23:30:54] ...trap/engines/engine.py.229: process: 0.56539 | load: 0.00246 [I 2022-10-26 23:30:54] ...trap/engines/engine.py.230: loss: 0.14238 [I 2022-10-26 23:30:54] ...optimizers/trijoint.py.151: total_norm: 16.476912 [I 2022-10-26 23:31:00] ...trap/engines/engine.py.224: train: epoch 35 | batch 1990/2304 [I 2022-10-26 23:31:00] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:55 [I 2022-10-26 23:31:00] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00244 [I 2022-10-26 23:31:00] ...trap/engines/engine.py.230: loss: 0.12131 [I 2022-10-26 23:31:00] ...optimizers/trijoint.py.151: total_norm: 22.884190 [I 2022-10-26 23:31:05] ...trap/engines/engine.py.224: train: epoch 35 | batch 2000/2304 [I 2022-10-26 23:31:05] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:47 [I 2022-10-26 23:31:05] ...trap/engines/engine.py.229: process: 0.54737 | load: 0.00244 [I 2022-10-26 23:31:05] ...trap/engines/engine.py.230: loss: 0.18534 [I 2022-10-26 23:31:05] ...optimizers/trijoint.py.151: total_norm: 42.431965 [I 2022-10-26 23:31:11] ...trap/engines/engine.py.224: train: epoch 35 | batch 2010/2304 [I 2022-10-26 23:31:11] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:43 [I 2022-10-26 23:31:11] ...trap/engines/engine.py.229: process: 0.56114 | load: 0.00240 [I 2022-10-26 23:31:11] ...trap/engines/engine.py.230: loss: 0.11613 [I 2022-10-26 23:31:11] ...optimizers/trijoint.py.151: total_norm: 9.838066 [I 2022-10-26 23:31:17] ...trap/engines/engine.py.224: train: epoch 35 | batch 2020/2304 [I 2022-10-26 23:31:17] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:38 [I 2022-10-26 23:31:17] ...trap/engines/engine.py.229: process: 0.56336 | load: 0.00242 [I 2022-10-26 23:31:17] ...trap/engines/engine.py.230: loss: 0.16209 [I 2022-10-26 23:31:17] ...optimizers/trijoint.py.151: total_norm: 12.901439 [I 2022-10-26 23:31:22] ...trap/engines/engine.py.224: train: epoch 35 | batch 2030/2304 [I 2022-10-26 23:31:22] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:31 [I 2022-10-26 23:31:22] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00240 [I 2022-10-26 23:31:22] ...trap/engines/engine.py.230: loss: 0.18913 [I 2022-10-26 23:31:22] ...optimizers/trijoint.py.151: total_norm: 19.788034 [I 2022-10-26 23:31:28] ...trap/engines/engine.py.224: train: epoch 35 | batch 2040/2304 [I 2022-10-26 23:31:28] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:26 [I 2022-10-26 23:31:28] ...trap/engines/engine.py.229: process: 0.55299 | load: 0.00245 [I 2022-10-26 23:31:28] ...trap/engines/engine.py.230: loss: 0.16860 [I 2022-10-26 23:31:28] ...optimizers/trijoint.py.151: total_norm: 19.269968 [I 2022-10-26 23:31:33] ...trap/engines/engine.py.224: train: epoch 35 | batch 2050/2304 [I 2022-10-26 23:31:33] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:21 [I 2022-10-26 23:31:33] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00244 [I 2022-10-26 23:31:33] ...trap/engines/engine.py.230: loss: 0.18690 [I 2022-10-26 23:31:33] ...optimizers/trijoint.py.151: total_norm: 19.406576 [I 2022-10-26 23:31:39] ...trap/engines/engine.py.224: train: epoch 35 | batch 2060/2304 [I 2022-10-26 23:31:39] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:02:15 [I 2022-10-26 23:31:39] ...trap/engines/engine.py.229: process: 0.55474 | load: 0.00242 [I 2022-10-26 23:31:39] ...trap/engines/engine.py.230: loss: 0.18566 [I 2022-10-26 23:31:39] ...optimizers/trijoint.py.151: total_norm: 16.898676 [I 2022-10-26 23:31:45] ...trap/engines/engine.py.224: train: epoch 35 | batch 2070/2304 [I 2022-10-26 23:31:45] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:02:11 [I 2022-10-26 23:31:45] ...trap/engines/engine.py.229: process: 0.57183 | load: 0.00249 [I 2022-10-26 23:31:45] ...trap/engines/engine.py.230: loss: 0.21006 [I 2022-10-26 23:31:45] ...optimizers/trijoint.py.151: total_norm: 16.923250 [I 2022-10-26 23:31:50] ...trap/engines/engine.py.224: train: epoch 35 | batch 2080/2304 [I 2022-10-26 23:31:50] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:02:05 [I 2022-10-26 23:31:50] ...trap/engines/engine.py.229: process: 0.56792 | load: 0.00241 [I 2022-10-26 23:31:50] ...trap/engines/engine.py.230: loss: 0.15698 [I 2022-10-26 23:31:50] ...optimizers/trijoint.py.151: total_norm: 13.655272 [I 2022-10-26 23:31:56] ...trap/engines/engine.py.224: train: epoch 35 | batch 2090/2304 [I 2022-10-26 23:31:56] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:59 [I 2022-10-26 23:31:56] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00244 [I 2022-10-26 23:31:56] ...trap/engines/engine.py.230: loss: 0.26191 [I 2022-10-26 23:31:56] ...optimizers/trijoint.py.151: total_norm: 21.937857 [I 2022-10-26 23:32:01] ...trap/engines/engine.py.224: train: epoch 35 | batch 2100/2304 [I 2022-10-26 23:32:01] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:53 [I 2022-10-26 23:32:01] ...trap/engines/engine.py.229: process: 0.55451 | load: 0.00239 [I 2022-10-26 23:32:01] ...trap/engines/engine.py.230: loss: 0.12105 [I 2022-10-26 23:32:01] ...optimizers/trijoint.py.151: total_norm: 13.307090 [I 2022-10-26 23:32:24] ...trap/engines/engine.py.224: train: epoch 35 | batch 2110/2304 [I 2022-10-26 23:32:24] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:05:21 [I 2022-10-26 23:32:24] ...trap/engines/engine.py.229: process: 0.56695 | load: 0.00256 [I 2022-10-26 23:32:24] ...trap/engines/engine.py.230: loss: 0.21739 [I 2022-10-26 23:32:24] ...optimizers/trijoint.py.151: total_norm: 19.637857 [I 2022-10-26 23:32:29] ...trap/engines/engine.py.224: train: epoch 35 | batch 2120/2304 [I 2022-10-26 23:32:29] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:02:02 [I 2022-10-26 23:32:29] ...trap/engines/engine.py.229: process: 0.54537 | load: 0.00237 [I 2022-10-26 23:32:29] ...trap/engines/engine.py.230: loss: 0.20521 [I 2022-10-26 23:32:29] ...optimizers/trijoint.py.151: total_norm: 27.390839 [I 2022-10-26 23:32:35] ...trap/engines/engine.py.224: train: epoch 35 | batch 2130/2304 [I 2022-10-26 23:32:35] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:38 [I 2022-10-26 23:32:35] ...trap/engines/engine.py.229: process: 0.55035 | load: 0.00256 [I 2022-10-26 23:32:35] ...trap/engines/engine.py.230: loss: 0.15505 [I 2022-10-26 23:32:35] ...optimizers/trijoint.py.151: total_norm: 21.259199 [I 2022-10-26 23:32:40] ...trap/engines/engine.py.224: train: epoch 35 | batch 2140/2304 [I 2022-10-26 23:32:40] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:31 [I 2022-10-26 23:32:40] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00263 [I 2022-10-26 23:32:40] ...trap/engines/engine.py.230: loss: 0.21387 [I 2022-10-26 23:32:40] ...optimizers/trijoint.py.151: total_norm: 28.455444 [I 2022-10-26 23:32:46] ...trap/engines/engine.py.224: train: epoch 35 | batch 2150/2304 [I 2022-10-26 23:32:46] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:25 [I 2022-10-26 23:32:46] ...trap/engines/engine.py.229: process: 0.54899 | load: 0.00262 [I 2022-10-26 23:32:46] ...trap/engines/engine.py.230: loss: 0.20044 [I 2022-10-26 23:32:46] ...optimizers/trijoint.py.151: total_norm: 16.337868 [I 2022-10-26 23:32:51] ...trap/engines/engine.py.224: train: epoch 35 | batch 2160/2304 [I 2022-10-26 23:32:51] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:01:20 [I 2022-10-26 23:32:51] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00255 [I 2022-10-26 23:32:51] ...trap/engines/engine.py.230: loss: 0.17927 [I 2022-10-26 23:32:51] ...optimizers/trijoint.py.151: total_norm: 18.316830 [I 2022-10-26 23:32:57] ...trap/engines/engine.py.224: train: epoch 35 | batch 2170/2304 [I 2022-10-26 23:32:57] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:01:14 [I 2022-10-26 23:32:57] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00259 [I 2022-10-26 23:32:57] ...trap/engines/engine.py.230: loss: 0.13196 [I 2022-10-26 23:32:57] ...optimizers/trijoint.py.151: total_norm: 26.388447 [I 2022-10-26 23:33:03] ...trap/engines/engine.py.224: train: epoch 35 | batch 2180/2304 [I 2022-10-26 23:33:03] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:01:08 [I 2022-10-26 23:33:03] ...trap/engines/engine.py.229: process: 0.54946 | load: 0.00256 [I 2022-10-26 23:33:03] ...trap/engines/engine.py.230: loss: 0.25431 [I 2022-10-26 23:33:03] ...optimizers/trijoint.py.151: total_norm: 26.024202 [I 2022-10-26 23:33:08] ...trap/engines/engine.py.224: train: epoch 35 | batch 2190/2304 [I 2022-10-26 23:33:08] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:01:03 [I 2022-10-26 23:33:08] ...trap/engines/engine.py.229: process: 0.54593 | load: 0.00249 [I 2022-10-26 23:33:08] ...trap/engines/engine.py.230: loss: 0.29666 [I 2022-10-26 23:33:08] ...optimizers/trijoint.py.151: total_norm: 28.536917 [I 2022-10-26 23:33:14] ...trap/engines/engine.py.224: train: epoch 35 | batch 2200/2304 [I 2022-10-26 23:33:14] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:57 [I 2022-10-26 23:33:14] ...trap/engines/engine.py.229: process: 0.55576 | load: 0.00253 [I 2022-10-26 23:33:14] ...trap/engines/engine.py.230: loss: 0.11586 [I 2022-10-26 23:33:14] ...optimizers/trijoint.py.151: total_norm: 11.604992 [I 2022-10-26 23:33:19] ...trap/engines/engine.py.224: train: epoch 35 | batch 2210/2304 [I 2022-10-26 23:33:19] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:52 [I 2022-10-26 23:33:19] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00245 [I 2022-10-26 23:33:19] ...trap/engines/engine.py.230: loss: 0.10689 [I 2022-10-26 23:33:19] ...optimizers/trijoint.py.151: total_norm: 12.360133 [I 2022-10-26 23:33:25] ...trap/engines/engine.py.224: train: epoch 35 | batch 2220/2304 [I 2022-10-26 23:33:25] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:46 [I 2022-10-26 23:33:25] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00256 [I 2022-10-26 23:33:25] ...trap/engines/engine.py.230: loss: 0.11155 [I 2022-10-26 23:33:25] ...optimizers/trijoint.py.151: total_norm: 17.249743 [I 2022-10-26 23:33:30] ...trap/engines/engine.py.224: train: epoch 35 | batch 2230/2304 [I 2022-10-26 23:33:30] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:41 [I 2022-10-26 23:33:30] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00255 [I 2022-10-26 23:33:30] ...trap/engines/engine.py.230: loss: 0.20177 [I 2022-10-26 23:33:30] ...optimizers/trijoint.py.151: total_norm: 23.051947 [I 2022-10-26 23:33:36] ...trap/engines/engine.py.224: train: epoch 35 | batch 2240/2304 [I 2022-10-26 23:33:36] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:35 [I 2022-10-26 23:33:36] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00260 [I 2022-10-26 23:33:36] ...trap/engines/engine.py.230: loss: 0.16716 [I 2022-10-26 23:33:36] ...optimizers/trijoint.py.151: total_norm: 16.639196 [I 2022-10-26 23:33:41] ...trap/engines/engine.py.224: train: epoch 35 | batch 2250/2304 [I 2022-10-26 23:33:41] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:30 [I 2022-10-26 23:33:41] ...trap/engines/engine.py.229: process: 0.56186 | load: 0.00260 [I 2022-10-26 23:33:41] ...trap/engines/engine.py.230: loss: 0.17850 [I 2022-10-26 23:33:41] ...optimizers/trijoint.py.151: total_norm: 16.536882 [I 2022-10-26 23:33:47] ...trap/engines/engine.py.224: train: epoch 35 | batch 2260/2304 [I 2022-10-26 23:33:47] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:00:24 [I 2022-10-26 23:33:47] ...trap/engines/engine.py.229: process: 0.56521 | load: 0.00257 [I 2022-10-26 23:33:47] ...trap/engines/engine.py.230: loss: 0.15374 [I 2022-10-26 23:33:47] ...optimizers/trijoint.py.151: total_norm: 22.507912 [I 2022-10-26 23:33:53] ...trap/engines/engine.py.224: train: epoch 35 | batch 2270/2304 [I 2022-10-26 23:33:53] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:18 [I 2022-10-26 23:33:53] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00257 [I 2022-10-26 23:33:53] ...trap/engines/engine.py.230: loss: 0.13350 [I 2022-10-26 23:33:53] ...optimizers/trijoint.py.151: total_norm: 12.454589 [I 2022-10-26 23:33:58] ...trap/engines/engine.py.224: train: epoch 35 | batch 2280/2304 [I 2022-10-26 23:33:58] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:00:13 [I 2022-10-26 23:33:58] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00249 [I 2022-10-26 23:33:58] ...trap/engines/engine.py.230: loss: 0.15874 [I 2022-10-26 23:33:58] ...optimizers/trijoint.py.151: total_norm: 12.783130 [I 2022-10-26 23:34:04] ...trap/engines/engine.py.224: train: epoch 35 | batch 2290/2304 [I 2022-10-26 23:34:04] ...trap/engines/engine.py.228: elapsed: 0:22:00 | left: 0:00:07 [I 2022-10-26 23:34:04] ...trap/engines/engine.py.229: process: 0.55121 | load: 0.00253 [I 2022-10-26 23:34:04] ...trap/engines/engine.py.230: loss: 0.21523 [I 2022-10-26 23:34:04] ...optimizers/trijoint.py.151: total_norm: 21.377892 [I 2022-10-26 23:34:09] ...trap/engines/engine.py.224: train: epoch 35 | batch 2300/2304 [I 2022-10-26 23:34:09] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:00:02 [I 2022-10-26 23:34:09] ...trap/engines/engine.py.229: process: 0.55534 | load: 0.00016 [I 2022-10-26 23:34:09] ...trap/engines/engine.py.230: loss: 0.14394 [I 2022-10-26 23:34:09] ...optimizers/trijoint.py.151: total_norm: 15.376253 [I 2022-10-26 23:34:12] ...trap/engines/engine.py.224: train: epoch 35 | batch 2304/2304 [I 2022-10-26 23:34:12] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:00:00 [I 2022-10-26 23:34:12] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00015 [I 2022-10-26 23:34:12] ...trap/engines/engine.py.230: loss: 0.17127 [I 2022-10-26 23:34:12] ...optimizers/trijoint.py.151: total_norm: 22.233763 [S 2022-10-26 23:34:13] ...trap/engines/engine.py.129: train_epoch.epoch: 35 [S 2022-10-26 23:34:13] ...trap/engines/engine.py.129: train_epoch.loss: 0.17545555349350492 [I 2022-10-26 23:34:13] ...s/criterions/triplet.py.84: margin = 0.23000000000000015 [I 2022-10-26 23:34:13] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 35 [I 2022-10-26 23:34:13] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 23:34:15] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 23:34:15] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 23:34:15] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 23:34:16] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 23:34:17] ...trap/engines/engine.py.321: eval: epoch 35 | batch 0/510 [I 2022-10-26 23:34:17] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:28:19 [I 2022-10-26 23:34:17] ...trap/engines/engine.py.326: process: 3.33298 | load: 2.87652 [I 2022-10-26 23:34:20] ...trap/engines/engine.py.321: eval: epoch 35 | batch 10/510 [I 2022-10-26 23:34:20] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:29 [I 2022-10-26 23:34:20] ...trap/engines/engine.py.326: process: 0.35289 | load: 0.00017 [I 2022-10-26 23:34:23] ...trap/engines/engine.py.321: eval: epoch 35 | batch 20/510 [I 2022-10-26 23:34:23] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:02:58 [I 2022-10-26 23:34:23] ...trap/engines/engine.py.326: process: 0.31715 | load: 0.00016 [I 2022-10-26 23:34:27] ...trap/engines/engine.py.321: eval: epoch 35 | batch 30/510 [I 2022-10-26 23:34:27] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:43 [I 2022-10-26 23:34:27] ...trap/engines/engine.py.326: process: 0.32359 | load: 0.00017 [I 2022-10-26 23:34:30] ...trap/engines/engine.py.321: eval: epoch 35 | batch 40/510 [I 2022-10-26 23:34:30] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:37 [I 2022-10-26 23:34:30] ...trap/engines/engine.py.326: process: 0.34013 | load: 0.00016 [I 2022-10-26 23:34:33] ...trap/engines/engine.py.321: eval: epoch 35 | batch 50/510 [I 2022-10-26 23:34:33] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:33 [I 2022-10-26 23:34:33] ...trap/engines/engine.py.326: process: 0.33906 | load: 0.00019 [I 2022-10-26 23:34:37] ...trap/engines/engine.py.321: eval: epoch 35 | batch 60/510 [I 2022-10-26 23:34:37] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:31 [I 2022-10-26 23:34:37] ...trap/engines/engine.py.326: process: 0.32966 | load: 0.00017 [I 2022-10-26 23:34:40] ...trap/engines/engine.py.321: eval: epoch 35 | batch 70/510 [I 2022-10-26 23:34:40] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-26 23:34:40] ...trap/engines/engine.py.326: process: 0.33108 | load: 0.00020 [I 2022-10-26 23:34:44] ...trap/engines/engine.py.321: eval: epoch 35 | batch 80/510 [I 2022-10-26 23:34:44] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:25 [I 2022-10-26 23:34:44] ...trap/engines/engine.py.326: process: 0.33431 | load: 0.00024 [I 2022-10-26 23:34:47] ...trap/engines/engine.py.321: eval: epoch 35 | batch 90/510 [I 2022-10-26 23:34:47] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:24 [I 2022-10-26 23:34:47] ...trap/engines/engine.py.326: process: 0.35692 | load: 0.00015 [I 2022-10-26 23:34:50] ...trap/engines/engine.py.321: eval: epoch 35 | batch 100/510 [I 2022-10-26 23:34:50] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-26 23:34:50] ...trap/engines/engine.py.326: process: 0.29255 | load: 0.00016 [I 2022-10-26 23:34:53] ...trap/engines/engine.py.321: eval: epoch 35 | batch 110/510 [I 2022-10-26 23:34:53] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-26 23:34:53] ...trap/engines/engine.py.326: process: 0.29552 | load: 0.00016 [I 2022-10-26 23:34:56] ...trap/engines/engine.py.321: eval: epoch 35 | batch 120/510 [I 2022-10-26 23:34:56] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:51 [I 2022-10-26 23:34:56] ...trap/engines/engine.py.326: process: 0.28812 | load: 0.00016 [I 2022-10-26 23:34:59] ...trap/engines/engine.py.321: eval: epoch 35 | batch 130/510 [I 2022-10-26 23:34:59] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:48 [I 2022-10-26 23:34:59] ...trap/engines/engine.py.326: process: 0.28997 | load: 0.00017 [I 2022-10-26 23:35:02] ...trap/engines/engine.py.321: eval: epoch 35 | batch 140/510 [I 2022-10-26 23:35:02] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-26 23:35:02] ...trap/engines/engine.py.326: process: 0.27827 | load: 0.00023 [I 2022-10-26 23:35:05] ...trap/engines/engine.py.321: eval: epoch 35 | batch 150/510 [I 2022-10-26 23:35:05] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-26 23:35:05] ...trap/engines/engine.py.326: process: 0.28591 | load: 0.00015 [I 2022-10-26 23:35:08] ...trap/engines/engine.py.321: eval: epoch 35 | batch 160/510 [I 2022-10-26 23:35:08] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-26 23:35:08] ...trap/engines/engine.py.326: process: 0.28764 | load: 0.00033 [I 2022-10-26 23:35:11] ...trap/engines/engine.py.321: eval: epoch 35 | batch 170/510 [I 2022-10-26 23:35:11] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-26 23:35:11] ...trap/engines/engine.py.326: process: 0.30425 | load: 0.00015 [I 2022-10-26 23:35:13] ...trap/engines/engine.py.321: eval: epoch 35 | batch 180/510 [I 2022-10-26 23:35:13] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-26 23:35:13] ...trap/engines/engine.py.326: process: 0.28754 | load: 0.00015 [I 2022-10-26 23:35:16] ...trap/engines/engine.py.321: eval: epoch 35 | batch 190/510 [I 2022-10-26 23:35:16] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-26 23:35:16] ...trap/engines/engine.py.326: process: 0.28135 | load: 0.00038 [I 2022-10-26 23:35:19] ...trap/engines/engine.py.321: eval: epoch 35 | batch 200/510 [I 2022-10-26 23:35:19] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-26 23:35:19] ...trap/engines/engine.py.326: process: 0.28512 | load: 0.00016 [I 2022-10-26 23:35:22] ...trap/engines/engine.py.321: eval: epoch 35 | batch 210/510 [I 2022-10-26 23:35:22] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-26 23:35:22] ...trap/engines/engine.py.326: process: 0.29078 | load: 0.00016 [I 2022-10-26 23:35:25] ...trap/engines/engine.py.321: eval: epoch 35 | batch 220/510 [I 2022-10-26 23:35:25] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:25 [I 2022-10-26 23:35:25] ...trap/engines/engine.py.326: process: 0.28845 | load: 0.00024 [I 2022-10-26 23:35:28] ...trap/engines/engine.py.321: eval: epoch 35 | batch 230/510 [I 2022-10-26 23:35:28] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 23:35:28] ...trap/engines/engine.py.326: process: 0.29251 | load: 0.00015 [I 2022-10-26 23:35:31] ...trap/engines/engine.py.321: eval: epoch 35 | batch 240/510 [I 2022-10-26 23:35:31] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-26 23:35:31] ...trap/engines/engine.py.326: process: 0.28873 | load: 0.00015 [I 2022-10-26 23:35:34] ...trap/engines/engine.py.321: eval: epoch 35 | batch 250/510 [I 2022-10-26 23:35:34] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:16 [I 2022-10-26 23:35:34] ...trap/engines/engine.py.326: process: 0.28846 | load: 0.00021 [I 2022-10-26 23:35:37] ...trap/engines/engine.py.321: eval: epoch 35 | batch 260/510 [I 2022-10-26 23:35:37] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-26 23:35:37] ...trap/engines/engine.py.326: process: 0.28136 | load: 0.00021 [I 2022-10-26 23:35:40] ...trap/engines/engine.py.321: eval: epoch 35 | batch 270/510 [I 2022-10-26 23:35:40] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-26 23:35:40] ...trap/engines/engine.py.326: process: 0.28389 | load: 0.00016 [I 2022-10-26 23:35:42] ...trap/engines/engine.py.321: eval: epoch 35 | batch 280/510 [I 2022-10-26 23:35:42] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:05 [I 2022-10-26 23:35:42] ...trap/engines/engine.py.326: process: 0.28530 | load: 0.00015 [I 2022-10-26 23:35:45] ...trap/engines/engine.py.321: eval: epoch 35 | batch 290/510 [I 2022-10-26 23:35:45] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-26 23:35:45] ...trap/engines/engine.py.326: process: 0.28858 | load: 0.00018 [I 2022-10-26 23:35:48] ...trap/engines/engine.py.321: eval: epoch 35 | batch 300/510 [I 2022-10-26 23:35:48] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-26 23:35:48] ...trap/engines/engine.py.326: process: 0.28588 | load: 0.00016 [I 2022-10-26 23:35:51] ...trap/engines/engine.py.321: eval: epoch 35 | batch 310/510 [I 2022-10-26 23:35:51] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-26 23:35:51] ...trap/engines/engine.py.326: process: 0.29177 | load: 0.00015 [I 2022-10-26 23:35:54] ...trap/engines/engine.py.321: eval: epoch 35 | batch 320/510 [I 2022-10-26 23:35:54] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-26 23:35:54] ...trap/engines/engine.py.326: process: 0.28339 | load: 0.00027 [I 2022-10-26 23:35:57] ...trap/engines/engine.py.321: eval: epoch 35 | batch 330/510 [I 2022-10-26 23:35:57] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:52 [I 2022-10-26 23:35:57] ...trap/engines/engine.py.326: process: 0.28657 | load: 0.00015 [I 2022-10-26 23:36:00] ...trap/engines/engine.py.321: eval: epoch 35 | batch 340/510 [I 2022-10-26 23:36:00] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:49 [I 2022-10-26 23:36:00] ...trap/engines/engine.py.326: process: 0.28215 | load: 0.00023 [I 2022-10-26 23:36:03] ...trap/engines/engine.py.321: eval: epoch 35 | batch 350/510 [I 2022-10-26 23:36:03] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-26 23:36:03] ...trap/engines/engine.py.326: process: 0.28884 | load: 0.00015 [I 2022-10-26 23:36:06] ...trap/engines/engine.py.321: eval: epoch 35 | batch 360/510 [I 2022-10-26 23:36:06] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-26 23:36:06] ...trap/engines/engine.py.326: process: 0.28509 | load: 0.00015 [I 2022-10-26 23:36:08] ...trap/engines/engine.py.321: eval: epoch 35 | batch 370/510 [I 2022-10-26 23:36:08] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-26 23:36:08] ...trap/engines/engine.py.326: process: 0.28850 | load: 0.00022 [I 2022-10-26 23:36:11] ...trap/engines/engine.py.321: eval: epoch 35 | batch 380/510 [I 2022-10-26 23:36:11] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-26 23:36:11] ...trap/engines/engine.py.326: process: 0.28236 | load: 0.00015 [I 2022-10-26 23:36:14] ...trap/engines/engine.py.321: eval: epoch 35 | batch 390/510 [I 2022-10-26 23:36:14] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-26 23:36:14] ...trap/engines/engine.py.326: process: 0.29110 | load: 0.00019 [I 2022-10-26 23:36:17] ...trap/engines/engine.py.321: eval: epoch 35 | batch 400/510 [I 2022-10-26 23:36:17] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:33 [I 2022-10-26 23:36:17] ...trap/engines/engine.py.326: process: 0.28808 | load: 0.00015 [I 2022-10-26 23:36:20] ...trap/engines/engine.py.321: eval: epoch 35 | batch 410/510 [I 2022-10-26 23:36:20] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-26 23:36:20] ...trap/engines/engine.py.326: process: 0.29010 | load: 0.00015 [I 2022-10-26 23:36:23] ...trap/engines/engine.py.321: eval: epoch 35 | batch 420/510 [I 2022-10-26 23:36:23] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-26 23:36:23] ...trap/engines/engine.py.326: process: 0.28553 | load: 0.00015 [I 2022-10-26 23:36:26] ...trap/engines/engine.py.321: eval: epoch 35 | batch 430/510 [I 2022-10-26 23:36:26] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-26 23:36:26] ...trap/engines/engine.py.326: process: 0.28977 | load: 0.00015 [I 2022-10-26 23:36:29] ...trap/engines/engine.py.321: eval: epoch 35 | batch 440/510 [I 2022-10-26 23:36:29] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-26 23:36:29] ...trap/engines/engine.py.326: process: 0.28449 | load: 0.00015 [I 2022-10-26 23:36:32] ...trap/engines/engine.py.321: eval: epoch 35 | batch 450/510 [I 2022-10-26 23:36:32] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-26 23:36:32] ...trap/engines/engine.py.326: process: 0.28739 | load: 0.00017 [I 2022-10-26 23:36:34] ...trap/engines/engine.py.321: eval: epoch 35 | batch 460/510 [I 2022-10-26 23:36:34] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-26 23:36:34] ...trap/engines/engine.py.326: process: 0.29293 | load: 0.00016 [I 2022-10-26 23:36:37] ...trap/engines/engine.py.321: eval: epoch 35 | batch 470/510 [I 2022-10-26 23:36:37] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-26 23:36:37] ...trap/engines/engine.py.326: process: 0.28530 | load: 0.00018 [I 2022-10-26 23:36:40] ...trap/engines/engine.py.321: eval: epoch 35 | batch 480/510 [I 2022-10-26 23:36:40] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-26 23:36:40] ...trap/engines/engine.py.326: process: 0.29387 | load: 0.00020 [I 2022-10-26 23:36:43] ...trap/engines/engine.py.321: eval: epoch 35 | batch 490/510 [I 2022-10-26 23:36:43] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-26 23:36:43] ...trap/engines/engine.py.326: process: 0.28553 | load: 0.00021 [I 2022-10-26 23:36:46] ...trap/engines/engine.py.321: eval: epoch 35 | batch 500/510 [I 2022-10-26 23:36:46] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-26 23:36:46] ...trap/engines/engine.py.326: process: 0.28382 | load: 0.00016 [I 2022-10-26 23:36:49] ...trap/engines/engine.py.321: eval: epoch 35 | batch 510/510 [I 2022-10-26 23:36:49] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-26 23:36:49] ...trap/engines/engine.py.326: process: 0.28796 | load: 0.00018 [S 2022-10-26 23:36:50] ...trap/engines/engine.py.132: eval_epoch.epoch: 35 [I 2022-10-26 23:36:50] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.676700 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.886200 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.926200 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.661300 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.885100 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.922000 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012248 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008920 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007600 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014927 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008561 [S 2022-10-26 23:36:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009176 [I 2022-10-26 23:36:55] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-26 23:36:55] ...trap/engines/engine.py.470: Saving model... [W 2022-10-26 23:36:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 23:36:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 23:36:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 23:36:56] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 23:36:59] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-26 23:37:05] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-26 23:37:05] ...trap/engines/engine.py.163: Training model on trainset for epoch 36 [I 2022-10-26 23:37:05] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-26 23:37:08] ...trap/engines/engine.py.224: train: epoch 36 | batch 0/2304 [I 2022-10-26 23:37:08] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:35 [I 2022-10-26 23:37:08] ...trap/engines/engine.py.229: process: 2.72386 | load: 2.08632 [I 2022-10-26 23:37:08] ...trap/engines/engine.py.230: loss: 0.12346 [I 2022-10-26 23:37:08] ...optimizers/trijoint.py.151: total_norm: 19.564634 [I 2022-10-26 23:37:14] ...trap/engines/engine.py.224: train: epoch 36 | batch 10/2304 [I 2022-10-26 23:37:14] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:18 [I 2022-10-26 23:37:14] ...trap/engines/engine.py.229: process: 0.56564 | load: 0.00357 [I 2022-10-26 23:37:14] ...trap/engines/engine.py.230: loss: 0.21228 [I 2022-10-26 23:37:14] ...optimizers/trijoint.py.151: total_norm: 11.763556 [I 2022-10-26 23:37:19] ...trap/engines/engine.py.224: train: epoch 36 | batch 20/2304 [I 2022-10-26 23:37:19] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:08 [I 2022-10-26 23:37:19] ...trap/engines/engine.py.229: process: 0.55635 | load: 0.00365 [I 2022-10-26 23:37:19] ...trap/engines/engine.py.230: loss: 0.16476 [I 2022-10-26 23:37:19] ...optimizers/trijoint.py.151: total_norm: 10.352063 [I 2022-10-26 23:37:25] ...trap/engines/engine.py.224: train: epoch 36 | batch 30/2304 [I 2022-10-26 23:37:25] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:11 [I 2022-10-26 23:37:25] ...trap/engines/engine.py.229: process: 0.56160 | load: 0.00352 [I 2022-10-26 23:37:25] ...trap/engines/engine.py.230: loss: 0.19129 [I 2022-10-26 23:37:25] ...optimizers/trijoint.py.151: total_norm: 21.510126 [I 2022-10-26 23:37:30] ...trap/engines/engine.py.224: train: epoch 36 | batch 40/2304 [I 2022-10-26 23:37:30] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:06 [I 2022-10-26 23:37:30] ...trap/engines/engine.py.229: process: 0.56543 | load: 0.00293 [I 2022-10-26 23:37:30] ...trap/engines/engine.py.230: loss: 0.16508 [I 2022-10-26 23:37:30] ...optimizers/trijoint.py.151: total_norm: 16.938606 [I 2022-10-26 23:37:36] ...trap/engines/engine.py.224: train: epoch 36 | batch 50/2304 [I 2022-10-26 23:37:36] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:00 [I 2022-10-26 23:37:36] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00334 [I 2022-10-26 23:37:36] ...trap/engines/engine.py.230: loss: 0.14876 [I 2022-10-26 23:37:36] ...optimizers/trijoint.py.151: total_norm: 9.522408 [I 2022-10-26 23:37:41] ...trap/engines/engine.py.224: train: epoch 36 | batch 60/2304 [I 2022-10-26 23:37:41] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:59 [I 2022-10-26 23:37:41] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00268 [I 2022-10-26 23:37:41] ...trap/engines/engine.py.230: loss: 0.19427 [I 2022-10-26 23:37:41] ...optimizers/trijoint.py.151: total_norm: 18.377987 [I 2022-10-26 23:37:47] ...trap/engines/engine.py.224: train: epoch 36 | batch 70/2304 [I 2022-10-26 23:37:47] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:56 [I 2022-10-26 23:37:47] ...trap/engines/engine.py.229: process: 0.56540 | load: 0.00291 [I 2022-10-26 23:37:47] ...trap/engines/engine.py.230: loss: 0.16036 [I 2022-10-26 23:37:47] ...optimizers/trijoint.py.151: total_norm: 22.241634 [I 2022-10-26 23:37:53] ...trap/engines/engine.py.224: train: epoch 36 | batch 80/2304 [I 2022-10-26 23:37:53] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:41 [I 2022-10-26 23:37:53] ...trap/engines/engine.py.229: process: 0.55341 | load: 0.00283 [I 2022-10-26 23:37:53] ...trap/engines/engine.py.230: loss: 0.15518 [I 2022-10-26 23:37:53] ...optimizers/trijoint.py.151: total_norm: 25.245155 [I 2022-10-26 23:37:58] ...trap/engines/engine.py.224: train: epoch 36 | batch 90/2304 [I 2022-10-26 23:37:58] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:30 [I 2022-10-26 23:37:58] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00260 [I 2022-10-26 23:37:58] ...trap/engines/engine.py.230: loss: 0.10133 [I 2022-10-26 23:37:58] ...optimizers/trijoint.py.151: total_norm: 7.154160 [I 2022-10-26 23:38:04] ...trap/engines/engine.py.224: train: epoch 36 | batch 100/2304 [I 2022-10-26 23:38:04] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:37 [I 2022-10-26 23:38:04] ...trap/engines/engine.py.229: process: 0.55070 | load: 0.00267 [I 2022-10-26 23:38:04] ...trap/engines/engine.py.230: loss: 0.13232 [I 2022-10-26 23:38:04] ...optimizers/trijoint.py.151: total_norm: 15.180148 [I 2022-10-26 23:38:10] ...trap/engines/engine.py.224: train: epoch 36 | batch 110/2304 [I 2022-10-26 23:38:10] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:31 [I 2022-10-26 23:38:10] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00256 [I 2022-10-26 23:38:10] ...trap/engines/engine.py.230: loss: 0.13694 [I 2022-10-26 23:38:10] ...optimizers/trijoint.py.151: total_norm: 30.232042 [I 2022-10-26 23:38:15] ...trap/engines/engine.py.224: train: epoch 36 | batch 120/2304 [I 2022-10-26 23:38:15] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:16 [I 2022-10-26 23:38:15] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00257 [I 2022-10-26 23:38:15] ...trap/engines/engine.py.230: loss: 0.22627 [I 2022-10-26 23:38:15] ...optimizers/trijoint.py.151: total_norm: 32.064499 [I 2022-10-26 23:38:21] ...trap/engines/engine.py.224: train: epoch 36 | batch 130/2304 [I 2022-10-26 23:38:21] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:07 [I 2022-10-26 23:38:21] ...trap/engines/engine.py.229: process: 0.54284 | load: 0.00247 [I 2022-10-26 23:38:21] ...trap/engines/engine.py.230: loss: 0.18746 [I 2022-10-26 23:38:21] ...optimizers/trijoint.py.151: total_norm: 14.016462 [I 2022-10-26 23:38:26] ...trap/engines/engine.py.224: train: epoch 36 | batch 140/2304 [I 2022-10-26 23:38:26] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:19:58 [I 2022-10-26 23:38:26] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00262 [I 2022-10-26 23:38:26] ...trap/engines/engine.py.230: loss: 0.18129 [I 2022-10-26 23:38:26] ...optimizers/trijoint.py.151: total_norm: 16.495705 [I 2022-10-26 23:38:32] ...trap/engines/engine.py.224: train: epoch 36 | batch 150/2304 [I 2022-10-26 23:38:32] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:52 [I 2022-10-26 23:38:32] ...trap/engines/engine.py.229: process: 0.54039 | load: 0.00252 [I 2022-10-26 23:38:32] ...trap/engines/engine.py.230: loss: 0.16460 [I 2022-10-26 23:38:32] ...optimizers/trijoint.py.151: total_norm: 18.513926 [I 2022-10-26 23:38:37] ...trap/engines/engine.py.224: train: epoch 36 | batch 160/2304 [I 2022-10-26 23:38:37] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:53 [I 2022-10-26 23:38:37] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00262 [I 2022-10-26 23:38:37] ...trap/engines/engine.py.230: loss: 0.20829 [I 2022-10-26 23:38:37] ...optimizers/trijoint.py.151: total_norm: 24.363134 [I 2022-10-26 23:38:43] ...trap/engines/engine.py.224: train: epoch 36 | batch 170/2304 [I 2022-10-26 23:38:43] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:49 [I 2022-10-26 23:38:43] ...trap/engines/engine.py.229: process: 0.56568 | load: 0.00260 [I 2022-10-26 23:38:43] ...trap/engines/engine.py.230: loss: 0.11337 [I 2022-10-26 23:38:43] ...optimizers/trijoint.py.151: total_norm: 9.841911 [I 2022-10-26 23:38:49] ...trap/engines/engine.py.224: train: epoch 36 | batch 180/2304 [I 2022-10-26 23:38:49] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:40 [I 2022-10-26 23:38:49] ...trap/engines/engine.py.229: process: 0.55724 | load: 0.00250 [I 2022-10-26 23:38:49] ...trap/engines/engine.py.230: loss: 0.17884 [I 2022-10-26 23:38:49] ...optimizers/trijoint.py.151: total_norm: 14.815094 [I 2022-10-26 23:38:54] ...trap/engines/engine.py.224: train: epoch 36 | batch 190/2304 [I 2022-10-26 23:38:54] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:38 [I 2022-10-26 23:38:54] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00247 [I 2022-10-26 23:38:54] ...trap/engines/engine.py.230: loss: 0.19436 [I 2022-10-26 23:38:54] ...optimizers/trijoint.py.151: total_norm: 17.572626 [I 2022-10-26 23:39:00] ...trap/engines/engine.py.224: train: epoch 36 | batch 200/2304 [I 2022-10-26 23:39:00] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:25 [I 2022-10-26 23:39:00] ...trap/engines/engine.py.229: process: 0.55264 | load: 0.00259 [I 2022-10-26 23:39:00] ...trap/engines/engine.py.230: loss: 0.18783 [I 2022-10-26 23:39:00] ...optimizers/trijoint.py.151: total_norm: 28.492544 [I 2022-10-26 23:39:05] ...trap/engines/engine.py.224: train: epoch 36 | batch 210/2304 [I 2022-10-26 23:39:05] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:50 [I 2022-10-26 23:39:05] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00250 [I 2022-10-26 23:39:05] ...trap/engines/engine.py.230: loss: 0.14383 [I 2022-10-26 23:39:05] ...optimizers/trijoint.py.151: total_norm: 16.159727 [I 2022-10-26 23:39:11] ...trap/engines/engine.py.224: train: epoch 36 | batch 220/2304 [I 2022-10-26 23:39:11] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:20 [I 2022-10-26 23:39:11] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00257 [I 2022-10-26 23:39:11] ...trap/engines/engine.py.230: loss: 0.19428 [I 2022-10-26 23:39:11] ...optimizers/trijoint.py.151: total_norm: 22.949120 [I 2022-10-26 23:39:17] ...trap/engines/engine.py.224: train: epoch 36 | batch 230/2304 [I 2022-10-26 23:39:17] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:18 [I 2022-10-26 23:39:17] ...trap/engines/engine.py.229: process: 0.57071 | load: 0.00268 [I 2022-10-26 23:39:17] ...trap/engines/engine.py.230: loss: 0.17824 [I 2022-10-26 23:39:17] ...optimizers/trijoint.py.151: total_norm: 16.788944 [I 2022-10-26 23:39:22] ...trap/engines/engine.py.224: train: epoch 36 | batch 240/2304 [I 2022-10-26 23:39:22] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:10 [I 2022-10-26 23:39:22] ...trap/engines/engine.py.229: process: 0.56225 | load: 0.00255 [I 2022-10-26 23:39:22] ...trap/engines/engine.py.230: loss: 0.20897 [I 2022-10-26 23:39:22] ...optimizers/trijoint.py.151: total_norm: 17.261763 [I 2022-10-26 23:39:28] ...trap/engines/engine.py.224: train: epoch 36 | batch 250/2304 [I 2022-10-26 23:39:28] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:57 [I 2022-10-26 23:39:28] ...trap/engines/engine.py.229: process: 0.55271 | load: 0.00251 [I 2022-10-26 23:39:28] ...trap/engines/engine.py.230: loss: 0.18634 [I 2022-10-26 23:39:28] ...optimizers/trijoint.py.151: total_norm: 31.823393 [I 2022-10-26 23:39:33] ...trap/engines/engine.py.224: train: epoch 36 | batch 260/2304 [I 2022-10-26 23:39:33] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:56 [I 2022-10-26 23:39:33] ...trap/engines/engine.py.229: process: 0.54485 | load: 0.00251 [I 2022-10-26 23:39:33] ...trap/engines/engine.py.230: loss: 0.14940 [I 2022-10-26 23:39:33] ...optimizers/trijoint.py.151: total_norm: 22.879490 [I 2022-10-26 23:39:39] ...trap/engines/engine.py.224: train: epoch 36 | batch 270/2304 [I 2022-10-26 23:39:39] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:49 [I 2022-10-26 23:39:39] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00274 [I 2022-10-26 23:39:39] ...trap/engines/engine.py.230: loss: 0.20092 [I 2022-10-26 23:39:39] ...optimizers/trijoint.py.151: total_norm: 19.244131 [I 2022-10-26 23:39:44] ...trap/engines/engine.py.224: train: epoch 36 | batch 280/2304 [I 2022-10-26 23:39:44] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:37 [I 2022-10-26 23:39:44] ...trap/engines/engine.py.229: process: 0.55302 | load: 0.00247 [I 2022-10-26 23:39:44] ...trap/engines/engine.py.230: loss: 0.13428 [I 2022-10-26 23:39:44] ...optimizers/trijoint.py.151: total_norm: 26.177393 [I 2022-10-26 23:39:50] ...trap/engines/engine.py.224: train: epoch 36 | batch 290/2304 [I 2022-10-26 23:39:50] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:31 [I 2022-10-26 23:39:50] ...trap/engines/engine.py.229: process: 0.54951 | load: 0.00245 [I 2022-10-26 23:39:50] ...trap/engines/engine.py.230: loss: 0.20511 [I 2022-10-26 23:39:50] ...optimizers/trijoint.py.151: total_norm: 21.390257 [I 2022-10-26 23:39:55] ...trap/engines/engine.py.224: train: epoch 36 | batch 300/2304 [I 2022-10-26 23:39:55] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:32 [I 2022-10-26 23:39:55] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00245 [I 2022-10-26 23:39:55] ...trap/engines/engine.py.230: loss: 0.19727 [I 2022-10-26 23:39:55] ...optimizers/trijoint.py.151: total_norm: 18.033510 [I 2022-10-26 23:40:01] ...trap/engines/engine.py.224: train: epoch 36 | batch 310/2304 [I 2022-10-26 23:40:01] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:28 [I 2022-10-26 23:40:01] ...trap/engines/engine.py.229: process: 0.54423 | load: 0.00257 [I 2022-10-26 23:40:01] ...trap/engines/engine.py.230: loss: 0.18716 [I 2022-10-26 23:40:01] ...optimizers/trijoint.py.151: total_norm: 48.580227 [I 2022-10-26 23:40:07] ...trap/engines/engine.py.224: train: epoch 36 | batch 320/2304 [I 2022-10-26 23:40:07] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:10 [I 2022-10-26 23:40:07] ...trap/engines/engine.py.229: process: 0.53904 | load: 0.00245 [I 2022-10-26 23:40:07] ...trap/engines/engine.py.230: loss: 0.13710 [I 2022-10-26 23:40:07] ...optimizers/trijoint.py.151: total_norm: 14.871130 [I 2022-10-26 23:40:12] ...trap/engines/engine.py.224: train: epoch 36 | batch 330/2304 [I 2022-10-26 23:40:12] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:21 [I 2022-10-26 23:40:12] ...trap/engines/engine.py.229: process: 0.55307 | load: 0.00249 [I 2022-10-26 23:40:12] ...trap/engines/engine.py.230: loss: 0.16041 [I 2022-10-26 23:40:12] ...optimizers/trijoint.py.151: total_norm: 23.653595 [I 2022-10-26 23:40:18] ...trap/engines/engine.py.224: train: epoch 36 | batch 340/2304 [I 2022-10-26 23:40:18] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:05 [I 2022-10-26 23:40:18] ...trap/engines/engine.py.229: process: 0.54717 | load: 0.00255 [I 2022-10-26 23:40:18] ...trap/engines/engine.py.230: loss: 0.15171 [I 2022-10-26 23:40:18] ...optimizers/trijoint.py.151: total_norm: 16.237143 [I 2022-10-26 23:40:23] ...trap/engines/engine.py.224: train: epoch 36 | batch 350/2304 [I 2022-10-26 23:40:23] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:12 [I 2022-10-26 23:40:23] ...trap/engines/engine.py.229: process: 0.56786 | load: 0.00240 [I 2022-10-26 23:40:23] ...trap/engines/engine.py.230: loss: 0.17145 [I 2022-10-26 23:40:23] ...optimizers/trijoint.py.151: total_norm: 18.204908 [I 2022-10-26 23:40:29] ...trap/engines/engine.py.224: train: epoch 36 | batch 360/2304 [I 2022-10-26 23:40:29] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:02 [I 2022-10-26 23:40:29] ...trap/engines/engine.py.229: process: 0.56294 | load: 0.00248 [I 2022-10-26 23:40:29] ...trap/engines/engine.py.230: loss: 0.18738 [I 2022-10-26 23:40:29] ...optimizers/trijoint.py.151: total_norm: 25.603758 [I 2022-10-26 23:40:34] ...trap/engines/engine.py.224: train: epoch 36 | batch 370/2304 [I 2022-10-26 23:40:34] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:54 [I 2022-10-26 23:40:34] ...trap/engines/engine.py.229: process: 0.54564 | load: 0.00248 [I 2022-10-26 23:40:34] ...trap/engines/engine.py.230: loss: 0.17189 [I 2022-10-26 23:40:34] ...optimizers/trijoint.py.151: total_norm: 13.411998 [I 2022-10-26 23:40:40] ...trap/engines/engine.py.224: train: epoch 36 | batch 380/2304 [I 2022-10-26 23:40:40] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:56 [I 2022-10-26 23:40:40] ...trap/engines/engine.py.229: process: 0.56214 | load: 0.00241 [I 2022-10-26 23:40:40] ...trap/engines/engine.py.230: loss: 0.19498 [I 2022-10-26 23:40:40] ...optimizers/trijoint.py.151: total_norm: 24.153801 [I 2022-10-26 23:40:46] ...trap/engines/engine.py.224: train: epoch 36 | batch 390/2304 [I 2022-10-26 23:40:46] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:45 [I 2022-10-26 23:40:46] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00244 [I 2022-10-26 23:40:46] ...trap/engines/engine.py.230: loss: 0.14318 [I 2022-10-26 23:40:46] ...optimizers/trijoint.py.151: total_norm: 18.665701 [I 2022-10-26 23:40:51] ...trap/engines/engine.py.224: train: epoch 36 | batch 400/2304 [I 2022-10-26 23:40:51] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:35 [I 2022-10-26 23:40:51] ...trap/engines/engine.py.229: process: 0.54386 | load: 0.00242 [I 2022-10-26 23:40:51] ...trap/engines/engine.py.230: loss: 0.15413 [I 2022-10-26 23:40:51] ...optimizers/trijoint.py.151: total_norm: 14.793795 [I 2022-10-26 23:40:57] ...trap/engines/engine.py.224: train: epoch 36 | batch 410/2304 [I 2022-10-26 23:40:57] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:30 [I 2022-10-26 23:40:57] ...trap/engines/engine.py.229: process: 0.55472 | load: 0.00245 [I 2022-10-26 23:40:57] ...trap/engines/engine.py.230: loss: 0.15082 [I 2022-10-26 23:40:57] ...optimizers/trijoint.py.151: total_norm: 16.419415 [I 2022-10-26 23:41:02] ...trap/engines/engine.py.224: train: epoch 36 | batch 420/2304 [I 2022-10-26 23:41:02] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:26 [I 2022-10-26 23:41:02] ...trap/engines/engine.py.229: process: 0.55115 | load: 0.00247 [I 2022-10-26 23:41:02] ...trap/engines/engine.py.230: loss: 0.16576 [I 2022-10-26 23:41:02] ...optimizers/trijoint.py.151: total_norm: 26.039242 [I 2022-10-26 23:41:08] ...trap/engines/engine.py.224: train: epoch 36 | batch 430/2304 [I 2022-10-26 23:41:08] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:25 [I 2022-10-26 23:41:08] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00243 [I 2022-10-26 23:41:08] ...trap/engines/engine.py.230: loss: 0.24922 [I 2022-10-26 23:41:08] ...optimizers/trijoint.py.151: total_norm: 16.863848 [I 2022-10-26 23:41:13] ...trap/engines/engine.py.224: train: epoch 36 | batch 440/2304 [I 2022-10-26 23:41:13] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:20 [I 2022-10-26 23:41:13] ...trap/engines/engine.py.229: process: 0.56514 | load: 0.00243 [I 2022-10-26 23:41:13] ...trap/engines/engine.py.230: loss: 0.23107 [I 2022-10-26 23:41:13] ...optimizers/trijoint.py.151: total_norm: 21.212387 [I 2022-10-26 23:41:19] ...trap/engines/engine.py.224: train: epoch 36 | batch 450/2304 [I 2022-10-26 23:41:19] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:20 [I 2022-10-26 23:41:19] ...trap/engines/engine.py.229: process: 0.56056 | load: 0.00239 [I 2022-10-26 23:41:19] ...trap/engines/engine.py.230: loss: 0.14375 [I 2022-10-26 23:41:19] ...optimizers/trijoint.py.151: total_norm: 15.818292 [I 2022-10-26 23:41:25] ...trap/engines/engine.py.224: train: epoch 36 | batch 460/2304 [I 2022-10-26 23:41:25] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:02 [I 2022-10-26 23:41:25] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00250 [I 2022-10-26 23:41:25] ...trap/engines/engine.py.230: loss: 0.21995 [I 2022-10-26 23:41:25] ...optimizers/trijoint.py.151: total_norm: 35.517307 [I 2022-10-26 23:41:30] ...trap/engines/engine.py.224: train: epoch 36 | batch 470/2304 [I 2022-10-26 23:41:30] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:50 [I 2022-10-26 23:41:30] ...trap/engines/engine.py.229: process: 0.54339 | load: 0.00245 [I 2022-10-26 23:41:30] ...trap/engines/engine.py.230: loss: 0.15536 [I 2022-10-26 23:41:30] ...optimizers/trijoint.py.151: total_norm: 18.889435 [I 2022-10-26 23:41:36] ...trap/engines/engine.py.224: train: epoch 36 | batch 480/2304 [I 2022-10-26 23:41:36] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:58 [I 2022-10-26 23:41:36] ...trap/engines/engine.py.229: process: 0.57000 | load: 0.00240 [I 2022-10-26 23:41:36] ...trap/engines/engine.py.230: loss: 0.13649 [I 2022-10-26 23:41:36] ...optimizers/trijoint.py.151: total_norm: 24.122549 [I 2022-10-26 23:41:41] ...trap/engines/engine.py.224: train: epoch 36 | batch 490/2304 [I 2022-10-26 23:41:41] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:45 [I 2022-10-26 23:41:41] ...trap/engines/engine.py.229: process: 0.55513 | load: 0.00248 [I 2022-10-26 23:41:41] ...trap/engines/engine.py.230: loss: 0.12152 [I 2022-10-26 23:41:41] ...optimizers/trijoint.py.151: total_norm: 21.481052 [I 2022-10-26 23:41:47] ...trap/engines/engine.py.224: train: epoch 36 | batch 500/2304 [I 2022-10-26 23:41:47] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:38 [I 2022-10-26 23:41:47] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00243 [I 2022-10-26 23:41:47] ...trap/engines/engine.py.230: loss: 0.25435 [I 2022-10-26 23:41:47] ...optimizers/trijoint.py.151: total_norm: 12.699296 [I 2022-10-26 23:41:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 510/2304 [I 2022-10-26 23:41:52] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:38 [I 2022-10-26 23:41:52] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00240 [I 2022-10-26 23:41:52] ...trap/engines/engine.py.230: loss: 0.14866 [I 2022-10-26 23:41:52] ...optimizers/trijoint.py.151: total_norm: 31.114273 [I 2022-10-26 23:41:58] ...trap/engines/engine.py.224: train: epoch 36 | batch 520/2304 [I 2022-10-26 23:41:58] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:16:36 [I 2022-10-26 23:41:58] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00246 [I 2022-10-26 23:41:58] ...trap/engines/engine.py.230: loss: 0.21275 [I 2022-10-26 23:41:58] ...optimizers/trijoint.py.151: total_norm: 23.709131 [I 2022-10-26 23:42:04] ...trap/engines/engine.py.224: train: epoch 36 | batch 530/2304 [I 2022-10-26 23:42:04] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:26 [I 2022-10-26 23:42:04] ...trap/engines/engine.py.229: process: 0.55980 | load: 0.00240 [I 2022-10-26 23:42:04] ...trap/engines/engine.py.230: loss: 0.16335 [I 2022-10-26 23:42:04] ...optimizers/trijoint.py.151: total_norm: 13.741837 [I 2022-10-26 23:42:09] ...trap/engines/engine.py.224: train: epoch 36 | batch 540/2304 [I 2022-10-26 23:42:09] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:24 [I 2022-10-26 23:42:09] ...trap/engines/engine.py.229: process: 0.55064 | load: 0.00244 [I 2022-10-26 23:42:09] ...trap/engines/engine.py.230: loss: 0.13071 [I 2022-10-26 23:42:09] ...optimizers/trijoint.py.151: total_norm: 13.891228 [I 2022-10-26 23:42:15] ...trap/engines/engine.py.224: train: epoch 36 | batch 550/2304 [I 2022-10-26 23:42:15] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:14 [I 2022-10-26 23:42:15] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00245 [I 2022-10-26 23:42:15] ...trap/engines/engine.py.230: loss: 0.13957 [I 2022-10-26 23:42:15] ...optimizers/trijoint.py.151: total_norm: 21.498005 [I 2022-10-26 23:42:20] ...trap/engines/engine.py.224: train: epoch 36 | batch 560/2304 [I 2022-10-26 23:42:20] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:03 [I 2022-10-26 23:42:20] ...trap/engines/engine.py.229: process: 0.54800 | load: 0.00251 [I 2022-10-26 23:42:20] ...trap/engines/engine.py.230: loss: 0.17777 [I 2022-10-26 23:42:20] ...optimizers/trijoint.py.151: total_norm: 21.739958 [I 2022-10-26 23:42:26] ...trap/engines/engine.py.224: train: epoch 36 | batch 570/2304 [I 2022-10-26 23:42:26] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:16:00 [I 2022-10-26 23:42:26] ...trap/engines/engine.py.229: process: 0.54714 | load: 0.00246 [I 2022-10-26 23:42:26] ...trap/engines/engine.py.230: loss: 0.18575 [I 2022-10-26 23:42:26] ...optimizers/trijoint.py.151: total_norm: 21.633959 [I 2022-10-26 23:42:31] ...trap/engines/engine.py.224: train: epoch 36 | batch 580/2304 [I 2022-10-26 23:42:31] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:15:53 [I 2022-10-26 23:42:31] ...trap/engines/engine.py.229: process: 0.54391 | load: 0.00247 [I 2022-10-26 23:42:31] ...trap/engines/engine.py.230: loss: 0.19972 [I 2022-10-26 23:42:31] ...optimizers/trijoint.py.151: total_norm: 23.362116 [I 2022-10-26 23:42:37] ...trap/engines/engine.py.224: train: epoch 36 | batch 590/2304 [I 2022-10-26 23:42:37] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:15:59 [I 2022-10-26 23:42:37] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00248 [I 2022-10-26 23:42:37] ...trap/engines/engine.py.230: loss: 0.13732 [I 2022-10-26 23:42:37] ...optimizers/trijoint.py.151: total_norm: 18.603502 [I 2022-10-26 23:42:43] ...trap/engines/engine.py.224: train: epoch 36 | batch 600/2304 [I 2022-10-26 23:42:43] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:49 [I 2022-10-26 23:42:43] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00251 [I 2022-10-26 23:42:43] ...trap/engines/engine.py.230: loss: 0.16883 [I 2022-10-26 23:42:43] ...optimizers/trijoint.py.151: total_norm: 24.050219 [I 2022-10-26 23:42:48] ...trap/engines/engine.py.224: train: epoch 36 | batch 610/2304 [I 2022-10-26 23:42:48] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:49 [I 2022-10-26 23:42:48] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00243 [I 2022-10-26 23:42:48] ...trap/engines/engine.py.230: loss: 0.20739 [I 2022-10-26 23:42:48] ...optimizers/trijoint.py.151: total_norm: 21.789112 [I 2022-10-26 23:42:54] ...trap/engines/engine.py.224: train: epoch 36 | batch 620/2304 [I 2022-10-26 23:42:54] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:38 [I 2022-10-26 23:42:54] ...trap/engines/engine.py.229: process: 0.54362 | load: 0.00241 [I 2022-10-26 23:42:54] ...trap/engines/engine.py.230: loss: 0.16202 [I 2022-10-26 23:42:54] ...optimizers/trijoint.py.151: total_norm: 12.815001 [I 2022-10-26 23:42:59] ...trap/engines/engine.py.224: train: epoch 36 | batch 630/2304 [I 2022-10-26 23:42:59] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:27 [I 2022-10-26 23:42:59] ...trap/engines/engine.py.229: process: 0.54670 | load: 0.00241 [I 2022-10-26 23:42:59] ...trap/engines/engine.py.230: loss: 0.15461 [I 2022-10-26 23:42:59] ...optimizers/trijoint.py.151: total_norm: 19.028545 [I 2022-10-26 23:43:05] ...trap/engines/engine.py.224: train: epoch 36 | batch 640/2304 [I 2022-10-26 23:43:05] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:23 [I 2022-10-26 23:43:05] ...trap/engines/engine.py.229: process: 0.54896 | load: 0.00246 [I 2022-10-26 23:43:05] ...trap/engines/engine.py.230: loss: 0.24594 [I 2022-10-26 23:43:05] ...optimizers/trijoint.py.151: total_norm: 22.401529 [I 2022-10-26 23:43:11] ...trap/engines/engine.py.224: train: epoch 36 | batch 650/2304 [I 2022-10-26 23:43:11] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:24 [I 2022-10-26 23:43:11] ...trap/engines/engine.py.229: process: 0.58371 | load: 0.00250 [I 2022-10-26 23:43:11] ...trap/engines/engine.py.230: loss: 0.20494 [I 2022-10-26 23:43:11] ...optimizers/trijoint.py.151: total_norm: 22.708538 [I 2022-10-26 23:43:16] ...trap/engines/engine.py.224: train: epoch 36 | batch 660/2304 [I 2022-10-26 23:43:16] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:15:10 [I 2022-10-26 23:43:16] ...trap/engines/engine.py.229: process: 0.55108 | load: 0.00247 [I 2022-10-26 23:43:16] ...trap/engines/engine.py.230: loss: 0.16359 [I 2022-10-26 23:43:16] ...optimizers/trijoint.py.151: total_norm: 24.389544 [I 2022-10-26 23:43:22] ...trap/engines/engine.py.224: train: epoch 36 | batch 670/2304 [I 2022-10-26 23:43:22] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:09 [I 2022-10-26 23:43:22] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00248 [I 2022-10-26 23:43:22] ...trap/engines/engine.py.230: loss: 0.19483 [I 2022-10-26 23:43:22] ...optimizers/trijoint.py.151: total_norm: 15.272071 [I 2022-10-26 23:43:27] ...trap/engines/engine.py.224: train: epoch 36 | batch 680/2304 [I 2022-10-26 23:43:27] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:07 [I 2022-10-26 23:43:27] ...trap/engines/engine.py.229: process: 0.56407 | load: 0.00249 [I 2022-10-26 23:43:27] ...trap/engines/engine.py.230: loss: 0.27809 [I 2022-10-26 23:43:27] ...optimizers/trijoint.py.151: total_norm: 15.861837 [I 2022-10-26 23:43:33] ...trap/engines/engine.py.224: train: epoch 36 | batch 690/2304 [I 2022-10-26 23:43:33] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:14:55 [I 2022-10-26 23:43:33] ...trap/engines/engine.py.229: process: 0.55347 | load: 0.00240 [I 2022-10-26 23:43:33] ...trap/engines/engine.py.230: loss: 0.15845 [I 2022-10-26 23:43:33] ...optimizers/trijoint.py.151: total_norm: 12.242260 [I 2022-10-26 23:43:38] ...trap/engines/engine.py.224: train: epoch 36 | batch 700/2304 [I 2022-10-26 23:43:38] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:44 [I 2022-10-26 23:43:38] ...trap/engines/engine.py.229: process: 0.54601 | load: 0.00240 [I 2022-10-26 23:43:38] ...trap/engines/engine.py.230: loss: 0.18109 [I 2022-10-26 23:43:38] ...optimizers/trijoint.py.151: total_norm: 16.639595 [I 2022-10-26 23:43:44] ...trap/engines/engine.py.224: train: epoch 36 | batch 710/2304 [I 2022-10-26 23:43:44] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:14:37 [I 2022-10-26 23:43:44] ...trap/engines/engine.py.229: process: 0.54437 | load: 0.00246 [I 2022-10-26 23:43:44] ...trap/engines/engine.py.230: loss: 0.12196 [I 2022-10-26 23:43:44] ...optimizers/trijoint.py.151: total_norm: 20.699457 [I 2022-10-26 23:43:50] ...trap/engines/engine.py.224: train: epoch 36 | batch 720/2304 [I 2022-10-26 23:43:50] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:40 [I 2022-10-26 23:43:50] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00247 [I 2022-10-26 23:43:50] ...trap/engines/engine.py.230: loss: 0.18913 [I 2022-10-26 23:43:50] ...optimizers/trijoint.py.151: total_norm: 15.014894 [I 2022-10-26 23:43:55] ...trap/engines/engine.py.224: train: epoch 36 | batch 730/2304 [I 2022-10-26 23:43:55] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:14:40 [I 2022-10-26 23:43:55] ...trap/engines/engine.py.229: process: 0.54916 | load: 0.00241 [I 2022-10-26 23:43:55] ...trap/engines/engine.py.230: loss: 0.15368 [I 2022-10-26 23:43:55] ...optimizers/trijoint.py.151: total_norm: 23.788956 [I 2022-10-26 23:44:01] ...trap/engines/engine.py.224: train: epoch 36 | batch 740/2304 [I 2022-10-26 23:44:01] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:21 [I 2022-10-26 23:44:01] ...trap/engines/engine.py.229: process: 0.54386 | load: 0.00256 [I 2022-10-26 23:44:01] ...trap/engines/engine.py.230: loss: 0.17489 [I 2022-10-26 23:44:01] ...optimizers/trijoint.py.151: total_norm: 15.406088 [I 2022-10-26 23:44:06] ...trap/engines/engine.py.224: train: epoch 36 | batch 750/2304 [I 2022-10-26 23:44:06] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:20 [I 2022-10-26 23:44:06] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00244 [I 2022-10-26 23:44:06] ...trap/engines/engine.py.230: loss: 0.13596 [I 2022-10-26 23:44:06] ...optimizers/trijoint.py.151: total_norm: 22.529627 [I 2022-10-26 23:44:12] ...trap/engines/engine.py.224: train: epoch 36 | batch 760/2304 [I 2022-10-26 23:44:12] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:18 [I 2022-10-26 23:44:12] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00244 [I 2022-10-26 23:44:12] ...trap/engines/engine.py.230: loss: 0.16200 [I 2022-10-26 23:44:12] ...optimizers/trijoint.py.151: total_norm: 17.402414 [I 2022-10-26 23:44:17] ...trap/engines/engine.py.224: train: epoch 36 | batch 770/2304 [I 2022-10-26 23:44:17] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:17 [I 2022-10-26 23:44:17] ...trap/engines/engine.py.229: process: 0.56073 | load: 0.00283 [I 2022-10-26 23:44:17] ...trap/engines/engine.py.230: loss: 0.16004 [I 2022-10-26 23:44:17] ...optimizers/trijoint.py.151: total_norm: 23.128977 [I 2022-10-26 23:44:23] ...trap/engines/engine.py.224: train: epoch 36 | batch 780/2304 [I 2022-10-26 23:44:23] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:14:12 [I 2022-10-26 23:44:23] ...trap/engines/engine.py.229: process: 0.56287 | load: 0.00240 [I 2022-10-26 23:44:23] ...trap/engines/engine.py.230: loss: 0.15936 [I 2022-10-26 23:44:23] ...optimizers/trijoint.py.151: total_norm: 20.492739 [I 2022-10-26 23:44:28] ...trap/engines/engine.py.224: train: epoch 36 | batch 790/2304 [I 2022-10-26 23:44:28] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:13:58 [I 2022-10-26 23:44:28] ...trap/engines/engine.py.229: process: 0.54662 | load: 0.00246 [I 2022-10-26 23:44:28] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-26 23:44:28] ...optimizers/trijoint.py.151: total_norm: 9.649138 [I 2022-10-26 23:44:34] ...trap/engines/engine.py.224: train: epoch 36 | batch 800/2304 [I 2022-10-26 23:44:34] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:13:53 [I 2022-10-26 23:44:34] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00241 [I 2022-10-26 23:44:34] ...trap/engines/engine.py.230: loss: 0.13156 [I 2022-10-26 23:44:34] ...optimizers/trijoint.py.151: total_norm: 34.773411 [I 2022-10-26 23:44:40] ...trap/engines/engine.py.224: train: epoch 36 | batch 810/2304 [I 2022-10-26 23:44:40] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:13:53 [I 2022-10-26 23:44:40] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00244 [I 2022-10-26 23:44:40] ...trap/engines/engine.py.230: loss: 0.22234 [I 2022-10-26 23:44:40] ...optimizers/trijoint.py.151: total_norm: 17.108604 [I 2022-10-26 23:44:45] ...trap/engines/engine.py.224: train: epoch 36 | batch 820/2304 [I 2022-10-26 23:44:45] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:13:38 [I 2022-10-26 23:44:45] ...trap/engines/engine.py.229: process: 0.54827 | load: 0.00241 [I 2022-10-26 23:44:45] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-26 23:44:45] ...optimizers/trijoint.py.151: total_norm: 7.312800 [I 2022-10-26 23:44:51] ...trap/engines/engine.py.224: train: epoch 36 | batch 830/2304 [I 2022-10-26 23:44:51] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:13:27 [I 2022-10-26 23:44:51] ...trap/engines/engine.py.229: process: 0.54407 | load: 0.00240 [I 2022-10-26 23:44:51] ...trap/engines/engine.py.230: loss: 0.21210 [I 2022-10-26 23:44:51] ...optimizers/trijoint.py.151: total_norm: 35.522480 [I 2022-10-26 23:44:56] ...trap/engines/engine.py.224: train: epoch 36 | batch 840/2304 [I 2022-10-26 23:44:56] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:32 [I 2022-10-26 23:44:56] ...trap/engines/engine.py.229: process: 0.57361 | load: 0.00241 [I 2022-10-26 23:44:56] ...trap/engines/engine.py.230: loss: 0.17051 [I 2022-10-26 23:44:56] ...optimizers/trijoint.py.151: total_norm: 54.685349 [I 2022-10-26 23:45:02] ...trap/engines/engine.py.224: train: epoch 36 | batch 850/2304 [I 2022-10-26 23:45:02] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:26 [I 2022-10-26 23:45:02] ...trap/engines/engine.py.229: process: 0.55968 | load: 0.00243 [I 2022-10-26 23:45:02] ...trap/engines/engine.py.230: loss: 0.19865 [I 2022-10-26 23:45:02] ...optimizers/trijoint.py.151: total_norm: 16.675489 [I 2022-10-26 23:45:07] ...trap/engines/engine.py.224: train: epoch 36 | batch 860/2304 [I 2022-10-26 23:45:07] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:17 [I 2022-10-26 23:45:07] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00239 [I 2022-10-26 23:45:07] ...trap/engines/engine.py.230: loss: 0.16658 [I 2022-10-26 23:45:07] ...optimizers/trijoint.py.151: total_norm: 20.542910 [I 2022-10-26 23:45:13] ...trap/engines/engine.py.224: train: epoch 36 | batch 870/2304 [I 2022-10-26 23:45:13] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:17 [I 2022-10-26 23:45:13] ...trap/engines/engine.py.229: process: 0.54938 | load: 0.00239 [I 2022-10-26 23:45:13] ...trap/engines/engine.py.230: loss: 0.18207 [I 2022-10-26 23:45:13] ...optimizers/trijoint.py.151: total_norm: 15.014990 [I 2022-10-26 23:45:18] ...trap/engines/engine.py.224: train: epoch 36 | batch 880/2304 [I 2022-10-26 23:45:18] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:12 [I 2022-10-26 23:45:18] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00242 [I 2022-10-26 23:45:18] ...trap/engines/engine.py.230: loss: 0.22980 [I 2022-10-26 23:45:18] ...optimizers/trijoint.py.151: total_norm: 37.868141 [I 2022-10-26 23:45:24] ...trap/engines/engine.py.224: train: epoch 36 | batch 890/2304 [I 2022-10-26 23:45:24] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:07 [I 2022-10-26 23:45:24] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00246 [I 2022-10-26 23:45:24] ...trap/engines/engine.py.230: loss: 0.15685 [I 2022-10-26 23:45:24] ...optimizers/trijoint.py.151: total_norm: 15.197879 [I 2022-10-26 23:45:30] ...trap/engines/engine.py.224: train: epoch 36 | batch 900/2304 [I 2022-10-26 23:45:30] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:13 [I 2022-10-26 23:45:30] ...trap/engines/engine.py.229: process: 0.54966 | load: 0.00249 [I 2022-10-26 23:45:30] ...trap/engines/engine.py.230: loss: 0.11803 [I 2022-10-26 23:45:30] ...optimizers/trijoint.py.151: total_norm: 14.078775 [I 2022-10-26 23:45:35] ...trap/engines/engine.py.224: train: epoch 36 | batch 910/2304 [I 2022-10-26 23:45:35] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:59 [I 2022-10-26 23:45:35] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00246 [I 2022-10-26 23:45:35] ...trap/engines/engine.py.230: loss: 0.17621 [I 2022-10-26 23:45:35] ...optimizers/trijoint.py.151: total_norm: 18.628641 [I 2022-10-26 23:45:41] ...trap/engines/engine.py.224: train: epoch 36 | batch 920/2304 [I 2022-10-26 23:45:41] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:05 [I 2022-10-26 23:45:41] ...trap/engines/engine.py.229: process: 0.56715 | load: 0.00250 [I 2022-10-26 23:45:41] ...trap/engines/engine.py.230: loss: 0.13895 [I 2022-10-26 23:45:41] ...optimizers/trijoint.py.151: total_norm: 15.033834 [I 2022-10-26 23:45:47] ...trap/engines/engine.py.224: train: epoch 36 | batch 930/2304 [I 2022-10-26 23:45:47] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:12:42 [I 2022-10-26 23:45:47] ...trap/engines/engine.py.229: process: 0.54631 | load: 0.00242 [I 2022-10-26 23:45:47] ...trap/engines/engine.py.230: loss: 0.15423 [I 2022-10-26 23:45:47] ...optimizers/trijoint.py.151: total_norm: 19.631353 [I 2022-10-26 23:45:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 940/2304 [I 2022-10-26 23:45:52] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:12:37 [I 2022-10-26 23:45:52] ...trap/engines/engine.py.229: process: 0.56592 | load: 0.00245 [I 2022-10-26 23:45:52] ...trap/engines/engine.py.230: loss: 0.19328 [I 2022-10-26 23:45:52] ...optimizers/trijoint.py.151: total_norm: 58.574554 [I 2022-10-26 23:45:58] ...trap/engines/engine.py.224: train: epoch 36 | batch 950/2304 [I 2022-10-26 23:45:58] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:35 [I 2022-10-26 23:45:58] ...trap/engines/engine.py.229: process: 0.55803 | load: 0.00244 [I 2022-10-26 23:45:58] ...trap/engines/engine.py.230: loss: 0.19543 [I 2022-10-26 23:45:58] ...optimizers/trijoint.py.151: total_norm: 18.954905 [I 2022-10-26 23:46:03] ...trap/engines/engine.py.224: train: epoch 36 | batch 960/2304 [I 2022-10-26 23:46:03] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:23 [I 2022-10-26 23:46:03] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00247 [I 2022-10-26 23:46:03] ...trap/engines/engine.py.230: loss: 0.18624 [I 2022-10-26 23:46:03] ...optimizers/trijoint.py.151: total_norm: 17.544216 [I 2022-10-26 23:46:09] ...trap/engines/engine.py.224: train: epoch 36 | batch 970/2304 [I 2022-10-26 23:46:09] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:21 [I 2022-10-26 23:46:09] ...trap/engines/engine.py.229: process: 0.55001 | load: 0.00247 [I 2022-10-26 23:46:09] ...trap/engines/engine.py.230: loss: 0.10236 [I 2022-10-26 23:46:09] ...optimizers/trijoint.py.151: total_norm: 10.524549 [I 2022-10-26 23:46:14] ...trap/engines/engine.py.224: train: epoch 36 | batch 980/2304 [I 2022-10-26 23:46:14] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:18 [I 2022-10-26 23:46:14] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00246 [I 2022-10-26 23:46:14] ...trap/engines/engine.py.230: loss: 0.18430 [I 2022-10-26 23:46:14] ...optimizers/trijoint.py.151: total_norm: 14.694613 [I 2022-10-26 23:46:20] ...trap/engines/engine.py.224: train: epoch 36 | batch 990/2304 [I 2022-10-26 23:46:20] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:12 [I 2022-10-26 23:46:20] ...trap/engines/engine.py.229: process: 0.56378 | load: 0.00244 [I 2022-10-26 23:46:20] ...trap/engines/engine.py.230: loss: 0.15923 [I 2022-10-26 23:46:20] ...optimizers/trijoint.py.151: total_norm: 16.227577 [I 2022-10-26 23:46:26] ...trap/engines/engine.py.224: train: epoch 36 | batch 1000/2304 [I 2022-10-26 23:46:26] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:10 [I 2022-10-26 23:46:26] ...trap/engines/engine.py.229: process: 0.56439 | load: 0.00241 [I 2022-10-26 23:46:26] ...trap/engines/engine.py.230: loss: 0.21021 [I 2022-10-26 23:46:26] ...optimizers/trijoint.py.151: total_norm: 18.449091 [I 2022-10-26 23:46:31] ...trap/engines/engine.py.224: train: epoch 36 | batch 1010/2304 [I 2022-10-26 23:46:31] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:12:02 [I 2022-10-26 23:46:31] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00242 [I 2022-10-26 23:46:31] ...trap/engines/engine.py.230: loss: 0.14688 [I 2022-10-26 23:46:31] ...optimizers/trijoint.py.151: total_norm: 25.569590 [I 2022-10-26 23:46:37] ...trap/engines/engine.py.224: train: epoch 36 | batch 1020/2304 [I 2022-10-26 23:46:37] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:11:48 [I 2022-10-26 23:46:37] ...trap/engines/engine.py.229: process: 0.54600 | load: 0.00240 [I 2022-10-26 23:46:37] ...trap/engines/engine.py.230: loss: 0.21895 [I 2022-10-26 23:46:37] ...optimizers/trijoint.py.151: total_norm: 15.446052 [I 2022-10-26 23:46:42] ...trap/engines/engine.py.224: train: epoch 36 | batch 1030/2304 [I 2022-10-26 23:46:42] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:43 [I 2022-10-26 23:46:42] ...trap/engines/engine.py.229: process: 0.55054 | load: 0.00243 [I 2022-10-26 23:46:42] ...trap/engines/engine.py.230: loss: 0.12389 [I 2022-10-26 23:46:42] ...optimizers/trijoint.py.151: total_norm: 14.330095 [I 2022-10-26 23:46:48] ...trap/engines/engine.py.224: train: epoch 36 | batch 1040/2304 [I 2022-10-26 23:46:48] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:11:43 [I 2022-10-26 23:46:48] ...trap/engines/engine.py.229: process: 0.55474 | load: 0.00247 [I 2022-10-26 23:46:48] ...trap/engines/engine.py.230: loss: 0.14263 [I 2022-10-26 23:46:48] ...optimizers/trijoint.py.151: total_norm: 13.823580 [I 2022-10-26 23:46:53] ...trap/engines/engine.py.224: train: epoch 36 | batch 1050/2304 [I 2022-10-26 23:46:53] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:32 [I 2022-10-26 23:46:53] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00242 [I 2022-10-26 23:46:53] ...trap/engines/engine.py.230: loss: 0.12315 [I 2022-10-26 23:46:53] ...optimizers/trijoint.py.151: total_norm: 15.501608 [I 2022-10-26 23:46:59] ...trap/engines/engine.py.224: train: epoch 36 | batch 1060/2304 [I 2022-10-26 23:46:59] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:46 [I 2022-10-26 23:46:59] ...trap/engines/engine.py.229: process: 0.58041 | load: 0.00240 [I 2022-10-26 23:46:59] ...trap/engines/engine.py.230: loss: 0.24825 [I 2022-10-26 23:46:59] ...optimizers/trijoint.py.151: total_norm: 38.556641 [I 2022-10-26 23:47:05] ...trap/engines/engine.py.224: train: epoch 36 | batch 1070/2304 [I 2022-10-26 23:47:05] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:34 [I 2022-10-26 23:47:05] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00258 [I 2022-10-26 23:47:05] ...trap/engines/engine.py.230: loss: 0.17385 [I 2022-10-26 23:47:05] ...optimizers/trijoint.py.151: total_norm: 21.901264 [I 2022-10-26 23:47:10] ...trap/engines/engine.py.224: train: epoch 36 | batch 1080/2304 [I 2022-10-26 23:47:10] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:20 [I 2022-10-26 23:47:10] ...trap/engines/engine.py.229: process: 0.54300 | load: 0.00245 [I 2022-10-26 23:47:10] ...trap/engines/engine.py.230: loss: 0.18555 [I 2022-10-26 23:47:10] ...optimizers/trijoint.py.151: total_norm: 37.465618 [I 2022-10-26 23:47:16] ...trap/engines/engine.py.224: train: epoch 36 | batch 1090/2304 [I 2022-10-26 23:47:16] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:14 [I 2022-10-26 23:47:16] ...trap/engines/engine.py.229: process: 0.54438 | load: 0.00241 [I 2022-10-26 23:47:16] ...trap/engines/engine.py.230: loss: 0.20908 [I 2022-10-26 23:47:16] ...optimizers/trijoint.py.151: total_norm: 36.983860 [I 2022-10-26 23:47:21] ...trap/engines/engine.py.224: train: epoch 36 | batch 1100/2304 [I 2022-10-26 23:47:21] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:09 [I 2022-10-26 23:47:21] ...trap/engines/engine.py.229: process: 0.55029 | load: 0.00246 [I 2022-10-26 23:47:21] ...trap/engines/engine.py.230: loss: 0.20371 [I 2022-10-26 23:47:21] ...optimizers/trijoint.py.151: total_norm: 14.525907 [I 2022-10-26 23:47:27] ...trap/engines/engine.py.224: train: epoch 36 | batch 1110/2304 [I 2022-10-26 23:47:27] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:03 [I 2022-10-26 23:47:27] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00247 [I 2022-10-26 23:47:27] ...trap/engines/engine.py.230: loss: 0.13923 [I 2022-10-26 23:47:27] ...optimizers/trijoint.py.151: total_norm: 14.279392 [I 2022-10-26 23:47:33] ...trap/engines/engine.py.224: train: epoch 36 | batch 1120/2304 [I 2022-10-26 23:47:33] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:10:58 [I 2022-10-26 23:47:33] ...trap/engines/engine.py.229: process: 0.56457 | load: 0.00242 [I 2022-10-26 23:47:33] ...trap/engines/engine.py.230: loss: 0.20203 [I 2022-10-26 23:47:33] ...optimizers/trijoint.py.151: total_norm: 40.048134 [I 2022-10-26 23:47:38] ...trap/engines/engine.py.224: train: epoch 36 | batch 1130/2304 [I 2022-10-26 23:47:38] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:10:55 [I 2022-10-26 23:47:38] ...trap/engines/engine.py.229: process: 0.56566 | load: 0.00242 [I 2022-10-26 23:47:38] ...trap/engines/engine.py.230: loss: 0.10387 [I 2022-10-26 23:47:38] ...optimizers/trijoint.py.151: total_norm: 13.651655 [I 2022-10-26 23:47:44] ...trap/engines/engine.py.224: train: epoch 36 | batch 1140/2304 [I 2022-10-26 23:47:44] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:10:42 [I 2022-10-26 23:47:44] ...trap/engines/engine.py.229: process: 0.53844 | load: 0.00243 [I 2022-10-26 23:47:44] ...trap/engines/engine.py.230: loss: 0.21187 [I 2022-10-26 23:47:44] ...optimizers/trijoint.py.151: total_norm: 19.765942 [I 2022-10-26 23:47:49] ...trap/engines/engine.py.224: train: epoch 36 | batch 1150/2304 [I 2022-10-26 23:47:49] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:10:40 [I 2022-10-26 23:47:49] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00240 [I 2022-10-26 23:47:49] ...trap/engines/engine.py.230: loss: 0.19411 [I 2022-10-26 23:47:49] ...optimizers/trijoint.py.151: total_norm: 29.193388 [I 2022-10-26 23:47:55] ...trap/engines/engine.py.224: train: epoch 36 | batch 1160/2304 [I 2022-10-26 23:47:55] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:10:37 [I 2022-10-26 23:47:55] ...trap/engines/engine.py.229: process: 0.56503 | load: 0.00247 [I 2022-10-26 23:47:55] ...trap/engines/engine.py.230: loss: 0.09716 [I 2022-10-26 23:47:55] ...optimizers/trijoint.py.151: total_norm: 24.933161 [I 2022-10-26 23:48:00] ...trap/engines/engine.py.224: train: epoch 36 | batch 1170/2304 [I 2022-10-26 23:48:00] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:34 [I 2022-10-26 23:48:00] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00243 [I 2022-10-26 23:48:00] ...trap/engines/engine.py.230: loss: 0.14893 [I 2022-10-26 23:48:00] ...optimizers/trijoint.py.151: total_norm: 16.246510 [I 2022-10-26 23:48:06] ...trap/engines/engine.py.224: train: epoch 36 | batch 1180/2304 [I 2022-10-26 23:48:06] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:27 [I 2022-10-26 23:48:06] ...trap/engines/engine.py.229: process: 0.56560 | load: 0.00244 [I 2022-10-26 23:48:06] ...trap/engines/engine.py.230: loss: 0.14135 [I 2022-10-26 23:48:06] ...optimizers/trijoint.py.151: total_norm: 20.947958 [I 2022-10-26 23:48:12] ...trap/engines/engine.py.224: train: epoch 36 | batch 1190/2304 [I 2022-10-26 23:48:12] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:28 [I 2022-10-26 23:48:12] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00248 [I 2022-10-26 23:48:12] ...trap/engines/engine.py.230: loss: 0.15101 [I 2022-10-26 23:48:12] ...optimizers/trijoint.py.151: total_norm: 15.024562 [I 2022-10-26 23:48:17] ...trap/engines/engine.py.224: train: epoch 36 | batch 1200/2304 [I 2022-10-26 23:48:17] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:11 [I 2022-10-26 23:48:17] ...trap/engines/engine.py.229: process: 0.55113 | load: 0.00247 [I 2022-10-26 23:48:17] ...trap/engines/engine.py.230: loss: 0.12253 [I 2022-10-26 23:48:17] ...optimizers/trijoint.py.151: total_norm: 11.090344 [I 2022-10-26 23:48:23] ...trap/engines/engine.py.224: train: epoch 36 | batch 1210/2304 [I 2022-10-26 23:48:23] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:08 [I 2022-10-26 23:48:23] ...trap/engines/engine.py.229: process: 0.56112 | load: 0.00242 [I 2022-10-26 23:48:23] ...trap/engines/engine.py.230: loss: 0.18786 [I 2022-10-26 23:48:23] ...optimizers/trijoint.py.151: total_norm: 35.501102 [I 2022-10-26 23:48:28] ...trap/engines/engine.py.224: train: epoch 36 | batch 1220/2304 [I 2022-10-26 23:48:28] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:02 [I 2022-10-26 23:48:28] ...trap/engines/engine.py.229: process: 0.54740 | load: 0.00247 [I 2022-10-26 23:48:28] ...trap/engines/engine.py.230: loss: 0.13328 [I 2022-10-26 23:48:28] ...optimizers/trijoint.py.151: total_norm: 12.788890 [I 2022-10-26 23:48:34] ...trap/engines/engine.py.224: train: epoch 36 | batch 1230/2304 [I 2022-10-26 23:48:34] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:09:57 [I 2022-10-26 23:48:34] ...trap/engines/engine.py.229: process: 0.54729 | load: 0.00245 [I 2022-10-26 23:48:34] ...trap/engines/engine.py.230: loss: 0.20508 [I 2022-10-26 23:48:34] ...optimizers/trijoint.py.151: total_norm: 32.631237 [I 2022-10-26 23:48:39] ...trap/engines/engine.py.224: train: epoch 36 | batch 1240/2304 [I 2022-10-26 23:48:39] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:09:51 [I 2022-10-26 23:48:39] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00243 [I 2022-10-26 23:48:39] ...trap/engines/engine.py.230: loss: 0.23526 [I 2022-10-26 23:48:39] ...optimizers/trijoint.py.151: total_norm: 28.233469 [I 2022-10-26 23:48:45] ...trap/engines/engine.py.224: train: epoch 36 | batch 1250/2304 [I 2022-10-26 23:48:45] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:09:46 [I 2022-10-26 23:48:45] ...trap/engines/engine.py.229: process: 0.55727 | load: 0.00249 [I 2022-10-26 23:48:45] ...trap/engines/engine.py.230: loss: 0.29392 [I 2022-10-26 23:48:45] ...optimizers/trijoint.py.151: total_norm: 14.715816 [I 2022-10-26 23:48:51] ...trap/engines/engine.py.224: train: epoch 36 | batch 1260/2304 [I 2022-10-26 23:48:51] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:09:40 [I 2022-10-26 23:48:51] ...trap/engines/engine.py.229: process: 0.54495 | load: 0.00250 [I 2022-10-26 23:48:51] ...trap/engines/engine.py.230: loss: 0.20109 [I 2022-10-26 23:48:51] ...optimizers/trijoint.py.151: total_norm: 17.958452 [I 2022-10-26 23:48:56] ...trap/engines/engine.py.224: train: epoch 36 | batch 1270/2304 [I 2022-10-26 23:48:56] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:37 [I 2022-10-26 23:48:56] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00247 [I 2022-10-26 23:48:56] ...trap/engines/engine.py.230: loss: 0.22537 [I 2022-10-26 23:48:56] ...optimizers/trijoint.py.151: total_norm: 27.325724 [I 2022-10-26 23:49:02] ...trap/engines/engine.py.224: train: epoch 36 | batch 1280/2304 [I 2022-10-26 23:49:02] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:28 [I 2022-10-26 23:49:02] ...trap/engines/engine.py.229: process: 0.55632 | load: 0.00255 [I 2022-10-26 23:49:02] ...trap/engines/engine.py.230: loss: 0.16639 [I 2022-10-26 23:49:02] ...optimizers/trijoint.py.151: total_norm: 17.274881 [I 2022-10-26 23:49:07] ...trap/engines/engine.py.224: train: epoch 36 | batch 1290/2304 [I 2022-10-26 23:49:07] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:25 [I 2022-10-26 23:49:07] ...trap/engines/engine.py.229: process: 0.56548 | load: 0.00279 [I 2022-10-26 23:49:07] ...trap/engines/engine.py.230: loss: 0.15851 [I 2022-10-26 23:49:07] ...optimizers/trijoint.py.151: total_norm: 13.045810 [I 2022-10-26 23:49:13] ...trap/engines/engine.py.224: train: epoch 36 | batch 1300/2304 [I 2022-10-26 23:49:13] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:35 [I 2022-10-26 23:49:13] ...trap/engines/engine.py.229: process: 0.64273 | load: 0.00249 [I 2022-10-26 23:49:13] ...trap/engines/engine.py.230: loss: 0.17901 [I 2022-10-26 23:49:13] ...optimizers/trijoint.py.151: total_norm: 10.634265 [I 2022-10-26 23:49:19] ...trap/engines/engine.py.224: train: epoch 36 | batch 1310/2304 [I 2022-10-26 23:49:19] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:19 [I 2022-10-26 23:49:19] ...trap/engines/engine.py.229: process: 0.56335 | load: 0.00244 [I 2022-10-26 23:49:19] ...trap/engines/engine.py.230: loss: 0.19709 [I 2022-10-26 23:49:19] ...optimizers/trijoint.py.151: total_norm: 23.912655 [I 2022-10-26 23:49:24] ...trap/engines/engine.py.224: train: epoch 36 | batch 1320/2304 [I 2022-10-26 23:49:24] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:03 [I 2022-10-26 23:49:24] ...trap/engines/engine.py.229: process: 0.53969 | load: 0.00241 [I 2022-10-26 23:49:24] ...trap/engines/engine.py.230: loss: 0.17741 [I 2022-10-26 23:49:24] ...optimizers/trijoint.py.151: total_norm: 12.803280 [I 2022-10-26 23:49:30] ...trap/engines/engine.py.224: train: epoch 36 | batch 1330/2304 [I 2022-10-26 23:49:30] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:08:58 [I 2022-10-26 23:49:30] ...trap/engines/engine.py.229: process: 0.54616 | load: 0.00240 [I 2022-10-26 23:49:30] ...trap/engines/engine.py.230: loss: 0.21590 [I 2022-10-26 23:49:30] ...optimizers/trijoint.py.151: total_norm: 28.316225 [I 2022-10-26 23:49:35] ...trap/engines/engine.py.224: train: epoch 36 | batch 1340/2304 [I 2022-10-26 23:49:35] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:08:56 [I 2022-10-26 23:49:35] ...trap/engines/engine.py.229: process: 0.56644 | load: 0.00247 [I 2022-10-26 23:49:35] ...trap/engines/engine.py.230: loss: 0.16908 [I 2022-10-26 23:49:35] ...optimizers/trijoint.py.151: total_norm: 16.797682 [I 2022-10-26 23:49:41] ...trap/engines/engine.py.224: train: epoch 36 | batch 1350/2304 [I 2022-10-26 23:49:41] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:08:48 [I 2022-10-26 23:49:41] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00244 [I 2022-10-26 23:49:41] ...trap/engines/engine.py.230: loss: 0.13825 [I 2022-10-26 23:49:41] ...optimizers/trijoint.py.151: total_norm: 18.522408 [I 2022-10-26 23:49:46] ...trap/engines/engine.py.224: train: epoch 36 | batch 1360/2304 [I 2022-10-26 23:49:46] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:08:41 [I 2022-10-26 23:49:46] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00246 [I 2022-10-26 23:49:46] ...trap/engines/engine.py.230: loss: 0.15532 [I 2022-10-26 23:49:46] ...optimizers/trijoint.py.151: total_norm: 13.571414 [I 2022-10-26 23:49:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 1370/2304 [I 2022-10-26 23:49:52] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:08:34 [I 2022-10-26 23:49:52] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00245 [I 2022-10-26 23:49:52] ...trap/engines/engine.py.230: loss: 0.15123 [I 2022-10-26 23:49:52] ...optimizers/trijoint.py.151: total_norm: 19.439131 [I 2022-10-26 23:49:57] ...trap/engines/engine.py.224: train: epoch 36 | batch 1380/2304 [I 2022-10-26 23:49:57] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:32 [I 2022-10-26 23:49:57] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00243 [I 2022-10-26 23:49:57] ...trap/engines/engine.py.230: loss: 0.23536 [I 2022-10-26 23:49:57] ...optimizers/trijoint.py.151: total_norm: 27.551601 [I 2022-10-26 23:50:03] ...trap/engines/engine.py.224: train: epoch 36 | batch 1390/2304 [I 2022-10-26 23:50:03] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:26 [I 2022-10-26 23:50:03] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00254 [I 2022-10-26 23:50:03] ...trap/engines/engine.py.230: loss: 0.14714 [I 2022-10-26 23:50:03] ...optimizers/trijoint.py.151: total_norm: 12.761254 [I 2022-10-26 23:50:09] ...trap/engines/engine.py.224: train: epoch 36 | batch 1400/2304 [I 2022-10-26 23:50:09] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:20 [I 2022-10-26 23:50:09] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00242 [I 2022-10-26 23:50:09] ...trap/engines/engine.py.230: loss: 0.12770 [I 2022-10-26 23:50:09] ...optimizers/trijoint.py.151: total_norm: 11.216953 [I 2022-10-26 23:50:14] ...trap/engines/engine.py.224: train: epoch 36 | batch 1410/2304 [I 2022-10-26 23:50:14] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:13 [I 2022-10-26 23:50:14] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00268 [I 2022-10-26 23:50:14] ...trap/engines/engine.py.230: loss: 0.13342 [I 2022-10-26 23:50:14] ...optimizers/trijoint.py.151: total_norm: 18.243711 [I 2022-10-26 23:50:20] ...trap/engines/engine.py.224: train: epoch 36 | batch 1420/2304 [I 2022-10-26 23:50:20] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:16 [I 2022-10-26 23:50:20] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00250 [I 2022-10-26 23:50:20] ...trap/engines/engine.py.230: loss: 0.18593 [I 2022-10-26 23:50:20] ...optimizers/trijoint.py.151: total_norm: 10.042602 [I 2022-10-26 23:50:25] ...trap/engines/engine.py.224: train: epoch 36 | batch 1430/2304 [I 2022-10-26 23:50:25] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:01 [I 2022-10-26 23:50:25] ...trap/engines/engine.py.229: process: 0.53844 | load: 0.00245 [I 2022-10-26 23:50:25] ...trap/engines/engine.py.230: loss: 0.11873 [I 2022-10-26 23:50:25] ...optimizers/trijoint.py.151: total_norm: 7.974567 [I 2022-10-26 23:50:31] ...trap/engines/engine.py.224: train: epoch 36 | batch 1440/2304 [I 2022-10-26 23:50:31] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:07:59 [I 2022-10-26 23:50:31] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00246 [I 2022-10-26 23:50:31] ...trap/engines/engine.py.230: loss: 0.20720 [I 2022-10-26 23:50:31] ...optimizers/trijoint.py.151: total_norm: 28.676874 [I 2022-10-26 23:50:36] ...trap/engines/engine.py.224: train: epoch 36 | batch 1450/2304 [I 2022-10-26 23:50:36] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:07:57 [I 2022-10-26 23:50:36] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00247 [I 2022-10-26 23:50:36] ...trap/engines/engine.py.230: loss: 0.12956 [I 2022-10-26 23:50:36] ...optimizers/trijoint.py.151: total_norm: 19.014822 [I 2022-10-26 23:50:42] ...trap/engines/engine.py.224: train: epoch 36 | batch 1460/2304 [I 2022-10-26 23:50:42] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:07:45 [I 2022-10-26 23:50:42] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00250 [I 2022-10-26 23:50:42] ...trap/engines/engine.py.230: loss: 0.15167 [I 2022-10-26 23:50:42] ...optimizers/trijoint.py.151: total_norm: 16.025610 [I 2022-10-26 23:50:48] ...trap/engines/engine.py.224: train: epoch 36 | batch 1470/2304 [I 2022-10-26 23:50:48] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:07:41 [I 2022-10-26 23:50:48] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00247 [I 2022-10-26 23:50:48] ...trap/engines/engine.py.230: loss: 0.10909 [I 2022-10-26 23:50:48] ...optimizers/trijoint.py.151: total_norm: 10.586671 [I 2022-10-26 23:50:53] ...trap/engines/engine.py.224: train: epoch 36 | batch 1480/2304 [I 2022-10-26 23:50:53] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:07:34 [I 2022-10-26 23:50:53] ...trap/engines/engine.py.229: process: 0.54820 | load: 0.00243 [I 2022-10-26 23:50:53] ...trap/engines/engine.py.230: loss: 0.15805 [I 2022-10-26 23:50:53] ...optimizers/trijoint.py.151: total_norm: 23.192053 [I 2022-10-26 23:50:59] ...trap/engines/engine.py.224: train: epoch 36 | batch 1490/2304 [I 2022-10-26 23:50:59] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:30 [I 2022-10-26 23:50:59] ...trap/engines/engine.py.229: process: 0.56070 | load: 0.00244 [I 2022-10-26 23:50:59] ...trap/engines/engine.py.230: loss: 0.15880 [I 2022-10-26 23:50:59] ...optimizers/trijoint.py.151: total_norm: 21.203108 [I 2022-10-26 23:51:04] ...trap/engines/engine.py.224: train: epoch 36 | batch 1500/2304 [I 2022-10-26 23:51:04] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:27 [I 2022-10-26 23:51:04] ...trap/engines/engine.py.229: process: 0.56242 | load: 0.00246 [I 2022-10-26 23:51:04] ...trap/engines/engine.py.230: loss: 0.16593 [I 2022-10-26 23:51:04] ...optimizers/trijoint.py.151: total_norm: 26.818951 [I 2022-10-26 23:51:10] ...trap/engines/engine.py.224: train: epoch 36 | batch 1510/2304 [I 2022-10-26 23:51:10] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:20 [I 2022-10-26 23:51:10] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00245 [I 2022-10-26 23:51:10] ...trap/engines/engine.py.230: loss: 0.13034 [I 2022-10-26 23:51:10] ...optimizers/trijoint.py.151: total_norm: 13.180681 [I 2022-10-26 23:51:15] ...trap/engines/engine.py.224: train: epoch 36 | batch 1520/2304 [I 2022-10-26 23:51:15] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:17 [I 2022-10-26 23:51:15] ...trap/engines/engine.py.229: process: 0.56972 | load: 0.00249 [I 2022-10-26 23:51:15] ...trap/engines/engine.py.230: loss: 0.19030 [I 2022-10-26 23:51:15] ...optimizers/trijoint.py.151: total_norm: 20.981327 [I 2022-10-26 23:51:21] ...trap/engines/engine.py.224: train: epoch 36 | batch 1530/2304 [I 2022-10-26 23:51:21] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:12 [I 2022-10-26 23:51:21] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00253 [I 2022-10-26 23:51:21] ...trap/engines/engine.py.230: loss: 0.13394 [I 2022-10-26 23:51:21] ...optimizers/trijoint.py.151: total_norm: 13.999169 [I 2022-10-26 23:51:27] ...trap/engines/engine.py.224: train: epoch 36 | batch 1540/2304 [I 2022-10-26 23:51:27] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:05 [I 2022-10-26 23:51:27] ...trap/engines/engine.py.229: process: 0.54852 | load: 0.00254 [I 2022-10-26 23:51:27] ...trap/engines/engine.py.230: loss: 0.19736 [I 2022-10-26 23:51:27] ...optimizers/trijoint.py.151: total_norm: 17.358442 [I 2022-10-26 23:51:32] ...trap/engines/engine.py.224: train: epoch 36 | batch 1550/2304 [I 2022-10-26 23:51:32] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:06:54 [I 2022-10-26 23:51:32] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00247 [I 2022-10-26 23:51:32] ...trap/engines/engine.py.230: loss: 0.22262 [I 2022-10-26 23:51:32] ...optimizers/trijoint.py.151: total_norm: 46.420856 [I 2022-10-26 23:51:38] ...trap/engines/engine.py.224: train: epoch 36 | batch 1560/2304 [I 2022-10-26 23:51:38] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:06:54 [I 2022-10-26 23:51:38] ...trap/engines/engine.py.229: process: 0.55190 | load: 0.00247 [I 2022-10-26 23:51:38] ...trap/engines/engine.py.230: loss: 0.17061 [I 2022-10-26 23:51:38] ...optimizers/trijoint.py.151: total_norm: 10.428583 [I 2022-10-26 23:51:43] ...trap/engines/engine.py.224: train: epoch 36 | batch 1570/2304 [I 2022-10-26 23:51:43] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:06:47 [I 2022-10-26 23:51:43] ...trap/engines/engine.py.229: process: 0.53941 | load: 0.00251 [I 2022-10-26 23:51:43] ...trap/engines/engine.py.230: loss: 0.15776 [I 2022-10-26 23:51:43] ...optimizers/trijoint.py.151: total_norm: 17.034805 [I 2022-10-26 23:51:49] ...trap/engines/engine.py.224: train: epoch 36 | batch 1580/2304 [I 2022-10-26 23:51:49] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:06:41 [I 2022-10-26 23:51:49] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00244 [I 2022-10-26 23:51:49] ...trap/engines/engine.py.230: loss: 0.17873 [I 2022-10-26 23:51:49] ...optimizers/trijoint.py.151: total_norm: 18.586233 [I 2022-10-26 23:51:54] ...trap/engines/engine.py.224: train: epoch 36 | batch 1590/2304 [I 2022-10-26 23:51:54] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:06:35 [I 2022-10-26 23:51:54] ...trap/engines/engine.py.229: process: 0.54432 | load: 0.00245 [I 2022-10-26 23:51:54] ...trap/engines/engine.py.230: loss: 0.16803 [I 2022-10-26 23:51:54] ...optimizers/trijoint.py.151: total_norm: 28.007753 [I 2022-10-26 23:52:00] ...trap/engines/engine.py.224: train: epoch 36 | batch 1600/2304 [I 2022-10-26 23:52:00] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:33 [I 2022-10-26 23:52:00] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00253 [I 2022-10-26 23:52:00] ...trap/engines/engine.py.230: loss: 0.22825 [I 2022-10-26 23:52:00] ...optimizers/trijoint.py.151: total_norm: 14.795447 [I 2022-10-26 23:52:05] ...trap/engines/engine.py.224: train: epoch 36 | batch 1610/2304 [I 2022-10-26 23:52:05] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:23 [I 2022-10-26 23:52:05] ...trap/engines/engine.py.229: process: 0.54767 | load: 0.00240 [I 2022-10-26 23:52:05] ...trap/engines/engine.py.230: loss: 0.17969 [I 2022-10-26 23:52:05] ...optimizers/trijoint.py.151: total_norm: 18.556898 [I 2022-10-26 23:52:11] ...trap/engines/engine.py.224: train: epoch 36 | batch 1620/2304 [I 2022-10-26 23:52:11] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:20 [I 2022-10-26 23:52:11] ...trap/engines/engine.py.229: process: 0.56403 | load: 0.00244 [I 2022-10-26 23:52:11] ...trap/engines/engine.py.230: loss: 0.24239 [I 2022-10-26 23:52:11] ...optimizers/trijoint.py.151: total_norm: 28.933159 [I 2022-10-26 23:52:17] ...trap/engines/engine.py.224: train: epoch 36 | batch 1630/2304 [I 2022-10-26 23:52:17] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:15 [I 2022-10-26 23:52:17] ...trap/engines/engine.py.229: process: 0.56157 | load: 0.00243 [I 2022-10-26 23:52:17] ...trap/engines/engine.py.230: loss: 0.18725 [I 2022-10-26 23:52:17] ...optimizers/trijoint.py.151: total_norm: 18.563837 [I 2022-10-26 23:52:22] ...trap/engines/engine.py.224: train: epoch 36 | batch 1640/2304 [I 2022-10-26 23:52:22] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:12 [I 2022-10-26 23:52:22] ...trap/engines/engine.py.229: process: 0.57682 | load: 0.00254 [I 2022-10-26 23:52:22] ...trap/engines/engine.py.230: loss: 0.15680 [I 2022-10-26 23:52:22] ...optimizers/trijoint.py.151: total_norm: 29.209509 [I 2022-10-26 23:52:28] ...trap/engines/engine.py.224: train: epoch 36 | batch 1650/2304 [I 2022-10-26 23:52:28] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:04 [I 2022-10-26 23:52:28] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00253 [I 2022-10-26 23:52:28] ...trap/engines/engine.py.230: loss: 0.15854 [I 2022-10-26 23:52:28] ...optimizers/trijoint.py.151: total_norm: 21.732222 [I 2022-10-26 23:52:33] ...trap/engines/engine.py.224: train: epoch 36 | batch 1660/2304 [I 2022-10-26 23:52:33] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:00 [I 2022-10-26 23:52:33] ...trap/engines/engine.py.229: process: 0.55800 | load: 0.00248 [I 2022-10-26 23:52:33] ...trap/engines/engine.py.230: loss: 0.25449 [I 2022-10-26 23:52:33] ...optimizers/trijoint.py.151: total_norm: 38.059025 [I 2022-10-26 23:52:39] ...trap/engines/engine.py.224: train: epoch 36 | batch 1670/2304 [I 2022-10-26 23:52:39] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:05:52 [I 2022-10-26 23:52:39] ...trap/engines/engine.py.229: process: 0.55156 | load: 0.00248 [I 2022-10-26 23:52:39] ...trap/engines/engine.py.230: loss: 0.14614 [I 2022-10-26 23:52:39] ...optimizers/trijoint.py.151: total_norm: 13.879348 [I 2022-10-26 23:52:45] ...trap/engines/engine.py.224: train: epoch 36 | batch 1680/2304 [I 2022-10-26 23:52:45] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:05:49 [I 2022-10-26 23:52:45] ...trap/engines/engine.py.229: process: 0.56588 | load: 0.00244 [I 2022-10-26 23:52:45] ...trap/engines/engine.py.230: loss: 0.19663 [I 2022-10-26 23:52:45] ...optimizers/trijoint.py.151: total_norm: 29.129488 [I 2022-10-26 23:52:50] ...trap/engines/engine.py.224: train: epoch 36 | batch 1690/2304 [I 2022-10-26 23:52:50] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:05:42 [I 2022-10-26 23:52:50] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00250 [I 2022-10-26 23:52:50] ...trap/engines/engine.py.230: loss: 0.14846 [I 2022-10-26 23:52:50] ...optimizers/trijoint.py.151: total_norm: 16.821222 [I 2022-10-26 23:52:56] ...trap/engines/engine.py.224: train: epoch 36 | batch 1700/2304 [I 2022-10-26 23:52:56] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:05:38 [I 2022-10-26 23:52:56] ...trap/engines/engine.py.229: process: 0.56387 | load: 0.00250 [I 2022-10-26 23:52:56] ...trap/engines/engine.py.230: loss: 0.12761 [I 2022-10-26 23:52:56] ...optimizers/trijoint.py.151: total_norm: 10.850982 [I 2022-10-26 23:53:01] ...trap/engines/engine.py.224: train: epoch 36 | batch 1710/2304 [I 2022-10-26 23:53:01] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:33 [I 2022-10-26 23:53:01] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00248 [I 2022-10-26 23:53:01] ...trap/engines/engine.py.230: loss: 0.26017 [I 2022-10-26 23:53:01] ...optimizers/trijoint.py.151: total_norm: 32.117676 [I 2022-10-26 23:53:07] ...trap/engines/engine.py.224: train: epoch 36 | batch 1720/2304 [I 2022-10-26 23:53:07] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:25 [I 2022-10-26 23:53:07] ...trap/engines/engine.py.229: process: 0.55539 | load: 0.00246 [I 2022-10-26 23:53:07] ...trap/engines/engine.py.230: loss: 0.12987 [I 2022-10-26 23:53:07] ...optimizers/trijoint.py.151: total_norm: 14.545124 [I 2022-10-26 23:53:13] ...trap/engines/engine.py.224: train: epoch 36 | batch 1730/2304 [I 2022-10-26 23:53:13] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:19 [I 2022-10-26 23:53:13] ...trap/engines/engine.py.229: process: 0.54583 | load: 0.00249 [I 2022-10-26 23:53:13] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-26 23:53:13] ...optimizers/trijoint.py.151: total_norm: 18.273628 [I 2022-10-26 23:53:18] ...trap/engines/engine.py.224: train: epoch 36 | batch 1740/2304 [I 2022-10-26 23:53:18] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:16 [I 2022-10-26 23:53:18] ...trap/engines/engine.py.229: process: 0.56126 | load: 0.00245 [I 2022-10-26 23:53:18] ...trap/engines/engine.py.230: loss: 0.15971 [I 2022-10-26 23:53:18] ...optimizers/trijoint.py.151: total_norm: 16.364471 [I 2022-10-26 23:53:24] ...trap/engines/engine.py.224: train: epoch 36 | batch 1750/2304 [I 2022-10-26 23:53:24] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:08 [I 2022-10-26 23:53:24] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00243 [I 2022-10-26 23:53:24] ...trap/engines/engine.py.230: loss: 0.15555 [I 2022-10-26 23:53:24] ...optimizers/trijoint.py.151: total_norm: 23.777157 [I 2022-10-26 23:53:29] ...trap/engines/engine.py.224: train: epoch 36 | batch 1760/2304 [I 2022-10-26 23:53:29] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:00 [I 2022-10-26 23:53:29] ...trap/engines/engine.py.229: process: 0.53930 | load: 0.00243 [I 2022-10-26 23:53:29] ...trap/engines/engine.py.230: loss: 0.14210 [I 2022-10-26 23:53:29] ...optimizers/trijoint.py.151: total_norm: 16.597157 [I 2022-10-26 23:53:35] ...trap/engines/engine.py.224: train: epoch 36 | batch 1770/2304 [I 2022-10-26 23:53:35] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:04:57 [I 2022-10-26 23:53:35] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00240 [I 2022-10-26 23:53:35] ...trap/engines/engine.py.230: loss: 0.13287 [I 2022-10-26 23:53:35] ...optimizers/trijoint.py.151: total_norm: 10.572513 [I 2022-10-26 23:53:41] ...trap/engines/engine.py.224: train: epoch 36 | batch 1780/2304 [I 2022-10-26 23:53:41] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:04:50 [I 2022-10-26 23:53:41] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00253 [I 2022-10-26 23:53:41] ...trap/engines/engine.py.230: loss: 0.12005 [I 2022-10-26 23:53:41] ...optimizers/trijoint.py.151: total_norm: 14.905603 [I 2022-10-26 23:53:46] ...trap/engines/engine.py.224: train: epoch 36 | batch 1790/2304 [I 2022-10-26 23:53:46] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:04:44 [I 2022-10-26 23:53:46] ...trap/engines/engine.py.229: process: 0.54772 | load: 0.00254 [I 2022-10-26 23:53:46] ...trap/engines/engine.py.230: loss: 0.16519 [I 2022-10-26 23:53:46] ...optimizers/trijoint.py.151: total_norm: 31.015995 [I 2022-10-26 23:53:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 1800/2304 [I 2022-10-26 23:53:52] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:04:42 [I 2022-10-26 23:53:52] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00251 [I 2022-10-26 23:53:52] ...trap/engines/engine.py.230: loss: 0.13687 [I 2022-10-26 23:53:52] ...optimizers/trijoint.py.151: total_norm: 9.461073 [I 2022-10-26 23:53:57] ...trap/engines/engine.py.224: train: epoch 36 | batch 1810/2304 [I 2022-10-26 23:53:57] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:35 [I 2022-10-26 23:53:57] ...trap/engines/engine.py.229: process: 0.54944 | load: 0.00252 [I 2022-10-26 23:53:57] ...trap/engines/engine.py.230: loss: 0.13200 [I 2022-10-26 23:53:57] ...optimizers/trijoint.py.151: total_norm: 13.987635 [I 2022-10-26 23:54:03] ...trap/engines/engine.py.224: train: epoch 36 | batch 1820/2304 [I 2022-10-26 23:54:03] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:27 [I 2022-10-26 23:54:03] ...trap/engines/engine.py.229: process: 0.55062 | load: 0.00244 [I 2022-10-26 23:54:03] ...trap/engines/engine.py.230: loss: 0.18229 [I 2022-10-26 23:54:03] ...optimizers/trijoint.py.151: total_norm: 18.165424 [I 2022-10-26 23:54:08] ...trap/engines/engine.py.224: train: epoch 36 | batch 1830/2304 [I 2022-10-26 23:54:08] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:04:21 [I 2022-10-26 23:54:08] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00247 [I 2022-10-26 23:54:08] ...trap/engines/engine.py.230: loss: 0.19613 [I 2022-10-26 23:54:08] ...optimizers/trijoint.py.151: total_norm: 12.673378 [I 2022-10-26 23:54:14] ...trap/engines/engine.py.224: train: epoch 36 | batch 1840/2304 [I 2022-10-26 23:54:14] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:17 [I 2022-10-26 23:54:14] ...trap/engines/engine.py.229: process: 0.56535 | load: 0.00242 [I 2022-10-26 23:54:14] ...trap/engines/engine.py.230: loss: 0.24392 [I 2022-10-26 23:54:14] ...optimizers/trijoint.py.151: total_norm: 16.286060 [I 2022-10-26 23:54:19] ...trap/engines/engine.py.224: train: epoch 36 | batch 1850/2304 [I 2022-10-26 23:54:19] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:13 [I 2022-10-26 23:54:19] ...trap/engines/engine.py.229: process: 0.56786 | load: 0.00246 [I 2022-10-26 23:54:19] ...trap/engines/engine.py.230: loss: 0.16911 [I 2022-10-26 23:54:19] ...optimizers/trijoint.py.151: total_norm: 20.156244 [I 2022-10-26 23:54:25] ...trap/engines/engine.py.224: train: epoch 36 | batch 1860/2304 [I 2022-10-26 23:54:25] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:09 [I 2022-10-26 23:54:25] ...trap/engines/engine.py.229: process: 0.55481 | load: 0.00268 [I 2022-10-26 23:54:25] ...trap/engines/engine.py.230: loss: 0.18847 [I 2022-10-26 23:54:25] ...optimizers/trijoint.py.151: total_norm: 22.610113 [I 2022-10-26 23:54:31] ...trap/engines/engine.py.224: train: epoch 36 | batch 1870/2304 [I 2022-10-26 23:54:31] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:04 [I 2022-10-26 23:54:31] ...trap/engines/engine.py.229: process: 0.57162 | load: 0.00248 [I 2022-10-26 23:54:31] ...trap/engines/engine.py.230: loss: 0.20609 [I 2022-10-26 23:54:31] ...optimizers/trijoint.py.151: total_norm: 30.555048 [I 2022-10-26 23:54:36] ...trap/engines/engine.py.224: train: epoch 36 | batch 1880/2304 [I 2022-10-26 23:54:36] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:03:57 [I 2022-10-26 23:54:36] ...trap/engines/engine.py.229: process: 0.55817 | load: 0.00247 [I 2022-10-26 23:54:36] ...trap/engines/engine.py.230: loss: 0.12381 [I 2022-10-26 23:54:36] ...optimizers/trijoint.py.151: total_norm: 11.899820 [I 2022-10-26 23:54:42] ...trap/engines/engine.py.224: train: epoch 36 | batch 1890/2304 [I 2022-10-26 23:54:42] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:03:51 [I 2022-10-26 23:54:42] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00250 [I 2022-10-26 23:54:42] ...trap/engines/engine.py.230: loss: 0.17146 [I 2022-10-26 23:54:42] ...optimizers/trijoint.py.151: total_norm: 20.502352 [I 2022-10-26 23:54:48] ...trap/engines/engine.py.224: train: epoch 36 | batch 1900/2304 [I 2022-10-26 23:54:48] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:03:46 [I 2022-10-26 23:54:48] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00242 [I 2022-10-26 23:54:48] ...trap/engines/engine.py.230: loss: 0.18483 [I 2022-10-26 23:54:48] ...optimizers/trijoint.py.151: total_norm: 15.382087 [I 2022-10-26 23:54:53] ...trap/engines/engine.py.224: train: epoch 36 | batch 1910/2304 [I 2022-10-26 23:54:53] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:03:40 [I 2022-10-26 23:54:53] ...trap/engines/engine.py.229: process: 0.55596 | load: 0.00249 [I 2022-10-26 23:54:53] ...trap/engines/engine.py.230: loss: 0.11372 [I 2022-10-26 23:54:53] ...optimizers/trijoint.py.151: total_norm: 12.033747 [I 2022-10-26 23:54:59] ...trap/engines/engine.py.224: train: epoch 36 | batch 1920/2304 [I 2022-10-26 23:54:59] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:33 [I 2022-10-26 23:54:59] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00249 [I 2022-10-26 23:54:59] ...trap/engines/engine.py.230: loss: 0.09784 [I 2022-10-26 23:54:59] ...optimizers/trijoint.py.151: total_norm: 15.692855 [I 2022-10-26 23:55:04] ...trap/engines/engine.py.224: train: epoch 36 | batch 1930/2304 [I 2022-10-26 23:55:04] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:26 [I 2022-10-26 23:55:04] ...trap/engines/engine.py.229: process: 0.54354 | load: 0.00245 [I 2022-10-26 23:55:04] ...trap/engines/engine.py.230: loss: 0.22237 [I 2022-10-26 23:55:04] ...optimizers/trijoint.py.151: total_norm: 26.990719 [I 2022-10-26 23:55:10] ...trap/engines/engine.py.224: train: epoch 36 | batch 1940/2304 [I 2022-10-26 23:55:10] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:21 [I 2022-10-26 23:55:10] ...trap/engines/engine.py.229: process: 0.55575 | load: 0.00268 [I 2022-10-26 23:55:10] ...trap/engines/engine.py.230: loss: 0.20229 [I 2022-10-26 23:55:10] ...optimizers/trijoint.py.151: total_norm: 14.374903 [I 2022-10-26 23:55:15] ...trap/engines/engine.py.224: train: epoch 36 | batch 1950/2304 [I 2022-10-26 23:55:15] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:03:14 [I 2022-10-26 23:55:15] ...trap/engines/engine.py.229: process: 0.54176 | load: 0.00241 [I 2022-10-26 23:55:15] ...trap/engines/engine.py.230: loss: 0.18463 [I 2022-10-26 23:55:15] ...optimizers/trijoint.py.151: total_norm: 15.119404 [I 2022-10-26 23:55:21] ...trap/engines/engine.py.224: train: epoch 36 | batch 1960/2304 [I 2022-10-26 23:55:21] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:09 [I 2022-10-26 23:55:21] ...trap/engines/engine.py.229: process: 0.55624 | load: 0.00250 [I 2022-10-26 23:55:21] ...trap/engines/engine.py.230: loss: 0.20295 [I 2022-10-26 23:55:21] ...optimizers/trijoint.py.151: total_norm: 13.783150 [I 2022-10-26 23:55:26] ...trap/engines/engine.py.224: train: epoch 36 | batch 1970/2304 [I 2022-10-26 23:55:26] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:05 [I 2022-10-26 23:55:26] ...trap/engines/engine.py.229: process: 0.55727 | load: 0.00249 [I 2022-10-26 23:55:26] ...trap/engines/engine.py.230: loss: 0.11429 [I 2022-10-26 23:55:26] ...optimizers/trijoint.py.151: total_norm: 12.312321 [I 2022-10-26 23:55:32] ...trap/engines/engine.py.224: train: epoch 36 | batch 1980/2304 [I 2022-10-26 23:55:32] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:02:59 [I 2022-10-26 23:55:32] ...trap/engines/engine.py.229: process: 0.55830 | load: 0.00244 [I 2022-10-26 23:55:32] ...trap/engines/engine.py.230: loss: 0.20018 [I 2022-10-26 23:55:32] ...optimizers/trijoint.py.151: total_norm: 18.746410 [I 2022-10-26 23:55:38] ...trap/engines/engine.py.224: train: epoch 36 | batch 1990/2304 [I 2022-10-26 23:55:38] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:02:57 [I 2022-10-26 23:55:38] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00267 [I 2022-10-26 23:55:38] ...trap/engines/engine.py.230: loss: 0.16748 [I 2022-10-26 23:55:38] ...optimizers/trijoint.py.151: total_norm: 18.849943 [I 2022-10-26 23:55:43] ...trap/engines/engine.py.224: train: epoch 36 | batch 2000/2304 [I 2022-10-26 23:55:43] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:02:49 [I 2022-10-26 23:55:43] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00242 [I 2022-10-26 23:55:43] ...trap/engines/engine.py.230: loss: 0.22111 [I 2022-10-26 23:55:43] ...optimizers/trijoint.py.151: total_norm: 21.784742 [I 2022-10-26 23:55:49] ...trap/engines/engine.py.224: train: epoch 36 | batch 2010/2304 [I 2022-10-26 23:55:49] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:02:45 [I 2022-10-26 23:55:49] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00247 [I 2022-10-26 23:55:49] ...trap/engines/engine.py.230: loss: 0.24122 [I 2022-10-26 23:55:49] ...optimizers/trijoint.py.151: total_norm: 32.032196 [I 2022-10-26 23:55:55] ...trap/engines/engine.py.224: train: epoch 36 | batch 2020/2304 [I 2022-10-26 23:55:55] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:02:36 [I 2022-10-26 23:55:55] ...trap/engines/engine.py.229: process: 0.53985 | load: 0.00245 [I 2022-10-26 23:55:55] ...trap/engines/engine.py.230: loss: 0.15302 [I 2022-10-26 23:55:55] ...optimizers/trijoint.py.151: total_norm: 21.299902 [I 2022-10-26 23:56:00] ...trap/engines/engine.py.224: train: epoch 36 | batch 2030/2304 [I 2022-10-26 23:56:00] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:30 [I 2022-10-26 23:56:00] ...trap/engines/engine.py.229: process: 0.53571 | load: 0.00242 [I 2022-10-26 23:56:00] ...trap/engines/engine.py.230: loss: 0.18192 [I 2022-10-26 23:56:00] ...optimizers/trijoint.py.151: total_norm: 26.860071 [I 2022-10-26 23:56:06] ...trap/engines/engine.py.224: train: epoch 36 | batch 2040/2304 [I 2022-10-26 23:56:06] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:25 [I 2022-10-26 23:56:06] ...trap/engines/engine.py.229: process: 0.54522 | load: 0.00244 [I 2022-10-26 23:56:06] ...trap/engines/engine.py.230: loss: 0.13812 [I 2022-10-26 23:56:06] ...optimizers/trijoint.py.151: total_norm: 11.377984 [I 2022-10-26 23:56:11] ...trap/engines/engine.py.224: train: epoch 36 | batch 2050/2304 [I 2022-10-26 23:56:11] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:20 [I 2022-10-26 23:56:11] ...trap/engines/engine.py.229: process: 0.54904 | load: 0.00250 [I 2022-10-26 23:56:11] ...trap/engines/engine.py.230: loss: 0.11584 [I 2022-10-26 23:56:11] ...optimizers/trijoint.py.151: total_norm: 16.294762 [I 2022-10-26 23:56:17] ...trap/engines/engine.py.224: train: epoch 36 | batch 2060/2304 [I 2022-10-26 23:56:17] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:15 [I 2022-10-26 23:56:17] ...trap/engines/engine.py.229: process: 0.54846 | load: 0.00248 [I 2022-10-26 23:56:17] ...trap/engines/engine.py.230: loss: 0.15158 [I 2022-10-26 23:56:17] ...optimizers/trijoint.py.151: total_norm: 33.278297 [I 2022-10-26 23:56:22] ...trap/engines/engine.py.224: train: epoch 36 | batch 2070/2304 [I 2022-10-26 23:56:22] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:12 [I 2022-10-26 23:56:22] ...trap/engines/engine.py.229: process: 0.60963 | load: 0.00242 [I 2022-10-26 23:56:22] ...trap/engines/engine.py.230: loss: 0.17293 [I 2022-10-26 23:56:22] ...optimizers/trijoint.py.151: total_norm: 15.466656 [I 2022-10-26 23:56:28] ...trap/engines/engine.py.224: train: epoch 36 | batch 2080/2304 [I 2022-10-26 23:56:28] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:06 [I 2022-10-26 23:56:28] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00244 [I 2022-10-26 23:56:28] ...trap/engines/engine.py.230: loss: 0.16923 [I 2022-10-26 23:56:28] ...optimizers/trijoint.py.151: total_norm: 29.399456 [I 2022-10-26 23:56:34] ...trap/engines/engine.py.224: train: epoch 36 | batch 2090/2304 [I 2022-10-26 23:56:34] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:01:59 [I 2022-10-26 23:56:34] ...trap/engines/engine.py.229: process: 0.56296 | load: 0.00244 [I 2022-10-26 23:56:34] ...trap/engines/engine.py.230: loss: 0.13838 [I 2022-10-26 23:56:34] ...optimizers/trijoint.py.151: total_norm: 21.982454 [I 2022-10-26 23:56:39] ...trap/engines/engine.py.224: train: epoch 36 | batch 2100/2304 [I 2022-10-26 23:56:39] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:01:52 [I 2022-10-26 23:56:39] ...trap/engines/engine.py.229: process: 0.54122 | load: 0.00244 [I 2022-10-26 23:56:39] ...trap/engines/engine.py.230: loss: 0.25025 [I 2022-10-26 23:56:39] ...optimizers/trijoint.py.151: total_norm: 24.746754 [I 2022-10-26 23:56:45] ...trap/engines/engine.py.224: train: epoch 36 | batch 2110/2304 [I 2022-10-26 23:56:45] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:01:47 [I 2022-10-26 23:56:45] ...trap/engines/engine.py.229: process: 0.54878 | load: 0.00252 [I 2022-10-26 23:56:45] ...trap/engines/engine.py.230: loss: 0.18606 [I 2022-10-26 23:56:45] ...optimizers/trijoint.py.151: total_norm: 21.374640 [I 2022-10-26 23:56:50] ...trap/engines/engine.py.224: train: epoch 36 | batch 2120/2304 [I 2022-10-26 23:56:50] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:01:42 [I 2022-10-26 23:56:50] ...trap/engines/engine.py.229: process: 0.55089 | load: 0.00245 [I 2022-10-26 23:56:50] ...trap/engines/engine.py.230: loss: 0.12444 [I 2022-10-26 23:56:50] ...optimizers/trijoint.py.151: total_norm: 19.058575 [I 2022-10-26 23:56:56] ...trap/engines/engine.py.224: train: epoch 36 | batch 2130/2304 [I 2022-10-26 23:56:56] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:37 [I 2022-10-26 23:56:56] ...trap/engines/engine.py.229: process: 0.55986 | load: 0.00244 [I 2022-10-26 23:56:56] ...trap/engines/engine.py.230: loss: 0.24705 [I 2022-10-26 23:56:56] ...optimizers/trijoint.py.151: total_norm: 36.250717 [I 2022-10-26 23:57:02] ...trap/engines/engine.py.224: train: epoch 36 | batch 2140/2304 [I 2022-10-26 23:57:02] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:32 [I 2022-10-26 23:57:02] ...trap/engines/engine.py.229: process: 0.56483 | load: 0.00245 [I 2022-10-26 23:57:02] ...trap/engines/engine.py.230: loss: 0.17428 [I 2022-10-26 23:57:02] ...optimizers/trijoint.py.151: total_norm: 17.542454 [I 2022-10-26 23:57:07] ...trap/engines/engine.py.224: train: epoch 36 | batch 2150/2304 [I 2022-10-26 23:57:07] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:25 [I 2022-10-26 23:57:07] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00248 [I 2022-10-26 23:57:07] ...trap/engines/engine.py.230: loss: 0.19714 [I 2022-10-26 23:57:07] ...optimizers/trijoint.py.151: total_norm: 17.511854 [I 2022-10-26 23:57:13] ...trap/engines/engine.py.224: train: epoch 36 | batch 2160/2304 [I 2022-10-26 23:57:13] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:20 [I 2022-10-26 23:57:13] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00245 [I 2022-10-26 23:57:13] ...trap/engines/engine.py.230: loss: 0.16028 [I 2022-10-26 23:57:13] ...optimizers/trijoint.py.151: total_norm: 21.135717 [I 2022-10-26 23:57:18] ...trap/engines/engine.py.224: train: epoch 36 | batch 2170/2304 [I 2022-10-26 23:57:18] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:14 [I 2022-10-26 23:57:18] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00244 [I 2022-10-26 23:57:18] ...trap/engines/engine.py.230: loss: 0.14852 [I 2022-10-26 23:57:18] ...optimizers/trijoint.py.151: total_norm: 25.501997 [I 2022-10-26 23:57:24] ...trap/engines/engine.py.224: train: epoch 36 | batch 2180/2304 [I 2022-10-26 23:57:24] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:09 [I 2022-10-26 23:57:24] ...trap/engines/engine.py.229: process: 0.56268 | load: 0.00243 [I 2022-10-26 23:57:24] ...trap/engines/engine.py.230: loss: 0.21708 [I 2022-10-26 23:57:24] ...optimizers/trijoint.py.151: total_norm: 31.554958 [I 2022-10-26 23:57:29] ...trap/engines/engine.py.224: train: epoch 36 | batch 2190/2304 [I 2022-10-26 23:57:29] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:02 [I 2022-10-26 23:57:29] ...trap/engines/engine.py.229: process: 0.53861 | load: 0.00240 [I 2022-10-26 23:57:29] ...trap/engines/engine.py.230: loss: 0.11069 [I 2022-10-26 23:57:29] ...optimizers/trijoint.py.151: total_norm: 7.787428 [I 2022-10-26 23:57:35] ...trap/engines/engine.py.224: train: epoch 36 | batch 2200/2304 [I 2022-10-26 23:57:35] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:00:57 [I 2022-10-26 23:57:35] ...trap/engines/engine.py.229: process: 0.55157 | load: 0.00254 [I 2022-10-26 23:57:35] ...trap/engines/engine.py.230: loss: 0.31060 [I 2022-10-26 23:57:35] ...optimizers/trijoint.py.151: total_norm: 23.782839 [I 2022-10-26 23:57:41] ...trap/engines/engine.py.224: train: epoch 36 | batch 2210/2304 [I 2022-10-26 23:57:41] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:00:52 [I 2022-10-26 23:57:41] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00248 [I 2022-10-26 23:57:41] ...trap/engines/engine.py.230: loss: 0.24909 [I 2022-10-26 23:57:41] ...optimizers/trijoint.py.151: total_norm: 21.478346 [I 2022-10-26 23:57:46] ...trap/engines/engine.py.224: train: epoch 36 | batch 2220/2304 [I 2022-10-26 23:57:46] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:00:46 [I 2022-10-26 23:57:46] ...trap/engines/engine.py.229: process: 0.55472 | load: 0.00248 [I 2022-10-26 23:57:46] ...trap/engines/engine.py.230: loss: 0.21720 [I 2022-10-26 23:57:46] ...optimizers/trijoint.py.151: total_norm: 23.182135 [I 2022-10-26 23:57:52] ...trap/engines/engine.py.224: train: epoch 36 | batch 2230/2304 [I 2022-10-26 23:57:52] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:00:41 [I 2022-10-26 23:57:52] ...trap/engines/engine.py.229: process: 0.56880 | load: 0.00243 [I 2022-10-26 23:57:52] ...trap/engines/engine.py.230: loss: 0.14778 [I 2022-10-26 23:57:52] ...optimizers/trijoint.py.151: total_norm: 21.844046 [I 2022-10-26 23:57:57] ...trap/engines/engine.py.224: train: epoch 36 | batch 2240/2304 [I 2022-10-26 23:57:57] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:35 [I 2022-10-26 23:57:57] ...trap/engines/engine.py.229: process: 0.56529 | load: 0.00247 [I 2022-10-26 23:57:57] ...trap/engines/engine.py.230: loss: 0.15108 [I 2022-10-26 23:57:57] ...optimizers/trijoint.py.151: total_norm: 10.703071 [I 2022-10-26 23:58:03] ...trap/engines/engine.py.224: train: epoch 36 | batch 2250/2304 [I 2022-10-26 23:58:03] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:30 [I 2022-10-26 23:58:03] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00248 [I 2022-10-26 23:58:03] ...trap/engines/engine.py.230: loss: 0.25618 [I 2022-10-26 23:58:03] ...optimizers/trijoint.py.151: total_norm: 36.614456 [I 2022-10-26 23:58:09] ...trap/engines/engine.py.224: train: epoch 36 | batch 2260/2304 [I 2022-10-26 23:58:09] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:24 [I 2022-10-26 23:58:09] ...trap/engines/engine.py.229: process: 0.55814 | load: 0.00248 [I 2022-10-26 23:58:09] ...trap/engines/engine.py.230: loss: 0.23515 [I 2022-10-26 23:58:09] ...optimizers/trijoint.py.151: total_norm: 28.639200 [I 2022-10-26 23:58:14] ...trap/engines/engine.py.224: train: epoch 36 | batch 2270/2304 [I 2022-10-26 23:58:14] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:18 [I 2022-10-26 23:58:14] ...trap/engines/engine.py.229: process: 0.55481 | load: 0.00250 [I 2022-10-26 23:58:14] ...trap/engines/engine.py.230: loss: 0.14858 [I 2022-10-26 23:58:14] ...optimizers/trijoint.py.151: total_norm: 9.826448 [I 2022-10-26 23:58:20] ...trap/engines/engine.py.224: train: epoch 36 | batch 2280/2304 [I 2022-10-26 23:58:20] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:13 [I 2022-10-26 23:58:20] ...trap/engines/engine.py.229: process: 0.54937 | load: 0.00255 [I 2022-10-26 23:58:20] ...trap/engines/engine.py.230: loss: 0.18217 [I 2022-10-26 23:58:20] ...optimizers/trijoint.py.151: total_norm: 19.859457 [I 2022-10-26 23:58:25] ...trap/engines/engine.py.224: train: epoch 36 | batch 2290/2304 [I 2022-10-26 23:58:25] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:07 [I 2022-10-26 23:58:25] ...trap/engines/engine.py.229: process: 0.56311 | load: 0.00249 [I 2022-10-26 23:58:25] ...trap/engines/engine.py.230: loss: 0.16097 [I 2022-10-26 23:58:25] ...optimizers/trijoint.py.151: total_norm: 13.352428 [I 2022-10-26 23:58:31] ...trap/engines/engine.py.224: train: epoch 36 | batch 2300/2304 [I 2022-10-26 23:58:31] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:02 [I 2022-10-26 23:58:31] ...trap/engines/engine.py.229: process: 0.54859 | load: 0.00014 [I 2022-10-26 23:58:31] ...trap/engines/engine.py.230: loss: 0.14376 [I 2022-10-26 23:58:31] ...optimizers/trijoint.py.151: total_norm: 15.781799 [I 2022-10-26 23:58:33] ...trap/engines/engine.py.224: train: epoch 36 | batch 2304/2304 [I 2022-10-26 23:58:33] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:00 [I 2022-10-26 23:58:33] ...trap/engines/engine.py.229: process: 0.55637 | load: 0.00015 [I 2022-10-26 23:58:33] ...trap/engines/engine.py.230: loss: 0.19909 [I 2022-10-26 23:58:33] ...optimizers/trijoint.py.151: total_norm: 32.450203 [S 2022-10-26 23:58:34] ...trap/engines/engine.py.129: train_epoch.epoch: 36 [S 2022-10-26 23:58:34] ...trap/engines/engine.py.129: train_epoch.loss: 0.17392466371462817 [I 2022-10-26 23:58:34] ...s/criterions/triplet.py.84: margin = 0.23500000000000015 [I 2022-10-26 23:58:35] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 36 [I 2022-10-26 23:58:35] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-26 23:58:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-26 23:58:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-26 23:58:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-26 23:58:38] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-26 23:58:38] ...trap/engines/engine.py.321: eval: epoch 36 | batch 0/510 [I 2022-10-26 23:58:38] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:29:00 [I 2022-10-26 23:58:38] ...trap/engines/engine.py.326: process: 3.41250 | load: 3.04231 [I 2022-10-26 23:58:42] ...trap/engines/engine.py.321: eval: epoch 36 | batch 10/510 [I 2022-10-26 23:58:42] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:32 [I 2022-10-26 23:58:42] ...trap/engines/engine.py.326: process: 0.35604 | load: 0.00016 [I 2022-10-26 23:58:45] ...trap/engines/engine.py.321: eval: epoch 36 | batch 20/510 [I 2022-10-26 23:58:45] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:01 [I 2022-10-26 23:58:45] ...trap/engines/engine.py.326: process: 0.32878 | load: 0.00017 [I 2022-10-26 23:58:48] ...trap/engines/engine.py.321: eval: epoch 36 | batch 30/510 [I 2022-10-26 23:58:48] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:44 [I 2022-10-26 23:58:48] ...trap/engines/engine.py.326: process: 0.34939 | load: 0.00016 [I 2022-10-26 23:58:52] ...trap/engines/engine.py.321: eval: epoch 36 | batch 40/510 [I 2022-10-26 23:58:52] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:36 [I 2022-10-26 23:58:52] ...trap/engines/engine.py.326: process: 0.32827 | load: 0.00016 [I 2022-10-26 23:58:55] ...trap/engines/engine.py.321: eval: epoch 36 | batch 50/510 [I 2022-10-26 23:58:55] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:32 [I 2022-10-26 23:58:55] ...trap/engines/engine.py.326: process: 0.32986 | load: 0.00016 [I 2022-10-26 23:58:58] ...trap/engines/engine.py.321: eval: epoch 36 | batch 60/510 [I 2022-10-26 23:58:58] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:31 [I 2022-10-26 23:58:58] ...trap/engines/engine.py.326: process: 0.35869 | load: 0.00015 [I 2022-10-26 23:59:02] ...trap/engines/engine.py.321: eval: epoch 36 | batch 70/510 [I 2022-10-26 23:59:02] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:26 [I 2022-10-26 23:59:02] ...trap/engines/engine.py.326: process: 0.33185 | load: 0.00017 [I 2022-10-26 23:59:05] ...trap/engines/engine.py.321: eval: epoch 36 | batch 80/510 [I 2022-10-26 23:59:05] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-26 23:59:05] ...trap/engines/engine.py.326: process: 0.34512 | load: 0.00016 [I 2022-10-26 23:59:08] ...trap/engines/engine.py.321: eval: epoch 36 | batch 90/510 [I 2022-10-26 23:59:08] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-26 23:59:08] ...trap/engines/engine.py.326: process: 0.35378 | load: 0.00017 [I 2022-10-26 23:59:12] ...trap/engines/engine.py.321: eval: epoch 36 | batch 100/510 [I 2022-10-26 23:59:12] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:17 [I 2022-10-26 23:59:12] ...trap/engines/engine.py.326: process: 0.28405 | load: 0.00015 [I 2022-10-26 23:59:15] ...trap/engines/engine.py.321: eval: epoch 36 | batch 110/510 [I 2022-10-26 23:59:15] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-26 23:59:15] ...trap/engines/engine.py.326: process: 0.27997 | load: 0.00014 [I 2022-10-26 23:59:17] ...trap/engines/engine.py.321: eval: epoch 36 | batch 120/510 [I 2022-10-26 23:59:17] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-26 23:59:17] ...trap/engines/engine.py.326: process: 0.28924 | load: 0.00016 [I 2022-10-26 23:59:20] ...trap/engines/engine.py.321: eval: epoch 36 | batch 130/510 [I 2022-10-26 23:59:20] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:48 [I 2022-10-26 23:59:20] ...trap/engines/engine.py.326: process: 0.28112 | load: 0.00014 [I 2022-10-26 23:59:23] ...trap/engines/engine.py.321: eval: epoch 36 | batch 140/510 [I 2022-10-26 23:59:23] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:45 [I 2022-10-26 23:59:23] ...trap/engines/engine.py.326: process: 0.28265 | load: 0.00014 [I 2022-10-26 23:59:26] ...trap/engines/engine.py.321: eval: epoch 36 | batch 150/510 [I 2022-10-26 23:59:26] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-26 23:59:26] ...trap/engines/engine.py.326: process: 0.28547 | load: 0.00014 [I 2022-10-26 23:59:29] ...trap/engines/engine.py.321: eval: epoch 36 | batch 160/510 [I 2022-10-26 23:59:29] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-26 23:59:29] ...trap/engines/engine.py.326: process: 0.28722 | load: 0.00015 [I 2022-10-26 23:59:32] ...trap/engines/engine.py.321: eval: epoch 36 | batch 170/510 [I 2022-10-26 23:59:32] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:38 [I 2022-10-26 23:59:32] ...trap/engines/engine.py.326: process: 0.29742 | load: 0.00014 [I 2022-10-26 23:59:35] ...trap/engines/engine.py.321: eval: epoch 36 | batch 180/510 [I 2022-10-26 23:59:35] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:34 [I 2022-10-26 23:59:35] ...trap/engines/engine.py.326: process: 0.27868 | load: 0.00022 [I 2022-10-26 23:59:37] ...trap/engines/engine.py.321: eval: epoch 36 | batch 190/510 [I 2022-10-26 23:59:37] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:31 [I 2022-10-26 23:59:37] ...trap/engines/engine.py.326: process: 0.28623 | load: 0.00019 [I 2022-10-26 23:59:40] ...trap/engines/engine.py.321: eval: epoch 36 | batch 200/510 [I 2022-10-26 23:59:40] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-26 23:59:40] ...trap/engines/engine.py.326: process: 0.29204 | load: 0.00015 [I 2022-10-26 23:59:43] ...trap/engines/engine.py.321: eval: epoch 36 | batch 210/510 [I 2022-10-26 23:59:43] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:25 [I 2022-10-26 23:59:43] ...trap/engines/engine.py.326: process: 0.28419 | load: 0.00017 [I 2022-10-26 23:59:46] ...trap/engines/engine.py.321: eval: epoch 36 | batch 220/510 [I 2022-10-26 23:59:46] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-26 23:59:46] ...trap/engines/engine.py.326: process: 0.28662 | load: 0.00014 [I 2022-10-26 23:59:49] ...trap/engines/engine.py.321: eval: epoch 36 | batch 230/510 [I 2022-10-26 23:59:49] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-26 23:59:49] ...trap/engines/engine.py.326: process: 0.29115 | load: 0.00014 [I 2022-10-26 23:59:52] ...trap/engines/engine.py.321: eval: epoch 36 | batch 240/510 [I 2022-10-26 23:59:52] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-26 23:59:52] ...trap/engines/engine.py.326: process: 0.29014 | load: 0.00016 [I 2022-10-26 23:59:55] ...trap/engines/engine.py.321: eval: epoch 36 | batch 250/510 [I 2022-10-26 23:59:55] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:15 [I 2022-10-26 23:59:55] ...trap/engines/engine.py.326: process: 0.28797 | load: 0.00015 [I 2022-10-26 23:59:58] ...trap/engines/engine.py.321: eval: epoch 36 | batch 260/510 [I 2022-10-26 23:59:58] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:12 [I 2022-10-26 23:59:58] ...trap/engines/engine.py.326: process: 0.28734 | load: 0.00015 [I 2022-10-27 00:00:01] ...trap/engines/engine.py.321: eval: epoch 36 | batch 270/510 [I 2022-10-27 00:00:01] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-27 00:00:01] ...trap/engines/engine.py.326: process: 0.29008 | load: 0.00015 [I 2022-10-27 00:00:03] ...trap/engines/engine.py.321: eval: epoch 36 | batch 280/510 [I 2022-10-27 00:00:03] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-27 00:00:03] ...trap/engines/engine.py.326: process: 0.28616 | load: 0.00015 [I 2022-10-27 00:00:06] ...trap/engines/engine.py.321: eval: epoch 36 | batch 290/510 [I 2022-10-27 00:00:06] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-27 00:00:06] ...trap/engines/engine.py.326: process: 0.28802 | load: 0.00015 [I 2022-10-27 00:00:09] ...trap/engines/engine.py.321: eval: epoch 36 | batch 300/510 [I 2022-10-27 00:00:09] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-27 00:00:09] ...trap/engines/engine.py.326: process: 0.30061 | load: 0.00015 [I 2022-10-27 00:00:12] ...trap/engines/engine.py.321: eval: epoch 36 | batch 310/510 [I 2022-10-27 00:00:12] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 00:00:12] ...trap/engines/engine.py.326: process: 0.29087 | load: 0.00017 [I 2022-10-27 00:00:15] ...trap/engines/engine.py.321: eval: epoch 36 | batch 320/510 [I 2022-10-27 00:00:15] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:56 [I 2022-10-27 00:00:15] ...trap/engines/engine.py.326: process: 0.28826 | load: 0.00015 [I 2022-10-27 00:00:18] ...trap/engines/engine.py.321: eval: epoch 36 | batch 330/510 [I 2022-10-27 00:00:18] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 00:00:18] ...trap/engines/engine.py.326: process: 0.28186 | load: 0.00014 [I 2022-10-27 00:00:21] ...trap/engines/engine.py.321: eval: epoch 36 | batch 340/510 [I 2022-10-27 00:00:21] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:49 [I 2022-10-27 00:00:21] ...trap/engines/engine.py.326: process: 0.29077 | load: 0.00016 [I 2022-10-27 00:00:24] ...trap/engines/engine.py.321: eval: epoch 36 | batch 350/510 [I 2022-10-27 00:00:24] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:45 [I 2022-10-27 00:00:24] ...trap/engines/engine.py.326: process: 0.29108 | load: 0.00015 [I 2022-10-27 00:00:26] ...trap/engines/engine.py.321: eval: epoch 36 | batch 360/510 [I 2022-10-27 00:00:26] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:43 [I 2022-10-27 00:00:26] ...trap/engines/engine.py.326: process: 0.28636 | load: 0.00015 [I 2022-10-27 00:00:29] ...trap/engines/engine.py.321: eval: epoch 36 | batch 370/510 [I 2022-10-27 00:00:29] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-27 00:00:29] ...trap/engines/engine.py.326: process: 0.28692 | load: 0.00015 [I 2022-10-27 00:00:32] ...trap/engines/engine.py.321: eval: epoch 36 | batch 380/510 [I 2022-10-27 00:00:32] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-27 00:00:32] ...trap/engines/engine.py.326: process: 0.28063 | load: 0.00014 [I 2022-10-27 00:00:35] ...trap/engines/engine.py.321: eval: epoch 36 | batch 390/510 [I 2022-10-27 00:00:35] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-27 00:00:35] ...trap/engines/engine.py.326: process: 0.28820 | load: 0.00015 [I 2022-10-27 00:00:38] ...trap/engines/engine.py.321: eval: epoch 36 | batch 400/510 [I 2022-10-27 00:00:38] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-27 00:00:38] ...trap/engines/engine.py.326: process: 0.29551 | load: 0.00015 [I 2022-10-27 00:00:41] ...trap/engines/engine.py.321: eval: epoch 36 | batch 410/510 [I 2022-10-27 00:00:41] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:28 [I 2022-10-27 00:00:41] ...trap/engines/engine.py.326: process: 0.27893 | load: 0.00015 [I 2022-10-27 00:00:44] ...trap/engines/engine.py.321: eval: epoch 36 | batch 420/510 [I 2022-10-27 00:00:44] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-27 00:00:44] ...trap/engines/engine.py.326: process: 0.28675 | load: 0.00014 [I 2022-10-27 00:00:47] ...trap/engines/engine.py.321: eval: epoch 36 | batch 430/510 [I 2022-10-27 00:00:47] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-27 00:00:47] ...trap/engines/engine.py.326: process: 0.29011 | load: 0.00014 [I 2022-10-27 00:00:50] ...trap/engines/engine.py.321: eval: epoch 36 | batch 440/510 [I 2022-10-27 00:00:50] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-27 00:00:50] ...trap/engines/engine.py.326: process: 0.29165 | load: 0.00015 [I 2022-10-27 00:00:52] ...trap/engines/engine.py.321: eval: epoch 36 | batch 450/510 [I 2022-10-27 00:00:52] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-27 00:00:52] ...trap/engines/engine.py.326: process: 0.28326 | load: 0.00020 [I 2022-10-27 00:00:55] ...trap/engines/engine.py.321: eval: epoch 36 | batch 460/510 [I 2022-10-27 00:00:55] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-27 00:00:55] ...trap/engines/engine.py.326: process: 0.27995 | load: 0.00021 [I 2022-10-27 00:00:58] ...trap/engines/engine.py.321: eval: epoch 36 | batch 470/510 [I 2022-10-27 00:00:58] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-27 00:00:58] ...trap/engines/engine.py.326: process: 0.28867 | load: 0.00016 [I 2022-10-27 00:01:01] ...trap/engines/engine.py.321: eval: epoch 36 | batch 480/510 [I 2022-10-27 00:01:01] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 00:01:01] ...trap/engines/engine.py.326: process: 0.28857 | load: 0.00017 [I 2022-10-27 00:01:04] ...trap/engines/engine.py.321: eval: epoch 36 | batch 490/510 [I 2022-10-27 00:01:04] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 00:01:04] ...trap/engines/engine.py.326: process: 0.28463 | load: 0.00019 [I 2022-10-27 00:01:07] ...trap/engines/engine.py.321: eval: epoch 36 | batch 500/510 [I 2022-10-27 00:01:07] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-27 00:01:07] ...trap/engines/engine.py.326: process: 0.28521 | load: 0.00015 [I 2022-10-27 00:01:10] ...trap/engines/engine.py.321: eval: epoch 36 | batch 510/510 [I 2022-10-27 00:01:10] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-27 00:01:10] ...trap/engines/engine.py.326: process: 0.28509 | load: 0.00012 [S 2022-10-27 00:01:10] ...trap/engines/engine.py.132: eval_epoch.epoch: 36 [I 2022-10-27 00:01:10] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.669300 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.887500 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.926900 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.656900 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.883300 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.925300 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013335 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006438 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006534 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013939 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008661 [S 2022-10-27 00:01:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004627 [I 2022-10-27 00:01:15] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 00:01:15] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 00:01:16] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 00:01:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 00:01:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 00:01:17] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 00:01:20] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 00:01:26] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 00:01:26] ...trap/engines/engine.py.163: Training model on trainset for epoch 37 [I 2022-10-27 00:01:26] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 00:01:28] ...trap/engines/engine.py.224: train: epoch 37 | batch 0/2304 [I 2022-10-27 00:01:28] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:08 [I 2022-10-27 00:01:28] ...trap/engines/engine.py.229: process: 2.71222 | load: 2.07702 [I 2022-10-27 00:01:28] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-27 00:01:28] ...optimizers/trijoint.py.151: total_norm: 16.786331 [I 2022-10-27 00:01:34] ...trap/engines/engine.py.224: train: epoch 37 | batch 10/2304 [I 2022-10-27 00:01:34] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:19 [I 2022-10-27 00:01:34] ...trap/engines/engine.py.229: process: 0.57424 | load: 0.00370 [I 2022-10-27 00:01:34] ...trap/engines/engine.py.230: loss: 0.19196 [I 2022-10-27 00:01:34] ...optimizers/trijoint.py.151: total_norm: 25.435419 [I 2022-10-27 00:01:40] ...trap/engines/engine.py.224: train: epoch 37 | batch 20/2304 [I 2022-10-27 00:01:40] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:12 [I 2022-10-27 00:01:40] ...trap/engines/engine.py.229: process: 0.56556 | load: 0.00487 [I 2022-10-27 00:01:40] ...trap/engines/engine.py.230: loss: 0.15833 [I 2022-10-27 00:01:40] ...optimizers/trijoint.py.151: total_norm: 29.230991 [I 2022-10-27 00:01:45] ...trap/engines/engine.py.224: train: epoch 37 | batch 30/2304 [I 2022-10-27 00:01:45] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:13 [I 2022-10-27 00:01:45] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00346 [I 2022-10-27 00:01:45] ...trap/engines/engine.py.230: loss: 0.13551 [I 2022-10-27 00:01:45] ...optimizers/trijoint.py.151: total_norm: 10.199122 [I 2022-10-27 00:01:51] ...trap/engines/engine.py.224: train: epoch 37 | batch 40/2304 [I 2022-10-27 00:01:51] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:56 [I 2022-10-27 00:01:51] ...trap/engines/engine.py.229: process: 0.55072 | load: 0.00311 [I 2022-10-27 00:01:51] ...trap/engines/engine.py.230: loss: 0.15597 [I 2022-10-27 00:01:51] ...optimizers/trijoint.py.151: total_norm: 13.975423 [I 2022-10-27 00:01:56] ...trap/engines/engine.py.224: train: epoch 37 | batch 50/2304 [I 2022-10-27 00:01:56] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:45 [I 2022-10-27 00:01:56] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00302 [I 2022-10-27 00:01:56] ...trap/engines/engine.py.230: loss: 0.15461 [I 2022-10-27 00:01:56] ...optimizers/trijoint.py.151: total_norm: 12.932978 [I 2022-10-27 00:02:02] ...trap/engines/engine.py.224: train: epoch 37 | batch 60/2304 [I 2022-10-27 00:02:02] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:45 [I 2022-10-27 00:02:02] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00313 [I 2022-10-27 00:02:02] ...trap/engines/engine.py.230: loss: 0.20595 [I 2022-10-27 00:02:02] ...optimizers/trijoint.py.151: total_norm: 18.346704 [I 2022-10-27 00:02:07] ...trap/engines/engine.py.224: train: epoch 37 | batch 70/2304 [I 2022-10-27 00:02:07] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:39 [I 2022-10-27 00:02:07] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00312 [I 2022-10-27 00:02:07] ...trap/engines/engine.py.230: loss: 0.12581 [I 2022-10-27 00:02:07] ...optimizers/trijoint.py.151: total_norm: 13.892397 [I 2022-10-27 00:02:13] ...trap/engines/engine.py.224: train: epoch 37 | batch 80/2304 [I 2022-10-27 00:02:13] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:41 [I 2022-10-27 00:02:13] ...trap/engines/engine.py.229: process: 0.57436 | load: 0.00268 [I 2022-10-27 00:02:13] ...trap/engines/engine.py.230: loss: 0.18833 [I 2022-10-27 00:02:13] ...optimizers/trijoint.py.151: total_norm: 21.705385 [I 2022-10-27 00:02:19] ...trap/engines/engine.py.224: train: epoch 37 | batch 90/2304 [I 2022-10-27 00:02:19] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:32 [I 2022-10-27 00:02:19] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00285 [I 2022-10-27 00:02:19] ...trap/engines/engine.py.230: loss: 0.15072 [I 2022-10-27 00:02:19] ...optimizers/trijoint.py.151: total_norm: 15.333862 [I 2022-10-27 00:02:24] ...trap/engines/engine.py.224: train: epoch 37 | batch 100/2304 [I 2022-10-27 00:02:24] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:33 [I 2022-10-27 00:02:24] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00286 [I 2022-10-27 00:02:24] ...trap/engines/engine.py.230: loss: 0.14722 [I 2022-10-27 00:02:24] ...optimizers/trijoint.py.151: total_norm: 15.488988 [I 2022-10-27 00:02:30] ...trap/engines/engine.py.224: train: epoch 37 | batch 110/2304 [I 2022-10-27 00:02:30] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:13 [I 2022-10-27 00:02:30] ...trap/engines/engine.py.229: process: 0.54644 | load: 0.00278 [I 2022-10-27 00:02:30] ...trap/engines/engine.py.230: loss: 0.11943 [I 2022-10-27 00:02:30] ...optimizers/trijoint.py.151: total_norm: 8.785601 [I 2022-10-27 00:02:35] ...trap/engines/engine.py.224: train: epoch 37 | batch 120/2304 [I 2022-10-27 00:02:35] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:14 [I 2022-10-27 00:02:35] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00287 [I 2022-10-27 00:02:35] ...trap/engines/engine.py.230: loss: 0.26199 [I 2022-10-27 00:02:35] ...optimizers/trijoint.py.151: total_norm: 25.030640 [I 2022-10-27 00:02:41] ...trap/engines/engine.py.224: train: epoch 37 | batch 130/2304 [I 2022-10-27 00:02:41] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:04 [I 2022-10-27 00:02:41] ...trap/engines/engine.py.229: process: 0.57344 | load: 0.00281 [I 2022-10-27 00:02:41] ...trap/engines/engine.py.230: loss: 0.21127 [I 2022-10-27 00:02:41] ...optimizers/trijoint.py.151: total_norm: 25.877039 [I 2022-10-27 00:02:46] ...trap/engines/engine.py.224: train: epoch 37 | batch 140/2304 [I 2022-10-27 00:02:46] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:59 [I 2022-10-27 00:02:46] ...trap/engines/engine.py.229: process: 0.54801 | load: 0.00270 [I 2022-10-27 00:02:46] ...trap/engines/engine.py.230: loss: 0.18218 [I 2022-10-27 00:02:46] ...optimizers/trijoint.py.151: total_norm: 27.388357 [I 2022-10-27 00:02:52] ...trap/engines/engine.py.224: train: epoch 37 | batch 150/2304 [I 2022-10-27 00:02:52] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:04 [I 2022-10-27 00:02:52] ...trap/engines/engine.py.229: process: 0.57646 | load: 0.00253 [I 2022-10-27 00:02:52] ...trap/engines/engine.py.230: loss: 0.16029 [I 2022-10-27 00:02:52] ...optimizers/trijoint.py.151: total_norm: 13.415154 [I 2022-10-27 00:02:58] ...trap/engines/engine.py.224: train: epoch 37 | batch 160/2304 [I 2022-10-27 00:02:58] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:54 [I 2022-10-27 00:02:58] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00253 [I 2022-10-27 00:02:58] ...trap/engines/engine.py.230: loss: 0.18140 [I 2022-10-27 00:02:58] ...optimizers/trijoint.py.151: total_norm: 11.743774 [I 2022-10-27 00:03:03] ...trap/engines/engine.py.224: train: epoch 37 | batch 170/2304 [I 2022-10-27 00:03:03] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:20:00 [I 2022-10-27 00:03:03] ...trap/engines/engine.py.229: process: 0.57560 | load: 0.00251 [I 2022-10-27 00:03:03] ...trap/engines/engine.py.230: loss: 0.16376 [I 2022-10-27 00:03:03] ...optimizers/trijoint.py.151: total_norm: 16.045929 [I 2022-10-27 00:03:09] ...trap/engines/engine.py.224: train: epoch 37 | batch 180/2304 [I 2022-10-27 00:03:09] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:56 [I 2022-10-27 00:03:09] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00245 [I 2022-10-27 00:03:09] ...trap/engines/engine.py.230: loss: 0.12740 [I 2022-10-27 00:03:09] ...optimizers/trijoint.py.151: total_norm: 17.797495 [I 2022-10-27 00:03:14] ...trap/engines/engine.py.224: train: epoch 37 | batch 190/2304 [I 2022-10-27 00:03:14] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:56 [I 2022-10-27 00:03:14] ...trap/engines/engine.py.229: process: 0.59138 | load: 0.00245 [I 2022-10-27 00:03:14] ...trap/engines/engine.py.230: loss: 0.13130 [I 2022-10-27 00:03:14] ...optimizers/trijoint.py.151: total_norm: 14.288838 [I 2022-10-27 00:03:20] ...trap/engines/engine.py.224: train: epoch 37 | batch 200/2304 [I 2022-10-27 00:03:20] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:43 [I 2022-10-27 00:03:20] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00250 [I 2022-10-27 00:03:20] ...trap/engines/engine.py.230: loss: 0.14383 [I 2022-10-27 00:03:20] ...optimizers/trijoint.py.151: total_norm: 14.248278 [I 2022-10-27 00:03:26] ...trap/engines/engine.py.224: train: epoch 37 | batch 210/2304 [I 2022-10-27 00:03:26] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:37 [I 2022-10-27 00:03:26] ...trap/engines/engine.py.229: process: 0.56868 | load: 0.00272 [I 2022-10-27 00:03:26] ...trap/engines/engine.py.230: loss: 0.21714 [I 2022-10-27 00:03:26] ...optimizers/trijoint.py.151: total_norm: 22.856672 [I 2022-10-27 00:03:31] ...trap/engines/engine.py.224: train: epoch 37 | batch 220/2304 [I 2022-10-27 00:03:31] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:13 [I 2022-10-27 00:03:31] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00262 [I 2022-10-27 00:03:31] ...trap/engines/engine.py.230: loss: 0.10699 [I 2022-10-27 00:03:31] ...optimizers/trijoint.py.151: total_norm: 16.069025 [I 2022-10-27 00:03:37] ...trap/engines/engine.py.224: train: epoch 37 | batch 230/2304 [I 2022-10-27 00:03:37] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:14 [I 2022-10-27 00:03:37] ...trap/engines/engine.py.229: process: 0.56746 | load: 0.00264 [I 2022-10-27 00:03:37] ...trap/engines/engine.py.230: loss: 0.18919 [I 2022-10-27 00:03:37] ...optimizers/trijoint.py.151: total_norm: 35.772190 [I 2022-10-27 00:03:42] ...trap/engines/engine.py.224: train: epoch 37 | batch 240/2304 [I 2022-10-27 00:03:42] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:12 [I 2022-10-27 00:03:42] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00249 [I 2022-10-27 00:03:42] ...trap/engines/engine.py.230: loss: 0.11189 [I 2022-10-27 00:03:42] ...optimizers/trijoint.py.151: total_norm: 16.069485 [I 2022-10-27 00:03:48] ...trap/engines/engine.py.224: train: epoch 37 | batch 250/2304 [I 2022-10-27 00:03:48] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:01 [I 2022-10-27 00:03:48] ...trap/engines/engine.py.229: process: 0.55268 | load: 0.00251 [I 2022-10-27 00:03:48] ...trap/engines/engine.py.230: loss: 0.12712 [I 2022-10-27 00:03:48] ...optimizers/trijoint.py.151: total_norm: 17.036087 [I 2022-10-27 00:03:54] ...trap/engines/engine.py.224: train: epoch 37 | batch 260/2304 [I 2022-10-27 00:03:54] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:01 [I 2022-10-27 00:03:54] ...trap/engines/engine.py.229: process: 0.55387 | load: 0.00300 [I 2022-10-27 00:03:54] ...trap/engines/engine.py.230: loss: 0.14963 [I 2022-10-27 00:03:54] ...optimizers/trijoint.py.151: total_norm: 11.982057 [I 2022-10-27 00:03:59] ...trap/engines/engine.py.224: train: epoch 37 | batch 270/2304 [I 2022-10-27 00:03:59] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:00 [I 2022-10-27 00:03:59] ...trap/engines/engine.py.229: process: 0.57565 | load: 0.00264 [I 2022-10-27 00:03:59] ...trap/engines/engine.py.230: loss: 0.17966 [I 2022-10-27 00:03:59] ...optimizers/trijoint.py.151: total_norm: 16.212889 [I 2022-10-27 00:04:05] ...trap/engines/engine.py.224: train: epoch 37 | batch 280/2304 [I 2022-10-27 00:04:05] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:52 [I 2022-10-27 00:04:05] ...trap/engines/engine.py.229: process: 0.56456 | load: 0.00279 [I 2022-10-27 00:04:05] ...trap/engines/engine.py.230: loss: 0.17660 [I 2022-10-27 00:04:05] ...optimizers/trijoint.py.151: total_norm: 11.470461 [I 2022-10-27 00:04:10] ...trap/engines/engine.py.224: train: epoch 37 | batch 290/2304 [I 2022-10-27 00:04:10] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:41 [I 2022-10-27 00:04:10] ...trap/engines/engine.py.229: process: 0.55098 | load: 0.00251 [I 2022-10-27 00:04:10] ...trap/engines/engine.py.230: loss: 0.19925 [I 2022-10-27 00:04:10] ...optimizers/trijoint.py.151: total_norm: 17.503174 [I 2022-10-27 00:04:16] ...trap/engines/engine.py.224: train: epoch 37 | batch 300/2304 [I 2022-10-27 00:04:16] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:36 [I 2022-10-27 00:04:16] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00263 [I 2022-10-27 00:04:16] ...trap/engines/engine.py.230: loss: 0.12141 [I 2022-10-27 00:04:16] ...optimizers/trijoint.py.151: total_norm: 10.155943 [I 2022-10-27 00:04:22] ...trap/engines/engine.py.224: train: epoch 37 | batch 310/2304 [I 2022-10-27 00:04:22] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:32 [I 2022-10-27 00:04:22] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00250 [I 2022-10-27 00:04:22] ...trap/engines/engine.py.230: loss: 0.17210 [I 2022-10-27 00:04:22] ...optimizers/trijoint.py.151: total_norm: 37.196198 [I 2022-10-27 00:04:27] ...trap/engines/engine.py.224: train: epoch 37 | batch 320/2304 [I 2022-10-27 00:04:27] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:25 [I 2022-10-27 00:04:27] ...trap/engines/engine.py.229: process: 0.56498 | load: 0.00247 [I 2022-10-27 00:04:27] ...trap/engines/engine.py.230: loss: 0.09602 [I 2022-10-27 00:04:27] ...optimizers/trijoint.py.151: total_norm: 10.063447 [I 2022-10-27 00:04:33] ...trap/engines/engine.py.224: train: epoch 37 | batch 330/2304 [I 2022-10-27 00:04:33] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:21 [I 2022-10-27 00:04:33] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00260 [I 2022-10-27 00:04:33] ...trap/engines/engine.py.230: loss: 0.19768 [I 2022-10-27 00:04:33] ...optimizers/trijoint.py.151: total_norm: 14.281293 [I 2022-10-27 00:04:38] ...trap/engines/engine.py.224: train: epoch 37 | batch 340/2304 [I 2022-10-27 00:04:38] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:13 [I 2022-10-27 00:04:38] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00251 [I 2022-10-27 00:04:38] ...trap/engines/engine.py.230: loss: 0.14973 [I 2022-10-27 00:04:38] ...optimizers/trijoint.py.151: total_norm: 24.730030 [I 2022-10-27 00:04:44] ...trap/engines/engine.py.224: train: epoch 37 | batch 350/2304 [I 2022-10-27 00:04:44] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:11 [I 2022-10-27 00:04:44] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00255 [I 2022-10-27 00:04:44] ...trap/engines/engine.py.230: loss: 0.14555 [I 2022-10-27 00:04:44] ...optimizers/trijoint.py.151: total_norm: 14.001089 [I 2022-10-27 00:04:50] ...trap/engines/engine.py.224: train: epoch 37 | batch 360/2304 [I 2022-10-27 00:04:50] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:05 [I 2022-10-27 00:04:50] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00249 [I 2022-10-27 00:04:50] ...trap/engines/engine.py.230: loss: 0.11135 [I 2022-10-27 00:04:50] ...optimizers/trijoint.py.151: total_norm: 19.799110 [I 2022-10-27 00:04:55] ...trap/engines/engine.py.224: train: epoch 37 | batch 370/2304 [I 2022-10-27 00:04:55] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:57 [I 2022-10-27 00:04:55] ...trap/engines/engine.py.229: process: 0.55713 | load: 0.00258 [I 2022-10-27 00:04:55] ...trap/engines/engine.py.230: loss: 0.20363 [I 2022-10-27 00:04:55] ...optimizers/trijoint.py.151: total_norm: 19.111439 [I 2022-10-27 00:05:01] ...trap/engines/engine.py.224: train: epoch 37 | batch 380/2304 [I 2022-10-27 00:05:01] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:43 [I 2022-10-27 00:05:01] ...trap/engines/engine.py.229: process: 0.55100 | load: 0.00276 [I 2022-10-27 00:05:01] ...trap/engines/engine.py.230: loss: 0.23780 [I 2022-10-27 00:05:01] ...optimizers/trijoint.py.151: total_norm: 42.034634 [I 2022-10-27 00:05:06] ...trap/engines/engine.py.224: train: epoch 37 | batch 390/2304 [I 2022-10-27 00:05:06] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:43 [I 2022-10-27 00:05:06] ...trap/engines/engine.py.229: process: 0.54767 | load: 0.00253 [I 2022-10-27 00:05:06] ...trap/engines/engine.py.230: loss: 0.19508 [I 2022-10-27 00:05:06] ...optimizers/trijoint.py.151: total_norm: 15.431437 [I 2022-10-27 00:05:12] ...trap/engines/engine.py.224: train: epoch 37 | batch 400/2304 [I 2022-10-27 00:05:12] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:32 [I 2022-10-27 00:05:12] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00268 [I 2022-10-27 00:05:12] ...trap/engines/engine.py.230: loss: 0.13041 [I 2022-10-27 00:05:12] ...optimizers/trijoint.py.151: total_norm: 22.256104 [I 2022-10-27 00:05:17] ...trap/engines/engine.py.224: train: epoch 37 | batch 410/2304 [I 2022-10-27 00:05:17] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:38 [I 2022-10-27 00:05:17] ...trap/engines/engine.py.229: process: 0.57301 | load: 0.00247 [I 2022-10-27 00:05:17] ...trap/engines/engine.py.230: loss: 0.18595 [I 2022-10-27 00:05:17] ...optimizers/trijoint.py.151: total_norm: 19.668911 [I 2022-10-27 00:05:23] ...trap/engines/engine.py.224: train: epoch 37 | batch 420/2304 [I 2022-10-27 00:05:23] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:21 [I 2022-10-27 00:05:23] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00241 [I 2022-10-27 00:05:23] ...trap/engines/engine.py.230: loss: 0.14083 [I 2022-10-27 00:05:23] ...optimizers/trijoint.py.151: total_norm: 20.823942 [I 2022-10-27 00:05:29] ...trap/engines/engine.py.224: train: epoch 37 | batch 430/2304 [I 2022-10-27 00:05:29] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:20 [I 2022-10-27 00:05:29] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00263 [I 2022-10-27 00:05:29] ...trap/engines/engine.py.230: loss: 0.12582 [I 2022-10-27 00:05:29] ...optimizers/trijoint.py.151: total_norm: 13.702042 [I 2022-10-27 00:05:34] ...trap/engines/engine.py.224: train: epoch 37 | batch 440/2304 [I 2022-10-27 00:05:34] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:16 [I 2022-10-27 00:05:34] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00261 [I 2022-10-27 00:05:34] ...trap/engines/engine.py.230: loss: 0.21775 [I 2022-10-27 00:05:34] ...optimizers/trijoint.py.151: total_norm: 22.477116 [I 2022-10-27 00:05:40] ...trap/engines/engine.py.224: train: epoch 37 | batch 450/2304 [I 2022-10-27 00:05:40] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:22 [I 2022-10-27 00:05:40] ...trap/engines/engine.py.229: process: 0.55341 | load: 0.00257 [I 2022-10-27 00:05:40] ...trap/engines/engine.py.230: loss: 0.19820 [I 2022-10-27 00:05:40] ...optimizers/trijoint.py.151: total_norm: 20.168657 [I 2022-10-27 00:05:45] ...trap/engines/engine.py.224: train: epoch 37 | batch 460/2304 [I 2022-10-27 00:05:45] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:11 [I 2022-10-27 00:05:45] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00251 [I 2022-10-27 00:05:45] ...trap/engines/engine.py.230: loss: 0.16188 [I 2022-10-27 00:05:45] ...optimizers/trijoint.py.151: total_norm: 26.351765 [I 2022-10-27 00:05:51] ...trap/engines/engine.py.224: train: epoch 37 | batch 470/2304 [I 2022-10-27 00:05:51] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:29 [I 2022-10-27 00:05:51] ...trap/engines/engine.py.229: process: 0.62925 | load: 0.00264 [I 2022-10-27 00:05:51] ...trap/engines/engine.py.230: loss: 0.21311 [I 2022-10-27 00:05:51] ...optimizers/trijoint.py.151: total_norm: 22.101851 [I 2022-10-27 00:05:57] ...trap/engines/engine.py.224: train: epoch 37 | batch 480/2304 [I 2022-10-27 00:05:57] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:05 [I 2022-10-27 00:05:57] ...trap/engines/engine.py.229: process: 0.55568 | load: 0.00254 [I 2022-10-27 00:05:57] ...trap/engines/engine.py.230: loss: 0.28124 [I 2022-10-27 00:05:57] ...optimizers/trijoint.py.151: total_norm: 24.659779 [I 2022-10-27 00:06:02] ...trap/engines/engine.py.224: train: epoch 37 | batch 490/2304 [I 2022-10-27 00:06:02] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:17:08 [I 2022-10-27 00:06:02] ...trap/engines/engine.py.229: process: 0.57979 | load: 0.00262 [I 2022-10-27 00:06:02] ...trap/engines/engine.py.230: loss: 0.15947 [I 2022-10-27 00:06:02] ...optimizers/trijoint.py.151: total_norm: 13.171288 [I 2022-10-27 00:06:08] ...trap/engines/engine.py.224: train: epoch 37 | batch 500/2304 [I 2022-10-27 00:06:08] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:50 [I 2022-10-27 00:06:08] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00267 [I 2022-10-27 00:06:08] ...trap/engines/engine.py.230: loss: 0.15073 [I 2022-10-27 00:06:08] ...optimizers/trijoint.py.151: total_norm: 15.788865 [I 2022-10-27 00:06:14] ...trap/engines/engine.py.224: train: epoch 37 | batch 510/2304 [I 2022-10-27 00:06:14] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:44 [I 2022-10-27 00:06:14] ...trap/engines/engine.py.229: process: 0.56525 | load: 0.00253 [I 2022-10-27 00:06:14] ...trap/engines/engine.py.230: loss: 0.17958 [I 2022-10-27 00:06:14] ...optimizers/trijoint.py.151: total_norm: 14.243489 [I 2022-10-27 00:06:19] ...trap/engines/engine.py.224: train: epoch 37 | batch 520/2304 [I 2022-10-27 00:06:19] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:37 [I 2022-10-27 00:06:19] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00251 [I 2022-10-27 00:06:19] ...trap/engines/engine.py.230: loss: 0.20661 [I 2022-10-27 00:06:19] ...optimizers/trijoint.py.151: total_norm: 16.331076 [I 2022-10-27 00:06:25] ...trap/engines/engine.py.224: train: epoch 37 | batch 530/2304 [I 2022-10-27 00:06:25] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:29 [I 2022-10-27 00:06:25] ...trap/engines/engine.py.229: process: 0.54693 | load: 0.00255 [I 2022-10-27 00:06:25] ...trap/engines/engine.py.230: loss: 0.15908 [I 2022-10-27 00:06:25] ...optimizers/trijoint.py.151: total_norm: 21.502111 [I 2022-10-27 00:06:31] ...trap/engines/engine.py.224: train: epoch 37 | batch 540/2304 [I 2022-10-27 00:06:31] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:34 [I 2022-10-27 00:06:31] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00253 [I 2022-10-27 00:06:31] ...trap/engines/engine.py.230: loss: 0.17258 [I 2022-10-27 00:06:31] ...optimizers/trijoint.py.151: total_norm: 11.482277 [I 2022-10-27 00:06:36] ...trap/engines/engine.py.224: train: epoch 37 | batch 550/2304 [I 2022-10-27 00:06:36] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:08 [I 2022-10-27 00:06:36] ...trap/engines/engine.py.229: process: 0.54867 | load: 0.00247 [I 2022-10-27 00:06:36] ...trap/engines/engine.py.230: loss: 0.15312 [I 2022-10-27 00:06:36] ...optimizers/trijoint.py.151: total_norm: 10.888031 [I 2022-10-27 00:06:42] ...trap/engines/engine.py.224: train: epoch 37 | batch 560/2304 [I 2022-10-27 00:06:42] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:17 [I 2022-10-27 00:06:42] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00247 [I 2022-10-27 00:06:42] ...trap/engines/engine.py.230: loss: 0.12410 [I 2022-10-27 00:06:42] ...optimizers/trijoint.py.151: total_norm: 11.231961 [I 2022-10-27 00:06:47] ...trap/engines/engine.py.224: train: epoch 37 | batch 570/2304 [I 2022-10-27 00:06:47] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:06 [I 2022-10-27 00:06:47] ...trap/engines/engine.py.229: process: 0.55288 | load: 0.00243 [I 2022-10-27 00:06:47] ...trap/engines/engine.py.230: loss: 0.18447 [I 2022-10-27 00:06:47] ...optimizers/trijoint.py.151: total_norm: 14.236836 [I 2022-10-27 00:06:53] ...trap/engines/engine.py.224: train: epoch 37 | batch 580/2304 [I 2022-10-27 00:06:53] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:52 [I 2022-10-27 00:06:53] ...trap/engines/engine.py.229: process: 0.54565 | load: 0.00249 [I 2022-10-27 00:06:53] ...trap/engines/engine.py.230: loss: 0.18445 [I 2022-10-27 00:06:53] ...optimizers/trijoint.py.151: total_norm: 17.266388 [I 2022-10-27 00:06:58] ...trap/engines/engine.py.224: train: epoch 37 | batch 590/2304 [I 2022-10-27 00:06:58] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:56 [I 2022-10-27 00:06:58] ...trap/engines/engine.py.229: process: 0.56466 | load: 0.00247 [I 2022-10-27 00:06:58] ...trap/engines/engine.py.230: loss: 0.18496 [I 2022-10-27 00:06:58] ...optimizers/trijoint.py.151: total_norm: 17.218306 [I 2022-10-27 00:07:04] ...trap/engines/engine.py.224: train: epoch 37 | batch 600/2304 [I 2022-10-27 00:07:04] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:48 [I 2022-10-27 00:07:04] ...trap/engines/engine.py.229: process: 0.56091 | load: 0.00243 [I 2022-10-27 00:07:04] ...trap/engines/engine.py.230: loss: 0.14741 [I 2022-10-27 00:07:04] ...optimizers/trijoint.py.151: total_norm: 9.438270 [I 2022-10-27 00:07:10] ...trap/engines/engine.py.224: train: epoch 37 | batch 610/2304 [I 2022-10-27 00:07:10] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:16:25 [I 2022-10-27 00:07:10] ...trap/engines/engine.py.229: process: 0.56744 | load: 0.00244 [I 2022-10-27 00:07:10] ...trap/engines/engine.py.230: loss: 0.16410 [I 2022-10-27 00:07:10] ...optimizers/trijoint.py.151: total_norm: 20.204348 [I 2022-10-27 00:07:15] ...trap/engines/engine.py.224: train: epoch 37 | batch 620/2304 [I 2022-10-27 00:07:15] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:40 [I 2022-10-27 00:07:15] ...trap/engines/engine.py.229: process: 0.54694 | load: 0.00240 [I 2022-10-27 00:07:15] ...trap/engines/engine.py.230: loss: 0.13025 [I 2022-10-27 00:07:15] ...optimizers/trijoint.py.151: total_norm: 12.833374 [I 2022-10-27 00:07:21] ...trap/engines/engine.py.224: train: epoch 37 | batch 630/2304 [I 2022-10-27 00:07:21] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:31 [I 2022-10-27 00:07:21] ...trap/engines/engine.py.229: process: 0.54513 | load: 0.00264 [I 2022-10-27 00:07:21] ...trap/engines/engine.py.230: loss: 0.14530 [I 2022-10-27 00:07:21] ...optimizers/trijoint.py.151: total_norm: 19.131332 [I 2022-10-27 00:07:26] ...trap/engines/engine.py.224: train: epoch 37 | batch 640/2304 [I 2022-10-27 00:07:26] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:26 [I 2022-10-27 00:07:26] ...trap/engines/engine.py.229: process: 0.55605 | load: 0.00256 [I 2022-10-27 00:07:26] ...trap/engines/engine.py.230: loss: 0.13063 [I 2022-10-27 00:07:26] ...optimizers/trijoint.py.151: total_norm: 18.316916 [I 2022-10-27 00:07:32] ...trap/engines/engine.py.224: train: epoch 37 | batch 650/2304 [I 2022-10-27 00:07:32] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:22 [I 2022-10-27 00:07:32] ...trap/engines/engine.py.229: process: 0.56852 | load: 0.00250 [I 2022-10-27 00:07:32] ...trap/engines/engine.py.230: loss: 0.12440 [I 2022-10-27 00:07:32] ...optimizers/trijoint.py.151: total_norm: 14.719334 [I 2022-10-27 00:07:38] ...trap/engines/engine.py.224: train: epoch 37 | batch 660/2304 [I 2022-10-27 00:07:38] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:19 [I 2022-10-27 00:07:38] ...trap/engines/engine.py.229: process: 0.56337 | load: 0.00249 [I 2022-10-27 00:07:38] ...trap/engines/engine.py.230: loss: 0.13227 [I 2022-10-27 00:07:38] ...optimizers/trijoint.py.151: total_norm: 17.403181 [I 2022-10-27 00:07:43] ...trap/engines/engine.py.224: train: epoch 37 | batch 670/2304 [I 2022-10-27 00:07:43] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:13 [I 2022-10-27 00:07:43] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00248 [I 2022-10-27 00:07:43] ...trap/engines/engine.py.230: loss: 0.15971 [I 2022-10-27 00:07:43] ...optimizers/trijoint.py.151: total_norm: 17.901344 [I 2022-10-27 00:07:49] ...trap/engines/engine.py.224: train: epoch 37 | batch 680/2304 [I 2022-10-27 00:07:49] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:01 [I 2022-10-27 00:07:49] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00251 [I 2022-10-27 00:07:49] ...trap/engines/engine.py.230: loss: 0.14916 [I 2022-10-27 00:07:49] ...optimizers/trijoint.py.151: total_norm: 7.836117 [I 2022-10-27 00:07:54] ...trap/engines/engine.py.224: train: epoch 37 | batch 690/2304 [I 2022-10-27 00:07:54] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:57 [I 2022-10-27 00:07:54] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00242 [I 2022-10-27 00:07:54] ...trap/engines/engine.py.230: loss: 0.15727 [I 2022-10-27 00:07:54] ...optimizers/trijoint.py.151: total_norm: 17.025705 [I 2022-10-27 00:08:00] ...trap/engines/engine.py.224: train: epoch 37 | batch 700/2304 [I 2022-10-27 00:08:00] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:15:01 [I 2022-10-27 00:08:00] ...trap/engines/engine.py.229: process: 0.57876 | load: 0.00254 [I 2022-10-27 00:08:00] ...trap/engines/engine.py.230: loss: 0.14811 [I 2022-10-27 00:08:00] ...optimizers/trijoint.py.151: total_norm: 17.121159 [I 2022-10-27 00:08:05] ...trap/engines/engine.py.224: train: epoch 37 | batch 710/2304 [I 2022-10-27 00:08:05] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:48 [I 2022-10-27 00:08:05] ...trap/engines/engine.py.229: process: 0.54877 | load: 0.00243 [I 2022-10-27 00:08:05] ...trap/engines/engine.py.230: loss: 0.15195 [I 2022-10-27 00:08:05] ...optimizers/trijoint.py.151: total_norm: 15.569204 [I 2022-10-27 00:08:11] ...trap/engines/engine.py.224: train: epoch 37 | batch 720/2304 [I 2022-10-27 00:08:11] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:51 [I 2022-10-27 00:08:11] ...trap/engines/engine.py.229: process: 0.55942 | load: 0.00243 [I 2022-10-27 00:08:11] ...trap/engines/engine.py.230: loss: 0.18227 [I 2022-10-27 00:08:11] ...optimizers/trijoint.py.151: total_norm: 13.497394 [I 2022-10-27 00:08:17] ...trap/engines/engine.py.224: train: epoch 37 | batch 730/2304 [I 2022-10-27 00:08:17] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:25 [I 2022-10-27 00:08:17] ...trap/engines/engine.py.229: process: 0.53513 | load: 0.00242 [I 2022-10-27 00:08:17] ...trap/engines/engine.py.230: loss: 0.17140 [I 2022-10-27 00:08:17] ...optimizers/trijoint.py.151: total_norm: 14.816159 [I 2022-10-27 00:08:22] ...trap/engines/engine.py.224: train: epoch 37 | batch 740/2304 [I 2022-10-27 00:08:22] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:30 [I 2022-10-27 00:08:22] ...trap/engines/engine.py.229: process: 0.56751 | load: 0.00246 [I 2022-10-27 00:08:22] ...trap/engines/engine.py.230: loss: 0.11619 [I 2022-10-27 00:08:22] ...optimizers/trijoint.py.151: total_norm: 10.346089 [I 2022-10-27 00:08:28] ...trap/engines/engine.py.224: train: epoch 37 | batch 750/2304 [I 2022-10-27 00:08:28] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:24 [I 2022-10-27 00:08:28] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00248 [I 2022-10-27 00:08:28] ...trap/engines/engine.py.230: loss: 0.17935 [I 2022-10-27 00:08:28] ...optimizers/trijoint.py.151: total_norm: 22.134089 [I 2022-10-27 00:08:33] ...trap/engines/engine.py.224: train: epoch 37 | batch 760/2304 [I 2022-10-27 00:08:33] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:16 [I 2022-10-27 00:08:33] ...trap/engines/engine.py.229: process: 0.54720 | load: 0.00247 [I 2022-10-27 00:08:33] ...trap/engines/engine.py.230: loss: 0.19841 [I 2022-10-27 00:08:33] ...optimizers/trijoint.py.151: total_norm: 19.497009 [I 2022-10-27 00:08:39] ...trap/engines/engine.py.224: train: epoch 37 | batch 770/2304 [I 2022-10-27 00:08:39] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:16 [I 2022-10-27 00:08:39] ...trap/engines/engine.py.229: process: 0.55532 | load: 0.00248 [I 2022-10-27 00:08:39] ...trap/engines/engine.py.230: loss: 0.12985 [I 2022-10-27 00:08:39] ...optimizers/trijoint.py.151: total_norm: 11.812024 [I 2022-10-27 00:08:45] ...trap/engines/engine.py.224: train: epoch 37 | batch 780/2304 [I 2022-10-27 00:08:45] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:11 [I 2022-10-27 00:08:45] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00252 [I 2022-10-27 00:08:45] ...trap/engines/engine.py.230: loss: 0.16116 [I 2022-10-27 00:08:45] ...optimizers/trijoint.py.151: total_norm: 19.650078 [I 2022-10-27 00:08:50] ...trap/engines/engine.py.224: train: epoch 37 | batch 790/2304 [I 2022-10-27 00:08:50] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:13:56 [I 2022-10-27 00:08:50] ...trap/engines/engine.py.229: process: 0.54484 | load: 0.00245 [I 2022-10-27 00:08:50] ...trap/engines/engine.py.230: loss: 0.19791 [I 2022-10-27 00:08:50] ...optimizers/trijoint.py.151: total_norm: 24.369070 [I 2022-10-27 00:08:56] ...trap/engines/engine.py.224: train: epoch 37 | batch 800/2304 [I 2022-10-27 00:08:56] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:50 [I 2022-10-27 00:08:56] ...trap/engines/engine.py.229: process: 0.56331 | load: 0.00245 [I 2022-10-27 00:08:56] ...trap/engines/engine.py.230: loss: 0.18379 [I 2022-10-27 00:08:56] ...optimizers/trijoint.py.151: total_norm: 14.412897 [I 2022-10-27 00:09:01] ...trap/engines/engine.py.224: train: epoch 37 | batch 810/2304 [I 2022-10-27 00:09:01] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:49 [I 2022-10-27 00:09:01] ...trap/engines/engine.py.229: process: 0.55421 | load: 0.00251 [I 2022-10-27 00:09:01] ...trap/engines/engine.py.230: loss: 0.16801 [I 2022-10-27 00:09:01] ...optimizers/trijoint.py.151: total_norm: 40.455242 [I 2022-10-27 00:09:07] ...trap/engines/engine.py.224: train: epoch 37 | batch 820/2304 [I 2022-10-27 00:09:07] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:39 [I 2022-10-27 00:09:07] ...trap/engines/engine.py.229: process: 0.53955 | load: 0.00245 [I 2022-10-27 00:09:07] ...trap/engines/engine.py.230: loss: 0.10501 [I 2022-10-27 00:09:07] ...optimizers/trijoint.py.151: total_norm: 11.892631 [I 2022-10-27 00:09:12] ...trap/engines/engine.py.224: train: epoch 37 | batch 830/2304 [I 2022-10-27 00:09:12] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:47 [I 2022-10-27 00:09:12] ...trap/engines/engine.py.229: process: 0.56980 | load: 0.00251 [I 2022-10-27 00:09:12] ...trap/engines/engine.py.230: loss: 0.19926 [I 2022-10-27 00:09:12] ...optimizers/trijoint.py.151: total_norm: 12.220870 [I 2022-10-27 00:09:18] ...trap/engines/engine.py.224: train: epoch 37 | batch 840/2304 [I 2022-10-27 00:09:18] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:41 [I 2022-10-27 00:09:18] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00247 [I 2022-10-27 00:09:18] ...trap/engines/engine.py.230: loss: 0.24507 [I 2022-10-27 00:09:18] ...optimizers/trijoint.py.151: total_norm: 27.519182 [I 2022-10-27 00:09:24] ...trap/engines/engine.py.224: train: epoch 37 | batch 850/2304 [I 2022-10-27 00:09:24] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:34 [I 2022-10-27 00:09:24] ...trap/engines/engine.py.229: process: 0.57029 | load: 0.00248 [I 2022-10-27 00:09:24] ...trap/engines/engine.py.230: loss: 0.16881 [I 2022-10-27 00:09:24] ...optimizers/trijoint.py.151: total_norm: 23.120615 [I 2022-10-27 00:09:29] ...trap/engines/engine.py.224: train: epoch 37 | batch 860/2304 [I 2022-10-27 00:09:29] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:29 [I 2022-10-27 00:09:29] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00245 [I 2022-10-27 00:09:29] ...trap/engines/engine.py.230: loss: 0.19184 [I 2022-10-27 00:09:29] ...optimizers/trijoint.py.151: total_norm: 15.836061 [I 2022-10-27 00:09:35] ...trap/engines/engine.py.224: train: epoch 37 | batch 870/2304 [I 2022-10-27 00:09:35] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:21 [I 2022-10-27 00:09:35] ...trap/engines/engine.py.229: process: 0.56734 | load: 0.00244 [I 2022-10-27 00:09:35] ...trap/engines/engine.py.230: loss: 0.17379 [I 2022-10-27 00:09:35] ...optimizers/trijoint.py.151: total_norm: 11.352807 [I 2022-10-27 00:09:40] ...trap/engines/engine.py.224: train: epoch 37 | batch 880/2304 [I 2022-10-27 00:09:40] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:16 [I 2022-10-27 00:09:40] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00248 [I 2022-10-27 00:09:40] ...trap/engines/engine.py.230: loss: 0.13867 [I 2022-10-27 00:09:40] ...optimizers/trijoint.py.151: total_norm: 10.259513 [I 2022-10-27 00:09:46] ...trap/engines/engine.py.224: train: epoch 37 | batch 890/2304 [I 2022-10-27 00:09:46] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:11 [I 2022-10-27 00:09:46] ...trap/engines/engine.py.229: process: 0.58014 | load: 0.00242 [I 2022-10-27 00:09:46] ...trap/engines/engine.py.230: loss: 0.15655 [I 2022-10-27 00:09:46] ...optimizers/trijoint.py.151: total_norm: 22.302771 [I 2022-10-27 00:09:52] ...trap/engines/engine.py.224: train: epoch 37 | batch 900/2304 [I 2022-10-27 00:09:52] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:12:58 [I 2022-10-27 00:09:52] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00244 [I 2022-10-27 00:09:52] ...trap/engines/engine.py.230: loss: 0.20394 [I 2022-10-27 00:09:52] ...optimizers/trijoint.py.151: total_norm: 15.091855 [I 2022-10-27 00:09:57] ...trap/engines/engine.py.224: train: epoch 37 | batch 910/2304 [I 2022-10-27 00:09:57] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:51 [I 2022-10-27 00:09:57] ...trap/engines/engine.py.229: process: 0.53968 | load: 0.00241 [I 2022-10-27 00:09:57] ...trap/engines/engine.py.230: loss: 0.13065 [I 2022-10-27 00:09:57] ...optimizers/trijoint.py.151: total_norm: 10.941700 [I 2022-10-27 00:10:03] ...trap/engines/engine.py.224: train: epoch 37 | batch 920/2304 [I 2022-10-27 00:10:03] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:46 [I 2022-10-27 00:10:03] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00241 [I 2022-10-27 00:10:03] ...trap/engines/engine.py.230: loss: 0.19175 [I 2022-10-27 00:10:03] ...optimizers/trijoint.py.151: total_norm: 20.341785 [I 2022-10-27 00:10:08] ...trap/engines/engine.py.224: train: epoch 37 | batch 930/2304 [I 2022-10-27 00:10:08] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:47 [I 2022-10-27 00:10:08] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00241 [I 2022-10-27 00:10:08] ...trap/engines/engine.py.230: loss: 0.10930 [I 2022-10-27 00:10:08] ...optimizers/trijoint.py.151: total_norm: 9.026203 [I 2022-10-27 00:10:14] ...trap/engines/engine.py.224: train: epoch 37 | batch 940/2304 [I 2022-10-27 00:10:14] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:42 [I 2022-10-27 00:10:14] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00246 [I 2022-10-27 00:10:14] ...trap/engines/engine.py.230: loss: 0.15337 [I 2022-10-27 00:10:14] ...optimizers/trijoint.py.151: total_norm: 22.350182 [I 2022-10-27 00:10:20] ...trap/engines/engine.py.224: train: epoch 37 | batch 950/2304 [I 2022-10-27 00:10:20] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:31 [I 2022-10-27 00:10:20] ...trap/engines/engine.py.229: process: 0.53905 | load: 0.00248 [I 2022-10-27 00:10:20] ...trap/engines/engine.py.230: loss: 0.21591 [I 2022-10-27 00:10:20] ...optimizers/trijoint.py.151: total_norm: 35.448956 [I 2022-10-27 00:10:25] ...trap/engines/engine.py.224: train: epoch 37 | batch 960/2304 [I 2022-10-27 00:10:25] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:27 [I 2022-10-27 00:10:25] ...trap/engines/engine.py.229: process: 0.55658 | load: 0.00240 [I 2022-10-27 00:10:25] ...trap/engines/engine.py.230: loss: 0.12398 [I 2022-10-27 00:10:25] ...optimizers/trijoint.py.151: total_norm: 11.098077 [I 2022-10-27 00:10:51] ...trap/engines/engine.py.224: train: epoch 37 | batch 970/2304 [I 2022-10-27 00:10:51] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:24:25 [I 2022-10-27 00:10:51] ...trap/engines/engine.py.229: process: 0.56292 | load: 0.00238 [I 2022-10-27 00:10:51] ...trap/engines/engine.py.230: loss: 0.16959 [I 2022-10-27 00:10:51] ...optimizers/trijoint.py.151: total_norm: 22.199808 [I 2022-10-27 00:10:56] ...trap/engines/engine.py.224: train: epoch 37 | batch 980/2304 [I 2022-10-27 00:10:56] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:13:27 [I 2022-10-27 00:10:56] ...trap/engines/engine.py.229: process: 0.55038 | load: 0.00239 [I 2022-10-27 00:10:56] ...trap/engines/engine.py.230: loss: 0.23413 [I 2022-10-27 00:10:56] ...optimizers/trijoint.py.151: total_norm: 19.557131 [I 2022-10-27 00:11:02] ...trap/engines/engine.py.224: train: epoch 37 | batch 990/2304 [I 2022-10-27 00:11:02] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:12:19 [I 2022-10-27 00:11:02] ...trap/engines/engine.py.229: process: 0.55175 | load: 0.00237 [I 2022-10-27 00:11:02] ...trap/engines/engine.py.230: loss: 0.28368 [I 2022-10-27 00:11:02] ...optimizers/trijoint.py.151: total_norm: 28.336264 [I 2022-10-27 00:11:08] ...trap/engines/engine.py.224: train: epoch 37 | batch 1000/2304 [I 2022-10-27 00:11:08] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:12:00 [I 2022-10-27 00:11:08] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00252 [I 2022-10-27 00:11:08] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-27 00:11:08] ...optimizers/trijoint.py.151: total_norm: 12.703121 [I 2022-10-27 00:11:13] ...trap/engines/engine.py.224: train: epoch 37 | batch 1010/2304 [I 2022-10-27 00:11:13] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:12:03 [I 2022-10-27 00:11:13] ...trap/engines/engine.py.229: process: 0.56419 | load: 0.00252 [I 2022-10-27 00:11:13] ...trap/engines/engine.py.230: loss: 0.22568 [I 2022-10-27 00:11:13] ...optimizers/trijoint.py.151: total_norm: 27.404558 [I 2022-10-27 00:11:19] ...trap/engines/engine.py.224: train: epoch 37 | batch 1020/2304 [I 2022-10-27 00:11:19] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:57 [I 2022-10-27 00:11:19] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00253 [I 2022-10-27 00:11:19] ...trap/engines/engine.py.230: loss: 0.21061 [I 2022-10-27 00:11:19] ...optimizers/trijoint.py.151: total_norm: 10.962254 [I 2022-10-27 00:11:24] ...trap/engines/engine.py.224: train: epoch 37 | batch 1030/2304 [I 2022-10-27 00:11:24] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:52 [I 2022-10-27 00:11:24] ...trap/engines/engine.py.229: process: 0.56720 | load: 0.00255 [I 2022-10-27 00:11:24] ...trap/engines/engine.py.230: loss: 0.11772 [I 2022-10-27 00:11:24] ...optimizers/trijoint.py.151: total_norm: 12.912547 [I 2022-10-27 00:11:30] ...trap/engines/engine.py.224: train: epoch 37 | batch 1040/2304 [I 2022-10-27 00:11:30] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:41 [I 2022-10-27 00:11:30] ...trap/engines/engine.py.229: process: 0.55289 | load: 0.00270 [I 2022-10-27 00:11:30] ...trap/engines/engine.py.230: loss: 0.21885 [I 2022-10-27 00:11:30] ...optimizers/trijoint.py.151: total_norm: 12.045242 [I 2022-10-27 00:11:35] ...trap/engines/engine.py.224: train: epoch 37 | batch 1050/2304 [I 2022-10-27 00:11:35] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:30 [I 2022-10-27 00:11:35] ...trap/engines/engine.py.229: process: 0.54061 | load: 0.00247 [I 2022-10-27 00:11:35] ...trap/engines/engine.py.230: loss: 0.16752 [I 2022-10-27 00:11:35] ...optimizers/trijoint.py.151: total_norm: 18.451229 [I 2022-10-27 00:11:41] ...trap/engines/engine.py.224: train: epoch 37 | batch 1060/2304 [I 2022-10-27 00:11:41] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:30 [I 2022-10-27 00:11:41] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00259 [I 2022-10-27 00:11:41] ...trap/engines/engine.py.230: loss: 0.13607 [I 2022-10-27 00:11:41] ...optimizers/trijoint.py.151: total_norm: 20.034142 [I 2022-10-27 00:11:47] ...trap/engines/engine.py.224: train: epoch 37 | batch 1070/2304 [I 2022-10-27 00:11:47] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:29 [I 2022-10-27 00:11:47] ...trap/engines/engine.py.229: process: 0.56653 | load: 0.00252 [I 2022-10-27 00:11:47] ...trap/engines/engine.py.230: loss: 0.19270 [I 2022-10-27 00:11:47] ...optimizers/trijoint.py.151: total_norm: 27.827673 [I 2022-10-27 00:11:52] ...trap/engines/engine.py.224: train: epoch 37 | batch 1080/2304 [I 2022-10-27 00:11:52] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:21 [I 2022-10-27 00:11:52] ...trap/engines/engine.py.229: process: 0.55264 | load: 0.00252 [I 2022-10-27 00:11:52] ...trap/engines/engine.py.230: loss: 0.15380 [I 2022-10-27 00:11:52] ...optimizers/trijoint.py.151: total_norm: 18.197912 [I 2022-10-27 00:11:58] ...trap/engines/engine.py.224: train: epoch 37 | batch 1090/2304 [I 2022-10-27 00:11:58] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:15 [I 2022-10-27 00:11:58] ...trap/engines/engine.py.229: process: 0.55312 | load: 0.00263 [I 2022-10-27 00:11:58] ...trap/engines/engine.py.230: loss: 0.27962 [I 2022-10-27 00:11:58] ...optimizers/trijoint.py.151: total_norm: 29.514633 [I 2022-10-27 00:12:03] ...trap/engines/engine.py.224: train: epoch 37 | batch 1100/2304 [I 2022-10-27 00:12:03] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:09 [I 2022-10-27 00:12:03] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00247 [I 2022-10-27 00:12:03] ...trap/engines/engine.py.230: loss: 0.18974 [I 2022-10-27 00:12:03] ...optimizers/trijoint.py.151: total_norm: 22.587671 [I 2022-10-27 00:12:09] ...trap/engines/engine.py.224: train: epoch 37 | batch 1110/2304 [I 2022-10-27 00:12:09] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:11:02 [I 2022-10-27 00:12:09] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00254 [I 2022-10-27 00:12:09] ...trap/engines/engine.py.230: loss: 0.15277 [I 2022-10-27 00:12:09] ...optimizers/trijoint.py.151: total_norm: 10.668144 [I 2022-10-27 00:12:14] ...trap/engines/engine.py.224: train: epoch 37 | batch 1120/2304 [I 2022-10-27 00:12:14] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:54 [I 2022-10-27 00:12:14] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00246 [I 2022-10-27 00:12:14] ...trap/engines/engine.py.230: loss: 0.15635 [I 2022-10-27 00:12:14] ...optimizers/trijoint.py.151: total_norm: 19.697767 [I 2022-10-27 00:12:20] ...trap/engines/engine.py.224: train: epoch 37 | batch 1130/2304 [I 2022-10-27 00:12:20] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:52 [I 2022-10-27 00:12:20] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00253 [I 2022-10-27 00:12:20] ...trap/engines/engine.py.230: loss: 0.20338 [I 2022-10-27 00:12:20] ...optimizers/trijoint.py.151: total_norm: 26.975605 [I 2022-10-27 00:12:26] ...trap/engines/engine.py.224: train: epoch 37 | batch 1140/2304 [I 2022-10-27 00:12:26] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:47 [I 2022-10-27 00:12:26] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00248 [I 2022-10-27 00:12:26] ...trap/engines/engine.py.230: loss: 0.15778 [I 2022-10-27 00:12:26] ...optimizers/trijoint.py.151: total_norm: 18.826702 [I 2022-10-27 00:12:31] ...trap/engines/engine.py.224: train: epoch 37 | batch 1150/2304 [I 2022-10-27 00:12:31] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:41 [I 2022-10-27 00:12:31] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00251 [I 2022-10-27 00:12:31] ...trap/engines/engine.py.230: loss: 0.20596 [I 2022-10-27 00:12:31] ...optimizers/trijoint.py.151: total_norm: 41.691212 [I 2022-10-27 00:12:37] ...trap/engines/engine.py.224: train: epoch 37 | batch 1160/2304 [I 2022-10-27 00:12:37] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:35 [I 2022-10-27 00:12:37] ...trap/engines/engine.py.229: process: 0.54682 | load: 0.00257 [I 2022-10-27 00:12:37] ...trap/engines/engine.py.230: loss: 0.28329 [I 2022-10-27 00:12:37] ...optimizers/trijoint.py.151: total_norm: 27.984936 [I 2022-10-27 00:12:42] ...trap/engines/engine.py.224: train: epoch 37 | batch 1170/2304 [I 2022-10-27 00:12:42] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:28 [I 2022-10-27 00:12:42] ...trap/engines/engine.py.229: process: 0.54780 | load: 0.00249 [I 2022-10-27 00:12:42] ...trap/engines/engine.py.230: loss: 0.21237 [I 2022-10-27 00:12:42] ...optimizers/trijoint.py.151: total_norm: 33.882160 [I 2022-10-27 00:12:48] ...trap/engines/engine.py.224: train: epoch 37 | batch 1180/2304 [I 2022-10-27 00:12:48] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:28 [I 2022-10-27 00:12:48] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00250 [I 2022-10-27 00:12:48] ...trap/engines/engine.py.230: loss: 0.14586 [I 2022-10-27 00:12:48] ...optimizers/trijoint.py.151: total_norm: 18.389240 [I 2022-10-27 00:12:53] ...trap/engines/engine.py.224: train: epoch 37 | batch 1190/2304 [I 2022-10-27 00:12:53] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:15 [I 2022-10-27 00:12:53] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00253 [I 2022-10-27 00:12:53] ...trap/engines/engine.py.230: loss: 0.15787 [I 2022-10-27 00:12:53] ...optimizers/trijoint.py.151: total_norm: 29.129711 [I 2022-10-27 00:12:59] ...trap/engines/engine.py.224: train: epoch 37 | batch 1200/2304 [I 2022-10-27 00:12:59] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:15 [I 2022-10-27 00:12:59] ...trap/engines/engine.py.229: process: 0.54671 | load: 0.00251 [I 2022-10-27 00:12:59] ...trap/engines/engine.py.230: loss: 0.14612 [I 2022-10-27 00:12:59] ...optimizers/trijoint.py.151: total_norm: 16.109232 [I 2022-10-27 00:13:05] ...trap/engines/engine.py.224: train: epoch 37 | batch 1210/2304 [I 2022-10-27 00:13:05] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:10:14 [I 2022-10-27 00:13:05] ...trap/engines/engine.py.229: process: 0.57315 | load: 0.00251 [I 2022-10-27 00:13:05] ...trap/engines/engine.py.230: loss: 0.19442 [I 2022-10-27 00:13:05] ...optimizers/trijoint.py.151: total_norm: 18.449373 [I 2022-10-27 00:13:10] ...trap/engines/engine.py.224: train: epoch 37 | batch 1220/2304 [I 2022-10-27 00:13:10] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:10:02 [I 2022-10-27 00:13:10] ...trap/engines/engine.py.229: process: 0.55703 | load: 0.00260 [I 2022-10-27 00:13:10] ...trap/engines/engine.py.230: loss: 0.17706 [I 2022-10-27 00:13:10] ...optimizers/trijoint.py.151: total_norm: 13.808333 [I 2022-10-27 00:13:16] ...trap/engines/engine.py.224: train: epoch 37 | batch 1230/2304 [I 2022-10-27 00:13:16] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:59 [I 2022-10-27 00:13:16] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00272 [I 2022-10-27 00:13:16] ...trap/engines/engine.py.230: loss: 0.21965 [I 2022-10-27 00:13:16] ...optimizers/trijoint.py.151: total_norm: 30.257425 [I 2022-10-27 00:13:21] ...trap/engines/engine.py.224: train: epoch 37 | batch 1240/2304 [I 2022-10-27 00:13:21] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:52 [I 2022-10-27 00:13:21] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00255 [I 2022-10-27 00:13:21] ...trap/engines/engine.py.230: loss: 0.16163 [I 2022-10-27 00:13:21] ...optimizers/trijoint.py.151: total_norm: 17.546383 [I 2022-10-27 00:13:27] ...trap/engines/engine.py.224: train: epoch 37 | batch 1250/2304 [I 2022-10-27 00:13:27] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:46 [I 2022-10-27 00:13:27] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00248 [I 2022-10-27 00:13:27] ...trap/engines/engine.py.230: loss: 0.17850 [I 2022-10-27 00:13:27] ...optimizers/trijoint.py.151: total_norm: 21.066706 [I 2022-10-27 00:13:32] ...trap/engines/engine.py.224: train: epoch 37 | batch 1260/2304 [I 2022-10-27 00:13:32] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:36 [I 2022-10-27 00:13:32] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00248 [I 2022-10-27 00:13:32] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-27 00:13:32] ...optimizers/trijoint.py.151: total_norm: 22.702171 [I 2022-10-27 00:13:38] ...trap/engines/engine.py.224: train: epoch 37 | batch 1270/2304 [I 2022-10-27 00:13:38] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:35 [I 2022-10-27 00:13:38] ...trap/engines/engine.py.229: process: 0.54277 | load: 0.00250 [I 2022-10-27 00:13:38] ...trap/engines/engine.py.230: loss: 0.22006 [I 2022-10-27 00:13:38] ...optimizers/trijoint.py.151: total_norm: 22.042053 [I 2022-10-27 00:13:44] ...trap/engines/engine.py.224: train: epoch 37 | batch 1280/2304 [I 2022-10-27 00:13:44] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:28 [I 2022-10-27 00:13:44] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00248 [I 2022-10-27 00:13:44] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-27 00:13:44] ...optimizers/trijoint.py.151: total_norm: 20.948257 [I 2022-10-27 00:13:49] ...trap/engines/engine.py.224: train: epoch 37 | batch 1290/2304 [I 2022-10-27 00:13:49] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:28 [I 2022-10-27 00:13:49] ...trap/engines/engine.py.229: process: 0.56912 | load: 0.00252 [I 2022-10-27 00:13:49] ...trap/engines/engine.py.230: loss: 0.16925 [I 2022-10-27 00:13:49] ...optimizers/trijoint.py.151: total_norm: 18.841003 [I 2022-10-27 00:13:55] ...trap/engines/engine.py.224: train: epoch 37 | batch 1300/2304 [I 2022-10-27 00:13:55] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:16 [I 2022-10-27 00:13:55] ...trap/engines/engine.py.229: process: 0.55923 | load: 0.00253 [I 2022-10-27 00:13:55] ...trap/engines/engine.py.230: loss: 0.14276 [I 2022-10-27 00:13:55] ...optimizers/trijoint.py.151: total_norm: 14.094179 [I 2022-10-27 00:14:00] ...trap/engines/engine.py.224: train: epoch 37 | batch 1310/2304 [I 2022-10-27 00:14:00] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:09:13 [I 2022-10-27 00:14:00] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00254 [I 2022-10-27 00:14:00] ...trap/engines/engine.py.230: loss: 0.20842 [I 2022-10-27 00:14:00] ...optimizers/trijoint.py.151: total_norm: 29.057381 [I 2022-10-27 00:14:06] ...trap/engines/engine.py.224: train: epoch 37 | batch 1320/2304 [I 2022-10-27 00:14:06] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:10 [I 2022-10-27 00:14:06] ...trap/engines/engine.py.229: process: 0.56561 | load: 0.00250 [I 2022-10-27 00:14:06] ...trap/engines/engine.py.230: loss: 0.23075 [I 2022-10-27 00:14:06] ...optimizers/trijoint.py.151: total_norm: 18.825039 [I 2022-10-27 00:14:12] ...trap/engines/engine.py.224: train: epoch 37 | batch 1330/2304 [I 2022-10-27 00:14:12] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:09:04 [I 2022-10-27 00:14:12] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00258 [I 2022-10-27 00:14:12] ...trap/engines/engine.py.230: loss: 0.17787 [I 2022-10-27 00:14:12] ...optimizers/trijoint.py.151: total_norm: 27.399401 [I 2022-10-27 00:14:17] ...trap/engines/engine.py.224: train: epoch 37 | batch 1340/2304 [I 2022-10-27 00:14:17] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:56 [I 2022-10-27 00:14:17] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00256 [I 2022-10-27 00:14:17] ...trap/engines/engine.py.230: loss: 0.11706 [I 2022-10-27 00:14:17] ...optimizers/trijoint.py.151: total_norm: 14.017277 [I 2022-10-27 00:14:23] ...trap/engines/engine.py.224: train: epoch 37 | batch 1350/2304 [I 2022-10-27 00:14:23] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:46 [I 2022-10-27 00:14:23] ...trap/engines/engine.py.229: process: 0.54653 | load: 0.00255 [I 2022-10-27 00:14:23] ...trap/engines/engine.py.230: loss: 0.28713 [I 2022-10-27 00:14:23] ...optimizers/trijoint.py.151: total_norm: 24.937468 [I 2022-10-27 00:14:28] ...trap/engines/engine.py.224: train: epoch 37 | batch 1360/2304 [I 2022-10-27 00:14:28] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:50 [I 2022-10-27 00:14:28] ...trap/engines/engine.py.229: process: 0.56748 | load: 0.00254 [I 2022-10-27 00:14:28] ...trap/engines/engine.py.230: loss: 0.18351 [I 2022-10-27 00:14:28] ...optimizers/trijoint.py.151: total_norm: 24.220514 [I 2022-10-27 00:14:34] ...trap/engines/engine.py.224: train: epoch 37 | batch 1370/2304 [I 2022-10-27 00:14:34] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:39 [I 2022-10-27 00:14:34] ...trap/engines/engine.py.229: process: 0.55637 | load: 0.00257 [I 2022-10-27 00:14:34] ...trap/engines/engine.py.230: loss: 0.22884 [I 2022-10-27 00:14:34] ...optimizers/trijoint.py.151: total_norm: 29.481979 [I 2022-10-27 00:14:39] ...trap/engines/engine.py.224: train: epoch 37 | batch 1380/2304 [I 2022-10-27 00:14:39] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:33 [I 2022-10-27 00:14:39] ...trap/engines/engine.py.229: process: 0.54909 | load: 0.00247 [I 2022-10-27 00:14:39] ...trap/engines/engine.py.230: loss: 0.12235 [I 2022-10-27 00:14:39] ...optimizers/trijoint.py.151: total_norm: 15.278639 [I 2022-10-27 00:14:45] ...trap/engines/engine.py.224: train: epoch 37 | batch 1390/2304 [I 2022-10-27 00:14:45] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:28 [I 2022-10-27 00:14:45] ...trap/engines/engine.py.229: process: 0.55932 | load: 0.00254 [I 2022-10-27 00:14:45] ...trap/engines/engine.py.230: loss: 0.19211 [I 2022-10-27 00:14:45] ...optimizers/trijoint.py.151: total_norm: 16.459862 [I 2022-10-27 00:14:50] ...trap/engines/engine.py.224: train: epoch 37 | batch 1400/2304 [I 2022-10-27 00:14:50] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:21 [I 2022-10-27 00:14:50] ...trap/engines/engine.py.229: process: 0.54777 | load: 0.00247 [I 2022-10-27 00:14:50] ...trap/engines/engine.py.230: loss: 0.17871 [I 2022-10-27 00:14:50] ...optimizers/trijoint.py.151: total_norm: 27.153738 [I 2022-10-27 00:14:56] ...trap/engines/engine.py.224: train: epoch 37 | batch 1410/2304 [I 2022-10-27 00:14:56] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:20 [I 2022-10-27 00:14:56] ...trap/engines/engine.py.229: process: 0.57308 | load: 0.00253 [I 2022-10-27 00:14:56] ...trap/engines/engine.py.230: loss: 0.19542 [I 2022-10-27 00:14:56] ...optimizers/trijoint.py.151: total_norm: 14.356398 [I 2022-10-27 00:15:02] ...trap/engines/engine.py.224: train: epoch 37 | batch 1420/2304 [I 2022-10-27 00:15:02] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:08:14 [I 2022-10-27 00:15:02] ...trap/engines/engine.py.229: process: 0.56682 | load: 0.00255 [I 2022-10-27 00:15:02] ...trap/engines/engine.py.230: loss: 0.18877 [I 2022-10-27 00:15:02] ...optimizers/trijoint.py.151: total_norm: 32.197220 [I 2022-10-27 00:15:07] ...trap/engines/engine.py.224: train: epoch 37 | batch 1430/2304 [I 2022-10-27 00:15:07] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:08:04 [I 2022-10-27 00:15:07] ...trap/engines/engine.py.229: process: 0.53932 | load: 0.00246 [I 2022-10-27 00:15:07] ...trap/engines/engine.py.230: loss: 0.19395 [I 2022-10-27 00:15:07] ...optimizers/trijoint.py.151: total_norm: 13.114161 [I 2022-10-27 00:15:13] ...trap/engines/engine.py.224: train: epoch 37 | batch 1440/2304 [I 2022-10-27 00:15:13] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:07:57 [I 2022-10-27 00:15:13] ...trap/engines/engine.py.229: process: 0.54353 | load: 0.00249 [I 2022-10-27 00:15:13] ...trap/engines/engine.py.230: loss: 0.15082 [I 2022-10-27 00:15:13] ...optimizers/trijoint.py.151: total_norm: 19.893032 [I 2022-10-27 00:15:18] ...trap/engines/engine.py.224: train: epoch 37 | batch 1450/2304 [I 2022-10-27 00:15:18] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:54 [I 2022-10-27 00:15:18] ...trap/engines/engine.py.229: process: 0.55130 | load: 0.00245 [I 2022-10-27 00:15:18] ...trap/engines/engine.py.230: loss: 0.14089 [I 2022-10-27 00:15:18] ...optimizers/trijoint.py.151: total_norm: 9.780064 [I 2022-10-27 00:15:24] ...trap/engines/engine.py.224: train: epoch 37 | batch 1460/2304 [I 2022-10-27 00:15:24] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:50 [I 2022-10-27 00:15:24] ...trap/engines/engine.py.229: process: 0.55848 | load: 0.00253 [I 2022-10-27 00:15:24] ...trap/engines/engine.py.230: loss: 0.11400 [I 2022-10-27 00:15:24] ...optimizers/trijoint.py.151: total_norm: 10.335748 [I 2022-10-27 00:15:29] ...trap/engines/engine.py.224: train: epoch 37 | batch 1470/2304 [I 2022-10-27 00:15:29] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:07:46 [I 2022-10-27 00:15:29] ...trap/engines/engine.py.229: process: 0.57459 | load: 0.00250 [I 2022-10-27 00:15:29] ...trap/engines/engine.py.230: loss: 0.14905 [I 2022-10-27 00:15:29] ...optimizers/trijoint.py.151: total_norm: 14.097787 [I 2022-10-27 00:15:35] ...trap/engines/engine.py.224: train: epoch 37 | batch 1480/2304 [I 2022-10-27 00:15:35] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:39 [I 2022-10-27 00:15:35] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00253 [I 2022-10-27 00:15:35] ...trap/engines/engine.py.230: loss: 0.11926 [I 2022-10-27 00:15:35] ...optimizers/trijoint.py.151: total_norm: 16.729074 [I 2022-10-27 00:15:41] ...trap/engines/engine.py.224: train: epoch 37 | batch 1490/2304 [I 2022-10-27 00:15:41] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:07:34 [I 2022-10-27 00:15:41] ...trap/engines/engine.py.229: process: 0.55694 | load: 0.00253 [I 2022-10-27 00:15:41] ...trap/engines/engine.py.230: loss: 0.13878 [I 2022-10-27 00:15:41] ...optimizers/trijoint.py.151: total_norm: 13.893117 [I 2022-10-27 00:15:46] ...trap/engines/engine.py.224: train: epoch 37 | batch 1500/2304 [I 2022-10-27 00:15:46] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:45 [I 2022-10-27 00:15:46] ...trap/engines/engine.py.229: process: 0.67476 | load: 0.00253 [I 2022-10-27 00:15:46] ...trap/engines/engine.py.230: loss: 0.14593 [I 2022-10-27 00:15:46] ...optimizers/trijoint.py.151: total_norm: 30.623024 [I 2022-10-27 00:15:52] ...trap/engines/engine.py.224: train: epoch 37 | batch 1510/2304 [I 2022-10-27 00:15:52] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:21 [I 2022-10-27 00:15:52] ...trap/engines/engine.py.229: process: 0.54655 | load: 0.00249 [I 2022-10-27 00:15:52] ...trap/engines/engine.py.230: loss: 0.18321 [I 2022-10-27 00:15:52] ...optimizers/trijoint.py.151: total_norm: 17.782049 [I 2022-10-27 00:15:58] ...trap/engines/engine.py.224: train: epoch 37 | batch 1520/2304 [I 2022-10-27 00:15:58] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:07:16 [I 2022-10-27 00:15:58] ...trap/engines/engine.py.229: process: 0.55378 | load: 0.00251 [I 2022-10-27 00:15:58] ...trap/engines/engine.py.230: loss: 0.25063 [I 2022-10-27 00:15:58] ...optimizers/trijoint.py.151: total_norm: 21.971859 [I 2022-10-27 00:16:03] ...trap/engines/engine.py.224: train: epoch 37 | batch 1530/2304 [I 2022-10-27 00:16:03] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:07:13 [I 2022-10-27 00:16:03] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00252 [I 2022-10-27 00:16:03] ...trap/engines/engine.py.230: loss: 0.16479 [I 2022-10-27 00:16:03] ...optimizers/trijoint.py.151: total_norm: 15.261612 [I 2022-10-27 00:16:09] ...trap/engines/engine.py.224: train: epoch 37 | batch 1540/2304 [I 2022-10-27 00:16:09] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:07:06 [I 2022-10-27 00:16:09] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00255 [I 2022-10-27 00:16:09] ...trap/engines/engine.py.230: loss: 0.17742 [I 2022-10-27 00:16:09] ...optimizers/trijoint.py.151: total_norm: 16.767952 [I 2022-10-27 00:16:14] ...trap/engines/engine.py.224: train: epoch 37 | batch 1550/2304 [I 2022-10-27 00:16:14] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:06:59 [I 2022-10-27 00:16:14] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00252 [I 2022-10-27 00:16:14] ...trap/engines/engine.py.230: loss: 0.13897 [I 2022-10-27 00:16:14] ...optimizers/trijoint.py.151: total_norm: 13.426669 [I 2022-10-27 00:16:20] ...trap/engines/engine.py.224: train: epoch 37 | batch 1560/2304 [I 2022-10-27 00:16:20] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:53 [I 2022-10-27 00:16:20] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00254 [I 2022-10-27 00:16:20] ...trap/engines/engine.py.230: loss: 0.27701 [I 2022-10-27 00:16:20] ...optimizers/trijoint.py.151: total_norm: 16.276163 [I 2022-10-27 00:16:25] ...trap/engines/engine.py.224: train: epoch 37 | batch 1570/2304 [I 2022-10-27 00:16:25] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:47 [I 2022-10-27 00:16:25] ...trap/engines/engine.py.229: process: 0.55241 | load: 0.00265 [I 2022-10-27 00:16:25] ...trap/engines/engine.py.230: loss: 0.20347 [I 2022-10-27 00:16:25] ...optimizers/trijoint.py.151: total_norm: 21.546770 [I 2022-10-27 00:16:31] ...trap/engines/engine.py.224: train: epoch 37 | batch 1580/2304 [I 2022-10-27 00:16:31] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:45 [I 2022-10-27 00:16:31] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00248 [I 2022-10-27 00:16:31] ...trap/engines/engine.py.230: loss: 0.17271 [I 2022-10-27 00:16:31] ...optimizers/trijoint.py.151: total_norm: 24.239510 [I 2022-10-27 00:16:37] ...trap/engines/engine.py.224: train: epoch 37 | batch 1590/2304 [I 2022-10-27 00:16:37] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:38 [I 2022-10-27 00:16:37] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00250 [I 2022-10-27 00:16:37] ...trap/engines/engine.py.230: loss: 0.14139 [I 2022-10-27 00:16:37] ...optimizers/trijoint.py.151: total_norm: 17.726404 [I 2022-10-27 00:16:42] ...trap/engines/engine.py.224: train: epoch 37 | batch 1600/2304 [I 2022-10-27 00:16:42] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:31 [I 2022-10-27 00:16:42] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00250 [I 2022-10-27 00:16:42] ...trap/engines/engine.py.230: loss: 0.16615 [I 2022-10-27 00:16:42] ...optimizers/trijoint.py.151: total_norm: 19.769592 [I 2022-10-27 00:16:48] ...trap/engines/engine.py.224: train: epoch 37 | batch 1610/2304 [I 2022-10-27 00:16:48] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:27 [I 2022-10-27 00:16:48] ...trap/engines/engine.py.229: process: 0.55010 | load: 0.00254 [I 2022-10-27 00:16:48] ...trap/engines/engine.py.230: loss: 0.14918 [I 2022-10-27 00:16:48] ...optimizers/trijoint.py.151: total_norm: 24.257370 [I 2022-10-27 00:16:53] ...trap/engines/engine.py.224: train: epoch 37 | batch 1620/2304 [I 2022-10-27 00:16:53] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:17 [I 2022-10-27 00:16:53] ...trap/engines/engine.py.229: process: 0.54632 | load: 0.00249 [I 2022-10-27 00:16:53] ...trap/engines/engine.py.230: loss: 0.15031 [I 2022-10-27 00:16:53] ...optimizers/trijoint.py.151: total_norm: 8.098861 [I 2022-10-27 00:16:59] ...trap/engines/engine.py.224: train: epoch 37 | batch 1630/2304 [I 2022-10-27 00:16:59] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:14 [I 2022-10-27 00:16:59] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00259 [I 2022-10-27 00:16:59] ...trap/engines/engine.py.230: loss: 0.16126 [I 2022-10-27 00:16:59] ...optimizers/trijoint.py.151: total_norm: 14.398198 [I 2022-10-27 00:17:05] ...trap/engines/engine.py.224: train: epoch 37 | batch 1640/2304 [I 2022-10-27 00:17:05] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:06:13 [I 2022-10-27 00:17:05] ...trap/engines/engine.py.229: process: 0.56521 | load: 0.00259 [I 2022-10-27 00:17:05] ...trap/engines/engine.py.230: loss: 0.13512 [I 2022-10-27 00:17:05] ...optimizers/trijoint.py.151: total_norm: 14.816310 [I 2022-10-27 00:17:10] ...trap/engines/engine.py.224: train: epoch 37 | batch 1650/2304 [I 2022-10-27 00:17:10] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:06:04 [I 2022-10-27 00:17:10] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00254 [I 2022-10-27 00:17:10] ...trap/engines/engine.py.230: loss: 0.20813 [I 2022-10-27 00:17:10] ...optimizers/trijoint.py.151: total_norm: 11.335582 [I 2022-10-27 00:17:16] ...trap/engines/engine.py.224: train: epoch 37 | batch 1660/2304 [I 2022-10-27 00:17:16] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:57 [I 2022-10-27 00:17:16] ...trap/engines/engine.py.229: process: 0.55892 | load: 0.00252 [I 2022-10-27 00:17:16] ...trap/engines/engine.py.230: loss: 0.17521 [I 2022-10-27 00:17:16] ...optimizers/trijoint.py.151: total_norm: 27.574245 [I 2022-10-27 00:17:21] ...trap/engines/engine.py.224: train: epoch 37 | batch 1670/2304 [I 2022-10-27 00:17:21] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:53 [I 2022-10-27 00:17:21] ...trap/engines/engine.py.229: process: 0.55552 | load: 0.00248 [I 2022-10-27 00:17:21] ...trap/engines/engine.py.230: loss: 0.11975 [I 2022-10-27 00:17:21] ...optimizers/trijoint.py.151: total_norm: 11.279840 [I 2022-10-27 00:17:27] ...trap/engines/engine.py.224: train: epoch 37 | batch 1680/2304 [I 2022-10-27 00:17:27] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:47 [I 2022-10-27 00:17:27] ...trap/engines/engine.py.229: process: 0.56445 | load: 0.00254 [I 2022-10-27 00:17:27] ...trap/engines/engine.py.230: loss: 0.15654 [I 2022-10-27 00:17:27] ...optimizers/trijoint.py.151: total_norm: 14.064058 [I 2022-10-27 00:17:32] ...trap/engines/engine.py.224: train: epoch 37 | batch 1690/2304 [I 2022-10-27 00:17:32] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:43 [I 2022-10-27 00:17:32] ...trap/engines/engine.py.229: process: 0.56741 | load: 0.00257 [I 2022-10-27 00:17:32] ...trap/engines/engine.py.230: loss: 0.22672 [I 2022-10-27 00:17:32] ...optimizers/trijoint.py.151: total_norm: 27.313171 [I 2022-10-27 00:17:38] ...trap/engines/engine.py.224: train: epoch 37 | batch 1700/2304 [I 2022-10-27 00:17:38] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:54 [I 2022-10-27 00:17:38] ...trap/engines/engine.py.229: process: 0.62690 | load: 0.00279 [I 2022-10-27 00:17:38] ...trap/engines/engine.py.230: loss: 0.23229 [I 2022-10-27 00:17:38] ...optimizers/trijoint.py.151: total_norm: 15.810428 [I 2022-10-27 00:17:44] ...trap/engines/engine.py.224: train: epoch 37 | batch 1710/2304 [I 2022-10-27 00:17:44] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:33 [I 2022-10-27 00:17:44] ...trap/engines/engine.py.229: process: 0.56284 | load: 0.00251 [I 2022-10-27 00:17:44] ...trap/engines/engine.py.230: loss: 0.18504 [I 2022-10-27 00:17:44] ...optimizers/trijoint.py.151: total_norm: 20.267042 [I 2022-10-27 00:17:49] ...trap/engines/engine.py.224: train: epoch 37 | batch 1720/2304 [I 2022-10-27 00:17:49] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:23 [I 2022-10-27 00:17:49] ...trap/engines/engine.py.229: process: 0.54657 | load: 0.00258 [I 2022-10-27 00:17:49] ...trap/engines/engine.py.230: loss: 0.17177 [I 2022-10-27 00:17:49] ...optimizers/trijoint.py.151: total_norm: 14.301290 [I 2022-10-27 00:17:55] ...trap/engines/engine.py.224: train: epoch 37 | batch 1730/2304 [I 2022-10-27 00:17:55] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:18 [I 2022-10-27 00:17:55] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00257 [I 2022-10-27 00:17:55] ...trap/engines/engine.py.230: loss: 0.20613 [I 2022-10-27 00:17:55] ...optimizers/trijoint.py.151: total_norm: 22.636379 [I 2022-10-27 00:18:01] ...trap/engines/engine.py.224: train: epoch 37 | batch 1740/2304 [I 2022-10-27 00:18:01] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:05:12 [I 2022-10-27 00:18:01] ...trap/engines/engine.py.229: process: 0.55138 | load: 0.00251 [I 2022-10-27 00:18:01] ...trap/engines/engine.py.230: loss: 0.21195 [I 2022-10-27 00:18:01] ...optimizers/trijoint.py.151: total_norm: 33.814045 [I 2022-10-27 00:18:06] ...trap/engines/engine.py.224: train: epoch 37 | batch 1750/2304 [I 2022-10-27 00:18:06] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:05:06 [I 2022-10-27 00:18:06] ...trap/engines/engine.py.229: process: 0.56156 | load: 0.00256 [I 2022-10-27 00:18:06] ...trap/engines/engine.py.230: loss: 0.15544 [I 2022-10-27 00:18:06] ...optimizers/trijoint.py.151: total_norm: 18.674740 [I 2022-10-27 00:18:12] ...trap/engines/engine.py.224: train: epoch 37 | batch 1760/2304 [I 2022-10-27 00:18:12] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:05:02 [I 2022-10-27 00:18:12] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00250 [I 2022-10-27 00:18:12] ...trap/engines/engine.py.230: loss: 0.19703 [I 2022-10-27 00:18:12] ...optimizers/trijoint.py.151: total_norm: 18.393158 [I 2022-10-27 00:18:17] ...trap/engines/engine.py.224: train: epoch 37 | batch 1770/2304 [I 2022-10-27 00:18:17] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:05:01 [I 2022-10-27 00:18:17] ...trap/engines/engine.py.229: process: 0.55945 | load: 0.00276 [I 2022-10-27 00:18:17] ...trap/engines/engine.py.230: loss: 0.14129 [I 2022-10-27 00:18:17] ...optimizers/trijoint.py.151: total_norm: 13.529289 [I 2022-10-27 00:18:23] ...trap/engines/engine.py.224: train: epoch 37 | batch 1780/2304 [I 2022-10-27 00:18:23] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:54 [I 2022-10-27 00:18:23] ...trap/engines/engine.py.229: process: 0.56881 | load: 0.00251 [I 2022-10-27 00:18:23] ...trap/engines/engine.py.230: loss: 0.13330 [I 2022-10-27 00:18:23] ...optimizers/trijoint.py.151: total_norm: 13.752542 [I 2022-10-27 00:18:29] ...trap/engines/engine.py.224: train: epoch 37 | batch 1790/2304 [I 2022-10-27 00:18:29] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:46 [I 2022-10-27 00:18:29] ...trap/engines/engine.py.229: process: 0.54922 | load: 0.00265 [I 2022-10-27 00:18:29] ...trap/engines/engine.py.230: loss: 0.15761 [I 2022-10-27 00:18:29] ...optimizers/trijoint.py.151: total_norm: 24.740604 [I 2022-10-27 00:18:34] ...trap/engines/engine.py.224: train: epoch 37 | batch 1800/2304 [I 2022-10-27 00:18:34] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:41 [I 2022-10-27 00:18:34] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00260 [I 2022-10-27 00:18:34] ...trap/engines/engine.py.230: loss: 0.16815 [I 2022-10-27 00:18:34] ...optimizers/trijoint.py.151: total_norm: 17.967821 [I 2022-10-27 00:18:40] ...trap/engines/engine.py.224: train: epoch 37 | batch 1810/2304 [I 2022-10-27 00:18:40] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:36 [I 2022-10-27 00:18:40] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00303 [I 2022-10-27 00:18:40] ...trap/engines/engine.py.230: loss: 0.15811 [I 2022-10-27 00:18:40] ...optimizers/trijoint.py.151: total_norm: 10.738526 [I 2022-10-27 00:18:45] ...trap/engines/engine.py.224: train: epoch 37 | batch 1820/2304 [I 2022-10-27 00:18:45] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:30 [I 2022-10-27 00:18:45] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00274 [I 2022-10-27 00:18:45] ...trap/engines/engine.py.230: loss: 0.16802 [I 2022-10-27 00:18:45] ...optimizers/trijoint.py.151: total_norm: 9.303673 [I 2022-10-27 00:18:51] ...trap/engines/engine.py.224: train: epoch 37 | batch 1830/2304 [I 2022-10-27 00:18:51] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:27 [I 2022-10-27 00:18:51] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00262 [I 2022-10-27 00:18:51] ...trap/engines/engine.py.230: loss: 0.10224 [I 2022-10-27 00:18:51] ...optimizers/trijoint.py.151: total_norm: 9.450618 [I 2022-10-27 00:18:57] ...trap/engines/engine.py.224: train: epoch 37 | batch 1840/2304 [I 2022-10-27 00:18:57] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:18 [I 2022-10-27 00:18:57] ...trap/engines/engine.py.229: process: 0.55657 | load: 0.00256 [I 2022-10-27 00:18:57] ...trap/engines/engine.py.230: loss: 0.17707 [I 2022-10-27 00:18:57] ...optimizers/trijoint.py.151: total_norm: 13.436199 [I 2022-10-27 00:19:02] ...trap/engines/engine.py.224: train: epoch 37 | batch 1850/2304 [I 2022-10-27 00:19:02] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:04:13 [I 2022-10-27 00:19:02] ...trap/engines/engine.py.229: process: 0.55924 | load: 0.00258 [I 2022-10-27 00:19:02] ...trap/engines/engine.py.230: loss: 0.17498 [I 2022-10-27 00:19:02] ...optimizers/trijoint.py.151: total_norm: 25.366348 [I 2022-10-27 00:19:08] ...trap/engines/engine.py.224: train: epoch 37 | batch 1860/2304 [I 2022-10-27 00:19:08] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:04:07 [I 2022-10-27 00:19:08] ...trap/engines/engine.py.229: process: 0.55506 | load: 0.00257 [I 2022-10-27 00:19:08] ...trap/engines/engine.py.230: loss: 0.16426 [I 2022-10-27 00:19:08] ...optimizers/trijoint.py.151: total_norm: 19.553343 [I 2022-10-27 00:19:13] ...trap/engines/engine.py.224: train: epoch 37 | batch 1870/2304 [I 2022-10-27 00:19:13] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:04:00 [I 2022-10-27 00:19:13] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00256 [I 2022-10-27 00:19:13] ...trap/engines/engine.py.230: loss: 0.12716 [I 2022-10-27 00:19:13] ...optimizers/trijoint.py.151: total_norm: 16.060652 [I 2022-10-27 00:19:19] ...trap/engines/engine.py.224: train: epoch 37 | batch 1880/2304 [I 2022-10-27 00:19:19] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:56 [I 2022-10-27 00:19:19] ...trap/engines/engine.py.229: process: 0.55437 | load: 0.00250 [I 2022-10-27 00:19:19] ...trap/engines/engine.py.230: loss: 0.17017 [I 2022-10-27 00:19:19] ...optimizers/trijoint.py.151: total_norm: 13.243914 [I 2022-10-27 00:19:25] ...trap/engines/engine.py.224: train: epoch 37 | batch 1890/2304 [I 2022-10-27 00:19:25] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:58 [I 2022-10-27 00:19:25] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00275 [I 2022-10-27 00:19:25] ...trap/engines/engine.py.230: loss: 0.14813 [I 2022-10-27 00:19:25] ...optimizers/trijoint.py.151: total_norm: 28.014462 [I 2022-10-27 00:19:30] ...trap/engines/engine.py.224: train: epoch 37 | batch 1900/2304 [I 2022-10-27 00:19:30] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:46 [I 2022-10-27 00:19:30] ...trap/engines/engine.py.229: process: 0.56923 | load: 0.00250 [I 2022-10-27 00:19:30] ...trap/engines/engine.py.230: loss: 0.22467 [I 2022-10-27 00:19:30] ...optimizers/trijoint.py.151: total_norm: 15.856661 [I 2022-10-27 00:19:36] ...trap/engines/engine.py.224: train: epoch 37 | batch 1910/2304 [I 2022-10-27 00:19:36] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:03:38 [I 2022-10-27 00:19:36] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00256 [I 2022-10-27 00:19:36] ...trap/engines/engine.py.230: loss: 0.21446 [I 2022-10-27 00:19:36] ...optimizers/trijoint.py.151: total_norm: 21.571859 [I 2022-10-27 00:19:41] ...trap/engines/engine.py.224: train: epoch 37 | batch 1920/2304 [I 2022-10-27 00:19:41] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:32 [I 2022-10-27 00:19:41] ...trap/engines/engine.py.229: process: 0.55491 | load: 0.00252 [I 2022-10-27 00:19:41] ...trap/engines/engine.py.230: loss: 0.14841 [I 2022-10-27 00:19:41] ...optimizers/trijoint.py.151: total_norm: 22.367407 [I 2022-10-27 00:19:47] ...trap/engines/engine.py.224: train: epoch 37 | batch 1930/2304 [I 2022-10-27 00:19:47] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:29 [I 2022-10-27 00:19:47] ...trap/engines/engine.py.229: process: 0.57054 | load: 0.00254 [I 2022-10-27 00:19:47] ...trap/engines/engine.py.230: loss: 0.14596 [I 2022-10-27 00:19:47] ...optimizers/trijoint.py.151: total_norm: 13.807660 [I 2022-10-27 00:19:53] ...trap/engines/engine.py.224: train: epoch 37 | batch 1940/2304 [I 2022-10-27 00:19:53] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:22 [I 2022-10-27 00:19:53] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00259 [I 2022-10-27 00:19:53] ...trap/engines/engine.py.230: loss: 0.23230 [I 2022-10-27 00:19:53] ...optimizers/trijoint.py.151: total_norm: 24.072323 [I 2022-10-27 00:19:58] ...trap/engines/engine.py.224: train: epoch 37 | batch 1950/2304 [I 2022-10-27 00:19:58] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:17 [I 2022-10-27 00:19:58] ...trap/engines/engine.py.229: process: 0.56520 | load: 0.00253 [I 2022-10-27 00:19:58] ...trap/engines/engine.py.230: loss: 0.17680 [I 2022-10-27 00:19:58] ...optimizers/trijoint.py.151: total_norm: 12.932487 [I 2022-10-27 00:20:04] ...trap/engines/engine.py.224: train: epoch 37 | batch 1960/2304 [I 2022-10-27 00:20:04] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:03:12 [I 2022-10-27 00:20:04] ...trap/engines/engine.py.229: process: 0.56050 | load: 0.00253 [I 2022-10-27 00:20:04] ...trap/engines/engine.py.230: loss: 0.17723 [I 2022-10-27 00:20:04] ...optimizers/trijoint.py.151: total_norm: 15.655293 [I 2022-10-27 00:20:09] ...trap/engines/engine.py.224: train: epoch 37 | batch 1970/2304 [I 2022-10-27 00:20:09] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:03:06 [I 2022-10-27 00:20:09] ...trap/engines/engine.py.229: process: 0.55216 | load: 0.00243 [I 2022-10-27 00:20:09] ...trap/engines/engine.py.230: loss: 0.16757 [I 2022-10-27 00:20:09] ...optimizers/trijoint.py.151: total_norm: 21.971449 [I 2022-10-27 00:20:15] ...trap/engines/engine.py.224: train: epoch 37 | batch 1980/2304 [I 2022-10-27 00:20:15] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:03:00 [I 2022-10-27 00:20:15] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00241 [I 2022-10-27 00:20:15] ...trap/engines/engine.py.230: loss: 0.13790 [I 2022-10-27 00:20:15] ...optimizers/trijoint.py.151: total_norm: 21.237431 [I 2022-10-27 00:20:20] ...trap/engines/engine.py.224: train: epoch 37 | batch 1990/2304 [I 2022-10-27 00:20:20] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:54 [I 2022-10-27 00:20:20] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00245 [I 2022-10-27 00:20:20] ...trap/engines/engine.py.230: loss: 0.12748 [I 2022-10-27 00:20:20] ...optimizers/trijoint.py.151: total_norm: 10.068330 [I 2022-10-27 00:20:26] ...trap/engines/engine.py.224: train: epoch 37 | batch 2000/2304 [I 2022-10-27 00:20:26] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:49 [I 2022-10-27 00:20:26] ...trap/engines/engine.py.229: process: 0.56221 | load: 0.00242 [I 2022-10-27 00:20:26] ...trap/engines/engine.py.230: loss: 0.12369 [I 2022-10-27 00:20:26] ...optimizers/trijoint.py.151: total_norm: 11.412054 [I 2022-10-27 00:20:32] ...trap/engines/engine.py.224: train: epoch 37 | batch 2010/2304 [I 2022-10-27 00:20:32] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:43 [I 2022-10-27 00:20:32] ...trap/engines/engine.py.229: process: 0.54731 | load: 0.00243 [I 2022-10-27 00:20:32] ...trap/engines/engine.py.230: loss: 0.16280 [I 2022-10-27 00:20:32] ...optimizers/trijoint.py.151: total_norm: 19.423792 [I 2022-10-27 00:20:37] ...trap/engines/engine.py.224: train: epoch 37 | batch 2020/2304 [I 2022-10-27 00:20:37] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:43 [I 2022-10-27 00:20:37] ...trap/engines/engine.py.229: process: 0.57041 | load: 0.00244 [I 2022-10-27 00:20:37] ...trap/engines/engine.py.230: loss: 0.20017 [I 2022-10-27 00:20:37] ...optimizers/trijoint.py.151: total_norm: 33.366608 [I 2022-10-27 00:20:43] ...trap/engines/engine.py.224: train: epoch 37 | batch 2030/2304 [I 2022-10-27 00:20:43] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:34 [I 2022-10-27 00:20:43] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00242 [I 2022-10-27 00:20:43] ...trap/engines/engine.py.230: loss: 0.13453 [I 2022-10-27 00:20:43] ...optimizers/trijoint.py.151: total_norm: 18.254934 [I 2022-10-27 00:20:49] ...trap/engines/engine.py.224: train: epoch 37 | batch 2040/2304 [I 2022-10-27 00:20:49] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:26 [I 2022-10-27 00:20:49] ...trap/engines/engine.py.229: process: 0.55474 | load: 0.00242 [I 2022-10-27 00:20:49] ...trap/engines/engine.py.230: loss: 0.16028 [I 2022-10-27 00:20:49] ...optimizers/trijoint.py.151: total_norm: 35.190548 [I 2022-10-27 00:20:54] ...trap/engines/engine.py.224: train: epoch 37 | batch 2050/2304 [I 2022-10-27 00:20:54] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:21 [I 2022-10-27 00:20:54] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00243 [I 2022-10-27 00:20:54] ...trap/engines/engine.py.230: loss: 0.27235 [I 2022-10-27 00:20:54] ...optimizers/trijoint.py.151: total_norm: 29.665091 [I 2022-10-27 00:21:00] ...trap/engines/engine.py.224: train: epoch 37 | batch 2060/2304 [I 2022-10-27 00:21:00] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:02:16 [I 2022-10-27 00:21:00] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00247 [I 2022-10-27 00:21:00] ...trap/engines/engine.py.230: loss: 0.16767 [I 2022-10-27 00:21:00] ...optimizers/trijoint.py.151: total_norm: 11.630731 [I 2022-10-27 00:21:05] ...trap/engines/engine.py.224: train: epoch 37 | batch 2070/2304 [I 2022-10-27 00:21:05] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:02:08 [I 2022-10-27 00:21:05] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00247 [I 2022-10-27 00:21:05] ...trap/engines/engine.py.230: loss: 0.16178 [I 2022-10-27 00:21:05] ...optimizers/trijoint.py.151: total_norm: 28.923796 [I 2022-10-27 00:21:11] ...trap/engines/engine.py.224: train: epoch 37 | batch 2080/2304 [I 2022-10-27 00:21:11] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:02:05 [I 2022-10-27 00:21:11] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00243 [I 2022-10-27 00:21:11] ...trap/engines/engine.py.230: loss: 0.25429 [I 2022-10-27 00:21:11] ...optimizers/trijoint.py.151: total_norm: 19.593395 [I 2022-10-27 00:21:16] ...trap/engines/engine.py.224: train: epoch 37 | batch 2090/2304 [I 2022-10-27 00:21:16] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:57 [I 2022-10-27 00:21:16] ...trap/engines/engine.py.229: process: 0.54585 | load: 0.00243 [I 2022-10-27 00:21:16] ...trap/engines/engine.py.230: loss: 0.20748 [I 2022-10-27 00:21:16] ...optimizers/trijoint.py.151: total_norm: 17.662701 [I 2022-10-27 00:21:22] ...trap/engines/engine.py.224: train: epoch 37 | batch 2100/2304 [I 2022-10-27 00:21:22] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:53 [I 2022-10-27 00:21:22] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00252 [I 2022-10-27 00:21:22] ...trap/engines/engine.py.230: loss: 0.23644 [I 2022-10-27 00:21:22] ...optimizers/trijoint.py.151: total_norm: 22.696224 [I 2022-10-27 00:21:28] ...trap/engines/engine.py.224: train: epoch 37 | batch 2110/2304 [I 2022-10-27 00:21:28] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:48 [I 2022-10-27 00:21:28] ...trap/engines/engine.py.229: process: 0.56694 | load: 0.00258 [I 2022-10-27 00:21:28] ...trap/engines/engine.py.230: loss: 0.17768 [I 2022-10-27 00:21:28] ...optimizers/trijoint.py.151: total_norm: 14.420608 [I 2022-10-27 00:21:33] ...trap/engines/engine.py.224: train: epoch 37 | batch 2120/2304 [I 2022-10-27 00:21:33] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:42 [I 2022-10-27 00:21:33] ...trap/engines/engine.py.229: process: 0.56477 | load: 0.00247 [I 2022-10-27 00:21:33] ...trap/engines/engine.py.230: loss: 0.17417 [I 2022-10-27 00:21:33] ...optimizers/trijoint.py.151: total_norm: 11.665674 [I 2022-10-27 00:21:39] ...trap/engines/engine.py.224: train: epoch 37 | batch 2130/2304 [I 2022-10-27 00:21:39] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:36 [I 2022-10-27 00:21:39] ...trap/engines/engine.py.229: process: 0.54475 | load: 0.00253 [I 2022-10-27 00:21:39] ...trap/engines/engine.py.230: loss: 0.17664 [I 2022-10-27 00:21:39] ...optimizers/trijoint.py.151: total_norm: 24.007502 [I 2022-10-27 00:21:44] ...trap/engines/engine.py.224: train: epoch 37 | batch 2140/2304 [I 2022-10-27 00:21:44] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:30 [I 2022-10-27 00:21:44] ...trap/engines/engine.py.229: process: 0.56347 | load: 0.00253 [I 2022-10-27 00:21:44] ...trap/engines/engine.py.230: loss: 0.18843 [I 2022-10-27 00:21:44] ...optimizers/trijoint.py.151: total_norm: 17.493555 [I 2022-10-27 00:21:50] ...trap/engines/engine.py.224: train: epoch 37 | batch 2150/2304 [I 2022-10-27 00:21:50] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:25 [I 2022-10-27 00:21:50] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00246 [I 2022-10-27 00:21:50] ...trap/engines/engine.py.230: loss: 0.19959 [I 2022-10-27 00:21:50] ...optimizers/trijoint.py.151: total_norm: 11.699128 [I 2022-10-27 00:21:55] ...trap/engines/engine.py.224: train: epoch 37 | batch 2160/2304 [I 2022-10-27 00:21:55] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:19 [I 2022-10-27 00:21:55] ...trap/engines/engine.py.229: process: 0.54160 | load: 0.00241 [I 2022-10-27 00:21:55] ...trap/engines/engine.py.230: loss: 0.18833 [I 2022-10-27 00:21:55] ...optimizers/trijoint.py.151: total_norm: 22.320166 [I 2022-10-27 00:22:01] ...trap/engines/engine.py.224: train: epoch 37 | batch 2170/2304 [I 2022-10-27 00:22:01] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:01:15 [I 2022-10-27 00:22:01] ...trap/engines/engine.py.229: process: 0.56904 | load: 0.00246 [I 2022-10-27 00:22:01] ...trap/engines/engine.py.230: loss: 0.16042 [I 2022-10-27 00:22:01] ...optimizers/trijoint.py.151: total_norm: 15.744212 [I 2022-10-27 00:22:07] ...trap/engines/engine.py.224: train: epoch 37 | batch 2180/2304 [I 2022-10-27 00:22:07] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:09 [I 2022-10-27 00:22:07] ...trap/engines/engine.py.229: process: 0.55045 | load: 0.00243 [I 2022-10-27 00:22:07] ...trap/engines/engine.py.230: loss: 0.15950 [I 2022-10-27 00:22:07] ...optimizers/trijoint.py.151: total_norm: 13.950507 [I 2022-10-27 00:22:12] ...trap/engines/engine.py.224: train: epoch 37 | batch 2190/2304 [I 2022-10-27 00:22:12] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:01:03 [I 2022-10-27 00:22:12] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00245 [I 2022-10-27 00:22:12] ...trap/engines/engine.py.230: loss: 0.14595 [I 2022-10-27 00:22:12] ...optimizers/trijoint.py.151: total_norm: 9.968465 [I 2022-10-27 00:22:18] ...trap/engines/engine.py.224: train: epoch 37 | batch 2200/2304 [I 2022-10-27 00:22:18] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:57 [I 2022-10-27 00:22:18] ...trap/engines/engine.py.229: process: 0.54339 | load: 0.00241 [I 2022-10-27 00:22:18] ...trap/engines/engine.py.230: loss: 0.18213 [I 2022-10-27 00:22:18] ...optimizers/trijoint.py.151: total_norm: 13.795487 [I 2022-10-27 00:22:23] ...trap/engines/engine.py.224: train: epoch 37 | batch 2210/2304 [I 2022-10-27 00:22:23] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:52 [I 2022-10-27 00:22:23] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00243 [I 2022-10-27 00:22:23] ...trap/engines/engine.py.230: loss: 0.15926 [I 2022-10-27 00:22:23] ...optimizers/trijoint.py.151: total_norm: 15.992170 [I 2022-10-27 00:22:29] ...trap/engines/engine.py.224: train: epoch 37 | batch 2220/2304 [I 2022-10-27 00:22:29] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:47 [I 2022-10-27 00:22:29] ...trap/engines/engine.py.229: process: 0.56288 | load: 0.00246 [I 2022-10-27 00:22:29] ...trap/engines/engine.py.230: loss: 0.15649 [I 2022-10-27 00:22:29] ...optimizers/trijoint.py.151: total_norm: 15.907653 [I 2022-10-27 00:22:35] ...trap/engines/engine.py.224: train: epoch 37 | batch 2230/2304 [I 2022-10-27 00:22:35] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:41 [I 2022-10-27 00:22:35] ...trap/engines/engine.py.229: process: 0.54169 | load: 0.00245 [I 2022-10-27 00:22:35] ...trap/engines/engine.py.230: loss: 0.26202 [I 2022-10-27 00:22:35] ...optimizers/trijoint.py.151: total_norm: 19.929174 [I 2022-10-27 00:22:40] ...trap/engines/engine.py.224: train: epoch 37 | batch 2240/2304 [I 2022-10-27 00:22:40] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:35 [I 2022-10-27 00:22:40] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00243 [I 2022-10-27 00:22:40] ...trap/engines/engine.py.230: loss: 0.19308 [I 2022-10-27 00:22:40] ...optimizers/trijoint.py.151: total_norm: 19.318712 [I 2022-10-27 00:22:46] ...trap/engines/engine.py.224: train: epoch 37 | batch 2250/2304 [I 2022-10-27 00:22:46] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:30 [I 2022-10-27 00:22:46] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00244 [I 2022-10-27 00:22:46] ...trap/engines/engine.py.230: loss: 0.13932 [I 2022-10-27 00:22:46] ...optimizers/trijoint.py.151: total_norm: 19.197201 [I 2022-10-27 00:22:51] ...trap/engines/engine.py.224: train: epoch 37 | batch 2260/2304 [I 2022-10-27 00:22:51] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:24 [I 2022-10-27 00:22:51] ...trap/engines/engine.py.229: process: 0.56198 | load: 0.00242 [I 2022-10-27 00:22:51] ...trap/engines/engine.py.230: loss: 0.17670 [I 2022-10-27 00:22:51] ...optimizers/trijoint.py.151: total_norm: 14.202565 [I 2022-10-27 00:22:57] ...trap/engines/engine.py.224: train: epoch 37 | batch 2270/2304 [I 2022-10-27 00:22:57] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:18 [I 2022-10-27 00:22:57] ...trap/engines/engine.py.229: process: 0.54432 | load: 0.00252 [I 2022-10-27 00:22:57] ...trap/engines/engine.py.230: loss: 0.13650 [I 2022-10-27 00:22:57] ...optimizers/trijoint.py.151: total_norm: 15.592791 [I 2022-10-27 00:23:02] ...trap/engines/engine.py.224: train: epoch 37 | batch 2280/2304 [I 2022-10-27 00:23:02] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:13 [I 2022-10-27 00:23:02] ...trap/engines/engine.py.229: process: 0.54485 | load: 0.00248 [I 2022-10-27 00:23:02] ...trap/engines/engine.py.230: loss: 0.13668 [I 2022-10-27 00:23:02] ...optimizers/trijoint.py.151: total_norm: 17.656298 [I 2022-10-27 00:23:08] ...trap/engines/engine.py.224: train: epoch 37 | batch 2290/2304 [I 2022-10-27 00:23:08] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:00:07 [I 2022-10-27 00:23:08] ...trap/engines/engine.py.229: process: 0.59127 | load: 0.00245 [I 2022-10-27 00:23:08] ...trap/engines/engine.py.230: loss: 0.13225 [I 2022-10-27 00:23:08] ...optimizers/trijoint.py.151: total_norm: 30.722050 [I 2022-10-27 00:23:14] ...trap/engines/engine.py.224: train: epoch 37 | batch 2300/2304 [I 2022-10-27 00:23:14] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:02 [I 2022-10-27 00:23:14] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00016 [I 2022-10-27 00:23:14] ...trap/engines/engine.py.230: loss: 0.16736 [I 2022-10-27 00:23:14] ...optimizers/trijoint.py.151: total_norm: 20.974085 [I 2022-10-27 00:23:16] ...trap/engines/engine.py.224: train: epoch 37 | batch 2304/2304 [I 2022-10-27 00:23:16] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:00 [I 2022-10-27 00:23:16] ...trap/engines/engine.py.229: process: 0.54873 | load: 0.00016 [I 2022-10-27 00:23:16] ...trap/engines/engine.py.230: loss: 0.16576 [I 2022-10-27 00:23:16] ...optimizers/trijoint.py.151: total_norm: 13.743776 [S 2022-10-27 00:23:17] ...trap/engines/engine.py.129: train_epoch.epoch: 37 [S 2022-10-27 00:23:17] ...trap/engines/engine.py.129: train_epoch.loss: 0.17238351194240525 [I 2022-10-27 00:23:17] ...s/criterions/triplet.py.84: margin = 0.24000000000000016 [I 2022-10-27 00:23:17] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 37 [I 2022-10-27 00:23:17] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 00:23:19] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 00:23:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 00:23:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 00:23:20] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 00:23:21] ...trap/engines/engine.py.321: eval: epoch 37 | batch 0/510 [I 2022-10-27 00:23:21] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:28:37 [I 2022-10-27 00:23:21] ...trap/engines/engine.py.326: process: 3.36838 | load: 3.00130 [I 2022-10-27 00:23:24] ...trap/engines/engine.py.321: eval: epoch 37 | batch 10/510 [I 2022-10-27 00:23:24] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:29 [I 2022-10-27 00:23:24] ...trap/engines/engine.py.326: process: 0.32942 | load: 0.00019 [I 2022-10-27 00:23:27] ...trap/engines/engine.py.321: eval: epoch 37 | batch 20/510 [I 2022-10-27 00:23:27] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:00 [I 2022-10-27 00:23:27] ...trap/engines/engine.py.326: process: 0.34676 | load: 0.00017 [I 2022-10-27 00:23:31] ...trap/engines/engine.py.321: eval: epoch 37 | batch 30/510 [I 2022-10-27 00:23:31] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:45 [I 2022-10-27 00:23:31] ...trap/engines/engine.py.326: process: 0.35012 | load: 0.00016 [I 2022-10-27 00:23:34] ...trap/engines/engine.py.321: eval: epoch 37 | batch 40/510 [I 2022-10-27 00:23:34] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:37 [I 2022-10-27 00:23:34] ...trap/engines/engine.py.326: process: 0.33760 | load: 0.00017 [I 2022-10-27 00:23:38] ...trap/engines/engine.py.321: eval: epoch 37 | batch 50/510 [I 2022-10-27 00:23:38] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:32 [I 2022-10-27 00:23:38] ...trap/engines/engine.py.326: process: 0.32257 | load: 0.00015 [I 2022-10-27 00:23:41] ...trap/engines/engine.py.321: eval: epoch 37 | batch 60/510 [I 2022-10-27 00:23:41] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:30 [I 2022-10-27 00:23:41] ...trap/engines/engine.py.326: process: 0.32924 | load: 0.00017 [I 2022-10-27 00:23:44] ...trap/engines/engine.py.321: eval: epoch 37 | batch 70/510 [I 2022-10-27 00:23:44] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:24 [I 2022-10-27 00:23:44] ...trap/engines/engine.py.326: process: 0.33177 | load: 0.00016 [I 2022-10-27 00:23:47] ...trap/engines/engine.py.321: eval: epoch 37 | batch 80/510 [I 2022-10-27 00:23:47] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:22 [I 2022-10-27 00:23:47] ...trap/engines/engine.py.326: process: 0.32634 | load: 0.00017 [I 2022-10-27 00:23:51] ...trap/engines/engine.py.321: eval: epoch 37 | batch 90/510 [I 2022-10-27 00:23:51] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:22 [I 2022-10-27 00:23:51] ...trap/engines/engine.py.326: process: 0.36838 | load: 0.00017 [I 2022-10-27 00:23:54] ...trap/engines/engine.py.321: eval: epoch 37 | batch 100/510 [I 2022-10-27 00:23:54] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:14 [I 2022-10-27 00:23:54] ...trap/engines/engine.py.326: process: 0.29729 | load: 0.00018 [I 2022-10-27 00:23:57] ...trap/engines/engine.py.321: eval: epoch 37 | batch 110/510 [I 2022-10-27 00:23:57] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-27 00:23:57] ...trap/engines/engine.py.326: process: 0.27982 | load: 0.00014 [I 2022-10-27 00:24:00] ...trap/engines/engine.py.321: eval: epoch 37 | batch 120/510 [I 2022-10-27 00:24:00] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-27 00:24:00] ...trap/engines/engine.py.326: process: 0.28611 | load: 0.00014 [I 2022-10-27 00:24:03] ...trap/engines/engine.py.321: eval: epoch 37 | batch 130/510 [I 2022-10-27 00:24:03] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:48 [I 2022-10-27 00:24:03] ...trap/engines/engine.py.326: process: 0.28399 | load: 0.00018 [I 2022-10-27 00:24:06] ...trap/engines/engine.py.321: eval: epoch 37 | batch 140/510 [I 2022-10-27 00:24:06] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-27 00:24:06] ...trap/engines/engine.py.326: process: 0.28706 | load: 0.00020 [I 2022-10-27 00:24:09] ...trap/engines/engine.py.321: eval: epoch 37 | batch 150/510 [I 2022-10-27 00:24:09] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-27 00:24:09] ...trap/engines/engine.py.326: process: 0.29084 | load: 0.00019 [I 2022-10-27 00:24:11] ...trap/engines/engine.py.321: eval: epoch 37 | batch 160/510 [I 2022-10-27 00:24:11] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:40 [I 2022-10-27 00:24:11] ...trap/engines/engine.py.326: process: 0.28740 | load: 0.00021 [I 2022-10-27 00:24:14] ...trap/engines/engine.py.321: eval: epoch 37 | batch 170/510 [I 2022-10-27 00:24:14] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:37 [I 2022-10-27 00:24:14] ...trap/engines/engine.py.326: process: 0.28285 | load: 0.00025 [I 2022-10-27 00:24:17] ...trap/engines/engine.py.321: eval: epoch 37 | batch 180/510 [I 2022-10-27 00:24:17] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:34 [I 2022-10-27 00:24:17] ...trap/engines/engine.py.326: process: 0.28473 | load: 0.00020 [I 2022-10-27 00:24:20] ...trap/engines/engine.py.321: eval: epoch 37 | batch 190/510 [I 2022-10-27 00:24:20] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:33 [I 2022-10-27 00:24:20] ...trap/engines/engine.py.326: process: 0.28841 | load: 0.00021 [I 2022-10-27 00:24:23] ...trap/engines/engine.py.321: eval: epoch 37 | batch 200/510 [I 2022-10-27 00:24:23] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-27 00:24:23] ...trap/engines/engine.py.326: process: 0.31851 | load: 0.00015 [I 2022-10-27 00:24:26] ...trap/engines/engine.py.321: eval: epoch 37 | batch 210/510 [I 2022-10-27 00:24:26] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-27 00:24:26] ...trap/engines/engine.py.326: process: 0.28728 | load: 0.00015 [I 2022-10-27 00:24:29] ...trap/engines/engine.py.321: eval: epoch 37 | batch 220/510 [I 2022-10-27 00:24:29] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-27 00:24:29] ...trap/engines/engine.py.326: process: 0.28711 | load: 0.00015 [I 2022-10-27 00:24:32] ...trap/engines/engine.py.321: eval: epoch 37 | batch 230/510 [I 2022-10-27 00:24:32] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-27 00:24:32] ...trap/engines/engine.py.326: process: 0.28600 | load: 0.00014 [I 2022-10-27 00:24:35] ...trap/engines/engine.py.321: eval: epoch 37 | batch 240/510 [I 2022-10-27 00:24:35] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-27 00:24:35] ...trap/engines/engine.py.326: process: 0.28584 | load: 0.00015 [I 2022-10-27 00:24:37] ...trap/engines/engine.py.321: eval: epoch 37 | batch 250/510 [I 2022-10-27 00:24:37] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:14 [I 2022-10-27 00:24:37] ...trap/engines/engine.py.326: process: 0.28363 | load: 0.00015 [I 2022-10-27 00:24:40] ...trap/engines/engine.py.321: eval: epoch 37 | batch 260/510 [I 2022-10-27 00:24:40] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:11 [I 2022-10-27 00:24:40] ...trap/engines/engine.py.326: process: 0.29133 | load: 0.00015 [I 2022-10-27 00:24:43] ...trap/engines/engine.py.321: eval: epoch 37 | batch 270/510 [I 2022-10-27 00:24:43] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-27 00:24:43] ...trap/engines/engine.py.326: process: 0.28937 | load: 0.00016 [I 2022-10-27 00:24:46] ...trap/engines/engine.py.321: eval: epoch 37 | batch 280/510 [I 2022-10-27 00:24:46] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:05 [I 2022-10-27 00:24:46] ...trap/engines/engine.py.326: process: 0.28613 | load: 0.00016 [I 2022-10-27 00:24:49] ...trap/engines/engine.py.321: eval: epoch 37 | batch 290/510 [I 2022-10-27 00:24:49] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:02 [I 2022-10-27 00:24:49] ...trap/engines/engine.py.326: process: 0.28523 | load: 0.00015 [I 2022-10-27 00:24:52] ...trap/engines/engine.py.321: eval: epoch 37 | batch 300/510 [I 2022-10-27 00:24:52] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-27 00:24:52] ...trap/engines/engine.py.326: process: 0.29194 | load: 0.00015 [I 2022-10-27 00:24:55] ...trap/engines/engine.py.321: eval: epoch 37 | batch 310/510 [I 2022-10-27 00:24:55] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 00:24:55] ...trap/engines/engine.py.326: process: 0.29119 | load: 0.00015 [I 2022-10-27 00:24:57] ...trap/engines/engine.py.321: eval: epoch 37 | batch 320/510 [I 2022-10-27 00:24:57] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 00:24:57] ...trap/engines/engine.py.326: process: 0.28388 | load: 0.00015 [I 2022-10-27 00:25:00] ...trap/engines/engine.py.321: eval: epoch 37 | batch 330/510 [I 2022-10-27 00:25:00] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:51 [I 2022-10-27 00:25:00] ...trap/engines/engine.py.326: process: 0.28844 | load: 0.00017 [I 2022-10-27 00:25:03] ...trap/engines/engine.py.321: eval: epoch 37 | batch 340/510 [I 2022-10-27 00:25:03] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:48 [I 2022-10-27 00:25:03] ...trap/engines/engine.py.326: process: 0.29171 | load: 0.00014 [I 2022-10-27 00:25:06] ...trap/engines/engine.py.321: eval: epoch 37 | batch 350/510 [I 2022-10-27 00:25:06] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:45 [I 2022-10-27 00:25:06] ...trap/engines/engine.py.326: process: 0.28884 | load: 0.00015 [I 2022-10-27 00:25:09] ...trap/engines/engine.py.321: eval: epoch 37 | batch 360/510 [I 2022-10-27 00:25:09] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:42 [I 2022-10-27 00:25:09] ...trap/engines/engine.py.326: process: 0.28450 | load: 0.00017 [I 2022-10-27 00:25:12] ...trap/engines/engine.py.321: eval: epoch 37 | batch 370/510 [I 2022-10-27 00:25:12] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:39 [I 2022-10-27 00:25:12] ...trap/engines/engine.py.326: process: 0.28114 | load: 0.00016 [I 2022-10-27 00:25:15] ...trap/engines/engine.py.321: eval: epoch 37 | batch 380/510 [I 2022-10-27 00:25:15] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-27 00:25:15] ...trap/engines/engine.py.326: process: 0.28861 | load: 0.00015 [I 2022-10-27 00:25:18] ...trap/engines/engine.py.321: eval: epoch 37 | batch 390/510 [I 2022-10-27 00:25:18] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-27 00:25:18] ...trap/engines/engine.py.326: process: 0.28791 | load: 0.00020 [I 2022-10-27 00:25:20] ...trap/engines/engine.py.321: eval: epoch 37 | batch 400/510 [I 2022-10-27 00:25:20] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:31 [I 2022-10-27 00:25:20] ...trap/engines/engine.py.326: process: 0.28336 | load: 0.00015 [I 2022-10-27 00:25:23] ...trap/engines/engine.py.321: eval: epoch 37 | batch 410/510 [I 2022-10-27 00:25:23] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:28 [I 2022-10-27 00:25:23] ...trap/engines/engine.py.326: process: 0.28752 | load: 0.00015 [I 2022-10-27 00:25:26] ...trap/engines/engine.py.321: eval: epoch 37 | batch 420/510 [I 2022-10-27 00:25:26] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-27 00:25:26] ...trap/engines/engine.py.326: process: 0.29013 | load: 0.00019 [I 2022-10-27 00:25:29] ...trap/engines/engine.py.321: eval: epoch 37 | batch 430/510 [I 2022-10-27 00:25:29] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:23 [I 2022-10-27 00:25:29] ...trap/engines/engine.py.326: process: 0.28246 | load: 0.00014 [I 2022-10-27 00:25:32] ...trap/engines/engine.py.321: eval: epoch 37 | batch 440/510 [I 2022-10-27 00:25:32] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-27 00:25:32] ...trap/engines/engine.py.326: process: 0.28757 | load: 0.00015 [I 2022-10-27 00:25:35] ...trap/engines/engine.py.321: eval: epoch 37 | batch 450/510 [I 2022-10-27 00:25:35] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-27 00:25:35] ...trap/engines/engine.py.326: process: 0.28384 | load: 0.00019 [I 2022-10-27 00:25:38] ...trap/engines/engine.py.321: eval: epoch 37 | batch 460/510 [I 2022-10-27 00:25:38] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:15 [I 2022-10-27 00:25:38] ...trap/engines/engine.py.326: process: 0.36927 | load: 0.00021 [I 2022-10-27 00:25:41] ...trap/engines/engine.py.321: eval: epoch 37 | batch 470/510 [I 2022-10-27 00:25:41] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-27 00:25:41] ...trap/engines/engine.py.326: process: 0.28709 | load: 0.00019 [I 2022-10-27 00:25:44] ...trap/engines/engine.py.321: eval: epoch 37 | batch 480/510 [I 2022-10-27 00:25:44] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 00:25:44] ...trap/engines/engine.py.326: process: 0.28536 | load: 0.00018 [I 2022-10-27 00:25:46] ...trap/engines/engine.py.321: eval: epoch 37 | batch 490/510 [I 2022-10-27 00:25:46] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 00:25:46] ...trap/engines/engine.py.326: process: 0.28409 | load: 0.00015 [I 2022-10-27 00:25:49] ...trap/engines/engine.py.321: eval: epoch 37 | batch 500/510 [I 2022-10-27 00:25:49] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-27 00:25:49] ...trap/engines/engine.py.326: process: 0.28168 | load: 0.00015 [I 2022-10-27 00:25:52] ...trap/engines/engine.py.321: eval: epoch 37 | batch 510/510 [I 2022-10-27 00:25:52] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-27 00:25:52] ...trap/engines/engine.py.326: process: 0.28378 | load: 0.00012 [S 2022-10-27 00:25:53] ...trap/engines/engine.py.132: eval_epoch.epoch: 37 [I 2022-10-27 00:25:53] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.673900 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.890700 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.930100 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.656700 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.885500 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.925200 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009874 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009456 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008130 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008568 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012118 [S 2022-10-27 00:25:57] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010313 [I 2022-10-27 00:25:58] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 00:25:58] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 00:25:58] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 00:25:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 00:25:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 00:26:00] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 00:26:02] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 00:26:08] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 00:26:08] ...trap/engines/engine.py.163: Training model on trainset for epoch 38 [I 2022-10-27 00:26:08] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 00:26:11] ...trap/engines/engine.py.224: train: epoch 38 | batch 0/2304 [I 2022-10-27 00:26:11] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:40:29 [I 2022-10-27 00:26:11] ...trap/engines/engine.py.229: process: 2.61680 | load: 1.98988 [I 2022-10-27 00:26:11] ...trap/engines/engine.py.230: loss: 0.21461 [I 2022-10-27 00:26:11] ...optimizers/trijoint.py.151: total_norm: 28.757238 [I 2022-10-27 00:26:17] ...trap/engines/engine.py.224: train: epoch 38 | batch 10/2304 [I 2022-10-27 00:26:17] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:41 [I 2022-10-27 00:26:17] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00424 [I 2022-10-27 00:26:17] ...trap/engines/engine.py.230: loss: 0.15136 [I 2022-10-27 00:26:17] ...optimizers/trijoint.py.151: total_norm: 20.799162 [I 2022-10-27 00:26:22] ...trap/engines/engine.py.224: train: epoch 38 | batch 20/2304 [I 2022-10-27 00:26:22] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:06 [I 2022-10-27 00:26:22] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00355 [I 2022-10-27 00:26:22] ...trap/engines/engine.py.230: loss: 0.13931 [I 2022-10-27 00:26:22] ...optimizers/trijoint.py.151: total_norm: 11.819368 [I 2022-10-27 00:26:28] ...trap/engines/engine.py.224: train: epoch 38 | batch 30/2304 [I 2022-10-27 00:26:28] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:17 [I 2022-10-27 00:26:28] ...trap/engines/engine.py.229: process: 0.55031 | load: 0.00289 [I 2022-10-27 00:26:28] ...trap/engines/engine.py.230: loss: 0.17090 [I 2022-10-27 00:26:28] ...optimizers/trijoint.py.151: total_norm: 14.072762 [I 2022-10-27 00:26:33] ...trap/engines/engine.py.224: train: epoch 38 | batch 40/2304 [I 2022-10-27 00:26:33] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:02 [I 2022-10-27 00:26:33] ...trap/engines/engine.py.229: process: 0.54796 | load: 0.00276 [I 2022-10-27 00:26:33] ...trap/engines/engine.py.230: loss: 0.19672 [I 2022-10-27 00:26:33] ...optimizers/trijoint.py.151: total_norm: 23.337507 [I 2022-10-27 00:26:39] ...trap/engines/engine.py.224: train: epoch 38 | batch 50/2304 [I 2022-10-27 00:26:39] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:00 [I 2022-10-27 00:26:39] ...trap/engines/engine.py.229: process: 0.57339 | load: 0.00322 [I 2022-10-27 00:26:39] ...trap/engines/engine.py.230: loss: 0.23193 [I 2022-10-27 00:26:39] ...optimizers/trijoint.py.151: total_norm: 19.984341 [I 2022-10-27 00:26:45] ...trap/engines/engine.py.224: train: epoch 38 | batch 60/2304 [I 2022-10-27 00:26:45] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:49 [I 2022-10-27 00:26:45] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00266 [I 2022-10-27 00:26:45] ...trap/engines/engine.py.230: loss: 0.20341 [I 2022-10-27 00:26:45] ...optimizers/trijoint.py.151: total_norm: 16.571503 [I 2022-10-27 00:26:50] ...trap/engines/engine.py.224: train: epoch 38 | batch 70/2304 [I 2022-10-27 00:26:50] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:50 [I 2022-10-27 00:26:50] ...trap/engines/engine.py.229: process: 0.56320 | load: 0.00293 [I 2022-10-27 00:26:50] ...trap/engines/engine.py.230: loss: 0.21359 [I 2022-10-27 00:26:50] ...optimizers/trijoint.py.151: total_norm: 24.642258 [I 2022-10-27 00:26:56] ...trap/engines/engine.py.224: train: epoch 38 | batch 80/2304 [I 2022-10-27 00:26:56] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:46 [I 2022-10-27 00:26:56] ...trap/engines/engine.py.229: process: 0.57442 | load: 0.00251 [I 2022-10-27 00:26:56] ...trap/engines/engine.py.230: loss: 0.11494 [I 2022-10-27 00:26:56] ...optimizers/trijoint.py.151: total_norm: 16.091927 [I 2022-10-27 00:27:01] ...trap/engines/engine.py.224: train: epoch 38 | batch 90/2304 [I 2022-10-27 00:27:01] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:40 [I 2022-10-27 00:27:01] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00249 [I 2022-10-27 00:27:01] ...trap/engines/engine.py.230: loss: 0.15965 [I 2022-10-27 00:27:01] ...optimizers/trijoint.py.151: total_norm: 11.620275 [I 2022-10-27 00:27:07] ...trap/engines/engine.py.224: train: epoch 38 | batch 100/2304 [I 2022-10-27 00:27:07] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:17 [I 2022-10-27 00:27:07] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00262 [I 2022-10-27 00:27:07] ...trap/engines/engine.py.230: loss: 0.14301 [I 2022-10-27 00:27:07] ...optimizers/trijoint.py.151: total_norm: 19.645893 [I 2022-10-27 00:27:13] ...trap/engines/engine.py.224: train: epoch 38 | batch 110/2304 [I 2022-10-27 00:27:13] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:17 [I 2022-10-27 00:27:13] ...trap/engines/engine.py.229: process: 0.55426 | load: 0.00241 [I 2022-10-27 00:27:13] ...trap/engines/engine.py.230: loss: 0.16727 [I 2022-10-27 00:27:13] ...optimizers/trijoint.py.151: total_norm: 12.687413 [I 2022-10-27 00:27:18] ...trap/engines/engine.py.224: train: epoch 38 | batch 120/2304 [I 2022-10-27 00:27:18] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:06 [I 2022-10-27 00:27:18] ...trap/engines/engine.py.229: process: 0.54391 | load: 0.00253 [I 2022-10-27 00:27:18] ...trap/engines/engine.py.230: loss: 0.20544 [I 2022-10-27 00:27:18] ...optimizers/trijoint.py.151: total_norm: 32.426659 [I 2022-10-27 00:27:24] ...trap/engines/engine.py.224: train: epoch 38 | batch 130/2304 [I 2022-10-27 00:27:24] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:00 [I 2022-10-27 00:27:24] ...trap/engines/engine.py.229: process: 0.54843 | load: 0.00280 [I 2022-10-27 00:27:24] ...trap/engines/engine.py.230: loss: 0.17903 [I 2022-10-27 00:27:24] ...optimizers/trijoint.py.151: total_norm: 17.922901 [I 2022-10-27 00:27:29] ...trap/engines/engine.py.224: train: epoch 38 | batch 140/2304 [I 2022-10-27 00:27:29] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:04 [I 2022-10-27 00:27:29] ...trap/engines/engine.py.229: process: 0.56686 | load: 0.00255 [I 2022-10-27 00:27:29] ...trap/engines/engine.py.230: loss: 0.21540 [I 2022-10-27 00:27:29] ...optimizers/trijoint.py.151: total_norm: 26.124630 [I 2022-10-27 00:27:35] ...trap/engines/engine.py.224: train: epoch 38 | batch 150/2304 [I 2022-10-27 00:27:35] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:09 [I 2022-10-27 00:27:35] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00260 [I 2022-10-27 00:27:35] ...trap/engines/engine.py.230: loss: 0.14490 [I 2022-10-27 00:27:35] ...optimizers/trijoint.py.151: total_norm: 21.008451 [I 2022-10-27 00:27:40] ...trap/engines/engine.py.224: train: epoch 38 | batch 160/2304 [I 2022-10-27 00:27:40] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:54 [I 2022-10-27 00:27:40] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00253 [I 2022-10-27 00:27:40] ...trap/engines/engine.py.230: loss: 0.17255 [I 2022-10-27 00:27:40] ...optimizers/trijoint.py.151: total_norm: 17.447290 [I 2022-10-27 00:27:46] ...trap/engines/engine.py.224: train: epoch 38 | batch 170/2304 [I 2022-10-27 00:27:46] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:57 [I 2022-10-27 00:27:46] ...trap/engines/engine.py.229: process: 0.56697 | load: 0.00271 [I 2022-10-27 00:27:46] ...trap/engines/engine.py.230: loss: 0.14690 [I 2022-10-27 00:27:46] ...optimizers/trijoint.py.151: total_norm: 46.326015 [I 2022-10-27 00:27:52] ...trap/engines/engine.py.224: train: epoch 38 | batch 180/2304 [I 2022-10-27 00:27:52] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:47 [I 2022-10-27 00:27:52] ...trap/engines/engine.py.229: process: 0.57565 | load: 0.00249 [I 2022-10-27 00:27:52] ...trap/engines/engine.py.230: loss: 0.07981 [I 2022-10-27 00:27:52] ...optimizers/trijoint.py.151: total_norm: 8.088311 [I 2022-10-27 00:27:57] ...trap/engines/engine.py.224: train: epoch 38 | batch 190/2304 [I 2022-10-27 00:27:57] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:36 [I 2022-10-27 00:27:57] ...trap/engines/engine.py.229: process: 0.54731 | load: 0.00249 [I 2022-10-27 00:27:57] ...trap/engines/engine.py.230: loss: 0.20172 [I 2022-10-27 00:27:57] ...optimizers/trijoint.py.151: total_norm: 25.783381 [I 2022-10-27 00:28:03] ...trap/engines/engine.py.224: train: epoch 38 | batch 200/2304 [I 2022-10-27 00:28:03] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:34 [I 2022-10-27 00:28:03] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00263 [I 2022-10-27 00:28:03] ...trap/engines/engine.py.230: loss: 0.19973 [I 2022-10-27 00:28:03] ...optimizers/trijoint.py.151: total_norm: 25.535107 [I 2022-10-27 00:28:09] ...trap/engines/engine.py.224: train: epoch 38 | batch 210/2304 [I 2022-10-27 00:28:09] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:41 [I 2022-10-27 00:28:09] ...trap/engines/engine.py.229: process: 0.54304 | load: 0.00256 [I 2022-10-27 00:28:09] ...trap/engines/engine.py.230: loss: 0.15624 [I 2022-10-27 00:28:09] ...optimizers/trijoint.py.151: total_norm: 17.657438 [I 2022-10-27 00:28:14] ...trap/engines/engine.py.224: train: epoch 38 | batch 220/2304 [I 2022-10-27 00:28:14] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:26 [I 2022-10-27 00:28:14] ...trap/engines/engine.py.229: process: 0.57043 | load: 0.00262 [I 2022-10-27 00:28:14] ...trap/engines/engine.py.230: loss: 0.16360 [I 2022-10-27 00:28:14] ...optimizers/trijoint.py.151: total_norm: 13.173593 [I 2022-10-27 00:28:20] ...trap/engines/engine.py.224: train: epoch 38 | batch 230/2304 [I 2022-10-27 00:28:20] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:24 [I 2022-10-27 00:28:20] ...trap/engines/engine.py.229: process: 0.55302 | load: 0.00248 [I 2022-10-27 00:28:20] ...trap/engines/engine.py.230: loss: 0.15341 [I 2022-10-27 00:28:20] ...optimizers/trijoint.py.151: total_norm: 12.522096 [I 2022-10-27 00:28:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 240/2304 [I 2022-10-27 00:28:25] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:20 [I 2022-10-27 00:28:25] ...trap/engines/engine.py.229: process: 0.56577 | load: 0.00287 [I 2022-10-27 00:28:25] ...trap/engines/engine.py.230: loss: 0.21742 [I 2022-10-27 00:28:25] ...optimizers/trijoint.py.151: total_norm: 41.946568 [I 2022-10-27 00:28:31] ...trap/engines/engine.py.224: train: epoch 38 | batch 250/2304 [I 2022-10-27 00:28:31] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:06 [I 2022-10-27 00:28:31] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00275 [I 2022-10-27 00:28:31] ...trap/engines/engine.py.230: loss: 0.24684 [I 2022-10-27 00:28:31] ...optimizers/trijoint.py.151: total_norm: 16.318714 [I 2022-10-27 00:28:37] ...trap/engines/engine.py.224: train: epoch 38 | batch 260/2304 [I 2022-10-27 00:28:37] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:58 [I 2022-10-27 00:28:37] ...trap/engines/engine.py.229: process: 0.54909 | load: 0.00271 [I 2022-10-27 00:28:37] ...trap/engines/engine.py.230: loss: 0.10649 [I 2022-10-27 00:28:37] ...optimizers/trijoint.py.151: total_norm: 9.064106 [I 2022-10-27 00:28:42] ...trap/engines/engine.py.224: train: epoch 38 | batch 270/2304 [I 2022-10-27 00:28:42] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:58 [I 2022-10-27 00:28:42] ...trap/engines/engine.py.229: process: 0.55510 | load: 0.00246 [I 2022-10-27 00:28:42] ...trap/engines/engine.py.230: loss: 0.15294 [I 2022-10-27 00:28:42] ...optimizers/trijoint.py.151: total_norm: 10.177667 [I 2022-10-27 00:28:48] ...trap/engines/engine.py.224: train: epoch 38 | batch 280/2304 [I 2022-10-27 00:28:48] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:56 [I 2022-10-27 00:28:48] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00247 [I 2022-10-27 00:28:48] ...trap/engines/engine.py.230: loss: 0.11647 [I 2022-10-27 00:28:48] ...optimizers/trijoint.py.151: total_norm: 13.196318 [I 2022-10-27 00:28:53] ...trap/engines/engine.py.224: train: epoch 38 | batch 290/2304 [I 2022-10-27 00:28:53] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:50 [I 2022-10-27 00:28:53] ...trap/engines/engine.py.229: process: 0.56773 | load: 0.00244 [I 2022-10-27 00:28:53] ...trap/engines/engine.py.230: loss: 0.17114 [I 2022-10-27 00:28:53] ...optimizers/trijoint.py.151: total_norm: 20.320889 [I 2022-10-27 00:28:59] ...trap/engines/engine.py.224: train: epoch 38 | batch 300/2304 [I 2022-10-27 00:28:59] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:23 [I 2022-10-27 00:28:59] ...trap/engines/engine.py.229: process: 0.54700 | load: 0.00246 [I 2022-10-27 00:28:59] ...trap/engines/engine.py.230: loss: 0.16597 [I 2022-10-27 00:28:59] ...optimizers/trijoint.py.151: total_norm: 22.568453 [I 2022-10-27 00:29:05] ...trap/engines/engine.py.224: train: epoch 38 | batch 310/2304 [I 2022-10-27 00:29:05] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:29 [I 2022-10-27 00:29:05] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00241 [I 2022-10-27 00:29:05] ...trap/engines/engine.py.230: loss: 0.17633 [I 2022-10-27 00:29:05] ...optimizers/trijoint.py.151: total_norm: 20.121906 [I 2022-10-27 00:29:10] ...trap/engines/engine.py.224: train: epoch 38 | batch 320/2304 [I 2022-10-27 00:29:10] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:34 [I 2022-10-27 00:29:10] ...trap/engines/engine.py.229: process: 0.56724 | load: 0.00252 [I 2022-10-27 00:29:10] ...trap/engines/engine.py.230: loss: 0.13291 [I 2022-10-27 00:29:10] ...optimizers/trijoint.py.151: total_norm: 15.217400 [I 2022-10-27 00:29:16] ...trap/engines/engine.py.224: train: epoch 38 | batch 330/2304 [I 2022-10-27 00:29:16] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:29 [I 2022-10-27 00:29:16] ...trap/engines/engine.py.229: process: 0.57545 | load: 0.00244 [I 2022-10-27 00:29:16] ...trap/engines/engine.py.230: loss: 0.14209 [I 2022-10-27 00:29:16] ...optimizers/trijoint.py.151: total_norm: 12.853844 [I 2022-10-27 00:29:21] ...trap/engines/engine.py.224: train: epoch 38 | batch 340/2304 [I 2022-10-27 00:29:21] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:21 [I 2022-10-27 00:29:21] ...trap/engines/engine.py.229: process: 0.55968 | load: 0.00246 [I 2022-10-27 00:29:21] ...trap/engines/engine.py.230: loss: 0.16340 [I 2022-10-27 00:29:21] ...optimizers/trijoint.py.151: total_norm: 11.283895 [I 2022-10-27 00:29:27] ...trap/engines/engine.py.224: train: epoch 38 | batch 350/2304 [I 2022-10-27 00:29:27] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:11 [I 2022-10-27 00:29:27] ...trap/engines/engine.py.229: process: 0.57088 | load: 0.00246 [I 2022-10-27 00:29:27] ...trap/engines/engine.py.230: loss: 0.14641 [I 2022-10-27 00:29:27] ...optimizers/trijoint.py.151: total_norm: 15.762143 [I 2022-10-27 00:29:33] ...trap/engines/engine.py.224: train: epoch 38 | batch 360/2304 [I 2022-10-27 00:29:33] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:09 [I 2022-10-27 00:29:33] ...trap/engines/engine.py.229: process: 0.55243 | load: 0.00241 [I 2022-10-27 00:29:33] ...trap/engines/engine.py.230: loss: 0.15553 [I 2022-10-27 00:29:33] ...optimizers/trijoint.py.151: total_norm: 15.365349 [I 2022-10-27 00:29:38] ...trap/engines/engine.py.224: train: epoch 38 | batch 370/2304 [I 2022-10-27 00:29:38] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:58 [I 2022-10-27 00:29:38] ...trap/engines/engine.py.229: process: 0.54548 | load: 0.00245 [I 2022-10-27 00:29:38] ...trap/engines/engine.py.230: loss: 0.17395 [I 2022-10-27 00:29:38] ...optimizers/trijoint.py.151: total_norm: 25.179546 [I 2022-10-27 00:29:44] ...trap/engines/engine.py.224: train: epoch 38 | batch 380/2304 [I 2022-10-27 00:29:44] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:18:03 [I 2022-10-27 00:29:44] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00242 [I 2022-10-27 00:29:44] ...trap/engines/engine.py.230: loss: 0.16132 [I 2022-10-27 00:29:44] ...optimizers/trijoint.py.151: total_norm: 34.587856 [I 2022-10-27 00:29:49] ...trap/engines/engine.py.224: train: epoch 38 | batch 390/2304 [I 2022-10-27 00:29:49] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:51 [I 2022-10-27 00:29:49] ...trap/engines/engine.py.229: process: 0.57266 | load: 0.00267 [I 2022-10-27 00:29:49] ...trap/engines/engine.py.230: loss: 0.20499 [I 2022-10-27 00:29:49] ...optimizers/trijoint.py.151: total_norm: 22.131151 [I 2022-10-27 00:29:55] ...trap/engines/engine.py.224: train: epoch 38 | batch 400/2304 [I 2022-10-27 00:29:55] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:41 [I 2022-10-27 00:29:55] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00247 [I 2022-10-27 00:29:55] ...trap/engines/engine.py.230: loss: 0.17607 [I 2022-10-27 00:29:55] ...optimizers/trijoint.py.151: total_norm: 23.278687 [I 2022-10-27 00:30:01] ...trap/engines/engine.py.224: train: epoch 38 | batch 410/2304 [I 2022-10-27 00:30:01] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:38 [I 2022-10-27 00:30:01] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00250 [I 2022-10-27 00:30:01] ...trap/engines/engine.py.230: loss: 0.16191 [I 2022-10-27 00:30:01] ...optimizers/trijoint.py.151: total_norm: 42.024464 [I 2022-10-27 00:30:06] ...trap/engines/engine.py.224: train: epoch 38 | batch 420/2304 [I 2022-10-27 00:30:06] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:25 [I 2022-10-27 00:30:06] ...trap/engines/engine.py.229: process: 0.54403 | load: 0.00257 [I 2022-10-27 00:30:06] ...trap/engines/engine.py.230: loss: 0.16716 [I 2022-10-27 00:30:06] ...optimizers/trijoint.py.151: total_norm: 15.261100 [I 2022-10-27 00:30:12] ...trap/engines/engine.py.224: train: epoch 38 | batch 430/2304 [I 2022-10-27 00:30:12] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:18:07 [I 2022-10-27 00:30:12] ...trap/engines/engine.py.229: process: 0.56568 | load: 0.00247 [I 2022-10-27 00:30:12] ...trap/engines/engine.py.230: loss: 0.12991 [I 2022-10-27 00:30:12] ...optimizers/trijoint.py.151: total_norm: 10.032465 [I 2022-10-27 00:30:18] ...trap/engines/engine.py.224: train: epoch 38 | batch 440/2304 [I 2022-10-27 00:30:18] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:26 [I 2022-10-27 00:30:18] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00244 [I 2022-10-27 00:30:18] ...trap/engines/engine.py.230: loss: 0.20340 [I 2022-10-27 00:30:18] ...optimizers/trijoint.py.151: total_norm: 21.544048 [I 2022-10-27 00:30:23] ...trap/engines/engine.py.224: train: epoch 38 | batch 450/2304 [I 2022-10-27 00:30:23] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:09 [I 2022-10-27 00:30:23] ...trap/engines/engine.py.229: process: 0.54228 | load: 0.00261 [I 2022-10-27 00:30:23] ...trap/engines/engine.py.230: loss: 0.16628 [I 2022-10-27 00:30:23] ...optimizers/trijoint.py.151: total_norm: 25.351355 [I 2022-10-27 00:30:29] ...trap/engines/engine.py.224: train: epoch 38 | batch 460/2304 [I 2022-10-27 00:30:29] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:08 [I 2022-10-27 00:30:29] ...trap/engines/engine.py.229: process: 0.54808 | load: 0.00242 [I 2022-10-27 00:30:29] ...trap/engines/engine.py.230: loss: 0.27153 [I 2022-10-27 00:30:29] ...optimizers/trijoint.py.151: total_norm: 29.828104 [I 2022-10-27 00:30:34] ...trap/engines/engine.py.224: train: epoch 38 | batch 470/2304 [I 2022-10-27 00:30:34] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:01 [I 2022-10-27 00:30:34] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00243 [I 2022-10-27 00:30:34] ...trap/engines/engine.py.230: loss: 0.13777 [I 2022-10-27 00:30:34] ...optimizers/trijoint.py.151: total_norm: 9.673835 [I 2022-10-27 00:30:40] ...trap/engines/engine.py.224: train: epoch 38 | batch 480/2304 [I 2022-10-27 00:30:40] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:54 [I 2022-10-27 00:30:40] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00247 [I 2022-10-27 00:30:40] ...trap/engines/engine.py.230: loss: 0.22773 [I 2022-10-27 00:30:40] ...optimizers/trijoint.py.151: total_norm: 18.694996 [I 2022-10-27 00:30:45] ...trap/engines/engine.py.224: train: epoch 38 | batch 490/2304 [I 2022-10-27 00:30:45] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:50 [I 2022-10-27 00:30:45] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00241 [I 2022-10-27 00:30:45] ...trap/engines/engine.py.230: loss: 0.16000 [I 2022-10-27 00:30:45] ...optimizers/trijoint.py.151: total_norm: 13.410194 [I 2022-10-27 00:30:51] ...trap/engines/engine.py.224: train: epoch 38 | batch 500/2304 [I 2022-10-27 00:30:51] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:51 [I 2022-10-27 00:30:51] ...trap/engines/engine.py.229: process: 0.54645 | load: 0.00243 [I 2022-10-27 00:30:51] ...trap/engines/engine.py.230: loss: 0.16718 [I 2022-10-27 00:30:51] ...optimizers/trijoint.py.151: total_norm: 24.084112 [I 2022-10-27 00:30:57] ...trap/engines/engine.py.224: train: epoch 38 | batch 510/2304 [I 2022-10-27 00:30:57] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:42 [I 2022-10-27 00:30:57] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00243 [I 2022-10-27 00:30:57] ...trap/engines/engine.py.230: loss: 0.22261 [I 2022-10-27 00:30:57] ...optimizers/trijoint.py.151: total_norm: 22.442062 [I 2022-10-27 00:31:02] ...trap/engines/engine.py.224: train: epoch 38 | batch 520/2304 [I 2022-10-27 00:31:02] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:31 [I 2022-10-27 00:31:02] ...trap/engines/engine.py.229: process: 0.54865 | load: 0.00239 [I 2022-10-27 00:31:02] ...trap/engines/engine.py.230: loss: 0.14161 [I 2022-10-27 00:31:02] ...optimizers/trijoint.py.151: total_norm: 15.818089 [I 2022-10-27 00:31:08] ...trap/engines/engine.py.224: train: epoch 38 | batch 530/2304 [I 2022-10-27 00:31:08] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:26 [I 2022-10-27 00:31:08] ...trap/engines/engine.py.229: process: 0.56048 | load: 0.00241 [I 2022-10-27 00:31:08] ...trap/engines/engine.py.230: loss: 0.15755 [I 2022-10-27 00:31:08] ...optimizers/trijoint.py.151: total_norm: 16.820150 [I 2022-10-27 00:31:13] ...trap/engines/engine.py.224: train: epoch 38 | batch 540/2304 [I 2022-10-27 00:31:13] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:18 [I 2022-10-27 00:31:13] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00238 [I 2022-10-27 00:31:13] ...trap/engines/engine.py.230: loss: 0.26233 [I 2022-10-27 00:31:13] ...optimizers/trijoint.py.151: total_norm: 15.218698 [I 2022-10-27 00:31:19] ...trap/engines/engine.py.224: train: epoch 38 | batch 550/2304 [I 2022-10-27 00:31:19] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:10 [I 2022-10-27 00:31:19] ...trap/engines/engine.py.229: process: 0.56116 | load: 0.00251 [I 2022-10-27 00:31:19] ...trap/engines/engine.py.230: loss: 0.12542 [I 2022-10-27 00:31:19] ...optimizers/trijoint.py.151: total_norm: 10.178721 [I 2022-10-27 00:31:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 560/2304 [I 2022-10-27 00:31:25] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:11 [I 2022-10-27 00:31:25] ...trap/engines/engine.py.229: process: 0.55669 | load: 0.00251 [I 2022-10-27 00:31:25] ...trap/engines/engine.py.230: loss: 0.17687 [I 2022-10-27 00:31:25] ...optimizers/trijoint.py.151: total_norm: 20.973469 [I 2022-10-27 00:31:30] ...trap/engines/engine.py.224: train: epoch 38 | batch 570/2304 [I 2022-10-27 00:31:30] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:01 [I 2022-10-27 00:31:30] ...trap/engines/engine.py.229: process: 0.56048 | load: 0.00262 [I 2022-10-27 00:31:30] ...trap/engines/engine.py.230: loss: 0.23534 [I 2022-10-27 00:31:30] ...optimizers/trijoint.py.151: total_norm: 26.948488 [I 2022-10-27 00:31:36] ...trap/engines/engine.py.224: train: epoch 38 | batch 580/2304 [I 2022-10-27 00:31:36] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:59 [I 2022-10-27 00:31:36] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00242 [I 2022-10-27 00:31:36] ...trap/engines/engine.py.230: loss: 0.14840 [I 2022-10-27 00:31:36] ...optimizers/trijoint.py.151: total_norm: 21.998062 [I 2022-10-27 00:31:41] ...trap/engines/engine.py.224: train: epoch 38 | batch 590/2304 [I 2022-10-27 00:31:41] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:52 [I 2022-10-27 00:31:41] ...trap/engines/engine.py.229: process: 0.56432 | load: 0.00243 [I 2022-10-27 00:31:41] ...trap/engines/engine.py.230: loss: 0.18173 [I 2022-10-27 00:31:41] ...optimizers/trijoint.py.151: total_norm: 16.303835 [I 2022-10-27 00:31:47] ...trap/engines/engine.py.224: train: epoch 38 | batch 600/2304 [I 2022-10-27 00:31:47] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:49 [I 2022-10-27 00:31:47] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00242 [I 2022-10-27 00:31:47] ...trap/engines/engine.py.230: loss: 0.21352 [I 2022-10-27 00:31:47] ...optimizers/trijoint.py.151: total_norm: 14.678257 [I 2022-10-27 00:31:52] ...trap/engines/engine.py.224: train: epoch 38 | batch 610/2304 [I 2022-10-27 00:31:52] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:43 [I 2022-10-27 00:31:52] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00247 [I 2022-10-27 00:31:52] ...trap/engines/engine.py.230: loss: 0.20661 [I 2022-10-27 00:31:52] ...optimizers/trijoint.py.151: total_norm: 15.423936 [I 2022-10-27 00:31:58] ...trap/engines/engine.py.224: train: epoch 38 | batch 620/2304 [I 2022-10-27 00:31:58] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:47 [I 2022-10-27 00:31:58] ...trap/engines/engine.py.229: process: 0.58216 | load: 0.00243 [I 2022-10-27 00:31:58] ...trap/engines/engine.py.230: loss: 0.22313 [I 2022-10-27 00:31:58] ...optimizers/trijoint.py.151: total_norm: 32.883774 [I 2022-10-27 00:32:04] ...trap/engines/engine.py.224: train: epoch 38 | batch 630/2304 [I 2022-10-27 00:32:04] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:37 [I 2022-10-27 00:32:04] ...trap/engines/engine.py.229: process: 0.56949 | load: 0.00248 [I 2022-10-27 00:32:04] ...trap/engines/engine.py.230: loss: 0.15008 [I 2022-10-27 00:32:04] ...optimizers/trijoint.py.151: total_norm: 23.455824 [I 2022-10-27 00:32:09] ...trap/engines/engine.py.224: train: epoch 38 | batch 640/2304 [I 2022-10-27 00:32:09] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:30 [I 2022-10-27 00:32:09] ...trap/engines/engine.py.229: process: 0.56968 | load: 0.00246 [I 2022-10-27 00:32:09] ...trap/engines/engine.py.230: loss: 0.11688 [I 2022-10-27 00:32:09] ...optimizers/trijoint.py.151: total_norm: 20.459793 [I 2022-10-27 00:32:15] ...trap/engines/engine.py.224: train: epoch 38 | batch 650/2304 [I 2022-10-27 00:32:15] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:20 [I 2022-10-27 00:32:15] ...trap/engines/engine.py.229: process: 0.55426 | load: 0.00245 [I 2022-10-27 00:32:15] ...trap/engines/engine.py.230: loss: 0.19392 [I 2022-10-27 00:32:15] ...optimizers/trijoint.py.151: total_norm: 22.679264 [I 2022-10-27 00:32:20] ...trap/engines/engine.py.224: train: epoch 38 | batch 660/2304 [I 2022-10-27 00:32:20] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:14 [I 2022-10-27 00:32:20] ...trap/engines/engine.py.229: process: 0.54817 | load: 0.00241 [I 2022-10-27 00:32:20] ...trap/engines/engine.py.230: loss: 0.17989 [I 2022-10-27 00:32:20] ...optimizers/trijoint.py.151: total_norm: 28.190506 [I 2022-10-27 00:32:26] ...trap/engines/engine.py.224: train: epoch 38 | batch 670/2304 [I 2022-10-27 00:32:26] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:09 [I 2022-10-27 00:32:26] ...trap/engines/engine.py.229: process: 0.55800 | load: 0.00247 [I 2022-10-27 00:32:26] ...trap/engines/engine.py.230: loss: 0.16177 [I 2022-10-27 00:32:26] ...optimizers/trijoint.py.151: total_norm: 13.781311 [I 2022-10-27 00:32:32] ...trap/engines/engine.py.224: train: epoch 38 | batch 680/2304 [I 2022-10-27 00:32:32] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:07 [I 2022-10-27 00:32:32] ...trap/engines/engine.py.229: process: 0.55202 | load: 0.00246 [I 2022-10-27 00:32:32] ...trap/engines/engine.py.230: loss: 0.22808 [I 2022-10-27 00:32:32] ...optimizers/trijoint.py.151: total_norm: 13.887913 [I 2022-10-27 00:32:37] ...trap/engines/engine.py.224: train: epoch 38 | batch 690/2304 [I 2022-10-27 00:32:37] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:55 [I 2022-10-27 00:32:37] ...trap/engines/engine.py.229: process: 0.55451 | load: 0.00241 [I 2022-10-27 00:32:37] ...trap/engines/engine.py.230: loss: 0.12643 [I 2022-10-27 00:32:37] ...optimizers/trijoint.py.151: total_norm: 16.908705 [I 2022-10-27 00:32:43] ...trap/engines/engine.py.224: train: epoch 38 | batch 700/2304 [I 2022-10-27 00:32:43] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:49 [I 2022-10-27 00:32:43] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00242 [I 2022-10-27 00:32:43] ...trap/engines/engine.py.230: loss: 0.18806 [I 2022-10-27 00:32:43] ...optimizers/trijoint.py.151: total_norm: 22.430443 [I 2022-10-27 00:32:48] ...trap/engines/engine.py.224: train: epoch 38 | batch 710/2304 [I 2022-10-27 00:32:48] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:46 [I 2022-10-27 00:32:48] ...trap/engines/engine.py.229: process: 0.55596 | load: 0.00245 [I 2022-10-27 00:32:48] ...trap/engines/engine.py.230: loss: 0.18252 [I 2022-10-27 00:32:48] ...optimizers/trijoint.py.151: total_norm: 48.315681 [I 2022-10-27 00:32:54] ...trap/engines/engine.py.224: train: epoch 38 | batch 720/2304 [I 2022-10-27 00:32:54] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:39 [I 2022-10-27 00:32:54] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00242 [I 2022-10-27 00:32:54] ...trap/engines/engine.py.230: loss: 0.16983 [I 2022-10-27 00:32:54] ...optimizers/trijoint.py.151: total_norm: 13.474473 [I 2022-10-27 00:32:59] ...trap/engines/engine.py.224: train: epoch 38 | batch 730/2304 [I 2022-10-27 00:32:59] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:36 [I 2022-10-27 00:32:59] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00243 [I 2022-10-27 00:32:59] ...trap/engines/engine.py.230: loss: 0.18150 [I 2022-10-27 00:32:59] ...optimizers/trijoint.py.151: total_norm: 29.423626 [I 2022-10-27 00:33:05] ...trap/engines/engine.py.224: train: epoch 38 | batch 740/2304 [I 2022-10-27 00:33:05] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:32 [I 2022-10-27 00:33:05] ...trap/engines/engine.py.229: process: 0.55225 | load: 0.00239 [I 2022-10-27 00:33:05] ...trap/engines/engine.py.230: loss: 0.21933 [I 2022-10-27 00:33:05] ...optimizers/trijoint.py.151: total_norm: 13.351288 [I 2022-10-27 00:33:11] ...trap/engines/engine.py.224: train: epoch 38 | batch 750/2304 [I 2022-10-27 00:33:11] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:24 [I 2022-10-27 00:33:11] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00255 [I 2022-10-27 00:33:11] ...trap/engines/engine.py.230: loss: 0.24018 [I 2022-10-27 00:33:11] ...optimizers/trijoint.py.151: total_norm: 33.540821 [I 2022-10-27 00:33:16] ...trap/engines/engine.py.224: train: epoch 38 | batch 760/2304 [I 2022-10-27 00:33:16] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:15 [I 2022-10-27 00:33:16] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00247 [I 2022-10-27 00:33:16] ...trap/engines/engine.py.230: loss: 0.15222 [I 2022-10-27 00:33:16] ...optimizers/trijoint.py.151: total_norm: 13.559117 [I 2022-10-27 00:33:22] ...trap/engines/engine.py.224: train: epoch 38 | batch 770/2304 [I 2022-10-27 00:33:22] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:14 [I 2022-10-27 00:33:22] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00247 [I 2022-10-27 00:33:22] ...trap/engines/engine.py.230: loss: 0.17956 [I 2022-10-27 00:33:22] ...optimizers/trijoint.py.151: total_norm: 18.900021 [I 2022-10-27 00:33:27] ...trap/engines/engine.py.224: train: epoch 38 | batch 780/2304 [I 2022-10-27 00:33:27] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:14 [I 2022-10-27 00:33:27] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00243 [I 2022-10-27 00:33:27] ...trap/engines/engine.py.230: loss: 0.21624 [I 2022-10-27 00:33:27] ...optimizers/trijoint.py.151: total_norm: 11.628942 [I 2022-10-27 00:33:33] ...trap/engines/engine.py.224: train: epoch 38 | batch 790/2304 [I 2022-10-27 00:33:33] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:13:57 [I 2022-10-27 00:33:33] ...trap/engines/engine.py.229: process: 0.54164 | load: 0.00247 [I 2022-10-27 00:33:33] ...trap/engines/engine.py.230: loss: 0.18714 [I 2022-10-27 00:33:33] ...optimizers/trijoint.py.151: total_norm: 16.690563 [I 2022-10-27 00:33:39] ...trap/engines/engine.py.224: train: epoch 38 | batch 800/2304 [I 2022-10-27 00:33:39] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:03 [I 2022-10-27 00:33:39] ...trap/engines/engine.py.229: process: 0.55342 | load: 0.00243 [I 2022-10-27 00:33:39] ...trap/engines/engine.py.230: loss: 0.13933 [I 2022-10-27 00:33:39] ...optimizers/trijoint.py.151: total_norm: 8.124052 [I 2022-10-27 00:33:44] ...trap/engines/engine.py.224: train: epoch 38 | batch 810/2304 [I 2022-10-27 00:33:44] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:48 [I 2022-10-27 00:33:44] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00243 [I 2022-10-27 00:33:44] ...trap/engines/engine.py.230: loss: 0.17742 [I 2022-10-27 00:33:44] ...optimizers/trijoint.py.151: total_norm: 18.877115 [I 2022-10-27 00:33:50] ...trap/engines/engine.py.224: train: epoch 38 | batch 820/2304 [I 2022-10-27 00:33:50] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:47 [I 2022-10-27 00:33:50] ...trap/engines/engine.py.229: process: 0.54947 | load: 0.00244 [I 2022-10-27 00:33:50] ...trap/engines/engine.py.230: loss: 0.13006 [I 2022-10-27 00:33:50] ...optimizers/trijoint.py.151: total_norm: 19.308880 [I 2022-10-27 00:33:55] ...trap/engines/engine.py.224: train: epoch 38 | batch 830/2304 [I 2022-10-27 00:33:55] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:42 [I 2022-10-27 00:33:55] ...trap/engines/engine.py.229: process: 0.56154 | load: 0.00248 [I 2022-10-27 00:33:55] ...trap/engines/engine.py.230: loss: 0.22804 [I 2022-10-27 00:33:55] ...optimizers/trijoint.py.151: total_norm: 16.495972 [I 2022-10-27 00:34:01] ...trap/engines/engine.py.224: train: epoch 38 | batch 840/2304 [I 2022-10-27 00:34:01] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:35 [I 2022-10-27 00:34:01] ...trap/engines/engine.py.229: process: 0.54901 | load: 0.00247 [I 2022-10-27 00:34:01] ...trap/engines/engine.py.230: loss: 0.17142 [I 2022-10-27 00:34:01] ...optimizers/trijoint.py.151: total_norm: 20.250416 [I 2022-10-27 00:34:06] ...trap/engines/engine.py.224: train: epoch 38 | batch 850/2304 [I 2022-10-27 00:34:06] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:26 [I 2022-10-27 00:34:06] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00244 [I 2022-10-27 00:34:06] ...trap/engines/engine.py.230: loss: 0.14003 [I 2022-10-27 00:34:06] ...optimizers/trijoint.py.151: total_norm: 11.086071 [I 2022-10-27 00:34:12] ...trap/engines/engine.py.224: train: epoch 38 | batch 860/2304 [I 2022-10-27 00:34:12] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:22 [I 2022-10-27 00:34:12] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00244 [I 2022-10-27 00:34:12] ...trap/engines/engine.py.230: loss: 0.13517 [I 2022-10-27 00:34:12] ...optimizers/trijoint.py.151: total_norm: 19.384211 [I 2022-10-27 00:34:18] ...trap/engines/engine.py.224: train: epoch 38 | batch 870/2304 [I 2022-10-27 00:34:18] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:19 [I 2022-10-27 00:34:18] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00245 [I 2022-10-27 00:34:18] ...trap/engines/engine.py.230: loss: 0.27963 [I 2022-10-27 00:34:18] ...optimizers/trijoint.py.151: total_norm: 31.621532 [I 2022-10-27 00:34:23] ...trap/engines/engine.py.224: train: epoch 38 | batch 880/2304 [I 2022-10-27 00:34:23] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:13 [I 2022-10-27 00:34:23] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00251 [I 2022-10-27 00:34:23] ...trap/engines/engine.py.230: loss: 0.16981 [I 2022-10-27 00:34:23] ...optimizers/trijoint.py.151: total_norm: 16.204973 [I 2022-10-27 00:34:29] ...trap/engines/engine.py.224: train: epoch 38 | batch 890/2304 [I 2022-10-27 00:34:29] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:07 [I 2022-10-27 00:34:29] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00246 [I 2022-10-27 00:34:29] ...trap/engines/engine.py.230: loss: 0.23638 [I 2022-10-27 00:34:29] ...optimizers/trijoint.py.151: total_norm: 39.742748 [I 2022-10-27 00:34:34] ...trap/engines/engine.py.224: train: epoch 38 | batch 900/2304 [I 2022-10-27 00:34:34] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:12:59 [I 2022-10-27 00:34:34] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00241 [I 2022-10-27 00:34:34] ...trap/engines/engine.py.230: loss: 0.15933 [I 2022-10-27 00:34:34] ...optimizers/trijoint.py.151: total_norm: 13.838303 [I 2022-10-27 00:34:40] ...trap/engines/engine.py.224: train: epoch 38 | batch 910/2304 [I 2022-10-27 00:34:40] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:48 [I 2022-10-27 00:34:40] ...trap/engines/engine.py.229: process: 0.54722 | load: 0.00243 [I 2022-10-27 00:34:40] ...trap/engines/engine.py.230: loss: 0.23285 [I 2022-10-27 00:34:40] ...optimizers/trijoint.py.151: total_norm: 20.363224 [I 2022-10-27 00:34:45] ...trap/engines/engine.py.224: train: epoch 38 | batch 920/2304 [I 2022-10-27 00:34:45] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:50 [I 2022-10-27 00:34:45] ...trap/engines/engine.py.229: process: 0.55335 | load: 0.00243 [I 2022-10-27 00:34:45] ...trap/engines/engine.py.230: loss: 0.25400 [I 2022-10-27 00:34:45] ...optimizers/trijoint.py.151: total_norm: 22.316366 [I 2022-10-27 00:34:51] ...trap/engines/engine.py.224: train: epoch 38 | batch 930/2304 [I 2022-10-27 00:34:51] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:49 [I 2022-10-27 00:34:51] ...trap/engines/engine.py.229: process: 0.56318 | load: 0.00241 [I 2022-10-27 00:34:51] ...trap/engines/engine.py.230: loss: 0.17192 [I 2022-10-27 00:34:51] ...optimizers/trijoint.py.151: total_norm: 24.231424 [I 2022-10-27 00:34:57] ...trap/engines/engine.py.224: train: epoch 38 | batch 940/2304 [I 2022-10-27 00:34:57] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:48 [I 2022-10-27 00:34:57] ...trap/engines/engine.py.229: process: 0.55372 | load: 0.00244 [I 2022-10-27 00:34:57] ...trap/engines/engine.py.230: loss: 0.18903 [I 2022-10-27 00:34:57] ...optimizers/trijoint.py.151: total_norm: 27.993214 [I 2022-10-27 00:35:02] ...trap/engines/engine.py.224: train: epoch 38 | batch 950/2304 [I 2022-10-27 00:35:02] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:36 [I 2022-10-27 00:35:02] ...trap/engines/engine.py.229: process: 0.56411 | load: 0.00250 [I 2022-10-27 00:35:02] ...trap/engines/engine.py.230: loss: 0.14113 [I 2022-10-27 00:35:02] ...optimizers/trijoint.py.151: total_norm: 9.454770 [I 2022-10-27 00:35:08] ...trap/engines/engine.py.224: train: epoch 38 | batch 960/2304 [I 2022-10-27 00:35:08] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:26 [I 2022-10-27 00:35:08] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00245 [I 2022-10-27 00:35:08] ...trap/engines/engine.py.230: loss: 0.17648 [I 2022-10-27 00:35:08] ...optimizers/trijoint.py.151: total_norm: 13.284973 [I 2022-10-27 00:35:13] ...trap/engines/engine.py.224: train: epoch 38 | batch 970/2304 [I 2022-10-27 00:35:13] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:26 [I 2022-10-27 00:35:13] ...trap/engines/engine.py.229: process: 0.56950 | load: 0.00243 [I 2022-10-27 00:35:13] ...trap/engines/engine.py.230: loss: 0.19383 [I 2022-10-27 00:35:13] ...optimizers/trijoint.py.151: total_norm: 16.727766 [I 2022-10-27 00:35:19] ...trap/engines/engine.py.224: train: epoch 38 | batch 980/2304 [I 2022-10-27 00:35:19] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:15 [I 2022-10-27 00:35:19] ...trap/engines/engine.py.229: process: 0.54630 | load: 0.00244 [I 2022-10-27 00:35:19] ...trap/engines/engine.py.230: loss: 0.11689 [I 2022-10-27 00:35:19] ...optimizers/trijoint.py.151: total_norm: 12.947189 [I 2022-10-27 00:35:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 990/2304 [I 2022-10-27 00:35:25] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:13 [I 2022-10-27 00:35:25] ...trap/engines/engine.py.229: process: 0.54922 | load: 0.00246 [I 2022-10-27 00:35:25] ...trap/engines/engine.py.230: loss: 0.17859 [I 2022-10-27 00:35:25] ...optimizers/trijoint.py.151: total_norm: 22.846905 [I 2022-10-27 00:35:30] ...trap/engines/engine.py.224: train: epoch 38 | batch 1000/2304 [I 2022-10-27 00:35:30] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:04 [I 2022-10-27 00:35:30] ...trap/engines/engine.py.229: process: 0.54795 | load: 0.00241 [I 2022-10-27 00:35:30] ...trap/engines/engine.py.230: loss: 0.12231 [I 2022-10-27 00:35:30] ...optimizers/trijoint.py.151: total_norm: 12.830652 [I 2022-10-27 00:35:36] ...trap/engines/engine.py.224: train: epoch 38 | batch 1010/2304 [I 2022-10-27 00:35:36] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:10 [I 2022-10-27 00:35:36] ...trap/engines/engine.py.229: process: 0.58065 | load: 0.00247 [I 2022-10-27 00:35:36] ...trap/engines/engine.py.230: loss: 0.19940 [I 2022-10-27 00:35:36] ...optimizers/trijoint.py.151: total_norm: 10.681020 [I 2022-10-27 00:35:41] ...trap/engines/engine.py.224: train: epoch 38 | batch 1020/2304 [I 2022-10-27 00:35:41] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:12:08 [I 2022-10-27 00:35:41] ...trap/engines/engine.py.229: process: 0.57576 | load: 0.00246 [I 2022-10-27 00:35:41] ...trap/engines/engine.py.230: loss: 0.15632 [I 2022-10-27 00:35:41] ...optimizers/trijoint.py.151: total_norm: 9.208802 [I 2022-10-27 00:35:47] ...trap/engines/engine.py.224: train: epoch 38 | batch 1030/2304 [I 2022-10-27 00:35:47] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:12:32 [I 2022-10-27 00:35:47] ...trap/engines/engine.py.229: process: 0.66187 | load: 0.00274 [I 2022-10-27 00:35:47] ...trap/engines/engine.py.230: loss: 0.13535 [I 2022-10-27 00:35:47] ...optimizers/trijoint.py.151: total_norm: 14.114039 [I 2022-10-27 00:35:53] ...trap/engines/engine.py.224: train: epoch 38 | batch 1040/2304 [I 2022-10-27 00:35:53] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:53 [I 2022-10-27 00:35:53] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00249 [I 2022-10-27 00:35:53] ...trap/engines/engine.py.230: loss: 0.12288 [I 2022-10-27 00:35:53] ...optimizers/trijoint.py.151: total_norm: 16.197306 [I 2022-10-27 00:35:58] ...trap/engines/engine.py.224: train: epoch 38 | batch 1050/2304 [I 2022-10-27 00:35:58] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:41 [I 2022-10-27 00:35:58] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00244 [I 2022-10-27 00:35:58] ...trap/engines/engine.py.230: loss: 0.16447 [I 2022-10-27 00:35:58] ...optimizers/trijoint.py.151: total_norm: 28.133703 [I 2022-10-27 00:36:04] ...trap/engines/engine.py.224: train: epoch 38 | batch 1060/2304 [I 2022-10-27 00:36:04] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:35 [I 2022-10-27 00:36:04] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00239 [I 2022-10-27 00:36:04] ...trap/engines/engine.py.230: loss: 0.17455 [I 2022-10-27 00:36:04] ...optimizers/trijoint.py.151: total_norm: 17.246681 [I 2022-10-27 00:36:10] ...trap/engines/engine.py.224: train: epoch 38 | batch 1070/2304 [I 2022-10-27 00:36:10] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:28 [I 2022-10-27 00:36:10] ...trap/engines/engine.py.229: process: 0.56215 | load: 0.00243 [I 2022-10-27 00:36:10] ...trap/engines/engine.py.230: loss: 0.17311 [I 2022-10-27 00:36:10] ...optimizers/trijoint.py.151: total_norm: 27.669901 [I 2022-10-27 00:36:15] ...trap/engines/engine.py.224: train: epoch 38 | batch 1080/2304 [I 2022-10-27 00:36:15] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:20 [I 2022-10-27 00:36:15] ...trap/engines/engine.py.229: process: 0.56430 | load: 0.00243 [I 2022-10-27 00:36:15] ...trap/engines/engine.py.230: loss: 0.23282 [I 2022-10-27 00:36:15] ...optimizers/trijoint.py.151: total_norm: 14.026813 [I 2022-10-27 00:36:21] ...trap/engines/engine.py.224: train: epoch 38 | batch 1090/2304 [I 2022-10-27 00:36:21] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:12 [I 2022-10-27 00:36:21] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00244 [I 2022-10-27 00:36:21] ...trap/engines/engine.py.230: loss: 0.11210 [I 2022-10-27 00:36:21] ...optimizers/trijoint.py.151: total_norm: 13.339304 [I 2022-10-27 00:36:26] ...trap/engines/engine.py.224: train: epoch 38 | batch 1100/2304 [I 2022-10-27 00:36:26] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:21 [I 2022-10-27 00:36:26] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00243 [I 2022-10-27 00:36:26] ...trap/engines/engine.py.230: loss: 0.14760 [I 2022-10-27 00:36:26] ...optimizers/trijoint.py.151: total_norm: 9.234676 [I 2022-10-27 00:36:32] ...trap/engines/engine.py.224: train: epoch 38 | batch 1110/2304 [I 2022-10-27 00:36:32] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:06 [I 2022-10-27 00:36:32] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00241 [I 2022-10-27 00:36:32] ...trap/engines/engine.py.230: loss: 0.15052 [I 2022-10-27 00:36:32] ...optimizers/trijoint.py.151: total_norm: 8.838582 [I 2022-10-27 00:36:38] ...trap/engines/engine.py.224: train: epoch 38 | batch 1120/2304 [I 2022-10-27 00:36:38] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:10:57 [I 2022-10-27 00:36:38] ...trap/engines/engine.py.229: process: 0.55938 | load: 0.00240 [I 2022-10-27 00:36:38] ...trap/engines/engine.py.230: loss: 0.16500 [I 2022-10-27 00:36:38] ...optimizers/trijoint.py.151: total_norm: 15.499138 [I 2022-10-27 00:36:43] ...trap/engines/engine.py.224: train: epoch 38 | batch 1130/2304 [I 2022-10-27 00:36:43] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:56 [I 2022-10-27 00:36:43] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00241 [I 2022-10-27 00:36:43] ...trap/engines/engine.py.230: loss: 0.17814 [I 2022-10-27 00:36:43] ...optimizers/trijoint.py.151: total_norm: 26.014557 [I 2022-10-27 00:36:49] ...trap/engines/engine.py.224: train: epoch 38 | batch 1140/2304 [I 2022-10-27 00:36:49] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:49 [I 2022-10-27 00:36:49] ...trap/engines/engine.py.229: process: 0.56577 | load: 0.00246 [I 2022-10-27 00:36:49] ...trap/engines/engine.py.230: loss: 0.12803 [I 2022-10-27 00:36:49] ...optimizers/trijoint.py.151: total_norm: 13.328624 [I 2022-10-27 00:36:54] ...trap/engines/engine.py.224: train: epoch 38 | batch 1150/2304 [I 2022-10-27 00:36:54] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:39 [I 2022-10-27 00:36:54] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00242 [I 2022-10-27 00:36:54] ...trap/engines/engine.py.230: loss: 0.14132 [I 2022-10-27 00:36:54] ...optimizers/trijoint.py.151: total_norm: 32.513191 [I 2022-10-27 00:37:00] ...trap/engines/engine.py.224: train: epoch 38 | batch 1160/2304 [I 2022-10-27 00:37:00] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:41 [I 2022-10-27 00:37:00] ...trap/engines/engine.py.229: process: 0.57007 | load: 0.00245 [I 2022-10-27 00:37:00] ...trap/engines/engine.py.230: loss: 0.21444 [I 2022-10-27 00:37:00] ...optimizers/trijoint.py.151: total_norm: 23.607311 [I 2022-10-27 00:37:06] ...trap/engines/engine.py.224: train: epoch 38 | batch 1170/2304 [I 2022-10-27 00:37:06] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:39 [I 2022-10-27 00:37:06] ...trap/engines/engine.py.229: process: 0.54852 | load: 0.00246 [I 2022-10-27 00:37:06] ...trap/engines/engine.py.230: loss: 0.18296 [I 2022-10-27 00:37:06] ...optimizers/trijoint.py.151: total_norm: 18.924767 [I 2022-10-27 00:37:11] ...trap/engines/engine.py.224: train: epoch 38 | batch 1180/2304 [I 2022-10-27 00:37:11] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:25 [I 2022-10-27 00:37:11] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00245 [I 2022-10-27 00:37:11] ...trap/engines/engine.py.230: loss: 0.21259 [I 2022-10-27 00:37:11] ...optimizers/trijoint.py.151: total_norm: 12.087228 [I 2022-10-27 00:37:17] ...trap/engines/engine.py.224: train: epoch 38 | batch 1190/2304 [I 2022-10-27 00:37:17] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:23 [I 2022-10-27 00:37:17] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00243 [I 2022-10-27 00:37:17] ...trap/engines/engine.py.230: loss: 0.22867 [I 2022-10-27 00:37:17] ...optimizers/trijoint.py.151: total_norm: 26.737665 [I 2022-10-27 00:37:22] ...trap/engines/engine.py.224: train: epoch 38 | batch 1200/2304 [I 2022-10-27 00:37:22] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:17 [I 2022-10-27 00:37:22] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00246 [I 2022-10-27 00:37:22] ...trap/engines/engine.py.230: loss: 0.23830 [I 2022-10-27 00:37:22] ...optimizers/trijoint.py.151: total_norm: 20.457905 [I 2022-10-27 00:37:28] ...trap/engines/engine.py.224: train: epoch 38 | batch 1210/2304 [I 2022-10-27 00:37:28] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:09 [I 2022-10-27 00:37:28] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00244 [I 2022-10-27 00:37:28] ...trap/engines/engine.py.230: loss: 0.19590 [I 2022-10-27 00:37:28] ...optimizers/trijoint.py.151: total_norm: 19.194704 [I 2022-10-27 00:37:34] ...trap/engines/engine.py.224: train: epoch 38 | batch 1220/2304 [I 2022-10-27 00:37:34] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:09:56 [I 2022-10-27 00:37:34] ...trap/engines/engine.py.229: process: 0.54187 | load: 0.00250 [I 2022-10-27 00:37:34] ...trap/engines/engine.py.230: loss: 0.13861 [I 2022-10-27 00:37:34] ...optimizers/trijoint.py.151: total_norm: 7.698445 [I 2022-10-27 00:37:39] ...trap/engines/engine.py.224: train: epoch 38 | batch 1230/2304 [I 2022-10-27 00:37:39] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:59 [I 2022-10-27 00:37:39] ...trap/engines/engine.py.229: process: 0.57151 | load: 0.00369 [I 2022-10-27 00:37:39] ...trap/engines/engine.py.230: loss: 0.11836 [I 2022-10-27 00:37:39] ...optimizers/trijoint.py.151: total_norm: 13.277673 [I 2022-10-27 00:37:45] ...trap/engines/engine.py.224: train: epoch 38 | batch 1240/2304 [I 2022-10-27 00:37:45] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:53 [I 2022-10-27 00:37:45] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00251 [I 2022-10-27 00:37:45] ...trap/engines/engine.py.230: loss: 0.23849 [I 2022-10-27 00:37:45] ...optimizers/trijoint.py.151: total_norm: 28.418756 [I 2022-10-27 00:37:50] ...trap/engines/engine.py.224: train: epoch 38 | batch 1250/2304 [I 2022-10-27 00:37:50] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:46 [I 2022-10-27 00:37:50] ...trap/engines/engine.py.229: process: 0.56186 | load: 0.00241 [I 2022-10-27 00:37:50] ...trap/engines/engine.py.230: loss: 0.19877 [I 2022-10-27 00:37:50] ...optimizers/trijoint.py.151: total_norm: 17.401131 [I 2022-10-27 00:37:56] ...trap/engines/engine.py.224: train: epoch 38 | batch 1260/2304 [I 2022-10-27 00:37:56] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:40 [I 2022-10-27 00:37:56] ...trap/engines/engine.py.229: process: 0.55863 | load: 0.00244 [I 2022-10-27 00:37:56] ...trap/engines/engine.py.230: loss: 0.12254 [I 2022-10-27 00:37:56] ...optimizers/trijoint.py.151: total_norm: 17.555519 [I 2022-10-27 00:38:01] ...trap/engines/engine.py.224: train: epoch 38 | batch 1270/2304 [I 2022-10-27 00:38:01] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:33 [I 2022-10-27 00:38:01] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00242 [I 2022-10-27 00:38:01] ...trap/engines/engine.py.230: loss: 0.17101 [I 2022-10-27 00:38:01] ...optimizers/trijoint.py.151: total_norm: 14.985929 [I 2022-10-27 00:38:07] ...trap/engines/engine.py.224: train: epoch 38 | batch 1280/2304 [I 2022-10-27 00:38:07] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:32 [I 2022-10-27 00:38:07] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00241 [I 2022-10-27 00:38:07] ...trap/engines/engine.py.230: loss: 0.20481 [I 2022-10-27 00:38:07] ...optimizers/trijoint.py.151: total_norm: 26.289215 [I 2022-10-27 00:38:13] ...trap/engines/engine.py.224: train: epoch 38 | batch 1290/2304 [I 2022-10-27 00:38:13] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:27 [I 2022-10-27 00:38:13] ...trap/engines/engine.py.229: process: 0.56522 | load: 0.00245 [I 2022-10-27 00:38:13] ...trap/engines/engine.py.230: loss: 0.14777 [I 2022-10-27 00:38:13] ...optimizers/trijoint.py.151: total_norm: 13.959620 [I 2022-10-27 00:38:18] ...trap/engines/engine.py.224: train: epoch 38 | batch 1300/2304 [I 2022-10-27 00:38:18] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:21 [I 2022-10-27 00:38:18] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00240 [I 2022-10-27 00:38:18] ...trap/engines/engine.py.230: loss: 0.12854 [I 2022-10-27 00:38:18] ...optimizers/trijoint.py.151: total_norm: 13.880474 [I 2022-10-27 00:38:24] ...trap/engines/engine.py.224: train: epoch 38 | batch 1310/2304 [I 2022-10-27 00:38:24] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:12 [I 2022-10-27 00:38:24] ...trap/engines/engine.py.229: process: 0.54987 | load: 0.00245 [I 2022-10-27 00:38:24] ...trap/engines/engine.py.230: loss: 0.20365 [I 2022-10-27 00:38:24] ...optimizers/trijoint.py.151: total_norm: 14.355788 [I 2022-10-27 00:38:30] ...trap/engines/engine.py.224: train: epoch 38 | batch 1320/2304 [I 2022-10-27 00:38:30] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:09 [I 2022-10-27 00:38:30] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00263 [I 2022-10-27 00:38:30] ...trap/engines/engine.py.230: loss: 0.10342 [I 2022-10-27 00:38:30] ...optimizers/trijoint.py.151: total_norm: 16.093290 [I 2022-10-27 00:38:35] ...trap/engines/engine.py.224: train: epoch 38 | batch 1330/2304 [I 2022-10-27 00:38:35] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:00 [I 2022-10-27 00:38:35] ...trap/engines/engine.py.229: process: 0.55028 | load: 0.00251 [I 2022-10-27 00:38:35] ...trap/engines/engine.py.230: loss: 0.23048 [I 2022-10-27 00:38:35] ...optimizers/trijoint.py.151: total_norm: 16.973621 [I 2022-10-27 00:38:41] ...trap/engines/engine.py.224: train: epoch 38 | batch 1340/2304 [I 2022-10-27 00:38:41] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:56 [I 2022-10-27 00:38:41] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00284 [I 2022-10-27 00:38:41] ...trap/engines/engine.py.230: loss: 0.12130 [I 2022-10-27 00:38:41] ...optimizers/trijoint.py.151: total_norm: 13.106389 [I 2022-10-27 00:38:46] ...trap/engines/engine.py.224: train: epoch 38 | batch 1350/2304 [I 2022-10-27 00:38:46] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:54 [I 2022-10-27 00:38:46] ...trap/engines/engine.py.229: process: 0.56940 | load: 0.00243 [I 2022-10-27 00:38:46] ...trap/engines/engine.py.230: loss: 0.13811 [I 2022-10-27 00:38:46] ...optimizers/trijoint.py.151: total_norm: 16.545929 [I 2022-10-27 00:38:52] ...trap/engines/engine.py.224: train: epoch 38 | batch 1360/2304 [I 2022-10-27 00:38:52] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:43 [I 2022-10-27 00:38:52] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00245 [I 2022-10-27 00:38:52] ...trap/engines/engine.py.230: loss: 0.18559 [I 2022-10-27 00:38:52] ...optimizers/trijoint.py.151: total_norm: 19.849483 [I 2022-10-27 00:38:57] ...trap/engines/engine.py.224: train: epoch 38 | batch 1370/2304 [I 2022-10-27 00:38:57] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:38 [I 2022-10-27 00:38:57] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00245 [I 2022-10-27 00:38:57] ...trap/engines/engine.py.230: loss: 0.22137 [I 2022-10-27 00:38:57] ...optimizers/trijoint.py.151: total_norm: 23.313595 [I 2022-10-27 00:39:03] ...trap/engines/engine.py.224: train: epoch 38 | batch 1380/2304 [I 2022-10-27 00:39:03] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:35 [I 2022-10-27 00:39:03] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00244 [I 2022-10-27 00:39:03] ...trap/engines/engine.py.230: loss: 0.17188 [I 2022-10-27 00:39:03] ...optimizers/trijoint.py.151: total_norm: 18.766872 [I 2022-10-27 00:39:09] ...trap/engines/engine.py.224: train: epoch 38 | batch 1390/2304 [I 2022-10-27 00:39:09] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:33 [I 2022-10-27 00:39:09] ...trap/engines/engine.py.229: process: 0.54861 | load: 0.00242 [I 2022-10-27 00:39:09] ...trap/engines/engine.py.230: loss: 0.18669 [I 2022-10-27 00:39:09] ...optimizers/trijoint.py.151: total_norm: 16.912727 [I 2022-10-27 00:39:14] ...trap/engines/engine.py.224: train: epoch 38 | batch 1400/2304 [I 2022-10-27 00:39:14] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:26 [I 2022-10-27 00:39:14] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00246 [I 2022-10-27 00:39:14] ...trap/engines/engine.py.230: loss: 0.17992 [I 2022-10-27 00:39:14] ...optimizers/trijoint.py.151: total_norm: 30.591881 [I 2022-10-27 00:39:20] ...trap/engines/engine.py.224: train: epoch 38 | batch 1410/2304 [I 2022-10-27 00:39:20] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:17 [I 2022-10-27 00:39:20] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00250 [I 2022-10-27 00:39:20] ...trap/engines/engine.py.230: loss: 0.20004 [I 2022-10-27 00:39:20] ...optimizers/trijoint.py.151: total_norm: 30.913185 [I 2022-10-27 00:39:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 1420/2304 [I 2022-10-27 00:39:25] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:16 [I 2022-10-27 00:39:25] ...trap/engines/engine.py.229: process: 0.56178 | load: 0.00247 [I 2022-10-27 00:39:25] ...trap/engines/engine.py.230: loss: 0.17780 [I 2022-10-27 00:39:25] ...optimizers/trijoint.py.151: total_norm: 17.876829 [I 2022-10-27 00:39:31] ...trap/engines/engine.py.224: train: epoch 38 | batch 1430/2304 [I 2022-10-27 00:39:31] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:02 [I 2022-10-27 00:39:31] ...trap/engines/engine.py.229: process: 0.54126 | load: 0.00269 [I 2022-10-27 00:39:31] ...trap/engines/engine.py.230: loss: 0.22155 [I 2022-10-27 00:39:31] ...optimizers/trijoint.py.151: total_norm: 13.428474 [I 2022-10-27 00:39:37] ...trap/engines/engine.py.224: train: epoch 38 | batch 1440/2304 [I 2022-10-27 00:39:37] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:01 [I 2022-10-27 00:39:37] ...trap/engines/engine.py.229: process: 0.57053 | load: 0.00245 [I 2022-10-27 00:39:37] ...trap/engines/engine.py.230: loss: 0.19017 [I 2022-10-27 00:39:37] ...optimizers/trijoint.py.151: total_norm: 21.165321 [I 2022-10-27 00:39:42] ...trap/engines/engine.py.224: train: epoch 38 | batch 1450/2304 [I 2022-10-27 00:39:42] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:53 [I 2022-10-27 00:39:42] ...trap/engines/engine.py.229: process: 0.55348 | load: 0.00242 [I 2022-10-27 00:39:42] ...trap/engines/engine.py.230: loss: 0.26268 [I 2022-10-27 00:39:42] ...optimizers/trijoint.py.151: total_norm: 20.765917 [I 2022-10-27 00:39:48] ...trap/engines/engine.py.224: train: epoch 38 | batch 1460/2304 [I 2022-10-27 00:39:48] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:47 [I 2022-10-27 00:39:48] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00246 [I 2022-10-27 00:39:48] ...trap/engines/engine.py.230: loss: 0.26067 [I 2022-10-27 00:39:48] ...optimizers/trijoint.py.151: total_norm: 17.053307 [I 2022-10-27 00:39:53] ...trap/engines/engine.py.224: train: epoch 38 | batch 1470/2304 [I 2022-10-27 00:39:53] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:44 [I 2022-10-27 00:39:53] ...trap/engines/engine.py.229: process: 0.56578 | load: 0.00246 [I 2022-10-27 00:39:53] ...trap/engines/engine.py.230: loss: 0.16831 [I 2022-10-27 00:39:53] ...optimizers/trijoint.py.151: total_norm: 31.289965 [I 2022-10-27 00:39:59] ...trap/engines/engine.py.224: train: epoch 38 | batch 1480/2304 [I 2022-10-27 00:39:59] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:39 [I 2022-10-27 00:39:59] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00243 [I 2022-10-27 00:39:59] ...trap/engines/engine.py.230: loss: 0.20118 [I 2022-10-27 00:39:59] ...optimizers/trijoint.py.151: total_norm: 17.680330 [I 2022-10-27 00:40:04] ...trap/engines/engine.py.224: train: epoch 38 | batch 1490/2304 [I 2022-10-27 00:40:04] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:30 [I 2022-10-27 00:40:04] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00240 [I 2022-10-27 00:40:04] ...trap/engines/engine.py.230: loss: 0.14805 [I 2022-10-27 00:40:04] ...optimizers/trijoint.py.151: total_norm: 26.020784 [I 2022-10-27 00:40:10] ...trap/engines/engine.py.224: train: epoch 38 | batch 1500/2304 [I 2022-10-27 00:40:10] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:24 [I 2022-10-27 00:40:10] ...trap/engines/engine.py.229: process: 0.54993 | load: 0.00243 [I 2022-10-27 00:40:10] ...trap/engines/engine.py.230: loss: 0.14278 [I 2022-10-27 00:40:10] ...optimizers/trijoint.py.151: total_norm: 11.789375 [I 2022-10-27 00:40:16] ...trap/engines/engine.py.224: train: epoch 38 | batch 1510/2304 [I 2022-10-27 00:40:16] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:22 [I 2022-10-27 00:40:16] ...trap/engines/engine.py.229: process: 0.55175 | load: 0.00252 [I 2022-10-27 00:40:16] ...trap/engines/engine.py.230: loss: 0.10688 [I 2022-10-27 00:40:16] ...optimizers/trijoint.py.151: total_norm: 13.815675 [I 2022-10-27 00:40:21] ...trap/engines/engine.py.224: train: epoch 38 | batch 1520/2304 [I 2022-10-27 00:40:21] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:14 [I 2022-10-27 00:40:21] ...trap/engines/engine.py.229: process: 0.55605 | load: 0.00250 [I 2022-10-27 00:40:21] ...trap/engines/engine.py.230: loss: 0.17171 [I 2022-10-27 00:40:21] ...optimizers/trijoint.py.151: total_norm: 22.155102 [I 2022-10-27 00:40:27] ...trap/engines/engine.py.224: train: epoch 38 | batch 1530/2304 [I 2022-10-27 00:40:27] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:09 [I 2022-10-27 00:40:27] ...trap/engines/engine.py.229: process: 0.56750 | load: 0.00244 [I 2022-10-27 00:40:27] ...trap/engines/engine.py.230: loss: 0.19342 [I 2022-10-27 00:40:27] ...optimizers/trijoint.py.151: total_norm: 21.471355 [I 2022-10-27 00:40:32] ...trap/engines/engine.py.224: train: epoch 38 | batch 1540/2304 [I 2022-10-27 00:40:32] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:03 [I 2022-10-27 00:40:32] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00244 [I 2022-10-27 00:40:32] ...trap/engines/engine.py.230: loss: 0.20749 [I 2022-10-27 00:40:32] ...optimizers/trijoint.py.151: total_norm: 23.690531 [I 2022-10-27 00:40:38] ...trap/engines/engine.py.224: train: epoch 38 | batch 1550/2304 [I 2022-10-27 00:40:38] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:06:58 [I 2022-10-27 00:40:38] ...trap/engines/engine.py.229: process: 0.55023 | load: 0.00248 [I 2022-10-27 00:40:38] ...trap/engines/engine.py.230: loss: 0.11636 [I 2022-10-27 00:40:38] ...optimizers/trijoint.py.151: total_norm: 19.807999 [I 2022-10-27 00:40:43] ...trap/engines/engine.py.224: train: epoch 38 | batch 1560/2304 [I 2022-10-27 00:40:43] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:55 [I 2022-10-27 00:40:43] ...trap/engines/engine.py.229: process: 0.56425 | load: 0.00244 [I 2022-10-27 00:40:43] ...trap/engines/engine.py.230: loss: 0.15562 [I 2022-10-27 00:40:43] ...optimizers/trijoint.py.151: total_norm: 30.116966 [I 2022-10-27 00:40:49] ...trap/engines/engine.py.224: train: epoch 38 | batch 1570/2304 [I 2022-10-27 00:40:49] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:49 [I 2022-10-27 00:40:49] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00246 [I 2022-10-27 00:40:49] ...trap/engines/engine.py.230: loss: 0.19547 [I 2022-10-27 00:40:49] ...optimizers/trijoint.py.151: total_norm: 50.142166 [I 2022-10-27 00:40:55] ...trap/engines/engine.py.224: train: epoch 38 | batch 1580/2304 [I 2022-10-27 00:40:55] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:41 [I 2022-10-27 00:40:55] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00242 [I 2022-10-27 00:40:55] ...trap/engines/engine.py.230: loss: 0.21695 [I 2022-10-27 00:40:55] ...optimizers/trijoint.py.151: total_norm: 24.853477 [I 2022-10-27 00:41:00] ...trap/engines/engine.py.224: train: epoch 38 | batch 1590/2304 [I 2022-10-27 00:41:00] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:36 [I 2022-10-27 00:41:00] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00243 [I 2022-10-27 00:41:00] ...trap/engines/engine.py.230: loss: 0.20430 [I 2022-10-27 00:41:00] ...optimizers/trijoint.py.151: total_norm: 30.512863 [I 2022-10-27 00:41:06] ...trap/engines/engine.py.224: train: epoch 38 | batch 1600/2304 [I 2022-10-27 00:41:06] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:33 [I 2022-10-27 00:41:06] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00250 [I 2022-10-27 00:41:06] ...trap/engines/engine.py.230: loss: 0.16778 [I 2022-10-27 00:41:06] ...optimizers/trijoint.py.151: total_norm: 19.190109 [I 2022-10-27 00:41:11] ...trap/engines/engine.py.224: train: epoch 38 | batch 1610/2304 [I 2022-10-27 00:41:11] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:25 [I 2022-10-27 00:41:11] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00244 [I 2022-10-27 00:41:11] ...trap/engines/engine.py.230: loss: 0.18761 [I 2022-10-27 00:41:11] ...optimizers/trijoint.py.151: total_norm: 27.426807 [I 2022-10-27 00:41:17] ...trap/engines/engine.py.224: train: epoch 38 | batch 1620/2304 [I 2022-10-27 00:41:17] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:20 [I 2022-10-27 00:41:17] ...trap/engines/engine.py.229: process: 0.55624 | load: 0.00259 [I 2022-10-27 00:41:17] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-27 00:41:17] ...optimizers/trijoint.py.151: total_norm: 11.265005 [I 2022-10-27 00:41:22] ...trap/engines/engine.py.224: train: epoch 38 | batch 1630/2304 [I 2022-10-27 00:41:22] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:16 [I 2022-10-27 00:41:22] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00247 [I 2022-10-27 00:41:22] ...trap/engines/engine.py.230: loss: 0.13829 [I 2022-10-27 00:41:22] ...optimizers/trijoint.py.151: total_norm: 11.202774 [I 2022-10-27 00:41:28] ...trap/engines/engine.py.224: train: epoch 38 | batch 1640/2304 [I 2022-10-27 00:41:28] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:07 [I 2022-10-27 00:41:28] ...trap/engines/engine.py.229: process: 0.54517 | load: 0.00249 [I 2022-10-27 00:41:28] ...trap/engines/engine.py.230: loss: 0.18643 [I 2022-10-27 00:41:28] ...optimizers/trijoint.py.151: total_norm: 15.057564 [I 2022-10-27 00:41:34] ...trap/engines/engine.py.224: train: epoch 38 | batch 1650/2304 [I 2022-10-27 00:41:34] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:04 [I 2022-10-27 00:41:34] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00248 [I 2022-10-27 00:41:34] ...trap/engines/engine.py.230: loss: 0.12283 [I 2022-10-27 00:41:34] ...optimizers/trijoint.py.151: total_norm: 12.361231 [I 2022-10-27 00:41:39] ...trap/engines/engine.py.224: train: epoch 38 | batch 1660/2304 [I 2022-10-27 00:41:39] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:05:55 [I 2022-10-27 00:41:39] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00240 [I 2022-10-27 00:41:39] ...trap/engines/engine.py.230: loss: 0.16429 [I 2022-10-27 00:41:39] ...optimizers/trijoint.py.151: total_norm: 17.841606 [I 2022-10-27 00:41:45] ...trap/engines/engine.py.224: train: epoch 38 | batch 1670/2304 [I 2022-10-27 00:41:45] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:54 [I 2022-10-27 00:41:45] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00249 [I 2022-10-27 00:41:45] ...trap/engines/engine.py.230: loss: 0.17392 [I 2022-10-27 00:41:45] ...optimizers/trijoint.py.151: total_norm: 31.124460 [I 2022-10-27 00:41:50] ...trap/engines/engine.py.224: train: epoch 38 | batch 1680/2304 [I 2022-10-27 00:41:50] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:48 [I 2022-10-27 00:41:50] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00243 [I 2022-10-27 00:41:50] ...trap/engines/engine.py.230: loss: 0.19157 [I 2022-10-27 00:41:50] ...optimizers/trijoint.py.151: total_norm: 16.157862 [I 2022-10-27 00:41:56] ...trap/engines/engine.py.224: train: epoch 38 | batch 1690/2304 [I 2022-10-27 00:41:56] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:39 [I 2022-10-27 00:41:56] ...trap/engines/engine.py.229: process: 0.54545 | load: 0.00262 [I 2022-10-27 00:41:56] ...trap/engines/engine.py.230: loss: 0.12256 [I 2022-10-27 00:41:56] ...optimizers/trijoint.py.151: total_norm: 10.404716 [I 2022-10-27 00:42:01] ...trap/engines/engine.py.224: train: epoch 38 | batch 1700/2304 [I 2022-10-27 00:42:01] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:34 [I 2022-10-27 00:42:01] ...trap/engines/engine.py.229: process: 0.54818 | load: 0.00241 [I 2022-10-27 00:42:01] ...trap/engines/engine.py.230: loss: 0.20092 [I 2022-10-27 00:42:01] ...optimizers/trijoint.py.151: total_norm: 16.147890 [I 2022-10-27 00:42:07] ...trap/engines/engine.py.224: train: epoch 38 | batch 1710/2304 [I 2022-10-27 00:42:07] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:29 [I 2022-10-27 00:42:07] ...trap/engines/engine.py.229: process: 0.56096 | load: 0.00244 [I 2022-10-27 00:42:07] ...trap/engines/engine.py.230: loss: 0.20493 [I 2022-10-27 00:42:07] ...optimizers/trijoint.py.151: total_norm: 21.107328 [I 2022-10-27 00:42:13] ...trap/engines/engine.py.224: train: epoch 38 | batch 1720/2304 [I 2022-10-27 00:42:13] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:25 [I 2022-10-27 00:42:13] ...trap/engines/engine.py.229: process: 0.56389 | load: 0.00240 [I 2022-10-27 00:42:13] ...trap/engines/engine.py.230: loss: 0.17627 [I 2022-10-27 00:42:13] ...optimizers/trijoint.py.151: total_norm: 13.531997 [I 2022-10-27 00:42:18] ...trap/engines/engine.py.224: train: epoch 38 | batch 1730/2304 [I 2022-10-27 00:42:18] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:17 [I 2022-10-27 00:42:18] ...trap/engines/engine.py.229: process: 0.55198 | load: 0.00245 [I 2022-10-27 00:42:18] ...trap/engines/engine.py.230: loss: 0.20672 [I 2022-10-27 00:42:18] ...optimizers/trijoint.py.151: total_norm: 19.177315 [I 2022-10-27 00:42:24] ...trap/engines/engine.py.224: train: epoch 38 | batch 1740/2304 [I 2022-10-27 00:42:24] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:13 [I 2022-10-27 00:42:24] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00244 [I 2022-10-27 00:42:24] ...trap/engines/engine.py.230: loss: 0.14527 [I 2022-10-27 00:42:24] ...optimizers/trijoint.py.151: total_norm: 12.650792 [I 2022-10-27 00:42:29] ...trap/engines/engine.py.224: train: epoch 38 | batch 1750/2304 [I 2022-10-27 00:42:29] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:07 [I 2022-10-27 00:42:29] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00242 [I 2022-10-27 00:42:29] ...trap/engines/engine.py.230: loss: 0.18412 [I 2022-10-27 00:42:29] ...optimizers/trijoint.py.151: total_norm: 22.283848 [I 2022-10-27 00:42:35] ...trap/engines/engine.py.224: train: epoch 38 | batch 1760/2304 [I 2022-10-27 00:42:35] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:03 [I 2022-10-27 00:42:35] ...trap/engines/engine.py.229: process: 0.55481 | load: 0.00241 [I 2022-10-27 00:42:35] ...trap/engines/engine.py.230: loss: 0.12550 [I 2022-10-27 00:42:35] ...optimizers/trijoint.py.151: total_norm: 12.312166 [I 2022-10-27 00:42:40] ...trap/engines/engine.py.224: train: epoch 38 | batch 1770/2304 [I 2022-10-27 00:42:40] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:04:55 [I 2022-10-27 00:42:40] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00241 [I 2022-10-27 00:42:40] ...trap/engines/engine.py.230: loss: 0.12031 [I 2022-10-27 00:42:40] ...optimizers/trijoint.py.151: total_norm: 12.553378 [I 2022-10-27 00:42:46] ...trap/engines/engine.py.224: train: epoch 38 | batch 1780/2304 [I 2022-10-27 00:42:46] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:04:52 [I 2022-10-27 00:42:46] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00242 [I 2022-10-27 00:42:46] ...trap/engines/engine.py.230: loss: 0.14460 [I 2022-10-27 00:42:46] ...optimizers/trijoint.py.151: total_norm: 18.751839 [I 2022-10-27 00:42:51] ...trap/engines/engine.py.224: train: epoch 38 | batch 1790/2304 [I 2022-10-27 00:42:51] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:46 [I 2022-10-27 00:42:51] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00242 [I 2022-10-27 00:42:52] ...trap/engines/engine.py.230: loss: 0.13203 [I 2022-10-27 00:42:52] ...optimizers/trijoint.py.151: total_norm: 16.105930 [I 2022-10-27 00:42:57] ...trap/engines/engine.py.224: train: epoch 38 | batch 1800/2304 [I 2022-10-27 00:42:57] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:04:41 [I 2022-10-27 00:42:57] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00244 [I 2022-10-27 00:42:57] ...trap/engines/engine.py.230: loss: 0.16949 [I 2022-10-27 00:42:57] ...optimizers/trijoint.py.151: total_norm: 21.649164 [I 2022-10-27 00:43:03] ...trap/engines/engine.py.224: train: epoch 38 | batch 1810/2304 [I 2022-10-27 00:43:03] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:33 [I 2022-10-27 00:43:03] ...trap/engines/engine.py.229: process: 0.56156 | load: 0.00251 [I 2022-10-27 00:43:03] ...trap/engines/engine.py.230: loss: 0.12588 [I 2022-10-27 00:43:03] ...optimizers/trijoint.py.151: total_norm: 20.585440 [I 2022-10-27 00:43:08] ...trap/engines/engine.py.224: train: epoch 38 | batch 1820/2304 [I 2022-10-27 00:43:08] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:28 [I 2022-10-27 00:43:08] ...trap/engines/engine.py.229: process: 0.54993 | load: 0.00244 [I 2022-10-27 00:43:08] ...trap/engines/engine.py.230: loss: 0.20565 [I 2022-10-27 00:43:08] ...optimizers/trijoint.py.151: total_norm: 23.058823 [I 2022-10-27 00:43:14] ...trap/engines/engine.py.224: train: epoch 38 | batch 1830/2304 [I 2022-10-27 00:43:14] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:22 [I 2022-10-27 00:43:14] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00244 [I 2022-10-27 00:43:14] ...trap/engines/engine.py.230: loss: 0.19397 [I 2022-10-27 00:43:14] ...optimizers/trijoint.py.151: total_norm: 27.301594 [I 2022-10-27 00:43:19] ...trap/engines/engine.py.224: train: epoch 38 | batch 1840/2304 [I 2022-10-27 00:43:19] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:18 [I 2022-10-27 00:43:19] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00244 [I 2022-10-27 00:43:19] ...trap/engines/engine.py.230: loss: 0.21868 [I 2022-10-27 00:43:19] ...optimizers/trijoint.py.151: total_norm: 17.346712 [I 2022-10-27 00:43:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 1850/2304 [I 2022-10-27 00:43:25] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:12 [I 2022-10-27 00:43:25] ...trap/engines/engine.py.229: process: 0.56287 | load: 0.00244 [I 2022-10-27 00:43:25] ...trap/engines/engine.py.230: loss: 0.16877 [I 2022-10-27 00:43:25] ...optimizers/trijoint.py.151: total_norm: 17.890015 [I 2022-10-27 00:43:30] ...trap/engines/engine.py.224: train: epoch 38 | batch 1860/2304 [I 2022-10-27 00:43:30] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:07 [I 2022-10-27 00:43:30] ...trap/engines/engine.py.229: process: 0.56154 | load: 0.00244 [I 2022-10-27 00:43:30] ...trap/engines/engine.py.230: loss: 0.15726 [I 2022-10-27 00:43:30] ...optimizers/trijoint.py.151: total_norm: 23.507244 [I 2022-10-27 00:43:55] ...trap/engines/engine.py.224: train: epoch 38 | batch 1870/2304 [I 2022-10-27 00:43:55] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:26:02 [I 2022-10-27 00:43:55] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00241 [I 2022-10-27 00:43:55] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-27 00:43:55] ...optimizers/trijoint.py.151: total_norm: 28.844055 [I 2022-10-27 00:44:01] ...trap/engines/engine.py.224: train: epoch 38 | batch 1880/2304 [I 2022-10-27 00:44:01] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:06:15 [I 2022-10-27 00:44:01] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00238 [I 2022-10-27 00:44:01] ...trap/engines/engine.py.230: loss: 0.19634 [I 2022-10-27 00:44:01] ...optimizers/trijoint.py.151: total_norm: 18.474371 [I 2022-10-27 00:44:06] ...trap/engines/engine.py.224: train: epoch 38 | batch 1890/2304 [I 2022-10-27 00:44:06] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:04:03 [I 2022-10-27 00:44:06] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00241 [I 2022-10-27 00:44:06] ...trap/engines/engine.py.230: loss: 0.15993 [I 2022-10-27 00:44:06] ...optimizers/trijoint.py.151: total_norm: 25.534527 [I 2022-10-27 00:44:12] ...trap/engines/engine.py.224: train: epoch 38 | batch 1900/2304 [I 2022-10-27 00:44:12] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:46 [I 2022-10-27 00:44:12] ...trap/engines/engine.py.229: process: 0.55513 | load: 0.00238 [I 2022-10-27 00:44:12] ...trap/engines/engine.py.230: loss: 0.19312 [I 2022-10-27 00:44:12] ...optimizers/trijoint.py.151: total_norm: 25.023985 [I 2022-10-27 00:44:17] ...trap/engines/engine.py.224: train: epoch 38 | batch 1910/2304 [I 2022-10-27 00:44:17] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:39 [I 2022-10-27 00:44:17] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00240 [I 2022-10-27 00:44:17] ...trap/engines/engine.py.230: loss: 0.14915 [I 2022-10-27 00:44:17] ...optimizers/trijoint.py.151: total_norm: 19.252371 [I 2022-10-27 00:44:23] ...trap/engines/engine.py.224: train: epoch 38 | batch 1920/2304 [I 2022-10-27 00:44:23] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:34 [I 2022-10-27 00:44:23] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00243 [I 2022-10-27 00:44:23] ...trap/engines/engine.py.230: loss: 0.15487 [I 2022-10-27 00:44:23] ...optimizers/trijoint.py.151: total_norm: 19.006659 [I 2022-10-27 00:44:29] ...trap/engines/engine.py.224: train: epoch 38 | batch 1930/2304 [I 2022-10-27 00:44:29] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:29 [I 2022-10-27 00:44:29] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00241 [I 2022-10-27 00:44:29] ...trap/engines/engine.py.230: loss: 0.15564 [I 2022-10-27 00:44:29] ...optimizers/trijoint.py.151: total_norm: 12.615180 [I 2022-10-27 00:44:34] ...trap/engines/engine.py.224: train: epoch 38 | batch 1940/2304 [I 2022-10-27 00:44:34] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:22 [I 2022-10-27 00:44:34] ...trap/engines/engine.py.229: process: 0.54515 | load: 0.00243 [I 2022-10-27 00:44:34] ...trap/engines/engine.py.230: loss: 0.23862 [I 2022-10-27 00:44:34] ...optimizers/trijoint.py.151: total_norm: 18.449707 [I 2022-10-27 00:44:40] ...trap/engines/engine.py.224: train: epoch 38 | batch 1950/2304 [I 2022-10-27 00:44:40] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:19 [I 2022-10-27 00:44:40] ...trap/engines/engine.py.229: process: 0.57075 | load: 0.00240 [I 2022-10-27 00:44:40] ...trap/engines/engine.py.230: loss: 0.18369 [I 2022-10-27 00:44:40] ...optimizers/trijoint.py.151: total_norm: 18.475294 [I 2022-10-27 00:44:45] ...trap/engines/engine.py.224: train: epoch 38 | batch 1960/2304 [I 2022-10-27 00:44:45] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:03:12 [I 2022-10-27 00:44:45] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00246 [I 2022-10-27 00:44:45] ...trap/engines/engine.py.230: loss: 0.19288 [I 2022-10-27 00:44:45] ...optimizers/trijoint.py.151: total_norm: 26.765228 [I 2022-10-27 00:44:51] ...trap/engines/engine.py.224: train: epoch 38 | batch 1970/2304 [I 2022-10-27 00:44:51] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:03:06 [I 2022-10-27 00:44:51] ...trap/engines/engine.py.229: process: 0.57010 | load: 0.00245 [I 2022-10-27 00:44:51] ...trap/engines/engine.py.230: loss: 0.13577 [I 2022-10-27 00:44:51] ...optimizers/trijoint.py.151: total_norm: 18.026905 [I 2022-10-27 00:44:57] ...trap/engines/engine.py.224: train: epoch 38 | batch 1980/2304 [I 2022-10-27 00:44:57] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:59 [I 2022-10-27 00:44:57] ...trap/engines/engine.py.229: process: 0.54766 | load: 0.00242 [I 2022-10-27 00:44:57] ...trap/engines/engine.py.230: loss: 0.17382 [I 2022-10-27 00:44:57] ...optimizers/trijoint.py.151: total_norm: 19.244581 [I 2022-10-27 00:45:02] ...trap/engines/engine.py.224: train: epoch 38 | batch 1990/2304 [I 2022-10-27 00:45:02] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:56 [I 2022-10-27 00:45:02] ...trap/engines/engine.py.229: process: 0.56467 | load: 0.00244 [I 2022-10-27 00:45:02] ...trap/engines/engine.py.230: loss: 0.28972 [I 2022-10-27 00:45:02] ...optimizers/trijoint.py.151: total_norm: 22.357420 [I 2022-10-27 00:45:08] ...trap/engines/engine.py.224: train: epoch 38 | batch 2000/2304 [I 2022-10-27 00:45:08] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:51 [I 2022-10-27 00:45:08] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00243 [I 2022-10-27 00:45:08] ...trap/engines/engine.py.230: loss: 0.18086 [I 2022-10-27 00:45:08] ...optimizers/trijoint.py.151: total_norm: 21.638821 [I 2022-10-27 00:45:13] ...trap/engines/engine.py.224: train: epoch 38 | batch 2010/2304 [I 2022-10-27 00:45:13] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:45 [I 2022-10-27 00:45:13] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00238 [I 2022-10-27 00:45:13] ...trap/engines/engine.py.230: loss: 0.14119 [I 2022-10-27 00:45:13] ...optimizers/trijoint.py.151: total_norm: 16.441967 [I 2022-10-27 00:45:19] ...trap/engines/engine.py.224: train: epoch 38 | batch 2020/2304 [I 2022-10-27 00:45:19] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:38 [I 2022-10-27 00:45:19] ...trap/engines/engine.py.229: process: 0.55561 | load: 0.00244 [I 2022-10-27 00:45:19] ...trap/engines/engine.py.230: loss: 0.19765 [I 2022-10-27 00:45:19] ...optimizers/trijoint.py.151: total_norm: 14.366727 [I 2022-10-27 00:45:25] ...trap/engines/engine.py.224: train: epoch 38 | batch 2030/2304 [I 2022-10-27 00:45:25] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:32 [I 2022-10-27 00:45:25] ...trap/engines/engine.py.229: process: 0.55925 | load: 0.00240 [I 2022-10-27 00:45:25] ...trap/engines/engine.py.230: loss: 0.18076 [I 2022-10-27 00:45:25] ...optimizers/trijoint.py.151: total_norm: 26.588831 [I 2022-10-27 00:45:30] ...trap/engines/engine.py.224: train: epoch 38 | batch 2040/2304 [I 2022-10-27 00:45:30] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:27 [I 2022-10-27 00:45:30] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00242 [I 2022-10-27 00:45:30] ...trap/engines/engine.py.230: loss: 0.12898 [I 2022-10-27 00:45:30] ...optimizers/trijoint.py.151: total_norm: 11.485668 [I 2022-10-27 00:45:36] ...trap/engines/engine.py.224: train: epoch 38 | batch 2050/2304 [I 2022-10-27 00:45:36] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:21 [I 2022-10-27 00:45:36] ...trap/engines/engine.py.229: process: 0.54840 | load: 0.00256 [I 2022-10-27 00:45:36] ...trap/engines/engine.py.230: loss: 0.27331 [I 2022-10-27 00:45:36] ...optimizers/trijoint.py.151: total_norm: 21.272484 [I 2022-10-27 00:45:42] ...trap/engines/engine.py.224: train: epoch 38 | batch 2060/2304 [I 2022-10-27 00:45:42] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:18 [I 2022-10-27 00:45:42] ...trap/engines/engine.py.229: process: 0.54651 | load: 0.00249 [I 2022-10-27 00:45:42] ...trap/engines/engine.py.230: loss: 0.20234 [I 2022-10-27 00:45:42] ...optimizers/trijoint.py.151: total_norm: 24.648373 [I 2022-10-27 00:45:47] ...trap/engines/engine.py.224: train: epoch 38 | batch 2070/2304 [I 2022-10-27 00:45:47] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:11 [I 2022-10-27 00:45:47] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00251 [I 2022-10-27 00:45:47] ...trap/engines/engine.py.230: loss: 0.21352 [I 2022-10-27 00:45:47] ...optimizers/trijoint.py.151: total_norm: 22.698612 [I 2022-10-27 00:45:53] ...trap/engines/engine.py.224: train: epoch 38 | batch 2080/2304 [I 2022-10-27 00:45:53] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:02:04 [I 2022-10-27 00:45:53] ...trap/engines/engine.py.229: process: 0.55157 | load: 0.00244 [I 2022-10-27 00:45:53] ...trap/engines/engine.py.230: loss: 0.15412 [I 2022-10-27 00:45:53] ...optimizers/trijoint.py.151: total_norm: 14.161116 [I 2022-10-27 00:45:58] ...trap/engines/engine.py.224: train: epoch 38 | batch 2090/2304 [I 2022-10-27 00:45:58] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:59 [I 2022-10-27 00:45:58] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00243 [I 2022-10-27 00:45:58] ...trap/engines/engine.py.230: loss: 0.21741 [I 2022-10-27 00:45:58] ...optimizers/trijoint.py.151: total_norm: 21.436836 [I 2022-10-27 00:46:04] ...trap/engines/engine.py.224: train: epoch 38 | batch 2100/2304 [I 2022-10-27 00:46:04] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:53 [I 2022-10-27 00:46:04] ...trap/engines/engine.py.229: process: 0.56316 | load: 0.00244 [I 2022-10-27 00:46:04] ...trap/engines/engine.py.230: loss: 0.21098 [I 2022-10-27 00:46:04] ...optimizers/trijoint.py.151: total_norm: 18.504061 [I 2022-10-27 00:46:10] ...trap/engines/engine.py.224: train: epoch 38 | batch 2110/2304 [I 2022-10-27 00:46:10] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:48 [I 2022-10-27 00:46:10] ...trap/engines/engine.py.229: process: 0.56699 | load: 0.00248 [I 2022-10-27 00:46:10] ...trap/engines/engine.py.230: loss: 0.11967 [I 2022-10-27 00:46:10] ...optimizers/trijoint.py.151: total_norm: 14.226178 [I 2022-10-27 00:46:15] ...trap/engines/engine.py.224: train: epoch 38 | batch 2120/2304 [I 2022-10-27 00:46:15] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:41 [I 2022-10-27 00:46:15] ...trap/engines/engine.py.229: process: 0.54434 | load: 0.00272 [I 2022-10-27 00:46:15] ...trap/engines/engine.py.230: loss: 0.15544 [I 2022-10-27 00:46:15] ...optimizers/trijoint.py.151: total_norm: 18.162186 [I 2022-10-27 00:46:21] ...trap/engines/engine.py.224: train: epoch 38 | batch 2130/2304 [I 2022-10-27 00:46:21] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:37 [I 2022-10-27 00:46:21] ...trap/engines/engine.py.229: process: 0.57543 | load: 0.00258 [I 2022-10-27 00:46:21] ...trap/engines/engine.py.230: loss: 0.15389 [I 2022-10-27 00:46:21] ...optimizers/trijoint.py.151: total_norm: 11.157032 [I 2022-10-27 00:46:26] ...trap/engines/engine.py.224: train: epoch 38 | batch 2140/2304 [I 2022-10-27 00:46:26] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:31 [I 2022-10-27 00:46:26] ...trap/engines/engine.py.229: process: 0.54270 | load: 0.00253 [I 2022-10-27 00:46:26] ...trap/engines/engine.py.230: loss: 0.11152 [I 2022-10-27 00:46:26] ...optimizers/trijoint.py.151: total_norm: 9.578971 [I 2022-10-27 00:46:32] ...trap/engines/engine.py.224: train: epoch 38 | batch 2150/2304 [I 2022-10-27 00:46:32] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:24 [I 2022-10-27 00:46:32] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00251 [I 2022-10-27 00:46:32] ...trap/engines/engine.py.230: loss: 0.20447 [I 2022-10-27 00:46:32] ...optimizers/trijoint.py.151: total_norm: 15.588998 [I 2022-10-27 00:46:37] ...trap/engines/engine.py.224: train: epoch 38 | batch 2160/2304 [I 2022-10-27 00:46:37] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:20 [I 2022-10-27 00:46:37] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00252 [I 2022-10-27 00:46:37] ...trap/engines/engine.py.230: loss: 0.25340 [I 2022-10-27 00:46:37] ...optimizers/trijoint.py.151: total_norm: 22.200933 [I 2022-10-27 00:46:43] ...trap/engines/engine.py.224: train: epoch 38 | batch 2170/2304 [I 2022-10-27 00:46:43] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:01:14 [I 2022-10-27 00:46:43] ...trap/engines/engine.py.229: process: 0.55925 | load: 0.00251 [I 2022-10-27 00:46:43] ...trap/engines/engine.py.230: loss: 0.22921 [I 2022-10-27 00:46:43] ...optimizers/trijoint.py.151: total_norm: 24.157852 [I 2022-10-27 00:46:49] ...trap/engines/engine.py.224: train: epoch 38 | batch 2180/2304 [I 2022-10-27 00:46:49] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:08 [I 2022-10-27 00:46:49] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00244 [I 2022-10-27 00:46:49] ...trap/engines/engine.py.230: loss: 0.12459 [I 2022-10-27 00:46:49] ...optimizers/trijoint.py.151: total_norm: 9.794606 [I 2022-10-27 00:46:54] ...trap/engines/engine.py.224: train: epoch 38 | batch 2190/2304 [I 2022-10-27 00:46:54] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:01:03 [I 2022-10-27 00:46:54] ...trap/engines/engine.py.229: process: 0.55411 | load: 0.00241 [I 2022-10-27 00:46:54] ...trap/engines/engine.py.230: loss: 0.19875 [I 2022-10-27 00:46:54] ...optimizers/trijoint.py.151: total_norm: 21.852974 [I 2022-10-27 00:47:00] ...trap/engines/engine.py.224: train: epoch 38 | batch 2200/2304 [I 2022-10-27 00:47:00] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:57 [I 2022-10-27 00:47:00] ...trap/engines/engine.py.229: process: 0.54764 | load: 0.00244 [I 2022-10-27 00:47:00] ...trap/engines/engine.py.230: loss: 0.16144 [I 2022-10-27 00:47:00] ...optimizers/trijoint.py.151: total_norm: 20.456076 [I 2022-10-27 00:47:05] ...trap/engines/engine.py.224: train: epoch 38 | batch 2210/2304 [I 2022-10-27 00:47:05] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:52 [I 2022-10-27 00:47:05] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00246 [I 2022-10-27 00:47:05] ...trap/engines/engine.py.230: loss: 0.13713 [I 2022-10-27 00:47:05] ...optimizers/trijoint.py.151: total_norm: 33.139942 [I 2022-10-27 00:47:11] ...trap/engines/engine.py.224: train: epoch 38 | batch 2220/2304 [I 2022-10-27 00:47:11] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:46 [I 2022-10-27 00:47:11] ...trap/engines/engine.py.229: process: 0.56044 | load: 0.00244 [I 2022-10-27 00:47:11] ...trap/engines/engine.py.230: loss: 0.14669 [I 2022-10-27 00:47:11] ...optimizers/trijoint.py.151: total_norm: 22.508068 [I 2022-10-27 00:47:17] ...trap/engines/engine.py.224: train: epoch 38 | batch 2230/2304 [I 2022-10-27 00:47:17] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:41 [I 2022-10-27 00:47:17] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00264 [I 2022-10-27 00:47:17] ...trap/engines/engine.py.230: loss: 0.17480 [I 2022-10-27 00:47:17] ...optimizers/trijoint.py.151: total_norm: 24.413746 [I 2022-10-27 00:47:22] ...trap/engines/engine.py.224: train: epoch 38 | batch 2240/2304 [I 2022-10-27 00:47:22] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:35 [I 2022-10-27 00:47:22] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00257 [I 2022-10-27 00:47:22] ...trap/engines/engine.py.230: loss: 0.19542 [I 2022-10-27 00:47:22] ...optimizers/trijoint.py.151: total_norm: 15.533811 [I 2022-10-27 00:47:28] ...trap/engines/engine.py.224: train: epoch 38 | batch 2250/2304 [I 2022-10-27 00:47:28] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:30 [I 2022-10-27 00:47:28] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00251 [I 2022-10-27 00:47:28] ...trap/engines/engine.py.230: loss: 0.12232 [I 2022-10-27 00:47:28] ...optimizers/trijoint.py.151: total_norm: 8.913651 [I 2022-10-27 00:47:33] ...trap/engines/engine.py.224: train: epoch 38 | batch 2260/2304 [I 2022-10-27 00:47:33] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:24 [I 2022-10-27 00:47:33] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00274 [I 2022-10-27 00:47:33] ...trap/engines/engine.py.230: loss: 0.13487 [I 2022-10-27 00:47:33] ...optimizers/trijoint.py.151: total_norm: 11.170938 [I 2022-10-27 00:47:39] ...trap/engines/engine.py.224: train: epoch 38 | batch 2270/2304 [I 2022-10-27 00:47:39] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:18 [I 2022-10-27 00:47:39] ...trap/engines/engine.py.229: process: 0.54608 | load: 0.00256 [I 2022-10-27 00:47:39] ...trap/engines/engine.py.230: loss: 0.17379 [I 2022-10-27 00:47:39] ...optimizers/trijoint.py.151: total_norm: 27.305220 [I 2022-10-27 00:47:44] ...trap/engines/engine.py.224: train: epoch 38 | batch 2280/2304 [I 2022-10-27 00:47:44] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:13 [I 2022-10-27 00:47:44] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00259 [I 2022-10-27 00:47:44] ...trap/engines/engine.py.230: loss: 0.13802 [I 2022-10-27 00:47:44] ...optimizers/trijoint.py.151: total_norm: 13.685729 [I 2022-10-27 00:47:50] ...trap/engines/engine.py.224: train: epoch 38 | batch 2290/2304 [I 2022-10-27 00:47:50] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:00:07 [I 2022-10-27 00:47:50] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00255 [I 2022-10-27 00:47:50] ...trap/engines/engine.py.230: loss: 0.17611 [I 2022-10-27 00:47:50] ...optimizers/trijoint.py.151: total_norm: 14.635297 [I 2022-10-27 00:47:56] ...trap/engines/engine.py.224: train: epoch 38 | batch 2300/2304 [I 2022-10-27 00:47:56] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:02 [I 2022-10-27 00:47:56] ...trap/engines/engine.py.229: process: 0.54719 | load: 0.00036 [I 2022-10-27 00:47:56] ...trap/engines/engine.py.230: loss: 0.14846 [I 2022-10-27 00:47:56] ...optimizers/trijoint.py.151: total_norm: 17.968250 [I 2022-10-27 00:47:58] ...trap/engines/engine.py.224: train: epoch 38 | batch 2304/2304 [I 2022-10-27 00:47:58] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:00 [I 2022-10-27 00:47:58] ...trap/engines/engine.py.229: process: 0.55097 | load: 0.00019 [I 2022-10-27 00:47:58] ...trap/engines/engine.py.230: loss: 0.16726 [I 2022-10-27 00:47:58] ...optimizers/trijoint.py.151: total_norm: 14.068600 [S 2022-10-27 00:47:59] ...trap/engines/engine.py.129: train_epoch.epoch: 38 [S 2022-10-27 00:47:59] ...trap/engines/engine.py.129: train_epoch.loss: 0.1713035567463608 [I 2022-10-27 00:47:59] ...s/criterions/triplet.py.84: margin = 0.24500000000000016 [I 2022-10-27 00:48:00] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 38 [I 2022-10-27 00:48:00] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 00:48:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 00:48:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 00:48:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 00:48:03] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 00:48:03] ...trap/engines/engine.py.321: eval: epoch 38 | batch 0/510 [I 2022-10-27 00:48:03] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:29:46 [I 2022-10-27 00:48:03] ...trap/engines/engine.py.326: process: 3.50373 | load: 3.13594 [I 2022-10-27 00:48:07] ...trap/engines/engine.py.321: eval: epoch 38 | batch 10/510 [I 2022-10-27 00:48:07] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:38 [I 2022-10-27 00:48:07] ...trap/engines/engine.py.326: process: 0.33302 | load: 0.00018 [I 2022-10-27 00:48:10] ...trap/engines/engine.py.321: eval: epoch 38 | batch 20/510 [I 2022-10-27 00:48:10] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:03 [I 2022-10-27 00:48:10] ...trap/engines/engine.py.326: process: 0.32919 | load: 0.00018 [I 2022-10-27 00:48:13] ...trap/engines/engine.py.321: eval: epoch 38 | batch 30/510 [I 2022-10-27 00:48:13] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:42 [I 2022-10-27 00:48:13] ...trap/engines/engine.py.326: process: 0.33747 | load: 0.00017 [I 2022-10-27 00:48:17] ...trap/engines/engine.py.321: eval: epoch 38 | batch 40/510 [I 2022-10-27 00:48:17] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:36 [I 2022-10-27 00:48:17] ...trap/engines/engine.py.326: process: 0.33197 | load: 0.00040 [I 2022-10-27 00:48:20] ...trap/engines/engine.py.321: eval: epoch 38 | batch 50/510 [I 2022-10-27 00:48:20] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:35 [I 2022-10-27 00:48:20] ...trap/engines/engine.py.326: process: 0.33657 | load: 0.00018 [I 2022-10-27 00:48:23] ...trap/engines/engine.py.321: eval: epoch 38 | batch 60/510 [I 2022-10-27 00:48:23] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:29 [I 2022-10-27 00:48:23] ...trap/engines/engine.py.326: process: 0.32635 | load: 0.00020 [I 2022-10-27 00:48:27] ...trap/engines/engine.py.321: eval: epoch 38 | batch 70/510 [I 2022-10-27 00:48:27] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:27 [I 2022-10-27 00:48:27] ...trap/engines/engine.py.326: process: 0.33264 | load: 0.00027 [I 2022-10-27 00:48:30] ...trap/engines/engine.py.321: eval: epoch 38 | batch 80/510 [I 2022-10-27 00:48:30] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:25 [I 2022-10-27 00:48:30] ...trap/engines/engine.py.326: process: 0.33909 | load: 0.00018 [I 2022-10-27 00:48:33] ...trap/engines/engine.py.321: eval: epoch 38 | batch 90/510 [I 2022-10-27 00:48:33] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:23 [I 2022-10-27 00:48:33] ...trap/engines/engine.py.326: process: 0.36161 | load: 0.00018 [I 2022-10-27 00:48:37] ...trap/engines/engine.py.321: eval: epoch 38 | batch 100/510 [I 2022-10-27 00:48:37] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:12 [I 2022-10-27 00:48:37] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00017 [I 2022-10-27 00:48:40] ...trap/engines/engine.py.321: eval: epoch 38 | batch 110/510 [I 2022-10-27 00:48:40] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:58 [I 2022-10-27 00:48:40] ...trap/engines/engine.py.326: process: 0.29031 | load: 0.00025 [I 2022-10-27 00:48:43] ...trap/engines/engine.py.321: eval: epoch 38 | batch 120/510 [I 2022-10-27 00:48:43] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-27 00:48:43] ...trap/engines/engine.py.326: process: 0.28975 | load: 0.00023 [I 2022-10-27 00:48:45] ...trap/engines/engine.py.321: eval: epoch 38 | batch 130/510 [I 2022-10-27 00:48:45] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-27 00:48:45] ...trap/engines/engine.py.326: process: 0.29624 | load: 0.00016 [I 2022-10-27 00:48:48] ...trap/engines/engine.py.321: eval: epoch 38 | batch 140/510 [I 2022-10-27 00:48:48] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-27 00:48:48] ...trap/engines/engine.py.326: process: 0.28739 | load: 0.00018 [I 2022-10-27 00:48:51] ...trap/engines/engine.py.321: eval: epoch 38 | batch 150/510 [I 2022-10-27 00:48:51] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-27 00:48:51] ...trap/engines/engine.py.326: process: 0.29160 | load: 0.00027 [I 2022-10-27 00:48:54] ...trap/engines/engine.py.321: eval: epoch 38 | batch 160/510 [I 2022-10-27 00:48:54] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-27 00:48:54] ...trap/engines/engine.py.326: process: 0.27942 | load: 0.00021 [I 2022-10-27 00:48:57] ...trap/engines/engine.py.321: eval: epoch 38 | batch 170/510 [I 2022-10-27 00:48:57] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-27 00:48:57] ...trap/engines/engine.py.326: process: 0.29129 | load: 0.00016 [I 2022-10-27 00:49:00] ...trap/engines/engine.py.321: eval: epoch 38 | batch 180/510 [I 2022-10-27 00:49:00] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 00:49:00] ...trap/engines/engine.py.326: process: 0.29406 | load: 0.00018 [I 2022-10-27 00:49:03] ...trap/engines/engine.py.321: eval: epoch 38 | batch 190/510 [I 2022-10-27 00:49:03] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 00:49:03] ...trap/engines/engine.py.326: process: 0.28726 | load: 0.00020 [I 2022-10-27 00:49:06] ...trap/engines/engine.py.321: eval: epoch 38 | batch 200/510 [I 2022-10-27 00:49:06] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:34 [I 2022-10-27 00:49:06] ...trap/engines/engine.py.326: process: 0.28938 | load: 0.00020 [I 2022-10-27 00:49:09] ...trap/engines/engine.py.321: eval: epoch 38 | batch 210/510 [I 2022-10-27 00:49:09] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:27 [I 2022-10-27 00:49:09] ...trap/engines/engine.py.326: process: 0.28338 | load: 0.00025 [I 2022-10-27 00:49:12] ...trap/engines/engine.py.321: eval: epoch 38 | batch 220/510 [I 2022-10-27 00:49:12] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:24 [I 2022-10-27 00:49:12] ...trap/engines/engine.py.326: process: 0.28431 | load: 0.00025 [I 2022-10-27 00:49:15] ...trap/engines/engine.py.321: eval: epoch 38 | batch 230/510 [I 2022-10-27 00:49:15] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:21 [I 2022-10-27 00:49:15] ...trap/engines/engine.py.326: process: 0.29281 | load: 0.00019 [I 2022-10-27 00:49:18] ...trap/engines/engine.py.321: eval: epoch 38 | batch 240/510 [I 2022-10-27 00:49:18] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-27 00:49:18] ...trap/engines/engine.py.326: process: 0.29014 | load: 0.00015 [I 2022-10-27 00:49:20] ...trap/engines/engine.py.321: eval: epoch 38 | batch 250/510 [I 2022-10-27 00:49:20] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-27 00:49:20] ...trap/engines/engine.py.326: process: 0.28692 | load: 0.00019 [I 2022-10-27 00:49:23] ...trap/engines/engine.py.321: eval: epoch 38 | batch 260/510 [I 2022-10-27 00:49:23] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-27 00:49:23] ...trap/engines/engine.py.326: process: 0.29287 | load: 0.00018 [I 2022-10-27 00:49:26] ...trap/engines/engine.py.321: eval: epoch 38 | batch 270/510 [I 2022-10-27 00:49:26] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 00:49:26] ...trap/engines/engine.py.326: process: 0.29472 | load: 0.00023 [I 2022-10-27 00:49:29] ...trap/engines/engine.py.321: eval: epoch 38 | batch 280/510 [I 2022-10-27 00:49:29] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 00:49:29] ...trap/engines/engine.py.326: process: 0.29143 | load: 0.00017 [I 2022-10-27 00:49:32] ...trap/engines/engine.py.321: eval: epoch 38 | batch 290/510 [I 2022-10-27 00:49:32] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 00:49:32] ...trap/engines/engine.py.326: process: 0.29562 | load: 0.00015 [I 2022-10-27 00:49:35] ...trap/engines/engine.py.321: eval: epoch 38 | batch 300/510 [I 2022-10-27 00:49:35] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 00:49:35] ...trap/engines/engine.py.326: process: 0.29430 | load: 0.00015 [I 2022-10-27 00:49:38] ...trap/engines/engine.py.321: eval: epoch 38 | batch 310/510 [I 2022-10-27 00:49:38] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 00:49:38] ...trap/engines/engine.py.326: process: 0.28973 | load: 0.00022 [I 2022-10-27 00:49:41] ...trap/engines/engine.py.321: eval: epoch 38 | batch 320/510 [I 2022-10-27 00:49:41] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:55 [I 2022-10-27 00:49:41] ...trap/engines/engine.py.326: process: 0.28991 | load: 0.00023 [I 2022-10-27 00:49:44] ...trap/engines/engine.py.321: eval: epoch 38 | batch 330/510 [I 2022-10-27 00:49:44] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:52 [I 2022-10-27 00:49:44] ...trap/engines/engine.py.326: process: 0.29374 | load: 0.00023 [I 2022-10-27 00:49:47] ...trap/engines/engine.py.321: eval: epoch 38 | batch 340/510 [I 2022-10-27 00:49:47] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:49 [I 2022-10-27 00:49:47] ...trap/engines/engine.py.326: process: 0.28778 | load: 0.00021 [I 2022-10-27 00:49:49] ...trap/engines/engine.py.321: eval: epoch 38 | batch 350/510 [I 2022-10-27 00:49:49] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-27 00:49:49] ...trap/engines/engine.py.326: process: 0.29060 | load: 0.00021 [I 2022-10-27 00:49:52] ...trap/engines/engine.py.321: eval: epoch 38 | batch 360/510 [I 2022-10-27 00:49:52] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 00:49:52] ...trap/engines/engine.py.326: process: 0.28723 | load: 0.00014 [I 2022-10-27 00:49:55] ...trap/engines/engine.py.321: eval: epoch 38 | batch 370/510 [I 2022-10-27 00:49:55] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 00:49:55] ...trap/engines/engine.py.326: process: 0.28590 | load: 0.00018 [I 2022-10-27 00:49:58] ...trap/engines/engine.py.321: eval: epoch 38 | batch 380/510 [I 2022-10-27 00:49:58] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 00:49:58] ...trap/engines/engine.py.326: process: 0.28555 | load: 0.00016 [I 2022-10-27 00:50:01] ...trap/engines/engine.py.321: eval: epoch 38 | batch 390/510 [I 2022-10-27 00:50:01] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 00:50:01] ...trap/engines/engine.py.326: process: 0.29134 | load: 0.00017 [I 2022-10-27 00:50:04] ...trap/engines/engine.py.321: eval: epoch 38 | batch 400/510 [I 2022-10-27 00:50:04] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 00:50:04] ...trap/engines/engine.py.326: process: 0.28817 | load: 0.00015 [I 2022-10-27 00:50:07] ...trap/engines/engine.py.321: eval: epoch 38 | batch 410/510 [I 2022-10-27 00:50:07] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 00:50:07] ...trap/engines/engine.py.326: process: 0.28781 | load: 0.00020 [I 2022-10-27 00:50:10] ...trap/engines/engine.py.321: eval: epoch 38 | batch 420/510 [I 2022-10-27 00:50:10] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:27 [I 2022-10-27 00:50:10] ...trap/engines/engine.py.326: process: 0.38212 | load: 0.00016 [I 2022-10-27 00:50:13] ...trap/engines/engine.py.321: eval: epoch 38 | batch 430/510 [I 2022-10-27 00:50:13] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 00:50:13] ...trap/engines/engine.py.326: process: 0.29743 | load: 0.00018 [I 2022-10-27 00:50:16] ...trap/engines/engine.py.321: eval: epoch 38 | batch 440/510 [I 2022-10-27 00:50:16] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 00:50:16] ...trap/engines/engine.py.326: process: 0.29156 | load: 0.00029 [I 2022-10-27 00:50:19] ...trap/engines/engine.py.321: eval: epoch 38 | batch 450/510 [I 2022-10-27 00:50:19] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 00:50:19] ...trap/engines/engine.py.326: process: 0.28719 | load: 0.00022 [I 2022-10-27 00:50:21] ...trap/engines/engine.py.321: eval: epoch 38 | batch 460/510 [I 2022-10-27 00:50:21] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 00:50:21] ...trap/engines/engine.py.326: process: 0.28872 | load: 0.00015 [I 2022-10-27 00:50:24] ...trap/engines/engine.py.321: eval: epoch 38 | batch 470/510 [I 2022-10-27 00:50:24] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 00:50:24] ...trap/engines/engine.py.326: process: 0.29215 | load: 0.00021 [I 2022-10-27 00:50:27] ...trap/engines/engine.py.321: eval: epoch 38 | batch 480/510 [I 2022-10-27 00:50:27] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 00:50:27] ...trap/engines/engine.py.326: process: 0.29010 | load: 0.00020 [I 2022-10-27 00:50:30] ...trap/engines/engine.py.321: eval: epoch 38 | batch 490/510 [I 2022-10-27 00:50:30] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 00:50:30] ...trap/engines/engine.py.326: process: 0.28964 | load: 0.00019 [I 2022-10-27 00:50:33] ...trap/engines/engine.py.321: eval: epoch 38 | batch 500/510 [I 2022-10-27 00:50:33] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 00:50:33] ...trap/engines/engine.py.326: process: 0.29012 | load: 0.00015 [I 2022-10-27 00:50:36] ...trap/engines/engine.py.321: eval: epoch 38 | batch 510/510 [I 2022-10-27 00:50:36] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 00:50:36] ...trap/engines/engine.py.326: process: 0.28812 | load: 0.00016 [S 2022-10-27 00:50:37] ...trap/engines/engine.py.132: eval_epoch.epoch: 38 [I 2022-10-27 00:50:37] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.683200 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.892800 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.928300 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.666000 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.887300 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.923800 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017342 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010731 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008798 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012538 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010863 [S 2022-10-27 00:50:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009196 [I 2022-10-27 00:50:42] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 00:50:42] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 00:50:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 00:50:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 00:50:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 00:50:43] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 00:50:46] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 00:50:52] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 00:50:52] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 00:50:52] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 00:50:55] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 00:51:01] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 00:51:01] ...trap/engines/engine.py.163: Training model on trainset for epoch 39 [I 2022-10-27 00:51:01] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 00:51:04] ...trap/engines/engine.py.224: train: epoch 39 | batch 0/2304 [I 2022-10-27 00:51:04] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:56 [I 2022-10-27 00:51:04] ...trap/engines/engine.py.229: process: 2.65465 | load: 2.01469 [I 2022-10-27 00:51:04] ...trap/engines/engine.py.230: loss: 0.16255 [I 2022-10-27 00:51:04] ...optimizers/trijoint.py.151: total_norm: 35.724388 [I 2022-10-27 00:51:09] ...trap/engines/engine.py.224: train: epoch 39 | batch 10/2304 [I 2022-10-27 00:51:09] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:02 [I 2022-10-27 00:51:09] ...trap/engines/engine.py.229: process: 0.55375 | load: 0.00374 [I 2022-10-27 00:51:09] ...trap/engines/engine.py.230: loss: 0.20521 [I 2022-10-27 00:51:09] ...optimizers/trijoint.py.151: total_norm: 46.812393 [I 2022-10-27 00:51:15] ...trap/engines/engine.py.224: train: epoch 39 | batch 20/2304 [I 2022-10-27 00:51:15] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:26 [I 2022-10-27 00:51:15] ...trap/engines/engine.py.229: process: 0.56268 | load: 0.00329 [I 2022-10-27 00:51:15] ...trap/engines/engine.py.230: loss: 0.16586 [I 2022-10-27 00:51:15] ...optimizers/trijoint.py.151: total_norm: 13.868319 [I 2022-10-27 00:51:21] ...trap/engines/engine.py.224: train: epoch 39 | batch 30/2304 [I 2022-10-27 00:51:21] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:24 [I 2022-10-27 00:51:21] ...trap/engines/engine.py.229: process: 0.58784 | load: 0.00328 [I 2022-10-27 00:51:21] ...trap/engines/engine.py.230: loss: 0.11561 [I 2022-10-27 00:51:21] ...optimizers/trijoint.py.151: total_norm: 8.829072 [I 2022-10-27 00:51:26] ...trap/engines/engine.py.224: train: epoch 39 | batch 40/2304 [I 2022-10-27 00:51:26] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:13 [I 2022-10-27 00:51:26] ...trap/engines/engine.py.229: process: 0.57712 | load: 0.00369 [I 2022-10-27 00:51:26] ...trap/engines/engine.py.230: loss: 0.18205 [I 2022-10-27 00:51:26] ...optimizers/trijoint.py.151: total_norm: 25.331835 [I 2022-10-27 00:51:32] ...trap/engines/engine.py.224: train: epoch 39 | batch 50/2304 [I 2022-10-27 00:51:32] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:11 [I 2022-10-27 00:51:32] ...trap/engines/engine.py.229: process: 0.57355 | load: 0.00299 [I 2022-10-27 00:51:32] ...trap/engines/engine.py.230: loss: 0.15180 [I 2022-10-27 00:51:32] ...optimizers/trijoint.py.151: total_norm: 17.717348 [I 2022-10-27 00:51:37] ...trap/engines/engine.py.224: train: epoch 39 | batch 60/2304 [I 2022-10-27 00:51:37] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:50 [I 2022-10-27 00:51:37] ...trap/engines/engine.py.229: process: 0.54697 | load: 0.00259 [I 2022-10-27 00:51:37] ...trap/engines/engine.py.230: loss: 0.18165 [I 2022-10-27 00:51:37] ...optimizers/trijoint.py.151: total_norm: 42.871258 [I 2022-10-27 00:51:43] ...trap/engines/engine.py.224: train: epoch 39 | batch 70/2304 [I 2022-10-27 00:51:43] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:40 [I 2022-10-27 00:51:43] ...trap/engines/engine.py.229: process: 0.56722 | load: 0.00291 [I 2022-10-27 00:51:43] ...trap/engines/engine.py.230: loss: 0.19043 [I 2022-10-27 00:51:43] ...optimizers/trijoint.py.151: total_norm: 40.732433 [I 2022-10-27 00:51:48] ...trap/engines/engine.py.224: train: epoch 39 | batch 80/2304 [I 2022-10-27 00:51:48] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:36 [I 2022-10-27 00:51:48] ...trap/engines/engine.py.229: process: 0.55156 | load: 0.00261 [I 2022-10-27 00:51:48] ...trap/engines/engine.py.230: loss: 0.12320 [I 2022-10-27 00:51:48] ...optimizers/trijoint.py.151: total_norm: 18.024492 [I 2022-10-27 00:51:54] ...trap/engines/engine.py.224: train: epoch 39 | batch 90/2304 [I 2022-10-27 00:51:54] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:40 [I 2022-10-27 00:51:54] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00277 [I 2022-10-27 00:51:54] ...trap/engines/engine.py.230: loss: 0.24433 [I 2022-10-27 00:51:54] ...optimizers/trijoint.py.151: total_norm: 27.645176 [I 2022-10-27 00:52:00] ...trap/engines/engine.py.224: train: epoch 39 | batch 100/2304 [I 2022-10-27 00:52:00] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:19 [I 2022-10-27 00:52:00] ...trap/engines/engine.py.229: process: 0.54683 | load: 0.00264 [I 2022-10-27 00:52:00] ...trap/engines/engine.py.230: loss: 0.12399 [I 2022-10-27 00:52:00] ...optimizers/trijoint.py.151: total_norm: 12.571500 [I 2022-10-27 00:52:05] ...trap/engines/engine.py.224: train: epoch 39 | batch 110/2304 [I 2022-10-27 00:52:05] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:13 [I 2022-10-27 00:52:05] ...trap/engines/engine.py.229: process: 0.54200 | load: 0.00269 [I 2022-10-27 00:52:05] ...trap/engines/engine.py.230: loss: 0.18236 [I 2022-10-27 00:52:05] ...optimizers/trijoint.py.151: total_norm: 17.113134 [I 2022-10-27 00:52:11] ...trap/engines/engine.py.224: train: epoch 39 | batch 120/2304 [I 2022-10-27 00:52:11] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:16 [I 2022-10-27 00:52:11] ...trap/engines/engine.py.229: process: 0.56638 | load: 0.00245 [I 2022-10-27 00:52:11] ...trap/engines/engine.py.230: loss: 0.19055 [I 2022-10-27 00:52:11] ...optimizers/trijoint.py.151: total_norm: 18.183390 [I 2022-10-27 00:52:16] ...trap/engines/engine.py.224: train: epoch 39 | batch 130/2304 [I 2022-10-27 00:52:16] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:19 [I 2022-10-27 00:52:16] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00251 [I 2022-10-27 00:52:16] ...trap/engines/engine.py.230: loss: 0.16824 [I 2022-10-27 00:52:16] ...optimizers/trijoint.py.151: total_norm: 17.330423 [I 2022-10-27 00:52:22] ...trap/engines/engine.py.224: train: epoch 39 | batch 140/2304 [I 2022-10-27 00:52:22] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:13 [I 2022-10-27 00:52:22] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00250 [I 2022-10-27 00:52:22] ...trap/engines/engine.py.230: loss: 0.13061 [I 2022-10-27 00:52:22] ...optimizers/trijoint.py.151: total_norm: 17.598917 [I 2022-10-27 00:52:28] ...trap/engines/engine.py.224: train: epoch 39 | batch 150/2304 [I 2022-10-27 00:52:28] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:59 [I 2022-10-27 00:52:28] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00273 [I 2022-10-27 00:52:28] ...trap/engines/engine.py.230: loss: 0.16675 [I 2022-10-27 00:52:28] ...optimizers/trijoint.py.151: total_norm: 17.318880 [I 2022-10-27 00:52:33] ...trap/engines/engine.py.224: train: epoch 39 | batch 160/2304 [I 2022-10-27 00:52:33] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:56 [I 2022-10-27 00:52:33] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00260 [I 2022-10-27 00:52:33] ...trap/engines/engine.py.230: loss: 0.15069 [I 2022-10-27 00:52:33] ...optimizers/trijoint.py.151: total_norm: 17.929268 [I 2022-10-27 00:52:39] ...trap/engines/engine.py.224: train: epoch 39 | batch 170/2304 [I 2022-10-27 00:52:39] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:43 [I 2022-10-27 00:52:39] ...trap/engines/engine.py.229: process: 0.55790 | load: 0.00276 [I 2022-10-27 00:52:39] ...trap/engines/engine.py.230: loss: 0.17043 [I 2022-10-27 00:52:39] ...optimizers/trijoint.py.151: total_norm: 19.108677 [I 2022-10-27 00:52:44] ...trap/engines/engine.py.224: train: epoch 39 | batch 180/2304 [I 2022-10-27 00:52:44] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:53 [I 2022-10-27 00:52:44] ...trap/engines/engine.py.229: process: 0.56479 | load: 0.00280 [I 2022-10-27 00:52:44] ...trap/engines/engine.py.230: loss: 0.27353 [I 2022-10-27 00:52:44] ...optimizers/trijoint.py.151: total_norm: 11.171173 [I 2022-10-27 00:52:50] ...trap/engines/engine.py.224: train: epoch 39 | batch 190/2304 [I 2022-10-27 00:52:50] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:50 [I 2022-10-27 00:52:50] ...trap/engines/engine.py.229: process: 0.55842 | load: 0.00258 [I 2022-10-27 00:52:50] ...trap/engines/engine.py.230: loss: 0.16332 [I 2022-10-27 00:52:50] ...optimizers/trijoint.py.151: total_norm: 18.824055 [I 2022-10-27 00:52:56] ...trap/engines/engine.py.224: train: epoch 39 | batch 200/2304 [I 2022-10-27 00:52:56] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:37 [I 2022-10-27 00:52:56] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00263 [I 2022-10-27 00:52:56] ...trap/engines/engine.py.230: loss: 0.14351 [I 2022-10-27 00:52:56] ...optimizers/trijoint.py.151: total_norm: 24.177469 [I 2022-10-27 00:53:01] ...trap/engines/engine.py.224: train: epoch 39 | batch 210/2304 [I 2022-10-27 00:53:01] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:30 [I 2022-10-27 00:53:01] ...trap/engines/engine.py.229: process: 0.56366 | load: 0.00253 [I 2022-10-27 00:53:01] ...trap/engines/engine.py.230: loss: 0.13176 [I 2022-10-27 00:53:01] ...optimizers/trijoint.py.151: total_norm: 13.684412 [I 2022-10-27 00:53:07] ...trap/engines/engine.py.224: train: epoch 39 | batch 220/2304 [I 2022-10-27 00:53:07] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:30 [I 2022-10-27 00:53:07] ...trap/engines/engine.py.229: process: 0.56822 | load: 0.00266 [I 2022-10-27 00:53:07] ...trap/engines/engine.py.230: loss: 0.13279 [I 2022-10-27 00:53:07] ...optimizers/trijoint.py.151: total_norm: 12.782692 [I 2022-10-27 00:53:12] ...trap/engines/engine.py.224: train: epoch 39 | batch 230/2304 [I 2022-10-27 00:53:12] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:27 [I 2022-10-27 00:53:12] ...trap/engines/engine.py.229: process: 0.56723 | load: 0.00246 [I 2022-10-27 00:53:12] ...trap/engines/engine.py.230: loss: 0.17931 [I 2022-10-27 00:53:12] ...optimizers/trijoint.py.151: total_norm: 17.444016 [I 2022-10-27 00:53:18] ...trap/engines/engine.py.224: train: epoch 39 | batch 240/2304 [I 2022-10-27 00:53:18] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:27 [I 2022-10-27 00:53:18] ...trap/engines/engine.py.229: process: 0.57611 | load: 0.00247 [I 2022-10-27 00:53:18] ...trap/engines/engine.py.230: loss: 0.19363 [I 2022-10-27 00:53:18] ...optimizers/trijoint.py.151: total_norm: 21.293753 [I 2022-10-27 00:53:24] ...trap/engines/engine.py.224: train: epoch 39 | batch 250/2304 [I 2022-10-27 00:53:24] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:09 [I 2022-10-27 00:53:24] ...trap/engines/engine.py.229: process: 0.56448 | load: 0.00244 [I 2022-10-27 00:53:24] ...trap/engines/engine.py.230: loss: 0.14169 [I 2022-10-27 00:53:24] ...optimizers/trijoint.py.151: total_norm: 17.557011 [I 2022-10-27 00:53:29] ...trap/engines/engine.py.224: train: epoch 39 | batch 260/2304 [I 2022-10-27 00:53:29] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:05 [I 2022-10-27 00:53:29] ...trap/engines/engine.py.229: process: 0.57187 | load: 0.00249 [I 2022-10-27 00:53:29] ...trap/engines/engine.py.230: loss: 0.21596 [I 2022-10-27 00:53:29] ...optimizers/trijoint.py.151: total_norm: 25.724102 [I 2022-10-27 00:53:35] ...trap/engines/engine.py.224: train: epoch 39 | batch 270/2304 [I 2022-10-27 00:53:35] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:59 [I 2022-10-27 00:53:35] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00252 [I 2022-10-27 00:53:35] ...trap/engines/engine.py.230: loss: 0.19179 [I 2022-10-27 00:53:35] ...optimizers/trijoint.py.151: total_norm: 36.054836 [I 2022-10-27 00:53:40] ...trap/engines/engine.py.224: train: epoch 39 | batch 280/2304 [I 2022-10-27 00:53:40] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:52 [I 2022-10-27 00:53:40] ...trap/engines/engine.py.229: process: 0.56947 | load: 0.00243 [I 2022-10-27 00:53:40] ...trap/engines/engine.py.230: loss: 0.15859 [I 2022-10-27 00:53:40] ...optimizers/trijoint.py.151: total_norm: 19.080942 [I 2022-10-27 00:53:46] ...trap/engines/engine.py.224: train: epoch 39 | batch 290/2304 [I 2022-10-27 00:53:46] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:41 [I 2022-10-27 00:53:46] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00246 [I 2022-10-27 00:53:46] ...trap/engines/engine.py.230: loss: 0.13907 [I 2022-10-27 00:53:46] ...optimizers/trijoint.py.151: total_norm: 19.329742 [I 2022-10-27 00:53:52] ...trap/engines/engine.py.224: train: epoch 39 | batch 300/2304 [I 2022-10-27 00:53:52] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:37 [I 2022-10-27 00:53:52] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00254 [I 2022-10-27 00:53:52] ...trap/engines/engine.py.230: loss: 0.13416 [I 2022-10-27 00:53:52] ...optimizers/trijoint.py.151: total_norm: 18.676012 [I 2022-10-27 00:53:57] ...trap/engines/engine.py.224: train: epoch 39 | batch 310/2304 [I 2022-10-27 00:53:57] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:38 [I 2022-10-27 00:53:57] ...trap/engines/engine.py.229: process: 0.55072 | load: 0.00262 [I 2022-10-27 00:53:57] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-27 00:53:57] ...optimizers/trijoint.py.151: total_norm: 16.420343 [I 2022-10-27 00:54:03] ...trap/engines/engine.py.224: train: epoch 39 | batch 320/2304 [I 2022-10-27 00:54:03] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:27 [I 2022-10-27 00:54:03] ...trap/engines/engine.py.229: process: 0.56288 | load: 0.00246 [I 2022-10-27 00:54:03] ...trap/engines/engine.py.230: loss: 0.18911 [I 2022-10-27 00:54:03] ...optimizers/trijoint.py.151: total_norm: 36.820122 [I 2022-10-27 00:54:08] ...trap/engines/engine.py.224: train: epoch 39 | batch 330/2304 [I 2022-10-27 00:54:08] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:25 [I 2022-10-27 00:54:08] ...trap/engines/engine.py.229: process: 0.56532 | load: 0.00244 [I 2022-10-27 00:54:08] ...trap/engines/engine.py.230: loss: 0.16422 [I 2022-10-27 00:54:08] ...optimizers/trijoint.py.151: total_norm: 16.559872 [I 2022-10-27 00:54:14] ...trap/engines/engine.py.224: train: epoch 39 | batch 340/2304 [I 2022-10-27 00:54:14] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:16 [I 2022-10-27 00:54:14] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00258 [I 2022-10-27 00:54:14] ...trap/engines/engine.py.230: loss: 0.13821 [I 2022-10-27 00:54:14] ...optimizers/trijoint.py.151: total_norm: 26.548067 [I 2022-10-27 00:54:20] ...trap/engines/engine.py.224: train: epoch 39 | batch 350/2304 [I 2022-10-27 00:54:20] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:01 [I 2022-10-27 00:54:20] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00244 [I 2022-10-27 00:54:20] ...trap/engines/engine.py.230: loss: 0.16904 [I 2022-10-27 00:54:20] ...optimizers/trijoint.py.151: total_norm: 13.313154 [I 2022-10-27 00:54:25] ...trap/engines/engine.py.224: train: epoch 39 | batch 360/2304 [I 2022-10-27 00:54:25] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:58 [I 2022-10-27 00:54:25] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00246 [I 2022-10-27 00:54:25] ...trap/engines/engine.py.230: loss: 0.16190 [I 2022-10-27 00:54:25] ...optimizers/trijoint.py.151: total_norm: 33.222721 [I 2022-10-27 00:54:31] ...trap/engines/engine.py.224: train: epoch 39 | batch 370/2304 [I 2022-10-27 00:54:31] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:53 [I 2022-10-27 00:54:31] ...trap/engines/engine.py.229: process: 0.54271 | load: 0.00262 [I 2022-10-27 00:54:31] ...trap/engines/engine.py.230: loss: 0.14983 [I 2022-10-27 00:54:31] ...optimizers/trijoint.py.151: total_norm: 14.720654 [I 2022-10-27 00:54:36] ...trap/engines/engine.py.224: train: epoch 39 | batch 380/2304 [I 2022-10-27 00:54:36] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:53 [I 2022-10-27 00:54:36] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00247 [I 2022-10-27 00:54:36] ...trap/engines/engine.py.230: loss: 0.17190 [I 2022-10-27 00:54:36] ...optimizers/trijoint.py.151: total_norm: 14.106965 [I 2022-10-27 00:54:42] ...trap/engines/engine.py.224: train: epoch 39 | batch 390/2304 [I 2022-10-27 00:54:42] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:45 [I 2022-10-27 00:54:42] ...trap/engines/engine.py.229: process: 0.55228 | load: 0.00245 [I 2022-10-27 00:54:42] ...trap/engines/engine.py.230: loss: 0.13137 [I 2022-10-27 00:54:42] ...optimizers/trijoint.py.151: total_norm: 19.831095 [I 2022-10-27 00:54:47] ...trap/engines/engine.py.224: train: epoch 39 | batch 400/2304 [I 2022-10-27 00:54:47] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:48 [I 2022-10-27 00:54:47] ...trap/engines/engine.py.229: process: 0.55858 | load: 0.00301 [I 2022-10-27 00:54:47] ...trap/engines/engine.py.230: loss: 0.11835 [I 2022-10-27 00:54:47] ...optimizers/trijoint.py.151: total_norm: 13.788038 [I 2022-10-27 00:54:53] ...trap/engines/engine.py.224: train: epoch 39 | batch 410/2304 [I 2022-10-27 00:54:53] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:36 [I 2022-10-27 00:54:53] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00252 [I 2022-10-27 00:54:53] ...trap/engines/engine.py.230: loss: 0.16578 [I 2022-10-27 00:54:53] ...optimizers/trijoint.py.151: total_norm: 21.243797 [I 2022-10-27 00:54:59] ...trap/engines/engine.py.224: train: epoch 39 | batch 420/2304 [I 2022-10-27 00:54:59] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:25 [I 2022-10-27 00:54:59] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00251 [I 2022-10-27 00:54:59] ...trap/engines/engine.py.230: loss: 0.14920 [I 2022-10-27 00:54:59] ...optimizers/trijoint.py.151: total_norm: 14.850437 [I 2022-10-27 00:55:04] ...trap/engines/engine.py.224: train: epoch 39 | batch 430/2304 [I 2022-10-27 00:55:04] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:17 [I 2022-10-27 00:55:04] ...trap/engines/engine.py.229: process: 0.54708 | load: 0.00250 [I 2022-10-27 00:55:04] ...trap/engines/engine.py.230: loss: 0.18024 [I 2022-10-27 00:55:04] ...optimizers/trijoint.py.151: total_norm: 17.534508 [I 2022-10-27 00:55:10] ...trap/engines/engine.py.224: train: epoch 39 | batch 440/2304 [I 2022-10-27 00:55:10] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:24 [I 2022-10-27 00:55:10] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00248 [I 2022-10-27 00:55:10] ...trap/engines/engine.py.230: loss: 0.15388 [I 2022-10-27 00:55:10] ...optimizers/trijoint.py.151: total_norm: 13.594280 [I 2022-10-27 00:55:15] ...trap/engines/engine.py.224: train: epoch 39 | batch 450/2304 [I 2022-10-27 00:55:15] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:10 [I 2022-10-27 00:55:15] ...trap/engines/engine.py.229: process: 0.55525 | load: 0.00243 [I 2022-10-27 00:55:15] ...trap/engines/engine.py.230: loss: 0.19161 [I 2022-10-27 00:55:15] ...optimizers/trijoint.py.151: total_norm: 32.956192 [I 2022-10-27 00:55:21] ...trap/engines/engine.py.224: train: epoch 39 | batch 460/2304 [I 2022-10-27 00:55:21] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:04 [I 2022-10-27 00:55:21] ...trap/engines/engine.py.229: process: 0.54885 | load: 0.00249 [I 2022-10-27 00:55:21] ...trap/engines/engine.py.230: loss: 0.18440 [I 2022-10-27 00:55:21] ...optimizers/trijoint.py.151: total_norm: 28.267071 [I 2022-10-27 00:55:27] ...trap/engines/engine.py.224: train: epoch 39 | batch 470/2304 [I 2022-10-27 00:55:27] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:08 [I 2022-10-27 00:55:27] ...trap/engines/engine.py.229: process: 0.55932 | load: 0.00249 [I 2022-10-27 00:55:27] ...trap/engines/engine.py.230: loss: 0.16152 [I 2022-10-27 00:55:27] ...optimizers/trijoint.py.151: total_norm: 27.170467 [I 2022-10-27 00:55:32] ...trap/engines/engine.py.224: train: epoch 39 | batch 480/2304 [I 2022-10-27 00:55:32] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:53 [I 2022-10-27 00:55:32] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00252 [I 2022-10-27 00:55:32] ...trap/engines/engine.py.230: loss: 0.14979 [I 2022-10-27 00:55:32] ...optimizers/trijoint.py.151: total_norm: 18.150333 [I 2022-10-27 00:55:38] ...trap/engines/engine.py.224: train: epoch 39 | batch 490/2304 [I 2022-10-27 00:55:38] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:51 [I 2022-10-27 00:55:38] ...trap/engines/engine.py.229: process: 0.55529 | load: 0.00245 [I 2022-10-27 00:55:38] ...trap/engines/engine.py.230: loss: 0.11447 [I 2022-10-27 00:55:38] ...optimizers/trijoint.py.151: total_norm: 12.588352 [I 2022-10-27 00:55:43] ...trap/engines/engine.py.224: train: epoch 39 | batch 500/2304 [I 2022-10-27 00:55:43] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:17:04 [I 2022-10-27 00:55:43] ...trap/engines/engine.py.229: process: 0.57018 | load: 0.00248 [I 2022-10-27 00:55:43] ...trap/engines/engine.py.230: loss: 0.15175 [I 2022-10-27 00:55:43] ...optimizers/trijoint.py.151: total_norm: 14.979614 [I 2022-10-27 00:55:49] ...trap/engines/engine.py.224: train: epoch 39 | batch 510/2304 [I 2022-10-27 00:55:49] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:42 [I 2022-10-27 00:55:49] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00263 [I 2022-10-27 00:55:49] ...trap/engines/engine.py.230: loss: 0.20190 [I 2022-10-27 00:55:49] ...optimizers/trijoint.py.151: total_norm: 23.726568 [I 2022-10-27 00:55:55] ...trap/engines/engine.py.224: train: epoch 39 | batch 520/2304 [I 2022-10-27 00:55:55] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:46 [I 2022-10-27 00:55:55] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00252 [I 2022-10-27 00:55:55] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-27 00:55:55] ...optimizers/trijoint.py.151: total_norm: 22.290337 [I 2022-10-27 00:56:00] ...trap/engines/engine.py.224: train: epoch 39 | batch 530/2304 [I 2022-10-27 00:56:00] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:30 [I 2022-10-27 00:56:00] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00256 [I 2022-10-27 00:56:00] ...trap/engines/engine.py.230: loss: 0.14148 [I 2022-10-27 00:56:00] ...optimizers/trijoint.py.151: total_norm: 9.227839 [I 2022-10-27 00:56:06] ...trap/engines/engine.py.224: train: epoch 39 | batch 540/2304 [I 2022-10-27 00:56:06] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:16 [I 2022-10-27 00:56:06] ...trap/engines/engine.py.229: process: 0.54381 | load: 0.00266 [I 2022-10-27 00:56:06] ...trap/engines/engine.py.230: loss: 0.16657 [I 2022-10-27 00:56:06] ...optimizers/trijoint.py.151: total_norm: 27.907648 [I 2022-10-27 00:56:11] ...trap/engines/engine.py.224: train: epoch 39 | batch 550/2304 [I 2022-10-27 00:56:11] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:09 [I 2022-10-27 00:56:11] ...trap/engines/engine.py.229: process: 0.54583 | load: 0.00259 [I 2022-10-27 00:56:11] ...trap/engines/engine.py.230: loss: 0.16232 [I 2022-10-27 00:56:11] ...optimizers/trijoint.py.151: total_norm: 31.739986 [I 2022-10-27 00:56:17] ...trap/engines/engine.py.224: train: epoch 39 | batch 560/2304 [I 2022-10-27 00:56:17] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:12 [I 2022-10-27 00:56:17] ...trap/engines/engine.py.229: process: 0.57145 | load: 0.00267 [I 2022-10-27 00:56:17] ...trap/engines/engine.py.230: loss: 0.22321 [I 2022-10-27 00:56:17] ...optimizers/trijoint.py.151: total_norm: 15.529819 [I 2022-10-27 00:56:23] ...trap/engines/engine.py.224: train: epoch 39 | batch 570/2304 [I 2022-10-27 00:56:23] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:10 [I 2022-10-27 00:56:23] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00272 [I 2022-10-27 00:56:23] ...trap/engines/engine.py.230: loss: 0.11114 [I 2022-10-27 00:56:23] ...optimizers/trijoint.py.151: total_norm: 8.458391 [I 2022-10-27 00:56:28] ...trap/engines/engine.py.224: train: epoch 39 | batch 580/2304 [I 2022-10-27 00:56:28] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:02 [I 2022-10-27 00:56:28] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00269 [I 2022-10-27 00:56:28] ...trap/engines/engine.py.230: loss: 0.11627 [I 2022-10-27 00:56:28] ...optimizers/trijoint.py.151: total_norm: 19.898108 [I 2022-10-27 00:56:34] ...trap/engines/engine.py.224: train: epoch 39 | batch 590/2304 [I 2022-10-27 00:56:34] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:16:10 [I 2022-10-27 00:56:34] ...trap/engines/engine.py.229: process: 0.56680 | load: 0.00255 [I 2022-10-27 00:56:34] ...trap/engines/engine.py.230: loss: 0.13258 [I 2022-10-27 00:56:34] ...optimizers/trijoint.py.151: total_norm: 10.054523 [I 2022-10-27 00:56:39] ...trap/engines/engine.py.224: train: epoch 39 | batch 600/2304 [I 2022-10-27 00:56:39] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:48 [I 2022-10-27 00:56:39] ...trap/engines/engine.py.229: process: 0.55033 | load: 0.00254 [I 2022-10-27 00:56:39] ...trap/engines/engine.py.230: loss: 0.23371 [I 2022-10-27 00:56:39] ...optimizers/trijoint.py.151: total_norm: 19.623224 [I 2022-10-27 00:56:45] ...trap/engines/engine.py.224: train: epoch 39 | batch 610/2304 [I 2022-10-27 00:56:45] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:37 [I 2022-10-27 00:56:45] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00255 [I 2022-10-27 00:56:45] ...trap/engines/engine.py.230: loss: 0.19811 [I 2022-10-27 00:56:45] ...optimizers/trijoint.py.151: total_norm: 37.457363 [I 2022-10-27 00:56:50] ...trap/engines/engine.py.224: train: epoch 39 | batch 620/2304 [I 2022-10-27 00:56:50] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:34 [I 2022-10-27 00:56:50] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00265 [I 2022-10-27 00:56:50] ...trap/engines/engine.py.230: loss: 0.17035 [I 2022-10-27 00:56:50] ...optimizers/trijoint.py.151: total_norm: 10.195940 [I 2022-10-27 00:56:56] ...trap/engines/engine.py.224: train: epoch 39 | batch 630/2304 [I 2022-10-27 00:56:56] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:39 [I 2022-10-27 00:56:56] ...trap/engines/engine.py.229: process: 0.57407 | load: 0.00244 [I 2022-10-27 00:56:56] ...trap/engines/engine.py.230: loss: 0.16680 [I 2022-10-27 00:56:56] ...optimizers/trijoint.py.151: total_norm: 20.863918 [I 2022-10-27 00:57:02] ...trap/engines/engine.py.224: train: epoch 39 | batch 640/2304 [I 2022-10-27 00:57:02] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:27 [I 2022-10-27 00:57:02] ...trap/engines/engine.py.229: process: 0.55335 | load: 0.00250 [I 2022-10-27 00:57:02] ...trap/engines/engine.py.230: loss: 0.14798 [I 2022-10-27 00:57:02] ...optimizers/trijoint.py.151: total_norm: 15.572764 [I 2022-10-27 00:57:07] ...trap/engines/engine.py.224: train: epoch 39 | batch 650/2304 [I 2022-10-27 00:57:07] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:28 [I 2022-10-27 00:57:07] ...trap/engines/engine.py.229: process: 0.57814 | load: 0.00250 [I 2022-10-27 00:57:07] ...trap/engines/engine.py.230: loss: 0.17260 [I 2022-10-27 00:57:07] ...optimizers/trijoint.py.151: total_norm: 21.376444 [I 2022-10-27 00:57:13] ...trap/engines/engine.py.224: train: epoch 39 | batch 660/2304 [I 2022-10-27 00:57:13] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:37 [I 2022-10-27 00:57:13] ...trap/engines/engine.py.229: process: 0.56668 | load: 0.00246 [I 2022-10-27 00:57:13] ...trap/engines/engine.py.230: loss: 0.18016 [I 2022-10-27 00:57:13] ...optimizers/trijoint.py.151: total_norm: 15.858694 [I 2022-10-27 00:57:19] ...trap/engines/engine.py.224: train: epoch 39 | batch 670/2304 [I 2022-10-27 00:57:19] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:23 [I 2022-10-27 00:57:19] ...trap/engines/engine.py.229: process: 0.57284 | load: 0.00261 [I 2022-10-27 00:57:19] ...trap/engines/engine.py.230: loss: 0.14132 [I 2022-10-27 00:57:19] ...optimizers/trijoint.py.151: total_norm: 24.218533 [I 2022-10-27 00:57:24] ...trap/engines/engine.py.224: train: epoch 39 | batch 680/2304 [I 2022-10-27 00:57:24] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:02 [I 2022-10-27 00:57:24] ...trap/engines/engine.py.229: process: 0.55263 | load: 0.00245 [I 2022-10-27 00:57:24] ...trap/engines/engine.py.230: loss: 0.16254 [I 2022-10-27 00:57:24] ...optimizers/trijoint.py.151: total_norm: 13.245507 [I 2022-10-27 00:57:30] ...trap/engines/engine.py.224: train: epoch 39 | batch 690/2304 [I 2022-10-27 00:57:30] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:06 [I 2022-10-27 00:57:30] ...trap/engines/engine.py.229: process: 0.57713 | load: 0.00248 [I 2022-10-27 00:57:30] ...trap/engines/engine.py.230: loss: 0.16454 [I 2022-10-27 00:57:30] ...optimizers/trijoint.py.151: total_norm: 23.264736 [I 2022-10-27 00:57:35] ...trap/engines/engine.py.224: train: epoch 39 | batch 700/2304 [I 2022-10-27 00:57:35] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:15:08 [I 2022-10-27 00:57:35] ...trap/engines/engine.py.229: process: 0.58229 | load: 0.00245 [I 2022-10-27 00:57:35] ...trap/engines/engine.py.230: loss: 0.19186 [I 2022-10-27 00:57:35] ...optimizers/trijoint.py.151: total_norm: 22.750120 [I 2022-10-27 00:57:41] ...trap/engines/engine.py.224: train: epoch 39 | batch 710/2304 [I 2022-10-27 00:57:41] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:51 [I 2022-10-27 00:57:41] ...trap/engines/engine.py.229: process: 0.55198 | load: 0.00251 [I 2022-10-27 00:57:41] ...trap/engines/engine.py.230: loss: 0.18124 [I 2022-10-27 00:57:41] ...optimizers/trijoint.py.151: total_norm: 20.655569 [I 2022-10-27 00:57:47] ...trap/engines/engine.py.224: train: epoch 39 | batch 720/2304 [I 2022-10-27 00:57:47] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:47 [I 2022-10-27 00:57:47] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00248 [I 2022-10-27 00:57:47] ...trap/engines/engine.py.230: loss: 0.13698 [I 2022-10-27 00:57:47] ...optimizers/trijoint.py.151: total_norm: 18.332958 [I 2022-10-27 00:57:52] ...trap/engines/engine.py.224: train: epoch 39 | batch 730/2304 [I 2022-10-27 00:57:52] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:40 [I 2022-10-27 00:57:52] ...trap/engines/engine.py.229: process: 0.56265 | load: 0.00250 [I 2022-10-27 00:57:52] ...trap/engines/engine.py.230: loss: 0.14330 [I 2022-10-27 00:57:52] ...optimizers/trijoint.py.151: total_norm: 9.941998 [I 2022-10-27 00:57:58] ...trap/engines/engine.py.224: train: epoch 39 | batch 740/2304 [I 2022-10-27 00:57:58] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:29 [I 2022-10-27 00:57:58] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00249 [I 2022-10-27 00:57:58] ...trap/engines/engine.py.230: loss: 0.19023 [I 2022-10-27 00:57:58] ...optimizers/trijoint.py.151: total_norm: 17.936121 [I 2022-10-27 00:58:03] ...trap/engines/engine.py.224: train: epoch 39 | batch 750/2304 [I 2022-10-27 00:58:03] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:26 [I 2022-10-27 00:58:03] ...trap/engines/engine.py.229: process: 0.55518 | load: 0.00287 [I 2022-10-27 00:58:03] ...trap/engines/engine.py.230: loss: 0.23348 [I 2022-10-27 00:58:03] ...optimizers/trijoint.py.151: total_norm: 15.329699 [I 2022-10-27 00:58:09] ...trap/engines/engine.py.224: train: epoch 39 | batch 760/2304 [I 2022-10-27 00:58:09] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:23 [I 2022-10-27 00:58:09] ...trap/engines/engine.py.229: process: 0.56519 | load: 0.00244 [I 2022-10-27 00:58:09] ...trap/engines/engine.py.230: loss: 0.19437 [I 2022-10-27 00:58:09] ...optimizers/trijoint.py.151: total_norm: 17.802153 [I 2022-10-27 00:58:15] ...trap/engines/engine.py.224: train: epoch 39 | batch 770/2304 [I 2022-10-27 00:58:15] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:30 [I 2022-10-27 00:58:15] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00243 [I 2022-10-27 00:58:15] ...trap/engines/engine.py.230: loss: 0.17113 [I 2022-10-27 00:58:15] ...optimizers/trijoint.py.151: total_norm: 19.873007 [I 2022-10-27 00:58:20] ...trap/engines/engine.py.224: train: epoch 39 | batch 780/2304 [I 2022-10-27 00:58:20] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:15 [I 2022-10-27 00:58:20] ...trap/engines/engine.py.229: process: 0.57153 | load: 0.00246 [I 2022-10-27 00:58:20] ...trap/engines/engine.py.230: loss: 0.17720 [I 2022-10-27 00:58:20] ...optimizers/trijoint.py.151: total_norm: 17.049372 [I 2022-10-27 00:58:26] ...trap/engines/engine.py.224: train: epoch 39 | batch 790/2304 [I 2022-10-27 00:58:26] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:02 [I 2022-10-27 00:58:26] ...trap/engines/engine.py.229: process: 0.56126 | load: 0.00252 [I 2022-10-27 00:58:26] ...trap/engines/engine.py.230: loss: 0.14073 [I 2022-10-27 00:58:26] ...optimizers/trijoint.py.151: total_norm: 17.988689 [I 2022-10-27 00:58:31] ...trap/engines/engine.py.224: train: epoch 39 | batch 800/2304 [I 2022-10-27 00:58:31] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:59 [I 2022-10-27 00:58:31] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00249 [I 2022-10-27 00:58:31] ...trap/engines/engine.py.230: loss: 0.18010 [I 2022-10-27 00:58:31] ...optimizers/trijoint.py.151: total_norm: 14.535227 [I 2022-10-27 00:58:37] ...trap/engines/engine.py.224: train: epoch 39 | batch 810/2304 [I 2022-10-27 00:58:37] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:54 [I 2022-10-27 00:58:37] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00243 [I 2022-10-27 00:58:37] ...trap/engines/engine.py.230: loss: 0.16874 [I 2022-10-27 00:58:37] ...optimizers/trijoint.py.151: total_norm: 14.636301 [I 2022-10-27 00:58:42] ...trap/engines/engine.py.224: train: epoch 39 | batch 820/2304 [I 2022-10-27 00:58:42] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:46 [I 2022-10-27 00:58:42] ...trap/engines/engine.py.229: process: 0.54585 | load: 0.00248 [I 2022-10-27 00:58:42] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-27 00:58:42] ...optimizers/trijoint.py.151: total_norm: 11.028107 [I 2022-10-27 00:58:48] ...trap/engines/engine.py.224: train: epoch 39 | batch 830/2304 [I 2022-10-27 00:58:48] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:42 [I 2022-10-27 00:58:48] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00247 [I 2022-10-27 00:58:48] ...trap/engines/engine.py.230: loss: 0.15455 [I 2022-10-27 00:58:48] ...optimizers/trijoint.py.151: total_norm: 15.542529 [I 2022-10-27 00:58:54] ...trap/engines/engine.py.224: train: epoch 39 | batch 840/2304 [I 2022-10-27 00:58:54] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:38 [I 2022-10-27 00:58:54] ...trap/engines/engine.py.229: process: 0.56280 | load: 0.00247 [I 2022-10-27 00:58:54] ...trap/engines/engine.py.230: loss: 0.14009 [I 2022-10-27 00:58:54] ...optimizers/trijoint.py.151: total_norm: 31.687763 [I 2022-10-27 00:58:59] ...trap/engines/engine.py.224: train: epoch 39 | batch 850/2304 [I 2022-10-27 00:58:59] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:29 [I 2022-10-27 00:58:59] ...trap/engines/engine.py.229: process: 0.55140 | load: 0.00247 [I 2022-10-27 00:58:59] ...trap/engines/engine.py.230: loss: 0.11509 [I 2022-10-27 00:58:59] ...optimizers/trijoint.py.151: total_norm: 6.879703 [I 2022-10-27 00:59:05] ...trap/engines/engine.py.224: train: epoch 39 | batch 860/2304 [I 2022-10-27 00:59:05] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:25 [I 2022-10-27 00:59:05] ...trap/engines/engine.py.229: process: 0.55614 | load: 0.00249 [I 2022-10-27 00:59:05] ...trap/engines/engine.py.230: loss: 0.13081 [I 2022-10-27 00:59:05] ...optimizers/trijoint.py.151: total_norm: 16.400288 [I 2022-10-27 00:59:10] ...trap/engines/engine.py.224: train: epoch 39 | batch 870/2304 [I 2022-10-27 00:59:10] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:28 [I 2022-10-27 00:59:10] ...trap/engines/engine.py.229: process: 0.57001 | load: 0.00252 [I 2022-10-27 00:59:10] ...trap/engines/engine.py.230: loss: 0.17511 [I 2022-10-27 00:59:10] ...optimizers/trijoint.py.151: total_norm: 21.640762 [I 2022-10-27 00:59:16] ...trap/engines/engine.py.224: train: epoch 39 | batch 880/2304 [I 2022-10-27 00:59:16] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:18 [I 2022-10-27 00:59:16] ...trap/engines/engine.py.229: process: 0.56704 | load: 0.00247 [I 2022-10-27 00:59:16] ...trap/engines/engine.py.230: loss: 0.23055 [I 2022-10-27 00:59:16] ...optimizers/trijoint.py.151: total_norm: 18.507441 [I 2022-10-27 00:59:22] ...trap/engines/engine.py.224: train: epoch 39 | batch 890/2304 [I 2022-10-27 00:59:22] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:17 [I 2022-10-27 00:59:22] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00244 [I 2022-10-27 00:59:22] ...trap/engines/engine.py.230: loss: 0.19671 [I 2022-10-27 00:59:22] ...optimizers/trijoint.py.151: total_norm: 31.836227 [I 2022-10-27 00:59:27] ...trap/engines/engine.py.224: train: epoch 39 | batch 900/2304 [I 2022-10-27 00:59:27] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:04 [I 2022-10-27 00:59:27] ...trap/engines/engine.py.229: process: 0.54570 | load: 0.00245 [I 2022-10-27 00:59:27] ...trap/engines/engine.py.230: loss: 0.11279 [I 2022-10-27 00:59:27] ...optimizers/trijoint.py.151: total_norm: 7.593923 [I 2022-10-27 00:59:33] ...trap/engines/engine.py.224: train: epoch 39 | batch 910/2304 [I 2022-10-27 00:59:33] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:54 [I 2022-10-27 00:59:33] ...trap/engines/engine.py.229: process: 0.54860 | load: 0.00244 [I 2022-10-27 00:59:33] ...trap/engines/engine.py.230: loss: 0.15437 [I 2022-10-27 00:59:33] ...optimizers/trijoint.py.151: total_norm: 36.943626 [I 2022-10-27 00:59:39] ...trap/engines/engine.py.224: train: epoch 39 | batch 920/2304 [I 2022-10-27 00:59:39] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:51 [I 2022-10-27 00:59:39] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00242 [I 2022-10-27 00:59:39] ...trap/engines/engine.py.230: loss: 0.16128 [I 2022-10-27 00:59:39] ...optimizers/trijoint.py.151: total_norm: 17.964359 [I 2022-10-27 00:59:44] ...trap/engines/engine.py.224: train: epoch 39 | batch 930/2304 [I 2022-10-27 00:59:44] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:44 [I 2022-10-27 00:59:44] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00247 [I 2022-10-27 00:59:44] ...trap/engines/engine.py.230: loss: 0.18019 [I 2022-10-27 00:59:44] ...optimizers/trijoint.py.151: total_norm: 44.297024 [I 2022-10-27 00:59:50] ...trap/engines/engine.py.224: train: epoch 39 | batch 940/2304 [I 2022-10-27 00:59:50] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:43 [I 2022-10-27 00:59:50] ...trap/engines/engine.py.229: process: 0.56293 | load: 0.00243 [I 2022-10-27 00:59:50] ...trap/engines/engine.py.230: loss: 0.16651 [I 2022-10-27 00:59:50] ...optimizers/trijoint.py.151: total_norm: 12.800626 [I 2022-10-27 00:59:55] ...trap/engines/engine.py.224: train: epoch 39 | batch 950/2304 [I 2022-10-27 00:59:55] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:37 [I 2022-10-27 00:59:55] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00246 [I 2022-10-27 00:59:55] ...trap/engines/engine.py.230: loss: 0.16614 [I 2022-10-27 00:59:55] ...optimizers/trijoint.py.151: total_norm: 23.310255 [I 2022-10-27 01:00:01] ...trap/engines/engine.py.224: train: epoch 39 | batch 960/2304 [I 2022-10-27 01:00:01] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:31 [I 2022-10-27 01:00:01] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00241 [I 2022-10-27 01:00:01] ...trap/engines/engine.py.230: loss: 0.16356 [I 2022-10-27 01:00:01] ...optimizers/trijoint.py.151: total_norm: 18.771902 [I 2022-10-27 01:00:07] ...trap/engines/engine.py.224: train: epoch 39 | batch 970/2304 [I 2022-10-27 01:00:07] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:28 [I 2022-10-27 01:00:07] ...trap/engines/engine.py.229: process: 0.56402 | load: 0.00253 [I 2022-10-27 01:00:07] ...trap/engines/engine.py.230: loss: 0.18062 [I 2022-10-27 01:00:07] ...optimizers/trijoint.py.151: total_norm: 25.678719 [I 2022-10-27 01:00:12] ...trap/engines/engine.py.224: train: epoch 39 | batch 980/2304 [I 2022-10-27 01:00:12] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:23 [I 2022-10-27 01:00:12] ...trap/engines/engine.py.229: process: 0.55421 | load: 0.00247 [I 2022-10-27 01:00:12] ...trap/engines/engine.py.230: loss: 0.17885 [I 2022-10-27 01:00:12] ...optimizers/trijoint.py.151: total_norm: 16.751040 [I 2022-10-27 01:00:18] ...trap/engines/engine.py.224: train: epoch 39 | batch 990/2304 [I 2022-10-27 01:00:18] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:19 [I 2022-10-27 01:00:18] ...trap/engines/engine.py.229: process: 0.56822 | load: 0.00295 [I 2022-10-27 01:00:18] ...trap/engines/engine.py.230: loss: 0.16752 [I 2022-10-27 01:00:18] ...optimizers/trijoint.py.151: total_norm: 13.735640 [I 2022-10-27 01:00:23] ...trap/engines/engine.py.224: train: epoch 39 | batch 1000/2304 [I 2022-10-27 01:00:23] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:16 [I 2022-10-27 01:00:23] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00249 [I 2022-10-27 01:00:23] ...trap/engines/engine.py.230: loss: 0.21311 [I 2022-10-27 01:00:23] ...optimizers/trijoint.py.151: total_norm: 21.533037 [I 2022-10-27 01:00:29] ...trap/engines/engine.py.224: train: epoch 39 | batch 1010/2304 [I 2022-10-27 01:00:29] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:00 [I 2022-10-27 01:00:29] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00246 [I 2022-10-27 01:00:29] ...trap/engines/engine.py.230: loss: 0.14448 [I 2022-10-27 01:00:29] ...optimizers/trijoint.py.151: total_norm: 12.062490 [I 2022-10-27 01:00:35] ...trap/engines/engine.py.224: train: epoch 39 | batch 1020/2304 [I 2022-10-27 01:00:35] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:52 [I 2022-10-27 01:00:35] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00247 [I 2022-10-27 01:00:35] ...trap/engines/engine.py.230: loss: 0.14022 [I 2022-10-27 01:00:35] ...optimizers/trijoint.py.151: total_norm: 12.810843 [I 2022-10-27 01:00:40] ...trap/engines/engine.py.224: train: epoch 39 | batch 1030/2304 [I 2022-10-27 01:00:40] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:47 [I 2022-10-27 01:00:40] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00246 [I 2022-10-27 01:00:40] ...trap/engines/engine.py.230: loss: 0.21247 [I 2022-10-27 01:00:40] ...optimizers/trijoint.py.151: total_norm: 24.931992 [I 2022-10-27 01:00:46] ...trap/engines/engine.py.224: train: epoch 39 | batch 1040/2304 [I 2022-10-27 01:00:46] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:44 [I 2022-10-27 01:00:46] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00249 [I 2022-10-27 01:00:46] ...trap/engines/engine.py.230: loss: 0.15544 [I 2022-10-27 01:00:46] ...optimizers/trijoint.py.151: total_norm: 16.589443 [I 2022-10-27 01:00:51] ...trap/engines/engine.py.224: train: epoch 39 | batch 1050/2304 [I 2022-10-27 01:00:51] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:38 [I 2022-10-27 01:00:51] ...trap/engines/engine.py.229: process: 0.54071 | load: 0.00255 [I 2022-10-27 01:00:51] ...trap/engines/engine.py.230: loss: 0.09928 [I 2022-10-27 01:00:51] ...optimizers/trijoint.py.151: total_norm: 7.308919 [I 2022-10-27 01:00:57] ...trap/engines/engine.py.224: train: epoch 39 | batch 1060/2304 [I 2022-10-27 01:00:57] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:30 [I 2022-10-27 01:00:57] ...trap/engines/engine.py.229: process: 0.54439 | load: 0.00247 [I 2022-10-27 01:00:57] ...trap/engines/engine.py.230: loss: 0.19066 [I 2022-10-27 01:00:57] ...optimizers/trijoint.py.151: total_norm: 12.048664 [I 2022-10-27 01:01:03] ...trap/engines/engine.py.224: train: epoch 39 | batch 1070/2304 [I 2022-10-27 01:01:03] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:26 [I 2022-10-27 01:01:03] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00245 [I 2022-10-27 01:01:03] ...trap/engines/engine.py.230: loss: 0.17443 [I 2022-10-27 01:01:03] ...optimizers/trijoint.py.151: total_norm: 18.411337 [I 2022-10-27 01:01:08] ...trap/engines/engine.py.224: train: epoch 39 | batch 1080/2304 [I 2022-10-27 01:01:08] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:23 [I 2022-10-27 01:01:08] ...trap/engines/engine.py.229: process: 0.56572 | load: 0.00248 [I 2022-10-27 01:01:08] ...trap/engines/engine.py.230: loss: 0.15464 [I 2022-10-27 01:01:08] ...optimizers/trijoint.py.151: total_norm: 10.046968 [I 2022-10-27 01:01:14] ...trap/engines/engine.py.224: train: epoch 39 | batch 1090/2304 [I 2022-10-27 01:01:14] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:17 [I 2022-10-27 01:01:14] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00251 [I 2022-10-27 01:01:14] ...trap/engines/engine.py.230: loss: 0.15111 [I 2022-10-27 01:01:14] ...optimizers/trijoint.py.151: total_norm: 14.933035 [I 2022-10-27 01:01:19] ...trap/engines/engine.py.224: train: epoch 39 | batch 1100/2304 [I 2022-10-27 01:01:19] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:11 [I 2022-10-27 01:01:19] ...trap/engines/engine.py.229: process: 0.55185 | load: 0.00245 [I 2022-10-27 01:01:19] ...trap/engines/engine.py.230: loss: 0.16567 [I 2022-10-27 01:01:19] ...optimizers/trijoint.py.151: total_norm: 29.853947 [I 2022-10-27 01:01:25] ...trap/engines/engine.py.224: train: epoch 39 | batch 1110/2304 [I 2022-10-27 01:01:25] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:02 [I 2022-10-27 01:01:25] ...trap/engines/engine.py.229: process: 0.54967 | load: 0.00245 [I 2022-10-27 01:01:25] ...trap/engines/engine.py.230: loss: 0.12436 [I 2022-10-27 01:01:25] ...optimizers/trijoint.py.151: total_norm: 15.329826 [I 2022-10-27 01:01:30] ...trap/engines/engine.py.224: train: epoch 39 | batch 1120/2304 [I 2022-10-27 01:01:30] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:10:59 [I 2022-10-27 01:01:30] ...trap/engines/engine.py.229: process: 0.54784 | load: 0.00243 [I 2022-10-27 01:01:30] ...trap/engines/engine.py.230: loss: 0.15249 [I 2022-10-27 01:01:30] ...optimizers/trijoint.py.151: total_norm: 14.964885 [I 2022-10-27 01:01:36] ...trap/engines/engine.py.224: train: epoch 39 | batch 1130/2304 [I 2022-10-27 01:01:36] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:56 [I 2022-10-27 01:01:36] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00248 [I 2022-10-27 01:01:36] ...trap/engines/engine.py.230: loss: 0.23575 [I 2022-10-27 01:01:36] ...optimizers/trijoint.py.151: total_norm: 33.383179 [I 2022-10-27 01:01:42] ...trap/engines/engine.py.224: train: epoch 39 | batch 1140/2304 [I 2022-10-27 01:01:42] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:43 [I 2022-10-27 01:01:42] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00255 [I 2022-10-27 01:01:42] ...trap/engines/engine.py.230: loss: 0.10307 [I 2022-10-27 01:01:42] ...optimizers/trijoint.py.151: total_norm: 13.591725 [I 2022-10-27 01:01:47] ...trap/engines/engine.py.224: train: epoch 39 | batch 1150/2304 [I 2022-10-27 01:01:47] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:48 [I 2022-10-27 01:01:47] ...trap/engines/engine.py.229: process: 0.56783 | load: 0.00245 [I 2022-10-27 01:01:47] ...trap/engines/engine.py.230: loss: 0.15104 [I 2022-10-27 01:01:47] ...optimizers/trijoint.py.151: total_norm: 12.898218 [I 2022-10-27 01:01:53] ...trap/engines/engine.py.224: train: epoch 39 | batch 1160/2304 [I 2022-10-27 01:01:53] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:34 [I 2022-10-27 01:01:53] ...trap/engines/engine.py.229: process: 0.55105 | load: 0.00244 [I 2022-10-27 01:01:53] ...trap/engines/engine.py.230: loss: 0.18404 [I 2022-10-27 01:01:53] ...optimizers/trijoint.py.151: total_norm: 22.123354 [I 2022-10-27 01:01:58] ...trap/engines/engine.py.224: train: epoch 39 | batch 1170/2304 [I 2022-10-27 01:01:58] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:34 [I 2022-10-27 01:01:58] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00244 [I 2022-10-27 01:01:58] ...trap/engines/engine.py.230: loss: 0.14008 [I 2022-10-27 01:01:58] ...optimizers/trijoint.py.151: total_norm: 10.735907 [I 2022-10-27 01:02:04] ...trap/engines/engine.py.224: train: epoch 39 | batch 1180/2304 [I 2022-10-27 01:02:04] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:26 [I 2022-10-27 01:02:04] ...trap/engines/engine.py.229: process: 0.55624 | load: 0.00243 [I 2022-10-27 01:02:04] ...trap/engines/engine.py.230: loss: 0.25761 [I 2022-10-27 01:02:04] ...optimizers/trijoint.py.151: total_norm: 28.432167 [I 2022-10-27 01:02:10] ...trap/engines/engine.py.224: train: epoch 39 | batch 1190/2304 [I 2022-10-27 01:02:10] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:25 [I 2022-10-27 01:02:10] ...trap/engines/engine.py.229: process: 0.56507 | load: 0.00249 [I 2022-10-27 01:02:10] ...trap/engines/engine.py.230: loss: 0.12838 [I 2022-10-27 01:02:10] ...optimizers/trijoint.py.151: total_norm: 10.003495 [I 2022-10-27 01:02:15] ...trap/engines/engine.py.224: train: epoch 39 | batch 1200/2304 [I 2022-10-27 01:02:15] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:14 [I 2022-10-27 01:02:15] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00249 [I 2022-10-27 01:02:15] ...trap/engines/engine.py.230: loss: 0.11903 [I 2022-10-27 01:02:15] ...optimizers/trijoint.py.151: total_norm: 34.575375 [I 2022-10-27 01:02:21] ...trap/engines/engine.py.224: train: epoch 39 | batch 1210/2304 [I 2022-10-27 01:02:21] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:10 [I 2022-10-27 01:02:21] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00243 [I 2022-10-27 01:02:21] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-27 01:02:21] ...optimizers/trijoint.py.151: total_norm: 21.662296 [I 2022-10-27 01:02:26] ...trap/engines/engine.py.224: train: epoch 39 | batch 1220/2304 [I 2022-10-27 01:02:26] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:01 [I 2022-10-27 01:02:26] ...trap/engines/engine.py.229: process: 0.55038 | load: 0.00243 [I 2022-10-27 01:02:26] ...trap/engines/engine.py.230: loss: 0.17202 [I 2022-10-27 01:02:26] ...optimizers/trijoint.py.151: total_norm: 15.079398 [I 2022-10-27 01:02:32] ...trap/engines/engine.py.224: train: epoch 39 | batch 1230/2304 [I 2022-10-27 01:02:32] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:01 [I 2022-10-27 01:02:32] ...trap/engines/engine.py.229: process: 0.56056 | load: 0.00245 [I 2022-10-27 01:02:32] ...trap/engines/engine.py.230: loss: 0.14330 [I 2022-10-27 01:02:32] ...optimizers/trijoint.py.151: total_norm: 9.340468 [I 2022-10-27 01:02:38] ...trap/engines/engine.py.224: train: epoch 39 | batch 1240/2304 [I 2022-10-27 01:02:38] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:53 [I 2022-10-27 01:02:38] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00245 [I 2022-10-27 01:02:38] ...trap/engines/engine.py.230: loss: 0.14704 [I 2022-10-27 01:02:38] ...optimizers/trijoint.py.151: total_norm: 16.242105 [I 2022-10-27 01:02:43] ...trap/engines/engine.py.224: train: epoch 39 | batch 1250/2304 [I 2022-10-27 01:02:43] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:46 [I 2022-10-27 01:02:43] ...trap/engines/engine.py.229: process: 0.54192 | load: 0.00245 [I 2022-10-27 01:02:43] ...trap/engines/engine.py.230: loss: 0.12582 [I 2022-10-27 01:02:43] ...optimizers/trijoint.py.151: total_norm: 11.166065 [I 2022-10-27 01:02:49] ...trap/engines/engine.py.224: train: epoch 39 | batch 1260/2304 [I 2022-10-27 01:02:49] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:43 [I 2022-10-27 01:02:49] ...trap/engines/engine.py.229: process: 0.56762 | load: 0.00243 [I 2022-10-27 01:02:49] ...trap/engines/engine.py.230: loss: 0.13556 [I 2022-10-27 01:02:49] ...optimizers/trijoint.py.151: total_norm: 16.452311 [I 2022-10-27 01:02:54] ...trap/engines/engine.py.224: train: epoch 39 | batch 1270/2304 [I 2022-10-27 01:02:54] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:38 [I 2022-10-27 01:02:54] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00242 [I 2022-10-27 01:02:54] ...trap/engines/engine.py.230: loss: 0.23931 [I 2022-10-27 01:02:54] ...optimizers/trijoint.py.151: total_norm: 30.360914 [I 2022-10-27 01:03:00] ...trap/engines/engine.py.224: train: epoch 39 | batch 1280/2304 [I 2022-10-27 01:03:00] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:26 [I 2022-10-27 01:03:00] ...trap/engines/engine.py.229: process: 0.54842 | load: 0.00250 [I 2022-10-27 01:03:00] ...trap/engines/engine.py.230: loss: 0.10461 [I 2022-10-27 01:03:00] ...optimizers/trijoint.py.151: total_norm: 14.486964 [I 2022-10-27 01:03:05] ...trap/engines/engine.py.224: train: epoch 39 | batch 1290/2304 [I 2022-10-27 01:03:05] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:25 [I 2022-10-27 01:03:05] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00247 [I 2022-10-27 01:03:05] ...trap/engines/engine.py.230: loss: 0.16392 [I 2022-10-27 01:03:05] ...optimizers/trijoint.py.151: total_norm: 12.566799 [I 2022-10-27 01:03:11] ...trap/engines/engine.py.224: train: epoch 39 | batch 1300/2304 [I 2022-10-27 01:03:11] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:20 [I 2022-10-27 01:03:11] ...trap/engines/engine.py.229: process: 0.56415 | load: 0.00244 [I 2022-10-27 01:03:11] ...trap/engines/engine.py.230: loss: 0.15886 [I 2022-10-27 01:03:11] ...optimizers/trijoint.py.151: total_norm: 18.862080 [I 2022-10-27 01:03:17] ...trap/engines/engine.py.224: train: epoch 39 | batch 1310/2304 [I 2022-10-27 01:03:17] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:13 [I 2022-10-27 01:03:17] ...trap/engines/engine.py.229: process: 0.54828 | load: 0.00245 [I 2022-10-27 01:03:17] ...trap/engines/engine.py.230: loss: 0.23203 [I 2022-10-27 01:03:17] ...optimizers/trijoint.py.151: total_norm: 70.836411 [I 2022-10-27 01:03:22] ...trap/engines/engine.py.224: train: epoch 39 | batch 1320/2304 [I 2022-10-27 01:03:22] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:07 [I 2022-10-27 01:03:22] ...trap/engines/engine.py.229: process: 0.55065 | load: 0.00243 [I 2022-10-27 01:03:22] ...trap/engines/engine.py.230: loss: 0.19455 [I 2022-10-27 01:03:22] ...optimizers/trijoint.py.151: total_norm: 11.614648 [I 2022-10-27 01:03:28] ...trap/engines/engine.py.224: train: epoch 39 | batch 1330/2304 [I 2022-10-27 01:03:28] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:01 [I 2022-10-27 01:03:28] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00246 [I 2022-10-27 01:03:28] ...trap/engines/engine.py.230: loss: 0.17534 [I 2022-10-27 01:03:28] ...optimizers/trijoint.py.151: total_norm: 17.801754 [I 2022-10-27 01:03:33] ...trap/engines/engine.py.224: train: epoch 39 | batch 1340/2304 [I 2022-10-27 01:03:33] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:53 [I 2022-10-27 01:03:33] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00247 [I 2022-10-27 01:03:33] ...trap/engines/engine.py.230: loss: 0.15666 [I 2022-10-27 01:03:33] ...optimizers/trijoint.py.151: total_norm: 13.397850 [I 2022-10-27 01:03:39] ...trap/engines/engine.py.224: train: epoch 39 | batch 1350/2304 [I 2022-10-27 01:03:39] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:50 [I 2022-10-27 01:03:39] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00250 [I 2022-10-27 01:03:39] ...trap/engines/engine.py.230: loss: 0.12778 [I 2022-10-27 01:03:39] ...optimizers/trijoint.py.151: total_norm: 17.220039 [I 2022-10-27 01:03:44] ...trap/engines/engine.py.224: train: epoch 39 | batch 1360/2304 [I 2022-10-27 01:03:44] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:48 [I 2022-10-27 01:03:44] ...trap/engines/engine.py.229: process: 0.55509 | load: 0.00265 [I 2022-10-27 01:03:44] ...trap/engines/engine.py.230: loss: 0.17071 [I 2022-10-27 01:03:44] ...optimizers/trijoint.py.151: total_norm: 15.519669 [I 2022-10-27 01:03:50] ...trap/engines/engine.py.224: train: epoch 39 | batch 1370/2304 [I 2022-10-27 01:03:50] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:42 [I 2022-10-27 01:03:50] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00243 [I 2022-10-27 01:03:50] ...trap/engines/engine.py.230: loss: 0.17736 [I 2022-10-27 01:03:50] ...optimizers/trijoint.py.151: total_norm: 14.533373 [I 2022-10-27 01:03:56] ...trap/engines/engine.py.224: train: epoch 39 | batch 1380/2304 [I 2022-10-27 01:03:56] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:32 [I 2022-10-27 01:03:56] ...trap/engines/engine.py.229: process: 0.54628 | load: 0.00247 [I 2022-10-27 01:03:56] ...trap/engines/engine.py.230: loss: 0.17126 [I 2022-10-27 01:03:56] ...optimizers/trijoint.py.151: total_norm: 22.663929 [I 2022-10-27 01:04:01] ...trap/engines/engine.py.224: train: epoch 39 | batch 1390/2304 [I 2022-10-27 01:04:01] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:29 [I 2022-10-27 01:04:01] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00249 [I 2022-10-27 01:04:01] ...trap/engines/engine.py.230: loss: 0.13513 [I 2022-10-27 01:04:01] ...optimizers/trijoint.py.151: total_norm: 14.480375 [I 2022-10-27 01:04:07] ...trap/engines/engine.py.224: train: epoch 39 | batch 1400/2304 [I 2022-10-27 01:04:07] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:20 [I 2022-10-27 01:04:07] ...trap/engines/engine.py.229: process: 0.55131 | load: 0.00244 [I 2022-10-27 01:04:07] ...trap/engines/engine.py.230: loss: 0.12736 [I 2022-10-27 01:04:07] ...optimizers/trijoint.py.151: total_norm: 31.368181 [I 2022-10-27 01:04:12] ...trap/engines/engine.py.224: train: epoch 39 | batch 1410/2304 [I 2022-10-27 01:04:12] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:16 [I 2022-10-27 01:04:12] ...trap/engines/engine.py.229: process: 0.55926 | load: 0.00243 [I 2022-10-27 01:04:12] ...trap/engines/engine.py.230: loss: 0.21318 [I 2022-10-27 01:04:12] ...optimizers/trijoint.py.151: total_norm: 17.569170 [I 2022-10-27 01:04:18] ...trap/engines/engine.py.224: train: epoch 39 | batch 1420/2304 [I 2022-10-27 01:04:18] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:13 [I 2022-10-27 01:04:18] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00243 [I 2022-10-27 01:04:18] ...trap/engines/engine.py.230: loss: 0.14569 [I 2022-10-27 01:04:18] ...optimizers/trijoint.py.151: total_norm: 8.580936 [I 2022-10-27 01:04:24] ...trap/engines/engine.py.224: train: epoch 39 | batch 1430/2304 [I 2022-10-27 01:04:24] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:08 [I 2022-10-27 01:04:24] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00259 [I 2022-10-27 01:04:24] ...trap/engines/engine.py.230: loss: 0.16900 [I 2022-10-27 01:04:24] ...optimizers/trijoint.py.151: total_norm: 31.544466 [I 2022-10-27 01:04:29] ...trap/engines/engine.py.224: train: epoch 39 | batch 1440/2304 [I 2022-10-27 01:04:29] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:00 [I 2022-10-27 01:04:29] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00249 [I 2022-10-27 01:04:29] ...trap/engines/engine.py.230: loss: 0.17292 [I 2022-10-27 01:04:29] ...optimizers/trijoint.py.151: total_norm: 23.087606 [I 2022-10-27 01:04:35] ...trap/engines/engine.py.224: train: epoch 39 | batch 1450/2304 [I 2022-10-27 01:04:35] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:54 [I 2022-10-27 01:04:35] ...trap/engines/engine.py.229: process: 0.54859 | load: 0.00302 [I 2022-10-27 01:04:35] ...trap/engines/engine.py.230: loss: 0.15103 [I 2022-10-27 01:04:35] ...optimizers/trijoint.py.151: total_norm: 29.550055 [I 2022-10-27 01:04:40] ...trap/engines/engine.py.224: train: epoch 39 | batch 1460/2304 [I 2022-10-27 01:04:40] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:46 [I 2022-10-27 01:04:40] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00265 [I 2022-10-27 01:04:40] ...trap/engines/engine.py.230: loss: 0.11958 [I 2022-10-27 01:04:40] ...optimizers/trijoint.py.151: total_norm: 9.869635 [I 2022-10-27 01:04:46] ...trap/engines/engine.py.224: train: epoch 39 | batch 1470/2304 [I 2022-10-27 01:04:46] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:42 [I 2022-10-27 01:04:46] ...trap/engines/engine.py.229: process: 0.55298 | load: 0.00243 [I 2022-10-27 01:04:46] ...trap/engines/engine.py.230: loss: 0.16642 [I 2022-10-27 01:04:46] ...optimizers/trijoint.py.151: total_norm: 22.231073 [I 2022-10-27 01:04:51] ...trap/engines/engine.py.224: train: epoch 39 | batch 1480/2304 [I 2022-10-27 01:04:51] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:37 [I 2022-10-27 01:04:51] ...trap/engines/engine.py.229: process: 0.56348 | load: 0.00247 [I 2022-10-27 01:04:51] ...trap/engines/engine.py.230: loss: 0.12946 [I 2022-10-27 01:04:51] ...optimizers/trijoint.py.151: total_norm: 10.363946 [I 2022-10-27 01:04:57] ...trap/engines/engine.py.224: train: epoch 39 | batch 1490/2304 [I 2022-10-27 01:04:57] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:34 [I 2022-10-27 01:04:57] ...trap/engines/engine.py.229: process: 0.56416 | load: 0.00247 [I 2022-10-27 01:04:57] ...trap/engines/engine.py.230: loss: 0.16530 [I 2022-10-27 01:04:57] ...optimizers/trijoint.py.151: total_norm: 13.849879 [I 2022-10-27 01:05:02] ...trap/engines/engine.py.224: train: epoch 39 | batch 1500/2304 [I 2022-10-27 01:05:02] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:27 [I 2022-10-27 01:05:02] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00246 [I 2022-10-27 01:05:02] ...trap/engines/engine.py.230: loss: 0.15345 [I 2022-10-27 01:05:02] ...optimizers/trijoint.py.151: total_norm: 20.262035 [I 2022-10-27 01:05:08] ...trap/engines/engine.py.224: train: epoch 39 | batch 1510/2304 [I 2022-10-27 01:05:08] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:23 [I 2022-10-27 01:05:08] ...trap/engines/engine.py.229: process: 0.56299 | load: 0.00247 [I 2022-10-27 01:05:08] ...trap/engines/engine.py.230: loss: 0.22029 [I 2022-10-27 01:05:08] ...optimizers/trijoint.py.151: total_norm: 14.166153 [I 2022-10-27 01:05:14] ...trap/engines/engine.py.224: train: epoch 39 | batch 1520/2304 [I 2022-10-27 01:05:14] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:15 [I 2022-10-27 01:05:14] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00247 [I 2022-10-27 01:05:14] ...trap/engines/engine.py.230: loss: 0.16374 [I 2022-10-27 01:05:14] ...optimizers/trijoint.py.151: total_norm: 19.234203 [I 2022-10-27 01:05:19] ...trap/engines/engine.py.224: train: epoch 39 | batch 1530/2304 [I 2022-10-27 01:05:19] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:11 [I 2022-10-27 01:05:19] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00248 [I 2022-10-27 01:05:19] ...trap/engines/engine.py.230: loss: 0.15827 [I 2022-10-27 01:05:19] ...optimizers/trijoint.py.151: total_norm: 15.116952 [I 2022-10-27 01:05:25] ...trap/engines/engine.py.224: train: epoch 39 | batch 1540/2304 [I 2022-10-27 01:05:25] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:09 [I 2022-10-27 01:05:25] ...trap/engines/engine.py.229: process: 0.57635 | load: 0.00539 [I 2022-10-27 01:05:25] ...trap/engines/engine.py.230: loss: 0.18921 [I 2022-10-27 01:05:25] ...optimizers/trijoint.py.151: total_norm: 21.253897 [I 2022-10-27 01:05:31] ...trap/engines/engine.py.224: train: epoch 39 | batch 1550/2304 [I 2022-10-27 01:05:31] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:03 [I 2022-10-27 01:05:31] ...trap/engines/engine.py.229: process: 0.57230 | load: 0.00250 [I 2022-10-27 01:05:31] ...trap/engines/engine.py.230: loss: 0.20215 [I 2022-10-27 01:05:31] ...optimizers/trijoint.py.151: total_norm: 22.855795 [I 2022-10-27 01:05:36] ...trap/engines/engine.py.224: train: epoch 39 | batch 1560/2304 [I 2022-10-27 01:05:36] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:55 [I 2022-10-27 01:05:36] ...trap/engines/engine.py.229: process: 0.56282 | load: 0.00245 [I 2022-10-27 01:05:36] ...trap/engines/engine.py.230: loss: 0.16237 [I 2022-10-27 01:05:36] ...optimizers/trijoint.py.151: total_norm: 16.586563 [I 2022-10-27 01:05:42] ...trap/engines/engine.py.224: train: epoch 39 | batch 1570/2304 [I 2022-10-27 01:05:42] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:51 [I 2022-10-27 01:05:42] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00243 [I 2022-10-27 01:05:42] ...trap/engines/engine.py.230: loss: 0.12543 [I 2022-10-27 01:05:42] ...optimizers/trijoint.py.151: total_norm: 11.829862 [I 2022-10-27 01:05:47] ...trap/engines/engine.py.224: train: epoch 39 | batch 1580/2304 [I 2022-10-27 01:05:47] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:42 [I 2022-10-27 01:05:47] ...trap/engines/engine.py.229: process: 0.55688 | load: 0.00263 [I 2022-10-27 01:05:47] ...trap/engines/engine.py.230: loss: 0.21356 [I 2022-10-27 01:05:47] ...optimizers/trijoint.py.151: total_norm: 23.360289 [I 2022-10-27 01:05:53] ...trap/engines/engine.py.224: train: epoch 39 | batch 1590/2304 [I 2022-10-27 01:05:53] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:38 [I 2022-10-27 01:05:53] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00244 [I 2022-10-27 01:05:53] ...trap/engines/engine.py.230: loss: 0.18043 [I 2022-10-27 01:05:53] ...optimizers/trijoint.py.151: total_norm: 26.795446 [I 2022-10-27 01:05:58] ...trap/engines/engine.py.224: train: epoch 39 | batch 1600/2304 [I 2022-10-27 01:05:58] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:29 [I 2022-10-27 01:05:58] ...trap/engines/engine.py.229: process: 0.54557 | load: 0.00260 [I 2022-10-27 01:05:58] ...trap/engines/engine.py.230: loss: 0.15089 [I 2022-10-27 01:05:58] ...optimizers/trijoint.py.151: total_norm: 78.196381 [I 2022-10-27 01:06:04] ...trap/engines/engine.py.224: train: epoch 39 | batch 1610/2304 [I 2022-10-27 01:06:04] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:24 [I 2022-10-27 01:06:04] ...trap/engines/engine.py.229: process: 0.54466 | load: 0.00248 [I 2022-10-27 01:06:04] ...trap/engines/engine.py.230: loss: 0.17101 [I 2022-10-27 01:06:04] ...optimizers/trijoint.py.151: total_norm: 16.223488 [I 2022-10-27 01:06:10] ...trap/engines/engine.py.224: train: epoch 39 | batch 1620/2304 [I 2022-10-27 01:06:10] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:20 [I 2022-10-27 01:06:10] ...trap/engines/engine.py.229: process: 0.55150 | load: 0.00247 [I 2022-10-27 01:06:10] ...trap/engines/engine.py.230: loss: 0.14381 [I 2022-10-27 01:06:10] ...optimizers/trijoint.py.151: total_norm: 15.210781 [I 2022-10-27 01:06:15] ...trap/engines/engine.py.224: train: epoch 39 | batch 1630/2304 [I 2022-10-27 01:06:15] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:31 [I 2022-10-27 01:06:15] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00243 [I 2022-10-27 01:06:15] ...trap/engines/engine.py.230: loss: 0.18785 [I 2022-10-27 01:06:15] ...optimizers/trijoint.py.151: total_norm: 26.471083 [I 2022-10-27 01:06:21] ...trap/engines/engine.py.224: train: epoch 39 | batch 1640/2304 [I 2022-10-27 01:06:21] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:08 [I 2022-10-27 01:06:21] ...trap/engines/engine.py.229: process: 0.54601 | load: 0.00247 [I 2022-10-27 01:06:21] ...trap/engines/engine.py.230: loss: 0.15788 [I 2022-10-27 01:06:21] ...optimizers/trijoint.py.151: total_norm: 9.578330 [I 2022-10-27 01:06:26] ...trap/engines/engine.py.224: train: epoch 39 | batch 1650/2304 [I 2022-10-27 01:06:26] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:03 [I 2022-10-27 01:06:26] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00243 [I 2022-10-27 01:06:26] ...trap/engines/engine.py.230: loss: 0.10903 [I 2022-10-27 01:06:26] ...optimizers/trijoint.py.151: total_norm: 19.394119 [I 2022-10-27 01:06:32] ...trap/engines/engine.py.224: train: epoch 39 | batch 1660/2304 [I 2022-10-27 01:06:32] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:58 [I 2022-10-27 01:06:32] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00242 [I 2022-10-27 01:06:32] ...trap/engines/engine.py.230: loss: 0.18564 [I 2022-10-27 01:06:32] ...optimizers/trijoint.py.151: total_norm: 26.740898 [I 2022-10-27 01:06:38] ...trap/engines/engine.py.224: train: epoch 39 | batch 1670/2304 [I 2022-10-27 01:06:38] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:51 [I 2022-10-27 01:06:38] ...trap/engines/engine.py.229: process: 0.55936 | load: 0.00243 [I 2022-10-27 01:06:38] ...trap/engines/engine.py.230: loss: 0.20551 [I 2022-10-27 01:06:38] ...optimizers/trijoint.py.151: total_norm: 32.868546 [I 2022-10-27 01:06:43] ...trap/engines/engine.py.224: train: epoch 39 | batch 1680/2304 [I 2022-10-27 01:06:43] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:48 [I 2022-10-27 01:06:43] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00248 [I 2022-10-27 01:06:43] ...trap/engines/engine.py.230: loss: 0.15491 [I 2022-10-27 01:06:43] ...optimizers/trijoint.py.151: total_norm: 23.521198 [I 2022-10-27 01:06:49] ...trap/engines/engine.py.224: train: epoch 39 | batch 1690/2304 [I 2022-10-27 01:06:49] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:44 [I 2022-10-27 01:06:49] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00248 [I 2022-10-27 01:06:49] ...trap/engines/engine.py.230: loss: 0.17419 [I 2022-10-27 01:06:49] ...optimizers/trijoint.py.151: total_norm: 12.409233 [I 2022-10-27 01:06:55] ...trap/engines/engine.py.224: train: epoch 39 | batch 1700/2304 [I 2022-10-27 01:06:55] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:47 [I 2022-10-27 01:06:55] ...trap/engines/engine.py.229: process: 0.55924 | load: 0.00253 [I 2022-10-27 01:06:55] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-27 01:06:55] ...optimizers/trijoint.py.151: total_norm: 9.726923 [I 2022-10-27 01:07:00] ...trap/engines/engine.py.224: train: epoch 39 | batch 1710/2304 [I 2022-10-27 01:07:00] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:30 [I 2022-10-27 01:07:00] ...trap/engines/engine.py.229: process: 0.54421 | load: 0.00247 [I 2022-10-27 01:07:00] ...trap/engines/engine.py.230: loss: 0.17172 [I 2022-10-27 01:07:00] ...optimizers/trijoint.py.151: total_norm: 15.440467 [I 2022-10-27 01:07:06] ...trap/engines/engine.py.224: train: epoch 39 | batch 1720/2304 [I 2022-10-27 01:07:06] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:25 [I 2022-10-27 01:07:06] ...trap/engines/engine.py.229: process: 0.54820 | load: 0.00243 [I 2022-10-27 01:07:06] ...trap/engines/engine.py.230: loss: 0.16533 [I 2022-10-27 01:07:06] ...optimizers/trijoint.py.151: total_norm: 66.281113 [I 2022-10-27 01:07:11] ...trap/engines/engine.py.224: train: epoch 39 | batch 1730/2304 [I 2022-10-27 01:07:11] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:17 [I 2022-10-27 01:07:11] ...trap/engines/engine.py.229: process: 0.55450 | load: 0.00259 [I 2022-10-27 01:07:11] ...trap/engines/engine.py.230: loss: 0.19398 [I 2022-10-27 01:07:11] ...optimizers/trijoint.py.151: total_norm: 9.933645 [I 2022-10-27 01:07:17] ...trap/engines/engine.py.224: train: epoch 39 | batch 1740/2304 [I 2022-10-27 01:07:17] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:14 [I 2022-10-27 01:07:17] ...trap/engines/engine.py.229: process: 0.55094 | load: 0.00259 [I 2022-10-27 01:07:17] ...trap/engines/engine.py.230: loss: 0.18058 [I 2022-10-27 01:07:17] ...optimizers/trijoint.py.151: total_norm: 16.467098 [I 2022-10-27 01:07:22] ...trap/engines/engine.py.224: train: epoch 39 | batch 1750/2304 [I 2022-10-27 01:07:22] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:10 [I 2022-10-27 01:07:22] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00244 [I 2022-10-27 01:07:22] ...trap/engines/engine.py.230: loss: 0.17907 [I 2022-10-27 01:07:22] ...optimizers/trijoint.py.151: total_norm: 24.525127 [I 2022-10-27 01:07:28] ...trap/engines/engine.py.224: train: epoch 39 | batch 1760/2304 [I 2022-10-27 01:07:28] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:04 [I 2022-10-27 01:07:28] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00245 [I 2022-10-27 01:07:28] ...trap/engines/engine.py.230: loss: 0.21064 [I 2022-10-27 01:07:28] ...optimizers/trijoint.py.151: total_norm: 28.413713 [I 2022-10-27 01:07:34] ...trap/engines/engine.py.224: train: epoch 39 | batch 1770/2304 [I 2022-10-27 01:07:34] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:04:53 [I 2022-10-27 01:07:34] ...trap/engines/engine.py.229: process: 0.54290 | load: 0.00243 [I 2022-10-27 01:07:34] ...trap/engines/engine.py.230: loss: 0.15161 [I 2022-10-27 01:07:34] ...optimizers/trijoint.py.151: total_norm: 14.211951 [I 2022-10-27 01:07:39] ...trap/engines/engine.py.224: train: epoch 39 | batch 1780/2304 [I 2022-10-27 01:07:39] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:51 [I 2022-10-27 01:07:39] ...trap/engines/engine.py.229: process: 0.55005 | load: 0.00242 [I 2022-10-27 01:07:39] ...trap/engines/engine.py.230: loss: 0.19066 [I 2022-10-27 01:07:39] ...optimizers/trijoint.py.151: total_norm: 9.814835 [I 2022-10-27 01:07:45] ...trap/engines/engine.py.224: train: epoch 39 | batch 1790/2304 [I 2022-10-27 01:07:45] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:43 [I 2022-10-27 01:07:45] ...trap/engines/engine.py.229: process: 0.55034 | load: 0.00241 [I 2022-10-27 01:07:45] ...trap/engines/engine.py.230: loss: 0.23639 [I 2022-10-27 01:07:45] ...optimizers/trijoint.py.151: total_norm: 18.878590 [I 2022-10-27 01:07:50] ...trap/engines/engine.py.224: train: epoch 39 | batch 1800/2304 [I 2022-10-27 01:07:50] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:40 [I 2022-10-27 01:07:50] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00253 [I 2022-10-27 01:07:50] ...trap/engines/engine.py.230: loss: 0.16813 [I 2022-10-27 01:07:50] ...optimizers/trijoint.py.151: total_norm: 13.505497 [I 2022-10-27 01:07:56] ...trap/engines/engine.py.224: train: epoch 39 | batch 1810/2304 [I 2022-10-27 01:07:56] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:37 [I 2022-10-27 01:07:56] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00249 [I 2022-10-27 01:07:56] ...trap/engines/engine.py.230: loss: 0.17183 [I 2022-10-27 01:07:56] ...optimizers/trijoint.py.151: total_norm: 14.438131 [I 2022-10-27 01:08:02] ...trap/engines/engine.py.224: train: epoch 39 | batch 1820/2304 [I 2022-10-27 01:08:02] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:32 [I 2022-10-27 01:08:02] ...trap/engines/engine.py.229: process: 0.56989 | load: 0.00244 [I 2022-10-27 01:08:02] ...trap/engines/engine.py.230: loss: 0.14721 [I 2022-10-27 01:08:02] ...optimizers/trijoint.py.151: total_norm: 9.984577 [I 2022-10-27 01:08:07] ...trap/engines/engine.py.224: train: epoch 39 | batch 1830/2304 [I 2022-10-27 01:08:07] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:24 [I 2022-10-27 01:08:07] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00244 [I 2022-10-27 01:08:07] ...trap/engines/engine.py.230: loss: 0.25841 [I 2022-10-27 01:08:07] ...optimizers/trijoint.py.151: total_norm: 28.537521 [I 2022-10-27 01:08:13] ...trap/engines/engine.py.224: train: epoch 39 | batch 1840/2304 [I 2022-10-27 01:08:13] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:18 [I 2022-10-27 01:08:13] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00243 [I 2022-10-27 01:08:13] ...trap/engines/engine.py.230: loss: 0.25808 [I 2022-10-27 01:08:13] ...optimizers/trijoint.py.151: total_norm: 19.755447 [I 2022-10-27 01:08:18] ...trap/engines/engine.py.224: train: epoch 39 | batch 1850/2304 [I 2022-10-27 01:08:18] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:12 [I 2022-10-27 01:08:18] ...trap/engines/engine.py.229: process: 0.54514 | load: 0.00247 [I 2022-10-27 01:08:18] ...trap/engines/engine.py.230: loss: 0.21325 [I 2022-10-27 01:08:18] ...optimizers/trijoint.py.151: total_norm: 20.338846 [I 2022-10-27 01:08:24] ...trap/engines/engine.py.224: train: epoch 39 | batch 1860/2304 [I 2022-10-27 01:08:24] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:07 [I 2022-10-27 01:08:24] ...trap/engines/engine.py.229: process: 0.55174 | load: 0.00246 [I 2022-10-27 01:08:24] ...trap/engines/engine.py.230: loss: 0.17722 [I 2022-10-27 01:08:24] ...optimizers/trijoint.py.151: total_norm: 14.914718 [I 2022-10-27 01:08:29] ...trap/engines/engine.py.224: train: epoch 39 | batch 1870/2304 [I 2022-10-27 01:08:29] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:02 [I 2022-10-27 01:08:29] ...trap/engines/engine.py.229: process: 0.57289 | load: 0.00242 [I 2022-10-27 01:08:29] ...trap/engines/engine.py.230: loss: 0.20177 [I 2022-10-27 01:08:29] ...optimizers/trijoint.py.151: total_norm: 13.895332 [I 2022-10-27 01:08:35] ...trap/engines/engine.py.224: train: epoch 39 | batch 1880/2304 [I 2022-10-27 01:08:35] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:54 [I 2022-10-27 01:08:35] ...trap/engines/engine.py.229: process: 0.55437 | load: 0.00247 [I 2022-10-27 01:08:35] ...trap/engines/engine.py.230: loss: 0.19407 [I 2022-10-27 01:08:35] ...optimizers/trijoint.py.151: total_norm: 20.586061 [I 2022-10-27 01:08:41] ...trap/engines/engine.py.224: train: epoch 39 | batch 1890/2304 [I 2022-10-27 01:08:41] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:03:51 [I 2022-10-27 01:08:41] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00242 [I 2022-10-27 01:08:41] ...trap/engines/engine.py.230: loss: 0.14782 [I 2022-10-27 01:08:41] ...optimizers/trijoint.py.151: total_norm: 17.714239 [I 2022-10-27 01:08:46] ...trap/engines/engine.py.224: train: epoch 39 | batch 1900/2304 [I 2022-10-27 01:08:46] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:44 [I 2022-10-27 01:08:46] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00245 [I 2022-10-27 01:08:46] ...trap/engines/engine.py.230: loss: 0.14720 [I 2022-10-27 01:08:46] ...optimizers/trijoint.py.151: total_norm: 18.173357 [I 2022-10-27 01:08:52] ...trap/engines/engine.py.224: train: epoch 39 | batch 1910/2304 [I 2022-10-27 01:08:52] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:42 [I 2022-10-27 01:08:52] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00242 [I 2022-10-27 01:08:52] ...trap/engines/engine.py.230: loss: 0.18365 [I 2022-10-27 01:08:52] ...optimizers/trijoint.py.151: total_norm: 22.216091 [I 2022-10-27 01:08:58] ...trap/engines/engine.py.224: train: epoch 39 | batch 1920/2304 [I 2022-10-27 01:08:58] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:34 [I 2022-10-27 01:08:58] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00245 [I 2022-10-27 01:08:58] ...trap/engines/engine.py.230: loss: 0.28886 [I 2022-10-27 01:08:58] ...optimizers/trijoint.py.151: total_norm: 30.495712 [I 2022-10-27 01:09:03] ...trap/engines/engine.py.224: train: epoch 39 | batch 1930/2304 [I 2022-10-27 01:09:03] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:27 [I 2022-10-27 01:09:03] ...trap/engines/engine.py.229: process: 0.56086 | load: 0.00250 [I 2022-10-27 01:09:03] ...trap/engines/engine.py.230: loss: 0.12752 [I 2022-10-27 01:09:03] ...optimizers/trijoint.py.151: total_norm: 22.377007 [I 2022-10-27 01:09:09] ...trap/engines/engine.py.224: train: epoch 39 | batch 1940/2304 [I 2022-10-27 01:09:09] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:21 [I 2022-10-27 01:09:09] ...trap/engines/engine.py.229: process: 0.54428 | load: 0.00242 [I 2022-10-27 01:09:09] ...trap/engines/engine.py.230: loss: 0.15827 [I 2022-10-27 01:09:09] ...optimizers/trijoint.py.151: total_norm: 18.617945 [I 2022-10-27 01:09:14] ...trap/engines/engine.py.224: train: epoch 39 | batch 1950/2304 [I 2022-10-27 01:09:14] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:17 [I 2022-10-27 01:09:14] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00255 [I 2022-10-27 01:09:14] ...trap/engines/engine.py.230: loss: 0.19941 [I 2022-10-27 01:09:14] ...optimizers/trijoint.py.151: total_norm: 15.893400 [I 2022-10-27 01:09:20] ...trap/engines/engine.py.224: train: epoch 39 | batch 1960/2304 [I 2022-10-27 01:09:20] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:11 [I 2022-10-27 01:09:20] ...trap/engines/engine.py.229: process: 0.55758 | load: 0.00250 [I 2022-10-27 01:09:20] ...trap/engines/engine.py.230: loss: 0.16454 [I 2022-10-27 01:09:20] ...optimizers/trijoint.py.151: total_norm: 18.081585 [I 2022-10-27 01:09:25] ...trap/engines/engine.py.224: train: epoch 39 | batch 1970/2304 [I 2022-10-27 01:09:25] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:05 [I 2022-10-27 01:09:25] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00246 [I 2022-10-27 01:09:25] ...trap/engines/engine.py.230: loss: 0.14332 [I 2022-10-27 01:09:25] ...optimizers/trijoint.py.151: total_norm: 18.275841 [I 2022-10-27 01:09:31] ...trap/engines/engine.py.224: train: epoch 39 | batch 1980/2304 [I 2022-10-27 01:09:31] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:00 [I 2022-10-27 01:09:31] ...trap/engines/engine.py.229: process: 0.55341 | load: 0.00251 [I 2022-10-27 01:09:31] ...trap/engines/engine.py.230: loss: 0.15955 [I 2022-10-27 01:09:31] ...optimizers/trijoint.py.151: total_norm: 16.755489 [I 2022-10-27 01:09:37] ...trap/engines/engine.py.224: train: epoch 39 | batch 1990/2304 [I 2022-10-27 01:09:37] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:55 [I 2022-10-27 01:09:37] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00260 [I 2022-10-27 01:09:37] ...trap/engines/engine.py.230: loss: 0.15563 [I 2022-10-27 01:09:37] ...optimizers/trijoint.py.151: total_norm: 20.192507 [I 2022-10-27 01:09:42] ...trap/engines/engine.py.224: train: epoch 39 | batch 2000/2304 [I 2022-10-27 01:09:42] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:02:49 [I 2022-10-27 01:09:42] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00250 [I 2022-10-27 01:09:42] ...trap/engines/engine.py.230: loss: 0.14482 [I 2022-10-27 01:09:42] ...optimizers/trijoint.py.151: total_norm: 19.769979 [I 2022-10-27 01:09:48] ...trap/engines/engine.py.224: train: epoch 39 | batch 2010/2304 [I 2022-10-27 01:09:48] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:42 [I 2022-10-27 01:09:48] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00245 [I 2022-10-27 01:09:48] ...trap/engines/engine.py.230: loss: 0.19439 [I 2022-10-27 01:09:48] ...optimizers/trijoint.py.151: total_norm: 16.033310 [I 2022-10-27 01:09:53] ...trap/engines/engine.py.224: train: epoch 39 | batch 2020/2304 [I 2022-10-27 01:09:53] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:39 [I 2022-10-27 01:09:53] ...trap/engines/engine.py.229: process: 0.57115 | load: 0.00246 [I 2022-10-27 01:09:53] ...trap/engines/engine.py.230: loss: 0.20303 [I 2022-10-27 01:09:53] ...optimizers/trijoint.py.151: total_norm: 21.070044 [I 2022-10-27 01:09:59] ...trap/engines/engine.py.224: train: epoch 39 | batch 2030/2304 [I 2022-10-27 01:09:59] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:34 [I 2022-10-27 01:09:59] ...trap/engines/engine.py.229: process: 0.55176 | load: 0.00250 [I 2022-10-27 01:09:59] ...trap/engines/engine.py.230: loss: 0.21079 [I 2022-10-27 01:09:59] ...optimizers/trijoint.py.151: total_norm: 16.159182 [I 2022-10-27 01:10:05] ...trap/engines/engine.py.224: train: epoch 39 | batch 2040/2304 [I 2022-10-27 01:10:05] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:27 [I 2022-10-27 01:10:05] ...trap/engines/engine.py.229: process: 0.56340 | load: 0.00249 [I 2022-10-27 01:10:05] ...trap/engines/engine.py.230: loss: 0.15594 [I 2022-10-27 01:10:05] ...optimizers/trijoint.py.151: total_norm: 16.073973 [I 2022-10-27 01:10:10] ...trap/engines/engine.py.224: train: epoch 39 | batch 2050/2304 [I 2022-10-27 01:10:10] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:20 [I 2022-10-27 01:10:10] ...trap/engines/engine.py.229: process: 0.54713 | load: 0.00249 [I 2022-10-27 01:10:10] ...trap/engines/engine.py.230: loss: 0.18435 [I 2022-10-27 01:10:10] ...optimizers/trijoint.py.151: total_norm: 13.582131 [I 2022-10-27 01:10:16] ...trap/engines/engine.py.224: train: epoch 39 | batch 2060/2304 [I 2022-10-27 01:10:16] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:18 [I 2022-10-27 01:10:16] ...trap/engines/engine.py.229: process: 0.56737 | load: 0.00246 [I 2022-10-27 01:10:16] ...trap/engines/engine.py.230: loss: 0.11664 [I 2022-10-27 01:10:16] ...optimizers/trijoint.py.151: total_norm: 16.009254 [I 2022-10-27 01:10:22] ...trap/engines/engine.py.224: train: epoch 39 | batch 2070/2304 [I 2022-10-27 01:10:22] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:12 [I 2022-10-27 01:10:22] ...trap/engines/engine.py.229: process: 0.54193 | load: 0.00244 [I 2022-10-27 01:10:22] ...trap/engines/engine.py.230: loss: 0.14441 [I 2022-10-27 01:10:22] ...optimizers/trijoint.py.151: total_norm: 14.484775 [I 2022-10-27 01:10:27] ...trap/engines/engine.py.224: train: epoch 39 | batch 2080/2304 [I 2022-10-27 01:10:27] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:07 [I 2022-10-27 01:10:27] ...trap/engines/engine.py.229: process: 0.57119 | load: 0.00247 [I 2022-10-27 01:10:27] ...trap/engines/engine.py.230: loss: 0.18442 [I 2022-10-27 01:10:27] ...optimizers/trijoint.py.151: total_norm: 11.248058 [I 2022-10-27 01:10:33] ...trap/engines/engine.py.224: train: epoch 39 | batch 2090/2304 [I 2022-10-27 01:10:33] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:02:02 [I 2022-10-27 01:10:33] ...trap/engines/engine.py.229: process: 0.56492 | load: 0.00242 [I 2022-10-27 01:10:33] ...trap/engines/engine.py.230: loss: 0.15562 [I 2022-10-27 01:10:33] ...optimizers/trijoint.py.151: total_norm: 18.388107 [I 2022-10-27 01:10:39] ...trap/engines/engine.py.224: train: epoch 39 | batch 2100/2304 [I 2022-10-27 01:10:39] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:56 [I 2022-10-27 01:10:39] ...trap/engines/engine.py.229: process: 0.58780 | load: 0.00245 [I 2022-10-27 01:10:39] ...trap/engines/engine.py.230: loss: 0.14728 [I 2022-10-27 01:10:39] ...optimizers/trijoint.py.151: total_norm: 13.147845 [I 2022-10-27 01:10:44] ...trap/engines/engine.py.224: train: epoch 39 | batch 2110/2304 [I 2022-10-27 01:10:44] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:01:50 [I 2022-10-27 01:10:44] ...trap/engines/engine.py.229: process: 0.56215 | load: 0.00262 [I 2022-10-27 01:10:44] ...trap/engines/engine.py.230: loss: 0.26607 [I 2022-10-27 01:10:44] ...optimizers/trijoint.py.151: total_norm: 31.196478 [I 2022-10-27 01:10:50] ...trap/engines/engine.py.224: train: epoch 39 | batch 2120/2304 [I 2022-10-27 01:10:50] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:44 [I 2022-10-27 01:10:50] ...trap/engines/engine.py.229: process: 0.57420 | load: 0.00258 [I 2022-10-27 01:10:50] ...trap/engines/engine.py.230: loss: 0.17940 [I 2022-10-27 01:10:50] ...optimizers/trijoint.py.151: total_norm: 15.075290 [I 2022-10-27 01:10:56] ...trap/engines/engine.py.224: train: epoch 39 | batch 2130/2304 [I 2022-10-27 01:10:56] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:38 [I 2022-10-27 01:10:56] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00258 [I 2022-10-27 01:10:56] ...trap/engines/engine.py.230: loss: 0.17777 [I 2022-10-27 01:10:56] ...optimizers/trijoint.py.151: total_norm: 25.829924 [I 2022-10-27 01:11:02] ...trap/engines/engine.py.224: train: epoch 39 | batch 2140/2304 [I 2022-10-27 01:11:02] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:32 [I 2022-10-27 01:11:02] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00263 [I 2022-10-27 01:11:02] ...trap/engines/engine.py.230: loss: 0.12610 [I 2022-10-27 01:11:02] ...optimizers/trijoint.py.151: total_norm: 12.811975 [I 2022-10-27 01:11:07] ...trap/engines/engine.py.224: train: epoch 39 | batch 2150/2304 [I 2022-10-27 01:11:07] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:28 [I 2022-10-27 01:11:07] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00254 [I 2022-10-27 01:11:07] ...trap/engines/engine.py.230: loss: 0.20395 [I 2022-10-27 01:11:07] ...optimizers/trijoint.py.151: total_norm: 22.617231 [I 2022-10-27 01:11:13] ...trap/engines/engine.py.224: train: epoch 39 | batch 2160/2304 [I 2022-10-27 01:11:13] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:20 [I 2022-10-27 01:11:13] ...trap/engines/engine.py.229: process: 0.55081 | load: 0.00246 [I 2022-10-27 01:11:13] ...trap/engines/engine.py.230: loss: 0.18480 [I 2022-10-27 01:11:13] ...optimizers/trijoint.py.151: total_norm: 61.176952 [I 2022-10-27 01:11:19] ...trap/engines/engine.py.224: train: epoch 39 | batch 2170/2304 [I 2022-10-27 01:11:19] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:14 [I 2022-10-27 01:11:19] ...trap/engines/engine.py.229: process: 0.55337 | load: 0.00253 [I 2022-10-27 01:11:19] ...trap/engines/engine.py.230: loss: 0.14943 [I 2022-10-27 01:11:19] ...optimizers/trijoint.py.151: total_norm: 7.846202 [I 2022-10-27 01:11:24] ...trap/engines/engine.py.224: train: epoch 39 | batch 2180/2304 [I 2022-10-27 01:11:24] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:09 [I 2022-10-27 01:11:24] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00252 [I 2022-10-27 01:11:24] ...trap/engines/engine.py.230: loss: 0.14982 [I 2022-10-27 01:11:24] ...optimizers/trijoint.py.151: total_norm: 12.210979 [I 2022-10-27 01:11:30] ...trap/engines/engine.py.224: train: epoch 39 | batch 2190/2304 [I 2022-10-27 01:11:30] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:03 [I 2022-10-27 01:11:30] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00275 [I 2022-10-27 01:11:30] ...trap/engines/engine.py.230: loss: 0.13473 [I 2022-10-27 01:11:30] ...optimizers/trijoint.py.151: total_norm: 11.818476 [I 2022-10-27 01:11:35] ...trap/engines/engine.py.224: train: epoch 39 | batch 2200/2304 [I 2022-10-27 01:11:35] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:00:58 [I 2022-10-27 01:11:35] ...trap/engines/engine.py.229: process: 0.56640 | load: 0.00256 [I 2022-10-27 01:11:35] ...trap/engines/engine.py.230: loss: 0.18703 [I 2022-10-27 01:11:35] ...optimizers/trijoint.py.151: total_norm: 24.103220 [I 2022-10-27 01:11:41] ...trap/engines/engine.py.224: train: epoch 39 | batch 2210/2304 [I 2022-10-27 01:11:41] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:00:52 [I 2022-10-27 01:11:41] ...trap/engines/engine.py.229: process: 0.54674 | load: 0.00258 [I 2022-10-27 01:11:41] ...trap/engines/engine.py.230: loss: 0.24630 [I 2022-10-27 01:11:41] ...optimizers/trijoint.py.151: total_norm: 15.553031 [I 2022-10-27 01:11:46] ...trap/engines/engine.py.224: train: epoch 39 | batch 2220/2304 [I 2022-10-27 01:11:46] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:00:46 [I 2022-10-27 01:11:46] ...trap/engines/engine.py.229: process: 0.56428 | load: 0.00250 [I 2022-10-27 01:11:46] ...trap/engines/engine.py.230: loss: 0.14735 [I 2022-10-27 01:11:46] ...optimizers/trijoint.py.151: total_norm: 11.423699 [I 2022-10-27 01:11:52] ...trap/engines/engine.py.224: train: epoch 39 | batch 2230/2304 [I 2022-10-27 01:11:52] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:41 [I 2022-10-27 01:11:52] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00270 [I 2022-10-27 01:11:52] ...trap/engines/engine.py.230: loss: 0.22511 [I 2022-10-27 01:11:52] ...optimizers/trijoint.py.151: total_norm: 36.881474 [I 2022-10-27 01:11:58] ...trap/engines/engine.py.224: train: epoch 39 | batch 2240/2304 [I 2022-10-27 01:11:58] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:35 [I 2022-10-27 01:11:58] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00300 [I 2022-10-27 01:11:58] ...trap/engines/engine.py.230: loss: 0.17289 [I 2022-10-27 01:11:58] ...optimizers/trijoint.py.151: total_norm: 32.817543 [I 2022-10-27 01:12:03] ...trap/engines/engine.py.224: train: epoch 39 | batch 2250/2304 [I 2022-10-27 01:12:03] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:30 [I 2022-10-27 01:12:03] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00246 [I 2022-10-27 01:12:03] ...trap/engines/engine.py.230: loss: 0.14878 [I 2022-10-27 01:12:03] ...optimizers/trijoint.py.151: total_norm: 29.096424 [I 2022-10-27 01:12:09] ...trap/engines/engine.py.224: train: epoch 39 | batch 2260/2304 [I 2022-10-27 01:12:09] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:24 [I 2022-10-27 01:12:09] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00248 [I 2022-10-27 01:12:09] ...trap/engines/engine.py.230: loss: 0.16285 [I 2022-10-27 01:12:09] ...optimizers/trijoint.py.151: total_norm: 10.938041 [I 2022-10-27 01:12:14] ...trap/engines/engine.py.224: train: epoch 39 | batch 2270/2304 [I 2022-10-27 01:12:14] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:19 [I 2022-10-27 01:12:14] ...trap/engines/engine.py.229: process: 0.57887 | load: 0.00247 [I 2022-10-27 01:12:14] ...trap/engines/engine.py.230: loss: 0.17483 [I 2022-10-27 01:12:14] ...optimizers/trijoint.py.151: total_norm: 17.911694 [I 2022-10-27 01:12:20] ...trap/engines/engine.py.224: train: epoch 39 | batch 2280/2304 [I 2022-10-27 01:12:20] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:13 [I 2022-10-27 01:12:20] ...trap/engines/engine.py.229: process: 0.56234 | load: 0.00280 [I 2022-10-27 01:12:20] ...trap/engines/engine.py.230: loss: 0.15454 [I 2022-10-27 01:12:20] ...optimizers/trijoint.py.151: total_norm: 25.173986 [I 2022-10-27 01:12:26] ...trap/engines/engine.py.224: train: epoch 39 | batch 2290/2304 [I 2022-10-27 01:12:26] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:07 [I 2022-10-27 01:12:26] ...trap/engines/engine.py.229: process: 0.54924 | load: 0.00315 [I 2022-10-27 01:12:26] ...trap/engines/engine.py.230: loss: 0.22649 [I 2022-10-27 01:12:26] ...optimizers/trijoint.py.151: total_norm: 12.021285 [I 2022-10-27 01:12:31] ...trap/engines/engine.py.224: train: epoch 39 | batch 2300/2304 [I 2022-10-27 01:12:31] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:02 [I 2022-10-27 01:12:31] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00014 [I 2022-10-27 01:12:31] ...trap/engines/engine.py.230: loss: 0.14475 [I 2022-10-27 01:12:31] ...optimizers/trijoint.py.151: total_norm: 34.788303 [I 2022-10-27 01:12:34] ...trap/engines/engine.py.224: train: epoch 39 | batch 2304/2304 [I 2022-10-27 01:12:34] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:00 [I 2022-10-27 01:12:34] ...trap/engines/engine.py.229: process: 0.54067 | load: 0.00016 [I 2022-10-27 01:12:34] ...trap/engines/engine.py.230: loss: 0.15194 [I 2022-10-27 01:12:34] ...optimizers/trijoint.py.151: total_norm: 10.435064 [S 2022-10-27 01:12:35] ...trap/engines/engine.py.129: train_epoch.epoch: 39 [S 2022-10-27 01:12:35] ...trap/engines/engine.py.129: train_epoch.loss: 0.17181774801584748 [I 2022-10-27 01:12:35] ...s/criterions/triplet.py.84: margin = 0.25000000000000017 [I 2022-10-27 01:12:35] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 39 [I 2022-10-27 01:12:35] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 01:12:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 01:12:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 01:12:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 01:12:38] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 01:12:39] ...trap/engines/engine.py.321: eval: epoch 39 | batch 0/510 [I 2022-10-27 01:12:39] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:29:04 [I 2022-10-27 01:12:39] ...trap/engines/engine.py.326: process: 3.42113 | load: 3.05084 [I 2022-10-27 01:12:42] ...trap/engines/engine.py.321: eval: epoch 39 | batch 10/510 [I 2022-10-27 01:12:42] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:29 [I 2022-10-27 01:12:42] ...trap/engines/engine.py.326: process: 0.33011 | load: 0.00017 [I 2022-10-27 01:12:45] ...trap/engines/engine.py.321: eval: epoch 39 | batch 20/510 [I 2022-10-27 01:12:45] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:03:00 [I 2022-10-27 01:12:45] ...trap/engines/engine.py.326: process: 0.33150 | load: 0.00019 [I 2022-10-27 01:12:48] ...trap/engines/engine.py.321: eval: epoch 39 | batch 30/510 [I 2022-10-27 01:12:48] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:42 [I 2022-10-27 01:12:48] ...trap/engines/engine.py.326: process: 0.33604 | load: 0.00020 [I 2022-10-27 01:12:52] ...trap/engines/engine.py.321: eval: epoch 39 | batch 40/510 [I 2022-10-27 01:12:52] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:37 [I 2022-10-27 01:12:52] ...trap/engines/engine.py.326: process: 0.33695 | load: 0.00026 [I 2022-10-27 01:12:55] ...trap/engines/engine.py.321: eval: epoch 39 | batch 50/510 [I 2022-10-27 01:12:55] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:35 [I 2022-10-27 01:12:55] ...trap/engines/engine.py.326: process: 0.33092 | load: 0.00044 [I 2022-10-27 01:12:59] ...trap/engines/engine.py.321: eval: epoch 39 | batch 60/510 [I 2022-10-27 01:12:59] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:32 [I 2022-10-27 01:12:59] ...trap/engines/engine.py.326: process: 0.33080 | load: 0.00019 [I 2022-10-27 01:13:02] ...trap/engines/engine.py.321: eval: epoch 39 | batch 70/510 [I 2022-10-27 01:13:02] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:29 [I 2022-10-27 01:13:02] ...trap/engines/engine.py.326: process: 0.34205 | load: 0.00019 [I 2022-10-27 01:13:05] ...trap/engines/engine.py.321: eval: epoch 39 | batch 80/510 [I 2022-10-27 01:13:05] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-27 01:13:05] ...trap/engines/engine.py.326: process: 0.31567 | load: 0.00020 [I 2022-10-27 01:13:09] ...trap/engines/engine.py.321: eval: epoch 39 | batch 90/510 [I 2022-10-27 01:13:09] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:19 [I 2022-10-27 01:13:09] ...trap/engines/engine.py.326: process: 0.33525 | load: 0.00021 [I 2022-10-27 01:13:12] ...trap/engines/engine.py.321: eval: epoch 39 | batch 100/510 [I 2022-10-27 01:13:12] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:12 [I 2022-10-27 01:13:12] ...trap/engines/engine.py.326: process: 0.27770 | load: 0.00017 [I 2022-10-27 01:13:15] ...trap/engines/engine.py.321: eval: epoch 39 | batch 110/510 [I 2022-10-27 01:13:15] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-27 01:13:15] ...trap/engines/engine.py.326: process: 0.29001 | load: 0.00017 [I 2022-10-27 01:13:18] ...trap/engines/engine.py.321: eval: epoch 39 | batch 120/510 [I 2022-10-27 01:13:18] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:51 [I 2022-10-27 01:13:18] ...trap/engines/engine.py.326: process: 0.27955 | load: 0.00016 [I 2022-10-27 01:13:21] ...trap/engines/engine.py.321: eval: epoch 39 | batch 130/510 [I 2022-10-27 01:13:21] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:49 [I 2022-10-27 01:13:21] ...trap/engines/engine.py.326: process: 0.28633 | load: 0.00018 [I 2022-10-27 01:13:24] ...trap/engines/engine.py.321: eval: epoch 39 | batch 140/510 [I 2022-10-27 01:13:24] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-27 01:13:24] ...trap/engines/engine.py.326: process: 0.28507 | load: 0.00017 [I 2022-10-27 01:13:26] ...trap/engines/engine.py.321: eval: epoch 39 | batch 150/510 [I 2022-10-27 01:13:26] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-27 01:13:26] ...trap/engines/engine.py.326: process: 0.28765 | load: 0.00017 [I 2022-10-27 01:13:29] ...trap/engines/engine.py.321: eval: epoch 39 | batch 160/510 [I 2022-10-27 01:13:29] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-27 01:13:29] ...trap/engines/engine.py.326: process: 0.28787 | load: 0.00018 [I 2022-10-27 01:13:32] ...trap/engines/engine.py.321: eval: epoch 39 | batch 170/510 [I 2022-10-27 01:13:32] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-27 01:13:32] ...trap/engines/engine.py.326: process: 0.28862 | load: 0.00016 [I 2022-10-27 01:13:35] ...trap/engines/engine.py.321: eval: epoch 39 | batch 180/510 [I 2022-10-27 01:13:35] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:35 [I 2022-10-27 01:13:35] ...trap/engines/engine.py.326: process: 0.28369 | load: 0.00017 [I 2022-10-27 01:13:38] ...trap/engines/engine.py.321: eval: epoch 39 | batch 190/510 [I 2022-10-27 01:13:38] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:32 [I 2022-10-27 01:13:38] ...trap/engines/engine.py.326: process: 0.28731 | load: 0.00019 [I 2022-10-27 01:13:41] ...trap/engines/engine.py.321: eval: epoch 39 | batch 200/510 [I 2022-10-27 01:13:41] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:28 [I 2022-10-27 01:13:41] ...trap/engines/engine.py.326: process: 0.28268 | load: 0.00014 [I 2022-10-27 01:13:44] ...trap/engines/engine.py.321: eval: epoch 39 | batch 210/510 [I 2022-10-27 01:13:44] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:26 [I 2022-10-27 01:13:44] ...trap/engines/engine.py.326: process: 0.29028 | load: 0.00017 [I 2022-10-27 01:13:47] ...trap/engines/engine.py.321: eval: epoch 39 | batch 220/510 [I 2022-10-27 01:13:47] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-27 01:13:47] ...trap/engines/engine.py.326: process: 0.28680 | load: 0.00019 [I 2022-10-27 01:13:49] ...trap/engines/engine.py.321: eval: epoch 39 | batch 230/510 [I 2022-10-27 01:13:49] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-27 01:13:49] ...trap/engines/engine.py.326: process: 0.28711 | load: 0.00017 [I 2022-10-27 01:13:52] ...trap/engines/engine.py.321: eval: epoch 39 | batch 240/510 [I 2022-10-27 01:13:52] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-27 01:13:52] ...trap/engines/engine.py.326: process: 0.28344 | load: 0.00017 [I 2022-10-27 01:13:55] ...trap/engines/engine.py.321: eval: epoch 39 | batch 250/510 [I 2022-10-27 01:13:55] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-27 01:13:55] ...trap/engines/engine.py.326: process: 0.29186 | load: 0.00018 [I 2022-10-27 01:13:58] ...trap/engines/engine.py.321: eval: epoch 39 | batch 260/510 [I 2022-10-27 01:13:58] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:11 [I 2022-10-27 01:13:58] ...trap/engines/engine.py.326: process: 0.28670 | load: 0.00019 [I 2022-10-27 01:14:01] ...trap/engines/engine.py.321: eval: epoch 39 | batch 270/510 [I 2022-10-27 01:14:01] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:08 [I 2022-10-27 01:14:01] ...trap/engines/engine.py.326: process: 0.29025 | load: 0.00021 [I 2022-10-27 01:14:04] ...trap/engines/engine.py.321: eval: epoch 39 | batch 280/510 [I 2022-10-27 01:14:04] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:06 [I 2022-10-27 01:14:04] ...trap/engines/engine.py.326: process: 0.28903 | load: 0.00017 [I 2022-10-27 01:14:07] ...trap/engines/engine.py.321: eval: epoch 39 | batch 290/510 [I 2022-10-27 01:14:07] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:02 [I 2022-10-27 01:14:07] ...trap/engines/engine.py.326: process: 0.29245 | load: 0.00015 [I 2022-10-27 01:14:10] ...trap/engines/engine.py.321: eval: epoch 39 | batch 300/510 [I 2022-10-27 01:14:10] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-27 01:14:10] ...trap/engines/engine.py.326: process: 0.28643 | load: 0.00014 [I 2022-10-27 01:14:12] ...trap/engines/engine.py.321: eval: epoch 39 | batch 310/510 [I 2022-10-27 01:14:12] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 01:14:12] ...trap/engines/engine.py.326: process: 0.28721 | load: 0.00019 [I 2022-10-27 01:14:15] ...trap/engines/engine.py.321: eval: epoch 39 | batch 320/510 [I 2022-10-27 01:14:15] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 01:14:15] ...trap/engines/engine.py.326: process: 0.29217 | load: 0.00016 [I 2022-10-27 01:14:18] ...trap/engines/engine.py.321: eval: epoch 39 | batch 330/510 [I 2022-10-27 01:14:18] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:51 [I 2022-10-27 01:14:18] ...trap/engines/engine.py.326: process: 0.28424 | load: 0.00017 [I 2022-10-27 01:14:21] ...trap/engines/engine.py.321: eval: epoch 39 | batch 340/510 [I 2022-10-27 01:14:21] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:48 [I 2022-10-27 01:14:21] ...trap/engines/engine.py.326: process: 0.27832 | load: 0.00017 [I 2022-10-27 01:14:24] ...trap/engines/engine.py.321: eval: epoch 39 | batch 350/510 [I 2022-10-27 01:14:24] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:45 [I 2022-10-27 01:14:24] ...trap/engines/engine.py.326: process: 0.28243 | load: 0.00017 [I 2022-10-27 01:14:27] ...trap/engines/engine.py.321: eval: epoch 39 | batch 360/510 [I 2022-10-27 01:14:27] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:42 [I 2022-10-27 01:14:27] ...trap/engines/engine.py.326: process: 0.27943 | load: 0.00024 [I 2022-10-27 01:14:30] ...trap/engines/engine.py.321: eval: epoch 39 | batch 370/510 [I 2022-10-27 01:14:30] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-27 01:14:30] ...trap/engines/engine.py.326: process: 0.28817 | load: 0.00016 [I 2022-10-27 01:14:32] ...trap/engines/engine.py.321: eval: epoch 39 | batch 380/510 [I 2022-10-27 01:14:32] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-27 01:14:32] ...trap/engines/engine.py.326: process: 0.28551 | load: 0.00016 [I 2022-10-27 01:14:35] ...trap/engines/engine.py.321: eval: epoch 39 | batch 390/510 [I 2022-10-27 01:14:35] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-27 01:14:35] ...trap/engines/engine.py.326: process: 0.28563 | load: 0.00014 [I 2022-10-27 01:14:38] ...trap/engines/engine.py.321: eval: epoch 39 | batch 400/510 [I 2022-10-27 01:14:38] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:31 [I 2022-10-27 01:14:38] ...trap/engines/engine.py.326: process: 0.28813 | load: 0.00019 [I 2022-10-27 01:14:41] ...trap/engines/engine.py.321: eval: epoch 39 | batch 410/510 [I 2022-10-27 01:14:41] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:28 [I 2022-10-27 01:14:41] ...trap/engines/engine.py.326: process: 0.28907 | load: 0.00024 [I 2022-10-27 01:14:44] ...trap/engines/engine.py.321: eval: epoch 39 | batch 420/510 [I 2022-10-27 01:14:44] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:25 [I 2022-10-27 01:14:44] ...trap/engines/engine.py.326: process: 0.28703 | load: 0.00021 [I 2022-10-27 01:14:47] ...trap/engines/engine.py.321: eval: epoch 39 | batch 430/510 [I 2022-10-27 01:14:47] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:22 [I 2022-10-27 01:14:47] ...trap/engines/engine.py.326: process: 0.28614 | load: 0.00021 [I 2022-10-27 01:14:50] ...trap/engines/engine.py.321: eval: epoch 39 | batch 440/510 [I 2022-10-27 01:14:50] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-27 01:14:50] ...trap/engines/engine.py.326: process: 0.28986 | load: 0.00021 [I 2022-10-27 01:14:52] ...trap/engines/engine.py.321: eval: epoch 39 | batch 450/510 [I 2022-10-27 01:14:52] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-27 01:14:52] ...trap/engines/engine.py.326: process: 0.28936 | load: 0.00023 [I 2022-10-27 01:14:55] ...trap/engines/engine.py.321: eval: epoch 39 | batch 460/510 [I 2022-10-27 01:14:55] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-27 01:14:55] ...trap/engines/engine.py.326: process: 0.28094 | load: 0.00031 [I 2022-10-27 01:14:58] ...trap/engines/engine.py.321: eval: epoch 39 | batch 470/510 [I 2022-10-27 01:14:58] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-27 01:14:58] ...trap/engines/engine.py.326: process: 0.28667 | load: 0.00024 [I 2022-10-27 01:15:01] ...trap/engines/engine.py.321: eval: epoch 39 | batch 480/510 [I 2022-10-27 01:15:01] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:08 [I 2022-10-27 01:15:01] ...trap/engines/engine.py.326: process: 0.29206 | load: 0.00021 [I 2022-10-27 01:15:04] ...trap/engines/engine.py.321: eval: epoch 39 | batch 490/510 [I 2022-10-27 01:15:04] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:05 [I 2022-10-27 01:15:04] ...trap/engines/engine.py.326: process: 0.29082 | load: 0.00017 [I 2022-10-27 01:15:07] ...trap/engines/engine.py.321: eval: epoch 39 | batch 500/510 [I 2022-10-27 01:15:07] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:02 [I 2022-10-27 01:15:07] ...trap/engines/engine.py.326: process: 0.28871 | load: 0.00021 [I 2022-10-27 01:15:10] ...trap/engines/engine.py.321: eval: epoch 39 | batch 510/510 [I 2022-10-27 01:15:10] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:00 [I 2022-10-27 01:15:10] ...trap/engines/engine.py.326: process: 0.28184 | load: 0.00010 [S 2022-10-27 01:15:10] ...trap/engines/engine.py.132: eval_epoch.epoch: 39 [I 2022-10-27 01:15:10] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.679500 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.884300 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.925300 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.669200 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.882000 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.922100 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017642 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011145 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007267 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014063 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009033 [S 2022-10-27 01:15:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007867 [I 2022-10-27 01:15:16] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 01:15:16] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 01:15:16] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 01:15:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 01:15:16] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 01:15:17] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 01:15:20] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 01:15:26] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 01:15:26] ...trap/engines/engine.py.163: Training model on trainset for epoch 40 [I 2022-10-27 01:15:26] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 01:15:29] ...trap/engines/engine.py.224: train: epoch 40 | batch 0/2304 [I 2022-10-27 01:15:29] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:23 [I 2022-10-27 01:15:29] ...trap/engines/engine.py.229: process: 2.66660 | load: 2.04647 [I 2022-10-27 01:15:29] ...trap/engines/engine.py.230: loss: 0.17835 [I 2022-10-27 01:15:29] ...optimizers/trijoint.py.151: total_norm: 14.178853 [I 2022-10-27 01:15:34] ...trap/engines/engine.py.224: train: epoch 40 | batch 10/2304 [I 2022-10-27 01:15:34] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:38 [I 2022-10-27 01:15:34] ...trap/engines/engine.py.229: process: 0.62912 | load: 0.00353 [I 2022-10-27 01:15:34] ...trap/engines/engine.py.230: loss: 0.16812 [I 2022-10-27 01:15:34] ...optimizers/trijoint.py.151: total_norm: 13.767098 [I 2022-10-27 01:15:40] ...trap/engines/engine.py.224: train: epoch 40 | batch 20/2304 [I 2022-10-27 01:15:40] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:10 [I 2022-10-27 01:15:40] ...trap/engines/engine.py.229: process: 0.55023 | load: 0.00396 [I 2022-10-27 01:15:40] ...trap/engines/engine.py.230: loss: 0.14507 [I 2022-10-27 01:15:40] ...optimizers/trijoint.py.151: total_norm: 43.680004 [I 2022-10-27 01:15:46] ...trap/engines/engine.py.224: train: epoch 40 | batch 30/2304 [I 2022-10-27 01:15:46] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:22:03 [I 2022-10-27 01:15:46] ...trap/engines/engine.py.229: process: 0.56540 | load: 0.00344 [I 2022-10-27 01:15:46] ...trap/engines/engine.py.230: loss: 0.14304 [I 2022-10-27 01:15:46] ...optimizers/trijoint.py.151: total_norm: 17.391212 [I 2022-10-27 01:15:51] ...trap/engines/engine.py.224: train: epoch 40 | batch 40/2304 [I 2022-10-27 01:15:51] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:57 [I 2022-10-27 01:15:51] ...trap/engines/engine.py.229: process: 0.54439 | load: 0.00312 [I 2022-10-27 01:15:51] ...trap/engines/engine.py.230: loss: 0.17886 [I 2022-10-27 01:15:51] ...optimizers/trijoint.py.151: total_norm: 21.686716 [I 2022-10-27 01:15:57] ...trap/engines/engine.py.224: train: epoch 40 | batch 50/2304 [I 2022-10-27 01:15:57] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:46 [I 2022-10-27 01:15:57] ...trap/engines/engine.py.229: process: 0.54877 | load: 0.00311 [I 2022-10-27 01:15:57] ...trap/engines/engine.py.230: loss: 0.22019 [I 2022-10-27 01:15:57] ...optimizers/trijoint.py.151: total_norm: 19.326216 [I 2022-10-27 01:16:02] ...trap/engines/engine.py.224: train: epoch 40 | batch 60/2304 [I 2022-10-27 01:16:02] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:37 [I 2022-10-27 01:16:02] ...trap/engines/engine.py.229: process: 0.54165 | load: 0.00255 [I 2022-10-27 01:16:02] ...trap/engines/engine.py.230: loss: 0.21452 [I 2022-10-27 01:16:02] ...optimizers/trijoint.py.151: total_norm: 35.020691 [I 2022-10-27 01:16:08] ...trap/engines/engine.py.224: train: epoch 40 | batch 70/2304 [I 2022-10-27 01:16:08] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:36 [I 2022-10-27 01:16:08] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00276 [I 2022-10-27 01:16:08] ...trap/engines/engine.py.230: loss: 0.16657 [I 2022-10-27 01:16:08] ...optimizers/trijoint.py.151: total_norm: 10.539457 [I 2022-10-27 01:16:13] ...trap/engines/engine.py.224: train: epoch 40 | batch 80/2304 [I 2022-10-27 01:16:13] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:38 [I 2022-10-27 01:16:13] ...trap/engines/engine.py.229: process: 0.55803 | load: 0.00303 [I 2022-10-27 01:16:13] ...trap/engines/engine.py.230: loss: 0.09603 [I 2022-10-27 01:16:13] ...optimizers/trijoint.py.151: total_norm: 12.017610 [I 2022-10-27 01:16:19] ...trap/engines/engine.py.224: train: epoch 40 | batch 90/2304 [I 2022-10-27 01:16:19] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:22 [I 2022-10-27 01:16:19] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00276 [I 2022-10-27 01:16:19] ...trap/engines/engine.py.230: loss: 0.25162 [I 2022-10-27 01:16:19] ...optimizers/trijoint.py.151: total_norm: 32.080582 [I 2022-10-27 01:16:25] ...trap/engines/engine.py.224: train: epoch 40 | batch 100/2304 [I 2022-10-27 01:16:25] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:29 [I 2022-10-27 01:16:25] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00250 [I 2022-10-27 01:16:25] ...trap/engines/engine.py.230: loss: 0.17241 [I 2022-10-27 01:16:25] ...optimizers/trijoint.py.151: total_norm: 14.790412 [I 2022-10-27 01:16:30] ...trap/engines/engine.py.224: train: epoch 40 | batch 110/2304 [I 2022-10-27 01:16:30] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:40 [I 2022-10-27 01:16:30] ...trap/engines/engine.py.229: process: 0.57061 | load: 0.00262 [I 2022-10-27 01:16:30] ...trap/engines/engine.py.230: loss: 0.29540 [I 2022-10-27 01:16:30] ...optimizers/trijoint.py.151: total_norm: 48.871235 [I 2022-10-27 01:16:36] ...trap/engines/engine.py.224: train: epoch 40 | batch 120/2304 [I 2022-10-27 01:16:36] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:16 [I 2022-10-27 01:16:36] ...trap/engines/engine.py.229: process: 0.56859 | load: 0.00269 [I 2022-10-27 01:16:36] ...trap/engines/engine.py.230: loss: 0.19887 [I 2022-10-27 01:16:36] ...optimizers/trijoint.py.151: total_norm: 15.009751 [I 2022-10-27 01:16:41] ...trap/engines/engine.py.224: train: epoch 40 | batch 130/2304 [I 2022-10-27 01:16:41] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:08 [I 2022-10-27 01:16:41] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00251 [I 2022-10-27 01:16:41] ...trap/engines/engine.py.230: loss: 0.25496 [I 2022-10-27 01:16:41] ...optimizers/trijoint.py.151: total_norm: 14.812691 [I 2022-10-27 01:16:47] ...trap/engines/engine.py.224: train: epoch 40 | batch 140/2304 [I 2022-10-27 01:16:47] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:04 [I 2022-10-27 01:16:47] ...trap/engines/engine.py.229: process: 0.55141 | load: 0.00251 [I 2022-10-27 01:16:47] ...trap/engines/engine.py.230: loss: 0.21083 [I 2022-10-27 01:16:47] ...optimizers/trijoint.py.151: total_norm: 59.082272 [I 2022-10-27 01:16:53] ...trap/engines/engine.py.224: train: epoch 40 | batch 150/2304 [I 2022-10-27 01:16:53] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:01 [I 2022-10-27 01:16:53] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00266 [I 2022-10-27 01:16:53] ...trap/engines/engine.py.230: loss: 0.17297 [I 2022-10-27 01:16:53] ...optimizers/trijoint.py.151: total_norm: 67.387787 [I 2022-10-27 01:16:58] ...trap/engines/engine.py.224: train: epoch 40 | batch 160/2304 [I 2022-10-27 01:16:58] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:44 [I 2022-10-27 01:16:58] ...trap/engines/engine.py.229: process: 0.55231 | load: 0.00259 [I 2022-10-27 01:16:58] ...trap/engines/engine.py.230: loss: 0.17697 [I 2022-10-27 01:16:58] ...optimizers/trijoint.py.151: total_norm: 16.089191 [I 2022-10-27 01:17:04] ...trap/engines/engine.py.224: train: epoch 40 | batch 170/2304 [I 2022-10-27 01:17:04] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:20:04 [I 2022-10-27 01:17:04] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00267 [I 2022-10-27 01:17:04] ...trap/engines/engine.py.230: loss: 0.17244 [I 2022-10-27 01:17:04] ...optimizers/trijoint.py.151: total_norm: 19.705509 [I 2022-10-27 01:17:09] ...trap/engines/engine.py.224: train: epoch 40 | batch 180/2304 [I 2022-10-27 01:17:09] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:50 [I 2022-10-27 01:17:09] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00246 [I 2022-10-27 01:17:09] ...trap/engines/engine.py.230: loss: 0.18155 [I 2022-10-27 01:17:09] ...optimizers/trijoint.py.151: total_norm: 16.552217 [I 2022-10-27 01:17:15] ...trap/engines/engine.py.224: train: epoch 40 | batch 190/2304 [I 2022-10-27 01:17:15] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:37 [I 2022-10-27 01:17:15] ...trap/engines/engine.py.229: process: 0.55354 | load: 0.00249 [I 2022-10-27 01:17:15] ...trap/engines/engine.py.230: loss: 0.16898 [I 2022-10-27 01:17:15] ...optimizers/trijoint.py.151: total_norm: 20.085770 [I 2022-10-27 01:17:21] ...trap/engines/engine.py.224: train: epoch 40 | batch 200/2304 [I 2022-10-27 01:17:21] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:25 [I 2022-10-27 01:17:21] ...trap/engines/engine.py.229: process: 0.55009 | load: 0.00262 [I 2022-10-27 01:17:21] ...trap/engines/engine.py.230: loss: 0.14767 [I 2022-10-27 01:17:21] ...optimizers/trijoint.py.151: total_norm: 13.433071 [I 2022-10-27 01:17:26] ...trap/engines/engine.py.224: train: epoch 40 | batch 210/2304 [I 2022-10-27 01:17:26] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:17 [I 2022-10-27 01:17:26] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00254 [I 2022-10-27 01:17:26] ...trap/engines/engine.py.230: loss: 0.15835 [I 2022-10-27 01:17:26] ...optimizers/trijoint.py.151: total_norm: 31.251968 [I 2022-10-27 01:17:32] ...trap/engines/engine.py.224: train: epoch 40 | batch 220/2304 [I 2022-10-27 01:17:32] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:15 [I 2022-10-27 01:17:32] ...trap/engines/engine.py.229: process: 0.56418 | load: 0.00254 [I 2022-10-27 01:17:32] ...trap/engines/engine.py.230: loss: 0.12664 [I 2022-10-27 01:17:32] ...optimizers/trijoint.py.151: total_norm: 12.421594 [I 2022-10-27 01:17:37] ...trap/engines/engine.py.224: train: epoch 40 | batch 230/2304 [I 2022-10-27 01:17:37] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:15 [I 2022-10-27 01:17:37] ...trap/engines/engine.py.229: process: 0.55000 | load: 0.00265 [I 2022-10-27 01:17:37] ...trap/engines/engine.py.230: loss: 0.17431 [I 2022-10-27 01:17:37] ...optimizers/trijoint.py.151: total_norm: 8.682381 [I 2022-10-27 01:17:43] ...trap/engines/engine.py.224: train: epoch 40 | batch 240/2304 [I 2022-10-27 01:17:43] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:06 [I 2022-10-27 01:17:43] ...trap/engines/engine.py.229: process: 0.55209 | load: 0.00247 [I 2022-10-27 01:17:43] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-27 01:17:43] ...optimizers/trijoint.py.151: total_norm: 30.902304 [I 2022-10-27 01:17:48] ...trap/engines/engine.py.224: train: epoch 40 | batch 250/2304 [I 2022-10-27 01:17:48] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:05 [I 2022-10-27 01:17:48] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00244 [I 2022-10-27 01:17:48] ...trap/engines/engine.py.230: loss: 0.13197 [I 2022-10-27 01:17:48] ...optimizers/trijoint.py.151: total_norm: 13.579486 [I 2022-10-27 01:17:54] ...trap/engines/engine.py.224: train: epoch 40 | batch 260/2304 [I 2022-10-27 01:17:54] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:45 [I 2022-10-27 01:17:54] ...trap/engines/engine.py.229: process: 0.54221 | load: 0.00256 [I 2022-10-27 01:17:54] ...trap/engines/engine.py.230: loss: 0.15661 [I 2022-10-27 01:17:54] ...optimizers/trijoint.py.151: total_norm: 11.898062 [I 2022-10-27 01:18:00] ...trap/engines/engine.py.224: train: epoch 40 | batch 270/2304 [I 2022-10-27 01:18:00] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:45 [I 2022-10-27 01:18:00] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00247 [I 2022-10-27 01:18:00] ...trap/engines/engine.py.230: loss: 0.14007 [I 2022-10-27 01:18:00] ...optimizers/trijoint.py.151: total_norm: 14.235985 [I 2022-10-27 01:18:05] ...trap/engines/engine.py.224: train: epoch 40 | batch 280/2304 [I 2022-10-27 01:18:05] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:19:36 [I 2022-10-27 01:18:05] ...trap/engines/engine.py.229: process: 0.63351 | load: 0.00276 [I 2022-10-27 01:18:05] ...trap/engines/engine.py.230: loss: 0.19253 [I 2022-10-27 01:18:05] ...optimizers/trijoint.py.151: total_norm: 17.522333 [I 2022-10-27 01:18:11] ...trap/engines/engine.py.224: train: epoch 40 | batch 290/2304 [I 2022-10-27 01:18:11] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:48 [I 2022-10-27 01:18:11] ...trap/engines/engine.py.229: process: 0.56340 | load: 0.00277 [I 2022-10-27 01:18:11] ...trap/engines/engine.py.230: loss: 0.11112 [I 2022-10-27 01:18:11] ...optimizers/trijoint.py.151: total_norm: 14.032351 [I 2022-10-27 01:18:16] ...trap/engines/engine.py.224: train: epoch 40 | batch 300/2304 [I 2022-10-27 01:18:16] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:32 [I 2022-10-27 01:18:16] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00246 [I 2022-10-27 01:18:16] ...trap/engines/engine.py.230: loss: 0.12951 [I 2022-10-27 01:18:16] ...optimizers/trijoint.py.151: total_norm: 9.541320 [I 2022-10-27 01:18:22] ...trap/engines/engine.py.224: train: epoch 40 | batch 310/2304 [I 2022-10-27 01:18:22] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:37 [I 2022-10-27 01:18:22] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00264 [I 2022-10-27 01:18:22] ...trap/engines/engine.py.230: loss: 0.13719 [I 2022-10-27 01:18:22] ...optimizers/trijoint.py.151: total_norm: 11.777657 [I 2022-10-27 01:18:28] ...trap/engines/engine.py.224: train: epoch 40 | batch 320/2304 [I 2022-10-27 01:18:28] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:25 [I 2022-10-27 01:18:28] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00249 [I 2022-10-27 01:18:28] ...trap/engines/engine.py.230: loss: 0.16880 [I 2022-10-27 01:18:28] ...optimizers/trijoint.py.151: total_norm: 24.022120 [I 2022-10-27 01:18:33] ...trap/engines/engine.py.224: train: epoch 40 | batch 330/2304 [I 2022-10-27 01:18:33] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:23 [I 2022-10-27 01:18:33] ...trap/engines/engine.py.229: process: 0.56299 | load: 0.00246 [I 2022-10-27 01:18:33] ...trap/engines/engine.py.230: loss: 0.12421 [I 2022-10-27 01:18:33] ...optimizers/trijoint.py.151: total_norm: 13.385641 [I 2022-10-27 01:18:39] ...trap/engines/engine.py.224: train: epoch 40 | batch 340/2304 [I 2022-10-27 01:18:39] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:10 [I 2022-10-27 01:18:39] ...trap/engines/engine.py.229: process: 0.55931 | load: 0.00244 [I 2022-10-27 01:18:39] ...trap/engines/engine.py.230: loss: 0.16222 [I 2022-10-27 01:18:39] ...optimizers/trijoint.py.151: total_norm: 17.924900 [I 2022-10-27 01:18:44] ...trap/engines/engine.py.224: train: epoch 40 | batch 350/2304 [I 2022-10-27 01:18:44] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:07 [I 2022-10-27 01:18:44] ...trap/engines/engine.py.229: process: 0.54965 | load: 0.00248 [I 2022-10-27 01:18:44] ...trap/engines/engine.py.230: loss: 0.12650 [I 2022-10-27 01:18:44] ...optimizers/trijoint.py.151: total_norm: 9.625489 [I 2022-10-27 01:18:50] ...trap/engines/engine.py.224: train: epoch 40 | batch 360/2304 [I 2022-10-27 01:18:50] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:57 [I 2022-10-27 01:18:50] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00252 [I 2022-10-27 01:18:50] ...trap/engines/engine.py.230: loss: 0.11191 [I 2022-10-27 01:18:50] ...optimizers/trijoint.py.151: total_norm: 14.487951 [I 2022-10-27 01:18:55] ...trap/engines/engine.py.224: train: epoch 40 | batch 370/2304 [I 2022-10-27 01:18:55] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:52 [I 2022-10-27 01:18:55] ...trap/engines/engine.py.229: process: 0.55764 | load: 0.00245 [I 2022-10-27 01:18:55] ...trap/engines/engine.py.230: loss: 0.25001 [I 2022-10-27 01:18:55] ...optimizers/trijoint.py.151: total_norm: 26.521814 [I 2022-10-27 01:19:01] ...trap/engines/engine.py.224: train: epoch 40 | batch 380/2304 [I 2022-10-27 01:19:01] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:46 [I 2022-10-27 01:19:01] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00251 [I 2022-10-27 01:19:01] ...trap/engines/engine.py.230: loss: 0.18338 [I 2022-10-27 01:19:01] ...optimizers/trijoint.py.151: total_norm: 23.079985 [I 2022-10-27 01:19:07] ...trap/engines/engine.py.224: train: epoch 40 | batch 390/2304 [I 2022-10-27 01:19:07] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:43 [I 2022-10-27 01:19:07] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00248 [I 2022-10-27 01:19:07] ...trap/engines/engine.py.230: loss: 0.12493 [I 2022-10-27 01:19:07] ...optimizers/trijoint.py.151: total_norm: 21.129463 [I 2022-10-27 01:19:12] ...trap/engines/engine.py.224: train: epoch 40 | batch 400/2304 [I 2022-10-27 01:19:12] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:49 [I 2022-10-27 01:19:12] ...trap/engines/engine.py.229: process: 0.56555 | load: 0.00255 [I 2022-10-27 01:19:12] ...trap/engines/engine.py.230: loss: 0.13917 [I 2022-10-27 01:19:12] ...optimizers/trijoint.py.151: total_norm: 67.840111 [I 2022-10-27 01:19:18] ...trap/engines/engine.py.224: train: epoch 40 | batch 410/2304 [I 2022-10-27 01:19:18] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:34 [I 2022-10-27 01:19:18] ...trap/engines/engine.py.229: process: 0.56401 | load: 0.00243 [I 2022-10-27 01:19:18] ...trap/engines/engine.py.230: loss: 0.24203 [I 2022-10-27 01:19:18] ...optimizers/trijoint.py.151: total_norm: 29.773195 [I 2022-10-27 01:19:23] ...trap/engines/engine.py.224: train: epoch 40 | batch 420/2304 [I 2022-10-27 01:19:23] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:28 [I 2022-10-27 01:19:23] ...trap/engines/engine.py.229: process: 0.54965 | load: 0.00247 [I 2022-10-27 01:19:23] ...trap/engines/engine.py.230: loss: 0.17960 [I 2022-10-27 01:19:23] ...optimizers/trijoint.py.151: total_norm: 20.643957 [I 2022-10-27 01:19:29] ...trap/engines/engine.py.224: train: epoch 40 | batch 430/2304 [I 2022-10-27 01:19:29] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:24 [I 2022-10-27 01:19:29] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00262 [I 2022-10-27 01:19:29] ...trap/engines/engine.py.230: loss: 0.19312 [I 2022-10-27 01:19:29] ...optimizers/trijoint.py.151: total_norm: 16.198536 [I 2022-10-27 01:19:35] ...trap/engines/engine.py.224: train: epoch 40 | batch 440/2304 [I 2022-10-27 01:19:35] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:24 [I 2022-10-27 01:19:35] ...trap/engines/engine.py.229: process: 0.56787 | load: 0.00250 [I 2022-10-27 01:19:35] ...trap/engines/engine.py.230: loss: 0.14538 [I 2022-10-27 01:19:35] ...optimizers/trijoint.py.151: total_norm: 24.291994 [I 2022-10-27 01:19:40] ...trap/engines/engine.py.224: train: epoch 40 | batch 450/2304 [I 2022-10-27 01:19:40] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:08 [I 2022-10-27 01:19:40] ...trap/engines/engine.py.229: process: 0.54872 | load: 0.00249 [I 2022-10-27 01:19:40] ...trap/engines/engine.py.230: loss: 0.17600 [I 2022-10-27 01:19:40] ...optimizers/trijoint.py.151: total_norm: 13.688766 [I 2022-10-27 01:19:46] ...trap/engines/engine.py.224: train: epoch 40 | batch 460/2304 [I 2022-10-27 01:19:46] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:04 [I 2022-10-27 01:19:46] ...trap/engines/engine.py.229: process: 0.55015 | load: 0.00246 [I 2022-10-27 01:19:46] ...trap/engines/engine.py.230: loss: 0.16355 [I 2022-10-27 01:19:46] ...optimizers/trijoint.py.151: total_norm: 13.809516 [I 2022-10-27 01:19:51] ...trap/engines/engine.py.224: train: epoch 40 | batch 470/2304 [I 2022-10-27 01:19:51] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:02 [I 2022-10-27 01:19:51] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00243 [I 2022-10-27 01:19:51] ...trap/engines/engine.py.230: loss: 0.14957 [I 2022-10-27 01:19:51] ...optimizers/trijoint.py.151: total_norm: 15.262053 [I 2022-10-27 01:19:57] ...trap/engines/engine.py.224: train: epoch 40 | batch 480/2304 [I 2022-10-27 01:19:57] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:17:00 [I 2022-10-27 01:19:57] ...trap/engines/engine.py.229: process: 0.56221 | load: 0.00244 [I 2022-10-27 01:19:57] ...trap/engines/engine.py.230: loss: 0.16126 [I 2022-10-27 01:19:57] ...optimizers/trijoint.py.151: total_norm: 16.269363 [I 2022-10-27 01:20:03] ...trap/engines/engine.py.224: train: epoch 40 | batch 490/2304 [I 2022-10-27 01:20:03] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:57 [I 2022-10-27 01:20:03] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00242 [I 2022-10-27 01:20:03] ...trap/engines/engine.py.230: loss: 0.15323 [I 2022-10-27 01:20:03] ...optimizers/trijoint.py.151: total_norm: 14.901132 [I 2022-10-27 01:20:08] ...trap/engines/engine.py.224: train: epoch 40 | batch 500/2304 [I 2022-10-27 01:20:08] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:43 [I 2022-10-27 01:20:08] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00241 [I 2022-10-27 01:20:08] ...trap/engines/engine.py.230: loss: 0.13937 [I 2022-10-27 01:20:08] ...optimizers/trijoint.py.151: total_norm: 29.775150 [I 2022-10-27 01:20:14] ...trap/engines/engine.py.224: train: epoch 40 | batch 510/2304 [I 2022-10-27 01:20:14] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:41 [I 2022-10-27 01:20:14] ...trap/engines/engine.py.229: process: 0.54766 | load: 0.00271 [I 2022-10-27 01:20:14] ...trap/engines/engine.py.230: loss: 0.11845 [I 2022-10-27 01:20:14] ...optimizers/trijoint.py.151: total_norm: 11.151985 [I 2022-10-27 01:20:19] ...trap/engines/engine.py.224: train: epoch 40 | batch 520/2304 [I 2022-10-27 01:20:19] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:17:03 [I 2022-10-27 01:20:19] ...trap/engines/engine.py.229: process: 0.55315 | load: 0.00247 [I 2022-10-27 01:20:19] ...trap/engines/engine.py.230: loss: 0.19895 [I 2022-10-27 01:20:19] ...optimizers/trijoint.py.151: total_norm: 32.108616 [I 2022-10-27 01:20:25] ...trap/engines/engine.py.224: train: epoch 40 | batch 530/2304 [I 2022-10-27 01:20:25] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:33 [I 2022-10-27 01:20:25] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00266 [I 2022-10-27 01:20:25] ...trap/engines/engine.py.230: loss: 0.18077 [I 2022-10-27 01:20:25] ...optimizers/trijoint.py.151: total_norm: 19.334698 [I 2022-10-27 01:20:31] ...trap/engines/engine.py.224: train: epoch 40 | batch 540/2304 [I 2022-10-27 01:20:31] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:16 [I 2022-10-27 01:20:31] ...trap/engines/engine.py.229: process: 0.54843 | load: 0.00244 [I 2022-10-27 01:20:31] ...trap/engines/engine.py.230: loss: 0.13290 [I 2022-10-27 01:20:31] ...optimizers/trijoint.py.151: total_norm: 17.950626 [I 2022-10-27 01:20:36] ...trap/engines/engine.py.224: train: epoch 40 | batch 550/2304 [I 2022-10-27 01:20:36] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:25 [I 2022-10-27 01:20:36] ...trap/engines/engine.py.229: process: 0.57067 | load: 0.00241 [I 2022-10-27 01:20:36] ...trap/engines/engine.py.230: loss: 0.19556 [I 2022-10-27 01:20:36] ...optimizers/trijoint.py.151: total_norm: 31.629316 [I 2022-10-27 01:21:02] ...trap/engines/engine.py.224: train: epoch 40 | batch 560/2304 [I 2022-10-27 01:21:02] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 1:49:22 [I 2022-10-27 01:21:02] ...trap/engines/engine.py.229: process: 0.53497 | load: 0.00240 [I 2022-10-27 01:21:02] ...trap/engines/engine.py.230: loss: 0.11831 [I 2022-10-27 01:21:02] ...optimizers/trijoint.py.151: total_norm: 7.597689 [I 2022-10-27 01:21:07] ...trap/engines/engine.py.224: train: epoch 40 | batch 570/2304 [I 2022-10-27 01:21:07] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:26:00 [I 2022-10-27 01:21:07] ...trap/engines/engine.py.229: process: 0.55297 | load: 0.00237 [I 2022-10-27 01:21:07] ...trap/engines/engine.py.230: loss: 0.16076 [I 2022-10-27 01:21:07] ...optimizers/trijoint.py.151: total_norm: 12.161521 [I 2022-10-27 01:21:13] ...trap/engines/engine.py.224: train: epoch 40 | batch 580/2304 [I 2022-10-27 01:21:13] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:17:01 [I 2022-10-27 01:21:13] ...trap/engines/engine.py.229: process: 0.55284 | load: 0.00237 [I 2022-10-27 01:21:13] ...trap/engines/engine.py.230: loss: 0.19656 [I 2022-10-27 01:21:13] ...optimizers/trijoint.py.151: total_norm: 16.941666 [I 2022-10-27 01:21:19] ...trap/engines/engine.py.224: train: epoch 40 | batch 590/2304 [I 2022-10-27 01:21:19] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:15:56 [I 2022-10-27 01:21:19] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00235 [I 2022-10-27 01:21:19] ...trap/engines/engine.py.230: loss: 0.24637 [I 2022-10-27 01:21:19] ...optimizers/trijoint.py.151: total_norm: 34.090141 [I 2022-10-27 01:21:24] ...trap/engines/engine.py.224: train: epoch 40 | batch 600/2304 [I 2022-10-27 01:21:24] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:50 [I 2022-10-27 01:21:24] ...trap/engines/engine.py.229: process: 0.54668 | load: 0.00247 [I 2022-10-27 01:21:24] ...trap/engines/engine.py.230: loss: 0.17628 [I 2022-10-27 01:21:24] ...optimizers/trijoint.py.151: total_norm: 16.115217 [I 2022-10-27 01:21:30] ...trap/engines/engine.py.224: train: epoch 40 | batch 610/2304 [I 2022-10-27 01:21:30] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:15:46 [I 2022-10-27 01:21:30] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00248 [I 2022-10-27 01:21:30] ...trap/engines/engine.py.230: loss: 0.14335 [I 2022-10-27 01:21:30] ...optimizers/trijoint.py.151: total_norm: 17.853737 [I 2022-10-27 01:21:35] ...trap/engines/engine.py.224: train: epoch 40 | batch 620/2304 [I 2022-10-27 01:21:35] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:31 [I 2022-10-27 01:21:35] ...trap/engines/engine.py.229: process: 0.54321 | load: 0.00240 [I 2022-10-27 01:21:35] ...trap/engines/engine.py.230: loss: 0.35438 [I 2022-10-27 01:21:35] ...optimizers/trijoint.py.151: total_norm: 25.030294 [I 2022-10-27 01:21:41] ...trap/engines/engine.py.224: train: epoch 40 | batch 630/2304 [I 2022-10-27 01:21:41] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:15:31 [I 2022-10-27 01:21:41] ...trap/engines/engine.py.229: process: 0.55419 | load: 0.00240 [I 2022-10-27 01:21:41] ...trap/engines/engine.py.230: loss: 0.12576 [I 2022-10-27 01:21:41] ...optimizers/trijoint.py.151: total_norm: 14.311825 [I 2022-10-27 01:21:46] ...trap/engines/engine.py.224: train: epoch 40 | batch 640/2304 [I 2022-10-27 01:21:46] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:19 [I 2022-10-27 01:21:46] ...trap/engines/engine.py.229: process: 0.55462 | load: 0.00243 [I 2022-10-27 01:21:46] ...trap/engines/engine.py.230: loss: 0.17115 [I 2022-10-27 01:21:46] ...optimizers/trijoint.py.151: total_norm: 16.946825 [I 2022-10-27 01:21:52] ...trap/engines/engine.py.224: train: epoch 40 | batch 650/2304 [I 2022-10-27 01:21:52] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:15:17 [I 2022-10-27 01:21:52] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00240 [I 2022-10-27 01:21:52] ...trap/engines/engine.py.230: loss: 0.11579 [I 2022-10-27 01:21:52] ...optimizers/trijoint.py.151: total_norm: 13.068045 [I 2022-10-27 01:21:58] ...trap/engines/engine.py.224: train: epoch 40 | batch 660/2304 [I 2022-10-27 01:21:58] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:15:16 [I 2022-10-27 01:21:58] ...trap/engines/engine.py.229: process: 0.56633 | load: 0.00248 [I 2022-10-27 01:21:58] ...trap/engines/engine.py.230: loss: 0.19015 [I 2022-10-27 01:21:58] ...optimizers/trijoint.py.151: total_norm: 41.453320 [I 2022-10-27 01:22:03] ...trap/engines/engine.py.224: train: epoch 40 | batch 670/2304 [I 2022-10-27 01:22:03] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:12 [I 2022-10-27 01:22:03] ...trap/engines/engine.py.229: process: 0.55417 | load: 0.00241 [I 2022-10-27 01:22:03] ...trap/engines/engine.py.230: loss: 0.18803 [I 2022-10-27 01:22:03] ...optimizers/trijoint.py.151: total_norm: 22.548758 [I 2022-10-27 01:22:09] ...trap/engines/engine.py.224: train: epoch 40 | batch 680/2304 [I 2022-10-27 01:22:09] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:15:05 [I 2022-10-27 01:22:09] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00238 [I 2022-10-27 01:22:09] ...trap/engines/engine.py.230: loss: 0.23019 [I 2022-10-27 01:22:09] ...optimizers/trijoint.py.151: total_norm: 20.185040 [I 2022-10-27 01:22:14] ...trap/engines/engine.py.224: train: epoch 40 | batch 690/2304 [I 2022-10-27 01:22:14] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:14:52 [I 2022-10-27 01:22:14] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00241 [I 2022-10-27 01:22:14] ...trap/engines/engine.py.230: loss: 0.17908 [I 2022-10-27 01:22:14] ...optimizers/trijoint.py.151: total_norm: 13.045463 [I 2022-10-27 01:22:20] ...trap/engines/engine.py.224: train: epoch 40 | batch 700/2304 [I 2022-10-27 01:22:20] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:14:50 [I 2022-10-27 01:22:20] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00238 [I 2022-10-27 01:22:20] ...trap/engines/engine.py.230: loss: 0.15610 [I 2022-10-27 01:22:20] ...optimizers/trijoint.py.151: total_norm: 15.584665 [I 2022-10-27 01:22:25] ...trap/engines/engine.py.224: train: epoch 40 | batch 710/2304 [I 2022-10-27 01:22:25] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:14:46 [I 2022-10-27 01:22:25] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00254 [I 2022-10-27 01:22:25] ...trap/engines/engine.py.230: loss: 0.14444 [I 2022-10-27 01:22:25] ...optimizers/trijoint.py.151: total_norm: 42.791153 [I 2022-10-27 01:22:31] ...trap/engines/engine.py.224: train: epoch 40 | batch 720/2304 [I 2022-10-27 01:22:31] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:14:36 [I 2022-10-27 01:22:31] ...trap/engines/engine.py.229: process: 0.55043 | load: 0.00246 [I 2022-10-27 01:22:31] ...trap/engines/engine.py.230: loss: 0.16152 [I 2022-10-27 01:22:31] ...optimizers/trijoint.py.151: total_norm: 24.086826 [I 2022-10-27 01:22:37] ...trap/engines/engine.py.224: train: epoch 40 | batch 730/2304 [I 2022-10-27 01:22:37] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:14:26 [I 2022-10-27 01:22:37] ...trap/engines/engine.py.229: process: 0.54519 | load: 0.00241 [I 2022-10-27 01:22:37] ...trap/engines/engine.py.230: loss: 0.14067 [I 2022-10-27 01:22:37] ...optimizers/trijoint.py.151: total_norm: 46.739929 [I 2022-10-27 01:22:42] ...trap/engines/engine.py.224: train: epoch 40 | batch 740/2304 [I 2022-10-27 01:22:42] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:30 [I 2022-10-27 01:22:42] ...trap/engines/engine.py.229: process: 0.54253 | load: 0.00245 [I 2022-10-27 01:22:42] ...trap/engines/engine.py.230: loss: 0.12850 [I 2022-10-27 01:22:42] ...optimizers/trijoint.py.151: total_norm: 16.567984 [I 2022-10-27 01:22:48] ...trap/engines/engine.py.224: train: epoch 40 | batch 750/2304 [I 2022-10-27 01:22:48] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:14:25 [I 2022-10-27 01:22:48] ...trap/engines/engine.py.229: process: 0.56786 | load: 0.00244 [I 2022-10-27 01:22:48] ...trap/engines/engine.py.230: loss: 0.14405 [I 2022-10-27 01:22:48] ...optimizers/trijoint.py.151: total_norm: 69.108818 [I 2022-10-27 01:22:53] ...trap/engines/engine.py.224: train: epoch 40 | batch 760/2304 [I 2022-10-27 01:22:53] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:14:16 [I 2022-10-27 01:22:53] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00247 [I 2022-10-27 01:22:53] ...trap/engines/engine.py.230: loss: 0.11124 [I 2022-10-27 01:22:53] ...optimizers/trijoint.py.151: total_norm: 16.875973 [I 2022-10-27 01:22:59] ...trap/engines/engine.py.224: train: epoch 40 | batch 770/2304 [I 2022-10-27 01:22:59] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:14:15 [I 2022-10-27 01:22:59] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00242 [I 2022-10-27 01:22:59] ...trap/engines/engine.py.230: loss: 0.17702 [I 2022-10-27 01:22:59] ...optimizers/trijoint.py.151: total_norm: 14.710927 [I 2022-10-27 01:23:04] ...trap/engines/engine.py.224: train: epoch 40 | batch 780/2304 [I 2022-10-27 01:23:04] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:14:13 [I 2022-10-27 01:23:04] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00264 [I 2022-10-27 01:23:04] ...trap/engines/engine.py.230: loss: 0.19199 [I 2022-10-27 01:23:04] ...optimizers/trijoint.py.151: total_norm: 31.177532 [I 2022-10-27 01:23:10] ...trap/engines/engine.py.224: train: epoch 40 | batch 790/2304 [I 2022-10-27 01:23:10] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:14:08 [I 2022-10-27 01:23:10] ...trap/engines/engine.py.229: process: 0.54955 | load: 0.00245 [I 2022-10-27 01:23:10] ...trap/engines/engine.py.230: loss: 0.13157 [I 2022-10-27 01:23:10] ...optimizers/trijoint.py.151: total_norm: 10.742023 [I 2022-10-27 01:23:16] ...trap/engines/engine.py.224: train: epoch 40 | batch 800/2304 [I 2022-10-27 01:23:16] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:13:59 [I 2022-10-27 01:23:16] ...trap/engines/engine.py.229: process: 0.54709 | load: 0.00242 [I 2022-10-27 01:23:16] ...trap/engines/engine.py.230: loss: 0.18586 [I 2022-10-27 01:23:16] ...optimizers/trijoint.py.151: total_norm: 25.394396 [I 2022-10-27 01:23:21] ...trap/engines/engine.py.224: train: epoch 40 | batch 810/2304 [I 2022-10-27 01:23:21] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:13:56 [I 2022-10-27 01:23:21] ...trap/engines/engine.py.229: process: 0.56765 | load: 0.00243 [I 2022-10-27 01:23:21] ...trap/engines/engine.py.230: loss: 0.13526 [I 2022-10-27 01:23:21] ...optimizers/trijoint.py.151: total_norm: 14.066173 [I 2022-10-27 01:23:27] ...trap/engines/engine.py.224: train: epoch 40 | batch 820/2304 [I 2022-10-27 01:23:27] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:13:44 [I 2022-10-27 01:23:27] ...trap/engines/engine.py.229: process: 0.54449 | load: 0.00250 [I 2022-10-27 01:23:27] ...trap/engines/engine.py.230: loss: 0.16752 [I 2022-10-27 01:23:27] ...optimizers/trijoint.py.151: total_norm: 31.886620 [I 2022-10-27 01:23:32] ...trap/engines/engine.py.224: train: epoch 40 | batch 830/2304 [I 2022-10-27 01:23:32] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:13:35 [I 2022-10-27 01:23:32] ...trap/engines/engine.py.229: process: 0.54433 | load: 0.00246 [I 2022-10-27 01:23:32] ...trap/engines/engine.py.230: loss: 0.14407 [I 2022-10-27 01:23:32] ...optimizers/trijoint.py.151: total_norm: 8.949467 [I 2022-10-27 01:23:38] ...trap/engines/engine.py.224: train: epoch 40 | batch 840/2304 [I 2022-10-27 01:23:38] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:34 [I 2022-10-27 01:23:38] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00246 [I 2022-10-27 01:23:38] ...trap/engines/engine.py.230: loss: 0.21295 [I 2022-10-27 01:23:38] ...optimizers/trijoint.py.151: total_norm: 15.043786 [I 2022-10-27 01:23:44] ...trap/engines/engine.py.224: train: epoch 40 | batch 850/2304 [I 2022-10-27 01:23:44] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:13:31 [I 2022-10-27 01:23:44] ...trap/engines/engine.py.229: process: 0.54126 | load: 0.00248 [I 2022-10-27 01:23:44] ...trap/engines/engine.py.230: loss: 0.12924 [I 2022-10-27 01:23:44] ...optimizers/trijoint.py.151: total_norm: 11.895494 [I 2022-10-27 01:23:49] ...trap/engines/engine.py.224: train: epoch 40 | batch 860/2304 [I 2022-10-27 01:23:49] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:13:23 [I 2022-10-27 01:23:49] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00250 [I 2022-10-27 01:23:49] ...trap/engines/engine.py.230: loss: 0.19112 [I 2022-10-27 01:23:49] ...optimizers/trijoint.py.151: total_norm: 18.281157 [I 2022-10-27 01:23:55] ...trap/engines/engine.py.224: train: epoch 40 | batch 870/2304 [I 2022-10-27 01:23:55] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:19 [I 2022-10-27 01:23:55] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00243 [I 2022-10-27 01:23:55] ...trap/engines/engine.py.230: loss: 0.19159 [I 2022-10-27 01:23:55] ...optimizers/trijoint.py.151: total_norm: 38.439808 [I 2022-10-27 01:24:00] ...trap/engines/engine.py.224: train: epoch 40 | batch 880/2304 [I 2022-10-27 01:24:00] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:13:11 [I 2022-10-27 01:24:00] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00244 [I 2022-10-27 01:24:00] ...trap/engines/engine.py.230: loss: 0.21511 [I 2022-10-27 01:24:00] ...optimizers/trijoint.py.151: total_norm: 26.940725 [I 2022-10-27 01:24:06] ...trap/engines/engine.py.224: train: epoch 40 | batch 890/2304 [I 2022-10-27 01:24:06] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:13:07 [I 2022-10-27 01:24:06] ...trap/engines/engine.py.229: process: 0.56472 | load: 0.00244 [I 2022-10-27 01:24:06] ...trap/engines/engine.py.230: loss: 0.21841 [I 2022-10-27 01:24:06] ...optimizers/trijoint.py.151: total_norm: 32.151318 [I 2022-10-27 01:24:12] ...trap/engines/engine.py.224: train: epoch 40 | batch 900/2304 [I 2022-10-27 01:24:12] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:13:06 [I 2022-10-27 01:24:12] ...trap/engines/engine.py.229: process: 0.55662 | load: 0.00239 [I 2022-10-27 01:24:12] ...trap/engines/engine.py.230: loss: 0.22065 [I 2022-10-27 01:24:12] ...optimizers/trijoint.py.151: total_norm: 15.692192 [I 2022-10-27 01:24:17] ...trap/engines/engine.py.224: train: epoch 40 | batch 910/2304 [I 2022-10-27 01:24:17] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:12:55 [I 2022-10-27 01:24:17] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00242 [I 2022-10-27 01:24:17] ...trap/engines/engine.py.230: loss: 0.19986 [I 2022-10-27 01:24:17] ...optimizers/trijoint.py.151: total_norm: 31.110102 [I 2022-10-27 01:24:23] ...trap/engines/engine.py.224: train: epoch 40 | batch 920/2304 [I 2022-10-27 01:24:23] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:12:56 [I 2022-10-27 01:24:23] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00248 [I 2022-10-27 01:24:23] ...trap/engines/engine.py.230: loss: 0.20508 [I 2022-10-27 01:24:23] ...optimizers/trijoint.py.151: total_norm: 12.990531 [I 2022-10-27 01:24:28] ...trap/engines/engine.py.224: train: epoch 40 | batch 930/2304 [I 2022-10-27 01:24:28] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:12:53 [I 2022-10-27 01:24:28] ...trap/engines/engine.py.229: process: 0.56808 | load: 0.00241 [I 2022-10-27 01:24:28] ...trap/engines/engine.py.230: loss: 0.22345 [I 2022-10-27 01:24:28] ...optimizers/trijoint.py.151: total_norm: 27.677448 [I 2022-10-27 01:24:34] ...trap/engines/engine.py.224: train: epoch 40 | batch 940/2304 [I 2022-10-27 01:24:34] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:46 [I 2022-10-27 01:24:34] ...trap/engines/engine.py.229: process: 0.56476 | load: 0.00240 [I 2022-10-27 01:24:34] ...trap/engines/engine.py.230: loss: 0.24544 [I 2022-10-27 01:24:34] ...optimizers/trijoint.py.151: total_norm: 23.783388 [I 2022-10-27 01:24:40] ...trap/engines/engine.py.224: train: epoch 40 | batch 950/2304 [I 2022-10-27 01:24:40] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:44 [I 2022-10-27 01:24:40] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00250 [I 2022-10-27 01:24:40] ...trap/engines/engine.py.230: loss: 0.12643 [I 2022-10-27 01:24:40] ...optimizers/trijoint.py.151: total_norm: 16.691027 [I 2022-10-27 01:24:45] ...trap/engines/engine.py.224: train: epoch 40 | batch 960/2304 [I 2022-10-27 01:24:45] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:12:26 [I 2022-10-27 01:24:45] ...trap/engines/engine.py.229: process: 0.54397 | load: 0.00245 [I 2022-10-27 01:24:45] ...trap/engines/engine.py.230: loss: 0.16451 [I 2022-10-27 01:24:45] ...optimizers/trijoint.py.151: total_norm: 12.096339 [I 2022-10-27 01:24:51] ...trap/engines/engine.py.224: train: epoch 40 | batch 970/2304 [I 2022-10-27 01:24:51] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:24 [I 2022-10-27 01:24:51] ...trap/engines/engine.py.229: process: 0.54734 | load: 0.00246 [I 2022-10-27 01:24:51] ...trap/engines/engine.py.230: loss: 0.24562 [I 2022-10-27 01:24:51] ...optimizers/trijoint.py.151: total_norm: 23.116764 [I 2022-10-27 01:24:56] ...trap/engines/engine.py.224: train: epoch 40 | batch 980/2304 [I 2022-10-27 01:24:56] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:10 [I 2022-10-27 01:24:56] ...trap/engines/engine.py.229: process: 0.54838 | load: 0.00244 [I 2022-10-27 01:24:56] ...trap/engines/engine.py.230: loss: 0.13515 [I 2022-10-27 01:24:56] ...optimizers/trijoint.py.151: total_norm: 16.199713 [I 2022-10-27 01:25:02] ...trap/engines/engine.py.224: train: epoch 40 | batch 990/2304 [I 2022-10-27 01:25:02] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:12:12 [I 2022-10-27 01:25:02] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00243 [I 2022-10-27 01:25:02] ...trap/engines/engine.py.230: loss: 0.13697 [I 2022-10-27 01:25:02] ...optimizers/trijoint.py.151: total_norm: 12.229625 [I 2022-10-27 01:25:08] ...trap/engines/engine.py.224: train: epoch 40 | batch 1000/2304 [I 2022-10-27 01:25:08] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:12:08 [I 2022-10-27 01:25:08] ...trap/engines/engine.py.229: process: 0.54856 | load: 0.00259 [I 2022-10-27 01:25:08] ...trap/engines/engine.py.230: loss: 0.15158 [I 2022-10-27 01:25:08] ...optimizers/trijoint.py.151: total_norm: 11.579884 [I 2022-10-27 01:25:13] ...trap/engines/engine.py.224: train: epoch 40 | batch 1010/2304 [I 2022-10-27 01:25:13] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:11:55 [I 2022-10-27 01:25:13] ...trap/engines/engine.py.229: process: 0.54629 | load: 0.00245 [I 2022-10-27 01:25:13] ...trap/engines/engine.py.230: loss: 0.19301 [I 2022-10-27 01:25:13] ...optimizers/trijoint.py.151: total_norm: 25.889465 [I 2022-10-27 01:25:19] ...trap/engines/engine.py.224: train: epoch 40 | batch 1020/2304 [I 2022-10-27 01:25:19] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:55 [I 2022-10-27 01:25:19] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00240 [I 2022-10-27 01:25:19] ...trap/engines/engine.py.230: loss: 0.27173 [I 2022-10-27 01:25:19] ...optimizers/trijoint.py.151: total_norm: 19.669292 [I 2022-10-27 01:25:24] ...trap/engines/engine.py.224: train: epoch 40 | batch 1030/2304 [I 2022-10-27 01:25:24] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:43 [I 2022-10-27 01:25:24] ...trap/engines/engine.py.229: process: 0.54316 | load: 0.00260 [I 2022-10-27 01:25:24] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-27 01:25:24] ...optimizers/trijoint.py.151: total_norm: 28.544891 [I 2022-10-27 01:25:30] ...trap/engines/engine.py.224: train: epoch 40 | batch 1040/2304 [I 2022-10-27 01:25:30] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:49 [I 2022-10-27 01:25:30] ...trap/engines/engine.py.229: process: 0.56446 | load: 0.00244 [I 2022-10-27 01:25:30] ...trap/engines/engine.py.230: loss: 0.15477 [I 2022-10-27 01:25:30] ...optimizers/trijoint.py.151: total_norm: 26.828463 [I 2022-10-27 01:25:36] ...trap/engines/engine.py.224: train: epoch 40 | batch 1050/2304 [I 2022-10-27 01:25:36] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:50 [I 2022-10-27 01:25:36] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00243 [I 2022-10-27 01:25:36] ...trap/engines/engine.py.230: loss: 0.16143 [I 2022-10-27 01:25:36] ...optimizers/trijoint.py.151: total_norm: 29.647175 [I 2022-10-27 01:25:41] ...trap/engines/engine.py.224: train: epoch 40 | batch 1060/2304 [I 2022-10-27 01:25:41] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:34 [I 2022-10-27 01:25:41] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00243 [I 2022-10-27 01:25:41] ...trap/engines/engine.py.230: loss: 0.17738 [I 2022-10-27 01:25:41] ...optimizers/trijoint.py.151: total_norm: 8.627552 [I 2022-10-27 01:25:47] ...trap/engines/engine.py.224: train: epoch 40 | batch 1070/2304 [I 2022-10-27 01:25:47] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:47 [I 2022-10-27 01:25:47] ...trap/engines/engine.py.229: process: 0.56879 | load: 0.00246 [I 2022-10-27 01:25:47] ...trap/engines/engine.py.230: loss: 0.17104 [I 2022-10-27 01:25:47] ...optimizers/trijoint.py.151: total_norm: 11.659749 [I 2022-10-27 01:25:52] ...trap/engines/engine.py.224: train: epoch 40 | batch 1080/2304 [I 2022-10-27 01:25:52] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:26 [I 2022-10-27 01:25:52] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00245 [I 2022-10-27 01:25:52] ...trap/engines/engine.py.230: loss: 0.13832 [I 2022-10-27 01:25:52] ...optimizers/trijoint.py.151: total_norm: 10.343196 [I 2022-10-27 01:25:58] ...trap/engines/engine.py.224: train: epoch 40 | batch 1090/2304 [I 2022-10-27 01:25:58] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:18 [I 2022-10-27 01:25:58] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00244 [I 2022-10-27 01:25:58] ...trap/engines/engine.py.230: loss: 0.21390 [I 2022-10-27 01:25:58] ...optimizers/trijoint.py.151: total_norm: 19.147684 [I 2022-10-27 01:26:04] ...trap/engines/engine.py.224: train: epoch 40 | batch 1100/2304 [I 2022-10-27 01:26:04] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:17 [I 2022-10-27 01:26:04] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00248 [I 2022-10-27 01:26:04] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-27 01:26:04] ...optimizers/trijoint.py.151: total_norm: 14.900786 [I 2022-10-27 01:26:09] ...trap/engines/engine.py.224: train: epoch 40 | batch 1110/2304 [I 2022-10-27 01:26:09] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:11:04 [I 2022-10-27 01:26:09] ...trap/engines/engine.py.229: process: 0.54474 | load: 0.00244 [I 2022-10-27 01:26:09] ...trap/engines/engine.py.230: loss: 0.23657 [I 2022-10-27 01:26:09] ...optimizers/trijoint.py.151: total_norm: 16.028217 [I 2022-10-27 01:26:15] ...trap/engines/engine.py.224: train: epoch 40 | batch 1120/2304 [I 2022-10-27 01:26:15] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:55 [I 2022-10-27 01:26:15] ...trap/engines/engine.py.229: process: 0.54733 | load: 0.00241 [I 2022-10-27 01:26:15] ...trap/engines/engine.py.230: loss: 0.20310 [I 2022-10-27 01:26:15] ...optimizers/trijoint.py.151: total_norm: 28.882694 [I 2022-10-27 01:26:21] ...trap/engines/engine.py.224: train: epoch 40 | batch 1130/2304 [I 2022-10-27 01:26:21] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:11:24 [I 2022-10-27 01:26:21] ...trap/engines/engine.py.229: process: 0.62766 | load: 0.00270 [I 2022-10-27 01:26:21] ...trap/engines/engine.py.230: loss: 0.15397 [I 2022-10-27 01:26:21] ...optimizers/trijoint.py.151: total_norm: 16.028320 [I 2022-10-27 01:26:26] ...trap/engines/engine.py.224: train: epoch 40 | batch 1140/2304 [I 2022-10-27 01:26:26] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:52 [I 2022-10-27 01:26:26] ...trap/engines/engine.py.229: process: 0.55013 | load: 0.00241 [I 2022-10-27 01:26:26] ...trap/engines/engine.py.230: loss: 0.13924 [I 2022-10-27 01:26:26] ...optimizers/trijoint.py.151: total_norm: 11.449268 [I 2022-10-27 01:26:32] ...trap/engines/engine.py.224: train: epoch 40 | batch 1150/2304 [I 2022-10-27 01:26:32] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:38 [I 2022-10-27 01:26:32] ...trap/engines/engine.py.229: process: 0.54801 | load: 0.00240 [I 2022-10-27 01:26:32] ...trap/engines/engine.py.230: loss: 0.22085 [I 2022-10-27 01:26:32] ...optimizers/trijoint.py.151: total_norm: 15.985775 [I 2022-10-27 01:26:37] ...trap/engines/engine.py.224: train: epoch 40 | batch 1160/2304 [I 2022-10-27 01:26:37] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:37 [I 2022-10-27 01:26:37] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00243 [I 2022-10-27 01:26:37] ...trap/engines/engine.py.230: loss: 0.17590 [I 2022-10-27 01:26:37] ...optimizers/trijoint.py.151: total_norm: 12.940084 [I 2022-10-27 01:26:43] ...trap/engines/engine.py.224: train: epoch 40 | batch 1170/2304 [I 2022-10-27 01:26:43] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:30 [I 2022-10-27 01:26:43] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00257 [I 2022-10-27 01:26:43] ...trap/engines/engine.py.230: loss: 0.13645 [I 2022-10-27 01:26:43] ...optimizers/trijoint.py.151: total_norm: 9.520549 [I 2022-10-27 01:26:48] ...trap/engines/engine.py.224: train: epoch 40 | batch 1180/2304 [I 2022-10-27 01:26:48] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:27 [I 2022-10-27 01:26:48] ...trap/engines/engine.py.229: process: 0.56410 | load: 0.00241 [I 2022-10-27 01:26:48] ...trap/engines/engine.py.230: loss: 0.17346 [I 2022-10-27 01:26:48] ...optimizers/trijoint.py.151: total_norm: 17.059940 [I 2022-10-27 01:26:54] ...trap/engines/engine.py.224: train: epoch 40 | batch 1190/2304 [I 2022-10-27 01:26:54] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:22 [I 2022-10-27 01:26:54] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00242 [I 2022-10-27 01:26:54] ...trap/engines/engine.py.230: loss: 0.19742 [I 2022-10-27 01:26:54] ...optimizers/trijoint.py.151: total_norm: 130.158081 [I 2022-10-27 01:27:00] ...trap/engines/engine.py.224: train: epoch 40 | batch 1200/2304 [I 2022-10-27 01:27:00] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:19 [I 2022-10-27 01:27:00] ...trap/engines/engine.py.229: process: 0.54627 | load: 0.00243 [I 2022-10-27 01:27:00] ...trap/engines/engine.py.230: loss: 0.21041 [I 2022-10-27 01:27:00] ...optimizers/trijoint.py.151: total_norm: 17.121531 [I 2022-10-27 01:27:05] ...trap/engines/engine.py.224: train: epoch 40 | batch 1210/2304 [I 2022-10-27 01:27:05] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:10:15 [I 2022-10-27 01:27:05] ...trap/engines/engine.py.229: process: 0.56514 | load: 0.00244 [I 2022-10-27 01:27:05] ...trap/engines/engine.py.230: loss: 0.13136 [I 2022-10-27 01:27:05] ...optimizers/trijoint.py.151: total_norm: 20.363297 [I 2022-10-27 01:27:11] ...trap/engines/engine.py.224: train: epoch 40 | batch 1220/2304 [I 2022-10-27 01:27:11] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:10:03 [I 2022-10-27 01:27:11] ...trap/engines/engine.py.229: process: 0.55387 | load: 0.00243 [I 2022-10-27 01:27:11] ...trap/engines/engine.py.230: loss: 0.23675 [I 2022-10-27 01:27:11] ...optimizers/trijoint.py.151: total_norm: 23.439501 [I 2022-10-27 01:27:17] ...trap/engines/engine.py.224: train: epoch 40 | batch 1230/2304 [I 2022-10-27 01:27:17] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:59 [I 2022-10-27 01:27:17] ...trap/engines/engine.py.229: process: 0.55583 | load: 0.00240 [I 2022-10-27 01:27:17] ...trap/engines/engine.py.230: loss: 0.15974 [I 2022-10-27 01:27:17] ...optimizers/trijoint.py.151: total_norm: 13.736112 [I 2022-10-27 01:27:22] ...trap/engines/engine.py.224: train: epoch 40 | batch 1240/2304 [I 2022-10-27 01:27:22] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:52 [I 2022-10-27 01:27:22] ...trap/engines/engine.py.229: process: 0.54581 | load: 0.00250 [I 2022-10-27 01:27:22] ...trap/engines/engine.py.230: loss: 0.17299 [I 2022-10-27 01:27:22] ...optimizers/trijoint.py.151: total_norm: 15.154415 [I 2022-10-27 01:27:28] ...trap/engines/engine.py.224: train: epoch 40 | batch 1250/2304 [I 2022-10-27 01:27:28] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:44 [I 2022-10-27 01:27:28] ...trap/engines/engine.py.229: process: 0.55235 | load: 0.00253 [I 2022-10-27 01:27:28] ...trap/engines/engine.py.230: loss: 0.25610 [I 2022-10-27 01:27:28] ...optimizers/trijoint.py.151: total_norm: 28.039967 [I 2022-10-27 01:27:33] ...trap/engines/engine.py.224: train: epoch 40 | batch 1260/2304 [I 2022-10-27 01:27:33] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:50 [I 2022-10-27 01:27:33] ...trap/engines/engine.py.229: process: 0.56791 | load: 0.00246 [I 2022-10-27 01:27:33] ...trap/engines/engine.py.230: loss: 0.23470 [I 2022-10-27 01:27:33] ...optimizers/trijoint.py.151: total_norm: 33.184418 [I 2022-10-27 01:27:39] ...trap/engines/engine.py.224: train: epoch 40 | batch 1270/2304 [I 2022-10-27 01:27:39] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:35 [I 2022-10-27 01:27:39] ...trap/engines/engine.py.229: process: 0.55161 | load: 0.00247 [I 2022-10-27 01:27:39] ...trap/engines/engine.py.230: loss: 0.20530 [I 2022-10-27 01:27:39] ...optimizers/trijoint.py.151: total_norm: 17.013130 [I 2022-10-27 01:27:45] ...trap/engines/engine.py.224: train: epoch 40 | batch 1280/2304 [I 2022-10-27 01:27:45] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:31 [I 2022-10-27 01:27:45] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00250 [I 2022-10-27 01:27:45] ...trap/engines/engine.py.230: loss: 0.16626 [I 2022-10-27 01:27:45] ...optimizers/trijoint.py.151: total_norm: 16.661226 [I 2022-10-27 01:27:50] ...trap/engines/engine.py.224: train: epoch 40 | batch 1290/2304 [I 2022-10-27 01:27:50] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:23 [I 2022-10-27 01:27:50] ...trap/engines/engine.py.229: process: 0.55347 | load: 0.00243 [I 2022-10-27 01:27:50] ...trap/engines/engine.py.230: loss: 0.19139 [I 2022-10-27 01:27:50] ...optimizers/trijoint.py.151: total_norm: 15.146454 [I 2022-10-27 01:27:56] ...trap/engines/engine.py.224: train: epoch 40 | batch 1300/2304 [I 2022-10-27 01:27:56] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:18 [I 2022-10-27 01:27:56] ...trap/engines/engine.py.229: process: 0.56355 | load: 0.00242 [I 2022-10-27 01:27:56] ...trap/engines/engine.py.230: loss: 0.14202 [I 2022-10-27 01:27:56] ...optimizers/trijoint.py.151: total_norm: 20.028452 [I 2022-10-27 01:28:01] ...trap/engines/engine.py.224: train: epoch 40 | batch 1310/2304 [I 2022-10-27 01:28:01] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:19 [I 2022-10-27 01:28:01] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00264 [I 2022-10-27 01:28:01] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-27 01:28:01] ...optimizers/trijoint.py.151: total_norm: 13.361519 [I 2022-10-27 01:28:07] ...trap/engines/engine.py.224: train: epoch 40 | batch 1320/2304 [I 2022-10-27 01:28:07] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:09 [I 2022-10-27 01:28:07] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00241 [I 2022-10-27 01:28:07] ...trap/engines/engine.py.230: loss: 0.25528 [I 2022-10-27 01:28:07] ...optimizers/trijoint.py.151: total_norm: 19.267143 [I 2022-10-27 01:28:13] ...trap/engines/engine.py.224: train: epoch 40 | batch 1330/2304 [I 2022-10-27 01:28:13] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:09:02 [I 2022-10-27 01:28:13] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00245 [I 2022-10-27 01:28:13] ...trap/engines/engine.py.230: loss: 0.18944 [I 2022-10-27 01:28:13] ...optimizers/trijoint.py.151: total_norm: 24.129095 [I 2022-10-27 01:28:18] ...trap/engines/engine.py.224: train: epoch 40 | batch 1340/2304 [I 2022-10-27 01:28:18] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:09:06 [I 2022-10-27 01:28:18] ...trap/engines/engine.py.229: process: 0.56416 | load: 0.00277 [I 2022-10-27 01:28:18] ...trap/engines/engine.py.230: loss: 0.28363 [I 2022-10-27 01:28:18] ...optimizers/trijoint.py.151: total_norm: 25.998789 [I 2022-10-27 01:28:24] ...trap/engines/engine.py.224: train: epoch 40 | batch 1350/2304 [I 2022-10-27 01:28:24] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:53 [I 2022-10-27 01:28:24] ...trap/engines/engine.py.229: process: 0.54671 | load: 0.00243 [I 2022-10-27 01:28:24] ...trap/engines/engine.py.230: loss: 0.16449 [I 2022-10-27 01:28:24] ...optimizers/trijoint.py.151: total_norm: 19.122915 [I 2022-10-27 01:28:29] ...trap/engines/engine.py.224: train: epoch 40 | batch 1360/2304 [I 2022-10-27 01:28:29] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:50 [I 2022-10-27 01:28:29] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00245 [I 2022-10-27 01:28:29] ...trap/engines/engine.py.230: loss: 0.22418 [I 2022-10-27 01:28:29] ...optimizers/trijoint.py.151: total_norm: 14.776217 [I 2022-10-27 01:28:35] ...trap/engines/engine.py.224: train: epoch 40 | batch 1370/2304 [I 2022-10-27 01:28:35] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:36 [I 2022-10-27 01:28:35] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00243 [I 2022-10-27 01:28:35] ...trap/engines/engine.py.230: loss: 0.17177 [I 2022-10-27 01:28:35] ...optimizers/trijoint.py.151: total_norm: 15.832424 [I 2022-10-27 01:28:41] ...trap/engines/engine.py.224: train: epoch 40 | batch 1380/2304 [I 2022-10-27 01:28:41] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:34 [I 2022-10-27 01:28:41] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00246 [I 2022-10-27 01:28:41] ...trap/engines/engine.py.230: loss: 0.14425 [I 2022-10-27 01:28:41] ...optimizers/trijoint.py.151: total_norm: 10.947547 [I 2022-10-27 01:28:46] ...trap/engines/engine.py.224: train: epoch 40 | batch 1390/2304 [I 2022-10-27 01:28:46] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:36 [I 2022-10-27 01:28:46] ...trap/engines/engine.py.229: process: 0.57138 | load: 0.00246 [I 2022-10-27 01:28:46] ...trap/engines/engine.py.230: loss: 0.17013 [I 2022-10-27 01:28:46] ...optimizers/trijoint.py.151: total_norm: 15.459274 [I 2022-10-27 01:28:52] ...trap/engines/engine.py.224: train: epoch 40 | batch 1400/2304 [I 2022-10-27 01:28:52] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:24 [I 2022-10-27 01:28:52] ...trap/engines/engine.py.229: process: 0.55138 | load: 0.00244 [I 2022-10-27 01:28:52] ...trap/engines/engine.py.230: loss: 0.17740 [I 2022-10-27 01:28:52] ...optimizers/trijoint.py.151: total_norm: 24.350355 [I 2022-10-27 01:28:57] ...trap/engines/engine.py.224: train: epoch 40 | batch 1410/2304 [I 2022-10-27 01:28:57] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:17 [I 2022-10-27 01:28:57] ...trap/engines/engine.py.229: process: 0.54799 | load: 0.00245 [I 2022-10-27 01:28:57] ...trap/engines/engine.py.230: loss: 0.21675 [I 2022-10-27 01:28:57] ...optimizers/trijoint.py.151: total_norm: 19.696054 [I 2022-10-27 01:29:03] ...trap/engines/engine.py.224: train: epoch 40 | batch 1420/2304 [I 2022-10-27 01:29:03] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:11 [I 2022-10-27 01:29:03] ...trap/engines/engine.py.229: process: 0.54942 | load: 0.00243 [I 2022-10-27 01:29:03] ...trap/engines/engine.py.230: loss: 0.14652 [I 2022-10-27 01:29:03] ...optimizers/trijoint.py.151: total_norm: 21.889441 [I 2022-10-27 01:29:09] ...trap/engines/engine.py.224: train: epoch 40 | batch 1430/2304 [I 2022-10-27 01:29:09] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:08:02 [I 2022-10-27 01:29:09] ...trap/engines/engine.py.229: process: 0.54630 | load: 0.00241 [I 2022-10-27 01:29:09] ...trap/engines/engine.py.230: loss: 0.12961 [I 2022-10-27 01:29:09] ...optimizers/trijoint.py.151: total_norm: 22.985439 [I 2022-10-27 01:29:14] ...trap/engines/engine.py.224: train: epoch 40 | batch 1440/2304 [I 2022-10-27 01:29:14] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:08:00 [I 2022-10-27 01:29:14] ...trap/engines/engine.py.229: process: 0.54916 | load: 0.00243 [I 2022-10-27 01:29:14] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-27 01:29:14] ...optimizers/trijoint.py.151: total_norm: 20.739113 [I 2022-10-27 01:29:20] ...trap/engines/engine.py.224: train: epoch 40 | batch 1450/2304 [I 2022-10-27 01:29:20] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:08:10 [I 2022-10-27 01:29:20] ...trap/engines/engine.py.229: process: 0.65230 | load: 0.00292 [I 2022-10-27 01:29:20] ...trap/engines/engine.py.230: loss: 0.15578 [I 2022-10-27 01:29:20] ...optimizers/trijoint.py.151: total_norm: 10.831314 [I 2022-10-27 01:29:25] ...trap/engines/engine.py.224: train: epoch 40 | batch 1460/2304 [I 2022-10-27 01:29:25] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:53 [I 2022-10-27 01:29:25] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00244 [I 2022-10-27 01:29:25] ...trap/engines/engine.py.230: loss: 0.16150 [I 2022-10-27 01:29:25] ...optimizers/trijoint.py.151: total_norm: 19.468470 [I 2022-10-27 01:29:31] ...trap/engines/engine.py.224: train: epoch 40 | batch 1470/2304 [I 2022-10-27 01:29:31] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:45 [I 2022-10-27 01:29:31] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00250 [I 2022-10-27 01:29:31] ...trap/engines/engine.py.230: loss: 0.12659 [I 2022-10-27 01:29:31] ...optimizers/trijoint.py.151: total_norm: 26.346861 [I 2022-10-27 01:29:37] ...trap/engines/engine.py.224: train: epoch 40 | batch 1480/2304 [I 2022-10-27 01:29:37] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:37 [I 2022-10-27 01:29:37] ...trap/engines/engine.py.229: process: 0.54191 | load: 0.00251 [I 2022-10-27 01:29:37] ...trap/engines/engine.py.230: loss: 0.17681 [I 2022-10-27 01:29:37] ...optimizers/trijoint.py.151: total_norm: 14.839733 [I 2022-10-27 01:29:42] ...trap/engines/engine.py.224: train: epoch 40 | batch 1490/2304 [I 2022-10-27 01:29:42] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:34 [I 2022-10-27 01:29:42] ...trap/engines/engine.py.229: process: 0.55727 | load: 0.00246 [I 2022-10-27 01:29:42] ...trap/engines/engine.py.230: loss: 0.23007 [I 2022-10-27 01:29:42] ...optimizers/trijoint.py.151: total_norm: 25.180862 [I 2022-10-27 01:29:48] ...trap/engines/engine.py.224: train: epoch 40 | batch 1500/2304 [I 2022-10-27 01:29:48] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:28 [I 2022-10-27 01:29:48] ...trap/engines/engine.py.229: process: 0.55347 | load: 0.00243 [I 2022-10-27 01:29:48] ...trap/engines/engine.py.230: loss: 0.17701 [I 2022-10-27 01:29:48] ...optimizers/trijoint.py.151: total_norm: 23.545305 [I 2022-10-27 01:29:53] ...trap/engines/engine.py.224: train: epoch 40 | batch 1510/2304 [I 2022-10-27 01:29:53] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:24 [I 2022-10-27 01:29:53] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00245 [I 2022-10-27 01:29:53] ...trap/engines/engine.py.230: loss: 0.16998 [I 2022-10-27 01:29:53] ...optimizers/trijoint.py.151: total_norm: 19.374304 [I 2022-10-27 01:29:59] ...trap/engines/engine.py.224: train: epoch 40 | batch 1520/2304 [I 2022-10-27 01:29:59] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:21 [I 2022-10-27 01:29:59] ...trap/engines/engine.py.229: process: 0.56300 | load: 0.00264 [I 2022-10-27 01:29:59] ...trap/engines/engine.py.230: loss: 0.15215 [I 2022-10-27 01:29:59] ...optimizers/trijoint.py.151: total_norm: 16.446384 [I 2022-10-27 01:30:05] ...trap/engines/engine.py.224: train: epoch 40 | batch 1530/2304 [I 2022-10-27 01:30:05] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:07:12 [I 2022-10-27 01:30:05] ...trap/engines/engine.py.229: process: 0.54956 | load: 0.00242 [I 2022-10-27 01:30:05] ...trap/engines/engine.py.230: loss: 0.13482 [I 2022-10-27 01:30:05] ...optimizers/trijoint.py.151: total_norm: 10.983131 [I 2022-10-27 01:30:10] ...trap/engines/engine.py.224: train: epoch 40 | batch 1540/2304 [I 2022-10-27 01:30:10] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:07:06 [I 2022-10-27 01:30:10] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00248 [I 2022-10-27 01:30:10] ...trap/engines/engine.py.230: loss: 0.13055 [I 2022-10-27 01:30:10] ...optimizers/trijoint.py.151: total_norm: 19.081087 [I 2022-10-27 01:30:16] ...trap/engines/engine.py.224: train: epoch 40 | batch 1550/2304 [I 2022-10-27 01:30:16] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:06:59 [I 2022-10-27 01:30:16] ...trap/engines/engine.py.229: process: 0.54926 | load: 0.00247 [I 2022-10-27 01:30:16] ...trap/engines/engine.py.230: loss: 0.14010 [I 2022-10-27 01:30:16] ...optimizers/trijoint.py.151: total_norm: 13.406511 [I 2022-10-27 01:30:21] ...trap/engines/engine.py.224: train: epoch 40 | batch 1560/2304 [I 2022-10-27 01:30:21] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:57 [I 2022-10-27 01:30:21] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00249 [I 2022-10-27 01:30:21] ...trap/engines/engine.py.230: loss: 0.13836 [I 2022-10-27 01:30:21] ...optimizers/trijoint.py.151: total_norm: 16.952410 [I 2022-10-27 01:30:27] ...trap/engines/engine.py.224: train: epoch 40 | batch 1570/2304 [I 2022-10-27 01:30:27] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:51 [I 2022-10-27 01:30:27] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00246 [I 2022-10-27 01:30:27] ...trap/engines/engine.py.230: loss: 0.16878 [I 2022-10-27 01:30:27] ...optimizers/trijoint.py.151: total_norm: 18.936287 [I 2022-10-27 01:30:33] ...trap/engines/engine.py.224: train: epoch 40 | batch 1580/2304 [I 2022-10-27 01:30:33] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:47 [I 2022-10-27 01:30:33] ...trap/engines/engine.py.229: process: 0.53795 | load: 0.00248 [I 2022-10-27 01:30:33] ...trap/engines/engine.py.230: loss: 0.10519 [I 2022-10-27 01:30:33] ...optimizers/trijoint.py.151: total_norm: 7.249398 [I 2022-10-27 01:30:38] ...trap/engines/engine.py.224: train: epoch 40 | batch 1590/2304 [I 2022-10-27 01:30:38] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:42 [I 2022-10-27 01:30:38] ...trap/engines/engine.py.229: process: 0.57291 | load: 0.00246 [I 2022-10-27 01:30:38] ...trap/engines/engine.py.230: loss: 0.22054 [I 2022-10-27 01:30:38] ...optimizers/trijoint.py.151: total_norm: 26.509190 [I 2022-10-27 01:30:44] ...trap/engines/engine.py.224: train: epoch 40 | batch 1600/2304 [I 2022-10-27 01:30:44] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:35 [I 2022-10-27 01:30:44] ...trap/engines/engine.py.229: process: 0.55524 | load: 0.00243 [I 2022-10-27 01:30:44] ...trap/engines/engine.py.230: loss: 0.13945 [I 2022-10-27 01:30:44] ...optimizers/trijoint.py.151: total_norm: 14.572700 [I 2022-10-27 01:30:50] ...trap/engines/engine.py.224: train: epoch 40 | batch 1610/2304 [I 2022-10-27 01:30:50] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:27 [I 2022-10-27 01:30:50] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00244 [I 2022-10-27 01:30:50] ...trap/engines/engine.py.230: loss: 0.16685 [I 2022-10-27 01:30:50] ...optimizers/trijoint.py.151: total_norm: 13.740663 [I 2022-10-27 01:30:55] ...trap/engines/engine.py.224: train: epoch 40 | batch 1620/2304 [I 2022-10-27 01:30:55] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:18 [I 2022-10-27 01:30:55] ...trap/engines/engine.py.229: process: 0.54971 | load: 0.00242 [I 2022-10-27 01:30:55] ...trap/engines/engine.py.230: loss: 0.17400 [I 2022-10-27 01:30:55] ...optimizers/trijoint.py.151: total_norm: 19.778028 [I 2022-10-27 01:31:01] ...trap/engines/engine.py.224: train: epoch 40 | batch 1630/2304 [I 2022-10-27 01:31:01] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:06:15 [I 2022-10-27 01:31:01] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00244 [I 2022-10-27 01:31:01] ...trap/engines/engine.py.230: loss: 0.14163 [I 2022-10-27 01:31:01] ...optimizers/trijoint.py.151: total_norm: 12.345838 [I 2022-10-27 01:31:06] ...trap/engines/engine.py.224: train: epoch 40 | batch 1640/2304 [I 2022-10-27 01:31:06] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:06:08 [I 2022-10-27 01:31:06] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00243 [I 2022-10-27 01:31:06] ...trap/engines/engine.py.230: loss: 0.14734 [I 2022-10-27 01:31:06] ...optimizers/trijoint.py.151: total_norm: 11.537921 [I 2022-10-27 01:31:12] ...trap/engines/engine.py.224: train: epoch 40 | batch 1650/2304 [I 2022-10-27 01:31:12] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:06:05 [I 2022-10-27 01:31:12] ...trap/engines/engine.py.229: process: 0.55081 | load: 0.00244 [I 2022-10-27 01:31:12] ...trap/engines/engine.py.230: loss: 0.12554 [I 2022-10-27 01:31:12] ...optimizers/trijoint.py.151: total_norm: 15.106909 [I 2022-10-27 01:31:18] ...trap/engines/engine.py.224: train: epoch 40 | batch 1660/2304 [I 2022-10-27 01:31:18] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:59 [I 2022-10-27 01:31:18] ...trap/engines/engine.py.229: process: 0.55043 | load: 0.00250 [I 2022-10-27 01:31:18] ...trap/engines/engine.py.230: loss: 0.15025 [I 2022-10-27 01:31:18] ...optimizers/trijoint.py.151: total_norm: 17.801865 [I 2022-10-27 01:31:23] ...trap/engines/engine.py.224: train: epoch 40 | batch 1670/2304 [I 2022-10-27 01:31:23] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:56 [I 2022-10-27 01:31:23] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00246 [I 2022-10-27 01:31:23] ...trap/engines/engine.py.230: loss: 0.16400 [I 2022-10-27 01:31:23] ...optimizers/trijoint.py.151: total_norm: 17.904823 [I 2022-10-27 01:31:29] ...trap/engines/engine.py.224: train: epoch 40 | batch 1680/2304 [I 2022-10-27 01:31:29] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:48 [I 2022-10-27 01:31:29] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00254 [I 2022-10-27 01:31:29] ...trap/engines/engine.py.230: loss: 0.13149 [I 2022-10-27 01:31:29] ...optimizers/trijoint.py.151: total_norm: 11.920258 [I 2022-10-27 01:31:34] ...trap/engines/engine.py.224: train: epoch 40 | batch 1690/2304 [I 2022-10-27 01:31:34] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:43 [I 2022-10-27 01:31:34] ...trap/engines/engine.py.229: process: 0.56039 | load: 0.00245 [I 2022-10-27 01:31:34] ...trap/engines/engine.py.230: loss: 0.18251 [I 2022-10-27 01:31:34] ...optimizers/trijoint.py.151: total_norm: 12.506905 [I 2022-10-27 01:31:40] ...trap/engines/engine.py.224: train: epoch 40 | batch 1700/2304 [I 2022-10-27 01:31:40] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:37 [I 2022-10-27 01:31:40] ...trap/engines/engine.py.229: process: 0.56728 | load: 0.00253 [I 2022-10-27 01:31:40] ...trap/engines/engine.py.230: loss: 0.15659 [I 2022-10-27 01:31:40] ...optimizers/trijoint.py.151: total_norm: 9.976618 [I 2022-10-27 01:31:46] ...trap/engines/engine.py.224: train: epoch 40 | batch 1710/2304 [I 2022-10-27 01:31:46] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:33 [I 2022-10-27 01:31:46] ...trap/engines/engine.py.229: process: 0.56535 | load: 0.00248 [I 2022-10-27 01:31:46] ...trap/engines/engine.py.230: loss: 0.16007 [I 2022-10-27 01:31:46] ...optimizers/trijoint.py.151: total_norm: 24.051350 [I 2022-10-27 01:31:51] ...trap/engines/engine.py.224: train: epoch 40 | batch 1720/2304 [I 2022-10-27 01:31:51] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:24 [I 2022-10-27 01:31:51] ...trap/engines/engine.py.229: process: 0.54730 | load: 0.00245 [I 2022-10-27 01:31:51] ...trap/engines/engine.py.230: loss: 0.11274 [I 2022-10-27 01:31:51] ...optimizers/trijoint.py.151: total_norm: 11.636668 [I 2022-10-27 01:31:57] ...trap/engines/engine.py.224: train: epoch 40 | batch 1730/2304 [I 2022-10-27 01:31:57] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:19 [I 2022-10-27 01:31:57] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00241 [I 2022-10-27 01:31:57] ...trap/engines/engine.py.230: loss: 0.18001 [I 2022-10-27 01:31:57] ...optimizers/trijoint.py.151: total_norm: 10.779585 [I 2022-10-27 01:32:02] ...trap/engines/engine.py.224: train: epoch 40 | batch 1740/2304 [I 2022-10-27 01:32:02] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:05:15 [I 2022-10-27 01:32:02] ...trap/engines/engine.py.229: process: 0.56127 | load: 0.00243 [I 2022-10-27 01:32:02] ...trap/engines/engine.py.230: loss: 0.20093 [I 2022-10-27 01:32:02] ...optimizers/trijoint.py.151: total_norm: 22.415981 [I 2022-10-27 01:32:08] ...trap/engines/engine.py.224: train: epoch 40 | batch 1750/2304 [I 2022-10-27 01:32:08] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:11 [I 2022-10-27 01:32:08] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00245 [I 2022-10-27 01:32:08] ...trap/engines/engine.py.230: loss: 0.17592 [I 2022-10-27 01:32:08] ...optimizers/trijoint.py.151: total_norm: 22.192778 [I 2022-10-27 01:32:14] ...trap/engines/engine.py.224: train: epoch 40 | batch 1760/2304 [I 2022-10-27 01:32:14] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:05:06 [I 2022-10-27 01:32:14] ...trap/engines/engine.py.229: process: 0.55539 | load: 0.00247 [I 2022-10-27 01:32:14] ...trap/engines/engine.py.230: loss: 0.17267 [I 2022-10-27 01:32:14] ...optimizers/trijoint.py.151: total_norm: 16.841604 [I 2022-10-27 01:32:19] ...trap/engines/engine.py.224: train: epoch 40 | batch 1770/2304 [I 2022-10-27 01:32:19] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:57 [I 2022-10-27 01:32:19] ...trap/engines/engine.py.229: process: 0.54888 | load: 0.00246 [I 2022-10-27 01:32:19] ...trap/engines/engine.py.230: loss: 0.18328 [I 2022-10-27 01:32:19] ...optimizers/trijoint.py.151: total_norm: 13.831361 [I 2022-10-27 01:32:25] ...trap/engines/engine.py.224: train: epoch 40 | batch 1780/2304 [I 2022-10-27 01:32:25] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:52 [I 2022-10-27 01:32:25] ...trap/engines/engine.py.229: process: 0.56117 | load: 0.00249 [I 2022-10-27 01:32:25] ...trap/engines/engine.py.230: loss: 0.18156 [I 2022-10-27 01:32:25] ...optimizers/trijoint.py.151: total_norm: 16.970568 [I 2022-10-27 01:32:30] ...trap/engines/engine.py.224: train: epoch 40 | batch 1790/2304 [I 2022-10-27 01:32:30] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:47 [I 2022-10-27 01:32:30] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00242 [I 2022-10-27 01:32:30] ...trap/engines/engine.py.230: loss: 0.21637 [I 2022-10-27 01:32:30] ...optimizers/trijoint.py.151: total_norm: 30.449909 [I 2022-10-27 01:32:36] ...trap/engines/engine.py.224: train: epoch 40 | batch 1800/2304 [I 2022-10-27 01:32:36] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:42 [I 2022-10-27 01:32:36] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00244 [I 2022-10-27 01:32:36] ...trap/engines/engine.py.230: loss: 0.20962 [I 2022-10-27 01:32:36] ...optimizers/trijoint.py.151: total_norm: 21.123392 [I 2022-10-27 01:32:42] ...trap/engines/engine.py.224: train: epoch 40 | batch 1810/2304 [I 2022-10-27 01:32:42] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:38 [I 2022-10-27 01:32:42] ...trap/engines/engine.py.229: process: 0.58901 | load: 0.00246 [I 2022-10-27 01:32:42] ...trap/engines/engine.py.230: loss: 0.22895 [I 2022-10-27 01:32:42] ...optimizers/trijoint.py.151: total_norm: 20.432777 [I 2022-10-27 01:32:47] ...trap/engines/engine.py.224: train: epoch 40 | batch 1820/2304 [I 2022-10-27 01:32:47] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:30 [I 2022-10-27 01:32:47] ...trap/engines/engine.py.229: process: 0.56593 | load: 0.00243 [I 2022-10-27 01:32:47] ...trap/engines/engine.py.230: loss: 0.16660 [I 2022-10-27 01:32:47] ...optimizers/trijoint.py.151: total_norm: 16.096600 [I 2022-10-27 01:32:53] ...trap/engines/engine.py.224: train: epoch 40 | batch 1830/2304 [I 2022-10-27 01:32:53] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:24 [I 2022-10-27 01:32:53] ...trap/engines/engine.py.229: process: 0.55881 | load: 0.00245 [I 2022-10-27 01:32:53] ...trap/engines/engine.py.230: loss: 0.15072 [I 2022-10-27 01:32:53] ...optimizers/trijoint.py.151: total_norm: 13.782861 [I 2022-10-27 01:32:59] ...trap/engines/engine.py.224: train: epoch 40 | batch 1840/2304 [I 2022-10-27 01:32:59] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:20 [I 2022-10-27 01:32:59] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00243 [I 2022-10-27 01:32:59] ...trap/engines/engine.py.230: loss: 0.14942 [I 2022-10-27 01:32:59] ...optimizers/trijoint.py.151: total_norm: 19.859198 [I 2022-10-27 01:33:04] ...trap/engines/engine.py.224: train: epoch 40 | batch 1850/2304 [I 2022-10-27 01:33:04] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:04:13 [I 2022-10-27 01:33:04] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00249 [I 2022-10-27 01:33:04] ...trap/engines/engine.py.230: loss: 0.18500 [I 2022-10-27 01:33:04] ...optimizers/trijoint.py.151: total_norm: 18.489845 [I 2022-10-27 01:33:10] ...trap/engines/engine.py.224: train: epoch 40 | batch 1860/2304 [I 2022-10-27 01:33:10] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:04:08 [I 2022-10-27 01:33:10] ...trap/engines/engine.py.229: process: 0.56585 | load: 0.00278 [I 2022-10-27 01:33:10] ...trap/engines/engine.py.230: loss: 0.15751 [I 2022-10-27 01:33:10] ...optimizers/trijoint.py.151: total_norm: 29.304604 [I 2022-10-27 01:33:15] ...trap/engines/engine.py.224: train: epoch 40 | batch 1870/2304 [I 2022-10-27 01:33:15] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:04:02 [I 2022-10-27 01:33:15] ...trap/engines/engine.py.229: process: 0.56102 | load: 0.00244 [I 2022-10-27 01:33:15] ...trap/engines/engine.py.230: loss: 0.16537 [I 2022-10-27 01:33:15] ...optimizers/trijoint.py.151: total_norm: 7.835804 [I 2022-10-27 01:33:21] ...trap/engines/engine.py.224: train: epoch 40 | batch 1880/2304 [I 2022-10-27 01:33:21] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:58 [I 2022-10-27 01:33:21] ...trap/engines/engine.py.229: process: 0.56874 | load: 0.00248 [I 2022-10-27 01:33:21] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-27 01:33:21] ...optimizers/trijoint.py.151: total_norm: 14.435734 [I 2022-10-27 01:33:27] ...trap/engines/engine.py.224: train: epoch 40 | batch 1890/2304 [I 2022-10-27 01:33:27] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:53 [I 2022-10-27 01:33:27] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00245 [I 2022-10-27 01:33:27] ...trap/engines/engine.py.230: loss: 0.18480 [I 2022-10-27 01:33:27] ...optimizers/trijoint.py.151: total_norm: 15.546157 [I 2022-10-27 01:33:32] ...trap/engines/engine.py.224: train: epoch 40 | batch 1900/2304 [I 2022-10-27 01:33:32] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:45 [I 2022-10-27 01:33:32] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00241 [I 2022-10-27 01:33:32] ...trap/engines/engine.py.230: loss: 0.22979 [I 2022-10-27 01:33:32] ...optimizers/trijoint.py.151: total_norm: 22.996664 [I 2022-10-27 01:33:38] ...trap/engines/engine.py.224: train: epoch 40 | batch 1910/2304 [I 2022-10-27 01:33:38] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:41 [I 2022-10-27 01:33:38] ...trap/engines/engine.py.229: process: 0.58229 | load: 0.00250 [I 2022-10-27 01:33:38] ...trap/engines/engine.py.230: loss: 0.16772 [I 2022-10-27 01:33:38] ...optimizers/trijoint.py.151: total_norm: 15.083998 [I 2022-10-27 01:33:43] ...trap/engines/engine.py.224: train: epoch 40 | batch 1920/2304 [I 2022-10-27 01:33:43] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:36 [I 2022-10-27 01:33:43] ...trap/engines/engine.py.229: process: 0.55325 | load: 0.00246 [I 2022-10-27 01:33:43] ...trap/engines/engine.py.230: loss: 0.16163 [I 2022-10-27 01:33:43] ...optimizers/trijoint.py.151: total_norm: 23.405207 [I 2022-10-27 01:33:49] ...trap/engines/engine.py.224: train: epoch 40 | batch 1930/2304 [I 2022-10-27 01:33:49] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:28 [I 2022-10-27 01:33:49] ...trap/engines/engine.py.229: process: 0.55950 | load: 0.00248 [I 2022-10-27 01:33:49] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-27 01:33:49] ...optimizers/trijoint.py.151: total_norm: 19.428558 [I 2022-10-27 01:33:55] ...trap/engines/engine.py.224: train: epoch 40 | batch 1940/2304 [I 2022-10-27 01:33:55] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:23 [I 2022-10-27 01:33:55] ...trap/engines/engine.py.229: process: 0.55531 | load: 0.00245 [I 2022-10-27 01:33:55] ...trap/engines/engine.py.230: loss: 0.14115 [I 2022-10-27 01:33:55] ...optimizers/trijoint.py.151: total_norm: 8.994965 [I 2022-10-27 01:34:00] ...trap/engines/engine.py.224: train: epoch 40 | batch 1950/2304 [I 2022-10-27 01:34:00] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:03:17 [I 2022-10-27 01:34:00] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00252 [I 2022-10-27 01:34:00] ...trap/engines/engine.py.230: loss: 0.16052 [I 2022-10-27 01:34:00] ...optimizers/trijoint.py.151: total_norm: 14.104752 [I 2022-10-27 01:34:06] ...trap/engines/engine.py.224: train: epoch 40 | batch 1960/2304 [I 2022-10-27 01:34:06] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:03:11 [I 2022-10-27 01:34:06] ...trap/engines/engine.py.229: process: 0.54606 | load: 0.00245 [I 2022-10-27 01:34:06] ...trap/engines/engine.py.230: loss: 0.26447 [I 2022-10-27 01:34:06] ...optimizers/trijoint.py.151: total_norm: 30.538570 [I 2022-10-27 01:34:11] ...trap/engines/engine.py.224: train: epoch 40 | batch 1970/2304 [I 2022-10-27 01:34:11] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:03:05 [I 2022-10-27 01:34:11] ...trap/engines/engine.py.229: process: 0.56593 | load: 0.00251 [I 2022-10-27 01:34:11] ...trap/engines/engine.py.230: loss: 0.13248 [I 2022-10-27 01:34:11] ...optimizers/trijoint.py.151: total_norm: 9.404349 [I 2022-10-27 01:34:17] ...trap/engines/engine.py.224: train: epoch 40 | batch 1980/2304 [I 2022-10-27 01:34:17] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:03:00 [I 2022-10-27 01:34:17] ...trap/engines/engine.py.229: process: 0.54952 | load: 0.00243 [I 2022-10-27 01:34:17] ...trap/engines/engine.py.230: loss: 0.16791 [I 2022-10-27 01:34:17] ...optimizers/trijoint.py.151: total_norm: 18.401384 [I 2022-10-27 01:34:23] ...trap/engines/engine.py.224: train: epoch 40 | batch 1990/2304 [I 2022-10-27 01:34:23] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:54 [I 2022-10-27 01:34:23] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00244 [I 2022-10-27 01:34:23] ...trap/engines/engine.py.230: loss: 0.21413 [I 2022-10-27 01:34:23] ...optimizers/trijoint.py.151: total_norm: 27.647699 [I 2022-10-27 01:34:28] ...trap/engines/engine.py.224: train: epoch 40 | batch 2000/2304 [I 2022-10-27 01:34:28] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:49 [I 2022-10-27 01:34:28] ...trap/engines/engine.py.229: process: 0.56492 | load: 0.00244 [I 2022-10-27 01:34:28] ...trap/engines/engine.py.230: loss: 0.16058 [I 2022-10-27 01:34:28] ...optimizers/trijoint.py.151: total_norm: 17.180819 [I 2022-10-27 01:34:34] ...trap/engines/engine.py.224: train: epoch 40 | batch 2010/2304 [I 2022-10-27 01:34:34] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:43 [I 2022-10-27 01:34:34] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00258 [I 2022-10-27 01:34:34] ...trap/engines/engine.py.230: loss: 0.20084 [I 2022-10-27 01:34:34] ...optimizers/trijoint.py.151: total_norm: 14.672409 [I 2022-10-27 01:34:39] ...trap/engines/engine.py.224: train: epoch 40 | batch 2020/2304 [I 2022-10-27 01:34:39] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:37 [I 2022-10-27 01:34:39] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00256 [I 2022-10-27 01:34:39] ...trap/engines/engine.py.230: loss: 0.16182 [I 2022-10-27 01:34:39] ...optimizers/trijoint.py.151: total_norm: 26.430990 [I 2022-10-27 01:34:45] ...trap/engines/engine.py.224: train: epoch 40 | batch 2030/2304 [I 2022-10-27 01:34:45] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:33 [I 2022-10-27 01:34:45] ...trap/engines/engine.py.229: process: 0.57140 | load: 0.00247 [I 2022-10-27 01:34:45] ...trap/engines/engine.py.230: loss: 0.19599 [I 2022-10-27 01:34:45] ...optimizers/trijoint.py.151: total_norm: 30.084356 [I 2022-10-27 01:34:50] ...trap/engines/engine.py.224: train: epoch 40 | batch 2040/2304 [I 2022-10-27 01:34:50] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:27 [I 2022-10-27 01:34:50] ...trap/engines/engine.py.229: process: 0.56688 | load: 0.00246 [I 2022-10-27 01:34:50] ...trap/engines/engine.py.230: loss: 0.14773 [I 2022-10-27 01:34:50] ...optimizers/trijoint.py.151: total_norm: 20.279985 [I 2022-10-27 01:34:56] ...trap/engines/engine.py.224: train: epoch 40 | batch 2050/2304 [I 2022-10-27 01:34:56] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:21 [I 2022-10-27 01:34:56] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00245 [I 2022-10-27 01:34:56] ...trap/engines/engine.py.230: loss: 0.18747 [I 2022-10-27 01:34:56] ...optimizers/trijoint.py.151: total_norm: 18.382746 [I 2022-10-27 01:35:02] ...trap/engines/engine.py.224: train: epoch 40 | batch 2060/2304 [I 2022-10-27 01:35:02] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:02:16 [I 2022-10-27 01:35:02] ...trap/engines/engine.py.229: process: 0.56565 | load: 0.00244 [I 2022-10-27 01:35:02] ...trap/engines/engine.py.230: loss: 0.12203 [I 2022-10-27 01:35:02] ...optimizers/trijoint.py.151: total_norm: 9.926373 [I 2022-10-27 01:35:07] ...trap/engines/engine.py.224: train: epoch 40 | batch 2070/2304 [I 2022-10-27 01:35:07] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:02:11 [I 2022-10-27 01:35:07] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00246 [I 2022-10-27 01:35:07] ...trap/engines/engine.py.230: loss: 0.14101 [I 2022-10-27 01:35:07] ...optimizers/trijoint.py.151: total_norm: 10.209906 [I 2022-10-27 01:35:13] ...trap/engines/engine.py.224: train: epoch 40 | batch 2080/2304 [I 2022-10-27 01:35:13] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:02:05 [I 2022-10-27 01:35:13] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00246 [I 2022-10-27 01:35:13] ...trap/engines/engine.py.230: loss: 0.16676 [I 2022-10-27 01:35:13] ...optimizers/trijoint.py.151: total_norm: 23.159227 [I 2022-10-27 01:35:18] ...trap/engines/engine.py.224: train: epoch 40 | batch 2090/2304 [I 2022-10-27 01:35:18] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:59 [I 2022-10-27 01:35:18] ...trap/engines/engine.py.229: process: 0.55941 | load: 0.00240 [I 2022-10-27 01:35:18] ...trap/engines/engine.py.230: loss: 0.16862 [I 2022-10-27 01:35:18] ...optimizers/trijoint.py.151: total_norm: 18.619247 [I 2022-10-27 01:35:24] ...trap/engines/engine.py.224: train: epoch 40 | batch 2100/2304 [I 2022-10-27 01:35:24] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:53 [I 2022-10-27 01:35:24] ...trap/engines/engine.py.229: process: 0.55106 | load: 0.00247 [I 2022-10-27 01:35:24] ...trap/engines/engine.py.230: loss: 0.12178 [I 2022-10-27 01:35:24] ...optimizers/trijoint.py.151: total_norm: 13.128738 [I 2022-10-27 01:35:29] ...trap/engines/engine.py.224: train: epoch 40 | batch 2110/2304 [I 2022-10-27 01:35:29] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:47 [I 2022-10-27 01:35:29] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00248 [I 2022-10-27 01:35:29] ...trap/engines/engine.py.230: loss: 0.11639 [I 2022-10-27 01:35:29] ...optimizers/trijoint.py.151: total_norm: 6.155101 [I 2022-10-27 01:35:35] ...trap/engines/engine.py.224: train: epoch 40 | batch 2120/2304 [I 2022-10-27 01:35:35] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:45 [I 2022-10-27 01:35:35] ...trap/engines/engine.py.229: process: 0.59050 | load: 0.00257 [I 2022-10-27 01:35:35] ...trap/engines/engine.py.230: loss: 0.13760 [I 2022-10-27 01:35:35] ...optimizers/trijoint.py.151: total_norm: 21.679234 [I 2022-10-27 01:35:41] ...trap/engines/engine.py.224: train: epoch 40 | batch 2130/2304 [I 2022-10-27 01:35:41] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:37 [I 2022-10-27 01:35:41] ...trap/engines/engine.py.229: process: 0.55642 | load: 0.00247 [I 2022-10-27 01:35:41] ...trap/engines/engine.py.230: loss: 0.17047 [I 2022-10-27 01:35:41] ...optimizers/trijoint.py.151: total_norm: 12.948647 [I 2022-10-27 01:35:46] ...trap/engines/engine.py.224: train: epoch 40 | batch 2140/2304 [I 2022-10-27 01:35:46] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:31 [I 2022-10-27 01:35:46] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00244 [I 2022-10-27 01:35:46] ...trap/engines/engine.py.230: loss: 0.15451 [I 2022-10-27 01:35:46] ...optimizers/trijoint.py.151: total_norm: 34.233742 [I 2022-10-27 01:35:52] ...trap/engines/engine.py.224: train: epoch 40 | batch 2150/2304 [I 2022-10-27 01:35:52] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:26 [I 2022-10-27 01:35:52] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00242 [I 2022-10-27 01:35:52] ...trap/engines/engine.py.230: loss: 0.28512 [I 2022-10-27 01:35:52] ...optimizers/trijoint.py.151: total_norm: 30.592138 [I 2022-10-27 01:35:58] ...trap/engines/engine.py.224: train: epoch 40 | batch 2160/2304 [I 2022-10-27 01:35:58] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:19 [I 2022-10-27 01:35:58] ...trap/engines/engine.py.229: process: 0.54786 | load: 0.00245 [I 2022-10-27 01:35:58] ...trap/engines/engine.py.230: loss: 0.11427 [I 2022-10-27 01:35:58] ...optimizers/trijoint.py.151: total_norm: 41.062054 [I 2022-10-27 01:36:03] ...trap/engines/engine.py.224: train: epoch 40 | batch 2170/2304 [I 2022-10-27 01:36:03] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:01:14 [I 2022-10-27 01:36:03] ...trap/engines/engine.py.229: process: 0.55555 | load: 0.00245 [I 2022-10-27 01:36:03] ...trap/engines/engine.py.230: loss: 0.15456 [I 2022-10-27 01:36:03] ...optimizers/trijoint.py.151: total_norm: 19.957298 [I 2022-10-27 01:36:09] ...trap/engines/engine.py.224: train: epoch 40 | batch 2180/2304 [I 2022-10-27 01:36:09] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:08 [I 2022-10-27 01:36:09] ...trap/engines/engine.py.229: process: 0.55372 | load: 0.00241 [I 2022-10-27 01:36:09] ...trap/engines/engine.py.230: loss: 0.12088 [I 2022-10-27 01:36:09] ...optimizers/trijoint.py.151: total_norm: 10.378080 [I 2022-10-27 01:36:14] ...trap/engines/engine.py.224: train: epoch 40 | batch 2190/2304 [I 2022-10-27 01:36:14] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:01:03 [I 2022-10-27 01:36:14] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00244 [I 2022-10-27 01:36:14] ...trap/engines/engine.py.230: loss: 0.30208 [I 2022-10-27 01:36:14] ...optimizers/trijoint.py.151: total_norm: 17.018839 [I 2022-10-27 01:36:20] ...trap/engines/engine.py.224: train: epoch 40 | batch 2200/2304 [I 2022-10-27 01:36:20] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:57 [I 2022-10-27 01:36:20] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00242 [I 2022-10-27 01:36:20] ...trap/engines/engine.py.230: loss: 0.15801 [I 2022-10-27 01:36:20] ...optimizers/trijoint.py.151: total_norm: 20.076860 [I 2022-10-27 01:36:25] ...trap/engines/engine.py.224: train: epoch 40 | batch 2210/2304 [I 2022-10-27 01:36:25] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:52 [I 2022-10-27 01:36:25] ...trap/engines/engine.py.229: process: 0.54675 | load: 0.00242 [I 2022-10-27 01:36:25] ...trap/engines/engine.py.230: loss: 0.16665 [I 2022-10-27 01:36:25] ...optimizers/trijoint.py.151: total_norm: 13.702971 [I 2022-10-27 01:36:31] ...trap/engines/engine.py.224: train: epoch 40 | batch 2220/2304 [I 2022-10-27 01:36:31] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:47 [I 2022-10-27 01:36:31] ...trap/engines/engine.py.229: process: 0.56917 | load: 0.00244 [I 2022-10-27 01:36:31] ...trap/engines/engine.py.230: loss: 0.17963 [I 2022-10-27 01:36:31] ...optimizers/trijoint.py.151: total_norm: 15.730715 [I 2022-10-27 01:36:37] ...trap/engines/engine.py.224: train: epoch 40 | batch 2230/2304 [I 2022-10-27 01:36:37] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:41 [I 2022-10-27 01:36:37] ...trap/engines/engine.py.229: process: 0.55063 | load: 0.00253 [I 2022-10-27 01:36:37] ...trap/engines/engine.py.230: loss: 0.19834 [I 2022-10-27 01:36:37] ...optimizers/trijoint.py.151: total_norm: 34.410885 [I 2022-10-27 01:36:42] ...trap/engines/engine.py.224: train: epoch 40 | batch 2240/2304 [I 2022-10-27 01:36:42] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:35 [I 2022-10-27 01:36:42] ...trap/engines/engine.py.229: process: 0.56331 | load: 0.00247 [I 2022-10-27 01:36:42] ...trap/engines/engine.py.230: loss: 0.19720 [I 2022-10-27 01:36:42] ...optimizers/trijoint.py.151: total_norm: 17.473793 [I 2022-10-27 01:36:48] ...trap/engines/engine.py.224: train: epoch 40 | batch 2250/2304 [I 2022-10-27 01:36:48] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:29 [I 2022-10-27 01:36:48] ...trap/engines/engine.py.229: process: 0.54151 | load: 0.00244 [I 2022-10-27 01:36:48] ...trap/engines/engine.py.230: loss: 0.21703 [I 2022-10-27 01:36:48] ...optimizers/trijoint.py.151: total_norm: 25.151695 [I 2022-10-27 01:36:53] ...trap/engines/engine.py.224: train: epoch 40 | batch 2260/2304 [I 2022-10-27 01:36:53] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:24 [I 2022-10-27 01:36:53] ...trap/engines/engine.py.229: process: 0.54793 | load: 0.00244 [I 2022-10-27 01:36:53] ...trap/engines/engine.py.230: loss: 0.17150 [I 2022-10-27 01:36:53] ...optimizers/trijoint.py.151: total_norm: 7.930744 [I 2022-10-27 01:36:59] ...trap/engines/engine.py.224: train: epoch 40 | batch 2270/2304 [I 2022-10-27 01:36:59] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:18 [I 2022-10-27 01:36:59] ...trap/engines/engine.py.229: process: 0.55259 | load: 0.00243 [I 2022-10-27 01:36:59] ...trap/engines/engine.py.230: loss: 0.13063 [I 2022-10-27 01:36:59] ...optimizers/trijoint.py.151: total_norm: 17.643911 [I 2022-10-27 01:37:05] ...trap/engines/engine.py.224: train: epoch 40 | batch 2280/2304 [I 2022-10-27 01:37:05] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:13 [I 2022-10-27 01:37:05] ...trap/engines/engine.py.229: process: 0.56080 | load: 0.00241 [I 2022-10-27 01:37:05] ...trap/engines/engine.py.230: loss: 0.20769 [I 2022-10-27 01:37:05] ...optimizers/trijoint.py.151: total_norm: 18.613853 [I 2022-10-27 01:37:10] ...trap/engines/engine.py.224: train: epoch 40 | batch 2290/2304 [I 2022-10-27 01:37:10] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:00:07 [I 2022-10-27 01:37:10] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00243 [I 2022-10-27 01:37:10] ...trap/engines/engine.py.230: loss: 0.14901 [I 2022-10-27 01:37:10] ...optimizers/trijoint.py.151: total_norm: 10.165250 [I 2022-10-27 01:37:16] ...trap/engines/engine.py.224: train: epoch 40 | batch 2300/2304 [I 2022-10-27 01:37:16] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:02 [I 2022-10-27 01:37:16] ...trap/engines/engine.py.229: process: 0.55014 | load: 0.00016 [I 2022-10-27 01:37:16] ...trap/engines/engine.py.230: loss: 0.15963 [I 2022-10-27 01:37:16] ...optimizers/trijoint.py.151: total_norm: 20.217621 [I 2022-10-27 01:37:18] ...trap/engines/engine.py.224: train: epoch 40 | batch 2304/2304 [I 2022-10-27 01:37:18] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:00:00 [I 2022-10-27 01:37:18] ...trap/engines/engine.py.229: process: 0.55069 | load: 0.00017 [I 2022-10-27 01:37:18] ...trap/engines/engine.py.230: loss: 0.19039 [I 2022-10-27 01:37:18] ...optimizers/trijoint.py.151: total_norm: 15.112459 [S 2022-10-27 01:37:19] ...trap/engines/engine.py.129: train_epoch.epoch: 40 [S 2022-10-27 01:37:19] ...trap/engines/engine.py.129: train_epoch.loss: 0.17067846419883653 [I 2022-10-27 01:37:19] ...s/criterions/triplet.py.84: margin = 0.25500000000000017 [I 2022-10-27 01:37:20] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 40 [I 2022-10-27 01:37:20] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 01:37:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 01:37:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 01:37:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 01:37:23] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 01:37:23] ...trap/engines/engine.py.321: eval: epoch 40 | batch 0/510 [I 2022-10-27 01:37:23] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:29:52 [I 2022-10-27 01:37:23] ...trap/engines/engine.py.326: process: 3.51535 | load: 3.12321 [I 2022-10-27 01:37:27] ...trap/engines/engine.py.321: eval: epoch 40 | batch 10/510 [I 2022-10-27 01:37:27] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:36 [I 2022-10-27 01:37:27] ...trap/engines/engine.py.326: process: 0.31982 | load: 0.00017 [I 2022-10-27 01:37:30] ...trap/engines/engine.py.321: eval: epoch 40 | batch 20/510 [I 2022-10-27 01:37:30] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:02 [I 2022-10-27 01:37:30] ...trap/engines/engine.py.326: process: 0.32218 | load: 0.00029 [I 2022-10-27 01:37:33] ...trap/engines/engine.py.321: eval: epoch 40 | batch 30/510 [I 2022-10-27 01:37:33] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:45 [I 2022-10-27 01:37:33] ...trap/engines/engine.py.326: process: 0.36323 | load: 0.00018 [I 2022-10-27 01:37:37] ...trap/engines/engine.py.321: eval: epoch 40 | batch 40/510 [I 2022-10-27 01:37:37] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:38 [I 2022-10-27 01:37:37] ...trap/engines/engine.py.326: process: 0.33506 | load: 0.00018 [I 2022-10-27 01:37:40] ...trap/engines/engine.py.321: eval: epoch 40 | batch 50/510 [I 2022-10-27 01:37:40] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:34 [I 2022-10-27 01:37:40] ...trap/engines/engine.py.326: process: 0.32480 | load: 0.00018 [I 2022-10-27 01:37:44] ...trap/engines/engine.py.321: eval: epoch 40 | batch 60/510 [I 2022-10-27 01:37:44] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:33 [I 2022-10-27 01:37:44] ...trap/engines/engine.py.326: process: 0.37410 | load: 0.00017 [I 2022-10-27 01:37:47] ...trap/engines/engine.py.321: eval: epoch 40 | batch 70/510 [I 2022-10-27 01:37:47] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:26 [I 2022-10-27 01:37:47] ...trap/engines/engine.py.326: process: 0.33636 | load: 0.00016 [I 2022-10-27 01:37:50] ...trap/engines/engine.py.321: eval: epoch 40 | batch 80/510 [I 2022-10-27 01:37:50] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:22 [I 2022-10-27 01:37:50] ...trap/engines/engine.py.326: process: 0.32849 | load: 0.00017 [I 2022-10-27 01:37:54] ...trap/engines/engine.py.321: eval: epoch 40 | batch 90/510 [I 2022-10-27 01:37:54] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:21 [I 2022-10-27 01:37:54] ...trap/engines/engine.py.326: process: 0.35707 | load: 0.00017 [I 2022-10-27 01:37:57] ...trap/engines/engine.py.321: eval: epoch 40 | batch 100/510 [I 2022-10-27 01:37:57] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:13 [I 2022-10-27 01:37:57] ...trap/engines/engine.py.326: process: 0.29135 | load: 0.00024 [I 2022-10-27 01:38:00] ...trap/engines/engine.py.321: eval: epoch 40 | batch 110/510 [I 2022-10-27 01:38:00] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-27 01:38:00] ...trap/engines/engine.py.326: process: 0.28719 | load: 0.00015 [I 2022-10-27 01:38:03] ...trap/engines/engine.py.321: eval: epoch 40 | batch 120/510 [I 2022-10-27 01:38:03] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-27 01:38:03] ...trap/engines/engine.py.326: process: 0.28414 | load: 0.00020 [I 2022-10-27 01:38:06] ...trap/engines/engine.py.321: eval: epoch 40 | batch 130/510 [I 2022-10-27 01:38:06] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:56 [I 2022-10-27 01:38:06] ...trap/engines/engine.py.326: process: 0.28754 | load: 0.00019 [I 2022-10-27 01:38:09] ...trap/engines/engine.py.321: eval: epoch 40 | batch 140/510 [I 2022-10-27 01:38:09] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-27 01:38:09] ...trap/engines/engine.py.326: process: 0.29018 | load: 0.00019 [I 2022-10-27 01:38:11] ...trap/engines/engine.py.321: eval: epoch 40 | batch 150/510 [I 2022-10-27 01:38:11] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-27 01:38:11] ...trap/engines/engine.py.326: process: 0.28417 | load: 0.00019 [I 2022-10-27 01:38:14] ...trap/engines/engine.py.321: eval: epoch 40 | batch 160/510 [I 2022-10-27 01:38:14] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-27 01:38:14] ...trap/engines/engine.py.326: process: 0.28228 | load: 0.00016 [I 2022-10-27 01:38:17] ...trap/engines/engine.py.321: eval: epoch 40 | batch 170/510 [I 2022-10-27 01:38:17] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-27 01:38:17] ...trap/engines/engine.py.326: process: 0.28854 | load: 0.00023 [I 2022-10-27 01:38:20] ...trap/engines/engine.py.321: eval: epoch 40 | batch 180/510 [I 2022-10-27 01:38:20] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:36 [I 2022-10-27 01:38:20] ...trap/engines/engine.py.326: process: 0.29091 | load: 0.00023 [I 2022-10-27 01:38:23] ...trap/engines/engine.py.321: eval: epoch 40 | batch 190/510 [I 2022-10-27 01:38:23] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 01:38:23] ...trap/engines/engine.py.326: process: 0.29164 | load: 0.00015 [I 2022-10-27 01:38:26] ...trap/engines/engine.py.321: eval: epoch 40 | batch 200/510 [I 2022-10-27 01:38:26] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 01:38:26] ...trap/engines/engine.py.326: process: 0.28507 | load: 0.00020 [I 2022-10-27 01:38:29] ...trap/engines/engine.py.321: eval: epoch 40 | batch 210/510 [I 2022-10-27 01:38:29] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:27 [I 2022-10-27 01:38:29] ...trap/engines/engine.py.326: process: 0.29244 | load: 0.00018 [I 2022-10-27 01:38:32] ...trap/engines/engine.py.321: eval: epoch 40 | batch 220/510 [I 2022-10-27 01:38:32] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-27 01:38:32] ...trap/engines/engine.py.326: process: 0.29475 | load: 0.00023 [I 2022-10-27 01:38:35] ...trap/engines/engine.py.321: eval: epoch 40 | batch 230/510 [I 2022-10-27 01:38:35] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-27 01:38:35] ...trap/engines/engine.py.326: process: 0.29299 | load: 0.00017 [I 2022-10-27 01:38:38] ...trap/engines/engine.py.321: eval: epoch 40 | batch 240/510 [I 2022-10-27 01:38:38] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-27 01:38:38] ...trap/engines/engine.py.326: process: 0.28921 | load: 0.00022 [I 2022-10-27 01:38:40] ...trap/engines/engine.py.321: eval: epoch 40 | batch 250/510 [I 2022-10-27 01:38:40] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-27 01:38:40] ...trap/engines/engine.py.326: process: 0.28944 | load: 0.00018 [I 2022-10-27 01:38:43] ...trap/engines/engine.py.321: eval: epoch 40 | batch 260/510 [I 2022-10-27 01:38:43] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 01:38:43] ...trap/engines/engine.py.326: process: 0.28596 | load: 0.00017 [I 2022-10-27 01:38:46] ...trap/engines/engine.py.321: eval: epoch 40 | batch 270/510 [I 2022-10-27 01:38:46] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-27 01:38:46] ...trap/engines/engine.py.326: process: 0.28822 | load: 0.00018 [I 2022-10-27 01:38:49] ...trap/engines/engine.py.321: eval: epoch 40 | batch 280/510 [I 2022-10-27 01:38:49] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 01:38:49] ...trap/engines/engine.py.326: process: 0.28855 | load: 0.00016 [I 2022-10-27 01:38:52] ...trap/engines/engine.py.321: eval: epoch 40 | batch 290/510 [I 2022-10-27 01:38:52] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 01:38:52] ...trap/engines/engine.py.326: process: 0.28685 | load: 0.00018 [I 2022-10-27 01:38:55] ...trap/engines/engine.py.321: eval: epoch 40 | batch 300/510 [I 2022-10-27 01:38:55] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 01:38:55] ...trap/engines/engine.py.326: process: 0.29208 | load: 0.00014 [I 2022-10-27 01:38:58] ...trap/engines/engine.py.321: eval: epoch 40 | batch 310/510 [I 2022-10-27 01:38:58] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 01:38:58] ...trap/engines/engine.py.326: process: 0.29132 | load: 0.00018 [I 2022-10-27 01:39:01] ...trap/engines/engine.py.321: eval: epoch 40 | batch 320/510 [I 2022-10-27 01:39:01] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 01:39:01] ...trap/engines/engine.py.326: process: 0.28649 | load: 0.00024 [I 2022-10-27 01:39:04] ...trap/engines/engine.py.321: eval: epoch 40 | batch 330/510 [I 2022-10-27 01:39:04] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 01:39:04] ...trap/engines/engine.py.326: process: 0.29004 | load: 0.00015 [I 2022-10-27 01:39:07] ...trap/engines/engine.py.321: eval: epoch 40 | batch 340/510 [I 2022-10-27 01:39:07] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:49 [I 2022-10-27 01:39:07] ...trap/engines/engine.py.326: process: 0.28405 | load: 0.00020 [I 2022-10-27 01:39:09] ...trap/engines/engine.py.321: eval: epoch 40 | batch 350/510 [I 2022-10-27 01:39:09] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-27 01:39:09] ...trap/engines/engine.py.326: process: 0.29316 | load: 0.00023 [I 2022-10-27 01:39:12] ...trap/engines/engine.py.321: eval: epoch 40 | batch 360/510 [I 2022-10-27 01:39:12] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 01:39:12] ...trap/engines/engine.py.326: process: 0.30101 | load: 0.00016 [I 2022-10-27 01:39:15] ...trap/engines/engine.py.321: eval: epoch 40 | batch 370/510 [I 2022-10-27 01:39:15] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:41 [I 2022-10-27 01:39:15] ...trap/engines/engine.py.326: process: 0.30396 | load: 0.00019 [I 2022-10-27 01:39:18] ...trap/engines/engine.py.321: eval: epoch 40 | batch 380/510 [I 2022-10-27 01:39:18] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 01:39:18] ...trap/engines/engine.py.326: process: 0.29140 | load: 0.00015 [I 2022-10-27 01:39:21] ...trap/engines/engine.py.321: eval: epoch 40 | batch 390/510 [I 2022-10-27 01:39:21] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:35 [I 2022-10-27 01:39:21] ...trap/engines/engine.py.326: process: 0.29008 | load: 0.00018 [I 2022-10-27 01:39:24] ...trap/engines/engine.py.321: eval: epoch 40 | batch 400/510 [I 2022-10-27 01:39:24] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 01:39:24] ...trap/engines/engine.py.326: process: 0.29027 | load: 0.00046 [I 2022-10-27 01:39:27] ...trap/engines/engine.py.321: eval: epoch 40 | batch 410/510 [I 2022-10-27 01:39:27] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 01:39:27] ...trap/engines/engine.py.326: process: 0.29171 | load: 0.00020 [I 2022-10-27 01:39:30] ...trap/engines/engine.py.321: eval: epoch 40 | batch 420/510 [I 2022-10-27 01:39:30] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:26 [I 2022-10-27 01:39:30] ...trap/engines/engine.py.326: process: 0.29014 | load: 0.00015 [I 2022-10-27 01:39:33] ...trap/engines/engine.py.321: eval: epoch 40 | batch 430/510 [I 2022-10-27 01:39:33] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-27 01:39:33] ...trap/engines/engine.py.326: process: 0.28740 | load: 0.00018 [I 2022-10-27 01:39:36] ...trap/engines/engine.py.321: eval: epoch 40 | batch 440/510 [I 2022-10-27 01:39:36] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 01:39:36] ...trap/engines/engine.py.326: process: 0.28974 | load: 0.00017 [I 2022-10-27 01:39:39] ...trap/engines/engine.py.321: eval: epoch 40 | batch 450/510 [I 2022-10-27 01:39:39] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 01:39:39] ...trap/engines/engine.py.326: process: 0.28517 | load: 0.00021 [I 2022-10-27 01:39:41] ...trap/engines/engine.py.321: eval: epoch 40 | batch 460/510 [I 2022-10-27 01:39:41] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 01:39:41] ...trap/engines/engine.py.326: process: 0.29203 | load: 0.00016 [I 2022-10-27 01:39:44] ...trap/engines/engine.py.321: eval: epoch 40 | batch 470/510 [I 2022-10-27 01:39:44] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 01:39:44] ...trap/engines/engine.py.326: process: 0.28837 | load: 0.00019 [I 2022-10-27 01:39:47] ...trap/engines/engine.py.321: eval: epoch 40 | batch 480/510 [I 2022-10-27 01:39:47] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 01:39:47] ...trap/engines/engine.py.326: process: 0.28997 | load: 0.00016 [I 2022-10-27 01:39:50] ...trap/engines/engine.py.321: eval: epoch 40 | batch 490/510 [I 2022-10-27 01:39:50] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 01:39:50] ...trap/engines/engine.py.326: process: 0.29137 | load: 0.00018 [I 2022-10-27 01:39:53] ...trap/engines/engine.py.321: eval: epoch 40 | batch 500/510 [I 2022-10-27 01:39:53] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 01:39:53] ...trap/engines/engine.py.326: process: 0.28055 | load: 0.00014 [I 2022-10-27 01:39:56] ...trap/engines/engine.py.321: eval: epoch 40 | batch 510/510 [I 2022-10-27 01:39:56] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 01:39:56] ...trap/engines/engine.py.326: process: 0.28459 | load: 0.00011 [S 2022-10-27 01:39:57] ...trap/engines/engine.py.132: eval_epoch.epoch: 40 [I 2022-10-27 01:39:57] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.686500 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.897300 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933800 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.676600 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.893700 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.929000 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.018326 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011917 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008506 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011952 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008013 [S 2022-10-27 01:40:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009539 [I 2022-10-27 01:40:02] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 01:40:02] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 01:40:02] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 01:40:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 01:40:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 01:40:04] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 01:40:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 01:40:12] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 01:40:12] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 01:40:12] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 01:40:15] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 01:40:21] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 01:40:21] ...trap/engines/engine.py.163: Training model on trainset for epoch 41 [I 2022-10-27 01:40:21] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 01:40:23] ...trap/engines/engine.py.224: train: epoch 41 | batch 0/2304 [I 2022-10-27 01:40:23] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:39:07 [I 2022-10-27 01:40:23] ...trap/engines/engine.py.229: process: 2.58140 | load: 1.96596 [I 2022-10-27 01:40:23] ...trap/engines/engine.py.230: loss: 0.16552 [I 2022-10-27 01:40:23] ...optimizers/trijoint.py.151: total_norm: 23.887398 [I 2022-10-27 01:40:29] ...trap/engines/engine.py.224: train: epoch 41 | batch 10/2304 [I 2022-10-27 01:40:29] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:32 [I 2022-10-27 01:40:29] ...trap/engines/engine.py.229: process: 0.54884 | load: 0.00454 [I 2022-10-27 01:40:29] ...trap/engines/engine.py.230: loss: 0.14781 [I 2022-10-27 01:40:29] ...optimizers/trijoint.py.151: total_norm: 23.704962 [I 2022-10-27 01:40:34] ...trap/engines/engine.py.224: train: epoch 41 | batch 20/2304 [I 2022-10-27 01:40:34] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:07 [I 2022-10-27 01:40:34] ...trap/engines/engine.py.229: process: 0.56331 | load: 0.00345 [I 2022-10-27 01:40:34] ...trap/engines/engine.py.230: loss: 0.18141 [I 2022-10-27 01:40:34] ...optimizers/trijoint.py.151: total_norm: 11.053868 [I 2022-10-27 01:40:40] ...trap/engines/engine.py.224: train: epoch 41 | batch 30/2304 [I 2022-10-27 01:40:40] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:04 [I 2022-10-27 01:40:40] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00297 [I 2022-10-27 01:40:40] ...trap/engines/engine.py.230: loss: 0.14649 [I 2022-10-27 01:40:40] ...optimizers/trijoint.py.151: total_norm: 13.804669 [I 2022-10-27 01:40:46] ...trap/engines/engine.py.224: train: epoch 41 | batch 40/2304 [I 2022-10-27 01:40:46] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:21:19 [I 2022-10-27 01:40:46] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00273 [I 2022-10-27 01:40:46] ...trap/engines/engine.py.230: loss: 0.13350 [I 2022-10-27 01:40:46] ...optimizers/trijoint.py.151: total_norm: 14.803306 [I 2022-10-27 01:40:51] ...trap/engines/engine.py.224: train: epoch 41 | batch 50/2304 [I 2022-10-27 01:40:51] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:02 [I 2022-10-27 01:40:51] ...trap/engines/engine.py.229: process: 0.55462 | load: 0.00252 [I 2022-10-27 01:40:51] ...trap/engines/engine.py.230: loss: 0.13354 [I 2022-10-27 01:40:51] ...optimizers/trijoint.py.151: total_norm: 19.372507 [I 2022-10-27 01:40:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 60/2304 [I 2022-10-27 01:40:57] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:49 [I 2022-10-27 01:40:57] ...trap/engines/engine.py.229: process: 0.55227 | load: 0.00284 [I 2022-10-27 01:40:57] ...trap/engines/engine.py.230: loss: 0.21415 [I 2022-10-27 01:40:57] ...optimizers/trijoint.py.151: total_norm: 23.785511 [I 2022-10-27 01:41:02] ...trap/engines/engine.py.224: train: epoch 41 | batch 70/2304 [I 2022-10-27 01:41:02] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:41 [I 2022-10-27 01:41:02] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00281 [I 2022-10-27 01:41:02] ...trap/engines/engine.py.230: loss: 0.13447 [I 2022-10-27 01:41:02] ...optimizers/trijoint.py.151: total_norm: 13.849535 [I 2022-10-27 01:41:08] ...trap/engines/engine.py.224: train: epoch 41 | batch 80/2304 [I 2022-10-27 01:41:08] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:33 [I 2022-10-27 01:41:08] ...trap/engines/engine.py.229: process: 0.54865 | load: 0.00283 [I 2022-10-27 01:41:08] ...trap/engines/engine.py.230: loss: 0.15927 [I 2022-10-27 01:41:08] ...optimizers/trijoint.py.151: total_norm: 16.209795 [I 2022-10-27 01:41:14] ...trap/engines/engine.py.224: train: epoch 41 | batch 90/2304 [I 2022-10-27 01:41:14] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:38 [I 2022-10-27 01:41:14] ...trap/engines/engine.py.229: process: 0.56923 | load: 0.00273 [I 2022-10-27 01:41:14] ...trap/engines/engine.py.230: loss: 0.14344 [I 2022-10-27 01:41:14] ...optimizers/trijoint.py.151: total_norm: 9.037621 [I 2022-10-27 01:41:19] ...trap/engines/engine.py.224: train: epoch 41 | batch 100/2304 [I 2022-10-27 01:41:19] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:32 [I 2022-10-27 01:41:19] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00248 [I 2022-10-27 01:41:19] ...trap/engines/engine.py.230: loss: 0.19052 [I 2022-10-27 01:41:19] ...optimizers/trijoint.py.151: total_norm: 27.746508 [I 2022-10-27 01:41:25] ...trap/engines/engine.py.224: train: epoch 41 | batch 110/2304 [I 2022-10-27 01:41:25] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:16 [I 2022-10-27 01:41:25] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00248 [I 2022-10-27 01:41:25] ...trap/engines/engine.py.230: loss: 0.22286 [I 2022-10-27 01:41:25] ...optimizers/trijoint.py.151: total_norm: 11.063179 [I 2022-10-27 01:41:30] ...trap/engines/engine.py.224: train: epoch 41 | batch 120/2304 [I 2022-10-27 01:41:30] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:03 [I 2022-10-27 01:41:30] ...trap/engines/engine.py.229: process: 0.54629 | load: 0.00262 [I 2022-10-27 01:41:30] ...trap/engines/engine.py.230: loss: 0.34796 [I 2022-10-27 01:41:30] ...optimizers/trijoint.py.151: total_norm: 32.804966 [I 2022-10-27 01:41:36] ...trap/engines/engine.py.224: train: epoch 41 | batch 130/2304 [I 2022-10-27 01:41:36] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:11 [I 2022-10-27 01:41:36] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00258 [I 2022-10-27 01:41:36] ...trap/engines/engine.py.230: loss: 0.14433 [I 2022-10-27 01:41:36] ...optimizers/trijoint.py.151: total_norm: 11.177431 [I 2022-10-27 01:41:41] ...trap/engines/engine.py.224: train: epoch 41 | batch 140/2304 [I 2022-10-27 01:41:41] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:00 [I 2022-10-27 01:41:41] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00243 [I 2022-10-27 01:41:41] ...trap/engines/engine.py.230: loss: 0.12672 [I 2022-10-27 01:41:41] ...optimizers/trijoint.py.151: total_norm: 11.191431 [I 2022-10-27 01:41:47] ...trap/engines/engine.py.224: train: epoch 41 | batch 150/2304 [I 2022-10-27 01:41:47] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:54 [I 2022-10-27 01:41:47] ...trap/engines/engine.py.229: process: 0.55371 | load: 0.00268 [I 2022-10-27 01:41:47] ...trap/engines/engine.py.230: loss: 0.13363 [I 2022-10-27 01:41:47] ...optimizers/trijoint.py.151: total_norm: 8.788732 [I 2022-10-27 01:41:53] ...trap/engines/engine.py.224: train: epoch 41 | batch 160/2304 [I 2022-10-27 01:41:53] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:47 [I 2022-10-27 01:41:53] ...trap/engines/engine.py.229: process: 0.54868 | load: 0.00261 [I 2022-10-27 01:41:53] ...trap/engines/engine.py.230: loss: 0.15676 [I 2022-10-27 01:41:53] ...optimizers/trijoint.py.151: total_norm: 9.532579 [I 2022-10-27 01:41:58] ...trap/engines/engine.py.224: train: epoch 41 | batch 170/2304 [I 2022-10-27 01:41:58] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:46 [I 2022-10-27 01:41:58] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00256 [I 2022-10-27 01:41:58] ...trap/engines/engine.py.230: loss: 0.16569 [I 2022-10-27 01:41:58] ...optimizers/trijoint.py.151: total_norm: 16.953562 [I 2022-10-27 01:42:04] ...trap/engines/engine.py.224: train: epoch 41 | batch 180/2304 [I 2022-10-27 01:42:04] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:19:39 [I 2022-10-27 01:42:04] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00239 [I 2022-10-27 01:42:04] ...trap/engines/engine.py.230: loss: 0.20926 [I 2022-10-27 01:42:04] ...optimizers/trijoint.py.151: total_norm: 20.369753 [I 2022-10-27 01:42:09] ...trap/engines/engine.py.224: train: epoch 41 | batch 190/2304 [I 2022-10-27 01:42:09] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:37 [I 2022-10-27 01:42:09] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00241 [I 2022-10-27 01:42:09] ...trap/engines/engine.py.230: loss: 0.13107 [I 2022-10-27 01:42:09] ...optimizers/trijoint.py.151: total_norm: 15.800259 [I 2022-10-27 01:42:15] ...trap/engines/engine.py.224: train: epoch 41 | batch 200/2304 [I 2022-10-27 01:42:15] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:36 [I 2022-10-27 01:42:15] ...trap/engines/engine.py.229: process: 0.56992 | load: 0.00243 [I 2022-10-27 01:42:15] ...trap/engines/engine.py.230: loss: 0.25888 [I 2022-10-27 01:42:15] ...optimizers/trijoint.py.151: total_norm: 60.926983 [I 2022-10-27 01:42:20] ...trap/engines/engine.py.224: train: epoch 41 | batch 210/2304 [I 2022-10-27 01:42:20] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:22 [I 2022-10-27 01:42:20] ...trap/engines/engine.py.229: process: 0.54539 | load: 0.00242 [I 2022-10-27 01:42:20] ...trap/engines/engine.py.230: loss: 0.18691 [I 2022-10-27 01:42:20] ...optimizers/trijoint.py.151: total_norm: 21.951056 [I 2022-10-27 01:42:26] ...trap/engines/engine.py.224: train: epoch 41 | batch 220/2304 [I 2022-10-27 01:42:26] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:30 [I 2022-10-27 01:42:26] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00245 [I 2022-10-27 01:42:26] ...trap/engines/engine.py.230: loss: 0.15571 [I 2022-10-27 01:42:26] ...optimizers/trijoint.py.151: total_norm: 12.116575 [I 2022-10-27 01:42:32] ...trap/engines/engine.py.224: train: epoch 41 | batch 230/2304 [I 2022-10-27 01:42:32] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:15 [I 2022-10-27 01:42:32] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00250 [I 2022-10-27 01:42:32] ...trap/engines/engine.py.230: loss: 0.14610 [I 2022-10-27 01:42:32] ...optimizers/trijoint.py.151: total_norm: 22.429134 [I 2022-10-27 01:42:37] ...trap/engines/engine.py.224: train: epoch 41 | batch 240/2304 [I 2022-10-27 01:42:37] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:16 [I 2022-10-27 01:42:37] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00255 [I 2022-10-27 01:42:37] ...trap/engines/engine.py.230: loss: 0.15475 [I 2022-10-27 01:42:37] ...optimizers/trijoint.py.151: total_norm: 19.736198 [I 2022-10-27 01:42:43] ...trap/engines/engine.py.224: train: epoch 41 | batch 250/2304 [I 2022-10-27 01:42:43] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:00 [I 2022-10-27 01:42:43] ...trap/engines/engine.py.229: process: 0.55198 | load: 0.00274 [I 2022-10-27 01:42:43] ...trap/engines/engine.py.230: loss: 0.18552 [I 2022-10-27 01:42:43] ...optimizers/trijoint.py.151: total_norm: 16.760071 [I 2022-10-27 01:42:48] ...trap/engines/engine.py.224: train: epoch 41 | batch 260/2304 [I 2022-10-27 01:42:48] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:56 [I 2022-10-27 01:42:48] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00257 [I 2022-10-27 01:42:48] ...trap/engines/engine.py.230: loss: 0.16541 [I 2022-10-27 01:42:48] ...optimizers/trijoint.py.151: total_norm: 27.442005 [I 2022-10-27 01:42:54] ...trap/engines/engine.py.224: train: epoch 41 | batch 270/2304 [I 2022-10-27 01:42:54] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:59 [I 2022-10-27 01:42:54] ...trap/engines/engine.py.229: process: 0.56365 | load: 0.00242 [I 2022-10-27 01:42:54] ...trap/engines/engine.py.230: loss: 0.11182 [I 2022-10-27 01:42:54] ...optimizers/trijoint.py.151: total_norm: 12.071847 [I 2022-10-27 01:43:00] ...trap/engines/engine.py.224: train: epoch 41 | batch 280/2304 [I 2022-10-27 01:43:00] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:44 [I 2022-10-27 01:43:00] ...trap/engines/engine.py.229: process: 0.55242 | load: 0.00244 [I 2022-10-27 01:43:00] ...trap/engines/engine.py.230: loss: 0.21593 [I 2022-10-27 01:43:00] ...optimizers/trijoint.py.151: total_norm: 31.953033 [I 2022-10-27 01:43:05] ...trap/engines/engine.py.224: train: epoch 41 | batch 290/2304 [I 2022-10-27 01:43:05] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:28 [I 2022-10-27 01:43:05] ...trap/engines/engine.py.229: process: 0.55250 | load: 0.00243 [I 2022-10-27 01:43:05] ...trap/engines/engine.py.230: loss: 0.14425 [I 2022-10-27 01:43:05] ...optimizers/trijoint.py.151: total_norm: 10.900831 [I 2022-10-27 01:43:11] ...trap/engines/engine.py.224: train: epoch 41 | batch 300/2304 [I 2022-10-27 01:43:11] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:18:30 [I 2022-10-27 01:43:11] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00255 [I 2022-10-27 01:43:11] ...trap/engines/engine.py.230: loss: 0.17799 [I 2022-10-27 01:43:11] ...optimizers/trijoint.py.151: total_norm: 22.802845 [I 2022-10-27 01:43:16] ...trap/engines/engine.py.224: train: epoch 41 | batch 310/2304 [I 2022-10-27 01:43:16] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:12 [I 2022-10-27 01:43:16] ...trap/engines/engine.py.229: process: 0.54174 | load: 0.00236 [I 2022-10-27 01:43:16] ...trap/engines/engine.py.230: loss: 0.14860 [I 2022-10-27 01:43:16] ...optimizers/trijoint.py.151: total_norm: 12.147490 [I 2022-10-27 01:43:22] ...trap/engines/engine.py.224: train: epoch 41 | batch 320/2304 [I 2022-10-27 01:43:22] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:18:26 [I 2022-10-27 01:43:22] ...trap/engines/engine.py.229: process: 0.55131 | load: 0.00252 [I 2022-10-27 01:43:22] ...trap/engines/engine.py.230: loss: 0.14668 [I 2022-10-27 01:43:22] ...optimizers/trijoint.py.151: total_norm: 21.250139 [I 2022-10-27 01:43:27] ...trap/engines/engine.py.224: train: epoch 41 | batch 330/2304 [I 2022-10-27 01:43:27] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:17 [I 2022-10-27 01:43:27] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00269 [I 2022-10-27 01:43:27] ...trap/engines/engine.py.230: loss: 0.14947 [I 2022-10-27 01:43:27] ...optimizers/trijoint.py.151: total_norm: 13.727276 [I 2022-10-27 01:43:33] ...trap/engines/engine.py.224: train: epoch 41 | batch 340/2304 [I 2022-10-27 01:43:33] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:11 [I 2022-10-27 01:43:33] ...trap/engines/engine.py.229: process: 0.54440 | load: 0.00249 [I 2022-10-27 01:43:33] ...trap/engines/engine.py.230: loss: 0.14874 [I 2022-10-27 01:43:33] ...optimizers/trijoint.py.151: total_norm: 13.570134 [I 2022-10-27 01:43:38] ...trap/engines/engine.py.224: train: epoch 41 | batch 350/2304 [I 2022-10-27 01:43:38] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:18:04 [I 2022-10-27 01:43:38] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00247 [I 2022-10-27 01:43:38] ...trap/engines/engine.py.230: loss: 0.13767 [I 2022-10-27 01:43:38] ...optimizers/trijoint.py.151: total_norm: 14.958857 [I 2022-10-27 01:43:44] ...trap/engines/engine.py.224: train: epoch 41 | batch 360/2304 [I 2022-10-27 01:43:44] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:02 [I 2022-10-27 01:43:44] ...trap/engines/engine.py.229: process: 0.55494 | load: 0.00243 [I 2022-10-27 01:43:44] ...trap/engines/engine.py.230: loss: 0.12913 [I 2022-10-27 01:43:44] ...optimizers/trijoint.py.151: total_norm: 11.717484 [I 2022-10-27 01:43:50] ...trap/engines/engine.py.224: train: epoch 41 | batch 370/2304 [I 2022-10-27 01:43:50] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:18:04 [I 2022-10-27 01:43:50] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00245 [I 2022-10-27 01:43:50] ...trap/engines/engine.py.230: loss: 0.20164 [I 2022-10-27 01:43:50] ...optimizers/trijoint.py.151: total_norm: 17.127649 [I 2022-10-27 01:43:55] ...trap/engines/engine.py.224: train: epoch 41 | batch 380/2304 [I 2022-10-27 01:43:55] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:50 [I 2022-10-27 01:43:55] ...trap/engines/engine.py.229: process: 0.55341 | load: 0.00251 [I 2022-10-27 01:43:55] ...trap/engines/engine.py.230: loss: 0.13785 [I 2022-10-27 01:43:55] ...optimizers/trijoint.py.151: total_norm: 14.858088 [I 2022-10-27 01:44:01] ...trap/engines/engine.py.224: train: epoch 41 | batch 390/2304 [I 2022-10-27 01:44:01] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:46 [I 2022-10-27 01:44:01] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00247 [I 2022-10-27 01:44:01] ...trap/engines/engine.py.230: loss: 0.19391 [I 2022-10-27 01:44:01] ...optimizers/trijoint.py.151: total_norm: 29.706779 [I 2022-10-27 01:44:06] ...trap/engines/engine.py.224: train: epoch 41 | batch 400/2304 [I 2022-10-27 01:44:06] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:43 [I 2022-10-27 01:44:06] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00246 [I 2022-10-27 01:44:06] ...trap/engines/engine.py.230: loss: 0.23124 [I 2022-10-27 01:44:06] ...optimizers/trijoint.py.151: total_norm: 44.599751 [I 2022-10-27 01:44:12] ...trap/engines/engine.py.224: train: epoch 41 | batch 410/2304 [I 2022-10-27 01:44:12] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:39 [I 2022-10-27 01:44:12] ...trap/engines/engine.py.229: process: 0.56751 | load: 0.00242 [I 2022-10-27 01:44:12] ...trap/engines/engine.py.230: loss: 0.20114 [I 2022-10-27 01:44:12] ...optimizers/trijoint.py.151: total_norm: 26.533863 [I 2022-10-27 01:44:18] ...trap/engines/engine.py.224: train: epoch 41 | batch 420/2304 [I 2022-10-27 01:44:18] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:17:32 [I 2022-10-27 01:44:18] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00259 [I 2022-10-27 01:44:18] ...trap/engines/engine.py.230: loss: 0.15659 [I 2022-10-27 01:44:18] ...optimizers/trijoint.py.151: total_norm: 19.307713 [I 2022-10-27 01:44:23] ...trap/engines/engine.py.224: train: epoch 41 | batch 430/2304 [I 2022-10-27 01:44:23] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:24 [I 2022-10-27 01:44:23] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00250 [I 2022-10-27 01:44:23] ...trap/engines/engine.py.230: loss: 0.14269 [I 2022-10-27 01:44:23] ...optimizers/trijoint.py.151: total_norm: 13.790274 [I 2022-10-27 01:44:29] ...trap/engines/engine.py.224: train: epoch 41 | batch 440/2304 [I 2022-10-27 01:44:29] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:17:11 [I 2022-10-27 01:44:29] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00244 [I 2022-10-27 01:44:29] ...trap/engines/engine.py.230: loss: 0.22287 [I 2022-10-27 01:44:29] ...optimizers/trijoint.py.151: total_norm: 18.229740 [I 2022-10-27 01:44:34] ...trap/engines/engine.py.224: train: epoch 41 | batch 450/2304 [I 2022-10-27 01:44:34] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:17 [I 2022-10-27 01:44:34] ...trap/engines/engine.py.229: process: 0.56683 | load: 0.00245 [I 2022-10-27 01:44:34] ...trap/engines/engine.py.230: loss: 0.14810 [I 2022-10-27 01:44:34] ...optimizers/trijoint.py.151: total_norm: 15.859167 [I 2022-10-27 01:44:40] ...trap/engines/engine.py.224: train: epoch 41 | batch 460/2304 [I 2022-10-27 01:44:40] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:08 [I 2022-10-27 01:44:40] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00254 [I 2022-10-27 01:44:40] ...trap/engines/engine.py.230: loss: 0.16090 [I 2022-10-27 01:44:40] ...optimizers/trijoint.py.151: total_norm: 29.293089 [I 2022-10-27 01:44:45] ...trap/engines/engine.py.224: train: epoch 41 | batch 470/2304 [I 2022-10-27 01:44:45] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:16:57 [I 2022-10-27 01:44:45] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00251 [I 2022-10-27 01:44:45] ...trap/engines/engine.py.230: loss: 0.17683 [I 2022-10-27 01:44:45] ...optimizers/trijoint.py.151: total_norm: 35.502934 [I 2022-10-27 01:44:51] ...trap/engines/engine.py.224: train: epoch 41 | batch 480/2304 [I 2022-10-27 01:44:51] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:58 [I 2022-10-27 01:44:51] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00260 [I 2022-10-27 01:44:51] ...trap/engines/engine.py.230: loss: 0.20129 [I 2022-10-27 01:44:51] ...optimizers/trijoint.py.151: total_norm: 22.854788 [I 2022-10-27 01:44:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 490/2304 [I 2022-10-27 01:44:57] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:16:41 [I 2022-10-27 01:44:57] ...trap/engines/engine.py.229: process: 0.55025 | load: 0.00249 [I 2022-10-27 01:44:57] ...trap/engines/engine.py.230: loss: 0.14762 [I 2022-10-27 01:44:57] ...optimizers/trijoint.py.151: total_norm: 14.311223 [I 2022-10-27 01:45:02] ...trap/engines/engine.py.224: train: epoch 41 | batch 500/2304 [I 2022-10-27 01:45:02] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:39 [I 2022-10-27 01:45:02] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00246 [I 2022-10-27 01:45:02] ...trap/engines/engine.py.230: loss: 0.14846 [I 2022-10-27 01:45:02] ...optimizers/trijoint.py.151: total_norm: 16.243158 [I 2022-10-27 01:45:08] ...trap/engines/engine.py.224: train: epoch 41 | batch 510/2304 [I 2022-10-27 01:45:08] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:16:34 [I 2022-10-27 01:45:08] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00248 [I 2022-10-27 01:45:08] ...trap/engines/engine.py.230: loss: 0.15257 [I 2022-10-27 01:45:08] ...optimizers/trijoint.py.151: total_norm: 10.064287 [I 2022-10-27 01:45:13] ...trap/engines/engine.py.224: train: epoch 41 | batch 520/2304 [I 2022-10-27 01:45:13] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:16:32 [I 2022-10-27 01:45:13] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00245 [I 2022-10-27 01:45:13] ...trap/engines/engine.py.230: loss: 0.15940 [I 2022-10-27 01:45:13] ...optimizers/trijoint.py.151: total_norm: 20.238008 [I 2022-10-27 01:45:19] ...trap/engines/engine.py.224: train: epoch 41 | batch 530/2304 [I 2022-10-27 01:45:19] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:21 [I 2022-10-27 01:45:19] ...trap/engines/engine.py.229: process: 0.54659 | load: 0.00245 [I 2022-10-27 01:45:19] ...trap/engines/engine.py.230: loss: 0.21116 [I 2022-10-27 01:45:19] ...optimizers/trijoint.py.151: total_norm: 16.968807 [I 2022-10-27 01:45:24] ...trap/engines/engine.py.224: train: epoch 41 | batch 540/2304 [I 2022-10-27 01:45:24] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:16:14 [I 2022-10-27 01:45:24] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00260 [I 2022-10-27 01:45:24] ...trap/engines/engine.py.230: loss: 0.17251 [I 2022-10-27 01:45:24] ...optimizers/trijoint.py.151: total_norm: 32.181984 [I 2022-10-27 01:45:30] ...trap/engines/engine.py.224: train: epoch 41 | batch 550/2304 [I 2022-10-27 01:45:30] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:13 [I 2022-10-27 01:45:30] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00251 [I 2022-10-27 01:45:30] ...trap/engines/engine.py.230: loss: 0.17807 [I 2022-10-27 01:45:30] ...optimizers/trijoint.py.151: total_norm: 15.978430 [I 2022-10-27 01:45:36] ...trap/engines/engine.py.224: train: epoch 41 | batch 560/2304 [I 2022-10-27 01:45:36] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:16:16 [I 2022-10-27 01:45:36] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00260 [I 2022-10-27 01:45:36] ...trap/engines/engine.py.230: loss: 0.13530 [I 2022-10-27 01:45:36] ...optimizers/trijoint.py.151: total_norm: 17.449965 [I 2022-10-27 01:45:41] ...trap/engines/engine.py.224: train: epoch 41 | batch 570/2304 [I 2022-10-27 01:45:41] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:16:34 [I 2022-10-27 01:45:41] ...trap/engines/engine.py.229: process: 0.55937 | load: 0.00266 [I 2022-10-27 01:45:41] ...trap/engines/engine.py.230: loss: 0.11795 [I 2022-10-27 01:45:41] ...optimizers/trijoint.py.151: total_norm: 13.886396 [I 2022-10-27 01:45:47] ...trap/engines/engine.py.224: train: epoch 41 | batch 580/2304 [I 2022-10-27 01:45:47] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:15:58 [I 2022-10-27 01:45:47] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00246 [I 2022-10-27 01:45:47] ...trap/engines/engine.py.230: loss: 0.14819 [I 2022-10-27 01:45:47] ...optimizers/trijoint.py.151: total_norm: 8.599825 [I 2022-10-27 01:45:52] ...trap/engines/engine.py.224: train: epoch 41 | batch 590/2304 [I 2022-10-27 01:45:52] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:15:53 [I 2022-10-27 01:45:52] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00247 [I 2022-10-27 01:45:52] ...trap/engines/engine.py.230: loss: 0.09050 [I 2022-10-27 01:45:52] ...optimizers/trijoint.py.151: total_norm: 8.496205 [I 2022-10-27 01:45:58] ...trap/engines/engine.py.224: train: epoch 41 | batch 600/2304 [I 2022-10-27 01:45:58] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:46 [I 2022-10-27 01:45:58] ...trap/engines/engine.py.229: process: 0.55414 | load: 0.00256 [I 2022-10-27 01:45:58] ...trap/engines/engine.py.230: loss: 0.17980 [I 2022-10-27 01:45:58] ...optimizers/trijoint.py.151: total_norm: 21.359488 [I 2022-10-27 01:46:04] ...trap/engines/engine.py.224: train: epoch 41 | batch 610/2304 [I 2022-10-27 01:46:04] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:15:45 [I 2022-10-27 01:46:04] ...trap/engines/engine.py.229: process: 0.57030 | load: 0.00252 [I 2022-10-27 01:46:04] ...trap/engines/engine.py.230: loss: 0.18162 [I 2022-10-27 01:46:04] ...optimizers/trijoint.py.151: total_norm: 13.390689 [I 2022-10-27 01:46:09] ...trap/engines/engine.py.224: train: epoch 41 | batch 620/2304 [I 2022-10-27 01:46:09] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:36 [I 2022-10-27 01:46:09] ...trap/engines/engine.py.229: process: 0.54766 | load: 0.00255 [I 2022-10-27 01:46:09] ...trap/engines/engine.py.230: loss: 0.12487 [I 2022-10-27 01:46:09] ...optimizers/trijoint.py.151: total_norm: 8.697601 [I 2022-10-27 01:46:15] ...trap/engines/engine.py.224: train: epoch 41 | batch 630/2304 [I 2022-10-27 01:46:15] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:15:37 [I 2022-10-27 01:46:15] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00242 [I 2022-10-27 01:46:15] ...trap/engines/engine.py.230: loss: 0.22331 [I 2022-10-27 01:46:15] ...optimizers/trijoint.py.151: total_norm: 17.152573 [I 2022-10-27 01:46:20] ...trap/engines/engine.py.224: train: epoch 41 | batch 640/2304 [I 2022-10-27 01:46:20] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:31 [I 2022-10-27 01:46:20] ...trap/engines/engine.py.229: process: 0.57158 | load: 0.00256 [I 2022-10-27 01:46:20] ...trap/engines/engine.py.230: loss: 0.11290 [I 2022-10-27 01:46:20] ...optimizers/trijoint.py.151: total_norm: 9.496069 [I 2022-10-27 01:46:26] ...trap/engines/engine.py.224: train: epoch 41 | batch 650/2304 [I 2022-10-27 01:46:26] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:33 [I 2022-10-27 01:46:26] ...trap/engines/engine.py.229: process: 0.54778 | load: 0.00241 [I 2022-10-27 01:46:26] ...trap/engines/engine.py.230: loss: 0.17966 [I 2022-10-27 01:46:26] ...optimizers/trijoint.py.151: total_norm: 16.706240 [I 2022-10-27 01:46:32] ...trap/engines/engine.py.224: train: epoch 41 | batch 660/2304 [I 2022-10-27 01:46:32] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:15:14 [I 2022-10-27 01:46:32] ...trap/engines/engine.py.229: process: 0.54540 | load: 0.00241 [I 2022-10-27 01:46:32] ...trap/engines/engine.py.230: loss: 0.21346 [I 2022-10-27 01:46:32] ...optimizers/trijoint.py.151: total_norm: 28.090061 [I 2022-10-27 01:46:37] ...trap/engines/engine.py.224: train: epoch 41 | batch 670/2304 [I 2022-10-27 01:46:37] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:11 [I 2022-10-27 01:46:37] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00242 [I 2022-10-27 01:46:37] ...trap/engines/engine.py.230: loss: 0.11179 [I 2022-10-27 01:46:37] ...optimizers/trijoint.py.151: total_norm: 41.380634 [I 2022-10-27 01:46:43] ...trap/engines/engine.py.224: train: epoch 41 | batch 680/2304 [I 2022-10-27 01:46:43] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:15:02 [I 2022-10-27 01:46:43] ...trap/engines/engine.py.229: process: 0.55597 | load: 0.00243 [I 2022-10-27 01:46:43] ...trap/engines/engine.py.230: loss: 0.16396 [I 2022-10-27 01:46:43] ...optimizers/trijoint.py.151: total_norm: 16.096422 [I 2022-10-27 01:46:48] ...trap/engines/engine.py.224: train: epoch 41 | batch 690/2304 [I 2022-10-27 01:46:48] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:14:59 [I 2022-10-27 01:46:48] ...trap/engines/engine.py.229: process: 0.55576 | load: 0.00244 [I 2022-10-27 01:46:48] ...trap/engines/engine.py.230: loss: 0.19830 [I 2022-10-27 01:46:48] ...optimizers/trijoint.py.151: total_norm: 17.383949 [I 2022-10-27 01:46:54] ...trap/engines/engine.py.224: train: epoch 41 | batch 700/2304 [I 2022-10-27 01:46:54] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:58 [I 2022-10-27 01:46:54] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00246 [I 2022-10-27 01:46:54] ...trap/engines/engine.py.230: loss: 0.10986 [I 2022-10-27 01:46:54] ...optimizers/trijoint.py.151: total_norm: 10.938099 [I 2022-10-27 01:47:00] ...trap/engines/engine.py.224: train: epoch 41 | batch 710/2304 [I 2022-10-27 01:47:00] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:15:15 [I 2022-10-27 01:47:00] ...trap/engines/engine.py.229: process: 0.62422 | load: 0.00248 [I 2022-10-27 01:47:00] ...trap/engines/engine.py.230: loss: 0.16320 [I 2022-10-27 01:47:00] ...optimizers/trijoint.py.151: total_norm: 18.861578 [I 2022-10-27 01:47:05] ...trap/engines/engine.py.224: train: epoch 41 | batch 720/2304 [I 2022-10-27 01:47:05] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:45 [I 2022-10-27 01:47:05] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00243 [I 2022-10-27 01:47:05] ...trap/engines/engine.py.230: loss: 0.15629 [I 2022-10-27 01:47:05] ...optimizers/trijoint.py.151: total_norm: 17.735672 [I 2022-10-27 01:47:11] ...trap/engines/engine.py.224: train: epoch 41 | batch 730/2304 [I 2022-10-27 01:47:11] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:14:36 [I 2022-10-27 01:47:11] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00257 [I 2022-10-27 01:47:11] ...trap/engines/engine.py.230: loss: 0.16350 [I 2022-10-27 01:47:11] ...optimizers/trijoint.py.151: total_norm: 8.402011 [I 2022-10-27 01:47:16] ...trap/engines/engine.py.224: train: epoch 41 | batch 740/2304 [I 2022-10-27 01:47:16] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:35 [I 2022-10-27 01:47:16] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00246 [I 2022-10-27 01:47:16] ...trap/engines/engine.py.230: loss: 0.20886 [I 2022-10-27 01:47:16] ...optimizers/trijoint.py.151: total_norm: 16.326298 [I 2022-10-27 01:47:22] ...trap/engines/engine.py.224: train: epoch 41 | batch 750/2304 [I 2022-10-27 01:47:22] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:27 [I 2022-10-27 01:47:22] ...trap/engines/engine.py.229: process: 0.54707 | load: 0.00252 [I 2022-10-27 01:47:22] ...trap/engines/engine.py.230: loss: 0.18316 [I 2022-10-27 01:47:22] ...optimizers/trijoint.py.151: total_norm: 23.996613 [I 2022-10-27 01:47:28] ...trap/engines/engine.py.224: train: epoch 41 | batch 760/2304 [I 2022-10-27 01:47:28] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:23 [I 2022-10-27 01:47:28] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00244 [I 2022-10-27 01:47:28] ...trap/engines/engine.py.230: loss: 0.17289 [I 2022-10-27 01:47:28] ...optimizers/trijoint.py.151: total_norm: 20.017426 [I 2022-10-27 01:47:33] ...trap/engines/engine.py.224: train: epoch 41 | batch 770/2304 [I 2022-10-27 01:47:33] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:19 [I 2022-10-27 01:47:33] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00243 [I 2022-10-27 01:47:33] ...trap/engines/engine.py.230: loss: 0.14594 [I 2022-10-27 01:47:33] ...optimizers/trijoint.py.151: total_norm: 23.518286 [I 2022-10-27 01:47:39] ...trap/engines/engine.py.224: train: epoch 41 | batch 780/2304 [I 2022-10-27 01:47:39] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:18 [I 2022-10-27 01:47:39] ...trap/engines/engine.py.229: process: 0.56181 | load: 0.00242 [I 2022-10-27 01:47:39] ...trap/engines/engine.py.230: loss: 0.16191 [I 2022-10-27 01:47:39] ...optimizers/trijoint.py.151: total_norm: 10.759656 [I 2022-10-27 01:47:44] ...trap/engines/engine.py.224: train: epoch 41 | batch 790/2304 [I 2022-10-27 01:47:44] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:01 [I 2022-10-27 01:47:44] ...trap/engines/engine.py.229: process: 0.54088 | load: 0.00245 [I 2022-10-27 01:47:44] ...trap/engines/engine.py.230: loss: 0.21243 [I 2022-10-27 01:47:44] ...optimizers/trijoint.py.151: total_norm: 32.300606 [I 2022-10-27 01:47:50] ...trap/engines/engine.py.224: train: epoch 41 | batch 800/2304 [I 2022-10-27 01:47:50] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:59 [I 2022-10-27 01:47:50] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00245 [I 2022-10-27 01:47:50] ...trap/engines/engine.py.230: loss: 0.17855 [I 2022-10-27 01:47:50] ...optimizers/trijoint.py.151: total_norm: 19.157709 [I 2022-10-27 01:47:56] ...trap/engines/engine.py.224: train: epoch 41 | batch 810/2304 [I 2022-10-27 01:47:56] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:13:49 [I 2022-10-27 01:47:56] ...trap/engines/engine.py.229: process: 0.54366 | load: 0.00248 [I 2022-10-27 01:47:56] ...trap/engines/engine.py.230: loss: 0.21849 [I 2022-10-27 01:47:56] ...optimizers/trijoint.py.151: total_norm: 18.313925 [I 2022-10-27 01:48:01] ...trap/engines/engine.py.224: train: epoch 41 | batch 820/2304 [I 2022-10-27 01:48:01] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:47 [I 2022-10-27 01:48:01] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00247 [I 2022-10-27 01:48:01] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-27 01:48:01] ...optimizers/trijoint.py.151: total_norm: 7.294928 [I 2022-10-27 01:48:07] ...trap/engines/engine.py.224: train: epoch 41 | batch 830/2304 [I 2022-10-27 01:48:07] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:55 [I 2022-10-27 01:48:07] ...trap/engines/engine.py.229: process: 0.56335 | load: 0.00246 [I 2022-10-27 01:48:07] ...trap/engines/engine.py.230: loss: 0.15743 [I 2022-10-27 01:48:07] ...optimizers/trijoint.py.151: total_norm: 12.579627 [I 2022-10-27 01:48:12] ...trap/engines/engine.py.224: train: epoch 41 | batch 840/2304 [I 2022-10-27 01:48:12] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:35 [I 2022-10-27 01:48:12] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00241 [I 2022-10-27 01:48:12] ...trap/engines/engine.py.230: loss: 0.16744 [I 2022-10-27 01:48:12] ...optimizers/trijoint.py.151: total_norm: 17.329771 [I 2022-10-27 01:48:18] ...trap/engines/engine.py.224: train: epoch 41 | batch 850/2304 [I 2022-10-27 01:48:18] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:32 [I 2022-10-27 01:48:18] ...trap/engines/engine.py.229: process: 0.56731 | load: 0.00246 [I 2022-10-27 01:48:18] ...trap/engines/engine.py.230: loss: 0.15366 [I 2022-10-27 01:48:18] ...optimizers/trijoint.py.151: total_norm: 20.394798 [I 2022-10-27 01:48:24] ...trap/engines/engine.py.224: train: epoch 41 | batch 860/2304 [I 2022-10-27 01:48:24] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:21 [I 2022-10-27 01:48:24] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00243 [I 2022-10-27 01:48:24] ...trap/engines/engine.py.230: loss: 0.16585 [I 2022-10-27 01:48:24] ...optimizers/trijoint.py.151: total_norm: 16.538296 [I 2022-10-27 01:48:29] ...trap/engines/engine.py.224: train: epoch 41 | batch 870/2304 [I 2022-10-27 01:48:29] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:19 [I 2022-10-27 01:48:29] ...trap/engines/engine.py.229: process: 0.56632 | load: 0.00250 [I 2022-10-27 01:48:29] ...trap/engines/engine.py.230: loss: 0.25379 [I 2022-10-27 01:48:29] ...optimizers/trijoint.py.151: total_norm: 17.405630 [I 2022-10-27 01:48:35] ...trap/engines/engine.py.224: train: epoch 41 | batch 880/2304 [I 2022-10-27 01:48:35] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:07 [I 2022-10-27 01:48:35] ...trap/engines/engine.py.229: process: 0.54543 | load: 0.00268 [I 2022-10-27 01:48:35] ...trap/engines/engine.py.230: loss: 0.14407 [I 2022-10-27 01:48:35] ...optimizers/trijoint.py.151: total_norm: 23.682823 [I 2022-10-27 01:48:40] ...trap/engines/engine.py.224: train: epoch 41 | batch 890/2304 [I 2022-10-27 01:48:40] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:07 [I 2022-10-27 01:48:40] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00245 [I 2022-10-27 01:48:40] ...trap/engines/engine.py.230: loss: 0.14260 [I 2022-10-27 01:48:40] ...optimizers/trijoint.py.151: total_norm: 16.455454 [I 2022-10-27 01:48:46] ...trap/engines/engine.py.224: train: epoch 41 | batch 900/2304 [I 2022-10-27 01:48:46] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:12:58 [I 2022-10-27 01:48:46] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00247 [I 2022-10-27 01:48:46] ...trap/engines/engine.py.230: loss: 0.17857 [I 2022-10-27 01:48:46] ...optimizers/trijoint.py.151: total_norm: 13.960356 [I 2022-10-27 01:48:51] ...trap/engines/engine.py.224: train: epoch 41 | batch 910/2304 [I 2022-10-27 01:48:51] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:58 [I 2022-10-27 01:48:51] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00242 [I 2022-10-27 01:48:51] ...trap/engines/engine.py.230: loss: 0.11849 [I 2022-10-27 01:48:51] ...optimizers/trijoint.py.151: total_norm: 12.332849 [I 2022-10-27 01:48:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 920/2304 [I 2022-10-27 01:48:57] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:50 [I 2022-10-27 01:48:57] ...trap/engines/engine.py.229: process: 0.54774 | load: 0.00245 [I 2022-10-27 01:48:57] ...trap/engines/engine.py.230: loss: 0.12577 [I 2022-10-27 01:48:57] ...optimizers/trijoint.py.151: total_norm: 16.047239 [I 2022-10-27 01:49:03] ...trap/engines/engine.py.224: train: epoch 41 | batch 930/2304 [I 2022-10-27 01:49:03] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:12:46 [I 2022-10-27 01:49:03] ...trap/engines/engine.py.229: process: 0.55624 | load: 0.00256 [I 2022-10-27 01:49:03] ...trap/engines/engine.py.230: loss: 0.19792 [I 2022-10-27 01:49:03] ...optimizers/trijoint.py.151: total_norm: 25.895105 [I 2022-10-27 01:49:08] ...trap/engines/engine.py.224: train: epoch 41 | batch 940/2304 [I 2022-10-27 01:49:08] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:50 [I 2022-10-27 01:49:08] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00247 [I 2022-10-27 01:49:08] ...trap/engines/engine.py.230: loss: 0.18546 [I 2022-10-27 01:49:08] ...optimizers/trijoint.py.151: total_norm: 25.707941 [I 2022-10-27 01:49:14] ...trap/engines/engine.py.224: train: epoch 41 | batch 950/2304 [I 2022-10-27 01:49:14] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:33 [I 2022-10-27 01:49:14] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00246 [I 2022-10-27 01:49:14] ...trap/engines/engine.py.230: loss: 0.18011 [I 2022-10-27 01:49:14] ...optimizers/trijoint.py.151: total_norm: 14.046640 [I 2022-10-27 01:49:19] ...trap/engines/engine.py.224: train: epoch 41 | batch 960/2304 [I 2022-10-27 01:49:19] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:23 [I 2022-10-27 01:49:19] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00244 [I 2022-10-27 01:49:19] ...trap/engines/engine.py.230: loss: 0.15312 [I 2022-10-27 01:49:19] ...optimizers/trijoint.py.151: total_norm: 22.320942 [I 2022-10-27 01:49:25] ...trap/engines/engine.py.224: train: epoch 41 | batch 970/2304 [I 2022-10-27 01:49:25] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:23 [I 2022-10-27 01:49:25] ...trap/engines/engine.py.229: process: 0.56457 | load: 0.00243 [I 2022-10-27 01:49:25] ...trap/engines/engine.py.230: loss: 0.16349 [I 2022-10-27 01:49:25] ...optimizers/trijoint.py.151: total_norm: 14.900766 [I 2022-10-27 01:49:31] ...trap/engines/engine.py.224: train: epoch 41 | batch 980/2304 [I 2022-10-27 01:49:31] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:12 [I 2022-10-27 01:49:31] ...trap/engines/engine.py.229: process: 0.54006 | load: 0.00249 [I 2022-10-27 01:49:31] ...trap/engines/engine.py.230: loss: 0.18356 [I 2022-10-27 01:49:31] ...optimizers/trijoint.py.151: total_norm: 17.444942 [I 2022-10-27 01:49:36] ...trap/engines/engine.py.224: train: epoch 41 | batch 990/2304 [I 2022-10-27 01:49:36] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:13 [I 2022-10-27 01:49:36] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00245 [I 2022-10-27 01:49:36] ...trap/engines/engine.py.230: loss: 0.20819 [I 2022-10-27 01:49:36] ...optimizers/trijoint.py.151: total_norm: 13.396506 [I 2022-10-27 01:49:42] ...trap/engines/engine.py.224: train: epoch 41 | batch 1000/2304 [I 2022-10-27 01:49:42] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:06 [I 2022-10-27 01:49:42] ...trap/engines/engine.py.229: process: 0.55316 | load: 0.00241 [I 2022-10-27 01:49:42] ...trap/engines/engine.py.230: loss: 0.11353 [I 2022-10-27 01:49:42] ...optimizers/trijoint.py.151: total_norm: 12.223730 [I 2022-10-27 01:49:47] ...trap/engines/engine.py.224: train: epoch 41 | batch 1010/2304 [I 2022-10-27 01:49:47] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:11:59 [I 2022-10-27 01:49:47] ...trap/engines/engine.py.229: process: 0.56798 | load: 0.00269 [I 2022-10-27 01:49:47] ...trap/engines/engine.py.230: loss: 0.15706 [I 2022-10-27 01:49:47] ...optimizers/trijoint.py.151: total_norm: 19.597389 [I 2022-10-27 01:49:53] ...trap/engines/engine.py.224: train: epoch 41 | batch 1020/2304 [I 2022-10-27 01:49:53] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:51 [I 2022-10-27 01:49:53] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00249 [I 2022-10-27 01:49:53] ...trap/engines/engine.py.230: loss: 0.19795 [I 2022-10-27 01:49:53] ...optimizers/trijoint.py.151: total_norm: 13.745212 [I 2022-10-27 01:49:58] ...trap/engines/engine.py.224: train: epoch 41 | batch 1030/2304 [I 2022-10-27 01:49:58] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:48 [I 2022-10-27 01:49:58] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00242 [I 2022-10-27 01:49:58] ...trap/engines/engine.py.230: loss: 0.11436 [I 2022-10-27 01:49:58] ...optimizers/trijoint.py.151: total_norm: 20.102091 [I 2022-10-27 01:50:04] ...trap/engines/engine.py.224: train: epoch 41 | batch 1040/2304 [I 2022-10-27 01:50:04] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:44 [I 2022-10-27 01:50:04] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00244 [I 2022-10-27 01:50:04] ...trap/engines/engine.py.230: loss: 0.24870 [I 2022-10-27 01:50:04] ...optimizers/trijoint.py.151: total_norm: 27.548235 [I 2022-10-27 01:50:10] ...trap/engines/engine.py.224: train: epoch 41 | batch 1050/2304 [I 2022-10-27 01:50:10] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:39 [I 2022-10-27 01:50:10] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00244 [I 2022-10-27 01:50:10] ...trap/engines/engine.py.230: loss: 0.16202 [I 2022-10-27 01:50:10] ...optimizers/trijoint.py.151: total_norm: 12.635800 [I 2022-10-27 01:50:15] ...trap/engines/engine.py.224: train: epoch 41 | batch 1060/2304 [I 2022-10-27 01:50:15] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:37 [I 2022-10-27 01:50:15] ...trap/engines/engine.py.229: process: 0.56502 | load: 0.00251 [I 2022-10-27 01:50:15] ...trap/engines/engine.py.230: loss: 0.13874 [I 2022-10-27 01:50:15] ...optimizers/trijoint.py.151: total_norm: 15.787620 [I 2022-10-27 01:50:21] ...trap/engines/engine.py.224: train: epoch 41 | batch 1070/2304 [I 2022-10-27 01:50:21] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:35 [I 2022-10-27 01:50:21] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00246 [I 2022-10-27 01:50:21] ...trap/engines/engine.py.230: loss: 0.10990 [I 2022-10-27 01:50:21] ...optimizers/trijoint.py.151: total_norm: 15.869437 [I 2022-10-27 01:50:26] ...trap/engines/engine.py.224: train: epoch 41 | batch 1080/2304 [I 2022-10-27 01:50:26] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:20 [I 2022-10-27 01:50:26] ...trap/engines/engine.py.229: process: 0.54554 | load: 0.00247 [I 2022-10-27 01:50:26] ...trap/engines/engine.py.230: loss: 0.12010 [I 2022-10-27 01:50:26] ...optimizers/trijoint.py.151: total_norm: 8.118836 [I 2022-10-27 01:50:32] ...trap/engines/engine.py.224: train: epoch 41 | batch 1090/2304 [I 2022-10-27 01:50:32] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:13 [I 2022-10-27 01:50:32] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00245 [I 2022-10-27 01:50:32] ...trap/engines/engine.py.230: loss: 0.17917 [I 2022-10-27 01:50:32] ...optimizers/trijoint.py.151: total_norm: 21.712305 [I 2022-10-27 01:50:38] ...trap/engines/engine.py.224: train: epoch 41 | batch 1100/2304 [I 2022-10-27 01:50:38] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:06 [I 2022-10-27 01:50:38] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00243 [I 2022-10-27 01:50:38] ...trap/engines/engine.py.230: loss: 0.19093 [I 2022-10-27 01:50:38] ...optimizers/trijoint.py.151: total_norm: 18.390549 [I 2022-10-27 01:50:43] ...trap/engines/engine.py.224: train: epoch 41 | batch 1110/2304 [I 2022-10-27 01:50:43] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:01 [I 2022-10-27 01:50:43] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00241 [I 2022-10-27 01:50:43] ...trap/engines/engine.py.230: loss: 0.19645 [I 2022-10-27 01:50:43] ...optimizers/trijoint.py.151: total_norm: 14.391628 [I 2022-10-27 01:50:49] ...trap/engines/engine.py.224: train: epoch 41 | batch 1120/2304 [I 2022-10-27 01:50:49] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:00 [I 2022-10-27 01:50:49] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00243 [I 2022-10-27 01:50:49] ...trap/engines/engine.py.230: loss: 0.24693 [I 2022-10-27 01:50:49] ...optimizers/trijoint.py.151: total_norm: 22.465305 [I 2022-10-27 01:50:54] ...trap/engines/engine.py.224: train: epoch 41 | batch 1130/2304 [I 2022-10-27 01:50:54] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:51 [I 2022-10-27 01:50:54] ...trap/engines/engine.py.229: process: 0.55307 | load: 0.00243 [I 2022-10-27 01:50:54] ...trap/engines/engine.py.230: loss: 0.10346 [I 2022-10-27 01:50:54] ...optimizers/trijoint.py.151: total_norm: 9.298560 [I 2022-10-27 01:51:00] ...trap/engines/engine.py.224: train: epoch 41 | batch 1140/2304 [I 2022-10-27 01:51:00] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:48 [I 2022-10-27 01:51:00] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00243 [I 2022-10-27 01:51:00] ...trap/engines/engine.py.230: loss: 0.16757 [I 2022-10-27 01:51:00] ...optimizers/trijoint.py.151: total_norm: 15.259403 [I 2022-10-27 01:51:05] ...trap/engines/engine.py.224: train: epoch 41 | batch 1150/2304 [I 2022-10-27 01:51:05] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:10:41 [I 2022-10-27 01:51:05] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00253 [I 2022-10-27 01:51:05] ...trap/engines/engine.py.230: loss: 0.15837 [I 2022-10-27 01:51:05] ...optimizers/trijoint.py.151: total_norm: 16.046593 [I 2022-10-27 01:51:11] ...trap/engines/engine.py.224: train: epoch 41 | batch 1160/2304 [I 2022-10-27 01:51:11] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:39 [I 2022-10-27 01:51:11] ...trap/engines/engine.py.229: process: 0.56220 | load: 0.00247 [I 2022-10-27 01:51:11] ...trap/engines/engine.py.230: loss: 0.15808 [I 2022-10-27 01:51:11] ...optimizers/trijoint.py.151: total_norm: 20.484449 [I 2022-10-27 01:51:17] ...trap/engines/engine.py.224: train: epoch 41 | batch 1170/2304 [I 2022-10-27 01:51:17] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:32 [I 2022-10-27 01:51:17] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00245 [I 2022-10-27 01:51:17] ...trap/engines/engine.py.230: loss: 0.12055 [I 2022-10-27 01:51:17] ...optimizers/trijoint.py.151: total_norm: 13.544292 [I 2022-10-27 01:51:22] ...trap/engines/engine.py.224: train: epoch 41 | batch 1180/2304 [I 2022-10-27 01:51:22] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:22 [I 2022-10-27 01:51:22] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00246 [I 2022-10-27 01:51:22] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-27 01:51:22] ...optimizers/trijoint.py.151: total_norm: 10.694432 [I 2022-10-27 01:51:28] ...trap/engines/engine.py.224: train: epoch 41 | batch 1190/2304 [I 2022-10-27 01:51:28] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:18 [I 2022-10-27 01:51:28] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00248 [I 2022-10-27 01:51:28] ...trap/engines/engine.py.230: loss: 0.22292 [I 2022-10-27 01:51:28] ...optimizers/trijoint.py.151: total_norm: 29.216755 [I 2022-10-27 01:51:33] ...trap/engines/engine.py.224: train: epoch 41 | batch 1200/2304 [I 2022-10-27 01:51:33] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:16 [I 2022-10-27 01:51:33] ...trap/engines/engine.py.229: process: 0.56263 | load: 0.00248 [I 2022-10-27 01:51:33] ...trap/engines/engine.py.230: loss: 0.22810 [I 2022-10-27 01:51:33] ...optimizers/trijoint.py.151: total_norm: 29.617149 [I 2022-10-27 01:51:39] ...trap/engines/engine.py.224: train: epoch 41 | batch 1210/2304 [I 2022-10-27 01:51:39] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:12 [I 2022-10-27 01:51:39] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00251 [I 2022-10-27 01:51:39] ...trap/engines/engine.py.230: loss: 0.22995 [I 2022-10-27 01:51:39] ...optimizers/trijoint.py.151: total_norm: 22.349283 [I 2022-10-27 01:51:44] ...trap/engines/engine.py.224: train: epoch 41 | batch 1220/2304 [I 2022-10-27 01:51:44] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:03 [I 2022-10-27 01:51:44] ...trap/engines/engine.py.229: process: 0.55007 | load: 0.00250 [I 2022-10-27 01:51:44] ...trap/engines/engine.py.230: loss: 0.19578 [I 2022-10-27 01:51:44] ...optimizers/trijoint.py.151: total_norm: 17.320904 [I 2022-10-27 01:51:50] ...trap/engines/engine.py.224: train: epoch 41 | batch 1230/2304 [I 2022-10-27 01:51:50] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:09:54 [I 2022-10-27 01:51:50] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00244 [I 2022-10-27 01:51:50] ...trap/engines/engine.py.230: loss: 0.15501 [I 2022-10-27 01:51:50] ...optimizers/trijoint.py.151: total_norm: 10.973569 [I 2022-10-27 01:51:56] ...trap/engines/engine.py.224: train: epoch 41 | batch 1240/2304 [I 2022-10-27 01:51:56] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:09:51 [I 2022-10-27 01:51:56] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00242 [I 2022-10-27 01:51:56] ...trap/engines/engine.py.230: loss: 0.19143 [I 2022-10-27 01:51:56] ...optimizers/trijoint.py.151: total_norm: 21.911755 [I 2022-10-27 01:52:01] ...trap/engines/engine.py.224: train: epoch 41 | batch 1250/2304 [I 2022-10-27 01:52:01] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:09:44 [I 2022-10-27 01:52:01] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00242 [I 2022-10-27 01:52:01] ...trap/engines/engine.py.230: loss: 0.17103 [I 2022-10-27 01:52:01] ...optimizers/trijoint.py.151: total_norm: 15.996536 [I 2022-10-27 01:52:07] ...trap/engines/engine.py.224: train: epoch 41 | batch 1260/2304 [I 2022-10-27 01:52:07] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:09:40 [I 2022-10-27 01:52:07] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00242 [I 2022-10-27 01:52:07] ...trap/engines/engine.py.230: loss: 0.09612 [I 2022-10-27 01:52:07] ...optimizers/trijoint.py.151: total_norm: 14.630529 [I 2022-10-27 01:52:12] ...trap/engines/engine.py.224: train: epoch 41 | batch 1270/2304 [I 2022-10-27 01:52:12] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:33 [I 2022-10-27 01:52:12] ...trap/engines/engine.py.229: process: 0.54136 | load: 0.00242 [I 2022-10-27 01:52:12] ...trap/engines/engine.py.230: loss: 0.14444 [I 2022-10-27 01:52:12] ...optimizers/trijoint.py.151: total_norm: 16.528040 [I 2022-10-27 01:52:18] ...trap/engines/engine.py.224: train: epoch 41 | batch 1280/2304 [I 2022-10-27 01:52:18] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:26 [I 2022-10-27 01:52:18] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00251 [I 2022-10-27 01:52:18] ...trap/engines/engine.py.230: loss: 0.12333 [I 2022-10-27 01:52:18] ...optimizers/trijoint.py.151: total_norm: 11.542328 [I 2022-10-27 01:52:23] ...trap/engines/engine.py.224: train: epoch 41 | batch 1290/2304 [I 2022-10-27 01:52:23] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:23 [I 2022-10-27 01:52:23] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00248 [I 2022-10-27 01:52:23] ...trap/engines/engine.py.230: loss: 0.12845 [I 2022-10-27 01:52:23] ...optimizers/trijoint.py.151: total_norm: 9.843001 [I 2022-10-27 01:52:29] ...trap/engines/engine.py.224: train: epoch 41 | batch 1300/2304 [I 2022-10-27 01:52:29] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:18 [I 2022-10-27 01:52:29] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00244 [I 2022-10-27 01:52:29] ...trap/engines/engine.py.230: loss: 0.13735 [I 2022-10-27 01:52:29] ...optimizers/trijoint.py.151: total_norm: 15.392530 [I 2022-10-27 01:52:35] ...trap/engines/engine.py.224: train: epoch 41 | batch 1310/2304 [I 2022-10-27 01:52:35] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:15 [I 2022-10-27 01:52:35] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00245 [I 2022-10-27 01:52:35] ...trap/engines/engine.py.230: loss: 0.16938 [I 2022-10-27 01:52:35] ...optimizers/trijoint.py.151: total_norm: 27.003532 [I 2022-10-27 01:52:40] ...trap/engines/engine.py.224: train: epoch 41 | batch 1320/2304 [I 2022-10-27 01:52:40] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:08 [I 2022-10-27 01:52:40] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00246 [I 2022-10-27 01:52:40] ...trap/engines/engine.py.230: loss: 0.17409 [I 2022-10-27 01:52:40] ...optimizers/trijoint.py.151: total_norm: 20.840508 [I 2022-10-27 01:52:46] ...trap/engines/engine.py.224: train: epoch 41 | batch 1330/2304 [I 2022-10-27 01:52:46] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:04 [I 2022-10-27 01:52:46] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00247 [I 2022-10-27 01:52:46] ...trap/engines/engine.py.230: loss: 0.21931 [I 2022-10-27 01:52:46] ...optimizers/trijoint.py.151: total_norm: 20.419052 [I 2022-10-27 01:52:51] ...trap/engines/engine.py.224: train: epoch 41 | batch 1340/2304 [I 2022-10-27 01:52:51] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:08:58 [I 2022-10-27 01:52:51] ...trap/engines/engine.py.229: process: 0.56245 | load: 0.00253 [I 2022-10-27 01:52:51] ...trap/engines/engine.py.230: loss: 0.14899 [I 2022-10-27 01:52:51] ...optimizers/trijoint.py.151: total_norm: 19.221548 [I 2022-10-27 01:52:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 1350/2304 [I 2022-10-27 01:52:57] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:52 [I 2022-10-27 01:52:57] ...trap/engines/engine.py.229: process: 0.55831 | load: 0.00246 [I 2022-10-27 01:52:57] ...trap/engines/engine.py.230: loss: 0.14428 [I 2022-10-27 01:52:57] ...optimizers/trijoint.py.151: total_norm: 21.582104 [I 2022-10-27 01:53:02] ...trap/engines/engine.py.224: train: epoch 41 | batch 1360/2304 [I 2022-10-27 01:53:02] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:08:46 [I 2022-10-27 01:53:02] ...trap/engines/engine.py.229: process: 0.56330 | load: 0.00248 [I 2022-10-27 01:53:02] ...trap/engines/engine.py.230: loss: 0.10995 [I 2022-10-27 01:53:02] ...optimizers/trijoint.py.151: total_norm: 13.667179 [I 2022-10-27 01:53:08] ...trap/engines/engine.py.224: train: epoch 41 | batch 1370/2304 [I 2022-10-27 01:53:08] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:35 [I 2022-10-27 01:53:08] ...trap/engines/engine.py.229: process: 0.55144 | load: 0.00244 [I 2022-10-27 01:53:08] ...trap/engines/engine.py.230: loss: 0.19738 [I 2022-10-27 01:53:08] ...optimizers/trijoint.py.151: total_norm: 16.279448 [I 2022-10-27 01:53:14] ...trap/engines/engine.py.224: train: epoch 41 | batch 1380/2304 [I 2022-10-27 01:53:14] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:34 [I 2022-10-27 01:53:14] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00262 [I 2022-10-27 01:53:14] ...trap/engines/engine.py.230: loss: 0.14654 [I 2022-10-27 01:53:14] ...optimizers/trijoint.py.151: total_norm: 42.404552 [I 2022-10-27 01:53:19] ...trap/engines/engine.py.224: train: epoch 41 | batch 1390/2304 [I 2022-10-27 01:53:19] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:29 [I 2022-10-27 01:53:19] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00245 [I 2022-10-27 01:53:19] ...trap/engines/engine.py.230: loss: 0.16993 [I 2022-10-27 01:53:19] ...optimizers/trijoint.py.151: total_norm: 17.705219 [I 2022-10-27 01:53:25] ...trap/engines/engine.py.224: train: epoch 41 | batch 1400/2304 [I 2022-10-27 01:53:25] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:24 [I 2022-10-27 01:53:25] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00242 [I 2022-10-27 01:53:25] ...trap/engines/engine.py.230: loss: 0.14333 [I 2022-10-27 01:53:25] ...optimizers/trijoint.py.151: total_norm: 22.736931 [I 2022-10-27 01:53:30] ...trap/engines/engine.py.224: train: epoch 41 | batch 1410/2304 [I 2022-10-27 01:53:30] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:12 [I 2022-10-27 01:53:30] ...trap/engines/engine.py.229: process: 0.54707 | load: 0.00242 [I 2022-10-27 01:53:30] ...trap/engines/engine.py.230: loss: 0.20448 [I 2022-10-27 01:53:30] ...optimizers/trijoint.py.151: total_norm: 17.916105 [I 2022-10-27 01:53:36] ...trap/engines/engine.py.224: train: epoch 41 | batch 1420/2304 [I 2022-10-27 01:53:36] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:10 [I 2022-10-27 01:53:36] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00245 [I 2022-10-27 01:53:36] ...trap/engines/engine.py.230: loss: 0.17994 [I 2022-10-27 01:53:36] ...optimizers/trijoint.py.151: total_norm: 13.417986 [I 2022-10-27 01:53:42] ...trap/engines/engine.py.224: train: epoch 41 | batch 1430/2304 [I 2022-10-27 01:53:42] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:09 [I 2022-10-27 01:53:42] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00250 [I 2022-10-27 01:53:42] ...trap/engines/engine.py.230: loss: 0.13529 [I 2022-10-27 01:53:42] ...optimizers/trijoint.py.151: total_norm: 11.612898 [I 2022-10-27 01:53:47] ...trap/engines/engine.py.224: train: epoch 41 | batch 1440/2304 [I 2022-10-27 01:53:47] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:01 [I 2022-10-27 01:53:47] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00246 [I 2022-10-27 01:53:47] ...trap/engines/engine.py.230: loss: 0.14474 [I 2022-10-27 01:53:47] ...optimizers/trijoint.py.151: total_norm: 17.432537 [I 2022-10-27 01:53:53] ...trap/engines/engine.py.224: train: epoch 41 | batch 1450/2304 [I 2022-10-27 01:53:53] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:07:57 [I 2022-10-27 01:53:53] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00243 [I 2022-10-27 01:53:53] ...trap/engines/engine.py.230: loss: 0.18688 [I 2022-10-27 01:53:53] ...optimizers/trijoint.py.151: total_norm: 18.401272 [I 2022-10-27 01:53:58] ...trap/engines/engine.py.224: train: epoch 41 | batch 1460/2304 [I 2022-10-27 01:53:58] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:07:51 [I 2022-10-27 01:53:58] ...trap/engines/engine.py.229: process: 0.56224 | load: 0.00244 [I 2022-10-27 01:53:58] ...trap/engines/engine.py.230: loss: 0.28897 [I 2022-10-27 01:53:58] ...optimizers/trijoint.py.151: total_norm: 55.831451 [I 2022-10-27 01:54:04] ...trap/engines/engine.py.224: train: epoch 41 | batch 1470/2304 [I 2022-10-27 01:54:04] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:07:44 [I 2022-10-27 01:54:04] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00247 [I 2022-10-27 01:54:04] ...trap/engines/engine.py.230: loss: 0.18465 [I 2022-10-27 01:54:04] ...optimizers/trijoint.py.151: total_norm: 16.877728 [I 2022-10-27 01:54:09] ...trap/engines/engine.py.224: train: epoch 41 | batch 1480/2304 [I 2022-10-27 01:54:09] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:37 [I 2022-10-27 01:54:09] ...trap/engines/engine.py.229: process: 0.55305 | load: 0.00249 [I 2022-10-27 01:54:09] ...trap/engines/engine.py.230: loss: 0.15954 [I 2022-10-27 01:54:09] ...optimizers/trijoint.py.151: total_norm: 12.806235 [I 2022-10-27 01:54:15] ...trap/engines/engine.py.224: train: epoch 41 | batch 1490/2304 [I 2022-10-27 01:54:15] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:34 [I 2022-10-27 01:54:15] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00244 [I 2022-10-27 01:54:15] ...trap/engines/engine.py.230: loss: 0.15344 [I 2022-10-27 01:54:15] ...optimizers/trijoint.py.151: total_norm: 19.584814 [I 2022-10-27 01:54:21] ...trap/engines/engine.py.224: train: epoch 41 | batch 1500/2304 [I 2022-10-27 01:54:21] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:23 [I 2022-10-27 01:54:21] ...trap/engines/engine.py.229: process: 0.54710 | load: 0.00247 [I 2022-10-27 01:54:21] ...trap/engines/engine.py.230: loss: 0.17754 [I 2022-10-27 01:54:21] ...optimizers/trijoint.py.151: total_norm: 10.140773 [I 2022-10-27 01:54:26] ...trap/engines/engine.py.224: train: epoch 41 | batch 1510/2304 [I 2022-10-27 01:54:26] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:20 [I 2022-10-27 01:54:26] ...trap/engines/engine.py.229: process: 0.56357 | load: 0.00248 [I 2022-10-27 01:54:26] ...trap/engines/engine.py.230: loss: 0.13742 [I 2022-10-27 01:54:26] ...optimizers/trijoint.py.151: total_norm: 8.503686 [I 2022-10-27 01:54:32] ...trap/engines/engine.py.224: train: epoch 41 | batch 1520/2304 [I 2022-10-27 01:54:32] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:16 [I 2022-10-27 01:54:32] ...trap/engines/engine.py.229: process: 0.55902 | load: 0.00244 [I 2022-10-27 01:54:32] ...trap/engines/engine.py.230: loss: 0.11663 [I 2022-10-27 01:54:32] ...optimizers/trijoint.py.151: total_norm: 13.706249 [I 2022-10-27 01:54:37] ...trap/engines/engine.py.224: train: epoch 41 | batch 1530/2304 [I 2022-10-27 01:54:37] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:08 [I 2022-10-27 01:54:37] ...trap/engines/engine.py.229: process: 0.54588 | load: 0.00242 [I 2022-10-27 01:54:37] ...trap/engines/engine.py.230: loss: 0.14812 [I 2022-10-27 01:54:37] ...optimizers/trijoint.py.151: total_norm: 19.763376 [I 2022-10-27 01:54:43] ...trap/engines/engine.py.224: train: epoch 41 | batch 1540/2304 [I 2022-10-27 01:54:43] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:02 [I 2022-10-27 01:54:43] ...trap/engines/engine.py.229: process: 0.55234 | load: 0.00245 [I 2022-10-27 01:54:43] ...trap/engines/engine.py.230: loss: 0.13214 [I 2022-10-27 01:54:43] ...optimizers/trijoint.py.151: total_norm: 21.237106 [I 2022-10-27 01:54:48] ...trap/engines/engine.py.224: train: epoch 41 | batch 1550/2304 [I 2022-10-27 01:54:48] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:06:58 [I 2022-10-27 01:54:48] ...trap/engines/engine.py.229: process: 0.56374 | load: 0.00245 [I 2022-10-27 01:54:48] ...trap/engines/engine.py.230: loss: 0.11158 [I 2022-10-27 01:54:48] ...optimizers/trijoint.py.151: total_norm: 14.609963 [I 2022-10-27 01:54:54] ...trap/engines/engine.py.224: train: epoch 41 | batch 1560/2304 [I 2022-10-27 01:54:54] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:06:54 [I 2022-10-27 01:54:54] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00242 [I 2022-10-27 01:54:54] ...trap/engines/engine.py.230: loss: 0.21845 [I 2022-10-27 01:54:54] ...optimizers/trijoint.py.151: total_norm: 21.003389 [I 2022-10-27 01:55:00] ...trap/engines/engine.py.224: train: epoch 41 | batch 1570/2304 [I 2022-10-27 01:55:00] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:06:50 [I 2022-10-27 01:55:00] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00243 [I 2022-10-27 01:55:00] ...trap/engines/engine.py.230: loss: 0.19931 [I 2022-10-27 01:55:00] ...optimizers/trijoint.py.151: total_norm: 17.973543 [I 2022-10-27 01:55:05] ...trap/engines/engine.py.224: train: epoch 41 | batch 1580/2304 [I 2022-10-27 01:55:05] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:06:41 [I 2022-10-27 01:55:05] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00239 [I 2022-10-27 01:55:05] ...trap/engines/engine.py.230: loss: 0.14399 [I 2022-10-27 01:55:05] ...optimizers/trijoint.py.151: total_norm: 15.353761 [I 2022-10-27 01:55:11] ...trap/engines/engine.py.224: train: epoch 41 | batch 1590/2304 [I 2022-10-27 01:55:11] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:06:38 [I 2022-10-27 01:55:11] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00246 [I 2022-10-27 01:55:11] ...trap/engines/engine.py.230: loss: 0.14743 [I 2022-10-27 01:55:11] ...optimizers/trijoint.py.151: total_norm: 12.771951 [I 2022-10-27 01:55:16] ...trap/engines/engine.py.224: train: epoch 41 | batch 1600/2304 [I 2022-10-27 01:55:16] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:34 [I 2022-10-27 01:55:16] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00244 [I 2022-10-27 01:55:16] ...trap/engines/engine.py.230: loss: 0.15715 [I 2022-10-27 01:55:16] ...optimizers/trijoint.py.151: total_norm: 11.989613 [I 2022-10-27 01:55:22] ...trap/engines/engine.py.224: train: epoch 41 | batch 1610/2304 [I 2022-10-27 01:55:22] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:25 [I 2022-10-27 01:55:22] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00251 [I 2022-10-27 01:55:22] ...trap/engines/engine.py.230: loss: 0.16647 [I 2022-10-27 01:55:22] ...optimizers/trijoint.py.151: total_norm: 19.899099 [I 2022-10-27 01:55:27] ...trap/engines/engine.py.224: train: epoch 41 | batch 1620/2304 [I 2022-10-27 01:55:27] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:17 [I 2022-10-27 01:55:27] ...trap/engines/engine.py.229: process: 0.54741 | load: 0.00251 [I 2022-10-27 01:55:27] ...trap/engines/engine.py.230: loss: 0.23227 [I 2022-10-27 01:55:27] ...optimizers/trijoint.py.151: total_norm: 21.575331 [I 2022-10-27 01:55:33] ...trap/engines/engine.py.224: train: epoch 41 | batch 1630/2304 [I 2022-10-27 01:55:33] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:15 [I 2022-10-27 01:55:33] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00246 [I 2022-10-27 01:55:33] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-27 01:55:33] ...optimizers/trijoint.py.151: total_norm: 19.945786 [I 2022-10-27 01:55:39] ...trap/engines/engine.py.224: train: epoch 41 | batch 1640/2304 [I 2022-10-27 01:55:39] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:09 [I 2022-10-27 01:55:39] ...trap/engines/engine.py.229: process: 0.55061 | load: 0.00242 [I 2022-10-27 01:55:39] ...trap/engines/engine.py.230: loss: 0.23979 [I 2022-10-27 01:55:39] ...optimizers/trijoint.py.151: total_norm: 20.814909 [I 2022-10-27 01:55:44] ...trap/engines/engine.py.224: train: epoch 41 | batch 1650/2304 [I 2022-10-27 01:55:44] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:16 [I 2022-10-27 01:55:44] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00256 [I 2022-10-27 01:55:44] ...trap/engines/engine.py.230: loss: 0.15531 [I 2022-10-27 01:55:44] ...optimizers/trijoint.py.151: total_norm: 17.805540 [I 2022-10-27 01:55:50] ...trap/engines/engine.py.224: train: epoch 41 | batch 1660/2304 [I 2022-10-27 01:55:50] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:01 [I 2022-10-27 01:55:50] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00242 [I 2022-10-27 01:55:50] ...trap/engines/engine.py.230: loss: 0.15445 [I 2022-10-27 01:55:50] ...optimizers/trijoint.py.151: total_norm: 18.668104 [I 2022-10-27 01:55:56] ...trap/engines/engine.py.224: train: epoch 41 | batch 1670/2304 [I 2022-10-27 01:55:56] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:05:53 [I 2022-10-27 01:55:56] ...trap/engines/engine.py.229: process: 0.55524 | load: 0.00244 [I 2022-10-27 01:55:56] ...trap/engines/engine.py.230: loss: 0.13038 [I 2022-10-27 01:55:56] ...optimizers/trijoint.py.151: total_norm: 15.973003 [I 2022-10-27 01:56:01] ...trap/engines/engine.py.224: train: epoch 41 | batch 1680/2304 [I 2022-10-27 01:56:01] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:05:47 [I 2022-10-27 01:56:01] ...trap/engines/engine.py.229: process: 0.56025 | load: 0.00244 [I 2022-10-27 01:56:01] ...trap/engines/engine.py.230: loss: 0.20407 [I 2022-10-27 01:56:01] ...optimizers/trijoint.py.151: total_norm: 59.498741 [I 2022-10-27 01:56:07] ...trap/engines/engine.py.224: train: epoch 41 | batch 1690/2304 [I 2022-10-27 01:56:07] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:05:41 [I 2022-10-27 01:56:07] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00243 [I 2022-10-27 01:56:07] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-27 01:56:07] ...optimizers/trijoint.py.151: total_norm: 60.888302 [I 2022-10-27 01:56:12] ...trap/engines/engine.py.224: train: epoch 41 | batch 1700/2304 [I 2022-10-27 01:56:12] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:37 [I 2022-10-27 01:56:12] ...trap/engines/engine.py.229: process: 0.56002 | load: 0.00245 [I 2022-10-27 01:56:12] ...trap/engines/engine.py.230: loss: 0.14142 [I 2022-10-27 01:56:12] ...optimizers/trijoint.py.151: total_norm: 12.538176 [I 2022-10-27 01:56:18] ...trap/engines/engine.py.224: train: epoch 41 | batch 1710/2304 [I 2022-10-27 01:56:18] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:28 [I 2022-10-27 01:56:18] ...trap/engines/engine.py.229: process: 0.55019 | load: 0.00250 [I 2022-10-27 01:56:18] ...trap/engines/engine.py.230: loss: 0.17752 [I 2022-10-27 01:56:18] ...optimizers/trijoint.py.151: total_norm: 14.868531 [I 2022-10-27 01:56:24] ...trap/engines/engine.py.224: train: epoch 41 | batch 1720/2304 [I 2022-10-27 01:56:24] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:26 [I 2022-10-27 01:56:24] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00255 [I 2022-10-27 01:56:24] ...trap/engines/engine.py.230: loss: 0.16592 [I 2022-10-27 01:56:24] ...optimizers/trijoint.py.151: total_norm: 15.255678 [I 2022-10-27 01:56:29] ...trap/engines/engine.py.224: train: epoch 41 | batch 1730/2304 [I 2022-10-27 01:56:29] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:18 [I 2022-10-27 01:56:29] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00250 [I 2022-10-27 01:56:29] ...trap/engines/engine.py.230: loss: 0.24748 [I 2022-10-27 01:56:29] ...optimizers/trijoint.py.151: total_norm: 19.176823 [I 2022-10-27 01:56:35] ...trap/engines/engine.py.224: train: epoch 41 | batch 1740/2304 [I 2022-10-27 01:56:35] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:15 [I 2022-10-27 01:56:35] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00249 [I 2022-10-27 01:56:35] ...trap/engines/engine.py.230: loss: 0.21171 [I 2022-10-27 01:56:35] ...optimizers/trijoint.py.151: total_norm: 12.116643 [I 2022-10-27 01:56:40] ...trap/engines/engine.py.224: train: epoch 41 | batch 1750/2304 [I 2022-10-27 01:56:40] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:10 [I 2022-10-27 01:56:40] ...trap/engines/engine.py.229: process: 0.56126 | load: 0.00244 [I 2022-10-27 01:56:40] ...trap/engines/engine.py.230: loss: 0.17489 [I 2022-10-27 01:56:40] ...optimizers/trijoint.py.151: total_norm: 19.134298 [I 2022-10-27 01:56:46] ...trap/engines/engine.py.224: train: epoch 41 | batch 1760/2304 [I 2022-10-27 01:56:46] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:04:59 [I 2022-10-27 01:56:46] ...trap/engines/engine.py.229: process: 0.53792 | load: 0.00245 [I 2022-10-27 01:56:46] ...trap/engines/engine.py.230: loss: 0.15851 [I 2022-10-27 01:56:46] ...optimizers/trijoint.py.151: total_norm: 16.603941 [I 2022-10-27 01:56:51] ...trap/engines/engine.py.224: train: epoch 41 | batch 1770/2304 [I 2022-10-27 01:56:51] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:04:55 [I 2022-10-27 01:56:51] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00247 [I 2022-10-27 01:56:51] ...trap/engines/engine.py.230: loss: 0.17842 [I 2022-10-27 01:56:51] ...optimizers/trijoint.py.151: total_norm: 15.811173 [I 2022-10-27 01:56:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 1780/2304 [I 2022-10-27 01:56:57] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:04:51 [I 2022-10-27 01:56:57] ...trap/engines/engine.py.229: process: 0.56490 | load: 0.00241 [I 2022-10-27 01:56:57] ...trap/engines/engine.py.230: loss: 0.15029 [I 2022-10-27 01:56:57] ...optimizers/trijoint.py.151: total_norm: 23.031252 [I 2022-10-27 01:57:03] ...trap/engines/engine.py.224: train: epoch 41 | batch 1790/2304 [I 2022-10-27 01:57:03] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:04:47 [I 2022-10-27 01:57:03] ...trap/engines/engine.py.229: process: 0.57061 | load: 0.00240 [I 2022-10-27 01:57:03] ...trap/engines/engine.py.230: loss: 0.27488 [I 2022-10-27 01:57:03] ...optimizers/trijoint.py.151: total_norm: 17.526228 [I 2022-10-27 01:57:08] ...trap/engines/engine.py.224: train: epoch 41 | batch 1800/2304 [I 2022-10-27 01:57:08] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:38 [I 2022-10-27 01:57:08] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00244 [I 2022-10-27 01:57:08] ...trap/engines/engine.py.230: loss: 0.12469 [I 2022-10-27 01:57:08] ...optimizers/trijoint.py.151: total_norm: 11.726562 [I 2022-10-27 01:57:14] ...trap/engines/engine.py.224: train: epoch 41 | batch 1810/2304 [I 2022-10-27 01:57:14] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:37 [I 2022-10-27 01:57:14] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00245 [I 2022-10-27 01:57:14] ...trap/engines/engine.py.230: loss: 0.13299 [I 2022-10-27 01:57:14] ...optimizers/trijoint.py.151: total_norm: 15.304440 [I 2022-10-27 01:57:19] ...trap/engines/engine.py.224: train: epoch 41 | batch 1820/2304 [I 2022-10-27 01:57:19] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:29 [I 2022-10-27 01:57:19] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00241 [I 2022-10-27 01:57:19] ...trap/engines/engine.py.230: loss: 0.17311 [I 2022-10-27 01:57:19] ...optimizers/trijoint.py.151: total_norm: 24.217960 [I 2022-10-27 01:57:25] ...trap/engines/engine.py.224: train: epoch 41 | batch 1830/2304 [I 2022-10-27 01:57:25] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:24 [I 2022-10-27 01:57:25] ...trap/engines/engine.py.229: process: 0.56097 | load: 0.00250 [I 2022-10-27 01:57:25] ...trap/engines/engine.py.230: loss: 0.16643 [I 2022-10-27 01:57:25] ...optimizers/trijoint.py.151: total_norm: 17.355125 [I 2022-10-27 01:57:30] ...trap/engines/engine.py.224: train: epoch 41 | batch 1840/2304 [I 2022-10-27 01:57:30] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:17 [I 2022-10-27 01:57:30] ...trap/engines/engine.py.229: process: 0.55623 | load: 0.00247 [I 2022-10-27 01:57:30] ...trap/engines/engine.py.230: loss: 0.23571 [I 2022-10-27 01:57:30] ...optimizers/trijoint.py.151: total_norm: 15.491179 [I 2022-10-27 01:57:36] ...trap/engines/engine.py.224: train: epoch 41 | batch 1850/2304 [I 2022-10-27 01:57:36] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:11 [I 2022-10-27 01:57:36] ...trap/engines/engine.py.229: process: 0.54368 | load: 0.00246 [I 2022-10-27 01:57:36] ...trap/engines/engine.py.230: loss: 0.23331 [I 2022-10-27 01:57:36] ...optimizers/trijoint.py.151: total_norm: 42.713440 [I 2022-10-27 01:57:42] ...trap/engines/engine.py.224: train: epoch 41 | batch 1860/2304 [I 2022-10-27 01:57:42] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:07 [I 2022-10-27 01:57:42] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00247 [I 2022-10-27 01:57:42] ...trap/engines/engine.py.230: loss: 0.16443 [I 2022-10-27 01:57:42] ...optimizers/trijoint.py.151: total_norm: 19.388815 [I 2022-10-27 01:57:47] ...trap/engines/engine.py.224: train: epoch 41 | batch 1870/2304 [I 2022-10-27 01:57:47] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:01 [I 2022-10-27 01:57:47] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00245 [I 2022-10-27 01:57:47] ...trap/engines/engine.py.230: loss: 0.12236 [I 2022-10-27 01:57:47] ...optimizers/trijoint.py.151: total_norm: 11.128129 [I 2022-10-27 01:57:53] ...trap/engines/engine.py.224: train: epoch 41 | batch 1880/2304 [I 2022-10-27 01:57:53] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:03:56 [I 2022-10-27 01:57:53] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00245 [I 2022-10-27 01:57:53] ...trap/engines/engine.py.230: loss: 0.18694 [I 2022-10-27 01:57:53] ...optimizers/trijoint.py.151: total_norm: 14.261541 [I 2022-10-27 01:57:58] ...trap/engines/engine.py.224: train: epoch 41 | batch 1890/2304 [I 2022-10-27 01:57:58] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:03:51 [I 2022-10-27 01:57:58] ...trap/engines/engine.py.229: process: 0.56183 | load: 0.00242 [I 2022-10-27 01:57:58] ...trap/engines/engine.py.230: loss: 0.24690 [I 2022-10-27 01:57:58] ...optimizers/trijoint.py.151: total_norm: 24.242750 [I 2022-10-27 01:58:04] ...trap/engines/engine.py.224: train: epoch 41 | batch 1900/2304 [I 2022-10-27 01:58:04] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:03:43 [I 2022-10-27 01:58:04] ...trap/engines/engine.py.229: process: 0.55155 | load: 0.00248 [I 2022-10-27 01:58:04] ...trap/engines/engine.py.230: loss: 0.18877 [I 2022-10-27 01:58:04] ...optimizers/trijoint.py.151: total_norm: 13.523723 [I 2022-10-27 01:58:10] ...trap/engines/engine.py.224: train: epoch 41 | batch 1910/2304 [I 2022-10-27 01:58:10] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:03:41 [I 2022-10-27 01:58:10] ...trap/engines/engine.py.229: process: 0.56833 | load: 0.00244 [I 2022-10-27 01:58:10] ...trap/engines/engine.py.230: loss: 0.15586 [I 2022-10-27 01:58:10] ...optimizers/trijoint.py.151: total_norm: 24.227301 [I 2022-10-27 01:58:15] ...trap/engines/engine.py.224: train: epoch 41 | batch 1920/2304 [I 2022-10-27 01:58:15] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:44 [I 2022-10-27 01:58:15] ...trap/engines/engine.py.229: process: 0.66250 | load: 0.01743 [I 2022-10-27 01:58:15] ...trap/engines/engine.py.230: loss: 0.09709 [I 2022-10-27 01:58:15] ...optimizers/trijoint.py.151: total_norm: 19.973381 [I 2022-10-27 01:58:21] ...trap/engines/engine.py.224: train: epoch 41 | batch 1930/2304 [I 2022-10-27 01:58:21] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:29 [I 2022-10-27 01:58:21] ...trap/engines/engine.py.229: process: 0.55042 | load: 0.00246 [I 2022-10-27 01:58:21] ...trap/engines/engine.py.230: loss: 0.17672 [I 2022-10-27 01:58:21] ...optimizers/trijoint.py.151: total_norm: 28.101664 [I 2022-10-27 01:58:26] ...trap/engines/engine.py.224: train: epoch 41 | batch 1940/2304 [I 2022-10-27 01:58:26] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:22 [I 2022-10-27 01:58:26] ...trap/engines/engine.py.229: process: 0.55491 | load: 0.00245 [I 2022-10-27 01:58:26] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-27 01:58:26] ...optimizers/trijoint.py.151: total_norm: 17.481489 [I 2022-10-27 01:58:32] ...trap/engines/engine.py.224: train: epoch 41 | batch 1950/2304 [I 2022-10-27 01:58:32] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:16 [I 2022-10-27 01:58:32] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00240 [I 2022-10-27 01:58:32] ...trap/engines/engine.py.230: loss: 0.15099 [I 2022-10-27 01:58:32] ...optimizers/trijoint.py.151: total_norm: 28.895622 [I 2022-10-27 01:58:38] ...trap/engines/engine.py.224: train: epoch 41 | batch 1960/2304 [I 2022-10-27 01:58:38] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:10 [I 2022-10-27 01:58:38] ...trap/engines/engine.py.229: process: 0.54705 | load: 0.00244 [I 2022-10-27 01:58:38] ...trap/engines/engine.py.230: loss: 0.14672 [I 2022-10-27 01:58:38] ...optimizers/trijoint.py.151: total_norm: 18.027514 [I 2022-10-27 01:58:43] ...trap/engines/engine.py.224: train: epoch 41 | batch 1970/2304 [I 2022-10-27 01:58:43] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:04 [I 2022-10-27 01:58:43] ...trap/engines/engine.py.229: process: 0.54840 | load: 0.00242 [I 2022-10-27 01:58:43] ...trap/engines/engine.py.230: loss: 0.13106 [I 2022-10-27 01:58:43] ...optimizers/trijoint.py.151: total_norm: 10.641023 [I 2022-10-27 01:58:49] ...trap/engines/engine.py.224: train: epoch 41 | batch 1980/2304 [I 2022-10-27 01:58:49] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:02:59 [I 2022-10-27 01:58:49] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00242 [I 2022-10-27 01:58:49] ...trap/engines/engine.py.230: loss: 0.21045 [I 2022-10-27 01:58:49] ...optimizers/trijoint.py.151: total_norm: 14.436865 [I 2022-10-27 01:58:54] ...trap/engines/engine.py.224: train: epoch 41 | batch 1990/2304 [I 2022-10-27 01:58:54] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:02:53 [I 2022-10-27 01:58:54] ...trap/engines/engine.py.229: process: 0.54710 | load: 0.00247 [I 2022-10-27 01:58:54] ...trap/engines/engine.py.230: loss: 0.15112 [I 2022-10-27 01:58:54] ...optimizers/trijoint.py.151: total_norm: 12.953783 [I 2022-10-27 01:59:00] ...trap/engines/engine.py.224: train: epoch 41 | batch 2000/2304 [I 2022-10-27 01:59:00] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:02:49 [I 2022-10-27 01:59:00] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00243 [I 2022-10-27 01:59:00] ...trap/engines/engine.py.230: loss: 0.14096 [I 2022-10-27 01:59:00] ...optimizers/trijoint.py.151: total_norm: 13.968188 [I 2022-10-27 01:59:05] ...trap/engines/engine.py.224: train: epoch 41 | batch 2010/2304 [I 2022-10-27 01:59:05] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:02:44 [I 2022-10-27 01:59:05] ...trap/engines/engine.py.229: process: 0.56957 | load: 0.00240 [I 2022-10-27 01:59:05] ...trap/engines/engine.py.230: loss: 0.16180 [I 2022-10-27 01:59:05] ...optimizers/trijoint.py.151: total_norm: 19.614222 [I 2022-10-27 01:59:11] ...trap/engines/engine.py.224: train: epoch 41 | batch 2020/2304 [I 2022-10-27 01:59:11] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:37 [I 2022-10-27 01:59:11] ...trap/engines/engine.py.229: process: 0.56219 | load: 0.00242 [I 2022-10-27 01:59:11] ...trap/engines/engine.py.230: loss: 0.14442 [I 2022-10-27 01:59:11] ...optimizers/trijoint.py.151: total_norm: 13.841167 [I 2022-10-27 01:59:16] ...trap/engines/engine.py.224: train: epoch 41 | batch 2030/2304 [I 2022-10-27 01:59:16] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:32 [I 2022-10-27 01:59:16] ...trap/engines/engine.py.229: process: 0.56173 | load: 0.00252 [I 2022-10-27 01:59:16] ...trap/engines/engine.py.230: loss: 0.18798 [I 2022-10-27 01:59:16] ...optimizers/trijoint.py.151: total_norm: 18.440386 [I 2022-10-27 01:59:22] ...trap/engines/engine.py.224: train: epoch 41 | batch 2040/2304 [I 2022-10-27 01:59:22] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:31 [I 2022-10-27 01:59:22] ...trap/engines/engine.py.229: process: 0.56841 | load: 0.00250 [I 2022-10-27 01:59:22] ...trap/engines/engine.py.230: loss: 0.22264 [I 2022-10-27 01:59:22] ...optimizers/trijoint.py.151: total_norm: 12.443930 [I 2022-10-27 01:59:28] ...trap/engines/engine.py.224: train: epoch 41 | batch 2050/2304 [I 2022-10-27 01:59:28] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:22 [I 2022-10-27 01:59:28] ...trap/engines/engine.py.229: process: 0.55807 | load: 0.00248 [I 2022-10-27 01:59:28] ...trap/engines/engine.py.230: loss: 0.18025 [I 2022-10-27 01:59:28] ...optimizers/trijoint.py.151: total_norm: 23.704699 [I 2022-10-27 01:59:33] ...trap/engines/engine.py.224: train: epoch 41 | batch 2060/2304 [I 2022-10-27 01:59:33] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:16 [I 2022-10-27 01:59:33] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00243 [I 2022-10-27 01:59:33] ...trap/engines/engine.py.230: loss: 0.18199 [I 2022-10-27 01:59:33] ...optimizers/trijoint.py.151: total_norm: 25.637625 [I 2022-10-27 01:59:39] ...trap/engines/engine.py.224: train: epoch 41 | batch 2070/2304 [I 2022-10-27 01:59:39] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:09 [I 2022-10-27 01:59:39] ...trap/engines/engine.py.229: process: 0.56181 | load: 0.00244 [I 2022-10-27 01:59:39] ...trap/engines/engine.py.230: loss: 0.15046 [I 2022-10-27 01:59:39] ...optimizers/trijoint.py.151: total_norm: 19.891979 [I 2022-10-27 01:59:45] ...trap/engines/engine.py.224: train: epoch 41 | batch 2080/2304 [I 2022-10-27 01:59:45] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:04 [I 2022-10-27 01:59:45] ...trap/engines/engine.py.229: process: 0.56192 | load: 0.00245 [I 2022-10-27 01:59:45] ...trap/engines/engine.py.230: loss: 0.23196 [I 2022-10-27 01:59:45] ...optimizers/trijoint.py.151: total_norm: 17.238941 [I 2022-10-27 01:59:50] ...trap/engines/engine.py.224: train: epoch 41 | batch 2090/2304 [I 2022-10-27 01:59:50] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:01:59 [I 2022-10-27 01:59:50] ...trap/engines/engine.py.229: process: 0.57012 | load: 0.00244 [I 2022-10-27 01:59:50] ...trap/engines/engine.py.230: loss: 0.16411 [I 2022-10-27 01:59:50] ...optimizers/trijoint.py.151: total_norm: 21.564407 [I 2022-10-27 01:59:56] ...trap/engines/engine.py.224: train: epoch 41 | batch 2100/2304 [I 2022-10-27 01:59:56] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:01:52 [I 2022-10-27 01:59:56] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00245 [I 2022-10-27 01:59:56] ...trap/engines/engine.py.230: loss: 0.12832 [I 2022-10-27 01:59:56] ...optimizers/trijoint.py.151: total_norm: 12.172079 [I 2022-10-27 02:00:01] ...trap/engines/engine.py.224: train: epoch 41 | batch 2110/2304 [I 2022-10-27 02:00:01] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:01:47 [I 2022-10-27 02:00:01] ...trap/engines/engine.py.229: process: 0.55118 | load: 0.00249 [I 2022-10-27 02:00:01] ...trap/engines/engine.py.230: loss: 0.36262 [I 2022-10-27 02:00:01] ...optimizers/trijoint.py.151: total_norm: 23.453424 [I 2022-10-27 02:00:07] ...trap/engines/engine.py.224: train: epoch 41 | batch 2120/2304 [I 2022-10-27 02:00:07] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:01:41 [I 2022-10-27 02:00:07] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00246 [I 2022-10-27 02:00:07] ...trap/engines/engine.py.230: loss: 0.22461 [I 2022-10-27 02:00:07] ...optimizers/trijoint.py.151: total_norm: 25.541861 [I 2022-10-27 02:00:12] ...trap/engines/engine.py.224: train: epoch 41 | batch 2130/2304 [I 2022-10-27 02:00:12] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:36 [I 2022-10-27 02:00:12] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00243 [I 2022-10-27 02:00:12] ...trap/engines/engine.py.230: loss: 0.17311 [I 2022-10-27 02:00:12] ...optimizers/trijoint.py.151: total_norm: 12.053789 [I 2022-10-27 02:00:18] ...trap/engines/engine.py.224: train: epoch 41 | batch 2140/2304 [I 2022-10-27 02:00:18] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:31 [I 2022-10-27 02:00:18] ...trap/engines/engine.py.229: process: 0.56048 | load: 0.00242 [I 2022-10-27 02:00:18] ...trap/engines/engine.py.230: loss: 0.15601 [I 2022-10-27 02:00:18] ...optimizers/trijoint.py.151: total_norm: 26.490499 [I 2022-10-27 02:00:24] ...trap/engines/engine.py.224: train: epoch 41 | batch 2150/2304 [I 2022-10-27 02:00:24] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:28 [I 2022-10-27 02:00:24] ...trap/engines/engine.py.229: process: 0.64890 | load: 0.00246 [I 2022-10-27 02:00:24] ...trap/engines/engine.py.230: loss: 0.14785 [I 2022-10-27 02:00:24] ...optimizers/trijoint.py.151: total_norm: 10.972051 [I 2022-10-27 02:00:29] ...trap/engines/engine.py.224: train: epoch 41 | batch 2160/2304 [I 2022-10-27 02:00:29] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:19 [I 2022-10-27 02:00:29] ...trap/engines/engine.py.229: process: 0.54363 | load: 0.00244 [I 2022-10-27 02:00:29] ...trap/engines/engine.py.230: loss: 0.18990 [I 2022-10-27 02:00:29] ...optimizers/trijoint.py.151: total_norm: 13.248755 [I 2022-10-27 02:00:35] ...trap/engines/engine.py.224: train: epoch 41 | batch 2170/2304 [I 2022-10-27 02:00:35] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:14 [I 2022-10-27 02:00:35] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00244 [I 2022-10-27 02:00:35] ...trap/engines/engine.py.230: loss: 0.14456 [I 2022-10-27 02:00:35] ...optimizers/trijoint.py.151: total_norm: 14.594362 [I 2022-10-27 02:00:40] ...trap/engines/engine.py.224: train: epoch 41 | batch 2180/2304 [I 2022-10-27 02:00:40] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:08 [I 2022-10-27 02:00:40] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00246 [I 2022-10-27 02:00:40] ...trap/engines/engine.py.230: loss: 0.20721 [I 2022-10-27 02:00:40] ...optimizers/trijoint.py.151: total_norm: 20.201040 [I 2022-10-27 02:00:46] ...trap/engines/engine.py.224: train: epoch 41 | batch 2190/2304 [I 2022-10-27 02:00:46] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:03 [I 2022-10-27 02:00:46] ...trap/engines/engine.py.229: process: 0.56138 | load: 0.00244 [I 2022-10-27 02:00:46] ...trap/engines/engine.py.230: loss: 0.14224 [I 2022-10-27 02:00:46] ...optimizers/trijoint.py.151: total_norm: 17.433296 [I 2022-10-27 02:00:52] ...trap/engines/engine.py.224: train: epoch 41 | batch 2200/2304 [I 2022-10-27 02:00:52] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:00:57 [I 2022-10-27 02:00:52] ...trap/engines/engine.py.229: process: 0.56201 | load: 0.00241 [I 2022-10-27 02:00:52] ...trap/engines/engine.py.230: loss: 0.11496 [I 2022-10-27 02:00:52] ...optimizers/trijoint.py.151: total_norm: 12.409555 [I 2022-10-27 02:00:57] ...trap/engines/engine.py.224: train: epoch 41 | batch 2210/2304 [I 2022-10-27 02:00:57] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:00:52 [I 2022-10-27 02:00:57] ...trap/engines/engine.py.229: process: 0.55130 | load: 0.00242 [I 2022-10-27 02:00:57] ...trap/engines/engine.py.230: loss: 0.11969 [I 2022-10-27 02:00:57] ...optimizers/trijoint.py.151: total_norm: 9.708376 [I 2022-10-27 02:01:03] ...trap/engines/engine.py.224: train: epoch 41 | batch 2220/2304 [I 2022-10-27 02:01:03] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:00:46 [I 2022-10-27 02:01:03] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00248 [I 2022-10-27 02:01:03] ...trap/engines/engine.py.230: loss: 0.17997 [I 2022-10-27 02:01:03] ...optimizers/trijoint.py.151: total_norm: 20.646290 [I 2022-10-27 02:01:08] ...trap/engines/engine.py.224: train: epoch 41 | batch 2230/2304 [I 2022-10-27 02:01:08] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:00:41 [I 2022-10-27 02:01:08] ...trap/engines/engine.py.229: process: 0.54837 | load: 0.00244 [I 2022-10-27 02:01:08] ...trap/engines/engine.py.230: loss: 0.18270 [I 2022-10-27 02:01:08] ...optimizers/trijoint.py.151: total_norm: 19.650105 [I 2022-10-27 02:01:14] ...trap/engines/engine.py.224: train: epoch 41 | batch 2240/2304 [I 2022-10-27 02:01:14] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:35 [I 2022-10-27 02:01:14] ...trap/engines/engine.py.229: process: 0.54220 | load: 0.00243 [I 2022-10-27 02:01:14] ...trap/engines/engine.py.230: loss: 0.13164 [I 2022-10-27 02:01:14] ...optimizers/trijoint.py.151: total_norm: 8.595332 [I 2022-10-27 02:01:19] ...trap/engines/engine.py.224: train: epoch 41 | batch 2250/2304 [I 2022-10-27 02:01:19] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:29 [I 2022-10-27 02:01:19] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00244 [I 2022-10-27 02:01:19] ...trap/engines/engine.py.230: loss: 0.14614 [I 2022-10-27 02:01:19] ...optimizers/trijoint.py.151: total_norm: 9.433561 [I 2022-10-27 02:01:25] ...trap/engines/engine.py.224: train: epoch 41 | batch 2260/2304 [I 2022-10-27 02:01:25] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:24 [I 2022-10-27 02:01:25] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00243 [I 2022-10-27 02:01:25] ...trap/engines/engine.py.230: loss: 0.21416 [I 2022-10-27 02:01:25] ...optimizers/trijoint.py.151: total_norm: 18.984863 [I 2022-10-27 02:01:30] ...trap/engines/engine.py.224: train: epoch 41 | batch 2270/2304 [I 2022-10-27 02:01:30] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:18 [I 2022-10-27 02:01:30] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00245 [I 2022-10-27 02:01:30] ...trap/engines/engine.py.230: loss: 0.18441 [I 2022-10-27 02:01:30] ...optimizers/trijoint.py.151: total_norm: 23.582859 [I 2022-10-27 02:01:36] ...trap/engines/engine.py.224: train: epoch 41 | batch 2280/2304 [I 2022-10-27 02:01:36] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:13 [I 2022-10-27 02:01:36] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00245 [I 2022-10-27 02:01:36] ...trap/engines/engine.py.230: loss: 0.15580 [I 2022-10-27 02:01:36] ...optimizers/trijoint.py.151: total_norm: 18.730419 [I 2022-10-27 02:01:42] ...trap/engines/engine.py.224: train: epoch 41 | batch 2290/2304 [I 2022-10-27 02:01:42] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:07 [I 2022-10-27 02:01:42] ...trap/engines/engine.py.229: process: 0.56303 | load: 0.00242 [I 2022-10-27 02:01:42] ...trap/engines/engine.py.230: loss: 0.18250 [I 2022-10-27 02:01:42] ...optimizers/trijoint.py.151: total_norm: 14.946813 [I 2022-10-27 02:01:47] ...trap/engines/engine.py.224: train: epoch 41 | batch 2300/2304 [I 2022-10-27 02:01:47] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:02 [I 2022-10-27 02:01:47] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00014 [I 2022-10-27 02:01:47] ...trap/engines/engine.py.230: loss: 0.16992 [I 2022-10-27 02:01:47] ...optimizers/trijoint.py.151: total_norm: 16.303234 [I 2022-10-27 02:01:49] ...trap/engines/engine.py.224: train: epoch 41 | batch 2304/2304 [I 2022-10-27 02:01:49] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:00 [I 2022-10-27 02:01:49] ...trap/engines/engine.py.229: process: 0.54493 | load: 0.00015 [I 2022-10-27 02:01:49] ...trap/engines/engine.py.230: loss: 0.11854 [I 2022-10-27 02:01:49] ...optimizers/trijoint.py.151: total_norm: 11.598368 [S 2022-10-27 02:01:50] ...trap/engines/engine.py.129: train_epoch.epoch: 41 [S 2022-10-27 02:01:50] ...trap/engines/engine.py.129: train_epoch.loss: 0.17043925753901681 [I 2022-10-27 02:01:50] ...s/criterions/triplet.py.84: margin = 0.2600000000000002 [I 2022-10-27 02:01:51] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 41 [I 2022-10-27 02:01:51] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 02:01:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 02:01:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 02:01:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 02:01:54] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 02:01:55] ...trap/engines/engine.py.321: eval: epoch 41 | batch 0/510 [I 2022-10-27 02:01:55] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:29:55 [I 2022-10-27 02:01:55] ...trap/engines/engine.py.326: process: 3.52007 | load: 2.95676 [I 2022-10-27 02:01:58] ...trap/engines/engine.py.321: eval: epoch 41 | batch 10/510 [I 2022-10-27 02:01:58] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:38 [I 2022-10-27 02:01:58] ...trap/engines/engine.py.326: process: 0.33292 | load: 0.00021 [I 2022-10-27 02:02:01] ...trap/engines/engine.py.321: eval: epoch 41 | batch 20/510 [I 2022-10-27 02:02:01] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:03 [I 2022-10-27 02:02:01] ...trap/engines/engine.py.326: process: 0.32623 | load: 0.00016 [I 2022-10-27 02:02:05] ...trap/engines/engine.py.321: eval: epoch 41 | batch 30/510 [I 2022-10-27 02:02:05] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:41 [I 2022-10-27 02:02:05] ...trap/engines/engine.py.326: process: 0.32761 | load: 0.00016 [I 2022-10-27 02:02:08] ...trap/engines/engine.py.321: eval: epoch 41 | batch 40/510 [I 2022-10-27 02:02:08] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:36 [I 2022-10-27 02:02:08] ...trap/engines/engine.py.326: process: 0.32182 | load: 0.00017 [I 2022-10-27 02:02:11] ...trap/engines/engine.py.321: eval: epoch 41 | batch 50/510 [I 2022-10-27 02:02:11] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:33 [I 2022-10-27 02:02:11] ...trap/engines/engine.py.326: process: 0.32730 | load: 0.00015 [I 2022-10-27 02:02:15] ...trap/engines/engine.py.321: eval: epoch 41 | batch 60/510 [I 2022-10-27 02:02:15] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:28 [I 2022-10-27 02:02:15] ...trap/engines/engine.py.326: process: 0.32527 | load: 0.00018 [I 2022-10-27 02:02:18] ...trap/engines/engine.py.321: eval: epoch 41 | batch 70/510 [I 2022-10-27 02:02:18] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:26 [I 2022-10-27 02:02:18] ...trap/engines/engine.py.326: process: 0.34016 | load: 0.00016 [I 2022-10-27 02:02:21] ...trap/engines/engine.py.321: eval: epoch 41 | batch 80/510 [I 2022-10-27 02:02:21] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:22 [I 2022-10-27 02:02:21] ...trap/engines/engine.py.326: process: 0.32086 | load: 0.00019 [I 2022-10-27 02:02:25] ...trap/engines/engine.py.321: eval: epoch 41 | batch 90/510 [I 2022-10-27 02:02:25] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:18 [I 2022-10-27 02:02:25] ...trap/engines/engine.py.326: process: 0.33000 | load: 0.00017 [I 2022-10-27 02:02:28] ...trap/engines/engine.py.321: eval: epoch 41 | batch 100/510 [I 2022-10-27 02:02:28] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:12 [I 2022-10-27 02:02:28] ...trap/engines/engine.py.326: process: 0.29018 | load: 0.00018 [I 2022-10-27 02:02:31] ...trap/engines/engine.py.321: eval: epoch 41 | batch 110/510 [I 2022-10-27 02:02:31] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-27 02:02:31] ...trap/engines/engine.py.326: process: 0.29392 | load: 0.00019 [I 2022-10-27 02:02:34] ...trap/engines/engine.py.321: eval: epoch 41 | batch 120/510 [I 2022-10-27 02:02:34] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:54 [I 2022-10-27 02:02:34] ...trap/engines/engine.py.326: process: 0.29477 | load: 0.00019 [I 2022-10-27 02:02:37] ...trap/engines/engine.py.321: eval: epoch 41 | batch 130/510 [I 2022-10-27 02:02:37] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-27 02:02:37] ...trap/engines/engine.py.326: process: 0.30380 | load: 0.00019 [I 2022-10-27 02:02:40] ...trap/engines/engine.py.321: eval: epoch 41 | batch 140/510 [I 2022-10-27 02:02:40] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:48 [I 2022-10-27 02:02:40] ...trap/engines/engine.py.326: process: 0.29251 | load: 0.00021 [I 2022-10-27 02:02:42] ...trap/engines/engine.py.321: eval: epoch 41 | batch 150/510 [I 2022-10-27 02:02:42] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:44 [I 2022-10-27 02:02:42] ...trap/engines/engine.py.326: process: 0.28997 | load: 0.00018 [I 2022-10-27 02:02:45] ...trap/engines/engine.py.321: eval: epoch 41 | batch 160/510 [I 2022-10-27 02:02:45] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-27 02:02:45] ...trap/engines/engine.py.326: process: 0.28595 | load: 0.00024 [I 2022-10-27 02:02:48] ...trap/engines/engine.py.321: eval: epoch 41 | batch 170/510 [I 2022-10-27 02:02:48] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-27 02:02:48] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00017 [I 2022-10-27 02:02:51] ...trap/engines/engine.py.321: eval: epoch 41 | batch 180/510 [I 2022-10-27 02:02:51] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 02:02:51] ...trap/engines/engine.py.326: process: 0.29448 | load: 0.00015 [I 2022-10-27 02:02:54] ...trap/engines/engine.py.321: eval: epoch 41 | batch 190/510 [I 2022-10-27 02:02:54] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 02:02:54] ...trap/engines/engine.py.326: process: 0.28918 | load: 0.00017 [I 2022-10-27 02:02:57] ...trap/engines/engine.py.321: eval: epoch 41 | batch 200/510 [I 2022-10-27 02:02:57] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:30 [I 2022-10-27 02:02:57] ...trap/engines/engine.py.326: process: 0.28783 | load: 0.00015 [I 2022-10-27 02:03:00] ...trap/engines/engine.py.321: eval: epoch 41 | batch 210/510 [I 2022-10-27 02:03:00] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-27 02:03:00] ...trap/engines/engine.py.326: process: 0.28973 | load: 0.00014 [I 2022-10-27 02:03:03] ...trap/engines/engine.py.321: eval: epoch 41 | batch 220/510 [I 2022-10-27 02:03:03] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-27 02:03:03] ...trap/engines/engine.py.326: process: 0.28112 | load: 0.00015 [I 2022-10-27 02:03:06] ...trap/engines/engine.py.321: eval: epoch 41 | batch 230/510 [I 2022-10-27 02:03:06] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-27 02:03:06] ...trap/engines/engine.py.326: process: 0.29329 | load: 0.00015 [I 2022-10-27 02:03:09] ...trap/engines/engine.py.321: eval: epoch 41 | batch 240/510 [I 2022-10-27 02:03:09] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:19 [I 2022-10-27 02:03:09] ...trap/engines/engine.py.326: process: 0.30146 | load: 0.00015 [I 2022-10-27 02:03:12] ...trap/engines/engine.py.321: eval: epoch 41 | batch 250/510 [I 2022-10-27 02:03:12] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-27 02:03:12] ...trap/engines/engine.py.326: process: 0.28870 | load: 0.00015 [I 2022-10-27 02:03:14] ...trap/engines/engine.py.321: eval: epoch 41 | batch 260/510 [I 2022-10-27 02:03:14] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 02:03:14] ...trap/engines/engine.py.326: process: 0.28817 | load: 0.00016 [I 2022-10-27 02:03:17] ...trap/engines/engine.py.321: eval: epoch 41 | batch 270/510 [I 2022-10-27 02:03:17] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 02:03:17] ...trap/engines/engine.py.326: process: 0.29101 | load: 0.00016 [I 2022-10-27 02:03:20] ...trap/engines/engine.py.321: eval: epoch 41 | batch 280/510 [I 2022-10-27 02:03:20] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:05 [I 2022-10-27 02:03:20] ...trap/engines/engine.py.326: process: 0.27963 | load: 0.00015 [I 2022-10-27 02:03:23] ...trap/engines/engine.py.321: eval: epoch 41 | batch 290/510 [I 2022-10-27 02:03:23] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 02:03:23] ...trap/engines/engine.py.326: process: 0.29011 | load: 0.00015 [I 2022-10-27 02:03:26] ...trap/engines/engine.py.321: eval: epoch 41 | batch 300/510 [I 2022-10-27 02:03:26] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-27 02:03:26] ...trap/engines/engine.py.326: process: 0.29116 | load: 0.00019 [I 2022-10-27 02:03:29] ...trap/engines/engine.py.321: eval: epoch 41 | batch 310/510 [I 2022-10-27 02:03:29] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 02:03:29] ...trap/engines/engine.py.326: process: 0.28493 | load: 0.00017 [I 2022-10-27 02:03:32] ...trap/engines/engine.py.321: eval: epoch 41 | batch 320/510 [I 2022-10-27 02:03:32] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 02:03:32] ...trap/engines/engine.py.326: process: 0.28527 | load: 0.00015 [I 2022-10-27 02:03:35] ...trap/engines/engine.py.321: eval: epoch 41 | batch 330/510 [I 2022-10-27 02:03:35] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 02:03:35] ...trap/engines/engine.py.326: process: 0.28520 | load: 0.00015 [I 2022-10-27 02:03:37] ...trap/engines/engine.py.321: eval: epoch 41 | batch 340/510 [I 2022-10-27 02:03:37] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:49 [I 2022-10-27 02:03:37] ...trap/engines/engine.py.326: process: 0.29389 | load: 0.00016 [I 2022-10-27 02:03:40] ...trap/engines/engine.py.321: eval: epoch 41 | batch 350/510 [I 2022-10-27 02:03:40] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-27 02:03:40] ...trap/engines/engine.py.326: process: 0.29311 | load: 0.00022 [I 2022-10-27 02:03:43] ...trap/engines/engine.py.321: eval: epoch 41 | batch 360/510 [I 2022-10-27 02:03:43] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 02:03:43] ...trap/engines/engine.py.326: process: 0.28725 | load: 0.00019 [I 2022-10-27 02:03:46] ...trap/engines/engine.py.321: eval: epoch 41 | batch 370/510 [I 2022-10-27 02:03:46] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 02:03:46] ...trap/engines/engine.py.326: process: 0.28478 | load: 0.00015 [I 2022-10-27 02:03:49] ...trap/engines/engine.py.321: eval: epoch 41 | batch 380/510 [I 2022-10-27 02:03:49] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-27 02:03:49] ...trap/engines/engine.py.326: process: 0.29180 | load: 0.00020 [I 2022-10-27 02:03:52] ...trap/engines/engine.py.321: eval: epoch 41 | batch 390/510 [I 2022-10-27 02:03:52] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-27 02:03:52] ...trap/engines/engine.py.326: process: 0.28917 | load: 0.00014 [I 2022-10-27 02:03:55] ...trap/engines/engine.py.321: eval: epoch 41 | batch 400/510 [I 2022-10-27 02:03:55] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:32 [I 2022-10-27 02:03:55] ...trap/engines/engine.py.326: process: 0.28609 | load: 0.00015 [I 2022-10-27 02:03:58] ...trap/engines/engine.py.321: eval: epoch 41 | batch 410/510 [I 2022-10-27 02:03:58] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:29 [I 2022-10-27 02:03:58] ...trap/engines/engine.py.326: process: 0.28260 | load: 0.00020 [I 2022-10-27 02:04:01] ...trap/engines/engine.py.321: eval: epoch 41 | batch 420/510 [I 2022-10-27 02:04:01] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:26 [I 2022-10-27 02:04:01] ...trap/engines/engine.py.326: process: 0.29255 | load: 0.00019 [I 2022-10-27 02:04:04] ...trap/engines/engine.py.321: eval: epoch 41 | batch 430/510 [I 2022-10-27 02:04:04] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-27 02:04:04] ...trap/engines/engine.py.326: process: 0.28836 | load: 0.00015 [I 2022-10-27 02:04:06] ...trap/engines/engine.py.321: eval: epoch 41 | batch 440/510 [I 2022-10-27 02:04:06] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 02:04:06] ...trap/engines/engine.py.326: process: 0.28935 | load: 0.00016 [I 2022-10-27 02:04:09] ...trap/engines/engine.py.321: eval: epoch 41 | batch 450/510 [I 2022-10-27 02:04:09] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 02:04:09] ...trap/engines/engine.py.326: process: 0.28311 | load: 0.00016 [I 2022-10-27 02:04:12] ...trap/engines/engine.py.321: eval: epoch 41 | batch 460/510 [I 2022-10-27 02:04:12] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 02:04:12] ...trap/engines/engine.py.326: process: 0.29341 | load: 0.00019 [I 2022-10-27 02:04:15] ...trap/engines/engine.py.321: eval: epoch 41 | batch 470/510 [I 2022-10-27 02:04:15] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 02:04:15] ...trap/engines/engine.py.326: process: 0.28674 | load: 0.00127 [I 2022-10-27 02:04:18] ...trap/engines/engine.py.321: eval: epoch 41 | batch 480/510 [I 2022-10-27 02:04:18] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 02:04:18] ...trap/engines/engine.py.326: process: 0.28807 | load: 0.00016 [I 2022-10-27 02:04:21] ...trap/engines/engine.py.321: eval: epoch 41 | batch 490/510 [I 2022-10-27 02:04:21] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 02:04:21] ...trap/engines/engine.py.326: process: 0.28711 | load: 0.00016 [I 2022-10-27 02:04:24] ...trap/engines/engine.py.321: eval: epoch 41 | batch 500/510 [I 2022-10-27 02:04:24] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 02:04:24] ...trap/engines/engine.py.326: process: 0.29271 | load: 0.00015 [I 2022-10-27 02:04:27] ...trap/engines/engine.py.321: eval: epoch 41 | batch 510/510 [I 2022-10-27 02:04:27] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 02:04:27] ...trap/engines/engine.py.326: process: 0.28704 | load: 0.00011 [S 2022-10-27 02:04:27] ...trap/engines/engine.py.132: eval_epoch.epoch: 41 [I 2022-10-27 02:04:27] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.695400 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.900500 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936200 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.682400 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.894400 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932300 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.018266 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008801 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.003092 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.019443 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007761 [S 2022-10-27 02:04:31] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004649 [I 2022-10-27 02:04:33] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 02:04:33] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 02:04:33] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 02:04:33] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 02:04:33] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 02:04:34] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 02:04:37] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 02:04:43] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 02:04:43] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 02:04:43] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 02:04:46] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 02:04:51] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 02:04:51] ...trap/engines/engine.py.163: Training model on trainset for epoch 42 [I 2022-10-27 02:04:51] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 02:05:13] ...trap/engines/engine.py.224: train: epoch 42 | batch 0/2304 [I 2022-10-27 02:05:13] ...trap/engines/engine.py.228: elapsed: 0:00:21 | left: 13:37:35 [I 2022-10-27 02:05:13] ...trap/engines/engine.py.229: process: 21.29158 | load: 20.65217 [I 2022-10-27 02:05:13] ...trap/engines/engine.py.230: loss: 0.17099 [I 2022-10-27 02:05:13] ...optimizers/trijoint.py.151: total_norm: 26.371225 [I 2022-10-27 02:05:18] ...trap/engines/engine.py.224: train: epoch 42 | batch 10/2304 [I 2022-10-27 02:05:18] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 1:46:25 [I 2022-10-27 02:05:18] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00424 [I 2022-10-27 02:05:18] ...trap/engines/engine.py.230: loss: 0.13829 [I 2022-10-27 02:05:18] ...optimizers/trijoint.py.151: total_norm: 15.296834 [I 2022-10-27 02:05:24] ...trap/engines/engine.py.224: train: epoch 42 | batch 20/2304 [I 2022-10-27 02:05:24] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:30:11 [I 2022-10-27 02:05:24] ...trap/engines/engine.py.229: process: 0.55351 | load: 0.00364 [I 2022-10-27 02:05:24] ...trap/engines/engine.py.230: loss: 0.10780 [I 2022-10-27 02:05:24] ...optimizers/trijoint.py.151: total_norm: 10.749940 [I 2022-10-27 02:05:29] ...trap/engines/engine.py.224: train: epoch 42 | batch 30/2304 [I 2022-10-27 02:05:29] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:22:04 [I 2022-10-27 02:05:29] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00309 [I 2022-10-27 02:05:29] ...trap/engines/engine.py.230: loss: 0.26857 [I 2022-10-27 02:05:29] ...optimizers/trijoint.py.151: total_norm: 18.339153 [I 2022-10-27 02:05:35] ...trap/engines/engine.py.224: train: epoch 42 | batch 40/2304 [I 2022-10-27 02:05:35] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:21:40 [I 2022-10-27 02:05:35] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00269 [I 2022-10-27 02:05:35] ...trap/engines/engine.py.230: loss: 0.12201 [I 2022-10-27 02:05:35] ...optimizers/trijoint.py.151: total_norm: 12.609972 [I 2022-10-27 02:05:41] ...trap/engines/engine.py.224: train: epoch 42 | batch 50/2304 [I 2022-10-27 02:05:41] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:20:59 [I 2022-10-27 02:05:41] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00304 [I 2022-10-27 02:05:41] ...trap/engines/engine.py.230: loss: 0.18560 [I 2022-10-27 02:05:41] ...optimizers/trijoint.py.151: total_norm: 14.597312 [I 2022-10-27 02:05:46] ...trap/engines/engine.py.224: train: epoch 42 | batch 60/2304 [I 2022-10-27 02:05:46] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:21:02 [I 2022-10-27 02:05:46] ...trap/engines/engine.py.229: process: 0.56765 | load: 0.00287 [I 2022-10-27 02:05:46] ...trap/engines/engine.py.230: loss: 0.16205 [I 2022-10-27 02:05:46] ...optimizers/trijoint.py.151: total_norm: 17.950874 [I 2022-10-27 02:05:52] ...trap/engines/engine.py.224: train: epoch 42 | batch 70/2304 [I 2022-10-27 02:05:52] ...trap/engines/engine.py.228: elapsed: 0:01:00 | left: 0:20:49 [I 2022-10-27 02:05:52] ...trap/engines/engine.py.229: process: 0.56537 | load: 0.00256 [I 2022-10-27 02:05:52] ...trap/engines/engine.py.230: loss: 0.15036 [I 2022-10-27 02:05:52] ...optimizers/trijoint.py.151: total_norm: 7.429071 [I 2022-10-27 02:06:15] ...trap/engines/engine.py.224: train: epoch 42 | batch 80/2304 [I 2022-10-27 02:06:15] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 1:14:21 [I 2022-10-27 02:06:15] ...trap/engines/engine.py.229: process: 0.56398 | load: 0.00259 [I 2022-10-27 02:06:15] ...trap/engines/engine.py.230: loss: 0.20036 [I 2022-10-27 02:06:15] ...optimizers/trijoint.py.151: total_norm: 19.855690 [I 2022-10-27 02:06:21] ...trap/engines/engine.py.224: train: epoch 42 | batch 90/2304 [I 2022-10-27 02:06:21] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:26:18 [I 2022-10-27 02:06:21] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00271 [I 2022-10-27 02:06:21] ...trap/engines/engine.py.230: loss: 0.19355 [I 2022-10-27 02:06:21] ...optimizers/trijoint.py.151: total_norm: 17.870659 [I 2022-10-27 02:06:26] ...trap/engines/engine.py.224: train: epoch 42 | batch 100/2304 [I 2022-10-27 02:06:26] ...trap/engines/engine.py.228: elapsed: 0:01:34 | left: 0:21:55 [I 2022-10-27 02:06:26] ...trap/engines/engine.py.229: process: 0.68265 | load: 0.00260 [I 2022-10-27 02:06:26] ...trap/engines/engine.py.230: loss: 0.16343 [I 2022-10-27 02:06:26] ...optimizers/trijoint.py.151: total_norm: 44.667702 [I 2022-10-27 02:06:32] ...trap/engines/engine.py.224: train: epoch 42 | batch 110/2304 [I 2022-10-27 02:06:32] ...trap/engines/engine.py.228: elapsed: 0:01:40 | left: 0:20:28 [I 2022-10-27 02:06:32] ...trap/engines/engine.py.229: process: 0.55413 | load: 0.00267 [I 2022-10-27 02:06:32] ...trap/engines/engine.py.230: loss: 0.16516 [I 2022-10-27 02:06:32] ...optimizers/trijoint.py.151: total_norm: 11.264261 [I 2022-10-27 02:06:37] ...trap/engines/engine.py.224: train: epoch 42 | batch 120/2304 [I 2022-10-27 02:06:37] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:20:19 [I 2022-10-27 02:06:37] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00250 [I 2022-10-27 02:06:37] ...trap/engines/engine.py.230: loss: 0.20202 [I 2022-10-27 02:06:37] ...optimizers/trijoint.py.151: total_norm: 15.367414 [I 2022-10-27 02:06:43] ...trap/engines/engine.py.224: train: epoch 42 | batch 130/2304 [I 2022-10-27 02:06:43] ...trap/engines/engine.py.228: elapsed: 0:01:51 | left: 0:20:04 [I 2022-10-27 02:06:43] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00265 [I 2022-10-27 02:06:43] ...trap/engines/engine.py.230: loss: 0.18341 [I 2022-10-27 02:06:43] ...optimizers/trijoint.py.151: total_norm: 19.176342 [I 2022-10-27 02:06:49] ...trap/engines/engine.py.224: train: epoch 42 | batch 140/2304 [I 2022-10-27 02:06:49] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:19:59 [I 2022-10-27 02:06:49] ...trap/engines/engine.py.229: process: 0.54382 | load: 0.00249 [I 2022-10-27 02:06:49] ...trap/engines/engine.py.230: loss: 0.14143 [I 2022-10-27 02:06:49] ...optimizers/trijoint.py.151: total_norm: 11.140613 [I 2022-10-27 02:06:54] ...trap/engines/engine.py.224: train: epoch 42 | batch 150/2304 [I 2022-10-27 02:06:54] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:19:49 [I 2022-10-27 02:06:54] ...trap/engines/engine.py.229: process: 0.54162 | load: 0.00259 [I 2022-10-27 02:06:54] ...trap/engines/engine.py.230: loss: 0.17800 [I 2022-10-27 02:06:54] ...optimizers/trijoint.py.151: total_norm: 14.790881 [I 2022-10-27 02:07:00] ...trap/engines/engine.py.224: train: epoch 42 | batch 160/2304 [I 2022-10-27 02:07:00] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:19:54 [I 2022-10-27 02:07:00] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00270 [I 2022-10-27 02:07:00] ...trap/engines/engine.py.230: loss: 0.18543 [I 2022-10-27 02:07:00] ...optimizers/trijoint.py.151: total_norm: 17.394571 [I 2022-10-27 02:07:05] ...trap/engines/engine.py.224: train: epoch 42 | batch 170/2304 [I 2022-10-27 02:07:05] ...trap/engines/engine.py.228: elapsed: 0:02:14 | left: 0:19:58 [I 2022-10-27 02:07:05] ...trap/engines/engine.py.229: process: 0.53736 | load: 0.00242 [I 2022-10-27 02:07:05] ...trap/engines/engine.py.230: loss: 0.14874 [I 2022-10-27 02:07:05] ...optimizers/trijoint.py.151: total_norm: 8.588070 [I 2022-10-27 02:07:11] ...trap/engines/engine.py.224: train: epoch 42 | batch 180/2304 [I 2022-10-27 02:07:11] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:19:54 [I 2022-10-27 02:07:11] ...trap/engines/engine.py.229: process: 0.56254 | load: 0.00255 [I 2022-10-27 02:07:11] ...trap/engines/engine.py.230: loss: 0.24340 [I 2022-10-27 02:07:11] ...optimizers/trijoint.py.151: total_norm: 61.781002 [I 2022-10-27 02:07:17] ...trap/engines/engine.py.224: train: epoch 42 | batch 190/2304 [I 2022-10-27 02:07:17] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:19:43 [I 2022-10-27 02:07:17] ...trap/engines/engine.py.229: process: 0.56163 | load: 0.00252 [I 2022-10-27 02:07:17] ...trap/engines/engine.py.230: loss: 0.17017 [I 2022-10-27 02:07:17] ...optimizers/trijoint.py.151: total_norm: 21.329832 [I 2022-10-27 02:07:22] ...trap/engines/engine.py.224: train: epoch 42 | batch 200/2304 [I 2022-10-27 02:07:22] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:19:25 [I 2022-10-27 02:07:22] ...trap/engines/engine.py.229: process: 0.55143 | load: 0.00251 [I 2022-10-27 02:07:22] ...trap/engines/engine.py.230: loss: 0.16550 [I 2022-10-27 02:07:22] ...optimizers/trijoint.py.151: total_norm: 17.957653 [I 2022-10-27 02:07:28] ...trap/engines/engine.py.224: train: epoch 42 | batch 210/2304 [I 2022-10-27 02:07:28] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:19:17 [I 2022-10-27 02:07:28] ...trap/engines/engine.py.229: process: 0.54115 | load: 0.00251 [I 2022-10-27 02:07:28] ...trap/engines/engine.py.230: loss: 0.13839 [I 2022-10-27 02:07:28] ...optimizers/trijoint.py.151: total_norm: 14.978521 [I 2022-10-27 02:07:33] ...trap/engines/engine.py.224: train: epoch 42 | batch 220/2304 [I 2022-10-27 02:07:33] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:19:27 [I 2022-10-27 02:07:33] ...trap/engines/engine.py.229: process: 0.56808 | load: 0.00251 [I 2022-10-27 02:07:33] ...trap/engines/engine.py.230: loss: 0.19981 [I 2022-10-27 02:07:33] ...optimizers/trijoint.py.151: total_norm: 20.695988 [I 2022-10-27 02:07:39] ...trap/engines/engine.py.224: train: epoch 42 | batch 230/2304 [I 2022-10-27 02:07:39] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:19:32 [I 2022-10-27 02:07:39] ...trap/engines/engine.py.229: process: 0.56897 | load: 0.00256 [I 2022-10-27 02:07:39] ...trap/engines/engine.py.230: loss: 0.14702 [I 2022-10-27 02:07:39] ...optimizers/trijoint.py.151: total_norm: 16.419622 [I 2022-10-27 02:07:45] ...trap/engines/engine.py.224: train: epoch 42 | batch 240/2304 [I 2022-10-27 02:07:45] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:19:17 [I 2022-10-27 02:07:45] ...trap/engines/engine.py.229: process: 0.55410 | load: 0.00260 [I 2022-10-27 02:07:45] ...trap/engines/engine.py.230: loss: 0.12661 [I 2022-10-27 02:07:45] ...optimizers/trijoint.py.151: total_norm: 14.497417 [I 2022-10-27 02:07:50] ...trap/engines/engine.py.224: train: epoch 42 | batch 250/2304 [I 2022-10-27 02:07:50] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:19:12 [I 2022-10-27 02:07:50] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00253 [I 2022-10-27 02:07:50] ...trap/engines/engine.py.230: loss: 0.17199 [I 2022-10-27 02:07:50] ...optimizers/trijoint.py.151: total_norm: 35.452652 [I 2022-10-27 02:07:56] ...trap/engines/engine.py.224: train: epoch 42 | batch 260/2304 [I 2022-10-27 02:07:56] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:19:59 [I 2022-10-27 02:07:56] ...trap/engines/engine.py.229: process: 0.68593 | load: 0.00262 [I 2022-10-27 02:07:56] ...trap/engines/engine.py.230: loss: 0.10921 [I 2022-10-27 02:07:56] ...optimizers/trijoint.py.151: total_norm: 8.869187 [I 2022-10-27 02:08:02] ...trap/engines/engine.py.224: train: epoch 42 | batch 270/2304 [I 2022-10-27 02:08:02] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:19:15 [I 2022-10-27 02:08:02] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00253 [I 2022-10-27 02:08:02] ...trap/engines/engine.py.230: loss: 0.17301 [I 2022-10-27 02:08:02] ...optimizers/trijoint.py.151: total_norm: 17.517368 [I 2022-10-27 02:08:07] ...trap/engines/engine.py.224: train: epoch 42 | batch 280/2304 [I 2022-10-27 02:08:07] ...trap/engines/engine.py.228: elapsed: 0:03:15 | left: 0:18:50 [I 2022-10-27 02:08:07] ...trap/engines/engine.py.229: process: 0.54823 | load: 0.00284 [I 2022-10-27 02:08:07] ...trap/engines/engine.py.230: loss: 0.17717 [I 2022-10-27 02:08:07] ...optimizers/trijoint.py.151: total_norm: 23.163267 [I 2022-10-27 02:08:13] ...trap/engines/engine.py.224: train: epoch 42 | batch 290/2304 [I 2022-10-27 02:08:13] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:18:46 [I 2022-10-27 02:08:13] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00266 [I 2022-10-27 02:08:13] ...trap/engines/engine.py.230: loss: 0.19009 [I 2022-10-27 02:08:13] ...optimizers/trijoint.py.151: total_norm: 44.935333 [I 2022-10-27 02:08:18] ...trap/engines/engine.py.224: train: epoch 42 | batch 300/2304 [I 2022-10-27 02:08:18] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:18:28 [I 2022-10-27 02:08:18] ...trap/engines/engine.py.229: process: 0.54073 | load: 0.00254 [I 2022-10-27 02:08:18] ...trap/engines/engine.py.230: loss: 0.16481 [I 2022-10-27 02:08:18] ...optimizers/trijoint.py.151: total_norm: 11.819236 [I 2022-10-27 02:08:24] ...trap/engines/engine.py.224: train: epoch 42 | batch 310/2304 [I 2022-10-27 02:08:24] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:18:31 [I 2022-10-27 02:08:24] ...trap/engines/engine.py.229: process: 0.56420 | load: 0.00260 [I 2022-10-27 02:08:24] ...trap/engines/engine.py.230: loss: 0.10698 [I 2022-10-27 02:08:24] ...optimizers/trijoint.py.151: total_norm: 13.879682 [I 2022-10-27 02:08:30] ...trap/engines/engine.py.224: train: epoch 42 | batch 320/2304 [I 2022-10-27 02:08:30] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:18:29 [I 2022-10-27 02:08:30] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00247 [I 2022-10-27 02:08:30] ...trap/engines/engine.py.230: loss: 0.17970 [I 2022-10-27 02:08:30] ...optimizers/trijoint.py.151: total_norm: 17.785828 [I 2022-10-27 02:08:35] ...trap/engines/engine.py.224: train: epoch 42 | batch 330/2304 [I 2022-10-27 02:08:35] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:18:22 [I 2022-10-27 02:08:35] ...trap/engines/engine.py.229: process: 0.56403 | load: 0.00247 [I 2022-10-27 02:08:35] ...trap/engines/engine.py.230: loss: 0.17053 [I 2022-10-27 02:08:35] ...optimizers/trijoint.py.151: total_norm: 18.076132 [I 2022-10-27 02:08:41] ...trap/engines/engine.py.224: train: epoch 42 | batch 340/2304 [I 2022-10-27 02:08:41] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:18:18 [I 2022-10-27 02:08:41] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00284 [I 2022-10-27 02:08:41] ...trap/engines/engine.py.230: loss: 0.12365 [I 2022-10-27 02:08:41] ...optimizers/trijoint.py.151: total_norm: 22.276159 [I 2022-10-27 02:08:46] ...trap/engines/engine.py.224: train: epoch 42 | batch 350/2304 [I 2022-10-27 02:08:46] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:18:15 [I 2022-10-27 02:08:46] ...trap/engines/engine.py.229: process: 0.56160 | load: 0.00246 [I 2022-10-27 02:08:46] ...trap/engines/engine.py.230: loss: 0.16811 [I 2022-10-27 02:08:46] ...optimizers/trijoint.py.151: total_norm: 17.420906 [I 2022-10-27 02:08:52] ...trap/engines/engine.py.224: train: epoch 42 | batch 360/2304 [I 2022-10-27 02:08:52] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:18:08 [I 2022-10-27 02:08:52] ...trap/engines/engine.py.229: process: 0.56898 | load: 0.00265 [I 2022-10-27 02:08:52] ...trap/engines/engine.py.230: loss: 0.24304 [I 2022-10-27 02:08:52] ...optimizers/trijoint.py.151: total_norm: 18.553228 [I 2022-10-27 02:08:58] ...trap/engines/engine.py.224: train: epoch 42 | batch 370/2304 [I 2022-10-27 02:08:58] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:18:09 [I 2022-10-27 02:08:58] ...trap/engines/engine.py.229: process: 0.54922 | load: 0.00256 [I 2022-10-27 02:08:58] ...trap/engines/engine.py.230: loss: 0.17743 [I 2022-10-27 02:08:58] ...optimizers/trijoint.py.151: total_norm: 12.672919 [I 2022-10-27 02:09:03] ...trap/engines/engine.py.224: train: epoch 42 | batch 380/2304 [I 2022-10-27 02:09:03] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:17:55 [I 2022-10-27 02:09:03] ...trap/engines/engine.py.229: process: 0.55338 | load: 0.00261 [I 2022-10-27 02:09:03] ...trap/engines/engine.py.230: loss: 0.23261 [I 2022-10-27 02:09:03] ...optimizers/trijoint.py.151: total_norm: 14.624433 [I 2022-10-27 02:09:09] ...trap/engines/engine.py.224: train: epoch 42 | batch 390/2304 [I 2022-10-27 02:09:09] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:17:57 [I 2022-10-27 02:09:09] ...trap/engines/engine.py.229: process: 0.57026 | load: 0.00254 [I 2022-10-27 02:09:09] ...trap/engines/engine.py.230: loss: 0.18575 [I 2022-10-27 02:09:09] ...optimizers/trijoint.py.151: total_norm: 18.648285 [I 2022-10-27 02:09:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 400/2304 [I 2022-10-27 02:09:14] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:17:47 [I 2022-10-27 02:09:14] ...trap/engines/engine.py.229: process: 0.57120 | load: 0.00254 [I 2022-10-27 02:09:14] ...trap/engines/engine.py.230: loss: 0.21142 [I 2022-10-27 02:09:14] ...optimizers/trijoint.py.151: total_norm: 48.710793 [I 2022-10-27 02:09:20] ...trap/engines/engine.py.224: train: epoch 42 | batch 410/2304 [I 2022-10-27 02:09:20] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:17:30 [I 2022-10-27 02:09:20] ...trap/engines/engine.py.229: process: 0.54297 | load: 0.00247 [I 2022-10-27 02:09:20] ...trap/engines/engine.py.230: loss: 0.16009 [I 2022-10-27 02:09:20] ...optimizers/trijoint.py.151: total_norm: 20.848673 [I 2022-10-27 02:09:26] ...trap/engines/engine.py.224: train: epoch 42 | batch 420/2304 [I 2022-10-27 02:09:26] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:17:34 [I 2022-10-27 02:09:26] ...trap/engines/engine.py.229: process: 0.55075 | load: 0.00285 [I 2022-10-27 02:09:26] ...trap/engines/engine.py.230: loss: 0.23122 [I 2022-10-27 02:09:26] ...optimizers/trijoint.py.151: total_norm: 16.754944 [I 2022-10-27 02:09:31] ...trap/engines/engine.py.224: train: epoch 42 | batch 430/2304 [I 2022-10-27 02:09:31] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:17:38 [I 2022-10-27 02:09:31] ...trap/engines/engine.py.229: process: 0.57726 | load: 0.00252 [I 2022-10-27 02:09:31] ...trap/engines/engine.py.230: loss: 0.18014 [I 2022-10-27 02:09:31] ...optimizers/trijoint.py.151: total_norm: 13.729536 [I 2022-10-27 02:09:37] ...trap/engines/engine.py.224: train: epoch 42 | batch 440/2304 [I 2022-10-27 02:09:37] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:17:20 [I 2022-10-27 02:09:37] ...trap/engines/engine.py.229: process: 0.54059 | load: 0.00268 [I 2022-10-27 02:09:37] ...trap/engines/engine.py.230: loss: 0.09782 [I 2022-10-27 02:09:37] ...optimizers/trijoint.py.151: total_norm: 7.719182 [I 2022-10-27 02:09:43] ...trap/engines/engine.py.224: train: epoch 42 | batch 450/2304 [I 2022-10-27 02:09:43] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:17:14 [I 2022-10-27 02:09:43] ...trap/engines/engine.py.229: process: 0.54540 | load: 0.00256 [I 2022-10-27 02:09:43] ...trap/engines/engine.py.230: loss: 0.21208 [I 2022-10-27 02:09:43] ...optimizers/trijoint.py.151: total_norm: 14.919442 [I 2022-10-27 02:09:48] ...trap/engines/engine.py.224: train: epoch 42 | batch 460/2304 [I 2022-10-27 02:09:48] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:17:02 [I 2022-10-27 02:09:48] ...trap/engines/engine.py.229: process: 0.55258 | load: 0.00253 [I 2022-10-27 02:09:48] ...trap/engines/engine.py.230: loss: 0.15585 [I 2022-10-27 02:09:48] ...optimizers/trijoint.py.151: total_norm: 26.674971 [I 2022-10-27 02:09:54] ...trap/engines/engine.py.224: train: epoch 42 | batch 470/2304 [I 2022-10-27 02:09:54] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:16:57 [I 2022-10-27 02:09:54] ...trap/engines/engine.py.229: process: 0.56533 | load: 0.00252 [I 2022-10-27 02:09:54] ...trap/engines/engine.py.230: loss: 0.16054 [I 2022-10-27 02:09:54] ...optimizers/trijoint.py.151: total_norm: 31.580095 [I 2022-10-27 02:09:59] ...trap/engines/engine.py.224: train: epoch 42 | batch 480/2304 [I 2022-10-27 02:09:59] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:16:46 [I 2022-10-27 02:09:59] ...trap/engines/engine.py.229: process: 0.55059 | load: 0.00546 [I 2022-10-27 02:09:59] ...trap/engines/engine.py.230: loss: 0.17210 [I 2022-10-27 02:09:59] ...optimizers/trijoint.py.151: total_norm: 17.086531 [I 2022-10-27 02:10:05] ...trap/engines/engine.py.224: train: epoch 42 | batch 490/2304 [I 2022-10-27 02:10:05] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:17:00 [I 2022-10-27 02:10:05] ...trap/engines/engine.py.229: process: 0.56428 | load: 0.00248 [I 2022-10-27 02:10:05] ...trap/engines/engine.py.230: loss: 0.11119 [I 2022-10-27 02:10:05] ...optimizers/trijoint.py.151: total_norm: 11.745748 [I 2022-10-27 02:10:10] ...trap/engines/engine.py.224: train: epoch 42 | batch 500/2304 [I 2022-10-27 02:10:10] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:16:44 [I 2022-10-27 02:10:10] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00252 [I 2022-10-27 02:10:10] ...trap/engines/engine.py.230: loss: 0.15437 [I 2022-10-27 02:10:10] ...optimizers/trijoint.py.151: total_norm: 19.772532 [I 2022-10-27 02:10:16] ...trap/engines/engine.py.224: train: epoch 42 | batch 510/2304 [I 2022-10-27 02:10:16] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:16:41 [I 2022-10-27 02:10:16] ...trap/engines/engine.py.229: process: 0.56159 | load: 0.00247 [I 2022-10-27 02:10:16] ...trap/engines/engine.py.230: loss: 0.13338 [I 2022-10-27 02:10:16] ...optimizers/trijoint.py.151: total_norm: 30.069105 [I 2022-10-27 02:10:22] ...trap/engines/engine.py.224: train: epoch 42 | batch 520/2304 [I 2022-10-27 02:10:22] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:16:30 [I 2022-10-27 02:10:22] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00252 [I 2022-10-27 02:10:22] ...trap/engines/engine.py.230: loss: 0.13316 [I 2022-10-27 02:10:22] ...optimizers/trijoint.py.151: total_norm: 11.603992 [I 2022-10-27 02:10:27] ...trap/engines/engine.py.224: train: epoch 42 | batch 530/2304 [I 2022-10-27 02:10:27] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:16:26 [I 2022-10-27 02:10:27] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00242 [I 2022-10-27 02:10:27] ...trap/engines/engine.py.230: loss: 0.24494 [I 2022-10-27 02:10:27] ...optimizers/trijoint.py.151: total_norm: 47.716324 [I 2022-10-27 02:10:33] ...trap/engines/engine.py.224: train: epoch 42 | batch 540/2304 [I 2022-10-27 02:10:33] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:16:26 [I 2022-10-27 02:10:33] ...trap/engines/engine.py.229: process: 0.56519 | load: 0.00258 [I 2022-10-27 02:10:33] ...trap/engines/engine.py.230: loss: 0.19548 [I 2022-10-27 02:10:33] ...optimizers/trijoint.py.151: total_norm: 8.311226 [I 2022-10-27 02:10:38] ...trap/engines/engine.py.224: train: epoch 42 | batch 550/2304 [I 2022-10-27 02:10:38] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:16:24 [I 2022-10-27 02:10:38] ...trap/engines/engine.py.229: process: 0.58297 | load: 0.00251 [I 2022-10-27 02:10:38] ...trap/engines/engine.py.230: loss: 0.21709 [I 2022-10-27 02:10:38] ...optimizers/trijoint.py.151: total_norm: 18.203430 [I 2022-10-27 02:10:44] ...trap/engines/engine.py.224: train: epoch 42 | batch 560/2304 [I 2022-10-27 02:10:44] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:16:18 [I 2022-10-27 02:10:44] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00247 [I 2022-10-27 02:10:44] ...trap/engines/engine.py.230: loss: 0.11694 [I 2022-10-27 02:10:44] ...optimizers/trijoint.py.151: total_norm: 7.758325 [I 2022-10-27 02:10:50] ...trap/engines/engine.py.224: train: epoch 42 | batch 570/2304 [I 2022-10-27 02:10:50] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:16:02 [I 2022-10-27 02:10:50] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00247 [I 2022-10-27 02:10:50] ...trap/engines/engine.py.230: loss: 0.22460 [I 2022-10-27 02:10:50] ...optimizers/trijoint.py.151: total_norm: 9.636867 [I 2022-10-27 02:10:55] ...trap/engines/engine.py.224: train: epoch 42 | batch 580/2304 [I 2022-10-27 02:10:55] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:16:05 [I 2022-10-27 02:10:55] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00252 [I 2022-10-27 02:10:55] ...trap/engines/engine.py.230: loss: 0.19829 [I 2022-10-27 02:10:55] ...optimizers/trijoint.py.151: total_norm: 77.704697 [I 2022-10-27 02:11:01] ...trap/engines/engine.py.224: train: epoch 42 | batch 590/2304 [I 2022-10-27 02:11:01] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:16:01 [I 2022-10-27 02:11:01] ...trap/engines/engine.py.229: process: 0.56691 | load: 0.00257 [I 2022-10-27 02:11:01] ...trap/engines/engine.py.230: loss: 0.22610 [I 2022-10-27 02:11:01] ...optimizers/trijoint.py.151: total_norm: 24.886024 [I 2022-10-27 02:11:06] ...trap/engines/engine.py.224: train: epoch 42 | batch 600/2304 [I 2022-10-27 02:11:06] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:56 [I 2022-10-27 02:11:06] ...trap/engines/engine.py.229: process: 0.57601 | load: 0.00257 [I 2022-10-27 02:11:06] ...trap/engines/engine.py.230: loss: 0.18263 [I 2022-10-27 02:11:06] ...optimizers/trijoint.py.151: total_norm: 23.588509 [I 2022-10-27 02:11:12] ...trap/engines/engine.py.224: train: epoch 42 | batch 610/2304 [I 2022-10-27 02:11:12] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:55 [I 2022-10-27 02:11:12] ...trap/engines/engine.py.229: process: 0.56508 | load: 0.00268 [I 2022-10-27 02:11:12] ...trap/engines/engine.py.230: loss: 0.16552 [I 2022-10-27 02:11:12] ...optimizers/trijoint.py.151: total_norm: 12.853106 [I 2022-10-27 02:11:18] ...trap/engines/engine.py.224: train: epoch 42 | batch 620/2304 [I 2022-10-27 02:11:18] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:51 [I 2022-10-27 02:11:18] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00251 [I 2022-10-27 02:11:18] ...trap/engines/engine.py.230: loss: 0.14119 [I 2022-10-27 02:11:18] ...optimizers/trijoint.py.151: total_norm: 15.326571 [I 2022-10-27 02:11:23] ...trap/engines/engine.py.224: train: epoch 42 | batch 630/2304 [I 2022-10-27 02:11:23] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:15:35 [I 2022-10-27 02:11:23] ...trap/engines/engine.py.229: process: 0.55562 | load: 0.00254 [I 2022-10-27 02:11:23] ...trap/engines/engine.py.230: loss: 0.18189 [I 2022-10-27 02:11:23] ...optimizers/trijoint.py.151: total_norm: 15.798462 [I 2022-10-27 02:11:29] ...trap/engines/engine.py.224: train: epoch 42 | batch 640/2304 [I 2022-10-27 02:11:29] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:25 [I 2022-10-27 02:11:29] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00294 [I 2022-10-27 02:11:29] ...trap/engines/engine.py.230: loss: 0.17567 [I 2022-10-27 02:11:29] ...optimizers/trijoint.py.151: total_norm: 20.053251 [I 2022-10-27 02:11:35] ...trap/engines/engine.py.224: train: epoch 42 | batch 650/2304 [I 2022-10-27 02:11:35] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:30 [I 2022-10-27 02:11:35] ...trap/engines/engine.py.229: process: 0.56587 | load: 0.00252 [I 2022-10-27 02:11:35] ...trap/engines/engine.py.230: loss: 0.13071 [I 2022-10-27 02:11:35] ...optimizers/trijoint.py.151: total_norm: 19.875599 [I 2022-10-27 02:11:40] ...trap/engines/engine.py.224: train: epoch 42 | batch 660/2304 [I 2022-10-27 02:11:40] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:15:25 [I 2022-10-27 02:11:40] ...trap/engines/engine.py.229: process: 0.56446 | load: 0.00252 [I 2022-10-27 02:11:40] ...trap/engines/engine.py.230: loss: 0.16062 [I 2022-10-27 02:11:40] ...optimizers/trijoint.py.151: total_norm: 22.489290 [I 2022-10-27 02:11:46] ...trap/engines/engine.py.224: train: epoch 42 | batch 670/2304 [I 2022-10-27 02:11:46] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:15:12 [I 2022-10-27 02:11:46] ...trap/engines/engine.py.229: process: 0.54371 | load: 0.00267 [I 2022-10-27 02:11:46] ...trap/engines/engine.py.230: loss: 0.23314 [I 2022-10-27 02:11:46] ...optimizers/trijoint.py.151: total_norm: 16.100136 [I 2022-10-27 02:11:51] ...trap/engines/engine.py.224: train: epoch 42 | batch 680/2304 [I 2022-10-27 02:11:51] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:15:10 [I 2022-10-27 02:11:51] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00243 [I 2022-10-27 02:11:51] ...trap/engines/engine.py.230: loss: 0.21110 [I 2022-10-27 02:11:51] ...optimizers/trijoint.py.151: total_norm: 66.279213 [I 2022-10-27 02:11:57] ...trap/engines/engine.py.224: train: epoch 42 | batch 690/2304 [I 2022-10-27 02:11:57] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:15:02 [I 2022-10-27 02:11:57] ...trap/engines/engine.py.229: process: 0.56165 | load: 0.00246 [I 2022-10-27 02:11:57] ...trap/engines/engine.py.230: loss: 0.15323 [I 2022-10-27 02:11:57] ...optimizers/trijoint.py.151: total_norm: 13.629110 [I 2022-10-27 02:12:03] ...trap/engines/engine.py.224: train: epoch 42 | batch 700/2304 [I 2022-10-27 02:12:03] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:52 [I 2022-10-27 02:12:03] ...trap/engines/engine.py.229: process: 0.54934 | load: 0.00247 [I 2022-10-27 02:12:03] ...trap/engines/engine.py.230: loss: 0.29472 [I 2022-10-27 02:12:03] ...optimizers/trijoint.py.151: total_norm: 28.589582 [I 2022-10-27 02:12:08] ...trap/engines/engine.py.224: train: epoch 42 | batch 710/2304 [I 2022-10-27 02:12:08] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:51 [I 2022-10-27 02:12:08] ...trap/engines/engine.py.229: process: 0.56407 | load: 0.00250 [I 2022-10-27 02:12:08] ...trap/engines/engine.py.230: loss: 0.20017 [I 2022-10-27 02:12:08] ...optimizers/trijoint.py.151: total_norm: 15.911231 [I 2022-10-27 02:12:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 720/2304 [I 2022-10-27 02:12:14] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:48 [I 2022-10-27 02:12:14] ...trap/engines/engine.py.229: process: 0.56846 | load: 0.00244 [I 2022-10-27 02:12:14] ...trap/engines/engine.py.230: loss: 0.14361 [I 2022-10-27 02:12:14] ...optimizers/trijoint.py.151: total_norm: 12.328101 [I 2022-10-27 02:12:19] ...trap/engines/engine.py.224: train: epoch 42 | batch 730/2304 [I 2022-10-27 02:12:19] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:45 [I 2022-10-27 02:12:19] ...trap/engines/engine.py.229: process: 0.56773 | load: 0.00244 [I 2022-10-27 02:12:19] ...trap/engines/engine.py.230: loss: 0.12549 [I 2022-10-27 02:12:19] ...optimizers/trijoint.py.151: total_norm: 13.644199 [I 2022-10-27 02:12:25] ...trap/engines/engine.py.224: train: epoch 42 | batch 740/2304 [I 2022-10-27 02:12:25] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:14:42 [I 2022-10-27 02:12:25] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00244 [I 2022-10-27 02:12:25] ...trap/engines/engine.py.230: loss: 0.19994 [I 2022-10-27 02:12:25] ...optimizers/trijoint.py.151: total_norm: 17.480312 [I 2022-10-27 02:12:31] ...trap/engines/engine.py.224: train: epoch 42 | batch 750/2304 [I 2022-10-27 02:12:31] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:29 [I 2022-10-27 02:12:31] ...trap/engines/engine.py.229: process: 0.55804 | load: 0.00245 [I 2022-10-27 02:12:31] ...trap/engines/engine.py.230: loss: 0.19619 [I 2022-10-27 02:12:31] ...optimizers/trijoint.py.151: total_norm: 19.004871 [I 2022-10-27 02:12:36] ...trap/engines/engine.py.224: train: epoch 42 | batch 760/2304 [I 2022-10-27 02:12:36] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:14:17 [I 2022-10-27 02:12:36] ...trap/engines/engine.py.229: process: 0.54635 | load: 0.00249 [I 2022-10-27 02:12:36] ...trap/engines/engine.py.230: loss: 0.19253 [I 2022-10-27 02:12:36] ...optimizers/trijoint.py.151: total_norm: 17.590488 [I 2022-10-27 02:12:42] ...trap/engines/engine.py.224: train: epoch 42 | batch 770/2304 [I 2022-10-27 02:12:42] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:14:15 [I 2022-10-27 02:12:42] ...trap/engines/engine.py.229: process: 0.56119 | load: 0.00247 [I 2022-10-27 02:12:42] ...trap/engines/engine.py.230: loss: 0.17429 [I 2022-10-27 02:12:42] ...optimizers/trijoint.py.151: total_norm: 28.545385 [I 2022-10-27 02:12:47] ...trap/engines/engine.py.224: train: epoch 42 | batch 780/2304 [I 2022-10-27 02:12:47] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:14:18 [I 2022-10-27 02:12:47] ...trap/engines/engine.py.229: process: 0.56390 | load: 0.00248 [I 2022-10-27 02:12:47] ...trap/engines/engine.py.230: loss: 0.18984 [I 2022-10-27 02:12:47] ...optimizers/trijoint.py.151: total_norm: 25.753105 [I 2022-10-27 02:12:53] ...trap/engines/engine.py.224: train: epoch 42 | batch 790/2304 [I 2022-10-27 02:12:53] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:14:09 [I 2022-10-27 02:12:53] ...trap/engines/engine.py.229: process: 0.57116 | load: 0.00245 [I 2022-10-27 02:12:53] ...trap/engines/engine.py.230: loss: 0.24528 [I 2022-10-27 02:12:53] ...optimizers/trijoint.py.151: total_norm: 11.560809 [I 2022-10-27 02:12:59] ...trap/engines/engine.py.224: train: epoch 42 | batch 800/2304 [I 2022-10-27 02:12:59] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:14:03 [I 2022-10-27 02:12:59] ...trap/engines/engine.py.229: process: 0.56218 | load: 0.00250 [I 2022-10-27 02:12:59] ...trap/engines/engine.py.230: loss: 0.14998 [I 2022-10-27 02:12:59] ...optimizers/trijoint.py.151: total_norm: 17.749170 [I 2022-10-27 02:13:04] ...trap/engines/engine.py.224: train: epoch 42 | batch 810/2304 [I 2022-10-27 02:13:04] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:13:53 [I 2022-10-27 02:13:04] ...trap/engines/engine.py.229: process: 0.54472 | load: 0.00245 [I 2022-10-27 02:13:04] ...trap/engines/engine.py.230: loss: 0.28150 [I 2022-10-27 02:13:04] ...optimizers/trijoint.py.151: total_norm: 28.713831 [I 2022-10-27 02:13:10] ...trap/engines/engine.py.224: train: epoch 42 | batch 820/2304 [I 2022-10-27 02:13:10] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:54 [I 2022-10-27 02:13:10] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00260 [I 2022-10-27 02:13:10] ...trap/engines/engine.py.230: loss: 0.14442 [I 2022-10-27 02:13:10] ...optimizers/trijoint.py.151: total_norm: 11.242889 [I 2022-10-27 02:13:15] ...trap/engines/engine.py.224: train: epoch 42 | batch 830/2304 [I 2022-10-27 02:13:15] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:38 [I 2022-10-27 02:13:15] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00246 [I 2022-10-27 02:13:15] ...trap/engines/engine.py.230: loss: 0.14322 [I 2022-10-27 02:13:15] ...optimizers/trijoint.py.151: total_norm: 21.005743 [I 2022-10-27 02:13:21] ...trap/engines/engine.py.224: train: epoch 42 | batch 840/2304 [I 2022-10-27 02:13:21] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:13:35 [I 2022-10-27 02:13:21] ...trap/engines/engine.py.229: process: 0.55937 | load: 0.00248 [I 2022-10-27 02:13:21] ...trap/engines/engine.py.230: loss: 0.17035 [I 2022-10-27 02:13:21] ...optimizers/trijoint.py.151: total_norm: 28.096003 [I 2022-10-27 02:13:27] ...trap/engines/engine.py.224: train: epoch 42 | batch 850/2304 [I 2022-10-27 02:13:27] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:32 [I 2022-10-27 02:13:27] ...trap/engines/engine.py.229: process: 0.56619 | load: 0.00244 [I 2022-10-27 02:13:27] ...trap/engines/engine.py.230: loss: 0.15192 [I 2022-10-27 02:13:27] ...optimizers/trijoint.py.151: total_norm: 13.101361 [I 2022-10-27 02:13:32] ...trap/engines/engine.py.224: train: epoch 42 | batch 860/2304 [I 2022-10-27 02:13:32] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:13:30 [I 2022-10-27 02:13:32] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00242 [I 2022-10-27 02:13:32] ...trap/engines/engine.py.230: loss: 0.14184 [I 2022-10-27 02:13:32] ...optimizers/trijoint.py.151: total_norm: 18.543306 [I 2022-10-27 02:13:38] ...trap/engines/engine.py.224: train: epoch 42 | batch 870/2304 [I 2022-10-27 02:13:38] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:13:13 [I 2022-10-27 02:13:38] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00242 [I 2022-10-27 02:13:38] ...trap/engines/engine.py.230: loss: 0.19342 [I 2022-10-27 02:13:38] ...optimizers/trijoint.py.151: total_norm: 13.508548 [I 2022-10-27 02:13:43] ...trap/engines/engine.py.224: train: epoch 42 | batch 880/2304 [I 2022-10-27 02:13:43] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:13:20 [I 2022-10-27 02:13:43] ...trap/engines/engine.py.229: process: 0.56568 | load: 0.00256 [I 2022-10-27 02:13:43] ...trap/engines/engine.py.230: loss: 0.18451 [I 2022-10-27 02:13:43] ...optimizers/trijoint.py.151: total_norm: 16.243563 [I 2022-10-27 02:13:49] ...trap/engines/engine.py.224: train: epoch 42 | batch 890/2304 [I 2022-10-27 02:13:49] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:13:06 [I 2022-10-27 02:13:49] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00246 [I 2022-10-27 02:13:49] ...trap/engines/engine.py.230: loss: 0.14692 [I 2022-10-27 02:13:49] ...optimizers/trijoint.py.151: total_norm: 12.939246 [I 2022-10-27 02:13:55] ...trap/engines/engine.py.224: train: epoch 42 | batch 900/2304 [I 2022-10-27 02:13:55] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:13:08 [I 2022-10-27 02:13:55] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00242 [I 2022-10-27 02:13:55] ...trap/engines/engine.py.230: loss: 0.24767 [I 2022-10-27 02:13:55] ...optimizers/trijoint.py.151: total_norm: 20.402460 [I 2022-10-27 02:14:00] ...trap/engines/engine.py.224: train: epoch 42 | batch 910/2304 [I 2022-10-27 02:14:00] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:13:02 [I 2022-10-27 02:14:00] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00252 [I 2022-10-27 02:14:00] ...trap/engines/engine.py.230: loss: 0.15648 [I 2022-10-27 02:14:00] ...optimizers/trijoint.py.151: total_norm: 15.291940 [I 2022-10-27 02:14:06] ...trap/engines/engine.py.224: train: epoch 42 | batch 920/2304 [I 2022-10-27 02:14:06] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:57 [I 2022-10-27 02:14:06] ...trap/engines/engine.py.229: process: 0.55828 | load: 0.00289 [I 2022-10-27 02:14:06] ...trap/engines/engine.py.230: loss: 0.13717 [I 2022-10-27 02:14:06] ...optimizers/trijoint.py.151: total_norm: 11.251869 [I 2022-10-27 02:14:12] ...trap/engines/engine.py.224: train: epoch 42 | batch 930/2304 [I 2022-10-27 02:14:12] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:54 [I 2022-10-27 02:14:12] ...trap/engines/engine.py.229: process: 0.56254 | load: 0.00255 [I 2022-10-27 02:14:12] ...trap/engines/engine.py.230: loss: 0.19156 [I 2022-10-27 02:14:12] ...optimizers/trijoint.py.151: total_norm: 25.534967 [I 2022-10-27 02:14:17] ...trap/engines/engine.py.224: train: epoch 42 | batch 940/2304 [I 2022-10-27 02:14:17] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:12:49 [I 2022-10-27 02:14:17] ...trap/engines/engine.py.229: process: 0.56455 | load: 0.00245 [I 2022-10-27 02:14:17] ...trap/engines/engine.py.230: loss: 0.18850 [I 2022-10-27 02:14:17] ...optimizers/trijoint.py.151: total_norm: 16.929775 [I 2022-10-27 02:14:23] ...trap/engines/engine.py.224: train: epoch 42 | batch 950/2304 [I 2022-10-27 02:14:23] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:41 [I 2022-10-27 02:14:23] ...trap/engines/engine.py.229: process: 0.55803 | load: 0.00252 [I 2022-10-27 02:14:23] ...trap/engines/engine.py.230: loss: 0.11295 [I 2022-10-27 02:14:23] ...optimizers/trijoint.py.151: total_norm: 8.454814 [I 2022-10-27 02:14:28] ...trap/engines/engine.py.224: train: epoch 42 | batch 960/2304 [I 2022-10-27 02:14:28] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:12:33 [I 2022-10-27 02:14:28] ...trap/engines/engine.py.229: process: 0.56550 | load: 0.00261 [I 2022-10-27 02:14:28] ...trap/engines/engine.py.230: loss: 0.15954 [I 2022-10-27 02:14:28] ...optimizers/trijoint.py.151: total_norm: 28.719883 [I 2022-10-27 02:14:34] ...trap/engines/engine.py.224: train: epoch 42 | batch 970/2304 [I 2022-10-27 02:14:34] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:28 [I 2022-10-27 02:14:34] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00252 [I 2022-10-27 02:14:34] ...trap/engines/engine.py.230: loss: 0.13383 [I 2022-10-27 02:14:34] ...optimizers/trijoint.py.151: total_norm: 12.721210 [I 2022-10-27 02:14:40] ...trap/engines/engine.py.224: train: epoch 42 | batch 980/2304 [I 2022-10-27 02:14:40] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:12:23 [I 2022-10-27 02:14:40] ...trap/engines/engine.py.229: process: 0.57968 | load: 0.00257 [I 2022-10-27 02:14:40] ...trap/engines/engine.py.230: loss: 0.10567 [I 2022-10-27 02:14:40] ...optimizers/trijoint.py.151: total_norm: 8.096378 [I 2022-10-27 02:14:45] ...trap/engines/engine.py.224: train: epoch 42 | batch 990/2304 [I 2022-10-27 02:14:45] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:12:23 [I 2022-10-27 02:14:45] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00300 [I 2022-10-27 02:14:45] ...trap/engines/engine.py.230: loss: 0.15309 [I 2022-10-27 02:14:45] ...optimizers/trijoint.py.151: total_norm: 24.481619 [I 2022-10-27 02:14:51] ...trap/engines/engine.py.224: train: epoch 42 | batch 1000/2304 [I 2022-10-27 02:14:51] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:12:13 [I 2022-10-27 02:14:51] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00256 [I 2022-10-27 02:14:51] ...trap/engines/engine.py.230: loss: 0.12573 [I 2022-10-27 02:14:51] ...optimizers/trijoint.py.151: total_norm: 13.541076 [I 2022-10-27 02:14:57] ...trap/engines/engine.py.224: train: epoch 42 | batch 1010/2304 [I 2022-10-27 02:14:57] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:12:05 [I 2022-10-27 02:14:57] ...trap/engines/engine.py.229: process: 0.56707 | load: 0.00253 [I 2022-10-27 02:14:57] ...trap/engines/engine.py.230: loss: 0.14952 [I 2022-10-27 02:14:57] ...optimizers/trijoint.py.151: total_norm: 19.756704 [I 2022-10-27 02:15:02] ...trap/engines/engine.py.224: train: epoch 42 | batch 1020/2304 [I 2022-10-27 02:15:02] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:12:00 [I 2022-10-27 02:15:02] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00258 [I 2022-10-27 02:15:02] ...trap/engines/engine.py.230: loss: 0.26176 [I 2022-10-27 02:15:02] ...optimizers/trijoint.py.151: total_norm: 21.916876 [I 2022-10-27 02:15:08] ...trap/engines/engine.py.224: train: epoch 42 | batch 1030/2304 [I 2022-10-27 02:15:08] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:56 [I 2022-10-27 02:15:08] ...trap/engines/engine.py.229: process: 0.56520 | load: 0.00257 [I 2022-10-27 02:15:08] ...trap/engines/engine.py.230: loss: 0.17458 [I 2022-10-27 02:15:08] ...optimizers/trijoint.py.151: total_norm: 20.599020 [I 2022-10-27 02:15:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 1040/2304 [I 2022-10-27 02:15:14] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:50 [I 2022-10-27 02:15:14] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00250 [I 2022-10-27 02:15:14] ...trap/engines/engine.py.230: loss: 0.17664 [I 2022-10-27 02:15:14] ...optimizers/trijoint.py.151: total_norm: 20.562796 [I 2022-10-27 02:15:19] ...trap/engines/engine.py.224: train: epoch 42 | batch 1050/2304 [I 2022-10-27 02:15:19] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:45 [I 2022-10-27 02:15:19] ...trap/engines/engine.py.229: process: 0.56816 | load: 0.00258 [I 2022-10-27 02:15:19] ...trap/engines/engine.py.230: loss: 0.18017 [I 2022-10-27 02:15:19] ...optimizers/trijoint.py.151: total_norm: 19.435547 [I 2022-10-27 02:15:25] ...trap/engines/engine.py.224: train: epoch 42 | batch 1060/2304 [I 2022-10-27 02:15:25] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:11:37 [I 2022-10-27 02:15:25] ...trap/engines/engine.py.229: process: 0.54587 | load: 0.00266 [I 2022-10-27 02:15:25] ...trap/engines/engine.py.230: loss: 0.12430 [I 2022-10-27 02:15:25] ...optimizers/trijoint.py.151: total_norm: 33.130310 [I 2022-10-27 02:15:30] ...trap/engines/engine.py.224: train: epoch 42 | batch 1070/2304 [I 2022-10-27 02:15:30] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:11:34 [I 2022-10-27 02:15:30] ...trap/engines/engine.py.229: process: 0.56953 | load: 0.00249 [I 2022-10-27 02:15:30] ...trap/engines/engine.py.230: loss: 0.16692 [I 2022-10-27 02:15:30] ...optimizers/trijoint.py.151: total_norm: 18.773920 [I 2022-10-27 02:15:36] ...trap/engines/engine.py.224: train: epoch 42 | batch 1080/2304 [I 2022-10-27 02:15:36] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:11:33 [I 2022-10-27 02:15:36] ...trap/engines/engine.py.229: process: 0.57036 | load: 0.00259 [I 2022-10-27 02:15:36] ...trap/engines/engine.py.230: loss: 0.16905 [I 2022-10-27 02:15:36] ...optimizers/trijoint.py.151: total_norm: 15.336414 [I 2022-10-27 02:15:42] ...trap/engines/engine.py.224: train: epoch 42 | batch 1090/2304 [I 2022-10-27 02:15:42] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:11:26 [I 2022-10-27 02:15:42] ...trap/engines/engine.py.229: process: 0.56976 | load: 0.00263 [I 2022-10-27 02:15:42] ...trap/engines/engine.py.230: loss: 0.17345 [I 2022-10-27 02:15:42] ...optimizers/trijoint.py.151: total_norm: 20.046202 [I 2022-10-27 02:15:48] ...trap/engines/engine.py.224: train: epoch 42 | batch 1100/2304 [I 2022-10-27 02:15:48] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:11:28 [I 2022-10-27 02:15:48] ...trap/engines/engine.py.229: process: 0.56549 | load: 0.00256 [I 2022-10-27 02:15:48] ...trap/engines/engine.py.230: loss: 0.12033 [I 2022-10-27 02:15:48] ...optimizers/trijoint.py.151: total_norm: 12.111421 [I 2022-10-27 02:15:53] ...trap/engines/engine.py.224: train: epoch 42 | batch 1110/2304 [I 2022-10-27 02:15:53] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:11:17 [I 2022-10-27 02:15:53] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00256 [I 2022-10-27 02:15:53] ...trap/engines/engine.py.230: loss: 0.15713 [I 2022-10-27 02:15:53] ...optimizers/trijoint.py.151: total_norm: 22.900829 [I 2022-10-27 02:15:59] ...trap/engines/engine.py.224: train: epoch 42 | batch 1120/2304 [I 2022-10-27 02:15:59] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:11:09 [I 2022-10-27 02:15:59] ...trap/engines/engine.py.229: process: 0.55897 | load: 0.00255 [I 2022-10-27 02:15:59] ...trap/engines/engine.py.230: loss: 0.13362 [I 2022-10-27 02:15:59] ...optimizers/trijoint.py.151: total_norm: 10.529916 [I 2022-10-27 02:16:05] ...trap/engines/engine.py.224: train: epoch 42 | batch 1130/2304 [I 2022-10-27 02:16:05] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:11:04 [I 2022-10-27 02:16:05] ...trap/engines/engine.py.229: process: 0.57956 | load: 0.00260 [I 2022-10-27 02:16:05] ...trap/engines/engine.py.230: loss: 0.14507 [I 2022-10-27 02:16:05] ...optimizers/trijoint.py.151: total_norm: 16.705383 [I 2022-10-27 02:16:10] ...trap/engines/engine.py.224: train: epoch 42 | batch 1140/2304 [I 2022-10-27 02:16:10] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:11:00 [I 2022-10-27 02:16:10] ...trap/engines/engine.py.229: process: 0.56802 | load: 0.00275 [I 2022-10-27 02:16:10] ...trap/engines/engine.py.230: loss: 0.30548 [I 2022-10-27 02:16:10] ...optimizers/trijoint.py.151: total_norm: 20.539848 [I 2022-10-27 02:16:16] ...trap/engines/engine.py.224: train: epoch 42 | batch 1150/2304 [I 2022-10-27 02:16:16] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:53 [I 2022-10-27 02:16:16] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00266 [I 2022-10-27 02:16:16] ...trap/engines/engine.py.230: loss: 0.31926 [I 2022-10-27 02:16:16] ...optimizers/trijoint.py.151: total_norm: 16.522539 [I 2022-10-27 02:16:22] ...trap/engines/engine.py.224: train: epoch 42 | batch 1160/2304 [I 2022-10-27 02:16:22] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:45 [I 2022-10-27 02:16:22] ...trap/engines/engine.py.229: process: 0.56404 | load: 0.00250 [I 2022-10-27 02:16:22] ...trap/engines/engine.py.230: loss: 0.15044 [I 2022-10-27 02:16:22] ...optimizers/trijoint.py.151: total_norm: 17.325151 [I 2022-10-27 02:16:27] ...trap/engines/engine.py.224: train: epoch 42 | batch 1170/2304 [I 2022-10-27 02:16:27] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:10:38 [I 2022-10-27 02:16:27] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00256 [I 2022-10-27 02:16:27] ...trap/engines/engine.py.230: loss: 0.15017 [I 2022-10-27 02:16:27] ...optimizers/trijoint.py.151: total_norm: 20.198370 [I 2022-10-27 02:16:33] ...trap/engines/engine.py.224: train: epoch 42 | batch 1180/2304 [I 2022-10-27 02:16:33] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:10:34 [I 2022-10-27 02:16:33] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00295 [I 2022-10-27 02:16:33] ...trap/engines/engine.py.230: loss: 0.21298 [I 2022-10-27 02:16:33] ...optimizers/trijoint.py.151: total_norm: 14.354254 [I 2022-10-27 02:16:39] ...trap/engines/engine.py.224: train: epoch 42 | batch 1190/2304 [I 2022-10-27 02:16:39] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:10:25 [I 2022-10-27 02:16:39] ...trap/engines/engine.py.229: process: 0.56726 | load: 0.00259 [I 2022-10-27 02:16:39] ...trap/engines/engine.py.230: loss: 0.13808 [I 2022-10-27 02:16:39] ...optimizers/trijoint.py.151: total_norm: 15.328521 [I 2022-10-27 02:16:44] ...trap/engines/engine.py.224: train: epoch 42 | batch 1200/2304 [I 2022-10-27 02:16:44] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:10:17 [I 2022-10-27 02:16:44] ...trap/engines/engine.py.229: process: 0.58029 | load: 0.00255 [I 2022-10-27 02:16:44] ...trap/engines/engine.py.230: loss: 0.15169 [I 2022-10-27 02:16:44] ...optimizers/trijoint.py.151: total_norm: 19.455492 [I 2022-10-27 02:16:50] ...trap/engines/engine.py.224: train: epoch 42 | batch 1210/2304 [I 2022-10-27 02:16:50] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:10:08 [I 2022-10-27 02:16:50] ...trap/engines/engine.py.229: process: 0.54668 | load: 0.00265 [I 2022-10-27 02:16:50] ...trap/engines/engine.py.230: loss: 0.25682 [I 2022-10-27 02:16:50] ...optimizers/trijoint.py.151: total_norm: 35.972092 [I 2022-10-27 02:16:55] ...trap/engines/engine.py.224: train: epoch 42 | batch 1220/2304 [I 2022-10-27 02:16:55] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:59 [I 2022-10-27 02:16:55] ...trap/engines/engine.py.229: process: 0.55491 | load: 0.00246 [I 2022-10-27 02:16:55] ...trap/engines/engine.py.230: loss: 0.21435 [I 2022-10-27 02:16:55] ...optimizers/trijoint.py.151: total_norm: 25.871656 [I 2022-10-27 02:17:01] ...trap/engines/engine.py.224: train: epoch 42 | batch 1230/2304 [I 2022-10-27 02:17:01] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:10:10 [I 2022-10-27 02:17:01] ...trap/engines/engine.py.229: process: 0.55015 | load: 0.00244 [I 2022-10-27 02:17:01] ...trap/engines/engine.py.230: loss: 0.16848 [I 2022-10-27 02:17:01] ...optimizers/trijoint.py.151: total_norm: 15.646873 [I 2022-10-27 02:17:07] ...trap/engines/engine.py.224: train: epoch 42 | batch 1240/2304 [I 2022-10-27 02:17:07] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:58 [I 2022-10-27 02:17:07] ...trap/engines/engine.py.229: process: 0.56957 | load: 0.00252 [I 2022-10-27 02:17:07] ...trap/engines/engine.py.230: loss: 0.18847 [I 2022-10-27 02:17:07] ...optimizers/trijoint.py.151: total_norm: 25.783978 [I 2022-10-27 02:17:12] ...trap/engines/engine.py.224: train: epoch 42 | batch 1250/2304 [I 2022-10-27 02:17:12] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:42 [I 2022-10-27 02:17:12] ...trap/engines/engine.py.229: process: 0.54793 | load: 0.00242 [I 2022-10-27 02:17:12] ...trap/engines/engine.py.230: loss: 0.19106 [I 2022-10-27 02:17:12] ...optimizers/trijoint.py.151: total_norm: 21.440641 [I 2022-10-27 02:17:18] ...trap/engines/engine.py.224: train: epoch 42 | batch 1260/2304 [I 2022-10-27 02:17:18] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:39 [I 2022-10-27 02:17:18] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00245 [I 2022-10-27 02:17:18] ...trap/engines/engine.py.230: loss: 0.19876 [I 2022-10-27 02:17:18] ...optimizers/trijoint.py.151: total_norm: 13.665696 [I 2022-10-27 02:17:23] ...trap/engines/engine.py.224: train: epoch 42 | batch 1270/2304 [I 2022-10-27 02:17:23] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:09:33 [I 2022-10-27 02:17:23] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00244 [I 2022-10-27 02:17:23] ...trap/engines/engine.py.230: loss: 0.16076 [I 2022-10-27 02:17:23] ...optimizers/trijoint.py.151: total_norm: 13.122738 [I 2022-10-27 02:17:29] ...trap/engines/engine.py.224: train: epoch 42 | batch 1280/2304 [I 2022-10-27 02:17:29] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:09:30 [I 2022-10-27 02:17:29] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00244 [I 2022-10-27 02:17:29] ...trap/engines/engine.py.230: loss: 0.14282 [I 2022-10-27 02:17:29] ...optimizers/trijoint.py.151: total_norm: 9.476092 [I 2022-10-27 02:17:34] ...trap/engines/engine.py.224: train: epoch 42 | batch 1290/2304 [I 2022-10-27 02:17:34] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:09:26 [I 2022-10-27 02:17:34] ...trap/engines/engine.py.229: process: 0.56411 | load: 0.00244 [I 2022-10-27 02:17:34] ...trap/engines/engine.py.230: loss: 0.20048 [I 2022-10-27 02:17:34] ...optimizers/trijoint.py.151: total_norm: 13.463615 [I 2022-10-27 02:17:40] ...trap/engines/engine.py.224: train: epoch 42 | batch 1300/2304 [I 2022-10-27 02:17:40] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:09:19 [I 2022-10-27 02:17:40] ...trap/engines/engine.py.229: process: 0.56674 | load: 0.00241 [I 2022-10-27 02:17:40] ...trap/engines/engine.py.230: loss: 0.25062 [I 2022-10-27 02:17:40] ...optimizers/trijoint.py.151: total_norm: 19.503189 [I 2022-10-27 02:17:46] ...trap/engines/engine.py.224: train: epoch 42 | batch 1310/2304 [I 2022-10-27 02:17:46] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:09:12 [I 2022-10-27 02:17:46] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00242 [I 2022-10-27 02:17:46] ...trap/engines/engine.py.230: loss: 0.20156 [I 2022-10-27 02:17:46] ...optimizers/trijoint.py.151: total_norm: 43.383862 [I 2022-10-27 02:17:51] ...trap/engines/engine.py.224: train: epoch 42 | batch 1320/2304 [I 2022-10-27 02:17:51] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:09:08 [I 2022-10-27 02:17:51] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00241 [I 2022-10-27 02:17:51] ...trap/engines/engine.py.230: loss: 0.16410 [I 2022-10-27 02:17:51] ...optimizers/trijoint.py.151: total_norm: 15.518565 [I 2022-10-27 02:17:57] ...trap/engines/engine.py.224: train: epoch 42 | batch 1330/2304 [I 2022-10-27 02:17:57] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:09:04 [I 2022-10-27 02:17:57] ...trap/engines/engine.py.229: process: 0.56512 | load: 0.00251 [I 2022-10-27 02:17:57] ...trap/engines/engine.py.230: loss: 0.29639 [I 2022-10-27 02:17:57] ...optimizers/trijoint.py.151: total_norm: 36.612579 [I 2022-10-27 02:18:02] ...trap/engines/engine.py.224: train: epoch 42 | batch 1340/2304 [I 2022-10-27 02:18:02] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:56 [I 2022-10-27 02:18:02] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00247 [I 2022-10-27 02:18:02] ...trap/engines/engine.py.230: loss: 0.16842 [I 2022-10-27 02:18:02] ...optimizers/trijoint.py.151: total_norm: 13.807766 [I 2022-10-27 02:18:08] ...trap/engines/engine.py.224: train: epoch 42 | batch 1350/2304 [I 2022-10-27 02:18:08] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:54 [I 2022-10-27 02:18:08] ...trap/engines/engine.py.229: process: 0.55506 | load: 0.00247 [I 2022-10-27 02:18:08] ...trap/engines/engine.py.230: loss: 0.13546 [I 2022-10-27 02:18:08] ...optimizers/trijoint.py.151: total_norm: 18.857588 [I 2022-10-27 02:18:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 1360/2304 [I 2022-10-27 02:18:14] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:43 [I 2022-10-27 02:18:14] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00245 [I 2022-10-27 02:18:14] ...trap/engines/engine.py.230: loss: 0.20556 [I 2022-10-27 02:18:14] ...optimizers/trijoint.py.151: total_norm: 19.628202 [I 2022-10-27 02:18:19] ...trap/engines/engine.py.224: train: epoch 42 | batch 1370/2304 [I 2022-10-27 02:18:19] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:41 [I 2022-10-27 02:18:19] ...trap/engines/engine.py.229: process: 0.55835 | load: 0.00243 [I 2022-10-27 02:18:19] ...trap/engines/engine.py.230: loss: 0.13918 [I 2022-10-27 02:18:19] ...optimizers/trijoint.py.151: total_norm: 15.205879 [I 2022-10-27 02:18:25] ...trap/engines/engine.py.224: train: epoch 42 | batch 1380/2304 [I 2022-10-27 02:18:25] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:08:32 [I 2022-10-27 02:18:25] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00264 [I 2022-10-27 02:18:25] ...trap/engines/engine.py.230: loss: 0.16719 [I 2022-10-27 02:18:25] ...optimizers/trijoint.py.151: total_norm: 14.301820 [I 2022-10-27 02:18:30] ...trap/engines/engine.py.224: train: epoch 42 | batch 1390/2304 [I 2022-10-27 02:18:30] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:08:26 [I 2022-10-27 02:18:30] ...trap/engines/engine.py.229: process: 0.55211 | load: 0.00243 [I 2022-10-27 02:18:30] ...trap/engines/engine.py.230: loss: 0.21621 [I 2022-10-27 02:18:30] ...optimizers/trijoint.py.151: total_norm: 18.353300 [I 2022-10-27 02:18:36] ...trap/engines/engine.py.224: train: epoch 42 | batch 1400/2304 [I 2022-10-27 02:18:36] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:08:19 [I 2022-10-27 02:18:36] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00243 [I 2022-10-27 02:18:36] ...trap/engines/engine.py.230: loss: 0.16813 [I 2022-10-27 02:18:36] ...optimizers/trijoint.py.151: total_norm: 15.854406 [I 2022-10-27 02:18:41] ...trap/engines/engine.py.224: train: epoch 42 | batch 1410/2304 [I 2022-10-27 02:18:41] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:08:18 [I 2022-10-27 02:18:41] ...trap/engines/engine.py.229: process: 0.54493 | load: 0.00246 [I 2022-10-27 02:18:41] ...trap/engines/engine.py.230: loss: 0.23355 [I 2022-10-27 02:18:41] ...optimizers/trijoint.py.151: total_norm: 13.177122 [I 2022-10-27 02:18:47] ...trap/engines/engine.py.224: train: epoch 42 | batch 1420/2304 [I 2022-10-27 02:18:47] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:08:12 [I 2022-10-27 02:18:47] ...trap/engines/engine.py.229: process: 0.54928 | load: 0.00246 [I 2022-10-27 02:18:47] ...trap/engines/engine.py.230: loss: 0.23679 [I 2022-10-27 02:18:47] ...optimizers/trijoint.py.151: total_norm: 20.832624 [I 2022-10-27 02:18:53] ...trap/engines/engine.py.224: train: epoch 42 | batch 1430/2304 [I 2022-10-27 02:18:53] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:08:08 [I 2022-10-27 02:18:53] ...trap/engines/engine.py.229: process: 0.55509 | load: 0.00245 [I 2022-10-27 02:18:53] ...trap/engines/engine.py.230: loss: 0.15020 [I 2022-10-27 02:18:53] ...optimizers/trijoint.py.151: total_norm: 21.675123 [I 2022-10-27 02:18:58] ...trap/engines/engine.py.224: train: epoch 42 | batch 1440/2304 [I 2022-10-27 02:18:58] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:59 [I 2022-10-27 02:18:58] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00253 [I 2022-10-27 02:18:58] ...trap/engines/engine.py.230: loss: 0.14137 [I 2022-10-27 02:18:58] ...optimizers/trijoint.py.151: total_norm: 14.264924 [I 2022-10-27 02:19:04] ...trap/engines/engine.py.224: train: epoch 42 | batch 1450/2304 [I 2022-10-27 02:19:04] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:50 [I 2022-10-27 02:19:04] ...trap/engines/engine.py.229: process: 0.53976 | load: 0.00248 [I 2022-10-27 02:19:04] ...trap/engines/engine.py.230: loss: 0.13468 [I 2022-10-27 02:19:04] ...optimizers/trijoint.py.151: total_norm: 9.350828 [I 2022-10-27 02:19:09] ...trap/engines/engine.py.224: train: epoch 42 | batch 1460/2304 [I 2022-10-27 02:19:09] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:53 [I 2022-10-27 02:19:09] ...trap/engines/engine.py.229: process: 0.54551 | load: 0.00246 [I 2022-10-27 02:19:09] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-27 02:19:09] ...optimizers/trijoint.py.151: total_norm: 13.670996 [I 2022-10-27 02:19:15] ...trap/engines/engine.py.224: train: epoch 42 | batch 1470/2304 [I 2022-10-27 02:19:15] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:44 [I 2022-10-27 02:19:15] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00242 [I 2022-10-27 02:19:15] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-27 02:19:15] ...optimizers/trijoint.py.151: total_norm: 13.047704 [I 2022-10-27 02:19:21] ...trap/engines/engine.py.224: train: epoch 42 | batch 1480/2304 [I 2022-10-27 02:19:21] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:38 [I 2022-10-27 02:19:21] ...trap/engines/engine.py.229: process: 0.56585 | load: 0.00242 [I 2022-10-27 02:19:21] ...trap/engines/engine.py.230: loss: 0.12547 [I 2022-10-27 02:19:21] ...optimizers/trijoint.py.151: total_norm: 16.823441 [I 2022-10-27 02:19:26] ...trap/engines/engine.py.224: train: epoch 42 | batch 1490/2304 [I 2022-10-27 02:19:26] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:07:36 [I 2022-10-27 02:19:26] ...trap/engines/engine.py.229: process: 0.56930 | load: 0.00250 [I 2022-10-27 02:19:26] ...trap/engines/engine.py.230: loss: 0.20038 [I 2022-10-27 02:19:26] ...optimizers/trijoint.py.151: total_norm: 30.964207 [I 2022-10-27 02:19:32] ...trap/engines/engine.py.224: train: epoch 42 | batch 1500/2304 [I 2022-10-27 02:19:32] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:07:27 [I 2022-10-27 02:19:32] ...trap/engines/engine.py.229: process: 0.56491 | load: 0.00244 [I 2022-10-27 02:19:32] ...trap/engines/engine.py.230: loss: 0.17285 [I 2022-10-27 02:19:32] ...optimizers/trijoint.py.151: total_norm: 29.756632 [I 2022-10-27 02:19:37] ...trap/engines/engine.py.224: train: epoch 42 | batch 1510/2304 [I 2022-10-27 02:19:37] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:07:20 [I 2022-10-27 02:19:37] ...trap/engines/engine.py.229: process: 0.54908 | load: 0.00242 [I 2022-10-27 02:19:37] ...trap/engines/engine.py.230: loss: 0.12958 [I 2022-10-27 02:19:37] ...optimizers/trijoint.py.151: total_norm: 21.128735 [I 2022-10-27 02:19:43] ...trap/engines/engine.py.224: train: epoch 42 | batch 1520/2304 [I 2022-10-27 02:19:43] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:07:17 [I 2022-10-27 02:19:43] ...trap/engines/engine.py.229: process: 0.55578 | load: 0.00241 [I 2022-10-27 02:19:43] ...trap/engines/engine.py.230: loss: 0.16034 [I 2022-10-27 02:19:43] ...optimizers/trijoint.py.151: total_norm: 12.235829 [I 2022-10-27 02:19:48] ...trap/engines/engine.py.224: train: epoch 42 | batch 1530/2304 [I 2022-10-27 02:19:48] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:07:11 [I 2022-10-27 02:19:48] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00243 [I 2022-10-27 02:19:48] ...trap/engines/engine.py.230: loss: 0.19424 [I 2022-10-27 02:19:48] ...optimizers/trijoint.py.151: total_norm: 14.461953 [I 2022-10-27 02:19:54] ...trap/engines/engine.py.224: train: epoch 42 | batch 1540/2304 [I 2022-10-27 02:19:54] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:07:01 [I 2022-10-27 02:19:54] ...trap/engines/engine.py.229: process: 0.55046 | load: 0.00242 [I 2022-10-27 02:19:54] ...trap/engines/engine.py.230: loss: 0.15128 [I 2022-10-27 02:19:54] ...optimizers/trijoint.py.151: total_norm: 16.718803 [I 2022-10-27 02:20:00] ...trap/engines/engine.py.224: train: epoch 42 | batch 1550/2304 [I 2022-10-27 02:20:00] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:07:00 [I 2022-10-27 02:20:00] ...trap/engines/engine.py.229: process: 0.56393 | load: 0.00251 [I 2022-10-27 02:20:00] ...trap/engines/engine.py.230: loss: 0.23909 [I 2022-10-27 02:20:00] ...optimizers/trijoint.py.151: total_norm: 30.379444 [I 2022-10-27 02:20:05] ...trap/engines/engine.py.224: train: epoch 42 | batch 1560/2304 [I 2022-10-27 02:20:05] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:50 [I 2022-10-27 02:20:05] ...trap/engines/engine.py.229: process: 0.54414 | load: 0.00249 [I 2022-10-27 02:20:05] ...trap/engines/engine.py.230: loss: 0.17419 [I 2022-10-27 02:20:05] ...optimizers/trijoint.py.151: total_norm: 21.518837 [I 2022-10-27 02:20:11] ...trap/engines/engine.py.224: train: epoch 42 | batch 1570/2304 [I 2022-10-27 02:20:11] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:52 [I 2022-10-27 02:20:11] ...trap/engines/engine.py.229: process: 0.56406 | load: 0.00245 [I 2022-10-27 02:20:11] ...trap/engines/engine.py.230: loss: 0.21289 [I 2022-10-27 02:20:11] ...optimizers/trijoint.py.151: total_norm: 13.946424 [I 2022-10-27 02:20:16] ...trap/engines/engine.py.224: train: epoch 42 | batch 1580/2304 [I 2022-10-27 02:20:16] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:44 [I 2022-10-27 02:20:16] ...trap/engines/engine.py.229: process: 0.56393 | load: 0.00246 [I 2022-10-27 02:20:16] ...trap/engines/engine.py.230: loss: 0.12630 [I 2022-10-27 02:20:16] ...optimizers/trijoint.py.151: total_norm: 15.886836 [I 2022-10-27 02:20:22] ...trap/engines/engine.py.224: train: epoch 42 | batch 1590/2304 [I 2022-10-27 02:20:22] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:39 [I 2022-10-27 02:20:22] ...trap/engines/engine.py.229: process: 0.56414 | load: 0.00243 [I 2022-10-27 02:20:22] ...trap/engines/engine.py.230: loss: 0.20439 [I 2022-10-27 02:20:22] ...optimizers/trijoint.py.151: total_norm: 16.901236 [I 2022-10-27 02:20:28] ...trap/engines/engine.py.224: train: epoch 42 | batch 1600/2304 [I 2022-10-27 02:20:28] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:34 [I 2022-10-27 02:20:28] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00246 [I 2022-10-27 02:20:28] ...trap/engines/engine.py.230: loss: 0.17330 [I 2022-10-27 02:20:28] ...optimizers/trijoint.py.151: total_norm: 22.976049 [I 2022-10-27 02:20:33] ...trap/engines/engine.py.224: train: epoch 42 | batch 1610/2304 [I 2022-10-27 02:20:33] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:06:28 [I 2022-10-27 02:20:33] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00243 [I 2022-10-27 02:20:33] ...trap/engines/engine.py.230: loss: 0.16842 [I 2022-10-27 02:20:33] ...optimizers/trijoint.py.151: total_norm: 12.828766 [I 2022-10-27 02:20:39] ...trap/engines/engine.py.224: train: epoch 42 | batch 1620/2304 [I 2022-10-27 02:20:39] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:06:23 [I 2022-10-27 02:20:39] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00242 [I 2022-10-27 02:20:39] ...trap/engines/engine.py.230: loss: 0.18470 [I 2022-10-27 02:20:39] ...optimizers/trijoint.py.151: total_norm: 14.089228 [I 2022-10-27 02:20:45] ...trap/engines/engine.py.224: train: epoch 42 | batch 1630/2304 [I 2022-10-27 02:20:45] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:06:27 [I 2022-10-27 02:20:45] ...trap/engines/engine.py.229: process: 0.55737 | load: 0.00246 [I 2022-10-27 02:20:45] ...trap/engines/engine.py.230: loss: 0.20090 [I 2022-10-27 02:20:45] ...optimizers/trijoint.py.151: total_norm: 22.111795 [I 2022-10-27 02:20:50] ...trap/engines/engine.py.224: train: epoch 42 | batch 1640/2304 [I 2022-10-27 02:20:50] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:06:11 [I 2022-10-27 02:20:50] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00259 [I 2022-10-27 02:20:50] ...trap/engines/engine.py.230: loss: 0.15536 [I 2022-10-27 02:20:50] ...optimizers/trijoint.py.151: total_norm: 12.705949 [I 2022-10-27 02:20:56] ...trap/engines/engine.py.224: train: epoch 42 | batch 1650/2304 [I 2022-10-27 02:20:56] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:06:05 [I 2022-10-27 02:20:56] ...trap/engines/engine.py.229: process: 0.54898 | load: 0.00255 [I 2022-10-27 02:20:56] ...trap/engines/engine.py.230: loss: 0.13104 [I 2022-10-27 02:20:56] ...optimizers/trijoint.py.151: total_norm: 13.683065 [I 2022-10-27 02:21:01] ...trap/engines/engine.py.224: train: epoch 42 | batch 1660/2304 [I 2022-10-27 02:21:01] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:58 [I 2022-10-27 02:21:01] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00257 [I 2022-10-27 02:21:01] ...trap/engines/engine.py.230: loss: 0.14217 [I 2022-10-27 02:21:01] ...optimizers/trijoint.py.151: total_norm: 8.579741 [I 2022-10-27 02:21:07] ...trap/engines/engine.py.224: train: epoch 42 | batch 1670/2304 [I 2022-10-27 02:21:07] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:55 [I 2022-10-27 02:21:07] ...trap/engines/engine.py.229: process: 0.55272 | load: 0.00265 [I 2022-10-27 02:21:07] ...trap/engines/engine.py.230: loss: 0.13781 [I 2022-10-27 02:21:07] ...optimizers/trijoint.py.151: total_norm: 12.089967 [I 2022-10-27 02:21:13] ...trap/engines/engine.py.224: train: epoch 42 | batch 1680/2304 [I 2022-10-27 02:21:13] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:49 [I 2022-10-27 02:21:13] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00244 [I 2022-10-27 02:21:13] ...trap/engines/engine.py.230: loss: 0.14188 [I 2022-10-27 02:21:13] ...optimizers/trijoint.py.151: total_norm: 20.975571 [I 2022-10-27 02:21:18] ...trap/engines/engine.py.224: train: epoch 42 | batch 1690/2304 [I 2022-10-27 02:21:18] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:44 [I 2022-10-27 02:21:18] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00245 [I 2022-10-27 02:21:18] ...trap/engines/engine.py.230: loss: 0.14461 [I 2022-10-27 02:21:18] ...optimizers/trijoint.py.151: total_norm: 12.883057 [I 2022-10-27 02:21:24] ...trap/engines/engine.py.224: train: epoch 42 | batch 1700/2304 [I 2022-10-27 02:21:24] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:36 [I 2022-10-27 02:21:24] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00248 [I 2022-10-27 02:21:24] ...trap/engines/engine.py.230: loss: 0.12394 [I 2022-10-27 02:21:24] ...optimizers/trijoint.py.151: total_norm: 13.053879 [I 2022-10-27 02:21:29] ...trap/engines/engine.py.224: train: epoch 42 | batch 1710/2304 [I 2022-10-27 02:21:29] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:05:30 [I 2022-10-27 02:21:29] ...trap/engines/engine.py.229: process: 0.56335 | load: 0.00245 [I 2022-10-27 02:21:29] ...trap/engines/engine.py.230: loss: 0.21091 [I 2022-10-27 02:21:29] ...optimizers/trijoint.py.151: total_norm: 14.507259 [I 2022-10-27 02:21:35] ...trap/engines/engine.py.224: train: epoch 42 | batch 1720/2304 [I 2022-10-27 02:21:35] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:27 [I 2022-10-27 02:21:35] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00242 [I 2022-10-27 02:21:35] ...trap/engines/engine.py.230: loss: 0.21165 [I 2022-10-27 02:21:35] ...optimizers/trijoint.py.151: total_norm: 14.589071 [I 2022-10-27 02:21:40] ...trap/engines/engine.py.224: train: epoch 42 | batch 1730/2304 [I 2022-10-27 02:21:40] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:05:19 [I 2022-10-27 02:21:40] ...trap/engines/engine.py.229: process: 0.55100 | load: 0.00260 [I 2022-10-27 02:21:40] ...trap/engines/engine.py.230: loss: 0.16569 [I 2022-10-27 02:21:40] ...optimizers/trijoint.py.151: total_norm: 19.160124 [I 2022-10-27 02:21:46] ...trap/engines/engine.py.224: train: epoch 42 | batch 1740/2304 [I 2022-10-27 02:21:46] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:05:14 [I 2022-10-27 02:21:46] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00245 [I 2022-10-27 02:21:46] ...trap/engines/engine.py.230: loss: 0.19891 [I 2022-10-27 02:21:46] ...optimizers/trijoint.py.151: total_norm: 23.896151 [I 2022-10-27 02:21:52] ...trap/engines/engine.py.224: train: epoch 42 | batch 1750/2304 [I 2022-10-27 02:21:52] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:05:08 [I 2022-10-27 02:21:52] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00244 [I 2022-10-27 02:21:52] ...trap/engines/engine.py.230: loss: 0.15321 [I 2022-10-27 02:21:52] ...optimizers/trijoint.py.151: total_norm: 16.968876 [I 2022-10-27 02:21:57] ...trap/engines/engine.py.224: train: epoch 42 | batch 1760/2304 [I 2022-10-27 02:21:57] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:05:00 [I 2022-10-27 02:21:57] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00243 [I 2022-10-27 02:21:57] ...trap/engines/engine.py.230: loss: 0.14895 [I 2022-10-27 02:21:57] ...optimizers/trijoint.py.151: total_norm: 14.130565 [I 2022-10-27 02:22:03] ...trap/engines/engine.py.224: train: epoch 42 | batch 1770/2304 [I 2022-10-27 02:22:03] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:57 [I 2022-10-27 02:22:03] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00241 [I 2022-10-27 02:22:03] ...trap/engines/engine.py.230: loss: 0.18612 [I 2022-10-27 02:22:03] ...optimizers/trijoint.py.151: total_norm: 17.525787 [I 2022-10-27 02:22:08] ...trap/engines/engine.py.224: train: epoch 42 | batch 1780/2304 [I 2022-10-27 02:22:08] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:50 [I 2022-10-27 02:22:08] ...trap/engines/engine.py.229: process: 0.54540 | load: 0.00244 [I 2022-10-27 02:22:08] ...trap/engines/engine.py.230: loss: 0.23355 [I 2022-10-27 02:22:08] ...optimizers/trijoint.py.151: total_norm: 23.853045 [I 2022-10-27 02:22:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 1790/2304 [I 2022-10-27 02:22:14] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:45 [I 2022-10-27 02:22:14] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00251 [I 2022-10-27 02:22:14] ...trap/engines/engine.py.230: loss: 0.13658 [I 2022-10-27 02:22:14] ...optimizers/trijoint.py.151: total_norm: 11.896947 [I 2022-10-27 02:22:19] ...trap/engines/engine.py.224: train: epoch 42 | batch 1800/2304 [I 2022-10-27 02:22:19] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:39 [I 2022-10-27 02:22:19] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00245 [I 2022-10-27 02:22:19] ...trap/engines/engine.py.230: loss: 0.21272 [I 2022-10-27 02:22:19] ...optimizers/trijoint.py.151: total_norm: 28.425575 [I 2022-10-27 02:22:25] ...trap/engines/engine.py.224: train: epoch 42 | batch 1810/2304 [I 2022-10-27 02:22:25] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:04:35 [I 2022-10-27 02:22:25] ...trap/engines/engine.py.229: process: 0.56450 | load: 0.00243 [I 2022-10-27 02:22:25] ...trap/engines/engine.py.230: loss: 0.25115 [I 2022-10-27 02:22:25] ...optimizers/trijoint.py.151: total_norm: 22.827707 [I 2022-10-27 02:22:31] ...trap/engines/engine.py.224: train: epoch 42 | batch 1820/2304 [I 2022-10-27 02:22:31] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:04:30 [I 2022-10-27 02:22:31] ...trap/engines/engine.py.229: process: 0.56343 | load: 0.00244 [I 2022-10-27 02:22:31] ...trap/engines/engine.py.230: loss: 0.15188 [I 2022-10-27 02:22:31] ...optimizers/trijoint.py.151: total_norm: 20.243090 [I 2022-10-27 02:22:36] ...trap/engines/engine.py.224: train: epoch 42 | batch 1830/2304 [I 2022-10-27 02:22:36] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:04:25 [I 2022-10-27 02:22:36] ...trap/engines/engine.py.229: process: 0.56434 | load: 0.00245 [I 2022-10-27 02:22:36] ...trap/engines/engine.py.230: loss: 0.14267 [I 2022-10-27 02:22:36] ...optimizers/trijoint.py.151: total_norm: 14.297784 [I 2022-10-27 02:22:42] ...trap/engines/engine.py.224: train: epoch 42 | batch 1840/2304 [I 2022-10-27 02:22:42] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:04:18 [I 2022-10-27 02:22:42] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00243 [I 2022-10-27 02:22:42] ...trap/engines/engine.py.230: loss: 0.24183 [I 2022-10-27 02:22:42] ...optimizers/trijoint.py.151: total_norm: 31.364735 [I 2022-10-27 02:22:47] ...trap/engines/engine.py.224: train: epoch 42 | batch 1850/2304 [I 2022-10-27 02:22:47] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:04:12 [I 2022-10-27 02:22:47] ...trap/engines/engine.py.229: process: 0.56086 | load: 0.00243 [I 2022-10-27 02:22:47] ...trap/engines/engine.py.230: loss: 0.18537 [I 2022-10-27 02:22:47] ...optimizers/trijoint.py.151: total_norm: 21.648031 [I 2022-10-27 02:22:53] ...trap/engines/engine.py.224: train: epoch 42 | batch 1860/2304 [I 2022-10-27 02:22:53] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:04:06 [I 2022-10-27 02:22:53] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00245 [I 2022-10-27 02:22:53] ...trap/engines/engine.py.230: loss: 0.14025 [I 2022-10-27 02:22:53] ...optimizers/trijoint.py.151: total_norm: 13.735847 [I 2022-10-27 02:22:59] ...trap/engines/engine.py.224: train: epoch 42 | batch 1870/2304 [I 2022-10-27 02:22:59] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:04:02 [I 2022-10-27 02:22:59] ...trap/engines/engine.py.229: process: 0.55293 | load: 0.00250 [I 2022-10-27 02:22:59] ...trap/engines/engine.py.230: loss: 0.19950 [I 2022-10-27 02:22:59] ...optimizers/trijoint.py.151: total_norm: 23.333298 [I 2022-10-27 02:23:04] ...trap/engines/engine.py.224: train: epoch 42 | batch 1880/2304 [I 2022-10-27 02:23:04] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:56 [I 2022-10-27 02:23:04] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00245 [I 2022-10-27 02:23:04] ...trap/engines/engine.py.230: loss: 0.14759 [I 2022-10-27 02:23:04] ...optimizers/trijoint.py.151: total_norm: 19.806736 [I 2022-10-27 02:23:10] ...trap/engines/engine.py.224: train: epoch 42 | batch 1890/2304 [I 2022-10-27 02:23:10] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:49 [I 2022-10-27 02:23:10] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00247 [I 2022-10-27 02:23:10] ...trap/engines/engine.py.230: loss: 0.17176 [I 2022-10-27 02:23:10] ...optimizers/trijoint.py.151: total_norm: 17.843969 [I 2022-10-27 02:23:15] ...trap/engines/engine.py.224: train: epoch 42 | batch 1900/2304 [I 2022-10-27 02:23:15] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:43 [I 2022-10-27 02:23:15] ...trap/engines/engine.py.229: process: 0.55001 | load: 0.00246 [I 2022-10-27 02:23:15] ...trap/engines/engine.py.230: loss: 0.19461 [I 2022-10-27 02:23:15] ...optimizers/trijoint.py.151: total_norm: 17.345636 [I 2022-10-27 02:23:21] ...trap/engines/engine.py.224: train: epoch 42 | batch 1910/2304 [I 2022-10-27 02:23:21] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:39 [I 2022-10-27 02:23:21] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00243 [I 2022-10-27 02:23:21] ...trap/engines/engine.py.230: loss: 0.19906 [I 2022-10-27 02:23:21] ...optimizers/trijoint.py.151: total_norm: 14.822495 [I 2022-10-27 02:23:26] ...trap/engines/engine.py.224: train: epoch 42 | batch 1920/2304 [I 2022-10-27 02:23:26] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:32 [I 2022-10-27 02:23:26] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00243 [I 2022-10-27 02:23:26] ...trap/engines/engine.py.230: loss: 0.22130 [I 2022-10-27 02:23:26] ...optimizers/trijoint.py.151: total_norm: 17.611307 [I 2022-10-27 02:23:32] ...trap/engines/engine.py.224: train: epoch 42 | batch 1930/2304 [I 2022-10-27 02:23:32] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:03:26 [I 2022-10-27 02:23:32] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00243 [I 2022-10-27 02:23:32] ...trap/engines/engine.py.230: loss: 0.19997 [I 2022-10-27 02:23:32] ...optimizers/trijoint.py.151: total_norm: 27.846413 [I 2022-10-27 02:23:37] ...trap/engines/engine.py.224: train: epoch 42 | batch 1940/2304 [I 2022-10-27 02:23:37] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:03:22 [I 2022-10-27 02:23:37] ...trap/engines/engine.py.229: process: 0.55202 | load: 0.00241 [I 2022-10-27 02:23:37] ...trap/engines/engine.py.230: loss: 0.14974 [I 2022-10-27 02:23:37] ...optimizers/trijoint.py.151: total_norm: 18.227819 [I 2022-10-27 02:23:43] ...trap/engines/engine.py.224: train: epoch 42 | batch 1950/2304 [I 2022-10-27 02:23:43] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:03:16 [I 2022-10-27 02:23:43] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00240 [I 2022-10-27 02:23:43] ...trap/engines/engine.py.230: loss: 0.12672 [I 2022-10-27 02:23:43] ...optimizers/trijoint.py.151: total_norm: 11.958494 [I 2022-10-27 02:23:49] ...trap/engines/engine.py.224: train: epoch 42 | batch 1960/2304 [I 2022-10-27 02:23:49] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:03:11 [I 2022-10-27 02:23:49] ...trap/engines/engine.py.229: process: 0.56703 | load: 0.00288 [I 2022-10-27 02:23:49] ...trap/engines/engine.py.230: loss: 0.21849 [I 2022-10-27 02:23:49] ...optimizers/trijoint.py.151: total_norm: 25.094034 [I 2022-10-27 02:23:54] ...trap/engines/engine.py.224: train: epoch 42 | batch 1970/2304 [I 2022-10-27 02:23:54] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:03:06 [I 2022-10-27 02:23:54] ...trap/engines/engine.py.229: process: 0.56723 | load: 0.00251 [I 2022-10-27 02:23:54] ...trap/engines/engine.py.230: loss: 0.28237 [I 2022-10-27 02:23:54] ...optimizers/trijoint.py.151: total_norm: 18.849314 [I 2022-10-27 02:24:00] ...trap/engines/engine.py.224: train: epoch 42 | batch 1980/2304 [I 2022-10-27 02:24:00] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:03:00 [I 2022-10-27 02:24:00] ...trap/engines/engine.py.229: process: 0.55305 | load: 0.00245 [I 2022-10-27 02:24:00] ...trap/engines/engine.py.230: loss: 0.09698 [I 2022-10-27 02:24:00] ...optimizers/trijoint.py.151: total_norm: 8.545424 [I 2022-10-27 02:24:05] ...trap/engines/engine.py.224: train: epoch 42 | batch 1990/2304 [I 2022-10-27 02:24:05] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:54 [I 2022-10-27 02:24:05] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00240 [I 2022-10-27 02:24:05] ...trap/engines/engine.py.230: loss: 0.19913 [I 2022-10-27 02:24:05] ...optimizers/trijoint.py.151: total_norm: 25.184441 [I 2022-10-27 02:24:11] ...trap/engines/engine.py.224: train: epoch 42 | batch 2000/2304 [I 2022-10-27 02:24:11] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:48 [I 2022-10-27 02:24:11] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00244 [I 2022-10-27 02:24:11] ...trap/engines/engine.py.230: loss: 0.13015 [I 2022-10-27 02:24:11] ...optimizers/trijoint.py.151: total_norm: 19.772181 [I 2022-10-27 02:24:16] ...trap/engines/engine.py.224: train: epoch 42 | batch 2010/2304 [I 2022-10-27 02:24:16] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:42 [I 2022-10-27 02:24:16] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00263 [I 2022-10-27 02:24:16] ...trap/engines/engine.py.230: loss: 0.16642 [I 2022-10-27 02:24:16] ...optimizers/trijoint.py.151: total_norm: 17.968330 [I 2022-10-27 02:24:22] ...trap/engines/engine.py.224: train: epoch 42 | batch 2020/2304 [I 2022-10-27 02:24:22] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:02:36 [I 2022-10-27 02:24:22] ...trap/engines/engine.py.229: process: 0.55189 | load: 0.00243 [I 2022-10-27 02:24:22] ...trap/engines/engine.py.230: loss: 0.22959 [I 2022-10-27 02:24:22] ...optimizers/trijoint.py.151: total_norm: 16.771893 [I 2022-10-27 02:24:28] ...trap/engines/engine.py.224: train: epoch 42 | batch 2030/2304 [I 2022-10-27 02:24:28] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:02:32 [I 2022-10-27 02:24:28] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00241 [I 2022-10-27 02:24:28] ...trap/engines/engine.py.230: loss: 0.19587 [I 2022-10-27 02:24:28] ...optimizers/trijoint.py.151: total_norm: 19.095205 [I 2022-10-27 02:24:33] ...trap/engines/engine.py.224: train: epoch 42 | batch 2040/2304 [I 2022-10-27 02:24:33] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:02:26 [I 2022-10-27 02:24:33] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00240 [I 2022-10-27 02:24:33] ...trap/engines/engine.py.230: loss: 0.13969 [I 2022-10-27 02:24:33] ...optimizers/trijoint.py.151: total_norm: 20.529219 [I 2022-10-27 02:24:39] ...trap/engines/engine.py.224: train: epoch 42 | batch 2050/2304 [I 2022-10-27 02:24:39] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:02:22 [I 2022-10-27 02:24:39] ...trap/engines/engine.py.229: process: 0.56565 | load: 0.00244 [I 2022-10-27 02:24:39] ...trap/engines/engine.py.230: loss: 0.15758 [I 2022-10-27 02:24:39] ...optimizers/trijoint.py.151: total_norm: 14.983127 [I 2022-10-27 02:24:44] ...trap/engines/engine.py.224: train: epoch 42 | batch 2060/2304 [I 2022-10-27 02:24:44] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:02:15 [I 2022-10-27 02:24:44] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00244 [I 2022-10-27 02:24:44] ...trap/engines/engine.py.230: loss: 0.21376 [I 2022-10-27 02:24:44] ...optimizers/trijoint.py.151: total_norm: 20.265913 [I 2022-10-27 02:24:50] ...trap/engines/engine.py.224: train: epoch 42 | batch 2070/2304 [I 2022-10-27 02:24:50] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:02:09 [I 2022-10-27 02:24:50] ...trap/engines/engine.py.229: process: 0.54731 | load: 0.00245 [I 2022-10-27 02:24:50] ...trap/engines/engine.py.230: loss: 0.17583 [I 2022-10-27 02:24:50] ...optimizers/trijoint.py.151: total_norm: 19.326317 [I 2022-10-27 02:24:55] ...trap/engines/engine.py.224: train: epoch 42 | batch 2080/2304 [I 2022-10-27 02:24:55] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:02:04 [I 2022-10-27 02:24:55] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00247 [I 2022-10-27 02:24:55] ...trap/engines/engine.py.230: loss: 0.09993 [I 2022-10-27 02:24:55] ...optimizers/trijoint.py.151: total_norm: 10.096854 [I 2022-10-27 02:25:01] ...trap/engines/engine.py.224: train: epoch 42 | batch 2090/2304 [I 2022-10-27 02:25:01] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:57 [I 2022-10-27 02:25:01] ...trap/engines/engine.py.229: process: 0.54683 | load: 0.00249 [I 2022-10-27 02:25:01] ...trap/engines/engine.py.230: loss: 0.13632 [I 2022-10-27 02:25:01] ...optimizers/trijoint.py.151: total_norm: 43.348183 [I 2022-10-27 02:25:07] ...trap/engines/engine.py.224: train: epoch 42 | batch 2100/2304 [I 2022-10-27 02:25:07] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:53 [I 2022-10-27 02:25:07] ...trap/engines/engine.py.229: process: 0.55034 | load: 0.00247 [I 2022-10-27 02:25:07] ...trap/engines/engine.py.230: loss: 0.20822 [I 2022-10-27 02:25:07] ...optimizers/trijoint.py.151: total_norm: 22.650114 [I 2022-10-27 02:25:12] ...trap/engines/engine.py.224: train: epoch 42 | batch 2110/2304 [I 2022-10-27 02:25:12] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:47 [I 2022-10-27 02:25:12] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00247 [I 2022-10-27 02:25:12] ...trap/engines/engine.py.230: loss: 0.15405 [I 2022-10-27 02:25:12] ...optimizers/trijoint.py.151: total_norm: 18.147570 [I 2022-10-27 02:25:18] ...trap/engines/engine.py.224: train: epoch 42 | batch 2120/2304 [I 2022-10-27 02:25:18] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:42 [I 2022-10-27 02:25:18] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00247 [I 2022-10-27 02:25:18] ...trap/engines/engine.py.230: loss: 0.17294 [I 2022-10-27 02:25:18] ...optimizers/trijoint.py.151: total_norm: 14.736547 [I 2022-10-27 02:25:23] ...trap/engines/engine.py.224: train: epoch 42 | batch 2130/2304 [I 2022-10-27 02:25:23] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:36 [I 2022-10-27 02:25:23] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00244 [I 2022-10-27 02:25:23] ...trap/engines/engine.py.230: loss: 0.16491 [I 2022-10-27 02:25:23] ...optimizers/trijoint.py.151: total_norm: 21.429758 [I 2022-10-27 02:25:29] ...trap/engines/engine.py.224: train: epoch 42 | batch 2140/2304 [I 2022-10-27 02:25:29] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:01:31 [I 2022-10-27 02:25:29] ...trap/engines/engine.py.229: process: 0.57130 | load: 0.00243 [I 2022-10-27 02:25:29] ...trap/engines/engine.py.230: loss: 0.13742 [I 2022-10-27 02:25:29] ...optimizers/trijoint.py.151: total_norm: 26.183868 [I 2022-10-27 02:25:34] ...trap/engines/engine.py.224: train: epoch 42 | batch 2150/2304 [I 2022-10-27 02:25:34] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:01:26 [I 2022-10-27 02:25:34] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00242 [I 2022-10-27 02:25:34] ...trap/engines/engine.py.230: loss: 0.18513 [I 2022-10-27 02:25:34] ...optimizers/trijoint.py.151: total_norm: 18.422928 [I 2022-10-27 02:25:40] ...trap/engines/engine.py.224: train: epoch 42 | batch 2160/2304 [I 2022-10-27 02:25:40] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:01:21 [I 2022-10-27 02:25:40] ...trap/engines/engine.py.229: process: 0.56611 | load: 0.00245 [I 2022-10-27 02:25:40] ...trap/engines/engine.py.230: loss: 0.11323 [I 2022-10-27 02:25:40] ...optimizers/trijoint.py.151: total_norm: 17.472731 [I 2022-10-27 02:25:46] ...trap/engines/engine.py.224: train: epoch 42 | batch 2170/2304 [I 2022-10-27 02:25:46] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:01:15 [I 2022-10-27 02:25:46] ...trap/engines/engine.py.229: process: 0.56533 | load: 0.00261 [I 2022-10-27 02:25:46] ...trap/engines/engine.py.230: loss: 0.17710 [I 2022-10-27 02:25:46] ...optimizers/trijoint.py.151: total_norm: 13.646282 [I 2022-10-27 02:25:51] ...trap/engines/engine.py.224: train: epoch 42 | batch 2180/2304 [I 2022-10-27 02:25:51] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:01:09 [I 2022-10-27 02:25:51] ...trap/engines/engine.py.229: process: 0.55352 | load: 0.00254 [I 2022-10-27 02:25:51] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-27 02:25:51] ...optimizers/trijoint.py.151: total_norm: 10.028369 [I 2022-10-27 02:26:14] ...trap/engines/engine.py.224: train: epoch 42 | batch 2190/2304 [I 2022-10-27 02:26:14] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:01:54 [I 2022-10-27 02:26:14] ...trap/engines/engine.py.229: process: 0.56629 | load: 0.00240 [I 2022-10-27 02:26:14] ...trap/engines/engine.py.230: loss: 0.20109 [I 2022-10-27 02:26:14] ...optimizers/trijoint.py.151: total_norm: 24.262707 [I 2022-10-27 02:26:19] ...trap/engines/engine.py.224: train: epoch 42 | batch 2200/2304 [I 2022-10-27 02:26:19] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:01:02 [I 2022-10-27 02:26:19] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00239 [I 2022-10-27 02:26:19] ...trap/engines/engine.py.230: loss: 0.13680 [I 2022-10-27 02:26:19] ...optimizers/trijoint.py.151: total_norm: 25.066990 [I 2022-10-27 02:26:25] ...trap/engines/engine.py.224: train: epoch 42 | batch 2210/2304 [I 2022-10-27 02:26:25] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:53 [I 2022-10-27 02:26:25] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00246 [I 2022-10-27 02:26:25] ...trap/engines/engine.py.230: loss: 0.29751 [I 2022-10-27 02:26:25] ...optimizers/trijoint.py.151: total_norm: 30.333305 [I 2022-10-27 02:26:30] ...trap/engines/engine.py.224: train: epoch 42 | batch 2220/2304 [I 2022-10-27 02:26:30] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:46 [I 2022-10-27 02:26:30] ...trap/engines/engine.py.229: process: 0.56224 | load: 0.00280 [I 2022-10-27 02:26:30] ...trap/engines/engine.py.230: loss: 0.13068 [I 2022-10-27 02:26:30] ...optimizers/trijoint.py.151: total_norm: 20.845758 [I 2022-10-27 02:26:36] ...trap/engines/engine.py.224: train: epoch 42 | batch 2230/2304 [I 2022-10-27 02:26:36] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:00:41 [I 2022-10-27 02:26:36] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00242 [I 2022-10-27 02:26:36] ...trap/engines/engine.py.230: loss: 0.18175 [I 2022-10-27 02:26:36] ...optimizers/trijoint.py.151: total_norm: 42.302212 [I 2022-10-27 02:26:42] ...trap/engines/engine.py.224: train: epoch 42 | batch 2240/2304 [I 2022-10-27 02:26:42] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:35 [I 2022-10-27 02:26:42] ...trap/engines/engine.py.229: process: 0.54980 | load: 0.00248 [I 2022-10-27 02:26:42] ...trap/engines/engine.py.230: loss: 0.14617 [I 2022-10-27 02:26:42] ...optimizers/trijoint.py.151: total_norm: 32.750999 [I 2022-10-27 02:26:47] ...trap/engines/engine.py.224: train: epoch 42 | batch 2250/2304 [I 2022-10-27 02:26:47] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:00:29 [I 2022-10-27 02:26:47] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00244 [I 2022-10-27 02:26:47] ...trap/engines/engine.py.230: loss: 0.19312 [I 2022-10-27 02:26:47] ...optimizers/trijoint.py.151: total_norm: 16.255690 [I 2022-10-27 02:26:53] ...trap/engines/engine.py.224: train: epoch 42 | batch 2260/2304 [I 2022-10-27 02:26:53] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:00:24 [I 2022-10-27 02:26:53] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00264 [I 2022-10-27 02:26:53] ...trap/engines/engine.py.230: loss: 0.14522 [I 2022-10-27 02:26:53] ...optimizers/trijoint.py.151: total_norm: 17.704416 [I 2022-10-27 02:26:58] ...trap/engines/engine.py.224: train: epoch 42 | batch 2270/2304 [I 2022-10-27 02:26:58] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:00:18 [I 2022-10-27 02:26:58] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00241 [I 2022-10-27 02:26:58] ...trap/engines/engine.py.230: loss: 0.18185 [I 2022-10-27 02:26:58] ...optimizers/trijoint.py.151: total_norm: 12.810517 [I 2022-10-27 02:27:04] ...trap/engines/engine.py.224: train: epoch 42 | batch 2280/2304 [I 2022-10-27 02:27:04] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:00:13 [I 2022-10-27 02:27:04] ...trap/engines/engine.py.229: process: 0.55115 | load: 0.00242 [I 2022-10-27 02:27:04] ...trap/engines/engine.py.230: loss: 0.17931 [I 2022-10-27 02:27:04] ...optimizers/trijoint.py.151: total_norm: 30.516773 [I 2022-10-27 02:27:10] ...trap/engines/engine.py.224: train: epoch 42 | batch 2290/2304 [I 2022-10-27 02:27:10] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:00:07 [I 2022-10-27 02:27:10] ...trap/engines/engine.py.229: process: 0.56558 | load: 0.00240 [I 2022-10-27 02:27:10] ...trap/engines/engine.py.230: loss: 0.23090 [I 2022-10-27 02:27:10] ...optimizers/trijoint.py.151: total_norm: 17.158825 [I 2022-10-27 02:27:15] ...trap/engines/engine.py.224: train: epoch 42 | batch 2300/2304 [I 2022-10-27 02:27:15] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:00:02 [I 2022-10-27 02:27:15] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00016 [I 2022-10-27 02:27:15] ...trap/engines/engine.py.230: loss: 0.12577 [I 2022-10-27 02:27:15] ...optimizers/trijoint.py.151: total_norm: 15.464439 [I 2022-10-27 02:27:17] ...trap/engines/engine.py.224: train: epoch 42 | batch 2304/2304 [I 2022-10-27 02:27:17] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:00:00 [I 2022-10-27 02:27:17] ...trap/engines/engine.py.229: process: 0.55140 | load: 0.00017 [I 2022-10-27 02:27:17] ...trap/engines/engine.py.230: loss: 0.20573 [I 2022-10-27 02:27:17] ...optimizers/trijoint.py.151: total_norm: 21.277010 [S 2022-10-27 02:27:19] ...trap/engines/engine.py.129: train_epoch.epoch: 42 [S 2022-10-27 02:27:19] ...trap/engines/engine.py.129: train_epoch.loss: 0.1696646894944464 [I 2022-10-27 02:27:19] ...s/criterions/triplet.py.84: margin = 0.2650000000000002 [I 2022-10-27 02:27:19] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 42 [I 2022-10-27 02:27:19] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 02:27:21] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 02:27:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 02:27:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 02:27:23] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 02:27:23] ...trap/engines/engine.py.321: eval: epoch 42 | batch 0/510 [I 2022-10-27 02:27:23] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:30:19 [I 2022-10-27 02:27:23] ...trap/engines/engine.py.326: process: 3.56807 | load: 3.20982 [I 2022-10-27 02:27:26] ...trap/engines/engine.py.321: eval: epoch 42 | batch 10/510 [I 2022-10-27 02:27:26] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:40 [I 2022-10-27 02:27:26] ...trap/engines/engine.py.326: process: 0.33047 | load: 0.00020 [I 2022-10-27 02:27:30] ...trap/engines/engine.py.321: eval: epoch 42 | batch 20/510 [I 2022-10-27 02:27:30] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:05 [I 2022-10-27 02:27:30] ...trap/engines/engine.py.326: process: 0.32362 | load: 0.00041 [I 2022-10-27 02:27:33] ...trap/engines/engine.py.321: eval: epoch 42 | batch 30/510 [I 2022-10-27 02:27:33] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:47 [I 2022-10-27 02:27:33] ...trap/engines/engine.py.326: process: 0.35579 | load: 0.00020 [I 2022-10-27 02:27:37] ...trap/engines/engine.py.321: eval: epoch 42 | batch 40/510 [I 2022-10-27 02:27:37] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:38 [I 2022-10-27 02:27:37] ...trap/engines/engine.py.326: process: 0.34140 | load: 0.00024 [I 2022-10-27 02:27:40] ...trap/engines/engine.py.321: eval: epoch 42 | batch 50/510 [I 2022-10-27 02:27:40] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:36 [I 2022-10-27 02:27:40] ...trap/engines/engine.py.326: process: 0.34662 | load: 0.00017 [I 2022-10-27 02:27:43] ...trap/engines/engine.py.321: eval: epoch 42 | batch 60/510 [I 2022-10-27 02:27:43] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:33 [I 2022-10-27 02:27:43] ...trap/engines/engine.py.326: process: 0.34993 | load: 0.00017 [I 2022-10-27 02:27:47] ...trap/engines/engine.py.321: eval: epoch 42 | batch 70/510 [I 2022-10-27 02:27:47] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:29 [I 2022-10-27 02:27:47] ...trap/engines/engine.py.326: process: 0.32957 | load: 0.00021 [I 2022-10-27 02:27:50] ...trap/engines/engine.py.321: eval: epoch 42 | batch 80/510 [I 2022-10-27 02:27:50] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-27 02:27:50] ...trap/engines/engine.py.326: process: 0.33447 | load: 0.00021 [I 2022-10-27 02:27:54] ...trap/engines/engine.py.321: eval: epoch 42 | batch 90/510 [I 2022-10-27 02:27:54] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:20 [I 2022-10-27 02:27:54] ...trap/engines/engine.py.326: process: 0.33191 | load: 0.00019 [I 2022-10-27 02:27:57] ...trap/engines/engine.py.321: eval: epoch 42 | batch 100/510 [I 2022-10-27 02:27:57] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:14 [I 2022-10-27 02:27:57] ...trap/engines/engine.py.326: process: 0.30323 | load: 0.00025 [I 2022-10-27 02:28:00] ...trap/engines/engine.py.321: eval: epoch 42 | batch 110/510 [I 2022-10-27 02:28:00] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:57 [I 2022-10-27 02:28:00] ...trap/engines/engine.py.326: process: 0.28382 | load: 0.00015 [I 2022-10-27 02:28:03] ...trap/engines/engine.py.321: eval: epoch 42 | batch 120/510 [I 2022-10-27 02:28:03] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:54 [I 2022-10-27 02:28:03] ...trap/engines/engine.py.326: process: 0.29023 | load: 0.00031 [I 2022-10-27 02:28:06] ...trap/engines/engine.py.321: eval: epoch 42 | batch 130/510 [I 2022-10-27 02:28:06] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:51 [I 2022-10-27 02:28:06] ...trap/engines/engine.py.326: process: 0.30028 | load: 0.00015 [I 2022-10-27 02:28:09] ...trap/engines/engine.py.321: eval: epoch 42 | batch 140/510 [I 2022-10-27 02:28:09] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:51 [I 2022-10-27 02:28:09] ...trap/engines/engine.py.326: process: 0.28867 | load: 0.00014 [I 2022-10-27 02:28:11] ...trap/engines/engine.py.321: eval: epoch 42 | batch 150/510 [I 2022-10-27 02:28:11] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 02:28:11] ...trap/engines/engine.py.326: process: 0.28886 | load: 0.00015 [I 2022-10-27 02:28:14] ...trap/engines/engine.py.321: eval: epoch 42 | batch 160/510 [I 2022-10-27 02:28:14] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-27 02:28:14] ...trap/engines/engine.py.326: process: 0.29213 | load: 0.00020 [I 2022-10-27 02:28:17] ...trap/engines/engine.py.321: eval: epoch 42 | batch 170/510 [I 2022-10-27 02:28:17] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-27 02:28:17] ...trap/engines/engine.py.326: process: 0.29071 | load: 0.00019 [I 2022-10-27 02:28:20] ...trap/engines/engine.py.321: eval: epoch 42 | batch 180/510 [I 2022-10-27 02:28:20] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 02:28:20] ...trap/engines/engine.py.326: process: 0.28627 | load: 0.00019 [I 2022-10-27 02:28:23] ...trap/engines/engine.py.321: eval: epoch 42 | batch 190/510 [I 2022-10-27 02:28:23] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 02:28:23] ...trap/engines/engine.py.326: process: 0.28470 | load: 0.00016 [I 2022-10-27 02:28:26] ...trap/engines/engine.py.321: eval: epoch 42 | batch 200/510 [I 2022-10-27 02:28:26] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 02:28:26] ...trap/engines/engine.py.326: process: 0.28768 | load: 0.00020 [I 2022-10-27 02:28:29] ...trap/engines/engine.py.321: eval: epoch 42 | batch 210/510 [I 2022-10-27 02:28:29] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 02:28:29] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00025 [I 2022-10-27 02:28:32] ...trap/engines/engine.py.321: eval: epoch 42 | batch 220/510 [I 2022-10-27 02:28:32] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 02:28:32] ...trap/engines/engine.py.326: process: 0.28886 | load: 0.00016 [I 2022-10-27 02:28:35] ...trap/engines/engine.py.321: eval: epoch 42 | batch 230/510 [I 2022-10-27 02:28:35] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 02:28:35] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00022 [I 2022-10-27 02:28:38] ...trap/engines/engine.py.321: eval: epoch 42 | batch 240/510 [I 2022-10-27 02:28:38] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:19 [I 2022-10-27 02:28:38] ...trap/engines/engine.py.326: process: 0.29079 | load: 0.00022 [I 2022-10-27 02:28:40] ...trap/engines/engine.py.321: eval: epoch 42 | batch 250/510 [I 2022-10-27 02:28:40] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 02:28:40] ...trap/engines/engine.py.326: process: 0.29161 | load: 0.00016 [I 2022-10-27 02:28:43] ...trap/engines/engine.py.321: eval: epoch 42 | batch 260/510 [I 2022-10-27 02:28:43] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 02:28:43] ...trap/engines/engine.py.326: process: 0.28424 | load: 0.00020 [I 2022-10-27 02:28:46] ...trap/engines/engine.py.321: eval: epoch 42 | batch 270/510 [I 2022-10-27 02:28:46] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 02:28:46] ...trap/engines/engine.py.326: process: 0.28765 | load: 0.00016 [I 2022-10-27 02:28:49] ...trap/engines/engine.py.321: eval: epoch 42 | batch 280/510 [I 2022-10-27 02:28:49] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 02:28:49] ...trap/engines/engine.py.326: process: 0.28849 | load: 0.00016 [I 2022-10-27 02:28:52] ...trap/engines/engine.py.321: eval: epoch 42 | batch 290/510 [I 2022-10-27 02:28:52] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 02:28:52] ...trap/engines/engine.py.326: process: 0.28879 | load: 0.00020 [I 2022-10-27 02:28:55] ...trap/engines/engine.py.321: eval: epoch 42 | batch 300/510 [I 2022-10-27 02:28:55] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 02:28:55] ...trap/engines/engine.py.326: process: 0.28845 | load: 0.00021 [I 2022-10-27 02:28:58] ...trap/engines/engine.py.321: eval: epoch 42 | batch 310/510 [I 2022-10-27 02:28:58] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 02:28:58] ...trap/engines/engine.py.326: process: 0.28420 | load: 0.00015 [I 2022-10-27 02:29:01] ...trap/engines/engine.py.321: eval: epoch 42 | batch 320/510 [I 2022-10-27 02:29:01] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 02:29:01] ...trap/engines/engine.py.326: process: 0.29303 | load: 0.00016 [I 2022-10-27 02:29:04] ...trap/engines/engine.py.321: eval: epoch 42 | batch 330/510 [I 2022-10-27 02:29:04] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 02:29:04] ...trap/engines/engine.py.326: process: 0.28501 | load: 0.00015 [I 2022-10-27 02:29:06] ...trap/engines/engine.py.321: eval: epoch 42 | batch 340/510 [I 2022-10-27 02:29:06] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 02:29:06] ...trap/engines/engine.py.326: process: 0.30658 | load: 0.00015 [I 2022-10-27 02:29:09] ...trap/engines/engine.py.321: eval: epoch 42 | batch 350/510 [I 2022-10-27 02:29:09] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-27 02:29:09] ...trap/engines/engine.py.326: process: 0.28807 | load: 0.00015 [I 2022-10-27 02:29:12] ...trap/engines/engine.py.321: eval: epoch 42 | batch 360/510 [I 2022-10-27 02:29:12] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 02:29:12] ...trap/engines/engine.py.326: process: 0.29721 | load: 0.00021 [I 2022-10-27 02:29:15] ...trap/engines/engine.py.321: eval: epoch 42 | batch 370/510 [I 2022-10-27 02:29:15] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 02:29:15] ...trap/engines/engine.py.326: process: 0.29324 | load: 0.00021 [I 2022-10-27 02:29:18] ...trap/engines/engine.py.321: eval: epoch 42 | batch 380/510 [I 2022-10-27 02:29:18] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 02:29:18] ...trap/engines/engine.py.326: process: 0.28462 | load: 0.00020 [I 2022-10-27 02:29:21] ...trap/engines/engine.py.321: eval: epoch 42 | batch 390/510 [I 2022-10-27 02:29:21] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 02:29:21] ...trap/engines/engine.py.326: process: 0.28112 | load: 0.00015 [I 2022-10-27 02:29:24] ...trap/engines/engine.py.321: eval: epoch 42 | batch 400/510 [I 2022-10-27 02:29:24] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 02:29:24] ...trap/engines/engine.py.326: process: 0.28357 | load: 0.00017 [I 2022-10-27 02:29:27] ...trap/engines/engine.py.321: eval: epoch 42 | batch 410/510 [I 2022-10-27 02:29:27] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 02:29:27] ...trap/engines/engine.py.326: process: 0.28963 | load: 0.00020 [I 2022-10-27 02:29:30] ...trap/engines/engine.py.321: eval: epoch 42 | batch 420/510 [I 2022-10-27 02:29:30] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 02:29:30] ...trap/engines/engine.py.326: process: 0.28240 | load: 0.00020 [I 2022-10-27 02:29:32] ...trap/engines/engine.py.321: eval: epoch 42 | batch 430/510 [I 2022-10-27 02:29:32] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:22 [I 2022-10-27 02:29:32] ...trap/engines/engine.py.326: process: 0.28601 | load: 0.00018 [I 2022-10-27 02:29:35] ...trap/engines/engine.py.321: eval: epoch 42 | batch 440/510 [I 2022-10-27 02:29:35] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 02:29:35] ...trap/engines/engine.py.326: process: 0.29258 | load: 0.00016 [I 2022-10-27 02:29:38] ...trap/engines/engine.py.321: eval: epoch 42 | batch 450/510 [I 2022-10-27 02:29:38] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 02:29:38] ...trap/engines/engine.py.326: process: 0.29032 | load: 0.00017 [I 2022-10-27 02:29:41] ...trap/engines/engine.py.321: eval: epoch 42 | batch 460/510 [I 2022-10-27 02:29:41] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 02:29:41] ...trap/engines/engine.py.326: process: 0.30817 | load: 0.00017 [I 2022-10-27 02:29:44] ...trap/engines/engine.py.321: eval: epoch 42 | batch 470/510 [I 2022-10-27 02:29:44] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 02:29:44] ...trap/engines/engine.py.326: process: 0.28990 | load: 0.00022 [I 2022-10-27 02:29:47] ...trap/engines/engine.py.321: eval: epoch 42 | batch 480/510 [I 2022-10-27 02:29:47] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 02:29:47] ...trap/engines/engine.py.326: process: 0.28370 | load: 0.00021 [I 2022-10-27 02:29:50] ...trap/engines/engine.py.321: eval: epoch 42 | batch 490/510 [I 2022-10-27 02:29:50] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 02:29:50] ...trap/engines/engine.py.326: process: 0.28870 | load: 0.00017 [I 2022-10-27 02:29:53] ...trap/engines/engine.py.321: eval: epoch 42 | batch 500/510 [I 2022-10-27 02:29:53] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 02:29:53] ...trap/engines/engine.py.326: process: 0.28976 | load: 0.00020 [I 2022-10-27 02:29:56] ...trap/engines/engine.py.321: eval: epoch 42 | batch 510/510 [I 2022-10-27 02:29:56] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 02:29:56] ...trap/engines/engine.py.326: process: 0.29817 | load: 0.00011 [S 2022-10-27 02:29:56] ...trap/engines/engine.py.132: eval_epoch.epoch: 42 [I 2022-10-27 02:29:56] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.690900 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.896500 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.932500 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.680800 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.891700 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.928300 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015534 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006859 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004588 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012758 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006451 [S 2022-10-27 02:30:00] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006165 [I 2022-10-27 02:30:02] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 02:30:02] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 02:30:02] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 02:30:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 02:30:02] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 02:30:03] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 02:30:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 02:30:12] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 02:30:12] ...trap/engines/engine.py.163: Training model on trainset for epoch 43 [I 2022-10-27 02:30:12] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 02:30:14] ...trap/engines/engine.py.224: train: epoch 43 | batch 0/2304 [I 2022-10-27 02:30:14] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:40:20 [I 2022-10-27 02:30:14] ...trap/engines/engine.py.229: process: 2.61327 | load: 1.96825 [I 2022-10-27 02:30:14] ...trap/engines/engine.py.230: loss: 0.14172 [I 2022-10-27 02:30:14] ...optimizers/trijoint.py.151: total_norm: 12.408951 [I 2022-10-27 02:30:20] ...trap/engines/engine.py.224: train: epoch 43 | batch 10/2304 [I 2022-10-27 02:30:20] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:48 [I 2022-10-27 02:30:20] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00516 [I 2022-10-27 02:30:20] ...trap/engines/engine.py.230: loss: 0.12230 [I 2022-10-27 02:30:20] ...optimizers/trijoint.py.151: total_norm: 10.727498 [I 2022-10-27 02:30:25] ...trap/engines/engine.py.224: train: epoch 43 | batch 20/2304 [I 2022-10-27 02:30:25] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:14 [I 2022-10-27 02:30:25] ...trap/engines/engine.py.229: process: 0.56426 | load: 0.00364 [I 2022-10-27 02:30:25] ...trap/engines/engine.py.230: loss: 0.25334 [I 2022-10-27 02:30:25] ...optimizers/trijoint.py.151: total_norm: 29.505589 [I 2022-10-27 02:30:31] ...trap/engines/engine.py.224: train: epoch 43 | batch 30/2304 [I 2022-10-27 02:30:31] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:13 [I 2022-10-27 02:30:31] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00336 [I 2022-10-27 02:30:31] ...trap/engines/engine.py.230: loss: 0.19150 [I 2022-10-27 02:30:31] ...optimizers/trijoint.py.151: total_norm: 32.789085 [I 2022-10-27 02:30:37] ...trap/engines/engine.py.224: train: epoch 43 | batch 40/2304 [I 2022-10-27 02:30:37] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:21:06 [I 2022-10-27 02:30:37] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00340 [I 2022-10-27 02:30:37] ...trap/engines/engine.py.230: loss: 0.16820 [I 2022-10-27 02:30:37] ...optimizers/trijoint.py.151: total_norm: 15.001837 [I 2022-10-27 02:30:42] ...trap/engines/engine.py.224: train: epoch 43 | batch 50/2304 [I 2022-10-27 02:30:42] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:50 [I 2022-10-27 02:30:42] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00314 [I 2022-10-27 02:30:42] ...trap/engines/engine.py.230: loss: 0.10694 [I 2022-10-27 02:30:42] ...optimizers/trijoint.py.151: total_norm: 10.987430 [I 2022-10-27 02:30:48] ...trap/engines/engine.py.224: train: epoch 43 | batch 60/2304 [I 2022-10-27 02:30:48] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:55 [I 2022-10-27 02:30:48] ...trap/engines/engine.py.229: process: 0.55231 | load: 0.00335 [I 2022-10-27 02:30:48] ...trap/engines/engine.py.230: loss: 0.13263 [I 2022-10-27 02:30:48] ...optimizers/trijoint.py.151: total_norm: 9.682769 [I 2022-10-27 02:30:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 70/2304 [I 2022-10-27 02:30:54] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:21:08 [I 2022-10-27 02:30:54] ...trap/engines/engine.py.229: process: 0.56351 | load: 0.00281 [I 2022-10-27 02:30:54] ...trap/engines/engine.py.230: loss: 0.18992 [I 2022-10-27 02:30:54] ...optimizers/trijoint.py.151: total_norm: 26.321306 [I 2022-10-27 02:30:59] ...trap/engines/engine.py.224: train: epoch 43 | batch 80/2304 [I 2022-10-27 02:30:59] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:57 [I 2022-10-27 02:30:59] ...trap/engines/engine.py.229: process: 0.57201 | load: 0.00271 [I 2022-10-27 02:30:59] ...trap/engines/engine.py.230: loss: 0.20788 [I 2022-10-27 02:30:59] ...optimizers/trijoint.py.151: total_norm: 20.319773 [I 2022-10-27 02:31:05] ...trap/engines/engine.py.224: train: epoch 43 | batch 90/2304 [I 2022-10-27 02:31:05] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:40 [I 2022-10-27 02:31:05] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00266 [I 2022-10-27 02:31:05] ...trap/engines/engine.py.230: loss: 0.14318 [I 2022-10-27 02:31:05] ...optimizers/trijoint.py.151: total_norm: 16.801699 [I 2022-10-27 02:31:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 100/2304 [I 2022-10-27 02:31:10] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:27 [I 2022-10-27 02:31:10] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00262 [I 2022-10-27 02:31:10] ...trap/engines/engine.py.230: loss: 0.15744 [I 2022-10-27 02:31:10] ...optimizers/trijoint.py.151: total_norm: 14.205797 [I 2022-10-27 02:31:16] ...trap/engines/engine.py.224: train: epoch 43 | batch 110/2304 [I 2022-10-27 02:31:16] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:20 [I 2022-10-27 02:31:16] ...trap/engines/engine.py.229: process: 0.54650 | load: 0.00288 [I 2022-10-27 02:31:16] ...trap/engines/engine.py.230: loss: 0.12247 [I 2022-10-27 02:31:16] ...optimizers/trijoint.py.151: total_norm: 10.309443 [I 2022-10-27 02:31:22] ...trap/engines/engine.py.224: train: epoch 43 | batch 120/2304 [I 2022-10-27 02:31:22] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:26 [I 2022-10-27 02:31:22] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00263 [I 2022-10-27 02:31:22] ...trap/engines/engine.py.230: loss: 0.16136 [I 2022-10-27 02:31:22] ...optimizers/trijoint.py.151: total_norm: 14.569499 [I 2022-10-27 02:31:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 130/2304 [I 2022-10-27 02:31:27] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:09 [I 2022-10-27 02:31:27] ...trap/engines/engine.py.229: process: 0.54716 | load: 0.00252 [I 2022-10-27 02:31:27] ...trap/engines/engine.py.230: loss: 0.12185 [I 2022-10-27 02:31:27] ...optimizers/trijoint.py.151: total_norm: 19.169432 [I 2022-10-27 02:31:33] ...trap/engines/engine.py.224: train: epoch 43 | batch 140/2304 [I 2022-10-27 02:31:33] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:04 [I 2022-10-27 02:31:33] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00273 [I 2022-10-27 02:31:33] ...trap/engines/engine.py.230: loss: 0.14289 [I 2022-10-27 02:31:33] ...optimizers/trijoint.py.151: total_norm: 20.605907 [I 2022-10-27 02:31:38] ...trap/engines/engine.py.224: train: epoch 43 | batch 150/2304 [I 2022-10-27 02:31:38] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:59 [I 2022-10-27 02:31:38] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00253 [I 2022-10-27 02:31:38] ...trap/engines/engine.py.230: loss: 0.17377 [I 2022-10-27 02:31:38] ...optimizers/trijoint.py.151: total_norm: 13.043013 [I 2022-10-27 02:31:44] ...trap/engines/engine.py.224: train: epoch 43 | batch 160/2304 [I 2022-10-27 02:31:44] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:50 [I 2022-10-27 02:31:44] ...trap/engines/engine.py.229: process: 0.54385 | load: 0.00269 [I 2022-10-27 02:31:44] ...trap/engines/engine.py.230: loss: 0.15596 [I 2022-10-27 02:31:44] ...optimizers/trijoint.py.151: total_norm: 18.692423 [I 2022-10-27 02:31:49] ...trap/engines/engine.py.224: train: epoch 43 | batch 170/2304 [I 2022-10-27 02:31:49] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:41 [I 2022-10-27 02:31:49] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00271 [I 2022-10-27 02:31:49] ...trap/engines/engine.py.230: loss: 0.17722 [I 2022-10-27 02:31:49] ...optimizers/trijoint.py.151: total_norm: 21.924347 [I 2022-10-27 02:31:55] ...trap/engines/engine.py.224: train: epoch 43 | batch 180/2304 [I 2022-10-27 02:31:55] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:43 [I 2022-10-27 02:31:55] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00254 [I 2022-10-27 02:31:55] ...trap/engines/engine.py.230: loss: 0.13910 [I 2022-10-27 02:31:55] ...optimizers/trijoint.py.151: total_norm: 18.291222 [I 2022-10-27 02:32:01] ...trap/engines/engine.py.224: train: epoch 43 | batch 190/2304 [I 2022-10-27 02:32:01] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:32 [I 2022-10-27 02:32:01] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00276 [I 2022-10-27 02:32:01] ...trap/engines/engine.py.230: loss: 0.11404 [I 2022-10-27 02:32:01] ...optimizers/trijoint.py.151: total_norm: 9.765124 [I 2022-10-27 02:32:06] ...trap/engines/engine.py.224: train: epoch 43 | batch 200/2304 [I 2022-10-27 02:32:06] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:34 [I 2022-10-27 02:32:06] ...trap/engines/engine.py.229: process: 0.55341 | load: 0.00253 [I 2022-10-27 02:32:06] ...trap/engines/engine.py.230: loss: 0.19502 [I 2022-10-27 02:32:06] ...optimizers/trijoint.py.151: total_norm: 16.226316 [I 2022-10-27 02:32:12] ...trap/engines/engine.py.224: train: epoch 43 | batch 210/2304 [I 2022-10-27 02:32:12] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:22 [I 2022-10-27 02:32:12] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00270 [I 2022-10-27 02:32:12] ...trap/engines/engine.py.230: loss: 0.13000 [I 2022-10-27 02:32:12] ...optimizers/trijoint.py.151: total_norm: 8.694268 [I 2022-10-27 02:32:17] ...trap/engines/engine.py.224: train: epoch 43 | batch 220/2304 [I 2022-10-27 02:32:17] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:12 [I 2022-10-27 02:32:17] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00249 [I 2022-10-27 02:32:17] ...trap/engines/engine.py.230: loss: 0.17414 [I 2022-10-27 02:32:17] ...optimizers/trijoint.py.151: total_norm: 12.436926 [I 2022-10-27 02:32:23] ...trap/engines/engine.py.224: train: epoch 43 | batch 230/2304 [I 2022-10-27 02:32:23] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:10 [I 2022-10-27 02:32:23] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00244 [I 2022-10-27 02:32:23] ...trap/engines/engine.py.230: loss: 0.15040 [I 2022-10-27 02:32:23] ...optimizers/trijoint.py.151: total_norm: 15.940412 [I 2022-10-27 02:32:28] ...trap/engines/engine.py.224: train: epoch 43 | batch 240/2304 [I 2022-10-27 02:32:28] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:03 [I 2022-10-27 02:32:28] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00253 [I 2022-10-27 02:32:28] ...trap/engines/engine.py.230: loss: 0.12196 [I 2022-10-27 02:32:28] ...optimizers/trijoint.py.151: total_norm: 14.766629 [I 2022-10-27 02:32:34] ...trap/engines/engine.py.224: train: epoch 43 | batch 250/2304 [I 2022-10-27 02:32:34] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:02 [I 2022-10-27 02:32:34] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00256 [I 2022-10-27 02:32:34] ...trap/engines/engine.py.230: loss: 0.23476 [I 2022-10-27 02:32:34] ...optimizers/trijoint.py.151: total_norm: 10.976855 [I 2022-10-27 02:32:40] ...trap/engines/engine.py.224: train: epoch 43 | batch 260/2304 [I 2022-10-27 02:32:40] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:58 [I 2022-10-27 02:32:40] ...trap/engines/engine.py.229: process: 0.56638 | load: 0.00244 [I 2022-10-27 02:32:40] ...trap/engines/engine.py.230: loss: 0.18557 [I 2022-10-27 02:32:40] ...optimizers/trijoint.py.151: total_norm: 14.115457 [I 2022-10-27 02:32:45] ...trap/engines/engine.py.224: train: epoch 43 | batch 270/2304 [I 2022-10-27 02:32:45] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:49 [I 2022-10-27 02:32:45] ...trap/engines/engine.py.229: process: 0.55350 | load: 0.00245 [I 2022-10-27 02:32:45] ...trap/engines/engine.py.230: loss: 0.13789 [I 2022-10-27 02:32:45] ...optimizers/trijoint.py.151: total_norm: 15.856751 [I 2022-10-27 02:32:51] ...trap/engines/engine.py.224: train: epoch 43 | batch 280/2304 [I 2022-10-27 02:32:51] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:54 [I 2022-10-27 02:32:51] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00251 [I 2022-10-27 02:32:51] ...trap/engines/engine.py.230: loss: 0.23557 [I 2022-10-27 02:32:51] ...optimizers/trijoint.py.151: total_norm: 18.626934 [I 2022-10-27 02:32:56] ...trap/engines/engine.py.224: train: epoch 43 | batch 290/2304 [I 2022-10-27 02:32:56] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:47 [I 2022-10-27 02:32:56] ...trap/engines/engine.py.229: process: 0.55499 | load: 0.00243 [I 2022-10-27 02:32:56] ...trap/engines/engine.py.230: loss: 0.16416 [I 2022-10-27 02:32:56] ...optimizers/trijoint.py.151: total_norm: 16.534998 [I 2022-10-27 02:33:02] ...trap/engines/engine.py.224: train: epoch 43 | batch 300/2304 [I 2022-10-27 02:33:02] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:38 [I 2022-10-27 02:33:02] ...trap/engines/engine.py.229: process: 0.56553 | load: 0.00243 [I 2022-10-27 02:33:02] ...trap/engines/engine.py.230: loss: 0.15649 [I 2022-10-27 02:33:02] ...optimizers/trijoint.py.151: total_norm: 22.322697 [I 2022-10-27 02:33:08] ...trap/engines/engine.py.224: train: epoch 43 | batch 310/2304 [I 2022-10-27 02:33:08] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:27 [I 2022-10-27 02:33:08] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00254 [I 2022-10-27 02:33:08] ...trap/engines/engine.py.230: loss: 0.14272 [I 2022-10-27 02:33:08] ...optimizers/trijoint.py.151: total_norm: 16.201830 [I 2022-10-27 02:33:13] ...trap/engines/engine.py.224: train: epoch 43 | batch 320/2304 [I 2022-10-27 02:33:13] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:21 [I 2022-10-27 02:33:13] ...trap/engines/engine.py.229: process: 0.54597 | load: 0.00249 [I 2022-10-27 02:33:13] ...trap/engines/engine.py.230: loss: 0.16727 [I 2022-10-27 02:33:13] ...optimizers/trijoint.py.151: total_norm: 9.102259 [I 2022-10-27 02:33:19] ...trap/engines/engine.py.224: train: epoch 43 | batch 330/2304 [I 2022-10-27 02:33:19] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:12 [I 2022-10-27 02:33:19] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00252 [I 2022-10-27 02:33:19] ...trap/engines/engine.py.230: loss: 0.18852 [I 2022-10-27 02:33:19] ...optimizers/trijoint.py.151: total_norm: 20.377964 [I 2022-10-27 02:33:24] ...trap/engines/engine.py.224: train: epoch 43 | batch 340/2304 [I 2022-10-27 02:33:24] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:11 [I 2022-10-27 02:33:24] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00244 [I 2022-10-27 02:33:24] ...trap/engines/engine.py.230: loss: 0.14650 [I 2022-10-27 02:33:24] ...optimizers/trijoint.py.151: total_norm: 27.612270 [I 2022-10-27 02:33:30] ...trap/engines/engine.py.224: train: epoch 43 | batch 350/2304 [I 2022-10-27 02:33:30] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:11 [I 2022-10-27 02:33:30] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00243 [I 2022-10-27 02:33:30] ...trap/engines/engine.py.230: loss: 0.16882 [I 2022-10-27 02:33:30] ...optimizers/trijoint.py.151: total_norm: 20.846617 [I 2022-10-27 02:33:35] ...trap/engines/engine.py.224: train: epoch 43 | batch 360/2304 [I 2022-10-27 02:33:35] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:58 [I 2022-10-27 02:33:35] ...trap/engines/engine.py.229: process: 0.55316 | load: 0.00245 [I 2022-10-27 02:33:35] ...trap/engines/engine.py.230: loss: 0.20394 [I 2022-10-27 02:33:35] ...optimizers/trijoint.py.151: total_norm: 42.821484 [I 2022-10-27 02:33:41] ...trap/engines/engine.py.224: train: epoch 43 | batch 370/2304 [I 2022-10-27 02:33:41] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:48 [I 2022-10-27 02:33:41] ...trap/engines/engine.py.229: process: 0.54612 | load: 0.00245 [I 2022-10-27 02:33:41] ...trap/engines/engine.py.230: loss: 0.18674 [I 2022-10-27 02:33:41] ...optimizers/trijoint.py.151: total_norm: 20.277828 [I 2022-10-27 02:33:47] ...trap/engines/engine.py.224: train: epoch 43 | batch 380/2304 [I 2022-10-27 02:33:47] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:52 [I 2022-10-27 02:33:47] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00245 [I 2022-10-27 02:33:47] ...trap/engines/engine.py.230: loss: 0.11088 [I 2022-10-27 02:33:47] ...optimizers/trijoint.py.151: total_norm: 13.942186 [I 2022-10-27 02:33:52] ...trap/engines/engine.py.224: train: epoch 43 | batch 390/2304 [I 2022-10-27 02:33:52] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:46 [I 2022-10-27 02:33:52] ...trap/engines/engine.py.229: process: 0.56119 | load: 0.00254 [I 2022-10-27 02:33:52] ...trap/engines/engine.py.230: loss: 0.26104 [I 2022-10-27 02:33:52] ...optimizers/trijoint.py.151: total_norm: 20.900192 [I 2022-10-27 02:33:58] ...trap/engines/engine.py.224: train: epoch 43 | batch 400/2304 [I 2022-10-27 02:33:58] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:31 [I 2022-10-27 02:33:58] ...trap/engines/engine.py.229: process: 0.53541 | load: 0.00243 [I 2022-10-27 02:33:58] ...trap/engines/engine.py.230: loss: 0.11021 [I 2022-10-27 02:33:58] ...optimizers/trijoint.py.151: total_norm: 6.080127 [I 2022-10-27 02:34:03] ...trap/engines/engine.py.224: train: epoch 43 | batch 410/2304 [I 2022-10-27 02:34:03] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:30 [I 2022-10-27 02:34:03] ...trap/engines/engine.py.229: process: 0.55118 | load: 0.00245 [I 2022-10-27 02:34:03] ...trap/engines/engine.py.230: loss: 0.16026 [I 2022-10-27 02:34:03] ...optimizers/trijoint.py.151: total_norm: 29.720867 [I 2022-10-27 02:34:09] ...trap/engines/engine.py.224: train: epoch 43 | batch 420/2304 [I 2022-10-27 02:34:09] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:22 [I 2022-10-27 02:34:09] ...trap/engines/engine.py.229: process: 0.54628 | load: 0.00241 [I 2022-10-27 02:34:09] ...trap/engines/engine.py.230: loss: 0.18827 [I 2022-10-27 02:34:09] ...optimizers/trijoint.py.151: total_norm: 22.582561 [I 2022-10-27 02:34:14] ...trap/engines/engine.py.224: train: epoch 43 | batch 430/2304 [I 2022-10-27 02:34:14] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:17 [I 2022-10-27 02:34:14] ...trap/engines/engine.py.229: process: 0.55147 | load: 0.00252 [I 2022-10-27 02:34:14] ...trap/engines/engine.py.230: loss: 0.17203 [I 2022-10-27 02:34:14] ...optimizers/trijoint.py.151: total_norm: 14.316195 [I 2022-10-27 02:34:20] ...trap/engines/engine.py.224: train: epoch 43 | batch 440/2304 [I 2022-10-27 02:34:20] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:11 [I 2022-10-27 02:34:20] ...trap/engines/engine.py.229: process: 0.54949 | load: 0.00257 [I 2022-10-27 02:34:20] ...trap/engines/engine.py.230: loss: 0.11358 [I 2022-10-27 02:34:20] ...optimizers/trijoint.py.151: total_norm: 14.369102 [I 2022-10-27 02:34:25] ...trap/engines/engine.py.224: train: epoch 43 | batch 450/2304 [I 2022-10-27 02:34:25] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:15 [I 2022-10-27 02:34:25] ...trap/engines/engine.py.229: process: 0.56181 | load: 0.00251 [I 2022-10-27 02:34:25] ...trap/engines/engine.py.230: loss: 0.18689 [I 2022-10-27 02:34:25] ...optimizers/trijoint.py.151: total_norm: 17.973333 [I 2022-10-27 02:34:31] ...trap/engines/engine.py.224: train: epoch 43 | batch 460/2304 [I 2022-10-27 02:34:31] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:06 [I 2022-10-27 02:34:31] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00246 [I 2022-10-27 02:34:31] ...trap/engines/engine.py.230: loss: 0.15208 [I 2022-10-27 02:34:31] ...optimizers/trijoint.py.151: total_norm: 15.116899 [I 2022-10-27 02:34:37] ...trap/engines/engine.py.224: train: epoch 43 | batch 470/2304 [I 2022-10-27 02:34:37] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:07 [I 2022-10-27 02:34:37] ...trap/engines/engine.py.229: process: 0.56835 | load: 0.00243 [I 2022-10-27 02:34:37] ...trap/engines/engine.py.230: loss: 0.19049 [I 2022-10-27 02:34:37] ...optimizers/trijoint.py.151: total_norm: 14.877422 [I 2022-10-27 02:34:42] ...trap/engines/engine.py.224: train: epoch 43 | batch 480/2304 [I 2022-10-27 02:34:42] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:51 [I 2022-10-27 02:34:42] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00253 [I 2022-10-27 02:34:42] ...trap/engines/engine.py.230: loss: 0.22356 [I 2022-10-27 02:34:42] ...optimizers/trijoint.py.151: total_norm: 14.797159 [I 2022-10-27 02:34:48] ...trap/engines/engine.py.224: train: epoch 43 | batch 490/2304 [I 2022-10-27 02:34:48] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:55 [I 2022-10-27 02:34:48] ...trap/engines/engine.py.229: process: 0.57837 | load: 0.00260 [I 2022-10-27 02:34:48] ...trap/engines/engine.py.230: loss: 0.16821 [I 2022-10-27 02:34:48] ...optimizers/trijoint.py.151: total_norm: 19.560646 [I 2022-10-27 02:34:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 500/2304 [I 2022-10-27 02:34:53] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:40 [I 2022-10-27 02:34:53] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00247 [I 2022-10-27 02:34:53] ...trap/engines/engine.py.230: loss: 0.14886 [I 2022-10-27 02:34:53] ...optimizers/trijoint.py.151: total_norm: 13.841654 [I 2022-10-27 02:34:59] ...trap/engines/engine.py.224: train: epoch 43 | batch 510/2304 [I 2022-10-27 02:34:59] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:29 [I 2022-10-27 02:34:59] ...trap/engines/engine.py.229: process: 0.54595 | load: 0.00245 [I 2022-10-27 02:34:59] ...trap/engines/engine.py.230: loss: 0.17121 [I 2022-10-27 02:34:59] ...optimizers/trijoint.py.151: total_norm: 20.239040 [I 2022-10-27 02:35:04] ...trap/engines/engine.py.224: train: epoch 43 | batch 520/2304 [I 2022-10-27 02:35:04] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:16:34 [I 2022-10-27 02:35:04] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00244 [I 2022-10-27 02:35:04] ...trap/engines/engine.py.230: loss: 0.17765 [I 2022-10-27 02:35:04] ...optimizers/trijoint.py.151: total_norm: 15.614037 [I 2022-10-27 02:35:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 530/2304 [I 2022-10-27 02:35:10] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:27 [I 2022-10-27 02:35:10] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00245 [I 2022-10-27 02:35:10] ...trap/engines/engine.py.230: loss: 0.14897 [I 2022-10-27 02:35:10] ...optimizers/trijoint.py.151: total_norm: 12.560698 [I 2022-10-27 02:35:16] ...trap/engines/engine.py.224: train: epoch 43 | batch 540/2304 [I 2022-10-27 02:35:16] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:16:18 [I 2022-10-27 02:35:16] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00275 [I 2022-10-27 02:35:16] ...trap/engines/engine.py.230: loss: 0.15853 [I 2022-10-27 02:35:16] ...optimizers/trijoint.py.151: total_norm: 15.771267 [I 2022-10-27 02:35:21] ...trap/engines/engine.py.224: train: epoch 43 | batch 550/2304 [I 2022-10-27 02:35:21] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:11 [I 2022-10-27 02:35:21] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00242 [I 2022-10-27 02:35:21] ...trap/engines/engine.py.230: loss: 0.21828 [I 2022-10-27 02:35:21] ...optimizers/trijoint.py.151: total_norm: 15.922993 [I 2022-10-27 02:35:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 560/2304 [I 2022-10-27 02:35:27] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:12 [I 2022-10-27 02:35:27] ...trap/engines/engine.py.229: process: 0.56801 | load: 0.00240 [I 2022-10-27 02:35:27] ...trap/engines/engine.py.230: loss: 0.14841 [I 2022-10-27 02:35:27] ...optimizers/trijoint.py.151: total_norm: 16.865990 [I 2022-10-27 02:35:32] ...trap/engines/engine.py.224: train: epoch 43 | batch 570/2304 [I 2022-10-27 02:35:32] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:16:08 [I 2022-10-27 02:35:32] ...trap/engines/engine.py.229: process: 0.56950 | load: 0.00249 [I 2022-10-27 02:35:32] ...trap/engines/engine.py.230: loss: 0.20713 [I 2022-10-27 02:35:32] ...optimizers/trijoint.py.151: total_norm: 12.820150 [I 2022-10-27 02:35:38] ...trap/engines/engine.py.224: train: epoch 43 | batch 580/2304 [I 2022-10-27 02:35:38] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:01 [I 2022-10-27 02:35:38] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00245 [I 2022-10-27 02:35:38] ...trap/engines/engine.py.230: loss: 0.14009 [I 2022-10-27 02:35:38] ...optimizers/trijoint.py.151: total_norm: 16.417149 [I 2022-10-27 02:35:44] ...trap/engines/engine.py.224: train: epoch 43 | batch 590/2304 [I 2022-10-27 02:35:44] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:16:13 [I 2022-10-27 02:35:44] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00246 [I 2022-10-27 02:35:44] ...trap/engines/engine.py.230: loss: 0.10241 [I 2022-10-27 02:35:44] ...optimizers/trijoint.py.151: total_norm: 8.932893 [I 2022-10-27 02:35:49] ...trap/engines/engine.py.224: train: epoch 43 | batch 600/2304 [I 2022-10-27 02:35:49] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:55 [I 2022-10-27 02:35:49] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00245 [I 2022-10-27 02:35:49] ...trap/engines/engine.py.230: loss: 0.18429 [I 2022-10-27 02:35:49] ...optimizers/trijoint.py.151: total_norm: 13.387713 [I 2022-10-27 02:35:55] ...trap/engines/engine.py.224: train: epoch 43 | batch 610/2304 [I 2022-10-27 02:35:55] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:16:04 [I 2022-10-27 02:35:55] ...trap/engines/engine.py.229: process: 0.56957 | load: 0.00285 [I 2022-10-27 02:35:55] ...trap/engines/engine.py.230: loss: 0.20284 [I 2022-10-27 02:35:55] ...optimizers/trijoint.py.151: total_norm: 17.561932 [I 2022-10-27 02:36:00] ...trap/engines/engine.py.224: train: epoch 43 | batch 620/2304 [I 2022-10-27 02:36:00] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:36 [I 2022-10-27 02:36:00] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00245 [I 2022-10-27 02:36:00] ...trap/engines/engine.py.230: loss: 0.19052 [I 2022-10-27 02:36:00] ...optimizers/trijoint.py.151: total_norm: 8.720566 [I 2022-10-27 02:36:06] ...trap/engines/engine.py.224: train: epoch 43 | batch 630/2304 [I 2022-10-27 02:36:06] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:26 [I 2022-10-27 02:36:06] ...trap/engines/engine.py.229: process: 0.55174 | load: 0.00256 [I 2022-10-27 02:36:06] ...trap/engines/engine.py.230: loss: 0.15081 [I 2022-10-27 02:36:06] ...optimizers/trijoint.py.151: total_norm: 9.631719 [I 2022-10-27 02:36:12] ...trap/engines/engine.py.224: train: epoch 43 | batch 640/2304 [I 2022-10-27 02:36:12] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:26 [I 2022-10-27 02:36:12] ...trap/engines/engine.py.229: process: 0.55638 | load: 0.00244 [I 2022-10-27 02:36:12] ...trap/engines/engine.py.230: loss: 0.11665 [I 2022-10-27 02:36:12] ...optimizers/trijoint.py.151: total_norm: 9.987569 [I 2022-10-27 02:36:17] ...trap/engines/engine.py.224: train: epoch 43 | batch 650/2304 [I 2022-10-27 02:36:17] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:20 [I 2022-10-27 02:36:17] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00246 [I 2022-10-27 02:36:17] ...trap/engines/engine.py.230: loss: 0.15528 [I 2022-10-27 02:36:17] ...optimizers/trijoint.py.151: total_norm: 10.231243 [I 2022-10-27 02:36:23] ...trap/engines/engine.py.224: train: epoch 43 | batch 660/2304 [I 2022-10-27 02:36:23] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:14 [I 2022-10-27 02:36:23] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00244 [I 2022-10-27 02:36:23] ...trap/engines/engine.py.230: loss: 0.10162 [I 2022-10-27 02:36:23] ...optimizers/trijoint.py.151: total_norm: 11.983916 [I 2022-10-27 02:36:28] ...trap/engines/engine.py.224: train: epoch 43 | batch 670/2304 [I 2022-10-27 02:36:28] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:07 [I 2022-10-27 02:36:28] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00241 [I 2022-10-27 02:36:28] ...trap/engines/engine.py.230: loss: 0.15715 [I 2022-10-27 02:36:28] ...optimizers/trijoint.py.151: total_norm: 11.062195 [I 2022-10-27 02:36:34] ...trap/engines/engine.py.224: train: epoch 43 | batch 680/2304 [I 2022-10-27 02:36:34] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:34 [I 2022-10-27 02:36:34] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00251 [I 2022-10-27 02:36:34] ...trap/engines/engine.py.230: loss: 0.10380 [I 2022-10-27 02:36:34] ...optimizers/trijoint.py.151: total_norm: 9.669131 [I 2022-10-27 02:36:40] ...trap/engines/engine.py.224: train: epoch 43 | batch 690/2304 [I 2022-10-27 02:36:40] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:14:58 [I 2022-10-27 02:36:40] ...trap/engines/engine.py.229: process: 0.55408 | load: 0.00240 [I 2022-10-27 02:36:40] ...trap/engines/engine.py.230: loss: 0.13719 [I 2022-10-27 02:36:40] ...optimizers/trijoint.py.151: total_norm: 15.946466 [I 2022-10-27 02:36:45] ...trap/engines/engine.py.224: train: epoch 43 | batch 700/2304 [I 2022-10-27 02:36:45] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:46 [I 2022-10-27 02:36:45] ...trap/engines/engine.py.229: process: 0.54825 | load: 0.00240 [I 2022-10-27 02:36:45] ...trap/engines/engine.py.230: loss: 0.26883 [I 2022-10-27 02:36:45] ...optimizers/trijoint.py.151: total_norm: 13.246411 [I 2022-10-27 02:36:51] ...trap/engines/engine.py.224: train: epoch 43 | batch 710/2304 [I 2022-10-27 02:36:51] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:40 [I 2022-10-27 02:36:51] ...trap/engines/engine.py.229: process: 0.55077 | load: 0.00244 [I 2022-10-27 02:36:51] ...trap/engines/engine.py.230: loss: 0.23139 [I 2022-10-27 02:36:51] ...optimizers/trijoint.py.151: total_norm: 40.089859 [I 2022-10-27 02:36:56] ...trap/engines/engine.py.224: train: epoch 43 | batch 720/2304 [I 2022-10-27 02:36:56] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:35 [I 2022-10-27 02:36:56] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00251 [I 2022-10-27 02:36:56] ...trap/engines/engine.py.230: loss: 0.18670 [I 2022-10-27 02:36:56] ...optimizers/trijoint.py.151: total_norm: 26.132900 [I 2022-10-27 02:37:02] ...trap/engines/engine.py.224: train: epoch 43 | batch 730/2304 [I 2022-10-27 02:37:02] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:38 [I 2022-10-27 02:37:02] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00268 [I 2022-10-27 02:37:02] ...trap/engines/engine.py.230: loss: 0.23317 [I 2022-10-27 02:37:02] ...optimizers/trijoint.py.151: total_norm: 15.644597 [I 2022-10-27 02:37:07] ...trap/engines/engine.py.224: train: epoch 43 | batch 740/2304 [I 2022-10-27 02:37:07] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:29 [I 2022-10-27 02:37:07] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00242 [I 2022-10-27 02:37:07] ...trap/engines/engine.py.230: loss: 0.17706 [I 2022-10-27 02:37:07] ...optimizers/trijoint.py.151: total_norm: 15.693339 [I 2022-10-27 02:37:13] ...trap/engines/engine.py.224: train: epoch 43 | batch 750/2304 [I 2022-10-27 02:37:13] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:49 [I 2022-10-27 02:37:13] ...trap/engines/engine.py.229: process: 0.64373 | load: 0.00245 [I 2022-10-27 02:37:13] ...trap/engines/engine.py.230: loss: 0.11525 [I 2022-10-27 02:37:13] ...optimizers/trijoint.py.151: total_norm: 10.101971 [I 2022-10-27 02:37:19] ...trap/engines/engine.py.224: train: epoch 43 | batch 760/2304 [I 2022-10-27 02:37:19] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:19 [I 2022-10-27 02:37:19] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00248 [I 2022-10-27 02:37:19] ...trap/engines/engine.py.230: loss: 0.16194 [I 2022-10-27 02:37:19] ...optimizers/trijoint.py.151: total_norm: 22.735701 [I 2022-10-27 02:37:24] ...trap/engines/engine.py.224: train: epoch 43 | batch 770/2304 [I 2022-10-27 02:37:24] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:07 [I 2022-10-27 02:37:24] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00261 [I 2022-10-27 02:37:24] ...trap/engines/engine.py.230: loss: 0.20831 [I 2022-10-27 02:37:24] ...optimizers/trijoint.py.151: total_norm: 17.235550 [I 2022-10-27 02:37:30] ...trap/engines/engine.py.224: train: epoch 43 | batch 780/2304 [I 2022-10-27 02:37:30] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:16 [I 2022-10-27 02:37:30] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00244 [I 2022-10-27 02:37:30] ...trap/engines/engine.py.230: loss: 0.16623 [I 2022-10-27 02:37:30] ...optimizers/trijoint.py.151: total_norm: 17.302343 [I 2022-10-27 02:37:35] ...trap/engines/engine.py.224: train: epoch 43 | batch 790/2304 [I 2022-10-27 02:37:35] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:07 [I 2022-10-27 02:37:35] ...trap/engines/engine.py.229: process: 0.54902 | load: 0.00247 [I 2022-10-27 02:37:35] ...trap/engines/engine.py.230: loss: 0.14984 [I 2022-10-27 02:37:35] ...optimizers/trijoint.py.151: total_norm: 13.429585 [I 2022-10-27 02:37:41] ...trap/engines/engine.py.224: train: epoch 43 | batch 800/2304 [I 2022-10-27 02:37:41] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:14:01 [I 2022-10-27 02:37:41] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00248 [I 2022-10-27 02:37:41] ...trap/engines/engine.py.230: loss: 0.18516 [I 2022-10-27 02:37:41] ...optimizers/trijoint.py.151: total_norm: 14.422413 [I 2022-10-27 02:37:47] ...trap/engines/engine.py.224: train: epoch 43 | batch 810/2304 [I 2022-10-27 02:37:47] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:13:48 [I 2022-10-27 02:37:47] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00246 [I 2022-10-27 02:37:47] ...trap/engines/engine.py.230: loss: 0.17891 [I 2022-10-27 02:37:47] ...optimizers/trijoint.py.151: total_norm: 17.339983 [I 2022-10-27 02:37:52] ...trap/engines/engine.py.224: train: epoch 43 | batch 820/2304 [I 2022-10-27 02:37:52] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:41 [I 2022-10-27 02:37:52] ...trap/engines/engine.py.229: process: 0.54534 | load: 0.00244 [I 2022-10-27 02:37:52] ...trap/engines/engine.py.230: loss: 0.12556 [I 2022-10-27 02:37:52] ...optimizers/trijoint.py.151: total_norm: 12.395575 [I 2022-10-27 02:37:58] ...trap/engines/engine.py.224: train: epoch 43 | batch 830/2304 [I 2022-10-27 02:37:58] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:41 [I 2022-10-27 02:37:58] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00243 [I 2022-10-27 02:37:58] ...trap/engines/engine.py.230: loss: 0.13186 [I 2022-10-27 02:37:58] ...optimizers/trijoint.py.151: total_norm: 25.665356 [I 2022-10-27 02:38:03] ...trap/engines/engine.py.224: train: epoch 43 | batch 840/2304 [I 2022-10-27 02:38:03] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:33 [I 2022-10-27 02:38:03] ...trap/engines/engine.py.229: process: 0.54811 | load: 0.00248 [I 2022-10-27 02:38:03] ...trap/engines/engine.py.230: loss: 0.20507 [I 2022-10-27 02:38:03] ...optimizers/trijoint.py.151: total_norm: 13.539679 [I 2022-10-27 02:38:09] ...trap/engines/engine.py.224: train: epoch 43 | batch 850/2304 [I 2022-10-27 02:38:09] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:27 [I 2022-10-27 02:38:09] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00241 [I 2022-10-27 02:38:09] ...trap/engines/engine.py.230: loss: 0.12900 [I 2022-10-27 02:38:09] ...optimizers/trijoint.py.151: total_norm: 13.907870 [I 2022-10-27 02:38:15] ...trap/engines/engine.py.224: train: epoch 43 | batch 860/2304 [I 2022-10-27 02:38:15] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:37 [I 2022-10-27 02:38:15] ...trap/engines/engine.py.229: process: 0.54875 | load: 0.00243 [I 2022-10-27 02:38:15] ...trap/engines/engine.py.230: loss: 0.18706 [I 2022-10-27 02:38:15] ...optimizers/trijoint.py.151: total_norm: 28.767962 [I 2022-10-27 02:38:20] ...trap/engines/engine.py.224: train: epoch 43 | batch 870/2304 [I 2022-10-27 02:38:20] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:18 [I 2022-10-27 02:38:20] ...trap/engines/engine.py.229: process: 0.55481 | load: 0.00243 [I 2022-10-27 02:38:20] ...trap/engines/engine.py.230: loss: 0.18191 [I 2022-10-27 02:38:20] ...optimizers/trijoint.py.151: total_norm: 16.235256 [I 2022-10-27 02:38:26] ...trap/engines/engine.py.224: train: epoch 43 | batch 880/2304 [I 2022-10-27 02:38:26] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:11 [I 2022-10-27 02:38:26] ...trap/engines/engine.py.229: process: 0.54274 | load: 0.00245 [I 2022-10-27 02:38:26] ...trap/engines/engine.py.230: loss: 0.16943 [I 2022-10-27 02:38:26] ...optimizers/trijoint.py.151: total_norm: 14.957270 [I 2022-10-27 02:38:31] ...trap/engines/engine.py.224: train: epoch 43 | batch 890/2304 [I 2022-10-27 02:38:31] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:06 [I 2022-10-27 02:38:31] ...trap/engines/engine.py.229: process: 0.56044 | load: 0.00250 [I 2022-10-27 02:38:31] ...trap/engines/engine.py.230: loss: 0.22031 [I 2022-10-27 02:38:31] ...optimizers/trijoint.py.151: total_norm: 19.996902 [I 2022-10-27 02:38:37] ...trap/engines/engine.py.224: train: epoch 43 | batch 900/2304 [I 2022-10-27 02:38:37] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:02 [I 2022-10-27 02:38:37] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00249 [I 2022-10-27 02:38:37] ...trap/engines/engine.py.230: loss: 0.27169 [I 2022-10-27 02:38:37] ...optimizers/trijoint.py.151: total_norm: 13.692407 [I 2022-10-27 02:38:42] ...trap/engines/engine.py.224: train: epoch 43 | batch 910/2304 [I 2022-10-27 02:38:42] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:55 [I 2022-10-27 02:38:42] ...trap/engines/engine.py.229: process: 0.56573 | load: 0.00243 [I 2022-10-27 02:38:42] ...trap/engines/engine.py.230: loss: 0.15487 [I 2022-10-27 02:38:42] ...optimizers/trijoint.py.151: total_norm: 16.038195 [I 2022-10-27 02:38:48] ...trap/engines/engine.py.224: train: epoch 43 | batch 920/2304 [I 2022-10-27 02:38:48] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:46 [I 2022-10-27 02:38:48] ...trap/engines/engine.py.229: process: 0.54519 | load: 0.00243 [I 2022-10-27 02:38:48] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-27 02:38:48] ...optimizers/trijoint.py.151: total_norm: 18.960232 [I 2022-10-27 02:38:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 930/2304 [I 2022-10-27 02:38:53] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:12:48 [I 2022-10-27 02:38:53] ...trap/engines/engine.py.229: process: 0.56037 | load: 0.00246 [I 2022-10-27 02:38:53] ...trap/engines/engine.py.230: loss: 0.13364 [I 2022-10-27 02:38:53] ...optimizers/trijoint.py.151: total_norm: 12.469489 [I 2022-10-27 02:38:59] ...trap/engines/engine.py.224: train: epoch 43 | batch 940/2304 [I 2022-10-27 02:38:59] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:34 [I 2022-10-27 02:38:59] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00249 [I 2022-10-27 02:38:59] ...trap/engines/engine.py.230: loss: 0.20731 [I 2022-10-27 02:38:59] ...optimizers/trijoint.py.151: total_norm: 26.613104 [I 2022-10-27 02:39:05] ...trap/engines/engine.py.224: train: epoch 43 | batch 950/2304 [I 2022-10-27 02:39:05] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:33 [I 2022-10-27 02:39:05] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00243 [I 2022-10-27 02:39:05] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-27 02:39:05] ...optimizers/trijoint.py.151: total_norm: 16.481333 [I 2022-10-27 02:39:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 960/2304 [I 2022-10-27 02:39:10] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:26 [I 2022-10-27 02:39:10] ...trap/engines/engine.py.229: process: 0.55138 | load: 0.00249 [I 2022-10-27 02:39:10] ...trap/engines/engine.py.230: loss: 0.15866 [I 2022-10-27 02:39:10] ...optimizers/trijoint.py.151: total_norm: 16.677565 [I 2022-10-27 02:39:16] ...trap/engines/engine.py.224: train: epoch 43 | batch 970/2304 [I 2022-10-27 02:39:16] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:24 [I 2022-10-27 02:39:16] ...trap/engines/engine.py.229: process: 0.55597 | load: 0.00252 [I 2022-10-27 02:39:16] ...trap/engines/engine.py.230: loss: 0.12140 [I 2022-10-27 02:39:16] ...optimizers/trijoint.py.151: total_norm: 14.092415 [I 2022-10-27 02:39:22] ...trap/engines/engine.py.224: train: epoch 43 | batch 980/2304 [I 2022-10-27 02:39:22] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:26 [I 2022-10-27 02:39:22] ...trap/engines/engine.py.229: process: 0.55758 | load: 0.00248 [I 2022-10-27 02:39:22] ...trap/engines/engine.py.230: loss: 0.12714 [I 2022-10-27 02:39:22] ...optimizers/trijoint.py.151: total_norm: 16.806873 [I 2022-10-27 02:39:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 990/2304 [I 2022-10-27 02:39:27] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:08 [I 2022-10-27 02:39:27] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00244 [I 2022-10-27 02:39:27] ...trap/engines/engine.py.230: loss: 0.26364 [I 2022-10-27 02:39:27] ...optimizers/trijoint.py.151: total_norm: 33.262367 [I 2022-10-27 02:39:33] ...trap/engines/engine.py.224: train: epoch 43 | batch 1000/2304 [I 2022-10-27 02:39:33] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:08 [I 2022-10-27 02:39:33] ...trap/engines/engine.py.229: process: 0.56365 | load: 0.00242 [I 2022-10-27 02:39:33] ...trap/engines/engine.py.230: loss: 0.14284 [I 2022-10-27 02:39:33] ...optimizers/trijoint.py.151: total_norm: 14.204202 [I 2022-10-27 02:39:38] ...trap/engines/engine.py.224: train: epoch 43 | batch 1010/2304 [I 2022-10-27 02:39:38] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:00 [I 2022-10-27 02:39:38] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00243 [I 2022-10-27 02:39:38] ...trap/engines/engine.py.230: loss: 0.23614 [I 2022-10-27 02:39:38] ...optimizers/trijoint.py.151: total_norm: 25.977314 [I 2022-10-27 02:39:44] ...trap/engines/engine.py.224: train: epoch 43 | batch 1020/2304 [I 2022-10-27 02:39:44] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:56 [I 2022-10-27 02:39:44] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00246 [I 2022-10-27 02:39:44] ...trap/engines/engine.py.230: loss: 0.16565 [I 2022-10-27 02:39:44] ...optimizers/trijoint.py.151: total_norm: 21.490648 [I 2022-10-27 02:39:49] ...trap/engines/engine.py.224: train: epoch 43 | batch 1030/2304 [I 2022-10-27 02:39:49] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:49 [I 2022-10-27 02:39:49] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00240 [I 2022-10-27 02:39:49] ...trap/engines/engine.py.230: loss: 0.17770 [I 2022-10-27 02:39:49] ...optimizers/trijoint.py.151: total_norm: 32.471466 [I 2022-10-27 02:39:55] ...trap/engines/engine.py.224: train: epoch 43 | batch 1040/2304 [I 2022-10-27 02:39:55] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:43 [I 2022-10-27 02:39:55] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00238 [I 2022-10-27 02:39:55] ...trap/engines/engine.py.230: loss: 0.15099 [I 2022-10-27 02:39:55] ...optimizers/trijoint.py.151: total_norm: 11.235795 [I 2022-10-27 02:40:00] ...trap/engines/engine.py.224: train: epoch 43 | batch 1050/2304 [I 2022-10-27 02:40:00] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:29 [I 2022-10-27 02:40:00] ...trap/engines/engine.py.229: process: 0.55144 | load: 0.00241 [I 2022-10-27 02:40:00] ...trap/engines/engine.py.230: loss: 0.15512 [I 2022-10-27 02:40:00] ...optimizers/trijoint.py.151: total_norm: 24.072386 [I 2022-10-27 02:40:06] ...trap/engines/engine.py.224: train: epoch 43 | batch 1060/2304 [I 2022-10-27 02:40:06] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:29 [I 2022-10-27 02:40:06] ...trap/engines/engine.py.229: process: 0.56684 | load: 0.00252 [I 2022-10-27 02:40:06] ...trap/engines/engine.py.230: loss: 0.17074 [I 2022-10-27 02:40:06] ...optimizers/trijoint.py.151: total_norm: 10.660618 [I 2022-10-27 02:40:12] ...trap/engines/engine.py.224: train: epoch 43 | batch 1070/2304 [I 2022-10-27 02:40:12] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:25 [I 2022-10-27 02:40:12] ...trap/engines/engine.py.229: process: 0.55072 | load: 0.00247 [I 2022-10-27 02:40:12] ...trap/engines/engine.py.230: loss: 0.20558 [I 2022-10-27 02:40:12] ...optimizers/trijoint.py.151: total_norm: 19.773380 [I 2022-10-27 02:40:17] ...trap/engines/engine.py.224: train: epoch 43 | batch 1080/2304 [I 2022-10-27 02:40:17] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:18 [I 2022-10-27 02:40:17] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00244 [I 2022-10-27 02:40:17] ...trap/engines/engine.py.230: loss: 0.20362 [I 2022-10-27 02:40:17] ...optimizers/trijoint.py.151: total_norm: 28.253717 [I 2022-10-27 02:40:23] ...trap/engines/engine.py.224: train: epoch 43 | batch 1090/2304 [I 2022-10-27 02:40:23] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:24 [I 2022-10-27 02:40:23] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00244 [I 2022-10-27 02:40:23] ...trap/engines/engine.py.230: loss: 0.20954 [I 2022-10-27 02:40:23] ...optimizers/trijoint.py.151: total_norm: 10.131123 [I 2022-10-27 02:40:28] ...trap/engines/engine.py.224: train: epoch 43 | batch 1100/2304 [I 2022-10-27 02:40:28] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:12 [I 2022-10-27 02:40:28] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00245 [I 2022-10-27 02:40:28] ...trap/engines/engine.py.230: loss: 0.16664 [I 2022-10-27 02:40:28] ...optimizers/trijoint.py.151: total_norm: 24.341381 [I 2022-10-27 02:40:34] ...trap/engines/engine.py.224: train: epoch 43 | batch 1110/2304 [I 2022-10-27 02:40:34] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:13 [I 2022-10-27 02:40:34] ...trap/engines/engine.py.229: process: 0.58084 | load: 0.00242 [I 2022-10-27 02:40:34] ...trap/engines/engine.py.230: loss: 0.17311 [I 2022-10-27 02:40:34] ...optimizers/trijoint.py.151: total_norm: 13.641846 [I 2022-10-27 02:40:40] ...trap/engines/engine.py.224: train: epoch 43 | batch 1120/2304 [I 2022-10-27 02:40:40] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:05 [I 2022-10-27 02:40:40] ...trap/engines/engine.py.229: process: 0.58011 | load: 0.00247 [I 2022-10-27 02:40:40] ...trap/engines/engine.py.230: loss: 0.20624 [I 2022-10-27 02:40:40] ...optimizers/trijoint.py.151: total_norm: 30.034288 [I 2022-10-27 02:40:45] ...trap/engines/engine.py.224: train: epoch 43 | batch 1130/2304 [I 2022-10-27 02:40:45] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:54 [I 2022-10-27 02:40:45] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00247 [I 2022-10-27 02:40:45] ...trap/engines/engine.py.230: loss: 0.21070 [I 2022-10-27 02:40:45] ...optimizers/trijoint.py.151: total_norm: 10.749197 [I 2022-10-27 02:40:51] ...trap/engines/engine.py.224: train: epoch 43 | batch 1140/2304 [I 2022-10-27 02:40:51] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:46 [I 2022-10-27 02:40:51] ...trap/engines/engine.py.229: process: 0.54036 | load: 0.00245 [I 2022-10-27 02:40:51] ...trap/engines/engine.py.230: loss: 0.17136 [I 2022-10-27 02:40:51] ...optimizers/trijoint.py.151: total_norm: 21.061846 [I 2022-10-27 02:40:56] ...trap/engines/engine.py.224: train: epoch 43 | batch 1150/2304 [I 2022-10-27 02:40:56] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:10:42 [I 2022-10-27 02:40:56] ...trap/engines/engine.py.229: process: 0.55097 | load: 0.00244 [I 2022-10-27 02:40:56] ...trap/engines/engine.py.230: loss: 0.18378 [I 2022-10-27 02:40:56] ...optimizers/trijoint.py.151: total_norm: 29.939808 [I 2022-10-27 02:41:02] ...trap/engines/engine.py.224: train: epoch 43 | batch 1160/2304 [I 2022-10-27 02:41:02] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:37 [I 2022-10-27 02:41:02] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00316 [I 2022-10-27 02:41:02] ...trap/engines/engine.py.230: loss: 0.15709 [I 2022-10-27 02:41:02] ...optimizers/trijoint.py.151: total_norm: 8.892610 [I 2022-10-27 02:41:08] ...trap/engines/engine.py.224: train: epoch 43 | batch 1170/2304 [I 2022-10-27 02:41:08] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:33 [I 2022-10-27 02:41:08] ...trap/engines/engine.py.229: process: 0.55857 | load: 0.00242 [I 2022-10-27 02:41:08] ...trap/engines/engine.py.230: loss: 0.23429 [I 2022-10-27 02:41:08] ...optimizers/trijoint.py.151: total_norm: 28.017872 [I 2022-10-27 02:41:13] ...trap/engines/engine.py.224: train: epoch 43 | batch 1180/2304 [I 2022-10-27 02:41:13] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:27 [I 2022-10-27 02:41:13] ...trap/engines/engine.py.229: process: 0.56117 | load: 0.00243 [I 2022-10-27 02:41:13] ...trap/engines/engine.py.230: loss: 0.12603 [I 2022-10-27 02:41:13] ...optimizers/trijoint.py.151: total_norm: 11.142003 [I 2022-10-27 02:41:19] ...trap/engines/engine.py.224: train: epoch 43 | batch 1190/2304 [I 2022-10-27 02:41:19] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:19 [I 2022-10-27 02:41:19] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00245 [I 2022-10-27 02:41:19] ...trap/engines/engine.py.230: loss: 0.17511 [I 2022-10-27 02:41:19] ...optimizers/trijoint.py.151: total_norm: 19.755688 [I 2022-10-27 02:41:24] ...trap/engines/engine.py.224: train: epoch 43 | batch 1200/2304 [I 2022-10-27 02:41:24] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:17 [I 2022-10-27 02:41:24] ...trap/engines/engine.py.229: process: 0.57482 | load: 0.00264 [I 2022-10-27 02:41:24] ...trap/engines/engine.py.230: loss: 0.10698 [I 2022-10-27 02:41:24] ...optimizers/trijoint.py.151: total_norm: 19.565971 [I 2022-10-27 02:41:30] ...trap/engines/engine.py.224: train: epoch 43 | batch 1210/2304 [I 2022-10-27 02:41:30] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:10 [I 2022-10-27 02:41:30] ...trap/engines/engine.py.229: process: 0.56168 | load: 0.00250 [I 2022-10-27 02:41:30] ...trap/engines/engine.py.230: loss: 0.18876 [I 2022-10-27 02:41:30] ...optimizers/trijoint.py.151: total_norm: 20.531540 [I 2022-10-27 02:41:36] ...trap/engines/engine.py.224: train: epoch 43 | batch 1220/2304 [I 2022-10-27 02:41:36] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:03 [I 2022-10-27 02:41:36] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00246 [I 2022-10-27 02:41:36] ...trap/engines/engine.py.230: loss: 0.14954 [I 2022-10-27 02:41:36] ...optimizers/trijoint.py.151: total_norm: 22.651848 [I 2022-10-27 02:41:41] ...trap/engines/engine.py.224: train: epoch 43 | batch 1230/2304 [I 2022-10-27 02:41:41] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:09:53 [I 2022-10-27 02:41:41] ...trap/engines/engine.py.229: process: 0.54868 | load: 0.00245 [I 2022-10-27 02:41:41] ...trap/engines/engine.py.230: loss: 0.19866 [I 2022-10-27 02:41:41] ...optimizers/trijoint.py.151: total_norm: 14.958564 [I 2022-10-27 02:41:47] ...trap/engines/engine.py.224: train: epoch 43 | batch 1240/2304 [I 2022-10-27 02:41:47] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:48 [I 2022-10-27 02:41:47] ...trap/engines/engine.py.229: process: 0.54567 | load: 0.00246 [I 2022-10-27 02:41:47] ...trap/engines/engine.py.230: loss: 0.21594 [I 2022-10-27 02:41:47] ...optimizers/trijoint.py.151: total_norm: 19.445549 [I 2022-10-27 02:41:52] ...trap/engines/engine.py.224: train: epoch 43 | batch 1250/2304 [I 2022-10-27 02:41:52] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:09:44 [I 2022-10-27 02:41:52] ...trap/engines/engine.py.229: process: 0.55941 | load: 0.00245 [I 2022-10-27 02:41:52] ...trap/engines/engine.py.230: loss: 0.17266 [I 2022-10-27 02:41:52] ...optimizers/trijoint.py.151: total_norm: 15.718976 [I 2022-10-27 02:41:58] ...trap/engines/engine.py.224: train: epoch 43 | batch 1260/2304 [I 2022-10-27 02:41:58] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:36 [I 2022-10-27 02:41:58] ...trap/engines/engine.py.229: process: 0.54148 | load: 0.00247 [I 2022-10-27 02:41:58] ...trap/engines/engine.py.230: loss: 0.16096 [I 2022-10-27 02:41:58] ...optimizers/trijoint.py.151: total_norm: 21.436556 [I 2022-10-27 02:42:03] ...trap/engines/engine.py.224: train: epoch 43 | batch 1270/2304 [I 2022-10-27 02:42:03] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:36 [I 2022-10-27 02:42:03] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00244 [I 2022-10-27 02:42:03] ...trap/engines/engine.py.230: loss: 0.21993 [I 2022-10-27 02:42:03] ...optimizers/trijoint.py.151: total_norm: 16.770784 [I 2022-10-27 02:42:09] ...trap/engines/engine.py.224: train: epoch 43 | batch 1280/2304 [I 2022-10-27 02:42:09] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:33 [I 2022-10-27 02:42:09] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00252 [I 2022-10-27 02:42:09] ...trap/engines/engine.py.230: loss: 0.22356 [I 2022-10-27 02:42:09] ...optimizers/trijoint.py.151: total_norm: 19.491564 [I 2022-10-27 02:42:15] ...trap/engines/engine.py.224: train: epoch 43 | batch 1290/2304 [I 2022-10-27 02:42:15] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:24 [I 2022-10-27 02:42:15] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00250 [I 2022-10-27 02:42:15] ...trap/engines/engine.py.230: loss: 0.20148 [I 2022-10-27 02:42:15] ...optimizers/trijoint.py.151: total_norm: 14.022934 [I 2022-10-27 02:42:20] ...trap/engines/engine.py.224: train: epoch 43 | batch 1300/2304 [I 2022-10-27 02:42:20] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:20 [I 2022-10-27 02:42:20] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00246 [I 2022-10-27 02:42:20] ...trap/engines/engine.py.230: loss: 0.18499 [I 2022-10-27 02:42:20] ...optimizers/trijoint.py.151: total_norm: 17.638277 [I 2022-10-27 02:42:26] ...trap/engines/engine.py.224: train: epoch 43 | batch 1310/2304 [I 2022-10-27 02:42:26] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:14 [I 2022-10-27 02:42:26] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00244 [I 2022-10-27 02:42:26] ...trap/engines/engine.py.230: loss: 0.19016 [I 2022-10-27 02:42:26] ...optimizers/trijoint.py.151: total_norm: 16.926790 [I 2022-10-27 02:42:31] ...trap/engines/engine.py.224: train: epoch 43 | batch 1320/2304 [I 2022-10-27 02:42:31] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:06 [I 2022-10-27 02:42:31] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00244 [I 2022-10-27 02:42:31] ...trap/engines/engine.py.230: loss: 0.17636 [I 2022-10-27 02:42:31] ...optimizers/trijoint.py.151: total_norm: 25.207125 [I 2022-10-27 02:42:37] ...trap/engines/engine.py.224: train: epoch 43 | batch 1330/2304 [I 2022-10-27 02:42:37] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:00 [I 2022-10-27 02:42:37] ...trap/engines/engine.py.229: process: 0.55865 | load: 0.00244 [I 2022-10-27 02:42:37] ...trap/engines/engine.py.230: loss: 0.20475 [I 2022-10-27 02:42:37] ...optimizers/trijoint.py.151: total_norm: 38.496277 [I 2022-10-27 02:42:42] ...trap/engines/engine.py.224: train: epoch 43 | batch 1340/2304 [I 2022-10-27 02:42:42] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:08:51 [I 2022-10-27 02:42:42] ...trap/engines/engine.py.229: process: 0.54709 | load: 0.00247 [I 2022-10-27 02:42:42] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-27 02:42:42] ...optimizers/trijoint.py.151: total_norm: 9.781887 [I 2022-10-27 02:42:48] ...trap/engines/engine.py.224: train: epoch 43 | batch 1350/2304 [I 2022-10-27 02:42:48] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:48 [I 2022-10-27 02:42:48] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00243 [I 2022-10-27 02:42:48] ...trap/engines/engine.py.230: loss: 0.18918 [I 2022-10-27 02:42:48] ...optimizers/trijoint.py.151: total_norm: 27.057695 [I 2022-10-27 02:42:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 1360/2304 [I 2022-10-27 02:42:53] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:08:43 [I 2022-10-27 02:42:53] ...trap/engines/engine.py.229: process: 0.55609 | load: 0.00246 [I 2022-10-27 02:42:53] ...trap/engines/engine.py.230: loss: 0.20903 [I 2022-10-27 02:42:53] ...optimizers/trijoint.py.151: total_norm: 22.104055 [I 2022-10-27 02:42:59] ...trap/engines/engine.py.224: train: epoch 43 | batch 1370/2304 [I 2022-10-27 02:42:59] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:37 [I 2022-10-27 02:42:59] ...trap/engines/engine.py.229: process: 0.54476 | load: 0.00255 [I 2022-10-27 02:42:59] ...trap/engines/engine.py.230: loss: 0.12084 [I 2022-10-27 02:42:59] ...optimizers/trijoint.py.151: total_norm: 24.888374 [I 2022-10-27 02:43:05] ...trap/engines/engine.py.224: train: epoch 43 | batch 1380/2304 [I 2022-10-27 02:43:05] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:35 [I 2022-10-27 02:43:05] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00250 [I 2022-10-27 02:43:05] ...trap/engines/engine.py.230: loss: 0.14382 [I 2022-10-27 02:43:05] ...optimizers/trijoint.py.151: total_norm: 14.671345 [I 2022-10-27 02:43:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 1390/2304 [I 2022-10-27 02:43:10] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:23 [I 2022-10-27 02:43:10] ...trap/engines/engine.py.229: process: 0.53966 | load: 0.00246 [I 2022-10-27 02:43:10] ...trap/engines/engine.py.230: loss: 0.23806 [I 2022-10-27 02:43:10] ...optimizers/trijoint.py.151: total_norm: 19.471817 [I 2022-10-27 02:43:16] ...trap/engines/engine.py.224: train: epoch 43 | batch 1400/2304 [I 2022-10-27 02:43:16] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:19 [I 2022-10-27 02:43:16] ...trap/engines/engine.py.229: process: 0.54535 | load: 0.00242 [I 2022-10-27 02:43:16] ...trap/engines/engine.py.230: loss: 0.15954 [I 2022-10-27 02:43:16] ...optimizers/trijoint.py.151: total_norm: 11.151417 [I 2022-10-27 02:43:21] ...trap/engines/engine.py.224: train: epoch 43 | batch 1410/2304 [I 2022-10-27 02:43:21] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:17 [I 2022-10-27 02:43:21] ...trap/engines/engine.py.229: process: 0.56398 | load: 0.00277 [I 2022-10-27 02:43:21] ...trap/engines/engine.py.230: loss: 0.20211 [I 2022-10-27 02:43:21] ...optimizers/trijoint.py.151: total_norm: 12.748966 [I 2022-10-27 02:43:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 1420/2304 [I 2022-10-27 02:43:27] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:12 [I 2022-10-27 02:43:27] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00246 [I 2022-10-27 02:43:27] ...trap/engines/engine.py.230: loss: 0.12009 [I 2022-10-27 02:43:27] ...optimizers/trijoint.py.151: total_norm: 14.268893 [I 2022-10-27 02:43:32] ...trap/engines/engine.py.224: train: epoch 43 | batch 1430/2304 [I 2022-10-27 02:43:32] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:06 [I 2022-10-27 02:43:32] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00253 [I 2022-10-27 02:43:32] ...trap/engines/engine.py.230: loss: 0.19041 [I 2022-10-27 02:43:32] ...optimizers/trijoint.py.151: total_norm: 20.084225 [I 2022-10-27 02:43:38] ...trap/engines/engine.py.224: train: epoch 43 | batch 1440/2304 [I 2022-10-27 02:43:38] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:05 [I 2022-10-27 02:43:38] ...trap/engines/engine.py.229: process: 0.56416 | load: 0.00256 [I 2022-10-27 02:43:38] ...trap/engines/engine.py.230: loss: 0.18599 [I 2022-10-27 02:43:38] ...optimizers/trijoint.py.151: total_norm: 27.501862 [I 2022-10-27 02:43:44] ...trap/engines/engine.py.224: train: epoch 43 | batch 1450/2304 [I 2022-10-27 02:43:44] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:07:55 [I 2022-10-27 02:43:44] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00250 [I 2022-10-27 02:43:44] ...trap/engines/engine.py.230: loss: 0.11999 [I 2022-10-27 02:43:44] ...optimizers/trijoint.py.151: total_norm: 16.971085 [I 2022-10-27 02:43:49] ...trap/engines/engine.py.224: train: epoch 43 | batch 1460/2304 [I 2022-10-27 02:43:49] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:07:52 [I 2022-10-27 02:43:49] ...trap/engines/engine.py.229: process: 0.56862 | load: 0.00249 [I 2022-10-27 02:43:49] ...trap/engines/engine.py.230: loss: 0.16409 [I 2022-10-27 02:43:49] ...optimizers/trijoint.py.151: total_norm: 15.050187 [I 2022-10-27 02:43:55] ...trap/engines/engine.py.224: train: epoch 43 | batch 1470/2304 [I 2022-10-27 02:43:55] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:07:42 [I 2022-10-27 02:43:55] ...trap/engines/engine.py.229: process: 0.56313 | load: 0.00250 [I 2022-10-27 02:43:55] ...trap/engines/engine.py.230: loss: 0.20073 [I 2022-10-27 02:43:55] ...optimizers/trijoint.py.151: total_norm: 28.703302 [I 2022-10-27 02:44:00] ...trap/engines/engine.py.224: train: epoch 43 | batch 1480/2304 [I 2022-10-27 02:44:00] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:39 [I 2022-10-27 02:44:00] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00244 [I 2022-10-27 02:44:00] ...trap/engines/engine.py.230: loss: 0.17509 [I 2022-10-27 02:44:00] ...optimizers/trijoint.py.151: total_norm: 18.162012 [I 2022-10-27 02:44:06] ...trap/engines/engine.py.224: train: epoch 43 | batch 1490/2304 [I 2022-10-27 02:44:06] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:34 [I 2022-10-27 02:44:06] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00243 [I 2022-10-27 02:44:06] ...trap/engines/engine.py.230: loss: 0.19628 [I 2022-10-27 02:44:06] ...optimizers/trijoint.py.151: total_norm: 13.145856 [I 2022-10-27 02:44:12] ...trap/engines/engine.py.224: train: epoch 43 | batch 1500/2304 [I 2022-10-27 02:44:12] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:29 [I 2022-10-27 02:44:12] ...trap/engines/engine.py.229: process: 0.56715 | load: 0.00268 [I 2022-10-27 02:44:12] ...trap/engines/engine.py.230: loss: 0.15126 [I 2022-10-27 02:44:12] ...optimizers/trijoint.py.151: total_norm: 15.396052 [I 2022-10-27 02:44:17] ...trap/engines/engine.py.224: train: epoch 43 | batch 1510/2304 [I 2022-10-27 02:44:17] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:22 [I 2022-10-27 02:44:17] ...trap/engines/engine.py.229: process: 0.56414 | load: 0.00244 [I 2022-10-27 02:44:17] ...trap/engines/engine.py.230: loss: 0.18332 [I 2022-10-27 02:44:17] ...optimizers/trijoint.py.151: total_norm: 18.713985 [I 2022-10-27 02:44:23] ...trap/engines/engine.py.224: train: epoch 43 | batch 1520/2304 [I 2022-10-27 02:44:23] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:16 [I 2022-10-27 02:44:23] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00245 [I 2022-10-27 02:44:23] ...trap/engines/engine.py.230: loss: 0.18394 [I 2022-10-27 02:44:23] ...optimizers/trijoint.py.151: total_norm: 29.536915 [I 2022-10-27 02:44:28] ...trap/engines/engine.py.224: train: epoch 43 | batch 1530/2304 [I 2022-10-27 02:44:28] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:10 [I 2022-10-27 02:44:28] ...trap/engines/engine.py.229: process: 0.55552 | load: 0.00257 [I 2022-10-27 02:44:28] ...trap/engines/engine.py.230: loss: 0.19714 [I 2022-10-27 02:44:28] ...optimizers/trijoint.py.151: total_norm: 20.518793 [I 2022-10-27 02:44:34] ...trap/engines/engine.py.224: train: epoch 43 | batch 1540/2304 [I 2022-10-27 02:44:34] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:02 [I 2022-10-27 02:44:34] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00248 [I 2022-10-27 02:44:34] ...trap/engines/engine.py.230: loss: 0.16497 [I 2022-10-27 02:44:34] ...optimizers/trijoint.py.151: total_norm: 24.009350 [I 2022-10-27 02:44:39] ...trap/engines/engine.py.224: train: epoch 43 | batch 1550/2304 [I 2022-10-27 02:44:39] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:00 [I 2022-10-27 02:44:39] ...trap/engines/engine.py.229: process: 0.56316 | load: 0.00247 [I 2022-10-27 02:44:39] ...trap/engines/engine.py.230: loss: 0.17058 [I 2022-10-27 02:44:39] ...optimizers/trijoint.py.151: total_norm: 14.730866 [I 2022-10-27 02:44:45] ...trap/engines/engine.py.224: train: epoch 43 | batch 1560/2304 [I 2022-10-27 02:44:45] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:06:49 [I 2022-10-27 02:44:45] ...trap/engines/engine.py.229: process: 0.53967 | load: 0.00244 [I 2022-10-27 02:44:45] ...trap/engines/engine.py.230: loss: 0.11995 [I 2022-10-27 02:44:45] ...optimizers/trijoint.py.151: total_norm: 12.238226 [I 2022-10-27 02:44:50] ...trap/engines/engine.py.224: train: epoch 43 | batch 1570/2304 [I 2022-10-27 02:44:50] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:06:48 [I 2022-10-27 02:44:50] ...trap/engines/engine.py.229: process: 0.56293 | load: 0.00243 [I 2022-10-27 02:44:50] ...trap/engines/engine.py.230: loss: 0.23216 [I 2022-10-27 02:44:50] ...optimizers/trijoint.py.151: total_norm: 36.698429 [I 2022-10-27 02:44:56] ...trap/engines/engine.py.224: train: epoch 43 | batch 1580/2304 [I 2022-10-27 02:44:56] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:06:43 [I 2022-10-27 02:44:56] ...trap/engines/engine.py.229: process: 0.55568 | load: 0.00245 [I 2022-10-27 02:44:56] ...trap/engines/engine.py.230: loss: 0.15124 [I 2022-10-27 02:44:56] ...optimizers/trijoint.py.151: total_norm: 13.716401 [I 2022-10-27 02:45:02] ...trap/engines/engine.py.224: train: epoch 43 | batch 1590/2304 [I 2022-10-27 02:45:02] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:37 [I 2022-10-27 02:45:02] ...trap/engines/engine.py.229: process: 0.54303 | load: 0.00243 [I 2022-10-27 02:45:02] ...trap/engines/engine.py.230: loss: 0.14457 [I 2022-10-27 02:45:02] ...optimizers/trijoint.py.151: total_norm: 29.636568 [I 2022-10-27 02:45:07] ...trap/engines/engine.py.224: train: epoch 43 | batch 1600/2304 [I 2022-10-27 02:45:07] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:33 [I 2022-10-27 02:45:07] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00252 [I 2022-10-27 02:45:07] ...trap/engines/engine.py.230: loss: 0.18800 [I 2022-10-27 02:45:07] ...optimizers/trijoint.py.151: total_norm: 13.997344 [I 2022-10-27 02:45:13] ...trap/engines/engine.py.224: train: epoch 43 | batch 1610/2304 [I 2022-10-27 02:45:13] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:21 [I 2022-10-27 02:45:13] ...trap/engines/engine.py.229: process: 0.54258 | load: 0.00247 [I 2022-10-27 02:45:13] ...trap/engines/engine.py.230: loss: 0.23031 [I 2022-10-27 02:45:13] ...optimizers/trijoint.py.151: total_norm: 14.759890 [I 2022-10-27 02:45:18] ...trap/engines/engine.py.224: train: epoch 43 | batch 1620/2304 [I 2022-10-27 02:45:18] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:19 [I 2022-10-27 02:45:18] ...trap/engines/engine.py.229: process: 0.55485 | load: 0.00244 [I 2022-10-27 02:45:18] ...trap/engines/engine.py.230: loss: 0.13445 [I 2022-10-27 02:45:18] ...optimizers/trijoint.py.151: total_norm: 20.286097 [I 2022-10-27 02:45:24] ...trap/engines/engine.py.224: train: epoch 43 | batch 1630/2304 [I 2022-10-27 02:45:24] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:12 [I 2022-10-27 02:45:24] ...trap/engines/engine.py.229: process: 0.54741 | load: 0.00243 [I 2022-10-27 02:45:24] ...trap/engines/engine.py.230: loss: 0.16653 [I 2022-10-27 02:45:24] ...optimizers/trijoint.py.151: total_norm: 20.584721 [I 2022-10-27 02:45:29] ...trap/engines/engine.py.224: train: epoch 43 | batch 1640/2304 [I 2022-10-27 02:45:29] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:08 [I 2022-10-27 02:45:29] ...trap/engines/engine.py.229: process: 0.55079 | load: 0.00241 [I 2022-10-27 02:45:29] ...trap/engines/engine.py.230: loss: 0.20425 [I 2022-10-27 02:45:29] ...optimizers/trijoint.py.151: total_norm: 23.838171 [I 2022-10-27 02:45:35] ...trap/engines/engine.py.224: train: epoch 43 | batch 1650/2304 [I 2022-10-27 02:45:35] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:02 [I 2022-10-27 02:45:35] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00262 [I 2022-10-27 02:45:35] ...trap/engines/engine.py.230: loss: 0.16331 [I 2022-10-27 02:45:35] ...optimizers/trijoint.py.151: total_norm: 12.655931 [I 2022-10-27 02:45:41] ...trap/engines/engine.py.224: train: epoch 43 | batch 1660/2304 [I 2022-10-27 02:45:41] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:06 [I 2022-10-27 02:45:41] ...trap/engines/engine.py.229: process: 0.54816 | load: 0.00244 [I 2022-10-27 02:45:41] ...trap/engines/engine.py.230: loss: 0.19234 [I 2022-10-27 02:45:41] ...optimizers/trijoint.py.151: total_norm: 37.613842 [I 2022-10-27 02:45:46] ...trap/engines/engine.py.224: train: epoch 43 | batch 1670/2304 [I 2022-10-27 02:45:46] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:05:52 [I 2022-10-27 02:45:46] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00242 [I 2022-10-27 02:45:46] ...trap/engines/engine.py.230: loss: 0.20673 [I 2022-10-27 02:45:46] ...optimizers/trijoint.py.151: total_norm: 34.878597 [I 2022-10-27 02:45:52] ...trap/engines/engine.py.224: train: epoch 43 | batch 1680/2304 [I 2022-10-27 02:45:52] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:05:49 [I 2022-10-27 02:45:52] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00254 [I 2022-10-27 02:45:52] ...trap/engines/engine.py.230: loss: 0.23761 [I 2022-10-27 02:45:52] ...optimizers/trijoint.py.151: total_norm: 23.328587 [I 2022-10-27 02:45:58] ...trap/engines/engine.py.224: train: epoch 43 | batch 1690/2304 [I 2022-10-27 02:45:58] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:05:42 [I 2022-10-27 02:45:58] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00251 [I 2022-10-27 02:45:58] ...trap/engines/engine.py.230: loss: 0.18701 [I 2022-10-27 02:45:58] ...optimizers/trijoint.py.151: total_norm: 33.743721 [I 2022-10-27 02:46:03] ...trap/engines/engine.py.224: train: epoch 43 | batch 1700/2304 [I 2022-10-27 02:46:03] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:37 [I 2022-10-27 02:46:03] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00247 [I 2022-10-27 02:46:03] ...trap/engines/engine.py.230: loss: 0.17658 [I 2022-10-27 02:46:03] ...optimizers/trijoint.py.151: total_norm: 29.026119 [I 2022-10-27 02:46:09] ...trap/engines/engine.py.224: train: epoch 43 | batch 1710/2304 [I 2022-10-27 02:46:09] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:28 [I 2022-10-27 02:46:09] ...trap/engines/engine.py.229: process: 0.54777 | load: 0.00242 [I 2022-10-27 02:46:09] ...trap/engines/engine.py.230: loss: 0.10902 [I 2022-10-27 02:46:09] ...optimizers/trijoint.py.151: total_norm: 7.074737 [I 2022-10-27 02:46:14] ...trap/engines/engine.py.224: train: epoch 43 | batch 1720/2304 [I 2022-10-27 02:46:14] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:24 [I 2022-10-27 02:46:14] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00242 [I 2022-10-27 02:46:14] ...trap/engines/engine.py.230: loss: 0.19888 [I 2022-10-27 02:46:14] ...optimizers/trijoint.py.151: total_norm: 16.387861 [I 2022-10-27 02:46:20] ...trap/engines/engine.py.224: train: epoch 43 | batch 1730/2304 [I 2022-10-27 02:46:20] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:17 [I 2022-10-27 02:46:20] ...trap/engines/engine.py.229: process: 0.53986 | load: 0.00242 [I 2022-10-27 02:46:20] ...trap/engines/engine.py.230: loss: 0.20493 [I 2022-10-27 02:46:20] ...optimizers/trijoint.py.151: total_norm: 17.473461 [I 2022-10-27 02:46:25] ...trap/engines/engine.py.224: train: epoch 43 | batch 1740/2304 [I 2022-10-27 02:46:25] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:13 [I 2022-10-27 02:46:25] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00250 [I 2022-10-27 02:46:25] ...trap/engines/engine.py.230: loss: 0.15340 [I 2022-10-27 02:46:25] ...optimizers/trijoint.py.151: total_norm: 20.649502 [I 2022-10-27 02:46:31] ...trap/engines/engine.py.224: train: epoch 43 | batch 1750/2304 [I 2022-10-27 02:46:31] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:08 [I 2022-10-27 02:46:31] ...trap/engines/engine.py.229: process: 0.56234 | load: 0.00270 [I 2022-10-27 02:46:31] ...trap/engines/engine.py.230: loss: 0.14051 [I 2022-10-27 02:46:31] ...optimizers/trijoint.py.151: total_norm: 15.568248 [I 2022-10-27 02:46:37] ...trap/engines/engine.py.224: train: epoch 43 | batch 1760/2304 [I 2022-10-27 02:46:37] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:02 [I 2022-10-27 02:46:37] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00248 [I 2022-10-27 02:46:37] ...trap/engines/engine.py.230: loss: 0.15008 [I 2022-10-27 02:46:37] ...optimizers/trijoint.py.151: total_norm: 20.007265 [I 2022-10-27 02:46:42] ...trap/engines/engine.py.224: train: epoch 43 | batch 1770/2304 [I 2022-10-27 02:46:42] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:04:57 [I 2022-10-27 02:46:42] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00248 [I 2022-10-27 02:46:42] ...trap/engines/engine.py.230: loss: 0.17011 [I 2022-10-27 02:46:42] ...optimizers/trijoint.py.151: total_norm: 30.914454 [I 2022-10-27 02:46:48] ...trap/engines/engine.py.224: train: epoch 43 | batch 1780/2304 [I 2022-10-27 02:46:48] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:04:53 [I 2022-10-27 02:46:48] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00249 [I 2022-10-27 02:46:48] ...trap/engines/engine.py.230: loss: 0.17035 [I 2022-10-27 02:46:48] ...optimizers/trijoint.py.151: total_norm: 16.724546 [I 2022-10-27 02:46:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 1790/2304 [I 2022-10-27 02:46:53] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:04:45 [I 2022-10-27 02:46:53] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00245 [I 2022-10-27 02:46:53] ...trap/engines/engine.py.230: loss: 0.12192 [I 2022-10-27 02:46:53] ...optimizers/trijoint.py.151: total_norm: 13.642365 [I 2022-10-27 02:46:59] ...trap/engines/engine.py.224: train: epoch 43 | batch 1800/2304 [I 2022-10-27 02:46:59] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:41 [I 2022-10-27 02:46:59] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00250 [I 2022-10-27 02:46:59] ...trap/engines/engine.py.230: loss: 0.22791 [I 2022-10-27 02:46:59] ...optimizers/trijoint.py.151: total_norm: 20.931028 [I 2022-10-27 02:47:05] ...trap/engines/engine.py.224: train: epoch 43 | batch 1810/2304 [I 2022-10-27 02:47:05] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:44 [I 2022-10-27 02:47:05] ...trap/engines/engine.py.229: process: 0.64763 | load: 0.00252 [I 2022-10-27 02:47:05] ...trap/engines/engine.py.230: loss: 0.11995 [I 2022-10-27 02:47:05] ...optimizers/trijoint.py.151: total_norm: 8.606693 [I 2022-10-27 02:47:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 1820/2304 [I 2022-10-27 02:47:10] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:29 [I 2022-10-27 02:47:10] ...trap/engines/engine.py.229: process: 0.54020 | load: 0.00243 [I 2022-10-27 02:47:10] ...trap/engines/engine.py.230: loss: 0.13663 [I 2022-10-27 02:47:10] ...optimizers/trijoint.py.151: total_norm: 9.205798 [I 2022-10-27 02:47:16] ...trap/engines/engine.py.224: train: epoch 43 | batch 1830/2304 [I 2022-10-27 02:47:16] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:22 [I 2022-10-27 02:47:16] ...trap/engines/engine.py.229: process: 0.55113 | load: 0.00245 [I 2022-10-27 02:47:16] ...trap/engines/engine.py.230: loss: 0.18329 [I 2022-10-27 02:47:16] ...optimizers/trijoint.py.151: total_norm: 16.433475 [I 2022-10-27 02:47:21] ...trap/engines/engine.py.224: train: epoch 43 | batch 1840/2304 [I 2022-10-27 02:47:21] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:19 [I 2022-10-27 02:47:21] ...trap/engines/engine.py.229: process: 0.54750 | load: 0.00243 [I 2022-10-27 02:47:21] ...trap/engines/engine.py.230: loss: 0.14223 [I 2022-10-27 02:47:21] ...optimizers/trijoint.py.151: total_norm: 10.606918 [I 2022-10-27 02:47:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 1850/2304 [I 2022-10-27 02:47:27] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:10 [I 2022-10-27 02:47:27] ...trap/engines/engine.py.229: process: 0.55260 | load: 0.00241 [I 2022-10-27 02:47:27] ...trap/engines/engine.py.230: loss: 0.12359 [I 2022-10-27 02:47:27] ...optimizers/trijoint.py.151: total_norm: 21.430870 [I 2022-10-27 02:47:33] ...trap/engines/engine.py.224: train: epoch 43 | batch 1860/2304 [I 2022-10-27 02:47:33] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:08 [I 2022-10-27 02:47:33] ...trap/engines/engine.py.229: process: 0.56475 | load: 0.00254 [I 2022-10-27 02:47:33] ...trap/engines/engine.py.230: loss: 0.13469 [I 2022-10-27 02:47:33] ...optimizers/trijoint.py.151: total_norm: 12.629156 [I 2022-10-27 02:47:38] ...trap/engines/engine.py.224: train: epoch 43 | batch 1870/2304 [I 2022-10-27 02:47:38] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:01 [I 2022-10-27 02:47:38] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00251 [I 2022-10-27 02:47:38] ...trap/engines/engine.py.230: loss: 0.23609 [I 2022-10-27 02:47:38] ...optimizers/trijoint.py.151: total_norm: 64.127769 [I 2022-10-27 02:47:44] ...trap/engines/engine.py.224: train: epoch 43 | batch 1880/2304 [I 2022-10-27 02:47:44] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:03:57 [I 2022-10-27 02:47:44] ...trap/engines/engine.py.229: process: 0.56256 | load: 0.00250 [I 2022-10-27 02:47:44] ...trap/engines/engine.py.230: loss: 0.16000 [I 2022-10-27 02:47:44] ...optimizers/trijoint.py.151: total_norm: 15.870718 [I 2022-10-27 02:47:49] ...trap/engines/engine.py.224: train: epoch 43 | batch 1890/2304 [I 2022-10-27 02:47:49] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:03:49 [I 2022-10-27 02:47:49] ...trap/engines/engine.py.229: process: 0.55787 | load: 0.00245 [I 2022-10-27 02:47:49] ...trap/engines/engine.py.230: loss: 0.16772 [I 2022-10-27 02:47:49] ...optimizers/trijoint.py.151: total_norm: 16.634775 [I 2022-10-27 02:47:55] ...trap/engines/engine.py.224: train: epoch 43 | batch 1900/2304 [I 2022-10-27 02:47:55] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:03:44 [I 2022-10-27 02:47:55] ...trap/engines/engine.py.229: process: 0.54967 | load: 0.00242 [I 2022-10-27 02:47:55] ...trap/engines/engine.py.230: loss: 0.15729 [I 2022-10-27 02:47:55] ...optimizers/trijoint.py.151: total_norm: 12.914694 [I 2022-10-27 02:48:00] ...trap/engines/engine.py.224: train: epoch 43 | batch 1910/2304 [I 2022-10-27 02:48:00] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:03:39 [I 2022-10-27 02:48:00] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00242 [I 2022-10-27 02:48:00] ...trap/engines/engine.py.230: loss: 0.16153 [I 2022-10-27 02:48:00] ...optimizers/trijoint.py.151: total_norm: 26.003580 [I 2022-10-27 02:48:06] ...trap/engines/engine.py.224: train: epoch 43 | batch 1920/2304 [I 2022-10-27 02:48:06] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:32 [I 2022-10-27 02:48:06] ...trap/engines/engine.py.229: process: 0.54924 | load: 0.00243 [I 2022-10-27 02:48:06] ...trap/engines/engine.py.230: loss: 0.15094 [I 2022-10-27 02:48:06] ...optimizers/trijoint.py.151: total_norm: 21.595242 [I 2022-10-27 02:48:12] ...trap/engines/engine.py.224: train: epoch 43 | batch 1930/2304 [I 2022-10-27 02:48:12] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:31 [I 2022-10-27 02:48:12] ...trap/engines/engine.py.229: process: 0.55383 | load: 0.00252 [I 2022-10-27 02:48:12] ...trap/engines/engine.py.230: loss: 0.22096 [I 2022-10-27 02:48:12] ...optimizers/trijoint.py.151: total_norm: 11.543752 [I 2022-10-27 02:48:17] ...trap/engines/engine.py.224: train: epoch 43 | batch 1940/2304 [I 2022-10-27 02:48:17] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:23 [I 2022-10-27 02:48:17] ...trap/engines/engine.py.229: process: 0.55618 | load: 0.00248 [I 2022-10-27 02:48:17] ...trap/engines/engine.py.230: loss: 0.13555 [I 2022-10-27 02:48:17] ...optimizers/trijoint.py.151: total_norm: 26.913975 [I 2022-10-27 02:48:23] ...trap/engines/engine.py.224: train: epoch 43 | batch 1950/2304 [I 2022-10-27 02:48:23] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:16 [I 2022-10-27 02:48:23] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00246 [I 2022-10-27 02:48:23] ...trap/engines/engine.py.230: loss: 0.17863 [I 2022-10-27 02:48:23] ...optimizers/trijoint.py.151: total_norm: 16.788866 [I 2022-10-27 02:48:28] ...trap/engines/engine.py.224: train: epoch 43 | batch 1960/2304 [I 2022-10-27 02:48:28] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:11 [I 2022-10-27 02:48:28] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00246 [I 2022-10-27 02:48:28] ...trap/engines/engine.py.230: loss: 0.17087 [I 2022-10-27 02:48:28] ...optimizers/trijoint.py.151: total_norm: 16.016071 [I 2022-10-27 02:48:34] ...trap/engines/engine.py.224: train: epoch 43 | batch 1970/2304 [I 2022-10-27 02:48:34] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:06 [I 2022-10-27 02:48:34] ...trap/engines/engine.py.229: process: 0.56505 | load: 0.00243 [I 2022-10-27 02:48:34] ...trap/engines/engine.py.230: loss: 0.18330 [I 2022-10-27 02:48:34] ...optimizers/trijoint.py.151: total_norm: 22.809206 [I 2022-10-27 02:48:40] ...trap/engines/engine.py.224: train: epoch 43 | batch 1980/2304 [I 2022-10-27 02:48:40] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:02:59 [I 2022-10-27 02:48:40] ...trap/engines/engine.py.229: process: 0.53882 | load: 0.00247 [I 2022-10-27 02:48:40] ...trap/engines/engine.py.230: loss: 0.14523 [I 2022-10-27 02:48:40] ...optimizers/trijoint.py.151: total_norm: 18.982138 [I 2022-10-27 02:48:45] ...trap/engines/engine.py.224: train: epoch 43 | batch 1990/2304 [I 2022-10-27 02:48:45] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:02:54 [I 2022-10-27 02:48:45] ...trap/engines/engine.py.229: process: 0.54821 | load: 0.00244 [I 2022-10-27 02:48:45] ...trap/engines/engine.py.230: loss: 0.16352 [I 2022-10-27 02:48:45] ...optimizers/trijoint.py.151: total_norm: 17.803104 [I 2022-10-27 02:48:51] ...trap/engines/engine.py.224: train: epoch 43 | batch 2000/2304 [I 2022-10-27 02:48:51] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:02:49 [I 2022-10-27 02:48:51] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00245 [I 2022-10-27 02:48:51] ...trap/engines/engine.py.230: loss: 0.12812 [I 2022-10-27 02:48:51] ...optimizers/trijoint.py.151: total_norm: 10.784468 [I 2022-10-27 02:48:56] ...trap/engines/engine.py.224: train: epoch 43 | batch 2010/2304 [I 2022-10-27 02:48:56] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:02:44 [I 2022-10-27 02:48:56] ...trap/engines/engine.py.229: process: 0.55641 | load: 0.00243 [I 2022-10-27 02:48:56] ...trap/engines/engine.py.230: loss: 0.19211 [I 2022-10-27 02:48:56] ...optimizers/trijoint.py.151: total_norm: 19.875252 [I 2022-10-27 02:49:02] ...trap/engines/engine.py.224: train: epoch 43 | batch 2020/2304 [I 2022-10-27 02:49:02] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:37 [I 2022-10-27 02:49:02] ...trap/engines/engine.py.229: process: 0.54402 | load: 0.00264 [I 2022-10-27 02:49:02] ...trap/engines/engine.py.230: loss: 0.12827 [I 2022-10-27 02:49:02] ...optimizers/trijoint.py.151: total_norm: 11.262583 [I 2022-10-27 02:49:07] ...trap/engines/engine.py.224: train: epoch 43 | batch 2030/2304 [I 2022-10-27 02:49:07] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:32 [I 2022-10-27 02:49:07] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00250 [I 2022-10-27 02:49:07] ...trap/engines/engine.py.230: loss: 0.14511 [I 2022-10-27 02:49:07] ...optimizers/trijoint.py.151: total_norm: 30.856934 [I 2022-10-27 02:49:13] ...trap/engines/engine.py.224: train: epoch 43 | batch 2040/2304 [I 2022-10-27 02:49:13] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:30 [I 2022-10-27 02:49:13] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00272 [I 2022-10-27 02:49:13] ...trap/engines/engine.py.230: loss: 0.21898 [I 2022-10-27 02:49:13] ...optimizers/trijoint.py.151: total_norm: 14.408333 [I 2022-10-27 02:49:19] ...trap/engines/engine.py.224: train: epoch 43 | batch 2050/2304 [I 2022-10-27 02:49:19] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:20 [I 2022-10-27 02:49:19] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00249 [I 2022-10-27 02:49:19] ...trap/engines/engine.py.230: loss: 0.26162 [I 2022-10-27 02:49:19] ...optimizers/trijoint.py.151: total_norm: 22.921677 [I 2022-10-27 02:49:24] ...trap/engines/engine.py.224: train: epoch 43 | batch 2060/2304 [I 2022-10-27 02:49:24] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:15 [I 2022-10-27 02:49:24] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00248 [I 2022-10-27 02:49:24] ...trap/engines/engine.py.230: loss: 0.20427 [I 2022-10-27 02:49:24] ...optimizers/trijoint.py.151: total_norm: 21.702547 [I 2022-10-27 02:49:30] ...trap/engines/engine.py.224: train: epoch 43 | batch 2070/2304 [I 2022-10-27 02:49:30] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:10 [I 2022-10-27 02:49:30] ...trap/engines/engine.py.229: process: 0.54938 | load: 0.00248 [I 2022-10-27 02:49:30] ...trap/engines/engine.py.230: loss: 0.17523 [I 2022-10-27 02:49:30] ...optimizers/trijoint.py.151: total_norm: 17.089012 [I 2022-10-27 02:49:35] ...trap/engines/engine.py.224: train: epoch 43 | batch 2080/2304 [I 2022-10-27 02:49:35] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:03 [I 2022-10-27 02:49:35] ...trap/engines/engine.py.229: process: 0.54520 | load: 0.00249 [I 2022-10-27 02:49:35] ...trap/engines/engine.py.230: loss: 0.14102 [I 2022-10-27 02:49:35] ...optimizers/trijoint.py.151: total_norm: 11.646385 [I 2022-10-27 02:49:41] ...trap/engines/engine.py.224: train: epoch 43 | batch 2090/2304 [I 2022-10-27 02:49:41] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:01:58 [I 2022-10-27 02:49:41] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00252 [I 2022-10-27 02:49:41] ...trap/engines/engine.py.230: loss: 0.13765 [I 2022-10-27 02:49:41] ...optimizers/trijoint.py.151: total_norm: 14.354198 [I 2022-10-27 02:49:47] ...trap/engines/engine.py.224: train: epoch 43 | batch 2100/2304 [I 2022-10-27 02:49:47] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:01:52 [I 2022-10-27 02:49:47] ...trap/engines/engine.py.229: process: 0.54696 | load: 0.00245 [I 2022-10-27 02:49:47] ...trap/engines/engine.py.230: loss: 0.14382 [I 2022-10-27 02:49:47] ...optimizers/trijoint.py.151: total_norm: 27.216793 [I 2022-10-27 02:49:52] ...trap/engines/engine.py.224: train: epoch 43 | batch 2110/2304 [I 2022-10-27 02:49:52] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:01:47 [I 2022-10-27 02:49:52] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00251 [I 2022-10-27 02:49:52] ...trap/engines/engine.py.230: loss: 0.23159 [I 2022-10-27 02:49:52] ...optimizers/trijoint.py.151: total_norm: 18.025450 [I 2022-10-27 02:49:58] ...trap/engines/engine.py.224: train: epoch 43 | batch 2120/2304 [I 2022-10-27 02:49:58] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:01:41 [I 2022-10-27 02:49:58] ...trap/engines/engine.py.229: process: 0.54524 | load: 0.00246 [I 2022-10-27 02:49:58] ...trap/engines/engine.py.230: loss: 0.11564 [I 2022-10-27 02:49:58] ...optimizers/trijoint.py.151: total_norm: 11.743950 [I 2022-10-27 02:50:03] ...trap/engines/engine.py.224: train: epoch 43 | batch 2130/2304 [I 2022-10-27 02:50:03] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:37 [I 2022-10-27 02:50:03] ...trap/engines/engine.py.229: process: 0.57011 | load: 0.00244 [I 2022-10-27 02:50:03] ...trap/engines/engine.py.230: loss: 0.17636 [I 2022-10-27 02:50:03] ...optimizers/trijoint.py.151: total_norm: 19.378822 [I 2022-10-27 02:50:09] ...trap/engines/engine.py.224: train: epoch 43 | batch 2140/2304 [I 2022-10-27 02:50:09] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:30 [I 2022-10-27 02:50:09] ...trap/engines/engine.py.229: process: 0.56966 | load: 0.00244 [I 2022-10-27 02:50:09] ...trap/engines/engine.py.230: loss: 0.18822 [I 2022-10-27 02:50:09] ...optimizers/trijoint.py.151: total_norm: 11.148664 [I 2022-10-27 02:50:14] ...trap/engines/engine.py.224: train: epoch 43 | batch 2150/2304 [I 2022-10-27 02:50:14] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:26 [I 2022-10-27 02:50:14] ...trap/engines/engine.py.229: process: 0.55488 | load: 0.00250 [I 2022-10-27 02:50:14] ...trap/engines/engine.py.230: loss: 0.13107 [I 2022-10-27 02:50:14] ...optimizers/trijoint.py.151: total_norm: 19.690201 [I 2022-10-27 02:50:20] ...trap/engines/engine.py.224: train: epoch 43 | batch 2160/2304 [I 2022-10-27 02:50:20] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:22 [I 2022-10-27 02:50:20] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00274 [I 2022-10-27 02:50:20] ...trap/engines/engine.py.230: loss: 0.20194 [I 2022-10-27 02:50:20] ...optimizers/trijoint.py.151: total_norm: 21.931606 [I 2022-10-27 02:50:26] ...trap/engines/engine.py.224: train: epoch 43 | batch 2170/2304 [I 2022-10-27 02:50:26] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:14 [I 2022-10-27 02:50:26] ...trap/engines/engine.py.229: process: 0.56155 | load: 0.00253 [I 2022-10-27 02:50:26] ...trap/engines/engine.py.230: loss: 0.10075 [I 2022-10-27 02:50:26] ...optimizers/trijoint.py.151: total_norm: 8.659497 [I 2022-10-27 02:50:31] ...trap/engines/engine.py.224: train: epoch 43 | batch 2180/2304 [I 2022-10-27 02:50:31] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:09 [I 2022-10-27 02:50:31] ...trap/engines/engine.py.229: process: 0.57283 | load: 0.00250 [I 2022-10-27 02:50:31] ...trap/engines/engine.py.230: loss: 0.17088 [I 2022-10-27 02:50:31] ...optimizers/trijoint.py.151: total_norm: 15.571127 [I 2022-10-27 02:50:37] ...trap/engines/engine.py.224: train: epoch 43 | batch 2190/2304 [I 2022-10-27 02:50:37] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:03 [I 2022-10-27 02:50:37] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00247 [I 2022-10-27 02:50:37] ...trap/engines/engine.py.230: loss: 0.11772 [I 2022-10-27 02:50:37] ...optimizers/trijoint.py.151: total_norm: 15.676153 [I 2022-10-27 02:50:42] ...trap/engines/engine.py.224: train: epoch 43 | batch 2200/2304 [I 2022-10-27 02:50:42] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:00:57 [I 2022-10-27 02:50:42] ...trap/engines/engine.py.229: process: 0.54830 | load: 0.00261 [I 2022-10-27 02:50:42] ...trap/engines/engine.py.230: loss: 0.25048 [I 2022-10-27 02:50:42] ...optimizers/trijoint.py.151: total_norm: 16.152260 [I 2022-10-27 02:50:48] ...trap/engines/engine.py.224: train: epoch 43 | batch 2210/2304 [I 2022-10-27 02:50:48] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:00:52 [I 2022-10-27 02:50:48] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00246 [I 2022-10-27 02:50:48] ...trap/engines/engine.py.230: loss: 0.13686 [I 2022-10-27 02:50:48] ...optimizers/trijoint.py.151: total_norm: 10.337017 [I 2022-10-27 02:50:53] ...trap/engines/engine.py.224: train: epoch 43 | batch 2220/2304 [I 2022-10-27 02:50:53] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:00:46 [I 2022-10-27 02:50:53] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00247 [I 2022-10-27 02:50:53] ...trap/engines/engine.py.230: loss: 0.17445 [I 2022-10-27 02:50:53] ...optimizers/trijoint.py.151: total_norm: 18.859415 [I 2022-10-27 02:50:59] ...trap/engines/engine.py.224: train: epoch 43 | batch 2230/2304 [I 2022-10-27 02:50:59] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:00:41 [I 2022-10-27 02:50:59] ...trap/engines/engine.py.229: process: 0.54431 | load: 0.00242 [I 2022-10-27 02:50:59] ...trap/engines/engine.py.230: loss: 0.21835 [I 2022-10-27 02:50:59] ...optimizers/trijoint.py.151: total_norm: 24.947935 [I 2022-10-27 02:51:05] ...trap/engines/engine.py.224: train: epoch 43 | batch 2240/2304 [I 2022-10-27 02:51:05] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:35 [I 2022-10-27 02:51:05] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00244 [I 2022-10-27 02:51:05] ...trap/engines/engine.py.230: loss: 0.13448 [I 2022-10-27 02:51:05] ...optimizers/trijoint.py.151: total_norm: 10.048786 [I 2022-10-27 02:51:10] ...trap/engines/engine.py.224: train: epoch 43 | batch 2250/2304 [I 2022-10-27 02:51:10] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:30 [I 2022-10-27 02:51:10] ...trap/engines/engine.py.229: process: 0.56005 | load: 0.00250 [I 2022-10-27 02:51:10] ...trap/engines/engine.py.230: loss: 0.16008 [I 2022-10-27 02:51:10] ...optimizers/trijoint.py.151: total_norm: 18.872229 [I 2022-10-27 02:51:16] ...trap/engines/engine.py.224: train: epoch 43 | batch 2260/2304 [I 2022-10-27 02:51:16] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:24 [I 2022-10-27 02:51:16] ...trap/engines/engine.py.229: process: 0.55977 | load: 0.00246 [I 2022-10-27 02:51:16] ...trap/engines/engine.py.230: loss: 0.12492 [I 2022-10-27 02:51:16] ...optimizers/trijoint.py.151: total_norm: 10.860131 [I 2022-10-27 02:51:21] ...trap/engines/engine.py.224: train: epoch 43 | batch 2270/2304 [I 2022-10-27 02:51:21] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:18 [I 2022-10-27 02:51:21] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00247 [I 2022-10-27 02:51:21] ...trap/engines/engine.py.230: loss: 0.20372 [I 2022-10-27 02:51:21] ...optimizers/trijoint.py.151: total_norm: 22.743259 [I 2022-10-27 02:51:27] ...trap/engines/engine.py.224: train: epoch 43 | batch 2280/2304 [I 2022-10-27 02:51:27] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:13 [I 2022-10-27 02:51:27] ...trap/engines/engine.py.229: process: 0.54271 | load: 0.00247 [I 2022-10-27 02:51:27] ...trap/engines/engine.py.230: loss: 0.18377 [I 2022-10-27 02:51:27] ...optimizers/trijoint.py.151: total_norm: 15.717462 [I 2022-10-27 02:51:32] ...trap/engines/engine.py.224: train: epoch 43 | batch 2290/2304 [I 2022-10-27 02:51:32] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:07 [I 2022-10-27 02:51:32] ...trap/engines/engine.py.229: process: 0.55140 | load: 0.00246 [I 2022-10-27 02:51:32] ...trap/engines/engine.py.230: loss: 0.16379 [I 2022-10-27 02:51:32] ...optimizers/trijoint.py.151: total_norm: 21.088760 [I 2022-10-27 02:51:38] ...trap/engines/engine.py.224: train: epoch 43 | batch 2300/2304 [I 2022-10-27 02:51:38] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:02 [I 2022-10-27 02:51:38] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00014 [I 2022-10-27 02:51:38] ...trap/engines/engine.py.230: loss: 0.17283 [I 2022-10-27 02:51:38] ...optimizers/trijoint.py.151: total_norm: 21.051126 [I 2022-10-27 02:51:40] ...trap/engines/engine.py.224: train: epoch 43 | batch 2304/2304 [I 2022-10-27 02:51:40] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:00 [I 2022-10-27 02:51:40] ...trap/engines/engine.py.229: process: 0.55047 | load: 0.00015 [I 2022-10-27 02:51:40] ...trap/engines/engine.py.230: loss: 0.17041 [I 2022-10-27 02:51:40] ...optimizers/trijoint.py.151: total_norm: 13.125138 [S 2022-10-27 02:51:41] ...trap/engines/engine.py.129: train_epoch.epoch: 43 [S 2022-10-27 02:51:41] ...trap/engines/engine.py.129: train_epoch.loss: 0.16818271534044676 [I 2022-10-27 02:51:41] ...s/criterions/triplet.py.84: margin = 0.2700000000000002 [I 2022-10-27 02:51:42] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 43 [I 2022-10-27 02:51:42] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 02:51:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 02:51:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 02:51:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 02:51:45] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 02:51:45] ...trap/engines/engine.py.321: eval: epoch 43 | batch 0/510 [I 2022-10-27 02:51:45] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:30:20 [I 2022-10-27 02:51:45] ...trap/engines/engine.py.326: process: 3.57008 | load: 3.15748 [I 2022-10-27 02:51:49] ...trap/engines/engine.py.321: eval: epoch 43 | batch 10/510 [I 2022-10-27 02:51:49] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:39 [I 2022-10-27 02:51:49] ...trap/engines/engine.py.326: process: 0.33323 | load: 0.00017 [I 2022-10-27 02:51:52] ...trap/engines/engine.py.321: eval: epoch 43 | batch 20/510 [I 2022-10-27 02:51:52] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:02 [I 2022-10-27 02:51:52] ...trap/engines/engine.py.326: process: 0.32433 | load: 0.00019 [I 2022-10-27 02:51:55] ...trap/engines/engine.py.321: eval: epoch 43 | batch 30/510 [I 2022-10-27 02:51:55] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:45 [I 2022-10-27 02:51:55] ...trap/engines/engine.py.326: process: 0.35429 | load: 0.00021 [I 2022-10-27 02:51:59] ...trap/engines/engine.py.321: eval: epoch 43 | batch 40/510 [I 2022-10-27 02:51:59] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:38 [I 2022-10-27 02:51:59] ...trap/engines/engine.py.326: process: 0.33070 | load: 0.00022 [I 2022-10-27 02:52:02] ...trap/engines/engine.py.321: eval: epoch 43 | batch 50/510 [I 2022-10-27 02:52:02] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:35 [I 2022-10-27 02:52:02] ...trap/engines/engine.py.326: process: 0.32564 | load: 0.00020 [I 2022-10-27 02:52:06] ...trap/engines/engine.py.321: eval: epoch 43 | batch 60/510 [I 2022-10-27 02:52:06] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:31 [I 2022-10-27 02:52:06] ...trap/engines/engine.py.326: process: 0.35475 | load: 0.00018 [I 2022-10-27 02:52:09] ...trap/engines/engine.py.321: eval: epoch 43 | batch 70/510 [I 2022-10-27 02:52:09] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:26 [I 2022-10-27 02:52:09] ...trap/engines/engine.py.326: process: 0.33244 | load: 0.00017 [I 2022-10-27 02:52:12] ...trap/engines/engine.py.321: eval: epoch 43 | batch 80/510 [I 2022-10-27 02:52:12] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-27 02:52:12] ...trap/engines/engine.py.326: process: 0.33328 | load: 0.00020 [I 2022-10-27 02:52:16] ...trap/engines/engine.py.321: eval: epoch 43 | batch 90/510 [I 2022-10-27 02:52:16] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-27 02:52:16] ...trap/engines/engine.py.326: process: 0.34862 | load: 0.00018 [I 2022-10-27 02:52:19] ...trap/engines/engine.py.321: eval: epoch 43 | batch 100/510 [I 2022-10-27 02:52:19] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:12 [I 2022-10-27 02:52:19] ...trap/engines/engine.py.326: process: 0.29286 | load: 0.00028 [I 2022-10-27 02:52:22] ...trap/engines/engine.py.321: eval: epoch 43 | batch 110/510 [I 2022-10-27 02:52:22] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-27 02:52:22] ...trap/engines/engine.py.326: process: 0.27907 | load: 0.00029 [I 2022-10-27 02:52:25] ...trap/engines/engine.py.321: eval: epoch 43 | batch 120/510 [I 2022-10-27 02:52:25] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:53 [I 2022-10-27 02:52:25] ...trap/engines/engine.py.326: process: 0.28810 | load: 0.00024 [I 2022-10-27 02:52:28] ...trap/engines/engine.py.321: eval: epoch 43 | batch 130/510 [I 2022-10-27 02:52:28] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-27 02:52:28] ...trap/engines/engine.py.326: process: 0.28660 | load: 0.00021 [I 2022-10-27 02:52:31] ...trap/engines/engine.py.321: eval: epoch 43 | batch 140/510 [I 2022-10-27 02:52:31] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-27 02:52:31] ...trap/engines/engine.py.326: process: 0.27856 | load: 0.00016 [I 2022-10-27 02:52:33] ...trap/engines/engine.py.321: eval: epoch 43 | batch 150/510 [I 2022-10-27 02:52:33] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-27 02:52:33] ...trap/engines/engine.py.326: process: 0.28622 | load: 0.00027 [I 2022-10-27 02:52:36] ...trap/engines/engine.py.321: eval: epoch 43 | batch 160/510 [I 2022-10-27 02:52:36] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-27 02:52:36] ...trap/engines/engine.py.326: process: 0.28711 | load: 0.00024 [I 2022-10-27 02:52:39] ...trap/engines/engine.py.321: eval: epoch 43 | batch 170/510 [I 2022-10-27 02:52:39] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-27 02:52:39] ...trap/engines/engine.py.326: process: 0.28884 | load: 0.00020 [I 2022-10-27 02:52:42] ...trap/engines/engine.py.321: eval: epoch 43 | batch 180/510 [I 2022-10-27 02:52:42] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 02:52:42] ...trap/engines/engine.py.326: process: 0.29306 | load: 0.00025 [I 2022-10-27 02:52:45] ...trap/engines/engine.py.321: eval: epoch 43 | batch 190/510 [I 2022-10-27 02:52:45] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 02:52:45] ...trap/engines/engine.py.326: process: 0.29541 | load: 0.00018 [I 2022-10-27 02:52:48] ...trap/engines/engine.py.321: eval: epoch 43 | batch 200/510 [I 2022-10-27 02:52:48] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 02:52:48] ...trap/engines/engine.py.326: process: 0.28767 | load: 0.00037 [I 2022-10-27 02:52:51] ...trap/engines/engine.py.321: eval: epoch 43 | batch 210/510 [I 2022-10-27 02:52:51] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-27 02:52:51] ...trap/engines/engine.py.326: process: 0.29456 | load: 0.00014 [I 2022-10-27 02:52:54] ...trap/engines/engine.py.321: eval: epoch 43 | batch 220/510 [I 2022-10-27 02:52:54] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:24 [I 2022-10-27 02:52:54] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00019 [I 2022-10-27 02:52:57] ...trap/engines/engine.py.321: eval: epoch 43 | batch 230/510 [I 2022-10-27 02:52:57] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:22 [I 2022-10-27 02:52:57] ...trap/engines/engine.py.326: process: 0.30294 | load: 0.00014 [I 2022-10-27 02:53:00] ...trap/engines/engine.py.321: eval: epoch 43 | batch 240/510 [I 2022-10-27 02:53:00] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-27 02:53:00] ...trap/engines/engine.py.326: process: 0.28321 | load: 0.00022 [I 2022-10-27 02:53:02] ...trap/engines/engine.py.321: eval: epoch 43 | batch 250/510 [I 2022-10-27 02:53:02] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-27 02:53:02] ...trap/engines/engine.py.326: process: 0.28685 | load: 0.00020 [I 2022-10-27 02:53:05] ...trap/engines/engine.py.321: eval: epoch 43 | batch 260/510 [I 2022-10-27 02:53:05] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 02:53:05] ...trap/engines/engine.py.326: process: 0.28137 | load: 0.00028 [I 2022-10-27 02:53:08] ...trap/engines/engine.py.321: eval: epoch 43 | batch 270/510 [I 2022-10-27 02:53:08] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-27 02:53:08] ...trap/engines/engine.py.326: process: 0.28103 | load: 0.00020 [I 2022-10-27 02:53:11] ...trap/engines/engine.py.321: eval: epoch 43 | batch 280/510 [I 2022-10-27 02:53:11] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:05 [I 2022-10-27 02:53:11] ...trap/engines/engine.py.326: process: 0.28881 | load: 0.00025 [I 2022-10-27 02:53:14] ...trap/engines/engine.py.321: eval: epoch 43 | batch 290/510 [I 2022-10-27 02:53:14] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 02:53:14] ...trap/engines/engine.py.326: process: 0.28740 | load: 0.00016 [I 2022-10-27 02:53:17] ...trap/engines/engine.py.321: eval: epoch 43 | batch 300/510 [I 2022-10-27 02:53:17] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-27 02:53:17] ...trap/engines/engine.py.326: process: 0.28998 | load: 0.00018 [I 2022-10-27 02:53:20] ...trap/engines/engine.py.321: eval: epoch 43 | batch 310/510 [I 2022-10-27 02:53:20] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 02:53:20] ...trap/engines/engine.py.326: process: 0.29056 | load: 0.00014 [I 2022-10-27 02:53:22] ...trap/engines/engine.py.321: eval: epoch 43 | batch 320/510 [I 2022-10-27 02:53:22] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 02:53:22] ...trap/engines/engine.py.326: process: 0.28713 | load: 0.00018 [I 2022-10-27 02:53:25] ...trap/engines/engine.py.321: eval: epoch 43 | batch 330/510 [I 2022-10-27 02:53:25] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 02:53:25] ...trap/engines/engine.py.326: process: 0.29039 | load: 0.00016 [I 2022-10-27 02:53:28] ...trap/engines/engine.py.321: eval: epoch 43 | batch 340/510 [I 2022-10-27 02:53:28] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-27 02:53:28] ...trap/engines/engine.py.326: process: 0.28528 | load: 0.00019 [I 2022-10-27 02:53:31] ...trap/engines/engine.py.321: eval: epoch 43 | batch 350/510 [I 2022-10-27 02:53:31] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-27 02:53:31] ...trap/engines/engine.py.326: process: 0.28449 | load: 0.00019 [I 2022-10-27 02:53:34] ...trap/engines/engine.py.321: eval: epoch 43 | batch 360/510 [I 2022-10-27 02:53:34] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 02:53:34] ...trap/engines/engine.py.326: process: 0.28191 | load: 0.00015 [I 2022-10-27 02:53:37] ...trap/engines/engine.py.321: eval: epoch 43 | batch 370/510 [I 2022-10-27 02:53:37] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 02:53:37] ...trap/engines/engine.py.326: process: 0.28443 | load: 0.00015 [I 2022-10-27 02:53:40] ...trap/engines/engine.py.321: eval: epoch 43 | batch 380/510 [I 2022-10-27 02:53:40] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-27 02:53:40] ...trap/engines/engine.py.326: process: 0.28448 | load: 0.00026 [I 2022-10-27 02:53:43] ...trap/engines/engine.py.321: eval: epoch 43 | batch 390/510 [I 2022-10-27 02:53:43] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-27 02:53:43] ...trap/engines/engine.py.326: process: 0.28768 | load: 0.00016 [I 2022-10-27 02:53:46] ...trap/engines/engine.py.321: eval: epoch 43 | batch 400/510 [I 2022-10-27 02:53:46] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:32 [I 2022-10-27 02:53:46] ...trap/engines/engine.py.326: process: 0.31143 | load: 0.00025 [I 2022-10-27 02:53:48] ...trap/engines/engine.py.321: eval: epoch 43 | batch 410/510 [I 2022-10-27 02:53:48] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:29 [I 2022-10-27 02:53:48] ...trap/engines/engine.py.326: process: 0.29245 | load: 0.00016 [I 2022-10-27 02:53:51] ...trap/engines/engine.py.321: eval: epoch 43 | batch 420/510 [I 2022-10-27 02:53:51] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:26 [I 2022-10-27 02:53:51] ...trap/engines/engine.py.326: process: 0.28307 | load: 0.00024 [I 2022-10-27 02:53:54] ...trap/engines/engine.py.321: eval: epoch 43 | batch 430/510 [I 2022-10-27 02:53:54] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-27 02:53:54] ...trap/engines/engine.py.326: process: 0.28878 | load: 0.00015 [I 2022-10-27 02:53:57] ...trap/engines/engine.py.321: eval: epoch 43 | batch 440/510 [I 2022-10-27 02:53:57] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 02:53:57] ...trap/engines/engine.py.326: process: 0.28742 | load: 0.00016 [I 2022-10-27 02:54:00] ...trap/engines/engine.py.321: eval: epoch 43 | batch 450/510 [I 2022-10-27 02:54:00] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 02:54:00] ...trap/engines/engine.py.326: process: 0.29530 | load: 0.00017 [I 2022-10-27 02:54:03] ...trap/engines/engine.py.321: eval: epoch 43 | batch 460/510 [I 2022-10-27 02:54:03] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 02:54:03] ...trap/engines/engine.py.326: process: 0.28382 | load: 0.00022 [I 2022-10-27 02:54:06] ...trap/engines/engine.py.321: eval: epoch 43 | batch 470/510 [I 2022-10-27 02:54:06] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-27 02:54:06] ...trap/engines/engine.py.326: process: 0.28655 | load: 0.00016 [I 2022-10-27 02:54:09] ...trap/engines/engine.py.321: eval: epoch 43 | batch 480/510 [I 2022-10-27 02:54:09] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 02:54:09] ...trap/engines/engine.py.326: process: 0.28356 | load: 0.00016 [I 2022-10-27 02:54:11] ...trap/engines/engine.py.321: eval: epoch 43 | batch 490/510 [I 2022-10-27 02:54:11] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 02:54:11] ...trap/engines/engine.py.326: process: 0.28937 | load: 0.00015 [I 2022-10-27 02:54:14] ...trap/engines/engine.py.321: eval: epoch 43 | batch 500/510 [I 2022-10-27 02:54:14] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 02:54:14] ...trap/engines/engine.py.326: process: 0.28513 | load: 0.00020 [I 2022-10-27 02:54:17] ...trap/engines/engine.py.321: eval: epoch 43 | batch 510/510 [I 2022-10-27 02:54:17] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 02:54:17] ...trap/engines/engine.py.326: process: 0.28951 | load: 0.00011 [S 2022-10-27 02:54:18] ...trap/engines/engine.py.132: eval_epoch.epoch: 43 [I 2022-10-27 02:54:18] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.687200 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.893600 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935500 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.676000 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.891700 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.930300 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017848 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007645 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006054 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016100 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006512 [S 2022-10-27 02:54:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007564 [I 2022-10-27 02:54:23] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 02:54:23] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 02:54:23] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 02:54:23] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 02:54:23] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 02:54:25] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 02:54:27] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 02:54:33] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 02:54:33] ...trap/engines/engine.py.163: Training model on trainset for epoch 44 [I 2022-10-27 02:54:33] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 02:54:36] ...trap/engines/engine.py.224: train: epoch 44 | batch 0/2304 [I 2022-10-27 02:54:36] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:06 [I 2022-10-27 02:54:36] ...trap/engines/engine.py.229: process: 2.63321 | load: 1.96634 [I 2022-10-27 02:54:36] ...trap/engines/engine.py.230: loss: 0.15514 [I 2022-10-27 02:54:36] ...optimizers/trijoint.py.151: total_norm: 17.738348 [I 2022-10-27 02:54:42] ...trap/engines/engine.py.224: train: epoch 44 | batch 10/2304 [I 2022-10-27 02:54:42] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:50 [I 2022-10-27 02:54:42] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00387 [I 2022-10-27 02:54:42] ...trap/engines/engine.py.230: loss: 0.12739 [I 2022-10-27 02:54:42] ...optimizers/trijoint.py.151: total_norm: 9.539544 [I 2022-10-27 02:54:47] ...trap/engines/engine.py.224: train: epoch 44 | batch 20/2304 [I 2022-10-27 02:54:47] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:21:52 [I 2022-10-27 02:54:47] ...trap/engines/engine.py.229: process: 0.54894 | load: 0.00294 [I 2022-10-27 02:54:47] ...trap/engines/engine.py.230: loss: 0.19481 [I 2022-10-27 02:54:47] ...optimizers/trijoint.py.151: total_norm: 12.832799 [I 2022-10-27 02:54:53] ...trap/engines/engine.py.224: train: epoch 44 | batch 30/2304 [I 2022-10-27 02:54:53] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:11 [I 2022-10-27 02:54:53] ...trap/engines/engine.py.229: process: 0.55275 | load: 0.00265 [I 2022-10-27 02:54:53] ...trap/engines/engine.py.230: loss: 0.12763 [I 2022-10-27 02:54:53] ...optimizers/trijoint.py.151: total_norm: 14.826558 [I 2022-10-27 02:54:58] ...trap/engines/engine.py.224: train: epoch 44 | batch 40/2304 [I 2022-10-27 02:54:58] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:21:02 [I 2022-10-27 02:54:58] ...trap/engines/engine.py.229: process: 0.55460 | load: 0.00321 [I 2022-10-27 02:54:58] ...trap/engines/engine.py.230: loss: 0.20741 [I 2022-10-27 02:54:58] ...optimizers/trijoint.py.151: total_norm: 18.337166 [I 2022-10-27 02:55:04] ...trap/engines/engine.py.224: train: epoch 44 | batch 50/2304 [I 2022-10-27 02:55:04] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:51 [I 2022-10-27 02:55:04] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00268 [I 2022-10-27 02:55:04] ...trap/engines/engine.py.230: loss: 0.11123 [I 2022-10-27 02:55:04] ...optimizers/trijoint.py.151: total_norm: 12.627272 [I 2022-10-27 02:55:09] ...trap/engines/engine.py.224: train: epoch 44 | batch 60/2304 [I 2022-10-27 02:55:09] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:48 [I 2022-10-27 02:55:09] ...trap/engines/engine.py.229: process: 0.55047 | load: 0.00266 [I 2022-10-27 02:55:09] ...trap/engines/engine.py.230: loss: 0.13101 [I 2022-10-27 02:55:09] ...optimizers/trijoint.py.151: total_norm: 13.068885 [I 2022-10-27 02:55:15] ...trap/engines/engine.py.224: train: epoch 44 | batch 70/2304 [I 2022-10-27 02:55:15] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:51 [I 2022-10-27 02:55:15] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00295 [I 2022-10-27 02:55:15] ...trap/engines/engine.py.230: loss: 0.11384 [I 2022-10-27 02:55:15] ...optimizers/trijoint.py.151: total_norm: 15.563994 [I 2022-10-27 02:55:21] ...trap/engines/engine.py.224: train: epoch 44 | batch 80/2304 [I 2022-10-27 02:55:21] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:40 [I 2022-10-27 02:55:21] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00256 [I 2022-10-27 02:55:21] ...trap/engines/engine.py.230: loss: 0.18142 [I 2022-10-27 02:55:21] ...optimizers/trijoint.py.151: total_norm: 24.812675 [I 2022-10-27 02:55:26] ...trap/engines/engine.py.224: train: epoch 44 | batch 90/2304 [I 2022-10-27 02:55:26] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:41 [I 2022-10-27 02:55:26] ...trap/engines/engine.py.229: process: 0.57245 | load: 0.00255 [I 2022-10-27 02:55:26] ...trap/engines/engine.py.230: loss: 0.16488 [I 2022-10-27 02:55:26] ...optimizers/trijoint.py.151: total_norm: 16.041298 [I 2022-10-27 02:55:32] ...trap/engines/engine.py.224: train: epoch 44 | batch 100/2304 [I 2022-10-27 02:55:32] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:38 [I 2022-10-27 02:55:32] ...trap/engines/engine.py.229: process: 0.57549 | load: 0.00263 [I 2022-10-27 02:55:32] ...trap/engines/engine.py.230: loss: 0.13413 [I 2022-10-27 02:55:32] ...optimizers/trijoint.py.151: total_norm: 14.035535 [I 2022-10-27 02:55:38] ...trap/engines/engine.py.224: train: epoch 44 | batch 110/2304 [I 2022-10-27 02:55:38] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:50 [I 2022-10-27 02:55:38] ...trap/engines/engine.py.229: process: 0.59322 | load: 0.00273 [I 2022-10-27 02:55:38] ...trap/engines/engine.py.230: loss: 0.11724 [I 2022-10-27 02:55:38] ...optimizers/trijoint.py.151: total_norm: 9.212236 [I 2022-10-27 02:55:43] ...trap/engines/engine.py.224: train: epoch 44 | batch 120/2304 [I 2022-10-27 02:55:43] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:29 [I 2022-10-27 02:55:43] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00273 [I 2022-10-27 02:55:43] ...trap/engines/engine.py.230: loss: 0.13211 [I 2022-10-27 02:55:43] ...optimizers/trijoint.py.151: total_norm: 16.987288 [I 2022-10-27 02:55:49] ...trap/engines/engine.py.224: train: epoch 44 | batch 130/2304 [I 2022-10-27 02:55:49] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:24 [I 2022-10-27 02:55:49] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00349 [I 2022-10-27 02:55:49] ...trap/engines/engine.py.230: loss: 0.15776 [I 2022-10-27 02:55:49] ...optimizers/trijoint.py.151: total_norm: 24.510891 [I 2022-10-27 02:55:54] ...trap/engines/engine.py.224: train: epoch 44 | batch 140/2304 [I 2022-10-27 02:55:54] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:13 [I 2022-10-27 02:55:54] ...trap/engines/engine.py.229: process: 0.55667 | load: 0.00250 [I 2022-10-27 02:55:54] ...trap/engines/engine.py.230: loss: 0.12685 [I 2022-10-27 02:55:54] ...optimizers/trijoint.py.151: total_norm: 26.239998 [I 2022-10-27 02:56:00] ...trap/engines/engine.py.224: train: epoch 44 | batch 150/2304 [I 2022-10-27 02:56:00] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:03 [I 2022-10-27 02:56:00] ...trap/engines/engine.py.229: process: 0.56350 | load: 0.00248 [I 2022-10-27 02:56:00] ...trap/engines/engine.py.230: loss: 0.18299 [I 2022-10-27 02:56:00] ...optimizers/trijoint.py.151: total_norm: 38.007816 [I 2022-10-27 02:56:06] ...trap/engines/engine.py.224: train: epoch 44 | batch 160/2304 [I 2022-10-27 02:56:06] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:51 [I 2022-10-27 02:56:06] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00250 [I 2022-10-27 02:56:06] ...trap/engines/engine.py.230: loss: 0.15318 [I 2022-10-27 02:56:06] ...optimizers/trijoint.py.151: total_norm: 23.292940 [I 2022-10-27 02:56:11] ...trap/engines/engine.py.224: train: epoch 44 | batch 170/2304 [I 2022-10-27 02:56:11] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:59 [I 2022-10-27 02:56:11] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00280 [I 2022-10-27 02:56:11] ...trap/engines/engine.py.230: loss: 0.17027 [I 2022-10-27 02:56:11] ...optimizers/trijoint.py.151: total_norm: 40.338329 [I 2022-10-27 02:56:17] ...trap/engines/engine.py.224: train: epoch 44 | batch 180/2304 [I 2022-10-27 02:56:17] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:49 [I 2022-10-27 02:56:17] ...trap/engines/engine.py.229: process: 0.56869 | load: 0.00251 [I 2022-10-27 02:56:17] ...trap/engines/engine.py.230: loss: 0.18522 [I 2022-10-27 02:56:17] ...optimizers/trijoint.py.151: total_norm: 17.518358 [I 2022-10-27 02:56:22] ...trap/engines/engine.py.224: train: epoch 44 | batch 190/2304 [I 2022-10-27 02:56:22] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:40 [I 2022-10-27 02:56:22] ...trap/engines/engine.py.229: process: 0.56397 | load: 0.00286 [I 2022-10-27 02:56:22] ...trap/engines/engine.py.230: loss: 0.20866 [I 2022-10-27 02:56:22] ...optimizers/trijoint.py.151: total_norm: 23.654348 [I 2022-10-27 02:56:28] ...trap/engines/engine.py.224: train: epoch 44 | batch 200/2304 [I 2022-10-27 02:56:28] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:20:27 [I 2022-10-27 02:56:28] ...trap/engines/engine.py.229: process: 0.61778 | load: 0.01016 [I 2022-10-27 02:56:28] ...trap/engines/engine.py.230: loss: 0.15905 [I 2022-10-27 02:56:28] ...optimizers/trijoint.py.151: total_norm: 16.440319 [I 2022-10-27 02:56:34] ...trap/engines/engine.py.224: train: epoch 44 | batch 210/2304 [I 2022-10-27 02:56:34] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:27 [I 2022-10-27 02:56:34] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00267 [I 2022-10-27 02:56:34] ...trap/engines/engine.py.230: loss: 0.14350 [I 2022-10-27 02:56:34] ...optimizers/trijoint.py.151: total_norm: 18.026796 [I 2022-10-27 02:56:39] ...trap/engines/engine.py.224: train: epoch 44 | batch 220/2304 [I 2022-10-27 02:56:39] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:22 [I 2022-10-27 02:56:39] ...trap/engines/engine.py.229: process: 0.56463 | load: 0.00244 [I 2022-10-27 02:56:39] ...trap/engines/engine.py.230: loss: 0.19755 [I 2022-10-27 02:56:39] ...optimizers/trijoint.py.151: total_norm: 11.779814 [I 2022-10-27 02:56:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 230/2304 [I 2022-10-27 02:56:45] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:18 [I 2022-10-27 02:56:45] ...trap/engines/engine.py.229: process: 0.57458 | load: 0.00318 [I 2022-10-27 02:56:45] ...trap/engines/engine.py.230: loss: 0.16797 [I 2022-10-27 02:56:45] ...optimizers/trijoint.py.151: total_norm: 8.733174 [I 2022-10-27 02:56:51] ...trap/engines/engine.py.224: train: epoch 44 | batch 240/2304 [I 2022-10-27 02:56:51] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:18 [I 2022-10-27 02:56:51] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00259 [I 2022-10-27 02:56:51] ...trap/engines/engine.py.230: loss: 0.13688 [I 2022-10-27 02:56:51] ...optimizers/trijoint.py.151: total_norm: 8.748451 [I 2022-10-27 02:56:56] ...trap/engines/engine.py.224: train: epoch 44 | batch 250/2304 [I 2022-10-27 02:56:56] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:15 [I 2022-10-27 02:56:56] ...trap/engines/engine.py.229: process: 0.55768 | load: 0.00282 [I 2022-10-27 02:56:56] ...trap/engines/engine.py.230: loss: 0.15595 [I 2022-10-27 02:56:56] ...optimizers/trijoint.py.151: total_norm: 20.799498 [I 2022-10-27 02:57:02] ...trap/engines/engine.py.224: train: epoch 44 | batch 260/2304 [I 2022-10-27 02:57:02] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:57 [I 2022-10-27 02:57:02] ...trap/engines/engine.py.229: process: 0.56507 | load: 0.00271 [I 2022-10-27 02:57:02] ...trap/engines/engine.py.230: loss: 0.12295 [I 2022-10-27 02:57:02] ...optimizers/trijoint.py.151: total_norm: 13.852406 [I 2022-10-27 02:57:07] ...trap/engines/engine.py.224: train: epoch 44 | batch 270/2304 [I 2022-10-27 02:57:07] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:04 [I 2022-10-27 02:57:07] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00261 [I 2022-10-27 02:57:07] ...trap/engines/engine.py.230: loss: 0.14395 [I 2022-10-27 02:57:07] ...optimizers/trijoint.py.151: total_norm: 22.681398 [I 2022-10-27 02:57:13] ...trap/engines/engine.py.224: train: epoch 44 | batch 280/2304 [I 2022-10-27 02:57:13] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:44 [I 2022-10-27 02:57:13] ...trap/engines/engine.py.229: process: 0.55114 | load: 0.00256 [I 2022-10-27 02:57:13] ...trap/engines/engine.py.230: loss: 0.17665 [I 2022-10-27 02:57:13] ...optimizers/trijoint.py.151: total_norm: 41.609310 [I 2022-10-27 02:57:19] ...trap/engines/engine.py.224: train: epoch 44 | batch 290/2304 [I 2022-10-27 02:57:19] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:37 [I 2022-10-27 02:57:19] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00248 [I 2022-10-27 02:57:19] ...trap/engines/engine.py.230: loss: 0.19723 [I 2022-10-27 02:57:19] ...optimizers/trijoint.py.151: total_norm: 46.677589 [I 2022-10-27 02:57:24] ...trap/engines/engine.py.224: train: epoch 44 | batch 300/2304 [I 2022-10-27 02:57:24] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:22 [I 2022-10-27 02:57:24] ...trap/engines/engine.py.229: process: 0.54088 | load: 0.00263 [I 2022-10-27 02:57:24] ...trap/engines/engine.py.230: loss: 0.16757 [I 2022-10-27 02:57:24] ...optimizers/trijoint.py.151: total_norm: 14.593193 [I 2022-10-27 02:57:30] ...trap/engines/engine.py.224: train: epoch 44 | batch 310/2304 [I 2022-10-27 02:57:30] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:29 [I 2022-10-27 02:57:30] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00251 [I 2022-10-27 02:57:30] ...trap/engines/engine.py.230: loss: 0.24076 [I 2022-10-27 02:57:30] ...optimizers/trijoint.py.151: total_norm: 17.418243 [I 2022-10-27 02:57:35] ...trap/engines/engine.py.224: train: epoch 44 | batch 320/2304 [I 2022-10-27 02:57:35] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:27 [I 2022-10-27 02:57:35] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00248 [I 2022-10-27 02:57:35] ...trap/engines/engine.py.230: loss: 0.17300 [I 2022-10-27 02:57:35] ...optimizers/trijoint.py.151: total_norm: 19.058250 [I 2022-10-27 02:57:41] ...trap/engines/engine.py.224: train: epoch 44 | batch 330/2304 [I 2022-10-27 02:57:41] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:16 [I 2022-10-27 02:57:41] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00255 [I 2022-10-27 02:57:41] ...trap/engines/engine.py.230: loss: 0.13571 [I 2022-10-27 02:57:41] ...optimizers/trijoint.py.151: total_norm: 14.200386 [I 2022-10-27 02:57:46] ...trap/engines/engine.py.224: train: epoch 44 | batch 340/2304 [I 2022-10-27 02:57:46] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:16 [I 2022-10-27 02:57:46] ...trap/engines/engine.py.229: process: 0.56764 | load: 0.00246 [I 2022-10-27 02:57:46] ...trap/engines/engine.py.230: loss: 0.16781 [I 2022-10-27 02:57:46] ...optimizers/trijoint.py.151: total_norm: 16.907410 [I 2022-10-27 02:57:52] ...trap/engines/engine.py.224: train: epoch 44 | batch 350/2304 [I 2022-10-27 02:57:52] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:11 [I 2022-10-27 02:57:52] ...trap/engines/engine.py.229: process: 0.56476 | load: 0.00248 [I 2022-10-27 02:57:52] ...trap/engines/engine.py.230: loss: 0.13233 [I 2022-10-27 02:57:52] ...optimizers/trijoint.py.151: total_norm: 24.971464 [I 2022-10-27 02:57:57] ...trap/engines/engine.py.224: train: epoch 44 | batch 360/2304 [I 2022-10-27 02:57:57] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:49 [I 2022-10-27 02:57:57] ...trap/engines/engine.py.229: process: 0.54351 | load: 0.00239 [I 2022-10-27 02:57:57] ...trap/engines/engine.py.230: loss: 0.18602 [I 2022-10-27 02:57:57] ...optimizers/trijoint.py.151: total_norm: 22.836393 [I 2022-10-27 02:58:03] ...trap/engines/engine.py.224: train: epoch 44 | batch 370/2304 [I 2022-10-27 02:58:03] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:59 [I 2022-10-27 02:58:03] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00250 [I 2022-10-27 02:58:03] ...trap/engines/engine.py.230: loss: 0.18830 [I 2022-10-27 02:58:03] ...optimizers/trijoint.py.151: total_norm: 21.958403 [I 2022-10-27 02:58:09] ...trap/engines/engine.py.224: train: epoch 44 | batch 380/2304 [I 2022-10-27 02:58:09] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:46 [I 2022-10-27 02:58:09] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00248 [I 2022-10-27 02:58:09] ...trap/engines/engine.py.230: loss: 0.16876 [I 2022-10-27 02:58:09] ...optimizers/trijoint.py.151: total_norm: 11.452054 [I 2022-10-27 02:58:14] ...trap/engines/engine.py.224: train: epoch 44 | batch 390/2304 [I 2022-10-27 02:58:14] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:18:02 [I 2022-10-27 02:58:14] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00270 [I 2022-10-27 02:58:14] ...trap/engines/engine.py.230: loss: 0.16074 [I 2022-10-27 02:58:14] ...optimizers/trijoint.py.151: total_norm: 13.037040 [I 2022-10-27 02:58:20] ...trap/engines/engine.py.224: train: epoch 44 | batch 400/2304 [I 2022-10-27 02:58:20] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:48 [I 2022-10-27 02:58:20] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00248 [I 2022-10-27 02:58:20] ...trap/engines/engine.py.230: loss: 0.12989 [I 2022-10-27 02:58:20] ...optimizers/trijoint.py.151: total_norm: 13.343450 [I 2022-10-27 02:58:26] ...trap/engines/engine.py.224: train: epoch 44 | batch 410/2304 [I 2022-10-27 02:58:26] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:35 [I 2022-10-27 02:58:26] ...trap/engines/engine.py.229: process: 0.56153 | load: 0.00247 [I 2022-10-27 02:58:26] ...trap/engines/engine.py.230: loss: 0.17134 [I 2022-10-27 02:58:26] ...optimizers/trijoint.py.151: total_norm: 18.169493 [I 2022-10-27 02:58:31] ...trap/engines/engine.py.224: train: epoch 44 | batch 420/2304 [I 2022-10-27 02:58:31] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:25 [I 2022-10-27 02:58:31] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00261 [I 2022-10-27 02:58:31] ...trap/engines/engine.py.230: loss: 0.16517 [I 2022-10-27 02:58:31] ...optimizers/trijoint.py.151: total_norm: 9.993334 [I 2022-10-27 02:58:37] ...trap/engines/engine.py.224: train: epoch 44 | batch 430/2304 [I 2022-10-27 02:58:37] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:17 [I 2022-10-27 02:58:37] ...trap/engines/engine.py.229: process: 0.54963 | load: 0.00241 [I 2022-10-27 02:58:37] ...trap/engines/engine.py.230: loss: 0.19572 [I 2022-10-27 02:58:37] ...optimizers/trijoint.py.151: total_norm: 27.823223 [I 2022-10-27 02:58:42] ...trap/engines/engine.py.224: train: epoch 44 | batch 440/2304 [I 2022-10-27 02:58:42] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:08 [I 2022-10-27 02:58:42] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00242 [I 2022-10-27 02:58:42] ...trap/engines/engine.py.230: loss: 0.14834 [I 2022-10-27 02:58:42] ...optimizers/trijoint.py.151: total_norm: 22.180243 [I 2022-10-27 02:58:48] ...trap/engines/engine.py.224: train: epoch 44 | batch 450/2304 [I 2022-10-27 02:58:48] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:14 [I 2022-10-27 02:58:48] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00249 [I 2022-10-27 02:58:48] ...trap/engines/engine.py.230: loss: 0.14942 [I 2022-10-27 02:58:48] ...optimizers/trijoint.py.151: total_norm: 11.718694 [I 2022-10-27 02:58:53] ...trap/engines/engine.py.224: train: epoch 44 | batch 460/2304 [I 2022-10-27 02:58:53] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:12 [I 2022-10-27 02:58:53] ...trap/engines/engine.py.229: process: 0.57786 | load: 0.00259 [I 2022-10-27 02:58:53] ...trap/engines/engine.py.230: loss: 0.12699 [I 2022-10-27 02:58:53] ...optimizers/trijoint.py.151: total_norm: 18.680094 [I 2022-10-27 02:58:59] ...trap/engines/engine.py.224: train: epoch 44 | batch 470/2304 [I 2022-10-27 02:58:59] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:04 [I 2022-10-27 02:58:59] ...trap/engines/engine.py.229: process: 0.54874 | load: 0.00266 [I 2022-10-27 02:58:59] ...trap/engines/engine.py.230: loss: 0.15412 [I 2022-10-27 02:58:59] ...optimizers/trijoint.py.151: total_norm: 26.506231 [I 2022-10-27 02:59:05] ...trap/engines/engine.py.224: train: epoch 44 | batch 480/2304 [I 2022-10-27 02:59:05] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:58 [I 2022-10-27 02:59:05] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00255 [I 2022-10-27 02:59:05] ...trap/engines/engine.py.230: loss: 0.13729 [I 2022-10-27 02:59:05] ...optimizers/trijoint.py.151: total_norm: 11.316172 [I 2022-10-27 02:59:10] ...trap/engines/engine.py.224: train: epoch 44 | batch 490/2304 [I 2022-10-27 02:59:10] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:49 [I 2022-10-27 02:59:10] ...trap/engines/engine.py.229: process: 0.55026 | load: 0.00242 [I 2022-10-27 02:59:10] ...trap/engines/engine.py.230: loss: 0.12677 [I 2022-10-27 02:59:10] ...optimizers/trijoint.py.151: total_norm: 11.539726 [I 2022-10-27 02:59:16] ...trap/engines/engine.py.224: train: epoch 44 | batch 500/2304 [I 2022-10-27 02:59:16] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:42 [I 2022-10-27 02:59:16] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00242 [I 2022-10-27 02:59:16] ...trap/engines/engine.py.230: loss: 0.12836 [I 2022-10-27 02:59:16] ...optimizers/trijoint.py.151: total_norm: 28.633015 [I 2022-10-27 02:59:21] ...trap/engines/engine.py.224: train: epoch 44 | batch 510/2304 [I 2022-10-27 02:59:21] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:49 [I 2022-10-27 02:59:21] ...trap/engines/engine.py.229: process: 0.56446 | load: 0.00239 [I 2022-10-27 02:59:21] ...trap/engines/engine.py.230: loss: 0.15058 [I 2022-10-27 02:59:21] ...optimizers/trijoint.py.151: total_norm: 17.282747 [I 2022-10-27 02:59:27] ...trap/engines/engine.py.224: train: epoch 44 | batch 520/2304 [I 2022-10-27 02:59:27] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:36 [I 2022-10-27 02:59:27] ...trap/engines/engine.py.229: process: 0.54446 | load: 0.00240 [I 2022-10-27 02:59:27] ...trap/engines/engine.py.230: loss: 0.12126 [I 2022-10-27 02:59:27] ...optimizers/trijoint.py.151: total_norm: 19.785013 [I 2022-10-27 02:59:33] ...trap/engines/engine.py.224: train: epoch 44 | batch 530/2304 [I 2022-10-27 02:59:33] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:28 [I 2022-10-27 02:59:33] ...trap/engines/engine.py.229: process: 0.55828 | load: 0.00251 [I 2022-10-27 02:59:33] ...trap/engines/engine.py.230: loss: 0.15506 [I 2022-10-27 02:59:33] ...optimizers/trijoint.py.151: total_norm: 16.652245 [I 2022-10-27 02:59:38] ...trap/engines/engine.py.224: train: epoch 44 | batch 540/2304 [I 2022-10-27 02:59:38] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:23 [I 2022-10-27 02:59:38] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00248 [I 2022-10-27 02:59:38] ...trap/engines/engine.py.230: loss: 0.24287 [I 2022-10-27 02:59:38] ...optimizers/trijoint.py.151: total_norm: 21.176298 [I 2022-10-27 02:59:44] ...trap/engines/engine.py.224: train: epoch 44 | batch 550/2304 [I 2022-10-27 02:59:44] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:12 [I 2022-10-27 02:59:44] ...trap/engines/engine.py.229: process: 0.55192 | load: 0.00261 [I 2022-10-27 02:59:44] ...trap/engines/engine.py.230: loss: 0.14617 [I 2022-10-27 02:59:44] ...optimizers/trijoint.py.151: total_norm: 14.203443 [I 2022-10-27 02:59:49] ...trap/engines/engine.py.224: train: epoch 44 | batch 560/2304 [I 2022-10-27 02:59:49] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:16 [I 2022-10-27 02:59:49] ...trap/engines/engine.py.229: process: 0.56284 | load: 0.00252 [I 2022-10-27 02:59:49] ...trap/engines/engine.py.230: loss: 0.11886 [I 2022-10-27 02:59:49] ...optimizers/trijoint.py.151: total_norm: 10.005203 [I 2022-10-27 02:59:55] ...trap/engines/engine.py.224: train: epoch 44 | batch 570/2304 [I 2022-10-27 02:59:55] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:11 [I 2022-10-27 02:59:55] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00243 [I 2022-10-27 02:59:55] ...trap/engines/engine.py.230: loss: 0.20088 [I 2022-10-27 02:59:55] ...optimizers/trijoint.py.151: total_norm: 23.126253 [I 2022-10-27 03:00:00] ...trap/engines/engine.py.224: train: epoch 44 | batch 580/2304 [I 2022-10-27 03:00:00] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:02 [I 2022-10-27 03:00:00] ...trap/engines/engine.py.229: process: 0.56109 | load: 0.00242 [I 2022-10-27 03:00:00] ...trap/engines/engine.py.230: loss: 0.22285 [I 2022-10-27 03:00:00] ...optimizers/trijoint.py.151: total_norm: 14.909767 [I 2022-10-27 03:00:06] ...trap/engines/engine.py.224: train: epoch 44 | batch 590/2304 [I 2022-10-27 03:00:06] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:52 [I 2022-10-27 03:00:06] ...trap/engines/engine.py.229: process: 0.54725 | load: 0.00247 [I 2022-10-27 03:00:06] ...trap/engines/engine.py.230: loss: 0.11977 [I 2022-10-27 03:00:06] ...optimizers/trijoint.py.151: total_norm: 7.173799 [I 2022-10-27 03:00:12] ...trap/engines/engine.py.224: train: epoch 44 | batch 600/2304 [I 2022-10-27 03:00:12] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:47 [I 2022-10-27 03:00:12] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00242 [I 2022-10-27 03:00:12] ...trap/engines/engine.py.230: loss: 0.11747 [I 2022-10-27 03:00:12] ...optimizers/trijoint.py.151: total_norm: 8.505254 [I 2022-10-27 03:00:17] ...trap/engines/engine.py.224: train: epoch 44 | batch 610/2304 [I 2022-10-27 03:00:17] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:44 [I 2022-10-27 03:00:17] ...trap/engines/engine.py.229: process: 0.56796 | load: 0.00247 [I 2022-10-27 03:00:17] ...trap/engines/engine.py.230: loss: 0.14469 [I 2022-10-27 03:00:17] ...optimizers/trijoint.py.151: total_norm: 10.291063 [I 2022-10-27 03:00:23] ...trap/engines/engine.py.224: train: epoch 44 | batch 620/2304 [I 2022-10-27 03:00:23] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:44 [I 2022-10-27 03:00:23] ...trap/engines/engine.py.229: process: 0.54280 | load: 0.00243 [I 2022-10-27 03:00:23] ...trap/engines/engine.py.230: loss: 0.12928 [I 2022-10-27 03:00:23] ...optimizers/trijoint.py.151: total_norm: 9.732619 [I 2022-10-27 03:00:29] ...trap/engines/engine.py.224: train: epoch 44 | batch 630/2304 [I 2022-10-27 03:00:29] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:41 [I 2022-10-27 03:00:29] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00249 [I 2022-10-27 03:00:29] ...trap/engines/engine.py.230: loss: 0.21000 [I 2022-10-27 03:00:29] ...optimizers/trijoint.py.151: total_norm: 12.424277 [I 2022-10-27 03:00:34] ...trap/engines/engine.py.224: train: epoch 44 | batch 640/2304 [I 2022-10-27 03:00:34] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:22 [I 2022-10-27 03:00:34] ...trap/engines/engine.py.229: process: 0.54980 | load: 0.00244 [I 2022-10-27 03:00:34] ...trap/engines/engine.py.230: loss: 0.17105 [I 2022-10-27 03:00:34] ...optimizers/trijoint.py.151: total_norm: 30.426497 [I 2022-10-27 03:00:40] ...trap/engines/engine.py.224: train: epoch 44 | batch 650/2304 [I 2022-10-27 03:00:40] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:24 [I 2022-10-27 03:00:40] ...trap/engines/engine.py.229: process: 0.56116 | load: 0.00246 [I 2022-10-27 03:00:40] ...trap/engines/engine.py.230: loss: 0.22493 [I 2022-10-27 03:00:40] ...optimizers/trijoint.py.151: total_norm: 27.718048 [I 2022-10-27 03:00:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 660/2304 [I 2022-10-27 03:00:45] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:13 [I 2022-10-27 03:00:45] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00246 [I 2022-10-27 03:00:45] ...trap/engines/engine.py.230: loss: 0.15146 [I 2022-10-27 03:00:45] ...optimizers/trijoint.py.151: total_norm: 15.797359 [I 2022-10-27 03:00:51] ...trap/engines/engine.py.224: train: epoch 44 | batch 670/2304 [I 2022-10-27 03:00:51] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:17 [I 2022-10-27 03:00:51] ...trap/engines/engine.py.229: process: 0.56578 | load: 0.00263 [I 2022-10-27 03:00:51] ...trap/engines/engine.py.230: loss: 0.20556 [I 2022-10-27 03:00:51] ...optimizers/trijoint.py.151: total_norm: 13.721708 [I 2022-10-27 03:00:56] ...trap/engines/engine.py.224: train: epoch 44 | batch 680/2304 [I 2022-10-27 03:00:56] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:14:59 [I 2022-10-27 03:00:56] ...trap/engines/engine.py.229: process: 0.54472 | load: 0.00243 [I 2022-10-27 03:00:56] ...trap/engines/engine.py.230: loss: 0.27774 [I 2022-10-27 03:00:56] ...optimizers/trijoint.py.151: total_norm: 24.798502 [I 2022-10-27 03:01:20] ...trap/engines/engine.py.224: train: epoch 44 | batch 690/2304 [I 2022-10-27 03:01:20] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 1:32:25 [I 2022-10-27 03:01:20] ...trap/engines/engine.py.229: process: 0.58040 | load: 0.00243 [I 2022-10-27 03:01:20] ...trap/engines/engine.py.230: loss: 0.20028 [I 2022-10-27 03:01:20] ...optimizers/trijoint.py.151: total_norm: 22.455240 [I 2022-10-27 03:01:26] ...trap/engines/engine.py.224: train: epoch 44 | batch 700/2304 [I 2022-10-27 03:01:26] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:23:19 [I 2022-10-27 03:01:26] ...trap/engines/engine.py.229: process: 0.57105 | load: 0.00250 [I 2022-10-27 03:01:26] ...trap/engines/engine.py.230: loss: 0.16238 [I 2022-10-27 03:01:26] ...optimizers/trijoint.py.151: total_norm: 32.789642 [I 2022-10-27 03:01:31] ...trap/engines/engine.py.224: train: epoch 44 | batch 710/2304 [I 2022-10-27 03:01:31] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:15:40 [I 2022-10-27 03:01:31] ...trap/engines/engine.py.229: process: 0.54636 | load: 0.00257 [I 2022-10-27 03:01:31] ...trap/engines/engine.py.230: loss: 0.17698 [I 2022-10-27 03:01:31] ...optimizers/trijoint.py.151: total_norm: 32.778683 [I 2022-10-27 03:01:37] ...trap/engines/engine.py.224: train: epoch 44 | batch 720/2304 [I 2022-10-27 03:01:37] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:49 [I 2022-10-27 03:01:37] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00248 [I 2022-10-27 03:01:37] ...trap/engines/engine.py.230: loss: 0.18580 [I 2022-10-27 03:01:37] ...optimizers/trijoint.py.151: total_norm: 13.462669 [I 2022-10-27 03:01:42] ...trap/engines/engine.py.224: train: epoch 44 | batch 730/2304 [I 2022-10-27 03:01:42] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:28 [I 2022-10-27 03:01:42] ...trap/engines/engine.py.229: process: 0.53765 | load: 0.00246 [I 2022-10-27 03:01:42] ...trap/engines/engine.py.230: loss: 0.18768 [I 2022-10-27 03:01:42] ...optimizers/trijoint.py.151: total_norm: 16.190985 [I 2022-10-27 03:01:48] ...trap/engines/engine.py.224: train: epoch 44 | batch 740/2304 [I 2022-10-27 03:01:48] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:30 [I 2022-10-27 03:01:48] ...trap/engines/engine.py.229: process: 0.55041 | load: 0.00249 [I 2022-10-27 03:01:48] ...trap/engines/engine.py.230: loss: 0.17093 [I 2022-10-27 03:01:48] ...optimizers/trijoint.py.151: total_norm: 20.307520 [I 2022-10-27 03:01:54] ...trap/engines/engine.py.224: train: epoch 44 | batch 750/2304 [I 2022-10-27 03:01:54] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:30 [I 2022-10-27 03:01:54] ...trap/engines/engine.py.229: process: 0.56832 | load: 0.00240 [I 2022-10-27 03:01:54] ...trap/engines/engine.py.230: loss: 0.19553 [I 2022-10-27 03:01:54] ...optimizers/trijoint.py.151: total_norm: 11.422015 [I 2022-10-27 03:01:59] ...trap/engines/engine.py.224: train: epoch 44 | batch 760/2304 [I 2022-10-27 03:01:59] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:21 [I 2022-10-27 03:01:59] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00244 [I 2022-10-27 03:01:59] ...trap/engines/engine.py.230: loss: 0.27606 [I 2022-10-27 03:01:59] ...optimizers/trijoint.py.151: total_norm: 19.023552 [I 2022-10-27 03:02:05] ...trap/engines/engine.py.224: train: epoch 44 | batch 770/2304 [I 2022-10-27 03:02:05] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:14:23 [I 2022-10-27 03:02:05] ...trap/engines/engine.py.229: process: 0.56903 | load: 0.00246 [I 2022-10-27 03:02:05] ...trap/engines/engine.py.230: loss: 0.15784 [I 2022-10-27 03:02:05] ...optimizers/trijoint.py.151: total_norm: 9.888513 [I 2022-10-27 03:02:10] ...trap/engines/engine.py.224: train: epoch 44 | batch 780/2304 [I 2022-10-27 03:02:10] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:14:09 [I 2022-10-27 03:02:10] ...trap/engines/engine.py.229: process: 0.56101 | load: 0.00243 [I 2022-10-27 03:02:10] ...trap/engines/engine.py.230: loss: 0.12543 [I 2022-10-27 03:02:10] ...optimizers/trijoint.py.151: total_norm: 10.513021 [I 2022-10-27 03:02:16] ...trap/engines/engine.py.224: train: epoch 44 | batch 790/2304 [I 2022-10-27 03:02:16] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:14:00 [I 2022-10-27 03:02:16] ...trap/engines/engine.py.229: process: 0.56086 | load: 0.00246 [I 2022-10-27 03:02:16] ...trap/engines/engine.py.230: loss: 0.13106 [I 2022-10-27 03:02:16] ...optimizers/trijoint.py.151: total_norm: 17.701231 [I 2022-10-27 03:02:22] ...trap/engines/engine.py.224: train: epoch 44 | batch 800/2304 [I 2022-10-27 03:02:22] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:59 [I 2022-10-27 03:02:22] ...trap/engines/engine.py.229: process: 0.56337 | load: 0.00250 [I 2022-10-27 03:02:22] ...trap/engines/engine.py.230: loss: 0.19342 [I 2022-10-27 03:02:22] ...optimizers/trijoint.py.151: total_norm: 20.825655 [I 2022-10-27 03:02:27] ...trap/engines/engine.py.224: train: epoch 44 | batch 810/2304 [I 2022-10-27 03:02:27] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:47 [I 2022-10-27 03:02:27] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00245 [I 2022-10-27 03:02:27] ...trap/engines/engine.py.230: loss: 0.15700 [I 2022-10-27 03:02:27] ...optimizers/trijoint.py.151: total_norm: 8.829585 [I 2022-10-27 03:02:33] ...trap/engines/engine.py.224: train: epoch 44 | batch 820/2304 [I 2022-10-27 03:02:33] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:43 [I 2022-10-27 03:02:33] ...trap/engines/engine.py.229: process: 0.55421 | load: 0.00243 [I 2022-10-27 03:02:33] ...trap/engines/engine.py.230: loss: 0.16386 [I 2022-10-27 03:02:33] ...optimizers/trijoint.py.151: total_norm: 12.140160 [I 2022-10-27 03:02:38] ...trap/engines/engine.py.224: train: epoch 44 | batch 830/2304 [I 2022-10-27 03:02:38] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:32 [I 2022-10-27 03:02:38] ...trap/engines/engine.py.229: process: 0.54875 | load: 0.00250 [I 2022-10-27 03:02:38] ...trap/engines/engine.py.230: loss: 0.19495 [I 2022-10-27 03:02:38] ...optimizers/trijoint.py.151: total_norm: 20.172438 [I 2022-10-27 03:02:44] ...trap/engines/engine.py.224: train: epoch 44 | batch 840/2304 [I 2022-10-27 03:02:44] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:35 [I 2022-10-27 03:02:44] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00242 [I 2022-10-27 03:02:44] ...trap/engines/engine.py.230: loss: 0.13587 [I 2022-10-27 03:02:44] ...optimizers/trijoint.py.151: total_norm: 8.951878 [I 2022-10-27 03:02:49] ...trap/engines/engine.py.224: train: epoch 44 | batch 850/2304 [I 2022-10-27 03:02:49] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:31 [I 2022-10-27 03:02:49] ...trap/engines/engine.py.229: process: 0.56101 | load: 0.00242 [I 2022-10-27 03:02:49] ...trap/engines/engine.py.230: loss: 0.16952 [I 2022-10-27 03:02:49] ...optimizers/trijoint.py.151: total_norm: 14.458358 [I 2022-10-27 03:02:55] ...trap/engines/engine.py.224: train: epoch 44 | batch 860/2304 [I 2022-10-27 03:02:55] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:26 [I 2022-10-27 03:02:55] ...trap/engines/engine.py.229: process: 0.55089 | load: 0.00246 [I 2022-10-27 03:02:55] ...trap/engines/engine.py.230: loss: 0.11131 [I 2022-10-27 03:02:55] ...optimizers/trijoint.py.151: total_norm: 8.402741 [I 2022-10-27 03:03:00] ...trap/engines/engine.py.224: train: epoch 44 | batch 870/2304 [I 2022-10-27 03:03:00] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:12 [I 2022-10-27 03:03:00] ...trap/engines/engine.py.229: process: 0.54691 | load: 0.00242 [I 2022-10-27 03:03:00] ...trap/engines/engine.py.230: loss: 0.12477 [I 2022-10-27 03:03:00] ...optimizers/trijoint.py.151: total_norm: 25.133055 [I 2022-10-27 03:03:06] ...trap/engines/engine.py.224: train: epoch 44 | batch 880/2304 [I 2022-10-27 03:03:06] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:13:03 [I 2022-10-27 03:03:06] ...trap/engines/engine.py.229: process: 0.54349 | load: 0.00244 [I 2022-10-27 03:03:06] ...trap/engines/engine.py.230: loss: 0.15746 [I 2022-10-27 03:03:06] ...optimizers/trijoint.py.151: total_norm: 21.736849 [I 2022-10-27 03:03:12] ...trap/engines/engine.py.224: train: epoch 44 | batch 890/2304 [I 2022-10-27 03:03:12] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:13:10 [I 2022-10-27 03:03:12] ...trap/engines/engine.py.229: process: 0.56652 | load: 0.00244 [I 2022-10-27 03:03:12] ...trap/engines/engine.py.230: loss: 0.11226 [I 2022-10-27 03:03:12] ...optimizers/trijoint.py.151: total_norm: 26.866663 [I 2022-10-27 03:03:17] ...trap/engines/engine.py.224: train: epoch 44 | batch 900/2304 [I 2022-10-27 03:03:17] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:59 [I 2022-10-27 03:03:17] ...trap/engines/engine.py.229: process: 0.55243 | load: 0.00243 [I 2022-10-27 03:03:17] ...trap/engines/engine.py.230: loss: 0.15281 [I 2022-10-27 03:03:17] ...optimizers/trijoint.py.151: total_norm: 10.104349 [I 2022-10-27 03:03:23] ...trap/engines/engine.py.224: train: epoch 44 | batch 910/2304 [I 2022-10-27 03:03:23] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:55 [I 2022-10-27 03:03:23] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00241 [I 2022-10-27 03:03:23] ...trap/engines/engine.py.230: loss: 0.14456 [I 2022-10-27 03:03:23] ...optimizers/trijoint.py.151: total_norm: 26.468485 [I 2022-10-27 03:03:28] ...trap/engines/engine.py.224: train: epoch 44 | batch 920/2304 [I 2022-10-27 03:03:28] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:57 [I 2022-10-27 03:03:28] ...trap/engines/engine.py.229: process: 0.56699 | load: 0.00243 [I 2022-10-27 03:03:28] ...trap/engines/engine.py.230: loss: 0.19222 [I 2022-10-27 03:03:28] ...optimizers/trijoint.py.151: total_norm: 18.511381 [I 2022-10-27 03:03:34] ...trap/engines/engine.py.224: train: epoch 44 | batch 930/2304 [I 2022-10-27 03:03:34] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:45 [I 2022-10-27 03:03:34] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00245 [I 2022-10-27 03:03:34] ...trap/engines/engine.py.230: loss: 0.20754 [I 2022-10-27 03:03:34] ...optimizers/trijoint.py.151: total_norm: 23.059378 [I 2022-10-27 03:03:39] ...trap/engines/engine.py.224: train: epoch 44 | batch 940/2304 [I 2022-10-27 03:03:39] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:31 [I 2022-10-27 03:03:39] ...trap/engines/engine.py.229: process: 0.54739 | load: 0.00247 [I 2022-10-27 03:03:39] ...trap/engines/engine.py.230: loss: 0.11670 [I 2022-10-27 03:03:39] ...optimizers/trijoint.py.151: total_norm: 13.862687 [I 2022-10-27 03:03:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 950/2304 [I 2022-10-27 03:03:45] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:34 [I 2022-10-27 03:03:45] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00241 [I 2022-10-27 03:03:45] ...trap/engines/engine.py.230: loss: 0.17913 [I 2022-10-27 03:03:45] ...optimizers/trijoint.py.151: total_norm: 81.483795 [I 2022-10-27 03:03:51] ...trap/engines/engine.py.224: train: epoch 44 | batch 960/2304 [I 2022-10-27 03:03:51] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:30 [I 2022-10-27 03:03:51] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00243 [I 2022-10-27 03:03:51] ...trap/engines/engine.py.230: loss: 0.15319 [I 2022-10-27 03:03:51] ...optimizers/trijoint.py.151: total_norm: 9.659677 [I 2022-10-27 03:03:56] ...trap/engines/engine.py.224: train: epoch 44 | batch 970/2304 [I 2022-10-27 03:03:56] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:29 [I 2022-10-27 03:03:56] ...trap/engines/engine.py.229: process: 0.55120 | load: 0.00245 [I 2022-10-27 03:03:56] ...trap/engines/engine.py.230: loss: 0.18329 [I 2022-10-27 03:03:56] ...optimizers/trijoint.py.151: total_norm: 18.053411 [I 2022-10-27 03:04:02] ...trap/engines/engine.py.224: train: epoch 44 | batch 980/2304 [I 2022-10-27 03:04:02] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:20 [I 2022-10-27 03:04:02] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00245 [I 2022-10-27 03:04:02] ...trap/engines/engine.py.230: loss: 0.15666 [I 2022-10-27 03:04:02] ...optimizers/trijoint.py.151: total_norm: 28.245853 [I 2022-10-27 03:04:07] ...trap/engines/engine.py.224: train: epoch 44 | batch 990/2304 [I 2022-10-27 03:04:07] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:12:10 [I 2022-10-27 03:04:07] ...trap/engines/engine.py.229: process: 0.56398 | load: 0.00248 [I 2022-10-27 03:04:07] ...trap/engines/engine.py.230: loss: 0.10532 [I 2022-10-27 03:04:07] ...optimizers/trijoint.py.151: total_norm: 12.164881 [I 2022-10-27 03:04:13] ...trap/engines/engine.py.224: train: epoch 44 | batch 1000/2304 [I 2022-10-27 03:04:13] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:12:07 [I 2022-10-27 03:04:13] ...trap/engines/engine.py.229: process: 0.56504 | load: 0.00244 [I 2022-10-27 03:04:13] ...trap/engines/engine.py.230: loss: 0.15870 [I 2022-10-27 03:04:13] ...optimizers/trijoint.py.151: total_norm: 17.393301 [I 2022-10-27 03:04:19] ...trap/engines/engine.py.224: train: epoch 44 | batch 1010/2304 [I 2022-10-27 03:04:19] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:59 [I 2022-10-27 03:04:19] ...trap/engines/engine.py.229: process: 0.54740 | load: 0.00239 [I 2022-10-27 03:04:19] ...trap/engines/engine.py.230: loss: 0.11783 [I 2022-10-27 03:04:19] ...optimizers/trijoint.py.151: total_norm: 13.282952 [I 2022-10-27 03:04:24] ...trap/engines/engine.py.224: train: epoch 44 | batch 1020/2304 [I 2022-10-27 03:04:24] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:54 [I 2022-10-27 03:04:24] ...trap/engines/engine.py.229: process: 0.56182 | load: 0.00243 [I 2022-10-27 03:04:24] ...trap/engines/engine.py.230: loss: 0.19304 [I 2022-10-27 03:04:24] ...optimizers/trijoint.py.151: total_norm: 23.404669 [I 2022-10-27 03:04:30] ...trap/engines/engine.py.224: train: epoch 44 | batch 1030/2304 [I 2022-10-27 03:04:30] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:45 [I 2022-10-27 03:04:30] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00244 [I 2022-10-27 03:04:30] ...trap/engines/engine.py.230: loss: 0.14572 [I 2022-10-27 03:04:30] ...optimizers/trijoint.py.151: total_norm: 15.279230 [I 2022-10-27 03:04:35] ...trap/engines/engine.py.224: train: epoch 44 | batch 1040/2304 [I 2022-10-27 03:04:35] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:39 [I 2022-10-27 03:04:35] ...trap/engines/engine.py.229: process: 0.54825 | load: 0.00246 [I 2022-10-27 03:04:35] ...trap/engines/engine.py.230: loss: 0.19439 [I 2022-10-27 03:04:35] ...optimizers/trijoint.py.151: total_norm: 18.430758 [I 2022-10-27 03:04:41] ...trap/engines/engine.py.224: train: epoch 44 | batch 1050/2304 [I 2022-10-27 03:04:41] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:38 [I 2022-10-27 03:04:41] ...trap/engines/engine.py.229: process: 0.54941 | load: 0.00243 [I 2022-10-27 03:04:41] ...trap/engines/engine.py.230: loss: 0.14559 [I 2022-10-27 03:04:41] ...optimizers/trijoint.py.151: total_norm: 13.682855 [I 2022-10-27 03:04:46] ...trap/engines/engine.py.224: train: epoch 44 | batch 1060/2304 [I 2022-10-27 03:04:46] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:32 [I 2022-10-27 03:04:46] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00242 [I 2022-10-27 03:04:46] ...trap/engines/engine.py.230: loss: 0.12606 [I 2022-10-27 03:04:46] ...optimizers/trijoint.py.151: total_norm: 29.665442 [I 2022-10-27 03:04:52] ...trap/engines/engine.py.224: train: epoch 44 | batch 1070/2304 [I 2022-10-27 03:04:52] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:22 [I 2022-10-27 03:04:52] ...trap/engines/engine.py.229: process: 0.55274 | load: 0.00244 [I 2022-10-27 03:04:52] ...trap/engines/engine.py.230: loss: 0.16722 [I 2022-10-27 03:04:52] ...optimizers/trijoint.py.151: total_norm: 12.910938 [I 2022-10-27 03:04:57] ...trap/engines/engine.py.224: train: epoch 44 | batch 1080/2304 [I 2022-10-27 03:04:57] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:13 [I 2022-10-27 03:04:57] ...trap/engines/engine.py.229: process: 0.53846 | load: 0.00243 [I 2022-10-27 03:04:57] ...trap/engines/engine.py.230: loss: 0.19579 [I 2022-10-27 03:04:57] ...optimizers/trijoint.py.151: total_norm: 31.283987 [I 2022-10-27 03:05:03] ...trap/engines/engine.py.224: train: epoch 44 | batch 1090/2304 [I 2022-10-27 03:05:03] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:16 [I 2022-10-27 03:05:03] ...trap/engines/engine.py.229: process: 0.54998 | load: 0.00244 [I 2022-10-27 03:05:03] ...trap/engines/engine.py.230: loss: 0.21073 [I 2022-10-27 03:05:03] ...optimizers/trijoint.py.151: total_norm: 16.976746 [I 2022-10-27 03:05:09] ...trap/engines/engine.py.224: train: epoch 44 | batch 1100/2304 [I 2022-10-27 03:05:09] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:11:08 [I 2022-10-27 03:05:09] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00245 [I 2022-10-27 03:05:09] ...trap/engines/engine.py.230: loss: 0.16943 [I 2022-10-27 03:05:09] ...optimizers/trijoint.py.151: total_norm: 18.474165 [I 2022-10-27 03:05:14] ...trap/engines/engine.py.224: train: epoch 44 | batch 1110/2304 [I 2022-10-27 03:05:14] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:11:01 [I 2022-10-27 03:05:14] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00243 [I 2022-10-27 03:05:14] ...trap/engines/engine.py.230: loss: 0.17634 [I 2022-10-27 03:05:14] ...optimizers/trijoint.py.151: total_norm: 14.019888 [I 2022-10-27 03:05:20] ...trap/engines/engine.py.224: train: epoch 44 | batch 1120/2304 [I 2022-10-27 03:05:20] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:57 [I 2022-10-27 03:05:20] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00243 [I 2022-10-27 03:05:20] ...trap/engines/engine.py.230: loss: 0.23087 [I 2022-10-27 03:05:20] ...optimizers/trijoint.py.151: total_norm: 19.228500 [I 2022-10-27 03:05:25] ...trap/engines/engine.py.224: train: epoch 44 | batch 1130/2304 [I 2022-10-27 03:05:25] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:11:18 [I 2022-10-27 03:05:25] ...trap/engines/engine.py.229: process: 0.58308 | load: 0.00255 [I 2022-10-27 03:05:25] ...trap/engines/engine.py.230: loss: 0.11969 [I 2022-10-27 03:05:25] ...optimizers/trijoint.py.151: total_norm: 9.268918 [I 2022-10-27 03:05:31] ...trap/engines/engine.py.224: train: epoch 44 | batch 1140/2304 [I 2022-10-27 03:05:31] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:52 [I 2022-10-27 03:05:31] ...trap/engines/engine.py.229: process: 0.56402 | load: 0.00242 [I 2022-10-27 03:05:31] ...trap/engines/engine.py.230: loss: 0.16604 [I 2022-10-27 03:05:31] ...optimizers/trijoint.py.151: total_norm: 16.855570 [I 2022-10-27 03:05:37] ...trap/engines/engine.py.224: train: epoch 44 | batch 1150/2304 [I 2022-10-27 03:05:37] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:48 [I 2022-10-27 03:05:37] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00244 [I 2022-10-27 03:05:37] ...trap/engines/engine.py.230: loss: 0.13166 [I 2022-10-27 03:05:37] ...optimizers/trijoint.py.151: total_norm: 12.661087 [I 2022-10-27 03:05:42] ...trap/engines/engine.py.224: train: epoch 44 | batch 1160/2304 [I 2022-10-27 03:05:42] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:39 [I 2022-10-27 03:05:42] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00242 [I 2022-10-27 03:05:42] ...trap/engines/engine.py.230: loss: 0.14950 [I 2022-10-27 03:05:42] ...optimizers/trijoint.py.151: total_norm: 21.042078 [I 2022-10-27 03:05:48] ...trap/engines/engine.py.224: train: epoch 44 | batch 1170/2304 [I 2022-10-27 03:05:48] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:32 [I 2022-10-27 03:05:48] ...trap/engines/engine.py.229: process: 0.56876 | load: 0.00243 [I 2022-10-27 03:05:48] ...trap/engines/engine.py.230: loss: 0.22446 [I 2022-10-27 03:05:48] ...optimizers/trijoint.py.151: total_norm: 17.814186 [I 2022-10-27 03:05:54] ...trap/engines/engine.py.224: train: epoch 44 | batch 1180/2304 [I 2022-10-27 03:05:54] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:23 [I 2022-10-27 03:05:54] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00244 [I 2022-10-27 03:05:54] ...trap/engines/engine.py.230: loss: 0.16897 [I 2022-10-27 03:05:54] ...optimizers/trijoint.py.151: total_norm: 18.252478 [I 2022-10-27 03:05:59] ...trap/engines/engine.py.224: train: epoch 44 | batch 1190/2304 [I 2022-10-27 03:05:59] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:15 [I 2022-10-27 03:05:59] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00245 [I 2022-10-27 03:05:59] ...trap/engines/engine.py.230: loss: 0.17272 [I 2022-10-27 03:05:59] ...optimizers/trijoint.py.151: total_norm: 16.867760 [I 2022-10-27 03:06:05] ...trap/engines/engine.py.224: train: epoch 44 | batch 1200/2304 [I 2022-10-27 03:06:05] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:10:11 [I 2022-10-27 03:06:05] ...trap/engines/engine.py.229: process: 0.54861 | load: 0.00246 [I 2022-10-27 03:06:05] ...trap/engines/engine.py.230: loss: 0.20110 [I 2022-10-27 03:06:05] ...optimizers/trijoint.py.151: total_norm: 13.751605 [I 2022-10-27 03:06:10] ...trap/engines/engine.py.224: train: epoch 44 | batch 1210/2304 [I 2022-10-27 03:06:10] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:10:10 [I 2022-10-27 03:06:10] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00244 [I 2022-10-27 03:06:10] ...trap/engines/engine.py.230: loss: 0.13381 [I 2022-10-27 03:06:10] ...optimizers/trijoint.py.151: total_norm: 16.831039 [I 2022-10-27 03:06:16] ...trap/engines/engine.py.224: train: epoch 44 | batch 1220/2304 [I 2022-10-27 03:06:16] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:10:19 [I 2022-10-27 03:06:16] ...trap/engines/engine.py.229: process: 0.56377 | load: 0.00240 [I 2022-10-27 03:06:16] ...trap/engines/engine.py.230: loss: 0.19870 [I 2022-10-27 03:06:16] ...optimizers/trijoint.py.151: total_norm: 12.368603 [I 2022-10-27 03:06:21] ...trap/engines/engine.py.224: train: epoch 44 | batch 1230/2304 [I 2022-10-27 03:06:21] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:10:00 [I 2022-10-27 03:06:21] ...trap/engines/engine.py.229: process: 0.55425 | load: 0.00243 [I 2022-10-27 03:06:21] ...trap/engines/engine.py.230: loss: 0.16586 [I 2022-10-27 03:06:21] ...optimizers/trijoint.py.151: total_norm: 19.809622 [I 2022-10-27 03:06:27] ...trap/engines/engine.py.224: train: epoch 44 | batch 1240/2304 [I 2022-10-27 03:06:27] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:53 [I 2022-10-27 03:06:27] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00241 [I 2022-10-27 03:06:27] ...trap/engines/engine.py.230: loss: 0.13004 [I 2022-10-27 03:06:27] ...optimizers/trijoint.py.151: total_norm: 41.575882 [I 2022-10-27 03:06:33] ...trap/engines/engine.py.224: train: epoch 44 | batch 1250/2304 [I 2022-10-27 03:06:33] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:46 [I 2022-10-27 03:06:33] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00241 [I 2022-10-27 03:06:33] ...trap/engines/engine.py.230: loss: 0.15466 [I 2022-10-27 03:06:33] ...optimizers/trijoint.py.151: total_norm: 10.769591 [I 2022-10-27 03:06:38] ...trap/engines/engine.py.224: train: epoch 44 | batch 1260/2304 [I 2022-10-27 03:06:38] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:38 [I 2022-10-27 03:06:38] ...trap/engines/engine.py.229: process: 0.55587 | load: 0.00244 [I 2022-10-27 03:06:38] ...trap/engines/engine.py.230: loss: 0.14745 [I 2022-10-27 03:06:38] ...optimizers/trijoint.py.151: total_norm: 9.897137 [I 2022-10-27 03:06:44] ...trap/engines/engine.py.224: train: epoch 44 | batch 1270/2304 [I 2022-10-27 03:06:44] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:34 [I 2022-10-27 03:06:44] ...trap/engines/engine.py.229: process: 0.55865 | load: 0.00242 [I 2022-10-27 03:06:44] ...trap/engines/engine.py.230: loss: 0.20098 [I 2022-10-27 03:06:44] ...optimizers/trijoint.py.151: total_norm: 11.330934 [I 2022-10-27 03:06:49] ...trap/engines/engine.py.224: train: epoch 44 | batch 1280/2304 [I 2022-10-27 03:06:49] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:28 [I 2022-10-27 03:06:49] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00242 [I 2022-10-27 03:06:49] ...trap/engines/engine.py.230: loss: 0.12524 [I 2022-10-27 03:06:49] ...optimizers/trijoint.py.151: total_norm: 12.823562 [I 2022-10-27 03:06:55] ...trap/engines/engine.py.224: train: epoch 44 | batch 1290/2304 [I 2022-10-27 03:06:55] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:27 [I 2022-10-27 03:06:55] ...trap/engines/engine.py.229: process: 0.55288 | load: 0.00242 [I 2022-10-27 03:06:55] ...trap/engines/engine.py.230: loss: 0.14253 [I 2022-10-27 03:06:55] ...optimizers/trijoint.py.151: total_norm: 11.607598 [I 2022-10-27 03:07:01] ...trap/engines/engine.py.224: train: epoch 44 | batch 1300/2304 [I 2022-10-27 03:07:01] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:17 [I 2022-10-27 03:07:01] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00241 [I 2022-10-27 03:07:01] ...trap/engines/engine.py.230: loss: 0.21820 [I 2022-10-27 03:07:01] ...optimizers/trijoint.py.151: total_norm: 27.260910 [I 2022-10-27 03:07:06] ...trap/engines/engine.py.224: train: epoch 44 | batch 1310/2304 [I 2022-10-27 03:07:06] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:09:15 [I 2022-10-27 03:07:06] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00242 [I 2022-10-27 03:07:06] ...trap/engines/engine.py.230: loss: 0.12406 [I 2022-10-27 03:07:06] ...optimizers/trijoint.py.151: total_norm: 8.938237 [I 2022-10-27 03:07:12] ...trap/engines/engine.py.224: train: epoch 44 | batch 1320/2304 [I 2022-10-27 03:07:12] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:09:05 [I 2022-10-27 03:07:12] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00240 [I 2022-10-27 03:07:12] ...trap/engines/engine.py.230: loss: 0.14595 [I 2022-10-27 03:07:12] ...optimizers/trijoint.py.151: total_norm: 14.012324 [I 2022-10-27 03:07:17] ...trap/engines/engine.py.224: train: epoch 44 | batch 1330/2304 [I 2022-10-27 03:07:17] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:58 [I 2022-10-27 03:07:17] ...trap/engines/engine.py.229: process: 0.56091 | load: 0.00242 [I 2022-10-27 03:07:17] ...trap/engines/engine.py.230: loss: 0.13420 [I 2022-10-27 03:07:17] ...optimizers/trijoint.py.151: total_norm: 57.290787 [I 2022-10-27 03:07:23] ...trap/engines/engine.py.224: train: epoch 44 | batch 1340/2304 [I 2022-10-27 03:07:23] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:59 [I 2022-10-27 03:07:23] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00261 [I 2022-10-27 03:07:23] ...trap/engines/engine.py.230: loss: 0.16743 [I 2022-10-27 03:07:23] ...optimizers/trijoint.py.151: total_norm: 13.759303 [I 2022-10-27 03:07:28] ...trap/engines/engine.py.224: train: epoch 44 | batch 1350/2304 [I 2022-10-27 03:07:28] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:09:03 [I 2022-10-27 03:07:28] ...trap/engines/engine.py.229: process: 0.62806 | load: 0.00254 [I 2022-10-27 03:07:28] ...trap/engines/engine.py.230: loss: 0.15153 [I 2022-10-27 03:07:28] ...optimizers/trijoint.py.151: total_norm: 12.363160 [I 2022-10-27 03:07:34] ...trap/engines/engine.py.224: train: epoch 44 | batch 1360/2304 [I 2022-10-27 03:07:34] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:45 [I 2022-10-27 03:07:34] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00252 [I 2022-10-27 03:07:34] ...trap/engines/engine.py.230: loss: 0.14910 [I 2022-10-27 03:07:34] ...optimizers/trijoint.py.151: total_norm: 9.068710 [I 2022-10-27 03:07:40] ...trap/engines/engine.py.224: train: epoch 44 | batch 1370/2304 [I 2022-10-27 03:07:40] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:40 [I 2022-10-27 03:07:40] ...trap/engines/engine.py.229: process: 0.56852 | load: 0.00249 [I 2022-10-27 03:07:40] ...trap/engines/engine.py.230: loss: 0.17909 [I 2022-10-27 03:07:40] ...optimizers/trijoint.py.151: total_norm: 19.960428 [I 2022-10-27 03:07:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 1380/2304 [I 2022-10-27 03:07:45] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:35 [I 2022-10-27 03:07:45] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00251 [I 2022-10-27 03:07:45] ...trap/engines/engine.py.230: loss: 0.27617 [I 2022-10-27 03:07:45] ...optimizers/trijoint.py.151: total_norm: 24.777277 [I 2022-10-27 03:07:51] ...trap/engines/engine.py.224: train: epoch 44 | batch 1390/2304 [I 2022-10-27 03:07:51] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:24 [I 2022-10-27 03:07:51] ...trap/engines/engine.py.229: process: 0.54505 | load: 0.00242 [I 2022-10-27 03:07:51] ...trap/engines/engine.py.230: loss: 0.18505 [I 2022-10-27 03:07:51] ...optimizers/trijoint.py.151: total_norm: 20.061016 [I 2022-10-27 03:07:56] ...trap/engines/engine.py.224: train: epoch 44 | batch 1400/2304 [I 2022-10-27 03:07:56] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:23 [I 2022-10-27 03:07:56] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00242 [I 2022-10-27 03:07:56] ...trap/engines/engine.py.230: loss: 0.11943 [I 2022-10-27 03:07:56] ...optimizers/trijoint.py.151: total_norm: 8.866001 [I 2022-10-27 03:08:02] ...trap/engines/engine.py.224: train: epoch 44 | batch 1410/2304 [I 2022-10-27 03:08:02] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:13 [I 2022-10-27 03:08:02] ...trap/engines/engine.py.229: process: 0.54484 | load: 0.00245 [I 2022-10-27 03:08:02] ...trap/engines/engine.py.230: loss: 0.18896 [I 2022-10-27 03:08:02] ...optimizers/trijoint.py.151: total_norm: 16.991909 [I 2022-10-27 03:08:08] ...trap/engines/engine.py.224: train: epoch 44 | batch 1420/2304 [I 2022-10-27 03:08:08] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:08:16 [I 2022-10-27 03:08:08] ...trap/engines/engine.py.229: process: 0.56221 | load: 0.00243 [I 2022-10-27 03:08:08] ...trap/engines/engine.py.230: loss: 0.22009 [I 2022-10-27 03:08:08] ...optimizers/trijoint.py.151: total_norm: 15.131104 [I 2022-10-27 03:08:13] ...trap/engines/engine.py.224: train: epoch 44 | batch 1430/2304 [I 2022-10-27 03:08:13] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:08:06 [I 2022-10-27 03:08:13] ...trap/engines/engine.py.229: process: 0.56708 | load: 0.00244 [I 2022-10-27 03:08:13] ...trap/engines/engine.py.230: loss: 0.17512 [I 2022-10-27 03:08:13] ...optimizers/trijoint.py.151: total_norm: 47.996666 [I 2022-10-27 03:08:19] ...trap/engines/engine.py.224: train: epoch 44 | batch 1440/2304 [I 2022-10-27 03:08:19] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:58 [I 2022-10-27 03:08:19] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00250 [I 2022-10-27 03:08:19] ...trap/engines/engine.py.230: loss: 0.13443 [I 2022-10-27 03:08:19] ...optimizers/trijoint.py.151: total_norm: 16.343805 [I 2022-10-27 03:08:24] ...trap/engines/engine.py.224: train: epoch 44 | batch 1450/2304 [I 2022-10-27 03:08:24] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:53 [I 2022-10-27 03:08:24] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00247 [I 2022-10-27 03:08:24] ...trap/engines/engine.py.230: loss: 0.13913 [I 2022-10-27 03:08:24] ...optimizers/trijoint.py.151: total_norm: 22.732559 [I 2022-10-27 03:08:30] ...trap/engines/engine.py.224: train: epoch 44 | batch 1460/2304 [I 2022-10-27 03:08:30] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:50 [I 2022-10-27 03:08:30] ...trap/engines/engine.py.229: process: 0.56599 | load: 0.00251 [I 2022-10-27 03:08:30] ...trap/engines/engine.py.230: loss: 0.15652 [I 2022-10-27 03:08:30] ...optimizers/trijoint.py.151: total_norm: 22.894846 [I 2022-10-27 03:08:35] ...trap/engines/engine.py.224: train: epoch 44 | batch 1470/2304 [I 2022-10-27 03:08:35] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:43 [I 2022-10-27 03:08:35] ...trap/engines/engine.py.229: process: 0.54538 | load: 0.00252 [I 2022-10-27 03:08:35] ...trap/engines/engine.py.230: loss: 0.12649 [I 2022-10-27 03:08:35] ...optimizers/trijoint.py.151: total_norm: 7.830833 [I 2022-10-27 03:08:41] ...trap/engines/engine.py.224: train: epoch 44 | batch 1480/2304 [I 2022-10-27 03:08:41] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:39 [I 2022-10-27 03:08:41] ...trap/engines/engine.py.229: process: 0.54221 | load: 0.00245 [I 2022-10-27 03:08:41] ...trap/engines/engine.py.230: loss: 0.15668 [I 2022-10-27 03:08:41] ...optimizers/trijoint.py.151: total_norm: 19.689943 [I 2022-10-27 03:08:47] ...trap/engines/engine.py.224: train: epoch 44 | batch 1490/2304 [I 2022-10-27 03:08:47] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:32 [I 2022-10-27 03:08:47] ...trap/engines/engine.py.229: process: 0.55834 | load: 0.00252 [I 2022-10-27 03:08:47] ...trap/engines/engine.py.230: loss: 0.11817 [I 2022-10-27 03:08:47] ...optimizers/trijoint.py.151: total_norm: 15.099657 [I 2022-10-27 03:08:52] ...trap/engines/engine.py.224: train: epoch 44 | batch 1500/2304 [I 2022-10-27 03:08:52] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:28 [I 2022-10-27 03:08:52] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00251 [I 2022-10-27 03:08:52] ...trap/engines/engine.py.230: loss: 0.16528 [I 2022-10-27 03:08:52] ...optimizers/trijoint.py.151: total_norm: 19.796675 [I 2022-10-27 03:08:58] ...trap/engines/engine.py.224: train: epoch 44 | batch 1510/2304 [I 2022-10-27 03:08:58] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:24 [I 2022-10-27 03:08:58] ...trap/engines/engine.py.229: process: 0.56806 | load: 0.00248 [I 2022-10-27 03:08:58] ...trap/engines/engine.py.230: loss: 0.20590 [I 2022-10-27 03:08:58] ...optimizers/trijoint.py.151: total_norm: 18.010336 [I 2022-10-27 03:09:03] ...trap/engines/engine.py.224: train: epoch 44 | batch 1520/2304 [I 2022-10-27 03:09:03] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:15 [I 2022-10-27 03:09:03] ...trap/engines/engine.py.229: process: 0.55817 | load: 0.00251 [I 2022-10-27 03:09:03] ...trap/engines/engine.py.230: loss: 0.12477 [I 2022-10-27 03:09:03] ...optimizers/trijoint.py.151: total_norm: 17.107660 [I 2022-10-27 03:09:09] ...trap/engines/engine.py.224: train: epoch 44 | batch 1530/2304 [I 2022-10-27 03:09:09] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:07:11 [I 2022-10-27 03:09:09] ...trap/engines/engine.py.229: process: 0.55552 | load: 0.00246 [I 2022-10-27 03:09:09] ...trap/engines/engine.py.230: loss: 0.18907 [I 2022-10-27 03:09:09] ...optimizers/trijoint.py.151: total_norm: 11.320907 [I 2022-10-27 03:09:15] ...trap/engines/engine.py.224: train: epoch 44 | batch 1540/2304 [I 2022-10-27 03:09:15] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:07:17 [I 2022-10-27 03:09:15] ...trap/engines/engine.py.229: process: 0.64588 | load: 0.00246 [I 2022-10-27 03:09:15] ...trap/engines/engine.py.230: loss: 0.15878 [I 2022-10-27 03:09:15] ...optimizers/trijoint.py.151: total_norm: 15.707649 [I 2022-10-27 03:09:20] ...trap/engines/engine.py.224: train: epoch 44 | batch 1550/2304 [I 2022-10-27 03:09:20] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:07:00 [I 2022-10-27 03:09:20] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00248 [I 2022-10-27 03:09:20] ...trap/engines/engine.py.230: loss: 0.13368 [I 2022-10-27 03:09:20] ...optimizers/trijoint.py.151: total_norm: 11.720806 [I 2022-10-27 03:09:26] ...trap/engines/engine.py.224: train: epoch 44 | batch 1560/2304 [I 2022-10-27 03:09:26] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:55 [I 2022-10-27 03:09:26] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00250 [I 2022-10-27 03:09:26] ...trap/engines/engine.py.230: loss: 0.18911 [I 2022-10-27 03:09:26] ...optimizers/trijoint.py.151: total_norm: 10.441572 [I 2022-10-27 03:09:31] ...trap/engines/engine.py.224: train: epoch 44 | batch 1570/2304 [I 2022-10-27 03:09:31] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:48 [I 2022-10-27 03:09:31] ...trap/engines/engine.py.229: process: 0.56907 | load: 0.00252 [I 2022-10-27 03:09:31] ...trap/engines/engine.py.230: loss: 0.15205 [I 2022-10-27 03:09:31] ...optimizers/trijoint.py.151: total_norm: 29.315622 [I 2022-10-27 03:09:37] ...trap/engines/engine.py.224: train: epoch 44 | batch 1580/2304 [I 2022-10-27 03:09:37] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:43 [I 2022-10-27 03:09:37] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00241 [I 2022-10-27 03:09:37] ...trap/engines/engine.py.230: loss: 0.12811 [I 2022-10-27 03:09:37] ...optimizers/trijoint.py.151: total_norm: 11.601230 [I 2022-10-27 03:09:43] ...trap/engines/engine.py.224: train: epoch 44 | batch 1590/2304 [I 2022-10-27 03:09:43] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:40 [I 2022-10-27 03:09:43] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00281 [I 2022-10-27 03:09:43] ...trap/engines/engine.py.230: loss: 0.11851 [I 2022-10-27 03:09:43] ...optimizers/trijoint.py.151: total_norm: 16.319489 [I 2022-10-27 03:09:48] ...trap/engines/engine.py.224: train: epoch 44 | batch 1600/2304 [I 2022-10-27 03:09:48] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:34 [I 2022-10-27 03:09:48] ...trap/engines/engine.py.229: process: 0.55030 | load: 0.00265 [I 2022-10-27 03:09:48] ...trap/engines/engine.py.230: loss: 0.16942 [I 2022-10-27 03:09:48] ...optimizers/trijoint.py.151: total_norm: 16.473986 [I 2022-10-27 03:09:54] ...trap/engines/engine.py.224: train: epoch 44 | batch 1610/2304 [I 2022-10-27 03:09:54] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:29 [I 2022-10-27 03:09:54] ...trap/engines/engine.py.229: process: 0.54782 | load: 0.00261 [I 2022-10-27 03:09:54] ...trap/engines/engine.py.230: loss: 0.23930 [I 2022-10-27 03:09:54] ...optimizers/trijoint.py.151: total_norm: 13.367125 [I 2022-10-27 03:09:59] ...trap/engines/engine.py.224: train: epoch 44 | batch 1620/2304 [I 2022-10-27 03:09:59] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:22 [I 2022-10-27 03:09:59] ...trap/engines/engine.py.229: process: 0.56337 | load: 0.00252 [I 2022-10-27 03:09:59] ...trap/engines/engine.py.230: loss: 0.24060 [I 2022-10-27 03:09:59] ...optimizers/trijoint.py.151: total_norm: 21.199741 [I 2022-10-27 03:10:05] ...trap/engines/engine.py.224: train: epoch 44 | batch 1630/2304 [I 2022-10-27 03:10:05] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:16 [I 2022-10-27 03:10:05] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00289 [I 2022-10-27 03:10:05] ...trap/engines/engine.py.230: loss: 0.15744 [I 2022-10-27 03:10:05] ...optimizers/trijoint.py.151: total_norm: 18.983421 [I 2022-10-27 03:10:11] ...trap/engines/engine.py.224: train: epoch 44 | batch 1640/2304 [I 2022-10-27 03:10:11] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:06:08 [I 2022-10-27 03:10:11] ...trap/engines/engine.py.229: process: 0.55472 | load: 0.00258 [I 2022-10-27 03:10:11] ...trap/engines/engine.py.230: loss: 0.25191 [I 2022-10-27 03:10:11] ...optimizers/trijoint.py.151: total_norm: 23.846376 [I 2022-10-27 03:10:16] ...trap/engines/engine.py.224: train: epoch 44 | batch 1650/2304 [I 2022-10-27 03:10:16] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:06:04 [I 2022-10-27 03:10:16] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00277 [I 2022-10-27 03:10:16] ...trap/engines/engine.py.230: loss: 0.16120 [I 2022-10-27 03:10:16] ...optimizers/trijoint.py.151: total_norm: 8.876250 [I 2022-10-27 03:10:22] ...trap/engines/engine.py.224: train: epoch 44 | batch 1660/2304 [I 2022-10-27 03:10:22] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:55 [I 2022-10-27 03:10:22] ...trap/engines/engine.py.229: process: 0.54317 | load: 0.00251 [I 2022-10-27 03:10:22] ...trap/engines/engine.py.230: loss: 0.13396 [I 2022-10-27 03:10:22] ...optimizers/trijoint.py.151: total_norm: 10.812675 [I 2022-10-27 03:10:27] ...trap/engines/engine.py.224: train: epoch 44 | batch 1670/2304 [I 2022-10-27 03:10:27] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:59 [I 2022-10-27 03:10:27] ...trap/engines/engine.py.229: process: 0.55776 | load: 0.00244 [I 2022-10-27 03:10:27] ...trap/engines/engine.py.230: loss: 0.20731 [I 2022-10-27 03:10:27] ...optimizers/trijoint.py.151: total_norm: 43.941555 [I 2022-10-27 03:10:33] ...trap/engines/engine.py.224: train: epoch 44 | batch 1680/2304 [I 2022-10-27 03:10:33] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:47 [I 2022-10-27 03:10:33] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00255 [I 2022-10-27 03:10:33] ...trap/engines/engine.py.230: loss: 0.16043 [I 2022-10-27 03:10:33] ...optimizers/trijoint.py.151: total_norm: 60.671917 [I 2022-10-27 03:10:39] ...trap/engines/engine.py.224: train: epoch 44 | batch 1690/2304 [I 2022-10-27 03:10:39] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:42 [I 2022-10-27 03:10:39] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00251 [I 2022-10-27 03:10:39] ...trap/engines/engine.py.230: loss: 0.14085 [I 2022-10-27 03:10:39] ...optimizers/trijoint.py.151: total_norm: 11.803919 [I 2022-10-27 03:10:44] ...trap/engines/engine.py.224: train: epoch 44 | batch 1700/2304 [I 2022-10-27 03:10:44] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:34 [I 2022-10-27 03:10:44] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00248 [I 2022-10-27 03:10:44] ...trap/engines/engine.py.230: loss: 0.17190 [I 2022-10-27 03:10:44] ...optimizers/trijoint.py.151: total_norm: 21.005444 [I 2022-10-27 03:10:50] ...trap/engines/engine.py.224: train: epoch 44 | batch 1710/2304 [I 2022-10-27 03:10:50] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:28 [I 2022-10-27 03:10:50] ...trap/engines/engine.py.229: process: 0.54159 | load: 0.00247 [I 2022-10-27 03:10:50] ...trap/engines/engine.py.230: loss: 0.24859 [I 2022-10-27 03:10:50] ...optimizers/trijoint.py.151: total_norm: 14.594427 [I 2022-10-27 03:10:55] ...trap/engines/engine.py.224: train: epoch 44 | batch 1720/2304 [I 2022-10-27 03:10:55] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:25 [I 2022-10-27 03:10:55] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00243 [I 2022-10-27 03:10:55] ...trap/engines/engine.py.230: loss: 0.21081 [I 2022-10-27 03:10:55] ...optimizers/trijoint.py.151: total_norm: 22.428495 [I 2022-10-27 03:11:01] ...trap/engines/engine.py.224: train: epoch 44 | batch 1730/2304 [I 2022-10-27 03:11:01] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:18 [I 2022-10-27 03:11:01] ...trap/engines/engine.py.229: process: 0.55433 | load: 0.00247 [I 2022-10-27 03:11:01] ...trap/engines/engine.py.230: loss: 0.18752 [I 2022-10-27 03:11:01] ...optimizers/trijoint.py.151: total_norm: 20.416374 [I 2022-10-27 03:11:06] ...trap/engines/engine.py.224: train: epoch 44 | batch 1740/2304 [I 2022-10-27 03:11:06] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:11 [I 2022-10-27 03:11:06] ...trap/engines/engine.py.229: process: 0.54222 | load: 0.00243 [I 2022-10-27 03:11:06] ...trap/engines/engine.py.230: loss: 0.20317 [I 2022-10-27 03:11:06] ...optimizers/trijoint.py.151: total_norm: 16.427544 [I 2022-10-27 03:11:12] ...trap/engines/engine.py.224: train: epoch 44 | batch 1750/2304 [I 2022-10-27 03:11:12] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:05:07 [I 2022-10-27 03:11:12] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00245 [I 2022-10-27 03:11:12] ...trap/engines/engine.py.230: loss: 0.13597 [I 2022-10-27 03:11:12] ...optimizers/trijoint.py.151: total_norm: 15.365932 [I 2022-10-27 03:11:18] ...trap/engines/engine.py.224: train: epoch 44 | batch 1760/2304 [I 2022-10-27 03:11:18] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:05:01 [I 2022-10-27 03:11:18] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00241 [I 2022-10-27 03:11:18] ...trap/engines/engine.py.230: loss: 0.14542 [I 2022-10-27 03:11:18] ...optimizers/trijoint.py.151: total_norm: 15.564760 [I 2022-10-27 03:11:23] ...trap/engines/engine.py.224: train: epoch 44 | batch 1770/2304 [I 2022-10-27 03:11:23] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:57 [I 2022-10-27 03:11:23] ...trap/engines/engine.py.229: process: 0.57183 | load: 0.00245 [I 2022-10-27 03:11:23] ...trap/engines/engine.py.230: loss: 0.17916 [I 2022-10-27 03:11:23] ...optimizers/trijoint.py.151: total_norm: 15.884100 [I 2022-10-27 03:11:29] ...trap/engines/engine.py.224: train: epoch 44 | batch 1780/2304 [I 2022-10-27 03:11:29] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:51 [I 2022-10-27 03:11:29] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00242 [I 2022-10-27 03:11:29] ...trap/engines/engine.py.230: loss: 0.22510 [I 2022-10-27 03:11:29] ...optimizers/trijoint.py.151: total_norm: 26.676254 [I 2022-10-27 03:11:34] ...trap/engines/engine.py.224: train: epoch 44 | batch 1790/2304 [I 2022-10-27 03:11:34] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:45 [I 2022-10-27 03:11:34] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00249 [I 2022-10-27 03:11:34] ...trap/engines/engine.py.230: loss: 0.20077 [I 2022-10-27 03:11:34] ...optimizers/trijoint.py.151: total_norm: 16.433784 [I 2022-10-27 03:11:40] ...trap/engines/engine.py.224: train: epoch 44 | batch 1800/2304 [I 2022-10-27 03:11:40] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:41 [I 2022-10-27 03:11:40] ...trap/engines/engine.py.229: process: 0.56182 | load: 0.00244 [I 2022-10-27 03:11:40] ...trap/engines/engine.py.230: loss: 0.14485 [I 2022-10-27 03:11:40] ...optimizers/trijoint.py.151: total_norm: 33.031677 [I 2022-10-27 03:11:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 1810/2304 [I 2022-10-27 03:11:45] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:34 [I 2022-10-27 03:11:45] ...trap/engines/engine.py.229: process: 0.56522 | load: 0.00244 [I 2022-10-27 03:11:45] ...trap/engines/engine.py.230: loss: 0.18439 [I 2022-10-27 03:11:45] ...optimizers/trijoint.py.151: total_norm: 10.215501 [I 2022-10-27 03:11:51] ...trap/engines/engine.py.224: train: epoch 44 | batch 1820/2304 [I 2022-10-27 03:11:51] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:29 [I 2022-10-27 03:11:51] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00245 [I 2022-10-27 03:11:51] ...trap/engines/engine.py.230: loss: 0.17139 [I 2022-10-27 03:11:51] ...optimizers/trijoint.py.151: total_norm: 22.812761 [I 2022-10-27 03:11:56] ...trap/engines/engine.py.224: train: epoch 44 | batch 1830/2304 [I 2022-10-27 03:11:56] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:22 [I 2022-10-27 03:11:56] ...trap/engines/engine.py.229: process: 0.54562 | load: 0.00242 [I 2022-10-27 03:11:56] ...trap/engines/engine.py.230: loss: 0.12915 [I 2022-10-27 03:11:56] ...optimizers/trijoint.py.151: total_norm: 16.843298 [I 2022-10-27 03:12:02] ...trap/engines/engine.py.224: train: epoch 44 | batch 1840/2304 [I 2022-10-27 03:12:02] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:16 [I 2022-10-27 03:12:02] ...trap/engines/engine.py.229: process: 0.54304 | load: 0.00241 [I 2022-10-27 03:12:02] ...trap/engines/engine.py.230: loss: 0.17955 [I 2022-10-27 03:12:02] ...optimizers/trijoint.py.151: total_norm: 24.994890 [I 2022-10-27 03:12:08] ...trap/engines/engine.py.224: train: epoch 44 | batch 1850/2304 [I 2022-10-27 03:12:08] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:04:14 [I 2022-10-27 03:12:08] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00245 [I 2022-10-27 03:12:08] ...trap/engines/engine.py.230: loss: 0.21129 [I 2022-10-27 03:12:08] ...optimizers/trijoint.py.151: total_norm: 16.470316 [I 2022-10-27 03:12:13] ...trap/engines/engine.py.224: train: epoch 44 | batch 1860/2304 [I 2022-10-27 03:12:13] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:04:05 [I 2022-10-27 03:12:13] ...trap/engines/engine.py.229: process: 0.54789 | load: 0.00248 [I 2022-10-27 03:12:13] ...trap/engines/engine.py.230: loss: 0.20108 [I 2022-10-27 03:12:13] ...optimizers/trijoint.py.151: total_norm: 16.662949 [I 2022-10-27 03:12:19] ...trap/engines/engine.py.224: train: epoch 44 | batch 1870/2304 [I 2022-10-27 03:12:19] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:04:00 [I 2022-10-27 03:12:19] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00245 [I 2022-10-27 03:12:19] ...trap/engines/engine.py.230: loss: 0.19214 [I 2022-10-27 03:12:19] ...optimizers/trijoint.py.151: total_norm: 18.085463 [I 2022-10-27 03:12:24] ...trap/engines/engine.py.224: train: epoch 44 | batch 1880/2304 [I 2022-10-27 03:12:24] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:55 [I 2022-10-27 03:12:24] ...trap/engines/engine.py.229: process: 0.56080 | load: 0.00245 [I 2022-10-27 03:12:24] ...trap/engines/engine.py.230: loss: 0.23375 [I 2022-10-27 03:12:24] ...optimizers/trijoint.py.151: total_norm: 13.599724 [I 2022-10-27 03:12:30] ...trap/engines/engine.py.224: train: epoch 44 | batch 1890/2304 [I 2022-10-27 03:12:30] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:50 [I 2022-10-27 03:12:30] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00247 [I 2022-10-27 03:12:30] ...trap/engines/engine.py.230: loss: 0.23764 [I 2022-10-27 03:12:30] ...optimizers/trijoint.py.151: total_norm: 11.960361 [I 2022-10-27 03:12:35] ...trap/engines/engine.py.224: train: epoch 44 | batch 1900/2304 [I 2022-10-27 03:12:35] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:44 [I 2022-10-27 03:12:35] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00244 [I 2022-10-27 03:12:35] ...trap/engines/engine.py.230: loss: 0.15009 [I 2022-10-27 03:12:35] ...optimizers/trijoint.py.151: total_norm: 19.347633 [I 2022-10-27 03:12:41] ...trap/engines/engine.py.224: train: epoch 44 | batch 1910/2304 [I 2022-10-27 03:12:41] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:39 [I 2022-10-27 03:12:41] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00245 [I 2022-10-27 03:12:41] ...trap/engines/engine.py.230: loss: 0.20814 [I 2022-10-27 03:12:41] ...optimizers/trijoint.py.151: total_norm: 16.188919 [I 2022-10-27 03:12:47] ...trap/engines/engine.py.224: train: epoch 44 | batch 1920/2304 [I 2022-10-27 03:12:47] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:35 [I 2022-10-27 03:12:47] ...trap/engines/engine.py.229: process: 0.56833 | load: 0.00245 [I 2022-10-27 03:12:47] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-27 03:12:47] ...optimizers/trijoint.py.151: total_norm: 23.401690 [I 2022-10-27 03:12:52] ...trap/engines/engine.py.224: train: epoch 44 | batch 1930/2304 [I 2022-10-27 03:12:52] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:29 [I 2022-10-27 03:12:52] ...trap/engines/engine.py.229: process: 0.56384 | load: 0.00281 [I 2022-10-27 03:12:52] ...trap/engines/engine.py.230: loss: 0.18054 [I 2022-10-27 03:12:52] ...optimizers/trijoint.py.151: total_norm: 20.673990 [I 2022-10-27 03:12:58] ...trap/engines/engine.py.224: train: epoch 44 | batch 1940/2304 [I 2022-10-27 03:12:58] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:24 [I 2022-10-27 03:12:58] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00264 [I 2022-10-27 03:12:58] ...trap/engines/engine.py.230: loss: 0.11991 [I 2022-10-27 03:12:58] ...optimizers/trijoint.py.151: total_norm: 9.207795 [I 2022-10-27 03:13:03] ...trap/engines/engine.py.224: train: epoch 44 | batch 1950/2304 [I 2022-10-27 03:13:03] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:18 [I 2022-10-27 03:13:03] ...trap/engines/engine.py.229: process: 0.55578 | load: 0.00255 [I 2022-10-27 03:13:03] ...trap/engines/engine.py.230: loss: 0.24073 [I 2022-10-27 03:13:03] ...optimizers/trijoint.py.151: total_norm: 15.875450 [I 2022-10-27 03:13:09] ...trap/engines/engine.py.224: train: epoch 44 | batch 1960/2304 [I 2022-10-27 03:13:09] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:11 [I 2022-10-27 03:13:09] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00254 [I 2022-10-27 03:13:09] ...trap/engines/engine.py.230: loss: 0.14610 [I 2022-10-27 03:13:09] ...optimizers/trijoint.py.151: total_norm: 20.175707 [I 2022-10-27 03:13:15] ...trap/engines/engine.py.224: train: epoch 44 | batch 1970/2304 [I 2022-10-27 03:13:15] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:03:04 [I 2022-10-27 03:13:15] ...trap/engines/engine.py.229: process: 0.54465 | load: 0.00254 [I 2022-10-27 03:13:15] ...trap/engines/engine.py.230: loss: 0.21005 [I 2022-10-27 03:13:15] ...optimizers/trijoint.py.151: total_norm: 21.346659 [I 2022-10-27 03:13:20] ...trap/engines/engine.py.224: train: epoch 44 | batch 1980/2304 [I 2022-10-27 03:13:20] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:03:01 [I 2022-10-27 03:13:20] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00246 [I 2022-10-27 03:13:20] ...trap/engines/engine.py.230: loss: 0.23225 [I 2022-10-27 03:13:20] ...optimizers/trijoint.py.151: total_norm: 25.987925 [I 2022-10-27 03:13:26] ...trap/engines/engine.py.224: train: epoch 44 | batch 1990/2304 [I 2022-10-27 03:13:26] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:54 [I 2022-10-27 03:13:26] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00277 [I 2022-10-27 03:13:26] ...trap/engines/engine.py.230: loss: 0.18010 [I 2022-10-27 03:13:26] ...optimizers/trijoint.py.151: total_norm: 14.410398 [I 2022-10-27 03:13:31] ...trap/engines/engine.py.224: train: epoch 44 | batch 2000/2304 [I 2022-10-27 03:13:31] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:48 [I 2022-10-27 03:13:31] ...trap/engines/engine.py.229: process: 0.54583 | load: 0.00242 [I 2022-10-27 03:13:31] ...trap/engines/engine.py.230: loss: 0.19465 [I 2022-10-27 03:13:31] ...optimizers/trijoint.py.151: total_norm: 27.083828 [I 2022-10-27 03:13:37] ...trap/engines/engine.py.224: train: epoch 44 | batch 2010/2304 [I 2022-10-27 03:13:37] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:42 [I 2022-10-27 03:13:37] ...trap/engines/engine.py.229: process: 0.54740 | load: 0.00244 [I 2022-10-27 03:13:37] ...trap/engines/engine.py.230: loss: 0.19914 [I 2022-10-27 03:13:37] ...optimizers/trijoint.py.151: total_norm: 16.982864 [I 2022-10-27 03:13:42] ...trap/engines/engine.py.224: train: epoch 44 | batch 2020/2304 [I 2022-10-27 03:13:42] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:36 [I 2022-10-27 03:13:42] ...trap/engines/engine.py.229: process: 0.54137 | load: 0.00244 [I 2022-10-27 03:13:42] ...trap/engines/engine.py.230: loss: 0.13179 [I 2022-10-27 03:13:42] ...optimizers/trijoint.py.151: total_norm: 16.488640 [I 2022-10-27 03:13:48] ...trap/engines/engine.py.224: train: epoch 44 | batch 2030/2304 [I 2022-10-27 03:13:48] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:33 [I 2022-10-27 03:13:48] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00256 [I 2022-10-27 03:13:48] ...trap/engines/engine.py.230: loss: 0.19403 [I 2022-10-27 03:13:48] ...optimizers/trijoint.py.151: total_norm: 12.774934 [I 2022-10-27 03:13:54] ...trap/engines/engine.py.224: train: epoch 44 | batch 2040/2304 [I 2022-10-27 03:13:54] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:28 [I 2022-10-27 03:13:54] ...trap/engines/engine.py.229: process: 0.56760 | load: 0.00258 [I 2022-10-27 03:13:54] ...trap/engines/engine.py.230: loss: 0.17803 [I 2022-10-27 03:13:54] ...optimizers/trijoint.py.151: total_norm: 12.837038 [I 2022-10-27 03:13:59] ...trap/engines/engine.py.224: train: epoch 44 | batch 2050/2304 [I 2022-10-27 03:13:59] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:22 [I 2022-10-27 03:13:59] ...trap/engines/engine.py.229: process: 0.56585 | load: 0.00253 [I 2022-10-27 03:13:59] ...trap/engines/engine.py.230: loss: 0.12914 [I 2022-10-27 03:13:59] ...optimizers/trijoint.py.151: total_norm: 14.567781 [I 2022-10-27 03:14:05] ...trap/engines/engine.py.224: train: epoch 44 | batch 2060/2304 [I 2022-10-27 03:14:05] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:16 [I 2022-10-27 03:14:05] ...trap/engines/engine.py.229: process: 0.56155 | load: 0.00247 [I 2022-10-27 03:14:05] ...trap/engines/engine.py.230: loss: 0.12547 [I 2022-10-27 03:14:05] ...optimizers/trijoint.py.151: total_norm: 16.203375 [I 2022-10-27 03:14:11] ...trap/engines/engine.py.224: train: epoch 44 | batch 2070/2304 [I 2022-10-27 03:14:11] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:11 [I 2022-10-27 03:14:11] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00251 [I 2022-10-27 03:14:11] ...trap/engines/engine.py.230: loss: 0.16744 [I 2022-10-27 03:14:11] ...optimizers/trijoint.py.151: total_norm: 14.332524 [I 2022-10-27 03:14:16] ...trap/engines/engine.py.224: train: epoch 44 | batch 2080/2304 [I 2022-10-27 03:14:16] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:05 [I 2022-10-27 03:14:16] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00259 [I 2022-10-27 03:14:16] ...trap/engines/engine.py.230: loss: 0.10676 [I 2022-10-27 03:14:16] ...optimizers/trijoint.py.151: total_norm: 8.241326 [I 2022-10-27 03:14:22] ...trap/engines/engine.py.224: train: epoch 44 | batch 2090/2304 [I 2022-10-27 03:14:22] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:02:00 [I 2022-10-27 03:14:22] ...trap/engines/engine.py.229: process: 0.56764 | load: 0.00247 [I 2022-10-27 03:14:22] ...trap/engines/engine.py.230: loss: 0.21044 [I 2022-10-27 03:14:22] ...optimizers/trijoint.py.151: total_norm: 10.911382 [I 2022-10-27 03:14:27] ...trap/engines/engine.py.224: train: epoch 44 | batch 2100/2304 [I 2022-10-27 03:14:27] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:53 [I 2022-10-27 03:14:27] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00250 [I 2022-10-27 03:14:27] ...trap/engines/engine.py.230: loss: 0.19446 [I 2022-10-27 03:14:27] ...optimizers/trijoint.py.151: total_norm: 20.002409 [I 2022-10-27 03:14:33] ...trap/engines/engine.py.224: train: epoch 44 | batch 2110/2304 [I 2022-10-27 03:14:33] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:48 [I 2022-10-27 03:14:33] ...trap/engines/engine.py.229: process: 0.55175 | load: 0.00252 [I 2022-10-27 03:14:33] ...trap/engines/engine.py.230: loss: 0.26507 [I 2022-10-27 03:14:33] ...optimizers/trijoint.py.151: total_norm: 15.566355 [I 2022-10-27 03:14:38] ...trap/engines/engine.py.224: train: epoch 44 | batch 2120/2304 [I 2022-10-27 03:14:38] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:42 [I 2022-10-27 03:14:38] ...trap/engines/engine.py.229: process: 0.55863 | load: 0.00248 [I 2022-10-27 03:14:38] ...trap/engines/engine.py.230: loss: 0.21994 [I 2022-10-27 03:14:38] ...optimizers/trijoint.py.151: total_norm: 21.259323 [I 2022-10-27 03:14:44] ...trap/engines/engine.py.224: train: epoch 44 | batch 2130/2304 [I 2022-10-27 03:14:44] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:37 [I 2022-10-27 03:14:44] ...trap/engines/engine.py.229: process: 0.56283 | load: 0.00257 [I 2022-10-27 03:14:44] ...trap/engines/engine.py.230: loss: 0.14801 [I 2022-10-27 03:14:44] ...optimizers/trijoint.py.151: total_norm: 13.651199 [I 2022-10-27 03:14:50] ...trap/engines/engine.py.224: train: epoch 44 | batch 2140/2304 [I 2022-10-27 03:14:50] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:31 [I 2022-10-27 03:14:50] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00250 [I 2022-10-27 03:14:50] ...trap/engines/engine.py.230: loss: 0.12224 [I 2022-10-27 03:14:50] ...optimizers/trijoint.py.151: total_norm: 9.529154 [I 2022-10-27 03:14:55] ...trap/engines/engine.py.224: train: epoch 44 | batch 2150/2304 [I 2022-10-27 03:14:55] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:26 [I 2022-10-27 03:14:55] ...trap/engines/engine.py.229: process: 0.56484 | load: 0.00253 [I 2022-10-27 03:14:55] ...trap/engines/engine.py.230: loss: 0.25438 [I 2022-10-27 03:14:55] ...optimizers/trijoint.py.151: total_norm: 20.168915 [I 2022-10-27 03:15:01] ...trap/engines/engine.py.224: train: epoch 44 | batch 2160/2304 [I 2022-10-27 03:15:01] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:20 [I 2022-10-27 03:15:01] ...trap/engines/engine.py.229: process: 0.57031 | load: 0.00251 [I 2022-10-27 03:15:01] ...trap/engines/engine.py.230: loss: 0.17426 [I 2022-10-27 03:15:01] ...optimizers/trijoint.py.151: total_norm: 14.222505 [I 2022-10-27 03:15:06] ...trap/engines/engine.py.224: train: epoch 44 | batch 2170/2304 [I 2022-10-27 03:15:06] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:01:14 [I 2022-10-27 03:15:06] ...trap/engines/engine.py.229: process: 0.55298 | load: 0.00249 [I 2022-10-27 03:15:06] ...trap/engines/engine.py.230: loss: 0.17476 [I 2022-10-27 03:15:06] ...optimizers/trijoint.py.151: total_norm: 16.764439 [I 2022-10-27 03:15:12] ...trap/engines/engine.py.224: train: epoch 44 | batch 2180/2304 [I 2022-10-27 03:15:12] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:08 [I 2022-10-27 03:15:12] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00243 [I 2022-10-27 03:15:12] ...trap/engines/engine.py.230: loss: 0.14025 [I 2022-10-27 03:15:12] ...optimizers/trijoint.py.151: total_norm: 17.755604 [I 2022-10-27 03:15:17] ...trap/engines/engine.py.224: train: epoch 44 | batch 2190/2304 [I 2022-10-27 03:15:17] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:03 [I 2022-10-27 03:15:17] ...trap/engines/engine.py.229: process: 0.56154 | load: 0.00241 [I 2022-10-27 03:15:17] ...trap/engines/engine.py.230: loss: 0.14251 [I 2022-10-27 03:15:17] ...optimizers/trijoint.py.151: total_norm: 14.433016 [I 2022-10-27 03:15:23] ...trap/engines/engine.py.224: train: epoch 44 | batch 2200/2304 [I 2022-10-27 03:15:23] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:57 [I 2022-10-27 03:15:23] ...trap/engines/engine.py.229: process: 0.56242 | load: 0.00247 [I 2022-10-27 03:15:23] ...trap/engines/engine.py.230: loss: 0.23115 [I 2022-10-27 03:15:23] ...optimizers/trijoint.py.151: total_norm: 22.790579 [I 2022-10-27 03:15:29] ...trap/engines/engine.py.224: train: epoch 44 | batch 2210/2304 [I 2022-10-27 03:15:29] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:52 [I 2022-10-27 03:15:29] ...trap/engines/engine.py.229: process: 0.55804 | load: 0.00247 [I 2022-10-27 03:15:29] ...trap/engines/engine.py.230: loss: 0.25227 [I 2022-10-27 03:15:29] ...optimizers/trijoint.py.151: total_norm: 22.232969 [I 2022-10-27 03:15:34] ...trap/engines/engine.py.224: train: epoch 44 | batch 2220/2304 [I 2022-10-27 03:15:34] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:46 [I 2022-10-27 03:15:34] ...trap/engines/engine.py.229: process: 0.56833 | load: 0.00243 [I 2022-10-27 03:15:34] ...trap/engines/engine.py.230: loss: 0.12702 [I 2022-10-27 03:15:34] ...optimizers/trijoint.py.151: total_norm: 12.998353 [I 2022-10-27 03:15:40] ...trap/engines/engine.py.224: train: epoch 44 | batch 2230/2304 [I 2022-10-27 03:15:40] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:41 [I 2022-10-27 03:15:40] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00243 [I 2022-10-27 03:15:40] ...trap/engines/engine.py.230: loss: 0.17068 [I 2022-10-27 03:15:40] ...optimizers/trijoint.py.151: total_norm: 36.182789 [I 2022-10-27 03:15:45] ...trap/engines/engine.py.224: train: epoch 44 | batch 2240/2304 [I 2022-10-27 03:15:45] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:35 [I 2022-10-27 03:15:45] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00250 [I 2022-10-27 03:15:45] ...trap/engines/engine.py.230: loss: 0.17542 [I 2022-10-27 03:15:45] ...optimizers/trijoint.py.151: total_norm: 17.836235 [I 2022-10-27 03:15:51] ...trap/engines/engine.py.224: train: epoch 44 | batch 2250/2304 [I 2022-10-27 03:15:51] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:30 [I 2022-10-27 03:15:51] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00248 [I 2022-10-27 03:15:51] ...trap/engines/engine.py.230: loss: 0.14104 [I 2022-10-27 03:15:51] ...optimizers/trijoint.py.151: total_norm: 18.207920 [I 2022-10-27 03:15:57] ...trap/engines/engine.py.224: train: epoch 44 | batch 2260/2304 [I 2022-10-27 03:15:57] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:25 [I 2022-10-27 03:15:57] ...trap/engines/engine.py.229: process: 0.66168 | load: 0.00240 [I 2022-10-27 03:15:57] ...trap/engines/engine.py.230: loss: 0.19435 [I 2022-10-27 03:15:57] ...optimizers/trijoint.py.151: total_norm: 22.238768 [I 2022-10-27 03:16:02] ...trap/engines/engine.py.224: train: epoch 44 | batch 2270/2304 [I 2022-10-27 03:16:02] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:19 [I 2022-10-27 03:16:02] ...trap/engines/engine.py.229: process: 0.54839 | load: 0.00250 [I 2022-10-27 03:16:02] ...trap/engines/engine.py.230: loss: 0.15391 [I 2022-10-27 03:16:02] ...optimizers/trijoint.py.151: total_norm: 14.262175 [I 2022-10-27 03:16:08] ...trap/engines/engine.py.224: train: epoch 44 | batch 2280/2304 [I 2022-10-27 03:16:08] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:13 [I 2022-10-27 03:16:08] ...trap/engines/engine.py.229: process: 0.55679 | load: 0.00249 [I 2022-10-27 03:16:08] ...trap/engines/engine.py.230: loss: 0.21757 [I 2022-10-27 03:16:08] ...optimizers/trijoint.py.151: total_norm: 46.833977 [I 2022-10-27 03:16:13] ...trap/engines/engine.py.224: train: epoch 44 | batch 2290/2304 [I 2022-10-27 03:16:13] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:07 [I 2022-10-27 03:16:13] ...trap/engines/engine.py.229: process: 0.56292 | load: 0.00241 [I 2022-10-27 03:16:13] ...trap/engines/engine.py.230: loss: 0.18991 [I 2022-10-27 03:16:13] ...optimizers/trijoint.py.151: total_norm: 20.886934 [I 2022-10-27 03:16:19] ...trap/engines/engine.py.224: train: epoch 44 | batch 2300/2304 [I 2022-10-27 03:16:19] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:02 [I 2022-10-27 03:16:19] ...trap/engines/engine.py.229: process: 0.54673 | load: 0.00016 [I 2022-10-27 03:16:19] ...trap/engines/engine.py.230: loss: 0.20747 [I 2022-10-27 03:16:19] ...optimizers/trijoint.py.151: total_norm: 19.605001 [I 2022-10-27 03:16:21] ...trap/engines/engine.py.224: train: epoch 44 | batch 2304/2304 [I 2022-10-27 03:16:21] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:00 [I 2022-10-27 03:16:21] ...trap/engines/engine.py.229: process: 0.54432 | load: 0.00016 [I 2022-10-27 03:16:21] ...trap/engines/engine.py.230: loss: 0.12752 [I 2022-10-27 03:16:21] ...optimizers/trijoint.py.151: total_norm: 11.021195 [S 2022-10-27 03:16:22] ...trap/engines/engine.py.129: train_epoch.epoch: 44 [S 2022-10-27 03:16:22] ...trap/engines/engine.py.129: train_epoch.loss: 0.1676469684067362 [I 2022-10-27 03:16:22] ...s/criterions/triplet.py.84: margin = 0.2750000000000002 [I 2022-10-27 03:16:23] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 44 [I 2022-10-27 03:16:23] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 03:16:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 03:16:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 03:16:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 03:16:26] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 03:16:27] ...trap/engines/engine.py.321: eval: epoch 44 | batch 0/510 [I 2022-10-27 03:16:27] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:31:08 [I 2022-10-27 03:16:27] ...trap/engines/engine.py.326: process: 3.66280 | load: 3.28470 [I 2022-10-27 03:16:30] ...trap/engines/engine.py.321: eval: epoch 44 | batch 10/510 [I 2022-10-27 03:16:30] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:05:44 [I 2022-10-27 03:16:30] ...trap/engines/engine.py.326: process: 0.32961 | load: 0.00019 [I 2022-10-27 03:16:33] ...trap/engines/engine.py.321: eval: epoch 44 | batch 20/510 [I 2022-10-27 03:16:33] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:00 [I 2022-10-27 03:16:33] ...trap/engines/engine.py.326: process: 0.32438 | load: 0.00017 [I 2022-10-27 03:16:37] ...trap/engines/engine.py.321: eval: epoch 44 | batch 30/510 [I 2022-10-27 03:16:37] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:46 [I 2022-10-27 03:16:37] ...trap/engines/engine.py.326: process: 0.33934 | load: 0.00016 [I 2022-10-27 03:16:40] ...trap/engines/engine.py.321: eval: epoch 44 | batch 40/510 [I 2022-10-27 03:16:40] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:36 [I 2022-10-27 03:16:40] ...trap/engines/engine.py.326: process: 0.33274 | load: 0.00016 [I 2022-10-27 03:16:43] ...trap/engines/engine.py.321: eval: epoch 44 | batch 50/510 [I 2022-10-27 03:16:43] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:31 [I 2022-10-27 03:16:43] ...trap/engines/engine.py.326: process: 0.32686 | load: 0.00016 [I 2022-10-27 03:16:47] ...trap/engines/engine.py.321: eval: epoch 44 | batch 60/510 [I 2022-10-27 03:16:47] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:28 [I 2022-10-27 03:16:47] ...trap/engines/engine.py.326: process: 0.33066 | load: 0.00016 [I 2022-10-27 03:16:50] ...trap/engines/engine.py.321: eval: epoch 44 | batch 70/510 [I 2022-10-27 03:16:50] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:25 [I 2022-10-27 03:16:50] ...trap/engines/engine.py.326: process: 0.32785 | load: 0.00016 [I 2022-10-27 03:16:53] ...trap/engines/engine.py.321: eval: epoch 44 | batch 80/510 [I 2022-10-27 03:16:53] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:22 [I 2022-10-27 03:16:53] ...trap/engines/engine.py.326: process: 0.33224 | load: 0.00015 [I 2022-10-27 03:16:57] ...trap/engines/engine.py.321: eval: epoch 44 | batch 90/510 [I 2022-10-27 03:16:57] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:19 [I 2022-10-27 03:16:57] ...trap/engines/engine.py.326: process: 0.32876 | load: 0.00016 [I 2022-10-27 03:17:00] ...trap/engines/engine.py.321: eval: epoch 44 | batch 100/510 [I 2022-10-27 03:17:00] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:13 [I 2022-10-27 03:17:00] ...trap/engines/engine.py.326: process: 0.28922 | load: 0.00018 [I 2022-10-27 03:17:03] ...trap/engines/engine.py.321: eval: epoch 44 | batch 110/510 [I 2022-10-27 03:17:03] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:57 [I 2022-10-27 03:17:03] ...trap/engines/engine.py.326: process: 0.29264 | load: 0.00014 [I 2022-10-27 03:17:06] ...trap/engines/engine.py.321: eval: epoch 44 | batch 120/510 [I 2022-10-27 03:17:06] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-27 03:17:06] ...trap/engines/engine.py.326: process: 0.28091 | load: 0.00017 [I 2022-10-27 03:17:09] ...trap/engines/engine.py.321: eval: epoch 44 | batch 130/510 [I 2022-10-27 03:17:09] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:50 [I 2022-10-27 03:17:09] ...trap/engines/engine.py.326: process: 0.29062 | load: 0.00025 [I 2022-10-27 03:17:12] ...trap/engines/engine.py.321: eval: epoch 44 | batch 140/510 [I 2022-10-27 03:17:12] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-27 03:17:12] ...trap/engines/engine.py.326: process: 0.29787 | load: 0.00014 [I 2022-10-27 03:17:15] ...trap/engines/engine.py.321: eval: epoch 44 | batch 150/510 [I 2022-10-27 03:17:15] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:45 [I 2022-10-27 03:17:15] ...trap/engines/engine.py.326: process: 0.28392 | load: 0.00019 [I 2022-10-27 03:17:18] ...trap/engines/engine.py.321: eval: epoch 44 | batch 160/510 [I 2022-10-27 03:17:18] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-27 03:17:18] ...trap/engines/engine.py.326: process: 0.28666 | load: 0.00016 [I 2022-10-27 03:17:20] ...trap/engines/engine.py.321: eval: epoch 44 | batch 170/510 [I 2022-10-27 03:17:20] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-27 03:17:20] ...trap/engines/engine.py.326: process: 0.28735 | load: 0.00018 [I 2022-10-27 03:17:23] ...trap/engines/engine.py.321: eval: epoch 44 | batch 180/510 [I 2022-10-27 03:17:23] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 03:17:23] ...trap/engines/engine.py.326: process: 0.28868 | load: 0.00018 [I 2022-10-27 03:17:26] ...trap/engines/engine.py.321: eval: epoch 44 | batch 190/510 [I 2022-10-27 03:17:26] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 03:17:26] ...trap/engines/engine.py.326: process: 0.28994 | load: 0.00020 [I 2022-10-27 03:17:29] ...trap/engines/engine.py.321: eval: epoch 44 | batch 200/510 [I 2022-10-27 03:17:29] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 03:17:29] ...trap/engines/engine.py.326: process: 0.28292 | load: 0.00017 [I 2022-10-27 03:17:32] ...trap/engines/engine.py.321: eval: epoch 44 | batch 210/510 [I 2022-10-27 03:17:32] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:27 [I 2022-10-27 03:17:32] ...trap/engines/engine.py.326: process: 0.28948 | load: 0.00023 [I 2022-10-27 03:17:35] ...trap/engines/engine.py.321: eval: epoch 44 | batch 220/510 [I 2022-10-27 03:17:35] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-27 03:17:35] ...trap/engines/engine.py.326: process: 0.28847 | load: 0.00021 [I 2022-10-27 03:17:38] ...trap/engines/engine.py.321: eval: epoch 44 | batch 230/510 [I 2022-10-27 03:17:38] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-27 03:17:38] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00027 [I 2022-10-27 03:17:41] ...trap/engines/engine.py.321: eval: epoch 44 | batch 240/510 [I 2022-10-27 03:17:41] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-27 03:17:41] ...trap/engines/engine.py.326: process: 0.28449 | load: 0.00020 [I 2022-10-27 03:17:44] ...trap/engines/engine.py.321: eval: epoch 44 | batch 250/510 [I 2022-10-27 03:17:44] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-27 03:17:44] ...trap/engines/engine.py.326: process: 0.29177 | load: 0.00032 [I 2022-10-27 03:17:46] ...trap/engines/engine.py.321: eval: epoch 44 | batch 260/510 [I 2022-10-27 03:17:46] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 03:17:46] ...trap/engines/engine.py.326: process: 0.27932 | load: 0.00021 [I 2022-10-27 03:17:49] ...trap/engines/engine.py.321: eval: epoch 44 | batch 270/510 [I 2022-10-27 03:17:49] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:10 [I 2022-10-27 03:17:49] ...trap/engines/engine.py.326: process: 0.28825 | load: 0.00019 [I 2022-10-27 03:17:52] ...trap/engines/engine.py.321: eval: epoch 44 | batch 280/510 [I 2022-10-27 03:17:52] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 03:17:52] ...trap/engines/engine.py.326: process: 0.28870 | load: 0.00024 [I 2022-10-27 03:17:55] ...trap/engines/engine.py.321: eval: epoch 44 | batch 290/510 [I 2022-10-27 03:17:55] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 03:17:55] ...trap/engines/engine.py.326: process: 0.28609 | load: 0.00021 [I 2022-10-27 03:17:58] ...trap/engines/engine.py.321: eval: epoch 44 | batch 300/510 [I 2022-10-27 03:17:58] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-27 03:17:58] ...trap/engines/engine.py.326: process: 0.28556 | load: 0.00018 [I 2022-10-27 03:18:01] ...trap/engines/engine.py.321: eval: epoch 44 | batch 310/510 [I 2022-10-27 03:18:01] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 03:18:01] ...trap/engines/engine.py.326: process: 0.28625 | load: 0.00015 [I 2022-10-27 03:18:04] ...trap/engines/engine.py.321: eval: epoch 44 | batch 320/510 [I 2022-10-27 03:18:04] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 03:18:04] ...trap/engines/engine.py.326: process: 0.28662 | load: 0.00017 [I 2022-10-27 03:18:07] ...trap/engines/engine.py.321: eval: epoch 44 | batch 330/510 [I 2022-10-27 03:18:07] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 03:18:07] ...trap/engines/engine.py.326: process: 0.28835 | load: 0.00017 [I 2022-10-27 03:18:10] ...trap/engines/engine.py.321: eval: epoch 44 | batch 340/510 [I 2022-10-27 03:18:10] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-27 03:18:10] ...trap/engines/engine.py.326: process: 0.28939 | load: 0.00022 [I 2022-10-27 03:18:12] ...trap/engines/engine.py.321: eval: epoch 44 | batch 350/510 [I 2022-10-27 03:18:12] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-27 03:18:12] ...trap/engines/engine.py.326: process: 0.28770 | load: 0.00017 [I 2022-10-27 03:18:15] ...trap/engines/engine.py.321: eval: epoch 44 | batch 360/510 [I 2022-10-27 03:18:15] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 03:18:15] ...trap/engines/engine.py.326: process: 0.28750 | load: 0.00023 [I 2022-10-27 03:18:18] ...trap/engines/engine.py.321: eval: epoch 44 | batch 370/510 [I 2022-10-27 03:18:18] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 03:18:18] ...trap/engines/engine.py.326: process: 0.29035 | load: 0.00015 [I 2022-10-27 03:18:21] ...trap/engines/engine.py.321: eval: epoch 44 | batch 380/510 [I 2022-10-27 03:18:21] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:39 [I 2022-10-27 03:18:21] ...trap/engines/engine.py.326: process: 0.29895 | load: 0.00382 [I 2022-10-27 03:18:24] ...trap/engines/engine.py.321: eval: epoch 44 | batch 390/510 [I 2022-10-27 03:18:24] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-27 03:18:24] ...trap/engines/engine.py.326: process: 0.28921 | load: 0.00023 [I 2022-10-27 03:18:27] ...trap/engines/engine.py.321: eval: epoch 44 | batch 400/510 [I 2022-10-27 03:18:27] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-27 03:18:27] ...trap/engines/engine.py.326: process: 0.28520 | load: 0.00018 [I 2022-10-27 03:18:30] ...trap/engines/engine.py.321: eval: epoch 44 | batch 410/510 [I 2022-10-27 03:18:30] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-27 03:18:30] ...trap/engines/engine.py.326: process: 0.29325 | load: 0.00015 [I 2022-10-27 03:18:33] ...trap/engines/engine.py.321: eval: epoch 44 | batch 420/510 [I 2022-10-27 03:18:33] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-27 03:18:33] ...trap/engines/engine.py.326: process: 0.28569 | load: 0.00017 [I 2022-10-27 03:18:36] ...trap/engines/engine.py.321: eval: epoch 44 | batch 430/510 [I 2022-10-27 03:18:36] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-27 03:18:36] ...trap/engines/engine.py.326: process: 0.29317 | load: 0.00017 [I 2022-10-27 03:18:38] ...trap/engines/engine.py.321: eval: epoch 44 | batch 440/510 [I 2022-10-27 03:18:38] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 03:18:38] ...trap/engines/engine.py.326: process: 0.28511 | load: 0.00017 [I 2022-10-27 03:18:41] ...trap/engines/engine.py.321: eval: epoch 44 | batch 450/510 [I 2022-10-27 03:18:41] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 03:18:41] ...trap/engines/engine.py.326: process: 0.29612 | load: 0.00018 [I 2022-10-27 03:18:44] ...trap/engines/engine.py.321: eval: epoch 44 | batch 460/510 [I 2022-10-27 03:18:44] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 03:18:44] ...trap/engines/engine.py.326: process: 0.28968 | load: 0.00019 [I 2022-10-27 03:18:47] ...trap/engines/engine.py.321: eval: epoch 44 | batch 470/510 [I 2022-10-27 03:18:47] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 03:18:47] ...trap/engines/engine.py.326: process: 0.28530 | load: 0.00019 [I 2022-10-27 03:18:50] ...trap/engines/engine.py.321: eval: epoch 44 | batch 480/510 [I 2022-10-27 03:18:50] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 03:18:50] ...trap/engines/engine.py.326: process: 0.28813 | load: 0.00016 [I 2022-10-27 03:18:53] ...trap/engines/engine.py.321: eval: epoch 44 | batch 490/510 [I 2022-10-27 03:18:53] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 03:18:53] ...trap/engines/engine.py.326: process: 0.27594 | load: 0.00020 [I 2022-10-27 03:18:56] ...trap/engines/engine.py.321: eval: epoch 44 | batch 500/510 [I 2022-10-27 03:18:56] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 03:18:56] ...trap/engines/engine.py.326: process: 0.28984 | load: 0.00019 [I 2022-10-27 03:18:59] ...trap/engines/engine.py.321: eval: epoch 44 | batch 510/510 [I 2022-10-27 03:18:59] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 03:18:59] ...trap/engines/engine.py.326: process: 0.28029 | load: 0.00015 [S 2022-10-27 03:18:59] ...trap/engines/engine.py.132: eval_epoch.epoch: 44 [I 2022-10-27 03:18:59] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.687200 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.890200 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.931500 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.677200 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.887700 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.928100 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009704 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008400 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007953 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013710 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009900 [S 2022-10-27 03:19:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006204 [I 2022-10-27 03:19:05] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 03:19:05] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 03:19:05] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 03:19:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 03:19:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 03:19:07] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 03:19:09] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 03:19:15] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 03:19:15] ...trap/engines/engine.py.163: Training model on trainset for epoch 45 [I 2022-10-27 03:19:15] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 03:19:17] ...trap/engines/engine.py.224: train: epoch 45 | batch 0/2304 [I 2022-10-27 03:19:17] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:58 [I 2022-10-27 03:19:17] ...trap/engines/engine.py.229: process: 2.68161 | load: 2.03386 [I 2022-10-27 03:19:17] ...trap/engines/engine.py.230: loss: 0.15755 [I 2022-10-27 03:19:17] ...optimizers/trijoint.py.151: total_norm: 16.435175 [I 2022-10-27 03:19:23] ...trap/engines/engine.py.224: train: epoch 45 | batch 10/2304 [I 2022-10-27 03:19:23] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:59 [I 2022-10-27 03:19:23] ...trap/engines/engine.py.229: process: 0.56497 | load: 0.00484 [I 2022-10-27 03:19:23] ...trap/engines/engine.py.230: loss: 0.14291 [I 2022-10-27 03:19:23] ...optimizers/trijoint.py.151: total_norm: 11.685925 [I 2022-10-27 03:19:28] ...trap/engines/engine.py.224: train: epoch 45 | batch 20/2304 [I 2022-10-27 03:19:28] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:09 [I 2022-10-27 03:19:28] ...trap/engines/engine.py.229: process: 0.54407 | load: 0.00359 [I 2022-10-27 03:19:28] ...trap/engines/engine.py.230: loss: 0.15433 [I 2022-10-27 03:19:28] ...optimizers/trijoint.py.151: total_norm: 18.120564 [I 2022-10-27 03:19:34] ...trap/engines/engine.py.224: train: epoch 45 | batch 30/2304 [I 2022-10-27 03:19:34] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:14 [I 2022-10-27 03:19:34] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00306 [I 2022-10-27 03:19:34] ...trap/engines/engine.py.230: loss: 0.17792 [I 2022-10-27 03:19:34] ...optimizers/trijoint.py.151: total_norm: 24.358494 [I 2022-10-27 03:19:40] ...trap/engines/engine.py.224: train: epoch 45 | batch 40/2304 [I 2022-10-27 03:19:40] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:14 [I 2022-10-27 03:19:40] ...trap/engines/engine.py.229: process: 0.55611 | load: 0.00305 [I 2022-10-27 03:19:40] ...trap/engines/engine.py.230: loss: 0.12201 [I 2022-10-27 03:19:40] ...optimizers/trijoint.py.151: total_norm: 12.603523 [I 2022-10-27 03:19:45] ...trap/engines/engine.py.224: train: epoch 45 | batch 50/2304 [I 2022-10-27 03:19:45] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:49 [I 2022-10-27 03:19:45] ...trap/engines/engine.py.229: process: 0.54962 | load: 0.00289 [I 2022-10-27 03:19:45] ...trap/engines/engine.py.230: loss: 0.18272 [I 2022-10-27 03:19:45] ...optimizers/trijoint.py.151: total_norm: 8.964714 [I 2022-10-27 03:19:51] ...trap/engines/engine.py.224: train: epoch 45 | batch 60/2304 [I 2022-10-27 03:19:51] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:51 [I 2022-10-27 03:19:51] ...trap/engines/engine.py.229: process: 0.56127 | load: 0.00285 [I 2022-10-27 03:19:51] ...trap/engines/engine.py.230: loss: 0.14259 [I 2022-10-27 03:19:51] ...optimizers/trijoint.py.151: total_norm: 20.517046 [I 2022-10-27 03:19:56] ...trap/engines/engine.py.224: train: epoch 45 | batch 70/2304 [I 2022-10-27 03:19:56] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:53 [I 2022-10-27 03:19:56] ...trap/engines/engine.py.229: process: 0.56927 | load: 0.00262 [I 2022-10-27 03:19:56] ...trap/engines/engine.py.230: loss: 0.11111 [I 2022-10-27 03:19:56] ...optimizers/trijoint.py.151: total_norm: 15.029550 [I 2022-10-27 03:20:02] ...trap/engines/engine.py.224: train: epoch 45 | batch 80/2304 [I 2022-10-27 03:20:02] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:41 [I 2022-10-27 03:20:02] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00269 [I 2022-10-27 03:20:02] ...trap/engines/engine.py.230: loss: 0.23210 [I 2022-10-27 03:20:02] ...optimizers/trijoint.py.151: total_norm: 40.335056 [I 2022-10-27 03:20:08] ...trap/engines/engine.py.224: train: epoch 45 | batch 90/2304 [I 2022-10-27 03:20:08] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:24 [I 2022-10-27 03:20:08] ...trap/engines/engine.py.229: process: 0.53936 | load: 0.00285 [I 2022-10-27 03:20:08] ...trap/engines/engine.py.230: loss: 0.13647 [I 2022-10-27 03:20:08] ...optimizers/trijoint.py.151: total_norm: 22.405975 [I 2022-10-27 03:20:13] ...trap/engines/engine.py.224: train: epoch 45 | batch 100/2304 [I 2022-10-27 03:20:13] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:51 [I 2022-10-27 03:20:13] ...trap/engines/engine.py.229: process: 0.56684 | load: 0.00280 [I 2022-10-27 03:20:13] ...trap/engines/engine.py.230: loss: 0.13815 [I 2022-10-27 03:20:13] ...optimizers/trijoint.py.151: total_norm: 15.786194 [I 2022-10-27 03:20:19] ...trap/engines/engine.py.224: train: epoch 45 | batch 110/2304 [I 2022-10-27 03:20:19] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:12 [I 2022-10-27 03:20:19] ...trap/engines/engine.py.229: process: 0.54082 | load: 0.00254 [I 2022-10-27 03:20:19] ...trap/engines/engine.py.230: loss: 0.12569 [I 2022-10-27 03:20:19] ...optimizers/trijoint.py.151: total_norm: 10.517109 [I 2022-10-27 03:20:24] ...trap/engines/engine.py.224: train: epoch 45 | batch 120/2304 [I 2022-10-27 03:20:24] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:03 [I 2022-10-27 03:20:24] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00259 [I 2022-10-27 03:20:24] ...trap/engines/engine.py.230: loss: 0.23648 [I 2022-10-27 03:20:24] ...optimizers/trijoint.py.151: total_norm: 39.393650 [I 2022-10-27 03:20:30] ...trap/engines/engine.py.224: train: epoch 45 | batch 130/2304 [I 2022-10-27 03:20:30] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:02 [I 2022-10-27 03:20:30] ...trap/engines/engine.py.229: process: 0.54828 | load: 0.00240 [I 2022-10-27 03:20:30] ...trap/engines/engine.py.230: loss: 0.14710 [I 2022-10-27 03:20:30] ...optimizers/trijoint.py.151: total_norm: 26.460781 [I 2022-10-27 03:20:36] ...trap/engines/engine.py.224: train: epoch 45 | batch 140/2304 [I 2022-10-27 03:20:36] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:11 [I 2022-10-27 03:20:36] ...trap/engines/engine.py.229: process: 0.57111 | load: 0.00260 [I 2022-10-27 03:20:36] ...trap/engines/engine.py.230: loss: 0.17935 [I 2022-10-27 03:20:36] ...optimizers/trijoint.py.151: total_norm: 46.672104 [I 2022-10-27 03:20:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 150/2304 [I 2022-10-27 03:20:41] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:51 [I 2022-10-27 03:20:41] ...trap/engines/engine.py.229: process: 0.54814 | load: 0.00254 [I 2022-10-27 03:20:41] ...trap/engines/engine.py.230: loss: 0.16077 [I 2022-10-27 03:20:41] ...optimizers/trijoint.py.151: total_norm: 10.851941 [I 2022-10-27 03:20:47] ...trap/engines/engine.py.224: train: epoch 45 | batch 160/2304 [I 2022-10-27 03:20:47] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:59 [I 2022-10-27 03:20:47] ...trap/engines/engine.py.229: process: 0.54384 | load: 0.00288 [I 2022-10-27 03:20:47] ...trap/engines/engine.py.230: loss: 0.23425 [I 2022-10-27 03:20:47] ...optimizers/trijoint.py.151: total_norm: 21.704802 [I 2022-10-27 03:20:52] ...trap/engines/engine.py.224: train: epoch 45 | batch 170/2304 [I 2022-10-27 03:20:52] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:44 [I 2022-10-27 03:20:52] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00254 [I 2022-10-27 03:20:52] ...trap/engines/engine.py.230: loss: 0.23008 [I 2022-10-27 03:20:52] ...optimizers/trijoint.py.151: total_norm: 17.505629 [I 2022-10-27 03:20:58] ...trap/engines/engine.py.224: train: epoch 45 | batch 180/2304 [I 2022-10-27 03:20:58] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:30 [I 2022-10-27 03:20:58] ...trap/engines/engine.py.229: process: 0.54677 | load: 0.00240 [I 2022-10-27 03:20:58] ...trap/engines/engine.py.230: loss: 0.22416 [I 2022-10-27 03:20:58] ...optimizers/trijoint.py.151: total_norm: 19.285177 [I 2022-10-27 03:21:03] ...trap/engines/engine.py.224: train: epoch 45 | batch 190/2304 [I 2022-10-27 03:21:03] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:39 [I 2022-10-27 03:21:03] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00262 [I 2022-10-27 03:21:03] ...trap/engines/engine.py.230: loss: 0.12507 [I 2022-10-27 03:21:03] ...optimizers/trijoint.py.151: total_norm: 8.910580 [I 2022-10-27 03:21:09] ...trap/engines/engine.py.224: train: epoch 45 | batch 200/2304 [I 2022-10-27 03:21:09] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:20 [I 2022-10-27 03:21:09] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00251 [I 2022-10-27 03:21:09] ...trap/engines/engine.py.230: loss: 0.15978 [I 2022-10-27 03:21:09] ...optimizers/trijoint.py.151: total_norm: 19.777678 [I 2022-10-27 03:21:14] ...trap/engines/engine.py.224: train: epoch 45 | batch 210/2304 [I 2022-10-27 03:21:14] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:21 [I 2022-10-27 03:21:14] ...trap/engines/engine.py.229: process: 0.56369 | load: 0.00252 [I 2022-10-27 03:21:14] ...trap/engines/engine.py.230: loss: 0.23064 [I 2022-10-27 03:21:14] ...optimizers/trijoint.py.151: total_norm: 13.774998 [I 2022-10-27 03:21:20] ...trap/engines/engine.py.224: train: epoch 45 | batch 220/2304 [I 2022-10-27 03:21:20] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:12 [I 2022-10-27 03:21:20] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00241 [I 2022-10-27 03:21:20] ...trap/engines/engine.py.230: loss: 0.29385 [I 2022-10-27 03:21:20] ...optimizers/trijoint.py.151: total_norm: 15.073911 [I 2022-10-27 03:21:25] ...trap/engines/engine.py.224: train: epoch 45 | batch 230/2304 [I 2022-10-27 03:21:25] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:12 [I 2022-10-27 03:21:25] ...trap/engines/engine.py.229: process: 0.58186 | load: 0.00243 [I 2022-10-27 03:21:25] ...trap/engines/engine.py.230: loss: 0.16002 [I 2022-10-27 03:21:25] ...optimizers/trijoint.py.151: total_norm: 8.468118 [I 2022-10-27 03:21:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 240/2304 [I 2022-10-27 03:21:31] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:18:58 [I 2022-10-27 03:21:31] ...trap/engines/engine.py.229: process: 0.54946 | load: 0.00251 [I 2022-10-27 03:21:31] ...trap/engines/engine.py.230: loss: 0.22520 [I 2022-10-27 03:21:31] ...optimizers/trijoint.py.151: total_norm: 17.458433 [I 2022-10-27 03:21:37] ...trap/engines/engine.py.224: train: epoch 45 | batch 250/2304 [I 2022-10-27 03:21:37] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:19:02 [I 2022-10-27 03:21:37] ...trap/engines/engine.py.229: process: 0.55350 | load: 0.00238 [I 2022-10-27 03:21:37] ...trap/engines/engine.py.230: loss: 0.12429 [I 2022-10-27 03:21:37] ...optimizers/trijoint.py.151: total_norm: 10.540018 [I 2022-10-27 03:21:42] ...trap/engines/engine.py.224: train: epoch 45 | batch 260/2304 [I 2022-10-27 03:21:42] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:56 [I 2022-10-27 03:21:42] ...trap/engines/engine.py.229: process: 0.56549 | load: 0.00241 [I 2022-10-27 03:21:42] ...trap/engines/engine.py.230: loss: 0.15882 [I 2022-10-27 03:21:42] ...optimizers/trijoint.py.151: total_norm: 15.283778 [I 2022-10-27 03:21:48] ...trap/engines/engine.py.224: train: epoch 45 | batch 270/2304 [I 2022-10-27 03:21:48] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:36 [I 2022-10-27 03:21:48] ...trap/engines/engine.py.229: process: 0.54368 | load: 0.00258 [I 2022-10-27 03:21:48] ...trap/engines/engine.py.230: loss: 0.11945 [I 2022-10-27 03:21:48] ...optimizers/trijoint.py.151: total_norm: 14.292012 [I 2022-10-27 03:21:53] ...trap/engines/engine.py.224: train: epoch 45 | batch 280/2304 [I 2022-10-27 03:21:53] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:38 [I 2022-10-27 03:21:53] ...trap/engines/engine.py.229: process: 0.55113 | load: 0.00245 [I 2022-10-27 03:21:53] ...trap/engines/engine.py.230: loss: 0.23443 [I 2022-10-27 03:21:53] ...optimizers/trijoint.py.151: total_norm: 25.254215 [I 2022-10-27 03:21:59] ...trap/engines/engine.py.224: train: epoch 45 | batch 290/2304 [I 2022-10-27 03:21:59] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:35 [I 2022-10-27 03:21:59] ...trap/engines/engine.py.229: process: 0.55171 | load: 0.00243 [I 2022-10-27 03:21:59] ...trap/engines/engine.py.230: loss: 0.17944 [I 2022-10-27 03:21:59] ...optimizers/trijoint.py.151: total_norm: 14.351814 [I 2022-10-27 03:22:04] ...trap/engines/engine.py.224: train: epoch 45 | batch 300/2304 [I 2022-10-27 03:22:04] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:18:29 [I 2022-10-27 03:22:04] ...trap/engines/engine.py.229: process: 0.55186 | load: 0.00244 [I 2022-10-27 03:22:04] ...trap/engines/engine.py.230: loss: 0.17489 [I 2022-10-27 03:22:04] ...optimizers/trijoint.py.151: total_norm: 23.540527 [I 2022-10-27 03:22:10] ...trap/engines/engine.py.224: train: epoch 45 | batch 310/2304 [I 2022-10-27 03:22:10] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:25 [I 2022-10-27 03:22:10] ...trap/engines/engine.py.229: process: 0.55895 | load: 0.00242 [I 2022-10-27 03:22:10] ...trap/engines/engine.py.230: loss: 0.13644 [I 2022-10-27 03:22:10] ...optimizers/trijoint.py.151: total_norm: 20.927576 [I 2022-10-27 03:22:15] ...trap/engines/engine.py.224: train: epoch 45 | batch 320/2304 [I 2022-10-27 03:22:15] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:18:17 [I 2022-10-27 03:22:15] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00243 [I 2022-10-27 03:22:15] ...trap/engines/engine.py.230: loss: 0.18540 [I 2022-10-27 03:22:15] ...optimizers/trijoint.py.151: total_norm: 17.648031 [I 2022-10-27 03:22:21] ...trap/engines/engine.py.224: train: epoch 45 | batch 330/2304 [I 2022-10-27 03:22:21] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:14 [I 2022-10-27 03:22:21] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00252 [I 2022-10-27 03:22:21] ...trap/engines/engine.py.230: loss: 0.12623 [I 2022-10-27 03:22:21] ...optimizers/trijoint.py.151: total_norm: 16.141048 [I 2022-10-27 03:22:27] ...trap/engines/engine.py.224: train: epoch 45 | batch 340/2304 [I 2022-10-27 03:22:27] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:18:08 [I 2022-10-27 03:22:27] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00247 [I 2022-10-27 03:22:27] ...trap/engines/engine.py.230: loss: 0.20369 [I 2022-10-27 03:22:27] ...optimizers/trijoint.py.151: total_norm: 24.762096 [I 2022-10-27 03:22:32] ...trap/engines/engine.py.224: train: epoch 45 | batch 350/2304 [I 2022-10-27 03:22:32] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:18:03 [I 2022-10-27 03:22:32] ...trap/engines/engine.py.229: process: 0.55829 | load: 0.00239 [I 2022-10-27 03:22:32] ...trap/engines/engine.py.230: loss: 0.27501 [I 2022-10-27 03:22:32] ...optimizers/trijoint.py.151: total_norm: 19.302744 [I 2022-10-27 03:22:38] ...trap/engines/engine.py.224: train: epoch 45 | batch 360/2304 [I 2022-10-27 03:22:38] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:04 [I 2022-10-27 03:22:38] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00245 [I 2022-10-27 03:22:38] ...trap/engines/engine.py.230: loss: 0.11533 [I 2022-10-27 03:22:38] ...optimizers/trijoint.py.151: total_norm: 9.073875 [I 2022-10-27 03:22:43] ...trap/engines/engine.py.224: train: epoch 45 | batch 370/2304 [I 2022-10-27 03:22:43] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:17:58 [I 2022-10-27 03:22:43] ...trap/engines/engine.py.229: process: 0.56073 | load: 0.00249 [I 2022-10-27 03:22:43] ...trap/engines/engine.py.230: loss: 0.17854 [I 2022-10-27 03:22:43] ...optimizers/trijoint.py.151: total_norm: 21.334978 [I 2022-10-27 03:22:49] ...trap/engines/engine.py.224: train: epoch 45 | batch 380/2304 [I 2022-10-27 03:22:49] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:52 [I 2022-10-27 03:22:49] ...trap/engines/engine.py.229: process: 0.55799 | load: 0.00253 [I 2022-10-27 03:22:49] ...trap/engines/engine.py.230: loss: 0.14172 [I 2022-10-27 03:22:49] ...optimizers/trijoint.py.151: total_norm: 11.700254 [I 2022-10-27 03:22:54] ...trap/engines/engine.py.224: train: epoch 45 | batch 390/2304 [I 2022-10-27 03:22:54] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:17:44 [I 2022-10-27 03:22:54] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00247 [I 2022-10-27 03:22:54] ...trap/engines/engine.py.230: loss: 0.16222 [I 2022-10-27 03:22:54] ...optimizers/trijoint.py.151: total_norm: 14.589802 [I 2022-10-27 03:23:00] ...trap/engines/engine.py.224: train: epoch 45 | batch 400/2304 [I 2022-10-27 03:23:00] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:28 [I 2022-10-27 03:23:00] ...trap/engines/engine.py.229: process: 0.55351 | load: 0.00244 [I 2022-10-27 03:23:00] ...trap/engines/engine.py.230: loss: 0.11880 [I 2022-10-27 03:23:00] ...optimizers/trijoint.py.151: total_norm: 25.068441 [I 2022-10-27 03:23:25] ...trap/engines/engine.py.224: train: epoch 45 | batch 410/2304 [I 2022-10-27 03:23:25] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 1:54:42 [I 2022-10-27 03:23:25] ...trap/engines/engine.py.229: process: 0.55130 | load: 0.00239 [I 2022-10-27 03:23:25] ...trap/engines/engine.py.230: loss: 0.14127 [I 2022-10-27 03:23:25] ...optimizers/trijoint.py.151: total_norm: 14.683704 [I 2022-10-27 03:23:30] ...trap/engines/engine.py.224: train: epoch 45 | batch 420/2304 [I 2022-10-27 03:23:30] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:27:52 [I 2022-10-27 03:23:30] ...trap/engines/engine.py.229: process: 0.55926 | load: 0.00254 [I 2022-10-27 03:23:30] ...trap/engines/engine.py.230: loss: 0.17480 [I 2022-10-27 03:23:30] ...optimizers/trijoint.py.151: total_norm: 18.308306 [I 2022-10-27 03:23:36] ...trap/engines/engine.py.224: train: epoch 45 | batch 430/2304 [I 2022-10-27 03:23:36] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:18:30 [I 2022-10-27 03:23:36] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00240 [I 2022-10-27 03:23:36] ...trap/engines/engine.py.230: loss: 0.34535 [I 2022-10-27 03:23:36] ...optimizers/trijoint.py.151: total_norm: 49.410980 [I 2022-10-27 03:23:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 440/2304 [I 2022-10-27 03:23:41] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:25 [I 2022-10-27 03:23:41] ...trap/engines/engine.py.229: process: 0.55828 | load: 0.00237 [I 2022-10-27 03:23:41] ...trap/engines/engine.py.230: loss: 0.17266 [I 2022-10-27 03:23:41] ...optimizers/trijoint.py.151: total_norm: 14.010056 [I 2022-10-27 03:23:47] ...trap/engines/engine.py.224: train: epoch 45 | batch 450/2304 [I 2022-10-27 03:23:47] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:17:06 [I 2022-10-27 03:23:47] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00236 [I 2022-10-27 03:23:47] ...trap/engines/engine.py.230: loss: 0.19174 [I 2022-10-27 03:23:47] ...optimizers/trijoint.py.151: total_norm: 22.556044 [I 2022-10-27 03:23:53] ...trap/engines/engine.py.224: train: epoch 45 | batch 460/2304 [I 2022-10-27 03:23:53] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:17:01 [I 2022-10-27 03:23:53] ...trap/engines/engine.py.229: process: 0.54903 | load: 0.00236 [I 2022-10-27 03:23:53] ...trap/engines/engine.py.230: loss: 0.24234 [I 2022-10-27 03:23:53] ...optimizers/trijoint.py.151: total_norm: 23.582640 [I 2022-10-27 03:23:58] ...trap/engines/engine.py.224: train: epoch 45 | batch 470/2304 [I 2022-10-27 03:23:58] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:57 [I 2022-10-27 03:23:58] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00236 [I 2022-10-27 03:23:58] ...trap/engines/engine.py.230: loss: 0.18383 [I 2022-10-27 03:23:58] ...optimizers/trijoint.py.151: total_norm: 26.449953 [I 2022-10-27 03:24:04] ...trap/engines/engine.py.224: train: epoch 45 | batch 480/2304 [I 2022-10-27 03:24:04] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:51 [I 2022-10-27 03:24:04] ...trap/engines/engine.py.229: process: 0.54510 | load: 0.00237 [I 2022-10-27 03:24:04] ...trap/engines/engine.py.230: loss: 0.15435 [I 2022-10-27 03:24:04] ...optimizers/trijoint.py.151: total_norm: 15.051207 [I 2022-10-27 03:24:09] ...trap/engines/engine.py.224: train: epoch 45 | batch 490/2304 [I 2022-10-27 03:24:09] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:41 [I 2022-10-27 03:24:09] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00243 [I 2022-10-27 03:24:09] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-27 03:24:09] ...optimizers/trijoint.py.151: total_norm: 17.130846 [I 2022-10-27 03:24:15] ...trap/engines/engine.py.224: train: epoch 45 | batch 500/2304 [I 2022-10-27 03:24:15] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:35 [I 2022-10-27 03:24:15] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00237 [I 2022-10-27 03:24:15] ...trap/engines/engine.py.230: loss: 0.18962 [I 2022-10-27 03:24:15] ...optimizers/trijoint.py.151: total_norm: 26.727608 [I 2022-10-27 03:24:20] ...trap/engines/engine.py.224: train: epoch 45 | batch 510/2304 [I 2022-10-27 03:24:20] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:34 [I 2022-10-27 03:24:20] ...trap/engines/engine.py.229: process: 0.55194 | load: 0.00244 [I 2022-10-27 03:24:20] ...trap/engines/engine.py.230: loss: 0.14881 [I 2022-10-27 03:24:20] ...optimizers/trijoint.py.151: total_norm: 17.125713 [I 2022-10-27 03:24:26] ...trap/engines/engine.py.224: train: epoch 45 | batch 520/2304 [I 2022-10-27 03:24:26] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:31 [I 2022-10-27 03:24:26] ...trap/engines/engine.py.229: process: 0.54627 | load: 0.00234 [I 2022-10-27 03:24:26] ...trap/engines/engine.py.230: loss: 0.16404 [I 2022-10-27 03:24:26] ...optimizers/trijoint.py.151: total_norm: 34.971725 [I 2022-10-27 03:24:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 530/2304 [I 2022-10-27 03:24:31] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:22 [I 2022-10-27 03:24:31] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00234 [I 2022-10-27 03:24:31] ...trap/engines/engine.py.230: loss: 0.11288 [I 2022-10-27 03:24:31] ...optimizers/trijoint.py.151: total_norm: 20.484068 [I 2022-10-27 03:24:37] ...trap/engines/engine.py.224: train: epoch 45 | batch 540/2304 [I 2022-10-27 03:24:37] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:23 [I 2022-10-27 03:24:37] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00237 [I 2022-10-27 03:24:37] ...trap/engines/engine.py.230: loss: 0.12333 [I 2022-10-27 03:24:37] ...optimizers/trijoint.py.151: total_norm: 9.750987 [I 2022-10-27 03:24:43] ...trap/engines/engine.py.224: train: epoch 45 | batch 550/2304 [I 2022-10-27 03:24:43] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:17 [I 2022-10-27 03:24:43] ...trap/engines/engine.py.229: process: 0.55414 | load: 0.00236 [I 2022-10-27 03:24:43] ...trap/engines/engine.py.230: loss: 0.17612 [I 2022-10-27 03:24:43] ...optimizers/trijoint.py.151: total_norm: 15.370061 [I 2022-10-27 03:24:48] ...trap/engines/engine.py.224: train: epoch 45 | batch 560/2304 [I 2022-10-27 03:24:48] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:16 [I 2022-10-27 03:24:48] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00235 [I 2022-10-27 03:24:48] ...trap/engines/engine.py.230: loss: 0.19303 [I 2022-10-27 03:24:48] ...optimizers/trijoint.py.151: total_norm: 53.243469 [I 2022-10-27 03:24:54] ...trap/engines/engine.py.224: train: epoch 45 | batch 570/2304 [I 2022-10-27 03:24:54] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:16:02 [I 2022-10-27 03:24:54] ...trap/engines/engine.py.229: process: 0.54438 | load: 0.00235 [I 2022-10-27 03:24:54] ...trap/engines/engine.py.230: loss: 0.15303 [I 2022-10-27 03:24:54] ...optimizers/trijoint.py.151: total_norm: 24.347616 [I 2022-10-27 03:24:59] ...trap/engines/engine.py.224: train: epoch 45 | batch 580/2304 [I 2022-10-27 03:24:59] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:57 [I 2022-10-27 03:24:59] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00249 [I 2022-10-27 03:24:59] ...trap/engines/engine.py.230: loss: 0.11362 [I 2022-10-27 03:24:59] ...optimizers/trijoint.py.151: total_norm: 10.481275 [I 2022-10-27 03:25:05] ...trap/engines/engine.py.224: train: epoch 45 | batch 590/2304 [I 2022-10-27 03:25:05] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:16:01 [I 2022-10-27 03:25:05] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00244 [I 2022-10-27 03:25:05] ...trap/engines/engine.py.230: loss: 0.14045 [I 2022-10-27 03:25:05] ...optimizers/trijoint.py.151: total_norm: 10.368474 [I 2022-10-27 03:25:10] ...trap/engines/engine.py.224: train: epoch 45 | batch 600/2304 [I 2022-10-27 03:25:10] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:40 [I 2022-10-27 03:25:10] ...trap/engines/engine.py.229: process: 0.54966 | load: 0.00243 [I 2022-10-27 03:25:10] ...trap/engines/engine.py.230: loss: 0.13982 [I 2022-10-27 03:25:10] ...optimizers/trijoint.py.151: total_norm: 18.186983 [I 2022-10-27 03:25:16] ...trap/engines/engine.py.224: train: epoch 45 | batch 610/2304 [I 2022-10-27 03:25:16] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:39 [I 2022-10-27 03:25:16] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00241 [I 2022-10-27 03:25:16] ...trap/engines/engine.py.230: loss: 0.20792 [I 2022-10-27 03:25:16] ...optimizers/trijoint.py.151: total_norm: 21.378462 [I 2022-10-27 03:25:22] ...trap/engines/engine.py.224: train: epoch 45 | batch 620/2304 [I 2022-10-27 03:25:22] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:27 [I 2022-10-27 03:25:22] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00236 [I 2022-10-27 03:25:22] ...trap/engines/engine.py.230: loss: 0.21982 [I 2022-10-27 03:25:22] ...optimizers/trijoint.py.151: total_norm: 16.285900 [I 2022-10-27 03:25:28] ...trap/engines/engine.py.224: train: epoch 45 | batch 630/2304 [I 2022-10-27 03:25:28] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:19:38 [I 2022-10-27 03:25:28] ...trap/engines/engine.py.229: process: 0.77418 | load: 0.00528 [I 2022-10-27 03:25:28] ...trap/engines/engine.py.230: loss: 0.16950 [I 2022-10-27 03:25:28] ...optimizers/trijoint.py.151: total_norm: 9.699490 [I 2022-10-27 03:25:34] ...trap/engines/engine.py.224: train: epoch 45 | batch 640/2304 [I 2022-10-27 03:25:34] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:15:48 [I 2022-10-27 03:25:34] ...trap/engines/engine.py.229: process: 0.56192 | load: 0.00238 [I 2022-10-27 03:25:34] ...trap/engines/engine.py.230: loss: 0.17306 [I 2022-10-27 03:25:34] ...optimizers/trijoint.py.151: total_norm: 18.410566 [I 2022-10-27 03:25:40] ...trap/engines/engine.py.224: train: epoch 45 | batch 650/2304 [I 2022-10-27 03:25:40] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:15:36 [I 2022-10-27 03:25:40] ...trap/engines/engine.py.229: process: 0.55354 | load: 0.00241 [I 2022-10-27 03:25:40] ...trap/engines/engine.py.230: loss: 0.10934 [I 2022-10-27 03:25:40] ...optimizers/trijoint.py.151: total_norm: 7.190953 [I 2022-10-27 03:25:45] ...trap/engines/engine.py.224: train: epoch 45 | batch 660/2304 [I 2022-10-27 03:25:45] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:15:17 [I 2022-10-27 03:25:45] ...trap/engines/engine.py.229: process: 0.54532 | load: 0.00241 [I 2022-10-27 03:25:45] ...trap/engines/engine.py.230: loss: 0.12342 [I 2022-10-27 03:25:45] ...optimizers/trijoint.py.151: total_norm: 8.817265 [I 2022-10-27 03:25:51] ...trap/engines/engine.py.224: train: epoch 45 | batch 670/2304 [I 2022-10-27 03:25:51] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:15:22 [I 2022-10-27 03:25:51] ...trap/engines/engine.py.229: process: 0.55926 | load: 0.00240 [I 2022-10-27 03:25:51] ...trap/engines/engine.py.230: loss: 0.14630 [I 2022-10-27 03:25:51] ...optimizers/trijoint.py.151: total_norm: 12.482257 [I 2022-10-27 03:25:56] ...trap/engines/engine.py.224: train: epoch 45 | batch 680/2304 [I 2022-10-27 03:25:56] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:15:02 [I 2022-10-27 03:25:56] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00241 [I 2022-10-27 03:25:56] ...trap/engines/engine.py.230: loss: 0.12698 [I 2022-10-27 03:25:56] ...optimizers/trijoint.py.151: total_norm: 32.661423 [I 2022-10-27 03:26:02] ...trap/engines/engine.py.224: train: epoch 45 | batch 690/2304 [I 2022-10-27 03:26:02] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:14:56 [I 2022-10-27 03:26:02] ...trap/engines/engine.py.229: process: 0.56480 | load: 0.00241 [I 2022-10-27 03:26:02] ...trap/engines/engine.py.230: loss: 0.15168 [I 2022-10-27 03:26:02] ...optimizers/trijoint.py.151: total_norm: 10.994422 [I 2022-10-27 03:26:08] ...trap/engines/engine.py.224: train: epoch 45 | batch 700/2304 [I 2022-10-27 03:26:08] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:14:51 [I 2022-10-27 03:26:08] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00241 [I 2022-10-27 03:26:08] ...trap/engines/engine.py.230: loss: 0.16057 [I 2022-10-27 03:26:08] ...optimizers/trijoint.py.151: total_norm: 14.718276 [I 2022-10-27 03:26:13] ...trap/engines/engine.py.224: train: epoch 45 | batch 710/2304 [I 2022-10-27 03:26:13] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:14:50 [I 2022-10-27 03:26:13] ...trap/engines/engine.py.229: process: 0.56866 | load: 0.00247 [I 2022-10-27 03:26:13] ...trap/engines/engine.py.230: loss: 0.14972 [I 2022-10-27 03:26:13] ...optimizers/trijoint.py.151: total_norm: 12.941651 [I 2022-10-27 03:26:19] ...trap/engines/engine.py.224: train: epoch 45 | batch 720/2304 [I 2022-10-27 03:26:19] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:14:43 [I 2022-10-27 03:26:19] ...trap/engines/engine.py.229: process: 0.54478 | load: 0.00241 [I 2022-10-27 03:26:19] ...trap/engines/engine.py.230: loss: 0.11956 [I 2022-10-27 03:26:19] ...optimizers/trijoint.py.151: total_norm: 10.568241 [I 2022-10-27 03:26:24] ...trap/engines/engine.py.224: train: epoch 45 | batch 730/2304 [I 2022-10-27 03:26:24] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:35 [I 2022-10-27 03:26:24] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00242 [I 2022-10-27 03:26:24] ...trap/engines/engine.py.230: loss: 0.21376 [I 2022-10-27 03:26:24] ...optimizers/trijoint.py.151: total_norm: 35.525074 [I 2022-10-27 03:26:30] ...trap/engines/engine.py.224: train: epoch 45 | batch 740/2304 [I 2022-10-27 03:26:30] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:14:44 [I 2022-10-27 03:26:30] ...trap/engines/engine.py.229: process: 0.56572 | load: 0.00239 [I 2022-10-27 03:26:30] ...trap/engines/engine.py.230: loss: 0.12197 [I 2022-10-27 03:26:30] ...optimizers/trijoint.py.151: total_norm: 9.817052 [I 2022-10-27 03:26:36] ...trap/engines/engine.py.224: train: epoch 45 | batch 750/2304 [I 2022-10-27 03:26:36] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:14:28 [I 2022-10-27 03:26:36] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00244 [I 2022-10-27 03:26:36] ...trap/engines/engine.py.230: loss: 0.10316 [I 2022-10-27 03:26:36] ...optimizers/trijoint.py.151: total_norm: 12.073255 [I 2022-10-27 03:26:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 760/2304 [I 2022-10-27 03:26:41] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:14:12 [I 2022-10-27 03:26:41] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00238 [I 2022-10-27 03:26:41] ...trap/engines/engine.py.230: loss: 0.12895 [I 2022-10-27 03:26:41] ...optimizers/trijoint.py.151: total_norm: 8.069835 [I 2022-10-27 03:26:47] ...trap/engines/engine.py.224: train: epoch 45 | batch 770/2304 [I 2022-10-27 03:26:47] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:14:10 [I 2022-10-27 03:26:47] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00244 [I 2022-10-27 03:26:47] ...trap/engines/engine.py.230: loss: 0.22175 [I 2022-10-27 03:26:47] ...optimizers/trijoint.py.151: total_norm: 13.437338 [I 2022-10-27 03:26:52] ...trap/engines/engine.py.224: train: epoch 45 | batch 780/2304 [I 2022-10-27 03:26:52] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:14:04 [I 2022-10-27 03:26:52] ...trap/engines/engine.py.229: process: 0.55936 | load: 0.00241 [I 2022-10-27 03:26:52] ...trap/engines/engine.py.230: loss: 0.18231 [I 2022-10-27 03:26:52] ...optimizers/trijoint.py.151: total_norm: 16.720722 [I 2022-10-27 03:26:58] ...trap/engines/engine.py.224: train: epoch 45 | batch 790/2304 [I 2022-10-27 03:26:58] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:13:57 [I 2022-10-27 03:26:58] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00241 [I 2022-10-27 03:26:58] ...trap/engines/engine.py.230: loss: 0.12202 [I 2022-10-27 03:26:58] ...optimizers/trijoint.py.151: total_norm: 9.624380 [I 2022-10-27 03:27:03] ...trap/engines/engine.py.224: train: epoch 45 | batch 800/2304 [I 2022-10-27 03:27:03] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:14:14 [I 2022-10-27 03:27:03] ...trap/engines/engine.py.229: process: 0.62402 | load: 0.00241 [I 2022-10-27 03:27:03] ...trap/engines/engine.py.230: loss: 0.26857 [I 2022-10-27 03:27:03] ...optimizers/trijoint.py.151: total_norm: 27.135817 [I 2022-10-27 03:27:09] ...trap/engines/engine.py.224: train: epoch 45 | batch 810/2304 [I 2022-10-27 03:27:09] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:13:51 [I 2022-10-27 03:27:09] ...trap/engines/engine.py.229: process: 0.55376 | load: 0.00242 [I 2022-10-27 03:27:09] ...trap/engines/engine.py.230: loss: 0.22532 [I 2022-10-27 03:27:09] ...optimizers/trijoint.py.151: total_norm: 20.864222 [I 2022-10-27 03:27:15] ...trap/engines/engine.py.224: train: epoch 45 | batch 820/2304 [I 2022-10-27 03:27:15] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:46 [I 2022-10-27 03:27:15] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00241 [I 2022-10-27 03:27:15] ...trap/engines/engine.py.230: loss: 0.14050 [I 2022-10-27 03:27:15] ...optimizers/trijoint.py.151: total_norm: 29.845596 [I 2022-10-27 03:27:20] ...trap/engines/engine.py.224: train: epoch 45 | batch 830/2304 [I 2022-10-27 03:27:20] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:13:36 [I 2022-10-27 03:27:20] ...trap/engines/engine.py.229: process: 0.55342 | load: 0.00243 [I 2022-10-27 03:27:20] ...trap/engines/engine.py.230: loss: 0.12174 [I 2022-10-27 03:27:20] ...optimizers/trijoint.py.151: total_norm: 9.260906 [I 2022-10-27 03:27:26] ...trap/engines/engine.py.224: train: epoch 45 | batch 840/2304 [I 2022-10-27 03:27:26] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:38 [I 2022-10-27 03:27:26] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00248 [I 2022-10-27 03:27:26] ...trap/engines/engine.py.230: loss: 0.15522 [I 2022-10-27 03:27:26] ...optimizers/trijoint.py.151: total_norm: 15.774415 [I 2022-10-27 03:27:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 850/2304 [I 2022-10-27 03:27:31] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:25 [I 2022-10-27 03:27:31] ...trap/engines/engine.py.229: process: 0.55007 | load: 0.00247 [I 2022-10-27 03:27:31] ...trap/engines/engine.py.230: loss: 0.17549 [I 2022-10-27 03:27:31] ...optimizers/trijoint.py.151: total_norm: 32.737053 [I 2022-10-27 03:27:37] ...trap/engines/engine.py.224: train: epoch 45 | batch 860/2304 [I 2022-10-27 03:27:37] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:13:23 [I 2022-10-27 03:27:37] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00241 [I 2022-10-27 03:27:37] ...trap/engines/engine.py.230: loss: 0.13085 [I 2022-10-27 03:27:37] ...optimizers/trijoint.py.151: total_norm: 12.212784 [I 2022-10-27 03:27:42] ...trap/engines/engine.py.224: train: epoch 45 | batch 870/2304 [I 2022-10-27 03:27:42] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:10 [I 2022-10-27 03:27:42] ...trap/engines/engine.py.229: process: 0.54750 | load: 0.00244 [I 2022-10-27 03:27:42] ...trap/engines/engine.py.230: loss: 0.18070 [I 2022-10-27 03:27:42] ...optimizers/trijoint.py.151: total_norm: 19.882149 [I 2022-10-27 03:27:48] ...trap/engines/engine.py.224: train: epoch 45 | batch 880/2304 [I 2022-10-27 03:27:48] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:13:10 [I 2022-10-27 03:27:48] ...trap/engines/engine.py.229: process: 0.55694 | load: 0.00243 [I 2022-10-27 03:27:48] ...trap/engines/engine.py.230: loss: 0.15942 [I 2022-10-27 03:27:48] ...optimizers/trijoint.py.151: total_norm: 19.971889 [I 2022-10-27 03:27:54] ...trap/engines/engine.py.224: train: epoch 45 | batch 890/2304 [I 2022-10-27 03:27:54] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:13:09 [I 2022-10-27 03:27:54] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00243 [I 2022-10-27 03:27:54] ...trap/engines/engine.py.230: loss: 0.13507 [I 2022-10-27 03:27:54] ...optimizers/trijoint.py.151: total_norm: 17.371262 [I 2022-10-27 03:27:59] ...trap/engines/engine.py.224: train: epoch 45 | batch 900/2304 [I 2022-10-27 03:27:59] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:13:00 [I 2022-10-27 03:27:59] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00238 [I 2022-10-27 03:27:59] ...trap/engines/engine.py.230: loss: 0.18818 [I 2022-10-27 03:27:59] ...optimizers/trijoint.py.151: total_norm: 28.222847 [I 2022-10-27 03:28:05] ...trap/engines/engine.py.224: train: epoch 45 | batch 910/2304 [I 2022-10-27 03:28:05] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:13:06 [I 2022-10-27 03:28:05] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00242 [I 2022-10-27 03:28:05] ...trap/engines/engine.py.230: loss: 0.15358 [I 2022-10-27 03:28:05] ...optimizers/trijoint.py.151: total_norm: 11.824626 [I 2022-10-27 03:28:10] ...trap/engines/engine.py.224: train: epoch 45 | batch 920/2304 [I 2022-10-27 03:28:10] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:50 [I 2022-10-27 03:28:10] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00274 [I 2022-10-27 03:28:10] ...trap/engines/engine.py.230: loss: 0.15687 [I 2022-10-27 03:28:10] ...optimizers/trijoint.py.151: total_norm: 16.424980 [I 2022-10-27 03:28:16] ...trap/engines/engine.py.224: train: epoch 45 | batch 930/2304 [I 2022-10-27 03:28:16] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:46 [I 2022-10-27 03:28:16] ...trap/engines/engine.py.229: process: 0.54983 | load: 0.00240 [I 2022-10-27 03:28:16] ...trap/engines/engine.py.230: loss: 0.16917 [I 2022-10-27 03:28:16] ...optimizers/trijoint.py.151: total_norm: 28.700251 [I 2022-10-27 03:28:22] ...trap/engines/engine.py.224: train: epoch 45 | batch 940/2304 [I 2022-10-27 03:28:22] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:37 [I 2022-10-27 03:28:22] ...trap/engines/engine.py.229: process: 0.55072 | load: 0.00241 [I 2022-10-27 03:28:22] ...trap/engines/engine.py.230: loss: 0.20643 [I 2022-10-27 03:28:22] ...optimizers/trijoint.py.151: total_norm: 17.975182 [I 2022-10-27 03:28:27] ...trap/engines/engine.py.224: train: epoch 45 | batch 950/2304 [I 2022-10-27 03:28:27] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:12:31 [I 2022-10-27 03:28:27] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00242 [I 2022-10-27 03:28:27] ...trap/engines/engine.py.230: loss: 0.16516 [I 2022-10-27 03:28:27] ...optimizers/trijoint.py.151: total_norm: 14.414730 [I 2022-10-27 03:28:33] ...trap/engines/engine.py.224: train: epoch 45 | batch 960/2304 [I 2022-10-27 03:28:33] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:33 [I 2022-10-27 03:28:33] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00263 [I 2022-10-27 03:28:33] ...trap/engines/engine.py.230: loss: 0.16792 [I 2022-10-27 03:28:33] ...optimizers/trijoint.py.151: total_norm: 12.908353 [I 2022-10-27 03:28:38] ...trap/engines/engine.py.224: train: epoch 45 | batch 970/2304 [I 2022-10-27 03:28:38] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:18 [I 2022-10-27 03:28:38] ...trap/engines/engine.py.229: process: 0.54175 | load: 0.00239 [I 2022-10-27 03:28:38] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-27 03:28:38] ...optimizers/trijoint.py.151: total_norm: 9.867120 [I 2022-10-27 03:28:44] ...trap/engines/engine.py.224: train: epoch 45 | batch 980/2304 [I 2022-10-27 03:28:44] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:12:15 [I 2022-10-27 03:28:44] ...trap/engines/engine.py.229: process: 0.54407 | load: 0.00239 [I 2022-10-27 03:28:44] ...trap/engines/engine.py.230: loss: 0.15611 [I 2022-10-27 03:28:44] ...optimizers/trijoint.py.151: total_norm: 17.068829 [I 2022-10-27 03:28:49] ...trap/engines/engine.py.224: train: epoch 45 | batch 990/2304 [I 2022-10-27 03:28:49] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:12:13 [I 2022-10-27 03:28:49] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00239 [I 2022-10-27 03:28:49] ...trap/engines/engine.py.230: loss: 0.17894 [I 2022-10-27 03:28:49] ...optimizers/trijoint.py.151: total_norm: 17.470890 [I 2022-10-27 03:28:55] ...trap/engines/engine.py.224: train: epoch 45 | batch 1000/2304 [I 2022-10-27 03:28:55] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:12:07 [I 2022-10-27 03:28:55] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00239 [I 2022-10-27 03:28:55] ...trap/engines/engine.py.230: loss: 0.12912 [I 2022-10-27 03:28:55] ...optimizers/trijoint.py.151: total_norm: 16.042612 [I 2022-10-27 03:29:01] ...trap/engines/engine.py.224: train: epoch 45 | batch 1010/2304 [I 2022-10-27 03:29:01] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:12:05 [I 2022-10-27 03:29:01] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00250 [I 2022-10-27 03:29:01] ...trap/engines/engine.py.230: loss: 0.16646 [I 2022-10-27 03:29:01] ...optimizers/trijoint.py.151: total_norm: 14.990505 [I 2022-10-27 03:29:06] ...trap/engines/engine.py.224: train: epoch 45 | batch 1020/2304 [I 2022-10-27 03:29:06] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:12:02 [I 2022-10-27 03:29:06] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00251 [I 2022-10-27 03:29:06] ...trap/engines/engine.py.230: loss: 0.22154 [I 2022-10-27 03:29:06] ...optimizers/trijoint.py.151: total_norm: 20.484619 [I 2022-10-27 03:29:12] ...trap/engines/engine.py.224: train: epoch 45 | batch 1030/2304 [I 2022-10-27 03:29:12] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:50 [I 2022-10-27 03:29:12] ...trap/engines/engine.py.229: process: 0.55266 | load: 0.00244 [I 2022-10-27 03:29:12] ...trap/engines/engine.py.230: loss: 0.16096 [I 2022-10-27 03:29:12] ...optimizers/trijoint.py.151: total_norm: 19.155718 [I 2022-10-27 03:29:17] ...trap/engines/engine.py.224: train: epoch 45 | batch 1040/2304 [I 2022-10-27 03:29:17] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:37 [I 2022-10-27 03:29:17] ...trap/engines/engine.py.229: process: 0.55038 | load: 0.00240 [I 2022-10-27 03:29:17] ...trap/engines/engine.py.230: loss: 0.16676 [I 2022-10-27 03:29:17] ...optimizers/trijoint.py.151: total_norm: 18.614685 [I 2022-10-27 03:29:23] ...trap/engines/engine.py.224: train: epoch 45 | batch 1050/2304 [I 2022-10-27 03:29:23] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:33 [I 2022-10-27 03:29:23] ...trap/engines/engine.py.229: process: 0.54097 | load: 0.00244 [I 2022-10-27 03:29:23] ...trap/engines/engine.py.230: loss: 0.20696 [I 2022-10-27 03:29:23] ...optimizers/trijoint.py.151: total_norm: 13.616455 [I 2022-10-27 03:29:29] ...trap/engines/engine.py.224: train: epoch 45 | batch 1060/2304 [I 2022-10-27 03:29:29] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:33 [I 2022-10-27 03:29:29] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00244 [I 2022-10-27 03:29:29] ...trap/engines/engine.py.230: loss: 0.16026 [I 2022-10-27 03:29:29] ...optimizers/trijoint.py.151: total_norm: 14.053442 [I 2022-10-27 03:29:34] ...trap/engines/engine.py.224: train: epoch 45 | batch 1070/2304 [I 2022-10-27 03:29:34] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:25 [I 2022-10-27 03:29:34] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00260 [I 2022-10-27 03:29:34] ...trap/engines/engine.py.230: loss: 0.15540 [I 2022-10-27 03:29:34] ...optimizers/trijoint.py.151: total_norm: 26.852249 [I 2022-10-27 03:29:40] ...trap/engines/engine.py.224: train: epoch 45 | batch 1080/2304 [I 2022-10-27 03:29:40] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:18 [I 2022-10-27 03:29:40] ...trap/engines/engine.py.229: process: 0.55602 | load: 0.00241 [I 2022-10-27 03:29:40] ...trap/engines/engine.py.230: loss: 0.21576 [I 2022-10-27 03:29:40] ...optimizers/trijoint.py.151: total_norm: 25.028494 [I 2022-10-27 03:29:45] ...trap/engines/engine.py.224: train: epoch 45 | batch 1090/2304 [I 2022-10-27 03:29:45] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:14 [I 2022-10-27 03:29:45] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00240 [I 2022-10-27 03:29:45] ...trap/engines/engine.py.230: loss: 0.21181 [I 2022-10-27 03:29:45] ...optimizers/trijoint.py.151: total_norm: 22.776722 [I 2022-10-27 03:29:51] ...trap/engines/engine.py.224: train: epoch 45 | batch 1100/2304 [I 2022-10-27 03:29:51] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:11:08 [I 2022-10-27 03:29:51] ...trap/engines/engine.py.229: process: 0.56538 | load: 0.00241 [I 2022-10-27 03:29:51] ...trap/engines/engine.py.230: loss: 0.16627 [I 2022-10-27 03:29:51] ...optimizers/trijoint.py.151: total_norm: 12.601072 [I 2022-10-27 03:29:56] ...trap/engines/engine.py.224: train: epoch 45 | batch 1110/2304 [I 2022-10-27 03:29:56] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:11:01 [I 2022-10-27 03:29:56] ...trap/engines/engine.py.229: process: 0.55496 | load: 0.00278 [I 2022-10-27 03:29:56] ...trap/engines/engine.py.230: loss: 0.23585 [I 2022-10-27 03:29:56] ...optimizers/trijoint.py.151: total_norm: 18.438200 [I 2022-10-27 03:30:02] ...trap/engines/engine.py.224: train: epoch 45 | batch 1120/2304 [I 2022-10-27 03:30:02] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:57 [I 2022-10-27 03:30:02] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00241 [I 2022-10-27 03:30:02] ...trap/engines/engine.py.230: loss: 0.14237 [I 2022-10-27 03:30:02] ...optimizers/trijoint.py.151: total_norm: 12.990153 [I 2022-10-27 03:30:08] ...trap/engines/engine.py.224: train: epoch 45 | batch 1130/2304 [I 2022-10-27 03:30:08] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:11:06 [I 2022-10-27 03:30:08] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00238 [I 2022-10-27 03:30:08] ...trap/engines/engine.py.230: loss: 0.16228 [I 2022-10-27 03:30:08] ...optimizers/trijoint.py.151: total_norm: 12.670660 [I 2022-10-27 03:30:13] ...trap/engines/engine.py.224: train: epoch 45 | batch 1140/2304 [I 2022-10-27 03:30:13] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:47 [I 2022-10-27 03:30:13] ...trap/engines/engine.py.229: process: 0.56240 | load: 0.00240 [I 2022-10-27 03:30:13] ...trap/engines/engine.py.230: loss: 0.17993 [I 2022-10-27 03:30:13] ...optimizers/trijoint.py.151: total_norm: 16.376703 [I 2022-10-27 03:30:19] ...trap/engines/engine.py.224: train: epoch 45 | batch 1150/2304 [I 2022-10-27 03:30:19] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:43 [I 2022-10-27 03:30:19] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00246 [I 2022-10-27 03:30:19] ...trap/engines/engine.py.230: loss: 0.18134 [I 2022-10-27 03:30:19] ...optimizers/trijoint.py.151: total_norm: 16.967512 [I 2022-10-27 03:30:24] ...trap/engines/engine.py.224: train: epoch 45 | batch 1160/2304 [I 2022-10-27 03:30:24] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:37 [I 2022-10-27 03:30:24] ...trap/engines/engine.py.229: process: 0.54871 | load: 0.00245 [I 2022-10-27 03:30:24] ...trap/engines/engine.py.230: loss: 0.28129 [I 2022-10-27 03:30:24] ...optimizers/trijoint.py.151: total_norm: 15.138592 [I 2022-10-27 03:30:30] ...trap/engines/engine.py.224: train: epoch 45 | batch 1170/2304 [I 2022-10-27 03:30:30] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:34 [I 2022-10-27 03:30:30] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00245 [I 2022-10-27 03:30:30] ...trap/engines/engine.py.230: loss: 0.17152 [I 2022-10-27 03:30:30] ...optimizers/trijoint.py.151: total_norm: 9.334013 [I 2022-10-27 03:30:35] ...trap/engines/engine.py.224: train: epoch 45 | batch 1180/2304 [I 2022-10-27 03:30:35] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:28 [I 2022-10-27 03:30:35] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00252 [I 2022-10-27 03:30:35] ...trap/engines/engine.py.230: loss: 0.23035 [I 2022-10-27 03:30:35] ...optimizers/trijoint.py.151: total_norm: 28.988871 [I 2022-10-27 03:30:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 1190/2304 [I 2022-10-27 03:30:41] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:21 [I 2022-10-27 03:30:41] ...trap/engines/engine.py.229: process: 0.56280 | load: 0.00248 [I 2022-10-27 03:30:41] ...trap/engines/engine.py.230: loss: 0.18341 [I 2022-10-27 03:30:41] ...optimizers/trijoint.py.151: total_norm: 15.871373 [I 2022-10-27 03:30:47] ...trap/engines/engine.py.224: train: epoch 45 | batch 1200/2304 [I 2022-10-27 03:30:47] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:10:21 [I 2022-10-27 03:30:47] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00243 [I 2022-10-27 03:30:47] ...trap/engines/engine.py.230: loss: 0.23494 [I 2022-10-27 03:30:47] ...optimizers/trijoint.py.151: total_norm: 11.828157 [I 2022-10-27 03:30:52] ...trap/engines/engine.py.224: train: epoch 45 | batch 1210/2304 [I 2022-10-27 03:30:52] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:10:06 [I 2022-10-27 03:30:52] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00266 [I 2022-10-27 03:30:52] ...trap/engines/engine.py.230: loss: 0.15278 [I 2022-10-27 03:30:52] ...optimizers/trijoint.py.151: total_norm: 11.729461 [I 2022-10-27 03:30:58] ...trap/engines/engine.py.224: train: epoch 45 | batch 1220/2304 [I 2022-10-27 03:30:58] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:10:02 [I 2022-10-27 03:30:58] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00245 [I 2022-10-27 03:30:58] ...trap/engines/engine.py.230: loss: 0.19728 [I 2022-10-27 03:30:58] ...optimizers/trijoint.py.151: total_norm: 12.006989 [I 2022-10-27 03:31:03] ...trap/engines/engine.py.224: train: epoch 45 | batch 1230/2304 [I 2022-10-27 03:31:03] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:59 [I 2022-10-27 03:31:03] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00244 [I 2022-10-27 03:31:03] ...trap/engines/engine.py.230: loss: 0.22852 [I 2022-10-27 03:31:03] ...optimizers/trijoint.py.151: total_norm: 11.700315 [I 2022-10-27 03:31:09] ...trap/engines/engine.py.224: train: epoch 45 | batch 1240/2304 [I 2022-10-27 03:31:09] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:49 [I 2022-10-27 03:31:09] ...trap/engines/engine.py.229: process: 0.54161 | load: 0.00243 [I 2022-10-27 03:31:09] ...trap/engines/engine.py.230: loss: 0.18521 [I 2022-10-27 03:31:09] ...optimizers/trijoint.py.151: total_norm: 25.741293 [I 2022-10-27 03:31:15] ...trap/engines/engine.py.224: train: epoch 45 | batch 1250/2304 [I 2022-10-27 03:31:15] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:43 [I 2022-10-27 03:31:15] ...trap/engines/engine.py.229: process: 0.55271 | load: 0.00243 [I 2022-10-27 03:31:15] ...trap/engines/engine.py.230: loss: 0.19648 [I 2022-10-27 03:31:15] ...optimizers/trijoint.py.151: total_norm: 17.235800 [I 2022-10-27 03:31:20] ...trap/engines/engine.py.224: train: epoch 45 | batch 1260/2304 [I 2022-10-27 03:31:20] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:44 [I 2022-10-27 03:31:20] ...trap/engines/engine.py.229: process: 0.57103 | load: 0.00245 [I 2022-10-27 03:31:20] ...trap/engines/engine.py.230: loss: 0.12008 [I 2022-10-27 03:31:20] ...optimizers/trijoint.py.151: total_norm: 13.388158 [I 2022-10-27 03:31:26] ...trap/engines/engine.py.224: train: epoch 45 | batch 1270/2304 [I 2022-10-27 03:31:26] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:36 [I 2022-10-27 03:31:26] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00241 [I 2022-10-27 03:31:26] ...trap/engines/engine.py.230: loss: 0.11831 [I 2022-10-27 03:31:26] ...optimizers/trijoint.py.151: total_norm: 14.813740 [I 2022-10-27 03:31:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 1280/2304 [I 2022-10-27 03:31:31] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:29 [I 2022-10-27 03:31:31] ...trap/engines/engine.py.229: process: 0.54645 | load: 0.00242 [I 2022-10-27 03:31:31] ...trap/engines/engine.py.230: loss: 0.13995 [I 2022-10-27 03:31:31] ...optimizers/trijoint.py.151: total_norm: 12.497066 [I 2022-10-27 03:31:37] ...trap/engines/engine.py.224: train: epoch 45 | batch 1290/2304 [I 2022-10-27 03:31:37] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:25 [I 2022-10-27 03:31:37] ...trap/engines/engine.py.229: process: 0.56669 | load: 0.00239 [I 2022-10-27 03:31:37] ...trap/engines/engine.py.230: loss: 0.20048 [I 2022-10-27 03:31:37] ...optimizers/trijoint.py.151: total_norm: 16.350012 [I 2022-10-27 03:31:43] ...trap/engines/engine.py.224: train: epoch 45 | batch 1300/2304 [I 2022-10-27 03:31:43] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:20 [I 2022-10-27 03:31:43] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00241 [I 2022-10-27 03:31:43] ...trap/engines/engine.py.230: loss: 0.16695 [I 2022-10-27 03:31:43] ...optimizers/trijoint.py.151: total_norm: 15.252201 [I 2022-10-27 03:31:48] ...trap/engines/engine.py.224: train: epoch 45 | batch 1310/2304 [I 2022-10-27 03:31:48] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:09:12 [I 2022-10-27 03:31:48] ...trap/engines/engine.py.229: process: 0.55602 | load: 0.00255 [I 2022-10-27 03:31:48] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-27 03:31:48] ...optimizers/trijoint.py.151: total_norm: 12.737521 [I 2022-10-27 03:31:54] ...trap/engines/engine.py.224: train: epoch 45 | batch 1320/2304 [I 2022-10-27 03:31:54] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:09:07 [I 2022-10-27 03:31:54] ...trap/engines/engine.py.229: process: 0.55184 | load: 0.00252 [I 2022-10-27 03:31:54] ...trap/engines/engine.py.230: loss: 0.15110 [I 2022-10-27 03:31:54] ...optimizers/trijoint.py.151: total_norm: 14.269241 [I 2022-10-27 03:31:59] ...trap/engines/engine.py.224: train: epoch 45 | batch 1330/2304 [I 2022-10-27 03:31:59] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:09:02 [I 2022-10-27 03:31:59] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00247 [I 2022-10-27 03:31:59] ...trap/engines/engine.py.230: loss: 0.11054 [I 2022-10-27 03:31:59] ...optimizers/trijoint.py.151: total_norm: 11.401880 [I 2022-10-27 03:32:05] ...trap/engines/engine.py.224: train: epoch 45 | batch 1340/2304 [I 2022-10-27 03:32:05] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:56 [I 2022-10-27 03:32:05] ...trap/engines/engine.py.229: process: 0.54778 | load: 0.00240 [I 2022-10-27 03:32:05] ...trap/engines/engine.py.230: loss: 0.15877 [I 2022-10-27 03:32:05] ...optimizers/trijoint.py.151: total_norm: 18.959764 [I 2022-10-27 03:32:10] ...trap/engines/engine.py.224: train: epoch 45 | batch 1350/2304 [I 2022-10-27 03:32:10] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:52 [I 2022-10-27 03:32:10] ...trap/engines/engine.py.229: process: 0.56984 | load: 0.00245 [I 2022-10-27 03:32:10] ...trap/engines/engine.py.230: loss: 0.19600 [I 2022-10-27 03:32:10] ...optimizers/trijoint.py.151: total_norm: 22.273123 [I 2022-10-27 03:32:16] ...trap/engines/engine.py.224: train: epoch 45 | batch 1360/2304 [I 2022-10-27 03:32:16] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:45 [I 2022-10-27 03:32:16] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00244 [I 2022-10-27 03:32:16] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-27 03:32:16] ...optimizers/trijoint.py.151: total_norm: 8.328578 [I 2022-10-27 03:32:22] ...trap/engines/engine.py.224: train: epoch 45 | batch 1370/2304 [I 2022-10-27 03:32:22] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:37 [I 2022-10-27 03:32:22] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00242 [I 2022-10-27 03:32:22] ...trap/engines/engine.py.230: loss: 0.20687 [I 2022-10-27 03:32:22] ...optimizers/trijoint.py.151: total_norm: 26.101871 [I 2022-10-27 03:32:27] ...trap/engines/engine.py.224: train: epoch 45 | batch 1380/2304 [I 2022-10-27 03:32:27] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:36 [I 2022-10-27 03:32:27] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00245 [I 2022-10-27 03:32:27] ...trap/engines/engine.py.230: loss: 0.12383 [I 2022-10-27 03:32:27] ...optimizers/trijoint.py.151: total_norm: 8.085554 [I 2022-10-27 03:32:33] ...trap/engines/engine.py.224: train: epoch 45 | batch 1390/2304 [I 2022-10-27 03:32:33] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:27 [I 2022-10-27 03:32:33] ...trap/engines/engine.py.229: process: 0.56742 | load: 0.00252 [I 2022-10-27 03:32:33] ...trap/engines/engine.py.230: loss: 0.20877 [I 2022-10-27 03:32:33] ...optimizers/trijoint.py.151: total_norm: 32.468365 [I 2022-10-27 03:32:38] ...trap/engines/engine.py.224: train: epoch 45 | batch 1400/2304 [I 2022-10-27 03:32:38] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:19 [I 2022-10-27 03:32:38] ...trap/engines/engine.py.229: process: 0.56492 | load: 0.00239 [I 2022-10-27 03:32:38] ...trap/engines/engine.py.230: loss: 0.13677 [I 2022-10-27 03:32:38] ...optimizers/trijoint.py.151: total_norm: 21.961884 [I 2022-10-27 03:32:44] ...trap/engines/engine.py.224: train: epoch 45 | batch 1410/2304 [I 2022-10-27 03:32:44] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:08:15 [I 2022-10-27 03:32:44] ...trap/engines/engine.py.229: process: 0.54915 | load: 0.00243 [I 2022-10-27 03:32:44] ...trap/engines/engine.py.230: loss: 0.14184 [I 2022-10-27 03:32:44] ...optimizers/trijoint.py.151: total_norm: 16.831148 [I 2022-10-27 03:32:49] ...trap/engines/engine.py.224: train: epoch 45 | batch 1420/2304 [I 2022-10-27 03:32:49] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:08:08 [I 2022-10-27 03:32:49] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00242 [I 2022-10-27 03:32:49] ...trap/engines/engine.py.230: loss: 0.10970 [I 2022-10-27 03:32:49] ...optimizers/trijoint.py.151: total_norm: 15.020215 [I 2022-10-27 03:32:55] ...trap/engines/engine.py.224: train: epoch 45 | batch 1430/2304 [I 2022-10-27 03:32:55] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:08:05 [I 2022-10-27 03:32:55] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00241 [I 2022-10-27 03:32:55] ...trap/engines/engine.py.230: loss: 0.15235 [I 2022-10-27 03:32:55] ...optimizers/trijoint.py.151: total_norm: 10.673717 [I 2022-10-27 03:33:00] ...trap/engines/engine.py.224: train: epoch 45 | batch 1440/2304 [I 2022-10-27 03:33:00] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:08:02 [I 2022-10-27 03:33:00] ...trap/engines/engine.py.229: process: 0.55128 | load: 0.00243 [I 2022-10-27 03:33:00] ...trap/engines/engine.py.230: loss: 0.17690 [I 2022-10-27 03:33:00] ...optimizers/trijoint.py.151: total_norm: 26.992693 [I 2022-10-27 03:33:06] ...trap/engines/engine.py.224: train: epoch 45 | batch 1450/2304 [I 2022-10-27 03:33:06] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:55 [I 2022-10-27 03:33:06] ...trap/engines/engine.py.229: process: 0.56256 | load: 0.00245 [I 2022-10-27 03:33:06] ...trap/engines/engine.py.230: loss: 0.15918 [I 2022-10-27 03:33:06] ...optimizers/trijoint.py.151: total_norm: 18.514811 [I 2022-10-27 03:33:12] ...trap/engines/engine.py.224: train: epoch 45 | batch 1460/2304 [I 2022-10-27 03:33:12] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:49 [I 2022-10-27 03:33:12] ...trap/engines/engine.py.229: process: 0.55145 | load: 0.00286 [I 2022-10-27 03:33:12] ...trap/engines/engine.py.230: loss: 0.14816 [I 2022-10-27 03:33:12] ...optimizers/trijoint.py.151: total_norm: 11.731411 [I 2022-10-27 03:33:17] ...trap/engines/engine.py.224: train: epoch 45 | batch 1470/2304 [I 2022-10-27 03:33:17] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:43 [I 2022-10-27 03:33:17] ...trap/engines/engine.py.229: process: 0.55737 | load: 0.00247 [I 2022-10-27 03:33:17] ...trap/engines/engine.py.230: loss: 0.14868 [I 2022-10-27 03:33:17] ...optimizers/trijoint.py.151: total_norm: 15.714552 [I 2022-10-27 03:33:23] ...trap/engines/engine.py.224: train: epoch 45 | batch 1480/2304 [I 2022-10-27 03:33:23] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:38 [I 2022-10-27 03:33:23] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00241 [I 2022-10-27 03:33:23] ...trap/engines/engine.py.230: loss: 0.12754 [I 2022-10-27 03:33:23] ...optimizers/trijoint.py.151: total_norm: 17.832441 [I 2022-10-27 03:33:28] ...trap/engines/engine.py.224: train: epoch 45 | batch 1490/2304 [I 2022-10-27 03:33:28] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:34 [I 2022-10-27 03:33:28] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00244 [I 2022-10-27 03:33:28] ...trap/engines/engine.py.230: loss: 0.14382 [I 2022-10-27 03:33:28] ...optimizers/trijoint.py.151: total_norm: 16.744261 [I 2022-10-27 03:33:34] ...trap/engines/engine.py.224: train: epoch 45 | batch 1500/2304 [I 2022-10-27 03:33:34] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:27 [I 2022-10-27 03:33:34] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00245 [I 2022-10-27 03:33:34] ...trap/engines/engine.py.230: loss: 0.14642 [I 2022-10-27 03:33:34] ...optimizers/trijoint.py.151: total_norm: 15.897593 [I 2022-10-27 03:33:39] ...trap/engines/engine.py.224: train: epoch 45 | batch 1510/2304 [I 2022-10-27 03:33:39] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:23 [I 2022-10-27 03:33:39] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00244 [I 2022-10-27 03:33:39] ...trap/engines/engine.py.230: loss: 0.16272 [I 2022-10-27 03:33:39] ...optimizers/trijoint.py.151: total_norm: 23.495695 [I 2022-10-27 03:33:45] ...trap/engines/engine.py.224: train: epoch 45 | batch 1520/2304 [I 2022-10-27 03:33:45] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:14 [I 2022-10-27 03:33:45] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00243 [I 2022-10-27 03:33:45] ...trap/engines/engine.py.230: loss: 0.14245 [I 2022-10-27 03:33:45] ...optimizers/trijoint.py.151: total_norm: 15.941817 [I 2022-10-27 03:33:51] ...trap/engines/engine.py.224: train: epoch 45 | batch 1530/2304 [I 2022-10-27 03:33:51] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:07:09 [I 2022-10-27 03:33:51] ...trap/engines/engine.py.229: process: 0.55737 | load: 0.00242 [I 2022-10-27 03:33:51] ...trap/engines/engine.py.230: loss: 0.19873 [I 2022-10-27 03:33:51] ...optimizers/trijoint.py.151: total_norm: 18.412235 [I 2022-10-27 03:33:56] ...trap/engines/engine.py.224: train: epoch 45 | batch 1540/2304 [I 2022-10-27 03:33:56] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:07:04 [I 2022-10-27 03:33:56] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00243 [I 2022-10-27 03:33:56] ...trap/engines/engine.py.230: loss: 0.20635 [I 2022-10-27 03:33:56] ...optimizers/trijoint.py.151: total_norm: 14.884275 [I 2022-10-27 03:34:02] ...trap/engines/engine.py.224: train: epoch 45 | batch 1550/2304 [I 2022-10-27 03:34:02] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:59 [I 2022-10-27 03:34:02] ...trap/engines/engine.py.229: process: 0.54959 | load: 0.00243 [I 2022-10-27 03:34:02] ...trap/engines/engine.py.230: loss: 0.22308 [I 2022-10-27 03:34:02] ...optimizers/trijoint.py.151: total_norm: 23.723164 [I 2022-10-27 03:34:07] ...trap/engines/engine.py.224: train: epoch 45 | batch 1560/2304 [I 2022-10-27 03:34:07] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:51 [I 2022-10-27 03:34:07] ...trap/engines/engine.py.229: process: 0.55399 | load: 0.00242 [I 2022-10-27 03:34:07] ...trap/engines/engine.py.230: loss: 0.25673 [I 2022-10-27 03:34:07] ...optimizers/trijoint.py.151: total_norm: 17.409924 [I 2022-10-27 03:34:13] ...trap/engines/engine.py.224: train: epoch 45 | batch 1570/2304 [I 2022-10-27 03:34:13] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:47 [I 2022-10-27 03:34:13] ...trap/engines/engine.py.229: process: 0.55950 | load: 0.00242 [I 2022-10-27 03:34:13] ...trap/engines/engine.py.230: loss: 0.17989 [I 2022-10-27 03:34:13] ...optimizers/trijoint.py.151: total_norm: 33.234062 [I 2022-10-27 03:34:18] ...trap/engines/engine.py.224: train: epoch 45 | batch 1580/2304 [I 2022-10-27 03:34:18] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:43 [I 2022-10-27 03:34:18] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00239 [I 2022-10-27 03:34:18] ...trap/engines/engine.py.230: loss: 0.17195 [I 2022-10-27 03:34:18] ...optimizers/trijoint.py.151: total_norm: 11.778092 [I 2022-10-27 03:34:24] ...trap/engines/engine.py.224: train: epoch 45 | batch 1590/2304 [I 2022-10-27 03:34:24] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:37 [I 2022-10-27 03:34:24] ...trap/engines/engine.py.229: process: 0.56067 | load: 0.00245 [I 2022-10-27 03:34:24] ...trap/engines/engine.py.230: loss: 0.16703 [I 2022-10-27 03:34:24] ...optimizers/trijoint.py.151: total_norm: 9.982102 [I 2022-10-27 03:34:30] ...trap/engines/engine.py.224: train: epoch 45 | batch 1600/2304 [I 2022-10-27 03:34:30] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:32 [I 2022-10-27 03:34:30] ...trap/engines/engine.py.229: process: 0.56546 | load: 0.00244 [I 2022-10-27 03:34:30] ...trap/engines/engine.py.230: loss: 0.15119 [I 2022-10-27 03:34:30] ...optimizers/trijoint.py.151: total_norm: 15.799737 [I 2022-10-27 03:34:35] ...trap/engines/engine.py.224: train: epoch 45 | batch 1610/2304 [I 2022-10-27 03:34:35] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:25 [I 2022-10-27 03:34:35] ...trap/engines/engine.py.229: process: 0.55426 | load: 0.00242 [I 2022-10-27 03:34:35] ...trap/engines/engine.py.230: loss: 0.17787 [I 2022-10-27 03:34:35] ...optimizers/trijoint.py.151: total_norm: 19.207750 [I 2022-10-27 03:34:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 1620/2304 [I 2022-10-27 03:34:41] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:19 [I 2022-10-27 03:34:41] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00241 [I 2022-10-27 03:34:41] ...trap/engines/engine.py.230: loss: 0.25607 [I 2022-10-27 03:34:41] ...optimizers/trijoint.py.151: total_norm: 19.792351 [I 2022-10-27 03:34:46] ...trap/engines/engine.py.224: train: epoch 45 | batch 1630/2304 [I 2022-10-27 03:34:46] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:14 [I 2022-10-27 03:34:46] ...trap/engines/engine.py.229: process: 0.56750 | load: 0.00254 [I 2022-10-27 03:34:46] ...trap/engines/engine.py.230: loss: 0.16917 [I 2022-10-27 03:34:46] ...optimizers/trijoint.py.151: total_norm: 11.043113 [I 2022-10-27 03:34:52] ...trap/engines/engine.py.224: train: epoch 45 | batch 1640/2304 [I 2022-10-27 03:34:52] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:06:12 [I 2022-10-27 03:34:52] ...trap/engines/engine.py.229: process: 0.56156 | load: 0.00249 [I 2022-10-27 03:34:52] ...trap/engines/engine.py.230: loss: 0.24022 [I 2022-10-27 03:34:52] ...optimizers/trijoint.py.151: total_norm: 26.547483 [I 2022-10-27 03:34:57] ...trap/engines/engine.py.224: train: epoch 45 | batch 1650/2304 [I 2022-10-27 03:34:57] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:06:04 [I 2022-10-27 03:34:57] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00249 [I 2022-10-27 03:34:57] ...trap/engines/engine.py.230: loss: 0.17959 [I 2022-10-27 03:34:57] ...optimizers/trijoint.py.151: total_norm: 20.667389 [I 2022-10-27 03:35:03] ...trap/engines/engine.py.224: train: epoch 45 | batch 1660/2304 [I 2022-10-27 03:35:03] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:58 [I 2022-10-27 03:35:03] ...trap/engines/engine.py.229: process: 0.55117 | load: 0.00243 [I 2022-10-27 03:35:03] ...trap/engines/engine.py.230: loss: 0.14245 [I 2022-10-27 03:35:03] ...optimizers/trijoint.py.151: total_norm: 22.349329 [I 2022-10-27 03:35:09] ...trap/engines/engine.py.224: train: epoch 45 | batch 1670/2304 [I 2022-10-27 03:35:09] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:51 [I 2022-10-27 03:35:09] ...trap/engines/engine.py.229: process: 0.56439 | load: 0.00244 [I 2022-10-27 03:35:09] ...trap/engines/engine.py.230: loss: 0.17299 [I 2022-10-27 03:35:09] ...optimizers/trijoint.py.151: total_norm: 16.432623 [I 2022-10-27 03:35:14] ...trap/engines/engine.py.224: train: epoch 45 | batch 1680/2304 [I 2022-10-27 03:35:14] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:46 [I 2022-10-27 03:35:14] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00243 [I 2022-10-27 03:35:14] ...trap/engines/engine.py.230: loss: 0.16584 [I 2022-10-27 03:35:14] ...optimizers/trijoint.py.151: total_norm: 21.549545 [I 2022-10-27 03:35:20] ...trap/engines/engine.py.224: train: epoch 45 | batch 1690/2304 [I 2022-10-27 03:35:20] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:41 [I 2022-10-27 03:35:20] ...trap/engines/engine.py.229: process: 0.55719 | load: 0.00241 [I 2022-10-27 03:35:20] ...trap/engines/engine.py.230: loss: 0.19427 [I 2022-10-27 03:35:20] ...optimizers/trijoint.py.151: total_norm: 45.443432 [I 2022-10-27 03:35:25] ...trap/engines/engine.py.224: train: epoch 45 | batch 1700/2304 [I 2022-10-27 03:35:25] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:35 [I 2022-10-27 03:35:25] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00240 [I 2022-10-27 03:35:25] ...trap/engines/engine.py.230: loss: 0.14614 [I 2022-10-27 03:35:25] ...optimizers/trijoint.py.151: total_norm: 15.049033 [I 2022-10-27 03:35:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 1710/2304 [I 2022-10-27 03:35:31] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:27 [I 2022-10-27 03:35:31] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00243 [I 2022-10-27 03:35:31] ...trap/engines/engine.py.230: loss: 0.20516 [I 2022-10-27 03:35:31] ...optimizers/trijoint.py.151: total_norm: 14.545440 [I 2022-10-27 03:35:36] ...trap/engines/engine.py.224: train: epoch 45 | batch 1720/2304 [I 2022-10-27 03:35:36] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:23 [I 2022-10-27 03:35:36] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00245 [I 2022-10-27 03:35:36] ...trap/engines/engine.py.230: loss: 0.21400 [I 2022-10-27 03:35:36] ...optimizers/trijoint.py.151: total_norm: 17.879423 [I 2022-10-27 03:35:42] ...trap/engines/engine.py.224: train: epoch 45 | batch 1730/2304 [I 2022-10-27 03:35:42] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:18 [I 2022-10-27 03:35:42] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00244 [I 2022-10-27 03:35:42] ...trap/engines/engine.py.230: loss: 0.14267 [I 2022-10-27 03:35:42] ...optimizers/trijoint.py.151: total_norm: 23.900419 [I 2022-10-27 03:35:48] ...trap/engines/engine.py.224: train: epoch 45 | batch 1740/2304 [I 2022-10-27 03:35:48] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:05:18 [I 2022-10-27 03:35:48] ...trap/engines/engine.py.229: process: 0.55194 | load: 0.00243 [I 2022-10-27 03:35:48] ...trap/engines/engine.py.230: loss: 0.14035 [I 2022-10-27 03:35:48] ...optimizers/trijoint.py.151: total_norm: 11.264959 [I 2022-10-27 03:35:53] ...trap/engines/engine.py.224: train: epoch 45 | batch 1750/2304 [I 2022-10-27 03:35:53] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:05:15 [I 2022-10-27 03:35:53] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00247 [I 2022-10-27 03:35:53] ...trap/engines/engine.py.230: loss: 0.18046 [I 2022-10-27 03:35:53] ...optimizers/trijoint.py.151: total_norm: 46.923512 [I 2022-10-27 03:35:59] ...trap/engines/engine.py.224: train: epoch 45 | batch 1760/2304 [I 2022-10-27 03:35:59] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:05:01 [I 2022-10-27 03:35:59] ...trap/engines/engine.py.229: process: 0.54136 | load: 0.00243 [I 2022-10-27 03:35:59] ...trap/engines/engine.py.230: loss: 0.13889 [I 2022-10-27 03:35:59] ...optimizers/trijoint.py.151: total_norm: 13.038370 [I 2022-10-27 03:36:04] ...trap/engines/engine.py.224: train: epoch 45 | batch 1770/2304 [I 2022-10-27 03:36:04] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:56 [I 2022-10-27 03:36:04] ...trap/engines/engine.py.229: process: 0.56193 | load: 0.00244 [I 2022-10-27 03:36:04] ...trap/engines/engine.py.230: loss: 0.11683 [I 2022-10-27 03:36:04] ...optimizers/trijoint.py.151: total_norm: 10.343525 [I 2022-10-27 03:36:10] ...trap/engines/engine.py.224: train: epoch 45 | batch 1780/2304 [I 2022-10-27 03:36:10] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:51 [I 2022-10-27 03:36:10] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00244 [I 2022-10-27 03:36:10] ...trap/engines/engine.py.230: loss: 0.12083 [I 2022-10-27 03:36:10] ...optimizers/trijoint.py.151: total_norm: 11.987480 [I 2022-10-27 03:36:16] ...trap/engines/engine.py.224: train: epoch 45 | batch 1790/2304 [I 2022-10-27 03:36:16] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:46 [I 2022-10-27 03:36:16] ...trap/engines/engine.py.229: process: 0.54820 | load: 0.00267 [I 2022-10-27 03:36:16] ...trap/engines/engine.py.230: loss: 0.14999 [I 2022-10-27 03:36:16] ...optimizers/trijoint.py.151: total_norm: 21.343634 [I 2022-10-27 03:36:21] ...trap/engines/engine.py.224: train: epoch 45 | batch 1800/2304 [I 2022-10-27 03:36:21] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:38 [I 2022-10-27 03:36:21] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00245 [I 2022-10-27 03:36:21] ...trap/engines/engine.py.230: loss: 0.12953 [I 2022-10-27 03:36:21] ...optimizers/trijoint.py.151: total_norm: 11.923706 [I 2022-10-27 03:36:27] ...trap/engines/engine.py.224: train: epoch 45 | batch 1810/2304 [I 2022-10-27 03:36:27] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:35 [I 2022-10-27 03:36:27] ...trap/engines/engine.py.229: process: 0.56619 | load: 0.00245 [I 2022-10-27 03:36:27] ...trap/engines/engine.py.230: loss: 0.21973 [I 2022-10-27 03:36:27] ...optimizers/trijoint.py.151: total_norm: 20.979332 [I 2022-10-27 03:36:32] ...trap/engines/engine.py.224: train: epoch 45 | batch 1820/2304 [I 2022-10-27 03:36:32] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:29 [I 2022-10-27 03:36:32] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00244 [I 2022-10-27 03:36:32] ...trap/engines/engine.py.230: loss: 0.12323 [I 2022-10-27 03:36:32] ...optimizers/trijoint.py.151: total_norm: 19.240204 [I 2022-10-27 03:36:38] ...trap/engines/engine.py.224: train: epoch 45 | batch 1830/2304 [I 2022-10-27 03:36:38] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:27 [I 2022-10-27 03:36:38] ...trap/engines/engine.py.229: process: 0.56119 | load: 0.00249 [I 2022-10-27 03:36:38] ...trap/engines/engine.py.230: loss: 0.22005 [I 2022-10-27 03:36:38] ...optimizers/trijoint.py.151: total_norm: 16.702606 [I 2022-10-27 03:36:44] ...trap/engines/engine.py.224: train: epoch 45 | batch 1840/2304 [I 2022-10-27 03:36:44] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:17 [I 2022-10-27 03:36:44] ...trap/engines/engine.py.229: process: 0.56221 | load: 0.00242 [I 2022-10-27 03:36:44] ...trap/engines/engine.py.230: loss: 0.15104 [I 2022-10-27 03:36:44] ...optimizers/trijoint.py.151: total_norm: 16.403118 [I 2022-10-27 03:36:49] ...trap/engines/engine.py.224: train: epoch 45 | batch 1850/2304 [I 2022-10-27 03:36:49] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:04:12 [I 2022-10-27 03:36:49] ...trap/engines/engine.py.229: process: 0.54596 | load: 0.00244 [I 2022-10-27 03:36:49] ...trap/engines/engine.py.230: loss: 0.13450 [I 2022-10-27 03:36:49] ...optimizers/trijoint.py.151: total_norm: 13.677171 [I 2022-10-27 03:36:55] ...trap/engines/engine.py.224: train: epoch 45 | batch 1860/2304 [I 2022-10-27 03:36:55] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:04:07 [I 2022-10-27 03:36:55] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00246 [I 2022-10-27 03:36:55] ...trap/engines/engine.py.230: loss: 0.22059 [I 2022-10-27 03:36:55] ...optimizers/trijoint.py.151: total_norm: 16.196302 [I 2022-10-27 03:37:00] ...trap/engines/engine.py.224: train: epoch 45 | batch 1870/2304 [I 2022-10-27 03:37:00] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:04:02 [I 2022-10-27 03:37:00] ...trap/engines/engine.py.229: process: 0.57041 | load: 0.00243 [I 2022-10-27 03:37:00] ...trap/engines/engine.py.230: loss: 0.16055 [I 2022-10-27 03:37:00] ...optimizers/trijoint.py.151: total_norm: 11.346919 [I 2022-10-27 03:37:06] ...trap/engines/engine.py.224: train: epoch 45 | batch 1880/2304 [I 2022-10-27 03:37:06] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:56 [I 2022-10-27 03:37:06] ...trap/engines/engine.py.229: process: 0.55345 | load: 0.00243 [I 2022-10-27 03:37:06] ...trap/engines/engine.py.230: loss: 0.17251 [I 2022-10-27 03:37:06] ...optimizers/trijoint.py.151: total_norm: 14.669804 [I 2022-10-27 03:37:11] ...trap/engines/engine.py.224: train: epoch 45 | batch 1890/2304 [I 2022-10-27 03:37:11] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:49 [I 2022-10-27 03:37:11] ...trap/engines/engine.py.229: process: 0.54969 | load: 0.00244 [I 2022-10-27 03:37:11] ...trap/engines/engine.py.230: loss: 0.14665 [I 2022-10-27 03:37:11] ...optimizers/trijoint.py.151: total_norm: 17.294548 [I 2022-10-27 03:37:17] ...trap/engines/engine.py.224: train: epoch 45 | batch 1900/2304 [I 2022-10-27 03:37:17] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:46 [I 2022-10-27 03:37:17] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00245 [I 2022-10-27 03:37:17] ...trap/engines/engine.py.230: loss: 0.13063 [I 2022-10-27 03:37:17] ...optimizers/trijoint.py.151: total_norm: 13.768467 [I 2022-10-27 03:37:23] ...trap/engines/engine.py.224: train: epoch 45 | batch 1910/2304 [I 2022-10-27 03:37:23] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:38 [I 2022-10-27 03:37:23] ...trap/engines/engine.py.229: process: 0.55132 | load: 0.00246 [I 2022-10-27 03:37:23] ...trap/engines/engine.py.230: loss: 0.19384 [I 2022-10-27 03:37:23] ...optimizers/trijoint.py.151: total_norm: 20.200188 [I 2022-10-27 03:37:28] ...trap/engines/engine.py.224: train: epoch 45 | batch 1920/2304 [I 2022-10-27 03:37:28] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:32 [I 2022-10-27 03:37:28] ...trap/engines/engine.py.229: process: 0.54018 | load: 0.00242 [I 2022-10-27 03:37:28] ...trap/engines/engine.py.230: loss: 0.18454 [I 2022-10-27 03:37:28] ...optimizers/trijoint.py.151: total_norm: 36.947746 [I 2022-10-27 03:37:34] ...trap/engines/engine.py.224: train: epoch 45 | batch 1930/2304 [I 2022-10-27 03:37:34] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:27 [I 2022-10-27 03:37:34] ...trap/engines/engine.py.229: process: 0.54190 | load: 0.00241 [I 2022-10-27 03:37:34] ...trap/engines/engine.py.230: loss: 0.18828 [I 2022-10-27 03:37:34] ...optimizers/trijoint.py.151: total_norm: 36.518654 [I 2022-10-27 03:37:39] ...trap/engines/engine.py.224: train: epoch 45 | batch 1940/2304 [I 2022-10-27 03:37:39] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:22 [I 2022-10-27 03:37:39] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00241 [I 2022-10-27 03:37:39] ...trap/engines/engine.py.230: loss: 0.23037 [I 2022-10-27 03:37:39] ...optimizers/trijoint.py.151: total_norm: 21.330404 [I 2022-10-27 03:37:45] ...trap/engines/engine.py.224: train: epoch 45 | batch 1950/2304 [I 2022-10-27 03:37:45] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:15 [I 2022-10-27 03:37:45] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00242 [I 2022-10-27 03:37:45] ...trap/engines/engine.py.230: loss: 0.14947 [I 2022-10-27 03:37:45] ...optimizers/trijoint.py.151: total_norm: 20.205481 [I 2022-10-27 03:37:50] ...trap/engines/engine.py.224: train: epoch 45 | batch 1960/2304 [I 2022-10-27 03:37:50] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:10 [I 2022-10-27 03:37:50] ...trap/engines/engine.py.229: process: 0.55200 | load: 0.00244 [I 2022-10-27 03:37:50] ...trap/engines/engine.py.230: loss: 0.12744 [I 2022-10-27 03:37:50] ...optimizers/trijoint.py.151: total_norm: 15.676940 [I 2022-10-27 03:37:56] ...trap/engines/engine.py.224: train: epoch 45 | batch 1970/2304 [I 2022-10-27 03:37:56] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:03:06 [I 2022-10-27 03:37:56] ...trap/engines/engine.py.229: process: 0.56210 | load: 0.00242 [I 2022-10-27 03:37:56] ...trap/engines/engine.py.230: loss: 0.14196 [I 2022-10-27 03:37:56] ...optimizers/trijoint.py.151: total_norm: 34.060658 [I 2022-10-27 03:38:02] ...trap/engines/engine.py.224: train: epoch 45 | batch 1980/2304 [I 2022-10-27 03:38:02] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:58 [I 2022-10-27 03:38:02] ...trap/engines/engine.py.229: process: 0.53817 | load: 0.00251 [I 2022-10-27 03:38:02] ...trap/engines/engine.py.230: loss: 0.14035 [I 2022-10-27 03:38:02] ...optimizers/trijoint.py.151: total_norm: 17.457588 [I 2022-10-27 03:38:07] ...trap/engines/engine.py.224: train: epoch 45 | batch 1990/2304 [I 2022-10-27 03:38:07] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:55 [I 2022-10-27 03:38:07] ...trap/engines/engine.py.229: process: 0.57129 | load: 0.00250 [I 2022-10-27 03:38:07] ...trap/engines/engine.py.230: loss: 0.27331 [I 2022-10-27 03:38:07] ...optimizers/trijoint.py.151: total_norm: 20.587233 [I 2022-10-27 03:38:13] ...trap/engines/engine.py.224: train: epoch 45 | batch 2000/2304 [I 2022-10-27 03:38:13] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:49 [I 2022-10-27 03:38:13] ...trap/engines/engine.py.229: process: 0.56312 | load: 0.00245 [I 2022-10-27 03:38:13] ...trap/engines/engine.py.230: loss: 0.17717 [I 2022-10-27 03:38:13] ...optimizers/trijoint.py.151: total_norm: 21.680885 [I 2022-10-27 03:38:18] ...trap/engines/engine.py.224: train: epoch 45 | batch 2010/2304 [I 2022-10-27 03:38:18] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:48 [I 2022-10-27 03:38:18] ...trap/engines/engine.py.229: process: 0.57161 | load: 0.00242 [I 2022-10-27 03:38:18] ...trap/engines/engine.py.230: loss: 0.09252 [I 2022-10-27 03:38:18] ...optimizers/trijoint.py.151: total_norm: 14.786398 [I 2022-10-27 03:38:24] ...trap/engines/engine.py.224: train: epoch 45 | batch 2020/2304 [I 2022-10-27 03:38:24] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:40 [I 2022-10-27 03:38:24] ...trap/engines/engine.py.229: process: 0.56738 | load: 0.00244 [I 2022-10-27 03:38:24] ...trap/engines/engine.py.230: loss: 0.14404 [I 2022-10-27 03:38:24] ...optimizers/trijoint.py.151: total_norm: 16.573235 [I 2022-10-27 03:38:30] ...trap/engines/engine.py.224: train: epoch 45 | batch 2030/2304 [I 2022-10-27 03:38:30] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:32 [I 2022-10-27 03:38:30] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00242 [I 2022-10-27 03:38:30] ...trap/engines/engine.py.230: loss: 0.12096 [I 2022-10-27 03:38:30] ...optimizers/trijoint.py.151: total_norm: 9.047210 [I 2022-10-27 03:38:35] ...trap/engines/engine.py.224: train: epoch 45 | batch 2040/2304 [I 2022-10-27 03:38:35] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:27 [I 2022-10-27 03:38:35] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00242 [I 2022-10-27 03:38:35] ...trap/engines/engine.py.230: loss: 0.13037 [I 2022-10-27 03:38:35] ...optimizers/trijoint.py.151: total_norm: 7.120000 [I 2022-10-27 03:38:41] ...trap/engines/engine.py.224: train: epoch 45 | batch 2050/2304 [I 2022-10-27 03:38:41] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:22 [I 2022-10-27 03:38:41] ...trap/engines/engine.py.229: process: 0.56623 | load: 0.00243 [I 2022-10-27 03:38:41] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-27 03:38:41] ...optimizers/trijoint.py.151: total_norm: 13.236756 [I 2022-10-27 03:38:47] ...trap/engines/engine.py.224: train: epoch 45 | batch 2060/2304 [I 2022-10-27 03:38:47] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:16 [I 2022-10-27 03:38:47] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00240 [I 2022-10-27 03:38:47] ...trap/engines/engine.py.230: loss: 0.17492 [I 2022-10-27 03:38:47] ...optimizers/trijoint.py.151: total_norm: 21.151533 [I 2022-10-27 03:38:52] ...trap/engines/engine.py.224: train: epoch 45 | batch 2070/2304 [I 2022-10-27 03:38:52] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:09 [I 2022-10-27 03:38:52] ...trap/engines/engine.py.229: process: 0.54985 | load: 0.00259 [I 2022-10-27 03:38:52] ...trap/engines/engine.py.230: loss: 0.15421 [I 2022-10-27 03:38:52] ...optimizers/trijoint.py.151: total_norm: 15.220963 [I 2022-10-27 03:38:58] ...trap/engines/engine.py.224: train: epoch 45 | batch 2080/2304 [I 2022-10-27 03:38:58] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:02:04 [I 2022-10-27 03:38:58] ...trap/engines/engine.py.229: process: 0.54313 | load: 0.00244 [I 2022-10-27 03:38:58] ...trap/engines/engine.py.230: loss: 0.18294 [I 2022-10-27 03:38:58] ...optimizers/trijoint.py.151: total_norm: 17.640589 [I 2022-10-27 03:39:03] ...trap/engines/engine.py.224: train: epoch 45 | batch 2090/2304 [I 2022-10-27 03:39:03] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:58 [I 2022-10-27 03:39:03] ...trap/engines/engine.py.229: process: 0.55106 | load: 0.00243 [I 2022-10-27 03:39:03] ...trap/engines/engine.py.230: loss: 0.14129 [I 2022-10-27 03:39:03] ...optimizers/trijoint.py.151: total_norm: 10.477550 [I 2022-10-27 03:39:09] ...trap/engines/engine.py.224: train: epoch 45 | batch 2100/2304 [I 2022-10-27 03:39:09] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:54 [I 2022-10-27 03:39:09] ...trap/engines/engine.py.229: process: 0.55027 | load: 0.00244 [I 2022-10-27 03:39:09] ...trap/engines/engine.py.230: loss: 0.18364 [I 2022-10-27 03:39:09] ...optimizers/trijoint.py.151: total_norm: 37.960434 [I 2022-10-27 03:39:14] ...trap/engines/engine.py.224: train: epoch 45 | batch 2110/2304 [I 2022-10-27 03:39:14] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:48 [I 2022-10-27 03:39:14] ...trap/engines/engine.py.229: process: 0.56553 | load: 0.00251 [I 2022-10-27 03:39:14] ...trap/engines/engine.py.230: loss: 0.11880 [I 2022-10-27 03:39:14] ...optimizers/trijoint.py.151: total_norm: 17.489202 [I 2022-10-27 03:39:20] ...trap/engines/engine.py.224: train: epoch 45 | batch 2120/2304 [I 2022-10-27 03:39:20] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:45 [I 2022-10-27 03:39:20] ...trap/engines/engine.py.229: process: 0.57283 | load: 0.00247 [I 2022-10-27 03:39:20] ...trap/engines/engine.py.230: loss: 0.17939 [I 2022-10-27 03:39:20] ...optimizers/trijoint.py.151: total_norm: 13.393689 [I 2022-10-27 03:39:26] ...trap/engines/engine.py.224: train: epoch 45 | batch 2130/2304 [I 2022-10-27 03:39:26] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:38 [I 2022-10-27 03:39:26] ...trap/engines/engine.py.229: process: 0.56839 | load: 0.00242 [I 2022-10-27 03:39:26] ...trap/engines/engine.py.230: loss: 0.14675 [I 2022-10-27 03:39:26] ...optimizers/trijoint.py.151: total_norm: 13.774637 [I 2022-10-27 03:39:31] ...trap/engines/engine.py.224: train: epoch 45 | batch 2140/2304 [I 2022-10-27 03:39:31] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:31 [I 2022-10-27 03:39:31] ...trap/engines/engine.py.229: process: 0.55764 | load: 0.00242 [I 2022-10-27 03:39:31] ...trap/engines/engine.py.230: loss: 0.18071 [I 2022-10-27 03:39:31] ...optimizers/trijoint.py.151: total_norm: 35.046803 [I 2022-10-27 03:39:37] ...trap/engines/engine.py.224: train: epoch 45 | batch 2150/2304 [I 2022-10-27 03:39:37] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:26 [I 2022-10-27 03:39:37] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00242 [I 2022-10-27 03:39:37] ...trap/engines/engine.py.230: loss: 0.11975 [I 2022-10-27 03:39:37] ...optimizers/trijoint.py.151: total_norm: 10.655304 [I 2022-10-27 03:39:43] ...trap/engines/engine.py.224: train: epoch 45 | batch 2160/2304 [I 2022-10-27 03:39:43] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:20 [I 2022-10-27 03:39:43] ...trap/engines/engine.py.229: process: 0.56613 | load: 0.00243 [I 2022-10-27 03:39:43] ...trap/engines/engine.py.230: loss: 0.17591 [I 2022-10-27 03:39:43] ...optimizers/trijoint.py.151: total_norm: 14.426183 [I 2022-10-27 03:39:48] ...trap/engines/engine.py.224: train: epoch 45 | batch 2170/2304 [I 2022-10-27 03:39:48] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:14 [I 2022-10-27 03:39:48] ...trap/engines/engine.py.229: process: 0.54052 | load: 0.00245 [I 2022-10-27 03:39:48] ...trap/engines/engine.py.230: loss: 0.13009 [I 2022-10-27 03:39:48] ...optimizers/trijoint.py.151: total_norm: 7.800979 [I 2022-10-27 03:39:54] ...trap/engines/engine.py.224: train: epoch 45 | batch 2180/2304 [I 2022-10-27 03:39:54] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:01:09 [I 2022-10-27 03:39:54] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00245 [I 2022-10-27 03:39:54] ...trap/engines/engine.py.230: loss: 0.16446 [I 2022-10-27 03:39:54] ...optimizers/trijoint.py.151: total_norm: 20.332823 [I 2022-10-27 03:39:59] ...trap/engines/engine.py.224: train: epoch 45 | batch 2190/2304 [I 2022-10-27 03:39:59] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:03 [I 2022-10-27 03:39:59] ...trap/engines/engine.py.229: process: 0.54948 | load: 0.00277 [I 2022-10-27 03:39:59] ...trap/engines/engine.py.230: loss: 0.17116 [I 2022-10-27 03:39:59] ...optimizers/trijoint.py.151: total_norm: 21.812435 [I 2022-10-27 03:40:05] ...trap/engines/engine.py.224: train: epoch 45 | batch 2200/2304 [I 2022-10-27 03:40:05] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:57 [I 2022-10-27 03:40:05] ...trap/engines/engine.py.229: process: 0.54602 | load: 0.00241 [I 2022-10-27 03:40:05] ...trap/engines/engine.py.230: loss: 0.12668 [I 2022-10-27 03:40:05] ...optimizers/trijoint.py.151: total_norm: 11.423258 [I 2022-10-27 03:40:11] ...trap/engines/engine.py.224: train: epoch 45 | batch 2210/2304 [I 2022-10-27 03:40:11] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:52 [I 2022-10-27 03:40:11] ...trap/engines/engine.py.229: process: 0.56878 | load: 0.00242 [I 2022-10-27 03:40:11] ...trap/engines/engine.py.230: loss: 0.21837 [I 2022-10-27 03:40:11] ...optimizers/trijoint.py.151: total_norm: 27.333622 [I 2022-10-27 03:40:16] ...trap/engines/engine.py.224: train: epoch 45 | batch 2220/2304 [I 2022-10-27 03:40:16] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:46 [I 2022-10-27 03:40:16] ...trap/engines/engine.py.229: process: 0.56292 | load: 0.00244 [I 2022-10-27 03:40:16] ...trap/engines/engine.py.230: loss: 0.16107 [I 2022-10-27 03:40:16] ...optimizers/trijoint.py.151: total_norm: 16.276251 [I 2022-10-27 03:40:22] ...trap/engines/engine.py.224: train: epoch 45 | batch 2230/2304 [I 2022-10-27 03:40:22] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:40 [I 2022-10-27 03:40:22] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00239 [I 2022-10-27 03:40:22] ...trap/engines/engine.py.230: loss: 0.18017 [I 2022-10-27 03:40:22] ...optimizers/trijoint.py.151: total_norm: 52.171917 [I 2022-10-27 03:40:27] ...trap/engines/engine.py.224: train: epoch 45 | batch 2240/2304 [I 2022-10-27 03:40:27] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:35 [I 2022-10-27 03:40:27] ...trap/engines/engine.py.229: process: 0.55072 | load: 0.00254 [I 2022-10-27 03:40:27] ...trap/engines/engine.py.230: loss: 0.21656 [I 2022-10-27 03:40:27] ...optimizers/trijoint.py.151: total_norm: 14.949409 [I 2022-10-27 03:40:33] ...trap/engines/engine.py.224: train: epoch 45 | batch 2250/2304 [I 2022-10-27 03:40:33] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:30 [I 2022-10-27 03:40:33] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00248 [I 2022-10-27 03:40:33] ...trap/engines/engine.py.230: loss: 0.18150 [I 2022-10-27 03:40:33] ...optimizers/trijoint.py.151: total_norm: 17.237185 [I 2022-10-27 03:40:38] ...trap/engines/engine.py.224: train: epoch 45 | batch 2260/2304 [I 2022-10-27 03:40:38] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:24 [I 2022-10-27 03:40:38] ...trap/engines/engine.py.229: process: 0.56689 | load: 0.00241 [I 2022-10-27 03:40:38] ...trap/engines/engine.py.230: loss: 0.13207 [I 2022-10-27 03:40:38] ...optimizers/trijoint.py.151: total_norm: 11.723426 [I 2022-10-27 03:40:44] ...trap/engines/engine.py.224: train: epoch 45 | batch 2270/2304 [I 2022-10-27 03:40:44] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:18 [I 2022-10-27 03:40:44] ...trap/engines/engine.py.229: process: 0.56216 | load: 0.00245 [I 2022-10-27 03:40:44] ...trap/engines/engine.py.230: loss: 0.17400 [I 2022-10-27 03:40:44] ...optimizers/trijoint.py.151: total_norm: 17.247005 [I 2022-10-27 03:40:50] ...trap/engines/engine.py.224: train: epoch 45 | batch 2280/2304 [I 2022-10-27 03:40:50] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:13 [I 2022-10-27 03:40:50] ...trap/engines/engine.py.229: process: 0.54459 | load: 0.00245 [I 2022-10-27 03:40:50] ...trap/engines/engine.py.230: loss: 0.16038 [I 2022-10-27 03:40:50] ...optimizers/trijoint.py.151: total_norm: 16.537464 [I 2022-10-27 03:40:55] ...trap/engines/engine.py.224: train: epoch 45 | batch 2290/2304 [I 2022-10-27 03:40:55] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:07 [I 2022-10-27 03:40:55] ...trap/engines/engine.py.229: process: 0.56701 | load: 0.00244 [I 2022-10-27 03:40:55] ...trap/engines/engine.py.230: loss: 0.18255 [I 2022-10-27 03:40:55] ...optimizers/trijoint.py.151: total_norm: 13.405624 [I 2022-10-27 03:41:01] ...trap/engines/engine.py.224: train: epoch 45 | batch 2300/2304 [I 2022-10-27 03:41:01] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:02 [I 2022-10-27 03:41:01] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00015 [I 2022-10-27 03:41:01] ...trap/engines/engine.py.230: loss: 0.23654 [I 2022-10-27 03:41:01] ...optimizers/trijoint.py.151: total_norm: 21.678385 [I 2022-10-27 03:41:03] ...trap/engines/engine.py.224: train: epoch 45 | batch 2304/2304 [I 2022-10-27 03:41:03] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:00 [I 2022-10-27 03:41:03] ...trap/engines/engine.py.229: process: 0.54551 | load: 0.00015 [I 2022-10-27 03:41:03] ...trap/engines/engine.py.230: loss: 0.13725 [I 2022-10-27 03:41:03] ...optimizers/trijoint.py.151: total_norm: 15.374808 [S 2022-10-27 03:41:04] ...trap/engines/engine.py.129: train_epoch.epoch: 45 [S 2022-10-27 03:41:04] ...trap/engines/engine.py.129: train_epoch.loss: 0.1675761140963778 [I 2022-10-27 03:41:04] ...s/criterions/triplet.py.84: margin = 0.2800000000000002 [I 2022-10-27 03:41:05] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 45 [I 2022-10-27 03:41:05] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 03:41:07] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 03:41:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 03:41:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 03:41:08] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 03:41:09] ...trap/engines/engine.py.321: eval: epoch 45 | batch 0/510 [I 2022-10-27 03:41:09] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:32:18 [I 2022-10-27 03:41:09] ...trap/engines/engine.py.326: process: 3.80128 | load: 3.40685 [I 2022-10-27 03:41:12] ...trap/engines/engine.py.321: eval: epoch 45 | batch 10/510 [I 2022-10-27 03:41:12] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:05:54 [I 2022-10-27 03:41:12] ...trap/engines/engine.py.326: process: 0.33335 | load: 0.00017 [I 2022-10-27 03:41:15] ...trap/engines/engine.py.321: eval: epoch 45 | batch 20/510 [I 2022-10-27 03:41:15] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:01 [I 2022-10-27 03:41:15] ...trap/engines/engine.py.326: process: 0.32969 | load: 0.00020 [I 2022-10-27 03:41:19] ...trap/engines/engine.py.321: eval: epoch 45 | batch 30/510 [I 2022-10-27 03:41:19] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:43 [I 2022-10-27 03:41:19] ...trap/engines/engine.py.326: process: 0.32207 | load: 0.00019 [I 2022-10-27 03:41:22] ...trap/engines/engine.py.321: eval: epoch 45 | batch 40/510 [I 2022-10-27 03:41:22] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:39 [I 2022-10-27 03:41:22] ...trap/engines/engine.py.326: process: 0.33602 | load: 0.00022 [I 2022-10-27 03:41:25] ...trap/engines/engine.py.321: eval: epoch 45 | batch 50/510 [I 2022-10-27 03:41:25] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:33 [I 2022-10-27 03:41:25] ...trap/engines/engine.py.326: process: 0.33098 | load: 0.00045 [I 2022-10-27 03:41:29] ...trap/engines/engine.py.321: eval: epoch 45 | batch 60/510 [I 2022-10-27 03:41:29] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:30 [I 2022-10-27 03:41:29] ...trap/engines/engine.py.326: process: 0.33853 | load: 0.00023 [I 2022-10-27 03:41:32] ...trap/engines/engine.py.321: eval: epoch 45 | batch 70/510 [I 2022-10-27 03:41:32] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:28 [I 2022-10-27 03:41:32] ...trap/engines/engine.py.326: process: 0.33123 | load: 0.00027 [I 2022-10-27 03:41:36] ...trap/engines/engine.py.321: eval: epoch 45 | batch 80/510 [I 2022-10-27 03:41:36] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:22 [I 2022-10-27 03:41:36] ...trap/engines/engine.py.326: process: 0.32982 | load: 0.00023 [I 2022-10-27 03:41:39] ...trap/engines/engine.py.321: eval: epoch 45 | batch 90/510 [I 2022-10-27 03:41:39] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:18 [I 2022-10-27 03:41:39] ...trap/engines/engine.py.326: process: 0.32928 | load: 0.00045 [I 2022-10-27 03:41:42] ...trap/engines/engine.py.321: eval: epoch 45 | batch 100/510 [I 2022-10-27 03:41:42] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:14 [I 2022-10-27 03:41:42] ...trap/engines/engine.py.326: process: 0.29600 | load: 0.00020 [I 2022-10-27 03:41:45] ...trap/engines/engine.py.321: eval: epoch 45 | batch 110/510 [I 2022-10-27 03:41:45] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:58 [I 2022-10-27 03:41:45] ...trap/engines/engine.py.326: process: 0.29350 | load: 0.00015 [I 2022-10-27 03:41:48] ...trap/engines/engine.py.321: eval: epoch 45 | batch 120/510 [I 2022-10-27 03:41:48] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:53 [I 2022-10-27 03:41:48] ...trap/engines/engine.py.326: process: 0.28820 | load: 0.00027 [I 2022-10-27 03:41:51] ...trap/engines/engine.py.321: eval: epoch 45 | batch 130/510 [I 2022-10-27 03:41:51] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:50 [I 2022-10-27 03:41:51] ...trap/engines/engine.py.326: process: 0.29132 | load: 0.00020 [I 2022-10-27 03:41:54] ...trap/engines/engine.py.321: eval: epoch 45 | batch 140/510 [I 2022-10-27 03:41:54] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-27 03:41:54] ...trap/engines/engine.py.326: process: 0.29304 | load: 0.00025 [I 2022-10-27 03:41:57] ...trap/engines/engine.py.321: eval: epoch 45 | batch 150/510 [I 2022-10-27 03:41:57] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:45 [I 2022-10-27 03:41:57] ...trap/engines/engine.py.326: process: 0.30053 | load: 0.00022 [I 2022-10-27 03:42:00] ...trap/engines/engine.py.321: eval: epoch 45 | batch 160/510 [I 2022-10-27 03:42:00] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-27 03:42:00] ...trap/engines/engine.py.326: process: 0.28901 | load: 0.00020 [I 2022-10-27 03:42:02] ...trap/engines/engine.py.321: eval: epoch 45 | batch 170/510 [I 2022-10-27 03:42:02] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:39 [I 2022-10-27 03:42:02] ...trap/engines/engine.py.326: process: 0.28938 | load: 0.00034 [I 2022-10-27 03:42:05] ...trap/engines/engine.py.321: eval: epoch 45 | batch 180/510 [I 2022-10-27 03:42:05] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:36 [I 2022-10-27 03:42:05] ...trap/engines/engine.py.326: process: 0.28822 | load: 0.00024 [I 2022-10-27 03:42:08] ...trap/engines/engine.py.321: eval: epoch 45 | batch 190/510 [I 2022-10-27 03:42:08] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:33 [I 2022-10-27 03:42:08] ...trap/engines/engine.py.326: process: 0.29400 | load: 0.00030 [I 2022-10-27 03:42:11] ...trap/engines/engine.py.321: eval: epoch 45 | batch 200/510 [I 2022-10-27 03:42:11] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 03:42:11] ...trap/engines/engine.py.326: process: 0.28910 | load: 0.00024 [I 2022-10-27 03:42:14] ...trap/engines/engine.py.321: eval: epoch 45 | batch 210/510 [I 2022-10-27 03:42:14] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:27 [I 2022-10-27 03:42:14] ...trap/engines/engine.py.326: process: 0.29402 | load: 0.00014 [I 2022-10-27 03:42:17] ...trap/engines/engine.py.321: eval: epoch 45 | batch 220/510 [I 2022-10-27 03:42:17] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 03:42:17] ...trap/engines/engine.py.326: process: 0.28398 | load: 0.00021 [I 2022-10-27 03:42:20] ...trap/engines/engine.py.321: eval: epoch 45 | batch 230/510 [I 2022-10-27 03:42:20] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 03:42:20] ...trap/engines/engine.py.326: process: 0.28681 | load: 0.00025 [I 2022-10-27 03:42:23] ...trap/engines/engine.py.321: eval: epoch 45 | batch 240/510 [I 2022-10-27 03:42:23] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:18 [I 2022-10-27 03:42:23] ...trap/engines/engine.py.326: process: 0.29180 | load: 0.00025 [I 2022-10-27 03:42:26] ...trap/engines/engine.py.321: eval: epoch 45 | batch 250/510 [I 2022-10-27 03:42:26] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-27 03:42:26] ...trap/engines/engine.py.326: process: 0.29409 | load: 0.00018 [I 2022-10-27 03:42:29] ...trap/engines/engine.py.321: eval: epoch 45 | batch 260/510 [I 2022-10-27 03:42:29] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-27 03:42:29] ...trap/engines/engine.py.326: process: 0.29011 | load: 0.00022 [I 2022-10-27 03:42:31] ...trap/engines/engine.py.321: eval: epoch 45 | batch 270/510 [I 2022-10-27 03:42:31] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-27 03:42:31] ...trap/engines/engine.py.326: process: 0.28241 | load: 0.00018 [I 2022-10-27 03:42:34] ...trap/engines/engine.py.321: eval: epoch 45 | batch 280/510 [I 2022-10-27 03:42:34] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 03:42:34] ...trap/engines/engine.py.326: process: 0.28886 | load: 0.00015 [I 2022-10-27 03:42:37] ...trap/engines/engine.py.321: eval: epoch 45 | batch 290/510 [I 2022-10-27 03:42:37] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:02 [I 2022-10-27 03:42:37] ...trap/engines/engine.py.326: process: 0.28337 | load: 0.00017 [I 2022-10-27 03:42:40] ...trap/engines/engine.py.321: eval: epoch 45 | batch 300/510 [I 2022-10-27 03:42:40] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 03:42:40] ...trap/engines/engine.py.326: process: 0.28701 | load: 0.00019 [I 2022-10-27 03:42:43] ...trap/engines/engine.py.321: eval: epoch 45 | batch 310/510 [I 2022-10-27 03:42:43] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 03:42:43] ...trap/engines/engine.py.326: process: 0.28722 | load: 0.00017 [I 2022-10-27 03:42:46] ...trap/engines/engine.py.321: eval: epoch 45 | batch 320/510 [I 2022-10-27 03:42:46] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 03:42:46] ...trap/engines/engine.py.326: process: 0.28355 | load: 0.00017 [I 2022-10-27 03:42:49] ...trap/engines/engine.py.321: eval: epoch 45 | batch 330/510 [I 2022-10-27 03:42:49] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 03:42:49] ...trap/engines/engine.py.326: process: 0.28591 | load: 0.00017 [I 2022-10-27 03:42:52] ...trap/engines/engine.py.321: eval: epoch 45 | batch 340/510 [I 2022-10-27 03:42:52] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-27 03:42:52] ...trap/engines/engine.py.326: process: 0.29102 | load: 0.00019 [I 2022-10-27 03:42:54] ...trap/engines/engine.py.321: eval: epoch 45 | batch 350/510 [I 2022-10-27 03:42:54] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-27 03:42:54] ...trap/engines/engine.py.326: process: 0.29014 | load: 0.00017 [I 2022-10-27 03:42:57] ...trap/engines/engine.py.321: eval: epoch 45 | batch 360/510 [I 2022-10-27 03:42:57] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 03:42:57] ...trap/engines/engine.py.326: process: 0.28848 | load: 0.00016 [I 2022-10-27 03:43:00] ...trap/engines/engine.py.321: eval: epoch 45 | batch 370/510 [I 2022-10-27 03:43:00] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 03:43:00] ...trap/engines/engine.py.326: process: 0.28337 | load: 0.00018 [I 2022-10-27 03:43:03] ...trap/engines/engine.py.321: eval: epoch 45 | batch 380/510 [I 2022-10-27 03:43:03] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 03:43:03] ...trap/engines/engine.py.326: process: 0.29168 | load: 0.00017 [I 2022-10-27 03:43:06] ...trap/engines/engine.py.321: eval: epoch 45 | batch 390/510 [I 2022-10-27 03:43:06] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 03:43:06] ...trap/engines/engine.py.326: process: 0.28151 | load: 0.00017 [I 2022-10-27 03:43:09] ...trap/engines/engine.py.321: eval: epoch 45 | batch 400/510 [I 2022-10-27 03:43:09] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-27 03:43:09] ...trap/engines/engine.py.326: process: 0.28738 | load: 0.00014 [I 2022-10-27 03:43:12] ...trap/engines/engine.py.321: eval: epoch 45 | batch 410/510 [I 2022-10-27 03:43:12] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-27 03:43:12] ...trap/engines/engine.py.326: process: 0.28523 | load: 0.00016 [I 2022-10-27 03:43:15] ...trap/engines/engine.py.321: eval: epoch 45 | batch 420/510 [I 2022-10-27 03:43:15] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-27 03:43:15] ...trap/engines/engine.py.326: process: 0.29303 | load: 0.00017 [I 2022-10-27 03:43:17] ...trap/engines/engine.py.321: eval: epoch 45 | batch 430/510 [I 2022-10-27 03:43:17] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-27 03:43:17] ...trap/engines/engine.py.326: process: 0.28893 | load: 0.00016 [I 2022-10-27 03:43:20] ...trap/engines/engine.py.321: eval: epoch 45 | batch 440/510 [I 2022-10-27 03:43:20] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 03:43:20] ...trap/engines/engine.py.326: process: 0.28983 | load: 0.00017 [I 2022-10-27 03:43:23] ...trap/engines/engine.py.321: eval: epoch 45 | batch 450/510 [I 2022-10-27 03:43:23] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 03:43:23] ...trap/engines/engine.py.326: process: 0.28634 | load: 0.00018 [I 2022-10-27 03:43:26] ...trap/engines/engine.py.321: eval: epoch 45 | batch 460/510 [I 2022-10-27 03:43:26] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 03:43:26] ...trap/engines/engine.py.326: process: 0.29001 | load: 0.00024 [I 2022-10-27 03:43:29] ...trap/engines/engine.py.321: eval: epoch 45 | batch 470/510 [I 2022-10-27 03:43:29] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 03:43:29] ...trap/engines/engine.py.326: process: 0.28664 | load: 0.00019 [I 2022-10-27 03:43:32] ...trap/engines/engine.py.321: eval: epoch 45 | batch 480/510 [I 2022-10-27 03:43:32] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 03:43:32] ...trap/engines/engine.py.326: process: 0.28945 | load: 0.00017 [I 2022-10-27 03:43:35] ...trap/engines/engine.py.321: eval: epoch 45 | batch 490/510 [I 2022-10-27 03:43:35] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 03:43:35] ...trap/engines/engine.py.326: process: 0.28889 | load: 0.00015 [I 2022-10-27 03:43:38] ...trap/engines/engine.py.321: eval: epoch 45 | batch 500/510 [I 2022-10-27 03:43:38] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 03:43:38] ...trap/engines/engine.py.326: process: 0.28603 | load: 0.00014 [I 2022-10-27 03:43:40] ...trap/engines/engine.py.321: eval: epoch 45 | batch 510/510 [I 2022-10-27 03:43:40] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 03:43:40] ...trap/engines/engine.py.326: process: 0.28792 | load: 0.00013 [S 2022-10-27 03:43:41] ...trap/engines/engine.py.132: eval_epoch.epoch: 45 [I 2022-10-27 03:43:41] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.694700 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.901600 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936800 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.680600 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.894100 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932500 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011091 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009035 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008022 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017614 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009575 [S 2022-10-27 03:43:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009594 [I 2022-10-27 03:43:46] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 03:43:46] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 03:43:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 03:43:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 03:43:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 03:43:48] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 03:43:51] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 03:43:56] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 03:43:56] ...trap/engines/engine.py.163: Training model on trainset for epoch 46 [I 2022-10-27 03:43:56] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 03:43:59] ...trap/engines/engine.py.224: train: epoch 46 | batch 0/2304 [I 2022-10-27 03:43:59] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:27 [I 2022-10-27 03:43:59] ...trap/engines/engine.py.229: process: 2.66827 | load: 2.03830 [I 2022-10-27 03:43:59] ...trap/engines/engine.py.230: loss: 0.16084 [I 2022-10-27 03:43:59] ...optimizers/trijoint.py.151: total_norm: 21.725506 [I 2022-10-27 03:44:05] ...trap/engines/engine.py.224: train: epoch 46 | batch 10/2304 [I 2022-10-27 03:44:05] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:00 [I 2022-10-27 03:44:05] ...trap/engines/engine.py.229: process: 0.55776 | load: 0.00428 [I 2022-10-27 03:44:05] ...trap/engines/engine.py.230: loss: 0.22799 [I 2022-10-27 03:44:05] ...optimizers/trijoint.py.151: total_norm: 12.657980 [I 2022-10-27 03:44:10] ...trap/engines/engine.py.224: train: epoch 46 | batch 20/2304 [I 2022-10-27 03:44:10] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:21:59 [I 2022-10-27 03:44:10] ...trap/engines/engine.py.229: process: 0.53920 | load: 0.00374 [I 2022-10-27 03:44:10] ...trap/engines/engine.py.230: loss: 0.16459 [I 2022-10-27 03:44:10] ...optimizers/trijoint.py.151: total_norm: 17.131001 [I 2022-10-27 03:44:16] ...trap/engines/engine.py.224: train: epoch 46 | batch 30/2304 [I 2022-10-27 03:44:16] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-27 03:44:16] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00317 [I 2022-10-27 03:44:16] ...trap/engines/engine.py.230: loss: 0.23051 [I 2022-10-27 03:44:16] ...optimizers/trijoint.py.151: total_norm: 18.859282 [I 2022-10-27 03:44:21] ...trap/engines/engine.py.224: train: epoch 46 | batch 40/2304 [I 2022-10-27 03:44:21] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:59 [I 2022-10-27 03:44:21] ...trap/engines/engine.py.229: process: 0.55258 | load: 0.00293 [I 2022-10-27 03:44:21] ...trap/engines/engine.py.230: loss: 0.13724 [I 2022-10-27 03:44:21] ...optimizers/trijoint.py.151: total_norm: 13.345886 [I 2022-10-27 03:44:27] ...trap/engines/engine.py.224: train: epoch 46 | batch 50/2304 [I 2022-10-27 03:44:27] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:47 [I 2022-10-27 03:44:27] ...trap/engines/engine.py.229: process: 0.54932 | load: 0.00308 [I 2022-10-27 03:44:27] ...trap/engines/engine.py.230: loss: 0.14291 [I 2022-10-27 03:44:27] ...optimizers/trijoint.py.151: total_norm: 15.199904 [I 2022-10-27 03:44:32] ...trap/engines/engine.py.224: train: epoch 46 | batch 60/2304 [I 2022-10-27 03:44:32] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:34 [I 2022-10-27 03:44:32] ...trap/engines/engine.py.229: process: 0.54100 | load: 0.00294 [I 2022-10-27 03:44:32] ...trap/engines/engine.py.230: loss: 0.13410 [I 2022-10-27 03:44:32] ...optimizers/trijoint.py.151: total_norm: 10.922031 [I 2022-10-27 03:44:38] ...trap/engines/engine.py.224: train: epoch 46 | batch 70/2304 [I 2022-10-27 03:44:38] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:42 [I 2022-10-27 03:44:38] ...trap/engines/engine.py.229: process: 0.56343 | load: 0.00279 [I 2022-10-27 03:44:38] ...trap/engines/engine.py.230: loss: 0.24263 [I 2022-10-27 03:44:38] ...optimizers/trijoint.py.151: total_norm: 15.853878 [I 2022-10-27 03:44:44] ...trap/engines/engine.py.224: train: epoch 46 | batch 80/2304 [I 2022-10-27 03:44:44] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:42 [I 2022-10-27 03:44:44] ...trap/engines/engine.py.229: process: 0.56410 | load: 0.00289 [I 2022-10-27 03:44:44] ...trap/engines/engine.py.230: loss: 0.14952 [I 2022-10-27 03:44:44] ...optimizers/trijoint.py.151: total_norm: 9.755498 [I 2022-10-27 03:44:49] ...trap/engines/engine.py.224: train: epoch 46 | batch 90/2304 [I 2022-10-27 03:44:49] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:33 [I 2022-10-27 03:44:49] ...trap/engines/engine.py.229: process: 0.55462 | load: 0.00264 [I 2022-10-27 03:44:49] ...trap/engines/engine.py.230: loss: 0.12801 [I 2022-10-27 03:44:49] ...optimizers/trijoint.py.151: total_norm: 10.787011 [I 2022-10-27 03:44:55] ...trap/engines/engine.py.224: train: epoch 46 | batch 100/2304 [I 2022-10-27 03:44:55] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:24 [I 2022-10-27 03:44:55] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00272 [I 2022-10-27 03:44:55] ...trap/engines/engine.py.230: loss: 0.13849 [I 2022-10-27 03:44:55] ...optimizers/trijoint.py.151: total_norm: 18.171757 [I 2022-10-27 03:45:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 110/2304 [I 2022-10-27 03:45:00] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:20:26 [I 2022-10-27 03:45:00] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00364 [I 2022-10-27 03:45:00] ...trap/engines/engine.py.230: loss: 0.16944 [I 2022-10-27 03:45:00] ...optimizers/trijoint.py.151: total_norm: 16.116804 [I 2022-10-27 03:45:06] ...trap/engines/engine.py.224: train: epoch 46 | batch 120/2304 [I 2022-10-27 03:45:06] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:26 [I 2022-10-27 03:45:06] ...trap/engines/engine.py.229: process: 0.55844 | load: 0.00265 [I 2022-10-27 03:45:06] ...trap/engines/engine.py.230: loss: 0.19562 [I 2022-10-27 03:45:06] ...optimizers/trijoint.py.151: total_norm: 16.674822 [I 2022-10-27 03:45:12] ...trap/engines/engine.py.224: train: epoch 46 | batch 130/2304 [I 2022-10-27 03:45:12] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:19 [I 2022-10-27 03:45:12] ...trap/engines/engine.py.229: process: 0.56319 | load: 0.00247 [I 2022-10-27 03:45:12] ...trap/engines/engine.py.230: loss: 0.10232 [I 2022-10-27 03:45:12] ...optimizers/trijoint.py.151: total_norm: 11.653824 [I 2022-10-27 03:45:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 140/2304 [I 2022-10-27 03:45:17] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:50 [I 2022-10-27 03:45:17] ...trap/engines/engine.py.229: process: 0.53477 | load: 0.00286 [I 2022-10-27 03:45:17] ...trap/engines/engine.py.230: loss: 0.17251 [I 2022-10-27 03:45:17] ...optimizers/trijoint.py.151: total_norm: 40.649403 [I 2022-10-27 03:45:23] ...trap/engines/engine.py.224: train: epoch 46 | batch 150/2304 [I 2022-10-27 03:45:23] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:09 [I 2022-10-27 03:45:23] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00250 [I 2022-10-27 03:45:23] ...trap/engines/engine.py.230: loss: 0.21489 [I 2022-10-27 03:45:23] ...optimizers/trijoint.py.151: total_norm: 15.064097 [I 2022-10-27 03:45:28] ...trap/engines/engine.py.224: train: epoch 46 | batch 160/2304 [I 2022-10-27 03:45:28] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:20:01 [I 2022-10-27 03:45:28] ...trap/engines/engine.py.229: process: 0.57005 | load: 0.00255 [I 2022-10-27 03:45:28] ...trap/engines/engine.py.230: loss: 0.19539 [I 2022-10-27 03:45:28] ...optimizers/trijoint.py.151: total_norm: 19.758316 [I 2022-10-27 03:45:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 170/2304 [I 2022-10-27 03:45:34] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:49 [I 2022-10-27 03:45:34] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00247 [I 2022-10-27 03:45:34] ...trap/engines/engine.py.230: loss: 0.16556 [I 2022-10-27 03:45:34] ...optimizers/trijoint.py.151: total_norm: 13.741737 [I 2022-10-27 03:45:40] ...trap/engines/engine.py.224: train: epoch 46 | batch 180/2304 [I 2022-10-27 03:45:40] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:46 [I 2022-10-27 03:45:40] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00261 [I 2022-10-27 03:45:40] ...trap/engines/engine.py.230: loss: 0.16470 [I 2022-10-27 03:45:40] ...optimizers/trijoint.py.151: total_norm: 18.018461 [I 2022-10-27 03:45:45] ...trap/engines/engine.py.224: train: epoch 46 | batch 190/2304 [I 2022-10-27 03:45:45] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:49 [I 2022-10-27 03:45:45] ...trap/engines/engine.py.229: process: 0.57658 | load: 0.00252 [I 2022-10-27 03:45:45] ...trap/engines/engine.py.230: loss: 0.14554 [I 2022-10-27 03:45:45] ...optimizers/trijoint.py.151: total_norm: 11.944115 [I 2022-10-27 03:45:51] ...trap/engines/engine.py.224: train: epoch 46 | batch 200/2304 [I 2022-10-27 03:45:51] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:32 [I 2022-10-27 03:45:51] ...trap/engines/engine.py.229: process: 0.54994 | load: 0.00263 [I 2022-10-27 03:45:51] ...trap/engines/engine.py.230: loss: 0.19567 [I 2022-10-27 03:45:51] ...optimizers/trijoint.py.151: total_norm: 36.596424 [I 2022-10-27 03:45:56] ...trap/engines/engine.py.224: train: epoch 46 | batch 210/2304 [I 2022-10-27 03:45:56] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:48 [I 2022-10-27 03:45:56] ...trap/engines/engine.py.229: process: 0.56216 | load: 0.00255 [I 2022-10-27 03:45:56] ...trap/engines/engine.py.230: loss: 0.11969 [I 2022-10-27 03:45:56] ...optimizers/trijoint.py.151: total_norm: 35.786236 [I 2022-10-27 03:46:02] ...trap/engines/engine.py.224: train: epoch 46 | batch 220/2304 [I 2022-10-27 03:46:02] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:36 [I 2022-10-27 03:46:02] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00269 [I 2022-10-27 03:46:02] ...trap/engines/engine.py.230: loss: 0.12440 [I 2022-10-27 03:46:02] ...optimizers/trijoint.py.151: total_norm: 13.518152 [I 2022-10-27 03:46:08] ...trap/engines/engine.py.224: train: epoch 46 | batch 230/2304 [I 2022-10-27 03:46:08] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:27 [I 2022-10-27 03:46:08] ...trap/engines/engine.py.229: process: 0.55657 | load: 0.00276 [I 2022-10-27 03:46:08] ...trap/engines/engine.py.230: loss: 0.18183 [I 2022-10-27 03:46:08] ...optimizers/trijoint.py.151: total_norm: 17.706774 [I 2022-10-27 03:46:13] ...trap/engines/engine.py.224: train: epoch 46 | batch 240/2304 [I 2022-10-27 03:46:13] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:08 [I 2022-10-27 03:46:13] ...trap/engines/engine.py.229: process: 0.55026 | load: 0.00254 [I 2022-10-27 03:46:13] ...trap/engines/engine.py.230: loss: 0.25057 [I 2022-10-27 03:46:13] ...optimizers/trijoint.py.151: total_norm: 32.524818 [I 2022-10-27 03:46:19] ...trap/engines/engine.py.224: train: epoch 46 | batch 250/2304 [I 2022-10-27 03:46:19] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:02 [I 2022-10-27 03:46:19] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00245 [I 2022-10-27 03:46:19] ...trap/engines/engine.py.230: loss: 0.17864 [I 2022-10-27 03:46:19] ...optimizers/trijoint.py.151: total_norm: 37.394894 [I 2022-10-27 03:46:24] ...trap/engines/engine.py.224: train: epoch 46 | batch 260/2304 [I 2022-10-27 03:46:24] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:02 [I 2022-10-27 03:46:24] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00257 [I 2022-10-27 03:46:24] ...trap/engines/engine.py.230: loss: 0.22700 [I 2022-10-27 03:46:24] ...optimizers/trijoint.py.151: total_norm: 53.217300 [I 2022-10-27 03:46:30] ...trap/engines/engine.py.224: train: epoch 46 | batch 270/2304 [I 2022-10-27 03:46:30] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:05 [I 2022-10-27 03:46:30] ...trap/engines/engine.py.229: process: 0.57122 | load: 0.00252 [I 2022-10-27 03:46:30] ...trap/engines/engine.py.230: loss: 0.14298 [I 2022-10-27 03:46:30] ...optimizers/trijoint.py.151: total_norm: 12.878510 [I 2022-10-27 03:46:36] ...trap/engines/engine.py.224: train: epoch 46 | batch 280/2304 [I 2022-10-27 03:46:36] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:19:25 [I 2022-10-27 03:46:36] ...trap/engines/engine.py.229: process: 0.55162 | load: 0.00262 [I 2022-10-27 03:46:36] ...trap/engines/engine.py.230: loss: 0.13612 [I 2022-10-27 03:46:36] ...optimizers/trijoint.py.151: total_norm: 15.296090 [I 2022-10-27 03:46:41] ...trap/engines/engine.py.224: train: epoch 46 | batch 290/2304 [I 2022-10-27 03:46:41] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:50 [I 2022-10-27 03:46:41] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00253 [I 2022-10-27 03:46:41] ...trap/engines/engine.py.230: loss: 0.17493 [I 2022-10-27 03:46:41] ...optimizers/trijoint.py.151: total_norm: 22.265442 [I 2022-10-27 03:46:47] ...trap/engines/engine.py.224: train: epoch 46 | batch 300/2304 [I 2022-10-27 03:46:47] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:45 [I 2022-10-27 03:46:47] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00248 [I 2022-10-27 03:46:47] ...trap/engines/engine.py.230: loss: 0.21235 [I 2022-10-27 03:46:47] ...optimizers/trijoint.py.151: total_norm: 21.792881 [I 2022-10-27 03:46:53] ...trap/engines/engine.py.224: train: epoch 46 | batch 310/2304 [I 2022-10-27 03:46:53] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:36 [I 2022-10-27 03:46:53] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00244 [I 2022-10-27 03:46:53] ...trap/engines/engine.py.230: loss: 0.13777 [I 2022-10-27 03:46:53] ...optimizers/trijoint.py.151: total_norm: 18.605850 [I 2022-10-27 03:46:58] ...trap/engines/engine.py.224: train: epoch 46 | batch 320/2304 [I 2022-10-27 03:46:58] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:27 [I 2022-10-27 03:46:58] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00244 [I 2022-10-27 03:46:58] ...trap/engines/engine.py.230: loss: 0.14964 [I 2022-10-27 03:46:58] ...optimizers/trijoint.py.151: total_norm: 20.234751 [I 2022-10-27 03:47:04] ...trap/engines/engine.py.224: train: epoch 46 | batch 330/2304 [I 2022-10-27 03:47:04] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:15 [I 2022-10-27 03:47:04] ...trap/engines/engine.py.229: process: 0.54760 | load: 0.00252 [I 2022-10-27 03:47:04] ...trap/engines/engine.py.230: loss: 0.12438 [I 2022-10-27 03:47:04] ...optimizers/trijoint.py.151: total_norm: 19.651209 [I 2022-10-27 03:47:09] ...trap/engines/engine.py.224: train: epoch 46 | batch 340/2304 [I 2022-10-27 03:47:09] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:14 [I 2022-10-27 03:47:09] ...trap/engines/engine.py.229: process: 0.56865 | load: 0.00254 [I 2022-10-27 03:47:09] ...trap/engines/engine.py.230: loss: 0.21147 [I 2022-10-27 03:47:09] ...optimizers/trijoint.py.151: total_norm: 19.718706 [I 2022-10-27 03:47:15] ...trap/engines/engine.py.224: train: epoch 46 | batch 350/2304 [I 2022-10-27 03:47:15] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:54 [I 2022-10-27 03:47:15] ...trap/engines/engine.py.229: process: 0.59278 | load: 0.00277 [I 2022-10-27 03:47:15] ...trap/engines/engine.py.230: loss: 0.19108 [I 2022-10-27 03:47:15] ...optimizers/trijoint.py.151: total_norm: 42.349831 [I 2022-10-27 03:47:21] ...trap/engines/engine.py.224: train: epoch 46 | batch 360/2304 [I 2022-10-27 03:47:21] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:07 [I 2022-10-27 03:47:21] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00273 [I 2022-10-27 03:47:21] ...trap/engines/engine.py.230: loss: 0.22196 [I 2022-10-27 03:47:21] ...optimizers/trijoint.py.151: total_norm: 26.528648 [I 2022-10-27 03:47:26] ...trap/engines/engine.py.224: train: epoch 46 | batch 370/2304 [I 2022-10-27 03:47:26] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:03 [I 2022-10-27 03:47:26] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00249 [I 2022-10-27 03:47:26] ...trap/engines/engine.py.230: loss: 0.15170 [I 2022-10-27 03:47:26] ...optimizers/trijoint.py.151: total_norm: 21.091591 [I 2022-10-27 03:47:32] ...trap/engines/engine.py.224: train: epoch 46 | batch 380/2304 [I 2022-10-27 03:47:32] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:52 [I 2022-10-27 03:47:32] ...trap/engines/engine.py.229: process: 0.56764 | load: 0.00247 [I 2022-10-27 03:47:32] ...trap/engines/engine.py.230: loss: 0.24659 [I 2022-10-27 03:47:32] ...optimizers/trijoint.py.151: total_norm: 14.573962 [I 2022-10-27 03:47:37] ...trap/engines/engine.py.224: train: epoch 46 | batch 390/2304 [I 2022-10-27 03:47:37] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:42 [I 2022-10-27 03:47:37] ...trap/engines/engine.py.229: process: 0.54259 | load: 0.00244 [I 2022-10-27 03:47:37] ...trap/engines/engine.py.230: loss: 0.12915 [I 2022-10-27 03:47:37] ...optimizers/trijoint.py.151: total_norm: 11.296709 [I 2022-10-27 03:47:43] ...trap/engines/engine.py.224: train: epoch 46 | batch 400/2304 [I 2022-10-27 03:47:43] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:41 [I 2022-10-27 03:47:43] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00252 [I 2022-10-27 03:47:43] ...trap/engines/engine.py.230: loss: 0.18937 [I 2022-10-27 03:47:43] ...optimizers/trijoint.py.151: total_norm: 31.887064 [I 2022-10-27 03:47:49] ...trap/engines/engine.py.224: train: epoch 46 | batch 410/2304 [I 2022-10-27 03:47:49] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:41 [I 2022-10-27 03:47:49] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00247 [I 2022-10-27 03:47:49] ...trap/engines/engine.py.230: loss: 0.16756 [I 2022-10-27 03:47:49] ...optimizers/trijoint.py.151: total_norm: 25.016296 [I 2022-10-27 03:47:54] ...trap/engines/engine.py.224: train: epoch 46 | batch 420/2304 [I 2022-10-27 03:47:54] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:28 [I 2022-10-27 03:47:54] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00245 [I 2022-10-27 03:47:54] ...trap/engines/engine.py.230: loss: 0.14399 [I 2022-10-27 03:47:54] ...optimizers/trijoint.py.151: total_norm: 19.824495 [I 2022-10-27 03:48:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 430/2304 [I 2022-10-27 03:48:00] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:25 [I 2022-10-27 03:48:00] ...trap/engines/engine.py.229: process: 0.56156 | load: 0.00241 [I 2022-10-27 03:48:00] ...trap/engines/engine.py.230: loss: 0.10464 [I 2022-10-27 03:48:00] ...optimizers/trijoint.py.151: total_norm: 10.008117 [I 2022-10-27 03:48:05] ...trap/engines/engine.py.224: train: epoch 46 | batch 440/2304 [I 2022-10-27 03:48:05] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:27 [I 2022-10-27 03:48:05] ...trap/engines/engine.py.229: process: 0.55339 | load: 0.00245 [I 2022-10-27 03:48:05] ...trap/engines/engine.py.230: loss: 0.18100 [I 2022-10-27 03:48:05] ...optimizers/trijoint.py.151: total_norm: 17.171144 [I 2022-10-27 03:48:11] ...trap/engines/engine.py.224: train: epoch 46 | batch 450/2304 [I 2022-10-27 03:48:11] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:23 [I 2022-10-27 03:48:11] ...trap/engines/engine.py.229: process: 0.57258 | load: 0.00247 [I 2022-10-27 03:48:11] ...trap/engines/engine.py.230: loss: 0.21806 [I 2022-10-27 03:48:11] ...optimizers/trijoint.py.151: total_norm: 19.551441 [I 2022-10-27 03:48:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 460/2304 [I 2022-10-27 03:48:17] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:08 [I 2022-10-27 03:48:17] ...trap/engines/engine.py.229: process: 0.55234 | load: 0.00248 [I 2022-10-27 03:48:17] ...trap/engines/engine.py.230: loss: 0.12061 [I 2022-10-27 03:48:17] ...optimizers/trijoint.py.151: total_norm: 11.735091 [I 2022-10-27 03:48:22] ...trap/engines/engine.py.224: train: epoch 46 | batch 470/2304 [I 2022-10-27 03:48:22] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:59 [I 2022-10-27 03:48:22] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00243 [I 2022-10-27 03:48:22] ...trap/engines/engine.py.230: loss: 0.15952 [I 2022-10-27 03:48:22] ...optimizers/trijoint.py.151: total_norm: 11.334324 [I 2022-10-27 03:48:28] ...trap/engines/engine.py.224: train: epoch 46 | batch 480/2304 [I 2022-10-27 03:48:28] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:10 [I 2022-10-27 03:48:28] ...trap/engines/engine.py.229: process: 0.55623 | load: 0.00302 [I 2022-10-27 03:48:28] ...trap/engines/engine.py.230: loss: 0.17220 [I 2022-10-27 03:48:28] ...optimizers/trijoint.py.151: total_norm: 22.269217 [I 2022-10-27 03:48:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 490/2304 [I 2022-10-27 03:48:34] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:57 [I 2022-10-27 03:48:34] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00244 [I 2022-10-27 03:48:34] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-27 03:48:34] ...optimizers/trijoint.py.151: total_norm: 13.764141 [I 2022-10-27 03:48:39] ...trap/engines/engine.py.224: train: epoch 46 | batch 500/2304 [I 2022-10-27 03:48:39] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:44 [I 2022-10-27 03:48:39] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00244 [I 2022-10-27 03:48:39] ...trap/engines/engine.py.230: loss: 0.21502 [I 2022-10-27 03:48:39] ...optimizers/trijoint.py.151: total_norm: 21.584585 [I 2022-10-27 03:48:45] ...trap/engines/engine.py.224: train: epoch 46 | batch 510/2304 [I 2022-10-27 03:48:45] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:44 [I 2022-10-27 03:48:45] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00245 [I 2022-10-27 03:48:45] ...trap/engines/engine.py.230: loss: 0.17140 [I 2022-10-27 03:48:45] ...optimizers/trijoint.py.151: total_norm: 12.697563 [I 2022-10-27 03:48:50] ...trap/engines/engine.py.224: train: epoch 46 | batch 520/2304 [I 2022-10-27 03:48:50] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:30 [I 2022-10-27 03:48:50] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00244 [I 2022-10-27 03:48:50] ...trap/engines/engine.py.230: loss: 0.14621 [I 2022-10-27 03:48:50] ...optimizers/trijoint.py.151: total_norm: 14.224754 [I 2022-10-27 03:48:56] ...trap/engines/engine.py.224: train: epoch 46 | batch 530/2304 [I 2022-10-27 03:48:56] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:32 [I 2022-10-27 03:48:56] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00242 [I 2022-10-27 03:48:56] ...trap/engines/engine.py.230: loss: 0.19014 [I 2022-10-27 03:48:56] ...optimizers/trijoint.py.151: total_norm: 13.494854 [I 2022-10-27 03:49:02] ...trap/engines/engine.py.224: train: epoch 46 | batch 540/2304 [I 2022-10-27 03:49:02] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:27 [I 2022-10-27 03:49:02] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00246 [I 2022-10-27 03:49:02] ...trap/engines/engine.py.230: loss: 0.15233 [I 2022-10-27 03:49:02] ...optimizers/trijoint.py.151: total_norm: 9.338285 [I 2022-10-27 03:49:07] ...trap/engines/engine.py.224: train: epoch 46 | batch 550/2304 [I 2022-10-27 03:49:07] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:25 [I 2022-10-27 03:49:07] ...trap/engines/engine.py.229: process: 0.54696 | load: 0.00249 [I 2022-10-27 03:49:07] ...trap/engines/engine.py.230: loss: 0.16619 [I 2022-10-27 03:49:07] ...optimizers/trijoint.py.151: total_norm: 19.316395 [I 2022-10-27 03:49:13] ...trap/engines/engine.py.224: train: epoch 46 | batch 560/2304 [I 2022-10-27 03:49:13] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:14 [I 2022-10-27 03:49:13] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00240 [I 2022-10-27 03:49:13] ...trap/engines/engine.py.230: loss: 0.18503 [I 2022-10-27 03:49:13] ...optimizers/trijoint.py.151: total_norm: 30.715422 [I 2022-10-27 03:49:18] ...trap/engines/engine.py.224: train: epoch 46 | batch 570/2304 [I 2022-10-27 03:49:18] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:12 [I 2022-10-27 03:49:18] ...trap/engines/engine.py.229: process: 0.56280 | load: 0.00254 [I 2022-10-27 03:49:18] ...trap/engines/engine.py.230: loss: 0.15270 [I 2022-10-27 03:49:18] ...optimizers/trijoint.py.151: total_norm: 24.344364 [I 2022-10-27 03:49:24] ...trap/engines/engine.py.224: train: epoch 46 | batch 580/2304 [I 2022-10-27 03:49:24] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:01 [I 2022-10-27 03:49:24] ...trap/engines/engine.py.229: process: 0.54867 | load: 0.00243 [I 2022-10-27 03:49:24] ...trap/engines/engine.py.230: loss: 0.15457 [I 2022-10-27 03:49:24] ...optimizers/trijoint.py.151: total_norm: 14.110253 [I 2022-10-27 03:49:30] ...trap/engines/engine.py.224: train: epoch 46 | batch 590/2304 [I 2022-10-27 03:49:30] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:16 [I 2022-10-27 03:49:30] ...trap/engines/engine.py.229: process: 0.54674 | load: 0.00264 [I 2022-10-27 03:49:30] ...trap/engines/engine.py.230: loss: 0.10798 [I 2022-10-27 03:49:30] ...optimizers/trijoint.py.151: total_norm: 8.094156 [I 2022-10-27 03:49:35] ...trap/engines/engine.py.224: train: epoch 46 | batch 600/2304 [I 2022-10-27 03:49:35] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:55 [I 2022-10-27 03:49:35] ...trap/engines/engine.py.229: process: 0.57114 | load: 0.00238 [I 2022-10-27 03:49:35] ...trap/engines/engine.py.230: loss: 0.21864 [I 2022-10-27 03:49:35] ...optimizers/trijoint.py.151: total_norm: 22.461304 [I 2022-10-27 03:49:41] ...trap/engines/engine.py.224: train: epoch 46 | batch 610/2304 [I 2022-10-27 03:49:41] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:50 [I 2022-10-27 03:49:41] ...trap/engines/engine.py.229: process: 0.57120 | load: 0.00243 [I 2022-10-27 03:49:41] ...trap/engines/engine.py.230: loss: 0.16334 [I 2022-10-27 03:49:41] ...optimizers/trijoint.py.151: total_norm: 26.985256 [I 2022-10-27 03:49:46] ...trap/engines/engine.py.224: train: epoch 46 | batch 620/2304 [I 2022-10-27 03:49:46] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:44 [I 2022-10-27 03:49:46] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00241 [I 2022-10-27 03:49:46] ...trap/engines/engine.py.230: loss: 0.24584 [I 2022-10-27 03:49:46] ...optimizers/trijoint.py.151: total_norm: 13.045388 [I 2022-10-27 03:49:52] ...trap/engines/engine.py.224: train: epoch 46 | batch 630/2304 [I 2022-10-27 03:49:52] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:30 [I 2022-10-27 03:49:52] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00240 [I 2022-10-27 03:49:52] ...trap/engines/engine.py.230: loss: 0.13574 [I 2022-10-27 03:49:52] ...optimizers/trijoint.py.151: total_norm: 11.327558 [I 2022-10-27 03:49:58] ...trap/engines/engine.py.224: train: epoch 46 | batch 640/2304 [I 2022-10-27 03:49:58] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:29 [I 2022-10-27 03:49:58] ...trap/engines/engine.py.229: process: 0.54808 | load: 0.00239 [I 2022-10-27 03:49:58] ...trap/engines/engine.py.230: loss: 0.13947 [I 2022-10-27 03:49:58] ...optimizers/trijoint.py.151: total_norm: 17.315199 [I 2022-10-27 03:50:03] ...trap/engines/engine.py.224: train: epoch 46 | batch 650/2304 [I 2022-10-27 03:50:03] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:20 [I 2022-10-27 03:50:03] ...trap/engines/engine.py.229: process: 0.56351 | load: 0.00238 [I 2022-10-27 03:50:03] ...trap/engines/engine.py.230: loss: 0.16452 [I 2022-10-27 03:50:03] ...optimizers/trijoint.py.151: total_norm: 20.725611 [I 2022-10-27 03:50:09] ...trap/engines/engine.py.224: train: epoch 46 | batch 660/2304 [I 2022-10-27 03:50:09] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:14 [I 2022-10-27 03:50:09] ...trap/engines/engine.py.229: process: 0.55819 | load: 0.00242 [I 2022-10-27 03:50:09] ...trap/engines/engine.py.230: loss: 0.21726 [I 2022-10-27 03:50:09] ...optimizers/trijoint.py.151: total_norm: 24.998795 [I 2022-10-27 03:50:14] ...trap/engines/engine.py.224: train: epoch 46 | batch 670/2304 [I 2022-10-27 03:50:14] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:09 [I 2022-10-27 03:50:14] ...trap/engines/engine.py.229: process: 0.54794 | load: 0.00242 [I 2022-10-27 03:50:14] ...trap/engines/engine.py.230: loss: 0.15516 [I 2022-10-27 03:50:14] ...optimizers/trijoint.py.151: total_norm: 11.915636 [I 2022-10-27 03:50:20] ...trap/engines/engine.py.224: train: epoch 46 | batch 680/2304 [I 2022-10-27 03:50:20] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:14:55 [I 2022-10-27 03:50:20] ...trap/engines/engine.py.229: process: 0.53847 | load: 0.00241 [I 2022-10-27 03:50:20] ...trap/engines/engine.py.230: loss: 0.14637 [I 2022-10-27 03:50:20] ...optimizers/trijoint.py.151: total_norm: 17.958111 [I 2022-10-27 03:50:25] ...trap/engines/engine.py.224: train: epoch 46 | batch 690/2304 [I 2022-10-27 03:50:25] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:14:58 [I 2022-10-27 03:50:25] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00242 [I 2022-10-27 03:50:25] ...trap/engines/engine.py.230: loss: 0.16634 [I 2022-10-27 03:50:25] ...optimizers/trijoint.py.151: total_norm: 15.280036 [I 2022-10-27 03:50:31] ...trap/engines/engine.py.224: train: epoch 46 | batch 700/2304 [I 2022-10-27 03:50:31] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:15:07 [I 2022-10-27 03:50:31] ...trap/engines/engine.py.229: process: 0.56564 | load: 0.00238 [I 2022-10-27 03:50:31] ...trap/engines/engine.py.230: loss: 0.19747 [I 2022-10-27 03:50:31] ...optimizers/trijoint.py.151: total_norm: 12.951024 [I 2022-10-27 03:50:37] ...trap/engines/engine.py.224: train: epoch 46 | batch 710/2304 [I 2022-10-27 03:50:37] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:48 [I 2022-10-27 03:50:37] ...trap/engines/engine.py.229: process: 0.55803 | load: 0.00243 [I 2022-10-27 03:50:37] ...trap/engines/engine.py.230: loss: 0.12023 [I 2022-10-27 03:50:37] ...optimizers/trijoint.py.151: total_norm: 12.475975 [I 2022-10-27 03:50:42] ...trap/engines/engine.py.224: train: epoch 46 | batch 720/2304 [I 2022-10-27 03:50:42] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:41 [I 2022-10-27 03:50:42] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00243 [I 2022-10-27 03:50:42] ...trap/engines/engine.py.230: loss: 0.21150 [I 2022-10-27 03:50:42] ...optimizers/trijoint.py.151: total_norm: 24.491432 [I 2022-10-27 03:50:48] ...trap/engines/engine.py.224: train: epoch 46 | batch 730/2304 [I 2022-10-27 03:50:48] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:32 [I 2022-10-27 03:50:48] ...trap/engines/engine.py.229: process: 0.54016 | load: 0.00243 [I 2022-10-27 03:50:48] ...trap/engines/engine.py.230: loss: 0.16765 [I 2022-10-27 03:50:48] ...optimizers/trijoint.py.151: total_norm: 16.343864 [I 2022-10-27 03:50:53] ...trap/engines/engine.py.224: train: epoch 46 | batch 740/2304 [I 2022-10-27 03:50:53] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:31 [I 2022-10-27 03:50:53] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00243 [I 2022-10-27 03:50:53] ...trap/engines/engine.py.230: loss: 0.14250 [I 2022-10-27 03:50:53] ...optimizers/trijoint.py.151: total_norm: 16.047987 [I 2022-10-27 03:50:59] ...trap/engines/engine.py.224: train: epoch 46 | batch 750/2304 [I 2022-10-27 03:50:59] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:28 [I 2022-10-27 03:50:59] ...trap/engines/engine.py.229: process: 0.55158 | load: 0.00257 [I 2022-10-27 03:50:59] ...trap/engines/engine.py.230: loss: 0.19930 [I 2022-10-27 03:50:59] ...optimizers/trijoint.py.151: total_norm: 15.921304 [I 2022-10-27 03:51:05] ...trap/engines/engine.py.224: train: epoch 46 | batch 760/2304 [I 2022-10-27 03:51:05] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:18 [I 2022-10-27 03:51:05] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00244 [I 2022-10-27 03:51:05] ...trap/engines/engine.py.230: loss: 0.14718 [I 2022-10-27 03:51:05] ...optimizers/trijoint.py.151: total_norm: 30.148682 [I 2022-10-27 03:51:10] ...trap/engines/engine.py.224: train: epoch 46 | batch 770/2304 [I 2022-10-27 03:51:10] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:16 [I 2022-10-27 03:51:10] ...trap/engines/engine.py.229: process: 0.56583 | load: 0.00247 [I 2022-10-27 03:51:10] ...trap/engines/engine.py.230: loss: 0.20391 [I 2022-10-27 03:51:10] ...optimizers/trijoint.py.151: total_norm: 16.221283 [I 2022-10-27 03:51:16] ...trap/engines/engine.py.224: train: epoch 46 | batch 780/2304 [I 2022-10-27 03:51:16] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:08 [I 2022-10-27 03:51:16] ...trap/engines/engine.py.229: process: 0.55206 | load: 0.00241 [I 2022-10-27 03:51:16] ...trap/engines/engine.py.230: loss: 0.17649 [I 2022-10-27 03:51:16] ...optimizers/trijoint.py.151: total_norm: 36.010227 [I 2022-10-27 03:51:21] ...trap/engines/engine.py.224: train: epoch 46 | batch 790/2304 [I 2022-10-27 03:51:21] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:02 [I 2022-10-27 03:51:21] ...trap/engines/engine.py.229: process: 0.55050 | load: 0.00243 [I 2022-10-27 03:51:21] ...trap/engines/engine.py.230: loss: 0.17296 [I 2022-10-27 03:51:21] ...optimizers/trijoint.py.151: total_norm: 13.345845 [I 2022-10-27 03:51:27] ...trap/engines/engine.py.224: train: epoch 46 | batch 800/2304 [I 2022-10-27 03:51:27] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:54 [I 2022-10-27 03:51:27] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00243 [I 2022-10-27 03:51:27] ...trap/engines/engine.py.230: loss: 0.20116 [I 2022-10-27 03:51:27] ...optimizers/trijoint.py.151: total_norm: 14.840629 [I 2022-10-27 03:51:32] ...trap/engines/engine.py.224: train: epoch 46 | batch 810/2304 [I 2022-10-27 03:51:32] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:57 [I 2022-10-27 03:51:32] ...trap/engines/engine.py.229: process: 0.56472 | load: 0.00240 [I 2022-10-27 03:51:32] ...trap/engines/engine.py.230: loss: 0.14025 [I 2022-10-27 03:51:32] ...optimizers/trijoint.py.151: total_norm: 33.409561 [I 2022-10-27 03:51:38] ...trap/engines/engine.py.224: train: epoch 46 | batch 820/2304 [I 2022-10-27 03:51:38] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:51 [I 2022-10-27 03:51:38] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00248 [I 2022-10-27 03:51:38] ...trap/engines/engine.py.230: loss: 0.15920 [I 2022-10-27 03:51:38] ...optimizers/trijoint.py.151: total_norm: 25.206919 [I 2022-10-27 03:51:44] ...trap/engines/engine.py.224: train: epoch 46 | batch 830/2304 [I 2022-10-27 03:51:44] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:44 [I 2022-10-27 03:51:44] ...trap/engines/engine.py.229: process: 0.56537 | load: 0.00243 [I 2022-10-27 03:51:44] ...trap/engines/engine.py.230: loss: 0.10915 [I 2022-10-27 03:51:44] ...optimizers/trijoint.py.151: total_norm: 12.849238 [I 2022-10-27 03:51:49] ...trap/engines/engine.py.224: train: epoch 46 | batch 840/2304 [I 2022-10-27 03:51:49] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:36 [I 2022-10-27 03:51:49] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00241 [I 2022-10-27 03:51:49] ...trap/engines/engine.py.230: loss: 0.14162 [I 2022-10-27 03:51:49] ...optimizers/trijoint.py.151: total_norm: 11.100258 [I 2022-10-27 03:51:55] ...trap/engines/engine.py.224: train: epoch 46 | batch 850/2304 [I 2022-10-27 03:51:55] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:39 [I 2022-10-27 03:51:55] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00244 [I 2022-10-27 03:51:55] ...trap/engines/engine.py.230: loss: 0.12483 [I 2022-10-27 03:51:55] ...optimizers/trijoint.py.151: total_norm: 8.524649 [I 2022-10-27 03:52:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 860/2304 [I 2022-10-27 03:52:00] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:25 [I 2022-10-27 03:52:00] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00244 [I 2022-10-27 03:52:00] ...trap/engines/engine.py.230: loss: 0.24185 [I 2022-10-27 03:52:00] ...optimizers/trijoint.py.151: total_norm: 34.146400 [I 2022-10-27 03:52:06] ...trap/engines/engine.py.224: train: epoch 46 | batch 870/2304 [I 2022-10-27 03:52:06] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:16 [I 2022-10-27 03:52:06] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00252 [I 2022-10-27 03:52:06] ...trap/engines/engine.py.230: loss: 0.15427 [I 2022-10-27 03:52:06] ...optimizers/trijoint.py.151: total_norm: 20.303371 [I 2022-10-27 03:52:12] ...trap/engines/engine.py.224: train: epoch 46 | batch 880/2304 [I 2022-10-27 03:52:12] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:08 [I 2022-10-27 03:52:12] ...trap/engines/engine.py.229: process: 0.54543 | load: 0.00247 [I 2022-10-27 03:52:12] ...trap/engines/engine.py.230: loss: 0.15330 [I 2022-10-27 03:52:12] ...optimizers/trijoint.py.151: total_norm: 11.015024 [I 2022-10-27 03:52:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 890/2304 [I 2022-10-27 03:52:17] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:06 [I 2022-10-27 03:52:17] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00244 [I 2022-10-27 03:52:17] ...trap/engines/engine.py.230: loss: 0.11607 [I 2022-10-27 03:52:17] ...optimizers/trijoint.py.151: total_norm: 14.258985 [I 2022-10-27 03:52:23] ...trap/engines/engine.py.224: train: epoch 46 | batch 900/2304 [I 2022-10-27 03:52:23] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:12:57 [I 2022-10-27 03:52:23] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00242 [I 2022-10-27 03:52:23] ...trap/engines/engine.py.230: loss: 0.21480 [I 2022-10-27 03:52:23] ...optimizers/trijoint.py.151: total_norm: 25.318665 [I 2022-10-27 03:52:28] ...trap/engines/engine.py.224: train: epoch 46 | batch 910/2304 [I 2022-10-27 03:52:28] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:57 [I 2022-10-27 03:52:28] ...trap/engines/engine.py.229: process: 0.56412 | load: 0.00243 [I 2022-10-27 03:52:28] ...trap/engines/engine.py.230: loss: 0.17875 [I 2022-10-27 03:52:28] ...optimizers/trijoint.py.151: total_norm: 17.037693 [I 2022-10-27 03:52:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 920/2304 [I 2022-10-27 03:52:34] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:48 [I 2022-10-27 03:52:34] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00243 [I 2022-10-27 03:52:34] ...trap/engines/engine.py.230: loss: 0.12893 [I 2022-10-27 03:52:34] ...optimizers/trijoint.py.151: total_norm: 14.819620 [I 2022-10-27 03:52:39] ...trap/engines/engine.py.224: train: epoch 46 | batch 930/2304 [I 2022-10-27 03:52:39] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:47 [I 2022-10-27 03:52:39] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00244 [I 2022-10-27 03:52:39] ...trap/engines/engine.py.230: loss: 0.11478 [I 2022-10-27 03:52:39] ...optimizers/trijoint.py.151: total_norm: 17.631348 [I 2022-10-27 03:52:45] ...trap/engines/engine.py.224: train: epoch 46 | batch 940/2304 [I 2022-10-27 03:52:45] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:38 [I 2022-10-27 03:52:45] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00245 [I 2022-10-27 03:52:45] ...trap/engines/engine.py.230: loss: 0.21991 [I 2022-10-27 03:52:45] ...optimizers/trijoint.py.151: total_norm: 20.093058 [I 2022-10-27 03:52:51] ...trap/engines/engine.py.224: train: epoch 46 | batch 950/2304 [I 2022-10-27 03:52:51] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:29 [I 2022-10-27 03:52:51] ...trap/engines/engine.py.229: process: 0.54942 | load: 0.00240 [I 2022-10-27 03:52:51] ...trap/engines/engine.py.230: loss: 0.18295 [I 2022-10-27 03:52:51] ...optimizers/trijoint.py.151: total_norm: 16.286352 [I 2022-10-27 03:52:56] ...trap/engines/engine.py.224: train: epoch 46 | batch 960/2304 [I 2022-10-27 03:52:56] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:27 [I 2022-10-27 03:52:56] ...trap/engines/engine.py.229: process: 0.55000 | load: 0.00242 [I 2022-10-27 03:52:56] ...trap/engines/engine.py.230: loss: 0.14666 [I 2022-10-27 03:52:56] ...optimizers/trijoint.py.151: total_norm: 16.000792 [I 2022-10-27 03:53:02] ...trap/engines/engine.py.224: train: epoch 46 | batch 970/2304 [I 2022-10-27 03:53:02] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:22 [I 2022-10-27 03:53:02] ...trap/engines/engine.py.229: process: 0.56055 | load: 0.00243 [I 2022-10-27 03:53:02] ...trap/engines/engine.py.230: loss: 0.24027 [I 2022-10-27 03:53:02] ...optimizers/trijoint.py.151: total_norm: 23.300592 [I 2022-10-27 03:53:07] ...trap/engines/engine.py.224: train: epoch 46 | batch 980/2304 [I 2022-10-27 03:53:07] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:13 [I 2022-10-27 03:53:07] ...trap/engines/engine.py.229: process: 0.55787 | load: 0.00252 [I 2022-10-27 03:53:07] ...trap/engines/engine.py.230: loss: 0.18552 [I 2022-10-27 03:53:07] ...optimizers/trijoint.py.151: total_norm: 30.630501 [I 2022-10-27 03:53:13] ...trap/engines/engine.py.224: train: epoch 46 | batch 990/2304 [I 2022-10-27 03:53:13] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:08 [I 2022-10-27 03:53:13] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00247 [I 2022-10-27 03:53:13] ...trap/engines/engine.py.230: loss: 0.18194 [I 2022-10-27 03:53:13] ...optimizers/trijoint.py.151: total_norm: 13.003393 [I 2022-10-27 03:53:18] ...trap/engines/engine.py.224: train: epoch 46 | batch 1000/2304 [I 2022-10-27 03:53:18] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:03 [I 2022-10-27 03:53:18] ...trap/engines/engine.py.229: process: 0.55391 | load: 0.00245 [I 2022-10-27 03:53:18] ...trap/engines/engine.py.230: loss: 0.14386 [I 2022-10-27 03:53:18] ...optimizers/trijoint.py.151: total_norm: 13.382320 [I 2022-10-27 03:53:24] ...trap/engines/engine.py.224: train: epoch 46 | batch 1010/2304 [I 2022-10-27 03:53:24] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:11:53 [I 2022-10-27 03:53:24] ...trap/engines/engine.py.229: process: 0.54821 | load: 0.00247 [I 2022-10-27 03:53:24] ...trap/engines/engine.py.230: loss: 0.13665 [I 2022-10-27 03:53:24] ...optimizers/trijoint.py.151: total_norm: 18.341993 [I 2022-10-27 03:53:30] ...trap/engines/engine.py.224: train: epoch 46 | batch 1020/2304 [I 2022-10-27 03:53:30] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:50 [I 2022-10-27 03:53:30] ...trap/engines/engine.py.229: process: 0.54764 | load: 0.00242 [I 2022-10-27 03:53:30] ...trap/engines/engine.py.230: loss: 0.16599 [I 2022-10-27 03:53:30] ...optimizers/trijoint.py.151: total_norm: 17.286663 [I 2022-10-27 03:53:35] ...trap/engines/engine.py.224: train: epoch 46 | batch 1030/2304 [I 2022-10-27 03:53:35] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:47 [I 2022-10-27 03:53:35] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00243 [I 2022-10-27 03:53:35] ...trap/engines/engine.py.230: loss: 0.18417 [I 2022-10-27 03:53:35] ...optimizers/trijoint.py.151: total_norm: 19.520287 [I 2022-10-27 03:53:41] ...trap/engines/engine.py.224: train: epoch 46 | batch 1040/2304 [I 2022-10-27 03:53:41] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:46 [I 2022-10-27 03:53:41] ...trap/engines/engine.py.229: process: 0.56352 | load: 0.00244 [I 2022-10-27 03:53:41] ...trap/engines/engine.py.230: loss: 0.18488 [I 2022-10-27 03:53:41] ...optimizers/trijoint.py.151: total_norm: 11.308979 [I 2022-10-27 03:53:46] ...trap/engines/engine.py.224: train: epoch 46 | batch 1050/2304 [I 2022-10-27 03:53:46] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:40 [I 2022-10-27 03:53:46] ...trap/engines/engine.py.229: process: 0.56370 | load: 0.00243 [I 2022-10-27 03:53:46] ...trap/engines/engine.py.230: loss: 0.19840 [I 2022-10-27 03:53:46] ...optimizers/trijoint.py.151: total_norm: 24.335566 [I 2022-10-27 03:53:52] ...trap/engines/engine.py.224: train: epoch 46 | batch 1060/2304 [I 2022-10-27 03:53:52] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:35 [I 2022-10-27 03:53:52] ...trap/engines/engine.py.229: process: 0.56405 | load: 0.00244 [I 2022-10-27 03:53:52] ...trap/engines/engine.py.230: loss: 0.14920 [I 2022-10-27 03:53:52] ...optimizers/trijoint.py.151: total_norm: 33.123169 [I 2022-10-27 03:53:57] ...trap/engines/engine.py.224: train: epoch 46 | batch 1070/2304 [I 2022-10-27 03:53:57] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:30 [I 2022-10-27 03:53:57] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00244 [I 2022-10-27 03:53:57] ...trap/engines/engine.py.230: loss: 0.13197 [I 2022-10-27 03:53:57] ...optimizers/trijoint.py.151: total_norm: 10.350373 [I 2022-10-27 03:54:03] ...trap/engines/engine.py.224: train: epoch 46 | batch 1080/2304 [I 2022-10-27 03:54:03] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:16 [I 2022-10-27 03:54:03] ...trap/engines/engine.py.229: process: 0.54949 | load: 0.00248 [I 2022-10-27 03:54:03] ...trap/engines/engine.py.230: loss: 0.20209 [I 2022-10-27 03:54:03] ...optimizers/trijoint.py.151: total_norm: 22.062151 [I 2022-10-27 03:54:09] ...trap/engines/engine.py.224: train: epoch 46 | batch 1090/2304 [I 2022-10-27 03:54:09] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:15 [I 2022-10-27 03:54:09] ...trap/engines/engine.py.229: process: 0.55767 | load: 0.00247 [I 2022-10-27 03:54:09] ...trap/engines/engine.py.230: loss: 0.09203 [I 2022-10-27 03:54:09] ...optimizers/trijoint.py.151: total_norm: 8.989199 [I 2022-10-27 03:54:14] ...trap/engines/engine.py.224: train: epoch 46 | batch 1100/2304 [I 2022-10-27 03:54:14] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:08 [I 2022-10-27 03:54:14] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00244 [I 2022-10-27 03:54:14] ...trap/engines/engine.py.230: loss: 0.15753 [I 2022-10-27 03:54:14] ...optimizers/trijoint.py.151: total_norm: 20.216560 [I 2022-10-27 03:54:20] ...trap/engines/engine.py.224: train: epoch 46 | batch 1110/2304 [I 2022-10-27 03:54:20] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:10:58 [I 2022-10-27 03:54:20] ...trap/engines/engine.py.229: process: 0.55025 | load: 0.00245 [I 2022-10-27 03:54:20] ...trap/engines/engine.py.230: loss: 0.16241 [I 2022-10-27 03:54:20] ...optimizers/trijoint.py.151: total_norm: 15.360669 [I 2022-10-27 03:54:25] ...trap/engines/engine.py.224: train: epoch 46 | batch 1120/2304 [I 2022-10-27 03:54:25] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:10:53 [I 2022-10-27 03:54:25] ...trap/engines/engine.py.229: process: 0.54216 | load: 0.00241 [I 2022-10-27 03:54:25] ...trap/engines/engine.py.230: loss: 0.18443 [I 2022-10-27 03:54:25] ...optimizers/trijoint.py.151: total_norm: 46.469784 [I 2022-10-27 03:54:31] ...trap/engines/engine.py.224: train: epoch 46 | batch 1130/2304 [I 2022-10-27 03:54:31] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:52 [I 2022-10-27 03:54:31] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00242 [I 2022-10-27 03:54:31] ...trap/engines/engine.py.230: loss: 0.14175 [I 2022-10-27 03:54:31] ...optimizers/trijoint.py.151: total_norm: 14.307374 [I 2022-10-27 03:54:36] ...trap/engines/engine.py.224: train: epoch 46 | batch 1140/2304 [I 2022-10-27 03:54:36] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:43 [I 2022-10-27 03:54:36] ...trap/engines/engine.py.229: process: 0.54332 | load: 0.00242 [I 2022-10-27 03:54:36] ...trap/engines/engine.py.230: loss: 0.14545 [I 2022-10-27 03:54:36] ...optimizers/trijoint.py.151: total_norm: 17.595150 [I 2022-10-27 03:54:42] ...trap/engines/engine.py.224: train: epoch 46 | batch 1150/2304 [I 2022-10-27 03:54:42] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:37 [I 2022-10-27 03:54:42] ...trap/engines/engine.py.229: process: 0.54442 | load: 0.00241 [I 2022-10-27 03:54:42] ...trap/engines/engine.py.230: loss: 0.14341 [I 2022-10-27 03:54:42] ...optimizers/trijoint.py.151: total_norm: 11.255208 [I 2022-10-27 03:54:47] ...trap/engines/engine.py.224: train: epoch 46 | batch 1160/2304 [I 2022-10-27 03:54:47] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:34 [I 2022-10-27 03:54:47] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00243 [I 2022-10-27 03:54:47] ...trap/engines/engine.py.230: loss: 0.15943 [I 2022-10-27 03:54:47] ...optimizers/trijoint.py.151: total_norm: 23.400866 [I 2022-10-27 03:54:53] ...trap/engines/engine.py.224: train: epoch 46 | batch 1170/2304 [I 2022-10-27 03:54:53] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:28 [I 2022-10-27 03:54:53] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00244 [I 2022-10-27 03:54:53] ...trap/engines/engine.py.230: loss: 0.13414 [I 2022-10-27 03:54:53] ...optimizers/trijoint.py.151: total_norm: 11.592614 [I 2022-10-27 03:54:59] ...trap/engines/engine.py.224: train: epoch 46 | batch 1180/2304 [I 2022-10-27 03:54:59] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:25 [I 2022-10-27 03:54:59] ...trap/engines/engine.py.229: process: 0.57228 | load: 0.00242 [I 2022-10-27 03:54:59] ...trap/engines/engine.py.230: loss: 0.13218 [I 2022-10-27 03:54:59] ...optimizers/trijoint.py.151: total_norm: 26.684973 [I 2022-10-27 03:55:04] ...trap/engines/engine.py.224: train: epoch 46 | batch 1190/2304 [I 2022-10-27 03:55:04] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:23 [I 2022-10-27 03:55:04] ...trap/engines/engine.py.229: process: 0.56910 | load: 0.00253 [I 2022-10-27 03:55:04] ...trap/engines/engine.py.230: loss: 0.23069 [I 2022-10-27 03:55:04] ...optimizers/trijoint.py.151: total_norm: 31.949190 [I 2022-10-27 03:55:10] ...trap/engines/engine.py.224: train: epoch 46 | batch 1200/2304 [I 2022-10-27 03:55:10] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:13 [I 2022-10-27 03:55:10] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00248 [I 2022-10-27 03:55:10] ...trap/engines/engine.py.230: loss: 0.11048 [I 2022-10-27 03:55:10] ...optimizers/trijoint.py.151: total_norm: 14.262064 [I 2022-10-27 03:55:15] ...trap/engines/engine.py.224: train: epoch 46 | batch 1210/2304 [I 2022-10-27 03:55:15] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:09 [I 2022-10-27 03:55:15] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00246 [I 2022-10-27 03:55:15] ...trap/engines/engine.py.230: loss: 0.19154 [I 2022-10-27 03:55:15] ...optimizers/trijoint.py.151: total_norm: 14.900125 [I 2022-10-27 03:55:21] ...trap/engines/engine.py.224: train: epoch 46 | batch 1220/2304 [I 2022-10-27 03:55:21] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:02 [I 2022-10-27 03:55:21] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00250 [I 2022-10-27 03:55:21] ...trap/engines/engine.py.230: loss: 0.15669 [I 2022-10-27 03:55:21] ...optimizers/trijoint.py.151: total_norm: 17.752598 [I 2022-10-27 03:55:27] ...trap/engines/engine.py.224: train: epoch 46 | batch 1230/2304 [I 2022-10-27 03:55:27] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:00 [I 2022-10-27 03:55:27] ...trap/engines/engine.py.229: process: 0.56223 | load: 0.00241 [I 2022-10-27 03:55:27] ...trap/engines/engine.py.230: loss: 0.13088 [I 2022-10-27 03:55:27] ...optimizers/trijoint.py.151: total_norm: 10.919782 [I 2022-10-27 03:55:32] ...trap/engines/engine.py.224: train: epoch 46 | batch 1240/2304 [I 2022-10-27 03:55:32] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:47 [I 2022-10-27 03:55:32] ...trap/engines/engine.py.229: process: 0.53757 | load: 0.00247 [I 2022-10-27 03:55:32] ...trap/engines/engine.py.230: loss: 0.11326 [I 2022-10-27 03:55:32] ...optimizers/trijoint.py.151: total_norm: 7.398765 [I 2022-10-27 03:55:38] ...trap/engines/engine.py.224: train: epoch 46 | batch 1250/2304 [I 2022-10-27 03:55:38] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:44 [I 2022-10-27 03:55:38] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00246 [I 2022-10-27 03:55:38] ...trap/engines/engine.py.230: loss: 0.21793 [I 2022-10-27 03:55:38] ...optimizers/trijoint.py.151: total_norm: 18.414478 [I 2022-10-27 03:55:43] ...trap/engines/engine.py.224: train: epoch 46 | batch 1260/2304 [I 2022-10-27 03:55:43] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:37 [I 2022-10-27 03:55:43] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00246 [I 2022-10-27 03:55:43] ...trap/engines/engine.py.230: loss: 0.16508 [I 2022-10-27 03:55:43] ...optimizers/trijoint.py.151: total_norm: 21.808867 [I 2022-10-27 03:55:49] ...trap/engines/engine.py.224: train: epoch 46 | batch 1270/2304 [I 2022-10-27 03:55:49] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:55 [I 2022-10-27 03:55:49] ...trap/engines/engine.py.229: process: 0.61495 | load: 0.00260 [I 2022-10-27 03:55:49] ...trap/engines/engine.py.230: loss: 0.17275 [I 2022-10-27 03:55:49] ...optimizers/trijoint.py.151: total_norm: 18.809175 [I 2022-10-27 03:55:54] ...trap/engines/engine.py.224: train: epoch 46 | batch 1280/2304 [I 2022-10-27 03:55:54] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:29 [I 2022-10-27 03:55:54] ...trap/engines/engine.py.229: process: 0.53739 | load: 0.00243 [I 2022-10-27 03:55:54] ...trap/engines/engine.py.230: loss: 0.14893 [I 2022-10-27 03:55:54] ...optimizers/trijoint.py.151: total_norm: 16.388426 [I 2022-10-27 03:56:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 1290/2304 [I 2022-10-27 03:56:00] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:34 [I 2022-10-27 03:56:00] ...trap/engines/engine.py.229: process: 0.55099 | load: 0.00266 [I 2022-10-27 03:56:00] ...trap/engines/engine.py.230: loss: 0.14476 [I 2022-10-27 03:56:00] ...optimizers/trijoint.py.151: total_norm: 17.882559 [I 2022-10-27 03:56:06] ...trap/engines/engine.py.224: train: epoch 46 | batch 1300/2304 [I 2022-10-27 03:56:06] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:18 [I 2022-10-27 03:56:06] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00265 [I 2022-10-27 03:56:06] ...trap/engines/engine.py.230: loss: 0.16622 [I 2022-10-27 03:56:06] ...optimizers/trijoint.py.151: total_norm: 45.589237 [I 2022-10-27 03:56:11] ...trap/engines/engine.py.224: train: epoch 46 | batch 1310/2304 [I 2022-10-27 03:56:11] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:15 [I 2022-10-27 03:56:11] ...trap/engines/engine.py.229: process: 0.56534 | load: 0.00246 [I 2022-10-27 03:56:11] ...trap/engines/engine.py.230: loss: 0.12592 [I 2022-10-27 03:56:11] ...optimizers/trijoint.py.151: total_norm: 27.936798 [I 2022-10-27 03:56:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 1320/2304 [I 2022-10-27 03:56:17] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:07 [I 2022-10-27 03:56:17] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00246 [I 2022-10-27 03:56:17] ...trap/engines/engine.py.230: loss: 0.11115 [I 2022-10-27 03:56:17] ...optimizers/trijoint.py.151: total_norm: 8.137931 [I 2022-10-27 03:56:22] ...trap/engines/engine.py.224: train: epoch 46 | batch 1330/2304 [I 2022-10-27 03:56:22] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:01 [I 2022-10-27 03:56:22] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00244 [I 2022-10-27 03:56:22] ...trap/engines/engine.py.230: loss: 0.20414 [I 2022-10-27 03:56:22] ...optimizers/trijoint.py.151: total_norm: 29.196569 [I 2022-10-27 03:56:28] ...trap/engines/engine.py.224: train: epoch 46 | batch 1340/2304 [I 2022-10-27 03:56:28] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:55 [I 2022-10-27 03:56:28] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00244 [I 2022-10-27 03:56:28] ...trap/engines/engine.py.230: loss: 0.18392 [I 2022-10-27 03:56:28] ...optimizers/trijoint.py.151: total_norm: 14.970863 [I 2022-10-27 03:56:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 1350/2304 [I 2022-10-27 03:56:34] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:51 [I 2022-10-27 03:56:34] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00246 [I 2022-10-27 03:56:34] ...trap/engines/engine.py.230: loss: 0.12995 [I 2022-10-27 03:56:34] ...optimizers/trijoint.py.151: total_norm: 9.202128 [I 2022-10-27 03:56:39] ...trap/engines/engine.py.224: train: epoch 46 | batch 1360/2304 [I 2022-10-27 03:56:39] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:43 [I 2022-10-27 03:56:39] ...trap/engines/engine.py.229: process: 0.55204 | load: 0.00241 [I 2022-10-27 03:56:39] ...trap/engines/engine.py.230: loss: 0.14784 [I 2022-10-27 03:56:39] ...optimizers/trijoint.py.151: total_norm: 11.229879 [I 2022-10-27 03:56:45] ...trap/engines/engine.py.224: train: epoch 46 | batch 1370/2304 [I 2022-10-27 03:56:45] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:44 [I 2022-10-27 03:56:45] ...trap/engines/engine.py.229: process: 0.55152 | load: 0.00239 [I 2022-10-27 03:56:45] ...trap/engines/engine.py.230: loss: 0.15292 [I 2022-10-27 03:56:45] ...optimizers/trijoint.py.151: total_norm: 23.504351 [I 2022-10-27 03:56:50] ...trap/engines/engine.py.224: train: epoch 46 | batch 1380/2304 [I 2022-10-27 03:56:50] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:35 [I 2022-10-27 03:56:50] ...trap/engines/engine.py.229: process: 0.56450 | load: 0.00243 [I 2022-10-27 03:56:50] ...trap/engines/engine.py.230: loss: 0.21785 [I 2022-10-27 03:56:50] ...optimizers/trijoint.py.151: total_norm: 13.984472 [I 2022-10-27 03:56:56] ...trap/engines/engine.py.224: train: epoch 46 | batch 1390/2304 [I 2022-10-27 03:56:56] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:29 [I 2022-10-27 03:56:56] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00240 [I 2022-10-27 03:56:56] ...trap/engines/engine.py.230: loss: 0.16188 [I 2022-10-27 03:56:56] ...optimizers/trijoint.py.151: total_norm: 11.898565 [I 2022-10-27 03:57:02] ...trap/engines/engine.py.224: train: epoch 46 | batch 1400/2304 [I 2022-10-27 03:57:02] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:22 [I 2022-10-27 03:57:02] ...trap/engines/engine.py.229: process: 0.54882 | load: 0.00253 [I 2022-10-27 03:57:02] ...trap/engines/engine.py.230: loss: 0.13392 [I 2022-10-27 03:57:02] ...optimizers/trijoint.py.151: total_norm: 17.809010 [I 2022-10-27 03:57:07] ...trap/engines/engine.py.224: train: epoch 46 | batch 1410/2304 [I 2022-10-27 03:57:07] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:17 [I 2022-10-27 03:57:07] ...trap/engines/engine.py.229: process: 0.56576 | load: 0.00248 [I 2022-10-27 03:57:07] ...trap/engines/engine.py.230: loss: 0.18501 [I 2022-10-27 03:57:07] ...optimizers/trijoint.py.151: total_norm: 18.067570 [I 2022-10-27 03:57:13] ...trap/engines/engine.py.224: train: epoch 46 | batch 1420/2304 [I 2022-10-27 03:57:13] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:10 [I 2022-10-27 03:57:13] ...trap/engines/engine.py.229: process: 0.54167 | load: 0.00246 [I 2022-10-27 03:57:13] ...trap/engines/engine.py.230: loss: 0.18670 [I 2022-10-27 03:57:13] ...optimizers/trijoint.py.151: total_norm: 8.550818 [I 2022-10-27 03:57:18] ...trap/engines/engine.py.224: train: epoch 46 | batch 1430/2304 [I 2022-10-27 03:57:18] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:10 [I 2022-10-27 03:57:18] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00243 [I 2022-10-27 03:57:18] ...trap/engines/engine.py.230: loss: 0.14695 [I 2022-10-27 03:57:18] ...optimizers/trijoint.py.151: total_norm: 13.506839 [I 2022-10-27 03:57:24] ...trap/engines/engine.py.224: train: epoch 46 | batch 1440/2304 [I 2022-10-27 03:57:24] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:02 [I 2022-10-27 03:57:24] ...trap/engines/engine.py.229: process: 0.56535 | load: 0.00242 [I 2022-10-27 03:57:24] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-27 03:57:24] ...optimizers/trijoint.py.151: total_norm: 21.777506 [I 2022-10-27 03:57:30] ...trap/engines/engine.py.224: train: epoch 46 | batch 1450/2304 [I 2022-10-27 03:57:30] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:53 [I 2022-10-27 03:57:30] ...trap/engines/engine.py.229: process: 0.54201 | load: 0.00244 [I 2022-10-27 03:57:30] ...trap/engines/engine.py.230: loss: 0.15221 [I 2022-10-27 03:57:30] ...optimizers/trijoint.py.151: total_norm: 19.421148 [I 2022-10-27 03:57:35] ...trap/engines/engine.py.224: train: epoch 46 | batch 1460/2304 [I 2022-10-27 03:57:35] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:41 [I 2022-10-27 03:57:35] ...trap/engines/engine.py.229: process: 0.54519 | load: 0.00244 [I 2022-10-27 03:57:35] ...trap/engines/engine.py.230: loss: 0.09906 [I 2022-10-27 03:57:35] ...optimizers/trijoint.py.151: total_norm: 19.603191 [I 2022-10-27 03:57:41] ...trap/engines/engine.py.224: train: epoch 46 | batch 1470/2304 [I 2022-10-27 03:57:41] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:42 [I 2022-10-27 03:57:41] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00247 [I 2022-10-27 03:57:41] ...trap/engines/engine.py.230: loss: 0.32366 [I 2022-10-27 03:57:41] ...optimizers/trijoint.py.151: total_norm: 19.322275 [I 2022-10-27 03:57:46] ...trap/engines/engine.py.224: train: epoch 46 | batch 1480/2304 [I 2022-10-27 03:57:46] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:35 [I 2022-10-27 03:57:46] ...trap/engines/engine.py.229: process: 0.54623 | load: 0.00248 [I 2022-10-27 03:57:46] ...trap/engines/engine.py.230: loss: 0.15702 [I 2022-10-27 03:57:46] ...optimizers/trijoint.py.151: total_norm: 9.896242 [I 2022-10-27 03:57:52] ...trap/engines/engine.py.224: train: epoch 46 | batch 1490/2304 [I 2022-10-27 03:57:52] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:31 [I 2022-10-27 03:57:52] ...trap/engines/engine.py.229: process: 0.54285 | load: 0.00253 [I 2022-10-27 03:57:52] ...trap/engines/engine.py.230: loss: 0.19033 [I 2022-10-27 03:57:52] ...optimizers/trijoint.py.151: total_norm: 26.071705 [I 2022-10-27 03:57:57] ...trap/engines/engine.py.224: train: epoch 46 | batch 1500/2304 [I 2022-10-27 03:57:57] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:24 [I 2022-10-27 03:57:57] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00246 [I 2022-10-27 03:57:57] ...trap/engines/engine.py.230: loss: 0.16793 [I 2022-10-27 03:57:57] ...optimizers/trijoint.py.151: total_norm: 21.222292 [I 2022-10-27 03:58:03] ...trap/engines/engine.py.224: train: epoch 46 | batch 1510/2304 [I 2022-10-27 03:58:03] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:20 [I 2022-10-27 03:58:03] ...trap/engines/engine.py.229: process: 0.54592 | load: 0.00246 [I 2022-10-27 03:58:03] ...trap/engines/engine.py.230: loss: 0.11662 [I 2022-10-27 03:58:03] ...optimizers/trijoint.py.151: total_norm: 7.650623 [I 2022-10-27 03:58:08] ...trap/engines/engine.py.224: train: epoch 46 | batch 1520/2304 [I 2022-10-27 03:58:08] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:16 [I 2022-10-27 03:58:08] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00243 [I 2022-10-27 03:58:08] ...trap/engines/engine.py.230: loss: 0.15699 [I 2022-10-27 03:58:08] ...optimizers/trijoint.py.151: total_norm: 15.224534 [I 2022-10-27 03:58:14] ...trap/engines/engine.py.224: train: epoch 46 | batch 1530/2304 [I 2022-10-27 03:58:14] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:09 [I 2022-10-27 03:58:14] ...trap/engines/engine.py.229: process: 0.54870 | load: 0.00241 [I 2022-10-27 03:58:14] ...trap/engines/engine.py.230: loss: 0.18590 [I 2022-10-27 03:58:14] ...optimizers/trijoint.py.151: total_norm: 21.085556 [I 2022-10-27 03:58:20] ...trap/engines/engine.py.224: train: epoch 46 | batch 1540/2304 [I 2022-10-27 03:58:20] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:05 [I 2022-10-27 03:58:20] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00244 [I 2022-10-27 03:58:20] ...trap/engines/engine.py.230: loss: 0.14198 [I 2022-10-27 03:58:20] ...optimizers/trijoint.py.151: total_norm: 9.787806 [I 2022-10-27 03:58:25] ...trap/engines/engine.py.224: train: epoch 46 | batch 1550/2304 [I 2022-10-27 03:58:25] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:02 [I 2022-10-27 03:58:25] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00240 [I 2022-10-27 03:58:25] ...trap/engines/engine.py.230: loss: 0.19112 [I 2022-10-27 03:58:25] ...optimizers/trijoint.py.151: total_norm: 10.908174 [I 2022-10-27 03:58:31] ...trap/engines/engine.py.224: train: epoch 46 | batch 1560/2304 [I 2022-10-27 03:58:31] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:06:54 [I 2022-10-27 03:58:31] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00243 [I 2022-10-27 03:58:31] ...trap/engines/engine.py.230: loss: 0.12081 [I 2022-10-27 03:58:31] ...optimizers/trijoint.py.151: total_norm: 11.773115 [I 2022-10-27 03:58:36] ...trap/engines/engine.py.224: train: epoch 46 | batch 1570/2304 [I 2022-10-27 03:58:36] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:48 [I 2022-10-27 03:58:36] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00247 [I 2022-10-27 03:58:36] ...trap/engines/engine.py.230: loss: 0.15504 [I 2022-10-27 03:58:36] ...optimizers/trijoint.py.151: total_norm: 17.125881 [I 2022-10-27 03:58:42] ...trap/engines/engine.py.224: train: epoch 46 | batch 1580/2304 [I 2022-10-27 03:58:42] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:41 [I 2022-10-27 03:58:42] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00242 [I 2022-10-27 03:58:42] ...trap/engines/engine.py.230: loss: 0.14693 [I 2022-10-27 03:58:42] ...optimizers/trijoint.py.151: total_norm: 27.673248 [I 2022-10-27 03:58:48] ...trap/engines/engine.py.224: train: epoch 46 | batch 1590/2304 [I 2022-10-27 03:58:48] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:37 [I 2022-10-27 03:58:48] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00242 [I 2022-10-27 03:58:48] ...trap/engines/engine.py.230: loss: 0.14866 [I 2022-10-27 03:58:48] ...optimizers/trijoint.py.151: total_norm: 16.430412 [I 2022-10-27 03:58:53] ...trap/engines/engine.py.224: train: epoch 46 | batch 1600/2304 [I 2022-10-27 03:58:53] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:32 [I 2022-10-27 03:58:53] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00239 [I 2022-10-27 03:58:53] ...trap/engines/engine.py.230: loss: 0.12363 [I 2022-10-27 03:58:53] ...optimizers/trijoint.py.151: total_norm: 12.662384 [I 2022-10-27 03:58:59] ...trap/engines/engine.py.224: train: epoch 46 | batch 1610/2304 [I 2022-10-27 03:58:59] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:26 [I 2022-10-27 03:58:59] ...trap/engines/engine.py.229: process: 0.55267 | load: 0.00247 [I 2022-10-27 03:58:59] ...trap/engines/engine.py.230: loss: 0.14022 [I 2022-10-27 03:58:59] ...optimizers/trijoint.py.151: total_norm: 18.586348 [I 2022-10-27 03:59:04] ...trap/engines/engine.py.224: train: epoch 46 | batch 1620/2304 [I 2022-10-27 03:59:04] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:18 [I 2022-10-27 03:59:04] ...trap/engines/engine.py.229: process: 0.54777 | load: 0.00247 [I 2022-10-27 03:59:04] ...trap/engines/engine.py.230: loss: 0.16052 [I 2022-10-27 03:59:04] ...optimizers/trijoint.py.151: total_norm: 16.400188 [I 2022-10-27 03:59:10] ...trap/engines/engine.py.224: train: epoch 46 | batch 1630/2304 [I 2022-10-27 03:59:10] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:13 [I 2022-10-27 03:59:10] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00242 [I 2022-10-27 03:59:10] ...trap/engines/engine.py.230: loss: 0.17366 [I 2022-10-27 03:59:10] ...optimizers/trijoint.py.151: total_norm: 10.845243 [I 2022-10-27 03:59:15] ...trap/engines/engine.py.224: train: epoch 46 | batch 1640/2304 [I 2022-10-27 03:59:15] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:09 [I 2022-10-27 03:59:15] ...trap/engines/engine.py.229: process: 0.54637 | load: 0.00244 [I 2022-10-27 03:59:15] ...trap/engines/engine.py.230: loss: 0.09896 [I 2022-10-27 03:59:15] ...optimizers/trijoint.py.151: total_norm: 8.439720 [I 2022-10-27 03:59:21] ...trap/engines/engine.py.224: train: epoch 46 | batch 1650/2304 [I 2022-10-27 03:59:21] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:02 [I 2022-10-27 03:59:21] ...trap/engines/engine.py.229: process: 0.55177 | load: 0.00242 [I 2022-10-27 03:59:21] ...trap/engines/engine.py.230: loss: 0.11271 [I 2022-10-27 03:59:21] ...optimizers/trijoint.py.151: total_norm: 16.302176 [I 2022-10-27 03:59:27] ...trap/engines/engine.py.224: train: epoch 46 | batch 1660/2304 [I 2022-10-27 03:59:27] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:09 [I 2022-10-27 03:59:27] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00245 [I 2022-10-27 03:59:27] ...trap/engines/engine.py.230: loss: 0.18522 [I 2022-10-27 03:59:27] ...optimizers/trijoint.py.151: total_norm: 18.282398 [I 2022-10-27 03:59:32] ...trap/engines/engine.py.224: train: epoch 46 | batch 1670/2304 [I 2022-10-27 03:59:32] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:05:53 [I 2022-10-27 03:59:32] ...trap/engines/engine.py.229: process: 0.55229 | load: 0.00245 [I 2022-10-27 03:59:32] ...trap/engines/engine.py.230: loss: 0.22613 [I 2022-10-27 03:59:32] ...optimizers/trijoint.py.151: total_norm: 22.103266 [I 2022-10-27 03:59:38] ...trap/engines/engine.py.224: train: epoch 46 | batch 1680/2304 [I 2022-10-27 03:59:38] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:47 [I 2022-10-27 03:59:38] ...trap/engines/engine.py.229: process: 0.56544 | load: 0.00247 [I 2022-10-27 03:59:38] ...trap/engines/engine.py.230: loss: 0.20609 [I 2022-10-27 03:59:38] ...optimizers/trijoint.py.151: total_norm: 16.846371 [I 2022-10-27 03:59:43] ...trap/engines/engine.py.224: train: epoch 46 | batch 1690/2304 [I 2022-10-27 03:59:43] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:43 [I 2022-10-27 03:59:43] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00256 [I 2022-10-27 03:59:43] ...trap/engines/engine.py.230: loss: 0.12635 [I 2022-10-27 03:59:43] ...optimizers/trijoint.py.151: total_norm: 12.940578 [I 2022-10-27 03:59:49] ...trap/engines/engine.py.224: train: epoch 46 | batch 1700/2304 [I 2022-10-27 03:59:49] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:35 [I 2022-10-27 03:59:49] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00249 [I 2022-10-27 03:59:49] ...trap/engines/engine.py.230: loss: 0.10237 [I 2022-10-27 03:59:49] ...optimizers/trijoint.py.151: total_norm: 10.711648 [I 2022-10-27 03:59:55] ...trap/engines/engine.py.224: train: epoch 46 | batch 1710/2304 [I 2022-10-27 03:59:55] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:32 [I 2022-10-27 03:59:55] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00248 [I 2022-10-27 03:59:55] ...trap/engines/engine.py.230: loss: 0.20139 [I 2022-10-27 03:59:55] ...optimizers/trijoint.py.151: total_norm: 13.544003 [I 2022-10-27 04:00:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 1720/2304 [I 2022-10-27 04:00:00] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:25 [I 2022-10-27 04:00:00] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00243 [I 2022-10-27 04:00:00] ...trap/engines/engine.py.230: loss: 0.15506 [I 2022-10-27 04:00:00] ...optimizers/trijoint.py.151: total_norm: 18.872166 [I 2022-10-27 04:00:06] ...trap/engines/engine.py.224: train: epoch 46 | batch 1730/2304 [I 2022-10-27 04:00:06] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:17 [I 2022-10-27 04:00:06] ...trap/engines/engine.py.229: process: 0.55623 | load: 0.00245 [I 2022-10-27 04:00:06] ...trap/engines/engine.py.230: loss: 0.17805 [I 2022-10-27 04:00:06] ...optimizers/trijoint.py.151: total_norm: 18.963717 [I 2022-10-27 04:00:11] ...trap/engines/engine.py.224: train: epoch 46 | batch 1740/2304 [I 2022-10-27 04:00:11] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:14 [I 2022-10-27 04:00:11] ...trap/engines/engine.py.229: process: 0.56134 | load: 0.00245 [I 2022-10-27 04:00:11] ...trap/engines/engine.py.230: loss: 0.16371 [I 2022-10-27 04:00:11] ...optimizers/trijoint.py.151: total_norm: 10.351077 [I 2022-10-27 04:00:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 1750/2304 [I 2022-10-27 04:00:17] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:07 [I 2022-10-27 04:00:17] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00262 [I 2022-10-27 04:00:17] ...trap/engines/engine.py.230: loss: 0.12068 [I 2022-10-27 04:00:17] ...optimizers/trijoint.py.151: total_norm: 13.391779 [I 2022-10-27 04:00:22] ...trap/engines/engine.py.224: train: epoch 46 | batch 1760/2304 [I 2022-10-27 04:00:22] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:01 [I 2022-10-27 04:00:22] ...trap/engines/engine.py.229: process: 0.55013 | load: 0.00246 [I 2022-10-27 04:00:22] ...trap/engines/engine.py.230: loss: 0.15442 [I 2022-10-27 04:00:22] ...optimizers/trijoint.py.151: total_norm: 20.179306 [I 2022-10-27 04:00:28] ...trap/engines/engine.py.224: train: epoch 46 | batch 1770/2304 [I 2022-10-27 04:00:28] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:05:07 [I 2022-10-27 04:00:28] ...trap/engines/engine.py.229: process: 0.56716 | load: 0.00241 [I 2022-10-27 04:00:28] ...trap/engines/engine.py.230: loss: 0.31661 [I 2022-10-27 04:00:28] ...optimizers/trijoint.py.151: total_norm: 21.506369 [I 2022-10-27 04:00:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 1780/2304 [I 2022-10-27 04:00:34] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:04:54 [I 2022-10-27 04:00:34] ...trap/engines/engine.py.229: process: 0.57583 | load: 0.00244 [I 2022-10-27 04:00:34] ...trap/engines/engine.py.230: loss: 0.11757 [I 2022-10-27 04:00:34] ...optimizers/trijoint.py.151: total_norm: 14.795401 [I 2022-10-27 04:00:39] ...trap/engines/engine.py.224: train: epoch 46 | batch 1790/2304 [I 2022-10-27 04:00:39] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:04:44 [I 2022-10-27 04:00:39] ...trap/engines/engine.py.229: process: 0.54599 | load: 0.00253 [I 2022-10-27 04:00:39] ...trap/engines/engine.py.230: loss: 0.11872 [I 2022-10-27 04:00:39] ...optimizers/trijoint.py.151: total_norm: 21.013088 [I 2022-10-27 04:00:45] ...trap/engines/engine.py.224: train: epoch 46 | batch 1800/2304 [I 2022-10-27 04:00:45] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:04:41 [I 2022-10-27 04:00:45] ...trap/engines/engine.py.229: process: 0.56599 | load: 0.00253 [I 2022-10-27 04:00:45] ...trap/engines/engine.py.230: loss: 0.19926 [I 2022-10-27 04:00:45] ...optimizers/trijoint.py.151: total_norm: 22.651834 [I 2022-10-27 04:00:50] ...trap/engines/engine.py.224: train: epoch 46 | batch 1810/2304 [I 2022-10-27 04:00:50] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:33 [I 2022-10-27 04:00:50] ...trap/engines/engine.py.229: process: 0.54385 | load: 0.00249 [I 2022-10-27 04:00:50] ...trap/engines/engine.py.230: loss: 0.17246 [I 2022-10-27 04:00:50] ...optimizers/trijoint.py.151: total_norm: 15.123329 [I 2022-10-27 04:00:56] ...trap/engines/engine.py.224: train: epoch 46 | batch 1820/2304 [I 2022-10-27 04:00:56] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:29 [I 2022-10-27 04:00:56] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00243 [I 2022-10-27 04:00:56] ...trap/engines/engine.py.230: loss: 0.14896 [I 2022-10-27 04:00:56] ...optimizers/trijoint.py.151: total_norm: 14.674053 [I 2022-10-27 04:01:02] ...trap/engines/engine.py.224: train: epoch 46 | batch 1830/2304 [I 2022-10-27 04:01:02] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:24 [I 2022-10-27 04:01:02] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00246 [I 2022-10-27 04:01:02] ...trap/engines/engine.py.230: loss: 0.13897 [I 2022-10-27 04:01:02] ...optimizers/trijoint.py.151: total_norm: 8.720452 [I 2022-10-27 04:01:07] ...trap/engines/engine.py.224: train: epoch 46 | batch 1840/2304 [I 2022-10-27 04:01:07] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:17 [I 2022-10-27 04:01:07] ...trap/engines/engine.py.229: process: 0.55025 | load: 0.00246 [I 2022-10-27 04:01:07] ...trap/engines/engine.py.230: loss: 0.16904 [I 2022-10-27 04:01:07] ...optimizers/trijoint.py.151: total_norm: 13.274937 [I 2022-10-27 04:01:13] ...trap/engines/engine.py.224: train: epoch 46 | batch 1850/2304 [I 2022-10-27 04:01:13] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:12 [I 2022-10-27 04:01:13] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00243 [I 2022-10-27 04:01:13] ...trap/engines/engine.py.230: loss: 0.23565 [I 2022-10-27 04:01:13] ...optimizers/trijoint.py.151: total_norm: 21.099953 [I 2022-10-27 04:01:18] ...trap/engines/engine.py.224: train: epoch 46 | batch 1860/2304 [I 2022-10-27 04:01:18] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:06 [I 2022-10-27 04:01:18] ...trap/engines/engine.py.229: process: 0.56312 | load: 0.00248 [I 2022-10-27 04:01:18] ...trap/engines/engine.py.230: loss: 0.12301 [I 2022-10-27 04:01:18] ...optimizers/trijoint.py.151: total_norm: 11.389294 [I 2022-10-27 04:01:24] ...trap/engines/engine.py.224: train: epoch 46 | batch 1870/2304 [I 2022-10-27 04:01:24] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:01 [I 2022-10-27 04:01:24] ...trap/engines/engine.py.229: process: 0.55036 | load: 0.00248 [I 2022-10-27 04:01:24] ...trap/engines/engine.py.230: loss: 0.23637 [I 2022-10-27 04:01:24] ...optimizers/trijoint.py.151: total_norm: 13.619505 [I 2022-10-27 04:01:29] ...trap/engines/engine.py.224: train: epoch 46 | batch 1880/2304 [I 2022-10-27 04:01:29] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:03:57 [I 2022-10-27 04:01:29] ...trap/engines/engine.py.229: process: 0.55679 | load: 0.00249 [I 2022-10-27 04:01:29] ...trap/engines/engine.py.230: loss: 0.16823 [I 2022-10-27 04:01:29] ...optimizers/trijoint.py.151: total_norm: 11.081709 [I 2022-10-27 04:01:35] ...trap/engines/engine.py.224: train: epoch 46 | batch 1890/2304 [I 2022-10-27 04:01:35] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:03:51 [I 2022-10-27 04:01:35] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00242 [I 2022-10-27 04:01:35] ...trap/engines/engine.py.230: loss: 0.15942 [I 2022-10-27 04:01:35] ...optimizers/trijoint.py.151: total_norm: 15.454742 [I 2022-10-27 04:01:41] ...trap/engines/engine.py.224: train: epoch 46 | batch 1900/2304 [I 2022-10-27 04:01:41] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:03:44 [I 2022-10-27 04:01:41] ...trap/engines/engine.py.229: process: 0.55767 | load: 0.00243 [I 2022-10-27 04:01:41] ...trap/engines/engine.py.230: loss: 0.19794 [I 2022-10-27 04:01:41] ...optimizers/trijoint.py.151: total_norm: 22.619774 [I 2022-10-27 04:01:46] ...trap/engines/engine.py.224: train: epoch 46 | batch 1910/2304 [I 2022-10-27 04:01:46] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:03:39 [I 2022-10-27 04:01:46] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00244 [I 2022-10-27 04:01:46] ...trap/engines/engine.py.230: loss: 0.14351 [I 2022-10-27 04:01:46] ...optimizers/trijoint.py.151: total_norm: 10.756970 [I 2022-10-27 04:01:52] ...trap/engines/engine.py.224: train: epoch 46 | batch 1920/2304 [I 2022-10-27 04:01:52] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:35 [I 2022-10-27 04:01:52] ...trap/engines/engine.py.229: process: 0.56705 | load: 0.00241 [I 2022-10-27 04:01:52] ...trap/engines/engine.py.230: loss: 0.19515 [I 2022-10-27 04:01:52] ...optimizers/trijoint.py.151: total_norm: 24.104815 [I 2022-10-27 04:01:57] ...trap/engines/engine.py.224: train: epoch 46 | batch 1930/2304 [I 2022-10-27 04:01:57] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:28 [I 2022-10-27 04:01:57] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00243 [I 2022-10-27 04:01:57] ...trap/engines/engine.py.230: loss: 0.14980 [I 2022-10-27 04:01:57] ...optimizers/trijoint.py.151: total_norm: 14.621790 [I 2022-10-27 04:02:03] ...trap/engines/engine.py.224: train: epoch 46 | batch 1940/2304 [I 2022-10-27 04:02:03] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:23 [I 2022-10-27 04:02:03] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00239 [I 2022-10-27 04:02:03] ...trap/engines/engine.py.230: loss: 0.11826 [I 2022-10-27 04:02:03] ...optimizers/trijoint.py.151: total_norm: 16.396544 [I 2022-10-27 04:02:09] ...trap/engines/engine.py.224: train: epoch 46 | batch 1950/2304 [I 2022-10-27 04:02:09] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:16 [I 2022-10-27 04:02:09] ...trap/engines/engine.py.229: process: 0.54697 | load: 0.00243 [I 2022-10-27 04:02:09] ...trap/engines/engine.py.230: loss: 0.18486 [I 2022-10-27 04:02:09] ...optimizers/trijoint.py.151: total_norm: 15.548596 [I 2022-10-27 04:02:14] ...trap/engines/engine.py.224: train: epoch 46 | batch 1960/2304 [I 2022-10-27 04:02:14] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:10 [I 2022-10-27 04:02:14] ...trap/engines/engine.py.229: process: 0.55157 | load: 0.00248 [I 2022-10-27 04:02:14] ...trap/engines/engine.py.230: loss: 0.16130 [I 2022-10-27 04:02:14] ...optimizers/trijoint.py.151: total_norm: 15.040301 [I 2022-10-27 04:02:20] ...trap/engines/engine.py.224: train: epoch 46 | batch 1970/2304 [I 2022-10-27 04:02:20] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:04 [I 2022-10-27 04:02:20] ...trap/engines/engine.py.229: process: 0.55194 | load: 0.00248 [I 2022-10-27 04:02:20] ...trap/engines/engine.py.230: loss: 0.24016 [I 2022-10-27 04:02:20] ...optimizers/trijoint.py.151: total_norm: 14.113820 [I 2022-10-27 04:02:25] ...trap/engines/engine.py.224: train: epoch 46 | batch 1980/2304 [I 2022-10-27 04:02:25] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:01 [I 2022-10-27 04:02:25] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00244 [I 2022-10-27 04:02:25] ...trap/engines/engine.py.230: loss: 0.15532 [I 2022-10-27 04:02:25] ...optimizers/trijoint.py.151: total_norm: 11.439336 [I 2022-10-27 04:02:31] ...trap/engines/engine.py.224: train: epoch 46 | batch 1990/2304 [I 2022-10-27 04:02:31] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:02:54 [I 2022-10-27 04:02:31] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00243 [I 2022-10-27 04:02:31] ...trap/engines/engine.py.230: loss: 0.15414 [I 2022-10-27 04:02:31] ...optimizers/trijoint.py.151: total_norm: 19.592199 [I 2022-10-27 04:02:36] ...trap/engines/engine.py.224: train: epoch 46 | batch 2000/2304 [I 2022-10-27 04:02:36] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:02:48 [I 2022-10-27 04:02:36] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00246 [I 2022-10-27 04:02:36] ...trap/engines/engine.py.230: loss: 0.13118 [I 2022-10-27 04:02:36] ...optimizers/trijoint.py.151: total_norm: 9.752800 [I 2022-10-27 04:02:42] ...trap/engines/engine.py.224: train: epoch 46 | batch 2010/2304 [I 2022-10-27 04:02:42] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:02:43 [I 2022-10-27 04:02:42] ...trap/engines/engine.py.229: process: 0.55137 | load: 0.00243 [I 2022-10-27 04:02:42] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-27 04:02:42] ...optimizers/trijoint.py.151: total_norm: 26.001925 [I 2022-10-27 04:02:48] ...trap/engines/engine.py.224: train: epoch 46 | batch 2020/2304 [I 2022-10-27 04:02:48] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:37 [I 2022-10-27 04:02:48] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00246 [I 2022-10-27 04:02:48] ...trap/engines/engine.py.230: loss: 0.12275 [I 2022-10-27 04:02:48] ...optimizers/trijoint.py.151: total_norm: 15.250999 [I 2022-10-27 04:02:53] ...trap/engines/engine.py.224: train: epoch 46 | batch 2030/2304 [I 2022-10-27 04:02:53] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:32 [I 2022-10-27 04:02:53] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00242 [I 2022-10-27 04:02:53] ...trap/engines/engine.py.230: loss: 0.23575 [I 2022-10-27 04:02:53] ...optimizers/trijoint.py.151: total_norm: 32.970997 [I 2022-10-27 04:02:59] ...trap/engines/engine.py.224: train: epoch 46 | batch 2040/2304 [I 2022-10-27 04:02:59] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:26 [I 2022-10-27 04:02:59] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00255 [I 2022-10-27 04:02:59] ...trap/engines/engine.py.230: loss: 0.18959 [I 2022-10-27 04:02:59] ...optimizers/trijoint.py.151: total_norm: 20.761732 [I 2022-10-27 04:03:04] ...trap/engines/engine.py.224: train: epoch 46 | batch 2050/2304 [I 2022-10-27 04:03:04] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:21 [I 2022-10-27 04:03:04] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00242 [I 2022-10-27 04:03:04] ...trap/engines/engine.py.230: loss: 0.14390 [I 2022-10-27 04:03:04] ...optimizers/trijoint.py.151: total_norm: 19.897779 [I 2022-10-27 04:03:10] ...trap/engines/engine.py.224: train: epoch 46 | batch 2060/2304 [I 2022-10-27 04:03:10] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:15 [I 2022-10-27 04:03:10] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00245 [I 2022-10-27 04:03:10] ...trap/engines/engine.py.230: loss: 0.18315 [I 2022-10-27 04:03:10] ...optimizers/trijoint.py.151: total_norm: 15.811973 [I 2022-10-27 04:03:15] ...trap/engines/engine.py.224: train: epoch 46 | batch 2070/2304 [I 2022-10-27 04:03:15] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:09 [I 2022-10-27 04:03:15] ...trap/engines/engine.py.229: process: 0.54967 | load: 0.00242 [I 2022-10-27 04:03:15] ...trap/engines/engine.py.230: loss: 0.11259 [I 2022-10-27 04:03:15] ...optimizers/trijoint.py.151: total_norm: 6.860389 [I 2022-10-27 04:03:21] ...trap/engines/engine.py.224: train: epoch 46 | batch 2080/2304 [I 2022-10-27 04:03:21] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:04 [I 2022-10-27 04:03:21] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00239 [I 2022-10-27 04:03:21] ...trap/engines/engine.py.230: loss: 0.21509 [I 2022-10-27 04:03:21] ...optimizers/trijoint.py.151: total_norm: 27.166145 [I 2022-10-27 04:03:26] ...trap/engines/engine.py.224: train: epoch 46 | batch 2090/2304 [I 2022-10-27 04:03:26] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:01:58 [I 2022-10-27 04:03:26] ...trap/engines/engine.py.229: process: 0.55013 | load: 0.00243 [I 2022-10-27 04:03:26] ...trap/engines/engine.py.230: loss: 0.13884 [I 2022-10-27 04:03:26] ...optimizers/trijoint.py.151: total_norm: 22.760822 [I 2022-10-27 04:03:32] ...trap/engines/engine.py.224: train: epoch 46 | batch 2100/2304 [I 2022-10-27 04:03:32] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:01:53 [I 2022-10-27 04:03:32] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00241 [I 2022-10-27 04:03:32] ...trap/engines/engine.py.230: loss: 0.25758 [I 2022-10-27 04:03:32] ...optimizers/trijoint.py.151: total_norm: 20.637131 [I 2022-10-27 04:03:38] ...trap/engines/engine.py.224: train: epoch 46 | batch 2110/2304 [I 2022-10-27 04:03:38] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:01:46 [I 2022-10-27 04:03:38] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00243 [I 2022-10-27 04:03:38] ...trap/engines/engine.py.230: loss: 0.10272 [I 2022-10-27 04:03:38] ...optimizers/trijoint.py.151: total_norm: 9.619782 [I 2022-10-27 04:03:43] ...trap/engines/engine.py.224: train: epoch 46 | batch 2120/2304 [I 2022-10-27 04:03:43] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:01:42 [I 2022-10-27 04:03:43] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00256 [I 2022-10-27 04:03:43] ...trap/engines/engine.py.230: loss: 0.19486 [I 2022-10-27 04:03:43] ...optimizers/trijoint.py.151: total_norm: 29.110033 [I 2022-10-27 04:03:49] ...trap/engines/engine.py.224: train: epoch 46 | batch 2130/2304 [I 2022-10-27 04:03:49] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:36 [I 2022-10-27 04:03:49] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00240 [I 2022-10-27 04:03:49] ...trap/engines/engine.py.230: loss: 0.21193 [I 2022-10-27 04:03:49] ...optimizers/trijoint.py.151: total_norm: 16.119488 [I 2022-10-27 04:03:54] ...trap/engines/engine.py.224: train: epoch 46 | batch 2140/2304 [I 2022-10-27 04:03:54] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:31 [I 2022-10-27 04:03:54] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00282 [I 2022-10-27 04:03:54] ...trap/engines/engine.py.230: loss: 0.20079 [I 2022-10-27 04:03:54] ...optimizers/trijoint.py.151: total_norm: 22.568407 [I 2022-10-27 04:04:00] ...trap/engines/engine.py.224: train: epoch 46 | batch 2150/2304 [I 2022-10-27 04:04:00] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:26 [I 2022-10-27 04:04:00] ...trap/engines/engine.py.229: process: 0.56254 | load: 0.00256 [I 2022-10-27 04:04:00] ...trap/engines/engine.py.230: loss: 0.24458 [I 2022-10-27 04:04:00] ...optimizers/trijoint.py.151: total_norm: 25.978041 [I 2022-10-27 04:04:06] ...trap/engines/engine.py.224: train: epoch 46 | batch 2160/2304 [I 2022-10-27 04:04:06] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:20 [I 2022-10-27 04:04:06] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00248 [I 2022-10-27 04:04:06] ...trap/engines/engine.py.230: loss: 0.12405 [I 2022-10-27 04:04:06] ...optimizers/trijoint.py.151: total_norm: 10.268967 [I 2022-10-27 04:04:11] ...trap/engines/engine.py.224: train: epoch 46 | batch 2170/2304 [I 2022-10-27 04:04:11] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:14 [I 2022-10-27 04:04:11] ...trap/engines/engine.py.229: process: 0.53989 | load: 0.00246 [I 2022-10-27 04:04:11] ...trap/engines/engine.py.230: loss: 0.16856 [I 2022-10-27 04:04:11] ...optimizers/trijoint.py.151: total_norm: 54.774788 [I 2022-10-27 04:04:17] ...trap/engines/engine.py.224: train: epoch 46 | batch 2180/2304 [I 2022-10-27 04:04:17] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:09 [I 2022-10-27 04:04:17] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00244 [I 2022-10-27 04:04:17] ...trap/engines/engine.py.230: loss: 0.12743 [I 2022-10-27 04:04:17] ...optimizers/trijoint.py.151: total_norm: 13.312584 [I 2022-10-27 04:04:22] ...trap/engines/engine.py.224: train: epoch 46 | batch 2190/2304 [I 2022-10-27 04:04:22] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:03 [I 2022-10-27 04:04:22] ...trap/engines/engine.py.229: process: 0.55039 | load: 0.00245 [I 2022-10-27 04:04:22] ...trap/engines/engine.py.230: loss: 0.17813 [I 2022-10-27 04:04:22] ...optimizers/trijoint.py.151: total_norm: 15.142546 [I 2022-10-27 04:04:28] ...trap/engines/engine.py.224: train: epoch 46 | batch 2200/2304 [I 2022-10-27 04:04:28] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:00:58 [I 2022-10-27 04:04:28] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00243 [I 2022-10-27 04:04:28] ...trap/engines/engine.py.230: loss: 0.19420 [I 2022-10-27 04:04:28] ...optimizers/trijoint.py.151: total_norm: 23.394054 [I 2022-10-27 04:04:34] ...trap/engines/engine.py.224: train: epoch 46 | batch 2210/2304 [I 2022-10-27 04:04:34] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:00:52 [I 2022-10-27 04:04:34] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00247 [I 2022-10-27 04:04:34] ...trap/engines/engine.py.230: loss: 0.18682 [I 2022-10-27 04:04:34] ...optimizers/trijoint.py.151: total_norm: 13.739632 [I 2022-10-27 04:04:39] ...trap/engines/engine.py.224: train: epoch 46 | batch 2220/2304 [I 2022-10-27 04:04:39] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:00:46 [I 2022-10-27 04:04:39] ...trap/engines/engine.py.229: process: 0.54518 | load: 0.00242 [I 2022-10-27 04:04:39] ...trap/engines/engine.py.230: loss: 0.24822 [I 2022-10-27 04:04:39] ...optimizers/trijoint.py.151: total_norm: 42.797737 [I 2022-10-27 04:04:45] ...trap/engines/engine.py.224: train: epoch 46 | batch 2230/2304 [I 2022-10-27 04:04:45] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:41 [I 2022-10-27 04:04:45] ...trap/engines/engine.py.229: process: 0.57221 | load: 0.00245 [I 2022-10-27 04:04:45] ...trap/engines/engine.py.230: loss: 0.14809 [I 2022-10-27 04:04:45] ...optimizers/trijoint.py.151: total_norm: 15.356673 [I 2022-10-27 04:04:50] ...trap/engines/engine.py.224: train: epoch 46 | batch 2240/2304 [I 2022-10-27 04:04:50] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:35 [I 2022-10-27 04:04:50] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00260 [I 2022-10-27 04:04:50] ...trap/engines/engine.py.230: loss: 0.16508 [I 2022-10-27 04:04:50] ...optimizers/trijoint.py.151: total_norm: 24.156757 [I 2022-10-27 04:04:56] ...trap/engines/engine.py.224: train: epoch 46 | batch 2250/2304 [I 2022-10-27 04:04:56] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:29 [I 2022-10-27 04:04:56] ...trap/engines/engine.py.229: process: 0.57098 | load: 0.00249 [I 2022-10-27 04:04:56] ...trap/engines/engine.py.230: loss: 0.23853 [I 2022-10-27 04:04:56] ...optimizers/trijoint.py.151: total_norm: 27.443079 [I 2022-10-27 04:05:01] ...trap/engines/engine.py.224: train: epoch 46 | batch 2260/2304 [I 2022-10-27 04:05:01] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:24 [I 2022-10-27 04:05:01] ...trap/engines/engine.py.229: process: 0.54867 | load: 0.00248 [I 2022-10-27 04:05:01] ...trap/engines/engine.py.230: loss: 0.21152 [I 2022-10-27 04:05:01] ...optimizers/trijoint.py.151: total_norm: 21.896038 [I 2022-10-27 04:05:07] ...trap/engines/engine.py.224: train: epoch 46 | batch 2270/2304 [I 2022-10-27 04:05:07] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:19 [I 2022-10-27 04:05:07] ...trap/engines/engine.py.229: process: 0.56654 | load: 0.00243 [I 2022-10-27 04:05:07] ...trap/engines/engine.py.230: loss: 0.10028 [I 2022-10-27 04:05:07] ...optimizers/trijoint.py.151: total_norm: 9.462231 [I 2022-10-27 04:05:13] ...trap/engines/engine.py.224: train: epoch 46 | batch 2280/2304 [I 2022-10-27 04:05:13] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:13 [I 2022-10-27 04:05:13] ...trap/engines/engine.py.229: process: 0.55079 | load: 0.00243 [I 2022-10-27 04:05:13] ...trap/engines/engine.py.230: loss: 0.10597 [I 2022-10-27 04:05:13] ...optimizers/trijoint.py.151: total_norm: 10.683883 [I 2022-10-27 04:05:18] ...trap/engines/engine.py.224: train: epoch 46 | batch 2290/2304 [I 2022-10-27 04:05:18] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:07 [I 2022-10-27 04:05:18] ...trap/engines/engine.py.229: process: 0.56336 | load: 0.00243 [I 2022-10-27 04:05:18] ...trap/engines/engine.py.230: loss: 0.25542 [I 2022-10-27 04:05:18] ...optimizers/trijoint.py.151: total_norm: 14.251669 [I 2022-10-27 04:05:24] ...trap/engines/engine.py.224: train: epoch 46 | batch 2300/2304 [I 2022-10-27 04:05:24] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:02 [I 2022-10-27 04:05:24] ...trap/engines/engine.py.229: process: 0.54695 | load: 0.00015 [I 2022-10-27 04:05:24] ...trap/engines/engine.py.230: loss: 0.17581 [I 2022-10-27 04:05:24] ...optimizers/trijoint.py.151: total_norm: 23.951218 [I 2022-10-27 04:05:26] ...trap/engines/engine.py.224: train: epoch 46 | batch 2304/2304 [I 2022-10-27 04:05:26] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:00 [I 2022-10-27 04:05:26] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00015 [I 2022-10-27 04:05:26] ...trap/engines/engine.py.230: loss: 0.16457 [I 2022-10-27 04:05:26] ...optimizers/trijoint.py.151: total_norm: 27.107281 [S 2022-10-27 04:05:27] ...trap/engines/engine.py.129: train_epoch.epoch: 46 [S 2022-10-27 04:05:27] ...trap/engines/engine.py.129: train_epoch.loss: 0.16640649427884055 [I 2022-10-27 04:05:27] ...s/criterions/triplet.py.84: margin = 0.2850000000000002 [I 2022-10-27 04:05:28] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 46 [I 2022-10-27 04:05:28] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 04:05:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 04:05:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 04:05:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 04:05:31] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 04:05:32] ...trap/engines/engine.py.321: eval: epoch 46 | batch 0/510 [I 2022-10-27 04:05:32] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:32:27 [I 2022-10-27 04:05:32] ...trap/engines/engine.py.326: process: 3.81809 | load: 3.43329 [I 2022-10-27 04:05:35] ...trap/engines/engine.py.321: eval: epoch 46 | batch 10/510 [I 2022-10-27 04:05:35] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:05:53 [I 2022-10-27 04:05:35] ...trap/engines/engine.py.326: process: 0.33040 | load: 0.00017 [I 2022-10-27 04:05:38] ...trap/engines/engine.py.321: eval: epoch 46 | batch 20/510 [I 2022-10-27 04:05:38] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:01 [I 2022-10-27 04:05:38] ...trap/engines/engine.py.326: process: 0.32345 | load: 0.00020 [I 2022-10-27 04:05:42] ...trap/engines/engine.py.321: eval: epoch 46 | batch 30/510 [I 2022-10-27 04:05:42] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:42 [I 2022-10-27 04:05:42] ...trap/engines/engine.py.326: process: 0.33933 | load: 0.00018 [I 2022-10-27 04:05:45] ...trap/engines/engine.py.321: eval: epoch 46 | batch 40/510 [I 2022-10-27 04:05:45] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:35 [I 2022-10-27 04:05:45] ...trap/engines/engine.py.326: process: 0.33160 | load: 0.00022 [I 2022-10-27 04:05:48] ...trap/engines/engine.py.321: eval: epoch 46 | batch 50/510 [I 2022-10-27 04:05:48] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:33 [I 2022-10-27 04:05:48] ...trap/engines/engine.py.326: process: 0.33453 | load: 0.00022 [I 2022-10-27 04:05:52] ...trap/engines/engine.py.321: eval: epoch 46 | batch 60/510 [I 2022-10-27 04:05:52] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:33 [I 2022-10-27 04:05:52] ...trap/engines/engine.py.326: process: 0.33836 | load: 0.00019 [I 2022-10-27 04:05:55] ...trap/engines/engine.py.321: eval: epoch 46 | batch 70/510 [I 2022-10-27 04:05:55] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:27 [I 2022-10-27 04:05:55] ...trap/engines/engine.py.326: process: 0.33620 | load: 0.00023 [I 2022-10-27 04:05:58] ...trap/engines/engine.py.321: eval: epoch 46 | batch 80/510 [I 2022-10-27 04:05:58] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:25 [I 2022-10-27 04:05:58] ...trap/engines/engine.py.326: process: 0.33879 | load: 0.00022 [I 2022-10-27 04:06:02] ...trap/engines/engine.py.321: eval: epoch 46 | batch 90/510 [I 2022-10-27 04:06:02] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:23 [I 2022-10-27 04:06:02] ...trap/engines/engine.py.326: process: 0.32486 | load: 0.00015 [I 2022-10-27 04:06:05] ...trap/engines/engine.py.321: eval: epoch 46 | batch 100/510 [I 2022-10-27 04:06:05] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:13 [I 2022-10-27 04:06:05] ...trap/engines/engine.py.326: process: 0.28333 | load: 0.00018 [I 2022-10-27 04:06:08] ...trap/engines/engine.py.321: eval: epoch 46 | batch 110/510 [I 2022-10-27 04:06:08] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:58 [I 2022-10-27 04:06:08] ...trap/engines/engine.py.326: process: 0.31132 | load: 0.00015 [I 2022-10-27 04:06:11] ...trap/engines/engine.py.321: eval: epoch 46 | batch 120/510 [I 2022-10-27 04:06:11] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:53 [I 2022-10-27 04:06:11] ...trap/engines/engine.py.326: process: 0.28207 | load: 0.00016 [I 2022-10-27 04:06:14] ...trap/engines/engine.py.321: eval: epoch 46 | batch 130/510 [I 2022-10-27 04:06:14] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:51 [I 2022-10-27 04:06:14] ...trap/engines/engine.py.326: process: 0.30834 | load: 0.00016 [I 2022-10-27 04:06:17] ...trap/engines/engine.py.321: eval: epoch 46 | batch 140/510 [I 2022-10-27 04:06:17] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:47 [I 2022-10-27 04:06:17] ...trap/engines/engine.py.326: process: 0.28911 | load: 0.00014 [I 2022-10-27 04:06:20] ...trap/engines/engine.py.321: eval: epoch 46 | batch 150/510 [I 2022-10-27 04:06:20] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:45 [I 2022-10-27 04:06:20] ...trap/engines/engine.py.326: process: 0.28971 | load: 0.00019 [I 2022-10-27 04:06:23] ...trap/engines/engine.py.321: eval: epoch 46 | batch 160/510 [I 2022-10-27 04:06:23] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-27 04:06:23] ...trap/engines/engine.py.326: process: 0.28552 | load: 0.00020 [I 2022-10-27 04:06:25] ...trap/engines/engine.py.321: eval: epoch 46 | batch 170/510 [I 2022-10-27 04:06:25] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-27 04:06:25] ...trap/engines/engine.py.326: process: 0.28083 | load: 0.00020 [I 2022-10-27 04:06:28] ...trap/engines/engine.py.321: eval: epoch 46 | batch 180/510 [I 2022-10-27 04:06:28] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 04:06:28] ...trap/engines/engine.py.326: process: 0.28899 | load: 0.00021 [I 2022-10-27 04:06:31] ...trap/engines/engine.py.321: eval: epoch 46 | batch 190/510 [I 2022-10-27 04:06:31] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 04:06:31] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00015 [I 2022-10-27 04:06:34] ...trap/engines/engine.py.321: eval: epoch 46 | batch 200/510 [I 2022-10-27 04:06:34] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:30 [I 2022-10-27 04:06:34] ...trap/engines/engine.py.326: process: 0.29058 | load: 0.00021 [I 2022-10-27 04:06:37] ...trap/engines/engine.py.321: eval: epoch 46 | batch 210/510 [I 2022-10-27 04:06:37] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 04:06:37] ...trap/engines/engine.py.326: process: 0.28617 | load: 0.00023 [I 2022-10-27 04:06:40] ...trap/engines/engine.py.321: eval: epoch 46 | batch 220/510 [I 2022-10-27 04:06:40] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:24 [I 2022-10-27 04:06:40] ...trap/engines/engine.py.326: process: 0.28810 | load: 0.00026 [I 2022-10-27 04:06:43] ...trap/engines/engine.py.321: eval: epoch 46 | batch 230/510 [I 2022-10-27 04:06:43] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:24 [I 2022-10-27 04:06:43] ...trap/engines/engine.py.326: process: 0.29248 | load: 0.00021 [I 2022-10-27 04:06:46] ...trap/engines/engine.py.321: eval: epoch 46 | batch 240/510 [I 2022-10-27 04:06:46] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 04:06:46] ...trap/engines/engine.py.326: process: 0.28960 | load: 0.00025 [I 2022-10-27 04:06:49] ...trap/engines/engine.py.321: eval: epoch 46 | batch 250/510 [I 2022-10-27 04:06:49] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:16 [I 2022-10-27 04:06:49] ...trap/engines/engine.py.326: process: 0.29937 | load: 0.00016 [I 2022-10-27 04:06:52] ...trap/engines/engine.py.321: eval: epoch 46 | batch 260/510 [I 2022-10-27 04:06:52] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 04:06:52] ...trap/engines/engine.py.326: process: 0.29100 | load: 0.00025 [I 2022-10-27 04:06:55] ...trap/engines/engine.py.321: eval: epoch 46 | batch 270/510 [I 2022-10-27 04:06:55] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 04:06:55] ...trap/engines/engine.py.326: process: 0.29299 | load: 0.00016 [I 2022-10-27 04:06:58] ...trap/engines/engine.py.321: eval: epoch 46 | batch 280/510 [I 2022-10-27 04:06:58] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 04:06:58] ...trap/engines/engine.py.326: process: 0.29465 | load: 0.00016 [I 2022-10-27 04:07:00] ...trap/engines/engine.py.321: eval: epoch 46 | batch 290/510 [I 2022-10-27 04:07:00] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 04:07:00] ...trap/engines/engine.py.326: process: 0.28808 | load: 0.00016 [I 2022-10-27 04:07:03] ...trap/engines/engine.py.321: eval: epoch 46 | batch 300/510 [I 2022-10-27 04:07:03] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 04:07:03] ...trap/engines/engine.py.326: process: 0.29314 | load: 0.00016 [I 2022-10-27 04:07:06] ...trap/engines/engine.py.321: eval: epoch 46 | batch 310/510 [I 2022-10-27 04:07:06] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 04:07:06] ...trap/engines/engine.py.326: process: 0.29184 | load: 0.00015 [I 2022-10-27 04:07:09] ...trap/engines/engine.py.321: eval: epoch 46 | batch 320/510 [I 2022-10-27 04:07:09] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 04:07:09] ...trap/engines/engine.py.326: process: 0.29170 | load: 0.00027 [I 2022-10-27 04:07:12] ...trap/engines/engine.py.321: eval: epoch 46 | batch 330/510 [I 2022-10-27 04:07:12] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 04:07:12] ...trap/engines/engine.py.326: process: 0.28655 | load: 0.00019 [I 2022-10-27 04:07:15] ...trap/engines/engine.py.321: eval: epoch 46 | batch 340/510 [I 2022-10-27 04:07:15] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 04:07:15] ...trap/engines/engine.py.326: process: 0.29186 | load: 0.00024 [I 2022-10-27 04:07:18] ...trap/engines/engine.py.321: eval: epoch 46 | batch 350/510 [I 2022-10-27 04:07:18] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:49 [I 2022-10-27 04:07:18] ...trap/engines/engine.py.326: process: 0.29065 | load: 0.00035 [I 2022-10-27 04:07:21] ...trap/engines/engine.py.321: eval: epoch 46 | batch 360/510 [I 2022-10-27 04:07:21] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:44 [I 2022-10-27 04:07:21] ...trap/engines/engine.py.326: process: 0.30138 | load: 0.00024 [I 2022-10-27 04:07:24] ...trap/engines/engine.py.321: eval: epoch 46 | batch 370/510 [I 2022-10-27 04:07:24] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 04:07:24] ...trap/engines/engine.py.326: process: 0.27940 | load: 0.00015 [I 2022-10-27 04:07:27] ...trap/engines/engine.py.321: eval: epoch 46 | batch 380/510 [I 2022-10-27 04:07:27] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 04:07:27] ...trap/engines/engine.py.326: process: 0.29556 | load: 0.00040 [I 2022-10-27 04:07:29] ...trap/engines/engine.py.321: eval: epoch 46 | batch 390/510 [I 2022-10-27 04:07:29] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 04:07:29] ...trap/engines/engine.py.326: process: 0.29073 | load: 0.00016 [I 2022-10-27 04:07:32] ...trap/engines/engine.py.321: eval: epoch 46 | batch 400/510 [I 2022-10-27 04:07:32] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:32 [I 2022-10-27 04:07:32] ...trap/engines/engine.py.326: process: 0.31066 | load: 0.00023 [I 2022-10-27 04:07:35] ...trap/engines/engine.py.321: eval: epoch 46 | batch 410/510 [I 2022-10-27 04:07:35] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:29 [I 2022-10-27 04:07:35] ...trap/engines/engine.py.326: process: 0.29581 | load: 0.00016 [I 2022-10-27 04:07:38] ...trap/engines/engine.py.321: eval: epoch 46 | batch 420/510 [I 2022-10-27 04:07:38] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:26 [I 2022-10-27 04:07:38] ...trap/engines/engine.py.326: process: 0.28812 | load: 0.00024 [I 2022-10-27 04:07:41] ...trap/engines/engine.py.321: eval: epoch 46 | batch 430/510 [I 2022-10-27 04:07:41] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 04:07:41] ...trap/engines/engine.py.326: process: 0.28686 | load: 0.00014 [I 2022-10-27 04:07:44] ...trap/engines/engine.py.321: eval: epoch 46 | batch 440/510 [I 2022-10-27 04:07:44] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 04:07:44] ...trap/engines/engine.py.326: process: 0.29228 | load: 0.00015 [I 2022-10-27 04:07:47] ...trap/engines/engine.py.321: eval: epoch 46 | batch 450/510 [I 2022-10-27 04:07:47] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 04:07:47] ...trap/engines/engine.py.326: process: 0.30543 | load: 0.00016 [I 2022-10-27 04:07:50] ...trap/engines/engine.py.321: eval: epoch 46 | batch 460/510 [I 2022-10-27 04:07:50] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 04:07:50] ...trap/engines/engine.py.326: process: 0.28980 | load: 0.00026 [I 2022-10-27 04:07:53] ...trap/engines/engine.py.321: eval: epoch 46 | batch 470/510 [I 2022-10-27 04:07:53] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 04:07:53] ...trap/engines/engine.py.326: process: 0.28247 | load: 0.00015 [I 2022-10-27 04:07:56] ...trap/engines/engine.py.321: eval: epoch 46 | batch 480/510 [I 2022-10-27 04:07:56] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 04:07:56] ...trap/engines/engine.py.326: process: 0.28064 | load: 0.00017 [I 2022-10-27 04:07:58] ...trap/engines/engine.py.321: eval: epoch 46 | batch 490/510 [I 2022-10-27 04:07:58] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 04:07:58] ...trap/engines/engine.py.326: process: 0.29018 | load: 0.00016 [I 2022-10-27 04:08:01] ...trap/engines/engine.py.321: eval: epoch 46 | batch 500/510 [I 2022-10-27 04:08:01] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 04:08:01] ...trap/engines/engine.py.326: process: 0.28496 | load: 0.00015 [I 2022-10-27 04:08:04] ...trap/engines/engine.py.321: eval: epoch 46 | batch 510/510 [I 2022-10-27 04:08:04] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 04:08:04] ...trap/engines/engine.py.326: process: 0.28542 | load: 0.00012 [S 2022-10-27 04:08:05] ...trap/engines/engine.py.132: eval_epoch.epoch: 46 [I 2022-10-27 04:08:05] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.699500 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.897500 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935300 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.689400 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.893400 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932100 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.018173 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007284 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007001 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012651 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006545 [S 2022-10-27 04:08:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004908 [I 2022-10-27 04:08:10] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 04:08:10] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 04:08:10] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 04:08:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 04:08:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 04:08:12] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 04:08:15] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 04:08:21] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 04:08:21] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 04:08:21] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 04:08:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 04:08:30] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 04:08:30] ...trap/engines/engine.py.163: Training model on trainset for epoch 47 [I 2022-10-27 04:08:30] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 04:08:33] ...trap/engines/engine.py.224: train: epoch 47 | batch 0/2304 [I 2022-10-27 04:08:33] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:07 [I 2022-10-27 04:08:33] ...trap/engines/engine.py.229: process: 2.68566 | load: 2.05216 [I 2022-10-27 04:08:33] ...trap/engines/engine.py.230: loss: 0.12034 [I 2022-10-27 04:08:33] ...optimizers/trijoint.py.151: total_norm: 13.798473 [I 2022-10-27 04:08:39] ...trap/engines/engine.py.224: train: epoch 47 | batch 10/2304 [I 2022-10-27 04:08:39] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:12 [I 2022-10-27 04:08:39] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00499 [I 2022-10-27 04:08:39] ...trap/engines/engine.py.230: loss: 0.15922 [I 2022-10-27 04:08:39] ...optimizers/trijoint.py.151: total_norm: 17.753641 [I 2022-10-27 04:08:44] ...trap/engines/engine.py.224: train: epoch 47 | batch 20/2304 [I 2022-10-27 04:08:44] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:16 [I 2022-10-27 04:08:44] ...trap/engines/engine.py.229: process: 0.56272 | load: 0.00416 [I 2022-10-27 04:08:44] ...trap/engines/engine.py.230: loss: 0.12037 [I 2022-10-27 04:08:44] ...optimizers/trijoint.py.151: total_norm: 10.275192 [I 2022-10-27 04:08:50] ...trap/engines/engine.py.224: train: epoch 47 | batch 30/2304 [I 2022-10-27 04:08:50] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:16 [I 2022-10-27 04:08:50] ...trap/engines/engine.py.229: process: 0.56403 | load: 0.00292 [I 2022-10-27 04:08:50] ...trap/engines/engine.py.230: loss: 0.19023 [I 2022-10-27 04:08:50] ...optimizers/trijoint.py.151: total_norm: 21.558546 [I 2022-10-27 04:08:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 40/2304 [I 2022-10-27 04:08:55] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:58 [I 2022-10-27 04:08:55] ...trap/engines/engine.py.229: process: 0.54635 | load: 0.00354 [I 2022-10-27 04:08:55] ...trap/engines/engine.py.230: loss: 0.12838 [I 2022-10-27 04:08:55] ...optimizers/trijoint.py.151: total_norm: 7.414867 [I 2022-10-27 04:09:01] ...trap/engines/engine.py.224: train: epoch 47 | batch 50/2304 [I 2022-10-27 04:09:01] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:48 [I 2022-10-27 04:09:01] ...trap/engines/engine.py.229: process: 0.54484 | load: 0.00286 [I 2022-10-27 04:09:01] ...trap/engines/engine.py.230: loss: 0.17207 [I 2022-10-27 04:09:01] ...optimizers/trijoint.py.151: total_norm: 16.185619 [I 2022-10-27 04:09:06] ...trap/engines/engine.py.224: train: epoch 47 | batch 60/2304 [I 2022-10-27 04:09:06] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:54 [I 2022-10-27 04:09:06] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00330 [I 2022-10-27 04:09:06] ...trap/engines/engine.py.230: loss: 0.16258 [I 2022-10-27 04:09:06] ...optimizers/trijoint.py.151: total_norm: 15.792325 [I 2022-10-27 04:09:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 70/2304 [I 2022-10-27 04:09:12] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:40 [I 2022-10-27 04:09:12] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00271 [I 2022-10-27 04:09:12] ...trap/engines/engine.py.230: loss: 0.16600 [I 2022-10-27 04:09:12] ...optimizers/trijoint.py.151: total_norm: 19.373779 [I 2022-10-27 04:09:18] ...trap/engines/engine.py.224: train: epoch 47 | batch 80/2304 [I 2022-10-27 04:09:18] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:45 [I 2022-10-27 04:09:18] ...trap/engines/engine.py.229: process: 0.57183 | load: 0.00243 [I 2022-10-27 04:09:18] ...trap/engines/engine.py.230: loss: 0.13075 [I 2022-10-27 04:09:18] ...optimizers/trijoint.py.151: total_norm: 12.425591 [I 2022-10-27 04:09:23] ...trap/engines/engine.py.224: train: epoch 47 | batch 90/2304 [I 2022-10-27 04:09:23] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:21:10 [I 2022-10-27 04:09:23] ...trap/engines/engine.py.229: process: 0.56390 | load: 0.00273 [I 2022-10-27 04:09:23] ...trap/engines/engine.py.230: loss: 0.13903 [I 2022-10-27 04:09:23] ...optimizers/trijoint.py.151: total_norm: 21.510262 [I 2022-10-27 04:09:29] ...trap/engines/engine.py.224: train: epoch 47 | batch 100/2304 [I 2022-10-27 04:09:29] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:35 [I 2022-10-27 04:09:29] ...trap/engines/engine.py.229: process: 0.56776 | load: 0.00265 [I 2022-10-27 04:09:29] ...trap/engines/engine.py.230: loss: 0.16958 [I 2022-10-27 04:09:29] ...optimizers/trijoint.py.151: total_norm: 24.628147 [I 2022-10-27 04:09:35] ...trap/engines/engine.py.224: train: epoch 47 | batch 110/2304 [I 2022-10-27 04:09:35] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:17 [I 2022-10-27 04:09:35] ...trap/engines/engine.py.229: process: 0.55191 | load: 0.00251 [I 2022-10-27 04:09:35] ...trap/engines/engine.py.230: loss: 0.16516 [I 2022-10-27 04:09:35] ...optimizers/trijoint.py.151: total_norm: 9.910140 [I 2022-10-27 04:09:40] ...trap/engines/engine.py.224: train: epoch 47 | batch 120/2304 [I 2022-10-27 04:09:40] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:19 [I 2022-10-27 04:09:40] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00281 [I 2022-10-27 04:09:40] ...trap/engines/engine.py.230: loss: 0.17397 [I 2022-10-27 04:09:40] ...optimizers/trijoint.py.151: total_norm: 40.312294 [I 2022-10-27 04:09:46] ...trap/engines/engine.py.224: train: epoch 47 | batch 130/2304 [I 2022-10-27 04:09:46] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:07 [I 2022-10-27 04:09:46] ...trap/engines/engine.py.229: process: 0.55320 | load: 0.00266 [I 2022-10-27 04:09:46] ...trap/engines/engine.py.230: loss: 0.15765 [I 2022-10-27 04:09:46] ...optimizers/trijoint.py.151: total_norm: 15.651850 [I 2022-10-27 04:09:51] ...trap/engines/engine.py.224: train: epoch 47 | batch 140/2304 [I 2022-10-27 04:09:51] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:19:55 [I 2022-10-27 04:09:51] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00256 [I 2022-10-27 04:09:51] ...trap/engines/engine.py.230: loss: 0.20899 [I 2022-10-27 04:09:51] ...optimizers/trijoint.py.151: total_norm: 17.246698 [I 2022-10-27 04:09:57] ...trap/engines/engine.py.224: train: epoch 47 | batch 150/2304 [I 2022-10-27 04:09:57] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:55 [I 2022-10-27 04:09:57] ...trap/engines/engine.py.229: process: 0.54680 | load: 0.00255 [I 2022-10-27 04:09:57] ...trap/engines/engine.py.230: loss: 0.14182 [I 2022-10-27 04:09:57] ...optimizers/trijoint.py.151: total_norm: 11.474174 [I 2022-10-27 04:10:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 160/2304 [I 2022-10-27 04:10:02] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:54 [I 2022-10-27 04:10:02] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00260 [I 2022-10-27 04:10:02] ...trap/engines/engine.py.230: loss: 0.23615 [I 2022-10-27 04:10:02] ...optimizers/trijoint.py.151: total_norm: 61.452145 [I 2022-10-27 04:10:08] ...trap/engines/engine.py.224: train: epoch 47 | batch 170/2304 [I 2022-10-27 04:10:08] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:51 [I 2022-10-27 04:10:08] ...trap/engines/engine.py.229: process: 0.56322 | load: 0.00246 [I 2022-10-27 04:10:08] ...trap/engines/engine.py.230: loss: 0.12535 [I 2022-10-27 04:10:08] ...optimizers/trijoint.py.151: total_norm: 27.982420 [I 2022-10-27 04:10:14] ...trap/engines/engine.py.224: train: epoch 47 | batch 180/2304 [I 2022-10-27 04:10:14] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:41 [I 2022-10-27 04:10:14] ...trap/engines/engine.py.229: process: 0.56368 | load: 0.00265 [I 2022-10-27 04:10:14] ...trap/engines/engine.py.230: loss: 0.15781 [I 2022-10-27 04:10:14] ...optimizers/trijoint.py.151: total_norm: 16.442064 [I 2022-10-27 04:10:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 190/2304 [I 2022-10-27 04:10:19] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:38 [I 2022-10-27 04:10:19] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00256 [I 2022-10-27 04:10:19] ...trap/engines/engine.py.230: loss: 0.15249 [I 2022-10-27 04:10:19] ...optimizers/trijoint.py.151: total_norm: 24.671415 [I 2022-10-27 04:10:25] ...trap/engines/engine.py.224: train: epoch 47 | batch 200/2304 [I 2022-10-27 04:10:25] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:35 [I 2022-10-27 04:10:25] ...trap/engines/engine.py.229: process: 0.56459 | load: 0.00247 [I 2022-10-27 04:10:25] ...trap/engines/engine.py.230: loss: 0.20160 [I 2022-10-27 04:10:25] ...optimizers/trijoint.py.151: total_norm: 16.432604 [I 2022-10-27 04:10:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 210/2304 [I 2022-10-27 04:10:30] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:35 [I 2022-10-27 04:10:30] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00263 [I 2022-10-27 04:10:30] ...trap/engines/engine.py.230: loss: 0.15943 [I 2022-10-27 04:10:30] ...optimizers/trijoint.py.151: total_norm: 8.819211 [I 2022-10-27 04:10:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 220/2304 [I 2022-10-27 04:10:36] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:30 [I 2022-10-27 04:10:36] ...trap/engines/engine.py.229: process: 0.56431 | load: 0.00265 [I 2022-10-27 04:10:36] ...trap/engines/engine.py.230: loss: 0.11986 [I 2022-10-27 04:10:36] ...optimizers/trijoint.py.151: total_norm: 10.665605 [I 2022-10-27 04:10:42] ...trap/engines/engine.py.224: train: epoch 47 | batch 230/2304 [I 2022-10-27 04:10:42] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:33 [I 2022-10-27 04:10:42] ...trap/engines/engine.py.229: process: 0.57802 | load: 0.00252 [I 2022-10-27 04:10:42] ...trap/engines/engine.py.230: loss: 0.18397 [I 2022-10-27 04:10:42] ...optimizers/trijoint.py.151: total_norm: 37.018852 [I 2022-10-27 04:10:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 240/2304 [I 2022-10-27 04:10:47] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:21 [I 2022-10-27 04:10:47] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00260 [I 2022-10-27 04:10:47] ...trap/engines/engine.py.230: loss: 0.20205 [I 2022-10-27 04:10:47] ...optimizers/trijoint.py.151: total_norm: 26.607336 [I 2022-10-27 04:10:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 250/2304 [I 2022-10-27 04:10:53] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:13 [I 2022-10-27 04:10:53] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00251 [I 2022-10-27 04:10:53] ...trap/engines/engine.py.230: loss: 0.15283 [I 2022-10-27 04:10:53] ...optimizers/trijoint.py.151: total_norm: 8.926592 [I 2022-10-27 04:10:59] ...trap/engines/engine.py.224: train: epoch 47 | batch 260/2304 [I 2022-10-27 04:10:59] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:08 [I 2022-10-27 04:10:59] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00249 [I 2022-10-27 04:10:59] ...trap/engines/engine.py.230: loss: 0.11306 [I 2022-10-27 04:10:59] ...optimizers/trijoint.py.151: total_norm: 13.199943 [I 2022-10-27 04:11:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 270/2304 [I 2022-10-27 04:11:04] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:54 [I 2022-10-27 04:11:04] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00253 [I 2022-10-27 04:11:04] ...trap/engines/engine.py.230: loss: 0.18130 [I 2022-10-27 04:11:04] ...optimizers/trijoint.py.151: total_norm: 19.618023 [I 2022-10-27 04:11:10] ...trap/engines/engine.py.224: train: epoch 47 | batch 280/2304 [I 2022-10-27 04:11:10] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:52 [I 2022-10-27 04:11:10] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00254 [I 2022-10-27 04:11:10] ...trap/engines/engine.py.230: loss: 0.18446 [I 2022-10-27 04:11:10] ...optimizers/trijoint.py.151: total_norm: 26.107864 [I 2022-10-27 04:11:15] ...trap/engines/engine.py.224: train: epoch 47 | batch 290/2304 [I 2022-10-27 04:11:15] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:40 [I 2022-10-27 04:11:15] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00249 [I 2022-10-27 04:11:15] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-27 04:11:15] ...optimizers/trijoint.py.151: total_norm: 12.004995 [I 2022-10-27 04:11:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 300/2304 [I 2022-10-27 04:11:21] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:28 [I 2022-10-27 04:11:21] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00247 [I 2022-10-27 04:11:21] ...trap/engines/engine.py.230: loss: 0.18055 [I 2022-10-27 04:11:21] ...optimizers/trijoint.py.151: total_norm: 17.780182 [I 2022-10-27 04:11:26] ...trap/engines/engine.py.224: train: epoch 47 | batch 310/2304 [I 2022-10-27 04:11:26] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:27 [I 2022-10-27 04:11:26] ...trap/engines/engine.py.229: process: 0.55693 | load: 0.00241 [I 2022-10-27 04:11:26] ...trap/engines/engine.py.230: loss: 0.17738 [I 2022-10-27 04:11:26] ...optimizers/trijoint.py.151: total_norm: 18.314106 [I 2022-10-27 04:11:32] ...trap/engines/engine.py.224: train: epoch 47 | batch 320/2304 [I 2022-10-27 04:11:32] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:24 [I 2022-10-27 04:11:32] ...trap/engines/engine.py.229: process: 0.55198 | load: 0.00251 [I 2022-10-27 04:11:32] ...trap/engines/engine.py.230: loss: 0.17026 [I 2022-10-27 04:11:32] ...optimizers/trijoint.py.151: total_norm: 19.920889 [I 2022-10-27 04:11:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 330/2304 [I 2022-10-27 04:11:38] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:22 [I 2022-10-27 04:11:38] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00266 [I 2022-10-27 04:11:38] ...trap/engines/engine.py.230: loss: 0.17991 [I 2022-10-27 04:11:38] ...optimizers/trijoint.py.151: total_norm: 22.103466 [I 2022-10-27 04:11:43] ...trap/engines/engine.py.224: train: epoch 47 | batch 340/2304 [I 2022-10-27 04:11:43] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:13 [I 2022-10-27 04:11:43] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00249 [I 2022-10-27 04:11:43] ...trap/engines/engine.py.230: loss: 0.19670 [I 2022-10-27 04:11:43] ...optimizers/trijoint.py.151: total_norm: 21.752230 [I 2022-10-27 04:11:49] ...trap/engines/engine.py.224: train: epoch 47 | batch 350/2304 [I 2022-10-27 04:11:49] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:11 [I 2022-10-27 04:11:49] ...trap/engines/engine.py.229: process: 0.55145 | load: 0.00242 [I 2022-10-27 04:11:49] ...trap/engines/engine.py.230: loss: 0.11490 [I 2022-10-27 04:11:49] ...optimizers/trijoint.py.151: total_norm: 12.847648 [I 2022-10-27 04:11:54] ...trap/engines/engine.py.224: train: epoch 47 | batch 360/2304 [I 2022-10-27 04:11:54] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:04 [I 2022-10-27 04:11:54] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00250 [I 2022-10-27 04:11:54] ...trap/engines/engine.py.230: loss: 0.12932 [I 2022-10-27 04:11:54] ...optimizers/trijoint.py.151: total_norm: 12.305580 [I 2022-10-27 04:12:00] ...trap/engines/engine.py.224: train: epoch 47 | batch 370/2304 [I 2022-10-27 04:12:00] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:59 [I 2022-10-27 04:12:00] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00255 [I 2022-10-27 04:12:00] ...trap/engines/engine.py.230: loss: 0.15800 [I 2022-10-27 04:12:00] ...optimizers/trijoint.py.151: total_norm: 16.766939 [I 2022-10-27 04:12:06] ...trap/engines/engine.py.224: train: epoch 47 | batch 380/2304 [I 2022-10-27 04:12:06] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:46 [I 2022-10-27 04:12:06] ...trap/engines/engine.py.229: process: 0.54473 | load: 0.00246 [I 2022-10-27 04:12:06] ...trap/engines/engine.py.230: loss: 0.19775 [I 2022-10-27 04:12:06] ...optimizers/trijoint.py.151: total_norm: 23.933731 [I 2022-10-27 04:12:11] ...trap/engines/engine.py.224: train: epoch 47 | batch 390/2304 [I 2022-10-27 04:12:11] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:43 [I 2022-10-27 04:12:11] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00243 [I 2022-10-27 04:12:11] ...trap/engines/engine.py.230: loss: 0.17296 [I 2022-10-27 04:12:11] ...optimizers/trijoint.py.151: total_norm: 15.705978 [I 2022-10-27 04:12:17] ...trap/engines/engine.py.224: train: epoch 47 | batch 400/2304 [I 2022-10-27 04:12:17] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:35 [I 2022-10-27 04:12:17] ...trap/engines/engine.py.229: process: 0.53753 | load: 0.00279 [I 2022-10-27 04:12:17] ...trap/engines/engine.py.230: loss: 0.15028 [I 2022-10-27 04:12:17] ...optimizers/trijoint.py.151: total_norm: 6.304016 [I 2022-10-27 04:12:22] ...trap/engines/engine.py.224: train: epoch 47 | batch 410/2304 [I 2022-10-27 04:12:22] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:37 [I 2022-10-27 04:12:22] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00246 [I 2022-10-27 04:12:22] ...trap/engines/engine.py.230: loss: 0.17163 [I 2022-10-27 04:12:22] ...optimizers/trijoint.py.151: total_norm: 12.752514 [I 2022-10-27 04:12:28] ...trap/engines/engine.py.224: train: epoch 47 | batch 420/2304 [I 2022-10-27 04:12:28] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:29 [I 2022-10-27 04:12:28] ...trap/engines/engine.py.229: process: 0.55153 | load: 0.00248 [I 2022-10-27 04:12:28] ...trap/engines/engine.py.230: loss: 0.20027 [I 2022-10-27 04:12:28] ...optimizers/trijoint.py.151: total_norm: 12.155362 [I 2022-10-27 04:12:33] ...trap/engines/engine.py.224: train: epoch 47 | batch 430/2304 [I 2022-10-27 04:12:33] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:30 [I 2022-10-27 04:12:33] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00243 [I 2022-10-27 04:12:33] ...trap/engines/engine.py.230: loss: 0.28116 [I 2022-10-27 04:12:33] ...optimizers/trijoint.py.151: total_norm: 46.179375 [I 2022-10-27 04:12:39] ...trap/engines/engine.py.224: train: epoch 47 | batch 440/2304 [I 2022-10-27 04:12:39] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:29 [I 2022-10-27 04:12:39] ...trap/engines/engine.py.229: process: 0.56598 | load: 0.00254 [I 2022-10-27 04:12:39] ...trap/engines/engine.py.230: loss: 0.14144 [I 2022-10-27 04:12:39] ...optimizers/trijoint.py.151: total_norm: 12.915695 [I 2022-10-27 04:12:45] ...trap/engines/engine.py.224: train: epoch 47 | batch 450/2304 [I 2022-10-27 04:12:45] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:21 [I 2022-10-27 04:12:45] ...trap/engines/engine.py.229: process: 0.57254 | load: 0.00246 [I 2022-10-27 04:12:45] ...trap/engines/engine.py.230: loss: 0.15870 [I 2022-10-27 04:12:45] ...optimizers/trijoint.py.151: total_norm: 14.713066 [I 2022-10-27 04:12:50] ...trap/engines/engine.py.224: train: epoch 47 | batch 460/2304 [I 2022-10-27 04:12:50] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:11 [I 2022-10-27 04:12:50] ...trap/engines/engine.py.229: process: 0.56502 | load: 0.00255 [I 2022-10-27 04:12:50] ...trap/engines/engine.py.230: loss: 0.11926 [I 2022-10-27 04:12:50] ...optimizers/trijoint.py.151: total_norm: 14.695803 [I 2022-10-27 04:12:56] ...trap/engines/engine.py.224: train: epoch 47 | batch 470/2304 [I 2022-10-27 04:12:56] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:08 [I 2022-10-27 04:12:56] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00251 [I 2022-10-27 04:12:56] ...trap/engines/engine.py.230: loss: 0.15659 [I 2022-10-27 04:12:56] ...optimizers/trijoint.py.151: total_norm: 11.105989 [I 2022-10-27 04:13:01] ...trap/engines/engine.py.224: train: epoch 47 | batch 480/2304 [I 2022-10-27 04:13:01] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:51 [I 2022-10-27 04:13:01] ...trap/engines/engine.py.229: process: 0.54274 | load: 0.00245 [I 2022-10-27 04:13:01] ...trap/engines/engine.py.230: loss: 0.14534 [I 2022-10-27 04:13:01] ...optimizers/trijoint.py.151: total_norm: 13.650101 [I 2022-10-27 04:13:07] ...trap/engines/engine.py.224: train: epoch 47 | batch 490/2304 [I 2022-10-27 04:13:07] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:56 [I 2022-10-27 04:13:07] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00243 [I 2022-10-27 04:13:07] ...trap/engines/engine.py.230: loss: 0.15596 [I 2022-10-27 04:13:07] ...optimizers/trijoint.py.151: total_norm: 15.048470 [I 2022-10-27 04:13:13] ...trap/engines/engine.py.224: train: epoch 47 | batch 500/2304 [I 2022-10-27 04:13:13] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:44 [I 2022-10-27 04:13:13] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00242 [I 2022-10-27 04:13:13] ...trap/engines/engine.py.230: loss: 0.15589 [I 2022-10-27 04:13:13] ...optimizers/trijoint.py.151: total_norm: 14.139461 [I 2022-10-27 04:13:18] ...trap/engines/engine.py.224: train: epoch 47 | batch 510/2304 [I 2022-10-27 04:13:18] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:43 [I 2022-10-27 04:13:18] ...trap/engines/engine.py.229: process: 0.56355 | load: 0.00243 [I 2022-10-27 04:13:18] ...trap/engines/engine.py.230: loss: 0.16010 [I 2022-10-27 04:13:18] ...optimizers/trijoint.py.151: total_norm: 27.808365 [I 2022-10-27 04:13:24] ...trap/engines/engine.py.224: train: epoch 47 | batch 520/2304 [I 2022-10-27 04:13:24] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:39 [I 2022-10-27 04:13:24] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00239 [I 2022-10-27 04:13:24] ...trap/engines/engine.py.230: loss: 0.13571 [I 2022-10-27 04:13:24] ...optimizers/trijoint.py.151: total_norm: 10.082449 [I 2022-10-27 04:13:29] ...trap/engines/engine.py.224: train: epoch 47 | batch 530/2304 [I 2022-10-27 04:13:29] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:19 [I 2022-10-27 04:13:29] ...trap/engines/engine.py.229: process: 0.55066 | load: 0.00254 [I 2022-10-27 04:13:29] ...trap/engines/engine.py.230: loss: 0.11035 [I 2022-10-27 04:13:29] ...optimizers/trijoint.py.151: total_norm: 10.749260 [I 2022-10-27 04:13:35] ...trap/engines/engine.py.224: train: epoch 47 | batch 540/2304 [I 2022-10-27 04:13:35] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:22 [I 2022-10-27 04:13:35] ...trap/engines/engine.py.229: process: 0.56483 | load: 0.00249 [I 2022-10-27 04:13:35] ...trap/engines/engine.py.230: loss: 0.23132 [I 2022-10-27 04:13:35] ...optimizers/trijoint.py.151: total_norm: 30.587439 [I 2022-10-27 04:13:40] ...trap/engines/engine.py.224: train: epoch 47 | batch 550/2304 [I 2022-10-27 04:13:40] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:07 [I 2022-10-27 04:13:40] ...trap/engines/engine.py.229: process: 0.54620 | load: 0.00245 [I 2022-10-27 04:13:40] ...trap/engines/engine.py.230: loss: 0.13351 [I 2022-10-27 04:13:40] ...optimizers/trijoint.py.151: total_norm: 16.995689 [I 2022-10-27 04:13:46] ...trap/engines/engine.py.224: train: epoch 47 | batch 560/2304 [I 2022-10-27 04:13:46] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:17 [I 2022-10-27 04:13:46] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00245 [I 2022-10-27 04:13:46] ...trap/engines/engine.py.230: loss: 0.21030 [I 2022-10-27 04:13:46] ...optimizers/trijoint.py.151: total_norm: 16.280495 [I 2022-10-27 04:13:52] ...trap/engines/engine.py.224: train: epoch 47 | batch 570/2304 [I 2022-10-27 04:13:52] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:03 [I 2022-10-27 04:13:52] ...trap/engines/engine.py.229: process: 0.55612 | load: 0.00246 [I 2022-10-27 04:13:52] ...trap/engines/engine.py.230: loss: 0.14266 [I 2022-10-27 04:13:52] ...optimizers/trijoint.py.151: total_norm: 12.154321 [I 2022-10-27 04:13:57] ...trap/engines/engine.py.224: train: epoch 47 | batch 580/2304 [I 2022-10-27 04:13:57] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:59 [I 2022-10-27 04:13:57] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00244 [I 2022-10-27 04:13:57] ...trap/engines/engine.py.230: loss: 0.21278 [I 2022-10-27 04:13:57] ...optimizers/trijoint.py.151: total_norm: 12.160976 [I 2022-10-27 04:14:03] ...trap/engines/engine.py.224: train: epoch 47 | batch 590/2304 [I 2022-10-27 04:14:03] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:54 [I 2022-10-27 04:14:03] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00247 [I 2022-10-27 04:14:03] ...trap/engines/engine.py.230: loss: 0.16487 [I 2022-10-27 04:14:03] ...optimizers/trijoint.py.151: total_norm: 15.981115 [I 2022-10-27 04:14:08] ...trap/engines/engine.py.224: train: epoch 47 | batch 600/2304 [I 2022-10-27 04:14:08] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:50 [I 2022-10-27 04:14:08] ...trap/engines/engine.py.229: process: 0.56701 | load: 0.00250 [I 2022-10-27 04:14:08] ...trap/engines/engine.py.230: loss: 0.13593 [I 2022-10-27 04:14:08] ...optimizers/trijoint.py.151: total_norm: 18.593931 [I 2022-10-27 04:14:14] ...trap/engines/engine.py.224: train: epoch 47 | batch 610/2304 [I 2022-10-27 04:14:14] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:41 [I 2022-10-27 04:14:14] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00248 [I 2022-10-27 04:14:14] ...trap/engines/engine.py.230: loss: 0.11568 [I 2022-10-27 04:14:14] ...optimizers/trijoint.py.151: total_norm: 10.708093 [I 2022-10-27 04:14:20] ...trap/engines/engine.py.224: train: epoch 47 | batch 620/2304 [I 2022-10-27 04:14:20] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:40 [I 2022-10-27 04:14:20] ...trap/engines/engine.py.229: process: 0.56332 | load: 0.00243 [I 2022-10-27 04:14:20] ...trap/engines/engine.py.230: loss: 0.16526 [I 2022-10-27 04:14:20] ...optimizers/trijoint.py.151: total_norm: 17.307180 [I 2022-10-27 04:14:25] ...trap/engines/engine.py.224: train: epoch 47 | batch 630/2304 [I 2022-10-27 04:14:25] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:33 [I 2022-10-27 04:14:25] ...trap/engines/engine.py.229: process: 0.55147 | load: 0.00240 [I 2022-10-27 04:14:25] ...trap/engines/engine.py.230: loss: 0.15207 [I 2022-10-27 04:14:25] ...optimizers/trijoint.py.151: total_norm: 14.497874 [I 2022-10-27 04:14:31] ...trap/engines/engine.py.224: train: epoch 47 | batch 640/2304 [I 2022-10-27 04:14:31] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:29 [I 2022-10-27 04:14:31] ...trap/engines/engine.py.229: process: 0.56404 | load: 0.00239 [I 2022-10-27 04:14:31] ...trap/engines/engine.py.230: loss: 0.26061 [I 2022-10-27 04:14:31] ...optimizers/trijoint.py.151: total_norm: 31.261562 [I 2022-10-27 04:14:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 650/2304 [I 2022-10-27 04:14:36] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:20 [I 2022-10-27 04:14:36] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00244 [I 2022-10-27 04:14:36] ...trap/engines/engine.py.230: loss: 0.14838 [I 2022-10-27 04:14:36] ...optimizers/trijoint.py.151: total_norm: 11.687224 [I 2022-10-27 04:14:42] ...trap/engines/engine.py.224: train: epoch 47 | batch 660/2304 [I 2022-10-27 04:14:42] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:12 [I 2022-10-27 04:14:42] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00243 [I 2022-10-27 04:14:42] ...trap/engines/engine.py.230: loss: 0.23173 [I 2022-10-27 04:14:42] ...optimizers/trijoint.py.151: total_norm: 26.228271 [I 2022-10-27 04:14:48] ...trap/engines/engine.py.224: train: epoch 47 | batch 670/2304 [I 2022-10-27 04:14:48] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:11 [I 2022-10-27 04:14:48] ...trap/engines/engine.py.229: process: 0.56023 | load: 0.00242 [I 2022-10-27 04:14:48] ...trap/engines/engine.py.230: loss: 0.17255 [I 2022-10-27 04:14:48] ...optimizers/trijoint.py.151: total_norm: 18.117773 [I 2022-10-27 04:14:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 680/2304 [I 2022-10-27 04:14:53] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:03 [I 2022-10-27 04:14:53] ...trap/engines/engine.py.229: process: 0.54857 | load: 0.00253 [I 2022-10-27 04:14:53] ...trap/engines/engine.py.230: loss: 0.15093 [I 2022-10-27 04:14:53] ...optimizers/trijoint.py.151: total_norm: 21.137754 [I 2022-10-27 04:14:59] ...trap/engines/engine.py.224: train: epoch 47 | batch 690/2304 [I 2022-10-27 04:14:59] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:56 [I 2022-10-27 04:14:59] ...trap/engines/engine.py.229: process: 0.54576 | load: 0.00250 [I 2022-10-27 04:14:59] ...trap/engines/engine.py.230: loss: 0.16703 [I 2022-10-27 04:14:59] ...optimizers/trijoint.py.151: total_norm: 11.999814 [I 2022-10-27 04:15:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 700/2304 [I 2022-10-27 04:15:04] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:56 [I 2022-10-27 04:15:04] ...trap/engines/engine.py.229: process: 0.56294 | load: 0.00250 [I 2022-10-27 04:15:04] ...trap/engines/engine.py.230: loss: 0.17887 [I 2022-10-27 04:15:04] ...optimizers/trijoint.py.151: total_norm: 18.788435 [I 2022-10-27 04:15:10] ...trap/engines/engine.py.224: train: epoch 47 | batch 710/2304 [I 2022-10-27 04:15:10] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:46 [I 2022-10-27 04:15:10] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00241 [I 2022-10-27 04:15:10] ...trap/engines/engine.py.230: loss: 0.16759 [I 2022-10-27 04:15:10] ...optimizers/trijoint.py.151: total_norm: 12.916718 [I 2022-10-27 04:15:15] ...trap/engines/engine.py.224: train: epoch 47 | batch 720/2304 [I 2022-10-27 04:15:15] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:44 [I 2022-10-27 04:15:15] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00253 [I 2022-10-27 04:15:15] ...trap/engines/engine.py.230: loss: 0.18950 [I 2022-10-27 04:15:15] ...optimizers/trijoint.py.151: total_norm: 16.576735 [I 2022-10-27 04:15:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 730/2304 [I 2022-10-27 04:15:21] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:36 [I 2022-10-27 04:15:21] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00248 [I 2022-10-27 04:15:21] ...trap/engines/engine.py.230: loss: 0.13708 [I 2022-10-27 04:15:21] ...optimizers/trijoint.py.151: total_norm: 18.612242 [I 2022-10-27 04:15:27] ...trap/engines/engine.py.224: train: epoch 47 | batch 740/2304 [I 2022-10-27 04:15:27] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:32 [I 2022-10-27 04:15:27] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00244 [I 2022-10-27 04:15:27] ...trap/engines/engine.py.230: loss: 0.22861 [I 2022-10-27 04:15:27] ...optimizers/trijoint.py.151: total_norm: 20.328182 [I 2022-10-27 04:15:32] ...trap/engines/engine.py.224: train: epoch 47 | batch 750/2304 [I 2022-10-27 04:15:32] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:24 [I 2022-10-27 04:15:32] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00247 [I 2022-10-27 04:15:32] ...trap/engines/engine.py.230: loss: 0.11455 [I 2022-10-27 04:15:32] ...optimizers/trijoint.py.151: total_norm: 28.161272 [I 2022-10-27 04:15:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 760/2304 [I 2022-10-27 04:15:38] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:52 [I 2022-10-27 04:15:38] ...trap/engines/engine.py.229: process: 0.62645 | load: 0.00307 [I 2022-10-27 04:15:38] ...trap/engines/engine.py.230: loss: 0.13253 [I 2022-10-27 04:15:38] ...optimizers/trijoint.py.151: total_norm: 13.492771 [I 2022-10-27 04:15:43] ...trap/engines/engine.py.224: train: epoch 47 | batch 770/2304 [I 2022-10-27 04:15:43] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:16 [I 2022-10-27 04:15:43] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00249 [I 2022-10-27 04:15:43] ...trap/engines/engine.py.230: loss: 0.12945 [I 2022-10-27 04:15:43] ...optimizers/trijoint.py.151: total_norm: 18.416084 [I 2022-10-27 04:15:49] ...trap/engines/engine.py.224: train: epoch 47 | batch 780/2304 [I 2022-10-27 04:15:49] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:37 [I 2022-10-27 04:15:49] ...trap/engines/engine.py.229: process: 0.66756 | load: 0.00243 [I 2022-10-27 04:15:49] ...trap/engines/engine.py.230: loss: 0.18016 [I 2022-10-27 04:15:49] ...optimizers/trijoint.py.151: total_norm: 24.304079 [I 2022-10-27 04:15:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 790/2304 [I 2022-10-27 04:15:55] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:09 [I 2022-10-27 04:15:55] ...trap/engines/engine.py.229: process: 0.56515 | load: 0.00380 [I 2022-10-27 04:15:55] ...trap/engines/engine.py.230: loss: 0.14822 [I 2022-10-27 04:15:55] ...optimizers/trijoint.py.151: total_norm: 20.229839 [I 2022-10-27 04:16:00] ...trap/engines/engine.py.224: train: epoch 47 | batch 800/2304 [I 2022-10-27 04:16:00] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:02 [I 2022-10-27 04:16:00] ...trap/engines/engine.py.229: process: 0.55711 | load: 0.00245 [I 2022-10-27 04:16:00] ...trap/engines/engine.py.230: loss: 0.21073 [I 2022-10-27 04:16:00] ...optimizers/trijoint.py.151: total_norm: 15.915186 [I 2022-10-27 04:16:06] ...trap/engines/engine.py.224: train: epoch 47 | batch 810/2304 [I 2022-10-27 04:16:06] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:53 [I 2022-10-27 04:16:06] ...trap/engines/engine.py.229: process: 0.55356 | load: 0.00260 [I 2022-10-27 04:16:06] ...trap/engines/engine.py.230: loss: 0.26671 [I 2022-10-27 04:16:06] ...optimizers/trijoint.py.151: total_norm: 23.880182 [I 2022-10-27 04:16:11] ...trap/engines/engine.py.224: train: epoch 47 | batch 820/2304 [I 2022-10-27 04:16:11] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:44 [I 2022-10-27 04:16:11] ...trap/engines/engine.py.229: process: 0.55949 | load: 0.00246 [I 2022-10-27 04:16:11] ...trap/engines/engine.py.230: loss: 0.18600 [I 2022-10-27 04:16:11] ...optimizers/trijoint.py.151: total_norm: 21.172525 [I 2022-10-27 04:16:17] ...trap/engines/engine.py.224: train: epoch 47 | batch 830/2304 [I 2022-10-27 04:16:17] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:38 [I 2022-10-27 04:16:17] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00244 [I 2022-10-27 04:16:17] ...trap/engines/engine.py.230: loss: 0.17027 [I 2022-10-27 04:16:17] ...optimizers/trijoint.py.151: total_norm: 21.551577 [I 2022-10-27 04:16:23] ...trap/engines/engine.py.224: train: epoch 47 | batch 840/2304 [I 2022-10-27 04:16:23] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:37 [I 2022-10-27 04:16:23] ...trap/engines/engine.py.229: process: 0.56445 | load: 0.00244 [I 2022-10-27 04:16:23] ...trap/engines/engine.py.230: loss: 0.14807 [I 2022-10-27 04:16:23] ...optimizers/trijoint.py.151: total_norm: 11.239274 [I 2022-10-27 04:16:28] ...trap/engines/engine.py.224: train: epoch 47 | batch 850/2304 [I 2022-10-27 04:16:28] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:39 [I 2022-10-27 04:16:28] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00251 [I 2022-10-27 04:16:28] ...trap/engines/engine.py.230: loss: 0.16197 [I 2022-10-27 04:16:28] ...optimizers/trijoint.py.151: total_norm: 13.774490 [I 2022-10-27 04:16:34] ...trap/engines/engine.py.224: train: epoch 47 | batch 860/2304 [I 2022-10-27 04:16:34] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:24 [I 2022-10-27 04:16:34] ...trap/engines/engine.py.229: process: 0.56606 | load: 0.00250 [I 2022-10-27 04:16:34] ...trap/engines/engine.py.230: loss: 0.13415 [I 2022-10-27 04:16:34] ...optimizers/trijoint.py.151: total_norm: 13.840841 [I 2022-10-27 04:16:40] ...trap/engines/engine.py.224: train: epoch 47 | batch 870/2304 [I 2022-10-27 04:16:40] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:18 [I 2022-10-27 04:16:40] ...trap/engines/engine.py.229: process: 0.55637 | load: 0.00245 [I 2022-10-27 04:16:40] ...trap/engines/engine.py.230: loss: 0.11341 [I 2022-10-27 04:16:40] ...optimizers/trijoint.py.151: total_norm: 12.196062 [I 2022-10-27 04:16:45] ...trap/engines/engine.py.224: train: epoch 47 | batch 880/2304 [I 2022-10-27 04:16:45] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:15 [I 2022-10-27 04:16:45] ...trap/engines/engine.py.229: process: 0.56395 | load: 0.00247 [I 2022-10-27 04:16:45] ...trap/engines/engine.py.230: loss: 0.12378 [I 2022-10-27 04:16:45] ...optimizers/trijoint.py.151: total_norm: 9.145249 [I 2022-10-27 04:16:51] ...trap/engines/engine.py.224: train: epoch 47 | batch 890/2304 [I 2022-10-27 04:16:51] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:00 [I 2022-10-27 04:16:51] ...trap/engines/engine.py.229: process: 0.54658 | load: 0.00246 [I 2022-10-27 04:16:51] ...trap/engines/engine.py.230: loss: 0.16259 [I 2022-10-27 04:16:51] ...optimizers/trijoint.py.151: total_norm: 32.962875 [I 2022-10-27 04:16:56] ...trap/engines/engine.py.224: train: epoch 47 | batch 900/2304 [I 2022-10-27 04:16:56] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:12:59 [I 2022-10-27 04:16:56] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00244 [I 2022-10-27 04:16:56] ...trap/engines/engine.py.230: loss: 0.13856 [I 2022-10-27 04:16:56] ...optimizers/trijoint.py.151: total_norm: 10.665844 [I 2022-10-27 04:17:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 910/2304 [I 2022-10-27 04:17:02] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:56 [I 2022-10-27 04:17:02] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00245 [I 2022-10-27 04:17:02] ...trap/engines/engine.py.230: loss: 0.16975 [I 2022-10-27 04:17:02] ...optimizers/trijoint.py.151: total_norm: 23.818096 [I 2022-10-27 04:17:07] ...trap/engines/engine.py.224: train: epoch 47 | batch 920/2304 [I 2022-10-27 04:17:07] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:57 [I 2022-10-27 04:17:07] ...trap/engines/engine.py.229: process: 0.57320 | load: 0.00247 [I 2022-10-27 04:17:07] ...trap/engines/engine.py.230: loss: 0.15739 [I 2022-10-27 04:17:07] ...optimizers/trijoint.py.151: total_norm: 11.412453 [I 2022-10-27 04:17:13] ...trap/engines/engine.py.224: train: epoch 47 | batch 930/2304 [I 2022-10-27 04:17:13] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:44 [I 2022-10-27 04:17:13] ...trap/engines/engine.py.229: process: 0.55667 | load: 0.00248 [I 2022-10-27 04:17:13] ...trap/engines/engine.py.230: loss: 0.15939 [I 2022-10-27 04:17:13] ...optimizers/trijoint.py.151: total_norm: 11.819195 [I 2022-10-27 04:17:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 940/2304 [I 2022-10-27 04:17:19] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:39 [I 2022-10-27 04:17:19] ...trap/engines/engine.py.229: process: 0.56533 | load: 0.00248 [I 2022-10-27 04:17:19] ...trap/engines/engine.py.230: loss: 0.11246 [I 2022-10-27 04:17:19] ...optimizers/trijoint.py.151: total_norm: 8.192150 [I 2022-10-27 04:17:24] ...trap/engines/engine.py.224: train: epoch 47 | batch 950/2304 [I 2022-10-27 04:17:24] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:34 [I 2022-10-27 04:17:24] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00243 [I 2022-10-27 04:17:24] ...trap/engines/engine.py.230: loss: 0.20706 [I 2022-10-27 04:17:24] ...optimizers/trijoint.py.151: total_norm: 23.486864 [I 2022-10-27 04:17:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 960/2304 [I 2022-10-27 04:17:30] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:26 [I 2022-10-27 04:17:30] ...trap/engines/engine.py.229: process: 0.55342 | load: 0.00242 [I 2022-10-27 04:17:30] ...trap/engines/engine.py.230: loss: 0.14659 [I 2022-10-27 04:17:30] ...optimizers/trijoint.py.151: total_norm: 14.064366 [I 2022-10-27 04:17:35] ...trap/engines/engine.py.224: train: epoch 47 | batch 970/2304 [I 2022-10-27 04:17:35] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:19 [I 2022-10-27 04:17:35] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00243 [I 2022-10-27 04:17:35] ...trap/engines/engine.py.230: loss: 0.15941 [I 2022-10-27 04:17:35] ...optimizers/trijoint.py.151: total_norm: 11.817434 [I 2022-10-27 04:17:41] ...trap/engines/engine.py.224: train: epoch 47 | batch 980/2304 [I 2022-10-27 04:17:41] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:18 [I 2022-10-27 04:17:41] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00242 [I 2022-10-27 04:17:41] ...trap/engines/engine.py.230: loss: 0.17624 [I 2022-10-27 04:17:41] ...optimizers/trijoint.py.151: total_norm: 17.340185 [I 2022-10-27 04:17:46] ...trap/engines/engine.py.224: train: epoch 47 | batch 990/2304 [I 2022-10-27 04:17:46] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:14 [I 2022-10-27 04:17:46] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00244 [I 2022-10-27 04:17:46] ...trap/engines/engine.py.230: loss: 0.20638 [I 2022-10-27 04:17:46] ...optimizers/trijoint.py.151: total_norm: 34.843773 [I 2022-10-27 04:17:52] ...trap/engines/engine.py.224: train: epoch 47 | batch 1000/2304 [I 2022-10-27 04:17:52] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:11:59 [I 2022-10-27 04:17:52] ...trap/engines/engine.py.229: process: 0.54203 | load: 0.00243 [I 2022-10-27 04:17:52] ...trap/engines/engine.py.230: loss: 0.12263 [I 2022-10-27 04:17:52] ...optimizers/trijoint.py.151: total_norm: 11.877534 [I 2022-10-27 04:17:58] ...trap/engines/engine.py.224: train: epoch 47 | batch 1010/2304 [I 2022-10-27 04:17:58] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:02 [I 2022-10-27 04:17:58] ...trap/engines/engine.py.229: process: 0.56298 | load: 0.00244 [I 2022-10-27 04:17:58] ...trap/engines/engine.py.230: loss: 0.16243 [I 2022-10-27 04:17:58] ...optimizers/trijoint.py.151: total_norm: 36.044792 [I 2022-10-27 04:18:03] ...trap/engines/engine.py.224: train: epoch 47 | batch 1020/2304 [I 2022-10-27 04:18:03] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:55 [I 2022-10-27 04:18:03] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00244 [I 2022-10-27 04:18:03] ...trap/engines/engine.py.230: loss: 0.14619 [I 2022-10-27 04:18:03] ...optimizers/trijoint.py.151: total_norm: 13.169113 [I 2022-10-27 04:18:09] ...trap/engines/engine.py.224: train: epoch 47 | batch 1030/2304 [I 2022-10-27 04:18:09] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:49 [I 2022-10-27 04:18:09] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00253 [I 2022-10-27 04:18:09] ...trap/engines/engine.py.230: loss: 0.12308 [I 2022-10-27 04:18:09] ...optimizers/trijoint.py.151: total_norm: 11.606595 [I 2022-10-27 04:18:14] ...trap/engines/engine.py.224: train: epoch 47 | batch 1040/2304 [I 2022-10-27 04:18:14] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:12:05 [I 2022-10-27 04:18:14] ...trap/engines/engine.py.229: process: 0.56525 | load: 0.00248 [I 2022-10-27 04:18:14] ...trap/engines/engine.py.230: loss: 0.17574 [I 2022-10-27 04:18:14] ...optimizers/trijoint.py.151: total_norm: 17.065540 [I 2022-10-27 04:18:20] ...trap/engines/engine.py.224: train: epoch 47 | batch 1050/2304 [I 2022-10-27 04:18:20] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:40 [I 2022-10-27 04:18:20] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00248 [I 2022-10-27 04:18:20] ...trap/engines/engine.py.230: loss: 0.15876 [I 2022-10-27 04:18:20] ...optimizers/trijoint.py.151: total_norm: 11.096892 [I 2022-10-27 04:18:26] ...trap/engines/engine.py.224: train: epoch 47 | batch 1060/2304 [I 2022-10-27 04:18:26] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:32 [I 2022-10-27 04:18:26] ...trap/engines/engine.py.229: process: 0.55578 | load: 0.00243 [I 2022-10-27 04:18:26] ...trap/engines/engine.py.230: loss: 0.17185 [I 2022-10-27 04:18:26] ...optimizers/trijoint.py.151: total_norm: 19.603804 [I 2022-10-27 04:18:31] ...trap/engines/engine.py.224: train: epoch 47 | batch 1070/2304 [I 2022-10-27 04:18:31] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:25 [I 2022-10-27 04:18:31] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00248 [I 2022-10-27 04:18:31] ...trap/engines/engine.py.230: loss: 0.17957 [I 2022-10-27 04:18:31] ...optimizers/trijoint.py.151: total_norm: 16.228659 [I 2022-10-27 04:18:37] ...trap/engines/engine.py.224: train: epoch 47 | batch 1080/2304 [I 2022-10-27 04:18:37] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:21 [I 2022-10-27 04:18:37] ...trap/engines/engine.py.229: process: 0.54928 | load: 0.00240 [I 2022-10-27 04:18:37] ...trap/engines/engine.py.230: loss: 0.16388 [I 2022-10-27 04:18:37] ...optimizers/trijoint.py.151: total_norm: 10.248874 [I 2022-10-27 04:18:42] ...trap/engines/engine.py.224: train: epoch 47 | batch 1090/2304 [I 2022-10-27 04:18:42] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:19 [I 2022-10-27 04:18:42] ...trap/engines/engine.py.229: process: 0.56222 | load: 0.00243 [I 2022-10-27 04:18:42] ...trap/engines/engine.py.230: loss: 0.23619 [I 2022-10-27 04:18:42] ...optimizers/trijoint.py.151: total_norm: 29.776789 [I 2022-10-27 04:18:48] ...trap/engines/engine.py.224: train: epoch 47 | batch 1100/2304 [I 2022-10-27 04:18:48] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:12 [I 2022-10-27 04:18:48] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00244 [I 2022-10-27 04:18:48] ...trap/engines/engine.py.230: loss: 0.12592 [I 2022-10-27 04:18:48] ...optimizers/trijoint.py.151: total_norm: 10.601834 [I 2022-10-27 04:18:54] ...trap/engines/engine.py.224: train: epoch 47 | batch 1110/2304 [I 2022-10-27 04:18:54] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:02 [I 2022-10-27 04:18:54] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00244 [I 2022-10-27 04:18:54] ...trap/engines/engine.py.230: loss: 0.14119 [I 2022-10-27 04:18:54] ...optimizers/trijoint.py.151: total_norm: 31.737732 [I 2022-10-27 04:18:59] ...trap/engines/engine.py.224: train: epoch 47 | batch 1120/2304 [I 2022-10-27 04:18:59] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:03 [I 2022-10-27 04:18:59] ...trap/engines/engine.py.229: process: 0.55703 | load: 0.00249 [I 2022-10-27 04:18:59] ...trap/engines/engine.py.230: loss: 0.13929 [I 2022-10-27 04:18:59] ...optimizers/trijoint.py.151: total_norm: 21.774345 [I 2022-10-27 04:19:05] ...trap/engines/engine.py.224: train: epoch 47 | batch 1130/2304 [I 2022-10-27 04:19:05] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:52 [I 2022-10-27 04:19:05] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00245 [I 2022-10-27 04:19:05] ...trap/engines/engine.py.230: loss: 0.19554 [I 2022-10-27 04:19:05] ...optimizers/trijoint.py.151: total_norm: 20.064674 [I 2022-10-27 04:19:10] ...trap/engines/engine.py.224: train: epoch 47 | batch 1140/2304 [I 2022-10-27 04:19:10] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:47 [I 2022-10-27 04:19:10] ...trap/engines/engine.py.229: process: 0.55115 | load: 0.00246 [I 2022-10-27 04:19:10] ...trap/engines/engine.py.230: loss: 0.17045 [I 2022-10-27 04:19:10] ...optimizers/trijoint.py.151: total_norm: 16.667019 [I 2022-10-27 04:19:16] ...trap/engines/engine.py.224: train: epoch 47 | batch 1150/2304 [I 2022-10-27 04:19:16] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:43 [I 2022-10-27 04:19:16] ...trap/engines/engine.py.229: process: 0.55941 | load: 0.00244 [I 2022-10-27 04:19:16] ...trap/engines/engine.py.230: loss: 0.16241 [I 2022-10-27 04:19:16] ...optimizers/trijoint.py.151: total_norm: 24.537943 [I 2022-10-27 04:19:22] ...trap/engines/engine.py.224: train: epoch 47 | batch 1160/2304 [I 2022-10-27 04:19:22] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:50 [I 2022-10-27 04:19:22] ...trap/engines/engine.py.229: process: 0.57045 | load: 0.00245 [I 2022-10-27 04:19:22] ...trap/engines/engine.py.230: loss: 0.18669 [I 2022-10-27 04:19:22] ...optimizers/trijoint.py.151: total_norm: 12.180271 [I 2022-10-27 04:19:27] ...trap/engines/engine.py.224: train: epoch 47 | batch 1170/2304 [I 2022-10-27 04:19:27] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:33 [I 2022-10-27 04:19:27] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00246 [I 2022-10-27 04:19:27] ...trap/engines/engine.py.230: loss: 0.16065 [I 2022-10-27 04:19:27] ...optimizers/trijoint.py.151: total_norm: 24.879229 [I 2022-10-27 04:19:33] ...trap/engines/engine.py.224: train: epoch 47 | batch 1180/2304 [I 2022-10-27 04:19:33] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:26 [I 2022-10-27 04:19:33] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00244 [I 2022-10-27 04:19:33] ...trap/engines/engine.py.230: loss: 0.14253 [I 2022-10-27 04:19:33] ...optimizers/trijoint.py.151: total_norm: 11.264537 [I 2022-10-27 04:19:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 1190/2304 [I 2022-10-27 04:19:38] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:15 [I 2022-10-27 04:19:38] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00257 [I 2022-10-27 04:19:38] ...trap/engines/engine.py.230: loss: 0.16018 [I 2022-10-27 04:19:38] ...optimizers/trijoint.py.151: total_norm: 28.807180 [I 2022-10-27 04:19:44] ...trap/engines/engine.py.224: train: epoch 47 | batch 1200/2304 [I 2022-10-27 04:19:44] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:15 [I 2022-10-27 04:19:44] ...trap/engines/engine.py.229: process: 0.56562 | load: 0.00241 [I 2022-10-27 04:19:44] ...trap/engines/engine.py.230: loss: 0.14745 [I 2022-10-27 04:19:44] ...optimizers/trijoint.py.151: total_norm: 20.693930 [I 2022-10-27 04:19:49] ...trap/engines/engine.py.224: train: epoch 47 | batch 1210/2304 [I 2022-10-27 04:19:49] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:09 [I 2022-10-27 04:19:49] ...trap/engines/engine.py.229: process: 0.56339 | load: 0.00241 [I 2022-10-27 04:19:49] ...trap/engines/engine.py.230: loss: 0.14619 [I 2022-10-27 04:19:49] ...optimizers/trijoint.py.151: total_norm: 14.890409 [I 2022-10-27 04:19:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 1220/2304 [I 2022-10-27 04:19:55] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:07 [I 2022-10-27 04:19:55] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00257 [I 2022-10-27 04:19:55] ...trap/engines/engine.py.230: loss: 0.20455 [I 2022-10-27 04:19:55] ...optimizers/trijoint.py.151: total_norm: 14.502685 [I 2022-10-27 04:20:01] ...trap/engines/engine.py.224: train: epoch 47 | batch 1230/2304 [I 2022-10-27 04:20:01] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:56 [I 2022-10-27 04:20:01] ...trap/engines/engine.py.229: process: 0.55299 | load: 0.00248 [I 2022-10-27 04:20:01] ...trap/engines/engine.py.230: loss: 0.14671 [I 2022-10-27 04:20:01] ...optimizers/trijoint.py.151: total_norm: 20.503731 [I 2022-10-27 04:20:06] ...trap/engines/engine.py.224: train: epoch 47 | batch 1240/2304 [I 2022-10-27 04:20:06] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:53 [I 2022-10-27 04:20:06] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00248 [I 2022-10-27 04:20:06] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-27 04:20:06] ...optimizers/trijoint.py.151: total_norm: 17.850769 [I 2022-10-27 04:20:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 1250/2304 [I 2022-10-27 04:20:12] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:45 [I 2022-10-27 04:20:12] ...trap/engines/engine.py.229: process: 0.54919 | load: 0.00266 [I 2022-10-27 04:20:12] ...trap/engines/engine.py.230: loss: 0.12461 [I 2022-10-27 04:20:12] ...optimizers/trijoint.py.151: total_norm: 28.479715 [I 2022-10-27 04:20:17] ...trap/engines/engine.py.224: train: epoch 47 | batch 1260/2304 [I 2022-10-27 04:20:17] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:38 [I 2022-10-27 04:20:17] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00244 [I 2022-10-27 04:20:17] ...trap/engines/engine.py.230: loss: 0.12241 [I 2022-10-27 04:20:17] ...optimizers/trijoint.py.151: total_norm: 12.119843 [I 2022-10-27 04:20:23] ...trap/engines/engine.py.224: train: epoch 47 | batch 1270/2304 [I 2022-10-27 04:20:23] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:35 [I 2022-10-27 04:20:23] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00243 [I 2022-10-27 04:20:23] ...trap/engines/engine.py.230: loss: 0.12549 [I 2022-10-27 04:20:23] ...optimizers/trijoint.py.151: total_norm: 39.362198 [I 2022-10-27 04:20:29] ...trap/engines/engine.py.224: train: epoch 47 | batch 1280/2304 [I 2022-10-27 04:20:29] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:34 [I 2022-10-27 04:20:29] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00245 [I 2022-10-27 04:20:29] ...trap/engines/engine.py.230: loss: 0.19118 [I 2022-10-27 04:20:29] ...optimizers/trijoint.py.151: total_norm: 30.944254 [I 2022-10-27 04:20:34] ...trap/engines/engine.py.224: train: epoch 47 | batch 1290/2304 [I 2022-10-27 04:20:34] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:23 [I 2022-10-27 04:20:34] ...trap/engines/engine.py.229: process: 0.55036 | load: 0.00249 [I 2022-10-27 04:20:34] ...trap/engines/engine.py.230: loss: 0.24139 [I 2022-10-27 04:20:34] ...optimizers/trijoint.py.151: total_norm: 12.929945 [I 2022-10-27 04:20:40] ...trap/engines/engine.py.224: train: epoch 47 | batch 1300/2304 [I 2022-10-27 04:20:40] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:18 [I 2022-10-27 04:20:40] ...trap/engines/engine.py.229: process: 0.55272 | load: 0.00245 [I 2022-10-27 04:20:40] ...trap/engines/engine.py.230: loss: 0.15170 [I 2022-10-27 04:20:40] ...optimizers/trijoint.py.151: total_norm: 9.977188 [I 2022-10-27 04:20:45] ...trap/engines/engine.py.224: train: epoch 47 | batch 1310/2304 [I 2022-10-27 04:20:45] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:18 [I 2022-10-27 04:20:45] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00257 [I 2022-10-27 04:20:45] ...trap/engines/engine.py.230: loss: 0.18720 [I 2022-10-27 04:20:45] ...optimizers/trijoint.py.151: total_norm: 14.785009 [I 2022-10-27 04:20:51] ...trap/engines/engine.py.224: train: epoch 47 | batch 1320/2304 [I 2022-10-27 04:20:51] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:06 [I 2022-10-27 04:20:51] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00248 [I 2022-10-27 04:20:51] ...trap/engines/engine.py.230: loss: 0.15607 [I 2022-10-27 04:20:51] ...optimizers/trijoint.py.151: total_norm: 15.723776 [I 2022-10-27 04:20:57] ...trap/engines/engine.py.224: train: epoch 47 | batch 1330/2304 [I 2022-10-27 04:20:57] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:03 [I 2022-10-27 04:20:57] ...trap/engines/engine.py.229: process: 0.54452 | load: 0.00243 [I 2022-10-27 04:20:57] ...trap/engines/engine.py.230: loss: 0.13688 [I 2022-10-27 04:20:57] ...optimizers/trijoint.py.151: total_norm: 13.369497 [I 2022-10-27 04:21:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 1340/2304 [I 2022-10-27 04:21:02] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:55 [I 2022-10-27 04:21:02] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00244 [I 2022-10-27 04:21:02] ...trap/engines/engine.py.230: loss: 0.17618 [I 2022-10-27 04:21:02] ...optimizers/trijoint.py.151: total_norm: 17.421738 [I 2022-10-27 04:21:08] ...trap/engines/engine.py.224: train: epoch 47 | batch 1350/2304 [I 2022-10-27 04:21:08] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:52 [I 2022-10-27 04:21:08] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00246 [I 2022-10-27 04:21:08] ...trap/engines/engine.py.230: loss: 0.12033 [I 2022-10-27 04:21:08] ...optimizers/trijoint.py.151: total_norm: 25.053959 [I 2022-10-27 04:21:13] ...trap/engines/engine.py.224: train: epoch 47 | batch 1360/2304 [I 2022-10-27 04:21:13] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:41 [I 2022-10-27 04:21:13] ...trap/engines/engine.py.229: process: 0.54020 | load: 0.00244 [I 2022-10-27 04:21:13] ...trap/engines/engine.py.230: loss: 0.10307 [I 2022-10-27 04:21:13] ...optimizers/trijoint.py.151: total_norm: 12.761479 [I 2022-10-27 04:21:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 1370/2304 [I 2022-10-27 04:21:19] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:36 [I 2022-10-27 04:21:19] ...trap/engines/engine.py.229: process: 0.54571 | load: 0.00238 [I 2022-10-27 04:21:19] ...trap/engines/engine.py.230: loss: 0.20594 [I 2022-10-27 04:21:19] ...optimizers/trijoint.py.151: total_norm: 16.976299 [I 2022-10-27 04:21:24] ...trap/engines/engine.py.224: train: epoch 47 | batch 1380/2304 [I 2022-10-27 04:21:24] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:32 [I 2022-10-27 04:21:24] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00244 [I 2022-10-27 04:21:24] ...trap/engines/engine.py.230: loss: 0.20626 [I 2022-10-27 04:21:24] ...optimizers/trijoint.py.151: total_norm: 41.770012 [I 2022-10-27 04:21:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 1390/2304 [I 2022-10-27 04:21:30] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:27 [I 2022-10-27 04:21:30] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00251 [I 2022-10-27 04:21:30] ...trap/engines/engine.py.230: loss: 0.11980 [I 2022-10-27 04:21:30] ...optimizers/trijoint.py.151: total_norm: 14.672317 [I 2022-10-27 04:21:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 1400/2304 [I 2022-10-27 04:21:36] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:28 [I 2022-10-27 04:21:36] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00248 [I 2022-10-27 04:21:36] ...trap/engines/engine.py.230: loss: 0.15968 [I 2022-10-27 04:21:36] ...optimizers/trijoint.py.151: total_norm: 26.467913 [I 2022-10-27 04:21:41] ...trap/engines/engine.py.224: train: epoch 47 | batch 1410/2304 [I 2022-10-27 04:21:41] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:17 [I 2022-10-27 04:21:41] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00244 [I 2022-10-27 04:21:41] ...trap/engines/engine.py.230: loss: 0.22365 [I 2022-10-27 04:21:41] ...optimizers/trijoint.py.151: total_norm: 12.283740 [I 2022-10-27 04:21:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 1420/2304 [I 2022-10-27 04:21:47] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:10 [I 2022-10-27 04:21:47] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00243 [I 2022-10-27 04:21:47] ...trap/engines/engine.py.230: loss: 0.21571 [I 2022-10-27 04:21:47] ...optimizers/trijoint.py.151: total_norm: 18.414003 [I 2022-10-27 04:21:52] ...trap/engines/engine.py.224: train: epoch 47 | batch 1430/2304 [I 2022-10-27 04:21:52] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:03 [I 2022-10-27 04:21:52] ...trap/engines/engine.py.229: process: 0.56097 | load: 0.00246 [I 2022-10-27 04:21:52] ...trap/engines/engine.py.230: loss: 0.13477 [I 2022-10-27 04:21:52] ...optimizers/trijoint.py.151: total_norm: 26.101156 [I 2022-10-27 04:21:58] ...trap/engines/engine.py.224: train: epoch 47 | batch 1440/2304 [I 2022-10-27 04:21:58] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:03 [I 2022-10-27 04:21:58] ...trap/engines/engine.py.229: process: 0.57080 | load: 0.00242 [I 2022-10-27 04:21:58] ...trap/engines/engine.py.230: loss: 0.13673 [I 2022-10-27 04:21:58] ...optimizers/trijoint.py.151: total_norm: 18.376356 [I 2022-10-27 04:22:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 1450/2304 [I 2022-10-27 04:22:04] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:59 [I 2022-10-27 04:22:04] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00244 [I 2022-10-27 04:22:04] ...trap/engines/engine.py.230: loss: 0.14466 [I 2022-10-27 04:22:04] ...optimizers/trijoint.py.151: total_norm: 23.149025 [I 2022-10-27 04:22:09] ...trap/engines/engine.py.224: train: epoch 47 | batch 1460/2304 [I 2022-10-27 04:22:09] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:51 [I 2022-10-27 04:22:09] ...trap/engines/engine.py.229: process: 0.56372 | load: 0.00263 [I 2022-10-27 04:22:09] ...trap/engines/engine.py.230: loss: 0.16334 [I 2022-10-27 04:22:09] ...optimizers/trijoint.py.151: total_norm: 11.872330 [I 2022-10-27 04:22:15] ...trap/engines/engine.py.224: train: epoch 47 | batch 1470/2304 [I 2022-10-27 04:22:15] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:38 [I 2022-10-27 04:22:15] ...trap/engines/engine.py.229: process: 0.53651 | load: 0.00243 [I 2022-10-27 04:22:15] ...trap/engines/engine.py.230: loss: 0.14965 [I 2022-10-27 04:22:15] ...optimizers/trijoint.py.151: total_norm: 7.965667 [I 2022-10-27 04:22:20] ...trap/engines/engine.py.224: train: epoch 47 | batch 1480/2304 [I 2022-10-27 04:22:20] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:37 [I 2022-10-27 04:22:20] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00245 [I 2022-10-27 04:22:20] ...trap/engines/engine.py.230: loss: 0.17239 [I 2022-10-27 04:22:20] ...optimizers/trijoint.py.151: total_norm: 31.979692 [I 2022-10-27 04:22:26] ...trap/engines/engine.py.224: train: epoch 47 | batch 1490/2304 [I 2022-10-27 04:22:26] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:34 [I 2022-10-27 04:22:26] ...trap/engines/engine.py.229: process: 0.56530 | load: 0.00243 [I 2022-10-27 04:22:26] ...trap/engines/engine.py.230: loss: 0.12582 [I 2022-10-27 04:22:26] ...optimizers/trijoint.py.151: total_norm: 13.528225 [I 2022-10-27 04:22:31] ...trap/engines/engine.py.224: train: epoch 47 | batch 1500/2304 [I 2022-10-27 04:22:31] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:27 [I 2022-10-27 04:22:31] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00245 [I 2022-10-27 04:22:31] ...trap/engines/engine.py.230: loss: 0.27430 [I 2022-10-27 04:22:31] ...optimizers/trijoint.py.151: total_norm: 26.729986 [I 2022-10-27 04:22:37] ...trap/engines/engine.py.224: train: epoch 47 | batch 1510/2304 [I 2022-10-27 04:22:37] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:19 [I 2022-10-27 04:22:37] ...trap/engines/engine.py.229: process: 0.54622 | load: 0.00259 [I 2022-10-27 04:22:37] ...trap/engines/engine.py.230: loss: 0.17185 [I 2022-10-27 04:22:37] ...optimizers/trijoint.py.151: total_norm: 19.817362 [I 2022-10-27 04:22:43] ...trap/engines/engine.py.224: train: epoch 47 | batch 1520/2304 [I 2022-10-27 04:22:43] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:17 [I 2022-10-27 04:22:43] ...trap/engines/engine.py.229: process: 0.55544 | load: 0.00243 [I 2022-10-27 04:22:43] ...trap/engines/engine.py.230: loss: 0.21517 [I 2022-10-27 04:22:43] ...optimizers/trijoint.py.151: total_norm: 17.826405 [I 2022-10-27 04:22:48] ...trap/engines/engine.py.224: train: epoch 47 | batch 1530/2304 [I 2022-10-27 04:22:48] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:08 [I 2022-10-27 04:22:48] ...trap/engines/engine.py.229: process: 0.54767 | load: 0.00240 [I 2022-10-27 04:22:48] ...trap/engines/engine.py.230: loss: 0.22775 [I 2022-10-27 04:22:48] ...optimizers/trijoint.py.151: total_norm: 23.749571 [I 2022-10-27 04:22:54] ...trap/engines/engine.py.224: train: epoch 47 | batch 1540/2304 [I 2022-10-27 04:22:54] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:05 [I 2022-10-27 04:22:54] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00242 [I 2022-10-27 04:22:54] ...trap/engines/engine.py.230: loss: 0.23151 [I 2022-10-27 04:22:54] ...optimizers/trijoint.py.151: total_norm: 13.041939 [I 2022-10-27 04:22:59] ...trap/engines/engine.py.224: train: epoch 47 | batch 1550/2304 [I 2022-10-27 04:22:59] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:00 [I 2022-10-27 04:22:59] ...trap/engines/engine.py.229: process: 0.56306 | load: 0.00242 [I 2022-10-27 04:22:59] ...trap/engines/engine.py.230: loss: 0.21398 [I 2022-10-27 04:22:59] ...optimizers/trijoint.py.151: total_norm: 16.257742 [I 2022-10-27 04:23:05] ...trap/engines/engine.py.224: train: epoch 47 | batch 1560/2304 [I 2022-10-27 04:23:05] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:06:53 [I 2022-10-27 04:23:05] ...trap/engines/engine.py.229: process: 0.54628 | load: 0.00252 [I 2022-10-27 04:23:05] ...trap/engines/engine.py.230: loss: 0.13498 [I 2022-10-27 04:23:05] ...optimizers/trijoint.py.151: total_norm: 15.728540 [I 2022-10-27 04:23:10] ...trap/engines/engine.py.224: train: epoch 47 | batch 1570/2304 [I 2022-10-27 04:23:10] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:48 [I 2022-10-27 04:23:10] ...trap/engines/engine.py.229: process: 0.56568 | load: 0.00246 [I 2022-10-27 04:23:10] ...trap/engines/engine.py.230: loss: 0.14501 [I 2022-10-27 04:23:10] ...optimizers/trijoint.py.151: total_norm: 12.220543 [I 2022-10-27 04:23:16] ...trap/engines/engine.py.224: train: epoch 47 | batch 1580/2304 [I 2022-10-27 04:23:16] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:44 [I 2022-10-27 04:23:16] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00245 [I 2022-10-27 04:23:16] ...trap/engines/engine.py.230: loss: 0.30148 [I 2022-10-27 04:23:16] ...optimizers/trijoint.py.151: total_norm: 24.637457 [I 2022-10-27 04:23:21] ...trap/engines/engine.py.224: train: epoch 47 | batch 1590/2304 [I 2022-10-27 04:23:21] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:35 [I 2022-10-27 04:23:21] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00245 [I 2022-10-27 04:23:21] ...trap/engines/engine.py.230: loss: 0.17371 [I 2022-10-27 04:23:21] ...optimizers/trijoint.py.151: total_norm: 11.642099 [I 2022-10-27 04:23:27] ...trap/engines/engine.py.224: train: epoch 47 | batch 1600/2304 [I 2022-10-27 04:23:27] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:32 [I 2022-10-27 04:23:27] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00243 [I 2022-10-27 04:23:27] ...trap/engines/engine.py.230: loss: 0.20012 [I 2022-10-27 04:23:27] ...optimizers/trijoint.py.151: total_norm: 9.532632 [I 2022-10-27 04:23:33] ...trap/engines/engine.py.224: train: epoch 47 | batch 1610/2304 [I 2022-10-27 04:23:33] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:26 [I 2022-10-27 04:23:33] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00246 [I 2022-10-27 04:23:33] ...trap/engines/engine.py.230: loss: 0.10882 [I 2022-10-27 04:23:33] ...optimizers/trijoint.py.151: total_norm: 12.950416 [I 2022-10-27 04:23:38] ...trap/engines/engine.py.224: train: epoch 47 | batch 1620/2304 [I 2022-10-27 04:23:38] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:22 [I 2022-10-27 04:23:38] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00243 [I 2022-10-27 04:23:38] ...trap/engines/engine.py.230: loss: 0.18497 [I 2022-10-27 04:23:38] ...optimizers/trijoint.py.151: total_norm: 19.027555 [I 2022-10-27 04:23:44] ...trap/engines/engine.py.224: train: epoch 47 | batch 1630/2304 [I 2022-10-27 04:23:44] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:15 [I 2022-10-27 04:23:44] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00244 [I 2022-10-27 04:23:44] ...trap/engines/engine.py.230: loss: 0.14500 [I 2022-10-27 04:23:44] ...optimizers/trijoint.py.151: total_norm: 24.114922 [I 2022-10-27 04:23:49] ...trap/engines/engine.py.224: train: epoch 47 | batch 1640/2304 [I 2022-10-27 04:23:49] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:09 [I 2022-10-27 04:23:49] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00249 [I 2022-10-27 04:23:49] ...trap/engines/engine.py.230: loss: 0.15047 [I 2022-10-27 04:23:49] ...optimizers/trijoint.py.151: total_norm: 24.416653 [I 2022-10-27 04:23:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 1650/2304 [I 2022-10-27 04:23:55] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:06 [I 2022-10-27 04:23:55] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00250 [I 2022-10-27 04:23:55] ...trap/engines/engine.py.230: loss: 0.17012 [I 2022-10-27 04:23:55] ...optimizers/trijoint.py.151: total_norm: 9.456228 [I 2022-10-27 04:24:01] ...trap/engines/engine.py.224: train: epoch 47 | batch 1660/2304 [I 2022-10-27 04:24:01] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:05:58 [I 2022-10-27 04:24:01] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00246 [I 2022-10-27 04:24:01] ...trap/engines/engine.py.230: loss: 0.30195 [I 2022-10-27 04:24:01] ...optimizers/trijoint.py.151: total_norm: 26.846361 [I 2022-10-27 04:24:06] ...trap/engines/engine.py.224: train: epoch 47 | batch 1670/2304 [I 2022-10-27 04:24:06] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:55 [I 2022-10-27 04:24:06] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00242 [I 2022-10-27 04:24:06] ...trap/engines/engine.py.230: loss: 0.26383 [I 2022-10-27 04:24:06] ...optimizers/trijoint.py.151: total_norm: 21.317993 [I 2022-10-27 04:24:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 1680/2304 [I 2022-10-27 04:24:12] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:48 [I 2022-10-27 04:24:12] ...trap/engines/engine.py.229: process: 0.57407 | load: 0.00245 [I 2022-10-27 04:24:12] ...trap/engines/engine.py.230: loss: 0.18929 [I 2022-10-27 04:24:12] ...optimizers/trijoint.py.151: total_norm: 15.040672 [I 2022-10-27 04:24:17] ...trap/engines/engine.py.224: train: epoch 47 | batch 1690/2304 [I 2022-10-27 04:24:17] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:43 [I 2022-10-27 04:24:17] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00244 [I 2022-10-27 04:24:17] ...trap/engines/engine.py.230: loss: 0.20141 [I 2022-10-27 04:24:17] ...optimizers/trijoint.py.151: total_norm: 48.629913 [I 2022-10-27 04:24:23] ...trap/engines/engine.py.224: train: epoch 47 | batch 1700/2304 [I 2022-10-27 04:24:23] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:36 [I 2022-10-27 04:24:23] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00249 [I 2022-10-27 04:24:23] ...trap/engines/engine.py.230: loss: 0.15025 [I 2022-10-27 04:24:23] ...optimizers/trijoint.py.151: total_norm: 13.043670 [I 2022-10-27 04:24:28] ...trap/engines/engine.py.224: train: epoch 47 | batch 1710/2304 [I 2022-10-27 04:24:28] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:29 [I 2022-10-27 04:24:28] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00261 [I 2022-10-27 04:24:28] ...trap/engines/engine.py.230: loss: 0.17066 [I 2022-10-27 04:24:28] ...optimizers/trijoint.py.151: total_norm: 13.145305 [I 2022-10-27 04:24:34] ...trap/engines/engine.py.224: train: epoch 47 | batch 1720/2304 [I 2022-10-27 04:24:34] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:25 [I 2022-10-27 04:24:34] ...trap/engines/engine.py.229: process: 0.55095 | load: 0.00241 [I 2022-10-27 04:24:34] ...trap/engines/engine.py.230: loss: 0.15889 [I 2022-10-27 04:24:34] ...optimizers/trijoint.py.151: total_norm: 22.800037 [I 2022-10-27 04:24:40] ...trap/engines/engine.py.224: train: epoch 47 | batch 1730/2304 [I 2022-10-27 04:24:40] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:18 [I 2022-10-27 04:24:40] ...trap/engines/engine.py.229: process: 0.54944 | load: 0.00242 [I 2022-10-27 04:24:40] ...trap/engines/engine.py.230: loss: 0.14567 [I 2022-10-27 04:24:40] ...optimizers/trijoint.py.151: total_norm: 24.378805 [I 2022-10-27 04:24:45] ...trap/engines/engine.py.224: train: epoch 47 | batch 1740/2304 [I 2022-10-27 04:24:45] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:14 [I 2022-10-27 04:24:45] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00250 [I 2022-10-27 04:24:45] ...trap/engines/engine.py.230: loss: 0.22141 [I 2022-10-27 04:24:45] ...optimizers/trijoint.py.151: total_norm: 22.934361 [I 2022-10-27 04:24:51] ...trap/engines/engine.py.224: train: epoch 47 | batch 1750/2304 [I 2022-10-27 04:24:51] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:09 [I 2022-10-27 04:24:51] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00246 [I 2022-10-27 04:24:51] ...trap/engines/engine.py.230: loss: 0.17752 [I 2022-10-27 04:24:51] ...optimizers/trijoint.py.151: total_norm: 27.619537 [I 2022-10-27 04:24:56] ...trap/engines/engine.py.224: train: epoch 47 | batch 1760/2304 [I 2022-10-27 04:24:56] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:03 [I 2022-10-27 04:24:56] ...trap/engines/engine.py.229: process: 0.56525 | load: 0.00245 [I 2022-10-27 04:24:56] ...trap/engines/engine.py.230: loss: 0.25249 [I 2022-10-27 04:24:56] ...optimizers/trijoint.py.151: total_norm: 23.343159 [I 2022-10-27 04:25:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 1770/2304 [I 2022-10-27 04:25:02] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:04:56 [I 2022-10-27 04:25:02] ...trap/engines/engine.py.229: process: 0.55335 | load: 0.00244 [I 2022-10-27 04:25:02] ...trap/engines/engine.py.230: loss: 0.13223 [I 2022-10-27 04:25:02] ...optimizers/trijoint.py.151: total_norm: 17.980530 [I 2022-10-27 04:25:08] ...trap/engines/engine.py.224: train: epoch 47 | batch 1780/2304 [I 2022-10-27 04:25:08] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:04:53 [I 2022-10-27 04:25:08] ...trap/engines/engine.py.229: process: 0.57043 | load: 0.00249 [I 2022-10-27 04:25:08] ...trap/engines/engine.py.230: loss: 0.23110 [I 2022-10-27 04:25:08] ...optimizers/trijoint.py.151: total_norm: 13.539369 [I 2022-10-27 04:25:13] ...trap/engines/engine.py.224: train: epoch 47 | batch 1790/2304 [I 2022-10-27 04:25:13] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:47 [I 2022-10-27 04:25:13] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00245 [I 2022-10-27 04:25:13] ...trap/engines/engine.py.230: loss: 0.19747 [I 2022-10-27 04:25:13] ...optimizers/trijoint.py.151: total_norm: 15.340276 [I 2022-10-27 04:25:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 1800/2304 [I 2022-10-27 04:25:19] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:04:38 [I 2022-10-27 04:25:19] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00244 [I 2022-10-27 04:25:19] ...trap/engines/engine.py.230: loss: 0.11143 [I 2022-10-27 04:25:19] ...optimizers/trijoint.py.151: total_norm: 18.147753 [I 2022-10-27 04:25:24] ...trap/engines/engine.py.224: train: epoch 47 | batch 1810/2304 [I 2022-10-27 04:25:24] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:36 [I 2022-10-27 04:25:24] ...trap/engines/engine.py.229: process: 0.55437 | load: 0.00247 [I 2022-10-27 04:25:24] ...trap/engines/engine.py.230: loss: 0.14314 [I 2022-10-27 04:25:24] ...optimizers/trijoint.py.151: total_norm: 13.290102 [I 2022-10-27 04:25:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 1820/2304 [I 2022-10-27 04:25:30] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:30 [I 2022-10-27 04:25:30] ...trap/engines/engine.py.229: process: 0.56539 | load: 0.00248 [I 2022-10-27 04:25:30] ...trap/engines/engine.py.230: loss: 0.14648 [I 2022-10-27 04:25:30] ...optimizers/trijoint.py.151: total_norm: 14.869207 [I 2022-10-27 04:25:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 1830/2304 [I 2022-10-27 04:25:36] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:24 [I 2022-10-27 04:25:36] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00244 [I 2022-10-27 04:25:36] ...trap/engines/engine.py.230: loss: 0.19767 [I 2022-10-27 04:25:36] ...optimizers/trijoint.py.151: total_norm: 19.528669 [I 2022-10-27 04:25:41] ...trap/engines/engine.py.224: train: epoch 47 | batch 1840/2304 [I 2022-10-27 04:25:41] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:22 [I 2022-10-27 04:25:41] ...trap/engines/engine.py.229: process: 0.55895 | load: 0.00241 [I 2022-10-27 04:25:41] ...trap/engines/engine.py.230: loss: 0.15025 [I 2022-10-27 04:25:41] ...optimizers/trijoint.py.151: total_norm: 14.865959 [I 2022-10-27 04:25:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 1850/2304 [I 2022-10-27 04:25:47] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:14 [I 2022-10-27 04:25:47] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00241 [I 2022-10-27 04:25:47] ...trap/engines/engine.py.230: loss: 0.14402 [I 2022-10-27 04:25:47] ...optimizers/trijoint.py.151: total_norm: 20.752234 [I 2022-10-27 04:25:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 1860/2304 [I 2022-10-27 04:25:53] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:09 [I 2022-10-27 04:25:53] ...trap/engines/engine.py.229: process: 0.54685 | load: 0.00260 [I 2022-10-27 04:25:53] ...trap/engines/engine.py.230: loss: 0.21494 [I 2022-10-27 04:25:53] ...optimizers/trijoint.py.151: total_norm: 57.022957 [I 2022-10-27 04:25:58] ...trap/engines/engine.py.224: train: epoch 47 | batch 1870/2304 [I 2022-10-27 04:25:58] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:00 [I 2022-10-27 04:25:58] ...trap/engines/engine.py.229: process: 0.54965 | load: 0.00244 [I 2022-10-27 04:25:58] ...trap/engines/engine.py.230: loss: 0.18343 [I 2022-10-27 04:25:58] ...optimizers/trijoint.py.151: total_norm: 24.394779 [I 2022-10-27 04:26:04] ...trap/engines/engine.py.224: train: epoch 47 | batch 1880/2304 [I 2022-10-27 04:26:04] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:03:55 [I 2022-10-27 04:26:04] ...trap/engines/engine.py.229: process: 0.54598 | load: 0.00241 [I 2022-10-27 04:26:04] ...trap/engines/engine.py.230: loss: 0.12728 [I 2022-10-27 04:26:04] ...optimizers/trijoint.py.151: total_norm: 12.389785 [I 2022-10-27 04:26:09] ...trap/engines/engine.py.224: train: epoch 47 | batch 1890/2304 [I 2022-10-27 04:26:09] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:03:51 [I 2022-10-27 04:26:09] ...trap/engines/engine.py.229: process: 0.56729 | load: 0.00242 [I 2022-10-27 04:26:09] ...trap/engines/engine.py.230: loss: 0.14557 [I 2022-10-27 04:26:09] ...optimizers/trijoint.py.151: total_norm: 22.333004 [I 2022-10-27 04:26:15] ...trap/engines/engine.py.224: train: epoch 47 | batch 1900/2304 [I 2022-10-27 04:26:15] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:03:45 [I 2022-10-27 04:26:15] ...trap/engines/engine.py.229: process: 0.56288 | load: 0.00242 [I 2022-10-27 04:26:15] ...trap/engines/engine.py.230: loss: 0.17117 [I 2022-10-27 04:26:15] ...optimizers/trijoint.py.151: total_norm: 16.234087 [I 2022-10-27 04:26:20] ...trap/engines/engine.py.224: train: epoch 47 | batch 1910/2304 [I 2022-10-27 04:26:20] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:38 [I 2022-10-27 04:26:20] ...trap/engines/engine.py.229: process: 0.55337 | load: 0.00248 [I 2022-10-27 04:26:20] ...trap/engines/engine.py.230: loss: 0.18585 [I 2022-10-27 04:26:20] ...optimizers/trijoint.py.151: total_norm: 21.030935 [I 2022-10-27 04:26:26] ...trap/engines/engine.py.224: train: epoch 47 | batch 1920/2304 [I 2022-10-27 04:26:26] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:44 [I 2022-10-27 04:26:26] ...trap/engines/engine.py.229: process: 0.69515 | load: 0.00247 [I 2022-10-27 04:26:26] ...trap/engines/engine.py.230: loss: 0.14165 [I 2022-10-27 04:26:26] ...optimizers/trijoint.py.151: total_norm: 15.415678 [I 2022-10-27 04:26:32] ...trap/engines/engine.py.224: train: epoch 47 | batch 1930/2304 [I 2022-10-27 04:26:32] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:29 [I 2022-10-27 04:26:32] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00245 [I 2022-10-27 04:26:32] ...trap/engines/engine.py.230: loss: 0.18190 [I 2022-10-27 04:26:32] ...optimizers/trijoint.py.151: total_norm: 32.048340 [I 2022-10-27 04:26:37] ...trap/engines/engine.py.224: train: epoch 47 | batch 1940/2304 [I 2022-10-27 04:26:37] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:22 [I 2022-10-27 04:26:37] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00243 [I 2022-10-27 04:26:37] ...trap/engines/engine.py.230: loss: 0.12163 [I 2022-10-27 04:26:37] ...optimizers/trijoint.py.151: total_norm: 14.994824 [I 2022-10-27 04:26:43] ...trap/engines/engine.py.224: train: epoch 47 | batch 1950/2304 [I 2022-10-27 04:26:43] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:16 [I 2022-10-27 04:26:43] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00245 [I 2022-10-27 04:26:43] ...trap/engines/engine.py.230: loss: 0.22948 [I 2022-10-27 04:26:43] ...optimizers/trijoint.py.151: total_norm: 16.771851 [I 2022-10-27 04:26:48] ...trap/engines/engine.py.224: train: epoch 47 | batch 1960/2304 [I 2022-10-27 04:26:48] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:11 [I 2022-10-27 04:26:48] ...trap/engines/engine.py.229: process: 0.55153 | load: 0.00239 [I 2022-10-27 04:26:48] ...trap/engines/engine.py.230: loss: 0.18823 [I 2022-10-27 04:26:48] ...optimizers/trijoint.py.151: total_norm: 14.726063 [I 2022-10-27 04:26:54] ...trap/engines/engine.py.224: train: epoch 47 | batch 1970/2304 [I 2022-10-27 04:26:54] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:06 [I 2022-10-27 04:26:54] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00260 [I 2022-10-27 04:26:54] ...trap/engines/engine.py.230: loss: 0.18533 [I 2022-10-27 04:26:54] ...optimizers/trijoint.py.151: total_norm: 26.114458 [I 2022-10-27 04:27:00] ...trap/engines/engine.py.224: train: epoch 47 | batch 1980/2304 [I 2022-10-27 04:27:00] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:02:59 [I 2022-10-27 04:27:00] ...trap/engines/engine.py.229: process: 0.54594 | load: 0.00243 [I 2022-10-27 04:27:00] ...trap/engines/engine.py.230: loss: 0.11785 [I 2022-10-27 04:27:00] ...optimizers/trijoint.py.151: total_norm: 47.856407 [I 2022-10-27 04:27:05] ...trap/engines/engine.py.224: train: epoch 47 | batch 1990/2304 [I 2022-10-27 04:27:05] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:00 [I 2022-10-27 04:27:05] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00245 [I 2022-10-27 04:27:05] ...trap/engines/engine.py.230: loss: 0.16494 [I 2022-10-27 04:27:05] ...optimizers/trijoint.py.151: total_norm: 8.584880 [I 2022-10-27 04:27:11] ...trap/engines/engine.py.224: train: epoch 47 | batch 2000/2304 [I 2022-10-27 04:27:11] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:02:49 [I 2022-10-27 04:27:11] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00249 [I 2022-10-27 04:27:11] ...trap/engines/engine.py.230: loss: 0.13195 [I 2022-10-27 04:27:11] ...optimizers/trijoint.py.151: total_norm: 10.858307 [I 2022-10-27 04:27:16] ...trap/engines/engine.py.224: train: epoch 47 | batch 2010/2304 [I 2022-10-27 04:27:16] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:44 [I 2022-10-27 04:27:16] ...trap/engines/engine.py.229: process: 0.57005 | load: 0.00244 [I 2022-10-27 04:27:16] ...trap/engines/engine.py.230: loss: 0.12811 [I 2022-10-27 04:27:16] ...optimizers/trijoint.py.151: total_norm: 11.215577 [I 2022-10-27 04:27:22] ...trap/engines/engine.py.224: train: epoch 47 | batch 2020/2304 [I 2022-10-27 04:27:22] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:37 [I 2022-10-27 04:27:22] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00243 [I 2022-10-27 04:27:22] ...trap/engines/engine.py.230: loss: 0.15579 [I 2022-10-27 04:27:22] ...optimizers/trijoint.py.151: total_norm: 11.006960 [I 2022-10-27 04:27:28] ...trap/engines/engine.py.224: train: epoch 47 | batch 2030/2304 [I 2022-10-27 04:27:28] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:33 [I 2022-10-27 04:27:28] ...trap/engines/engine.py.229: process: 0.55309 | load: 0.00241 [I 2022-10-27 04:27:28] ...trap/engines/engine.py.230: loss: 0.10028 [I 2022-10-27 04:27:28] ...optimizers/trijoint.py.151: total_norm: 8.917505 [I 2022-10-27 04:27:33] ...trap/engines/engine.py.224: train: epoch 47 | batch 2040/2304 [I 2022-10-27 04:27:33] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:26 [I 2022-10-27 04:27:33] ...trap/engines/engine.py.229: process: 0.55578 | load: 0.00244 [I 2022-10-27 04:27:33] ...trap/engines/engine.py.230: loss: 0.17222 [I 2022-10-27 04:27:33] ...optimizers/trijoint.py.151: total_norm: 20.164515 [I 2022-10-27 04:27:39] ...trap/engines/engine.py.224: train: epoch 47 | batch 2050/2304 [I 2022-10-27 04:27:39] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:20 [I 2022-10-27 04:27:39] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00242 [I 2022-10-27 04:27:39] ...trap/engines/engine.py.230: loss: 0.18409 [I 2022-10-27 04:27:39] ...optimizers/trijoint.py.151: total_norm: 61.265934 [I 2022-10-27 04:27:44] ...trap/engines/engine.py.224: train: epoch 47 | batch 2060/2304 [I 2022-10-27 04:27:44] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:14 [I 2022-10-27 04:27:44] ...trap/engines/engine.py.229: process: 0.53747 | load: 0.00244 [I 2022-10-27 04:27:44] ...trap/engines/engine.py.230: loss: 0.14338 [I 2022-10-27 04:27:44] ...optimizers/trijoint.py.151: total_norm: 7.722403 [I 2022-10-27 04:27:50] ...trap/engines/engine.py.224: train: epoch 47 | batch 2070/2304 [I 2022-10-27 04:27:50] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:10 [I 2022-10-27 04:27:50] ...trap/engines/engine.py.229: process: 0.56422 | load: 0.00255 [I 2022-10-27 04:27:50] ...trap/engines/engine.py.230: loss: 0.19424 [I 2022-10-27 04:27:50] ...optimizers/trijoint.py.151: total_norm: 29.971514 [I 2022-10-27 04:27:55] ...trap/engines/engine.py.224: train: epoch 47 | batch 2080/2304 [I 2022-10-27 04:27:55] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:05 [I 2022-10-27 04:27:55] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00250 [I 2022-10-27 04:27:55] ...trap/engines/engine.py.230: loss: 0.15045 [I 2022-10-27 04:27:55] ...optimizers/trijoint.py.151: total_norm: 14.579189 [I 2022-10-27 04:28:01] ...trap/engines/engine.py.224: train: epoch 47 | batch 2090/2304 [I 2022-10-27 04:28:01] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:01:59 [I 2022-10-27 04:28:01] ...trap/engines/engine.py.229: process: 0.55534 | load: 0.00245 [I 2022-10-27 04:28:01] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-27 04:28:01] ...optimizers/trijoint.py.151: total_norm: 20.753237 [I 2022-10-27 04:28:07] ...trap/engines/engine.py.224: train: epoch 47 | batch 2100/2304 [I 2022-10-27 04:28:07] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:01:56 [I 2022-10-27 04:28:07] ...trap/engines/engine.py.229: process: 0.57157 | load: 0.00248 [I 2022-10-27 04:28:07] ...trap/engines/engine.py.230: loss: 0.13401 [I 2022-10-27 04:28:07] ...optimizers/trijoint.py.151: total_norm: 21.246698 [I 2022-10-27 04:28:12] ...trap/engines/engine.py.224: train: epoch 47 | batch 2110/2304 [I 2022-10-27 04:28:12] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:01:48 [I 2022-10-27 04:28:12] ...trap/engines/engine.py.229: process: 0.55257 | load: 0.00244 [I 2022-10-27 04:28:12] ...trap/engines/engine.py.230: loss: 0.15480 [I 2022-10-27 04:28:12] ...optimizers/trijoint.py.151: total_norm: 13.849966 [I 2022-10-27 04:28:18] ...trap/engines/engine.py.224: train: epoch 47 | batch 2120/2304 [I 2022-10-27 04:28:18] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:01:42 [I 2022-10-27 04:28:18] ...trap/engines/engine.py.229: process: 0.56590 | load: 0.00244 [I 2022-10-27 04:28:18] ...trap/engines/engine.py.230: loss: 0.13434 [I 2022-10-27 04:28:18] ...optimizers/trijoint.py.151: total_norm: 15.609242 [I 2022-10-27 04:28:23] ...trap/engines/engine.py.224: train: epoch 47 | batch 2130/2304 [I 2022-10-27 04:28:23] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:36 [I 2022-10-27 04:28:23] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00260 [I 2022-10-27 04:28:23] ...trap/engines/engine.py.230: loss: 0.15190 [I 2022-10-27 04:28:23] ...optimizers/trijoint.py.151: total_norm: 15.448728 [I 2022-10-27 04:28:29] ...trap/engines/engine.py.224: train: epoch 47 | batch 2140/2304 [I 2022-10-27 04:28:29] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:31 [I 2022-10-27 04:28:29] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00245 [I 2022-10-27 04:28:29] ...trap/engines/engine.py.230: loss: 0.17562 [I 2022-10-27 04:28:29] ...optimizers/trijoint.py.151: total_norm: 20.683043 [I 2022-10-27 04:28:35] ...trap/engines/engine.py.224: train: epoch 47 | batch 2150/2304 [I 2022-10-27 04:28:35] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:25 [I 2022-10-27 04:28:35] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00246 [I 2022-10-27 04:28:35] ...trap/engines/engine.py.230: loss: 0.10939 [I 2022-10-27 04:28:35] ...optimizers/trijoint.py.151: total_norm: 8.953795 [I 2022-10-27 04:28:40] ...trap/engines/engine.py.224: train: epoch 47 | batch 2160/2304 [I 2022-10-27 04:28:40] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:20 [I 2022-10-27 04:28:40] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00282 [I 2022-10-27 04:28:40] ...trap/engines/engine.py.230: loss: 0.12893 [I 2022-10-27 04:28:40] ...optimizers/trijoint.py.151: total_norm: 10.672777 [I 2022-10-27 04:28:46] ...trap/engines/engine.py.224: train: epoch 47 | batch 2170/2304 [I 2022-10-27 04:28:46] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:14 [I 2022-10-27 04:28:46] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00255 [I 2022-10-27 04:28:46] ...trap/engines/engine.py.230: loss: 0.16797 [I 2022-10-27 04:28:46] ...optimizers/trijoint.py.151: total_norm: 21.271957 [I 2022-10-27 04:28:51] ...trap/engines/engine.py.224: train: epoch 47 | batch 2180/2304 [I 2022-10-27 04:28:51] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:09 [I 2022-10-27 04:28:51] ...trap/engines/engine.py.229: process: 0.57111 | load: 0.00244 [I 2022-10-27 04:28:51] ...trap/engines/engine.py.230: loss: 0.17159 [I 2022-10-27 04:28:51] ...optimizers/trijoint.py.151: total_norm: 36.555420 [I 2022-10-27 04:28:57] ...trap/engines/engine.py.224: train: epoch 47 | batch 2190/2304 [I 2022-10-27 04:28:57] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:03 [I 2022-10-27 04:28:57] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00245 [I 2022-10-27 04:28:57] ...trap/engines/engine.py.230: loss: 0.10705 [I 2022-10-27 04:28:57] ...optimizers/trijoint.py.151: total_norm: 8.933554 [I 2022-10-27 04:29:02] ...trap/engines/engine.py.224: train: epoch 47 | batch 2200/2304 [I 2022-10-27 04:29:02] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:00:58 [I 2022-10-27 04:29:02] ...trap/engines/engine.py.229: process: 0.60936 | load: 0.00242 [I 2022-10-27 04:29:02] ...trap/engines/engine.py.230: loss: 0.27313 [I 2022-10-27 04:29:02] ...optimizers/trijoint.py.151: total_norm: 37.015659 [I 2022-10-27 04:29:08] ...trap/engines/engine.py.224: train: epoch 47 | batch 2210/2304 [I 2022-10-27 04:29:08] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:00:52 [I 2022-10-27 04:29:08] ...trap/engines/engine.py.229: process: 0.55362 | load: 0.00242 [I 2022-10-27 04:29:08] ...trap/engines/engine.py.230: loss: 0.11917 [I 2022-10-27 04:29:08] ...optimizers/trijoint.py.151: total_norm: 10.953919 [I 2022-10-27 04:29:14] ...trap/engines/engine.py.224: train: epoch 47 | batch 2220/2304 [I 2022-10-27 04:29:14] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:00:46 [I 2022-10-27 04:29:14] ...trap/engines/engine.py.229: process: 0.54900 | load: 0.00265 [I 2022-10-27 04:29:14] ...trap/engines/engine.py.230: loss: 0.11686 [I 2022-10-27 04:29:14] ...optimizers/trijoint.py.151: total_norm: 13.391455 [I 2022-10-27 04:29:19] ...trap/engines/engine.py.224: train: epoch 47 | batch 2230/2304 [I 2022-10-27 04:29:19] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:40 [I 2022-10-27 04:29:19] ...trap/engines/engine.py.229: process: 0.54902 | load: 0.00244 [I 2022-10-27 04:29:19] ...trap/engines/engine.py.230: loss: 0.22296 [I 2022-10-27 04:29:19] ...optimizers/trijoint.py.151: total_norm: 23.801714 [I 2022-10-27 04:29:25] ...trap/engines/engine.py.224: train: epoch 47 | batch 2240/2304 [I 2022-10-27 04:29:25] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:35 [I 2022-10-27 04:29:25] ...trap/engines/engine.py.229: process: 0.55233 | load: 0.00269 [I 2022-10-27 04:29:25] ...trap/engines/engine.py.230: loss: 0.21957 [I 2022-10-27 04:29:25] ...optimizers/trijoint.py.151: total_norm: 23.588345 [I 2022-10-27 04:29:30] ...trap/engines/engine.py.224: train: epoch 47 | batch 2250/2304 [I 2022-10-27 04:29:30] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:29 [I 2022-10-27 04:29:30] ...trap/engines/engine.py.229: process: 0.54397 | load: 0.00243 [I 2022-10-27 04:29:30] ...trap/engines/engine.py.230: loss: 0.15344 [I 2022-10-27 04:29:30] ...optimizers/trijoint.py.151: total_norm: 12.237689 [I 2022-10-27 04:29:36] ...trap/engines/engine.py.224: train: epoch 47 | batch 2260/2304 [I 2022-10-27 04:29:36] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:24 [I 2022-10-27 04:29:36] ...trap/engines/engine.py.229: process: 0.56188 | load: 0.00239 [I 2022-10-27 04:29:36] ...trap/engines/engine.py.230: loss: 0.17659 [I 2022-10-27 04:29:36] ...optimizers/trijoint.py.151: total_norm: 24.088163 [I 2022-10-27 04:29:41] ...trap/engines/engine.py.224: train: epoch 47 | batch 2270/2304 [I 2022-10-27 04:29:41] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:18 [I 2022-10-27 04:29:41] ...trap/engines/engine.py.229: process: 0.55032 | load: 0.00247 [I 2022-10-27 04:29:42] ...trap/engines/engine.py.230: loss: 0.21863 [I 2022-10-27 04:29:42] ...optimizers/trijoint.py.151: total_norm: 17.329620 [I 2022-10-27 04:29:47] ...trap/engines/engine.py.224: train: epoch 47 | batch 2280/2304 [I 2022-10-27 04:29:47] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:13 [I 2022-10-27 04:29:47] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00244 [I 2022-10-27 04:29:47] ...trap/engines/engine.py.230: loss: 0.11781 [I 2022-10-27 04:29:47] ...optimizers/trijoint.py.151: total_norm: 10.960862 [I 2022-10-27 04:29:53] ...trap/engines/engine.py.224: train: epoch 47 | batch 2290/2304 [I 2022-10-27 04:29:53] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:07 [I 2022-10-27 04:29:53] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00245 [I 2022-10-27 04:29:53] ...trap/engines/engine.py.230: loss: 0.13129 [I 2022-10-27 04:29:53] ...optimizers/trijoint.py.151: total_norm: 13.207216 [I 2022-10-27 04:29:58] ...trap/engines/engine.py.224: train: epoch 47 | batch 2300/2304 [I 2022-10-27 04:29:58] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:02 [I 2022-10-27 04:29:58] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00014 [I 2022-10-27 04:29:58] ...trap/engines/engine.py.230: loss: 0.17830 [I 2022-10-27 04:29:58] ...optimizers/trijoint.py.151: total_norm: 38.257977 [I 2022-10-27 04:30:00] ...trap/engines/engine.py.224: train: epoch 47 | batch 2304/2304 [I 2022-10-27 04:30:00] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:00 [I 2022-10-27 04:30:00] ...trap/engines/engine.py.229: process: 0.54727 | load: 0.00015 [I 2022-10-27 04:30:00] ...trap/engines/engine.py.230: loss: 0.16945 [I 2022-10-27 04:30:00] ...optimizers/trijoint.py.151: total_norm: 20.261517 [S 2022-10-27 04:30:02] ...trap/engines/engine.py.129: train_epoch.epoch: 47 [S 2022-10-27 04:30:02] ...trap/engines/engine.py.129: train_epoch.loss: 0.16629361233263884 [I 2022-10-27 04:30:02] ...s/criterions/triplet.py.84: margin = 0.2900000000000002 [I 2022-10-27 04:30:02] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 47 [I 2022-10-27 04:30:02] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 04:30:04] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 04:30:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 04:30:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 04:30:06] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 04:30:06] ...trap/engines/engine.py.321: eval: epoch 47 | batch 0/510 [I 2022-10-27 04:30:06] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:32:21 [I 2022-10-27 04:30:06] ...trap/engines/engine.py.326: process: 3.80597 | load: 2.94583 [I 2022-10-27 04:30:09] ...trap/engines/engine.py.321: eval: epoch 47 | batch 10/510 [I 2022-10-27 04:30:09] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:05:52 [I 2022-10-27 04:30:09] ...trap/engines/engine.py.326: process: 0.32445 | load: 0.00020 [I 2022-10-27 04:30:13] ...trap/engines/engine.py.321: eval: epoch 47 | batch 20/510 [I 2022-10-27 04:30:13] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:08 [I 2022-10-27 04:30:13] ...trap/engines/engine.py.326: process: 0.32234 | load: 0.00019 [I 2022-10-27 04:30:16] ...trap/engines/engine.py.321: eval: epoch 47 | batch 30/510 [I 2022-10-27 04:30:16] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:45 [I 2022-10-27 04:30:16] ...trap/engines/engine.py.326: process: 0.33595 | load: 0.00017 [I 2022-10-27 04:30:20] ...trap/engines/engine.py.321: eval: epoch 47 | batch 40/510 [I 2022-10-27 04:30:20] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:39 [I 2022-10-27 04:30:20] ...trap/engines/engine.py.326: process: 0.35967 | load: 0.00016 [I 2022-10-27 04:30:23] ...trap/engines/engine.py.321: eval: epoch 47 | batch 50/510 [I 2022-10-27 04:30:23] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:34 [I 2022-10-27 04:30:23] ...trap/engines/engine.py.326: process: 0.32908 | load: 0.00019 [I 2022-10-27 04:30:26] ...trap/engines/engine.py.321: eval: epoch 47 | batch 60/510 [I 2022-10-27 04:30:26] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:30 [I 2022-10-27 04:30:26] ...trap/engines/engine.py.326: process: 0.34061 | load: 0.00022 [I 2022-10-27 04:30:30] ...trap/engines/engine.py.321: eval: epoch 47 | batch 70/510 [I 2022-10-27 04:30:30] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:26 [I 2022-10-27 04:30:30] ...trap/engines/engine.py.326: process: 0.32230 | load: 0.00017 [I 2022-10-27 04:30:33] ...trap/engines/engine.py.321: eval: epoch 47 | batch 80/510 [I 2022-10-27 04:30:33] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:24 [I 2022-10-27 04:30:33] ...trap/engines/engine.py.326: process: 0.32910 | load: 0.00018 [I 2022-10-27 04:30:36] ...trap/engines/engine.py.321: eval: epoch 47 | batch 90/510 [I 2022-10-27 04:30:36] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:22 [I 2022-10-27 04:30:36] ...trap/engines/engine.py.326: process: 0.34523 | load: 0.00020 [I 2022-10-27 04:30:40] ...trap/engines/engine.py.321: eval: epoch 47 | batch 100/510 [I 2022-10-27 04:30:40] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:15 [I 2022-10-27 04:30:40] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00022 [I 2022-10-27 04:30:43] ...trap/engines/engine.py.321: eval: epoch 47 | batch 110/510 [I 2022-10-27 04:30:43] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:02:00 [I 2022-10-27 04:30:43] ...trap/engines/engine.py.326: process: 0.30514 | load: 0.00018 [I 2022-10-27 04:30:46] ...trap/engines/engine.py.321: eval: epoch 47 | batch 120/510 [I 2022-10-27 04:30:46] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:53 [I 2022-10-27 04:30:46] ...trap/engines/engine.py.326: process: 0.29046 | load: 0.00022 [I 2022-10-27 04:30:49] ...trap/engines/engine.py.321: eval: epoch 47 | batch 130/510 [I 2022-10-27 04:30:49] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:50 [I 2022-10-27 04:30:49] ...trap/engines/engine.py.326: process: 0.28718 | load: 0.00019 [I 2022-10-27 04:30:51] ...trap/engines/engine.py.321: eval: epoch 47 | batch 140/510 [I 2022-10-27 04:30:51] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 04:30:51] ...trap/engines/engine.py.326: process: 0.28912 | load: 0.00017 [I 2022-10-27 04:30:54] ...trap/engines/engine.py.321: eval: epoch 47 | batch 150/510 [I 2022-10-27 04:30:54] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 04:30:54] ...trap/engines/engine.py.326: process: 0.28720 | load: 0.00019 [I 2022-10-27 04:30:57] ...trap/engines/engine.py.321: eval: epoch 47 | batch 160/510 [I 2022-10-27 04:30:57] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 04:30:57] ...trap/engines/engine.py.326: process: 0.29196 | load: 0.00019 [I 2022-10-27 04:31:00] ...trap/engines/engine.py.321: eval: epoch 47 | batch 170/510 [I 2022-10-27 04:31:00] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:39 [I 2022-10-27 04:31:00] ...trap/engines/engine.py.326: process: 0.29328 | load: 0.00021 [I 2022-10-27 04:31:03] ...trap/engines/engine.py.321: eval: epoch 47 | batch 180/510 [I 2022-10-27 04:31:03] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 04:31:03] ...trap/engines/engine.py.326: process: 0.28235 | load: 0.00015 [I 2022-10-27 04:31:06] ...trap/engines/engine.py.321: eval: epoch 47 | batch 190/510 [I 2022-10-27 04:31:06] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:33 [I 2022-10-27 04:31:06] ...trap/engines/engine.py.326: process: 0.29014 | load: 0.00022 [I 2022-10-27 04:31:09] ...trap/engines/engine.py.321: eval: epoch 47 | batch 200/510 [I 2022-10-27 04:31:09] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:30 [I 2022-10-27 04:31:09] ...trap/engines/engine.py.326: process: 0.29725 | load: 0.00014 [I 2022-10-27 04:31:12] ...trap/engines/engine.py.321: eval: epoch 47 | batch 210/510 [I 2022-10-27 04:31:12] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 04:31:12] ...trap/engines/engine.py.326: process: 0.28216 | load: 0.00015 [I 2022-10-27 04:31:15] ...trap/engines/engine.py.321: eval: epoch 47 | batch 220/510 [I 2022-10-27 04:31:15] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 04:31:15] ...trap/engines/engine.py.326: process: 0.28616 | load: 0.00015 [I 2022-10-27 04:31:18] ...trap/engines/engine.py.321: eval: epoch 47 | batch 230/510 [I 2022-10-27 04:31:18] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 04:31:18] ...trap/engines/engine.py.326: process: 0.29414 | load: 0.00015 [I 2022-10-27 04:31:21] ...trap/engines/engine.py.321: eval: epoch 47 | batch 240/510 [I 2022-10-27 04:31:21] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 04:31:21] ...trap/engines/engine.py.326: process: 0.28890 | load: 0.00015 [I 2022-10-27 04:31:23] ...trap/engines/engine.py.321: eval: epoch 47 | batch 250/510 [I 2022-10-27 04:31:23] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 04:31:23] ...trap/engines/engine.py.326: process: 0.28601 | load: 0.00015 [I 2022-10-27 04:31:26] ...trap/engines/engine.py.321: eval: epoch 47 | batch 260/510 [I 2022-10-27 04:31:26] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 04:31:26] ...trap/engines/engine.py.326: process: 0.29103 | load: 0.00017 [I 2022-10-27 04:31:29] ...trap/engines/engine.py.321: eval: epoch 47 | batch 270/510 [I 2022-10-27 04:31:29] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-27 04:31:29] ...trap/engines/engine.py.326: process: 0.28834 | load: 0.00018 [I 2022-10-27 04:31:32] ...trap/engines/engine.py.321: eval: epoch 47 | batch 280/510 [I 2022-10-27 04:31:32] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 04:31:32] ...trap/engines/engine.py.326: process: 0.30189 | load: 0.00015 [I 2022-10-27 04:31:35] ...trap/engines/engine.py.321: eval: epoch 47 | batch 290/510 [I 2022-10-27 04:31:35] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:04 [I 2022-10-27 04:31:35] ...trap/engines/engine.py.326: process: 0.30349 | load: 0.00016 [I 2022-10-27 04:31:38] ...trap/engines/engine.py.321: eval: epoch 47 | batch 300/510 [I 2022-10-27 04:31:38] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 04:31:38] ...trap/engines/engine.py.326: process: 0.29014 | load: 0.00014 [I 2022-10-27 04:31:41] ...trap/engines/engine.py.321: eval: epoch 47 | batch 310/510 [I 2022-10-27 04:31:41] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 04:31:41] ...trap/engines/engine.py.326: process: 0.29238 | load: 0.00015 [I 2022-10-27 04:31:44] ...trap/engines/engine.py.321: eval: epoch 47 | batch 320/510 [I 2022-10-27 04:31:44] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 04:31:44] ...trap/engines/engine.py.326: process: 0.28550 | load: 0.00015 [I 2022-10-27 04:31:46] ...trap/engines/engine.py.321: eval: epoch 47 | batch 330/510 [I 2022-10-27 04:31:46] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 04:31:47] ...trap/engines/engine.py.326: process: 0.28802 | load: 0.00017 [I 2022-10-27 04:31:49] ...trap/engines/engine.py.321: eval: epoch 47 | batch 340/510 [I 2022-10-27 04:31:49] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 04:31:49] ...trap/engines/engine.py.326: process: 0.28357 | load: 0.00020 [I 2022-10-27 04:31:52] ...trap/engines/engine.py.321: eval: epoch 47 | batch 350/510 [I 2022-10-27 04:31:52] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 04:31:52] ...trap/engines/engine.py.326: process: 0.29671 | load: 0.00015 [I 2022-10-27 04:31:55] ...trap/engines/engine.py.321: eval: epoch 47 | batch 360/510 [I 2022-10-27 04:31:55] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 04:31:55] ...trap/engines/engine.py.326: process: 0.28512 | load: 0.00016 [I 2022-10-27 04:31:58] ...trap/engines/engine.py.321: eval: epoch 47 | batch 370/510 [I 2022-10-27 04:31:58] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 04:31:58] ...trap/engines/engine.py.326: process: 0.28497 | load: 0.00014 [I 2022-10-27 04:32:01] ...trap/engines/engine.py.321: eval: epoch 47 | batch 380/510 [I 2022-10-27 04:32:01] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 04:32:01] ...trap/engines/engine.py.326: process: 0.28954 | load: 0.00020 [I 2022-10-27 04:32:04] ...trap/engines/engine.py.321: eval: epoch 47 | batch 390/510 [I 2022-10-27 04:32:04] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 04:32:04] ...trap/engines/engine.py.326: process: 0.30435 | load: 0.00014 [I 2022-10-27 04:32:07] ...trap/engines/engine.py.321: eval: epoch 47 | batch 400/510 [I 2022-10-27 04:32:07] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 04:32:07] ...trap/engines/engine.py.326: process: 0.29073 | load: 0.00015 [I 2022-10-27 04:32:10] ...trap/engines/engine.py.321: eval: epoch 47 | batch 410/510 [I 2022-10-27 04:32:10] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 04:32:10] ...trap/engines/engine.py.326: process: 0.28516 | load: 0.00014 [I 2022-10-27 04:32:12] ...trap/engines/engine.py.321: eval: epoch 47 | batch 420/510 [I 2022-10-27 04:32:12] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 04:32:12] ...trap/engines/engine.py.326: process: 0.28405 | load: 0.00016 [I 2022-10-27 04:32:15] ...trap/engines/engine.py.321: eval: epoch 47 | batch 430/510 [I 2022-10-27 04:32:15] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 04:32:15] ...trap/engines/engine.py.326: process: 0.29318 | load: 0.00016 [I 2022-10-27 04:32:18] ...trap/engines/engine.py.321: eval: epoch 47 | batch 440/510 [I 2022-10-27 04:32:18] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 04:32:18] ...trap/engines/engine.py.326: process: 0.28903 | load: 0.00015 [I 2022-10-27 04:32:21] ...trap/engines/engine.py.321: eval: epoch 47 | batch 450/510 [I 2022-10-27 04:32:21] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 04:32:21] ...trap/engines/engine.py.326: process: 0.28655 | load: 0.00020 [I 2022-10-27 04:32:24] ...trap/engines/engine.py.321: eval: epoch 47 | batch 460/510 [I 2022-10-27 04:32:24] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 04:32:24] ...trap/engines/engine.py.326: process: 0.28813 | load: 0.00015 [I 2022-10-27 04:32:27] ...trap/engines/engine.py.321: eval: epoch 47 | batch 470/510 [I 2022-10-27 04:32:27] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 04:32:27] ...trap/engines/engine.py.326: process: 0.29006 | load: 0.00015 [I 2022-10-27 04:32:30] ...trap/engines/engine.py.321: eval: epoch 47 | batch 480/510 [I 2022-10-27 04:32:30] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 04:32:30] ...trap/engines/engine.py.326: process: 0.30075 | load: 0.00016 [I 2022-10-27 04:32:33] ...trap/engines/engine.py.321: eval: epoch 47 | batch 490/510 [I 2022-10-27 04:32:33] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 04:32:33] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00014 [I 2022-10-27 04:32:36] ...trap/engines/engine.py.321: eval: epoch 47 | batch 500/510 [I 2022-10-27 04:32:36] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 04:32:36] ...trap/engines/engine.py.326: process: 0.28926 | load: 0.00014 [I 2022-10-27 04:32:38] ...trap/engines/engine.py.321: eval: epoch 47 | batch 510/510 [I 2022-10-27 04:32:38] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 04:32:38] ...trap/engines/engine.py.326: process: 0.28789 | load: 0.00011 [S 2022-10-27 04:32:39] ...trap/engines/engine.py.132: eval_epoch.epoch: 47 [I 2022-10-27 04:32:39] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.695800 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.901000 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936600 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.685200 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.896300 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931900 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013934 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005797 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.003878 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014586 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006133 [S 2022-10-27 04:32:43] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005166 [I 2022-10-27 04:32:44] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 04:32:44] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 04:32:45] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 04:32:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 04:32:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 04:32:46] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 04:32:49] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 04:32:55] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 04:32:55] ...trap/engines/engine.py.163: Training model on trainset for epoch 48 [I 2022-10-27 04:32:55] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 04:32:58] ...trap/engines/engine.py.224: train: epoch 48 | batch 0/2304 [I 2022-10-27 04:32:58] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:49 [I 2022-10-27 04:32:58] ...trap/engines/engine.py.229: process: 2.70364 | load: 2.08294 [I 2022-10-27 04:32:58] ...trap/engines/engine.py.230: loss: 0.19867 [I 2022-10-27 04:32:58] ...optimizers/trijoint.py.151: total_norm: 14.205376 [I 2022-10-27 04:33:03] ...trap/engines/engine.py.224: train: epoch 48 | batch 10/2304 [I 2022-10-27 04:33:03] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:02 [I 2022-10-27 04:33:03] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00405 [I 2022-10-27 04:33:03] ...trap/engines/engine.py.230: loss: 0.13682 [I 2022-10-27 04:33:03] ...optimizers/trijoint.py.151: total_norm: 15.831185 [I 2022-10-27 04:33:09] ...trap/engines/engine.py.224: train: epoch 48 | batch 20/2304 [I 2022-10-27 04:33:09] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:12 [I 2022-10-27 04:33:09] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00376 [I 2022-10-27 04:33:09] ...trap/engines/engine.py.230: loss: 0.19750 [I 2022-10-27 04:33:09] ...optimizers/trijoint.py.151: total_norm: 14.912331 [I 2022-10-27 04:33:14] ...trap/engines/engine.py.224: train: epoch 48 | batch 30/2304 [I 2022-10-27 04:33:14] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:02 [I 2022-10-27 04:33:14] ...trap/engines/engine.py.229: process: 0.54674 | load: 0.00290 [I 2022-10-27 04:33:14] ...trap/engines/engine.py.230: loss: 0.26245 [I 2022-10-27 04:33:14] ...optimizers/trijoint.py.151: total_norm: 17.512096 [I 2022-10-27 04:33:20] ...trap/engines/engine.py.224: train: epoch 48 | batch 40/2304 [I 2022-10-27 04:33:20] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:46 [I 2022-10-27 04:33:20] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00275 [I 2022-10-27 04:33:20] ...trap/engines/engine.py.230: loss: 0.14368 [I 2022-10-27 04:33:20] ...optimizers/trijoint.py.151: total_norm: 17.647833 [I 2022-10-27 04:33:26] ...trap/engines/engine.py.224: train: epoch 48 | batch 50/2304 [I 2022-10-27 04:33:26] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:55 [I 2022-10-27 04:33:26] ...trap/engines/engine.py.229: process: 0.55171 | load: 0.00339 [I 2022-10-27 04:33:26] ...trap/engines/engine.py.230: loss: 0.12921 [I 2022-10-27 04:33:26] ...optimizers/trijoint.py.151: total_norm: 17.662592 [I 2022-10-27 04:33:31] ...trap/engines/engine.py.224: train: epoch 48 | batch 60/2304 [I 2022-10-27 04:33:31] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:47 [I 2022-10-27 04:33:31] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00279 [I 2022-10-27 04:33:31] ...trap/engines/engine.py.230: loss: 0.17195 [I 2022-10-27 04:33:31] ...optimizers/trijoint.py.151: total_norm: 22.932774 [I 2022-10-27 04:33:37] ...trap/engines/engine.py.224: train: epoch 48 | batch 70/2304 [I 2022-10-27 04:33:37] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:35 [I 2022-10-27 04:33:37] ...trap/engines/engine.py.229: process: 0.55181 | load: 0.00267 [I 2022-10-27 04:33:37] ...trap/engines/engine.py.230: loss: 0.17737 [I 2022-10-27 04:33:37] ...optimizers/trijoint.py.151: total_norm: 27.762924 [I 2022-10-27 04:33:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 80/2304 [I 2022-10-27 04:33:42] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:41 [I 2022-10-27 04:33:42] ...trap/engines/engine.py.229: process: 0.55442 | load: 0.00284 [I 2022-10-27 04:33:42] ...trap/engines/engine.py.230: loss: 0.11960 [I 2022-10-27 04:33:42] ...optimizers/trijoint.py.151: total_norm: 14.353405 [I 2022-10-27 04:33:48] ...trap/engines/engine.py.224: train: epoch 48 | batch 90/2304 [I 2022-10-27 04:33:48] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:31 [I 2022-10-27 04:33:48] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00256 [I 2022-10-27 04:33:48] ...trap/engines/engine.py.230: loss: 0.14768 [I 2022-10-27 04:33:48] ...optimizers/trijoint.py.151: total_norm: 13.802814 [I 2022-10-27 04:33:53] ...trap/engines/engine.py.224: train: epoch 48 | batch 100/2304 [I 2022-10-27 04:33:53] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:24 [I 2022-10-27 04:33:53] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00251 [I 2022-10-27 04:33:53] ...trap/engines/engine.py.230: loss: 0.10115 [I 2022-10-27 04:33:53] ...optimizers/trijoint.py.151: total_norm: 9.325130 [I 2022-10-27 04:33:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 110/2304 [I 2022-10-27 04:33:59] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:20:23 [I 2022-10-27 04:33:59] ...trap/engines/engine.py.229: process: 0.56166 | load: 0.00252 [I 2022-10-27 04:33:59] ...trap/engines/engine.py.230: loss: 0.15737 [I 2022-10-27 04:33:59] ...optimizers/trijoint.py.151: total_norm: 16.427584 [I 2022-10-27 04:34:04] ...trap/engines/engine.py.224: train: epoch 48 | batch 120/2304 [I 2022-10-27 04:34:04] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:16 [I 2022-10-27 04:34:04] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00240 [I 2022-10-27 04:34:04] ...trap/engines/engine.py.230: loss: 0.15354 [I 2022-10-27 04:34:04] ...optimizers/trijoint.py.151: total_norm: 13.059623 [I 2022-10-27 04:34:10] ...trap/engines/engine.py.224: train: epoch 48 | batch 130/2304 [I 2022-10-27 04:34:10] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:02 [I 2022-10-27 04:34:10] ...trap/engines/engine.py.229: process: 0.54896 | load: 0.00261 [I 2022-10-27 04:34:10] ...trap/engines/engine.py.230: loss: 0.17839 [I 2022-10-27 04:34:10] ...optimizers/trijoint.py.151: total_norm: 8.323633 [I 2022-10-27 04:34:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 140/2304 [I 2022-10-27 04:34:16] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:05 [I 2022-10-27 04:34:16] ...trap/engines/engine.py.229: process: 0.56539 | load: 0.00264 [I 2022-10-27 04:34:16] ...trap/engines/engine.py.230: loss: 0.15484 [I 2022-10-27 04:34:16] ...optimizers/trijoint.py.151: total_norm: 8.987595 [I 2022-10-27 04:34:21] ...trap/engines/engine.py.224: train: epoch 48 | batch 150/2304 [I 2022-10-27 04:34:21] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:53 [I 2022-10-27 04:34:21] ...trap/engines/engine.py.229: process: 0.54960 | load: 0.00249 [I 2022-10-27 04:34:21] ...trap/engines/engine.py.230: loss: 0.21532 [I 2022-10-27 04:34:21] ...optimizers/trijoint.py.151: total_norm: 29.778204 [I 2022-10-27 04:34:27] ...trap/engines/engine.py.224: train: epoch 48 | batch 160/2304 [I 2022-10-27 04:34:27] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:37 [I 2022-10-27 04:34:27] ...trap/engines/engine.py.229: process: 0.53896 | load: 0.00249 [I 2022-10-27 04:34:27] ...trap/engines/engine.py.230: loss: 0.14411 [I 2022-10-27 04:34:27] ...optimizers/trijoint.py.151: total_norm: 14.788894 [I 2022-10-27 04:34:32] ...trap/engines/engine.py.224: train: epoch 48 | batch 170/2304 [I 2022-10-27 04:34:32] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:44 [I 2022-10-27 04:34:32] ...trap/engines/engine.py.229: process: 0.55481 | load: 0.00254 [I 2022-10-27 04:34:32] ...trap/engines/engine.py.230: loss: 0.14955 [I 2022-10-27 04:34:32] ...optimizers/trijoint.py.151: total_norm: 21.009777 [I 2022-10-27 04:34:38] ...trap/engines/engine.py.224: train: epoch 48 | batch 180/2304 [I 2022-10-27 04:34:38] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:19:38 [I 2022-10-27 04:34:38] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00264 [I 2022-10-27 04:34:38] ...trap/engines/engine.py.230: loss: 0.15229 [I 2022-10-27 04:34:38] ...optimizers/trijoint.py.151: total_norm: 24.564024 [I 2022-10-27 04:34:43] ...trap/engines/engine.py.224: train: epoch 48 | batch 190/2304 [I 2022-10-27 04:34:43] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:35 [I 2022-10-27 04:34:43] ...trap/engines/engine.py.229: process: 0.56690 | load: 0.00255 [I 2022-10-27 04:34:43] ...trap/engines/engine.py.230: loss: 0.10814 [I 2022-10-27 04:34:43] ...optimizers/trijoint.py.151: total_norm: 13.752899 [I 2022-10-27 04:34:49] ...trap/engines/engine.py.224: train: epoch 48 | batch 200/2304 [I 2022-10-27 04:34:49] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:19:42 [I 2022-10-27 04:34:49] ...trap/engines/engine.py.229: process: 0.57830 | load: 0.00265 [I 2022-10-27 04:34:49] ...trap/engines/engine.py.230: loss: 0.12813 [I 2022-10-27 04:34:49] ...optimizers/trijoint.py.151: total_norm: 17.336725 [I 2022-10-27 04:34:55] ...trap/engines/engine.py.224: train: epoch 48 | batch 210/2304 [I 2022-10-27 04:34:55] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:34 [I 2022-10-27 04:34:55] ...trap/engines/engine.py.229: process: 0.56430 | load: 0.00244 [I 2022-10-27 04:34:55] ...trap/engines/engine.py.230: loss: 0.15811 [I 2022-10-27 04:34:55] ...optimizers/trijoint.py.151: total_norm: 18.156456 [I 2022-10-27 04:35:00] ...trap/engines/engine.py.224: train: epoch 48 | batch 220/2304 [I 2022-10-27 04:35:00] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:26 [I 2022-10-27 04:35:00] ...trap/engines/engine.py.229: process: 0.56651 | load: 0.00276 [I 2022-10-27 04:35:00] ...trap/engines/engine.py.230: loss: 0.14739 [I 2022-10-27 04:35:00] ...optimizers/trijoint.py.151: total_norm: 18.054020 [I 2022-10-27 04:35:06] ...trap/engines/engine.py.224: train: epoch 48 | batch 230/2304 [I 2022-10-27 04:35:06] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:17 [I 2022-10-27 04:35:06] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00262 [I 2022-10-27 04:35:06] ...trap/engines/engine.py.230: loss: 0.18033 [I 2022-10-27 04:35:06] ...optimizers/trijoint.py.151: total_norm: 25.976336 [I 2022-10-27 04:35:11] ...trap/engines/engine.py.224: train: epoch 48 | batch 240/2304 [I 2022-10-27 04:35:11] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:12 [I 2022-10-27 04:35:11] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00244 [I 2022-10-27 04:35:11] ...trap/engines/engine.py.230: loss: 0.14021 [I 2022-10-27 04:35:11] ...optimizers/trijoint.py.151: total_norm: 15.380247 [I 2022-10-27 04:35:17] ...trap/engines/engine.py.224: train: epoch 48 | batch 250/2304 [I 2022-10-27 04:35:17] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:18:57 [I 2022-10-27 04:35:17] ...trap/engines/engine.py.229: process: 0.54523 | load: 0.00246 [I 2022-10-27 04:35:17] ...trap/engines/engine.py.230: loss: 0.15305 [I 2022-10-27 04:35:17] ...optimizers/trijoint.py.151: total_norm: 12.294641 [I 2022-10-27 04:35:22] ...trap/engines/engine.py.224: train: epoch 48 | batch 260/2304 [I 2022-10-27 04:35:22] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:57 [I 2022-10-27 04:35:22] ...trap/engines/engine.py.229: process: 0.56470 | load: 0.00245 [I 2022-10-27 04:35:22] ...trap/engines/engine.py.230: loss: 0.12841 [I 2022-10-27 04:35:22] ...optimizers/trijoint.py.151: total_norm: 14.124335 [I 2022-10-27 04:35:28] ...trap/engines/engine.py.224: train: epoch 48 | batch 270/2304 [I 2022-10-27 04:35:28] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:18:43 [I 2022-10-27 04:35:28] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00253 [I 2022-10-27 04:35:28] ...trap/engines/engine.py.230: loss: 0.16870 [I 2022-10-27 04:35:28] ...optimizers/trijoint.py.151: total_norm: 20.384483 [I 2022-10-27 04:35:34] ...trap/engines/engine.py.224: train: epoch 48 | batch 280/2304 [I 2022-10-27 04:35:34] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:45 [I 2022-10-27 04:35:34] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00246 [I 2022-10-27 04:35:34] ...trap/engines/engine.py.230: loss: 0.15225 [I 2022-10-27 04:35:34] ...optimizers/trijoint.py.151: total_norm: 23.340105 [I 2022-10-27 04:35:39] ...trap/engines/engine.py.224: train: epoch 48 | batch 290/2304 [I 2022-10-27 04:35:39] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:19:16 [I 2022-10-27 04:35:39] ...trap/engines/engine.py.229: process: 0.58317 | load: 0.00252 [I 2022-10-27 04:35:39] ...trap/engines/engine.py.230: loss: 0.12601 [I 2022-10-27 04:35:39] ...optimizers/trijoint.py.151: total_norm: 11.367623 [I 2022-10-27 04:35:45] ...trap/engines/engine.py.224: train: epoch 48 | batch 300/2304 [I 2022-10-27 04:35:45] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:18:42 [I 2022-10-27 04:35:45] ...trap/engines/engine.py.229: process: 0.55477 | load: 0.00268 [I 2022-10-27 04:35:45] ...trap/engines/engine.py.230: loss: 0.23945 [I 2022-10-27 04:35:45] ...optimizers/trijoint.py.151: total_norm: 19.838501 [I 2022-10-27 04:35:51] ...trap/engines/engine.py.224: train: epoch 48 | batch 310/2304 [I 2022-10-27 04:35:51] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:19:00 [I 2022-10-27 04:35:51] ...trap/engines/engine.py.229: process: 0.55131 | load: 0.00254 [I 2022-10-27 04:35:51] ...trap/engines/engine.py.230: loss: 0.17948 [I 2022-10-27 04:35:51] ...optimizers/trijoint.py.151: total_norm: 34.171276 [I 2022-10-27 04:35:56] ...trap/engines/engine.py.224: train: epoch 48 | batch 320/2304 [I 2022-10-27 04:35:56] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:17 [I 2022-10-27 04:35:56] ...trap/engines/engine.py.229: process: 0.54514 | load: 0.00246 [I 2022-10-27 04:35:56] ...trap/engines/engine.py.230: loss: 0.18248 [I 2022-10-27 04:35:56] ...optimizers/trijoint.py.151: total_norm: 18.428642 [I 2022-10-27 04:36:02] ...trap/engines/engine.py.224: train: epoch 48 | batch 330/2304 [I 2022-10-27 04:36:02] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:06 [I 2022-10-27 04:36:02] ...trap/engines/engine.py.229: process: 0.54527 | load: 0.00242 [I 2022-10-27 04:36:02] ...trap/engines/engine.py.230: loss: 0.18787 [I 2022-10-27 04:36:02] ...optimizers/trijoint.py.151: total_norm: 10.542010 [I 2022-10-27 04:36:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 340/2304 [I 2022-10-27 04:36:07] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:09 [I 2022-10-27 04:36:07] ...trap/engines/engine.py.229: process: 0.55641 | load: 0.00251 [I 2022-10-27 04:36:07] ...trap/engines/engine.py.230: loss: 0.11393 [I 2022-10-27 04:36:07] ...optimizers/trijoint.py.151: total_norm: 8.976599 [I 2022-10-27 04:36:13] ...trap/engines/engine.py.224: train: epoch 48 | batch 350/2304 [I 2022-10-27 04:36:13] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:17:59 [I 2022-10-27 04:36:13] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00246 [I 2022-10-27 04:36:13] ...trap/engines/engine.py.230: loss: 0.17467 [I 2022-10-27 04:36:13] ...optimizers/trijoint.py.151: total_norm: 19.753157 [I 2022-10-27 04:36:18] ...trap/engines/engine.py.224: train: epoch 48 | batch 360/2304 [I 2022-10-27 04:36:18] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:59 [I 2022-10-27 04:36:18] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00241 [I 2022-10-27 04:36:18] ...trap/engines/engine.py.230: loss: 0.15778 [I 2022-10-27 04:36:18] ...optimizers/trijoint.py.151: total_norm: 17.010458 [I 2022-10-27 04:36:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 370/2304 [I 2022-10-27 04:36:24] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:17:53 [I 2022-10-27 04:36:24] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00242 [I 2022-10-27 04:36:24] ...trap/engines/engine.py.230: loss: 0.21719 [I 2022-10-27 04:36:24] ...optimizers/trijoint.py.151: total_norm: 29.924709 [I 2022-10-27 04:36:30] ...trap/engines/engine.py.224: train: epoch 48 | batch 380/2304 [I 2022-10-27 04:36:30] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:18:01 [I 2022-10-27 04:36:30] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00257 [I 2022-10-27 04:36:30] ...trap/engines/engine.py.230: loss: 0.23265 [I 2022-10-27 04:36:30] ...optimizers/trijoint.py.151: total_norm: 23.603415 [I 2022-10-27 04:36:35] ...trap/engines/engine.py.224: train: epoch 48 | batch 390/2304 [I 2022-10-27 04:36:35] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:43 [I 2022-10-27 04:36:35] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00246 [I 2022-10-27 04:36:35] ...trap/engines/engine.py.230: loss: 0.11283 [I 2022-10-27 04:36:35] ...optimizers/trijoint.py.151: total_norm: 12.022892 [I 2022-10-27 04:36:41] ...trap/engines/engine.py.224: train: epoch 48 | batch 400/2304 [I 2022-10-27 04:36:41] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:42 [I 2022-10-27 04:36:41] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00260 [I 2022-10-27 04:36:41] ...trap/engines/engine.py.230: loss: 0.17125 [I 2022-10-27 04:36:41] ...optimizers/trijoint.py.151: total_norm: 9.983664 [I 2022-10-27 04:36:46] ...trap/engines/engine.py.224: train: epoch 48 | batch 410/2304 [I 2022-10-27 04:36:46] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:33 [I 2022-10-27 04:36:46] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00253 [I 2022-10-27 04:36:46] ...trap/engines/engine.py.230: loss: 0.14033 [I 2022-10-27 04:36:46] ...optimizers/trijoint.py.151: total_norm: 16.685614 [I 2022-10-27 04:36:52] ...trap/engines/engine.py.224: train: epoch 48 | batch 420/2304 [I 2022-10-27 04:36:52] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:17:29 [I 2022-10-27 04:36:52] ...trap/engines/engine.py.229: process: 0.56617 | load: 0.00244 [I 2022-10-27 04:36:52] ...trap/engines/engine.py.230: loss: 0.11228 [I 2022-10-27 04:36:52] ...optimizers/trijoint.py.151: total_norm: 8.473251 [I 2022-10-27 04:36:57] ...trap/engines/engine.py.224: train: epoch 48 | batch 430/2304 [I 2022-10-27 04:36:57] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:13 [I 2022-10-27 04:36:57] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00246 [I 2022-10-27 04:36:57] ...trap/engines/engine.py.230: loss: 0.15670 [I 2022-10-27 04:36:57] ...optimizers/trijoint.py.151: total_norm: 16.578421 [I 2022-10-27 04:37:03] ...trap/engines/engine.py.224: train: epoch 48 | batch 440/2304 [I 2022-10-27 04:37:03] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:17:19 [I 2022-10-27 04:37:03] ...trap/engines/engine.py.229: process: 0.54657 | load: 0.00269 [I 2022-10-27 04:37:03] ...trap/engines/engine.py.230: loss: 0.17540 [I 2022-10-27 04:37:03] ...optimizers/trijoint.py.151: total_norm: 13.648680 [I 2022-10-27 04:37:09] ...trap/engines/engine.py.224: train: epoch 48 | batch 450/2304 [I 2022-10-27 04:37:09] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:20 [I 2022-10-27 04:37:09] ...trap/engines/engine.py.229: process: 0.56778 | load: 0.00243 [I 2022-10-27 04:37:09] ...trap/engines/engine.py.230: loss: 0.15593 [I 2022-10-27 04:37:09] ...optimizers/trijoint.py.151: total_norm: 32.578434 [I 2022-10-27 04:37:14] ...trap/engines/engine.py.224: train: epoch 48 | batch 460/2304 [I 2022-10-27 04:37:14] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:09 [I 2022-10-27 04:37:14] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00268 [I 2022-10-27 04:37:14] ...trap/engines/engine.py.230: loss: 0.16966 [I 2022-10-27 04:37:14] ...optimizers/trijoint.py.151: total_norm: 14.560047 [I 2022-10-27 04:37:20] ...trap/engines/engine.py.224: train: epoch 48 | batch 470/2304 [I 2022-10-27 04:37:20] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:17:05 [I 2022-10-27 04:37:20] ...trap/engines/engine.py.229: process: 0.56390 | load: 0.00245 [I 2022-10-27 04:37:20] ...trap/engines/engine.py.230: loss: 0.19696 [I 2022-10-27 04:37:20] ...optimizers/trijoint.py.151: total_norm: 35.601494 [I 2022-10-27 04:37:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 480/2304 [I 2022-10-27 04:37:25] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:54 [I 2022-10-27 04:37:25] ...trap/engines/engine.py.229: process: 0.54801 | load: 0.00245 [I 2022-10-27 04:37:25] ...trap/engines/engine.py.230: loss: 0.17496 [I 2022-10-27 04:37:25] ...optimizers/trijoint.py.151: total_norm: 18.759876 [I 2022-10-27 04:37:31] ...trap/engines/engine.py.224: train: epoch 48 | batch 490/2304 [I 2022-10-27 04:37:31] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:16:51 [I 2022-10-27 04:37:31] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00244 [I 2022-10-27 04:37:31] ...trap/engines/engine.py.230: loss: 0.13394 [I 2022-10-27 04:37:31] ...optimizers/trijoint.py.151: total_norm: 8.044709 [I 2022-10-27 04:37:36] ...trap/engines/engine.py.224: train: epoch 48 | batch 500/2304 [I 2022-10-27 04:37:36] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:43 [I 2022-10-27 04:37:36] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00249 [I 2022-10-27 04:37:36] ...trap/engines/engine.py.230: loss: 0.28847 [I 2022-10-27 04:37:36] ...optimizers/trijoint.py.151: total_norm: 41.968273 [I 2022-10-27 04:37:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 510/2304 [I 2022-10-27 04:37:42] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:39 [I 2022-10-27 04:37:42] ...trap/engines/engine.py.229: process: 0.55204 | load: 0.00258 [I 2022-10-27 04:37:42] ...trap/engines/engine.py.230: loss: 0.11330 [I 2022-10-27 04:37:42] ...optimizers/trijoint.py.151: total_norm: 9.715753 [I 2022-10-27 04:37:48] ...trap/engines/engine.py.224: train: epoch 48 | batch 520/2304 [I 2022-10-27 04:37:48] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:16:38 [I 2022-10-27 04:37:48] ...trap/engines/engine.py.229: process: 0.56338 | load: 0.00259 [I 2022-10-27 04:37:48] ...trap/engines/engine.py.230: loss: 0.22068 [I 2022-10-27 04:37:48] ...optimizers/trijoint.py.151: total_norm: 20.629671 [I 2022-10-27 04:37:53] ...trap/engines/engine.py.224: train: epoch 48 | batch 530/2304 [I 2022-10-27 04:37:53] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:32 [I 2022-10-27 04:37:53] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00250 [I 2022-10-27 04:37:53] ...trap/engines/engine.py.230: loss: 0.20233 [I 2022-10-27 04:37:53] ...optimizers/trijoint.py.151: total_norm: 23.696600 [I 2022-10-27 04:37:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 540/2304 [I 2022-10-27 04:37:59] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:16:22 [I 2022-10-27 04:37:59] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00245 [I 2022-10-27 04:37:59] ...trap/engines/engine.py.230: loss: 0.13055 [I 2022-10-27 04:37:59] ...optimizers/trijoint.py.151: total_norm: 17.653400 [I 2022-10-27 04:38:04] ...trap/engines/engine.py.224: train: epoch 48 | batch 550/2304 [I 2022-10-27 04:38:04] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:24 [I 2022-10-27 04:38:04] ...trap/engines/engine.py.229: process: 0.55242 | load: 0.00251 [I 2022-10-27 04:38:04] ...trap/engines/engine.py.230: loss: 0.14751 [I 2022-10-27 04:38:04] ...optimizers/trijoint.py.151: total_norm: 15.926420 [I 2022-10-27 04:38:10] ...trap/engines/engine.py.224: train: epoch 48 | batch 560/2304 [I 2022-10-27 04:38:10] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:15 [I 2022-10-27 04:38:10] ...trap/engines/engine.py.229: process: 0.57083 | load: 0.00249 [I 2022-10-27 04:38:10] ...trap/engines/engine.py.230: loss: 0.19677 [I 2022-10-27 04:38:10] ...optimizers/trijoint.py.151: total_norm: 21.338200 [I 2022-10-27 04:38:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 570/2304 [I 2022-10-27 04:38:16] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:16:09 [I 2022-10-27 04:38:16] ...trap/engines/engine.py.229: process: 0.56966 | load: 0.00246 [I 2022-10-27 04:38:16] ...trap/engines/engine.py.230: loss: 0.14122 [I 2022-10-27 04:38:16] ...optimizers/trijoint.py.151: total_norm: 10.319559 [I 2022-10-27 04:38:21] ...trap/engines/engine.py.224: train: epoch 48 | batch 580/2304 [I 2022-10-27 04:38:21] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:00 [I 2022-10-27 04:38:21] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00258 [I 2022-10-27 04:38:21] ...trap/engines/engine.py.230: loss: 0.18488 [I 2022-10-27 04:38:21] ...optimizers/trijoint.py.151: total_norm: 16.677158 [I 2022-10-27 04:38:27] ...trap/engines/engine.py.224: train: epoch 48 | batch 590/2304 [I 2022-10-27 04:38:27] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:15:55 [I 2022-10-27 04:38:27] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00245 [I 2022-10-27 04:38:27] ...trap/engines/engine.py.230: loss: 0.22409 [I 2022-10-27 04:38:27] ...optimizers/trijoint.py.151: total_norm: 17.553858 [I 2022-10-27 04:38:32] ...trap/engines/engine.py.224: train: epoch 48 | batch 600/2304 [I 2022-10-27 04:38:32] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:54 [I 2022-10-27 04:38:32] ...trap/engines/engine.py.229: process: 0.57433 | load: 0.00245 [I 2022-10-27 04:38:32] ...trap/engines/engine.py.230: loss: 0.14304 [I 2022-10-27 04:38:32] ...optimizers/trijoint.py.151: total_norm: 10.824957 [I 2022-10-27 04:38:38] ...trap/engines/engine.py.224: train: epoch 48 | batch 610/2304 [I 2022-10-27 04:38:38] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:15:43 [I 2022-10-27 04:38:38] ...trap/engines/engine.py.229: process: 0.55848 | load: 0.00243 [I 2022-10-27 04:38:38] ...trap/engines/engine.py.230: loss: 0.18259 [I 2022-10-27 04:38:38] ...optimizers/trijoint.py.151: total_norm: 25.507421 [I 2022-10-27 04:38:44] ...trap/engines/engine.py.224: train: epoch 48 | batch 620/2304 [I 2022-10-27 04:38:44] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:38 [I 2022-10-27 04:38:44] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00250 [I 2022-10-27 04:38:44] ...trap/engines/engine.py.230: loss: 0.15287 [I 2022-10-27 04:38:44] ...optimizers/trijoint.py.151: total_norm: 18.642212 [I 2022-10-27 04:38:49] ...trap/engines/engine.py.224: train: epoch 48 | batch 630/2304 [I 2022-10-27 04:38:49] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:30 [I 2022-10-27 04:38:49] ...trap/engines/engine.py.229: process: 0.55321 | load: 0.00246 [I 2022-10-27 04:38:49] ...trap/engines/engine.py.230: loss: 0.16839 [I 2022-10-27 04:38:49] ...optimizers/trijoint.py.151: total_norm: 15.921444 [I 2022-10-27 04:38:55] ...trap/engines/engine.py.224: train: epoch 48 | batch 640/2304 [I 2022-10-27 04:38:55] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:29 [I 2022-10-27 04:38:55] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00253 [I 2022-10-27 04:38:55] ...trap/engines/engine.py.230: loss: 0.15198 [I 2022-10-27 04:38:55] ...optimizers/trijoint.py.151: total_norm: 10.729609 [I 2022-10-27 04:39:19] ...trap/engines/engine.py.224: train: epoch 48 | batch 650/2304 [I 2022-10-27 04:39:19] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:42:59 [I 2022-10-27 04:39:19] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00242 [I 2022-10-27 04:39:19] ...trap/engines/engine.py.230: loss: 0.16279 [I 2022-10-27 04:39:19] ...optimizers/trijoint.py.151: total_norm: 11.626272 [I 2022-10-27 04:39:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 660/2304 [I 2022-10-27 04:39:25] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:18:12 [I 2022-10-27 04:39:25] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00248 [I 2022-10-27 04:39:25] ...trap/engines/engine.py.230: loss: 0.12586 [I 2022-10-27 04:39:25] ...optimizers/trijoint.py.151: total_norm: 10.021543 [I 2022-10-27 04:39:31] ...trap/engines/engine.py.224: train: epoch 48 | batch 670/2304 [I 2022-10-27 04:39:31] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:15:29 [I 2022-10-27 04:39:31] ...trap/engines/engine.py.229: process: 0.56358 | load: 0.00244 [I 2022-10-27 04:39:31] ...trap/engines/engine.py.230: loss: 0.16564 [I 2022-10-27 04:39:31] ...optimizers/trijoint.py.151: total_norm: 14.424556 [I 2022-10-27 04:39:36] ...trap/engines/engine.py.224: train: epoch 48 | batch 680/2304 [I 2022-10-27 04:39:36] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:15:11 [I 2022-10-27 04:39:36] ...trap/engines/engine.py.229: process: 0.56094 | load: 0.00242 [I 2022-10-27 04:39:36] ...trap/engines/engine.py.230: loss: 0.16746 [I 2022-10-27 04:39:36] ...optimizers/trijoint.py.151: total_norm: 25.270582 [I 2022-10-27 04:39:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 690/2304 [I 2022-10-27 04:39:42] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:15:02 [I 2022-10-27 04:39:42] ...trap/engines/engine.py.229: process: 0.56402 | load: 0.00239 [I 2022-10-27 04:39:42] ...trap/engines/engine.py.230: loss: 0.16223 [I 2022-10-27 04:39:42] ...optimizers/trijoint.py.151: total_norm: 20.357742 [I 2022-10-27 04:39:47] ...trap/engines/engine.py.224: train: epoch 48 | batch 700/2304 [I 2022-10-27 04:39:47] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:14:56 [I 2022-10-27 04:39:47] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00241 [I 2022-10-27 04:39:47] ...trap/engines/engine.py.230: loss: 0.18124 [I 2022-10-27 04:39:47] ...optimizers/trijoint.py.151: total_norm: 13.949478 [I 2022-10-27 04:39:53] ...trap/engines/engine.py.224: train: epoch 48 | batch 710/2304 [I 2022-10-27 04:39:53] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:38 [I 2022-10-27 04:39:53] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00245 [I 2022-10-27 04:39:53] ...trap/engines/engine.py.230: loss: 0.17553 [I 2022-10-27 04:39:53] ...optimizers/trijoint.py.151: total_norm: 11.411236 [I 2022-10-27 04:39:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 720/2304 [I 2022-10-27 04:39:59] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:45 [I 2022-10-27 04:39:59] ...trap/engines/engine.py.229: process: 0.55842 | load: 0.00242 [I 2022-10-27 04:39:59] ...trap/engines/engine.py.230: loss: 0.16970 [I 2022-10-27 04:39:59] ...optimizers/trijoint.py.151: total_norm: 12.848315 [I 2022-10-27 04:40:04] ...trap/engines/engine.py.224: train: epoch 48 | batch 730/2304 [I 2022-10-27 04:40:04] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:36 [I 2022-10-27 04:40:04] ...trap/engines/engine.py.229: process: 0.55936 | load: 0.00240 [I 2022-10-27 04:40:04] ...trap/engines/engine.py.230: loss: 0.13797 [I 2022-10-27 04:40:04] ...optimizers/trijoint.py.151: total_norm: 19.157320 [I 2022-10-27 04:40:10] ...trap/engines/engine.py.224: train: epoch 48 | batch 740/2304 [I 2022-10-27 04:40:10] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:41 [I 2022-10-27 04:40:10] ...trap/engines/engine.py.229: process: 0.55985 | load: 0.00241 [I 2022-10-27 04:40:10] ...trap/engines/engine.py.230: loss: 0.21939 [I 2022-10-27 04:40:10] ...optimizers/trijoint.py.151: total_norm: 9.593802 [I 2022-10-27 04:40:15] ...trap/engines/engine.py.224: train: epoch 48 | batch 750/2304 [I 2022-10-27 04:40:15] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:26 [I 2022-10-27 04:40:15] ...trap/engines/engine.py.229: process: 0.56260 | load: 0.00247 [I 2022-10-27 04:40:15] ...trap/engines/engine.py.230: loss: 0.17273 [I 2022-10-27 04:40:15] ...optimizers/trijoint.py.151: total_norm: 14.665307 [I 2022-10-27 04:40:21] ...trap/engines/engine.py.224: train: epoch 48 | batch 760/2304 [I 2022-10-27 04:40:21] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:14:20 [I 2022-10-27 04:40:21] ...trap/engines/engine.py.229: process: 0.55481 | load: 0.00243 [I 2022-10-27 04:40:21] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-27 04:40:21] ...optimizers/trijoint.py.151: total_norm: 23.262510 [I 2022-10-27 04:40:27] ...trap/engines/engine.py.224: train: epoch 48 | batch 770/2304 [I 2022-10-27 04:40:27] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:14:14 [I 2022-10-27 04:40:27] ...trap/engines/engine.py.229: process: 0.55602 | load: 0.00245 [I 2022-10-27 04:40:27] ...trap/engines/engine.py.230: loss: 0.17914 [I 2022-10-27 04:40:27] ...optimizers/trijoint.py.151: total_norm: 36.660267 [I 2022-10-27 04:40:32] ...trap/engines/engine.py.224: train: epoch 48 | batch 780/2304 [I 2022-10-27 04:40:32] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:14:11 [I 2022-10-27 04:40:32] ...trap/engines/engine.py.229: process: 0.56188 | load: 0.00242 [I 2022-10-27 04:40:32] ...trap/engines/engine.py.230: loss: 0.12543 [I 2022-10-27 04:40:32] ...optimizers/trijoint.py.151: total_norm: 13.571784 [I 2022-10-27 04:40:38] ...trap/engines/engine.py.224: train: epoch 48 | batch 790/2304 [I 2022-10-27 04:40:38] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:14:01 [I 2022-10-27 04:40:38] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00241 [I 2022-10-27 04:40:38] ...trap/engines/engine.py.230: loss: 0.11406 [I 2022-10-27 04:40:38] ...optimizers/trijoint.py.151: total_norm: 9.028930 [I 2022-10-27 04:40:43] ...trap/engines/engine.py.224: train: epoch 48 | batch 800/2304 [I 2022-10-27 04:40:43] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:14:06 [I 2022-10-27 04:40:43] ...trap/engines/engine.py.229: process: 0.55233 | load: 0.00241 [I 2022-10-27 04:40:43] ...trap/engines/engine.py.230: loss: 0.14720 [I 2022-10-27 04:40:43] ...optimizers/trijoint.py.151: total_norm: 19.945549 [I 2022-10-27 04:40:49] ...trap/engines/engine.py.224: train: epoch 48 | batch 810/2304 [I 2022-10-27 04:40:49] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:13:48 [I 2022-10-27 04:40:49] ...trap/engines/engine.py.229: process: 0.54825 | load: 0.00241 [I 2022-10-27 04:40:49] ...trap/engines/engine.py.230: loss: 0.12082 [I 2022-10-27 04:40:49] ...optimizers/trijoint.py.151: total_norm: 7.212705 [I 2022-10-27 04:40:55] ...trap/engines/engine.py.224: train: epoch 48 | batch 820/2304 [I 2022-10-27 04:40:55] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:39 [I 2022-10-27 04:40:55] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00245 [I 2022-10-27 04:40:55] ...trap/engines/engine.py.230: loss: 0.19944 [I 2022-10-27 04:40:55] ...optimizers/trijoint.py.151: total_norm: 19.193388 [I 2022-10-27 04:41:00] ...trap/engines/engine.py.224: train: epoch 48 | batch 830/2304 [I 2022-10-27 04:41:00] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:13:41 [I 2022-10-27 04:41:00] ...trap/engines/engine.py.229: process: 0.55023 | load: 0.00243 [I 2022-10-27 04:41:00] ...trap/engines/engine.py.230: loss: 0.22814 [I 2022-10-27 04:41:00] ...optimizers/trijoint.py.151: total_norm: 20.704964 [I 2022-10-27 04:41:06] ...trap/engines/engine.py.224: train: epoch 48 | batch 840/2304 [I 2022-10-27 04:41:06] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:33 [I 2022-10-27 04:41:06] ...trap/engines/engine.py.229: process: 0.55033 | load: 0.00244 [I 2022-10-27 04:41:06] ...trap/engines/engine.py.230: loss: 0.19900 [I 2022-10-27 04:41:06] ...optimizers/trijoint.py.151: total_norm: 19.557022 [I 2022-10-27 04:41:11] ...trap/engines/engine.py.224: train: epoch 48 | batch 850/2304 [I 2022-10-27 04:41:11] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:23 [I 2022-10-27 04:41:11] ...trap/engines/engine.py.229: process: 0.55478 | load: 0.00246 [I 2022-10-27 04:41:11] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-27 04:41:11] ...optimizers/trijoint.py.151: total_norm: 12.437110 [I 2022-10-27 04:41:17] ...trap/engines/engine.py.224: train: epoch 48 | batch 860/2304 [I 2022-10-27 04:41:17] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:21 [I 2022-10-27 04:41:17] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00242 [I 2022-10-27 04:41:17] ...trap/engines/engine.py.230: loss: 0.13103 [I 2022-10-27 04:41:17] ...optimizers/trijoint.py.151: total_norm: 22.259907 [I 2022-10-27 04:41:22] ...trap/engines/engine.py.224: train: epoch 48 | batch 870/2304 [I 2022-10-27 04:41:22] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:07 [I 2022-10-27 04:41:22] ...trap/engines/engine.py.229: process: 0.53555 | load: 0.00253 [I 2022-10-27 04:41:22] ...trap/engines/engine.py.230: loss: 0.11324 [I 2022-10-27 04:41:22] ...optimizers/trijoint.py.151: total_norm: 7.811186 [I 2022-10-27 04:41:28] ...trap/engines/engine.py.224: train: epoch 48 | batch 880/2304 [I 2022-10-27 04:41:28] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:13:04 [I 2022-10-27 04:41:28] ...trap/engines/engine.py.229: process: 0.55202 | load: 0.00245 [I 2022-10-27 04:41:28] ...trap/engines/engine.py.230: loss: 0.26340 [I 2022-10-27 04:41:28] ...optimizers/trijoint.py.151: total_norm: 31.119568 [I 2022-10-27 04:41:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 890/2304 [I 2022-10-27 04:41:33] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:13:08 [I 2022-10-27 04:41:33] ...trap/engines/engine.py.229: process: 0.55809 | load: 0.00244 [I 2022-10-27 04:41:33] ...trap/engines/engine.py.230: loss: 0.15230 [I 2022-10-27 04:41:33] ...optimizers/trijoint.py.151: total_norm: 11.376965 [I 2022-10-27 04:41:39] ...trap/engines/engine.py.224: train: epoch 48 | batch 900/2304 [I 2022-10-27 04:41:39] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:13:00 [I 2022-10-27 04:41:39] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00245 [I 2022-10-27 04:41:39] ...trap/engines/engine.py.230: loss: 0.17231 [I 2022-10-27 04:41:39] ...optimizers/trijoint.py.151: total_norm: 13.225059 [I 2022-10-27 04:41:45] ...trap/engines/engine.py.224: train: epoch 48 | batch 910/2304 [I 2022-10-27 04:41:45] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:51 [I 2022-10-27 04:41:45] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00241 [I 2022-10-27 04:41:45] ...trap/engines/engine.py.230: loss: 0.10310 [I 2022-10-27 04:41:45] ...optimizers/trijoint.py.151: total_norm: 14.845319 [I 2022-10-27 04:41:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 920/2304 [I 2022-10-27 04:41:50] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:50 [I 2022-10-27 04:41:50] ...trap/engines/engine.py.229: process: 0.56672 | load: 0.00244 [I 2022-10-27 04:41:50] ...trap/engines/engine.py.230: loss: 0.11822 [I 2022-10-27 04:41:50] ...optimizers/trijoint.py.151: total_norm: 18.571856 [I 2022-10-27 04:41:56] ...trap/engines/engine.py.224: train: epoch 48 | batch 930/2304 [I 2022-10-27 04:41:56] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:40 [I 2022-10-27 04:41:56] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00246 [I 2022-10-27 04:41:56] ...trap/engines/engine.py.230: loss: 0.16796 [I 2022-10-27 04:41:56] ...optimizers/trijoint.py.151: total_norm: 15.327951 [I 2022-10-27 04:42:01] ...trap/engines/engine.py.224: train: epoch 48 | batch 940/2304 [I 2022-10-27 04:42:01] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:41 [I 2022-10-27 04:42:01] ...trap/engines/engine.py.229: process: 0.54519 | load: 0.00241 [I 2022-10-27 04:42:01] ...trap/engines/engine.py.230: loss: 0.18886 [I 2022-10-27 04:42:01] ...optimizers/trijoint.py.151: total_norm: 24.013943 [I 2022-10-27 04:42:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 950/2304 [I 2022-10-27 04:42:07] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:40 [I 2022-10-27 04:42:07] ...trap/engines/engine.py.229: process: 0.57442 | load: 0.00244 [I 2022-10-27 04:42:07] ...trap/engines/engine.py.230: loss: 0.26235 [I 2022-10-27 04:42:07] ...optimizers/trijoint.py.151: total_norm: 17.974407 [I 2022-10-27 04:42:12] ...trap/engines/engine.py.224: train: epoch 48 | batch 960/2304 [I 2022-10-27 04:42:12] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:28 [I 2022-10-27 04:42:12] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00261 [I 2022-10-27 04:42:12] ...trap/engines/engine.py.230: loss: 0.17704 [I 2022-10-27 04:42:12] ...optimizers/trijoint.py.151: total_norm: 14.192229 [I 2022-10-27 04:42:18] ...trap/engines/engine.py.224: train: epoch 48 | batch 970/2304 [I 2022-10-27 04:42:18] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:16 [I 2022-10-27 04:42:18] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00246 [I 2022-10-27 04:42:18] ...trap/engines/engine.py.230: loss: 0.14173 [I 2022-10-27 04:42:18] ...optimizers/trijoint.py.151: total_norm: 11.982974 [I 2022-10-27 04:42:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 980/2304 [I 2022-10-27 04:42:24] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:23 [I 2022-10-27 04:42:24] ...trap/engines/engine.py.229: process: 0.58541 | load: 0.00250 [I 2022-10-27 04:42:24] ...trap/engines/engine.py.230: loss: 0.14999 [I 2022-10-27 04:42:24] ...optimizers/trijoint.py.151: total_norm: 14.063141 [I 2022-10-27 04:42:29] ...trap/engines/engine.py.224: train: epoch 48 | batch 990/2304 [I 2022-10-27 04:42:29] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:12:15 [I 2022-10-27 04:42:29] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00244 [I 2022-10-27 04:42:29] ...trap/engines/engine.py.230: loss: 0.16410 [I 2022-10-27 04:42:29] ...optimizers/trijoint.py.151: total_norm: 14.202646 [I 2022-10-27 04:42:35] ...trap/engines/engine.py.224: train: epoch 48 | batch 1000/2304 [I 2022-10-27 04:42:35] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:12:06 [I 2022-10-27 04:42:35] ...trap/engines/engine.py.229: process: 0.55082 | load: 0.00253 [I 2022-10-27 04:42:35] ...trap/engines/engine.py.230: loss: 0.15788 [I 2022-10-27 04:42:35] ...optimizers/trijoint.py.151: total_norm: 14.387177 [I 2022-10-27 04:42:40] ...trap/engines/engine.py.224: train: epoch 48 | batch 1010/2304 [I 2022-10-27 04:42:40] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:12:03 [I 2022-10-27 04:42:40] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00247 [I 2022-10-27 04:42:40] ...trap/engines/engine.py.230: loss: 0.21686 [I 2022-10-27 04:42:40] ...optimizers/trijoint.py.151: total_norm: 19.970385 [I 2022-10-27 04:42:46] ...trap/engines/engine.py.224: train: epoch 48 | batch 1020/2304 [I 2022-10-27 04:42:46] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:53 [I 2022-10-27 04:42:46] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00244 [I 2022-10-27 04:42:46] ...trap/engines/engine.py.230: loss: 0.15654 [I 2022-10-27 04:42:46] ...optimizers/trijoint.py.151: total_norm: 15.863606 [I 2022-10-27 04:42:52] ...trap/engines/engine.py.224: train: epoch 48 | batch 1030/2304 [I 2022-10-27 04:42:52] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:46 [I 2022-10-27 04:42:52] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00242 [I 2022-10-27 04:42:52] ...trap/engines/engine.py.230: loss: 0.16143 [I 2022-10-27 04:42:52] ...optimizers/trijoint.py.151: total_norm: 16.449072 [I 2022-10-27 04:42:57] ...trap/engines/engine.py.224: train: epoch 48 | batch 1040/2304 [I 2022-10-27 04:42:57] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:43 [I 2022-10-27 04:42:57] ...trap/engines/engine.py.229: process: 0.54354 | load: 0.00242 [I 2022-10-27 04:42:57] ...trap/engines/engine.py.230: loss: 0.14331 [I 2022-10-27 04:42:57] ...optimizers/trijoint.py.151: total_norm: 18.771526 [I 2022-10-27 04:43:03] ...trap/engines/engine.py.224: train: epoch 48 | batch 1050/2304 [I 2022-10-27 04:43:03] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:36 [I 2022-10-27 04:43:03] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00244 [I 2022-10-27 04:43:03] ...trap/engines/engine.py.230: loss: 0.18613 [I 2022-10-27 04:43:03] ...optimizers/trijoint.py.151: total_norm: 13.528822 [I 2022-10-27 04:43:08] ...trap/engines/engine.py.224: train: epoch 48 | batch 1060/2304 [I 2022-10-27 04:43:08] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:29 [I 2022-10-27 04:43:08] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00242 [I 2022-10-27 04:43:08] ...trap/engines/engine.py.230: loss: 0.11879 [I 2022-10-27 04:43:08] ...optimizers/trijoint.py.151: total_norm: 27.075819 [I 2022-10-27 04:43:14] ...trap/engines/engine.py.224: train: epoch 48 | batch 1070/2304 [I 2022-10-27 04:43:14] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:33 [I 2022-10-27 04:43:14] ...trap/engines/engine.py.229: process: 0.56527 | load: 0.00241 [I 2022-10-27 04:43:14] ...trap/engines/engine.py.230: loss: 0.14333 [I 2022-10-27 04:43:14] ...optimizers/trijoint.py.151: total_norm: 16.459297 [I 2022-10-27 04:43:19] ...trap/engines/engine.py.224: train: epoch 48 | batch 1080/2304 [I 2022-10-27 04:43:19] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:22 [I 2022-10-27 04:43:19] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00244 [I 2022-10-27 04:43:19] ...trap/engines/engine.py.230: loss: 0.13673 [I 2022-10-27 04:43:19] ...optimizers/trijoint.py.151: total_norm: 17.441807 [I 2022-10-27 04:43:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 1090/2304 [I 2022-10-27 04:43:25] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:15 [I 2022-10-27 04:43:25] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00239 [I 2022-10-27 04:43:25] ...trap/engines/engine.py.230: loss: 0.14434 [I 2022-10-27 04:43:25] ...optimizers/trijoint.py.151: total_norm: 30.839268 [I 2022-10-27 04:43:31] ...trap/engines/engine.py.224: train: epoch 48 | batch 1100/2304 [I 2022-10-27 04:43:31] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:11:07 [I 2022-10-27 04:43:31] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00242 [I 2022-10-27 04:43:31] ...trap/engines/engine.py.230: loss: 0.23826 [I 2022-10-27 04:43:31] ...optimizers/trijoint.py.151: total_norm: 35.415794 [I 2022-10-27 04:43:36] ...trap/engines/engine.py.224: train: epoch 48 | batch 1110/2304 [I 2022-10-27 04:43:36] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:11:03 [I 2022-10-27 04:43:36] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00240 [I 2022-10-27 04:43:36] ...trap/engines/engine.py.230: loss: 0.15850 [I 2022-10-27 04:43:36] ...optimizers/trijoint.py.151: total_norm: 26.550913 [I 2022-10-27 04:43:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 1120/2304 [I 2022-10-27 04:43:42] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:58 [I 2022-10-27 04:43:42] ...trap/engines/engine.py.229: process: 0.55194 | load: 0.00241 [I 2022-10-27 04:43:42] ...trap/engines/engine.py.230: loss: 0.14948 [I 2022-10-27 04:43:42] ...optimizers/trijoint.py.151: total_norm: 19.929686 [I 2022-10-27 04:43:47] ...trap/engines/engine.py.224: train: epoch 48 | batch 1130/2304 [I 2022-10-27 04:43:47] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:47 [I 2022-10-27 04:43:47] ...trap/engines/engine.py.229: process: 0.55346 | load: 0.00242 [I 2022-10-27 04:43:47] ...trap/engines/engine.py.230: loss: 0.18538 [I 2022-10-27 04:43:47] ...optimizers/trijoint.py.151: total_norm: 94.226448 [I 2022-10-27 04:43:53] ...trap/engines/engine.py.224: train: epoch 48 | batch 1140/2304 [I 2022-10-27 04:43:53] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:45 [I 2022-10-27 04:43:53] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00240 [I 2022-10-27 04:43:53] ...trap/engines/engine.py.230: loss: 0.18947 [I 2022-10-27 04:43:53] ...optimizers/trijoint.py.151: total_norm: 21.346012 [I 2022-10-27 04:43:58] ...trap/engines/engine.py.224: train: epoch 48 | batch 1150/2304 [I 2022-10-27 04:43:58] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:40 [I 2022-10-27 04:43:58] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00243 [I 2022-10-27 04:43:58] ...trap/engines/engine.py.230: loss: 0.11411 [I 2022-10-27 04:43:58] ...optimizers/trijoint.py.151: total_norm: 11.507085 [I 2022-10-27 04:44:04] ...trap/engines/engine.py.224: train: epoch 48 | batch 1160/2304 [I 2022-10-27 04:44:04] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:34 [I 2022-10-27 04:44:04] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00242 [I 2022-10-27 04:44:04] ...trap/engines/engine.py.230: loss: 0.18367 [I 2022-10-27 04:44:04] ...optimizers/trijoint.py.151: total_norm: 34.439491 [I 2022-10-27 04:44:10] ...trap/engines/engine.py.224: train: epoch 48 | batch 1170/2304 [I 2022-10-27 04:44:10] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:30 [I 2022-10-27 04:44:10] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00240 [I 2022-10-27 04:44:10] ...trap/engines/engine.py.230: loss: 0.17603 [I 2022-10-27 04:44:10] ...optimizers/trijoint.py.151: total_norm: 18.253326 [I 2022-10-27 04:44:15] ...trap/engines/engine.py.224: train: epoch 48 | batch 1180/2304 [I 2022-10-27 04:44:15] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:22 [I 2022-10-27 04:44:15] ...trap/engines/engine.py.229: process: 0.54930 | load: 0.00243 [I 2022-10-27 04:44:15] ...trap/engines/engine.py.230: loss: 0.16500 [I 2022-10-27 04:44:15] ...optimizers/trijoint.py.151: total_norm: 9.691786 [I 2022-10-27 04:44:21] ...trap/engines/engine.py.224: train: epoch 48 | batch 1190/2304 [I 2022-10-27 04:44:21] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:19 [I 2022-10-27 04:44:21] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00241 [I 2022-10-27 04:44:21] ...trap/engines/engine.py.230: loss: 0.25341 [I 2022-10-27 04:44:21] ...optimizers/trijoint.py.151: total_norm: 33.780621 [I 2022-10-27 04:44:26] ...trap/engines/engine.py.224: train: epoch 48 | batch 1200/2304 [I 2022-10-27 04:44:26] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:10:19 [I 2022-10-27 04:44:26] ...trap/engines/engine.py.229: process: 0.57122 | load: 0.00243 [I 2022-10-27 04:44:26] ...trap/engines/engine.py.230: loss: 0.16188 [I 2022-10-27 04:44:26] ...optimizers/trijoint.py.151: total_norm: 15.254595 [I 2022-10-27 04:44:32] ...trap/engines/engine.py.224: train: epoch 48 | batch 1210/2304 [I 2022-10-27 04:44:32] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:10:08 [I 2022-10-27 04:44:32] ...trap/engines/engine.py.229: process: 0.55347 | load: 0.00239 [I 2022-10-27 04:44:32] ...trap/engines/engine.py.230: loss: 0.13597 [I 2022-10-27 04:44:32] ...optimizers/trijoint.py.151: total_norm: 21.842209 [I 2022-10-27 04:44:37] ...trap/engines/engine.py.224: train: epoch 48 | batch 1220/2304 [I 2022-10-27 04:44:37] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:10:04 [I 2022-10-27 04:44:37] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00255 [I 2022-10-27 04:44:37] ...trap/engines/engine.py.230: loss: 0.14208 [I 2022-10-27 04:44:37] ...optimizers/trijoint.py.151: total_norm: 17.153912 [I 2022-10-27 04:44:43] ...trap/engines/engine.py.224: train: epoch 48 | batch 1230/2304 [I 2022-10-27 04:44:43] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:56 [I 2022-10-27 04:44:43] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00248 [I 2022-10-27 04:44:43] ...trap/engines/engine.py.230: loss: 0.10982 [I 2022-10-27 04:44:43] ...optimizers/trijoint.py.151: total_norm: 8.720960 [I 2022-10-27 04:44:49] ...trap/engines/engine.py.224: train: epoch 48 | batch 1240/2304 [I 2022-10-27 04:44:49] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:52 [I 2022-10-27 04:44:49] ...trap/engines/engine.py.229: process: 0.56294 | load: 0.00244 [I 2022-10-27 04:44:49] ...trap/engines/engine.py.230: loss: 0.22732 [I 2022-10-27 04:44:49] ...optimizers/trijoint.py.151: total_norm: 31.074982 [I 2022-10-27 04:44:54] ...trap/engines/engine.py.224: train: epoch 48 | batch 1250/2304 [I 2022-10-27 04:44:54] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:48 [I 2022-10-27 04:44:54] ...trap/engines/engine.py.229: process: 0.55218 | load: 0.00249 [I 2022-10-27 04:44:54] ...trap/engines/engine.py.230: loss: 0.17581 [I 2022-10-27 04:44:54] ...optimizers/trijoint.py.151: total_norm: 14.495613 [I 2022-10-27 04:45:00] ...trap/engines/engine.py.224: train: epoch 48 | batch 1260/2304 [I 2022-10-27 04:45:00] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:42 [I 2022-10-27 04:45:00] ...trap/engines/engine.py.229: process: 0.56508 | load: 0.00247 [I 2022-10-27 04:45:00] ...trap/engines/engine.py.230: loss: 0.16226 [I 2022-10-27 04:45:00] ...optimizers/trijoint.py.151: total_norm: 18.394365 [I 2022-10-27 04:45:05] ...trap/engines/engine.py.224: train: epoch 48 | batch 1270/2304 [I 2022-10-27 04:45:05] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:36 [I 2022-10-27 04:45:05] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00245 [I 2022-10-27 04:45:05] ...trap/engines/engine.py.230: loss: 0.15099 [I 2022-10-27 04:45:05] ...optimizers/trijoint.py.151: total_norm: 13.707173 [I 2022-10-27 04:45:11] ...trap/engines/engine.py.224: train: epoch 48 | batch 1280/2304 [I 2022-10-27 04:45:11] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:32 [I 2022-10-27 04:45:11] ...trap/engines/engine.py.229: process: 0.55242 | load: 0.00242 [I 2022-10-27 04:45:11] ...trap/engines/engine.py.230: loss: 0.18358 [I 2022-10-27 04:45:11] ...optimizers/trijoint.py.151: total_norm: 17.715263 [I 2022-10-27 04:45:17] ...trap/engines/engine.py.224: train: epoch 48 | batch 1290/2304 [I 2022-10-27 04:45:17] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:28 [I 2022-10-27 04:45:17] ...trap/engines/engine.py.229: process: 0.55793 | load: 0.00241 [I 2022-10-27 04:45:17] ...trap/engines/engine.py.230: loss: 0.09871 [I 2022-10-27 04:45:17] ...optimizers/trijoint.py.151: total_norm: 10.727249 [I 2022-10-27 04:45:22] ...trap/engines/engine.py.224: train: epoch 48 | batch 1300/2304 [I 2022-10-27 04:45:22] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:20 [I 2022-10-27 04:45:22] ...trap/engines/engine.py.229: process: 0.56730 | load: 0.00242 [I 2022-10-27 04:45:22] ...trap/engines/engine.py.230: loss: 0.15978 [I 2022-10-27 04:45:22] ...optimizers/trijoint.py.151: total_norm: 23.426460 [I 2022-10-27 04:45:28] ...trap/engines/engine.py.224: train: epoch 48 | batch 1310/2304 [I 2022-10-27 04:45:28] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:09:12 [I 2022-10-27 04:45:28] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00240 [I 2022-10-27 04:45:28] ...trap/engines/engine.py.230: loss: 0.16775 [I 2022-10-27 04:45:28] ...optimizers/trijoint.py.151: total_norm: 20.658817 [I 2022-10-27 04:45:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 1320/2304 [I 2022-10-27 04:45:33] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:09:09 [I 2022-10-27 04:45:33] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00242 [I 2022-10-27 04:45:33] ...trap/engines/engine.py.230: loss: 0.15901 [I 2022-10-27 04:45:33] ...optimizers/trijoint.py.151: total_norm: 31.379847 [I 2022-10-27 04:45:39] ...trap/engines/engine.py.224: train: epoch 48 | batch 1330/2304 [I 2022-10-27 04:45:39] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:09:08 [I 2022-10-27 04:45:39] ...trap/engines/engine.py.229: process: 0.58320 | load: 0.00241 [I 2022-10-27 04:45:39] ...trap/engines/engine.py.230: loss: 0.10794 [I 2022-10-27 04:45:39] ...optimizers/trijoint.py.151: total_norm: 11.569569 [I 2022-10-27 04:45:45] ...trap/engines/engine.py.224: train: epoch 48 | batch 1340/2304 [I 2022-10-27 04:45:45] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:09:00 [I 2022-10-27 04:45:45] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00241 [I 2022-10-27 04:45:45] ...trap/engines/engine.py.230: loss: 0.11607 [I 2022-10-27 04:45:45] ...optimizers/trijoint.py.151: total_norm: 8.874756 [I 2022-10-27 04:45:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 1350/2304 [I 2022-10-27 04:45:50] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:09:03 [I 2022-10-27 04:45:50] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00243 [I 2022-10-27 04:45:50] ...trap/engines/engine.py.230: loss: 0.19293 [I 2022-10-27 04:45:50] ...optimizers/trijoint.py.151: total_norm: 10.156228 [I 2022-10-27 04:45:56] ...trap/engines/engine.py.224: train: epoch 48 | batch 1360/2304 [I 2022-10-27 04:45:56] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:47 [I 2022-10-27 04:45:56] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00241 [I 2022-10-27 04:45:56] ...trap/engines/engine.py.230: loss: 0.15236 [I 2022-10-27 04:45:56] ...optimizers/trijoint.py.151: total_norm: 12.662716 [I 2022-10-27 04:46:01] ...trap/engines/engine.py.224: train: epoch 48 | batch 1370/2304 [I 2022-10-27 04:46:01] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:38 [I 2022-10-27 04:46:01] ...trap/engines/engine.py.229: process: 0.55830 | load: 0.00241 [I 2022-10-27 04:46:01] ...trap/engines/engine.py.230: loss: 0.13108 [I 2022-10-27 04:46:01] ...optimizers/trijoint.py.151: total_norm: 15.886689 [I 2022-10-27 04:46:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 1380/2304 [I 2022-10-27 04:46:07] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:35 [I 2022-10-27 04:46:07] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00244 [I 2022-10-27 04:46:07] ...trap/engines/engine.py.230: loss: 0.18633 [I 2022-10-27 04:46:07] ...optimizers/trijoint.py.151: total_norm: 16.695112 [I 2022-10-27 04:46:13] ...trap/engines/engine.py.224: train: epoch 48 | batch 1390/2304 [I 2022-10-27 04:46:13] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:35 [I 2022-10-27 04:46:13] ...trap/engines/engine.py.229: process: 0.57582 | load: 0.00243 [I 2022-10-27 04:46:13] ...trap/engines/engine.py.230: loss: 0.16213 [I 2022-10-27 04:46:13] ...optimizers/trijoint.py.151: total_norm: 14.588374 [I 2022-10-27 04:46:18] ...trap/engines/engine.py.224: train: epoch 48 | batch 1400/2304 [I 2022-10-27 04:46:18] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:24 [I 2022-10-27 04:46:18] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00245 [I 2022-10-27 04:46:18] ...trap/engines/engine.py.230: loss: 0.18852 [I 2022-10-27 04:46:18] ...optimizers/trijoint.py.151: total_norm: 34.741032 [I 2022-10-27 04:46:24] ...trap/engines/engine.py.224: train: epoch 48 | batch 1410/2304 [I 2022-10-27 04:46:24] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:18 [I 2022-10-27 04:46:24] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00252 [I 2022-10-27 04:46:24] ...trap/engines/engine.py.230: loss: 0.16050 [I 2022-10-27 04:46:24] ...optimizers/trijoint.py.151: total_norm: 16.219133 [I 2022-10-27 04:46:30] ...trap/engines/engine.py.224: train: epoch 48 | batch 1420/2304 [I 2022-10-27 04:46:30] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:08:31 [I 2022-10-27 04:46:30] ...trap/engines/engine.py.229: process: 0.66890 | load: 0.00248 [I 2022-10-27 04:46:30] ...trap/engines/engine.py.230: loss: 0.16115 [I 2022-10-27 04:46:30] ...optimizers/trijoint.py.151: total_norm: 22.895613 [I 2022-10-27 04:46:35] ...trap/engines/engine.py.224: train: epoch 48 | batch 1430/2304 [I 2022-10-27 04:46:35] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:08:10 [I 2022-10-27 04:46:35] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00248 [I 2022-10-27 04:46:35] ...trap/engines/engine.py.230: loss: 0.14561 [I 2022-10-27 04:46:35] ...optimizers/trijoint.py.151: total_norm: 8.434873 [I 2022-10-27 04:46:41] ...trap/engines/engine.py.224: train: epoch 48 | batch 1440/2304 [I 2022-10-27 04:46:41] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:08:03 [I 2022-10-27 04:46:41] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00242 [I 2022-10-27 04:46:41] ...trap/engines/engine.py.230: loss: 0.11179 [I 2022-10-27 04:46:41] ...optimizers/trijoint.py.151: total_norm: 7.779246 [I 2022-10-27 04:46:46] ...trap/engines/engine.py.224: train: epoch 48 | batch 1450/2304 [I 2022-10-27 04:46:46] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:56 [I 2022-10-27 04:46:46] ...trap/engines/engine.py.229: process: 0.56156 | load: 0.00263 [I 2022-10-27 04:46:46] ...trap/engines/engine.py.230: loss: 0.24220 [I 2022-10-27 04:46:46] ...optimizers/trijoint.py.151: total_norm: 31.978670 [I 2022-10-27 04:46:52] ...trap/engines/engine.py.224: train: epoch 48 | batch 1460/2304 [I 2022-10-27 04:46:52] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:48 [I 2022-10-27 04:46:52] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00245 [I 2022-10-27 04:46:52] ...trap/engines/engine.py.230: loss: 0.17870 [I 2022-10-27 04:46:52] ...optimizers/trijoint.py.151: total_norm: 13.265339 [I 2022-10-27 04:46:57] ...trap/engines/engine.py.224: train: epoch 48 | batch 1470/2304 [I 2022-10-27 04:46:57] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:43 [I 2022-10-27 04:46:57] ...trap/engines/engine.py.229: process: 0.56233 | load: 0.00244 [I 2022-10-27 04:46:57] ...trap/engines/engine.py.230: loss: 0.13703 [I 2022-10-27 04:46:57] ...optimizers/trijoint.py.151: total_norm: 8.658340 [I 2022-10-27 04:47:03] ...trap/engines/engine.py.224: train: epoch 48 | batch 1480/2304 [I 2022-10-27 04:47:03] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:38 [I 2022-10-27 04:47:03] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00244 [I 2022-10-27 04:47:03] ...trap/engines/engine.py.230: loss: 0.11208 [I 2022-10-27 04:47:03] ...optimizers/trijoint.py.151: total_norm: 15.563871 [I 2022-10-27 04:47:09] ...trap/engines/engine.py.224: train: epoch 48 | batch 1490/2304 [I 2022-10-27 04:47:09] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:42 [I 2022-10-27 04:47:09] ...trap/engines/engine.py.229: process: 0.62369 | load: 0.00245 [I 2022-10-27 04:47:09] ...trap/engines/engine.py.230: loss: 0.14064 [I 2022-10-27 04:47:09] ...optimizers/trijoint.py.151: total_norm: 41.459358 [I 2022-10-27 04:47:14] ...trap/engines/engine.py.224: train: epoch 48 | batch 1500/2304 [I 2022-10-27 04:47:14] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:34 [I 2022-10-27 04:47:14] ...trap/engines/engine.py.229: process: 0.56508 | load: 0.00245 [I 2022-10-27 04:47:14] ...trap/engines/engine.py.230: loss: 0.18976 [I 2022-10-27 04:47:14] ...optimizers/trijoint.py.151: total_norm: 15.837806 [I 2022-10-27 04:47:20] ...trap/engines/engine.py.224: train: epoch 48 | batch 1510/2304 [I 2022-10-27 04:47:20] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:23 [I 2022-10-27 04:47:20] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00241 [I 2022-10-27 04:47:20] ...trap/engines/engine.py.230: loss: 0.23633 [I 2022-10-27 04:47:20] ...optimizers/trijoint.py.151: total_norm: 12.786430 [I 2022-10-27 04:47:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 1520/2304 [I 2022-10-27 04:47:25] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:14 [I 2022-10-27 04:47:25] ...trap/engines/engine.py.229: process: 0.55250 | load: 0.00243 [I 2022-10-27 04:47:25] ...trap/engines/engine.py.230: loss: 0.14125 [I 2022-10-27 04:47:25] ...optimizers/trijoint.py.151: total_norm: 13.690486 [I 2022-10-27 04:47:31] ...trap/engines/engine.py.224: train: epoch 48 | batch 1530/2304 [I 2022-10-27 04:47:31] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:07:08 [I 2022-10-27 04:47:31] ...trap/engines/engine.py.229: process: 0.55069 | load: 0.00244 [I 2022-10-27 04:47:31] ...trap/engines/engine.py.230: loss: 0.16118 [I 2022-10-27 04:47:31] ...optimizers/trijoint.py.151: total_norm: 15.986922 [I 2022-10-27 04:47:37] ...trap/engines/engine.py.224: train: epoch 48 | batch 1540/2304 [I 2022-10-27 04:47:37] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:07:09 [I 2022-10-27 04:47:37] ...trap/engines/engine.py.229: process: 0.56303 | load: 0.00244 [I 2022-10-27 04:47:37] ...trap/engines/engine.py.230: loss: 0.19491 [I 2022-10-27 04:47:37] ...optimizers/trijoint.py.151: total_norm: 22.336870 [I 2022-10-27 04:47:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 1550/2304 [I 2022-10-27 04:47:42] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:07:01 [I 2022-10-27 04:47:42] ...trap/engines/engine.py.229: process: 0.56223 | load: 0.00243 [I 2022-10-27 04:47:42] ...trap/engines/engine.py.230: loss: 0.12508 [I 2022-10-27 04:47:42] ...optimizers/trijoint.py.151: total_norm: 9.314479 [I 2022-10-27 04:47:48] ...trap/engines/engine.py.224: train: epoch 48 | batch 1560/2304 [I 2022-10-27 04:47:48] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:56 [I 2022-10-27 04:47:48] ...trap/engines/engine.py.229: process: 0.56625 | load: 0.00242 [I 2022-10-27 04:47:48] ...trap/engines/engine.py.230: loss: 0.14133 [I 2022-10-27 04:47:48] ...optimizers/trijoint.py.151: total_norm: 9.201881 [I 2022-10-27 04:47:53] ...trap/engines/engine.py.224: train: epoch 48 | batch 1570/2304 [I 2022-10-27 04:47:53] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:51 [I 2022-10-27 04:47:53] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00272 [I 2022-10-27 04:47:53] ...trap/engines/engine.py.230: loss: 0.11352 [I 2022-10-27 04:47:53] ...optimizers/trijoint.py.151: total_norm: 22.026138 [I 2022-10-27 04:47:59] ...trap/engines/engine.py.224: train: epoch 48 | batch 1580/2304 [I 2022-10-27 04:47:59] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:40 [I 2022-10-27 04:47:59] ...trap/engines/engine.py.229: process: 0.55184 | load: 0.00239 [I 2022-10-27 04:47:59] ...trap/engines/engine.py.230: loss: 0.15221 [I 2022-10-27 04:47:59] ...optimizers/trijoint.py.151: total_norm: 10.704069 [I 2022-10-27 04:48:05] ...trap/engines/engine.py.224: train: epoch 48 | batch 1590/2304 [I 2022-10-27 04:48:05] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:35 [I 2022-10-27 04:48:05] ...trap/engines/engine.py.229: process: 0.54566 | load: 0.00241 [I 2022-10-27 04:48:05] ...trap/engines/engine.py.230: loss: 0.20614 [I 2022-10-27 04:48:05] ...optimizers/trijoint.py.151: total_norm: 11.269505 [I 2022-10-27 04:48:10] ...trap/engines/engine.py.224: train: epoch 48 | batch 1600/2304 [I 2022-10-27 04:48:10] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:32 [I 2022-10-27 04:48:10] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00261 [I 2022-10-27 04:48:10] ...trap/engines/engine.py.230: loss: 0.12201 [I 2022-10-27 04:48:10] ...optimizers/trijoint.py.151: total_norm: 18.782158 [I 2022-10-27 04:48:16] ...trap/engines/engine.py.224: train: epoch 48 | batch 1610/2304 [I 2022-10-27 04:48:16] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:29 [I 2022-10-27 04:48:16] ...trap/engines/engine.py.229: process: 0.56480 | load: 0.00245 [I 2022-10-27 04:48:16] ...trap/engines/engine.py.230: loss: 0.19530 [I 2022-10-27 04:48:16] ...optimizers/trijoint.py.151: total_norm: 25.765902 [I 2022-10-27 04:48:21] ...trap/engines/engine.py.224: train: epoch 48 | batch 1620/2304 [I 2022-10-27 04:48:21] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:22 [I 2022-10-27 04:48:21] ...trap/engines/engine.py.229: process: 0.56443 | load: 0.00243 [I 2022-10-27 04:48:21] ...trap/engines/engine.py.230: loss: 0.17277 [I 2022-10-27 04:48:21] ...optimizers/trijoint.py.151: total_norm: 25.808010 [I 2022-10-27 04:48:27] ...trap/engines/engine.py.224: train: epoch 48 | batch 1630/2304 [I 2022-10-27 04:48:27] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:06:16 [I 2022-10-27 04:48:27] ...trap/engines/engine.py.229: process: 0.56045 | load: 0.00270 [I 2022-10-27 04:48:27] ...trap/engines/engine.py.230: loss: 0.18881 [I 2022-10-27 04:48:27] ...optimizers/trijoint.py.151: total_norm: 22.528978 [I 2022-10-27 04:48:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 1640/2304 [I 2022-10-27 04:48:33] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:06:09 [I 2022-10-27 04:48:33] ...trap/engines/engine.py.229: process: 0.55434 | load: 0.00249 [I 2022-10-27 04:48:33] ...trap/engines/engine.py.230: loss: 0.16282 [I 2022-10-27 04:48:33] ...optimizers/trijoint.py.151: total_norm: 11.215896 [I 2022-10-27 04:48:38] ...trap/engines/engine.py.224: train: epoch 48 | batch 1650/2304 [I 2022-10-27 04:48:38] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:06:03 [I 2022-10-27 04:48:38] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00247 [I 2022-10-27 04:48:38] ...trap/engines/engine.py.230: loss: 0.18491 [I 2022-10-27 04:48:38] ...optimizers/trijoint.py.151: total_norm: 52.350906 [I 2022-10-27 04:48:44] ...trap/engines/engine.py.224: train: epoch 48 | batch 1660/2304 [I 2022-10-27 04:48:44] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:06:00 [I 2022-10-27 04:48:44] ...trap/engines/engine.py.229: process: 0.56234 | load: 0.00244 [I 2022-10-27 04:48:44] ...trap/engines/engine.py.230: loss: 0.18376 [I 2022-10-27 04:48:44] ...optimizers/trijoint.py.151: total_norm: 19.910891 [I 2022-10-27 04:48:49] ...trap/engines/engine.py.224: train: epoch 48 | batch 1670/2304 [I 2022-10-27 04:48:49] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:53 [I 2022-10-27 04:48:49] ...trap/engines/engine.py.229: process: 0.54493 | load: 0.00243 [I 2022-10-27 04:48:49] ...trap/engines/engine.py.230: loss: 0.17914 [I 2022-10-27 04:48:49] ...optimizers/trijoint.py.151: total_norm: 19.123409 [I 2022-10-27 04:48:55] ...trap/engines/engine.py.224: train: epoch 48 | batch 1680/2304 [I 2022-10-27 04:48:55] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:50 [I 2022-10-27 04:48:55] ...trap/engines/engine.py.229: process: 0.56493 | load: 0.00264 [I 2022-10-27 04:48:55] ...trap/engines/engine.py.230: loss: 0.16672 [I 2022-10-27 04:48:55] ...optimizers/trijoint.py.151: total_norm: 24.298922 [I 2022-10-27 04:49:01] ...trap/engines/engine.py.224: train: epoch 48 | batch 1690/2304 [I 2022-10-27 04:49:01] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:42 [I 2022-10-27 04:49:01] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00246 [I 2022-10-27 04:49:01] ...trap/engines/engine.py.230: loss: 0.11287 [I 2022-10-27 04:49:01] ...optimizers/trijoint.py.151: total_norm: 8.764709 [I 2022-10-27 04:49:06] ...trap/engines/engine.py.224: train: epoch 48 | batch 1700/2304 [I 2022-10-27 04:49:06] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:35 [I 2022-10-27 04:49:06] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00246 [I 2022-10-27 04:49:06] ...trap/engines/engine.py.230: loss: 0.21017 [I 2022-10-27 04:49:06] ...optimizers/trijoint.py.151: total_norm: 16.317553 [I 2022-10-27 04:49:12] ...trap/engines/engine.py.224: train: epoch 48 | batch 1710/2304 [I 2022-10-27 04:49:12] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:36 [I 2022-10-27 04:49:12] ...trap/engines/engine.py.229: process: 0.55980 | load: 0.00242 [I 2022-10-27 04:49:12] ...trap/engines/engine.py.230: loss: 0.13056 [I 2022-10-27 04:49:12] ...optimizers/trijoint.py.151: total_norm: 19.286667 [I 2022-10-27 04:49:17] ...trap/engines/engine.py.224: train: epoch 48 | batch 1720/2304 [I 2022-10-27 04:49:17] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:26 [I 2022-10-27 04:49:17] ...trap/engines/engine.py.229: process: 0.56630 | load: 0.00240 [I 2022-10-27 04:49:17] ...trap/engines/engine.py.230: loss: 0.16025 [I 2022-10-27 04:49:17] ...optimizers/trijoint.py.151: total_norm: 14.425334 [I 2022-10-27 04:49:23] ...trap/engines/engine.py.224: train: epoch 48 | batch 1730/2304 [I 2022-10-27 04:49:23] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:21 [I 2022-10-27 04:49:23] ...trap/engines/engine.py.229: process: 0.56439 | load: 0.00245 [I 2022-10-27 04:49:23] ...trap/engines/engine.py.230: loss: 0.15012 [I 2022-10-27 04:49:23] ...optimizers/trijoint.py.151: total_norm: 22.991987 [I 2022-10-27 04:49:29] ...trap/engines/engine.py.224: train: epoch 48 | batch 1740/2304 [I 2022-10-27 04:49:29] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:05:14 [I 2022-10-27 04:49:29] ...trap/engines/engine.py.229: process: 0.56558 | load: 0.00243 [I 2022-10-27 04:49:29] ...trap/engines/engine.py.230: loss: 0.14792 [I 2022-10-27 04:49:29] ...optimizers/trijoint.py.151: total_norm: 11.973142 [I 2022-10-27 04:49:34] ...trap/engines/engine.py.224: train: epoch 48 | batch 1750/2304 [I 2022-10-27 04:49:34] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:05:09 [I 2022-10-27 04:49:34] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00239 [I 2022-10-27 04:49:34] ...trap/engines/engine.py.230: loss: 0.13042 [I 2022-10-27 04:49:34] ...optimizers/trijoint.py.151: total_norm: 18.041452 [I 2022-10-27 04:49:40] ...trap/engines/engine.py.224: train: epoch 48 | batch 1760/2304 [I 2022-10-27 04:49:40] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:05:03 [I 2022-10-27 04:49:40] ...trap/engines/engine.py.229: process: 0.55688 | load: 0.00243 [I 2022-10-27 04:49:40] ...trap/engines/engine.py.230: loss: 0.15450 [I 2022-10-27 04:49:40] ...optimizers/trijoint.py.151: total_norm: 36.562328 [I 2022-10-27 04:49:45] ...trap/engines/engine.py.224: train: epoch 48 | batch 1770/2304 [I 2022-10-27 04:49:45] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:57 [I 2022-10-27 04:49:45] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00242 [I 2022-10-27 04:49:45] ...trap/engines/engine.py.230: loss: 0.15967 [I 2022-10-27 04:49:45] ...optimizers/trijoint.py.151: total_norm: 13.210365 [I 2022-10-27 04:49:51] ...trap/engines/engine.py.224: train: epoch 48 | batch 1780/2304 [I 2022-10-27 04:49:51] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:51 [I 2022-10-27 04:49:51] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00243 [I 2022-10-27 04:49:51] ...trap/engines/engine.py.230: loss: 0.16048 [I 2022-10-27 04:49:51] ...optimizers/trijoint.py.151: total_norm: 15.692945 [I 2022-10-27 04:49:57] ...trap/engines/engine.py.224: train: epoch 48 | batch 1790/2304 [I 2022-10-27 04:49:57] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:47 [I 2022-10-27 04:49:57] ...trap/engines/engine.py.229: process: 0.57299 | load: 0.00241 [I 2022-10-27 04:49:57] ...trap/engines/engine.py.230: loss: 0.20749 [I 2022-10-27 04:49:57] ...optimizers/trijoint.py.151: total_norm: 17.074915 [I 2022-10-27 04:50:02] ...trap/engines/engine.py.224: train: epoch 48 | batch 1800/2304 [I 2022-10-27 04:50:02] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:40 [I 2022-10-27 04:50:02] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00243 [I 2022-10-27 04:50:02] ...trap/engines/engine.py.230: loss: 0.12178 [I 2022-10-27 04:50:02] ...optimizers/trijoint.py.151: total_norm: 12.949091 [I 2022-10-27 04:50:08] ...trap/engines/engine.py.224: train: epoch 48 | batch 1810/2304 [I 2022-10-27 04:50:08] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:34 [I 2022-10-27 04:50:08] ...trap/engines/engine.py.229: process: 0.55510 | load: 0.00244 [I 2022-10-27 04:50:08] ...trap/engines/engine.py.230: loss: 0.11682 [I 2022-10-27 04:50:08] ...optimizers/trijoint.py.151: total_norm: 14.821064 [I 2022-10-27 04:50:13] ...trap/engines/engine.py.224: train: epoch 48 | batch 1820/2304 [I 2022-10-27 04:50:13] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:30 [I 2022-10-27 04:50:13] ...trap/engines/engine.py.229: process: 0.54980 | load: 0.00243 [I 2022-10-27 04:50:13] ...trap/engines/engine.py.230: loss: 0.15009 [I 2022-10-27 04:50:13] ...optimizers/trijoint.py.151: total_norm: 19.043163 [I 2022-10-27 04:50:19] ...trap/engines/engine.py.224: train: epoch 48 | batch 1830/2304 [I 2022-10-27 04:50:19] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:24 [I 2022-10-27 04:50:19] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00245 [I 2022-10-27 04:50:19] ...trap/engines/engine.py.230: loss: 0.12540 [I 2022-10-27 04:50:19] ...optimizers/trijoint.py.151: total_norm: 16.353548 [I 2022-10-27 04:50:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 1840/2304 [I 2022-10-27 04:50:25] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:21 [I 2022-10-27 04:50:25] ...trap/engines/engine.py.229: process: 0.56411 | load: 0.00243 [I 2022-10-27 04:50:25] ...trap/engines/engine.py.230: loss: 0.18368 [I 2022-10-27 04:50:25] ...optimizers/trijoint.py.151: total_norm: 23.111868 [I 2022-10-27 04:50:30] ...trap/engines/engine.py.224: train: epoch 48 | batch 1850/2304 [I 2022-10-27 04:50:30] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:14 [I 2022-10-27 04:50:30] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00243 [I 2022-10-27 04:50:30] ...trap/engines/engine.py.230: loss: 0.15603 [I 2022-10-27 04:50:30] ...optimizers/trijoint.py.151: total_norm: 25.104780 [I 2022-10-27 04:50:36] ...trap/engines/engine.py.224: train: epoch 48 | batch 1860/2304 [I 2022-10-27 04:50:36] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:04:08 [I 2022-10-27 04:50:36] ...trap/engines/engine.py.229: process: 0.56413 | load: 0.00250 [I 2022-10-27 04:50:36] ...trap/engines/engine.py.230: loss: 0.24065 [I 2022-10-27 04:50:36] ...optimizers/trijoint.py.151: total_norm: 17.972185 [I 2022-10-27 04:50:41] ...trap/engines/engine.py.224: train: epoch 48 | batch 1870/2304 [I 2022-10-27 04:50:41] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:04:01 [I 2022-10-27 04:50:41] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00251 [I 2022-10-27 04:50:41] ...trap/engines/engine.py.230: loss: 0.17081 [I 2022-10-27 04:50:41] ...optimizers/trijoint.py.151: total_norm: 12.746644 [I 2022-10-27 04:50:47] ...trap/engines/engine.py.224: train: epoch 48 | batch 1880/2304 [I 2022-10-27 04:50:47] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:59 [I 2022-10-27 04:50:47] ...trap/engines/engine.py.229: process: 0.56293 | load: 0.00245 [I 2022-10-27 04:50:47] ...trap/engines/engine.py.230: loss: 0.15099 [I 2022-10-27 04:50:47] ...optimizers/trijoint.py.151: total_norm: 10.957297 [I 2022-10-27 04:50:53] ...trap/engines/engine.py.224: train: epoch 48 | batch 1890/2304 [I 2022-10-27 04:50:53] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:52 [I 2022-10-27 04:50:53] ...trap/engines/engine.py.229: process: 0.55688 | load: 0.00240 [I 2022-10-27 04:50:53] ...trap/engines/engine.py.230: loss: 0.18625 [I 2022-10-27 04:50:53] ...optimizers/trijoint.py.151: total_norm: 11.311819 [I 2022-10-27 04:50:58] ...trap/engines/engine.py.224: train: epoch 48 | batch 1900/2304 [I 2022-10-27 04:50:58] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:46 [I 2022-10-27 04:50:58] ...trap/engines/engine.py.229: process: 0.56522 | load: 0.00245 [I 2022-10-27 04:50:58] ...trap/engines/engine.py.230: loss: 0.16363 [I 2022-10-27 04:50:58] ...optimizers/trijoint.py.151: total_norm: 11.165432 [I 2022-10-27 04:51:04] ...trap/engines/engine.py.224: train: epoch 48 | batch 1910/2304 [I 2022-10-27 04:51:04] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:38 [I 2022-10-27 04:51:04] ...trap/engines/engine.py.229: process: 0.54987 | load: 0.00247 [I 2022-10-27 04:51:04] ...trap/engines/engine.py.230: loss: 0.21766 [I 2022-10-27 04:51:04] ...optimizers/trijoint.py.151: total_norm: 29.127460 [I 2022-10-27 04:51:09] ...trap/engines/engine.py.224: train: epoch 48 | batch 1920/2304 [I 2022-10-27 04:51:09] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:34 [I 2022-10-27 04:51:09] ...trap/engines/engine.py.229: process: 0.55104 | load: 0.00240 [I 2022-10-27 04:51:09] ...trap/engines/engine.py.230: loss: 0.17691 [I 2022-10-27 04:51:09] ...optimizers/trijoint.py.151: total_norm: 17.401306 [I 2022-10-27 04:51:15] ...trap/engines/engine.py.224: train: epoch 48 | batch 1930/2304 [I 2022-10-27 04:51:15] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:28 [I 2022-10-27 04:51:15] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00244 [I 2022-10-27 04:51:15] ...trap/engines/engine.py.230: loss: 0.11177 [I 2022-10-27 04:51:15] ...optimizers/trijoint.py.151: total_norm: 16.471140 [I 2022-10-27 04:51:21] ...trap/engines/engine.py.224: train: epoch 48 | batch 1940/2304 [I 2022-10-27 04:51:21] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:24 [I 2022-10-27 04:51:21] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00245 [I 2022-10-27 04:51:21] ...trap/engines/engine.py.230: loss: 0.10327 [I 2022-10-27 04:51:21] ...optimizers/trijoint.py.151: total_norm: 9.762386 [I 2022-10-27 04:51:26] ...trap/engines/engine.py.224: train: epoch 48 | batch 1950/2304 [I 2022-10-27 04:51:26] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:18 [I 2022-10-27 04:51:26] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00243 [I 2022-10-27 04:51:26] ...trap/engines/engine.py.230: loss: 0.16526 [I 2022-10-27 04:51:26] ...optimizers/trijoint.py.151: total_norm: 30.447834 [I 2022-10-27 04:51:32] ...trap/engines/engine.py.224: train: epoch 48 | batch 1960/2304 [I 2022-10-27 04:51:32] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:03:12 [I 2022-10-27 04:51:32] ...trap/engines/engine.py.229: process: 0.56622 | load: 0.00243 [I 2022-10-27 04:51:32] ...trap/engines/engine.py.230: loss: 0.23622 [I 2022-10-27 04:51:32] ...optimizers/trijoint.py.151: total_norm: 44.287308 [I 2022-10-27 04:51:37] ...trap/engines/engine.py.224: train: epoch 48 | batch 1970/2304 [I 2022-10-27 04:51:37] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:03:06 [I 2022-10-27 04:51:37] ...trap/engines/engine.py.229: process: 0.55419 | load: 0.00242 [I 2022-10-27 04:51:37] ...trap/engines/engine.py.230: loss: 0.13403 [I 2022-10-27 04:51:37] ...optimizers/trijoint.py.151: total_norm: 28.935890 [I 2022-10-27 04:51:43] ...trap/engines/engine.py.224: train: epoch 48 | batch 1980/2304 [I 2022-10-27 04:51:43] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:03:00 [I 2022-10-27 04:51:43] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00263 [I 2022-10-27 04:51:43] ...trap/engines/engine.py.230: loss: 0.17644 [I 2022-10-27 04:51:43] ...optimizers/trijoint.py.151: total_norm: 20.799431 [I 2022-10-27 04:51:49] ...trap/engines/engine.py.224: train: epoch 48 | batch 1990/2304 [I 2022-10-27 04:51:49] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:54 [I 2022-10-27 04:51:49] ...trap/engines/engine.py.229: process: 0.54512 | load: 0.00242 [I 2022-10-27 04:51:49] ...trap/engines/engine.py.230: loss: 0.15377 [I 2022-10-27 04:51:49] ...optimizers/trijoint.py.151: total_norm: 26.829231 [I 2022-10-27 04:51:54] ...trap/engines/engine.py.224: train: epoch 48 | batch 2000/2304 [I 2022-10-27 04:51:54] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:50 [I 2022-10-27 04:51:54] ...trap/engines/engine.py.229: process: 0.55509 | load: 0.00242 [I 2022-10-27 04:51:54] ...trap/engines/engine.py.230: loss: 0.17518 [I 2022-10-27 04:51:54] ...optimizers/trijoint.py.151: total_norm: 25.191317 [I 2022-10-27 04:52:00] ...trap/engines/engine.py.224: train: epoch 48 | batch 2010/2304 [I 2022-10-27 04:52:00] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:43 [I 2022-10-27 04:52:00] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00245 [I 2022-10-27 04:52:00] ...trap/engines/engine.py.230: loss: 0.20476 [I 2022-10-27 04:52:00] ...optimizers/trijoint.py.151: total_norm: 20.906353 [I 2022-10-27 04:52:05] ...trap/engines/engine.py.224: train: epoch 48 | batch 2020/2304 [I 2022-10-27 04:52:05] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:38 [I 2022-10-27 04:52:05] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00243 [I 2022-10-27 04:52:05] ...trap/engines/engine.py.230: loss: 0.22506 [I 2022-10-27 04:52:05] ...optimizers/trijoint.py.151: total_norm: 22.559015 [I 2022-10-27 04:52:11] ...trap/engines/engine.py.224: train: epoch 48 | batch 2030/2304 [I 2022-10-27 04:52:11] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:33 [I 2022-10-27 04:52:11] ...trap/engines/engine.py.229: process: 0.57078 | load: 0.00244 [I 2022-10-27 04:52:11] ...trap/engines/engine.py.230: loss: 0.16394 [I 2022-10-27 04:52:11] ...optimizers/trijoint.py.151: total_norm: 18.797956 [I 2022-10-27 04:52:17] ...trap/engines/engine.py.224: train: epoch 48 | batch 2040/2304 [I 2022-10-27 04:52:17] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:27 [I 2022-10-27 04:52:17] ...trap/engines/engine.py.229: process: 0.55985 | load: 0.00244 [I 2022-10-27 04:52:17] ...trap/engines/engine.py.230: loss: 0.15881 [I 2022-10-27 04:52:17] ...optimizers/trijoint.py.151: total_norm: 12.175453 [I 2022-10-27 04:52:22] ...trap/engines/engine.py.224: train: epoch 48 | batch 2050/2304 [I 2022-10-27 04:52:22] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:22 [I 2022-10-27 04:52:22] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00243 [I 2022-10-27 04:52:22] ...trap/engines/engine.py.230: loss: 0.13332 [I 2022-10-27 04:52:22] ...optimizers/trijoint.py.151: total_norm: 14.066860 [I 2022-10-27 04:52:28] ...trap/engines/engine.py.224: train: epoch 48 | batch 2060/2304 [I 2022-10-27 04:52:28] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:02:15 [I 2022-10-27 04:52:28] ...trap/engines/engine.py.229: process: 0.53904 | load: 0.00242 [I 2022-10-27 04:52:28] ...trap/engines/engine.py.230: loss: 0.14330 [I 2022-10-27 04:52:28] ...optimizers/trijoint.py.151: total_norm: 12.601314 [I 2022-10-27 04:52:33] ...trap/engines/engine.py.224: train: epoch 48 | batch 2070/2304 [I 2022-10-27 04:52:33] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:09 [I 2022-10-27 04:52:33] ...trap/engines/engine.py.229: process: 0.54841 | load: 0.00242 [I 2022-10-27 04:52:33] ...trap/engines/engine.py.230: loss: 0.19164 [I 2022-10-27 04:52:33] ...optimizers/trijoint.py.151: total_norm: 21.842293 [I 2022-10-27 04:52:39] ...trap/engines/engine.py.224: train: epoch 48 | batch 2080/2304 [I 2022-10-27 04:52:39] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:02:04 [I 2022-10-27 04:52:39] ...trap/engines/engine.py.229: process: 0.56254 | load: 0.00244 [I 2022-10-27 04:52:39] ...trap/engines/engine.py.230: loss: 0.13454 [I 2022-10-27 04:52:39] ...optimizers/trijoint.py.151: total_norm: 21.460810 [I 2022-10-27 04:52:44] ...trap/engines/engine.py.224: train: epoch 48 | batch 2090/2304 [I 2022-10-27 04:52:44] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:59 [I 2022-10-27 04:52:44] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00239 [I 2022-10-27 04:52:44] ...trap/engines/engine.py.230: loss: 0.18905 [I 2022-10-27 04:52:44] ...optimizers/trijoint.py.151: total_norm: 15.600285 [I 2022-10-27 04:52:50] ...trap/engines/engine.py.224: train: epoch 48 | batch 2100/2304 [I 2022-10-27 04:52:50] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:53 [I 2022-10-27 04:52:50] ...trap/engines/engine.py.229: process: 0.55289 | load: 0.00242 [I 2022-10-27 04:52:50] ...trap/engines/engine.py.230: loss: 0.22819 [I 2022-10-27 04:52:50] ...optimizers/trijoint.py.151: total_norm: 27.254549 [I 2022-10-27 04:52:56] ...trap/engines/engine.py.224: train: epoch 48 | batch 2110/2304 [I 2022-10-27 04:52:56] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:47 [I 2022-10-27 04:52:56] ...trap/engines/engine.py.229: process: 0.55366 | load: 0.00239 [I 2022-10-27 04:52:56] ...trap/engines/engine.py.230: loss: 0.10920 [I 2022-10-27 04:52:56] ...optimizers/trijoint.py.151: total_norm: 6.472292 [I 2022-10-27 04:53:01] ...trap/engines/engine.py.224: train: epoch 48 | batch 2120/2304 [I 2022-10-27 04:53:01] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:43 [I 2022-10-27 04:53:01] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00254 [I 2022-10-27 04:53:01] ...trap/engines/engine.py.230: loss: 0.13619 [I 2022-10-27 04:53:01] ...optimizers/trijoint.py.151: total_norm: 14.997573 [I 2022-10-27 04:53:07] ...trap/engines/engine.py.224: train: epoch 48 | batch 2130/2304 [I 2022-10-27 04:53:07] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:36 [I 2022-10-27 04:53:07] ...trap/engines/engine.py.229: process: 0.55543 | load: 0.00248 [I 2022-10-27 04:53:07] ...trap/engines/engine.py.230: loss: 0.14949 [I 2022-10-27 04:53:07] ...optimizers/trijoint.py.151: total_norm: 12.729290 [I 2022-10-27 04:53:12] ...trap/engines/engine.py.224: train: epoch 48 | batch 2140/2304 [I 2022-10-27 04:53:12] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:31 [I 2022-10-27 04:53:12] ...trap/engines/engine.py.229: process: 0.54593 | load: 0.00246 [I 2022-10-27 04:53:12] ...trap/engines/engine.py.230: loss: 0.16266 [I 2022-10-27 04:53:12] ...optimizers/trijoint.py.151: total_norm: 11.647478 [I 2022-10-27 04:53:18] ...trap/engines/engine.py.224: train: epoch 48 | batch 2150/2304 [I 2022-10-27 04:53:18] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:25 [I 2022-10-27 04:53:18] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00245 [I 2022-10-27 04:53:18] ...trap/engines/engine.py.230: loss: 0.18026 [I 2022-10-27 04:53:18] ...optimizers/trijoint.py.151: total_norm: 18.777214 [I 2022-10-27 04:53:23] ...trap/engines/engine.py.224: train: epoch 48 | batch 2160/2304 [I 2022-10-27 04:53:23] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:19 [I 2022-10-27 04:53:23] ...trap/engines/engine.py.229: process: 0.54916 | load: 0.00242 [I 2022-10-27 04:53:23] ...trap/engines/engine.py.230: loss: 0.16142 [I 2022-10-27 04:53:23] ...optimizers/trijoint.py.151: total_norm: 10.294850 [I 2022-10-27 04:53:29] ...trap/engines/engine.py.224: train: epoch 48 | batch 2170/2304 [I 2022-10-27 04:53:29] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:14 [I 2022-10-27 04:53:29] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00243 [I 2022-10-27 04:53:29] ...trap/engines/engine.py.230: loss: 0.17100 [I 2022-10-27 04:53:29] ...optimizers/trijoint.py.151: total_norm: 13.065360 [I 2022-10-27 04:53:35] ...trap/engines/engine.py.224: train: epoch 48 | batch 2180/2304 [I 2022-10-27 04:53:35] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:01:09 [I 2022-10-27 04:53:35] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00243 [I 2022-10-27 04:53:35] ...trap/engines/engine.py.230: loss: 0.12642 [I 2022-10-27 04:53:35] ...optimizers/trijoint.py.151: total_norm: 9.578702 [I 2022-10-27 04:53:40] ...trap/engines/engine.py.224: train: epoch 48 | batch 2190/2304 [I 2022-10-27 04:53:40] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:01:03 [I 2022-10-27 04:53:40] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00244 [I 2022-10-27 04:53:40] ...trap/engines/engine.py.230: loss: 0.18661 [I 2022-10-27 04:53:40] ...optimizers/trijoint.py.151: total_norm: 10.555009 [I 2022-10-27 04:53:46] ...trap/engines/engine.py.224: train: epoch 48 | batch 2200/2304 [I 2022-10-27 04:53:46] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:58 [I 2022-10-27 04:53:46] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00243 [I 2022-10-27 04:53:46] ...trap/engines/engine.py.230: loss: 0.18336 [I 2022-10-27 04:53:46] ...optimizers/trijoint.py.151: total_norm: 12.063947 [I 2022-10-27 04:53:51] ...trap/engines/engine.py.224: train: epoch 48 | batch 2210/2304 [I 2022-10-27 04:53:51] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:52 [I 2022-10-27 04:53:51] ...trap/engines/engine.py.229: process: 0.56225 | load: 0.00247 [I 2022-10-27 04:53:51] ...trap/engines/engine.py.230: loss: 0.18629 [I 2022-10-27 04:53:51] ...optimizers/trijoint.py.151: total_norm: 26.055763 [I 2022-10-27 04:53:57] ...trap/engines/engine.py.224: train: epoch 48 | batch 2220/2304 [I 2022-10-27 04:53:57] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:46 [I 2022-10-27 04:53:57] ...trap/engines/engine.py.229: process: 0.54450 | load: 0.00243 [I 2022-10-27 04:53:57] ...trap/engines/engine.py.230: loss: 0.16762 [I 2022-10-27 04:53:57] ...optimizers/trijoint.py.151: total_norm: 20.039499 [I 2022-10-27 04:54:02] ...trap/engines/engine.py.224: train: epoch 48 | batch 2230/2304 [I 2022-10-27 04:54:03] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:41 [I 2022-10-27 04:54:03] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00242 [I 2022-10-27 04:54:03] ...trap/engines/engine.py.230: loss: 0.12953 [I 2022-10-27 04:54:03] ...optimizers/trijoint.py.151: total_norm: 11.279520 [I 2022-10-27 04:54:08] ...trap/engines/engine.py.224: train: epoch 48 | batch 2240/2304 [I 2022-10-27 04:54:08] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:35 [I 2022-10-27 04:54:08] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00247 [I 2022-10-27 04:54:08] ...trap/engines/engine.py.230: loss: 0.14156 [I 2022-10-27 04:54:08] ...optimizers/trijoint.py.151: total_norm: 16.447525 [I 2022-10-27 04:54:14] ...trap/engines/engine.py.224: train: epoch 48 | batch 2250/2304 [I 2022-10-27 04:54:14] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:29 [I 2022-10-27 04:54:14] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00245 [I 2022-10-27 04:54:14] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-27 04:54:14] ...optimizers/trijoint.py.151: total_norm: 11.448596 [I 2022-10-27 04:54:19] ...trap/engines/engine.py.224: train: epoch 48 | batch 2260/2304 [I 2022-10-27 04:54:19] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:24 [I 2022-10-27 04:54:19] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00244 [I 2022-10-27 04:54:19] ...trap/engines/engine.py.230: loss: 0.20898 [I 2022-10-27 04:54:19] ...optimizers/trijoint.py.151: total_norm: 11.997907 [I 2022-10-27 04:54:25] ...trap/engines/engine.py.224: train: epoch 48 | batch 2270/2304 [I 2022-10-27 04:54:25] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:18 [I 2022-10-27 04:54:25] ...trap/engines/engine.py.229: process: 0.56869 | load: 0.00246 [I 2022-10-27 04:54:25] ...trap/engines/engine.py.230: loss: 0.17835 [I 2022-10-27 04:54:25] ...optimizers/trijoint.py.151: total_norm: 16.132305 [I 2022-10-27 04:54:30] ...trap/engines/engine.py.224: train: epoch 48 | batch 2280/2304 [I 2022-10-27 04:54:30] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:13 [I 2022-10-27 04:54:30] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00246 [I 2022-10-27 04:54:30] ...trap/engines/engine.py.230: loss: 0.21996 [I 2022-10-27 04:54:30] ...optimizers/trijoint.py.151: total_norm: 16.296072 [I 2022-10-27 04:54:36] ...trap/engines/engine.py.224: train: epoch 48 | batch 2290/2304 [I 2022-10-27 04:54:36] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:07 [I 2022-10-27 04:54:36] ...trap/engines/engine.py.229: process: 0.53739 | load: 0.00248 [I 2022-10-27 04:54:36] ...trap/engines/engine.py.230: loss: 0.18542 [I 2022-10-27 04:54:36] ...optimizers/trijoint.py.151: total_norm: 21.677607 [I 2022-10-27 04:54:42] ...trap/engines/engine.py.224: train: epoch 48 | batch 2300/2304 [I 2022-10-27 04:54:42] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:02 [I 2022-10-27 04:54:42] ...trap/engines/engine.py.229: process: 0.56827 | load: 0.00016 [I 2022-10-27 04:54:42] ...trap/engines/engine.py.230: loss: 0.21989 [I 2022-10-27 04:54:42] ...optimizers/trijoint.py.151: total_norm: 43.804688 [I 2022-10-27 04:54:44] ...trap/engines/engine.py.224: train: epoch 48 | batch 2304/2304 [I 2022-10-27 04:54:44] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:00 [I 2022-10-27 04:54:44] ...trap/engines/engine.py.229: process: 0.54034 | load: 0.00021 [I 2022-10-27 04:54:44] ...trap/engines/engine.py.230: loss: 0.11567 [I 2022-10-27 04:54:44] ...optimizers/trijoint.py.151: total_norm: 9.583032 [S 2022-10-27 04:54:45] ...trap/engines/engine.py.129: train_epoch.epoch: 48 [S 2022-10-27 04:54:45] ...trap/engines/engine.py.129: train_epoch.loss: 0.16543704613946264 [I 2022-10-27 04:54:45] ...s/criterions/triplet.py.84: margin = 0.2950000000000002 [I 2022-10-27 04:54:46] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 48 [I 2022-10-27 04:54:46] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 04:54:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 04:54:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 04:54:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 04:54:49] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 04:54:49] ...trap/engines/engine.py.321: eval: epoch 48 | batch 0/510 [I 2022-10-27 04:54:49] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:32:10 [I 2022-10-27 04:54:49] ...trap/engines/engine.py.326: process: 3.78514 | load: 2.96591 [I 2022-10-27 04:54:53] ...trap/engines/engine.py.321: eval: epoch 48 | batch 10/510 [I 2022-10-27 04:54:53] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:05:49 [I 2022-10-27 04:54:53] ...trap/engines/engine.py.326: process: 0.32010 | load: 0.00016 [I 2022-10-27 04:54:56] ...trap/engines/engine.py.321: eval: epoch 48 | batch 20/510 [I 2022-10-27 04:54:56] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:02 [I 2022-10-27 04:54:56] ...trap/engines/engine.py.326: process: 0.33431 | load: 0.00016 [I 2022-10-27 04:54:59] ...trap/engines/engine.py.321: eval: epoch 48 | batch 30/510 [I 2022-10-27 04:54:59] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:43 [I 2022-10-27 04:54:59] ...trap/engines/engine.py.326: process: 0.32439 | load: 0.00015 [I 2022-10-27 04:55:03] ...trap/engines/engine.py.321: eval: epoch 48 | batch 40/510 [I 2022-10-27 04:55:03] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:36 [I 2022-10-27 04:55:03] ...trap/engines/engine.py.326: process: 0.33656 | load: 0.00015 [I 2022-10-27 04:55:06] ...trap/engines/engine.py.321: eval: epoch 48 | batch 50/510 [I 2022-10-27 04:55:06] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:32 [I 2022-10-27 04:55:06] ...trap/engines/engine.py.326: process: 0.32838 | load: 0.00015 [I 2022-10-27 04:55:09] ...trap/engines/engine.py.321: eval: epoch 48 | batch 60/510 [I 2022-10-27 04:55:09] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:30 [I 2022-10-27 04:55:09] ...trap/engines/engine.py.326: process: 0.33051 | load: 0.00015 [I 2022-10-27 04:55:13] ...trap/engines/engine.py.321: eval: epoch 48 | batch 70/510 [I 2022-10-27 04:55:13] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:25 [I 2022-10-27 04:55:13] ...trap/engines/engine.py.326: process: 0.32161 | load: 0.00020 [I 2022-10-27 04:55:16] ...trap/engines/engine.py.321: eval: epoch 48 | batch 80/510 [I 2022-10-27 04:55:16] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-27 04:55:16] ...trap/engines/engine.py.326: process: 0.33775 | load: 0.00018 [I 2022-10-27 04:55:19] ...trap/engines/engine.py.321: eval: epoch 48 | batch 90/510 [I 2022-10-27 04:55:19] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:20 [I 2022-10-27 04:55:19] ...trap/engines/engine.py.326: process: 0.34384 | load: 0.00016 [I 2022-10-27 04:55:23] ...trap/engines/engine.py.321: eval: epoch 48 | batch 100/510 [I 2022-10-27 04:55:23] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:12 [I 2022-10-27 04:55:23] ...trap/engines/engine.py.326: process: 0.29548 | load: 0.00016 [I 2022-10-27 04:55:26] ...trap/engines/engine.py.321: eval: epoch 48 | batch 110/510 [I 2022-10-27 04:55:26] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:01:56 [I 2022-10-27 04:55:26] ...trap/engines/engine.py.326: process: 0.28612 | load: 0.00013 [I 2022-10-27 04:55:28] ...trap/engines/engine.py.321: eval: epoch 48 | batch 120/510 [I 2022-10-27 04:55:28] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:52 [I 2022-10-27 04:55:28] ...trap/engines/engine.py.326: process: 0.28803 | load: 0.00013 [I 2022-10-27 04:55:31] ...trap/engines/engine.py.321: eval: epoch 48 | batch 130/510 [I 2022-10-27 04:55:31] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:48 [I 2022-10-27 04:55:31] ...trap/engines/engine.py.326: process: 0.28183 | load: 0.00018 [I 2022-10-27 04:55:34] ...trap/engines/engine.py.321: eval: epoch 48 | batch 140/510 [I 2022-10-27 04:55:34] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-27 04:55:34] ...trap/engines/engine.py.326: process: 0.28903 | load: 0.00017 [I 2022-10-27 04:55:37] ...trap/engines/engine.py.321: eval: epoch 48 | batch 150/510 [I 2022-10-27 04:55:37] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-27 04:55:37] ...trap/engines/engine.py.326: process: 0.29036 | load: 0.00014 [I 2022-10-27 04:55:40] ...trap/engines/engine.py.321: eval: epoch 48 | batch 160/510 [I 2022-10-27 04:55:40] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-27 04:55:40] ...trap/engines/engine.py.326: process: 0.29012 | load: 0.00020 [I 2022-10-27 04:55:43] ...trap/engines/engine.py.321: eval: epoch 48 | batch 170/510 [I 2022-10-27 04:55:43] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:39 [I 2022-10-27 04:55:43] ...trap/engines/engine.py.326: process: 0.29878 | load: 0.00015 [I 2022-10-27 04:55:46] ...trap/engines/engine.py.321: eval: epoch 48 | batch 180/510 [I 2022-10-27 04:55:46] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:37 [I 2022-10-27 04:55:46] ...trap/engines/engine.py.326: process: 0.29402 | load: 0.00017 [I 2022-10-27 04:55:49] ...trap/engines/engine.py.321: eval: epoch 48 | batch 190/510 [I 2022-10-27 04:55:49] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 04:55:49] ...trap/engines/engine.py.326: process: 0.28737 | load: 0.00019 [I 2022-10-27 04:55:52] ...trap/engines/engine.py.321: eval: epoch 48 | batch 200/510 [I 2022-10-27 04:55:52] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:29 [I 2022-10-27 04:55:52] ...trap/engines/engine.py.326: process: 0.29399 | load: 0.00014 [I 2022-10-27 04:55:55] ...trap/engines/engine.py.321: eval: epoch 48 | batch 210/510 [I 2022-10-27 04:55:55] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:31 [I 2022-10-27 04:55:55] ...trap/engines/engine.py.326: process: 0.35622 | load: 0.00015 [I 2022-10-27 04:55:57] ...trap/engines/engine.py.321: eval: epoch 48 | batch 220/510 [I 2022-10-27 04:55:57] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-27 04:55:57] ...trap/engines/engine.py.326: process: 0.29395 | load: 0.00014 [I 2022-10-27 04:56:00] ...trap/engines/engine.py.321: eval: epoch 48 | batch 230/510 [I 2022-10-27 04:56:00] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-27 04:56:00] ...trap/engines/engine.py.326: process: 0.27871 | load: 0.00015 [I 2022-10-27 04:56:03] ...trap/engines/engine.py.321: eval: epoch 48 | batch 240/510 [I 2022-10-27 04:56:03] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-27 04:56:03] ...trap/engines/engine.py.326: process: 0.29093 | load: 0.00016 [I 2022-10-27 04:56:06] ...trap/engines/engine.py.321: eval: epoch 48 | batch 250/510 [I 2022-10-27 04:56:06] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-27 04:56:06] ...trap/engines/engine.py.326: process: 0.28087 | load: 0.00017 [I 2022-10-27 04:56:09] ...trap/engines/engine.py.321: eval: epoch 48 | batch 260/510 [I 2022-10-27 04:56:09] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-27 04:56:09] ...trap/engines/engine.py.326: process: 0.29371 | load: 0.00017 [I 2022-10-27 04:56:12] ...trap/engines/engine.py.321: eval: epoch 48 | batch 270/510 [I 2022-10-27 04:56:12] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 04:56:12] ...trap/engines/engine.py.326: process: 0.28764 | load: 0.00017 [I 2022-10-27 04:56:15] ...trap/engines/engine.py.321: eval: epoch 48 | batch 280/510 [I 2022-10-27 04:56:15] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 04:56:15] ...trap/engines/engine.py.326: process: 0.29035 | load: 0.00014 [I 2022-10-27 04:56:18] ...trap/engines/engine.py.321: eval: epoch 48 | batch 290/510 [I 2022-10-27 04:56:18] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-27 04:56:18] ...trap/engines/engine.py.326: process: 0.28551 | load: 0.00015 [I 2022-10-27 04:56:20] ...trap/engines/engine.py.321: eval: epoch 48 | batch 300/510 [I 2022-10-27 04:56:20] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-27 04:56:20] ...trap/engines/engine.py.326: process: 0.29617 | load: 0.00015 [I 2022-10-27 04:56:23] ...trap/engines/engine.py.321: eval: epoch 48 | batch 310/510 [I 2022-10-27 04:56:23] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 04:56:23] ...trap/engines/engine.py.326: process: 0.28556 | load: 0.00020 [I 2022-10-27 04:56:26] ...trap/engines/engine.py.321: eval: epoch 48 | batch 320/510 [I 2022-10-27 04:56:26] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 04:56:26] ...trap/engines/engine.py.326: process: 0.29417 | load: 0.00018 [I 2022-10-27 04:56:29] ...trap/engines/engine.py.321: eval: epoch 48 | batch 330/510 [I 2022-10-27 04:56:29] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 04:56:29] ...trap/engines/engine.py.326: process: 0.28280 | load: 0.00015 [I 2022-10-27 04:56:32] ...trap/engines/engine.py.321: eval: epoch 48 | batch 340/510 [I 2022-10-27 04:56:32] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-27 04:56:32] ...trap/engines/engine.py.326: process: 0.28736 | load: 0.00014 [I 2022-10-27 04:56:35] ...trap/engines/engine.py.321: eval: epoch 48 | batch 350/510 [I 2022-10-27 04:56:35] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:48 [I 2022-10-27 04:56:35] ...trap/engines/engine.py.326: process: 0.29378 | load: 0.00018 [I 2022-10-27 04:56:38] ...trap/engines/engine.py.321: eval: epoch 48 | batch 360/510 [I 2022-10-27 04:56:38] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 04:56:38] ...trap/engines/engine.py.326: process: 0.29136 | load: 0.00016 [I 2022-10-27 04:56:41] ...trap/engines/engine.py.321: eval: epoch 48 | batch 370/510 [I 2022-10-27 04:56:41] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 04:56:41] ...trap/engines/engine.py.326: process: 0.27898 | load: 0.00017 [I 2022-10-27 04:56:44] ...trap/engines/engine.py.321: eval: epoch 48 | batch 380/510 [I 2022-10-27 04:56:44] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-27 04:56:44] ...trap/engines/engine.py.326: process: 0.29688 | load: 0.00017 [I 2022-10-27 04:56:47] ...trap/engines/engine.py.321: eval: epoch 48 | batch 390/510 [I 2022-10-27 04:56:47] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-27 04:56:47] ...trap/engines/engine.py.326: process: 0.28802 | load: 0.00016 [I 2022-10-27 04:56:49] ...trap/engines/engine.py.321: eval: epoch 48 | batch 400/510 [I 2022-10-27 04:56:49] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-27 04:56:49] ...trap/engines/engine.py.326: process: 0.28813 | load: 0.00016 [I 2022-10-27 04:56:52] ...trap/engines/engine.py.321: eval: epoch 48 | batch 410/510 [I 2022-10-27 04:56:52] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-27 04:56:52] ...trap/engines/engine.py.326: process: 0.28973 | load: 0.00016 [I 2022-10-27 04:56:55] ...trap/engines/engine.py.321: eval: epoch 48 | batch 420/510 [I 2022-10-27 04:56:55] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-27 04:56:55] ...trap/engines/engine.py.326: process: 0.29229 | load: 0.00022 [I 2022-10-27 04:56:58] ...trap/engines/engine.py.321: eval: epoch 48 | batch 430/510 [I 2022-10-27 04:56:58] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-27 04:56:58] ...trap/engines/engine.py.326: process: 0.29097 | load: 0.00023 [I 2022-10-27 04:57:01] ...trap/engines/engine.py.321: eval: epoch 48 | batch 440/510 [I 2022-10-27 04:57:01] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 04:57:01] ...trap/engines/engine.py.326: process: 0.29174 | load: 0.00016 [I 2022-10-27 04:57:04] ...trap/engines/engine.py.321: eval: epoch 48 | batch 450/510 [I 2022-10-27 04:57:04] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 04:57:04] ...trap/engines/engine.py.326: process: 0.28590 | load: 0.00018 [I 2022-10-27 04:57:07] ...trap/engines/engine.py.321: eval: epoch 48 | batch 460/510 [I 2022-10-27 04:57:07] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 04:57:07] ...trap/engines/engine.py.326: process: 0.29275 | load: 0.00017 [I 2022-10-27 04:57:10] ...trap/engines/engine.py.321: eval: epoch 48 | batch 470/510 [I 2022-10-27 04:57:10] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-27 04:57:10] ...trap/engines/engine.py.326: process: 0.28893 | load: 0.00021 [I 2022-10-27 04:57:12] ...trap/engines/engine.py.321: eval: epoch 48 | batch 480/510 [I 2022-10-27 04:57:12] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 04:57:12] ...trap/engines/engine.py.326: process: 0.29264 | load: 0.00027 [I 2022-10-27 04:57:15] ...trap/engines/engine.py.321: eval: epoch 48 | batch 490/510 [I 2022-10-27 04:57:15] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 04:57:15] ...trap/engines/engine.py.326: process: 0.28260 | load: 0.00016 [I 2022-10-27 04:57:18] ...trap/engines/engine.py.321: eval: epoch 48 | batch 500/510 [I 2022-10-27 04:57:18] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 04:57:18] ...trap/engines/engine.py.326: process: 0.29168 | load: 0.00017 [I 2022-10-27 04:57:21] ...trap/engines/engine.py.321: eval: epoch 48 | batch 510/510 [I 2022-10-27 04:57:21] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 04:57:21] ...trap/engines/engine.py.326: process: 0.29105 | load: 0.00010 [S 2022-10-27 04:57:22] ...trap/engines/engine.py.132: eval_epoch.epoch: 48 [I 2022-10-27 04:57:22] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.703200 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.903100 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936400 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.689500 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.900000 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933900 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014985 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007489 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006053 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013735 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008602 [S 2022-10-27 04:57:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005243 [I 2022-10-27 04:57:27] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 04:57:27] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 04:57:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 04:57:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 04:57:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 04:57:29] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 04:57:32] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 04:57:38] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 04:57:38] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 04:57:38] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 04:57:41] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 04:57:47] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 04:57:47] ...trap/engines/engine.py.163: Training model on trainset for epoch 49 [I 2022-10-27 04:57:47] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 04:57:50] ...trap/engines/engine.py.224: train: epoch 49 | batch 0/2304 [I 2022-10-27 04:57:50] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:53 [I 2022-10-27 04:57:50] ...trap/engines/engine.py.229: process: 2.65355 | load: 1.98960 [I 2022-10-27 04:57:50] ...trap/engines/engine.py.230: loss: 0.21314 [I 2022-10-27 04:57:50] ...optimizers/trijoint.py.151: total_norm: 23.479115 [I 2022-10-27 04:57:55] ...trap/engines/engine.py.224: train: epoch 49 | batch 10/2304 [I 2022-10-27 04:57:55] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:56 [I 2022-10-27 04:57:55] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00392 [I 2022-10-27 04:57:55] ...trap/engines/engine.py.230: loss: 0.18697 [I 2022-10-27 04:57:55] ...optimizers/trijoint.py.151: total_norm: 12.796022 [I 2022-10-27 04:58:01] ...trap/engines/engine.py.224: train: epoch 49 | batch 20/2304 [I 2022-10-27 04:58:01] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:11 [I 2022-10-27 04:58:01] ...trap/engines/engine.py.229: process: 0.56162 | load: 0.00324 [I 2022-10-27 04:58:01] ...trap/engines/engine.py.230: loss: 0.18199 [I 2022-10-27 04:58:01] ...optimizers/trijoint.py.151: total_norm: 17.319719 [I 2022-10-27 04:58:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 30/2304 [I 2022-10-27 04:58:06] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:19 [I 2022-10-27 04:58:06] ...trap/engines/engine.py.229: process: 0.56585 | load: 0.00290 [I 2022-10-27 04:58:06] ...trap/engines/engine.py.230: loss: 0.18632 [I 2022-10-27 04:58:06] ...optimizers/trijoint.py.151: total_norm: 19.667336 [I 2022-10-27 04:58:12] ...trap/engines/engine.py.224: train: epoch 49 | batch 40/2304 [I 2022-10-27 04:58:12] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:06 [I 2022-10-27 04:58:12] ...trap/engines/engine.py.229: process: 0.57013 | load: 0.00295 [I 2022-10-27 04:58:12] ...trap/engines/engine.py.230: loss: 0.17717 [I 2022-10-27 04:58:12] ...optimizers/trijoint.py.151: total_norm: 20.168432 [I 2022-10-27 04:58:18] ...trap/engines/engine.py.224: train: epoch 49 | batch 50/2304 [I 2022-10-27 04:58:18] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:26 [I 2022-10-27 04:58:18] ...trap/engines/engine.py.229: process: 0.62068 | load: 0.00264 [I 2022-10-27 04:58:18] ...trap/engines/engine.py.230: loss: 0.20240 [I 2022-10-27 04:58:18] ...optimizers/trijoint.py.151: total_norm: 20.164982 [I 2022-10-27 04:58:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 60/2304 [I 2022-10-27 04:58:23] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:59 [I 2022-10-27 04:58:23] ...trap/engines/engine.py.229: process: 0.56843 | load: 0.00288 [I 2022-10-27 04:58:23] ...trap/engines/engine.py.230: loss: 0.12580 [I 2022-10-27 04:58:23] ...optimizers/trijoint.py.151: total_norm: 8.941293 [I 2022-10-27 04:58:29] ...trap/engines/engine.py.224: train: epoch 49 | batch 70/2304 [I 2022-10-27 04:58:29] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:46 [I 2022-10-27 04:58:29] ...trap/engines/engine.py.229: process: 0.55065 | load: 0.00264 [I 2022-10-27 04:58:29] ...trap/engines/engine.py.230: loss: 0.15800 [I 2022-10-27 04:58:29] ...optimizers/trijoint.py.151: total_norm: 12.682878 [I 2022-10-27 04:58:35] ...trap/engines/engine.py.224: train: epoch 49 | batch 80/2304 [I 2022-10-27 04:58:35] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:53 [I 2022-10-27 04:58:35] ...trap/engines/engine.py.229: process: 0.56736 | load: 0.00262 [I 2022-10-27 04:58:35] ...trap/engines/engine.py.230: loss: 0.17564 [I 2022-10-27 04:58:35] ...optimizers/trijoint.py.151: total_norm: 21.880457 [I 2022-10-27 04:58:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 90/2304 [I 2022-10-27 04:58:40] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:50 [I 2022-10-27 04:58:40] ...trap/engines/engine.py.229: process: 0.57029 | load: 0.00263 [I 2022-10-27 04:58:40] ...trap/engines/engine.py.230: loss: 0.22109 [I 2022-10-27 04:58:40] ...optimizers/trijoint.py.151: total_norm: 12.755409 [I 2022-10-27 04:58:46] ...trap/engines/engine.py.224: train: epoch 49 | batch 100/2304 [I 2022-10-27 04:58:46] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:45 [I 2022-10-27 04:58:46] ...trap/engines/engine.py.229: process: 0.57270 | load: 0.00260 [I 2022-10-27 04:58:46] ...trap/engines/engine.py.230: loss: 0.16856 [I 2022-10-27 04:58:46] ...optimizers/trijoint.py.151: total_norm: 13.204910 [I 2022-10-27 04:58:51] ...trap/engines/engine.py.224: train: epoch 49 | batch 110/2304 [I 2022-10-27 04:58:51] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:31 [I 2022-10-27 04:58:51] ...trap/engines/engine.py.229: process: 0.56414 | load: 0.00266 [I 2022-10-27 04:58:51] ...trap/engines/engine.py.230: loss: 0.21216 [I 2022-10-27 04:58:51] ...optimizers/trijoint.py.151: total_norm: 14.137503 [I 2022-10-27 04:58:57] ...trap/engines/engine.py.224: train: epoch 49 | batch 120/2304 [I 2022-10-27 04:58:57] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:12 [I 2022-10-27 04:58:57] ...trap/engines/engine.py.229: process: 0.55888 | load: 0.00267 [I 2022-10-27 04:58:57] ...trap/engines/engine.py.230: loss: 0.18053 [I 2022-10-27 04:58:57] ...optimizers/trijoint.py.151: total_norm: 22.054173 [I 2022-10-27 04:59:03] ...trap/engines/engine.py.224: train: epoch 49 | batch 130/2304 [I 2022-10-27 04:59:03] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:20 [I 2022-10-27 04:59:03] ...trap/engines/engine.py.229: process: 0.57481 | load: 0.00255 [I 2022-10-27 04:59:03] ...trap/engines/engine.py.230: loss: 0.17027 [I 2022-10-27 04:59:03] ...optimizers/trijoint.py.151: total_norm: 15.830151 [I 2022-10-27 04:59:08] ...trap/engines/engine.py.224: train: epoch 49 | batch 140/2304 [I 2022-10-27 04:59:08] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:08 [I 2022-10-27 04:59:08] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00262 [I 2022-10-27 04:59:08] ...trap/engines/engine.py.230: loss: 0.11984 [I 2022-10-27 04:59:08] ...optimizers/trijoint.py.151: total_norm: 29.746338 [I 2022-10-27 04:59:14] ...trap/engines/engine.py.224: train: epoch 49 | batch 150/2304 [I 2022-10-27 04:59:14] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:08 [I 2022-10-27 04:59:14] ...trap/engines/engine.py.229: process: 0.58270 | load: 0.00258 [I 2022-10-27 04:59:14] ...trap/engines/engine.py.230: loss: 0.28188 [I 2022-10-27 04:59:14] ...optimizers/trijoint.py.151: total_norm: 23.464464 [I 2022-10-27 04:59:19] ...trap/engines/engine.py.224: train: epoch 49 | batch 160/2304 [I 2022-10-27 04:59:19] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:05 [I 2022-10-27 04:59:19] ...trap/engines/engine.py.229: process: 0.56329 | load: 0.00246 [I 2022-10-27 04:59:19] ...trap/engines/engine.py.230: loss: 0.18057 [I 2022-10-27 04:59:19] ...optimizers/trijoint.py.151: total_norm: 15.814065 [I 2022-10-27 04:59:25] ...trap/engines/engine.py.224: train: epoch 49 | batch 170/2304 [I 2022-10-27 04:59:25] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:20:04 [I 2022-10-27 04:59:25] ...trap/engines/engine.py.229: process: 0.55696 | load: 0.00266 [I 2022-10-27 04:59:25] ...trap/engines/engine.py.230: loss: 0.19345 [I 2022-10-27 04:59:25] ...optimizers/trijoint.py.151: total_norm: 17.537634 [I 2022-10-27 04:59:31] ...trap/engines/engine.py.224: train: epoch 49 | batch 180/2304 [I 2022-10-27 04:59:31] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:43 [I 2022-10-27 04:59:31] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00255 [I 2022-10-27 04:59:31] ...trap/engines/engine.py.230: loss: 0.18225 [I 2022-10-27 04:59:31] ...optimizers/trijoint.py.151: total_norm: 33.307167 [I 2022-10-27 04:59:36] ...trap/engines/engine.py.224: train: epoch 49 | batch 190/2304 [I 2022-10-27 04:59:36] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:38 [I 2022-10-27 04:59:36] ...trap/engines/engine.py.229: process: 0.56849 | load: 0.00279 [I 2022-10-27 04:59:36] ...trap/engines/engine.py.230: loss: 0.18900 [I 2022-10-27 04:59:36] ...optimizers/trijoint.py.151: total_norm: 18.855310 [I 2022-10-27 04:59:42] ...trap/engines/engine.py.224: train: epoch 49 | batch 200/2304 [I 2022-10-27 04:59:42] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:40 [I 2022-10-27 04:59:42] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00247 [I 2022-10-27 04:59:42] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-27 04:59:42] ...optimizers/trijoint.py.151: total_norm: 6.869907 [I 2022-10-27 04:59:48] ...trap/engines/engine.py.224: train: epoch 49 | batch 210/2304 [I 2022-10-27 04:59:48] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:44 [I 2022-10-27 04:59:48] ...trap/engines/engine.py.229: process: 0.56784 | load: 0.00253 [I 2022-10-27 04:59:48] ...trap/engines/engine.py.230: loss: 0.21972 [I 2022-10-27 04:59:48] ...optimizers/trijoint.py.151: total_norm: 43.255554 [I 2022-10-27 04:59:53] ...trap/engines/engine.py.224: train: epoch 49 | batch 220/2304 [I 2022-10-27 04:59:53] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:28 [I 2022-10-27 04:59:53] ...trap/engines/engine.py.229: process: 0.55338 | load: 0.00253 [I 2022-10-27 04:59:53] ...trap/engines/engine.py.230: loss: 0.23206 [I 2022-10-27 04:59:53] ...optimizers/trijoint.py.151: total_norm: 11.054659 [I 2022-10-27 04:59:59] ...trap/engines/engine.py.224: train: epoch 49 | batch 230/2304 [I 2022-10-27 04:59:59] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:31 [I 2022-10-27 04:59:59] ...trap/engines/engine.py.229: process: 0.57306 | load: 0.00260 [I 2022-10-27 04:59:59] ...trap/engines/engine.py.230: loss: 0.16054 [I 2022-10-27 04:59:59] ...optimizers/trijoint.py.151: total_norm: 14.351658 [I 2022-10-27 05:00:04] ...trap/engines/engine.py.224: train: epoch 49 | batch 240/2304 [I 2022-10-27 05:00:04] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:04 [I 2022-10-27 05:00:04] ...trap/engines/engine.py.229: process: 0.54633 | load: 0.00255 [I 2022-10-27 05:00:04] ...trap/engines/engine.py.230: loss: 0.13022 [I 2022-10-27 05:00:04] ...optimizers/trijoint.py.151: total_norm: 21.833651 [I 2022-10-27 05:00:10] ...trap/engines/engine.py.224: train: epoch 49 | batch 250/2304 [I 2022-10-27 05:00:10] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:11 [I 2022-10-27 05:00:10] ...trap/engines/engine.py.229: process: 0.56621 | load: 0.00264 [I 2022-10-27 05:00:10] ...trap/engines/engine.py.230: loss: 0.16956 [I 2022-10-27 05:00:10] ...optimizers/trijoint.py.151: total_norm: 13.449978 [I 2022-10-27 05:00:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 260/2304 [I 2022-10-27 05:00:15] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:58 [I 2022-10-27 05:00:15] ...trap/engines/engine.py.229: process: 0.57383 | load: 0.00253 [I 2022-10-27 05:00:15] ...trap/engines/engine.py.230: loss: 0.21726 [I 2022-10-27 05:00:15] ...optimizers/trijoint.py.151: total_norm: 23.551558 [I 2022-10-27 05:00:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 270/2304 [I 2022-10-27 05:00:21] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:54 [I 2022-10-27 05:00:21] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00259 [I 2022-10-27 05:00:21] ...trap/engines/engine.py.230: loss: 0.18073 [I 2022-10-27 05:00:21] ...optimizers/trijoint.py.151: total_norm: 15.511172 [I 2022-10-27 05:00:27] ...trap/engines/engine.py.224: train: epoch 49 | batch 280/2304 [I 2022-10-27 05:00:27] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:45 [I 2022-10-27 05:00:27] ...trap/engines/engine.py.229: process: 0.55061 | load: 0.00253 [I 2022-10-27 05:00:27] ...trap/engines/engine.py.230: loss: 0.12853 [I 2022-10-27 05:00:27] ...optimizers/trijoint.py.151: total_norm: 8.793893 [I 2022-10-27 05:00:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 290/2304 [I 2022-10-27 05:00:32] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:51 [I 2022-10-27 05:00:32] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00265 [I 2022-10-27 05:00:32] ...trap/engines/engine.py.230: loss: 0.13662 [I 2022-10-27 05:00:32] ...optimizers/trijoint.py.151: total_norm: 15.515978 [I 2022-10-27 05:00:38] ...trap/engines/engine.py.224: train: epoch 49 | batch 300/2304 [I 2022-10-27 05:00:38] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:32 [I 2022-10-27 05:00:38] ...trap/engines/engine.py.229: process: 0.54563 | load: 0.00261 [I 2022-10-27 05:00:38] ...trap/engines/engine.py.230: loss: 0.21709 [I 2022-10-27 05:00:38] ...optimizers/trijoint.py.151: total_norm: 18.019882 [I 2022-10-27 05:00:44] ...trap/engines/engine.py.224: train: epoch 49 | batch 310/2304 [I 2022-10-27 05:00:44] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:33 [I 2022-10-27 05:00:44] ...trap/engines/engine.py.229: process: 0.56284 | load: 0.00256 [I 2022-10-27 05:00:44] ...trap/engines/engine.py.230: loss: 0.20538 [I 2022-10-27 05:00:44] ...optimizers/trijoint.py.151: total_norm: 19.631754 [I 2022-10-27 05:00:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 320/2304 [I 2022-10-27 05:00:49] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:36 [I 2022-10-27 05:00:49] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00258 [I 2022-10-27 05:00:49] ...trap/engines/engine.py.230: loss: 0.16587 [I 2022-10-27 05:00:49] ...optimizers/trijoint.py.151: total_norm: 21.852722 [I 2022-10-27 05:00:55] ...trap/engines/engine.py.224: train: epoch 49 | batch 330/2304 [I 2022-10-27 05:00:55] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:18 [I 2022-10-27 05:00:55] ...trap/engines/engine.py.229: process: 0.54971 | load: 0.00271 [I 2022-10-27 05:00:55] ...trap/engines/engine.py.230: loss: 0.19535 [I 2022-10-27 05:00:55] ...optimizers/trijoint.py.151: total_norm: 26.381273 [I 2022-10-27 05:01:00] ...trap/engines/engine.py.224: train: epoch 49 | batch 340/2304 [I 2022-10-27 05:01:00] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:09 [I 2022-10-27 05:01:00] ...trap/engines/engine.py.229: process: 0.56522 | load: 0.00245 [I 2022-10-27 05:01:00] ...trap/engines/engine.py.230: loss: 0.20262 [I 2022-10-27 05:01:00] ...optimizers/trijoint.py.151: total_norm: 9.198767 [I 2022-10-27 05:01:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 350/2304 [I 2022-10-27 05:01:06] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:10 [I 2022-10-27 05:01:06] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00242 [I 2022-10-27 05:01:06] ...trap/engines/engine.py.230: loss: 0.14836 [I 2022-10-27 05:01:06] ...optimizers/trijoint.py.151: total_norm: 14.653107 [I 2022-10-27 05:01:12] ...trap/engines/engine.py.224: train: epoch 49 | batch 360/2304 [I 2022-10-27 05:01:12] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:06 [I 2022-10-27 05:01:12] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00257 [I 2022-10-27 05:01:12] ...trap/engines/engine.py.230: loss: 0.18981 [I 2022-10-27 05:01:12] ...optimizers/trijoint.py.151: total_norm: 21.635889 [I 2022-10-27 05:01:17] ...trap/engines/engine.py.224: train: epoch 49 | batch 370/2304 [I 2022-10-27 05:01:17] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:17:57 [I 2022-10-27 05:01:17] ...trap/engines/engine.py.229: process: 0.56387 | load: 0.00251 [I 2022-10-27 05:01:17] ...trap/engines/engine.py.230: loss: 0.21227 [I 2022-10-27 05:01:17] ...optimizers/trijoint.py.151: total_norm: 34.024479 [I 2022-10-27 05:01:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 380/2304 [I 2022-10-27 05:01:23] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:50 [I 2022-10-27 05:01:23] ...trap/engines/engine.py.229: process: 0.54569 | load: 0.00247 [I 2022-10-27 05:01:23] ...trap/engines/engine.py.230: loss: 0.15118 [I 2022-10-27 05:01:23] ...optimizers/trijoint.py.151: total_norm: 15.163183 [I 2022-10-27 05:01:28] ...trap/engines/engine.py.224: train: epoch 49 | batch 390/2304 [I 2022-10-27 05:01:28] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:41 [I 2022-10-27 05:01:28] ...trap/engines/engine.py.229: process: 0.56769 | load: 0.00258 [I 2022-10-27 05:01:28] ...trap/engines/engine.py.230: loss: 0.18095 [I 2022-10-27 05:01:28] ...optimizers/trijoint.py.151: total_norm: 16.092356 [I 2022-10-27 05:01:34] ...trap/engines/engine.py.224: train: epoch 49 | batch 400/2304 [I 2022-10-27 05:01:34] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:52 [I 2022-10-27 05:01:34] ...trap/engines/engine.py.229: process: 0.58356 | load: 0.00247 [I 2022-10-27 05:01:34] ...trap/engines/engine.py.230: loss: 0.11405 [I 2022-10-27 05:01:34] ...optimizers/trijoint.py.151: total_norm: 9.479456 [I 2022-10-27 05:01:39] ...trap/engines/engine.py.224: train: epoch 49 | batch 410/2304 [I 2022-10-27 05:01:39] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:32 [I 2022-10-27 05:01:39] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00252 [I 2022-10-27 05:01:39] ...trap/engines/engine.py.230: loss: 0.21288 [I 2022-10-27 05:01:39] ...optimizers/trijoint.py.151: total_norm: 20.489326 [I 2022-10-27 05:01:45] ...trap/engines/engine.py.224: train: epoch 49 | batch 420/2304 [I 2022-10-27 05:01:45] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:22 [I 2022-10-27 05:01:45] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00246 [I 2022-10-27 05:01:45] ...trap/engines/engine.py.230: loss: 0.18127 [I 2022-10-27 05:01:45] ...optimizers/trijoint.py.151: total_norm: 25.049187 [I 2022-10-27 05:01:51] ...trap/engines/engine.py.224: train: epoch 49 | batch 430/2304 [I 2022-10-27 05:01:51] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:20 [I 2022-10-27 05:01:51] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00243 [I 2022-10-27 05:01:51] ...trap/engines/engine.py.230: loss: 0.14102 [I 2022-10-27 05:01:51] ...optimizers/trijoint.py.151: total_norm: 13.669154 [I 2022-10-27 05:01:56] ...trap/engines/engine.py.224: train: epoch 49 | batch 440/2304 [I 2022-10-27 05:01:56] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:27 [I 2022-10-27 05:01:56] ...trap/engines/engine.py.229: process: 0.57486 | load: 0.00250 [I 2022-10-27 05:01:56] ...trap/engines/engine.py.230: loss: 0.13124 [I 2022-10-27 05:01:56] ...optimizers/trijoint.py.151: total_norm: 12.148689 [I 2022-10-27 05:02:02] ...trap/engines/engine.py.224: train: epoch 49 | batch 450/2304 [I 2022-10-27 05:02:02] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:17 [I 2022-10-27 05:02:02] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00254 [I 2022-10-27 05:02:02] ...trap/engines/engine.py.230: loss: 0.15116 [I 2022-10-27 05:02:02] ...optimizers/trijoint.py.151: total_norm: 20.590761 [I 2022-10-27 05:02:07] ...trap/engines/engine.py.224: train: epoch 49 | batch 460/2304 [I 2022-10-27 05:02:07] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:09 [I 2022-10-27 05:02:07] ...trap/engines/engine.py.229: process: 0.56330 | load: 0.00244 [I 2022-10-27 05:02:07] ...trap/engines/engine.py.230: loss: 0.14878 [I 2022-10-27 05:02:07] ...optimizers/trijoint.py.151: total_norm: 14.573594 [I 2022-10-27 05:02:13] ...trap/engines/engine.py.224: train: epoch 49 | batch 470/2304 [I 2022-10-27 05:02:13] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:16:57 [I 2022-10-27 05:02:13] ...trap/engines/engine.py.229: process: 0.56302 | load: 0.00247 [I 2022-10-27 05:02:13] ...trap/engines/engine.py.230: loss: 0.17888 [I 2022-10-27 05:02:13] ...optimizers/trijoint.py.151: total_norm: 20.689861 [I 2022-10-27 05:02:18] ...trap/engines/engine.py.224: train: epoch 49 | batch 480/2304 [I 2022-10-27 05:02:18] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:50 [I 2022-10-27 05:02:18] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00245 [I 2022-10-27 05:02:18] ...trap/engines/engine.py.230: loss: 0.19331 [I 2022-10-27 05:02:18] ...optimizers/trijoint.py.151: total_norm: 13.134155 [I 2022-10-27 05:02:24] ...trap/engines/engine.py.224: train: epoch 49 | batch 490/2304 [I 2022-10-27 05:02:24] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:48 [I 2022-10-27 05:02:24] ...trap/engines/engine.py.229: process: 0.56417 | load: 0.00241 [I 2022-10-27 05:02:24] ...trap/engines/engine.py.230: loss: 0.17710 [I 2022-10-27 05:02:24] ...optimizers/trijoint.py.151: total_norm: 19.035191 [I 2022-10-27 05:02:30] ...trap/engines/engine.py.224: train: epoch 49 | batch 500/2304 [I 2022-10-27 05:02:30] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:36 [I 2022-10-27 05:02:30] ...trap/engines/engine.py.229: process: 0.54153 | load: 0.00242 [I 2022-10-27 05:02:30] ...trap/engines/engine.py.230: loss: 0.21705 [I 2022-10-27 05:02:30] ...optimizers/trijoint.py.151: total_norm: 21.483755 [I 2022-10-27 05:02:35] ...trap/engines/engine.py.224: train: epoch 49 | batch 510/2304 [I 2022-10-27 05:02:35] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:42 [I 2022-10-27 05:02:35] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00240 [I 2022-10-27 05:02:35] ...trap/engines/engine.py.230: loss: 0.16826 [I 2022-10-27 05:02:35] ...optimizers/trijoint.py.151: total_norm: 20.298193 [I 2022-10-27 05:02:41] ...trap/engines/engine.py.224: train: epoch 49 | batch 520/2304 [I 2022-10-27 05:02:41] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:42 [I 2022-10-27 05:02:41] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00241 [I 2022-10-27 05:02:41] ...trap/engines/engine.py.230: loss: 0.15314 [I 2022-10-27 05:02:41] ...optimizers/trijoint.py.151: total_norm: 28.869558 [I 2022-10-27 05:02:46] ...trap/engines/engine.py.224: train: epoch 49 | batch 530/2304 [I 2022-10-27 05:02:46] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:33 [I 2022-10-27 05:02:46] ...trap/engines/engine.py.229: process: 0.56283 | load: 0.00251 [I 2022-10-27 05:02:46] ...trap/engines/engine.py.230: loss: 0.13052 [I 2022-10-27 05:02:46] ...optimizers/trijoint.py.151: total_norm: 11.290586 [I 2022-10-27 05:02:52] ...trap/engines/engine.py.224: train: epoch 49 | batch 540/2304 [I 2022-10-27 05:02:52] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:30 [I 2022-10-27 05:02:52] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00247 [I 2022-10-27 05:02:52] ...trap/engines/engine.py.230: loss: 0.11607 [I 2022-10-27 05:02:52] ...optimizers/trijoint.py.151: total_norm: 15.216502 [I 2022-10-27 05:02:58] ...trap/engines/engine.py.224: train: epoch 49 | batch 550/2304 [I 2022-10-27 05:02:58] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:27 [I 2022-10-27 05:02:58] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00247 [I 2022-10-27 05:02:58] ...trap/engines/engine.py.230: loss: 0.20367 [I 2022-10-27 05:02:58] ...optimizers/trijoint.py.151: total_norm: 30.633339 [I 2022-10-27 05:03:03] ...trap/engines/engine.py.224: train: epoch 49 | batch 560/2304 [I 2022-10-27 05:03:03] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:14 [I 2022-10-27 05:03:03] ...trap/engines/engine.py.229: process: 0.55486 | load: 0.00244 [I 2022-10-27 05:03:03] ...trap/engines/engine.py.230: loss: 0.14999 [I 2022-10-27 05:03:03] ...optimizers/trijoint.py.151: total_norm: 16.325932 [I 2022-10-27 05:03:09] ...trap/engines/engine.py.224: train: epoch 49 | batch 570/2304 [I 2022-10-27 05:03:09] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:15 [I 2022-10-27 05:03:09] ...trap/engines/engine.py.229: process: 0.56592 | load: 0.00241 [I 2022-10-27 05:03:09] ...trap/engines/engine.py.230: loss: 0.17719 [I 2022-10-27 05:03:09] ...optimizers/trijoint.py.151: total_norm: 19.132532 [I 2022-10-27 05:03:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 580/2304 [I 2022-10-27 05:03:15] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:00 [I 2022-10-27 05:03:15] ...trap/engines/engine.py.229: process: 0.55094 | load: 0.00240 [I 2022-10-27 05:03:15] ...trap/engines/engine.py.230: loss: 0.15267 [I 2022-10-27 05:03:15] ...optimizers/trijoint.py.151: total_norm: 17.778368 [I 2022-10-27 05:03:20] ...trap/engines/engine.py.224: train: epoch 49 | batch 590/2304 [I 2022-10-27 05:03:20] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:00 [I 2022-10-27 05:03:20] ...trap/engines/engine.py.229: process: 0.56376 | load: 0.00250 [I 2022-10-27 05:03:20] ...trap/engines/engine.py.230: loss: 0.13119 [I 2022-10-27 05:03:20] ...optimizers/trijoint.py.151: total_norm: 11.788279 [I 2022-10-27 05:03:26] ...trap/engines/engine.py.224: train: epoch 49 | batch 600/2304 [I 2022-10-27 05:03:26] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:57 [I 2022-10-27 05:03:26] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00256 [I 2022-10-27 05:03:26] ...trap/engines/engine.py.230: loss: 0.18648 [I 2022-10-27 05:03:26] ...optimizers/trijoint.py.151: total_norm: 23.717133 [I 2022-10-27 05:03:31] ...trap/engines/engine.py.224: train: epoch 49 | batch 610/2304 [I 2022-10-27 05:03:31] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:43 [I 2022-10-27 05:03:31] ...trap/engines/engine.py.229: process: 0.54491 | load: 0.00256 [I 2022-10-27 05:03:31] ...trap/engines/engine.py.230: loss: 0.12298 [I 2022-10-27 05:03:31] ...optimizers/trijoint.py.151: total_norm: 48.045418 [I 2022-10-27 05:03:37] ...trap/engines/engine.py.224: train: epoch 49 | batch 620/2304 [I 2022-10-27 05:03:37] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:37 [I 2022-10-27 05:03:37] ...trap/engines/engine.py.229: process: 0.55360 | load: 0.00250 [I 2022-10-27 05:03:37] ...trap/engines/engine.py.230: loss: 0.25545 [I 2022-10-27 05:03:37] ...optimizers/trijoint.py.151: total_norm: 28.208830 [I 2022-10-27 05:03:43] ...trap/engines/engine.py.224: train: epoch 49 | batch 630/2304 [I 2022-10-27 05:03:43] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:42 [I 2022-10-27 05:03:43] ...trap/engines/engine.py.229: process: 0.55758 | load: 0.00244 [I 2022-10-27 05:03:43] ...trap/engines/engine.py.230: loss: 0.16714 [I 2022-10-27 05:03:43] ...optimizers/trijoint.py.151: total_norm: 13.919004 [I 2022-10-27 05:03:48] ...trap/engines/engine.py.224: train: epoch 49 | batch 640/2304 [I 2022-10-27 05:03:48] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:27 [I 2022-10-27 05:03:48] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00244 [I 2022-10-27 05:03:48] ...trap/engines/engine.py.230: loss: 0.19279 [I 2022-10-27 05:03:48] ...optimizers/trijoint.py.151: total_norm: 12.051338 [I 2022-10-27 05:03:54] ...trap/engines/engine.py.224: train: epoch 49 | batch 650/2304 [I 2022-10-27 05:03:54] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:22 [I 2022-10-27 05:03:54] ...trap/engines/engine.py.229: process: 0.55019 | load: 0.00245 [I 2022-10-27 05:03:54] ...trap/engines/engine.py.230: loss: 0.15899 [I 2022-10-27 05:03:54] ...optimizers/trijoint.py.151: total_norm: 12.363913 [I 2022-10-27 05:03:59] ...trap/engines/engine.py.224: train: epoch 49 | batch 660/2304 [I 2022-10-27 05:03:59] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:18 [I 2022-10-27 05:03:59] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00242 [I 2022-10-27 05:03:59] ...trap/engines/engine.py.230: loss: 0.19529 [I 2022-10-27 05:03:59] ...optimizers/trijoint.py.151: total_norm: 22.756355 [I 2022-10-27 05:04:05] ...trap/engines/engine.py.224: train: epoch 49 | batch 670/2304 [I 2022-10-27 05:04:05] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:12 [I 2022-10-27 05:04:05] ...trap/engines/engine.py.229: process: 0.56805 | load: 0.00310 [I 2022-10-27 05:04:05] ...trap/engines/engine.py.230: loss: 0.16109 [I 2022-10-27 05:04:05] ...optimizers/trijoint.py.151: total_norm: 14.928482 [I 2022-10-27 05:04:11] ...trap/engines/engine.py.224: train: epoch 49 | batch 680/2304 [I 2022-10-27 05:04:11] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:10 [I 2022-10-27 05:04:11] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00238 [I 2022-10-27 05:04:11] ...trap/engines/engine.py.230: loss: 0.15357 [I 2022-10-27 05:04:11] ...optimizers/trijoint.py.151: total_norm: 18.510239 [I 2022-10-27 05:04:16] ...trap/engines/engine.py.224: train: epoch 49 | batch 690/2304 [I 2022-10-27 05:04:16] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:03 [I 2022-10-27 05:04:16] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00246 [I 2022-10-27 05:04:16] ...trap/engines/engine.py.230: loss: 0.17407 [I 2022-10-27 05:04:16] ...optimizers/trijoint.py.151: total_norm: 19.205517 [I 2022-10-27 05:04:22] ...trap/engines/engine.py.224: train: epoch 49 | batch 700/2304 [I 2022-10-27 05:04:22] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:55 [I 2022-10-27 05:04:22] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00266 [I 2022-10-27 05:04:22] ...trap/engines/engine.py.230: loss: 0.13910 [I 2022-10-27 05:04:22] ...optimizers/trijoint.py.151: total_norm: 12.037354 [I 2022-10-27 05:04:27] ...trap/engines/engine.py.224: train: epoch 49 | batch 710/2304 [I 2022-10-27 05:04:27] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:48 [I 2022-10-27 05:04:27] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00254 [I 2022-10-27 05:04:27] ...trap/engines/engine.py.230: loss: 0.12712 [I 2022-10-27 05:04:27] ...optimizers/trijoint.py.151: total_norm: 10.325548 [I 2022-10-27 05:04:33] ...trap/engines/engine.py.224: train: epoch 49 | batch 720/2304 [I 2022-10-27 05:04:33] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:49 [I 2022-10-27 05:04:33] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00247 [I 2022-10-27 05:04:33] ...trap/engines/engine.py.230: loss: 0.17768 [I 2022-10-27 05:04:33] ...optimizers/trijoint.py.151: total_norm: 20.996584 [I 2022-10-27 05:04:39] ...trap/engines/engine.py.224: train: epoch 49 | batch 730/2304 [I 2022-10-27 05:04:39] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:37 [I 2022-10-27 05:04:39] ...trap/engines/engine.py.229: process: 0.54498 | load: 0.00243 [I 2022-10-27 05:04:39] ...trap/engines/engine.py.230: loss: 0.11731 [I 2022-10-27 05:04:39] ...optimizers/trijoint.py.151: total_norm: 8.887493 [I 2022-10-27 05:04:44] ...trap/engines/engine.py.224: train: epoch 49 | batch 740/2304 [I 2022-10-27 05:04:44] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:32 [I 2022-10-27 05:04:44] ...trap/engines/engine.py.229: process: 0.54544 | load: 0.00244 [I 2022-10-27 05:04:44] ...trap/engines/engine.py.230: loss: 0.13830 [I 2022-10-27 05:04:44] ...optimizers/trijoint.py.151: total_norm: 10.132977 [I 2022-10-27 05:04:50] ...trap/engines/engine.py.224: train: epoch 49 | batch 750/2304 [I 2022-10-27 05:04:50] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:22 [I 2022-10-27 05:04:50] ...trap/engines/engine.py.229: process: 0.54244 | load: 0.00245 [I 2022-10-27 05:04:50] ...trap/engines/engine.py.230: loss: 0.12498 [I 2022-10-27 05:04:50] ...optimizers/trijoint.py.151: total_norm: 16.658783 [I 2022-10-27 05:04:55] ...trap/engines/engine.py.224: train: epoch 49 | batch 760/2304 [I 2022-10-27 05:04:55] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:20 [I 2022-10-27 05:04:55] ...trap/engines/engine.py.229: process: 0.55881 | load: 0.00243 [I 2022-10-27 05:04:55] ...trap/engines/engine.py.230: loss: 0.18135 [I 2022-10-27 05:04:55] ...optimizers/trijoint.py.151: total_norm: 18.258402 [I 2022-10-27 05:05:01] ...trap/engines/engine.py.224: train: epoch 49 | batch 770/2304 [I 2022-10-27 05:05:01] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:18 [I 2022-10-27 05:05:01] ...trap/engines/engine.py.229: process: 0.57191 | load: 0.00246 [I 2022-10-27 05:05:01] ...trap/engines/engine.py.230: loss: 0.12959 [I 2022-10-27 05:05:01] ...optimizers/trijoint.py.151: total_norm: 19.810123 [I 2022-10-27 05:05:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 780/2304 [I 2022-10-27 05:05:06] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:08 [I 2022-10-27 05:05:06] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00245 [I 2022-10-27 05:05:06] ...trap/engines/engine.py.230: loss: 0.13467 [I 2022-10-27 05:05:06] ...optimizers/trijoint.py.151: total_norm: 8.396638 [I 2022-10-27 05:05:12] ...trap/engines/engine.py.224: train: epoch 49 | batch 790/2304 [I 2022-10-27 05:05:12] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:13:58 [I 2022-10-27 05:05:12] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00240 [I 2022-10-27 05:05:12] ...trap/engines/engine.py.230: loss: 0.14837 [I 2022-10-27 05:05:12] ...optimizers/trijoint.py.151: total_norm: 13.977304 [I 2022-10-27 05:05:18] ...trap/engines/engine.py.224: train: epoch 49 | batch 800/2304 [I 2022-10-27 05:05:18] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:57 [I 2022-10-27 05:05:18] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00246 [I 2022-10-27 05:05:18] ...trap/engines/engine.py.230: loss: 0.10617 [I 2022-10-27 05:05:18] ...optimizers/trijoint.py.151: total_norm: 11.618866 [I 2022-10-27 05:05:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 810/2304 [I 2022-10-27 05:05:23] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:46 [I 2022-10-27 05:05:23] ...trap/engines/engine.py.229: process: 0.54790 | load: 0.00243 [I 2022-10-27 05:05:23] ...trap/engines/engine.py.230: loss: 0.15003 [I 2022-10-27 05:05:23] ...optimizers/trijoint.py.151: total_norm: 13.869082 [I 2022-10-27 05:05:29] ...trap/engines/engine.py.224: train: epoch 49 | batch 820/2304 [I 2022-10-27 05:05:29] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:42 [I 2022-10-27 05:05:29] ...trap/engines/engine.py.229: process: 0.54588 | load: 0.00247 [I 2022-10-27 05:05:29] ...trap/engines/engine.py.230: loss: 0.18867 [I 2022-10-27 05:05:29] ...optimizers/trijoint.py.151: total_norm: 15.310925 [I 2022-10-27 05:05:34] ...trap/engines/engine.py.224: train: epoch 49 | batch 830/2304 [I 2022-10-27 05:05:34] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:51 [I 2022-10-27 05:05:34] ...trap/engines/engine.py.229: process: 0.56778 | load: 0.00285 [I 2022-10-27 05:05:34] ...trap/engines/engine.py.230: loss: 0.13468 [I 2022-10-27 05:05:34] ...optimizers/trijoint.py.151: total_norm: 35.446049 [I 2022-10-27 05:05:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 840/2304 [I 2022-10-27 05:05:40] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:31 [I 2022-10-27 05:05:40] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00246 [I 2022-10-27 05:05:40] ...trap/engines/engine.py.230: loss: 0.15396 [I 2022-10-27 05:05:40] ...optimizers/trijoint.py.151: total_norm: 19.580887 [I 2022-10-27 05:05:45] ...trap/engines/engine.py.224: train: epoch 49 | batch 850/2304 [I 2022-10-27 05:05:45] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:21 [I 2022-10-27 05:05:45] ...trap/engines/engine.py.229: process: 0.55152 | load: 0.00251 [I 2022-10-27 05:05:45] ...trap/engines/engine.py.230: loss: 0.15696 [I 2022-10-27 05:05:45] ...optimizers/trijoint.py.151: total_norm: 17.022583 [I 2022-10-27 05:05:51] ...trap/engines/engine.py.224: train: epoch 49 | batch 860/2304 [I 2022-10-27 05:05:51] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:35 [I 2022-10-27 05:05:51] ...trap/engines/engine.py.229: process: 0.55293 | load: 0.00250 [I 2022-10-27 05:05:51] ...trap/engines/engine.py.230: loss: 0.11983 [I 2022-10-27 05:05:51] ...optimizers/trijoint.py.151: total_norm: 12.953871 [I 2022-10-27 05:05:57] ...trap/engines/engine.py.224: train: epoch 49 | batch 870/2304 [I 2022-10-27 05:05:57] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:27 [I 2022-10-27 05:05:57] ...trap/engines/engine.py.229: process: 0.57923 | load: 0.00244 [I 2022-10-27 05:05:57] ...trap/engines/engine.py.230: loss: 0.14515 [I 2022-10-27 05:05:57] ...optimizers/trijoint.py.151: total_norm: 17.829794 [I 2022-10-27 05:06:02] ...trap/engines/engine.py.224: train: epoch 49 | batch 880/2304 [I 2022-10-27 05:06:02] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:18 [I 2022-10-27 05:06:02] ...trap/engines/engine.py.229: process: 0.55024 | load: 0.00259 [I 2022-10-27 05:06:02] ...trap/engines/engine.py.230: loss: 0.17154 [I 2022-10-27 05:06:02] ...optimizers/trijoint.py.151: total_norm: 19.908512 [I 2022-10-27 05:06:08] ...trap/engines/engine.py.224: train: epoch 49 | batch 890/2304 [I 2022-10-27 05:06:08] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:06 [I 2022-10-27 05:06:08] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00244 [I 2022-10-27 05:06:08] ...trap/engines/engine.py.230: loss: 0.19061 [I 2022-10-27 05:06:08] ...optimizers/trijoint.py.151: total_norm: 15.232133 [I 2022-10-27 05:06:14] ...trap/engines/engine.py.224: train: epoch 49 | batch 900/2304 [I 2022-10-27 05:06:14] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:03 [I 2022-10-27 05:06:14] ...trap/engines/engine.py.229: process: 0.54990 | load: 0.00242 [I 2022-10-27 05:06:14] ...trap/engines/engine.py.230: loss: 0.20953 [I 2022-10-27 05:06:14] ...optimizers/trijoint.py.151: total_norm: 51.598618 [I 2022-10-27 05:06:19] ...trap/engines/engine.py.224: train: epoch 49 | batch 910/2304 [I 2022-10-27 05:06:19] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:51 [I 2022-10-27 05:06:19] ...trap/engines/engine.py.229: process: 0.55127 | load: 0.00247 [I 2022-10-27 05:06:19] ...trap/engines/engine.py.230: loss: 0.18595 [I 2022-10-27 05:06:19] ...optimizers/trijoint.py.151: total_norm: 25.095701 [I 2022-10-27 05:06:25] ...trap/engines/engine.py.224: train: epoch 49 | batch 920/2304 [I 2022-10-27 05:06:25] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:51 [I 2022-10-27 05:06:25] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00248 [I 2022-10-27 05:06:25] ...trap/engines/engine.py.230: loss: 0.12219 [I 2022-10-27 05:06:25] ...optimizers/trijoint.py.151: total_norm: 14.267974 [I 2022-10-27 05:06:30] ...trap/engines/engine.py.224: train: epoch 49 | batch 930/2304 [I 2022-10-27 05:06:30] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:46 [I 2022-10-27 05:06:30] ...trap/engines/engine.py.229: process: 0.56401 | load: 0.00255 [I 2022-10-27 05:06:30] ...trap/engines/engine.py.230: loss: 0.11944 [I 2022-10-27 05:06:30] ...optimizers/trijoint.py.151: total_norm: 16.860605 [I 2022-10-27 05:06:36] ...trap/engines/engine.py.224: train: epoch 49 | batch 940/2304 [I 2022-10-27 05:06:36] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:40 [I 2022-10-27 05:06:36] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00248 [I 2022-10-27 05:06:36] ...trap/engines/engine.py.230: loss: 0.13553 [I 2022-10-27 05:06:36] ...optimizers/trijoint.py.151: total_norm: 11.768117 [I 2022-10-27 05:06:42] ...trap/engines/engine.py.224: train: epoch 49 | batch 950/2304 [I 2022-10-27 05:06:42] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:13:02 [I 2022-10-27 05:06:42] ...trap/engines/engine.py.229: process: 0.56883 | load: 0.00263 [I 2022-10-27 05:06:42] ...trap/engines/engine.py.230: loss: 0.12020 [I 2022-10-27 05:06:42] ...optimizers/trijoint.py.151: total_norm: 12.901462 [I 2022-10-27 05:06:47] ...trap/engines/engine.py.224: train: epoch 49 | batch 960/2304 [I 2022-10-27 05:06:47] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:33 [I 2022-10-27 05:06:47] ...trap/engines/engine.py.229: process: 0.55547 | load: 0.00247 [I 2022-10-27 05:06:47] ...trap/engines/engine.py.230: loss: 0.18435 [I 2022-10-27 05:06:47] ...optimizers/trijoint.py.151: total_norm: 21.001406 [I 2022-10-27 05:06:53] ...trap/engines/engine.py.224: train: epoch 49 | batch 970/2304 [I 2022-10-27 05:06:53] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:27 [I 2022-10-27 05:06:53] ...trap/engines/engine.py.229: process: 0.55422 | load: 0.00246 [I 2022-10-27 05:06:53] ...trap/engines/engine.py.230: loss: 0.19091 [I 2022-10-27 05:06:53] ...optimizers/trijoint.py.151: total_norm: 16.625811 [I 2022-10-27 05:06:59] ...trap/engines/engine.py.224: train: epoch 49 | batch 980/2304 [I 2022-10-27 05:06:59] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:24 [I 2022-10-27 05:06:59] ...trap/engines/engine.py.229: process: 0.56786 | load: 0.00245 [I 2022-10-27 05:06:59] ...trap/engines/engine.py.230: loss: 0.14738 [I 2022-10-27 05:06:59] ...optimizers/trijoint.py.151: total_norm: 10.725912 [I 2022-10-27 05:07:04] ...trap/engines/engine.py.224: train: epoch 49 | batch 990/2304 [I 2022-10-27 05:07:04] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:12 [I 2022-10-27 05:07:04] ...trap/engines/engine.py.229: process: 0.54511 | load: 0.00249 [I 2022-10-27 05:07:04] ...trap/engines/engine.py.230: loss: 0.16262 [I 2022-10-27 05:07:04] ...optimizers/trijoint.py.151: total_norm: 14.088763 [I 2022-10-27 05:07:10] ...trap/engines/engine.py.224: train: epoch 49 | batch 1000/2304 [I 2022-10-27 05:07:10] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:13 [I 2022-10-27 05:07:10] ...trap/engines/engine.py.229: process: 0.56547 | load: 0.00250 [I 2022-10-27 05:07:10] ...trap/engines/engine.py.230: loss: 0.13254 [I 2022-10-27 05:07:10] ...optimizers/trijoint.py.151: total_norm: 8.000979 [I 2022-10-27 05:07:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 1010/2304 [I 2022-10-27 05:07:15] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:11 [I 2022-10-27 05:07:15] ...trap/engines/engine.py.229: process: 0.57573 | load: 0.00266 [I 2022-10-27 05:07:15] ...trap/engines/engine.py.230: loss: 0.17528 [I 2022-10-27 05:07:15] ...optimizers/trijoint.py.151: total_norm: 12.535915 [I 2022-10-27 05:07:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 1020/2304 [I 2022-10-27 05:07:21] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:12:10 [I 2022-10-27 05:07:21] ...trap/engines/engine.py.229: process: 0.56823 | load: 0.00245 [I 2022-10-27 05:07:21] ...trap/engines/engine.py.230: loss: 0.14549 [I 2022-10-27 05:07:21] ...optimizers/trijoint.py.151: total_norm: 12.310528 [I 2022-10-27 05:07:27] ...trap/engines/engine.py.224: train: epoch 49 | batch 1030/2304 [I 2022-10-27 05:07:27] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:55 [I 2022-10-27 05:07:27] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00245 [I 2022-10-27 05:07:27] ...trap/engines/engine.py.230: loss: 0.24782 [I 2022-10-27 05:07:27] ...optimizers/trijoint.py.151: total_norm: 15.172998 [I 2022-10-27 05:07:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 1040/2304 [I 2022-10-27 05:07:32] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:48 [I 2022-10-27 05:07:32] ...trap/engines/engine.py.229: process: 0.56801 | load: 0.00246 [I 2022-10-27 05:07:32] ...trap/engines/engine.py.230: loss: 0.10362 [I 2022-10-27 05:07:32] ...optimizers/trijoint.py.151: total_norm: 12.043618 [I 2022-10-27 05:07:38] ...trap/engines/engine.py.224: train: epoch 49 | batch 1050/2304 [I 2022-10-27 05:07:38] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:46 [I 2022-10-27 05:07:38] ...trap/engines/engine.py.229: process: 0.56396 | load: 0.00244 [I 2022-10-27 05:07:38] ...trap/engines/engine.py.230: loss: 0.13520 [I 2022-10-27 05:07:38] ...optimizers/trijoint.py.151: total_norm: 12.425714 [I 2022-10-27 05:07:44] ...trap/engines/engine.py.224: train: epoch 49 | batch 1060/2304 [I 2022-10-27 05:07:44] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:37 [I 2022-10-27 05:07:44] ...trap/engines/engine.py.229: process: 0.56711 | load: 0.00258 [I 2022-10-27 05:07:44] ...trap/engines/engine.py.230: loss: 0.11540 [I 2022-10-27 05:07:44] ...optimizers/trijoint.py.151: total_norm: 8.951475 [I 2022-10-27 05:07:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 1070/2304 [I 2022-10-27 05:07:49] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:31 [I 2022-10-27 05:07:49] ...trap/engines/engine.py.229: process: 0.56301 | load: 0.00249 [I 2022-10-27 05:07:49] ...trap/engines/engine.py.230: loss: 0.14272 [I 2022-10-27 05:07:49] ...optimizers/trijoint.py.151: total_norm: 17.351009 [I 2022-10-27 05:07:55] ...trap/engines/engine.py.224: train: epoch 49 | batch 1080/2304 [I 2022-10-27 05:07:55] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:33 [I 2022-10-27 05:07:55] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00248 [I 2022-10-27 05:07:55] ...trap/engines/engine.py.230: loss: 0.12598 [I 2022-10-27 05:07:55] ...optimizers/trijoint.py.151: total_norm: 9.662559 [I 2022-10-27 05:08:01] ...trap/engines/engine.py.224: train: epoch 49 | batch 1090/2304 [I 2022-10-27 05:08:01] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:28 [I 2022-10-27 05:08:01] ...trap/engines/engine.py.229: process: 0.57343 | load: 0.00246 [I 2022-10-27 05:08:01] ...trap/engines/engine.py.230: loss: 0.20043 [I 2022-10-27 05:08:01] ...optimizers/trijoint.py.151: total_norm: 16.947098 [I 2022-10-27 05:08:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 1100/2304 [I 2022-10-27 05:08:06] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:20 [I 2022-10-27 05:08:06] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00244 [I 2022-10-27 05:08:06] ...trap/engines/engine.py.230: loss: 0.17280 [I 2022-10-27 05:08:06] ...optimizers/trijoint.py.151: total_norm: 15.392205 [I 2022-10-27 05:08:12] ...trap/engines/engine.py.224: train: epoch 49 | batch 1110/2304 [I 2022-10-27 05:08:12] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:11 [I 2022-10-27 05:08:12] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00245 [I 2022-10-27 05:08:12] ...trap/engines/engine.py.230: loss: 0.19151 [I 2022-10-27 05:08:12] ...optimizers/trijoint.py.151: total_norm: 22.613506 [I 2022-10-27 05:08:18] ...trap/engines/engine.py.224: train: epoch 49 | batch 1120/2304 [I 2022-10-27 05:08:18] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:06 [I 2022-10-27 05:08:18] ...trap/engines/engine.py.229: process: 0.57068 | load: 0.00252 [I 2022-10-27 05:08:18] ...trap/engines/engine.py.230: loss: 0.14827 [I 2022-10-27 05:08:18] ...optimizers/trijoint.py.151: total_norm: 7.042850 [I 2022-10-27 05:08:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 1130/2304 [I 2022-10-27 05:08:23] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:10:59 [I 2022-10-27 05:08:23] ...trap/engines/engine.py.229: process: 0.55937 | load: 0.00247 [I 2022-10-27 05:08:23] ...trap/engines/engine.py.230: loss: 0.13134 [I 2022-10-27 05:08:23] ...optimizers/trijoint.py.151: total_norm: 11.938236 [I 2022-10-27 05:08:29] ...trap/engines/engine.py.224: train: epoch 49 | batch 1140/2304 [I 2022-10-27 05:08:29] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:11:05 [I 2022-10-27 05:08:29] ...trap/engines/engine.py.229: process: 0.62431 | load: 0.00243 [I 2022-10-27 05:08:29] ...trap/engines/engine.py.230: loss: 0.14240 [I 2022-10-27 05:08:29] ...optimizers/trijoint.py.151: total_norm: 16.522842 [I 2022-10-27 05:08:34] ...trap/engines/engine.py.224: train: epoch 49 | batch 1150/2304 [I 2022-10-27 05:08:34] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:44 [I 2022-10-27 05:08:34] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00243 [I 2022-10-27 05:08:34] ...trap/engines/engine.py.230: loss: 0.18883 [I 2022-10-27 05:08:34] ...optimizers/trijoint.py.151: total_norm: 18.844170 [I 2022-10-27 05:08:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 1160/2304 [I 2022-10-27 05:08:40] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:38 [I 2022-10-27 05:08:40] ...trap/engines/engine.py.229: process: 0.56299 | load: 0.00244 [I 2022-10-27 05:08:40] ...trap/engines/engine.py.230: loss: 0.11726 [I 2022-10-27 05:08:40] ...optimizers/trijoint.py.151: total_norm: 12.996392 [I 2022-10-27 05:08:46] ...trap/engines/engine.py.224: train: epoch 49 | batch 1170/2304 [I 2022-10-27 05:08:46] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:32 [I 2022-10-27 05:08:46] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00244 [I 2022-10-27 05:08:46] ...trap/engines/engine.py.230: loss: 0.16799 [I 2022-10-27 05:08:46] ...optimizers/trijoint.py.151: total_norm: 21.307707 [I 2022-10-27 05:08:51] ...trap/engines/engine.py.224: train: epoch 49 | batch 1180/2304 [I 2022-10-27 05:08:51] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:25 [I 2022-10-27 05:08:51] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00245 [I 2022-10-27 05:08:51] ...trap/engines/engine.py.230: loss: 0.15136 [I 2022-10-27 05:08:51] ...optimizers/trijoint.py.151: total_norm: 11.030133 [I 2022-10-27 05:08:57] ...trap/engines/engine.py.224: train: epoch 49 | batch 1190/2304 [I 2022-10-27 05:08:57] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:16 [I 2022-10-27 05:08:57] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00251 [I 2022-10-27 05:08:57] ...trap/engines/engine.py.230: loss: 0.15362 [I 2022-10-27 05:08:57] ...optimizers/trijoint.py.151: total_norm: 21.223717 [I 2022-10-27 05:09:02] ...trap/engines/engine.py.224: train: epoch 49 | batch 1200/2304 [I 2022-10-27 05:09:02] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:17 [I 2022-10-27 05:09:02] ...trap/engines/engine.py.229: process: 0.56300 | load: 0.00248 [I 2022-10-27 05:09:02] ...trap/engines/engine.py.230: loss: 0.14092 [I 2022-10-27 05:09:02] ...optimizers/trijoint.py.151: total_norm: 24.471348 [I 2022-10-27 05:09:08] ...trap/engines/engine.py.224: train: epoch 49 | batch 1210/2304 [I 2022-10-27 05:09:08] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:14 [I 2022-10-27 05:09:08] ...trap/engines/engine.py.229: process: 0.54263 | load: 0.00243 [I 2022-10-27 05:09:08] ...trap/engines/engine.py.230: loss: 0.25878 [I 2022-10-27 05:09:08] ...optimizers/trijoint.py.151: total_norm: 20.593513 [I 2022-10-27 05:09:14] ...trap/engines/engine.py.224: train: epoch 49 | batch 1220/2304 [I 2022-10-27 05:09:14] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:10 [I 2022-10-27 05:09:14] ...trap/engines/engine.py.229: process: 0.56463 | load: 0.00248 [I 2022-10-27 05:09:14] ...trap/engines/engine.py.230: loss: 0.11069 [I 2022-10-27 05:09:14] ...optimizers/trijoint.py.151: total_norm: 12.384827 [I 2022-10-27 05:09:19] ...trap/engines/engine.py.224: train: epoch 49 | batch 1230/2304 [I 2022-10-27 05:09:19] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:10:00 [I 2022-10-27 05:09:19] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00241 [I 2022-10-27 05:09:19] ...trap/engines/engine.py.230: loss: 0.14815 [I 2022-10-27 05:09:19] ...optimizers/trijoint.py.151: total_norm: 14.422433 [I 2022-10-27 05:09:25] ...trap/engines/engine.py.224: train: epoch 49 | batch 1240/2304 [I 2022-10-27 05:09:25] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:09:54 [I 2022-10-27 05:09:25] ...trap/engines/engine.py.229: process: 0.55346 | load: 0.00245 [I 2022-10-27 05:09:25] ...trap/engines/engine.py.230: loss: 0.14788 [I 2022-10-27 05:09:25] ...optimizers/trijoint.py.151: total_norm: 27.819778 [I 2022-10-27 05:09:31] ...trap/engines/engine.py.224: train: epoch 49 | batch 1250/2304 [I 2022-10-27 05:09:31] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:53 [I 2022-10-27 05:09:31] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00248 [I 2022-10-27 05:09:31] ...trap/engines/engine.py.230: loss: 0.14546 [I 2022-10-27 05:09:31] ...optimizers/trijoint.py.151: total_norm: 25.868980 [I 2022-10-27 05:09:36] ...trap/engines/engine.py.224: train: epoch 49 | batch 1260/2304 [I 2022-10-27 05:09:36] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:09:47 [I 2022-10-27 05:09:36] ...trap/engines/engine.py.229: process: 0.57167 | load: 0.00249 [I 2022-10-27 05:09:36] ...trap/engines/engine.py.230: loss: 0.13045 [I 2022-10-27 05:09:36] ...optimizers/trijoint.py.151: total_norm: 16.604567 [I 2022-10-27 05:09:42] ...trap/engines/engine.py.224: train: epoch 49 | batch 1270/2304 [I 2022-10-27 05:09:42] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:45 [I 2022-10-27 05:09:42] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00243 [I 2022-10-27 05:09:42] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-27 05:09:42] ...optimizers/trijoint.py.151: total_norm: 18.046150 [I 2022-10-27 05:09:48] ...trap/engines/engine.py.224: train: epoch 49 | batch 1280/2304 [I 2022-10-27 05:09:48] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:09:29 [I 2022-10-27 05:09:48] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00246 [I 2022-10-27 05:09:48] ...trap/engines/engine.py.230: loss: 0.13852 [I 2022-10-27 05:09:48] ...optimizers/trijoint.py.151: total_norm: 17.652746 [I 2022-10-27 05:09:53] ...trap/engines/engine.py.224: train: epoch 49 | batch 1290/2304 [I 2022-10-27 05:09:53] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:25 [I 2022-10-27 05:09:53] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00245 [I 2022-10-27 05:09:53] ...trap/engines/engine.py.230: loss: 0.11940 [I 2022-10-27 05:09:53] ...optimizers/trijoint.py.151: total_norm: 11.633784 [I 2022-10-27 05:09:59] ...trap/engines/engine.py.224: train: epoch 49 | batch 1300/2304 [I 2022-10-27 05:09:59] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:21 [I 2022-10-27 05:09:59] ...trap/engines/engine.py.229: process: 0.56583 | load: 0.00242 [I 2022-10-27 05:09:59] ...trap/engines/engine.py.230: loss: 0.14524 [I 2022-10-27 05:09:59] ...optimizers/trijoint.py.151: total_norm: 18.074362 [I 2022-10-27 05:10:04] ...trap/engines/engine.py.224: train: epoch 49 | batch 1310/2304 [I 2022-10-27 05:10:04] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:12 [I 2022-10-27 05:10:04] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00240 [I 2022-10-27 05:10:04] ...trap/engines/engine.py.230: loss: 0.15595 [I 2022-10-27 05:10:04] ...optimizers/trijoint.py.151: total_norm: 11.130770 [I 2022-10-27 05:10:10] ...trap/engines/engine.py.224: train: epoch 49 | batch 1320/2304 [I 2022-10-27 05:10:10] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:06 [I 2022-10-27 05:10:10] ...trap/engines/engine.py.229: process: 0.54939 | load: 0.00241 [I 2022-10-27 05:10:10] ...trap/engines/engine.py.230: loss: 0.17843 [I 2022-10-27 05:10:10] ...optimizers/trijoint.py.151: total_norm: 16.711487 [I 2022-10-27 05:10:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 1330/2304 [I 2022-10-27 05:10:15] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:09:04 [I 2022-10-27 05:10:15] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00242 [I 2022-10-27 05:10:15] ...trap/engines/engine.py.230: loss: 0.11221 [I 2022-10-27 05:10:15] ...optimizers/trijoint.py.151: total_norm: 10.874192 [I 2022-10-27 05:10:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 1340/2304 [I 2022-10-27 05:10:21] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:09:03 [I 2022-10-27 05:10:21] ...trap/engines/engine.py.229: process: 0.56725 | load: 0.00243 [I 2022-10-27 05:10:21] ...trap/engines/engine.py.230: loss: 0.19259 [I 2022-10-27 05:10:21] ...optimizers/trijoint.py.151: total_norm: 32.935497 [I 2022-10-27 05:10:27] ...trap/engines/engine.py.224: train: epoch 49 | batch 1350/2304 [I 2022-10-27 05:10:27] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:08:53 [I 2022-10-27 05:10:27] ...trap/engines/engine.py.229: process: 0.56692 | load: 0.00245 [I 2022-10-27 05:10:27] ...trap/engines/engine.py.230: loss: 0.23182 [I 2022-10-27 05:10:27] ...optimizers/trijoint.py.151: total_norm: 36.460258 [I 2022-10-27 05:10:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 1360/2304 [I 2022-10-27 05:10:32] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:08:47 [I 2022-10-27 05:10:32] ...trap/engines/engine.py.229: process: 0.56606 | load: 0.00252 [I 2022-10-27 05:10:32] ...trap/engines/engine.py.230: loss: 0.14709 [I 2022-10-27 05:10:32] ...optimizers/trijoint.py.151: total_norm: 44.308544 [I 2022-10-27 05:10:38] ...trap/engines/engine.py.224: train: epoch 49 | batch 1370/2304 [I 2022-10-27 05:10:38] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:38 [I 2022-10-27 05:10:38] ...trap/engines/engine.py.229: process: 0.55949 | load: 0.00246 [I 2022-10-27 05:10:38] ...trap/engines/engine.py.230: loss: 0.13781 [I 2022-10-27 05:10:38] ...optimizers/trijoint.py.151: total_norm: 25.058165 [I 2022-10-27 05:10:44] ...trap/engines/engine.py.224: train: epoch 49 | batch 1380/2304 [I 2022-10-27 05:10:44] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:36 [I 2022-10-27 05:10:44] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00243 [I 2022-10-27 05:10:44] ...trap/engines/engine.py.230: loss: 0.15444 [I 2022-10-27 05:10:44] ...optimizers/trijoint.py.151: total_norm: 9.981571 [I 2022-10-27 05:10:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 1390/2304 [I 2022-10-27 05:10:49] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:33 [I 2022-10-27 05:10:49] ...trap/engines/engine.py.229: process: 0.56044 | load: 0.00242 [I 2022-10-27 05:10:49] ...trap/engines/engine.py.230: loss: 0.15632 [I 2022-10-27 05:10:49] ...optimizers/trijoint.py.151: total_norm: 27.202456 [I 2022-10-27 05:10:55] ...trap/engines/engine.py.224: train: epoch 49 | batch 1400/2304 [I 2022-10-27 05:10:55] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:39 [I 2022-10-27 05:10:55] ...trap/engines/engine.py.229: process: 0.56390 | load: 0.00244 [I 2022-10-27 05:10:55] ...trap/engines/engine.py.230: loss: 0.16616 [I 2022-10-27 05:10:55] ...optimizers/trijoint.py.151: total_norm: 13.125843 [I 2022-10-27 05:11:00] ...trap/engines/engine.py.224: train: epoch 49 | batch 1410/2304 [I 2022-10-27 05:11:00] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:19 [I 2022-10-27 05:11:00] ...trap/engines/engine.py.229: process: 0.54761 | load: 0.00243 [I 2022-10-27 05:11:00] ...trap/engines/engine.py.230: loss: 0.13131 [I 2022-10-27 05:11:00] ...optimizers/trijoint.py.151: total_norm: 17.227097 [I 2022-10-27 05:11:06] ...trap/engines/engine.py.224: train: epoch 49 | batch 1420/2304 [I 2022-10-27 05:11:06] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:11 [I 2022-10-27 05:11:06] ...trap/engines/engine.py.229: process: 0.55104 | load: 0.00246 [I 2022-10-27 05:11:06] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-27 05:11:06] ...optimizers/trijoint.py.151: total_norm: 9.871200 [I 2022-10-27 05:11:12] ...trap/engines/engine.py.224: train: epoch 49 | batch 1430/2304 [I 2022-10-27 05:11:12] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:10 [I 2022-10-27 05:11:12] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00250 [I 2022-10-27 05:11:12] ...trap/engines/engine.py.230: loss: 0.11107 [I 2022-10-27 05:11:12] ...optimizers/trijoint.py.151: total_norm: 17.477991 [I 2022-10-27 05:11:17] ...trap/engines/engine.py.224: train: epoch 49 | batch 1440/2304 [I 2022-10-27 05:11:17] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:04 [I 2022-10-27 05:11:17] ...trap/engines/engine.py.229: process: 0.56449 | load: 0.00250 [I 2022-10-27 05:11:17] ...trap/engines/engine.py.230: loss: 0.17184 [I 2022-10-27 05:11:17] ...optimizers/trijoint.py.151: total_norm: 30.509274 [I 2022-10-27 05:11:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 1450/2304 [I 2022-10-27 05:11:23] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:07:54 [I 2022-10-27 05:11:23] ...trap/engines/engine.py.229: process: 0.55235 | load: 0.00247 [I 2022-10-27 05:11:23] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-27 05:11:23] ...optimizers/trijoint.py.151: total_norm: 20.659786 [I 2022-10-27 05:11:28] ...trap/engines/engine.py.224: train: epoch 49 | batch 1460/2304 [I 2022-10-27 05:11:28] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:07:48 [I 2022-10-27 05:11:28] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00244 [I 2022-10-27 05:11:28] ...trap/engines/engine.py.230: loss: 0.17034 [I 2022-10-27 05:11:28] ...optimizers/trijoint.py.151: total_norm: 25.579828 [I 2022-10-27 05:11:34] ...trap/engines/engine.py.224: train: epoch 49 | batch 1470/2304 [I 2022-10-27 05:11:34] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:07:50 [I 2022-10-27 05:11:34] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00246 [I 2022-10-27 05:11:34] ...trap/engines/engine.py.230: loss: 0.15445 [I 2022-10-27 05:11:34] ...optimizers/trijoint.py.151: total_norm: 29.214439 [I 2022-10-27 05:11:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 1480/2304 [I 2022-10-27 05:11:40] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:39 [I 2022-10-27 05:11:40] ...trap/engines/engine.py.229: process: 0.55111 | load: 0.00245 [I 2022-10-27 05:11:40] ...trap/engines/engine.py.230: loss: 0.13082 [I 2022-10-27 05:11:40] ...optimizers/trijoint.py.151: total_norm: 18.734344 [I 2022-10-27 05:11:45] ...trap/engines/engine.py.224: train: epoch 49 | batch 1490/2304 [I 2022-10-27 05:11:45] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:33 [I 2022-10-27 05:11:45] ...trap/engines/engine.py.229: process: 0.55299 | load: 0.00246 [I 2022-10-27 05:11:45] ...trap/engines/engine.py.230: loss: 0.21029 [I 2022-10-27 05:11:45] ...optimizers/trijoint.py.151: total_norm: 21.515388 [I 2022-10-27 05:11:51] ...trap/engines/engine.py.224: train: epoch 49 | batch 1500/2304 [I 2022-10-27 05:11:51] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:29 [I 2022-10-27 05:11:51] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00245 [I 2022-10-27 05:11:51] ...trap/engines/engine.py.230: loss: 0.20821 [I 2022-10-27 05:11:51] ...optimizers/trijoint.py.151: total_norm: 17.181627 [I 2022-10-27 05:11:56] ...trap/engines/engine.py.224: train: epoch 49 | batch 1510/2304 [I 2022-10-27 05:11:56] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:20 [I 2022-10-27 05:11:56] ...trap/engines/engine.py.229: process: 0.55093 | load: 0.00244 [I 2022-10-27 05:11:56] ...trap/engines/engine.py.230: loss: 0.18803 [I 2022-10-27 05:11:56] ...optimizers/trijoint.py.151: total_norm: 14.716207 [I 2022-10-27 05:12:02] ...trap/engines/engine.py.224: train: epoch 49 | batch 1520/2304 [I 2022-10-27 05:12:02] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:17 [I 2022-10-27 05:12:02] ...trap/engines/engine.py.229: process: 0.56352 | load: 0.00262 [I 2022-10-27 05:12:02] ...trap/engines/engine.py.230: loss: 0.19532 [I 2022-10-27 05:12:02] ...optimizers/trijoint.py.151: total_norm: 24.814259 [I 2022-10-27 05:12:08] ...trap/engines/engine.py.224: train: epoch 49 | batch 1530/2304 [I 2022-10-27 05:12:08] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:12 [I 2022-10-27 05:12:08] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00251 [I 2022-10-27 05:12:08] ...trap/engines/engine.py.230: loss: 0.12496 [I 2022-10-27 05:12:08] ...optimizers/trijoint.py.151: total_norm: 13.206209 [I 2022-10-27 05:12:13] ...trap/engines/engine.py.224: train: epoch 49 | batch 1540/2304 [I 2022-10-27 05:12:13] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:07 [I 2022-10-27 05:12:13] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00249 [I 2022-10-27 05:12:13] ...trap/engines/engine.py.230: loss: 0.17200 [I 2022-10-27 05:12:13] ...optimizers/trijoint.py.151: total_norm: 24.733858 [I 2022-10-27 05:12:19] ...trap/engines/engine.py.224: train: epoch 49 | batch 1550/2304 [I 2022-10-27 05:12:19] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:07:01 [I 2022-10-27 05:12:19] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00247 [I 2022-10-27 05:12:19] ...trap/engines/engine.py.230: loss: 0.23109 [I 2022-10-27 05:12:19] ...optimizers/trijoint.py.151: total_norm: 16.739645 [I 2022-10-27 05:12:24] ...trap/engines/engine.py.224: train: epoch 49 | batch 1560/2304 [I 2022-10-27 05:12:24] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:06:53 [I 2022-10-27 05:12:24] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00243 [I 2022-10-27 05:12:24] ...trap/engines/engine.py.230: loss: 0.15344 [I 2022-10-27 05:12:24] ...optimizers/trijoint.py.151: total_norm: 17.016291 [I 2022-10-27 05:12:30] ...trap/engines/engine.py.224: train: epoch 49 | batch 1570/2304 [I 2022-10-27 05:12:30] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:06:51 [I 2022-10-27 05:12:30] ...trap/engines/engine.py.229: process: 0.56390 | load: 0.00249 [I 2022-10-27 05:12:30] ...trap/engines/engine.py.230: loss: 0.14680 [I 2022-10-27 05:12:30] ...optimizers/trijoint.py.151: total_norm: 8.511711 [I 2022-10-27 05:12:36] ...trap/engines/engine.py.224: train: epoch 49 | batch 1580/2304 [I 2022-10-27 05:12:36] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:06:46 [I 2022-10-27 05:12:36] ...trap/engines/engine.py.229: process: 0.56619 | load: 0.00248 [I 2022-10-27 05:12:36] ...trap/engines/engine.py.230: loss: 0.19349 [I 2022-10-27 05:12:36] ...optimizers/trijoint.py.151: total_norm: 17.868643 [I 2022-10-27 05:12:41] ...trap/engines/engine.py.224: train: epoch 49 | batch 1590/2304 [I 2022-10-27 05:12:41] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:40 [I 2022-10-27 05:12:41] ...trap/engines/engine.py.229: process: 0.57241 | load: 0.00244 [I 2022-10-27 05:12:41] ...trap/engines/engine.py.230: loss: 0.18121 [I 2022-10-27 05:12:41] ...optimizers/trijoint.py.151: total_norm: 28.617628 [I 2022-10-27 05:12:47] ...trap/engines/engine.py.224: train: epoch 49 | batch 1600/2304 [I 2022-10-27 05:12:47] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:33 [I 2022-10-27 05:12:47] ...trap/engines/engine.py.229: process: 0.55042 | load: 0.00244 [I 2022-10-27 05:12:47] ...trap/engines/engine.py.230: loss: 0.19232 [I 2022-10-27 05:12:47] ...optimizers/trijoint.py.151: total_norm: 11.387190 [I 2022-10-27 05:12:53] ...trap/engines/engine.py.224: train: epoch 49 | batch 1610/2304 [I 2022-10-27 05:12:53] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:29 [I 2022-10-27 05:12:53] ...trap/engines/engine.py.229: process: 0.56436 | load: 0.00255 [I 2022-10-27 05:12:53] ...trap/engines/engine.py.230: loss: 0.18624 [I 2022-10-27 05:12:53] ...optimizers/trijoint.py.151: total_norm: 21.418459 [I 2022-10-27 05:12:58] ...trap/engines/engine.py.224: train: epoch 49 | batch 1620/2304 [I 2022-10-27 05:12:58] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:20 [I 2022-10-27 05:12:58] ...trap/engines/engine.py.229: process: 0.54902 | load: 0.00266 [I 2022-10-27 05:12:58] ...trap/engines/engine.py.230: loss: 0.22421 [I 2022-10-27 05:12:58] ...optimizers/trijoint.py.151: total_norm: 33.910465 [I 2022-10-27 05:13:04] ...trap/engines/engine.py.224: train: epoch 49 | batch 1630/2304 [I 2022-10-27 05:13:04] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:21 [I 2022-10-27 05:13:04] ...trap/engines/engine.py.229: process: 0.57315 | load: 0.00248 [I 2022-10-27 05:13:04] ...trap/engines/engine.py.230: loss: 0.15419 [I 2022-10-27 05:13:04] ...optimizers/trijoint.py.151: total_norm: 20.031765 [I 2022-10-27 05:13:09] ...trap/engines/engine.py.224: train: epoch 49 | batch 1640/2304 [I 2022-10-27 05:13:09] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:12 [I 2022-10-27 05:13:09] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00244 [I 2022-10-27 05:13:09] ...trap/engines/engine.py.230: loss: 0.15745 [I 2022-10-27 05:13:09] ...optimizers/trijoint.py.151: total_norm: 63.531071 [I 2022-10-27 05:13:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 1650/2304 [I 2022-10-27 05:13:15] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:05 [I 2022-10-27 05:13:15] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00244 [I 2022-10-27 05:13:15] ...trap/engines/engine.py.230: loss: 0.15284 [I 2022-10-27 05:13:15] ...optimizers/trijoint.py.151: total_norm: 17.194693 [I 2022-10-27 05:13:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 1660/2304 [I 2022-10-27 05:13:21] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:00 [I 2022-10-27 05:13:21] ...trap/engines/engine.py.229: process: 0.56182 | load: 0.00251 [I 2022-10-27 05:13:21] ...trap/engines/engine.py.230: loss: 0.18284 [I 2022-10-27 05:13:21] ...optimizers/trijoint.py.151: total_norm: 20.829342 [I 2022-10-27 05:13:26] ...trap/engines/engine.py.224: train: epoch 49 | batch 1670/2304 [I 2022-10-27 05:13:26] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:05:54 [I 2022-10-27 05:13:26] ...trap/engines/engine.py.229: process: 0.57493 | load: 0.00244 [I 2022-10-27 05:13:26] ...trap/engines/engine.py.230: loss: 0.11920 [I 2022-10-27 05:13:26] ...optimizers/trijoint.py.151: total_norm: 9.228819 [I 2022-10-27 05:13:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 1680/2304 [I 2022-10-27 05:13:32] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:05:50 [I 2022-10-27 05:13:32] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00243 [I 2022-10-27 05:13:32] ...trap/engines/engine.py.230: loss: 0.17919 [I 2022-10-27 05:13:32] ...optimizers/trijoint.py.151: total_norm: 17.768112 [I 2022-10-27 05:13:37] ...trap/engines/engine.py.224: train: epoch 49 | batch 1690/2304 [I 2022-10-27 05:13:37] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:05:46 [I 2022-10-27 05:13:37] ...trap/engines/engine.py.229: process: 0.57236 | load: 0.00252 [I 2022-10-27 05:13:37] ...trap/engines/engine.py.230: loss: 0.16129 [I 2022-10-27 05:13:37] ...optimizers/trijoint.py.151: total_norm: 15.821955 [I 2022-10-27 05:13:43] ...trap/engines/engine.py.224: train: epoch 49 | batch 1700/2304 [I 2022-10-27 05:13:43] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:38 [I 2022-10-27 05:13:43] ...trap/engines/engine.py.229: process: 0.56722 | load: 0.00249 [I 2022-10-27 05:13:43] ...trap/engines/engine.py.230: loss: 0.16408 [I 2022-10-27 05:13:43] ...optimizers/trijoint.py.151: total_norm: 15.005626 [I 2022-10-27 05:13:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 1710/2304 [I 2022-10-27 05:13:49] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:35 [I 2022-10-27 05:13:49] ...trap/engines/engine.py.229: process: 0.57289 | load: 0.00247 [I 2022-10-27 05:13:49] ...trap/engines/engine.py.230: loss: 0.14619 [I 2022-10-27 05:13:49] ...optimizers/trijoint.py.151: total_norm: 21.507854 [I 2022-10-27 05:13:54] ...trap/engines/engine.py.224: train: epoch 49 | batch 1720/2304 [I 2022-10-27 05:13:54] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:26 [I 2022-10-27 05:13:54] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00245 [I 2022-10-27 05:13:54] ...trap/engines/engine.py.230: loss: 0.22230 [I 2022-10-27 05:13:54] ...optimizers/trijoint.py.151: total_norm: 16.145285 [I 2022-10-27 05:14:00] ...trap/engines/engine.py.224: train: epoch 49 | batch 1730/2304 [I 2022-10-27 05:14:00] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:19 [I 2022-10-27 05:14:00] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00248 [I 2022-10-27 05:14:00] ...trap/engines/engine.py.230: loss: 0.13210 [I 2022-10-27 05:14:00] ...optimizers/trijoint.py.151: total_norm: 56.211094 [I 2022-10-27 05:14:05] ...trap/engines/engine.py.224: train: epoch 49 | batch 1740/2304 [I 2022-10-27 05:14:05] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:13 [I 2022-10-27 05:14:05] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00245 [I 2022-10-27 05:14:05] ...trap/engines/engine.py.230: loss: 0.24268 [I 2022-10-27 05:14:05] ...optimizers/trijoint.py.151: total_norm: 38.856789 [I 2022-10-27 05:14:11] ...trap/engines/engine.py.224: train: epoch 49 | batch 1750/2304 [I 2022-10-27 05:14:11] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:10 [I 2022-10-27 05:14:11] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00248 [I 2022-10-27 05:14:11] ...trap/engines/engine.py.230: loss: 0.17817 [I 2022-10-27 05:14:11] ...optimizers/trijoint.py.151: total_norm: 14.906913 [I 2022-10-27 05:14:17] ...trap/engines/engine.py.224: train: epoch 49 | batch 1760/2304 [I 2022-10-27 05:14:17] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:02 [I 2022-10-27 05:14:17] ...trap/engines/engine.py.229: process: 0.55637 | load: 0.00246 [I 2022-10-27 05:14:17] ...trap/engines/engine.py.230: loss: 0.14932 [I 2022-10-27 05:14:17] ...optimizers/trijoint.py.151: total_norm: 13.370972 [I 2022-10-27 05:14:22] ...trap/engines/engine.py.224: train: epoch 49 | batch 1770/2304 [I 2022-10-27 05:14:22] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:04:55 [I 2022-10-27 05:14:22] ...trap/engines/engine.py.229: process: 0.55395 | load: 0.00243 [I 2022-10-27 05:14:22] ...trap/engines/engine.py.230: loss: 0.19923 [I 2022-10-27 05:14:22] ...optimizers/trijoint.py.151: total_norm: 21.687511 [I 2022-10-27 05:14:28] ...trap/engines/engine.py.224: train: epoch 49 | batch 1780/2304 [I 2022-10-27 05:14:28] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:04:49 [I 2022-10-27 05:14:28] ...trap/engines/engine.py.229: process: 0.55534 | load: 0.00244 [I 2022-10-27 05:14:28] ...trap/engines/engine.py.230: loss: 0.14204 [I 2022-10-27 05:14:28] ...optimizers/trijoint.py.151: total_norm: 27.463779 [I 2022-10-27 05:14:33] ...trap/engines/engine.py.224: train: epoch 49 | batch 1790/2304 [I 2022-10-27 05:14:33] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:04:45 [I 2022-10-27 05:14:33] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00244 [I 2022-10-27 05:14:33] ...trap/engines/engine.py.230: loss: 0.15542 [I 2022-10-27 05:14:33] ...optimizers/trijoint.py.151: total_norm: 17.414082 [I 2022-10-27 05:14:39] ...trap/engines/engine.py.224: train: epoch 49 | batch 1800/2304 [I 2022-10-27 05:14:39] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:38 [I 2022-10-27 05:14:39] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00243 [I 2022-10-27 05:14:39] ...trap/engines/engine.py.230: loss: 0.17624 [I 2022-10-27 05:14:39] ...optimizers/trijoint.py.151: total_norm: 29.255638 [I 2022-10-27 05:14:44] ...trap/engines/engine.py.224: train: epoch 49 | batch 1810/2304 [I 2022-10-27 05:14:44] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:35 [I 2022-10-27 05:14:44] ...trap/engines/engine.py.229: process: 0.55614 | load: 0.00242 [I 2022-10-27 05:14:44] ...trap/engines/engine.py.230: loss: 0.30065 [I 2022-10-27 05:14:44] ...optimizers/trijoint.py.151: total_norm: 18.193052 [I 2022-10-27 05:14:50] ...trap/engines/engine.py.224: train: epoch 49 | batch 1820/2304 [I 2022-10-27 05:14:50] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:04:28 [I 2022-10-27 05:14:50] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00242 [I 2022-10-27 05:14:50] ...trap/engines/engine.py.230: loss: 0.14176 [I 2022-10-27 05:14:50] ...optimizers/trijoint.py.151: total_norm: 18.837955 [I 2022-10-27 05:14:56] ...trap/engines/engine.py.224: train: epoch 49 | batch 1830/2304 [I 2022-10-27 05:14:56] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:22 [I 2022-10-27 05:14:56] ...trap/engines/engine.py.229: process: 0.55068 | load: 0.00246 [I 2022-10-27 05:14:56] ...trap/engines/engine.py.230: loss: 0.18576 [I 2022-10-27 05:14:56] ...optimizers/trijoint.py.151: total_norm: 123.150032 [I 2022-10-27 05:15:01] ...trap/engines/engine.py.224: train: epoch 49 | batch 1840/2304 [I 2022-10-27 05:15:01] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:19 [I 2022-10-27 05:15:01] ...trap/engines/engine.py.229: process: 0.56339 | load: 0.00262 [I 2022-10-27 05:15:01] ...trap/engines/engine.py.230: loss: 0.15829 [I 2022-10-27 05:15:01] ...optimizers/trijoint.py.151: total_norm: 17.164316 [I 2022-10-27 05:15:07] ...trap/engines/engine.py.224: train: epoch 49 | batch 1850/2304 [I 2022-10-27 05:15:07] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:10 [I 2022-10-27 05:15:07] ...trap/engines/engine.py.229: process: 0.54567 | load: 0.00239 [I 2022-10-27 05:15:07] ...trap/engines/engine.py.230: loss: 0.24862 [I 2022-10-27 05:15:07] ...optimizers/trijoint.py.151: total_norm: 33.534115 [I 2022-10-27 05:15:12] ...trap/engines/engine.py.224: train: epoch 49 | batch 1860/2304 [I 2022-10-27 05:15:12] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:04 [I 2022-10-27 05:15:12] ...trap/engines/engine.py.229: process: 0.54693 | load: 0.00248 [I 2022-10-27 05:15:12] ...trap/engines/engine.py.230: loss: 0.17867 [I 2022-10-27 05:15:12] ...optimizers/trijoint.py.151: total_norm: 28.745878 [I 2022-10-27 05:15:18] ...trap/engines/engine.py.224: train: epoch 49 | batch 1870/2304 [I 2022-10-27 05:15:18] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:02 [I 2022-10-27 05:15:18] ...trap/engines/engine.py.229: process: 0.55387 | load: 0.00245 [I 2022-10-27 05:15:18] ...trap/engines/engine.py.230: loss: 0.13061 [I 2022-10-27 05:15:18] ...optimizers/trijoint.py.151: total_norm: 12.422631 [I 2022-10-27 05:15:23] ...trap/engines/engine.py.224: train: epoch 49 | batch 1880/2304 [I 2022-10-27 05:15:23] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:03:55 [I 2022-10-27 05:15:23] ...trap/engines/engine.py.229: process: 0.55372 | load: 0.00242 [I 2022-10-27 05:15:23] ...trap/engines/engine.py.230: loss: 0.18196 [I 2022-10-27 05:15:23] ...optimizers/trijoint.py.151: total_norm: 11.618311 [I 2022-10-27 05:15:29] ...trap/engines/engine.py.224: train: epoch 49 | batch 1890/2304 [I 2022-10-27 05:15:29] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:03:49 [I 2022-10-27 05:15:29] ...trap/engines/engine.py.229: process: 0.54769 | load: 0.00245 [I 2022-10-27 05:15:29] ...trap/engines/engine.py.230: loss: 0.18055 [I 2022-10-27 05:15:29] ...optimizers/trijoint.py.151: total_norm: 18.955816 [I 2022-10-27 05:15:35] ...trap/engines/engine.py.224: train: epoch 49 | batch 1900/2304 [I 2022-10-27 05:15:35] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:03:44 [I 2022-10-27 05:15:35] ...trap/engines/engine.py.229: process: 0.54968 | load: 0.00243 [I 2022-10-27 05:15:35] ...trap/engines/engine.py.230: loss: 0.21459 [I 2022-10-27 05:15:35] ...optimizers/trijoint.py.151: total_norm: 17.286337 [I 2022-10-27 05:15:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 1910/2304 [I 2022-10-27 05:15:40] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:39 [I 2022-10-27 05:15:40] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00242 [I 2022-10-27 05:15:40] ...trap/engines/engine.py.230: loss: 0.17060 [I 2022-10-27 05:15:40] ...optimizers/trijoint.py.151: total_norm: 17.932539 [I 2022-10-27 05:15:46] ...trap/engines/engine.py.224: train: epoch 49 | batch 1920/2304 [I 2022-10-27 05:15:46] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:33 [I 2022-10-27 05:15:46] ...trap/engines/engine.py.229: process: 0.55822 | load: 0.00242 [I 2022-10-27 05:15:46] ...trap/engines/engine.py.230: loss: 0.13419 [I 2022-10-27 05:15:46] ...optimizers/trijoint.py.151: total_norm: 9.215398 [I 2022-10-27 05:15:51] ...trap/engines/engine.py.224: train: epoch 49 | batch 1930/2304 [I 2022-10-27 05:15:51] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:26 [I 2022-10-27 05:15:51] ...trap/engines/engine.py.229: process: 0.54363 | load: 0.00242 [I 2022-10-27 05:15:51] ...trap/engines/engine.py.230: loss: 0.13125 [I 2022-10-27 05:15:51] ...optimizers/trijoint.py.151: total_norm: 15.582932 [I 2022-10-27 05:15:57] ...trap/engines/engine.py.224: train: epoch 49 | batch 1940/2304 [I 2022-10-27 05:15:57] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:03:23 [I 2022-10-27 05:15:57] ...trap/engines/engine.py.229: process: 0.54830 | load: 0.00244 [I 2022-10-27 05:15:57] ...trap/engines/engine.py.230: loss: 0.13866 [I 2022-10-27 05:15:57] ...optimizers/trijoint.py.151: total_norm: 16.459063 [I 2022-10-27 05:16:03] ...trap/engines/engine.py.224: train: epoch 49 | batch 1950/2304 [I 2022-10-27 05:16:03] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:25 [I 2022-10-27 05:16:03] ...trap/engines/engine.py.229: process: 0.68609 | load: 0.00240 [I 2022-10-27 05:16:03] ...trap/engines/engine.py.230: loss: 0.22576 [I 2022-10-27 05:16:03] ...optimizers/trijoint.py.151: total_norm: 21.502724 [I 2022-10-27 05:16:08] ...trap/engines/engine.py.224: train: epoch 49 | batch 1960/2304 [I 2022-10-27 05:16:08] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:13 [I 2022-10-27 05:16:08] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00246 [I 2022-10-27 05:16:08] ...trap/engines/engine.py.230: loss: 0.23397 [I 2022-10-27 05:16:08] ...optimizers/trijoint.py.151: total_norm: 28.381449 [I 2022-10-27 05:16:14] ...trap/engines/engine.py.224: train: epoch 49 | batch 1970/2304 [I 2022-10-27 05:16:14] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:05 [I 2022-10-27 05:16:14] ...trap/engines/engine.py.229: process: 0.54757 | load: 0.00243 [I 2022-10-27 05:16:14] ...trap/engines/engine.py.230: loss: 0.12445 [I 2022-10-27 05:16:14] ...optimizers/trijoint.py.151: total_norm: 28.335068 [I 2022-10-27 05:16:19] ...trap/engines/engine.py.224: train: epoch 49 | batch 1980/2304 [I 2022-10-27 05:16:19] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:02:59 [I 2022-10-27 05:16:19] ...trap/engines/engine.py.229: process: 0.54988 | load: 0.00248 [I 2022-10-27 05:16:19] ...trap/engines/engine.py.230: loss: 0.20766 [I 2022-10-27 05:16:19] ...optimizers/trijoint.py.151: total_norm: 20.845760 [I 2022-10-27 05:16:25] ...trap/engines/engine.py.224: train: epoch 49 | batch 1990/2304 [I 2022-10-27 05:16:25] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:02:52 [I 2022-10-27 05:16:25] ...trap/engines/engine.py.229: process: 0.54549 | load: 0.00244 [I 2022-10-27 05:16:25] ...trap/engines/engine.py.230: loss: 0.21714 [I 2022-10-27 05:16:25] ...optimizers/trijoint.py.151: total_norm: 20.326908 [I 2022-10-27 05:16:31] ...trap/engines/engine.py.224: train: epoch 49 | batch 2000/2304 [I 2022-10-27 05:16:31] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:02:47 [I 2022-10-27 05:16:31] ...trap/engines/engine.py.229: process: 0.54773 | load: 0.00244 [I 2022-10-27 05:16:31] ...trap/engines/engine.py.230: loss: 0.11431 [I 2022-10-27 05:16:31] ...optimizers/trijoint.py.151: total_norm: 13.089014 [I 2022-10-27 05:16:36] ...trap/engines/engine.py.224: train: epoch 49 | batch 2010/2304 [I 2022-10-27 05:16:36] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:02:43 [I 2022-10-27 05:16:36] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00243 [I 2022-10-27 05:16:36] ...trap/engines/engine.py.230: loss: 0.15882 [I 2022-10-27 05:16:36] ...optimizers/trijoint.py.151: total_norm: 18.185457 [I 2022-10-27 05:16:42] ...trap/engines/engine.py.224: train: epoch 49 | batch 2020/2304 [I 2022-10-27 05:16:42] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:41 [I 2022-10-27 05:16:42] ...trap/engines/engine.py.229: process: 0.61445 | load: 0.00245 [I 2022-10-27 05:16:42] ...trap/engines/engine.py.230: loss: 0.10332 [I 2022-10-27 05:16:42] ...optimizers/trijoint.py.151: total_norm: 8.074180 [I 2022-10-27 05:16:47] ...trap/engines/engine.py.224: train: epoch 49 | batch 2030/2304 [I 2022-10-27 05:16:47] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:32 [I 2022-10-27 05:16:47] ...trap/engines/engine.py.229: process: 0.54961 | load: 0.00243 [I 2022-10-27 05:16:47] ...trap/engines/engine.py.230: loss: 0.12494 [I 2022-10-27 05:16:47] ...optimizers/trijoint.py.151: total_norm: 7.204905 [I 2022-10-27 05:16:53] ...trap/engines/engine.py.224: train: epoch 49 | batch 2040/2304 [I 2022-10-27 05:16:53] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:02:25 [I 2022-10-27 05:16:53] ...trap/engines/engine.py.229: process: 0.54826 | load: 0.00240 [I 2022-10-27 05:16:53] ...trap/engines/engine.py.230: loss: 0.12126 [I 2022-10-27 05:16:53] ...optimizers/trijoint.py.151: total_norm: 13.656651 [I 2022-10-27 05:16:58] ...trap/engines/engine.py.224: train: epoch 49 | batch 2050/2304 [I 2022-10-27 05:16:58] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:20 [I 2022-10-27 05:16:58] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00259 [I 2022-10-27 05:16:58] ...trap/engines/engine.py.230: loss: 0.22145 [I 2022-10-27 05:16:58] ...optimizers/trijoint.py.151: total_norm: 19.592400 [I 2022-10-27 05:17:04] ...trap/engines/engine.py.224: train: epoch 49 | batch 2060/2304 [I 2022-10-27 05:17:04] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:15 [I 2022-10-27 05:17:04] ...trap/engines/engine.py.229: process: 0.55169 | load: 0.00244 [I 2022-10-27 05:17:04] ...trap/engines/engine.py.230: loss: 0.10929 [I 2022-10-27 05:17:04] ...optimizers/trijoint.py.151: total_norm: 11.717068 [I 2022-10-27 05:17:10] ...trap/engines/engine.py.224: train: epoch 49 | batch 2070/2304 [I 2022-10-27 05:17:10] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:09 [I 2022-10-27 05:17:10] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00258 [I 2022-10-27 05:17:10] ...trap/engines/engine.py.230: loss: 0.15581 [I 2022-10-27 05:17:10] ...optimizers/trijoint.py.151: total_norm: 9.980812 [I 2022-10-27 05:17:15] ...trap/engines/engine.py.224: train: epoch 49 | batch 2080/2304 [I 2022-10-27 05:17:15] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:04 [I 2022-10-27 05:17:15] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00258 [I 2022-10-27 05:17:15] ...trap/engines/engine.py.230: loss: 0.13801 [I 2022-10-27 05:17:15] ...optimizers/trijoint.py.151: total_norm: 11.718925 [I 2022-10-27 05:17:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 2090/2304 [I 2022-10-27 05:17:21] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:00 [I 2022-10-27 05:17:21] ...trap/engines/engine.py.229: process: 0.54158 | load: 0.00241 [I 2022-10-27 05:17:21] ...trap/engines/engine.py.230: loss: 0.16116 [I 2022-10-27 05:17:21] ...optimizers/trijoint.py.151: total_norm: 8.943064 [I 2022-10-27 05:17:26] ...trap/engines/engine.py.224: train: epoch 49 | batch 2100/2304 [I 2022-10-27 05:17:26] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:01:53 [I 2022-10-27 05:17:26] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00241 [I 2022-10-27 05:17:26] ...trap/engines/engine.py.230: loss: 0.15140 [I 2022-10-27 05:17:26] ...optimizers/trijoint.py.151: total_norm: 22.191055 [I 2022-10-27 05:17:32] ...trap/engines/engine.py.224: train: epoch 49 | batch 2110/2304 [I 2022-10-27 05:17:32] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:47 [I 2022-10-27 05:17:32] ...trap/engines/engine.py.229: process: 0.55161 | load: 0.00239 [I 2022-10-27 05:17:32] ...trap/engines/engine.py.230: loss: 0.17047 [I 2022-10-27 05:17:32] ...optimizers/trijoint.py.151: total_norm: 11.073457 [I 2022-10-27 05:17:37] ...trap/engines/engine.py.224: train: epoch 49 | batch 2120/2304 [I 2022-10-27 05:17:37] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:41 [I 2022-10-27 05:17:37] ...trap/engines/engine.py.229: process: 0.54714 | load: 0.00240 [I 2022-10-27 05:17:37] ...trap/engines/engine.py.230: loss: 0.17810 [I 2022-10-27 05:17:37] ...optimizers/trijoint.py.151: total_norm: 8.969058 [I 2022-10-27 05:17:43] ...trap/engines/engine.py.224: train: epoch 49 | batch 2130/2304 [I 2022-10-27 05:17:43] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:37 [I 2022-10-27 05:17:43] ...trap/engines/engine.py.229: process: 0.56832 | load: 0.00241 [I 2022-10-27 05:17:43] ...trap/engines/engine.py.230: loss: 0.17113 [I 2022-10-27 05:17:43] ...optimizers/trijoint.py.151: total_norm: 20.464182 [I 2022-10-27 05:17:49] ...trap/engines/engine.py.224: train: epoch 49 | batch 2140/2304 [I 2022-10-27 05:17:49] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:31 [I 2022-10-27 05:17:49] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00241 [I 2022-10-27 05:17:49] ...trap/engines/engine.py.230: loss: 0.19702 [I 2022-10-27 05:17:49] ...optimizers/trijoint.py.151: total_norm: 57.722988 [I 2022-10-27 05:17:54] ...trap/engines/engine.py.224: train: epoch 49 | batch 2150/2304 [I 2022-10-27 05:17:54] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:25 [I 2022-10-27 05:17:54] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00243 [I 2022-10-27 05:17:54] ...trap/engines/engine.py.230: loss: 0.25772 [I 2022-10-27 05:17:54] ...optimizers/trijoint.py.151: total_norm: 24.976830 [I 2022-10-27 05:18:00] ...trap/engines/engine.py.224: train: epoch 49 | batch 2160/2304 [I 2022-10-27 05:18:00] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:20 [I 2022-10-27 05:18:00] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00243 [I 2022-10-27 05:18:00] ...trap/engines/engine.py.230: loss: 0.20842 [I 2022-10-27 05:18:00] ...optimizers/trijoint.py.151: total_norm: 16.047642 [I 2022-10-27 05:18:05] ...trap/engines/engine.py.224: train: epoch 49 | batch 2170/2304 [I 2022-10-27 05:18:05] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:14 [I 2022-10-27 05:18:05] ...trap/engines/engine.py.229: process: 0.56399 | load: 0.00254 [I 2022-10-27 05:18:05] ...trap/engines/engine.py.230: loss: 0.13759 [I 2022-10-27 05:18:05] ...optimizers/trijoint.py.151: total_norm: 12.588633 [I 2022-10-27 05:18:11] ...trap/engines/engine.py.224: train: epoch 49 | batch 2180/2304 [I 2022-10-27 05:18:11] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:09 [I 2022-10-27 05:18:11] ...trap/engines/engine.py.229: process: 0.56455 | load: 0.00247 [I 2022-10-27 05:18:11] ...trap/engines/engine.py.230: loss: 0.20347 [I 2022-10-27 05:18:11] ...optimizers/trijoint.py.151: total_norm: 20.201910 [I 2022-10-27 05:18:17] ...trap/engines/engine.py.224: train: epoch 49 | batch 2190/2304 [I 2022-10-27 05:18:17] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:03 [I 2022-10-27 05:18:17] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00245 [I 2022-10-27 05:18:17] ...trap/engines/engine.py.230: loss: 0.23051 [I 2022-10-27 05:18:17] ...optimizers/trijoint.py.151: total_norm: 15.814366 [I 2022-10-27 05:18:22] ...trap/engines/engine.py.224: train: epoch 49 | batch 2200/2304 [I 2022-10-27 05:18:22] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:00:58 [I 2022-10-27 05:18:22] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00243 [I 2022-10-27 05:18:22] ...trap/engines/engine.py.230: loss: 0.16365 [I 2022-10-27 05:18:22] ...optimizers/trijoint.py.151: total_norm: 14.612318 [I 2022-10-27 05:18:28] ...trap/engines/engine.py.224: train: epoch 49 | batch 2210/2304 [I 2022-10-27 05:18:28] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:00:52 [I 2022-10-27 05:18:28] ...trap/engines/engine.py.229: process: 0.56238 | load: 0.00248 [I 2022-10-27 05:18:28] ...trap/engines/engine.py.230: loss: 0.15817 [I 2022-10-27 05:18:28] ...optimizers/trijoint.py.151: total_norm: 16.201498 [I 2022-10-27 05:18:33] ...trap/engines/engine.py.224: train: epoch 49 | batch 2220/2304 [I 2022-10-27 05:18:33] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:00:46 [I 2022-10-27 05:18:33] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00238 [I 2022-10-27 05:18:33] ...trap/engines/engine.py.230: loss: 0.17911 [I 2022-10-27 05:18:33] ...optimizers/trijoint.py.151: total_norm: 14.000566 [I 2022-10-27 05:18:39] ...trap/engines/engine.py.224: train: epoch 49 | batch 2230/2304 [I 2022-10-27 05:18:39] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:41 [I 2022-10-27 05:18:39] ...trap/engines/engine.py.229: process: 0.54448 | load: 0.00242 [I 2022-10-27 05:18:39] ...trap/engines/engine.py.230: loss: 0.14331 [I 2022-10-27 05:18:39] ...optimizers/trijoint.py.151: total_norm: 14.247681 [I 2022-10-27 05:18:45] ...trap/engines/engine.py.224: train: epoch 49 | batch 2240/2304 [I 2022-10-27 05:18:45] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:35 [I 2022-10-27 05:18:45] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00242 [I 2022-10-27 05:18:45] ...trap/engines/engine.py.230: loss: 0.14213 [I 2022-10-27 05:18:45] ...optimizers/trijoint.py.151: total_norm: 9.980879 [I 2022-10-27 05:18:50] ...trap/engines/engine.py.224: train: epoch 49 | batch 2250/2304 [I 2022-10-27 05:18:50] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:30 [I 2022-10-27 05:18:50] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00244 [I 2022-10-27 05:18:50] ...trap/engines/engine.py.230: loss: 0.20043 [I 2022-10-27 05:18:50] ...optimizers/trijoint.py.151: total_norm: 17.340614 [I 2022-10-27 05:19:16] ...trap/engines/engine.py.224: train: epoch 49 | batch 2260/2304 [I 2022-10-27 05:19:16] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:01:01 [I 2022-10-27 05:19:16] ...trap/engines/engine.py.229: process: 0.55809 | load: 0.00238 [I 2022-10-27 05:19:16] ...trap/engines/engine.py.230: loss: 0.20512 [I 2022-10-27 05:19:16] ...optimizers/trijoint.py.151: total_norm: 24.208281 [I 2022-10-27 05:19:21] ...trap/engines/engine.py.224: train: epoch 49 | batch 2270/2304 [I 2022-10-27 05:19:21] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:21 [I 2022-10-27 05:19:21] ...trap/engines/engine.py.229: process: 0.54771 | load: 0.00238 [I 2022-10-27 05:19:21] ...trap/engines/engine.py.230: loss: 0.14831 [I 2022-10-27 05:19:21] ...optimizers/trijoint.py.151: total_norm: 36.236847 [I 2022-10-27 05:19:27] ...trap/engines/engine.py.224: train: epoch 49 | batch 2280/2304 [I 2022-10-27 05:19:27] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:14 [I 2022-10-27 05:19:27] ...trap/engines/engine.py.229: process: 0.59533 | load: 0.00831 [I 2022-10-27 05:19:27] ...trap/engines/engine.py.230: loss: 0.14624 [I 2022-10-27 05:19:27] ...optimizers/trijoint.py.151: total_norm: 15.335945 [I 2022-10-27 05:19:33] ...trap/engines/engine.py.224: train: epoch 49 | batch 2290/2304 [I 2022-10-27 05:19:33] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:07 [I 2022-10-27 05:19:33] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00238 [I 2022-10-27 05:19:33] ...trap/engines/engine.py.230: loss: 0.21746 [I 2022-10-27 05:19:33] ...optimizers/trijoint.py.151: total_norm: 19.480808 [I 2022-10-27 05:19:38] ...trap/engines/engine.py.224: train: epoch 49 | batch 2300/2304 [I 2022-10-27 05:19:38] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:02 [I 2022-10-27 05:19:38] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00014 [I 2022-10-27 05:19:38] ...trap/engines/engine.py.230: loss: 0.13896 [I 2022-10-27 05:19:38] ...optimizers/trijoint.py.151: total_norm: 12.432261 [I 2022-10-27 05:19:40] ...trap/engines/engine.py.224: train: epoch 49 | batch 2304/2304 [I 2022-10-27 05:19:40] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:00:00 [I 2022-10-27 05:19:40] ...trap/engines/engine.py.229: process: 0.54764 | load: 0.00014 [I 2022-10-27 05:19:40] ...trap/engines/engine.py.230: loss: 0.16867 [I 2022-10-27 05:19:40] ...optimizers/trijoint.py.151: total_norm: 16.231852 [S 2022-10-27 05:19:41] ...trap/engines/engine.py.129: train_epoch.epoch: 49 [S 2022-10-27 05:19:41] ...trap/engines/engine.py.129: train_epoch.loss: 0.16712351782073684 [I 2022-10-27 05:19:41] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 05:19:42] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 49 [I 2022-10-27 05:19:42] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 05:19:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 05:19:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 05:19:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 05:19:46] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 05:19:46] ...trap/engines/engine.py.321: eval: epoch 49 | batch 0/510 [I 2022-10-27 05:19:46] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:34:21 [I 2022-10-27 05:19:46] ...trap/engines/engine.py.326: process: 4.04232 | load: 3.63704 [I 2022-10-27 05:19:50] ...trap/engines/engine.py.321: eval: epoch 49 | batch 10/510 [I 2022-10-27 05:19:50] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:08 [I 2022-10-27 05:19:50] ...trap/engines/engine.py.326: process: 0.32764 | load: 0.00019 [I 2022-10-27 05:19:53] ...trap/engines/engine.py.321: eval: epoch 49 | batch 20/510 [I 2022-10-27 05:19:53] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:03 [I 2022-10-27 05:19:53] ...trap/engines/engine.py.326: process: 0.33543 | load: 0.00018 [I 2022-10-27 05:19:56] ...trap/engines/engine.py.321: eval: epoch 49 | batch 30/510 [I 2022-10-27 05:19:56] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:41 [I 2022-10-27 05:19:56] ...trap/engines/engine.py.326: process: 0.32348 | load: 0.00017 [I 2022-10-27 05:20:00] ...trap/engines/engine.py.321: eval: epoch 49 | batch 40/510 [I 2022-10-27 05:20:00] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:36 [I 2022-10-27 05:20:00] ...trap/engines/engine.py.326: process: 0.32786 | load: 0.00018 [I 2022-10-27 05:20:03] ...trap/engines/engine.py.321: eval: epoch 49 | batch 50/510 [I 2022-10-27 05:20:03] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:31 [I 2022-10-27 05:20:03] ...trap/engines/engine.py.326: process: 0.32201 | load: 0.00017 [I 2022-10-27 05:20:06] ...trap/engines/engine.py.321: eval: epoch 49 | batch 60/510 [I 2022-10-27 05:20:06] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:30 [I 2022-10-27 05:20:06] ...trap/engines/engine.py.326: process: 0.35862 | load: 0.00036 [I 2022-10-27 05:20:10] ...trap/engines/engine.py.321: eval: epoch 49 | batch 70/510 [I 2022-10-27 05:20:10] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:26 [I 2022-10-27 05:20:10] ...trap/engines/engine.py.326: process: 0.34762 | load: 0.00021 [I 2022-10-27 05:20:13] ...trap/engines/engine.py.321: eval: epoch 49 | batch 80/510 [I 2022-10-27 05:20:13] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:24 [I 2022-10-27 05:20:13] ...trap/engines/engine.py.326: process: 0.33005 | load: 0.00019 [I 2022-10-27 05:20:16] ...trap/engines/engine.py.321: eval: epoch 49 | batch 90/510 [I 2022-10-27 05:20:16] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:22 [I 2022-10-27 05:20:16] ...trap/engines/engine.py.326: process: 0.37148 | load: 0.00018 [I 2022-10-27 05:20:20] ...trap/engines/engine.py.321: eval: epoch 49 | batch 100/510 [I 2022-10-27 05:20:20] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:14 [I 2022-10-27 05:20:20] ...trap/engines/engine.py.326: process: 0.29054 | load: 0.00016 [I 2022-10-27 05:20:22] ...trap/engines/engine.py.321: eval: epoch 49 | batch 110/510 [I 2022-10-27 05:20:22] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:58 [I 2022-10-27 05:20:22] ...trap/engines/engine.py.326: process: 0.28889 | load: 0.00015 [I 2022-10-27 05:20:25] ...trap/engines/engine.py.321: eval: epoch 49 | batch 120/510 [I 2022-10-27 05:20:25] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:54 [I 2022-10-27 05:20:25] ...trap/engines/engine.py.326: process: 0.29078 | load: 0.00016 [I 2022-10-27 05:20:28] ...trap/engines/engine.py.321: eval: epoch 49 | batch 130/510 [I 2022-10-27 05:20:28] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:51 [I 2022-10-27 05:20:28] ...trap/engines/engine.py.326: process: 0.28855 | load: 0.00025 [I 2022-10-27 05:20:31] ...trap/engines/engine.py.321: eval: epoch 49 | batch 140/510 [I 2022-10-27 05:20:31] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-27 05:20:31] ...trap/engines/engine.py.326: process: 0.28841 | load: 0.00015 [I 2022-10-27 05:20:34] ...trap/engines/engine.py.321: eval: epoch 49 | batch 150/510 [I 2022-10-27 05:20:34] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-27 05:20:34] ...trap/engines/engine.py.326: process: 0.28582 | load: 0.00014 [I 2022-10-27 05:20:37] ...trap/engines/engine.py.321: eval: epoch 49 | batch 160/510 [I 2022-10-27 05:20:37] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-27 05:20:37] ...trap/engines/engine.py.326: process: 0.29192 | load: 0.00015 [I 2022-10-27 05:20:40] ...trap/engines/engine.py.321: eval: epoch 49 | batch 170/510 [I 2022-10-27 05:20:40] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-27 05:20:40] ...trap/engines/engine.py.326: process: 0.28508 | load: 0.00015 [I 2022-10-27 05:20:43] ...trap/engines/engine.py.321: eval: epoch 49 | batch 180/510 [I 2022-10-27 05:20:43] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 05:20:43] ...trap/engines/engine.py.326: process: 0.28692 | load: 0.00018 [I 2022-10-27 05:20:46] ...trap/engines/engine.py.321: eval: epoch 49 | batch 190/510 [I 2022-10-27 05:20:46] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 05:20:46] ...trap/engines/engine.py.326: process: 0.28945 | load: 0.00018 [I 2022-10-27 05:20:49] ...trap/engines/engine.py.321: eval: epoch 49 | batch 200/510 [I 2022-10-27 05:20:49] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 05:20:49] ...trap/engines/engine.py.326: process: 0.29092 | load: 0.00019 [I 2022-10-27 05:20:51] ...trap/engines/engine.py.321: eval: epoch 49 | batch 210/510 [I 2022-10-27 05:20:51] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 05:20:51] ...trap/engines/engine.py.326: process: 0.29278 | load: 0.00015 [I 2022-10-27 05:20:54] ...trap/engines/engine.py.321: eval: epoch 49 | batch 220/510 [I 2022-10-27 05:20:54] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 05:20:54] ...trap/engines/engine.py.326: process: 0.28599 | load: 0.00020 [I 2022-10-27 05:20:57] ...trap/engines/engine.py.321: eval: epoch 49 | batch 230/510 [I 2022-10-27 05:20:57] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-27 05:20:57] ...trap/engines/engine.py.326: process: 0.28676 | load: 0.00015 [I 2022-10-27 05:21:00] ...trap/engines/engine.py.321: eval: epoch 49 | batch 240/510 [I 2022-10-27 05:21:00] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-27 05:21:00] ...trap/engines/engine.py.326: process: 0.28935 | load: 0.00015 [I 2022-10-27 05:21:03] ...trap/engines/engine.py.321: eval: epoch 49 | batch 250/510 [I 2022-10-27 05:21:03] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-27 05:21:03] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00018 [I 2022-10-27 05:21:06] ...trap/engines/engine.py.321: eval: epoch 49 | batch 260/510 [I 2022-10-27 05:21:06] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 05:21:06] ...trap/engines/engine.py.326: process: 0.29102 | load: 0.00015 [I 2022-10-27 05:21:09] ...trap/engines/engine.py.321: eval: epoch 49 | batch 270/510 [I 2022-10-27 05:21:09] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 05:21:09] ...trap/engines/engine.py.326: process: 0.29175 | load: 0.00015 [I 2022-10-27 05:21:12] ...trap/engines/engine.py.321: eval: epoch 49 | batch 280/510 [I 2022-10-27 05:21:12] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 05:21:12] ...trap/engines/engine.py.326: process: 0.28812 | load: 0.00014 [I 2022-10-27 05:21:14] ...trap/engines/engine.py.321: eval: epoch 49 | batch 290/510 [I 2022-10-27 05:21:14] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:02 [I 2022-10-27 05:21:14] ...trap/engines/engine.py.326: process: 0.27315 | load: 0.00015 [I 2022-10-27 05:21:17] ...trap/engines/engine.py.321: eval: epoch 49 | batch 300/510 [I 2022-10-27 05:21:17] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:00:59 [I 2022-10-27 05:21:17] ...trap/engines/engine.py.326: process: 0.28083 | load: 0.00014 [I 2022-10-27 05:21:20] ...trap/engines/engine.py.321: eval: epoch 49 | batch 310/510 [I 2022-10-27 05:21:20] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 05:21:20] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00015 [I 2022-10-27 05:21:23] ...trap/engines/engine.py.321: eval: epoch 49 | batch 320/510 [I 2022-10-27 05:21:23] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 05:21:23] ...trap/engines/engine.py.326: process: 0.28723 | load: 0.00015 [I 2022-10-27 05:21:26] ...trap/engines/engine.py.321: eval: epoch 49 | batch 330/510 [I 2022-10-27 05:21:26] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 05:21:26] ...trap/engines/engine.py.326: process: 0.28350 | load: 0.00015 [I 2022-10-27 05:21:29] ...trap/engines/engine.py.321: eval: epoch 49 | batch 340/510 [I 2022-10-27 05:21:29] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-27 05:21:29] ...trap/engines/engine.py.326: process: 0.28788 | load: 0.00014 [I 2022-10-27 05:21:32] ...trap/engines/engine.py.321: eval: epoch 49 | batch 350/510 [I 2022-10-27 05:21:32] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-27 05:21:32] ...trap/engines/engine.py.326: process: 0.29000 | load: 0.00015 [I 2022-10-27 05:21:35] ...trap/engines/engine.py.321: eval: epoch 49 | batch 360/510 [I 2022-10-27 05:21:35] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 05:21:35] ...trap/engines/engine.py.326: process: 0.28354 | load: 0.00015 [I 2022-10-27 05:21:37] ...trap/engines/engine.py.321: eval: epoch 49 | batch 370/510 [I 2022-10-27 05:21:37] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 05:21:37] ...trap/engines/engine.py.326: process: 0.29223 | load: 0.00015 [I 2022-10-27 05:21:40] ...trap/engines/engine.py.321: eval: epoch 49 | batch 380/510 [I 2022-10-27 05:21:40] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 05:21:40] ...trap/engines/engine.py.326: process: 0.28306 | load: 0.00015 [I 2022-10-27 05:21:43] ...trap/engines/engine.py.321: eval: epoch 49 | batch 390/510 [I 2022-10-27 05:21:43] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 05:21:43] ...trap/engines/engine.py.326: process: 0.28963 | load: 0.00015 [I 2022-10-27 05:21:46] ...trap/engines/engine.py.321: eval: epoch 49 | batch 400/510 [I 2022-10-27 05:21:46] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-27 05:21:46] ...trap/engines/engine.py.326: process: 0.29012 | load: 0.00015 [I 2022-10-27 05:21:49] ...trap/engines/engine.py.321: eval: epoch 49 | batch 410/510 [I 2022-10-27 05:21:49] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-27 05:21:49] ...trap/engines/engine.py.326: process: 0.28584 | load: 0.00015 [I 2022-10-27 05:21:52] ...trap/engines/engine.py.321: eval: epoch 49 | batch 420/510 [I 2022-10-27 05:21:52] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-27 05:21:52] ...trap/engines/engine.py.326: process: 0.28154 | load: 0.00021 [I 2022-10-27 05:21:55] ...trap/engines/engine.py.321: eval: epoch 49 | batch 430/510 [I 2022-10-27 05:21:55] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:22 [I 2022-10-27 05:21:55] ...trap/engines/engine.py.326: process: 0.28388 | load: 0.00023 [I 2022-10-27 05:21:58] ...trap/engines/engine.py.321: eval: epoch 49 | batch 440/510 [I 2022-10-27 05:21:58] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 05:21:58] ...trap/engines/engine.py.326: process: 0.28326 | load: 0.00020 [I 2022-10-27 05:22:00] ...trap/engines/engine.py.321: eval: epoch 49 | batch 450/510 [I 2022-10-27 05:22:00] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 05:22:00] ...trap/engines/engine.py.326: process: 0.29251 | load: 0.00016 [I 2022-10-27 05:22:03] ...trap/engines/engine.py.321: eval: epoch 49 | batch 460/510 [I 2022-10-27 05:22:03] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 05:22:03] ...trap/engines/engine.py.326: process: 0.28330 | load: 0.00020 [I 2022-10-27 05:22:06] ...trap/engines/engine.py.321: eval: epoch 49 | batch 470/510 [I 2022-10-27 05:22:06] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 05:22:06] ...trap/engines/engine.py.326: process: 0.29088 | load: 0.00017 [I 2022-10-27 05:22:09] ...trap/engines/engine.py.321: eval: epoch 49 | batch 480/510 [I 2022-10-27 05:22:09] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 05:22:09] ...trap/engines/engine.py.326: process: 0.28736 | load: 0.00020 [I 2022-10-27 05:22:12] ...trap/engines/engine.py.321: eval: epoch 49 | batch 490/510 [I 2022-10-27 05:22:12] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 05:22:12] ...trap/engines/engine.py.326: process: 0.28638 | load: 0.00014 [I 2022-10-27 05:22:15] ...trap/engines/engine.py.321: eval: epoch 49 | batch 500/510 [I 2022-10-27 05:22:15] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 05:22:15] ...trap/engines/engine.py.326: process: 0.28610 | load: 0.00014 [I 2022-10-27 05:22:18] ...trap/engines/engine.py.321: eval: epoch 49 | batch 510/510 [I 2022-10-27 05:22:18] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 05:22:18] ...trap/engines/engine.py.326: process: 0.28242 | load: 0.00011 [S 2022-10-27 05:22:19] ...trap/engines/engine.py.132: eval_epoch.epoch: 49 [I 2022-10-27 05:22:19] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.697700 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.898500 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934800 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.690100 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.895200 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.930600 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015659 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010298 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009250 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016459 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009897 [S 2022-10-27 05:22:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006119 [I 2022-10-27 05:22:24] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 05:22:24] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 05:22:24] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 05:22:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 05:22:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 05:22:27] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 05:22:28] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 05:22:34] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 05:22:34] ...trap/engines/engine.py.163: Training model on trainset for epoch 50 [I 2022-10-27 05:22:34] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 05:22:37] ...trap/engines/engine.py.224: train: epoch 50 | batch 0/2304 [I 2022-10-27 05:22:37] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:53 [I 2022-10-27 05:22:37] ...trap/engines/engine.py.229: process: 2.80966 | load: 2.15918 [I 2022-10-27 05:22:37] ...trap/engines/engine.py.230: loss: 0.18404 [I 2022-10-27 05:22:37] ...optimizers/trijoint.py.151: total_norm: 14.864899 [I 2022-10-27 05:22:43] ...trap/engines/engine.py.224: train: epoch 50 | batch 10/2304 [I 2022-10-27 05:22:43] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:42 [I 2022-10-27 05:22:43] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00316 [I 2022-10-27 05:22:43] ...trap/engines/engine.py.230: loss: 0.14133 [I 2022-10-27 05:22:43] ...optimizers/trijoint.py.151: total_norm: 12.603571 [I 2022-10-27 05:22:48] ...trap/engines/engine.py.224: train: epoch 50 | batch 20/2304 [I 2022-10-27 05:22:48] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:06 [I 2022-10-27 05:22:48] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00365 [I 2022-10-27 05:22:48] ...trap/engines/engine.py.230: loss: 0.16653 [I 2022-10-27 05:22:48] ...optimizers/trijoint.py.151: total_norm: 11.909558 [I 2022-10-27 05:22:54] ...trap/engines/engine.py.224: train: epoch 50 | batch 30/2304 [I 2022-10-27 05:22:54] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-27 05:22:54] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00339 [I 2022-10-27 05:22:54] ...trap/engines/engine.py.230: loss: 0.13078 [I 2022-10-27 05:22:54] ...optimizers/trijoint.py.151: total_norm: 10.689669 [I 2022-10-27 05:23:00] ...trap/engines/engine.py.224: train: epoch 50 | batch 40/2304 [I 2022-10-27 05:23:00] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:54 [I 2022-10-27 05:23:00] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00350 [I 2022-10-27 05:23:00] ...trap/engines/engine.py.230: loss: 0.14702 [I 2022-10-27 05:23:00] ...optimizers/trijoint.py.151: total_norm: 18.858608 [I 2022-10-27 05:23:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 50/2304 [I 2022-10-27 05:23:05] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:57 [I 2022-10-27 05:23:05] ...trap/engines/engine.py.229: process: 0.56742 | load: 0.00293 [I 2022-10-27 05:23:05] ...trap/engines/engine.py.230: loss: 0.13697 [I 2022-10-27 05:23:05] ...optimizers/trijoint.py.151: total_norm: 11.095395 [I 2022-10-27 05:23:11] ...trap/engines/engine.py.224: train: epoch 50 | batch 60/2304 [I 2022-10-27 05:23:11] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:56 [I 2022-10-27 05:23:11] ...trap/engines/engine.py.229: process: 0.56515 | load: 0.00266 [I 2022-10-27 05:23:11] ...trap/engines/engine.py.230: loss: 0.21592 [I 2022-10-27 05:23:11] ...optimizers/trijoint.py.151: total_norm: 13.200082 [I 2022-10-27 05:23:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 70/2304 [I 2022-10-27 05:23:16] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:36 [I 2022-10-27 05:23:16] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00280 [I 2022-10-27 05:23:16] ...trap/engines/engine.py.230: loss: 0.16577 [I 2022-10-27 05:23:16] ...optimizers/trijoint.py.151: total_norm: 11.727552 [I 2022-10-27 05:23:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 80/2304 [I 2022-10-27 05:23:22] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:38 [I 2022-10-27 05:23:22] ...trap/engines/engine.py.229: process: 0.54512 | load: 0.00239 [I 2022-10-27 05:23:22] ...trap/engines/engine.py.230: loss: 0.17898 [I 2022-10-27 05:23:22] ...optimizers/trijoint.py.151: total_norm: 18.036116 [I 2022-10-27 05:23:27] ...trap/engines/engine.py.224: train: epoch 50 | batch 90/2304 [I 2022-10-27 05:23:27] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:35 [I 2022-10-27 05:23:27] ...trap/engines/engine.py.229: process: 0.56623 | load: 0.00245 [I 2022-10-27 05:23:27] ...trap/engines/engine.py.230: loss: 0.17436 [I 2022-10-27 05:23:27] ...optimizers/trijoint.py.151: total_norm: 13.326160 [I 2022-10-27 05:23:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 100/2304 [I 2022-10-27 05:23:33] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:17 [I 2022-10-27 05:23:33] ...trap/engines/engine.py.229: process: 0.53249 | load: 0.00289 [I 2022-10-27 05:23:33] ...trap/engines/engine.py.230: loss: 0.12150 [I 2022-10-27 05:23:33] ...optimizers/trijoint.py.151: total_norm: 7.637712 [I 2022-10-27 05:23:39] ...trap/engines/engine.py.224: train: epoch 50 | batch 110/2304 [I 2022-10-27 05:23:39] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:16 [I 2022-10-27 05:23:39] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00249 [I 2022-10-27 05:23:39] ...trap/engines/engine.py.230: loss: 0.12004 [I 2022-10-27 05:23:39] ...optimizers/trijoint.py.151: total_norm: 7.371334 [I 2022-10-27 05:23:44] ...trap/engines/engine.py.224: train: epoch 50 | batch 120/2304 [I 2022-10-27 05:23:44] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:17 [I 2022-10-27 05:23:44] ...trap/engines/engine.py.229: process: 0.55338 | load: 0.00281 [I 2022-10-27 05:23:44] ...trap/engines/engine.py.230: loss: 0.16375 [I 2022-10-27 05:23:44] ...optimizers/trijoint.py.151: total_norm: 16.670597 [I 2022-10-27 05:23:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 130/2304 [I 2022-10-27 05:23:50] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:31 [I 2022-10-27 05:23:50] ...trap/engines/engine.py.229: process: 0.58731 | load: 0.00252 [I 2022-10-27 05:23:50] ...trap/engines/engine.py.230: loss: 0.18622 [I 2022-10-27 05:23:50] ...optimizers/trijoint.py.151: total_norm: 18.412693 [I 2022-10-27 05:23:55] ...trap/engines/engine.py.224: train: epoch 50 | batch 140/2304 [I 2022-10-27 05:23:55] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:15 [I 2022-10-27 05:23:55] ...trap/engines/engine.py.229: process: 0.56557 | load: 0.00321 [I 2022-10-27 05:23:55] ...trap/engines/engine.py.230: loss: 0.15754 [I 2022-10-27 05:23:55] ...optimizers/trijoint.py.151: total_norm: 20.049450 [I 2022-10-27 05:24:01] ...trap/engines/engine.py.224: train: epoch 50 | batch 150/2304 [I 2022-10-27 05:24:01] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:05 [I 2022-10-27 05:24:01] ...trap/engines/engine.py.229: process: 0.56536 | load: 0.00257 [I 2022-10-27 05:24:01] ...trap/engines/engine.py.230: loss: 0.16932 [I 2022-10-27 05:24:01] ...optimizers/trijoint.py.151: total_norm: 15.251188 [I 2022-10-27 05:24:07] ...trap/engines/engine.py.224: train: epoch 50 | batch 160/2304 [I 2022-10-27 05:24:07] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:55 [I 2022-10-27 05:24:07] ...trap/engines/engine.py.229: process: 0.56410 | load: 0.00261 [I 2022-10-27 05:24:07] ...trap/engines/engine.py.230: loss: 0.23834 [I 2022-10-27 05:24:07] ...optimizers/trijoint.py.151: total_norm: 21.348579 [I 2022-10-27 05:24:12] ...trap/engines/engine.py.224: train: epoch 50 | batch 170/2304 [I 2022-10-27 05:24:12] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:45 [I 2022-10-27 05:24:12] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00253 [I 2022-10-27 05:24:12] ...trap/engines/engine.py.230: loss: 0.18776 [I 2022-10-27 05:24:12] ...optimizers/trijoint.py.151: total_norm: 35.542240 [I 2022-10-27 05:24:18] ...trap/engines/engine.py.224: train: epoch 50 | batch 180/2304 [I 2022-10-27 05:24:18] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:43 [I 2022-10-27 05:24:18] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00248 [I 2022-10-27 05:24:18] ...trap/engines/engine.py.230: loss: 0.14315 [I 2022-10-27 05:24:18] ...optimizers/trijoint.py.151: total_norm: 14.963073 [I 2022-10-27 05:24:23] ...trap/engines/engine.py.224: train: epoch 50 | batch 190/2304 [I 2022-10-27 05:24:23] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:43 [I 2022-10-27 05:24:23] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00250 [I 2022-10-27 05:24:23] ...trap/engines/engine.py.230: loss: 0.30328 [I 2022-10-27 05:24:23] ...optimizers/trijoint.py.151: total_norm: 18.580582 [I 2022-10-27 05:24:29] ...trap/engines/engine.py.224: train: epoch 50 | batch 200/2304 [I 2022-10-27 05:24:29] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:27 [I 2022-10-27 05:24:29] ...trap/engines/engine.py.229: process: 0.54802 | load: 0.00256 [I 2022-10-27 05:24:29] ...trap/engines/engine.py.230: loss: 0.17067 [I 2022-10-27 05:24:29] ...optimizers/trijoint.py.151: total_norm: 18.123373 [I 2022-10-27 05:24:35] ...trap/engines/engine.py.224: train: epoch 50 | batch 210/2304 [I 2022-10-27 05:24:35] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:39 [I 2022-10-27 05:24:35] ...trap/engines/engine.py.229: process: 0.57027 | load: 0.00257 [I 2022-10-27 05:24:35] ...trap/engines/engine.py.230: loss: 0.19015 [I 2022-10-27 05:24:35] ...optimizers/trijoint.py.151: total_norm: 11.308777 [I 2022-10-27 05:24:40] ...trap/engines/engine.py.224: train: epoch 50 | batch 220/2304 [I 2022-10-27 05:24:40] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:28 [I 2022-10-27 05:24:40] ...trap/engines/engine.py.229: process: 0.55302 | load: 0.00259 [I 2022-10-27 05:24:40] ...trap/engines/engine.py.230: loss: 0.12890 [I 2022-10-27 05:24:40] ...optimizers/trijoint.py.151: total_norm: 9.813362 [I 2022-10-27 05:24:46] ...trap/engines/engine.py.224: train: epoch 50 | batch 230/2304 [I 2022-10-27 05:24:46] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:17 [I 2022-10-27 05:24:46] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00270 [I 2022-10-27 05:24:46] ...trap/engines/engine.py.230: loss: 0.21802 [I 2022-10-27 05:24:46] ...optimizers/trijoint.py.151: total_norm: 18.707413 [I 2022-10-27 05:24:51] ...trap/engines/engine.py.224: train: epoch 50 | batch 240/2304 [I 2022-10-27 05:24:51] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:14 [I 2022-10-27 05:24:51] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00249 [I 2022-10-27 05:24:51] ...trap/engines/engine.py.230: loss: 0.17636 [I 2022-10-27 05:24:51] ...optimizers/trijoint.py.151: total_norm: 11.182321 [I 2022-10-27 05:24:57] ...trap/engines/engine.py.224: train: epoch 50 | batch 250/2304 [I 2022-10-27 05:24:57] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:06 [I 2022-10-27 05:24:57] ...trap/engines/engine.py.229: process: 0.56056 | load: 0.00252 [I 2022-10-27 05:24:57] ...trap/engines/engine.py.230: loss: 0.18989 [I 2022-10-27 05:24:57] ...optimizers/trijoint.py.151: total_norm: 21.742172 [I 2022-10-27 05:25:03] ...trap/engines/engine.py.224: train: epoch 50 | batch 260/2304 [I 2022-10-27 05:25:03] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:06 [I 2022-10-27 05:25:03] ...trap/engines/engine.py.229: process: 0.55845 | load: 0.00252 [I 2022-10-27 05:25:03] ...trap/engines/engine.py.230: loss: 0.13128 [I 2022-10-27 05:25:03] ...optimizers/trijoint.py.151: total_norm: 18.966925 [I 2022-10-27 05:25:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 270/2304 [I 2022-10-27 05:25:08] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:49 [I 2022-10-27 05:25:08] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00244 [I 2022-10-27 05:25:08] ...trap/engines/engine.py.230: loss: 0.11266 [I 2022-10-27 05:25:08] ...optimizers/trijoint.py.151: total_norm: 8.565110 [I 2022-10-27 05:25:14] ...trap/engines/engine.py.224: train: epoch 50 | batch 280/2304 [I 2022-10-27 05:25:14] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:34 [I 2022-10-27 05:25:14] ...trap/engines/engine.py.229: process: 0.54096 | load: 0.00247 [I 2022-10-27 05:25:14] ...trap/engines/engine.py.230: loss: 0.15181 [I 2022-10-27 05:25:14] ...optimizers/trijoint.py.151: total_norm: 18.389700 [I 2022-10-27 05:25:19] ...trap/engines/engine.py.224: train: epoch 50 | batch 290/2304 [I 2022-10-27 05:25:19] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:34 [I 2022-10-27 05:25:19] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00244 [I 2022-10-27 05:25:19] ...trap/engines/engine.py.230: loss: 0.22029 [I 2022-10-27 05:25:19] ...optimizers/trijoint.py.151: total_norm: 30.744619 [I 2022-10-27 05:25:25] ...trap/engines/engine.py.224: train: epoch 50 | batch 300/2304 [I 2022-10-27 05:25:25] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:43 [I 2022-10-27 05:25:25] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00245 [I 2022-10-27 05:25:25] ...trap/engines/engine.py.230: loss: 0.12122 [I 2022-10-27 05:25:25] ...optimizers/trijoint.py.151: total_norm: 12.225924 [I 2022-10-27 05:25:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 310/2304 [I 2022-10-27 05:25:30] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:27 [I 2022-10-27 05:25:30] ...trap/engines/engine.py.229: process: 0.54866 | load: 0.00242 [I 2022-10-27 05:25:30] ...trap/engines/engine.py.230: loss: 0.16727 [I 2022-10-27 05:25:30] ...optimizers/trijoint.py.151: total_norm: 18.465956 [I 2022-10-27 05:25:36] ...trap/engines/engine.py.224: train: epoch 50 | batch 320/2304 [I 2022-10-27 05:25:36] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:26 [I 2022-10-27 05:25:36] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00252 [I 2022-10-27 05:25:36] ...trap/engines/engine.py.230: loss: 0.18149 [I 2022-10-27 05:25:36] ...optimizers/trijoint.py.151: total_norm: 21.175371 [I 2022-10-27 05:25:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 330/2304 [I 2022-10-27 05:25:42] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:30 [I 2022-10-27 05:25:42] ...trap/engines/engine.py.229: process: 0.57034 | load: 0.00250 [I 2022-10-27 05:25:42] ...trap/engines/engine.py.230: loss: 0.21566 [I 2022-10-27 05:25:42] ...optimizers/trijoint.py.151: total_norm: 16.314993 [I 2022-10-27 05:25:47] ...trap/engines/engine.py.224: train: epoch 50 | batch 340/2304 [I 2022-10-27 05:25:47] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:11 [I 2022-10-27 05:25:47] ...trap/engines/engine.py.229: process: 0.55295 | load: 0.00242 [I 2022-10-27 05:25:47] ...trap/engines/engine.py.230: loss: 0.13257 [I 2022-10-27 05:25:47] ...optimizers/trijoint.py.151: total_norm: 18.893747 [I 2022-10-27 05:25:53] ...trap/engines/engine.py.224: train: epoch 50 | batch 350/2304 [I 2022-10-27 05:25:53] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:37 [I 2022-10-27 05:25:53] ...trap/engines/engine.py.229: process: 0.58380 | load: 0.00238 [I 2022-10-27 05:25:53] ...trap/engines/engine.py.230: loss: 0.20919 [I 2022-10-27 05:25:53] ...optimizers/trijoint.py.151: total_norm: 14.379623 [I 2022-10-27 05:25:58] ...trap/engines/engine.py.224: train: epoch 50 | batch 360/2304 [I 2022-10-27 05:25:58] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:05 [I 2022-10-27 05:25:58] ...trap/engines/engine.py.229: process: 0.55312 | load: 0.00243 [I 2022-10-27 05:25:58] ...trap/engines/engine.py.230: loss: 0.12986 [I 2022-10-27 05:25:58] ...optimizers/trijoint.py.151: total_norm: 10.728313 [I 2022-10-27 05:26:04] ...trap/engines/engine.py.224: train: epoch 50 | batch 370/2304 [I 2022-10-27 05:26:04] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:09 [I 2022-10-27 05:26:04] ...trap/engines/engine.py.229: process: 0.55482 | load: 0.00245 [I 2022-10-27 05:26:04] ...trap/engines/engine.py.230: loss: 0.20347 [I 2022-10-27 05:26:04] ...optimizers/trijoint.py.151: total_norm: 30.991167 [I 2022-10-27 05:26:10] ...trap/engines/engine.py.224: train: epoch 50 | batch 380/2304 [I 2022-10-27 05:26:10] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:55 [I 2022-10-27 05:26:10] ...trap/engines/engine.py.229: process: 0.56643 | load: 0.00249 [I 2022-10-27 05:26:10] ...trap/engines/engine.py.230: loss: 0.18369 [I 2022-10-27 05:26:10] ...optimizers/trijoint.py.151: total_norm: 30.424145 [I 2022-10-27 05:26:15] ...trap/engines/engine.py.224: train: epoch 50 | batch 390/2304 [I 2022-10-27 05:26:15] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:43 [I 2022-10-27 05:26:15] ...trap/engines/engine.py.229: process: 0.55491 | load: 0.00252 [I 2022-10-27 05:26:15] ...trap/engines/engine.py.230: loss: 0.13816 [I 2022-10-27 05:26:15] ...optimizers/trijoint.py.151: total_norm: 10.304427 [I 2022-10-27 05:26:21] ...trap/engines/engine.py.224: train: epoch 50 | batch 400/2304 [I 2022-10-27 05:26:21] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:41 [I 2022-10-27 05:26:21] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00246 [I 2022-10-27 05:26:21] ...trap/engines/engine.py.230: loss: 0.21208 [I 2022-10-27 05:26:21] ...optimizers/trijoint.py.151: total_norm: 19.054413 [I 2022-10-27 05:26:27] ...trap/engines/engine.py.224: train: epoch 50 | batch 410/2304 [I 2022-10-27 05:26:27] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:39 [I 2022-10-27 05:26:27] ...trap/engines/engine.py.229: process: 0.56218 | load: 0.00254 [I 2022-10-27 05:26:27] ...trap/engines/engine.py.230: loss: 0.10243 [I 2022-10-27 05:26:27] ...optimizers/trijoint.py.151: total_norm: 11.633665 [I 2022-10-27 05:26:32] ...trap/engines/engine.py.224: train: epoch 50 | batch 420/2304 [I 2022-10-27 05:26:32] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:31 [I 2022-10-27 05:26:32] ...trap/engines/engine.py.229: process: 0.56373 | load: 0.00250 [I 2022-10-27 05:26:32] ...trap/engines/engine.py.230: loss: 0.12734 [I 2022-10-27 05:26:32] ...optimizers/trijoint.py.151: total_norm: 14.079733 [I 2022-10-27 05:26:38] ...trap/engines/engine.py.224: train: epoch 50 | batch 430/2304 [I 2022-10-27 05:26:38] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:25 [I 2022-10-27 05:26:38] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00255 [I 2022-10-27 05:26:38] ...trap/engines/engine.py.230: loss: 0.24311 [I 2022-10-27 05:26:38] ...optimizers/trijoint.py.151: total_norm: 42.309669 [I 2022-10-27 05:26:43] ...trap/engines/engine.py.224: train: epoch 50 | batch 440/2304 [I 2022-10-27 05:26:43] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:25 [I 2022-10-27 05:26:43] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00246 [I 2022-10-27 05:26:43] ...trap/engines/engine.py.230: loss: 0.18930 [I 2022-10-27 05:26:43] ...optimizers/trijoint.py.151: total_norm: 13.166879 [I 2022-10-27 05:26:49] ...trap/engines/engine.py.224: train: epoch 50 | batch 450/2304 [I 2022-10-27 05:26:49] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:18 [I 2022-10-27 05:26:49] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00253 [I 2022-10-27 05:26:49] ...trap/engines/engine.py.230: loss: 0.17908 [I 2022-10-27 05:26:49] ...optimizers/trijoint.py.151: total_norm: 14.791205 [I 2022-10-27 05:26:55] ...trap/engines/engine.py.224: train: epoch 50 | batch 460/2304 [I 2022-10-27 05:26:55] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:05 [I 2022-10-27 05:26:55] ...trap/engines/engine.py.229: process: 0.54933 | load: 0.00250 [I 2022-10-27 05:26:55] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-27 05:26:55] ...optimizers/trijoint.py.151: total_norm: 9.601089 [I 2022-10-27 05:27:00] ...trap/engines/engine.py.224: train: epoch 50 | batch 470/2304 [I 2022-10-27 05:27:00] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:08 [I 2022-10-27 05:27:00] ...trap/engines/engine.py.229: process: 0.56918 | load: 0.00248 [I 2022-10-27 05:27:00] ...trap/engines/engine.py.230: loss: 0.15736 [I 2022-10-27 05:27:00] ...optimizers/trijoint.py.151: total_norm: 14.164909 [I 2022-10-27 05:27:06] ...trap/engines/engine.py.224: train: epoch 50 | batch 480/2304 [I 2022-10-27 05:27:06] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:57 [I 2022-10-27 05:27:06] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00241 [I 2022-10-27 05:27:06] ...trap/engines/engine.py.230: loss: 0.15455 [I 2022-10-27 05:27:06] ...optimizers/trijoint.py.151: total_norm: 9.910041 [I 2022-10-27 05:27:11] ...trap/engines/engine.py.224: train: epoch 50 | batch 490/2304 [I 2022-10-27 05:27:11] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:57 [I 2022-10-27 05:27:11] ...trap/engines/engine.py.229: process: 0.54922 | load: 0.00249 [I 2022-10-27 05:27:11] ...trap/engines/engine.py.230: loss: 0.16506 [I 2022-10-27 05:27:11] ...optimizers/trijoint.py.151: total_norm: 18.038641 [I 2022-10-27 05:27:17] ...trap/engines/engine.py.224: train: epoch 50 | batch 500/2304 [I 2022-10-27 05:27:17] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:48 [I 2022-10-27 05:27:17] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00247 [I 2022-10-27 05:27:17] ...trap/engines/engine.py.230: loss: 0.22062 [I 2022-10-27 05:27:17] ...optimizers/trijoint.py.151: total_norm: 14.916929 [I 2022-10-27 05:27:23] ...trap/engines/engine.py.224: train: epoch 50 | batch 510/2304 [I 2022-10-27 05:27:23] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:54 [I 2022-10-27 05:27:23] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00251 [I 2022-10-27 05:27:23] ...trap/engines/engine.py.230: loss: 0.14332 [I 2022-10-27 05:27:23] ...optimizers/trijoint.py.151: total_norm: 14.715385 [I 2022-10-27 05:27:28] ...trap/engines/engine.py.224: train: epoch 50 | batch 520/2304 [I 2022-10-27 05:27:28] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:41 [I 2022-10-27 05:27:28] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00255 [I 2022-10-27 05:27:28] ...trap/engines/engine.py.230: loss: 0.15797 [I 2022-10-27 05:27:28] ...optimizers/trijoint.py.151: total_norm: 20.716999 [I 2022-10-27 05:27:34] ...trap/engines/engine.py.224: train: epoch 50 | batch 530/2304 [I 2022-10-27 05:27:34] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:30 [I 2022-10-27 05:27:34] ...trap/engines/engine.py.229: process: 0.55364 | load: 0.00243 [I 2022-10-27 05:27:34] ...trap/engines/engine.py.230: loss: 0.17486 [I 2022-10-27 05:27:34] ...optimizers/trijoint.py.151: total_norm: 18.686426 [I 2022-10-27 05:27:39] ...trap/engines/engine.py.224: train: epoch 50 | batch 540/2304 [I 2022-10-27 05:27:39] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:21 [I 2022-10-27 05:27:39] ...trap/engines/engine.py.229: process: 0.55437 | load: 0.00244 [I 2022-10-27 05:27:39] ...trap/engines/engine.py.230: loss: 0.16047 [I 2022-10-27 05:27:39] ...optimizers/trijoint.py.151: total_norm: 22.805273 [I 2022-10-27 05:27:45] ...trap/engines/engine.py.224: train: epoch 50 | batch 550/2304 [I 2022-10-27 05:27:45] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:15 [I 2022-10-27 05:27:45] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00249 [I 2022-10-27 05:27:45] ...trap/engines/engine.py.230: loss: 0.28447 [I 2022-10-27 05:27:45] ...optimizers/trijoint.py.151: total_norm: 22.214256 [I 2022-10-27 05:27:51] ...trap/engines/engine.py.224: train: epoch 50 | batch 560/2304 [I 2022-10-27 05:27:51] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:11 [I 2022-10-27 05:27:51] ...trap/engines/engine.py.229: process: 0.57022 | load: 0.00243 [I 2022-10-27 05:27:51] ...trap/engines/engine.py.230: loss: 0.22503 [I 2022-10-27 05:27:51] ...optimizers/trijoint.py.151: total_norm: 19.703182 [I 2022-10-27 05:27:56] ...trap/engines/engine.py.224: train: epoch 50 | batch 570/2304 [I 2022-10-27 05:27:56] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:07 [I 2022-10-27 05:27:56] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00241 [I 2022-10-27 05:27:56] ...trap/engines/engine.py.230: loss: 0.14406 [I 2022-10-27 05:27:56] ...optimizers/trijoint.py.151: total_norm: 10.957931 [I 2022-10-27 05:28:02] ...trap/engines/engine.py.224: train: epoch 50 | batch 580/2304 [I 2022-10-27 05:28:02] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:13 [I 2022-10-27 05:28:02] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00243 [I 2022-10-27 05:28:02] ...trap/engines/engine.py.230: loss: 0.15532 [I 2022-10-27 05:28:02] ...optimizers/trijoint.py.151: total_norm: 11.934133 [I 2022-10-27 05:28:07] ...trap/engines/engine.py.224: train: epoch 50 | batch 590/2304 [I 2022-10-27 05:28:07] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:58 [I 2022-10-27 05:28:07] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00261 [I 2022-10-27 05:28:07] ...trap/engines/engine.py.230: loss: 0.17648 [I 2022-10-27 05:28:07] ...optimizers/trijoint.py.151: total_norm: 18.054199 [I 2022-10-27 05:28:13] ...trap/engines/engine.py.224: train: epoch 50 | batch 600/2304 [I 2022-10-27 05:28:13] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:57 [I 2022-10-27 05:28:13] ...trap/engines/engine.py.229: process: 0.56344 | load: 0.00242 [I 2022-10-27 05:28:13] ...trap/engines/engine.py.230: loss: 0.23249 [I 2022-10-27 05:28:13] ...optimizers/trijoint.py.151: total_norm: 17.656672 [I 2022-10-27 05:28:19] ...trap/engines/engine.py.224: train: epoch 50 | batch 610/2304 [I 2022-10-27 05:28:19] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:50 [I 2022-10-27 05:28:19] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00257 [I 2022-10-27 05:28:19] ...trap/engines/engine.py.230: loss: 0.15367 [I 2022-10-27 05:28:19] ...optimizers/trijoint.py.151: total_norm: 20.554340 [I 2022-10-27 05:28:24] ...trap/engines/engine.py.224: train: epoch 50 | batch 620/2304 [I 2022-10-27 05:28:24] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:47 [I 2022-10-27 05:28:24] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00251 [I 2022-10-27 05:28:24] ...trap/engines/engine.py.230: loss: 0.19946 [I 2022-10-27 05:28:24] ...optimizers/trijoint.py.151: total_norm: 23.666199 [I 2022-10-27 05:28:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 630/2304 [I 2022-10-27 05:28:30] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:37 [I 2022-10-27 05:28:30] ...trap/engines/engine.py.229: process: 0.56566 | load: 0.00244 [I 2022-10-27 05:28:30] ...trap/engines/engine.py.230: loss: 0.19633 [I 2022-10-27 05:28:30] ...optimizers/trijoint.py.151: total_norm: 8.890964 [I 2022-10-27 05:28:36] ...trap/engines/engine.py.224: train: epoch 50 | batch 640/2304 [I 2022-10-27 05:28:36] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:26 [I 2022-10-27 05:28:36] ...trap/engines/engine.py.229: process: 0.55413 | load: 0.00257 [I 2022-10-27 05:28:36] ...trap/engines/engine.py.230: loss: 0.14758 [I 2022-10-27 05:28:36] ...optimizers/trijoint.py.151: total_norm: 10.408252 [I 2022-10-27 05:28:41] ...trap/engines/engine.py.224: train: epoch 50 | batch 650/2304 [I 2022-10-27 05:28:41] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:23 [I 2022-10-27 05:28:41] ...trap/engines/engine.py.229: process: 0.55480 | load: 0.00250 [I 2022-10-27 05:28:41] ...trap/engines/engine.py.230: loss: 0.16862 [I 2022-10-27 05:28:41] ...optimizers/trijoint.py.151: total_norm: 24.283066 [I 2022-10-27 05:28:47] ...trap/engines/engine.py.224: train: epoch 50 | batch 660/2304 [I 2022-10-27 05:28:47] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:21 [I 2022-10-27 05:28:47] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00250 [I 2022-10-27 05:28:47] ...trap/engines/engine.py.230: loss: 0.20545 [I 2022-10-27 05:28:47] ...optimizers/trijoint.py.151: total_norm: 26.285675 [I 2022-10-27 05:28:52] ...trap/engines/engine.py.224: train: epoch 50 | batch 670/2304 [I 2022-10-27 05:28:52] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:16 [I 2022-10-27 05:28:52] ...trap/engines/engine.py.229: process: 0.56634 | load: 0.00245 [I 2022-10-27 05:28:52] ...trap/engines/engine.py.230: loss: 0.15273 [I 2022-10-27 05:28:52] ...optimizers/trijoint.py.151: total_norm: 18.884014 [I 2022-10-27 05:28:58] ...trap/engines/engine.py.224: train: epoch 50 | batch 680/2304 [I 2022-10-27 05:28:58] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:08 [I 2022-10-27 05:28:58] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00245 [I 2022-10-27 05:28:58] ...trap/engines/engine.py.230: loss: 0.16314 [I 2022-10-27 05:28:58] ...optimizers/trijoint.py.151: total_norm: 21.255869 [I 2022-10-27 05:29:04] ...trap/engines/engine.py.224: train: epoch 50 | batch 690/2304 [I 2022-10-27 05:29:04] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:07 [I 2022-10-27 05:29:04] ...trap/engines/engine.py.229: process: 0.56432 | load: 0.00258 [I 2022-10-27 05:29:04] ...trap/engines/engine.py.230: loss: 0.10420 [I 2022-10-27 05:29:04] ...optimizers/trijoint.py.151: total_norm: 10.853662 [I 2022-10-27 05:29:09] ...trap/engines/engine.py.224: train: epoch 50 | batch 700/2304 [I 2022-10-27 05:29:09] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:57 [I 2022-10-27 05:29:09] ...trap/engines/engine.py.229: process: 0.55127 | load: 0.00259 [I 2022-10-27 05:29:09] ...trap/engines/engine.py.230: loss: 0.11563 [I 2022-10-27 05:29:09] ...optimizers/trijoint.py.151: total_norm: 11.286668 [I 2022-10-27 05:29:15] ...trap/engines/engine.py.224: train: epoch 50 | batch 710/2304 [I 2022-10-27 05:29:15] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:56 [I 2022-10-27 05:29:15] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00259 [I 2022-10-27 05:29:15] ...trap/engines/engine.py.230: loss: 0.14381 [I 2022-10-27 05:29:15] ...optimizers/trijoint.py.151: total_norm: 24.267307 [I 2022-10-27 05:29:20] ...trap/engines/engine.py.224: train: epoch 50 | batch 720/2304 [I 2022-10-27 05:29:20] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:50 [I 2022-10-27 05:29:20] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00250 [I 2022-10-27 05:29:20] ...trap/engines/engine.py.230: loss: 0.12764 [I 2022-10-27 05:29:20] ...optimizers/trijoint.py.151: total_norm: 17.739918 [I 2022-10-27 05:29:26] ...trap/engines/engine.py.224: train: epoch 50 | batch 730/2304 [I 2022-10-27 05:29:26] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:58 [I 2022-10-27 05:29:26] ...trap/engines/engine.py.229: process: 0.54624 | load: 0.00253 [I 2022-10-27 05:29:26] ...trap/engines/engine.py.230: loss: 0.23707 [I 2022-10-27 05:29:26] ...optimizers/trijoint.py.151: total_norm: 15.556479 [I 2022-10-27 05:29:32] ...trap/engines/engine.py.224: train: epoch 50 | batch 740/2304 [I 2022-10-27 05:29:32] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:38 [I 2022-10-27 05:29:32] ...trap/engines/engine.py.229: process: 0.56406 | load: 0.00243 [I 2022-10-27 05:29:32] ...trap/engines/engine.py.230: loss: 0.22990 [I 2022-10-27 05:29:32] ...optimizers/trijoint.py.151: total_norm: 26.962166 [I 2022-10-27 05:29:37] ...trap/engines/engine.py.224: train: epoch 50 | batch 750/2304 [I 2022-10-27 05:29:37] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:29 [I 2022-10-27 05:29:37] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00245 [I 2022-10-27 05:29:37] ...trap/engines/engine.py.230: loss: 0.13654 [I 2022-10-27 05:29:37] ...optimizers/trijoint.py.151: total_norm: 12.633580 [I 2022-10-27 05:29:43] ...trap/engines/engine.py.224: train: epoch 50 | batch 760/2304 [I 2022-10-27 05:29:43] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:17 [I 2022-10-27 05:29:43] ...trap/engines/engine.py.229: process: 0.55071 | load: 0.00245 [I 2022-10-27 05:29:43] ...trap/engines/engine.py.230: loss: 0.10593 [I 2022-10-27 05:29:43] ...optimizers/trijoint.py.151: total_norm: 7.059679 [I 2022-10-27 05:29:49] ...trap/engines/engine.py.224: train: epoch 50 | batch 770/2304 [I 2022-10-27 05:29:49] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:23 [I 2022-10-27 05:29:49] ...trap/engines/engine.py.229: process: 0.58182 | load: 0.00255 [I 2022-10-27 05:29:49] ...trap/engines/engine.py.230: loss: 0.18213 [I 2022-10-27 05:29:49] ...optimizers/trijoint.py.151: total_norm: 15.514575 [I 2022-10-27 05:29:54] ...trap/engines/engine.py.224: train: epoch 50 | batch 780/2304 [I 2022-10-27 05:29:54] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:06 [I 2022-10-27 05:29:54] ...trap/engines/engine.py.229: process: 0.54617 | load: 0.00244 [I 2022-10-27 05:29:54] ...trap/engines/engine.py.230: loss: 0.17604 [I 2022-10-27 05:29:54] ...optimizers/trijoint.py.151: total_norm: 15.825873 [I 2022-10-27 05:30:00] ...trap/engines/engine.py.224: train: epoch 50 | batch 790/2304 [I 2022-10-27 05:30:00] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:08 [I 2022-10-27 05:30:00] ...trap/engines/engine.py.229: process: 0.56389 | load: 0.00251 [I 2022-10-27 05:30:00] ...trap/engines/engine.py.230: loss: 0.15980 [I 2022-10-27 05:30:00] ...optimizers/trijoint.py.151: total_norm: 13.951522 [I 2022-10-27 05:30:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 800/2304 [I 2022-10-27 05:30:05] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:14:03 [I 2022-10-27 05:30:05] ...trap/engines/engine.py.229: process: 0.55628 | load: 0.00245 [I 2022-10-27 05:30:05] ...trap/engines/engine.py.230: loss: 0.12826 [I 2022-10-27 05:30:05] ...optimizers/trijoint.py.151: total_norm: 13.404379 [I 2022-10-27 05:30:11] ...trap/engines/engine.py.224: train: epoch 50 | batch 810/2304 [I 2022-10-27 05:30:11] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:58 [I 2022-10-27 05:30:11] ...trap/engines/engine.py.229: process: 0.56992 | load: 0.00254 [I 2022-10-27 05:30:11] ...trap/engines/engine.py.230: loss: 0.13118 [I 2022-10-27 05:30:11] ...optimizers/trijoint.py.151: total_norm: 12.519377 [I 2022-10-27 05:30:17] ...trap/engines/engine.py.224: train: epoch 50 | batch 820/2304 [I 2022-10-27 05:30:17] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:50 [I 2022-10-27 05:30:17] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00255 [I 2022-10-27 05:30:17] ...trap/engines/engine.py.230: loss: 0.27276 [I 2022-10-27 05:30:17] ...optimizers/trijoint.py.151: total_norm: 19.849243 [I 2022-10-27 05:30:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 830/2304 [I 2022-10-27 05:30:22] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:44 [I 2022-10-27 05:30:22] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00254 [I 2022-10-27 05:30:22] ...trap/engines/engine.py.230: loss: 0.16928 [I 2022-10-27 05:30:22] ...optimizers/trijoint.py.151: total_norm: 16.286247 [I 2022-10-27 05:30:28] ...trap/engines/engine.py.224: train: epoch 50 | batch 840/2304 [I 2022-10-27 05:30:28] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:37 [I 2022-10-27 05:30:28] ...trap/engines/engine.py.229: process: 0.57077 | load: 0.00249 [I 2022-10-27 05:30:28] ...trap/engines/engine.py.230: loss: 0.15350 [I 2022-10-27 05:30:28] ...optimizers/trijoint.py.151: total_norm: 22.883854 [I 2022-10-27 05:30:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 850/2304 [I 2022-10-27 05:30:33] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:37 [I 2022-10-27 05:30:33] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00254 [I 2022-10-27 05:30:33] ...trap/engines/engine.py.230: loss: 0.12324 [I 2022-10-27 05:30:33] ...optimizers/trijoint.py.151: total_norm: 9.853389 [I 2022-10-27 05:30:39] ...trap/engines/engine.py.224: train: epoch 50 | batch 860/2304 [I 2022-10-27 05:30:39] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:43 [I 2022-10-27 05:30:39] ...trap/engines/engine.py.229: process: 0.56772 | load: 0.00246 [I 2022-10-27 05:30:39] ...trap/engines/engine.py.230: loss: 0.14683 [I 2022-10-27 05:30:39] ...optimizers/trijoint.py.151: total_norm: 9.596026 [I 2022-10-27 05:30:45] ...trap/engines/engine.py.224: train: epoch 50 | batch 870/2304 [I 2022-10-27 05:30:45] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:26 [I 2022-10-27 05:30:45] ...trap/engines/engine.py.229: process: 0.56539 | load: 0.00247 [I 2022-10-27 05:30:45] ...trap/engines/engine.py.230: loss: 0.11622 [I 2022-10-27 05:30:45] ...optimizers/trijoint.py.151: total_norm: 8.796211 [I 2022-10-27 05:30:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 880/2304 [I 2022-10-27 05:30:50] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:19 [I 2022-10-27 05:30:50] ...trap/engines/engine.py.229: process: 0.56713 | load: 0.00248 [I 2022-10-27 05:30:50] ...trap/engines/engine.py.230: loss: 0.12335 [I 2022-10-27 05:30:50] ...optimizers/trijoint.py.151: total_norm: 13.363289 [I 2022-10-27 05:30:56] ...trap/engines/engine.py.224: train: epoch 50 | batch 890/2304 [I 2022-10-27 05:30:56] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:08 [I 2022-10-27 05:30:56] ...trap/engines/engine.py.229: process: 0.54721 | load: 0.00243 [I 2022-10-27 05:30:56] ...trap/engines/engine.py.230: loss: 0.12652 [I 2022-10-27 05:30:56] ...optimizers/trijoint.py.151: total_norm: 15.527586 [I 2022-10-27 05:31:02] ...trap/engines/engine.py.224: train: epoch 50 | batch 900/2304 [I 2022-10-27 05:31:02] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:04 [I 2022-10-27 05:31:02] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00248 [I 2022-10-27 05:31:02] ...trap/engines/engine.py.230: loss: 0.13928 [I 2022-10-27 05:31:02] ...optimizers/trijoint.py.151: total_norm: 8.747209 [I 2022-10-27 05:31:07] ...trap/engines/engine.py.224: train: epoch 50 | batch 910/2304 [I 2022-10-27 05:31:07] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:58 [I 2022-10-27 05:31:07] ...trap/engines/engine.py.229: process: 0.54961 | load: 0.00242 [I 2022-10-27 05:31:07] ...trap/engines/engine.py.230: loss: 0.12290 [I 2022-10-27 05:31:07] ...optimizers/trijoint.py.151: total_norm: 9.667764 [I 2022-10-27 05:31:13] ...trap/engines/engine.py.224: train: epoch 50 | batch 920/2304 [I 2022-10-27 05:31:13] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:57 [I 2022-10-27 05:31:13] ...trap/engines/engine.py.229: process: 0.55923 | load: 0.00246 [I 2022-10-27 05:31:13] ...trap/engines/engine.py.230: loss: 0.16338 [I 2022-10-27 05:31:13] ...optimizers/trijoint.py.151: total_norm: 20.955185 [I 2022-10-27 05:31:18] ...trap/engines/engine.py.224: train: epoch 50 | batch 930/2304 [I 2022-10-27 05:31:18] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:12:44 [I 2022-10-27 05:31:18] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00244 [I 2022-10-27 05:31:18] ...trap/engines/engine.py.230: loss: 0.27801 [I 2022-10-27 05:31:18] ...optimizers/trijoint.py.151: total_norm: 15.517433 [I 2022-10-27 05:31:24] ...trap/engines/engine.py.224: train: epoch 50 | batch 940/2304 [I 2022-10-27 05:31:24] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:42 [I 2022-10-27 05:31:24] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00243 [I 2022-10-27 05:31:24] ...trap/engines/engine.py.230: loss: 0.15950 [I 2022-10-27 05:31:24] ...optimizers/trijoint.py.151: total_norm: 12.841149 [I 2022-10-27 05:31:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 950/2304 [I 2022-10-27 05:31:30] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:28 [I 2022-10-27 05:31:30] ...trap/engines/engine.py.229: process: 0.54365 | load: 0.00246 [I 2022-10-27 05:31:30] ...trap/engines/engine.py.230: loss: 0.12596 [I 2022-10-27 05:31:30] ...optimizers/trijoint.py.151: total_norm: 7.749218 [I 2022-10-27 05:31:35] ...trap/engines/engine.py.224: train: epoch 50 | batch 960/2304 [I 2022-10-27 05:31:35] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:27 [I 2022-10-27 05:31:35] ...trap/engines/engine.py.229: process: 0.57373 | load: 0.00243 [I 2022-10-27 05:31:35] ...trap/engines/engine.py.230: loss: 0.17596 [I 2022-10-27 05:31:35] ...optimizers/trijoint.py.151: total_norm: 10.739642 [I 2022-10-27 05:31:41] ...trap/engines/engine.py.224: train: epoch 50 | batch 970/2304 [I 2022-10-27 05:31:41] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:27 [I 2022-10-27 05:31:41] ...trap/engines/engine.py.229: process: 0.55501 | load: 0.00251 [I 2022-10-27 05:31:41] ...trap/engines/engine.py.230: loss: 0.13838 [I 2022-10-27 05:31:41] ...optimizers/trijoint.py.151: total_norm: 15.557383 [I 2022-10-27 05:31:46] ...trap/engines/engine.py.224: train: epoch 50 | batch 980/2304 [I 2022-10-27 05:31:46] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:12:20 [I 2022-10-27 05:31:46] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00248 [I 2022-10-27 05:31:46] ...trap/engines/engine.py.230: loss: 0.14886 [I 2022-10-27 05:31:46] ...optimizers/trijoint.py.151: total_norm: 11.270309 [I 2022-10-27 05:31:52] ...trap/engines/engine.py.224: train: epoch 50 | batch 990/2304 [I 2022-10-27 05:31:52] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:16 [I 2022-10-27 05:31:52] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00244 [I 2022-10-27 05:31:52] ...trap/engines/engine.py.230: loss: 0.16627 [I 2022-10-27 05:31:52] ...optimizers/trijoint.py.151: total_norm: 18.630981 [I 2022-10-27 05:31:58] ...trap/engines/engine.py.224: train: epoch 50 | batch 1000/2304 [I 2022-10-27 05:31:58] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:07 [I 2022-10-27 05:31:58] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00244 [I 2022-10-27 05:31:58] ...trap/engines/engine.py.230: loss: 0.14748 [I 2022-10-27 05:31:58] ...optimizers/trijoint.py.151: total_norm: 15.780804 [I 2022-10-27 05:32:03] ...trap/engines/engine.py.224: train: epoch 50 | batch 1010/2304 [I 2022-10-27 05:32:03] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:01 [I 2022-10-27 05:32:03] ...trap/engines/engine.py.229: process: 0.56300 | load: 0.00244 [I 2022-10-27 05:32:03] ...trap/engines/engine.py.230: loss: 0.15591 [I 2022-10-27 05:32:03] ...optimizers/trijoint.py.151: total_norm: 16.894409 [I 2022-10-27 05:32:09] ...trap/engines/engine.py.224: train: epoch 50 | batch 1020/2304 [I 2022-10-27 05:32:09] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:52 [I 2022-10-27 05:32:09] ...trap/engines/engine.py.229: process: 0.54523 | load: 0.00245 [I 2022-10-27 05:32:09] ...trap/engines/engine.py.230: loss: 0.19140 [I 2022-10-27 05:32:09] ...optimizers/trijoint.py.151: total_norm: 29.544580 [I 2022-10-27 05:32:14] ...trap/engines/engine.py.224: train: epoch 50 | batch 1030/2304 [I 2022-10-27 05:32:14] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:11:52 [I 2022-10-27 05:32:14] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00265 [I 2022-10-27 05:32:14] ...trap/engines/engine.py.230: loss: 0.14604 [I 2022-10-27 05:32:14] ...optimizers/trijoint.py.151: total_norm: 16.000954 [I 2022-10-27 05:32:20] ...trap/engines/engine.py.224: train: epoch 50 | batch 1040/2304 [I 2022-10-27 05:32:20] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:46 [I 2022-10-27 05:32:20] ...trap/engines/engine.py.229: process: 0.55237 | load: 0.00244 [I 2022-10-27 05:32:20] ...trap/engines/engine.py.230: loss: 0.19383 [I 2022-10-27 05:32:20] ...optimizers/trijoint.py.151: total_norm: 28.844992 [I 2022-10-27 05:32:26] ...trap/engines/engine.py.224: train: epoch 50 | batch 1050/2304 [I 2022-10-27 05:32:26] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:39 [I 2022-10-27 05:32:26] ...trap/engines/engine.py.229: process: 0.55297 | load: 0.00244 [I 2022-10-27 05:32:26] ...trap/engines/engine.py.230: loss: 0.13642 [I 2022-10-27 05:32:26] ...optimizers/trijoint.py.151: total_norm: 23.438862 [I 2022-10-27 05:32:31] ...trap/engines/engine.py.224: train: epoch 50 | batch 1060/2304 [I 2022-10-27 05:32:31] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:36 [I 2022-10-27 05:32:31] ...trap/engines/engine.py.229: process: 0.56453 | load: 0.00244 [I 2022-10-27 05:32:31] ...trap/engines/engine.py.230: loss: 0.13455 [I 2022-10-27 05:32:31] ...optimizers/trijoint.py.151: total_norm: 14.084478 [I 2022-10-27 05:32:37] ...trap/engines/engine.py.224: train: epoch 50 | batch 1070/2304 [I 2022-10-27 05:32:37] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:33 [I 2022-10-27 05:32:37] ...trap/engines/engine.py.229: process: 0.56529 | load: 0.00243 [I 2022-10-27 05:32:37] ...trap/engines/engine.py.230: loss: 0.14377 [I 2022-10-27 05:32:37] ...optimizers/trijoint.py.151: total_norm: 11.550729 [I 2022-10-27 05:32:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 1080/2304 [I 2022-10-27 05:32:42] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:26 [I 2022-10-27 05:32:42] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00246 [I 2022-10-27 05:32:42] ...trap/engines/engine.py.230: loss: 0.16148 [I 2022-10-27 05:32:42] ...optimizers/trijoint.py.151: total_norm: 37.012146 [I 2022-10-27 05:32:48] ...trap/engines/engine.py.224: train: epoch 50 | batch 1090/2304 [I 2022-10-27 05:32:48] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:20 [I 2022-10-27 05:32:48] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00244 [I 2022-10-27 05:32:48] ...trap/engines/engine.py.230: loss: 0.16431 [I 2022-10-27 05:32:48] ...optimizers/trijoint.py.151: total_norm: 48.919720 [I 2022-10-27 05:32:54] ...trap/engines/engine.py.224: train: epoch 50 | batch 1100/2304 [I 2022-10-27 05:32:54] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:13 [I 2022-10-27 05:32:54] ...trap/engines/engine.py.229: process: 0.54822 | load: 0.00241 [I 2022-10-27 05:32:54] ...trap/engines/engine.py.230: loss: 0.12475 [I 2022-10-27 05:32:54] ...optimizers/trijoint.py.151: total_norm: 19.623590 [I 2022-10-27 05:32:59] ...trap/engines/engine.py.224: train: epoch 50 | batch 1110/2304 [I 2022-10-27 05:32:59] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:04 [I 2022-10-27 05:32:59] ...trap/engines/engine.py.229: process: 0.54515 | load: 0.00245 [I 2022-10-27 05:32:59] ...trap/engines/engine.py.230: loss: 0.11694 [I 2022-10-27 05:32:59] ...optimizers/trijoint.py.151: total_norm: 6.971793 [I 2022-10-27 05:33:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 1120/2304 [I 2022-10-27 05:33:05] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:03 [I 2022-10-27 05:33:05] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00247 [I 2022-10-27 05:33:05] ...trap/engines/engine.py.230: loss: 0.12713 [I 2022-10-27 05:33:05] ...optimizers/trijoint.py.151: total_norm: 16.300087 [I 2022-10-27 05:33:10] ...trap/engines/engine.py.224: train: epoch 50 | batch 1130/2304 [I 2022-10-27 05:33:10] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:10:57 [I 2022-10-27 05:33:10] ...trap/engines/engine.py.229: process: 0.56062 | load: 0.00266 [I 2022-10-27 05:33:10] ...trap/engines/engine.py.230: loss: 0.13257 [I 2022-10-27 05:33:10] ...optimizers/trijoint.py.151: total_norm: 8.104003 [I 2022-10-27 05:33:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 1140/2304 [I 2022-10-27 05:33:16] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:10:51 [I 2022-10-27 05:33:16] ...trap/engines/engine.py.229: process: 0.56674 | load: 0.00242 [I 2022-10-27 05:33:16] ...trap/engines/engine.py.230: loss: 0.11549 [I 2022-10-27 05:33:16] ...optimizers/trijoint.py.151: total_norm: 22.721186 [I 2022-10-27 05:33:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 1150/2304 [I 2022-10-27 05:33:22] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:42 [I 2022-10-27 05:33:22] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00247 [I 2022-10-27 05:33:22] ...trap/engines/engine.py.230: loss: 0.22539 [I 2022-10-27 05:33:22] ...optimizers/trijoint.py.151: total_norm: 21.531145 [I 2022-10-27 05:33:27] ...trap/engines/engine.py.224: train: epoch 50 | batch 1160/2304 [I 2022-10-27 05:33:27] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:38 [I 2022-10-27 05:33:27] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00244 [I 2022-10-27 05:33:27] ...trap/engines/engine.py.230: loss: 0.22047 [I 2022-10-27 05:33:27] ...optimizers/trijoint.py.151: total_norm: 23.769968 [I 2022-10-27 05:33:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 1170/2304 [I 2022-10-27 05:33:33] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:28 [I 2022-10-27 05:33:33] ...trap/engines/engine.py.229: process: 0.55893 | load: 0.00253 [I 2022-10-27 05:33:33] ...trap/engines/engine.py.230: loss: 0.12581 [I 2022-10-27 05:33:33] ...optimizers/trijoint.py.151: total_norm: 22.627548 [I 2022-10-27 05:33:38] ...trap/engines/engine.py.224: train: epoch 50 | batch 1180/2304 [I 2022-10-27 05:33:38] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:24 [I 2022-10-27 05:33:38] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00249 [I 2022-10-27 05:33:38] ...trap/engines/engine.py.230: loss: 0.12387 [I 2022-10-27 05:33:38] ...optimizers/trijoint.py.151: total_norm: 11.188898 [I 2022-10-27 05:33:44] ...trap/engines/engine.py.224: train: epoch 50 | batch 1190/2304 [I 2022-10-27 05:33:44] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:24 [I 2022-10-27 05:33:44] ...trap/engines/engine.py.229: process: 0.57439 | load: 0.00247 [I 2022-10-27 05:33:44] ...trap/engines/engine.py.230: loss: 0.13281 [I 2022-10-27 05:33:44] ...optimizers/trijoint.py.151: total_norm: 20.200403 [I 2022-10-27 05:33:49] ...trap/engines/engine.py.224: train: epoch 50 | batch 1200/2304 [I 2022-10-27 05:33:49] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:18 [I 2022-10-27 05:33:49] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00246 [I 2022-10-27 05:33:49] ...trap/engines/engine.py.230: loss: 0.11329 [I 2022-10-27 05:33:49] ...optimizers/trijoint.py.151: total_norm: 9.582396 [I 2022-10-27 05:33:55] ...trap/engines/engine.py.224: train: epoch 50 | batch 1210/2304 [I 2022-10-27 05:33:55] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:09 [I 2022-10-27 05:33:55] ...trap/engines/engine.py.229: process: 0.56297 | load: 0.00245 [I 2022-10-27 05:33:55] ...trap/engines/engine.py.230: loss: 0.13000 [I 2022-10-27 05:33:55] ...optimizers/trijoint.py.151: total_norm: 20.470385 [I 2022-10-27 05:34:01] ...trap/engines/engine.py.224: train: epoch 50 | batch 1220/2304 [I 2022-10-27 05:34:01] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:03 [I 2022-10-27 05:34:01] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00245 [I 2022-10-27 05:34:01] ...trap/engines/engine.py.230: loss: 0.29779 [I 2022-10-27 05:34:01] ...optimizers/trijoint.py.151: total_norm: 41.809254 [I 2022-10-27 05:34:06] ...trap/engines/engine.py.224: train: epoch 50 | batch 1230/2304 [I 2022-10-27 05:34:06] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:09:56 [I 2022-10-27 05:34:06] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00241 [I 2022-10-27 05:34:06] ...trap/engines/engine.py.230: loss: 0.14254 [I 2022-10-27 05:34:06] ...optimizers/trijoint.py.151: total_norm: 14.079302 [I 2022-10-27 05:34:12] ...trap/engines/engine.py.224: train: epoch 50 | batch 1240/2304 [I 2022-10-27 05:34:12] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:52 [I 2022-10-27 05:34:12] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00240 [I 2022-10-27 05:34:12] ...trap/engines/engine.py.230: loss: 0.11662 [I 2022-10-27 05:34:12] ...optimizers/trijoint.py.151: total_norm: 13.667807 [I 2022-10-27 05:34:17] ...trap/engines/engine.py.224: train: epoch 50 | batch 1250/2304 [I 2022-10-27 05:34:17] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:49 [I 2022-10-27 05:34:17] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00245 [I 2022-10-27 05:34:17] ...trap/engines/engine.py.230: loss: 0.13554 [I 2022-10-27 05:34:17] ...optimizers/trijoint.py.151: total_norm: 18.030590 [I 2022-10-27 05:34:23] ...trap/engines/engine.py.224: train: epoch 50 | batch 1260/2304 [I 2022-10-27 05:34:23] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:42 [I 2022-10-27 05:34:23] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00254 [I 2022-10-27 05:34:23] ...trap/engines/engine.py.230: loss: 0.19705 [I 2022-10-27 05:34:23] ...optimizers/trijoint.py.151: total_norm: 10.924588 [I 2022-10-27 05:34:29] ...trap/engines/engine.py.224: train: epoch 50 | batch 1270/2304 [I 2022-10-27 05:34:29] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:37 [I 2022-10-27 05:34:29] ...trap/engines/engine.py.229: process: 0.55618 | load: 0.00243 [I 2022-10-27 05:34:29] ...trap/engines/engine.py.230: loss: 0.19031 [I 2022-10-27 05:34:29] ...optimizers/trijoint.py.151: total_norm: 22.573019 [I 2022-10-27 05:34:34] ...trap/engines/engine.py.224: train: epoch 50 | batch 1280/2304 [I 2022-10-27 05:34:34] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:28 [I 2022-10-27 05:34:34] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00248 [I 2022-10-27 05:34:34] ...trap/engines/engine.py.230: loss: 0.16728 [I 2022-10-27 05:34:34] ...optimizers/trijoint.py.151: total_norm: 29.845951 [I 2022-10-27 05:34:40] ...trap/engines/engine.py.224: train: epoch 50 | batch 1290/2304 [I 2022-10-27 05:34:40] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:27 [I 2022-10-27 05:34:40] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00250 [I 2022-10-27 05:34:40] ...trap/engines/engine.py.230: loss: 0.12434 [I 2022-10-27 05:34:40] ...optimizers/trijoint.py.151: total_norm: 12.999850 [I 2022-10-27 05:34:45] ...trap/engines/engine.py.224: train: epoch 50 | batch 1300/2304 [I 2022-10-27 05:34:45] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:17 [I 2022-10-27 05:34:45] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00251 [I 2022-10-27 05:34:45] ...trap/engines/engine.py.230: loss: 0.12494 [I 2022-10-27 05:34:45] ...optimizers/trijoint.py.151: total_norm: 11.601201 [I 2022-10-27 05:34:51] ...trap/engines/engine.py.224: train: epoch 50 | batch 1310/2304 [I 2022-10-27 05:34:51] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:13 [I 2022-10-27 05:34:51] ...trap/engines/engine.py.229: process: 0.54689 | load: 0.00250 [I 2022-10-27 05:34:51] ...trap/engines/engine.py.230: loss: 0.14167 [I 2022-10-27 05:34:51] ...optimizers/trijoint.py.151: total_norm: 7.088003 [I 2022-10-27 05:34:57] ...trap/engines/engine.py.224: train: epoch 50 | batch 1320/2304 [I 2022-10-27 05:34:57] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:06 [I 2022-10-27 05:34:57] ...trap/engines/engine.py.229: process: 0.56198 | load: 0.00248 [I 2022-10-27 05:34:57] ...trap/engines/engine.py.230: loss: 0.13379 [I 2022-10-27 05:34:57] ...optimizers/trijoint.py.151: total_norm: 29.296192 [I 2022-10-27 05:35:02] ...trap/engines/engine.py.224: train: epoch 50 | batch 1330/2304 [I 2022-10-27 05:35:02] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:00 [I 2022-10-27 05:35:02] ...trap/engines/engine.py.229: process: 0.55408 | load: 0.00244 [I 2022-10-27 05:35:02] ...trap/engines/engine.py.230: loss: 0.21657 [I 2022-10-27 05:35:02] ...optimizers/trijoint.py.151: total_norm: 13.533833 [I 2022-10-27 05:35:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 1340/2304 [I 2022-10-27 05:35:08] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:08:56 [I 2022-10-27 05:35:08] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00243 [I 2022-10-27 05:35:08] ...trap/engines/engine.py.230: loss: 0.16247 [I 2022-10-27 05:35:08] ...optimizers/trijoint.py.151: total_norm: 21.155426 [I 2022-10-27 05:35:13] ...trap/engines/engine.py.224: train: epoch 50 | batch 1350/2304 [I 2022-10-27 05:35:13] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:50 [I 2022-10-27 05:35:13] ...trap/engines/engine.py.229: process: 0.55949 | load: 0.00293 [I 2022-10-27 05:35:13] ...trap/engines/engine.py.230: loss: 0.25496 [I 2022-10-27 05:35:13] ...optimizers/trijoint.py.151: total_norm: 16.548033 [I 2022-10-27 05:35:19] ...trap/engines/engine.py.224: train: epoch 50 | batch 1360/2304 [I 2022-10-27 05:35:19] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:45 [I 2022-10-27 05:35:19] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00242 [I 2022-10-27 05:35:19] ...trap/engines/engine.py.230: loss: 0.16582 [I 2022-10-27 05:35:19] ...optimizers/trijoint.py.151: total_norm: 16.946865 [I 2022-10-27 05:35:24] ...trap/engines/engine.py.224: train: epoch 50 | batch 1370/2304 [I 2022-10-27 05:35:24] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:36 [I 2022-10-27 05:35:24] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00244 [I 2022-10-27 05:35:24] ...trap/engines/engine.py.230: loss: 0.17658 [I 2022-10-27 05:35:24] ...optimizers/trijoint.py.151: total_norm: 15.504144 [I 2022-10-27 05:35:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 1380/2304 [I 2022-10-27 05:35:30] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:32 [I 2022-10-27 05:35:30] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00242 [I 2022-10-27 05:35:30] ...trap/engines/engine.py.230: loss: 0.10498 [I 2022-10-27 05:35:30] ...optimizers/trijoint.py.151: total_norm: 9.065702 [I 2022-10-27 05:35:36] ...trap/engines/engine.py.224: train: epoch 50 | batch 1390/2304 [I 2022-10-27 05:35:36] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:42 [I 2022-10-27 05:35:36] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00241 [I 2022-10-27 05:35:36] ...trap/engines/engine.py.230: loss: 0.13122 [I 2022-10-27 05:35:36] ...optimizers/trijoint.py.151: total_norm: 21.152620 [I 2022-10-27 05:35:41] ...trap/engines/engine.py.224: train: epoch 50 | batch 1400/2304 [I 2022-10-27 05:35:41] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:24 [I 2022-10-27 05:35:41] ...trap/engines/engine.py.229: process: 0.56742 | load: 0.00247 [I 2022-10-27 05:35:41] ...trap/engines/engine.py.230: loss: 0.20718 [I 2022-10-27 05:35:41] ...optimizers/trijoint.py.151: total_norm: 18.216124 [I 2022-10-27 05:35:47] ...trap/engines/engine.py.224: train: epoch 50 | batch 1410/2304 [I 2022-10-27 05:35:47] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:29 [I 2022-10-27 05:35:47] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00299 [I 2022-10-27 05:35:47] ...trap/engines/engine.py.230: loss: 0.13632 [I 2022-10-27 05:35:47] ...optimizers/trijoint.py.151: total_norm: 8.707270 [I 2022-10-27 05:35:53] ...trap/engines/engine.py.224: train: epoch 50 | batch 1420/2304 [I 2022-10-27 05:35:53] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:13 [I 2022-10-27 05:35:53] ...trap/engines/engine.py.229: process: 0.54916 | load: 0.00262 [I 2022-10-27 05:35:53] ...trap/engines/engine.py.230: loss: 0.17302 [I 2022-10-27 05:35:53] ...optimizers/trijoint.py.151: total_norm: 14.664740 [I 2022-10-27 05:35:58] ...trap/engines/engine.py.224: train: epoch 50 | batch 1430/2304 [I 2022-10-27 05:35:58] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:09 [I 2022-10-27 05:35:58] ...trap/engines/engine.py.229: process: 0.56319 | load: 0.00253 [I 2022-10-27 05:35:58] ...trap/engines/engine.py.230: loss: 0.16155 [I 2022-10-27 05:35:58] ...optimizers/trijoint.py.151: total_norm: 28.692732 [I 2022-10-27 05:36:04] ...trap/engines/engine.py.224: train: epoch 50 | batch 1440/2304 [I 2022-10-27 05:36:04] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:08:04 [I 2022-10-27 05:36:04] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00250 [I 2022-10-27 05:36:04] ...trap/engines/engine.py.230: loss: 0.14430 [I 2022-10-27 05:36:04] ...optimizers/trijoint.py.151: total_norm: 15.037940 [I 2022-10-27 05:36:09] ...trap/engines/engine.py.224: train: epoch 50 | batch 1450/2304 [I 2022-10-27 05:36:09] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:07:54 [I 2022-10-27 05:36:09] ...trap/engines/engine.py.229: process: 0.54418 | load: 0.00248 [I 2022-10-27 05:36:09] ...trap/engines/engine.py.230: loss: 0.15044 [I 2022-10-27 05:36:09] ...optimizers/trijoint.py.151: total_norm: 14.344494 [I 2022-10-27 05:36:15] ...trap/engines/engine.py.224: train: epoch 50 | batch 1460/2304 [I 2022-10-27 05:36:15] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:52 [I 2022-10-27 05:36:15] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00249 [I 2022-10-27 05:36:15] ...trap/engines/engine.py.230: loss: 0.17893 [I 2022-10-27 05:36:15] ...optimizers/trijoint.py.151: total_norm: 22.175755 [I 2022-10-27 05:36:21] ...trap/engines/engine.py.224: train: epoch 50 | batch 1470/2304 [I 2022-10-27 05:36:21] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:08:09 [I 2022-10-27 05:36:21] ...trap/engines/engine.py.229: process: 0.67351 | load: 0.01041 [I 2022-10-27 05:36:21] ...trap/engines/engine.py.230: loss: 0.25097 [I 2022-10-27 05:36:21] ...optimizers/trijoint.py.151: total_norm: 57.990128 [I 2022-10-27 05:36:26] ...trap/engines/engine.py.224: train: epoch 50 | batch 1480/2304 [I 2022-10-27 05:36:26] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:45 [I 2022-10-27 05:36:26] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00248 [I 2022-10-27 05:36:26] ...trap/engines/engine.py.230: loss: 0.28123 [I 2022-10-27 05:36:26] ...optimizers/trijoint.py.151: total_norm: 24.531345 [I 2022-10-27 05:36:32] ...trap/engines/engine.py.224: train: epoch 50 | batch 1490/2304 [I 2022-10-27 05:36:32] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:38 [I 2022-10-27 05:36:32] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00245 [I 2022-10-27 05:36:32] ...trap/engines/engine.py.230: loss: 0.20445 [I 2022-10-27 05:36:32] ...optimizers/trijoint.py.151: total_norm: 33.715092 [I 2022-10-27 05:36:38] ...trap/engines/engine.py.224: train: epoch 50 | batch 1500/2304 [I 2022-10-27 05:36:38] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:07:31 [I 2022-10-27 05:36:38] ...trap/engines/engine.py.229: process: 0.56299 | load: 0.00254 [I 2022-10-27 05:36:38] ...trap/engines/engine.py.230: loss: 0.15839 [I 2022-10-27 05:36:38] ...optimizers/trijoint.py.151: total_norm: 20.722923 [I 2022-10-27 05:36:43] ...trap/engines/engine.py.224: train: epoch 50 | batch 1510/2304 [I 2022-10-27 05:36:43] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:23 [I 2022-10-27 05:36:43] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00247 [I 2022-10-27 05:36:43] ...trap/engines/engine.py.230: loss: 0.16354 [I 2022-10-27 05:36:43] ...optimizers/trijoint.py.151: total_norm: 21.312149 [I 2022-10-27 05:36:49] ...trap/engines/engine.py.224: train: epoch 50 | batch 1520/2304 [I 2022-10-27 05:36:49] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:07:13 [I 2022-10-27 05:36:49] ...trap/engines/engine.py.229: process: 0.54032 | load: 0.00243 [I 2022-10-27 05:36:49] ...trap/engines/engine.py.230: loss: 0.15537 [I 2022-10-27 05:36:49] ...optimizers/trijoint.py.151: total_norm: 15.383149 [I 2022-10-27 05:36:54] ...trap/engines/engine.py.224: train: epoch 50 | batch 1530/2304 [I 2022-10-27 05:36:54] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:10 [I 2022-10-27 05:36:54] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00244 [I 2022-10-27 05:36:54] ...trap/engines/engine.py.230: loss: 0.20461 [I 2022-10-27 05:36:54] ...optimizers/trijoint.py.151: total_norm: 31.401051 [I 2022-10-27 05:37:00] ...trap/engines/engine.py.224: train: epoch 50 | batch 1540/2304 [I 2022-10-27 05:37:00] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:15 [I 2022-10-27 05:37:00] ...trap/engines/engine.py.229: process: 0.56220 | load: 0.00248 [I 2022-10-27 05:37:00] ...trap/engines/engine.py.230: loss: 0.12113 [I 2022-10-27 05:37:00] ...optimizers/trijoint.py.151: total_norm: 9.558880 [I 2022-10-27 05:37:06] ...trap/engines/engine.py.224: train: epoch 50 | batch 1550/2304 [I 2022-10-27 05:37:06] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:07:01 [I 2022-10-27 05:37:06] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00241 [I 2022-10-27 05:37:06] ...trap/engines/engine.py.230: loss: 0.17413 [I 2022-10-27 05:37:06] ...optimizers/trijoint.py.151: total_norm: 16.806480 [I 2022-10-27 05:37:11] ...trap/engines/engine.py.224: train: epoch 50 | batch 1560/2304 [I 2022-10-27 05:37:11] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:55 [I 2022-10-27 05:37:11] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00246 [I 2022-10-27 05:37:11] ...trap/engines/engine.py.230: loss: 0.12701 [I 2022-10-27 05:37:11] ...optimizers/trijoint.py.151: total_norm: 9.242124 [I 2022-10-27 05:37:17] ...trap/engines/engine.py.224: train: epoch 50 | batch 1570/2304 [I 2022-10-27 05:37:17] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:06:48 [I 2022-10-27 05:37:17] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00244 [I 2022-10-27 05:37:17] ...trap/engines/engine.py.230: loss: 0.19087 [I 2022-10-27 05:37:17] ...optimizers/trijoint.py.151: total_norm: 19.163656 [I 2022-10-27 05:37:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 1580/2304 [I 2022-10-27 05:37:22] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:06:44 [I 2022-10-27 05:37:22] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00242 [I 2022-10-27 05:37:22] ...trap/engines/engine.py.230: loss: 0.30038 [I 2022-10-27 05:37:22] ...optimizers/trijoint.py.151: total_norm: 14.843354 [I 2022-10-27 05:37:28] ...trap/engines/engine.py.224: train: epoch 50 | batch 1590/2304 [I 2022-10-27 05:37:28] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:38 [I 2022-10-27 05:37:28] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00248 [I 2022-10-27 05:37:28] ...trap/engines/engine.py.230: loss: 0.25956 [I 2022-10-27 05:37:28] ...optimizers/trijoint.py.151: total_norm: 12.604767 [I 2022-10-27 05:37:34] ...trap/engines/engine.py.224: train: epoch 50 | batch 1600/2304 [I 2022-10-27 05:37:34] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:33 [I 2022-10-27 05:37:34] ...trap/engines/engine.py.229: process: 0.55265 | load: 0.00246 [I 2022-10-27 05:37:34] ...trap/engines/engine.py.230: loss: 0.20250 [I 2022-10-27 05:37:34] ...optimizers/trijoint.py.151: total_norm: 17.796097 [I 2022-10-27 05:37:39] ...trap/engines/engine.py.224: train: epoch 50 | batch 1610/2304 [I 2022-10-27 05:37:39] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:27 [I 2022-10-27 05:37:39] ...trap/engines/engine.py.229: process: 0.54450 | load: 0.00243 [I 2022-10-27 05:37:39] ...trap/engines/engine.py.230: loss: 0.24801 [I 2022-10-27 05:37:39] ...optimizers/trijoint.py.151: total_norm: 27.246990 [I 2022-10-27 05:37:45] ...trap/engines/engine.py.224: train: epoch 50 | batch 1620/2304 [I 2022-10-27 05:37:45] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:20 [I 2022-10-27 05:37:45] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00252 [I 2022-10-27 05:37:45] ...trap/engines/engine.py.230: loss: 0.19453 [I 2022-10-27 05:37:45] ...optimizers/trijoint.py.151: total_norm: 17.641670 [I 2022-10-27 05:37:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 1630/2304 [I 2022-10-27 05:37:50] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:12 [I 2022-10-27 05:37:50] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00254 [I 2022-10-27 05:37:50] ...trap/engines/engine.py.230: loss: 0.15531 [I 2022-10-27 05:37:50] ...optimizers/trijoint.py.151: total_norm: 15.650189 [I 2022-10-27 05:37:56] ...trap/engines/engine.py.224: train: epoch 50 | batch 1640/2304 [I 2022-10-27 05:37:56] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:10 [I 2022-10-27 05:37:56] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00245 [I 2022-10-27 05:37:56] ...trap/engines/engine.py.230: loss: 0.19114 [I 2022-10-27 05:37:56] ...optimizers/trijoint.py.151: total_norm: 45.598194 [I 2022-10-27 05:38:02] ...trap/engines/engine.py.224: train: epoch 50 | batch 1650/2304 [I 2022-10-27 05:38:02] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:14 [I 2022-10-27 05:38:02] ...trap/engines/engine.py.229: process: 0.56300 | load: 0.00246 [I 2022-10-27 05:38:02] ...trap/engines/engine.py.230: loss: 0.13852 [I 2022-10-27 05:38:02] ...optimizers/trijoint.py.151: total_norm: 20.086752 [I 2022-10-27 05:38:07] ...trap/engines/engine.py.224: train: epoch 50 | batch 1660/2304 [I 2022-10-27 05:38:07] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:06:02 [I 2022-10-27 05:38:07] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00240 [I 2022-10-27 05:38:07] ...trap/engines/engine.py.230: loss: 0.15310 [I 2022-10-27 05:38:07] ...optimizers/trijoint.py.151: total_norm: 14.576702 [I 2022-10-27 05:38:13] ...trap/engines/engine.py.224: train: epoch 50 | batch 1670/2304 [I 2022-10-27 05:38:13] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:56 [I 2022-10-27 05:38:13] ...trap/engines/engine.py.229: process: 0.56564 | load: 0.00243 [I 2022-10-27 05:38:13] ...trap/engines/engine.py.230: loss: 0.14132 [I 2022-10-27 05:38:13] ...optimizers/trijoint.py.151: total_norm: 17.989006 [I 2022-10-27 05:38:19] ...trap/engines/engine.py.224: train: epoch 50 | batch 1680/2304 [I 2022-10-27 05:38:19] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:05:50 [I 2022-10-27 05:38:19] ...trap/engines/engine.py.229: process: 0.56744 | load: 0.00243 [I 2022-10-27 05:38:19] ...trap/engines/engine.py.230: loss: 0.22236 [I 2022-10-27 05:38:19] ...optimizers/trijoint.py.151: total_norm: 11.106514 [I 2022-10-27 05:38:24] ...trap/engines/engine.py.224: train: epoch 50 | batch 1690/2304 [I 2022-10-27 05:38:24] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:42 [I 2022-10-27 05:38:24] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00246 [I 2022-10-27 05:38:24] ...trap/engines/engine.py.230: loss: 0.17130 [I 2022-10-27 05:38:24] ...optimizers/trijoint.py.151: total_norm: 18.919676 [I 2022-10-27 05:38:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 1700/2304 [I 2022-10-27 05:38:30] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:36 [I 2022-10-27 05:38:30] ...trap/engines/engine.py.229: process: 0.54674 | load: 0.00246 [I 2022-10-27 05:38:30] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-27 05:38:30] ...optimizers/trijoint.py.151: total_norm: 10.016657 [I 2022-10-27 05:38:35] ...trap/engines/engine.py.224: train: epoch 50 | batch 1710/2304 [I 2022-10-27 05:38:35] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:31 [I 2022-10-27 05:38:35] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00244 [I 2022-10-27 05:38:35] ...trap/engines/engine.py.230: loss: 0.13183 [I 2022-10-27 05:38:35] ...optimizers/trijoint.py.151: total_norm: 20.746405 [I 2022-10-27 05:38:41] ...trap/engines/engine.py.224: train: epoch 50 | batch 1720/2304 [I 2022-10-27 05:38:41] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:28 [I 2022-10-27 05:38:41] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00245 [I 2022-10-27 05:38:41] ...trap/engines/engine.py.230: loss: 0.19150 [I 2022-10-27 05:38:41] ...optimizers/trijoint.py.151: total_norm: 33.977245 [I 2022-10-27 05:38:47] ...trap/engines/engine.py.224: train: epoch 50 | batch 1730/2304 [I 2022-10-27 05:38:47] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:18 [I 2022-10-27 05:38:47] ...trap/engines/engine.py.229: process: 0.54085 | load: 0.00248 [I 2022-10-27 05:38:47] ...trap/engines/engine.py.230: loss: 0.15932 [I 2022-10-27 05:38:47] ...optimizers/trijoint.py.151: total_norm: 13.338024 [I 2022-10-27 05:38:52] ...trap/engines/engine.py.224: train: epoch 50 | batch 1740/2304 [I 2022-10-27 05:38:52] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:13 [I 2022-10-27 05:38:52] ...trap/engines/engine.py.229: process: 0.55369 | load: 0.00243 [I 2022-10-27 05:38:52] ...trap/engines/engine.py.230: loss: 0.19968 [I 2022-10-27 05:38:52] ...optimizers/trijoint.py.151: total_norm: 17.637590 [I 2022-10-27 05:38:58] ...trap/engines/engine.py.224: train: epoch 50 | batch 1750/2304 [I 2022-10-27 05:38:58] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:07 [I 2022-10-27 05:38:58] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00243 [I 2022-10-27 05:38:58] ...trap/engines/engine.py.230: loss: 0.19620 [I 2022-10-27 05:38:58] ...optimizers/trijoint.py.151: total_norm: 9.224093 [I 2022-10-27 05:39:03] ...trap/engines/engine.py.224: train: epoch 50 | batch 1760/2304 [I 2022-10-27 05:39:03] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:03 [I 2022-10-27 05:39:03] ...trap/engines/engine.py.229: process: 0.54857 | load: 0.00244 [I 2022-10-27 05:39:03] ...trap/engines/engine.py.230: loss: 0.18856 [I 2022-10-27 05:39:03] ...optimizers/trijoint.py.151: total_norm: 20.591690 [I 2022-10-27 05:39:09] ...trap/engines/engine.py.224: train: epoch 50 | batch 1770/2304 [I 2022-10-27 05:39:09] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:04:55 [I 2022-10-27 05:39:09] ...trap/engines/engine.py.229: process: 0.53657 | load: 0.00240 [I 2022-10-27 05:39:09] ...trap/engines/engine.py.230: loss: 0.16384 [I 2022-10-27 05:39:09] ...optimizers/trijoint.py.151: total_norm: 22.549471 [I 2022-10-27 05:39:14] ...trap/engines/engine.py.224: train: epoch 50 | batch 1780/2304 [I 2022-10-27 05:39:14] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:04:51 [I 2022-10-27 05:39:14] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00241 [I 2022-10-27 05:39:14] ...trap/engines/engine.py.230: loss: 0.20240 [I 2022-10-27 05:39:14] ...optimizers/trijoint.py.151: total_norm: 21.942860 [I 2022-10-27 05:39:20] ...trap/engines/engine.py.224: train: epoch 50 | batch 1790/2304 [I 2022-10-27 05:39:20] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:49 [I 2022-10-27 05:39:20] ...trap/engines/engine.py.229: process: 0.56588 | load: 0.00241 [I 2022-10-27 05:39:20] ...trap/engines/engine.py.230: loss: 0.18040 [I 2022-10-27 05:39:20] ...optimizers/trijoint.py.151: total_norm: 23.984587 [I 2022-10-27 05:39:26] ...trap/engines/engine.py.224: train: epoch 50 | batch 1800/2304 [I 2022-10-27 05:39:26] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:40 [I 2022-10-27 05:39:26] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00241 [I 2022-10-27 05:39:26] ...trap/engines/engine.py.230: loss: 0.18881 [I 2022-10-27 05:39:26] ...optimizers/trijoint.py.151: total_norm: 24.265867 [I 2022-10-27 05:39:31] ...trap/engines/engine.py.224: train: epoch 50 | batch 1810/2304 [I 2022-10-27 05:39:31] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:33 [I 2022-10-27 05:39:31] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00242 [I 2022-10-27 05:39:31] ...trap/engines/engine.py.230: loss: 0.15077 [I 2022-10-27 05:39:31] ...optimizers/trijoint.py.151: total_norm: 10.905948 [I 2022-10-27 05:39:37] ...trap/engines/engine.py.224: train: epoch 50 | batch 1820/2304 [I 2022-10-27 05:39:37] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:28 [I 2022-10-27 05:39:37] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00254 [I 2022-10-27 05:39:37] ...trap/engines/engine.py.230: loss: 0.10288 [I 2022-10-27 05:39:37] ...optimizers/trijoint.py.151: total_norm: 9.682730 [I 2022-10-27 05:39:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 1830/2304 [I 2022-10-27 05:39:42] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:23 [I 2022-10-27 05:39:42] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00249 [I 2022-10-27 05:39:42] ...trap/engines/engine.py.230: loss: 0.14545 [I 2022-10-27 05:39:42] ...optimizers/trijoint.py.151: total_norm: 18.236261 [I 2022-10-27 05:39:48] ...trap/engines/engine.py.224: train: epoch 50 | batch 1840/2304 [I 2022-10-27 05:39:48] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:17 [I 2022-10-27 05:39:48] ...trap/engines/engine.py.229: process: 0.56261 | load: 0.00249 [I 2022-10-27 05:39:48] ...trap/engines/engine.py.230: loss: 0.12040 [I 2022-10-27 05:39:48] ...optimizers/trijoint.py.151: total_norm: 16.782837 [I 2022-10-27 05:39:53] ...trap/engines/engine.py.224: train: epoch 50 | batch 1850/2304 [I 2022-10-27 05:39:53] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:10 [I 2022-10-27 05:39:53] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00244 [I 2022-10-27 05:39:53] ...trap/engines/engine.py.230: loss: 0.15745 [I 2022-10-27 05:39:53] ...optimizers/trijoint.py.151: total_norm: 11.350215 [I 2022-10-27 05:39:59] ...trap/engines/engine.py.224: train: epoch 50 | batch 1860/2304 [I 2022-10-27 05:39:59] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:07 [I 2022-10-27 05:39:59] ...trap/engines/engine.py.229: process: 0.56459 | load: 0.00244 [I 2022-10-27 05:39:59] ...trap/engines/engine.py.230: loss: 0.18746 [I 2022-10-27 05:39:59] ...optimizers/trijoint.py.151: total_norm: 20.689941 [I 2022-10-27 05:40:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 1870/2304 [I 2022-10-27 05:40:05] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:08 [I 2022-10-27 05:40:05] ...trap/engines/engine.py.229: process: 0.54892 | load: 0.00245 [I 2022-10-27 05:40:05] ...trap/engines/engine.py.230: loss: 0.16996 [I 2022-10-27 05:40:05] ...optimizers/trijoint.py.151: total_norm: 23.982306 [I 2022-10-27 05:40:10] ...trap/engines/engine.py.224: train: epoch 50 | batch 1880/2304 [I 2022-10-27 05:40:10] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:03:57 [I 2022-10-27 05:40:10] ...trap/engines/engine.py.229: process: 0.55305 | load: 0.00246 [I 2022-10-27 05:40:10] ...trap/engines/engine.py.230: loss: 0.14679 [I 2022-10-27 05:40:10] ...optimizers/trijoint.py.151: total_norm: 15.798258 [I 2022-10-27 05:40:16] ...trap/engines/engine.py.224: train: epoch 50 | batch 1890/2304 [I 2022-10-27 05:40:16] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:53 [I 2022-10-27 05:40:16] ...trap/engines/engine.py.229: process: 0.57056 | load: 0.00251 [I 2022-10-27 05:40:16] ...trap/engines/engine.py.230: loss: 0.25108 [I 2022-10-27 05:40:16] ...optimizers/trijoint.py.151: total_norm: 33.377831 [I 2022-10-27 05:40:22] ...trap/engines/engine.py.224: train: epoch 50 | batch 1900/2304 [I 2022-10-27 05:40:22] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:03:45 [I 2022-10-27 05:40:22] ...trap/engines/engine.py.229: process: 0.55343 | load: 0.00245 [I 2022-10-27 05:40:22] ...trap/engines/engine.py.230: loss: 0.15515 [I 2022-10-27 05:40:22] ...optimizers/trijoint.py.151: total_norm: 17.547007 [I 2022-10-27 05:40:27] ...trap/engines/engine.py.224: train: epoch 50 | batch 1910/2304 [I 2022-10-27 05:40:27] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:40 [I 2022-10-27 05:40:27] ...trap/engines/engine.py.229: process: 0.55968 | load: 0.00265 [I 2022-10-27 05:40:27] ...trap/engines/engine.py.230: loss: 0.14775 [I 2022-10-27 05:40:27] ...optimizers/trijoint.py.151: total_norm: 21.314840 [I 2022-10-27 05:40:33] ...trap/engines/engine.py.224: train: epoch 50 | batch 1920/2304 [I 2022-10-27 05:40:33] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:34 [I 2022-10-27 05:40:33] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00247 [I 2022-10-27 05:40:33] ...trap/engines/engine.py.230: loss: 0.19962 [I 2022-10-27 05:40:33] ...optimizers/trijoint.py.151: total_norm: 37.919971 [I 2022-10-27 05:40:38] ...trap/engines/engine.py.224: train: epoch 50 | batch 1930/2304 [I 2022-10-27 05:40:38] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:32 [I 2022-10-27 05:40:38] ...trap/engines/engine.py.229: process: 0.55058 | load: 0.00265 [I 2022-10-27 05:40:38] ...trap/engines/engine.py.230: loss: 0.10771 [I 2022-10-27 05:40:38] ...optimizers/trijoint.py.151: total_norm: 10.946409 [I 2022-10-27 05:40:44] ...trap/engines/engine.py.224: train: epoch 50 | batch 1940/2304 [I 2022-10-27 05:40:44] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:23 [I 2022-10-27 05:40:44] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00242 [I 2022-10-27 05:40:44] ...trap/engines/engine.py.230: loss: 0.09938 [I 2022-10-27 05:40:44] ...optimizers/trijoint.py.151: total_norm: 17.348068 [I 2022-10-27 05:40:50] ...trap/engines/engine.py.224: train: epoch 50 | batch 1950/2304 [I 2022-10-27 05:40:50] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:17 [I 2022-10-27 05:40:50] ...trap/engines/engine.py.229: process: 0.55474 | load: 0.00244 [I 2022-10-27 05:40:50] ...trap/engines/engine.py.230: loss: 0.18785 [I 2022-10-27 05:40:50] ...optimizers/trijoint.py.151: total_norm: 19.078186 [I 2022-10-27 05:40:55] ...trap/engines/engine.py.224: train: epoch 50 | batch 1960/2304 [I 2022-10-27 05:40:55] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:12 [I 2022-10-27 05:40:55] ...trap/engines/engine.py.229: process: 0.55013 | load: 0.00241 [I 2022-10-27 05:40:55] ...trap/engines/engine.py.230: loss: 0.12621 [I 2022-10-27 05:40:55] ...optimizers/trijoint.py.151: total_norm: 10.683452 [I 2022-10-27 05:41:01] ...trap/engines/engine.py.224: train: epoch 50 | batch 1970/2304 [I 2022-10-27 05:41:01] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:06 [I 2022-10-27 05:41:01] ...trap/engines/engine.py.229: process: 0.56210 | load: 0.00246 [I 2022-10-27 05:41:01] ...trap/engines/engine.py.230: loss: 0.16420 [I 2022-10-27 05:41:01] ...optimizers/trijoint.py.151: total_norm: 18.224609 [I 2022-10-27 05:41:06] ...trap/engines/engine.py.224: train: epoch 50 | batch 1980/2304 [I 2022-10-27 05:41:06] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:00 [I 2022-10-27 05:41:06] ...trap/engines/engine.py.229: process: 0.54528 | load: 0.00245 [I 2022-10-27 05:41:06] ...trap/engines/engine.py.230: loss: 0.14007 [I 2022-10-27 05:41:06] ...optimizers/trijoint.py.151: total_norm: 7.870603 [I 2022-10-27 05:41:12] ...trap/engines/engine.py.224: train: epoch 50 | batch 1990/2304 [I 2022-10-27 05:41:12] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:02:54 [I 2022-10-27 05:41:12] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00251 [I 2022-10-27 05:41:12] ...trap/engines/engine.py.230: loss: 0.20415 [I 2022-10-27 05:41:12] ...optimizers/trijoint.py.151: total_norm: 22.982317 [I 2022-10-27 05:41:18] ...trap/engines/engine.py.224: train: epoch 50 | batch 2000/2304 [I 2022-10-27 05:41:18] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:02:49 [I 2022-10-27 05:41:18] ...trap/engines/engine.py.229: process: 0.55938 | load: 0.00247 [I 2022-10-27 05:41:18] ...trap/engines/engine.py.230: loss: 0.16949 [I 2022-10-27 05:41:18] ...optimizers/trijoint.py.151: total_norm: 19.330431 [I 2022-10-27 05:41:23] ...trap/engines/engine.py.224: train: epoch 50 | batch 2010/2304 [I 2022-10-27 05:41:23] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:43 [I 2022-10-27 05:41:23] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00243 [I 2022-10-27 05:41:23] ...trap/engines/engine.py.230: loss: 0.25223 [I 2022-10-27 05:41:23] ...optimizers/trijoint.py.151: total_norm: 50.460861 [I 2022-10-27 05:41:29] ...trap/engines/engine.py.224: train: epoch 50 | batch 2020/2304 [I 2022-10-27 05:41:29] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:38 [I 2022-10-27 05:41:29] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00243 [I 2022-10-27 05:41:29] ...trap/engines/engine.py.230: loss: 0.13349 [I 2022-10-27 05:41:29] ...optimizers/trijoint.py.151: total_norm: 19.872065 [I 2022-10-27 05:41:34] ...trap/engines/engine.py.224: train: epoch 50 | batch 2030/2304 [I 2022-10-27 05:41:34] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:32 [I 2022-10-27 05:41:34] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00245 [I 2022-10-27 05:41:34] ...trap/engines/engine.py.230: loss: 0.15124 [I 2022-10-27 05:41:34] ...optimizers/trijoint.py.151: total_norm: 12.374045 [I 2022-10-27 05:41:40] ...trap/engines/engine.py.224: train: epoch 50 | batch 2040/2304 [I 2022-10-27 05:41:40] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:27 [I 2022-10-27 05:41:40] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00245 [I 2022-10-27 05:41:40] ...trap/engines/engine.py.230: loss: 0.16189 [I 2022-10-27 05:41:40] ...optimizers/trijoint.py.151: total_norm: 17.755863 [I 2022-10-27 05:41:45] ...trap/engines/engine.py.224: train: epoch 50 | batch 2050/2304 [I 2022-10-27 05:41:45] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:21 [I 2022-10-27 05:41:45] ...trap/engines/engine.py.229: process: 0.55696 | load: 0.00273 [I 2022-10-27 05:41:45] ...trap/engines/engine.py.230: loss: 0.12947 [I 2022-10-27 05:41:45] ...optimizers/trijoint.py.151: total_norm: 22.017908 [I 2022-10-27 05:41:51] ...trap/engines/engine.py.224: train: epoch 50 | batch 2060/2304 [I 2022-10-27 05:41:51] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:16 [I 2022-10-27 05:41:51] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00248 [I 2022-10-27 05:41:51] ...trap/engines/engine.py.230: loss: 0.16269 [I 2022-10-27 05:41:51] ...optimizers/trijoint.py.151: total_norm: 20.911463 [I 2022-10-27 05:41:57] ...trap/engines/engine.py.224: train: epoch 50 | batch 2070/2304 [I 2022-10-27 05:41:57] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:10 [I 2022-10-27 05:41:57] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00246 [I 2022-10-27 05:41:57] ...trap/engines/engine.py.230: loss: 0.19931 [I 2022-10-27 05:41:57] ...optimizers/trijoint.py.151: total_norm: 22.073677 [I 2022-10-27 05:42:02] ...trap/engines/engine.py.224: train: epoch 50 | batch 2080/2304 [I 2022-10-27 05:42:02] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:05 [I 2022-10-27 05:42:02] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00248 [I 2022-10-27 05:42:02] ...trap/engines/engine.py.230: loss: 0.13345 [I 2022-10-27 05:42:02] ...optimizers/trijoint.py.151: total_norm: 10.933619 [I 2022-10-27 05:42:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 2090/2304 [I 2022-10-27 05:42:08] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:01:59 [I 2022-10-27 05:42:08] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00246 [I 2022-10-27 05:42:08] ...trap/engines/engine.py.230: loss: 0.13797 [I 2022-10-27 05:42:08] ...optimizers/trijoint.py.151: total_norm: 25.501326 [I 2022-10-27 05:42:13] ...trap/engines/engine.py.224: train: epoch 50 | batch 2100/2304 [I 2022-10-27 05:42:13] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:01:54 [I 2022-10-27 05:42:13] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00254 [I 2022-10-27 05:42:13] ...trap/engines/engine.py.230: loss: 0.24058 [I 2022-10-27 05:42:13] ...optimizers/trijoint.py.151: total_norm: 15.651779 [I 2022-10-27 05:42:19] ...trap/engines/engine.py.224: train: epoch 50 | batch 2110/2304 [I 2022-10-27 05:42:19] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:48 [I 2022-10-27 05:42:19] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00247 [I 2022-10-27 05:42:19] ...trap/engines/engine.py.230: loss: 0.20784 [I 2022-10-27 05:42:19] ...optimizers/trijoint.py.151: total_norm: 19.358036 [I 2022-10-27 05:42:25] ...trap/engines/engine.py.224: train: epoch 50 | batch 2120/2304 [I 2022-10-27 05:42:25] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:42 [I 2022-10-27 05:42:25] ...trap/engines/engine.py.229: process: 0.56080 | load: 0.00246 [I 2022-10-27 05:42:25] ...trap/engines/engine.py.230: loss: 0.20100 [I 2022-10-27 05:42:25] ...optimizers/trijoint.py.151: total_norm: 19.622507 [I 2022-10-27 05:42:30] ...trap/engines/engine.py.224: train: epoch 50 | batch 2130/2304 [I 2022-10-27 05:42:30] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:37 [I 2022-10-27 05:42:30] ...trap/engines/engine.py.229: process: 0.56258 | load: 0.00250 [I 2022-10-27 05:42:30] ...trap/engines/engine.py.230: loss: 0.17797 [I 2022-10-27 05:42:30] ...optimizers/trijoint.py.151: total_norm: 28.754299 [I 2022-10-27 05:42:36] ...trap/engines/engine.py.224: train: epoch 50 | batch 2140/2304 [I 2022-10-27 05:42:36] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:31 [I 2022-10-27 05:42:36] ...trap/engines/engine.py.229: process: 0.54165 | load: 0.00257 [I 2022-10-27 05:42:36] ...trap/engines/engine.py.230: loss: 0.22887 [I 2022-10-27 05:42:36] ...optimizers/trijoint.py.151: total_norm: 15.943478 [I 2022-10-27 05:42:42] ...trap/engines/engine.py.224: train: epoch 50 | batch 2150/2304 [I 2022-10-27 05:42:42] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:26 [I 2022-10-27 05:42:42] ...trap/engines/engine.py.229: process: 0.55342 | load: 0.00254 [I 2022-10-27 05:42:42] ...trap/engines/engine.py.230: loss: 0.16370 [I 2022-10-27 05:42:42] ...optimizers/trijoint.py.151: total_norm: 12.683961 [I 2022-10-27 05:42:47] ...trap/engines/engine.py.224: train: epoch 50 | batch 2160/2304 [I 2022-10-27 05:42:47] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:20 [I 2022-10-27 05:42:47] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00252 [I 2022-10-27 05:42:47] ...trap/engines/engine.py.230: loss: 0.19122 [I 2022-10-27 05:42:47] ...optimizers/trijoint.py.151: total_norm: 35.540344 [I 2022-10-27 05:42:53] ...trap/engines/engine.py.224: train: epoch 50 | batch 2170/2304 [I 2022-10-27 05:42:53] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:14 [I 2022-10-27 05:42:53] ...trap/engines/engine.py.229: process: 0.56567 | load: 0.00245 [I 2022-10-27 05:42:53] ...trap/engines/engine.py.230: loss: 0.19314 [I 2022-10-27 05:42:53] ...optimizers/trijoint.py.151: total_norm: 17.638868 [I 2022-10-27 05:42:58] ...trap/engines/engine.py.224: train: epoch 50 | batch 2180/2304 [I 2022-10-27 05:42:58] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:09 [I 2022-10-27 05:42:58] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00244 [I 2022-10-27 05:42:58] ...trap/engines/engine.py.230: loss: 0.15305 [I 2022-10-27 05:42:58] ...optimizers/trijoint.py.151: total_norm: 14.746236 [I 2022-10-27 05:43:04] ...trap/engines/engine.py.224: train: epoch 50 | batch 2190/2304 [I 2022-10-27 05:43:04] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:03 [I 2022-10-27 05:43:04] ...trap/engines/engine.py.229: process: 0.55587 | load: 0.00247 [I 2022-10-27 05:43:04] ...trap/engines/engine.py.230: loss: 0.19338 [I 2022-10-27 05:43:04] ...optimizers/trijoint.py.151: total_norm: 10.213940 [I 2022-10-27 05:43:09] ...trap/engines/engine.py.224: train: epoch 50 | batch 2200/2304 [I 2022-10-27 05:43:09] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:00:58 [I 2022-10-27 05:43:09] ...trap/engines/engine.py.229: process: 0.55693 | load: 0.00247 [I 2022-10-27 05:43:09] ...trap/engines/engine.py.230: loss: 0.19361 [I 2022-10-27 05:43:09] ...optimizers/trijoint.py.151: total_norm: 11.776076 [I 2022-10-27 05:43:15] ...trap/engines/engine.py.224: train: epoch 50 | batch 2210/2304 [I 2022-10-27 05:43:15] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:00:52 [I 2022-10-27 05:43:15] ...trap/engines/engine.py.229: process: 0.56496 | load: 0.00252 [I 2022-10-27 05:43:15] ...trap/engines/engine.py.230: loss: 0.17416 [I 2022-10-27 05:43:15] ...optimizers/trijoint.py.151: total_norm: 8.508860 [I 2022-10-27 05:43:21] ...trap/engines/engine.py.224: train: epoch 50 | batch 2220/2304 [I 2022-10-27 05:43:21] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:00:46 [I 2022-10-27 05:43:21] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00244 [I 2022-10-27 05:43:21] ...trap/engines/engine.py.230: loss: 0.16749 [I 2022-10-27 05:43:21] ...optimizers/trijoint.py.151: total_norm: 19.578035 [I 2022-10-27 05:43:26] ...trap/engines/engine.py.224: train: epoch 50 | batch 2230/2304 [I 2022-10-27 05:43:26] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:41 [I 2022-10-27 05:43:26] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00242 [I 2022-10-27 05:43:26] ...trap/engines/engine.py.230: loss: 0.16316 [I 2022-10-27 05:43:26] ...optimizers/trijoint.py.151: total_norm: 16.168016 [I 2022-10-27 05:43:32] ...trap/engines/engine.py.224: train: epoch 50 | batch 2240/2304 [I 2022-10-27 05:43:32] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:35 [I 2022-10-27 05:43:32] ...trap/engines/engine.py.229: process: 0.55143 | load: 0.00344 [I 2022-10-27 05:43:32] ...trap/engines/engine.py.230: loss: 0.13102 [I 2022-10-27 05:43:32] ...optimizers/trijoint.py.151: total_norm: 14.003722 [I 2022-10-27 05:43:37] ...trap/engines/engine.py.224: train: epoch 50 | batch 2250/2304 [I 2022-10-27 05:43:37] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:30 [I 2022-10-27 05:43:37] ...trap/engines/engine.py.229: process: 0.55662 | load: 0.00241 [I 2022-10-27 05:43:37] ...trap/engines/engine.py.230: loss: 0.15482 [I 2022-10-27 05:43:37] ...optimizers/trijoint.py.151: total_norm: 17.792681 [I 2022-10-27 05:43:43] ...trap/engines/engine.py.224: train: epoch 50 | batch 2260/2304 [I 2022-10-27 05:43:43] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:24 [I 2022-10-27 05:43:43] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00242 [I 2022-10-27 05:43:43] ...trap/engines/engine.py.230: loss: 0.14769 [I 2022-10-27 05:43:43] ...optimizers/trijoint.py.151: total_norm: 9.351143 [I 2022-10-27 05:43:49] ...trap/engines/engine.py.224: train: epoch 50 | batch 2270/2304 [I 2022-10-27 05:43:49] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:18 [I 2022-10-27 05:43:49] ...trap/engines/engine.py.229: process: 0.55228 | load: 0.00243 [I 2022-10-27 05:43:49] ...trap/engines/engine.py.230: loss: 0.18585 [I 2022-10-27 05:43:49] ...optimizers/trijoint.py.151: total_norm: 17.648506 [I 2022-10-27 05:43:54] ...trap/engines/engine.py.224: train: epoch 50 | batch 2280/2304 [I 2022-10-27 05:43:54] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:13 [I 2022-10-27 05:43:54] ...trap/engines/engine.py.229: process: 0.56335 | load: 0.00243 [I 2022-10-27 05:43:54] ...trap/engines/engine.py.230: loss: 0.17644 [I 2022-10-27 05:43:54] ...optimizers/trijoint.py.151: total_norm: 14.112066 [I 2022-10-27 05:44:00] ...trap/engines/engine.py.224: train: epoch 50 | batch 2290/2304 [I 2022-10-27 05:44:00] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:07 [I 2022-10-27 05:44:00] ...trap/engines/engine.py.229: process: 0.55242 | load: 0.00252 [I 2022-10-27 05:44:00] ...trap/engines/engine.py.230: loss: 0.18118 [I 2022-10-27 05:44:00] ...optimizers/trijoint.py.151: total_norm: 18.142815 [I 2022-10-27 05:44:05] ...trap/engines/engine.py.224: train: epoch 50 | batch 2300/2304 [I 2022-10-27 05:44:05] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:02 [I 2022-10-27 05:44:05] ...trap/engines/engine.py.229: process: 0.54420 | load: 0.00015 [I 2022-10-27 05:44:05] ...trap/engines/engine.py.230: loss: 0.14152 [I 2022-10-27 05:44:05] ...optimizers/trijoint.py.151: total_norm: 12.966550 [I 2022-10-27 05:44:08] ...trap/engines/engine.py.224: train: epoch 50 | batch 2304/2304 [I 2022-10-27 05:44:08] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:00 [I 2022-10-27 05:44:08] ...trap/engines/engine.py.229: process: 0.55567 | load: 0.00014 [I 2022-10-27 05:44:08] ...trap/engines/engine.py.230: loss: 0.17101 [I 2022-10-27 05:44:08] ...optimizers/trijoint.py.151: total_norm: 13.138660 [S 2022-10-27 05:44:09] ...trap/engines/engine.py.129: train_epoch.epoch: 50 [S 2022-10-27 05:44:09] ...trap/engines/engine.py.129: train_epoch.loss: 0.1649913820747181 [I 2022-10-27 05:44:09] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 05:44:09] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 50 [I 2022-10-27 05:44:09] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 05:44:11] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 05:44:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 05:44:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 05:44:13] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 05:44:13] ...trap/engines/engine.py.321: eval: epoch 50 | batch 0/510 [I 2022-10-27 05:44:13] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:33:41 [I 2022-10-27 05:44:13] ...trap/engines/engine.py.326: process: 3.96456 | load: 3.59388 [I 2022-10-27 05:44:17] ...trap/engines/engine.py.321: eval: epoch 50 | batch 10/510 [I 2022-10-27 05:44:17] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:01 [I 2022-10-27 05:44:17] ...trap/engines/engine.py.326: process: 0.34840 | load: 0.00017 [I 2022-10-27 05:44:20] ...trap/engines/engine.py.321: eval: epoch 50 | batch 20/510 [I 2022-10-27 05:44:20] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:04 [I 2022-10-27 05:44:20] ...trap/engines/engine.py.326: process: 0.32366 | load: 0.00019 [I 2022-10-27 05:44:23] ...trap/engines/engine.py.321: eval: epoch 50 | batch 30/510 [I 2022-10-27 05:44:23] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:45 [I 2022-10-27 05:44:23] ...trap/engines/engine.py.326: process: 0.33060 | load: 0.00017 [I 2022-10-27 05:44:27] ...trap/engines/engine.py.321: eval: epoch 50 | batch 40/510 [I 2022-10-27 05:44:27] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:39 [I 2022-10-27 05:44:27] ...trap/engines/engine.py.326: process: 0.32433 | load: 0.00019 [I 2022-10-27 05:44:30] ...trap/engines/engine.py.321: eval: epoch 50 | batch 50/510 [I 2022-10-27 05:44:30] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:32 [I 2022-10-27 05:44:30] ...trap/engines/engine.py.326: process: 0.32806 | load: 0.00018 [I 2022-10-27 05:44:33] ...trap/engines/engine.py.321: eval: epoch 50 | batch 60/510 [I 2022-10-27 05:44:33] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:30 [I 2022-10-27 05:44:33] ...trap/engines/engine.py.326: process: 0.33197 | load: 0.00017 [I 2022-10-27 05:44:37] ...trap/engines/engine.py.321: eval: epoch 50 | batch 70/510 [I 2022-10-27 05:44:37] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:28 [I 2022-10-27 05:44:37] ...trap/engines/engine.py.326: process: 0.33631 | load: 0.00020 [I 2022-10-27 05:44:40] ...trap/engines/engine.py.321: eval: epoch 50 | batch 80/510 [I 2022-10-27 05:44:40] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:26 [I 2022-10-27 05:44:40] ...trap/engines/engine.py.326: process: 0.35356 | load: 0.00022 [I 2022-10-27 05:44:44] ...trap/engines/engine.py.321: eval: epoch 50 | batch 90/510 [I 2022-10-27 05:44:44] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:21 [I 2022-10-27 05:44:44] ...trap/engines/engine.py.326: process: 0.33026 | load: 0.00018 [I 2022-10-27 05:44:47] ...trap/engines/engine.py.321: eval: epoch 50 | batch 100/510 [I 2022-10-27 05:44:47] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:15 [I 2022-10-27 05:44:47] ...trap/engines/engine.py.326: process: 0.27970 | load: 0.00017 [I 2022-10-27 05:44:50] ...trap/engines/engine.py.321: eval: epoch 50 | batch 110/510 [I 2022-10-27 05:44:50] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:59 [I 2022-10-27 05:44:50] ...trap/engines/engine.py.326: process: 0.31443 | load: 0.00015 [I 2022-10-27 05:44:53] ...trap/engines/engine.py.321: eval: epoch 50 | batch 120/510 [I 2022-10-27 05:44:53] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:53 [I 2022-10-27 05:44:53] ...trap/engines/engine.py.326: process: 0.28918 | load: 0.00018 [I 2022-10-27 05:44:56] ...trap/engines/engine.py.321: eval: epoch 50 | batch 130/510 [I 2022-10-27 05:44:56] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:50 [I 2022-10-27 05:44:56] ...trap/engines/engine.py.326: process: 0.28735 | load: 0.00018 [I 2022-10-27 05:44:59] ...trap/engines/engine.py.321: eval: epoch 50 | batch 140/510 [I 2022-10-27 05:44:59] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:45 [I 2022-10-27 05:44:59] ...trap/engines/engine.py.326: process: 0.28018 | load: 0.00016 [I 2022-10-27 05:45:01] ...trap/engines/engine.py.321: eval: epoch 50 | batch 150/510 [I 2022-10-27 05:45:01] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:43 [I 2022-10-27 05:45:01] ...trap/engines/engine.py.326: process: 0.29336 | load: 0.00024 [I 2022-10-27 05:45:04] ...trap/engines/engine.py.321: eval: epoch 50 | batch 160/510 [I 2022-10-27 05:45:04] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 05:45:04] ...trap/engines/engine.py.326: process: 0.29132 | load: 0.00017 [I 2022-10-27 05:45:07] ...trap/engines/engine.py.321: eval: epoch 50 | batch 170/510 [I 2022-10-27 05:45:07] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-27 05:45:07] ...trap/engines/engine.py.326: process: 0.28858 | load: 0.00020 [I 2022-10-27 05:45:10] ...trap/engines/engine.py.321: eval: epoch 50 | batch 180/510 [I 2022-10-27 05:45:10] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:36 [I 2022-10-27 05:45:10] ...trap/engines/engine.py.326: process: 0.30163 | load: 0.00020 [I 2022-10-27 05:45:13] ...trap/engines/engine.py.321: eval: epoch 50 | batch 190/510 [I 2022-10-27 05:45:13] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:33 [I 2022-10-27 05:45:13] ...trap/engines/engine.py.326: process: 0.29016 | load: 0.00015 [I 2022-10-27 05:45:16] ...trap/engines/engine.py.321: eval: epoch 50 | batch 200/510 [I 2022-10-27 05:45:16] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:30 [I 2022-10-27 05:45:16] ...trap/engines/engine.py.326: process: 0.30255 | load: 0.00022 [I 2022-10-27 05:45:19] ...trap/engines/engine.py.321: eval: epoch 50 | batch 210/510 [I 2022-10-27 05:45:19] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:27 [I 2022-10-27 05:45:19] ...trap/engines/engine.py.326: process: 0.28329 | load: 0.00015 [I 2022-10-27 05:45:22] ...trap/engines/engine.py.321: eval: epoch 50 | batch 220/510 [I 2022-10-27 05:45:22] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:24 [I 2022-10-27 05:45:22] ...trap/engines/engine.py.326: process: 0.28957 | load: 0.00015 [I 2022-10-27 05:45:25] ...trap/engines/engine.py.321: eval: epoch 50 | batch 230/510 [I 2022-10-27 05:45:25] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:21 [I 2022-10-27 05:45:25] ...trap/engines/engine.py.326: process: 0.29964 | load: 0.00016 [I 2022-10-27 05:45:28] ...trap/engines/engine.py.321: eval: epoch 50 | batch 240/510 [I 2022-10-27 05:45:28] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 05:45:28] ...trap/engines/engine.py.326: process: 0.28766 | load: 0.00016 [I 2022-10-27 05:45:31] ...trap/engines/engine.py.321: eval: epoch 50 | batch 250/510 [I 2022-10-27 05:45:31] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 05:45:31] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00016 [I 2022-10-27 05:45:33] ...trap/engines/engine.py.321: eval: epoch 50 | batch 260/510 [I 2022-10-27 05:45:33] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:11 [I 2022-10-27 05:45:33] ...trap/engines/engine.py.326: process: 0.28702 | load: 0.00015 [I 2022-10-27 05:45:36] ...trap/engines/engine.py.321: eval: epoch 50 | batch 270/510 [I 2022-10-27 05:45:36] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 05:45:36] ...trap/engines/engine.py.326: process: 0.28950 | load: 0.00019 [I 2022-10-27 05:45:39] ...trap/engines/engine.py.321: eval: epoch 50 | batch 280/510 [I 2022-10-27 05:45:39] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:07 [I 2022-10-27 05:45:39] ...trap/engines/engine.py.326: process: 0.29190 | load: 0.00015 [I 2022-10-27 05:45:42] ...trap/engines/engine.py.321: eval: epoch 50 | batch 290/510 [I 2022-10-27 05:45:42] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:04 [I 2022-10-27 05:45:42] ...trap/engines/engine.py.326: process: 0.28557 | load: 0.00026 [I 2022-10-27 05:45:45] ...trap/engines/engine.py.321: eval: epoch 50 | batch 300/510 [I 2022-10-27 05:45:45] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 05:45:45] ...trap/engines/engine.py.326: process: 0.28650 | load: 0.00015 [I 2022-10-27 05:45:48] ...trap/engines/engine.py.321: eval: epoch 50 | batch 310/510 [I 2022-10-27 05:45:48] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:58 [I 2022-10-27 05:45:48] ...trap/engines/engine.py.326: process: 0.29301 | load: 0.00025 [I 2022-10-27 05:45:51] ...trap/engines/engine.py.321: eval: epoch 50 | batch 320/510 [I 2022-10-27 05:45:51] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 05:45:51] ...trap/engines/engine.py.326: process: 0.28535 | load: 0.00016 [I 2022-10-27 05:45:54] ...trap/engines/engine.py.321: eval: epoch 50 | batch 330/510 [I 2022-10-27 05:45:54] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:53 [I 2022-10-27 05:45:54] ...trap/engines/engine.py.326: process: 0.28841 | load: 0.00023 [I 2022-10-27 05:45:57] ...trap/engines/engine.py.321: eval: epoch 50 | batch 340/510 [I 2022-10-27 05:45:57] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 05:45:57] ...trap/engines/engine.py.326: process: 0.29373 | load: 0.00019 [I 2022-10-27 05:46:00] ...trap/engines/engine.py.321: eval: epoch 50 | batch 350/510 [I 2022-10-27 05:46:00] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:47 [I 2022-10-27 05:46:00] ...trap/engines/engine.py.326: process: 0.29501 | load: 0.00024 [I 2022-10-27 05:46:03] ...trap/engines/engine.py.321: eval: epoch 50 | batch 360/510 [I 2022-10-27 05:46:03] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:44 [I 2022-10-27 05:46:03] ...trap/engines/engine.py.326: process: 0.29266 | load: 0.00020 [I 2022-10-27 05:46:06] ...trap/engines/engine.py.321: eval: epoch 50 | batch 370/510 [I 2022-10-27 05:46:06] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:41 [I 2022-10-27 05:46:06] ...trap/engines/engine.py.326: process: 0.28417 | load: 0.00015 [I 2022-10-27 05:46:09] ...trap/engines/engine.py.321: eval: epoch 50 | batch 380/510 [I 2022-10-27 05:46:09] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 05:46:09] ...trap/engines/engine.py.326: process: 0.28365 | load: 0.00016 [I 2022-10-27 05:46:12] ...trap/engines/engine.py.321: eval: epoch 50 | batch 390/510 [I 2022-10-27 05:46:12] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 05:46:12] ...trap/engines/engine.py.326: process: 0.29271 | load: 0.00016 [I 2022-10-27 05:46:14] ...trap/engines/engine.py.321: eval: epoch 50 | batch 400/510 [I 2022-10-27 05:46:14] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 05:46:14] ...trap/engines/engine.py.326: process: 0.28760 | load: 0.00016 [I 2022-10-27 05:46:17] ...trap/engines/engine.py.321: eval: epoch 50 | batch 410/510 [I 2022-10-27 05:46:17] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 05:46:17] ...trap/engines/engine.py.326: process: 0.28821 | load: 0.00015 [I 2022-10-27 05:46:20] ...trap/engines/engine.py.321: eval: epoch 50 | batch 420/510 [I 2022-10-27 05:46:20] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:26 [I 2022-10-27 05:46:20] ...trap/engines/engine.py.326: process: 0.28952 | load: 0.00026 [I 2022-10-27 05:46:23] ...trap/engines/engine.py.321: eval: epoch 50 | batch 430/510 [I 2022-10-27 05:46:23] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 05:46:23] ...trap/engines/engine.py.326: process: 0.28703 | load: 0.00016 [I 2022-10-27 05:46:26] ...trap/engines/engine.py.321: eval: epoch 50 | batch 440/510 [I 2022-10-27 05:46:26] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 05:46:26] ...trap/engines/engine.py.326: process: 0.29147 | load: 0.00019 [I 2022-10-27 05:46:29] ...trap/engines/engine.py.321: eval: epoch 50 | batch 450/510 [I 2022-10-27 05:46:29] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 05:46:29] ...trap/engines/engine.py.326: process: 0.28547 | load: 0.00019 [I 2022-10-27 05:46:32] ...trap/engines/engine.py.321: eval: epoch 50 | batch 460/510 [I 2022-10-27 05:46:32] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 05:46:32] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00018 [I 2022-10-27 05:46:35] ...trap/engines/engine.py.321: eval: epoch 50 | batch 470/510 [I 2022-10-27 05:46:35] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 05:46:35] ...trap/engines/engine.py.326: process: 0.29021 | load: 0.00016 [I 2022-10-27 05:46:38] ...trap/engines/engine.py.321: eval: epoch 50 | batch 480/510 [I 2022-10-27 05:46:38] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 05:46:38] ...trap/engines/engine.py.326: process: 0.28818 | load: 0.00021 [I 2022-10-27 05:46:41] ...trap/engines/engine.py.321: eval: epoch 50 | batch 490/510 [I 2022-10-27 05:46:41] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 05:46:41] ...trap/engines/engine.py.326: process: 0.28511 | load: 0.00020 [I 2022-10-27 05:46:44] ...trap/engines/engine.py.321: eval: epoch 50 | batch 500/510 [I 2022-10-27 05:46:44] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 05:46:44] ...trap/engines/engine.py.326: process: 0.28469 | load: 0.00033 [I 2022-10-27 05:46:47] ...trap/engines/engine.py.321: eval: epoch 50 | batch 510/510 [I 2022-10-27 05:46:47] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 05:46:47] ...trap/engines/engine.py.326: process: 0.28651 | load: 0.00019 [S 2022-10-27 05:46:47] ...trap/engines/engine.py.132: eval_epoch.epoch: 50 [I 2022-10-27 05:46:47] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.703500 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.899900 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934900 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.692700 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.893800 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.928100 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013433 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009751 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007063 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008403 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005250 [S 2022-10-27 05:46:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005890 [I 2022-10-27 05:46:52] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 05:46:52] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 05:46:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 05:46:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 05:46:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 05:46:55] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 05:46:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 05:47:03] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 05:47:03] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 05:47:03] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 05:47:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 05:47:12] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 05:47:12] ...trap/engines/engine.py.163: Training model on trainset for epoch 51 [I 2022-10-27 05:47:12] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 05:47:15] ...trap/engines/engine.py.224: train: epoch 51 | batch 0/2304 [I 2022-10-27 05:47:15] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:56 [I 2022-10-27 05:47:15] ...trap/engines/engine.py.229: process: 2.78495 | load: 2.12836 [I 2022-10-27 05:47:15] ...trap/engines/engine.py.230: loss: 0.14852 [I 2022-10-27 05:47:15] ...optimizers/trijoint.py.151: total_norm: 17.984926 [I 2022-10-27 05:47:21] ...trap/engines/engine.py.224: train: epoch 51 | batch 10/2304 [I 2022-10-27 05:47:21] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:53 [I 2022-10-27 05:47:21] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00372 [I 2022-10-27 05:47:21] ...trap/engines/engine.py.230: loss: 0.15128 [I 2022-10-27 05:47:21] ...optimizers/trijoint.py.151: total_norm: 14.317826 [I 2022-10-27 05:47:26] ...trap/engines/engine.py.224: train: epoch 51 | batch 20/2304 [I 2022-10-27 05:47:26] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:20 [I 2022-10-27 05:47:26] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00314 [I 2022-10-27 05:47:26] ...trap/engines/engine.py.230: loss: 0.08920 [I 2022-10-27 05:47:26] ...optimizers/trijoint.py.151: total_norm: 8.556642 [I 2022-10-27 05:47:32] ...trap/engines/engine.py.224: train: epoch 51 | batch 30/2304 [I 2022-10-27 05:47:32] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:12 [I 2022-10-27 05:47:32] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00353 [I 2022-10-27 05:47:32] ...trap/engines/engine.py.230: loss: 0.14863 [I 2022-10-27 05:47:32] ...optimizers/trijoint.py.151: total_norm: 15.385058 [I 2022-10-27 05:47:37] ...trap/engines/engine.py.224: train: epoch 51 | batch 40/2304 [I 2022-10-27 05:47:37] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:01 [I 2022-10-27 05:47:37] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00315 [I 2022-10-27 05:47:37] ...trap/engines/engine.py.230: loss: 0.11317 [I 2022-10-27 05:47:37] ...optimizers/trijoint.py.151: total_norm: 12.265250 [I 2022-10-27 05:47:43] ...trap/engines/engine.py.224: train: epoch 51 | batch 50/2304 [I 2022-10-27 05:47:43] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:53 [I 2022-10-27 05:47:43] ...trap/engines/engine.py.229: process: 0.54953 | load: 0.00290 [I 2022-10-27 05:47:43] ...trap/engines/engine.py.230: loss: 0.14418 [I 2022-10-27 05:47:43] ...optimizers/trijoint.py.151: total_norm: 95.060722 [I 2022-10-27 05:47:49] ...trap/engines/engine.py.224: train: epoch 51 | batch 60/2304 [I 2022-10-27 05:47:49] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:06 [I 2022-10-27 05:47:49] ...trap/engines/engine.py.229: process: 0.58347 | load: 0.00265 [I 2022-10-27 05:47:49] ...trap/engines/engine.py.230: loss: 0.14123 [I 2022-10-27 05:47:49] ...optimizers/trijoint.py.151: total_norm: 8.301886 [I 2022-10-27 05:47:54] ...trap/engines/engine.py.224: train: epoch 51 | batch 70/2304 [I 2022-10-27 05:47:54] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:53 [I 2022-10-27 05:47:54] ...trap/engines/engine.py.229: process: 0.55209 | load: 0.00291 [I 2022-10-27 05:47:54] ...trap/engines/engine.py.230: loss: 0.14956 [I 2022-10-27 05:47:54] ...optimizers/trijoint.py.151: total_norm: 13.671075 [I 2022-10-27 05:48:00] ...trap/engines/engine.py.224: train: epoch 51 | batch 80/2304 [I 2022-10-27 05:48:00] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:21:02 [I 2022-10-27 05:48:00] ...trap/engines/engine.py.229: process: 0.58730 | load: 0.00268 [I 2022-10-27 05:48:00] ...trap/engines/engine.py.230: loss: 0.10548 [I 2022-10-27 05:48:00] ...optimizers/trijoint.py.151: total_norm: 9.075523 [I 2022-10-27 05:48:06] ...trap/engines/engine.py.224: train: epoch 51 | batch 90/2304 [I 2022-10-27 05:48:06] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:42 [I 2022-10-27 05:48:06] ...trap/engines/engine.py.229: process: 0.56660 | load: 0.00270 [I 2022-10-27 05:48:06] ...trap/engines/engine.py.230: loss: 0.15422 [I 2022-10-27 05:48:06] ...optimizers/trijoint.py.151: total_norm: 10.830184 [I 2022-10-27 05:48:11] ...trap/engines/engine.py.224: train: epoch 51 | batch 100/2304 [I 2022-10-27 05:48:11] ...trap/engines/engine.py.228: elapsed: 0:00:59 | left: 0:20:34 [I 2022-10-27 05:48:11] ...trap/engines/engine.py.229: process: 0.56468 | load: 0.00283 [I 2022-10-27 05:48:11] ...trap/engines/engine.py.230: loss: 0.18254 [I 2022-10-27 05:48:11] ...optimizers/trijoint.py.151: total_norm: 21.971973 [I 2022-10-27 05:48:17] ...trap/engines/engine.py.224: train: epoch 51 | batch 110/2304 [I 2022-10-27 05:48:17] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:31 [I 2022-10-27 05:48:17] ...trap/engines/engine.py.229: process: 0.56375 | load: 0.00276 [I 2022-10-27 05:48:17] ...trap/engines/engine.py.230: loss: 0.15766 [I 2022-10-27 05:48:17] ...optimizers/trijoint.py.151: total_norm: 14.839196 [I 2022-10-27 05:48:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 120/2304 [I 2022-10-27 05:48:22] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:35 [I 2022-10-27 05:48:22] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00247 [I 2022-10-27 05:48:22] ...trap/engines/engine.py.230: loss: 0.11682 [I 2022-10-27 05:48:22] ...optimizers/trijoint.py.151: total_norm: 11.068532 [I 2022-10-27 05:48:28] ...trap/engines/engine.py.224: train: epoch 51 | batch 130/2304 [I 2022-10-27 05:48:28] ...trap/engines/engine.py.228: elapsed: 0:01:16 | left: 0:20:25 [I 2022-10-27 05:48:28] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00267 [I 2022-10-27 05:48:28] ...trap/engines/engine.py.230: loss: 0.14668 [I 2022-10-27 05:48:28] ...optimizers/trijoint.py.151: total_norm: 13.993126 [I 2022-10-27 05:48:34] ...trap/engines/engine.py.224: train: epoch 51 | batch 140/2304 [I 2022-10-27 05:48:34] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:03 [I 2022-10-27 05:48:34] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00273 [I 2022-10-27 05:48:34] ...trap/engines/engine.py.230: loss: 0.17116 [I 2022-10-27 05:48:34] ...optimizers/trijoint.py.151: total_norm: 10.041659 [I 2022-10-27 05:48:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 150/2304 [I 2022-10-27 05:48:39] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:20:23 [I 2022-10-27 05:48:39] ...trap/engines/engine.py.229: process: 0.58678 | load: 0.00248 [I 2022-10-27 05:48:39] ...trap/engines/engine.py.230: loss: 0.18263 [I 2022-10-27 05:48:39] ...optimizers/trijoint.py.151: total_norm: 27.629757 [I 2022-10-27 05:48:45] ...trap/engines/engine.py.224: train: epoch 51 | batch 160/2304 [I 2022-10-27 05:48:45] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:05 [I 2022-10-27 05:48:45] ...trap/engines/engine.py.229: process: 0.56490 | load: 0.00250 [I 2022-10-27 05:48:45] ...trap/engines/engine.py.230: loss: 0.20711 [I 2022-10-27 05:48:45] ...optimizers/trijoint.py.151: total_norm: 23.262272 [I 2022-10-27 05:48:51] ...trap/engines/engine.py.224: train: epoch 51 | batch 170/2304 [I 2022-10-27 05:48:51] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:53 [I 2022-10-27 05:48:51] ...trap/engines/engine.py.229: process: 0.54958 | load: 0.00278 [I 2022-10-27 05:48:51] ...trap/engines/engine.py.230: loss: 0.20436 [I 2022-10-27 05:48:51] ...optimizers/trijoint.py.151: total_norm: 16.560959 [I 2022-10-27 05:48:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 180/2304 [I 2022-10-27 05:48:56] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:19:41 [I 2022-10-27 05:48:56] ...trap/engines/engine.py.229: process: 0.56215 | load: 0.00263 [I 2022-10-27 05:48:56] ...trap/engines/engine.py.230: loss: 0.14680 [I 2022-10-27 05:48:56] ...optimizers/trijoint.py.151: total_norm: 29.415476 [I 2022-10-27 05:49:02] ...trap/engines/engine.py.224: train: epoch 51 | batch 190/2304 [I 2022-10-27 05:49:02] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:39 [I 2022-10-27 05:49:02] ...trap/engines/engine.py.229: process: 0.55257 | load: 0.00249 [I 2022-10-27 05:49:02] ...trap/engines/engine.py.230: loss: 0.20150 [I 2022-10-27 05:49:02] ...optimizers/trijoint.py.151: total_norm: 22.747231 [I 2022-10-27 05:49:07] ...trap/engines/engine.py.224: train: epoch 51 | batch 200/2304 [I 2022-10-27 05:49:07] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:35 [I 2022-10-27 05:49:07] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00263 [I 2022-10-27 05:49:07] ...trap/engines/engine.py.230: loss: 0.13993 [I 2022-10-27 05:49:07] ...optimizers/trijoint.py.151: total_norm: 11.324037 [I 2022-10-27 05:49:13] ...trap/engines/engine.py.224: train: epoch 51 | batch 210/2304 [I 2022-10-27 05:49:13] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:31 [I 2022-10-27 05:49:13] ...trap/engines/engine.py.229: process: 0.54541 | load: 0.00248 [I 2022-10-27 05:49:13] ...trap/engines/engine.py.230: loss: 0.10505 [I 2022-10-27 05:49:13] ...optimizers/trijoint.py.151: total_norm: 13.204535 [I 2022-10-27 05:49:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 220/2304 [I 2022-10-27 05:49:19] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:25 [I 2022-10-27 05:49:19] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00253 [I 2022-10-27 05:49:19] ...trap/engines/engine.py.230: loss: 0.17089 [I 2022-10-27 05:49:19] ...optimizers/trijoint.py.151: total_norm: 13.987307 [I 2022-10-27 05:49:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 230/2304 [I 2022-10-27 05:49:24] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:19:21 [I 2022-10-27 05:49:24] ...trap/engines/engine.py.229: process: 0.55193 | load: 0.00248 [I 2022-10-27 05:49:24] ...trap/engines/engine.py.230: loss: 0.14402 [I 2022-10-27 05:49:24] ...optimizers/trijoint.py.151: total_norm: 9.810560 [I 2022-10-27 05:49:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 240/2304 [I 2022-10-27 05:49:30] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:21 [I 2022-10-27 05:49:30] ...trap/engines/engine.py.229: process: 0.56725 | load: 0.00249 [I 2022-10-27 05:49:30] ...trap/engines/engine.py.230: loss: 0.11627 [I 2022-10-27 05:49:30] ...optimizers/trijoint.py.151: total_norm: 8.725601 [I 2022-10-27 05:49:35] ...trap/engines/engine.py.224: train: epoch 51 | batch 250/2304 [I 2022-10-27 05:49:35] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:13 [I 2022-10-27 05:49:35] ...trap/engines/engine.py.229: process: 0.56823 | load: 0.00250 [I 2022-10-27 05:49:35] ...trap/engines/engine.py.230: loss: 0.17343 [I 2022-10-27 05:49:35] ...optimizers/trijoint.py.151: total_norm: 14.304892 [I 2022-10-27 05:49:41] ...trap/engines/engine.py.224: train: epoch 51 | batch 260/2304 [I 2022-10-27 05:49:41] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:19:13 [I 2022-10-27 05:49:41] ...trap/engines/engine.py.229: process: 0.55676 | load: 0.00260 [I 2022-10-27 05:49:41] ...trap/engines/engine.py.230: loss: 0.12872 [I 2022-10-27 05:49:41] ...optimizers/trijoint.py.151: total_norm: 14.341029 [I 2022-10-27 05:49:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 270/2304 [I 2022-10-27 05:49:47] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:19:02 [I 2022-10-27 05:49:47] ...trap/engines/engine.py.229: process: 0.56309 | load: 0.00249 [I 2022-10-27 05:49:47] ...trap/engines/engine.py.230: loss: 0.14349 [I 2022-10-27 05:49:47] ...optimizers/trijoint.py.151: total_norm: 17.168257 [I 2022-10-27 05:49:52] ...trap/engines/engine.py.224: train: epoch 51 | batch 280/2304 [I 2022-10-27 05:49:52] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:18:53 [I 2022-10-27 05:49:52] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00244 [I 2022-10-27 05:49:52] ...trap/engines/engine.py.230: loss: 0.12687 [I 2022-10-27 05:49:52] ...optimizers/trijoint.py.151: total_norm: 14.734282 [I 2022-10-27 05:49:58] ...trap/engines/engine.py.224: train: epoch 51 | batch 290/2304 [I 2022-10-27 05:49:58] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:52 [I 2022-10-27 05:49:58] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00248 [I 2022-10-27 05:49:58] ...trap/engines/engine.py.230: loss: 0.11767 [I 2022-10-27 05:49:58] ...optimizers/trijoint.py.151: total_norm: 7.494445 [I 2022-10-27 05:50:04] ...trap/engines/engine.py.224: train: epoch 51 | batch 300/2304 [I 2022-10-27 05:50:04] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:41 [I 2022-10-27 05:50:04] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00252 [I 2022-10-27 05:50:04] ...trap/engines/engine.py.230: loss: 0.20624 [I 2022-10-27 05:50:04] ...optimizers/trijoint.py.151: total_norm: 22.012699 [I 2022-10-27 05:50:09] ...trap/engines/engine.py.224: train: epoch 51 | batch 310/2304 [I 2022-10-27 05:50:09] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:18:39 [I 2022-10-27 05:50:09] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00247 [I 2022-10-27 05:50:09] ...trap/engines/engine.py.230: loss: 0.13351 [I 2022-10-27 05:50:09] ...optimizers/trijoint.py.151: total_norm: 21.773758 [I 2022-10-27 05:50:15] ...trap/engines/engine.py.224: train: epoch 51 | batch 320/2304 [I 2022-10-27 05:50:15] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:23 [I 2022-10-27 05:50:15] ...trap/engines/engine.py.229: process: 0.54742 | load: 0.00259 [I 2022-10-27 05:50:15] ...trap/engines/engine.py.230: loss: 0.16521 [I 2022-10-27 05:50:15] ...optimizers/trijoint.py.151: total_norm: 18.981895 [I 2022-10-27 05:50:20] ...trap/engines/engine.py.224: train: epoch 51 | batch 330/2304 [I 2022-10-27 05:50:20] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:18:37 [I 2022-10-27 05:50:20] ...trap/engines/engine.py.229: process: 0.57372 | load: 0.00252 [I 2022-10-27 05:50:20] ...trap/engines/engine.py.230: loss: 0.12402 [I 2022-10-27 05:50:20] ...optimizers/trijoint.py.151: total_norm: 17.537136 [I 2022-10-27 05:50:26] ...trap/engines/engine.py.224: train: epoch 51 | batch 340/2304 [I 2022-10-27 05:50:26] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:27 [I 2022-10-27 05:50:26] ...trap/engines/engine.py.229: process: 0.57462 | load: 0.00244 [I 2022-10-27 05:50:26] ...trap/engines/engine.py.230: loss: 0.13001 [I 2022-10-27 05:50:26] ...optimizers/trijoint.py.151: total_norm: 26.539661 [I 2022-10-27 05:50:32] ...trap/engines/engine.py.224: train: epoch 51 | batch 350/2304 [I 2022-10-27 05:50:32] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:44 [I 2022-10-27 05:50:32] ...trap/engines/engine.py.229: process: 0.63895 | load: 0.00247 [I 2022-10-27 05:50:32] ...trap/engines/engine.py.230: loss: 0.22060 [I 2022-10-27 05:50:32] ...optimizers/trijoint.py.151: total_norm: 29.883184 [I 2022-10-27 05:50:37] ...trap/engines/engine.py.224: train: epoch 51 | batch 360/2304 [I 2022-10-27 05:50:37] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:18:26 [I 2022-10-27 05:50:37] ...trap/engines/engine.py.229: process: 0.56623 | load: 0.00251 [I 2022-10-27 05:50:37] ...trap/engines/engine.py.230: loss: 0.15073 [I 2022-10-27 05:50:37] ...optimizers/trijoint.py.151: total_norm: 12.181643 [I 2022-10-27 05:50:43] ...trap/engines/engine.py.224: train: epoch 51 | batch 370/2304 [I 2022-10-27 05:50:43] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:03 [I 2022-10-27 05:50:43] ...trap/engines/engine.py.229: process: 0.55694 | load: 0.00247 [I 2022-10-27 05:50:43] ...trap/engines/engine.py.230: loss: 0.12560 [I 2022-10-27 05:50:43] ...optimizers/trijoint.py.151: total_norm: 11.477717 [I 2022-10-27 05:50:49] ...trap/engines/engine.py.224: train: epoch 51 | batch 380/2304 [I 2022-10-27 05:50:49] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:17:58 [I 2022-10-27 05:50:49] ...trap/engines/engine.py.229: process: 0.56593 | load: 0.00249 [I 2022-10-27 05:50:49] ...trap/engines/engine.py.230: loss: 0.10581 [I 2022-10-27 05:50:49] ...optimizers/trijoint.py.151: total_norm: 12.269348 [I 2022-10-27 05:50:54] ...trap/engines/engine.py.224: train: epoch 51 | batch 390/2304 [I 2022-10-27 05:50:54] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:17:50 [I 2022-10-27 05:50:54] ...trap/engines/engine.py.229: process: 0.56210 | load: 0.00265 [I 2022-10-27 05:50:54] ...trap/engines/engine.py.230: loss: 0.13840 [I 2022-10-27 05:50:54] ...optimizers/trijoint.py.151: total_norm: 9.090575 [I 2022-10-27 05:51:00] ...trap/engines/engine.py.224: train: epoch 51 | batch 400/2304 [I 2022-10-27 05:51:00] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:37 [I 2022-10-27 05:51:00] ...trap/engines/engine.py.229: process: 0.54886 | load: 0.00263 [I 2022-10-27 05:51:00] ...trap/engines/engine.py.230: loss: 0.15052 [I 2022-10-27 05:51:00] ...optimizers/trijoint.py.151: total_norm: 16.137825 [I 2022-10-27 05:51:05] ...trap/engines/engine.py.224: train: epoch 51 | batch 410/2304 [I 2022-10-27 05:51:05] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:17:35 [I 2022-10-27 05:51:05] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00245 [I 2022-10-27 05:51:05] ...trap/engines/engine.py.230: loss: 0.16534 [I 2022-10-27 05:51:05] ...optimizers/trijoint.py.151: total_norm: 18.375597 [I 2022-10-27 05:51:11] ...trap/engines/engine.py.224: train: epoch 51 | batch 420/2304 [I 2022-10-27 05:51:11] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:31 [I 2022-10-27 05:51:11] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00245 [I 2022-10-27 05:51:11] ...trap/engines/engine.py.230: loss: 0.17760 [I 2022-10-27 05:51:11] ...optimizers/trijoint.py.151: total_norm: 26.676600 [I 2022-10-27 05:51:17] ...trap/engines/engine.py.224: train: epoch 51 | batch 430/2304 [I 2022-10-27 05:51:17] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:31 [I 2022-10-27 05:51:17] ...trap/engines/engine.py.229: process: 0.56841 | load: 0.00245 [I 2022-10-27 05:51:17] ...trap/engines/engine.py.230: loss: 0.17720 [I 2022-10-27 05:51:17] ...optimizers/trijoint.py.151: total_norm: 29.276779 [I 2022-10-27 05:51:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 440/2304 [I 2022-10-27 05:51:22] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:17:26 [I 2022-10-27 05:51:22] ...trap/engines/engine.py.229: process: 0.56711 | load: 0.00248 [I 2022-10-27 05:51:22] ...trap/engines/engine.py.230: loss: 0.15569 [I 2022-10-27 05:51:22] ...optimizers/trijoint.py.151: total_norm: 15.090934 [I 2022-10-27 05:51:28] ...trap/engines/engine.py.224: train: epoch 51 | batch 450/2304 [I 2022-10-27 05:51:28] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:18 [I 2022-10-27 05:51:28] ...trap/engines/engine.py.229: process: 0.55033 | load: 0.00245 [I 2022-10-27 05:51:28] ...trap/engines/engine.py.230: loss: 0.19535 [I 2022-10-27 05:51:28] ...optimizers/trijoint.py.151: total_norm: 12.009468 [I 2022-10-27 05:51:33] ...trap/engines/engine.py.224: train: epoch 51 | batch 460/2304 [I 2022-10-27 05:51:33] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:16:58 [I 2022-10-27 05:51:33] ...trap/engines/engine.py.229: process: 0.54690 | load: 0.00247 [I 2022-10-27 05:51:33] ...trap/engines/engine.py.230: loss: 0.15801 [I 2022-10-27 05:51:33] ...optimizers/trijoint.py.151: total_norm: 12.665017 [I 2022-10-27 05:51:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 470/2304 [I 2022-10-27 05:51:39] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:16:58 [I 2022-10-27 05:51:39] ...trap/engines/engine.py.229: process: 0.54848 | load: 0.00240 [I 2022-10-27 05:51:39] ...trap/engines/engine.py.230: loss: 0.21148 [I 2022-10-27 05:51:39] ...optimizers/trijoint.py.151: total_norm: 18.393990 [I 2022-10-27 05:51:45] ...trap/engines/engine.py.224: train: epoch 51 | batch 480/2304 [I 2022-10-27 05:51:45] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:17:03 [I 2022-10-27 05:51:45] ...trap/engines/engine.py.229: process: 0.56282 | load: 0.00247 [I 2022-10-27 05:51:45] ...trap/engines/engine.py.230: loss: 0.12758 [I 2022-10-27 05:51:45] ...optimizers/trijoint.py.151: total_norm: 10.352041 [I 2022-10-27 05:51:50] ...trap/engines/engine.py.224: train: epoch 51 | batch 490/2304 [I 2022-10-27 05:51:50] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:16:58 [I 2022-10-27 05:51:50] ...trap/engines/engine.py.229: process: 0.56556 | load: 0.00244 [I 2022-10-27 05:51:50] ...trap/engines/engine.py.230: loss: 0.12768 [I 2022-10-27 05:51:50] ...optimizers/trijoint.py.151: total_norm: 15.251980 [I 2022-10-27 05:51:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 500/2304 [I 2022-10-27 05:51:56] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:52 [I 2022-10-27 05:51:56] ...trap/engines/engine.py.229: process: 0.56989 | load: 0.00245 [I 2022-10-27 05:51:56] ...trap/engines/engine.py.230: loss: 0.18255 [I 2022-10-27 05:51:56] ...optimizers/trijoint.py.151: total_norm: 17.906208 [I 2022-10-27 05:52:01] ...trap/engines/engine.py.224: train: epoch 51 | batch 510/2304 [I 2022-10-27 05:52:01] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:51 [I 2022-10-27 05:52:01] ...trap/engines/engine.py.229: process: 0.58198 | load: 0.00252 [I 2022-10-27 05:52:01] ...trap/engines/engine.py.230: loss: 0.18036 [I 2022-10-27 05:52:01] ...optimizers/trijoint.py.151: total_norm: 10.342044 [I 2022-10-27 05:52:07] ...trap/engines/engine.py.224: train: epoch 51 | batch 520/2304 [I 2022-10-27 05:52:07] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:39 [I 2022-10-27 05:52:07] ...trap/engines/engine.py.229: process: 0.56764 | load: 0.00241 [I 2022-10-27 05:52:07] ...trap/engines/engine.py.230: loss: 0.12728 [I 2022-10-27 05:52:07] ...optimizers/trijoint.py.151: total_norm: 14.884630 [I 2022-10-27 05:52:12] ...trap/engines/engine.py.224: train: epoch 51 | batch 530/2304 [I 2022-10-27 05:52:12] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:25 [I 2022-10-27 05:52:12] ...trap/engines/engine.py.229: process: 0.54324 | load: 0.00251 [I 2022-10-27 05:52:12] ...trap/engines/engine.py.230: loss: 0.18755 [I 2022-10-27 05:52:12] ...optimizers/trijoint.py.151: total_norm: 15.374168 [I 2022-10-27 05:52:18] ...trap/engines/engine.py.224: train: epoch 51 | batch 540/2304 [I 2022-10-27 05:52:18] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:16:25 [I 2022-10-27 05:52:18] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00250 [I 2022-10-27 05:52:18] ...trap/engines/engine.py.230: loss: 0.12923 [I 2022-10-27 05:52:18] ...optimizers/trijoint.py.151: total_norm: 28.499914 [I 2022-10-27 05:52:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 550/2304 [I 2022-10-27 05:52:24] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:18 [I 2022-10-27 05:52:24] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00249 [I 2022-10-27 05:52:24] ...trap/engines/engine.py.230: loss: 0.19980 [I 2022-10-27 05:52:24] ...optimizers/trijoint.py.151: total_norm: 19.121029 [I 2022-10-27 05:52:29] ...trap/engines/engine.py.224: train: epoch 51 | batch 560/2304 [I 2022-10-27 05:52:29] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:09 [I 2022-10-27 05:52:29] ...trap/engines/engine.py.229: process: 0.56420 | load: 0.00243 [I 2022-10-27 05:52:29] ...trap/engines/engine.py.230: loss: 0.18439 [I 2022-10-27 05:52:29] ...optimizers/trijoint.py.151: total_norm: 12.870963 [I 2022-10-27 05:52:35] ...trap/engines/engine.py.224: train: epoch 51 | batch 570/2304 [I 2022-10-27 05:52:35] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:09 [I 2022-10-27 05:52:35] ...trap/engines/engine.py.229: process: 0.56902 | load: 0.00239 [I 2022-10-27 05:52:35] ...trap/engines/engine.py.230: loss: 0.13299 [I 2022-10-27 05:52:35] ...optimizers/trijoint.py.151: total_norm: 12.249777 [I 2022-10-27 05:52:40] ...trap/engines/engine.py.224: train: epoch 51 | batch 580/2304 [I 2022-10-27 05:52:40] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:15:59 [I 2022-10-27 05:52:40] ...trap/engines/engine.py.229: process: 0.56036 | load: 0.00241 [I 2022-10-27 05:52:40] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-27 05:52:40] ...optimizers/trijoint.py.151: total_norm: 19.921125 [I 2022-10-27 05:52:46] ...trap/engines/engine.py.224: train: epoch 51 | batch 590/2304 [I 2022-10-27 05:52:46] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:55 [I 2022-10-27 05:52:46] ...trap/engines/engine.py.229: process: 0.55274 | load: 0.00242 [I 2022-10-27 05:52:46] ...trap/engines/engine.py.230: loss: 0.13721 [I 2022-10-27 05:52:46] ...optimizers/trijoint.py.151: total_norm: 13.233988 [I 2022-10-27 05:52:52] ...trap/engines/engine.py.224: train: epoch 51 | batch 600/2304 [I 2022-10-27 05:52:52] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:54 [I 2022-10-27 05:52:52] ...trap/engines/engine.py.229: process: 0.56465 | load: 0.00255 [I 2022-10-27 05:52:52] ...trap/engines/engine.py.230: loss: 0.09698 [I 2022-10-27 05:52:52] ...optimizers/trijoint.py.151: total_norm: 9.924128 [I 2022-10-27 05:52:57] ...trap/engines/engine.py.224: train: epoch 51 | batch 610/2304 [I 2022-10-27 05:52:57] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:15:37 [I 2022-10-27 05:52:57] ...trap/engines/engine.py.229: process: 0.54884 | load: 0.00248 [I 2022-10-27 05:52:57] ...trap/engines/engine.py.230: loss: 0.19520 [I 2022-10-27 05:52:57] ...optimizers/trijoint.py.151: total_norm: 20.743376 [I 2022-10-27 05:53:03] ...trap/engines/engine.py.224: train: epoch 51 | batch 620/2304 [I 2022-10-27 05:53:03] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:36 [I 2022-10-27 05:53:03] ...trap/engines/engine.py.229: process: 0.56541 | load: 0.00246 [I 2022-10-27 05:53:03] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-27 05:53:03] ...optimizers/trijoint.py.151: total_norm: 19.084055 [I 2022-10-27 05:53:08] ...trap/engines/engine.py.224: train: epoch 51 | batch 630/2304 [I 2022-10-27 05:53:08] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:35 [I 2022-10-27 05:53:08] ...trap/engines/engine.py.229: process: 0.56115 | load: 0.00245 [I 2022-10-27 05:53:08] ...trap/engines/engine.py.230: loss: 0.15169 [I 2022-10-27 05:53:08] ...optimizers/trijoint.py.151: total_norm: 13.557111 [I 2022-10-27 05:53:14] ...trap/engines/engine.py.224: train: epoch 51 | batch 640/2304 [I 2022-10-27 05:53:14] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:25 [I 2022-10-27 05:53:14] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00244 [I 2022-10-27 05:53:14] ...trap/engines/engine.py.230: loss: 0.17843 [I 2022-10-27 05:53:14] ...optimizers/trijoint.py.151: total_norm: 13.304688 [I 2022-10-27 05:53:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 650/2304 [I 2022-10-27 05:53:19] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:25 [I 2022-10-27 05:53:19] ...trap/engines/engine.py.229: process: 0.55888 | load: 0.00244 [I 2022-10-27 05:53:19] ...trap/engines/engine.py.230: loss: 0.11707 [I 2022-10-27 05:53:19] ...optimizers/trijoint.py.151: total_norm: 10.718732 [I 2022-10-27 05:53:25] ...trap/engines/engine.py.224: train: epoch 51 | batch 660/2304 [I 2022-10-27 05:53:25] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:15:22 [I 2022-10-27 05:53:25] ...trap/engines/engine.py.229: process: 0.56328 | load: 0.00244 [I 2022-10-27 05:53:25] ...trap/engines/engine.py.230: loss: 0.10128 [I 2022-10-27 05:53:25] ...optimizers/trijoint.py.151: total_norm: 8.823228 [I 2022-10-27 05:53:31] ...trap/engines/engine.py.224: train: epoch 51 | batch 670/2304 [I 2022-10-27 05:53:31] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:05 [I 2022-10-27 05:53:31] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00241 [I 2022-10-27 05:53:31] ...trap/engines/engine.py.230: loss: 0.19202 [I 2022-10-27 05:53:31] ...optimizers/trijoint.py.151: total_norm: 23.021219 [I 2022-10-27 05:53:36] ...trap/engines/engine.py.224: train: epoch 51 | batch 680/2304 [I 2022-10-27 05:53:36] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:14:53 [I 2022-10-27 05:53:36] ...trap/engines/engine.py.229: process: 0.54206 | load: 0.00242 [I 2022-10-27 05:53:36] ...trap/engines/engine.py.230: loss: 0.14997 [I 2022-10-27 05:53:36] ...optimizers/trijoint.py.151: total_norm: 7.660795 [I 2022-10-27 05:53:42] ...trap/engines/engine.py.224: train: epoch 51 | batch 690/2304 [I 2022-10-27 05:53:42] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:14:55 [I 2022-10-27 05:53:42] ...trap/engines/engine.py.229: process: 0.56491 | load: 0.00241 [I 2022-10-27 05:53:42] ...trap/engines/engine.py.230: loss: 0.19801 [I 2022-10-27 05:53:42] ...optimizers/trijoint.py.151: total_norm: 16.470793 [I 2022-10-27 05:53:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 700/2304 [I 2022-10-27 05:53:47] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:14:54 [I 2022-10-27 05:53:47] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00240 [I 2022-10-27 05:53:47] ...trap/engines/engine.py.230: loss: 0.10652 [I 2022-10-27 05:53:47] ...optimizers/trijoint.py.151: total_norm: 8.787455 [I 2022-10-27 05:53:53] ...trap/engines/engine.py.224: train: epoch 51 | batch 710/2304 [I 2022-10-27 05:53:53] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:45 [I 2022-10-27 05:53:53] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00255 [I 2022-10-27 05:53:53] ...trap/engines/engine.py.230: loss: 0.19580 [I 2022-10-27 05:53:53] ...optimizers/trijoint.py.151: total_norm: 21.056408 [I 2022-10-27 05:54:18] ...trap/engines/engine.py.224: train: epoch 51 | batch 720/2304 [I 2022-10-27 05:54:18] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 1:37:05 [I 2022-10-27 05:54:18] ...trap/engines/engine.py.229: process: 0.56184 | load: 0.00239 [I 2022-10-27 05:54:18] ...trap/engines/engine.py.230: loss: 0.17256 [I 2022-10-27 05:54:18] ...optimizers/trijoint.py.151: total_norm: 12.811170 [I 2022-10-27 05:54:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 730/2304 [I 2022-10-27 05:54:24] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:23:27 [I 2022-10-27 05:54:24] ...trap/engines/engine.py.229: process: 0.56412 | load: 0.00256 [I 2022-10-27 05:54:24] ...trap/engines/engine.py.230: loss: 0.16052 [I 2022-10-27 05:54:24] ...optimizers/trijoint.py.151: total_norm: 18.913685 [I 2022-10-27 05:54:29] ...trap/engines/engine.py.224: train: epoch 51 | batch 740/2304 [I 2022-10-27 05:54:29] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:15:23 [I 2022-10-27 05:54:29] ...trap/engines/engine.py.229: process: 0.54897 | load: 0.00251 [I 2022-10-27 05:54:29] ...trap/engines/engine.py.230: loss: 0.21314 [I 2022-10-27 05:54:29] ...optimizers/trijoint.py.151: total_norm: 53.558033 [I 2022-10-27 05:54:35] ...trap/engines/engine.py.224: train: epoch 51 | batch 750/2304 [I 2022-10-27 05:54:35] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:22 [I 2022-10-27 05:54:35] ...trap/engines/engine.py.229: process: 0.54573 | load: 0.00250 [I 2022-10-27 05:54:35] ...trap/engines/engine.py.230: loss: 0.10045 [I 2022-10-27 05:54:35] ...optimizers/trijoint.py.151: total_norm: 20.198391 [I 2022-10-27 05:54:40] ...trap/engines/engine.py.224: train: epoch 51 | batch 760/2304 [I 2022-10-27 05:54:40] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:15 [I 2022-10-27 05:54:40] ...trap/engines/engine.py.229: process: 0.55672 | load: 0.00254 [I 2022-10-27 05:54:40] ...trap/engines/engine.py.230: loss: 0.20767 [I 2022-10-27 05:54:40] ...optimizers/trijoint.py.151: total_norm: 17.895006 [I 2022-10-27 05:54:46] ...trap/engines/engine.py.224: train: epoch 51 | batch 770/2304 [I 2022-10-27 05:54:46] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:14:10 [I 2022-10-27 05:54:46] ...trap/engines/engine.py.229: process: 0.55594 | load: 0.00254 [I 2022-10-27 05:54:46] ...trap/engines/engine.py.230: loss: 0.17634 [I 2022-10-27 05:54:46] ...optimizers/trijoint.py.151: total_norm: 25.113104 [I 2022-10-27 05:54:51] ...trap/engines/engine.py.224: train: epoch 51 | batch 780/2304 [I 2022-10-27 05:54:51] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:04 [I 2022-10-27 05:54:51] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00268 [I 2022-10-27 05:54:51] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-27 05:54:51] ...optimizers/trijoint.py.151: total_norm: 17.945158 [I 2022-10-27 05:54:57] ...trap/engines/engine.py.224: train: epoch 51 | batch 790/2304 [I 2022-10-27 05:54:57] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:14:02 [I 2022-10-27 05:54:57] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00248 [I 2022-10-27 05:54:57] ...trap/engines/engine.py.230: loss: 0.26605 [I 2022-10-27 05:54:57] ...optimizers/trijoint.py.151: total_norm: 18.286404 [I 2022-10-27 05:55:03] ...trap/engines/engine.py.224: train: epoch 51 | batch 800/2304 [I 2022-10-27 05:55:03] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:13:58 [I 2022-10-27 05:55:03] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00252 [I 2022-10-27 05:55:03] ...trap/engines/engine.py.230: loss: 0.14839 [I 2022-10-27 05:55:03] ...optimizers/trijoint.py.151: total_norm: 17.280050 [I 2022-10-27 05:55:08] ...trap/engines/engine.py.224: train: epoch 51 | batch 810/2304 [I 2022-10-27 05:55:08] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:51 [I 2022-10-27 05:55:08] ...trap/engines/engine.py.229: process: 0.55496 | load: 0.00247 [I 2022-10-27 05:55:08] ...trap/engines/engine.py.230: loss: 0.16711 [I 2022-10-27 05:55:08] ...optimizers/trijoint.py.151: total_norm: 18.948355 [I 2022-10-27 05:55:14] ...trap/engines/engine.py.224: train: epoch 51 | batch 820/2304 [I 2022-10-27 05:55:14] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:13:44 [I 2022-10-27 05:55:14] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00251 [I 2022-10-27 05:55:14] ...trap/engines/engine.py.230: loss: 0.16407 [I 2022-10-27 05:55:14] ...optimizers/trijoint.py.151: total_norm: 14.191171 [I 2022-10-27 05:55:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 830/2304 [I 2022-10-27 05:55:19] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:41 [I 2022-10-27 05:55:19] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00250 [I 2022-10-27 05:55:19] ...trap/engines/engine.py.230: loss: 0.16687 [I 2022-10-27 05:55:19] ...optimizers/trijoint.py.151: total_norm: 15.420749 [I 2022-10-27 05:55:25] ...trap/engines/engine.py.224: train: epoch 51 | batch 840/2304 [I 2022-10-27 05:55:25] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:13:34 [I 2022-10-27 05:55:25] ...trap/engines/engine.py.229: process: 0.55895 | load: 0.00251 [I 2022-10-27 05:55:25] ...trap/engines/engine.py.230: loss: 0.14225 [I 2022-10-27 05:55:25] ...optimizers/trijoint.py.151: total_norm: 9.591413 [I 2022-10-27 05:55:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 850/2304 [I 2022-10-27 05:55:30] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:25 [I 2022-10-27 05:55:30] ...trap/engines/engine.py.229: process: 0.55155 | load: 0.00253 [I 2022-10-27 05:55:30] ...trap/engines/engine.py.230: loss: 0.19174 [I 2022-10-27 05:55:30] ...optimizers/trijoint.py.151: total_norm: 23.203012 [I 2022-10-27 05:55:36] ...trap/engines/engine.py.224: train: epoch 51 | batch 860/2304 [I 2022-10-27 05:55:36] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:13:20 [I 2022-10-27 05:55:36] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00249 [I 2022-10-27 05:55:36] ...trap/engines/engine.py.230: loss: 0.15866 [I 2022-10-27 05:55:36] ...optimizers/trijoint.py.151: total_norm: 15.736050 [I 2022-10-27 05:55:41] ...trap/engines/engine.py.224: train: epoch 51 | batch 870/2304 [I 2022-10-27 05:55:41] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:13:12 [I 2022-10-27 05:55:41] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00248 [I 2022-10-27 05:55:41] ...trap/engines/engine.py.230: loss: 0.16515 [I 2022-10-27 05:55:41] ...optimizers/trijoint.py.151: total_norm: 32.052402 [I 2022-10-27 05:55:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 880/2304 [I 2022-10-27 05:55:47] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:29 [I 2022-10-27 05:55:47] ...trap/engines/engine.py.229: process: 0.55979 | load: 0.00252 [I 2022-10-27 05:55:47] ...trap/engines/engine.py.230: loss: 0.12840 [I 2022-10-27 05:55:47] ...optimizers/trijoint.py.151: total_norm: 32.466492 [I 2022-10-27 05:55:53] ...trap/engines/engine.py.224: train: epoch 51 | batch 890/2304 [I 2022-10-27 05:55:53] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:13:08 [I 2022-10-27 05:55:53] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00249 [I 2022-10-27 05:55:53] ...trap/engines/engine.py.230: loss: 0.09674 [I 2022-10-27 05:55:53] ...optimizers/trijoint.py.151: total_norm: 6.975206 [I 2022-10-27 05:55:58] ...trap/engines/engine.py.224: train: epoch 51 | batch 900/2304 [I 2022-10-27 05:55:58] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:13:06 [I 2022-10-27 05:55:58] ...trap/engines/engine.py.229: process: 0.56903 | load: 0.00249 [I 2022-10-27 05:55:58] ...trap/engines/engine.py.230: loss: 0.15660 [I 2022-10-27 05:55:58] ...optimizers/trijoint.py.151: total_norm: 21.870729 [I 2022-10-27 05:56:04] ...trap/engines/engine.py.224: train: epoch 51 | batch 910/2304 [I 2022-10-27 05:56:04] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:12:55 [I 2022-10-27 05:56:04] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00257 [I 2022-10-27 05:56:04] ...trap/engines/engine.py.230: loss: 0.13066 [I 2022-10-27 05:56:04] ...optimizers/trijoint.py.151: total_norm: 11.113618 [I 2022-10-27 05:56:09] ...trap/engines/engine.py.224: train: epoch 51 | batch 920/2304 [I 2022-10-27 05:56:09] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:47 [I 2022-10-27 05:56:09] ...trap/engines/engine.py.229: process: 0.54882 | load: 0.00256 [I 2022-10-27 05:56:09] ...trap/engines/engine.py.230: loss: 0.13354 [I 2022-10-27 05:56:09] ...optimizers/trijoint.py.151: total_norm: 16.411472 [I 2022-10-27 05:56:15] ...trap/engines/engine.py.224: train: epoch 51 | batch 930/2304 [I 2022-10-27 05:56:15] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:12:41 [I 2022-10-27 05:56:15] ...trap/engines/engine.py.229: process: 0.54751 | load: 0.00251 [I 2022-10-27 05:56:15] ...trap/engines/engine.py.230: loss: 0.25321 [I 2022-10-27 05:56:15] ...optimizers/trijoint.py.151: total_norm: 18.525612 [I 2022-10-27 05:56:21] ...trap/engines/engine.py.224: train: epoch 51 | batch 940/2304 [I 2022-10-27 05:56:21] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:12:43 [I 2022-10-27 05:56:21] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00254 [I 2022-10-27 05:56:21] ...trap/engines/engine.py.230: loss: 0.21236 [I 2022-10-27 05:56:21] ...optimizers/trijoint.py.151: total_norm: 19.708130 [I 2022-10-27 05:56:26] ...trap/engines/engine.py.224: train: epoch 51 | batch 950/2304 [I 2022-10-27 05:56:26] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:31 [I 2022-10-27 05:56:26] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00245 [I 2022-10-27 05:56:26] ...trap/engines/engine.py.230: loss: 0.13653 [I 2022-10-27 05:56:26] ...optimizers/trijoint.py.151: total_norm: 10.044029 [I 2022-10-27 05:56:32] ...trap/engines/engine.py.224: train: epoch 51 | batch 960/2304 [I 2022-10-27 05:56:32] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:12:23 [I 2022-10-27 05:56:32] ...trap/engines/engine.py.229: process: 0.54858 | load: 0.00248 [I 2022-10-27 05:56:32] ...trap/engines/engine.py.230: loss: 0.12047 [I 2022-10-27 05:56:32] ...optimizers/trijoint.py.151: total_norm: 10.788618 [I 2022-10-27 05:56:38] ...trap/engines/engine.py.224: train: epoch 51 | batch 970/2304 [I 2022-10-27 05:56:38] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:12:29 [I 2022-10-27 05:56:38] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00250 [I 2022-10-27 05:56:38] ...trap/engines/engine.py.230: loss: 0.13212 [I 2022-10-27 05:56:38] ...optimizers/trijoint.py.151: total_norm: 13.324834 [I 2022-10-27 05:56:43] ...trap/engines/engine.py.224: train: epoch 51 | batch 980/2304 [I 2022-10-27 05:56:43] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:12 [I 2022-10-27 05:56:43] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00253 [I 2022-10-27 05:56:43] ...trap/engines/engine.py.230: loss: 0.19553 [I 2022-10-27 05:56:43] ...optimizers/trijoint.py.151: total_norm: 16.448135 [I 2022-10-27 05:56:49] ...trap/engines/engine.py.224: train: epoch 51 | batch 990/2304 [I 2022-10-27 05:56:49] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:12:15 [I 2022-10-27 05:56:49] ...trap/engines/engine.py.229: process: 0.56359 | load: 0.00250 [I 2022-10-27 05:56:49] ...trap/engines/engine.py.230: loss: 0.17098 [I 2022-10-27 05:56:49] ...optimizers/trijoint.py.151: total_norm: 22.817669 [I 2022-10-27 05:56:54] ...trap/engines/engine.py.224: train: epoch 51 | batch 1000/2304 [I 2022-10-27 05:56:54] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:04 [I 2022-10-27 05:56:54] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00251 [I 2022-10-27 05:56:54] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-27 05:56:54] ...optimizers/trijoint.py.151: total_norm: 19.058538 [I 2022-10-27 05:57:00] ...trap/engines/engine.py.224: train: epoch 51 | batch 1010/2304 [I 2022-10-27 05:57:00] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:11:54 [I 2022-10-27 05:57:00] ...trap/engines/engine.py.229: process: 0.54769 | load: 0.00251 [I 2022-10-27 05:57:00] ...trap/engines/engine.py.230: loss: 0.13838 [I 2022-10-27 05:57:00] ...optimizers/trijoint.py.151: total_norm: 14.044434 [I 2022-10-27 05:57:05] ...trap/engines/engine.py.224: train: epoch 51 | batch 1020/2304 [I 2022-10-27 05:57:05] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:54 [I 2022-10-27 05:57:05] ...trap/engines/engine.py.229: process: 0.56004 | load: 0.00247 [I 2022-10-27 05:57:05] ...trap/engines/engine.py.230: loss: 0.16067 [I 2022-10-27 05:57:05] ...optimizers/trijoint.py.151: total_norm: 18.039612 [I 2022-10-27 05:57:11] ...trap/engines/engine.py.224: train: epoch 51 | batch 1030/2304 [I 2022-10-27 05:57:11] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:12:21 [I 2022-10-27 05:57:11] ...trap/engines/engine.py.229: process: 0.65318 | load: 0.00275 [I 2022-10-27 05:57:11] ...trap/engines/engine.py.230: loss: 0.16990 [I 2022-10-27 05:57:11] ...optimizers/trijoint.py.151: total_norm: 13.468649 [I 2022-10-27 05:57:17] ...trap/engines/engine.py.224: train: epoch 51 | batch 1040/2304 [I 2022-10-27 05:57:17] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:46 [I 2022-10-27 05:57:17] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00247 [I 2022-10-27 05:57:17] ...trap/engines/engine.py.230: loss: 0.19303 [I 2022-10-27 05:57:17] ...optimizers/trijoint.py.151: total_norm: 15.558277 [I 2022-10-27 05:57:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 1050/2304 [I 2022-10-27 05:57:22] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:36 [I 2022-10-27 05:57:22] ...trap/engines/engine.py.229: process: 0.55458 | load: 0.00251 [I 2022-10-27 05:57:22] ...trap/engines/engine.py.230: loss: 0.15480 [I 2022-10-27 05:57:22] ...optimizers/trijoint.py.151: total_norm: 11.031509 [I 2022-10-27 05:57:28] ...trap/engines/engine.py.224: train: epoch 51 | batch 1060/2304 [I 2022-10-27 05:57:28] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:31 [I 2022-10-27 05:57:28] ...trap/engines/engine.py.229: process: 0.55602 | load: 0.00274 [I 2022-10-27 05:57:28] ...trap/engines/engine.py.230: loss: 0.20444 [I 2022-10-27 05:57:28] ...optimizers/trijoint.py.151: total_norm: 24.456553 [I 2022-10-27 05:57:33] ...trap/engines/engine.py.224: train: epoch 51 | batch 1070/2304 [I 2022-10-27 05:57:33] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:24 [I 2022-10-27 05:57:33] ...trap/engines/engine.py.229: process: 0.56370 | load: 0.00249 [I 2022-10-27 05:57:33] ...trap/engines/engine.py.230: loss: 0.16771 [I 2022-10-27 05:57:33] ...optimizers/trijoint.py.151: total_norm: 11.964339 [I 2022-10-27 05:57:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 1080/2304 [I 2022-10-27 05:57:39] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:16 [I 2022-10-27 05:57:39] ...trap/engines/engine.py.229: process: 0.55079 | load: 0.00264 [I 2022-10-27 05:57:39] ...trap/engines/engine.py.230: loss: 0.13186 [I 2022-10-27 05:57:39] ...optimizers/trijoint.py.151: total_norm: 17.903484 [I 2022-10-27 05:57:44] ...trap/engines/engine.py.224: train: epoch 51 | batch 1090/2304 [I 2022-10-27 05:57:44] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:15 [I 2022-10-27 05:57:44] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00255 [I 2022-10-27 05:57:44] ...trap/engines/engine.py.230: loss: 0.25963 [I 2022-10-27 05:57:44] ...optimizers/trijoint.py.151: total_norm: 23.889137 [I 2022-10-27 05:57:50] ...trap/engines/engine.py.224: train: epoch 51 | batch 1100/2304 [I 2022-10-27 05:57:50] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:14 [I 2022-10-27 05:57:50] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00253 [I 2022-10-27 05:57:50] ...trap/engines/engine.py.230: loss: 0.16296 [I 2022-10-27 05:57:50] ...optimizers/trijoint.py.151: total_norm: 10.804322 [I 2022-10-27 05:57:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 1110/2304 [I 2022-10-27 05:57:56] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:11:04 [I 2022-10-27 05:57:56] ...trap/engines/engine.py.229: process: 0.55945 | load: 0.00253 [I 2022-10-27 05:57:56] ...trap/engines/engine.py.230: loss: 0.19102 [I 2022-10-27 05:57:56] ...optimizers/trijoint.py.151: total_norm: 22.802557 [I 2022-10-27 05:58:01] ...trap/engines/engine.py.224: train: epoch 51 | batch 1120/2304 [I 2022-10-27 05:58:01] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:10:54 [I 2022-10-27 05:58:01] ...trap/engines/engine.py.229: process: 0.54813 | load: 0.00247 [I 2022-10-27 05:58:01] ...trap/engines/engine.py.230: loss: 0.16388 [I 2022-10-27 05:58:01] ...optimizers/trijoint.py.151: total_norm: 18.334591 [I 2022-10-27 05:58:07] ...trap/engines/engine.py.224: train: epoch 51 | batch 1130/2304 [I 2022-10-27 05:58:07] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:50 [I 2022-10-27 05:58:07] ...trap/engines/engine.py.229: process: 0.55532 | load: 0.00264 [I 2022-10-27 05:58:07] ...trap/engines/engine.py.230: loss: 0.14649 [I 2022-10-27 05:58:07] ...optimizers/trijoint.py.151: total_norm: 15.695535 [I 2022-10-27 05:58:12] ...trap/engines/engine.py.224: train: epoch 51 | batch 1140/2304 [I 2022-10-27 05:58:12] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:49 [I 2022-10-27 05:58:12] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00258 [I 2022-10-27 05:58:12] ...trap/engines/engine.py.230: loss: 0.17786 [I 2022-10-27 05:58:12] ...optimizers/trijoint.py.151: total_norm: 17.383007 [I 2022-10-27 05:58:18] ...trap/engines/engine.py.224: train: epoch 51 | batch 1150/2304 [I 2022-10-27 05:58:18] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:38 [I 2022-10-27 05:58:18] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00256 [I 2022-10-27 05:58:18] ...trap/engines/engine.py.230: loss: 0.20108 [I 2022-10-27 05:58:18] ...optimizers/trijoint.py.151: total_norm: 25.147713 [I 2022-10-27 05:58:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 1160/2304 [I 2022-10-27 05:58:24] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:49 [I 2022-10-27 05:58:24] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00252 [I 2022-10-27 05:58:24] ...trap/engines/engine.py.230: loss: 0.16442 [I 2022-10-27 05:58:24] ...optimizers/trijoint.py.151: total_norm: 37.424942 [I 2022-10-27 05:58:29] ...trap/engines/engine.py.224: train: epoch 51 | batch 1170/2304 [I 2022-10-27 05:58:29] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:34 [I 2022-10-27 05:58:29] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00270 [I 2022-10-27 05:58:29] ...trap/engines/engine.py.230: loss: 0.14730 [I 2022-10-27 05:58:29] ...optimizers/trijoint.py.151: total_norm: 16.692225 [I 2022-10-27 05:58:35] ...trap/engines/engine.py.224: train: epoch 51 | batch 1180/2304 [I 2022-10-27 05:58:35] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:29 [I 2022-10-27 05:58:35] ...trap/engines/engine.py.229: process: 0.56137 | load: 0.00263 [I 2022-10-27 05:58:35] ...trap/engines/engine.py.230: loss: 0.15469 [I 2022-10-27 05:58:35] ...optimizers/trijoint.py.151: total_norm: 18.615248 [I 2022-10-27 05:58:40] ...trap/engines/engine.py.224: train: epoch 51 | batch 1190/2304 [I 2022-10-27 05:58:40] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:12 [I 2022-10-27 05:58:40] ...trap/engines/engine.py.229: process: 0.54400 | load: 0.00285 [I 2022-10-27 05:58:40] ...trap/engines/engine.py.230: loss: 0.14590 [I 2022-10-27 05:58:40] ...optimizers/trijoint.py.151: total_norm: 14.049029 [I 2022-10-27 05:58:46] ...trap/engines/engine.py.224: train: epoch 51 | batch 1200/2304 [I 2022-10-27 05:58:46] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:10 [I 2022-10-27 05:58:46] ...trap/engines/engine.py.229: process: 0.55013 | load: 0.00264 [I 2022-10-27 05:58:46] ...trap/engines/engine.py.230: loss: 0.21335 [I 2022-10-27 05:58:46] ...optimizers/trijoint.py.151: total_norm: 16.102602 [I 2022-10-27 05:58:51] ...trap/engines/engine.py.224: train: epoch 51 | batch 1210/2304 [I 2022-10-27 05:58:51] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:10:05 [I 2022-10-27 05:58:51] ...trap/engines/engine.py.229: process: 0.54703 | load: 0.00258 [I 2022-10-27 05:58:51] ...trap/engines/engine.py.230: loss: 0.29844 [I 2022-10-27 05:58:51] ...optimizers/trijoint.py.151: total_norm: 21.141096 [I 2022-10-27 05:58:57] ...trap/engines/engine.py.224: train: epoch 51 | batch 1220/2304 [I 2022-10-27 05:58:57] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:10:16 [I 2022-10-27 05:58:57] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00260 [I 2022-10-27 05:58:57] ...trap/engines/engine.py.230: loss: 0.18927 [I 2022-10-27 05:58:57] ...optimizers/trijoint.py.151: total_norm: 17.241373 [I 2022-10-27 05:59:03] ...trap/engines/engine.py.224: train: epoch 51 | batch 1230/2304 [I 2022-10-27 05:59:03] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:10:00 [I 2022-10-27 05:59:03] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00262 [I 2022-10-27 05:59:03] ...trap/engines/engine.py.230: loss: 0.12710 [I 2022-10-27 05:59:03] ...optimizers/trijoint.py.151: total_norm: 10.832169 [I 2022-10-27 05:59:08] ...trap/engines/engine.py.224: train: epoch 51 | batch 1240/2304 [I 2022-10-27 05:59:08] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:50 [I 2022-10-27 05:59:08] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00258 [I 2022-10-27 05:59:08] ...trap/engines/engine.py.230: loss: 0.13377 [I 2022-10-27 05:59:08] ...optimizers/trijoint.py.151: total_norm: 7.671384 [I 2022-10-27 05:59:14] ...trap/engines/engine.py.224: train: epoch 51 | batch 1250/2304 [I 2022-10-27 05:59:14] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:44 [I 2022-10-27 05:59:14] ...trap/engines/engine.py.229: process: 0.54912 | load: 0.00256 [I 2022-10-27 05:59:14] ...trap/engines/engine.py.230: loss: 0.18206 [I 2022-10-27 05:59:14] ...optimizers/trijoint.py.151: total_norm: 16.963316 [I 2022-10-27 05:59:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 1260/2304 [I 2022-10-27 05:59:19] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:41 [I 2022-10-27 05:59:19] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00259 [I 2022-10-27 05:59:19] ...trap/engines/engine.py.230: loss: 0.24120 [I 2022-10-27 05:59:19] ...optimizers/trijoint.py.151: total_norm: 40.955925 [I 2022-10-27 05:59:25] ...trap/engines/engine.py.224: train: epoch 51 | batch 1270/2304 [I 2022-10-27 05:59:25] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:31 [I 2022-10-27 05:59:25] ...trap/engines/engine.py.229: process: 0.54595 | load: 0.00261 [I 2022-10-27 05:59:25] ...trap/engines/engine.py.230: loss: 0.17669 [I 2022-10-27 05:59:25] ...optimizers/trijoint.py.151: total_norm: 24.411396 [I 2022-10-27 05:59:31] ...trap/engines/engine.py.224: train: epoch 51 | batch 1280/2304 [I 2022-10-27 05:59:31] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:27 [I 2022-10-27 05:59:31] ...trap/engines/engine.py.229: process: 0.56344 | load: 0.00255 [I 2022-10-27 05:59:31] ...trap/engines/engine.py.230: loss: 0.11910 [I 2022-10-27 05:59:31] ...optimizers/trijoint.py.151: total_norm: 13.400563 [I 2022-10-27 05:59:36] ...trap/engines/engine.py.224: train: epoch 51 | batch 1290/2304 [I 2022-10-27 05:59:36] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:27 [I 2022-10-27 05:59:36] ...trap/engines/engine.py.229: process: 0.54598 | load: 0.00254 [I 2022-10-27 05:59:36] ...trap/engines/engine.py.230: loss: 0.12397 [I 2022-10-27 05:59:36] ...optimizers/trijoint.py.151: total_norm: 14.719548 [I 2022-10-27 05:59:42] ...trap/engines/engine.py.224: train: epoch 51 | batch 1300/2304 [I 2022-10-27 05:59:42] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:20 [I 2022-10-27 05:59:42] ...trap/engines/engine.py.229: process: 0.56306 | load: 0.00255 [I 2022-10-27 05:59:42] ...trap/engines/engine.py.230: loss: 0.17308 [I 2022-10-27 05:59:42] ...optimizers/trijoint.py.151: total_norm: 19.359331 [I 2022-10-27 05:59:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 1310/2304 [I 2022-10-27 05:59:47] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:14 [I 2022-10-27 05:59:47] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00250 [I 2022-10-27 05:59:47] ...trap/engines/engine.py.230: loss: 0.14129 [I 2022-10-27 05:59:47] ...optimizers/trijoint.py.151: total_norm: 20.235577 [I 2022-10-27 05:59:53] ...trap/engines/engine.py.224: train: epoch 51 | batch 1320/2304 [I 2022-10-27 05:59:53] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:08 [I 2022-10-27 05:59:53] ...trap/engines/engine.py.229: process: 0.56985 | load: 0.00289 [I 2022-10-27 05:59:53] ...trap/engines/engine.py.230: loss: 0.12654 [I 2022-10-27 05:59:53] ...optimizers/trijoint.py.151: total_norm: 25.375103 [I 2022-10-27 05:59:58] ...trap/engines/engine.py.224: train: epoch 51 | batch 1330/2304 [I 2022-10-27 05:59:58] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:08:58 [I 2022-10-27 05:59:58] ...trap/engines/engine.py.229: process: 0.54183 | load: 0.00280 [I 2022-10-27 05:59:58] ...trap/engines/engine.py.230: loss: 0.12989 [I 2022-10-27 05:59:58] ...optimizers/trijoint.py.151: total_norm: 7.537788 [I 2022-10-27 06:00:04] ...trap/engines/engine.py.224: train: epoch 51 | batch 1340/2304 [I 2022-10-27 06:00:04] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:58 [I 2022-10-27 06:00:04] ...trap/engines/engine.py.229: process: 0.56505 | load: 0.00264 [I 2022-10-27 06:00:04] ...trap/engines/engine.py.230: loss: 0.17442 [I 2022-10-27 06:00:04] ...optimizers/trijoint.py.151: total_norm: 17.724503 [I 2022-10-27 06:00:10] ...trap/engines/engine.py.224: train: epoch 51 | batch 1350/2304 [I 2022-10-27 06:00:10] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:09:00 [I 2022-10-27 06:00:10] ...trap/engines/engine.py.229: process: 0.56526 | load: 0.00262 [I 2022-10-27 06:00:10] ...trap/engines/engine.py.230: loss: 0.21567 [I 2022-10-27 06:00:10] ...optimizers/trijoint.py.151: total_norm: 35.264610 [I 2022-10-27 06:00:15] ...trap/engines/engine.py.224: train: epoch 51 | batch 1360/2304 [I 2022-10-27 06:00:15] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:46 [I 2022-10-27 06:00:15] ...trap/engines/engine.py.229: process: 0.55688 | load: 0.00256 [I 2022-10-27 06:00:15] ...trap/engines/engine.py.230: loss: 0.16440 [I 2022-10-27 06:00:15] ...optimizers/trijoint.py.151: total_norm: 11.835131 [I 2022-10-27 06:00:21] ...trap/engines/engine.py.224: train: epoch 51 | batch 1370/2304 [I 2022-10-27 06:00:21] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:37 [I 2022-10-27 06:00:21] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00254 [I 2022-10-27 06:00:21] ...trap/engines/engine.py.230: loss: 0.13492 [I 2022-10-27 06:00:21] ...optimizers/trijoint.py.151: total_norm: 15.041372 [I 2022-10-27 06:00:26] ...trap/engines/engine.py.224: train: epoch 51 | batch 1380/2304 [I 2022-10-27 06:00:26] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:33 [I 2022-10-27 06:00:26] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00250 [I 2022-10-27 06:00:26] ...trap/engines/engine.py.230: loss: 0.12175 [I 2022-10-27 06:00:26] ...optimizers/trijoint.py.151: total_norm: 15.398518 [I 2022-10-27 06:00:32] ...trap/engines/engine.py.224: train: epoch 51 | batch 1390/2304 [I 2022-10-27 06:00:32] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:24 [I 2022-10-27 06:00:32] ...trap/engines/engine.py.229: process: 0.55102 | load: 0.00257 [I 2022-10-27 06:00:32] ...trap/engines/engine.py.230: loss: 0.19366 [I 2022-10-27 06:00:32] ...optimizers/trijoint.py.151: total_norm: 23.583235 [I 2022-10-27 06:00:38] ...trap/engines/engine.py.224: train: epoch 51 | batch 1400/2304 [I 2022-10-27 06:00:38] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:19 [I 2022-10-27 06:00:38] ...trap/engines/engine.py.229: process: 0.54778 | load: 0.00261 [I 2022-10-27 06:00:38] ...trap/engines/engine.py.230: loss: 0.23354 [I 2022-10-27 06:00:38] ...optimizers/trijoint.py.151: total_norm: 18.565344 [I 2022-10-27 06:00:43] ...trap/engines/engine.py.224: train: epoch 51 | batch 1410/2304 [I 2022-10-27 06:00:43] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:33 [I 2022-10-27 06:00:43] ...trap/engines/engine.py.229: process: 0.66425 | load: 0.00258 [I 2022-10-27 06:00:43] ...trap/engines/engine.py.230: loss: 0.18785 [I 2022-10-27 06:00:43] ...optimizers/trijoint.py.151: total_norm: 22.554182 [I 2022-10-27 06:00:49] ...trap/engines/engine.py.224: train: epoch 51 | batch 1420/2304 [I 2022-10-27 06:00:49] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:15 [I 2022-10-27 06:00:49] ...trap/engines/engine.py.229: process: 0.56923 | load: 0.00249 [I 2022-10-27 06:00:49] ...trap/engines/engine.py.230: loss: 0.15766 [I 2022-10-27 06:00:49] ...optimizers/trijoint.py.151: total_norm: 25.500893 [I 2022-10-27 06:00:54] ...trap/engines/engine.py.224: train: epoch 51 | batch 1430/2304 [I 2022-10-27 06:00:54] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:08:08 [I 2022-10-27 06:00:54] ...trap/engines/engine.py.229: process: 0.56720 | load: 0.00269 [I 2022-10-27 06:00:54] ...trap/engines/engine.py.230: loss: 0.18860 [I 2022-10-27 06:00:54] ...optimizers/trijoint.py.151: total_norm: 34.649921 [I 2022-10-27 06:01:00] ...trap/engines/engine.py.224: train: epoch 51 | batch 1440/2304 [I 2022-10-27 06:01:00] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:08:00 [I 2022-10-27 06:01:00] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00255 [I 2022-10-27 06:01:00] ...trap/engines/engine.py.230: loss: 0.13616 [I 2022-10-27 06:01:00] ...optimizers/trijoint.py.151: total_norm: 15.132449 [I 2022-10-27 06:01:06] ...trap/engines/engine.py.224: train: epoch 51 | batch 1450/2304 [I 2022-10-27 06:01:06] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:55 [I 2022-10-27 06:01:06] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00253 [I 2022-10-27 06:01:06] ...trap/engines/engine.py.230: loss: 0.16570 [I 2022-10-27 06:01:06] ...optimizers/trijoint.py.151: total_norm: 12.683097 [I 2022-10-27 06:01:11] ...trap/engines/engine.py.224: train: epoch 51 | batch 1460/2304 [I 2022-10-27 06:01:11] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:47 [I 2022-10-27 06:01:11] ...trap/engines/engine.py.229: process: 0.55003 | load: 0.00266 [I 2022-10-27 06:01:11] ...trap/engines/engine.py.230: loss: 0.20531 [I 2022-10-27 06:01:11] ...optimizers/trijoint.py.151: total_norm: 17.709867 [I 2022-10-27 06:01:17] ...trap/engines/engine.py.224: train: epoch 51 | batch 1470/2304 [I 2022-10-27 06:01:17] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:43 [I 2022-10-27 06:01:17] ...trap/engines/engine.py.229: process: 0.55954 | load: 0.00258 [I 2022-10-27 06:01:17] ...trap/engines/engine.py.230: loss: 0.19283 [I 2022-10-27 06:01:17] ...optimizers/trijoint.py.151: total_norm: 21.198296 [I 2022-10-27 06:01:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 1480/2304 [I 2022-10-27 06:01:22] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:38 [I 2022-10-27 06:01:22] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00259 [I 2022-10-27 06:01:22] ...trap/engines/engine.py.230: loss: 0.14606 [I 2022-10-27 06:01:22] ...optimizers/trijoint.py.151: total_norm: 16.075447 [I 2022-10-27 06:01:28] ...trap/engines/engine.py.224: train: epoch 51 | batch 1490/2304 [I 2022-10-27 06:01:28] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:31 [I 2022-10-27 06:01:28] ...trap/engines/engine.py.229: process: 0.55063 | load: 0.00257 [I 2022-10-27 06:01:28] ...trap/engines/engine.py.230: loss: 0.14498 [I 2022-10-27 06:01:28] ...optimizers/trijoint.py.151: total_norm: 8.975272 [I 2022-10-27 06:01:33] ...trap/engines/engine.py.224: train: epoch 51 | batch 1500/2304 [I 2022-10-27 06:01:33] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:27 [I 2022-10-27 06:01:33] ...trap/engines/engine.py.229: process: 0.56303 | load: 0.00277 [I 2022-10-27 06:01:33] ...trap/engines/engine.py.230: loss: 0.12592 [I 2022-10-27 06:01:33] ...optimizers/trijoint.py.151: total_norm: 10.773322 [I 2022-10-27 06:01:39] ...trap/engines/engine.py.224: train: epoch 51 | batch 1510/2304 [I 2022-10-27 06:01:39] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:22 [I 2022-10-27 06:01:39] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00257 [I 2022-10-27 06:01:39] ...trap/engines/engine.py.230: loss: 0.13330 [I 2022-10-27 06:01:39] ...optimizers/trijoint.py.151: total_norm: 17.537792 [I 2022-10-27 06:01:45] ...trap/engines/engine.py.224: train: epoch 51 | batch 1520/2304 [I 2022-10-27 06:01:45] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:13 [I 2022-10-27 06:01:45] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00253 [I 2022-10-27 06:01:45] ...trap/engines/engine.py.230: loss: 0.18297 [I 2022-10-27 06:01:45] ...optimizers/trijoint.py.151: total_norm: 31.784502 [I 2022-10-27 06:01:50] ...trap/engines/engine.py.224: train: epoch 51 | batch 1530/2304 [I 2022-10-27 06:01:50] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:07:06 [I 2022-10-27 06:01:50] ...trap/engines/engine.py.229: process: 0.54674 | load: 0.00253 [I 2022-10-27 06:01:50] ...trap/engines/engine.py.230: loss: 0.16850 [I 2022-10-27 06:01:50] ...optimizers/trijoint.py.151: total_norm: 29.363949 [I 2022-10-27 06:01:56] ...trap/engines/engine.py.224: train: epoch 51 | batch 1540/2304 [I 2022-10-27 06:01:56] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:07:06 [I 2022-10-27 06:01:56] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00254 [I 2022-10-27 06:01:56] ...trap/engines/engine.py.230: loss: 0.20839 [I 2022-10-27 06:01:56] ...optimizers/trijoint.py.151: total_norm: 18.746988 [I 2022-10-27 06:02:01] ...trap/engines/engine.py.224: train: epoch 51 | batch 1550/2304 [I 2022-10-27 06:02:01] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:07:00 [I 2022-10-27 06:02:01] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00250 [I 2022-10-27 06:02:01] ...trap/engines/engine.py.230: loss: 0.13787 [I 2022-10-27 06:02:01] ...optimizers/trijoint.py.151: total_norm: 11.038043 [I 2022-10-27 06:02:07] ...trap/engines/engine.py.224: train: epoch 51 | batch 1560/2304 [I 2022-10-27 06:02:07] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:53 [I 2022-10-27 06:02:07] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00254 [I 2022-10-27 06:02:07] ...trap/engines/engine.py.230: loss: 0.19081 [I 2022-10-27 06:02:07] ...optimizers/trijoint.py.151: total_norm: 28.280903 [I 2022-10-27 06:02:12] ...trap/engines/engine.py.224: train: epoch 51 | batch 1570/2304 [I 2022-10-27 06:02:12] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:48 [I 2022-10-27 06:02:12] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00253 [I 2022-10-27 06:02:12] ...trap/engines/engine.py.230: loss: 0.17863 [I 2022-10-27 06:02:12] ...optimizers/trijoint.py.151: total_norm: 27.948004 [I 2022-10-27 06:02:18] ...trap/engines/engine.py.224: train: epoch 51 | batch 1580/2304 [I 2022-10-27 06:02:18] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:46 [I 2022-10-27 06:02:18] ...trap/engines/engine.py.229: process: 0.55893 | load: 0.00251 [I 2022-10-27 06:02:18] ...trap/engines/engine.py.230: loss: 0.10706 [I 2022-10-27 06:02:18] ...optimizers/trijoint.py.151: total_norm: 10.838453 [I 2022-10-27 06:02:24] ...trap/engines/engine.py.224: train: epoch 51 | batch 1590/2304 [I 2022-10-27 06:02:24] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:38 [I 2022-10-27 06:02:24] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00250 [I 2022-10-27 06:02:24] ...trap/engines/engine.py.230: loss: 0.12548 [I 2022-10-27 06:02:24] ...optimizers/trijoint.py.151: total_norm: 13.308208 [I 2022-10-27 06:02:29] ...trap/engines/engine.py.224: train: epoch 51 | batch 1600/2304 [I 2022-10-27 06:02:29] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:31 [I 2022-10-27 06:02:29] ...trap/engines/engine.py.229: process: 0.56160 | load: 0.00246 [I 2022-10-27 06:02:29] ...trap/engines/engine.py.230: loss: 0.16063 [I 2022-10-27 06:02:29] ...optimizers/trijoint.py.151: total_norm: 14.997616 [I 2022-10-27 06:02:35] ...trap/engines/engine.py.224: train: epoch 51 | batch 1610/2304 [I 2022-10-27 06:02:35] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:25 [I 2022-10-27 06:02:35] ...trap/engines/engine.py.229: process: 0.54417 | load: 0.00248 [I 2022-10-27 06:02:35] ...trap/engines/engine.py.230: loss: 0.16455 [I 2022-10-27 06:02:35] ...optimizers/trijoint.py.151: total_norm: 71.519958 [I 2022-10-27 06:02:40] ...trap/engines/engine.py.224: train: epoch 51 | batch 1620/2304 [I 2022-10-27 06:02:40] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:20 [I 2022-10-27 06:02:40] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00256 [I 2022-10-27 06:02:40] ...trap/engines/engine.py.230: loss: 0.15416 [I 2022-10-27 06:02:40] ...optimizers/trijoint.py.151: total_norm: 22.625853 [I 2022-10-27 06:02:46] ...trap/engines/engine.py.224: train: epoch 51 | batch 1630/2304 [I 2022-10-27 06:02:46] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:14 [I 2022-10-27 06:02:46] ...trap/engines/engine.py.229: process: 0.54801 | load: 0.00257 [I 2022-10-27 06:02:46] ...trap/engines/engine.py.230: loss: 0.13782 [I 2022-10-27 06:02:46] ...optimizers/trijoint.py.151: total_norm: 12.152843 [I 2022-10-27 06:02:51] ...trap/engines/engine.py.224: train: epoch 51 | batch 1640/2304 [I 2022-10-27 06:02:51] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:06:06 [I 2022-10-27 06:02:51] ...trap/engines/engine.py.229: process: 0.53693 | load: 0.00252 [I 2022-10-27 06:02:51] ...trap/engines/engine.py.230: loss: 0.11689 [I 2022-10-27 06:02:51] ...optimizers/trijoint.py.151: total_norm: 7.367417 [I 2022-10-27 06:02:57] ...trap/engines/engine.py.224: train: epoch 51 | batch 1650/2304 [I 2022-10-27 06:02:57] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:06:03 [I 2022-10-27 06:02:57] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00253 [I 2022-10-27 06:02:57] ...trap/engines/engine.py.230: loss: 0.12533 [I 2022-10-27 06:02:57] ...optimizers/trijoint.py.151: total_norm: 18.401678 [I 2022-10-27 06:03:02] ...trap/engines/engine.py.224: train: epoch 51 | batch 1660/2304 [I 2022-10-27 06:03:02] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:05:57 [I 2022-10-27 06:03:02] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00253 [I 2022-10-27 06:03:02] ...trap/engines/engine.py.230: loss: 0.17149 [I 2022-10-27 06:03:02] ...optimizers/trijoint.py.151: total_norm: 21.644373 [I 2022-10-27 06:03:08] ...trap/engines/engine.py.224: train: epoch 51 | batch 1670/2304 [I 2022-10-27 06:03:08] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:52 [I 2022-10-27 06:03:08] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00253 [I 2022-10-27 06:03:08] ...trap/engines/engine.py.230: loss: 0.16447 [I 2022-10-27 06:03:08] ...optimizers/trijoint.py.151: total_norm: 18.904066 [I 2022-10-27 06:03:14] ...trap/engines/engine.py.224: train: epoch 51 | batch 1680/2304 [I 2022-10-27 06:03:14] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:44 [I 2022-10-27 06:03:14] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00249 [I 2022-10-27 06:03:14] ...trap/engines/engine.py.230: loss: 0.24838 [I 2022-10-27 06:03:14] ...optimizers/trijoint.py.151: total_norm: 21.691303 [I 2022-10-27 06:03:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 1690/2304 [I 2022-10-27 06:03:19] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:42 [I 2022-10-27 06:03:19] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00254 [I 2022-10-27 06:03:19] ...trap/engines/engine.py.230: loss: 0.22397 [I 2022-10-27 06:03:19] ...optimizers/trijoint.py.151: total_norm: 29.021015 [I 2022-10-27 06:03:25] ...trap/engines/engine.py.224: train: epoch 51 | batch 1700/2304 [I 2022-10-27 06:03:25] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:39 [I 2022-10-27 06:03:25] ...trap/engines/engine.py.229: process: 0.57630 | load: 0.00251 [I 2022-10-27 06:03:25] ...trap/engines/engine.py.230: loss: 0.10549 [I 2022-10-27 06:03:25] ...optimizers/trijoint.py.151: total_norm: 16.740189 [I 2022-10-27 06:03:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 1710/2304 [I 2022-10-27 06:03:30] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:33 [I 2022-10-27 06:03:30] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00258 [I 2022-10-27 06:03:30] ...trap/engines/engine.py.230: loss: 0.18233 [I 2022-10-27 06:03:30] ...optimizers/trijoint.py.151: total_norm: 15.329435 [I 2022-10-27 06:03:36] ...trap/engines/engine.py.224: train: epoch 51 | batch 1720/2304 [I 2022-10-27 06:03:36] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:23 [I 2022-10-27 06:03:36] ...trap/engines/engine.py.229: process: 0.54839 | load: 0.00263 [I 2022-10-27 06:03:36] ...trap/engines/engine.py.230: loss: 0.14208 [I 2022-10-27 06:03:36] ...optimizers/trijoint.py.151: total_norm: 10.939945 [I 2022-10-27 06:03:42] ...trap/engines/engine.py.224: train: epoch 51 | batch 1730/2304 [I 2022-10-27 06:03:42] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:18 [I 2022-10-27 06:03:42] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00258 [I 2022-10-27 06:03:42] ...trap/engines/engine.py.230: loss: 0.14025 [I 2022-10-27 06:03:42] ...optimizers/trijoint.py.151: total_norm: 12.920955 [I 2022-10-27 06:03:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 1740/2304 [I 2022-10-27 06:03:47] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:05:14 [I 2022-10-27 06:03:47] ...trap/engines/engine.py.229: process: 0.56071 | load: 0.00255 [I 2022-10-27 06:03:47] ...trap/engines/engine.py.230: loss: 0.10805 [I 2022-10-27 06:03:47] ...optimizers/trijoint.py.151: total_norm: 20.146988 [I 2022-10-27 06:03:53] ...trap/engines/engine.py.224: train: epoch 51 | batch 1750/2304 [I 2022-10-27 06:03:53] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:05:07 [I 2022-10-27 06:03:53] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00255 [I 2022-10-27 06:03:53] ...trap/engines/engine.py.230: loss: 0.13258 [I 2022-10-27 06:03:53] ...optimizers/trijoint.py.151: total_norm: 11.282787 [I 2022-10-27 06:03:58] ...trap/engines/engine.py.224: train: epoch 51 | batch 1760/2304 [I 2022-10-27 06:03:58] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:05:01 [I 2022-10-27 06:03:58] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00262 [I 2022-10-27 06:03:58] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-27 06:03:58] ...optimizers/trijoint.py.151: total_norm: 10.289731 [I 2022-10-27 06:04:04] ...trap/engines/engine.py.224: train: epoch 51 | batch 1770/2304 [I 2022-10-27 06:04:04] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:58 [I 2022-10-27 06:04:04] ...trap/engines/engine.py.229: process: 0.55711 | load: 0.00254 [I 2022-10-27 06:04:04] ...trap/engines/engine.py.230: loss: 0.17466 [I 2022-10-27 06:04:04] ...optimizers/trijoint.py.151: total_norm: 17.383854 [I 2022-10-27 06:04:09] ...trap/engines/engine.py.224: train: epoch 51 | batch 1780/2304 [I 2022-10-27 06:04:09] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:52 [I 2022-10-27 06:04:09] ...trap/engines/engine.py.229: process: 0.56282 | load: 0.00260 [I 2022-10-27 06:04:09] ...trap/engines/engine.py.230: loss: 0.15588 [I 2022-10-27 06:04:09] ...optimizers/trijoint.py.151: total_norm: 14.033800 [I 2022-10-27 06:04:15] ...trap/engines/engine.py.224: train: epoch 51 | batch 1790/2304 [I 2022-10-27 06:04:15] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:46 [I 2022-10-27 06:04:15] ...trap/engines/engine.py.229: process: 0.55147 | load: 0.00251 [I 2022-10-27 06:04:15] ...trap/engines/engine.py.230: loss: 0.13510 [I 2022-10-27 06:04:15] ...optimizers/trijoint.py.151: total_norm: 16.358473 [I 2022-10-27 06:04:21] ...trap/engines/engine.py.224: train: epoch 51 | batch 1800/2304 [I 2022-10-27 06:04:21] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:39 [I 2022-10-27 06:04:21] ...trap/engines/engine.py.229: process: 0.54488 | load: 0.00254 [I 2022-10-27 06:04:21] ...trap/engines/engine.py.230: loss: 0.12103 [I 2022-10-27 06:04:21] ...optimizers/trijoint.py.151: total_norm: 11.584539 [I 2022-10-27 06:04:26] ...trap/engines/engine.py.224: train: epoch 51 | batch 1810/2304 [I 2022-10-27 06:04:26] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:34 [I 2022-10-27 06:04:26] ...trap/engines/engine.py.229: process: 0.55353 | load: 0.00256 [I 2022-10-27 06:04:26] ...trap/engines/engine.py.230: loss: 0.17479 [I 2022-10-27 06:04:26] ...optimizers/trijoint.py.151: total_norm: 13.557235 [I 2022-10-27 06:04:32] ...trap/engines/engine.py.224: train: epoch 51 | batch 1820/2304 [I 2022-10-27 06:04:32] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:28 [I 2022-10-27 06:04:32] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00256 [I 2022-10-27 06:04:32] ...trap/engines/engine.py.230: loss: 0.15291 [I 2022-10-27 06:04:32] ...optimizers/trijoint.py.151: total_norm: 9.310243 [I 2022-10-27 06:04:37] ...trap/engines/engine.py.224: train: epoch 51 | batch 1830/2304 [I 2022-10-27 06:04:37] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:23 [I 2022-10-27 06:04:37] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00254 [I 2022-10-27 06:04:37] ...trap/engines/engine.py.230: loss: 0.12989 [I 2022-10-27 06:04:37] ...optimizers/trijoint.py.151: total_norm: 13.413177 [I 2022-10-27 06:04:43] ...trap/engines/engine.py.224: train: epoch 51 | batch 1840/2304 [I 2022-10-27 06:04:43] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:16 [I 2022-10-27 06:04:43] ...trap/engines/engine.py.229: process: 0.54845 | load: 0.00248 [I 2022-10-27 06:04:43] ...trap/engines/engine.py.230: loss: 0.21000 [I 2022-10-27 06:04:43] ...optimizers/trijoint.py.151: total_norm: 29.918468 [I 2022-10-27 06:04:48] ...trap/engines/engine.py.224: train: epoch 51 | batch 1850/2304 [I 2022-10-27 06:04:48] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:04:10 [I 2022-10-27 06:04:48] ...trap/engines/engine.py.229: process: 0.53559 | load: 0.00252 [I 2022-10-27 06:04:48] ...trap/engines/engine.py.230: loss: 0.21658 [I 2022-10-27 06:04:48] ...optimizers/trijoint.py.151: total_norm: 24.558464 [I 2022-10-27 06:04:54] ...trap/engines/engine.py.224: train: epoch 51 | batch 1860/2304 [I 2022-10-27 06:04:54] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:04:05 [I 2022-10-27 06:04:54] ...trap/engines/engine.py.229: process: 0.54277 | load: 0.00252 [I 2022-10-27 06:04:54] ...trap/engines/engine.py.230: loss: 0.14162 [I 2022-10-27 06:04:54] ...optimizers/trijoint.py.151: total_norm: 7.061522 [I 2022-10-27 06:04:59] ...trap/engines/engine.py.224: train: epoch 51 | batch 1870/2304 [I 2022-10-27 06:04:59] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:04:01 [I 2022-10-27 06:04:59] ...trap/engines/engine.py.229: process: 0.56240 | load: 0.00252 [I 2022-10-27 06:04:59] ...trap/engines/engine.py.230: loss: 0.18113 [I 2022-10-27 06:04:59] ...optimizers/trijoint.py.151: total_norm: 40.109386 [I 2022-10-27 06:05:05] ...trap/engines/engine.py.224: train: epoch 51 | batch 1880/2304 [I 2022-10-27 06:05:05] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:55 [I 2022-10-27 06:05:05] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00250 [I 2022-10-27 06:05:05] ...trap/engines/engine.py.230: loss: 0.15660 [I 2022-10-27 06:05:05] ...optimizers/trijoint.py.151: total_norm: 19.459288 [I 2022-10-27 06:05:11] ...trap/engines/engine.py.224: train: epoch 51 | batch 1890/2304 [I 2022-10-27 06:05:11] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:49 [I 2022-10-27 06:05:11] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00252 [I 2022-10-27 06:05:11] ...trap/engines/engine.py.230: loss: 0.13796 [I 2022-10-27 06:05:11] ...optimizers/trijoint.py.151: total_norm: 13.248407 [I 2022-10-27 06:05:16] ...trap/engines/engine.py.224: train: epoch 51 | batch 1900/2304 [I 2022-10-27 06:05:16] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:44 [I 2022-10-27 06:05:16] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00255 [I 2022-10-27 06:05:16] ...trap/engines/engine.py.230: loss: 0.15309 [I 2022-10-27 06:05:16] ...optimizers/trijoint.py.151: total_norm: 15.116953 [I 2022-10-27 06:05:22] ...trap/engines/engine.py.224: train: epoch 51 | batch 1910/2304 [I 2022-10-27 06:05:22] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:38 [I 2022-10-27 06:05:22] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00250 [I 2022-10-27 06:05:22] ...trap/engines/engine.py.230: loss: 0.13949 [I 2022-10-27 06:05:22] ...optimizers/trijoint.py.151: total_norm: 14.729136 [I 2022-10-27 06:05:27] ...trap/engines/engine.py.224: train: epoch 51 | batch 1920/2304 [I 2022-10-27 06:05:27] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:32 [I 2022-10-27 06:05:27] ...trap/engines/engine.py.229: process: 0.55183 | load: 0.00250 [I 2022-10-27 06:05:27] ...trap/engines/engine.py.230: loss: 0.13240 [I 2022-10-27 06:05:27] ...optimizers/trijoint.py.151: total_norm: 15.495605 [I 2022-10-27 06:05:33] ...trap/engines/engine.py.224: train: epoch 51 | batch 1930/2304 [I 2022-10-27 06:05:33] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:27 [I 2022-10-27 06:05:33] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00255 [I 2022-10-27 06:05:33] ...trap/engines/engine.py.230: loss: 0.16427 [I 2022-10-27 06:05:33] ...optimizers/trijoint.py.151: total_norm: 11.972871 [I 2022-10-27 06:05:38] ...trap/engines/engine.py.224: train: epoch 51 | batch 1940/2304 [I 2022-10-27 06:05:38] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:22 [I 2022-10-27 06:05:38] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00260 [I 2022-10-27 06:05:38] ...trap/engines/engine.py.230: loss: 0.14889 [I 2022-10-27 06:05:38] ...optimizers/trijoint.py.151: total_norm: 12.375679 [I 2022-10-27 06:05:44] ...trap/engines/engine.py.224: train: epoch 51 | batch 1950/2304 [I 2022-10-27 06:05:44] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:17 [I 2022-10-27 06:05:44] ...trap/engines/engine.py.229: process: 0.55466 | load: 0.00270 [I 2022-10-27 06:05:44] ...trap/engines/engine.py.230: loss: 0.16483 [I 2022-10-27 06:05:44] ...optimizers/trijoint.py.151: total_norm: 11.973245 [I 2022-10-27 06:05:50] ...trap/engines/engine.py.224: train: epoch 51 | batch 1960/2304 [I 2022-10-27 06:05:50] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:03:13 [I 2022-10-27 06:05:50] ...trap/engines/engine.py.229: process: 0.55539 | load: 0.00251 [I 2022-10-27 06:05:50] ...trap/engines/engine.py.230: loss: 0.14364 [I 2022-10-27 06:05:50] ...optimizers/trijoint.py.151: total_norm: 18.534460 [I 2022-10-27 06:05:55] ...trap/engines/engine.py.224: train: epoch 51 | batch 1970/2304 [I 2022-10-27 06:05:55] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:03:05 [I 2022-10-27 06:05:55] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00260 [I 2022-10-27 06:05:55] ...trap/engines/engine.py.230: loss: 0.18188 [I 2022-10-27 06:05:55] ...optimizers/trijoint.py.151: total_norm: 37.413624 [I 2022-10-27 06:06:01] ...trap/engines/engine.py.224: train: epoch 51 | batch 1980/2304 [I 2022-10-27 06:06:01] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:03:02 [I 2022-10-27 06:06:01] ...trap/engines/engine.py.229: process: 0.55048 | load: 0.00256 [I 2022-10-27 06:06:01] ...trap/engines/engine.py.230: loss: 0.21504 [I 2022-10-27 06:06:01] ...optimizers/trijoint.py.151: total_norm: 19.344296 [I 2022-10-27 06:06:06] ...trap/engines/engine.py.224: train: epoch 51 | batch 1990/2304 [I 2022-10-27 06:06:06] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:54 [I 2022-10-27 06:06:06] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00249 [I 2022-10-27 06:06:06] ...trap/engines/engine.py.230: loss: 0.18540 [I 2022-10-27 06:06:06] ...optimizers/trijoint.py.151: total_norm: 51.818260 [I 2022-10-27 06:06:12] ...trap/engines/engine.py.224: train: epoch 51 | batch 2000/2304 [I 2022-10-27 06:06:12] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:48 [I 2022-10-27 06:06:12] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00256 [I 2022-10-27 06:06:12] ...trap/engines/engine.py.230: loss: 0.15532 [I 2022-10-27 06:06:12] ...optimizers/trijoint.py.151: total_norm: 32.294373 [I 2022-10-27 06:06:18] ...trap/engines/engine.py.224: train: epoch 51 | batch 2010/2304 [I 2022-10-27 06:06:18] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:44 [I 2022-10-27 06:06:18] ...trap/engines/engine.py.229: process: 0.56581 | load: 0.00253 [I 2022-10-27 06:06:18] ...trap/engines/engine.py.230: loss: 0.17589 [I 2022-10-27 06:06:18] ...optimizers/trijoint.py.151: total_norm: 10.771769 [I 2022-10-27 06:06:23] ...trap/engines/engine.py.224: train: epoch 51 | batch 2020/2304 [I 2022-10-27 06:06:23] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:36 [I 2022-10-27 06:06:23] ...trap/engines/engine.py.229: process: 0.55510 | load: 0.00249 [I 2022-10-27 06:06:23] ...trap/engines/engine.py.230: loss: 0.14301 [I 2022-10-27 06:06:23] ...optimizers/trijoint.py.151: total_norm: 20.599991 [I 2022-10-27 06:06:29] ...trap/engines/engine.py.224: train: epoch 51 | batch 2030/2304 [I 2022-10-27 06:06:29] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:31 [I 2022-10-27 06:06:29] ...trap/engines/engine.py.229: process: 0.54641 | load: 0.00254 [I 2022-10-27 06:06:29] ...trap/engines/engine.py.230: loss: 0.17356 [I 2022-10-27 06:06:29] ...optimizers/trijoint.py.151: total_norm: 9.484963 [I 2022-10-27 06:06:34] ...trap/engines/engine.py.224: train: epoch 51 | batch 2040/2304 [I 2022-10-27 06:06:34] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:27 [I 2022-10-27 06:06:34] ...trap/engines/engine.py.229: process: 0.56306 | load: 0.00252 [I 2022-10-27 06:06:34] ...trap/engines/engine.py.230: loss: 0.20216 [I 2022-10-27 06:06:34] ...optimizers/trijoint.py.151: total_norm: 11.356580 [I 2022-10-27 06:06:40] ...trap/engines/engine.py.224: train: epoch 51 | batch 2050/2304 [I 2022-10-27 06:06:40] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:22 [I 2022-10-27 06:06:40] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00256 [I 2022-10-27 06:06:40] ...trap/engines/engine.py.230: loss: 0.18996 [I 2022-10-27 06:06:40] ...optimizers/trijoint.py.151: total_norm: 22.016039 [I 2022-10-27 06:06:45] ...trap/engines/engine.py.224: train: epoch 51 | batch 2060/2304 [I 2022-10-27 06:06:45] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:15 [I 2022-10-27 06:06:45] ...trap/engines/engine.py.229: process: 0.54620 | load: 0.00252 [I 2022-10-27 06:06:45] ...trap/engines/engine.py.230: loss: 0.17672 [I 2022-10-27 06:06:45] ...optimizers/trijoint.py.151: total_norm: 37.013348 [I 2022-10-27 06:06:51] ...trap/engines/engine.py.224: train: epoch 51 | batch 2070/2304 [I 2022-10-27 06:06:51] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:08 [I 2022-10-27 06:06:51] ...trap/engines/engine.py.229: process: 0.54103 | load: 0.00252 [I 2022-10-27 06:06:51] ...trap/engines/engine.py.230: loss: 0.15900 [I 2022-10-27 06:06:51] ...optimizers/trijoint.py.151: total_norm: 18.852659 [I 2022-10-27 06:06:57] ...trap/engines/engine.py.224: train: epoch 51 | batch 2080/2304 [I 2022-10-27 06:06:57] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:02:04 [I 2022-10-27 06:06:57] ...trap/engines/engine.py.229: process: 0.54841 | load: 0.00251 [I 2022-10-27 06:06:57] ...trap/engines/engine.py.230: loss: 0.16888 [I 2022-10-27 06:06:57] ...optimizers/trijoint.py.151: total_norm: 16.977827 [I 2022-10-27 06:07:02] ...trap/engines/engine.py.224: train: epoch 51 | batch 2090/2304 [I 2022-10-27 06:07:02] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:58 [I 2022-10-27 06:07:02] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00249 [I 2022-10-27 06:07:02] ...trap/engines/engine.py.230: loss: 0.22416 [I 2022-10-27 06:07:02] ...optimizers/trijoint.py.151: total_norm: 36.790356 [I 2022-10-27 06:07:08] ...trap/engines/engine.py.224: train: epoch 51 | batch 2100/2304 [I 2022-10-27 06:07:08] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:53 [I 2022-10-27 06:07:08] ...trap/engines/engine.py.229: process: 0.55410 | load: 0.00253 [I 2022-10-27 06:07:08] ...trap/engines/engine.py.230: loss: 0.15709 [I 2022-10-27 06:07:08] ...optimizers/trijoint.py.151: total_norm: 15.732106 [I 2022-10-27 06:07:13] ...trap/engines/engine.py.224: train: epoch 51 | batch 2110/2304 [I 2022-10-27 06:07:13] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:48 [I 2022-10-27 06:07:13] ...trap/engines/engine.py.229: process: 0.54613 | load: 0.00246 [I 2022-10-27 06:07:13] ...trap/engines/engine.py.230: loss: 0.18696 [I 2022-10-27 06:07:13] ...optimizers/trijoint.py.151: total_norm: 15.514925 [I 2022-10-27 06:07:19] ...trap/engines/engine.py.224: train: epoch 51 | batch 2120/2304 [I 2022-10-27 06:07:19] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:42 [I 2022-10-27 06:07:19] ...trap/engines/engine.py.229: process: 0.54333 | load: 0.00256 [I 2022-10-27 06:07:19] ...trap/engines/engine.py.230: loss: 0.12699 [I 2022-10-27 06:07:19] ...optimizers/trijoint.py.151: total_norm: 9.742587 [I 2022-10-27 06:07:25] ...trap/engines/engine.py.224: train: epoch 51 | batch 2130/2304 [I 2022-10-27 06:07:25] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:36 [I 2022-10-27 06:07:25] ...trap/engines/engine.py.229: process: 0.55205 | load: 0.00256 [I 2022-10-27 06:07:25] ...trap/engines/engine.py.230: loss: 0.18009 [I 2022-10-27 06:07:25] ...optimizers/trijoint.py.151: total_norm: 26.327084 [I 2022-10-27 06:07:30] ...trap/engines/engine.py.224: train: epoch 51 | batch 2140/2304 [I 2022-10-27 06:07:30] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:30 [I 2022-10-27 06:07:30] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00254 [I 2022-10-27 06:07:30] ...trap/engines/engine.py.230: loss: 0.14510 [I 2022-10-27 06:07:30] ...optimizers/trijoint.py.151: total_norm: 23.301476 [I 2022-10-27 06:07:36] ...trap/engines/engine.py.224: train: epoch 51 | batch 2150/2304 [I 2022-10-27 06:07:36] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:25 [I 2022-10-27 06:07:36] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00252 [I 2022-10-27 06:07:36] ...trap/engines/engine.py.230: loss: 0.15197 [I 2022-10-27 06:07:36] ...optimizers/trijoint.py.151: total_norm: 13.405154 [I 2022-10-27 06:07:41] ...trap/engines/engine.py.224: train: epoch 51 | batch 2160/2304 [I 2022-10-27 06:07:41] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:19 [I 2022-10-27 06:07:41] ...trap/engines/engine.py.229: process: 0.54294 | load: 0.00250 [I 2022-10-27 06:07:41] ...trap/engines/engine.py.230: loss: 0.22697 [I 2022-10-27 06:07:41] ...optimizers/trijoint.py.151: total_norm: 21.421665 [I 2022-10-27 06:07:47] ...trap/engines/engine.py.224: train: epoch 51 | batch 2170/2304 [I 2022-10-27 06:07:47] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:01:14 [I 2022-10-27 06:07:47] ...trap/engines/engine.py.229: process: 0.55395 | load: 0.00251 [I 2022-10-27 06:07:47] ...trap/engines/engine.py.230: loss: 0.16745 [I 2022-10-27 06:07:47] ...optimizers/trijoint.py.151: total_norm: 30.705442 [I 2022-10-27 06:07:52] ...trap/engines/engine.py.224: train: epoch 51 | batch 2180/2304 [I 2022-10-27 06:07:52] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:08 [I 2022-10-27 06:07:52] ...trap/engines/engine.py.229: process: 0.54940 | load: 0.00249 [I 2022-10-27 06:07:52] ...trap/engines/engine.py.230: loss: 0.22813 [I 2022-10-27 06:07:52] ...optimizers/trijoint.py.151: total_norm: 17.603697 [I 2022-10-27 06:07:58] ...trap/engines/engine.py.224: train: epoch 51 | batch 2190/2304 [I 2022-10-27 06:07:58] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:01:03 [I 2022-10-27 06:07:58] ...trap/engines/engine.py.229: process: 0.55079 | load: 0.00252 [I 2022-10-27 06:07:58] ...trap/engines/engine.py.230: loss: 0.31959 [I 2022-10-27 06:07:58] ...optimizers/trijoint.py.151: total_norm: 13.866005 [I 2022-10-27 06:08:03] ...trap/engines/engine.py.224: train: epoch 51 | batch 2200/2304 [I 2022-10-27 06:08:03] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:57 [I 2022-10-27 06:08:03] ...trap/engines/engine.py.229: process: 0.55911 | load: 0.00248 [I 2022-10-27 06:08:03] ...trap/engines/engine.py.230: loss: 0.14179 [I 2022-10-27 06:08:03] ...optimizers/trijoint.py.151: total_norm: 15.718462 [I 2022-10-27 06:08:09] ...trap/engines/engine.py.224: train: epoch 51 | batch 2210/2304 [I 2022-10-27 06:08:09] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:51 [I 2022-10-27 06:08:09] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00251 [I 2022-10-27 06:08:09] ...trap/engines/engine.py.230: loss: 0.15991 [I 2022-10-27 06:08:09] ...optimizers/trijoint.py.151: total_norm: 25.793701 [I 2022-10-27 06:08:15] ...trap/engines/engine.py.224: train: epoch 51 | batch 2220/2304 [I 2022-10-27 06:08:15] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:46 [I 2022-10-27 06:08:15] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00249 [I 2022-10-27 06:08:15] ...trap/engines/engine.py.230: loss: 0.14202 [I 2022-10-27 06:08:15] ...optimizers/trijoint.py.151: total_norm: 19.792318 [I 2022-10-27 06:08:20] ...trap/engines/engine.py.224: train: epoch 51 | batch 2230/2304 [I 2022-10-27 06:08:20] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:41 [I 2022-10-27 06:08:20] ...trap/engines/engine.py.229: process: 0.56342 | load: 0.00275 [I 2022-10-27 06:08:20] ...trap/engines/engine.py.230: loss: 0.19930 [I 2022-10-27 06:08:20] ...optimizers/trijoint.py.151: total_norm: 22.757023 [I 2022-10-27 06:08:26] ...trap/engines/engine.py.224: train: epoch 51 | batch 2240/2304 [I 2022-10-27 06:08:26] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:35 [I 2022-10-27 06:08:26] ...trap/engines/engine.py.229: process: 0.54900 | load: 0.00260 [I 2022-10-27 06:08:26] ...trap/engines/engine.py.230: loss: 0.18209 [I 2022-10-27 06:08:26] ...optimizers/trijoint.py.151: total_norm: 19.676670 [I 2022-10-27 06:08:31] ...trap/engines/engine.py.224: train: epoch 51 | batch 2250/2304 [I 2022-10-27 06:08:31] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:29 [I 2022-10-27 06:08:31] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00258 [I 2022-10-27 06:08:31] ...trap/engines/engine.py.230: loss: 0.13058 [I 2022-10-27 06:08:31] ...optimizers/trijoint.py.151: total_norm: 15.993332 [I 2022-10-27 06:08:37] ...trap/engines/engine.py.224: train: epoch 51 | batch 2260/2304 [I 2022-10-27 06:08:37] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:24 [I 2022-10-27 06:08:37] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00254 [I 2022-10-27 06:08:37] ...trap/engines/engine.py.230: loss: 0.12010 [I 2022-10-27 06:08:37] ...optimizers/trijoint.py.151: total_norm: 10.219632 [I 2022-10-27 06:08:42] ...trap/engines/engine.py.224: train: epoch 51 | batch 2270/2304 [I 2022-10-27 06:08:42] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:18 [I 2022-10-27 06:08:42] ...trap/engines/engine.py.229: process: 0.55280 | load: 0.00254 [I 2022-10-27 06:08:42] ...trap/engines/engine.py.230: loss: 0.13583 [I 2022-10-27 06:08:42] ...optimizers/trijoint.py.151: total_norm: 42.556309 [I 2022-10-27 06:08:48] ...trap/engines/engine.py.224: train: epoch 51 | batch 2280/2304 [I 2022-10-27 06:08:48] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:13 [I 2022-10-27 06:08:48] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00258 [I 2022-10-27 06:08:48] ...trap/engines/engine.py.230: loss: 0.19979 [I 2022-10-27 06:08:48] ...optimizers/trijoint.py.151: total_norm: 27.174786 [I 2022-10-27 06:08:54] ...trap/engines/engine.py.224: train: epoch 51 | batch 2290/2304 [I 2022-10-27 06:08:54] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:00:07 [I 2022-10-27 06:08:54] ...trap/engines/engine.py.229: process: 0.55145 | load: 0.00253 [I 2022-10-27 06:08:54] ...trap/engines/engine.py.230: loss: 0.15604 [I 2022-10-27 06:08:54] ...optimizers/trijoint.py.151: total_norm: 15.096873 [I 2022-10-27 06:08:59] ...trap/engines/engine.py.224: train: epoch 51 | batch 2300/2304 [I 2022-10-27 06:08:59] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:02 [I 2022-10-27 06:08:59] ...trap/engines/engine.py.229: process: 0.55057 | load: 0.00014 [I 2022-10-27 06:08:59] ...trap/engines/engine.py.230: loss: 0.11273 [I 2022-10-27 06:08:59] ...optimizers/trijoint.py.151: total_norm: 8.285833 [I 2022-10-27 06:09:01] ...trap/engines/engine.py.224: train: epoch 51 | batch 2304/2304 [I 2022-10-27 06:09:01] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:00 [I 2022-10-27 06:09:01] ...trap/engines/engine.py.229: process: 0.54874 | load: 0.00017 [I 2022-10-27 06:09:01] ...trap/engines/engine.py.230: loss: 0.20052 [I 2022-10-27 06:09:01] ...optimizers/trijoint.py.151: total_norm: 24.186544 [S 2022-10-27 06:09:03] ...trap/engines/engine.py.129: train_epoch.epoch: 51 [S 2022-10-27 06:09:03] ...trap/engines/engine.py.129: train_epoch.loss: 0.1641575624685484 [I 2022-10-27 06:09:03] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 06:09:03] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 51 [I 2022-10-27 06:09:03] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 06:09:05] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 06:09:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 06:09:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 06:09:07] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 06:09:07] ...trap/engines/engine.py.321: eval: epoch 51 | batch 0/510 [I 2022-10-27 06:09:07] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:33:48 [I 2022-10-27 06:09:07] ...trap/engines/engine.py.326: process: 3.97747 | load: 3.06740 [I 2022-10-27 06:09:11] ...trap/engines/engine.py.321: eval: epoch 51 | batch 10/510 [I 2022-10-27 06:09:11] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:01 [I 2022-10-27 06:09:11] ...trap/engines/engine.py.326: process: 0.33111 | load: 0.00016 [I 2022-10-27 06:09:14] ...trap/engines/engine.py.321: eval: epoch 51 | batch 20/510 [I 2022-10-27 06:09:14] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:05 [I 2022-10-27 06:09:14] ...trap/engines/engine.py.326: process: 0.32392 | load: 0.00021 [I 2022-10-27 06:09:17] ...trap/engines/engine.py.321: eval: epoch 51 | batch 30/510 [I 2022-10-27 06:09:17] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:41 [I 2022-10-27 06:09:17] ...trap/engines/engine.py.326: process: 0.32821 | load: 0.00018 [I 2022-10-27 06:09:21] ...trap/engines/engine.py.321: eval: epoch 51 | batch 40/510 [I 2022-10-27 06:09:21] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:36 [I 2022-10-27 06:09:21] ...trap/engines/engine.py.326: process: 0.33151 | load: 0.00017 [I 2022-10-27 06:09:24] ...trap/engines/engine.py.321: eval: epoch 51 | batch 50/510 [I 2022-10-27 06:09:24] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:36 [I 2022-10-27 06:09:24] ...trap/engines/engine.py.326: process: 0.34366 | load: 0.00020 [I 2022-10-27 06:09:28] ...trap/engines/engine.py.321: eval: epoch 51 | batch 60/510 [I 2022-10-27 06:09:28] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:35 [I 2022-10-27 06:09:28] ...trap/engines/engine.py.326: process: 0.36843 | load: 0.00017 [I 2022-10-27 06:09:31] ...trap/engines/engine.py.321: eval: epoch 51 | batch 70/510 [I 2022-10-27 06:09:31] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:28 [I 2022-10-27 06:09:31] ...trap/engines/engine.py.326: process: 0.32672 | load: 0.00017 [I 2022-10-27 06:09:34] ...trap/engines/engine.py.321: eval: epoch 51 | batch 80/510 [I 2022-10-27 06:09:34] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:27 [I 2022-10-27 06:09:34] ...trap/engines/engine.py.326: process: 0.34524 | load: 0.00017 [I 2022-10-27 06:09:38] ...trap/engines/engine.py.321: eval: epoch 51 | batch 90/510 [I 2022-10-27 06:09:38] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:23 [I 2022-10-27 06:09:38] ...trap/engines/engine.py.326: process: 0.36267 | load: 0.00017 [I 2022-10-27 06:09:41] ...trap/engines/engine.py.321: eval: epoch 51 | batch 100/510 [I 2022-10-27 06:09:41] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:13 [I 2022-10-27 06:09:41] ...trap/engines/engine.py.326: process: 0.29660 | load: 0.00020 [I 2022-10-27 06:09:44] ...trap/engines/engine.py.321: eval: epoch 51 | batch 110/510 [I 2022-10-27 06:09:44] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:56 [I 2022-10-27 06:09:44] ...trap/engines/engine.py.326: process: 0.28797 | load: 0.00023 [I 2022-10-27 06:09:47] ...trap/engines/engine.py.321: eval: epoch 51 | batch 120/510 [I 2022-10-27 06:09:47] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:52 [I 2022-10-27 06:09:47] ...trap/engines/engine.py.326: process: 0.28800 | load: 0.00021 [I 2022-10-27 06:09:50] ...trap/engines/engine.py.321: eval: epoch 51 | batch 130/510 [I 2022-10-27 06:09:50] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:49 [I 2022-10-27 06:09:50] ...trap/engines/engine.py.326: process: 0.28927 | load: 0.00015 [I 2022-10-27 06:09:53] ...trap/engines/engine.py.321: eval: epoch 51 | batch 140/510 [I 2022-10-27 06:09:53] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 06:09:53] ...trap/engines/engine.py.326: process: 0.29197 | load: 0.00020 [I 2022-10-27 06:09:56] ...trap/engines/engine.py.321: eval: epoch 51 | batch 150/510 [I 2022-10-27 06:09:56] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:45 [I 2022-10-27 06:09:56] ...trap/engines/engine.py.326: process: 0.28351 | load: 0.00025 [I 2022-10-27 06:09:58] ...trap/engines/engine.py.321: eval: epoch 51 | batch 160/510 [I 2022-10-27 06:09:58] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 06:09:58] ...trap/engines/engine.py.326: process: 0.29370 | load: 0.00016 [I 2022-10-27 06:10:01] ...trap/engines/engine.py.321: eval: epoch 51 | batch 170/510 [I 2022-10-27 06:10:01] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 06:10:01] ...trap/engines/engine.py.326: process: 0.29057 | load: 0.00028 [I 2022-10-27 06:10:04] ...trap/engines/engine.py.321: eval: epoch 51 | batch 180/510 [I 2022-10-27 06:10:04] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 06:10:04] ...trap/engines/engine.py.326: process: 0.29152 | load: 0.00029 [I 2022-10-27 06:10:07] ...trap/engines/engine.py.321: eval: epoch 51 | batch 190/510 [I 2022-10-27 06:10:07] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:31 [I 2022-10-27 06:10:07] ...trap/engines/engine.py.326: process: 0.27985 | load: 0.00027 [I 2022-10-27 06:10:10] ...trap/engines/engine.py.321: eval: epoch 51 | batch 200/510 [I 2022-10-27 06:10:10] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 06:10:10] ...trap/engines/engine.py.326: process: 0.28854 | load: 0.00017 [I 2022-10-27 06:10:13] ...trap/engines/engine.py.321: eval: epoch 51 | batch 210/510 [I 2022-10-27 06:10:13] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 06:10:13] ...trap/engines/engine.py.326: process: 0.28592 | load: 0.00016 [I 2022-10-27 06:10:16] ...trap/engines/engine.py.321: eval: epoch 51 | batch 220/510 [I 2022-10-27 06:10:16] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 06:10:16] ...trap/engines/engine.py.326: process: 0.28185 | load: 0.00019 [I 2022-10-27 06:10:19] ...trap/engines/engine.py.321: eval: epoch 51 | batch 230/510 [I 2022-10-27 06:10:19] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 06:10:19] ...trap/engines/engine.py.326: process: 0.28814 | load: 0.00034 [I 2022-10-27 06:10:22] ...trap/engines/engine.py.321: eval: epoch 51 | batch 240/510 [I 2022-10-27 06:10:22] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 06:10:22] ...trap/engines/engine.py.326: process: 0.29412 | load: 0.00024 [I 2022-10-27 06:10:25] ...trap/engines/engine.py.321: eval: epoch 51 | batch 250/510 [I 2022-10-27 06:10:25] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 06:10:25] ...trap/engines/engine.py.326: process: 0.29027 | load: 0.00028 [I 2022-10-27 06:10:28] ...trap/engines/engine.py.321: eval: epoch 51 | batch 260/510 [I 2022-10-27 06:10:28] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:14 [I 2022-10-27 06:10:28] ...trap/engines/engine.py.326: process: 0.29140 | load: 0.00017 [I 2022-10-27 06:10:30] ...trap/engines/engine.py.321: eval: epoch 51 | batch 270/510 [I 2022-10-27 06:10:30] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 06:10:30] ...trap/engines/engine.py.326: process: 0.28740 | load: 0.00030 [I 2022-10-27 06:10:33] ...trap/engines/engine.py.321: eval: epoch 51 | batch 280/510 [I 2022-10-27 06:10:33] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 06:10:33] ...trap/engines/engine.py.326: process: 0.29377 | load: 0.00018 [I 2022-10-27 06:10:36] ...trap/engines/engine.py.321: eval: epoch 51 | batch 290/510 [I 2022-10-27 06:10:36] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 06:10:36] ...trap/engines/engine.py.326: process: 0.28684 | load: 0.00025 [I 2022-10-27 06:10:39] ...trap/engines/engine.py.321: eval: epoch 51 | batch 300/510 [I 2022-10-27 06:10:39] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:01 [I 2022-10-27 06:10:39] ...trap/engines/engine.py.326: process: 0.29319 | load: 0.00022 [I 2022-10-27 06:10:42] ...trap/engines/engine.py.321: eval: epoch 51 | batch 310/510 [I 2022-10-27 06:10:42] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:58 [I 2022-10-27 06:10:42] ...trap/engines/engine.py.326: process: 0.30818 | load: 0.00025 [I 2022-10-27 06:10:45] ...trap/engines/engine.py.321: eval: epoch 51 | batch 320/510 [I 2022-10-27 06:10:45] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:55 [I 2022-10-27 06:10:45] ...trap/engines/engine.py.326: process: 0.28867 | load: 0.00017 [I 2022-10-27 06:10:48] ...trap/engines/engine.py.321: eval: epoch 51 | batch 330/510 [I 2022-10-27 06:10:48] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:52 [I 2022-10-27 06:10:48] ...trap/engines/engine.py.326: process: 0.29320 | load: 0.00025 [I 2022-10-27 06:10:51] ...trap/engines/engine.py.321: eval: epoch 51 | batch 340/510 [I 2022-10-27 06:10:51] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 06:10:51] ...trap/engines/engine.py.326: process: 0.29273 | load: 0.00015 [I 2022-10-27 06:10:54] ...trap/engines/engine.py.321: eval: epoch 51 | batch 350/510 [I 2022-10-27 06:10:54] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 06:10:54] ...trap/engines/engine.py.326: process: 0.29038 | load: 0.00018 [I 2022-10-27 06:10:56] ...trap/engines/engine.py.321: eval: epoch 51 | batch 360/510 [I 2022-10-27 06:10:56] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 06:10:56] ...trap/engines/engine.py.326: process: 0.29315 | load: 0.00019 [I 2022-10-27 06:10:59] ...trap/engines/engine.py.321: eval: epoch 51 | batch 370/510 [I 2022-10-27 06:10:59] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 06:10:59] ...trap/engines/engine.py.326: process: 0.27962 | load: 0.00024 [I 2022-10-27 06:11:02] ...trap/engines/engine.py.321: eval: epoch 51 | batch 380/510 [I 2022-10-27 06:11:02] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 06:11:02] ...trap/engines/engine.py.326: process: 0.29282 | load: 0.00016 [I 2022-10-27 06:11:05] ...trap/engines/engine.py.321: eval: epoch 51 | batch 390/510 [I 2022-10-27 06:11:05] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 06:11:05] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00016 [I 2022-10-27 06:11:08] ...trap/engines/engine.py.321: eval: epoch 51 | batch 400/510 [I 2022-10-27 06:11:08] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 06:11:08] ...trap/engines/engine.py.326: process: 0.29153 | load: 0.00016 [I 2022-10-27 06:11:11] ...trap/engines/engine.py.321: eval: epoch 51 | batch 410/510 [I 2022-10-27 06:11:11] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 06:11:11] ...trap/engines/engine.py.326: process: 0.28232 | load: 0.00024 [I 2022-10-27 06:11:14] ...trap/engines/engine.py.321: eval: epoch 51 | batch 420/510 [I 2022-10-27 06:11:14] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 06:11:14] ...trap/engines/engine.py.326: process: 0.28118 | load: 0.00016 [I 2022-10-27 06:11:17] ...trap/engines/engine.py.321: eval: epoch 51 | batch 430/510 [I 2022-10-27 06:11:17] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 06:11:17] ...trap/engines/engine.py.326: process: 0.28400 | load: 0.00022 [I 2022-10-27 06:11:20] ...trap/engines/engine.py.321: eval: epoch 51 | batch 440/510 [I 2022-10-27 06:11:20] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 06:11:20] ...trap/engines/engine.py.326: process: 0.29054 | load: 0.00017 [I 2022-10-27 06:11:22] ...trap/engines/engine.py.321: eval: epoch 51 | batch 450/510 [I 2022-10-27 06:11:22] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 06:11:22] ...trap/engines/engine.py.326: process: 0.28969 | load: 0.00016 [I 2022-10-27 06:11:25] ...trap/engines/engine.py.321: eval: epoch 51 | batch 460/510 [I 2022-10-27 06:11:25] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 06:11:25] ...trap/engines/engine.py.326: process: 0.29163 | load: 0.00015 [I 2022-10-27 06:11:28] ...trap/engines/engine.py.321: eval: epoch 51 | batch 470/510 [I 2022-10-27 06:11:28] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 06:11:28] ...trap/engines/engine.py.326: process: 0.28055 | load: 0.00022 [I 2022-10-27 06:11:31] ...trap/engines/engine.py.321: eval: epoch 51 | batch 480/510 [I 2022-10-27 06:11:31] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 06:11:31] ...trap/engines/engine.py.326: process: 0.29154 | load: 0.00017 [I 2022-10-27 06:11:34] ...trap/engines/engine.py.321: eval: epoch 51 | batch 490/510 [I 2022-10-27 06:11:34] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 06:11:34] ...trap/engines/engine.py.326: process: 0.28519 | load: 0.00021 [I 2022-10-27 06:11:37] ...trap/engines/engine.py.321: eval: epoch 51 | batch 500/510 [I 2022-10-27 06:11:37] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 06:11:37] ...trap/engines/engine.py.326: process: 0.28967 | load: 0.00016 [I 2022-10-27 06:11:40] ...trap/engines/engine.py.321: eval: epoch 51 | batch 510/510 [I 2022-10-27 06:11:40] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 06:11:40] ...trap/engines/engine.py.326: process: 0.28673 | load: 0.00016 [S 2022-10-27 06:11:40] ...trap/engines/engine.py.132: eval_epoch.epoch: 51 [I 2022-10-27 06:11:40] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.700100 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.897100 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.929700 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.683300 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.890500 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.926800 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011104 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006426 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007142 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012594 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008512 [S 2022-10-27 06:11:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005381 [I 2022-10-27 06:11:46] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 06:11:46] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 06:11:46] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 06:11:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 06:11:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 06:11:49] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 06:11:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 06:11:57] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 06:11:57] ...trap/engines/engine.py.163: Training model on trainset for epoch 52 [I 2022-10-27 06:11:57] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 06:11:59] ...trap/engines/engine.py.224: train: epoch 52 | batch 0/2304 [I 2022-10-27 06:11:59] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:50 [I 2022-10-27 06:11:59] ...trap/engines/engine.py.229: process: 2.70415 | load: 2.06388 [I 2022-10-27 06:11:59] ...trap/engines/engine.py.230: loss: 0.15481 [I 2022-10-27 06:11:59] ...optimizers/trijoint.py.151: total_norm: 16.395031 [I 2022-10-27 06:12:05] ...trap/engines/engine.py.224: train: epoch 52 | batch 10/2304 [I 2022-10-27 06:12:05] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:10 [I 2022-10-27 06:12:05] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00374 [I 2022-10-27 06:12:05] ...trap/engines/engine.py.230: loss: 0.17748 [I 2022-10-27 06:12:05] ...optimizers/trijoint.py.151: total_norm: 17.764139 [I 2022-10-27 06:12:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 20/2304 [I 2022-10-27 06:12:10] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:18 [I 2022-10-27 06:12:10] ...trap/engines/engine.py.229: process: 0.58355 | load: 0.00311 [I 2022-10-27 06:12:10] ...trap/engines/engine.py.230: loss: 0.21707 [I 2022-10-27 06:12:10] ...optimizers/trijoint.py.151: total_norm: 26.636299 [I 2022-10-27 06:12:16] ...trap/engines/engine.py.224: train: epoch 52 | batch 30/2304 [I 2022-10-27 06:12:16] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:19 [I 2022-10-27 06:12:16] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00312 [I 2022-10-27 06:12:16] ...trap/engines/engine.py.230: loss: 0.13565 [I 2022-10-27 06:12:16] ...optimizers/trijoint.py.151: total_norm: 22.764151 [I 2022-10-27 06:12:22] ...trap/engines/engine.py.224: train: epoch 52 | batch 40/2304 [I 2022-10-27 06:12:22] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:50 [I 2022-10-27 06:12:22] ...trap/engines/engine.py.229: process: 0.53999 | load: 0.00289 [I 2022-10-27 06:12:22] ...trap/engines/engine.py.230: loss: 0.13285 [I 2022-10-27 06:12:22] ...optimizers/trijoint.py.151: total_norm: 18.750086 [I 2022-10-27 06:12:27] ...trap/engines/engine.py.224: train: epoch 52 | batch 50/2304 [I 2022-10-27 06:12:27] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:49 [I 2022-10-27 06:12:27] ...trap/engines/engine.py.229: process: 0.54781 | load: 0.00292 [I 2022-10-27 06:12:27] ...trap/engines/engine.py.230: loss: 0.19383 [I 2022-10-27 06:12:27] ...optimizers/trijoint.py.151: total_norm: 19.330465 [I 2022-10-27 06:12:33] ...trap/engines/engine.py.224: train: epoch 52 | batch 60/2304 [I 2022-10-27 06:12:33] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:50 [I 2022-10-27 06:12:33] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00290 [I 2022-10-27 06:12:33] ...trap/engines/engine.py.230: loss: 0.25619 [I 2022-10-27 06:12:33] ...optimizers/trijoint.py.151: total_norm: 13.767030 [I 2022-10-27 06:12:38] ...trap/engines/engine.py.224: train: epoch 52 | batch 70/2304 [I 2022-10-27 06:12:38] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:37 [I 2022-10-27 06:12:38] ...trap/engines/engine.py.229: process: 0.55005 | load: 0.00263 [I 2022-10-27 06:12:38] ...trap/engines/engine.py.230: loss: 0.11929 [I 2022-10-27 06:12:38] ...optimizers/trijoint.py.151: total_norm: 9.235011 [I 2022-10-27 06:12:44] ...trap/engines/engine.py.224: train: epoch 52 | batch 80/2304 [I 2022-10-27 06:12:44] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:35 [I 2022-10-27 06:12:44] ...trap/engines/engine.py.229: process: 0.54966 | load: 0.00261 [I 2022-10-27 06:12:44] ...trap/engines/engine.py.230: loss: 0.12845 [I 2022-10-27 06:12:44] ...optimizers/trijoint.py.151: total_norm: 21.150242 [I 2022-10-27 06:12:49] ...trap/engines/engine.py.224: train: epoch 52 | batch 90/2304 [I 2022-10-27 06:12:49] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:29 [I 2022-10-27 06:12:49] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00241 [I 2022-10-27 06:12:49] ...trap/engines/engine.py.230: loss: 0.10682 [I 2022-10-27 06:12:49] ...optimizers/trijoint.py.151: total_norm: 9.248483 [I 2022-10-27 06:12:55] ...trap/engines/engine.py.224: train: epoch 52 | batch 100/2304 [I 2022-10-27 06:12:55] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:19 [I 2022-10-27 06:12:55] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00256 [I 2022-10-27 06:12:55] ...trap/engines/engine.py.230: loss: 0.15355 [I 2022-10-27 06:12:55] ...optimizers/trijoint.py.151: total_norm: 14.529830 [I 2022-10-27 06:13:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 110/2304 [I 2022-10-27 06:13:00] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:20:18 [I 2022-10-27 06:13:00] ...trap/engines/engine.py.229: process: 0.54963 | load: 0.00271 [I 2022-10-27 06:13:00] ...trap/engines/engine.py.230: loss: 0.09083 [I 2022-10-27 06:13:00] ...optimizers/trijoint.py.151: total_norm: 7.118284 [I 2022-10-27 06:13:06] ...trap/engines/engine.py.224: train: epoch 52 | batch 120/2304 [I 2022-10-27 06:13:06] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:24 [I 2022-10-27 06:13:06] ...trap/engines/engine.py.229: process: 0.57099 | load: 0.00281 [I 2022-10-27 06:13:06] ...trap/engines/engine.py.230: loss: 0.23576 [I 2022-10-27 06:13:06] ...optimizers/trijoint.py.151: total_norm: 34.894108 [I 2022-10-27 06:13:12] ...trap/engines/engine.py.224: train: epoch 52 | batch 130/2304 [I 2022-10-27 06:13:12] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:02 [I 2022-10-27 06:13:12] ...trap/engines/engine.py.229: process: 0.55030 | load: 0.00280 [I 2022-10-27 06:13:12] ...trap/engines/engine.py.230: loss: 0.21226 [I 2022-10-27 06:13:12] ...optimizers/trijoint.py.151: total_norm: 17.971176 [I 2022-10-27 06:13:17] ...trap/engines/engine.py.224: train: epoch 52 | batch 140/2304 [I 2022-10-27 06:13:17] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:57 [I 2022-10-27 06:13:17] ...trap/engines/engine.py.229: process: 0.53833 | load: 0.00248 [I 2022-10-27 06:13:17] ...trap/engines/engine.py.230: loss: 0.17584 [I 2022-10-27 06:13:17] ...optimizers/trijoint.py.151: total_norm: 13.443110 [I 2022-10-27 06:13:23] ...trap/engines/engine.py.224: train: epoch 52 | batch 150/2304 [I 2022-10-27 06:13:23] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:52 [I 2022-10-27 06:13:23] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00243 [I 2022-10-27 06:13:23] ...trap/engines/engine.py.230: loss: 0.19608 [I 2022-10-27 06:13:23] ...optimizers/trijoint.py.151: total_norm: 19.554993 [I 2022-10-27 06:13:28] ...trap/engines/engine.py.224: train: epoch 52 | batch 160/2304 [I 2022-10-27 06:13:28] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:50 [I 2022-10-27 06:13:28] ...trap/engines/engine.py.229: process: 0.55267 | load: 0.00250 [I 2022-10-27 06:13:28] ...trap/engines/engine.py.230: loss: 0.18151 [I 2022-10-27 06:13:28] ...optimizers/trijoint.py.151: total_norm: 26.517029 [I 2022-10-27 06:13:34] ...trap/engines/engine.py.224: train: epoch 52 | batch 170/2304 [I 2022-10-27 06:13:34] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:43 [I 2022-10-27 06:13:34] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00244 [I 2022-10-27 06:13:34] ...trap/engines/engine.py.230: loss: 0.15121 [I 2022-10-27 06:13:34] ...optimizers/trijoint.py.151: total_norm: 38.896545 [I 2022-10-27 06:13:39] ...trap/engines/engine.py.224: train: epoch 52 | batch 180/2304 [I 2022-10-27 06:13:39] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:19:44 [I 2022-10-27 06:13:39] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00242 [I 2022-10-27 06:13:39] ...trap/engines/engine.py.230: loss: 0.18591 [I 2022-10-27 06:13:39] ...optimizers/trijoint.py.151: total_norm: 34.076416 [I 2022-10-27 06:13:45] ...trap/engines/engine.py.224: train: epoch 52 | batch 190/2304 [I 2022-10-27 06:13:45] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:20 [I 2022-10-27 06:13:45] ...trap/engines/engine.py.229: process: 0.54319 | load: 0.00252 [I 2022-10-27 06:13:45] ...trap/engines/engine.py.230: loss: 0.14143 [I 2022-10-27 06:13:45] ...optimizers/trijoint.py.151: total_norm: 17.769281 [I 2022-10-27 06:13:50] ...trap/engines/engine.py.224: train: epoch 52 | batch 200/2304 [I 2022-10-27 06:13:50] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:19:35 [I 2022-10-27 06:13:50] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00243 [I 2022-10-27 06:13:50] ...trap/engines/engine.py.230: loss: 0.14030 [I 2022-10-27 06:13:50] ...optimizers/trijoint.py.151: total_norm: 12.782337 [I 2022-10-27 06:13:56] ...trap/engines/engine.py.224: train: epoch 52 | batch 210/2304 [I 2022-10-27 06:13:56] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:40 [I 2022-10-27 06:13:56] ...trap/engines/engine.py.229: process: 0.56683 | load: 0.00246 [I 2022-10-27 06:13:56] ...trap/engines/engine.py.230: loss: 0.27795 [I 2022-10-27 06:13:56] ...optimizers/trijoint.py.151: total_norm: 24.063377 [I 2022-10-27 06:14:02] ...trap/engines/engine.py.224: train: epoch 52 | batch 220/2304 [I 2022-10-27 06:14:02] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:33 [I 2022-10-27 06:14:02] ...trap/engines/engine.py.229: process: 0.57155 | load: 0.00245 [I 2022-10-27 06:14:02] ...trap/engines/engine.py.230: loss: 0.15958 [I 2022-10-27 06:14:02] ...optimizers/trijoint.py.151: total_norm: 14.874297 [I 2022-10-27 06:14:07] ...trap/engines/engine.py.224: train: epoch 52 | batch 230/2304 [I 2022-10-27 06:14:07] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:14 [I 2022-10-27 06:14:07] ...trap/engines/engine.py.229: process: 0.54761 | load: 0.00255 [I 2022-10-27 06:14:07] ...trap/engines/engine.py.230: loss: 0.14843 [I 2022-10-27 06:14:07] ...optimizers/trijoint.py.151: total_norm: 11.624843 [I 2022-10-27 06:14:13] ...trap/engines/engine.py.224: train: epoch 52 | batch 240/2304 [I 2022-10-27 06:14:13] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:18:58 [I 2022-10-27 06:14:13] ...trap/engines/engine.py.229: process: 0.54756 | load: 0.00260 [I 2022-10-27 06:14:13] ...trap/engines/engine.py.230: loss: 0.20286 [I 2022-10-27 06:14:13] ...optimizers/trijoint.py.151: total_norm: 22.217203 [I 2022-10-27 06:14:18] ...trap/engines/engine.py.224: train: epoch 52 | batch 250/2304 [I 2022-10-27 06:14:18] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:19:02 [I 2022-10-27 06:14:18] ...trap/engines/engine.py.229: process: 0.56354 | load: 0.00242 [I 2022-10-27 06:14:18] ...trap/engines/engine.py.230: loss: 0.17179 [I 2022-10-27 06:14:18] ...optimizers/trijoint.py.151: total_norm: 15.399228 [I 2022-10-27 06:14:24] ...trap/engines/engine.py.224: train: epoch 52 | batch 260/2304 [I 2022-10-27 06:14:24] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:52 [I 2022-10-27 06:14:24] ...trap/engines/engine.py.229: process: 0.55301 | load: 0.00250 [I 2022-10-27 06:14:24] ...trap/engines/engine.py.230: loss: 0.17507 [I 2022-10-27 06:14:24] ...optimizers/trijoint.py.151: total_norm: 24.273863 [I 2022-10-27 06:14:30] ...trap/engines/engine.py.224: train: epoch 52 | batch 270/2304 [I 2022-10-27 06:14:30] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:53 [I 2022-10-27 06:14:30] ...trap/engines/engine.py.229: process: 0.55902 | load: 0.00254 [I 2022-10-27 06:14:30] ...trap/engines/engine.py.230: loss: 0.12178 [I 2022-10-27 06:14:30] ...optimizers/trijoint.py.151: total_norm: 11.104610 [I 2022-10-27 06:14:35] ...trap/engines/engine.py.224: train: epoch 52 | batch 280/2304 [I 2022-10-27 06:14:35] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:45 [I 2022-10-27 06:14:35] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00250 [I 2022-10-27 06:14:35] ...trap/engines/engine.py.230: loss: 0.14953 [I 2022-10-27 06:14:35] ...optimizers/trijoint.py.151: total_norm: 10.610158 [I 2022-10-27 06:14:41] ...trap/engines/engine.py.224: train: epoch 52 | batch 290/2304 [I 2022-10-27 06:14:41] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:41 [I 2022-10-27 06:14:41] ...trap/engines/engine.py.229: process: 0.55127 | load: 0.00252 [I 2022-10-27 06:14:41] ...trap/engines/engine.py.230: loss: 0.20159 [I 2022-10-27 06:14:41] ...optimizers/trijoint.py.151: total_norm: 20.418640 [I 2022-10-27 06:14:46] ...trap/engines/engine.py.224: train: epoch 52 | batch 300/2304 [I 2022-10-27 06:14:46] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:18:36 [I 2022-10-27 06:14:46] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00266 [I 2022-10-27 06:14:46] ...trap/engines/engine.py.230: loss: 0.15392 [I 2022-10-27 06:14:46] ...optimizers/trijoint.py.151: total_norm: 15.059071 [I 2022-10-27 06:14:52] ...trap/engines/engine.py.224: train: epoch 52 | batch 310/2304 [I 2022-10-27 06:14:52] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:27 [I 2022-10-27 06:14:52] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00273 [I 2022-10-27 06:14:52] ...trap/engines/engine.py.230: loss: 0.15881 [I 2022-10-27 06:14:52] ...optimizers/trijoint.py.151: total_norm: 28.848568 [I 2022-10-27 06:14:57] ...trap/engines/engine.py.224: train: epoch 52 | batch 320/2304 [I 2022-10-27 06:14:57] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:18:18 [I 2022-10-27 06:14:57] ...trap/engines/engine.py.229: process: 0.56529 | load: 0.00244 [I 2022-10-27 06:14:57] ...trap/engines/engine.py.230: loss: 0.17932 [I 2022-10-27 06:14:57] ...optimizers/trijoint.py.151: total_norm: 18.827919 [I 2022-10-27 06:15:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 330/2304 [I 2022-10-27 06:15:03] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:17 [I 2022-10-27 06:15:03] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00245 [I 2022-10-27 06:15:03] ...trap/engines/engine.py.230: loss: 0.15420 [I 2022-10-27 06:15:03] ...optimizers/trijoint.py.151: total_norm: 10.923378 [I 2022-10-27 06:15:08] ...trap/engines/engine.py.224: train: epoch 52 | batch 340/2304 [I 2022-10-27 06:15:08] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:18:05 [I 2022-10-27 06:15:08] ...trap/engines/engine.py.229: process: 0.54965 | load: 0.00244 [I 2022-10-27 06:15:08] ...trap/engines/engine.py.230: loss: 0.17553 [I 2022-10-27 06:15:08] ...optimizers/trijoint.py.151: total_norm: 40.552883 [I 2022-10-27 06:15:14] ...trap/engines/engine.py.224: train: epoch 52 | batch 350/2304 [I 2022-10-27 06:15:14] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:18:07 [I 2022-10-27 06:15:14] ...trap/engines/engine.py.229: process: 0.56389 | load: 0.00244 [I 2022-10-27 06:15:14] ...trap/engines/engine.py.230: loss: 0.18274 [I 2022-10-27 06:15:14] ...optimizers/trijoint.py.151: total_norm: 14.949423 [I 2022-10-27 06:15:20] ...trap/engines/engine.py.224: train: epoch 52 | batch 360/2304 [I 2022-10-27 06:15:20] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:55 [I 2022-10-27 06:15:20] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00252 [I 2022-10-27 06:15:20] ...trap/engines/engine.py.230: loss: 0.21308 [I 2022-10-27 06:15:20] ...optimizers/trijoint.py.151: total_norm: 13.737285 [I 2022-10-27 06:15:25] ...trap/engines/engine.py.224: train: epoch 52 | batch 370/2304 [I 2022-10-27 06:15:25] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:18:02 [I 2022-10-27 06:15:25] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00263 [I 2022-10-27 06:15:25] ...trap/engines/engine.py.230: loss: 0.14444 [I 2022-10-27 06:15:25] ...optimizers/trijoint.py.151: total_norm: 27.224455 [I 2022-10-27 06:15:31] ...trap/engines/engine.py.224: train: epoch 52 | batch 380/2304 [I 2022-10-27 06:15:31] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:46 [I 2022-10-27 06:15:31] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00245 [I 2022-10-27 06:15:31] ...trap/engines/engine.py.230: loss: 0.12958 [I 2022-10-27 06:15:31] ...optimizers/trijoint.py.151: total_norm: 12.317231 [I 2022-10-27 06:15:36] ...trap/engines/engine.py.224: train: epoch 52 | batch 390/2304 [I 2022-10-27 06:15:36] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:17:43 [I 2022-10-27 06:15:36] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00245 [I 2022-10-27 06:15:36] ...trap/engines/engine.py.230: loss: 0.15008 [I 2022-10-27 06:15:36] ...optimizers/trijoint.py.151: total_norm: 47.635090 [I 2022-10-27 06:15:42] ...trap/engines/engine.py.224: train: epoch 52 | batch 400/2304 [I 2022-10-27 06:15:42] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:38 [I 2022-10-27 06:15:42] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00252 [I 2022-10-27 06:15:42] ...trap/engines/engine.py.230: loss: 0.15669 [I 2022-10-27 06:15:42] ...optimizers/trijoint.py.151: total_norm: 68.489662 [I 2022-10-27 06:15:47] ...trap/engines/engine.py.224: train: epoch 52 | batch 410/2304 [I 2022-10-27 06:15:47] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:17:32 [I 2022-10-27 06:15:47] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00244 [I 2022-10-27 06:15:47] ...trap/engines/engine.py.230: loss: 0.23594 [I 2022-10-27 06:15:47] ...optimizers/trijoint.py.151: total_norm: 31.854856 [I 2022-10-27 06:15:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 420/2304 [I 2022-10-27 06:15:53] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:17:28 [I 2022-10-27 06:15:53] ...trap/engines/engine.py.229: process: 0.55938 | load: 0.00252 [I 2022-10-27 06:15:53] ...trap/engines/engine.py.230: loss: 0.19682 [I 2022-10-27 06:15:53] ...optimizers/trijoint.py.151: total_norm: 14.255404 [I 2022-10-27 06:15:59] ...trap/engines/engine.py.224: train: epoch 52 | batch 430/2304 [I 2022-10-27 06:15:59] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:16 [I 2022-10-27 06:15:59] ...trap/engines/engine.py.229: process: 0.53718 | load: 0.00248 [I 2022-10-27 06:15:59] ...trap/engines/engine.py.230: loss: 0.12321 [I 2022-10-27 06:15:59] ...optimizers/trijoint.py.151: total_norm: 12.661036 [I 2022-10-27 06:16:04] ...trap/engines/engine.py.224: train: epoch 52 | batch 440/2304 [I 2022-10-27 06:16:04] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:17:49 [I 2022-10-27 06:16:04] ...trap/engines/engine.py.229: process: 0.54959 | load: 0.00244 [I 2022-10-27 06:16:04] ...trap/engines/engine.py.230: loss: 0.16368 [I 2022-10-27 06:16:04] ...optimizers/trijoint.py.151: total_norm: 10.929623 [I 2022-10-27 06:16:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 450/2304 [I 2022-10-27 06:16:10] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:20 [I 2022-10-27 06:16:10] ...trap/engines/engine.py.229: process: 0.56087 | load: 0.00244 [I 2022-10-27 06:16:10] ...trap/engines/engine.py.230: loss: 0.14090 [I 2022-10-27 06:16:10] ...optimizers/trijoint.py.151: total_norm: 9.415635 [I 2022-10-27 06:16:15] ...trap/engines/engine.py.224: train: epoch 52 | batch 460/2304 [I 2022-10-27 06:16:15] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:17:01 [I 2022-10-27 06:16:15] ...trap/engines/engine.py.229: process: 0.55587 | load: 0.00245 [I 2022-10-27 06:16:15] ...trap/engines/engine.py.230: loss: 0.17586 [I 2022-10-27 06:16:15] ...optimizers/trijoint.py.151: total_norm: 18.560907 [I 2022-10-27 06:16:21] ...trap/engines/engine.py.224: train: epoch 52 | batch 470/2304 [I 2022-10-27 06:16:21] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:16:59 [I 2022-10-27 06:16:21] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00243 [I 2022-10-27 06:16:21] ...trap/engines/engine.py.230: loss: 0.18007 [I 2022-10-27 06:16:21] ...optimizers/trijoint.py.151: total_norm: 24.995123 [I 2022-10-27 06:16:27] ...trap/engines/engine.py.224: train: epoch 52 | batch 480/2304 [I 2022-10-27 06:16:27] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:58 [I 2022-10-27 06:16:27] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00248 [I 2022-10-27 06:16:27] ...trap/engines/engine.py.230: loss: 0.21056 [I 2022-10-27 06:16:27] ...optimizers/trijoint.py.151: total_norm: 13.533566 [I 2022-10-27 06:16:32] ...trap/engines/engine.py.224: train: epoch 52 | batch 490/2304 [I 2022-10-27 06:16:32] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:16:45 [I 2022-10-27 06:16:32] ...trap/engines/engine.py.229: process: 0.53817 | load: 0.00254 [I 2022-10-27 06:16:32] ...trap/engines/engine.py.230: loss: 0.12203 [I 2022-10-27 06:16:32] ...optimizers/trijoint.py.151: total_norm: 9.363820 [I 2022-10-27 06:16:38] ...trap/engines/engine.py.224: train: epoch 52 | batch 500/2304 [I 2022-10-27 06:16:38] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:36 [I 2022-10-27 06:16:38] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00244 [I 2022-10-27 06:16:38] ...trap/engines/engine.py.230: loss: 0.13282 [I 2022-10-27 06:16:38] ...optimizers/trijoint.py.151: total_norm: 25.091269 [I 2022-10-27 06:16:43] ...trap/engines/engine.py.224: train: epoch 52 | batch 510/2304 [I 2022-10-27 06:16:43] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:16:28 [I 2022-10-27 06:16:43] ...trap/engines/engine.py.229: process: 0.55518 | load: 0.00244 [I 2022-10-27 06:16:43] ...trap/engines/engine.py.230: loss: 0.15303 [I 2022-10-27 06:16:43] ...optimizers/trijoint.py.151: total_norm: 20.827070 [I 2022-10-27 06:16:49] ...trap/engines/engine.py.224: train: epoch 52 | batch 520/2304 [I 2022-10-27 06:16:49] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:17:00 [I 2022-10-27 06:16:49] ...trap/engines/engine.py.229: process: 0.56997 | load: 0.00246 [I 2022-10-27 06:16:49] ...trap/engines/engine.py.230: loss: 0.20455 [I 2022-10-27 06:16:49] ...optimizers/trijoint.py.151: total_norm: 11.998700 [I 2022-10-27 06:16:55] ...trap/engines/engine.py.224: train: epoch 52 | batch 530/2304 [I 2022-10-27 06:16:55] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:34 [I 2022-10-27 06:16:55] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00242 [I 2022-10-27 06:16:55] ...trap/engines/engine.py.230: loss: 0.20945 [I 2022-10-27 06:16:55] ...optimizers/trijoint.py.151: total_norm: 14.071026 [I 2022-10-27 06:17:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 540/2304 [I 2022-10-27 06:17:00] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:16:20 [I 2022-10-27 06:17:00] ...trap/engines/engine.py.229: process: 0.55015 | load: 0.00243 [I 2022-10-27 06:17:00] ...trap/engines/engine.py.230: loss: 0.13765 [I 2022-10-27 06:17:00] ...optimizers/trijoint.py.151: total_norm: 17.754011 [I 2022-10-27 06:17:06] ...trap/engines/engine.py.224: train: epoch 52 | batch 550/2304 [I 2022-10-27 06:17:06] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:11 [I 2022-10-27 06:17:06] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00246 [I 2022-10-27 06:17:06] ...trap/engines/engine.py.230: loss: 0.15034 [I 2022-10-27 06:17:06] ...optimizers/trijoint.py.151: total_norm: 24.166433 [I 2022-10-27 06:17:11] ...trap/engines/engine.py.224: train: epoch 52 | batch 560/2304 [I 2022-10-27 06:17:11] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:16:14 [I 2022-10-27 06:17:11] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00242 [I 2022-10-27 06:17:11] ...trap/engines/engine.py.230: loss: 0.13211 [I 2022-10-27 06:17:11] ...optimizers/trijoint.py.151: total_norm: 11.796625 [I 2022-10-27 06:17:17] ...trap/engines/engine.py.224: train: epoch 52 | batch 570/2304 [I 2022-10-27 06:17:17] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:16:07 [I 2022-10-27 06:17:17] ...trap/engines/engine.py.229: process: 0.57251 | load: 0.00249 [I 2022-10-27 06:17:17] ...trap/engines/engine.py.230: loss: 0.18987 [I 2022-10-27 06:17:17] ...optimizers/trijoint.py.151: total_norm: 21.079409 [I 2022-10-27 06:17:23] ...trap/engines/engine.py.224: train: epoch 52 | batch 580/2304 [I 2022-10-27 06:17:23] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:33 [I 2022-10-27 06:17:23] ...trap/engines/engine.py.229: process: 0.65282 | load: 0.00245 [I 2022-10-27 06:17:23] ...trap/engines/engine.py.230: loss: 0.13993 [I 2022-10-27 06:17:23] ...optimizers/trijoint.py.151: total_norm: 16.242674 [I 2022-10-27 06:17:28] ...trap/engines/engine.py.224: train: epoch 52 | batch 590/2304 [I 2022-10-27 06:17:28] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:15:53 [I 2022-10-27 06:17:28] ...trap/engines/engine.py.229: process: 0.56489 | load: 0.00244 [I 2022-10-27 06:17:28] ...trap/engines/engine.py.230: loss: 0.15348 [I 2022-10-27 06:17:28] ...optimizers/trijoint.py.151: total_norm: 9.531148 [I 2022-10-27 06:17:34] ...trap/engines/engine.py.224: train: epoch 52 | batch 600/2304 [I 2022-10-27 06:17:34] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:45 [I 2022-10-27 06:17:34] ...trap/engines/engine.py.229: process: 0.54354 | load: 0.00242 [I 2022-10-27 06:17:34] ...trap/engines/engine.py.230: loss: 0.10774 [I 2022-10-27 06:17:34] ...optimizers/trijoint.py.151: total_norm: 8.368947 [I 2022-10-27 06:17:39] ...trap/engines/engine.py.224: train: epoch 52 | batch 610/2304 [I 2022-10-27 06:17:39] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:15:40 [I 2022-10-27 06:17:39] ...trap/engines/engine.py.229: process: 0.54920 | load: 0.00239 [I 2022-10-27 06:17:39] ...trap/engines/engine.py.230: loss: 0.14710 [I 2022-10-27 06:17:39] ...optimizers/trijoint.py.151: total_norm: 16.741808 [I 2022-10-27 06:17:45] ...trap/engines/engine.py.224: train: epoch 52 | batch 620/2304 [I 2022-10-27 06:17:45] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:29 [I 2022-10-27 06:17:45] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00242 [I 2022-10-27 06:17:45] ...trap/engines/engine.py.230: loss: 0.15330 [I 2022-10-27 06:17:45] ...optimizers/trijoint.py.151: total_norm: 17.228481 [I 2022-10-27 06:17:50] ...trap/engines/engine.py.224: train: epoch 52 | batch 630/2304 [I 2022-10-27 06:17:50] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:15:30 [I 2022-10-27 06:17:50] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00244 [I 2022-10-27 06:17:50] ...trap/engines/engine.py.230: loss: 0.22014 [I 2022-10-27 06:17:50] ...optimizers/trijoint.py.151: total_norm: 24.103004 [I 2022-10-27 06:17:56] ...trap/engines/engine.py.224: train: epoch 52 | batch 640/2304 [I 2022-10-27 06:17:56] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:22 [I 2022-10-27 06:17:56] ...trap/engines/engine.py.229: process: 0.55032 | load: 0.00240 [I 2022-10-27 06:17:56] ...trap/engines/engine.py.230: loss: 0.16128 [I 2022-10-27 06:17:56] ...optimizers/trijoint.py.151: total_norm: 18.555302 [I 2022-10-27 06:18:02] ...trap/engines/engine.py.224: train: epoch 52 | batch 650/2304 [I 2022-10-27 06:18:02] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:24 [I 2022-10-27 06:18:02] ...trap/engines/engine.py.229: process: 0.56920 | load: 0.00243 [I 2022-10-27 06:18:02] ...trap/engines/engine.py.230: loss: 0.15567 [I 2022-10-27 06:18:02] ...optimizers/trijoint.py.151: total_norm: 38.986713 [I 2022-10-27 06:18:07] ...trap/engines/engine.py.224: train: epoch 52 | batch 660/2304 [I 2022-10-27 06:18:07] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:15:09 [I 2022-10-27 06:18:07] ...trap/engines/engine.py.229: process: 0.54337 | load: 0.00252 [I 2022-10-27 06:18:07] ...trap/engines/engine.py.230: loss: 0.11964 [I 2022-10-27 06:18:07] ...optimizers/trijoint.py.151: total_norm: 7.438667 [I 2022-10-27 06:18:13] ...trap/engines/engine.py.224: train: epoch 52 | batch 670/2304 [I 2022-10-27 06:18:13] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:05 [I 2022-10-27 06:18:13] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00247 [I 2022-10-27 06:18:13] ...trap/engines/engine.py.230: loss: 0.11139 [I 2022-10-27 06:18:13] ...optimizers/trijoint.py.151: total_norm: 8.812719 [I 2022-10-27 06:18:18] ...trap/engines/engine.py.224: train: epoch 52 | batch 680/2304 [I 2022-10-27 06:18:18] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:15:22 [I 2022-10-27 06:18:18] ...trap/engines/engine.py.229: process: 0.53775 | load: 0.00244 [I 2022-10-27 06:18:18] ...trap/engines/engine.py.230: loss: 0.11790 [I 2022-10-27 06:18:18] ...optimizers/trijoint.py.151: total_norm: 17.570578 [I 2022-10-27 06:18:24] ...trap/engines/engine.py.224: train: epoch 52 | batch 690/2304 [I 2022-10-27 06:18:24] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:15:01 [I 2022-10-27 06:18:24] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00253 [I 2022-10-27 06:18:24] ...trap/engines/engine.py.230: loss: 0.17199 [I 2022-10-27 06:18:24] ...optimizers/trijoint.py.151: total_norm: 19.542652 [I 2022-10-27 06:18:29] ...trap/engines/engine.py.224: train: epoch 52 | batch 700/2304 [I 2022-10-27 06:18:29] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:14:52 [I 2022-10-27 06:18:29] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00242 [I 2022-10-27 06:18:29] ...trap/engines/engine.py.230: loss: 0.25306 [I 2022-10-27 06:18:29] ...optimizers/trijoint.py.151: total_norm: 25.578739 [I 2022-10-27 06:18:35] ...trap/engines/engine.py.224: train: epoch 52 | batch 710/2304 [I 2022-10-27 06:18:35] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:14:43 [I 2022-10-27 06:18:35] ...trap/engines/engine.py.229: process: 0.56210 | load: 0.00242 [I 2022-10-27 06:18:35] ...trap/engines/engine.py.230: loss: 0.13740 [I 2022-10-27 06:18:35] ...optimizers/trijoint.py.151: total_norm: 9.813609 [I 2022-10-27 06:18:41] ...trap/engines/engine.py.224: train: epoch 52 | batch 720/2304 [I 2022-10-27 06:18:41] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:36 [I 2022-10-27 06:18:41] ...trap/engines/engine.py.229: process: 0.55141 | load: 0.00242 [I 2022-10-27 06:18:41] ...trap/engines/engine.py.230: loss: 0.19138 [I 2022-10-27 06:18:41] ...optimizers/trijoint.py.151: total_norm: 18.444380 [I 2022-10-27 06:18:46] ...trap/engines/engine.py.224: train: epoch 52 | batch 730/2304 [I 2022-10-27 06:18:46] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:14:39 [I 2022-10-27 06:18:46] ...trap/engines/engine.py.229: process: 0.56878 | load: 0.00244 [I 2022-10-27 06:18:46] ...trap/engines/engine.py.230: loss: 0.12769 [I 2022-10-27 06:18:46] ...optimizers/trijoint.py.151: total_norm: 9.701260 [I 2022-10-27 06:18:52] ...trap/engines/engine.py.224: train: epoch 52 | batch 740/2304 [I 2022-10-27 06:18:52] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:31 [I 2022-10-27 06:18:52] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00240 [I 2022-10-27 06:18:52] ...trap/engines/engine.py.230: loss: 0.22396 [I 2022-10-27 06:18:52] ...optimizers/trijoint.py.151: total_norm: 28.196814 [I 2022-10-27 06:18:57] ...trap/engines/engine.py.224: train: epoch 52 | batch 750/2304 [I 2022-10-27 06:18:57] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:24 [I 2022-10-27 06:18:57] ...trap/engines/engine.py.229: process: 0.55075 | load: 0.00238 [I 2022-10-27 06:18:57] ...trap/engines/engine.py.230: loss: 0.17731 [I 2022-10-27 06:18:57] ...optimizers/trijoint.py.151: total_norm: 12.546540 [I 2022-10-27 06:19:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 760/2304 [I 2022-10-27 06:19:03] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:20 [I 2022-10-27 06:19:03] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00241 [I 2022-10-27 06:19:03] ...trap/engines/engine.py.230: loss: 0.18785 [I 2022-10-27 06:19:03] ...optimizers/trijoint.py.151: total_norm: 18.225170 [I 2022-10-27 06:19:08] ...trap/engines/engine.py.224: train: epoch 52 | batch 770/2304 [I 2022-10-27 06:19:08] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:15 [I 2022-10-27 06:19:08] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00249 [I 2022-10-27 06:19:08] ...trap/engines/engine.py.230: loss: 0.18111 [I 2022-10-27 06:19:08] ...optimizers/trijoint.py.151: total_norm: 14.269255 [I 2022-10-27 06:19:14] ...trap/engines/engine.py.224: train: epoch 52 | batch 780/2304 [I 2022-10-27 06:19:14] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:14:07 [I 2022-10-27 06:19:14] ...trap/engines/engine.py.229: process: 0.55460 | load: 0.00247 [I 2022-10-27 06:19:14] ...trap/engines/engine.py.230: loss: 0.16330 [I 2022-10-27 06:19:14] ...optimizers/trijoint.py.151: total_norm: 15.965148 [I 2022-10-27 06:19:20] ...trap/engines/engine.py.224: train: epoch 52 | batch 790/2304 [I 2022-10-27 06:19:20] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:13:59 [I 2022-10-27 06:19:20] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00241 [I 2022-10-27 06:19:20] ...trap/engines/engine.py.230: loss: 0.15435 [I 2022-10-27 06:19:20] ...optimizers/trijoint.py.151: total_norm: 16.435642 [I 2022-10-27 06:19:25] ...trap/engines/engine.py.224: train: epoch 52 | batch 800/2304 [I 2022-10-27 06:19:25] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:06 [I 2022-10-27 06:19:25] ...trap/engines/engine.py.229: process: 0.55277 | load: 0.00239 [I 2022-10-27 06:19:25] ...trap/engines/engine.py.230: loss: 0.14121 [I 2022-10-27 06:19:25] ...optimizers/trijoint.py.151: total_norm: 17.269968 [I 2022-10-27 06:19:31] ...trap/engines/engine.py.224: train: epoch 52 | batch 810/2304 [I 2022-10-27 06:19:31] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:13:53 [I 2022-10-27 06:19:31] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00241 [I 2022-10-27 06:19:31] ...trap/engines/engine.py.230: loss: 0.11484 [I 2022-10-27 06:19:31] ...optimizers/trijoint.py.151: total_norm: 14.363688 [I 2022-10-27 06:19:37] ...trap/engines/engine.py.224: train: epoch 52 | batch 820/2304 [I 2022-10-27 06:19:37] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:49 [I 2022-10-27 06:19:37] ...trap/engines/engine.py.229: process: 0.55945 | load: 0.00247 [I 2022-10-27 06:19:37] ...trap/engines/engine.py.230: loss: 0.20801 [I 2022-10-27 06:19:37] ...optimizers/trijoint.py.151: total_norm: 16.644276 [I 2022-10-27 06:19:42] ...trap/engines/engine.py.224: train: epoch 52 | batch 830/2304 [I 2022-10-27 06:19:42] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:13:30 [I 2022-10-27 06:19:42] ...trap/engines/engine.py.229: process: 0.54425 | load: 0.00265 [I 2022-10-27 06:19:42] ...trap/engines/engine.py.230: loss: 0.14974 [I 2022-10-27 06:19:42] ...optimizers/trijoint.py.151: total_norm: 12.988351 [I 2022-10-27 06:19:48] ...trap/engines/engine.py.224: train: epoch 52 | batch 840/2304 [I 2022-10-27 06:19:48] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:26 [I 2022-10-27 06:19:48] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00247 [I 2022-10-27 06:19:48] ...trap/engines/engine.py.230: loss: 0.16852 [I 2022-10-27 06:19:48] ...optimizers/trijoint.py.151: total_norm: 20.105579 [I 2022-10-27 06:19:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 850/2304 [I 2022-10-27 06:19:53] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:25 [I 2022-10-27 06:19:53] ...trap/engines/engine.py.229: process: 0.54237 | load: 0.00252 [I 2022-10-27 06:19:53] ...trap/engines/engine.py.230: loss: 0.22333 [I 2022-10-27 06:19:53] ...optimizers/trijoint.py.151: total_norm: 26.803404 [I 2022-10-27 06:19:59] ...trap/engines/engine.py.224: train: epoch 52 | batch 860/2304 [I 2022-10-27 06:19:59] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:21 [I 2022-10-27 06:19:59] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00250 [I 2022-10-27 06:19:59] ...trap/engines/engine.py.230: loss: 0.16894 [I 2022-10-27 06:19:59] ...optimizers/trijoint.py.151: total_norm: 16.382290 [I 2022-10-27 06:20:04] ...trap/engines/engine.py.224: train: epoch 52 | batch 870/2304 [I 2022-10-27 06:20:04] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:14 [I 2022-10-27 06:20:04] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00244 [I 2022-10-27 06:20:04] ...trap/engines/engine.py.230: loss: 0.12987 [I 2022-10-27 06:20:04] ...optimizers/trijoint.py.151: total_norm: 24.942612 [I 2022-10-27 06:20:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 880/2304 [I 2022-10-27 06:20:10] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:04 [I 2022-10-27 06:20:10] ...trap/engines/engine.py.229: process: 0.54224 | load: 0.00243 [I 2022-10-27 06:20:10] ...trap/engines/engine.py.230: loss: 0.16155 [I 2022-10-27 06:20:10] ...optimizers/trijoint.py.151: total_norm: 13.880853 [I 2022-10-27 06:20:15] ...trap/engines/engine.py.224: train: epoch 52 | batch 890/2304 [I 2022-10-27 06:20:15] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:02 [I 2022-10-27 06:20:15] ...trap/engines/engine.py.229: process: 0.54774 | load: 0.00240 [I 2022-10-27 06:20:15] ...trap/engines/engine.py.230: loss: 0.13693 [I 2022-10-27 06:20:15] ...optimizers/trijoint.py.151: total_norm: 13.005116 [I 2022-10-27 06:20:21] ...trap/engines/engine.py.224: train: epoch 52 | batch 900/2304 [I 2022-10-27 06:20:21] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:12:55 [I 2022-10-27 06:20:21] ...trap/engines/engine.py.229: process: 0.54272 | load: 0.00242 [I 2022-10-27 06:20:21] ...trap/engines/engine.py.230: loss: 0.14470 [I 2022-10-27 06:20:21] ...optimizers/trijoint.py.151: total_norm: 14.789239 [I 2022-10-27 06:20:26] ...trap/engines/engine.py.224: train: epoch 52 | batch 910/2304 [I 2022-10-27 06:20:26] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:12:51 [I 2022-10-27 06:20:26] ...trap/engines/engine.py.229: process: 0.55851 | load: 0.00239 [I 2022-10-27 06:20:26] ...trap/engines/engine.py.230: loss: 0.21597 [I 2022-10-27 06:20:26] ...optimizers/trijoint.py.151: total_norm: 12.196958 [I 2022-10-27 06:20:32] ...trap/engines/engine.py.224: train: epoch 52 | batch 920/2304 [I 2022-10-27 06:20:32] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:12:49 [I 2022-10-27 06:20:32] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00240 [I 2022-10-27 06:20:32] ...trap/engines/engine.py.230: loss: 0.17541 [I 2022-10-27 06:20:32] ...optimizers/trijoint.py.151: total_norm: 12.472981 [I 2022-10-27 06:20:38] ...trap/engines/engine.py.224: train: epoch 52 | batch 930/2304 [I 2022-10-27 06:20:38] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:12:46 [I 2022-10-27 06:20:38] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00249 [I 2022-10-27 06:20:38] ...trap/engines/engine.py.230: loss: 0.16067 [I 2022-10-27 06:20:38] ...optimizers/trijoint.py.151: total_norm: 12.527541 [I 2022-10-27 06:20:43] ...trap/engines/engine.py.224: train: epoch 52 | batch 940/2304 [I 2022-10-27 06:20:43] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:12:37 [I 2022-10-27 06:20:43] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00245 [I 2022-10-27 06:20:43] ...trap/engines/engine.py.230: loss: 0.31145 [I 2022-10-27 06:20:43] ...optimizers/trijoint.py.151: total_norm: 22.479158 [I 2022-10-27 06:20:49] ...trap/engines/engine.py.224: train: epoch 52 | batch 950/2304 [I 2022-10-27 06:20:49] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:31 [I 2022-10-27 06:20:49] ...trap/engines/engine.py.229: process: 0.54576 | load: 0.00242 [I 2022-10-27 06:20:49] ...trap/engines/engine.py.230: loss: 0.20219 [I 2022-10-27 06:20:49] ...optimizers/trijoint.py.151: total_norm: 15.743891 [I 2022-10-27 06:20:54] ...trap/engines/engine.py.224: train: epoch 52 | batch 960/2304 [I 2022-10-27 06:20:54] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:24 [I 2022-10-27 06:20:54] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00243 [I 2022-10-27 06:20:54] ...trap/engines/engine.py.230: loss: 0.18473 [I 2022-10-27 06:20:54] ...optimizers/trijoint.py.151: total_norm: 15.333391 [I 2022-10-27 06:21:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 970/2304 [I 2022-10-27 06:21:00] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:17 [I 2022-10-27 06:21:00] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00242 [I 2022-10-27 06:21:00] ...trap/engines/engine.py.230: loss: 0.14937 [I 2022-10-27 06:21:00] ...optimizers/trijoint.py.151: total_norm: 18.193428 [I 2022-10-27 06:21:05] ...trap/engines/engine.py.224: train: epoch 52 | batch 980/2304 [I 2022-10-27 06:21:05] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:12:11 [I 2022-10-27 06:21:05] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00244 [I 2022-10-27 06:21:05] ...trap/engines/engine.py.230: loss: 0.15941 [I 2022-10-27 06:21:05] ...optimizers/trijoint.py.151: total_norm: 18.743557 [I 2022-10-27 06:21:11] ...trap/engines/engine.py.224: train: epoch 52 | batch 990/2304 [I 2022-10-27 06:21:11] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:09 [I 2022-10-27 06:21:11] ...trap/engines/engine.py.229: process: 0.55040 | load: 0.00243 [I 2022-10-27 06:21:11] ...trap/engines/engine.py.230: loss: 0.12547 [I 2022-10-27 06:21:11] ...optimizers/trijoint.py.151: total_norm: 25.019215 [I 2022-10-27 06:21:17] ...trap/engines/engine.py.224: train: epoch 52 | batch 1000/2304 [I 2022-10-27 06:21:17] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:03 [I 2022-10-27 06:21:17] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00241 [I 2022-10-27 06:21:17] ...trap/engines/engine.py.230: loss: 0.13463 [I 2022-10-27 06:21:17] ...optimizers/trijoint.py.151: total_norm: 12.530663 [I 2022-10-27 06:21:22] ...trap/engines/engine.py.224: train: epoch 52 | batch 1010/2304 [I 2022-10-27 06:21:22] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:11:56 [I 2022-10-27 06:21:22] ...trap/engines/engine.py.229: process: 0.55057 | load: 0.00246 [I 2022-10-27 06:21:22] ...trap/engines/engine.py.230: loss: 0.16496 [I 2022-10-27 06:21:22] ...optimizers/trijoint.py.151: total_norm: 11.083923 [I 2022-10-27 06:21:28] ...trap/engines/engine.py.224: train: epoch 52 | batch 1020/2304 [I 2022-10-27 06:21:28] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:11:50 [I 2022-10-27 06:21:28] ...trap/engines/engine.py.229: process: 0.54082 | load: 0.00258 [I 2022-10-27 06:21:28] ...trap/engines/engine.py.230: loss: 0.17168 [I 2022-10-27 06:21:28] ...optimizers/trijoint.py.151: total_norm: 7.532626 [I 2022-10-27 06:21:33] ...trap/engines/engine.py.224: train: epoch 52 | batch 1030/2304 [I 2022-10-27 06:21:33] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:11:42 [I 2022-10-27 06:21:33] ...trap/engines/engine.py.229: process: 0.55672 | load: 0.00249 [I 2022-10-27 06:21:33] ...trap/engines/engine.py.230: loss: 0.16097 [I 2022-10-27 06:21:33] ...optimizers/trijoint.py.151: total_norm: 18.442410 [I 2022-10-27 06:21:39] ...trap/engines/engine.py.224: train: epoch 52 | batch 1040/2304 [I 2022-10-27 06:21:39] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:11:38 [I 2022-10-27 06:21:39] ...trap/engines/engine.py.229: process: 0.54565 | load: 0.00241 [I 2022-10-27 06:21:39] ...trap/engines/engine.py.230: loss: 0.14011 [I 2022-10-27 06:21:39] ...optimizers/trijoint.py.151: total_norm: 8.684001 [I 2022-10-27 06:21:44] ...trap/engines/engine.py.224: train: epoch 52 | batch 1050/2304 [I 2022-10-27 06:21:44] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:11:35 [I 2022-10-27 06:21:44] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00241 [I 2022-10-27 06:21:44] ...trap/engines/engine.py.230: loss: 0.11180 [I 2022-10-27 06:21:44] ...optimizers/trijoint.py.151: total_norm: 12.667521 [I 2022-10-27 06:21:50] ...trap/engines/engine.py.224: train: epoch 52 | batch 1060/2304 [I 2022-10-27 06:21:50] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:27 [I 2022-10-27 06:21:50] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00241 [I 2022-10-27 06:21:50] ...trap/engines/engine.py.230: loss: 0.15721 [I 2022-10-27 06:21:50] ...optimizers/trijoint.py.151: total_norm: 14.421325 [I 2022-10-27 06:21:55] ...trap/engines/engine.py.224: train: epoch 52 | batch 1070/2304 [I 2022-10-27 06:21:55] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:25 [I 2022-10-27 06:21:55] ...trap/engines/engine.py.229: process: 0.54811 | load: 0.00243 [I 2022-10-27 06:21:55] ...trap/engines/engine.py.230: loss: 0.17927 [I 2022-10-27 06:21:55] ...optimizers/trijoint.py.151: total_norm: 12.060846 [I 2022-10-27 06:22:01] ...trap/engines/engine.py.224: train: epoch 52 | batch 1080/2304 [I 2022-10-27 06:22:01] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:21 [I 2022-10-27 06:22:01] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00248 [I 2022-10-27 06:22:01] ...trap/engines/engine.py.230: loss: 0.13672 [I 2022-10-27 06:22:01] ...optimizers/trijoint.py.151: total_norm: 23.224600 [I 2022-10-27 06:22:07] ...trap/engines/engine.py.224: train: epoch 52 | batch 1090/2304 [I 2022-10-27 06:22:07] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:17 [I 2022-10-27 06:22:07] ...trap/engines/engine.py.229: process: 0.55168 | load: 0.00242 [I 2022-10-27 06:22:07] ...trap/engines/engine.py.230: loss: 0.11524 [I 2022-10-27 06:22:07] ...optimizers/trijoint.py.151: total_norm: 13.553215 [I 2022-10-27 06:22:12] ...trap/engines/engine.py.224: train: epoch 52 | batch 1100/2304 [I 2022-10-27 06:22:12] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:12 [I 2022-10-27 06:22:12] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00241 [I 2022-10-27 06:22:12] ...trap/engines/engine.py.230: loss: 0.13741 [I 2022-10-27 06:22:12] ...optimizers/trijoint.py.151: total_norm: 15.072460 [I 2022-10-27 06:22:18] ...trap/engines/engine.py.224: train: epoch 52 | batch 1110/2304 [I 2022-10-27 06:22:18] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:04 [I 2022-10-27 06:22:18] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00243 [I 2022-10-27 06:22:18] ...trap/engines/engine.py.230: loss: 0.13454 [I 2022-10-27 06:22:18] ...optimizers/trijoint.py.151: total_norm: 22.445856 [I 2022-10-27 06:22:23] ...trap/engines/engine.py.224: train: epoch 52 | batch 1120/2304 [I 2022-10-27 06:22:23] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:10:53 [I 2022-10-27 06:22:23] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00241 [I 2022-10-27 06:22:23] ...trap/engines/engine.py.230: loss: 0.21298 [I 2022-10-27 06:22:23] ...optimizers/trijoint.py.151: total_norm: 23.218140 [I 2022-10-27 06:22:29] ...trap/engines/engine.py.224: train: epoch 52 | batch 1130/2304 [I 2022-10-27 06:22:29] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:10:50 [I 2022-10-27 06:22:29] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00243 [I 2022-10-27 06:22:29] ...trap/engines/engine.py.230: loss: 0.13928 [I 2022-10-27 06:22:29] ...optimizers/trijoint.py.151: total_norm: 16.596071 [I 2022-10-27 06:22:34] ...trap/engines/engine.py.224: train: epoch 52 | batch 1140/2304 [I 2022-10-27 06:22:34] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:10:49 [I 2022-10-27 06:22:34] ...trap/engines/engine.py.229: process: 0.55597 | load: 0.00253 [I 2022-10-27 06:22:34] ...trap/engines/engine.py.230: loss: 0.13371 [I 2022-10-27 06:22:34] ...optimizers/trijoint.py.151: total_norm: 23.018866 [I 2022-10-27 06:22:40] ...trap/engines/engine.py.224: train: epoch 52 | batch 1150/2304 [I 2022-10-27 06:22:40] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:10:38 [I 2022-10-27 06:22:40] ...trap/engines/engine.py.229: process: 0.54231 | load: 0.00247 [I 2022-10-27 06:22:40] ...trap/engines/engine.py.230: loss: 0.15489 [I 2022-10-27 06:22:40] ...optimizers/trijoint.py.151: total_norm: 25.504911 [I 2022-10-27 06:22:46] ...trap/engines/engine.py.224: train: epoch 52 | batch 1160/2304 [I 2022-10-27 06:22:46] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:31 [I 2022-10-27 06:22:46] ...trap/engines/engine.py.229: process: 0.55354 | load: 0.00249 [I 2022-10-27 06:22:46] ...trap/engines/engine.py.230: loss: 0.16351 [I 2022-10-27 06:22:46] ...optimizers/trijoint.py.151: total_norm: 13.535021 [I 2022-10-27 06:22:51] ...trap/engines/engine.py.224: train: epoch 52 | batch 1170/2304 [I 2022-10-27 06:22:51] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:30 [I 2022-10-27 06:22:51] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00248 [I 2022-10-27 06:22:51] ...trap/engines/engine.py.230: loss: 0.17151 [I 2022-10-27 06:22:51] ...optimizers/trijoint.py.151: total_norm: 23.486710 [I 2022-10-27 06:22:57] ...trap/engines/engine.py.224: train: epoch 52 | batch 1180/2304 [I 2022-10-27 06:22:57] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:19 [I 2022-10-27 06:22:57] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00247 [I 2022-10-27 06:22:57] ...trap/engines/engine.py.230: loss: 0.12233 [I 2022-10-27 06:22:57] ...optimizers/trijoint.py.151: total_norm: 29.884693 [I 2022-10-27 06:23:02] ...trap/engines/engine.py.224: train: epoch 52 | batch 1190/2304 [I 2022-10-27 06:23:02] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:15 [I 2022-10-27 06:23:02] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00244 [I 2022-10-27 06:23:02] ...trap/engines/engine.py.230: loss: 0.16912 [I 2022-10-27 06:23:02] ...optimizers/trijoint.py.151: total_norm: 17.577421 [I 2022-10-27 06:23:08] ...trap/engines/engine.py.224: train: epoch 52 | batch 1200/2304 [I 2022-10-27 06:23:08] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:14 [I 2022-10-27 06:23:08] ...trap/engines/engine.py.229: process: 0.56245 | load: 0.00241 [I 2022-10-27 06:23:08] ...trap/engines/engine.py.230: loss: 0.17871 [I 2022-10-27 06:23:08] ...optimizers/trijoint.py.151: total_norm: 21.106562 [I 2022-10-27 06:23:13] ...trap/engines/engine.py.224: train: epoch 52 | batch 1210/2304 [I 2022-10-27 06:23:13] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:07 [I 2022-10-27 06:23:13] ...trap/engines/engine.py.229: process: 0.55602 | load: 0.00244 [I 2022-10-27 06:23:13] ...trap/engines/engine.py.230: loss: 0.18959 [I 2022-10-27 06:23:13] ...optimizers/trijoint.py.151: total_norm: 19.122837 [I 2022-10-27 06:23:19] ...trap/engines/engine.py.224: train: epoch 52 | batch 1220/2304 [I 2022-10-27 06:23:19] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:00 [I 2022-10-27 06:23:19] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00243 [I 2022-10-27 06:23:19] ...trap/engines/engine.py.230: loss: 0.19403 [I 2022-10-27 06:23:19] ...optimizers/trijoint.py.151: total_norm: 30.107544 [I 2022-10-27 06:23:24] ...trap/engines/engine.py.224: train: epoch 52 | batch 1230/2304 [I 2022-10-27 06:23:24] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:09:55 [I 2022-10-27 06:23:24] ...trap/engines/engine.py.229: process: 0.55400 | load: 0.00243 [I 2022-10-27 06:23:24] ...trap/engines/engine.py.230: loss: 0.15461 [I 2022-10-27 06:23:24] ...optimizers/trijoint.py.151: total_norm: 14.840995 [I 2022-10-27 06:23:30] ...trap/engines/engine.py.224: train: epoch 52 | batch 1240/2304 [I 2022-10-27 06:23:30] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:09:51 [I 2022-10-27 06:23:30] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00245 [I 2022-10-27 06:23:30] ...trap/engines/engine.py.230: loss: 0.13252 [I 2022-10-27 06:23:30] ...optimizers/trijoint.py.151: total_norm: 19.598333 [I 2022-10-27 06:23:35] ...trap/engines/engine.py.224: train: epoch 52 | batch 1250/2304 [I 2022-10-27 06:23:35] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:09:45 [I 2022-10-27 06:23:35] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00248 [I 2022-10-27 06:23:35] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-27 06:23:35] ...optimizers/trijoint.py.151: total_norm: 7.266222 [I 2022-10-27 06:23:41] ...trap/engines/engine.py.224: train: epoch 52 | batch 1260/2304 [I 2022-10-27 06:23:41] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:09:32 [I 2022-10-27 06:23:41] ...trap/engines/engine.py.229: process: 0.53635 | load: 0.00247 [I 2022-10-27 06:23:41] ...trap/engines/engine.py.230: loss: 0.21100 [I 2022-10-27 06:23:41] ...optimizers/trijoint.py.151: total_norm: 18.618513 [I 2022-10-27 06:23:47] ...trap/engines/engine.py.224: train: epoch 52 | batch 1270/2304 [I 2022-10-27 06:23:47] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:36 [I 2022-10-27 06:23:47] ...trap/engines/engine.py.229: process: 0.57456 | load: 0.00247 [I 2022-10-27 06:23:47] ...trap/engines/engine.py.230: loss: 0.19681 [I 2022-10-27 06:23:47] ...optimizers/trijoint.py.151: total_norm: 15.031209 [I 2022-10-27 06:23:52] ...trap/engines/engine.py.224: train: epoch 52 | batch 1280/2304 [I 2022-10-27 06:23:52] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:27 [I 2022-10-27 06:23:52] ...trap/engines/engine.py.229: process: 0.55231 | load: 0.00258 [I 2022-10-27 06:23:52] ...trap/engines/engine.py.230: loss: 0.18586 [I 2022-10-27 06:23:52] ...optimizers/trijoint.py.151: total_norm: 46.506191 [I 2022-10-27 06:23:58] ...trap/engines/engine.py.224: train: epoch 52 | batch 1290/2304 [I 2022-10-27 06:23:58] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:24 [I 2022-10-27 06:23:58] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00247 [I 2022-10-27 06:23:58] ...trap/engines/engine.py.230: loss: 0.16505 [I 2022-10-27 06:23:58] ...optimizers/trijoint.py.151: total_norm: 41.417385 [I 2022-10-27 06:24:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 1300/2304 [I 2022-10-27 06:24:03] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:15 [I 2022-10-27 06:24:03] ...trap/engines/engine.py.229: process: 0.54684 | load: 0.00244 [I 2022-10-27 06:24:03] ...trap/engines/engine.py.230: loss: 0.13520 [I 2022-10-27 06:24:03] ...optimizers/trijoint.py.151: total_norm: 10.229666 [I 2022-10-27 06:24:09] ...trap/engines/engine.py.224: train: epoch 52 | batch 1310/2304 [I 2022-10-27 06:24:09] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:11 [I 2022-10-27 06:24:09] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00245 [I 2022-10-27 06:24:09] ...trap/engines/engine.py.230: loss: 0.20774 [I 2022-10-27 06:24:09] ...optimizers/trijoint.py.151: total_norm: 26.653772 [I 2022-10-27 06:24:14] ...trap/engines/engine.py.224: train: epoch 52 | batch 1320/2304 [I 2022-10-27 06:24:14] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:02 [I 2022-10-27 06:24:14] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00243 [I 2022-10-27 06:24:14] ...trap/engines/engine.py.230: loss: 0.13799 [I 2022-10-27 06:24:14] ...optimizers/trijoint.py.151: total_norm: 13.442083 [I 2022-10-27 06:24:20] ...trap/engines/engine.py.224: train: epoch 52 | batch 1330/2304 [I 2022-10-27 06:24:20] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:08:57 [I 2022-10-27 06:24:20] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00241 [I 2022-10-27 06:24:20] ...trap/engines/engine.py.230: loss: 0.17231 [I 2022-10-27 06:24:20] ...optimizers/trijoint.py.151: total_norm: 19.583796 [I 2022-10-27 06:24:25] ...trap/engines/engine.py.224: train: epoch 52 | batch 1340/2304 [I 2022-10-27 06:24:25] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:08:57 [I 2022-10-27 06:24:25] ...trap/engines/engine.py.229: process: 0.56460 | load: 0.00243 [I 2022-10-27 06:24:25] ...trap/engines/engine.py.230: loss: 0.15956 [I 2022-10-27 06:24:25] ...optimizers/trijoint.py.151: total_norm: 23.508417 [I 2022-10-27 06:24:31] ...trap/engines/engine.py.224: train: epoch 52 | batch 1350/2304 [I 2022-10-27 06:24:31] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:08:50 [I 2022-10-27 06:24:31] ...trap/engines/engine.py.229: process: 0.56512 | load: 0.00242 [I 2022-10-27 06:24:31] ...trap/engines/engine.py.230: loss: 0.12885 [I 2022-10-27 06:24:31] ...optimizers/trijoint.py.151: total_norm: 15.844387 [I 2022-10-27 06:24:37] ...trap/engines/engine.py.224: train: epoch 52 | batch 1360/2304 [I 2022-10-27 06:24:37] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:08:42 [I 2022-10-27 06:24:37] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00243 [I 2022-10-27 06:24:37] ...trap/engines/engine.py.230: loss: 0.12431 [I 2022-10-27 06:24:37] ...optimizers/trijoint.py.151: total_norm: 14.915721 [I 2022-10-27 06:24:42] ...trap/engines/engine.py.224: train: epoch 52 | batch 1370/2304 [I 2022-10-27 06:24:42] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:08:35 [I 2022-10-27 06:24:42] ...trap/engines/engine.py.229: process: 0.54450 | load: 0.00240 [I 2022-10-27 06:24:42] ...trap/engines/engine.py.230: loss: 0.18494 [I 2022-10-27 06:24:42] ...optimizers/trijoint.py.151: total_norm: 15.404209 [I 2022-10-27 06:24:48] ...trap/engines/engine.py.224: train: epoch 52 | batch 1380/2304 [I 2022-10-27 06:24:48] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:32 [I 2022-10-27 06:24:48] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00244 [I 2022-10-27 06:24:48] ...trap/engines/engine.py.230: loss: 0.15584 [I 2022-10-27 06:24:48] ...optimizers/trijoint.py.151: total_norm: 10.368512 [I 2022-10-27 06:24:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 1390/2304 [I 2022-10-27 06:24:53] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:29 [I 2022-10-27 06:24:53] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00240 [I 2022-10-27 06:24:53] ...trap/engines/engine.py.230: loss: 0.17558 [I 2022-10-27 06:24:53] ...optimizers/trijoint.py.151: total_norm: 34.264729 [I 2022-10-27 06:24:59] ...trap/engines/engine.py.224: train: epoch 52 | batch 1400/2304 [I 2022-10-27 06:24:59] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:18 [I 2022-10-27 06:24:59] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00242 [I 2022-10-27 06:24:59] ...trap/engines/engine.py.230: loss: 0.24100 [I 2022-10-27 06:24:59] ...optimizers/trijoint.py.151: total_norm: 20.057072 [I 2022-10-27 06:25:04] ...trap/engines/engine.py.224: train: epoch 52 | batch 1410/2304 [I 2022-10-27 06:25:04] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:08:16 [I 2022-10-27 06:25:04] ...trap/engines/engine.py.229: process: 0.55339 | load: 0.00239 [I 2022-10-27 06:25:04] ...trap/engines/engine.py.230: loss: 0.20231 [I 2022-10-27 06:25:04] ...optimizers/trijoint.py.151: total_norm: 10.137030 [I 2022-10-27 06:25:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 1420/2304 [I 2022-10-27 06:25:10] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:10 [I 2022-10-27 06:25:10] ...trap/engines/engine.py.229: process: 0.56828 | load: 0.00252 [I 2022-10-27 06:25:10] ...trap/engines/engine.py.230: loss: 0.13838 [I 2022-10-27 06:25:10] ...optimizers/trijoint.py.151: total_norm: 14.215402 [I 2022-10-27 06:25:15] ...trap/engines/engine.py.224: train: epoch 52 | batch 1430/2304 [I 2022-10-27 06:25:15] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:03 [I 2022-10-27 06:25:15] ...trap/engines/engine.py.229: process: 0.54336 | load: 0.00246 [I 2022-10-27 06:25:15] ...trap/engines/engine.py.230: loss: 0.18751 [I 2022-10-27 06:25:15] ...optimizers/trijoint.py.151: total_norm: 23.859617 [I 2022-10-27 06:25:21] ...trap/engines/engine.py.224: train: epoch 52 | batch 1440/2304 [I 2022-10-27 06:25:21] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:07:56 [I 2022-10-27 06:25:21] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00245 [I 2022-10-27 06:25:21] ...trap/engines/engine.py.230: loss: 0.18469 [I 2022-10-27 06:25:21] ...optimizers/trijoint.py.151: total_norm: 54.914398 [I 2022-10-27 06:25:26] ...trap/engines/engine.py.224: train: epoch 52 | batch 1450/2304 [I 2022-10-27 06:25:26] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:07:51 [I 2022-10-27 06:25:26] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00245 [I 2022-10-27 06:25:26] ...trap/engines/engine.py.230: loss: 0.14563 [I 2022-10-27 06:25:26] ...optimizers/trijoint.py.151: total_norm: 22.920687 [I 2022-10-27 06:25:32] ...trap/engines/engine.py.224: train: epoch 52 | batch 1460/2304 [I 2022-10-27 06:25:32] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:07:47 [I 2022-10-27 06:25:32] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00243 [I 2022-10-27 06:25:32] ...trap/engines/engine.py.230: loss: 0.20342 [I 2022-10-27 06:25:32] ...optimizers/trijoint.py.151: total_norm: 23.786106 [I 2022-10-27 06:25:38] ...trap/engines/engine.py.224: train: epoch 52 | batch 1470/2304 [I 2022-10-27 06:25:38] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:07:42 [I 2022-10-27 06:25:38] ...trap/engines/engine.py.229: process: 0.55141 | load: 0.00237 [I 2022-10-27 06:25:38] ...trap/engines/engine.py.230: loss: 0.15157 [I 2022-10-27 06:25:38] ...optimizers/trijoint.py.151: total_norm: 17.475639 [I 2022-10-27 06:25:43] ...trap/engines/engine.py.224: train: epoch 52 | batch 1480/2304 [I 2022-10-27 06:25:43] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:47 [I 2022-10-27 06:25:43] ...trap/engines/engine.py.229: process: 0.60570 | load: 0.00246 [I 2022-10-27 06:25:43] ...trap/engines/engine.py.230: loss: 0.12677 [I 2022-10-27 06:25:43] ...optimizers/trijoint.py.151: total_norm: 9.566939 [I 2022-10-27 06:25:49] ...trap/engines/engine.py.224: train: epoch 52 | batch 1490/2304 [I 2022-10-27 06:25:49] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:35 [I 2022-10-27 06:25:49] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00241 [I 2022-10-27 06:25:49] ...trap/engines/engine.py.230: loss: 0.17421 [I 2022-10-27 06:25:49] ...optimizers/trijoint.py.151: total_norm: 27.881916 [I 2022-10-27 06:25:55] ...trap/engines/engine.py.224: train: epoch 52 | batch 1500/2304 [I 2022-10-27 06:25:55] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:34 [I 2022-10-27 06:25:55] ...trap/engines/engine.py.229: process: 0.62069 | load: 0.00238 [I 2022-10-27 06:25:55] ...trap/engines/engine.py.230: loss: 0.19005 [I 2022-10-27 06:25:55] ...optimizers/trijoint.py.151: total_norm: 12.653311 [I 2022-10-27 06:26:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 1510/2304 [I 2022-10-27 06:26:00] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:07:22 [I 2022-10-27 06:26:00] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00249 [I 2022-10-27 06:26:00] ...trap/engines/engine.py.230: loss: 0.14515 [I 2022-10-27 06:26:00] ...optimizers/trijoint.py.151: total_norm: 11.735122 [I 2022-10-27 06:26:06] ...trap/engines/engine.py.224: train: epoch 52 | batch 1520/2304 [I 2022-10-27 06:26:06] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:18 [I 2022-10-27 06:26:06] ...trap/engines/engine.py.229: process: 0.57042 | load: 0.00251 [I 2022-10-27 06:26:06] ...trap/engines/engine.py.230: loss: 0.09936 [I 2022-10-27 06:26:06] ...optimizers/trijoint.py.151: total_norm: 7.734646 [I 2022-10-27 06:26:11] ...trap/engines/engine.py.224: train: epoch 52 | batch 1530/2304 [I 2022-10-27 06:26:11] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:07:09 [I 2022-10-27 06:26:11] ...trap/engines/engine.py.229: process: 0.54527 | load: 0.00248 [I 2022-10-27 06:26:11] ...trap/engines/engine.py.230: loss: 0.19517 [I 2022-10-27 06:26:11] ...optimizers/trijoint.py.151: total_norm: 19.913477 [I 2022-10-27 06:26:17] ...trap/engines/engine.py.224: train: epoch 52 | batch 1540/2304 [I 2022-10-27 06:26:17] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:00 [I 2022-10-27 06:26:17] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00246 [I 2022-10-27 06:26:17] ...trap/engines/engine.py.230: loss: 0.14931 [I 2022-10-27 06:26:17] ...optimizers/trijoint.py.151: total_norm: 9.478630 [I 2022-10-27 06:26:22] ...trap/engines/engine.py.224: train: epoch 52 | batch 1550/2304 [I 2022-10-27 06:26:22] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:06:54 [I 2022-10-27 06:26:22] ...trap/engines/engine.py.229: process: 0.54441 | load: 0.00244 [I 2022-10-27 06:26:22] ...trap/engines/engine.py.230: loss: 0.11391 [I 2022-10-27 06:26:22] ...optimizers/trijoint.py.151: total_norm: 16.646252 [I 2022-10-27 06:26:28] ...trap/engines/engine.py.224: train: epoch 52 | batch 1560/2304 [I 2022-10-27 06:26:28] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:06:49 [I 2022-10-27 06:26:28] ...trap/engines/engine.py.229: process: 0.54319 | load: 0.00244 [I 2022-10-27 06:26:28] ...trap/engines/engine.py.230: loss: 0.14065 [I 2022-10-27 06:26:28] ...optimizers/trijoint.py.151: total_norm: 10.389597 [I 2022-10-27 06:26:33] ...trap/engines/engine.py.224: train: epoch 52 | batch 1570/2304 [I 2022-10-27 06:26:33] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:51 [I 2022-10-27 06:26:33] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00242 [I 2022-10-27 06:26:33] ...trap/engines/engine.py.230: loss: 0.16442 [I 2022-10-27 06:26:33] ...optimizers/trijoint.py.151: total_norm: 30.351101 [I 2022-10-27 06:26:39] ...trap/engines/engine.py.224: train: epoch 52 | batch 1580/2304 [I 2022-10-27 06:26:39] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:06:40 [I 2022-10-27 06:26:39] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00244 [I 2022-10-27 06:26:39] ...trap/engines/engine.py.230: loss: 0.23363 [I 2022-10-27 06:26:39] ...optimizers/trijoint.py.151: total_norm: 11.345996 [I 2022-10-27 06:26:45] ...trap/engines/engine.py.224: train: epoch 52 | batch 1590/2304 [I 2022-10-27 06:26:45] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:06:35 [I 2022-10-27 06:26:45] ...trap/engines/engine.py.229: process: 0.54446 | load: 0.00240 [I 2022-10-27 06:26:45] ...trap/engines/engine.py.230: loss: 0.21930 [I 2022-10-27 06:26:45] ...optimizers/trijoint.py.151: total_norm: 29.055397 [I 2022-10-27 06:26:50] ...trap/engines/engine.py.224: train: epoch 52 | batch 1600/2304 [I 2022-10-27 06:26:50] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:31 [I 2022-10-27 06:26:50] ...trap/engines/engine.py.229: process: 0.56412 | load: 0.00243 [I 2022-10-27 06:26:50] ...trap/engines/engine.py.230: loss: 0.14273 [I 2022-10-27 06:26:50] ...optimizers/trijoint.py.151: total_norm: 13.091466 [I 2022-10-27 06:26:56] ...trap/engines/engine.py.224: train: epoch 52 | batch 1610/2304 [I 2022-10-27 06:26:56] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:25 [I 2022-10-27 06:26:56] ...trap/engines/engine.py.229: process: 0.54789 | load: 0.00241 [I 2022-10-27 06:26:56] ...trap/engines/engine.py.230: loss: 0.21757 [I 2022-10-27 06:26:56] ...optimizers/trijoint.py.151: total_norm: 30.741814 [I 2022-10-27 06:27:01] ...trap/engines/engine.py.224: train: epoch 52 | batch 1620/2304 [I 2022-10-27 06:27:01] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:20 [I 2022-10-27 06:27:01] ...trap/engines/engine.py.229: process: 0.54516 | load: 0.00270 [I 2022-10-27 06:27:01] ...trap/engines/engine.py.230: loss: 0.12819 [I 2022-10-27 06:27:01] ...optimizers/trijoint.py.151: total_norm: 15.472162 [I 2022-10-27 06:27:07] ...trap/engines/engine.py.224: train: epoch 52 | batch 1630/2304 [I 2022-10-27 06:27:07] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:13 [I 2022-10-27 06:27:07] ...trap/engines/engine.py.229: process: 0.55152 | load: 0.00244 [I 2022-10-27 06:27:07] ...trap/engines/engine.py.230: loss: 0.12429 [I 2022-10-27 06:27:07] ...optimizers/trijoint.py.151: total_norm: 8.289287 [I 2022-10-27 06:27:13] ...trap/engines/engine.py.224: train: epoch 52 | batch 1640/2304 [I 2022-10-27 06:27:13] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:13 [I 2022-10-27 06:27:13] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00244 [I 2022-10-27 06:27:13] ...trap/engines/engine.py.230: loss: 0.23368 [I 2022-10-27 06:27:13] ...optimizers/trijoint.py.151: total_norm: 19.718096 [I 2022-10-27 06:27:18] ...trap/engines/engine.py.224: train: epoch 52 | batch 1650/2304 [I 2022-10-27 06:27:18] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:01 [I 2022-10-27 06:27:18] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00244 [I 2022-10-27 06:27:18] ...trap/engines/engine.py.230: loss: 0.14212 [I 2022-10-27 06:27:18] ...optimizers/trijoint.py.151: total_norm: 14.946502 [I 2022-10-27 06:27:24] ...trap/engines/engine.py.224: train: epoch 52 | batch 1660/2304 [I 2022-10-27 06:27:24] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:05:57 [I 2022-10-27 06:27:24] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00251 [I 2022-10-27 06:27:24] ...trap/engines/engine.py.230: loss: 0.18505 [I 2022-10-27 06:27:24] ...optimizers/trijoint.py.151: total_norm: 26.123720 [I 2022-10-27 06:27:29] ...trap/engines/engine.py.224: train: epoch 52 | batch 1670/2304 [I 2022-10-27 06:27:29] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:05:54 [I 2022-10-27 06:27:29] ...trap/engines/engine.py.229: process: 0.56595 | load: 0.00246 [I 2022-10-27 06:27:29] ...trap/engines/engine.py.230: loss: 0.15731 [I 2022-10-27 06:27:29] ...optimizers/trijoint.py.151: total_norm: 26.609180 [I 2022-10-27 06:27:35] ...trap/engines/engine.py.224: train: epoch 52 | batch 1680/2304 [I 2022-10-27 06:27:35] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:46 [I 2022-10-27 06:27:35] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00246 [I 2022-10-27 06:27:35] ...trap/engines/engine.py.230: loss: 0.19931 [I 2022-10-27 06:27:35] ...optimizers/trijoint.py.151: total_norm: 19.383821 [I 2022-10-27 06:27:40] ...trap/engines/engine.py.224: train: epoch 52 | batch 1690/2304 [I 2022-10-27 06:27:40] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:41 [I 2022-10-27 06:27:40] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00251 [I 2022-10-27 06:27:40] ...trap/engines/engine.py.230: loss: 0.22658 [I 2022-10-27 06:27:40] ...optimizers/trijoint.py.151: total_norm: 13.334404 [I 2022-10-27 06:27:46] ...trap/engines/engine.py.224: train: epoch 52 | batch 1700/2304 [I 2022-10-27 06:27:46] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:40 [I 2022-10-27 06:27:46] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00249 [I 2022-10-27 06:27:46] ...trap/engines/engine.py.230: loss: 0.22221 [I 2022-10-27 06:27:46] ...optimizers/trijoint.py.151: total_norm: 19.420702 [I 2022-10-27 06:27:52] ...trap/engines/engine.py.224: train: epoch 52 | batch 1710/2304 [I 2022-10-27 06:27:52] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:29 [I 2022-10-27 06:27:52] ...trap/engines/engine.py.229: process: 0.55214 | load: 0.00246 [I 2022-10-27 06:27:52] ...trap/engines/engine.py.230: loss: 0.17575 [I 2022-10-27 06:27:52] ...optimizers/trijoint.py.151: total_norm: 15.465753 [I 2022-10-27 06:27:57] ...trap/engines/engine.py.224: train: epoch 52 | batch 1720/2304 [I 2022-10-27 06:27:57] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:23 [I 2022-10-27 06:27:57] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00241 [I 2022-10-27 06:27:57] ...trap/engines/engine.py.230: loss: 0.16173 [I 2022-10-27 06:27:57] ...optimizers/trijoint.py.151: total_norm: 18.997219 [I 2022-10-27 06:28:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 1730/2304 [I 2022-10-27 06:28:03] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:17 [I 2022-10-27 06:28:03] ...trap/engines/engine.py.229: process: 0.53791 | load: 0.00243 [I 2022-10-27 06:28:03] ...trap/engines/engine.py.230: loss: 0.12067 [I 2022-10-27 06:28:03] ...optimizers/trijoint.py.151: total_norm: 10.654854 [I 2022-10-27 06:28:08] ...trap/engines/engine.py.224: train: epoch 52 | batch 1740/2304 [I 2022-10-27 06:28:08] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:11 [I 2022-10-27 06:28:08] ...trap/engines/engine.py.229: process: 0.55292 | load: 0.00239 [I 2022-10-27 06:28:08] ...trap/engines/engine.py.230: loss: 0.11987 [I 2022-10-27 06:28:08] ...optimizers/trijoint.py.151: total_norm: 21.782721 [I 2022-10-27 06:28:14] ...trap/engines/engine.py.224: train: epoch 52 | batch 1750/2304 [I 2022-10-27 06:28:14] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:05 [I 2022-10-27 06:28:14] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00244 [I 2022-10-27 06:28:14] ...trap/engines/engine.py.230: loss: 0.17608 [I 2022-10-27 06:28:14] ...optimizers/trijoint.py.151: total_norm: 20.637089 [I 2022-10-27 06:28:19] ...trap/engines/engine.py.224: train: epoch 52 | batch 1760/2304 [I 2022-10-27 06:28:19] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:12 [I 2022-10-27 06:28:19] ...trap/engines/engine.py.229: process: 0.57317 | load: 0.00244 [I 2022-10-27 06:28:19] ...trap/engines/engine.py.230: loss: 0.26404 [I 2022-10-27 06:28:19] ...optimizers/trijoint.py.151: total_norm: 23.855324 [I 2022-10-27 06:28:25] ...trap/engines/engine.py.224: train: epoch 52 | batch 1770/2304 [I 2022-10-27 06:28:25] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:04:57 [I 2022-10-27 06:28:25] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00240 [I 2022-10-27 06:28:25] ...trap/engines/engine.py.230: loss: 0.13043 [I 2022-10-27 06:28:25] ...optimizers/trijoint.py.151: total_norm: 24.013874 [I 2022-10-27 06:28:31] ...trap/engines/engine.py.224: train: epoch 52 | batch 1780/2304 [I 2022-10-27 06:28:31] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:04:51 [I 2022-10-27 06:28:31] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00243 [I 2022-10-27 06:28:31] ...trap/engines/engine.py.230: loss: 0.13948 [I 2022-10-27 06:28:31] ...optimizers/trijoint.py.151: total_norm: 13.107023 [I 2022-10-27 06:28:36] ...trap/engines/engine.py.224: train: epoch 52 | batch 1790/2304 [I 2022-10-27 06:28:36] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:45 [I 2022-10-27 06:28:36] ...trap/engines/engine.py.229: process: 0.54615 | load: 0.00246 [I 2022-10-27 06:28:36] ...trap/engines/engine.py.230: loss: 0.18110 [I 2022-10-27 06:28:36] ...optimizers/trijoint.py.151: total_norm: 15.332445 [I 2022-10-27 06:28:42] ...trap/engines/engine.py.224: train: epoch 52 | batch 1800/2304 [I 2022-10-27 06:28:42] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:40 [I 2022-10-27 06:28:42] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00244 [I 2022-10-27 06:28:42] ...trap/engines/engine.py.230: loss: 0.14341 [I 2022-10-27 06:28:42] ...optimizers/trijoint.py.151: total_norm: 10.425927 [I 2022-10-27 06:28:47] ...trap/engines/engine.py.224: train: epoch 52 | batch 1810/2304 [I 2022-10-27 06:28:47] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:34 [I 2022-10-27 06:28:47] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00249 [I 2022-10-27 06:28:47] ...trap/engines/engine.py.230: loss: 0.23399 [I 2022-10-27 06:28:47] ...optimizers/trijoint.py.151: total_norm: 44.080502 [I 2022-10-27 06:28:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 1820/2304 [I 2022-10-27 06:28:53] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:28 [I 2022-10-27 06:28:53] ...trap/engines/engine.py.229: process: 0.55720 | load: 0.00243 [I 2022-10-27 06:28:53] ...trap/engines/engine.py.230: loss: 0.16100 [I 2022-10-27 06:28:53] ...optimizers/trijoint.py.151: total_norm: 12.458228 [I 2022-10-27 06:28:59] ...trap/engines/engine.py.224: train: epoch 52 | batch 1830/2304 [I 2022-10-27 06:28:59] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:25 [I 2022-10-27 06:28:59] ...trap/engines/engine.py.229: process: 0.54975 | load: 0.00240 [I 2022-10-27 06:28:59] ...trap/engines/engine.py.230: loss: 0.16720 [I 2022-10-27 06:28:59] ...optimizers/trijoint.py.151: total_norm: 18.007725 [I 2022-10-27 06:29:04] ...trap/engines/engine.py.224: train: epoch 52 | batch 1840/2304 [I 2022-10-27 06:29:04] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:18 [I 2022-10-27 06:29:04] ...trap/engines/engine.py.229: process: 0.54687 | load: 0.00245 [I 2022-10-27 06:29:04] ...trap/engines/engine.py.230: loss: 0.10855 [I 2022-10-27 06:29:04] ...optimizers/trijoint.py.151: total_norm: 7.738553 [I 2022-10-27 06:29:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 1850/2304 [I 2022-10-27 06:29:10] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:12 [I 2022-10-27 06:29:10] ...trap/engines/engine.py.229: process: 0.55770 | load: 0.00260 [I 2022-10-27 06:29:10] ...trap/engines/engine.py.230: loss: 0.16764 [I 2022-10-27 06:29:10] ...optimizers/trijoint.py.151: total_norm: 14.562916 [I 2022-10-27 06:29:15] ...trap/engines/engine.py.224: train: epoch 52 | batch 1860/2304 [I 2022-10-27 06:29:15] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:07 [I 2022-10-27 06:29:15] ...trap/engines/engine.py.229: process: 0.57048 | load: 0.00241 [I 2022-10-27 06:29:15] ...trap/engines/engine.py.230: loss: 0.15370 [I 2022-10-27 06:29:15] ...optimizers/trijoint.py.151: total_norm: 8.738348 [I 2022-10-27 06:29:21] ...trap/engines/engine.py.224: train: epoch 52 | batch 1870/2304 [I 2022-10-27 06:29:21] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:02 [I 2022-10-27 06:29:21] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00248 [I 2022-10-27 06:29:21] ...trap/engines/engine.py.230: loss: 0.18064 [I 2022-10-27 06:29:21] ...optimizers/trijoint.py.151: total_norm: 16.213232 [I 2022-10-27 06:29:26] ...trap/engines/engine.py.224: train: epoch 52 | batch 1880/2304 [I 2022-10-27 06:29:26] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:03:57 [I 2022-10-27 06:29:26] ...trap/engines/engine.py.229: process: 0.56879 | load: 0.00244 [I 2022-10-27 06:29:26] ...trap/engines/engine.py.230: loss: 0.16560 [I 2022-10-27 06:29:26] ...optimizers/trijoint.py.151: total_norm: 17.290785 [I 2022-10-27 06:29:32] ...trap/engines/engine.py.224: train: epoch 52 | batch 1890/2304 [I 2022-10-27 06:29:32] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:03:52 [I 2022-10-27 06:29:32] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00245 [I 2022-10-27 06:29:32] ...trap/engines/engine.py.230: loss: 0.17700 [I 2022-10-27 06:29:32] ...optimizers/trijoint.py.151: total_norm: 11.572375 [I 2022-10-27 06:29:38] ...trap/engines/engine.py.224: train: epoch 52 | batch 1900/2304 [I 2022-10-27 06:29:38] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:45 [I 2022-10-27 06:29:38] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00242 [I 2022-10-27 06:29:38] ...trap/engines/engine.py.230: loss: 0.12104 [I 2022-10-27 06:29:38] ...optimizers/trijoint.py.151: total_norm: 14.935038 [I 2022-10-27 06:29:43] ...trap/engines/engine.py.224: train: epoch 52 | batch 1910/2304 [I 2022-10-27 06:29:43] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:40 [I 2022-10-27 06:29:43] ...trap/engines/engine.py.229: process: 0.57583 | load: 0.00246 [I 2022-10-27 06:29:43] ...trap/engines/engine.py.230: loss: 0.22138 [I 2022-10-27 06:29:43] ...optimizers/trijoint.py.151: total_norm: 12.105436 [I 2022-10-27 06:29:49] ...trap/engines/engine.py.224: train: epoch 52 | batch 1920/2304 [I 2022-10-27 06:29:49] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:33 [I 2022-10-27 06:29:49] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00244 [I 2022-10-27 06:29:49] ...trap/engines/engine.py.230: loss: 0.23974 [I 2022-10-27 06:29:49] ...optimizers/trijoint.py.151: total_norm: 31.431351 [I 2022-10-27 06:29:54] ...trap/engines/engine.py.224: train: epoch 52 | batch 1930/2304 [I 2022-10-27 06:29:54] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:28 [I 2022-10-27 06:29:54] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00260 [I 2022-10-27 06:29:54] ...trap/engines/engine.py.230: loss: 0.16362 [I 2022-10-27 06:29:54] ...optimizers/trijoint.py.151: total_norm: 18.258961 [I 2022-10-27 06:30:00] ...trap/engines/engine.py.224: train: epoch 52 | batch 1940/2304 [I 2022-10-27 06:30:00] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:23 [I 2022-10-27 06:30:00] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00249 [I 2022-10-27 06:30:00] ...trap/engines/engine.py.230: loss: 0.11637 [I 2022-10-27 06:30:00] ...optimizers/trijoint.py.151: total_norm: 14.533192 [I 2022-10-27 06:30:06] ...trap/engines/engine.py.224: train: epoch 52 | batch 1950/2304 [I 2022-10-27 06:30:06] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:23 [I 2022-10-27 06:30:06] ...trap/engines/engine.py.229: process: 0.55819 | load: 0.00246 [I 2022-10-27 06:30:06] ...trap/engines/engine.py.230: loss: 0.16361 [I 2022-10-27 06:30:06] ...optimizers/trijoint.py.151: total_norm: 22.746799 [I 2022-10-27 06:30:11] ...trap/engines/engine.py.224: train: epoch 52 | batch 1960/2304 [I 2022-10-27 06:30:11] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:09 [I 2022-10-27 06:30:11] ...trap/engines/engine.py.229: process: 0.54685 | load: 0.00243 [I 2022-10-27 06:30:11] ...trap/engines/engine.py.230: loss: 0.11944 [I 2022-10-27 06:30:11] ...optimizers/trijoint.py.151: total_norm: 11.521620 [I 2022-10-27 06:30:17] ...trap/engines/engine.py.224: train: epoch 52 | batch 1970/2304 [I 2022-10-27 06:30:17] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:05 [I 2022-10-27 06:30:17] ...trap/engines/engine.py.229: process: 0.55054 | load: 0.00243 [I 2022-10-27 06:30:17] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-27 06:30:17] ...optimizers/trijoint.py.151: total_norm: 14.342809 [I 2022-10-27 06:30:22] ...trap/engines/engine.py.224: train: epoch 52 | batch 1980/2304 [I 2022-10-27 06:30:22] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:02:58 [I 2022-10-27 06:30:22] ...trap/engines/engine.py.229: process: 0.55076 | load: 0.00246 [I 2022-10-27 06:30:22] ...trap/engines/engine.py.230: loss: 0.11359 [I 2022-10-27 06:30:22] ...optimizers/trijoint.py.151: total_norm: 11.161669 [I 2022-10-27 06:30:28] ...trap/engines/engine.py.224: train: epoch 52 | batch 1990/2304 [I 2022-10-27 06:30:28] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:02:55 [I 2022-10-27 06:30:28] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00246 [I 2022-10-27 06:30:28] ...trap/engines/engine.py.230: loss: 0.12528 [I 2022-10-27 06:30:28] ...optimizers/trijoint.py.151: total_norm: 13.540389 [I 2022-10-27 06:30:33] ...trap/engines/engine.py.224: train: epoch 52 | batch 2000/2304 [I 2022-10-27 06:30:33] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:02:49 [I 2022-10-27 06:30:33] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00243 [I 2022-10-27 06:30:33] ...trap/engines/engine.py.230: loss: 0.17035 [I 2022-10-27 06:30:33] ...optimizers/trijoint.py.151: total_norm: 15.255202 [I 2022-10-27 06:30:39] ...trap/engines/engine.py.224: train: epoch 52 | batch 2010/2304 [I 2022-10-27 06:30:39] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:43 [I 2022-10-27 06:30:39] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00248 [I 2022-10-27 06:30:39] ...trap/engines/engine.py.230: loss: 0.13513 [I 2022-10-27 06:30:39] ...optimizers/trijoint.py.151: total_norm: 12.057213 [I 2022-10-27 06:30:45] ...trap/engines/engine.py.224: train: epoch 52 | batch 2020/2304 [I 2022-10-27 06:30:45] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:40 [I 2022-10-27 06:30:45] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00249 [I 2022-10-27 06:30:45] ...trap/engines/engine.py.230: loss: 0.15898 [I 2022-10-27 06:30:45] ...optimizers/trijoint.py.151: total_norm: 18.889639 [I 2022-10-27 06:30:50] ...trap/engines/engine.py.224: train: epoch 52 | batch 2030/2304 [I 2022-10-27 06:30:50] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:33 [I 2022-10-27 06:30:50] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00257 [I 2022-10-27 06:30:50] ...trap/engines/engine.py.230: loss: 0.18430 [I 2022-10-27 06:30:50] ...optimizers/trijoint.py.151: total_norm: 25.168968 [I 2022-10-27 06:30:56] ...trap/engines/engine.py.224: train: epoch 52 | batch 2040/2304 [I 2022-10-27 06:30:56] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:27 [I 2022-10-27 06:30:56] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00250 [I 2022-10-27 06:30:56] ...trap/engines/engine.py.230: loss: 0.14902 [I 2022-10-27 06:30:56] ...optimizers/trijoint.py.151: total_norm: 28.790964 [I 2022-10-27 06:31:02] ...trap/engines/engine.py.224: train: epoch 52 | batch 2050/2304 [I 2022-10-27 06:31:02] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:21 [I 2022-10-27 06:31:02] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00248 [I 2022-10-27 06:31:02] ...trap/engines/engine.py.230: loss: 0.14063 [I 2022-10-27 06:31:02] ...optimizers/trijoint.py.151: total_norm: 44.037857 [I 2022-10-27 06:31:07] ...trap/engines/engine.py.224: train: epoch 52 | batch 2060/2304 [I 2022-10-27 06:31:07] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:15 [I 2022-10-27 06:31:07] ...trap/engines/engine.py.229: process: 0.55045 | load: 0.00242 [I 2022-10-27 06:31:07] ...trap/engines/engine.py.230: loss: 0.21935 [I 2022-10-27 06:31:07] ...optimizers/trijoint.py.151: total_norm: 37.287312 [I 2022-10-27 06:31:13] ...trap/engines/engine.py.224: train: epoch 52 | batch 2070/2304 [I 2022-10-27 06:31:13] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:10 [I 2022-10-27 06:31:13] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00242 [I 2022-10-27 06:31:13] ...trap/engines/engine.py.230: loss: 0.13017 [I 2022-10-27 06:31:13] ...optimizers/trijoint.py.151: total_norm: 15.588721 [I 2022-10-27 06:31:18] ...trap/engines/engine.py.224: train: epoch 52 | batch 2080/2304 [I 2022-10-27 06:31:18] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:05 [I 2022-10-27 06:31:18] ...trap/engines/engine.py.229: process: 0.56354 | load: 0.00245 [I 2022-10-27 06:31:18] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-27 06:31:18] ...optimizers/trijoint.py.151: total_norm: 12.914425 [I 2022-10-27 06:31:24] ...trap/engines/engine.py.224: train: epoch 52 | batch 2090/2304 [I 2022-10-27 06:31:24] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:01:58 [I 2022-10-27 06:31:24] ...trap/engines/engine.py.229: process: 0.54837 | load: 0.00244 [I 2022-10-27 06:31:24] ...trap/engines/engine.py.230: loss: 0.14847 [I 2022-10-27 06:31:24] ...optimizers/trijoint.py.151: total_norm: 15.025548 [I 2022-10-27 06:31:29] ...trap/engines/engine.py.224: train: epoch 52 | batch 2100/2304 [I 2022-10-27 06:31:29] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:01:52 [I 2022-10-27 06:31:29] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00244 [I 2022-10-27 06:31:29] ...trap/engines/engine.py.230: loss: 0.10191 [I 2022-10-27 06:31:29] ...optimizers/trijoint.py.151: total_norm: 8.976438 [I 2022-10-27 06:31:35] ...trap/engines/engine.py.224: train: epoch 52 | batch 2110/2304 [I 2022-10-27 06:31:35] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:01:48 [I 2022-10-27 06:31:35] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00242 [I 2022-10-27 06:31:35] ...trap/engines/engine.py.230: loss: 0.14736 [I 2022-10-27 06:31:35] ...optimizers/trijoint.py.151: total_norm: 18.219101 [I 2022-10-27 06:31:41] ...trap/engines/engine.py.224: train: epoch 52 | batch 2120/2304 [I 2022-10-27 06:31:41] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:43 [I 2022-10-27 06:31:41] ...trap/engines/engine.py.229: process: 0.57255 | load: 0.00243 [I 2022-10-27 06:31:41] ...trap/engines/engine.py.230: loss: 0.16779 [I 2022-10-27 06:31:41] ...optimizers/trijoint.py.151: total_norm: 8.511955 [I 2022-10-27 06:31:46] ...trap/engines/engine.py.224: train: epoch 52 | batch 2130/2304 [I 2022-10-27 06:31:46] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:37 [I 2022-10-27 06:31:46] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00258 [I 2022-10-27 06:31:46] ...trap/engines/engine.py.230: loss: 0.18362 [I 2022-10-27 06:31:46] ...optimizers/trijoint.py.151: total_norm: 21.841810 [I 2022-10-27 06:31:52] ...trap/engines/engine.py.224: train: epoch 52 | batch 2140/2304 [I 2022-10-27 06:31:52] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:31 [I 2022-10-27 06:31:52] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00248 [I 2022-10-27 06:31:52] ...trap/engines/engine.py.230: loss: 0.25628 [I 2022-10-27 06:31:52] ...optimizers/trijoint.py.151: total_norm: 50.419273 [I 2022-10-27 06:31:57] ...trap/engines/engine.py.224: train: epoch 52 | batch 2150/2304 [I 2022-10-27 06:31:57] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:25 [I 2022-10-27 06:31:57] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00249 [I 2022-10-27 06:31:57] ...trap/engines/engine.py.230: loss: 0.15706 [I 2022-10-27 06:31:57] ...optimizers/trijoint.py.151: total_norm: 25.811056 [I 2022-10-27 06:32:03] ...trap/engines/engine.py.224: train: epoch 52 | batch 2160/2304 [I 2022-10-27 06:32:03] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:20 [I 2022-10-27 06:32:03] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00242 [I 2022-10-27 06:32:03] ...trap/engines/engine.py.230: loss: 0.13910 [I 2022-10-27 06:32:03] ...optimizers/trijoint.py.151: total_norm: 10.117384 [I 2022-10-27 06:32:09] ...trap/engines/engine.py.224: train: epoch 52 | batch 2170/2304 [I 2022-10-27 06:32:09] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:14 [I 2022-10-27 06:32:09] ...trap/engines/engine.py.229: process: 0.54606 | load: 0.00244 [I 2022-10-27 06:32:09] ...trap/engines/engine.py.230: loss: 0.14656 [I 2022-10-27 06:32:09] ...optimizers/trijoint.py.151: total_norm: 16.658918 [I 2022-10-27 06:32:14] ...trap/engines/engine.py.224: train: epoch 52 | batch 2180/2304 [I 2022-10-27 06:32:14] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:08 [I 2022-10-27 06:32:14] ...trap/engines/engine.py.229: process: 0.56332 | load: 0.00244 [I 2022-10-27 06:32:14] ...trap/engines/engine.py.230: loss: 0.14913 [I 2022-10-27 06:32:14] ...optimizers/trijoint.py.151: total_norm: 14.244056 [I 2022-10-27 06:32:20] ...trap/engines/engine.py.224: train: epoch 52 | batch 2190/2304 [I 2022-10-27 06:32:20] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:04 [I 2022-10-27 06:32:20] ...trap/engines/engine.py.229: process: 0.56910 | load: 0.00241 [I 2022-10-27 06:32:20] ...trap/engines/engine.py.230: loss: 0.12109 [I 2022-10-27 06:32:20] ...optimizers/trijoint.py.151: total_norm: 9.580967 [I 2022-10-27 06:32:25] ...trap/engines/engine.py.224: train: epoch 52 | batch 2200/2304 [I 2022-10-27 06:32:25] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:00:57 [I 2022-10-27 06:32:25] ...trap/engines/engine.py.229: process: 0.54708 | load: 0.00242 [I 2022-10-27 06:32:25] ...trap/engines/engine.py.230: loss: 0.19503 [I 2022-10-27 06:32:25] ...optimizers/trijoint.py.151: total_norm: 23.415739 [I 2022-10-27 06:32:31] ...trap/engines/engine.py.224: train: epoch 52 | batch 2210/2304 [I 2022-10-27 06:32:31] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:00:52 [I 2022-10-27 06:32:31] ...trap/engines/engine.py.229: process: 0.54841 | load: 0.00242 [I 2022-10-27 06:32:31] ...trap/engines/engine.py.230: loss: 0.17921 [I 2022-10-27 06:32:31] ...optimizers/trijoint.py.151: total_norm: 37.556980 [I 2022-10-27 06:32:37] ...trap/engines/engine.py.224: train: epoch 52 | batch 2220/2304 [I 2022-10-27 06:32:37] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:00:46 [I 2022-10-27 06:32:37] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00246 [I 2022-10-27 06:32:37] ...trap/engines/engine.py.230: loss: 0.21096 [I 2022-10-27 06:32:37] ...optimizers/trijoint.py.151: total_norm: 18.323730 [I 2022-10-27 06:32:42] ...trap/engines/engine.py.224: train: epoch 52 | batch 2230/2304 [I 2022-10-27 06:32:42] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:00:41 [I 2022-10-27 06:32:42] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00244 [I 2022-10-27 06:32:42] ...trap/engines/engine.py.230: loss: 0.10681 [I 2022-10-27 06:32:42] ...optimizers/trijoint.py.151: total_norm: 11.868918 [I 2022-10-27 06:32:48] ...trap/engines/engine.py.224: train: epoch 52 | batch 2240/2304 [I 2022-10-27 06:32:48] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:35 [I 2022-10-27 06:32:48] ...trap/engines/engine.py.229: process: 0.56149 | load: 0.00249 [I 2022-10-27 06:32:48] ...trap/engines/engine.py.230: loss: 0.25445 [I 2022-10-27 06:32:48] ...optimizers/trijoint.py.151: total_norm: 16.443020 [I 2022-10-27 06:32:53] ...trap/engines/engine.py.224: train: epoch 52 | batch 2250/2304 [I 2022-10-27 06:32:53] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:29 [I 2022-10-27 06:32:53] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00242 [I 2022-10-27 06:32:53] ...trap/engines/engine.py.230: loss: 0.14307 [I 2022-10-27 06:32:53] ...optimizers/trijoint.py.151: total_norm: 16.962128 [I 2022-10-27 06:32:59] ...trap/engines/engine.py.224: train: epoch 52 | batch 2260/2304 [I 2022-10-27 06:32:59] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:24 [I 2022-10-27 06:32:59] ...trap/engines/engine.py.229: process: 0.56643 | load: 0.00243 [I 2022-10-27 06:32:59] ...trap/engines/engine.py.230: loss: 0.14727 [I 2022-10-27 06:32:59] ...optimizers/trijoint.py.151: total_norm: 14.350116 [I 2022-10-27 06:33:04] ...trap/engines/engine.py.224: train: epoch 52 | batch 2270/2304 [I 2022-10-27 06:33:04] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:18 [I 2022-10-27 06:33:04] ...trap/engines/engine.py.229: process: 0.56134 | load: 0.00280 [I 2022-10-27 06:33:04] ...trap/engines/engine.py.230: loss: 0.12440 [I 2022-10-27 06:33:04] ...optimizers/trijoint.py.151: total_norm: 17.033451 [I 2022-10-27 06:33:10] ...trap/engines/engine.py.224: train: epoch 52 | batch 2280/2304 [I 2022-10-27 06:33:10] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:13 [I 2022-10-27 06:33:10] ...trap/engines/engine.py.229: process: 0.54971 | load: 0.00242 [I 2022-10-27 06:33:10] ...trap/engines/engine.py.230: loss: 0.10025 [I 2022-10-27 06:33:10] ...optimizers/trijoint.py.151: total_norm: 7.091880 [I 2022-10-27 06:33:15] ...trap/engines/engine.py.224: train: epoch 52 | batch 2290/2304 [I 2022-10-27 06:33:15] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:07 [I 2022-10-27 06:33:15] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00242 [I 2022-10-27 06:33:15] ...trap/engines/engine.py.230: loss: 0.15487 [I 2022-10-27 06:33:16] ...optimizers/trijoint.py.151: total_norm: 27.174519 [I 2022-10-27 06:33:21] ...trap/engines/engine.py.224: train: epoch 52 | batch 2300/2304 [I 2022-10-27 06:33:21] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:02 [I 2022-10-27 06:33:21] ...trap/engines/engine.py.229: process: 0.53735 | load: 0.00014 [I 2022-10-27 06:33:21] ...trap/engines/engine.py.230: loss: 0.18716 [I 2022-10-27 06:33:21] ...optimizers/trijoint.py.151: total_norm: 28.661854 [I 2022-10-27 06:33:23] ...trap/engines/engine.py.224: train: epoch 52 | batch 2304/2304 [I 2022-10-27 06:33:23] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:00 [I 2022-10-27 06:33:23] ...trap/engines/engine.py.229: process: 0.54943 | load: 0.00014 [I 2022-10-27 06:33:23] ...trap/engines/engine.py.230: loss: 0.13321 [I 2022-10-27 06:33:23] ...optimizers/trijoint.py.151: total_norm: 13.716231 [S 2022-10-27 06:33:25] ...trap/engines/engine.py.129: train_epoch.epoch: 52 [S 2022-10-27 06:33:25] ...trap/engines/engine.py.129: train_epoch.loss: 0.16318814065347784 [I 2022-10-27 06:33:25] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 06:33:25] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 52 [I 2022-10-27 06:33:25] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 06:33:27] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 06:33:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 06:33:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 06:33:29] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 06:33:29] ...trap/engines/engine.py.321: eval: epoch 52 | batch 0/510 [I 2022-10-27 06:33:29] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:34:13 [I 2022-10-27 06:33:29] ...trap/engines/engine.py.326: process: 4.02557 | load: 3.60752 [I 2022-10-27 06:33:33] ...trap/engines/engine.py.321: eval: epoch 52 | batch 10/510 [I 2022-10-27 06:33:33] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:04 [I 2022-10-27 06:33:33] ...trap/engines/engine.py.326: process: 0.34959 | load: 0.00017 [I 2022-10-27 06:33:36] ...trap/engines/engine.py.321: eval: epoch 52 | batch 20/510 [I 2022-10-27 06:33:36] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:02 [I 2022-10-27 06:33:36] ...trap/engines/engine.py.326: process: 0.32187 | load: 0.00018 [I 2022-10-27 06:33:39] ...trap/engines/engine.py.321: eval: epoch 52 | batch 30/510 [I 2022-10-27 06:33:39] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:42 [I 2022-10-27 06:33:39] ...trap/engines/engine.py.326: process: 0.33220 | load: 0.00020 [I 2022-10-27 06:33:43] ...trap/engines/engine.py.321: eval: epoch 52 | batch 40/510 [I 2022-10-27 06:33:43] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:38 [I 2022-10-27 06:33:43] ...trap/engines/engine.py.326: process: 0.35382 | load: 0.00019 [I 2022-10-27 06:33:46] ...trap/engines/engine.py.321: eval: epoch 52 | batch 50/510 [I 2022-10-27 06:33:46] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:32 [I 2022-10-27 06:33:46] ...trap/engines/engine.py.326: process: 0.32553 | load: 0.00019 [I 2022-10-27 06:33:49] ...trap/engines/engine.py.321: eval: epoch 52 | batch 60/510 [I 2022-10-27 06:33:49] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:30 [I 2022-10-27 06:33:49] ...trap/engines/engine.py.326: process: 0.33966 | load: 0.00036 [I 2022-10-27 06:33:53] ...trap/engines/engine.py.321: eval: epoch 52 | batch 70/510 [I 2022-10-27 06:33:53] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:28 [I 2022-10-27 06:33:53] ...trap/engines/engine.py.326: process: 0.36835 | load: 0.00018 [I 2022-10-27 06:33:56] ...trap/engines/engine.py.321: eval: epoch 52 | batch 80/510 [I 2022-10-27 06:33:56] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-27 06:33:56] ...trap/engines/engine.py.326: process: 0.32528 | load: 0.00018 [I 2022-10-27 06:33:59] ...trap/engines/engine.py.321: eval: epoch 52 | batch 90/510 [I 2022-10-27 06:33:59] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:21 [I 2022-10-27 06:33:59] ...trap/engines/engine.py.326: process: 0.35149 | load: 0.00019 [I 2022-10-27 06:34:03] ...trap/engines/engine.py.321: eval: epoch 52 | batch 100/510 [I 2022-10-27 06:34:03] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:13 [I 2022-10-27 06:34:03] ...trap/engines/engine.py.326: process: 0.28999 | load: 0.00016 [I 2022-10-27 06:34:06] ...trap/engines/engine.py.321: eval: epoch 52 | batch 110/510 [I 2022-10-27 06:34:06] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:57 [I 2022-10-27 06:34:06] ...trap/engines/engine.py.326: process: 0.28806 | load: 0.00017 [I 2022-10-27 06:34:08] ...trap/engines/engine.py.321: eval: epoch 52 | batch 120/510 [I 2022-10-27 06:34:08] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:55 [I 2022-10-27 06:34:08] ...trap/engines/engine.py.326: process: 0.30905 | load: 0.00016 [I 2022-10-27 06:34:11] ...trap/engines/engine.py.321: eval: epoch 52 | batch 130/510 [I 2022-10-27 06:34:11] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:51 [I 2022-10-27 06:34:11] ...trap/engines/engine.py.326: process: 0.30353 | load: 0.00021 [I 2022-10-27 06:34:14] ...trap/engines/engine.py.321: eval: epoch 52 | batch 140/510 [I 2022-10-27 06:34:14] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-27 06:34:14] ...trap/engines/engine.py.326: process: 0.28490 | load: 0.00019 [I 2022-10-27 06:34:17] ...trap/engines/engine.py.321: eval: epoch 52 | batch 150/510 [I 2022-10-27 06:34:17] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-27 06:34:17] ...trap/engines/engine.py.326: process: 0.28848 | load: 0.00014 [I 2022-10-27 06:34:20] ...trap/engines/engine.py.321: eval: epoch 52 | batch 160/510 [I 2022-10-27 06:34:20] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-27 06:34:20] ...trap/engines/engine.py.326: process: 0.28841 | load: 0.00015 [I 2022-10-27 06:34:23] ...trap/engines/engine.py.321: eval: epoch 52 | batch 170/510 [I 2022-10-27 06:34:23] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-27 06:34:23] ...trap/engines/engine.py.326: process: 0.29053 | load: 0.00016 [I 2022-10-27 06:34:26] ...trap/engines/engine.py.321: eval: epoch 52 | batch 180/510 [I 2022-10-27 06:34:26] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 06:34:26] ...trap/engines/engine.py.326: process: 0.28716 | load: 0.00022 [I 2022-10-27 06:34:29] ...trap/engines/engine.py.321: eval: epoch 52 | batch 190/510 [I 2022-10-27 06:34:29] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 06:34:29] ...trap/engines/engine.py.326: process: 0.28675 | load: 0.00015 [I 2022-10-27 06:34:32] ...trap/engines/engine.py.321: eval: epoch 52 | batch 200/510 [I 2022-10-27 06:34:32] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:30 [I 2022-10-27 06:34:32] ...trap/engines/engine.py.326: process: 0.30564 | load: 0.00015 [I 2022-10-27 06:34:34] ...trap/engines/engine.py.321: eval: epoch 52 | batch 210/510 [I 2022-10-27 06:34:34] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 06:34:34] ...trap/engines/engine.py.326: process: 0.28487 | load: 0.00015 [I 2022-10-27 06:34:37] ...trap/engines/engine.py.321: eval: epoch 52 | batch 220/510 [I 2022-10-27 06:34:37] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:24 [I 2022-10-27 06:34:37] ...trap/engines/engine.py.326: process: 0.28726 | load: 0.00020 [I 2022-10-27 06:34:40] ...trap/engines/engine.py.321: eval: epoch 52 | batch 230/510 [I 2022-10-27 06:34:40] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:21 [I 2022-10-27 06:34:40] ...trap/engines/engine.py.326: process: 0.29031 | load: 0.00014 [I 2022-10-27 06:34:43] ...trap/engines/engine.py.321: eval: epoch 52 | batch 240/510 [I 2022-10-27 06:34:43] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-27 06:34:43] ...trap/engines/engine.py.326: process: 0.28617 | load: 0.00016 [I 2022-10-27 06:34:46] ...trap/engines/engine.py.321: eval: epoch 52 | batch 250/510 [I 2022-10-27 06:34:46] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-27 06:34:46] ...trap/engines/engine.py.326: process: 0.28846 | load: 0.00015 [I 2022-10-27 06:34:49] ...trap/engines/engine.py.321: eval: epoch 52 | batch 260/510 [I 2022-10-27 06:34:49] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 06:34:49] ...trap/engines/engine.py.326: process: 0.28984 | load: 0.00017 [I 2022-10-27 06:34:52] ...trap/engines/engine.py.321: eval: epoch 52 | batch 270/510 [I 2022-10-27 06:34:52] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-27 06:34:52] ...trap/engines/engine.py.326: process: 0.28037 | load: 0.00015 [I 2022-10-27 06:34:55] ...trap/engines/engine.py.321: eval: epoch 52 | batch 280/510 [I 2022-10-27 06:34:55] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:05 [I 2022-10-27 06:34:55] ...trap/engines/engine.py.326: process: 0.28993 | load: 0.00015 [I 2022-10-27 06:34:57] ...trap/engines/engine.py.321: eval: epoch 52 | batch 290/510 [I 2022-10-27 06:34:57] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 06:34:57] ...trap/engines/engine.py.326: process: 0.28892 | load: 0.00016 [I 2022-10-27 06:35:00] ...trap/engines/engine.py.321: eval: epoch 52 | batch 300/510 [I 2022-10-27 06:35:00] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 06:35:00] ...trap/engines/engine.py.326: process: 0.29987 | load: 0.00015 [I 2022-10-27 06:35:03] ...trap/engines/engine.py.321: eval: epoch 52 | batch 310/510 [I 2022-10-27 06:35:03] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 06:35:03] ...trap/engines/engine.py.326: process: 0.28855 | load: 0.00022 [I 2022-10-27 06:35:06] ...trap/engines/engine.py.321: eval: epoch 52 | batch 320/510 [I 2022-10-27 06:35:06] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 06:35:06] ...trap/engines/engine.py.326: process: 0.28718 | load: 0.00015 [I 2022-10-27 06:35:09] ...trap/engines/engine.py.321: eval: epoch 52 | batch 330/510 [I 2022-10-27 06:35:09] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 06:35:09] ...trap/engines/engine.py.326: process: 0.28476 | load: 0.00016 [I 2022-10-27 06:35:12] ...trap/engines/engine.py.321: eval: epoch 52 | batch 340/510 [I 2022-10-27 06:35:12] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-27 06:35:12] ...trap/engines/engine.py.326: process: 0.28217 | load: 0.00015 [I 2022-10-27 06:35:15] ...trap/engines/engine.py.321: eval: epoch 52 | batch 350/510 [I 2022-10-27 06:35:15] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-27 06:35:15] ...trap/engines/engine.py.326: process: 0.28421 | load: 0.00016 [I 2022-10-27 06:35:18] ...trap/engines/engine.py.321: eval: epoch 52 | batch 360/510 [I 2022-10-27 06:35:18] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:42 [I 2022-10-27 06:35:18] ...trap/engines/engine.py.326: process: 0.28904 | load: 0.00016 [I 2022-10-27 06:35:20] ...trap/engines/engine.py.321: eval: epoch 52 | batch 370/510 [I 2022-10-27 06:35:20] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 06:35:20] ...trap/engines/engine.py.326: process: 0.28028 | load: 0.00016 [I 2022-10-27 06:35:23] ...trap/engines/engine.py.321: eval: epoch 52 | batch 380/510 [I 2022-10-27 06:35:23] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 06:35:23] ...trap/engines/engine.py.326: process: 0.30697 | load: 0.00016 [I 2022-10-27 06:35:26] ...trap/engines/engine.py.321: eval: epoch 52 | batch 390/510 [I 2022-10-27 06:35:26] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-27 06:35:26] ...trap/engines/engine.py.326: process: 0.28915 | load: 0.00015 [I 2022-10-27 06:35:29] ...trap/engines/engine.py.321: eval: epoch 52 | batch 400/510 [I 2022-10-27 06:35:29] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-27 06:35:29] ...trap/engines/engine.py.326: process: 0.28292 | load: 0.00014 [I 2022-10-27 06:35:32] ...trap/engines/engine.py.321: eval: epoch 52 | batch 410/510 [I 2022-10-27 06:35:32] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-27 06:35:32] ...trap/engines/engine.py.326: process: 0.28395 | load: 0.00019 [I 2022-10-27 06:35:35] ...trap/engines/engine.py.321: eval: epoch 52 | batch 420/510 [I 2022-10-27 06:35:35] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-27 06:35:35] ...trap/engines/engine.py.326: process: 0.28694 | load: 0.00019 [I 2022-10-27 06:35:38] ...trap/engines/engine.py.321: eval: epoch 52 | batch 430/510 [I 2022-10-27 06:35:38] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:22 [I 2022-10-27 06:35:38] ...trap/engines/engine.py.326: process: 0.28817 | load: 0.00015 [I 2022-10-27 06:35:41] ...trap/engines/engine.py.321: eval: epoch 52 | batch 440/510 [I 2022-10-27 06:35:41] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 06:35:41] ...trap/engines/engine.py.326: process: 0.28049 | load: 0.00015 [I 2022-10-27 06:35:43] ...trap/engines/engine.py.321: eval: epoch 52 | batch 450/510 [I 2022-10-27 06:35:43] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 06:35:43] ...trap/engines/engine.py.326: process: 0.28817 | load: 0.00016 [I 2022-10-27 06:35:46] ...trap/engines/engine.py.321: eval: epoch 52 | batch 460/510 [I 2022-10-27 06:35:46] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 06:35:46] ...trap/engines/engine.py.326: process: 0.28613 | load: 0.00016 [I 2022-10-27 06:35:49] ...trap/engines/engine.py.321: eval: epoch 52 | batch 470/510 [I 2022-10-27 06:35:49] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-27 06:35:49] ...trap/engines/engine.py.326: process: 0.28469 | load: 0.00015 [I 2022-10-27 06:35:52] ...trap/engines/engine.py.321: eval: epoch 52 | batch 480/510 [I 2022-10-27 06:35:52] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 06:35:52] ...trap/engines/engine.py.326: process: 0.29277 | load: 0.00015 [I 2022-10-27 06:35:55] ...trap/engines/engine.py.321: eval: epoch 52 | batch 490/510 [I 2022-10-27 06:35:55] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 06:35:55] ...trap/engines/engine.py.326: process: 0.28815 | load: 0.00015 [I 2022-10-27 06:35:58] ...trap/engines/engine.py.321: eval: epoch 52 | batch 500/510 [I 2022-10-27 06:35:58] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 06:35:58] ...trap/engines/engine.py.326: process: 0.28526 | load: 0.00015 [I 2022-10-27 06:36:01] ...trap/engines/engine.py.321: eval: epoch 52 | batch 510/510 [I 2022-10-27 06:36:01] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 06:36:01] ...trap/engines/engine.py.326: process: 0.28592 | load: 0.00015 [S 2022-10-27 06:36:02] ...trap/engines/engine.py.132: eval_epoch.epoch: 52 [I 2022-10-27 06:36:02] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.703600 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905500 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937700 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.688900 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.896700 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933400 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009002 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006185 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004173 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009005 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.004776 [S 2022-10-27 06:36:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004432 [I 2022-10-27 06:36:07] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 06:36:07] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 06:36:07] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 06:36:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 06:36:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 06:36:09] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 06:36:12] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 06:36:18] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 06:36:18] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 06:36:18] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 06:36:20] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 06:36:26] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 06:36:26] ...trap/engines/engine.py.163: Training model on trainset for epoch 53 [I 2022-10-27 06:36:26] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 06:36:29] ...trap/engines/engine.py.224: train: epoch 53 | batch 0/2304 [I 2022-10-27 06:36:29] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:59 [I 2022-10-27 06:36:29] ...trap/engines/engine.py.229: process: 2.70808 | load: 2.08117 [I 2022-10-27 06:36:29] ...trap/engines/engine.py.230: loss: 0.11166 [I 2022-10-27 06:36:29] ...optimizers/trijoint.py.151: total_norm: 12.104914 [I 2022-10-27 06:36:34] ...trap/engines/engine.py.224: train: epoch 53 | batch 10/2304 [I 2022-10-27 06:36:34] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:05 [I 2022-10-27 06:36:34] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00511 [I 2022-10-27 06:36:34] ...trap/engines/engine.py.230: loss: 0.15367 [I 2022-10-27 06:36:34] ...optimizers/trijoint.py.151: total_norm: 14.056501 [I 2022-10-27 06:36:40] ...trap/engines/engine.py.224: train: epoch 53 | batch 20/2304 [I 2022-10-27 06:36:40] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:13 [I 2022-10-27 06:36:40] ...trap/engines/engine.py.229: process: 0.55346 | load: 0.00284 [I 2022-10-27 06:36:40] ...trap/engines/engine.py.230: loss: 0.16468 [I 2022-10-27 06:36:40] ...optimizers/trijoint.py.151: total_norm: 15.796700 [I 2022-10-27 06:36:46] ...trap/engines/engine.py.224: train: epoch 53 | batch 30/2304 [I 2022-10-27 06:36:46] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:20 [I 2022-10-27 06:36:46] ...trap/engines/engine.py.229: process: 0.54618 | load: 0.00337 [I 2022-10-27 06:36:46] ...trap/engines/engine.py.230: loss: 0.14462 [I 2022-10-27 06:36:46] ...optimizers/trijoint.py.151: total_norm: 17.173851 [I 2022-10-27 06:36:51] ...trap/engines/engine.py.224: train: epoch 53 | batch 40/2304 [I 2022-10-27 06:36:51] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:00 [I 2022-10-27 06:36:51] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00280 [I 2022-10-27 06:36:51] ...trap/engines/engine.py.230: loss: 0.18306 [I 2022-10-27 06:36:51] ...optimizers/trijoint.py.151: total_norm: 30.338303 [I 2022-10-27 06:36:57] ...trap/engines/engine.py.224: train: epoch 53 | batch 50/2304 [I 2022-10-27 06:36:57] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:58 [I 2022-10-27 06:36:57] ...trap/engines/engine.py.229: process: 0.56886 | load: 0.00265 [I 2022-10-27 06:36:57] ...trap/engines/engine.py.230: loss: 0.15644 [I 2022-10-27 06:36:57] ...optimizers/trijoint.py.151: total_norm: 13.084792 [I 2022-10-27 06:37:02] ...trap/engines/engine.py.224: train: epoch 53 | batch 60/2304 [I 2022-10-27 06:37:02] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:46 [I 2022-10-27 06:37:02] ...trap/engines/engine.py.229: process: 0.56394 | load: 0.00260 [I 2022-10-27 06:37:02] ...trap/engines/engine.py.230: loss: 0.14355 [I 2022-10-27 06:37:02] ...optimizers/trijoint.py.151: total_norm: 23.533367 [I 2022-10-27 06:37:08] ...trap/engines/engine.py.224: train: epoch 53 | batch 70/2304 [I 2022-10-27 06:37:08] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:49 [I 2022-10-27 06:37:08] ...trap/engines/engine.py.229: process: 0.55406 | load: 0.00291 [I 2022-10-27 06:37:08] ...trap/engines/engine.py.230: loss: 0.12727 [I 2022-10-27 06:37:08] ...optimizers/trijoint.py.151: total_norm: 23.305513 [I 2022-10-27 06:37:14] ...trap/engines/engine.py.224: train: epoch 53 | batch 80/2304 [I 2022-10-27 06:37:14] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:36 [I 2022-10-27 06:37:14] ...trap/engines/engine.py.229: process: 0.54886 | load: 0.00267 [I 2022-10-27 06:37:14] ...trap/engines/engine.py.230: loss: 0.11097 [I 2022-10-27 06:37:14] ...optimizers/trijoint.py.151: total_norm: 7.223969 [I 2022-10-27 06:37:19] ...trap/engines/engine.py.224: train: epoch 53 | batch 90/2304 [I 2022-10-27 06:37:19] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:28 [I 2022-10-27 06:37:19] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00256 [I 2022-10-27 06:37:19] ...trap/engines/engine.py.230: loss: 0.13716 [I 2022-10-27 06:37:19] ...optimizers/trijoint.py.151: total_norm: 12.643002 [I 2022-10-27 06:37:25] ...trap/engines/engine.py.224: train: epoch 53 | batch 100/2304 [I 2022-10-27 06:37:25] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:35 [I 2022-10-27 06:37:25] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00261 [I 2022-10-27 06:37:25] ...trap/engines/engine.py.230: loss: 0.18415 [I 2022-10-27 06:37:25] ...optimizers/trijoint.py.151: total_norm: 19.100851 [I 2022-10-27 06:37:30] ...trap/engines/engine.py.224: train: epoch 53 | batch 110/2304 [I 2022-10-27 06:37:30] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:16 [I 2022-10-27 06:37:30] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00309 [I 2022-10-27 06:37:30] ...trap/engines/engine.py.230: loss: 0.11503 [I 2022-10-27 06:37:30] ...optimizers/trijoint.py.151: total_norm: 23.345442 [I 2022-10-27 06:37:36] ...trap/engines/engine.py.224: train: epoch 53 | batch 120/2304 [I 2022-10-27 06:37:36] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:13 [I 2022-10-27 06:37:36] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00288 [I 2022-10-27 06:37:36] ...trap/engines/engine.py.230: loss: 0.13473 [I 2022-10-27 06:37:36] ...optimizers/trijoint.py.151: total_norm: 18.092773 [I 2022-10-27 06:37:42] ...trap/engines/engine.py.224: train: epoch 53 | batch 130/2304 [I 2022-10-27 06:37:42] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:07 [I 2022-10-27 06:37:42] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00275 [I 2022-10-27 06:37:42] ...trap/engines/engine.py.230: loss: 0.17272 [I 2022-10-27 06:37:42] ...optimizers/trijoint.py.151: total_norm: 15.486836 [I 2022-10-27 06:37:47] ...trap/engines/engine.py.224: train: epoch 53 | batch 140/2304 [I 2022-10-27 06:37:47] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:12 [I 2022-10-27 06:37:47] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00268 [I 2022-10-27 06:37:47] ...trap/engines/engine.py.230: loss: 0.12591 [I 2022-10-27 06:37:47] ...optimizers/trijoint.py.151: total_norm: 20.420361 [I 2022-10-27 06:37:53] ...trap/engines/engine.py.224: train: epoch 53 | batch 150/2304 [I 2022-10-27 06:37:53] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:58 [I 2022-10-27 06:37:53] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00302 [I 2022-10-27 06:37:53] ...trap/engines/engine.py.230: loss: 0.22000 [I 2022-10-27 06:37:53] ...optimizers/trijoint.py.151: total_norm: 23.030249 [I 2022-10-27 06:37:58] ...trap/engines/engine.py.224: train: epoch 53 | batch 160/2304 [I 2022-10-27 06:37:58] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:02 [I 2022-10-27 06:37:58] ...trap/engines/engine.py.229: process: 0.57026 | load: 0.00286 [I 2022-10-27 06:37:58] ...trap/engines/engine.py.230: loss: 0.17772 [I 2022-10-27 06:37:58] ...optimizers/trijoint.py.151: total_norm: 14.534390 [I 2022-10-27 06:38:04] ...trap/engines/engine.py.224: train: epoch 53 | batch 170/2304 [I 2022-10-27 06:38:04] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:57 [I 2022-10-27 06:38:04] ...trap/engines/engine.py.229: process: 0.56254 | load: 0.00286 [I 2022-10-27 06:38:04] ...trap/engines/engine.py.230: loss: 0.13948 [I 2022-10-27 06:38:04] ...optimizers/trijoint.py.151: total_norm: 23.063192 [I 2022-10-27 06:38:10] ...trap/engines/engine.py.224: train: epoch 53 | batch 180/2304 [I 2022-10-27 06:38:10] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:46 [I 2022-10-27 06:38:10] ...trap/engines/engine.py.229: process: 0.56606 | load: 0.00303 [I 2022-10-27 06:38:10] ...trap/engines/engine.py.230: loss: 0.12825 [I 2022-10-27 06:38:10] ...optimizers/trijoint.py.151: total_norm: 11.005174 [I 2022-10-27 06:38:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 190/2304 [I 2022-10-27 06:38:15] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:40 [I 2022-10-27 06:38:15] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00264 [I 2022-10-27 06:38:15] ...trap/engines/engine.py.230: loss: 0.18835 [I 2022-10-27 06:38:15] ...optimizers/trijoint.py.151: total_norm: 17.485037 [I 2022-10-27 06:38:21] ...trap/engines/engine.py.224: train: epoch 53 | batch 200/2304 [I 2022-10-27 06:38:21] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:33 [I 2022-10-27 06:38:21] ...trap/engines/engine.py.229: process: 0.56451 | load: 0.00288 [I 2022-10-27 06:38:21] ...trap/engines/engine.py.230: loss: 0.15311 [I 2022-10-27 06:38:21] ...optimizers/trijoint.py.151: total_norm: 13.766386 [I 2022-10-27 06:38:27] ...trap/engines/engine.py.224: train: epoch 53 | batch 210/2304 [I 2022-10-27 06:38:27] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:20:05 [I 2022-10-27 06:38:27] ...trap/engines/engine.py.229: process: 0.56130 | load: 0.00251 [I 2022-10-27 06:38:27] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-27 06:38:27] ...optimizers/trijoint.py.151: total_norm: 8.879542 [I 2022-10-27 06:38:32] ...trap/engines/engine.py.224: train: epoch 53 | batch 220/2304 [I 2022-10-27 06:38:32] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:30 [I 2022-10-27 06:38:32] ...trap/engines/engine.py.229: process: 0.56256 | load: 0.00271 [I 2022-10-27 06:38:32] ...trap/engines/engine.py.230: loss: 0.10364 [I 2022-10-27 06:38:32] ...optimizers/trijoint.py.151: total_norm: 12.039759 [I 2022-10-27 06:38:38] ...trap/engines/engine.py.224: train: epoch 53 | batch 230/2304 [I 2022-10-27 06:38:38] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:15 [I 2022-10-27 06:38:38] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00262 [I 2022-10-27 06:38:38] ...trap/engines/engine.py.230: loss: 0.13037 [I 2022-10-27 06:38:38] ...optimizers/trijoint.py.151: total_norm: 16.636803 [I 2022-10-27 06:38:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 240/2304 [I 2022-10-27 06:38:43] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:07 [I 2022-10-27 06:38:43] ...trap/engines/engine.py.229: process: 0.54718 | load: 0.00247 [I 2022-10-27 06:38:43] ...trap/engines/engine.py.230: loss: 0.14230 [I 2022-10-27 06:38:43] ...optimizers/trijoint.py.151: total_norm: 15.862822 [I 2022-10-27 06:38:49] ...trap/engines/engine.py.224: train: epoch 53 | batch 250/2304 [I 2022-10-27 06:38:49] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:55 [I 2022-10-27 06:38:49] ...trap/engines/engine.py.229: process: 0.54718 | load: 0.00251 [I 2022-10-27 06:38:49] ...trap/engines/engine.py.230: loss: 0.11206 [I 2022-10-27 06:38:49] ...optimizers/trijoint.py.151: total_norm: 8.246532 [I 2022-10-27 06:38:54] ...trap/engines/engine.py.224: train: epoch 53 | batch 260/2304 [I 2022-10-27 06:38:54] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:03 [I 2022-10-27 06:38:54] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00248 [I 2022-10-27 06:38:54] ...trap/engines/engine.py.230: loss: 0.12475 [I 2022-10-27 06:38:54] ...optimizers/trijoint.py.151: total_norm: 14.139107 [I 2022-10-27 06:39:00] ...trap/engines/engine.py.224: train: epoch 53 | batch 270/2304 [I 2022-10-27 06:39:00] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:51 [I 2022-10-27 06:39:00] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00253 [I 2022-10-27 06:39:00] ...trap/engines/engine.py.230: loss: 0.13821 [I 2022-10-27 06:39:00] ...optimizers/trijoint.py.151: total_norm: 44.199917 [I 2022-10-27 06:39:06] ...trap/engines/engine.py.224: train: epoch 53 | batch 280/2304 [I 2022-10-27 06:39:06] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:51 [I 2022-10-27 06:39:06] ...trap/engines/engine.py.229: process: 0.56371 | load: 0.00246 [I 2022-10-27 06:39:06] ...trap/engines/engine.py.230: loss: 0.15322 [I 2022-10-27 06:39:06] ...optimizers/trijoint.py.151: total_norm: 14.600567 [I 2022-10-27 06:39:11] ...trap/engines/engine.py.224: train: epoch 53 | batch 290/2304 [I 2022-10-27 06:39:11] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:49 [I 2022-10-27 06:39:11] ...trap/engines/engine.py.229: process: 0.54497 | load: 0.00281 [I 2022-10-27 06:39:11] ...trap/engines/engine.py.230: loss: 0.16223 [I 2022-10-27 06:39:11] ...optimizers/trijoint.py.151: total_norm: 20.160185 [I 2022-10-27 06:39:17] ...trap/engines/engine.py.224: train: epoch 53 | batch 300/2304 [I 2022-10-27 06:39:17] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:39 [I 2022-10-27 06:39:17] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00263 [I 2022-10-27 06:39:17] ...trap/engines/engine.py.230: loss: 0.18268 [I 2022-10-27 06:39:17] ...optimizers/trijoint.py.151: total_norm: 14.131691 [I 2022-10-27 06:39:22] ...trap/engines/engine.py.224: train: epoch 53 | batch 310/2304 [I 2022-10-27 06:39:22] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:32 [I 2022-10-27 06:39:22] ...trap/engines/engine.py.229: process: 0.54905 | load: 0.00247 [I 2022-10-27 06:39:22] ...trap/engines/engine.py.230: loss: 0.22551 [I 2022-10-27 06:39:22] ...optimizers/trijoint.py.151: total_norm: 27.448668 [I 2022-10-27 06:39:28] ...trap/engines/engine.py.224: train: epoch 53 | batch 320/2304 [I 2022-10-27 06:39:28] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:20 [I 2022-10-27 06:39:28] ...trap/engines/engine.py.229: process: 0.55319 | load: 0.00254 [I 2022-10-27 06:39:28] ...trap/engines/engine.py.230: loss: 0.12907 [I 2022-10-27 06:39:28] ...optimizers/trijoint.py.151: total_norm: 19.955763 [I 2022-10-27 06:39:34] ...trap/engines/engine.py.224: train: epoch 53 | batch 330/2304 [I 2022-10-27 06:39:34] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:37 [I 2022-10-27 06:39:34] ...trap/engines/engine.py.229: process: 0.59887 | load: 0.00255 [I 2022-10-27 06:39:34] ...trap/engines/engine.py.230: loss: 0.12823 [I 2022-10-27 06:39:34] ...optimizers/trijoint.py.151: total_norm: 14.668050 [I 2022-10-27 06:39:39] ...trap/engines/engine.py.224: train: epoch 53 | batch 340/2304 [I 2022-10-27 06:39:39] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:15 [I 2022-10-27 06:39:39] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00255 [I 2022-10-27 06:39:39] ...trap/engines/engine.py.230: loss: 0.17675 [I 2022-10-27 06:39:39] ...optimizers/trijoint.py.151: total_norm: 11.128544 [I 2022-10-27 06:39:45] ...trap/engines/engine.py.224: train: epoch 53 | batch 350/2304 [I 2022-10-27 06:39:45] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:05 [I 2022-10-27 06:39:45] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00249 [I 2022-10-27 06:39:45] ...trap/engines/engine.py.230: loss: 0.16554 [I 2022-10-27 06:39:45] ...optimizers/trijoint.py.151: total_norm: 28.661976 [I 2022-10-27 06:39:50] ...trap/engines/engine.py.224: train: epoch 53 | batch 360/2304 [I 2022-10-27 06:39:50] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:57 [I 2022-10-27 06:39:50] ...trap/engines/engine.py.229: process: 0.55234 | load: 0.00268 [I 2022-10-27 06:39:50] ...trap/engines/engine.py.230: loss: 0.20501 [I 2022-10-27 06:39:50] ...optimizers/trijoint.py.151: total_norm: 18.467773 [I 2022-10-27 06:39:56] ...trap/engines/engine.py.224: train: epoch 53 | batch 370/2304 [I 2022-10-27 06:39:56] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:00 [I 2022-10-27 06:39:56] ...trap/engines/engine.py.229: process: 0.54681 | load: 0.00266 [I 2022-10-27 06:39:56] ...trap/engines/engine.py.230: loss: 0.19796 [I 2022-10-27 06:39:56] ...optimizers/trijoint.py.151: total_norm: 15.307481 [I 2022-10-27 06:40:02] ...trap/engines/engine.py.224: train: epoch 53 | batch 380/2304 [I 2022-10-27 06:40:02] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:18:04 [I 2022-10-27 06:40:02] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00265 [I 2022-10-27 06:40:02] ...trap/engines/engine.py.230: loss: 0.22593 [I 2022-10-27 06:40:02] ...optimizers/trijoint.py.151: total_norm: 34.187424 [I 2022-10-27 06:40:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 390/2304 [I 2022-10-27 06:40:07] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:50 [I 2022-10-27 06:40:07] ...trap/engines/engine.py.229: process: 0.54781 | load: 0.00259 [I 2022-10-27 06:40:07] ...trap/engines/engine.py.230: loss: 0.18799 [I 2022-10-27 06:40:07] ...optimizers/trijoint.py.151: total_norm: 22.433737 [I 2022-10-27 06:40:13] ...trap/engines/engine.py.224: train: epoch 53 | batch 400/2304 [I 2022-10-27 06:40:13] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:23 [I 2022-10-27 06:40:13] ...trap/engines/engine.py.229: process: 0.54345 | load: 0.00247 [I 2022-10-27 06:40:13] ...trap/engines/engine.py.230: loss: 0.17533 [I 2022-10-27 06:40:13] ...optimizers/trijoint.py.151: total_norm: 19.285994 [I 2022-10-27 06:40:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 410/2304 [I 2022-10-27 06:40:18] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:33 [I 2022-10-27 06:40:18] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00246 [I 2022-10-27 06:40:18] ...trap/engines/engine.py.230: loss: 0.15135 [I 2022-10-27 06:40:18] ...optimizers/trijoint.py.151: total_norm: 11.267690 [I 2022-10-27 06:40:24] ...trap/engines/engine.py.224: train: epoch 53 | batch 420/2304 [I 2022-10-27 06:40:24] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:20:20 [I 2022-10-27 06:40:24] ...trap/engines/engine.py.229: process: 0.70616 | load: 0.00254 [I 2022-10-27 06:40:24] ...trap/engines/engine.py.230: loss: 0.14449 [I 2022-10-27 06:40:24] ...optimizers/trijoint.py.151: total_norm: 9.361802 [I 2022-10-27 06:40:31] ...trap/engines/engine.py.224: train: epoch 53 | batch 430/2304 [I 2022-10-27 06:40:31] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:18:30 [I 2022-10-27 06:40:31] ...trap/engines/engine.py.229: process: 0.55711 | load: 0.00249 [I 2022-10-27 06:40:31] ...trap/engines/engine.py.230: loss: 0.13131 [I 2022-10-27 06:40:31] ...optimizers/trijoint.py.151: total_norm: 10.337689 [I 2022-10-27 06:40:36] ...trap/engines/engine.py.224: train: epoch 53 | batch 440/2304 [I 2022-10-27 06:40:36] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:17:29 [I 2022-10-27 06:40:36] ...trap/engines/engine.py.229: process: 0.56727 | load: 0.00247 [I 2022-10-27 06:40:36] ...trap/engines/engine.py.230: loss: 0.14336 [I 2022-10-27 06:40:36] ...optimizers/trijoint.py.151: total_norm: 20.364269 [I 2022-10-27 06:40:42] ...trap/engines/engine.py.224: train: epoch 53 | batch 450/2304 [I 2022-10-27 06:40:42] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:13 [I 2022-10-27 06:40:42] ...trap/engines/engine.py.229: process: 0.55347 | load: 0.00238 [I 2022-10-27 06:40:42] ...trap/engines/engine.py.230: loss: 0.15154 [I 2022-10-27 06:40:42] ...optimizers/trijoint.py.151: total_norm: 13.509254 [I 2022-10-27 06:40:48] ...trap/engines/engine.py.224: train: epoch 53 | batch 460/2304 [I 2022-10-27 06:40:48] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:17:14 [I 2022-10-27 06:40:48] ...trap/engines/engine.py.229: process: 0.56245 | load: 0.00246 [I 2022-10-27 06:40:48] ...trap/engines/engine.py.230: loss: 0.19018 [I 2022-10-27 06:40:48] ...optimizers/trijoint.py.151: total_norm: 63.277664 [I 2022-10-27 06:40:53] ...trap/engines/engine.py.224: train: epoch 53 | batch 470/2304 [I 2022-10-27 06:40:53] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:17:05 [I 2022-10-27 06:40:53] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00247 [I 2022-10-27 06:40:53] ...trap/engines/engine.py.230: loss: 0.14518 [I 2022-10-27 06:40:53] ...optimizers/trijoint.py.151: total_norm: 16.628450 [I 2022-10-27 06:40:59] ...trap/engines/engine.py.224: train: epoch 53 | batch 480/2304 [I 2022-10-27 06:40:59] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:17:00 [I 2022-10-27 06:40:59] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00255 [I 2022-10-27 06:40:59] ...trap/engines/engine.py.230: loss: 0.11850 [I 2022-10-27 06:40:59] ...optimizers/trijoint.py.151: total_norm: 14.413348 [I 2022-10-27 06:41:04] ...trap/engines/engine.py.224: train: epoch 53 | batch 490/2304 [I 2022-10-27 06:41:04] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:16:45 [I 2022-10-27 06:41:04] ...trap/engines/engine.py.229: process: 0.54367 | load: 0.00251 [I 2022-10-27 06:41:04] ...trap/engines/engine.py.230: loss: 0.15045 [I 2022-10-27 06:41:04] ...optimizers/trijoint.py.151: total_norm: 7.113201 [I 2022-10-27 06:41:10] ...trap/engines/engine.py.224: train: epoch 53 | batch 500/2304 [I 2022-10-27 06:41:10] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:44 [I 2022-10-27 06:41:10] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00248 [I 2022-10-27 06:41:10] ...trap/engines/engine.py.230: loss: 0.10457 [I 2022-10-27 06:41:10] ...optimizers/trijoint.py.151: total_norm: 9.033864 [I 2022-10-27 06:41:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 510/2304 [I 2022-10-27 06:41:15] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:41 [I 2022-10-27 06:41:15] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00244 [I 2022-10-27 06:41:15] ...trap/engines/engine.py.230: loss: 0.14846 [I 2022-10-27 06:41:15] ...optimizers/trijoint.py.151: total_norm: 13.223879 [I 2022-10-27 06:41:21] ...trap/engines/engine.py.224: train: epoch 53 | batch 520/2304 [I 2022-10-27 06:41:21] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:36 [I 2022-10-27 06:41:21] ...trap/engines/engine.py.229: process: 0.56302 | load: 0.00255 [I 2022-10-27 06:41:21] ...trap/engines/engine.py.230: loss: 0.25677 [I 2022-10-27 06:41:21] ...optimizers/trijoint.py.151: total_norm: 23.839914 [I 2022-10-27 06:41:27] ...trap/engines/engine.py.224: train: epoch 53 | batch 530/2304 [I 2022-10-27 06:41:27] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:19 [I 2022-10-27 06:41:27] ...trap/engines/engine.py.229: process: 0.54424 | load: 0.00243 [I 2022-10-27 06:41:27] ...trap/engines/engine.py.230: loss: 0.14012 [I 2022-10-27 06:41:27] ...optimizers/trijoint.py.151: total_norm: 18.835506 [I 2022-10-27 06:41:32] ...trap/engines/engine.py.224: train: epoch 53 | batch 540/2304 [I 2022-10-27 06:41:32] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:16:17 [I 2022-10-27 06:41:32] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00248 [I 2022-10-27 06:41:32] ...trap/engines/engine.py.230: loss: 0.14612 [I 2022-10-27 06:41:32] ...optimizers/trijoint.py.151: total_norm: 8.610361 [I 2022-10-27 06:41:38] ...trap/engines/engine.py.224: train: epoch 53 | batch 550/2304 [I 2022-10-27 06:41:38] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:17 [I 2022-10-27 06:41:38] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00242 [I 2022-10-27 06:41:38] ...trap/engines/engine.py.230: loss: 0.16900 [I 2022-10-27 06:41:38] ...optimizers/trijoint.py.151: total_norm: 11.708872 [I 2022-10-27 06:41:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 560/2304 [I 2022-10-27 06:41:43] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:01 [I 2022-10-27 06:41:43] ...trap/engines/engine.py.229: process: 0.54901 | load: 0.00253 [I 2022-10-27 06:41:43] ...trap/engines/engine.py.230: loss: 0.15688 [I 2022-10-27 06:41:43] ...optimizers/trijoint.py.151: total_norm: 34.728386 [I 2022-10-27 06:41:49] ...trap/engines/engine.py.224: train: epoch 53 | batch 570/2304 [I 2022-10-27 06:41:49] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:01 [I 2022-10-27 06:41:49] ...trap/engines/engine.py.229: process: 0.55378 | load: 0.00243 [I 2022-10-27 06:41:49] ...trap/engines/engine.py.230: loss: 0.16109 [I 2022-10-27 06:41:49] ...optimizers/trijoint.py.151: total_norm: 26.058134 [I 2022-10-27 06:41:54] ...trap/engines/engine.py.224: train: epoch 53 | batch 580/2304 [I 2022-10-27 06:41:54] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:15:48 [I 2022-10-27 06:41:54] ...trap/engines/engine.py.229: process: 0.54857 | load: 0.00239 [I 2022-10-27 06:41:54] ...trap/engines/engine.py.230: loss: 0.13265 [I 2022-10-27 06:41:54] ...optimizers/trijoint.py.151: total_norm: 20.232660 [I 2022-10-27 06:42:00] ...trap/engines/engine.py.224: train: epoch 53 | batch 590/2304 [I 2022-10-27 06:42:00] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:53 [I 2022-10-27 06:42:00] ...trap/engines/engine.py.229: process: 0.56784 | load: 0.00246 [I 2022-10-27 06:42:00] ...trap/engines/engine.py.230: loss: 0.18525 [I 2022-10-27 06:42:00] ...optimizers/trijoint.py.151: total_norm: 26.141132 [I 2022-10-27 06:42:05] ...trap/engines/engine.py.224: train: epoch 53 | batch 600/2304 [I 2022-10-27 06:42:05] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:53 [I 2022-10-27 06:42:05] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00248 [I 2022-10-27 06:42:05] ...trap/engines/engine.py.230: loss: 0.14846 [I 2022-10-27 06:42:05] ...optimizers/trijoint.py.151: total_norm: 12.126839 [I 2022-10-27 06:42:11] ...trap/engines/engine.py.224: train: epoch 53 | batch 610/2304 [I 2022-10-27 06:42:11] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:41 [I 2022-10-27 06:42:11] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00245 [I 2022-10-27 06:42:11] ...trap/engines/engine.py.230: loss: 0.15584 [I 2022-10-27 06:42:11] ...optimizers/trijoint.py.151: total_norm: 30.710716 [I 2022-10-27 06:42:17] ...trap/engines/engine.py.224: train: epoch 53 | batch 620/2304 [I 2022-10-27 06:42:17] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:38 [I 2022-10-27 06:42:17] ...trap/engines/engine.py.229: process: 0.56184 | load: 0.00245 [I 2022-10-27 06:42:17] ...trap/engines/engine.py.230: loss: 0.17558 [I 2022-10-27 06:42:17] ...optimizers/trijoint.py.151: total_norm: 22.347282 [I 2022-10-27 06:42:22] ...trap/engines/engine.py.224: train: epoch 53 | batch 630/2304 [I 2022-10-27 06:42:22] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:26 [I 2022-10-27 06:42:22] ...trap/engines/engine.py.229: process: 0.54393 | load: 0.00243 [I 2022-10-27 06:42:22] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-27 06:42:22] ...optimizers/trijoint.py.151: total_norm: 10.018900 [I 2022-10-27 06:42:28] ...trap/engines/engine.py.224: train: epoch 53 | batch 640/2304 [I 2022-10-27 06:42:28] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:18 [I 2022-10-27 06:42:28] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00240 [I 2022-10-27 06:42:28] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-27 06:42:28] ...optimizers/trijoint.py.151: total_norm: 15.306993 [I 2022-10-27 06:42:33] ...trap/engines/engine.py.224: train: epoch 53 | batch 650/2304 [I 2022-10-27 06:42:33] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:22 [I 2022-10-27 06:42:33] ...trap/engines/engine.py.229: process: 0.55925 | load: 0.00244 [I 2022-10-27 06:42:33] ...trap/engines/engine.py.230: loss: 0.15668 [I 2022-10-27 06:42:33] ...optimizers/trijoint.py.151: total_norm: 13.383714 [I 2022-10-27 06:42:39] ...trap/engines/engine.py.224: train: epoch 53 | batch 660/2304 [I 2022-10-27 06:42:39] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:11 [I 2022-10-27 06:42:39] ...trap/engines/engine.py.229: process: 0.54580 | load: 0.00247 [I 2022-10-27 06:42:39] ...trap/engines/engine.py.230: loss: 0.12920 [I 2022-10-27 06:42:39] ...optimizers/trijoint.py.151: total_norm: 24.912100 [I 2022-10-27 06:42:44] ...trap/engines/engine.py.224: train: epoch 53 | batch 670/2304 [I 2022-10-27 06:42:44] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:10 [I 2022-10-27 06:42:44] ...trap/engines/engine.py.229: process: 0.57087 | load: 0.00241 [I 2022-10-27 06:42:44] ...trap/engines/engine.py.230: loss: 0.11225 [I 2022-10-27 06:42:44] ...optimizers/trijoint.py.151: total_norm: 16.334280 [I 2022-10-27 06:42:50] ...trap/engines/engine.py.224: train: epoch 53 | batch 680/2304 [I 2022-10-27 06:42:50] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:14:57 [I 2022-10-27 06:42:50] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00244 [I 2022-10-27 06:42:50] ...trap/engines/engine.py.230: loss: 0.15318 [I 2022-10-27 06:42:50] ...optimizers/trijoint.py.151: total_norm: 29.021509 [I 2022-10-27 06:42:56] ...trap/engines/engine.py.224: train: epoch 53 | batch 690/2304 [I 2022-10-27 06:42:56] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:14:59 [I 2022-10-27 06:42:56] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00255 [I 2022-10-27 06:42:56] ...trap/engines/engine.py.230: loss: 0.12306 [I 2022-10-27 06:42:56] ...optimizers/trijoint.py.151: total_norm: 11.884977 [I 2022-10-27 06:43:01] ...trap/engines/engine.py.224: train: epoch 53 | batch 700/2304 [I 2022-10-27 06:43:01] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:14:53 [I 2022-10-27 06:43:01] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00246 [I 2022-10-27 06:43:01] ...trap/engines/engine.py.230: loss: 0.18808 [I 2022-10-27 06:43:01] ...optimizers/trijoint.py.151: total_norm: 28.016630 [I 2022-10-27 06:43:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 710/2304 [I 2022-10-27 06:43:07] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:45 [I 2022-10-27 06:43:07] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00259 [I 2022-10-27 06:43:07] ...trap/engines/engine.py.230: loss: 0.25055 [I 2022-10-27 06:43:07] ...optimizers/trijoint.py.151: total_norm: 21.624079 [I 2022-10-27 06:43:12] ...trap/engines/engine.py.224: train: epoch 53 | batch 720/2304 [I 2022-10-27 06:43:12] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:37 [I 2022-10-27 06:43:12] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00254 [I 2022-10-27 06:43:12] ...trap/engines/engine.py.230: loss: 0.14816 [I 2022-10-27 06:43:12] ...optimizers/trijoint.py.151: total_norm: 24.107727 [I 2022-10-27 06:43:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 730/2304 [I 2022-10-27 06:43:18] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:37 [I 2022-10-27 06:43:18] ...trap/engines/engine.py.229: process: 0.54831 | load: 0.00248 [I 2022-10-27 06:43:18] ...trap/engines/engine.py.230: loss: 0.13664 [I 2022-10-27 06:43:18] ...optimizers/trijoint.py.151: total_norm: 13.712795 [I 2022-10-27 06:43:23] ...trap/engines/engine.py.224: train: epoch 53 | batch 740/2304 [I 2022-10-27 06:43:23] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:31 [I 2022-10-27 06:43:23] ...trap/engines/engine.py.229: process: 0.56306 | load: 0.00243 [I 2022-10-27 06:43:23] ...trap/engines/engine.py.230: loss: 0.10342 [I 2022-10-27 06:43:23] ...optimizers/trijoint.py.151: total_norm: 11.024541 [I 2022-10-27 06:43:29] ...trap/engines/engine.py.224: train: epoch 53 | batch 750/2304 [I 2022-10-27 06:43:29] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:24 [I 2022-10-27 06:43:29] ...trap/engines/engine.py.229: process: 0.56193 | load: 0.00249 [I 2022-10-27 06:43:29] ...trap/engines/engine.py.230: loss: 0.14661 [I 2022-10-27 06:43:29] ...optimizers/trijoint.py.151: total_norm: 13.634949 [I 2022-10-27 06:43:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 760/2304 [I 2022-10-27 06:43:35] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:14 [I 2022-10-27 06:43:35] ...trap/engines/engine.py.229: process: 0.55521 | load: 0.00248 [I 2022-10-27 06:43:35] ...trap/engines/engine.py.230: loss: 0.15477 [I 2022-10-27 06:43:35] ...optimizers/trijoint.py.151: total_norm: 15.001751 [I 2022-10-27 06:43:40] ...trap/engines/engine.py.224: train: epoch 53 | batch 770/2304 [I 2022-10-27 06:43:40] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:08 [I 2022-10-27 06:43:40] ...trap/engines/engine.py.229: process: 0.54764 | load: 0.00307 [I 2022-10-27 06:43:40] ...trap/engines/engine.py.230: loss: 0.14343 [I 2022-10-27 06:43:40] ...optimizers/trijoint.py.151: total_norm: 12.295546 [I 2022-10-27 06:43:46] ...trap/engines/engine.py.224: train: epoch 53 | batch 780/2304 [I 2022-10-27 06:43:46] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:06 [I 2022-10-27 06:43:46] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00247 [I 2022-10-27 06:43:46] ...trap/engines/engine.py.230: loss: 0.27802 [I 2022-10-27 06:43:46] ...optimizers/trijoint.py.151: total_norm: 28.560268 [I 2022-10-27 06:43:51] ...trap/engines/engine.py.224: train: epoch 53 | batch 790/2304 [I 2022-10-27 06:43:51] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:13:52 [I 2022-10-27 06:43:51] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00249 [I 2022-10-27 06:43:51] ...trap/engines/engine.py.230: loss: 0.14705 [I 2022-10-27 06:43:51] ...optimizers/trijoint.py.151: total_norm: 11.502328 [I 2022-10-27 06:43:57] ...trap/engines/engine.py.224: train: epoch 53 | batch 800/2304 [I 2022-10-27 06:43:57] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:49 [I 2022-10-27 06:43:57] ...trap/engines/engine.py.229: process: 0.54435 | load: 0.00257 [I 2022-10-27 06:43:57] ...trap/engines/engine.py.230: loss: 0.14777 [I 2022-10-27 06:43:57] ...optimizers/trijoint.py.151: total_norm: 13.777554 [I 2022-10-27 06:44:02] ...trap/engines/engine.py.224: train: epoch 53 | batch 810/2304 [I 2022-10-27 06:44:02] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:51 [I 2022-10-27 06:44:02] ...trap/engines/engine.py.229: process: 0.55711 | load: 0.00247 [I 2022-10-27 06:44:02] ...trap/engines/engine.py.230: loss: 0.12477 [I 2022-10-27 06:44:02] ...optimizers/trijoint.py.151: total_norm: 14.951648 [I 2022-10-27 06:44:08] ...trap/engines/engine.py.224: train: epoch 53 | batch 820/2304 [I 2022-10-27 06:44:08] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:49 [I 2022-10-27 06:44:08] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00247 [I 2022-10-27 06:44:08] ...trap/engines/engine.py.230: loss: 0.22663 [I 2022-10-27 06:44:08] ...optimizers/trijoint.py.151: total_norm: 17.293108 [I 2022-10-27 06:44:13] ...trap/engines/engine.py.224: train: epoch 53 | batch 830/2304 [I 2022-10-27 06:44:13] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:36 [I 2022-10-27 06:44:13] ...trap/engines/engine.py.229: process: 0.54284 | load: 0.00248 [I 2022-10-27 06:44:13] ...trap/engines/engine.py.230: loss: 0.17529 [I 2022-10-27 06:44:13] ...optimizers/trijoint.py.151: total_norm: 22.205967 [I 2022-10-27 06:44:19] ...trap/engines/engine.py.224: train: epoch 53 | batch 840/2304 [I 2022-10-27 06:44:19] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:29 [I 2022-10-27 06:44:19] ...trap/engines/engine.py.229: process: 0.55248 | load: 0.00246 [I 2022-10-27 06:44:19] ...trap/engines/engine.py.230: loss: 0.17511 [I 2022-10-27 06:44:19] ...optimizers/trijoint.py.151: total_norm: 12.696736 [I 2022-10-27 06:44:25] ...trap/engines/engine.py.224: train: epoch 53 | batch 850/2304 [I 2022-10-27 06:44:25] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:30 [I 2022-10-27 06:44:25] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00249 [I 2022-10-27 06:44:25] ...trap/engines/engine.py.230: loss: 0.16776 [I 2022-10-27 06:44:25] ...optimizers/trijoint.py.151: total_norm: 30.852646 [I 2022-10-27 06:44:30] ...trap/engines/engine.py.224: train: epoch 53 | batch 860/2304 [I 2022-10-27 06:44:30] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:22 [I 2022-10-27 06:44:30] ...trap/engines/engine.py.229: process: 0.54857 | load: 0.00250 [I 2022-10-27 06:44:30] ...trap/engines/engine.py.230: loss: 0.17244 [I 2022-10-27 06:44:30] ...optimizers/trijoint.py.151: total_norm: 19.972908 [I 2022-10-27 06:44:36] ...trap/engines/engine.py.224: train: epoch 53 | batch 870/2304 [I 2022-10-27 06:44:36] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:16 [I 2022-10-27 06:44:36] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00240 [I 2022-10-27 06:44:36] ...trap/engines/engine.py.230: loss: 0.13814 [I 2022-10-27 06:44:36] ...optimizers/trijoint.py.151: total_norm: 11.345560 [I 2022-10-27 06:44:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 880/2304 [I 2022-10-27 06:44:41] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:13 [I 2022-10-27 06:44:41] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00241 [I 2022-10-27 06:44:41] ...trap/engines/engine.py.230: loss: 0.28758 [I 2022-10-27 06:44:41] ...optimizers/trijoint.py.151: total_norm: 18.668358 [I 2022-10-27 06:44:47] ...trap/engines/engine.py.224: train: epoch 53 | batch 890/2304 [I 2022-10-27 06:44:47] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:12 [I 2022-10-27 06:44:47] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00246 [I 2022-10-27 06:44:47] ...trap/engines/engine.py.230: loss: 0.17688 [I 2022-10-27 06:44:47] ...optimizers/trijoint.py.151: total_norm: 17.594631 [I 2022-10-27 06:44:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 900/2304 [I 2022-10-27 06:44:52] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:12:57 [I 2022-10-27 06:44:52] ...trap/engines/engine.py.229: process: 0.54995 | load: 0.00242 [I 2022-10-27 06:44:52] ...trap/engines/engine.py.230: loss: 0.14579 [I 2022-10-27 06:44:52] ...optimizers/trijoint.py.151: total_norm: 11.369840 [I 2022-10-27 06:44:58] ...trap/engines/engine.py.224: train: epoch 53 | batch 910/2304 [I 2022-10-27 06:44:58] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:55 [I 2022-10-27 06:44:58] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00246 [I 2022-10-27 06:44:58] ...trap/engines/engine.py.230: loss: 0.14276 [I 2022-10-27 06:44:58] ...optimizers/trijoint.py.151: total_norm: 10.965034 [I 2022-10-27 06:45:04] ...trap/engines/engine.py.224: train: epoch 53 | batch 920/2304 [I 2022-10-27 06:45:04] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:50 [I 2022-10-27 06:45:04] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00284 [I 2022-10-27 06:45:04] ...trap/engines/engine.py.230: loss: 0.21467 [I 2022-10-27 06:45:04] ...optimizers/trijoint.py.151: total_norm: 17.014290 [I 2022-10-27 06:45:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 930/2304 [I 2022-10-27 06:45:09] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:42 [I 2022-10-27 06:45:09] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00256 [I 2022-10-27 06:45:09] ...trap/engines/engine.py.230: loss: 0.15024 [I 2022-10-27 06:45:09] ...optimizers/trijoint.py.151: total_norm: 14.779428 [I 2022-10-27 06:45:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 940/2304 [I 2022-10-27 06:45:15] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:39 [I 2022-10-27 06:45:15] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00249 [I 2022-10-27 06:45:15] ...trap/engines/engine.py.230: loss: 0.19666 [I 2022-10-27 06:45:15] ...optimizers/trijoint.py.151: total_norm: 20.359161 [I 2022-10-27 06:45:20] ...trap/engines/engine.py.224: train: epoch 53 | batch 950/2304 [I 2022-10-27 06:45:20] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:34 [I 2022-10-27 06:45:20] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00244 [I 2022-10-27 06:45:20] ...trap/engines/engine.py.230: loss: 0.19474 [I 2022-10-27 06:45:20] ...optimizers/trijoint.py.151: total_norm: 15.827174 [I 2022-10-27 06:45:26] ...trap/engines/engine.py.224: train: epoch 53 | batch 960/2304 [I 2022-10-27 06:45:26] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:35 [I 2022-10-27 06:45:26] ...trap/engines/engine.py.229: process: 0.57493 | load: 0.00242 [I 2022-10-27 06:45:26] ...trap/engines/engine.py.230: loss: 0.16935 [I 2022-10-27 06:45:26] ...optimizers/trijoint.py.151: total_norm: 11.084193 [I 2022-10-27 06:45:32] ...trap/engines/engine.py.224: train: epoch 53 | batch 970/2304 [I 2022-10-27 06:45:32] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:22 [I 2022-10-27 06:45:32] ...trap/engines/engine.py.229: process: 0.55241 | load: 0.00244 [I 2022-10-27 06:45:32] ...trap/engines/engine.py.230: loss: 0.17885 [I 2022-10-27 06:45:32] ...optimizers/trijoint.py.151: total_norm: 16.876690 [I 2022-10-27 06:45:37] ...trap/engines/engine.py.224: train: epoch 53 | batch 980/2304 [I 2022-10-27 06:45:37] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:21 [I 2022-10-27 06:45:37] ...trap/engines/engine.py.229: process: 0.57334 | load: 0.00245 [I 2022-10-27 06:45:37] ...trap/engines/engine.py.230: loss: 0.11403 [I 2022-10-27 06:45:37] ...optimizers/trijoint.py.151: total_norm: 14.305195 [I 2022-10-27 06:45:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 990/2304 [I 2022-10-27 06:45:43] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:06 [I 2022-10-27 06:45:43] ...trap/engines/engine.py.229: process: 0.54199 | load: 0.00242 [I 2022-10-27 06:45:43] ...trap/engines/engine.py.230: loss: 0.19608 [I 2022-10-27 06:45:43] ...optimizers/trijoint.py.151: total_norm: 14.636522 [I 2022-10-27 06:45:48] ...trap/engines/engine.py.224: train: epoch 53 | batch 1000/2304 [I 2022-10-27 06:45:48] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:21 [I 2022-10-27 06:45:48] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00250 [I 2022-10-27 06:45:48] ...trap/engines/engine.py.230: loss: 0.11058 [I 2022-10-27 06:45:48] ...optimizers/trijoint.py.151: total_norm: 7.449645 [I 2022-10-27 06:45:54] ...trap/engines/engine.py.224: train: epoch 53 | batch 1010/2304 [I 2022-10-27 06:45:54] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:05 [I 2022-10-27 06:45:54] ...trap/engines/engine.py.229: process: 0.55317 | load: 0.00243 [I 2022-10-27 06:45:54] ...trap/engines/engine.py.230: loss: 0.14109 [I 2022-10-27 06:45:54] ...optimizers/trijoint.py.151: total_norm: 11.030809 [I 2022-10-27 06:46:00] ...trap/engines/engine.py.224: train: epoch 53 | batch 1020/2304 [I 2022-10-27 06:46:00] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:12:06 [I 2022-10-27 06:46:00] ...trap/engines/engine.py.229: process: 0.56729 | load: 0.00246 [I 2022-10-27 06:46:00] ...trap/engines/engine.py.230: loss: 0.15269 [I 2022-10-27 06:46:00] ...optimizers/trijoint.py.151: total_norm: 22.178930 [I 2022-10-27 06:46:05] ...trap/engines/engine.py.224: train: epoch 53 | batch 1030/2304 [I 2022-10-27 06:46:05] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:49 [I 2022-10-27 06:46:05] ...trap/engines/engine.py.229: process: 0.56488 | load: 0.00242 [I 2022-10-27 06:46:05] ...trap/engines/engine.py.230: loss: 0.19870 [I 2022-10-27 06:46:05] ...optimizers/trijoint.py.151: total_norm: 12.487544 [I 2022-10-27 06:46:11] ...trap/engines/engine.py.224: train: epoch 53 | batch 1040/2304 [I 2022-10-27 06:46:11] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:47 [I 2022-10-27 06:46:11] ...trap/engines/engine.py.229: process: 0.55824 | load: 0.00248 [I 2022-10-27 06:46:11] ...trap/engines/engine.py.230: loss: 0.11219 [I 2022-10-27 06:46:11] ...optimizers/trijoint.py.151: total_norm: 17.053110 [I 2022-10-27 06:46:17] ...trap/engines/engine.py.224: train: epoch 53 | batch 1050/2304 [I 2022-10-27 06:46:17] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:35 [I 2022-10-27 06:46:17] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00247 [I 2022-10-27 06:46:17] ...trap/engines/engine.py.230: loss: 0.14775 [I 2022-10-27 06:46:17] ...optimizers/trijoint.py.151: total_norm: 20.723505 [I 2022-10-27 06:46:22] ...trap/engines/engine.py.224: train: epoch 53 | batch 1060/2304 [I 2022-10-27 06:46:22] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:32 [I 2022-10-27 06:46:22] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00243 [I 2022-10-27 06:46:22] ...trap/engines/engine.py.230: loss: 0.16217 [I 2022-10-27 06:46:22] ...optimizers/trijoint.py.151: total_norm: 20.259480 [I 2022-10-27 06:46:28] ...trap/engines/engine.py.224: train: epoch 53 | batch 1070/2304 [I 2022-10-27 06:46:28] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:34 [I 2022-10-27 06:46:28] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00242 [I 2022-10-27 06:46:28] ...trap/engines/engine.py.230: loss: 0.24091 [I 2022-10-27 06:46:28] ...optimizers/trijoint.py.151: total_norm: 24.420498 [I 2022-10-27 06:46:33] ...trap/engines/engine.py.224: train: epoch 53 | batch 1080/2304 [I 2022-10-27 06:46:33] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:21 [I 2022-10-27 06:46:33] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00250 [I 2022-10-27 06:46:33] ...trap/engines/engine.py.230: loss: 0.21682 [I 2022-10-27 06:46:33] ...optimizers/trijoint.py.151: total_norm: 31.367495 [I 2022-10-27 06:46:39] ...trap/engines/engine.py.224: train: epoch 53 | batch 1090/2304 [I 2022-10-27 06:46:39] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:23 [I 2022-10-27 06:46:39] ...trap/engines/engine.py.229: process: 0.55184 | load: 0.00251 [I 2022-10-27 06:46:39] ...trap/engines/engine.py.230: loss: 0.19548 [I 2022-10-27 06:46:39] ...optimizers/trijoint.py.151: total_norm: 21.950497 [I 2022-10-27 06:46:45] ...trap/engines/engine.py.224: train: epoch 53 | batch 1100/2304 [I 2022-10-27 06:46:45] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:15 [I 2022-10-27 06:46:45] ...trap/engines/engine.py.229: process: 0.56814 | load: 0.00249 [I 2022-10-27 06:46:45] ...trap/engines/engine.py.230: loss: 0.12182 [I 2022-10-27 06:46:45] ...optimizers/trijoint.py.151: total_norm: 12.557364 [I 2022-10-27 06:46:50] ...trap/engines/engine.py.224: train: epoch 53 | batch 1110/2304 [I 2022-10-27 06:46:50] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:04 [I 2022-10-27 06:46:50] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00245 [I 2022-10-27 06:46:50] ...trap/engines/engine.py.230: loss: 0.16944 [I 2022-10-27 06:46:50] ...optimizers/trijoint.py.151: total_norm: 14.373978 [I 2022-10-27 06:46:56] ...trap/engines/engine.py.224: train: epoch 53 | batch 1120/2304 [I 2022-10-27 06:46:56] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:01 [I 2022-10-27 06:46:56] ...trap/engines/engine.py.229: process: 0.56354 | load: 0.00242 [I 2022-10-27 06:46:56] ...trap/engines/engine.py.230: loss: 0.17372 [I 2022-10-27 06:46:56] ...optimizers/trijoint.py.151: total_norm: 17.173986 [I 2022-10-27 06:47:01] ...trap/engines/engine.py.224: train: epoch 53 | batch 1130/2304 [I 2022-10-27 06:47:01] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:55 [I 2022-10-27 06:47:01] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00244 [I 2022-10-27 06:47:01] ...trap/engines/engine.py.230: loss: 0.15550 [I 2022-10-27 06:47:01] ...optimizers/trijoint.py.151: total_norm: 23.508661 [I 2022-10-27 06:47:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 1140/2304 [I 2022-10-27 06:47:07] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:45 [I 2022-10-27 06:47:07] ...trap/engines/engine.py.229: process: 0.54695 | load: 0.00263 [I 2022-10-27 06:47:07] ...trap/engines/engine.py.230: loss: 0.19727 [I 2022-10-27 06:47:07] ...optimizers/trijoint.py.151: total_norm: 20.454790 [I 2022-10-27 06:47:13] ...trap/engines/engine.py.224: train: epoch 53 | batch 1150/2304 [I 2022-10-27 06:47:13] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:54 [I 2022-10-27 06:47:13] ...trap/engines/engine.py.229: process: 0.61876 | load: 0.00252 [I 2022-10-27 06:47:13] ...trap/engines/engine.py.230: loss: 0.11515 [I 2022-10-27 06:47:13] ...optimizers/trijoint.py.151: total_norm: 9.660673 [I 2022-10-27 06:47:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 1160/2304 [I 2022-10-27 06:47:18] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:39 [I 2022-10-27 06:47:18] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00248 [I 2022-10-27 06:47:18] ...trap/engines/engine.py.230: loss: 0.17092 [I 2022-10-27 06:47:18] ...optimizers/trijoint.py.151: total_norm: 20.594273 [I 2022-10-27 06:47:24] ...trap/engines/engine.py.224: train: epoch 53 | batch 1170/2304 [I 2022-10-27 06:47:24] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:32 [I 2022-10-27 06:47:24] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00244 [I 2022-10-27 06:47:24] ...trap/engines/engine.py.230: loss: 0.12055 [I 2022-10-27 06:47:24] ...optimizers/trijoint.py.151: total_norm: 12.837365 [I 2022-10-27 06:47:29] ...trap/engines/engine.py.224: train: epoch 53 | batch 1180/2304 [I 2022-10-27 06:47:29] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:25 [I 2022-10-27 06:47:29] ...trap/engines/engine.py.229: process: 0.54951 | load: 0.00249 [I 2022-10-27 06:47:29] ...trap/engines/engine.py.230: loss: 0.16420 [I 2022-10-27 06:47:29] ...optimizers/trijoint.py.151: total_norm: 32.903835 [I 2022-10-27 06:47:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 1190/2304 [I 2022-10-27 06:47:35] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:25 [I 2022-10-27 06:47:35] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00244 [I 2022-10-27 06:47:35] ...trap/engines/engine.py.230: loss: 0.15597 [I 2022-10-27 06:47:35] ...optimizers/trijoint.py.151: total_norm: 11.121008 [I 2022-10-27 06:47:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 1200/2304 [I 2022-10-27 06:47:41] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:15 [I 2022-10-27 06:47:41] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00245 [I 2022-10-27 06:47:41] ...trap/engines/engine.py.230: loss: 0.18239 [I 2022-10-27 06:47:41] ...optimizers/trijoint.py.151: total_norm: 31.762819 [I 2022-10-27 06:47:46] ...trap/engines/engine.py.224: train: epoch 53 | batch 1210/2304 [I 2022-10-27 06:47:46] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:09 [I 2022-10-27 06:47:46] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00249 [I 2022-10-27 06:47:46] ...trap/engines/engine.py.230: loss: 0.17414 [I 2022-10-27 06:47:46] ...optimizers/trijoint.py.151: total_norm: 17.472143 [I 2022-10-27 06:47:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 1220/2304 [I 2022-10-27 06:47:52] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:01 [I 2022-10-27 06:47:52] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00242 [I 2022-10-27 06:47:52] ...trap/engines/engine.py.230: loss: 0.13819 [I 2022-10-27 06:47:52] ...optimizers/trijoint.py.151: total_norm: 24.780119 [I 2022-10-27 06:47:57] ...trap/engines/engine.py.224: train: epoch 53 | batch 1230/2304 [I 2022-10-27 06:47:57] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:09:57 [I 2022-10-27 06:47:57] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00243 [I 2022-10-27 06:47:57] ...trap/engines/engine.py.230: loss: 0.15228 [I 2022-10-27 06:47:57] ...optimizers/trijoint.py.151: total_norm: 18.731228 [I 2022-10-27 06:48:03] ...trap/engines/engine.py.224: train: epoch 53 | batch 1240/2304 [I 2022-10-27 06:48:03] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:53 [I 2022-10-27 06:48:03] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00504 [I 2022-10-27 06:48:03] ...trap/engines/engine.py.230: loss: 0.17420 [I 2022-10-27 06:48:03] ...optimizers/trijoint.py.151: total_norm: 19.472767 [I 2022-10-27 06:48:08] ...trap/engines/engine.py.224: train: epoch 53 | batch 1250/2304 [I 2022-10-27 06:48:08] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:49 [I 2022-10-27 06:48:08] ...trap/engines/engine.py.229: process: 0.56299 | load: 0.00244 [I 2022-10-27 06:48:08] ...trap/engines/engine.py.230: loss: 0.12636 [I 2022-10-27 06:48:08] ...optimizers/trijoint.py.151: total_norm: 11.351644 [I 2022-10-27 06:48:14] ...trap/engines/engine.py.224: train: epoch 53 | batch 1260/2304 [I 2022-10-27 06:48:14] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:41 [I 2022-10-27 06:48:14] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00240 [I 2022-10-27 06:48:14] ...trap/engines/engine.py.230: loss: 0.13815 [I 2022-10-27 06:48:14] ...optimizers/trijoint.py.151: total_norm: 11.434973 [I 2022-10-27 06:48:20] ...trap/engines/engine.py.224: train: epoch 53 | batch 1270/2304 [I 2022-10-27 06:48:20] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:47 [I 2022-10-27 06:48:20] ...trap/engines/engine.py.229: process: 0.56350 | load: 0.00245 [I 2022-10-27 06:48:20] ...trap/engines/engine.py.230: loss: 0.17320 [I 2022-10-27 06:48:20] ...optimizers/trijoint.py.151: total_norm: 29.429354 [I 2022-10-27 06:48:25] ...trap/engines/engine.py.224: train: epoch 53 | batch 1280/2304 [I 2022-10-27 06:48:25] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:36 [I 2022-10-27 06:48:25] ...trap/engines/engine.py.229: process: 0.57253 | load: 0.00244 [I 2022-10-27 06:48:25] ...trap/engines/engine.py.230: loss: 0.15291 [I 2022-10-27 06:48:25] ...optimizers/trijoint.py.151: total_norm: 20.470058 [I 2022-10-27 06:48:31] ...trap/engines/engine.py.224: train: epoch 53 | batch 1290/2304 [I 2022-10-27 06:48:31] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:21 [I 2022-10-27 06:48:31] ...trap/engines/engine.py.229: process: 0.54675 | load: 0.00239 [I 2022-10-27 06:48:31] ...trap/engines/engine.py.230: loss: 0.14096 [I 2022-10-27 06:48:31] ...optimizers/trijoint.py.151: total_norm: 17.490746 [I 2022-10-27 06:48:36] ...trap/engines/engine.py.224: train: epoch 53 | batch 1300/2304 [I 2022-10-27 06:48:36] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:23 [I 2022-10-27 06:48:36] ...trap/engines/engine.py.229: process: 0.56270 | load: 0.00251 [I 2022-10-27 06:48:36] ...trap/engines/engine.py.230: loss: 0.17237 [I 2022-10-27 06:48:36] ...optimizers/trijoint.py.151: total_norm: 18.338320 [I 2022-10-27 06:48:42] ...trap/engines/engine.py.224: train: epoch 53 | batch 1310/2304 [I 2022-10-27 06:48:42] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:11 [I 2022-10-27 06:48:42] ...trap/engines/engine.py.229: process: 0.55135 | load: 0.00245 [I 2022-10-27 06:48:42] ...trap/engines/engine.py.230: loss: 0.17803 [I 2022-10-27 06:48:42] ...optimizers/trijoint.py.151: total_norm: 14.199433 [I 2022-10-27 06:48:48] ...trap/engines/engine.py.224: train: epoch 53 | batch 1320/2304 [I 2022-10-27 06:48:48] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:06 [I 2022-10-27 06:48:48] ...trap/engines/engine.py.229: process: 0.55031 | load: 0.00245 [I 2022-10-27 06:48:48] ...trap/engines/engine.py.230: loss: 0.10236 [I 2022-10-27 06:48:48] ...optimizers/trijoint.py.151: total_norm: 10.148722 [I 2022-10-27 06:48:53] ...trap/engines/engine.py.224: train: epoch 53 | batch 1330/2304 [I 2022-10-27 06:48:53] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:08:59 [I 2022-10-27 06:48:53] ...trap/engines/engine.py.229: process: 0.54735 | load: 0.00242 [I 2022-10-27 06:48:53] ...trap/engines/engine.py.230: loss: 0.17241 [I 2022-10-27 06:48:53] ...optimizers/trijoint.py.151: total_norm: 20.028275 [I 2022-10-27 06:48:59] ...trap/engines/engine.py.224: train: epoch 53 | batch 1340/2304 [I 2022-10-27 06:48:59] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:57 [I 2022-10-27 06:48:59] ...trap/engines/engine.py.229: process: 0.56593 | load: 0.00249 [I 2022-10-27 06:48:59] ...trap/engines/engine.py.230: loss: 0.15638 [I 2022-10-27 06:48:59] ...optimizers/trijoint.py.151: total_norm: 24.977566 [I 2022-10-27 06:49:04] ...trap/engines/engine.py.224: train: epoch 53 | batch 1350/2304 [I 2022-10-27 06:49:04] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:48 [I 2022-10-27 06:49:04] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00243 [I 2022-10-27 06:49:04] ...trap/engines/engine.py.230: loss: 0.13312 [I 2022-10-27 06:49:04] ...optimizers/trijoint.py.151: total_norm: 13.602883 [I 2022-10-27 06:49:10] ...trap/engines/engine.py.224: train: epoch 53 | batch 1360/2304 [I 2022-10-27 06:49:10] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:46 [I 2022-10-27 06:49:10] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00253 [I 2022-10-27 06:49:10] ...trap/engines/engine.py.230: loss: 0.11734 [I 2022-10-27 06:49:10] ...optimizers/trijoint.py.151: total_norm: 11.317420 [I 2022-10-27 06:49:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 1370/2304 [I 2022-10-27 06:49:15] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:42 [I 2022-10-27 06:49:15] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00248 [I 2022-10-27 06:49:15] ...trap/engines/engine.py.230: loss: 0.10905 [I 2022-10-27 06:49:15] ...optimizers/trijoint.py.151: total_norm: 107.396217 [I 2022-10-27 06:49:21] ...trap/engines/engine.py.224: train: epoch 53 | batch 1380/2304 [I 2022-10-27 06:49:21] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:44 [I 2022-10-27 06:49:21] ...trap/engines/engine.py.229: process: 0.56841 | load: 0.00244 [I 2022-10-27 06:49:21] ...trap/engines/engine.py.230: loss: 0.18558 [I 2022-10-27 06:49:21] ...optimizers/trijoint.py.151: total_norm: 18.152609 [I 2022-10-27 06:49:27] ...trap/engines/engine.py.224: train: epoch 53 | batch 1390/2304 [I 2022-10-27 06:49:27] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:30 [I 2022-10-27 06:49:27] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00246 [I 2022-10-27 06:49:27] ...trap/engines/engine.py.230: loss: 0.13322 [I 2022-10-27 06:49:27] ...optimizers/trijoint.py.151: total_norm: 13.812907 [I 2022-10-27 06:49:32] ...trap/engines/engine.py.224: train: epoch 53 | batch 1400/2304 [I 2022-10-27 06:49:32] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:21 [I 2022-10-27 06:49:32] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00243 [I 2022-10-27 06:49:32] ...trap/engines/engine.py.230: loss: 0.22944 [I 2022-10-27 06:49:32] ...optimizers/trijoint.py.151: total_norm: 18.351068 [I 2022-10-27 06:49:38] ...trap/engines/engine.py.224: train: epoch 53 | batch 1410/2304 [I 2022-10-27 06:49:38] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:19 [I 2022-10-27 06:49:38] ...trap/engines/engine.py.229: process: 0.56216 | load: 0.00243 [I 2022-10-27 06:49:38] ...trap/engines/engine.py.230: loss: 0.13188 [I 2022-10-27 06:49:38] ...optimizers/trijoint.py.151: total_norm: 10.455933 [I 2022-10-27 06:49:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 1420/2304 [I 2022-10-27 06:49:43] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:16 [I 2022-10-27 06:49:43] ...trap/engines/engine.py.229: process: 0.56359 | load: 0.00253 [I 2022-10-27 06:49:43] ...trap/engines/engine.py.230: loss: 0.10271 [I 2022-10-27 06:49:43] ...optimizers/trijoint.py.151: total_norm: 11.630297 [I 2022-10-27 06:49:49] ...trap/engines/engine.py.224: train: epoch 53 | batch 1430/2304 [I 2022-10-27 06:49:49] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:08 [I 2022-10-27 06:49:49] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00245 [I 2022-10-27 06:49:49] ...trap/engines/engine.py.230: loss: 0.17174 [I 2022-10-27 06:49:49] ...optimizers/trijoint.py.151: total_norm: 13.911680 [I 2022-10-27 06:49:55] ...trap/engines/engine.py.224: train: epoch 53 | batch 1440/2304 [I 2022-10-27 06:49:55] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:00 [I 2022-10-27 06:49:55] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00250 [I 2022-10-27 06:49:55] ...trap/engines/engine.py.230: loss: 0.14541 [I 2022-10-27 06:49:55] ...optimizers/trijoint.py.151: total_norm: 26.679256 [I 2022-10-27 06:50:00] ...trap/engines/engine.py.224: train: epoch 53 | batch 1450/2304 [I 2022-10-27 06:50:00] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:07:56 [I 2022-10-27 06:50:00] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00242 [I 2022-10-27 06:50:00] ...trap/engines/engine.py.230: loss: 0.18297 [I 2022-10-27 06:50:00] ...optimizers/trijoint.py.151: total_norm: 17.715782 [I 2022-10-27 06:50:06] ...trap/engines/engine.py.224: train: epoch 53 | batch 1460/2304 [I 2022-10-27 06:50:06] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:47 [I 2022-10-27 06:50:06] ...trap/engines/engine.py.229: process: 0.55348 | load: 0.00241 [I 2022-10-27 06:50:06] ...trap/engines/engine.py.230: loss: 0.14745 [I 2022-10-27 06:50:06] ...optimizers/trijoint.py.151: total_norm: 25.579933 [I 2022-10-27 06:50:11] ...trap/engines/engine.py.224: train: epoch 53 | batch 1470/2304 [I 2022-10-27 06:50:11] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:43 [I 2022-10-27 06:50:11] ...trap/engines/engine.py.229: process: 0.55561 | load: 0.00243 [I 2022-10-27 06:50:11] ...trap/engines/engine.py.230: loss: 0.12840 [I 2022-10-27 06:50:11] ...optimizers/trijoint.py.151: total_norm: 7.405160 [I 2022-10-27 06:50:17] ...trap/engines/engine.py.224: train: epoch 53 | batch 1480/2304 [I 2022-10-27 06:50:17] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:35 [I 2022-10-27 06:50:17] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00242 [I 2022-10-27 06:50:17] ...trap/engines/engine.py.230: loss: 0.17034 [I 2022-10-27 06:50:17] ...optimizers/trijoint.py.151: total_norm: 31.466715 [I 2022-10-27 06:50:22] ...trap/engines/engine.py.224: train: epoch 53 | batch 1490/2304 [I 2022-10-27 06:50:22] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:31 [I 2022-10-27 06:50:22] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00241 [I 2022-10-27 06:50:22] ...trap/engines/engine.py.230: loss: 0.12696 [I 2022-10-27 06:50:22] ...optimizers/trijoint.py.151: total_norm: 12.735457 [I 2022-10-27 06:50:28] ...trap/engines/engine.py.224: train: epoch 53 | batch 1500/2304 [I 2022-10-27 06:50:28] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:33 [I 2022-10-27 06:50:28] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00252 [I 2022-10-27 06:50:28] ...trap/engines/engine.py.230: loss: 0.13752 [I 2022-10-27 06:50:28] ...optimizers/trijoint.py.151: total_norm: 13.938869 [I 2022-10-27 06:50:34] ...trap/engines/engine.py.224: train: epoch 53 | batch 1510/2304 [I 2022-10-27 06:50:34] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:19 [I 2022-10-27 06:50:34] ...trap/engines/engine.py.229: process: 0.54376 | load: 0.00247 [I 2022-10-27 06:50:34] ...trap/engines/engine.py.230: loss: 0.19546 [I 2022-10-27 06:50:34] ...optimizers/trijoint.py.151: total_norm: 33.923912 [I 2022-10-27 06:50:39] ...trap/engines/engine.py.224: train: epoch 53 | batch 1520/2304 [I 2022-10-27 06:50:39] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:14 [I 2022-10-27 06:50:39] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00245 [I 2022-10-27 06:50:39] ...trap/engines/engine.py.230: loss: 0.15624 [I 2022-10-27 06:50:39] ...optimizers/trijoint.py.151: total_norm: 19.005396 [I 2022-10-27 06:50:45] ...trap/engines/engine.py.224: train: epoch 53 | batch 1530/2304 [I 2022-10-27 06:50:45] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:08 [I 2022-10-27 06:50:45] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00243 [I 2022-10-27 06:50:45] ...trap/engines/engine.py.230: loss: 0.15367 [I 2022-10-27 06:50:45] ...optimizers/trijoint.py.151: total_norm: 14.890698 [I 2022-10-27 06:50:50] ...trap/engines/engine.py.224: train: epoch 53 | batch 1540/2304 [I 2022-10-27 06:50:50] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:04 [I 2022-10-27 06:50:50] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00243 [I 2022-10-27 06:50:50] ...trap/engines/engine.py.230: loss: 0.21737 [I 2022-10-27 06:50:50] ...optimizers/trijoint.py.151: total_norm: 21.212997 [I 2022-10-27 06:50:56] ...trap/engines/engine.py.224: train: epoch 53 | batch 1550/2304 [I 2022-10-27 06:50:56] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:06:56 [I 2022-10-27 06:50:56] ...trap/engines/engine.py.229: process: 0.54316 | load: 0.00244 [I 2022-10-27 06:50:56] ...trap/engines/engine.py.230: loss: 0.13760 [I 2022-10-27 06:50:56] ...optimizers/trijoint.py.151: total_norm: 16.122108 [I 2022-10-27 06:51:02] ...trap/engines/engine.py.224: train: epoch 53 | batch 1560/2304 [I 2022-10-27 06:51:02] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:49 [I 2022-10-27 06:51:02] ...trap/engines/engine.py.229: process: 0.53634 | load: 0.00265 [I 2022-10-27 06:51:02] ...trap/engines/engine.py.230: loss: 0.13392 [I 2022-10-27 06:51:02] ...optimizers/trijoint.py.151: total_norm: 40.865471 [I 2022-10-27 06:51:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 1570/2304 [I 2022-10-27 06:51:07] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:46 [I 2022-10-27 06:51:07] ...trap/engines/engine.py.229: process: 0.54440 | load: 0.00251 [I 2022-10-27 06:51:07] ...trap/engines/engine.py.230: loss: 0.18083 [I 2022-10-27 06:51:07] ...optimizers/trijoint.py.151: total_norm: 24.140886 [I 2022-10-27 06:51:13] ...trap/engines/engine.py.224: train: epoch 53 | batch 1580/2304 [I 2022-10-27 06:51:13] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:42 [I 2022-10-27 06:51:13] ...trap/engines/engine.py.229: process: 0.56534 | load: 0.00246 [I 2022-10-27 06:51:13] ...trap/engines/engine.py.230: loss: 0.13593 [I 2022-10-27 06:51:13] ...optimizers/trijoint.py.151: total_norm: 12.559421 [I 2022-10-27 06:51:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 1590/2304 [I 2022-10-27 06:51:18] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:36 [I 2022-10-27 06:51:18] ...trap/engines/engine.py.229: process: 0.55558 | load: 0.00250 [I 2022-10-27 06:51:18] ...trap/engines/engine.py.230: loss: 0.13919 [I 2022-10-27 06:51:18] ...optimizers/trijoint.py.151: total_norm: 16.400249 [I 2022-10-27 06:51:24] ...trap/engines/engine.py.224: train: epoch 53 | batch 1600/2304 [I 2022-10-27 06:51:24] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:30 [I 2022-10-27 06:51:24] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00248 [I 2022-10-27 06:51:24] ...trap/engines/engine.py.230: loss: 0.16948 [I 2022-10-27 06:51:24] ...optimizers/trijoint.py.151: total_norm: 33.969547 [I 2022-10-27 06:51:29] ...trap/engines/engine.py.224: train: epoch 53 | batch 1610/2304 [I 2022-10-27 06:51:29] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:24 [I 2022-10-27 06:51:29] ...trap/engines/engine.py.229: process: 0.54322 | load: 0.00244 [I 2022-10-27 06:51:29] ...trap/engines/engine.py.230: loss: 0.15740 [I 2022-10-27 06:51:29] ...optimizers/trijoint.py.151: total_norm: 13.499503 [I 2022-10-27 06:51:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 1620/2304 [I 2022-10-27 06:51:35] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:21 [I 2022-10-27 06:51:35] ...trap/engines/engine.py.229: process: 0.56037 | load: 0.00243 [I 2022-10-27 06:51:35] ...trap/engines/engine.py.230: loss: 0.23168 [I 2022-10-27 06:51:35] ...optimizers/trijoint.py.151: total_norm: 20.812988 [I 2022-10-27 06:51:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 1630/2304 [I 2022-10-27 06:51:41] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:16 [I 2022-10-27 06:51:41] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00245 [I 2022-10-27 06:51:41] ...trap/engines/engine.py.230: loss: 0.16696 [I 2022-10-27 06:51:41] ...optimizers/trijoint.py.151: total_norm: 14.131072 [I 2022-10-27 06:51:46] ...trap/engines/engine.py.224: train: epoch 53 | batch 1640/2304 [I 2022-10-27 06:51:46] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:09 [I 2022-10-27 06:51:46] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00244 [I 2022-10-27 06:51:46] ...trap/engines/engine.py.230: loss: 0.14623 [I 2022-10-27 06:51:46] ...optimizers/trijoint.py.151: total_norm: 24.534178 [I 2022-10-27 06:51:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 1650/2304 [I 2022-10-27 06:51:52] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:00 [I 2022-10-27 06:51:52] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00243 [I 2022-10-27 06:51:52] ...trap/engines/engine.py.230: loss: 0.14601 [I 2022-10-27 06:51:52] ...optimizers/trijoint.py.151: total_norm: 18.458879 [I 2022-10-27 06:51:57] ...trap/engines/engine.py.224: train: epoch 53 | batch 1660/2304 [I 2022-10-27 06:51:57] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:58 [I 2022-10-27 06:51:57] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00244 [I 2022-10-27 06:51:57] ...trap/engines/engine.py.230: loss: 0.16767 [I 2022-10-27 06:51:57] ...optimizers/trijoint.py.151: total_norm: 12.192497 [I 2022-10-27 06:52:03] ...trap/engines/engine.py.224: train: epoch 53 | batch 1670/2304 [I 2022-10-27 06:52:03] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:53 [I 2022-10-27 06:52:03] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00245 [I 2022-10-27 06:52:03] ...trap/engines/engine.py.230: loss: 0.16531 [I 2022-10-27 06:52:03] ...optimizers/trijoint.py.151: total_norm: 23.322947 [I 2022-10-27 06:52:08] ...trap/engines/engine.py.224: train: epoch 53 | batch 1680/2304 [I 2022-10-27 06:52:08] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:47 [I 2022-10-27 06:52:08] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00245 [I 2022-10-27 06:52:08] ...trap/engines/engine.py.230: loss: 0.22626 [I 2022-10-27 06:52:08] ...optimizers/trijoint.py.151: total_norm: 19.439964 [I 2022-10-27 06:52:14] ...trap/engines/engine.py.224: train: epoch 53 | batch 1690/2304 [I 2022-10-27 06:52:14] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:38 [I 2022-10-27 06:52:14] ...trap/engines/engine.py.229: process: 0.55472 | load: 0.00267 [I 2022-10-27 06:52:14] ...trap/engines/engine.py.230: loss: 0.16038 [I 2022-10-27 06:52:14] ...optimizers/trijoint.py.151: total_norm: 19.782770 [I 2022-10-27 06:52:19] ...trap/engines/engine.py.224: train: epoch 53 | batch 1700/2304 [I 2022-10-27 06:52:19] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:35 [I 2022-10-27 06:52:19] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00250 [I 2022-10-27 06:52:19] ...trap/engines/engine.py.230: loss: 0.12945 [I 2022-10-27 06:52:19] ...optimizers/trijoint.py.151: total_norm: 13.254167 [I 2022-10-27 06:52:25] ...trap/engines/engine.py.224: train: epoch 53 | batch 1710/2304 [I 2022-10-27 06:52:25] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:29 [I 2022-10-27 06:52:25] ...trap/engines/engine.py.229: process: 0.56229 | load: 0.00247 [I 2022-10-27 06:52:25] ...trap/engines/engine.py.230: loss: 0.19089 [I 2022-10-27 06:52:25] ...optimizers/trijoint.py.151: total_norm: 16.811029 [I 2022-10-27 06:52:50] ...trap/engines/engine.py.224: train: epoch 53 | batch 1720/2304 [I 2022-10-27 06:52:50] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:10:30 [I 2022-10-27 06:52:50] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00237 [I 2022-10-27 06:52:50] ...trap/engines/engine.py.230: loss: 0.23716 [I 2022-10-27 06:52:50] ...optimizers/trijoint.py.151: total_norm: 30.160170 [I 2022-10-27 06:52:56] ...trap/engines/engine.py.224: train: epoch 53 | batch 1730/2304 [I 2022-10-27 06:52:56] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:51 [I 2022-10-27 06:52:56] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00241 [I 2022-10-27 06:52:56] ...trap/engines/engine.py.230: loss: 0.12729 [I 2022-10-27 06:52:56] ...optimizers/trijoint.py.151: total_norm: 15.754387 [I 2022-10-27 06:53:01] ...trap/engines/engine.py.224: train: epoch 53 | batch 1740/2304 [I 2022-10-27 06:53:01] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:05:17 [I 2022-10-27 06:53:01] ...trap/engines/engine.py.229: process: 0.56286 | load: 0.00240 [I 2022-10-27 06:53:01] ...trap/engines/engine.py.230: loss: 0.20313 [I 2022-10-27 06:53:01] ...optimizers/trijoint.py.151: total_norm: 65.348816 [I 2022-10-27 06:53:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 1750/2304 [I 2022-10-27 06:53:07] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:05:05 [I 2022-10-27 06:53:07] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00239 [I 2022-10-27 06:53:07] ...trap/engines/engine.py.230: loss: 0.13019 [I 2022-10-27 06:53:07] ...optimizers/trijoint.py.151: total_norm: 18.398867 [I 2022-10-27 06:53:12] ...trap/engines/engine.py.224: train: epoch 53 | batch 1760/2304 [I 2022-10-27 06:53:12] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:05:02 [I 2022-10-27 06:53:12] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00246 [I 2022-10-27 06:53:12] ...trap/engines/engine.py.230: loss: 0.14180 [I 2022-10-27 06:53:12] ...optimizers/trijoint.py.151: total_norm: 14.865043 [I 2022-10-27 06:53:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 1770/2304 [I 2022-10-27 06:53:18] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:54 [I 2022-10-27 06:53:18] ...trap/engines/engine.py.229: process: 0.53567 | load: 0.00240 [I 2022-10-27 06:53:18] ...trap/engines/engine.py.230: loss: 0.13170 [I 2022-10-27 06:53:18] ...optimizers/trijoint.py.151: total_norm: 17.625179 [I 2022-10-27 06:53:23] ...trap/engines/engine.py.224: train: epoch 53 | batch 1780/2304 [I 2022-10-27 06:53:23] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:51 [I 2022-10-27 06:53:23] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00241 [I 2022-10-27 06:53:23] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-27 06:53:23] ...optimizers/trijoint.py.151: total_norm: 10.450663 [I 2022-10-27 06:53:29] ...trap/engines/engine.py.224: train: epoch 53 | batch 1790/2304 [I 2022-10-27 06:53:29] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:43 [I 2022-10-27 06:53:29] ...trap/engines/engine.py.229: process: 0.55231 | load: 0.00240 [I 2022-10-27 06:53:29] ...trap/engines/engine.py.230: loss: 0.10641 [I 2022-10-27 06:53:29] ...optimizers/trijoint.py.151: total_norm: 7.377651 [I 2022-10-27 06:53:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 1800/2304 [I 2022-10-27 06:53:35] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:39 [I 2022-10-27 06:53:35] ...trap/engines/engine.py.229: process: 0.55064 | load: 0.00244 [I 2022-10-27 06:53:35] ...trap/engines/engine.py.230: loss: 0.19870 [I 2022-10-27 06:53:35] ...optimizers/trijoint.py.151: total_norm: 21.107639 [I 2022-10-27 06:53:40] ...trap/engines/engine.py.224: train: epoch 53 | batch 1810/2304 [I 2022-10-27 06:53:40] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:35 [I 2022-10-27 06:53:40] ...trap/engines/engine.py.229: process: 0.56573 | load: 0.00244 [I 2022-10-27 06:53:40] ...trap/engines/engine.py.230: loss: 0.17028 [I 2022-10-27 06:53:40] ...optimizers/trijoint.py.151: total_norm: 18.019865 [I 2022-10-27 06:53:46] ...trap/engines/engine.py.224: train: epoch 53 | batch 1820/2304 [I 2022-10-27 06:53:46] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:28 [I 2022-10-27 06:53:46] ...trap/engines/engine.py.229: process: 0.56407 | load: 0.00244 [I 2022-10-27 06:53:46] ...trap/engines/engine.py.230: loss: 0.19165 [I 2022-10-27 06:53:46] ...optimizers/trijoint.py.151: total_norm: 25.474331 [I 2022-10-27 06:53:51] ...trap/engines/engine.py.224: train: epoch 53 | batch 1830/2304 [I 2022-10-27 06:53:51] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:22 [I 2022-10-27 06:53:51] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00241 [I 2022-10-27 06:53:51] ...trap/engines/engine.py.230: loss: 0.17490 [I 2022-10-27 06:53:51] ...optimizers/trijoint.py.151: total_norm: 21.248066 [I 2022-10-27 06:53:57] ...trap/engines/engine.py.224: train: epoch 53 | batch 1840/2304 [I 2022-10-27 06:53:57] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:17 [I 2022-10-27 06:53:57] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00240 [I 2022-10-27 06:53:57] ...trap/engines/engine.py.230: loss: 0.14685 [I 2022-10-27 06:53:57] ...optimizers/trijoint.py.151: total_norm: 8.640152 [I 2022-10-27 06:54:02] ...trap/engines/engine.py.224: train: epoch 53 | batch 1850/2304 [I 2022-10-27 06:54:02] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:04:12 [I 2022-10-27 06:54:02] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00240 [I 2022-10-27 06:54:02] ...trap/engines/engine.py.230: loss: 0.15188 [I 2022-10-27 06:54:02] ...optimizers/trijoint.py.151: total_norm: 11.799617 [I 2022-10-27 06:54:08] ...trap/engines/engine.py.224: train: epoch 53 | batch 1860/2304 [I 2022-10-27 06:54:08] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:04:05 [I 2022-10-27 06:54:08] ...trap/engines/engine.py.229: process: 0.54387 | load: 0.00242 [I 2022-10-27 06:54:08] ...trap/engines/engine.py.230: loss: 0.15314 [I 2022-10-27 06:54:08] ...optimizers/trijoint.py.151: total_norm: 17.118521 [I 2022-10-27 06:54:13] ...trap/engines/engine.py.224: train: epoch 53 | batch 1870/2304 [I 2022-10-27 06:54:13] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:04:00 [I 2022-10-27 06:54:13] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00239 [I 2022-10-27 06:54:13] ...trap/engines/engine.py.230: loss: 0.14024 [I 2022-10-27 06:54:13] ...optimizers/trijoint.py.151: total_norm: 15.265390 [I 2022-10-27 06:54:19] ...trap/engines/engine.py.224: train: epoch 53 | batch 1880/2304 [I 2022-10-27 06:54:19] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:57 [I 2022-10-27 06:54:19] ...trap/engines/engine.py.229: process: 0.57055 | load: 0.00240 [I 2022-10-27 06:54:19] ...trap/engines/engine.py.230: loss: 0.16443 [I 2022-10-27 06:54:19] ...optimizers/trijoint.py.151: total_norm: 16.790318 [I 2022-10-27 06:54:25] ...trap/engines/engine.py.224: train: epoch 53 | batch 1890/2304 [I 2022-10-27 06:54:25] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:49 [I 2022-10-27 06:54:25] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00239 [I 2022-10-27 06:54:25] ...trap/engines/engine.py.230: loss: 0.20717 [I 2022-10-27 06:54:25] ...optimizers/trijoint.py.151: total_norm: 26.230133 [I 2022-10-27 06:54:30] ...trap/engines/engine.py.224: train: epoch 53 | batch 1900/2304 [I 2022-10-27 06:54:30] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:44 [I 2022-10-27 06:54:30] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00239 [I 2022-10-27 06:54:30] ...trap/engines/engine.py.230: loss: 0.13450 [I 2022-10-27 06:54:30] ...optimizers/trijoint.py.151: total_norm: 7.998646 [I 2022-10-27 06:54:36] ...trap/engines/engine.py.224: train: epoch 53 | batch 1910/2304 [I 2022-10-27 06:54:36] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:37 [I 2022-10-27 06:54:36] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00240 [I 2022-10-27 06:54:36] ...trap/engines/engine.py.230: loss: 0.16279 [I 2022-10-27 06:54:36] ...optimizers/trijoint.py.151: total_norm: 11.727777 [I 2022-10-27 06:54:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 1920/2304 [I 2022-10-27 06:54:41] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:33 [I 2022-10-27 06:54:41] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00238 [I 2022-10-27 06:54:41] ...trap/engines/engine.py.230: loss: 0.11566 [I 2022-10-27 06:54:41] ...optimizers/trijoint.py.151: total_norm: 12.589458 [I 2022-10-27 06:54:47] ...trap/engines/engine.py.224: train: epoch 53 | batch 1930/2304 [I 2022-10-27 06:54:47] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:28 [I 2022-10-27 06:54:47] ...trap/engines/engine.py.229: process: 0.56273 | load: 0.00249 [I 2022-10-27 06:54:47] ...trap/engines/engine.py.230: loss: 0.15867 [I 2022-10-27 06:54:47] ...optimizers/trijoint.py.151: total_norm: 39.022629 [I 2022-10-27 06:54:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 1940/2304 [I 2022-10-27 06:54:52] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:23 [I 2022-10-27 06:54:52] ...trap/engines/engine.py.229: process: 0.56549 | load: 0.00245 [I 2022-10-27 06:54:52] ...trap/engines/engine.py.230: loss: 0.14019 [I 2022-10-27 06:54:52] ...optimizers/trijoint.py.151: total_norm: 16.330442 [I 2022-10-27 06:54:58] ...trap/engines/engine.py.224: train: epoch 53 | batch 1950/2304 [I 2022-10-27 06:54:58] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:17 [I 2022-10-27 06:54:58] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00243 [I 2022-10-27 06:54:58] ...trap/engines/engine.py.230: loss: 0.13459 [I 2022-10-27 06:54:58] ...optimizers/trijoint.py.151: total_norm: 14.416483 [I 2022-10-27 06:55:04] ...trap/engines/engine.py.224: train: epoch 53 | batch 1960/2304 [I 2022-10-27 06:55:04] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:03:12 [I 2022-10-27 06:55:04] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00241 [I 2022-10-27 06:55:04] ...trap/engines/engine.py.230: loss: 0.11633 [I 2022-10-27 06:55:04] ...optimizers/trijoint.py.151: total_norm: 11.907597 [I 2022-10-27 06:55:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 1970/2304 [I 2022-10-27 06:55:09] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:03:05 [I 2022-10-27 06:55:09] ...trap/engines/engine.py.229: process: 0.56568 | load: 0.00241 [I 2022-10-27 06:55:09] ...trap/engines/engine.py.230: loss: 0.11095 [I 2022-10-27 06:55:09] ...optimizers/trijoint.py.151: total_norm: 10.837382 [I 2022-10-27 06:55:15] ...trap/engines/engine.py.224: train: epoch 53 | batch 1980/2304 [I 2022-10-27 06:55:15] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:03:00 [I 2022-10-27 06:55:15] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00245 [I 2022-10-27 06:55:15] ...trap/engines/engine.py.230: loss: 0.22768 [I 2022-10-27 06:55:15] ...optimizers/trijoint.py.151: total_norm: 27.133780 [I 2022-10-27 06:55:20] ...trap/engines/engine.py.224: train: epoch 53 | batch 1990/2304 [I 2022-10-27 06:55:20] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:54 [I 2022-10-27 06:55:20] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00244 [I 2022-10-27 06:55:20] ...trap/engines/engine.py.230: loss: 0.13246 [I 2022-10-27 06:55:20] ...optimizers/trijoint.py.151: total_norm: 10.787448 [I 2022-10-27 06:55:26] ...trap/engines/engine.py.224: train: epoch 53 | batch 2000/2304 [I 2022-10-27 06:55:26] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:48 [I 2022-10-27 06:55:26] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00243 [I 2022-10-27 06:55:26] ...trap/engines/engine.py.230: loss: 0.22863 [I 2022-10-27 06:55:26] ...optimizers/trijoint.py.151: total_norm: 16.814890 [I 2022-10-27 06:55:31] ...trap/engines/engine.py.224: train: epoch 53 | batch 2010/2304 [I 2022-10-27 06:55:31] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:43 [I 2022-10-27 06:55:31] ...trap/engines/engine.py.229: process: 0.55108 | load: 0.00248 [I 2022-10-27 06:55:31] ...trap/engines/engine.py.230: loss: 0.17515 [I 2022-10-27 06:55:31] ...optimizers/trijoint.py.151: total_norm: 15.535471 [I 2022-10-27 06:55:37] ...trap/engines/engine.py.224: train: epoch 53 | batch 2020/2304 [I 2022-10-27 06:55:37] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:37 [I 2022-10-27 06:55:37] ...trap/engines/engine.py.229: process: 0.55213 | load: 0.00261 [I 2022-10-27 06:55:37] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-27 06:55:37] ...optimizers/trijoint.py.151: total_norm: 8.554188 [I 2022-10-27 06:55:43] ...trap/engines/engine.py.224: train: epoch 53 | batch 2030/2304 [I 2022-10-27 06:55:43] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:31 [I 2022-10-27 06:55:43] ...trap/engines/engine.py.229: process: 0.53815 | load: 0.00253 [I 2022-10-27 06:55:43] ...trap/engines/engine.py.230: loss: 0.12135 [I 2022-10-27 06:55:43] ...optimizers/trijoint.py.151: total_norm: 21.586561 [I 2022-10-27 06:55:48] ...trap/engines/engine.py.224: train: epoch 53 | batch 2040/2304 [I 2022-10-27 06:55:48] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:31 [I 2022-10-27 06:55:48] ...trap/engines/engine.py.229: process: 0.56759 | load: 0.00253 [I 2022-10-27 06:55:48] ...trap/engines/engine.py.230: loss: 0.23720 [I 2022-10-27 06:55:48] ...optimizers/trijoint.py.151: total_norm: 34.455364 [I 2022-10-27 06:55:54] ...trap/engines/engine.py.224: train: epoch 53 | batch 2050/2304 [I 2022-10-27 06:55:54] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:20 [I 2022-10-27 06:55:54] ...trap/engines/engine.py.229: process: 0.54657 | load: 0.00245 [I 2022-10-27 06:55:54] ...trap/engines/engine.py.230: loss: 0.14676 [I 2022-10-27 06:55:54] ...optimizers/trijoint.py.151: total_norm: 5.632672 [I 2022-10-27 06:55:59] ...trap/engines/engine.py.224: train: epoch 53 | batch 2060/2304 [I 2022-10-27 06:55:59] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:17 [I 2022-10-27 06:55:59] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00245 [I 2022-10-27 06:55:59] ...trap/engines/engine.py.230: loss: 0.13373 [I 2022-10-27 06:55:59] ...optimizers/trijoint.py.151: total_norm: 17.337000 [I 2022-10-27 06:56:05] ...trap/engines/engine.py.224: train: epoch 53 | batch 2070/2304 [I 2022-10-27 06:56:05] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:10 [I 2022-10-27 06:56:05] ...trap/engines/engine.py.229: process: 0.56491 | load: 0.00243 [I 2022-10-27 06:56:05] ...trap/engines/engine.py.230: loss: 0.16852 [I 2022-10-27 06:56:05] ...optimizers/trijoint.py.151: total_norm: 10.170003 [I 2022-10-27 06:56:11] ...trap/engines/engine.py.224: train: epoch 53 | batch 2080/2304 [I 2022-10-27 06:56:11] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:02:06 [I 2022-10-27 06:56:11] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00246 [I 2022-10-27 06:56:11] ...trap/engines/engine.py.230: loss: 0.20468 [I 2022-10-27 06:56:11] ...optimizers/trijoint.py.151: total_norm: 14.533102 [I 2022-10-27 06:56:16] ...trap/engines/engine.py.224: train: epoch 53 | batch 2090/2304 [I 2022-10-27 06:56:16] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:02:01 [I 2022-10-27 06:56:16] ...trap/engines/engine.py.229: process: 0.56127 | load: 0.00247 [I 2022-10-27 06:56:16] ...trap/engines/engine.py.230: loss: 0.19685 [I 2022-10-27 06:56:16] ...optimizers/trijoint.py.151: total_norm: 16.486977 [I 2022-10-27 06:56:22] ...trap/engines/engine.py.224: train: epoch 53 | batch 2100/2304 [I 2022-10-27 06:56:22] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:55 [I 2022-10-27 06:56:22] ...trap/engines/engine.py.229: process: 0.57453 | load: 0.00246 [I 2022-10-27 06:56:22] ...trap/engines/engine.py.230: loss: 0.17517 [I 2022-10-27 06:56:22] ...optimizers/trijoint.py.151: total_norm: 13.760667 [I 2022-10-27 06:56:28] ...trap/engines/engine.py.224: train: epoch 53 | batch 2110/2304 [I 2022-10-27 06:56:28] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:48 [I 2022-10-27 06:56:28] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00240 [I 2022-10-27 06:56:28] ...trap/engines/engine.py.230: loss: 0.12361 [I 2022-10-27 06:56:28] ...optimizers/trijoint.py.151: total_norm: 12.591005 [I 2022-10-27 06:56:33] ...trap/engines/engine.py.224: train: epoch 53 | batch 2120/2304 [I 2022-10-27 06:56:33] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:42 [I 2022-10-27 06:56:33] ...trap/engines/engine.py.229: process: 0.56493 | load: 0.00284 [I 2022-10-27 06:56:33] ...trap/engines/engine.py.230: loss: 0.15052 [I 2022-10-27 06:56:33] ...optimizers/trijoint.py.151: total_norm: 20.579676 [I 2022-10-27 06:56:39] ...trap/engines/engine.py.224: train: epoch 53 | batch 2130/2304 [I 2022-10-27 06:56:39] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:38 [I 2022-10-27 06:56:39] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00261 [I 2022-10-27 06:56:39] ...trap/engines/engine.py.230: loss: 0.13352 [I 2022-10-27 06:56:39] ...optimizers/trijoint.py.151: total_norm: 14.429404 [I 2022-10-27 06:56:45] ...trap/engines/engine.py.224: train: epoch 53 | batch 2140/2304 [I 2022-10-27 06:56:45] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:31 [I 2022-10-27 06:56:45] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00248 [I 2022-10-27 06:56:45] ...trap/engines/engine.py.230: loss: 0.15491 [I 2022-10-27 06:56:45] ...optimizers/trijoint.py.151: total_norm: 15.921553 [I 2022-10-27 06:56:50] ...trap/engines/engine.py.224: train: epoch 53 | batch 2150/2304 [I 2022-10-27 06:56:50] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:26 [I 2022-10-27 06:56:50] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00245 [I 2022-10-27 06:56:50] ...trap/engines/engine.py.230: loss: 0.14947 [I 2022-10-27 06:56:50] ...optimizers/trijoint.py.151: total_norm: 21.664558 [I 2022-10-27 06:56:56] ...trap/engines/engine.py.224: train: epoch 53 | batch 2160/2304 [I 2022-10-27 06:56:56] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:20 [I 2022-10-27 06:56:56] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00253 [I 2022-10-27 06:56:56] ...trap/engines/engine.py.230: loss: 0.16002 [I 2022-10-27 06:56:56] ...optimizers/trijoint.py.151: total_norm: 64.073845 [I 2022-10-27 06:57:01] ...trap/engines/engine.py.224: train: epoch 53 | batch 2170/2304 [I 2022-10-27 06:57:01] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:01:14 [I 2022-10-27 06:57:01] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00253 [I 2022-10-27 06:57:01] ...trap/engines/engine.py.230: loss: 0.16586 [I 2022-10-27 06:57:01] ...optimizers/trijoint.py.151: total_norm: 30.195604 [I 2022-10-27 06:57:07] ...trap/engines/engine.py.224: train: epoch 53 | batch 2180/2304 [I 2022-10-27 06:57:07] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:08 [I 2022-10-27 06:57:07] ...trap/engines/engine.py.229: process: 0.55177 | load: 0.00249 [I 2022-10-27 06:57:07] ...trap/engines/engine.py.230: loss: 0.17819 [I 2022-10-27 06:57:07] ...optimizers/trijoint.py.151: total_norm: 28.907444 [I 2022-10-27 06:57:13] ...trap/engines/engine.py.224: train: epoch 53 | batch 2190/2304 [I 2022-10-27 06:57:13] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:01:05 [I 2022-10-27 06:57:13] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00246 [I 2022-10-27 06:57:13] ...trap/engines/engine.py.230: loss: 0.13202 [I 2022-10-27 06:57:13] ...optimizers/trijoint.py.151: total_norm: 12.695639 [I 2022-10-27 06:57:18] ...trap/engines/engine.py.224: train: epoch 53 | batch 2200/2304 [I 2022-10-27 06:57:18] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:58 [I 2022-10-27 06:57:18] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00250 [I 2022-10-27 06:57:18] ...trap/engines/engine.py.230: loss: 0.14262 [I 2022-10-27 06:57:18] ...optimizers/trijoint.py.151: total_norm: 10.379545 [I 2022-10-27 06:57:24] ...trap/engines/engine.py.224: train: epoch 53 | batch 2210/2304 [I 2022-10-27 06:57:24] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:52 [I 2022-10-27 06:57:24] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00243 [I 2022-10-27 06:57:24] ...trap/engines/engine.py.230: loss: 0.22629 [I 2022-10-27 06:57:24] ...optimizers/trijoint.py.151: total_norm: 14.132760 [I 2022-10-27 06:57:29] ...trap/engines/engine.py.224: train: epoch 53 | batch 2220/2304 [I 2022-10-27 06:57:29] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:46 [I 2022-10-27 06:57:29] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00243 [I 2022-10-27 06:57:29] ...trap/engines/engine.py.230: loss: 0.11965 [I 2022-10-27 06:57:29] ...optimizers/trijoint.py.151: total_norm: 8.837432 [I 2022-10-27 06:57:35] ...trap/engines/engine.py.224: train: epoch 53 | batch 2230/2304 [I 2022-10-27 06:57:35] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:41 [I 2022-10-27 06:57:35] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00251 [I 2022-10-27 06:57:35] ...trap/engines/engine.py.230: loss: 0.13155 [I 2022-10-27 06:57:35] ...optimizers/trijoint.py.151: total_norm: 19.461481 [I 2022-10-27 06:57:41] ...trap/engines/engine.py.224: train: epoch 53 | batch 2240/2304 [I 2022-10-27 06:57:41] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:35 [I 2022-10-27 06:57:41] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00248 [I 2022-10-27 06:57:41] ...trap/engines/engine.py.230: loss: 0.11153 [I 2022-10-27 06:57:41] ...optimizers/trijoint.py.151: total_norm: 9.972693 [I 2022-10-27 06:57:46] ...trap/engines/engine.py.224: train: epoch 53 | batch 2250/2304 [I 2022-10-27 06:57:46] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:29 [I 2022-10-27 06:57:46] ...trap/engines/engine.py.229: process: 0.53949 | load: 0.00250 [I 2022-10-27 06:57:46] ...trap/engines/engine.py.230: loss: 0.17379 [I 2022-10-27 06:57:46] ...optimizers/trijoint.py.151: total_norm: 30.606432 [I 2022-10-27 06:57:52] ...trap/engines/engine.py.224: train: epoch 53 | batch 2260/2304 [I 2022-10-27 06:57:52] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:25 [I 2022-10-27 06:57:52] ...trap/engines/engine.py.229: process: 0.55150 | load: 0.00245 [I 2022-10-27 06:57:52] ...trap/engines/engine.py.230: loss: 0.15371 [I 2022-10-27 06:57:52] ...optimizers/trijoint.py.151: total_norm: 15.162138 [I 2022-10-27 06:57:57] ...trap/engines/engine.py.224: train: epoch 53 | batch 2270/2304 [I 2022-10-27 06:57:57] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:18 [I 2022-10-27 06:57:57] ...trap/engines/engine.py.229: process: 0.54288 | load: 0.00252 [I 2022-10-27 06:57:57] ...trap/engines/engine.py.230: loss: 0.20679 [I 2022-10-27 06:57:57] ...optimizers/trijoint.py.151: total_norm: 22.157413 [I 2022-10-27 06:58:03] ...trap/engines/engine.py.224: train: epoch 53 | batch 2280/2304 [I 2022-10-27 06:58:03] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:13 [I 2022-10-27 06:58:03] ...trap/engines/engine.py.229: process: 0.53991 | load: 0.00243 [I 2022-10-27 06:58:03] ...trap/engines/engine.py.230: loss: 0.14516 [I 2022-10-27 06:58:03] ...optimizers/trijoint.py.151: total_norm: 18.105238 [I 2022-10-27 06:58:09] ...trap/engines/engine.py.224: train: epoch 53 | batch 2290/2304 [I 2022-10-27 06:58:09] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:00:07 [I 2022-10-27 06:58:09] ...trap/engines/engine.py.229: process: 0.54379 | load: 0.00247 [I 2022-10-27 06:58:09] ...trap/engines/engine.py.230: loss: 0.21705 [I 2022-10-27 06:58:09] ...optimizers/trijoint.py.151: total_norm: 16.288479 [I 2022-10-27 06:58:14] ...trap/engines/engine.py.224: train: epoch 53 | batch 2300/2304 [I 2022-10-27 06:58:14] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:02 [I 2022-10-27 06:58:14] ...trap/engines/engine.py.229: process: 0.54460 | load: 0.00015 [I 2022-10-27 06:58:14] ...trap/engines/engine.py.230: loss: 0.14426 [I 2022-10-27 06:58:14] ...optimizers/trijoint.py.151: total_norm: 16.733894 [I 2022-10-27 06:58:16] ...trap/engines/engine.py.224: train: epoch 53 | batch 2304/2304 [I 2022-10-27 06:58:16] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:00 [I 2022-10-27 06:58:16] ...trap/engines/engine.py.229: process: 0.55137 | load: 0.00015 [I 2022-10-27 06:58:16] ...trap/engines/engine.py.230: loss: 0.13354 [I 2022-10-27 06:58:16] ...optimizers/trijoint.py.151: total_norm: 21.776375 [S 2022-10-27 06:58:18] ...trap/engines/engine.py.129: train_epoch.epoch: 53 [S 2022-10-27 06:58:18] ...trap/engines/engine.py.129: train_epoch.loss: 0.1606916065943991 [I 2022-10-27 06:58:18] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 06:58:18] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 53 [I 2022-10-27 06:58:18] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 06:58:20] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 06:58:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 06:58:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 06:58:22] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 06:58:22] ...trap/engines/engine.py.321: eval: epoch 53 | batch 0/510 [I 2022-10-27 06:58:22] ...trap/engines/engine.py.325: elapsed: 0:00:03 | left: 0:33:52 [I 2022-10-27 06:58:22] ...trap/engines/engine.py.326: process: 3.98563 | load: 2.96738 [I 2022-10-27 06:58:26] ...trap/engines/engine.py.321: eval: epoch 53 | batch 10/510 [I 2022-10-27 06:58:26] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:10 [I 2022-10-27 06:58:26] ...trap/engines/engine.py.326: process: 0.40545 | load: 0.00017 [I 2022-10-27 06:58:29] ...trap/engines/engine.py.321: eval: epoch 53 | batch 20/510 [I 2022-10-27 06:58:29] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:04 [I 2022-10-27 06:58:29] ...trap/engines/engine.py.326: process: 0.31862 | load: 0.00017 [I 2022-10-27 06:58:33] ...trap/engines/engine.py.321: eval: epoch 53 | batch 30/510 [I 2022-10-27 06:58:33] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:40 [I 2022-10-27 06:58:33] ...trap/engines/engine.py.326: process: 0.32714 | load: 0.00016 [I 2022-10-27 06:58:36] ...trap/engines/engine.py.321: eval: epoch 53 | batch 40/510 [I 2022-10-27 06:58:36] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:35 [I 2022-10-27 06:58:36] ...trap/engines/engine.py.326: process: 0.32494 | load: 0.00015 [I 2022-10-27 06:58:39] ...trap/engines/engine.py.321: eval: epoch 53 | batch 50/510 [I 2022-10-27 06:58:39] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:32 [I 2022-10-27 06:58:39] ...trap/engines/engine.py.326: process: 0.32886 | load: 0.00018 [I 2022-10-27 06:58:43] ...trap/engines/engine.py.321: eval: epoch 53 | batch 60/510 [I 2022-10-27 06:58:43] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:30 [I 2022-10-27 06:58:43] ...trap/engines/engine.py.326: process: 0.32738 | load: 0.00017 [I 2022-10-27 06:58:46] ...trap/engines/engine.py.321: eval: epoch 53 | batch 70/510 [I 2022-10-27 06:58:46] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:25 [I 2022-10-27 06:58:46] ...trap/engines/engine.py.326: process: 0.31756 | load: 0.00016 [I 2022-10-27 06:58:49] ...trap/engines/engine.py.321: eval: epoch 53 | batch 80/510 [I 2022-10-27 06:58:49] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:22 [I 2022-10-27 06:58:49] ...trap/engines/engine.py.326: process: 0.32684 | load: 0.00021 [I 2022-10-27 06:58:53] ...trap/engines/engine.py.321: eval: epoch 53 | batch 90/510 [I 2022-10-27 06:58:53] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:20 [I 2022-10-27 06:58:53] ...trap/engines/engine.py.326: process: 0.32869 | load: 0.00015 [I 2022-10-27 06:58:56] ...trap/engines/engine.py.321: eval: epoch 53 | batch 100/510 [I 2022-10-27 06:58:56] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:12 [I 2022-10-27 06:58:56] ...trap/engines/engine.py.326: process: 0.28066 | load: 0.00017 [I 2022-10-27 06:58:59] ...trap/engines/engine.py.321: eval: epoch 53 | batch 110/510 [I 2022-10-27 06:58:59] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:57 [I 2022-10-27 06:58:59] ...trap/engines/engine.py.326: process: 0.29047 | load: 0.00017 [I 2022-10-27 06:59:02] ...trap/engines/engine.py.321: eval: epoch 53 | batch 120/510 [I 2022-10-27 06:59:02] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:52 [I 2022-10-27 06:59:02] ...trap/engines/engine.py.326: process: 0.28721 | load: 0.00014 [I 2022-10-27 06:59:05] ...trap/engines/engine.py.321: eval: epoch 53 | batch 130/510 [I 2022-10-27 06:59:05] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:51 [I 2022-10-27 06:59:05] ...trap/engines/engine.py.326: process: 0.29002 | load: 0.00016 [I 2022-10-27 06:59:07] ...trap/engines/engine.py.321: eval: epoch 53 | batch 140/510 [I 2022-10-27 06:59:07] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 06:59:07] ...trap/engines/engine.py.326: process: 0.28553 | load: 0.00021 [I 2022-10-27 06:59:10] ...trap/engines/engine.py.321: eval: epoch 53 | batch 150/510 [I 2022-10-27 06:59:10] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-27 06:59:10] ...trap/engines/engine.py.326: process: 0.28091 | load: 0.00015 [I 2022-10-27 06:59:13] ...trap/engines/engine.py.321: eval: epoch 53 | batch 160/510 [I 2022-10-27 06:59:13] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-27 06:59:13] ...trap/engines/engine.py.326: process: 0.28732 | load: 0.00015 [I 2022-10-27 06:59:16] ...trap/engines/engine.py.321: eval: epoch 53 | batch 170/510 [I 2022-10-27 06:59:16] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-27 06:59:16] ...trap/engines/engine.py.326: process: 0.28807 | load: 0.00015 [I 2022-10-27 06:59:19] ...trap/engines/engine.py.321: eval: epoch 53 | batch 180/510 [I 2022-10-27 06:59:19] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 06:59:19] ...trap/engines/engine.py.326: process: 0.29323 | load: 0.00015 [I 2022-10-27 06:59:22] ...trap/engines/engine.py.321: eval: epoch 53 | batch 190/510 [I 2022-10-27 06:59:22] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 06:59:22] ...trap/engines/engine.py.326: process: 0.28944 | load: 0.00014 [I 2022-10-27 06:59:25] ...trap/engines/engine.py.321: eval: epoch 53 | batch 200/510 [I 2022-10-27 06:59:25] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 06:59:25] ...trap/engines/engine.py.326: process: 0.28502 | load: 0.00015 [I 2022-10-27 06:59:28] ...trap/engines/engine.py.321: eval: epoch 53 | batch 210/510 [I 2022-10-27 06:59:28] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 06:59:28] ...trap/engines/engine.py.326: process: 0.28375 | load: 0.00021 [I 2022-10-27 06:59:31] ...trap/engines/engine.py.321: eval: epoch 53 | batch 220/510 [I 2022-10-27 06:59:31] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:22 [I 2022-10-27 06:59:31] ...trap/engines/engine.py.326: process: 0.28565 | load: 0.00016 [I 2022-10-27 06:59:33] ...trap/engines/engine.py.321: eval: epoch 53 | batch 230/510 [I 2022-10-27 06:59:33] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:19 [I 2022-10-27 06:59:33] ...trap/engines/engine.py.326: process: 0.28403 | load: 0.00014 [I 2022-10-27 06:59:36] ...trap/engines/engine.py.321: eval: epoch 53 | batch 240/510 [I 2022-10-27 06:59:36] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:22 [I 2022-10-27 06:59:36] ...trap/engines/engine.py.326: process: 0.29186 | load: 0.00015 [I 2022-10-27 06:59:39] ...trap/engines/engine.py.321: eval: epoch 53 | batch 250/510 [I 2022-10-27 06:59:39] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-27 06:59:39] ...trap/engines/engine.py.326: process: 0.29026 | load: 0.00016 [I 2022-10-27 06:59:42] ...trap/engines/engine.py.321: eval: epoch 53 | batch 260/510 [I 2022-10-27 06:59:42] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 06:59:42] ...trap/engines/engine.py.326: process: 0.28878 | load: 0.00015 [I 2022-10-27 06:59:45] ...trap/engines/engine.py.321: eval: epoch 53 | batch 270/510 [I 2022-10-27 06:59:45] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 06:59:45] ...trap/engines/engine.py.326: process: 0.28715 | load: 0.00016 [I 2022-10-27 06:59:48] ...trap/engines/engine.py.321: eval: epoch 53 | batch 280/510 [I 2022-10-27 06:59:48] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 06:59:48] ...trap/engines/engine.py.326: process: 0.29098 | load: 0.00016 [I 2022-10-27 06:59:51] ...trap/engines/engine.py.321: eval: epoch 53 | batch 290/510 [I 2022-10-27 06:59:51] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 06:59:51] ...trap/engines/engine.py.326: process: 0.28858 | load: 0.00019 [I 2022-10-27 06:59:54] ...trap/engines/engine.py.321: eval: epoch 53 | batch 300/510 [I 2022-10-27 06:59:54] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 06:59:54] ...trap/engines/engine.py.326: process: 0.28629 | load: 0.00017 [I 2022-10-27 06:59:57] ...trap/engines/engine.py.321: eval: epoch 53 | batch 310/510 [I 2022-10-27 06:59:57] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 06:59:57] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00015 [I 2022-10-27 06:59:59] ...trap/engines/engine.py.321: eval: epoch 53 | batch 320/510 [I 2022-10-27 06:59:59] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 06:59:59] ...trap/engines/engine.py.326: process: 0.28735 | load: 0.00020 [I 2022-10-27 07:00:02] ...trap/engines/engine.py.321: eval: epoch 53 | batch 330/510 [I 2022-10-27 07:00:02] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 07:00:02] ...trap/engines/engine.py.326: process: 0.28599 | load: 0.00014 [I 2022-10-27 07:00:05] ...trap/engines/engine.py.321: eval: epoch 53 | batch 340/510 [I 2022-10-27 07:00:05] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-27 07:00:05] ...trap/engines/engine.py.326: process: 0.28080 | load: 0.00015 [I 2022-10-27 07:00:08] ...trap/engines/engine.py.321: eval: epoch 53 | batch 350/510 [I 2022-10-27 07:00:08] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-27 07:00:08] ...trap/engines/engine.py.326: process: 0.28874 | load: 0.00014 [I 2022-10-27 07:00:11] ...trap/engines/engine.py.321: eval: epoch 53 | batch 360/510 [I 2022-10-27 07:00:11] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 07:00:11] ...trap/engines/engine.py.326: process: 0.28870 | load: 0.00015 [I 2022-10-27 07:00:14] ...trap/engines/engine.py.321: eval: epoch 53 | batch 370/510 [I 2022-10-27 07:00:14] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:42 [I 2022-10-27 07:00:14] ...trap/engines/engine.py.326: process: 0.33205 | load: 0.00023 [I 2022-10-27 07:00:17] ...trap/engines/engine.py.321: eval: epoch 53 | batch 380/510 [I 2022-10-27 07:00:17] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 07:00:17] ...trap/engines/engine.py.326: process: 0.29323 | load: 0.00021 [I 2022-10-27 07:00:20] ...trap/engines/engine.py.321: eval: epoch 53 | batch 390/510 [I 2022-10-27 07:00:20] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 07:00:20] ...trap/engines/engine.py.326: process: 0.29374 | load: 0.00015 [I 2022-10-27 07:00:22] ...trap/engines/engine.py.321: eval: epoch 53 | batch 400/510 [I 2022-10-27 07:00:22] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 07:00:22] ...trap/engines/engine.py.326: process: 0.28837 | load: 0.00016 [I 2022-10-27 07:00:25] ...trap/engines/engine.py.321: eval: epoch 53 | batch 410/510 [I 2022-10-27 07:00:25] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-27 07:00:25] ...trap/engines/engine.py.326: process: 0.28830 | load: 0.00015 [I 2022-10-27 07:00:28] ...trap/engines/engine.py.321: eval: epoch 53 | batch 420/510 [I 2022-10-27 07:00:28] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-27 07:00:28] ...trap/engines/engine.py.326: process: 0.28498 | load: 0.00017 [I 2022-10-27 07:00:31] ...trap/engines/engine.py.321: eval: epoch 53 | batch 430/510 [I 2022-10-27 07:00:31] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-27 07:00:31] ...trap/engines/engine.py.326: process: 0.29514 | load: 0.00025 [I 2022-10-27 07:00:34] ...trap/engines/engine.py.321: eval: epoch 53 | batch 440/510 [I 2022-10-27 07:00:34] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 07:00:34] ...trap/engines/engine.py.326: process: 0.29154 | load: 0.00016 [I 2022-10-27 07:00:37] ...trap/engines/engine.py.321: eval: epoch 53 | batch 450/510 [I 2022-10-27 07:00:37] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 07:00:37] ...trap/engines/engine.py.326: process: 0.28231 | load: 0.00015 [I 2022-10-27 07:00:40] ...trap/engines/engine.py.321: eval: epoch 53 | batch 460/510 [I 2022-10-27 07:00:40] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 07:00:40] ...trap/engines/engine.py.326: process: 0.28274 | load: 0.00015 [I 2022-10-27 07:00:43] ...trap/engines/engine.py.321: eval: epoch 53 | batch 470/510 [I 2022-10-27 07:00:43] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 07:00:43] ...trap/engines/engine.py.326: process: 0.29045 | load: 0.00016 [I 2022-10-27 07:00:45] ...trap/engines/engine.py.321: eval: epoch 53 | batch 480/510 [I 2022-10-27 07:00:45] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 07:00:45] ...trap/engines/engine.py.326: process: 0.28411 | load: 0.00021 [I 2022-10-27 07:00:48] ...trap/engines/engine.py.321: eval: epoch 53 | batch 490/510 [I 2022-10-27 07:00:48] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:06 [I 2022-10-27 07:00:48] ...trap/engines/engine.py.326: process: 0.28158 | load: 0.00016 [I 2022-10-27 07:00:51] ...trap/engines/engine.py.321: eval: epoch 53 | batch 500/510 [I 2022-10-27 07:00:51] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 07:00:51] ...trap/engines/engine.py.326: process: 0.28772 | load: 0.00015 [I 2022-10-27 07:00:54] ...trap/engines/engine.py.321: eval: epoch 53 | batch 510/510 [I 2022-10-27 07:00:54] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 07:00:54] ...trap/engines/engine.py.326: process: 0.28571 | load: 0.00010 [S 2022-10-27 07:00:55] ...trap/engines/engine.py.132: eval_epoch.epoch: 53 [I 2022-10-27 07:00:55] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.703600 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.896500 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934100 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.695400 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.895500 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.930300 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010210 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007839 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004415 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014814 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007864 [S 2022-10-27 07:00:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007226 [I 2022-10-27 07:01:00] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 07:01:00] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 07:01:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 07:01:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 07:01:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 07:01:03] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 07:01:05] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 07:01:11] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 07:01:11] ...trap/engines/engine.py.163: Training model on trainset for epoch 54 [I 2022-10-27 07:01:11] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 07:01:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 0/2304 [I 2022-10-27 07:01:14] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:55 [I 2022-10-27 07:01:14] ...trap/engines/engine.py.229: process: 2.70648 | load: 2.05505 [I 2022-10-27 07:01:14] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-27 07:01:14] ...optimizers/trijoint.py.151: total_norm: 39.867325 [I 2022-10-27 07:01:19] ...trap/engines/engine.py.224: train: epoch 54 | batch 10/2304 [I 2022-10-27 07:01:19] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:01 [I 2022-10-27 07:01:19] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00481 [I 2022-10-27 07:01:19] ...trap/engines/engine.py.230: loss: 0.12382 [I 2022-10-27 07:01:19] ...optimizers/trijoint.py.151: total_norm: 10.228413 [I 2022-10-27 07:01:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 20/2304 [I 2022-10-27 07:01:25] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:05 [I 2022-10-27 07:01:25] ...trap/engines/engine.py.229: process: 0.54539 | load: 0.00354 [I 2022-10-27 07:01:25] ...trap/engines/engine.py.230: loss: 0.19715 [I 2022-10-27 07:01:25] ...optimizers/trijoint.py.151: total_norm: 33.954559 [I 2022-10-27 07:01:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 30/2304 [I 2022-10-27 07:01:31] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:03 [I 2022-10-27 07:01:31] ...trap/engines/engine.py.229: process: 0.55552 | load: 0.00290 [I 2022-10-27 07:01:31] ...trap/engines/engine.py.230: loss: 0.17269 [I 2022-10-27 07:01:31] ...optimizers/trijoint.py.151: total_norm: 12.436623 [I 2022-10-27 07:01:36] ...trap/engines/engine.py.224: train: epoch 54 | batch 40/2304 [I 2022-10-27 07:01:36] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:21:00 [I 2022-10-27 07:01:36] ...trap/engines/engine.py.229: process: 0.56165 | load: 0.00305 [I 2022-10-27 07:01:36] ...trap/engines/engine.py.230: loss: 0.17504 [I 2022-10-27 07:01:36] ...optimizers/trijoint.py.151: total_norm: 20.181423 [I 2022-10-27 07:01:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 50/2304 [I 2022-10-27 07:01:42] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:55 [I 2022-10-27 07:01:42] ...trap/engines/engine.py.229: process: 0.55433 | load: 0.00256 [I 2022-10-27 07:01:42] ...trap/engines/engine.py.230: loss: 0.14469 [I 2022-10-27 07:01:42] ...optimizers/trijoint.py.151: total_norm: 17.933599 [I 2022-10-27 07:01:47] ...trap/engines/engine.py.224: train: epoch 54 | batch 60/2304 [I 2022-10-27 07:01:47] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:52 [I 2022-10-27 07:01:47] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00278 [I 2022-10-27 07:01:47] ...trap/engines/engine.py.230: loss: 0.16587 [I 2022-10-27 07:01:47] ...optimizers/trijoint.py.151: total_norm: 20.373846 [I 2022-10-27 07:01:53] ...trap/engines/engine.py.224: train: epoch 54 | batch 70/2304 [I 2022-10-27 07:01:53] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:43 [I 2022-10-27 07:01:53] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00305 [I 2022-10-27 07:01:53] ...trap/engines/engine.py.230: loss: 0.15402 [I 2022-10-27 07:01:53] ...optimizers/trijoint.py.151: total_norm: 16.936745 [I 2022-10-27 07:01:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 80/2304 [I 2022-10-27 07:01:59] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:53 [I 2022-10-27 07:01:59] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00315 [I 2022-10-27 07:01:59] ...trap/engines/engine.py.230: loss: 0.19293 [I 2022-10-27 07:01:59] ...optimizers/trijoint.py.151: total_norm: 20.891226 [I 2022-10-27 07:02:04] ...trap/engines/engine.py.224: train: epoch 54 | batch 90/2304 [I 2022-10-27 07:02:04] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:46 [I 2022-10-27 07:02:04] ...trap/engines/engine.py.229: process: 0.55413 | load: 0.00268 [I 2022-10-27 07:02:04] ...trap/engines/engine.py.230: loss: 0.16495 [I 2022-10-27 07:02:04] ...optimizers/trijoint.py.151: total_norm: 14.675883 [I 2022-10-27 07:02:10] ...trap/engines/engine.py.224: train: epoch 54 | batch 100/2304 [I 2022-10-27 07:02:10] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:42 [I 2022-10-27 07:02:10] ...trap/engines/engine.py.229: process: 0.57682 | load: 0.00258 [I 2022-10-27 07:02:10] ...trap/engines/engine.py.230: loss: 0.15200 [I 2022-10-27 07:02:10] ...optimizers/trijoint.py.151: total_norm: 12.182674 [I 2022-10-27 07:02:15] ...trap/engines/engine.py.224: train: epoch 54 | batch 110/2304 [I 2022-10-27 07:02:15] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:14 [I 2022-10-27 07:02:15] ...trap/engines/engine.py.229: process: 0.54207 | load: 0.00259 [I 2022-10-27 07:02:15] ...trap/engines/engine.py.230: loss: 0.14928 [I 2022-10-27 07:02:15] ...optimizers/trijoint.py.151: total_norm: 22.657555 [I 2022-10-27 07:02:21] ...trap/engines/engine.py.224: train: epoch 54 | batch 120/2304 [I 2022-10-27 07:02:21] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:11 [I 2022-10-27 07:02:21] ...trap/engines/engine.py.229: process: 0.56995 | load: 0.00254 [I 2022-10-27 07:02:21] ...trap/engines/engine.py.230: loss: 0.14257 [I 2022-10-27 07:02:21] ...optimizers/trijoint.py.151: total_norm: 10.317854 [I 2022-10-27 07:02:27] ...trap/engines/engine.py.224: train: epoch 54 | batch 130/2304 [I 2022-10-27 07:02:27] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:16 [I 2022-10-27 07:02:27] ...trap/engines/engine.py.229: process: 0.57012 | load: 0.00245 [I 2022-10-27 07:02:27] ...trap/engines/engine.py.230: loss: 0.21009 [I 2022-10-27 07:02:27] ...optimizers/trijoint.py.151: total_norm: 19.593643 [I 2022-10-27 07:02:32] ...trap/engines/engine.py.224: train: epoch 54 | batch 140/2304 [I 2022-10-27 07:02:32] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:57 [I 2022-10-27 07:02:32] ...trap/engines/engine.py.229: process: 0.54970 | load: 0.00254 [I 2022-10-27 07:02:32] ...trap/engines/engine.py.230: loss: 0.23931 [I 2022-10-27 07:02:32] ...optimizers/trijoint.py.151: total_norm: 16.154724 [I 2022-10-27 07:02:38] ...trap/engines/engine.py.224: train: epoch 54 | batch 150/2304 [I 2022-10-27 07:02:38] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:57 [I 2022-10-27 07:02:38] ...trap/engines/engine.py.229: process: 0.55786 | load: 0.00251 [I 2022-10-27 07:02:38] ...trap/engines/engine.py.230: loss: 0.13409 [I 2022-10-27 07:02:38] ...optimizers/trijoint.py.151: total_norm: 15.537606 [I 2022-10-27 07:02:43] ...trap/engines/engine.py.224: train: epoch 54 | batch 160/2304 [I 2022-10-27 07:02:43] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:45 [I 2022-10-27 07:02:43] ...trap/engines/engine.py.229: process: 0.55460 | load: 0.00267 [I 2022-10-27 07:02:43] ...trap/engines/engine.py.230: loss: 0.13669 [I 2022-10-27 07:02:43] ...optimizers/trijoint.py.151: total_norm: 8.655306 [I 2022-10-27 07:02:49] ...trap/engines/engine.py.224: train: epoch 54 | batch 170/2304 [I 2022-10-27 07:02:49] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:43 [I 2022-10-27 07:02:49] ...trap/engines/engine.py.229: process: 0.53749 | load: 0.00256 [I 2022-10-27 07:02:49] ...trap/engines/engine.py.230: loss: 0.14691 [I 2022-10-27 07:02:49] ...optimizers/trijoint.py.151: total_norm: 24.448700 [I 2022-10-27 07:02:54] ...trap/engines/engine.py.224: train: epoch 54 | batch 180/2304 [I 2022-10-27 07:02:54] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:40 [I 2022-10-27 07:02:54] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00253 [I 2022-10-27 07:02:54] ...trap/engines/engine.py.230: loss: 0.11844 [I 2022-10-27 07:02:54] ...optimizers/trijoint.py.151: total_norm: 13.566503 [I 2022-10-27 07:03:00] ...trap/engines/engine.py.224: train: epoch 54 | batch 190/2304 [I 2022-10-27 07:03:00] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:43 [I 2022-10-27 07:03:00] ...trap/engines/engine.py.229: process: 0.56168 | load: 0.00268 [I 2022-10-27 07:03:00] ...trap/engines/engine.py.230: loss: 0.14420 [I 2022-10-27 07:03:00] ...optimizers/trijoint.py.151: total_norm: 21.546452 [I 2022-10-27 07:03:06] ...trap/engines/engine.py.224: train: epoch 54 | batch 200/2304 [I 2022-10-27 07:03:06] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:18 [I 2022-10-27 07:03:06] ...trap/engines/engine.py.229: process: 0.54367 | load: 0.00254 [I 2022-10-27 07:03:06] ...trap/engines/engine.py.230: loss: 0.14329 [I 2022-10-27 07:03:06] ...optimizers/trijoint.py.151: total_norm: 10.385084 [I 2022-10-27 07:03:11] ...trap/engines/engine.py.224: train: epoch 54 | batch 210/2304 [I 2022-10-27 07:03:11] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:25 [I 2022-10-27 07:03:11] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00243 [I 2022-10-27 07:03:11] ...trap/engines/engine.py.230: loss: 0.28729 [I 2022-10-27 07:03:11] ...optimizers/trijoint.py.151: total_norm: 23.358250 [I 2022-10-27 07:03:17] ...trap/engines/engine.py.224: train: epoch 54 | batch 220/2304 [I 2022-10-27 07:03:17] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:20 [I 2022-10-27 07:03:17] ...trap/engines/engine.py.229: process: 0.56157 | load: 0.00243 [I 2022-10-27 07:03:17] ...trap/engines/engine.py.230: loss: 0.17280 [I 2022-10-27 07:03:17] ...optimizers/trijoint.py.151: total_norm: 11.195691 [I 2022-10-27 07:03:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 230/2304 [I 2022-10-27 07:03:22] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:14 [I 2022-10-27 07:03:22] ...trap/engines/engine.py.229: process: 0.55975 | load: 0.00261 [I 2022-10-27 07:03:22] ...trap/engines/engine.py.230: loss: 0.20351 [I 2022-10-27 07:03:22] ...optimizers/trijoint.py.151: total_norm: 16.180706 [I 2022-10-27 07:03:28] ...trap/engines/engine.py.224: train: epoch 54 | batch 240/2304 [I 2022-10-27 07:03:28] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:10 [I 2022-10-27 07:03:28] ...trap/engines/engine.py.229: process: 0.55461 | load: 0.00249 [I 2022-10-27 07:03:28] ...trap/engines/engine.py.230: loss: 0.14648 [I 2022-10-27 07:03:28] ...optimizers/trijoint.py.151: total_norm: 14.916819 [I 2022-10-27 07:03:34] ...trap/engines/engine.py.224: train: epoch 54 | batch 250/2304 [I 2022-10-27 07:03:34] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:03 [I 2022-10-27 07:03:34] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00264 [I 2022-10-27 07:03:34] ...trap/engines/engine.py.230: loss: 0.11335 [I 2022-10-27 07:03:34] ...optimizers/trijoint.py.151: total_norm: 16.342222 [I 2022-10-27 07:03:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 260/2304 [I 2022-10-27 07:03:39] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:59 [I 2022-10-27 07:03:39] ...trap/engines/engine.py.229: process: 0.56114 | load: 0.00262 [I 2022-10-27 07:03:39] ...trap/engines/engine.py.230: loss: 0.14818 [I 2022-10-27 07:03:39] ...optimizers/trijoint.py.151: total_norm: 17.175968 [I 2022-10-27 07:03:45] ...trap/engines/engine.py.224: train: epoch 54 | batch 270/2304 [I 2022-10-27 07:03:45] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:51 [I 2022-10-27 07:03:45] ...trap/engines/engine.py.229: process: 0.56329 | load: 0.00248 [I 2022-10-27 07:03:45] ...trap/engines/engine.py.230: loss: 0.26202 [I 2022-10-27 07:03:45] ...optimizers/trijoint.py.151: total_norm: 16.218357 [I 2022-10-27 07:03:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 280/2304 [I 2022-10-27 07:03:50] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:52 [I 2022-10-27 07:03:50] ...trap/engines/engine.py.229: process: 0.56598 | load: 0.00274 [I 2022-10-27 07:03:50] ...trap/engines/engine.py.230: loss: 0.17676 [I 2022-10-27 07:03:50] ...optimizers/trijoint.py.151: total_norm: 21.273010 [I 2022-10-27 07:03:56] ...trap/engines/engine.py.224: train: epoch 54 | batch 290/2304 [I 2022-10-27 07:03:56] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:35 [I 2022-10-27 07:03:56] ...trap/engines/engine.py.229: process: 0.54719 | load: 0.00241 [I 2022-10-27 07:03:56] ...trap/engines/engine.py.230: loss: 0.19974 [I 2022-10-27 07:03:56] ...optimizers/trijoint.py.151: total_norm: 28.096157 [I 2022-10-27 07:04:01] ...trap/engines/engine.py.224: train: epoch 54 | batch 300/2304 [I 2022-10-27 07:04:01] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:34 [I 2022-10-27 07:04:01] ...trap/engines/engine.py.229: process: 0.55044 | load: 0.00243 [I 2022-10-27 07:04:01] ...trap/engines/engine.py.230: loss: 0.15364 [I 2022-10-27 07:04:01] ...optimizers/trijoint.py.151: total_norm: 14.886305 [I 2022-10-27 07:04:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 310/2304 [I 2022-10-27 07:04:07] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:24 [I 2022-10-27 07:04:07] ...trap/engines/engine.py.229: process: 0.54930 | load: 0.00258 [I 2022-10-27 07:04:07] ...trap/engines/engine.py.230: loss: 0.18044 [I 2022-10-27 07:04:07] ...optimizers/trijoint.py.151: total_norm: 17.361727 [I 2022-10-27 07:04:13] ...trap/engines/engine.py.224: train: epoch 54 | batch 320/2304 [I 2022-10-27 07:04:13] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:17 [I 2022-10-27 07:04:13] ...trap/engines/engine.py.229: process: 0.54660 | load: 0.00249 [I 2022-10-27 07:04:13] ...trap/engines/engine.py.230: loss: 0.24365 [I 2022-10-27 07:04:13] ...optimizers/trijoint.py.151: total_norm: 43.124546 [I 2022-10-27 07:04:18] ...trap/engines/engine.py.224: train: epoch 54 | batch 330/2304 [I 2022-10-27 07:04:18] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:13 [I 2022-10-27 07:04:18] ...trap/engines/engine.py.229: process: 0.55492 | load: 0.00247 [I 2022-10-27 07:04:18] ...trap/engines/engine.py.230: loss: 0.11295 [I 2022-10-27 07:04:18] ...optimizers/trijoint.py.151: total_norm: 13.478672 [I 2022-10-27 07:04:24] ...trap/engines/engine.py.224: train: epoch 54 | batch 340/2304 [I 2022-10-27 07:04:24] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:09 [I 2022-10-27 07:04:24] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00243 [I 2022-10-27 07:04:24] ...trap/engines/engine.py.230: loss: 0.20884 [I 2022-10-27 07:04:24] ...optimizers/trijoint.py.151: total_norm: 17.210340 [I 2022-10-27 07:04:29] ...trap/engines/engine.py.224: train: epoch 54 | batch 350/2304 [I 2022-10-27 07:04:29] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:18:07 [I 2022-10-27 07:04:29] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00244 [I 2022-10-27 07:04:29] ...trap/engines/engine.py.230: loss: 0.19423 [I 2022-10-27 07:04:29] ...optimizers/trijoint.py.151: total_norm: 26.391312 [I 2022-10-27 07:04:35] ...trap/engines/engine.py.224: train: epoch 54 | batch 360/2304 [I 2022-10-27 07:04:35] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:00 [I 2022-10-27 07:04:35] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00244 [I 2022-10-27 07:04:35] ...trap/engines/engine.py.230: loss: 0.11653 [I 2022-10-27 07:04:35] ...optimizers/trijoint.py.151: total_norm: 14.961768 [I 2022-10-27 07:04:40] ...trap/engines/engine.py.224: train: epoch 54 | batch 370/2304 [I 2022-10-27 07:04:40] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:00 [I 2022-10-27 07:04:40] ...trap/engines/engine.py.229: process: 0.57202 | load: 0.00242 [I 2022-10-27 07:04:40] ...trap/engines/engine.py.230: loss: 0.14133 [I 2022-10-27 07:04:40] ...optimizers/trijoint.py.151: total_norm: 11.156613 [I 2022-10-27 07:04:46] ...trap/engines/engine.py.224: train: epoch 54 | batch 380/2304 [I 2022-10-27 07:04:46] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:54 [I 2022-10-27 07:04:46] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00253 [I 2022-10-27 07:04:46] ...trap/engines/engine.py.230: loss: 0.13912 [I 2022-10-27 07:04:46] ...optimizers/trijoint.py.151: total_norm: 14.754540 [I 2022-10-27 07:04:51] ...trap/engines/engine.py.224: train: epoch 54 | batch 390/2304 [I 2022-10-27 07:04:51] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:44 [I 2022-10-27 07:04:51] ...trap/engines/engine.py.229: process: 0.55880 | load: 0.00247 [I 2022-10-27 07:04:51] ...trap/engines/engine.py.230: loss: 0.20636 [I 2022-10-27 07:04:51] ...optimizers/trijoint.py.151: total_norm: 18.580519 [I 2022-10-27 07:04:57] ...trap/engines/engine.py.224: train: epoch 54 | batch 400/2304 [I 2022-10-27 07:04:57] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:42 [I 2022-10-27 07:04:57] ...trap/engines/engine.py.229: process: 0.55687 | load: 0.00248 [I 2022-10-27 07:04:57] ...trap/engines/engine.py.230: loss: 0.11220 [I 2022-10-27 07:04:57] ...optimizers/trijoint.py.151: total_norm: 8.879385 [I 2022-10-27 07:05:03] ...trap/engines/engine.py.224: train: epoch 54 | batch 410/2304 [I 2022-10-27 07:05:03] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:33 [I 2022-10-27 07:05:03] ...trap/engines/engine.py.229: process: 0.55234 | load: 0.00244 [I 2022-10-27 07:05:03] ...trap/engines/engine.py.230: loss: 0.10933 [I 2022-10-27 07:05:03] ...optimizers/trijoint.py.151: total_norm: 8.791206 [I 2022-10-27 07:05:08] ...trap/engines/engine.py.224: train: epoch 54 | batch 420/2304 [I 2022-10-27 07:05:08] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:27 [I 2022-10-27 07:05:08] ...trap/engines/engine.py.229: process: 0.55434 | load: 0.00244 [I 2022-10-27 07:05:08] ...trap/engines/engine.py.230: loss: 0.14838 [I 2022-10-27 07:05:08] ...optimizers/trijoint.py.151: total_norm: 12.741290 [I 2022-10-27 07:05:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 430/2304 [I 2022-10-27 07:05:14] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:24 [I 2022-10-27 07:05:14] ...trap/engines/engine.py.229: process: 0.56351 | load: 0.00251 [I 2022-10-27 07:05:14] ...trap/engines/engine.py.230: loss: 0.12510 [I 2022-10-27 07:05:14] ...optimizers/trijoint.py.151: total_norm: 10.944051 [I 2022-10-27 07:05:19] ...trap/engines/engine.py.224: train: epoch 54 | batch 440/2304 [I 2022-10-27 07:05:19] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:23 [I 2022-10-27 07:05:19] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00245 [I 2022-10-27 07:05:19] ...trap/engines/engine.py.230: loss: 0.14411 [I 2022-10-27 07:05:19] ...optimizers/trijoint.py.151: total_norm: 33.977989 [I 2022-10-27 07:05:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 450/2304 [I 2022-10-27 07:05:25] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:13 [I 2022-10-27 07:05:25] ...trap/engines/engine.py.229: process: 0.55897 | load: 0.00268 [I 2022-10-27 07:05:25] ...trap/engines/engine.py.230: loss: 0.19223 [I 2022-10-27 07:05:25] ...optimizers/trijoint.py.151: total_norm: 18.814756 [I 2022-10-27 07:05:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 460/2304 [I 2022-10-27 07:05:31] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:04 [I 2022-10-27 07:05:31] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00246 [I 2022-10-27 07:05:31] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-27 07:05:31] ...optimizers/trijoint.py.151: total_norm: 13.281053 [I 2022-10-27 07:05:36] ...trap/engines/engine.py.224: train: epoch 54 | batch 470/2304 [I 2022-10-27 07:05:36] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:16:58 [I 2022-10-27 07:05:36] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00247 [I 2022-10-27 07:05:36] ...trap/engines/engine.py.230: loss: 0.13049 [I 2022-10-27 07:05:36] ...optimizers/trijoint.py.151: total_norm: 14.318516 [I 2022-10-27 07:05:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 480/2304 [I 2022-10-27 07:05:42] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:51 [I 2022-10-27 07:05:42] ...trap/engines/engine.py.229: process: 0.54801 | load: 0.00244 [I 2022-10-27 07:05:42] ...trap/engines/engine.py.230: loss: 0.16885 [I 2022-10-27 07:05:42] ...optimizers/trijoint.py.151: total_norm: 17.121353 [I 2022-10-27 07:05:47] ...trap/engines/engine.py.224: train: epoch 54 | batch 490/2304 [I 2022-10-27 07:05:47] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:47 [I 2022-10-27 07:05:47] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00258 [I 2022-10-27 07:05:47] ...trap/engines/engine.py.230: loss: 0.12624 [I 2022-10-27 07:05:47] ...optimizers/trijoint.py.151: total_norm: 8.385905 [I 2022-10-27 07:05:53] ...trap/engines/engine.py.224: train: epoch 54 | batch 500/2304 [I 2022-10-27 07:05:53] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:17:11 [I 2022-10-27 07:05:53] ...trap/engines/engine.py.229: process: 0.54209 | load: 0.00255 [I 2022-10-27 07:05:53] ...trap/engines/engine.py.230: loss: 0.12910 [I 2022-10-27 07:05:53] ...optimizers/trijoint.py.151: total_norm: 18.689461 [I 2022-10-27 07:05:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 510/2304 [I 2022-10-27 07:05:59] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:39 [I 2022-10-27 07:05:59] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00244 [I 2022-10-27 07:05:59] ...trap/engines/engine.py.230: loss: 0.13959 [I 2022-10-27 07:05:59] ...optimizers/trijoint.py.151: total_norm: 29.199234 [I 2022-10-27 07:06:04] ...trap/engines/engine.py.224: train: epoch 54 | batch 520/2304 [I 2022-10-27 07:06:04] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:43 [I 2022-10-27 07:06:04] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00246 [I 2022-10-27 07:06:04] ...trap/engines/engine.py.230: loss: 0.15775 [I 2022-10-27 07:06:04] ...optimizers/trijoint.py.151: total_norm: 8.757054 [I 2022-10-27 07:06:10] ...trap/engines/engine.py.224: train: epoch 54 | batch 530/2304 [I 2022-10-27 07:06:10] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:22 [I 2022-10-27 07:06:10] ...trap/engines/engine.py.229: process: 0.54828 | load: 0.00242 [I 2022-10-27 07:06:10] ...trap/engines/engine.py.230: loss: 0.15916 [I 2022-10-27 07:06:10] ...optimizers/trijoint.py.151: total_norm: 27.097506 [I 2022-10-27 07:06:15] ...trap/engines/engine.py.224: train: epoch 54 | batch 540/2304 [I 2022-10-27 07:06:15] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:19 [I 2022-10-27 07:06:15] ...trap/engines/engine.py.229: process: 0.55233 | load: 0.00244 [I 2022-10-27 07:06:15] ...trap/engines/engine.py.230: loss: 0.22585 [I 2022-10-27 07:06:15] ...optimizers/trijoint.py.151: total_norm: 21.133089 [I 2022-10-27 07:06:21] ...trap/engines/engine.py.224: train: epoch 54 | batch 550/2304 [I 2022-10-27 07:06:21] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:13 [I 2022-10-27 07:06:21] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00245 [I 2022-10-27 07:06:21] ...trap/engines/engine.py.230: loss: 0.17984 [I 2022-10-27 07:06:21] ...optimizers/trijoint.py.151: total_norm: 16.407202 [I 2022-10-27 07:06:26] ...trap/engines/engine.py.224: train: epoch 54 | batch 560/2304 [I 2022-10-27 07:06:26] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:03 [I 2022-10-27 07:06:26] ...trap/engines/engine.py.229: process: 0.54892 | load: 0.00246 [I 2022-10-27 07:06:26] ...trap/engines/engine.py.230: loss: 0.17998 [I 2022-10-27 07:06:26] ...optimizers/trijoint.py.151: total_norm: 11.318196 [I 2022-10-27 07:06:32] ...trap/engines/engine.py.224: train: epoch 54 | batch 570/2304 [I 2022-10-27 07:06:32] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:16:00 [I 2022-10-27 07:06:32] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00245 [I 2022-10-27 07:06:32] ...trap/engines/engine.py.230: loss: 0.10902 [I 2022-10-27 07:06:32] ...optimizers/trijoint.py.151: total_norm: 7.910810 [I 2022-10-27 07:06:38] ...trap/engines/engine.py.224: train: epoch 54 | batch 580/2304 [I 2022-10-27 07:06:38] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:22 [I 2022-10-27 07:06:38] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00247 [I 2022-10-27 07:06:38] ...trap/engines/engine.py.230: loss: 0.19749 [I 2022-10-27 07:06:38] ...optimizers/trijoint.py.151: total_norm: 23.066141 [I 2022-10-27 07:06:43] ...trap/engines/engine.py.224: train: epoch 54 | batch 590/2304 [I 2022-10-27 07:06:43] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:54 [I 2022-10-27 07:06:43] ...trap/engines/engine.py.229: process: 0.54885 | load: 0.00244 [I 2022-10-27 07:06:43] ...trap/engines/engine.py.230: loss: 0.19610 [I 2022-10-27 07:06:43] ...optimizers/trijoint.py.151: total_norm: 26.922647 [I 2022-10-27 07:06:49] ...trap/engines/engine.py.224: train: epoch 54 | batch 600/2304 [I 2022-10-27 07:06:49] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:47 [I 2022-10-27 07:06:49] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00256 [I 2022-10-27 07:06:49] ...trap/engines/engine.py.230: loss: 0.14393 [I 2022-10-27 07:06:49] ...optimizers/trijoint.py.151: total_norm: 21.042704 [I 2022-10-27 07:06:54] ...trap/engines/engine.py.224: train: epoch 54 | batch 610/2304 [I 2022-10-27 07:06:54] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:40 [I 2022-10-27 07:06:54] ...trap/engines/engine.py.229: process: 0.54768 | load: 0.00254 [I 2022-10-27 07:06:54] ...trap/engines/engine.py.230: loss: 0.17235 [I 2022-10-27 07:06:54] ...optimizers/trijoint.py.151: total_norm: 13.324110 [I 2022-10-27 07:07:00] ...trap/engines/engine.py.224: train: epoch 54 | batch 620/2304 [I 2022-10-27 07:07:00] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:37 [I 2022-10-27 07:07:00] ...trap/engines/engine.py.229: process: 0.55229 | load: 0.00250 [I 2022-10-27 07:07:00] ...trap/engines/engine.py.230: loss: 0.14473 [I 2022-10-27 07:07:00] ...optimizers/trijoint.py.151: total_norm: 14.256777 [I 2022-10-27 07:07:06] ...trap/engines/engine.py.224: train: epoch 54 | batch 630/2304 [I 2022-10-27 07:07:06] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:24 [I 2022-10-27 07:07:06] ...trap/engines/engine.py.229: process: 0.55097 | load: 0.00240 [I 2022-10-27 07:07:06] ...trap/engines/engine.py.230: loss: 0.17165 [I 2022-10-27 07:07:06] ...optimizers/trijoint.py.151: total_norm: 13.660293 [I 2022-10-27 07:07:11] ...trap/engines/engine.py.224: train: epoch 54 | batch 640/2304 [I 2022-10-27 07:07:11] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:17 [I 2022-10-27 07:07:11] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00254 [I 2022-10-27 07:07:11] ...trap/engines/engine.py.230: loss: 0.18442 [I 2022-10-27 07:07:11] ...optimizers/trijoint.py.151: total_norm: 12.349983 [I 2022-10-27 07:07:17] ...trap/engines/engine.py.224: train: epoch 54 | batch 650/2304 [I 2022-10-27 07:07:17] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:14 [I 2022-10-27 07:07:17] ...trap/engines/engine.py.229: process: 0.56214 | load: 0.00243 [I 2022-10-27 07:07:17] ...trap/engines/engine.py.230: loss: 0.17206 [I 2022-10-27 07:07:17] ...optimizers/trijoint.py.151: total_norm: 14.647139 [I 2022-10-27 07:07:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 660/2304 [I 2022-10-27 07:07:22] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:16 [I 2022-10-27 07:07:22] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00247 [I 2022-10-27 07:07:22] ...trap/engines/engine.py.230: loss: 0.16860 [I 2022-10-27 07:07:22] ...optimizers/trijoint.py.151: total_norm: 11.857418 [I 2022-10-27 07:07:28] ...trap/engines/engine.py.224: train: epoch 54 | batch 670/2304 [I 2022-10-27 07:07:28] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:04 [I 2022-10-27 07:07:28] ...trap/engines/engine.py.229: process: 0.54620 | load: 0.00241 [I 2022-10-27 07:07:28] ...trap/engines/engine.py.230: loss: 0.21175 [I 2022-10-27 07:07:28] ...optimizers/trijoint.py.151: total_norm: 99.169754 [I 2022-10-27 07:07:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 680/2304 [I 2022-10-27 07:07:33] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:03 [I 2022-10-27 07:07:33] ...trap/engines/engine.py.229: process: 0.55873 | load: 0.00244 [I 2022-10-27 07:07:33] ...trap/engines/engine.py.230: loss: 0.14198 [I 2022-10-27 07:07:33] ...optimizers/trijoint.py.151: total_norm: 16.582481 [I 2022-10-27 07:07:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 690/2304 [I 2022-10-27 07:07:39] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:15:07 [I 2022-10-27 07:07:39] ...trap/engines/engine.py.229: process: 0.57154 | load: 0.00254 [I 2022-10-27 07:07:39] ...trap/engines/engine.py.230: loss: 0.18991 [I 2022-10-27 07:07:39] ...optimizers/trijoint.py.151: total_norm: 23.806574 [I 2022-10-27 07:07:45] ...trap/engines/engine.py.224: train: epoch 54 | batch 700/2304 [I 2022-10-27 07:07:45] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:51 [I 2022-10-27 07:07:45] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00255 [I 2022-10-27 07:07:45] ...trap/engines/engine.py.230: loss: 0.15234 [I 2022-10-27 07:07:45] ...optimizers/trijoint.py.151: total_norm: 39.950588 [I 2022-10-27 07:07:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 710/2304 [I 2022-10-27 07:07:50] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:14:47 [I 2022-10-27 07:07:50] ...trap/engines/engine.py.229: process: 0.56227 | load: 0.00242 [I 2022-10-27 07:07:50] ...trap/engines/engine.py.230: loss: 0.11345 [I 2022-10-27 07:07:50] ...optimizers/trijoint.py.151: total_norm: 11.529327 [I 2022-10-27 07:07:56] ...trap/engines/engine.py.224: train: epoch 54 | batch 720/2304 [I 2022-10-27 07:07:56] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:44 [I 2022-10-27 07:07:56] ...trap/engines/engine.py.229: process: 0.55684 | load: 0.00249 [I 2022-10-27 07:07:56] ...trap/engines/engine.py.230: loss: 0.15124 [I 2022-10-27 07:07:56] ...optimizers/trijoint.py.151: total_norm: 20.959505 [I 2022-10-27 07:08:01] ...trap/engines/engine.py.224: train: epoch 54 | batch 730/2304 [I 2022-10-27 07:08:01] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:35 [I 2022-10-27 07:08:01] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00248 [I 2022-10-27 07:08:01] ...trap/engines/engine.py.230: loss: 0.18669 [I 2022-10-27 07:08:01] ...optimizers/trijoint.py.151: total_norm: 19.999592 [I 2022-10-27 07:08:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 740/2304 [I 2022-10-27 07:08:07] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:25 [I 2022-10-27 07:08:07] ...trap/engines/engine.py.229: process: 0.54890 | load: 0.00261 [I 2022-10-27 07:08:07] ...trap/engines/engine.py.230: loss: 0.18658 [I 2022-10-27 07:08:07] ...optimizers/trijoint.py.151: total_norm: 16.368467 [I 2022-10-27 07:08:13] ...trap/engines/engine.py.224: train: epoch 54 | batch 750/2304 [I 2022-10-27 07:08:13] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:24 [I 2022-10-27 07:08:13] ...trap/engines/engine.py.229: process: 0.56268 | load: 0.00249 [I 2022-10-27 07:08:13] ...trap/engines/engine.py.230: loss: 0.14083 [I 2022-10-27 07:08:13] ...optimizers/trijoint.py.151: total_norm: 22.658049 [I 2022-10-27 07:08:18] ...trap/engines/engine.py.224: train: epoch 54 | batch 760/2304 [I 2022-10-27 07:08:18] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:22 [I 2022-10-27 07:08:18] ...trap/engines/engine.py.229: process: 0.56691 | load: 0.00244 [I 2022-10-27 07:08:18] ...trap/engines/engine.py.230: loss: 0.10210 [I 2022-10-27 07:08:18] ...optimizers/trijoint.py.151: total_norm: 9.333334 [I 2022-10-27 07:08:24] ...trap/engines/engine.py.224: train: epoch 54 | batch 770/2304 [I 2022-10-27 07:08:24] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:25 [I 2022-10-27 07:08:24] ...trap/engines/engine.py.229: process: 0.56366 | load: 0.00253 [I 2022-10-27 07:08:24] ...trap/engines/engine.py.230: loss: 0.11976 [I 2022-10-27 07:08:24] ...optimizers/trijoint.py.151: total_norm: 8.678212 [I 2022-10-27 07:08:29] ...trap/engines/engine.py.224: train: epoch 54 | batch 780/2304 [I 2022-10-27 07:08:29] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:06 [I 2022-10-27 07:08:29] ...trap/engines/engine.py.229: process: 0.54217 | load: 0.00252 [I 2022-10-27 07:08:29] ...trap/engines/engine.py.230: loss: 0.16530 [I 2022-10-27 07:08:29] ...optimizers/trijoint.py.151: total_norm: 10.014403 [I 2022-10-27 07:08:35] ...trap/engines/engine.py.224: train: epoch 54 | batch 790/2304 [I 2022-10-27 07:08:35] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:13:58 [I 2022-10-27 07:08:35] ...trap/engines/engine.py.229: process: 0.55155 | load: 0.00246 [I 2022-10-27 07:08:35] ...trap/engines/engine.py.230: loss: 0.15592 [I 2022-10-27 07:08:35] ...optimizers/trijoint.py.151: total_norm: 20.822172 [I 2022-10-27 07:08:41] ...trap/engines/engine.py.224: train: epoch 54 | batch 800/2304 [I 2022-10-27 07:08:41] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:14:01 [I 2022-10-27 07:08:41] ...trap/engines/engine.py.229: process: 0.56642 | load: 0.00245 [I 2022-10-27 07:08:41] ...trap/engines/engine.py.230: loss: 0.13491 [I 2022-10-27 07:08:41] ...optimizers/trijoint.py.151: total_norm: 16.308363 [I 2022-10-27 07:08:46] ...trap/engines/engine.py.224: train: epoch 54 | batch 810/2304 [I 2022-10-27 07:08:46] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:13:57 [I 2022-10-27 07:08:46] ...trap/engines/engine.py.229: process: 0.56436 | load: 0.00247 [I 2022-10-27 07:08:46] ...trap/engines/engine.py.230: loss: 0.14381 [I 2022-10-27 07:08:46] ...optimizers/trijoint.py.151: total_norm: 11.361503 [I 2022-10-27 07:08:52] ...trap/engines/engine.py.224: train: epoch 54 | batch 820/2304 [I 2022-10-27 07:08:52] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:42 [I 2022-10-27 07:08:52] ...trap/engines/engine.py.229: process: 0.55576 | load: 0.00246 [I 2022-10-27 07:08:52] ...trap/engines/engine.py.230: loss: 0.16711 [I 2022-10-27 07:08:52] ...optimizers/trijoint.py.151: total_norm: 25.977516 [I 2022-10-27 07:08:57] ...trap/engines/engine.py.224: train: epoch 54 | batch 830/2304 [I 2022-10-27 07:08:57] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:43 [I 2022-10-27 07:08:57] ...trap/engines/engine.py.229: process: 0.56620 | load: 0.00245 [I 2022-10-27 07:08:57] ...trap/engines/engine.py.230: loss: 0.12814 [I 2022-10-27 07:08:57] ...optimizers/trijoint.py.151: total_norm: 11.284010 [I 2022-10-27 07:09:03] ...trap/engines/engine.py.224: train: epoch 54 | batch 840/2304 [I 2022-10-27 07:09:03] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:41 [I 2022-10-27 07:09:03] ...trap/engines/engine.py.229: process: 0.55221 | load: 0.00257 [I 2022-10-27 07:09:03] ...trap/engines/engine.py.230: loss: 0.16959 [I 2022-10-27 07:09:03] ...optimizers/trijoint.py.151: total_norm: 30.552422 [I 2022-10-27 07:09:09] ...trap/engines/engine.py.224: train: epoch 54 | batch 850/2304 [I 2022-10-27 07:09:09] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:31 [I 2022-10-27 07:09:09] ...trap/engines/engine.py.229: process: 0.56377 | load: 0.00245 [I 2022-10-27 07:09:09] ...trap/engines/engine.py.230: loss: 0.13402 [I 2022-10-27 07:09:09] ...optimizers/trijoint.py.151: total_norm: 17.191683 [I 2022-10-27 07:09:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 860/2304 [I 2022-10-27 07:09:14] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:22 [I 2022-10-27 07:09:14] ...trap/engines/engine.py.229: process: 0.55346 | load: 0.00247 [I 2022-10-27 07:09:14] ...trap/engines/engine.py.230: loss: 0.10545 [I 2022-10-27 07:09:14] ...optimizers/trijoint.py.151: total_norm: 8.719842 [I 2022-10-27 07:09:20] ...trap/engines/engine.py.224: train: epoch 54 | batch 870/2304 [I 2022-10-27 07:09:20] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:15 [I 2022-10-27 07:09:20] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00243 [I 2022-10-27 07:09:20] ...trap/engines/engine.py.230: loss: 0.16738 [I 2022-10-27 07:09:20] ...optimizers/trijoint.py.151: total_norm: 24.982533 [I 2022-10-27 07:09:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 880/2304 [I 2022-10-27 07:09:25] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:16 [I 2022-10-27 07:09:25] ...trap/engines/engine.py.229: process: 0.55979 | load: 0.00243 [I 2022-10-27 07:09:25] ...trap/engines/engine.py.230: loss: 0.21210 [I 2022-10-27 07:09:25] ...optimizers/trijoint.py.151: total_norm: 23.019207 [I 2022-10-27 07:09:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 890/2304 [I 2022-10-27 07:09:31] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:21 [I 2022-10-27 07:09:31] ...trap/engines/engine.py.229: process: 0.54714 | load: 0.00242 [I 2022-10-27 07:09:31] ...trap/engines/engine.py.230: loss: 0.17227 [I 2022-10-27 07:09:31] ...optimizers/trijoint.py.151: total_norm: 13.827675 [I 2022-10-27 07:09:37] ...trap/engines/engine.py.224: train: epoch 54 | batch 900/2304 [I 2022-10-27 07:09:37] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:08 [I 2022-10-27 07:09:37] ...trap/engines/engine.py.229: process: 0.56604 | load: 0.00247 [I 2022-10-27 07:09:37] ...trap/engines/engine.py.230: loss: 0.23403 [I 2022-10-27 07:09:37] ...optimizers/trijoint.py.151: total_norm: 24.044399 [I 2022-10-27 07:09:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 910/2304 [I 2022-10-27 07:09:42] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:57 [I 2022-10-27 07:09:42] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00247 [I 2022-10-27 07:09:42] ...trap/engines/engine.py.230: loss: 0.22505 [I 2022-10-27 07:09:42] ...optimizers/trijoint.py.151: total_norm: 10.458892 [I 2022-10-27 07:09:48] ...trap/engines/engine.py.224: train: epoch 54 | batch 920/2304 [I 2022-10-27 07:09:48] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:53 [I 2022-10-27 07:09:48] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00247 [I 2022-10-27 07:09:48] ...trap/engines/engine.py.230: loss: 0.10557 [I 2022-10-27 07:09:48] ...optimizers/trijoint.py.151: total_norm: 8.418691 [I 2022-10-27 07:09:53] ...trap/engines/engine.py.224: train: epoch 54 | batch 930/2304 [I 2022-10-27 07:09:53] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:45 [I 2022-10-27 07:09:53] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00242 [I 2022-10-27 07:09:53] ...trap/engines/engine.py.230: loss: 0.22957 [I 2022-10-27 07:09:53] ...optimizers/trijoint.py.151: total_norm: 20.004765 [I 2022-10-27 07:09:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 940/2304 [I 2022-10-27 07:09:59] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:42 [I 2022-10-27 07:09:59] ...trap/engines/engine.py.229: process: 0.55657 | load: 0.00243 [I 2022-10-27 07:09:59] ...trap/engines/engine.py.230: loss: 0.12449 [I 2022-10-27 07:09:59] ...optimizers/trijoint.py.151: total_norm: 20.170053 [I 2022-10-27 07:10:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 950/2304 [I 2022-10-27 07:10:05] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:32 [I 2022-10-27 07:10:05] ...trap/engines/engine.py.229: process: 0.56278 | load: 0.00249 [I 2022-10-27 07:10:05] ...trap/engines/engine.py.230: loss: 0.13416 [I 2022-10-27 07:10:05] ...optimizers/trijoint.py.151: total_norm: 8.476651 [I 2022-10-27 07:10:10] ...trap/engines/engine.py.224: train: epoch 54 | batch 960/2304 [I 2022-10-27 07:10:10] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:21 [I 2022-10-27 07:10:10] ...trap/engines/engine.py.229: process: 0.54849 | load: 0.00242 [I 2022-10-27 07:10:10] ...trap/engines/engine.py.230: loss: 0.21943 [I 2022-10-27 07:10:10] ...optimizers/trijoint.py.151: total_norm: 22.850595 [I 2022-10-27 07:10:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 970/2304 [I 2022-10-27 07:10:16] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:14 [I 2022-10-27 07:10:16] ...trap/engines/engine.py.229: process: 0.54285 | load: 0.00244 [I 2022-10-27 07:10:16] ...trap/engines/engine.py.230: loss: 0.13909 [I 2022-10-27 07:10:16] ...optimizers/trijoint.py.151: total_norm: 15.796610 [I 2022-10-27 07:10:21] ...trap/engines/engine.py.224: train: epoch 54 | batch 980/2304 [I 2022-10-27 07:10:21] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:16 [I 2022-10-27 07:10:21] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00243 [I 2022-10-27 07:10:21] ...trap/engines/engine.py.230: loss: 0.12878 [I 2022-10-27 07:10:21] ...optimizers/trijoint.py.151: total_norm: 17.172775 [I 2022-10-27 07:10:27] ...trap/engines/engine.py.224: train: epoch 54 | batch 990/2304 [I 2022-10-27 07:10:27] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:08 [I 2022-10-27 07:10:27] ...trap/engines/engine.py.229: process: 0.56210 | load: 0.00244 [I 2022-10-27 07:10:27] ...trap/engines/engine.py.230: loss: 0.13308 [I 2022-10-27 07:10:27] ...optimizers/trijoint.py.151: total_norm: 17.043264 [I 2022-10-27 07:10:32] ...trap/engines/engine.py.224: train: epoch 54 | batch 1000/2304 [I 2022-10-27 07:10:32] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:04 [I 2022-10-27 07:10:32] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00241 [I 2022-10-27 07:10:32] ...trap/engines/engine.py.230: loss: 0.18108 [I 2022-10-27 07:10:32] ...optimizers/trijoint.py.151: total_norm: 13.976915 [I 2022-10-27 07:10:38] ...trap/engines/engine.py.224: train: epoch 54 | batch 1010/2304 [I 2022-10-27 07:10:38] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:10 [I 2022-10-27 07:10:38] ...trap/engines/engine.py.229: process: 0.56179 | load: 0.00243 [I 2022-10-27 07:10:38] ...trap/engines/engine.py.230: loss: 0.16814 [I 2022-10-27 07:10:38] ...optimizers/trijoint.py.151: total_norm: 32.251480 [I 2022-10-27 07:10:43] ...trap/engines/engine.py.224: train: epoch 54 | batch 1020/2304 [I 2022-10-27 07:10:43] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:52 [I 2022-10-27 07:10:43] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00245 [I 2022-10-27 07:10:43] ...trap/engines/engine.py.230: loss: 0.12094 [I 2022-10-27 07:10:43] ...optimizers/trijoint.py.151: total_norm: 9.096377 [I 2022-10-27 07:10:49] ...trap/engines/engine.py.224: train: epoch 54 | batch 1030/2304 [I 2022-10-27 07:10:49] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:51 [I 2022-10-27 07:10:49] ...trap/engines/engine.py.229: process: 0.56294 | load: 0.00248 [I 2022-10-27 07:10:49] ...trap/engines/engine.py.230: loss: 0.13559 [I 2022-10-27 07:10:49] ...optimizers/trijoint.py.151: total_norm: 10.459451 [I 2022-10-27 07:10:55] ...trap/engines/engine.py.224: train: epoch 54 | batch 1040/2304 [I 2022-10-27 07:10:55] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:40 [I 2022-10-27 07:10:55] ...trap/engines/engine.py.229: process: 0.54793 | load: 0.00241 [I 2022-10-27 07:10:55] ...trap/engines/engine.py.230: loss: 0.25759 [I 2022-10-27 07:10:55] ...optimizers/trijoint.py.151: total_norm: 31.444618 [I 2022-10-27 07:11:00] ...trap/engines/engine.py.224: train: epoch 54 | batch 1050/2304 [I 2022-10-27 07:11:00] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:33 [I 2022-10-27 07:11:00] ...trap/engines/engine.py.229: process: 0.54513 | load: 0.00246 [I 2022-10-27 07:11:00] ...trap/engines/engine.py.230: loss: 0.23788 [I 2022-10-27 07:11:00] ...optimizers/trijoint.py.151: total_norm: 21.069389 [I 2022-10-27 07:11:06] ...trap/engines/engine.py.224: train: epoch 54 | batch 1060/2304 [I 2022-10-27 07:11:06] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:29 [I 2022-10-27 07:11:06] ...trap/engines/engine.py.229: process: 0.55135 | load: 0.00248 [I 2022-10-27 07:11:06] ...trap/engines/engine.py.230: loss: 0.14227 [I 2022-10-27 07:11:06] ...optimizers/trijoint.py.151: total_norm: 18.134077 [I 2022-10-27 07:11:11] ...trap/engines/engine.py.224: train: epoch 54 | batch 1070/2304 [I 2022-10-27 07:11:11] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:24 [I 2022-10-27 07:11:11] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00245 [I 2022-10-27 07:11:11] ...trap/engines/engine.py.230: loss: 0.18147 [I 2022-10-27 07:11:11] ...optimizers/trijoint.py.151: total_norm: 27.329580 [I 2022-10-27 07:11:17] ...trap/engines/engine.py.224: train: epoch 54 | batch 1080/2304 [I 2022-10-27 07:11:17] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:19 [I 2022-10-27 07:11:17] ...trap/engines/engine.py.229: process: 0.56342 | load: 0.00240 [I 2022-10-27 07:11:17] ...trap/engines/engine.py.230: loss: 0.16597 [I 2022-10-27 07:11:17] ...optimizers/trijoint.py.151: total_norm: 10.145436 [I 2022-10-27 07:11:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 1090/2304 [I 2022-10-27 07:11:22] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:14 [I 2022-10-27 07:11:22] ...trap/engines/engine.py.229: process: 0.56354 | load: 0.00243 [I 2022-10-27 07:11:22] ...trap/engines/engine.py.230: loss: 0.12037 [I 2022-10-27 07:11:22] ...optimizers/trijoint.py.151: total_norm: 8.812201 [I 2022-10-27 07:11:28] ...trap/engines/engine.py.224: train: epoch 54 | batch 1100/2304 [I 2022-10-27 07:11:28] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:10 [I 2022-10-27 07:11:28] ...trap/engines/engine.py.229: process: 0.55321 | load: 0.00243 [I 2022-10-27 07:11:28] ...trap/engines/engine.py.230: loss: 0.18160 [I 2022-10-27 07:11:28] ...optimizers/trijoint.py.151: total_norm: 29.485640 [I 2022-10-27 07:11:34] ...trap/engines/engine.py.224: train: epoch 54 | batch 1110/2304 [I 2022-10-27 07:11:34] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:05 [I 2022-10-27 07:11:34] ...trap/engines/engine.py.229: process: 0.54792 | load: 0.00244 [I 2022-10-27 07:11:34] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-27 07:11:34] ...optimizers/trijoint.py.151: total_norm: 23.697910 [I 2022-10-27 07:11:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 1120/2304 [I 2022-10-27 07:11:39] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:10:58 [I 2022-10-27 07:11:39] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00242 [I 2022-10-27 07:11:39] ...trap/engines/engine.py.230: loss: 0.20569 [I 2022-10-27 07:11:39] ...optimizers/trijoint.py.151: total_norm: 23.890232 [I 2022-10-27 07:11:45] ...trap/engines/engine.py.224: train: epoch 54 | batch 1130/2304 [I 2022-10-27 07:11:45] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:54 [I 2022-10-27 07:11:45] ...trap/engines/engine.py.229: process: 0.56278 | load: 0.00239 [I 2022-10-27 07:11:45] ...trap/engines/engine.py.230: loss: 0.12355 [I 2022-10-27 07:11:45] ...optimizers/trijoint.py.151: total_norm: 12.720675 [I 2022-10-27 07:11:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 1140/2304 [I 2022-10-27 07:11:50] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:41 [I 2022-10-27 07:11:50] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00239 [I 2022-10-27 07:11:50] ...trap/engines/engine.py.230: loss: 0.19496 [I 2022-10-27 07:11:50] ...optimizers/trijoint.py.151: total_norm: 26.932159 [I 2022-10-27 07:11:56] ...trap/engines/engine.py.224: train: epoch 54 | batch 1150/2304 [I 2022-10-27 07:11:56] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:10:41 [I 2022-10-27 07:11:56] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00241 [I 2022-10-27 07:11:56] ...trap/engines/engine.py.230: loss: 0.12243 [I 2022-10-27 07:11:56] ...optimizers/trijoint.py.151: total_norm: 13.087779 [I 2022-10-27 07:12:01] ...trap/engines/engine.py.224: train: epoch 54 | batch 1160/2304 [I 2022-10-27 07:12:01] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:33 [I 2022-10-27 07:12:01] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00253 [I 2022-10-27 07:12:01] ...trap/engines/engine.py.230: loss: 0.17629 [I 2022-10-27 07:12:01] ...optimizers/trijoint.py.151: total_norm: 20.295290 [I 2022-10-27 07:12:07] ...trap/engines/engine.py.224: train: epoch 54 | batch 1170/2304 [I 2022-10-27 07:12:07] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:33 [I 2022-10-27 07:12:07] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00250 [I 2022-10-27 07:12:07] ...trap/engines/engine.py.230: loss: 0.17680 [I 2022-10-27 07:12:07] ...optimizers/trijoint.py.151: total_norm: 11.443965 [I 2022-10-27 07:12:13] ...trap/engines/engine.py.224: train: epoch 54 | batch 1180/2304 [I 2022-10-27 07:12:13] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:26 [I 2022-10-27 07:12:13] ...trap/engines/engine.py.229: process: 0.54167 | load: 0.00243 [I 2022-10-27 07:12:13] ...trap/engines/engine.py.230: loss: 0.15524 [I 2022-10-27 07:12:13] ...optimizers/trijoint.py.151: total_norm: 16.170307 [I 2022-10-27 07:12:18] ...trap/engines/engine.py.224: train: epoch 54 | batch 1190/2304 [I 2022-10-27 07:12:18] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:17 [I 2022-10-27 07:12:18] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00243 [I 2022-10-27 07:12:18] ...trap/engines/engine.py.230: loss: 0.17923 [I 2022-10-27 07:12:18] ...optimizers/trijoint.py.151: total_norm: 27.524090 [I 2022-10-27 07:12:24] ...trap/engines/engine.py.224: train: epoch 54 | batch 1200/2304 [I 2022-10-27 07:12:24] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:15 [I 2022-10-27 07:12:24] ...trap/engines/engine.py.229: process: 0.56552 | load: 0.00244 [I 2022-10-27 07:12:24] ...trap/engines/engine.py.230: loss: 0.15934 [I 2022-10-27 07:12:24] ...optimizers/trijoint.py.151: total_norm: 24.520857 [I 2022-10-27 07:12:29] ...trap/engines/engine.py.224: train: epoch 54 | batch 1210/2304 [I 2022-10-27 07:12:29] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:00 [I 2022-10-27 07:12:29] ...trap/engines/engine.py.229: process: 0.54590 | load: 0.00246 [I 2022-10-27 07:12:29] ...trap/engines/engine.py.230: loss: 0.16551 [I 2022-10-27 07:12:29] ...optimizers/trijoint.py.151: total_norm: 21.015354 [I 2022-10-27 07:12:35] ...trap/engines/engine.py.224: train: epoch 54 | batch 1220/2304 [I 2022-10-27 07:12:35] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:00 [I 2022-10-27 07:12:35] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00244 [I 2022-10-27 07:12:35] ...trap/engines/engine.py.230: loss: 0.14688 [I 2022-10-27 07:12:35] ...optimizers/trijoint.py.151: total_norm: 25.916275 [I 2022-10-27 07:12:40] ...trap/engines/engine.py.224: train: epoch 54 | batch 1230/2304 [I 2022-10-27 07:12:40] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:09:58 [I 2022-10-27 07:12:40] ...trap/engines/engine.py.229: process: 0.55658 | load: 0.00244 [I 2022-10-27 07:12:40] ...trap/engines/engine.py.230: loss: 0.20184 [I 2022-10-27 07:12:40] ...optimizers/trijoint.py.151: total_norm: 10.867509 [I 2022-10-27 07:12:46] ...trap/engines/engine.py.224: train: epoch 54 | batch 1240/2304 [I 2022-10-27 07:12:46] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:09:49 [I 2022-10-27 07:12:46] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00244 [I 2022-10-27 07:12:46] ...trap/engines/engine.py.230: loss: 0.18260 [I 2022-10-27 07:12:46] ...optimizers/trijoint.py.151: total_norm: 8.762187 [I 2022-10-27 07:12:52] ...trap/engines/engine.py.224: train: epoch 54 | batch 1250/2304 [I 2022-10-27 07:12:52] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:09:46 [I 2022-10-27 07:12:52] ...trap/engines/engine.py.229: process: 0.56871 | load: 0.00246 [I 2022-10-27 07:12:52] ...trap/engines/engine.py.230: loss: 0.16114 [I 2022-10-27 07:12:52] ...optimizers/trijoint.py.151: total_norm: 11.976489 [I 2022-10-27 07:12:57] ...trap/engines/engine.py.224: train: epoch 54 | batch 1260/2304 [I 2022-10-27 07:12:57] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:09:37 [I 2022-10-27 07:12:57] ...trap/engines/engine.py.229: process: 0.54742 | load: 0.00242 [I 2022-10-27 07:12:57] ...trap/engines/engine.py.230: loss: 0.08544 [I 2022-10-27 07:12:57] ...optimizers/trijoint.py.151: total_norm: 7.136682 [I 2022-10-27 07:13:03] ...trap/engines/engine.py.224: train: epoch 54 | batch 1270/2304 [I 2022-10-27 07:13:03] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:30 [I 2022-10-27 07:13:03] ...trap/engines/engine.py.229: process: 0.54909 | load: 0.00252 [I 2022-10-27 07:13:03] ...trap/engines/engine.py.230: loss: 0.20725 [I 2022-10-27 07:13:03] ...optimizers/trijoint.py.151: total_norm: 20.159107 [I 2022-10-27 07:13:08] ...trap/engines/engine.py.224: train: epoch 54 | batch 1280/2304 [I 2022-10-27 07:13:08] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:28 [I 2022-10-27 07:13:08] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00245 [I 2022-10-27 07:13:08] ...trap/engines/engine.py.230: loss: 0.19000 [I 2022-10-27 07:13:08] ...optimizers/trijoint.py.151: total_norm: 17.144157 [I 2022-10-27 07:13:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 1290/2304 [I 2022-10-27 07:13:14] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:27 [I 2022-10-27 07:13:14] ...trap/engines/engine.py.229: process: 0.56568 | load: 0.00246 [I 2022-10-27 07:13:14] ...trap/engines/engine.py.230: loss: 0.22269 [I 2022-10-27 07:13:14] ...optimizers/trijoint.py.151: total_norm: 21.247526 [I 2022-10-27 07:13:19] ...trap/engines/engine.py.224: train: epoch 54 | batch 1300/2304 [I 2022-10-27 07:13:19] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:22 [I 2022-10-27 07:13:19] ...trap/engines/engine.py.229: process: 0.56781 | load: 0.00241 [I 2022-10-27 07:13:19] ...trap/engines/engine.py.230: loss: 0.13170 [I 2022-10-27 07:13:19] ...optimizers/trijoint.py.151: total_norm: 13.944867 [I 2022-10-27 07:13:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 1310/2304 [I 2022-10-27 07:13:25] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:14 [I 2022-10-27 07:13:25] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00250 [I 2022-10-27 07:13:25] ...trap/engines/engine.py.230: loss: 0.14827 [I 2022-10-27 07:13:25] ...optimizers/trijoint.py.151: total_norm: 16.687490 [I 2022-10-27 07:13:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 1320/2304 [I 2022-10-27 07:13:31] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:06 [I 2022-10-27 07:13:31] ...trap/engines/engine.py.229: process: 0.55246 | load: 0.00246 [I 2022-10-27 07:13:31] ...trap/engines/engine.py.230: loss: 0.11466 [I 2022-10-27 07:13:31] ...optimizers/trijoint.py.151: total_norm: 12.360754 [I 2022-10-27 07:13:36] ...trap/engines/engine.py.224: train: epoch 54 | batch 1330/2304 [I 2022-10-27 07:13:36] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:01 [I 2022-10-27 07:13:36] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00262 [I 2022-10-27 07:13:36] ...trap/engines/engine.py.230: loss: 0.20970 [I 2022-10-27 07:13:36] ...optimizers/trijoint.py.151: total_norm: 20.488443 [I 2022-10-27 07:13:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 1340/2304 [I 2022-10-27 07:13:42] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:09:00 [I 2022-10-27 07:13:42] ...trap/engines/engine.py.229: process: 0.57030 | load: 0.00245 [I 2022-10-27 07:13:42] ...trap/engines/engine.py.230: loss: 0.16772 [I 2022-10-27 07:13:42] ...optimizers/trijoint.py.151: total_norm: 25.151257 [I 2022-10-27 07:13:47] ...trap/engines/engine.py.224: train: epoch 54 | batch 1350/2304 [I 2022-10-27 07:13:47] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:48 [I 2022-10-27 07:13:47] ...trap/engines/engine.py.229: process: 0.54672 | load: 0.00241 [I 2022-10-27 07:13:47] ...trap/engines/engine.py.230: loss: 0.19075 [I 2022-10-27 07:13:47] ...optimizers/trijoint.py.151: total_norm: 21.165359 [I 2022-10-27 07:13:53] ...trap/engines/engine.py.224: train: epoch 54 | batch 1360/2304 [I 2022-10-27 07:13:53] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:08:47 [I 2022-10-27 07:13:53] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00247 [I 2022-10-27 07:13:53] ...trap/engines/engine.py.230: loss: 0.18389 [I 2022-10-27 07:13:53] ...optimizers/trijoint.py.151: total_norm: 14.812280 [I 2022-10-27 07:13:58] ...trap/engines/engine.py.224: train: epoch 54 | batch 1370/2304 [I 2022-10-27 07:13:58] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:38 [I 2022-10-27 07:13:58] ...trap/engines/engine.py.229: process: 0.55115 | load: 0.00242 [I 2022-10-27 07:13:58] ...trap/engines/engine.py.230: loss: 0.21962 [I 2022-10-27 07:13:58] ...optimizers/trijoint.py.151: total_norm: 14.029869 [I 2022-10-27 07:14:04] ...trap/engines/engine.py.224: train: epoch 54 | batch 1380/2304 [I 2022-10-27 07:14:04] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:28 [I 2022-10-27 07:14:04] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00247 [I 2022-10-27 07:14:04] ...trap/engines/engine.py.230: loss: 0.15939 [I 2022-10-27 07:14:04] ...optimizers/trijoint.py.151: total_norm: 10.394245 [I 2022-10-27 07:14:10] ...trap/engines/engine.py.224: train: epoch 54 | batch 1390/2304 [I 2022-10-27 07:14:10] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:28 [I 2022-10-27 07:14:10] ...trap/engines/engine.py.229: process: 0.55079 | load: 0.00239 [I 2022-10-27 07:14:10] ...trap/engines/engine.py.230: loss: 0.12475 [I 2022-10-27 07:14:10] ...optimizers/trijoint.py.151: total_norm: 12.436610 [I 2022-10-27 07:14:15] ...trap/engines/engine.py.224: train: epoch 54 | batch 1400/2304 [I 2022-10-27 07:14:15] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:24 [I 2022-10-27 07:14:15] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00240 [I 2022-10-27 07:14:15] ...trap/engines/engine.py.230: loss: 0.19050 [I 2022-10-27 07:14:15] ...optimizers/trijoint.py.151: total_norm: 25.426092 [I 2022-10-27 07:14:21] ...trap/engines/engine.py.224: train: epoch 54 | batch 1410/2304 [I 2022-10-27 07:14:21] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:16 [I 2022-10-27 07:14:21] ...trap/engines/engine.py.229: process: 0.54846 | load: 0.00249 [I 2022-10-27 07:14:21] ...trap/engines/engine.py.230: loss: 0.09979 [I 2022-10-27 07:14:21] ...optimizers/trijoint.py.151: total_norm: 7.854025 [I 2022-10-27 07:14:26] ...trap/engines/engine.py.224: train: epoch 54 | batch 1420/2304 [I 2022-10-27 07:14:26] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:13 [I 2022-10-27 07:14:26] ...trap/engines/engine.py.229: process: 0.55433 | load: 0.00254 [I 2022-10-27 07:14:26] ...trap/engines/engine.py.230: loss: 0.12366 [I 2022-10-27 07:14:26] ...optimizers/trijoint.py.151: total_norm: 13.800331 [I 2022-10-27 07:14:32] ...trap/engines/engine.py.224: train: epoch 54 | batch 1430/2304 [I 2022-10-27 07:14:32] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:07 [I 2022-10-27 07:14:32] ...trap/engines/engine.py.229: process: 0.56160 | load: 0.00248 [I 2022-10-27 07:14:32] ...trap/engines/engine.py.230: loss: 0.22826 [I 2022-10-27 07:14:32] ...optimizers/trijoint.py.151: total_norm: 30.528971 [I 2022-10-27 07:14:37] ...trap/engines/engine.py.224: train: epoch 54 | batch 1440/2304 [I 2022-10-27 07:14:37] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:07:58 [I 2022-10-27 07:14:37] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00246 [I 2022-10-27 07:14:38] ...trap/engines/engine.py.230: loss: 0.15261 [I 2022-10-27 07:14:38] ...optimizers/trijoint.py.151: total_norm: 18.610420 [I 2022-10-27 07:14:43] ...trap/engines/engine.py.224: train: epoch 54 | batch 1450/2304 [I 2022-10-27 07:14:43] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:07:48 [I 2022-10-27 07:14:43] ...trap/engines/engine.py.229: process: 0.54637 | load: 0.00243 [I 2022-10-27 07:14:43] ...trap/engines/engine.py.230: loss: 0.19831 [I 2022-10-27 07:14:43] ...optimizers/trijoint.py.151: total_norm: 32.494625 [I 2022-10-27 07:14:49] ...trap/engines/engine.py.224: train: epoch 54 | batch 1460/2304 [I 2022-10-27 07:14:49] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:07:50 [I 2022-10-27 07:14:49] ...trap/engines/engine.py.229: process: 0.55614 | load: 0.00242 [I 2022-10-27 07:14:49] ...trap/engines/engine.py.230: loss: 0.15845 [I 2022-10-27 07:14:49] ...optimizers/trijoint.py.151: total_norm: 10.831134 [I 2022-10-27 07:14:54] ...trap/engines/engine.py.224: train: epoch 54 | batch 1470/2304 [I 2022-10-27 07:14:54] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:07:45 [I 2022-10-27 07:14:54] ...trap/engines/engine.py.229: process: 0.56680 | load: 0.00245 [I 2022-10-27 07:14:54] ...trap/engines/engine.py.230: loss: 0.17487 [I 2022-10-27 07:14:54] ...optimizers/trijoint.py.151: total_norm: 9.803880 [I 2022-10-27 07:15:00] ...trap/engines/engine.py.224: train: epoch 54 | batch 1480/2304 [I 2022-10-27 07:15:00] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:43 [I 2022-10-27 07:15:00] ...trap/engines/engine.py.229: process: 0.57339 | load: 0.00245 [I 2022-10-27 07:15:00] ...trap/engines/engine.py.230: loss: 0.10447 [I 2022-10-27 07:15:00] ...optimizers/trijoint.py.151: total_norm: 9.492514 [I 2022-10-27 07:15:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 1490/2304 [I 2022-10-27 07:15:05] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:34 [I 2022-10-27 07:15:05] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00248 [I 2022-10-27 07:15:05] ...trap/engines/engine.py.230: loss: 0.18636 [I 2022-10-27 07:15:05] ...optimizers/trijoint.py.151: total_norm: 34.513348 [I 2022-10-27 07:15:11] ...trap/engines/engine.py.224: train: epoch 54 | batch 1500/2304 [I 2022-10-27 07:15:11] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:35 [I 2022-10-27 07:15:11] ...trap/engines/engine.py.229: process: 0.57591 | load: 0.00281 [I 2022-10-27 07:15:11] ...trap/engines/engine.py.230: loss: 0.18957 [I 2022-10-27 07:15:11] ...optimizers/trijoint.py.151: total_norm: 19.186646 [I 2022-10-27 07:15:17] ...trap/engines/engine.py.224: train: epoch 54 | batch 1510/2304 [I 2022-10-27 07:15:17] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:29 [I 2022-10-27 07:15:17] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00247 [I 2022-10-27 07:15:17] ...trap/engines/engine.py.230: loss: 0.20599 [I 2022-10-27 07:15:17] ...optimizers/trijoint.py.151: total_norm: 11.931826 [I 2022-10-27 07:15:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 1520/2304 [I 2022-10-27 07:15:22] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:21 [I 2022-10-27 07:15:22] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00243 [I 2022-10-27 07:15:22] ...trap/engines/engine.py.230: loss: 0.13337 [I 2022-10-27 07:15:22] ...optimizers/trijoint.py.151: total_norm: 10.945239 [I 2022-10-27 07:15:28] ...trap/engines/engine.py.224: train: epoch 54 | batch 1530/2304 [I 2022-10-27 07:15:28] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:17 [I 2022-10-27 07:15:28] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00246 [I 2022-10-27 07:15:28] ...trap/engines/engine.py.230: loss: 0.21906 [I 2022-10-27 07:15:28] ...optimizers/trijoint.py.151: total_norm: 24.309383 [I 2022-10-27 07:15:34] ...trap/engines/engine.py.224: train: epoch 54 | batch 1540/2304 [I 2022-10-27 07:15:34] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:10 [I 2022-10-27 07:15:34] ...trap/engines/engine.py.229: process: 0.55135 | load: 0.00242 [I 2022-10-27 07:15:34] ...trap/engines/engine.py.230: loss: 0.16369 [I 2022-10-27 07:15:34] ...optimizers/trijoint.py.151: total_norm: 18.852032 [I 2022-10-27 07:15:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 1550/2304 [I 2022-10-27 07:15:39] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:06 [I 2022-10-27 07:15:39] ...trap/engines/engine.py.229: process: 0.56646 | load: 0.00244 [I 2022-10-27 07:15:39] ...trap/engines/engine.py.230: loss: 0.13340 [I 2022-10-27 07:15:39] ...optimizers/trijoint.py.151: total_norm: 20.875496 [I 2022-10-27 07:15:45] ...trap/engines/engine.py.224: train: epoch 54 | batch 1560/2304 [I 2022-10-27 07:15:45] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:00 [I 2022-10-27 07:15:45] ...trap/engines/engine.py.229: process: 0.55720 | load: 0.00247 [I 2022-10-27 07:15:45] ...trap/engines/engine.py.230: loss: 0.22678 [I 2022-10-27 07:15:45] ...optimizers/trijoint.py.151: total_norm: 13.149849 [I 2022-10-27 07:15:51] ...trap/engines/engine.py.224: train: epoch 54 | batch 1570/2304 [I 2022-10-27 07:15:51] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:06:53 [I 2022-10-27 07:15:51] ...trap/engines/engine.py.229: process: 0.55905 | load: 0.00244 [I 2022-10-27 07:15:51] ...trap/engines/engine.py.230: loss: 0.12756 [I 2022-10-27 07:15:51] ...optimizers/trijoint.py.151: total_norm: 13.285990 [I 2022-10-27 07:15:56] ...trap/engines/engine.py.224: train: epoch 54 | batch 1580/2304 [I 2022-10-27 07:15:56] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:51 [I 2022-10-27 07:15:56] ...trap/engines/engine.py.229: process: 0.56542 | load: 0.00248 [I 2022-10-27 07:15:56] ...trap/engines/engine.py.230: loss: 0.17284 [I 2022-10-27 07:15:56] ...optimizers/trijoint.py.151: total_norm: 8.884178 [I 2022-10-27 07:16:02] ...trap/engines/engine.py.224: train: epoch 54 | batch 1590/2304 [I 2022-10-27 07:16:02] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:53 [I 2022-10-27 07:16:02] ...trap/engines/engine.py.229: process: 0.56561 | load: 0.00250 [I 2022-10-27 07:16:02] ...trap/engines/engine.py.230: loss: 0.11742 [I 2022-10-27 07:16:02] ...optimizers/trijoint.py.151: total_norm: 15.579235 [I 2022-10-27 07:16:08] ...trap/engines/engine.py.224: train: epoch 54 | batch 1600/2304 [I 2022-10-27 07:16:08] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:36 [I 2022-10-27 07:16:08] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00246 [I 2022-10-27 07:16:08] ...trap/engines/engine.py.230: loss: 0.18240 [I 2022-10-27 07:16:08] ...optimizers/trijoint.py.151: total_norm: 10.364411 [I 2022-10-27 07:16:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 1610/2304 [I 2022-10-27 07:16:14] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:31 [I 2022-10-27 07:16:14] ...trap/engines/engine.py.229: process: 0.56561 | load: 0.00246 [I 2022-10-27 07:16:14] ...trap/engines/engine.py.230: loss: 0.12203 [I 2022-10-27 07:16:14] ...optimizers/trijoint.py.151: total_norm: 11.393595 [I 2022-10-27 07:16:19] ...trap/engines/engine.py.224: train: epoch 54 | batch 1620/2304 [I 2022-10-27 07:16:19] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:22 [I 2022-10-27 07:16:19] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00245 [I 2022-10-27 07:16:19] ...trap/engines/engine.py.230: loss: 0.21541 [I 2022-10-27 07:16:19] ...optimizers/trijoint.py.151: total_norm: 23.575647 [I 2022-10-27 07:16:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 1630/2304 [I 2022-10-27 07:16:25] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:15 [I 2022-10-27 07:16:25] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00242 [I 2022-10-27 07:16:25] ...trap/engines/engine.py.230: loss: 0.26039 [I 2022-10-27 07:16:25] ...optimizers/trijoint.py.151: total_norm: 27.285105 [I 2022-10-27 07:16:30] ...trap/engines/engine.py.224: train: epoch 54 | batch 1640/2304 [I 2022-10-27 07:16:30] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:08 [I 2022-10-27 07:16:30] ...trap/engines/engine.py.229: process: 0.55968 | load: 0.00247 [I 2022-10-27 07:16:30] ...trap/engines/engine.py.230: loss: 0.15935 [I 2022-10-27 07:16:30] ...optimizers/trijoint.py.151: total_norm: 10.606556 [I 2022-10-27 07:16:36] ...trap/engines/engine.py.224: train: epoch 54 | batch 1650/2304 [I 2022-10-27 07:16:36] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:02 [I 2022-10-27 07:16:36] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00250 [I 2022-10-27 07:16:36] ...trap/engines/engine.py.230: loss: 0.14663 [I 2022-10-27 07:16:36] ...optimizers/trijoint.py.151: total_norm: 20.896492 [I 2022-10-27 07:16:41] ...trap/engines/engine.py.224: train: epoch 54 | batch 1660/2304 [I 2022-10-27 07:16:41] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:05:55 [I 2022-10-27 07:16:41] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00248 [I 2022-10-27 07:16:41] ...trap/engines/engine.py.230: loss: 0.19428 [I 2022-10-27 07:16:41] ...optimizers/trijoint.py.151: total_norm: 11.797318 [I 2022-10-27 07:16:47] ...trap/engines/engine.py.224: train: epoch 54 | batch 1670/2304 [I 2022-10-27 07:16:47] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:05:55 [I 2022-10-27 07:16:47] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00244 [I 2022-10-27 07:16:47] ...trap/engines/engine.py.230: loss: 0.18905 [I 2022-10-27 07:16:47] ...optimizers/trijoint.py.151: total_norm: 40.411358 [I 2022-10-27 07:16:53] ...trap/engines/engine.py.224: train: epoch 54 | batch 1680/2304 [I 2022-10-27 07:16:53] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:45 [I 2022-10-27 07:16:53] ...trap/engines/engine.py.229: process: 0.54506 | load: 0.00245 [I 2022-10-27 07:16:53] ...trap/engines/engine.py.230: loss: 0.11184 [I 2022-10-27 07:16:53] ...optimizers/trijoint.py.151: total_norm: 8.506941 [I 2022-10-27 07:16:58] ...trap/engines/engine.py.224: train: epoch 54 | batch 1690/2304 [I 2022-10-27 07:16:58] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:05:42 [I 2022-10-27 07:16:58] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00250 [I 2022-10-27 07:16:58] ...trap/engines/engine.py.230: loss: 0.15560 [I 2022-10-27 07:16:58] ...optimizers/trijoint.py.151: total_norm: 12.110807 [I 2022-10-27 07:17:04] ...trap/engines/engine.py.224: train: epoch 54 | batch 1700/2304 [I 2022-10-27 07:17:04] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:34 [I 2022-10-27 07:17:04] ...trap/engines/engine.py.229: process: 0.56445 | load: 0.00245 [I 2022-10-27 07:17:04] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-27 07:17:04] ...optimizers/trijoint.py.151: total_norm: 21.655062 [I 2022-10-27 07:17:09] ...trap/engines/engine.py.224: train: epoch 54 | batch 1710/2304 [I 2022-10-27 07:17:09] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:29 [I 2022-10-27 07:17:09] ...trap/engines/engine.py.229: process: 0.56173 | load: 0.00241 [I 2022-10-27 07:17:09] ...trap/engines/engine.py.230: loss: 0.19762 [I 2022-10-27 07:17:09] ...optimizers/trijoint.py.151: total_norm: 21.068666 [I 2022-10-27 07:17:15] ...trap/engines/engine.py.224: train: epoch 54 | batch 1720/2304 [I 2022-10-27 07:17:15] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:25 [I 2022-10-27 07:17:15] ...trap/engines/engine.py.229: process: 0.56582 | load: 0.00241 [I 2022-10-27 07:17:15] ...trap/engines/engine.py.230: loss: 0.19723 [I 2022-10-27 07:17:15] ...optimizers/trijoint.py.151: total_norm: 17.396679 [I 2022-10-27 07:17:20] ...trap/engines/engine.py.224: train: epoch 54 | batch 1730/2304 [I 2022-10-27 07:17:20] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:16 [I 2022-10-27 07:17:20] ...trap/engines/engine.py.229: process: 0.54782 | load: 0.00241 [I 2022-10-27 07:17:20] ...trap/engines/engine.py.230: loss: 0.18429 [I 2022-10-27 07:17:20] ...optimizers/trijoint.py.151: total_norm: 15.809767 [I 2022-10-27 07:17:26] ...trap/engines/engine.py.224: train: epoch 54 | batch 1740/2304 [I 2022-10-27 07:17:26] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:16 [I 2022-10-27 07:17:26] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00244 [I 2022-10-27 07:17:26] ...trap/engines/engine.py.230: loss: 0.19360 [I 2022-10-27 07:17:26] ...optimizers/trijoint.py.151: total_norm: 14.752771 [I 2022-10-27 07:17:32] ...trap/engines/engine.py.224: train: epoch 54 | batch 1750/2304 [I 2022-10-27 07:17:32] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:09 [I 2022-10-27 07:17:32] ...trap/engines/engine.py.229: process: 0.56579 | load: 0.00260 [I 2022-10-27 07:17:32] ...trap/engines/engine.py.230: loss: 0.14624 [I 2022-10-27 07:17:32] ...optimizers/trijoint.py.151: total_norm: 42.095825 [I 2022-10-27 07:17:37] ...trap/engines/engine.py.224: train: epoch 54 | batch 1760/2304 [I 2022-10-27 07:17:37] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:05 [I 2022-10-27 07:17:37] ...trap/engines/engine.py.229: process: 0.56434 | load: 0.00249 [I 2022-10-27 07:17:37] ...trap/engines/engine.py.230: loss: 0.20030 [I 2022-10-27 07:17:37] ...optimizers/trijoint.py.151: total_norm: 19.657656 [I 2022-10-27 07:17:43] ...trap/engines/engine.py.224: train: epoch 54 | batch 1770/2304 [I 2022-10-27 07:17:43] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:04:58 [I 2022-10-27 07:17:43] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00245 [I 2022-10-27 07:17:43] ...trap/engines/engine.py.230: loss: 0.20761 [I 2022-10-27 07:17:43] ...optimizers/trijoint.py.151: total_norm: 23.099245 [I 2022-10-27 07:17:48] ...trap/engines/engine.py.224: train: epoch 54 | batch 1780/2304 [I 2022-10-27 07:17:48] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:04:50 [I 2022-10-27 07:17:48] ...trap/engines/engine.py.229: process: 0.54397 | load: 0.00243 [I 2022-10-27 07:17:48] ...trap/engines/engine.py.230: loss: 0.22222 [I 2022-10-27 07:17:48] ...optimizers/trijoint.py.151: total_norm: 32.806408 [I 2022-10-27 07:17:54] ...trap/engines/engine.py.224: train: epoch 54 | batch 1790/2304 [I 2022-10-27 07:17:54] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:04:44 [I 2022-10-27 07:17:54] ...trap/engines/engine.py.229: process: 0.55242 | load: 0.00244 [I 2022-10-27 07:17:54] ...trap/engines/engine.py.230: loss: 0.15686 [I 2022-10-27 07:17:54] ...optimizers/trijoint.py.151: total_norm: 15.742951 [I 2022-10-27 07:18:00] ...trap/engines/engine.py.224: train: epoch 54 | batch 1800/2304 [I 2022-10-27 07:18:00] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:04:42 [I 2022-10-27 07:18:00] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00244 [I 2022-10-27 07:18:00] ...trap/engines/engine.py.230: loss: 0.17864 [I 2022-10-27 07:18:00] ...optimizers/trijoint.py.151: total_norm: 20.656755 [I 2022-10-27 07:18:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 1810/2304 [I 2022-10-27 07:18:05] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:34 [I 2022-10-27 07:18:05] ...trap/engines/engine.py.229: process: 0.54594 | load: 0.00243 [I 2022-10-27 07:18:05] ...trap/engines/engine.py.230: loss: 0.18444 [I 2022-10-27 07:18:05] ...optimizers/trijoint.py.151: total_norm: 18.154449 [I 2022-10-27 07:18:11] ...trap/engines/engine.py.224: train: epoch 54 | batch 1820/2304 [I 2022-10-27 07:18:11] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:28 [I 2022-10-27 07:18:11] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00244 [I 2022-10-27 07:18:11] ...trap/engines/engine.py.230: loss: 0.15884 [I 2022-10-27 07:18:11] ...optimizers/trijoint.py.151: total_norm: 13.276395 [I 2022-10-27 07:18:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 1830/2304 [I 2022-10-27 07:18:16] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:23 [I 2022-10-27 07:18:16] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00253 [I 2022-10-27 07:18:16] ...trap/engines/engine.py.230: loss: 0.16224 [I 2022-10-27 07:18:16] ...optimizers/trijoint.py.151: total_norm: 16.160940 [I 2022-10-27 07:18:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 1840/2304 [I 2022-10-27 07:18:22] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:21 [I 2022-10-27 07:18:22] ...trap/engines/engine.py.229: process: 0.54986 | load: 0.00252 [I 2022-10-27 07:18:22] ...trap/engines/engine.py.230: loss: 0.10938 [I 2022-10-27 07:18:22] ...optimizers/trijoint.py.151: total_norm: 8.550398 [I 2022-10-27 07:18:28] ...trap/engines/engine.py.224: train: epoch 54 | batch 1850/2304 [I 2022-10-27 07:18:28] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:14 [I 2022-10-27 07:18:28] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00243 [I 2022-10-27 07:18:28] ...trap/engines/engine.py.230: loss: 0.16957 [I 2022-10-27 07:18:28] ...optimizers/trijoint.py.151: total_norm: 21.392807 [I 2022-10-27 07:18:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 1860/2304 [I 2022-10-27 07:18:33] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:07 [I 2022-10-27 07:18:33] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00247 [I 2022-10-27 07:18:33] ...trap/engines/engine.py.230: loss: 0.12378 [I 2022-10-27 07:18:33] ...optimizers/trijoint.py.151: total_norm: 9.936286 [I 2022-10-27 07:18:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 1870/2304 [I 2022-10-27 07:18:39] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:00 [I 2022-10-27 07:18:39] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00242 [I 2022-10-27 07:18:39] ...trap/engines/engine.py.230: loss: 0.15151 [I 2022-10-27 07:18:39] ...optimizers/trijoint.py.151: total_norm: 26.990374 [I 2022-10-27 07:18:44] ...trap/engines/engine.py.224: train: epoch 54 | batch 1880/2304 [I 2022-10-27 07:18:44] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:03:53 [I 2022-10-27 07:18:44] ...trap/engines/engine.py.229: process: 0.54483 | load: 0.00240 [I 2022-10-27 07:18:44] ...trap/engines/engine.py.230: loss: 0.13661 [I 2022-10-27 07:18:44] ...optimizers/trijoint.py.151: total_norm: 15.922730 [I 2022-10-27 07:18:50] ...trap/engines/engine.py.224: train: epoch 54 | batch 1890/2304 [I 2022-10-27 07:18:50] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:03:49 [I 2022-10-27 07:18:50] ...trap/engines/engine.py.229: process: 0.56049 | load: 0.00241 [I 2022-10-27 07:18:50] ...trap/engines/engine.py.230: loss: 0.15795 [I 2022-10-27 07:18:50] ...optimizers/trijoint.py.151: total_norm: 17.812452 [I 2022-10-27 07:18:55] ...trap/engines/engine.py.224: train: epoch 54 | batch 1900/2304 [I 2022-10-27 07:18:55] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:03:43 [I 2022-10-27 07:18:55] ...trap/engines/engine.py.229: process: 0.55451 | load: 0.00247 [I 2022-10-27 07:18:55] ...trap/engines/engine.py.230: loss: 0.11547 [I 2022-10-27 07:18:55] ...optimizers/trijoint.py.151: total_norm: 8.142047 [I 2022-10-27 07:19:01] ...trap/engines/engine.py.224: train: epoch 54 | batch 1910/2304 [I 2022-10-27 07:19:01] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:03:39 [I 2022-10-27 07:19:01] ...trap/engines/engine.py.229: process: 0.55767 | load: 0.00243 [I 2022-10-27 07:19:01] ...trap/engines/engine.py.230: loss: 0.17263 [I 2022-10-27 07:19:01] ...optimizers/trijoint.py.151: total_norm: 12.316011 [I 2022-10-27 07:19:06] ...trap/engines/engine.py.224: train: epoch 54 | batch 1920/2304 [I 2022-10-27 07:19:06] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:34 [I 2022-10-27 07:19:06] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00263 [I 2022-10-27 07:19:06] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-27 07:19:06] ...optimizers/trijoint.py.151: total_norm: 9.626097 [I 2022-10-27 07:19:12] ...trap/engines/engine.py.224: train: epoch 54 | batch 1930/2304 [I 2022-10-27 07:19:12] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:27 [I 2022-10-27 07:19:12] ...trap/engines/engine.py.229: process: 0.55597 | load: 0.00260 [I 2022-10-27 07:19:12] ...trap/engines/engine.py.230: loss: 0.21183 [I 2022-10-27 07:19:12] ...optimizers/trijoint.py.151: total_norm: 19.791409 [I 2022-10-27 07:19:18] ...trap/engines/engine.py.224: train: epoch 54 | batch 1940/2304 [I 2022-10-27 07:19:18] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:21 [I 2022-10-27 07:19:18] ...trap/engines/engine.py.229: process: 0.55154 | load: 0.00245 [I 2022-10-27 07:19:18] ...trap/engines/engine.py.230: loss: 0.18689 [I 2022-10-27 07:19:18] ...optimizers/trijoint.py.151: total_norm: 34.371986 [I 2022-10-27 07:19:23] ...trap/engines/engine.py.224: train: epoch 54 | batch 1950/2304 [I 2022-10-27 07:19:23] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:23 [I 2022-10-27 07:19:23] ...trap/engines/engine.py.229: process: 0.56394 | load: 0.00246 [I 2022-10-27 07:19:23] ...trap/engines/engine.py.230: loss: 0.16578 [I 2022-10-27 07:19:23] ...optimizers/trijoint.py.151: total_norm: 26.935026 [I 2022-10-27 07:19:29] ...trap/engines/engine.py.224: train: epoch 54 | batch 1960/2304 [I 2022-10-27 07:19:29] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:12 [I 2022-10-27 07:19:29] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00245 [I 2022-10-27 07:19:29] ...trap/engines/engine.py.230: loss: 0.14573 [I 2022-10-27 07:19:29] ...optimizers/trijoint.py.151: total_norm: 20.478441 [I 2022-10-27 07:19:35] ...trap/engines/engine.py.224: train: epoch 54 | batch 1970/2304 [I 2022-10-27 07:19:35] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:07 [I 2022-10-27 07:19:35] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00244 [I 2022-10-27 07:19:35] ...trap/engines/engine.py.230: loss: 0.15514 [I 2022-10-27 07:19:35] ...optimizers/trijoint.py.151: total_norm: 25.126633 [I 2022-10-27 07:19:40] ...trap/engines/engine.py.224: train: epoch 54 | batch 1980/2304 [I 2022-10-27 07:19:40] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:02 [I 2022-10-27 07:19:40] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00246 [I 2022-10-27 07:19:40] ...trap/engines/engine.py.230: loss: 0.20540 [I 2022-10-27 07:19:40] ...optimizers/trijoint.py.151: total_norm: 18.239559 [I 2022-10-27 07:19:46] ...trap/engines/engine.py.224: train: epoch 54 | batch 1990/2304 [I 2022-10-27 07:19:46] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:02:57 [I 2022-10-27 07:19:46] ...trap/engines/engine.py.229: process: 0.56796 | load: 0.00244 [I 2022-10-27 07:19:46] ...trap/engines/engine.py.230: loss: 0.25846 [I 2022-10-27 07:19:46] ...optimizers/trijoint.py.151: total_norm: 12.594311 [I 2022-10-27 07:19:51] ...trap/engines/engine.py.224: train: epoch 54 | batch 2000/2304 [I 2022-10-27 07:19:51] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:02:49 [I 2022-10-27 07:19:51] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00241 [I 2022-10-27 07:19:51] ...trap/engines/engine.py.230: loss: 0.13518 [I 2022-10-27 07:19:51] ...optimizers/trijoint.py.151: total_norm: 11.154040 [I 2022-10-27 07:19:57] ...trap/engines/engine.py.224: train: epoch 54 | batch 2010/2304 [I 2022-10-27 07:19:57] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:02:44 [I 2022-10-27 07:19:57] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00249 [I 2022-10-27 07:19:57] ...trap/engines/engine.py.230: loss: 0.12338 [I 2022-10-27 07:19:57] ...optimizers/trijoint.py.151: total_norm: 24.006348 [I 2022-10-27 07:20:03] ...trap/engines/engine.py.224: train: epoch 54 | batch 2020/2304 [I 2022-10-27 07:20:03] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:37 [I 2022-10-27 07:20:03] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00244 [I 2022-10-27 07:20:03] ...trap/engines/engine.py.230: loss: 0.19097 [I 2022-10-27 07:20:03] ...optimizers/trijoint.py.151: total_norm: 15.232383 [I 2022-10-27 07:20:08] ...trap/engines/engine.py.224: train: epoch 54 | batch 2030/2304 [I 2022-10-27 07:20:08] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:32 [I 2022-10-27 07:20:08] ...trap/engines/engine.py.229: process: 0.56523 | load: 0.00266 [I 2022-10-27 07:20:08] ...trap/engines/engine.py.230: loss: 0.15191 [I 2022-10-27 07:20:08] ...optimizers/trijoint.py.151: total_norm: 12.809869 [I 2022-10-27 07:20:14] ...trap/engines/engine.py.224: train: epoch 54 | batch 2040/2304 [I 2022-10-27 07:20:14] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:26 [I 2022-10-27 07:20:14] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00247 [I 2022-10-27 07:20:14] ...trap/engines/engine.py.230: loss: 0.13564 [I 2022-10-27 07:20:14] ...optimizers/trijoint.py.151: total_norm: 14.807354 [I 2022-10-27 07:20:19] ...trap/engines/engine.py.224: train: epoch 54 | batch 2050/2304 [I 2022-10-27 07:20:19] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:22 [I 2022-10-27 07:20:19] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00244 [I 2022-10-27 07:20:19] ...trap/engines/engine.py.230: loss: 0.21104 [I 2022-10-27 07:20:19] ...optimizers/trijoint.py.151: total_norm: 22.074842 [I 2022-10-27 07:20:25] ...trap/engines/engine.py.224: train: epoch 54 | batch 2060/2304 [I 2022-10-27 07:20:25] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:16 [I 2022-10-27 07:20:25] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00239 [I 2022-10-27 07:20:25] ...trap/engines/engine.py.230: loss: 0.15559 [I 2022-10-27 07:20:25] ...optimizers/trijoint.py.151: total_norm: 18.059107 [I 2022-10-27 07:20:31] ...trap/engines/engine.py.224: train: epoch 54 | batch 2070/2304 [I 2022-10-27 07:20:31] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:13 [I 2022-10-27 07:20:31] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00243 [I 2022-10-27 07:20:31] ...trap/engines/engine.py.230: loss: 0.16217 [I 2022-10-27 07:20:31] ...optimizers/trijoint.py.151: total_norm: 9.031117 [I 2022-10-27 07:20:36] ...trap/engines/engine.py.224: train: epoch 54 | batch 2080/2304 [I 2022-10-27 07:20:36] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:05 [I 2022-10-27 07:20:36] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00244 [I 2022-10-27 07:20:36] ...trap/engines/engine.py.230: loss: 0.21735 [I 2022-10-27 07:20:36] ...optimizers/trijoint.py.151: total_norm: 22.510036 [I 2022-10-27 07:20:42] ...trap/engines/engine.py.224: train: epoch 54 | batch 2090/2304 [I 2022-10-27 07:20:42] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:01:58 [I 2022-10-27 07:20:42] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00240 [I 2022-10-27 07:20:42] ...trap/engines/engine.py.230: loss: 0.17130 [I 2022-10-27 07:20:42] ...optimizers/trijoint.py.151: total_norm: 21.076559 [I 2022-10-27 07:20:47] ...trap/engines/engine.py.224: train: epoch 54 | batch 2100/2304 [I 2022-10-27 07:20:47] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:01:53 [I 2022-10-27 07:20:47] ...trap/engines/engine.py.229: process: 0.55023 | load: 0.00243 [I 2022-10-27 07:20:47] ...trap/engines/engine.py.230: loss: 0.14788 [I 2022-10-27 07:20:47] ...optimizers/trijoint.py.151: total_norm: 16.083582 [I 2022-10-27 07:20:53] ...trap/engines/engine.py.224: train: epoch 54 | batch 2110/2304 [I 2022-10-27 07:20:53] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:01:47 [I 2022-10-27 07:20:53] ...trap/engines/engine.py.229: process: 0.55259 | load: 0.00243 [I 2022-10-27 07:20:53] ...trap/engines/engine.py.230: loss: 0.14849 [I 2022-10-27 07:20:53] ...optimizers/trijoint.py.151: total_norm: 20.573742 [I 2022-10-27 07:20:59] ...trap/engines/engine.py.224: train: epoch 54 | batch 2120/2304 [I 2022-10-27 07:20:59] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:01:42 [I 2022-10-27 07:20:59] ...trap/engines/engine.py.229: process: 0.56757 | load: 0.00261 [I 2022-10-27 07:20:59] ...trap/engines/engine.py.230: loss: 0.17516 [I 2022-10-27 07:20:59] ...optimizers/trijoint.py.151: total_norm: 17.678936 [I 2022-10-27 07:21:04] ...trap/engines/engine.py.224: train: epoch 54 | batch 2130/2304 [I 2022-10-27 07:21:04] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:36 [I 2022-10-27 07:21:04] ...trap/engines/engine.py.229: process: 0.55737 | load: 0.00241 [I 2022-10-27 07:21:04] ...trap/engines/engine.py.230: loss: 0.15780 [I 2022-10-27 07:21:04] ...optimizers/trijoint.py.151: total_norm: 16.614269 [I 2022-10-27 07:21:10] ...trap/engines/engine.py.224: train: epoch 54 | batch 2140/2304 [I 2022-10-27 07:21:10] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:31 [I 2022-10-27 07:21:10] ...trap/engines/engine.py.229: process: 0.55127 | load: 0.00248 [I 2022-10-27 07:21:10] ...trap/engines/engine.py.230: loss: 0.24956 [I 2022-10-27 07:21:10] ...optimizers/trijoint.py.151: total_norm: 36.877613 [I 2022-10-27 07:21:15] ...trap/engines/engine.py.224: train: epoch 54 | batch 2150/2304 [I 2022-10-27 07:21:15] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:24 [I 2022-10-27 07:21:15] ...trap/engines/engine.py.229: process: 0.54614 | load: 0.00246 [I 2022-10-27 07:21:15] ...trap/engines/engine.py.230: loss: 0.13764 [I 2022-10-27 07:21:15] ...optimizers/trijoint.py.151: total_norm: 18.375540 [I 2022-10-27 07:21:21] ...trap/engines/engine.py.224: train: epoch 54 | batch 2160/2304 [I 2022-10-27 07:21:21] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:19 [I 2022-10-27 07:21:21] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00245 [I 2022-10-27 07:21:21] ...trap/engines/engine.py.230: loss: 0.19073 [I 2022-10-27 07:21:21] ...optimizers/trijoint.py.151: total_norm: 22.579029 [I 2022-10-27 07:21:26] ...trap/engines/engine.py.224: train: epoch 54 | batch 2170/2304 [I 2022-10-27 07:21:26] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:14 [I 2022-10-27 07:21:26] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00241 [I 2022-10-27 07:21:26] ...trap/engines/engine.py.230: loss: 0.11568 [I 2022-10-27 07:21:26] ...optimizers/trijoint.py.151: total_norm: 12.237228 [I 2022-10-27 07:21:32] ...trap/engines/engine.py.224: train: epoch 54 | batch 2180/2304 [I 2022-10-27 07:21:32] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:09 [I 2022-10-27 07:21:32] ...trap/engines/engine.py.229: process: 0.56570 | load: 0.00242 [I 2022-10-27 07:21:32] ...trap/engines/engine.py.230: loss: 0.12226 [I 2022-10-27 07:21:32] ...optimizers/trijoint.py.151: total_norm: 11.255841 [I 2022-10-27 07:21:37] ...trap/engines/engine.py.224: train: epoch 54 | batch 2190/2304 [I 2022-10-27 07:21:37] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:03 [I 2022-10-27 07:21:37] ...trap/engines/engine.py.229: process: 0.55293 | load: 0.00240 [I 2022-10-27 07:21:37] ...trap/engines/engine.py.230: loss: 0.14904 [I 2022-10-27 07:21:37] ...optimizers/trijoint.py.151: total_norm: 16.638536 [I 2022-10-27 07:21:43] ...trap/engines/engine.py.224: train: epoch 54 | batch 2200/2304 [I 2022-10-27 07:21:43] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:00:57 [I 2022-10-27 07:21:43] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00244 [I 2022-10-27 07:21:43] ...trap/engines/engine.py.230: loss: 0.14278 [I 2022-10-27 07:21:43] ...optimizers/trijoint.py.151: total_norm: 15.227504 [I 2022-10-27 07:21:49] ...trap/engines/engine.py.224: train: epoch 54 | batch 2210/2304 [I 2022-10-27 07:21:49] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:00:51 [I 2022-10-27 07:21:49] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00242 [I 2022-10-27 07:21:49] ...trap/engines/engine.py.230: loss: 0.20749 [I 2022-10-27 07:21:49] ...optimizers/trijoint.py.151: total_norm: 22.760462 [I 2022-10-27 07:21:54] ...trap/engines/engine.py.224: train: epoch 54 | batch 2220/2304 [I 2022-10-27 07:21:54] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:00:46 [I 2022-10-27 07:21:54] ...trap/engines/engine.py.229: process: 0.56294 | load: 0.00244 [I 2022-10-27 07:21:54] ...trap/engines/engine.py.230: loss: 0.11212 [I 2022-10-27 07:21:54] ...optimizers/trijoint.py.151: total_norm: 10.638462 [I 2022-10-27 07:22:00] ...trap/engines/engine.py.224: train: epoch 54 | batch 2230/2304 [I 2022-10-27 07:22:00] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:40 [I 2022-10-27 07:22:00] ...trap/engines/engine.py.229: process: 0.54515 | load: 0.00240 [I 2022-10-27 07:22:00] ...trap/engines/engine.py.230: loss: 0.11730 [I 2022-10-27 07:22:00] ...optimizers/trijoint.py.151: total_norm: 11.630404 [I 2022-10-27 07:22:05] ...trap/engines/engine.py.224: train: epoch 54 | batch 2240/2304 [I 2022-10-27 07:22:05] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:35 [I 2022-10-27 07:22:05] ...trap/engines/engine.py.229: process: 0.57494 | load: 0.00246 [I 2022-10-27 07:22:05] ...trap/engines/engine.py.230: loss: 0.18490 [I 2022-10-27 07:22:05] ...optimizers/trijoint.py.151: total_norm: 16.435009 [I 2022-10-27 07:22:11] ...trap/engines/engine.py.224: train: epoch 54 | batch 2250/2304 [I 2022-10-27 07:22:11] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:30 [I 2022-10-27 07:22:11] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00244 [I 2022-10-27 07:22:11] ...trap/engines/engine.py.230: loss: 0.12321 [I 2022-10-27 07:22:11] ...optimizers/trijoint.py.151: total_norm: 20.662415 [I 2022-10-27 07:22:16] ...trap/engines/engine.py.224: train: epoch 54 | batch 2260/2304 [I 2022-10-27 07:22:16] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:24 [I 2022-10-27 07:22:16] ...trap/engines/engine.py.229: process: 0.54041 | load: 0.00247 [I 2022-10-27 07:22:16] ...trap/engines/engine.py.230: loss: 0.19054 [I 2022-10-27 07:22:16] ...optimizers/trijoint.py.151: total_norm: 12.169174 [I 2022-10-27 07:22:22] ...trap/engines/engine.py.224: train: epoch 54 | batch 2270/2304 [I 2022-10-27 07:22:22] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:18 [I 2022-10-27 07:22:22] ...trap/engines/engine.py.229: process: 0.56219 | load: 0.00244 [I 2022-10-27 07:22:22] ...trap/engines/engine.py.230: loss: 0.12736 [I 2022-10-27 07:22:22] ...optimizers/trijoint.py.151: total_norm: 15.333617 [I 2022-10-27 07:22:28] ...trap/engines/engine.py.224: train: epoch 54 | batch 2280/2304 [I 2022-10-27 07:22:28] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:13 [I 2022-10-27 07:22:28] ...trap/engines/engine.py.229: process: 0.54977 | load: 0.00261 [I 2022-10-27 07:22:28] ...trap/engines/engine.py.230: loss: 0.12797 [I 2022-10-27 07:22:28] ...optimizers/trijoint.py.151: total_norm: 12.269856 [I 2022-10-27 07:22:33] ...trap/engines/engine.py.224: train: epoch 54 | batch 2290/2304 [I 2022-10-27 07:22:33] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:07 [I 2022-10-27 07:22:33] ...trap/engines/engine.py.229: process: 0.55345 | load: 0.00242 [I 2022-10-27 07:22:33] ...trap/engines/engine.py.230: loss: 0.19197 [I 2022-10-27 07:22:33] ...optimizers/trijoint.py.151: total_norm: 24.045656 [I 2022-10-27 07:22:39] ...trap/engines/engine.py.224: train: epoch 54 | batch 2300/2304 [I 2022-10-27 07:22:39] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:02 [I 2022-10-27 07:22:39] ...trap/engines/engine.py.229: process: 0.54751 | load: 0.00014 [I 2022-10-27 07:22:39] ...trap/engines/engine.py.230: loss: 0.13791 [I 2022-10-27 07:22:39] ...optimizers/trijoint.py.151: total_norm: 8.808983 [I 2022-10-27 07:22:41] ...trap/engines/engine.py.224: train: epoch 54 | batch 2304/2304 [I 2022-10-27 07:22:41] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:00 [I 2022-10-27 07:22:41] ...trap/engines/engine.py.229: process: 0.54320 | load: 0.00014 [I 2022-10-27 07:22:41] ...trap/engines/engine.py.230: loss: 0.15627 [I 2022-10-27 07:22:41] ...optimizers/trijoint.py.151: total_norm: 17.476067 [S 2022-10-27 07:22:42] ...trap/engines/engine.py.129: train_epoch.epoch: 54 [S 2022-10-27 07:22:42] ...trap/engines/engine.py.129: train_epoch.loss: 0.1606435924380565 [I 2022-10-27 07:22:42] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 07:22:43] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 54 [I 2022-10-27 07:22:43] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 07:22:45] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 07:22:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 07:22:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 07:22:46] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 07:22:47] ...trap/engines/engine.py.321: eval: epoch 54 | batch 0/510 [I 2022-10-27 07:22:47] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:34:22 [I 2022-10-27 07:22:47] ...trap/engines/engine.py.326: process: 4.04403 | load: 3.67297 [I 2022-10-27 07:22:50] ...trap/engines/engine.py.321: eval: epoch 54 | batch 10/510 [I 2022-10-27 07:22:50] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:05 [I 2022-10-27 07:22:50] ...trap/engines/engine.py.326: process: 0.32831 | load: 0.00017 [I 2022-10-27 07:22:53] ...trap/engines/engine.py.321: eval: epoch 54 | batch 20/510 [I 2022-10-27 07:22:53] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:04 [I 2022-10-27 07:22:53] ...trap/engines/engine.py.326: process: 0.33180 | load: 0.00023 [I 2022-10-27 07:22:57] ...trap/engines/engine.py.321: eval: epoch 54 | batch 30/510 [I 2022-10-27 07:22:57] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:02:39 [I 2022-10-27 07:22:57] ...trap/engines/engine.py.326: process: 0.32122 | load: 0.00017 [I 2022-10-27 07:23:00] ...trap/engines/engine.py.321: eval: epoch 54 | batch 40/510 [I 2022-10-27 07:23:00] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:36 [I 2022-10-27 07:23:00] ...trap/engines/engine.py.326: process: 0.33168 | load: 0.00018 [I 2022-10-27 07:23:03] ...trap/engines/engine.py.321: eval: epoch 54 | batch 50/510 [I 2022-10-27 07:23:03] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:36 [I 2022-10-27 07:23:03] ...trap/engines/engine.py.326: process: 0.35767 | load: 0.00026 [I 2022-10-27 07:23:07] ...trap/engines/engine.py.321: eval: epoch 54 | batch 60/510 [I 2022-10-27 07:23:07] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:32 [I 2022-10-27 07:23:07] ...trap/engines/engine.py.326: process: 0.33290 | load: 0.00020 [I 2022-10-27 07:23:10] ...trap/engines/engine.py.321: eval: epoch 54 | batch 70/510 [I 2022-10-27 07:23:10] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:30 [I 2022-10-27 07:23:10] ...trap/engines/engine.py.326: process: 0.36339 | load: 0.00017 [I 2022-10-27 07:23:14] ...trap/engines/engine.py.321: eval: epoch 54 | batch 80/510 [I 2022-10-27 07:23:14] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:25 [I 2022-10-27 07:23:14] ...trap/engines/engine.py.326: process: 0.34119 | load: 0.00018 [I 2022-10-27 07:23:17] ...trap/engines/engine.py.321: eval: epoch 54 | batch 90/510 [I 2022-10-27 07:23:17] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:21 [I 2022-10-27 07:23:17] ...trap/engines/engine.py.326: process: 0.32365 | load: 0.00017 [I 2022-10-27 07:23:20] ...trap/engines/engine.py.321: eval: epoch 54 | batch 100/510 [I 2022-10-27 07:23:20] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:13 [I 2022-10-27 07:23:20] ...trap/engines/engine.py.326: process: 0.29238 | load: 0.00016 [I 2022-10-27 07:23:23] ...trap/engines/engine.py.321: eval: epoch 54 | batch 110/510 [I 2022-10-27 07:23:23] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:56 [I 2022-10-27 07:23:23] ...trap/engines/engine.py.326: process: 0.28447 | load: 0.00014 [I 2022-10-27 07:23:26] ...trap/engines/engine.py.321: eval: epoch 54 | batch 120/510 [I 2022-10-27 07:23:26] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:53 [I 2022-10-27 07:23:26] ...trap/engines/engine.py.326: process: 0.29815 | load: 0.00013 [I 2022-10-27 07:23:29] ...trap/engines/engine.py.321: eval: epoch 54 | batch 130/510 [I 2022-10-27 07:23:29] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:51 [I 2022-10-27 07:23:29] ...trap/engines/engine.py.326: process: 0.30620 | load: 0.00017 [I 2022-10-27 07:23:32] ...trap/engines/engine.py.321: eval: epoch 54 | batch 140/510 [I 2022-10-27 07:23:32] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 07:23:32] ...trap/engines/engine.py.326: process: 0.28671 | load: 0.00015 [I 2022-10-27 07:23:35] ...trap/engines/engine.py.321: eval: epoch 54 | batch 150/510 [I 2022-10-27 07:23:35] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:45 [I 2022-10-27 07:23:35] ...trap/engines/engine.py.326: process: 0.30018 | load: 0.00016 [I 2022-10-27 07:23:38] ...trap/engines/engine.py.321: eval: epoch 54 | batch 160/510 [I 2022-10-27 07:23:38] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:42 [I 2022-10-27 07:23:38] ...trap/engines/engine.py.326: process: 0.29051 | load: 0.00021 [I 2022-10-27 07:23:41] ...trap/engines/engine.py.321: eval: epoch 54 | batch 170/510 [I 2022-10-27 07:23:41] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-27 07:23:41] ...trap/engines/engine.py.326: process: 0.29332 | load: 0.00015 [I 2022-10-27 07:23:44] ...trap/engines/engine.py.321: eval: epoch 54 | batch 180/510 [I 2022-10-27 07:23:44] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 07:23:44] ...trap/engines/engine.py.326: process: 0.28651 | load: 0.00021 [I 2022-10-27 07:23:46] ...trap/engines/engine.py.321: eval: epoch 54 | batch 190/510 [I 2022-10-27 07:23:46] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:31 [I 2022-10-27 07:23:46] ...trap/engines/engine.py.326: process: 0.28253 | load: 0.00014 [I 2022-10-27 07:23:49] ...trap/engines/engine.py.321: eval: epoch 54 | batch 200/510 [I 2022-10-27 07:23:49] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 07:23:49] ...trap/engines/engine.py.326: process: 0.29447 | load: 0.00017 [I 2022-10-27 07:23:52] ...trap/engines/engine.py.321: eval: epoch 54 | batch 210/510 [I 2022-10-27 07:23:52] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 07:23:52] ...trap/engines/engine.py.326: process: 0.28902 | load: 0.00019 [I 2022-10-27 07:23:55] ...trap/engines/engine.py.321: eval: epoch 54 | batch 220/510 [I 2022-10-27 07:23:55] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:24 [I 2022-10-27 07:23:55] ...trap/engines/engine.py.326: process: 0.29360 | load: 0.00014 [I 2022-10-27 07:23:58] ...trap/engines/engine.py.321: eval: epoch 54 | batch 230/510 [I 2022-10-27 07:23:58] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 07:23:58] ...trap/engines/engine.py.326: process: 0.28472 | load: 0.00033 [I 2022-10-27 07:24:01] ...trap/engines/engine.py.321: eval: epoch 54 | batch 240/510 [I 2022-10-27 07:24:01] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 07:24:01] ...trap/engines/engine.py.326: process: 0.29626 | load: 0.00016 [I 2022-10-27 07:24:04] ...trap/engines/engine.py.321: eval: epoch 54 | batch 250/510 [I 2022-10-27 07:24:04] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:15 [I 2022-10-27 07:24:04] ...trap/engines/engine.py.326: process: 0.28694 | load: 0.00028 [I 2022-10-27 07:24:07] ...trap/engines/engine.py.321: eval: epoch 54 | batch 260/510 [I 2022-10-27 07:24:07] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-27 07:24:07] ...trap/engines/engine.py.326: process: 0.29687 | load: 0.00019 [I 2022-10-27 07:24:10] ...trap/engines/engine.py.321: eval: epoch 54 | batch 270/510 [I 2022-10-27 07:24:10] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 07:24:10] ...trap/engines/engine.py.326: process: 0.28632 | load: 0.00021 [I 2022-10-27 07:24:12] ...trap/engines/engine.py.321: eval: epoch 54 | batch 280/510 [I 2022-10-27 07:24:12] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 07:24:12] ...trap/engines/engine.py.326: process: 0.28337 | load: 0.00016 [I 2022-10-27 07:24:15] ...trap/engines/engine.py.321: eval: epoch 54 | batch 290/510 [I 2022-10-27 07:24:15] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 07:24:15] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00016 [I 2022-10-27 07:24:18] ...trap/engines/engine.py.321: eval: epoch 54 | batch 300/510 [I 2022-10-27 07:24:18] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 07:24:18] ...trap/engines/engine.py.326: process: 0.28782 | load: 0.00021 [I 2022-10-27 07:24:21] ...trap/engines/engine.py.321: eval: epoch 54 | batch 310/510 [I 2022-10-27 07:24:21] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 07:24:21] ...trap/engines/engine.py.326: process: 0.28761 | load: 0.00019 [I 2022-10-27 07:24:24] ...trap/engines/engine.py.321: eval: epoch 54 | batch 320/510 [I 2022-10-27 07:24:24] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 07:24:24] ...trap/engines/engine.py.326: process: 0.28916 | load: 0.00014 [I 2022-10-27 07:24:27] ...trap/engines/engine.py.321: eval: epoch 54 | batch 330/510 [I 2022-10-27 07:24:27] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 07:24:27] ...trap/engines/engine.py.326: process: 0.28936 | load: 0.00015 [I 2022-10-27 07:24:30] ...trap/engines/engine.py.321: eval: epoch 54 | batch 340/510 [I 2022-10-27 07:24:30] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-27 07:24:30] ...trap/engines/engine.py.326: process: 0.28496 | load: 0.00015 [I 2022-10-27 07:24:33] ...trap/engines/engine.py.321: eval: epoch 54 | batch 350/510 [I 2022-10-27 07:24:33] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-27 07:24:33] ...trap/engines/engine.py.326: process: 0.28585 | load: 0.00018 [I 2022-10-27 07:24:35] ...trap/engines/engine.py.321: eval: epoch 54 | batch 360/510 [I 2022-10-27 07:24:35] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 07:24:35] ...trap/engines/engine.py.326: process: 0.29092 | load: 0.00014 [I 2022-10-27 07:24:38] ...trap/engines/engine.py.321: eval: epoch 54 | batch 370/510 [I 2022-10-27 07:24:38] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 07:24:38] ...trap/engines/engine.py.326: process: 0.29166 | load: 0.00027 [I 2022-10-27 07:24:41] ...trap/engines/engine.py.321: eval: epoch 54 | batch 380/510 [I 2022-10-27 07:24:41] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 07:24:41] ...trap/engines/engine.py.326: process: 0.28995 | load: 0.00017 [I 2022-10-27 07:24:44] ...trap/engines/engine.py.321: eval: epoch 54 | batch 390/510 [I 2022-10-27 07:24:44] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 07:24:44] ...trap/engines/engine.py.326: process: 0.28728 | load: 0.00024 [I 2022-10-27 07:24:47] ...trap/engines/engine.py.321: eval: epoch 54 | batch 400/510 [I 2022-10-27 07:24:47] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 07:24:47] ...trap/engines/engine.py.326: process: 0.28865 | load: 0.00015 [I 2022-10-27 07:24:50] ...trap/engines/engine.py.321: eval: epoch 54 | batch 410/510 [I 2022-10-27 07:24:50] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 07:24:50] ...trap/engines/engine.py.326: process: 0.28991 | load: 0.00018 [I 2022-10-27 07:24:53] ...trap/engines/engine.py.321: eval: epoch 54 | batch 420/510 [I 2022-10-27 07:24:53] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-27 07:24:53] ...trap/engines/engine.py.326: process: 0.28660 | load: 0.00014 [I 2022-10-27 07:24:56] ...trap/engines/engine.py.321: eval: epoch 54 | batch 430/510 [I 2022-10-27 07:24:56] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:22 [I 2022-10-27 07:24:56] ...trap/engines/engine.py.326: process: 0.27918 | load: 0.00021 [I 2022-10-27 07:24:58] ...trap/engines/engine.py.321: eval: epoch 54 | batch 440/510 [I 2022-10-27 07:24:58] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 07:24:58] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00017 [I 2022-10-27 07:25:01] ...trap/engines/engine.py.321: eval: epoch 54 | batch 450/510 [I 2022-10-27 07:25:01] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 07:25:01] ...trap/engines/engine.py.326: process: 0.28503 | load: 0.00021 [I 2022-10-27 07:25:04] ...trap/engines/engine.py.321: eval: epoch 54 | batch 460/510 [I 2022-10-27 07:25:04] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 07:25:04] ...trap/engines/engine.py.326: process: 0.28880 | load: 0.00021 [I 2022-10-27 07:25:07] ...trap/engines/engine.py.321: eval: epoch 54 | batch 470/510 [I 2022-10-27 07:25:07] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 07:25:07] ...trap/engines/engine.py.326: process: 0.28583 | load: 0.00022 [I 2022-10-27 07:25:10] ...trap/engines/engine.py.321: eval: epoch 54 | batch 480/510 [I 2022-10-27 07:25:10] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 07:25:10] ...trap/engines/engine.py.326: process: 0.28732 | load: 0.00020 [I 2022-10-27 07:25:13] ...trap/engines/engine.py.321: eval: epoch 54 | batch 490/510 [I 2022-10-27 07:25:13] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 07:25:13] ...trap/engines/engine.py.326: process: 0.29997 | load: 0.00016 [I 2022-10-27 07:25:16] ...trap/engines/engine.py.321: eval: epoch 54 | batch 500/510 [I 2022-10-27 07:25:16] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 07:25:16] ...trap/engines/engine.py.326: process: 0.28749 | load: 0.00015 [I 2022-10-27 07:25:19] ...trap/engines/engine.py.321: eval: epoch 54 | batch 510/510 [I 2022-10-27 07:25:19] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 07:25:19] ...trap/engines/engine.py.326: process: 0.28581 | load: 0.00011 [S 2022-10-27 07:25:19] ...trap/engines/engine.py.132: eval_epoch.epoch: 54 [I 2022-10-27 07:25:19] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.701700 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.898500 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933900 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.692800 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.897700 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931000 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017911 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005766 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006670 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015289 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011992 [S 2022-10-27 07:25:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008112 [I 2022-10-27 07:25:25] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 07:25:25] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 07:25:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 07:25:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 07:25:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 07:25:28] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 07:25:29] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 07:25:35] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 07:25:35] ...trap/engines/engine.py.163: Training model on trainset for epoch 55 [I 2022-10-27 07:25:36] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 07:25:38] ...trap/engines/engine.py.224: train: epoch 55 | batch 0/2304 [I 2022-10-27 07:25:38] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:04 [I 2022-10-27 07:25:38] ...trap/engines/engine.py.229: process: 2.65810 | load: 2.00927 [I 2022-10-27 07:25:38] ...trap/engines/engine.py.230: loss: 0.13393 [I 2022-10-27 07:25:38] ...optimizers/trijoint.py.151: total_norm: 13.737626 [I 2022-10-27 07:25:44] ...trap/engines/engine.py.224: train: epoch 55 | batch 10/2304 [I 2022-10-27 07:25:44] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:54 [I 2022-10-27 07:25:44] ...trap/engines/engine.py.229: process: 0.56279 | load: 0.00547 [I 2022-10-27 07:25:44] ...trap/engines/engine.py.230: loss: 0.19141 [I 2022-10-27 07:25:44] ...optimizers/trijoint.py.151: total_norm: 18.687790 [I 2022-10-27 07:26:09] ...trap/engines/engine.py.224: train: epoch 55 | batch 20/2304 [I 2022-10-27 07:26:09] ...trap/engines/engine.py.228: elapsed: 0:00:33 | left: 0:41:47 [I 2022-10-27 07:26:09] ...trap/engines/engine.py.229: process: 0.55688 | load: 0.00348 [I 2022-10-27 07:26:09] ...trap/engines/engine.py.230: loss: 0.13685 [I 2022-10-27 07:26:09] ...optimizers/trijoint.py.151: total_norm: 9.419835 [I 2022-10-27 07:26:14] ...trap/engines/engine.py.224: train: epoch 55 | batch 30/2304 [I 2022-10-27 07:26:14] ...trap/engines/engine.py.228: elapsed: 0:00:38 | left: 0:23:31 [I 2022-10-27 07:26:14] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00301 [I 2022-10-27 07:26:14] ...trap/engines/engine.py.230: loss: 0.16560 [I 2022-10-27 07:26:14] ...optimizers/trijoint.py.151: total_norm: 12.607836 [I 2022-10-27 07:26:20] ...trap/engines/engine.py.224: train: epoch 55 | batch 40/2304 [I 2022-10-27 07:26:20] ...trap/engines/engine.py.228: elapsed: 0:00:44 | left: 0:21:14 [I 2022-10-27 07:26:20] ...trap/engines/engine.py.229: process: 0.56817 | load: 0.00318 [I 2022-10-27 07:26:20] ...trap/engines/engine.py.230: loss: 0.15885 [I 2022-10-27 07:26:20] ...optimizers/trijoint.py.151: total_norm: 13.656840 [I 2022-10-27 07:26:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 50/2304 [I 2022-10-27 07:26:25] ...trap/engines/engine.py.228: elapsed: 0:00:49 | left: 0:20:59 [I 2022-10-27 07:26:25] ...trap/engines/engine.py.229: process: 0.55356 | load: 0.00289 [I 2022-10-27 07:26:25] ...trap/engines/engine.py.230: loss: 0.13997 [I 2022-10-27 07:26:25] ...optimizers/trijoint.py.151: total_norm: 17.970432 [I 2022-10-27 07:26:31] ...trap/engines/engine.py.224: train: epoch 55 | batch 60/2304 [I 2022-10-27 07:26:31] ...trap/engines/engine.py.228: elapsed: 0:00:55 | left: 0:20:54 [I 2022-10-27 07:26:31] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00279 [I 2022-10-27 07:26:31] ...trap/engines/engine.py.230: loss: 0.12317 [I 2022-10-27 07:26:31] ...optimizers/trijoint.py.151: total_norm: 15.269113 [I 2022-10-27 07:26:37] ...trap/engines/engine.py.224: train: epoch 55 | batch 70/2304 [I 2022-10-27 07:26:37] ...trap/engines/engine.py.228: elapsed: 0:01:01 | left: 0:20:58 [I 2022-10-27 07:26:37] ...trap/engines/engine.py.229: process: 0.57378 | load: 0.00283 [I 2022-10-27 07:26:37] ...trap/engines/engine.py.230: loss: 0.11762 [I 2022-10-27 07:26:37] ...optimizers/trijoint.py.151: total_norm: 13.561672 [I 2022-10-27 07:26:42] ...trap/engines/engine.py.224: train: epoch 55 | batch 80/2304 [I 2022-10-27 07:26:42] ...trap/engines/engine.py.228: elapsed: 0:01:06 | left: 0:20:55 [I 2022-10-27 07:26:42] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00296 [I 2022-10-27 07:26:42] ...trap/engines/engine.py.230: loss: 0.20408 [I 2022-10-27 07:26:42] ...optimizers/trijoint.py.151: total_norm: 54.547077 [I 2022-10-27 07:26:48] ...trap/engines/engine.py.224: train: epoch 55 | batch 90/2304 [I 2022-10-27 07:26:48] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 0:20:49 [I 2022-10-27 07:26:48] ...trap/engines/engine.py.229: process: 0.57428 | load: 0.00271 [I 2022-10-27 07:26:48] ...trap/engines/engine.py.230: loss: 0.11179 [I 2022-10-27 07:26:48] ...optimizers/trijoint.py.151: total_norm: 10.139607 [I 2022-10-27 07:26:54] ...trap/engines/engine.py.224: train: epoch 55 | batch 100/2304 [I 2022-10-27 07:26:54] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 0:21:04 [I 2022-10-27 07:26:54] ...trap/engines/engine.py.229: process: 0.63266 | load: 0.00253 [I 2022-10-27 07:26:54] ...trap/engines/engine.py.230: loss: 0.13161 [I 2022-10-27 07:26:54] ...optimizers/trijoint.py.151: total_norm: 10.479019 [I 2022-10-27 07:26:59] ...trap/engines/engine.py.224: train: epoch 55 | batch 110/2304 [I 2022-10-27 07:26:59] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:20:37 [I 2022-10-27 07:26:59] ...trap/engines/engine.py.229: process: 0.56458 | load: 0.00242 [I 2022-10-27 07:26:59] ...trap/engines/engine.py.230: loss: 0.18240 [I 2022-10-27 07:26:59] ...optimizers/trijoint.py.151: total_norm: 28.442989 [I 2022-10-27 07:27:05] ...trap/engines/engine.py.224: train: epoch 55 | batch 120/2304 [I 2022-10-27 07:27:05] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:20:21 [I 2022-10-27 07:27:05] ...trap/engines/engine.py.229: process: 0.55152 | load: 0.00266 [I 2022-10-27 07:27:05] ...trap/engines/engine.py.230: loss: 0.13946 [I 2022-10-27 07:27:05] ...optimizers/trijoint.py.151: total_norm: 12.736728 [I 2022-10-27 07:27:10] ...trap/engines/engine.py.224: train: epoch 55 | batch 130/2304 [I 2022-10-27 07:27:10] ...trap/engines/engine.py.228: elapsed: 0:01:34 | left: 0:20:18 [I 2022-10-27 07:27:10] ...trap/engines/engine.py.229: process: 0.56659 | load: 0.00262 [I 2022-10-27 07:27:10] ...trap/engines/engine.py.230: loss: 0.15366 [I 2022-10-27 07:27:10] ...optimizers/trijoint.py.151: total_norm: 12.160885 [I 2022-10-27 07:27:16] ...trap/engines/engine.py.224: train: epoch 55 | batch 140/2304 [I 2022-10-27 07:27:16] ...trap/engines/engine.py.228: elapsed: 0:01:40 | left: 0:20:12 [I 2022-10-27 07:27:16] ...trap/engines/engine.py.229: process: 0.55611 | load: 0.00267 [I 2022-10-27 07:27:16] ...trap/engines/engine.py.230: loss: 0.19659 [I 2022-10-27 07:27:16] ...optimizers/trijoint.py.151: total_norm: 18.904028 [I 2022-10-27 07:27:22] ...trap/engines/engine.py.224: train: epoch 55 | batch 150/2304 [I 2022-10-27 07:27:22] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:20:01 [I 2022-10-27 07:27:22] ...trap/engines/engine.py.229: process: 0.55305 | load: 0.00241 [I 2022-10-27 07:27:22] ...trap/engines/engine.py.230: loss: 0.12926 [I 2022-10-27 07:27:22] ...optimizers/trijoint.py.151: total_norm: 11.167348 [I 2022-10-27 07:27:27] ...trap/engines/engine.py.224: train: epoch 55 | batch 160/2304 [I 2022-10-27 07:27:27] ...trap/engines/engine.py.228: elapsed: 0:01:51 | left: 0:19:52 [I 2022-10-27 07:27:27] ...trap/engines/engine.py.229: process: 0.55597 | load: 0.00248 [I 2022-10-27 07:27:27] ...trap/engines/engine.py.230: loss: 0.22145 [I 2022-10-27 07:27:27] ...optimizers/trijoint.py.151: total_norm: 15.715934 [I 2022-10-27 07:27:33] ...trap/engines/engine.py.224: train: epoch 55 | batch 170/2304 [I 2022-10-27 07:27:33] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:20:06 [I 2022-10-27 07:27:33] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00271 [I 2022-10-27 07:27:33] ...trap/engines/engine.py.230: loss: 0.24635 [I 2022-10-27 07:27:33] ...optimizers/trijoint.py.151: total_norm: 34.988400 [I 2022-10-27 07:27:39] ...trap/engines/engine.py.224: train: epoch 55 | batch 180/2304 [I 2022-10-27 07:27:39] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:19:46 [I 2022-10-27 07:27:39] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00246 [I 2022-10-27 07:27:39] ...trap/engines/engine.py.230: loss: 0.13205 [I 2022-10-27 07:27:39] ...optimizers/trijoint.py.151: total_norm: 14.000471 [I 2022-10-27 07:27:44] ...trap/engines/engine.py.224: train: epoch 55 | batch 190/2304 [I 2022-10-27 07:27:44] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:19:38 [I 2022-10-27 07:27:44] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00238 [I 2022-10-27 07:27:44] ...trap/engines/engine.py.230: loss: 0.20720 [I 2022-10-27 07:27:44] ...optimizers/trijoint.py.151: total_norm: 20.777164 [I 2022-10-27 07:27:50] ...trap/engines/engine.py.224: train: epoch 55 | batch 200/2304 [I 2022-10-27 07:27:50] ...trap/engines/engine.py.228: elapsed: 0:02:14 | left: 0:19:37 [I 2022-10-27 07:27:50] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00254 [I 2022-10-27 07:27:50] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-27 07:27:50] ...optimizers/trijoint.py.151: total_norm: 32.392899 [I 2022-10-27 07:27:55] ...trap/engines/engine.py.224: train: epoch 55 | batch 210/2304 [I 2022-10-27 07:27:55] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:19:25 [I 2022-10-27 07:27:55] ...trap/engines/engine.py.229: process: 0.57256 | load: 0.00237 [I 2022-10-27 07:27:55] ...trap/engines/engine.py.230: loss: 0.21109 [I 2022-10-27 07:27:55] ...optimizers/trijoint.py.151: total_norm: 19.037733 [I 2022-10-27 07:28:01] ...trap/engines/engine.py.224: train: epoch 55 | batch 220/2304 [I 2022-10-27 07:28:01] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:19:22 [I 2022-10-27 07:28:01] ...trap/engines/engine.py.229: process: 0.55521 | load: 0.00247 [I 2022-10-27 07:28:01] ...trap/engines/engine.py.230: loss: 0.18866 [I 2022-10-27 07:28:01] ...optimizers/trijoint.py.151: total_norm: 16.784740 [I 2022-10-27 07:28:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 230/2304 [I 2022-10-27 07:28:06] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:19:10 [I 2022-10-27 07:28:06] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00263 [I 2022-10-27 07:28:06] ...trap/engines/engine.py.230: loss: 0.14721 [I 2022-10-27 07:28:06] ...optimizers/trijoint.py.151: total_norm: 9.051870 [I 2022-10-27 07:28:12] ...trap/engines/engine.py.224: train: epoch 55 | batch 240/2304 [I 2022-10-27 07:28:12] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:19:06 [I 2022-10-27 07:28:12] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00263 [I 2022-10-27 07:28:12] ...trap/engines/engine.py.230: loss: 0.15742 [I 2022-10-27 07:28:12] ...optimizers/trijoint.py.151: total_norm: 18.580032 [I 2022-10-27 07:28:18] ...trap/engines/engine.py.224: train: epoch 55 | batch 250/2304 [I 2022-10-27 07:28:18] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:18:59 [I 2022-10-27 07:28:18] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00240 [I 2022-10-27 07:28:18] ...trap/engines/engine.py.230: loss: 0.14393 [I 2022-10-27 07:28:18] ...optimizers/trijoint.py.151: total_norm: 26.357895 [I 2022-10-27 07:28:23] ...trap/engines/engine.py.224: train: epoch 55 | batch 260/2304 [I 2022-10-27 07:28:23] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:19:38 [I 2022-10-27 07:28:23] ...trap/engines/engine.py.229: process: 0.55030 | load: 0.00241 [I 2022-10-27 07:28:23] ...trap/engines/engine.py.230: loss: 0.14588 [I 2022-10-27 07:28:23] ...optimizers/trijoint.py.151: total_norm: 13.319446 [I 2022-10-27 07:28:29] ...trap/engines/engine.py.224: train: epoch 55 | batch 270/2304 [I 2022-10-27 07:28:29] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:18:50 [I 2022-10-27 07:28:29] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00250 [I 2022-10-27 07:28:29] ...trap/engines/engine.py.230: loss: 0.22018 [I 2022-10-27 07:28:29] ...optimizers/trijoint.py.151: total_norm: 15.077162 [I 2022-10-27 07:28:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 280/2304 [I 2022-10-27 07:28:34] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:18:49 [I 2022-10-27 07:28:34] ...trap/engines/engine.py.229: process: 0.56683 | load: 0.00253 [I 2022-10-27 07:28:34] ...trap/engines/engine.py.230: loss: 0.16313 [I 2022-10-27 07:28:34] ...optimizers/trijoint.py.151: total_norm: 14.646218 [I 2022-10-27 07:28:40] ...trap/engines/engine.py.224: train: epoch 55 | batch 290/2304 [I 2022-10-27 07:28:40] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:18:36 [I 2022-10-27 07:28:40] ...trap/engines/engine.py.229: process: 0.55025 | load: 0.00248 [I 2022-10-27 07:28:40] ...trap/engines/engine.py.230: loss: 0.14679 [I 2022-10-27 07:28:40] ...optimizers/trijoint.py.151: total_norm: 31.753807 [I 2022-10-27 07:28:46] ...trap/engines/engine.py.224: train: epoch 55 | batch 300/2304 [I 2022-10-27 07:28:46] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:18:31 [I 2022-10-27 07:28:46] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00240 [I 2022-10-27 07:28:46] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-27 07:28:46] ...optimizers/trijoint.py.151: total_norm: 24.098042 [I 2022-10-27 07:28:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 310/2304 [I 2022-10-27 07:28:51] ...trap/engines/engine.py.228: elapsed: 0:03:15 | left: 0:18:28 [I 2022-10-27 07:28:51] ...trap/engines/engine.py.229: process: 0.56485 | load: 0.00241 [I 2022-10-27 07:28:51] ...trap/engines/engine.py.230: loss: 0.20566 [I 2022-10-27 07:28:51] ...optimizers/trijoint.py.151: total_norm: 18.421820 [I 2022-10-27 07:28:57] ...trap/engines/engine.py.224: train: epoch 55 | batch 320/2304 [I 2022-10-27 07:28:57] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:18:25 [I 2022-10-27 07:28:57] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00243 [I 2022-10-27 07:28:57] ...trap/engines/engine.py.230: loss: 0.09483 [I 2022-10-27 07:28:57] ...optimizers/trijoint.py.151: total_norm: 11.317270 [I 2022-10-27 07:29:02] ...trap/engines/engine.py.224: train: epoch 55 | batch 330/2304 [I 2022-10-27 07:29:02] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:18:25 [I 2022-10-27 07:29:02] ...trap/engines/engine.py.229: process: 0.56480 | load: 0.00250 [I 2022-10-27 07:29:02] ...trap/engines/engine.py.230: loss: 0.25365 [I 2022-10-27 07:29:02] ...optimizers/trijoint.py.151: total_norm: 13.169198 [I 2022-10-27 07:29:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 340/2304 [I 2022-10-27 07:29:08] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:18:10 [I 2022-10-27 07:29:08] ...trap/engines/engine.py.229: process: 0.56265 | load: 0.00238 [I 2022-10-27 07:29:08] ...trap/engines/engine.py.230: loss: 0.16002 [I 2022-10-27 07:29:08] ...optimizers/trijoint.py.151: total_norm: 30.601997 [I 2022-10-27 07:29:13] ...trap/engines/engine.py.224: train: epoch 55 | batch 350/2304 [I 2022-10-27 07:29:13] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:18:00 [I 2022-10-27 07:29:13] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00245 [I 2022-10-27 07:29:13] ...trap/engines/engine.py.230: loss: 0.15449 [I 2022-10-27 07:29:13] ...optimizers/trijoint.py.151: total_norm: 15.308286 [I 2022-10-27 07:29:19] ...trap/engines/engine.py.224: train: epoch 55 | batch 360/2304 [I 2022-10-27 07:29:19] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:18:01 [I 2022-10-27 07:29:19] ...trap/engines/engine.py.229: process: 0.56329 | load: 0.00252 [I 2022-10-27 07:29:19] ...trap/engines/engine.py.230: loss: 0.09718 [I 2022-10-27 07:29:19] ...optimizers/trijoint.py.151: total_norm: 23.087069 [I 2022-10-27 07:29:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 370/2304 [I 2022-10-27 07:29:25] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:17:58 [I 2022-10-27 07:29:25] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00245 [I 2022-10-27 07:29:25] ...trap/engines/engine.py.230: loss: 0.10614 [I 2022-10-27 07:29:25] ...optimizers/trijoint.py.151: total_norm: 9.573545 [I 2022-10-27 07:29:30] ...trap/engines/engine.py.224: train: epoch 55 | batch 380/2304 [I 2022-10-27 07:29:30] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:17:52 [I 2022-10-27 07:29:30] ...trap/engines/engine.py.229: process: 0.55413 | load: 0.00252 [I 2022-10-27 07:29:30] ...trap/engines/engine.py.230: loss: 0.15302 [I 2022-10-27 07:29:30] ...optimizers/trijoint.py.151: total_norm: 33.007072 [I 2022-10-27 07:29:36] ...trap/engines/engine.py.224: train: epoch 55 | batch 390/2304 [I 2022-10-27 07:29:36] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:17:59 [I 2022-10-27 07:29:36] ...trap/engines/engine.py.229: process: 0.57675 | load: 0.00252 [I 2022-10-27 07:29:36] ...trap/engines/engine.py.230: loss: 0.14867 [I 2022-10-27 07:29:36] ...optimizers/trijoint.py.151: total_norm: 21.219784 [I 2022-10-27 07:29:42] ...trap/engines/engine.py.224: train: epoch 55 | batch 400/2304 [I 2022-10-27 07:29:42] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:17:48 [I 2022-10-27 07:29:42] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00243 [I 2022-10-27 07:29:42] ...trap/engines/engine.py.230: loss: 0.17099 [I 2022-10-27 07:29:42] ...optimizers/trijoint.py.151: total_norm: 15.102283 [I 2022-10-27 07:29:47] ...trap/engines/engine.py.224: train: epoch 55 | batch 410/2304 [I 2022-10-27 07:29:47] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:17:34 [I 2022-10-27 07:29:47] ...trap/engines/engine.py.229: process: 0.54463 | load: 0.00247 [I 2022-10-27 07:29:47] ...trap/engines/engine.py.230: loss: 0.13734 [I 2022-10-27 07:29:47] ...optimizers/trijoint.py.151: total_norm: 11.212281 [I 2022-10-27 07:29:53] ...trap/engines/engine.py.224: train: epoch 55 | batch 420/2304 [I 2022-10-27 07:29:53] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:17:30 [I 2022-10-27 07:29:53] ...trap/engines/engine.py.229: process: 0.55299 | load: 0.00248 [I 2022-10-27 07:29:53] ...trap/engines/engine.py.230: loss: 0.15937 [I 2022-10-27 07:29:53] ...optimizers/trijoint.py.151: total_norm: 13.332446 [I 2022-10-27 07:29:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 430/2304 [I 2022-10-27 07:29:58] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:17:33 [I 2022-10-27 07:29:58] ...trap/engines/engine.py.229: process: 0.56863 | load: 0.00243 [I 2022-10-27 07:29:58] ...trap/engines/engine.py.230: loss: 0.28515 [I 2022-10-27 07:29:58] ...optimizers/trijoint.py.151: total_norm: 23.445189 [I 2022-10-27 07:30:04] ...trap/engines/engine.py.224: train: epoch 55 | batch 440/2304 [I 2022-10-27 07:30:04] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:17:28 [I 2022-10-27 07:30:04] ...trap/engines/engine.py.229: process: 0.57261 | load: 0.00252 [I 2022-10-27 07:30:04] ...trap/engines/engine.py.230: loss: 0.13245 [I 2022-10-27 07:30:04] ...optimizers/trijoint.py.151: total_norm: 23.953339 [I 2022-10-27 07:30:10] ...trap/engines/engine.py.224: train: epoch 55 | batch 450/2304 [I 2022-10-27 07:30:10] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:17:32 [I 2022-10-27 07:30:10] ...trap/engines/engine.py.229: process: 0.57587 | load: 0.00278 [I 2022-10-27 07:30:10] ...trap/engines/engine.py.230: loss: 0.16998 [I 2022-10-27 07:30:10] ...optimizers/trijoint.py.151: total_norm: 17.908970 [I 2022-10-27 07:30:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 460/2304 [I 2022-10-27 07:30:15] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:17:21 [I 2022-10-27 07:30:15] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00255 [I 2022-10-27 07:30:15] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-27 07:30:15] ...optimizers/trijoint.py.151: total_norm: 7.783581 [I 2022-10-27 07:30:21] ...trap/engines/engine.py.224: train: epoch 55 | batch 470/2304 [I 2022-10-27 07:30:21] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:17:21 [I 2022-10-27 07:30:21] ...trap/engines/engine.py.229: process: 0.58176 | load: 0.00247 [I 2022-10-27 07:30:21] ...trap/engines/engine.py.230: loss: 0.17772 [I 2022-10-27 07:30:21] ...optimizers/trijoint.py.151: total_norm: 19.956835 [I 2022-10-27 07:30:27] ...trap/engines/engine.py.224: train: epoch 55 | batch 480/2304 [I 2022-10-27 07:30:27] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:17:20 [I 2022-10-27 07:30:27] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00250 [I 2022-10-27 07:30:27] ...trap/engines/engine.py.230: loss: 0.18607 [I 2022-10-27 07:30:27] ...optimizers/trijoint.py.151: total_norm: 12.275676 [I 2022-10-27 07:30:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 490/2304 [I 2022-10-27 07:30:32] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:17:09 [I 2022-10-27 07:30:32] ...trap/engines/engine.py.229: process: 0.58008 | load: 0.00281 [I 2022-10-27 07:30:32] ...trap/engines/engine.py.230: loss: 0.16158 [I 2022-10-27 07:30:32] ...optimizers/trijoint.py.151: total_norm: 17.537146 [I 2022-10-27 07:30:38] ...trap/engines/engine.py.224: train: epoch 55 | batch 500/2304 [I 2022-10-27 07:30:38] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:17:08 [I 2022-10-27 07:30:38] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00253 [I 2022-10-27 07:30:38] ...trap/engines/engine.py.230: loss: 0.19592 [I 2022-10-27 07:30:38] ...optimizers/trijoint.py.151: total_norm: 38.787502 [I 2022-10-27 07:30:44] ...trap/engines/engine.py.224: train: epoch 55 | batch 510/2304 [I 2022-10-27 07:30:44] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:16:53 [I 2022-10-27 07:30:44] ...trap/engines/engine.py.229: process: 0.56875 | load: 0.00275 [I 2022-10-27 07:30:44] ...trap/engines/engine.py.230: loss: 0.28188 [I 2022-10-27 07:30:44] ...optimizers/trijoint.py.151: total_norm: 31.671177 [I 2022-10-27 07:30:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 520/2304 [I 2022-10-27 07:30:49] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:16:40 [I 2022-10-27 07:30:49] ...trap/engines/engine.py.229: process: 0.54938 | load: 0.00244 [I 2022-10-27 07:30:49] ...trap/engines/engine.py.230: loss: 0.27302 [I 2022-10-27 07:30:49] ...optimizers/trijoint.py.151: total_norm: 30.603680 [I 2022-10-27 07:30:55] ...trap/engines/engine.py.224: train: epoch 55 | batch 530/2304 [I 2022-10-27 07:30:55] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:16:42 [I 2022-10-27 07:30:55] ...trap/engines/engine.py.229: process: 0.57724 | load: 0.00248 [I 2022-10-27 07:30:55] ...trap/engines/engine.py.230: loss: 0.09593 [I 2022-10-27 07:30:55] ...optimizers/trijoint.py.151: total_norm: 11.841958 [I 2022-10-27 07:31:01] ...trap/engines/engine.py.224: train: epoch 55 | batch 540/2304 [I 2022-10-27 07:31:01] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:16:26 [I 2022-10-27 07:31:01] ...trap/engines/engine.py.229: process: 0.55558 | load: 0.00250 [I 2022-10-27 07:31:01] ...trap/engines/engine.py.230: loss: 0.23974 [I 2022-10-27 07:31:01] ...optimizers/trijoint.py.151: total_norm: 30.598000 [I 2022-10-27 07:31:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 550/2304 [I 2022-10-27 07:31:06] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:16:24 [I 2022-10-27 07:31:06] ...trap/engines/engine.py.229: process: 0.56597 | load: 0.00315 [I 2022-10-27 07:31:06] ...trap/engines/engine.py.230: loss: 0.13997 [I 2022-10-27 07:31:06] ...optimizers/trijoint.py.151: total_norm: 27.210159 [I 2022-10-27 07:31:12] ...trap/engines/engine.py.224: train: epoch 55 | batch 560/2304 [I 2022-10-27 07:31:12] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:16:15 [I 2022-10-27 07:31:12] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00247 [I 2022-10-27 07:31:12] ...trap/engines/engine.py.230: loss: 0.13624 [I 2022-10-27 07:31:12] ...optimizers/trijoint.py.151: total_norm: 10.426721 [I 2022-10-27 07:31:18] ...trap/engines/engine.py.224: train: epoch 55 | batch 570/2304 [I 2022-10-27 07:31:18] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:16:07 [I 2022-10-27 07:31:18] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00244 [I 2022-10-27 07:31:18] ...trap/engines/engine.py.230: loss: 0.19223 [I 2022-10-27 07:31:18] ...optimizers/trijoint.py.151: total_norm: 27.133432 [I 2022-10-27 07:31:23] ...trap/engines/engine.py.224: train: epoch 55 | batch 580/2304 [I 2022-10-27 07:31:23] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:16:00 [I 2022-10-27 07:31:23] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00248 [I 2022-10-27 07:31:23] ...trap/engines/engine.py.230: loss: 0.20058 [I 2022-10-27 07:31:23] ...optimizers/trijoint.py.151: total_norm: 16.618206 [I 2022-10-27 07:31:29] ...trap/engines/engine.py.224: train: epoch 55 | batch 590/2304 [I 2022-10-27 07:31:29] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:15:58 [I 2022-10-27 07:31:29] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00253 [I 2022-10-27 07:31:29] ...trap/engines/engine.py.230: loss: 0.14967 [I 2022-10-27 07:31:29] ...optimizers/trijoint.py.151: total_norm: 17.310892 [I 2022-10-27 07:31:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 600/2304 [I 2022-10-27 07:31:34] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:53 [I 2022-10-27 07:31:34] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00251 [I 2022-10-27 07:31:34] ...trap/engines/engine.py.230: loss: 0.17404 [I 2022-10-27 07:31:34] ...optimizers/trijoint.py.151: total_norm: 24.253206 [I 2022-10-27 07:31:40] ...trap/engines/engine.py.224: train: epoch 55 | batch 610/2304 [I 2022-10-27 07:31:40] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:47 [I 2022-10-27 07:31:40] ...trap/engines/engine.py.229: process: 0.55360 | load: 0.00249 [I 2022-10-27 07:31:40] ...trap/engines/engine.py.230: loss: 0.22492 [I 2022-10-27 07:31:40] ...optimizers/trijoint.py.151: total_norm: 16.320251 [I 2022-10-27 07:31:46] ...trap/engines/engine.py.224: train: epoch 55 | batch 620/2304 [I 2022-10-27 07:31:46] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:15:46 [I 2022-10-27 07:31:46] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00249 [I 2022-10-27 07:31:46] ...trap/engines/engine.py.230: loss: 0.32504 [I 2022-10-27 07:31:46] ...optimizers/trijoint.py.151: total_norm: 19.060371 [I 2022-10-27 07:31:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 630/2304 [I 2022-10-27 07:31:51] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:37 [I 2022-10-27 07:31:51] ...trap/engines/engine.py.229: process: 0.56561 | load: 0.00245 [I 2022-10-27 07:31:51] ...trap/engines/engine.py.230: loss: 0.15589 [I 2022-10-27 07:31:51] ...optimizers/trijoint.py.151: total_norm: 8.321328 [I 2022-10-27 07:31:57] ...trap/engines/engine.py.224: train: epoch 55 | batch 640/2304 [I 2022-10-27 07:31:57] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:15:30 [I 2022-10-27 07:31:57] ...trap/engines/engine.py.229: process: 0.55164 | load: 0.00242 [I 2022-10-27 07:31:57] ...trap/engines/engine.py.230: loss: 0.14116 [I 2022-10-27 07:31:57] ...optimizers/trijoint.py.151: total_norm: 14.341396 [I 2022-10-27 07:32:02] ...trap/engines/engine.py.224: train: epoch 55 | batch 650/2304 [I 2022-10-27 07:32:02] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:36 [I 2022-10-27 07:32:02] ...trap/engines/engine.py.229: process: 0.58819 | load: 0.00247 [I 2022-10-27 07:32:02] ...trap/engines/engine.py.230: loss: 0.20904 [I 2022-10-27 07:32:02] ...optimizers/trijoint.py.151: total_norm: 15.833968 [I 2022-10-27 07:32:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 660/2304 [I 2022-10-27 07:32:08] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:15:35 [I 2022-10-27 07:32:08] ...trap/engines/engine.py.229: process: 0.57653 | load: 0.00256 [I 2022-10-27 07:32:08] ...trap/engines/engine.py.230: loss: 0.14960 [I 2022-10-27 07:32:08] ...optimizers/trijoint.py.151: total_norm: 9.079539 [I 2022-10-27 07:32:14] ...trap/engines/engine.py.224: train: epoch 55 | batch 670/2304 [I 2022-10-27 07:32:14] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:15:32 [I 2022-10-27 07:32:14] ...trap/engines/engine.py.229: process: 0.58109 | load: 0.00250 [I 2022-10-27 07:32:14] ...trap/engines/engine.py.230: loss: 0.16842 [I 2022-10-27 07:32:14] ...optimizers/trijoint.py.151: total_norm: 13.803220 [I 2022-10-27 07:32:19] ...trap/engines/engine.py.224: train: epoch 55 | batch 680/2304 [I 2022-10-27 07:32:19] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:12 [I 2022-10-27 07:32:19] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00242 [I 2022-10-27 07:32:19] ...trap/engines/engine.py.230: loss: 0.18179 [I 2022-10-27 07:32:19] ...optimizers/trijoint.py.151: total_norm: 26.616005 [I 2022-10-27 07:32:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 690/2304 [I 2022-10-27 07:32:25] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:15:08 [I 2022-10-27 07:32:25] ...trap/engines/engine.py.229: process: 0.58047 | load: 0.00247 [I 2022-10-27 07:32:25] ...trap/engines/engine.py.230: loss: 0.19529 [I 2022-10-27 07:32:25] ...optimizers/trijoint.py.151: total_norm: 24.096514 [I 2022-10-27 07:32:31] ...trap/engines/engine.py.224: train: epoch 55 | batch 700/2304 [I 2022-10-27 07:32:31] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:58 [I 2022-10-27 07:32:31] ...trap/engines/engine.py.229: process: 0.57346 | load: 0.00245 [I 2022-10-27 07:32:31] ...trap/engines/engine.py.230: loss: 0.09011 [I 2022-10-27 07:32:31] ...optimizers/trijoint.py.151: total_norm: 12.091787 [I 2022-10-27 07:32:36] ...trap/engines/engine.py.224: train: epoch 55 | batch 710/2304 [I 2022-10-27 07:32:36] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:54 [I 2022-10-27 07:32:36] ...trap/engines/engine.py.229: process: 0.56023 | load: 0.00248 [I 2022-10-27 07:32:36] ...trap/engines/engine.py.230: loss: 0.13229 [I 2022-10-27 07:32:36] ...optimizers/trijoint.py.151: total_norm: 12.219403 [I 2022-10-27 07:32:42] ...trap/engines/engine.py.224: train: epoch 55 | batch 720/2304 [I 2022-10-27 07:32:42] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:56 [I 2022-10-27 07:32:42] ...trap/engines/engine.py.229: process: 0.57461 | load: 0.00246 [I 2022-10-27 07:32:42] ...trap/engines/engine.py.230: loss: 0.16366 [I 2022-10-27 07:32:42] ...optimizers/trijoint.py.151: total_norm: 26.963980 [I 2022-10-27 07:32:47] ...trap/engines/engine.py.224: train: epoch 55 | batch 730/2304 [I 2022-10-27 07:32:47] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:36 [I 2022-10-27 07:32:47] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00249 [I 2022-10-27 07:32:47] ...trap/engines/engine.py.230: loss: 0.10949 [I 2022-10-27 07:32:47] ...optimizers/trijoint.py.151: total_norm: 10.241094 [I 2022-10-27 07:32:53] ...trap/engines/engine.py.224: train: epoch 55 | batch 740/2304 [I 2022-10-27 07:32:53] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:14:34 [I 2022-10-27 07:32:53] ...trap/engines/engine.py.229: process: 0.56959 | load: 0.00247 [I 2022-10-27 07:32:53] ...trap/engines/engine.py.230: loss: 0.19816 [I 2022-10-27 07:32:53] ...optimizers/trijoint.py.151: total_norm: 27.451010 [I 2022-10-27 07:32:59] ...trap/engines/engine.py.224: train: epoch 55 | batch 750/2304 [I 2022-10-27 07:32:59] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:33 [I 2022-10-27 07:32:59] ...trap/engines/engine.py.229: process: 0.56425 | load: 0.00244 [I 2022-10-27 07:32:59] ...trap/engines/engine.py.230: loss: 0.13672 [I 2022-10-27 07:32:59] ...optimizers/trijoint.py.151: total_norm: 12.626888 [I 2022-10-27 07:33:04] ...trap/engines/engine.py.224: train: epoch 55 | batch 760/2304 [I 2022-10-27 07:33:04] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:24 [I 2022-10-27 07:33:04] ...trap/engines/engine.py.229: process: 0.56225 | load: 0.00246 [I 2022-10-27 07:33:04] ...trap/engines/engine.py.230: loss: 0.14289 [I 2022-10-27 07:33:04] ...optimizers/trijoint.py.151: total_norm: 32.360191 [I 2022-10-27 07:33:10] ...trap/engines/engine.py.224: train: epoch 55 | batch 770/2304 [I 2022-10-27 07:33:10] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:14:19 [I 2022-10-27 07:33:10] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00247 [I 2022-10-27 07:33:10] ...trap/engines/engine.py.230: loss: 0.16372 [I 2022-10-27 07:33:10] ...optimizers/trijoint.py.151: total_norm: 21.570614 [I 2022-10-27 07:33:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 780/2304 [I 2022-10-27 07:33:15] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:16 [I 2022-10-27 07:33:15] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00247 [I 2022-10-27 07:33:15] ...trap/engines/engine.py.230: loss: 0.14675 [I 2022-10-27 07:33:15] ...optimizers/trijoint.py.151: total_norm: 20.809681 [I 2022-10-27 07:33:21] ...trap/engines/engine.py.224: train: epoch 55 | batch 790/2304 [I 2022-10-27 07:33:21] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:14:10 [I 2022-10-27 07:33:21] ...trap/engines/engine.py.229: process: 0.56419 | load: 0.00249 [I 2022-10-27 07:33:21] ...trap/engines/engine.py.230: loss: 0.13814 [I 2022-10-27 07:33:21] ...optimizers/trijoint.py.151: total_norm: 21.128527 [I 2022-10-27 07:33:27] ...trap/engines/engine.py.224: train: epoch 55 | batch 800/2304 [I 2022-10-27 07:33:27] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:14:02 [I 2022-10-27 07:33:27] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00260 [I 2022-10-27 07:33:27] ...trap/engines/engine.py.230: loss: 0.16246 [I 2022-10-27 07:33:27] ...optimizers/trijoint.py.151: total_norm: 17.659491 [I 2022-10-27 07:33:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 810/2304 [I 2022-10-27 07:33:32] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:55 [I 2022-10-27 07:33:32] ...trap/engines/engine.py.229: process: 0.54736 | load: 0.00251 [I 2022-10-27 07:33:32] ...trap/engines/engine.py.230: loss: 0.16000 [I 2022-10-27 07:33:32] ...optimizers/trijoint.py.151: total_norm: 10.555514 [I 2022-10-27 07:33:38] ...trap/engines/engine.py.224: train: epoch 55 | batch 820/2304 [I 2022-10-27 07:33:38] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:53 [I 2022-10-27 07:33:38] ...trap/engines/engine.py.229: process: 0.55346 | load: 0.00245 [I 2022-10-27 07:33:38] ...trap/engines/engine.py.230: loss: 0.12193 [I 2022-10-27 07:33:38] ...optimizers/trijoint.py.151: total_norm: 14.047109 [I 2022-10-27 07:33:44] ...trap/engines/engine.py.224: train: epoch 55 | batch 830/2304 [I 2022-10-27 07:33:44] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:50 [I 2022-10-27 07:33:44] ...trap/engines/engine.py.229: process: 0.56817 | load: 0.00243 [I 2022-10-27 07:33:44] ...trap/engines/engine.py.230: loss: 0.17530 [I 2022-10-27 07:33:44] ...optimizers/trijoint.py.151: total_norm: 11.947896 [I 2022-10-27 07:33:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 840/2304 [I 2022-10-27 07:33:49] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:35 [I 2022-10-27 07:33:49] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00244 [I 2022-10-27 07:33:49] ...trap/engines/engine.py.230: loss: 0.11606 [I 2022-10-27 07:33:49] ...optimizers/trijoint.py.151: total_norm: 19.158165 [I 2022-10-27 07:33:55] ...trap/engines/engine.py.224: train: epoch 55 | batch 850/2304 [I 2022-10-27 07:33:55] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:33 [I 2022-10-27 07:33:55] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00270 [I 2022-10-27 07:33:55] ...trap/engines/engine.py.230: loss: 0.16489 [I 2022-10-27 07:33:55] ...optimizers/trijoint.py.151: total_norm: 24.649324 [I 2022-10-27 07:34:01] ...trap/engines/engine.py.224: train: epoch 55 | batch 860/2304 [I 2022-10-27 07:34:01] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:33 [I 2022-10-27 07:34:01] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00246 [I 2022-10-27 07:34:01] ...trap/engines/engine.py.230: loss: 0.18021 [I 2022-10-27 07:34:01] ...optimizers/trijoint.py.151: total_norm: 14.435165 [I 2022-10-27 07:34:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 870/2304 [I 2022-10-27 07:34:06] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:13:28 [I 2022-10-27 07:34:06] ...trap/engines/engine.py.229: process: 0.57071 | load: 0.00252 [I 2022-10-27 07:34:06] ...trap/engines/engine.py.230: loss: 0.12631 [I 2022-10-27 07:34:06] ...optimizers/trijoint.py.151: total_norm: 11.436387 [I 2022-10-27 07:34:12] ...trap/engines/engine.py.224: train: epoch 55 | batch 880/2304 [I 2022-10-27 07:34:12] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:13:16 [I 2022-10-27 07:34:12] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00258 [I 2022-10-27 07:34:12] ...trap/engines/engine.py.230: loss: 0.15711 [I 2022-10-27 07:34:12] ...optimizers/trijoint.py.151: total_norm: 23.274734 [I 2022-10-27 07:34:17] ...trap/engines/engine.py.224: train: epoch 55 | batch 890/2304 [I 2022-10-27 07:34:17] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:13:12 [I 2022-10-27 07:34:17] ...trap/engines/engine.py.229: process: 0.56500 | load: 0.00280 [I 2022-10-27 07:34:17] ...trap/engines/engine.py.230: loss: 0.12530 [I 2022-10-27 07:34:17] ...optimizers/trijoint.py.151: total_norm: 13.908479 [I 2022-10-27 07:34:23] ...trap/engines/engine.py.224: train: epoch 55 | batch 900/2304 [I 2022-10-27 07:34:23] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:59 [I 2022-10-27 07:34:23] ...trap/engines/engine.py.229: process: 0.55065 | load: 0.00265 [I 2022-10-27 07:34:23] ...trap/engines/engine.py.230: loss: 0.16594 [I 2022-10-27 07:34:23] ...optimizers/trijoint.py.151: total_norm: 26.259483 [I 2022-10-27 07:34:29] ...trap/engines/engine.py.224: train: epoch 55 | batch 910/2304 [I 2022-10-27 07:34:29] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:13:03 [I 2022-10-27 07:34:29] ...trap/engines/engine.py.229: process: 0.56625 | load: 0.00241 [I 2022-10-27 07:34:29] ...trap/engines/engine.py.230: loss: 0.16463 [I 2022-10-27 07:34:29] ...optimizers/trijoint.py.151: total_norm: 13.176600 [I 2022-10-27 07:34:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 920/2304 [I 2022-10-27 07:34:34] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:13:02 [I 2022-10-27 07:34:34] ...trap/engines/engine.py.229: process: 0.57435 | load: 0.00241 [I 2022-10-27 07:34:34] ...trap/engines/engine.py.230: loss: 0.17447 [I 2022-10-27 07:34:34] ...optimizers/trijoint.py.151: total_norm: 15.205895 [I 2022-10-27 07:34:40] ...trap/engines/engine.py.224: train: epoch 55 | batch 930/2304 [I 2022-10-27 07:34:40] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:13:01 [I 2022-10-27 07:34:40] ...trap/engines/engine.py.229: process: 0.56913 | load: 0.00253 [I 2022-10-27 07:34:40] ...trap/engines/engine.py.230: loss: 0.13223 [I 2022-10-27 07:34:40] ...optimizers/trijoint.py.151: total_norm: 13.470860 [I 2022-10-27 07:34:46] ...trap/engines/engine.py.224: train: epoch 55 | batch 940/2304 [I 2022-10-27 07:34:46] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:42 [I 2022-10-27 07:34:46] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00251 [I 2022-10-27 07:34:46] ...trap/engines/engine.py.230: loss: 0.17072 [I 2022-10-27 07:34:46] ...optimizers/trijoint.py.151: total_norm: 15.892921 [I 2022-10-27 07:34:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 950/2304 [I 2022-10-27 07:34:51] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:41 [I 2022-10-27 07:34:51] ...trap/engines/engine.py.229: process: 0.56492 | load: 0.00243 [I 2022-10-27 07:34:51] ...trap/engines/engine.py.230: loss: 0.11370 [I 2022-10-27 07:34:51] ...optimizers/trijoint.py.151: total_norm: 10.690049 [I 2022-10-27 07:34:57] ...trap/engines/engine.py.224: train: epoch 55 | batch 960/2304 [I 2022-10-27 07:34:57] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:29 [I 2022-10-27 07:34:57] ...trap/engines/engine.py.229: process: 0.54710 | load: 0.00250 [I 2022-10-27 07:34:57] ...trap/engines/engine.py.230: loss: 0.24917 [I 2022-10-27 07:34:57] ...optimizers/trijoint.py.151: total_norm: 15.195395 [I 2022-10-27 07:35:02] ...trap/engines/engine.py.224: train: epoch 55 | batch 970/2304 [I 2022-10-27 07:35:02] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:25 [I 2022-10-27 07:35:02] ...trap/engines/engine.py.229: process: 0.56546 | load: 0.00245 [I 2022-10-27 07:35:02] ...trap/engines/engine.py.230: loss: 0.13301 [I 2022-10-27 07:35:02] ...optimizers/trijoint.py.151: total_norm: 12.167760 [I 2022-10-27 07:35:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 980/2304 [I 2022-10-27 07:35:08] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:12:23 [I 2022-10-27 07:35:08] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00247 [I 2022-10-27 07:35:08] ...trap/engines/engine.py.230: loss: 0.18377 [I 2022-10-27 07:35:08] ...optimizers/trijoint.py.151: total_norm: 9.458404 [I 2022-10-27 07:35:14] ...trap/engines/engine.py.224: train: epoch 55 | batch 990/2304 [I 2022-10-27 07:35:14] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:12:12 [I 2022-10-27 07:35:14] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00242 [I 2022-10-27 07:35:14] ...trap/engines/engine.py.230: loss: 0.21296 [I 2022-10-27 07:35:14] ...optimizers/trijoint.py.151: total_norm: 19.503754 [I 2022-10-27 07:35:19] ...trap/engines/engine.py.224: train: epoch 55 | batch 1000/2304 [I 2022-10-27 07:35:19] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:12:11 [I 2022-10-27 07:35:19] ...trap/engines/engine.py.229: process: 0.54226 | load: 0.00248 [I 2022-10-27 07:35:19] ...trap/engines/engine.py.230: loss: 0.23190 [I 2022-10-27 07:35:19] ...optimizers/trijoint.py.151: total_norm: 39.747395 [I 2022-10-27 07:35:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 1010/2304 [I 2022-10-27 07:35:25] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:12:08 [I 2022-10-27 07:35:25] ...trap/engines/engine.py.229: process: 0.57617 | load: 0.00250 [I 2022-10-27 07:35:25] ...trap/engines/engine.py.230: loss: 0.13412 [I 2022-10-27 07:35:25] ...optimizers/trijoint.py.151: total_norm: 12.257070 [I 2022-10-27 07:35:30] ...trap/engines/engine.py.224: train: epoch 55 | batch 1020/2304 [I 2022-10-27 07:35:30] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:59 [I 2022-10-27 07:35:30] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00248 [I 2022-10-27 07:35:30] ...trap/engines/engine.py.230: loss: 0.13074 [I 2022-10-27 07:35:30] ...optimizers/trijoint.py.151: total_norm: 12.855326 [I 2022-10-27 07:35:36] ...trap/engines/engine.py.224: train: epoch 55 | batch 1030/2304 [I 2022-10-27 07:35:36] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:52 [I 2022-10-27 07:35:36] ...trap/engines/engine.py.229: process: 0.56102 | load: 0.00248 [I 2022-10-27 07:35:36] ...trap/engines/engine.py.230: loss: 0.18413 [I 2022-10-27 07:35:36] ...optimizers/trijoint.py.151: total_norm: 20.709606 [I 2022-10-27 07:35:42] ...trap/engines/engine.py.224: train: epoch 55 | batch 1040/2304 [I 2022-10-27 07:35:42] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:42 [I 2022-10-27 07:35:42] ...trap/engines/engine.py.229: process: 0.53892 | load: 0.00242 [I 2022-10-27 07:35:42] ...trap/engines/engine.py.230: loss: 0.19902 [I 2022-10-27 07:35:42] ...optimizers/trijoint.py.151: total_norm: 14.628204 [I 2022-10-27 07:35:47] ...trap/engines/engine.py.224: train: epoch 55 | batch 1050/2304 [I 2022-10-27 07:35:47] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:43 [I 2022-10-27 07:35:47] ...trap/engines/engine.py.229: process: 0.55093 | load: 0.00246 [I 2022-10-27 07:35:47] ...trap/engines/engine.py.230: loss: 0.18462 [I 2022-10-27 07:35:47] ...optimizers/trijoint.py.151: total_norm: 29.453423 [I 2022-10-27 07:35:53] ...trap/engines/engine.py.224: train: epoch 55 | batch 1060/2304 [I 2022-10-27 07:35:53] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:49 [I 2022-10-27 07:35:53] ...trap/engines/engine.py.229: process: 0.56405 | load: 0.00268 [I 2022-10-27 07:35:53] ...trap/engines/engine.py.230: loss: 0.16017 [I 2022-10-27 07:35:53] ...optimizers/trijoint.py.151: total_norm: 45.464504 [I 2022-10-27 07:35:59] ...trap/engines/engine.py.224: train: epoch 55 | batch 1070/2304 [I 2022-10-27 07:35:59] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:35 [I 2022-10-27 07:35:59] ...trap/engines/engine.py.229: process: 0.56543 | load: 0.00256 [I 2022-10-27 07:35:59] ...trap/engines/engine.py.230: loss: 0.20876 [I 2022-10-27 07:35:59] ...optimizers/trijoint.py.151: total_norm: 26.686966 [I 2022-10-27 07:36:04] ...trap/engines/engine.py.224: train: epoch 55 | batch 1080/2304 [I 2022-10-27 07:36:04] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:34 [I 2022-10-27 07:36:04] ...trap/engines/engine.py.229: process: 0.56457 | load: 0.00250 [I 2022-10-27 07:36:04] ...trap/engines/engine.py.230: loss: 0.12597 [I 2022-10-27 07:36:04] ...optimizers/trijoint.py.151: total_norm: 25.380371 [I 2022-10-27 07:36:10] ...trap/engines/engine.py.224: train: epoch 55 | batch 1090/2304 [I 2022-10-27 07:36:10] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:11:17 [I 2022-10-27 07:36:10] ...trap/engines/engine.py.229: process: 0.55895 | load: 0.00245 [I 2022-10-27 07:36:10] ...trap/engines/engine.py.230: loss: 0.12817 [I 2022-10-27 07:36:10] ...optimizers/trijoint.py.151: total_norm: 20.943449 [I 2022-10-27 07:36:16] ...trap/engines/engine.py.224: train: epoch 55 | batch 1100/2304 [I 2022-10-27 07:36:16] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:11:05 [I 2022-10-27 07:36:16] ...trap/engines/engine.py.229: process: 0.54925 | load: 0.00246 [I 2022-10-27 07:36:16] ...trap/engines/engine.py.230: loss: 0.14693 [I 2022-10-27 07:36:16] ...optimizers/trijoint.py.151: total_norm: 9.901724 [I 2022-10-27 07:36:21] ...trap/engines/engine.py.224: train: epoch 55 | batch 1110/2304 [I 2022-10-27 07:36:21] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:11:11 [I 2022-10-27 07:36:21] ...trap/engines/engine.py.229: process: 0.56566 | load: 0.00249 [I 2022-10-27 07:36:21] ...trap/engines/engine.py.230: loss: 0.14675 [I 2022-10-27 07:36:21] ...optimizers/trijoint.py.151: total_norm: 11.627041 [I 2022-10-27 07:36:27] ...trap/engines/engine.py.224: train: epoch 55 | batch 1120/2304 [I 2022-10-27 07:36:27] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:11:00 [I 2022-10-27 07:36:27] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00251 [I 2022-10-27 07:36:27] ...trap/engines/engine.py.230: loss: 0.12606 [I 2022-10-27 07:36:27] ...optimizers/trijoint.py.151: total_norm: 11.307566 [I 2022-10-27 07:36:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 1130/2304 [I 2022-10-27 07:36:32] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:55 [I 2022-10-27 07:36:32] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00246 [I 2022-10-27 07:36:32] ...trap/engines/engine.py.230: loss: 0.14202 [I 2022-10-27 07:36:32] ...optimizers/trijoint.py.151: total_norm: 17.435413 [I 2022-10-27 07:36:38] ...trap/engines/engine.py.224: train: epoch 55 | batch 1140/2304 [I 2022-10-27 07:36:38] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:52 [I 2022-10-27 07:36:38] ...trap/engines/engine.py.229: process: 0.54713 | load: 0.00249 [I 2022-10-27 07:36:38] ...trap/engines/engine.py.230: loss: 0.14769 [I 2022-10-27 07:36:38] ...optimizers/trijoint.py.151: total_norm: 29.582312 [I 2022-10-27 07:36:44] ...trap/engines/engine.py.224: train: epoch 55 | batch 1150/2304 [I 2022-10-27 07:36:44] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:56 [I 2022-10-27 07:36:44] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00247 [I 2022-10-27 07:36:44] ...trap/engines/engine.py.230: loss: 0.11985 [I 2022-10-27 07:36:44] ...optimizers/trijoint.py.151: total_norm: 8.260889 [I 2022-10-27 07:36:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 1160/2304 [I 2022-10-27 07:36:49] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:42 [I 2022-10-27 07:36:49] ...trap/engines/engine.py.229: process: 0.56611 | load: 0.00249 [I 2022-10-27 07:36:49] ...trap/engines/engine.py.230: loss: 0.16644 [I 2022-10-27 07:36:49] ...optimizers/trijoint.py.151: total_norm: 18.959852 [I 2022-10-27 07:36:55] ...trap/engines/engine.py.224: train: epoch 55 | batch 1170/2304 [I 2022-10-27 07:36:55] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:37 [I 2022-10-27 07:36:55] ...trap/engines/engine.py.229: process: 0.56552 | load: 0.00243 [I 2022-10-27 07:36:55] ...trap/engines/engine.py.230: loss: 0.17676 [I 2022-10-27 07:36:55] ...optimizers/trijoint.py.151: total_norm: 15.837814 [I 2022-10-27 07:37:01] ...trap/engines/engine.py.224: train: epoch 55 | batch 1180/2304 [I 2022-10-27 07:37:01] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:31 [I 2022-10-27 07:37:01] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00256 [I 2022-10-27 07:37:01] ...trap/engines/engine.py.230: loss: 0.18874 [I 2022-10-27 07:37:01] ...optimizers/trijoint.py.151: total_norm: 40.404137 [I 2022-10-27 07:37:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 1190/2304 [I 2022-10-27 07:37:06] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:21 [I 2022-10-27 07:37:06] ...trap/engines/engine.py.229: process: 0.55531 | load: 0.00248 [I 2022-10-27 07:37:06] ...trap/engines/engine.py.230: loss: 0.18238 [I 2022-10-27 07:37:06] ...optimizers/trijoint.py.151: total_norm: 25.273674 [I 2022-10-27 07:37:12] ...trap/engines/engine.py.224: train: epoch 55 | batch 1200/2304 [I 2022-10-27 07:37:12] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:10:18 [I 2022-10-27 07:37:12] ...trap/engines/engine.py.229: process: 0.55687 | load: 0.00248 [I 2022-10-27 07:37:12] ...trap/engines/engine.py.230: loss: 0.15156 [I 2022-10-27 07:37:12] ...optimizers/trijoint.py.151: total_norm: 10.959611 [I 2022-10-27 07:37:18] ...trap/engines/engine.py.224: train: epoch 55 | batch 1210/2304 [I 2022-10-27 07:37:18] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:10:30 [I 2022-10-27 07:37:18] ...trap/engines/engine.py.229: process: 0.63836 | load: 0.00245 [I 2022-10-27 07:37:18] ...trap/engines/engine.py.230: loss: 0.13581 [I 2022-10-27 07:37:18] ...optimizers/trijoint.py.151: total_norm: 35.171597 [I 2022-10-27 07:37:23] ...trap/engines/engine.py.224: train: epoch 55 | batch 1220/2304 [I 2022-10-27 07:37:23] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:10:02 [I 2022-10-27 07:37:23] ...trap/engines/engine.py.229: process: 0.54411 | load: 0.00242 [I 2022-10-27 07:37:23] ...trap/engines/engine.py.230: loss: 0.15569 [I 2022-10-27 07:37:23] ...optimizers/trijoint.py.151: total_norm: 17.328068 [I 2022-10-27 07:37:29] ...trap/engines/engine.py.224: train: epoch 55 | batch 1230/2304 [I 2022-10-27 07:37:29] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:58 [I 2022-10-27 07:37:29] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00257 [I 2022-10-27 07:37:29] ...trap/engines/engine.py.230: loss: 0.16612 [I 2022-10-27 07:37:29] ...optimizers/trijoint.py.151: total_norm: 17.711100 [I 2022-10-27 07:37:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 1240/2304 [I 2022-10-27 07:37:34] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:57 [I 2022-10-27 07:37:34] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00249 [I 2022-10-27 07:37:34] ...trap/engines/engine.py.230: loss: 0.17238 [I 2022-10-27 07:37:34] ...optimizers/trijoint.py.151: total_norm: 11.079715 [I 2022-10-27 07:37:40] ...trap/engines/engine.py.224: train: epoch 55 | batch 1250/2304 [I 2022-10-27 07:37:40] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:49 [I 2022-10-27 07:37:40] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00243 [I 2022-10-27 07:37:40] ...trap/engines/engine.py.230: loss: 0.13891 [I 2022-10-27 07:37:40] ...optimizers/trijoint.py.151: total_norm: 11.529254 [I 2022-10-27 07:37:46] ...trap/engines/engine.py.224: train: epoch 55 | batch 1260/2304 [I 2022-10-27 07:37:46] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:44 [I 2022-10-27 07:37:46] ...trap/engines/engine.py.229: process: 0.57232 | load: 0.00250 [I 2022-10-27 07:37:46] ...trap/engines/engine.py.230: loss: 0.14879 [I 2022-10-27 07:37:46] ...optimizers/trijoint.py.151: total_norm: 18.485603 [I 2022-10-27 07:37:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 1270/2304 [I 2022-10-27 07:37:51] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:33 [I 2022-10-27 07:37:51] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00244 [I 2022-10-27 07:37:51] ...trap/engines/engine.py.230: loss: 0.15943 [I 2022-10-27 07:37:51] ...optimizers/trijoint.py.151: total_norm: 13.092530 [I 2022-10-27 07:37:57] ...trap/engines/engine.py.224: train: epoch 55 | batch 1280/2304 [I 2022-10-27 07:37:57] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:30 [I 2022-10-27 07:37:57] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00241 [I 2022-10-27 07:37:57] ...trap/engines/engine.py.230: loss: 0.21193 [I 2022-10-27 07:37:57] ...optimizers/trijoint.py.151: total_norm: 23.976938 [I 2022-10-27 07:38:02] ...trap/engines/engine.py.224: train: epoch 55 | batch 1290/2304 [I 2022-10-27 07:38:02] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:31 [I 2022-10-27 07:38:02] ...trap/engines/engine.py.229: process: 0.57275 | load: 0.00291 [I 2022-10-27 07:38:02] ...trap/engines/engine.py.230: loss: 0.12183 [I 2022-10-27 07:38:02] ...optimizers/trijoint.py.151: total_norm: 12.188529 [I 2022-10-27 07:38:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 1300/2304 [I 2022-10-27 07:38:08] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:09:22 [I 2022-10-27 07:38:08] ...trap/engines/engine.py.229: process: 0.56300 | load: 0.00254 [I 2022-10-27 07:38:08] ...trap/engines/engine.py.230: loss: 0.19085 [I 2022-10-27 07:38:08] ...optimizers/trijoint.py.151: total_norm: 15.578040 [I 2022-10-27 07:38:14] ...trap/engines/engine.py.224: train: epoch 55 | batch 1310/2304 [I 2022-10-27 07:38:14] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:09:12 [I 2022-10-27 07:38:14] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00244 [I 2022-10-27 07:38:14] ...trap/engines/engine.py.230: loss: 0.14810 [I 2022-10-27 07:38:14] ...optimizers/trijoint.py.151: total_norm: 17.943169 [I 2022-10-27 07:38:19] ...trap/engines/engine.py.224: train: epoch 55 | batch 1320/2304 [I 2022-10-27 07:38:19] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:09:07 [I 2022-10-27 07:38:19] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00246 [I 2022-10-27 07:38:19] ...trap/engines/engine.py.230: loss: 0.15033 [I 2022-10-27 07:38:19] ...optimizers/trijoint.py.151: total_norm: 19.371399 [I 2022-10-27 07:38:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 1330/2304 [I 2022-10-27 07:38:25] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:09:07 [I 2022-10-27 07:38:25] ...trap/engines/engine.py.229: process: 0.56578 | load: 0.00252 [I 2022-10-27 07:38:25] ...trap/engines/engine.py.230: loss: 0.13300 [I 2022-10-27 07:38:25] ...optimizers/trijoint.py.151: total_norm: 13.481406 [I 2022-10-27 07:38:31] ...trap/engines/engine.py.224: train: epoch 55 | batch 1340/2304 [I 2022-10-27 07:38:31] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:09:11 [I 2022-10-27 07:38:31] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00249 [I 2022-10-27 07:38:31] ...trap/engines/engine.py.230: loss: 0.13516 [I 2022-10-27 07:38:31] ...optimizers/trijoint.py.151: total_norm: 10.591864 [I 2022-10-27 07:38:36] ...trap/engines/engine.py.224: train: epoch 55 | batch 1350/2304 [I 2022-10-27 07:38:36] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:48 [I 2022-10-27 07:38:36] ...trap/engines/engine.py.229: process: 0.54280 | load: 0.00249 [I 2022-10-27 07:38:36] ...trap/engines/engine.py.230: loss: 0.13658 [I 2022-10-27 07:38:36] ...optimizers/trijoint.py.151: total_norm: 19.219423 [I 2022-10-27 07:38:42] ...trap/engines/engine.py.224: train: epoch 55 | batch 1360/2304 [I 2022-10-27 07:38:42] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:43 [I 2022-10-27 07:38:42] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00247 [I 2022-10-27 07:38:42] ...trap/engines/engine.py.230: loss: 0.13808 [I 2022-10-27 07:38:42] ...optimizers/trijoint.py.151: total_norm: 14.679686 [I 2022-10-27 07:38:47] ...trap/engines/engine.py.224: train: epoch 55 | batch 1370/2304 [I 2022-10-27 07:38:47] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:42 [I 2022-10-27 07:38:47] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00245 [I 2022-10-27 07:38:47] ...trap/engines/engine.py.230: loss: 0.16616 [I 2022-10-27 07:38:47] ...optimizers/trijoint.py.151: total_norm: 23.710028 [I 2022-10-27 07:38:53] ...trap/engines/engine.py.224: train: epoch 55 | batch 1380/2304 [I 2022-10-27 07:38:53] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:33 [I 2022-10-27 07:38:53] ...trap/engines/engine.py.229: process: 0.54676 | load: 0.00245 [I 2022-10-27 07:38:53] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-27 07:38:53] ...optimizers/trijoint.py.151: total_norm: 14.633958 [I 2022-10-27 07:38:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 1390/2304 [I 2022-10-27 07:38:58] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:34 [I 2022-10-27 07:38:58] ...trap/engines/engine.py.229: process: 0.57463 | load: 0.00243 [I 2022-10-27 07:38:58] ...trap/engines/engine.py.230: loss: 0.17537 [I 2022-10-27 07:38:58] ...optimizers/trijoint.py.151: total_norm: 35.766350 [I 2022-10-27 07:39:04] ...trap/engines/engine.py.224: train: epoch 55 | batch 1400/2304 [I 2022-10-27 07:39:04] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:26 [I 2022-10-27 07:39:04] ...trap/engines/engine.py.229: process: 0.54938 | load: 0.00248 [I 2022-10-27 07:39:04] ...trap/engines/engine.py.230: loss: 0.15553 [I 2022-10-27 07:39:04] ...optimizers/trijoint.py.151: total_norm: 11.743869 [I 2022-10-27 07:39:10] ...trap/engines/engine.py.224: train: epoch 55 | batch 1410/2304 [I 2022-10-27 07:39:10] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:08:25 [I 2022-10-27 07:39:10] ...trap/engines/engine.py.229: process: 0.56989 | load: 0.00244 [I 2022-10-27 07:39:10] ...trap/engines/engine.py.230: loss: 0.14228 [I 2022-10-27 07:39:10] ...optimizers/trijoint.py.151: total_norm: 11.206007 [I 2022-10-27 07:39:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 1420/2304 [I 2022-10-27 07:39:15] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:08:15 [I 2022-10-27 07:39:15] ...trap/engines/engine.py.229: process: 0.55618 | load: 0.00249 [I 2022-10-27 07:39:15] ...trap/engines/engine.py.230: loss: 0.22458 [I 2022-10-27 07:39:15] ...optimizers/trijoint.py.151: total_norm: 25.605890 [I 2022-10-27 07:39:21] ...trap/engines/engine.py.224: train: epoch 55 | batch 1430/2304 [I 2022-10-27 07:39:21] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:08:07 [I 2022-10-27 07:39:21] ...trap/engines/engine.py.229: process: 0.54490 | load: 0.00248 [I 2022-10-27 07:39:21] ...trap/engines/engine.py.230: loss: 0.16568 [I 2022-10-27 07:39:21] ...optimizers/trijoint.py.151: total_norm: 14.473698 [I 2022-10-27 07:39:27] ...trap/engines/engine.py.224: train: epoch 55 | batch 1440/2304 [I 2022-10-27 07:39:27] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:08:07 [I 2022-10-27 07:39:27] ...trap/engines/engine.py.229: process: 0.56229 | load: 0.00249 [I 2022-10-27 07:39:27] ...trap/engines/engine.py.230: loss: 0.14509 [I 2022-10-27 07:39:27] ...optimizers/trijoint.py.151: total_norm: 37.909981 [I 2022-10-27 07:39:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 1450/2304 [I 2022-10-27 07:39:32] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:08:02 [I 2022-10-27 07:39:32] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00253 [I 2022-10-27 07:39:32] ...trap/engines/engine.py.230: loss: 0.11842 [I 2022-10-27 07:39:32] ...optimizers/trijoint.py.151: total_norm: 12.729198 [I 2022-10-27 07:39:38] ...trap/engines/engine.py.224: train: epoch 55 | batch 1460/2304 [I 2022-10-27 07:39:38] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:51 [I 2022-10-27 07:39:38] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00248 [I 2022-10-27 07:39:38] ...trap/engines/engine.py.230: loss: 0.14546 [I 2022-10-27 07:39:38] ...optimizers/trijoint.py.151: total_norm: 15.371144 [I 2022-10-27 07:39:44] ...trap/engines/engine.py.224: train: epoch 55 | batch 1470/2304 [I 2022-10-27 07:39:44] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:53 [I 2022-10-27 07:39:44] ...trap/engines/engine.py.229: process: 0.55376 | load: 0.00246 [I 2022-10-27 07:39:44] ...trap/engines/engine.py.230: loss: 0.14116 [I 2022-10-27 07:39:44] ...optimizers/trijoint.py.151: total_norm: 9.856110 [I 2022-10-27 07:39:49] ...trap/engines/engine.py.224: train: epoch 55 | batch 1480/2304 [I 2022-10-27 07:39:49] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:43 [I 2022-10-27 07:39:49] ...trap/engines/engine.py.229: process: 0.57381 | load: 0.00247 [I 2022-10-27 07:39:49] ...trap/engines/engine.py.230: loss: 0.10929 [I 2022-10-27 07:39:49] ...optimizers/trijoint.py.151: total_norm: 7.371431 [I 2022-10-27 07:39:55] ...trap/engines/engine.py.224: train: epoch 55 | batch 1490/2304 [I 2022-10-27 07:39:55] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:38 [I 2022-10-27 07:39:55] ...trap/engines/engine.py.229: process: 0.56811 | load: 0.00257 [I 2022-10-27 07:39:55] ...trap/engines/engine.py.230: loss: 0.18542 [I 2022-10-27 07:39:55] ...optimizers/trijoint.py.151: total_norm: 18.863775 [I 2022-10-27 07:40:00] ...trap/engines/engine.py.224: train: epoch 55 | batch 1500/2304 [I 2022-10-27 07:40:00] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:30 [I 2022-10-27 07:40:00] ...trap/engines/engine.py.229: process: 0.56296 | load: 0.00248 [I 2022-10-27 07:40:00] ...trap/engines/engine.py.230: loss: 0.15732 [I 2022-10-27 07:40:00] ...optimizers/trijoint.py.151: total_norm: 15.225747 [I 2022-10-27 07:40:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 1510/2304 [I 2022-10-27 07:40:06] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:22 [I 2022-10-27 07:40:06] ...trap/engines/engine.py.229: process: 0.55804 | load: 0.00247 [I 2022-10-27 07:40:06] ...trap/engines/engine.py.230: loss: 0.26199 [I 2022-10-27 07:40:06] ...optimizers/trijoint.py.151: total_norm: 25.195091 [I 2022-10-27 07:40:12] ...trap/engines/engine.py.224: train: epoch 55 | batch 1520/2304 [I 2022-10-27 07:40:12] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:07:17 [I 2022-10-27 07:40:12] ...trap/engines/engine.py.229: process: 0.54550 | load: 0.00244 [I 2022-10-27 07:40:12] ...trap/engines/engine.py.230: loss: 0.11298 [I 2022-10-27 07:40:12] ...optimizers/trijoint.py.151: total_norm: 9.663962 [I 2022-10-27 07:40:17] ...trap/engines/engine.py.224: train: epoch 55 | batch 1530/2304 [I 2022-10-27 07:40:17] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:07:11 [I 2022-10-27 07:40:17] ...trap/engines/engine.py.229: process: 0.55055 | load: 0.00246 [I 2022-10-27 07:40:17] ...trap/engines/engine.py.230: loss: 0.15069 [I 2022-10-27 07:40:17] ...optimizers/trijoint.py.151: total_norm: 18.463268 [I 2022-10-27 07:40:23] ...trap/engines/engine.py.224: train: epoch 55 | batch 1540/2304 [I 2022-10-27 07:40:23] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:07:06 [I 2022-10-27 07:40:23] ...trap/engines/engine.py.229: process: 0.54896 | load: 0.00244 [I 2022-10-27 07:40:23] ...trap/engines/engine.py.230: loss: 0.14333 [I 2022-10-27 07:40:23] ...optimizers/trijoint.py.151: total_norm: 27.400782 [I 2022-10-27 07:40:28] ...trap/engines/engine.py.224: train: epoch 55 | batch 1550/2304 [I 2022-10-27 07:40:28] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:59 [I 2022-10-27 07:40:28] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00242 [I 2022-10-27 07:40:28] ...trap/engines/engine.py.230: loss: 0.15982 [I 2022-10-27 07:40:28] ...optimizers/trijoint.py.151: total_norm: 27.705132 [I 2022-10-27 07:40:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 1560/2304 [I 2022-10-27 07:40:34] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:54 [I 2022-10-27 07:40:34] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00250 [I 2022-10-27 07:40:34] ...trap/engines/engine.py.230: loss: 0.15620 [I 2022-10-27 07:40:34] ...optimizers/trijoint.py.151: total_norm: 14.113554 [I 2022-10-27 07:40:40] ...trap/engines/engine.py.224: train: epoch 55 | batch 1570/2304 [I 2022-10-27 07:40:40] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:49 [I 2022-10-27 07:40:40] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00242 [I 2022-10-27 07:40:40] ...trap/engines/engine.py.230: loss: 0.15545 [I 2022-10-27 07:40:40] ...optimizers/trijoint.py.151: total_norm: 25.315966 [I 2022-10-27 07:40:45] ...trap/engines/engine.py.224: train: epoch 55 | batch 1580/2304 [I 2022-10-27 07:40:45] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:44 [I 2022-10-27 07:40:45] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00266 [I 2022-10-27 07:40:45] ...trap/engines/engine.py.230: loss: 0.15813 [I 2022-10-27 07:40:45] ...optimizers/trijoint.py.151: total_norm: 13.024725 [I 2022-10-27 07:40:51] ...trap/engines/engine.py.224: train: epoch 55 | batch 1590/2304 [I 2022-10-27 07:40:51] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:42 [I 2022-10-27 07:40:51] ...trap/engines/engine.py.229: process: 0.56941 | load: 0.00253 [I 2022-10-27 07:40:51] ...trap/engines/engine.py.230: loss: 0.16319 [I 2022-10-27 07:40:51] ...optimizers/trijoint.py.151: total_norm: 11.643524 [I 2022-10-27 07:40:57] ...trap/engines/engine.py.224: train: epoch 55 | batch 1600/2304 [I 2022-10-27 07:40:57] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:37 [I 2022-10-27 07:40:57] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00246 [I 2022-10-27 07:40:57] ...trap/engines/engine.py.230: loss: 0.18851 [I 2022-10-27 07:40:57] ...optimizers/trijoint.py.151: total_norm: 9.270191 [I 2022-10-27 07:41:02] ...trap/engines/engine.py.224: train: epoch 55 | batch 1610/2304 [I 2022-10-27 07:41:02] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:28 [I 2022-10-27 07:41:02] ...trap/engines/engine.py.229: process: 0.54723 | load: 0.00254 [I 2022-10-27 07:41:02] ...trap/engines/engine.py.230: loss: 0.13219 [I 2022-10-27 07:41:02] ...optimizers/trijoint.py.151: total_norm: 13.897312 [I 2022-10-27 07:41:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 1620/2304 [I 2022-10-27 07:41:08] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:06:22 [I 2022-10-27 07:41:08] ...trap/engines/engine.py.229: process: 0.56493 | load: 0.00281 [I 2022-10-27 07:41:08] ...trap/engines/engine.py.230: loss: 0.11044 [I 2022-10-27 07:41:08] ...optimizers/trijoint.py.151: total_norm: 8.779006 [I 2022-10-27 07:41:13] ...trap/engines/engine.py.224: train: epoch 55 | batch 1630/2304 [I 2022-10-27 07:41:13] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:06:14 [I 2022-10-27 07:41:13] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00266 [I 2022-10-27 07:41:13] ...trap/engines/engine.py.230: loss: 0.13335 [I 2022-10-27 07:41:13] ...optimizers/trijoint.py.151: total_norm: 39.411316 [I 2022-10-27 07:41:19] ...trap/engines/engine.py.224: train: epoch 55 | batch 1640/2304 [I 2022-10-27 07:41:19] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:06:10 [I 2022-10-27 07:41:19] ...trap/engines/engine.py.229: process: 0.55215 | load: 0.00267 [I 2022-10-27 07:41:19] ...trap/engines/engine.py.230: loss: 0.17964 [I 2022-10-27 07:41:19] ...optimizers/trijoint.py.151: total_norm: 19.918686 [I 2022-10-27 07:41:25] ...trap/engines/engine.py.224: train: epoch 55 | batch 1650/2304 [I 2022-10-27 07:41:25] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:06:08 [I 2022-10-27 07:41:25] ...trap/engines/engine.py.229: process: 0.57311 | load: 0.00264 [I 2022-10-27 07:41:25] ...trap/engines/engine.py.230: loss: 0.18683 [I 2022-10-27 07:41:25] ...optimizers/trijoint.py.151: total_norm: 11.249937 [I 2022-10-27 07:41:30] ...trap/engines/engine.py.224: train: epoch 55 | batch 1660/2304 [I 2022-10-27 07:41:30] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:06:03 [I 2022-10-27 07:41:30] ...trap/engines/engine.py.229: process: 0.56606 | load: 0.00278 [I 2022-10-27 07:41:30] ...trap/engines/engine.py.230: loss: 0.12866 [I 2022-10-27 07:41:30] ...optimizers/trijoint.py.151: total_norm: 10.347914 [I 2022-10-27 07:41:36] ...trap/engines/engine.py.224: train: epoch 55 | batch 1670/2304 [I 2022-10-27 07:41:36] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:53 [I 2022-10-27 07:41:36] ...trap/engines/engine.py.229: process: 0.56320 | load: 0.00273 [I 2022-10-27 07:41:36] ...trap/engines/engine.py.230: loss: 0.16504 [I 2022-10-27 07:41:36] ...optimizers/trijoint.py.151: total_norm: 69.192802 [I 2022-10-27 07:41:41] ...trap/engines/engine.py.224: train: epoch 55 | batch 1680/2304 [I 2022-10-27 07:41:41] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:49 [I 2022-10-27 07:41:41] ...trap/engines/engine.py.229: process: 0.56654 | load: 0.00269 [I 2022-10-27 07:41:41] ...trap/engines/engine.py.230: loss: 0.15326 [I 2022-10-27 07:41:41] ...optimizers/trijoint.py.151: total_norm: 17.843222 [I 2022-10-27 07:41:47] ...trap/engines/engine.py.224: train: epoch 55 | batch 1690/2304 [I 2022-10-27 07:41:47] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:45 [I 2022-10-27 07:41:47] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00250 [I 2022-10-27 07:41:47] ...trap/engines/engine.py.230: loss: 0.16018 [I 2022-10-27 07:41:47] ...optimizers/trijoint.py.151: total_norm: 19.238091 [I 2022-10-27 07:41:53] ...trap/engines/engine.py.224: train: epoch 55 | batch 1700/2304 [I 2022-10-27 07:41:53] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:34 [I 2022-10-27 07:41:53] ...trap/engines/engine.py.229: process: 0.54421 | load: 0.00253 [I 2022-10-27 07:41:53] ...trap/engines/engine.py.230: loss: 0.10009 [I 2022-10-27 07:41:53] ...optimizers/trijoint.py.151: total_norm: 9.180081 [I 2022-10-27 07:41:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 1710/2304 [I 2022-10-27 07:41:58] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:29 [I 2022-10-27 07:41:58] ...trap/engines/engine.py.229: process: 0.54730 | load: 0.00248 [I 2022-10-27 07:41:58] ...trap/engines/engine.py.230: loss: 0.12098 [I 2022-10-27 07:41:58] ...optimizers/trijoint.py.151: total_norm: 20.484539 [I 2022-10-27 07:42:04] ...trap/engines/engine.py.224: train: epoch 55 | batch 1720/2304 [I 2022-10-27 07:42:04] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:24 [I 2022-10-27 07:42:04] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00254 [I 2022-10-27 07:42:04] ...trap/engines/engine.py.230: loss: 0.11217 [I 2022-10-27 07:42:04] ...optimizers/trijoint.py.151: total_norm: 29.989216 [I 2022-10-27 07:42:09] ...trap/engines/engine.py.224: train: epoch 55 | batch 1730/2304 [I 2022-10-27 07:42:09] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:05:20 [I 2022-10-27 07:42:09] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00251 [I 2022-10-27 07:42:09] ...trap/engines/engine.py.230: loss: 0.11771 [I 2022-10-27 07:42:09] ...optimizers/trijoint.py.151: total_norm: 6.349952 [I 2022-10-27 07:42:15] ...trap/engines/engine.py.224: train: epoch 55 | batch 1740/2304 [I 2022-10-27 07:42:15] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:05:16 [I 2022-10-27 07:42:15] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00253 [I 2022-10-27 07:42:15] ...trap/engines/engine.py.230: loss: 0.19872 [I 2022-10-27 07:42:15] ...optimizers/trijoint.py.151: total_norm: 39.620773 [I 2022-10-27 07:42:21] ...trap/engines/engine.py.224: train: epoch 55 | batch 1750/2304 [I 2022-10-27 07:42:21] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:05:09 [I 2022-10-27 07:42:21] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00249 [I 2022-10-27 07:42:21] ...trap/engines/engine.py.230: loss: 0.19507 [I 2022-10-27 07:42:21] ...optimizers/trijoint.py.151: total_norm: 15.983077 [I 2022-10-27 07:42:26] ...trap/engines/engine.py.224: train: epoch 55 | batch 1760/2304 [I 2022-10-27 07:42:26] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:05:02 [I 2022-10-27 07:42:26] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00293 [I 2022-10-27 07:42:26] ...trap/engines/engine.py.230: loss: 0.14673 [I 2022-10-27 07:42:26] ...optimizers/trijoint.py.151: total_norm: 19.939318 [I 2022-10-27 07:42:32] ...trap/engines/engine.py.224: train: epoch 55 | batch 1770/2304 [I 2022-10-27 07:42:32] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:55 [I 2022-10-27 07:42:32] ...trap/engines/engine.py.229: process: 0.54523 | load: 0.00298 [I 2022-10-27 07:42:32] ...trap/engines/engine.py.230: loss: 0.12815 [I 2022-10-27 07:42:32] ...optimizers/trijoint.py.151: total_norm: 37.193554 [I 2022-10-27 07:42:37] ...trap/engines/engine.py.224: train: epoch 55 | batch 1780/2304 [I 2022-10-27 07:42:37] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:51 [I 2022-10-27 07:42:37] ...trap/engines/engine.py.229: process: 0.56369 | load: 0.00258 [I 2022-10-27 07:42:37] ...trap/engines/engine.py.230: loss: 0.23617 [I 2022-10-27 07:42:37] ...optimizers/trijoint.py.151: total_norm: 22.145784 [I 2022-10-27 07:42:43] ...trap/engines/engine.py.224: train: epoch 55 | batch 1790/2304 [I 2022-10-27 07:42:43] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:47 [I 2022-10-27 07:42:43] ...trap/engines/engine.py.229: process: 0.57552 | load: 0.00244 [I 2022-10-27 07:42:43] ...trap/engines/engine.py.230: loss: 0.16900 [I 2022-10-27 07:42:43] ...optimizers/trijoint.py.151: total_norm: 16.532673 [I 2022-10-27 07:42:48] ...trap/engines/engine.py.224: train: epoch 55 | batch 1800/2304 [I 2022-10-27 07:42:48] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:40 [I 2022-10-27 07:42:48] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00267 [I 2022-10-27 07:42:48] ...trap/engines/engine.py.230: loss: 0.12561 [I 2022-10-27 07:42:48] ...optimizers/trijoint.py.151: total_norm: 14.422714 [I 2022-10-27 07:42:54] ...trap/engines/engine.py.224: train: epoch 55 | batch 1810/2304 [I 2022-10-27 07:42:54] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:33 [I 2022-10-27 07:42:54] ...trap/engines/engine.py.229: process: 0.54643 | load: 0.00281 [I 2022-10-27 07:42:54] ...trap/engines/engine.py.230: loss: 0.16476 [I 2022-10-27 07:42:54] ...optimizers/trijoint.py.151: total_norm: 21.535421 [I 2022-10-27 07:43:00] ...trap/engines/engine.py.224: train: epoch 55 | batch 1820/2304 [I 2022-10-27 07:43:00] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:30 [I 2022-10-27 07:43:00] ...trap/engines/engine.py.229: process: 0.56828 | load: 0.00257 [I 2022-10-27 07:43:00] ...trap/engines/engine.py.230: loss: 0.18272 [I 2022-10-27 07:43:00] ...optimizers/trijoint.py.151: total_norm: 16.191277 [I 2022-10-27 07:43:05] ...trap/engines/engine.py.224: train: epoch 55 | batch 1830/2304 [I 2022-10-27 07:43:05] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:23 [I 2022-10-27 07:43:05] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00251 [I 2022-10-27 07:43:05] ...trap/engines/engine.py.230: loss: 0.13000 [I 2022-10-27 07:43:05] ...optimizers/trijoint.py.151: total_norm: 13.449366 [I 2022-10-27 07:43:11] ...trap/engines/engine.py.224: train: epoch 55 | batch 1840/2304 [I 2022-10-27 07:43:11] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:17 [I 2022-10-27 07:43:11] ...trap/engines/engine.py.229: process: 0.54960 | load: 0.00269 [I 2022-10-27 07:43:11] ...trap/engines/engine.py.230: loss: 0.17017 [I 2022-10-27 07:43:11] ...optimizers/trijoint.py.151: total_norm: 27.884405 [I 2022-10-27 07:43:16] ...trap/engines/engine.py.224: train: epoch 55 | batch 1850/2304 [I 2022-10-27 07:43:16] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:04:12 [I 2022-10-27 07:43:16] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00255 [I 2022-10-27 07:43:16] ...trap/engines/engine.py.230: loss: 0.14221 [I 2022-10-27 07:43:16] ...optimizers/trijoint.py.151: total_norm: 17.442127 [I 2022-10-27 07:43:22] ...trap/engines/engine.py.224: train: epoch 55 | batch 1860/2304 [I 2022-10-27 07:43:22] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:04:05 [I 2022-10-27 07:43:22] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00253 [I 2022-10-27 07:43:22] ...trap/engines/engine.py.230: loss: 0.14686 [I 2022-10-27 07:43:22] ...optimizers/trijoint.py.151: total_norm: 32.860352 [I 2022-10-27 07:43:27] ...trap/engines/engine.py.224: train: epoch 55 | batch 1870/2304 [I 2022-10-27 07:43:27] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:04:02 [I 2022-10-27 07:43:27] ...trap/engines/engine.py.229: process: 0.56188 | load: 0.00249 [I 2022-10-27 07:43:27] ...trap/engines/engine.py.230: loss: 0.19771 [I 2022-10-27 07:43:27] ...optimizers/trijoint.py.151: total_norm: 17.144291 [I 2022-10-27 07:43:33] ...trap/engines/engine.py.224: train: epoch 55 | batch 1880/2304 [I 2022-10-27 07:43:33] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:56 [I 2022-10-27 07:43:33] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00250 [I 2022-10-27 07:43:33] ...trap/engines/engine.py.230: loss: 0.14781 [I 2022-10-27 07:43:33] ...optimizers/trijoint.py.151: total_norm: 20.053795 [I 2022-10-27 07:43:39] ...trap/engines/engine.py.224: train: epoch 55 | batch 1890/2304 [I 2022-10-27 07:43:39] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:52 [I 2022-10-27 07:43:39] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00253 [I 2022-10-27 07:43:39] ...trap/engines/engine.py.230: loss: 0.11912 [I 2022-10-27 07:43:39] ...optimizers/trijoint.py.151: total_norm: 19.811743 [I 2022-10-27 07:43:44] ...trap/engines/engine.py.224: train: epoch 55 | batch 1900/2304 [I 2022-10-27 07:43:44] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:45 [I 2022-10-27 07:43:44] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00255 [I 2022-10-27 07:43:44] ...trap/engines/engine.py.230: loss: 0.15185 [I 2022-10-27 07:43:44] ...optimizers/trijoint.py.151: total_norm: 25.160679 [I 2022-10-27 07:43:50] ...trap/engines/engine.py.224: train: epoch 55 | batch 1910/2304 [I 2022-10-27 07:43:50] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:38 [I 2022-10-27 07:43:50] ...trap/engines/engine.py.229: process: 0.54761 | load: 0.00243 [I 2022-10-27 07:43:50] ...trap/engines/engine.py.230: loss: 0.18902 [I 2022-10-27 07:43:50] ...optimizers/trijoint.py.151: total_norm: 15.392558 [I 2022-10-27 07:43:55] ...trap/engines/engine.py.224: train: epoch 55 | batch 1920/2304 [I 2022-10-27 07:43:55] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:34 [I 2022-10-27 07:43:55] ...trap/engines/engine.py.229: process: 0.56265 | load: 0.00294 [I 2022-10-27 07:43:55] ...trap/engines/engine.py.230: loss: 0.15012 [I 2022-10-27 07:43:55] ...optimizers/trijoint.py.151: total_norm: 11.853641 [I 2022-10-27 07:44:01] ...trap/engines/engine.py.224: train: epoch 55 | batch 1930/2304 [I 2022-10-27 07:44:01] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:29 [I 2022-10-27 07:44:01] ...trap/engines/engine.py.229: process: 0.56489 | load: 0.00260 [I 2022-10-27 07:44:01] ...trap/engines/engine.py.230: loss: 0.21569 [I 2022-10-27 07:44:01] ...optimizers/trijoint.py.151: total_norm: 14.550617 [I 2022-10-27 07:44:06] ...trap/engines/engine.py.224: train: epoch 55 | batch 1940/2304 [I 2022-10-27 07:44:06] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:23 [I 2022-10-27 07:44:06] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00256 [I 2022-10-27 07:44:06] ...trap/engines/engine.py.230: loss: 0.14810 [I 2022-10-27 07:44:06] ...optimizers/trijoint.py.151: total_norm: 14.247345 [I 2022-10-27 07:44:12] ...trap/engines/engine.py.224: train: epoch 55 | batch 1950/2304 [I 2022-10-27 07:44:12] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:03:16 [I 2022-10-27 07:44:12] ...trap/engines/engine.py.229: process: 0.55411 | load: 0.00249 [I 2022-10-27 07:44:12] ...trap/engines/engine.py.230: loss: 0.15407 [I 2022-10-27 07:44:12] ...optimizers/trijoint.py.151: total_norm: 18.946400 [I 2022-10-27 07:44:18] ...trap/engines/engine.py.224: train: epoch 55 | batch 1960/2304 [I 2022-10-27 07:44:18] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:03:12 [I 2022-10-27 07:44:18] ...trap/engines/engine.py.229: process: 0.56844 | load: 0.00247 [I 2022-10-27 07:44:18] ...trap/engines/engine.py.230: loss: 0.12172 [I 2022-10-27 07:44:18] ...optimizers/trijoint.py.151: total_norm: 12.603189 [I 2022-10-27 07:44:23] ...trap/engines/engine.py.224: train: epoch 55 | batch 1970/2304 [I 2022-10-27 07:44:23] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:03:06 [I 2022-10-27 07:44:23] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00247 [I 2022-10-27 07:44:23] ...trap/engines/engine.py.230: loss: 0.14178 [I 2022-10-27 07:44:23] ...optimizers/trijoint.py.151: total_norm: 9.136027 [I 2022-10-27 07:44:29] ...trap/engines/engine.py.224: train: epoch 55 | batch 1980/2304 [I 2022-10-27 07:44:29] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:03:00 [I 2022-10-27 07:44:29] ...trap/engines/engine.py.229: process: 0.55480 | load: 0.00244 [I 2022-10-27 07:44:29] ...trap/engines/engine.py.230: loss: 0.14358 [I 2022-10-27 07:44:29] ...optimizers/trijoint.py.151: total_norm: 17.505589 [I 2022-10-27 07:44:34] ...trap/engines/engine.py.224: train: epoch 55 | batch 1990/2304 [I 2022-10-27 07:44:34] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:54 [I 2022-10-27 07:44:34] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00248 [I 2022-10-27 07:44:34] ...trap/engines/engine.py.230: loss: 0.09600 [I 2022-10-27 07:44:34] ...optimizers/trijoint.py.151: total_norm: 8.277944 [I 2022-10-27 07:44:56] ...trap/engines/engine.py.224: train: epoch 55 | batch 2000/2304 [I 2022-10-27 07:44:56] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:13:25 [I 2022-10-27 07:44:56] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00254 [I 2022-10-27 07:44:56] ...trap/engines/engine.py.230: loss: 0.16198 [I 2022-10-27 07:44:56] ...optimizers/trijoint.py.151: total_norm: 12.809191 [I 2022-10-27 07:45:03] ...trap/engines/engine.py.224: train: epoch 55 | batch 2010/2304 [I 2022-10-27 07:45:03] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:03:57 [I 2022-10-27 07:45:03] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00260 [I 2022-10-27 07:45:03] ...trap/engines/engine.py.230: loss: 0.16888 [I 2022-10-27 07:45:03] ...optimizers/trijoint.py.151: total_norm: 18.644712 [I 2022-10-27 07:45:08] ...trap/engines/engine.py.224: train: epoch 55 | batch 2020/2304 [I 2022-10-27 07:45:08] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:02:44 [I 2022-10-27 07:45:08] ...trap/engines/engine.py.229: process: 0.54443 | load: 0.00248 [I 2022-10-27 07:45:08] ...trap/engines/engine.py.230: loss: 0.10988 [I 2022-10-27 07:45:08] ...optimizers/trijoint.py.151: total_norm: 19.879297 [I 2022-10-27 07:45:14] ...trap/engines/engine.py.224: train: epoch 55 | batch 2030/2304 [I 2022-10-27 07:45:14] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:33 [I 2022-10-27 07:45:14] ...trap/engines/engine.py.229: process: 0.55092 | load: 0.00250 [I 2022-10-27 07:45:14] ...trap/engines/engine.py.230: loss: 0.14581 [I 2022-10-27 07:45:14] ...optimizers/trijoint.py.151: total_norm: 21.733110 [I 2022-10-27 07:45:20] ...trap/engines/engine.py.224: train: epoch 55 | batch 2040/2304 [I 2022-10-27 07:45:20] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:02:26 [I 2022-10-27 07:45:20] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00249 [I 2022-10-27 07:45:20] ...trap/engines/engine.py.230: loss: 0.18328 [I 2022-10-27 07:45:20] ...optimizers/trijoint.py.151: total_norm: 22.215099 [I 2022-10-27 07:45:39] ...trap/engines/engine.py.224: train: epoch 55 | batch 2050/2304 [I 2022-10-27 07:45:39] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:05:28 [I 2022-10-27 07:45:39] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00242 [I 2022-10-27 07:45:39] ...trap/engines/engine.py.230: loss: 0.17094 [I 2022-10-27 07:45:39] ...optimizers/trijoint.py.151: total_norm: 20.668343 [I 2022-10-27 07:45:45] ...trap/engines/engine.py.224: train: epoch 55 | batch 2060/2304 [I 2022-10-27 07:45:45] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:02:34 [I 2022-10-27 07:45:45] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00248 [I 2022-10-27 07:45:45] ...trap/engines/engine.py.230: loss: 0.13359 [I 2022-10-27 07:45:45] ...optimizers/trijoint.py.151: total_norm: 15.229980 [I 2022-10-27 07:45:50] ...trap/engines/engine.py.224: train: epoch 55 | batch 2070/2304 [I 2022-10-27 07:45:50] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:02:11 [I 2022-10-27 07:45:50] ...trap/engines/engine.py.229: process: 0.54612 | load: 0.00252 [I 2022-10-27 07:45:50] ...trap/engines/engine.py.230: loss: 0.13930 [I 2022-10-27 07:45:50] ...optimizers/trijoint.py.151: total_norm: 15.209230 [I 2022-10-27 07:45:56] ...trap/engines/engine.py.224: train: epoch 55 | batch 2080/2304 [I 2022-10-27 07:45:56] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:02:06 [I 2022-10-27 07:45:56] ...trap/engines/engine.py.229: process: 0.54047 | load: 0.00246 [I 2022-10-27 07:45:56] ...trap/engines/engine.py.230: loss: 0.16343 [I 2022-10-27 07:45:56] ...optimizers/trijoint.py.151: total_norm: 25.767864 [I 2022-10-27 07:46:02] ...trap/engines/engine.py.224: train: epoch 55 | batch 2090/2304 [I 2022-10-27 07:46:02] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:58 [I 2022-10-27 07:46:02] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00255 [I 2022-10-27 07:46:02] ...trap/engines/engine.py.230: loss: 0.15887 [I 2022-10-27 07:46:02] ...optimizers/trijoint.py.151: total_norm: 19.924751 [I 2022-10-27 07:46:07] ...trap/engines/engine.py.224: train: epoch 55 | batch 2100/2304 [I 2022-10-27 07:46:07] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:55 [I 2022-10-27 07:46:07] ...trap/engines/engine.py.229: process: 0.56507 | load: 0.00245 [I 2022-10-27 07:46:07] ...trap/engines/engine.py.230: loss: 0.17100 [I 2022-10-27 07:46:07] ...optimizers/trijoint.py.151: total_norm: 18.457134 [I 2022-10-27 07:46:13] ...trap/engines/engine.py.224: train: epoch 55 | batch 2110/2304 [I 2022-10-27 07:46:13] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:01:48 [I 2022-10-27 07:46:13] ...trap/engines/engine.py.229: process: 0.54821 | load: 0.00245 [I 2022-10-27 07:46:13] ...trap/engines/engine.py.230: loss: 0.16906 [I 2022-10-27 07:46:13] ...optimizers/trijoint.py.151: total_norm: 26.273266 [I 2022-10-27 07:46:19] ...trap/engines/engine.py.224: train: epoch 55 | batch 2120/2304 [I 2022-10-27 07:46:19] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:01:41 [I 2022-10-27 07:46:19] ...trap/engines/engine.py.229: process: 0.55097 | load: 0.00244 [I 2022-10-27 07:46:19] ...trap/engines/engine.py.230: loss: 0.18987 [I 2022-10-27 07:46:19] ...optimizers/trijoint.py.151: total_norm: 23.223869 [I 2022-10-27 07:46:24] ...trap/engines/engine.py.224: train: epoch 55 | batch 2130/2304 [I 2022-10-27 07:46:24] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:01:36 [I 2022-10-27 07:46:24] ...trap/engines/engine.py.229: process: 0.55975 | load: 0.00245 [I 2022-10-27 07:46:24] ...trap/engines/engine.py.230: loss: 0.11202 [I 2022-10-27 07:46:24] ...optimizers/trijoint.py.151: total_norm: 9.728120 [I 2022-10-27 07:46:30] ...trap/engines/engine.py.224: train: epoch 55 | batch 2140/2304 [I 2022-10-27 07:46:30] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:01:31 [I 2022-10-27 07:46:30] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00243 [I 2022-10-27 07:46:30] ...trap/engines/engine.py.230: loss: 0.18283 [I 2022-10-27 07:46:30] ...optimizers/trijoint.py.151: total_norm: 27.732218 [I 2022-10-27 07:46:35] ...trap/engines/engine.py.224: train: epoch 55 | batch 2150/2304 [I 2022-10-27 07:46:35] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:01:25 [I 2022-10-27 07:46:35] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00245 [I 2022-10-27 07:46:35] ...trap/engines/engine.py.230: loss: 0.18208 [I 2022-10-27 07:46:35] ...optimizers/trijoint.py.151: total_norm: 15.526555 [I 2022-10-27 07:46:41] ...trap/engines/engine.py.224: train: epoch 55 | batch 2160/2304 [I 2022-10-27 07:46:41] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:01:20 [I 2022-10-27 07:46:41] ...trap/engines/engine.py.229: process: 0.55431 | load: 0.00246 [I 2022-10-27 07:46:41] ...trap/engines/engine.py.230: loss: 0.17066 [I 2022-10-27 07:46:41] ...optimizers/trijoint.py.151: total_norm: 35.877892 [I 2022-10-27 07:46:46] ...trap/engines/engine.py.224: train: epoch 55 | batch 2170/2304 [I 2022-10-27 07:46:46] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:01:14 [I 2022-10-27 07:46:46] ...trap/engines/engine.py.229: process: 0.54145 | load: 0.00247 [I 2022-10-27 07:46:46] ...trap/engines/engine.py.230: loss: 0.18144 [I 2022-10-27 07:46:46] ...optimizers/trijoint.py.151: total_norm: 22.923807 [I 2022-10-27 07:46:52] ...trap/engines/engine.py.224: train: epoch 55 | batch 2180/2304 [I 2022-10-27 07:46:52] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:01:08 [I 2022-10-27 07:46:52] ...trap/engines/engine.py.229: process: 0.55218 | load: 0.00248 [I 2022-10-27 07:46:52] ...trap/engines/engine.py.230: loss: 0.24006 [I 2022-10-27 07:46:52] ...optimizers/trijoint.py.151: total_norm: 18.732979 [I 2022-10-27 07:46:58] ...trap/engines/engine.py.224: train: epoch 55 | batch 2190/2304 [I 2022-10-27 07:46:58] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:01:03 [I 2022-10-27 07:46:58] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00250 [I 2022-10-27 07:46:58] ...trap/engines/engine.py.230: loss: 0.16409 [I 2022-10-27 07:46:58] ...optimizers/trijoint.py.151: total_norm: 10.075164 [I 2022-10-27 07:47:03] ...trap/engines/engine.py.224: train: epoch 55 | batch 2200/2304 [I 2022-10-27 07:47:03] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:57 [I 2022-10-27 07:47:03] ...trap/engines/engine.py.229: process: 0.56632 | load: 0.00245 [I 2022-10-27 07:47:03] ...trap/engines/engine.py.230: loss: 0.19575 [I 2022-10-27 07:47:03] ...optimizers/trijoint.py.151: total_norm: 19.381649 [I 2022-10-27 07:47:09] ...trap/engines/engine.py.224: train: epoch 55 | batch 2210/2304 [I 2022-10-27 07:47:09] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:52 [I 2022-10-27 07:47:09] ...trap/engines/engine.py.229: process: 0.56557 | load: 0.00244 [I 2022-10-27 07:47:09] ...trap/engines/engine.py.230: loss: 0.15519 [I 2022-10-27 07:47:09] ...optimizers/trijoint.py.151: total_norm: 15.336761 [I 2022-10-27 07:47:14] ...trap/engines/engine.py.224: train: epoch 55 | batch 2220/2304 [I 2022-10-27 07:47:14] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:47 [I 2022-10-27 07:47:14] ...trap/engines/engine.py.229: process: 0.57099 | load: 0.00244 [I 2022-10-27 07:47:14] ...trap/engines/engine.py.230: loss: 0.16063 [I 2022-10-27 07:47:14] ...optimizers/trijoint.py.151: total_norm: 96.278656 [I 2022-10-27 07:47:20] ...trap/engines/engine.py.224: train: epoch 55 | batch 2230/2304 [I 2022-10-27 07:47:20] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:00:42 [I 2022-10-27 07:47:20] ...trap/engines/engine.py.229: process: 0.54963 | load: 0.00251 [I 2022-10-27 07:47:20] ...trap/engines/engine.py.230: loss: 0.14115 [I 2022-10-27 07:47:20] ...optimizers/trijoint.py.151: total_norm: 10.683725 [I 2022-10-27 07:47:26] ...trap/engines/engine.py.224: train: epoch 55 | batch 2240/2304 [I 2022-10-27 07:47:26] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:35 [I 2022-10-27 07:47:26] ...trap/engines/engine.py.229: process: 0.56086 | load: 0.00245 [I 2022-10-27 07:47:26] ...trap/engines/engine.py.230: loss: 0.17447 [I 2022-10-27 07:47:26] ...optimizers/trijoint.py.151: total_norm: 19.266853 [I 2022-10-27 07:47:31] ...trap/engines/engine.py.224: train: epoch 55 | batch 2250/2304 [I 2022-10-27 07:47:31] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:00:30 [I 2022-10-27 07:47:31] ...trap/engines/engine.py.229: process: 0.56904 | load: 0.00256 [I 2022-10-27 07:47:31] ...trap/engines/engine.py.230: loss: 0.15704 [I 2022-10-27 07:47:31] ...optimizers/trijoint.py.151: total_norm: 18.446802 [I 2022-10-27 07:47:37] ...trap/engines/engine.py.224: train: epoch 55 | batch 2260/2304 [I 2022-10-27 07:47:37] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:00:24 [I 2022-10-27 07:47:37] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00249 [I 2022-10-27 07:47:37] ...trap/engines/engine.py.230: loss: 0.17260 [I 2022-10-27 07:47:37] ...optimizers/trijoint.py.151: total_norm: 12.423163 [I 2022-10-27 07:47:43] ...trap/engines/engine.py.224: train: epoch 55 | batch 2270/2304 [I 2022-10-27 07:47:43] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:00:19 [I 2022-10-27 07:47:43] ...trap/engines/engine.py.229: process: 0.57881 | load: 0.00247 [I 2022-10-27 07:47:43] ...trap/engines/engine.py.230: loss: 0.14998 [I 2022-10-27 07:47:43] ...optimizers/trijoint.py.151: total_norm: 32.929970 [I 2022-10-27 07:47:48] ...trap/engines/engine.py.224: train: epoch 55 | batch 2280/2304 [I 2022-10-27 07:47:48] ...trap/engines/engine.py.228: elapsed: 0:22:12 | left: 0:00:13 [I 2022-10-27 07:47:48] ...trap/engines/engine.py.229: process: 0.56835 | load: 0.00245 [I 2022-10-27 07:47:48] ...trap/engines/engine.py.230: loss: 0.17437 [I 2022-10-27 07:47:48] ...optimizers/trijoint.py.151: total_norm: 34.675026 [I 2022-10-27 07:47:54] ...trap/engines/engine.py.224: train: epoch 55 | batch 2290/2304 [I 2022-10-27 07:47:54] ...trap/engines/engine.py.228: elapsed: 0:22:18 | left: 0:00:07 [I 2022-10-27 07:47:54] ...trap/engines/engine.py.229: process: 0.57269 | load: 0.00246 [I 2022-10-27 07:47:54] ...trap/engines/engine.py.230: loss: 0.13177 [I 2022-10-27 07:47:54] ...optimizers/trijoint.py.151: total_norm: 17.274477 [I 2022-10-27 07:47:59] ...trap/engines/engine.py.224: train: epoch 55 | batch 2300/2304 [I 2022-10-27 07:47:59] ...trap/engines/engine.py.228: elapsed: 0:22:23 | left: 0:00:02 [I 2022-10-27 07:47:59] ...trap/engines/engine.py.229: process: 0.53771 | load: 0.00015 [I 2022-10-27 07:47:59] ...trap/engines/engine.py.230: loss: 0.10476 [I 2022-10-27 07:47:59] ...optimizers/trijoint.py.151: total_norm: 7.932204 [I 2022-10-27 07:48:02] ...trap/engines/engine.py.224: train: epoch 55 | batch 2304/2304 [I 2022-10-27 07:48:02] ...trap/engines/engine.py.228: elapsed: 0:22:26 | left: 0:00:00 [I 2022-10-27 07:48:02] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00018 [I 2022-10-27 07:48:02] ...trap/engines/engine.py.230: loss: 0.21068 [I 2022-10-27 07:48:02] ...optimizers/trijoint.py.151: total_norm: 20.497952 [S 2022-10-27 07:48:03] ...trap/engines/engine.py.129: train_epoch.epoch: 55 [S 2022-10-27 07:48:03] ...trap/engines/engine.py.129: train_epoch.loss: 0.15899053327496812 [I 2022-10-27 07:48:03] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 07:48:04] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 55 [I 2022-10-27 07:48:04] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 07:48:06] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 07:48:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 07:48:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 07:48:07] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 07:48:08] ...trap/engines/engine.py.321: eval: epoch 55 | batch 0/510 [I 2022-10-27 07:48:08] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:34:24 [I 2022-10-27 07:48:08] ...trap/engines/engine.py.326: process: 4.04790 | load: 2.92581 [I 2022-10-27 07:48:11] ...trap/engines/engine.py.321: eval: epoch 55 | batch 10/510 [I 2022-10-27 07:48:11] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:08 [I 2022-10-27 07:48:11] ...trap/engines/engine.py.326: process: 0.33066 | load: 0.00019 [I 2022-10-27 07:48:15] ...trap/engines/engine.py.321: eval: epoch 55 | batch 20/510 [I 2022-10-27 07:48:15] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:05 [I 2022-10-27 07:48:15] ...trap/engines/engine.py.326: process: 0.32838 | load: 0.00024 [I 2022-10-27 07:48:18] ...trap/engines/engine.py.321: eval: epoch 55 | batch 30/510 [I 2022-10-27 07:48:18] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:42 [I 2022-10-27 07:48:18] ...trap/engines/engine.py.326: process: 0.32414 | load: 0.00017 [I 2022-10-27 07:48:21] ...trap/engines/engine.py.321: eval: epoch 55 | batch 40/510 [I 2022-10-27 07:48:21] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:43 [I 2022-10-27 07:48:21] ...trap/engines/engine.py.326: process: 0.33837 | load: 0.00017 [I 2022-10-27 07:48:25] ...trap/engines/engine.py.321: eval: epoch 55 | batch 50/510 [I 2022-10-27 07:48:25] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:33 [I 2022-10-27 07:48:25] ...trap/engines/engine.py.326: process: 0.33162 | load: 0.00019 [I 2022-10-27 07:48:28] ...trap/engines/engine.py.321: eval: epoch 55 | batch 60/510 [I 2022-10-27 07:48:28] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:29 [I 2022-10-27 07:48:28] ...trap/engines/engine.py.326: process: 0.33317 | load: 0.00017 [I 2022-10-27 07:48:31] ...trap/engines/engine.py.321: eval: epoch 55 | batch 70/510 [I 2022-10-27 07:48:31] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:27 [I 2022-10-27 07:48:31] ...trap/engines/engine.py.326: process: 0.33748 | load: 0.00016 [I 2022-10-27 07:48:35] ...trap/engines/engine.py.321: eval: epoch 55 | batch 80/510 [I 2022-10-27 07:48:35] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-27 07:48:35] ...trap/engines/engine.py.326: process: 0.32448 | load: 0.00026 [I 2022-10-27 07:48:38] ...trap/engines/engine.py.321: eval: epoch 55 | batch 90/510 [I 2022-10-27 07:48:38] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:22 [I 2022-10-27 07:48:38] ...trap/engines/engine.py.326: process: 0.33048 | load: 0.00030 [I 2022-10-27 07:48:42] ...trap/engines/engine.py.321: eval: epoch 55 | batch 100/510 [I 2022-10-27 07:48:42] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:15 [I 2022-10-27 07:48:42] ...trap/engines/engine.py.326: process: 0.28369 | load: 0.00018 [I 2022-10-27 07:48:44] ...trap/engines/engine.py.321: eval: epoch 55 | batch 110/510 [I 2022-10-27 07:48:44] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:58 [I 2022-10-27 07:48:44] ...trap/engines/engine.py.326: process: 0.28609 | load: 0.00034 [I 2022-10-27 07:48:47] ...trap/engines/engine.py.321: eval: epoch 55 | batch 120/510 [I 2022-10-27 07:48:47] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:51 [I 2022-10-27 07:48:47] ...trap/engines/engine.py.326: process: 0.28167 | load: 0.00014 [I 2022-10-27 07:48:50] ...trap/engines/engine.py.321: eval: epoch 55 | batch 130/510 [I 2022-10-27 07:48:50] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:50 [I 2022-10-27 07:48:50] ...trap/engines/engine.py.326: process: 0.28703 | load: 0.00019 [I 2022-10-27 07:48:53] ...trap/engines/engine.py.321: eval: epoch 55 | batch 140/510 [I 2022-10-27 07:48:53] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 07:48:53] ...trap/engines/engine.py.326: process: 0.29332 | load: 0.00019 [I 2022-10-27 07:48:56] ...trap/engines/engine.py.321: eval: epoch 55 | batch 150/510 [I 2022-10-27 07:48:56] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 07:48:56] ...trap/engines/engine.py.326: process: 0.28925 | load: 0.00018 [I 2022-10-27 07:48:59] ...trap/engines/engine.py.321: eval: epoch 55 | batch 160/510 [I 2022-10-27 07:48:59] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:43 [I 2022-10-27 07:48:59] ...trap/engines/engine.py.326: process: 0.30738 | load: 0.00015 [I 2022-10-27 07:49:02] ...trap/engines/engine.py.321: eval: epoch 55 | batch 170/510 [I 2022-10-27 07:49:02] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 07:49:02] ...trap/engines/engine.py.326: process: 0.28920 | load: 0.00030 [I 2022-10-27 07:49:05] ...trap/engines/engine.py.321: eval: epoch 55 | batch 180/510 [I 2022-10-27 07:49:05] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:37 [I 2022-10-27 07:49:05] ...trap/engines/engine.py.326: process: 0.30628 | load: 0.00017 [I 2022-10-27 07:49:08] ...trap/engines/engine.py.321: eval: epoch 55 | batch 190/510 [I 2022-10-27 07:49:08] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:34 [I 2022-10-27 07:49:08] ...trap/engines/engine.py.326: process: 0.28979 | load: 0.00019 [I 2022-10-27 07:49:11] ...trap/engines/engine.py.321: eval: epoch 55 | batch 200/510 [I 2022-10-27 07:49:11] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:31 [I 2022-10-27 07:49:11] ...trap/engines/engine.py.326: process: 0.29264 | load: 0.00016 [I 2022-10-27 07:49:14] ...trap/engines/engine.py.321: eval: epoch 55 | batch 210/510 [I 2022-10-27 07:49:14] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:27 [I 2022-10-27 07:49:14] ...trap/engines/engine.py.326: process: 0.28979 | load: 0.00015 [I 2022-10-27 07:49:17] ...trap/engines/engine.py.321: eval: epoch 55 | batch 220/510 [I 2022-10-27 07:49:17] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:24 [I 2022-10-27 07:49:17] ...trap/engines/engine.py.326: process: 0.29156 | load: 0.00017 [I 2022-10-27 07:49:19] ...trap/engines/engine.py.321: eval: epoch 55 | batch 230/510 [I 2022-10-27 07:49:19] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:21 [I 2022-10-27 07:49:19] ...trap/engines/engine.py.326: process: 0.29220 | load: 0.00015 [I 2022-10-27 07:49:22] ...trap/engines/engine.py.321: eval: epoch 55 | batch 240/510 [I 2022-10-27 07:49:22] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 07:49:22] ...trap/engines/engine.py.326: process: 0.28608 | load: 0.00020 [I 2022-10-27 07:49:25] ...trap/engines/engine.py.321: eval: epoch 55 | batch 250/510 [I 2022-10-27 07:49:25] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:16 [I 2022-10-27 07:49:25] ...trap/engines/engine.py.326: process: 0.28347 | load: 0.00022 [I 2022-10-27 07:49:28] ...trap/engines/engine.py.321: eval: epoch 55 | batch 260/510 [I 2022-10-27 07:49:28] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 07:49:28] ...trap/engines/engine.py.326: process: 0.28835 | load: 0.00023 [I 2022-10-27 07:49:31] ...trap/engines/engine.py.321: eval: epoch 55 | batch 270/510 [I 2022-10-27 07:49:31] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 07:49:31] ...trap/engines/engine.py.326: process: 0.29018 | load: 0.00023 [I 2022-10-27 07:49:34] ...trap/engines/engine.py.321: eval: epoch 55 | batch 280/510 [I 2022-10-27 07:49:34] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:07 [I 2022-10-27 07:49:34] ...trap/engines/engine.py.326: process: 0.28551 | load: 0.00021 [I 2022-10-27 07:49:37] ...trap/engines/engine.py.321: eval: epoch 55 | batch 290/510 [I 2022-10-27 07:49:37] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:04 [I 2022-10-27 07:49:37] ...trap/engines/engine.py.326: process: 0.28806 | load: 0.00014 [I 2022-10-27 07:49:40] ...trap/engines/engine.py.321: eval: epoch 55 | batch 300/510 [I 2022-10-27 07:49:40] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:02 [I 2022-10-27 07:49:40] ...trap/engines/engine.py.326: process: 0.28732 | load: 0.00025 [I 2022-10-27 07:49:43] ...trap/engines/engine.py.321: eval: epoch 55 | batch 310/510 [I 2022-10-27 07:49:43] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 07:49:43] ...trap/engines/engine.py.326: process: 0.28981 | load: 0.00015 [I 2022-10-27 07:49:46] ...trap/engines/engine.py.321: eval: epoch 55 | batch 320/510 [I 2022-10-27 07:49:46] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 07:49:46] ...trap/engines/engine.py.326: process: 0.29098 | load: 0.00021 [I 2022-10-27 07:49:49] ...trap/engines/engine.py.321: eval: epoch 55 | batch 330/510 [I 2022-10-27 07:49:49] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:52 [I 2022-10-27 07:49:49] ...trap/engines/engine.py.326: process: 0.28792 | load: 0.00018 [I 2022-10-27 07:49:52] ...trap/engines/engine.py.321: eval: epoch 55 | batch 340/510 [I 2022-10-27 07:49:52] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-27 07:49:52] ...trap/engines/engine.py.326: process: 0.28168 | load: 0.00019 [I 2022-10-27 07:49:55] ...trap/engines/engine.py.321: eval: epoch 55 | batch 350/510 [I 2022-10-27 07:49:55] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 07:49:55] ...trap/engines/engine.py.326: process: 0.29034 | load: 0.00021 [I 2022-10-27 07:49:57] ...trap/engines/engine.py.321: eval: epoch 55 | batch 360/510 [I 2022-10-27 07:49:57] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 07:49:57] ...trap/engines/engine.py.326: process: 0.28630 | load: 0.00015 [I 2022-10-27 07:50:00] ...trap/engines/engine.py.321: eval: epoch 55 | batch 370/510 [I 2022-10-27 07:50:00] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 07:50:00] ...trap/engines/engine.py.326: process: 0.29023 | load: 0.00015 [I 2022-10-27 07:50:03] ...trap/engines/engine.py.321: eval: epoch 55 | batch 380/510 [I 2022-10-27 07:50:03] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 07:50:03] ...trap/engines/engine.py.326: process: 0.29291 | load: 0.00020 [I 2022-10-27 07:50:06] ...trap/engines/engine.py.321: eval: epoch 55 | batch 390/510 [I 2022-10-27 07:50:06] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 07:50:06] ...trap/engines/engine.py.326: process: 0.28051 | load: 0.00018 [I 2022-10-27 07:50:09] ...trap/engines/engine.py.321: eval: epoch 55 | batch 400/510 [I 2022-10-27 07:50:09] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 07:50:09] ...trap/engines/engine.py.326: process: 0.29073 | load: 0.00033 [I 2022-10-27 07:50:12] ...trap/engines/engine.py.321: eval: epoch 55 | batch 410/510 [I 2022-10-27 07:50:12] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 07:50:12] ...trap/engines/engine.py.326: process: 0.28417 | load: 0.00020 [I 2022-10-27 07:50:15] ...trap/engines/engine.py.321: eval: epoch 55 | batch 420/510 [I 2022-10-27 07:50:15] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-27 07:50:15] ...trap/engines/engine.py.326: process: 0.31036 | load: 0.00015 [I 2022-10-27 07:50:18] ...trap/engines/engine.py.321: eval: epoch 55 | batch 430/510 [I 2022-10-27 07:50:18] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 07:50:18] ...trap/engines/engine.py.326: process: 0.28534 | load: 0.00019 [I 2022-10-27 07:50:21] ...trap/engines/engine.py.321: eval: epoch 55 | batch 440/510 [I 2022-10-27 07:50:21] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 07:50:21] ...trap/engines/engine.py.326: process: 0.28790 | load: 0.00025 [I 2022-10-27 07:50:24] ...trap/engines/engine.py.321: eval: epoch 55 | batch 450/510 [I 2022-10-27 07:50:24] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:18 [I 2022-10-27 07:50:24] ...trap/engines/engine.py.326: process: 0.28770 | load: 0.00017 [I 2022-10-27 07:50:27] ...trap/engines/engine.py.321: eval: epoch 55 | batch 460/510 [I 2022-10-27 07:50:27] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 07:50:27] ...trap/engines/engine.py.326: process: 0.28788 | load: 0.00017 [I 2022-10-27 07:50:29] ...trap/engines/engine.py.321: eval: epoch 55 | batch 470/510 [I 2022-10-27 07:50:29] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 07:50:29] ...trap/engines/engine.py.326: process: 0.28862 | load: 0.00022 [I 2022-10-27 07:50:32] ...trap/engines/engine.py.321: eval: epoch 55 | batch 480/510 [I 2022-10-27 07:50:32] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 07:50:32] ...trap/engines/engine.py.326: process: 0.28812 | load: 0.00016 [I 2022-10-27 07:50:35] ...trap/engines/engine.py.321: eval: epoch 55 | batch 490/510 [I 2022-10-27 07:50:35] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 07:50:35] ...trap/engines/engine.py.326: process: 0.28360 | load: 0.00019 [I 2022-10-27 07:50:38] ...trap/engines/engine.py.321: eval: epoch 55 | batch 500/510 [I 2022-10-27 07:50:38] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 07:50:38] ...trap/engines/engine.py.326: process: 0.29277 | load: 0.00017 [I 2022-10-27 07:50:41] ...trap/engines/engine.py.321: eval: epoch 55 | batch 510/510 [I 2022-10-27 07:50:41] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 07:50:41] ...trap/engines/engine.py.326: process: 0.28803 | load: 0.00015 [S 2022-10-27 07:50:42] ...trap/engines/engine.py.132: eval_epoch.epoch: 55 [I 2022-10-27 07:50:42] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.707300 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.903700 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936900 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.701100 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.901400 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933600 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015100 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007590 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007582 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017683 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008947 [S 2022-10-27 07:50:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006711 [I 2022-10-27 07:50:47] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 07:50:47] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 07:50:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 07:50:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 07:50:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 07:50:50] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 07:50:52] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 07:50:58] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 07:50:58] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 07:50:58] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 07:51:01] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 07:51:07] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 07:51:07] ...trap/engines/engine.py.163: Training model on trainset for epoch 56 [I 2022-10-27 07:51:07] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 07:51:09] ...trap/engines/engine.py.224: train: epoch 56 | batch 0/2304 [I 2022-10-27 07:51:09] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:35 [I 2022-10-27 07:51:09] ...trap/engines/engine.py.229: process: 2.67177 | load: 2.02655 [I 2022-10-27 07:51:09] ...trap/engines/engine.py.230: loss: 0.10532 [I 2022-10-27 07:51:09] ...optimizers/trijoint.py.151: total_norm: 12.156528 [I 2022-10-27 07:51:15] ...trap/engines/engine.py.224: train: epoch 56 | batch 10/2304 [I 2022-10-27 07:51:15] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:02 [I 2022-10-27 07:51:15] ...trap/engines/engine.py.229: process: 0.56484 | load: 0.00309 [I 2022-10-27 07:51:15] ...trap/engines/engine.py.230: loss: 0.11340 [I 2022-10-27 07:51:15] ...optimizers/trijoint.py.151: total_norm: 8.004398 [I 2022-10-27 07:51:20] ...trap/engines/engine.py.224: train: epoch 56 | batch 20/2304 [I 2022-10-27 07:51:20] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:09 [I 2022-10-27 07:51:20] ...trap/engines/engine.py.229: process: 0.56389 | load: 0.00408 [I 2022-10-27 07:51:20] ...trap/engines/engine.py.230: loss: 0.15116 [I 2022-10-27 07:51:20] ...optimizers/trijoint.py.151: total_norm: 9.834578 [I 2022-10-27 07:51:26] ...trap/engines/engine.py.224: train: epoch 56 | batch 30/2304 [I 2022-10-27 07:51:26] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:19 [I 2022-10-27 07:51:26] ...trap/engines/engine.py.229: process: 0.57444 | load: 0.00332 [I 2022-10-27 07:51:26] ...trap/engines/engine.py.230: loss: 0.26875 [I 2022-10-27 07:51:26] ...optimizers/trijoint.py.151: total_norm: 18.354137 [I 2022-10-27 07:51:32] ...trap/engines/engine.py.224: train: epoch 56 | batch 40/2304 [I 2022-10-27 07:51:32] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:59 [I 2022-10-27 07:51:32] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00283 [I 2022-10-27 07:51:32] ...trap/engines/engine.py.230: loss: 0.14294 [I 2022-10-27 07:51:32] ...optimizers/trijoint.py.151: total_norm: 20.877226 [I 2022-10-27 07:51:37] ...trap/engines/engine.py.224: train: epoch 56 | batch 50/2304 [I 2022-10-27 07:51:37] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:00 [I 2022-10-27 07:51:37] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00269 [I 2022-10-27 07:51:37] ...trap/engines/engine.py.230: loss: 0.12925 [I 2022-10-27 07:51:37] ...optimizers/trijoint.py.151: total_norm: 14.531793 [I 2022-10-27 07:51:43] ...trap/engines/engine.py.224: train: epoch 56 | batch 60/2304 [I 2022-10-27 07:51:43] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:48 [I 2022-10-27 07:51:43] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00257 [I 2022-10-27 07:51:43] ...trap/engines/engine.py.230: loss: 0.19231 [I 2022-10-27 07:51:43] ...optimizers/trijoint.py.151: total_norm: 13.342943 [I 2022-10-27 07:51:48] ...trap/engines/engine.py.224: train: epoch 56 | batch 70/2304 [I 2022-10-27 07:51:48] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:41 [I 2022-10-27 07:51:48] ...trap/engines/engine.py.229: process: 0.54881 | load: 0.00274 [I 2022-10-27 07:51:48] ...trap/engines/engine.py.230: loss: 0.17972 [I 2022-10-27 07:51:48] ...optimizers/trijoint.py.151: total_norm: 19.162012 [I 2022-10-27 07:51:54] ...trap/engines/engine.py.224: train: epoch 56 | batch 80/2304 [I 2022-10-27 07:51:54] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:30 [I 2022-10-27 07:51:54] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00275 [I 2022-10-27 07:51:54] ...trap/engines/engine.py.230: loss: 0.13898 [I 2022-10-27 07:51:54] ...optimizers/trijoint.py.151: total_norm: 16.933138 [I 2022-10-27 07:51:59] ...trap/engines/engine.py.224: train: epoch 56 | batch 90/2304 [I 2022-10-27 07:51:59] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:36 [I 2022-10-27 07:51:59] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00254 [I 2022-10-27 07:51:59] ...trap/engines/engine.py.230: loss: 0.17254 [I 2022-10-27 07:51:59] ...optimizers/trijoint.py.151: total_norm: 16.890144 [I 2022-10-27 07:52:05] ...trap/engines/engine.py.224: train: epoch 56 | batch 100/2304 [I 2022-10-27 07:52:05] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:30 [I 2022-10-27 07:52:05] ...trap/engines/engine.py.229: process: 0.56718 | load: 0.00273 [I 2022-10-27 07:52:05] ...trap/engines/engine.py.230: loss: 0.15152 [I 2022-10-27 07:52:05] ...optimizers/trijoint.py.151: total_norm: 19.733747 [I 2022-10-27 07:52:11] ...trap/engines/engine.py.224: train: epoch 56 | batch 110/2304 [I 2022-10-27 07:52:11] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:25 [I 2022-10-27 07:52:11] ...trap/engines/engine.py.229: process: 0.57243 | load: 0.00243 [I 2022-10-27 07:52:11] ...trap/engines/engine.py.230: loss: 0.17262 [I 2022-10-27 07:52:11] ...optimizers/trijoint.py.151: total_norm: 20.921223 [I 2022-10-27 07:52:16] ...trap/engines/engine.py.224: train: epoch 56 | batch 120/2304 [I 2022-10-27 07:52:16] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:18 [I 2022-10-27 07:52:16] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00313 [I 2022-10-27 07:52:16] ...trap/engines/engine.py.230: loss: 0.16019 [I 2022-10-27 07:52:16] ...optimizers/trijoint.py.151: total_norm: 16.380465 [I 2022-10-27 07:52:22] ...trap/engines/engine.py.224: train: epoch 56 | batch 130/2304 [I 2022-10-27 07:52:22] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:16 [I 2022-10-27 07:52:22] ...trap/engines/engine.py.229: process: 0.55979 | load: 0.00261 [I 2022-10-27 07:52:22] ...trap/engines/engine.py.230: loss: 0.13115 [I 2022-10-27 07:52:22] ...optimizers/trijoint.py.151: total_norm: 12.629358 [I 2022-10-27 07:52:27] ...trap/engines/engine.py.224: train: epoch 56 | batch 140/2304 [I 2022-10-27 07:52:27] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:05 [I 2022-10-27 07:52:27] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00270 [I 2022-10-27 07:52:27] ...trap/engines/engine.py.230: loss: 0.14060 [I 2022-10-27 07:52:27] ...optimizers/trijoint.py.151: total_norm: 8.096193 [I 2022-10-27 07:52:33] ...trap/engines/engine.py.224: train: epoch 56 | batch 150/2304 [I 2022-10-27 07:52:33] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:01 [I 2022-10-27 07:52:33] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00251 [I 2022-10-27 07:52:33] ...trap/engines/engine.py.230: loss: 0.23184 [I 2022-10-27 07:52:33] ...optimizers/trijoint.py.151: total_norm: 15.604936 [I 2022-10-27 07:52:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 160/2304 [I 2022-10-27 07:52:38] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:45 [I 2022-10-27 07:52:38] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00266 [I 2022-10-27 07:52:38] ...trap/engines/engine.py.230: loss: 0.11790 [I 2022-10-27 07:52:38] ...optimizers/trijoint.py.151: total_norm: 11.230293 [I 2022-10-27 07:52:44] ...trap/engines/engine.py.224: train: epoch 56 | batch 170/2304 [I 2022-10-27 07:52:44] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:52 [I 2022-10-27 07:52:44] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00265 [I 2022-10-27 07:52:44] ...trap/engines/engine.py.230: loss: 0.13449 [I 2022-10-27 07:52:44] ...optimizers/trijoint.py.151: total_norm: 13.087867 [I 2022-10-27 07:52:50] ...trap/engines/engine.py.224: train: epoch 56 | batch 180/2304 [I 2022-10-27 07:52:50] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:41 [I 2022-10-27 07:52:50] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00258 [I 2022-10-27 07:52:50] ...trap/engines/engine.py.230: loss: 0.11846 [I 2022-10-27 07:52:50] ...optimizers/trijoint.py.151: total_norm: 11.480909 [I 2022-10-27 07:52:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 190/2304 [I 2022-10-27 07:52:55] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:45 [I 2022-10-27 07:52:55] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00269 [I 2022-10-27 07:52:55] ...trap/engines/engine.py.230: loss: 0.18886 [I 2022-10-27 07:52:55] ...optimizers/trijoint.py.151: total_norm: 12.912001 [I 2022-10-27 07:53:01] ...trap/engines/engine.py.224: train: epoch 56 | batch 200/2304 [I 2022-10-27 07:53:01] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:38 [I 2022-10-27 07:53:01] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00254 [I 2022-10-27 07:53:01] ...trap/engines/engine.py.230: loss: 0.14799 [I 2022-10-27 07:53:01] ...optimizers/trijoint.py.151: total_norm: 13.819438 [I 2022-10-27 07:53:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 210/2304 [I 2022-10-27 07:53:06] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:30 [I 2022-10-27 07:53:06] ...trap/engines/engine.py.229: process: 0.55305 | load: 0.00245 [I 2022-10-27 07:53:06] ...trap/engines/engine.py.230: loss: 0.12851 [I 2022-10-27 07:53:06] ...optimizers/trijoint.py.151: total_norm: 61.178402 [I 2022-10-27 07:53:12] ...trap/engines/engine.py.224: train: epoch 56 | batch 220/2304 [I 2022-10-27 07:53:12] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:27 [I 2022-10-27 07:53:12] ...trap/engines/engine.py.229: process: 0.57765 | load: 0.00246 [I 2022-10-27 07:53:12] ...trap/engines/engine.py.230: loss: 0.19797 [I 2022-10-27 07:53:12] ...optimizers/trijoint.py.151: total_norm: 23.416681 [I 2022-10-27 07:53:18] ...trap/engines/engine.py.224: train: epoch 56 | batch 230/2304 [I 2022-10-27 07:53:18] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:18 [I 2022-10-27 07:53:18] ...trap/engines/engine.py.229: process: 0.56294 | load: 0.00261 [I 2022-10-27 07:53:18] ...trap/engines/engine.py.230: loss: 0.13361 [I 2022-10-27 07:53:18] ...optimizers/trijoint.py.151: total_norm: 8.565531 [I 2022-10-27 07:53:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 240/2304 [I 2022-10-27 07:53:23] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:08 [I 2022-10-27 07:53:23] ...trap/engines/engine.py.229: process: 0.55895 | load: 0.00254 [I 2022-10-27 07:53:23] ...trap/engines/engine.py.230: loss: 0.18872 [I 2022-10-27 07:53:23] ...optimizers/trijoint.py.151: total_norm: 19.561674 [I 2022-10-27 07:53:29] ...trap/engines/engine.py.224: train: epoch 56 | batch 250/2304 [I 2022-10-27 07:53:29] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:05 [I 2022-10-27 07:53:29] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00255 [I 2022-10-27 07:53:29] ...trap/engines/engine.py.230: loss: 0.18886 [I 2022-10-27 07:53:29] ...optimizers/trijoint.py.151: total_norm: 26.552490 [I 2022-10-27 07:53:34] ...trap/engines/engine.py.224: train: epoch 56 | batch 260/2304 [I 2022-10-27 07:53:34] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:05 [I 2022-10-27 07:53:34] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00244 [I 2022-10-27 07:53:34] ...trap/engines/engine.py.230: loss: 0.11979 [I 2022-10-27 07:53:34] ...optimizers/trijoint.py.151: total_norm: 43.934273 [I 2022-10-27 07:53:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 270/2304 [I 2022-10-27 07:53:40] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:59 [I 2022-10-27 07:53:40] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00250 [I 2022-10-27 07:53:40] ...trap/engines/engine.py.230: loss: 0.12809 [I 2022-10-27 07:53:40] ...optimizers/trijoint.py.151: total_norm: 8.124309 [I 2022-10-27 07:53:46] ...trap/engines/engine.py.224: train: epoch 56 | batch 280/2304 [I 2022-10-27 07:53:46] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:53 [I 2022-10-27 07:53:46] ...trap/engines/engine.py.229: process: 0.55807 | load: 0.00252 [I 2022-10-27 07:53:46] ...trap/engines/engine.py.230: loss: 0.15198 [I 2022-10-27 07:53:46] ...optimizers/trijoint.py.151: total_norm: 36.369865 [I 2022-10-27 07:53:51] ...trap/engines/engine.py.224: train: epoch 56 | batch 290/2304 [I 2022-10-27 07:53:51] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:48 [I 2022-10-27 07:53:51] ...trap/engines/engine.py.229: process: 0.56472 | load: 0.00265 [I 2022-10-27 07:53:51] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-27 07:53:51] ...optimizers/trijoint.py.151: total_norm: 20.277719 [I 2022-10-27 07:53:57] ...trap/engines/engine.py.224: train: epoch 56 | batch 300/2304 [I 2022-10-27 07:53:57] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:39 [I 2022-10-27 07:53:57] ...trap/engines/engine.py.229: process: 0.56873 | load: 0.00255 [I 2022-10-27 07:53:57] ...trap/engines/engine.py.230: loss: 0.09500 [I 2022-10-27 07:53:57] ...optimizers/trijoint.py.151: total_norm: 11.547193 [I 2022-10-27 07:54:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 310/2304 [I 2022-10-27 07:54:02] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:35 [I 2022-10-27 07:54:02] ...trap/engines/engine.py.229: process: 0.56877 | load: 0.00295 [I 2022-10-27 07:54:02] ...trap/engines/engine.py.230: loss: 0.15296 [I 2022-10-27 07:54:02] ...optimizers/trijoint.py.151: total_norm: 22.390760 [I 2022-10-27 07:54:08] ...trap/engines/engine.py.224: train: epoch 56 | batch 320/2304 [I 2022-10-27 07:54:08] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:44 [I 2022-10-27 07:54:08] ...trap/engines/engine.py.229: process: 0.56778 | load: 0.00249 [I 2022-10-27 07:54:08] ...trap/engines/engine.py.230: loss: 0.11971 [I 2022-10-27 07:54:08] ...optimizers/trijoint.py.151: total_norm: 11.103043 [I 2022-10-27 07:54:14] ...trap/engines/engine.py.224: train: epoch 56 | batch 330/2304 [I 2022-10-27 07:54:14] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:21 [I 2022-10-27 07:54:14] ...trap/engines/engine.py.229: process: 0.56706 | load: 0.00248 [I 2022-10-27 07:54:14] ...trap/engines/engine.py.230: loss: 0.17591 [I 2022-10-27 07:54:14] ...optimizers/trijoint.py.151: total_norm: 10.572875 [I 2022-10-27 07:54:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 340/2304 [I 2022-10-27 07:54:19] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:18 [I 2022-10-27 07:54:19] ...trap/engines/engine.py.229: process: 0.54986 | load: 0.00248 [I 2022-10-27 07:54:19] ...trap/engines/engine.py.230: loss: 0.15115 [I 2022-10-27 07:54:19] ...optimizers/trijoint.py.151: total_norm: 18.839132 [I 2022-10-27 07:54:25] ...trap/engines/engine.py.224: train: epoch 56 | batch 350/2304 [I 2022-10-27 07:54:25] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:09 [I 2022-10-27 07:54:25] ...trap/engines/engine.py.229: process: 0.54862 | load: 0.00276 [I 2022-10-27 07:54:25] ...trap/engines/engine.py.230: loss: 0.10643 [I 2022-10-27 07:54:25] ...optimizers/trijoint.py.151: total_norm: 12.258416 [I 2022-10-27 07:54:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 360/2304 [I 2022-10-27 07:54:30] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:03 [I 2022-10-27 07:54:30] ...trap/engines/engine.py.229: process: 0.56273 | load: 0.00263 [I 2022-10-27 07:54:30] ...trap/engines/engine.py.230: loss: 0.16208 [I 2022-10-27 07:54:30] ...optimizers/trijoint.py.151: total_norm: 11.145655 [I 2022-10-27 07:54:36] ...trap/engines/engine.py.224: train: epoch 56 | batch 370/2304 [I 2022-10-27 07:54:36] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:06 [I 2022-10-27 07:54:36] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00257 [I 2022-10-27 07:54:36] ...trap/engines/engine.py.230: loss: 0.18657 [I 2022-10-27 07:54:36] ...optimizers/trijoint.py.151: total_norm: 11.296313 [I 2022-10-27 07:54:42] ...trap/engines/engine.py.224: train: epoch 56 | batch 380/2304 [I 2022-10-27 07:54:42] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:47 [I 2022-10-27 07:54:42] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00252 [I 2022-10-27 07:54:42] ...trap/engines/engine.py.230: loss: 0.15684 [I 2022-10-27 07:54:42] ...optimizers/trijoint.py.151: total_norm: 21.999271 [I 2022-10-27 07:54:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 390/2304 [I 2022-10-27 07:54:47] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:44 [I 2022-10-27 07:54:47] ...trap/engines/engine.py.229: process: 0.54449 | load: 0.00250 [I 2022-10-27 07:54:47] ...trap/engines/engine.py.230: loss: 0.13471 [I 2022-10-27 07:54:47] ...optimizers/trijoint.py.151: total_norm: 9.662373 [I 2022-10-27 07:54:53] ...trap/engines/engine.py.224: train: epoch 56 | batch 400/2304 [I 2022-10-27 07:54:53] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:44 [I 2022-10-27 07:54:53] ...trap/engines/engine.py.229: process: 0.55383 | load: 0.00249 [I 2022-10-27 07:54:53] ...trap/engines/engine.py.230: loss: 0.13403 [I 2022-10-27 07:54:53] ...optimizers/trijoint.py.151: total_norm: 8.384738 [I 2022-10-27 07:54:58] ...trap/engines/engine.py.224: train: epoch 56 | batch 410/2304 [I 2022-10-27 07:54:58] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:40 [I 2022-10-27 07:54:58] ...trap/engines/engine.py.229: process: 0.57010 | load: 0.00245 [I 2022-10-27 07:54:58] ...trap/engines/engine.py.230: loss: 0.14567 [I 2022-10-27 07:54:58] ...optimizers/trijoint.py.151: total_norm: 9.968159 [I 2022-10-27 07:55:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 420/2304 [I 2022-10-27 07:55:04] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:28 [I 2022-10-27 07:55:04] ...trap/engines/engine.py.229: process: 0.55258 | load: 0.00248 [I 2022-10-27 07:55:04] ...trap/engines/engine.py.230: loss: 0.18411 [I 2022-10-27 07:55:04] ...optimizers/trijoint.py.151: total_norm: 29.611059 [I 2022-10-27 07:55:10] ...trap/engines/engine.py.224: train: epoch 56 | batch 430/2304 [I 2022-10-27 07:55:10] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:22 [I 2022-10-27 07:55:10] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00251 [I 2022-10-27 07:55:10] ...trap/engines/engine.py.230: loss: 0.22739 [I 2022-10-27 07:55:10] ...optimizers/trijoint.py.151: total_norm: 24.959959 [I 2022-10-27 07:55:15] ...trap/engines/engine.py.224: train: epoch 56 | batch 440/2304 [I 2022-10-27 07:55:15] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:21 [I 2022-10-27 07:55:15] ...trap/engines/engine.py.229: process: 0.56458 | load: 0.00250 [I 2022-10-27 07:55:15] ...trap/engines/engine.py.230: loss: 0.17795 [I 2022-10-27 07:55:15] ...optimizers/trijoint.py.151: total_norm: 19.224510 [I 2022-10-27 07:55:21] ...trap/engines/engine.py.224: train: epoch 56 | batch 450/2304 [I 2022-10-27 07:55:21] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:22 [I 2022-10-27 07:55:21] ...trap/engines/engine.py.229: process: 0.56993 | load: 0.00248 [I 2022-10-27 07:55:21] ...trap/engines/engine.py.230: loss: 0.12729 [I 2022-10-27 07:55:21] ...optimizers/trijoint.py.151: total_norm: 28.191851 [I 2022-10-27 07:55:26] ...trap/engines/engine.py.224: train: epoch 56 | batch 460/2304 [I 2022-10-27 07:55:26] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:05 [I 2022-10-27 07:55:26] ...trap/engines/engine.py.229: process: 0.55081 | load: 0.00247 [I 2022-10-27 07:55:26] ...trap/engines/engine.py.230: loss: 0.14548 [I 2022-10-27 07:55:26] ...optimizers/trijoint.py.151: total_norm: 25.315248 [I 2022-10-27 07:55:32] ...trap/engines/engine.py.224: train: epoch 56 | batch 470/2304 [I 2022-10-27 07:55:32] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:03 [I 2022-10-27 07:55:32] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00254 [I 2022-10-27 07:55:32] ...trap/engines/engine.py.230: loss: 0.10645 [I 2022-10-27 07:55:32] ...optimizers/trijoint.py.151: total_norm: 6.897532 [I 2022-10-27 07:55:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 480/2304 [I 2022-10-27 07:55:38] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:00 [I 2022-10-27 07:55:38] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00251 [I 2022-10-27 07:55:38] ...trap/engines/engine.py.230: loss: 0.17796 [I 2022-10-27 07:55:38] ...optimizers/trijoint.py.151: total_norm: 23.165987 [I 2022-10-27 07:55:43] ...trap/engines/engine.py.224: train: epoch 56 | batch 490/2304 [I 2022-10-27 07:55:43] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:49 [I 2022-10-27 07:55:43] ...trap/engines/engine.py.229: process: 0.54252 | load: 0.00245 [I 2022-10-27 07:55:43] ...trap/engines/engine.py.230: loss: 0.16977 [I 2022-10-27 07:55:43] ...optimizers/trijoint.py.151: total_norm: 22.280045 [I 2022-10-27 07:55:49] ...trap/engines/engine.py.224: train: epoch 56 | batch 500/2304 [I 2022-10-27 07:55:49] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:48 [I 2022-10-27 07:55:49] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00243 [I 2022-10-27 07:55:49] ...trap/engines/engine.py.230: loss: 0.19228 [I 2022-10-27 07:55:49] ...optimizers/trijoint.py.151: total_norm: 23.817158 [I 2022-10-27 07:55:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 510/2304 [I 2022-10-27 07:55:55] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:57 [I 2022-10-27 07:55:55] ...trap/engines/engine.py.229: process: 0.56966 | load: 0.00248 [I 2022-10-27 07:55:55] ...trap/engines/engine.py.230: loss: 0.17659 [I 2022-10-27 07:55:55] ...optimizers/trijoint.py.151: total_norm: 24.950190 [I 2022-10-27 07:56:00] ...trap/engines/engine.py.224: train: epoch 56 | batch 520/2304 [I 2022-10-27 07:56:00] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:56 [I 2022-10-27 07:56:00] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00253 [I 2022-10-27 07:56:00] ...trap/engines/engine.py.230: loss: 0.12905 [I 2022-10-27 07:56:00] ...optimizers/trijoint.py.151: total_norm: 19.125059 [I 2022-10-27 07:56:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 530/2304 [I 2022-10-27 07:56:06] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:34 [I 2022-10-27 07:56:06] ...trap/engines/engine.py.229: process: 0.56712 | load: 0.00244 [I 2022-10-27 07:56:06] ...trap/engines/engine.py.230: loss: 0.15556 [I 2022-10-27 07:56:06] ...optimizers/trijoint.py.151: total_norm: 13.059616 [I 2022-10-27 07:56:11] ...trap/engines/engine.py.224: train: epoch 56 | batch 540/2304 [I 2022-10-27 07:56:11] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:29 [I 2022-10-27 07:56:11] ...trap/engines/engine.py.229: process: 0.55997 | load: 0.00243 [I 2022-10-27 07:56:11] ...trap/engines/engine.py.230: loss: 0.15726 [I 2022-10-27 07:56:11] ...optimizers/trijoint.py.151: total_norm: 18.413273 [I 2022-10-27 07:56:17] ...trap/engines/engine.py.224: train: epoch 56 | batch 550/2304 [I 2022-10-27 07:56:17] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:26 [I 2022-10-27 07:56:17] ...trap/engines/engine.py.229: process: 0.57340 | load: 0.00244 [I 2022-10-27 07:56:17] ...trap/engines/engine.py.230: loss: 0.14245 [I 2022-10-27 07:56:17] ...optimizers/trijoint.py.151: total_norm: 18.330019 [I 2022-10-27 07:56:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 560/2304 [I 2022-10-27 07:56:23] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:11 [I 2022-10-27 07:56:23] ...trap/engines/engine.py.229: process: 0.54803 | load: 0.00246 [I 2022-10-27 07:56:23] ...trap/engines/engine.py.230: loss: 0.17146 [I 2022-10-27 07:56:23] ...optimizers/trijoint.py.151: total_norm: 19.584211 [I 2022-10-27 07:56:28] ...trap/engines/engine.py.224: train: epoch 56 | batch 570/2304 [I 2022-10-27 07:56:28] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:12 [I 2022-10-27 07:56:28] ...trap/engines/engine.py.229: process: 0.56639 | load: 0.00244 [I 2022-10-27 07:56:28] ...trap/engines/engine.py.230: loss: 0.21385 [I 2022-10-27 07:56:28] ...optimizers/trijoint.py.151: total_norm: 21.176567 [I 2022-10-27 07:56:34] ...trap/engines/engine.py.224: train: epoch 56 | batch 580/2304 [I 2022-10-27 07:56:34] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:05 [I 2022-10-27 07:56:34] ...trap/engines/engine.py.229: process: 0.55485 | load: 0.00244 [I 2022-10-27 07:56:34] ...trap/engines/engine.py.230: loss: 0.13227 [I 2022-10-27 07:56:34] ...optimizers/trijoint.py.151: total_norm: 23.740335 [I 2022-10-27 07:56:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 590/2304 [I 2022-10-27 07:56:40] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:11 [I 2022-10-27 07:56:40] ...trap/engines/engine.py.229: process: 0.56130 | load: 0.00242 [I 2022-10-27 07:56:40] ...trap/engines/engine.py.230: loss: 0.14490 [I 2022-10-27 07:56:40] ...optimizers/trijoint.py.151: total_norm: 18.281700 [I 2022-10-27 07:56:45] ...trap/engines/engine.py.224: train: epoch 56 | batch 600/2304 [I 2022-10-27 07:56:45] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:59 [I 2022-10-27 07:56:45] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00243 [I 2022-10-27 07:56:45] ...trap/engines/engine.py.230: loss: 0.18016 [I 2022-10-27 07:56:45] ...optimizers/trijoint.py.151: total_norm: 25.653770 [I 2022-10-27 07:56:51] ...trap/engines/engine.py.224: train: epoch 56 | batch 610/2304 [I 2022-10-27 07:56:51] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:53 [I 2022-10-27 07:56:51] ...trap/engines/engine.py.229: process: 0.57352 | load: 0.00243 [I 2022-10-27 07:56:51] ...trap/engines/engine.py.230: loss: 0.12029 [I 2022-10-27 07:56:51] ...optimizers/trijoint.py.151: total_norm: 16.261667 [I 2022-10-27 07:56:56] ...trap/engines/engine.py.224: train: epoch 56 | batch 620/2304 [I 2022-10-27 07:56:56] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:42 [I 2022-10-27 07:56:56] ...trap/engines/engine.py.229: process: 0.56414 | load: 0.00244 [I 2022-10-27 07:56:56] ...trap/engines/engine.py.230: loss: 0.13952 [I 2022-10-27 07:56:56] ...optimizers/trijoint.py.151: total_norm: 26.535776 [I 2022-10-27 07:57:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 630/2304 [I 2022-10-27 07:57:02] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:36 [I 2022-10-27 07:57:02] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00241 [I 2022-10-27 07:57:02] ...trap/engines/engine.py.230: loss: 0.16591 [I 2022-10-27 07:57:02] ...optimizers/trijoint.py.151: total_norm: 17.844238 [I 2022-10-27 07:57:08] ...trap/engines/engine.py.224: train: epoch 56 | batch 640/2304 [I 2022-10-27 07:57:08] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:32 [I 2022-10-27 07:57:08] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00263 [I 2022-10-27 07:57:08] ...trap/engines/engine.py.230: loss: 0.19387 [I 2022-10-27 07:57:08] ...optimizers/trijoint.py.151: total_norm: 65.859436 [I 2022-10-27 07:57:13] ...trap/engines/engine.py.224: train: epoch 56 | batch 650/2304 [I 2022-10-27 07:57:13] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:24 [I 2022-10-27 07:57:13] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00249 [I 2022-10-27 07:57:13] ...trap/engines/engine.py.230: loss: 0.16402 [I 2022-10-27 07:57:13] ...optimizers/trijoint.py.151: total_norm: 15.414752 [I 2022-10-27 07:57:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 660/2304 [I 2022-10-27 07:57:19] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:17 [I 2022-10-27 07:57:19] ...trap/engines/engine.py.229: process: 0.55786 | load: 0.00245 [I 2022-10-27 07:57:19] ...trap/engines/engine.py.230: loss: 0.18459 [I 2022-10-27 07:57:19] ...optimizers/trijoint.py.151: total_norm: 28.006432 [I 2022-10-27 07:57:24] ...trap/engines/engine.py.224: train: epoch 56 | batch 670/2304 [I 2022-10-27 07:57:24] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:16 [I 2022-10-27 07:57:24] ...trap/engines/engine.py.229: process: 0.56002 | load: 0.00247 [I 2022-10-27 07:57:24] ...trap/engines/engine.py.230: loss: 0.11763 [I 2022-10-27 07:57:24] ...optimizers/trijoint.py.151: total_norm: 11.277247 [I 2022-10-27 07:57:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 680/2304 [I 2022-10-27 07:57:30] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:01 [I 2022-10-27 07:57:30] ...trap/engines/engine.py.229: process: 0.55492 | load: 0.00243 [I 2022-10-27 07:57:30] ...trap/engines/engine.py.230: loss: 0.13316 [I 2022-10-27 07:57:30] ...optimizers/trijoint.py.151: total_norm: 15.359552 [I 2022-10-27 07:57:36] ...trap/engines/engine.py.224: train: epoch 56 | batch 690/2304 [I 2022-10-27 07:57:36] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:14:59 [I 2022-10-27 07:57:36] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00241 [I 2022-10-27 07:57:36] ...trap/engines/engine.py.230: loss: 0.21724 [I 2022-10-27 07:57:36] ...optimizers/trijoint.py.151: total_norm: 13.636001 [I 2022-10-27 07:57:41] ...trap/engines/engine.py.224: train: epoch 56 | batch 700/2304 [I 2022-10-27 07:57:41] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:58 [I 2022-10-27 07:57:41] ...trap/engines/engine.py.229: process: 0.56632 | load: 0.00245 [I 2022-10-27 07:57:41] ...trap/engines/engine.py.230: loss: 0.23549 [I 2022-10-27 07:57:41] ...optimizers/trijoint.py.151: total_norm: 32.942791 [I 2022-10-27 07:57:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 710/2304 [I 2022-10-27 07:57:47] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:59 [I 2022-10-27 07:57:47] ...trap/engines/engine.py.229: process: 0.57268 | load: 0.00244 [I 2022-10-27 07:57:47] ...trap/engines/engine.py.230: loss: 0.14249 [I 2022-10-27 07:57:47] ...optimizers/trijoint.py.151: total_norm: 10.435460 [I 2022-10-27 07:57:53] ...trap/engines/engine.py.224: train: epoch 56 | batch 720/2304 [I 2022-10-27 07:57:53] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:15:07 [I 2022-10-27 07:57:53] ...trap/engines/engine.py.229: process: 0.55033 | load: 0.00243 [I 2022-10-27 07:57:53] ...trap/engines/engine.py.230: loss: 0.09807 [I 2022-10-27 07:57:53] ...optimizers/trijoint.py.151: total_norm: 7.216416 [I 2022-10-27 07:57:58] ...trap/engines/engine.py.224: train: epoch 56 | batch 730/2304 [I 2022-10-27 07:57:58] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:40 [I 2022-10-27 07:57:58] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00245 [I 2022-10-27 07:57:58] ...trap/engines/engine.py.230: loss: 0.13427 [I 2022-10-27 07:57:58] ...optimizers/trijoint.py.151: total_norm: 27.061167 [I 2022-10-27 07:58:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 740/2304 [I 2022-10-27 07:58:04] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:32 [I 2022-10-27 07:58:04] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00241 [I 2022-10-27 07:58:04] ...trap/engines/engine.py.230: loss: 0.15695 [I 2022-10-27 07:58:04] ...optimizers/trijoint.py.151: total_norm: 10.277581 [I 2022-10-27 07:58:09] ...trap/engines/engine.py.224: train: epoch 56 | batch 750/2304 [I 2022-10-27 07:58:09] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:34 [I 2022-10-27 07:58:09] ...trap/engines/engine.py.229: process: 0.56681 | load: 0.00246 [I 2022-10-27 07:58:09] ...trap/engines/engine.py.230: loss: 0.17348 [I 2022-10-27 07:58:09] ...optimizers/trijoint.py.151: total_norm: 18.828665 [I 2022-10-27 07:58:15] ...trap/engines/engine.py.224: train: epoch 56 | batch 760/2304 [I 2022-10-27 07:58:15] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:26 [I 2022-10-27 07:58:15] ...trap/engines/engine.py.229: process: 0.55265 | load: 0.00246 [I 2022-10-27 07:58:15] ...trap/engines/engine.py.230: loss: 0.20285 [I 2022-10-27 07:58:15] ...optimizers/trijoint.py.151: total_norm: 25.419292 [I 2022-10-27 07:58:21] ...trap/engines/engine.py.224: train: epoch 56 | batch 770/2304 [I 2022-10-27 07:58:21] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:15 [I 2022-10-27 07:58:21] ...trap/engines/engine.py.229: process: 0.56286 | load: 0.00247 [I 2022-10-27 07:58:21] ...trap/engines/engine.py.230: loss: 0.10474 [I 2022-10-27 07:58:21] ...optimizers/trijoint.py.151: total_norm: 10.947521 [I 2022-10-27 07:58:26] ...trap/engines/engine.py.224: train: epoch 56 | batch 780/2304 [I 2022-10-27 07:58:26] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:11 [I 2022-10-27 07:58:26] ...trap/engines/engine.py.229: process: 0.56119 | load: 0.00245 [I 2022-10-27 07:58:26] ...trap/engines/engine.py.230: loss: 0.20233 [I 2022-10-27 07:58:26] ...optimizers/trijoint.py.151: total_norm: 14.139473 [I 2022-10-27 07:58:32] ...trap/engines/engine.py.224: train: epoch 56 | batch 790/2304 [I 2022-10-27 07:58:32] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:14 [I 2022-10-27 07:58:32] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00251 [I 2022-10-27 07:58:32] ...trap/engines/engine.py.230: loss: 0.10020 [I 2022-10-27 07:58:32] ...optimizers/trijoint.py.151: total_norm: 12.045485 [I 2022-10-27 07:58:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 800/2304 [I 2022-10-27 07:58:38] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:14:03 [I 2022-10-27 07:58:38] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00247 [I 2022-10-27 07:58:38] ...trap/engines/engine.py.230: loss: 0.19069 [I 2022-10-27 07:58:38] ...optimizers/trijoint.py.151: total_norm: 29.496552 [I 2022-10-27 07:58:43] ...trap/engines/engine.py.224: train: epoch 56 | batch 810/2304 [I 2022-10-27 07:58:43] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:56 [I 2022-10-27 07:58:43] ...trap/engines/engine.py.229: process: 0.56686 | load: 0.00245 [I 2022-10-27 07:58:43] ...trap/engines/engine.py.230: loss: 0.14344 [I 2022-10-27 07:58:43] ...optimizers/trijoint.py.151: total_norm: 25.840515 [I 2022-10-27 07:58:49] ...trap/engines/engine.py.224: train: epoch 56 | batch 820/2304 [I 2022-10-27 07:58:49] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:40 [I 2022-10-27 07:58:49] ...trap/engines/engine.py.229: process: 0.55014 | load: 0.00244 [I 2022-10-27 07:58:49] ...trap/engines/engine.py.230: loss: 0.19688 [I 2022-10-27 07:58:49] ...optimizers/trijoint.py.151: total_norm: 16.487671 [I 2022-10-27 07:58:54] ...trap/engines/engine.py.224: train: epoch 56 | batch 830/2304 [I 2022-10-27 07:58:54] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:38 [I 2022-10-27 07:58:54] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00248 [I 2022-10-27 07:58:54] ...trap/engines/engine.py.230: loss: 0.16389 [I 2022-10-27 07:58:54] ...optimizers/trijoint.py.151: total_norm: 21.516546 [I 2022-10-27 07:59:00] ...trap/engines/engine.py.224: train: epoch 56 | batch 840/2304 [I 2022-10-27 07:59:00] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:40 [I 2022-10-27 07:59:00] ...trap/engines/engine.py.229: process: 0.56781 | load: 0.00248 [I 2022-10-27 07:59:00] ...trap/engines/engine.py.230: loss: 0.15734 [I 2022-10-27 07:59:00] ...optimizers/trijoint.py.151: total_norm: 8.759895 [I 2022-10-27 07:59:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 850/2304 [I 2022-10-27 07:59:06] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:42 [I 2022-10-27 07:59:06] ...trap/engines/engine.py.229: process: 0.54977 | load: 0.00247 [I 2022-10-27 07:59:06] ...trap/engines/engine.py.230: loss: 0.17252 [I 2022-10-27 07:59:06] ...optimizers/trijoint.py.151: total_norm: 17.511152 [I 2022-10-27 07:59:11] ...trap/engines/engine.py.224: train: epoch 56 | batch 860/2304 [I 2022-10-27 07:59:11] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:31 [I 2022-10-27 07:59:11] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00246 [I 2022-10-27 07:59:11] ...trap/engines/engine.py.230: loss: 0.14857 [I 2022-10-27 07:59:11] ...optimizers/trijoint.py.151: total_norm: 16.442892 [I 2022-10-27 07:59:17] ...trap/engines/engine.py.224: train: epoch 56 | batch 870/2304 [I 2022-10-27 07:59:17] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:18 [I 2022-10-27 07:59:17] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00244 [I 2022-10-27 07:59:17] ...trap/engines/engine.py.230: loss: 0.15680 [I 2022-10-27 07:59:17] ...optimizers/trijoint.py.151: total_norm: 16.114069 [I 2022-10-27 07:59:22] ...trap/engines/engine.py.224: train: epoch 56 | batch 880/2304 [I 2022-10-27 07:59:22] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:16 [I 2022-10-27 07:59:22] ...trap/engines/engine.py.229: process: 0.56956 | load: 0.00249 [I 2022-10-27 07:59:22] ...trap/engines/engine.py.230: loss: 0.20949 [I 2022-10-27 07:59:22] ...optimizers/trijoint.py.151: total_norm: 22.893599 [I 2022-10-27 07:59:28] ...trap/engines/engine.py.224: train: epoch 56 | batch 890/2304 [I 2022-10-27 07:59:28] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:14 [I 2022-10-27 07:59:28] ...trap/engines/engine.py.229: process: 0.56154 | load: 0.00246 [I 2022-10-27 07:59:28] ...trap/engines/engine.py.230: loss: 0.12510 [I 2022-10-27 07:59:28] ...optimizers/trijoint.py.151: total_norm: 16.290430 [I 2022-10-27 07:59:34] ...trap/engines/engine.py.224: train: epoch 56 | batch 900/2304 [I 2022-10-27 07:59:34] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:05 [I 2022-10-27 07:59:34] ...trap/engines/engine.py.229: process: 0.54976 | load: 0.00246 [I 2022-10-27 07:59:34] ...trap/engines/engine.py.230: loss: 0.16072 [I 2022-10-27 07:59:34] ...optimizers/trijoint.py.151: total_norm: 22.833982 [I 2022-10-27 07:59:39] ...trap/engines/engine.py.224: train: epoch 56 | batch 910/2304 [I 2022-10-27 07:59:39] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:13:10 [I 2022-10-27 07:59:39] ...trap/engines/engine.py.229: process: 0.56834 | load: 0.00246 [I 2022-10-27 07:59:39] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-27 07:59:39] ...optimizers/trijoint.py.151: total_norm: 28.355579 [I 2022-10-27 07:59:45] ...trap/engines/engine.py.224: train: epoch 56 | batch 920/2304 [I 2022-10-27 07:59:45] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:53 [I 2022-10-27 07:59:45] ...trap/engines/engine.py.229: process: 0.55351 | load: 0.00243 [I 2022-10-27 07:59:45] ...trap/engines/engine.py.230: loss: 0.20948 [I 2022-10-27 07:59:45] ...optimizers/trijoint.py.151: total_norm: 34.201782 [I 2022-10-27 07:59:51] ...trap/engines/engine.py.224: train: epoch 56 | batch 930/2304 [I 2022-10-27 07:59:51] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:12:52 [I 2022-10-27 07:59:51] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00241 [I 2022-10-27 07:59:51] ...trap/engines/engine.py.230: loss: 0.15711 [I 2022-10-27 07:59:51] ...optimizers/trijoint.py.151: total_norm: 18.192858 [I 2022-10-27 07:59:56] ...trap/engines/engine.py.224: train: epoch 56 | batch 940/2304 [I 2022-10-27 07:59:56] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:42 [I 2022-10-27 07:59:56] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00241 [I 2022-10-27 07:59:56] ...trap/engines/engine.py.230: loss: 0.18574 [I 2022-10-27 07:59:56] ...optimizers/trijoint.py.151: total_norm: 21.955542 [I 2022-10-27 08:00:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 950/2304 [I 2022-10-27 08:00:02] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:36 [I 2022-10-27 08:00:02] ...trap/engines/engine.py.229: process: 0.55153 | load: 0.00241 [I 2022-10-27 08:00:02] ...trap/engines/engine.py.230: loss: 0.11482 [I 2022-10-27 08:00:02] ...optimizers/trijoint.py.151: total_norm: 14.116043 [I 2022-10-27 08:00:07] ...trap/engines/engine.py.224: train: epoch 56 | batch 960/2304 [I 2022-10-27 08:00:07] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:31 [I 2022-10-27 08:00:07] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00245 [I 2022-10-27 08:00:07] ...trap/engines/engine.py.230: loss: 0.13132 [I 2022-10-27 08:00:07] ...optimizers/trijoint.py.151: total_norm: 20.892612 [I 2022-10-27 08:00:13] ...trap/engines/engine.py.224: train: epoch 56 | batch 970/2304 [I 2022-10-27 08:00:13] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:27 [I 2022-10-27 08:00:13] ...trap/engines/engine.py.229: process: 0.56336 | load: 0.00248 [I 2022-10-27 08:00:13] ...trap/engines/engine.py.230: loss: 0.12424 [I 2022-10-27 08:00:13] ...optimizers/trijoint.py.151: total_norm: 27.146173 [I 2022-10-27 08:00:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 980/2304 [I 2022-10-27 08:00:19] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:12:33 [I 2022-10-27 08:00:19] ...trap/engines/engine.py.229: process: 0.56326 | load: 0.00257 [I 2022-10-27 08:00:19] ...trap/engines/engine.py.230: loss: 0.14462 [I 2022-10-27 08:00:19] ...optimizers/trijoint.py.151: total_norm: 23.831959 [I 2022-10-27 08:00:24] ...trap/engines/engine.py.224: train: epoch 56 | batch 990/2304 [I 2022-10-27 08:00:24] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:17 [I 2022-10-27 08:00:24] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00247 [I 2022-10-27 08:00:24] ...trap/engines/engine.py.230: loss: 0.14112 [I 2022-10-27 08:00:24] ...optimizers/trijoint.py.151: total_norm: 17.190153 [I 2022-10-27 08:00:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 1000/2304 [I 2022-10-27 08:00:30] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:07 [I 2022-10-27 08:00:30] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00243 [I 2022-10-27 08:00:30] ...trap/engines/engine.py.230: loss: 0.13000 [I 2022-10-27 08:00:30] ...optimizers/trijoint.py.151: total_norm: 16.088423 [I 2022-10-27 08:00:36] ...trap/engines/engine.py.224: train: epoch 56 | batch 1010/2304 [I 2022-10-27 08:00:36] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:12:05 [I 2022-10-27 08:00:36] ...trap/engines/engine.py.229: process: 0.55830 | load: 0.00244 [I 2022-10-27 08:00:36] ...trap/engines/engine.py.230: loss: 0.18766 [I 2022-10-27 08:00:36] ...optimizers/trijoint.py.151: total_norm: 18.142136 [I 2022-10-27 08:00:41] ...trap/engines/engine.py.224: train: epoch 56 | batch 1020/2304 [I 2022-10-27 08:00:41] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:57 [I 2022-10-27 08:00:41] ...trap/engines/engine.py.229: process: 0.55494 | load: 0.00242 [I 2022-10-27 08:00:41] ...trap/engines/engine.py.230: loss: 0.11763 [I 2022-10-27 08:00:41] ...optimizers/trijoint.py.151: total_norm: 9.992691 [I 2022-10-27 08:00:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 1030/2304 [I 2022-10-27 08:00:47] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:11:47 [I 2022-10-27 08:00:47] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00244 [I 2022-10-27 08:00:47] ...trap/engines/engine.py.230: loss: 0.11509 [I 2022-10-27 08:00:47] ...optimizers/trijoint.py.151: total_norm: 8.156430 [I 2022-10-27 08:00:52] ...trap/engines/engine.py.224: train: epoch 56 | batch 1040/2304 [I 2022-10-27 08:00:52] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:12:03 [I 2022-10-27 08:00:52] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00245 [I 2022-10-27 08:00:52] ...trap/engines/engine.py.230: loss: 0.10801 [I 2022-10-27 08:00:52] ...optimizers/trijoint.py.151: total_norm: 8.087123 [I 2022-10-27 08:00:58] ...trap/engines/engine.py.224: train: epoch 56 | batch 1050/2304 [I 2022-10-27 08:00:58] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:46 [I 2022-10-27 08:00:58] ...trap/engines/engine.py.229: process: 0.56562 | load: 0.00241 [I 2022-10-27 08:00:58] ...trap/engines/engine.py.230: loss: 0.14663 [I 2022-10-27 08:00:58] ...optimizers/trijoint.py.151: total_norm: 14.690302 [I 2022-10-27 08:01:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 1060/2304 [I 2022-10-27 08:01:04] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:33 [I 2022-10-27 08:01:04] ...trap/engines/engine.py.229: process: 0.56726 | load: 0.00264 [I 2022-10-27 08:01:04] ...trap/engines/engine.py.230: loss: 0.17678 [I 2022-10-27 08:01:04] ...optimizers/trijoint.py.151: total_norm: 14.230270 [I 2022-10-27 08:01:09] ...trap/engines/engine.py.224: train: epoch 56 | batch 1070/2304 [I 2022-10-27 08:01:09] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:26 [I 2022-10-27 08:01:09] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00245 [I 2022-10-27 08:01:09] ...trap/engines/engine.py.230: loss: 0.17659 [I 2022-10-27 08:01:09] ...optimizers/trijoint.py.151: total_norm: 26.237579 [I 2022-10-27 08:01:15] ...trap/engines/engine.py.224: train: epoch 56 | batch 1080/2304 [I 2022-10-27 08:01:15] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:24 [I 2022-10-27 08:01:15] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00250 [I 2022-10-27 08:01:15] ...trap/engines/engine.py.230: loss: 0.20794 [I 2022-10-27 08:01:15] ...optimizers/trijoint.py.151: total_norm: 47.147167 [I 2022-10-27 08:01:20] ...trap/engines/engine.py.224: train: epoch 56 | batch 1090/2304 [I 2022-10-27 08:01:20] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:12 [I 2022-10-27 08:01:20] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00249 [I 2022-10-27 08:01:20] ...trap/engines/engine.py.230: loss: 0.13560 [I 2022-10-27 08:01:20] ...optimizers/trijoint.py.151: total_norm: 13.186142 [I 2022-10-27 08:01:26] ...trap/engines/engine.py.224: train: epoch 56 | batch 1100/2304 [I 2022-10-27 08:01:26] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:09 [I 2022-10-27 08:01:26] ...trap/engines/engine.py.229: process: 0.55047 | load: 0.00247 [I 2022-10-27 08:01:26] ...trap/engines/engine.py.230: loss: 0.09550 [I 2022-10-27 08:01:26] ...optimizers/trijoint.py.151: total_norm: 9.680999 [I 2022-10-27 08:01:31] ...trap/engines/engine.py.224: train: epoch 56 | batch 1110/2304 [I 2022-10-27 08:01:31] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:10:59 [I 2022-10-27 08:01:31] ...trap/engines/engine.py.229: process: 0.54997 | load: 0.00243 [I 2022-10-27 08:01:31] ...trap/engines/engine.py.230: loss: 0.16350 [I 2022-10-27 08:01:31] ...optimizers/trijoint.py.151: total_norm: 19.092560 [I 2022-10-27 08:01:37] ...trap/engines/engine.py.224: train: epoch 56 | batch 1120/2304 [I 2022-10-27 08:01:37] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:10:56 [I 2022-10-27 08:01:37] ...trap/engines/engine.py.229: process: 0.55008 | load: 0.00246 [I 2022-10-27 08:01:37] ...trap/engines/engine.py.230: loss: 0.12883 [I 2022-10-27 08:01:37] ...optimizers/trijoint.py.151: total_norm: 13.313735 [I 2022-10-27 08:01:43] ...trap/engines/engine.py.224: train: epoch 56 | batch 1130/2304 [I 2022-10-27 08:01:43] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:10:54 [I 2022-10-27 08:01:43] ...trap/engines/engine.py.229: process: 0.55786 | load: 0.00248 [I 2022-10-27 08:01:43] ...trap/engines/engine.py.230: loss: 0.13198 [I 2022-10-27 08:01:43] ...optimizers/trijoint.py.151: total_norm: 11.416050 [I 2022-10-27 08:01:48] ...trap/engines/engine.py.224: train: epoch 56 | batch 1140/2304 [I 2022-10-27 08:01:48] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:10:44 [I 2022-10-27 08:01:48] ...trap/engines/engine.py.229: process: 0.54839 | load: 0.00244 [I 2022-10-27 08:01:48] ...trap/engines/engine.py.230: loss: 0.14591 [I 2022-10-27 08:01:48] ...optimizers/trijoint.py.151: total_norm: 13.475865 [I 2022-10-27 08:01:54] ...trap/engines/engine.py.224: train: epoch 56 | batch 1150/2304 [I 2022-10-27 08:01:54] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:37 [I 2022-10-27 08:01:54] ...trap/engines/engine.py.229: process: 0.56371 | load: 0.00245 [I 2022-10-27 08:01:54] ...trap/engines/engine.py.230: loss: 0.19230 [I 2022-10-27 08:01:54] ...optimizers/trijoint.py.151: total_norm: 18.365984 [I 2022-10-27 08:01:59] ...trap/engines/engine.py.224: train: epoch 56 | batch 1160/2304 [I 2022-10-27 08:01:59] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:41 [I 2022-10-27 08:01:59] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00246 [I 2022-10-27 08:01:59] ...trap/engines/engine.py.230: loss: 0.11526 [I 2022-10-27 08:01:59] ...optimizers/trijoint.py.151: total_norm: 13.163342 [I 2022-10-27 08:02:05] ...trap/engines/engine.py.224: train: epoch 56 | batch 1170/2304 [I 2022-10-27 08:02:05] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:32 [I 2022-10-27 08:02:05] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00250 [I 2022-10-27 08:02:05] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-27 08:02:05] ...optimizers/trijoint.py.151: total_norm: 16.200718 [I 2022-10-27 08:02:10] ...trap/engines/engine.py.224: train: epoch 56 | batch 1180/2304 [I 2022-10-27 08:02:10] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:27 [I 2022-10-27 08:02:10] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00240 [I 2022-10-27 08:02:10] ...trap/engines/engine.py.230: loss: 0.12462 [I 2022-10-27 08:02:10] ...optimizers/trijoint.py.151: total_norm: 15.524543 [I 2022-10-27 08:02:16] ...trap/engines/engine.py.224: train: epoch 56 | batch 1190/2304 [I 2022-10-27 08:02:16] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:19 [I 2022-10-27 08:02:16] ...trap/engines/engine.py.229: process: 0.56582 | load: 0.00241 [I 2022-10-27 08:02:16] ...trap/engines/engine.py.230: loss: 0.17729 [I 2022-10-27 08:02:16] ...optimizers/trijoint.py.151: total_norm: 13.511736 [I 2022-10-27 08:02:22] ...trap/engines/engine.py.224: train: epoch 56 | batch 1200/2304 [I 2022-10-27 08:02:22] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:11 [I 2022-10-27 08:02:22] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00255 [I 2022-10-27 08:02:22] ...trap/engines/engine.py.230: loss: 0.19154 [I 2022-10-27 08:02:22] ...optimizers/trijoint.py.151: total_norm: 14.337749 [I 2022-10-27 08:02:27] ...trap/engines/engine.py.224: train: epoch 56 | batch 1210/2304 [I 2022-10-27 08:02:27] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:06 [I 2022-10-27 08:02:27] ...trap/engines/engine.py.229: process: 0.54839 | load: 0.00247 [I 2022-10-27 08:02:27] ...trap/engines/engine.py.230: loss: 0.10547 [I 2022-10-27 08:02:27] ...optimizers/trijoint.py.151: total_norm: 9.410111 [I 2022-10-27 08:02:33] ...trap/engines/engine.py.224: train: epoch 56 | batch 1220/2304 [I 2022-10-27 08:02:33] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:04 [I 2022-10-27 08:02:33] ...trap/engines/engine.py.229: process: 0.56411 | load: 0.00248 [I 2022-10-27 08:02:33] ...trap/engines/engine.py.230: loss: 0.11412 [I 2022-10-27 08:02:33] ...optimizers/trijoint.py.151: total_norm: 12.581126 [I 2022-10-27 08:02:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 1230/2304 [I 2022-10-27 08:02:38] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:10:01 [I 2022-10-27 08:02:38] ...trap/engines/engine.py.229: process: 0.55250 | load: 0.00254 [I 2022-10-27 08:02:38] ...trap/engines/engine.py.230: loss: 0.11857 [I 2022-10-27 08:02:38] ...optimizers/trijoint.py.151: total_norm: 7.661024 [I 2022-10-27 08:02:44] ...trap/engines/engine.py.224: train: epoch 56 | batch 1240/2304 [I 2022-10-27 08:02:44] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:56 [I 2022-10-27 08:02:44] ...trap/engines/engine.py.229: process: 0.56603 | load: 0.00243 [I 2022-10-27 08:02:44] ...trap/engines/engine.py.230: loss: 0.16632 [I 2022-10-27 08:02:44] ...optimizers/trijoint.py.151: total_norm: 15.219084 [I 2022-10-27 08:02:50] ...trap/engines/engine.py.224: train: epoch 56 | batch 1250/2304 [I 2022-10-27 08:02:50] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:49 [I 2022-10-27 08:02:50] ...trap/engines/engine.py.229: process: 0.56676 | load: 0.00248 [I 2022-10-27 08:02:50] ...trap/engines/engine.py.230: loss: 0.18016 [I 2022-10-27 08:02:50] ...optimizers/trijoint.py.151: total_norm: 25.982748 [I 2022-10-27 08:02:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 1260/2304 [I 2022-10-27 08:02:55] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:46 [I 2022-10-27 08:02:55] ...trap/engines/engine.py.229: process: 0.56476 | load: 0.00292 [I 2022-10-27 08:02:55] ...trap/engines/engine.py.230: loss: 0.25424 [I 2022-10-27 08:02:55] ...optimizers/trijoint.py.151: total_norm: 31.693649 [I 2022-10-27 08:03:01] ...trap/engines/engine.py.224: train: epoch 56 | batch 1270/2304 [I 2022-10-27 08:03:01] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:41 [I 2022-10-27 08:03:01] ...trap/engines/engine.py.229: process: 0.57248 | load: 0.00250 [I 2022-10-27 08:03:01] ...trap/engines/engine.py.230: loss: 0.17755 [I 2022-10-27 08:03:01] ...optimizers/trijoint.py.151: total_norm: 19.525423 [I 2022-10-27 08:03:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 1280/2304 [I 2022-10-27 08:03:06] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:30 [I 2022-10-27 08:03:06] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00254 [I 2022-10-27 08:03:06] ...trap/engines/engine.py.230: loss: 0.14340 [I 2022-10-27 08:03:06] ...optimizers/trijoint.py.151: total_norm: 13.275141 [I 2022-10-27 08:03:12] ...trap/engines/engine.py.224: train: epoch 56 | batch 1290/2304 [I 2022-10-27 08:03:12] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:31 [I 2022-10-27 08:03:12] ...trap/engines/engine.py.229: process: 0.56529 | load: 0.00245 [I 2022-10-27 08:03:12] ...trap/engines/engine.py.230: loss: 0.15513 [I 2022-10-27 08:03:12] ...optimizers/trijoint.py.151: total_norm: 18.963558 [I 2022-10-27 08:03:18] ...trap/engines/engine.py.224: train: epoch 56 | batch 1300/2304 [I 2022-10-27 08:03:18] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:20 [I 2022-10-27 08:03:18] ...trap/engines/engine.py.229: process: 0.55312 | load: 0.00247 [I 2022-10-27 08:03:18] ...trap/engines/engine.py.230: loss: 0.13686 [I 2022-10-27 08:03:18] ...optimizers/trijoint.py.151: total_norm: 14.723141 [I 2022-10-27 08:03:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 1310/2304 [I 2022-10-27 08:03:23] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:12 [I 2022-10-27 08:03:23] ...trap/engines/engine.py.229: process: 0.56730 | load: 0.00264 [I 2022-10-27 08:03:23] ...trap/engines/engine.py.230: loss: 0.12901 [I 2022-10-27 08:03:23] ...optimizers/trijoint.py.151: total_norm: 10.257561 [I 2022-10-27 08:03:29] ...trap/engines/engine.py.224: train: epoch 56 | batch 1320/2304 [I 2022-10-27 08:03:29] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:02 [I 2022-10-27 08:03:29] ...trap/engines/engine.py.229: process: 0.54990 | load: 0.00242 [I 2022-10-27 08:03:29] ...trap/engines/engine.py.230: loss: 0.16776 [I 2022-10-27 08:03:29] ...optimizers/trijoint.py.151: total_norm: 11.405772 [I 2022-10-27 08:03:34] ...trap/engines/engine.py.224: train: epoch 56 | batch 1330/2304 [I 2022-10-27 08:03:34] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:08:59 [I 2022-10-27 08:03:34] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00243 [I 2022-10-27 08:03:34] ...trap/engines/engine.py.230: loss: 0.09987 [I 2022-10-27 08:03:34] ...optimizers/trijoint.py.151: total_norm: 10.774535 [I 2022-10-27 08:03:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 1340/2304 [I 2022-10-27 08:03:40] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:08:55 [I 2022-10-27 08:03:40] ...trap/engines/engine.py.229: process: 0.55417 | load: 0.00244 [I 2022-10-27 08:03:40] ...trap/engines/engine.py.230: loss: 0.14033 [I 2022-10-27 08:03:40] ...optimizers/trijoint.py.151: total_norm: 33.174046 [I 2022-10-27 08:03:45] ...trap/engines/engine.py.224: train: epoch 56 | batch 1350/2304 [I 2022-10-27 08:03:45] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:50 [I 2022-10-27 08:03:45] ...trap/engines/engine.py.229: process: 0.54848 | load: 0.00241 [I 2022-10-27 08:03:45] ...trap/engines/engine.py.230: loss: 0.16068 [I 2022-10-27 08:03:45] ...optimizers/trijoint.py.151: total_norm: 12.465077 [I 2022-10-27 08:03:51] ...trap/engines/engine.py.224: train: epoch 56 | batch 1360/2304 [I 2022-10-27 08:03:51] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:45 [I 2022-10-27 08:03:51] ...trap/engines/engine.py.229: process: 0.55628 | load: 0.00243 [I 2022-10-27 08:03:51] ...trap/engines/engine.py.230: loss: 0.15149 [I 2022-10-27 08:03:51] ...optimizers/trijoint.py.151: total_norm: 21.179089 [I 2022-10-27 08:03:56] ...trap/engines/engine.py.224: train: epoch 56 | batch 1370/2304 [I 2022-10-27 08:03:56] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:38 [I 2022-10-27 08:03:56] ...trap/engines/engine.py.229: process: 0.55458 | load: 0.00243 [I 2022-10-27 08:03:56] ...trap/engines/engine.py.230: loss: 0.24761 [I 2022-10-27 08:03:56] ...optimizers/trijoint.py.151: total_norm: 23.468351 [I 2022-10-27 08:04:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 1380/2304 [I 2022-10-27 08:04:02] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:32 [I 2022-10-27 08:04:02] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00243 [I 2022-10-27 08:04:02] ...trap/engines/engine.py.230: loss: 0.13293 [I 2022-10-27 08:04:02] ...optimizers/trijoint.py.151: total_norm: 16.985985 [I 2022-10-27 08:04:08] ...trap/engines/engine.py.224: train: epoch 56 | batch 1390/2304 [I 2022-10-27 08:04:08] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:24 [I 2022-10-27 08:04:08] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00255 [I 2022-10-27 08:04:08] ...trap/engines/engine.py.230: loss: 0.11713 [I 2022-10-27 08:04:08] ...optimizers/trijoint.py.151: total_norm: 15.996703 [I 2022-10-27 08:04:13] ...trap/engines/engine.py.224: train: epoch 56 | batch 1400/2304 [I 2022-10-27 08:04:13] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:23 [I 2022-10-27 08:04:13] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00253 [I 2022-10-27 08:04:13] ...trap/engines/engine.py.230: loss: 0.12549 [I 2022-10-27 08:04:13] ...optimizers/trijoint.py.151: total_norm: 10.318450 [I 2022-10-27 08:04:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 1410/2304 [I 2022-10-27 08:04:19] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:16 [I 2022-10-27 08:04:19] ...trap/engines/engine.py.229: process: 0.54358 | load: 0.00247 [I 2022-10-27 08:04:19] ...trap/engines/engine.py.230: loss: 0.13993 [I 2022-10-27 08:04:19] ...optimizers/trijoint.py.151: total_norm: 8.757074 [I 2022-10-27 08:04:24] ...trap/engines/engine.py.224: train: epoch 56 | batch 1420/2304 [I 2022-10-27 08:04:24] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:09 [I 2022-10-27 08:04:24] ...trap/engines/engine.py.229: process: 0.55012 | load: 0.00244 [I 2022-10-27 08:04:24] ...trap/engines/engine.py.230: loss: 0.14281 [I 2022-10-27 08:04:24] ...optimizers/trijoint.py.151: total_norm: 21.756439 [I 2022-10-27 08:04:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 1430/2304 [I 2022-10-27 08:04:30] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:00 [I 2022-10-27 08:04:30] ...trap/engines/engine.py.229: process: 0.54654 | load: 0.00246 [I 2022-10-27 08:04:30] ...trap/engines/engine.py.230: loss: 0.18237 [I 2022-10-27 08:04:30] ...optimizers/trijoint.py.151: total_norm: 11.726759 [I 2022-10-27 08:04:35] ...trap/engines/engine.py.224: train: epoch 56 | batch 1440/2304 [I 2022-10-27 08:04:35] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:07:59 [I 2022-10-27 08:04:35] ...trap/engines/engine.py.229: process: 0.55227 | load: 0.00241 [I 2022-10-27 08:04:35] ...trap/engines/engine.py.230: loss: 0.11354 [I 2022-10-27 08:04:35] ...optimizers/trijoint.py.151: total_norm: 12.866979 [I 2022-10-27 08:04:41] ...trap/engines/engine.py.224: train: epoch 56 | batch 1450/2304 [I 2022-10-27 08:04:41] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:07:55 [I 2022-10-27 08:04:41] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00245 [I 2022-10-27 08:04:41] ...trap/engines/engine.py.230: loss: 0.12739 [I 2022-10-27 08:04:41] ...optimizers/trijoint.py.151: total_norm: 9.111571 [I 2022-10-27 08:04:46] ...trap/engines/engine.py.224: train: epoch 56 | batch 1460/2304 [I 2022-10-27 08:04:46] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:49 [I 2022-10-27 08:04:46] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00255 [I 2022-10-27 08:04:46] ...trap/engines/engine.py.230: loss: 0.11797 [I 2022-10-27 08:04:46] ...optimizers/trijoint.py.151: total_norm: 18.819906 [I 2022-10-27 08:04:52] ...trap/engines/engine.py.224: train: epoch 56 | batch 1470/2304 [I 2022-10-27 08:04:52] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:46 [I 2022-10-27 08:04:52] ...trap/engines/engine.py.229: process: 0.55830 | load: 0.00249 [I 2022-10-27 08:04:52] ...trap/engines/engine.py.230: loss: 0.16333 [I 2022-10-27 08:04:52] ...optimizers/trijoint.py.151: total_norm: 17.832735 [I 2022-10-27 08:04:58] ...trap/engines/engine.py.224: train: epoch 56 | batch 1480/2304 [I 2022-10-27 08:04:58] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:39 [I 2022-10-27 08:04:58] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00244 [I 2022-10-27 08:04:58] ...trap/engines/engine.py.230: loss: 0.14455 [I 2022-10-27 08:04:58] ...optimizers/trijoint.py.151: total_norm: 9.097177 [I 2022-10-27 08:05:03] ...trap/engines/engine.py.224: train: epoch 56 | batch 1490/2304 [I 2022-10-27 08:05:03] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:35 [I 2022-10-27 08:05:03] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00249 [I 2022-10-27 08:05:03] ...trap/engines/engine.py.230: loss: 0.21079 [I 2022-10-27 08:05:03] ...optimizers/trijoint.py.151: total_norm: 44.268276 [I 2022-10-27 08:05:09] ...trap/engines/engine.py.224: train: epoch 56 | batch 1500/2304 [I 2022-10-27 08:05:09] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:31 [I 2022-10-27 08:05:09] ...trap/engines/engine.py.229: process: 0.56591 | load: 0.00243 [I 2022-10-27 08:05:09] ...trap/engines/engine.py.230: loss: 0.11924 [I 2022-10-27 08:05:09] ...optimizers/trijoint.py.151: total_norm: 8.583656 [I 2022-10-27 08:05:14] ...trap/engines/engine.py.224: train: epoch 56 | batch 1510/2304 [I 2022-10-27 08:05:14] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:22 [I 2022-10-27 08:05:14] ...trap/engines/engine.py.229: process: 0.56508 | load: 0.00246 [I 2022-10-27 08:05:14] ...trap/engines/engine.py.230: loss: 0.13090 [I 2022-10-27 08:05:14] ...optimizers/trijoint.py.151: total_norm: 8.007838 [I 2022-10-27 08:05:20] ...trap/engines/engine.py.224: train: epoch 56 | batch 1520/2304 [I 2022-10-27 08:05:20] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:16 [I 2022-10-27 08:05:20] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00244 [I 2022-10-27 08:05:20] ...trap/engines/engine.py.230: loss: 0.18840 [I 2022-10-27 08:05:20] ...optimizers/trijoint.py.151: total_norm: 19.148258 [I 2022-10-27 08:05:26] ...trap/engines/engine.py.224: train: epoch 56 | batch 1530/2304 [I 2022-10-27 08:05:26] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:11 [I 2022-10-27 08:05:26] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00245 [I 2022-10-27 08:05:26] ...trap/engines/engine.py.230: loss: 0.12023 [I 2022-10-27 08:05:26] ...optimizers/trijoint.py.151: total_norm: 17.287062 [I 2022-10-27 08:05:31] ...trap/engines/engine.py.224: train: epoch 56 | batch 1540/2304 [I 2022-10-27 08:05:31] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:06 [I 2022-10-27 08:05:31] ...trap/engines/engine.py.229: process: 0.56155 | load: 0.00240 [I 2022-10-27 08:05:31] ...trap/engines/engine.py.230: loss: 0.10110 [I 2022-10-27 08:05:31] ...optimizers/trijoint.py.151: total_norm: 20.125072 [I 2022-10-27 08:05:37] ...trap/engines/engine.py.224: train: epoch 56 | batch 1550/2304 [I 2022-10-27 08:05:37] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:06:59 [I 2022-10-27 08:05:37] ...trap/engines/engine.py.229: process: 0.55128 | load: 0.00241 [I 2022-10-27 08:05:37] ...trap/engines/engine.py.230: loss: 0.15306 [I 2022-10-27 08:05:37] ...optimizers/trijoint.py.151: total_norm: 17.239388 [I 2022-10-27 08:05:42] ...trap/engines/engine.py.224: train: epoch 56 | batch 1560/2304 [I 2022-10-27 08:05:42] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:55 [I 2022-10-27 08:05:42] ...trap/engines/engine.py.229: process: 0.56023 | load: 0.00240 [I 2022-10-27 08:05:42] ...trap/engines/engine.py.230: loss: 0.13217 [I 2022-10-27 08:05:42] ...optimizers/trijoint.py.151: total_norm: 16.707630 [I 2022-10-27 08:05:48] ...trap/engines/engine.py.224: train: epoch 56 | batch 1570/2304 [I 2022-10-27 08:05:48] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:49 [I 2022-10-27 08:05:48] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00243 [I 2022-10-27 08:05:48] ...trap/engines/engine.py.230: loss: 0.19197 [I 2022-10-27 08:05:48] ...optimizers/trijoint.py.151: total_norm: 21.511858 [I 2022-10-27 08:05:54] ...trap/engines/engine.py.224: train: epoch 56 | batch 1580/2304 [I 2022-10-27 08:05:54] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:42 [I 2022-10-27 08:05:54] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00240 [I 2022-10-27 08:05:54] ...trap/engines/engine.py.230: loss: 0.14936 [I 2022-10-27 08:05:54] ...optimizers/trijoint.py.151: total_norm: 12.104177 [I 2022-10-27 08:05:59] ...trap/engines/engine.py.224: train: epoch 56 | batch 1590/2304 [I 2022-10-27 08:05:59] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:42 [I 2022-10-27 08:05:59] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00250 [I 2022-10-27 08:05:59] ...trap/engines/engine.py.230: loss: 0.17251 [I 2022-10-27 08:05:59] ...optimizers/trijoint.py.151: total_norm: 16.611849 [I 2022-10-27 08:06:05] ...trap/engines/engine.py.224: train: epoch 56 | batch 1600/2304 [I 2022-10-27 08:06:05] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:31 [I 2022-10-27 08:06:05] ...trap/engines/engine.py.229: process: 0.56250 | load: 0.00248 [I 2022-10-27 08:06:05] ...trap/engines/engine.py.230: loss: 0.15702 [I 2022-10-27 08:06:05] ...optimizers/trijoint.py.151: total_norm: 13.569097 [I 2022-10-27 08:06:10] ...trap/engines/engine.py.224: train: epoch 56 | batch 1610/2304 [I 2022-10-27 08:06:10] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:28 [I 2022-10-27 08:06:10] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00243 [I 2022-10-27 08:06:10] ...trap/engines/engine.py.230: loss: 0.17530 [I 2022-10-27 08:06:10] ...optimizers/trijoint.py.151: total_norm: 22.221994 [I 2022-10-27 08:06:16] ...trap/engines/engine.py.224: train: epoch 56 | batch 1620/2304 [I 2022-10-27 08:06:16] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:20 [I 2022-10-27 08:06:16] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00245 [I 2022-10-27 08:06:16] ...trap/engines/engine.py.230: loss: 0.20332 [I 2022-10-27 08:06:16] ...optimizers/trijoint.py.151: total_norm: 21.213177 [I 2022-10-27 08:06:22] ...trap/engines/engine.py.224: train: epoch 56 | batch 1630/2304 [I 2022-10-27 08:06:22] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:15 [I 2022-10-27 08:06:22] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00247 [I 2022-10-27 08:06:22] ...trap/engines/engine.py.230: loss: 0.17771 [I 2022-10-27 08:06:22] ...optimizers/trijoint.py.151: total_norm: 23.253780 [I 2022-10-27 08:06:27] ...trap/engines/engine.py.224: train: epoch 56 | batch 1640/2304 [I 2022-10-27 08:06:27] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:09 [I 2022-10-27 08:06:27] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00241 [I 2022-10-27 08:06:27] ...trap/engines/engine.py.230: loss: 0.16404 [I 2022-10-27 08:06:27] ...optimizers/trijoint.py.151: total_norm: 10.502170 [I 2022-10-27 08:06:33] ...trap/engines/engine.py.224: train: epoch 56 | batch 1650/2304 [I 2022-10-27 08:06:33] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:05 [I 2022-10-27 08:06:33] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00243 [I 2022-10-27 08:06:33] ...trap/engines/engine.py.230: loss: 0.17218 [I 2022-10-27 08:06:33] ...optimizers/trijoint.py.151: total_norm: 15.069335 [I 2022-10-27 08:06:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 1660/2304 [I 2022-10-27 08:06:38] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:58 [I 2022-10-27 08:06:38] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00246 [I 2022-10-27 08:06:38] ...trap/engines/engine.py.230: loss: 0.12363 [I 2022-10-27 08:06:38] ...optimizers/trijoint.py.151: total_norm: 10.134077 [I 2022-10-27 08:06:44] ...trap/engines/engine.py.224: train: epoch 56 | batch 1670/2304 [I 2022-10-27 08:06:44] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:06:09 [I 2022-10-27 08:06:44] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00251 [I 2022-10-27 08:06:44] ...trap/engines/engine.py.230: loss: 0.17796 [I 2022-10-27 08:06:44] ...optimizers/trijoint.py.151: total_norm: 17.701401 [I 2022-10-27 08:06:50] ...trap/engines/engine.py.224: train: epoch 56 | batch 1680/2304 [I 2022-10-27 08:06:50] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:51 [I 2022-10-27 08:06:50] ...trap/engines/engine.py.229: process: 0.56578 | load: 0.00250 [I 2022-10-27 08:06:50] ...trap/engines/engine.py.230: loss: 0.18797 [I 2022-10-27 08:06:50] ...optimizers/trijoint.py.151: total_norm: 17.195749 [I 2022-10-27 08:06:55] ...trap/engines/engine.py.224: train: epoch 56 | batch 1690/2304 [I 2022-10-27 08:06:55] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:43 [I 2022-10-27 08:06:55] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00247 [I 2022-10-27 08:06:55] ...trap/engines/engine.py.230: loss: 0.17974 [I 2022-10-27 08:06:55] ...optimizers/trijoint.py.151: total_norm: 22.602295 [I 2022-10-27 08:07:01] ...trap/engines/engine.py.224: train: epoch 56 | batch 1700/2304 [I 2022-10-27 08:07:01] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:34 [I 2022-10-27 08:07:01] ...trap/engines/engine.py.229: process: 0.54237 | load: 0.00246 [I 2022-10-27 08:07:01] ...trap/engines/engine.py.230: loss: 0.14268 [I 2022-10-27 08:07:01] ...optimizers/trijoint.py.151: total_norm: 23.268209 [I 2022-10-27 08:07:06] ...trap/engines/engine.py.224: train: epoch 56 | batch 1710/2304 [I 2022-10-27 08:07:06] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:30 [I 2022-10-27 08:07:06] ...trap/engines/engine.py.229: process: 0.56173 | load: 0.00242 [I 2022-10-27 08:07:06] ...trap/engines/engine.py.230: loss: 0.18446 [I 2022-10-27 08:07:06] ...optimizers/trijoint.py.151: total_norm: 17.444580 [I 2022-10-27 08:07:12] ...trap/engines/engine.py.224: train: epoch 56 | batch 1720/2304 [I 2022-10-27 08:07:12] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:24 [I 2022-10-27 08:07:12] ...trap/engines/engine.py.229: process: 0.54672 | load: 0.00241 [I 2022-10-27 08:07:12] ...trap/engines/engine.py.230: loss: 0.11269 [I 2022-10-27 08:07:12] ...optimizers/trijoint.py.151: total_norm: 12.983285 [I 2022-10-27 08:07:18] ...trap/engines/engine.py.224: train: epoch 56 | batch 1730/2304 [I 2022-10-27 08:07:18] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:19 [I 2022-10-27 08:07:18] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00243 [I 2022-10-27 08:07:18] ...trap/engines/engine.py.230: loss: 0.14749 [I 2022-10-27 08:07:18] ...optimizers/trijoint.py.151: total_norm: 13.160310 [I 2022-10-27 08:07:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 1740/2304 [I 2022-10-27 08:07:23] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:19 [I 2022-10-27 08:07:23] ...trap/engines/engine.py.229: process: 0.54915 | load: 0.00244 [I 2022-10-27 08:07:23] ...trap/engines/engine.py.230: loss: 0.18223 [I 2022-10-27 08:07:23] ...optimizers/trijoint.py.151: total_norm: 17.054083 [I 2022-10-27 08:07:29] ...trap/engines/engine.py.224: train: epoch 56 | batch 1750/2304 [I 2022-10-27 08:07:29] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:09 [I 2022-10-27 08:07:29] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00240 [I 2022-10-27 08:07:29] ...trap/engines/engine.py.230: loss: 0.17478 [I 2022-10-27 08:07:29] ...optimizers/trijoint.py.151: total_norm: 24.635441 [I 2022-10-27 08:07:34] ...trap/engines/engine.py.224: train: epoch 56 | batch 1760/2304 [I 2022-10-27 08:07:34] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:02 [I 2022-10-27 08:07:34] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00245 [I 2022-10-27 08:07:34] ...trap/engines/engine.py.230: loss: 0.21466 [I 2022-10-27 08:07:34] ...optimizers/trijoint.py.151: total_norm: 21.923008 [I 2022-10-27 08:07:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 1770/2304 [I 2022-10-27 08:07:40] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:04:58 [I 2022-10-27 08:07:40] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00241 [I 2022-10-27 08:07:40] ...trap/engines/engine.py.230: loss: 0.13643 [I 2022-10-27 08:07:40] ...optimizers/trijoint.py.151: total_norm: 18.053312 [I 2022-10-27 08:07:46] ...trap/engines/engine.py.224: train: epoch 56 | batch 1780/2304 [I 2022-10-27 08:07:46] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:51 [I 2022-10-27 08:07:46] ...trap/engines/engine.py.229: process: 0.55417 | load: 0.00238 [I 2022-10-27 08:07:46] ...trap/engines/engine.py.230: loss: 0.11352 [I 2022-10-27 08:07:46] ...optimizers/trijoint.py.151: total_norm: 7.468153 [I 2022-10-27 08:07:51] ...trap/engines/engine.py.224: train: epoch 56 | batch 1790/2304 [I 2022-10-27 08:07:51] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:45 [I 2022-10-27 08:07:51] ...trap/engines/engine.py.229: process: 0.55800 | load: 0.00236 [I 2022-10-27 08:07:51] ...trap/engines/engine.py.230: loss: 0.15773 [I 2022-10-27 08:07:51] ...optimizers/trijoint.py.151: total_norm: 22.205938 [I 2022-10-27 08:07:57] ...trap/engines/engine.py.224: train: epoch 56 | batch 1800/2304 [I 2022-10-27 08:07:57] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:40 [I 2022-10-27 08:07:57] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00240 [I 2022-10-27 08:07:57] ...trap/engines/engine.py.230: loss: 0.19130 [I 2022-10-27 08:07:57] ...optimizers/trijoint.py.151: total_norm: 13.023998 [I 2022-10-27 08:08:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 1810/2304 [I 2022-10-27 08:08:02] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:35 [I 2022-10-27 08:08:02] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00243 [I 2022-10-27 08:08:02] ...trap/engines/engine.py.230: loss: 0.20090 [I 2022-10-27 08:08:02] ...optimizers/trijoint.py.151: total_norm: 11.513431 [I 2022-10-27 08:08:08] ...trap/engines/engine.py.224: train: epoch 56 | batch 1820/2304 [I 2022-10-27 08:08:08] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:30 [I 2022-10-27 08:08:08] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00250 [I 2022-10-27 08:08:08] ...trap/engines/engine.py.230: loss: 0.14274 [I 2022-10-27 08:08:08] ...optimizers/trijoint.py.151: total_norm: 13.184980 [I 2022-10-27 08:08:14] ...trap/engines/engine.py.224: train: epoch 56 | batch 1830/2304 [I 2022-10-27 08:08:14] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:24 [I 2022-10-27 08:08:14] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00248 [I 2022-10-27 08:08:14] ...trap/engines/engine.py.230: loss: 0.14596 [I 2022-10-27 08:08:14] ...optimizers/trijoint.py.151: total_norm: 42.313618 [I 2022-10-27 08:08:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 1840/2304 [I 2022-10-27 08:08:19] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:20 [I 2022-10-27 08:08:19] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00243 [I 2022-10-27 08:08:19] ...trap/engines/engine.py.230: loss: 0.13864 [I 2022-10-27 08:08:19] ...optimizers/trijoint.py.151: total_norm: 8.063869 [I 2022-10-27 08:08:25] ...trap/engines/engine.py.224: train: epoch 56 | batch 1850/2304 [I 2022-10-27 08:08:25] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:16 [I 2022-10-27 08:08:25] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00242 [I 2022-10-27 08:08:25] ...trap/engines/engine.py.230: loss: 0.20728 [I 2022-10-27 08:08:25] ...optimizers/trijoint.py.151: total_norm: 16.428057 [I 2022-10-27 08:08:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 1860/2304 [I 2022-10-27 08:08:30] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:07 [I 2022-10-27 08:08:30] ...trap/engines/engine.py.229: process: 0.54840 | load: 0.00245 [I 2022-10-27 08:08:30] ...trap/engines/engine.py.230: loss: 0.13160 [I 2022-10-27 08:08:30] ...optimizers/trijoint.py.151: total_norm: 19.261156 [I 2022-10-27 08:08:36] ...trap/engines/engine.py.224: train: epoch 56 | batch 1870/2304 [I 2022-10-27 08:08:36] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:03 [I 2022-10-27 08:08:36] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00241 [I 2022-10-27 08:08:36] ...trap/engines/engine.py.230: loss: 0.17975 [I 2022-10-27 08:08:36] ...optimizers/trijoint.py.151: total_norm: 34.856403 [I 2022-10-27 08:08:42] ...trap/engines/engine.py.224: train: epoch 56 | batch 1880/2304 [I 2022-10-27 08:08:42] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:03:56 [I 2022-10-27 08:08:42] ...trap/engines/engine.py.229: process: 0.57103 | load: 0.00262 [I 2022-10-27 08:08:42] ...trap/engines/engine.py.230: loss: 0.16989 [I 2022-10-27 08:08:42] ...optimizers/trijoint.py.151: total_norm: 28.170107 [I 2022-10-27 08:08:47] ...trap/engines/engine.py.224: train: epoch 56 | batch 1890/2304 [I 2022-10-27 08:08:47] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:03:50 [I 2022-10-27 08:08:47] ...trap/engines/engine.py.229: process: 0.55512 | load: 0.00245 [I 2022-10-27 08:08:47] ...trap/engines/engine.py.230: loss: 0.13820 [I 2022-10-27 08:08:47] ...optimizers/trijoint.py.151: total_norm: 11.225066 [I 2022-10-27 08:08:53] ...trap/engines/engine.py.224: train: epoch 56 | batch 1900/2304 [I 2022-10-27 08:08:53] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:46 [I 2022-10-27 08:08:53] ...trap/engines/engine.py.229: process: 0.56378 | load: 0.00248 [I 2022-10-27 08:08:53] ...trap/engines/engine.py.230: loss: 0.14747 [I 2022-10-27 08:08:53] ...optimizers/trijoint.py.151: total_norm: 18.845976 [I 2022-10-27 08:08:58] ...trap/engines/engine.py.224: train: epoch 56 | batch 1910/2304 [I 2022-10-27 08:08:58] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:39 [I 2022-10-27 08:08:58] ...trap/engines/engine.py.229: process: 0.56597 | load: 0.00247 [I 2022-10-27 08:08:58] ...trap/engines/engine.py.230: loss: 0.19335 [I 2022-10-27 08:08:58] ...optimizers/trijoint.py.151: total_norm: 44.731747 [I 2022-10-27 08:09:04] ...trap/engines/engine.py.224: train: epoch 56 | batch 1920/2304 [I 2022-10-27 08:09:04] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:34 [I 2022-10-27 08:09:04] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00245 [I 2022-10-27 08:09:04] ...trap/engines/engine.py.230: loss: 0.19050 [I 2022-10-27 08:09:04] ...optimizers/trijoint.py.151: total_norm: 9.561020 [I 2022-10-27 08:09:10] ...trap/engines/engine.py.224: train: epoch 56 | batch 1930/2304 [I 2022-10-27 08:09:10] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:27 [I 2022-10-27 08:09:10] ...trap/engines/engine.py.229: process: 0.54379 | load: 0.00242 [I 2022-10-27 08:09:10] ...trap/engines/engine.py.230: loss: 0.17017 [I 2022-10-27 08:09:10] ...optimizers/trijoint.py.151: total_norm: 36.432411 [I 2022-10-27 08:09:15] ...trap/engines/engine.py.224: train: epoch 56 | batch 1940/2304 [I 2022-10-27 08:09:15] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:22 [I 2022-10-27 08:09:15] ...trap/engines/engine.py.229: process: 0.56073 | load: 0.00246 [I 2022-10-27 08:09:15] ...trap/engines/engine.py.230: loss: 0.14512 [I 2022-10-27 08:09:15] ...optimizers/trijoint.py.151: total_norm: 12.736382 [I 2022-10-27 08:09:21] ...trap/engines/engine.py.224: train: epoch 56 | batch 1950/2304 [I 2022-10-27 08:09:21] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:17 [I 2022-10-27 08:09:21] ...trap/engines/engine.py.229: process: 0.56258 | load: 0.00258 [I 2022-10-27 08:09:21] ...trap/engines/engine.py.230: loss: 0.11553 [I 2022-10-27 08:09:21] ...optimizers/trijoint.py.151: total_norm: 8.909358 [I 2022-10-27 08:09:26] ...trap/engines/engine.py.224: train: epoch 56 | batch 1960/2304 [I 2022-10-27 08:09:26] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:12 [I 2022-10-27 08:09:26] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00259 [I 2022-10-27 08:09:26] ...trap/engines/engine.py.230: loss: 0.13065 [I 2022-10-27 08:09:26] ...optimizers/trijoint.py.151: total_norm: 15.417222 [I 2022-10-27 08:09:32] ...trap/engines/engine.py.224: train: epoch 56 | batch 1970/2304 [I 2022-10-27 08:09:32] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:05 [I 2022-10-27 08:09:32] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00240 [I 2022-10-27 08:09:32] ...trap/engines/engine.py.230: loss: 0.28373 [I 2022-10-27 08:09:32] ...optimizers/trijoint.py.151: total_norm: 16.607738 [I 2022-10-27 08:09:37] ...trap/engines/engine.py.224: train: epoch 56 | batch 1980/2304 [I 2022-10-27 08:09:37] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:00 [I 2022-10-27 08:09:37] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00243 [I 2022-10-27 08:09:37] ...trap/engines/engine.py.230: loss: 0.15017 [I 2022-10-27 08:09:37] ...optimizers/trijoint.py.151: total_norm: 9.903346 [I 2022-10-27 08:09:43] ...trap/engines/engine.py.224: train: epoch 56 | batch 1990/2304 [I 2022-10-27 08:09:43] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:02:53 [I 2022-10-27 08:09:43] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00248 [I 2022-10-27 08:09:43] ...trap/engines/engine.py.230: loss: 0.13855 [I 2022-10-27 08:09:43] ...optimizers/trijoint.py.151: total_norm: 15.929436 [I 2022-10-27 08:09:49] ...trap/engines/engine.py.224: train: epoch 56 | batch 2000/2304 [I 2022-10-27 08:09:49] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:48 [I 2022-10-27 08:09:49] ...trap/engines/engine.py.229: process: 0.56037 | load: 0.00247 [I 2022-10-27 08:09:49] ...trap/engines/engine.py.230: loss: 0.14523 [I 2022-10-27 08:09:49] ...optimizers/trijoint.py.151: total_norm: 31.018127 [I 2022-10-27 08:09:54] ...trap/engines/engine.py.224: train: epoch 56 | batch 2010/2304 [I 2022-10-27 08:09:54] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:02:44 [I 2022-10-27 08:09:54] ...trap/engines/engine.py.229: process: 0.57373 | load: 0.00246 [I 2022-10-27 08:09:54] ...trap/engines/engine.py.230: loss: 0.10630 [I 2022-10-27 08:09:54] ...optimizers/trijoint.py.151: total_norm: 15.331011 [I 2022-10-27 08:10:00] ...trap/engines/engine.py.224: train: epoch 56 | batch 2020/2304 [I 2022-10-27 08:10:00] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:37 [I 2022-10-27 08:10:00] ...trap/engines/engine.py.229: process: 0.54949 | load: 0.00241 [I 2022-10-27 08:10:00] ...trap/engines/engine.py.230: loss: 0.17267 [I 2022-10-27 08:10:00] ...optimizers/trijoint.py.151: total_norm: 12.295405 [I 2022-10-27 08:10:05] ...trap/engines/engine.py.224: train: epoch 56 | batch 2030/2304 [I 2022-10-27 08:10:05] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:32 [I 2022-10-27 08:10:05] ...trap/engines/engine.py.229: process: 0.56801 | load: 0.00248 [I 2022-10-27 08:10:05] ...trap/engines/engine.py.230: loss: 0.14675 [I 2022-10-27 08:10:05] ...optimizers/trijoint.py.151: total_norm: 21.281139 [I 2022-10-27 08:10:11] ...trap/engines/engine.py.224: train: epoch 56 | batch 2040/2304 [I 2022-10-27 08:10:11] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:26 [I 2022-10-27 08:10:11] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00245 [I 2022-10-27 08:10:11] ...trap/engines/engine.py.230: loss: 0.10769 [I 2022-10-27 08:10:11] ...optimizers/trijoint.py.151: total_norm: 9.440623 [I 2022-10-27 08:10:16] ...trap/engines/engine.py.224: train: epoch 56 | batch 2050/2304 [I 2022-10-27 08:10:16] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:21 [I 2022-10-27 08:10:16] ...trap/engines/engine.py.229: process: 0.55361 | load: 0.00245 [I 2022-10-27 08:10:16] ...trap/engines/engine.py.230: loss: 0.14472 [I 2022-10-27 08:10:16] ...optimizers/trijoint.py.151: total_norm: 11.243083 [I 2022-10-27 08:10:22] ...trap/engines/engine.py.224: train: epoch 56 | batch 2060/2304 [I 2022-10-27 08:10:22] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:16 [I 2022-10-27 08:10:22] ...trap/engines/engine.py.229: process: 0.55895 | load: 0.00244 [I 2022-10-27 08:10:22] ...trap/engines/engine.py.230: loss: 0.13986 [I 2022-10-27 08:10:22] ...optimizers/trijoint.py.151: total_norm: 20.369854 [I 2022-10-27 08:10:28] ...trap/engines/engine.py.224: train: epoch 56 | batch 2070/2304 [I 2022-10-27 08:10:28] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:15 [I 2022-10-27 08:10:28] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00247 [I 2022-10-27 08:10:28] ...trap/engines/engine.py.230: loss: 0.18599 [I 2022-10-27 08:10:28] ...optimizers/trijoint.py.151: total_norm: 11.348441 [I 2022-10-27 08:10:33] ...trap/engines/engine.py.224: train: epoch 56 | batch 2080/2304 [I 2022-10-27 08:10:33] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:04 [I 2022-10-27 08:10:33] ...trap/engines/engine.py.229: process: 0.55725 | load: 0.00248 [I 2022-10-27 08:10:33] ...trap/engines/engine.py.230: loss: 0.15706 [I 2022-10-27 08:10:33] ...optimizers/trijoint.py.151: total_norm: 19.053593 [I 2022-10-27 08:10:39] ...trap/engines/engine.py.224: train: epoch 56 | batch 2090/2304 [I 2022-10-27 08:10:39] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:01:59 [I 2022-10-27 08:10:39] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00243 [I 2022-10-27 08:10:39] ...trap/engines/engine.py.230: loss: 0.18244 [I 2022-10-27 08:10:39] ...optimizers/trijoint.py.151: total_norm: 25.305387 [I 2022-10-27 08:10:44] ...trap/engines/engine.py.224: train: epoch 56 | batch 2100/2304 [I 2022-10-27 08:10:45] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:53 [I 2022-10-27 08:10:45] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00245 [I 2022-10-27 08:10:45] ...trap/engines/engine.py.230: loss: 0.16365 [I 2022-10-27 08:10:45] ...optimizers/trijoint.py.151: total_norm: 13.262597 [I 2022-10-27 08:10:50] ...trap/engines/engine.py.224: train: epoch 56 | batch 2110/2304 [I 2022-10-27 08:10:50] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:01:48 [I 2022-10-27 08:10:50] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00245 [I 2022-10-27 08:10:50] ...trap/engines/engine.py.230: loss: 0.11036 [I 2022-10-27 08:10:50] ...optimizers/trijoint.py.151: total_norm: 8.873450 [I 2022-10-27 08:10:56] ...trap/engines/engine.py.224: train: epoch 56 | batch 2120/2304 [I 2022-10-27 08:10:56] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:41 [I 2022-10-27 08:10:56] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00244 [I 2022-10-27 08:10:56] ...trap/engines/engine.py.230: loss: 0.20736 [I 2022-10-27 08:10:56] ...optimizers/trijoint.py.151: total_norm: 24.972694 [I 2022-10-27 08:11:01] ...trap/engines/engine.py.224: train: epoch 56 | batch 2130/2304 [I 2022-10-27 08:11:01] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:36 [I 2022-10-27 08:11:01] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00242 [I 2022-10-27 08:11:01] ...trap/engines/engine.py.230: loss: 0.11713 [I 2022-10-27 08:11:01] ...optimizers/trijoint.py.151: total_norm: 11.781945 [I 2022-10-27 08:11:07] ...trap/engines/engine.py.224: train: epoch 56 | batch 2140/2304 [I 2022-10-27 08:11:07] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:31 [I 2022-10-27 08:11:07] ...trap/engines/engine.py.229: process: 0.58431 | load: 0.00239 [I 2022-10-27 08:11:07] ...trap/engines/engine.py.230: loss: 0.13545 [I 2022-10-27 08:11:07] ...optimizers/trijoint.py.151: total_norm: 14.662943 [I 2022-10-27 08:11:12] ...trap/engines/engine.py.224: train: epoch 56 | batch 2150/2304 [I 2022-10-27 08:11:12] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:25 [I 2022-10-27 08:11:12] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00240 [I 2022-10-27 08:11:12] ...trap/engines/engine.py.230: loss: 0.15624 [I 2022-10-27 08:11:12] ...optimizers/trijoint.py.151: total_norm: 21.434586 [I 2022-10-27 08:11:18] ...trap/engines/engine.py.224: train: epoch 56 | batch 2160/2304 [I 2022-10-27 08:11:18] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:20 [I 2022-10-27 08:11:18] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00239 [I 2022-10-27 08:11:18] ...trap/engines/engine.py.230: loss: 0.13912 [I 2022-10-27 08:11:18] ...optimizers/trijoint.py.151: total_norm: 14.344530 [I 2022-10-27 08:11:23] ...trap/engines/engine.py.224: train: epoch 56 | batch 2170/2304 [I 2022-10-27 08:11:23] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:14 [I 2022-10-27 08:11:23] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00243 [I 2022-10-27 08:11:23] ...trap/engines/engine.py.230: loss: 0.18305 [I 2022-10-27 08:11:23] ...optimizers/trijoint.py.151: total_norm: 12.971897 [I 2022-10-27 08:11:29] ...trap/engines/engine.py.224: train: epoch 56 | batch 2180/2304 [I 2022-10-27 08:11:29] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:09 [I 2022-10-27 08:11:29] ...trap/engines/engine.py.229: process: 0.56475 | load: 0.00252 [I 2022-10-27 08:11:29] ...trap/engines/engine.py.230: loss: 0.12248 [I 2022-10-27 08:11:29] ...optimizers/trijoint.py.151: total_norm: 11.469173 [I 2022-10-27 08:11:35] ...trap/engines/engine.py.224: train: epoch 56 | batch 2190/2304 [I 2022-10-27 08:11:35] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:03 [I 2022-10-27 08:11:35] ...trap/engines/engine.py.229: process: 0.55684 | load: 0.00251 [I 2022-10-27 08:11:35] ...trap/engines/engine.py.230: loss: 0.16027 [I 2022-10-27 08:11:35] ...optimizers/trijoint.py.151: total_norm: 10.320047 [I 2022-10-27 08:11:40] ...trap/engines/engine.py.224: train: epoch 56 | batch 2200/2304 [I 2022-10-27 08:11:40] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:57 [I 2022-10-27 08:11:40] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00246 [I 2022-10-27 08:11:40] ...trap/engines/engine.py.230: loss: 0.16957 [I 2022-10-27 08:11:40] ...optimizers/trijoint.py.151: total_norm: 20.047806 [I 2022-10-27 08:11:46] ...trap/engines/engine.py.224: train: epoch 56 | batch 2210/2304 [I 2022-10-27 08:11:46] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:00:52 [I 2022-10-27 08:11:46] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00242 [I 2022-10-27 08:11:46] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-27 08:11:46] ...optimizers/trijoint.py.151: total_norm: 22.552216 [I 2022-10-27 08:11:51] ...trap/engines/engine.py.224: train: epoch 56 | batch 2220/2304 [I 2022-10-27 08:11:51] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:00:46 [I 2022-10-27 08:11:51] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00245 [I 2022-10-27 08:11:51] ...trap/engines/engine.py.230: loss: 0.21172 [I 2022-10-27 08:11:51] ...optimizers/trijoint.py.151: total_norm: 10.938113 [I 2022-10-27 08:11:57] ...trap/engines/engine.py.224: train: epoch 56 | batch 2230/2304 [I 2022-10-27 08:11:57] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:41 [I 2022-10-27 08:11:57] ...trap/engines/engine.py.229: process: 0.56420 | load: 0.00246 [I 2022-10-27 08:11:57] ...trap/engines/engine.py.230: loss: 0.12906 [I 2022-10-27 08:11:57] ...optimizers/trijoint.py.151: total_norm: 19.785116 [I 2022-10-27 08:12:02] ...trap/engines/engine.py.224: train: epoch 56 | batch 2240/2304 [I 2022-10-27 08:12:02] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:35 [I 2022-10-27 08:12:02] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00246 [I 2022-10-27 08:12:02] ...trap/engines/engine.py.230: loss: 0.15103 [I 2022-10-27 08:12:02] ...optimizers/trijoint.py.151: total_norm: 18.369043 [I 2022-10-27 08:12:08] ...trap/engines/engine.py.224: train: epoch 56 | batch 2250/2304 [I 2022-10-27 08:12:08] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:29 [I 2022-10-27 08:12:08] ...trap/engines/engine.py.229: process: 0.53743 | load: 0.00249 [I 2022-10-27 08:12:08] ...trap/engines/engine.py.230: loss: 0.16325 [I 2022-10-27 08:12:08] ...optimizers/trijoint.py.151: total_norm: 13.035399 [I 2022-10-27 08:12:14] ...trap/engines/engine.py.224: train: epoch 56 | batch 2260/2304 [I 2022-10-27 08:12:14] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:24 [I 2022-10-27 08:12:14] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00247 [I 2022-10-27 08:12:14] ...trap/engines/engine.py.230: loss: 0.14998 [I 2022-10-27 08:12:14] ...optimizers/trijoint.py.151: total_norm: 14.977621 [I 2022-10-27 08:12:19] ...trap/engines/engine.py.224: train: epoch 56 | batch 2270/2304 [I 2022-10-27 08:12:19] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:18 [I 2022-10-27 08:12:19] ...trap/engines/engine.py.229: process: 0.54350 | load: 0.00245 [I 2022-10-27 08:12:19] ...trap/engines/engine.py.230: loss: 0.13298 [I 2022-10-27 08:12:19] ...optimizers/trijoint.py.151: total_norm: 12.724559 [I 2022-10-27 08:12:25] ...trap/engines/engine.py.224: train: epoch 56 | batch 2280/2304 [I 2022-10-27 08:12:25] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:13 [I 2022-10-27 08:12:25] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00242 [I 2022-10-27 08:12:25] ...trap/engines/engine.py.230: loss: 0.13157 [I 2022-10-27 08:12:25] ...optimizers/trijoint.py.151: total_norm: 37.089176 [I 2022-10-27 08:12:30] ...trap/engines/engine.py.224: train: epoch 56 | batch 2290/2304 [I 2022-10-27 08:12:30] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:07 [I 2022-10-27 08:12:30] ...trap/engines/engine.py.229: process: 0.55103 | load: 0.00240 [I 2022-10-27 08:12:30] ...trap/engines/engine.py.230: loss: 0.11271 [I 2022-10-27 08:12:30] ...optimizers/trijoint.py.151: total_norm: 8.656657 [I 2022-10-27 08:12:36] ...trap/engines/engine.py.224: train: epoch 56 | batch 2300/2304 [I 2022-10-27 08:12:36] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:02 [I 2022-10-27 08:12:36] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00014 [I 2022-10-27 08:12:36] ...trap/engines/engine.py.230: loss: 0.13574 [I 2022-10-27 08:12:36] ...optimizers/trijoint.py.151: total_norm: 15.865309 [I 2022-10-27 08:12:38] ...trap/engines/engine.py.224: train: epoch 56 | batch 2304/2304 [I 2022-10-27 08:12:38] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:00 [I 2022-10-27 08:12:38] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00015 [I 2022-10-27 08:12:38] ...trap/engines/engine.py.230: loss: 0.13257 [I 2022-10-27 08:12:38] ...optimizers/trijoint.py.151: total_norm: 15.876279 [S 2022-10-27 08:12:39] ...trap/engines/engine.py.129: train_epoch.epoch: 56 [S 2022-10-27 08:12:39] ...trap/engines/engine.py.129: train_epoch.loss: 0.1590403896318082 [I 2022-10-27 08:12:39] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 08:12:40] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 56 [I 2022-10-27 08:12:40] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 08:12:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 08:12:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 08:12:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 08:12:44] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 08:12:44] ...trap/engines/engine.py.321: eval: epoch 56 | batch 0/510 [I 2022-10-27 08:12:44] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:35:11 [I 2022-10-27 08:12:44] ...trap/engines/engine.py.326: process: 4.14058 | load: 3.73470 [I 2022-10-27 08:12:48] ...trap/engines/engine.py.321: eval: epoch 56 | batch 10/510 [I 2022-10-27 08:12:48] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:09 [I 2022-10-27 08:12:48] ...trap/engines/engine.py.326: process: 0.32969 | load: 0.00017 [I 2022-10-27 08:12:51] ...trap/engines/engine.py.321: eval: epoch 56 | batch 20/510 [I 2022-10-27 08:12:51] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:04 [I 2022-10-27 08:12:51] ...trap/engines/engine.py.326: process: 0.33460 | load: 0.00016 [I 2022-10-27 08:12:54] ...trap/engines/engine.py.321: eval: epoch 56 | batch 30/510 [I 2022-10-27 08:12:54] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:39 [I 2022-10-27 08:12:54] ...trap/engines/engine.py.326: process: 0.31944 | load: 0.00016 [I 2022-10-27 08:12:58] ...trap/engines/engine.py.321: eval: epoch 56 | batch 40/510 [I 2022-10-27 08:12:58] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:34 [I 2022-10-27 08:12:58] ...trap/engines/engine.py.326: process: 0.32925 | load: 0.00017 [I 2022-10-27 08:13:01] ...trap/engines/engine.py.321: eval: epoch 56 | batch 50/510 [I 2022-10-27 08:13:01] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:32 [I 2022-10-27 08:13:01] ...trap/engines/engine.py.326: process: 0.32951 | load: 0.00020 [I 2022-10-27 08:13:04] ...trap/engines/engine.py.321: eval: epoch 56 | batch 60/510 [I 2022-10-27 08:13:04] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:33 [I 2022-10-27 08:13:04] ...trap/engines/engine.py.326: process: 0.33079 | load: 0.00025 [I 2022-10-27 08:13:08] ...trap/engines/engine.py.321: eval: epoch 56 | batch 70/510 [I 2022-10-27 08:13:08] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:27 [I 2022-10-27 08:13:08] ...trap/engines/engine.py.326: process: 0.31982 | load: 0.00019 [I 2022-10-27 08:13:11] ...trap/engines/engine.py.321: eval: epoch 56 | batch 80/510 [I 2022-10-27 08:13:11] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:26 [I 2022-10-27 08:13:11] ...trap/engines/engine.py.326: process: 0.32709 | load: 0.00017 [I 2022-10-27 08:13:14] ...trap/engines/engine.py.321: eval: epoch 56 | batch 90/510 [I 2022-10-27 08:13:14] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:22 [I 2022-10-27 08:13:14] ...trap/engines/engine.py.326: process: 0.32647 | load: 0.00019 [I 2022-10-27 08:13:18] ...trap/engines/engine.py.321: eval: epoch 56 | batch 100/510 [I 2022-10-27 08:13:18] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:13 [I 2022-10-27 08:13:18] ...trap/engines/engine.py.326: process: 0.28688 | load: 0.00017 [I 2022-10-27 08:13:21] ...trap/engines/engine.py.321: eval: epoch 56 | batch 110/510 [I 2022-10-27 08:13:21] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:56 [I 2022-10-27 08:13:21] ...trap/engines/engine.py.326: process: 0.28099 | load: 0.00018 [I 2022-10-27 08:13:23] ...trap/engines/engine.py.321: eval: epoch 56 | batch 120/510 [I 2022-10-27 08:13:23] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:52 [I 2022-10-27 08:13:23] ...trap/engines/engine.py.326: process: 0.28684 | load: 0.00014 [I 2022-10-27 08:13:26] ...trap/engines/engine.py.321: eval: epoch 56 | batch 130/510 [I 2022-10-27 08:13:26] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:49 [I 2022-10-27 08:13:26] ...trap/engines/engine.py.326: process: 0.29161 | load: 0.00016 [I 2022-10-27 08:13:29] ...trap/engines/engine.py.321: eval: epoch 56 | batch 140/510 [I 2022-10-27 08:13:29] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 08:13:29] ...trap/engines/engine.py.326: process: 0.29047 | load: 0.00019 [I 2022-10-27 08:13:32] ...trap/engines/engine.py.321: eval: epoch 56 | batch 150/510 [I 2022-10-27 08:13:32] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:45 [I 2022-10-27 08:13:32] ...trap/engines/engine.py.326: process: 0.29454 | load: 0.00022 [I 2022-10-27 08:13:35] ...trap/engines/engine.py.321: eval: epoch 56 | batch 160/510 [I 2022-10-27 08:13:35] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:43 [I 2022-10-27 08:13:35] ...trap/engines/engine.py.326: process: 0.30561 | load: 0.00014 [I 2022-10-27 08:13:38] ...trap/engines/engine.py.321: eval: epoch 56 | batch 170/510 [I 2022-10-27 08:13:38] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-27 08:13:38] ...trap/engines/engine.py.326: process: 0.28226 | load: 0.00014 [I 2022-10-27 08:13:41] ...trap/engines/engine.py.321: eval: epoch 56 | batch 180/510 [I 2022-10-27 08:13:41] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 08:13:41] ...trap/engines/engine.py.326: process: 0.28234 | load: 0.00017 [I 2022-10-27 08:13:44] ...trap/engines/engine.py.321: eval: epoch 56 | batch 190/510 [I 2022-10-27 08:13:44] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 08:13:44] ...trap/engines/engine.py.326: process: 0.28281 | load: 0.00022 [I 2022-10-27 08:13:47] ...trap/engines/engine.py.321: eval: epoch 56 | batch 200/510 [I 2022-10-27 08:13:47] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 08:13:47] ...trap/engines/engine.py.326: process: 0.28747 | load: 0.00016 [I 2022-10-27 08:13:50] ...trap/engines/engine.py.321: eval: epoch 56 | batch 210/510 [I 2022-10-27 08:13:50] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 08:13:50] ...trap/engines/engine.py.326: process: 0.29059 | load: 0.00016 [I 2022-10-27 08:13:53] ...trap/engines/engine.py.321: eval: epoch 56 | batch 220/510 [I 2022-10-27 08:13:53] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:24 [I 2022-10-27 08:13:53] ...trap/engines/engine.py.326: process: 0.29929 | load: 0.00016 [I 2022-10-27 08:13:55] ...trap/engines/engine.py.321: eval: epoch 56 | batch 230/510 [I 2022-10-27 08:13:55] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:21 [I 2022-10-27 08:13:55] ...trap/engines/engine.py.326: process: 0.28975 | load: 0.00025 [I 2022-10-27 08:13:58] ...trap/engines/engine.py.321: eval: epoch 56 | batch 240/510 [I 2022-10-27 08:13:58] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 08:13:58] ...trap/engines/engine.py.326: process: 0.28685 | load: 0.00033 [I 2022-10-27 08:14:01] ...trap/engines/engine.py.321: eval: epoch 56 | batch 250/510 [I 2022-10-27 08:14:01] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 08:14:01] ...trap/engines/engine.py.326: process: 0.29300 | load: 0.00019 [I 2022-10-27 08:14:04] ...trap/engines/engine.py.321: eval: epoch 56 | batch 260/510 [I 2022-10-27 08:14:04] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-27 08:14:04] ...trap/engines/engine.py.326: process: 0.29479 | load: 0.00025 [I 2022-10-27 08:14:07] ...trap/engines/engine.py.321: eval: epoch 56 | batch 270/510 [I 2022-10-27 08:14:07] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:10 [I 2022-10-27 08:14:07] ...trap/engines/engine.py.326: process: 0.28836 | load: 0.00024 [I 2022-10-27 08:14:10] ...trap/engines/engine.py.321: eval: epoch 56 | batch 280/510 [I 2022-10-27 08:14:10] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:07 [I 2022-10-27 08:14:10] ...trap/engines/engine.py.326: process: 0.28758 | load: 0.00017 [I 2022-10-27 08:14:13] ...trap/engines/engine.py.321: eval: epoch 56 | batch 290/510 [I 2022-10-27 08:14:13] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:04 [I 2022-10-27 08:14:13] ...trap/engines/engine.py.326: process: 0.28591 | load: 0.00024 [I 2022-10-27 08:14:16] ...trap/engines/engine.py.321: eval: epoch 56 | batch 300/510 [I 2022-10-27 08:14:16] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:01 [I 2022-10-27 08:14:16] ...trap/engines/engine.py.326: process: 0.29382 | load: 0.00016 [I 2022-10-27 08:14:19] ...trap/engines/engine.py.321: eval: epoch 56 | batch 310/510 [I 2022-10-27 08:14:19] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 08:14:19] ...trap/engines/engine.py.326: process: 0.28411 | load: 0.00017 [I 2022-10-27 08:14:22] ...trap/engines/engine.py.321: eval: epoch 56 | batch 320/510 [I 2022-10-27 08:14:22] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 08:14:22] ...trap/engines/engine.py.326: process: 0.28896 | load: 0.00016 [I 2022-10-27 08:14:24] ...trap/engines/engine.py.321: eval: epoch 56 | batch 330/510 [I 2022-10-27 08:14:24] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 08:14:24] ...trap/engines/engine.py.326: process: 0.28716 | load: 0.00016 [I 2022-10-27 08:14:27] ...trap/engines/engine.py.321: eval: epoch 56 | batch 340/510 [I 2022-10-27 08:14:27] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 08:14:27] ...trap/engines/engine.py.326: process: 0.28953 | load: 0.00017 [I 2022-10-27 08:14:30] ...trap/engines/engine.py.321: eval: epoch 56 | batch 350/510 [I 2022-10-27 08:14:30] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 08:14:30] ...trap/engines/engine.py.326: process: 0.29077 | load: 0.00017 [I 2022-10-27 08:14:33] ...trap/engines/engine.py.321: eval: epoch 56 | batch 360/510 [I 2022-10-27 08:14:33] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 08:14:33] ...trap/engines/engine.py.326: process: 0.28524 | load: 0.00015 [I 2022-10-27 08:14:36] ...trap/engines/engine.py.321: eval: epoch 56 | batch 370/510 [I 2022-10-27 08:14:36] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 08:14:36] ...trap/engines/engine.py.326: process: 0.28645 | load: 0.00015 [I 2022-10-27 08:14:39] ...trap/engines/engine.py.321: eval: epoch 56 | batch 380/510 [I 2022-10-27 08:14:39] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 08:14:39] ...trap/engines/engine.py.326: process: 0.29128 | load: 0.00023 [I 2022-10-27 08:14:42] ...trap/engines/engine.py.321: eval: epoch 56 | batch 390/510 [I 2022-10-27 08:14:42] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 08:14:42] ...trap/engines/engine.py.326: process: 0.28976 | load: 0.00019 [I 2022-10-27 08:14:45] ...trap/engines/engine.py.321: eval: epoch 56 | batch 400/510 [I 2022-10-27 08:14:45] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 08:14:45] ...trap/engines/engine.py.326: process: 0.29135 | load: 0.00017 [I 2022-10-27 08:14:48] ...trap/engines/engine.py.321: eval: epoch 56 | batch 410/510 [I 2022-10-27 08:14:48] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 08:14:48] ...trap/engines/engine.py.326: process: 0.29340 | load: 0.00022 [I 2022-10-27 08:14:51] ...trap/engines/engine.py.321: eval: epoch 56 | batch 420/510 [I 2022-10-27 08:14:51] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:26 [I 2022-10-27 08:14:51] ...trap/engines/engine.py.326: process: 0.29479 | load: 0.00016 [I 2022-10-27 08:14:53] ...trap/engines/engine.py.321: eval: epoch 56 | batch 430/510 [I 2022-10-27 08:14:53] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:22 [I 2022-10-27 08:14:53] ...trap/engines/engine.py.326: process: 0.28237 | load: 0.00024 [I 2022-10-27 08:14:56] ...trap/engines/engine.py.321: eval: epoch 56 | batch 440/510 [I 2022-10-27 08:14:56] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 08:14:56] ...trap/engines/engine.py.326: process: 0.29193 | load: 0.00016 [I 2022-10-27 08:14:59] ...trap/engines/engine.py.321: eval: epoch 56 | batch 450/510 [I 2022-10-27 08:14:59] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 08:14:59] ...trap/engines/engine.py.326: process: 0.28951 | load: 0.00019 [I 2022-10-27 08:15:02] ...trap/engines/engine.py.321: eval: epoch 56 | batch 460/510 [I 2022-10-27 08:15:02] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 08:15:02] ...trap/engines/engine.py.326: process: 0.30518 | load: 0.00018 [I 2022-10-27 08:15:05] ...trap/engines/engine.py.321: eval: epoch 56 | batch 470/510 [I 2022-10-27 08:15:05] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 08:15:05] ...trap/engines/engine.py.326: process: 0.28356 | load: 0.00018 [I 2022-10-27 08:15:08] ...trap/engines/engine.py.321: eval: epoch 56 | batch 480/510 [I 2022-10-27 08:15:08] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 08:15:08] ...trap/engines/engine.py.326: process: 0.28110 | load: 0.00023 [I 2022-10-27 08:15:11] ...trap/engines/engine.py.321: eval: epoch 56 | batch 490/510 [I 2022-10-27 08:15:11] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 08:15:11] ...trap/engines/engine.py.326: process: 0.28720 | load: 0.00017 [I 2022-10-27 08:15:14] ...trap/engines/engine.py.321: eval: epoch 56 | batch 500/510 [I 2022-10-27 08:15:14] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 08:15:14] ...trap/engines/engine.py.326: process: 0.30286 | load: 0.00016 [I 2022-10-27 08:15:17] ...trap/engines/engine.py.321: eval: epoch 56 | batch 510/510 [I 2022-10-27 08:15:17] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 08:15:17] ...trap/engines/engine.py.326: process: 0.28496 | load: 0.00010 [S 2022-10-27 08:15:17] ...trap/engines/engine.py.132: eval_epoch.epoch: 56 [I 2022-10-27 08:15:17] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.707000 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.897300 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935700 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.692100 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.896100 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932900 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014484 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009675 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007471 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010876 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005338 [S 2022-10-27 08:15:21] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006332 [I 2022-10-27 08:15:23] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 08:15:23] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 08:15:23] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 08:15:23] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 08:15:23] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 08:15:26] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 08:15:27] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 08:15:33] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 08:15:33] ...trap/engines/engine.py.163: Training model on trainset for epoch 57 [I 2022-10-27 08:15:33] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 08:15:36] ...trap/engines/engine.py.224: train: epoch 57 | batch 0/2304 [I 2022-10-27 08:15:36] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:41 [I 2022-10-27 08:15:36] ...trap/engines/engine.py.229: process: 2.70044 | load: 2.05171 [I 2022-10-27 08:15:36] ...trap/engines/engine.py.230: loss: 0.14400 [I 2022-10-27 08:15:36] ...optimizers/trijoint.py.151: total_norm: 13.028142 [I 2022-10-27 08:15:41] ...trap/engines/engine.py.224: train: epoch 57 | batch 10/2304 [I 2022-10-27 08:15:41] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:06 [I 2022-10-27 08:15:41] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00485 [I 2022-10-27 08:15:41] ...trap/engines/engine.py.230: loss: 0.13618 [I 2022-10-27 08:15:41] ...optimizers/trijoint.py.151: total_norm: 10.927043 [I 2022-10-27 08:15:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 20/2304 [I 2022-10-27 08:15:47] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:09 [I 2022-10-27 08:15:47] ...trap/engines/engine.py.229: process: 0.55830 | load: 0.00359 [I 2022-10-27 08:15:47] ...trap/engines/engine.py.230: loss: 0.19189 [I 2022-10-27 08:15:47] ...optimizers/trijoint.py.151: total_norm: 33.976440 [I 2022-10-27 08:15:53] ...trap/engines/engine.py.224: train: epoch 57 | batch 30/2304 [I 2022-10-27 08:15:53] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:05 [I 2022-10-27 08:15:53] ...trap/engines/engine.py.229: process: 0.55758 | load: 0.00273 [I 2022-10-27 08:15:53] ...trap/engines/engine.py.230: loss: 0.11745 [I 2022-10-27 08:15:53] ...optimizers/trijoint.py.151: total_norm: 18.315527 [I 2022-10-27 08:15:58] ...trap/engines/engine.py.224: train: epoch 57 | batch 40/2304 [I 2022-10-27 08:15:58] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:08 [I 2022-10-27 08:15:58] ...trap/engines/engine.py.229: process: 0.56926 | load: 0.00380 [I 2022-10-27 08:15:58] ...trap/engines/engine.py.230: loss: 0.12084 [I 2022-10-27 08:15:58] ...optimizers/trijoint.py.151: total_norm: 20.365276 [I 2022-10-27 08:16:04] ...trap/engines/engine.py.224: train: epoch 57 | batch 50/2304 [I 2022-10-27 08:16:04] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:21 [I 2022-10-27 08:16:04] ...trap/engines/engine.py.229: process: 0.57438 | load: 0.00275 [I 2022-10-27 08:16:04] ...trap/engines/engine.py.230: loss: 0.14001 [I 2022-10-27 08:16:04] ...optimizers/trijoint.py.151: total_norm: 13.406569 [I 2022-10-27 08:16:09] ...trap/engines/engine.py.224: train: epoch 57 | batch 60/2304 [I 2022-10-27 08:16:09] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:49 [I 2022-10-27 08:16:09] ...trap/engines/engine.py.229: process: 0.55597 | load: 0.00301 [I 2022-10-27 08:16:09] ...trap/engines/engine.py.230: loss: 0.15798 [I 2022-10-27 08:16:09] ...optimizers/trijoint.py.151: total_norm: 27.458237 [I 2022-10-27 08:16:15] ...trap/engines/engine.py.224: train: epoch 57 | batch 70/2304 [I 2022-10-27 08:16:15] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:54 [I 2022-10-27 08:16:15] ...trap/engines/engine.py.229: process: 0.56411 | load: 0.00294 [I 2022-10-27 08:16:15] ...trap/engines/engine.py.230: loss: 0.15723 [I 2022-10-27 08:16:15] ...optimizers/trijoint.py.151: total_norm: 16.819834 [I 2022-10-27 08:16:21] ...trap/engines/engine.py.224: train: epoch 57 | batch 80/2304 [I 2022-10-27 08:16:21] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:40 [I 2022-10-27 08:16:21] ...trap/engines/engine.py.229: process: 0.55641 | load: 0.00265 [I 2022-10-27 08:16:21] ...trap/engines/engine.py.230: loss: 0.11919 [I 2022-10-27 08:16:21] ...optimizers/trijoint.py.151: total_norm: 13.843981 [I 2022-10-27 08:16:26] ...trap/engines/engine.py.224: train: epoch 57 | batch 90/2304 [I 2022-10-27 08:16:26] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:43 [I 2022-10-27 08:16:26] ...trap/engines/engine.py.229: process: 0.56888 | load: 0.00275 [I 2022-10-27 08:16:26] ...trap/engines/engine.py.230: loss: 0.11243 [I 2022-10-27 08:16:26] ...optimizers/trijoint.py.151: total_norm: 10.903590 [I 2022-10-27 08:16:32] ...trap/engines/engine.py.224: train: epoch 57 | batch 100/2304 [I 2022-10-27 08:16:32] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:31 [I 2022-10-27 08:16:32] ...trap/engines/engine.py.229: process: 0.56869 | load: 0.00254 [I 2022-10-27 08:16:32] ...trap/engines/engine.py.230: loss: 0.11754 [I 2022-10-27 08:16:32] ...optimizers/trijoint.py.151: total_norm: 11.149212 [I 2022-10-27 08:16:37] ...trap/engines/engine.py.224: train: epoch 57 | batch 110/2304 [I 2022-10-27 08:16:37] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:27 [I 2022-10-27 08:16:37] ...trap/engines/engine.py.229: process: 0.57738 | load: 0.00312 [I 2022-10-27 08:16:37] ...trap/engines/engine.py.230: loss: 0.12241 [I 2022-10-27 08:16:37] ...optimizers/trijoint.py.151: total_norm: 17.482071 [I 2022-10-27 08:16:43] ...trap/engines/engine.py.224: train: epoch 57 | batch 120/2304 [I 2022-10-27 08:16:43] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:17 [I 2022-10-27 08:16:43] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00282 [I 2022-10-27 08:16:43] ...trap/engines/engine.py.230: loss: 0.16921 [I 2022-10-27 08:16:43] ...optimizers/trijoint.py.151: total_norm: 26.021927 [I 2022-10-27 08:16:49] ...trap/engines/engine.py.224: train: epoch 57 | batch 130/2304 [I 2022-10-27 08:16:49] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:07 [I 2022-10-27 08:16:49] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00284 [I 2022-10-27 08:16:49] ...trap/engines/engine.py.230: loss: 0.11956 [I 2022-10-27 08:16:49] ...optimizers/trijoint.py.151: total_norm: 11.996733 [I 2022-10-27 08:16:54] ...trap/engines/engine.py.224: train: epoch 57 | batch 140/2304 [I 2022-10-27 08:16:54] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:17 [I 2022-10-27 08:16:54] ...trap/engines/engine.py.229: process: 0.57568 | load: 0.00273 [I 2022-10-27 08:16:54] ...trap/engines/engine.py.230: loss: 0.22047 [I 2022-10-27 08:16:54] ...optimizers/trijoint.py.151: total_norm: 14.826098 [I 2022-10-27 08:17:00] ...trap/engines/engine.py.224: train: epoch 57 | batch 150/2304 [I 2022-10-27 08:17:00] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:07 [I 2022-10-27 08:17:00] ...trap/engines/engine.py.229: process: 0.54338 | load: 0.00247 [I 2022-10-27 08:17:00] ...trap/engines/engine.py.230: loss: 0.12373 [I 2022-10-27 08:17:00] ...optimizers/trijoint.py.151: total_norm: 10.343609 [I 2022-10-27 08:17:06] ...trap/engines/engine.py.224: train: epoch 57 | batch 160/2304 [I 2022-10-27 08:17:06] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:04 [I 2022-10-27 08:17:06] ...trap/engines/engine.py.229: process: 0.56261 | load: 0.00262 [I 2022-10-27 08:17:06] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-27 08:17:06] ...optimizers/trijoint.py.151: total_norm: 8.722656 [I 2022-10-27 08:17:11] ...trap/engines/engine.py.224: train: epoch 57 | batch 170/2304 [I 2022-10-27 08:17:11] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:54 [I 2022-10-27 08:17:11] ...trap/engines/engine.py.229: process: 0.56329 | load: 0.00270 [I 2022-10-27 08:17:11] ...trap/engines/engine.py.230: loss: 0.14899 [I 2022-10-27 08:17:11] ...optimizers/trijoint.py.151: total_norm: 12.440463 [I 2022-10-27 08:17:17] ...trap/engines/engine.py.224: train: epoch 57 | batch 180/2304 [I 2022-10-27 08:17:17] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:46 [I 2022-10-27 08:17:17] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00251 [I 2022-10-27 08:17:17] ...trap/engines/engine.py.230: loss: 0.17737 [I 2022-10-27 08:17:17] ...optimizers/trijoint.py.151: total_norm: 13.930489 [I 2022-10-27 08:17:22] ...trap/engines/engine.py.224: train: epoch 57 | batch 190/2304 [I 2022-10-27 08:17:22] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:47 [I 2022-10-27 08:17:22] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00272 [I 2022-10-27 08:17:22] ...trap/engines/engine.py.230: loss: 0.17221 [I 2022-10-27 08:17:22] ...optimizers/trijoint.py.151: total_norm: 14.969613 [I 2022-10-27 08:17:28] ...trap/engines/engine.py.224: train: epoch 57 | batch 200/2304 [I 2022-10-27 08:17:28] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:35 [I 2022-10-27 08:17:28] ...trap/engines/engine.py.229: process: 0.55847 | load: 0.00313 [I 2022-10-27 08:17:28] ...trap/engines/engine.py.230: loss: 0.13076 [I 2022-10-27 08:17:28] ...optimizers/trijoint.py.151: total_norm: 17.367590 [I 2022-10-27 08:17:34] ...trap/engines/engine.py.224: train: epoch 57 | batch 210/2304 [I 2022-10-27 08:17:34] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:38 [I 2022-10-27 08:17:34] ...trap/engines/engine.py.229: process: 0.55181 | load: 0.00268 [I 2022-10-27 08:17:34] ...trap/engines/engine.py.230: loss: 0.19978 [I 2022-10-27 08:17:34] ...optimizers/trijoint.py.151: total_norm: 16.738642 [I 2022-10-27 08:17:39] ...trap/engines/engine.py.224: train: epoch 57 | batch 220/2304 [I 2022-10-27 08:17:39] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:28 [I 2022-10-27 08:17:39] ...trap/engines/engine.py.229: process: 0.57065 | load: 0.00250 [I 2022-10-27 08:17:39] ...trap/engines/engine.py.230: loss: 0.17027 [I 2022-10-27 08:17:39] ...optimizers/trijoint.py.151: total_norm: 30.923075 [I 2022-10-27 08:17:45] ...trap/engines/engine.py.224: train: epoch 57 | batch 230/2304 [I 2022-10-27 08:17:45] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:20 [I 2022-10-27 08:17:45] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00256 [I 2022-10-27 08:17:45] ...trap/engines/engine.py.230: loss: 0.16391 [I 2022-10-27 08:17:45] ...optimizers/trijoint.py.151: total_norm: 42.080425 [I 2022-10-27 08:17:51] ...trap/engines/engine.py.224: train: epoch 57 | batch 240/2304 [I 2022-10-27 08:17:51] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:29 [I 2022-10-27 08:17:51] ...trap/engines/engine.py.229: process: 0.57018 | load: 0.00275 [I 2022-10-27 08:17:51] ...trap/engines/engine.py.230: loss: 0.17752 [I 2022-10-27 08:17:51] ...optimizers/trijoint.py.151: total_norm: 12.201991 [I 2022-10-27 08:17:56] ...trap/engines/engine.py.224: train: epoch 57 | batch 250/2304 [I 2022-10-27 08:17:56] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:18 [I 2022-10-27 08:17:56] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00257 [I 2022-10-27 08:17:56] ...trap/engines/engine.py.230: loss: 0.18692 [I 2022-10-27 08:17:56] ...optimizers/trijoint.py.151: total_norm: 19.006939 [I 2022-10-27 08:18:02] ...trap/engines/engine.py.224: train: epoch 57 | batch 260/2304 [I 2022-10-27 08:18:02] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:07 [I 2022-10-27 08:18:02] ...trap/engines/engine.py.229: process: 0.56598 | load: 0.00281 [I 2022-10-27 08:18:02] ...trap/engines/engine.py.230: loss: 0.15070 [I 2022-10-27 08:18:02] ...optimizers/trijoint.py.151: total_norm: 13.447901 [I 2022-10-27 08:18:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 270/2304 [I 2022-10-27 08:18:08] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:19:03 [I 2022-10-27 08:18:08] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00275 [I 2022-10-27 08:18:08] ...trap/engines/engine.py.230: loss: 0.14403 [I 2022-10-27 08:18:08] ...optimizers/trijoint.py.151: total_norm: 21.628235 [I 2022-10-27 08:18:13] ...trap/engines/engine.py.224: train: epoch 57 | batch 280/2304 [I 2022-10-27 08:18:13] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:19:00 [I 2022-10-27 08:18:13] ...trap/engines/engine.py.229: process: 0.58616 | load: 0.00251 [I 2022-10-27 08:18:13] ...trap/engines/engine.py.230: loss: 0.11732 [I 2022-10-27 08:18:13] ...optimizers/trijoint.py.151: total_norm: 17.323154 [I 2022-10-27 08:18:19] ...trap/engines/engine.py.224: train: epoch 57 | batch 290/2304 [I 2022-10-27 08:18:19] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:35 [I 2022-10-27 08:18:19] ...trap/engines/engine.py.229: process: 0.55038 | load: 0.00247 [I 2022-10-27 08:18:19] ...trap/engines/engine.py.230: loss: 0.24761 [I 2022-10-27 08:18:19] ...optimizers/trijoint.py.151: total_norm: 14.347175 [I 2022-10-27 08:18:44] ...trap/engines/engine.py.224: train: epoch 57 | batch 300/2304 [I 2022-10-27 08:18:44] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:36:34 [I 2022-10-27 08:18:44] ...trap/engines/engine.py.229: process: 0.54852 | load: 0.00240 [I 2022-10-27 08:18:44] ...trap/engines/engine.py.230: loss: 0.18957 [I 2022-10-27 08:18:44] ...optimizers/trijoint.py.151: total_norm: 16.241594 [I 2022-10-27 08:18:50] ...trap/engines/engine.py.224: train: epoch 57 | batch 310/2304 [I 2022-10-27 08:18:50] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:20:20 [I 2022-10-27 08:18:50] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00238 [I 2022-10-27 08:18:50] ...trap/engines/engine.py.230: loss: 0.19316 [I 2022-10-27 08:18:50] ...optimizers/trijoint.py.151: total_norm: 19.404978 [I 2022-10-27 08:18:56] ...trap/engines/engine.py.224: train: epoch 57 | batch 320/2304 [I 2022-10-27 08:18:56] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:18:35 [I 2022-10-27 08:18:56] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00243 [I 2022-10-27 08:18:56] ...trap/engines/engine.py.230: loss: 0.11635 [I 2022-10-27 08:18:56] ...optimizers/trijoint.py.151: total_norm: 8.564796 [I 2022-10-27 08:19:01] ...trap/engines/engine.py.224: train: epoch 57 | batch 330/2304 [I 2022-10-27 08:19:01] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:18:25 [I 2022-10-27 08:19:01] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00240 [I 2022-10-27 08:19:01] ...trap/engines/engine.py.230: loss: 0.15335 [I 2022-10-27 08:19:01] ...optimizers/trijoint.py.151: total_norm: 23.295395 [I 2022-10-27 08:19:07] ...trap/engines/engine.py.224: train: epoch 57 | batch 340/2304 [I 2022-10-27 08:19:07] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:18:13 [I 2022-10-27 08:19:07] ...trap/engines/engine.py.229: process: 0.55000 | load: 0.00239 [I 2022-10-27 08:19:07] ...trap/engines/engine.py.230: loss: 0.13863 [I 2022-10-27 08:19:07] ...optimizers/trijoint.py.151: total_norm: 15.711669 [I 2022-10-27 08:19:12] ...trap/engines/engine.py.224: train: epoch 57 | batch 350/2304 [I 2022-10-27 08:19:12] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:18:03 [I 2022-10-27 08:19:12] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00239 [I 2022-10-27 08:19:12] ...trap/engines/engine.py.230: loss: 0.11100 [I 2022-10-27 08:19:12] ...optimizers/trijoint.py.151: total_norm: 15.190936 [I 2022-10-27 08:19:18] ...trap/engines/engine.py.224: train: epoch 57 | batch 360/2304 [I 2022-10-27 08:19:18] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:18:11 [I 2022-10-27 08:19:18] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00248 [I 2022-10-27 08:19:18] ...trap/engines/engine.py.230: loss: 0.12158 [I 2022-10-27 08:19:18] ...optimizers/trijoint.py.151: total_norm: 16.550219 [I 2022-10-27 08:19:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 370/2304 [I 2022-10-27 08:19:23] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:17:57 [I 2022-10-27 08:19:23] ...trap/engines/engine.py.229: process: 0.55041 | load: 0.00256 [I 2022-10-27 08:19:23] ...trap/engines/engine.py.230: loss: 0.10365 [I 2022-10-27 08:19:23] ...optimizers/trijoint.py.151: total_norm: 13.597412 [I 2022-10-27 08:19:29] ...trap/engines/engine.py.224: train: epoch 57 | batch 380/2304 [I 2022-10-27 08:19:29] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:17:57 [I 2022-10-27 08:19:29] ...trap/engines/engine.py.229: process: 0.56928 | load: 0.00250 [I 2022-10-27 08:19:29] ...trap/engines/engine.py.230: loss: 0.16903 [I 2022-10-27 08:19:29] ...optimizers/trijoint.py.151: total_norm: 28.136242 [I 2022-10-27 08:19:35] ...trap/engines/engine.py.224: train: epoch 57 | batch 390/2304 [I 2022-10-27 08:19:35] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:17:51 [I 2022-10-27 08:19:35] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00237 [I 2022-10-27 08:19:35] ...trap/engines/engine.py.230: loss: 0.16618 [I 2022-10-27 08:19:35] ...optimizers/trijoint.py.151: total_norm: 16.499344 [I 2022-10-27 08:19:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 400/2304 [I 2022-10-27 08:19:40] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:17:48 [I 2022-10-27 08:19:40] ...trap/engines/engine.py.229: process: 0.56517 | load: 0.00246 [I 2022-10-27 08:19:40] ...trap/engines/engine.py.230: loss: 0.14577 [I 2022-10-27 08:19:40] ...optimizers/trijoint.py.151: total_norm: 12.795999 [I 2022-10-27 08:19:46] ...trap/engines/engine.py.224: train: epoch 57 | batch 410/2304 [I 2022-10-27 08:19:46] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:17:29 [I 2022-10-27 08:19:46] ...trap/engines/engine.py.229: process: 0.54692 | load: 0.00239 [I 2022-10-27 08:19:46] ...trap/engines/engine.py.230: loss: 0.18001 [I 2022-10-27 08:19:46] ...optimizers/trijoint.py.151: total_norm: 10.724237 [I 2022-10-27 08:19:51] ...trap/engines/engine.py.224: train: epoch 57 | batch 420/2304 [I 2022-10-27 08:19:51] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:17:31 [I 2022-10-27 08:19:52] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00241 [I 2022-10-27 08:19:52] ...trap/engines/engine.py.230: loss: 0.16077 [I 2022-10-27 08:19:52] ...optimizers/trijoint.py.151: total_norm: 7.601042 [I 2022-10-27 08:19:57] ...trap/engines/engine.py.224: train: epoch 57 | batch 430/2304 [I 2022-10-27 08:19:57] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:17:18 [I 2022-10-27 08:19:57] ...trap/engines/engine.py.229: process: 0.54594 | load: 0.00241 [I 2022-10-27 08:19:57] ...trap/engines/engine.py.230: loss: 0.13342 [I 2022-10-27 08:19:57] ...optimizers/trijoint.py.151: total_norm: 21.933088 [I 2022-10-27 08:20:03] ...trap/engines/engine.py.224: train: epoch 57 | batch 440/2304 [I 2022-10-27 08:20:03] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:17:20 [I 2022-10-27 08:20:03] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00249 [I 2022-10-27 08:20:03] ...trap/engines/engine.py.230: loss: 0.09468 [I 2022-10-27 08:20:03] ...optimizers/trijoint.py.151: total_norm: 11.096211 [I 2022-10-27 08:20:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 450/2304 [I 2022-10-27 08:20:08] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:17:14 [I 2022-10-27 08:20:08] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00244 [I 2022-10-27 08:20:08] ...trap/engines/engine.py.230: loss: 0.16287 [I 2022-10-27 08:20:08] ...optimizers/trijoint.py.151: total_norm: 14.201647 [I 2022-10-27 08:20:14] ...trap/engines/engine.py.224: train: epoch 57 | batch 460/2304 [I 2022-10-27 08:20:14] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:17:13 [I 2022-10-27 08:20:14] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00241 [I 2022-10-27 08:20:14] ...trap/engines/engine.py.230: loss: 0.20831 [I 2022-10-27 08:20:14] ...optimizers/trijoint.py.151: total_norm: 20.313911 [I 2022-10-27 08:20:19] ...trap/engines/engine.py.224: train: epoch 57 | batch 470/2304 [I 2022-10-27 08:20:19] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:17:06 [I 2022-10-27 08:20:19] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00245 [I 2022-10-27 08:20:19] ...trap/engines/engine.py.230: loss: 0.17145 [I 2022-10-27 08:20:19] ...optimizers/trijoint.py.151: total_norm: 19.837791 [I 2022-10-27 08:20:25] ...trap/engines/engine.py.224: train: epoch 57 | batch 480/2304 [I 2022-10-27 08:20:25] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:16:56 [I 2022-10-27 08:20:25] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00241 [I 2022-10-27 08:20:25] ...trap/engines/engine.py.230: loss: 0.20643 [I 2022-10-27 08:20:25] ...optimizers/trijoint.py.151: total_norm: 22.484873 [I 2022-10-27 08:20:31] ...trap/engines/engine.py.224: train: epoch 57 | batch 490/2304 [I 2022-10-27 08:20:31] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:16:39 [I 2022-10-27 08:20:31] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00240 [I 2022-10-27 08:20:31] ...trap/engines/engine.py.230: loss: 0.14094 [I 2022-10-27 08:20:31] ...optimizers/trijoint.py.151: total_norm: 21.352961 [I 2022-10-27 08:20:36] ...trap/engines/engine.py.224: train: epoch 57 | batch 500/2304 [I 2022-10-27 08:20:36] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:16:44 [I 2022-10-27 08:20:36] ...trap/engines/engine.py.229: process: 0.55190 | load: 0.00244 [I 2022-10-27 08:20:36] ...trap/engines/engine.py.230: loss: 0.18344 [I 2022-10-27 08:20:36] ...optimizers/trijoint.py.151: total_norm: 16.454140 [I 2022-10-27 08:20:42] ...trap/engines/engine.py.224: train: epoch 57 | batch 510/2304 [I 2022-10-27 08:20:42] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:16:45 [I 2022-10-27 08:20:42] ...trap/engines/engine.py.229: process: 0.55257 | load: 0.00249 [I 2022-10-27 08:20:42] ...trap/engines/engine.py.230: loss: 0.20464 [I 2022-10-27 08:20:42] ...optimizers/trijoint.py.151: total_norm: 28.543285 [I 2022-10-27 08:20:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 520/2304 [I 2022-10-27 08:20:47] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:16:26 [I 2022-10-27 08:20:47] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00244 [I 2022-10-27 08:20:47] ...trap/engines/engine.py.230: loss: 0.10356 [I 2022-10-27 08:20:47] ...optimizers/trijoint.py.151: total_norm: 10.673462 [I 2022-10-27 08:20:53] ...trap/engines/engine.py.224: train: epoch 57 | batch 530/2304 [I 2022-10-27 08:20:53] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:16:21 [I 2022-10-27 08:20:53] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00244 [I 2022-10-27 08:20:53] ...trap/engines/engine.py.230: loss: 0.20343 [I 2022-10-27 08:20:53] ...optimizers/trijoint.py.151: total_norm: 13.121799 [I 2022-10-27 08:20:58] ...trap/engines/engine.py.224: train: epoch 57 | batch 540/2304 [I 2022-10-27 08:20:58] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:16:19 [I 2022-10-27 08:20:58] ...trap/engines/engine.py.229: process: 0.56149 | load: 0.00244 [I 2022-10-27 08:20:58] ...trap/engines/engine.py.230: loss: 0.15849 [I 2022-10-27 08:20:58] ...optimizers/trijoint.py.151: total_norm: 47.447041 [I 2022-10-27 08:21:04] ...trap/engines/engine.py.224: train: epoch 57 | batch 550/2304 [I 2022-10-27 08:21:04] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:16:09 [I 2022-10-27 08:21:04] ...trap/engines/engine.py.229: process: 0.54190 | load: 0.00244 [I 2022-10-27 08:21:04] ...trap/engines/engine.py.230: loss: 0.11412 [I 2022-10-27 08:21:04] ...optimizers/trijoint.py.151: total_norm: 7.682362 [I 2022-10-27 08:21:10] ...trap/engines/engine.py.224: train: epoch 57 | batch 560/2304 [I 2022-10-27 08:21:10] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:16:06 [I 2022-10-27 08:21:10] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00243 [I 2022-10-27 08:21:10] ...trap/engines/engine.py.230: loss: 0.11265 [I 2022-10-27 08:21:10] ...optimizers/trijoint.py.151: total_norm: 8.363938 [I 2022-10-27 08:21:15] ...trap/engines/engine.py.224: train: epoch 57 | batch 570/2304 [I 2022-10-27 08:21:15] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:16:01 [I 2022-10-27 08:21:15] ...trap/engines/engine.py.229: process: 0.54188 | load: 0.00245 [I 2022-10-27 08:21:15] ...trap/engines/engine.py.230: loss: 0.12864 [I 2022-10-27 08:21:15] ...optimizers/trijoint.py.151: total_norm: 12.958393 [I 2022-10-27 08:21:21] ...trap/engines/engine.py.224: train: epoch 57 | batch 580/2304 [I 2022-10-27 08:21:21] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:15:57 [I 2022-10-27 08:21:21] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00247 [I 2022-10-27 08:21:21] ...trap/engines/engine.py.230: loss: 0.18157 [I 2022-10-27 08:21:21] ...optimizers/trijoint.py.151: total_norm: 16.187222 [I 2022-10-27 08:21:26] ...trap/engines/engine.py.224: train: epoch 57 | batch 590/2304 [I 2022-10-27 08:21:26] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:15:52 [I 2022-10-27 08:21:26] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00245 [I 2022-10-27 08:21:26] ...trap/engines/engine.py.230: loss: 0.14605 [I 2022-10-27 08:21:26] ...optimizers/trijoint.py.151: total_norm: 31.684559 [I 2022-10-27 08:21:32] ...trap/engines/engine.py.224: train: epoch 57 | batch 600/2304 [I 2022-10-27 08:21:32] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:47 [I 2022-10-27 08:21:32] ...trap/engines/engine.py.229: process: 0.55268 | load: 0.00244 [I 2022-10-27 08:21:32] ...trap/engines/engine.py.230: loss: 0.16824 [I 2022-10-27 08:21:32] ...optimizers/trijoint.py.151: total_norm: 35.170631 [I 2022-10-27 08:21:37] ...trap/engines/engine.py.224: train: epoch 57 | batch 610/2304 [I 2022-10-27 08:21:37] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:38 [I 2022-10-27 08:21:37] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00243 [I 2022-10-27 08:21:37] ...trap/engines/engine.py.230: loss: 0.15674 [I 2022-10-27 08:21:37] ...optimizers/trijoint.py.151: total_norm: 24.997561 [I 2022-10-27 08:21:43] ...trap/engines/engine.py.224: train: epoch 57 | batch 620/2304 [I 2022-10-27 08:21:43] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:38 [I 2022-10-27 08:21:43] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00244 [I 2022-10-27 08:21:43] ...trap/engines/engine.py.230: loss: 0.19839 [I 2022-10-27 08:21:43] ...optimizers/trijoint.py.151: total_norm: 15.275634 [I 2022-10-27 08:21:49] ...trap/engines/engine.py.224: train: epoch 57 | batch 630/2304 [I 2022-10-27 08:21:49] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:30 [I 2022-10-27 08:21:49] ...trap/engines/engine.py.229: process: 0.55865 | load: 0.00240 [I 2022-10-27 08:21:49] ...trap/engines/engine.py.230: loss: 0.14479 [I 2022-10-27 08:21:49] ...optimizers/trijoint.py.151: total_norm: 10.520628 [I 2022-10-27 08:21:54] ...trap/engines/engine.py.224: train: epoch 57 | batch 640/2304 [I 2022-10-27 08:21:54] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:24 [I 2022-10-27 08:21:54] ...trap/engines/engine.py.229: process: 0.55863 | load: 0.00245 [I 2022-10-27 08:21:54] ...trap/engines/engine.py.230: loss: 0.11187 [I 2022-10-27 08:21:54] ...optimizers/trijoint.py.151: total_norm: 7.257896 [I 2022-10-27 08:22:00] ...trap/engines/engine.py.224: train: epoch 57 | batch 650/2304 [I 2022-10-27 08:22:00] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:15 [I 2022-10-27 08:22:00] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00242 [I 2022-10-27 08:22:00] ...trap/engines/engine.py.230: loss: 0.18692 [I 2022-10-27 08:22:00] ...optimizers/trijoint.py.151: total_norm: 14.817110 [I 2022-10-27 08:22:05] ...trap/engines/engine.py.224: train: epoch 57 | batch 660/2304 [I 2022-10-27 08:22:05] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:15:17 [I 2022-10-27 08:22:05] ...trap/engines/engine.py.229: process: 0.54767 | load: 0.00243 [I 2022-10-27 08:22:05] ...trap/engines/engine.py.230: loss: 0.13771 [I 2022-10-27 08:22:05] ...optimizers/trijoint.py.151: total_norm: 19.081549 [I 2022-10-27 08:22:11] ...trap/engines/engine.py.224: train: epoch 57 | batch 670/2304 [I 2022-10-27 08:22:11] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:05 [I 2022-10-27 08:22:11] ...trap/engines/engine.py.229: process: 0.54857 | load: 0.00244 [I 2022-10-27 08:22:11] ...trap/engines/engine.py.230: loss: 0.15787 [I 2022-10-27 08:22:11] ...optimizers/trijoint.py.151: total_norm: 20.889633 [I 2022-10-27 08:22:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 680/2304 [I 2022-10-27 08:22:16] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:02 [I 2022-10-27 08:22:16] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00241 [I 2022-10-27 08:22:16] ...trap/engines/engine.py.230: loss: 0.15843 [I 2022-10-27 08:22:16] ...optimizers/trijoint.py.151: total_norm: 13.604065 [I 2022-10-27 08:22:22] ...trap/engines/engine.py.224: train: epoch 57 | batch 690/2304 [I 2022-10-27 08:22:22] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:14:54 [I 2022-10-27 08:22:22] ...trap/engines/engine.py.229: process: 0.55155 | load: 0.00242 [I 2022-10-27 08:22:22] ...trap/engines/engine.py.230: loss: 0.12223 [I 2022-10-27 08:22:22] ...optimizers/trijoint.py.151: total_norm: 9.663581 [I 2022-10-27 08:22:28] ...trap/engines/engine.py.224: train: epoch 57 | batch 700/2304 [I 2022-10-27 08:22:28] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:14:55 [I 2022-10-27 08:22:28] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00242 [I 2022-10-27 08:22:28] ...trap/engines/engine.py.230: loss: 0.17019 [I 2022-10-27 08:22:28] ...optimizers/trijoint.py.151: total_norm: 41.111389 [I 2022-10-27 08:22:33] ...trap/engines/engine.py.224: train: epoch 57 | batch 710/2304 [I 2022-10-27 08:22:33] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:14:40 [I 2022-10-27 08:22:33] ...trap/engines/engine.py.229: process: 0.52986 | load: 0.00251 [I 2022-10-27 08:22:33] ...trap/engines/engine.py.230: loss: 0.10610 [I 2022-10-27 08:22:33] ...optimizers/trijoint.py.151: total_norm: 7.076305 [I 2022-10-27 08:22:39] ...trap/engines/engine.py.224: train: epoch 57 | batch 720/2304 [I 2022-10-27 08:22:39] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:41 [I 2022-10-27 08:22:39] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00257 [I 2022-10-27 08:22:39] ...trap/engines/engine.py.230: loss: 0.17042 [I 2022-10-27 08:22:39] ...optimizers/trijoint.py.151: total_norm: 28.435406 [I 2022-10-27 08:22:44] ...trap/engines/engine.py.224: train: epoch 57 | batch 730/2304 [I 2022-10-27 08:22:44] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:37 [I 2022-10-27 08:22:44] ...trap/engines/engine.py.229: process: 0.57741 | load: 0.00251 [I 2022-10-27 08:22:44] ...trap/engines/engine.py.230: loss: 0.14330 [I 2022-10-27 08:22:44] ...optimizers/trijoint.py.151: total_norm: 24.797487 [I 2022-10-27 08:22:50] ...trap/engines/engine.py.224: train: epoch 57 | batch 740/2304 [I 2022-10-27 08:22:50] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:28 [I 2022-10-27 08:22:50] ...trap/engines/engine.py.229: process: 0.55724 | load: 0.00256 [I 2022-10-27 08:22:50] ...trap/engines/engine.py.230: loss: 0.23982 [I 2022-10-27 08:22:50] ...optimizers/trijoint.py.151: total_norm: 22.805201 [I 2022-10-27 08:22:55] ...trap/engines/engine.py.224: train: epoch 57 | batch 750/2304 [I 2022-10-27 08:22:55] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:18 [I 2022-10-27 08:22:55] ...trap/engines/engine.py.229: process: 0.54908 | load: 0.00249 [I 2022-10-27 08:22:55] ...trap/engines/engine.py.230: loss: 0.19537 [I 2022-10-27 08:22:55] ...optimizers/trijoint.py.151: total_norm: 34.203510 [I 2022-10-27 08:23:01] ...trap/engines/engine.py.224: train: epoch 57 | batch 760/2304 [I 2022-10-27 08:23:01] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:14:19 [I 2022-10-27 08:23:01] ...trap/engines/engine.py.229: process: 0.56328 | load: 0.00246 [I 2022-10-27 08:23:01] ...trap/engines/engine.py.230: loss: 0.15613 [I 2022-10-27 08:23:01] ...optimizers/trijoint.py.151: total_norm: 11.802211 [I 2022-10-27 08:23:07] ...trap/engines/engine.py.224: train: epoch 57 | batch 770/2304 [I 2022-10-27 08:23:07] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:14:12 [I 2022-10-27 08:23:07] ...trap/engines/engine.py.229: process: 0.55725 | load: 0.00258 [I 2022-10-27 08:23:07] ...trap/engines/engine.py.230: loss: 0.11900 [I 2022-10-27 08:23:07] ...optimizers/trijoint.py.151: total_norm: 17.932049 [I 2022-10-27 08:23:12] ...trap/engines/engine.py.224: train: epoch 57 | batch 780/2304 [I 2022-10-27 08:23:12] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:14:10 [I 2022-10-27 08:23:12] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00251 [I 2022-10-27 08:23:12] ...trap/engines/engine.py.230: loss: 0.14781 [I 2022-10-27 08:23:12] ...optimizers/trijoint.py.151: total_norm: 13.826713 [I 2022-10-27 08:23:18] ...trap/engines/engine.py.224: train: epoch 57 | batch 790/2304 [I 2022-10-27 08:23:18] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:13:59 [I 2022-10-27 08:23:18] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00253 [I 2022-10-27 08:23:18] ...trap/engines/engine.py.230: loss: 0.12142 [I 2022-10-27 08:23:18] ...optimizers/trijoint.py.151: total_norm: 19.156528 [I 2022-10-27 08:23:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 800/2304 [I 2022-10-27 08:23:23] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:13:48 [I 2022-10-27 08:23:23] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00256 [I 2022-10-27 08:23:23] ...trap/engines/engine.py.230: loss: 0.15391 [I 2022-10-27 08:23:23] ...optimizers/trijoint.py.151: total_norm: 19.370905 [I 2022-10-27 08:23:29] ...trap/engines/engine.py.224: train: epoch 57 | batch 810/2304 [I 2022-10-27 08:23:29] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:13:44 [I 2022-10-27 08:23:29] ...trap/engines/engine.py.229: process: 0.54489 | load: 0.00246 [I 2022-10-27 08:23:29] ...trap/engines/engine.py.230: loss: 0.15152 [I 2022-10-27 08:23:29] ...optimizers/trijoint.py.151: total_norm: 10.607250 [I 2022-10-27 08:23:34] ...trap/engines/engine.py.224: train: epoch 57 | batch 820/2304 [I 2022-10-27 08:23:34] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:13:39 [I 2022-10-27 08:23:34] ...trap/engines/engine.py.229: process: 0.54540 | load: 0.00276 [I 2022-10-27 08:23:34] ...trap/engines/engine.py.230: loss: 0.20023 [I 2022-10-27 08:23:34] ...optimizers/trijoint.py.151: total_norm: 21.129129 [I 2022-10-27 08:23:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 830/2304 [I 2022-10-27 08:23:40] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:13:38 [I 2022-10-27 08:23:40] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00256 [I 2022-10-27 08:23:40] ...trap/engines/engine.py.230: loss: 0.11960 [I 2022-10-27 08:23:40] ...optimizers/trijoint.py.151: total_norm: 18.429508 [I 2022-10-27 08:23:45] ...trap/engines/engine.py.224: train: epoch 57 | batch 840/2304 [I 2022-10-27 08:23:45] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:13:31 [I 2022-10-27 08:23:45] ...trap/engines/engine.py.229: process: 0.55283 | load: 0.00265 [I 2022-10-27 08:23:45] ...trap/engines/engine.py.230: loss: 0.14044 [I 2022-10-27 08:23:45] ...optimizers/trijoint.py.151: total_norm: 11.158282 [I 2022-10-27 08:23:51] ...trap/engines/engine.py.224: train: epoch 57 | batch 850/2304 [I 2022-10-27 08:23:51] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:13:24 [I 2022-10-27 08:23:51] ...trap/engines/engine.py.229: process: 0.54104 | load: 0.00259 [I 2022-10-27 08:23:51] ...trap/engines/engine.py.230: loss: 0.17807 [I 2022-10-27 08:23:51] ...optimizers/trijoint.py.151: total_norm: 10.745493 [I 2022-10-27 08:23:56] ...trap/engines/engine.py.224: train: epoch 57 | batch 860/2304 [I 2022-10-27 08:23:56] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:13:21 [I 2022-10-27 08:23:56] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00255 [I 2022-10-27 08:23:56] ...trap/engines/engine.py.230: loss: 0.15538 [I 2022-10-27 08:23:56] ...optimizers/trijoint.py.151: total_norm: 17.225460 [I 2022-10-27 08:24:02] ...trap/engines/engine.py.224: train: epoch 57 | batch 870/2304 [I 2022-10-27 08:24:02] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:18 [I 2022-10-27 08:24:02] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00254 [I 2022-10-27 08:24:02] ...trap/engines/engine.py.230: loss: 0.13780 [I 2022-10-27 08:24:02] ...optimizers/trijoint.py.151: total_norm: 28.690903 [I 2022-10-27 08:24:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 880/2304 [I 2022-10-27 08:24:08] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:13:12 [I 2022-10-27 08:24:08] ...trap/engines/engine.py.229: process: 0.55345 | load: 0.00254 [I 2022-10-27 08:24:08] ...trap/engines/engine.py.230: loss: 0.14246 [I 2022-10-27 08:24:08] ...optimizers/trijoint.py.151: total_norm: 11.581808 [I 2022-10-27 08:24:13] ...trap/engines/engine.py.224: train: epoch 57 | batch 890/2304 [I 2022-10-27 08:24:13] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:13:04 [I 2022-10-27 08:24:13] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00249 [I 2022-10-27 08:24:13] ...trap/engines/engine.py.230: loss: 0.16045 [I 2022-10-27 08:24:13] ...optimizers/trijoint.py.151: total_norm: 9.866398 [I 2022-10-27 08:24:19] ...trap/engines/engine.py.224: train: epoch 57 | batch 900/2304 [I 2022-10-27 08:24:19] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:12:59 [I 2022-10-27 08:24:19] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00245 [I 2022-10-27 08:24:19] ...trap/engines/engine.py.230: loss: 0.10416 [I 2022-10-27 08:24:19] ...optimizers/trijoint.py.151: total_norm: 18.458651 [I 2022-10-27 08:24:24] ...trap/engines/engine.py.224: train: epoch 57 | batch 910/2304 [I 2022-10-27 08:24:24] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:12:54 [I 2022-10-27 08:24:24] ...trap/engines/engine.py.229: process: 0.54537 | load: 0.00243 [I 2022-10-27 08:24:24] ...trap/engines/engine.py.230: loss: 0.15820 [I 2022-10-27 08:24:24] ...optimizers/trijoint.py.151: total_norm: 16.720686 [I 2022-10-27 08:24:30] ...trap/engines/engine.py.224: train: epoch 57 | batch 920/2304 [I 2022-10-27 08:24:30] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:12:52 [I 2022-10-27 08:24:30] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00246 [I 2022-10-27 08:24:30] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-27 08:24:30] ...optimizers/trijoint.py.151: total_norm: 11.221358 [I 2022-10-27 08:24:35] ...trap/engines/engine.py.224: train: epoch 57 | batch 930/2304 [I 2022-10-27 08:24:35] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:12:45 [I 2022-10-27 08:24:35] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00243 [I 2022-10-27 08:24:35] ...trap/engines/engine.py.230: loss: 0.14180 [I 2022-10-27 08:24:35] ...optimizers/trijoint.py.151: total_norm: 18.323120 [I 2022-10-27 08:24:41] ...trap/engines/engine.py.224: train: epoch 57 | batch 940/2304 [I 2022-10-27 08:24:41] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:36 [I 2022-10-27 08:24:41] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00249 [I 2022-10-27 08:24:41] ...trap/engines/engine.py.230: loss: 0.11457 [I 2022-10-27 08:24:41] ...optimizers/trijoint.py.151: total_norm: 15.395176 [I 2022-10-27 08:24:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 950/2304 [I 2022-10-27 08:24:47] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:36 [I 2022-10-27 08:24:47] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00249 [I 2022-10-27 08:24:47] ...trap/engines/engine.py.230: loss: 0.16568 [I 2022-10-27 08:24:47] ...optimizers/trijoint.py.151: total_norm: 15.009010 [I 2022-10-27 08:24:52] ...trap/engines/engine.py.224: train: epoch 57 | batch 960/2304 [I 2022-10-27 08:24:52] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:12:26 [I 2022-10-27 08:24:52] ...trap/engines/engine.py.229: process: 0.54734 | load: 0.00250 [I 2022-10-27 08:24:52] ...trap/engines/engine.py.230: loss: 0.14244 [I 2022-10-27 08:24:52] ...optimizers/trijoint.py.151: total_norm: 21.281097 [I 2022-10-27 08:24:58] ...trap/engines/engine.py.224: train: epoch 57 | batch 970/2304 [I 2022-10-27 08:24:58] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:22 [I 2022-10-27 08:24:58] ...trap/engines/engine.py.229: process: 0.54854 | load: 0.00244 [I 2022-10-27 08:24:58] ...trap/engines/engine.py.230: loss: 0.11996 [I 2022-10-27 08:24:58] ...optimizers/trijoint.py.151: total_norm: 14.729011 [I 2022-10-27 08:25:03] ...trap/engines/engine.py.224: train: epoch 57 | batch 980/2304 [I 2022-10-27 08:25:03] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:13 [I 2022-10-27 08:25:03] ...trap/engines/engine.py.229: process: 0.54775 | load: 0.00250 [I 2022-10-27 08:25:03] ...trap/engines/engine.py.230: loss: 0.11551 [I 2022-10-27 08:25:03] ...optimizers/trijoint.py.151: total_norm: 9.011323 [I 2022-10-27 08:25:09] ...trap/engines/engine.py.224: train: epoch 57 | batch 990/2304 [I 2022-10-27 08:25:09] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:12:13 [I 2022-10-27 08:25:09] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00293 [I 2022-10-27 08:25:09] ...trap/engines/engine.py.230: loss: 0.21245 [I 2022-10-27 08:25:09] ...optimizers/trijoint.py.151: total_norm: 10.988621 [I 2022-10-27 08:25:14] ...trap/engines/engine.py.224: train: epoch 57 | batch 1000/2304 [I 2022-10-27 08:25:14] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:11:58 [I 2022-10-27 08:25:14] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00271 [I 2022-10-27 08:25:14] ...trap/engines/engine.py.230: loss: 0.22154 [I 2022-10-27 08:25:14] ...optimizers/trijoint.py.151: total_norm: 22.625343 [I 2022-10-27 08:25:20] ...trap/engines/engine.py.224: train: epoch 57 | batch 1010/2304 [I 2022-10-27 08:25:20] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:12:00 [I 2022-10-27 08:25:20] ...trap/engines/engine.py.229: process: 0.55539 | load: 0.00262 [I 2022-10-27 08:25:20] ...trap/engines/engine.py.230: loss: 0.19994 [I 2022-10-27 08:25:20] ...optimizers/trijoint.py.151: total_norm: 10.691178 [I 2022-10-27 08:25:26] ...trap/engines/engine.py.224: train: epoch 57 | batch 1020/2304 [I 2022-10-27 08:25:26] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:54 [I 2022-10-27 08:25:26] ...trap/engines/engine.py.229: process: 0.55024 | load: 0.00256 [I 2022-10-27 08:25:26] ...trap/engines/engine.py.230: loss: 0.09630 [I 2022-10-27 08:25:26] ...optimizers/trijoint.py.151: total_norm: 11.259659 [I 2022-10-27 08:25:31] ...trap/engines/engine.py.224: train: epoch 57 | batch 1030/2304 [I 2022-10-27 08:25:31] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:41 [I 2022-10-27 08:25:31] ...trap/engines/engine.py.229: process: 0.53712 | load: 0.00249 [I 2022-10-27 08:25:31] ...trap/engines/engine.py.230: loss: 0.18577 [I 2022-10-27 08:25:31] ...optimizers/trijoint.py.151: total_norm: 12.379198 [I 2022-10-27 08:25:37] ...trap/engines/engine.py.224: train: epoch 57 | batch 1040/2304 [I 2022-10-27 08:25:37] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:39 [I 2022-10-27 08:25:37] ...trap/engines/engine.py.229: process: 0.54892 | load: 0.00259 [I 2022-10-27 08:25:37] ...trap/engines/engine.py.230: loss: 0.14670 [I 2022-10-27 08:25:37] ...optimizers/trijoint.py.151: total_norm: 23.925156 [I 2022-10-27 08:25:42] ...trap/engines/engine.py.224: train: epoch 57 | batch 1050/2304 [I 2022-10-27 08:25:42] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:38 [I 2022-10-27 08:25:42] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00248 [I 2022-10-27 08:25:42] ...trap/engines/engine.py.230: loss: 0.18045 [I 2022-10-27 08:25:42] ...optimizers/trijoint.py.151: total_norm: 19.675009 [I 2022-10-27 08:25:48] ...trap/engines/engine.py.224: train: epoch 57 | batch 1060/2304 [I 2022-10-27 08:25:48] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:31 [I 2022-10-27 08:25:48] ...trap/engines/engine.py.229: process: 0.54656 | load: 0.00248 [I 2022-10-27 08:25:48] ...trap/engines/engine.py.230: loss: 0.16803 [I 2022-10-27 08:25:48] ...optimizers/trijoint.py.151: total_norm: 13.551556 [I 2022-10-27 08:25:54] ...trap/engines/engine.py.224: train: epoch 57 | batch 1070/2304 [I 2022-10-27 08:25:54] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:50 [I 2022-10-27 08:25:54] ...trap/engines/engine.py.229: process: 0.58286 | load: 0.01167 [I 2022-10-27 08:25:54] ...trap/engines/engine.py.230: loss: 0.11061 [I 2022-10-27 08:25:54] ...optimizers/trijoint.py.151: total_norm: 30.428907 [I 2022-10-27 08:25:59] ...trap/engines/engine.py.224: train: epoch 57 | batch 1080/2304 [I 2022-10-27 08:25:59] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:20 [I 2022-10-27 08:25:59] ...trap/engines/engine.py.229: process: 0.55008 | load: 0.00249 [I 2022-10-27 08:25:59] ...trap/engines/engine.py.230: loss: 0.11890 [I 2022-10-27 08:25:59] ...optimizers/trijoint.py.151: total_norm: 11.875288 [I 2022-10-27 08:26:05] ...trap/engines/engine.py.224: train: epoch 57 | batch 1090/2304 [I 2022-10-27 08:26:05] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:22 [I 2022-10-27 08:26:05] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00251 [I 2022-10-27 08:26:05] ...trap/engines/engine.py.230: loss: 0.16392 [I 2022-10-27 08:26:05] ...optimizers/trijoint.py.151: total_norm: 14.562565 [I 2022-10-27 08:26:10] ...trap/engines/engine.py.224: train: epoch 57 | batch 1100/2304 [I 2022-10-27 08:26:10] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:13 [I 2022-10-27 08:26:10] ...trap/engines/engine.py.229: process: 0.56766 | load: 0.00258 [I 2022-10-27 08:26:10] ...trap/engines/engine.py.230: loss: 0.12958 [I 2022-10-27 08:26:10] ...optimizers/trijoint.py.151: total_norm: 12.877520 [I 2022-10-27 08:26:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 1110/2304 [I 2022-10-27 08:26:16] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:11:04 [I 2022-10-27 08:26:16] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00256 [I 2022-10-27 08:26:16] ...trap/engines/engine.py.230: loss: 0.13773 [I 2022-10-27 08:26:16] ...optimizers/trijoint.py.151: total_norm: 37.985355 [I 2022-10-27 08:26:22] ...trap/engines/engine.py.224: train: epoch 57 | batch 1120/2304 [I 2022-10-27 08:26:22] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:54 [I 2022-10-27 08:26:22] ...trap/engines/engine.py.229: process: 0.54436 | load: 0.00251 [I 2022-10-27 08:26:22] ...trap/engines/engine.py.230: loss: 0.13770 [I 2022-10-27 08:26:22] ...optimizers/trijoint.py.151: total_norm: 17.925993 [I 2022-10-27 08:26:27] ...trap/engines/engine.py.224: train: epoch 57 | batch 1130/2304 [I 2022-10-27 08:26:27] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:52 [I 2022-10-27 08:26:27] ...trap/engines/engine.py.229: process: 0.55375 | load: 0.00243 [I 2022-10-27 08:26:27] ...trap/engines/engine.py.230: loss: 0.13157 [I 2022-10-27 08:26:27] ...optimizers/trijoint.py.151: total_norm: 8.960571 [I 2022-10-27 08:26:33] ...trap/engines/engine.py.224: train: epoch 57 | batch 1140/2304 [I 2022-10-27 08:26:33] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:43 [I 2022-10-27 08:26:33] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00246 [I 2022-10-27 08:26:33] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-27 08:26:33] ...optimizers/trijoint.py.151: total_norm: 7.153107 [I 2022-10-27 08:26:38] ...trap/engines/engine.py.224: train: epoch 57 | batch 1150/2304 [I 2022-10-27 08:26:38] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:42 [I 2022-10-27 08:26:38] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00245 [I 2022-10-27 08:26:38] ...trap/engines/engine.py.230: loss: 0.12470 [I 2022-10-27 08:26:38] ...optimizers/trijoint.py.151: total_norm: 10.249830 [I 2022-10-27 08:26:44] ...trap/engines/engine.py.224: train: epoch 57 | batch 1160/2304 [I 2022-10-27 08:26:44] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:46 [I 2022-10-27 08:26:44] ...trap/engines/engine.py.229: process: 0.56181 | load: 0.00255 [I 2022-10-27 08:26:44] ...trap/engines/engine.py.230: loss: 0.15238 [I 2022-10-27 08:26:44] ...optimizers/trijoint.py.151: total_norm: 25.672295 [I 2022-10-27 08:26:50] ...trap/engines/engine.py.224: train: epoch 57 | batch 1170/2304 [I 2022-10-27 08:26:50] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:33 [I 2022-10-27 08:26:50] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00248 [I 2022-10-27 08:26:50] ...trap/engines/engine.py.230: loss: 0.15170 [I 2022-10-27 08:26:50] ...optimizers/trijoint.py.151: total_norm: 37.128666 [I 2022-10-27 08:26:55] ...trap/engines/engine.py.224: train: epoch 57 | batch 1180/2304 [I 2022-10-27 08:26:55] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:25 [I 2022-10-27 08:26:55] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00249 [I 2022-10-27 08:26:55] ...trap/engines/engine.py.230: loss: 0.15369 [I 2022-10-27 08:26:55] ...optimizers/trijoint.py.151: total_norm: 26.199011 [I 2022-10-27 08:27:01] ...trap/engines/engine.py.224: train: epoch 57 | batch 1190/2304 [I 2022-10-27 08:27:01] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:19 [I 2022-10-27 08:27:01] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00243 [I 2022-10-27 08:27:01] ...trap/engines/engine.py.230: loss: 0.19046 [I 2022-10-27 08:27:01] ...optimizers/trijoint.py.151: total_norm: 18.355072 [I 2022-10-27 08:27:06] ...trap/engines/engine.py.224: train: epoch 57 | batch 1200/2304 [I 2022-10-27 08:27:06] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:12 [I 2022-10-27 08:27:06] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00246 [I 2022-10-27 08:27:06] ...trap/engines/engine.py.230: loss: 0.14997 [I 2022-10-27 08:27:06] ...optimizers/trijoint.py.151: total_norm: 9.931168 [I 2022-10-27 08:27:12] ...trap/engines/engine.py.224: train: epoch 57 | batch 1210/2304 [I 2022-10-27 08:27:12] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:10:12 [I 2022-10-27 08:27:12] ...trap/engines/engine.py.229: process: 0.56579 | load: 0.00249 [I 2022-10-27 08:27:12] ...trap/engines/engine.py.230: loss: 0.19180 [I 2022-10-27 08:27:12] ...optimizers/trijoint.py.151: total_norm: 15.475661 [I 2022-10-27 08:27:17] ...trap/engines/engine.py.224: train: epoch 57 | batch 1220/2304 [I 2022-10-27 08:27:17] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:10:03 [I 2022-10-27 08:27:17] ...trap/engines/engine.py.229: process: 0.56502 | load: 0.00264 [I 2022-10-27 08:27:17] ...trap/engines/engine.py.230: loss: 0.21841 [I 2022-10-27 08:27:17] ...optimizers/trijoint.py.151: total_norm: 21.298086 [I 2022-10-27 08:27:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 1230/2304 [I 2022-10-27 08:27:23] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:10:12 [I 2022-10-27 08:27:23] ...trap/engines/engine.py.229: process: 0.56336 | load: 0.00256 [I 2022-10-27 08:27:23] ...trap/engines/engine.py.230: loss: 0.28974 [I 2022-10-27 08:27:23] ...optimizers/trijoint.py.151: total_norm: 27.851608 [I 2022-10-27 08:27:29] ...trap/engines/engine.py.224: train: epoch 57 | batch 1240/2304 [I 2022-10-27 08:27:29] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:58 [I 2022-10-27 08:27:29] ...trap/engines/engine.py.229: process: 0.56517 | load: 0.00251 [I 2022-10-27 08:27:29] ...trap/engines/engine.py.230: loss: 0.17897 [I 2022-10-27 08:27:29] ...optimizers/trijoint.py.151: total_norm: 17.098106 [I 2022-10-27 08:27:34] ...trap/engines/engine.py.224: train: epoch 57 | batch 1250/2304 [I 2022-10-27 08:27:34] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:51 [I 2022-10-27 08:27:34] ...trap/engines/engine.py.229: process: 0.57109 | load: 0.00244 [I 2022-10-27 08:27:34] ...trap/engines/engine.py.230: loss: 0.13614 [I 2022-10-27 08:27:34] ...optimizers/trijoint.py.151: total_norm: 8.094522 [I 2022-10-27 08:27:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 1260/2304 [I 2022-10-27 08:27:40] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:43 [I 2022-10-27 08:27:40] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00260 [I 2022-10-27 08:27:40] ...trap/engines/engine.py.230: loss: 0.17656 [I 2022-10-27 08:27:40] ...optimizers/trijoint.py.151: total_norm: 29.350920 [I 2022-10-27 08:27:45] ...trap/engines/engine.py.224: train: epoch 57 | batch 1270/2304 [I 2022-10-27 08:27:45] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:34 [I 2022-10-27 08:27:45] ...trap/engines/engine.py.229: process: 0.54171 | load: 0.00243 [I 2022-10-27 08:27:45] ...trap/engines/engine.py.230: loss: 0.14180 [I 2022-10-27 08:27:45] ...optimizers/trijoint.py.151: total_norm: 7.562045 [I 2022-10-27 08:27:51] ...trap/engines/engine.py.224: train: epoch 57 | batch 1280/2304 [I 2022-10-27 08:27:51] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:26 [I 2022-10-27 08:27:51] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00251 [I 2022-10-27 08:27:51] ...trap/engines/engine.py.230: loss: 0.11203 [I 2022-10-27 08:27:51] ...optimizers/trijoint.py.151: total_norm: 10.074268 [I 2022-10-27 08:27:57] ...trap/engines/engine.py.224: train: epoch 57 | batch 1290/2304 [I 2022-10-27 08:27:57] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:23 [I 2022-10-27 08:27:57] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00251 [I 2022-10-27 08:27:57] ...trap/engines/engine.py.230: loss: 0.20055 [I 2022-10-27 08:27:57] ...optimizers/trijoint.py.151: total_norm: 22.499454 [I 2022-10-27 08:28:02] ...trap/engines/engine.py.224: train: epoch 57 | batch 1300/2304 [I 2022-10-27 08:28:02] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:09:13 [I 2022-10-27 08:28:02] ...trap/engines/engine.py.229: process: 0.54926 | load: 0.00248 [I 2022-10-27 08:28:02] ...trap/engines/engine.py.230: loss: 0.14751 [I 2022-10-27 08:28:02] ...optimizers/trijoint.py.151: total_norm: 18.359713 [I 2022-10-27 08:28:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 1310/2304 [I 2022-10-27 08:28:08] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:09:07 [I 2022-10-27 08:28:08] ...trap/engines/engine.py.229: process: 0.54808 | load: 0.00241 [I 2022-10-27 08:28:08] ...trap/engines/engine.py.230: loss: 0.16075 [I 2022-10-27 08:28:08] ...optimizers/trijoint.py.151: total_norm: 13.760770 [I 2022-10-27 08:28:13] ...trap/engines/engine.py.224: train: epoch 57 | batch 1320/2304 [I 2022-10-27 08:28:13] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:05 [I 2022-10-27 08:28:13] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00254 [I 2022-10-27 08:28:13] ...trap/engines/engine.py.230: loss: 0.19746 [I 2022-10-27 08:28:13] ...optimizers/trijoint.py.151: total_norm: 10.098108 [I 2022-10-27 08:28:19] ...trap/engines/engine.py.224: train: epoch 57 | batch 1330/2304 [I 2022-10-27 08:28:19] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:08:59 [I 2022-10-27 08:28:19] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00259 [I 2022-10-27 08:28:19] ...trap/engines/engine.py.230: loss: 0.22826 [I 2022-10-27 08:28:19] ...optimizers/trijoint.py.151: total_norm: 16.284679 [I 2022-10-27 08:28:24] ...trap/engines/engine.py.224: train: epoch 57 | batch 1340/2304 [I 2022-10-27 08:28:24] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:56 [I 2022-10-27 08:28:24] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00260 [I 2022-10-27 08:28:24] ...trap/engines/engine.py.230: loss: 0.16389 [I 2022-10-27 08:28:24] ...optimizers/trijoint.py.151: total_norm: 15.372114 [I 2022-10-27 08:28:30] ...trap/engines/engine.py.224: train: epoch 57 | batch 1350/2304 [I 2022-10-27 08:28:30] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:56 [I 2022-10-27 08:28:30] ...trap/engines/engine.py.229: process: 0.54430 | load: 0.00257 [I 2022-10-27 08:28:30] ...trap/engines/engine.py.230: loss: 0.15221 [I 2022-10-27 08:28:30] ...optimizers/trijoint.py.151: total_norm: 13.464219 [I 2022-10-27 08:28:36] ...trap/engines/engine.py.224: train: epoch 57 | batch 1360/2304 [I 2022-10-27 08:28:36] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:48 [I 2022-10-27 08:28:36] ...trap/engines/engine.py.229: process: 0.56183 | load: 0.00250 [I 2022-10-27 08:28:36] ...trap/engines/engine.py.230: loss: 0.12154 [I 2022-10-27 08:28:36] ...optimizers/trijoint.py.151: total_norm: 19.897440 [I 2022-10-27 08:28:41] ...trap/engines/engine.py.224: train: epoch 57 | batch 1370/2304 [I 2022-10-27 08:28:41] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:42 [I 2022-10-27 08:28:41] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00251 [I 2022-10-27 08:28:41] ...trap/engines/engine.py.230: loss: 0.11870 [I 2022-10-27 08:28:41] ...optimizers/trijoint.py.151: total_norm: 7.297731 [I 2022-10-27 08:28:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 1380/2304 [I 2022-10-27 08:28:47] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:33 [I 2022-10-27 08:28:47] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00250 [I 2022-10-27 08:28:47] ...trap/engines/engine.py.230: loss: 0.16014 [I 2022-10-27 08:28:47] ...optimizers/trijoint.py.151: total_norm: 11.980431 [I 2022-10-27 08:28:52] ...trap/engines/engine.py.224: train: epoch 57 | batch 1390/2304 [I 2022-10-27 08:28:52] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:27 [I 2022-10-27 08:28:52] ...trap/engines/engine.py.229: process: 0.55720 | load: 0.00248 [I 2022-10-27 08:28:52] ...trap/engines/engine.py.230: loss: 0.18117 [I 2022-10-27 08:28:52] ...optimizers/trijoint.py.151: total_norm: 19.251640 [I 2022-10-27 08:28:58] ...trap/engines/engine.py.224: train: epoch 57 | batch 1400/2304 [I 2022-10-27 08:28:58] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:21 [I 2022-10-27 08:28:58] ...trap/engines/engine.py.229: process: 0.56427 | load: 0.00248 [I 2022-10-27 08:28:58] ...trap/engines/engine.py.230: loss: 0.21530 [I 2022-10-27 08:28:58] ...optimizers/trijoint.py.151: total_norm: 29.958303 [I 2022-10-27 08:29:04] ...trap/engines/engine.py.224: train: epoch 57 | batch 1410/2304 [I 2022-10-27 08:29:04] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:15 [I 2022-10-27 08:29:04] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00247 [I 2022-10-27 08:29:04] ...trap/engines/engine.py.230: loss: 0.25177 [I 2022-10-27 08:29:04] ...optimizers/trijoint.py.151: total_norm: 29.850815 [I 2022-10-27 08:29:09] ...trap/engines/engine.py.224: train: epoch 57 | batch 1420/2304 [I 2022-10-27 08:29:09] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:08:12 [I 2022-10-27 08:29:09] ...trap/engines/engine.py.229: process: 0.57097 | load: 0.00243 [I 2022-10-27 08:29:09] ...trap/engines/engine.py.230: loss: 0.14446 [I 2022-10-27 08:29:09] ...optimizers/trijoint.py.151: total_norm: 24.004810 [I 2022-10-27 08:29:15] ...trap/engines/engine.py.224: train: epoch 57 | batch 1430/2304 [I 2022-10-27 08:29:15] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:08:01 [I 2022-10-27 08:29:15] ...trap/engines/engine.py.229: process: 0.53778 | load: 0.00243 [I 2022-10-27 08:29:15] ...trap/engines/engine.py.230: loss: 0.12140 [I 2022-10-27 08:29:15] ...optimizers/trijoint.py.151: total_norm: 6.730029 [I 2022-10-27 08:29:20] ...trap/engines/engine.py.224: train: epoch 57 | batch 1440/2304 [I 2022-10-27 08:29:20] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:07:59 [I 2022-10-27 08:29:20] ...trap/engines/engine.py.229: process: 0.54503 | load: 0.00244 [I 2022-10-27 08:29:20] ...trap/engines/engine.py.230: loss: 0.13725 [I 2022-10-27 08:29:20] ...optimizers/trijoint.py.151: total_norm: 20.663153 [I 2022-10-27 08:29:26] ...trap/engines/engine.py.224: train: epoch 57 | batch 1450/2304 [I 2022-10-27 08:29:26] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:55 [I 2022-10-27 08:29:26] ...trap/engines/engine.py.229: process: 0.55237 | load: 0.00244 [I 2022-10-27 08:29:26] ...trap/engines/engine.py.230: loss: 0.17403 [I 2022-10-27 08:29:26] ...optimizers/trijoint.py.151: total_norm: 23.095724 [I 2022-10-27 08:29:32] ...trap/engines/engine.py.224: train: epoch 57 | batch 1460/2304 [I 2022-10-27 08:29:32] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:08:03 [I 2022-10-27 08:29:32] ...trap/engines/engine.py.229: process: 0.63321 | load: 0.00245 [I 2022-10-27 08:29:32] ...trap/engines/engine.py.230: loss: 0.13505 [I 2022-10-27 08:29:32] ...optimizers/trijoint.py.151: total_norm: 18.200222 [I 2022-10-27 08:29:37] ...trap/engines/engine.py.224: train: epoch 57 | batch 1470/2304 [I 2022-10-27 08:29:37] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:48 [I 2022-10-27 08:29:37] ...trap/engines/engine.py.229: process: 0.55200 | load: 0.00240 [I 2022-10-27 08:29:37] ...trap/engines/engine.py.230: loss: 0.16916 [I 2022-10-27 08:29:37] ...optimizers/trijoint.py.151: total_norm: 19.367315 [I 2022-10-27 08:29:43] ...trap/engines/engine.py.224: train: epoch 57 | batch 1480/2304 [I 2022-10-27 08:29:43] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:34 [I 2022-10-27 08:29:43] ...trap/engines/engine.py.229: process: 0.54643 | load: 0.00245 [I 2022-10-27 08:29:43] ...trap/engines/engine.py.230: loss: 0.12940 [I 2022-10-27 08:29:43] ...optimizers/trijoint.py.151: total_norm: 12.886059 [I 2022-10-27 08:29:48] ...trap/engines/engine.py.224: train: epoch 57 | batch 1490/2304 [I 2022-10-27 08:29:48] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:35 [I 2022-10-27 08:29:48] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00245 [I 2022-10-27 08:29:48] ...trap/engines/engine.py.230: loss: 0.16505 [I 2022-10-27 08:29:48] ...optimizers/trijoint.py.151: total_norm: 27.177952 [I 2022-10-27 08:29:54] ...trap/engines/engine.py.224: train: epoch 57 | batch 1500/2304 [I 2022-10-27 08:29:54] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:25 [I 2022-10-27 08:29:54] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00253 [I 2022-10-27 08:29:54] ...trap/engines/engine.py.230: loss: 0.17333 [I 2022-10-27 08:29:54] ...optimizers/trijoint.py.151: total_norm: 15.596912 [I 2022-10-27 08:29:59] ...trap/engines/engine.py.224: train: epoch 57 | batch 1510/2304 [I 2022-10-27 08:29:59] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:21 [I 2022-10-27 08:29:59] ...trap/engines/engine.py.229: process: 0.54533 | load: 0.00251 [I 2022-10-27 08:29:59] ...trap/engines/engine.py.230: loss: 0.22510 [I 2022-10-27 08:29:59] ...optimizers/trijoint.py.151: total_norm: 12.592340 [I 2022-10-27 08:30:05] ...trap/engines/engine.py.224: train: epoch 57 | batch 1520/2304 [I 2022-10-27 08:30:05] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:07:15 [I 2022-10-27 08:30:05] ...trap/engines/engine.py.229: process: 0.54100 | load: 0.00249 [I 2022-10-27 08:30:05] ...trap/engines/engine.py.230: loss: 0.10698 [I 2022-10-27 08:30:05] ...optimizers/trijoint.py.151: total_norm: 15.725015 [I 2022-10-27 08:30:11] ...trap/engines/engine.py.224: train: epoch 57 | batch 1530/2304 [I 2022-10-27 08:30:11] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:07:10 [I 2022-10-27 08:30:11] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00250 [I 2022-10-27 08:30:11] ...trap/engines/engine.py.230: loss: 0.13563 [I 2022-10-27 08:30:11] ...optimizers/trijoint.py.151: total_norm: 18.474392 [I 2022-10-27 08:30:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 1540/2304 [I 2022-10-27 08:30:16] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:07:03 [I 2022-10-27 08:30:16] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00243 [I 2022-10-27 08:30:16] ...trap/engines/engine.py.230: loss: 0.10306 [I 2022-10-27 08:30:16] ...optimizers/trijoint.py.151: total_norm: 9.528707 [I 2022-10-27 08:30:22] ...trap/engines/engine.py.224: train: epoch 57 | batch 1550/2304 [I 2022-10-27 08:30:22] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:06:57 [I 2022-10-27 08:30:22] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00242 [I 2022-10-27 08:30:22] ...trap/engines/engine.py.230: loss: 0.11384 [I 2022-10-27 08:30:22] ...optimizers/trijoint.py.151: total_norm: 9.461123 [I 2022-10-27 08:30:27] ...trap/engines/engine.py.224: train: epoch 57 | batch 1560/2304 [I 2022-10-27 08:30:27] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:54 [I 2022-10-27 08:30:27] ...trap/engines/engine.py.229: process: 0.56002 | load: 0.00249 [I 2022-10-27 08:30:27] ...trap/engines/engine.py.230: loss: 0.14624 [I 2022-10-27 08:30:27] ...optimizers/trijoint.py.151: total_norm: 8.951690 [I 2022-10-27 08:30:33] ...trap/engines/engine.py.224: train: epoch 57 | batch 1570/2304 [I 2022-10-27 08:30:33] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:46 [I 2022-10-27 08:30:33] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00252 [I 2022-10-27 08:30:33] ...trap/engines/engine.py.230: loss: 0.18694 [I 2022-10-27 08:30:33] ...optimizers/trijoint.py.151: total_norm: 33.626106 [I 2022-10-27 08:30:39] ...trap/engines/engine.py.224: train: epoch 57 | batch 1580/2304 [I 2022-10-27 08:30:39] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:07:05 [I 2022-10-27 08:30:39] ...trap/engines/engine.py.229: process: 0.71988 | load: 0.00253 [I 2022-10-27 08:30:39] ...trap/engines/engine.py.230: loss: 0.18136 [I 2022-10-27 08:30:39] ...optimizers/trijoint.py.151: total_norm: 10.247247 [I 2022-10-27 08:30:44] ...trap/engines/engine.py.224: train: epoch 57 | batch 1590/2304 [I 2022-10-27 08:30:44] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:37 [I 2022-10-27 08:30:44] ...trap/engines/engine.py.229: process: 0.55343 | load: 0.00243 [I 2022-10-27 08:30:44] ...trap/engines/engine.py.230: loss: 0.19587 [I 2022-10-27 08:30:44] ...optimizers/trijoint.py.151: total_norm: 25.714506 [I 2022-10-27 08:30:50] ...trap/engines/engine.py.224: train: epoch 57 | batch 1600/2304 [I 2022-10-27 08:30:50] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:33 [I 2022-10-27 08:30:50] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00250 [I 2022-10-27 08:30:50] ...trap/engines/engine.py.230: loss: 0.12249 [I 2022-10-27 08:30:50] ...optimizers/trijoint.py.151: total_norm: 12.726686 [I 2022-10-27 08:30:55] ...trap/engines/engine.py.224: train: epoch 57 | batch 1610/2304 [I 2022-10-27 08:30:55] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:29 [I 2022-10-27 08:30:55] ...trap/engines/engine.py.229: process: 0.57379 | load: 0.00248 [I 2022-10-27 08:30:55] ...trap/engines/engine.py.230: loss: 0.15047 [I 2022-10-27 08:30:55] ...optimizers/trijoint.py.151: total_norm: 18.658989 [I 2022-10-27 08:31:01] ...trap/engines/engine.py.224: train: epoch 57 | batch 1620/2304 [I 2022-10-27 08:31:01] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:23 [I 2022-10-27 08:31:01] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00246 [I 2022-10-27 08:31:01] ...trap/engines/engine.py.230: loss: 0.10556 [I 2022-10-27 08:31:01] ...optimizers/trijoint.py.151: total_norm: 8.150968 [I 2022-10-27 08:31:07] ...trap/engines/engine.py.224: train: epoch 57 | batch 1630/2304 [I 2022-10-27 08:31:07] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:21 [I 2022-10-27 08:31:07] ...trap/engines/engine.py.229: process: 0.56696 | load: 0.00247 [I 2022-10-27 08:31:07] ...trap/engines/engine.py.230: loss: 0.18517 [I 2022-10-27 08:31:07] ...optimizers/trijoint.py.151: total_norm: 13.847141 [I 2022-10-27 08:31:12] ...trap/engines/engine.py.224: train: epoch 57 | batch 1640/2304 [I 2022-10-27 08:31:12] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:06:11 [I 2022-10-27 08:31:12] ...trap/engines/engine.py.229: process: 0.56447 | load: 0.00258 [I 2022-10-27 08:31:12] ...trap/engines/engine.py.230: loss: 0.14416 [I 2022-10-27 08:31:12] ...optimizers/trijoint.py.151: total_norm: 9.577068 [I 2022-10-27 08:31:18] ...trap/engines/engine.py.224: train: epoch 57 | batch 1650/2304 [I 2022-10-27 08:31:18] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:06:04 [I 2022-10-27 08:31:18] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00253 [I 2022-10-27 08:31:18] ...trap/engines/engine.py.230: loss: 0.12741 [I 2022-10-27 08:31:18] ...optimizers/trijoint.py.151: total_norm: 16.610079 [I 2022-10-27 08:31:23] ...trap/engines/engine.py.224: train: epoch 57 | batch 1660/2304 [I 2022-10-27 08:31:23] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:05:59 [I 2022-10-27 08:31:23] ...trap/engines/engine.py.229: process: 0.56669 | load: 0.00244 [I 2022-10-27 08:31:23] ...trap/engines/engine.py.230: loss: 0.13381 [I 2022-10-27 08:31:23] ...optimizers/trijoint.py.151: total_norm: 10.206795 [I 2022-10-27 08:31:29] ...trap/engines/engine.py.224: train: epoch 57 | batch 1670/2304 [I 2022-10-27 08:31:29] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:52 [I 2022-10-27 08:31:29] ...trap/engines/engine.py.229: process: 0.54929 | load: 0.00251 [I 2022-10-27 08:31:29] ...trap/engines/engine.py.230: loss: 0.20680 [I 2022-10-27 08:31:29] ...optimizers/trijoint.py.151: total_norm: 26.808758 [I 2022-10-27 08:31:34] ...trap/engines/engine.py.224: train: epoch 57 | batch 1680/2304 [I 2022-10-27 08:31:34] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:47 [I 2022-10-27 08:31:34] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00250 [I 2022-10-27 08:31:34] ...trap/engines/engine.py.230: loss: 0.18135 [I 2022-10-27 08:31:34] ...optimizers/trijoint.py.151: total_norm: 25.494404 [I 2022-10-27 08:31:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 1690/2304 [I 2022-10-27 08:31:40] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:42 [I 2022-10-27 08:31:40] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00247 [I 2022-10-27 08:31:40] ...trap/engines/engine.py.230: loss: 0.10580 [I 2022-10-27 08:31:40] ...optimizers/trijoint.py.151: total_norm: 16.304663 [I 2022-10-27 08:31:46] ...trap/engines/engine.py.224: train: epoch 57 | batch 1700/2304 [I 2022-10-27 08:31:46] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:36 [I 2022-10-27 08:31:46] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00250 [I 2022-10-27 08:31:46] ...trap/engines/engine.py.230: loss: 0.17279 [I 2022-10-27 08:31:46] ...optimizers/trijoint.py.151: total_norm: 14.999853 [I 2022-10-27 08:31:51] ...trap/engines/engine.py.224: train: epoch 57 | batch 1710/2304 [I 2022-10-27 08:31:51] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:32 [I 2022-10-27 08:31:51] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00249 [I 2022-10-27 08:31:51] ...trap/engines/engine.py.230: loss: 0.18180 [I 2022-10-27 08:31:51] ...optimizers/trijoint.py.151: total_norm: 21.026373 [I 2022-10-27 08:31:57] ...trap/engines/engine.py.224: train: epoch 57 | batch 1720/2304 [I 2022-10-27 08:31:57] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:24 [I 2022-10-27 08:31:57] ...trap/engines/engine.py.229: process: 0.56243 | load: 0.00248 [I 2022-10-27 08:31:57] ...trap/engines/engine.py.230: loss: 0.20095 [I 2022-10-27 08:31:57] ...optimizers/trijoint.py.151: total_norm: 28.479200 [I 2022-10-27 08:32:02] ...trap/engines/engine.py.224: train: epoch 57 | batch 1730/2304 [I 2022-10-27 08:32:02] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:18 [I 2022-10-27 08:32:02] ...trap/engines/engine.py.229: process: 0.54539 | load: 0.00243 [I 2022-10-27 08:32:02] ...trap/engines/engine.py.230: loss: 0.20591 [I 2022-10-27 08:32:02] ...optimizers/trijoint.py.151: total_norm: 11.244102 [I 2022-10-27 08:32:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 1740/2304 [I 2022-10-27 08:32:08] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:05:14 [I 2022-10-27 08:32:08] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00247 [I 2022-10-27 08:32:08] ...trap/engines/engine.py.230: loss: 0.12200 [I 2022-10-27 08:32:08] ...optimizers/trijoint.py.151: total_norm: 8.143976 [I 2022-10-27 08:32:14] ...trap/engines/engine.py.224: train: epoch 57 | batch 1750/2304 [I 2022-10-27 08:32:14] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:05:06 [I 2022-10-27 08:32:14] ...trap/engines/engine.py.229: process: 0.54123 | load: 0.00255 [I 2022-10-27 08:32:14] ...trap/engines/engine.py.230: loss: 0.10970 [I 2022-10-27 08:32:14] ...optimizers/trijoint.py.151: total_norm: 7.587994 [I 2022-10-27 08:32:19] ...trap/engines/engine.py.224: train: epoch 57 | batch 1760/2304 [I 2022-10-27 08:32:19] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:05:04 [I 2022-10-27 08:32:19] ...trap/engines/engine.py.229: process: 0.56187 | load: 0.00247 [I 2022-10-27 08:32:19] ...trap/engines/engine.py.230: loss: 0.14608 [I 2022-10-27 08:32:19] ...optimizers/trijoint.py.151: total_norm: 18.525604 [I 2022-10-27 08:32:25] ...trap/engines/engine.py.224: train: epoch 57 | batch 1770/2304 [I 2022-10-27 08:32:25] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:05:00 [I 2022-10-27 08:32:25] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00252 [I 2022-10-27 08:32:25] ...trap/engines/engine.py.230: loss: 0.11204 [I 2022-10-27 08:32:25] ...optimizers/trijoint.py.151: total_norm: 10.369991 [I 2022-10-27 08:32:30] ...trap/engines/engine.py.224: train: epoch 57 | batch 1780/2304 [I 2022-10-27 08:32:30] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:52 [I 2022-10-27 08:32:30] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00249 [I 2022-10-27 08:32:30] ...trap/engines/engine.py.230: loss: 0.16838 [I 2022-10-27 08:32:30] ...optimizers/trijoint.py.151: total_norm: 16.477390 [I 2022-10-27 08:32:36] ...trap/engines/engine.py.224: train: epoch 57 | batch 1790/2304 [I 2022-10-27 08:32:36] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:48 [I 2022-10-27 08:32:36] ...trap/engines/engine.py.229: process: 0.56841 | load: 0.00259 [I 2022-10-27 08:32:36] ...trap/engines/engine.py.230: loss: 0.20630 [I 2022-10-27 08:32:36] ...optimizers/trijoint.py.151: total_norm: 9.250007 [I 2022-10-27 08:32:42] ...trap/engines/engine.py.224: train: epoch 57 | batch 1800/2304 [I 2022-10-27 08:32:42] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:40 [I 2022-10-27 08:32:42] ...trap/engines/engine.py.229: process: 0.54822 | load: 0.00255 [I 2022-10-27 08:32:42] ...trap/engines/engine.py.230: loss: 0.13431 [I 2022-10-27 08:32:42] ...optimizers/trijoint.py.151: total_norm: 29.773556 [I 2022-10-27 08:32:47] ...trap/engines/engine.py.224: train: epoch 57 | batch 1810/2304 [I 2022-10-27 08:32:47] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:36 [I 2022-10-27 08:32:47] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00255 [I 2022-10-27 08:32:47] ...trap/engines/engine.py.230: loss: 0.12411 [I 2022-10-27 08:32:47] ...optimizers/trijoint.py.151: total_norm: 10.011732 [I 2022-10-27 08:32:53] ...trap/engines/engine.py.224: train: epoch 57 | batch 1820/2304 [I 2022-10-27 08:32:53] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:30 [I 2022-10-27 08:32:53] ...trap/engines/engine.py.229: process: 0.56193 | load: 0.00264 [I 2022-10-27 08:32:53] ...trap/engines/engine.py.230: loss: 0.16246 [I 2022-10-27 08:32:53] ...optimizers/trijoint.py.151: total_norm: 15.444452 [I 2022-10-27 08:32:58] ...trap/engines/engine.py.224: train: epoch 57 | batch 1830/2304 [I 2022-10-27 08:32:58] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:24 [I 2022-10-27 08:32:58] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00257 [I 2022-10-27 08:32:58] ...trap/engines/engine.py.230: loss: 0.21561 [I 2022-10-27 08:32:58] ...optimizers/trijoint.py.151: total_norm: 37.322990 [I 2022-10-27 08:33:04] ...trap/engines/engine.py.224: train: epoch 57 | batch 1840/2304 [I 2022-10-27 08:33:04] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:18 [I 2022-10-27 08:33:04] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00252 [I 2022-10-27 08:33:04] ...trap/engines/engine.py.230: loss: 0.13335 [I 2022-10-27 08:33:04] ...optimizers/trijoint.py.151: total_norm: 10.545998 [I 2022-10-27 08:33:10] ...trap/engines/engine.py.224: train: epoch 57 | batch 1850/2304 [I 2022-10-27 08:33:10] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:04:15 [I 2022-10-27 08:33:10] ...trap/engines/engine.py.229: process: 0.58465 | load: 0.00263 [I 2022-10-27 08:33:10] ...trap/engines/engine.py.230: loss: 0.22048 [I 2022-10-27 08:33:10] ...optimizers/trijoint.py.151: total_norm: 24.213125 [I 2022-10-27 08:33:15] ...trap/engines/engine.py.224: train: epoch 57 | batch 1860/2304 [I 2022-10-27 08:33:15] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:04:08 [I 2022-10-27 08:33:15] ...trap/engines/engine.py.229: process: 0.56797 | load: 0.00253 [I 2022-10-27 08:33:15] ...trap/engines/engine.py.230: loss: 0.10362 [I 2022-10-27 08:33:15] ...optimizers/trijoint.py.151: total_norm: 12.716147 [I 2022-10-27 08:33:21] ...trap/engines/engine.py.224: train: epoch 57 | batch 1870/2304 [I 2022-10-27 08:33:21] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:04:02 [I 2022-10-27 08:33:21] ...trap/engines/engine.py.229: process: 0.54697 | load: 0.00248 [I 2022-10-27 08:33:21] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-27 08:33:21] ...optimizers/trijoint.py.151: total_norm: 9.596357 [I 2022-10-27 08:33:26] ...trap/engines/engine.py.224: train: epoch 57 | batch 1880/2304 [I 2022-10-27 08:33:26] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:56 [I 2022-10-27 08:33:26] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00243 [I 2022-10-27 08:33:26] ...trap/engines/engine.py.230: loss: 0.16139 [I 2022-10-27 08:33:26] ...optimizers/trijoint.py.151: total_norm: 18.446224 [I 2022-10-27 08:33:32] ...trap/engines/engine.py.224: train: epoch 57 | batch 1890/2304 [I 2022-10-27 08:33:32] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:48 [I 2022-10-27 08:33:32] ...trap/engines/engine.py.229: process: 0.54373 | load: 0.00242 [I 2022-10-27 08:33:32] ...trap/engines/engine.py.230: loss: 0.18133 [I 2022-10-27 08:33:32] ...optimizers/trijoint.py.151: total_norm: 24.428621 [I 2022-10-27 08:33:37] ...trap/engines/engine.py.224: train: epoch 57 | batch 1900/2304 [I 2022-10-27 08:33:37] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:46 [I 2022-10-27 08:33:37] ...trap/engines/engine.py.229: process: 0.56320 | load: 0.00245 [I 2022-10-27 08:33:37] ...trap/engines/engine.py.230: loss: 0.22042 [I 2022-10-27 08:33:37] ...optimizers/trijoint.py.151: total_norm: 39.483681 [I 2022-10-27 08:33:43] ...trap/engines/engine.py.224: train: epoch 57 | batch 1910/2304 [I 2022-10-27 08:33:43] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:40 [I 2022-10-27 08:33:43] ...trap/engines/engine.py.229: process: 0.55063 | load: 0.00248 [I 2022-10-27 08:33:43] ...trap/engines/engine.py.230: loss: 0.16127 [I 2022-10-27 08:33:43] ...optimizers/trijoint.py.151: total_norm: 17.526499 [I 2022-10-27 08:33:49] ...trap/engines/engine.py.224: train: epoch 57 | batch 1920/2304 [I 2022-10-27 08:33:49] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:35 [I 2022-10-27 08:33:49] ...trap/engines/engine.py.229: process: 0.56787 | load: 0.00249 [I 2022-10-27 08:33:49] ...trap/engines/engine.py.230: loss: 0.11152 [I 2022-10-27 08:33:49] ...optimizers/trijoint.py.151: total_norm: 10.037522 [I 2022-10-27 08:33:54] ...trap/engines/engine.py.224: train: epoch 57 | batch 1930/2304 [I 2022-10-27 08:33:54] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:29 [I 2022-10-27 08:33:54] ...trap/engines/engine.py.229: process: 0.56117 | load: 0.00246 [I 2022-10-27 08:33:54] ...trap/engines/engine.py.230: loss: 0.21466 [I 2022-10-27 08:33:54] ...optimizers/trijoint.py.151: total_norm: 28.725414 [I 2022-10-27 08:34:00] ...trap/engines/engine.py.224: train: epoch 57 | batch 1940/2304 [I 2022-10-27 08:34:00] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:22 [I 2022-10-27 08:34:00] ...trap/engines/engine.py.229: process: 0.54909 | load: 0.00243 [I 2022-10-27 08:34:00] ...trap/engines/engine.py.230: loss: 0.13638 [I 2022-10-27 08:34:00] ...optimizers/trijoint.py.151: total_norm: 24.125904 [I 2022-10-27 08:34:05] ...trap/engines/engine.py.224: train: epoch 57 | batch 1950/2304 [I 2022-10-27 08:34:05] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:17 [I 2022-10-27 08:34:05] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00250 [I 2022-10-27 08:34:05] ...trap/engines/engine.py.230: loss: 0.15473 [I 2022-10-27 08:34:05] ...optimizers/trijoint.py.151: total_norm: 16.770412 [I 2022-10-27 08:34:11] ...trap/engines/engine.py.224: train: epoch 57 | batch 1960/2304 [I 2022-10-27 08:34:11] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:03:11 [I 2022-10-27 08:34:11] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00249 [I 2022-10-27 08:34:11] ...trap/engines/engine.py.230: loss: 0.17253 [I 2022-10-27 08:34:11] ...optimizers/trijoint.py.151: total_norm: 15.851601 [I 2022-10-27 08:34:17] ...trap/engines/engine.py.224: train: epoch 57 | batch 1970/2304 [I 2022-10-27 08:34:17] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:03:06 [I 2022-10-27 08:34:17] ...trap/engines/engine.py.229: process: 0.56552 | load: 0.00243 [I 2022-10-27 08:34:17] ...trap/engines/engine.py.230: loss: 0.17314 [I 2022-10-27 08:34:17] ...optimizers/trijoint.py.151: total_norm: 22.447231 [I 2022-10-27 08:34:22] ...trap/engines/engine.py.224: train: epoch 57 | batch 1980/2304 [I 2022-10-27 08:34:22] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:03:01 [I 2022-10-27 08:34:22] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00245 [I 2022-10-27 08:34:22] ...trap/engines/engine.py.230: loss: 0.15818 [I 2022-10-27 08:34:22] ...optimizers/trijoint.py.151: total_norm: 9.968285 [I 2022-10-27 08:34:28] ...trap/engines/engine.py.224: train: epoch 57 | batch 1990/2304 [I 2022-10-27 08:34:28] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:55 [I 2022-10-27 08:34:28] ...trap/engines/engine.py.229: process: 0.55614 | load: 0.00252 [I 2022-10-27 08:34:28] ...trap/engines/engine.py.230: loss: 0.13285 [I 2022-10-27 08:34:28] ...optimizers/trijoint.py.151: total_norm: 11.176089 [I 2022-10-27 08:34:33] ...trap/engines/engine.py.224: train: epoch 57 | batch 2000/2304 [I 2022-10-27 08:34:33] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:48 [I 2022-10-27 08:34:33] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00253 [I 2022-10-27 08:34:33] ...trap/engines/engine.py.230: loss: 0.19060 [I 2022-10-27 08:34:33] ...optimizers/trijoint.py.151: total_norm: 16.657339 [I 2022-10-27 08:34:39] ...trap/engines/engine.py.224: train: epoch 57 | batch 2010/2304 [I 2022-10-27 08:34:39] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:42 [I 2022-10-27 08:34:39] ...trap/engines/engine.py.229: process: 0.55359 | load: 0.00251 [I 2022-10-27 08:34:39] ...trap/engines/engine.py.230: loss: 0.14531 [I 2022-10-27 08:34:39] ...optimizers/trijoint.py.151: total_norm: 74.400215 [I 2022-10-27 08:34:44] ...trap/engines/engine.py.224: train: epoch 57 | batch 2020/2304 [I 2022-10-27 08:34:44] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:38 [I 2022-10-27 08:34:44] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00249 [I 2022-10-27 08:34:44] ...trap/engines/engine.py.230: loss: 0.17201 [I 2022-10-27 08:34:44] ...optimizers/trijoint.py.151: total_norm: 14.863240 [I 2022-10-27 08:34:50] ...trap/engines/engine.py.224: train: epoch 57 | batch 2030/2304 [I 2022-10-27 08:34:50] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:33 [I 2022-10-27 08:34:50] ...trap/engines/engine.py.229: process: 0.55829 | load: 0.00249 [I 2022-10-27 08:34:50] ...trap/engines/engine.py.230: loss: 0.18256 [I 2022-10-27 08:34:50] ...optimizers/trijoint.py.151: total_norm: 39.737442 [I 2022-10-27 08:34:56] ...trap/engines/engine.py.224: train: epoch 57 | batch 2040/2304 [I 2022-10-27 08:34:56] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:29 [I 2022-10-27 08:34:56] ...trap/engines/engine.py.229: process: 0.58204 | load: 0.00262 [I 2022-10-27 08:34:56] ...trap/engines/engine.py.230: loss: 0.14023 [I 2022-10-27 08:34:56] ...optimizers/trijoint.py.151: total_norm: 19.923676 [I 2022-10-27 08:35:01] ...trap/engines/engine.py.224: train: epoch 57 | batch 2050/2304 [I 2022-10-27 08:35:01] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:22 [I 2022-10-27 08:35:01] ...trap/engines/engine.py.229: process: 0.56600 | load: 0.00261 [I 2022-10-27 08:35:01] ...trap/engines/engine.py.230: loss: 0.16831 [I 2022-10-27 08:35:01] ...optimizers/trijoint.py.151: total_norm: 31.755812 [I 2022-10-27 08:35:07] ...trap/engines/engine.py.224: train: epoch 57 | batch 2060/2304 [I 2022-10-27 08:35:07] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:16 [I 2022-10-27 08:35:07] ...trap/engines/engine.py.229: process: 0.55803 | load: 0.00275 [I 2022-10-27 08:35:07] ...trap/engines/engine.py.230: loss: 0.16092 [I 2022-10-27 08:35:07] ...optimizers/trijoint.py.151: total_norm: 20.055998 [I 2022-10-27 08:35:13] ...trap/engines/engine.py.224: train: epoch 57 | batch 2070/2304 [I 2022-10-27 08:35:13] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:02:10 [I 2022-10-27 08:35:13] ...trap/engines/engine.py.229: process: 0.54458 | load: 0.00249 [I 2022-10-27 08:35:13] ...trap/engines/engine.py.230: loss: 0.15091 [I 2022-10-27 08:35:13] ...optimizers/trijoint.py.151: total_norm: 14.004340 [I 2022-10-27 08:35:18] ...trap/engines/engine.py.224: train: epoch 57 | batch 2080/2304 [I 2022-10-27 08:35:18] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:02:04 [I 2022-10-27 08:35:18] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00244 [I 2022-10-27 08:35:18] ...trap/engines/engine.py.230: loss: 0.12216 [I 2022-10-27 08:35:18] ...optimizers/trijoint.py.151: total_norm: 11.399302 [I 2022-10-27 08:35:24] ...trap/engines/engine.py.224: train: epoch 57 | batch 2090/2304 [I 2022-10-27 08:35:24] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:57 [I 2022-10-27 08:35:24] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00245 [I 2022-10-27 08:35:24] ...trap/engines/engine.py.230: loss: 0.11797 [I 2022-10-27 08:35:24] ...optimizers/trijoint.py.151: total_norm: 12.779874 [I 2022-10-27 08:35:29] ...trap/engines/engine.py.224: train: epoch 57 | batch 2100/2304 [I 2022-10-27 08:35:29] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:53 [I 2022-10-27 08:35:29] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00248 [I 2022-10-27 08:35:29] ...trap/engines/engine.py.230: loss: 0.26677 [I 2022-10-27 08:35:29] ...optimizers/trijoint.py.151: total_norm: 22.279682 [I 2022-10-27 08:35:35] ...trap/engines/engine.py.224: train: epoch 57 | batch 2110/2304 [I 2022-10-27 08:35:35] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:48 [I 2022-10-27 08:35:35] ...trap/engines/engine.py.229: process: 0.55364 | load: 0.00249 [I 2022-10-27 08:35:35] ...trap/engines/engine.py.230: loss: 0.21121 [I 2022-10-27 08:35:35] ...optimizers/trijoint.py.151: total_norm: 33.764210 [I 2022-10-27 08:35:40] ...trap/engines/engine.py.224: train: epoch 57 | batch 2120/2304 [I 2022-10-27 08:35:40] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:42 [I 2022-10-27 08:35:40] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00251 [I 2022-10-27 08:35:40] ...trap/engines/engine.py.230: loss: 0.12777 [I 2022-10-27 08:35:40] ...optimizers/trijoint.py.151: total_norm: 30.487633 [I 2022-10-27 08:35:46] ...trap/engines/engine.py.224: train: epoch 57 | batch 2130/2304 [I 2022-10-27 08:35:46] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:37 [I 2022-10-27 08:35:46] ...trap/engines/engine.py.229: process: 0.54413 | load: 0.00247 [I 2022-10-27 08:35:46] ...trap/engines/engine.py.230: loss: 0.20703 [I 2022-10-27 08:35:46] ...optimizers/trijoint.py.151: total_norm: 8.777330 [I 2022-10-27 08:35:52] ...trap/engines/engine.py.224: train: epoch 57 | batch 2140/2304 [I 2022-10-27 08:35:52] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:30 [I 2022-10-27 08:35:52] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00244 [I 2022-10-27 08:35:52] ...trap/engines/engine.py.230: loss: 0.21472 [I 2022-10-27 08:35:52] ...optimizers/trijoint.py.151: total_norm: 12.710864 [I 2022-10-27 08:35:57] ...trap/engines/engine.py.224: train: epoch 57 | batch 2150/2304 [I 2022-10-27 08:35:57] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:25 [I 2022-10-27 08:35:57] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00241 [I 2022-10-27 08:35:57] ...trap/engines/engine.py.230: loss: 0.16324 [I 2022-10-27 08:35:57] ...optimizers/trijoint.py.151: total_norm: 25.106668 [I 2022-10-27 08:36:03] ...trap/engines/engine.py.224: train: epoch 57 | batch 2160/2304 [I 2022-10-27 08:36:03] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:22 [I 2022-10-27 08:36:03] ...trap/engines/engine.py.229: process: 0.59337 | load: 0.00254 [I 2022-10-27 08:36:03] ...trap/engines/engine.py.230: loss: 0.18979 [I 2022-10-27 08:36:03] ...optimizers/trijoint.py.151: total_norm: 15.075095 [I 2022-10-27 08:36:08] ...trap/engines/engine.py.224: train: epoch 57 | batch 2170/2304 [I 2022-10-27 08:36:08] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:01:13 [I 2022-10-27 08:36:08] ...trap/engines/engine.py.229: process: 0.54647 | load: 0.00251 [I 2022-10-27 08:36:08] ...trap/engines/engine.py.230: loss: 0.14751 [I 2022-10-27 08:36:08] ...optimizers/trijoint.py.151: total_norm: 11.536599 [I 2022-10-27 08:36:14] ...trap/engines/engine.py.224: train: epoch 57 | batch 2180/2304 [I 2022-10-27 08:36:14] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:09 [I 2022-10-27 08:36:14] ...trap/engines/engine.py.229: process: 0.54930 | load: 0.00244 [I 2022-10-27 08:36:14] ...trap/engines/engine.py.230: loss: 0.08965 [I 2022-10-27 08:36:14] ...optimizers/trijoint.py.151: total_norm: 14.345507 [I 2022-10-27 08:36:20] ...trap/engines/engine.py.224: train: epoch 57 | batch 2190/2304 [I 2022-10-27 08:36:20] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:01:03 [I 2022-10-27 08:36:20] ...trap/engines/engine.py.229: process: 0.55425 | load: 0.00243 [I 2022-10-27 08:36:20] ...trap/engines/engine.py.230: loss: 0.22624 [I 2022-10-27 08:36:20] ...optimizers/trijoint.py.151: total_norm: 28.858856 [I 2022-10-27 08:36:25] ...trap/engines/engine.py.224: train: epoch 57 | batch 2200/2304 [I 2022-10-27 08:36:25] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:58 [I 2022-10-27 08:36:25] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00252 [I 2022-10-27 08:36:25] ...trap/engines/engine.py.230: loss: 0.23690 [I 2022-10-27 08:36:25] ...optimizers/trijoint.py.151: total_norm: 23.246546 [I 2022-10-27 08:36:31] ...trap/engines/engine.py.224: train: epoch 57 | batch 2210/2304 [I 2022-10-27 08:36:31] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:51 [I 2022-10-27 08:36:31] ...trap/engines/engine.py.229: process: 0.54378 | load: 0.00260 [I 2022-10-27 08:36:31] ...trap/engines/engine.py.230: loss: 0.16342 [I 2022-10-27 08:36:31] ...optimizers/trijoint.py.151: total_norm: 15.360189 [I 2022-10-27 08:36:36] ...trap/engines/engine.py.224: train: epoch 57 | batch 2220/2304 [I 2022-10-27 08:36:36] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:46 [I 2022-10-27 08:36:36] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00256 [I 2022-10-27 08:36:36] ...trap/engines/engine.py.230: loss: 0.16574 [I 2022-10-27 08:36:36] ...optimizers/trijoint.py.151: total_norm: 12.402868 [I 2022-10-27 08:36:42] ...trap/engines/engine.py.224: train: epoch 57 | batch 2230/2304 [I 2022-10-27 08:36:42] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:41 [I 2022-10-27 08:36:42] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00262 [I 2022-10-27 08:36:42] ...trap/engines/engine.py.230: loss: 0.12940 [I 2022-10-27 08:36:42] ...optimizers/trijoint.py.151: total_norm: 17.423151 [I 2022-10-27 08:36:48] ...trap/engines/engine.py.224: train: epoch 57 | batch 2240/2304 [I 2022-10-27 08:36:48] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:35 [I 2022-10-27 08:36:48] ...trap/engines/engine.py.229: process: 0.57233 | load: 0.00254 [I 2022-10-27 08:36:48] ...trap/engines/engine.py.230: loss: 0.15662 [I 2022-10-27 08:36:48] ...optimizers/trijoint.py.151: total_norm: 11.262514 [I 2022-10-27 08:36:53] ...trap/engines/engine.py.224: train: epoch 57 | batch 2250/2304 [I 2022-10-27 08:36:53] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:30 [I 2022-10-27 08:36:53] ...trap/engines/engine.py.229: process: 0.55312 | load: 0.00246 [I 2022-10-27 08:36:53] ...trap/engines/engine.py.230: loss: 0.21658 [I 2022-10-27 08:36:53] ...optimizers/trijoint.py.151: total_norm: 16.339413 [I 2022-10-27 08:36:59] ...trap/engines/engine.py.224: train: epoch 57 | batch 2260/2304 [I 2022-10-27 08:36:59] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:24 [I 2022-10-27 08:36:59] ...trap/engines/engine.py.229: process: 0.54608 | load: 0.00252 [I 2022-10-27 08:36:59] ...trap/engines/engine.py.230: loss: 0.15331 [I 2022-10-27 08:36:59] ...optimizers/trijoint.py.151: total_norm: 15.129854 [I 2022-10-27 08:37:05] ...trap/engines/engine.py.224: train: epoch 57 | batch 2270/2304 [I 2022-10-27 08:37:05] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:19 [I 2022-10-27 08:37:05] ...trap/engines/engine.py.229: process: 0.57678 | load: 0.00250 [I 2022-10-27 08:37:05] ...trap/engines/engine.py.230: loss: 0.12222 [I 2022-10-27 08:37:05] ...optimizers/trijoint.py.151: total_norm: 22.789835 [I 2022-10-27 08:37:10] ...trap/engines/engine.py.224: train: epoch 57 | batch 2280/2304 [I 2022-10-27 08:37:10] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:13 [I 2022-10-27 08:37:10] ...trap/engines/engine.py.229: process: 0.56091 | load: 0.00254 [I 2022-10-27 08:37:10] ...trap/engines/engine.py.230: loss: 0.17453 [I 2022-10-27 08:37:10] ...optimizers/trijoint.py.151: total_norm: 24.829113 [I 2022-10-27 08:37:16] ...trap/engines/engine.py.224: train: epoch 57 | batch 2290/2304 [I 2022-10-27 08:37:16] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:00:07 [I 2022-10-27 08:37:16] ...trap/engines/engine.py.229: process: 0.55425 | load: 0.00252 [I 2022-10-27 08:37:16] ...trap/engines/engine.py.230: loss: 0.12180 [I 2022-10-27 08:37:16] ...optimizers/trijoint.py.151: total_norm: 11.782267 [I 2022-10-27 08:37:21] ...trap/engines/engine.py.224: train: epoch 57 | batch 2300/2304 [I 2022-10-27 08:37:21] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:02 [I 2022-10-27 08:37:21] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00015 [I 2022-10-27 08:37:21] ...trap/engines/engine.py.230: loss: 0.15145 [I 2022-10-27 08:37:21] ...optimizers/trijoint.py.151: total_norm: 21.529154 [I 2022-10-27 08:37:24] ...trap/engines/engine.py.224: train: epoch 57 | batch 2304/2304 [I 2022-10-27 08:37:24] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:00 [I 2022-10-27 08:37:24] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00015 [I 2022-10-27 08:37:24] ...trap/engines/engine.py.230: loss: 0.14361 [I 2022-10-27 08:37:24] ...optimizers/trijoint.py.151: total_norm: 10.290918 [S 2022-10-27 08:37:25] ...trap/engines/engine.py.129: train_epoch.epoch: 57 [S 2022-10-27 08:37:25] ...trap/engines/engine.py.129: train_epoch.loss: 0.15737552216797743 [I 2022-10-27 08:37:25] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 08:37:26] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 57 [I 2022-10-27 08:37:26] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 08:37:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 08:37:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 08:37:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 08:37:30] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 08:37:30] ...trap/engines/engine.py.321: eval: epoch 57 | batch 0/510 [I 2022-10-27 08:37:30] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:37:03 [I 2022-10-27 08:37:30] ...trap/engines/engine.py.326: process: 4.35894 | load: 2.99119 [I 2022-10-27 08:37:33] ...trap/engines/engine.py.321: eval: epoch 57 | batch 10/510 [I 2022-10-27 08:37:33] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:26 [I 2022-10-27 08:37:33] ...trap/engines/engine.py.326: process: 0.35238 | load: 0.00029 [I 2022-10-27 08:37:37] ...trap/engines/engine.py.321: eval: epoch 57 | batch 20/510 [I 2022-10-27 08:37:37] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:07 [I 2022-10-27 08:37:37] ...trap/engines/engine.py.326: process: 0.32982 | load: 0.00016 [I 2022-10-27 08:37:40] ...trap/engines/engine.py.321: eval: epoch 57 | batch 30/510 [I 2022-10-27 08:37:40] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:43 [I 2022-10-27 08:37:40] ...trap/engines/engine.py.326: process: 0.33741 | load: 0.00019 [I 2022-10-27 08:37:44] ...trap/engines/engine.py.321: eval: epoch 57 | batch 40/510 [I 2022-10-27 08:37:44] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:38 [I 2022-10-27 08:37:44] ...trap/engines/engine.py.326: process: 0.35421 | load: 0.00016 [I 2022-10-27 08:37:47] ...trap/engines/engine.py.321: eval: epoch 57 | batch 50/510 [I 2022-10-27 08:37:47] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:32 [I 2022-10-27 08:37:47] ...trap/engines/engine.py.326: process: 0.33095 | load: 0.00017 [I 2022-10-27 08:37:50] ...trap/engines/engine.py.321: eval: epoch 57 | batch 60/510 [I 2022-10-27 08:37:50] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:30 [I 2022-10-27 08:37:50] ...trap/engines/engine.py.326: process: 0.33657 | load: 0.00019 [I 2022-10-27 08:37:54] ...trap/engines/engine.py.321: eval: epoch 57 | batch 70/510 [I 2022-10-27 08:37:54] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:28 [I 2022-10-27 08:37:54] ...trap/engines/engine.py.326: process: 0.35946 | load: 0.00016 [I 2022-10-27 08:37:57] ...trap/engines/engine.py.321: eval: epoch 57 | batch 80/510 [I 2022-10-27 08:37:57] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:23 [I 2022-10-27 08:37:57] ...trap/engines/engine.py.326: process: 0.34081 | load: 0.00022 [I 2022-10-27 08:38:00] ...trap/engines/engine.py.321: eval: epoch 57 | batch 90/510 [I 2022-10-27 08:38:00] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:18 [I 2022-10-27 08:38:00] ...trap/engines/engine.py.326: process: 0.33058 | load: 0.00016 [I 2022-10-27 08:38:03] ...trap/engines/engine.py.321: eval: epoch 57 | batch 100/510 [I 2022-10-27 08:38:03] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:10 [I 2022-10-27 08:38:03] ...trap/engines/engine.py.326: process: 0.28316 | load: 0.00018 [I 2022-10-27 08:38:06] ...trap/engines/engine.py.321: eval: epoch 57 | batch 110/510 [I 2022-10-27 08:38:06] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:57 [I 2022-10-27 08:38:06] ...trap/engines/engine.py.326: process: 0.28861 | load: 0.00019 [I 2022-10-27 08:38:09] ...trap/engines/engine.py.321: eval: epoch 57 | batch 120/510 [I 2022-10-27 08:38:09] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:54 [I 2022-10-27 08:38:09] ...trap/engines/engine.py.326: process: 0.30835 | load: 0.00019 [I 2022-10-27 08:38:12] ...trap/engines/engine.py.321: eval: epoch 57 | batch 130/510 [I 2022-10-27 08:38:12] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:49 [I 2022-10-27 08:38:12] ...trap/engines/engine.py.326: process: 0.28680 | load: 0.00016 [I 2022-10-27 08:38:15] ...trap/engines/engine.py.321: eval: epoch 57 | batch 140/510 [I 2022-10-27 08:38:15] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 08:38:15] ...trap/engines/engine.py.326: process: 0.29027 | load: 0.00017 [I 2022-10-27 08:38:18] ...trap/engines/engine.py.321: eval: epoch 57 | batch 150/510 [I 2022-10-27 08:38:18] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 08:38:18] ...trap/engines/engine.py.326: process: 0.28604 | load: 0.00020 [I 2022-10-27 08:38:21] ...trap/engines/engine.py.321: eval: epoch 57 | batch 160/510 [I 2022-10-27 08:38:21] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:42 [I 2022-10-27 08:38:21] ...trap/engines/engine.py.326: process: 0.29085 | load: 0.00021 [I 2022-10-27 08:38:24] ...trap/engines/engine.py.321: eval: epoch 57 | batch 170/510 [I 2022-10-27 08:38:24] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-27 08:38:24] ...trap/engines/engine.py.326: process: 0.28926 | load: 0.00014 [I 2022-10-27 08:38:27] ...trap/engines/engine.py.321: eval: epoch 57 | batch 180/510 [I 2022-10-27 08:38:27] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 08:38:27] ...trap/engines/engine.py.326: process: 0.29184 | load: 0.00019 [I 2022-10-27 08:38:29] ...trap/engines/engine.py.321: eval: epoch 57 | batch 190/510 [I 2022-10-27 08:38:29] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:32 [I 2022-10-27 08:38:29] ...trap/engines/engine.py.326: process: 0.28446 | load: 0.00016 [I 2022-10-27 08:38:33] ...trap/engines/engine.py.321: eval: epoch 57 | batch 200/510 [I 2022-10-27 08:38:33] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:32 [I 2022-10-27 08:38:33] ...trap/engines/engine.py.326: process: 0.29350 | load: 0.00021 [I 2022-10-27 08:38:35] ...trap/engines/engine.py.321: eval: epoch 57 | batch 210/510 [I 2022-10-27 08:38:35] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 08:38:35] ...trap/engines/engine.py.326: process: 0.28539 | load: 0.00016 [I 2022-10-27 08:38:38] ...trap/engines/engine.py.321: eval: epoch 57 | batch 220/510 [I 2022-10-27 08:38:38] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 08:38:38] ...trap/engines/engine.py.326: process: 0.28936 | load: 0.00015 [I 2022-10-27 08:38:41] ...trap/engines/engine.py.321: eval: epoch 57 | batch 230/510 [I 2022-10-27 08:38:41] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 08:38:41] ...trap/engines/engine.py.326: process: 0.28542 | load: 0.00019 [I 2022-10-27 08:38:44] ...trap/engines/engine.py.321: eval: epoch 57 | batch 240/510 [I 2022-10-27 08:38:44] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 08:38:44] ...trap/engines/engine.py.326: process: 0.28700 | load: 0.00027 [I 2022-10-27 08:38:47] ...trap/engines/engine.py.321: eval: epoch 57 | batch 250/510 [I 2022-10-27 08:38:47] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 08:38:47] ...trap/engines/engine.py.326: process: 0.28796 | load: 0.00018 [I 2022-10-27 08:38:50] ...trap/engines/engine.py.321: eval: epoch 57 | batch 260/510 [I 2022-10-27 08:38:50] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 08:38:50] ...trap/engines/engine.py.326: process: 0.29234 | load: 0.00016 [I 2022-10-27 08:38:53] ...trap/engines/engine.py.321: eval: epoch 57 | batch 270/510 [I 2022-10-27 08:38:53] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 08:38:53] ...trap/engines/engine.py.326: process: 0.28162 | load: 0.00017 [I 2022-10-27 08:38:56] ...trap/engines/engine.py.321: eval: epoch 57 | batch 280/510 [I 2022-10-27 08:38:56] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 08:38:56] ...trap/engines/engine.py.326: process: 0.28405 | load: 0.00022 [I 2022-10-27 08:38:58] ...trap/engines/engine.py.321: eval: epoch 57 | batch 290/510 [I 2022-10-27 08:38:58] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 08:38:58] ...trap/engines/engine.py.326: process: 0.29262 | load: 0.00017 [I 2022-10-27 08:39:01] ...trap/engines/engine.py.321: eval: epoch 57 | batch 300/510 [I 2022-10-27 08:39:01] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 08:39:01] ...trap/engines/engine.py.326: process: 0.29647 | load: 0.00016 [I 2022-10-27 08:39:04] ...trap/engines/engine.py.321: eval: epoch 57 | batch 310/510 [I 2022-10-27 08:39:04] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 08:39:04] ...trap/engines/engine.py.326: process: 0.28043 | load: 0.00023 [I 2022-10-27 08:39:07] ...trap/engines/engine.py.321: eval: epoch 57 | batch 320/510 [I 2022-10-27 08:39:07] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 08:39:07] ...trap/engines/engine.py.326: process: 0.28754 | load: 0.00024 [I 2022-10-27 08:39:10] ...trap/engines/engine.py.321: eval: epoch 57 | batch 330/510 [I 2022-10-27 08:39:10] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 08:39:10] ...trap/engines/engine.py.326: process: 0.28300 | load: 0.00018 [I 2022-10-27 08:39:13] ...trap/engines/engine.py.321: eval: epoch 57 | batch 340/510 [I 2022-10-27 08:39:13] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-27 08:39:13] ...trap/engines/engine.py.326: process: 0.29184 | load: 0.00016 [I 2022-10-27 08:39:16] ...trap/engines/engine.py.321: eval: epoch 57 | batch 350/510 [I 2022-10-27 08:39:16] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-27 08:39:16] ...trap/engines/engine.py.326: process: 0.28863 | load: 0.00016 [I 2022-10-27 08:39:19] ...trap/engines/engine.py.321: eval: epoch 57 | batch 360/510 [I 2022-10-27 08:39:19] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 08:39:19] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00017 [I 2022-10-27 08:39:21] ...trap/engines/engine.py.321: eval: epoch 57 | batch 370/510 [I 2022-10-27 08:39:21] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 08:39:21] ...trap/engines/engine.py.326: process: 0.28696 | load: 0.00016 [I 2022-10-27 08:39:24] ...trap/engines/engine.py.321: eval: epoch 57 | batch 380/510 [I 2022-10-27 08:39:24] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 08:39:24] ...trap/engines/engine.py.326: process: 0.28995 | load: 0.00021 [I 2022-10-27 08:39:27] ...trap/engines/engine.py.321: eval: epoch 57 | batch 390/510 [I 2022-10-27 08:39:27] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 08:39:27] ...trap/engines/engine.py.326: process: 0.28887 | load: 0.00021 [I 2022-10-27 08:39:30] ...trap/engines/engine.py.321: eval: epoch 57 | batch 400/510 [I 2022-10-27 08:39:30] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 08:39:30] ...trap/engines/engine.py.326: process: 0.28132 | load: 0.00021 [I 2022-10-27 08:39:33] ...trap/engines/engine.py.321: eval: epoch 57 | batch 410/510 [I 2022-10-27 08:39:33] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 08:39:33] ...trap/engines/engine.py.326: process: 0.28877 | load: 0.00021 [I 2022-10-27 08:39:36] ...trap/engines/engine.py.321: eval: epoch 57 | batch 420/510 [I 2022-10-27 08:39:36] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:26 [I 2022-10-27 08:39:36] ...trap/engines/engine.py.326: process: 0.29194 | load: 0.00021 [I 2022-10-27 08:39:39] ...trap/engines/engine.py.321: eval: epoch 57 | batch 430/510 [I 2022-10-27 08:39:39] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 08:39:39] ...trap/engines/engine.py.326: process: 0.29077 | load: 0.00023 [I 2022-10-27 08:39:42] ...trap/engines/engine.py.321: eval: epoch 57 | batch 440/510 [I 2022-10-27 08:39:42] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 08:39:42] ...trap/engines/engine.py.326: process: 0.28667 | load: 0.00016 [I 2022-10-27 08:39:45] ...trap/engines/engine.py.321: eval: epoch 57 | batch 450/510 [I 2022-10-27 08:39:45] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 08:39:45] ...trap/engines/engine.py.326: process: 0.28955 | load: 0.00017 [I 2022-10-27 08:39:47] ...trap/engines/engine.py.321: eval: epoch 57 | batch 460/510 [I 2022-10-27 08:39:47] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 08:39:47] ...trap/engines/engine.py.326: process: 0.28806 | load: 0.00016 [I 2022-10-27 08:39:50] ...trap/engines/engine.py.321: eval: epoch 57 | batch 470/510 [I 2022-10-27 08:39:50] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 08:39:50] ...trap/engines/engine.py.326: process: 0.29044 | load: 0.00017 [I 2022-10-27 08:39:53] ...trap/engines/engine.py.321: eval: epoch 57 | batch 480/510 [I 2022-10-27 08:39:53] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 08:39:53] ...trap/engines/engine.py.326: process: 0.29116 | load: 0.00023 [I 2022-10-27 08:39:56] ...trap/engines/engine.py.321: eval: epoch 57 | batch 490/510 [I 2022-10-27 08:39:56] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 08:39:56] ...trap/engines/engine.py.326: process: 0.28578 | load: 0.00016 [I 2022-10-27 08:39:59] ...trap/engines/engine.py.321: eval: epoch 57 | batch 500/510 [I 2022-10-27 08:39:59] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 08:39:59] ...trap/engines/engine.py.326: process: 0.29404 | load: 0.00028 [I 2022-10-27 08:40:02] ...trap/engines/engine.py.321: eval: epoch 57 | batch 510/510 [I 2022-10-27 08:40:02] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 08:40:02] ...trap/engines/engine.py.326: process: 0.28692 | load: 0.00019 [S 2022-10-27 08:40:03] ...trap/engines/engine.py.132: eval_epoch.epoch: 57 [I 2022-10-27 08:40:03] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.710800 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.901000 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935200 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.694900 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.897500 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931900 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.007167 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005882 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004190 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008408 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006249 [S 2022-10-27 08:40:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.003646 [I 2022-10-27 08:40:08] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 08:40:08] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 08:40:08] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 08:40:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 08:40:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 08:40:11] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 08:40:13] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 08:40:19] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 08:40:19] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 08:40:19] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 08:40:22] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 08:40:28] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 08:40:28] ...trap/engines/engine.py.163: Training model on trainset for epoch 58 [I 2022-10-27 08:40:28] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 08:40:31] ...trap/engines/engine.py.224: train: epoch 58 | batch 0/2304 [I 2022-10-27 08:40:31] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:11 [I 2022-10-27 08:40:31] ...trap/engines/engine.py.229: process: 2.76562 | load: 2.14534 [I 2022-10-27 08:40:31] ...trap/engines/engine.py.230: loss: 0.19328 [I 2022-10-27 08:40:31] ...optimizers/trijoint.py.151: total_norm: 14.759839 [I 2022-10-27 08:40:36] ...trap/engines/engine.py.224: train: epoch 58 | batch 10/2304 [I 2022-10-27 08:40:36] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:41 [I 2022-10-27 08:40:36] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00384 [I 2022-10-27 08:40:36] ...trap/engines/engine.py.230: loss: 0.20524 [I 2022-10-27 08:40:36] ...optimizers/trijoint.py.151: total_norm: 19.070545 [I 2022-10-27 08:40:42] ...trap/engines/engine.py.224: train: epoch 58 | batch 20/2304 [I 2022-10-27 08:40:42] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:11 [I 2022-10-27 08:40:42] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00316 [I 2022-10-27 08:40:42] ...trap/engines/engine.py.230: loss: 0.09896 [I 2022-10-27 08:40:42] ...optimizers/trijoint.py.151: total_norm: 8.210584 [I 2022-10-27 08:40:48] ...trap/engines/engine.py.224: train: epoch 58 | batch 30/2304 [I 2022-10-27 08:40:48] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-27 08:40:48] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00381 [I 2022-10-27 08:40:48] ...trap/engines/engine.py.230: loss: 0.23674 [I 2022-10-27 08:40:48] ...optimizers/trijoint.py.151: total_norm: 21.944658 [I 2022-10-27 08:40:53] ...trap/engines/engine.py.224: train: epoch 58 | batch 40/2304 [I 2022-10-27 08:40:53] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:52 [I 2022-10-27 08:40:53] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00359 [I 2022-10-27 08:40:53] ...trap/engines/engine.py.230: loss: 0.13115 [I 2022-10-27 08:40:53] ...optimizers/trijoint.py.151: total_norm: 13.313618 [I 2022-10-27 08:40:59] ...trap/engines/engine.py.224: train: epoch 58 | batch 50/2304 [I 2022-10-27 08:40:59] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:41 [I 2022-10-27 08:40:59] ...trap/engines/engine.py.229: process: 0.54781 | load: 0.00295 [I 2022-10-27 08:40:59] ...trap/engines/engine.py.230: loss: 0.17214 [I 2022-10-27 08:40:59] ...optimizers/trijoint.py.151: total_norm: 21.158545 [I 2022-10-27 08:41:04] ...trap/engines/engine.py.224: train: epoch 58 | batch 60/2304 [I 2022-10-27 08:41:04] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:40 [I 2022-10-27 08:41:04] ...trap/engines/engine.py.229: process: 0.54861 | load: 0.00291 [I 2022-10-27 08:41:04] ...trap/engines/engine.py.230: loss: 0.11358 [I 2022-10-27 08:41:04] ...optimizers/trijoint.py.151: total_norm: 17.161226 [I 2022-10-27 08:41:10] ...trap/engines/engine.py.224: train: epoch 58 | batch 70/2304 [I 2022-10-27 08:41:10] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:45 [I 2022-10-27 08:41:10] ...trap/engines/engine.py.229: process: 0.55009 | load: 0.00253 [I 2022-10-27 08:41:10] ...trap/engines/engine.py.230: loss: 0.21501 [I 2022-10-27 08:41:10] ...optimizers/trijoint.py.151: total_norm: 23.581053 [I 2022-10-27 08:41:15] ...trap/engines/engine.py.224: train: epoch 58 | batch 80/2304 [I 2022-10-27 08:41:15] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:32 [I 2022-10-27 08:41:15] ...trap/engines/engine.py.229: process: 0.55201 | load: 0.00278 [I 2022-10-27 08:41:15] ...trap/engines/engine.py.230: loss: 0.22592 [I 2022-10-27 08:41:15] ...optimizers/trijoint.py.151: total_norm: 19.374344 [I 2022-10-27 08:41:21] ...trap/engines/engine.py.224: train: epoch 58 | batch 90/2304 [I 2022-10-27 08:41:21] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:31 [I 2022-10-27 08:41:21] ...trap/engines/engine.py.229: process: 0.55112 | load: 0.00278 [I 2022-10-27 08:41:21] ...trap/engines/engine.py.230: loss: 0.11707 [I 2022-10-27 08:41:21] ...optimizers/trijoint.py.151: total_norm: 17.906214 [I 2022-10-27 08:41:26] ...trap/engines/engine.py.224: train: epoch 58 | batch 100/2304 [I 2022-10-27 08:41:26] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:18 [I 2022-10-27 08:41:26] ...trap/engines/engine.py.229: process: 0.55044 | load: 0.00279 [I 2022-10-27 08:41:26] ...trap/engines/engine.py.230: loss: 0.16806 [I 2022-10-27 08:41:26] ...optimizers/trijoint.py.151: total_norm: 13.850656 [I 2022-10-27 08:41:32] ...trap/engines/engine.py.224: train: epoch 58 | batch 110/2304 [I 2022-10-27 08:41:32] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:17 [I 2022-10-27 08:41:32] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00257 [I 2022-10-27 08:41:32] ...trap/engines/engine.py.230: loss: 0.19469 [I 2022-10-27 08:41:32] ...optimizers/trijoint.py.151: total_norm: 24.399422 [I 2022-10-27 08:41:38] ...trap/engines/engine.py.224: train: epoch 58 | batch 120/2304 [I 2022-10-27 08:41:38] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:09 [I 2022-10-27 08:41:38] ...trap/engines/engine.py.229: process: 0.54494 | load: 0.00267 [I 2022-10-27 08:41:38] ...trap/engines/engine.py.230: loss: 0.11990 [I 2022-10-27 08:41:38] ...optimizers/trijoint.py.151: total_norm: 16.851742 [I 2022-10-27 08:41:43] ...trap/engines/engine.py.224: train: epoch 58 | batch 130/2304 [I 2022-10-27 08:41:43] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:11 [I 2022-10-27 08:41:43] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00285 [I 2022-10-27 08:41:43] ...trap/engines/engine.py.230: loss: 0.17773 [I 2022-10-27 08:41:43] ...optimizers/trijoint.py.151: total_norm: 25.493296 [I 2022-10-27 08:41:49] ...trap/engines/engine.py.224: train: epoch 58 | batch 140/2304 [I 2022-10-27 08:41:49] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:06 [I 2022-10-27 08:41:49] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00257 [I 2022-10-27 08:41:49] ...trap/engines/engine.py.230: loss: 0.16222 [I 2022-10-27 08:41:49] ...optimizers/trijoint.py.151: total_norm: 10.192004 [I 2022-10-27 08:41:54] ...trap/engines/engine.py.224: train: epoch 58 | batch 150/2304 [I 2022-10-27 08:41:54] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:57 [I 2022-10-27 08:41:54] ...trap/engines/engine.py.229: process: 0.55506 | load: 0.00263 [I 2022-10-27 08:41:54] ...trap/engines/engine.py.230: loss: 0.16761 [I 2022-10-27 08:41:54] ...optimizers/trijoint.py.151: total_norm: 16.260805 [I 2022-10-27 08:42:00] ...trap/engines/engine.py.224: train: epoch 58 | batch 160/2304 [I 2022-10-27 08:42:00] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:53 [I 2022-10-27 08:42:00] ...trap/engines/engine.py.229: process: 0.55406 | load: 0.00264 [I 2022-10-27 08:42:00] ...trap/engines/engine.py.230: loss: 0.21703 [I 2022-10-27 08:42:00] ...optimizers/trijoint.py.151: total_norm: 10.558330 [I 2022-10-27 08:42:05] ...trap/engines/engine.py.224: train: epoch 58 | batch 170/2304 [I 2022-10-27 08:42:05] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:40 [I 2022-10-27 08:42:05] ...trap/engines/engine.py.229: process: 0.54459 | load: 0.00282 [I 2022-10-27 08:42:05] ...trap/engines/engine.py.230: loss: 0.14924 [I 2022-10-27 08:42:05] ...optimizers/trijoint.py.151: total_norm: 19.073090 [I 2022-10-27 08:42:11] ...trap/engines/engine.py.224: train: epoch 58 | batch 180/2304 [I 2022-10-27 08:42:11] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:19:34 [I 2022-10-27 08:42:11] ...trap/engines/engine.py.229: process: 0.54915 | load: 0.00250 [I 2022-10-27 08:42:11] ...trap/engines/engine.py.230: loss: 0.13567 [I 2022-10-27 08:42:11] ...optimizers/trijoint.py.151: total_norm: 18.825560 [I 2022-10-27 08:42:16] ...trap/engines/engine.py.224: train: epoch 58 | batch 190/2304 [I 2022-10-27 08:42:16] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:28 [I 2022-10-27 08:42:16] ...trap/engines/engine.py.229: process: 0.54464 | load: 0.00244 [I 2022-10-27 08:42:16] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-27 08:42:17] ...optimizers/trijoint.py.151: total_norm: 16.589920 [I 2022-10-27 08:42:22] ...trap/engines/engine.py.224: train: epoch 58 | batch 200/2304 [I 2022-10-27 08:42:22] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:23 [I 2022-10-27 08:42:22] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00250 [I 2022-10-27 08:42:22] ...trap/engines/engine.py.230: loss: 0.12266 [I 2022-10-27 08:42:22] ...optimizers/trijoint.py.151: total_norm: 28.082497 [I 2022-10-27 08:42:28] ...trap/engines/engine.py.224: train: epoch 58 | batch 210/2304 [I 2022-10-27 08:42:28] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:16 [I 2022-10-27 08:42:28] ...trap/engines/engine.py.229: process: 0.55266 | load: 0.00255 [I 2022-10-27 08:42:28] ...trap/engines/engine.py.230: loss: 0.17666 [I 2022-10-27 08:42:28] ...optimizers/trijoint.py.151: total_norm: 24.295980 [I 2022-10-27 08:42:33] ...trap/engines/engine.py.224: train: epoch 58 | batch 220/2304 [I 2022-10-27 08:42:33] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:18 [I 2022-10-27 08:42:33] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00250 [I 2022-10-27 08:42:33] ...trap/engines/engine.py.230: loss: 0.21063 [I 2022-10-27 08:42:33] ...optimizers/trijoint.py.151: total_norm: 40.235653 [I 2022-10-27 08:42:39] ...trap/engines/engine.py.224: train: epoch 58 | batch 230/2304 [I 2022-10-27 08:42:39] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:07 [I 2022-10-27 08:42:39] ...trap/engines/engine.py.229: process: 0.54482 | load: 0.00255 [I 2022-10-27 08:42:39] ...trap/engines/engine.py.230: loss: 0.15403 [I 2022-10-27 08:42:39] ...optimizers/trijoint.py.151: total_norm: 26.465927 [I 2022-10-27 08:42:44] ...trap/engines/engine.py.224: train: epoch 58 | batch 240/2304 [I 2022-10-27 08:42:44] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:04 [I 2022-10-27 08:42:44] ...trap/engines/engine.py.229: process: 0.55218 | load: 0.00242 [I 2022-10-27 08:42:44] ...trap/engines/engine.py.230: loss: 0.19252 [I 2022-10-27 08:42:44] ...optimizers/trijoint.py.151: total_norm: 21.647564 [I 2022-10-27 08:42:50] ...trap/engines/engine.py.224: train: epoch 58 | batch 250/2304 [I 2022-10-27 08:42:50] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:18:57 [I 2022-10-27 08:42:50] ...trap/engines/engine.py.229: process: 0.55888 | load: 0.00250 [I 2022-10-27 08:42:50] ...trap/engines/engine.py.230: loss: 0.10416 [I 2022-10-27 08:42:50] ...optimizers/trijoint.py.151: total_norm: 9.570061 [I 2022-10-27 08:42:55] ...trap/engines/engine.py.224: train: epoch 58 | batch 260/2304 [I 2022-10-27 08:42:55] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:47 [I 2022-10-27 08:42:55] ...trap/engines/engine.py.229: process: 0.56592 | load: 0.00242 [I 2022-10-27 08:42:55] ...trap/engines/engine.py.230: loss: 0.19436 [I 2022-10-27 08:42:55] ...optimizers/trijoint.py.151: total_norm: 14.691307 [I 2022-10-27 08:43:01] ...trap/engines/engine.py.224: train: epoch 58 | batch 270/2304 [I 2022-10-27 08:43:01] ...trap/engines/engine.py.228: elapsed: 0:02:32 | left: 0:18:47 [I 2022-10-27 08:43:01] ...trap/engines/engine.py.229: process: 0.55425 | load: 0.00249 [I 2022-10-27 08:43:01] ...trap/engines/engine.py.230: loss: 0.15297 [I 2022-10-27 08:43:01] ...optimizers/trijoint.py.151: total_norm: 18.028009 [I 2022-10-27 08:43:06] ...trap/engines/engine.py.224: train: epoch 58 | batch 280/2304 [I 2022-10-27 08:43:06] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:46 [I 2022-10-27 08:43:06] ...trap/engines/engine.py.229: process: 0.54896 | load: 0.00249 [I 2022-10-27 08:43:06] ...trap/engines/engine.py.230: loss: 0.15977 [I 2022-10-27 08:43:06] ...optimizers/trijoint.py.151: total_norm: 33.242489 [I 2022-10-27 08:43:12] ...trap/engines/engine.py.224: train: epoch 58 | batch 290/2304 [I 2022-10-27 08:43:12] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:43 [I 2022-10-27 08:43:12] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00253 [I 2022-10-27 08:43:12] ...trap/engines/engine.py.230: loss: 0.17802 [I 2022-10-27 08:43:12] ...optimizers/trijoint.py.151: total_norm: 13.189454 [I 2022-10-27 08:43:18] ...trap/engines/engine.py.224: train: epoch 58 | batch 300/2304 [I 2022-10-27 08:43:18] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:18:30 [I 2022-10-27 08:43:18] ...trap/engines/engine.py.229: process: 0.55201 | load: 0.00245 [I 2022-10-27 08:43:18] ...trap/engines/engine.py.230: loss: 0.23360 [I 2022-10-27 08:43:18] ...optimizers/trijoint.py.151: total_norm: 15.683268 [I 2022-10-27 08:43:23] ...trap/engines/engine.py.224: train: epoch 58 | batch 310/2304 [I 2022-10-27 08:43:23] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:25 [I 2022-10-27 08:43:23] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00243 [I 2022-10-27 08:43:23] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-27 08:43:23] ...optimizers/trijoint.py.151: total_norm: 21.125881 [I 2022-10-27 08:43:29] ...trap/engines/engine.py.224: train: epoch 58 | batch 320/2304 [I 2022-10-27 08:43:29] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:18:16 [I 2022-10-27 08:43:29] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00240 [I 2022-10-27 08:43:29] ...trap/engines/engine.py.230: loss: 0.19139 [I 2022-10-27 08:43:29] ...optimizers/trijoint.py.151: total_norm: 25.918550 [I 2022-10-27 08:43:34] ...trap/engines/engine.py.224: train: epoch 58 | batch 330/2304 [I 2022-10-27 08:43:34] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:09 [I 2022-10-27 08:43:34] ...trap/engines/engine.py.229: process: 0.54458 | load: 0.00243 [I 2022-10-27 08:43:34] ...trap/engines/engine.py.230: loss: 0.14994 [I 2022-10-27 08:43:34] ...optimizers/trijoint.py.151: total_norm: 25.974167 [I 2022-10-27 08:43:40] ...trap/engines/engine.py.224: train: epoch 58 | batch 340/2304 [I 2022-10-27 08:43:40] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:18:06 [I 2022-10-27 08:43:40] ...trap/engines/engine.py.229: process: 0.55366 | load: 0.00244 [I 2022-10-27 08:43:40] ...trap/engines/engine.py.230: loss: 0.17265 [I 2022-10-27 08:43:40] ...optimizers/trijoint.py.151: total_norm: 16.770693 [I 2022-10-27 08:43:45] ...trap/engines/engine.py.224: train: epoch 58 | batch 350/2304 [I 2022-10-27 08:43:45] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:18:04 [I 2022-10-27 08:43:45] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00243 [I 2022-10-27 08:43:45] ...trap/engines/engine.py.230: loss: 0.18439 [I 2022-10-27 08:43:45] ...optimizers/trijoint.py.151: total_norm: 14.488562 [I 2022-10-27 08:43:51] ...trap/engines/engine.py.224: train: epoch 58 | batch 360/2304 [I 2022-10-27 08:43:51] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:17:54 [I 2022-10-27 08:43:51] ...trap/engines/engine.py.229: process: 0.55354 | load: 0.00247 [I 2022-10-27 08:43:51] ...trap/engines/engine.py.230: loss: 0.12720 [I 2022-10-27 08:43:51] ...optimizers/trijoint.py.151: total_norm: 47.892433 [I 2022-10-27 08:43:56] ...trap/engines/engine.py.224: train: epoch 58 | batch 370/2304 [I 2022-10-27 08:43:56] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:17:51 [I 2022-10-27 08:43:56] ...trap/engines/engine.py.229: process: 0.56529 | load: 0.00242 [I 2022-10-27 08:43:56] ...trap/engines/engine.py.230: loss: 0.21063 [I 2022-10-27 08:43:56] ...optimizers/trijoint.py.151: total_norm: 24.572744 [I 2022-10-27 08:44:02] ...trap/engines/engine.py.224: train: epoch 58 | batch 380/2304 [I 2022-10-27 08:44:02] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:55 [I 2022-10-27 08:44:02] ...trap/engines/engine.py.229: process: 0.56829 | load: 0.00248 [I 2022-10-27 08:44:02] ...trap/engines/engine.py.230: loss: 0.16073 [I 2022-10-27 08:44:02] ...optimizers/trijoint.py.151: total_norm: 8.953151 [I 2022-10-27 08:44:08] ...trap/engines/engine.py.224: train: epoch 58 | batch 390/2304 [I 2022-10-27 08:44:08] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:17:34 [I 2022-10-27 08:44:08] ...trap/engines/engine.py.229: process: 0.54620 | load: 0.00247 [I 2022-10-27 08:44:08] ...trap/engines/engine.py.230: loss: 0.12992 [I 2022-10-27 08:44:08] ...optimizers/trijoint.py.151: total_norm: 26.847147 [I 2022-10-27 08:44:13] ...trap/engines/engine.py.224: train: epoch 58 | batch 400/2304 [I 2022-10-27 08:44:13] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:43 [I 2022-10-27 08:44:13] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00244 [I 2022-10-27 08:44:13] ...trap/engines/engine.py.230: loss: 0.17662 [I 2022-10-27 08:44:13] ...optimizers/trijoint.py.151: total_norm: 16.409309 [I 2022-10-27 08:44:19] ...trap/engines/engine.py.224: train: epoch 58 | batch 410/2304 [I 2022-10-27 08:44:19] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:17:20 [I 2022-10-27 08:44:19] ...trap/engines/engine.py.229: process: 0.54015 | load: 0.00251 [I 2022-10-27 08:44:19] ...trap/engines/engine.py.230: loss: 0.14931 [I 2022-10-27 08:44:19] ...optimizers/trijoint.py.151: total_norm: 10.938636 [I 2022-10-27 08:44:24] ...trap/engines/engine.py.224: train: epoch 58 | batch 420/2304 [I 2022-10-27 08:44:24] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:17:24 [I 2022-10-27 08:44:24] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00253 [I 2022-10-27 08:44:24] ...trap/engines/engine.py.230: loss: 0.22606 [I 2022-10-27 08:44:24] ...optimizers/trijoint.py.151: total_norm: 24.260796 [I 2022-10-27 08:44:30] ...trap/engines/engine.py.224: train: epoch 58 | batch 430/2304 [I 2022-10-27 08:44:30] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:17:19 [I 2022-10-27 08:44:30] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00242 [I 2022-10-27 08:44:30] ...trap/engines/engine.py.230: loss: 0.22225 [I 2022-10-27 08:44:30] ...optimizers/trijoint.py.151: total_norm: 12.779326 [I 2022-10-27 08:44:35] ...trap/engines/engine.py.224: train: epoch 58 | batch 440/2304 [I 2022-10-27 08:44:35] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:17:10 [I 2022-10-27 08:44:35] ...trap/engines/engine.py.229: process: 0.55158 | load: 0.00238 [I 2022-10-27 08:44:35] ...trap/engines/engine.py.230: loss: 0.21494 [I 2022-10-27 08:44:35] ...optimizers/trijoint.py.151: total_norm: 39.579494 [I 2022-10-27 08:44:41] ...trap/engines/engine.py.224: train: epoch 58 | batch 450/2304 [I 2022-10-27 08:44:41] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:17:00 [I 2022-10-27 08:44:41] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00243 [I 2022-10-27 08:44:41] ...trap/engines/engine.py.230: loss: 0.13143 [I 2022-10-27 08:44:41] ...optimizers/trijoint.py.151: total_norm: 13.545737 [I 2022-10-27 08:44:46] ...trap/engines/engine.py.224: train: epoch 58 | batch 460/2304 [I 2022-10-27 08:44:46] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:17:02 [I 2022-10-27 08:44:46] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00242 [I 2022-10-27 08:44:46] ...trap/engines/engine.py.230: loss: 0.13629 [I 2022-10-27 08:44:46] ...optimizers/trijoint.py.151: total_norm: 12.618068 [I 2022-10-27 08:44:52] ...trap/engines/engine.py.224: train: epoch 58 | batch 470/2304 [I 2022-10-27 08:44:52] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:16:56 [I 2022-10-27 08:44:52] ...trap/engines/engine.py.229: process: 0.56851 | load: 0.00241 [I 2022-10-27 08:44:52] ...trap/engines/engine.py.230: loss: 0.15938 [I 2022-10-27 08:44:52] ...optimizers/trijoint.py.151: total_norm: 19.294600 [I 2022-10-27 08:44:57] ...trap/engines/engine.py.224: train: epoch 58 | batch 480/2304 [I 2022-10-27 08:44:57] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:16:53 [I 2022-10-27 08:44:57] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00241 [I 2022-10-27 08:44:57] ...trap/engines/engine.py.230: loss: 0.13342 [I 2022-10-27 08:44:57] ...optimizers/trijoint.py.151: total_norm: 10.500829 [I 2022-10-27 08:45:03] ...trap/engines/engine.py.224: train: epoch 58 | batch 490/2304 [I 2022-10-27 08:45:03] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:16:44 [I 2022-10-27 08:45:03] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00241 [I 2022-10-27 08:45:03] ...trap/engines/engine.py.230: loss: 0.15259 [I 2022-10-27 08:45:03] ...optimizers/trijoint.py.151: total_norm: 17.490387 [I 2022-10-27 08:45:08] ...trap/engines/engine.py.224: train: epoch 58 | batch 500/2304 [I 2022-10-27 08:45:08] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:16:43 [I 2022-10-27 08:45:08] ...trap/engines/engine.py.229: process: 0.55325 | load: 0.00247 [I 2022-10-27 08:45:08] ...trap/engines/engine.py.230: loss: 0.11971 [I 2022-10-27 08:45:08] ...optimizers/trijoint.py.151: total_norm: 13.422590 [I 2022-10-27 08:45:14] ...trap/engines/engine.py.224: train: epoch 58 | batch 510/2304 [I 2022-10-27 08:45:14] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:16:37 [I 2022-10-27 08:45:14] ...trap/engines/engine.py.229: process: 0.55175 | load: 0.00253 [I 2022-10-27 08:45:14] ...trap/engines/engine.py.230: loss: 0.16292 [I 2022-10-27 08:45:14] ...optimizers/trijoint.py.151: total_norm: 15.318450 [I 2022-10-27 08:45:20] ...trap/engines/engine.py.224: train: epoch 58 | batch 520/2304 [I 2022-10-27 08:45:20] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:16:33 [I 2022-10-27 08:45:20] ...trap/engines/engine.py.229: process: 0.55776 | load: 0.00245 [I 2022-10-27 08:45:20] ...trap/engines/engine.py.230: loss: 0.24465 [I 2022-10-27 08:45:20] ...optimizers/trijoint.py.151: total_norm: 31.468260 [I 2022-10-27 08:45:25] ...trap/engines/engine.py.224: train: epoch 58 | batch 530/2304 [I 2022-10-27 08:45:25] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:16:15 [I 2022-10-27 08:45:25] ...trap/engines/engine.py.229: process: 0.54746 | load: 0.00246 [I 2022-10-27 08:45:25] ...trap/engines/engine.py.230: loss: 0.17031 [I 2022-10-27 08:45:25] ...optimizers/trijoint.py.151: total_norm: 20.281401 [I 2022-10-27 08:45:31] ...trap/engines/engine.py.224: train: epoch 58 | batch 540/2304 [I 2022-10-27 08:45:31] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:16:15 [I 2022-10-27 08:45:31] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00240 [I 2022-10-27 08:45:31] ...trap/engines/engine.py.230: loss: 0.15952 [I 2022-10-27 08:45:31] ...optimizers/trijoint.py.151: total_norm: 12.434731 [I 2022-10-27 08:45:36] ...trap/engines/engine.py.224: train: epoch 58 | batch 550/2304 [I 2022-10-27 08:45:36] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:16:16 [I 2022-10-27 08:45:36] ...trap/engines/engine.py.229: process: 0.55360 | load: 0.00246 [I 2022-10-27 08:45:36] ...trap/engines/engine.py.230: loss: 0.22719 [I 2022-10-27 08:45:36] ...optimizers/trijoint.py.151: total_norm: 25.101740 [I 2022-10-27 08:45:42] ...trap/engines/engine.py.224: train: epoch 58 | batch 560/2304 [I 2022-10-27 08:45:42] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:16:05 [I 2022-10-27 08:45:42] ...trap/engines/engine.py.229: process: 0.55490 | load: 0.00244 [I 2022-10-27 08:45:42] ...trap/engines/engine.py.230: loss: 0.20957 [I 2022-10-27 08:45:42] ...optimizers/trijoint.py.151: total_norm: 12.984437 [I 2022-10-27 08:45:47] ...trap/engines/engine.py.224: train: epoch 58 | batch 570/2304 [I 2022-10-27 08:45:47] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:15:59 [I 2022-10-27 08:45:47] ...trap/engines/engine.py.229: process: 0.55391 | load: 0.00271 [I 2022-10-27 08:45:47] ...trap/engines/engine.py.230: loss: 0.17003 [I 2022-10-27 08:45:47] ...optimizers/trijoint.py.151: total_norm: 24.405138 [I 2022-10-27 08:45:53] ...trap/engines/engine.py.224: train: epoch 58 | batch 580/2304 [I 2022-10-27 08:45:53] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:15:53 [I 2022-10-27 08:45:53] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00240 [I 2022-10-27 08:45:53] ...trap/engines/engine.py.230: loss: 0.14786 [I 2022-10-27 08:45:53] ...optimizers/trijoint.py.151: total_norm: 16.535042 [I 2022-10-27 08:45:59] ...trap/engines/engine.py.224: train: epoch 58 | batch 590/2304 [I 2022-10-27 08:45:59] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:16:10 [I 2022-10-27 08:45:59] ...trap/engines/engine.py.229: process: 0.54687 | load: 0.00239 [I 2022-10-27 08:45:59] ...trap/engines/engine.py.230: loss: 0.14676 [I 2022-10-27 08:45:59] ...optimizers/trijoint.py.151: total_norm: 25.546185 [I 2022-10-27 08:46:04] ...trap/engines/engine.py.224: train: epoch 58 | batch 600/2304 [I 2022-10-27 08:46:04] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:15:39 [I 2022-10-27 08:46:04] ...trap/engines/engine.py.229: process: 0.54161 | load: 0.00247 [I 2022-10-27 08:46:04] ...trap/engines/engine.py.230: loss: 0.20936 [I 2022-10-27 08:46:04] ...optimizers/trijoint.py.151: total_norm: 19.133709 [I 2022-10-27 08:46:10] ...trap/engines/engine.py.224: train: epoch 58 | batch 610/2304 [I 2022-10-27 08:46:10] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:16:01 [I 2022-10-27 08:46:10] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00244 [I 2022-10-27 08:46:10] ...trap/engines/engine.py.230: loss: 0.11400 [I 2022-10-27 08:46:10] ...optimizers/trijoint.py.151: total_norm: 12.064735 [I 2022-10-27 08:46:15] ...trap/engines/engine.py.224: train: epoch 58 | batch 620/2304 [I 2022-10-27 08:46:15] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:15:42 [I 2022-10-27 08:46:15] ...trap/engines/engine.py.229: process: 0.56995 | load: 0.00255 [I 2022-10-27 08:46:15] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-27 08:46:15] ...optimizers/trijoint.py.151: total_norm: 15.292152 [I 2022-10-27 08:46:21] ...trap/engines/engine.py.224: train: epoch 58 | batch 630/2304 [I 2022-10-27 08:46:21] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:15:25 [I 2022-10-27 08:46:21] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00245 [I 2022-10-27 08:46:21] ...trap/engines/engine.py.230: loss: 0.17313 [I 2022-10-27 08:46:21] ...optimizers/trijoint.py.151: total_norm: 13.210387 [I 2022-10-27 08:46:26] ...trap/engines/engine.py.224: train: epoch 58 | batch 640/2304 [I 2022-10-27 08:46:26] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:19 [I 2022-10-27 08:46:26] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00242 [I 2022-10-27 08:46:26] ...trap/engines/engine.py.230: loss: 0.18423 [I 2022-10-27 08:46:26] ...optimizers/trijoint.py.151: total_norm: 22.655802 [I 2022-10-27 08:46:32] ...trap/engines/engine.py.224: train: epoch 58 | batch 650/2304 [I 2022-10-27 08:46:32] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:15 [I 2022-10-27 08:46:32] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00242 [I 2022-10-27 08:46:32] ...trap/engines/engine.py.230: loss: 0.15723 [I 2022-10-27 08:46:32] ...optimizers/trijoint.py.151: total_norm: 24.626131 [I 2022-10-27 08:46:38] ...trap/engines/engine.py.224: train: epoch 58 | batch 660/2304 [I 2022-10-27 08:46:38] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:07 [I 2022-10-27 08:46:38] ...trap/engines/engine.py.229: process: 0.54853 | load: 0.00243 [I 2022-10-27 08:46:38] ...trap/engines/engine.py.230: loss: 0.20739 [I 2022-10-27 08:46:38] ...optimizers/trijoint.py.151: total_norm: 37.482136 [I 2022-10-27 08:46:43] ...trap/engines/engine.py.224: train: epoch 58 | batch 670/2304 [I 2022-10-27 08:46:43] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:11 [I 2022-10-27 08:46:43] ...trap/engines/engine.py.229: process: 0.54557 | load: 0.00241 [I 2022-10-27 08:46:43] ...trap/engines/engine.py.230: loss: 0.16942 [I 2022-10-27 08:46:43] ...optimizers/trijoint.py.151: total_norm: 14.198271 [I 2022-10-27 08:46:49] ...trap/engines/engine.py.224: train: epoch 58 | batch 680/2304 [I 2022-10-27 08:46:49] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:14:58 [I 2022-10-27 08:46:49] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00240 [I 2022-10-27 08:46:49] ...trap/engines/engine.py.230: loss: 0.12251 [I 2022-10-27 08:46:49] ...optimizers/trijoint.py.151: total_norm: 25.389685 [I 2022-10-27 08:46:54] ...trap/engines/engine.py.224: train: epoch 58 | batch 690/2304 [I 2022-10-27 08:46:54] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:14:57 [I 2022-10-27 08:46:54] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00245 [I 2022-10-27 08:46:54] ...trap/engines/engine.py.230: loss: 0.16036 [I 2022-10-27 08:46:54] ...optimizers/trijoint.py.151: total_norm: 34.303688 [I 2022-10-27 08:47:00] ...trap/engines/engine.py.224: train: epoch 58 | batch 700/2304 [I 2022-10-27 08:47:00] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:14:46 [I 2022-10-27 08:47:00] ...trap/engines/engine.py.229: process: 0.54288 | load: 0.00239 [I 2022-10-27 08:47:00] ...trap/engines/engine.py.230: loss: 0.11777 [I 2022-10-27 08:47:00] ...optimizers/trijoint.py.151: total_norm: 8.160292 [I 2022-10-27 08:47:05] ...trap/engines/engine.py.224: train: epoch 58 | batch 710/2304 [I 2022-10-27 08:47:05] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:14:48 [I 2022-10-27 08:47:05] ...trap/engines/engine.py.229: process: 0.55010 | load: 0.00241 [I 2022-10-27 08:47:05] ...trap/engines/engine.py.230: loss: 0.10932 [I 2022-10-27 08:47:05] ...optimizers/trijoint.py.151: total_norm: 19.944683 [I 2022-10-27 08:47:11] ...trap/engines/engine.py.224: train: epoch 58 | batch 720/2304 [I 2022-10-27 08:47:11] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:14:48 [I 2022-10-27 08:47:11] ...trap/engines/engine.py.229: process: 0.56700 | load: 0.00241 [I 2022-10-27 08:47:11] ...trap/engines/engine.py.230: loss: 0.16572 [I 2022-10-27 08:47:11] ...optimizers/trijoint.py.151: total_norm: 13.802893 [I 2022-10-27 08:47:17] ...trap/engines/engine.py.224: train: epoch 58 | batch 730/2304 [I 2022-10-27 08:47:17] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:14:28 [I 2022-10-27 08:47:17] ...trap/engines/engine.py.229: process: 0.55197 | load: 0.00242 [I 2022-10-27 08:47:17] ...trap/engines/engine.py.230: loss: 0.16993 [I 2022-10-27 08:47:17] ...optimizers/trijoint.py.151: total_norm: 9.275338 [I 2022-10-27 08:47:22] ...trap/engines/engine.py.224: train: epoch 58 | batch 740/2304 [I 2022-10-27 08:47:22] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:14:21 [I 2022-10-27 08:47:22] ...trap/engines/engine.py.229: process: 0.55269 | load: 0.00240 [I 2022-10-27 08:47:22] ...trap/engines/engine.py.230: loss: 0.11171 [I 2022-10-27 08:47:22] ...optimizers/trijoint.py.151: total_norm: 13.305485 [I 2022-10-27 08:47:28] ...trap/engines/engine.py.224: train: epoch 58 | batch 750/2304 [I 2022-10-27 08:47:28] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:14:31 [I 2022-10-27 08:47:28] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00245 [I 2022-10-27 08:47:28] ...trap/engines/engine.py.230: loss: 0.11333 [I 2022-10-27 08:47:28] ...optimizers/trijoint.py.151: total_norm: 25.491467 [I 2022-10-27 08:47:33] ...trap/engines/engine.py.224: train: epoch 58 | batch 760/2304 [I 2022-10-27 08:47:33] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:20 [I 2022-10-27 08:47:33] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00251 [I 2022-10-27 08:47:33] ...trap/engines/engine.py.230: loss: 0.19635 [I 2022-10-27 08:47:33] ...optimizers/trijoint.py.151: total_norm: 27.623508 [I 2022-10-27 08:47:39] ...trap/engines/engine.py.224: train: epoch 58 | batch 770/2304 [I 2022-10-27 08:47:39] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:14:11 [I 2022-10-27 08:47:39] ...trap/engines/engine.py.229: process: 0.54659 | load: 0.00246 [I 2022-10-27 08:47:39] ...trap/engines/engine.py.230: loss: 0.14091 [I 2022-10-27 08:47:39] ...optimizers/trijoint.py.151: total_norm: 18.655615 [I 2022-10-27 08:47:45] ...trap/engines/engine.py.224: train: epoch 58 | batch 780/2304 [I 2022-10-27 08:47:45] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:07 [I 2022-10-27 08:47:45] ...trap/engines/engine.py.229: process: 0.55227 | load: 0.00244 [I 2022-10-27 08:47:45] ...trap/engines/engine.py.230: loss: 0.14034 [I 2022-10-27 08:47:45] ...optimizers/trijoint.py.151: total_norm: 19.508104 [I 2022-10-27 08:47:50] ...trap/engines/engine.py.224: train: epoch 58 | batch 790/2304 [I 2022-10-27 08:47:50] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:00 [I 2022-10-27 08:47:50] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00241 [I 2022-10-27 08:47:50] ...trap/engines/engine.py.230: loss: 0.11606 [I 2022-10-27 08:47:50] ...optimizers/trijoint.py.151: total_norm: 16.110516 [I 2022-10-27 08:47:56] ...trap/engines/engine.py.224: train: epoch 58 | batch 800/2304 [I 2022-10-27 08:47:56] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:13:55 [I 2022-10-27 08:47:56] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00242 [I 2022-10-27 08:47:56] ...trap/engines/engine.py.230: loss: 0.11371 [I 2022-10-27 08:47:56] ...optimizers/trijoint.py.151: total_norm: 11.115236 [I 2022-10-27 08:48:01] ...trap/engines/engine.py.224: train: epoch 58 | batch 810/2304 [I 2022-10-27 08:48:01] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:13:46 [I 2022-10-27 08:48:01] ...trap/engines/engine.py.229: process: 0.54738 | load: 0.00243 [I 2022-10-27 08:48:01] ...trap/engines/engine.py.230: loss: 0.16907 [I 2022-10-27 08:48:01] ...optimizers/trijoint.py.151: total_norm: 13.570739 [I 2022-10-27 08:48:07] ...trap/engines/engine.py.224: train: epoch 58 | batch 820/2304 [I 2022-10-27 08:48:07] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:13:40 [I 2022-10-27 08:48:07] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00243 [I 2022-10-27 08:48:07] ...trap/engines/engine.py.230: loss: 0.15766 [I 2022-10-27 08:48:07] ...optimizers/trijoint.py.151: total_norm: 17.943300 [I 2022-10-27 08:48:12] ...trap/engines/engine.py.224: train: epoch 58 | batch 830/2304 [I 2022-10-27 08:48:12] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:13:32 [I 2022-10-27 08:48:12] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00241 [I 2022-10-27 08:48:12] ...trap/engines/engine.py.230: loss: 0.17412 [I 2022-10-27 08:48:12] ...optimizers/trijoint.py.151: total_norm: 12.564731 [I 2022-10-27 08:48:18] ...trap/engines/engine.py.224: train: epoch 58 | batch 840/2304 [I 2022-10-27 08:48:18] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:13:37 [I 2022-10-27 08:48:18] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00248 [I 2022-10-27 08:48:18] ...trap/engines/engine.py.230: loss: 0.14248 [I 2022-10-27 08:48:18] ...optimizers/trijoint.py.151: total_norm: 19.359068 [I 2022-10-27 08:48:23] ...trap/engines/engine.py.224: train: epoch 58 | batch 850/2304 [I 2022-10-27 08:48:23] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:13:24 [I 2022-10-27 08:48:23] ...trap/engines/engine.py.229: process: 0.55851 | load: 0.00243 [I 2022-10-27 08:48:23] ...trap/engines/engine.py.230: loss: 0.16308 [I 2022-10-27 08:48:23] ...optimizers/trijoint.py.151: total_norm: 28.356314 [I 2022-10-27 08:48:29] ...trap/engines/engine.py.224: train: epoch 58 | batch 860/2304 [I 2022-10-27 08:48:29] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:13:32 [I 2022-10-27 08:48:29] ...trap/engines/engine.py.229: process: 0.59182 | load: 0.00244 [I 2022-10-27 08:48:29] ...trap/engines/engine.py.230: loss: 0.13273 [I 2022-10-27 08:48:29] ...optimizers/trijoint.py.151: total_norm: 11.543271 [I 2022-10-27 08:48:35] ...trap/engines/engine.py.224: train: epoch 58 | batch 870/2304 [I 2022-10-27 08:48:35] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:13:26 [I 2022-10-27 08:48:35] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00245 [I 2022-10-27 08:48:35] ...trap/engines/engine.py.230: loss: 0.17862 [I 2022-10-27 08:48:35] ...optimizers/trijoint.py.151: total_norm: 15.932020 [I 2022-10-27 08:48:40] ...trap/engines/engine.py.224: train: epoch 58 | batch 880/2304 [I 2022-10-27 08:48:40] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:13:08 [I 2022-10-27 08:48:40] ...trap/engines/engine.py.229: process: 0.56287 | load: 0.00248 [I 2022-10-27 08:48:40] ...trap/engines/engine.py.230: loss: 0.15106 [I 2022-10-27 08:48:40] ...optimizers/trijoint.py.151: total_norm: 33.363785 [I 2022-10-27 08:48:46] ...trap/engines/engine.py.224: train: epoch 58 | batch 890/2304 [I 2022-10-27 08:48:46] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:13:08 [I 2022-10-27 08:48:46] ...trap/engines/engine.py.229: process: 0.54973 | load: 0.00247 [I 2022-10-27 08:48:46] ...trap/engines/engine.py.230: loss: 0.22358 [I 2022-10-27 08:48:46] ...optimizers/trijoint.py.151: total_norm: 18.018177 [I 2022-10-27 08:48:51] ...trap/engines/engine.py.224: train: epoch 58 | batch 900/2304 [I 2022-10-27 08:48:51] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:12:56 [I 2022-10-27 08:48:51] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00242 [I 2022-10-27 08:48:51] ...trap/engines/engine.py.230: loss: 0.12752 [I 2022-10-27 08:48:51] ...optimizers/trijoint.py.151: total_norm: 52.245205 [I 2022-10-27 08:48:57] ...trap/engines/engine.py.224: train: epoch 58 | batch 910/2304 [I 2022-10-27 08:48:57] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:12:53 [I 2022-10-27 08:48:57] ...trap/engines/engine.py.229: process: 0.55129 | load: 0.00243 [I 2022-10-27 08:48:57] ...trap/engines/engine.py.230: loss: 0.17492 [I 2022-10-27 08:48:57] ...optimizers/trijoint.py.151: total_norm: 18.714516 [I 2022-10-27 08:49:02] ...trap/engines/engine.py.224: train: epoch 58 | batch 920/2304 [I 2022-10-27 08:49:02] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:12:45 [I 2022-10-27 08:49:02] ...trap/engines/engine.py.229: process: 0.54246 | load: 0.00246 [I 2022-10-27 08:49:02] ...trap/engines/engine.py.230: loss: 0.13254 [I 2022-10-27 08:49:02] ...optimizers/trijoint.py.151: total_norm: 12.612435 [I 2022-10-27 08:49:08] ...trap/engines/engine.py.224: train: epoch 58 | batch 930/2304 [I 2022-10-27 08:49:08] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:12:41 [I 2022-10-27 08:49:08] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00240 [I 2022-10-27 08:49:08] ...trap/engines/engine.py.230: loss: 0.13057 [I 2022-10-27 08:49:08] ...optimizers/trijoint.py.151: total_norm: 13.741798 [I 2022-10-27 08:49:14] ...trap/engines/engine.py.224: train: epoch 58 | batch 940/2304 [I 2022-10-27 08:49:14] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:12:34 [I 2022-10-27 08:49:14] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00242 [I 2022-10-27 08:49:14] ...trap/engines/engine.py.230: loss: 0.15791 [I 2022-10-27 08:49:14] ...optimizers/trijoint.py.151: total_norm: 16.330669 [I 2022-10-27 08:49:19] ...trap/engines/engine.py.224: train: epoch 58 | batch 950/2304 [I 2022-10-27 08:49:19] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:12:28 [I 2022-10-27 08:49:19] ...trap/engines/engine.py.229: process: 0.54457 | load: 0.00241 [I 2022-10-27 08:49:19] ...trap/engines/engine.py.230: loss: 0.12508 [I 2022-10-27 08:49:19] ...optimizers/trijoint.py.151: total_norm: 13.189667 [I 2022-10-27 08:49:25] ...trap/engines/engine.py.224: train: epoch 58 | batch 960/2304 [I 2022-10-27 08:49:25] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:12:15 [I 2022-10-27 08:49:25] ...trap/engines/engine.py.229: process: 0.54424 | load: 0.00239 [I 2022-10-27 08:49:25] ...trap/engines/engine.py.230: loss: 0.15539 [I 2022-10-27 08:49:25] ...optimizers/trijoint.py.151: total_norm: 12.729600 [I 2022-10-27 08:49:30] ...trap/engines/engine.py.224: train: epoch 58 | batch 970/2304 [I 2022-10-27 08:49:30] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:12:13 [I 2022-10-27 08:49:30] ...trap/engines/engine.py.229: process: 0.54733 | load: 0.00242 [I 2022-10-27 08:49:30] ...trap/engines/engine.py.230: loss: 0.15284 [I 2022-10-27 08:49:30] ...optimizers/trijoint.py.151: total_norm: 34.152615 [I 2022-10-27 08:49:36] ...trap/engines/engine.py.224: train: epoch 58 | batch 980/2304 [I 2022-10-27 08:49:36] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:15 [I 2022-10-27 08:49:36] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00242 [I 2022-10-27 08:49:36] ...trap/engines/engine.py.230: loss: 0.18515 [I 2022-10-27 08:49:36] ...optimizers/trijoint.py.151: total_norm: 11.179689 [I 2022-10-27 08:49:41] ...trap/engines/engine.py.224: train: epoch 58 | batch 990/2304 [I 2022-10-27 08:49:41] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:06 [I 2022-10-27 08:49:41] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00240 [I 2022-10-27 08:49:41] ...trap/engines/engine.py.230: loss: 0.11667 [I 2022-10-27 08:49:41] ...optimizers/trijoint.py.151: total_norm: 34.057816 [I 2022-10-27 08:49:47] ...trap/engines/engine.py.224: train: epoch 58 | batch 1000/2304 [I 2022-10-27 08:49:47] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:04 [I 2022-10-27 08:49:47] ...trap/engines/engine.py.229: process: 0.54493 | load: 0.00240 [I 2022-10-27 08:49:47] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-27 08:49:47] ...optimizers/trijoint.py.151: total_norm: 29.937336 [I 2022-10-27 08:49:52] ...trap/engines/engine.py.224: train: epoch 58 | batch 1010/2304 [I 2022-10-27 08:49:52] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:11:58 [I 2022-10-27 08:49:52] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00247 [I 2022-10-27 08:49:52] ...trap/engines/engine.py.230: loss: 0.20343 [I 2022-10-27 08:49:52] ...optimizers/trijoint.py.151: total_norm: 30.230886 [I 2022-10-27 08:49:58] ...trap/engines/engine.py.224: train: epoch 58 | batch 1020/2304 [I 2022-10-27 08:49:58] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:11:50 [I 2022-10-27 08:49:58] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00244 [I 2022-10-27 08:49:58] ...trap/engines/engine.py.230: loss: 0.18385 [I 2022-10-27 08:49:58] ...optimizers/trijoint.py.151: total_norm: 24.320688 [I 2022-10-27 08:50:04] ...trap/engines/engine.py.224: train: epoch 58 | batch 1030/2304 [I 2022-10-27 08:50:04] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:11:47 [I 2022-10-27 08:50:04] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00243 [I 2022-10-27 08:50:04] ...trap/engines/engine.py.230: loss: 0.12427 [I 2022-10-27 08:50:04] ...optimizers/trijoint.py.151: total_norm: 14.729740 [I 2022-10-27 08:50:09] ...trap/engines/engine.py.224: train: epoch 58 | batch 1040/2304 [I 2022-10-27 08:50:09] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:11:39 [I 2022-10-27 08:50:09] ...trap/engines/engine.py.229: process: 0.55419 | load: 0.00258 [I 2022-10-27 08:50:09] ...trap/engines/engine.py.230: loss: 0.18119 [I 2022-10-27 08:50:09] ...optimizers/trijoint.py.151: total_norm: 15.416788 [I 2022-10-27 08:50:15] ...trap/engines/engine.py.224: train: epoch 58 | batch 1050/2304 [I 2022-10-27 08:50:15] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:11:34 [I 2022-10-27 08:50:15] ...trap/engines/engine.py.229: process: 0.55110 | load: 0.00241 [I 2022-10-27 08:50:15] ...trap/engines/engine.py.230: loss: 0.19986 [I 2022-10-27 08:50:15] ...optimizers/trijoint.py.151: total_norm: 38.465569 [I 2022-10-27 08:50:20] ...trap/engines/engine.py.224: train: epoch 58 | batch 1060/2304 [I 2022-10-27 08:50:20] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:28 [I 2022-10-27 08:50:20] ...trap/engines/engine.py.229: process: 0.55539 | load: 0.00242 [I 2022-10-27 08:50:20] ...trap/engines/engine.py.230: loss: 0.18290 [I 2022-10-27 08:50:20] ...optimizers/trijoint.py.151: total_norm: 17.193174 [I 2022-10-27 08:50:26] ...trap/engines/engine.py.224: train: epoch 58 | batch 1070/2304 [I 2022-10-27 08:50:26] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:27 [I 2022-10-27 08:50:26] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00254 [I 2022-10-27 08:50:26] ...trap/engines/engine.py.230: loss: 0.12935 [I 2022-10-27 08:50:26] ...optimizers/trijoint.py.151: total_norm: 41.557415 [I 2022-10-27 08:50:31] ...trap/engines/engine.py.224: train: epoch 58 | batch 1080/2304 [I 2022-10-27 08:50:31] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:21 [I 2022-10-27 08:50:31] ...trap/engines/engine.py.229: process: 0.55578 | load: 0.00243 [I 2022-10-27 08:50:31] ...trap/engines/engine.py.230: loss: 0.17846 [I 2022-10-27 08:50:31] ...optimizers/trijoint.py.151: total_norm: 41.068375 [I 2022-10-27 08:50:37] ...trap/engines/engine.py.224: train: epoch 58 | batch 1090/2304 [I 2022-10-27 08:50:37] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:19 [I 2022-10-27 08:50:37] ...trap/engines/engine.py.229: process: 0.56545 | load: 0.00243 [I 2022-10-27 08:50:37] ...trap/engines/engine.py.230: loss: 0.15703 [I 2022-10-27 08:50:37] ...optimizers/trijoint.py.151: total_norm: 11.185137 [I 2022-10-27 08:50:43] ...trap/engines/engine.py.224: train: epoch 58 | batch 1100/2304 [I 2022-10-27 08:50:43] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:13 [I 2022-10-27 08:50:43] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00240 [I 2022-10-27 08:50:43] ...trap/engines/engine.py.230: loss: 0.14928 [I 2022-10-27 08:50:43] ...optimizers/trijoint.py.151: total_norm: 12.384467 [I 2022-10-27 08:50:48] ...trap/engines/engine.py.224: train: epoch 58 | batch 1110/2304 [I 2022-10-27 08:50:48] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:06 [I 2022-10-27 08:50:48] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00241 [I 2022-10-27 08:50:48] ...trap/engines/engine.py.230: loss: 0.09702 [I 2022-10-27 08:50:48] ...optimizers/trijoint.py.151: total_norm: 10.656582 [I 2022-10-27 08:50:54] ...trap/engines/engine.py.224: train: epoch 58 | batch 1120/2304 [I 2022-10-27 08:50:54] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:10:56 [I 2022-10-27 08:50:54] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00249 [I 2022-10-27 08:50:54] ...trap/engines/engine.py.230: loss: 0.11831 [I 2022-10-27 08:50:54] ...optimizers/trijoint.py.151: total_norm: 12.205748 [I 2022-10-27 08:50:59] ...trap/engines/engine.py.224: train: epoch 58 | batch 1130/2304 [I 2022-10-27 08:50:59] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:10:50 [I 2022-10-27 08:50:59] ...trap/engines/engine.py.229: process: 0.56325 | load: 0.00247 [I 2022-10-27 08:50:59] ...trap/engines/engine.py.230: loss: 0.16580 [I 2022-10-27 08:50:59] ...optimizers/trijoint.py.151: total_norm: 10.300720 [I 2022-10-27 08:51:05] ...trap/engines/engine.py.224: train: epoch 58 | batch 1140/2304 [I 2022-10-27 08:51:05] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:10:36 [I 2022-10-27 08:51:05] ...trap/engines/engine.py.229: process: 0.54055 | load: 0.00245 [I 2022-10-27 08:51:05] ...trap/engines/engine.py.230: loss: 0.16505 [I 2022-10-27 08:51:05] ...optimizers/trijoint.py.151: total_norm: 15.907651 [I 2022-10-27 08:51:10] ...trap/engines/engine.py.224: train: epoch 58 | batch 1150/2304 [I 2022-10-27 08:51:10] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:38 [I 2022-10-27 08:51:10] ...trap/engines/engine.py.229: process: 0.54724 | load: 0.00246 [I 2022-10-27 08:51:10] ...trap/engines/engine.py.230: loss: 0.16320 [I 2022-10-27 08:51:10] ...optimizers/trijoint.py.151: total_norm: 13.589491 [I 2022-10-27 08:51:16] ...trap/engines/engine.py.224: train: epoch 58 | batch 1160/2304 [I 2022-10-27 08:51:16] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:37 [I 2022-10-27 08:51:16] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00243 [I 2022-10-27 08:51:16] ...trap/engines/engine.py.230: loss: 0.13618 [I 2022-10-27 08:51:16] ...optimizers/trijoint.py.151: total_norm: 11.996819 [I 2022-10-27 08:51:21] ...trap/engines/engine.py.224: train: epoch 58 | batch 1170/2304 [I 2022-10-27 08:51:21] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:31 [I 2022-10-27 08:51:21] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00242 [I 2022-10-27 08:51:21] ...trap/engines/engine.py.230: loss: 0.15251 [I 2022-10-27 08:51:21] ...optimizers/trijoint.py.151: total_norm: 24.028149 [I 2022-10-27 08:51:27] ...trap/engines/engine.py.224: train: epoch 58 | batch 1180/2304 [I 2022-10-27 08:51:27] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:24 [I 2022-10-27 08:51:27] ...trap/engines/engine.py.229: process: 0.55829 | load: 0.00246 [I 2022-10-27 08:51:27] ...trap/engines/engine.py.230: loss: 0.15455 [I 2022-10-27 08:51:27] ...optimizers/trijoint.py.151: total_norm: 33.472424 [I 2022-10-27 08:51:33] ...trap/engines/engine.py.224: train: epoch 58 | batch 1190/2304 [I 2022-10-27 08:51:33] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:20 [I 2022-10-27 08:51:33] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00242 [I 2022-10-27 08:51:33] ...trap/engines/engine.py.230: loss: 0.14628 [I 2022-10-27 08:51:33] ...optimizers/trijoint.py.151: total_norm: 18.287769 [I 2022-10-27 08:51:38] ...trap/engines/engine.py.224: train: epoch 58 | batch 1200/2304 [I 2022-10-27 08:51:38] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:12 [I 2022-10-27 08:51:38] ...trap/engines/engine.py.229: process: 0.55347 | load: 0.00241 [I 2022-10-27 08:51:38] ...trap/engines/engine.py.230: loss: 0.20936 [I 2022-10-27 08:51:38] ...optimizers/trijoint.py.151: total_norm: 37.032387 [I 2022-10-27 08:51:44] ...trap/engines/engine.py.224: train: epoch 58 | batch 1210/2304 [I 2022-10-27 08:51:44] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:09 [I 2022-10-27 08:51:44] ...trap/engines/engine.py.229: process: 0.56373 | load: 0.00245 [I 2022-10-27 08:51:44] ...trap/engines/engine.py.230: loss: 0.10123 [I 2022-10-27 08:51:44] ...optimizers/trijoint.py.151: total_norm: 8.332401 [I 2022-10-27 08:51:49] ...trap/engines/engine.py.224: train: epoch 58 | batch 1220/2304 [I 2022-10-27 08:51:49] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:00 [I 2022-10-27 08:51:49] ...trap/engines/engine.py.229: process: 0.55121 | load: 0.00243 [I 2022-10-27 08:51:49] ...trap/engines/engine.py.230: loss: 0.18186 [I 2022-10-27 08:51:49] ...optimizers/trijoint.py.151: total_norm: 13.169751 [I 2022-10-27 08:51:55] ...trap/engines/engine.py.224: train: epoch 58 | batch 1230/2304 [I 2022-10-27 08:51:55] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:09:54 [I 2022-10-27 08:51:55] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00240 [I 2022-10-27 08:51:55] ...trap/engines/engine.py.230: loss: 0.16823 [I 2022-10-27 08:51:55] ...optimizers/trijoint.py.151: total_norm: 26.577879 [I 2022-10-27 08:52:00] ...trap/engines/engine.py.224: train: epoch 58 | batch 1240/2304 [I 2022-10-27 08:52:00] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:09:48 [I 2022-10-27 08:52:00] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00239 [I 2022-10-27 08:52:00] ...trap/engines/engine.py.230: loss: 0.16016 [I 2022-10-27 08:52:00] ...optimizers/trijoint.py.151: total_norm: 8.254823 [I 2022-10-27 08:52:06] ...trap/engines/engine.py.224: train: epoch 58 | batch 1250/2304 [I 2022-10-27 08:52:06] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:48 [I 2022-10-27 08:52:06] ...trap/engines/engine.py.229: process: 0.56583 | load: 0.00246 [I 2022-10-27 08:52:06] ...trap/engines/engine.py.230: loss: 0.13548 [I 2022-10-27 08:52:06] ...optimizers/trijoint.py.151: total_norm: 50.598206 [I 2022-10-27 08:52:12] ...trap/engines/engine.py.224: train: epoch 58 | batch 1260/2304 [I 2022-10-27 08:52:12] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:40 [I 2022-10-27 08:52:12] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00246 [I 2022-10-27 08:52:12] ...trap/engines/engine.py.230: loss: 0.14948 [I 2022-10-27 08:52:12] ...optimizers/trijoint.py.151: total_norm: 10.917118 [I 2022-10-27 08:52:17] ...trap/engines/engine.py.224: train: epoch 58 | batch 1270/2304 [I 2022-10-27 08:52:17] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:09:31 [I 2022-10-27 08:52:17] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00247 [I 2022-10-27 08:52:17] ...trap/engines/engine.py.230: loss: 0.12901 [I 2022-10-27 08:52:17] ...optimizers/trijoint.py.151: total_norm: 14.320620 [I 2022-10-27 08:52:23] ...trap/engines/engine.py.224: train: epoch 58 | batch 1280/2304 [I 2022-10-27 08:52:23] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:22 [I 2022-10-27 08:52:23] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00246 [I 2022-10-27 08:52:23] ...trap/engines/engine.py.230: loss: 0.17897 [I 2022-10-27 08:52:23] ...optimizers/trijoint.py.151: total_norm: 28.735903 [I 2022-10-27 08:52:28] ...trap/engines/engine.py.224: train: epoch 58 | batch 1290/2304 [I 2022-10-27 08:52:28] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:09:18 [I 2022-10-27 08:52:28] ...trap/engines/engine.py.229: process: 0.54827 | load: 0.00239 [I 2022-10-27 08:52:28] ...trap/engines/engine.py.230: loss: 0.14547 [I 2022-10-27 08:52:28] ...optimizers/trijoint.py.151: total_norm: 22.675070 [I 2022-10-27 08:52:34] ...trap/engines/engine.py.224: train: epoch 58 | batch 1300/2304 [I 2022-10-27 08:52:34] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:16 [I 2022-10-27 08:52:34] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00239 [I 2022-10-27 08:52:34] ...trap/engines/engine.py.230: loss: 0.18210 [I 2022-10-27 08:52:34] ...optimizers/trijoint.py.151: total_norm: 17.289320 [I 2022-10-27 08:52:39] ...trap/engines/engine.py.224: train: epoch 58 | batch 1310/2304 [I 2022-10-27 08:52:39] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:08 [I 2022-10-27 08:52:39] ...trap/engines/engine.py.229: process: 0.55283 | load: 0.00241 [I 2022-10-27 08:52:39] ...trap/engines/engine.py.230: loss: 0.20781 [I 2022-10-27 08:52:39] ...optimizers/trijoint.py.151: total_norm: 15.767129 [I 2022-10-27 08:52:45] ...trap/engines/engine.py.224: train: epoch 58 | batch 1320/2304 [I 2022-10-27 08:52:45] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:07 [I 2022-10-27 08:52:45] ...trap/engines/engine.py.229: process: 0.56504 | load: 0.00243 [I 2022-10-27 08:52:45] ...trap/engines/engine.py.230: loss: 0.13127 [I 2022-10-27 08:52:45] ...optimizers/trijoint.py.151: total_norm: 14.629189 [I 2022-10-27 08:52:50] ...trap/engines/engine.py.224: train: epoch 58 | batch 1330/2304 [I 2022-10-27 08:52:50] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:01 [I 2022-10-27 08:52:50] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00241 [I 2022-10-27 08:52:50] ...trap/engines/engine.py.230: loss: 0.15299 [I 2022-10-27 08:52:50] ...optimizers/trijoint.py.151: total_norm: 19.643774 [I 2022-10-27 08:52:56] ...trap/engines/engine.py.224: train: epoch 58 | batch 1340/2304 [I 2022-10-27 08:52:56] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:08:57 [I 2022-10-27 08:52:56] ...trap/engines/engine.py.229: process: 0.56348 | load: 0.00242 [I 2022-10-27 08:52:56] ...trap/engines/engine.py.230: loss: 0.20064 [I 2022-10-27 08:52:56] ...optimizers/trijoint.py.151: total_norm: 17.746351 [I 2022-10-27 08:53:01] ...trap/engines/engine.py.224: train: epoch 58 | batch 1350/2304 [I 2022-10-27 08:53:01] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:08:50 [I 2022-10-27 08:53:01] ...trap/engines/engine.py.229: process: 0.54896 | load: 0.00250 [I 2022-10-27 08:53:01] ...trap/engines/engine.py.230: loss: 0.15914 [I 2022-10-27 08:53:01] ...optimizers/trijoint.py.151: total_norm: 18.992504 [I 2022-10-27 08:53:07] ...trap/engines/engine.py.224: train: epoch 58 | batch 1360/2304 [I 2022-10-27 08:53:07] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:08:46 [I 2022-10-27 08:53:07] ...trap/engines/engine.py.229: process: 0.55555 | load: 0.00241 [I 2022-10-27 08:53:07] ...trap/engines/engine.py.230: loss: 0.12838 [I 2022-10-27 08:53:07] ...optimizers/trijoint.py.151: total_norm: 30.008961 [I 2022-10-27 08:53:13] ...trap/engines/engine.py.224: train: epoch 58 | batch 1370/2304 [I 2022-10-27 08:53:13] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:40 [I 2022-10-27 08:53:13] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00250 [I 2022-10-27 08:53:13] ...trap/engines/engine.py.230: loss: 0.15754 [I 2022-10-27 08:53:13] ...optimizers/trijoint.py.151: total_norm: 21.018946 [I 2022-10-27 08:53:18] ...trap/engines/engine.py.224: train: epoch 58 | batch 1380/2304 [I 2022-10-27 08:53:18] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:36 [I 2022-10-27 08:53:18] ...trap/engines/engine.py.229: process: 0.56300 | load: 0.00245 [I 2022-10-27 08:53:18] ...trap/engines/engine.py.230: loss: 0.17751 [I 2022-10-27 08:53:18] ...optimizers/trijoint.py.151: total_norm: 37.343563 [I 2022-10-27 08:53:24] ...trap/engines/engine.py.224: train: epoch 58 | batch 1390/2304 [I 2022-10-27 08:53:24] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:30 [I 2022-10-27 08:53:24] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00245 [I 2022-10-27 08:53:24] ...trap/engines/engine.py.230: loss: 0.21036 [I 2022-10-27 08:53:24] ...optimizers/trijoint.py.151: total_norm: 24.747618 [I 2022-10-27 08:53:29] ...trap/engines/engine.py.224: train: epoch 58 | batch 1400/2304 [I 2022-10-27 08:53:29] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:22 [I 2022-10-27 08:53:29] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00242 [I 2022-10-27 08:53:29] ...trap/engines/engine.py.230: loss: 0.18640 [I 2022-10-27 08:53:29] ...optimizers/trijoint.py.151: total_norm: 20.072773 [I 2022-10-27 08:53:35] ...trap/engines/engine.py.224: train: epoch 58 | batch 1410/2304 [I 2022-10-27 08:53:35] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:15 [I 2022-10-27 08:53:35] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00241 [I 2022-10-27 08:53:35] ...trap/engines/engine.py.230: loss: 0.11107 [I 2022-10-27 08:53:35] ...optimizers/trijoint.py.151: total_norm: 10.345216 [I 2022-10-27 08:53:40] ...trap/engines/engine.py.224: train: epoch 58 | batch 1420/2304 [I 2022-10-27 08:53:40] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:10 [I 2022-10-27 08:53:40] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00241 [I 2022-10-27 08:53:40] ...trap/engines/engine.py.230: loss: 0.15386 [I 2022-10-27 08:53:40] ...optimizers/trijoint.py.151: total_norm: 15.464968 [I 2022-10-27 08:53:46] ...trap/engines/engine.py.224: train: epoch 58 | batch 1430/2304 [I 2022-10-27 08:53:46] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:10 [I 2022-10-27 08:53:46] ...trap/engines/engine.py.229: process: 0.56337 | load: 0.00243 [I 2022-10-27 08:53:46] ...trap/engines/engine.py.230: loss: 0.10683 [I 2022-10-27 08:53:46] ...optimizers/trijoint.py.151: total_norm: 19.634087 [I 2022-10-27 08:53:52] ...trap/engines/engine.py.224: train: epoch 58 | batch 1440/2304 [I 2022-10-27 08:53:52] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:04 [I 2022-10-27 08:53:52] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00242 [I 2022-10-27 08:53:52] ...trap/engines/engine.py.230: loss: 0.18844 [I 2022-10-27 08:53:52] ...optimizers/trijoint.py.151: total_norm: 18.069387 [I 2022-10-27 08:53:57] ...trap/engines/engine.py.224: train: epoch 58 | batch 1450/2304 [I 2022-10-27 08:53:57] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:07:54 [I 2022-10-27 08:53:57] ...trap/engines/engine.py.229: process: 0.56366 | load: 0.00241 [I 2022-10-27 08:53:57] ...trap/engines/engine.py.230: loss: 0.13207 [I 2022-10-27 08:53:57] ...optimizers/trijoint.py.151: total_norm: 16.140266 [I 2022-10-27 08:54:03] ...trap/engines/engine.py.224: train: epoch 58 | batch 1460/2304 [I 2022-10-27 08:54:03] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:07:51 [I 2022-10-27 08:54:03] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00245 [I 2022-10-27 08:54:03] ...trap/engines/engine.py.230: loss: 0.10853 [I 2022-10-27 08:54:03] ...optimizers/trijoint.py.151: total_norm: 9.329835 [I 2022-10-27 08:54:08] ...trap/engines/engine.py.224: train: epoch 58 | batch 1470/2304 [I 2022-10-27 08:54:08] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:43 [I 2022-10-27 08:54:08] ...trap/engines/engine.py.229: process: 0.54908 | load: 0.00248 [I 2022-10-27 08:54:08] ...trap/engines/engine.py.230: loss: 0.11762 [I 2022-10-27 08:54:08] ...optimizers/trijoint.py.151: total_norm: 15.221484 [I 2022-10-27 08:54:14] ...trap/engines/engine.py.224: train: epoch 58 | batch 1480/2304 [I 2022-10-27 08:54:14] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:38 [I 2022-10-27 08:54:14] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00275 [I 2022-10-27 08:54:14] ...trap/engines/engine.py.230: loss: 0.21082 [I 2022-10-27 08:54:14] ...optimizers/trijoint.py.151: total_norm: 29.837353 [I 2022-10-27 08:54:20] ...trap/engines/engine.py.224: train: epoch 58 | batch 1490/2304 [I 2022-10-27 08:54:20] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:30 [I 2022-10-27 08:54:20] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00243 [I 2022-10-27 08:54:20] ...trap/engines/engine.py.230: loss: 0.22403 [I 2022-10-27 08:54:20] ...optimizers/trijoint.py.151: total_norm: 19.581343 [I 2022-10-27 08:54:25] ...trap/engines/engine.py.224: train: epoch 58 | batch 1500/2304 [I 2022-10-27 08:54:25] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:22 [I 2022-10-27 08:54:25] ...trap/engines/engine.py.229: process: 0.54771 | load: 0.00242 [I 2022-10-27 08:54:25] ...trap/engines/engine.py.230: loss: 0.15086 [I 2022-10-27 08:54:25] ...optimizers/trijoint.py.151: total_norm: 13.748260 [I 2022-10-27 08:54:31] ...trap/engines/engine.py.224: train: epoch 58 | batch 1510/2304 [I 2022-10-27 08:54:31] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:17 [I 2022-10-27 08:54:31] ...trap/engines/engine.py.229: process: 0.54949 | load: 0.00243 [I 2022-10-27 08:54:31] ...trap/engines/engine.py.230: loss: 0.18547 [I 2022-10-27 08:54:31] ...optimizers/trijoint.py.151: total_norm: 26.653091 [I 2022-10-27 08:54:36] ...trap/engines/engine.py.224: train: epoch 58 | batch 1520/2304 [I 2022-10-27 08:54:36] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:10 [I 2022-10-27 08:54:36] ...trap/engines/engine.py.229: process: 0.54636 | load: 0.00243 [I 2022-10-27 08:54:36] ...trap/engines/engine.py.230: loss: 0.15024 [I 2022-10-27 08:54:36] ...optimizers/trijoint.py.151: total_norm: 32.420498 [I 2022-10-27 08:54:42] ...trap/engines/engine.py.224: train: epoch 58 | batch 1530/2304 [I 2022-10-27 08:54:42] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:08 [I 2022-10-27 08:54:42] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00244 [I 2022-10-27 08:54:42] ...trap/engines/engine.py.230: loss: 0.15095 [I 2022-10-27 08:54:42] ...optimizers/trijoint.py.151: total_norm: 16.667831 [I 2022-10-27 08:54:47] ...trap/engines/engine.py.224: train: epoch 58 | batch 1540/2304 [I 2022-10-27 08:54:47] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:06 [I 2022-10-27 08:54:47] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00239 [I 2022-10-27 08:54:47] ...trap/engines/engine.py.230: loss: 0.11466 [I 2022-10-27 08:54:47] ...optimizers/trijoint.py.151: total_norm: 16.021032 [I 2022-10-27 08:54:53] ...trap/engines/engine.py.224: train: epoch 58 | batch 1550/2304 [I 2022-10-27 08:54:53] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:06:57 [I 2022-10-27 08:54:53] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00242 [I 2022-10-27 08:54:53] ...trap/engines/engine.py.230: loss: 0.14385 [I 2022-10-27 08:54:53] ...optimizers/trijoint.py.151: total_norm: 13.084004 [I 2022-10-27 08:54:58] ...trap/engines/engine.py.224: train: epoch 58 | batch 1560/2304 [I 2022-10-27 08:54:58] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:06:52 [I 2022-10-27 08:54:58] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00242 [I 2022-10-27 08:54:58] ...trap/engines/engine.py.230: loss: 0.13363 [I 2022-10-27 08:54:58] ...optimizers/trijoint.py.151: total_norm: 11.959492 [I 2022-10-27 08:55:04] ...trap/engines/engine.py.224: train: epoch 58 | batch 1570/2304 [I 2022-10-27 08:55:04] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:46 [I 2022-10-27 08:55:04] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00241 [I 2022-10-27 08:55:04] ...trap/engines/engine.py.230: loss: 0.17309 [I 2022-10-27 08:55:04] ...optimizers/trijoint.py.151: total_norm: 16.995567 [I 2022-10-27 08:55:10] ...trap/engines/engine.py.224: train: epoch 58 | batch 1580/2304 [I 2022-10-27 08:55:10] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:42 [I 2022-10-27 08:55:10] ...trap/engines/engine.py.229: process: 0.55614 | load: 0.00241 [I 2022-10-27 08:55:10] ...trap/engines/engine.py.230: loss: 0.15224 [I 2022-10-27 08:55:10] ...optimizers/trijoint.py.151: total_norm: 24.084063 [I 2022-10-27 08:55:15] ...trap/engines/engine.py.224: train: epoch 58 | batch 1590/2304 [I 2022-10-27 08:55:15] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:37 [I 2022-10-27 08:55:15] ...trap/engines/engine.py.229: process: 0.55215 | load: 0.00238 [I 2022-10-27 08:55:15] ...trap/engines/engine.py.230: loss: 0.23767 [I 2022-10-27 08:55:15] ...optimizers/trijoint.py.151: total_norm: 19.664072 [I 2022-10-27 08:55:21] ...trap/engines/engine.py.224: train: epoch 58 | batch 1600/2304 [I 2022-10-27 08:55:21] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:30 [I 2022-10-27 08:55:21] ...trap/engines/engine.py.229: process: 0.55571 | load: 0.00239 [I 2022-10-27 08:55:21] ...trap/engines/engine.py.230: loss: 0.10789 [I 2022-10-27 08:55:21] ...optimizers/trijoint.py.151: total_norm: 8.973921 [I 2022-10-27 08:55:26] ...trap/engines/engine.py.224: train: epoch 58 | batch 1610/2304 [I 2022-10-27 08:55:26] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:25 [I 2022-10-27 08:55:26] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00248 [I 2022-10-27 08:55:26] ...trap/engines/engine.py.230: loss: 0.14354 [I 2022-10-27 08:55:26] ...optimizers/trijoint.py.151: total_norm: 26.825891 [I 2022-10-27 08:55:32] ...trap/engines/engine.py.224: train: epoch 58 | batch 1620/2304 [I 2022-10-27 08:55:32] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:21 [I 2022-10-27 08:55:32] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00245 [I 2022-10-27 08:55:32] ...trap/engines/engine.py.230: loss: 0.12681 [I 2022-10-27 08:55:32] ...optimizers/trijoint.py.151: total_norm: 13.107255 [I 2022-10-27 08:55:37] ...trap/engines/engine.py.224: train: epoch 58 | batch 1630/2304 [I 2022-10-27 08:55:37] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:16 [I 2022-10-27 08:55:37] ...trap/engines/engine.py.229: process: 0.56116 | load: 0.00247 [I 2022-10-27 08:55:37] ...trap/engines/engine.py.230: loss: 0.13381 [I 2022-10-27 08:55:37] ...optimizers/trijoint.py.151: total_norm: 15.201035 [I 2022-10-27 08:55:43] ...trap/engines/engine.py.224: train: epoch 58 | batch 1640/2304 [I 2022-10-27 08:55:43] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:09 [I 2022-10-27 08:55:43] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00243 [I 2022-10-27 08:55:43] ...trap/engines/engine.py.230: loss: 0.15758 [I 2022-10-27 08:55:43] ...optimizers/trijoint.py.151: total_norm: 9.382826 [I 2022-10-27 08:55:49] ...trap/engines/engine.py.224: train: epoch 58 | batch 1650/2304 [I 2022-10-27 08:55:49] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:04 [I 2022-10-27 08:55:49] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00241 [I 2022-10-27 08:55:49] ...trap/engines/engine.py.230: loss: 0.12805 [I 2022-10-27 08:55:49] ...optimizers/trijoint.py.151: total_norm: 9.289576 [I 2022-10-27 08:55:54] ...trap/engines/engine.py.224: train: epoch 58 | batch 1660/2304 [I 2022-10-27 08:55:54] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:05:58 [I 2022-10-27 08:55:54] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00242 [I 2022-10-27 08:55:54] ...trap/engines/engine.py.230: loss: 0.13628 [I 2022-10-27 08:55:54] ...optimizers/trijoint.py.151: total_norm: 14.334218 [I 2022-10-27 08:56:00] ...trap/engines/engine.py.224: train: epoch 58 | batch 1670/2304 [I 2022-10-27 08:56:00] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:51 [I 2022-10-27 08:56:00] ...trap/engines/engine.py.229: process: 0.53319 | load: 0.00243 [I 2022-10-27 08:56:00] ...trap/engines/engine.py.230: loss: 0.17285 [I 2022-10-27 08:56:00] ...optimizers/trijoint.py.151: total_norm: 19.232109 [I 2022-10-27 08:56:05] ...trap/engines/engine.py.224: train: epoch 58 | batch 1680/2304 [I 2022-10-27 08:56:05] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:06:01 [I 2022-10-27 08:56:05] ...trap/engines/engine.py.229: process: 0.57047 | load: 0.00244 [I 2022-10-27 08:56:05] ...trap/engines/engine.py.230: loss: 0.10836 [I 2022-10-27 08:56:05] ...optimizers/trijoint.py.151: total_norm: 8.655925 [I 2022-10-27 08:56:11] ...trap/engines/engine.py.224: train: epoch 58 | batch 1690/2304 [I 2022-10-27 08:56:11] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:44 [I 2022-10-27 08:56:11] ...trap/engines/engine.py.229: process: 0.56329 | load: 0.00256 [I 2022-10-27 08:56:11] ...trap/engines/engine.py.230: loss: 0.16854 [I 2022-10-27 08:56:11] ...optimizers/trijoint.py.151: total_norm: 29.824394 [I 2022-10-27 08:56:17] ...trap/engines/engine.py.224: train: epoch 58 | batch 1700/2304 [I 2022-10-27 08:56:17] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:34 [I 2022-10-27 08:56:17] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00256 [I 2022-10-27 08:56:17] ...trap/engines/engine.py.230: loss: 0.13610 [I 2022-10-27 08:56:17] ...optimizers/trijoint.py.151: total_norm: 25.629709 [I 2022-10-27 08:56:22] ...trap/engines/engine.py.224: train: epoch 58 | batch 1710/2304 [I 2022-10-27 08:56:22] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:30 [I 2022-10-27 08:56:22] ...trap/engines/engine.py.229: process: 0.55157 | load: 0.00246 [I 2022-10-27 08:56:22] ...trap/engines/engine.py.230: loss: 0.18693 [I 2022-10-27 08:56:22] ...optimizers/trijoint.py.151: total_norm: 24.440098 [I 2022-10-27 08:56:28] ...trap/engines/engine.py.224: train: epoch 58 | batch 1720/2304 [I 2022-10-27 08:56:28] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:23 [I 2022-10-27 08:56:28] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00247 [I 2022-10-27 08:56:28] ...trap/engines/engine.py.230: loss: 0.11019 [I 2022-10-27 08:56:28] ...optimizers/trijoint.py.151: total_norm: 12.267218 [I 2022-10-27 08:56:33] ...trap/engines/engine.py.224: train: epoch 58 | batch 1730/2304 [I 2022-10-27 08:56:33] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:17 [I 2022-10-27 08:56:33] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00243 [I 2022-10-27 08:56:33] ...trap/engines/engine.py.230: loss: 0.11616 [I 2022-10-27 08:56:33] ...optimizers/trijoint.py.151: total_norm: 9.764240 [I 2022-10-27 08:56:39] ...trap/engines/engine.py.224: train: epoch 58 | batch 1740/2304 [I 2022-10-27 08:56:39] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:12 [I 2022-10-27 08:56:39] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00245 [I 2022-10-27 08:56:39] ...trap/engines/engine.py.230: loss: 0.20726 [I 2022-10-27 08:56:39] ...optimizers/trijoint.py.151: total_norm: 42.780575 [I 2022-10-27 08:56:44] ...trap/engines/engine.py.224: train: epoch 58 | batch 1750/2304 [I 2022-10-27 08:56:44] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:05 [I 2022-10-27 08:56:44] ...trap/engines/engine.py.229: process: 0.56555 | load: 0.00242 [I 2022-10-27 08:56:44] ...trap/engines/engine.py.230: loss: 0.13778 [I 2022-10-27 08:56:44] ...optimizers/trijoint.py.151: total_norm: 31.743101 [I 2022-10-27 08:56:50] ...trap/engines/engine.py.224: train: epoch 58 | batch 1760/2304 [I 2022-10-27 08:56:50] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:04 [I 2022-10-27 08:56:50] ...trap/engines/engine.py.229: process: 0.54598 | load: 0.00244 [I 2022-10-27 08:56:50] ...trap/engines/engine.py.230: loss: 0.16097 [I 2022-10-27 08:56:50] ...optimizers/trijoint.py.151: total_norm: 18.971558 [I 2022-10-27 08:56:56] ...trap/engines/engine.py.224: train: epoch 58 | batch 1770/2304 [I 2022-10-27 08:56:56] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:04:57 [I 2022-10-27 08:56:56] ...trap/engines/engine.py.229: process: 0.56212 | load: 0.00248 [I 2022-10-27 08:56:56] ...trap/engines/engine.py.230: loss: 0.17674 [I 2022-10-27 08:56:56] ...optimizers/trijoint.py.151: total_norm: 12.160769 [I 2022-10-27 08:57:01] ...trap/engines/engine.py.224: train: epoch 58 | batch 1780/2304 [I 2022-10-27 08:57:01] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:04:52 [I 2022-10-27 08:57:01] ...trap/engines/engine.py.229: process: 0.57474 | load: 0.00280 [I 2022-10-27 08:57:01] ...trap/engines/engine.py.230: loss: 0.13640 [I 2022-10-27 08:57:01] ...optimizers/trijoint.py.151: total_norm: 10.035147 [I 2022-10-27 08:57:07] ...trap/engines/engine.py.224: train: epoch 58 | batch 1790/2304 [I 2022-10-27 08:57:07] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:45 [I 2022-10-27 08:57:07] ...trap/engines/engine.py.229: process: 0.55911 | load: 0.00250 [I 2022-10-27 08:57:07] ...trap/engines/engine.py.230: loss: 0.19062 [I 2022-10-27 08:57:07] ...optimizers/trijoint.py.151: total_norm: 19.353411 [I 2022-10-27 08:57:12] ...trap/engines/engine.py.224: train: epoch 58 | batch 1800/2304 [I 2022-10-27 08:57:12] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:41 [I 2022-10-27 08:57:12] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00244 [I 2022-10-27 08:57:12] ...trap/engines/engine.py.230: loss: 0.13409 [I 2022-10-27 08:57:12] ...optimizers/trijoint.py.151: total_norm: 12.987362 [I 2022-10-27 08:57:18] ...trap/engines/engine.py.224: train: epoch 58 | batch 1810/2304 [I 2022-10-27 08:57:18] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:34 [I 2022-10-27 08:57:18] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00246 [I 2022-10-27 08:57:18] ...trap/engines/engine.py.230: loss: 0.12980 [I 2022-10-27 08:57:18] ...optimizers/trijoint.py.151: total_norm: 9.242129 [I 2022-10-27 08:57:23] ...trap/engines/engine.py.224: train: epoch 58 | batch 1820/2304 [I 2022-10-27 08:57:23] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:27 [I 2022-10-27 08:57:23] ...trap/engines/engine.py.229: process: 0.53642 | load: 0.00243 [I 2022-10-27 08:57:23] ...trap/engines/engine.py.230: loss: 0.17891 [I 2022-10-27 08:57:23] ...optimizers/trijoint.py.151: total_norm: 61.258812 [I 2022-10-27 08:57:29] ...trap/engines/engine.py.224: train: epoch 58 | batch 1830/2304 [I 2022-10-27 08:57:29] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:27 [I 2022-10-27 08:57:29] ...trap/engines/engine.py.229: process: 0.55258 | load: 0.00245 [I 2022-10-27 08:57:29] ...trap/engines/engine.py.230: loss: 0.18916 [I 2022-10-27 08:57:29] ...optimizers/trijoint.py.151: total_norm: 10.612127 [I 2022-10-27 08:57:35] ...trap/engines/engine.py.224: train: epoch 58 | batch 1840/2304 [I 2022-10-27 08:57:35] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:15 [I 2022-10-27 08:57:35] ...trap/engines/engine.py.229: process: 0.54185 | load: 0.00244 [I 2022-10-27 08:57:35] ...trap/engines/engine.py.230: loss: 0.18802 [I 2022-10-27 08:57:35] ...optimizers/trijoint.py.151: total_norm: 17.799973 [I 2022-10-27 08:57:40] ...trap/engines/engine.py.224: train: epoch 58 | batch 1850/2304 [I 2022-10-27 08:57:40] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:11 [I 2022-10-27 08:57:40] ...trap/engines/engine.py.229: process: 0.54980 | load: 0.00241 [I 2022-10-27 08:57:40] ...trap/engines/engine.py.230: loss: 0.18049 [I 2022-10-27 08:57:40] ...optimizers/trijoint.py.151: total_norm: 13.711423 [I 2022-10-27 08:57:46] ...trap/engines/engine.py.224: train: epoch 58 | batch 1860/2304 [I 2022-10-27 08:57:46] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:07 [I 2022-10-27 08:57:46] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00259 [I 2022-10-27 08:57:46] ...trap/engines/engine.py.230: loss: 0.20031 [I 2022-10-27 08:57:46] ...optimizers/trijoint.py.151: total_norm: 12.787619 [I 2022-10-27 08:57:51] ...trap/engines/engine.py.224: train: epoch 58 | batch 1870/2304 [I 2022-10-27 08:57:51] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:01 [I 2022-10-27 08:57:51] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00247 [I 2022-10-27 08:57:51] ...trap/engines/engine.py.230: loss: 0.09836 [I 2022-10-27 08:57:51] ...optimizers/trijoint.py.151: total_norm: 11.473418 [I 2022-10-27 08:57:57] ...trap/engines/engine.py.224: train: epoch 58 | batch 1880/2304 [I 2022-10-27 08:57:57] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:03:55 [I 2022-10-27 08:57:57] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00249 [I 2022-10-27 08:57:57] ...trap/engines/engine.py.230: loss: 0.14319 [I 2022-10-27 08:57:57] ...optimizers/trijoint.py.151: total_norm: 9.549806 [I 2022-10-27 08:58:02] ...trap/engines/engine.py.224: train: epoch 58 | batch 1890/2304 [I 2022-10-27 08:58:02] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:50 [I 2022-10-27 08:58:02] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00245 [I 2022-10-27 08:58:02] ...trap/engines/engine.py.230: loss: 0.22237 [I 2022-10-27 08:58:02] ...optimizers/trijoint.py.151: total_norm: 31.301813 [I 2022-10-27 08:58:08] ...trap/engines/engine.py.224: train: epoch 58 | batch 1900/2304 [I 2022-10-27 08:58:08] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:03:43 [I 2022-10-27 08:58:08] ...trap/engines/engine.py.229: process: 0.55054 | load: 0.00245 [I 2022-10-27 08:58:08] ...trap/engines/engine.py.230: loss: 0.12773 [I 2022-10-27 08:58:08] ...optimizers/trijoint.py.151: total_norm: 20.328350 [I 2022-10-27 08:58:13] ...trap/engines/engine.py.224: train: epoch 58 | batch 1910/2304 [I 2022-10-27 08:58:13] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:37 [I 2022-10-27 08:58:13] ...trap/engines/engine.py.229: process: 0.54927 | load: 0.00244 [I 2022-10-27 08:58:13] ...trap/engines/engine.py.230: loss: 0.16309 [I 2022-10-27 08:58:13] ...optimizers/trijoint.py.151: total_norm: 18.219995 [I 2022-10-27 08:58:19] ...trap/engines/engine.py.224: train: epoch 58 | batch 1920/2304 [I 2022-10-27 08:58:19] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:33 [I 2022-10-27 08:58:19] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00244 [I 2022-10-27 08:58:19] ...trap/engines/engine.py.230: loss: 0.16203 [I 2022-10-27 08:58:19] ...optimizers/trijoint.py.151: total_norm: 14.148050 [I 2022-10-27 08:58:25] ...trap/engines/engine.py.224: train: epoch 58 | batch 1930/2304 [I 2022-10-27 08:58:25] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:27 [I 2022-10-27 08:58:25] ...trap/engines/engine.py.229: process: 0.55168 | load: 0.00241 [I 2022-10-27 08:58:25] ...trap/engines/engine.py.230: loss: 0.15011 [I 2022-10-27 08:58:25] ...optimizers/trijoint.py.151: total_norm: 11.720832 [I 2022-10-27 08:58:30] ...trap/engines/engine.py.224: train: epoch 58 | batch 1940/2304 [I 2022-10-27 08:58:30] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:23 [I 2022-10-27 08:58:30] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00242 [I 2022-10-27 08:58:30] ...trap/engines/engine.py.230: loss: 0.15151 [I 2022-10-27 08:58:30] ...optimizers/trijoint.py.151: total_norm: 13.421196 [I 2022-10-27 08:58:36] ...trap/engines/engine.py.224: train: epoch 58 | batch 1950/2304 [I 2022-10-27 08:58:36] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:16 [I 2022-10-27 08:58:36] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00243 [I 2022-10-27 08:58:36] ...trap/engines/engine.py.230: loss: 0.11659 [I 2022-10-27 08:58:36] ...optimizers/trijoint.py.151: total_norm: 10.737529 [I 2022-10-27 08:58:41] ...trap/engines/engine.py.224: train: epoch 58 | batch 1960/2304 [I 2022-10-27 08:58:41] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:11 [I 2022-10-27 08:58:41] ...trap/engines/engine.py.229: process: 0.56416 | load: 0.00243 [I 2022-10-27 08:58:41] ...trap/engines/engine.py.230: loss: 0.21040 [I 2022-10-27 08:58:41] ...optimizers/trijoint.py.151: total_norm: 8.694670 [I 2022-10-27 08:58:47] ...trap/engines/engine.py.224: train: epoch 58 | batch 1970/2304 [I 2022-10-27 08:58:47] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:06 [I 2022-10-27 08:58:47] ...trap/engines/engine.py.229: process: 0.57045 | load: 0.00243 [I 2022-10-27 08:58:47] ...trap/engines/engine.py.230: loss: 0.16969 [I 2022-10-27 08:58:47] ...optimizers/trijoint.py.151: total_norm: 25.278061 [I 2022-10-27 08:58:53] ...trap/engines/engine.py.224: train: epoch 58 | batch 1980/2304 [I 2022-10-27 08:58:53] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:01 [I 2022-10-27 08:58:53] ...trap/engines/engine.py.229: process: 0.56541 | load: 0.00249 [I 2022-10-27 08:58:53] ...trap/engines/engine.py.230: loss: 0.18313 [I 2022-10-27 08:58:53] ...optimizers/trijoint.py.151: total_norm: 18.729565 [I 2022-10-27 08:58:58] ...trap/engines/engine.py.224: train: epoch 58 | batch 1990/2304 [I 2022-10-27 08:58:58] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:02:55 [I 2022-10-27 08:58:58] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00241 [I 2022-10-27 08:58:58] ...trap/engines/engine.py.230: loss: 0.14474 [I 2022-10-27 08:58:58] ...optimizers/trijoint.py.151: total_norm: 17.165016 [I 2022-10-27 08:59:04] ...trap/engines/engine.py.224: train: epoch 58 | batch 2000/2304 [I 2022-10-27 08:59:04] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:47 [I 2022-10-27 08:59:04] ...trap/engines/engine.py.229: process: 0.54765 | load: 0.00252 [I 2022-10-27 08:59:04] ...trap/engines/engine.py.230: loss: 0.11370 [I 2022-10-27 08:59:04] ...optimizers/trijoint.py.151: total_norm: 13.388242 [I 2022-10-27 08:59:09] ...trap/engines/engine.py.224: train: epoch 58 | batch 2010/2304 [I 2022-10-27 08:59:09] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:02:43 [I 2022-10-27 08:59:09] ...trap/engines/engine.py.229: process: 0.54542 | load: 0.00243 [I 2022-10-27 08:59:09] ...trap/engines/engine.py.230: loss: 0.15502 [I 2022-10-27 08:59:09] ...optimizers/trijoint.py.151: total_norm: 11.735768 [I 2022-10-27 08:59:15] ...trap/engines/engine.py.224: train: epoch 58 | batch 2020/2304 [I 2022-10-27 08:59:15] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:37 [I 2022-10-27 08:59:15] ...trap/engines/engine.py.229: process: 0.54616 | load: 0.00244 [I 2022-10-27 08:59:15] ...trap/engines/engine.py.230: loss: 0.15469 [I 2022-10-27 08:59:15] ...optimizers/trijoint.py.151: total_norm: 11.919996 [I 2022-10-27 08:59:20] ...trap/engines/engine.py.224: train: epoch 58 | batch 2030/2304 [I 2022-10-27 08:59:20] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:33 [I 2022-10-27 08:59:20] ...trap/engines/engine.py.229: process: 0.56369 | load: 0.00243 [I 2022-10-27 08:59:20] ...trap/engines/engine.py.230: loss: 0.15161 [I 2022-10-27 08:59:20] ...optimizers/trijoint.py.151: total_norm: 38.470932 [I 2022-10-27 08:59:26] ...trap/engines/engine.py.224: train: epoch 58 | batch 2040/2304 [I 2022-10-27 08:59:26] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:27 [I 2022-10-27 08:59:26] ...trap/engines/engine.py.229: process: 0.54882 | load: 0.00241 [I 2022-10-27 08:59:26] ...trap/engines/engine.py.230: loss: 0.16179 [I 2022-10-27 08:59:26] ...optimizers/trijoint.py.151: total_norm: 16.911119 [I 2022-10-27 08:59:32] ...trap/engines/engine.py.224: train: epoch 58 | batch 2050/2304 [I 2022-10-27 08:59:32] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:23 [I 2022-10-27 08:59:32] ...trap/engines/engine.py.229: process: 0.56290 | load: 0.00244 [I 2022-10-27 08:59:32] ...trap/engines/engine.py.230: loss: 0.12752 [I 2022-10-27 08:59:32] ...optimizers/trijoint.py.151: total_norm: 21.663177 [I 2022-10-27 08:59:37] ...trap/engines/engine.py.224: train: epoch 58 | batch 2060/2304 [I 2022-10-27 08:59:37] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:15 [I 2022-10-27 08:59:37] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00249 [I 2022-10-27 08:59:37] ...trap/engines/engine.py.230: loss: 0.21925 [I 2022-10-27 08:59:37] ...optimizers/trijoint.py.151: total_norm: 12.673952 [I 2022-10-27 08:59:43] ...trap/engines/engine.py.224: train: epoch 58 | batch 2070/2304 [I 2022-10-27 08:59:43] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:09 [I 2022-10-27 08:59:43] ...trap/engines/engine.py.229: process: 0.56427 | load: 0.00242 [I 2022-10-27 08:59:43] ...trap/engines/engine.py.230: loss: 0.17134 [I 2022-10-27 08:59:43] ...optimizers/trijoint.py.151: total_norm: 20.505705 [I 2022-10-27 08:59:48] ...trap/engines/engine.py.224: train: epoch 58 | batch 2080/2304 [I 2022-10-27 08:59:48] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:04 [I 2022-10-27 08:59:48] ...trap/engines/engine.py.229: process: 0.54571 | load: 0.00242 [I 2022-10-27 08:59:48] ...trap/engines/engine.py.230: loss: 0.11875 [I 2022-10-27 08:59:48] ...optimizers/trijoint.py.151: total_norm: 14.835335 [I 2022-10-27 08:59:54] ...trap/engines/engine.py.224: train: epoch 58 | batch 2090/2304 [I 2022-10-27 08:59:54] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:01:59 [I 2022-10-27 08:59:54] ...trap/engines/engine.py.229: process: 0.56840 | load: 0.00243 [I 2022-10-27 08:59:54] ...trap/engines/engine.py.230: loss: 0.12795 [I 2022-10-27 08:59:54] ...optimizers/trijoint.py.151: total_norm: 20.795933 [I 2022-10-27 08:59:59] ...trap/engines/engine.py.224: train: epoch 58 | batch 2100/2304 [I 2022-10-27 08:59:59] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:01:53 [I 2022-10-27 08:59:59] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00245 [I 2022-10-27 08:59:59] ...trap/engines/engine.py.230: loss: 0.11477 [I 2022-10-27 08:59:59] ...optimizers/trijoint.py.151: total_norm: 7.490132 [I 2022-10-27 09:00:05] ...trap/engines/engine.py.224: train: epoch 58 | batch 2110/2304 [I 2022-10-27 09:00:05] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:48 [I 2022-10-27 09:00:05] ...trap/engines/engine.py.229: process: 0.54908 | load: 0.00250 [I 2022-10-27 09:00:05] ...trap/engines/engine.py.230: loss: 0.12651 [I 2022-10-27 09:00:05] ...optimizers/trijoint.py.151: total_norm: 13.059843 [I 2022-10-27 09:00:11] ...trap/engines/engine.py.224: train: epoch 58 | batch 2120/2304 [I 2022-10-27 09:00:11] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:01:42 [I 2022-10-27 09:00:11] ...trap/engines/engine.py.229: process: 0.54974 | load: 0.00255 [I 2022-10-27 09:00:11] ...trap/engines/engine.py.230: loss: 0.16241 [I 2022-10-27 09:00:11] ...optimizers/trijoint.py.151: total_norm: 12.152762 [I 2022-10-27 09:00:16] ...trap/engines/engine.py.224: train: epoch 58 | batch 2130/2304 [I 2022-10-27 09:00:16] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:36 [I 2022-10-27 09:00:16] ...trap/engines/engine.py.229: process: 0.55378 | load: 0.00251 [I 2022-10-27 09:00:16] ...trap/engines/engine.py.230: loss: 0.17573 [I 2022-10-27 09:00:16] ...optimizers/trijoint.py.151: total_norm: 18.091534 [I 2022-10-27 09:00:22] ...trap/engines/engine.py.224: train: epoch 58 | batch 2140/2304 [I 2022-10-27 09:00:22] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:30 [I 2022-10-27 09:00:22] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00245 [I 2022-10-27 09:00:22] ...trap/engines/engine.py.230: loss: 0.13923 [I 2022-10-27 09:00:22] ...optimizers/trijoint.py.151: total_norm: 20.569918 [I 2022-10-27 09:00:27] ...trap/engines/engine.py.224: train: epoch 58 | batch 2150/2304 [I 2022-10-27 09:00:27] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:25 [I 2022-10-27 09:00:27] ...trap/engines/engine.py.229: process: 0.55458 | load: 0.00243 [I 2022-10-27 09:00:27] ...trap/engines/engine.py.230: loss: 0.11954 [I 2022-10-27 09:00:27] ...optimizers/trijoint.py.151: total_norm: 13.973683 [I 2022-10-27 09:00:33] ...trap/engines/engine.py.224: train: epoch 58 | batch 2160/2304 [I 2022-10-27 09:00:33] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:20 [I 2022-10-27 09:00:33] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00242 [I 2022-10-27 09:00:33] ...trap/engines/engine.py.230: loss: 0.11546 [I 2022-10-27 09:00:33] ...optimizers/trijoint.py.151: total_norm: 8.683443 [I 2022-10-27 09:00:38] ...trap/engines/engine.py.224: train: epoch 58 | batch 2170/2304 [I 2022-10-27 09:00:38] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:14 [I 2022-10-27 09:00:38] ...trap/engines/engine.py.229: process: 0.55376 | load: 0.00241 [I 2022-10-27 09:00:38] ...trap/engines/engine.py.230: loss: 0.19413 [I 2022-10-27 09:00:38] ...optimizers/trijoint.py.151: total_norm: 21.798883 [I 2022-10-27 09:00:44] ...trap/engines/engine.py.224: train: epoch 58 | batch 2180/2304 [I 2022-10-27 09:00:44] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:08 [I 2022-10-27 09:00:44] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00239 [I 2022-10-27 09:00:44] ...trap/engines/engine.py.230: loss: 0.21593 [I 2022-10-27 09:00:44] ...optimizers/trijoint.py.151: total_norm: 29.940393 [I 2022-10-27 09:00:50] ...trap/engines/engine.py.224: train: epoch 58 | batch 2190/2304 [I 2022-10-27 09:00:50] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:03 [I 2022-10-27 09:00:50] ...trap/engines/engine.py.229: process: 0.55583 | load: 0.00243 [I 2022-10-27 09:00:50] ...trap/engines/engine.py.230: loss: 0.18279 [I 2022-10-27 09:00:50] ...optimizers/trijoint.py.151: total_norm: 26.122747 [I 2022-10-27 09:00:55] ...trap/engines/engine.py.224: train: epoch 58 | batch 2200/2304 [I 2022-10-27 09:00:55] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:00:57 [I 2022-10-27 09:00:55] ...trap/engines/engine.py.229: process: 0.55400 | load: 0.00242 [I 2022-10-27 09:00:55] ...trap/engines/engine.py.230: loss: 0.12433 [I 2022-10-27 09:00:55] ...optimizers/trijoint.py.151: total_norm: 15.851807 [I 2022-10-27 09:01:01] ...trap/engines/engine.py.224: train: epoch 58 | batch 2210/2304 [I 2022-10-27 09:01:01] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:00:52 [I 2022-10-27 09:01:01] ...trap/engines/engine.py.229: process: 0.54316 | load: 0.00242 [I 2022-10-27 09:01:01] ...trap/engines/engine.py.230: loss: 0.21971 [I 2022-10-27 09:01:01] ...optimizers/trijoint.py.151: total_norm: 16.886124 [I 2022-10-27 09:01:06] ...trap/engines/engine.py.224: train: epoch 58 | batch 2220/2304 [I 2022-10-27 09:01:06] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:00:46 [I 2022-10-27 09:01:06] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00241 [I 2022-10-27 09:01:06] ...trap/engines/engine.py.230: loss: 0.15084 [I 2022-10-27 09:01:06] ...optimizers/trijoint.py.151: total_norm: 9.882042 [I 2022-10-27 09:01:12] ...trap/engines/engine.py.224: train: epoch 58 | batch 2230/2304 [I 2022-10-27 09:01:12] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:00:41 [I 2022-10-27 09:01:12] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00243 [I 2022-10-27 09:01:12] ...trap/engines/engine.py.230: loss: 0.14664 [I 2022-10-27 09:01:12] ...optimizers/trijoint.py.151: total_norm: 11.273548 [I 2022-10-27 09:01:17] ...trap/engines/engine.py.224: train: epoch 58 | batch 2240/2304 [I 2022-10-27 09:01:17] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:35 [I 2022-10-27 09:01:17] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00254 [I 2022-10-27 09:01:17] ...trap/engines/engine.py.230: loss: 0.18970 [I 2022-10-27 09:01:17] ...optimizers/trijoint.py.151: total_norm: 69.961823 [I 2022-10-27 09:01:23] ...trap/engines/engine.py.224: train: epoch 58 | batch 2250/2304 [I 2022-10-27 09:01:23] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:29 [I 2022-10-27 09:01:23] ...trap/engines/engine.py.229: process: 0.55003 | load: 0.00249 [I 2022-10-27 09:01:23] ...trap/engines/engine.py.230: loss: 0.27165 [I 2022-10-27 09:01:23] ...optimizers/trijoint.py.151: total_norm: 20.235516 [I 2022-10-27 09:01:29] ...trap/engines/engine.py.224: train: epoch 58 | batch 2260/2304 [I 2022-10-27 09:01:29] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:24 [I 2022-10-27 09:01:29] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00243 [I 2022-10-27 09:01:29] ...trap/engines/engine.py.230: loss: 0.14209 [I 2022-10-27 09:01:29] ...optimizers/trijoint.py.151: total_norm: 13.542171 [I 2022-10-27 09:01:34] ...trap/engines/engine.py.224: train: epoch 58 | batch 2270/2304 [I 2022-10-27 09:01:34] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:18 [I 2022-10-27 09:01:34] ...trap/engines/engine.py.229: process: 0.54853 | load: 0.00244 [I 2022-10-27 09:01:34] ...trap/engines/engine.py.230: loss: 0.19937 [I 2022-10-27 09:01:34] ...optimizers/trijoint.py.151: total_norm: 21.424326 [I 2022-10-27 09:01:40] ...trap/engines/engine.py.224: train: epoch 58 | batch 2280/2304 [I 2022-10-27 09:01:40] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:13 [I 2022-10-27 09:01:40] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00246 [I 2022-10-27 09:01:40] ...trap/engines/engine.py.230: loss: 0.11166 [I 2022-10-27 09:01:40] ...optimizers/trijoint.py.151: total_norm: 13.712509 [I 2022-10-27 09:01:45] ...trap/engines/engine.py.224: train: epoch 58 | batch 2290/2304 [I 2022-10-27 09:01:45] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:07 [I 2022-10-27 09:01:45] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00243 [I 2022-10-27 09:01:45] ...trap/engines/engine.py.230: loss: 0.13780 [I 2022-10-27 09:01:45] ...optimizers/trijoint.py.151: total_norm: 18.464434 [I 2022-10-27 09:01:51] ...trap/engines/engine.py.224: train: epoch 58 | batch 2300/2304 [I 2022-10-27 09:01:51] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:02 [I 2022-10-27 09:01:51] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00014 [I 2022-10-27 09:01:51] ...trap/engines/engine.py.230: loss: 0.11762 [I 2022-10-27 09:01:51] ...optimizers/trijoint.py.151: total_norm: 10.592303 [I 2022-10-27 09:01:53] ...trap/engines/engine.py.224: train: epoch 58 | batch 2304/2304 [I 2022-10-27 09:01:53] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:00 [I 2022-10-27 09:01:53] ...trap/engines/engine.py.229: process: 0.54537 | load: 0.00014 [I 2022-10-27 09:01:53] ...trap/engines/engine.py.230: loss: 0.15865 [I 2022-10-27 09:01:53] ...optimizers/trijoint.py.151: total_norm: 17.126549 [S 2022-10-27 09:01:54] ...trap/engines/engine.py.129: train_epoch.epoch: 58 [S 2022-10-27 09:01:54] ...trap/engines/engine.py.129: train_epoch.loss: 0.15672235862875192 [I 2022-10-27 09:01:54] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 09:01:55] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 58 [I 2022-10-27 09:01:55] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 09:01:57] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 09:01:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 09:01:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 09:01:59] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 09:01:59] ...trap/engines/engine.py.321: eval: epoch 58 | batch 0/510 [I 2022-10-27 09:01:59] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:35:08 [I 2022-10-27 09:01:59] ...trap/engines/engine.py.326: process: 4.13442 | load: 2.97042 [I 2022-10-27 09:02:02] ...trap/engines/engine.py.321: eval: epoch 58 | batch 10/510 [I 2022-10-27 09:02:02] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:10 [I 2022-10-27 09:02:02] ...trap/engines/engine.py.326: process: 0.31795 | load: 0.00017 [I 2022-10-27 09:02:06] ...trap/engines/engine.py.321: eval: epoch 58 | batch 20/510 [I 2022-10-27 09:02:06] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:04 [I 2022-10-27 09:02:06] ...trap/engines/engine.py.326: process: 0.32938 | load: 0.00026 [I 2022-10-27 09:02:09] ...trap/engines/engine.py.321: eval: epoch 58 | batch 30/510 [I 2022-10-27 09:02:09] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:44 [I 2022-10-27 09:02:09] ...trap/engines/engine.py.326: process: 0.32840 | load: 0.00017 [I 2022-10-27 09:02:13] ...trap/engines/engine.py.321: eval: epoch 58 | batch 40/510 [I 2022-10-27 09:02:13] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:38 [I 2022-10-27 09:02:13] ...trap/engines/engine.py.326: process: 0.32878 | load: 0.00018 [I 2022-10-27 09:02:16] ...trap/engines/engine.py.321: eval: epoch 58 | batch 50/510 [I 2022-10-27 09:02:16] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:33 [I 2022-10-27 09:02:16] ...trap/engines/engine.py.326: process: 0.32144 | load: 0.00017 [I 2022-10-27 09:02:19] ...trap/engines/engine.py.321: eval: epoch 58 | batch 60/510 [I 2022-10-27 09:02:19] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:31 [I 2022-10-27 09:02:19] ...trap/engines/engine.py.326: process: 0.33098 | load: 0.00017 [I 2022-10-27 09:02:23] ...trap/engines/engine.py.321: eval: epoch 58 | batch 70/510 [I 2022-10-27 09:02:23] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:25 [I 2022-10-27 09:02:23] ...trap/engines/engine.py.326: process: 0.32985 | load: 0.00022 [I 2022-10-27 09:02:26] ...trap/engines/engine.py.321: eval: epoch 58 | batch 80/510 [I 2022-10-27 09:02:26] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:22 [I 2022-10-27 09:02:26] ...trap/engines/engine.py.326: process: 0.32438 | load: 0.00016 [I 2022-10-27 09:02:29] ...trap/engines/engine.py.321: eval: epoch 58 | batch 90/510 [I 2022-10-27 09:02:29] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:19 [I 2022-10-27 09:02:29] ...trap/engines/engine.py.326: process: 0.32330 | load: 0.00017 [I 2022-10-27 09:02:32] ...trap/engines/engine.py.321: eval: epoch 58 | batch 100/510 [I 2022-10-27 09:02:32] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:10 [I 2022-10-27 09:02:32] ...trap/engines/engine.py.326: process: 0.28134 | load: 0.00016 [I 2022-10-27 09:02:35] ...trap/engines/engine.py.321: eval: epoch 58 | batch 110/510 [I 2022-10-27 09:02:35] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:56 [I 2022-10-27 09:02:35] ...trap/engines/engine.py.326: process: 0.28510 | load: 0.00021 [I 2022-10-27 09:02:38] ...trap/engines/engine.py.321: eval: epoch 58 | batch 120/510 [I 2022-10-27 09:02:38] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:52 [I 2022-10-27 09:02:38] ...trap/engines/engine.py.326: process: 0.28624 | load: 0.00015 [I 2022-10-27 09:02:41] ...trap/engines/engine.py.321: eval: epoch 58 | batch 130/510 [I 2022-10-27 09:02:41] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:49 [I 2022-10-27 09:02:41] ...trap/engines/engine.py.326: process: 0.28916 | load: 0.00023 [I 2022-10-27 09:02:44] ...trap/engines/engine.py.321: eval: epoch 58 | batch 140/510 [I 2022-10-27 09:02:44] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:46 [I 2022-10-27 09:02:44] ...trap/engines/engine.py.326: process: 0.28628 | load: 0.00014 [I 2022-10-27 09:02:47] ...trap/engines/engine.py.321: eval: epoch 58 | batch 150/510 [I 2022-10-27 09:02:47] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:42 [I 2022-10-27 09:02:47] ...trap/engines/engine.py.326: process: 0.28125 | load: 0.00024 [I 2022-10-27 09:02:50] ...trap/engines/engine.py.321: eval: epoch 58 | batch 160/510 [I 2022-10-27 09:02:50] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:40 [I 2022-10-27 09:02:50] ...trap/engines/engine.py.326: process: 0.29063 | load: 0.00015 [I 2022-10-27 09:02:53] ...trap/engines/engine.py.321: eval: epoch 58 | batch 170/510 [I 2022-10-27 09:02:53] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-27 09:02:53] ...trap/engines/engine.py.326: process: 0.28946 | load: 0.00014 [I 2022-10-27 09:02:55] ...trap/engines/engine.py.321: eval: epoch 58 | batch 180/510 [I 2022-10-27 09:02:55] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 09:02:55] ...trap/engines/engine.py.326: process: 0.28787 | load: 0.00016 [I 2022-10-27 09:02:58] ...trap/engines/engine.py.321: eval: epoch 58 | batch 190/510 [I 2022-10-27 09:02:58] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:31 [I 2022-10-27 09:02:58] ...trap/engines/engine.py.326: process: 0.28500 | load: 0.00018 [I 2022-10-27 09:03:01] ...trap/engines/engine.py.321: eval: epoch 58 | batch 200/510 [I 2022-10-27 09:03:01] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:28 [I 2022-10-27 09:03:01] ...trap/engines/engine.py.326: process: 0.28688 | load: 0.00016 [I 2022-10-27 09:03:04] ...trap/engines/engine.py.321: eval: epoch 58 | batch 210/510 [I 2022-10-27 09:03:04] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 09:03:04] ...trap/engines/engine.py.326: process: 0.29052 | load: 0.00024 [I 2022-10-27 09:03:07] ...trap/engines/engine.py.321: eval: epoch 58 | batch 220/510 [I 2022-10-27 09:03:07] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:23 [I 2022-10-27 09:03:07] ...trap/engines/engine.py.326: process: 0.28740 | load: 0.00015 [I 2022-10-27 09:03:10] ...trap/engines/engine.py.321: eval: epoch 58 | batch 230/510 [I 2022-10-27 09:03:10] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-27 09:03:10] ...trap/engines/engine.py.326: process: 0.28407 | load: 0.00024 [I 2022-10-27 09:03:13] ...trap/engines/engine.py.321: eval: epoch 58 | batch 240/510 [I 2022-10-27 09:03:13] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:16 [I 2022-10-27 09:03:13] ...trap/engines/engine.py.326: process: 0.28995 | load: 0.00018 [I 2022-10-27 09:03:16] ...trap/engines/engine.py.321: eval: epoch 58 | batch 250/510 [I 2022-10-27 09:03:16] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-27 09:03:16] ...trap/engines/engine.py.326: process: 0.29110 | load: 0.00023 [I 2022-10-27 09:03:18] ...trap/engines/engine.py.321: eval: epoch 58 | batch 260/510 [I 2022-10-27 09:03:18] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 09:03:18] ...trap/engines/engine.py.326: process: 0.28052 | load: 0.00015 [I 2022-10-27 09:03:21] ...trap/engines/engine.py.321: eval: epoch 58 | batch 270/510 [I 2022-10-27 09:03:21] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-27 09:03:21] ...trap/engines/engine.py.326: process: 0.28942 | load: 0.00015 [I 2022-10-27 09:03:24] ...trap/engines/engine.py.321: eval: epoch 58 | batch 280/510 [I 2022-10-27 09:03:24] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 09:03:24] ...trap/engines/engine.py.326: process: 0.29123 | load: 0.00015 [I 2022-10-27 09:03:27] ...trap/engines/engine.py.321: eval: epoch 58 | batch 290/510 [I 2022-10-27 09:03:27] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:03 [I 2022-10-27 09:03:27] ...trap/engines/engine.py.326: process: 0.28345 | load: 0.00025 [I 2022-10-27 09:03:30] ...trap/engines/engine.py.321: eval: epoch 58 | batch 300/510 [I 2022-10-27 09:03:30] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:00 [I 2022-10-27 09:03:30] ...trap/engines/engine.py.326: process: 0.28696 | load: 0.00016 [I 2022-10-27 09:03:33] ...trap/engines/engine.py.321: eval: epoch 58 | batch 310/510 [I 2022-10-27 09:03:33] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 09:03:33] ...trap/engines/engine.py.326: process: 0.28577 | load: 0.00025 [I 2022-10-27 09:03:36] ...trap/engines/engine.py.321: eval: epoch 58 | batch 320/510 [I 2022-10-27 09:03:36] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 09:03:36] ...trap/engines/engine.py.326: process: 0.28481 | load: 0.00016 [I 2022-10-27 09:03:38] ...trap/engines/engine.py.321: eval: epoch 58 | batch 330/510 [I 2022-10-27 09:03:38] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 09:03:38] ...trap/engines/engine.py.326: process: 0.28733 | load: 0.00025 [I 2022-10-27 09:03:41] ...trap/engines/engine.py.321: eval: epoch 58 | batch 340/510 [I 2022-10-27 09:03:41] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:48 [I 2022-10-27 09:03:41] ...trap/engines/engine.py.326: process: 0.28658 | load: 0.00014 [I 2022-10-27 09:03:44] ...trap/engines/engine.py.321: eval: epoch 58 | batch 350/510 [I 2022-10-27 09:03:44] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-27 09:03:44] ...trap/engines/engine.py.326: process: 0.28003 | load: 0.00015 [I 2022-10-27 09:03:47] ...trap/engines/engine.py.321: eval: epoch 58 | batch 360/510 [I 2022-10-27 09:03:47] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 09:03:47] ...trap/engines/engine.py.326: process: 0.29117 | load: 0.00015 [I 2022-10-27 09:03:50] ...trap/engines/engine.py.321: eval: epoch 58 | batch 370/510 [I 2022-10-27 09:03:50] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:40 [I 2022-10-27 09:03:50] ...trap/engines/engine.py.326: process: 0.28927 | load: 0.00016 [I 2022-10-27 09:03:53] ...trap/engines/engine.py.321: eval: epoch 58 | batch 380/510 [I 2022-10-27 09:03:53] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:37 [I 2022-10-27 09:03:53] ...trap/engines/engine.py.326: process: 0.28823 | load: 0.00015 [I 2022-10-27 09:03:56] ...trap/engines/engine.py.321: eval: epoch 58 | batch 390/510 [I 2022-10-27 09:03:56] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-27 09:03:56] ...trap/engines/engine.py.326: process: 0.28172 | load: 0.00015 [I 2022-10-27 09:03:59] ...trap/engines/engine.py.321: eval: epoch 58 | batch 400/510 [I 2022-10-27 09:03:59] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-27 09:03:59] ...trap/engines/engine.py.326: process: 0.28911 | load: 0.00015 [I 2022-10-27 09:04:01] ...trap/engines/engine.py.321: eval: epoch 58 | batch 410/510 [I 2022-10-27 09:04:01] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-27 09:04:01] ...trap/engines/engine.py.326: process: 0.28538 | load: 0.00020 [I 2022-10-27 09:04:04] ...trap/engines/engine.py.321: eval: epoch 58 | batch 420/510 [I 2022-10-27 09:04:04] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-27 09:04:04] ...trap/engines/engine.py.326: process: 0.28643 | load: 0.00015 [I 2022-10-27 09:04:07] ...trap/engines/engine.py.321: eval: epoch 58 | batch 430/510 [I 2022-10-27 09:04:07] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:22 [I 2022-10-27 09:04:07] ...trap/engines/engine.py.326: process: 0.28536 | load: 0.00015 [I 2022-10-27 09:04:10] ...trap/engines/engine.py.321: eval: epoch 58 | batch 440/510 [I 2022-10-27 09:04:10] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:20 [I 2022-10-27 09:04:10] ...trap/engines/engine.py.326: process: 0.29050 | load: 0.00020 [I 2022-10-27 09:04:13] ...trap/engines/engine.py.321: eval: epoch 58 | batch 450/510 [I 2022-10-27 09:04:13] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:17 [I 2022-10-27 09:04:13] ...trap/engines/engine.py.326: process: 0.29052 | load: 0.00016 [I 2022-10-27 09:04:16] ...trap/engines/engine.py.321: eval: epoch 58 | batch 460/510 [I 2022-10-27 09:04:16] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-27 09:04:16] ...trap/engines/engine.py.326: process: 0.28235 | load: 0.00016 [I 2022-10-27 09:04:19] ...trap/engines/engine.py.321: eval: epoch 58 | batch 470/510 [I 2022-10-27 09:04:19] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-27 09:04:19] ...trap/engines/engine.py.326: process: 0.28872 | load: 0.00021 [I 2022-10-27 09:04:21] ...trap/engines/engine.py.321: eval: epoch 58 | batch 480/510 [I 2022-10-27 09:04:21] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 09:04:21] ...trap/engines/engine.py.326: process: 0.28905 | load: 0.00015 [I 2022-10-27 09:04:24] ...trap/engines/engine.py.321: eval: epoch 58 | batch 490/510 [I 2022-10-27 09:04:24] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 09:04:24] ...trap/engines/engine.py.326: process: 0.28747 | load: 0.00014 [I 2022-10-27 09:04:27] ...trap/engines/engine.py.321: eval: epoch 58 | batch 500/510 [I 2022-10-27 09:04:27] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 09:04:27] ...trap/engines/engine.py.326: process: 0.29157 | load: 0.00015 [I 2022-10-27 09:04:30] ...trap/engines/engine.py.321: eval: epoch 58 | batch 510/510 [I 2022-10-27 09:04:30] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 09:04:30] ...trap/engines/engine.py.326: process: 0.28590 | load: 0.00010 [S 2022-10-27 09:04:31] ...trap/engines/engine.py.132: eval_epoch.epoch: 58 [I 2022-10-27 09:04:31] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.721700 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.904300 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936700 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.705600 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.901200 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934500 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.007253 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010909 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008730 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.007228 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009152 [S 2022-10-27 09:04:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007632 [I 2022-10-27 09:04:36] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 09:04:36] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 09:04:36] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 09:04:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 09:04:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 09:04:39] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 09:04:41] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 09:04:47] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 09:04:47] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 09:04:47] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 09:04:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 09:04:56] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 09:04:56] ...trap/engines/engine.py.163: Training model on trainset for epoch 59 [I 2022-10-27 09:04:56] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 09:04:59] ...trap/engines/engine.py.224: train: epoch 59 | batch 0/2304 [I 2022-10-27 09:04:59] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:26 [I 2022-10-27 09:04:59] ...trap/engines/engine.py.229: process: 2.69394 | load: 2.03606 [I 2022-10-27 09:04:59] ...trap/engines/engine.py.230: loss: 0.14977 [I 2022-10-27 09:04:59] ...optimizers/trijoint.py.151: total_norm: 31.756670 [I 2022-10-27 09:05:04] ...trap/engines/engine.py.224: train: epoch 59 | batch 10/2304 [I 2022-10-27 09:05:04] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:06 [I 2022-10-27 09:05:04] ...trap/engines/engine.py.229: process: 0.56242 | load: 0.00473 [I 2022-10-27 09:05:04] ...trap/engines/engine.py.230: loss: 0.15060 [I 2022-10-27 09:05:04] ...optimizers/trijoint.py.151: total_norm: 17.622789 [I 2022-10-27 09:05:10] ...trap/engines/engine.py.224: train: epoch 59 | batch 20/2304 [I 2022-10-27 09:05:10] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:21:58 [I 2022-10-27 09:05:10] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00329 [I 2022-10-27 09:05:10] ...trap/engines/engine.py.230: loss: 0.23727 [I 2022-10-27 09:05:10] ...optimizers/trijoint.py.151: total_norm: 21.798311 [I 2022-10-27 09:05:16] ...trap/engines/engine.py.224: train: epoch 59 | batch 30/2304 [I 2022-10-27 09:05:16] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:03 [I 2022-10-27 09:05:16] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00356 [I 2022-10-27 09:05:16] ...trap/engines/engine.py.230: loss: 0.13209 [I 2022-10-27 09:05:16] ...optimizers/trijoint.py.151: total_norm: 21.250618 [I 2022-10-27 09:05:21] ...trap/engines/engine.py.224: train: epoch 59 | batch 40/2304 [I 2022-10-27 09:05:21] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:48 [I 2022-10-27 09:05:21] ...trap/engines/engine.py.229: process: 0.53992 | load: 0.00305 [I 2022-10-27 09:05:21] ...trap/engines/engine.py.230: loss: 0.14672 [I 2022-10-27 09:05:21] ...optimizers/trijoint.py.151: total_norm: 15.333931 [I 2022-10-27 09:05:27] ...trap/engines/engine.py.224: train: epoch 59 | batch 50/2304 [I 2022-10-27 09:05:27] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:50 [I 2022-10-27 09:05:27] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00290 [I 2022-10-27 09:05:27] ...trap/engines/engine.py.230: loss: 0.16711 [I 2022-10-27 09:05:27] ...optimizers/trijoint.py.151: total_norm: 16.743650 [I 2022-10-27 09:05:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 60/2304 [I 2022-10-27 09:05:32] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:54 [I 2022-10-27 09:05:32] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00284 [I 2022-10-27 09:05:32] ...trap/engines/engine.py.230: loss: 0.21425 [I 2022-10-27 09:05:32] ...optimizers/trijoint.py.151: total_norm: 25.444756 [I 2022-10-27 09:05:38] ...trap/engines/engine.py.224: train: epoch 59 | batch 70/2304 [I 2022-10-27 09:05:38] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:45 [I 2022-10-27 09:05:38] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00265 [I 2022-10-27 09:05:38] ...trap/engines/engine.py.230: loss: 0.17640 [I 2022-10-27 09:05:38] ...optimizers/trijoint.py.151: total_norm: 13.743964 [I 2022-10-27 09:05:43] ...trap/engines/engine.py.224: train: epoch 59 | batch 80/2304 [I 2022-10-27 09:05:43] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:38 [I 2022-10-27 09:05:43] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00282 [I 2022-10-27 09:05:43] ...trap/engines/engine.py.230: loss: 0.13637 [I 2022-10-27 09:05:43] ...optimizers/trijoint.py.151: total_norm: 16.271255 [I 2022-10-27 09:05:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 90/2304 [I 2022-10-27 09:05:49] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:32 [I 2022-10-27 09:05:49] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00264 [I 2022-10-27 09:05:49] ...trap/engines/engine.py.230: loss: 0.16446 [I 2022-10-27 09:05:49] ...optimizers/trijoint.py.151: total_norm: 19.714140 [I 2022-10-27 09:05:55] ...trap/engines/engine.py.224: train: epoch 59 | batch 100/2304 [I 2022-10-27 09:05:55] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:30 [I 2022-10-27 09:05:55] ...trap/engines/engine.py.229: process: 0.55900 | load: 0.00279 [I 2022-10-27 09:05:55] ...trap/engines/engine.py.230: loss: 0.15938 [I 2022-10-27 09:05:55] ...optimizers/trijoint.py.151: total_norm: 15.470746 [I 2022-10-27 09:06:00] ...trap/engines/engine.py.224: train: epoch 59 | batch 110/2304 [I 2022-10-27 09:06:00] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:56 [I 2022-10-27 09:06:00] ...trap/engines/engine.py.229: process: 0.57345 | load: 0.00270 [I 2022-10-27 09:06:00] ...trap/engines/engine.py.230: loss: 0.13051 [I 2022-10-27 09:06:00] ...optimizers/trijoint.py.151: total_norm: 22.953285 [I 2022-10-27 09:06:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 120/2304 [I 2022-10-27 09:06:06] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:22 [I 2022-10-27 09:06:06] ...trap/engines/engine.py.229: process: 0.55283 | load: 0.00273 [I 2022-10-27 09:06:06] ...trap/engines/engine.py.230: loss: 0.12023 [I 2022-10-27 09:06:06] ...optimizers/trijoint.py.151: total_norm: 8.191651 [I 2022-10-27 09:06:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 130/2304 [I 2022-10-27 09:06:12] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:19 [I 2022-10-27 09:06:12] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00259 [I 2022-10-27 09:06:12] ...trap/engines/engine.py.230: loss: 0.16673 [I 2022-10-27 09:06:12] ...optimizers/trijoint.py.151: total_norm: 11.217971 [I 2022-10-27 09:06:17] ...trap/engines/engine.py.224: train: epoch 59 | batch 140/2304 [I 2022-10-27 09:06:17] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:03 [I 2022-10-27 09:06:17] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00272 [I 2022-10-27 09:06:17] ...trap/engines/engine.py.230: loss: 0.20653 [I 2022-10-27 09:06:17] ...optimizers/trijoint.py.151: total_norm: 30.791691 [I 2022-10-27 09:06:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 150/2304 [I 2022-10-27 09:06:23] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:00 [I 2022-10-27 09:06:23] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00272 [I 2022-10-27 09:06:23] ...trap/engines/engine.py.230: loss: 0.11843 [I 2022-10-27 09:06:23] ...optimizers/trijoint.py.151: total_norm: 10.450095 [I 2022-10-27 09:06:28] ...trap/engines/engine.py.224: train: epoch 59 | batch 160/2304 [I 2022-10-27 09:06:28] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:47 [I 2022-10-27 09:06:28] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00834 [I 2022-10-27 09:06:28] ...trap/engines/engine.py.230: loss: 0.12129 [I 2022-10-27 09:06:28] ...optimizers/trijoint.py.151: total_norm: 10.130692 [I 2022-10-27 09:06:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 170/2304 [I 2022-10-27 09:06:34] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:52 [I 2022-10-27 09:06:34] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00258 [I 2022-10-27 09:06:34] ...trap/engines/engine.py.230: loss: 0.14595 [I 2022-10-27 09:06:34] ...optimizers/trijoint.py.151: total_norm: 13.609168 [I 2022-10-27 09:06:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 180/2304 [I 2022-10-27 09:06:40] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:41 [I 2022-10-27 09:06:40] ...trap/engines/engine.py.229: process: 0.54799 | load: 0.00269 [I 2022-10-27 09:06:40] ...trap/engines/engine.py.230: loss: 0.12925 [I 2022-10-27 09:06:40] ...optimizers/trijoint.py.151: total_norm: 17.403996 [I 2022-10-27 09:06:45] ...trap/engines/engine.py.224: train: epoch 59 | batch 190/2304 [I 2022-10-27 09:06:45] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:27 [I 2022-10-27 09:06:45] ...trap/engines/engine.py.229: process: 0.54686 | load: 0.00250 [I 2022-10-27 09:06:45] ...trap/engines/engine.py.230: loss: 0.16834 [I 2022-10-27 09:06:45] ...optimizers/trijoint.py.151: total_norm: 13.612999 [I 2022-10-27 09:06:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 200/2304 [I 2022-10-27 09:06:51] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:41 [I 2022-10-27 09:06:51] ...trap/engines/engine.py.229: process: 0.59366 | load: 0.00263 [I 2022-10-27 09:06:51] ...trap/engines/engine.py.230: loss: 0.20292 [I 2022-10-27 09:06:51] ...optimizers/trijoint.py.151: total_norm: 17.303600 [I 2022-10-27 09:06:56] ...trap/engines/engine.py.224: train: epoch 59 | batch 210/2304 [I 2022-10-27 09:06:56] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:18 [I 2022-10-27 09:06:56] ...trap/engines/engine.py.229: process: 0.55423 | load: 0.00263 [I 2022-10-27 09:06:56] ...trap/engines/engine.py.230: loss: 0.16833 [I 2022-10-27 09:06:56] ...optimizers/trijoint.py.151: total_norm: 14.941779 [I 2022-10-27 09:07:21] ...trap/engines/engine.py.224: train: epoch 59 | batch 220/2304 [I 2022-10-27 09:07:21] ...trap/engines/engine.py.228: elapsed: 0:02:24 | left: 0:46:43 [I 2022-10-27 09:07:21] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00260 [I 2022-10-27 09:07:21] ...trap/engines/engine.py.230: loss: 0.13588 [I 2022-10-27 09:07:21] ...optimizers/trijoint.py.151: total_norm: 16.347151 [I 2022-10-27 09:07:26] ...trap/engines/engine.py.224: train: epoch 59 | batch 230/2304 [I 2022-10-27 09:07:26] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:22:05 [I 2022-10-27 09:07:26] ...trap/engines/engine.py.229: process: 0.55496 | load: 0.00256 [I 2022-10-27 09:07:26] ...trap/engines/engine.py.230: loss: 0.10978 [I 2022-10-27 09:07:26] ...optimizers/trijoint.py.151: total_norm: 14.709687 [I 2022-10-27 09:07:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 240/2304 [I 2022-10-27 09:07:32] ...trap/engines/engine.py.228: elapsed: 0:02:35 | left: 0:19:28 [I 2022-10-27 09:07:32] ...trap/engines/engine.py.229: process: 0.55101 | load: 0.00240 [I 2022-10-27 09:07:32] ...trap/engines/engine.py.230: loss: 0.15696 [I 2022-10-27 09:07:32] ...optimizers/trijoint.py.151: total_norm: 15.619649 [I 2022-10-27 09:07:37] ...trap/engines/engine.py.224: train: epoch 59 | batch 250/2304 [I 2022-10-27 09:07:37] ...trap/engines/engine.py.228: elapsed: 0:02:41 | left: 0:19:02 [I 2022-10-27 09:07:37] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00241 [I 2022-10-27 09:07:37] ...trap/engines/engine.py.230: loss: 0.17139 [I 2022-10-27 09:07:37] ...optimizers/trijoint.py.151: total_norm: 21.267248 [I 2022-10-27 09:07:43] ...trap/engines/engine.py.224: train: epoch 59 | batch 260/2304 [I 2022-10-27 09:07:43] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:18:45 [I 2022-10-27 09:07:43] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00240 [I 2022-10-27 09:07:43] ...trap/engines/engine.py.230: loss: 0.12574 [I 2022-10-27 09:07:43] ...optimizers/trijoint.py.151: total_norm: 11.862214 [I 2022-10-27 09:07:48] ...trap/engines/engine.py.224: train: epoch 59 | batch 270/2304 [I 2022-10-27 09:07:48] ...trap/engines/engine.py.228: elapsed: 0:02:52 | left: 0:18:53 [I 2022-10-27 09:07:48] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00246 [I 2022-10-27 09:07:48] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-27 09:07:48] ...optimizers/trijoint.py.151: total_norm: 13.012756 [I 2022-10-27 09:07:54] ...trap/engines/engine.py.224: train: epoch 59 | batch 280/2304 [I 2022-10-27 09:07:54] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:18:42 [I 2022-10-27 09:07:54] ...trap/engines/engine.py.229: process: 0.54397 | load: 0.00243 [I 2022-10-27 09:07:54] ...trap/engines/engine.py.230: loss: 0.12862 [I 2022-10-27 09:07:54] ...optimizers/trijoint.py.151: total_norm: 15.792113 [I 2022-10-27 09:08:00] ...trap/engines/engine.py.224: train: epoch 59 | batch 290/2304 [I 2022-10-27 09:08:00] ...trap/engines/engine.py.228: elapsed: 0:03:03 | left: 0:18:38 [I 2022-10-27 09:08:00] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00250 [I 2022-10-27 09:08:00] ...trap/engines/engine.py.230: loss: 0.18210 [I 2022-10-27 09:08:00] ...optimizers/trijoint.py.151: total_norm: 23.539339 [I 2022-10-27 09:08:05] ...trap/engines/engine.py.224: train: epoch 59 | batch 300/2304 [I 2022-10-27 09:08:05] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:18:30 [I 2022-10-27 09:08:05] ...trap/engines/engine.py.229: process: 0.55462 | load: 0.00237 [I 2022-10-27 09:08:05] ...trap/engines/engine.py.230: loss: 0.12673 [I 2022-10-27 09:08:05] ...optimizers/trijoint.py.151: total_norm: 12.233212 [I 2022-10-27 09:08:11] ...trap/engines/engine.py.224: train: epoch 59 | batch 310/2304 [I 2022-10-27 09:08:11] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:18:25 [I 2022-10-27 09:08:11] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00238 [I 2022-10-27 09:08:11] ...trap/engines/engine.py.230: loss: 0.16493 [I 2022-10-27 09:08:11] ...optimizers/trijoint.py.151: total_norm: 14.229905 [I 2022-10-27 09:08:16] ...trap/engines/engine.py.224: train: epoch 59 | batch 320/2304 [I 2022-10-27 09:08:16] ...trap/engines/engine.py.228: elapsed: 0:03:20 | left: 0:18:21 [I 2022-10-27 09:08:16] ...trap/engines/engine.py.229: process: 0.54418 | load: 0.00248 [I 2022-10-27 09:08:16] ...trap/engines/engine.py.230: loss: 0.17033 [I 2022-10-27 09:08:16] ...optimizers/trijoint.py.151: total_norm: 29.709648 [I 2022-10-27 09:08:22] ...trap/engines/engine.py.224: train: epoch 59 | batch 330/2304 [I 2022-10-27 09:08:22] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:18:49 [I 2022-10-27 09:08:22] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00241 [I 2022-10-27 09:08:22] ...trap/engines/engine.py.230: loss: 0.12540 [I 2022-10-27 09:08:22] ...optimizers/trijoint.py.151: total_norm: 20.098772 [I 2022-10-27 09:08:27] ...trap/engines/engine.py.224: train: epoch 59 | batch 340/2304 [I 2022-10-27 09:08:27] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:18:08 [I 2022-10-27 09:08:27] ...trap/engines/engine.py.229: process: 0.53688 | load: 0.00245 [I 2022-10-27 09:08:27] ...trap/engines/engine.py.230: loss: 0.20989 [I 2022-10-27 09:08:27] ...optimizers/trijoint.py.151: total_norm: 26.733213 [I 2022-10-27 09:08:33] ...trap/engines/engine.py.224: train: epoch 59 | batch 350/2304 [I 2022-10-27 09:08:33] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:18:04 [I 2022-10-27 09:08:33] ...trap/engines/engine.py.229: process: 0.55024 | load: 0.00241 [I 2022-10-27 09:08:33] ...trap/engines/engine.py.230: loss: 0.16420 [I 2022-10-27 09:08:33] ...optimizers/trijoint.py.151: total_norm: 29.010155 [I 2022-10-27 09:08:39] ...trap/engines/engine.py.224: train: epoch 59 | batch 360/2304 [I 2022-10-27 09:08:39] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:17:50 [I 2022-10-27 09:08:39] ...trap/engines/engine.py.229: process: 0.55007 | load: 0.00249 [I 2022-10-27 09:08:39] ...trap/engines/engine.py.230: loss: 0.16581 [I 2022-10-27 09:08:39] ...optimizers/trijoint.py.151: total_norm: 19.164270 [I 2022-10-27 09:08:44] ...trap/engines/engine.py.224: train: epoch 59 | batch 370/2304 [I 2022-10-27 09:08:44] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:52 [I 2022-10-27 09:08:44] ...trap/engines/engine.py.229: process: 0.55493 | load: 0.00243 [I 2022-10-27 09:08:44] ...trap/engines/engine.py.230: loss: 0.11844 [I 2022-10-27 09:08:44] ...optimizers/trijoint.py.151: total_norm: 20.526245 [I 2022-10-27 09:08:50] ...trap/engines/engine.py.224: train: epoch 59 | batch 380/2304 [I 2022-10-27 09:08:50] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:17:54 [I 2022-10-27 09:08:50] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00244 [I 2022-10-27 09:08:50] ...trap/engines/engine.py.230: loss: 0.17152 [I 2022-10-27 09:08:50] ...optimizers/trijoint.py.151: total_norm: 16.111378 [I 2022-10-27 09:08:55] ...trap/engines/engine.py.224: train: epoch 59 | batch 390/2304 [I 2022-10-27 09:08:55] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:17:45 [I 2022-10-27 09:08:55] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00242 [I 2022-10-27 09:08:55] ...trap/engines/engine.py.230: loss: 0.18069 [I 2022-10-27 09:08:55] ...optimizers/trijoint.py.151: total_norm: 11.461802 [I 2022-10-27 09:09:01] ...trap/engines/engine.py.224: train: epoch 59 | batch 400/2304 [I 2022-10-27 09:09:01] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:36 [I 2022-10-27 09:09:01] ...trap/engines/engine.py.229: process: 0.55082 | load: 0.00248 [I 2022-10-27 09:09:01] ...trap/engines/engine.py.230: loss: 0.12875 [I 2022-10-27 09:09:01] ...optimizers/trijoint.py.151: total_norm: 24.507883 [I 2022-10-27 09:09:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 410/2304 [I 2022-10-27 09:09:06] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:17:33 [I 2022-10-27 09:09:06] ...trap/engines/engine.py.229: process: 0.56405 | load: 0.00243 [I 2022-10-27 09:09:06] ...trap/engines/engine.py.230: loss: 0.15230 [I 2022-10-27 09:09:06] ...optimizers/trijoint.py.151: total_norm: 13.962363 [I 2022-10-27 09:09:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 420/2304 [I 2022-10-27 09:09:12] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:20 [I 2022-10-27 09:09:12] ...trap/engines/engine.py.229: process: 0.54389 | load: 0.00246 [I 2022-10-27 09:09:12] ...trap/engines/engine.py.230: loss: 0.15835 [I 2022-10-27 09:09:12] ...optimizers/trijoint.py.151: total_norm: 13.612311 [I 2022-10-27 09:09:18] ...trap/engines/engine.py.224: train: epoch 59 | batch 430/2304 [I 2022-10-27 09:09:18] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:17:27 [I 2022-10-27 09:09:18] ...trap/engines/engine.py.229: process: 0.56763 | load: 0.00266 [I 2022-10-27 09:09:18] ...trap/engines/engine.py.230: loss: 0.15212 [I 2022-10-27 09:09:18] ...optimizers/trijoint.py.151: total_norm: 27.731268 [I 2022-10-27 09:09:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 440/2304 [I 2022-10-27 09:09:23] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:17:56 [I 2022-10-27 09:09:23] ...trap/engines/engine.py.229: process: 0.54844 | load: 0.00242 [I 2022-10-27 09:09:23] ...trap/engines/engine.py.230: loss: 0.15265 [I 2022-10-27 09:09:23] ...optimizers/trijoint.py.151: total_norm: 21.565401 [I 2022-10-27 09:09:29] ...trap/engines/engine.py.224: train: epoch 59 | batch 450/2304 [I 2022-10-27 09:09:29] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:17:05 [I 2022-10-27 09:09:29] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00244 [I 2022-10-27 09:09:29] ...trap/engines/engine.py.230: loss: 0.18543 [I 2022-10-27 09:09:29] ...optimizers/trijoint.py.151: total_norm: 7.421409 [I 2022-10-27 09:09:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 460/2304 [I 2022-10-27 09:09:34] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:17:05 [I 2022-10-27 09:09:34] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00246 [I 2022-10-27 09:09:34] ...trap/engines/engine.py.230: loss: 0.11004 [I 2022-10-27 09:09:34] ...optimizers/trijoint.py.151: total_norm: 12.975837 [I 2022-10-27 09:09:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 470/2304 [I 2022-10-27 09:09:40] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:53 [I 2022-10-27 09:09:40] ...trap/engines/engine.py.229: process: 0.54659 | load: 0.00243 [I 2022-10-27 09:09:40] ...trap/engines/engine.py.230: loss: 0.15889 [I 2022-10-27 09:09:40] ...optimizers/trijoint.py.151: total_norm: 20.867416 [I 2022-10-27 09:09:45] ...trap/engines/engine.py.224: train: epoch 59 | batch 480/2304 [I 2022-10-27 09:09:45] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:54 [I 2022-10-27 09:09:45] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00250 [I 2022-10-27 09:09:45] ...trap/engines/engine.py.230: loss: 0.11176 [I 2022-10-27 09:09:45] ...optimizers/trijoint.py.151: total_norm: 8.475633 [I 2022-10-27 09:09:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 490/2304 [I 2022-10-27 09:09:51] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:43 [I 2022-10-27 09:09:51] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00251 [I 2022-10-27 09:09:51] ...trap/engines/engine.py.230: loss: 0.12321 [I 2022-10-27 09:09:51] ...optimizers/trijoint.py.151: total_norm: 15.338027 [I 2022-10-27 09:09:57] ...trap/engines/engine.py.224: train: epoch 59 | batch 500/2304 [I 2022-10-27 09:09:57] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:41 [I 2022-10-27 09:09:57] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00246 [I 2022-10-27 09:09:57] ...trap/engines/engine.py.230: loss: 0.15607 [I 2022-10-27 09:09:57] ...optimizers/trijoint.py.151: total_norm: 16.430248 [I 2022-10-27 09:10:02] ...trap/engines/engine.py.224: train: epoch 59 | batch 510/2304 [I 2022-10-27 09:10:02] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:35 [I 2022-10-27 09:10:02] ...trap/engines/engine.py.229: process: 0.54333 | load: 0.00250 [I 2022-10-27 09:10:02] ...trap/engines/engine.py.230: loss: 0.19497 [I 2022-10-27 09:10:02] ...optimizers/trijoint.py.151: total_norm: 19.613733 [I 2022-10-27 09:10:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 520/2304 [I 2022-10-27 09:10:08] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:36 [I 2022-10-27 09:10:08] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00245 [I 2022-10-27 09:10:08] ...trap/engines/engine.py.230: loss: 0.17020 [I 2022-10-27 09:10:08] ...optimizers/trijoint.py.151: total_norm: 15.689225 [I 2022-10-27 09:10:13] ...trap/engines/engine.py.224: train: epoch 59 | batch 530/2304 [I 2022-10-27 09:10:13] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:23 [I 2022-10-27 09:10:13] ...trap/engines/engine.py.229: process: 0.56220 | load: 0.00247 [I 2022-10-27 09:10:13] ...trap/engines/engine.py.230: loss: 0.12497 [I 2022-10-27 09:10:13] ...optimizers/trijoint.py.151: total_norm: 18.287992 [I 2022-10-27 09:10:19] ...trap/engines/engine.py.224: train: epoch 59 | batch 540/2304 [I 2022-10-27 09:10:19] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:19 [I 2022-10-27 09:10:19] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00242 [I 2022-10-27 09:10:19] ...trap/engines/engine.py.230: loss: 0.10188 [I 2022-10-27 09:10:19] ...optimizers/trijoint.py.151: total_norm: 15.250779 [I 2022-10-27 09:10:24] ...trap/engines/engine.py.224: train: epoch 59 | batch 550/2304 [I 2022-10-27 09:10:24] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:16:15 [I 2022-10-27 09:10:24] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00245 [I 2022-10-27 09:10:24] ...trap/engines/engine.py.230: loss: 0.10005 [I 2022-10-27 09:10:24] ...optimizers/trijoint.py.151: total_norm: 16.703035 [I 2022-10-27 09:10:30] ...trap/engines/engine.py.224: train: epoch 59 | batch 560/2304 [I 2022-10-27 09:10:30] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:11 [I 2022-10-27 09:10:30] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00277 [I 2022-10-27 09:10:30] ...trap/engines/engine.py.230: loss: 0.13169 [I 2022-10-27 09:10:30] ...optimizers/trijoint.py.151: total_norm: 16.865271 [I 2022-10-27 09:10:36] ...trap/engines/engine.py.224: train: epoch 59 | batch 570/2304 [I 2022-10-27 09:10:36] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:16:10 [I 2022-10-27 09:10:36] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00240 [I 2022-10-27 09:10:36] ...trap/engines/engine.py.230: loss: 0.15812 [I 2022-10-27 09:10:36] ...optimizers/trijoint.py.151: total_norm: 20.498077 [I 2022-10-27 09:10:41] ...trap/engines/engine.py.224: train: epoch 59 | batch 580/2304 [I 2022-10-27 09:10:41] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:52 [I 2022-10-27 09:10:41] ...trap/engines/engine.py.229: process: 0.55006 | load: 0.00269 [I 2022-10-27 09:10:41] ...trap/engines/engine.py.230: loss: 0.12292 [I 2022-10-27 09:10:41] ...optimizers/trijoint.py.151: total_norm: 11.608146 [I 2022-10-27 09:10:47] ...trap/engines/engine.py.224: train: epoch 59 | batch 590/2304 [I 2022-10-27 09:10:47] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:52 [I 2022-10-27 09:10:47] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00241 [I 2022-10-27 09:10:47] ...trap/engines/engine.py.230: loss: 0.21275 [I 2022-10-27 09:10:47] ...optimizers/trijoint.py.151: total_norm: 19.997944 [I 2022-10-27 09:10:52] ...trap/engines/engine.py.224: train: epoch 59 | batch 600/2304 [I 2022-10-27 09:10:52] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:49 [I 2022-10-27 09:10:52] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00245 [I 2022-10-27 09:10:52] ...trap/engines/engine.py.230: loss: 0.14394 [I 2022-10-27 09:10:52] ...optimizers/trijoint.py.151: total_norm: 45.521614 [I 2022-10-27 09:10:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 610/2304 [I 2022-10-27 09:10:58] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:51 [I 2022-10-27 09:10:58] ...trap/engines/engine.py.229: process: 0.56653 | load: 0.00242 [I 2022-10-27 09:10:58] ...trap/engines/engine.py.230: loss: 0.16157 [I 2022-10-27 09:10:58] ...optimizers/trijoint.py.151: total_norm: 16.632559 [I 2022-10-27 09:11:03] ...trap/engines/engine.py.224: train: epoch 59 | batch 620/2304 [I 2022-10-27 09:11:03] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:40 [I 2022-10-27 09:11:03] ...trap/engines/engine.py.229: process: 0.56082 | load: 0.00242 [I 2022-10-27 09:11:03] ...trap/engines/engine.py.230: loss: 0.23069 [I 2022-10-27 09:11:03] ...optimizers/trijoint.py.151: total_norm: 24.368851 [I 2022-10-27 09:11:09] ...trap/engines/engine.py.224: train: epoch 59 | batch 630/2304 [I 2022-10-27 09:11:09] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:34 [I 2022-10-27 09:11:09] ...trap/engines/engine.py.229: process: 0.56290 | load: 0.00244 [I 2022-10-27 09:11:09] ...trap/engines/engine.py.230: loss: 0.13771 [I 2022-10-27 09:11:09] ...optimizers/trijoint.py.151: total_norm: 22.034733 [I 2022-10-27 09:11:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 640/2304 [I 2022-10-27 09:11:15] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:31 [I 2022-10-27 09:11:15] ...trap/engines/engine.py.229: process: 0.54589 | load: 0.00241 [I 2022-10-27 09:11:15] ...trap/engines/engine.py.230: loss: 0.16681 [I 2022-10-27 09:11:15] ...optimizers/trijoint.py.151: total_norm: 36.403580 [I 2022-10-27 09:11:20] ...trap/engines/engine.py.224: train: epoch 59 | batch 650/2304 [I 2022-10-27 09:11:20] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:23 [I 2022-10-27 09:11:20] ...trap/engines/engine.py.229: process: 0.56339 | load: 0.00245 [I 2022-10-27 09:11:20] ...trap/engines/engine.py.230: loss: 0.16051 [I 2022-10-27 09:11:20] ...optimizers/trijoint.py.151: total_norm: 13.627833 [I 2022-10-27 09:11:26] ...trap/engines/engine.py.224: train: epoch 59 | batch 660/2304 [I 2022-10-27 09:11:26] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:18 [I 2022-10-27 09:11:26] ...trap/engines/engine.py.229: process: 0.56527 | load: 0.00243 [I 2022-10-27 09:11:26] ...trap/engines/engine.py.230: loss: 0.13078 [I 2022-10-27 09:11:26] ...optimizers/trijoint.py.151: total_norm: 18.352594 [I 2022-10-27 09:11:31] ...trap/engines/engine.py.224: train: epoch 59 | batch 670/2304 [I 2022-10-27 09:11:31] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:15:04 [I 2022-10-27 09:11:31] ...trap/engines/engine.py.229: process: 0.53874 | load: 0.00249 [I 2022-10-27 09:11:31] ...trap/engines/engine.py.230: loss: 0.10467 [I 2022-10-27 09:11:31] ...optimizers/trijoint.py.151: total_norm: 7.369617 [I 2022-10-27 09:11:37] ...trap/engines/engine.py.224: train: epoch 59 | batch 680/2304 [I 2022-10-27 09:11:37] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:15:05 [I 2022-10-27 09:11:37] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00240 [I 2022-10-27 09:11:37] ...trap/engines/engine.py.230: loss: 0.16699 [I 2022-10-27 09:11:37] ...optimizers/trijoint.py.151: total_norm: 31.164789 [I 2022-10-27 09:11:43] ...trap/engines/engine.py.224: train: epoch 59 | batch 690/2304 [I 2022-10-27 09:11:43] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:59 [I 2022-10-27 09:11:43] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00247 [I 2022-10-27 09:11:43] ...trap/engines/engine.py.230: loss: 0.17386 [I 2022-10-27 09:11:43] ...optimizers/trijoint.py.151: total_norm: 14.285080 [I 2022-10-27 09:11:48] ...trap/engines/engine.py.224: train: epoch 59 | batch 700/2304 [I 2022-10-27 09:11:48] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:57 [I 2022-10-27 09:11:48] ...trap/engines/engine.py.229: process: 0.57498 | load: 0.00244 [I 2022-10-27 09:11:48] ...trap/engines/engine.py.230: loss: 0.12761 [I 2022-10-27 09:11:48] ...optimizers/trijoint.py.151: total_norm: 9.430140 [I 2022-10-27 09:11:54] ...trap/engines/engine.py.224: train: epoch 59 | batch 710/2304 [I 2022-10-27 09:11:54] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:50 [I 2022-10-27 09:11:54] ...trap/engines/engine.py.229: process: 0.56539 | load: 0.00242 [I 2022-10-27 09:11:54] ...trap/engines/engine.py.230: loss: 0.18130 [I 2022-10-27 09:11:54] ...optimizers/trijoint.py.151: total_norm: 13.326267 [I 2022-10-27 09:11:59] ...trap/engines/engine.py.224: train: epoch 59 | batch 720/2304 [I 2022-10-27 09:11:59] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:38 [I 2022-10-27 09:11:59] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00247 [I 2022-10-27 09:11:59] ...trap/engines/engine.py.230: loss: 0.15529 [I 2022-10-27 09:11:59] ...optimizers/trijoint.py.151: total_norm: 13.834906 [I 2022-10-27 09:12:05] ...trap/engines/engine.py.224: train: epoch 59 | batch 730/2304 [I 2022-10-27 09:12:05] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:38 [I 2022-10-27 09:12:05] ...trap/engines/engine.py.229: process: 0.56579 | load: 0.00264 [I 2022-10-27 09:12:05] ...trap/engines/engine.py.230: loss: 0.14179 [I 2022-10-27 09:12:05] ...optimizers/trijoint.py.151: total_norm: 27.413103 [I 2022-10-27 09:12:10] ...trap/engines/engine.py.224: train: epoch 59 | batch 740/2304 [I 2022-10-27 09:12:10] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:25 [I 2022-10-27 09:12:10] ...trap/engines/engine.py.229: process: 0.54535 | load: 0.00243 [I 2022-10-27 09:12:10] ...trap/engines/engine.py.230: loss: 0.11073 [I 2022-10-27 09:12:10] ...optimizers/trijoint.py.151: total_norm: 7.793648 [I 2022-10-27 09:12:16] ...trap/engines/engine.py.224: train: epoch 59 | batch 750/2304 [I 2022-10-27 09:12:16] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:25 [I 2022-10-27 09:12:16] ...trap/engines/engine.py.229: process: 0.55694 | load: 0.00247 [I 2022-10-27 09:12:16] ...trap/engines/engine.py.230: loss: 0.17967 [I 2022-10-27 09:12:16] ...optimizers/trijoint.py.151: total_norm: 17.428446 [I 2022-10-27 09:12:22] ...trap/engines/engine.py.224: train: epoch 59 | batch 760/2304 [I 2022-10-27 09:12:22] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:18 [I 2022-10-27 09:12:22] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00242 [I 2022-10-27 09:12:22] ...trap/engines/engine.py.230: loss: 0.11530 [I 2022-10-27 09:12:22] ...optimizers/trijoint.py.151: total_norm: 10.048888 [I 2022-10-27 09:12:27] ...trap/engines/engine.py.224: train: epoch 59 | batch 770/2304 [I 2022-10-27 09:12:27] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:14 [I 2022-10-27 09:12:27] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00242 [I 2022-10-27 09:12:27] ...trap/engines/engine.py.230: loss: 0.11017 [I 2022-10-27 09:12:27] ...optimizers/trijoint.py.151: total_norm: 11.539710 [I 2022-10-27 09:12:33] ...trap/engines/engine.py.224: train: epoch 59 | batch 780/2304 [I 2022-10-27 09:12:33] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:14:01 [I 2022-10-27 09:12:33] ...trap/engines/engine.py.229: process: 0.55461 | load: 0.00243 [I 2022-10-27 09:12:33] ...trap/engines/engine.py.230: loss: 0.12680 [I 2022-10-27 09:12:33] ...optimizers/trijoint.py.151: total_norm: 11.007824 [I 2022-10-27 09:12:38] ...trap/engines/engine.py.224: train: epoch 59 | batch 790/2304 [I 2022-10-27 09:12:38] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:59 [I 2022-10-27 09:12:38] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00242 [I 2022-10-27 09:12:38] ...trap/engines/engine.py.230: loss: 0.19055 [I 2022-10-27 09:12:38] ...optimizers/trijoint.py.151: total_norm: 20.453646 [I 2022-10-27 09:12:44] ...trap/engines/engine.py.224: train: epoch 59 | batch 800/2304 [I 2022-10-27 09:12:44] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:55 [I 2022-10-27 09:12:44] ...trap/engines/engine.py.229: process: 0.56194 | load: 0.00241 [I 2022-10-27 09:12:44] ...trap/engines/engine.py.230: loss: 0.10176 [I 2022-10-27 09:12:44] ...optimizers/trijoint.py.151: total_norm: 8.729675 [I 2022-10-27 09:12:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 810/2304 [I 2022-10-27 09:12:49] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:49 [I 2022-10-27 09:12:49] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00243 [I 2022-10-27 09:12:49] ...trap/engines/engine.py.230: loss: 0.14174 [I 2022-10-27 09:12:49] ...optimizers/trijoint.py.151: total_norm: 16.371166 [I 2022-10-27 09:12:55] ...trap/engines/engine.py.224: train: epoch 59 | batch 820/2304 [I 2022-10-27 09:12:55] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:45 [I 2022-10-27 09:12:55] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00241 [I 2022-10-27 09:12:55] ...trap/engines/engine.py.230: loss: 0.17658 [I 2022-10-27 09:12:55] ...optimizers/trijoint.py.151: total_norm: 18.182013 [I 2022-10-27 09:13:01] ...trap/engines/engine.py.224: train: epoch 59 | batch 830/2304 [I 2022-10-27 09:13:01] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:35 [I 2022-10-27 09:13:01] ...trap/engines/engine.py.229: process: 0.54281 | load: 0.00242 [I 2022-10-27 09:13:01] ...trap/engines/engine.py.230: loss: 0.16060 [I 2022-10-27 09:13:01] ...optimizers/trijoint.py.151: total_norm: 36.212524 [I 2022-10-27 09:13:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 840/2304 [I 2022-10-27 09:13:06] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:33 [I 2022-10-27 09:13:06] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00241 [I 2022-10-27 09:13:06] ...trap/engines/engine.py.230: loss: 0.10522 [I 2022-10-27 09:13:06] ...optimizers/trijoint.py.151: total_norm: 9.573748 [I 2022-10-27 09:13:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 850/2304 [I 2022-10-27 09:13:12] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:26 [I 2022-10-27 09:13:12] ...trap/engines/engine.py.229: process: 0.55458 | load: 0.00243 [I 2022-10-27 09:13:12] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-27 09:13:12] ...optimizers/trijoint.py.151: total_norm: 12.765071 [I 2022-10-27 09:13:17] ...trap/engines/engine.py.224: train: epoch 59 | batch 860/2304 [I 2022-10-27 09:13:17] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:19 [I 2022-10-27 09:13:17] ...trap/engines/engine.py.229: process: 0.55459 | load: 0.00243 [I 2022-10-27 09:13:17] ...trap/engines/engine.py.230: loss: 0.17525 [I 2022-10-27 09:13:17] ...optimizers/trijoint.py.151: total_norm: 27.546097 [I 2022-10-27 09:13:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 870/2304 [I 2022-10-27 09:13:23] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:17 [I 2022-10-27 09:13:23] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00241 [I 2022-10-27 09:13:23] ...trap/engines/engine.py.230: loss: 0.15196 [I 2022-10-27 09:13:23] ...optimizers/trijoint.py.151: total_norm: 16.101589 [I 2022-10-27 09:13:28] ...trap/engines/engine.py.224: train: epoch 59 | batch 880/2304 [I 2022-10-27 09:13:28] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:13:19 [I 2022-10-27 09:13:28] ...trap/engines/engine.py.229: process: 0.56955 | load: 0.00243 [I 2022-10-27 09:13:28] ...trap/engines/engine.py.230: loss: 0.12933 [I 2022-10-27 09:13:28] ...optimizers/trijoint.py.151: total_norm: 17.132349 [I 2022-10-27 09:13:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 890/2304 [I 2022-10-27 09:13:34] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:13:07 [I 2022-10-27 09:13:34] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00248 [I 2022-10-27 09:13:34] ...trap/engines/engine.py.230: loss: 0.20224 [I 2022-10-27 09:13:34] ...optimizers/trijoint.py.151: total_norm: 15.151359 [I 2022-10-27 09:13:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 900/2304 [I 2022-10-27 09:13:40] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:58 [I 2022-10-27 09:13:40] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00248 [I 2022-10-27 09:13:40] ...trap/engines/engine.py.230: loss: 0.15259 [I 2022-10-27 09:13:40] ...optimizers/trijoint.py.151: total_norm: 22.475225 [I 2022-10-27 09:13:45] ...trap/engines/engine.py.224: train: epoch 59 | batch 910/2304 [I 2022-10-27 09:13:45] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:57 [I 2022-10-27 09:13:45] ...trap/engines/engine.py.229: process: 0.55395 | load: 0.00250 [I 2022-10-27 09:13:45] ...trap/engines/engine.py.230: loss: 0.12237 [I 2022-10-27 09:13:45] ...optimizers/trijoint.py.151: total_norm: 14.267630 [I 2022-10-27 09:13:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 920/2304 [I 2022-10-27 09:13:51] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:53 [I 2022-10-27 09:13:51] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00245 [I 2022-10-27 09:13:51] ...trap/engines/engine.py.230: loss: 0.12238 [I 2022-10-27 09:13:51] ...optimizers/trijoint.py.151: total_norm: 11.205517 [I 2022-10-27 09:13:56] ...trap/engines/engine.py.224: train: epoch 59 | batch 930/2304 [I 2022-10-27 09:13:56] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:45 [I 2022-10-27 09:13:56] ...trap/engines/engine.py.229: process: 0.56117 | load: 0.00245 [I 2022-10-27 09:13:56] ...trap/engines/engine.py.230: loss: 0.18819 [I 2022-10-27 09:13:56] ...optimizers/trijoint.py.151: total_norm: 20.874233 [I 2022-10-27 09:14:02] ...trap/engines/engine.py.224: train: epoch 59 | batch 940/2304 [I 2022-10-27 09:14:02] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:37 [I 2022-10-27 09:14:02] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00245 [I 2022-10-27 09:14:02] ...trap/engines/engine.py.230: loss: 0.25281 [I 2022-10-27 09:14:02] ...optimizers/trijoint.py.151: total_norm: 38.312580 [I 2022-10-27 09:14:07] ...trap/engines/engine.py.224: train: epoch 59 | batch 950/2304 [I 2022-10-27 09:14:07] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:32 [I 2022-10-27 09:14:07] ...trap/engines/engine.py.229: process: 0.55770 | load: 0.00245 [I 2022-10-27 09:14:07] ...trap/engines/engine.py.230: loss: 0.14752 [I 2022-10-27 09:14:07] ...optimizers/trijoint.py.151: total_norm: 19.566660 [I 2022-10-27 09:14:13] ...trap/engines/engine.py.224: train: epoch 59 | batch 960/2304 [I 2022-10-27 09:14:13] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:30 [I 2022-10-27 09:14:13] ...trap/engines/engine.py.229: process: 0.56002 | load: 0.00245 [I 2022-10-27 09:14:13] ...trap/engines/engine.py.230: loss: 0.17134 [I 2022-10-27 09:14:13] ...optimizers/trijoint.py.151: total_norm: 62.691261 [I 2022-10-27 09:14:19] ...trap/engines/engine.py.224: train: epoch 59 | batch 970/2304 [I 2022-10-27 09:14:19] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:24 [I 2022-10-27 09:14:19] ...trap/engines/engine.py.229: process: 0.56600 | load: 0.00244 [I 2022-10-27 09:14:19] ...trap/engines/engine.py.230: loss: 0.12413 [I 2022-10-27 09:14:19] ...optimizers/trijoint.py.151: total_norm: 15.585531 [I 2022-10-27 09:14:24] ...trap/engines/engine.py.224: train: epoch 59 | batch 980/2304 [I 2022-10-27 09:14:24] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:11 [I 2022-10-27 09:14:24] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00242 [I 2022-10-27 09:14:24] ...trap/engines/engine.py.230: loss: 0.15554 [I 2022-10-27 09:14:24] ...optimizers/trijoint.py.151: total_norm: 10.911736 [I 2022-10-27 09:14:30] ...trap/engines/engine.py.224: train: epoch 59 | batch 990/2304 [I 2022-10-27 09:14:30] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:12:04 [I 2022-10-27 09:14:30] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00262 [I 2022-10-27 09:14:30] ...trap/engines/engine.py.230: loss: 0.12664 [I 2022-10-27 09:14:30] ...optimizers/trijoint.py.151: total_norm: 14.248492 [I 2022-10-27 09:14:35] ...trap/engines/engine.py.224: train: epoch 59 | batch 1000/2304 [I 2022-10-27 09:14:35] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:12:00 [I 2022-10-27 09:14:35] ...trap/engines/engine.py.229: process: 0.54925 | load: 0.00240 [I 2022-10-27 09:14:35] ...trap/engines/engine.py.230: loss: 0.25465 [I 2022-10-27 09:14:35] ...optimizers/trijoint.py.151: total_norm: 17.118015 [I 2022-10-27 09:14:41] ...trap/engines/engine.py.224: train: epoch 59 | batch 1010/2304 [I 2022-10-27 09:14:41] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:57 [I 2022-10-27 09:14:41] ...trap/engines/engine.py.229: process: 0.56660 | load: 0.00262 [I 2022-10-27 09:14:41] ...trap/engines/engine.py.230: loss: 0.15213 [I 2022-10-27 09:14:41] ...optimizers/trijoint.py.151: total_norm: 17.558050 [I 2022-10-27 09:14:46] ...trap/engines/engine.py.224: train: epoch 59 | batch 1020/2304 [I 2022-10-27 09:14:46] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:51 [I 2022-10-27 09:14:46] ...trap/engines/engine.py.229: process: 0.54748 | load: 0.00239 [I 2022-10-27 09:14:46] ...trap/engines/engine.py.230: loss: 0.10856 [I 2022-10-27 09:14:46] ...optimizers/trijoint.py.151: total_norm: 7.720796 [I 2022-10-27 09:14:52] ...trap/engines/engine.py.224: train: epoch 59 | batch 1030/2304 [I 2022-10-27 09:14:52] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:45 [I 2022-10-27 09:14:52] ...trap/engines/engine.py.229: process: 0.54595 | load: 0.00242 [I 2022-10-27 09:14:52] ...trap/engines/engine.py.230: loss: 0.11956 [I 2022-10-27 09:14:52] ...optimizers/trijoint.py.151: total_norm: 24.643887 [I 2022-10-27 09:14:57] ...trap/engines/engine.py.224: train: epoch 59 | batch 1040/2304 [I 2022-10-27 09:14:57] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:43 [I 2022-10-27 09:14:57] ...trap/engines/engine.py.229: process: 0.55042 | load: 0.00241 [I 2022-10-27 09:14:57] ...trap/engines/engine.py.230: loss: 0.16405 [I 2022-10-27 09:14:57] ...optimizers/trijoint.py.151: total_norm: 13.380509 [I 2022-10-27 09:15:03] ...trap/engines/engine.py.224: train: epoch 59 | batch 1050/2304 [I 2022-10-27 09:15:03] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:41 [I 2022-10-27 09:15:03] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00244 [I 2022-10-27 09:15:03] ...trap/engines/engine.py.230: loss: 0.16693 [I 2022-10-27 09:15:03] ...optimizers/trijoint.py.151: total_norm: 17.467304 [I 2022-10-27 09:15:09] ...trap/engines/engine.py.224: train: epoch 59 | batch 1060/2304 [I 2022-10-27 09:15:09] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:33 [I 2022-10-27 09:15:09] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00240 [I 2022-10-27 09:15:09] ...trap/engines/engine.py.230: loss: 0.14655 [I 2022-10-27 09:15:09] ...optimizers/trijoint.py.151: total_norm: 14.677764 [I 2022-10-27 09:15:14] ...trap/engines/engine.py.224: train: epoch 59 | batch 1070/2304 [I 2022-10-27 09:15:14] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:27 [I 2022-10-27 09:15:14] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00243 [I 2022-10-27 09:15:14] ...trap/engines/engine.py.230: loss: 0.17193 [I 2022-10-27 09:15:14] ...optimizers/trijoint.py.151: total_norm: 28.665768 [I 2022-10-27 09:15:20] ...trap/engines/engine.py.224: train: epoch 59 | batch 1080/2304 [I 2022-10-27 09:15:20] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:20 [I 2022-10-27 09:15:20] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00240 [I 2022-10-27 09:15:20] ...trap/engines/engine.py.230: loss: 0.14159 [I 2022-10-27 09:15:20] ...optimizers/trijoint.py.151: total_norm: 19.307781 [I 2022-10-27 09:15:25] ...trap/engines/engine.py.224: train: epoch 59 | batch 1090/2304 [I 2022-10-27 09:15:25] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:12 [I 2022-10-27 09:15:25] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00242 [I 2022-10-27 09:15:25] ...trap/engines/engine.py.230: loss: 0.12183 [I 2022-10-27 09:15:25] ...optimizers/trijoint.py.151: total_norm: 14.607779 [I 2022-10-27 09:15:31] ...trap/engines/engine.py.224: train: epoch 59 | batch 1100/2304 [I 2022-10-27 09:15:31] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:11:11 [I 2022-10-27 09:15:31] ...trap/engines/engine.py.229: process: 0.56772 | load: 0.00244 [I 2022-10-27 09:15:31] ...trap/engines/engine.py.230: loss: 0.15661 [I 2022-10-27 09:15:31] ...optimizers/trijoint.py.151: total_norm: 15.977957 [I 2022-10-27 09:15:37] ...trap/engines/engine.py.224: train: epoch 59 | batch 1110/2304 [I 2022-10-27 09:15:37] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:11:04 [I 2022-10-27 09:15:37] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00245 [I 2022-10-27 09:15:37] ...trap/engines/engine.py.230: loss: 0.12913 [I 2022-10-27 09:15:37] ...optimizers/trijoint.py.151: total_norm: 17.313499 [I 2022-10-27 09:15:42] ...trap/engines/engine.py.224: train: epoch 59 | batch 1120/2304 [I 2022-10-27 09:15:42] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:55 [I 2022-10-27 09:15:42] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00241 [I 2022-10-27 09:15:42] ...trap/engines/engine.py.230: loss: 0.17179 [I 2022-10-27 09:15:42] ...optimizers/trijoint.py.151: total_norm: 19.009138 [I 2022-10-27 09:15:48] ...trap/engines/engine.py.224: train: epoch 59 | batch 1130/2304 [I 2022-10-27 09:15:48] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:56 [I 2022-10-27 09:15:48] ...trap/engines/engine.py.229: process: 0.56662 | load: 0.00242 [I 2022-10-27 09:15:48] ...trap/engines/engine.py.230: loss: 0.17893 [I 2022-10-27 09:15:48] ...optimizers/trijoint.py.151: total_norm: 18.149282 [I 2022-10-27 09:15:53] ...trap/engines/engine.py.224: train: epoch 59 | batch 1140/2304 [I 2022-10-27 09:15:53] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:11:01 [I 2022-10-27 09:15:53] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00241 [I 2022-10-27 09:15:53] ...trap/engines/engine.py.230: loss: 0.14103 [I 2022-10-27 09:15:53] ...optimizers/trijoint.py.151: total_norm: 12.818825 [I 2022-10-27 09:15:59] ...trap/engines/engine.py.224: train: epoch 59 | batch 1150/2304 [I 2022-10-27 09:15:59] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:43 [I 2022-10-27 09:15:59] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00240 [I 2022-10-27 09:15:59] ...trap/engines/engine.py.230: loss: 0.19137 [I 2022-10-27 09:15:59] ...optimizers/trijoint.py.151: total_norm: 22.670519 [I 2022-10-27 09:16:05] ...trap/engines/engine.py.224: train: epoch 59 | batch 1160/2304 [I 2022-10-27 09:16:05] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:41 [I 2022-10-27 09:16:05] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00241 [I 2022-10-27 09:16:05] ...trap/engines/engine.py.230: loss: 0.28453 [I 2022-10-27 09:16:05] ...optimizers/trijoint.py.151: total_norm: 16.459558 [I 2022-10-27 09:16:10] ...trap/engines/engine.py.224: train: epoch 59 | batch 1170/2304 [I 2022-10-27 09:16:10] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:29 [I 2022-10-27 09:16:10] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00243 [I 2022-10-27 09:16:10] ...trap/engines/engine.py.230: loss: 0.17759 [I 2022-10-27 09:16:10] ...optimizers/trijoint.py.151: total_norm: 16.549498 [I 2022-10-27 09:16:16] ...trap/engines/engine.py.224: train: epoch 59 | batch 1180/2304 [I 2022-10-27 09:16:16] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:21 [I 2022-10-27 09:16:16] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00242 [I 2022-10-27 09:16:16] ...trap/engines/engine.py.230: loss: 0.15068 [I 2022-10-27 09:16:16] ...optimizers/trijoint.py.151: total_norm: 13.809832 [I 2022-10-27 09:16:21] ...trap/engines/engine.py.224: train: epoch 59 | batch 1190/2304 [I 2022-10-27 09:16:21] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:19 [I 2022-10-27 09:16:21] ...trap/engines/engine.py.229: process: 0.56163 | load: 0.00240 [I 2022-10-27 09:16:21] ...trap/engines/engine.py.230: loss: 0.11090 [I 2022-10-27 09:16:21] ...optimizers/trijoint.py.151: total_norm: 16.253239 [I 2022-10-27 09:16:27] ...trap/engines/engine.py.224: train: epoch 59 | batch 1200/2304 [I 2022-10-27 09:16:27] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:15 [I 2022-10-27 09:16:27] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00242 [I 2022-10-27 09:16:27] ...trap/engines/engine.py.230: loss: 0.11151 [I 2022-10-27 09:16:27] ...optimizers/trijoint.py.151: total_norm: 14.730197 [I 2022-10-27 09:16:33] ...trap/engines/engine.py.224: train: epoch 59 | batch 1210/2304 [I 2022-10-27 09:16:33] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:10:11 [I 2022-10-27 09:16:33] ...trap/engines/engine.py.229: process: 0.56773 | load: 0.00241 [I 2022-10-27 09:16:33] ...trap/engines/engine.py.230: loss: 0.09634 [I 2022-10-27 09:16:33] ...optimizers/trijoint.py.151: total_norm: 9.286817 [I 2022-10-27 09:16:38] ...trap/engines/engine.py.224: train: epoch 59 | batch 1220/2304 [I 2022-10-27 09:16:38] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:10:34 [I 2022-10-27 09:16:38] ...trap/engines/engine.py.229: process: 0.63157 | load: 0.00270 [I 2022-10-27 09:16:38] ...trap/engines/engine.py.230: loss: 0.17355 [I 2022-10-27 09:16:38] ...optimizers/trijoint.py.151: total_norm: 9.911712 [I 2022-10-27 09:16:44] ...trap/engines/engine.py.224: train: epoch 59 | batch 1230/2304 [I 2022-10-27 09:16:44] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:57 [I 2022-10-27 09:16:44] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00256 [I 2022-10-27 09:16:44] ...trap/engines/engine.py.230: loss: 0.19051 [I 2022-10-27 09:16:44] ...optimizers/trijoint.py.151: total_norm: 75.201950 [I 2022-10-27 09:16:49] ...trap/engines/engine.py.224: train: epoch 59 | batch 1240/2304 [I 2022-10-27 09:16:49] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:52 [I 2022-10-27 09:16:49] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00246 [I 2022-10-27 09:16:49] ...trap/engines/engine.py.230: loss: 0.14883 [I 2022-10-27 09:16:49] ...optimizers/trijoint.py.151: total_norm: 19.640940 [I 2022-10-27 09:16:55] ...trap/engines/engine.py.224: train: epoch 59 | batch 1250/2304 [I 2022-10-27 09:16:55] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:45 [I 2022-10-27 09:16:55] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00244 [I 2022-10-27 09:16:55] ...trap/engines/engine.py.230: loss: 0.13854 [I 2022-10-27 09:16:55] ...optimizers/trijoint.py.151: total_norm: 16.690107 [I 2022-10-27 09:17:01] ...trap/engines/engine.py.224: train: epoch 59 | batch 1260/2304 [I 2022-10-27 09:17:01] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:40 [I 2022-10-27 09:17:01] ...trap/engines/engine.py.229: process: 0.56050 | load: 0.00245 [I 2022-10-27 09:17:01] ...trap/engines/engine.py.230: loss: 0.20455 [I 2022-10-27 09:17:01] ...optimizers/trijoint.py.151: total_norm: 11.634867 [I 2022-10-27 09:17:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 1270/2304 [I 2022-10-27 09:17:06] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:40 [I 2022-10-27 09:17:06] ...trap/engines/engine.py.229: process: 0.56617 | load: 0.00240 [I 2022-10-27 09:17:06] ...trap/engines/engine.py.230: loss: 0.13074 [I 2022-10-27 09:17:06] ...optimizers/trijoint.py.151: total_norm: 20.531641 [I 2022-10-27 09:17:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 1280/2304 [I 2022-10-27 09:17:12] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:27 [I 2022-10-27 09:17:12] ...trap/engines/engine.py.229: process: 0.54921 | load: 0.00242 [I 2022-10-27 09:17:12] ...trap/engines/engine.py.230: loss: 0.16608 [I 2022-10-27 09:17:12] ...optimizers/trijoint.py.151: total_norm: 16.256657 [I 2022-10-27 09:17:17] ...trap/engines/engine.py.224: train: epoch 59 | batch 1290/2304 [I 2022-10-27 09:17:17] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:29 [I 2022-10-27 09:17:17] ...trap/engines/engine.py.229: process: 0.54770 | load: 0.00242 [I 2022-10-27 09:17:17] ...trap/engines/engine.py.230: loss: 0.17537 [I 2022-10-27 09:17:17] ...optimizers/trijoint.py.151: total_norm: 25.777628 [I 2022-10-27 09:17:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 1300/2304 [I 2022-10-27 09:17:23] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:19 [I 2022-10-27 09:17:23] ...trap/engines/engine.py.229: process: 0.56411 | load: 0.00259 [I 2022-10-27 09:17:23] ...trap/engines/engine.py.230: loss: 0.14377 [I 2022-10-27 09:17:23] ...optimizers/trijoint.py.151: total_norm: 15.737576 [I 2022-10-27 09:17:29] ...trap/engines/engine.py.224: train: epoch 59 | batch 1310/2304 [I 2022-10-27 09:17:29] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:09:14 [I 2022-10-27 09:17:29] ...trap/engines/engine.py.229: process: 0.56514 | load: 0.00243 [I 2022-10-27 09:17:29] ...trap/engines/engine.py.230: loss: 0.18628 [I 2022-10-27 09:17:29] ...optimizers/trijoint.py.151: total_norm: 21.707664 [I 2022-10-27 09:17:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 1320/2304 [I 2022-10-27 09:17:34] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:09:07 [I 2022-10-27 09:17:34] ...trap/engines/engine.py.229: process: 0.55110 | load: 0.00242 [I 2022-10-27 09:17:34] ...trap/engines/engine.py.230: loss: 0.16752 [I 2022-10-27 09:17:34] ...optimizers/trijoint.py.151: total_norm: 19.262379 [I 2022-10-27 09:17:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 1330/2304 [I 2022-10-27 09:17:40] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:09:01 [I 2022-10-27 09:17:40] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00242 [I 2022-10-27 09:17:40] ...trap/engines/engine.py.230: loss: 0.09054 [I 2022-10-27 09:17:40] ...optimizers/trijoint.py.151: total_norm: 9.146585 [I 2022-10-27 09:17:45] ...trap/engines/engine.py.224: train: epoch 59 | batch 1340/2304 [I 2022-10-27 09:17:45] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:55 [I 2022-10-27 09:17:45] ...trap/engines/engine.py.229: process: 0.55048 | load: 0.00240 [I 2022-10-27 09:17:45] ...trap/engines/engine.py.230: loss: 0.12925 [I 2022-10-27 09:17:45] ...optimizers/trijoint.py.151: total_norm: 18.096346 [I 2022-10-27 09:17:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 1350/2304 [I 2022-10-27 09:17:51] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:51 [I 2022-10-27 09:17:51] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00245 [I 2022-10-27 09:17:51] ...trap/engines/engine.py.230: loss: 0.23211 [I 2022-10-27 09:17:51] ...optimizers/trijoint.py.151: total_norm: 14.586763 [I 2022-10-27 09:17:57] ...trap/engines/engine.py.224: train: epoch 59 | batch 1360/2304 [I 2022-10-27 09:17:57] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:49 [I 2022-10-27 09:17:57] ...trap/engines/engine.py.229: process: 0.54633 | load: 0.00243 [I 2022-10-27 09:17:57] ...trap/engines/engine.py.230: loss: 0.18028 [I 2022-10-27 09:17:57] ...optimizers/trijoint.py.151: total_norm: 16.715734 [I 2022-10-27 09:18:02] ...trap/engines/engine.py.224: train: epoch 59 | batch 1370/2304 [I 2022-10-27 09:18:02] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:40 [I 2022-10-27 09:18:02] ...trap/engines/engine.py.229: process: 0.56463 | load: 0.00243 [I 2022-10-27 09:18:02] ...trap/engines/engine.py.230: loss: 0.17105 [I 2022-10-27 09:18:02] ...optimizers/trijoint.py.151: total_norm: 18.705715 [I 2022-10-27 09:18:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 1380/2304 [I 2022-10-27 09:18:08] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:36 [I 2022-10-27 09:18:08] ...trap/engines/engine.py.229: process: 0.56402 | load: 0.00242 [I 2022-10-27 09:18:08] ...trap/engines/engine.py.230: loss: 0.12534 [I 2022-10-27 09:18:08] ...optimizers/trijoint.py.151: total_norm: 12.253850 [I 2022-10-27 09:18:13] ...trap/engines/engine.py.224: train: epoch 59 | batch 1390/2304 [I 2022-10-27 09:18:13] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:26 [I 2022-10-27 09:18:13] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00246 [I 2022-10-27 09:18:13] ...trap/engines/engine.py.230: loss: 0.20093 [I 2022-10-27 09:18:13] ...optimizers/trijoint.py.151: total_norm: 34.799515 [I 2022-10-27 09:18:19] ...trap/engines/engine.py.224: train: epoch 59 | batch 1400/2304 [I 2022-10-27 09:18:19] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:20 [I 2022-10-27 09:18:19] ...trap/engines/engine.py.229: process: 0.55094 | load: 0.00241 [I 2022-10-27 09:18:19] ...trap/engines/engine.py.230: loss: 0.13976 [I 2022-10-27 09:18:19] ...optimizers/trijoint.py.151: total_norm: 7.864932 [I 2022-10-27 09:18:24] ...trap/engines/engine.py.224: train: epoch 59 | batch 1410/2304 [I 2022-10-27 09:18:24] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:15 [I 2022-10-27 09:18:24] ...trap/engines/engine.py.229: process: 0.56557 | load: 0.00243 [I 2022-10-27 09:18:24] ...trap/engines/engine.py.230: loss: 0.18955 [I 2022-10-27 09:18:24] ...optimizers/trijoint.py.151: total_norm: 16.848080 [I 2022-10-27 09:18:30] ...trap/engines/engine.py.224: train: epoch 59 | batch 1420/2304 [I 2022-10-27 09:18:30] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:08:16 [I 2022-10-27 09:18:30] ...trap/engines/engine.py.229: process: 0.55905 | load: 0.00240 [I 2022-10-27 09:18:30] ...trap/engines/engine.py.230: loss: 0.14178 [I 2022-10-27 09:18:30] ...optimizers/trijoint.py.151: total_norm: 12.245479 [I 2022-10-27 09:18:36] ...trap/engines/engine.py.224: train: epoch 59 | batch 1430/2304 [I 2022-10-27 09:18:36] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:08:08 [I 2022-10-27 09:18:36] ...trap/engines/engine.py.229: process: 0.56329 | load: 0.00240 [I 2022-10-27 09:18:36] ...trap/engines/engine.py.230: loss: 0.12755 [I 2022-10-27 09:18:36] ...optimizers/trijoint.py.151: total_norm: 14.345275 [I 2022-10-27 09:18:41] ...trap/engines/engine.py.224: train: epoch 59 | batch 1440/2304 [I 2022-10-27 09:18:41] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:57 [I 2022-10-27 09:18:41] ...trap/engines/engine.py.229: process: 0.56163 | load: 0.00242 [I 2022-10-27 09:18:41] ...trap/engines/engine.py.230: loss: 0.23309 [I 2022-10-27 09:18:41] ...optimizers/trijoint.py.151: total_norm: 10.006476 [I 2022-10-27 09:18:47] ...trap/engines/engine.py.224: train: epoch 59 | batch 1450/2304 [I 2022-10-27 09:18:47] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:53 [I 2022-10-27 09:18:47] ...trap/engines/engine.py.229: process: 0.56268 | load: 0.00239 [I 2022-10-27 09:18:47] ...trap/engines/engine.py.230: loss: 0.12204 [I 2022-10-27 09:18:47] ...optimizers/trijoint.py.151: total_norm: 18.281155 [I 2022-10-27 09:18:52] ...trap/engines/engine.py.224: train: epoch 59 | batch 1460/2304 [I 2022-10-27 09:18:52] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:51 [I 2022-10-27 09:18:52] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00249 [I 2022-10-27 09:18:52] ...trap/engines/engine.py.230: loss: 0.12708 [I 2022-10-27 09:18:52] ...optimizers/trijoint.py.151: total_norm: 13.812140 [I 2022-10-27 09:18:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 1470/2304 [I 2022-10-27 09:18:58] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:43 [I 2022-10-27 09:18:58] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00242 [I 2022-10-27 09:18:58] ...trap/engines/engine.py.230: loss: 0.16161 [I 2022-10-27 09:18:58] ...optimizers/trijoint.py.151: total_norm: 17.876068 [I 2022-10-27 09:19:04] ...trap/engines/engine.py.224: train: epoch 59 | batch 1480/2304 [I 2022-10-27 09:19:04] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:40 [I 2022-10-27 09:19:04] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00242 [I 2022-10-27 09:19:04] ...trap/engines/engine.py.230: loss: 0.11751 [I 2022-10-27 09:19:04] ...optimizers/trijoint.py.151: total_norm: 13.212705 [I 2022-10-27 09:19:09] ...trap/engines/engine.py.224: train: epoch 59 | batch 1490/2304 [I 2022-10-27 09:19:09] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:37 [I 2022-10-27 09:19:09] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00245 [I 2022-10-27 09:19:09] ...trap/engines/engine.py.230: loss: 0.18206 [I 2022-10-27 09:19:09] ...optimizers/trijoint.py.151: total_norm: 21.728516 [I 2022-10-27 09:19:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 1500/2304 [I 2022-10-27 09:19:15] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:35 [I 2022-10-27 09:19:15] ...trap/engines/engine.py.229: process: 0.56850 | load: 0.00257 [I 2022-10-27 09:19:15] ...trap/engines/engine.py.230: loss: 0.13244 [I 2022-10-27 09:19:15] ...optimizers/trijoint.py.151: total_norm: 14.591120 [I 2022-10-27 09:19:21] ...trap/engines/engine.py.224: train: epoch 59 | batch 1510/2304 [I 2022-10-27 09:19:21] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:23 [I 2022-10-27 09:19:21] ...trap/engines/engine.py.229: process: 0.55793 | load: 0.00246 [I 2022-10-27 09:19:21] ...trap/engines/engine.py.230: loss: 0.13822 [I 2022-10-27 09:19:21] ...optimizers/trijoint.py.151: total_norm: 9.681518 [I 2022-10-27 09:19:26] ...trap/engines/engine.py.224: train: epoch 59 | batch 1520/2304 [I 2022-10-27 09:19:26] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:18 [I 2022-10-27 09:19:26] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00244 [I 2022-10-27 09:19:26] ...trap/engines/engine.py.230: loss: 0.16453 [I 2022-10-27 09:19:26] ...optimizers/trijoint.py.151: total_norm: 20.680147 [I 2022-10-27 09:19:32] ...trap/engines/engine.py.224: train: epoch 59 | batch 1530/2304 [I 2022-10-27 09:19:32] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:07:12 [I 2022-10-27 09:19:32] ...trap/engines/engine.py.229: process: 0.56807 | load: 0.00244 [I 2022-10-27 09:19:32] ...trap/engines/engine.py.230: loss: 0.21193 [I 2022-10-27 09:19:32] ...optimizers/trijoint.py.151: total_norm: 16.759762 [I 2022-10-27 09:19:37] ...trap/engines/engine.py.224: train: epoch 59 | batch 1540/2304 [I 2022-10-27 09:19:37] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:07:06 [I 2022-10-27 09:19:37] ...trap/engines/engine.py.229: process: 0.55496 | load: 0.00242 [I 2022-10-27 09:19:37] ...trap/engines/engine.py.230: loss: 0.11787 [I 2022-10-27 09:19:37] ...optimizers/trijoint.py.151: total_norm: 22.750671 [I 2022-10-27 09:19:43] ...trap/engines/engine.py.224: train: epoch 59 | batch 1550/2304 [I 2022-10-27 09:19:43] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:58 [I 2022-10-27 09:19:43] ...trap/engines/engine.py.229: process: 0.54542 | load: 0.00241 [I 2022-10-27 09:19:43] ...trap/engines/engine.py.230: loss: 0.16687 [I 2022-10-27 09:19:43] ...optimizers/trijoint.py.151: total_norm: 41.758007 [I 2022-10-27 09:19:48] ...trap/engines/engine.py.224: train: epoch 59 | batch 1560/2304 [I 2022-10-27 09:19:48] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:53 [I 2022-10-27 09:19:48] ...trap/engines/engine.py.229: process: 0.56744 | load: 0.00240 [I 2022-10-27 09:19:48] ...trap/engines/engine.py.230: loss: 0.17682 [I 2022-10-27 09:19:48] ...optimizers/trijoint.py.151: total_norm: 12.464038 [I 2022-10-27 09:19:54] ...trap/engines/engine.py.224: train: epoch 59 | batch 1570/2304 [I 2022-10-27 09:19:54] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:50 [I 2022-10-27 09:19:54] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00240 [I 2022-10-27 09:19:54] ...trap/engines/engine.py.230: loss: 0.19900 [I 2022-10-27 09:19:54] ...optimizers/trijoint.py.151: total_norm: 17.828609 [I 2022-10-27 09:20:00] ...trap/engines/engine.py.224: train: epoch 59 | batch 1580/2304 [I 2022-10-27 09:20:00] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:44 [I 2022-10-27 09:20:00] ...trap/engines/engine.py.229: process: 0.54904 | load: 0.00242 [I 2022-10-27 09:20:00] ...trap/engines/engine.py.230: loss: 0.11382 [I 2022-10-27 09:20:00] ...optimizers/trijoint.py.151: total_norm: 13.699507 [I 2022-10-27 09:20:05] ...trap/engines/engine.py.224: train: epoch 59 | batch 1590/2304 [I 2022-10-27 09:20:05] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:35 [I 2022-10-27 09:20:05] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00240 [I 2022-10-27 09:20:05] ...trap/engines/engine.py.230: loss: 0.23500 [I 2022-10-27 09:20:05] ...optimizers/trijoint.py.151: total_norm: 17.646605 [I 2022-10-27 09:20:11] ...trap/engines/engine.py.224: train: epoch 59 | batch 1600/2304 [I 2022-10-27 09:20:11] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:31 [I 2022-10-27 09:20:11] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00240 [I 2022-10-27 09:20:11] ...trap/engines/engine.py.230: loss: 0.14111 [I 2022-10-27 09:20:11] ...optimizers/trijoint.py.151: total_norm: 12.280763 [I 2022-10-27 09:20:16] ...trap/engines/engine.py.224: train: epoch 59 | batch 1610/2304 [I 2022-10-27 09:20:16] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:40 [I 2022-10-27 09:20:16] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00242 [I 2022-10-27 09:20:16] ...trap/engines/engine.py.230: loss: 0.17940 [I 2022-10-27 09:20:16] ...optimizers/trijoint.py.151: total_norm: 11.252295 [I 2022-10-27 09:20:22] ...trap/engines/engine.py.224: train: epoch 59 | batch 1620/2304 [I 2022-10-27 09:20:22] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:23 [I 2022-10-27 09:20:22] ...trap/engines/engine.py.229: process: 0.57163 | load: 0.00242 [I 2022-10-27 09:20:22] ...trap/engines/engine.py.230: loss: 0.20773 [I 2022-10-27 09:20:22] ...optimizers/trijoint.py.151: total_norm: 20.025240 [I 2022-10-27 09:20:28] ...trap/engines/engine.py.224: train: epoch 59 | batch 1630/2304 [I 2022-10-27 09:20:28] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:15 [I 2022-10-27 09:20:28] ...trap/engines/engine.py.229: process: 0.56137 | load: 0.00243 [I 2022-10-27 09:20:28] ...trap/engines/engine.py.230: loss: 0.15027 [I 2022-10-27 09:20:28] ...optimizers/trijoint.py.151: total_norm: 18.782440 [I 2022-10-27 09:20:33] ...trap/engines/engine.py.224: train: epoch 59 | batch 1640/2304 [I 2022-10-27 09:20:33] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:07 [I 2022-10-27 09:20:33] ...trap/engines/engine.py.229: process: 0.55804 | load: 0.00240 [I 2022-10-27 09:20:33] ...trap/engines/engine.py.230: loss: 0.13356 [I 2022-10-27 09:20:33] ...optimizers/trijoint.py.151: total_norm: 27.861816 [I 2022-10-27 09:20:39] ...trap/engines/engine.py.224: train: epoch 59 | batch 1650/2304 [I 2022-10-27 09:20:39] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:06:02 [I 2022-10-27 09:20:39] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00242 [I 2022-10-27 09:20:39] ...trap/engines/engine.py.230: loss: 0.17413 [I 2022-10-27 09:20:39] ...optimizers/trijoint.py.151: total_norm: 22.075411 [I 2022-10-27 09:20:44] ...trap/engines/engine.py.224: train: epoch 59 | batch 1660/2304 [I 2022-10-27 09:20:44] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:55 [I 2022-10-27 09:20:44] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00242 [I 2022-10-27 09:20:44] ...trap/engines/engine.py.230: loss: 0.18266 [I 2022-10-27 09:20:44] ...optimizers/trijoint.py.151: total_norm: 20.481194 [I 2022-10-27 09:20:50] ...trap/engines/engine.py.224: train: epoch 59 | batch 1670/2304 [I 2022-10-27 09:20:50] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:52 [I 2022-10-27 09:20:50] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00242 [I 2022-10-27 09:20:50] ...trap/engines/engine.py.230: loss: 0.12762 [I 2022-10-27 09:20:50] ...optimizers/trijoint.py.151: total_norm: 13.913330 [I 2022-10-27 09:20:56] ...trap/engines/engine.py.224: train: epoch 59 | batch 1680/2304 [I 2022-10-27 09:20:56] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:49 [I 2022-10-27 09:20:56] ...trap/engines/engine.py.229: process: 0.55342 | load: 0.00241 [I 2022-10-27 09:20:56] ...trap/engines/engine.py.230: loss: 0.13238 [I 2022-10-27 09:20:56] ...optimizers/trijoint.py.151: total_norm: 16.768929 [I 2022-10-27 09:21:01] ...trap/engines/engine.py.224: train: epoch 59 | batch 1690/2304 [I 2022-10-27 09:21:01] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:42 [I 2022-10-27 09:21:01] ...trap/engines/engine.py.229: process: 0.55623 | load: 0.00240 [I 2022-10-27 09:21:01] ...trap/engines/engine.py.230: loss: 0.15023 [I 2022-10-27 09:21:01] ...optimizers/trijoint.py.151: total_norm: 18.586857 [I 2022-10-27 09:21:07] ...trap/engines/engine.py.224: train: epoch 59 | batch 1700/2304 [I 2022-10-27 09:21:07] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:35 [I 2022-10-27 09:21:07] ...trap/engines/engine.py.229: process: 0.55562 | load: 0.00242 [I 2022-10-27 09:21:07] ...trap/engines/engine.py.230: loss: 0.14854 [I 2022-10-27 09:21:07] ...optimizers/trijoint.py.151: total_norm: 15.613017 [I 2022-10-27 09:21:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 1710/2304 [I 2022-10-27 09:21:12] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:30 [I 2022-10-27 09:21:12] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00242 [I 2022-10-27 09:21:12] ...trap/engines/engine.py.230: loss: 0.15052 [I 2022-10-27 09:21:12] ...optimizers/trijoint.py.151: total_norm: 25.492771 [I 2022-10-27 09:21:18] ...trap/engines/engine.py.224: train: epoch 59 | batch 1720/2304 [I 2022-10-27 09:21:18] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:22 [I 2022-10-27 09:21:18] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00242 [I 2022-10-27 09:21:18] ...trap/engines/engine.py.230: loss: 0.13656 [I 2022-10-27 09:21:18] ...optimizers/trijoint.py.151: total_norm: 17.517277 [I 2022-10-27 09:21:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 1730/2304 [I 2022-10-27 09:21:23] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:20 [I 2022-10-27 09:21:23] ...trap/engines/engine.py.229: process: 0.55518 | load: 0.00243 [I 2022-10-27 09:21:23] ...trap/engines/engine.py.230: loss: 0.14187 [I 2022-10-27 09:21:23] ...optimizers/trijoint.py.151: total_norm: 10.612102 [I 2022-10-27 09:21:29] ...trap/engines/engine.py.224: train: epoch 59 | batch 1740/2304 [I 2022-10-27 09:21:29] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:13 [I 2022-10-27 09:21:29] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00239 [I 2022-10-27 09:21:29] ...trap/engines/engine.py.230: loss: 0.12959 [I 2022-10-27 09:21:29] ...optimizers/trijoint.py.151: total_norm: 20.780535 [I 2022-10-27 09:21:35] ...trap/engines/engine.py.224: train: epoch 59 | batch 1750/2304 [I 2022-10-27 09:21:35] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:05:08 [I 2022-10-27 09:21:35] ...trap/engines/engine.py.229: process: 0.54862 | load: 0.00241 [I 2022-10-27 09:21:35] ...trap/engines/engine.py.230: loss: 0.16358 [I 2022-10-27 09:21:35] ...optimizers/trijoint.py.151: total_norm: 20.154997 [I 2022-10-27 09:21:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 1760/2304 [I 2022-10-27 09:21:40] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:01 [I 2022-10-27 09:21:40] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00242 [I 2022-10-27 09:21:40] ...trap/engines/engine.py.230: loss: 0.17265 [I 2022-10-27 09:21:40] ...optimizers/trijoint.py.151: total_norm: 15.833108 [I 2022-10-27 09:21:46] ...trap/engines/engine.py.224: train: epoch 59 | batch 1770/2304 [I 2022-10-27 09:21:46] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:56 [I 2022-10-27 09:21:46] ...trap/engines/engine.py.229: process: 0.56599 | load: 0.00241 [I 2022-10-27 09:21:46] ...trap/engines/engine.py.230: loss: 0.16606 [I 2022-10-27 09:21:46] ...optimizers/trijoint.py.151: total_norm: 40.908348 [I 2022-10-27 09:21:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 1780/2304 [I 2022-10-27 09:21:51] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:48 [I 2022-10-27 09:21:51] ...trap/engines/engine.py.229: process: 0.53695 | load: 0.00243 [I 2022-10-27 09:21:51] ...trap/engines/engine.py.230: loss: 0.16846 [I 2022-10-27 09:21:51] ...optimizers/trijoint.py.151: total_norm: 22.876019 [I 2022-10-27 09:21:57] ...trap/engines/engine.py.224: train: epoch 59 | batch 1790/2304 [I 2022-10-27 09:21:57] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:47 [I 2022-10-27 09:21:57] ...trap/engines/engine.py.229: process: 0.56744 | load: 0.00239 [I 2022-10-27 09:21:57] ...trap/engines/engine.py.230: loss: 0.11424 [I 2022-10-27 09:21:57] ...optimizers/trijoint.py.151: total_norm: 11.846224 [I 2022-10-27 09:22:02] ...trap/engines/engine.py.224: train: epoch 59 | batch 1800/2304 [I 2022-10-27 09:22:02] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:42 [I 2022-10-27 09:22:02] ...trap/engines/engine.py.229: process: 0.56296 | load: 0.00242 [I 2022-10-27 09:22:02] ...trap/engines/engine.py.230: loss: 0.20012 [I 2022-10-27 09:22:02] ...optimizers/trijoint.py.151: total_norm: 51.859020 [I 2022-10-27 09:22:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 1810/2304 [I 2022-10-27 09:22:08] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:31 [I 2022-10-27 09:22:08] ...trap/engines/engine.py.229: process: 0.54602 | load: 0.00240 [I 2022-10-27 09:22:08] ...trap/engines/engine.py.230: loss: 0.15541 [I 2022-10-27 09:22:08] ...optimizers/trijoint.py.151: total_norm: 10.219913 [I 2022-10-27 09:22:14] ...trap/engines/engine.py.224: train: epoch 59 | batch 1820/2304 [I 2022-10-27 09:22:14] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:28 [I 2022-10-27 09:22:14] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00253 [I 2022-10-27 09:22:14] ...trap/engines/engine.py.230: loss: 0.14111 [I 2022-10-27 09:22:14] ...optimizers/trijoint.py.151: total_norm: 14.023665 [I 2022-10-27 09:22:19] ...trap/engines/engine.py.224: train: epoch 59 | batch 1830/2304 [I 2022-10-27 09:22:19] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:24 [I 2022-10-27 09:22:19] ...trap/engines/engine.py.229: process: 0.54607 | load: 0.00248 [I 2022-10-27 09:22:19] ...trap/engines/engine.py.230: loss: 0.20953 [I 2022-10-27 09:22:19] ...optimizers/trijoint.py.151: total_norm: 21.635103 [I 2022-10-27 09:22:25] ...trap/engines/engine.py.224: train: epoch 59 | batch 1840/2304 [I 2022-10-27 09:22:25] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:17 [I 2022-10-27 09:22:25] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00246 [I 2022-10-27 09:22:25] ...trap/engines/engine.py.230: loss: 0.16392 [I 2022-10-27 09:22:25] ...optimizers/trijoint.py.151: total_norm: 12.811388 [I 2022-10-27 09:22:30] ...trap/engines/engine.py.224: train: epoch 59 | batch 1850/2304 [I 2022-10-27 09:22:30] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:04:12 [I 2022-10-27 09:22:30] ...trap/engines/engine.py.229: process: 0.55834 | load: 0.00242 [I 2022-10-27 09:22:30] ...trap/engines/engine.py.230: loss: 0.15214 [I 2022-10-27 09:22:30] ...optimizers/trijoint.py.151: total_norm: 13.996371 [I 2022-10-27 09:22:36] ...trap/engines/engine.py.224: train: epoch 59 | batch 1860/2304 [I 2022-10-27 09:22:36] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:04:06 [I 2022-10-27 09:22:36] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00242 [I 2022-10-27 09:22:36] ...trap/engines/engine.py.230: loss: 0.16725 [I 2022-10-27 09:22:36] ...optimizers/trijoint.py.151: total_norm: 19.255070 [I 2022-10-27 09:22:41] ...trap/engines/engine.py.224: train: epoch 59 | batch 1870/2304 [I 2022-10-27 09:22:41] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:04:01 [I 2022-10-27 09:22:41] ...trap/engines/engine.py.229: process: 0.56490 | load: 0.00240 [I 2022-10-27 09:22:41] ...trap/engines/engine.py.230: loss: 0.16351 [I 2022-10-27 09:22:41] ...optimizers/trijoint.py.151: total_norm: 14.043953 [I 2022-10-27 09:22:47] ...trap/engines/engine.py.224: train: epoch 59 | batch 1880/2304 [I 2022-10-27 09:22:47] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:56 [I 2022-10-27 09:22:47] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00243 [I 2022-10-27 09:22:47] ...trap/engines/engine.py.230: loss: 0.13889 [I 2022-10-27 09:22:47] ...optimizers/trijoint.py.151: total_norm: 20.988787 [I 2022-10-27 09:22:53] ...trap/engines/engine.py.224: train: epoch 59 | batch 1890/2304 [I 2022-10-27 09:22:53] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:50 [I 2022-10-27 09:22:53] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00243 [I 2022-10-27 09:22:53] ...trap/engines/engine.py.230: loss: 0.20584 [I 2022-10-27 09:22:53] ...optimizers/trijoint.py.151: total_norm: 23.880003 [I 2022-10-27 09:22:58] ...trap/engines/engine.py.224: train: epoch 59 | batch 1900/2304 [I 2022-10-27 09:22:58] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:44 [I 2022-10-27 09:22:58] ...trap/engines/engine.py.229: process: 0.56941 | load: 0.00241 [I 2022-10-27 09:22:58] ...trap/engines/engine.py.230: loss: 0.12800 [I 2022-10-27 09:22:58] ...optimizers/trijoint.py.151: total_norm: 16.009701 [I 2022-10-27 09:23:04] ...trap/engines/engine.py.224: train: epoch 59 | batch 1910/2304 [I 2022-10-27 09:23:04] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:38 [I 2022-10-27 09:23:04] ...trap/engines/engine.py.229: process: 0.53980 | load: 0.00245 [I 2022-10-27 09:23:04] ...trap/engines/engine.py.230: loss: 0.15238 [I 2022-10-27 09:23:04] ...optimizers/trijoint.py.151: total_norm: 15.607775 [I 2022-10-27 09:23:09] ...trap/engines/engine.py.224: train: epoch 59 | batch 1920/2304 [I 2022-10-27 09:23:09] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:32 [I 2022-10-27 09:23:09] ...trap/engines/engine.py.229: process: 0.54469 | load: 0.00241 [I 2022-10-27 09:23:09] ...trap/engines/engine.py.230: loss: 0.16235 [I 2022-10-27 09:23:09] ...optimizers/trijoint.py.151: total_norm: 16.493076 [I 2022-10-27 09:23:15] ...trap/engines/engine.py.224: train: epoch 59 | batch 1930/2304 [I 2022-10-27 09:23:15] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:26 [I 2022-10-27 09:23:15] ...trap/engines/engine.py.229: process: 0.54122 | load: 0.00242 [I 2022-10-27 09:23:15] ...trap/engines/engine.py.230: loss: 0.26671 [I 2022-10-27 09:23:15] ...optimizers/trijoint.py.151: total_norm: 29.175962 [I 2022-10-27 09:23:20] ...trap/engines/engine.py.224: train: epoch 59 | batch 1940/2304 [I 2022-10-27 09:23:20] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:22 [I 2022-10-27 09:23:20] ...trap/engines/engine.py.229: process: 0.54997 | load: 0.00242 [I 2022-10-27 09:23:20] ...trap/engines/engine.py.230: loss: 0.13109 [I 2022-10-27 09:23:20] ...optimizers/trijoint.py.151: total_norm: 17.263176 [I 2022-10-27 09:23:26] ...trap/engines/engine.py.224: train: epoch 59 | batch 1950/2304 [I 2022-10-27 09:23:26] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:16 [I 2022-10-27 09:23:26] ...trap/engines/engine.py.229: process: 0.56116 | load: 0.00242 [I 2022-10-27 09:23:26] ...trap/engines/engine.py.230: loss: 0.12948 [I 2022-10-27 09:23:26] ...optimizers/trijoint.py.151: total_norm: 10.793636 [I 2022-10-27 09:23:31] ...trap/engines/engine.py.224: train: epoch 59 | batch 1960/2304 [I 2022-10-27 09:23:31] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:10 [I 2022-10-27 09:23:31] ...trap/engines/engine.py.229: process: 0.54333 | load: 0.00239 [I 2022-10-27 09:23:31] ...trap/engines/engine.py.230: loss: 0.20530 [I 2022-10-27 09:23:31] ...optimizers/trijoint.py.151: total_norm: 15.793097 [I 2022-10-27 09:23:37] ...trap/engines/engine.py.224: train: epoch 59 | batch 1970/2304 [I 2022-10-27 09:23:37] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:03:04 [I 2022-10-27 09:23:37] ...trap/engines/engine.py.229: process: 0.54783 | load: 0.00241 [I 2022-10-27 09:23:37] ...trap/engines/engine.py.230: loss: 0.19118 [I 2022-10-27 09:23:37] ...optimizers/trijoint.py.151: total_norm: 11.444770 [I 2022-10-27 09:23:43] ...trap/engines/engine.py.224: train: epoch 59 | batch 1980/2304 [I 2022-10-27 09:23:43] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:59 [I 2022-10-27 09:23:43] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00242 [I 2022-10-27 09:23:43] ...trap/engines/engine.py.230: loss: 0.16758 [I 2022-10-27 09:23:43] ...optimizers/trijoint.py.151: total_norm: 30.444391 [I 2022-10-27 09:23:48] ...trap/engines/engine.py.224: train: epoch 59 | batch 1990/2304 [I 2022-10-27 09:23:48] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:54 [I 2022-10-27 09:23:48] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00246 [I 2022-10-27 09:23:48] ...trap/engines/engine.py.230: loss: 0.11227 [I 2022-10-27 09:23:48] ...optimizers/trijoint.py.151: total_norm: 16.592978 [I 2022-10-27 09:23:54] ...trap/engines/engine.py.224: train: epoch 59 | batch 2000/2304 [I 2022-10-27 09:23:54] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:49 [I 2022-10-27 09:23:54] ...trap/engines/engine.py.229: process: 0.56693 | load: 0.00244 [I 2022-10-27 09:23:54] ...trap/engines/engine.py.230: loss: 0.23088 [I 2022-10-27 09:23:54] ...optimizers/trijoint.py.151: total_norm: 29.542046 [I 2022-10-27 09:23:59] ...trap/engines/engine.py.224: train: epoch 59 | batch 2010/2304 [I 2022-10-27 09:23:59] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:43 [I 2022-10-27 09:23:59] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00245 [I 2022-10-27 09:23:59] ...trap/engines/engine.py.230: loss: 0.20101 [I 2022-10-27 09:23:59] ...optimizers/trijoint.py.151: total_norm: 14.681931 [I 2022-10-27 09:24:05] ...trap/engines/engine.py.224: train: epoch 59 | batch 2020/2304 [I 2022-10-27 09:24:05] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:38 [I 2022-10-27 09:24:05] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00243 [I 2022-10-27 09:24:05] ...trap/engines/engine.py.230: loss: 0.11101 [I 2022-10-27 09:24:05] ...optimizers/trijoint.py.151: total_norm: 9.306707 [I 2022-10-27 09:24:11] ...trap/engines/engine.py.224: train: epoch 59 | batch 2030/2304 [I 2022-10-27 09:24:11] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:32 [I 2022-10-27 09:24:11] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00248 [I 2022-10-27 09:24:11] ...trap/engines/engine.py.230: loss: 0.22220 [I 2022-10-27 09:24:11] ...optimizers/trijoint.py.151: total_norm: 17.212162 [I 2022-10-27 09:24:16] ...trap/engines/engine.py.224: train: epoch 59 | batch 2040/2304 [I 2022-10-27 09:24:16] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:27 [I 2022-10-27 09:24:16] ...trap/engines/engine.py.229: process: 0.55720 | load: 0.00248 [I 2022-10-27 09:24:16] ...trap/engines/engine.py.230: loss: 0.14450 [I 2022-10-27 09:24:16] ...optimizers/trijoint.py.151: total_norm: 14.846775 [I 2022-10-27 09:24:22] ...trap/engines/engine.py.224: train: epoch 59 | batch 2050/2304 [I 2022-10-27 09:24:22] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:21 [I 2022-10-27 09:24:22] ...trap/engines/engine.py.229: process: 0.54736 | load: 0.00264 [I 2022-10-27 09:24:22] ...trap/engines/engine.py.230: loss: 0.16151 [I 2022-10-27 09:24:22] ...optimizers/trijoint.py.151: total_norm: 10.171891 [I 2022-10-27 09:24:27] ...trap/engines/engine.py.224: train: epoch 59 | batch 2060/2304 [I 2022-10-27 09:24:27] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:16 [I 2022-10-27 09:24:27] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00247 [I 2022-10-27 09:24:27] ...trap/engines/engine.py.230: loss: 0.10396 [I 2022-10-27 09:24:27] ...optimizers/trijoint.py.151: total_norm: 12.417486 [I 2022-10-27 09:24:33] ...trap/engines/engine.py.224: train: epoch 59 | batch 2070/2304 [I 2022-10-27 09:24:33] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:02:10 [I 2022-10-27 09:24:33] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00240 [I 2022-10-27 09:24:33] ...trap/engines/engine.py.230: loss: 0.19913 [I 2022-10-27 09:24:33] ...optimizers/trijoint.py.151: total_norm: 21.979925 [I 2022-10-27 09:24:38] ...trap/engines/engine.py.224: train: epoch 59 | batch 2080/2304 [I 2022-10-27 09:24:38] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:05 [I 2022-10-27 09:24:38] ...trap/engines/engine.py.229: process: 0.56637 | load: 0.00247 [I 2022-10-27 09:24:38] ...trap/engines/engine.py.230: loss: 0.16509 [I 2022-10-27 09:24:38] ...optimizers/trijoint.py.151: total_norm: 25.290606 [I 2022-10-27 09:24:44] ...trap/engines/engine.py.224: train: epoch 59 | batch 2090/2304 [I 2022-10-27 09:24:44] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:01:58 [I 2022-10-27 09:24:44] ...trap/engines/engine.py.229: process: 0.54632 | load: 0.00242 [I 2022-10-27 09:24:44] ...trap/engines/engine.py.230: loss: 0.16210 [I 2022-10-27 09:24:44] ...optimizers/trijoint.py.151: total_norm: 15.208715 [I 2022-10-27 09:24:50] ...trap/engines/engine.py.224: train: epoch 59 | batch 2100/2304 [I 2022-10-27 09:24:50] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:53 [I 2022-10-27 09:24:50] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00247 [I 2022-10-27 09:24:50] ...trap/engines/engine.py.230: loss: 0.14730 [I 2022-10-27 09:24:50] ...optimizers/trijoint.py.151: total_norm: 13.702182 [I 2022-10-27 09:24:55] ...trap/engines/engine.py.224: train: epoch 59 | batch 2110/2304 [I 2022-10-27 09:24:55] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:48 [I 2022-10-27 09:24:55] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00248 [I 2022-10-27 09:24:55] ...trap/engines/engine.py.230: loss: 0.16120 [I 2022-10-27 09:24:55] ...optimizers/trijoint.py.151: total_norm: 26.815252 [I 2022-10-27 09:25:01] ...trap/engines/engine.py.224: train: epoch 59 | batch 2120/2304 [I 2022-10-27 09:25:01] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:42 [I 2022-10-27 09:25:01] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00246 [I 2022-10-27 09:25:01] ...trap/engines/engine.py.230: loss: 0.15405 [I 2022-10-27 09:25:01] ...optimizers/trijoint.py.151: total_norm: 13.101689 [I 2022-10-27 09:25:06] ...trap/engines/engine.py.224: train: epoch 59 | batch 2130/2304 [I 2022-10-27 09:25:06] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:36 [I 2022-10-27 09:25:06] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00245 [I 2022-10-27 09:25:06] ...trap/engines/engine.py.230: loss: 0.14212 [I 2022-10-27 09:25:06] ...optimizers/trijoint.py.151: total_norm: 19.531879 [I 2022-10-27 09:25:12] ...trap/engines/engine.py.224: train: epoch 59 | batch 2140/2304 [I 2022-10-27 09:25:12] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:31 [I 2022-10-27 09:25:12] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00248 [I 2022-10-27 09:25:12] ...trap/engines/engine.py.230: loss: 0.24999 [I 2022-10-27 09:25:12] ...optimizers/trijoint.py.151: total_norm: 10.332032 [I 2022-10-27 09:25:18] ...trap/engines/engine.py.224: train: epoch 59 | batch 2150/2304 [I 2022-10-27 09:25:18] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:25 [I 2022-10-27 09:25:18] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00243 [I 2022-10-27 09:25:18] ...trap/engines/engine.py.230: loss: 0.14194 [I 2022-10-27 09:25:18] ...optimizers/trijoint.py.151: total_norm: 30.738667 [I 2022-10-27 09:25:23] ...trap/engines/engine.py.224: train: epoch 59 | batch 2160/2304 [I 2022-10-27 09:25:23] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:20 [I 2022-10-27 09:25:23] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00245 [I 2022-10-27 09:25:23] ...trap/engines/engine.py.230: loss: 0.14529 [I 2022-10-27 09:25:23] ...optimizers/trijoint.py.151: total_norm: 9.402356 [I 2022-10-27 09:25:29] ...trap/engines/engine.py.224: train: epoch 59 | batch 2170/2304 [I 2022-10-27 09:25:29] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:01:14 [I 2022-10-27 09:25:29] ...trap/engines/engine.py.229: process: 0.56210 | load: 0.00239 [I 2022-10-27 09:25:29] ...trap/engines/engine.py.230: loss: 0.21071 [I 2022-10-27 09:25:29] ...optimizers/trijoint.py.151: total_norm: 15.105213 [I 2022-10-27 09:25:34] ...trap/engines/engine.py.224: train: epoch 59 | batch 2180/2304 [I 2022-10-27 09:25:34] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:01:08 [I 2022-10-27 09:25:34] ...trap/engines/engine.py.229: process: 0.54932 | load: 0.00243 [I 2022-10-27 09:25:34] ...trap/engines/engine.py.230: loss: 0.13991 [I 2022-10-27 09:25:34] ...optimizers/trijoint.py.151: total_norm: 11.448423 [I 2022-10-27 09:25:40] ...trap/engines/engine.py.224: train: epoch 59 | batch 2190/2304 [I 2022-10-27 09:25:40] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:01:03 [I 2022-10-27 09:25:40] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00242 [I 2022-10-27 09:25:40] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-27 09:25:40] ...optimizers/trijoint.py.151: total_norm: 14.455384 [I 2022-10-27 09:25:45] ...trap/engines/engine.py.224: train: epoch 59 | batch 2200/2304 [I 2022-10-27 09:25:45] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:57 [I 2022-10-27 09:25:45] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00244 [I 2022-10-27 09:25:45] ...trap/engines/engine.py.230: loss: 0.20377 [I 2022-10-27 09:25:45] ...optimizers/trijoint.py.151: total_norm: 24.643299 [I 2022-10-27 09:25:51] ...trap/engines/engine.py.224: train: epoch 59 | batch 2210/2304 [I 2022-10-27 09:25:51] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:52 [I 2022-10-27 09:25:51] ...trap/engines/engine.py.229: process: 0.56343 | load: 0.00241 [I 2022-10-27 09:25:51] ...trap/engines/engine.py.230: loss: 0.16939 [I 2022-10-27 09:25:51] ...optimizers/trijoint.py.151: total_norm: 15.136413 [I 2022-10-27 09:25:56] ...trap/engines/engine.py.224: train: epoch 59 | batch 2220/2304 [I 2022-10-27 09:25:56] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:46 [I 2022-10-27 09:25:56] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00243 [I 2022-10-27 09:25:56] ...trap/engines/engine.py.230: loss: 0.16699 [I 2022-10-27 09:25:56] ...optimizers/trijoint.py.151: total_norm: 24.115301 [I 2022-10-27 09:26:02] ...trap/engines/engine.py.224: train: epoch 59 | batch 2230/2304 [I 2022-10-27 09:26:02] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:41 [I 2022-10-27 09:26:02] ...trap/engines/engine.py.229: process: 0.58350 | load: 0.00241 [I 2022-10-27 09:26:02] ...trap/engines/engine.py.230: loss: 0.12707 [I 2022-10-27 09:26:02] ...optimizers/trijoint.py.151: total_norm: 14.167196 [I 2022-10-27 09:26:08] ...trap/engines/engine.py.224: train: epoch 59 | batch 2240/2304 [I 2022-10-27 09:26:08] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:35 [I 2022-10-27 09:26:08] ...trap/engines/engine.py.229: process: 0.56141 | load: 0.00241 [I 2022-10-27 09:26:08] ...trap/engines/engine.py.230: loss: 0.14486 [I 2022-10-27 09:26:08] ...optimizers/trijoint.py.151: total_norm: 37.945110 [I 2022-10-27 09:26:13] ...trap/engines/engine.py.224: train: epoch 59 | batch 2250/2304 [I 2022-10-27 09:26:13] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:30 [I 2022-10-27 09:26:13] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00239 [I 2022-10-27 09:26:13] ...trap/engines/engine.py.230: loss: 0.16520 [I 2022-10-27 09:26:13] ...optimizers/trijoint.py.151: total_norm: 19.756060 [I 2022-10-27 09:26:19] ...trap/engines/engine.py.224: train: epoch 59 | batch 2260/2304 [I 2022-10-27 09:26:19] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:24 [I 2022-10-27 09:26:19] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00244 [I 2022-10-27 09:26:19] ...trap/engines/engine.py.230: loss: 0.14360 [I 2022-10-27 09:26:19] ...optimizers/trijoint.py.151: total_norm: 18.472225 [I 2022-10-27 09:26:24] ...trap/engines/engine.py.224: train: epoch 59 | batch 2270/2304 [I 2022-10-27 09:26:24] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:18 [I 2022-10-27 09:26:24] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00242 [I 2022-10-27 09:26:24] ...trap/engines/engine.py.230: loss: 0.13231 [I 2022-10-27 09:26:24] ...optimizers/trijoint.py.151: total_norm: 12.562171 [I 2022-10-27 09:26:30] ...trap/engines/engine.py.224: train: epoch 59 | batch 2280/2304 [I 2022-10-27 09:26:30] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:13 [I 2022-10-27 09:26:30] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00238 [I 2022-10-27 09:26:30] ...trap/engines/engine.py.230: loss: 0.26895 [I 2022-10-27 09:26:30] ...optimizers/trijoint.py.151: total_norm: 17.364336 [I 2022-10-27 09:26:36] ...trap/engines/engine.py.224: train: epoch 59 | batch 2290/2304 [I 2022-10-27 09:26:36] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:07 [I 2022-10-27 09:26:36] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00242 [I 2022-10-27 09:26:36] ...trap/engines/engine.py.230: loss: 0.14310 [I 2022-10-27 09:26:36] ...optimizers/trijoint.py.151: total_norm: 16.176493 [I 2022-10-27 09:26:41] ...trap/engines/engine.py.224: train: epoch 59 | batch 2300/2304 [I 2022-10-27 09:26:41] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:02 [I 2022-10-27 09:26:41] ...trap/engines/engine.py.229: process: 0.55213 | load: 0.00014 [I 2022-10-27 09:26:41] ...trap/engines/engine.py.230: loss: 0.16872 [I 2022-10-27 09:26:41] ...optimizers/trijoint.py.151: total_norm: 17.363470 [I 2022-10-27 09:26:43] ...trap/engines/engine.py.224: train: epoch 59 | batch 2304/2304 [I 2022-10-27 09:26:43] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:00 [I 2022-10-27 09:26:43] ...trap/engines/engine.py.229: process: 0.55542 | load: 0.00014 [I 2022-10-27 09:26:43] ...trap/engines/engine.py.230: loss: 0.17080 [I 2022-10-27 09:26:43] ...optimizers/trijoint.py.151: total_norm: 22.098507 [S 2022-10-27 09:26:45] ...trap/engines/engine.py.129: train_epoch.epoch: 59 [S 2022-10-27 09:26:45] ...trap/engines/engine.py.129: train_epoch.loss: 0.1568345454438648 [I 2022-10-27 09:26:45] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 09:26:46] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 59 [I 2022-10-27 09:26:46] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 09:26:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 09:26:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 09:26:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 09:26:49] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 09:26:50] ...trap/engines/engine.py.321: eval: epoch 59 | batch 0/510 [I 2022-10-27 09:26:50] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:35:05 [I 2022-10-27 09:26:50] ...trap/engines/engine.py.326: process: 4.12837 | load: 2.82413 [I 2022-10-27 09:26:53] ...trap/engines/engine.py.321: eval: epoch 59 | batch 10/510 [I 2022-10-27 09:26:53] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:14 [I 2022-10-27 09:26:53] ...trap/engines/engine.py.326: process: 0.38477 | load: 0.00017 [I 2022-10-27 09:26:56] ...trap/engines/engine.py.321: eval: epoch 59 | batch 20/510 [I 2022-10-27 09:26:56] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:05 [I 2022-10-27 09:26:56] ...trap/engines/engine.py.326: process: 0.32712 | load: 0.00018 [I 2022-10-27 09:27:00] ...trap/engines/engine.py.321: eval: epoch 59 | batch 30/510 [I 2022-10-27 09:27:00] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:41 [I 2022-10-27 09:27:00] ...trap/engines/engine.py.326: process: 0.32659 | load: 0.00016 [I 2022-10-27 09:27:03] ...trap/engines/engine.py.321: eval: epoch 59 | batch 40/510 [I 2022-10-27 09:27:03] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:35 [I 2022-10-27 09:27:03] ...trap/engines/engine.py.326: process: 0.33600 | load: 0.00015 [I 2022-10-27 09:27:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 50/510 [I 2022-10-27 09:27:06] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:31 [I 2022-10-27 09:27:06] ...trap/engines/engine.py.326: process: 0.33343 | load: 0.00018 [I 2022-10-27 09:27:10] ...trap/engines/engine.py.321: eval: epoch 59 | batch 60/510 [I 2022-10-27 09:27:10] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:29 [I 2022-10-27 09:27:10] ...trap/engines/engine.py.326: process: 0.33158 | load: 0.00017 [I 2022-10-27 09:27:13] ...trap/engines/engine.py.321: eval: epoch 59 | batch 70/510 [I 2022-10-27 09:27:13] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:25 [I 2022-10-27 09:27:13] ...trap/engines/engine.py.326: process: 0.32987 | load: 0.00035 [I 2022-10-27 09:27:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 80/510 [I 2022-10-27 09:27:16] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:24 [I 2022-10-27 09:27:16] ...trap/engines/engine.py.326: process: 0.34498 | load: 0.00018 [I 2022-10-27 09:27:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 90/510 [I 2022-10-27 09:27:20] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:21 [I 2022-10-27 09:27:20] ...trap/engines/engine.py.326: process: 0.34090 | load: 0.00018 [I 2022-10-27 09:27:23] ...trap/engines/engine.py.321: eval: epoch 59 | batch 100/510 [I 2022-10-27 09:27:23] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:13 [I 2022-10-27 09:27:23] ...trap/engines/engine.py.326: process: 0.28394 | load: 0.00017 [I 2022-10-27 09:27:26] ...trap/engines/engine.py.321: eval: epoch 59 | batch 110/510 [I 2022-10-27 09:27:26] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:56 [I 2022-10-27 09:27:26] ...trap/engines/engine.py.326: process: 0.29665 | load: 0.00014 [I 2022-10-27 09:27:29] ...trap/engines/engine.py.321: eval: epoch 59 | batch 120/510 [I 2022-10-27 09:27:29] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:52 [I 2022-10-27 09:27:29] ...trap/engines/engine.py.326: process: 0.28394 | load: 0.00016 [I 2022-10-27 09:27:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 130/510 [I 2022-10-27 09:27:32] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:53 [I 2022-10-27 09:27:32] ...trap/engines/engine.py.326: process: 0.28675 | load: 0.00015 [I 2022-10-27 09:27:35] ...trap/engines/engine.py.321: eval: epoch 59 | batch 140/510 [I 2022-10-27 09:27:35] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 09:27:35] ...trap/engines/engine.py.326: process: 0.28660 | load: 0.00023 [I 2022-10-27 09:27:37] ...trap/engines/engine.py.321: eval: epoch 59 | batch 150/510 [I 2022-10-27 09:27:37] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:43 [I 2022-10-27 09:27:37] ...trap/engines/engine.py.326: process: 0.28760 | load: 0.00015 [I 2022-10-27 09:27:40] ...trap/engines/engine.py.321: eval: epoch 59 | batch 160/510 [I 2022-10-27 09:27:40] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:41 [I 2022-10-27 09:27:40] ...trap/engines/engine.py.326: process: 0.30672 | load: 0.00014 [I 2022-10-27 09:27:43] ...trap/engines/engine.py.321: eval: epoch 59 | batch 170/510 [I 2022-10-27 09:27:43] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:38 [I 2022-10-27 09:27:43] ...trap/engines/engine.py.326: process: 0.28467 | load: 0.00016 [I 2022-10-27 09:27:46] ...trap/engines/engine.py.321: eval: epoch 59 | batch 180/510 [I 2022-10-27 09:27:46] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:34 [I 2022-10-27 09:27:46] ...trap/engines/engine.py.326: process: 0.28675 | load: 0.00014 [I 2022-10-27 09:27:49] ...trap/engines/engine.py.321: eval: epoch 59 | batch 190/510 [I 2022-10-27 09:27:49] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:31 [I 2022-10-27 09:27:49] ...trap/engines/engine.py.326: process: 0.28331 | load: 0.00018 [I 2022-10-27 09:27:52] ...trap/engines/engine.py.321: eval: epoch 59 | batch 200/510 [I 2022-10-27 09:27:52] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:28 [I 2022-10-27 09:27:52] ...trap/engines/engine.py.326: process: 0.28631 | load: 0.00015 [I 2022-10-27 09:27:55] ...trap/engines/engine.py.321: eval: epoch 59 | batch 210/510 [I 2022-10-27 09:27:55] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:25 [I 2022-10-27 09:27:55] ...trap/engines/engine.py.326: process: 0.28401 | load: 0.00019 [I 2022-10-27 09:27:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 220/510 [I 2022-10-27 09:27:58] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 09:27:58] ...trap/engines/engine.py.326: process: 0.28280 | load: 0.00016 [I 2022-10-27 09:28:01] ...trap/engines/engine.py.321: eval: epoch 59 | batch 230/510 [I 2022-10-27 09:28:01] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:20 [I 2022-10-27 09:28:01] ...trap/engines/engine.py.326: process: 0.28613 | load: 0.00019 [I 2022-10-27 09:28:03] ...trap/engines/engine.py.321: eval: epoch 59 | batch 240/510 [I 2022-10-27 09:28:03] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:17 [I 2022-10-27 09:28:03] ...trap/engines/engine.py.326: process: 0.28547 | load: 0.00016 [I 2022-10-27 09:28:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 250/510 [I 2022-10-27 09:28:06] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-27 09:28:06] ...trap/engines/engine.py.326: process: 0.28456 | load: 0.00019 [I 2022-10-27 09:28:09] ...trap/engines/engine.py.321: eval: epoch 59 | batch 260/510 [I 2022-10-27 09:28:09] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 09:28:09] ...trap/engines/engine.py.326: process: 0.28177 | load: 0.00016 [I 2022-10-27 09:28:12] ...trap/engines/engine.py.321: eval: epoch 59 | batch 270/510 [I 2022-10-27 09:28:12] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-27 09:28:12] ...trap/engines/engine.py.326: process: 0.28570 | load: 0.00019 [I 2022-10-27 09:28:15] ...trap/engines/engine.py.321: eval: epoch 59 | batch 280/510 [I 2022-10-27 09:28:15] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 09:28:15] ...trap/engines/engine.py.326: process: 0.29057 | load: 0.00015 [I 2022-10-27 09:28:18] ...trap/engines/engine.py.321: eval: epoch 59 | batch 290/510 [I 2022-10-27 09:28:18] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 09:28:18] ...trap/engines/engine.py.326: process: 0.28545 | load: 0.00017 [I 2022-10-27 09:28:21] ...trap/engines/engine.py.321: eval: epoch 59 | batch 300/510 [I 2022-10-27 09:28:21] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 09:28:21] ...trap/engines/engine.py.326: process: 0.28666 | load: 0.00015 [I 2022-10-27 09:28:23] ...trap/engines/engine.py.321: eval: epoch 59 | batch 310/510 [I 2022-10-27 09:28:23] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:00:57 [I 2022-10-27 09:28:23] ...trap/engines/engine.py.326: process: 0.28303 | load: 0.00016 [I 2022-10-27 09:28:26] ...trap/engines/engine.py.321: eval: epoch 59 | batch 320/510 [I 2022-10-27 09:28:26] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:54 [I 2022-10-27 09:28:26] ...trap/engines/engine.py.326: process: 0.28779 | load: 0.00015 [I 2022-10-27 09:28:29] ...trap/engines/engine.py.321: eval: epoch 59 | batch 330/510 [I 2022-10-27 09:28:29] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:51 [I 2022-10-27 09:28:29] ...trap/engines/engine.py.326: process: 0.28406 | load: 0.00016 [I 2022-10-27 09:28:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 340/510 [I 2022-10-27 09:28:32] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:50 [I 2022-10-27 09:28:32] ...trap/engines/engine.py.326: process: 0.28827 | load: 0.00016 [I 2022-10-27 09:28:35] ...trap/engines/engine.py.321: eval: epoch 59 | batch 350/510 [I 2022-10-27 09:28:35] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-27 09:28:35] ...trap/engines/engine.py.326: process: 0.28735 | load: 0.00018 [I 2022-10-27 09:28:38] ...trap/engines/engine.py.321: eval: epoch 59 | batch 360/510 [I 2022-10-27 09:28:38] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:42 [I 2022-10-27 09:28:38] ...trap/engines/engine.py.326: process: 0.28393 | load: 0.00015 [I 2022-10-27 09:28:41] ...trap/engines/engine.py.321: eval: epoch 59 | batch 370/510 [I 2022-10-27 09:28:41] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 09:28:41] ...trap/engines/engine.py.326: process: 0.28796 | load: 0.00015 [I 2022-10-27 09:28:44] ...trap/engines/engine.py.321: eval: epoch 59 | batch 380/510 [I 2022-10-27 09:28:44] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 09:28:44] ...trap/engines/engine.py.326: process: 0.27948 | load: 0.00016 [I 2022-10-27 09:28:46] ...trap/engines/engine.py.321: eval: epoch 59 | batch 390/510 [I 2022-10-27 09:28:46] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:34 [I 2022-10-27 09:28:46] ...trap/engines/engine.py.326: process: 0.28693 | load: 0.00015 [I 2022-10-27 09:28:49] ...trap/engines/engine.py.321: eval: epoch 59 | batch 400/510 [I 2022-10-27 09:28:49] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:31 [I 2022-10-27 09:28:49] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00015 [I 2022-10-27 09:28:52] ...trap/engines/engine.py.321: eval: epoch 59 | batch 410/510 [I 2022-10-27 09:28:52] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:28 [I 2022-10-27 09:28:52] ...trap/engines/engine.py.326: process: 0.28442 | load: 0.00016 [I 2022-10-27 09:28:55] ...trap/engines/engine.py.321: eval: epoch 59 | batch 420/510 [I 2022-10-27 09:28:55] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-27 09:28:55] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00015 [I 2022-10-27 09:28:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 430/510 [I 2022-10-27 09:28:58] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:22 [I 2022-10-27 09:28:58] ...trap/engines/engine.py.326: process: 0.28538 | load: 0.00016 [I 2022-10-27 09:29:01] ...trap/engines/engine.py.321: eval: epoch 59 | batch 440/510 [I 2022-10-27 09:29:01] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 09:29:01] ...trap/engines/engine.py.326: process: 0.28482 | load: 0.00018 [I 2022-10-27 09:29:04] ...trap/engines/engine.py.321: eval: epoch 59 | batch 450/510 [I 2022-10-27 09:29:04] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 09:29:04] ...trap/engines/engine.py.326: process: 0.29319 | load: 0.00016 [I 2022-10-27 09:29:07] ...trap/engines/engine.py.321: eval: epoch 59 | batch 460/510 [I 2022-10-27 09:29:07] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:14 [I 2022-10-27 09:29:07] ...trap/engines/engine.py.326: process: 0.29088 | load: 0.00016 [I 2022-10-27 09:29:09] ...trap/engines/engine.py.321: eval: epoch 59 | batch 470/510 [I 2022-10-27 09:29:09] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:11 [I 2022-10-27 09:29:09] ...trap/engines/engine.py.326: process: 0.28494 | load: 0.00015 [I 2022-10-27 09:29:12] ...trap/engines/engine.py.321: eval: epoch 59 | batch 480/510 [I 2022-10-27 09:29:12] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:08 [I 2022-10-27 09:29:12] ...trap/engines/engine.py.326: process: 0.29008 | load: 0.00016 [I 2022-10-27 09:29:15] ...trap/engines/engine.py.321: eval: epoch 59 | batch 490/510 [I 2022-10-27 09:29:15] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:05 [I 2022-10-27 09:29:15] ...trap/engines/engine.py.326: process: 0.28994 | load: 0.00015 [I 2022-10-27 09:29:18] ...trap/engines/engine.py.321: eval: epoch 59 | batch 500/510 [I 2022-10-27 09:29:18] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 09:29:18] ...trap/engines/engine.py.326: process: 0.28439 | load: 0.00015 [I 2022-10-27 09:29:21] ...trap/engines/engine.py.321: eval: epoch 59 | batch 510/510 [I 2022-10-27 09:29:21] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 09:29:21] ...trap/engines/engine.py.326: process: 0.29063 | load: 0.00012 [S 2022-10-27 09:29:22] ...trap/engines/engine.py.132: eval_epoch.epoch: 59 [I 2022-10-27 09:29:22] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.710200 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.900000 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936400 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.701000 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.897300 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932400 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015204 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011384 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008811 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.017332 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009231 [S 2022-10-27 09:29:26] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008065 [I 2022-10-27 09:29:27] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 09:29:27] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 09:29:27] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 09:29:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 09:29:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 09:29:29] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 09:29:32] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 09:29:38] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 09:29:38] ...trap/engines/engine.py.163: Training model on trainset for epoch 60 [I 2022-10-27 09:29:38] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 09:29:41] ...trap/engines/engine.py.224: train: epoch 60 | batch 0/2304 [I 2022-10-27 09:29:41] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:04 [I 2022-10-27 09:29:41] ...trap/engines/engine.py.229: process: 2.63208 | load: 1.99294 [I 2022-10-27 09:29:41] ...trap/engines/engine.py.230: loss: 0.12005 [I 2022-10-27 09:29:41] ...optimizers/trijoint.py.151: total_norm: 11.247722 [I 2022-10-27 09:29:46] ...trap/engines/engine.py.224: train: epoch 60 | batch 10/2304 [I 2022-10-27 09:29:46] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:37 [I 2022-10-27 09:29:46] ...trap/engines/engine.py.229: process: 0.54717 | load: 0.00402 [I 2022-10-27 09:29:46] ...trap/engines/engine.py.230: loss: 0.16155 [I 2022-10-27 09:29:46] ...optimizers/trijoint.py.151: total_norm: 12.150875 [I 2022-10-27 09:29:52] ...trap/engines/engine.py.224: train: epoch 60 | batch 20/2304 [I 2022-10-27 09:29:52] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:11 [I 2022-10-27 09:29:52] ...trap/engines/engine.py.229: process: 0.55525 | load: 0.00399 [I 2022-10-27 09:29:52] ...trap/engines/engine.py.230: loss: 0.14764 [I 2022-10-27 09:29:52] ...optimizers/trijoint.py.151: total_norm: 10.941109 [I 2022-10-27 09:29:57] ...trap/engines/engine.py.224: train: epoch 60 | batch 30/2304 [I 2022-10-27 09:29:57] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:20 [I 2022-10-27 09:29:57] ...trap/engines/engine.py.229: process: 0.55235 | load: 0.00299 [I 2022-10-27 09:29:57] ...trap/engines/engine.py.230: loss: 0.16016 [I 2022-10-27 09:29:57] ...optimizers/trijoint.py.151: total_norm: 22.910196 [I 2022-10-27 09:30:03] ...trap/engines/engine.py.224: train: epoch 60 | batch 40/2304 [I 2022-10-27 09:30:03] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:21:05 [I 2022-10-27 09:30:03] ...trap/engines/engine.py.229: process: 0.56734 | load: 0.00353 [I 2022-10-27 09:30:03] ...trap/engines/engine.py.230: loss: 0.12984 [I 2022-10-27 09:30:03] ...optimizers/trijoint.py.151: total_norm: 10.824571 [I 2022-10-27 09:30:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 50/2304 [I 2022-10-27 09:30:08] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:47 [I 2022-10-27 09:30:08] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00299 [I 2022-10-27 09:30:08] ...trap/engines/engine.py.230: loss: 0.19876 [I 2022-10-27 09:30:08] ...optimizers/trijoint.py.151: total_norm: 14.219129 [I 2022-10-27 09:30:14] ...trap/engines/engine.py.224: train: epoch 60 | batch 60/2304 [I 2022-10-27 09:30:14] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:48 [I 2022-10-27 09:30:14] ...trap/engines/engine.py.229: process: 0.57168 | load: 0.00309 [I 2022-10-27 09:30:14] ...trap/engines/engine.py.230: loss: 0.21585 [I 2022-10-27 09:30:14] ...optimizers/trijoint.py.151: total_norm: 27.139406 [I 2022-10-27 09:30:19] ...trap/engines/engine.py.224: train: epoch 60 | batch 70/2304 [I 2022-10-27 09:30:19] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:39 [I 2022-10-27 09:30:19] ...trap/engines/engine.py.229: process: 0.55080 | load: 0.00318 [I 2022-10-27 09:30:19] ...trap/engines/engine.py.230: loss: 0.15534 [I 2022-10-27 09:30:19] ...optimizers/trijoint.py.151: total_norm: 25.016760 [I 2022-10-27 09:30:25] ...trap/engines/engine.py.224: train: epoch 60 | batch 80/2304 [I 2022-10-27 09:30:25] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:36 [I 2022-10-27 09:30:25] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00279 [I 2022-10-27 09:30:25] ...trap/engines/engine.py.230: loss: 0.22007 [I 2022-10-27 09:30:25] ...optimizers/trijoint.py.151: total_norm: 15.459357 [I 2022-10-27 09:30:31] ...trap/engines/engine.py.224: train: epoch 60 | batch 90/2304 [I 2022-10-27 09:30:31] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:27 [I 2022-10-27 09:30:31] ...trap/engines/engine.py.229: process: 0.54401 | load: 0.00297 [I 2022-10-27 09:30:31] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-27 09:30:31] ...optimizers/trijoint.py.151: total_norm: 15.597251 [I 2022-10-27 09:30:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 100/2304 [I 2022-10-27 09:30:36] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:11 [I 2022-10-27 09:30:36] ...trap/engines/engine.py.229: process: 0.53962 | load: 0.00278 [I 2022-10-27 09:30:36] ...trap/engines/engine.py.230: loss: 0.20453 [I 2022-10-27 09:30:36] ...optimizers/trijoint.py.151: total_norm: 50.056381 [I 2022-10-27 09:30:42] ...trap/engines/engine.py.224: train: epoch 60 | batch 110/2304 [I 2022-10-27 09:30:42] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:20:48 [I 2022-10-27 09:30:42] ...trap/engines/engine.py.229: process: 0.62364 | load: 0.00259 [I 2022-10-27 09:30:42] ...trap/engines/engine.py.230: loss: 0.19154 [I 2022-10-27 09:30:42] ...optimizers/trijoint.py.151: total_norm: 14.530430 [I 2022-10-27 09:30:47] ...trap/engines/engine.py.224: train: epoch 60 | batch 120/2304 [I 2022-10-27 09:30:47] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:25 [I 2022-10-27 09:30:47] ...trap/engines/engine.py.229: process: 0.58279 | load: 0.00254 [I 2022-10-27 09:30:47] ...trap/engines/engine.py.230: loss: 0.14661 [I 2022-10-27 09:30:47] ...optimizers/trijoint.py.151: total_norm: 10.556085 [I 2022-10-27 09:30:53] ...trap/engines/engine.py.224: train: epoch 60 | batch 130/2304 [I 2022-10-27 09:30:53] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:11 [I 2022-10-27 09:30:53] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00271 [I 2022-10-27 09:30:53] ...trap/engines/engine.py.230: loss: 0.10960 [I 2022-10-27 09:30:53] ...optimizers/trijoint.py.151: total_norm: 8.276265 [I 2022-10-27 09:30:58] ...trap/engines/engine.py.224: train: epoch 60 | batch 140/2304 [I 2022-10-27 09:30:58] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:57 [I 2022-10-27 09:30:58] ...trap/engines/engine.py.229: process: 0.54614 | load: 0.00256 [I 2022-10-27 09:30:58] ...trap/engines/engine.py.230: loss: 0.11814 [I 2022-10-27 09:30:58] ...optimizers/trijoint.py.151: total_norm: 13.619950 [I 2022-10-27 09:31:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 150/2304 [I 2022-10-27 09:31:04] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:51 [I 2022-10-27 09:31:04] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00255 [I 2022-10-27 09:31:04] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-27 09:31:04] ...optimizers/trijoint.py.151: total_norm: 12.444174 [I 2022-10-27 09:31:10] ...trap/engines/engine.py.224: train: epoch 60 | batch 160/2304 [I 2022-10-27 09:31:10] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:52 [I 2022-10-27 09:31:10] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00256 [I 2022-10-27 09:31:10] ...trap/engines/engine.py.230: loss: 0.12414 [I 2022-10-27 09:31:10] ...optimizers/trijoint.py.151: total_norm: 21.366589 [I 2022-10-27 09:31:15] ...trap/engines/engine.py.224: train: epoch 60 | batch 170/2304 [I 2022-10-27 09:31:15] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:39 [I 2022-10-27 09:31:15] ...trap/engines/engine.py.229: process: 0.55701 | load: 0.00266 [I 2022-10-27 09:31:15] ...trap/engines/engine.py.230: loss: 0.14061 [I 2022-10-27 09:31:15] ...optimizers/trijoint.py.151: total_norm: 13.625009 [I 2022-10-27 09:31:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 180/2304 [I 2022-10-27 09:31:21] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:19:35 [I 2022-10-27 09:31:21] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00251 [I 2022-10-27 09:31:21] ...trap/engines/engine.py.230: loss: 0.14596 [I 2022-10-27 09:31:21] ...optimizers/trijoint.py.151: total_norm: 18.418194 [I 2022-10-27 09:31:26] ...trap/engines/engine.py.224: train: epoch 60 | batch 190/2304 [I 2022-10-27 09:31:26] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:34 [I 2022-10-27 09:31:26] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00254 [I 2022-10-27 09:31:26] ...trap/engines/engine.py.230: loss: 0.14410 [I 2022-10-27 09:31:26] ...optimizers/trijoint.py.151: total_norm: 12.568336 [I 2022-10-27 09:31:32] ...trap/engines/engine.py.224: train: epoch 60 | batch 200/2304 [I 2022-10-27 09:31:32] ...trap/engines/engine.py.228: elapsed: 0:01:53 | left: 0:19:26 [I 2022-10-27 09:31:32] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00266 [I 2022-10-27 09:31:32] ...trap/engines/engine.py.230: loss: 0.10321 [I 2022-10-27 09:31:32] ...optimizers/trijoint.py.151: total_norm: 9.181932 [I 2022-10-27 09:31:37] ...trap/engines/engine.py.224: train: epoch 60 | batch 210/2304 [I 2022-10-27 09:31:37] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:19 [I 2022-10-27 09:31:37] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00258 [I 2022-10-27 09:31:37] ...trap/engines/engine.py.230: loss: 0.20977 [I 2022-10-27 09:31:37] ...optimizers/trijoint.py.151: total_norm: 20.088549 [I 2022-10-27 09:31:43] ...trap/engines/engine.py.224: train: epoch 60 | batch 220/2304 [I 2022-10-27 09:31:43] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:18 [I 2022-10-27 09:31:43] ...trap/engines/engine.py.229: process: 0.55787 | load: 0.00260 [I 2022-10-27 09:31:43] ...trap/engines/engine.py.230: loss: 0.16554 [I 2022-10-27 09:31:43] ...optimizers/trijoint.py.151: total_norm: 18.524033 [I 2022-10-27 09:31:49] ...trap/engines/engine.py.224: train: epoch 60 | batch 230/2304 [I 2022-10-27 09:31:49] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:18 [I 2022-10-27 09:31:49] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00290 [I 2022-10-27 09:31:49] ...trap/engines/engine.py.230: loss: 0.18501 [I 2022-10-27 09:31:49] ...optimizers/trijoint.py.151: total_norm: 12.784390 [I 2022-10-27 09:31:54] ...trap/engines/engine.py.224: train: epoch 60 | batch 240/2304 [I 2022-10-27 09:31:54] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:04 [I 2022-10-27 09:31:54] ...trap/engines/engine.py.229: process: 0.54670 | load: 0.00264 [I 2022-10-27 09:31:54] ...trap/engines/engine.py.230: loss: 0.15831 [I 2022-10-27 09:31:54] ...optimizers/trijoint.py.151: total_norm: 16.348011 [I 2022-10-27 09:32:00] ...trap/engines/engine.py.224: train: epoch 60 | batch 250/2304 [I 2022-10-27 09:32:00] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:19:05 [I 2022-10-27 09:32:00] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00264 [I 2022-10-27 09:32:00] ...trap/engines/engine.py.230: loss: 0.14940 [I 2022-10-27 09:32:00] ...optimizers/trijoint.py.151: total_norm: 24.669037 [I 2022-10-27 09:32:05] ...trap/engines/engine.py.224: train: epoch 60 | batch 260/2304 [I 2022-10-27 09:32:05] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:04 [I 2022-10-27 09:32:05] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00259 [I 2022-10-27 09:32:05] ...trap/engines/engine.py.230: loss: 0.15167 [I 2022-10-27 09:32:05] ...optimizers/trijoint.py.151: total_norm: 19.102180 [I 2022-10-27 09:32:11] ...trap/engines/engine.py.224: train: epoch 60 | batch 270/2304 [I 2022-10-27 09:32:11] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:02 [I 2022-10-27 09:32:11] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00258 [I 2022-10-27 09:32:11] ...trap/engines/engine.py.230: loss: 0.10798 [I 2022-10-27 09:32:11] ...optimizers/trijoint.py.151: total_norm: 12.091654 [I 2022-10-27 09:32:16] ...trap/engines/engine.py.224: train: epoch 60 | batch 280/2304 [I 2022-10-27 09:32:16] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:47 [I 2022-10-27 09:32:16] ...trap/engines/engine.py.229: process: 0.55381 | load: 0.00260 [I 2022-10-27 09:32:16] ...trap/engines/engine.py.230: loss: 0.12998 [I 2022-10-27 09:32:16] ...optimizers/trijoint.py.151: total_norm: 8.920390 [I 2022-10-27 09:32:22] ...trap/engines/engine.py.224: train: epoch 60 | batch 290/2304 [I 2022-10-27 09:32:22] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:39 [I 2022-10-27 09:32:22] ...trap/engines/engine.py.229: process: 0.54676 | load: 0.00267 [I 2022-10-27 09:32:22] ...trap/engines/engine.py.230: loss: 0.11174 [I 2022-10-27 09:32:22] ...optimizers/trijoint.py.151: total_norm: 10.949945 [I 2022-10-27 09:32:28] ...trap/engines/engine.py.224: train: epoch 60 | batch 300/2304 [I 2022-10-27 09:32:28] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:18:35 [I 2022-10-27 09:32:28] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00264 [I 2022-10-27 09:32:28] ...trap/engines/engine.py.230: loss: 0.30308 [I 2022-10-27 09:32:28] ...optimizers/trijoint.py.151: total_norm: 22.703405 [I 2022-10-27 09:32:33] ...trap/engines/engine.py.224: train: epoch 60 | batch 310/2304 [I 2022-10-27 09:32:33] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:37 [I 2022-10-27 09:32:33] ...trap/engines/engine.py.229: process: 0.56611 | load: 0.00262 [I 2022-10-27 09:32:33] ...trap/engines/engine.py.230: loss: 0.14006 [I 2022-10-27 09:32:33] ...optimizers/trijoint.py.151: total_norm: 19.296890 [I 2022-10-27 09:32:39] ...trap/engines/engine.py.224: train: epoch 60 | batch 320/2304 [I 2022-10-27 09:32:39] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:18:25 [I 2022-10-27 09:32:39] ...trap/engines/engine.py.229: process: 0.55204 | load: 0.00283 [I 2022-10-27 09:32:39] ...trap/engines/engine.py.230: loss: 0.15610 [I 2022-10-27 09:32:39] ...optimizers/trijoint.py.151: total_norm: 16.074690 [I 2022-10-27 09:32:44] ...trap/engines/engine.py.224: train: epoch 60 | batch 330/2304 [I 2022-10-27 09:32:44] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:18 [I 2022-10-27 09:32:44] ...trap/engines/engine.py.229: process: 0.54519 | load: 0.00252 [I 2022-10-27 09:32:44] ...trap/engines/engine.py.230: loss: 0.10175 [I 2022-10-27 09:32:44] ...optimizers/trijoint.py.151: total_norm: 10.320369 [I 2022-10-27 09:32:50] ...trap/engines/engine.py.224: train: epoch 60 | batch 340/2304 [I 2022-10-27 09:32:50] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:22 [I 2022-10-27 09:32:50] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00278 [I 2022-10-27 09:32:50] ...trap/engines/engine.py.230: loss: 0.21792 [I 2022-10-27 09:32:50] ...optimizers/trijoint.py.151: total_norm: 27.332350 [I 2022-10-27 09:32:56] ...trap/engines/engine.py.224: train: epoch 60 | batch 350/2304 [I 2022-10-27 09:32:56] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:18:12 [I 2022-10-27 09:32:56] ...trap/engines/engine.py.229: process: 0.55366 | load: 0.00271 [I 2022-10-27 09:32:56] ...trap/engines/engine.py.230: loss: 0.15698 [I 2022-10-27 09:32:56] ...optimizers/trijoint.py.151: total_norm: 15.558271 [I 2022-10-27 09:33:01] ...trap/engines/engine.py.224: train: epoch 60 | batch 360/2304 [I 2022-10-27 09:33:01] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:10 [I 2022-10-27 09:33:01] ...trap/engines/engine.py.229: process: 0.58497 | load: 0.00258 [I 2022-10-27 09:33:01] ...trap/engines/engine.py.230: loss: 0.12815 [I 2022-10-27 09:33:01] ...optimizers/trijoint.py.151: total_norm: 15.518594 [I 2022-10-27 09:33:07] ...trap/engines/engine.py.224: train: epoch 60 | batch 370/2304 [I 2022-10-27 09:33:07] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:17:55 [I 2022-10-27 09:33:07] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00261 [I 2022-10-27 09:33:07] ...trap/engines/engine.py.230: loss: 0.16137 [I 2022-10-27 09:33:07] ...optimizers/trijoint.py.151: total_norm: 19.617960 [I 2022-10-27 09:33:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 380/2304 [I 2022-10-27 09:33:12] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:47 [I 2022-10-27 09:33:12] ...trap/engines/engine.py.229: process: 0.54340 | load: 0.00258 [I 2022-10-27 09:33:12] ...trap/engines/engine.py.230: loss: 0.16473 [I 2022-10-27 09:33:12] ...optimizers/trijoint.py.151: total_norm: 14.690105 [I 2022-10-27 09:33:18] ...trap/engines/engine.py.224: train: epoch 60 | batch 390/2304 [I 2022-10-27 09:33:18] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:49 [I 2022-10-27 09:33:18] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00250 [I 2022-10-27 09:33:18] ...trap/engines/engine.py.230: loss: 0.17462 [I 2022-10-27 09:33:18] ...optimizers/trijoint.py.151: total_norm: 16.454435 [I 2022-10-27 09:33:24] ...trap/engines/engine.py.224: train: epoch 60 | batch 400/2304 [I 2022-10-27 09:33:24] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:36 [I 2022-10-27 09:33:24] ...trap/engines/engine.py.229: process: 0.54444 | load: 0.00260 [I 2022-10-27 09:33:24] ...trap/engines/engine.py.230: loss: 0.18567 [I 2022-10-27 09:33:24] ...optimizers/trijoint.py.151: total_norm: 18.944178 [I 2022-10-27 09:33:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 410/2304 [I 2022-10-27 09:33:29] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:31 [I 2022-10-27 09:33:29] ...trap/engines/engine.py.229: process: 0.54896 | load: 0.00254 [I 2022-10-27 09:33:29] ...trap/engines/engine.py.230: loss: 0.19738 [I 2022-10-27 09:33:29] ...optimizers/trijoint.py.151: total_norm: 12.597897 [I 2022-10-27 09:33:35] ...trap/engines/engine.py.224: train: epoch 60 | batch 420/2304 [I 2022-10-27 09:33:35] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:17:26 [I 2022-10-27 09:33:35] ...trap/engines/engine.py.229: process: 0.56819 | load: 0.00253 [I 2022-10-27 09:33:35] ...trap/engines/engine.py.230: loss: 0.12097 [I 2022-10-27 09:33:35] ...optimizers/trijoint.py.151: total_norm: 13.946402 [I 2022-10-27 09:33:40] ...trap/engines/engine.py.224: train: epoch 60 | batch 430/2304 [I 2022-10-27 09:33:40] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:25 [I 2022-10-27 09:33:40] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00254 [I 2022-10-27 09:33:40] ...trap/engines/engine.py.230: loss: 0.13364 [I 2022-10-27 09:33:40] ...optimizers/trijoint.py.151: total_norm: 14.921622 [I 2022-10-27 09:33:46] ...trap/engines/engine.py.224: train: epoch 60 | batch 440/2304 [I 2022-10-27 09:33:46] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:17:18 [I 2022-10-27 09:33:46] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00260 [I 2022-10-27 09:33:46] ...trap/engines/engine.py.230: loss: 0.14292 [I 2022-10-27 09:33:46] ...optimizers/trijoint.py.151: total_norm: 16.002157 [I 2022-10-27 09:33:51] ...trap/engines/engine.py.224: train: epoch 60 | batch 450/2304 [I 2022-10-27 09:33:51] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:14 [I 2022-10-27 09:33:51] ...trap/engines/engine.py.229: process: 0.55422 | load: 0.00264 [I 2022-10-27 09:33:51] ...trap/engines/engine.py.230: loss: 0.14327 [I 2022-10-27 09:33:51] ...optimizers/trijoint.py.151: total_norm: 11.165663 [I 2022-10-27 09:33:57] ...trap/engines/engine.py.224: train: epoch 60 | batch 460/2304 [I 2022-10-27 09:33:57] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:05 [I 2022-10-27 09:33:57] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00267 [I 2022-10-27 09:33:57] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-27 09:33:57] ...optimizers/trijoint.py.151: total_norm: 16.581099 [I 2022-10-27 09:34:03] ...trap/engines/engine.py.224: train: epoch 60 | batch 470/2304 [I 2022-10-27 09:34:03] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:17:00 [I 2022-10-27 09:34:03] ...trap/engines/engine.py.229: process: 0.56527 | load: 0.00254 [I 2022-10-27 09:34:03] ...trap/engines/engine.py.230: loss: 0.13390 [I 2022-10-27 09:34:03] ...optimizers/trijoint.py.151: total_norm: 14.557243 [I 2022-10-27 09:34:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 480/2304 [I 2022-10-27 09:34:08] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:53 [I 2022-10-27 09:34:08] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00264 [I 2022-10-27 09:34:08] ...trap/engines/engine.py.230: loss: 0.13378 [I 2022-10-27 09:34:08] ...optimizers/trijoint.py.151: total_norm: 17.871326 [I 2022-10-27 09:34:14] ...trap/engines/engine.py.224: train: epoch 60 | batch 490/2304 [I 2022-10-27 09:34:14] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:16:50 [I 2022-10-27 09:34:14] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00255 [I 2022-10-27 09:34:14] ...trap/engines/engine.py.230: loss: 0.13328 [I 2022-10-27 09:34:14] ...optimizers/trijoint.py.151: total_norm: 10.100652 [I 2022-10-27 09:34:19] ...trap/engines/engine.py.224: train: epoch 60 | batch 500/2304 [I 2022-10-27 09:34:19] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:39 [I 2022-10-27 09:34:19] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00259 [I 2022-10-27 09:34:19] ...trap/engines/engine.py.230: loss: 0.14187 [I 2022-10-27 09:34:19] ...optimizers/trijoint.py.151: total_norm: 14.661542 [I 2022-10-27 09:34:25] ...trap/engines/engine.py.224: train: epoch 60 | batch 510/2304 [I 2022-10-27 09:34:25] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:16:36 [I 2022-10-27 09:34:25] ...trap/engines/engine.py.229: process: 0.55637 | load: 0.00276 [I 2022-10-27 09:34:25] ...trap/engines/engine.py.230: loss: 0.16810 [I 2022-10-27 09:34:25] ...optimizers/trijoint.py.151: total_norm: 14.364796 [I 2022-10-27 09:34:30] ...trap/engines/engine.py.224: train: epoch 60 | batch 520/2304 [I 2022-10-27 09:34:30] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:16:28 [I 2022-10-27 09:34:30] ...trap/engines/engine.py.229: process: 0.55669 | load: 0.00253 [I 2022-10-27 09:34:30] ...trap/engines/engine.py.230: loss: 0.16145 [I 2022-10-27 09:34:30] ...optimizers/trijoint.py.151: total_norm: 11.507758 [I 2022-10-27 09:34:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 530/2304 [I 2022-10-27 09:34:36] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:21 [I 2022-10-27 09:34:36] ...trap/engines/engine.py.229: process: 0.54676 | load: 0.00250 [I 2022-10-27 09:34:36] ...trap/engines/engine.py.230: loss: 0.19188 [I 2022-10-27 09:34:36] ...optimizers/trijoint.py.151: total_norm: 20.318295 [I 2022-10-27 09:34:42] ...trap/engines/engine.py.224: train: epoch 60 | batch 540/2304 [I 2022-10-27 09:34:42] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:16:22 [I 2022-10-27 09:34:42] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00270 [I 2022-10-27 09:34:42] ...trap/engines/engine.py.230: loss: 0.11892 [I 2022-10-27 09:34:42] ...optimizers/trijoint.py.151: total_norm: 12.586328 [I 2022-10-27 09:34:47] ...trap/engines/engine.py.224: train: epoch 60 | batch 550/2304 [I 2022-10-27 09:34:47] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:15 [I 2022-10-27 09:34:47] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00265 [I 2022-10-27 09:34:47] ...trap/engines/engine.py.230: loss: 0.11840 [I 2022-10-27 09:34:47] ...optimizers/trijoint.py.151: total_norm: 18.146259 [I 2022-10-27 09:34:53] ...trap/engines/engine.py.224: train: epoch 60 | batch 560/2304 [I 2022-10-27 09:34:53] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:16:07 [I 2022-10-27 09:34:53] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00257 [I 2022-10-27 09:34:53] ...trap/engines/engine.py.230: loss: 0.14669 [I 2022-10-27 09:34:53] ...optimizers/trijoint.py.151: total_norm: 15.921618 [I 2022-10-27 09:34:58] ...trap/engines/engine.py.224: train: epoch 60 | batch 570/2304 [I 2022-10-27 09:34:58] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:16:04 [I 2022-10-27 09:34:58] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00252 [I 2022-10-27 09:34:58] ...trap/engines/engine.py.230: loss: 0.11649 [I 2022-10-27 09:34:58] ...optimizers/trijoint.py.151: total_norm: 8.699511 [I 2022-10-27 09:35:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 580/2304 [I 2022-10-27 09:35:04] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:16:00 [I 2022-10-27 09:35:04] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00275 [I 2022-10-27 09:35:04] ...trap/engines/engine.py.230: loss: 0.14729 [I 2022-10-27 09:35:04] ...optimizers/trijoint.py.151: total_norm: 35.118782 [I 2022-10-27 09:35:09] ...trap/engines/engine.py.224: train: epoch 60 | batch 590/2304 [I 2022-10-27 09:35:09] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:15:53 [I 2022-10-27 09:35:09] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00251 [I 2022-10-27 09:35:09] ...trap/engines/engine.py.230: loss: 0.14917 [I 2022-10-27 09:35:09] ...optimizers/trijoint.py.151: total_norm: 21.156324 [I 2022-10-27 09:35:15] ...trap/engines/engine.py.224: train: epoch 60 | batch 600/2304 [I 2022-10-27 09:35:15] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:44 [I 2022-10-27 09:35:15] ...trap/engines/engine.py.229: process: 0.56302 | load: 0.00251 [I 2022-10-27 09:35:15] ...trap/engines/engine.py.230: loss: 0.19168 [I 2022-10-27 09:35:15] ...optimizers/trijoint.py.151: total_norm: 21.689222 [I 2022-10-27 09:35:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 610/2304 [I 2022-10-27 09:35:21] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:15:42 [I 2022-10-27 09:35:21] ...trap/engines/engine.py.229: process: 0.55277 | load: 0.00254 [I 2022-10-27 09:35:21] ...trap/engines/engine.py.230: loss: 0.15163 [I 2022-10-27 09:35:21] ...optimizers/trijoint.py.151: total_norm: 11.313045 [I 2022-10-27 09:35:26] ...trap/engines/engine.py.224: train: epoch 60 | batch 620/2304 [I 2022-10-27 09:35:26] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:37 [I 2022-10-27 09:35:26] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00253 [I 2022-10-27 09:35:26] ...trap/engines/engine.py.230: loss: 0.11789 [I 2022-10-27 09:35:26] ...optimizers/trijoint.py.151: total_norm: 11.091979 [I 2022-10-27 09:35:32] ...trap/engines/engine.py.224: train: epoch 60 | batch 630/2304 [I 2022-10-27 09:35:32] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:15:26 [I 2022-10-27 09:35:32] ...trap/engines/engine.py.229: process: 0.55038 | load: 0.00277 [I 2022-10-27 09:35:32] ...trap/engines/engine.py.230: loss: 0.16040 [I 2022-10-27 09:35:32] ...optimizers/trijoint.py.151: total_norm: 21.034273 [I 2022-10-27 09:35:37] ...trap/engines/engine.py.224: train: epoch 60 | batch 640/2304 [I 2022-10-27 09:35:37] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:24 [I 2022-10-27 09:35:37] ...trap/engines/engine.py.229: process: 0.55997 | load: 0.00303 [I 2022-10-27 09:35:37] ...trap/engines/engine.py.230: loss: 0.17969 [I 2022-10-27 09:35:37] ...optimizers/trijoint.py.151: total_norm: 17.032642 [I 2022-10-27 09:35:43] ...trap/engines/engine.py.224: train: epoch 60 | batch 650/2304 [I 2022-10-27 09:35:43] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:17 [I 2022-10-27 09:35:43] ...trap/engines/engine.py.229: process: 0.54820 | load: 0.00259 [I 2022-10-27 09:35:43] ...trap/engines/engine.py.230: loss: 0.10671 [I 2022-10-27 09:35:43] ...optimizers/trijoint.py.151: total_norm: 7.840178 [I 2022-10-27 09:35:48] ...trap/engines/engine.py.224: train: epoch 60 | batch 660/2304 [I 2022-10-27 09:35:48] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:15:13 [I 2022-10-27 09:35:48] ...trap/engines/engine.py.229: process: 0.55338 | load: 0.00257 [I 2022-10-27 09:35:48] ...trap/engines/engine.py.230: loss: 0.18176 [I 2022-10-27 09:35:48] ...optimizers/trijoint.py.151: total_norm: 13.684339 [I 2022-10-27 09:35:54] ...trap/engines/engine.py.224: train: epoch 60 | batch 670/2304 [I 2022-10-27 09:35:54] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:09 [I 2022-10-27 09:35:54] ...trap/engines/engine.py.229: process: 0.55137 | load: 0.00258 [I 2022-10-27 09:35:54] ...trap/engines/engine.py.230: loss: 0.11914 [I 2022-10-27 09:35:54] ...optimizers/trijoint.py.151: total_norm: 15.449035 [I 2022-10-27 09:36:00] ...trap/engines/engine.py.224: train: epoch 60 | batch 680/2304 [I 2022-10-27 09:36:00] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:15:21 [I 2022-10-27 09:36:00] ...trap/engines/engine.py.229: process: 0.56761 | load: 0.00258 [I 2022-10-27 09:36:00] ...trap/engines/engine.py.230: loss: 0.20845 [I 2022-10-27 09:36:00] ...optimizers/trijoint.py.151: total_norm: 29.950069 [I 2022-10-27 09:36:05] ...trap/engines/engine.py.224: train: epoch 60 | batch 690/2304 [I 2022-10-27 09:36:05] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:15:05 [I 2022-10-27 09:36:05] ...trap/engines/engine.py.229: process: 0.55461 | load: 0.00255 [I 2022-10-27 09:36:05] ...trap/engines/engine.py.230: loss: 0.11280 [I 2022-10-27 09:36:05] ...optimizers/trijoint.py.151: total_norm: 10.457524 [I 2022-10-27 09:36:11] ...trap/engines/engine.py.224: train: epoch 60 | batch 700/2304 [I 2022-10-27 09:36:11] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:15:09 [I 2022-10-27 09:36:11] ...trap/engines/engine.py.229: process: 0.56512 | load: 0.00257 [I 2022-10-27 09:36:11] ...trap/engines/engine.py.230: loss: 0.11060 [I 2022-10-27 09:36:11] ...optimizers/trijoint.py.151: total_norm: 7.643708 [I 2022-10-27 09:36:17] ...trap/engines/engine.py.224: train: epoch 60 | batch 710/2304 [I 2022-10-27 09:36:17] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:14:52 [I 2022-10-27 09:36:17] ...trap/engines/engine.py.229: process: 0.54618 | load: 0.00253 [I 2022-10-27 09:36:17] ...trap/engines/engine.py.230: loss: 0.15228 [I 2022-10-27 09:36:17] ...optimizers/trijoint.py.151: total_norm: 50.290966 [I 2022-10-27 09:36:22] ...trap/engines/engine.py.224: train: epoch 60 | batch 720/2304 [I 2022-10-27 09:36:22] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:44 [I 2022-10-27 09:36:22] ...trap/engines/engine.py.229: process: 0.56709 | load: 0.00261 [I 2022-10-27 09:36:22] ...trap/engines/engine.py.230: loss: 0.14698 [I 2022-10-27 09:36:22] ...optimizers/trijoint.py.151: total_norm: 23.207664 [I 2022-10-27 09:36:28] ...trap/engines/engine.py.224: train: epoch 60 | batch 730/2304 [I 2022-10-27 09:36:28] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:14:38 [I 2022-10-27 09:36:28] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00254 [I 2022-10-27 09:36:28] ...trap/engines/engine.py.230: loss: 0.11818 [I 2022-10-27 09:36:28] ...optimizers/trijoint.py.151: total_norm: 16.473814 [I 2022-10-27 09:36:33] ...trap/engines/engine.py.224: train: epoch 60 | batch 740/2304 [I 2022-10-27 09:36:33] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:30 [I 2022-10-27 09:36:33] ...trap/engines/engine.py.229: process: 0.55271 | load: 0.00261 [I 2022-10-27 09:36:33] ...trap/engines/engine.py.230: loss: 0.19643 [I 2022-10-27 09:36:33] ...optimizers/trijoint.py.151: total_norm: 21.472538 [I 2022-10-27 09:36:39] ...trap/engines/engine.py.224: train: epoch 60 | batch 750/2304 [I 2022-10-27 09:36:39] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:26 [I 2022-10-27 09:36:39] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00263 [I 2022-10-27 09:36:39] ...trap/engines/engine.py.230: loss: 0.16557 [I 2022-10-27 09:36:39] ...optimizers/trijoint.py.151: total_norm: 15.573662 [I 2022-10-27 09:36:44] ...trap/engines/engine.py.224: train: epoch 60 | batch 760/2304 [I 2022-10-27 09:36:44] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:30 [I 2022-10-27 09:36:44] ...trap/engines/engine.py.229: process: 0.58251 | load: 0.00245 [I 2022-10-27 09:36:44] ...trap/engines/engine.py.230: loss: 0.22034 [I 2022-10-27 09:36:44] ...optimizers/trijoint.py.151: total_norm: 20.184731 [I 2022-10-27 09:36:50] ...trap/engines/engine.py.224: train: epoch 60 | batch 770/2304 [I 2022-10-27 09:36:50] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:48 [I 2022-10-27 09:36:50] ...trap/engines/engine.py.229: process: 0.65161 | load: 0.00253 [I 2022-10-27 09:36:50] ...trap/engines/engine.py.230: loss: 0.10887 [I 2022-10-27 09:36:50] ...optimizers/trijoint.py.151: total_norm: 10.804843 [I 2022-10-27 09:36:56] ...trap/engines/engine.py.224: train: epoch 60 | batch 780/2304 [I 2022-10-27 09:36:56] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:14:19 [I 2022-10-27 09:36:56] ...trap/engines/engine.py.229: process: 0.58288 | load: 0.00253 [I 2022-10-27 09:36:56] ...trap/engines/engine.py.230: loss: 0.11725 [I 2022-10-27 09:36:56] ...optimizers/trijoint.py.151: total_norm: 9.228213 [I 2022-10-27 09:37:01] ...trap/engines/engine.py.224: train: epoch 60 | batch 790/2304 [I 2022-10-27 09:37:01] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:13 [I 2022-10-27 09:37:01] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00251 [I 2022-10-27 09:37:01] ...trap/engines/engine.py.230: loss: 0.16408 [I 2022-10-27 09:37:01] ...optimizers/trijoint.py.151: total_norm: 18.631805 [I 2022-10-27 09:37:07] ...trap/engines/engine.py.224: train: epoch 60 | batch 800/2304 [I 2022-10-27 09:37:07] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:14:04 [I 2022-10-27 09:37:07] ...trap/engines/engine.py.229: process: 0.56678 | load: 0.00253 [I 2022-10-27 09:37:07] ...trap/engines/engine.py.230: loss: 0.16177 [I 2022-10-27 09:37:07] ...optimizers/trijoint.py.151: total_norm: 22.809982 [I 2022-10-27 09:37:13] ...trap/engines/engine.py.224: train: epoch 60 | batch 810/2304 [I 2022-10-27 09:37:13] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:13:52 [I 2022-10-27 09:37:13] ...trap/engines/engine.py.229: process: 0.55434 | load: 0.00248 [I 2022-10-27 09:37:13] ...trap/engines/engine.py.230: loss: 0.12927 [I 2022-10-27 09:37:13] ...optimizers/trijoint.py.151: total_norm: 9.878864 [I 2022-10-27 09:37:18] ...trap/engines/engine.py.224: train: epoch 60 | batch 820/2304 [I 2022-10-27 09:37:18] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:42 [I 2022-10-27 09:37:18] ...trap/engines/engine.py.229: process: 0.55433 | load: 0.00263 [I 2022-10-27 09:37:18] ...trap/engines/engine.py.230: loss: 0.21498 [I 2022-10-27 09:37:18] ...optimizers/trijoint.py.151: total_norm: 21.874851 [I 2022-10-27 09:37:24] ...trap/engines/engine.py.224: train: epoch 60 | batch 830/2304 [I 2022-10-27 09:37:24] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:13:32 [I 2022-10-27 09:37:24] ...trap/engines/engine.py.229: process: 0.54062 | load: 0.00254 [I 2022-10-27 09:37:24] ...trap/engines/engine.py.230: loss: 0.17123 [I 2022-10-27 09:37:24] ...optimizers/trijoint.py.151: total_norm: 14.009346 [I 2022-10-27 09:37:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 840/2304 [I 2022-10-27 09:37:29] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:35 [I 2022-10-27 09:37:29] ...trap/engines/engine.py.229: process: 0.53544 | load: 0.00244 [I 2022-10-27 09:37:29] ...trap/engines/engine.py.230: loss: 0.13832 [I 2022-10-27 09:37:29] ...optimizers/trijoint.py.151: total_norm: 8.364622 [I 2022-10-27 09:37:35] ...trap/engines/engine.py.224: train: epoch 60 | batch 850/2304 [I 2022-10-27 09:37:35] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:26 [I 2022-10-27 09:37:35] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00249 [I 2022-10-27 09:37:35] ...trap/engines/engine.py.230: loss: 0.18419 [I 2022-10-27 09:37:35] ...optimizers/trijoint.py.151: total_norm: 24.742678 [I 2022-10-27 09:37:41] ...trap/engines/engine.py.224: train: epoch 60 | batch 860/2304 [I 2022-10-27 09:37:41] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:26 [I 2022-10-27 09:37:41] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00245 [I 2022-10-27 09:37:41] ...trap/engines/engine.py.230: loss: 0.16837 [I 2022-10-27 09:37:41] ...optimizers/trijoint.py.151: total_norm: 25.251703 [I 2022-10-27 09:37:46] ...trap/engines/engine.py.224: train: epoch 60 | batch 870/2304 [I 2022-10-27 09:37:46] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:17 [I 2022-10-27 09:37:46] ...trap/engines/engine.py.229: process: 0.55070 | load: 0.00243 [I 2022-10-27 09:37:46] ...trap/engines/engine.py.230: loss: 0.18076 [I 2022-10-27 09:37:46] ...optimizers/trijoint.py.151: total_norm: 16.436363 [I 2022-10-27 09:37:52] ...trap/engines/engine.py.224: train: epoch 60 | batch 880/2304 [I 2022-10-27 09:37:52] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:18 [I 2022-10-27 09:37:52] ...trap/engines/engine.py.229: process: 0.57575 | load: 0.00249 [I 2022-10-27 09:37:52] ...trap/engines/engine.py.230: loss: 0.18013 [I 2022-10-27 09:37:52] ...optimizers/trijoint.py.151: total_norm: 21.476505 [I 2022-10-27 09:37:57] ...trap/engines/engine.py.224: train: epoch 60 | batch 890/2304 [I 2022-10-27 09:37:57] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:10 [I 2022-10-27 09:37:57] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00244 [I 2022-10-27 09:37:57] ...trap/engines/engine.py.230: loss: 0.12941 [I 2022-10-27 09:37:57] ...optimizers/trijoint.py.151: total_norm: 59.781754 [I 2022-10-27 09:38:03] ...trap/engines/engine.py.224: train: epoch 60 | batch 900/2304 [I 2022-10-27 09:38:03] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:00 [I 2022-10-27 09:38:03] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00243 [I 2022-10-27 09:38:03] ...trap/engines/engine.py.230: loss: 0.11789 [I 2022-10-27 09:38:03] ...optimizers/trijoint.py.151: total_norm: 13.954756 [I 2022-10-27 09:38:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 910/2304 [I 2022-10-27 09:38:08] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:57 [I 2022-10-27 09:38:08] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00243 [I 2022-10-27 09:38:08] ...trap/engines/engine.py.230: loss: 0.13213 [I 2022-10-27 09:38:08] ...optimizers/trijoint.py.151: total_norm: 14.968696 [I 2022-10-27 09:38:14] ...trap/engines/engine.py.224: train: epoch 60 | batch 920/2304 [I 2022-10-27 09:38:14] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:47 [I 2022-10-27 09:38:14] ...trap/engines/engine.py.229: process: 0.55029 | load: 0.00242 [I 2022-10-27 09:38:14] ...trap/engines/engine.py.230: loss: 0.09836 [I 2022-10-27 09:38:14] ...optimizers/trijoint.py.151: total_norm: 8.777648 [I 2022-10-27 09:38:20] ...trap/engines/engine.py.224: train: epoch 60 | batch 930/2304 [I 2022-10-27 09:38:20] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:12:40 [I 2022-10-27 09:38:20] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00243 [I 2022-10-27 09:38:20] ...trap/engines/engine.py.230: loss: 0.09955 [I 2022-10-27 09:38:20] ...optimizers/trijoint.py.151: total_norm: 10.164124 [I 2022-10-27 09:38:25] ...trap/engines/engine.py.224: train: epoch 60 | batch 940/2304 [I 2022-10-27 09:38:25] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:40 [I 2022-10-27 09:38:25] ...trap/engines/engine.py.229: process: 0.57369 | load: 0.00248 [I 2022-10-27 09:38:25] ...trap/engines/engine.py.230: loss: 0.18343 [I 2022-10-27 09:38:25] ...optimizers/trijoint.py.151: total_norm: 11.959047 [I 2022-10-27 09:38:31] ...trap/engines/engine.py.224: train: epoch 60 | batch 950/2304 [I 2022-10-27 09:38:31] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:45 [I 2022-10-27 09:38:31] ...trap/engines/engine.py.229: process: 0.56224 | load: 0.00253 [I 2022-10-27 09:38:31] ...trap/engines/engine.py.230: loss: 0.14736 [I 2022-10-27 09:38:31] ...optimizers/trijoint.py.151: total_norm: 11.845785 [I 2022-10-27 09:38:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 960/2304 [I 2022-10-27 09:38:36] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:26 [I 2022-10-27 09:38:36] ...trap/engines/engine.py.229: process: 0.54492 | load: 0.00255 [I 2022-10-27 09:38:36] ...trap/engines/engine.py.230: loss: 0.23611 [I 2022-10-27 09:38:36] ...optimizers/trijoint.py.151: total_norm: 32.455982 [I 2022-10-27 09:38:42] ...trap/engines/engine.py.224: train: epoch 60 | batch 970/2304 [I 2022-10-27 09:38:42] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:21 [I 2022-10-27 09:38:42] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00264 [I 2022-10-27 09:38:42] ...trap/engines/engine.py.230: loss: 0.22263 [I 2022-10-27 09:38:42] ...optimizers/trijoint.py.151: total_norm: 12.684024 [I 2022-10-27 09:38:48] ...trap/engines/engine.py.224: train: epoch 60 | batch 980/2304 [I 2022-10-27 09:38:48] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:13 [I 2022-10-27 09:38:48] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00256 [I 2022-10-27 09:38:48] ...trap/engines/engine.py.230: loss: 0.15190 [I 2022-10-27 09:38:48] ...optimizers/trijoint.py.151: total_norm: 12.641068 [I 2022-10-27 09:38:53] ...trap/engines/engine.py.224: train: epoch 60 | batch 990/2304 [I 2022-10-27 09:38:53] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:08 [I 2022-10-27 09:38:53] ...trap/engines/engine.py.229: process: 0.55209 | load: 0.00258 [I 2022-10-27 09:38:53] ...trap/engines/engine.py.230: loss: 0.13164 [I 2022-10-27 09:38:53] ...optimizers/trijoint.py.151: total_norm: 10.672304 [I 2022-10-27 09:38:59] ...trap/engines/engine.py.224: train: epoch 60 | batch 1000/2304 [I 2022-10-27 09:38:59] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:02 [I 2022-10-27 09:38:59] ...trap/engines/engine.py.229: process: 0.54479 | load: 0.00257 [I 2022-10-27 09:38:59] ...trap/engines/engine.py.230: loss: 0.13866 [I 2022-10-27 09:38:59] ...optimizers/trijoint.py.151: total_norm: 13.073300 [I 2022-10-27 09:39:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 1010/2304 [I 2022-10-27 09:39:04] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:11:57 [I 2022-10-27 09:39:04] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00263 [I 2022-10-27 09:39:04] ...trap/engines/engine.py.230: loss: 0.15128 [I 2022-10-27 09:39:04] ...optimizers/trijoint.py.151: total_norm: 21.274790 [I 2022-10-27 09:39:10] ...trap/engines/engine.py.224: train: epoch 60 | batch 1020/2304 [I 2022-10-27 09:39:10] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:11:51 [I 2022-10-27 09:39:10] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00252 [I 2022-10-27 09:39:10] ...trap/engines/engine.py.230: loss: 0.16204 [I 2022-10-27 09:39:10] ...optimizers/trijoint.py.151: total_norm: 26.955929 [I 2022-10-27 09:39:15] ...trap/engines/engine.py.224: train: epoch 60 | batch 1030/2304 [I 2022-10-27 09:39:15] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:46 [I 2022-10-27 09:39:15] ...trap/engines/engine.py.229: process: 0.55137 | load: 0.00259 [I 2022-10-27 09:39:15] ...trap/engines/engine.py.230: loss: 0.16456 [I 2022-10-27 09:39:15] ...optimizers/trijoint.py.151: total_norm: 12.158952 [I 2022-10-27 09:39:21] ...trap/engines/engine.py.224: train: epoch 60 | batch 1040/2304 [I 2022-10-27 09:39:21] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:11:41 [I 2022-10-27 09:39:21] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00260 [I 2022-10-27 09:39:21] ...trap/engines/engine.py.230: loss: 0.14998 [I 2022-10-27 09:39:21] ...optimizers/trijoint.py.151: total_norm: 16.335003 [I 2022-10-27 09:39:26] ...trap/engines/engine.py.224: train: epoch 60 | batch 1050/2304 [I 2022-10-27 09:39:26] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:31 [I 2022-10-27 09:39:26] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00270 [I 2022-10-27 09:39:26] ...trap/engines/engine.py.230: loss: 0.10641 [I 2022-10-27 09:39:26] ...optimizers/trijoint.py.151: total_norm: 15.966844 [I 2022-10-27 09:39:32] ...trap/engines/engine.py.224: train: epoch 60 | batch 1060/2304 [I 2022-10-27 09:39:32] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:28 [I 2022-10-27 09:39:32] ...trap/engines/engine.py.229: process: 0.54414 | load: 0.00261 [I 2022-10-27 09:39:32] ...trap/engines/engine.py.230: loss: 0.13808 [I 2022-10-27 09:39:32] ...optimizers/trijoint.py.151: total_norm: 20.322115 [I 2022-10-27 09:39:38] ...trap/engines/engine.py.224: train: epoch 60 | batch 1070/2304 [I 2022-10-27 09:39:38] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:28 [I 2022-10-27 09:39:38] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00260 [I 2022-10-27 09:39:38] ...trap/engines/engine.py.230: loss: 0.14193 [I 2022-10-27 09:39:38] ...optimizers/trijoint.py.151: total_norm: 16.940990 [I 2022-10-27 09:39:43] ...trap/engines/engine.py.224: train: epoch 60 | batch 1080/2304 [I 2022-10-27 09:39:43] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:21 [I 2022-10-27 09:39:43] ...trap/engines/engine.py.229: process: 0.55484 | load: 0.00258 [I 2022-10-27 09:39:43] ...trap/engines/engine.py.230: loss: 0.15399 [I 2022-10-27 09:39:43] ...optimizers/trijoint.py.151: total_norm: 11.796684 [I 2022-10-27 09:39:49] ...trap/engines/engine.py.224: train: epoch 60 | batch 1090/2304 [I 2022-10-27 09:39:49] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:17 [I 2022-10-27 09:39:49] ...trap/engines/engine.py.229: process: 0.56178 | load: 0.00252 [I 2022-10-27 09:39:49] ...trap/engines/engine.py.230: loss: 0.17603 [I 2022-10-27 09:39:49] ...optimizers/trijoint.py.151: total_norm: 18.039246 [I 2022-10-27 09:39:54] ...trap/engines/engine.py.224: train: epoch 60 | batch 1100/2304 [I 2022-10-27 09:39:54] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:10 [I 2022-10-27 09:39:54] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00260 [I 2022-10-27 09:39:54] ...trap/engines/engine.py.230: loss: 0.18263 [I 2022-10-27 09:39:54] ...optimizers/trijoint.py.151: total_norm: 17.084291 [I 2022-10-27 09:40:00] ...trap/engines/engine.py.224: train: epoch 60 | batch 1110/2304 [I 2022-10-27 09:40:00] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:03 [I 2022-10-27 09:40:00] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00256 [I 2022-10-27 09:40:00] ...trap/engines/engine.py.230: loss: 0.09887 [I 2022-10-27 09:40:00] ...optimizers/trijoint.py.151: total_norm: 10.290426 [I 2022-10-27 09:40:05] ...trap/engines/engine.py.224: train: epoch 60 | batch 1120/2304 [I 2022-10-27 09:40:05] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:10:58 [I 2022-10-27 09:40:05] ...trap/engines/engine.py.229: process: 0.55343 | load: 0.00260 [I 2022-10-27 09:40:05] ...trap/engines/engine.py.230: loss: 0.18352 [I 2022-10-27 09:40:05] ...optimizers/trijoint.py.151: total_norm: 17.560089 [I 2022-10-27 09:40:11] ...trap/engines/engine.py.224: train: epoch 60 | batch 1130/2304 [I 2022-10-27 09:40:11] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:52 [I 2022-10-27 09:40:11] ...trap/engines/engine.py.229: process: 0.54907 | load: 0.00251 [I 2022-10-27 09:40:11] ...trap/engines/engine.py.230: loss: 0.15916 [I 2022-10-27 09:40:11] ...optimizers/trijoint.py.151: total_norm: 38.147743 [I 2022-10-27 09:40:17] ...trap/engines/engine.py.224: train: epoch 60 | batch 1140/2304 [I 2022-10-27 09:40:17] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:10:48 [I 2022-10-27 09:40:17] ...trap/engines/engine.py.229: process: 0.55124 | load: 0.00253 [I 2022-10-27 09:40:17] ...trap/engines/engine.py.230: loss: 0.13005 [I 2022-10-27 09:40:17] ...optimizers/trijoint.py.151: total_norm: 12.948231 [I 2022-10-27 09:40:22] ...trap/engines/engine.py.224: train: epoch 60 | batch 1150/2304 [I 2022-10-27 09:40:22] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:10:40 [I 2022-10-27 09:40:22] ...trap/engines/engine.py.229: process: 0.55321 | load: 0.00261 [I 2022-10-27 09:40:22] ...trap/engines/engine.py.230: loss: 0.12323 [I 2022-10-27 09:40:22] ...optimizers/trijoint.py.151: total_norm: 14.120933 [I 2022-10-27 09:40:28] ...trap/engines/engine.py.224: train: epoch 60 | batch 1160/2304 [I 2022-10-27 09:40:28] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:10:33 [I 2022-10-27 09:40:28] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00257 [I 2022-10-27 09:40:28] ...trap/engines/engine.py.230: loss: 0.13285 [I 2022-10-27 09:40:28] ...optimizers/trijoint.py.151: total_norm: 38.388592 [I 2022-10-27 09:40:33] ...trap/engines/engine.py.224: train: epoch 60 | batch 1170/2304 [I 2022-10-27 09:40:33] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:28 [I 2022-10-27 09:40:33] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00262 [I 2022-10-27 09:40:33] ...trap/engines/engine.py.230: loss: 0.13933 [I 2022-10-27 09:40:33] ...optimizers/trijoint.py.151: total_norm: 36.694862 [I 2022-10-27 09:40:39] ...trap/engines/engine.py.224: train: epoch 60 | batch 1180/2304 [I 2022-10-27 09:40:39] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:40 [I 2022-10-27 09:40:39] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00262 [I 2022-10-27 09:40:39] ...trap/engines/engine.py.230: loss: 0.11101 [I 2022-10-27 09:40:39] ...optimizers/trijoint.py.151: total_norm: 15.083580 [I 2022-10-27 09:40:45] ...trap/engines/engine.py.224: train: epoch 60 | batch 1190/2304 [I 2022-10-27 09:40:45] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:21 [I 2022-10-27 09:40:45] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00262 [I 2022-10-27 09:40:45] ...trap/engines/engine.py.230: loss: 0.18465 [I 2022-10-27 09:40:45] ...optimizers/trijoint.py.151: total_norm: 17.435881 [I 2022-10-27 09:40:50] ...trap/engines/engine.py.224: train: epoch 60 | batch 1200/2304 [I 2022-10-27 09:40:50] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:11 [I 2022-10-27 09:40:50] ...trap/engines/engine.py.229: process: 0.55512 | load: 0.00252 [I 2022-10-27 09:40:50] ...trap/engines/engine.py.230: loss: 0.15454 [I 2022-10-27 09:40:50] ...optimizers/trijoint.py.151: total_norm: 12.855410 [I 2022-10-27 09:40:56] ...trap/engines/engine.py.224: train: epoch 60 | batch 1210/2304 [I 2022-10-27 09:40:56] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:09 [I 2022-10-27 09:40:56] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00257 [I 2022-10-27 09:40:56] ...trap/engines/engine.py.230: loss: 0.15260 [I 2022-10-27 09:40:56] ...optimizers/trijoint.py.151: total_norm: 10.209866 [I 2022-10-27 09:41:01] ...trap/engines/engine.py.224: train: epoch 60 | batch 1220/2304 [I 2022-10-27 09:41:01] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:00 [I 2022-10-27 09:41:01] ...trap/engines/engine.py.229: process: 0.55338 | load: 0.00272 [I 2022-10-27 09:41:01] ...trap/engines/engine.py.230: loss: 0.12946 [I 2022-10-27 09:41:01] ...optimizers/trijoint.py.151: total_norm: 21.535065 [I 2022-10-27 09:41:07] ...trap/engines/engine.py.224: train: epoch 60 | batch 1230/2304 [I 2022-10-27 09:41:07] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:09:58 [I 2022-10-27 09:41:07] ...trap/engines/engine.py.229: process: 0.56907 | load: 0.00276 [I 2022-10-27 09:41:07] ...trap/engines/engine.py.230: loss: 0.20576 [I 2022-10-27 09:41:07] ...optimizers/trijoint.py.151: total_norm: 15.287035 [I 2022-10-27 09:41:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 1240/2304 [I 2022-10-27 09:41:12] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:09:45 [I 2022-10-27 09:41:12] ...trap/engines/engine.py.229: process: 0.54342 | load: 0.00254 [I 2022-10-27 09:41:12] ...trap/engines/engine.py.230: loss: 0.15100 [I 2022-10-27 09:41:12] ...optimizers/trijoint.py.151: total_norm: 22.800495 [I 2022-10-27 09:41:18] ...trap/engines/engine.py.224: train: epoch 60 | batch 1250/2304 [I 2022-10-27 09:41:18] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:09:42 [I 2022-10-27 09:41:18] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00251 [I 2022-10-27 09:41:18] ...trap/engines/engine.py.230: loss: 0.23602 [I 2022-10-27 09:41:18] ...optimizers/trijoint.py.151: total_norm: 18.894859 [I 2022-10-27 09:41:23] ...trap/engines/engine.py.224: train: epoch 60 | batch 1260/2304 [I 2022-10-27 09:41:23] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:09:41 [I 2022-10-27 09:41:23] ...trap/engines/engine.py.229: process: 0.56475 | load: 0.00260 [I 2022-10-27 09:41:23] ...trap/engines/engine.py.230: loss: 0.16049 [I 2022-10-27 09:41:23] ...optimizers/trijoint.py.151: total_norm: 20.568605 [I 2022-10-27 09:41:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 1270/2304 [I 2022-10-27 09:41:29] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:37 [I 2022-10-27 09:41:29] ...trap/engines/engine.py.229: process: 0.55096 | load: 0.00283 [I 2022-10-27 09:41:29] ...trap/engines/engine.py.230: loss: 0.12822 [I 2022-10-27 09:41:29] ...optimizers/trijoint.py.151: total_norm: 10.191145 [I 2022-10-27 09:41:35] ...trap/engines/engine.py.224: train: epoch 60 | batch 1280/2304 [I 2022-10-27 09:41:35] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:25 [I 2022-10-27 09:41:35] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00258 [I 2022-10-27 09:41:35] ...trap/engines/engine.py.230: loss: 0.17061 [I 2022-10-27 09:41:35] ...optimizers/trijoint.py.151: total_norm: 17.938013 [I 2022-10-27 09:41:40] ...trap/engines/engine.py.224: train: epoch 60 | batch 1290/2304 [I 2022-10-27 09:41:40] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:19 [I 2022-10-27 09:41:40] ...trap/engines/engine.py.229: process: 0.55023 | load: 0.00251 [I 2022-10-27 09:41:40] ...trap/engines/engine.py.230: loss: 0.16000 [I 2022-10-27 09:41:40] ...optimizers/trijoint.py.151: total_norm: 13.507391 [I 2022-10-27 09:41:46] ...trap/engines/engine.py.224: train: epoch 60 | batch 1300/2304 [I 2022-10-27 09:41:46] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:18 [I 2022-10-27 09:41:46] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00250 [I 2022-10-27 09:41:46] ...trap/engines/engine.py.230: loss: 0.15781 [I 2022-10-27 09:41:46] ...optimizers/trijoint.py.151: total_norm: 16.092842 [I 2022-10-27 09:41:51] ...trap/engines/engine.py.224: train: epoch 60 | batch 1310/2304 [I 2022-10-27 09:41:51] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:12 [I 2022-10-27 09:41:51] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00248 [I 2022-10-27 09:41:51] ...trap/engines/engine.py.230: loss: 0.14447 [I 2022-10-27 09:41:51] ...optimizers/trijoint.py.151: total_norm: 8.795104 [I 2022-10-27 09:41:57] ...trap/engines/engine.py.224: train: epoch 60 | batch 1320/2304 [I 2022-10-27 09:41:57] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:04 [I 2022-10-27 09:41:57] ...trap/engines/engine.py.229: process: 0.55354 | load: 0.00254 [I 2022-10-27 09:41:57] ...trap/engines/engine.py.230: loss: 0.11285 [I 2022-10-27 09:41:57] ...optimizers/trijoint.py.151: total_norm: 26.936243 [I 2022-10-27 09:42:02] ...trap/engines/engine.py.224: train: epoch 60 | batch 1330/2304 [I 2022-10-27 09:42:02] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:03 [I 2022-10-27 09:42:02] ...trap/engines/engine.py.229: process: 0.56523 | load: 0.00251 [I 2022-10-27 09:42:02] ...trap/engines/engine.py.230: loss: 0.21064 [I 2022-10-27 09:42:02] ...optimizers/trijoint.py.151: total_norm: 20.192324 [I 2022-10-27 09:42:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 1340/2304 [I 2022-10-27 09:42:08] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:08:56 [I 2022-10-27 09:42:08] ...trap/engines/engine.py.229: process: 0.54606 | load: 0.00255 [I 2022-10-27 09:42:08] ...trap/engines/engine.py.230: loss: 0.13157 [I 2022-10-27 09:42:08] ...optimizers/trijoint.py.151: total_norm: 12.066617 [I 2022-10-27 09:42:14] ...trap/engines/engine.py.224: train: epoch 60 | batch 1350/2304 [I 2022-10-27 09:42:14] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:08:52 [I 2022-10-27 09:42:14] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00248 [I 2022-10-27 09:42:14] ...trap/engines/engine.py.230: loss: 0.12930 [I 2022-10-27 09:42:14] ...optimizers/trijoint.py.151: total_norm: 14.116836 [I 2022-10-27 09:42:19] ...trap/engines/engine.py.224: train: epoch 60 | batch 1360/2304 [I 2022-10-27 09:42:19] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:08:44 [I 2022-10-27 09:42:19] ...trap/engines/engine.py.229: process: 0.55353 | load: 0.00252 [I 2022-10-27 09:42:19] ...trap/engines/engine.py.230: loss: 0.17370 [I 2022-10-27 09:42:19] ...optimizers/trijoint.py.151: total_norm: 25.725798 [I 2022-10-27 09:42:25] ...trap/engines/engine.py.224: train: epoch 60 | batch 1370/2304 [I 2022-10-27 09:42:25] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:08:41 [I 2022-10-27 09:42:25] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00260 [I 2022-10-27 09:42:25] ...trap/engines/engine.py.230: loss: 0.12482 [I 2022-10-27 09:42:25] ...optimizers/trijoint.py.151: total_norm: 16.377056 [I 2022-10-27 09:42:30] ...trap/engines/engine.py.224: train: epoch 60 | batch 1380/2304 [I 2022-10-27 09:42:30] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:32 [I 2022-10-27 09:42:30] ...trap/engines/engine.py.229: process: 0.55274 | load: 0.00259 [I 2022-10-27 09:42:30] ...trap/engines/engine.py.230: loss: 0.17423 [I 2022-10-27 09:42:30] ...optimizers/trijoint.py.151: total_norm: 31.235203 [I 2022-10-27 09:42:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 1390/2304 [I 2022-10-27 09:42:36] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:29 [I 2022-10-27 09:42:36] ...trap/engines/engine.py.229: process: 0.56097 | load: 0.00255 [I 2022-10-27 09:42:36] ...trap/engines/engine.py.230: loss: 0.21983 [I 2022-10-27 09:42:36] ...optimizers/trijoint.py.151: total_norm: 11.374357 [I 2022-10-27 09:42:41] ...trap/engines/engine.py.224: train: epoch 60 | batch 1400/2304 [I 2022-10-27 09:42:41] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:25 [I 2022-10-27 09:42:41] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00251 [I 2022-10-27 09:42:41] ...trap/engines/engine.py.230: loss: 0.13102 [I 2022-10-27 09:42:41] ...optimizers/trijoint.py.151: total_norm: 15.870876 [I 2022-10-27 09:42:47] ...trap/engines/engine.py.224: train: epoch 60 | batch 1410/2304 [I 2022-10-27 09:42:47] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:18 [I 2022-10-27 09:42:47] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00253 [I 2022-10-27 09:42:47] ...trap/engines/engine.py.230: loss: 0.17034 [I 2022-10-27 09:42:47] ...optimizers/trijoint.py.151: total_norm: 18.913160 [I 2022-10-27 09:42:53] ...trap/engines/engine.py.224: train: epoch 60 | batch 1420/2304 [I 2022-10-27 09:42:53] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:12 [I 2022-10-27 09:42:53] ...trap/engines/engine.py.229: process: 0.55923 | load: 0.00255 [I 2022-10-27 09:42:53] ...trap/engines/engine.py.230: loss: 0.14007 [I 2022-10-27 09:42:53] ...optimizers/trijoint.py.151: total_norm: 11.894075 [I 2022-10-27 09:42:58] ...trap/engines/engine.py.224: train: epoch 60 | batch 1430/2304 [I 2022-10-27 09:42:58] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:04 [I 2022-10-27 09:42:58] ...trap/engines/engine.py.229: process: 0.54925 | load: 0.00262 [I 2022-10-27 09:42:58] ...trap/engines/engine.py.230: loss: 0.21727 [I 2022-10-27 09:42:58] ...optimizers/trijoint.py.151: total_norm: 10.710155 [I 2022-10-27 09:43:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 1440/2304 [I 2022-10-27 09:43:04] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:07:59 [I 2022-10-27 09:43:04] ...trap/engines/engine.py.229: process: 0.55110 | load: 0.00263 [I 2022-10-27 09:43:04] ...trap/engines/engine.py.230: loss: 0.18592 [I 2022-10-27 09:43:04] ...optimizers/trijoint.py.151: total_norm: 74.048141 [I 2022-10-27 09:43:09] ...trap/engines/engine.py.224: train: epoch 60 | batch 1450/2304 [I 2022-10-27 09:43:09] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:07:53 [I 2022-10-27 09:43:09] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00256 [I 2022-10-27 09:43:09] ...trap/engines/engine.py.230: loss: 0.15825 [I 2022-10-27 09:43:09] ...optimizers/trijoint.py.151: total_norm: 10.621824 [I 2022-10-27 09:43:15] ...trap/engines/engine.py.224: train: epoch 60 | batch 1460/2304 [I 2022-10-27 09:43:15] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:07:45 [I 2022-10-27 09:43:15] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00255 [I 2022-10-27 09:43:15] ...trap/engines/engine.py.230: loss: 0.11867 [I 2022-10-27 09:43:15] ...optimizers/trijoint.py.151: total_norm: 10.001581 [I 2022-10-27 09:43:20] ...trap/engines/engine.py.224: train: epoch 60 | batch 1470/2304 [I 2022-10-27 09:43:20] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:07:40 [I 2022-10-27 09:43:20] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00248 [I 2022-10-27 09:43:20] ...trap/engines/engine.py.230: loss: 0.15540 [I 2022-10-27 09:43:20] ...optimizers/trijoint.py.151: total_norm: 12.454268 [I 2022-10-27 09:43:26] ...trap/engines/engine.py.224: train: epoch 60 | batch 1480/2304 [I 2022-10-27 09:43:26] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:41 [I 2022-10-27 09:43:26] ...trap/engines/engine.py.229: process: 0.56692 | load: 0.00244 [I 2022-10-27 09:43:26] ...trap/engines/engine.py.230: loss: 0.17226 [I 2022-10-27 09:43:26] ...optimizers/trijoint.py.151: total_norm: 15.364786 [I 2022-10-27 09:43:32] ...trap/engines/engine.py.224: train: epoch 60 | batch 1490/2304 [I 2022-10-27 09:43:32] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:33 [I 2022-10-27 09:43:32] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00247 [I 2022-10-27 09:43:32] ...trap/engines/engine.py.230: loss: 0.17974 [I 2022-10-27 09:43:32] ...optimizers/trijoint.py.151: total_norm: 27.795732 [I 2022-10-27 09:43:37] ...trap/engines/engine.py.224: train: epoch 60 | batch 1500/2304 [I 2022-10-27 09:43:37] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:26 [I 2022-10-27 09:43:37] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00248 [I 2022-10-27 09:43:37] ...trap/engines/engine.py.230: loss: 0.14171 [I 2022-10-27 09:43:37] ...optimizers/trijoint.py.151: total_norm: 34.309475 [I 2022-10-27 09:43:43] ...trap/engines/engine.py.224: train: epoch 60 | batch 1510/2304 [I 2022-10-27 09:43:43] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:21 [I 2022-10-27 09:43:43] ...trap/engines/engine.py.229: process: 0.55799 | load: 0.00248 [I 2022-10-27 09:43:43] ...trap/engines/engine.py.230: loss: 0.19306 [I 2022-10-27 09:43:43] ...optimizers/trijoint.py.151: total_norm: 22.713820 [I 2022-10-27 09:43:48] ...trap/engines/engine.py.224: train: epoch 60 | batch 1520/2304 [I 2022-10-27 09:43:48] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:17 [I 2022-10-27 09:43:48] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00243 [I 2022-10-27 09:43:48] ...trap/engines/engine.py.230: loss: 0.12233 [I 2022-10-27 09:43:48] ...optimizers/trijoint.py.151: total_norm: 13.708283 [I 2022-10-27 09:43:54] ...trap/engines/engine.py.224: train: epoch 60 | batch 1530/2304 [I 2022-10-27 09:43:54] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:10 [I 2022-10-27 09:43:54] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00241 [I 2022-10-27 09:43:54] ...trap/engines/engine.py.230: loss: 0.12499 [I 2022-10-27 09:43:54] ...optimizers/trijoint.py.151: total_norm: 12.207406 [I 2022-10-27 09:43:59] ...trap/engines/engine.py.224: train: epoch 60 | batch 1540/2304 [I 2022-10-27 09:43:59] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:05 [I 2022-10-27 09:43:59] ...trap/engines/engine.py.229: process: 0.56273 | load: 0.00241 [I 2022-10-27 09:43:59] ...trap/engines/engine.py.230: loss: 0.11664 [I 2022-10-27 09:43:59] ...optimizers/trijoint.py.151: total_norm: 9.686278 [I 2022-10-27 09:44:05] ...trap/engines/engine.py.224: train: epoch 60 | batch 1550/2304 [I 2022-10-27 09:44:05] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:06:54 [I 2022-10-27 09:44:05] ...trap/engines/engine.py.229: process: 0.54326 | load: 0.00245 [I 2022-10-27 09:44:05] ...trap/engines/engine.py.230: loss: 0.16551 [I 2022-10-27 09:44:05] ...optimizers/trijoint.py.151: total_norm: 20.793253 [I 2022-10-27 09:44:10] ...trap/engines/engine.py.224: train: epoch 60 | batch 1560/2304 [I 2022-10-27 09:44:10] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:06:49 [I 2022-10-27 09:44:10] ...trap/engines/engine.py.229: process: 0.54315 | load: 0.00247 [I 2022-10-27 09:44:10] ...trap/engines/engine.py.230: loss: 0.09732 [I 2022-10-27 09:44:10] ...optimizers/trijoint.py.151: total_norm: 9.878388 [I 2022-10-27 09:44:34] ...trap/engines/engine.py.224: train: epoch 60 | batch 1570/2304 [I 2022-10-27 09:44:34] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:24:43 [I 2022-10-27 09:44:34] ...trap/engines/engine.py.229: process: 0.54686 | load: 0.00239 [I 2022-10-27 09:44:34] ...trap/engines/engine.py.230: loss: 0.13708 [I 2022-10-27 09:44:34] ...optimizers/trijoint.py.151: total_norm: 21.382696 [I 2022-10-27 09:44:39] ...trap/engines/engine.py.224: train: epoch 60 | batch 1580/2304 [I 2022-10-27 09:44:39] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:08:33 [I 2022-10-27 09:44:39] ...trap/engines/engine.py.229: process: 0.55406 | load: 0.00235 [I 2022-10-27 09:44:39] ...trap/engines/engine.py.230: loss: 0.15088 [I 2022-10-27 09:44:39] ...optimizers/trijoint.py.151: total_norm: 13.176688 [I 2022-10-27 09:44:45] ...trap/engines/engine.py.224: train: epoch 60 | batch 1590/2304 [I 2022-10-27 09:44:45] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:47 [I 2022-10-27 09:44:45] ...trap/engines/engine.py.229: process: 0.54668 | load: 0.00235 [I 2022-10-27 09:44:45] ...trap/engines/engine.py.230: loss: 0.16407 [I 2022-10-27 09:44:45] ...optimizers/trijoint.py.151: total_norm: 20.073793 [I 2022-10-27 09:44:51] ...trap/engines/engine.py.224: train: epoch 60 | batch 1600/2304 [I 2022-10-27 09:44:51] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:33 [I 2022-10-27 09:44:51] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00237 [I 2022-10-27 09:44:51] ...trap/engines/engine.py.230: loss: 0.11613 [I 2022-10-27 09:44:51] ...optimizers/trijoint.py.151: total_norm: 10.984839 [I 2022-10-27 09:44:56] ...trap/engines/engine.py.224: train: epoch 60 | batch 1610/2304 [I 2022-10-27 09:44:56] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:26 [I 2022-10-27 09:44:56] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00238 [I 2022-10-27 09:44:56] ...trap/engines/engine.py.230: loss: 0.19442 [I 2022-10-27 09:44:56] ...optimizers/trijoint.py.151: total_norm: 23.553177 [I 2022-10-27 09:45:02] ...trap/engines/engine.py.224: train: epoch 60 | batch 1620/2304 [I 2022-10-27 09:45:02] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:19 [I 2022-10-27 09:45:02] ...trap/engines/engine.py.229: process: 0.54857 | load: 0.00239 [I 2022-10-27 09:45:02] ...trap/engines/engine.py.230: loss: 0.18343 [I 2022-10-27 09:45:02] ...optimizers/trijoint.py.151: total_norm: 35.778809 [I 2022-10-27 09:45:07] ...trap/engines/engine.py.224: train: epoch 60 | batch 1630/2304 [I 2022-10-27 09:45:07] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:13 [I 2022-10-27 09:45:07] ...trap/engines/engine.py.229: process: 0.54740 | load: 0.00238 [I 2022-10-27 09:45:07] ...trap/engines/engine.py.230: loss: 0.13050 [I 2022-10-27 09:45:07] ...optimizers/trijoint.py.151: total_norm: 18.492920 [I 2022-10-27 09:45:13] ...trap/engines/engine.py.224: train: epoch 60 | batch 1640/2304 [I 2022-10-27 09:45:13] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:06:08 [I 2022-10-27 09:45:13] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00245 [I 2022-10-27 09:45:13] ...trap/engines/engine.py.230: loss: 0.13998 [I 2022-10-27 09:45:13] ...optimizers/trijoint.py.151: total_norm: 16.948212 [I 2022-10-27 09:45:18] ...trap/engines/engine.py.224: train: epoch 60 | batch 1650/2304 [I 2022-10-27 09:45:18] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:06:03 [I 2022-10-27 09:45:18] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00242 [I 2022-10-27 09:45:18] ...trap/engines/engine.py.230: loss: 0.20097 [I 2022-10-27 09:45:18] ...optimizers/trijoint.py.151: total_norm: 18.777130 [I 2022-10-27 09:45:24] ...trap/engines/engine.py.224: train: epoch 60 | batch 1660/2304 [I 2022-10-27 09:45:24] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:05:56 [I 2022-10-27 09:45:24] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00240 [I 2022-10-27 09:45:24] ...trap/engines/engine.py.230: loss: 0.10999 [I 2022-10-27 09:45:24] ...optimizers/trijoint.py.151: total_norm: 10.497849 [I 2022-10-27 09:45:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 1670/2304 [I 2022-10-27 09:45:29] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:52 [I 2022-10-27 09:45:29] ...trap/engines/engine.py.229: process: 0.54242 | load: 0.00238 [I 2022-10-27 09:45:29] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-27 09:45:29] ...optimizers/trijoint.py.151: total_norm: 10.503287 [I 2022-10-27 09:45:35] ...trap/engines/engine.py.224: train: epoch 60 | batch 1680/2304 [I 2022-10-27 09:45:35] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:46 [I 2022-10-27 09:45:35] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00240 [I 2022-10-27 09:45:35] ...trap/engines/engine.py.230: loss: 0.14268 [I 2022-10-27 09:45:35] ...optimizers/trijoint.py.151: total_norm: 17.467112 [I 2022-10-27 09:45:41] ...trap/engines/engine.py.224: train: epoch 60 | batch 1690/2304 [I 2022-10-27 09:45:41] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:40 [I 2022-10-27 09:45:41] ...trap/engines/engine.py.229: process: 0.54966 | load: 0.00252 [I 2022-10-27 09:45:41] ...trap/engines/engine.py.230: loss: 0.10413 [I 2022-10-27 09:45:41] ...optimizers/trijoint.py.151: total_norm: 13.692523 [I 2022-10-27 09:45:46] ...trap/engines/engine.py.224: train: epoch 60 | batch 1700/2304 [I 2022-10-27 09:45:46] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:35 [I 2022-10-27 09:45:46] ...trap/engines/engine.py.229: process: 0.55009 | load: 0.00244 [I 2022-10-27 09:45:46] ...trap/engines/engine.py.230: loss: 0.11335 [I 2022-10-27 09:45:46] ...optimizers/trijoint.py.151: total_norm: 9.497992 [I 2022-10-27 09:45:52] ...trap/engines/engine.py.224: train: epoch 60 | batch 1710/2304 [I 2022-10-27 09:45:52] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:36 [I 2022-10-27 09:45:52] ...trap/engines/engine.py.229: process: 0.59709 | load: 0.00243 [I 2022-10-27 09:45:52] ...trap/engines/engine.py.230: loss: 0.15564 [I 2022-10-27 09:45:52] ...optimizers/trijoint.py.151: total_norm: 15.040130 [I 2022-10-27 09:45:57] ...trap/engines/engine.py.224: train: epoch 60 | batch 1720/2304 [I 2022-10-27 09:45:57] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:26 [I 2022-10-27 09:45:57] ...trap/engines/engine.py.229: process: 0.55488 | load: 0.00245 [I 2022-10-27 09:45:57] ...trap/engines/engine.py.230: loss: 0.09742 [I 2022-10-27 09:45:57] ...optimizers/trijoint.py.151: total_norm: 7.965683 [I 2022-10-27 09:46:03] ...trap/engines/engine.py.224: train: epoch 60 | batch 1730/2304 [I 2022-10-27 09:46:03] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:28 [I 2022-10-27 09:46:03] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00241 [I 2022-10-27 09:46:03] ...trap/engines/engine.py.230: loss: 0.15208 [I 2022-10-27 09:46:03] ...optimizers/trijoint.py.151: total_norm: 12.343431 [I 2022-10-27 09:46:09] ...trap/engines/engine.py.224: train: epoch 60 | batch 1740/2304 [I 2022-10-27 09:46:09] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:13 [I 2022-10-27 09:46:09] ...trap/engines/engine.py.229: process: 0.55042 | load: 0.00240 [I 2022-10-27 09:46:09] ...trap/engines/engine.py.230: loss: 0.14018 [I 2022-10-27 09:46:09] ...optimizers/trijoint.py.151: total_norm: 15.615541 [I 2022-10-27 09:46:14] ...trap/engines/engine.py.224: train: epoch 60 | batch 1750/2304 [I 2022-10-27 09:46:14] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:05:07 [I 2022-10-27 09:46:14] ...trap/engines/engine.py.229: process: 0.53915 | load: 0.00247 [I 2022-10-27 09:46:14] ...trap/engines/engine.py.230: loss: 0.13581 [I 2022-10-27 09:46:14] ...optimizers/trijoint.py.151: total_norm: 14.517288 [I 2022-10-27 09:46:20] ...trap/engines/engine.py.224: train: epoch 60 | batch 1760/2304 [I 2022-10-27 09:46:20] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:03 [I 2022-10-27 09:46:20] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00243 [I 2022-10-27 09:46:20] ...trap/engines/engine.py.230: loss: 0.11544 [I 2022-10-27 09:46:20] ...optimizers/trijoint.py.151: total_norm: 12.830528 [I 2022-10-27 09:46:25] ...trap/engines/engine.py.224: train: epoch 60 | batch 1770/2304 [I 2022-10-27 09:46:25] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:57 [I 2022-10-27 09:46:25] ...trap/engines/engine.py.229: process: 0.55224 | load: 0.00240 [I 2022-10-27 09:46:25] ...trap/engines/engine.py.230: loss: 0.10602 [I 2022-10-27 09:46:25] ...optimizers/trijoint.py.151: total_norm: 13.043818 [I 2022-10-27 09:46:31] ...trap/engines/engine.py.224: train: epoch 60 | batch 1780/2304 [I 2022-10-27 09:46:31] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:49 [I 2022-10-27 09:46:31] ...trap/engines/engine.py.229: process: 0.53980 | load: 0.00244 [I 2022-10-27 09:46:31] ...trap/engines/engine.py.230: loss: 0.12545 [I 2022-10-27 09:46:31] ...optimizers/trijoint.py.151: total_norm: 7.953518 [I 2022-10-27 09:46:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 1790/2304 [I 2022-10-27 09:46:36] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:45 [I 2022-10-27 09:46:36] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00240 [I 2022-10-27 09:46:36] ...trap/engines/engine.py.230: loss: 0.15365 [I 2022-10-27 09:46:36] ...optimizers/trijoint.py.151: total_norm: 9.188285 [I 2022-10-27 09:46:42] ...trap/engines/engine.py.224: train: epoch 60 | batch 1800/2304 [I 2022-10-27 09:46:42] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:45 [I 2022-10-27 09:46:42] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00238 [I 2022-10-27 09:46:42] ...trap/engines/engine.py.230: loss: 0.11181 [I 2022-10-27 09:46:42] ...optimizers/trijoint.py.151: total_norm: 7.889379 [I 2022-10-27 09:46:48] ...trap/engines/engine.py.224: train: epoch 60 | batch 1810/2304 [I 2022-10-27 09:46:48] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:35 [I 2022-10-27 09:46:48] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00248 [I 2022-10-27 09:46:48] ...trap/engines/engine.py.230: loss: 0.12388 [I 2022-10-27 09:46:48] ...optimizers/trijoint.py.151: total_norm: 10.126650 [I 2022-10-27 09:46:53] ...trap/engines/engine.py.224: train: epoch 60 | batch 1820/2304 [I 2022-10-27 09:46:53] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:30 [I 2022-10-27 09:46:53] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00244 [I 2022-10-27 09:46:53] ...trap/engines/engine.py.230: loss: 0.12512 [I 2022-10-27 09:46:53] ...optimizers/trijoint.py.151: total_norm: 23.763727 [I 2022-10-27 09:46:59] ...trap/engines/engine.py.224: train: epoch 60 | batch 1830/2304 [I 2022-10-27 09:46:59] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:24 [I 2022-10-27 09:46:59] ...trap/engines/engine.py.229: process: 0.55293 | load: 0.00248 [I 2022-10-27 09:46:59] ...trap/engines/engine.py.230: loss: 0.17374 [I 2022-10-27 09:46:59] ...optimizers/trijoint.py.151: total_norm: 16.009607 [I 2022-10-27 09:47:05] ...trap/engines/engine.py.224: train: epoch 60 | batch 1840/2304 [I 2022-10-27 09:47:05] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:18 [I 2022-10-27 09:47:05] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00247 [I 2022-10-27 09:47:05] ...trap/engines/engine.py.230: loss: 0.16085 [I 2022-10-27 09:47:05] ...optimizers/trijoint.py.151: total_norm: 20.499554 [I 2022-10-27 09:47:10] ...trap/engines/engine.py.224: train: epoch 60 | batch 1850/2304 [I 2022-10-27 09:47:10] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:11 [I 2022-10-27 09:47:10] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00244 [I 2022-10-27 09:47:10] ...trap/engines/engine.py.230: loss: 0.14503 [I 2022-10-27 09:47:10] ...optimizers/trijoint.py.151: total_norm: 21.947729 [I 2022-10-27 09:47:16] ...trap/engines/engine.py.224: train: epoch 60 | batch 1860/2304 [I 2022-10-27 09:47:16] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:04:09 [I 2022-10-27 09:47:16] ...trap/engines/engine.py.229: process: 0.57534 | load: 0.00245 [I 2022-10-27 09:47:16] ...trap/engines/engine.py.230: loss: 0.13379 [I 2022-10-27 09:47:16] ...optimizers/trijoint.py.151: total_norm: 12.726461 [I 2022-10-27 09:47:22] ...trap/engines/engine.py.224: train: epoch 60 | batch 1870/2304 [I 2022-10-27 09:47:22] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:04:04 [I 2022-10-27 09:47:22] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00249 [I 2022-10-27 09:47:22] ...trap/engines/engine.py.230: loss: 0.13666 [I 2022-10-27 09:47:22] ...optimizers/trijoint.py.151: total_norm: 19.089851 [I 2022-10-27 09:47:27] ...trap/engines/engine.py.224: train: epoch 60 | batch 1880/2304 [I 2022-10-27 09:47:27] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:03:56 [I 2022-10-27 09:47:27] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00246 [I 2022-10-27 09:47:27] ...trap/engines/engine.py.230: loss: 0.19283 [I 2022-10-27 09:47:27] ...optimizers/trijoint.py.151: total_norm: 16.061998 [I 2022-10-27 09:47:33] ...trap/engines/engine.py.224: train: epoch 60 | batch 1890/2304 [I 2022-10-27 09:47:33] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:51 [I 2022-10-27 09:47:33] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00241 [I 2022-10-27 09:47:33] ...trap/engines/engine.py.230: loss: 0.13229 [I 2022-10-27 09:47:33] ...optimizers/trijoint.py.151: total_norm: 16.534195 [I 2022-10-27 09:47:38] ...trap/engines/engine.py.224: train: epoch 60 | batch 1900/2304 [I 2022-10-27 09:47:38] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:45 [I 2022-10-27 09:47:38] ...trap/engines/engine.py.229: process: 0.55713 | load: 0.00248 [I 2022-10-27 09:47:38] ...trap/engines/engine.py.230: loss: 0.12538 [I 2022-10-27 09:47:38] ...optimizers/trijoint.py.151: total_norm: 15.978372 [I 2022-10-27 09:47:44] ...trap/engines/engine.py.224: train: epoch 60 | batch 1910/2304 [I 2022-10-27 09:47:44] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:40 [I 2022-10-27 09:47:44] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00243 [I 2022-10-27 09:47:44] ...trap/engines/engine.py.230: loss: 0.09992 [I 2022-10-27 09:47:44] ...optimizers/trijoint.py.151: total_norm: 9.543121 [I 2022-10-27 09:47:49] ...trap/engines/engine.py.224: train: epoch 60 | batch 1920/2304 [I 2022-10-27 09:47:49] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:33 [I 2022-10-27 09:47:49] ...trap/engines/engine.py.229: process: 0.56229 | load: 0.00258 [I 2022-10-27 09:47:49] ...trap/engines/engine.py.230: loss: 0.14410 [I 2022-10-27 09:47:49] ...optimizers/trijoint.py.151: total_norm: 44.917843 [I 2022-10-27 09:47:55] ...trap/engines/engine.py.224: train: epoch 60 | batch 1930/2304 [I 2022-10-27 09:47:55] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:26 [I 2022-10-27 09:47:55] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00267 [I 2022-10-27 09:47:55] ...trap/engines/engine.py.230: loss: 0.25883 [I 2022-10-27 09:47:55] ...optimizers/trijoint.py.151: total_norm: 18.529985 [I 2022-10-27 09:48:01] ...trap/engines/engine.py.224: train: epoch 60 | batch 1940/2304 [I 2022-10-27 09:48:01] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:21 [I 2022-10-27 09:48:01] ...trap/engines/engine.py.229: process: 0.55490 | load: 0.00251 [I 2022-10-27 09:48:01] ...trap/engines/engine.py.230: loss: 0.14370 [I 2022-10-27 09:48:01] ...optimizers/trijoint.py.151: total_norm: 9.350691 [I 2022-10-27 09:48:06] ...trap/engines/engine.py.224: train: epoch 60 | batch 1950/2304 [I 2022-10-27 09:48:06] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:17 [I 2022-10-27 09:48:06] ...trap/engines/engine.py.229: process: 0.54338 | load: 0.00249 [I 2022-10-27 09:48:06] ...trap/engines/engine.py.230: loss: 0.15932 [I 2022-10-27 09:48:06] ...optimizers/trijoint.py.151: total_norm: 23.186134 [I 2022-10-27 09:48:12] ...trap/engines/engine.py.224: train: epoch 60 | batch 1960/2304 [I 2022-10-27 09:48:12] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:11 [I 2022-10-27 09:48:12] ...trap/engines/engine.py.229: process: 0.56264 | load: 0.00250 [I 2022-10-27 09:48:12] ...trap/engines/engine.py.230: loss: 0.11071 [I 2022-10-27 09:48:12] ...optimizers/trijoint.py.151: total_norm: 51.415356 [I 2022-10-27 09:48:17] ...trap/engines/engine.py.224: train: epoch 60 | batch 1970/2304 [I 2022-10-27 09:48:17] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:03:05 [I 2022-10-27 09:48:17] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00245 [I 2022-10-27 09:48:17] ...trap/engines/engine.py.230: loss: 0.14802 [I 2022-10-27 09:48:17] ...optimizers/trijoint.py.151: total_norm: 18.998257 [I 2022-10-27 09:48:23] ...trap/engines/engine.py.224: train: epoch 60 | batch 1980/2304 [I 2022-10-27 09:48:23] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:03:00 [I 2022-10-27 09:48:23] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00248 [I 2022-10-27 09:48:23] ...trap/engines/engine.py.230: loss: 0.13314 [I 2022-10-27 09:48:23] ...optimizers/trijoint.py.151: total_norm: 19.658594 [I 2022-10-27 09:48:29] ...trap/engines/engine.py.224: train: epoch 60 | batch 1990/2304 [I 2022-10-27 09:48:29] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:59 [I 2022-10-27 09:48:29] ...trap/engines/engine.py.229: process: 0.54980 | load: 0.00247 [I 2022-10-27 09:48:29] ...trap/engines/engine.py.230: loss: 0.21698 [I 2022-10-27 09:48:29] ...optimizers/trijoint.py.151: total_norm: 24.526432 [I 2022-10-27 09:48:34] ...trap/engines/engine.py.224: train: epoch 60 | batch 2000/2304 [I 2022-10-27 09:48:34] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:50 [I 2022-10-27 09:48:34] ...trap/engines/engine.py.229: process: 0.55216 | load: 0.00255 [I 2022-10-27 09:48:34] ...trap/engines/engine.py.230: loss: 0.15314 [I 2022-10-27 09:48:34] ...optimizers/trijoint.py.151: total_norm: 10.084506 [I 2022-10-27 09:48:40] ...trap/engines/engine.py.224: train: epoch 60 | batch 2010/2304 [I 2022-10-27 09:48:40] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:45 [I 2022-10-27 09:48:40] ...trap/engines/engine.py.229: process: 0.56382 | load: 0.00245 [I 2022-10-27 09:48:40] ...trap/engines/engine.py.230: loss: 0.13474 [I 2022-10-27 09:48:40] ...optimizers/trijoint.py.151: total_norm: 14.230339 [I 2022-10-27 09:48:45] ...trap/engines/engine.py.224: train: epoch 60 | batch 2020/2304 [I 2022-10-27 09:48:45] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:39 [I 2022-10-27 09:48:45] ...trap/engines/engine.py.229: process: 0.55975 | load: 0.00242 [I 2022-10-27 09:48:45] ...trap/engines/engine.py.230: loss: 0.11273 [I 2022-10-27 09:48:45] ...optimizers/trijoint.py.151: total_norm: 12.199843 [I 2022-10-27 09:48:51] ...trap/engines/engine.py.224: train: epoch 60 | batch 2030/2304 [I 2022-10-27 09:48:51] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:32 [I 2022-10-27 09:48:51] ...trap/engines/engine.py.229: process: 0.54562 | load: 0.00243 [I 2022-10-27 09:48:51] ...trap/engines/engine.py.230: loss: 0.18366 [I 2022-10-27 09:48:51] ...optimizers/trijoint.py.151: total_norm: 26.953293 [I 2022-10-27 09:48:57] ...trap/engines/engine.py.224: train: epoch 60 | batch 2040/2304 [I 2022-10-27 09:48:57] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:27 [I 2022-10-27 09:48:57] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00244 [I 2022-10-27 09:48:57] ...trap/engines/engine.py.230: loss: 0.18167 [I 2022-10-27 09:48:57] ...optimizers/trijoint.py.151: total_norm: 33.217941 [I 2022-10-27 09:49:02] ...trap/engines/engine.py.224: train: epoch 60 | batch 2050/2304 [I 2022-10-27 09:49:02] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:21 [I 2022-10-27 09:49:02] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00243 [I 2022-10-27 09:49:02] ...trap/engines/engine.py.230: loss: 0.17173 [I 2022-10-27 09:49:02] ...optimizers/trijoint.py.151: total_norm: 12.473586 [I 2022-10-27 09:49:08] ...trap/engines/engine.py.224: train: epoch 60 | batch 2060/2304 [I 2022-10-27 09:49:08] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:15 [I 2022-10-27 09:49:08] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00243 [I 2022-10-27 09:49:08] ...trap/engines/engine.py.230: loss: 0.18174 [I 2022-10-27 09:49:08] ...optimizers/trijoint.py.151: total_norm: 9.909616 [I 2022-10-27 09:49:13] ...trap/engines/engine.py.224: train: epoch 60 | batch 2070/2304 [I 2022-10-27 09:49:13] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:02:10 [I 2022-10-27 09:49:13] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00242 [I 2022-10-27 09:49:13] ...trap/engines/engine.py.230: loss: 0.10766 [I 2022-10-27 09:49:13] ...optimizers/trijoint.py.151: total_norm: 10.487118 [I 2022-10-27 09:49:19] ...trap/engines/engine.py.224: train: epoch 60 | batch 2080/2304 [I 2022-10-27 09:49:19] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:02:05 [I 2022-10-27 09:49:19] ...trap/engines/engine.py.229: process: 0.57604 | load: 0.00243 [I 2022-10-27 09:49:19] ...trap/engines/engine.py.230: loss: 0.11182 [I 2022-10-27 09:49:19] ...optimizers/trijoint.py.151: total_norm: 8.928127 [I 2022-10-27 09:49:24] ...trap/engines/engine.py.224: train: epoch 60 | batch 2090/2304 [I 2022-10-27 09:49:24] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:01:59 [I 2022-10-27 09:49:24] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00246 [I 2022-10-27 09:49:24] ...trap/engines/engine.py.230: loss: 0.13594 [I 2022-10-27 09:49:24] ...optimizers/trijoint.py.151: total_norm: 62.678448 [I 2022-10-27 09:49:30] ...trap/engines/engine.py.224: train: epoch 60 | batch 2100/2304 [I 2022-10-27 09:49:30] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:53 [I 2022-10-27 09:49:30] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00252 [I 2022-10-27 09:49:30] ...trap/engines/engine.py.230: loss: 0.17534 [I 2022-10-27 09:49:30] ...optimizers/trijoint.py.151: total_norm: 17.476835 [I 2022-10-27 09:49:36] ...trap/engines/engine.py.224: train: epoch 60 | batch 2110/2304 [I 2022-10-27 09:49:36] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:48 [I 2022-10-27 09:49:36] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00248 [I 2022-10-27 09:49:36] ...trap/engines/engine.py.230: loss: 0.12607 [I 2022-10-27 09:49:36] ...optimizers/trijoint.py.151: total_norm: 19.326557 [I 2022-10-27 09:49:41] ...trap/engines/engine.py.224: train: epoch 60 | batch 2120/2304 [I 2022-10-27 09:49:41] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:42 [I 2022-10-27 09:49:41] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00246 [I 2022-10-27 09:49:41] ...trap/engines/engine.py.230: loss: 0.20373 [I 2022-10-27 09:49:41] ...optimizers/trijoint.py.151: total_norm: 32.071537 [I 2022-10-27 09:49:47] ...trap/engines/engine.py.224: train: epoch 60 | batch 2130/2304 [I 2022-10-27 09:49:47] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:37 [I 2022-10-27 09:49:47] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00245 [I 2022-10-27 09:49:47] ...trap/engines/engine.py.230: loss: 0.18276 [I 2022-10-27 09:49:47] ...optimizers/trijoint.py.151: total_norm: 21.654837 [I 2022-10-27 09:49:52] ...trap/engines/engine.py.224: train: epoch 60 | batch 2140/2304 [I 2022-10-27 09:49:52] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:31 [I 2022-10-27 09:49:52] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00249 [I 2022-10-27 09:49:52] ...trap/engines/engine.py.230: loss: 0.14511 [I 2022-10-27 09:49:52] ...optimizers/trijoint.py.151: total_norm: 12.559175 [I 2022-10-27 09:49:58] ...trap/engines/engine.py.224: train: epoch 60 | batch 2150/2304 [I 2022-10-27 09:49:58] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:25 [I 2022-10-27 09:49:58] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00245 [I 2022-10-27 09:49:58] ...trap/engines/engine.py.230: loss: 0.15289 [I 2022-10-27 09:49:58] ...optimizers/trijoint.py.151: total_norm: 24.777021 [I 2022-10-27 09:50:04] ...trap/engines/engine.py.224: train: epoch 60 | batch 2160/2304 [I 2022-10-27 09:50:04] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:20 [I 2022-10-27 09:50:04] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00244 [I 2022-10-27 09:50:04] ...trap/engines/engine.py.230: loss: 0.18982 [I 2022-10-27 09:50:04] ...optimizers/trijoint.py.151: total_norm: 25.837221 [I 2022-10-27 09:50:09] ...trap/engines/engine.py.224: train: epoch 60 | batch 2170/2304 [I 2022-10-27 09:50:09] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:14 [I 2022-10-27 09:50:09] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00247 [I 2022-10-27 09:50:09] ...trap/engines/engine.py.230: loss: 0.15947 [I 2022-10-27 09:50:09] ...optimizers/trijoint.py.151: total_norm: 24.709333 [I 2022-10-27 09:50:15] ...trap/engines/engine.py.224: train: epoch 60 | batch 2180/2304 [I 2022-10-27 09:50:15] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:09 [I 2022-10-27 09:50:15] ...trap/engines/engine.py.229: process: 0.57349 | load: 0.00247 [I 2022-10-27 09:50:15] ...trap/engines/engine.py.230: loss: 0.16592 [I 2022-10-27 09:50:15] ...optimizers/trijoint.py.151: total_norm: 20.484644 [I 2022-10-27 09:50:20] ...trap/engines/engine.py.224: train: epoch 60 | batch 2190/2304 [I 2022-10-27 09:50:20] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:03 [I 2022-10-27 09:50:20] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00248 [I 2022-10-27 09:50:20] ...trap/engines/engine.py.230: loss: 0.10969 [I 2022-10-27 09:50:20] ...optimizers/trijoint.py.151: total_norm: 9.477120 [I 2022-10-27 09:50:26] ...trap/engines/engine.py.224: train: epoch 60 | batch 2200/2304 [I 2022-10-27 09:50:26] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:58 [I 2022-10-27 09:50:26] ...trap/engines/engine.py.229: process: 0.54696 | load: 0.00252 [I 2022-10-27 09:50:26] ...trap/engines/engine.py.230: loss: 0.16481 [I 2022-10-27 09:50:26] ...optimizers/trijoint.py.151: total_norm: 13.170387 [I 2022-10-27 09:50:32] ...trap/engines/engine.py.224: train: epoch 60 | batch 2210/2304 [I 2022-10-27 09:50:32] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:52 [I 2022-10-27 09:50:32] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00254 [I 2022-10-27 09:50:32] ...trap/engines/engine.py.230: loss: 0.14864 [I 2022-10-27 09:50:32] ...optimizers/trijoint.py.151: total_norm: 15.341332 [I 2022-10-27 09:50:37] ...trap/engines/engine.py.224: train: epoch 60 | batch 2220/2304 [I 2022-10-27 09:50:37] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:46 [I 2022-10-27 09:50:37] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00246 [I 2022-10-27 09:50:37] ...trap/engines/engine.py.230: loss: 0.19556 [I 2022-10-27 09:50:37] ...optimizers/trijoint.py.151: total_norm: 12.258398 [I 2022-10-27 09:50:43] ...trap/engines/engine.py.224: train: epoch 60 | batch 2230/2304 [I 2022-10-27 09:50:43] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:41 [I 2022-10-27 09:50:43] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00252 [I 2022-10-27 09:50:43] ...trap/engines/engine.py.230: loss: 0.13242 [I 2022-10-27 09:50:43] ...optimizers/trijoint.py.151: total_norm: 12.695422 [I 2022-10-27 09:50:48] ...trap/engines/engine.py.224: train: epoch 60 | batch 2240/2304 [I 2022-10-27 09:50:48] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:35 [I 2022-10-27 09:50:48] ...trap/engines/engine.py.229: process: 0.55384 | load: 0.00253 [I 2022-10-27 09:50:48] ...trap/engines/engine.py.230: loss: 0.22678 [I 2022-10-27 09:50:48] ...optimizers/trijoint.py.151: total_norm: 22.826014 [I 2022-10-27 09:50:54] ...trap/engines/engine.py.224: train: epoch 60 | batch 2250/2304 [I 2022-10-27 09:50:54] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:30 [I 2022-10-27 09:50:54] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00261 [I 2022-10-27 09:50:54] ...trap/engines/engine.py.230: loss: 0.17636 [I 2022-10-27 09:50:54] ...optimizers/trijoint.py.151: total_norm: 21.361263 [I 2022-10-27 09:51:00] ...trap/engines/engine.py.224: train: epoch 60 | batch 2260/2304 [I 2022-10-27 09:51:00] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:24 [I 2022-10-27 09:51:00] ...trap/engines/engine.py.229: process: 0.56775 | load: 0.00258 [I 2022-10-27 09:51:00] ...trap/engines/engine.py.230: loss: 0.17701 [I 2022-10-27 09:51:00] ...optimizers/trijoint.py.151: total_norm: 21.321758 [I 2022-10-27 09:51:05] ...trap/engines/engine.py.224: train: epoch 60 | batch 2270/2304 [I 2022-10-27 09:51:05] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:19 [I 2022-10-27 09:51:05] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00253 [I 2022-10-27 09:51:05] ...trap/engines/engine.py.230: loss: 0.17338 [I 2022-10-27 09:51:05] ...optimizers/trijoint.py.151: total_norm: 18.507120 [I 2022-10-27 09:51:11] ...trap/engines/engine.py.224: train: epoch 60 | batch 2280/2304 [I 2022-10-27 09:51:11] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:13 [I 2022-10-27 09:51:11] ...trap/engines/engine.py.229: process: 0.55950 | load: 0.00250 [I 2022-10-27 09:51:11] ...trap/engines/engine.py.230: loss: 0.16697 [I 2022-10-27 09:51:11] ...optimizers/trijoint.py.151: total_norm: 22.643892 [I 2022-10-27 09:51:17] ...trap/engines/engine.py.224: train: epoch 60 | batch 2290/2304 [I 2022-10-27 09:51:17] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:07 [I 2022-10-27 09:51:17] ...trap/engines/engine.py.229: process: 0.56740 | load: 0.00250 [I 2022-10-27 09:51:17] ...trap/engines/engine.py.230: loss: 0.15953 [I 2022-10-27 09:51:17] ...optimizers/trijoint.py.151: total_norm: 23.399754 [I 2022-10-27 09:51:22] ...trap/engines/engine.py.224: train: epoch 60 | batch 2300/2304 [I 2022-10-27 09:51:22] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:00:02 [I 2022-10-27 09:51:22] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00015 [I 2022-10-27 09:51:22] ...trap/engines/engine.py.230: loss: 0.17942 [I 2022-10-27 09:51:22] ...optimizers/trijoint.py.151: total_norm: 19.893400 [I 2022-10-27 09:51:24] ...trap/engines/engine.py.224: train: epoch 60 | batch 2304/2304 [I 2022-10-27 09:51:24] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:00 [I 2022-10-27 09:51:24] ...trap/engines/engine.py.229: process: 0.54858 | load: 0.00017 [I 2022-10-27 09:51:24] ...trap/engines/engine.py.230: loss: 0.15136 [I 2022-10-27 09:51:24] ...optimizers/trijoint.py.151: total_norm: 8.559058 [S 2022-10-27 09:51:26] ...trap/engines/engine.py.129: train_epoch.epoch: 60 [S 2022-10-27 09:51:26] ...trap/engines/engine.py.129: train_epoch.loss: 0.15441027773370453 [I 2022-10-27 09:51:26] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 09:51:26] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 60 [I 2022-10-27 09:51:26] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 09:51:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 09:51:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 09:51:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 09:51:30] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 09:51:31] ...trap/engines/engine.py.321: eval: epoch 60 | batch 0/510 [I 2022-10-27 09:51:31] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:36:54 [I 2022-10-27 09:51:31] ...trap/engines/engine.py.326: process: 4.34208 | load: 2.90247 [I 2022-10-27 09:51:34] ...trap/engines/engine.py.321: eval: epoch 60 | batch 10/510 [I 2022-10-27 09:51:34] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:23 [I 2022-10-27 09:51:34] ...trap/engines/engine.py.326: process: 0.32641 | load: 0.00016 [I 2022-10-27 09:51:37] ...trap/engines/engine.py.321: eval: epoch 60 | batch 20/510 [I 2022-10-27 09:51:37] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:04 [I 2022-10-27 09:51:37] ...trap/engines/engine.py.326: process: 0.32506 | load: 0.00018 [I 2022-10-27 09:51:41] ...trap/engines/engine.py.321: eval: epoch 60 | batch 30/510 [I 2022-10-27 09:51:41] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:42 [I 2022-10-27 09:51:41] ...trap/engines/engine.py.326: process: 0.34387 | load: 0.00018 [I 2022-10-27 09:51:44] ...trap/engines/engine.py.321: eval: epoch 60 | batch 40/510 [I 2022-10-27 09:51:44] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:36 [I 2022-10-27 09:51:44] ...trap/engines/engine.py.326: process: 0.35460 | load: 0.00016 [I 2022-10-27 09:51:47] ...trap/engines/engine.py.321: eval: epoch 60 | batch 50/510 [I 2022-10-27 09:51:47] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:32 [I 2022-10-27 09:51:47] ...trap/engines/engine.py.326: process: 0.33204 | load: 0.00018 [I 2022-10-27 09:51:51] ...trap/engines/engine.py.321: eval: epoch 60 | batch 60/510 [I 2022-10-27 09:51:51] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:28 [I 2022-10-27 09:51:51] ...trap/engines/engine.py.326: process: 0.32827 | load: 0.00018 [I 2022-10-27 09:51:54] ...trap/engines/engine.py.321: eval: epoch 60 | batch 70/510 [I 2022-10-27 09:51:54] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:28 [I 2022-10-27 09:51:54] ...trap/engines/engine.py.326: process: 0.36925 | load: 0.00018 [I 2022-10-27 09:51:57] ...trap/engines/engine.py.321: eval: epoch 60 | batch 80/510 [I 2022-10-27 09:51:57] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-27 09:51:57] ...trap/engines/engine.py.326: process: 0.33256 | load: 0.00019 [I 2022-10-27 09:52:01] ...trap/engines/engine.py.321: eval: epoch 60 | batch 90/510 [I 2022-10-27 09:52:01] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:21 [I 2022-10-27 09:52:01] ...trap/engines/engine.py.326: process: 0.34412 | load: 0.00020 [I 2022-10-27 09:52:04] ...trap/engines/engine.py.321: eval: epoch 60 | batch 100/510 [I 2022-10-27 09:52:04] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:15 [I 2022-10-27 09:52:04] ...trap/engines/engine.py.326: process: 0.30256 | load: 0.00020 [I 2022-10-27 09:52:07] ...trap/engines/engine.py.321: eval: epoch 60 | batch 110/510 [I 2022-10-27 09:52:07] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:58 [I 2022-10-27 09:52:07] ...trap/engines/engine.py.326: process: 0.28894 | load: 0.00023 [I 2022-10-27 09:52:10] ...trap/engines/engine.py.321: eval: epoch 60 | batch 120/510 [I 2022-10-27 09:52:10] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:54 [I 2022-10-27 09:52:10] ...trap/engines/engine.py.326: process: 0.30970 | load: 0.00024 [I 2022-10-27 09:52:13] ...trap/engines/engine.py.321: eval: epoch 60 | batch 130/510 [I 2022-10-27 09:52:13] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:49 [I 2022-10-27 09:52:13] ...trap/engines/engine.py.326: process: 0.28942 | load: 0.00015 [I 2022-10-27 09:52:16] ...trap/engines/engine.py.321: eval: epoch 60 | batch 140/510 [I 2022-10-27 09:52:16] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:46 [I 2022-10-27 09:52:16] ...trap/engines/engine.py.326: process: 0.28363 | load: 0.00030 [I 2022-10-27 09:52:19] ...trap/engines/engine.py.321: eval: epoch 60 | batch 150/510 [I 2022-10-27 09:52:19] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:43 [I 2022-10-27 09:52:19] ...trap/engines/engine.py.326: process: 0.28038 | load: 0.00020 [I 2022-10-27 09:52:22] ...trap/engines/engine.py.321: eval: epoch 60 | batch 160/510 [I 2022-10-27 09:52:22] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 09:52:22] ...trap/engines/engine.py.326: process: 0.28997 | load: 0.00023 [I 2022-10-27 09:52:24] ...trap/engines/engine.py.321: eval: epoch 60 | batch 170/510 [I 2022-10-27 09:52:24] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-27 09:52:24] ...trap/engines/engine.py.326: process: 0.28815 | load: 0.00023 [I 2022-10-27 09:52:27] ...trap/engines/engine.py.321: eval: epoch 60 | batch 180/510 [I 2022-10-27 09:52:27] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 09:52:27] ...trap/engines/engine.py.326: process: 0.29008 | load: 0.00016 [I 2022-10-27 09:52:30] ...trap/engines/engine.py.321: eval: epoch 60 | batch 190/510 [I 2022-10-27 09:52:30] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:31 [I 2022-10-27 09:52:30] ...trap/engines/engine.py.326: process: 0.29070 | load: 0.00015 [I 2022-10-27 09:52:33] ...trap/engines/engine.py.321: eval: epoch 60 | batch 200/510 [I 2022-10-27 09:52:33] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 09:52:33] ...trap/engines/engine.py.326: process: 0.28749 | load: 0.00020 [I 2022-10-27 09:52:36] ...trap/engines/engine.py.321: eval: epoch 60 | batch 210/510 [I 2022-10-27 09:52:36] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 09:52:36] ...trap/engines/engine.py.326: process: 0.29110 | load: 0.00014 [I 2022-10-27 09:52:39] ...trap/engines/engine.py.321: eval: epoch 60 | batch 220/510 [I 2022-10-27 09:52:39] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 09:52:39] ...trap/engines/engine.py.326: process: 0.29117 | load: 0.00017 [I 2022-10-27 09:52:42] ...trap/engines/engine.py.321: eval: epoch 60 | batch 230/510 [I 2022-10-27 09:52:42] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:21 [I 2022-10-27 09:52:42] ...trap/engines/engine.py.326: process: 0.28986 | load: 0.00026 [I 2022-10-27 09:52:45] ...trap/engines/engine.py.321: eval: epoch 60 | batch 240/510 [I 2022-10-27 09:52:45] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 09:52:45] ...trap/engines/engine.py.326: process: 0.28691 | load: 0.00027 [I 2022-10-27 09:52:47] ...trap/engines/engine.py.321: eval: epoch 60 | batch 250/510 [I 2022-10-27 09:52:47] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:14 [I 2022-10-27 09:52:47] ...trap/engines/engine.py.326: process: 0.28384 | load: 0.00025 [I 2022-10-27 09:52:50] ...trap/engines/engine.py.321: eval: epoch 60 | batch 260/510 [I 2022-10-27 09:52:50] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-27 09:52:50] ...trap/engines/engine.py.326: process: 0.28878 | load: 0.00026 [I 2022-10-27 09:52:53] ...trap/engines/engine.py.321: eval: epoch 60 | batch 270/510 [I 2022-10-27 09:52:53] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:08 [I 2022-10-27 09:52:53] ...trap/engines/engine.py.326: process: 0.28025 | load: 0.00016 [I 2022-10-27 09:52:56] ...trap/engines/engine.py.321: eval: epoch 60 | batch 280/510 [I 2022-10-27 09:52:56] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:05 [I 2022-10-27 09:52:56] ...trap/engines/engine.py.326: process: 0.28717 | load: 0.00019 [I 2022-10-27 09:52:59] ...trap/engines/engine.py.321: eval: epoch 60 | batch 290/510 [I 2022-10-27 09:52:59] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 09:52:59] ...trap/engines/engine.py.326: process: 0.28234 | load: 0.00019 [I 2022-10-27 09:53:02] ...trap/engines/engine.py.321: eval: epoch 60 | batch 300/510 [I 2022-10-27 09:53:02] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 09:53:02] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00020 [I 2022-10-27 09:53:05] ...trap/engines/engine.py.321: eval: epoch 60 | batch 310/510 [I 2022-10-27 09:53:05] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 09:53:05] ...trap/engines/engine.py.326: process: 0.28745 | load: 0.00017 [I 2022-10-27 09:53:08] ...trap/engines/engine.py.321: eval: epoch 60 | batch 320/510 [I 2022-10-27 09:53:08] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 09:53:08] ...trap/engines/engine.py.326: process: 0.28725 | load: 0.00024 [I 2022-10-27 09:53:11] ...trap/engines/engine.py.321: eval: epoch 60 | batch 330/510 [I 2022-10-27 09:53:11] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 09:53:11] ...trap/engines/engine.py.326: process: 0.28843 | load: 0.00016 [I 2022-10-27 09:53:13] ...trap/engines/engine.py.321: eval: epoch 60 | batch 340/510 [I 2022-10-27 09:53:13] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:49 [I 2022-10-27 09:53:13] ...trap/engines/engine.py.326: process: 0.28935 | load: 0.00016 [I 2022-10-27 09:53:16] ...trap/engines/engine.py.321: eval: epoch 60 | batch 350/510 [I 2022-10-27 09:53:16] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:46 [I 2022-10-27 09:53:16] ...trap/engines/engine.py.326: process: 0.29244 | load: 0.00015 [I 2022-10-27 09:53:19] ...trap/engines/engine.py.321: eval: epoch 60 | batch 360/510 [I 2022-10-27 09:53:19] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:42 [I 2022-10-27 09:53:19] ...trap/engines/engine.py.326: process: 0.28075 | load: 0.00016 [I 2022-10-27 09:53:22] ...trap/engines/engine.py.321: eval: epoch 60 | batch 370/510 [I 2022-10-27 09:53:22] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 09:53:22] ...trap/engines/engine.py.326: process: 0.28676 | load: 0.00014 [I 2022-10-27 09:53:25] ...trap/engines/engine.py.321: eval: epoch 60 | batch 380/510 [I 2022-10-27 09:53:25] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 09:53:25] ...trap/engines/engine.py.326: process: 0.28931 | load: 0.00016 [I 2022-10-27 09:53:28] ...trap/engines/engine.py.321: eval: epoch 60 | batch 390/510 [I 2022-10-27 09:53:28] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 09:53:28] ...trap/engines/engine.py.326: process: 0.29697 | load: 0.00030 [I 2022-10-27 09:53:31] ...trap/engines/engine.py.321: eval: epoch 60 | batch 400/510 [I 2022-10-27 09:53:31] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 09:53:31] ...trap/engines/engine.py.326: process: 0.29045 | load: 0.00025 [I 2022-10-27 09:53:34] ...trap/engines/engine.py.321: eval: epoch 60 | batch 410/510 [I 2022-10-27 09:53:34] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 09:53:34] ...trap/engines/engine.py.326: process: 0.28856 | load: 0.00015 [I 2022-10-27 09:53:37] ...trap/engines/engine.py.321: eval: epoch 60 | batch 420/510 [I 2022-10-27 09:53:37] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 09:53:37] ...trap/engines/engine.py.326: process: 0.28661 | load: 0.00015 [I 2022-10-27 09:53:39] ...trap/engines/engine.py.321: eval: epoch 60 | batch 430/510 [I 2022-10-27 09:53:39] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:22 [I 2022-10-27 09:53:39] ...trap/engines/engine.py.326: process: 0.29019 | load: 0.00015 [I 2022-10-27 09:53:42] ...trap/engines/engine.py.321: eval: epoch 60 | batch 440/510 [I 2022-10-27 09:53:42] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 09:53:42] ...trap/engines/engine.py.326: process: 0.28710 | load: 0.00015 [I 2022-10-27 09:53:45] ...trap/engines/engine.py.321: eval: epoch 60 | batch 450/510 [I 2022-10-27 09:53:45] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 09:53:45] ...trap/engines/engine.py.326: process: 0.27998 | load: 0.00016 [I 2022-10-27 09:53:48] ...trap/engines/engine.py.321: eval: epoch 60 | batch 460/510 [I 2022-10-27 09:53:48] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 09:53:48] ...trap/engines/engine.py.326: process: 0.28899 | load: 0.00019 [I 2022-10-27 09:53:51] ...trap/engines/engine.py.321: eval: epoch 60 | batch 470/510 [I 2022-10-27 09:53:51] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 09:53:51] ...trap/engines/engine.py.326: process: 0.28856 | load: 0.00016 [I 2022-10-27 09:53:54] ...trap/engines/engine.py.321: eval: epoch 60 | batch 480/510 [I 2022-10-27 09:53:54] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 09:53:54] ...trap/engines/engine.py.326: process: 0.29018 | load: 0.00016 [I 2022-10-27 09:53:57] ...trap/engines/engine.py.321: eval: epoch 60 | batch 490/510 [I 2022-10-27 09:53:57] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 09:53:57] ...trap/engines/engine.py.326: process: 0.28862 | load: 0.00015 [I 2022-10-27 09:53:59] ...trap/engines/engine.py.321: eval: epoch 60 | batch 500/510 [I 2022-10-27 09:53:59] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 09:53:59] ...trap/engines/engine.py.326: process: 0.28968 | load: 0.00021 [I 2022-10-27 09:54:02] ...trap/engines/engine.py.321: eval: epoch 60 | batch 510/510 [I 2022-10-27 09:54:02] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 09:54:02] ...trap/engines/engine.py.326: process: 0.29131 | load: 0.00013 [S 2022-10-27 09:54:03] ...trap/engines/engine.py.132: eval_epoch.epoch: 60 [I 2022-10-27 09:54:03] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.704600 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.899800 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935500 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.702800 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.900800 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933300 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010716 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008483 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008041 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012952 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009075 [S 2022-10-27 09:54:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009274 [I 2022-10-27 09:54:08] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 09:54:08] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 09:54:09] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 09:54:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 09:54:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 09:54:12] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 09:54:13] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 09:54:19] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 09:54:19] ...trap/engines/engine.py.163: Training model on trainset for epoch 61 [I 2022-10-27 09:54:19] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 09:54:22] ...trap/engines/engine.py.224: train: epoch 61 | batch 0/2304 [I 2022-10-27 09:54:22] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:04 [I 2022-10-27 09:54:22] ...trap/engines/engine.py.229: process: 2.71016 | load: 2.04998 [I 2022-10-27 09:54:22] ...trap/engines/engine.py.230: loss: 0.15300 [I 2022-10-27 09:54:22] ...optimizers/trijoint.py.151: total_norm: 30.035551 [I 2022-10-27 09:54:27] ...trap/engines/engine.py.224: train: epoch 61 | batch 10/2304 [I 2022-10-27 09:54:27] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:08 [I 2022-10-27 09:54:27] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00469 [I 2022-10-27 09:54:27] ...trap/engines/engine.py.230: loss: 0.15946 [I 2022-10-27 09:54:27] ...optimizers/trijoint.py.151: total_norm: 19.591139 [I 2022-10-27 09:54:33] ...trap/engines/engine.py.224: train: epoch 61 | batch 20/2304 [I 2022-10-27 09:54:33] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:08 [I 2022-10-27 09:54:33] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00285 [I 2022-10-27 09:54:33] ...trap/engines/engine.py.230: loss: 0.13628 [I 2022-10-27 09:54:33] ...optimizers/trijoint.py.151: total_norm: 62.913692 [I 2022-10-27 09:54:39] ...trap/engines/engine.py.224: train: epoch 61 | batch 30/2304 [I 2022-10-27 09:54:39] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-27 09:54:39] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00349 [I 2022-10-27 09:54:39] ...trap/engines/engine.py.230: loss: 0.09867 [I 2022-10-27 09:54:39] ...optimizers/trijoint.py.151: total_norm: 11.891935 [I 2022-10-27 09:54:44] ...trap/engines/engine.py.224: train: epoch 61 | batch 40/2304 [I 2022-10-27 09:54:44] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:57 [I 2022-10-27 09:54:44] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00331 [I 2022-10-27 09:54:44] ...trap/engines/engine.py.230: loss: 0.12184 [I 2022-10-27 09:54:44] ...optimizers/trijoint.py.151: total_norm: 10.055260 [I 2022-10-27 09:54:50] ...trap/engines/engine.py.224: train: epoch 61 | batch 50/2304 [I 2022-10-27 09:54:50] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:53 [I 2022-10-27 09:54:50] ...trap/engines/engine.py.229: process: 0.56049 | load: 0.00277 [I 2022-10-27 09:54:50] ...trap/engines/engine.py.230: loss: 0.16113 [I 2022-10-27 09:54:50] ...optimizers/trijoint.py.151: total_norm: 24.930666 [I 2022-10-27 09:54:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 60/2304 [I 2022-10-27 09:54:55] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:55 [I 2022-10-27 09:54:55] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00280 [I 2022-10-27 09:54:55] ...trap/engines/engine.py.230: loss: 0.11356 [I 2022-10-27 09:54:55] ...optimizers/trijoint.py.151: total_norm: 15.436494 [I 2022-10-27 09:55:01] ...trap/engines/engine.py.224: train: epoch 61 | batch 70/2304 [I 2022-10-27 09:55:01] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:35 [I 2022-10-27 09:55:01] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00305 [I 2022-10-27 09:55:01] ...trap/engines/engine.py.230: loss: 0.11862 [I 2022-10-27 09:55:01] ...optimizers/trijoint.py.151: total_norm: 9.928617 [I 2022-10-27 09:55:06] ...trap/engines/engine.py.224: train: epoch 61 | batch 80/2304 [I 2022-10-27 09:55:06] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:40 [I 2022-10-27 09:55:06] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00269 [I 2022-10-27 09:55:06] ...trap/engines/engine.py.230: loss: 0.23479 [I 2022-10-27 09:55:06] ...optimizers/trijoint.py.151: total_norm: 16.576122 [I 2022-10-27 09:55:12] ...trap/engines/engine.py.224: train: epoch 61 | batch 90/2304 [I 2022-10-27 09:55:12] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:39 [I 2022-10-27 09:55:12] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00274 [I 2022-10-27 09:55:12] ...trap/engines/engine.py.230: loss: 0.24479 [I 2022-10-27 09:55:12] ...optimizers/trijoint.py.151: total_norm: 15.857001 [I 2022-10-27 09:55:18] ...trap/engines/engine.py.224: train: epoch 61 | batch 100/2304 [I 2022-10-27 09:55:18] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:16 [I 2022-10-27 09:55:18] ...trap/engines/engine.py.229: process: 0.54674 | load: 0.00270 [I 2022-10-27 09:55:18] ...trap/engines/engine.py.230: loss: 0.16029 [I 2022-10-27 09:55:18] ...optimizers/trijoint.py.151: total_norm: 16.951138 [I 2022-10-27 09:55:23] ...trap/engines/engine.py.224: train: epoch 61 | batch 110/2304 [I 2022-10-27 09:55:23] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:22 [I 2022-10-27 09:55:23] ...trap/engines/engine.py.229: process: 0.55726 | load: 0.00260 [I 2022-10-27 09:55:23] ...trap/engines/engine.py.230: loss: 0.14261 [I 2022-10-27 09:55:23] ...optimizers/trijoint.py.151: total_norm: 12.606533 [I 2022-10-27 09:55:29] ...trap/engines/engine.py.224: train: epoch 61 | batch 120/2304 [I 2022-10-27 09:55:29] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:17 [I 2022-10-27 09:55:29] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00258 [I 2022-10-27 09:55:29] ...trap/engines/engine.py.230: loss: 0.12540 [I 2022-10-27 09:55:29] ...optimizers/trijoint.py.151: total_norm: 12.297636 [I 2022-10-27 09:55:34] ...trap/engines/engine.py.224: train: epoch 61 | batch 130/2304 [I 2022-10-27 09:55:34] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:21 [I 2022-10-27 09:55:34] ...trap/engines/engine.py.229: process: 0.56374 | load: 0.00257 [I 2022-10-27 09:55:34] ...trap/engines/engine.py.230: loss: 0.12873 [I 2022-10-27 09:55:34] ...optimizers/trijoint.py.151: total_norm: 11.673961 [I 2022-10-27 09:55:40] ...trap/engines/engine.py.224: train: epoch 61 | batch 140/2304 [I 2022-10-27 09:55:40] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:02 [I 2022-10-27 09:55:40] ...trap/engines/engine.py.229: process: 0.56223 | load: 0.00264 [I 2022-10-27 09:55:40] ...trap/engines/engine.py.230: loss: 0.20000 [I 2022-10-27 09:55:40] ...optimizers/trijoint.py.151: total_norm: 17.755358 [I 2022-10-27 09:55:45] ...trap/engines/engine.py.224: train: epoch 61 | batch 150/2304 [I 2022-10-27 09:55:45] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:51 [I 2022-10-27 09:55:45] ...trap/engines/engine.py.229: process: 0.54940 | load: 0.00256 [I 2022-10-27 09:55:45] ...trap/engines/engine.py.230: loss: 0.14667 [I 2022-10-27 09:55:45] ...optimizers/trijoint.py.151: total_norm: 44.331173 [I 2022-10-27 09:55:51] ...trap/engines/engine.py.224: train: epoch 61 | batch 160/2304 [I 2022-10-27 09:55:51] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:03 [I 2022-10-27 09:55:51] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00267 [I 2022-10-27 09:55:51] ...trap/engines/engine.py.230: loss: 0.13888 [I 2022-10-27 09:55:51] ...optimizers/trijoint.py.151: total_norm: 13.367483 [I 2022-10-27 09:55:57] ...trap/engines/engine.py.224: train: epoch 61 | batch 170/2304 [I 2022-10-27 09:55:57] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:52 [I 2022-10-27 09:55:57] ...trap/engines/engine.py.229: process: 0.55724 | load: 0.00277 [I 2022-10-27 09:55:57] ...trap/engines/engine.py.230: loss: 0.14254 [I 2022-10-27 09:55:57] ...optimizers/trijoint.py.151: total_norm: 18.211676 [I 2022-10-27 09:56:02] ...trap/engines/engine.py.224: train: epoch 61 | batch 180/2304 [I 2022-10-27 09:56:02] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:20:13 [I 2022-10-27 09:56:02] ...trap/engines/engine.py.229: process: 0.55135 | load: 0.00255 [I 2022-10-27 09:56:02] ...trap/engines/engine.py.230: loss: 0.18703 [I 2022-10-27 09:56:02] ...optimizers/trijoint.py.151: total_norm: 21.166431 [I 2022-10-27 09:56:08] ...trap/engines/engine.py.224: train: epoch 61 | batch 190/2304 [I 2022-10-27 09:56:08] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:41 [I 2022-10-27 09:56:08] ...trap/engines/engine.py.229: process: 0.54618 | load: 0.00250 [I 2022-10-27 09:56:08] ...trap/engines/engine.py.230: loss: 0.10957 [I 2022-10-27 09:56:08] ...optimizers/trijoint.py.151: total_norm: 7.287710 [I 2022-10-27 09:56:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 200/2304 [I 2022-10-27 09:56:14] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:30 [I 2022-10-27 09:56:14] ...trap/engines/engine.py.229: process: 0.55143 | load: 0.00242 [I 2022-10-27 09:56:14] ...trap/engines/engine.py.230: loss: 0.15534 [I 2022-10-27 09:56:14] ...optimizers/trijoint.py.151: total_norm: 18.420292 [I 2022-10-27 09:56:19] ...trap/engines/engine.py.224: train: epoch 61 | batch 210/2304 [I 2022-10-27 09:56:19] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:35 [I 2022-10-27 09:56:19] ...trap/engines/engine.py.229: process: 0.57183 | load: 0.00256 [I 2022-10-27 09:56:19] ...trap/engines/engine.py.230: loss: 0.11913 [I 2022-10-27 09:56:19] ...optimizers/trijoint.py.151: total_norm: 12.604733 [I 2022-10-27 09:56:25] ...trap/engines/engine.py.224: train: epoch 61 | batch 220/2304 [I 2022-10-27 09:56:25] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:25 [I 2022-10-27 09:56:25] ...trap/engines/engine.py.229: process: 0.57811 | load: 0.00241 [I 2022-10-27 09:56:25] ...trap/engines/engine.py.230: loss: 0.10968 [I 2022-10-27 09:56:25] ...optimizers/trijoint.py.151: total_norm: 11.582994 [I 2022-10-27 09:56:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 230/2304 [I 2022-10-27 09:56:30] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:21 [I 2022-10-27 09:56:30] ...trap/engines/engine.py.229: process: 0.56621 | load: 0.00253 [I 2022-10-27 09:56:30] ...trap/engines/engine.py.230: loss: 0.16686 [I 2022-10-27 09:56:30] ...optimizers/trijoint.py.151: total_norm: 23.060888 [I 2022-10-27 09:56:36] ...trap/engines/engine.py.224: train: epoch 61 | batch 240/2304 [I 2022-10-27 09:56:36] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:20 [I 2022-10-27 09:56:36] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00248 [I 2022-10-27 09:56:36] ...trap/engines/engine.py.230: loss: 0.17665 [I 2022-10-27 09:56:36] ...optimizers/trijoint.py.151: total_norm: 23.133392 [I 2022-10-27 09:56:42] ...trap/engines/engine.py.224: train: epoch 61 | batch 250/2304 [I 2022-10-27 09:56:42] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:24 [I 2022-10-27 09:56:42] ...trap/engines/engine.py.229: process: 0.55931 | load: 0.00253 [I 2022-10-27 09:56:42] ...trap/engines/engine.py.230: loss: 0.14470 [I 2022-10-27 09:56:42] ...optimizers/trijoint.py.151: total_norm: 18.476770 [I 2022-10-27 09:56:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 260/2304 [I 2022-10-27 09:56:47] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:09 [I 2022-10-27 09:56:47] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00257 [I 2022-10-27 09:56:47] ...trap/engines/engine.py.230: loss: 0.14875 [I 2022-10-27 09:56:47] ...optimizers/trijoint.py.151: total_norm: 17.388981 [I 2022-10-27 09:56:53] ...trap/engines/engine.py.224: train: epoch 61 | batch 270/2304 [I 2022-10-27 09:56:53] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:01 [I 2022-10-27 09:56:53] ...trap/engines/engine.py.229: process: 0.56861 | load: 0.00242 [I 2022-10-27 09:56:53] ...trap/engines/engine.py.230: loss: 0.14561 [I 2022-10-27 09:56:53] ...optimizers/trijoint.py.151: total_norm: 9.107183 [I 2022-10-27 09:56:59] ...trap/engines/engine.py.224: train: epoch 61 | batch 280/2304 [I 2022-10-27 09:56:59] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:55 [I 2022-10-27 09:56:59] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00251 [I 2022-10-27 09:56:59] ...trap/engines/engine.py.230: loss: 0.15277 [I 2022-10-27 09:56:59] ...optimizers/trijoint.py.151: total_norm: 10.345055 [I 2022-10-27 09:57:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 290/2304 [I 2022-10-27 09:57:04] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:43 [I 2022-10-27 09:57:04] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00244 [I 2022-10-27 09:57:04] ...trap/engines/engine.py.230: loss: 0.20008 [I 2022-10-27 09:57:04] ...optimizers/trijoint.py.151: total_norm: 52.529530 [I 2022-10-27 09:57:10] ...trap/engines/engine.py.224: train: epoch 61 | batch 300/2304 [I 2022-10-27 09:57:10] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:38 [I 2022-10-27 09:57:10] ...trap/engines/engine.py.229: process: 0.56431 | load: 0.00256 [I 2022-10-27 09:57:10] ...trap/engines/engine.py.230: loss: 0.17395 [I 2022-10-27 09:57:10] ...optimizers/trijoint.py.151: total_norm: 14.938951 [I 2022-10-27 09:57:15] ...trap/engines/engine.py.224: train: epoch 61 | batch 310/2304 [I 2022-10-27 09:57:15] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:47 [I 2022-10-27 09:57:15] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00244 [I 2022-10-27 09:57:15] ...trap/engines/engine.py.230: loss: 0.11312 [I 2022-10-27 09:57:15] ...optimizers/trijoint.py.151: total_norm: 14.161635 [I 2022-10-27 09:57:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 320/2304 [I 2022-10-27 09:57:21] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:25 [I 2022-10-27 09:57:21] ...trap/engines/engine.py.229: process: 0.55844 | load: 0.00244 [I 2022-10-27 09:57:21] ...trap/engines/engine.py.230: loss: 0.20248 [I 2022-10-27 09:57:21] ...optimizers/trijoint.py.151: total_norm: 35.449768 [I 2022-10-27 09:57:27] ...trap/engines/engine.py.224: train: epoch 61 | batch 330/2304 [I 2022-10-27 09:57:27] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:17 [I 2022-10-27 09:57:27] ...trap/engines/engine.py.229: process: 0.55986 | load: 0.00238 [I 2022-10-27 09:57:27] ...trap/engines/engine.py.230: loss: 0.14151 [I 2022-10-27 09:57:27] ...optimizers/trijoint.py.151: total_norm: 15.668967 [I 2022-10-27 09:57:32] ...trap/engines/engine.py.224: train: epoch 61 | batch 340/2304 [I 2022-10-27 09:57:32] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:10 [I 2022-10-27 09:57:32] ...trap/engines/engine.py.229: process: 0.55612 | load: 0.00250 [I 2022-10-27 09:57:32] ...trap/engines/engine.py.230: loss: 0.11354 [I 2022-10-27 09:57:32] ...optimizers/trijoint.py.151: total_norm: 17.349209 [I 2022-10-27 09:57:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 350/2304 [I 2022-10-27 09:57:38] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:00 [I 2022-10-27 09:57:38] ...trap/engines/engine.py.229: process: 0.55201 | load: 0.00241 [I 2022-10-27 09:57:38] ...trap/engines/engine.py.230: loss: 0.12369 [I 2022-10-27 09:57:38] ...optimizers/trijoint.py.151: total_norm: 16.249725 [I 2022-10-27 09:57:43] ...trap/engines/engine.py.224: train: epoch 61 | batch 360/2304 [I 2022-10-27 09:57:43] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:51 [I 2022-10-27 09:57:43] ...trap/engines/engine.py.229: process: 0.54009 | load: 0.00240 [I 2022-10-27 09:57:43] ...trap/engines/engine.py.230: loss: 0.13573 [I 2022-10-27 09:57:43] ...optimizers/trijoint.py.151: total_norm: 11.431803 [I 2022-10-27 09:57:49] ...trap/engines/engine.py.224: train: epoch 61 | batch 370/2304 [I 2022-10-27 09:57:49] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:03 [I 2022-10-27 09:57:49] ...trap/engines/engine.py.229: process: 0.57791 | load: 0.00241 [I 2022-10-27 09:57:49] ...trap/engines/engine.py.230: loss: 0.13208 [I 2022-10-27 09:57:49] ...optimizers/trijoint.py.151: total_norm: 15.501779 [I 2022-10-27 09:57:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 380/2304 [I 2022-10-27 09:57:55] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:57 [I 2022-10-27 09:57:55] ...trap/engines/engine.py.229: process: 0.54539 | load: 0.00243 [I 2022-10-27 09:57:55] ...trap/engines/engine.py.230: loss: 0.17294 [I 2022-10-27 09:57:55] ...optimizers/trijoint.py.151: total_norm: 27.879040 [I 2022-10-27 09:58:00] ...trap/engines/engine.py.224: train: epoch 61 | batch 390/2304 [I 2022-10-27 09:58:00] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:51 [I 2022-10-27 09:58:00] ...trap/engines/engine.py.229: process: 0.56799 | load: 0.00243 [I 2022-10-27 09:58:00] ...trap/engines/engine.py.230: loss: 0.15892 [I 2022-10-27 09:58:00] ...optimizers/trijoint.py.151: total_norm: 20.095070 [I 2022-10-27 09:58:06] ...trap/engines/engine.py.224: train: epoch 61 | batch 400/2304 [I 2022-10-27 09:58:06] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:32 [I 2022-10-27 09:58:06] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00242 [I 2022-10-27 09:58:06] ...trap/engines/engine.py.230: loss: 0.14353 [I 2022-10-27 09:58:06] ...optimizers/trijoint.py.151: total_norm: 8.973922 [I 2022-10-27 09:58:11] ...trap/engines/engine.py.224: train: epoch 61 | batch 410/2304 [I 2022-10-27 09:58:11] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:33 [I 2022-10-27 09:58:11] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00246 [I 2022-10-27 09:58:11] ...trap/engines/engine.py.230: loss: 0.13735 [I 2022-10-27 09:58:11] ...optimizers/trijoint.py.151: total_norm: 10.301601 [I 2022-10-27 09:58:17] ...trap/engines/engine.py.224: train: epoch 61 | batch 420/2304 [I 2022-10-27 09:58:17] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:27 [I 2022-10-27 09:58:17] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00246 [I 2022-10-27 09:58:17] ...trap/engines/engine.py.230: loss: 0.10253 [I 2022-10-27 09:58:17] ...optimizers/trijoint.py.151: total_norm: 11.705562 [I 2022-10-27 09:58:22] ...trap/engines/engine.py.224: train: epoch 61 | batch 430/2304 [I 2022-10-27 09:58:22] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:20 [I 2022-10-27 09:58:22] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00248 [I 2022-10-27 09:58:22] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-27 09:58:22] ...optimizers/trijoint.py.151: total_norm: 10.077203 [I 2022-10-27 09:58:28] ...trap/engines/engine.py.224: train: epoch 61 | batch 440/2304 [I 2022-10-27 09:58:28] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:42 [I 2022-10-27 09:58:28] ...trap/engines/engine.py.229: process: 0.54458 | load: 0.00252 [I 2022-10-27 09:58:28] ...trap/engines/engine.py.230: loss: 0.15353 [I 2022-10-27 09:58:28] ...optimizers/trijoint.py.151: total_norm: 13.259077 [I 2022-10-27 09:58:34] ...trap/engines/engine.py.224: train: epoch 61 | batch 450/2304 [I 2022-10-27 09:58:34] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:29 [I 2022-10-27 09:58:34] ...trap/engines/engine.py.229: process: 0.57729 | load: 0.00243 [I 2022-10-27 09:58:34] ...trap/engines/engine.py.230: loss: 0.11532 [I 2022-10-27 09:58:34] ...optimizers/trijoint.py.151: total_norm: 14.706365 [I 2022-10-27 09:58:39] ...trap/engines/engine.py.224: train: epoch 61 | batch 460/2304 [I 2022-10-27 09:58:39] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:20 [I 2022-10-27 09:58:39] ...trap/engines/engine.py.229: process: 0.56957 | load: 0.00243 [I 2022-10-27 09:58:39] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-27 09:58:39] ...optimizers/trijoint.py.151: total_norm: 36.878265 [I 2022-10-27 09:58:45] ...trap/engines/engine.py.224: train: epoch 61 | batch 470/2304 [I 2022-10-27 09:58:45] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:15 [I 2022-10-27 09:58:45] ...trap/engines/engine.py.229: process: 0.57121 | load: 0.00243 [I 2022-10-27 09:58:45] ...trap/engines/engine.py.230: loss: 0.13440 [I 2022-10-27 09:58:45] ...optimizers/trijoint.py.151: total_norm: 16.459717 [I 2022-10-27 09:58:51] ...trap/engines/engine.py.224: train: epoch 61 | batch 480/2304 [I 2022-10-27 09:58:51] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:07 [I 2022-10-27 09:58:51] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00241 [I 2022-10-27 09:58:51] ...trap/engines/engine.py.230: loss: 0.20711 [I 2022-10-27 09:58:51] ...optimizers/trijoint.py.151: total_norm: 18.352840 [I 2022-10-27 09:58:56] ...trap/engines/engine.py.224: train: epoch 61 | batch 490/2304 [I 2022-10-27 09:58:56] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:54 [I 2022-10-27 09:58:56] ...trap/engines/engine.py.229: process: 0.56025 | load: 0.00242 [I 2022-10-27 09:58:56] ...trap/engines/engine.py.230: loss: 0.21739 [I 2022-10-27 09:58:56] ...optimizers/trijoint.py.151: total_norm: 16.227146 [I 2022-10-27 09:59:02] ...trap/engines/engine.py.224: train: epoch 61 | batch 500/2304 [I 2022-10-27 09:59:02] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:17:03 [I 2022-10-27 09:59:02] ...trap/engines/engine.py.229: process: 0.54595 | load: 0.00245 [I 2022-10-27 09:59:02] ...trap/engines/engine.py.230: loss: 0.18331 [I 2022-10-27 09:59:02] ...optimizers/trijoint.py.151: total_norm: 17.871511 [I 2022-10-27 09:59:08] ...trap/engines/engine.py.224: train: epoch 61 | batch 510/2304 [I 2022-10-27 09:59:08] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:40 [I 2022-10-27 09:59:08] ...trap/engines/engine.py.229: process: 0.54356 | load: 0.00243 [I 2022-10-27 09:59:08] ...trap/engines/engine.py.230: loss: 0.14894 [I 2022-10-27 09:59:08] ...optimizers/trijoint.py.151: total_norm: 17.634539 [I 2022-10-27 09:59:13] ...trap/engines/engine.py.224: train: epoch 61 | batch 520/2304 [I 2022-10-27 09:59:13] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:42 [I 2022-10-27 09:59:13] ...trap/engines/engine.py.229: process: 0.56451 | load: 0.00240 [I 2022-10-27 09:59:13] ...trap/engines/engine.py.230: loss: 0.16676 [I 2022-10-27 09:59:13] ...optimizers/trijoint.py.151: total_norm: 30.656166 [I 2022-10-27 09:59:19] ...trap/engines/engine.py.224: train: epoch 61 | batch 530/2304 [I 2022-10-27 09:59:19] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:27 [I 2022-10-27 09:59:19] ...trap/engines/engine.py.229: process: 0.55167 | load: 0.00252 [I 2022-10-27 09:59:19] ...trap/engines/engine.py.230: loss: 0.15397 [I 2022-10-27 09:59:19] ...optimizers/trijoint.py.151: total_norm: 16.064487 [I 2022-10-27 09:59:24] ...trap/engines/engine.py.224: train: epoch 61 | batch 540/2304 [I 2022-10-27 09:59:24] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:25 [I 2022-10-27 09:59:24] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00248 [I 2022-10-27 09:59:24] ...trap/engines/engine.py.230: loss: 0.19996 [I 2022-10-27 09:59:24] ...optimizers/trijoint.py.151: total_norm: 21.405273 [I 2022-10-27 09:59:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 550/2304 [I 2022-10-27 09:59:30] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:16 [I 2022-10-27 09:59:30] ...trap/engines/engine.py.229: process: 0.56991 | load: 0.00248 [I 2022-10-27 09:59:30] ...trap/engines/engine.py.230: loss: 0.14135 [I 2022-10-27 09:59:30] ...optimizers/trijoint.py.151: total_norm: 14.513520 [I 2022-10-27 09:59:36] ...trap/engines/engine.py.224: train: epoch 61 | batch 560/2304 [I 2022-10-27 09:59:36] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:53 [I 2022-10-27 09:59:36] ...trap/engines/engine.py.229: process: 0.55950 | load: 0.00266 [I 2022-10-27 09:59:36] ...trap/engines/engine.py.230: loss: 0.21079 [I 2022-10-27 09:59:36] ...optimizers/trijoint.py.151: total_norm: 15.523570 [I 2022-10-27 09:59:41] ...trap/engines/engine.py.224: train: epoch 61 | batch 570/2304 [I 2022-10-27 09:59:41] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:08 [I 2022-10-27 09:59:41] ...trap/engines/engine.py.229: process: 0.54455 | load: 0.00248 [I 2022-10-27 09:59:41] ...trap/engines/engine.py.230: loss: 0.10120 [I 2022-10-27 09:59:41] ...optimizers/trijoint.py.151: total_norm: 9.133552 [I 2022-10-27 09:59:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 580/2304 [I 2022-10-27 09:59:47] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:06 [I 2022-10-27 09:59:47] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00246 [I 2022-10-27 09:59:47] ...trap/engines/engine.py.230: loss: 0.12496 [I 2022-10-27 09:59:47] ...optimizers/trijoint.py.151: total_norm: 12.195810 [I 2022-10-27 09:59:52] ...trap/engines/engine.py.224: train: epoch 61 | batch 590/2304 [I 2022-10-27 09:59:52] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:50 [I 2022-10-27 09:59:52] ...trap/engines/engine.py.229: process: 0.55505 | load: 0.00249 [I 2022-10-27 09:59:52] ...trap/engines/engine.py.230: loss: 0.10280 [I 2022-10-27 09:59:52] ...optimizers/trijoint.py.151: total_norm: 9.090474 [I 2022-10-27 09:59:58] ...trap/engines/engine.py.224: train: epoch 61 | batch 600/2304 [I 2022-10-27 09:59:58] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:56 [I 2022-10-27 09:59:58] ...trap/engines/engine.py.229: process: 0.56646 | load: 0.00249 [I 2022-10-27 09:59:58] ...trap/engines/engine.py.230: loss: 0.15678 [I 2022-10-27 09:59:58] ...optimizers/trijoint.py.151: total_norm: 13.130127 [I 2022-10-27 10:00:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 610/2304 [I 2022-10-27 10:00:04] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:48 [I 2022-10-27 10:00:04] ...trap/engines/engine.py.229: process: 0.55292 | load: 0.00270 [I 2022-10-27 10:00:04] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-27 10:00:04] ...optimizers/trijoint.py.151: total_norm: 10.171844 [I 2022-10-27 10:00:09] ...trap/engines/engine.py.224: train: epoch 61 | batch 620/2304 [I 2022-10-27 10:00:09] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:43 [I 2022-10-27 10:00:09] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00258 [I 2022-10-27 10:00:09] ...trap/engines/engine.py.230: loss: 0.17578 [I 2022-10-27 10:00:09] ...optimizers/trijoint.py.151: total_norm: 12.554525 [I 2022-10-27 10:00:15] ...trap/engines/engine.py.224: train: epoch 61 | batch 630/2304 [I 2022-10-27 10:00:15] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:57 [I 2022-10-27 10:00:15] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00254 [I 2022-10-27 10:00:15] ...trap/engines/engine.py.230: loss: 0.13882 [I 2022-10-27 10:00:15] ...optimizers/trijoint.py.151: total_norm: 20.369616 [I 2022-10-27 10:00:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 640/2304 [I 2022-10-27 10:00:21] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:27 [I 2022-10-27 10:00:21] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00274 [I 2022-10-27 10:00:21] ...trap/engines/engine.py.230: loss: 0.17228 [I 2022-10-27 10:00:21] ...optimizers/trijoint.py.151: total_norm: 23.555025 [I 2022-10-27 10:00:26] ...trap/engines/engine.py.224: train: epoch 61 | batch 650/2304 [I 2022-10-27 10:00:26] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:20 [I 2022-10-27 10:00:26] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00250 [I 2022-10-27 10:00:26] ...trap/engines/engine.py.230: loss: 0.16051 [I 2022-10-27 10:00:26] ...optimizers/trijoint.py.151: total_norm: 18.240568 [I 2022-10-27 10:00:32] ...trap/engines/engine.py.224: train: epoch 61 | batch 660/2304 [I 2022-10-27 10:00:32] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:22 [I 2022-10-27 10:00:32] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00252 [I 2022-10-27 10:00:32] ...trap/engines/engine.py.230: loss: 0.16209 [I 2022-10-27 10:00:32] ...optimizers/trijoint.py.151: total_norm: 49.446724 [I 2022-10-27 10:00:37] ...trap/engines/engine.py.224: train: epoch 61 | batch 670/2304 [I 2022-10-27 10:00:37] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:13 [I 2022-10-27 10:00:37] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00243 [I 2022-10-27 10:00:37] ...trap/engines/engine.py.230: loss: 0.14659 [I 2022-10-27 10:00:37] ...optimizers/trijoint.py.151: total_norm: 17.207321 [I 2022-10-27 10:00:43] ...trap/engines/engine.py.224: train: epoch 61 | batch 680/2304 [I 2022-10-27 10:00:43] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:02 [I 2022-10-27 10:00:43] ...trap/engines/engine.py.229: process: 0.54778 | load: 0.00251 [I 2022-10-27 10:00:43] ...trap/engines/engine.py.230: loss: 0.15951 [I 2022-10-27 10:00:43] ...optimizers/trijoint.py.151: total_norm: 15.456781 [I 2022-10-27 10:00:49] ...trap/engines/engine.py.224: train: epoch 61 | batch 690/2304 [I 2022-10-27 10:00:49] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:03 [I 2022-10-27 10:00:49] ...trap/engines/engine.py.229: process: 0.56480 | load: 0.00245 [I 2022-10-27 10:00:49] ...trap/engines/engine.py.230: loss: 0.18844 [I 2022-10-27 10:00:49] ...optimizers/trijoint.py.151: total_norm: 14.014410 [I 2022-10-27 10:00:54] ...trap/engines/engine.py.224: train: epoch 61 | batch 700/2304 [I 2022-10-27 10:00:54] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:14:57 [I 2022-10-27 10:00:54] ...trap/engines/engine.py.229: process: 0.53628 | load: 0.00240 [I 2022-10-27 10:00:54] ...trap/engines/engine.py.230: loss: 0.17189 [I 2022-10-27 10:00:54] ...optimizers/trijoint.py.151: total_norm: 17.960087 [I 2022-10-27 10:01:00] ...trap/engines/engine.py.224: train: epoch 61 | batch 710/2304 [I 2022-10-27 10:01:00] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:45 [I 2022-10-27 10:01:00] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00245 [I 2022-10-27 10:01:00] ...trap/engines/engine.py.230: loss: 0.09493 [I 2022-10-27 10:01:00] ...optimizers/trijoint.py.151: total_norm: 11.276816 [I 2022-10-27 10:01:05] ...trap/engines/engine.py.224: train: epoch 61 | batch 720/2304 [I 2022-10-27 10:01:05] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:39 [I 2022-10-27 10:01:05] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00240 [I 2022-10-27 10:01:05] ...trap/engines/engine.py.230: loss: 0.13597 [I 2022-10-27 10:01:05] ...optimizers/trijoint.py.151: total_norm: 16.275059 [I 2022-10-27 10:01:11] ...trap/engines/engine.py.224: train: epoch 61 | batch 730/2304 [I 2022-10-27 10:01:11] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:42 [I 2022-10-27 10:01:11] ...trap/engines/engine.py.229: process: 0.55977 | load: 0.00260 [I 2022-10-27 10:01:11] ...trap/engines/engine.py.230: loss: 0.23516 [I 2022-10-27 10:01:11] ...optimizers/trijoint.py.151: total_norm: 23.810244 [I 2022-10-27 10:01:17] ...trap/engines/engine.py.224: train: epoch 61 | batch 740/2304 [I 2022-10-27 10:01:17] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:31 [I 2022-10-27 10:01:17] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00253 [I 2022-10-27 10:01:17] ...trap/engines/engine.py.230: loss: 0.11619 [I 2022-10-27 10:01:17] ...optimizers/trijoint.py.151: total_norm: 15.003300 [I 2022-10-27 10:01:22] ...trap/engines/engine.py.224: train: epoch 61 | batch 750/2304 [I 2022-10-27 10:01:22] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:24 [I 2022-10-27 10:01:22] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00246 [I 2022-10-27 10:01:22] ...trap/engines/engine.py.230: loss: 0.18404 [I 2022-10-27 10:01:22] ...optimizers/trijoint.py.151: total_norm: 18.102726 [I 2022-10-27 10:01:28] ...trap/engines/engine.py.224: train: epoch 61 | batch 760/2304 [I 2022-10-27 10:01:28] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:19 [I 2022-10-27 10:01:28] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00241 [I 2022-10-27 10:01:28] ...trap/engines/engine.py.230: loss: 0.20464 [I 2022-10-27 10:01:28] ...optimizers/trijoint.py.151: total_norm: 10.541509 [I 2022-10-27 10:01:33] ...trap/engines/engine.py.224: train: epoch 61 | batch 770/2304 [I 2022-10-27 10:01:33] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:22 [I 2022-10-27 10:01:33] ...trap/engines/engine.py.229: process: 0.56592 | load: 0.00243 [I 2022-10-27 10:01:33] ...trap/engines/engine.py.230: loss: 0.13560 [I 2022-10-27 10:01:33] ...optimizers/trijoint.py.151: total_norm: 12.416867 [I 2022-10-27 10:01:39] ...trap/engines/engine.py.224: train: epoch 61 | batch 780/2304 [I 2022-10-27 10:01:39] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:11 [I 2022-10-27 10:01:39] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00263 [I 2022-10-27 10:01:39] ...trap/engines/engine.py.230: loss: 0.14295 [I 2022-10-27 10:01:39] ...optimizers/trijoint.py.151: total_norm: 23.603083 [I 2022-10-27 10:01:45] ...trap/engines/engine.py.224: train: epoch 61 | batch 790/2304 [I 2022-10-27 10:01:45] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:05 [I 2022-10-27 10:01:45] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00246 [I 2022-10-27 10:01:45] ...trap/engines/engine.py.230: loss: 0.16370 [I 2022-10-27 10:01:45] ...optimizers/trijoint.py.151: total_norm: 20.003208 [I 2022-10-27 10:01:50] ...trap/engines/engine.py.224: train: epoch 61 | batch 800/2304 [I 2022-10-27 10:01:50] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:13:56 [I 2022-10-27 10:01:50] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00256 [I 2022-10-27 10:01:50] ...trap/engines/engine.py.230: loss: 0.15347 [I 2022-10-27 10:01:50] ...optimizers/trijoint.py.151: total_norm: 19.467648 [I 2022-10-27 10:01:56] ...trap/engines/engine.py.224: train: epoch 61 | batch 810/2304 [I 2022-10-27 10:01:56] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:51 [I 2022-10-27 10:01:56] ...trap/engines/engine.py.229: process: 0.56283 | load: 0.00249 [I 2022-10-27 10:01:56] ...trap/engines/engine.py.230: loss: 0.19111 [I 2022-10-27 10:01:56] ...optimizers/trijoint.py.151: total_norm: 16.100445 [I 2022-10-27 10:02:01] ...trap/engines/engine.py.224: train: epoch 61 | batch 820/2304 [I 2022-10-27 10:02:01] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:53 [I 2022-10-27 10:02:01] ...trap/engines/engine.py.229: process: 0.57729 | load: 0.00242 [I 2022-10-27 10:02:01] ...trap/engines/engine.py.230: loss: 0.15771 [I 2022-10-27 10:02:01] ...optimizers/trijoint.py.151: total_norm: 12.234279 [I 2022-10-27 10:02:07] ...trap/engines/engine.py.224: train: epoch 61 | batch 830/2304 [I 2022-10-27 10:02:07] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:37 [I 2022-10-27 10:02:07] ...trap/engines/engine.py.229: process: 0.54480 | load: 0.00245 [I 2022-10-27 10:02:07] ...trap/engines/engine.py.230: loss: 0.16718 [I 2022-10-27 10:02:07] ...optimizers/trijoint.py.151: total_norm: 30.433569 [I 2022-10-27 10:02:12] ...trap/engines/engine.py.224: train: epoch 61 | batch 840/2304 [I 2022-10-27 10:02:12] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:41 [I 2022-10-27 10:02:12] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00247 [I 2022-10-27 10:02:12] ...trap/engines/engine.py.230: loss: 0.12810 [I 2022-10-27 10:02:12] ...optimizers/trijoint.py.151: total_norm: 13.045569 [I 2022-10-27 10:02:18] ...trap/engines/engine.py.224: train: epoch 61 | batch 850/2304 [I 2022-10-27 10:02:18] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:30 [I 2022-10-27 10:02:18] ...trap/engines/engine.py.229: process: 0.54500 | load: 0.00245 [I 2022-10-27 10:02:18] ...trap/engines/engine.py.230: loss: 0.13067 [I 2022-10-27 10:02:18] ...optimizers/trijoint.py.151: total_norm: 7.254784 [I 2022-10-27 10:02:24] ...trap/engines/engine.py.224: train: epoch 61 | batch 860/2304 [I 2022-10-27 10:02:24] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:28 [I 2022-10-27 10:02:24] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00279 [I 2022-10-27 10:02:24] ...trap/engines/engine.py.230: loss: 0.15056 [I 2022-10-27 10:02:24] ...optimizers/trijoint.py.151: total_norm: 21.534309 [I 2022-10-27 10:02:29] ...trap/engines/engine.py.224: train: epoch 61 | batch 870/2304 [I 2022-10-27 10:02:29] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:24 [I 2022-10-27 10:02:29] ...trap/engines/engine.py.229: process: 0.55977 | load: 0.00242 [I 2022-10-27 10:02:29] ...trap/engines/engine.py.230: loss: 0.19504 [I 2022-10-27 10:02:29] ...optimizers/trijoint.py.151: total_norm: 21.850777 [I 2022-10-27 10:02:35] ...trap/engines/engine.py.224: train: epoch 61 | batch 880/2304 [I 2022-10-27 10:02:35] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:16 [I 2022-10-27 10:02:35] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00245 [I 2022-10-27 10:02:35] ...trap/engines/engine.py.230: loss: 0.16351 [I 2022-10-27 10:02:35] ...optimizers/trijoint.py.151: total_norm: 14.087889 [I 2022-10-27 10:02:40] ...trap/engines/engine.py.224: train: epoch 61 | batch 890/2304 [I 2022-10-27 10:02:40] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:15 [I 2022-10-27 10:02:40] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00258 [I 2022-10-27 10:02:40] ...trap/engines/engine.py.230: loss: 0.21944 [I 2022-10-27 10:02:40] ...optimizers/trijoint.py.151: total_norm: 23.273458 [I 2022-10-27 10:02:46] ...trap/engines/engine.py.224: train: epoch 61 | batch 900/2304 [I 2022-10-27 10:02:46] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:04 [I 2022-10-27 10:02:46] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00247 [I 2022-10-27 10:02:46] ...trap/engines/engine.py.230: loss: 0.13574 [I 2022-10-27 10:02:46] ...optimizers/trijoint.py.151: total_norm: 12.090549 [I 2022-10-27 10:02:52] ...trap/engines/engine.py.224: train: epoch 61 | batch 910/2304 [I 2022-10-27 10:02:52] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:57 [I 2022-10-27 10:02:52] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00280 [I 2022-10-27 10:02:52] ...trap/engines/engine.py.230: loss: 0.13690 [I 2022-10-27 10:02:52] ...optimizers/trijoint.py.151: total_norm: 9.445785 [I 2022-10-27 10:02:57] ...trap/engines/engine.py.224: train: epoch 61 | batch 920/2304 [I 2022-10-27 10:02:57] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:50 [I 2022-10-27 10:02:57] ...trap/engines/engine.py.229: process: 0.55848 | load: 0.00247 [I 2022-10-27 10:02:57] ...trap/engines/engine.py.230: loss: 0.11868 [I 2022-10-27 10:02:57] ...optimizers/trijoint.py.151: total_norm: 14.946393 [I 2022-10-27 10:03:03] ...trap/engines/engine.py.224: train: epoch 61 | batch 930/2304 [I 2022-10-27 10:03:03] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:43 [I 2022-10-27 10:03:03] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00244 [I 2022-10-27 10:03:03] ...trap/engines/engine.py.230: loss: 0.12472 [I 2022-10-27 10:03:03] ...optimizers/trijoint.py.151: total_norm: 19.991352 [I 2022-10-27 10:03:08] ...trap/engines/engine.py.224: train: epoch 61 | batch 940/2304 [I 2022-10-27 10:03:08] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:44 [I 2022-10-27 10:03:08] ...trap/engines/engine.py.229: process: 0.56408 | load: 0.00247 [I 2022-10-27 10:03:08] ...trap/engines/engine.py.230: loss: 0.12642 [I 2022-10-27 10:03:08] ...optimizers/trijoint.py.151: total_norm: 23.509142 [I 2022-10-27 10:03:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 950/2304 [I 2022-10-27 10:03:14] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:39 [I 2022-10-27 10:03:14] ...trap/engines/engine.py.229: process: 0.56221 | load: 0.00273 [I 2022-10-27 10:03:14] ...trap/engines/engine.py.230: loss: 0.14738 [I 2022-10-27 10:03:14] ...optimizers/trijoint.py.151: total_norm: 9.218665 [I 2022-10-27 10:03:20] ...trap/engines/engine.py.224: train: epoch 61 | batch 960/2304 [I 2022-10-27 10:03:20] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:30 [I 2022-10-27 10:03:20] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00275 [I 2022-10-27 10:03:20] ...trap/engines/engine.py.230: loss: 0.09643 [I 2022-10-27 10:03:20] ...optimizers/trijoint.py.151: total_norm: 8.692229 [I 2022-10-27 10:03:25] ...trap/engines/engine.py.224: train: epoch 61 | batch 970/2304 [I 2022-10-27 10:03:25] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:28 [I 2022-10-27 10:03:25] ...trap/engines/engine.py.229: process: 0.55786 | load: 0.00251 [I 2022-10-27 10:03:25] ...trap/engines/engine.py.230: loss: 0.12995 [I 2022-10-27 10:03:25] ...optimizers/trijoint.py.151: total_norm: 11.745476 [I 2022-10-27 10:03:31] ...trap/engines/engine.py.224: train: epoch 61 | batch 980/2304 [I 2022-10-27 10:03:31] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:18 [I 2022-10-27 10:03:31] ...trap/engines/engine.py.229: process: 0.56819 | load: 0.00253 [I 2022-10-27 10:03:31] ...trap/engines/engine.py.230: loss: 0.18612 [I 2022-10-27 10:03:31] ...optimizers/trijoint.py.151: total_norm: 12.637683 [I 2022-10-27 10:03:37] ...trap/engines/engine.py.224: train: epoch 61 | batch 990/2304 [I 2022-10-27 10:03:37] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:16 [I 2022-10-27 10:03:37] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00248 [I 2022-10-27 10:03:37] ...trap/engines/engine.py.230: loss: 0.10712 [I 2022-10-27 10:03:37] ...optimizers/trijoint.py.151: total_norm: 15.222096 [I 2022-10-27 10:03:42] ...trap/engines/engine.py.224: train: epoch 61 | batch 1000/2304 [I 2022-10-27 10:03:42] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:07 [I 2022-10-27 10:03:42] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00248 [I 2022-10-27 10:03:42] ...trap/engines/engine.py.230: loss: 0.14837 [I 2022-10-27 10:03:42] ...optimizers/trijoint.py.151: total_norm: 16.039087 [I 2022-10-27 10:03:48] ...trap/engines/engine.py.224: train: epoch 61 | batch 1010/2304 [I 2022-10-27 10:03:48] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:10 [I 2022-10-27 10:03:48] ...trap/engines/engine.py.229: process: 0.56261 | load: 0.00242 [I 2022-10-27 10:03:48] ...trap/engines/engine.py.230: loss: 0.14227 [I 2022-10-27 10:03:48] ...optimizers/trijoint.py.151: total_norm: 20.311371 [I 2022-10-27 10:03:53] ...trap/engines/engine.py.224: train: epoch 61 | batch 1020/2304 [I 2022-10-27 10:03:53] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:55 [I 2022-10-27 10:03:53] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00247 [I 2022-10-27 10:03:53] ...trap/engines/engine.py.230: loss: 0.15924 [I 2022-10-27 10:03:53] ...optimizers/trijoint.py.151: total_norm: 13.033896 [I 2022-10-27 10:03:59] ...trap/engines/engine.py.224: train: epoch 61 | batch 1030/2304 [I 2022-10-27 10:03:59] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:52 [I 2022-10-27 10:03:59] ...trap/engines/engine.py.229: process: 0.56278 | load: 0.00241 [I 2022-10-27 10:03:59] ...trap/engines/engine.py.230: loss: 0.14552 [I 2022-10-27 10:03:59] ...optimizers/trijoint.py.151: total_norm: 11.511645 [I 2022-10-27 10:04:05] ...trap/engines/engine.py.224: train: epoch 61 | batch 1040/2304 [I 2022-10-27 10:04:05] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:47 [I 2022-10-27 10:04:05] ...trap/engines/engine.py.229: process: 0.56229 | load: 0.00262 [I 2022-10-27 10:04:05] ...trap/engines/engine.py.230: loss: 0.15065 [I 2022-10-27 10:04:05] ...optimizers/trijoint.py.151: total_norm: 11.194815 [I 2022-10-27 10:04:10] ...trap/engines/engine.py.224: train: epoch 61 | batch 1050/2304 [I 2022-10-27 10:04:10] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:43 [I 2022-10-27 10:04:10] ...trap/engines/engine.py.229: process: 0.56784 | load: 0.00246 [I 2022-10-27 10:04:10] ...trap/engines/engine.py.230: loss: 0.12028 [I 2022-10-27 10:04:10] ...optimizers/trijoint.py.151: total_norm: 12.836194 [I 2022-10-27 10:04:16] ...trap/engines/engine.py.224: train: epoch 61 | batch 1060/2304 [I 2022-10-27 10:04:16] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:34 [I 2022-10-27 10:04:16] ...trap/engines/engine.py.229: process: 0.54738 | load: 0.00262 [I 2022-10-27 10:04:16] ...trap/engines/engine.py.230: loss: 0.23754 [I 2022-10-27 10:04:16] ...optimizers/trijoint.py.151: total_norm: 16.367048 [I 2022-10-27 10:04:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 1070/2304 [I 2022-10-27 10:04:21] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:31 [I 2022-10-27 10:04:21] ...trap/engines/engine.py.229: process: 0.55391 | load: 0.00523 [I 2022-10-27 10:04:21] ...trap/engines/engine.py.230: loss: 0.15719 [I 2022-10-27 10:04:21] ...optimizers/trijoint.py.151: total_norm: 20.962677 [I 2022-10-27 10:04:27] ...trap/engines/engine.py.224: train: epoch 61 | batch 1080/2304 [I 2022-10-27 10:04:27] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:29 [I 2022-10-27 10:04:27] ...trap/engines/engine.py.229: process: 0.57494 | load: 0.00251 [I 2022-10-27 10:04:27] ...trap/engines/engine.py.230: loss: 0.12859 [I 2022-10-27 10:04:27] ...optimizers/trijoint.py.151: total_norm: 17.838156 [I 2022-10-27 10:04:33] ...trap/engines/engine.py.224: train: epoch 61 | batch 1090/2304 [I 2022-10-27 10:04:33] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:16 [I 2022-10-27 10:04:33] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00246 [I 2022-10-27 10:04:33] ...trap/engines/engine.py.230: loss: 0.14842 [I 2022-10-27 10:04:33] ...optimizers/trijoint.py.151: total_norm: 16.849915 [I 2022-10-27 10:04:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 1100/2304 [I 2022-10-27 10:04:38] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:12 [I 2022-10-27 10:04:38] ...trap/engines/engine.py.229: process: 0.57159 | load: 0.00243 [I 2022-10-27 10:04:38] ...trap/engines/engine.py.230: loss: 0.13859 [I 2022-10-27 10:04:38] ...optimizers/trijoint.py.151: total_norm: 20.954460 [I 2022-10-27 10:04:44] ...trap/engines/engine.py.224: train: epoch 61 | batch 1110/2304 [I 2022-10-27 10:04:44] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:07 [I 2022-10-27 10:04:44] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00248 [I 2022-10-27 10:04:44] ...trap/engines/engine.py.230: loss: 0.22287 [I 2022-10-27 10:04:44] ...optimizers/trijoint.py.151: total_norm: 39.061848 [I 2022-10-27 10:04:49] ...trap/engines/engine.py.224: train: epoch 61 | batch 1120/2304 [I 2022-10-27 10:04:49] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:01 [I 2022-10-27 10:04:49] ...trap/engines/engine.py.229: process: 0.56427 | load: 0.00249 [I 2022-10-27 10:04:49] ...trap/engines/engine.py.230: loss: 0.19699 [I 2022-10-27 10:04:49] ...optimizers/trijoint.py.151: total_norm: 10.851176 [I 2022-10-27 10:04:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 1130/2304 [I 2022-10-27 10:04:55] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:59 [I 2022-10-27 10:04:55] ...trap/engines/engine.py.229: process: 0.57984 | load: 0.00246 [I 2022-10-27 10:04:55] ...trap/engines/engine.py.230: loss: 0.10730 [I 2022-10-27 10:04:55] ...optimizers/trijoint.py.151: total_norm: 8.589327 [I 2022-10-27 10:05:01] ...trap/engines/engine.py.224: train: epoch 61 | batch 1140/2304 [I 2022-10-27 10:05:01] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:10:53 [I 2022-10-27 10:05:01] ...trap/engines/engine.py.229: process: 0.56887 | load: 0.00248 [I 2022-10-27 10:05:01] ...trap/engines/engine.py.230: loss: 0.15811 [I 2022-10-27 10:05:01] ...optimizers/trijoint.py.151: total_norm: 33.765301 [I 2022-10-27 10:05:06] ...trap/engines/engine.py.224: train: epoch 61 | batch 1150/2304 [I 2022-10-27 10:05:06] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:45 [I 2022-10-27 10:05:06] ...trap/engines/engine.py.229: process: 0.54826 | load: 0.00260 [I 2022-10-27 10:05:06] ...trap/engines/engine.py.230: loss: 0.12439 [I 2022-10-27 10:05:06] ...optimizers/trijoint.py.151: total_norm: 16.650206 [I 2022-10-27 10:05:12] ...trap/engines/engine.py.224: train: epoch 61 | batch 1160/2304 [I 2022-10-27 10:05:12] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:44 [I 2022-10-27 10:05:12] ...trap/engines/engine.py.229: process: 0.58165 | load: 0.00259 [I 2022-10-27 10:05:12] ...trap/engines/engine.py.230: loss: 0.19224 [I 2022-10-27 10:05:12] ...optimizers/trijoint.py.151: total_norm: 16.491175 [I 2022-10-27 10:05:17] ...trap/engines/engine.py.224: train: epoch 61 | batch 1170/2304 [I 2022-10-27 10:05:17] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:32 [I 2022-10-27 10:05:17] ...trap/engines/engine.py.229: process: 0.55404 | load: 0.00253 [I 2022-10-27 10:05:17] ...trap/engines/engine.py.230: loss: 0.12790 [I 2022-10-27 10:05:17] ...optimizers/trijoint.py.151: total_norm: 13.080561 [I 2022-10-27 10:05:23] ...trap/engines/engine.py.224: train: epoch 61 | batch 1180/2304 [I 2022-10-27 10:05:23] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:34 [I 2022-10-27 10:05:23] ...trap/engines/engine.py.229: process: 0.56908 | load: 0.00254 [I 2022-10-27 10:05:23] ...trap/engines/engine.py.230: loss: 0.11940 [I 2022-10-27 10:05:23] ...optimizers/trijoint.py.151: total_norm: 25.181082 [I 2022-10-27 10:05:29] ...trap/engines/engine.py.224: train: epoch 61 | batch 1190/2304 [I 2022-10-27 10:05:29] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:18 [I 2022-10-27 10:05:29] ...trap/engines/engine.py.229: process: 0.55008 | load: 0.00245 [I 2022-10-27 10:05:29] ...trap/engines/engine.py.230: loss: 0.21126 [I 2022-10-27 10:05:29] ...optimizers/trijoint.py.151: total_norm: 17.750675 [I 2022-10-27 10:05:34] ...trap/engines/engine.py.224: train: epoch 61 | batch 1200/2304 [I 2022-10-27 10:05:34] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:17 [I 2022-10-27 10:05:34] ...trap/engines/engine.py.229: process: 0.56936 | load: 0.00245 [I 2022-10-27 10:05:34] ...trap/engines/engine.py.230: loss: 0.14276 [I 2022-10-27 10:05:34] ...optimizers/trijoint.py.151: total_norm: 9.758381 [I 2022-10-27 10:05:40] ...trap/engines/engine.py.224: train: epoch 61 | batch 1210/2304 [I 2022-10-27 10:05:40] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:10 [I 2022-10-27 10:05:40] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00250 [I 2022-10-27 10:05:40] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-27 10:05:40] ...optimizers/trijoint.py.151: total_norm: 11.409650 [I 2022-10-27 10:05:45] ...trap/engines/engine.py.224: train: epoch 61 | batch 1220/2304 [I 2022-10-27 10:05:45] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:04 [I 2022-10-27 10:05:45] ...trap/engines/engine.py.229: process: 0.56548 | load: 0.00279 [I 2022-10-27 10:05:45] ...trap/engines/engine.py.230: loss: 0.14859 [I 2022-10-27 10:05:45] ...optimizers/trijoint.py.151: total_norm: 16.202927 [I 2022-10-27 10:05:51] ...trap/engines/engine.py.224: train: epoch 61 | batch 1230/2304 [I 2022-10-27 10:05:51] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:09:58 [I 2022-10-27 10:05:51] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00252 [I 2022-10-27 10:05:51] ...trap/engines/engine.py.230: loss: 0.13548 [I 2022-10-27 10:05:51] ...optimizers/trijoint.py.151: total_norm: 8.684528 [I 2022-10-27 10:05:57] ...trap/engines/engine.py.224: train: epoch 61 | batch 1240/2304 [I 2022-10-27 10:05:57] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:53 [I 2022-10-27 10:05:57] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00245 [I 2022-10-27 10:05:57] ...trap/engines/engine.py.230: loss: 0.14282 [I 2022-10-27 10:05:57] ...optimizers/trijoint.py.151: total_norm: 17.255751 [I 2022-10-27 10:06:02] ...trap/engines/engine.py.224: train: epoch 61 | batch 1250/2304 [I 2022-10-27 10:06:02] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:42 [I 2022-10-27 10:06:02] ...trap/engines/engine.py.229: process: 0.54366 | load: 0.00242 [I 2022-10-27 10:06:02] ...trap/engines/engine.py.230: loss: 0.17743 [I 2022-10-27 10:06:02] ...optimizers/trijoint.py.151: total_norm: 17.797646 [I 2022-10-27 10:06:08] ...trap/engines/engine.py.224: train: epoch 61 | batch 1260/2304 [I 2022-10-27 10:06:08] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:47 [I 2022-10-27 10:06:08] ...trap/engines/engine.py.229: process: 0.59256 | load: 0.00243 [I 2022-10-27 10:06:08] ...trap/engines/engine.py.230: loss: 0.18223 [I 2022-10-27 10:06:08] ...optimizers/trijoint.py.151: total_norm: 28.024916 [I 2022-10-27 10:06:13] ...trap/engines/engine.py.224: train: epoch 61 | batch 1270/2304 [I 2022-10-27 10:06:13] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:40 [I 2022-10-27 10:06:13] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00245 [I 2022-10-27 10:06:13] ...trap/engines/engine.py.230: loss: 0.19789 [I 2022-10-27 10:06:13] ...optimizers/trijoint.py.151: total_norm: 23.534161 [I 2022-10-27 10:06:19] ...trap/engines/engine.py.224: train: epoch 61 | batch 1280/2304 [I 2022-10-27 10:06:19] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:09:45 [I 2022-10-27 10:06:19] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00251 [I 2022-10-27 10:06:19] ...trap/engines/engine.py.230: loss: 0.11418 [I 2022-10-27 10:06:19] ...optimizers/trijoint.py.151: total_norm: 14.910152 [I 2022-10-27 10:06:25] ...trap/engines/engine.py.224: train: epoch 61 | batch 1290/2304 [I 2022-10-27 10:06:25] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:29 [I 2022-10-27 10:06:25] ...trap/engines/engine.py.229: process: 0.54930 | load: 0.00260 [I 2022-10-27 10:06:25] ...trap/engines/engine.py.230: loss: 0.10328 [I 2022-10-27 10:06:25] ...optimizers/trijoint.py.151: total_norm: 9.315770 [I 2022-10-27 10:06:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 1300/2304 [I 2022-10-27 10:06:30] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:24 [I 2022-10-27 10:06:30] ...trap/engines/engine.py.229: process: 0.56706 | load: 0.00256 [I 2022-10-27 10:06:30] ...trap/engines/engine.py.230: loss: 0.13159 [I 2022-10-27 10:06:30] ...optimizers/trijoint.py.151: total_norm: 13.932999 [I 2022-10-27 10:06:36] ...trap/engines/engine.py.224: train: epoch 61 | batch 1310/2304 [I 2022-10-27 10:06:36] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:11 [I 2022-10-27 10:06:36] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00249 [I 2022-10-27 10:06:36] ...trap/engines/engine.py.230: loss: 0.13031 [I 2022-10-27 10:06:36] ...optimizers/trijoint.py.151: total_norm: 11.626926 [I 2022-10-27 10:06:42] ...trap/engines/engine.py.224: train: epoch 61 | batch 1320/2304 [I 2022-10-27 10:06:42] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:06 [I 2022-10-27 10:06:42] ...trap/engines/engine.py.229: process: 0.55267 | load: 0.00250 [I 2022-10-27 10:06:42] ...trap/engines/engine.py.230: loss: 0.13312 [I 2022-10-27 10:06:42] ...optimizers/trijoint.py.151: total_norm: 9.067969 [I 2022-10-27 10:06:47] ...trap/engines/engine.py.224: train: epoch 61 | batch 1330/2304 [I 2022-10-27 10:06:47] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:09:00 [I 2022-10-27 10:06:47] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00248 [I 2022-10-27 10:06:47] ...trap/engines/engine.py.230: loss: 0.11148 [I 2022-10-27 10:06:47] ...optimizers/trijoint.py.151: total_norm: 12.243176 [I 2022-10-27 10:06:53] ...trap/engines/engine.py.224: train: epoch 61 | batch 1340/2304 [I 2022-10-27 10:06:53] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:08:52 [I 2022-10-27 10:06:53] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00244 [I 2022-10-27 10:06:53] ...trap/engines/engine.py.230: loss: 0.13841 [I 2022-10-27 10:06:53] ...optimizers/trijoint.py.151: total_norm: 9.264526 [I 2022-10-27 10:06:58] ...trap/engines/engine.py.224: train: epoch 61 | batch 1350/2304 [I 2022-10-27 10:06:58] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:09:00 [I 2022-10-27 10:06:58] ...trap/engines/engine.py.229: process: 0.55204 | load: 0.00251 [I 2022-10-27 10:06:58] ...trap/engines/engine.py.230: loss: 0.13905 [I 2022-10-27 10:06:58] ...optimizers/trijoint.py.151: total_norm: 23.036818 [I 2022-10-27 10:07:04] ...trap/engines/engine.py.224: train: epoch 61 | batch 1360/2304 [I 2022-10-27 10:07:04] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:45 [I 2022-10-27 10:07:04] ...trap/engines/engine.py.229: process: 0.55007 | load: 0.00244 [I 2022-10-27 10:07:04] ...trap/engines/engine.py.230: loss: 0.15506 [I 2022-10-27 10:07:04] ...optimizers/trijoint.py.151: total_norm: 15.848433 [I 2022-10-27 10:07:10] ...trap/engines/engine.py.224: train: epoch 61 | batch 1370/2304 [I 2022-10-27 10:07:10] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:41 [I 2022-10-27 10:07:10] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00254 [I 2022-10-27 10:07:10] ...trap/engines/engine.py.230: loss: 0.18633 [I 2022-10-27 10:07:10] ...optimizers/trijoint.py.151: total_norm: 39.928013 [I 2022-10-27 10:07:15] ...trap/engines/engine.py.224: train: epoch 61 | batch 1380/2304 [I 2022-10-27 10:07:15] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:40 [I 2022-10-27 10:07:15] ...trap/engines/engine.py.229: process: 0.57935 | load: 0.00251 [I 2022-10-27 10:07:15] ...trap/engines/engine.py.230: loss: 0.12642 [I 2022-10-27 10:07:15] ...optimizers/trijoint.py.151: total_norm: 10.530267 [I 2022-10-27 10:07:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 1390/2304 [I 2022-10-27 10:07:21] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:29 [I 2022-10-27 10:07:21] ...trap/engines/engine.py.229: process: 0.54249 | load: 0.00245 [I 2022-10-27 10:07:21] ...trap/engines/engine.py.230: loss: 0.13688 [I 2022-10-27 10:07:21] ...optimizers/trijoint.py.151: total_norm: 10.067738 [I 2022-10-27 10:07:26] ...trap/engines/engine.py.224: train: epoch 61 | batch 1400/2304 [I 2022-10-27 10:07:26] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:08:22 [I 2022-10-27 10:07:26] ...trap/engines/engine.py.229: process: 0.56168 | load: 0.00250 [I 2022-10-27 10:07:26] ...trap/engines/engine.py.230: loss: 0.09862 [I 2022-10-27 10:07:26] ...optimizers/trijoint.py.151: total_norm: 8.236267 [I 2022-10-27 10:07:32] ...trap/engines/engine.py.224: train: epoch 61 | batch 1410/2304 [I 2022-10-27 10:07:32] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:21 [I 2022-10-27 10:07:32] ...trap/engines/engine.py.229: process: 0.56644 | load: 0.00247 [I 2022-10-27 10:07:32] ...trap/engines/engine.py.230: loss: 0.16124 [I 2022-10-27 10:07:32] ...optimizers/trijoint.py.151: total_norm: 26.235750 [I 2022-10-27 10:07:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 1420/2304 [I 2022-10-27 10:07:38] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:16 [I 2022-10-27 10:07:38] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00244 [I 2022-10-27 10:07:38] ...trap/engines/engine.py.230: loss: 0.16453 [I 2022-10-27 10:07:38] ...optimizers/trijoint.py.151: total_norm: 10.636352 [I 2022-10-27 10:07:43] ...trap/engines/engine.py.224: train: epoch 61 | batch 1430/2304 [I 2022-10-27 10:07:43] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:03 [I 2022-10-27 10:07:43] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00248 [I 2022-10-27 10:07:43] ...trap/engines/engine.py.230: loss: 0.17191 [I 2022-10-27 10:07:43] ...optimizers/trijoint.py.151: total_norm: 14.583746 [I 2022-10-27 10:07:49] ...trap/engines/engine.py.224: train: epoch 61 | batch 1440/2304 [I 2022-10-27 10:07:49] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:07:58 [I 2022-10-27 10:07:49] ...trap/engines/engine.py.229: process: 0.54474 | load: 0.00258 [I 2022-10-27 10:07:49] ...trap/engines/engine.py.230: loss: 0.13022 [I 2022-10-27 10:07:49] ...optimizers/trijoint.py.151: total_norm: 12.929262 [I 2022-10-27 10:07:54] ...trap/engines/engine.py.224: train: epoch 61 | batch 1450/2304 [I 2022-10-27 10:07:54] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:07:55 [I 2022-10-27 10:07:54] ...trap/engines/engine.py.229: process: 0.55658 | load: 0.00252 [I 2022-10-27 10:07:54] ...trap/engines/engine.py.230: loss: 0.15739 [I 2022-10-27 10:07:54] ...optimizers/trijoint.py.151: total_norm: 10.187984 [I 2022-10-27 10:08:00] ...trap/engines/engine.py.224: train: epoch 61 | batch 1460/2304 [I 2022-10-27 10:08:00] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:51 [I 2022-10-27 10:08:00] ...trap/engines/engine.py.229: process: 0.56579 | load: 0.00243 [I 2022-10-27 10:08:00] ...trap/engines/engine.py.230: loss: 0.10867 [I 2022-10-27 10:08:00] ...optimizers/trijoint.py.151: total_norm: 14.382085 [I 2022-10-27 10:08:05] ...trap/engines/engine.py.224: train: epoch 61 | batch 1470/2304 [I 2022-10-27 10:08:05] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:43 [I 2022-10-27 10:08:05] ...trap/engines/engine.py.229: process: 0.55231 | load: 0.00245 [I 2022-10-27 10:08:05] ...trap/engines/engine.py.230: loss: 0.13327 [I 2022-10-27 10:08:05] ...optimizers/trijoint.py.151: total_norm: 22.630808 [I 2022-10-27 10:08:11] ...trap/engines/engine.py.224: train: epoch 61 | batch 1480/2304 [I 2022-10-27 10:08:11] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:39 [I 2022-10-27 10:08:11] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00244 [I 2022-10-27 10:08:11] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-27 10:08:11] ...optimizers/trijoint.py.151: total_norm: 15.218361 [I 2022-10-27 10:08:17] ...trap/engines/engine.py.224: train: epoch 61 | batch 1490/2304 [I 2022-10-27 10:08:17] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:36 [I 2022-10-27 10:08:17] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00248 [I 2022-10-27 10:08:17] ...trap/engines/engine.py.230: loss: 0.14520 [I 2022-10-27 10:08:17] ...optimizers/trijoint.py.151: total_norm: 30.538189 [I 2022-10-27 10:08:22] ...trap/engines/engine.py.224: train: epoch 61 | batch 1500/2304 [I 2022-10-27 10:08:22] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:07:25 [I 2022-10-27 10:08:22] ...trap/engines/engine.py.229: process: 0.54936 | load: 0.00250 [I 2022-10-27 10:08:22] ...trap/engines/engine.py.230: loss: 0.13090 [I 2022-10-27 10:08:22] ...optimizers/trijoint.py.151: total_norm: 17.540630 [I 2022-10-27 10:08:28] ...trap/engines/engine.py.224: train: epoch 61 | batch 1510/2304 [I 2022-10-27 10:08:28] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:23 [I 2022-10-27 10:08:28] ...trap/engines/engine.py.229: process: 0.55832 | load: 0.00246 [I 2022-10-27 10:08:28] ...trap/engines/engine.py.230: loss: 0.19821 [I 2022-10-27 10:08:28] ...optimizers/trijoint.py.151: total_norm: 20.816994 [I 2022-10-27 10:08:33] ...trap/engines/engine.py.224: train: epoch 61 | batch 1520/2304 [I 2022-10-27 10:08:33] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:07:21 [I 2022-10-27 10:08:33] ...trap/engines/engine.py.229: process: 0.56642 | load: 0.00286 [I 2022-10-27 10:08:33] ...trap/engines/engine.py.230: loss: 0.12257 [I 2022-10-27 10:08:33] ...optimizers/trijoint.py.151: total_norm: 13.067609 [I 2022-10-27 10:08:39] ...trap/engines/engine.py.224: train: epoch 61 | batch 1530/2304 [I 2022-10-27 10:08:39] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:14 [I 2022-10-27 10:08:39] ...trap/engines/engine.py.229: process: 0.55602 | load: 0.00277 [I 2022-10-27 10:08:39] ...trap/engines/engine.py.230: loss: 0.11854 [I 2022-10-27 10:08:39] ...optimizers/trijoint.py.151: total_norm: 9.254758 [I 2022-10-27 10:08:45] ...trap/engines/engine.py.224: train: epoch 61 | batch 1540/2304 [I 2022-10-27 10:08:45] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:06 [I 2022-10-27 10:08:45] ...trap/engines/engine.py.229: process: 0.56274 | load: 0.00258 [I 2022-10-27 10:08:45] ...trap/engines/engine.py.230: loss: 0.09665 [I 2022-10-27 10:08:45] ...optimizers/trijoint.py.151: total_norm: 8.779935 [I 2022-10-27 10:08:50] ...trap/engines/engine.py.224: train: epoch 61 | batch 1550/2304 [I 2022-10-27 10:08:50] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:07:00 [I 2022-10-27 10:08:50] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00267 [I 2022-10-27 10:08:50] ...trap/engines/engine.py.230: loss: 0.13139 [I 2022-10-27 10:08:50] ...optimizers/trijoint.py.151: total_norm: 19.957697 [I 2022-10-27 10:08:56] ...trap/engines/engine.py.224: train: epoch 61 | batch 1560/2304 [I 2022-10-27 10:08:56] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:54 [I 2022-10-27 10:08:56] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00249 [I 2022-10-27 10:08:56] ...trap/engines/engine.py.230: loss: 0.24091 [I 2022-10-27 10:08:56] ...optimizers/trijoint.py.151: total_norm: 20.093142 [I 2022-10-27 10:09:01] ...trap/engines/engine.py.224: train: epoch 61 | batch 1570/2304 [I 2022-10-27 10:09:01] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:06:54 [I 2022-10-27 10:09:01] ...trap/engines/engine.py.229: process: 0.56470 | load: 0.00248 [I 2022-10-27 10:09:01] ...trap/engines/engine.py.230: loss: 0.16489 [I 2022-10-27 10:09:01] ...optimizers/trijoint.py.151: total_norm: 19.361937 [I 2022-10-27 10:09:07] ...trap/engines/engine.py.224: train: epoch 61 | batch 1580/2304 [I 2022-10-27 10:09:07] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:42 [I 2022-10-27 10:09:07] ...trap/engines/engine.py.229: process: 0.55192 | load: 0.00254 [I 2022-10-27 10:09:07] ...trap/engines/engine.py.230: loss: 0.27246 [I 2022-10-27 10:09:07] ...optimizers/trijoint.py.151: total_norm: 12.156629 [I 2022-10-27 10:09:13] ...trap/engines/engine.py.224: train: epoch 61 | batch 1590/2304 [I 2022-10-27 10:09:13] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:42 [I 2022-10-27 10:09:13] ...trap/engines/engine.py.229: process: 0.56004 | load: 0.00252 [I 2022-10-27 10:09:13] ...trap/engines/engine.py.230: loss: 0.15500 [I 2022-10-27 10:09:13] ...optimizers/trijoint.py.151: total_norm: 12.321469 [I 2022-10-27 10:09:18] ...trap/engines/engine.py.224: train: epoch 61 | batch 1600/2304 [I 2022-10-27 10:09:18] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:36 [I 2022-10-27 10:09:18] ...trap/engines/engine.py.229: process: 0.56623 | load: 0.00254 [I 2022-10-27 10:09:18] ...trap/engines/engine.py.230: loss: 0.11842 [I 2022-10-27 10:09:18] ...optimizers/trijoint.py.151: total_norm: 19.379307 [I 2022-10-27 10:09:24] ...trap/engines/engine.py.224: train: epoch 61 | batch 1610/2304 [I 2022-10-27 10:09:24] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:26 [I 2022-10-27 10:09:24] ...trap/engines/engine.py.229: process: 0.56600 | load: 0.00244 [I 2022-10-27 10:09:24] ...trap/engines/engine.py.230: loss: 0.12593 [I 2022-10-27 10:09:24] ...optimizers/trijoint.py.151: total_norm: 14.816442 [I 2022-10-27 10:09:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 1620/2304 [I 2022-10-27 10:09:30] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:22 [I 2022-10-27 10:09:30] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00246 [I 2022-10-27 10:09:30] ...trap/engines/engine.py.230: loss: 0.16008 [I 2022-10-27 10:09:30] ...optimizers/trijoint.py.151: total_norm: 12.028849 [I 2022-10-27 10:09:35] ...trap/engines/engine.py.224: train: epoch 61 | batch 1630/2304 [I 2022-10-27 10:09:35] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:15 [I 2022-10-27 10:09:35] ...trap/engines/engine.py.229: process: 0.55888 | load: 0.00245 [I 2022-10-27 10:09:35] ...trap/engines/engine.py.230: loss: 0.15101 [I 2022-10-27 10:09:35] ...optimizers/trijoint.py.151: total_norm: 10.081757 [I 2022-10-27 10:09:41] ...trap/engines/engine.py.224: train: epoch 61 | batch 1640/2304 [I 2022-10-27 10:09:41] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:29 [I 2022-10-27 10:09:41] ...trap/engines/engine.py.229: process: 0.68494 | load: 0.00278 [I 2022-10-27 10:09:41] ...trap/engines/engine.py.230: loss: 0.11044 [I 2022-10-27 10:09:41] ...optimizers/trijoint.py.151: total_norm: 10.638468 [I 2022-10-27 10:09:46] ...trap/engines/engine.py.224: train: epoch 61 | batch 1650/2304 [I 2022-10-27 10:09:46] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:09 [I 2022-10-27 10:09:46] ...trap/engines/engine.py.229: process: 0.57213 | load: 0.00251 [I 2022-10-27 10:09:46] ...trap/engines/engine.py.230: loss: 0.11487 [I 2022-10-27 10:09:46] ...optimizers/trijoint.py.151: total_norm: 13.459076 [I 2022-10-27 10:09:52] ...trap/engines/engine.py.224: train: epoch 61 | batch 1660/2304 [I 2022-10-27 10:09:52] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:06:00 [I 2022-10-27 10:09:52] ...trap/engines/engine.py.229: process: 0.55555 | load: 0.00249 [I 2022-10-27 10:09:52] ...trap/engines/engine.py.230: loss: 0.19526 [I 2022-10-27 10:09:52] ...optimizers/trijoint.py.151: total_norm: 23.326077 [I 2022-10-27 10:09:58] ...trap/engines/engine.py.224: train: epoch 61 | batch 1670/2304 [I 2022-10-27 10:09:58] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:55 [I 2022-10-27 10:09:58] ...trap/engines/engine.py.229: process: 0.56048 | load: 0.00252 [I 2022-10-27 10:09:58] ...trap/engines/engine.py.230: loss: 0.15066 [I 2022-10-27 10:09:58] ...optimizers/trijoint.py.151: total_norm: 12.734469 [I 2022-10-27 10:10:03] ...trap/engines/engine.py.224: train: epoch 61 | batch 1680/2304 [I 2022-10-27 10:10:03] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:05:49 [I 2022-10-27 10:10:03] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00248 [I 2022-10-27 10:10:03] ...trap/engines/engine.py.230: loss: 0.24761 [I 2022-10-27 10:10:03] ...optimizers/trijoint.py.151: total_norm: 31.675871 [I 2022-10-27 10:10:09] ...trap/engines/engine.py.224: train: epoch 61 | batch 1690/2304 [I 2022-10-27 10:10:09] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:41 [I 2022-10-27 10:10:09] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00247 [I 2022-10-27 10:10:09] ...trap/engines/engine.py.230: loss: 0.09852 [I 2022-10-27 10:10:09] ...optimizers/trijoint.py.151: total_norm: 9.981147 [I 2022-10-27 10:10:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 1700/2304 [I 2022-10-27 10:10:14] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:33 [I 2022-10-27 10:10:14] ...trap/engines/engine.py.229: process: 0.55237 | load: 0.00240 [I 2022-10-27 10:10:14] ...trap/engines/engine.py.230: loss: 0.12044 [I 2022-10-27 10:10:14] ...optimizers/trijoint.py.151: total_norm: 19.854179 [I 2022-10-27 10:10:20] ...trap/engines/engine.py.224: train: epoch 61 | batch 1710/2304 [I 2022-10-27 10:10:20] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:30 [I 2022-10-27 10:10:20] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00256 [I 2022-10-27 10:10:20] ...trap/engines/engine.py.230: loss: 0.10174 [I 2022-10-27 10:10:20] ...optimizers/trijoint.py.151: total_norm: 19.677099 [I 2022-10-27 10:10:25] ...trap/engines/engine.py.224: train: epoch 61 | batch 1720/2304 [I 2022-10-27 10:10:25] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:22 [I 2022-10-27 10:10:25] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00248 [I 2022-10-27 10:10:25] ...trap/engines/engine.py.230: loss: 0.14564 [I 2022-10-27 10:10:25] ...optimizers/trijoint.py.151: total_norm: 27.169762 [I 2022-10-27 10:10:31] ...trap/engines/engine.py.224: train: epoch 61 | batch 1730/2304 [I 2022-10-27 10:10:31] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:17 [I 2022-10-27 10:10:31] ...trap/engines/engine.py.229: process: 0.55229 | load: 0.00242 [I 2022-10-27 10:10:31] ...trap/engines/engine.py.230: loss: 0.12607 [I 2022-10-27 10:10:31] ...optimizers/trijoint.py.151: total_norm: 19.035151 [I 2022-10-27 10:10:37] ...trap/engines/engine.py.224: train: epoch 61 | batch 1740/2304 [I 2022-10-27 10:10:37] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:13 [I 2022-10-27 10:10:37] ...trap/engines/engine.py.229: process: 0.54993 | load: 0.00249 [I 2022-10-27 10:10:37] ...trap/engines/engine.py.230: loss: 0.10534 [I 2022-10-27 10:10:37] ...optimizers/trijoint.py.151: total_norm: 8.592154 [I 2022-10-27 10:10:42] ...trap/engines/engine.py.224: train: epoch 61 | batch 1750/2304 [I 2022-10-27 10:10:42] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:08 [I 2022-10-27 10:10:42] ...trap/engines/engine.py.229: process: 0.56149 | load: 0.00251 [I 2022-10-27 10:10:42] ...trap/engines/engine.py.230: loss: 0.16141 [I 2022-10-27 10:10:42] ...optimizers/trijoint.py.151: total_norm: 21.138876 [I 2022-10-27 10:10:48] ...trap/engines/engine.py.224: train: epoch 61 | batch 1760/2304 [I 2022-10-27 10:10:48] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:06 [I 2022-10-27 10:10:48] ...trap/engines/engine.py.229: process: 0.54169 | load: 0.00243 [I 2022-10-27 10:10:48] ...trap/engines/engine.py.230: loss: 0.13044 [I 2022-10-27 10:10:48] ...optimizers/trijoint.py.151: total_norm: 23.017891 [I 2022-10-27 10:10:53] ...trap/engines/engine.py.224: train: epoch 61 | batch 1770/2304 [I 2022-10-27 10:10:53] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:04:57 [I 2022-10-27 10:10:53] ...trap/engines/engine.py.229: process: 0.55662 | load: 0.00252 [I 2022-10-27 10:10:53] ...trap/engines/engine.py.230: loss: 0.18977 [I 2022-10-27 10:10:53] ...optimizers/trijoint.py.151: total_norm: 26.572601 [I 2022-10-27 10:10:59] ...trap/engines/engine.py.224: train: epoch 61 | batch 1780/2304 [I 2022-10-27 10:10:59] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:04:52 [I 2022-10-27 10:10:59] ...trap/engines/engine.py.229: process: 0.56705 | load: 0.00251 [I 2022-10-27 10:10:59] ...trap/engines/engine.py.230: loss: 0.15773 [I 2022-10-27 10:10:59] ...optimizers/trijoint.py.151: total_norm: 13.654358 [I 2022-10-27 10:11:05] ...trap/engines/engine.py.224: train: epoch 61 | batch 1790/2304 [I 2022-10-27 10:11:05] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:43 [I 2022-10-27 10:11:05] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00250 [I 2022-10-27 10:11:05] ...trap/engines/engine.py.230: loss: 0.11053 [I 2022-10-27 10:11:05] ...optimizers/trijoint.py.151: total_norm: 11.657634 [I 2022-10-27 10:11:10] ...trap/engines/engine.py.224: train: epoch 61 | batch 1800/2304 [I 2022-10-27 10:11:10] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:37 [I 2022-10-27 10:11:10] ...trap/engines/engine.py.229: process: 0.53897 | load: 0.00266 [I 2022-10-27 10:11:10] ...trap/engines/engine.py.230: loss: 0.10176 [I 2022-10-27 10:11:10] ...optimizers/trijoint.py.151: total_norm: 10.014310 [I 2022-10-27 10:11:16] ...trap/engines/engine.py.224: train: epoch 61 | batch 1810/2304 [I 2022-10-27 10:11:16] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:36 [I 2022-10-27 10:11:16] ...trap/engines/engine.py.229: process: 0.57080 | load: 0.00253 [I 2022-10-27 10:11:16] ...trap/engines/engine.py.230: loss: 0.20194 [I 2022-10-27 10:11:16] ...optimizers/trijoint.py.151: total_norm: 21.452578 [I 2022-10-27 10:11:21] ...trap/engines/engine.py.224: train: epoch 61 | batch 1820/2304 [I 2022-10-27 10:11:21] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:30 [I 2022-10-27 10:11:21] ...trap/engines/engine.py.229: process: 0.56308 | load: 0.00248 [I 2022-10-27 10:11:21] ...trap/engines/engine.py.230: loss: 0.22405 [I 2022-10-27 10:11:21] ...optimizers/trijoint.py.151: total_norm: 28.577419 [I 2022-10-27 10:11:27] ...trap/engines/engine.py.224: train: epoch 61 | batch 1830/2304 [I 2022-10-27 10:11:27] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:23 [I 2022-10-27 10:11:27] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00245 [I 2022-10-27 10:11:27] ...trap/engines/engine.py.230: loss: 0.14183 [I 2022-10-27 10:11:27] ...optimizers/trijoint.py.151: total_norm: 14.643715 [I 2022-10-27 10:11:32] ...trap/engines/engine.py.224: train: epoch 61 | batch 1840/2304 [I 2022-10-27 10:11:32] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:17 [I 2022-10-27 10:11:32] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00242 [I 2022-10-27 10:11:32] ...trap/engines/engine.py.230: loss: 0.10208 [I 2022-10-27 10:11:32] ...optimizers/trijoint.py.151: total_norm: 12.759716 [I 2022-10-27 10:11:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 1850/2304 [I 2022-10-27 10:11:38] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:10 [I 2022-10-27 10:11:38] ...trap/engines/engine.py.229: process: 0.54669 | load: 0.00241 [I 2022-10-27 10:11:38] ...trap/engines/engine.py.230: loss: 0.14114 [I 2022-10-27 10:11:38] ...optimizers/trijoint.py.151: total_norm: 19.526495 [I 2022-10-27 10:11:44] ...trap/engines/engine.py.224: train: epoch 61 | batch 1860/2304 [I 2022-10-27 10:11:44] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:06 [I 2022-10-27 10:11:44] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00245 [I 2022-10-27 10:11:44] ...trap/engines/engine.py.230: loss: 0.17654 [I 2022-10-27 10:11:44] ...optimizers/trijoint.py.151: total_norm: 24.639807 [I 2022-10-27 10:11:49] ...trap/engines/engine.py.224: train: epoch 61 | batch 1870/2304 [I 2022-10-27 10:11:49] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:00 [I 2022-10-27 10:11:49] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00241 [I 2022-10-27 10:11:49] ...trap/engines/engine.py.230: loss: 0.16182 [I 2022-10-27 10:11:49] ...optimizers/trijoint.py.151: total_norm: 18.917767 [I 2022-10-27 10:11:55] ...trap/engines/engine.py.224: train: epoch 61 | batch 1880/2304 [I 2022-10-27 10:11:55] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:03:54 [I 2022-10-27 10:11:55] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00262 [I 2022-10-27 10:11:55] ...trap/engines/engine.py.230: loss: 0.12494 [I 2022-10-27 10:11:55] ...optimizers/trijoint.py.151: total_norm: 8.750798 [I 2022-10-27 10:12:00] ...trap/engines/engine.py.224: train: epoch 61 | batch 1890/2304 [I 2022-10-27 10:12:00] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:48 [I 2022-10-27 10:12:00] ...trap/engines/engine.py.229: process: 0.54294 | load: 0.00255 [I 2022-10-27 10:12:00] ...trap/engines/engine.py.230: loss: 0.09917 [I 2022-10-27 10:12:00] ...optimizers/trijoint.py.151: total_norm: 7.580214 [I 2022-10-27 10:12:06] ...trap/engines/engine.py.224: train: epoch 61 | batch 1900/2304 [I 2022-10-27 10:12:06] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:44 [I 2022-10-27 10:12:06] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00251 [I 2022-10-27 10:12:06] ...trap/engines/engine.py.230: loss: 0.14594 [I 2022-10-27 10:12:06] ...optimizers/trijoint.py.151: total_norm: 16.541796 [I 2022-10-27 10:12:11] ...trap/engines/engine.py.224: train: epoch 61 | batch 1910/2304 [I 2022-10-27 10:12:11] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:42 [I 2022-10-27 10:12:11] ...trap/engines/engine.py.229: process: 0.57350 | load: 0.00248 [I 2022-10-27 10:12:11] ...trap/engines/engine.py.230: loss: 0.16490 [I 2022-10-27 10:12:11] ...optimizers/trijoint.py.151: total_norm: 24.913023 [I 2022-10-27 10:12:17] ...trap/engines/engine.py.224: train: epoch 61 | batch 1920/2304 [I 2022-10-27 10:12:17] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:34 [I 2022-10-27 10:12:17] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00249 [I 2022-10-27 10:12:17] ...trap/engines/engine.py.230: loss: 0.10475 [I 2022-10-27 10:12:17] ...optimizers/trijoint.py.151: total_norm: 12.155497 [I 2022-10-27 10:12:23] ...trap/engines/engine.py.224: train: epoch 61 | batch 1930/2304 [I 2022-10-27 10:12:23] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:28 [I 2022-10-27 10:12:23] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00246 [I 2022-10-27 10:12:23] ...trap/engines/engine.py.230: loss: 0.15763 [I 2022-10-27 10:12:23] ...optimizers/trijoint.py.151: total_norm: 27.173763 [I 2022-10-27 10:12:28] ...trap/engines/engine.py.224: train: epoch 61 | batch 1940/2304 [I 2022-10-27 10:12:28] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:23 [I 2022-10-27 10:12:28] ...trap/engines/engine.py.229: process: 0.55793 | load: 0.00244 [I 2022-10-27 10:12:28] ...trap/engines/engine.py.230: loss: 0.18117 [I 2022-10-27 10:12:28] ...optimizers/trijoint.py.151: total_norm: 29.700054 [I 2022-10-27 10:12:34] ...trap/engines/engine.py.224: train: epoch 61 | batch 1950/2304 [I 2022-10-27 10:12:34] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:18 [I 2022-10-27 10:12:34] ...trap/engines/engine.py.229: process: 0.56674 | load: 0.00261 [I 2022-10-27 10:12:34] ...trap/engines/engine.py.230: loss: 0.11930 [I 2022-10-27 10:12:34] ...optimizers/trijoint.py.151: total_norm: 10.980131 [I 2022-10-27 10:12:39] ...trap/engines/engine.py.224: train: epoch 61 | batch 1960/2304 [I 2022-10-27 10:12:39] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:13 [I 2022-10-27 10:12:39] ...trap/engines/engine.py.229: process: 0.56096 | load: 0.00257 [I 2022-10-27 10:12:39] ...trap/engines/engine.py.230: loss: 0.13951 [I 2022-10-27 10:12:39] ...optimizers/trijoint.py.151: total_norm: 17.343512 [I 2022-10-27 10:12:45] ...trap/engines/engine.py.224: train: epoch 61 | batch 1970/2304 [I 2022-10-27 10:12:45] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:05 [I 2022-10-27 10:12:45] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00252 [I 2022-10-27 10:12:45] ...trap/engines/engine.py.230: loss: 0.19629 [I 2022-10-27 10:12:45] ...optimizers/trijoint.py.151: total_norm: 20.079964 [I 2022-10-27 10:12:50] ...trap/engines/engine.py.224: train: epoch 61 | batch 1980/2304 [I 2022-10-27 10:12:50] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:00 [I 2022-10-27 10:12:50] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00248 [I 2022-10-27 10:12:50] ...trap/engines/engine.py.230: loss: 0.11198 [I 2022-10-27 10:12:50] ...optimizers/trijoint.py.151: total_norm: 9.264554 [I 2022-10-27 10:12:56] ...trap/engines/engine.py.224: train: epoch 61 | batch 1990/2304 [I 2022-10-27 10:12:56] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:02:56 [I 2022-10-27 10:12:56] ...trap/engines/engine.py.229: process: 0.57357 | load: 0.00248 [I 2022-10-27 10:12:56] ...trap/engines/engine.py.230: loss: 0.11178 [I 2022-10-27 10:12:56] ...optimizers/trijoint.py.151: total_norm: 14.898045 [I 2022-10-27 10:13:02] ...trap/engines/engine.py.224: train: epoch 61 | batch 2000/2304 [I 2022-10-27 10:13:02] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:50 [I 2022-10-27 10:13:02] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00249 [I 2022-10-27 10:13:02] ...trap/engines/engine.py.230: loss: 0.24016 [I 2022-10-27 10:13:02] ...optimizers/trijoint.py.151: total_norm: 25.586840 [I 2022-10-27 10:13:07] ...trap/engines/engine.py.224: train: epoch 61 | batch 2010/2304 [I 2022-10-27 10:13:07] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:44 [I 2022-10-27 10:13:07] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00241 [I 2022-10-27 10:13:07] ...trap/engines/engine.py.230: loss: 0.21810 [I 2022-10-27 10:13:07] ...optimizers/trijoint.py.151: total_norm: 29.091404 [I 2022-10-27 10:13:13] ...trap/engines/engine.py.224: train: epoch 61 | batch 2020/2304 [I 2022-10-27 10:13:13] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:38 [I 2022-10-27 10:13:13] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00248 [I 2022-10-27 10:13:13] ...trap/engines/engine.py.230: loss: 0.16175 [I 2022-10-27 10:13:13] ...optimizers/trijoint.py.151: total_norm: 16.330864 [I 2022-10-27 10:13:18] ...trap/engines/engine.py.224: train: epoch 61 | batch 2030/2304 [I 2022-10-27 10:13:19] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:33 [I 2022-10-27 10:13:19] ...trap/engines/engine.py.229: process: 0.57281 | load: 0.00241 [I 2022-10-27 10:13:19] ...trap/engines/engine.py.230: loss: 0.11792 [I 2022-10-27 10:13:19] ...optimizers/trijoint.py.151: total_norm: 21.375526 [I 2022-10-27 10:13:24] ...trap/engines/engine.py.224: train: epoch 61 | batch 2040/2304 [I 2022-10-27 10:13:24] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:27 [I 2022-10-27 10:13:24] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00250 [I 2022-10-27 10:13:24] ...trap/engines/engine.py.230: loss: 0.18534 [I 2022-10-27 10:13:24] ...optimizers/trijoint.py.151: total_norm: 12.696329 [I 2022-10-27 10:13:30] ...trap/engines/engine.py.224: train: epoch 61 | batch 2050/2304 [I 2022-10-27 10:13:30] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:22 [I 2022-10-27 10:13:30] ...trap/engines/engine.py.229: process: 0.56845 | load: 0.00248 [I 2022-10-27 10:13:30] ...trap/engines/engine.py.230: loss: 0.13082 [I 2022-10-27 10:13:30] ...optimizers/trijoint.py.151: total_norm: 9.593901 [I 2022-10-27 10:13:35] ...trap/engines/engine.py.224: train: epoch 61 | batch 2060/2304 [I 2022-10-27 10:13:35] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:16 [I 2022-10-27 10:13:35] ...trap/engines/engine.py.229: process: 0.56183 | load: 0.00242 [I 2022-10-27 10:13:35] ...trap/engines/engine.py.230: loss: 0.09101 [I 2022-10-27 10:13:35] ...optimizers/trijoint.py.151: total_norm: 12.068971 [I 2022-10-27 10:13:41] ...trap/engines/engine.py.224: train: epoch 61 | batch 2070/2304 [I 2022-10-27 10:13:41] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:10 [I 2022-10-27 10:13:41] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00242 [I 2022-10-27 10:13:41] ...trap/engines/engine.py.230: loss: 0.17488 [I 2022-10-27 10:13:41] ...optimizers/trijoint.py.151: total_norm: 17.604399 [I 2022-10-27 10:13:46] ...trap/engines/engine.py.224: train: epoch 61 | batch 2080/2304 [I 2022-10-27 10:13:46] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:04 [I 2022-10-27 10:13:46] ...trap/engines/engine.py.229: process: 0.54782 | load: 0.00245 [I 2022-10-27 10:13:46] ...trap/engines/engine.py.230: loss: 0.15604 [I 2022-10-27 10:13:46] ...optimizers/trijoint.py.151: total_norm: 14.642331 [I 2022-10-27 10:13:52] ...trap/engines/engine.py.224: train: epoch 61 | batch 2090/2304 [I 2022-10-27 10:13:52] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:01:58 [I 2022-10-27 10:13:52] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00249 [I 2022-10-27 10:13:52] ...trap/engines/engine.py.230: loss: 0.13286 [I 2022-10-27 10:13:52] ...optimizers/trijoint.py.151: total_norm: 13.946451 [I 2022-10-27 10:13:58] ...trap/engines/engine.py.224: train: epoch 61 | batch 2100/2304 [I 2022-10-27 10:13:58] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:01:53 [I 2022-10-27 10:13:58] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00242 [I 2022-10-27 10:13:58] ...trap/engines/engine.py.230: loss: 0.11939 [I 2022-10-27 10:13:58] ...optimizers/trijoint.py.151: total_norm: 18.822266 [I 2022-10-27 10:14:03] ...trap/engines/engine.py.224: train: epoch 61 | batch 2110/2304 [I 2022-10-27 10:14:03] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:47 [I 2022-10-27 10:14:03] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00243 [I 2022-10-27 10:14:03] ...trap/engines/engine.py.230: loss: 0.14084 [I 2022-10-27 10:14:03] ...optimizers/trijoint.py.151: total_norm: 18.516607 [I 2022-10-27 10:14:09] ...trap/engines/engine.py.224: train: epoch 61 | batch 2120/2304 [I 2022-10-27 10:14:09] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:43 [I 2022-10-27 10:14:09] ...trap/engines/engine.py.229: process: 0.57122 | load: 0.00243 [I 2022-10-27 10:14:09] ...trap/engines/engine.py.230: loss: 0.16957 [I 2022-10-27 10:14:09] ...optimizers/trijoint.py.151: total_norm: 13.268156 [I 2022-10-27 10:14:14] ...trap/engines/engine.py.224: train: epoch 61 | batch 2130/2304 [I 2022-10-27 10:14:14] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:37 [I 2022-10-27 10:14:14] ...trap/engines/engine.py.229: process: 0.56270 | load: 0.00259 [I 2022-10-27 10:14:14] ...trap/engines/engine.py.230: loss: 0.10062 [I 2022-10-27 10:14:14] ...optimizers/trijoint.py.151: total_norm: 7.489282 [I 2022-10-27 10:14:20] ...trap/engines/engine.py.224: train: epoch 61 | batch 2140/2304 [I 2022-10-27 10:14:20] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:32 [I 2022-10-27 10:14:20] ...trap/engines/engine.py.229: process: 0.56983 | load: 0.00252 [I 2022-10-27 10:14:20] ...trap/engines/engine.py.230: loss: 0.12215 [I 2022-10-27 10:14:20] ...optimizers/trijoint.py.151: total_norm: 11.087936 [I 2022-10-27 10:14:26] ...trap/engines/engine.py.224: train: epoch 61 | batch 2150/2304 [I 2022-10-27 10:14:26] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:26 [I 2022-10-27 10:14:26] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00250 [I 2022-10-27 10:14:26] ...trap/engines/engine.py.230: loss: 0.14472 [I 2022-10-27 10:14:26] ...optimizers/trijoint.py.151: total_norm: 57.286396 [I 2022-10-27 10:14:31] ...trap/engines/engine.py.224: train: epoch 61 | batch 2160/2304 [I 2022-10-27 10:14:31] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:19 [I 2022-10-27 10:14:31] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00246 [I 2022-10-27 10:14:31] ...trap/engines/engine.py.230: loss: 0.13881 [I 2022-10-27 10:14:31] ...optimizers/trijoint.py.151: total_norm: 11.494776 [I 2022-10-27 10:14:37] ...trap/engines/engine.py.224: train: epoch 61 | batch 2170/2304 [I 2022-10-27 10:14:37] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:14 [I 2022-10-27 10:14:37] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00246 [I 2022-10-27 10:14:37] ...trap/engines/engine.py.230: loss: 0.10728 [I 2022-10-27 10:14:37] ...optimizers/trijoint.py.151: total_norm: 8.687839 [I 2022-10-27 10:14:42] ...trap/engines/engine.py.224: train: epoch 61 | batch 2180/2304 [I 2022-10-27 10:14:42] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:09 [I 2022-10-27 10:14:42] ...trap/engines/engine.py.229: process: 0.55986 | load: 0.00254 [I 2022-10-27 10:14:42] ...trap/engines/engine.py.230: loss: 0.11909 [I 2022-10-27 10:14:42] ...optimizers/trijoint.py.151: total_norm: 14.517881 [I 2022-10-27 10:14:48] ...trap/engines/engine.py.224: train: epoch 61 | batch 2190/2304 [I 2022-10-27 10:14:48] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:03 [I 2022-10-27 10:14:48] ...trap/engines/engine.py.229: process: 0.54927 | load: 0.00248 [I 2022-10-27 10:14:48] ...trap/engines/engine.py.230: loss: 0.18958 [I 2022-10-27 10:14:48] ...optimizers/trijoint.py.151: total_norm: 19.350254 [I 2022-10-27 10:14:53] ...trap/engines/engine.py.224: train: epoch 61 | batch 2200/2304 [I 2022-10-27 10:14:53] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:00:58 [I 2022-10-27 10:14:53] ...trap/engines/engine.py.229: process: 0.56467 | load: 0.00246 [I 2022-10-27 10:14:53] ...trap/engines/engine.py.230: loss: 0.23211 [I 2022-10-27 10:14:53] ...optimizers/trijoint.py.151: total_norm: 31.053516 [I 2022-10-27 10:14:59] ...trap/engines/engine.py.224: train: epoch 61 | batch 2210/2304 [I 2022-10-27 10:14:59] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:00:52 [I 2022-10-27 10:14:59] ...trap/engines/engine.py.229: process: 0.55669 | load: 0.00255 [I 2022-10-27 10:14:59] ...trap/engines/engine.py.230: loss: 0.12676 [I 2022-10-27 10:14:59] ...optimizers/trijoint.py.151: total_norm: 9.060373 [I 2022-10-27 10:15:05] ...trap/engines/engine.py.224: train: epoch 61 | batch 2220/2304 [I 2022-10-27 10:15:05] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:00:46 [I 2022-10-27 10:15:05] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00253 [I 2022-10-27 10:15:05] ...trap/engines/engine.py.230: loss: 0.15767 [I 2022-10-27 10:15:05] ...optimizers/trijoint.py.151: total_norm: 14.420569 [I 2022-10-27 10:15:10] ...trap/engines/engine.py.224: train: epoch 61 | batch 2230/2304 [I 2022-10-27 10:15:10] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:41 [I 2022-10-27 10:15:10] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00244 [I 2022-10-27 10:15:10] ...trap/engines/engine.py.230: loss: 0.17716 [I 2022-10-27 10:15:10] ...optimizers/trijoint.py.151: total_norm: 15.349332 [I 2022-10-27 10:15:16] ...trap/engines/engine.py.224: train: epoch 61 | batch 2240/2304 [I 2022-10-27 10:15:16] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:35 [I 2022-10-27 10:15:16] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00246 [I 2022-10-27 10:15:16] ...trap/engines/engine.py.230: loss: 0.09111 [I 2022-10-27 10:15:16] ...optimizers/trijoint.py.151: total_norm: 14.141845 [I 2022-10-27 10:15:22] ...trap/engines/engine.py.224: train: epoch 61 | batch 2250/2304 [I 2022-10-27 10:15:22] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:30 [I 2022-10-27 10:15:22] ...trap/engines/engine.py.229: process: 0.54936 | load: 0.00249 [I 2022-10-27 10:15:22] ...trap/engines/engine.py.230: loss: 0.17561 [I 2022-10-27 10:15:22] ...optimizers/trijoint.py.151: total_norm: 18.067471 [I 2022-10-27 10:15:27] ...trap/engines/engine.py.224: train: epoch 61 | batch 2260/2304 [I 2022-10-27 10:15:27] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:24 [I 2022-10-27 10:15:27] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00254 [I 2022-10-27 10:15:27] ...trap/engines/engine.py.230: loss: 0.14835 [I 2022-10-27 10:15:27] ...optimizers/trijoint.py.151: total_norm: 18.652945 [I 2022-10-27 10:15:33] ...trap/engines/engine.py.224: train: epoch 61 | batch 2270/2304 [I 2022-10-27 10:15:33] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:19 [I 2022-10-27 10:15:33] ...trap/engines/engine.py.229: process: 0.56646 | load: 0.00252 [I 2022-10-27 10:15:33] ...trap/engines/engine.py.230: loss: 0.14284 [I 2022-10-27 10:15:33] ...optimizers/trijoint.py.151: total_norm: 29.210215 [I 2022-10-27 10:15:38] ...trap/engines/engine.py.224: train: epoch 61 | batch 2280/2304 [I 2022-10-27 10:15:38] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:13 [I 2022-10-27 10:15:38] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00246 [I 2022-10-27 10:15:38] ...trap/engines/engine.py.230: loss: 0.09679 [I 2022-10-27 10:15:38] ...optimizers/trijoint.py.151: total_norm: 11.305049 [I 2022-10-27 10:15:44] ...trap/engines/engine.py.224: train: epoch 61 | batch 2290/2304 [I 2022-10-27 10:15:44] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:07 [I 2022-10-27 10:15:44] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00246 [I 2022-10-27 10:15:44] ...trap/engines/engine.py.230: loss: 0.12337 [I 2022-10-27 10:15:44] ...optimizers/trijoint.py.151: total_norm: 13.003961 [I 2022-10-27 10:15:50] ...trap/engines/engine.py.224: train: epoch 61 | batch 2300/2304 [I 2022-10-27 10:15:50] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:02 [I 2022-10-27 10:15:50] ...trap/engines/engine.py.229: process: 0.55478 | load: 0.00016 [I 2022-10-27 10:15:50] ...trap/engines/engine.py.230: loss: 0.18386 [I 2022-10-27 10:15:50] ...optimizers/trijoint.py.151: total_norm: 16.212420 [I 2022-10-27 10:15:52] ...trap/engines/engine.py.224: train: epoch 61 | batch 2304/2304 [I 2022-10-27 10:15:52] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:00 [I 2022-10-27 10:15:52] ...trap/engines/engine.py.229: process: 0.54852 | load: 0.00014 [I 2022-10-27 10:15:52] ...trap/engines/engine.py.230: loss: 0.13605 [I 2022-10-27 10:15:52] ...optimizers/trijoint.py.151: total_norm: 14.553873 [S 2022-10-27 10:15:53] ...trap/engines/engine.py.129: train_epoch.epoch: 61 [S 2022-10-27 10:15:53] ...trap/engines/engine.py.129: train_epoch.loss: 0.15404117954591867 [I 2022-10-27 10:15:53] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 10:15:54] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 61 [I 2022-10-27 10:15:54] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 10:15:56] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 10:15:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 10:15:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 10:15:58] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 10:15:58] ...trap/engines/engine.py.321: eval: epoch 61 | batch 0/510 [I 2022-10-27 10:15:58] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:36:47 [I 2022-10-27 10:15:58] ...trap/engines/engine.py.326: process: 4.32852 | load: 3.03705 [I 2022-10-27 10:16:02] ...trap/engines/engine.py.321: eval: epoch 61 | batch 10/510 [I 2022-10-27 10:16:02] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:25 [I 2022-10-27 10:16:02] ...trap/engines/engine.py.326: process: 0.34665 | load: 0.00016 [I 2022-10-27 10:16:05] ...trap/engines/engine.py.321: eval: epoch 61 | batch 20/510 [I 2022-10-27 10:16:05] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:06 [I 2022-10-27 10:16:05] ...trap/engines/engine.py.326: process: 0.34000 | load: 0.00017 [I 2022-10-27 10:16:08] ...trap/engines/engine.py.321: eval: epoch 61 | batch 30/510 [I 2022-10-27 10:16:08] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:41 [I 2022-10-27 10:16:08] ...trap/engines/engine.py.326: process: 0.32710 | load: 0.00016 [I 2022-10-27 10:16:12] ...trap/engines/engine.py.321: eval: epoch 61 | batch 40/510 [I 2022-10-27 10:16:12] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:41 [I 2022-10-27 10:16:12] ...trap/engines/engine.py.326: process: 0.33767 | load: 0.00026 [I 2022-10-27 10:16:15] ...trap/engines/engine.py.321: eval: epoch 61 | batch 50/510 [I 2022-10-27 10:16:15] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:34 [I 2022-10-27 10:16:15] ...trap/engines/engine.py.326: process: 0.33346 | load: 0.00016 [I 2022-10-27 10:16:18] ...trap/engines/engine.py.321: eval: epoch 61 | batch 60/510 [I 2022-10-27 10:16:18] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:31 [I 2022-10-27 10:16:18] ...trap/engines/engine.py.326: process: 0.35402 | load: 0.00023 [I 2022-10-27 10:16:22] ...trap/engines/engine.py.321: eval: epoch 61 | batch 70/510 [I 2022-10-27 10:16:22] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:28 [I 2022-10-27 10:16:22] ...trap/engines/engine.py.326: process: 0.34423 | load: 0.00018 [I 2022-10-27 10:16:25] ...trap/engines/engine.py.321: eval: epoch 61 | batch 80/510 [I 2022-10-27 10:16:25] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:24 [I 2022-10-27 10:16:25] ...trap/engines/engine.py.326: process: 0.33672 | load: 0.00018 [I 2022-10-27 10:16:28] ...trap/engines/engine.py.321: eval: epoch 61 | batch 90/510 [I 2022-10-27 10:16:28] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:20 [I 2022-10-27 10:16:28] ...trap/engines/engine.py.326: process: 0.33664 | load: 0.00018 [I 2022-10-27 10:16:32] ...trap/engines/engine.py.321: eval: epoch 61 | batch 100/510 [I 2022-10-27 10:16:32] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:18 [I 2022-10-27 10:16:32] ...trap/engines/engine.py.326: process: 0.29258 | load: 0.00024 [I 2022-10-27 10:16:35] ...trap/engines/engine.py.321: eval: epoch 61 | batch 110/510 [I 2022-10-27 10:16:35] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:58 [I 2022-10-27 10:16:35] ...trap/engines/engine.py.326: process: 0.28229 | load: 0.00019 [I 2022-10-27 10:16:38] ...trap/engines/engine.py.321: eval: epoch 61 | batch 120/510 [I 2022-10-27 10:16:38] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:52 [I 2022-10-27 10:16:38] ...trap/engines/engine.py.326: process: 0.28718 | load: 0.00015 [I 2022-10-27 10:16:41] ...trap/engines/engine.py.321: eval: epoch 61 | batch 130/510 [I 2022-10-27 10:16:41] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:49 [I 2022-10-27 10:16:41] ...trap/engines/engine.py.326: process: 0.28920 | load: 0.00019 [I 2022-10-27 10:16:43] ...trap/engines/engine.py.321: eval: epoch 61 | batch 140/510 [I 2022-10-27 10:16:43] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:45 [I 2022-10-27 10:16:43] ...trap/engines/engine.py.326: process: 0.27899 | load: 0.00017 [I 2022-10-27 10:16:46] ...trap/engines/engine.py.321: eval: epoch 61 | batch 150/510 [I 2022-10-27 10:16:46] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:43 [I 2022-10-27 10:16:46] ...trap/engines/engine.py.326: process: 0.28432 | load: 0.00015 [I 2022-10-27 10:16:49] ...trap/engines/engine.py.321: eval: epoch 61 | batch 160/510 [I 2022-10-27 10:16:49] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:42 [I 2022-10-27 10:16:49] ...trap/engines/engine.py.326: process: 0.28766 | load: 0.00017 [I 2022-10-27 10:16:52] ...trap/engines/engine.py.321: eval: epoch 61 | batch 170/510 [I 2022-10-27 10:16:52] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:39 [I 2022-10-27 10:16:52] ...trap/engines/engine.py.326: process: 0.29886 | load: 0.00021 [I 2022-10-27 10:16:55] ...trap/engines/engine.py.321: eval: epoch 61 | batch 180/510 [I 2022-10-27 10:16:55] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:37 [I 2022-10-27 10:16:55] ...trap/engines/engine.py.326: process: 0.30891 | load: 0.00025 [I 2022-10-27 10:16:58] ...trap/engines/engine.py.321: eval: epoch 61 | batch 190/510 [I 2022-10-27 10:16:58] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:34 [I 2022-10-27 10:16:58] ...trap/engines/engine.py.326: process: 0.30286 | load: 0.00014 [I 2022-10-27 10:17:01] ...trap/engines/engine.py.321: eval: epoch 61 | batch 200/510 [I 2022-10-27 10:17:01] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:30 [I 2022-10-27 10:17:01] ...trap/engines/engine.py.326: process: 0.29309 | load: 0.00017 [I 2022-10-27 10:17:04] ...trap/engines/engine.py.321: eval: epoch 61 | batch 210/510 [I 2022-10-27 10:17:04] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 10:17:04] ...trap/engines/engine.py.326: process: 0.29317 | load: 0.00019 [I 2022-10-27 10:17:07] ...trap/engines/engine.py.321: eval: epoch 61 | batch 220/510 [I 2022-10-27 10:17:07] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 10:17:07] ...trap/engines/engine.py.326: process: 0.28880 | load: 0.00015 [I 2022-10-27 10:17:10] ...trap/engines/engine.py.321: eval: epoch 61 | batch 230/510 [I 2022-10-27 10:17:10] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 10:17:10] ...trap/engines/engine.py.326: process: 0.28993 | load: 0.00024 [I 2022-10-27 10:17:12] ...trap/engines/engine.py.321: eval: epoch 61 | batch 240/510 [I 2022-10-27 10:17:12] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 10:17:12] ...trap/engines/engine.py.326: process: 0.28908 | load: 0.00015 [I 2022-10-27 10:17:15] ...trap/engines/engine.py.321: eval: epoch 61 | batch 250/510 [I 2022-10-27 10:17:15] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 10:17:15] ...trap/engines/engine.py.326: process: 0.28713 | load: 0.00015 [I 2022-10-27 10:17:18] ...trap/engines/engine.py.321: eval: epoch 61 | batch 260/510 [I 2022-10-27 10:17:18] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 10:17:18] ...trap/engines/engine.py.326: process: 0.29127 | load: 0.00020 [I 2022-10-27 10:17:21] ...trap/engines/engine.py.321: eval: epoch 61 | batch 270/510 [I 2022-10-27 10:17:21] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 10:17:21] ...trap/engines/engine.py.326: process: 0.28840 | load: 0.00025 [I 2022-10-27 10:17:24] ...trap/engines/engine.py.321: eval: epoch 61 | batch 280/510 [I 2022-10-27 10:17:24] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:10 [I 2022-10-27 10:17:24] ...trap/engines/engine.py.326: process: 0.36963 | load: 0.00018 [I 2022-10-27 10:17:27] ...trap/engines/engine.py.321: eval: epoch 61 | batch 290/510 [I 2022-10-27 10:17:27] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 10:17:27] ...trap/engines/engine.py.326: process: 0.28799 | load: 0.00018 [I 2022-10-27 10:17:30] ...trap/engines/engine.py.321: eval: epoch 61 | batch 300/510 [I 2022-10-27 10:17:30] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 10:17:30] ...trap/engines/engine.py.326: process: 0.29463 | load: 0.00016 [I 2022-10-27 10:17:33] ...trap/engines/engine.py.321: eval: epoch 61 | batch 310/510 [I 2022-10-27 10:17:33] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 10:17:33] ...trap/engines/engine.py.326: process: 0.28444 | load: 0.00017 [I 2022-10-27 10:17:36] ...trap/engines/engine.py.321: eval: epoch 61 | batch 320/510 [I 2022-10-27 10:17:36] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:55 [I 2022-10-27 10:17:36] ...trap/engines/engine.py.326: process: 0.30146 | load: 0.00015 [I 2022-10-27 10:17:39] ...trap/engines/engine.py.321: eval: epoch 61 | batch 330/510 [I 2022-10-27 10:17:39] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:52 [I 2022-10-27 10:17:39] ...trap/engines/engine.py.326: process: 0.28810 | load: 0.00021 [I 2022-10-27 10:17:41] ...trap/engines/engine.py.321: eval: epoch 61 | batch 340/510 [I 2022-10-27 10:17:41] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-27 10:17:41] ...trap/engines/engine.py.326: process: 0.28598 | load: 0.00019 [I 2022-10-27 10:17:44] ...trap/engines/engine.py.321: eval: epoch 61 | batch 350/510 [I 2022-10-27 10:17:44] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 10:17:44] ...trap/engines/engine.py.326: process: 0.28737 | load: 0.00016 [I 2022-10-27 10:17:47] ...trap/engines/engine.py.321: eval: epoch 61 | batch 360/510 [I 2022-10-27 10:17:47] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 10:17:47] ...trap/engines/engine.py.326: process: 0.28450 | load: 0.00018 [I 2022-10-27 10:17:50] ...trap/engines/engine.py.321: eval: epoch 61 | batch 370/510 [I 2022-10-27 10:17:50] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 10:17:50] ...trap/engines/engine.py.326: process: 0.28275 | load: 0.00019 [I 2022-10-27 10:17:53] ...trap/engines/engine.py.321: eval: epoch 61 | batch 380/510 [I 2022-10-27 10:17:53] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 10:17:53] ...trap/engines/engine.py.326: process: 0.28607 | load: 0.00017 [I 2022-10-27 10:17:56] ...trap/engines/engine.py.321: eval: epoch 61 | batch 390/510 [I 2022-10-27 10:17:56] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 10:17:56] ...trap/engines/engine.py.326: process: 0.28399 | load: 0.00016 [I 2022-10-27 10:17:59] ...trap/engines/engine.py.321: eval: epoch 61 | batch 400/510 [I 2022-10-27 10:17:59] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:33 [I 2022-10-27 10:17:59] ...trap/engines/engine.py.326: process: 0.28421 | load: 0.00019 [I 2022-10-27 10:18:02] ...trap/engines/engine.py.321: eval: epoch 61 | batch 410/510 [I 2022-10-27 10:18:02] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:29 [I 2022-10-27 10:18:02] ...trap/engines/engine.py.326: process: 0.29580 | load: 0.00017 [I 2022-10-27 10:18:05] ...trap/engines/engine.py.321: eval: epoch 61 | batch 420/510 [I 2022-10-27 10:18:05] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 10:18:05] ...trap/engines/engine.py.326: process: 0.27993 | load: 0.00020 [I 2022-10-27 10:18:07] ...trap/engines/engine.py.321: eval: epoch 61 | batch 430/510 [I 2022-10-27 10:18:07] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 10:18:07] ...trap/engines/engine.py.326: process: 0.29120 | load: 0.00020 [I 2022-10-27 10:18:10] ...trap/engines/engine.py.321: eval: epoch 61 | batch 440/510 [I 2022-10-27 10:18:10] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 10:18:10] ...trap/engines/engine.py.326: process: 0.27932 | load: 0.00019 [I 2022-10-27 10:18:13] ...trap/engines/engine.py.321: eval: epoch 61 | batch 450/510 [I 2022-10-27 10:18:13] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 10:18:13] ...trap/engines/engine.py.326: process: 0.28988 | load: 0.00020 [I 2022-10-27 10:18:16] ...trap/engines/engine.py.321: eval: epoch 61 | batch 460/510 [I 2022-10-27 10:18:16] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 10:18:16] ...trap/engines/engine.py.326: process: 0.28235 | load: 0.00019 [I 2022-10-27 10:18:19] ...trap/engines/engine.py.321: eval: epoch 61 | batch 470/510 [I 2022-10-27 10:18:19] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 10:18:19] ...trap/engines/engine.py.326: process: 0.28061 | load: 0.00024 [I 2022-10-27 10:18:22] ...trap/engines/engine.py.321: eval: epoch 61 | batch 480/510 [I 2022-10-27 10:18:22] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 10:18:22] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00019 [I 2022-10-27 10:18:25] ...trap/engines/engine.py.321: eval: epoch 61 | batch 490/510 [I 2022-10-27 10:18:25] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 10:18:25] ...trap/engines/engine.py.326: process: 0.28476 | load: 0.00017 [I 2022-10-27 10:18:28] ...trap/engines/engine.py.321: eval: epoch 61 | batch 500/510 [I 2022-10-27 10:18:28] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 10:18:28] ...trap/engines/engine.py.326: process: 0.28704 | load: 0.00020 [I 2022-10-27 10:18:31] ...trap/engines/engine.py.321: eval: epoch 61 | batch 510/510 [I 2022-10-27 10:18:31] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 10:18:31] ...trap/engines/engine.py.326: process: 0.28601 | load: 0.00013 [S 2022-10-27 10:18:31] ...trap/engines/engine.py.132: eval_epoch.epoch: 61 [I 2022-10-27 10:18:31] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.707400 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.900100 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933200 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.696300 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.897300 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.930100 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013581 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009648 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008207 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016150 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011662 [S 2022-10-27 10:18:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009224 [I 2022-10-27 10:18:37] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 10:18:37] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 10:18:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 10:18:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 10:18:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 10:18:40] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 10:18:42] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 10:18:48] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 10:18:48] ...trap/engines/engine.py.163: Training model on trainset for epoch 62 [I 2022-10-27 10:18:48] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 10:18:50] ...trap/engines/engine.py.224: train: epoch 62 | batch 0/2304 [I 2022-10-27 10:18:50] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:47 [I 2022-10-27 10:18:50] ...trap/engines/engine.py.229: process: 2.65074 | load: 2.01623 [I 2022-10-27 10:18:50] ...trap/engines/engine.py.230: loss: 0.14205 [I 2022-10-27 10:18:50] ...optimizers/trijoint.py.151: total_norm: 13.723946 [I 2022-10-27 10:18:56] ...trap/engines/engine.py.224: train: epoch 62 | batch 10/2304 [I 2022-10-27 10:18:56] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:44 [I 2022-10-27 10:18:56] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00414 [I 2022-10-27 10:18:56] ...trap/engines/engine.py.230: loss: 0.16331 [I 2022-10-27 10:18:56] ...optimizers/trijoint.py.151: total_norm: 15.042089 [I 2022-10-27 10:19:01] ...trap/engines/engine.py.224: train: epoch 62 | batch 20/2304 [I 2022-10-27 10:19:01] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:04 [I 2022-10-27 10:19:01] ...trap/engines/engine.py.229: process: 0.54955 | load: 0.00337 [I 2022-10-27 10:19:01] ...trap/engines/engine.py.230: loss: 0.18205 [I 2022-10-27 10:19:01] ...optimizers/trijoint.py.151: total_norm: 18.224855 [I 2022-10-27 10:19:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 30/2304 [I 2022-10-27 10:19:07] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:23 [I 2022-10-27 10:19:07] ...trap/engines/engine.py.229: process: 0.57466 | load: 0.00288 [I 2022-10-27 10:19:07] ...trap/engines/engine.py.230: loss: 0.21005 [I 2022-10-27 10:19:07] ...optimizers/trijoint.py.151: total_norm: 18.699060 [I 2022-10-27 10:19:13] ...trap/engines/engine.py.224: train: epoch 62 | batch 40/2304 [I 2022-10-27 10:19:13] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:10 [I 2022-10-27 10:19:13] ...trap/engines/engine.py.229: process: 0.54634 | load: 0.00301 [I 2022-10-27 10:19:13] ...trap/engines/engine.py.230: loss: 0.11221 [I 2022-10-27 10:19:13] ...optimizers/trijoint.py.151: total_norm: 14.471120 [I 2022-10-27 10:19:18] ...trap/engines/engine.py.224: train: epoch 62 | batch 50/2304 [I 2022-10-27 10:19:18] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:52 [I 2022-10-27 10:19:18] ...trap/engines/engine.py.229: process: 0.55081 | load: 0.00296 [I 2022-10-27 10:19:18] ...trap/engines/engine.py.230: loss: 0.14930 [I 2022-10-27 10:19:18] ...optimizers/trijoint.py.151: total_norm: 17.376215 [I 2022-10-27 10:19:24] ...trap/engines/engine.py.224: train: epoch 62 | batch 60/2304 [I 2022-10-27 10:19:24] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:55 [I 2022-10-27 10:19:24] ...trap/engines/engine.py.229: process: 0.56532 | load: 0.00283 [I 2022-10-27 10:19:24] ...trap/engines/engine.py.230: loss: 0.13418 [I 2022-10-27 10:19:24] ...optimizers/trijoint.py.151: total_norm: 26.215902 [I 2022-10-27 10:19:30] ...trap/engines/engine.py.224: train: epoch 62 | batch 70/2304 [I 2022-10-27 10:19:30] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:48 [I 2022-10-27 10:19:30] ...trap/engines/engine.py.229: process: 0.56375 | load: 0.00253 [I 2022-10-27 10:19:30] ...trap/engines/engine.py.230: loss: 0.18784 [I 2022-10-27 10:19:30] ...optimizers/trijoint.py.151: total_norm: 14.378687 [I 2022-10-27 10:19:35] ...trap/engines/engine.py.224: train: epoch 62 | batch 80/2304 [I 2022-10-27 10:19:35] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:52 [I 2022-10-27 10:19:35] ...trap/engines/engine.py.229: process: 0.56048 | load: 0.00283 [I 2022-10-27 10:19:35] ...trap/engines/engine.py.230: loss: 0.17898 [I 2022-10-27 10:19:35] ...optimizers/trijoint.py.151: total_norm: 13.028240 [I 2022-10-27 10:19:41] ...trap/engines/engine.py.224: train: epoch 62 | batch 90/2304 [I 2022-10-27 10:19:41] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:29 [I 2022-10-27 10:19:41] ...trap/engines/engine.py.229: process: 0.54848 | load: 0.00277 [I 2022-10-27 10:19:41] ...trap/engines/engine.py.230: loss: 0.14381 [I 2022-10-27 10:19:41] ...optimizers/trijoint.py.151: total_norm: 31.551699 [I 2022-10-27 10:19:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 100/2304 [I 2022-10-27 10:19:46] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:24 [I 2022-10-27 10:19:46] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00268 [I 2022-10-27 10:19:46] ...trap/engines/engine.py.230: loss: 0.16462 [I 2022-10-27 10:19:46] ...optimizers/trijoint.py.151: total_norm: 17.023336 [I 2022-10-27 10:19:52] ...trap/engines/engine.py.224: train: epoch 62 | batch 110/2304 [I 2022-10-27 10:19:52] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:37 [I 2022-10-27 10:19:52] ...trap/engines/engine.py.229: process: 0.57654 | load: 0.00276 [I 2022-10-27 10:19:52] ...trap/engines/engine.py.230: loss: 0.13138 [I 2022-10-27 10:19:52] ...optimizers/trijoint.py.151: total_norm: 19.059704 [I 2022-10-27 10:19:58] ...trap/engines/engine.py.224: train: epoch 62 | batch 120/2304 [I 2022-10-27 10:19:58] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:19 [I 2022-10-27 10:19:58] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00246 [I 2022-10-27 10:19:58] ...trap/engines/engine.py.230: loss: 0.11554 [I 2022-10-27 10:19:58] ...optimizers/trijoint.py.151: total_norm: 28.677660 [I 2022-10-27 10:20:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 130/2304 [I 2022-10-27 10:20:03] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:17 [I 2022-10-27 10:20:03] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00259 [I 2022-10-27 10:20:03] ...trap/engines/engine.py.230: loss: 0.10928 [I 2022-10-27 10:20:03] ...optimizers/trijoint.py.151: total_norm: 10.447481 [I 2022-10-27 10:20:09] ...trap/engines/engine.py.224: train: epoch 62 | batch 140/2304 [I 2022-10-27 10:20:09] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:03 [I 2022-10-27 10:20:09] ...trap/engines/engine.py.229: process: 0.54473 | load: 0.00257 [I 2022-10-27 10:20:09] ...trap/engines/engine.py.230: loss: 0.13964 [I 2022-10-27 10:20:09] ...optimizers/trijoint.py.151: total_norm: 18.713923 [I 2022-10-27 10:20:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 150/2304 [I 2022-10-27 10:20:14] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:01 [I 2022-10-27 10:20:14] ...trap/engines/engine.py.229: process: 0.55868 | load: 0.00261 [I 2022-10-27 10:20:14] ...trap/engines/engine.py.230: loss: 0.16619 [I 2022-10-27 10:20:14] ...optimizers/trijoint.py.151: total_norm: 12.853663 [I 2022-10-27 10:20:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 160/2304 [I 2022-10-27 10:20:20] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:04 [I 2022-10-27 10:20:20] ...trap/engines/engine.py.229: process: 0.56456 | load: 0.00266 [I 2022-10-27 10:20:20] ...trap/engines/engine.py.230: loss: 0.14460 [I 2022-10-27 10:20:20] ...optimizers/trijoint.py.151: total_norm: 33.721573 [I 2022-10-27 10:20:26] ...trap/engines/engine.py.224: train: epoch 62 | batch 170/2304 [I 2022-10-27 10:20:26] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:20:00 [I 2022-10-27 10:20:26] ...trap/engines/engine.py.229: process: 0.56564 | load: 0.00252 [I 2022-10-27 10:20:26] ...trap/engines/engine.py.230: loss: 0.12641 [I 2022-10-27 10:20:26] ...optimizers/trijoint.py.151: total_norm: 10.538051 [I 2022-10-27 10:20:31] ...trap/engines/engine.py.224: train: epoch 62 | batch 180/2304 [I 2022-10-27 10:20:31] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:51 [I 2022-10-27 10:20:31] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00275 [I 2022-10-27 10:20:31] ...trap/engines/engine.py.230: loss: 0.16987 [I 2022-10-27 10:20:31] ...optimizers/trijoint.py.151: total_norm: 20.820095 [I 2022-10-27 10:20:37] ...trap/engines/engine.py.224: train: epoch 62 | batch 190/2304 [I 2022-10-27 10:20:37] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:38 [I 2022-10-27 10:20:37] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00258 [I 2022-10-27 10:20:37] ...trap/engines/engine.py.230: loss: 0.20447 [I 2022-10-27 10:20:37] ...optimizers/trijoint.py.151: total_norm: 13.318764 [I 2022-10-27 10:20:43] ...trap/engines/engine.py.224: train: epoch 62 | batch 200/2304 [I 2022-10-27 10:20:43] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:39 [I 2022-10-27 10:20:43] ...trap/engines/engine.py.229: process: 0.55632 | load: 0.00257 [I 2022-10-27 10:20:43] ...trap/engines/engine.py.230: loss: 0.17095 [I 2022-10-27 10:20:43] ...optimizers/trijoint.py.151: total_norm: 17.942698 [I 2022-10-27 10:20:48] ...trap/engines/engine.py.224: train: epoch 62 | batch 210/2304 [I 2022-10-27 10:20:48] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:22 [I 2022-10-27 10:20:48] ...trap/engines/engine.py.229: process: 0.53879 | load: 0.00276 [I 2022-10-27 10:20:48] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-27 10:20:48] ...optimizers/trijoint.py.151: total_norm: 7.976187 [I 2022-10-27 10:20:54] ...trap/engines/engine.py.224: train: epoch 62 | batch 220/2304 [I 2022-10-27 10:20:54] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:25 [I 2022-10-27 10:20:54] ...trap/engines/engine.py.229: process: 0.54836 | load: 0.00254 [I 2022-10-27 10:20:54] ...trap/engines/engine.py.230: loss: 0.13000 [I 2022-10-27 10:20:54] ...optimizers/trijoint.py.151: total_norm: 8.314580 [I 2022-10-27 10:20:59] ...trap/engines/engine.py.224: train: epoch 62 | batch 230/2304 [I 2022-10-27 10:20:59] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:22 [I 2022-10-27 10:20:59] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00286 [I 2022-10-27 10:20:59] ...trap/engines/engine.py.230: loss: 0.13044 [I 2022-10-27 10:20:59] ...optimizers/trijoint.py.151: total_norm: 16.180706 [I 2022-10-27 10:21:05] ...trap/engines/engine.py.224: train: epoch 62 | batch 240/2304 [I 2022-10-27 10:21:05] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:01 [I 2022-10-27 10:21:05] ...trap/engines/engine.py.229: process: 0.54501 | load: 0.00259 [I 2022-10-27 10:21:05] ...trap/engines/engine.py.230: loss: 0.14941 [I 2022-10-27 10:21:05] ...optimizers/trijoint.py.151: total_norm: 13.016950 [I 2022-10-27 10:21:11] ...trap/engines/engine.py.224: train: epoch 62 | batch 250/2304 [I 2022-10-27 10:21:11] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:01 [I 2022-10-27 10:21:11] ...trap/engines/engine.py.229: process: 0.56834 | load: 0.00258 [I 2022-10-27 10:21:11] ...trap/engines/engine.py.230: loss: 0.14410 [I 2022-10-27 10:21:11] ...optimizers/trijoint.py.151: total_norm: 20.186163 [I 2022-10-27 10:21:16] ...trap/engines/engine.py.224: train: epoch 62 | batch 260/2304 [I 2022-10-27 10:21:16] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:11 [I 2022-10-27 10:21:16] ...trap/engines/engine.py.229: process: 0.58623 | load: 0.00250 [I 2022-10-27 10:21:16] ...trap/engines/engine.py.230: loss: 0.10001 [I 2022-10-27 10:21:16] ...optimizers/trijoint.py.151: total_norm: 9.491367 [I 2022-10-27 10:21:22] ...trap/engines/engine.py.224: train: epoch 62 | batch 270/2304 [I 2022-10-27 10:21:22] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:57 [I 2022-10-27 10:21:22] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00312 [I 2022-10-27 10:21:22] ...trap/engines/engine.py.230: loss: 0.16515 [I 2022-10-27 10:21:22] ...optimizers/trijoint.py.151: total_norm: 22.345106 [I 2022-10-27 10:21:27] ...trap/engines/engine.py.224: train: epoch 62 | batch 280/2304 [I 2022-10-27 10:21:27] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:48 [I 2022-10-27 10:21:27] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00270 [I 2022-10-27 10:21:27] ...trap/engines/engine.py.230: loss: 0.14492 [I 2022-10-27 10:21:27] ...optimizers/trijoint.py.151: total_norm: 14.040023 [I 2022-10-27 10:21:33] ...trap/engines/engine.py.224: train: epoch 62 | batch 290/2304 [I 2022-10-27 10:21:33] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:47 [I 2022-10-27 10:21:33] ...trap/engines/engine.py.229: process: 0.53603 | load: 0.00247 [I 2022-10-27 10:21:33] ...trap/engines/engine.py.230: loss: 0.13501 [I 2022-10-27 10:21:33] ...optimizers/trijoint.py.151: total_norm: 19.324560 [I 2022-10-27 10:21:39] ...trap/engines/engine.py.224: train: epoch 62 | batch 300/2304 [I 2022-10-27 10:21:39] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:39 [I 2022-10-27 10:21:39] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00247 [I 2022-10-27 10:21:39] ...trap/engines/engine.py.230: loss: 0.12066 [I 2022-10-27 10:21:39] ...optimizers/trijoint.py.151: total_norm: 16.182016 [I 2022-10-27 10:21:44] ...trap/engines/engine.py.224: train: epoch 62 | batch 310/2304 [I 2022-10-27 10:21:44] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:33 [I 2022-10-27 10:21:44] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00249 [I 2022-10-27 10:21:44] ...trap/engines/engine.py.230: loss: 0.12186 [I 2022-10-27 10:21:44] ...optimizers/trijoint.py.151: total_norm: 18.311464 [I 2022-10-27 10:21:50] ...trap/engines/engine.py.224: train: epoch 62 | batch 320/2304 [I 2022-10-27 10:21:50] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:40 [I 2022-10-27 10:21:50] ...trap/engines/engine.py.229: process: 0.56484 | load: 0.00260 [I 2022-10-27 10:21:50] ...trap/engines/engine.py.230: loss: 0.13591 [I 2022-10-27 10:21:50] ...optimizers/trijoint.py.151: total_norm: 14.538854 [I 2022-10-27 10:21:55] ...trap/engines/engine.py.224: train: epoch 62 | batch 330/2304 [I 2022-10-27 10:21:55] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:28 [I 2022-10-27 10:21:55] ...trap/engines/engine.py.229: process: 0.56155 | load: 0.00246 [I 2022-10-27 10:21:55] ...trap/engines/engine.py.230: loss: 0.13889 [I 2022-10-27 10:21:55] ...optimizers/trijoint.py.151: total_norm: 15.559281 [I 2022-10-27 10:22:01] ...trap/engines/engine.py.224: train: epoch 62 | batch 340/2304 [I 2022-10-27 10:22:01] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:23 [I 2022-10-27 10:22:01] ...trap/engines/engine.py.229: process: 0.56843 | load: 0.00267 [I 2022-10-27 10:22:01] ...trap/engines/engine.py.230: loss: 0.11884 [I 2022-10-27 10:22:01] ...optimizers/trijoint.py.151: total_norm: 16.380472 [I 2022-10-27 10:22:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 350/2304 [I 2022-10-27 10:22:07] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:08 [I 2022-10-27 10:22:07] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00252 [I 2022-10-27 10:22:07] ...trap/engines/engine.py.230: loss: 0.13934 [I 2022-10-27 10:22:07] ...optimizers/trijoint.py.151: total_norm: 17.028292 [I 2022-10-27 10:22:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 360/2304 [I 2022-10-27 10:22:12] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:02 [I 2022-10-27 10:22:12] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00244 [I 2022-10-27 10:22:12] ...trap/engines/engine.py.230: loss: 0.11116 [I 2022-10-27 10:22:12] ...optimizers/trijoint.py.151: total_norm: 10.993022 [I 2022-10-27 10:22:18] ...trap/engines/engine.py.224: train: epoch 62 | batch 370/2304 [I 2022-10-27 10:22:18] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:17:58 [I 2022-10-27 10:22:18] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00242 [I 2022-10-27 10:22:18] ...trap/engines/engine.py.230: loss: 0.14913 [I 2022-10-27 10:22:18] ...optimizers/trijoint.py.151: total_norm: 8.527431 [I 2022-10-27 10:22:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 380/2304 [I 2022-10-27 10:22:23] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:51 [I 2022-10-27 10:22:23] ...trap/engines/engine.py.229: process: 0.55000 | load: 0.00242 [I 2022-10-27 10:22:23] ...trap/engines/engine.py.230: loss: 0.10634 [I 2022-10-27 10:22:23] ...optimizers/trijoint.py.151: total_norm: 7.062266 [I 2022-10-27 10:22:29] ...trap/engines/engine.py.224: train: epoch 62 | batch 390/2304 [I 2022-10-27 10:22:29] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:55 [I 2022-10-27 10:22:29] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00253 [I 2022-10-27 10:22:29] ...trap/engines/engine.py.230: loss: 0.11041 [I 2022-10-27 10:22:29] ...optimizers/trijoint.py.151: total_norm: 8.407446 [I 2022-10-27 10:22:35] ...trap/engines/engine.py.224: train: epoch 62 | batch 400/2304 [I 2022-10-27 10:22:35] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:42 [I 2022-10-27 10:22:35] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00245 [I 2022-10-27 10:22:35] ...trap/engines/engine.py.230: loss: 0.11813 [I 2022-10-27 10:22:35] ...optimizers/trijoint.py.151: total_norm: 10.033020 [I 2022-10-27 10:22:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 410/2304 [I 2022-10-27 10:22:40] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:38 [I 2022-10-27 10:22:40] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00254 [I 2022-10-27 10:22:40] ...trap/engines/engine.py.230: loss: 0.11894 [I 2022-10-27 10:22:40] ...optimizers/trijoint.py.151: total_norm: 11.336815 [I 2022-10-27 10:22:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 420/2304 [I 2022-10-27 10:22:46] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:32 [I 2022-10-27 10:22:46] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00249 [I 2022-10-27 10:22:46] ...trap/engines/engine.py.230: loss: 0.13084 [I 2022-10-27 10:22:46] ...optimizers/trijoint.py.151: total_norm: 14.605490 [I 2022-10-27 10:22:51] ...trap/engines/engine.py.224: train: epoch 62 | batch 430/2304 [I 2022-10-27 10:22:51] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:27 [I 2022-10-27 10:22:51] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00244 [I 2022-10-27 10:22:51] ...trap/engines/engine.py.230: loss: 0.11511 [I 2022-10-27 10:22:51] ...optimizers/trijoint.py.151: total_norm: 15.774162 [I 2022-10-27 10:22:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 440/2304 [I 2022-10-27 10:22:57] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:18 [I 2022-10-27 10:22:57] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00243 [I 2022-10-27 10:22:57] ...trap/engines/engine.py.230: loss: 0.16772 [I 2022-10-27 10:22:57] ...optimizers/trijoint.py.151: total_norm: 13.550958 [I 2022-10-27 10:23:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 450/2304 [I 2022-10-27 10:23:03] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:24 [I 2022-10-27 10:23:03] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00255 [I 2022-10-27 10:23:03] ...trap/engines/engine.py.230: loss: 0.15661 [I 2022-10-27 10:23:03] ...optimizers/trijoint.py.151: total_norm: 16.799566 [I 2022-10-27 10:23:08] ...trap/engines/engine.py.224: train: epoch 62 | batch 460/2304 [I 2022-10-27 10:23:08] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:12 [I 2022-10-27 10:23:08] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00249 [I 2022-10-27 10:23:08] ...trap/engines/engine.py.230: loss: 0.12484 [I 2022-10-27 10:23:08] ...optimizers/trijoint.py.151: total_norm: 19.002153 [I 2022-10-27 10:23:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 470/2304 [I 2022-10-27 10:23:14] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:01 [I 2022-10-27 10:23:14] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00248 [I 2022-10-27 10:23:14] ...trap/engines/engine.py.230: loss: 0.17723 [I 2022-10-27 10:23:14] ...optimizers/trijoint.py.151: total_norm: 10.482069 [I 2022-10-27 10:23:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 480/2304 [I 2022-10-27 10:23:20] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:56 [I 2022-10-27 10:23:20] ...trap/engines/engine.py.229: process: 0.56883 | load: 0.00243 [I 2022-10-27 10:23:20] ...trap/engines/engine.py.230: loss: 0.14326 [I 2022-10-27 10:23:20] ...optimizers/trijoint.py.151: total_norm: 19.718147 [I 2022-10-27 10:23:25] ...trap/engines/engine.py.224: train: epoch 62 | batch 490/2304 [I 2022-10-27 10:23:25] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:47 [I 2022-10-27 10:23:25] ...trap/engines/engine.py.229: process: 0.54825 | load: 0.00242 [I 2022-10-27 10:23:25] ...trap/engines/engine.py.230: loss: 0.13447 [I 2022-10-27 10:23:25] ...optimizers/trijoint.py.151: total_norm: 16.915131 [I 2022-10-27 10:23:31] ...trap/engines/engine.py.224: train: epoch 62 | batch 500/2304 [I 2022-10-27 10:23:31] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:50 [I 2022-10-27 10:23:31] ...trap/engines/engine.py.229: process: 0.56418 | load: 0.00245 [I 2022-10-27 10:23:31] ...trap/engines/engine.py.230: loss: 0.20210 [I 2022-10-27 10:23:31] ...optimizers/trijoint.py.151: total_norm: 13.184299 [I 2022-10-27 10:23:36] ...trap/engines/engine.py.224: train: epoch 62 | batch 510/2304 [I 2022-10-27 10:23:36] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:46 [I 2022-10-27 10:23:36] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00244 [I 2022-10-27 10:23:36] ...trap/engines/engine.py.230: loss: 0.17861 [I 2022-10-27 10:23:36] ...optimizers/trijoint.py.151: total_norm: 19.856890 [I 2022-10-27 10:23:42] ...trap/engines/engine.py.224: train: epoch 62 | batch 520/2304 [I 2022-10-27 10:23:42] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:36 [I 2022-10-27 10:23:42] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00274 [I 2022-10-27 10:23:42] ...trap/engines/engine.py.230: loss: 0.16270 [I 2022-10-27 10:23:42] ...optimizers/trijoint.py.151: total_norm: 24.311451 [I 2022-10-27 10:23:48] ...trap/engines/engine.py.224: train: epoch 62 | batch 530/2304 [I 2022-10-27 10:23:48] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:32 [I 2022-10-27 10:23:48] ...trap/engines/engine.py.229: process: 0.55632 | load: 0.00253 [I 2022-10-27 10:23:48] ...trap/engines/engine.py.230: loss: 0.11765 [I 2022-10-27 10:23:48] ...optimizers/trijoint.py.151: total_norm: 16.532324 [I 2022-10-27 10:23:53] ...trap/engines/engine.py.224: train: epoch 62 | batch 540/2304 [I 2022-10-27 10:23:53] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:22 [I 2022-10-27 10:23:53] ...trap/engines/engine.py.229: process: 0.54982 | load: 0.00251 [I 2022-10-27 10:23:53] ...trap/engines/engine.py.230: loss: 0.12182 [I 2022-10-27 10:23:53] ...optimizers/trijoint.py.151: total_norm: 15.810650 [I 2022-10-27 10:23:59] ...trap/engines/engine.py.224: train: epoch 62 | batch 550/2304 [I 2022-10-27 10:23:59] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:24 [I 2022-10-27 10:23:59] ...trap/engines/engine.py.229: process: 0.56188 | load: 0.00252 [I 2022-10-27 10:23:59] ...trap/engines/engine.py.230: loss: 0.16532 [I 2022-10-27 10:23:59] ...optimizers/trijoint.py.151: total_norm: 8.296007 [I 2022-10-27 10:24:04] ...trap/engines/engine.py.224: train: epoch 62 | batch 560/2304 [I 2022-10-27 10:24:04] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:17 [I 2022-10-27 10:24:04] ...trap/engines/engine.py.229: process: 0.56607 | load: 0.00248 [I 2022-10-27 10:24:04] ...trap/engines/engine.py.230: loss: 0.18125 [I 2022-10-27 10:24:04] ...optimizers/trijoint.py.151: total_norm: 48.753269 [I 2022-10-27 10:24:10] ...trap/engines/engine.py.224: train: epoch 62 | batch 570/2304 [I 2022-10-27 10:24:10] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:07 [I 2022-10-27 10:24:10] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00245 [I 2022-10-27 10:24:10] ...trap/engines/engine.py.230: loss: 0.14908 [I 2022-10-27 10:24:10] ...optimizers/trijoint.py.151: total_norm: 31.726259 [I 2022-10-27 10:24:16] ...trap/engines/engine.py.224: train: epoch 62 | batch 580/2304 [I 2022-10-27 10:24:16] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:59 [I 2022-10-27 10:24:16] ...trap/engines/engine.py.229: process: 0.56183 | load: 0.00257 [I 2022-10-27 10:24:16] ...trap/engines/engine.py.230: loss: 0.13012 [I 2022-10-27 10:24:16] ...optimizers/trijoint.py.151: total_norm: 10.722518 [I 2022-10-27 10:24:21] ...trap/engines/engine.py.224: train: epoch 62 | batch 590/2304 [I 2022-10-27 10:24:21] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:54 [I 2022-10-27 10:24:21] ...trap/engines/engine.py.229: process: 0.55258 | load: 0.00250 [I 2022-10-27 10:24:21] ...trap/engines/engine.py.230: loss: 0.14802 [I 2022-10-27 10:24:21] ...optimizers/trijoint.py.151: total_norm: 13.395173 [I 2022-10-27 10:24:27] ...trap/engines/engine.py.224: train: epoch 62 | batch 600/2304 [I 2022-10-27 10:24:27] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:44 [I 2022-10-27 10:24:27] ...trap/engines/engine.py.229: process: 0.55491 | load: 0.00248 [I 2022-10-27 10:24:27] ...trap/engines/engine.py.230: loss: 0.18860 [I 2022-10-27 10:24:27] ...optimizers/trijoint.py.151: total_norm: 19.941557 [I 2022-10-27 10:24:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 610/2304 [I 2022-10-27 10:24:32] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:56 [I 2022-10-27 10:24:32] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00248 [I 2022-10-27 10:24:32] ...trap/engines/engine.py.230: loss: 0.14001 [I 2022-10-27 10:24:32] ...optimizers/trijoint.py.151: total_norm: 14.400456 [I 2022-10-27 10:24:38] ...trap/engines/engine.py.224: train: epoch 62 | batch 620/2304 [I 2022-10-27 10:24:38] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:45 [I 2022-10-27 10:24:38] ...trap/engines/engine.py.229: process: 0.54769 | load: 0.00243 [I 2022-10-27 10:24:38] ...trap/engines/engine.py.230: loss: 0.18186 [I 2022-10-27 10:24:38] ...optimizers/trijoint.py.151: total_norm: 25.714390 [I 2022-10-27 10:24:44] ...trap/engines/engine.py.224: train: epoch 62 | batch 630/2304 [I 2022-10-27 10:24:44] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:47 [I 2022-10-27 10:24:44] ...trap/engines/engine.py.229: process: 0.56926 | load: 0.00245 [I 2022-10-27 10:24:44] ...trap/engines/engine.py.230: loss: 0.26007 [I 2022-10-27 10:24:44] ...optimizers/trijoint.py.151: total_norm: 30.624428 [I 2022-10-27 10:24:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 640/2304 [I 2022-10-27 10:24:49] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:37 [I 2022-10-27 10:24:49] ...trap/engines/engine.py.229: process: 0.56511 | load: 0.00248 [I 2022-10-27 10:24:49] ...trap/engines/engine.py.230: loss: 0.11811 [I 2022-10-27 10:24:49] ...optimizers/trijoint.py.151: total_norm: 19.951195 [I 2022-10-27 10:24:55] ...trap/engines/engine.py.224: train: epoch 62 | batch 650/2304 [I 2022-10-27 10:24:55] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:26 [I 2022-10-27 10:24:55] ...trap/engines/engine.py.229: process: 0.56233 | load: 0.00243 [I 2022-10-27 10:24:55] ...trap/engines/engine.py.230: loss: 0.19471 [I 2022-10-27 10:24:55] ...optimizers/trijoint.py.151: total_norm: 20.697691 [I 2022-10-27 10:25:01] ...trap/engines/engine.py.224: train: epoch 62 | batch 660/2304 [I 2022-10-27 10:25:01] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:27 [I 2022-10-27 10:25:01] ...trap/engines/engine.py.229: process: 0.57064 | load: 0.00243 [I 2022-10-27 10:25:01] ...trap/engines/engine.py.230: loss: 0.14019 [I 2022-10-27 10:25:01] ...optimizers/trijoint.py.151: total_norm: 13.475968 [I 2022-10-27 10:25:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 670/2304 [I 2022-10-27 10:25:06] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:20 [I 2022-10-27 10:25:06] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00251 [I 2022-10-27 10:25:06] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-27 10:25:06] ...optimizers/trijoint.py.151: total_norm: 19.728403 [I 2022-10-27 10:25:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 680/2304 [I 2022-10-27 10:25:12] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:09 [I 2022-10-27 10:25:12] ...trap/engines/engine.py.229: process: 0.54978 | load: 0.00247 [I 2022-10-27 10:25:12] ...trap/engines/engine.py.230: loss: 0.29343 [I 2022-10-27 10:25:12] ...optimizers/trijoint.py.151: total_norm: 17.316702 [I 2022-10-27 10:25:17] ...trap/engines/engine.py.224: train: epoch 62 | batch 690/2304 [I 2022-10-27 10:25:17] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:14:57 [I 2022-10-27 10:25:17] ...trap/engines/engine.py.229: process: 0.55176 | load: 0.00246 [I 2022-10-27 10:25:17] ...trap/engines/engine.py.230: loss: 0.10387 [I 2022-10-27 10:25:17] ...optimizers/trijoint.py.151: total_norm: 11.318720 [I 2022-10-27 10:25:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 700/2304 [I 2022-10-27 10:25:23] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:14:57 [I 2022-10-27 10:25:23] ...trap/engines/engine.py.229: process: 0.55897 | load: 0.00242 [I 2022-10-27 10:25:23] ...trap/engines/engine.py.230: loss: 0.13541 [I 2022-10-27 10:25:23] ...optimizers/trijoint.py.151: total_norm: 23.358667 [I 2022-10-27 10:25:29] ...trap/engines/engine.py.224: train: epoch 62 | batch 710/2304 [I 2022-10-27 10:25:29] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:55 [I 2022-10-27 10:25:29] ...trap/engines/engine.py.229: process: 0.56939 | load: 0.00245 [I 2022-10-27 10:25:29] ...trap/engines/engine.py.230: loss: 0.13400 [I 2022-10-27 10:25:29] ...optimizers/trijoint.py.151: total_norm: 20.214361 [I 2022-10-27 10:25:34] ...trap/engines/engine.py.224: train: epoch 62 | batch 720/2304 [I 2022-10-27 10:25:34] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:43 [I 2022-10-27 10:25:34] ...trap/engines/engine.py.229: process: 0.54817 | load: 0.00242 [I 2022-10-27 10:25:34] ...trap/engines/engine.py.230: loss: 0.16066 [I 2022-10-27 10:25:34] ...optimizers/trijoint.py.151: total_norm: 15.010299 [I 2022-10-27 10:25:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 730/2304 [I 2022-10-27 10:25:40] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:14:41 [I 2022-10-27 10:25:40] ...trap/engines/engine.py.229: process: 0.55366 | load: 0.00243 [I 2022-10-27 10:25:40] ...trap/engines/engine.py.230: loss: 0.12210 [I 2022-10-27 10:25:40] ...optimizers/trijoint.py.151: total_norm: 18.864328 [I 2022-10-27 10:25:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 740/2304 [I 2022-10-27 10:25:46] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:41 [I 2022-10-27 10:25:46] ...trap/engines/engine.py.229: process: 0.56492 | load: 0.00247 [I 2022-10-27 10:25:46] ...trap/engines/engine.py.230: loss: 0.14843 [I 2022-10-27 10:25:46] ...optimizers/trijoint.py.151: total_norm: 18.591684 [I 2022-10-27 10:25:51] ...trap/engines/engine.py.224: train: epoch 62 | batch 750/2304 [I 2022-10-27 10:25:51] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:52 [I 2022-10-27 10:25:51] ...trap/engines/engine.py.229: process: 0.53836 | load: 0.00256 [I 2022-10-27 10:25:51] ...trap/engines/engine.py.230: loss: 0.18994 [I 2022-10-27 10:25:51] ...optimizers/trijoint.py.151: total_norm: 12.847925 [I 2022-10-27 10:25:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 760/2304 [I 2022-10-27 10:25:57] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:28 [I 2022-10-27 10:25:57] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00249 [I 2022-10-27 10:25:57] ...trap/engines/engine.py.230: loss: 0.14061 [I 2022-10-27 10:25:57] ...optimizers/trijoint.py.151: total_norm: 19.158821 [I 2022-10-27 10:26:02] ...trap/engines/engine.py.224: train: epoch 62 | batch 770/2304 [I 2022-10-27 10:26:02] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:21 [I 2022-10-27 10:26:02] ...trap/engines/engine.py.229: process: 0.57678 | load: 0.00246 [I 2022-10-27 10:26:02] ...trap/engines/engine.py.230: loss: 0.19809 [I 2022-10-27 10:26:02] ...optimizers/trijoint.py.151: total_norm: 21.443447 [I 2022-10-27 10:26:08] ...trap/engines/engine.py.224: train: epoch 62 | batch 780/2304 [I 2022-10-27 10:26:08] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:20 [I 2022-10-27 10:26:08] ...trap/engines/engine.py.229: process: 0.54944 | load: 0.00242 [I 2022-10-27 10:26:08] ...trap/engines/engine.py.230: loss: 0.12936 [I 2022-10-27 10:26:08] ...optimizers/trijoint.py.151: total_norm: 12.425167 [I 2022-10-27 10:26:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 790/2304 [I 2022-10-27 10:26:32] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:29:53 [I 2022-10-27 10:26:32] ...trap/engines/engine.py.229: process: 0.55437 | load: 0.00236 [I 2022-10-27 10:26:32] ...trap/engines/engine.py.230: loss: 0.15050 [I 2022-10-27 10:26:32] ...optimizers/trijoint.py.151: total_norm: 24.888481 [I 2022-10-27 10:26:38] ...trap/engines/engine.py.224: train: epoch 62 | batch 800/2304 [I 2022-10-27 10:26:38] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:15:56 [I 2022-10-27 10:26:38] ...trap/engines/engine.py.229: process: 0.55204 | load: 0.00235 [I 2022-10-27 10:26:38] ...trap/engines/engine.py.230: loss: 0.13573 [I 2022-10-27 10:26:38] ...optimizers/trijoint.py.151: total_norm: 23.271433 [I 2022-10-27 10:26:44] ...trap/engines/engine.py.224: train: epoch 62 | batch 810/2304 [I 2022-10-27 10:26:44] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:14:06 [I 2022-10-27 10:26:44] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00236 [I 2022-10-27 10:26:44] ...trap/engines/engine.py.230: loss: 0.10841 [I 2022-10-27 10:26:44] ...optimizers/trijoint.py.151: total_norm: 11.030629 [I 2022-10-27 10:26:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 820/2304 [I 2022-10-27 10:26:49] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:13:48 [I 2022-10-27 10:26:49] ...trap/engines/engine.py.229: process: 0.55450 | load: 0.00244 [I 2022-10-27 10:26:49] ...trap/engines/engine.py.230: loss: 0.13698 [I 2022-10-27 10:26:49] ...optimizers/trijoint.py.151: total_norm: 15.169885 [I 2022-10-27 10:26:55] ...trap/engines/engine.py.224: train: epoch 62 | batch 830/2304 [I 2022-10-27 10:26:55] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:44 [I 2022-10-27 10:26:55] ...trap/engines/engine.py.229: process: 0.56622 | load: 0.00244 [I 2022-10-27 10:26:55] ...trap/engines/engine.py.230: loss: 0.14955 [I 2022-10-27 10:26:55] ...optimizers/trijoint.py.151: total_norm: 13.396971 [I 2022-10-27 10:27:00] ...trap/engines/engine.py.224: train: epoch 62 | batch 840/2304 [I 2022-10-27 10:27:00] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:13:38 [I 2022-10-27 10:27:00] ...trap/engines/engine.py.229: process: 0.56433 | load: 0.00241 [I 2022-10-27 10:27:00] ...trap/engines/engine.py.230: loss: 0.14956 [I 2022-10-27 10:27:00] ...optimizers/trijoint.py.151: total_norm: 18.777220 [I 2022-10-27 10:27:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 850/2304 [I 2022-10-27 10:27:06] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:27 [I 2022-10-27 10:27:06] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00239 [I 2022-10-27 10:27:06] ...trap/engines/engine.py.230: loss: 0.16898 [I 2022-10-27 10:27:06] ...optimizers/trijoint.py.151: total_norm: 44.954739 [I 2022-10-27 10:27:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 860/2304 [I 2022-10-27 10:27:12] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:13:21 [I 2022-10-27 10:27:12] ...trap/engines/engine.py.229: process: 0.56720 | load: 0.00238 [I 2022-10-27 10:27:12] ...trap/engines/engine.py.230: loss: 0.14362 [I 2022-10-27 10:27:12] ...optimizers/trijoint.py.151: total_norm: 9.024646 [I 2022-10-27 10:27:17] ...trap/engines/engine.py.224: train: epoch 62 | batch 870/2304 [I 2022-10-27 10:27:17] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:13:53 [I 2022-10-27 10:27:17] ...trap/engines/engine.py.229: process: 0.62624 | load: 0.00264 [I 2022-10-27 10:27:17] ...trap/engines/engine.py.230: loss: 0.09791 [I 2022-10-27 10:27:17] ...optimizers/trijoint.py.151: total_norm: 7.636260 [I 2022-10-27 10:27:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 880/2304 [I 2022-10-27 10:27:23] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:07 [I 2022-10-27 10:27:23] ...trap/engines/engine.py.229: process: 0.54248 | load: 0.00239 [I 2022-10-27 10:27:23] ...trap/engines/engine.py.230: loss: 0.16575 [I 2022-10-27 10:27:23] ...optimizers/trijoint.py.151: total_norm: 27.222980 [I 2022-10-27 10:27:28] ...trap/engines/engine.py.224: train: epoch 62 | batch 890/2304 [I 2022-10-27 10:27:28] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:13:04 [I 2022-10-27 10:27:28] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00242 [I 2022-10-27 10:27:28] ...trap/engines/engine.py.230: loss: 0.15770 [I 2022-10-27 10:27:28] ...optimizers/trijoint.py.151: total_norm: 14.377048 [I 2022-10-27 10:27:34] ...trap/engines/engine.py.224: train: epoch 62 | batch 900/2304 [I 2022-10-27 10:27:34] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:12:57 [I 2022-10-27 10:27:34] ...trap/engines/engine.py.229: process: 0.56461 | load: 0.00241 [I 2022-10-27 10:27:34] ...trap/engines/engine.py.230: loss: 0.14872 [I 2022-10-27 10:27:34] ...optimizers/trijoint.py.151: total_norm: 11.391516 [I 2022-10-27 10:27:39] ...trap/engines/engine.py.224: train: epoch 62 | batch 910/2304 [I 2022-10-27 10:27:39] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:12:55 [I 2022-10-27 10:27:39] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00242 [I 2022-10-27 10:27:39] ...trap/engines/engine.py.230: loss: 0.14032 [I 2022-10-27 10:27:39] ...optimizers/trijoint.py.151: total_norm: 17.877110 [I 2022-10-27 10:27:45] ...trap/engines/engine.py.224: train: epoch 62 | batch 920/2304 [I 2022-10-27 10:27:45] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:48 [I 2022-10-27 10:27:45] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00245 [I 2022-10-27 10:27:45] ...trap/engines/engine.py.230: loss: 0.11615 [I 2022-10-27 10:27:45] ...optimizers/trijoint.py.151: total_norm: 16.728910 [I 2022-10-27 10:27:51] ...trap/engines/engine.py.224: train: epoch 62 | batch 930/2304 [I 2022-10-27 10:27:51] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:12:35 [I 2022-10-27 10:27:51] ...trap/engines/engine.py.229: process: 0.54103 | load: 0.00239 [I 2022-10-27 10:27:51] ...trap/engines/engine.py.230: loss: 0.17297 [I 2022-10-27 10:27:51] ...optimizers/trijoint.py.151: total_norm: 16.393332 [I 2022-10-27 10:27:56] ...trap/engines/engine.py.224: train: epoch 62 | batch 940/2304 [I 2022-10-27 10:27:56] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:12:42 [I 2022-10-27 10:27:56] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00249 [I 2022-10-27 10:27:56] ...trap/engines/engine.py.230: loss: 0.14797 [I 2022-10-27 10:27:56] ...optimizers/trijoint.py.151: total_norm: 17.252661 [I 2022-10-27 10:28:02] ...trap/engines/engine.py.224: train: epoch 62 | batch 950/2304 [I 2022-10-27 10:28:02] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:39 [I 2022-10-27 10:28:02] ...trap/engines/engine.py.229: process: 0.57216 | load: 0.00242 [I 2022-10-27 10:28:02] ...trap/engines/engine.py.230: loss: 0.15863 [I 2022-10-27 10:28:02] ...optimizers/trijoint.py.151: total_norm: 12.144146 [I 2022-10-27 10:28:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 960/2304 [I 2022-10-27 10:28:07] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:12:29 [I 2022-10-27 10:28:07] ...trap/engines/engine.py.229: process: 0.55727 | load: 0.00240 [I 2022-10-27 10:28:07] ...trap/engines/engine.py.230: loss: 0.14307 [I 2022-10-27 10:28:07] ...optimizers/trijoint.py.151: total_norm: 20.013008 [I 2022-10-27 10:28:13] ...trap/engines/engine.py.224: train: epoch 62 | batch 970/2304 [I 2022-10-27 10:28:13] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:12:21 [I 2022-10-27 10:28:13] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00243 [I 2022-10-27 10:28:13] ...trap/engines/engine.py.230: loss: 0.14189 [I 2022-10-27 10:28:13] ...optimizers/trijoint.py.151: total_norm: 22.861406 [I 2022-10-27 10:28:19] ...trap/engines/engine.py.224: train: epoch 62 | batch 980/2304 [I 2022-10-27 10:28:19] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:16 [I 2022-10-27 10:28:19] ...trap/engines/engine.py.229: process: 0.56571 | load: 0.00240 [I 2022-10-27 10:28:19] ...trap/engines/engine.py.230: loss: 0.20929 [I 2022-10-27 10:28:19] ...optimizers/trijoint.py.151: total_norm: 21.230730 [I 2022-10-27 10:28:24] ...trap/engines/engine.py.224: train: epoch 62 | batch 990/2304 [I 2022-10-27 10:28:24] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:12:14 [I 2022-10-27 10:28:24] ...trap/engines/engine.py.229: process: 0.54871 | load: 0.00241 [I 2022-10-27 10:28:24] ...trap/engines/engine.py.230: loss: 0.13266 [I 2022-10-27 10:28:24] ...optimizers/trijoint.py.151: total_norm: 15.614073 [I 2022-10-27 10:28:30] ...trap/engines/engine.py.224: train: epoch 62 | batch 1000/2304 [I 2022-10-27 10:28:30] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:10 [I 2022-10-27 10:28:30] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00241 [I 2022-10-27 10:28:30] ...trap/engines/engine.py.230: loss: 0.16192 [I 2022-10-27 10:28:30] ...optimizers/trijoint.py.151: total_norm: 23.315884 [I 2022-10-27 10:28:35] ...trap/engines/engine.py.224: train: epoch 62 | batch 1010/2304 [I 2022-10-27 10:28:35] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:12:03 [I 2022-10-27 10:28:35] ...trap/engines/engine.py.229: process: 0.56712 | load: 0.00242 [I 2022-10-27 10:28:35] ...trap/engines/engine.py.230: loss: 0.17145 [I 2022-10-27 10:28:35] ...optimizers/trijoint.py.151: total_norm: 19.237677 [I 2022-10-27 10:28:41] ...trap/engines/engine.py.224: train: epoch 62 | batch 1020/2304 [I 2022-10-27 10:28:41] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:54 [I 2022-10-27 10:28:41] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00242 [I 2022-10-27 10:28:41] ...trap/engines/engine.py.230: loss: 0.12506 [I 2022-10-27 10:28:41] ...optimizers/trijoint.py.151: total_norm: 22.484213 [I 2022-10-27 10:28:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 1030/2304 [I 2022-10-27 10:28:46] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:45 [I 2022-10-27 10:28:46] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00243 [I 2022-10-27 10:28:46] ...trap/engines/engine.py.230: loss: 0.16549 [I 2022-10-27 10:28:46] ...optimizers/trijoint.py.151: total_norm: 18.064550 [I 2022-10-27 10:28:52] ...trap/engines/engine.py.224: train: epoch 62 | batch 1040/2304 [I 2022-10-27 10:28:52] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:38 [I 2022-10-27 10:28:52] ...trap/engines/engine.py.229: process: 0.55007 | load: 0.00239 [I 2022-10-27 10:28:52] ...trap/engines/engine.py.230: loss: 0.13561 [I 2022-10-27 10:28:52] ...optimizers/trijoint.py.151: total_norm: 10.886548 [I 2022-10-27 10:28:58] ...trap/engines/engine.py.224: train: epoch 62 | batch 1050/2304 [I 2022-10-27 10:28:58] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:36 [I 2022-10-27 10:28:58] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00243 [I 2022-10-27 10:28:58] ...trap/engines/engine.py.230: loss: 0.14707 [I 2022-10-27 10:28:58] ...optimizers/trijoint.py.151: total_norm: 23.077063 [I 2022-10-27 10:29:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 1060/2304 [I 2022-10-27 10:29:03] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:33 [I 2022-10-27 10:29:03] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00239 [I 2022-10-27 10:29:03] ...trap/engines/engine.py.230: loss: 0.19787 [I 2022-10-27 10:29:03] ...optimizers/trijoint.py.151: total_norm: 24.462765 [I 2022-10-27 10:29:09] ...trap/engines/engine.py.224: train: epoch 62 | batch 1070/2304 [I 2022-10-27 10:29:09] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:25 [I 2022-10-27 10:29:09] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00241 [I 2022-10-27 10:29:09] ...trap/engines/engine.py.230: loss: 0.12087 [I 2022-10-27 10:29:09] ...optimizers/trijoint.py.151: total_norm: 15.241954 [I 2022-10-27 10:29:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 1080/2304 [I 2022-10-27 10:29:14] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:21 [I 2022-10-27 10:29:14] ...trap/engines/engine.py.229: process: 0.55062 | load: 0.00278 [I 2022-10-27 10:29:14] ...trap/engines/engine.py.230: loss: 0.23698 [I 2022-10-27 10:29:14] ...optimizers/trijoint.py.151: total_norm: 25.010719 [I 2022-10-27 10:29:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 1090/2304 [I 2022-10-27 10:29:20] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:10 [I 2022-10-27 10:29:20] ...trap/engines/engine.py.229: process: 0.55481 | load: 0.00242 [I 2022-10-27 10:29:20] ...trap/engines/engine.py.230: loss: 0.13768 [I 2022-10-27 10:29:20] ...optimizers/trijoint.py.151: total_norm: 12.266012 [I 2022-10-27 10:29:25] ...trap/engines/engine.py.224: train: epoch 62 | batch 1100/2304 [I 2022-10-27 10:29:25] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:10 [I 2022-10-27 10:29:25] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00243 [I 2022-10-27 10:29:25] ...trap/engines/engine.py.230: loss: 0.11618 [I 2022-10-27 10:29:25] ...optimizers/trijoint.py.151: total_norm: 21.298746 [I 2022-10-27 10:29:31] ...trap/engines/engine.py.224: train: epoch 62 | batch 1110/2304 [I 2022-10-27 10:29:31] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:11:05 [I 2022-10-27 10:29:31] ...trap/engines/engine.py.229: process: 0.56314 | load: 0.00241 [I 2022-10-27 10:29:31] ...trap/engines/engine.py.230: loss: 0.12856 [I 2022-10-27 10:29:31] ...optimizers/trijoint.py.151: total_norm: 17.663475 [I 2022-10-27 10:29:37] ...trap/engines/engine.py.224: train: epoch 62 | batch 1120/2304 [I 2022-10-27 10:29:37] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:11:00 [I 2022-10-27 10:29:37] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00244 [I 2022-10-27 10:29:37] ...trap/engines/engine.py.230: loss: 0.15936 [I 2022-10-27 10:29:37] ...optimizers/trijoint.py.151: total_norm: 17.662678 [I 2022-10-27 10:29:42] ...trap/engines/engine.py.224: train: epoch 62 | batch 1130/2304 [I 2022-10-27 10:29:42] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:52 [I 2022-10-27 10:29:42] ...trap/engines/engine.py.229: process: 0.55351 | load: 0.00242 [I 2022-10-27 10:29:42] ...trap/engines/engine.py.230: loss: 0.12636 [I 2022-10-27 10:29:42] ...optimizers/trijoint.py.151: total_norm: 10.675915 [I 2022-10-27 10:29:48] ...trap/engines/engine.py.224: train: epoch 62 | batch 1140/2304 [I 2022-10-27 10:29:48] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:46 [I 2022-10-27 10:29:48] ...trap/engines/engine.py.229: process: 0.54819 | load: 0.00243 [I 2022-10-27 10:29:48] ...trap/engines/engine.py.230: loss: 0.25426 [I 2022-10-27 10:29:48] ...optimizers/trijoint.py.151: total_norm: 31.229502 [I 2022-10-27 10:29:53] ...trap/engines/engine.py.224: train: epoch 62 | batch 1150/2304 [I 2022-10-27 10:29:53] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:37 [I 2022-10-27 10:29:53] ...trap/engines/engine.py.229: process: 0.55391 | load: 0.00240 [I 2022-10-27 10:29:53] ...trap/engines/engine.py.230: loss: 0.17303 [I 2022-10-27 10:29:53] ...optimizers/trijoint.py.151: total_norm: 19.109072 [I 2022-10-27 10:29:59] ...trap/engines/engine.py.224: train: epoch 62 | batch 1160/2304 [I 2022-10-27 10:29:59] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:31 [I 2022-10-27 10:29:59] ...trap/engines/engine.py.229: process: 0.53681 | load: 0.00240 [I 2022-10-27 10:29:59] ...trap/engines/engine.py.230: loss: 0.11366 [I 2022-10-27 10:29:59] ...optimizers/trijoint.py.151: total_norm: 6.134096 [I 2022-10-27 10:30:04] ...trap/engines/engine.py.224: train: epoch 62 | batch 1170/2304 [I 2022-10-27 10:30:04] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:31 [I 2022-10-27 10:30:04] ...trap/engines/engine.py.229: process: 0.56587 | load: 0.00243 [I 2022-10-27 10:30:04] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-27 10:30:04] ...optimizers/trijoint.py.151: total_norm: 24.453512 [I 2022-10-27 10:30:10] ...trap/engines/engine.py.224: train: epoch 62 | batch 1180/2304 [I 2022-10-27 10:30:10] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:22 [I 2022-10-27 10:30:10] ...trap/engines/engine.py.229: process: 0.54734 | load: 0.00241 [I 2022-10-27 10:30:10] ...trap/engines/engine.py.230: loss: 0.14109 [I 2022-10-27 10:30:10] ...optimizers/trijoint.py.151: total_norm: 17.651886 [I 2022-10-27 10:30:16] ...trap/engines/engine.py.224: train: epoch 62 | batch 1190/2304 [I 2022-10-27 10:30:16] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:22 [I 2022-10-27 10:30:16] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00279 [I 2022-10-27 10:30:16] ...trap/engines/engine.py.230: loss: 0.22880 [I 2022-10-27 10:30:16] ...optimizers/trijoint.py.151: total_norm: 37.275574 [I 2022-10-27 10:30:21] ...trap/engines/engine.py.224: train: epoch 62 | batch 1200/2304 [I 2022-10-27 10:30:21] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:11 [I 2022-10-27 10:30:21] ...trap/engines/engine.py.229: process: 0.55575 | load: 0.00240 [I 2022-10-27 10:30:21] ...trap/engines/engine.py.230: loss: 0.12054 [I 2022-10-27 10:30:21] ...optimizers/trijoint.py.151: total_norm: 17.854784 [I 2022-10-27 10:30:27] ...trap/engines/engine.py.224: train: epoch 62 | batch 1210/2304 [I 2022-10-27 10:30:27] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:10:11 [I 2022-10-27 10:30:27] ...trap/engines/engine.py.229: process: 0.55847 | load: 0.00242 [I 2022-10-27 10:30:27] ...trap/engines/engine.py.230: loss: 0.19388 [I 2022-10-27 10:30:27] ...optimizers/trijoint.py.151: total_norm: 23.648859 [I 2022-10-27 10:30:33] ...trap/engines/engine.py.224: train: epoch 62 | batch 1220/2304 [I 2022-10-27 10:30:33] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:10:14 [I 2022-10-27 10:30:33] ...trap/engines/engine.py.229: process: 0.56039 | load: 0.00239 [I 2022-10-27 10:30:33] ...trap/engines/engine.py.230: loss: 0.13257 [I 2022-10-27 10:30:33] ...optimizers/trijoint.py.151: total_norm: 23.964979 [I 2022-10-27 10:30:38] ...trap/engines/engine.py.224: train: epoch 62 | batch 1230/2304 [I 2022-10-27 10:30:38] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:10:01 [I 2022-10-27 10:30:38] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00243 [I 2022-10-27 10:30:38] ...trap/engines/engine.py.230: loss: 0.11099 [I 2022-10-27 10:30:38] ...optimizers/trijoint.py.151: total_norm: 8.026089 [I 2022-10-27 10:30:44] ...trap/engines/engine.py.224: train: epoch 62 | batch 1240/2304 [I 2022-10-27 10:30:44] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:50 [I 2022-10-27 10:30:44] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00243 [I 2022-10-27 10:30:44] ...trap/engines/engine.py.230: loss: 0.19071 [I 2022-10-27 10:30:44] ...optimizers/trijoint.py.151: total_norm: 27.069559 [I 2022-10-27 10:30:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 1250/2304 [I 2022-10-27 10:30:49] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:48 [I 2022-10-27 10:30:49] ...trap/engines/engine.py.229: process: 0.56097 | load: 0.00241 [I 2022-10-27 10:30:49] ...trap/engines/engine.py.230: loss: 0.14065 [I 2022-10-27 10:30:49] ...optimizers/trijoint.py.151: total_norm: 9.588800 [I 2022-10-27 10:30:55] ...trap/engines/engine.py.224: train: epoch 62 | batch 1260/2304 [I 2022-10-27 10:30:55] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:46 [I 2022-10-27 10:30:55] ...trap/engines/engine.py.229: process: 0.56126 | load: 0.00262 [I 2022-10-27 10:30:55] ...trap/engines/engine.py.230: loss: 0.19931 [I 2022-10-27 10:30:55] ...optimizers/trijoint.py.151: total_norm: 19.234440 [I 2022-10-27 10:31:01] ...trap/engines/engine.py.224: train: epoch 62 | batch 1270/2304 [I 2022-10-27 10:31:01] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:35 [I 2022-10-27 10:31:01] ...trap/engines/engine.py.229: process: 0.54429 | load: 0.00242 [I 2022-10-27 10:31:01] ...trap/engines/engine.py.230: loss: 0.17306 [I 2022-10-27 10:31:01] ...optimizers/trijoint.py.151: total_norm: 16.270664 [I 2022-10-27 10:31:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 1280/2304 [I 2022-10-27 10:31:06] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:31 [I 2022-10-27 10:31:06] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00243 [I 2022-10-27 10:31:06] ...trap/engines/engine.py.230: loss: 0.10814 [I 2022-10-27 10:31:06] ...optimizers/trijoint.py.151: total_norm: 15.877850 [I 2022-10-27 10:31:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 1290/2304 [I 2022-10-27 10:31:12] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:28 [I 2022-10-27 10:31:12] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00241 [I 2022-10-27 10:31:12] ...trap/engines/engine.py.230: loss: 0.20457 [I 2022-10-27 10:31:12] ...optimizers/trijoint.py.151: total_norm: 20.024879 [I 2022-10-27 10:31:17] ...trap/engines/engine.py.224: train: epoch 62 | batch 1300/2304 [I 2022-10-27 10:31:17] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:20 [I 2022-10-27 10:31:17] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00251 [I 2022-10-27 10:31:17] ...trap/engines/engine.py.230: loss: 0.17564 [I 2022-10-27 10:31:17] ...optimizers/trijoint.py.151: total_norm: 22.166458 [I 2022-10-27 10:31:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 1310/2304 [I 2022-10-27 10:31:23] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:17 [I 2022-10-27 10:31:23] ...trap/engines/engine.py.229: process: 0.57176 | load: 0.00250 [I 2022-10-27 10:31:23] ...trap/engines/engine.py.230: loss: 0.15090 [I 2022-10-27 10:31:23] ...optimizers/trijoint.py.151: total_norm: 23.510929 [I 2022-10-27 10:31:29] ...trap/engines/engine.py.224: train: epoch 62 | batch 1320/2304 [I 2022-10-27 10:31:29] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:09 [I 2022-10-27 10:31:29] ...trap/engines/engine.py.229: process: 0.54930 | load: 0.00245 [I 2022-10-27 10:31:29] ...trap/engines/engine.py.230: loss: 0.18856 [I 2022-10-27 10:31:29] ...optimizers/trijoint.py.151: total_norm: 17.872736 [I 2022-10-27 10:31:34] ...trap/engines/engine.py.224: train: epoch 62 | batch 1330/2304 [I 2022-10-27 10:31:34] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:08:59 [I 2022-10-27 10:31:34] ...trap/engines/engine.py.229: process: 0.54480 | load: 0.00246 [I 2022-10-27 10:31:34] ...trap/engines/engine.py.230: loss: 0.15432 [I 2022-10-27 10:31:34] ...optimizers/trijoint.py.151: total_norm: 14.762647 [I 2022-10-27 10:31:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 1340/2304 [I 2022-10-27 10:31:40] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:55 [I 2022-10-27 10:31:40] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00283 [I 2022-10-27 10:31:40] ...trap/engines/engine.py.230: loss: 0.13281 [I 2022-10-27 10:31:40] ...optimizers/trijoint.py.151: total_norm: 16.815096 [I 2022-10-27 10:31:45] ...trap/engines/engine.py.224: train: epoch 62 | batch 1350/2304 [I 2022-10-27 10:31:45] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:52 [I 2022-10-27 10:31:45] ...trap/engines/engine.py.229: process: 0.56353 | load: 0.00246 [I 2022-10-27 10:31:45] ...trap/engines/engine.py.230: loss: 0.13741 [I 2022-10-27 10:31:45] ...optimizers/trijoint.py.151: total_norm: 10.728944 [I 2022-10-27 10:31:51] ...trap/engines/engine.py.224: train: epoch 62 | batch 1360/2304 [I 2022-10-27 10:31:51] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:45 [I 2022-10-27 10:31:51] ...trap/engines/engine.py.229: process: 0.55299 | load: 0.00240 [I 2022-10-27 10:31:51] ...trap/engines/engine.py.230: loss: 0.10123 [I 2022-10-27 10:31:51] ...optimizers/trijoint.py.151: total_norm: 12.992202 [I 2022-10-27 10:31:56] ...trap/engines/engine.py.224: train: epoch 62 | batch 1370/2304 [I 2022-10-27 10:31:56] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:39 [I 2022-10-27 10:31:56] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00241 [I 2022-10-27 10:31:56] ...trap/engines/engine.py.230: loss: 0.11326 [I 2022-10-27 10:31:56] ...optimizers/trijoint.py.151: total_norm: 14.071891 [I 2022-10-27 10:32:02] ...trap/engines/engine.py.224: train: epoch 62 | batch 1380/2304 [I 2022-10-27 10:32:02] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:33 [I 2022-10-27 10:32:02] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00248 [I 2022-10-27 10:32:02] ...trap/engines/engine.py.230: loss: 0.11542 [I 2022-10-27 10:32:02] ...optimizers/trijoint.py.151: total_norm: 15.253662 [I 2022-10-27 10:32:08] ...trap/engines/engine.py.224: train: epoch 62 | batch 1390/2304 [I 2022-10-27 10:32:08] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:28 [I 2022-10-27 10:32:08] ...trap/engines/engine.py.229: process: 0.54782 | load: 0.00243 [I 2022-10-27 10:32:08] ...trap/engines/engine.py.230: loss: 0.19216 [I 2022-10-27 10:32:08] ...optimizers/trijoint.py.151: total_norm: 25.576525 [I 2022-10-27 10:32:13] ...trap/engines/engine.py.224: train: epoch 62 | batch 1400/2304 [I 2022-10-27 10:32:13] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:21 [I 2022-10-27 10:32:13] ...trap/engines/engine.py.229: process: 0.54592 | load: 0.00249 [I 2022-10-27 10:32:13] ...trap/engines/engine.py.230: loss: 0.14998 [I 2022-10-27 10:32:13] ...optimizers/trijoint.py.151: total_norm: 19.461023 [I 2022-10-27 10:32:19] ...trap/engines/engine.py.224: train: epoch 62 | batch 1410/2304 [I 2022-10-27 10:32:19] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:21 [I 2022-10-27 10:32:19] ...trap/engines/engine.py.229: process: 0.56226 | load: 0.00248 [I 2022-10-27 10:32:19] ...trap/engines/engine.py.230: loss: 0.14981 [I 2022-10-27 10:32:19] ...optimizers/trijoint.py.151: total_norm: 14.607346 [I 2022-10-27 10:32:24] ...trap/engines/engine.py.224: train: epoch 62 | batch 1420/2304 [I 2022-10-27 10:32:24] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:12 [I 2022-10-27 10:32:24] ...trap/engines/engine.py.229: process: 0.55413 | load: 0.00252 [I 2022-10-27 10:32:24] ...trap/engines/engine.py.230: loss: 0.11625 [I 2022-10-27 10:32:24] ...optimizers/trijoint.py.151: total_norm: 11.965731 [I 2022-10-27 10:32:30] ...trap/engines/engine.py.224: train: epoch 62 | batch 1430/2304 [I 2022-10-27 10:32:30] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:08:05 [I 2022-10-27 10:32:30] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00249 [I 2022-10-27 10:32:30] ...trap/engines/engine.py.230: loss: 0.11860 [I 2022-10-27 10:32:30] ...optimizers/trijoint.py.151: total_norm: 12.232288 [I 2022-10-27 10:32:36] ...trap/engines/engine.py.224: train: epoch 62 | batch 1440/2304 [I 2022-10-27 10:32:36] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:08:01 [I 2022-10-27 10:32:36] ...trap/engines/engine.py.229: process: 0.56361 | load: 0.00249 [I 2022-10-27 10:32:36] ...trap/engines/engine.py.230: loss: 0.11334 [I 2022-10-27 10:32:36] ...optimizers/trijoint.py.151: total_norm: 14.634765 [I 2022-10-27 10:32:41] ...trap/engines/engine.py.224: train: epoch 62 | batch 1450/2304 [I 2022-10-27 10:32:41] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:54 [I 2022-10-27 10:32:41] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00249 [I 2022-10-27 10:32:41] ...trap/engines/engine.py.230: loss: 0.12415 [I 2022-10-27 10:32:41] ...optimizers/trijoint.py.151: total_norm: 32.089798 [I 2022-10-27 10:32:47] ...trap/engines/engine.py.224: train: epoch 62 | batch 1460/2304 [I 2022-10-27 10:32:47] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:51 [I 2022-10-27 10:32:47] ...trap/engines/engine.py.229: process: 0.55803 | load: 0.00246 [I 2022-10-27 10:32:47] ...trap/engines/engine.py.230: loss: 0.11367 [I 2022-10-27 10:32:47] ...optimizers/trijoint.py.151: total_norm: 15.364387 [I 2022-10-27 10:32:52] ...trap/engines/engine.py.224: train: epoch 62 | batch 1470/2304 [I 2022-10-27 10:32:52] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:45 [I 2022-10-27 10:32:52] ...trap/engines/engine.py.229: process: 0.54624 | load: 0.00259 [I 2022-10-27 10:32:52] ...trap/engines/engine.py.230: loss: 0.17018 [I 2022-10-27 10:32:52] ...optimizers/trijoint.py.151: total_norm: 19.275023 [I 2022-10-27 10:32:58] ...trap/engines/engine.py.224: train: epoch 62 | batch 1480/2304 [I 2022-10-27 10:32:58] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:42 [I 2022-10-27 10:32:58] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00248 [I 2022-10-27 10:32:58] ...trap/engines/engine.py.230: loss: 0.14066 [I 2022-10-27 10:32:58] ...optimizers/trijoint.py.151: total_norm: 18.170444 [I 2022-10-27 10:33:04] ...trap/engines/engine.py.224: train: epoch 62 | batch 1490/2304 [I 2022-10-27 10:33:04] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:33 [I 2022-10-27 10:33:04] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00244 [I 2022-10-27 10:33:04] ...trap/engines/engine.py.230: loss: 0.19901 [I 2022-10-27 10:33:04] ...optimizers/trijoint.py.151: total_norm: 55.449707 [I 2022-10-27 10:33:09] ...trap/engines/engine.py.224: train: epoch 62 | batch 1500/2304 [I 2022-10-27 10:33:09] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:28 [I 2022-10-27 10:33:09] ...trap/engines/engine.py.229: process: 0.56473 | load: 0.00246 [I 2022-10-27 10:33:09] ...trap/engines/engine.py.230: loss: 0.13800 [I 2022-10-27 10:33:09] ...optimizers/trijoint.py.151: total_norm: 9.342223 [I 2022-10-27 10:33:15] ...trap/engines/engine.py.224: train: epoch 62 | batch 1510/2304 [I 2022-10-27 10:33:15] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:21 [I 2022-10-27 10:33:15] ...trap/engines/engine.py.229: process: 0.55103 | load: 0.00258 [I 2022-10-27 10:33:15] ...trap/engines/engine.py.230: loss: 0.17217 [I 2022-10-27 10:33:15] ...optimizers/trijoint.py.151: total_norm: 23.208447 [I 2022-10-27 10:33:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 1520/2304 [I 2022-10-27 10:33:20] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:18 [I 2022-10-27 10:33:20] ...trap/engines/engine.py.229: process: 0.56382 | load: 0.00242 [I 2022-10-27 10:33:20] ...trap/engines/engine.py.230: loss: 0.14305 [I 2022-10-27 10:33:20] ...optimizers/trijoint.py.151: total_norm: 8.185173 [I 2022-10-27 10:33:26] ...trap/engines/engine.py.224: train: epoch 62 | batch 1530/2304 [I 2022-10-27 10:33:26] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:07:14 [I 2022-10-27 10:33:26] ...trap/engines/engine.py.229: process: 0.57121 | load: 0.00242 [I 2022-10-27 10:33:26] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-27 10:33:26] ...optimizers/trijoint.py.151: total_norm: 16.949087 [I 2022-10-27 10:33:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 1540/2304 [I 2022-10-27 10:33:32] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:07:04 [I 2022-10-27 10:33:32] ...trap/engines/engine.py.229: process: 0.54287 | load: 0.00245 [I 2022-10-27 10:33:32] ...trap/engines/engine.py.230: loss: 0.13800 [I 2022-10-27 10:33:32] ...optimizers/trijoint.py.151: total_norm: 15.146606 [I 2022-10-27 10:33:37] ...trap/engines/engine.py.224: train: epoch 62 | batch 1550/2304 [I 2022-10-27 10:33:37] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:06:59 [I 2022-10-27 10:33:37] ...trap/engines/engine.py.229: process: 0.55160 | load: 0.00241 [I 2022-10-27 10:33:37] ...trap/engines/engine.py.230: loss: 0.17113 [I 2022-10-27 10:33:37] ...optimizers/trijoint.py.151: total_norm: 22.142082 [I 2022-10-27 10:33:43] ...trap/engines/engine.py.224: train: epoch 62 | batch 1560/2304 [I 2022-10-27 10:33:43] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:55 [I 2022-10-27 10:33:43] ...trap/engines/engine.py.229: process: 0.55068 | load: 0.00242 [I 2022-10-27 10:33:43] ...trap/engines/engine.py.230: loss: 0.15251 [I 2022-10-27 10:33:43] ...optimizers/trijoint.py.151: total_norm: 17.947668 [I 2022-10-27 10:33:48] ...trap/engines/engine.py.224: train: epoch 62 | batch 1570/2304 [I 2022-10-27 10:33:48] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:51 [I 2022-10-27 10:33:48] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00242 [I 2022-10-27 10:33:48] ...trap/engines/engine.py.230: loss: 0.19599 [I 2022-10-27 10:33:48] ...optimizers/trijoint.py.151: total_norm: 64.395126 [I 2022-10-27 10:33:54] ...trap/engines/engine.py.224: train: epoch 62 | batch 1580/2304 [I 2022-10-27 10:33:54] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:44 [I 2022-10-27 10:33:54] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00245 [I 2022-10-27 10:33:54] ...trap/engines/engine.py.230: loss: 0.12771 [I 2022-10-27 10:33:54] ...optimizers/trijoint.py.151: total_norm: 18.893532 [I 2022-10-27 10:34:00] ...trap/engines/engine.py.224: train: epoch 62 | batch 1590/2304 [I 2022-10-27 10:34:00] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:37 [I 2022-10-27 10:34:00] ...trap/engines/engine.py.229: process: 0.54822 | load: 0.00241 [I 2022-10-27 10:34:00] ...trap/engines/engine.py.230: loss: 0.14311 [I 2022-10-27 10:34:00] ...optimizers/trijoint.py.151: total_norm: 50.636600 [I 2022-10-27 10:34:05] ...trap/engines/engine.py.224: train: epoch 62 | batch 1600/2304 [I 2022-10-27 10:34:05] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:33 [I 2022-10-27 10:34:05] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00245 [I 2022-10-27 10:34:05] ...trap/engines/engine.py.230: loss: 0.11854 [I 2022-10-27 10:34:05] ...optimizers/trijoint.py.151: total_norm: 9.809464 [I 2022-10-27 10:34:11] ...trap/engines/engine.py.224: train: epoch 62 | batch 1610/2304 [I 2022-10-27 10:34:11] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:29 [I 2022-10-27 10:34:11] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00243 [I 2022-10-27 10:34:11] ...trap/engines/engine.py.230: loss: 0.17525 [I 2022-10-27 10:34:11] ...optimizers/trijoint.py.151: total_norm: 13.853207 [I 2022-10-27 10:34:16] ...trap/engines/engine.py.224: train: epoch 62 | batch 1620/2304 [I 2022-10-27 10:34:16] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:22 [I 2022-10-27 10:34:16] ...trap/engines/engine.py.229: process: 0.55274 | load: 0.00241 [I 2022-10-27 10:34:16] ...trap/engines/engine.py.230: loss: 0.16935 [I 2022-10-27 10:34:16] ...optimizers/trijoint.py.151: total_norm: 14.184102 [I 2022-10-27 10:34:22] ...trap/engines/engine.py.224: train: epoch 62 | batch 1630/2304 [I 2022-10-27 10:34:22] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:06:14 [I 2022-10-27 10:34:22] ...trap/engines/engine.py.229: process: 0.54355 | load: 0.00245 [I 2022-10-27 10:34:22] ...trap/engines/engine.py.230: loss: 0.11002 [I 2022-10-27 10:34:22] ...optimizers/trijoint.py.151: total_norm: 13.969452 [I 2022-10-27 10:34:28] ...trap/engines/engine.py.224: train: epoch 62 | batch 1640/2304 [I 2022-10-27 10:34:28] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:06:10 [I 2022-10-27 10:34:28] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00242 [I 2022-10-27 10:34:28] ...trap/engines/engine.py.230: loss: 0.19120 [I 2022-10-27 10:34:28] ...optimizers/trijoint.py.151: total_norm: 17.383020 [I 2022-10-27 10:34:33] ...trap/engines/engine.py.224: train: epoch 62 | batch 1650/2304 [I 2022-10-27 10:34:33] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:06:04 [I 2022-10-27 10:34:33] ...trap/engines/engine.py.229: process: 0.56370 | load: 0.00241 [I 2022-10-27 10:34:33] ...trap/engines/engine.py.230: loss: 0.15262 [I 2022-10-27 10:34:33] ...optimizers/trijoint.py.151: total_norm: 21.350004 [I 2022-10-27 10:34:39] ...trap/engines/engine.py.224: train: epoch 62 | batch 1660/2304 [I 2022-10-27 10:34:39] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:56 [I 2022-10-27 10:34:39] ...trap/engines/engine.py.229: process: 0.53658 | load: 0.00240 [I 2022-10-27 10:34:39] ...trap/engines/engine.py.230: loss: 0.20032 [I 2022-10-27 10:34:39] ...optimizers/trijoint.py.151: total_norm: 26.269842 [I 2022-10-27 10:34:44] ...trap/engines/engine.py.224: train: epoch 62 | batch 1670/2304 [I 2022-10-27 10:34:44] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:54 [I 2022-10-27 10:34:44] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00241 [I 2022-10-27 10:34:44] ...trap/engines/engine.py.230: loss: 0.16143 [I 2022-10-27 10:34:44] ...optimizers/trijoint.py.151: total_norm: 16.817896 [I 2022-10-27 10:34:50] ...trap/engines/engine.py.224: train: epoch 62 | batch 1680/2304 [I 2022-10-27 10:34:50] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:46 [I 2022-10-27 10:34:50] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00242 [I 2022-10-27 10:34:50] ...trap/engines/engine.py.230: loss: 0.15680 [I 2022-10-27 10:34:50] ...optimizers/trijoint.py.151: total_norm: 24.189600 [I 2022-10-27 10:34:55] ...trap/engines/engine.py.224: train: epoch 62 | batch 1690/2304 [I 2022-10-27 10:34:55] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:40 [I 2022-10-27 10:34:55] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00240 [I 2022-10-27 10:34:55] ...trap/engines/engine.py.230: loss: 0.15883 [I 2022-10-27 10:34:55] ...optimizers/trijoint.py.151: total_norm: 41.504349 [I 2022-10-27 10:35:01] ...trap/engines/engine.py.224: train: epoch 62 | batch 1700/2304 [I 2022-10-27 10:35:01] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:35 [I 2022-10-27 10:35:01] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00240 [I 2022-10-27 10:35:01] ...trap/engines/engine.py.230: loss: 0.13652 [I 2022-10-27 10:35:01] ...optimizers/trijoint.py.151: total_norm: 10.129092 [I 2022-10-27 10:35:07] ...trap/engines/engine.py.224: train: epoch 62 | batch 1710/2304 [I 2022-10-27 10:35:07] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:32 [I 2022-10-27 10:35:07] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00240 [I 2022-10-27 10:35:07] ...trap/engines/engine.py.230: loss: 0.13134 [I 2022-10-27 10:35:07] ...optimizers/trijoint.py.151: total_norm: 12.396560 [I 2022-10-27 10:35:12] ...trap/engines/engine.py.224: train: epoch 62 | batch 1720/2304 [I 2022-10-27 10:35:12] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:20 [I 2022-10-27 10:35:12] ...trap/engines/engine.py.229: process: 0.55315 | load: 0.00242 [I 2022-10-27 10:35:12] ...trap/engines/engine.py.230: loss: 0.29726 [I 2022-10-27 10:35:12] ...optimizers/trijoint.py.151: total_norm: 21.377390 [I 2022-10-27 10:35:18] ...trap/engines/engine.py.224: train: epoch 62 | batch 1730/2304 [I 2022-10-27 10:35:18] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:18 [I 2022-10-27 10:35:18] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00249 [I 2022-10-27 10:35:18] ...trap/engines/engine.py.230: loss: 0.11018 [I 2022-10-27 10:35:18] ...optimizers/trijoint.py.151: total_norm: 10.504887 [I 2022-10-27 10:35:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 1740/2304 [I 2022-10-27 10:35:23] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:05:13 [I 2022-10-27 10:35:23] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00245 [I 2022-10-27 10:35:23] ...trap/engines/engine.py.230: loss: 0.16846 [I 2022-10-27 10:35:23] ...optimizers/trijoint.py.151: total_norm: 17.184107 [I 2022-10-27 10:35:29] ...trap/engines/engine.py.224: train: epoch 62 | batch 1750/2304 [I 2022-10-27 10:35:29] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:06 [I 2022-10-27 10:35:29] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00242 [I 2022-10-27 10:35:29] ...trap/engines/engine.py.230: loss: 0.18881 [I 2022-10-27 10:35:29] ...optimizers/trijoint.py.151: total_norm: 10.304827 [I 2022-10-27 10:35:34] ...trap/engines/engine.py.224: train: epoch 62 | batch 1760/2304 [I 2022-10-27 10:35:34] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:05:03 [I 2022-10-27 10:35:34] ...trap/engines/engine.py.229: process: 0.54536 | load: 0.00246 [I 2022-10-27 10:35:34] ...trap/engines/engine.py.230: loss: 0.16261 [I 2022-10-27 10:35:34] ...optimizers/trijoint.py.151: total_norm: 21.629988 [I 2022-10-27 10:35:40] ...trap/engines/engine.py.224: train: epoch 62 | batch 1770/2304 [I 2022-10-27 10:35:40] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:57 [I 2022-10-27 10:35:40] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00240 [I 2022-10-27 10:35:40] ...trap/engines/engine.py.230: loss: 0.22570 [I 2022-10-27 10:35:40] ...optimizers/trijoint.py.151: total_norm: 19.861553 [I 2022-10-27 10:35:46] ...trap/engines/engine.py.224: train: epoch 62 | batch 1780/2304 [I 2022-10-27 10:35:46] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:51 [I 2022-10-27 10:35:46] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00242 [I 2022-10-27 10:35:46] ...trap/engines/engine.py.230: loss: 0.15121 [I 2022-10-27 10:35:46] ...optimizers/trijoint.py.151: total_norm: 12.223387 [I 2022-10-27 10:35:51] ...trap/engines/engine.py.224: train: epoch 62 | batch 1790/2304 [I 2022-10-27 10:35:51] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:04:58 [I 2022-10-27 10:35:51] ...trap/engines/engine.py.229: process: 0.61587 | load: 0.00251 [I 2022-10-27 10:35:51] ...trap/engines/engine.py.230: loss: 0.13756 [I 2022-10-27 10:35:51] ...optimizers/trijoint.py.151: total_norm: 10.257257 [I 2022-10-27 10:35:57] ...trap/engines/engine.py.224: train: epoch 62 | batch 1800/2304 [I 2022-10-27 10:35:57] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:40 [I 2022-10-27 10:35:57] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00245 [I 2022-10-27 10:35:57] ...trap/engines/engine.py.230: loss: 0.13982 [I 2022-10-27 10:35:57] ...optimizers/trijoint.py.151: total_norm: 26.335091 [I 2022-10-27 10:36:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 1810/2304 [I 2022-10-27 10:36:03] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:36 [I 2022-10-27 10:36:03] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00242 [I 2022-10-27 10:36:03] ...trap/engines/engine.py.230: loss: 0.18373 [I 2022-10-27 10:36:03] ...optimizers/trijoint.py.151: total_norm: 16.048073 [I 2022-10-27 10:36:08] ...trap/engines/engine.py.224: train: epoch 62 | batch 1820/2304 [I 2022-10-27 10:36:08] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:30 [I 2022-10-27 10:36:08] ...trap/engines/engine.py.229: process: 0.57026 | load: 0.00243 [I 2022-10-27 10:36:08] ...trap/engines/engine.py.230: loss: 0.13846 [I 2022-10-27 10:36:08] ...optimizers/trijoint.py.151: total_norm: 10.565214 [I 2022-10-27 10:36:14] ...trap/engines/engine.py.224: train: epoch 62 | batch 1830/2304 [I 2022-10-27 10:36:14] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:26 [I 2022-10-27 10:36:14] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00244 [I 2022-10-27 10:36:14] ...trap/engines/engine.py.230: loss: 0.14643 [I 2022-10-27 10:36:14] ...optimizers/trijoint.py.151: total_norm: 8.575758 [I 2022-10-27 10:36:19] ...trap/engines/engine.py.224: train: epoch 62 | batch 1840/2304 [I 2022-10-27 10:36:19] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:04:18 [I 2022-10-27 10:36:19] ...trap/engines/engine.py.229: process: 0.55186 | load: 0.00247 [I 2022-10-27 10:36:19] ...trap/engines/engine.py.230: loss: 0.14635 [I 2022-10-27 10:36:19] ...optimizers/trijoint.py.151: total_norm: 29.700312 [I 2022-10-27 10:36:25] ...trap/engines/engine.py.224: train: epoch 62 | batch 1850/2304 [I 2022-10-27 10:36:25] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:04:11 [I 2022-10-27 10:36:25] ...trap/engines/engine.py.229: process: 0.54565 | load: 0.00246 [I 2022-10-27 10:36:25] ...trap/engines/engine.py.230: loss: 0.08971 [I 2022-10-27 10:36:25] ...optimizers/trijoint.py.151: total_norm: 10.344003 [I 2022-10-27 10:36:31] ...trap/engines/engine.py.224: train: epoch 62 | batch 1860/2304 [I 2022-10-27 10:36:31] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:04:11 [I 2022-10-27 10:36:31] ...trap/engines/engine.py.229: process: 0.55432 | load: 0.00250 [I 2022-10-27 10:36:31] ...trap/engines/engine.py.230: loss: 0.18921 [I 2022-10-27 10:36:31] ...optimizers/trijoint.py.151: total_norm: 15.253275 [I 2022-10-27 10:36:36] ...trap/engines/engine.py.224: train: epoch 62 | batch 1870/2304 [I 2022-10-27 10:36:36] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:04:02 [I 2022-10-27 10:36:36] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00245 [I 2022-10-27 10:36:36] ...trap/engines/engine.py.230: loss: 0.12615 [I 2022-10-27 10:36:36] ...optimizers/trijoint.py.151: total_norm: 14.961324 [I 2022-10-27 10:36:42] ...trap/engines/engine.py.224: train: epoch 62 | batch 1880/2304 [I 2022-10-27 10:36:42] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:58 [I 2022-10-27 10:36:42] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00247 [I 2022-10-27 10:36:42] ...trap/engines/engine.py.230: loss: 0.11236 [I 2022-10-27 10:36:42] ...optimizers/trijoint.py.151: total_norm: 13.380989 [I 2022-10-27 10:36:47] ...trap/engines/engine.py.224: train: epoch 62 | batch 1890/2304 [I 2022-10-27 10:36:47] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:51 [I 2022-10-27 10:36:47] ...trap/engines/engine.py.229: process: 0.55845 | load: 0.00242 [I 2022-10-27 10:36:47] ...trap/engines/engine.py.230: loss: 0.09704 [I 2022-10-27 10:36:47] ...optimizers/trijoint.py.151: total_norm: 8.984562 [I 2022-10-27 10:36:53] ...trap/engines/engine.py.224: train: epoch 62 | batch 1900/2304 [I 2022-10-27 10:36:53] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:45 [I 2022-10-27 10:36:53] ...trap/engines/engine.py.229: process: 0.54988 | load: 0.00243 [I 2022-10-27 10:36:53] ...trap/engines/engine.py.230: loss: 0.15461 [I 2022-10-27 10:36:53] ...optimizers/trijoint.py.151: total_norm: 11.185192 [I 2022-10-27 10:36:59] ...trap/engines/engine.py.224: train: epoch 62 | batch 1910/2304 [I 2022-10-27 10:36:59] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:39 [I 2022-10-27 10:36:59] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00242 [I 2022-10-27 10:36:59] ...trap/engines/engine.py.230: loss: 0.12113 [I 2022-10-27 10:36:59] ...optimizers/trijoint.py.151: total_norm: 14.323020 [I 2022-10-27 10:37:04] ...trap/engines/engine.py.224: train: epoch 62 | batch 1920/2304 [I 2022-10-27 10:37:04] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:33 [I 2022-10-27 10:37:04] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00242 [I 2022-10-27 10:37:04] ...trap/engines/engine.py.230: loss: 0.18744 [I 2022-10-27 10:37:04] ...optimizers/trijoint.py.151: total_norm: 20.832724 [I 2022-10-27 10:37:10] ...trap/engines/engine.py.224: train: epoch 62 | batch 1930/2304 [I 2022-10-27 10:37:10] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:28 [I 2022-10-27 10:37:10] ...trap/engines/engine.py.229: process: 0.55954 | load: 0.00242 [I 2022-10-27 10:37:10] ...trap/engines/engine.py.230: loss: 0.12127 [I 2022-10-27 10:37:10] ...optimizers/trijoint.py.151: total_norm: 18.182894 [I 2022-10-27 10:37:15] ...trap/engines/engine.py.224: train: epoch 62 | batch 1940/2304 [I 2022-10-27 10:37:15] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:22 [I 2022-10-27 10:37:15] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00240 [I 2022-10-27 10:37:15] ...trap/engines/engine.py.230: loss: 0.13774 [I 2022-10-27 10:37:15] ...optimizers/trijoint.py.151: total_norm: 14.071605 [I 2022-10-27 10:37:21] ...trap/engines/engine.py.224: train: epoch 62 | batch 1950/2304 [I 2022-10-27 10:37:21] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:20 [I 2022-10-27 10:37:21] ...trap/engines/engine.py.229: process: 0.55770 | load: 0.00243 [I 2022-10-27 10:37:21] ...trap/engines/engine.py.230: loss: 0.17540 [I 2022-10-27 10:37:21] ...optimizers/trijoint.py.151: total_norm: 15.560832 [I 2022-10-27 10:37:27] ...trap/engines/engine.py.224: train: epoch 62 | batch 1960/2304 [I 2022-10-27 10:37:27] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:03:13 [I 2022-10-27 10:37:27] ...trap/engines/engine.py.229: process: 0.56933 | load: 0.00245 [I 2022-10-27 10:37:27] ...trap/engines/engine.py.230: loss: 0.10410 [I 2022-10-27 10:37:27] ...optimizers/trijoint.py.151: total_norm: 14.874843 [I 2022-10-27 10:37:32] ...trap/engines/engine.py.224: train: epoch 62 | batch 1970/2304 [I 2022-10-27 10:37:32] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:03:06 [I 2022-10-27 10:37:32] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00259 [I 2022-10-27 10:37:32] ...trap/engines/engine.py.230: loss: 0.19273 [I 2022-10-27 10:37:32] ...optimizers/trijoint.py.151: total_norm: 23.464802 [I 2022-10-27 10:37:38] ...trap/engines/engine.py.224: train: epoch 62 | batch 1980/2304 [I 2022-10-27 10:37:38] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:59 [I 2022-10-27 10:37:38] ...trap/engines/engine.py.229: process: 0.54657 | load: 0.00242 [I 2022-10-27 10:37:38] ...trap/engines/engine.py.230: loss: 0.12185 [I 2022-10-27 10:37:38] ...optimizers/trijoint.py.151: total_norm: 12.574632 [I 2022-10-27 10:37:43] ...trap/engines/engine.py.224: train: epoch 62 | batch 1990/2304 [I 2022-10-27 10:37:43] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:53 [I 2022-10-27 10:37:43] ...trap/engines/engine.py.229: process: 0.53665 | load: 0.00245 [I 2022-10-27 10:37:43] ...trap/engines/engine.py.230: loss: 0.16343 [I 2022-10-27 10:37:43] ...optimizers/trijoint.py.151: total_norm: 19.505297 [I 2022-10-27 10:37:49] ...trap/engines/engine.py.224: train: epoch 62 | batch 2000/2304 [I 2022-10-27 10:37:49] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:47 [I 2022-10-27 10:37:49] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00239 [I 2022-10-27 10:37:49] ...trap/engines/engine.py.230: loss: 0.11517 [I 2022-10-27 10:37:49] ...optimizers/trijoint.py.151: total_norm: 10.667833 [I 2022-10-27 10:37:55] ...trap/engines/engine.py.224: train: epoch 62 | batch 2010/2304 [I 2022-10-27 10:37:55] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:45 [I 2022-10-27 10:37:55] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00245 [I 2022-10-27 10:37:55] ...trap/engines/engine.py.230: loss: 0.13492 [I 2022-10-27 10:37:55] ...optimizers/trijoint.py.151: total_norm: 29.073601 [I 2022-10-27 10:38:00] ...trap/engines/engine.py.224: train: epoch 62 | batch 2020/2304 [I 2022-10-27 10:38:00] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:37 [I 2022-10-27 10:38:00] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00241 [I 2022-10-27 10:38:00] ...trap/engines/engine.py.230: loss: 0.25049 [I 2022-10-27 10:38:00] ...optimizers/trijoint.py.151: total_norm: 110.243050 [I 2022-10-27 10:38:06] ...trap/engines/engine.py.224: train: epoch 62 | batch 2030/2304 [I 2022-10-27 10:38:06] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:32 [I 2022-10-27 10:38:06] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00242 [I 2022-10-27 10:38:06] ...trap/engines/engine.py.230: loss: 0.15440 [I 2022-10-27 10:38:06] ...optimizers/trijoint.py.151: total_norm: 16.827591 [I 2022-10-27 10:38:11] ...trap/engines/engine.py.224: train: epoch 62 | batch 2040/2304 [I 2022-10-27 10:38:11] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:28 [I 2022-10-27 10:38:11] ...trap/engines/engine.py.229: process: 0.56425 | load: 0.00241 [I 2022-10-27 10:38:11] ...trap/engines/engine.py.230: loss: 0.17196 [I 2022-10-27 10:38:11] ...optimizers/trijoint.py.151: total_norm: 9.744213 [I 2022-10-27 10:38:17] ...trap/engines/engine.py.224: train: epoch 62 | batch 2050/2304 [I 2022-10-27 10:38:17] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:22 [I 2022-10-27 10:38:17] ...trap/engines/engine.py.229: process: 0.54932 | load: 0.00249 [I 2022-10-27 10:38:17] ...trap/engines/engine.py.230: loss: 0.10331 [I 2022-10-27 10:38:17] ...optimizers/trijoint.py.151: total_norm: 8.571842 [I 2022-10-27 10:38:23] ...trap/engines/engine.py.224: train: epoch 62 | batch 2060/2304 [I 2022-10-27 10:38:23] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:02:16 [I 2022-10-27 10:38:23] ...trap/engines/engine.py.229: process: 0.54820 | load: 0.00245 [I 2022-10-27 10:38:23] ...trap/engines/engine.py.230: loss: 0.15665 [I 2022-10-27 10:38:23] ...optimizers/trijoint.py.151: total_norm: 13.185364 [I 2022-10-27 10:38:28] ...trap/engines/engine.py.224: train: epoch 62 | batch 2070/2304 [I 2022-10-27 10:38:28] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:02:13 [I 2022-10-27 10:38:28] ...trap/engines/engine.py.229: process: 0.54631 | load: 0.00243 [I 2022-10-27 10:38:28] ...trap/engines/engine.py.230: loss: 0.11026 [I 2022-10-27 10:38:28] ...optimizers/trijoint.py.151: total_norm: 7.848366 [I 2022-10-27 10:38:34] ...trap/engines/engine.py.224: train: epoch 62 | batch 2080/2304 [I 2022-10-27 10:38:34] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:02:03 [I 2022-10-27 10:38:34] ...trap/engines/engine.py.229: process: 0.53734 | load: 0.00242 [I 2022-10-27 10:38:34] ...trap/engines/engine.py.230: loss: 0.15118 [I 2022-10-27 10:38:34] ...optimizers/trijoint.py.151: total_norm: 13.979292 [I 2022-10-27 10:38:39] ...trap/engines/engine.py.224: train: epoch 62 | batch 2090/2304 [I 2022-10-27 10:38:39] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:59 [I 2022-10-27 10:38:39] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00240 [I 2022-10-27 10:38:39] ...trap/engines/engine.py.230: loss: 0.13705 [I 2022-10-27 10:38:39] ...optimizers/trijoint.py.151: total_norm: 16.411007 [I 2022-10-27 10:38:45] ...trap/engines/engine.py.224: train: epoch 62 | batch 2100/2304 [I 2022-10-27 10:38:45] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:54 [I 2022-10-27 10:38:45] ...trap/engines/engine.py.229: process: 0.55656 | load: 0.00243 [I 2022-10-27 10:38:45] ...trap/engines/engine.py.230: loss: 0.14910 [I 2022-10-27 10:38:45] ...optimizers/trijoint.py.151: total_norm: 11.979051 [I 2022-10-27 10:38:51] ...trap/engines/engine.py.224: train: epoch 62 | batch 2110/2304 [I 2022-10-27 10:38:51] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:47 [I 2022-10-27 10:38:51] ...trap/engines/engine.py.229: process: 0.54477 | load: 0.00243 [I 2022-10-27 10:38:51] ...trap/engines/engine.py.230: loss: 0.10879 [I 2022-10-27 10:38:51] ...optimizers/trijoint.py.151: total_norm: 11.916160 [I 2022-10-27 10:38:56] ...trap/engines/engine.py.224: train: epoch 62 | batch 2120/2304 [I 2022-10-27 10:38:56] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:42 [I 2022-10-27 10:38:56] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00242 [I 2022-10-27 10:38:56] ...trap/engines/engine.py.230: loss: 0.12054 [I 2022-10-27 10:38:56] ...optimizers/trijoint.py.151: total_norm: 21.609751 [I 2022-10-27 10:39:02] ...trap/engines/engine.py.224: train: epoch 62 | batch 2130/2304 [I 2022-10-27 10:39:02] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:41 [I 2022-10-27 10:39:02] ...trap/engines/engine.py.229: process: 0.67146 | load: 0.00241 [I 2022-10-27 10:39:02] ...trap/engines/engine.py.230: loss: 0.15735 [I 2022-10-27 10:39:02] ...optimizers/trijoint.py.151: total_norm: 29.738636 [I 2022-10-27 10:39:08] ...trap/engines/engine.py.224: train: epoch 62 | batch 2140/2304 [I 2022-10-27 10:39:08] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:31 [I 2022-10-27 10:39:08] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00246 [I 2022-10-27 10:39:08] ...trap/engines/engine.py.230: loss: 0.12019 [I 2022-10-27 10:39:08] ...optimizers/trijoint.py.151: total_norm: 16.368984 [I 2022-10-27 10:39:13] ...trap/engines/engine.py.224: train: epoch 62 | batch 2150/2304 [I 2022-10-27 10:39:13] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:25 [I 2022-10-27 10:39:13] ...trap/engines/engine.py.229: process: 0.54837 | load: 0.00242 [I 2022-10-27 10:39:13] ...trap/engines/engine.py.230: loss: 0.11962 [I 2022-10-27 10:39:13] ...optimizers/trijoint.py.151: total_norm: 8.712249 [I 2022-10-27 10:39:19] ...trap/engines/engine.py.224: train: epoch 62 | batch 2160/2304 [I 2022-10-27 10:39:19] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:20 [I 2022-10-27 10:39:19] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00244 [I 2022-10-27 10:39:19] ...trap/engines/engine.py.230: loss: 0.10513 [I 2022-10-27 10:39:19] ...optimizers/trijoint.py.151: total_norm: 20.224262 [I 2022-10-27 10:39:24] ...trap/engines/engine.py.224: train: epoch 62 | batch 2170/2304 [I 2022-10-27 10:39:24] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:14 [I 2022-10-27 10:39:24] ...trap/engines/engine.py.229: process: 0.55547 | load: 0.00240 [I 2022-10-27 10:39:24] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-27 10:39:24] ...optimizers/trijoint.py.151: total_norm: 18.651865 [I 2022-10-27 10:39:30] ...trap/engines/engine.py.224: train: epoch 62 | batch 2180/2304 [I 2022-10-27 10:39:30] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:08 [I 2022-10-27 10:39:30] ...trap/engines/engine.py.229: process: 0.55339 | load: 0.00242 [I 2022-10-27 10:39:30] ...trap/engines/engine.py.230: loss: 0.15239 [I 2022-10-27 10:39:30] ...optimizers/trijoint.py.151: total_norm: 18.763365 [I 2022-10-27 10:39:35] ...trap/engines/engine.py.224: train: epoch 62 | batch 2190/2304 [I 2022-10-27 10:39:35] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:01:03 [I 2022-10-27 10:39:35] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00243 [I 2022-10-27 10:39:35] ...trap/engines/engine.py.230: loss: 0.20770 [I 2022-10-27 10:39:35] ...optimizers/trijoint.py.151: total_norm: 19.528368 [I 2022-10-27 10:39:41] ...trap/engines/engine.py.224: train: epoch 62 | batch 2200/2304 [I 2022-10-27 10:39:41] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:58 [I 2022-10-27 10:39:41] ...trap/engines/engine.py.229: process: 0.55372 | load: 0.00248 [I 2022-10-27 10:39:41] ...trap/engines/engine.py.230: loss: 0.19885 [I 2022-10-27 10:39:41] ...optimizers/trijoint.py.151: total_norm: 16.898659 [I 2022-10-27 10:39:47] ...trap/engines/engine.py.224: train: epoch 62 | batch 2210/2304 [I 2022-10-27 10:39:47] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:52 [I 2022-10-27 10:39:47] ...trap/engines/engine.py.229: process: 0.54529 | load: 0.00243 [I 2022-10-27 10:39:47] ...trap/engines/engine.py.230: loss: 0.12052 [I 2022-10-27 10:39:47] ...optimizers/trijoint.py.151: total_norm: 14.376557 [I 2022-10-27 10:39:52] ...trap/engines/engine.py.224: train: epoch 62 | batch 2220/2304 [I 2022-10-27 10:39:52] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:46 [I 2022-10-27 10:39:52] ...trap/engines/engine.py.229: process: 0.55635 | load: 0.00248 [I 2022-10-27 10:39:52] ...trap/engines/engine.py.230: loss: 0.11981 [I 2022-10-27 10:39:52] ...optimizers/trijoint.py.151: total_norm: 20.497824 [I 2022-10-27 10:39:58] ...trap/engines/engine.py.224: train: epoch 62 | batch 2230/2304 [I 2022-10-27 10:39:58] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:41 [I 2022-10-27 10:39:58] ...trap/engines/engine.py.229: process: 0.54786 | load: 0.00249 [I 2022-10-27 10:39:58] ...trap/engines/engine.py.230: loss: 0.15701 [I 2022-10-27 10:39:58] ...optimizers/trijoint.py.151: total_norm: 16.812420 [I 2022-10-27 10:40:03] ...trap/engines/engine.py.224: train: epoch 62 | batch 2240/2304 [I 2022-10-27 10:40:03] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:35 [I 2022-10-27 10:40:03] ...trap/engines/engine.py.229: process: 0.55505 | load: 0.00246 [I 2022-10-27 10:40:03] ...trap/engines/engine.py.230: loss: 0.13281 [I 2022-10-27 10:40:03] ...optimizers/trijoint.py.151: total_norm: 18.192408 [I 2022-10-27 10:40:09] ...trap/engines/engine.py.224: train: epoch 62 | batch 2250/2304 [I 2022-10-27 10:40:09] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:30 [I 2022-10-27 10:40:09] ...trap/engines/engine.py.229: process: 0.56375 | load: 0.00243 [I 2022-10-27 10:40:09] ...trap/engines/engine.py.230: loss: 0.26831 [I 2022-10-27 10:40:09] ...optimizers/trijoint.py.151: total_norm: 15.865292 [I 2022-10-27 10:40:15] ...trap/engines/engine.py.224: train: epoch 62 | batch 2260/2304 [I 2022-10-27 10:40:15] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:24 [I 2022-10-27 10:40:15] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00244 [I 2022-10-27 10:40:15] ...trap/engines/engine.py.230: loss: 0.28823 [I 2022-10-27 10:40:15] ...optimizers/trijoint.py.151: total_norm: 24.780388 [I 2022-10-27 10:40:20] ...trap/engines/engine.py.224: train: epoch 62 | batch 2270/2304 [I 2022-10-27 10:40:20] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:19 [I 2022-10-27 10:40:20] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00245 [I 2022-10-27 10:40:20] ...trap/engines/engine.py.230: loss: 0.10707 [I 2022-10-27 10:40:20] ...optimizers/trijoint.py.151: total_norm: 13.385393 [I 2022-10-27 10:40:26] ...trap/engines/engine.py.224: train: epoch 62 | batch 2280/2304 [I 2022-10-27 10:40:26] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:13 [I 2022-10-27 10:40:26] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00245 [I 2022-10-27 10:40:26] ...trap/engines/engine.py.230: loss: 0.18924 [I 2022-10-27 10:40:26] ...optimizers/trijoint.py.151: total_norm: 33.354805 [I 2022-10-27 10:40:31] ...trap/engines/engine.py.224: train: epoch 62 | batch 2290/2304 [I 2022-10-27 10:40:31] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:00:07 [I 2022-10-27 10:40:31] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00242 [I 2022-10-27 10:40:31] ...trap/engines/engine.py.230: loss: 0.14019 [I 2022-10-27 10:40:31] ...optimizers/trijoint.py.151: total_norm: 17.133635 [I 2022-10-27 10:40:37] ...trap/engines/engine.py.224: train: epoch 62 | batch 2300/2304 [I 2022-10-27 10:40:37] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:02 [I 2022-10-27 10:40:37] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00014 [I 2022-10-27 10:40:37] ...trap/engines/engine.py.230: loss: 0.12481 [I 2022-10-27 10:40:37] ...optimizers/trijoint.py.151: total_norm: 12.837965 [I 2022-10-27 10:40:39] ...trap/engines/engine.py.224: train: epoch 62 | batch 2304/2304 [I 2022-10-27 10:40:39] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:00 [I 2022-10-27 10:40:39] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00015 [I 2022-10-27 10:40:39] ...trap/engines/engine.py.230: loss: 0.13589 [I 2022-10-27 10:40:39] ...optimizers/trijoint.py.151: total_norm: 9.166397 [S 2022-10-27 10:40:41] ...trap/engines/engine.py.129: train_epoch.epoch: 62 [S 2022-10-27 10:40:41] ...trap/engines/engine.py.129: train_epoch.loss: 0.15241823591081802 [I 2022-10-27 10:40:41] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 10:40:41] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 62 [I 2022-10-27 10:40:41] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 10:40:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 10:40:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 10:40:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 10:40:45] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 10:40:46] ...trap/engines/engine.py.321: eval: epoch 62 | batch 0/510 [I 2022-10-27 10:40:46] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:37:01 [I 2022-10-27 10:40:46] ...trap/engines/engine.py.326: process: 4.35600 | load: 2.92838 [I 2022-10-27 10:40:49] ...trap/engines/engine.py.321: eval: epoch 62 | batch 10/510 [I 2022-10-27 10:40:49] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:19 [I 2022-10-27 10:40:49] ...trap/engines/engine.py.326: process: 0.32531 | load: 0.00017 [I 2022-10-27 10:40:52] ...trap/engines/engine.py.321: eval: epoch 62 | batch 20/510 [I 2022-10-27 10:40:52] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:05 [I 2022-10-27 10:40:52] ...trap/engines/engine.py.326: process: 0.33352 | load: 0.00016 [I 2022-10-27 10:40:56] ...trap/engines/engine.py.321: eval: epoch 62 | batch 30/510 [I 2022-10-27 10:40:56] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:43 [I 2022-10-27 10:40:56] ...trap/engines/engine.py.326: process: 0.32907 | load: 0.00017 [I 2022-10-27 10:40:59] ...trap/engines/engine.py.321: eval: epoch 62 | batch 40/510 [I 2022-10-27 10:40:59] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:35 [I 2022-10-27 10:40:59] ...trap/engines/engine.py.326: process: 0.32887 | load: 0.00019 [I 2022-10-27 10:41:02] ...trap/engines/engine.py.321: eval: epoch 62 | batch 50/510 [I 2022-10-27 10:41:02] ...trap/engines/engine.py.325: elapsed: 0:00:20 | left: 0:02:33 [I 2022-10-27 10:41:02] ...trap/engines/engine.py.326: process: 0.33956 | load: 0.00040 [I 2022-10-27 10:41:06] ...trap/engines/engine.py.321: eval: epoch 62 | batch 60/510 [I 2022-10-27 10:41:06] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:27 [I 2022-10-27 10:41:06] ...trap/engines/engine.py.326: process: 0.32501 | load: 0.00021 [I 2022-10-27 10:41:09] ...trap/engines/engine.py.321: eval: epoch 62 | batch 70/510 [I 2022-10-27 10:41:09] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:29 [I 2022-10-27 10:41:09] ...trap/engines/engine.py.326: process: 0.35257 | load: 0.00021 [I 2022-10-27 10:41:12] ...trap/engines/engine.py.321: eval: epoch 62 | batch 80/510 [I 2022-10-27 10:41:12] ...trap/engines/engine.py.325: elapsed: 0:00:30 | left: 0:02:23 [I 2022-10-27 10:41:12] ...trap/engines/engine.py.326: process: 0.33291 | load: 0.00019 [I 2022-10-27 10:41:16] ...trap/engines/engine.py.321: eval: epoch 62 | batch 90/510 [I 2022-10-27 10:41:16] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:21 [I 2022-10-27 10:41:16] ...trap/engines/engine.py.326: process: 0.33606 | load: 0.00021 [I 2022-10-27 10:41:19] ...trap/engines/engine.py.321: eval: epoch 62 | batch 100/510 [I 2022-10-27 10:41:19] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:15 [I 2022-10-27 10:41:19] ...trap/engines/engine.py.326: process: 0.31713 | load: 0.00018 [I 2022-10-27 10:41:22] ...trap/engines/engine.py.321: eval: epoch 62 | batch 110/510 [I 2022-10-27 10:41:22] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:59 [I 2022-10-27 10:41:22] ...trap/engines/engine.py.326: process: 0.32426 | load: 0.00014 [I 2022-10-27 10:41:25] ...trap/engines/engine.py.321: eval: epoch 62 | batch 120/510 [I 2022-10-27 10:41:25] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:53 [I 2022-10-27 10:41:25] ...trap/engines/engine.py.326: process: 0.29863 | load: 0.00015 [I 2022-10-27 10:41:28] ...trap/engines/engine.py.321: eval: epoch 62 | batch 130/510 [I 2022-10-27 10:41:28] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:49 [I 2022-10-27 10:41:28] ...trap/engines/engine.py.326: process: 0.28207 | load: 0.00015 [I 2022-10-27 10:41:31] ...trap/engines/engine.py.321: eval: epoch 62 | batch 140/510 [I 2022-10-27 10:41:31] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:46 [I 2022-10-27 10:41:31] ...trap/engines/engine.py.326: process: 0.29114 | load: 0.00014 [I 2022-10-27 10:41:34] ...trap/engines/engine.py.321: eval: epoch 62 | batch 150/510 [I 2022-10-27 10:41:34] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:43 [I 2022-10-27 10:41:34] ...trap/engines/engine.py.326: process: 0.29096 | load: 0.00014 [I 2022-10-27 10:41:37] ...trap/engines/engine.py.321: eval: epoch 62 | batch 160/510 [I 2022-10-27 10:41:37] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 10:41:37] ...trap/engines/engine.py.326: process: 0.28499 | load: 0.00015 [I 2022-10-27 10:41:39] ...trap/engines/engine.py.321: eval: epoch 62 | batch 170/510 [I 2022-10-27 10:41:39] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-27 10:41:39] ...trap/engines/engine.py.326: process: 0.28627 | load: 0.00016 [I 2022-10-27 10:41:42] ...trap/engines/engine.py.321: eval: epoch 62 | batch 180/510 [I 2022-10-27 10:41:42] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:34 [I 2022-10-27 10:41:42] ...trap/engines/engine.py.326: process: 0.28281 | load: 0.00021 [I 2022-10-27 10:41:45] ...trap/engines/engine.py.321: eval: epoch 62 | batch 190/510 [I 2022-10-27 10:41:45] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:31 [I 2022-10-27 10:41:45] ...trap/engines/engine.py.326: process: 0.28719 | load: 0.00016 [I 2022-10-27 10:41:48] ...trap/engines/engine.py.321: eval: epoch 62 | batch 200/510 [I 2022-10-27 10:41:48] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 10:41:48] ...trap/engines/engine.py.326: process: 0.29145 | load: 0.00015 [I 2022-10-27 10:41:51] ...trap/engines/engine.py.321: eval: epoch 62 | batch 210/510 [I 2022-10-27 10:41:51] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 10:41:51] ...trap/engines/engine.py.326: process: 0.28933 | load: 0.00019 [I 2022-10-27 10:41:54] ...trap/engines/engine.py.321: eval: epoch 62 | batch 220/510 [I 2022-10-27 10:41:54] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 10:41:54] ...trap/engines/engine.py.326: process: 0.28992 | load: 0.00015 [I 2022-10-27 10:41:57] ...trap/engines/engine.py.321: eval: epoch 62 | batch 230/510 [I 2022-10-27 10:41:57] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:21 [I 2022-10-27 10:41:57] ...trap/engines/engine.py.326: process: 0.28796 | load: 0.00020 [I 2022-10-27 10:42:00] ...trap/engines/engine.py.321: eval: epoch 62 | batch 240/510 [I 2022-10-27 10:42:00] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 10:42:00] ...trap/engines/engine.py.326: process: 0.28351 | load: 0.00015 [I 2022-10-27 10:42:03] ...trap/engines/engine.py.321: eval: epoch 62 | batch 250/510 [I 2022-10-27 10:42:03] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 10:42:03] ...trap/engines/engine.py.326: process: 0.28596 | load: 0.00016 [I 2022-10-27 10:42:05] ...trap/engines/engine.py.321: eval: epoch 62 | batch 260/510 [I 2022-10-27 10:42:05] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:12 [I 2022-10-27 10:42:05] ...trap/engines/engine.py.326: process: 0.28733 | load: 0.00017 [I 2022-10-27 10:42:08] ...trap/engines/engine.py.321: eval: epoch 62 | batch 270/510 [I 2022-10-27 10:42:08] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 10:42:08] ...trap/engines/engine.py.326: process: 0.29154 | load: 0.00023 [I 2022-10-27 10:42:11] ...trap/engines/engine.py.321: eval: epoch 62 | batch 280/510 [I 2022-10-27 10:42:11] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 10:42:11] ...trap/engines/engine.py.326: process: 0.28357 | load: 0.00017 [I 2022-10-27 10:42:14] ...trap/engines/engine.py.321: eval: epoch 62 | batch 290/510 [I 2022-10-27 10:42:14] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 10:42:14] ...trap/engines/engine.py.326: process: 0.28655 | load: 0.00017 [I 2022-10-27 10:42:17] ...trap/engines/engine.py.321: eval: epoch 62 | batch 300/510 [I 2022-10-27 10:42:17] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 10:42:17] ...trap/engines/engine.py.326: process: 0.28622 | load: 0.00015 [I 2022-10-27 10:42:20] ...trap/engines/engine.py.321: eval: epoch 62 | batch 310/510 [I 2022-10-27 10:42:20] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:58 [I 2022-10-27 10:42:20] ...trap/engines/engine.py.326: process: 0.29077 | load: 0.00016 [I 2022-10-27 10:42:23] ...trap/engines/engine.py.321: eval: epoch 62 | batch 320/510 [I 2022-10-27 10:42:23] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:55 [I 2022-10-27 10:42:23] ...trap/engines/engine.py.326: process: 0.28434 | load: 0.00015 [I 2022-10-27 10:42:26] ...trap/engines/engine.py.321: eval: epoch 62 | batch 330/510 [I 2022-10-27 10:42:26] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:52 [I 2022-10-27 10:42:26] ...trap/engines/engine.py.326: process: 0.28731 | load: 0.00020 [I 2022-10-27 10:42:29] ...trap/engines/engine.py.321: eval: epoch 62 | batch 340/510 [I 2022-10-27 10:42:29] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 10:42:29] ...trap/engines/engine.py.326: process: 0.28937 | load: 0.00015 [I 2022-10-27 10:42:32] ...trap/engines/engine.py.321: eval: epoch 62 | batch 350/510 [I 2022-10-27 10:42:32] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 10:42:32] ...trap/engines/engine.py.326: process: 0.29746 | load: 0.00014 [I 2022-10-27 10:42:34] ...trap/engines/engine.py.321: eval: epoch 62 | batch 360/510 [I 2022-10-27 10:42:34] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:43 [I 2022-10-27 10:42:34] ...trap/engines/engine.py.326: process: 0.28627 | load: 0.00017 [I 2022-10-27 10:42:37] ...trap/engines/engine.py.321: eval: epoch 62 | batch 370/510 [I 2022-10-27 10:42:37] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 10:42:37] ...trap/engines/engine.py.326: process: 0.28651 | load: 0.00015 [I 2022-10-27 10:42:40] ...trap/engines/engine.py.321: eval: epoch 62 | batch 380/510 [I 2022-10-27 10:42:40] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 10:42:40] ...trap/engines/engine.py.326: process: 0.28388 | load: 0.00015 [I 2022-10-27 10:42:43] ...trap/engines/engine.py.321: eval: epoch 62 | batch 390/510 [I 2022-10-27 10:42:43] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 10:42:43] ...trap/engines/engine.py.326: process: 0.29351 | load: 0.00016 [I 2022-10-27 10:42:46] ...trap/engines/engine.py.321: eval: epoch 62 | batch 400/510 [I 2022-10-27 10:42:46] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 10:42:46] ...trap/engines/engine.py.326: process: 0.28911 | load: 0.00017 [I 2022-10-27 10:42:49] ...trap/engines/engine.py.321: eval: epoch 62 | batch 410/510 [I 2022-10-27 10:42:49] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 10:42:49] ...trap/engines/engine.py.326: process: 0.28909 | load: 0.00016 [I 2022-10-27 10:42:52] ...trap/engines/engine.py.321: eval: epoch 62 | batch 420/510 [I 2022-10-27 10:42:52] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 10:42:52] ...trap/engines/engine.py.326: process: 0.28425 | load: 0.00016 [I 2022-10-27 10:42:55] ...trap/engines/engine.py.321: eval: epoch 62 | batch 430/510 [I 2022-10-27 10:42:55] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:22 [I 2022-10-27 10:42:55] ...trap/engines/engine.py.326: process: 0.28487 | load: 0.00016 [I 2022-10-27 10:42:58] ...trap/engines/engine.py.321: eval: epoch 62 | batch 440/510 [I 2022-10-27 10:42:58] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 10:42:58] ...trap/engines/engine.py.326: process: 0.28663 | load: 0.00015 [I 2022-10-27 10:43:00] ...trap/engines/engine.py.321: eval: epoch 62 | batch 450/510 [I 2022-10-27 10:43:00] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 10:43:00] ...trap/engines/engine.py.326: process: 0.28447 | load: 0.00020 [I 2022-10-27 10:43:03] ...trap/engines/engine.py.321: eval: epoch 62 | batch 460/510 [I 2022-10-27 10:43:03] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 10:43:03] ...trap/engines/engine.py.326: process: 0.28705 | load: 0.00016 [I 2022-10-27 10:43:06] ...trap/engines/engine.py.321: eval: epoch 62 | batch 470/510 [I 2022-10-27 10:43:06] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 10:43:06] ...trap/engines/engine.py.326: process: 0.28559 | load: 0.00016 [I 2022-10-27 10:43:09] ...trap/engines/engine.py.321: eval: epoch 62 | batch 480/510 [I 2022-10-27 10:43:09] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 10:43:09] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00019 [I 2022-10-27 10:43:12] ...trap/engines/engine.py.321: eval: epoch 62 | batch 490/510 [I 2022-10-27 10:43:12] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 10:43:12] ...trap/engines/engine.py.326: process: 0.28854 | load: 0.00017 [I 2022-10-27 10:43:15] ...trap/engines/engine.py.321: eval: epoch 62 | batch 500/510 [I 2022-10-27 10:43:15] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 10:43:15] ...trap/engines/engine.py.326: process: 0.28517 | load: 0.00016 [I 2022-10-27 10:43:18] ...trap/engines/engine.py.321: eval: epoch 62 | batch 510/510 [I 2022-10-27 10:43:18] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 10:43:18] ...trap/engines/engine.py.326: process: 0.28615 | load: 0.00017 [S 2022-10-27 10:43:18] ...trap/engines/engine.py.132: eval_epoch.epoch: 62 [I 2022-10-27 10:43:18] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.709400 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.904700 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.938400 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.697500 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.901000 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934000 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010121 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008462 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007977 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015068 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010109 [S 2022-10-27 10:43:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008602 [I 2022-10-27 10:43:24] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 10:43:24] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 10:43:24] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 10:43:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 10:43:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 10:43:27] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 10:43:29] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 10:43:34] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 10:43:34] ...trap/engines/engine.py.163: Training model on trainset for epoch 63 [I 2022-10-27 10:43:34] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 10:43:37] ...trap/engines/engine.py.224: train: epoch 63 | batch 0/2304 [I 2022-10-27 10:43:37] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:44 [I 2022-10-27 10:43:37] ...trap/engines/engine.py.229: process: 2.64936 | load: 1.96576 [I 2022-10-27 10:43:37] ...trap/engines/engine.py.230: loss: 0.13054 [I 2022-10-27 10:43:37] ...optimizers/trijoint.py.151: total_norm: 9.219667 [I 2022-10-27 10:43:43] ...trap/engines/engine.py.224: train: epoch 63 | batch 10/2304 [I 2022-10-27 10:43:43] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:59 [I 2022-10-27 10:43:43] ...trap/engines/engine.py.229: process: 0.56718 | load: 0.00392 [I 2022-10-27 10:43:43] ...trap/engines/engine.py.230: loss: 0.12291 [I 2022-10-27 10:43:43] ...optimizers/trijoint.py.151: total_norm: 25.293983 [I 2022-10-27 10:43:48] ...trap/engines/engine.py.224: train: epoch 63 | batch 20/2304 [I 2022-10-27 10:43:48] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:07 [I 2022-10-27 10:43:48] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00355 [I 2022-10-27 10:43:48] ...trap/engines/engine.py.230: loss: 0.14361 [I 2022-10-27 10:43:48] ...optimizers/trijoint.py.151: total_norm: 18.300034 [I 2022-10-27 10:43:54] ...trap/engines/engine.py.224: train: epoch 63 | batch 30/2304 [I 2022-10-27 10:43:54] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:19 [I 2022-10-27 10:43:54] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00330 [I 2022-10-27 10:43:54] ...trap/engines/engine.py.230: loss: 0.14847 [I 2022-10-27 10:43:54] ...optimizers/trijoint.py.151: total_norm: 28.423502 [I 2022-10-27 10:43:59] ...trap/engines/engine.py.224: train: epoch 63 | batch 40/2304 [I 2022-10-27 10:43:59] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:03 [I 2022-10-27 10:43:59] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00287 [I 2022-10-27 10:43:59] ...trap/engines/engine.py.230: loss: 0.19063 [I 2022-10-27 10:43:59] ...optimizers/trijoint.py.151: total_norm: 34.127068 [I 2022-10-27 10:44:05] ...trap/engines/engine.py.224: train: epoch 63 | batch 50/2304 [I 2022-10-27 10:44:05] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:52 [I 2022-10-27 10:44:05] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00265 [I 2022-10-27 10:44:05] ...trap/engines/engine.py.230: loss: 0.11003 [I 2022-10-27 10:44:05] ...optimizers/trijoint.py.151: total_norm: 10.962454 [I 2022-10-27 10:44:11] ...trap/engines/engine.py.224: train: epoch 63 | batch 60/2304 [I 2022-10-27 10:44:11] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:47 [I 2022-10-27 10:44:11] ...trap/engines/engine.py.229: process: 0.55243 | load: 0.00270 [I 2022-10-27 10:44:11] ...trap/engines/engine.py.230: loss: 0.12035 [I 2022-10-27 10:44:11] ...optimizers/trijoint.py.151: total_norm: 12.555880 [I 2022-10-27 10:44:16] ...trap/engines/engine.py.224: train: epoch 63 | batch 70/2304 [I 2022-10-27 10:44:16] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:47 [I 2022-10-27 10:44:16] ...trap/engines/engine.py.229: process: 0.55637 | load: 0.00259 [I 2022-10-27 10:44:16] ...trap/engines/engine.py.230: loss: 0.13394 [I 2022-10-27 10:44:16] ...optimizers/trijoint.py.151: total_norm: 13.828876 [I 2022-10-27 10:44:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 80/2304 [I 2022-10-27 10:44:22] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:44 [I 2022-10-27 10:44:22] ...trap/engines/engine.py.229: process: 0.55501 | load: 0.00265 [I 2022-10-27 10:44:22] ...trap/engines/engine.py.230: loss: 0.10485 [I 2022-10-27 10:44:22] ...optimizers/trijoint.py.151: total_norm: 10.663033 [I 2022-10-27 10:44:27] ...trap/engines/engine.py.224: train: epoch 63 | batch 90/2304 [I 2022-10-27 10:44:27] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:25 [I 2022-10-27 10:44:27] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00272 [I 2022-10-27 10:44:27] ...trap/engines/engine.py.230: loss: 0.14236 [I 2022-10-27 10:44:27] ...optimizers/trijoint.py.151: total_norm: 14.948230 [I 2022-10-27 10:44:33] ...trap/engines/engine.py.224: train: epoch 63 | batch 100/2304 [I 2022-10-27 10:44:33] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:18 [I 2022-10-27 10:44:33] ...trap/engines/engine.py.229: process: 0.55141 | load: 0.00267 [I 2022-10-27 10:44:33] ...trap/engines/engine.py.230: loss: 0.14689 [I 2022-10-27 10:44:33] ...optimizers/trijoint.py.151: total_norm: 13.097628 [I 2022-10-27 10:44:38] ...trap/engines/engine.py.224: train: epoch 63 | batch 110/2304 [I 2022-10-27 10:44:38] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:18 [I 2022-10-27 10:44:38] ...trap/engines/engine.py.229: process: 0.55156 | load: 0.00254 [I 2022-10-27 10:44:38] ...trap/engines/engine.py.230: loss: 0.12317 [I 2022-10-27 10:44:38] ...optimizers/trijoint.py.151: total_norm: 10.085058 [I 2022-10-27 10:44:44] ...trap/engines/engine.py.224: train: epoch 63 | batch 120/2304 [I 2022-10-27 10:44:44] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:09 [I 2022-10-27 10:44:44] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00242 [I 2022-10-27 10:44:44] ...trap/engines/engine.py.230: loss: 0.14594 [I 2022-10-27 10:44:44] ...optimizers/trijoint.py.151: total_norm: 21.231333 [I 2022-10-27 10:44:50] ...trap/engines/engine.py.224: train: epoch 63 | batch 130/2304 [I 2022-10-27 10:44:50] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:10 [I 2022-10-27 10:44:50] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00261 [I 2022-10-27 10:44:50] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-27 10:44:50] ...optimizers/trijoint.py.151: total_norm: 9.729261 [I 2022-10-27 10:44:55] ...trap/engines/engine.py.224: train: epoch 63 | batch 140/2304 [I 2022-10-27 10:44:55] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:09 [I 2022-10-27 10:44:55] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00253 [I 2022-10-27 10:44:55] ...trap/engines/engine.py.230: loss: 0.14917 [I 2022-10-27 10:44:55] ...optimizers/trijoint.py.151: total_norm: 14.484130 [I 2022-10-27 10:45:01] ...trap/engines/engine.py.224: train: epoch 63 | batch 150/2304 [I 2022-10-27 10:45:01] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:56 [I 2022-10-27 10:45:01] ...trap/engines/engine.py.229: process: 0.55776 | load: 0.00275 [I 2022-10-27 10:45:01] ...trap/engines/engine.py.230: loss: 0.13260 [I 2022-10-27 10:45:01] ...optimizers/trijoint.py.151: total_norm: 17.028856 [I 2022-10-27 10:45:06] ...trap/engines/engine.py.224: train: epoch 63 | batch 160/2304 [I 2022-10-27 10:45:06] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:56 [I 2022-10-27 10:45:06] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00248 [I 2022-10-27 10:45:06] ...trap/engines/engine.py.230: loss: 0.17087 [I 2022-10-27 10:45:06] ...optimizers/trijoint.py.151: total_norm: 17.502741 [I 2022-10-27 10:45:12] ...trap/engines/engine.py.224: train: epoch 63 | batch 170/2304 [I 2022-10-27 10:45:12] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:38 [I 2022-10-27 10:45:12] ...trap/engines/engine.py.229: process: 0.54093 | load: 0.00247 [I 2022-10-27 10:45:12] ...trap/engines/engine.py.230: loss: 0.13844 [I 2022-10-27 10:45:12] ...optimizers/trijoint.py.151: total_norm: 15.748552 [I 2022-10-27 10:45:18] ...trap/engines/engine.py.224: train: epoch 63 | batch 180/2304 [I 2022-10-27 10:45:18] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:41 [I 2022-10-27 10:45:18] ...trap/engines/engine.py.229: process: 0.55243 | load: 0.00259 [I 2022-10-27 10:45:18] ...trap/engines/engine.py.230: loss: 0.13010 [I 2022-10-27 10:45:18] ...optimizers/trijoint.py.151: total_norm: 8.604506 [I 2022-10-27 10:45:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 190/2304 [I 2022-10-27 10:45:23] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:36 [I 2022-10-27 10:45:23] ...trap/engines/engine.py.229: process: 0.56163 | load: 0.00245 [I 2022-10-27 10:45:23] ...trap/engines/engine.py.230: loss: 0.13161 [I 2022-10-27 10:45:23] ...optimizers/trijoint.py.151: total_norm: 12.889534 [I 2022-10-27 10:45:29] ...trap/engines/engine.py.224: train: epoch 63 | batch 200/2304 [I 2022-10-27 10:45:29] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:31 [I 2022-10-27 10:45:29] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00263 [I 2022-10-27 10:45:29] ...trap/engines/engine.py.230: loss: 0.16463 [I 2022-10-27 10:45:29] ...optimizers/trijoint.py.151: total_norm: 31.159170 [I 2022-10-27 10:45:34] ...trap/engines/engine.py.224: train: epoch 63 | batch 210/2304 [I 2022-10-27 10:45:34] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:23 [I 2022-10-27 10:45:34] ...trap/engines/engine.py.229: process: 0.56377 | load: 0.00253 [I 2022-10-27 10:45:34] ...trap/engines/engine.py.230: loss: 0.16522 [I 2022-10-27 10:45:34] ...optimizers/trijoint.py.151: total_norm: 23.786390 [I 2022-10-27 10:45:40] ...trap/engines/engine.py.224: train: epoch 63 | batch 220/2304 [I 2022-10-27 10:45:40] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:23 [I 2022-10-27 10:45:40] ...trap/engines/engine.py.229: process: 0.56536 | load: 0.00247 [I 2022-10-27 10:45:40] ...trap/engines/engine.py.230: loss: 0.14082 [I 2022-10-27 10:45:40] ...optimizers/trijoint.py.151: total_norm: 24.637138 [I 2022-10-27 10:45:45] ...trap/engines/engine.py.224: train: epoch 63 | batch 230/2304 [I 2022-10-27 10:45:45] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:16 [I 2022-10-27 10:45:45] ...trap/engines/engine.py.229: process: 0.54782 | load: 0.00249 [I 2022-10-27 10:45:45] ...trap/engines/engine.py.230: loss: 0.16251 [I 2022-10-27 10:45:45] ...optimizers/trijoint.py.151: total_norm: 14.840785 [I 2022-10-27 10:45:51] ...trap/engines/engine.py.224: train: epoch 63 | batch 240/2304 [I 2022-10-27 10:45:51] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:11 [I 2022-10-27 10:45:51] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00257 [I 2022-10-27 10:45:51] ...trap/engines/engine.py.230: loss: 0.16385 [I 2022-10-27 10:45:51] ...optimizers/trijoint.py.151: total_norm: 26.184109 [I 2022-10-27 10:45:57] ...trap/engines/engine.py.224: train: epoch 63 | batch 250/2304 [I 2022-10-27 10:45:57] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:06 [I 2022-10-27 10:45:57] ...trap/engines/engine.py.229: process: 0.54746 | load: 0.00243 [I 2022-10-27 10:45:57] ...trap/engines/engine.py.230: loss: 0.15527 [I 2022-10-27 10:45:57] ...optimizers/trijoint.py.151: total_norm: 13.038521 [I 2022-10-27 10:46:02] ...trap/engines/engine.py.224: train: epoch 63 | batch 260/2304 [I 2022-10-27 10:46:02] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:15 [I 2022-10-27 10:46:02] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00243 [I 2022-10-27 10:46:02] ...trap/engines/engine.py.230: loss: 0.12347 [I 2022-10-27 10:46:02] ...optimizers/trijoint.py.151: total_norm: 19.322554 [I 2022-10-27 10:46:08] ...trap/engines/engine.py.224: train: epoch 63 | batch 270/2304 [I 2022-10-27 10:46:08] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:45 [I 2022-10-27 10:46:08] ...trap/engines/engine.py.229: process: 0.54673 | load: 0.00279 [I 2022-10-27 10:46:08] ...trap/engines/engine.py.230: loss: 0.12638 [I 2022-10-27 10:46:08] ...optimizers/trijoint.py.151: total_norm: 22.442371 [I 2022-10-27 10:46:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 280/2304 [I 2022-10-27 10:46:13] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:47 [I 2022-10-27 10:46:13] ...trap/engines/engine.py.229: process: 0.56538 | load: 0.00277 [I 2022-10-27 10:46:13] ...trap/engines/engine.py.230: loss: 0.21390 [I 2022-10-27 10:46:13] ...optimizers/trijoint.py.151: total_norm: 11.596482 [I 2022-10-27 10:46:19] ...trap/engines/engine.py.224: train: epoch 63 | batch 290/2304 [I 2022-10-27 10:46:19] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:42 [I 2022-10-27 10:46:19] ...trap/engines/engine.py.229: process: 0.56344 | load: 0.00270 [I 2022-10-27 10:46:19] ...trap/engines/engine.py.230: loss: 0.16910 [I 2022-10-27 10:46:19] ...optimizers/trijoint.py.151: total_norm: 21.179111 [I 2022-10-27 10:46:25] ...trap/engines/engine.py.224: train: epoch 63 | batch 300/2304 [I 2022-10-27 10:46:25] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:32 [I 2022-10-27 10:46:25] ...trap/engines/engine.py.229: process: 0.54571 | load: 0.00258 [I 2022-10-27 10:46:25] ...trap/engines/engine.py.230: loss: 0.11785 [I 2022-10-27 10:46:25] ...optimizers/trijoint.py.151: total_norm: 9.551771 [I 2022-10-27 10:46:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 310/2304 [I 2022-10-27 10:46:30] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:28 [I 2022-10-27 10:46:30] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00273 [I 2022-10-27 10:46:30] ...trap/engines/engine.py.230: loss: 0.10984 [I 2022-10-27 10:46:30] ...optimizers/trijoint.py.151: total_norm: 16.573330 [I 2022-10-27 10:46:36] ...trap/engines/engine.py.224: train: epoch 63 | batch 320/2304 [I 2022-10-27 10:46:36] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:24 [I 2022-10-27 10:46:36] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00275 [I 2022-10-27 10:46:36] ...trap/engines/engine.py.230: loss: 0.18013 [I 2022-10-27 10:46:36] ...optimizers/trijoint.py.151: total_norm: 27.890295 [I 2022-10-27 10:46:41] ...trap/engines/engine.py.224: train: epoch 63 | batch 330/2304 [I 2022-10-27 10:46:41] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:18 [I 2022-10-27 10:46:41] ...trap/engines/engine.py.229: process: 0.54564 | load: 0.00258 [I 2022-10-27 10:46:41] ...trap/engines/engine.py.230: loss: 0.10376 [I 2022-10-27 10:46:41] ...optimizers/trijoint.py.151: total_norm: 9.445294 [I 2022-10-27 10:46:47] ...trap/engines/engine.py.224: train: epoch 63 | batch 340/2304 [I 2022-10-27 10:46:47] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:28 [I 2022-10-27 10:46:47] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00282 [I 2022-10-27 10:46:47] ...trap/engines/engine.py.230: loss: 0.10650 [I 2022-10-27 10:46:47] ...optimizers/trijoint.py.151: total_norm: 54.225822 [I 2022-10-27 10:46:53] ...trap/engines/engine.py.224: train: epoch 63 | batch 350/2304 [I 2022-10-27 10:46:53] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:12 [I 2022-10-27 10:46:53] ...trap/engines/engine.py.229: process: 0.55950 | load: 0.00258 [I 2022-10-27 10:46:53] ...trap/engines/engine.py.230: loss: 0.14462 [I 2022-10-27 10:46:53] ...optimizers/trijoint.py.151: total_norm: 17.794420 [I 2022-10-27 10:46:58] ...trap/engines/engine.py.224: train: epoch 63 | batch 360/2304 [I 2022-10-27 10:46:58] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:59 [I 2022-10-27 10:46:58] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00257 [I 2022-10-27 10:46:58] ...trap/engines/engine.py.230: loss: 0.13392 [I 2022-10-27 10:46:58] ...optimizers/trijoint.py.151: total_norm: 9.728905 [I 2022-10-27 10:47:04] ...trap/engines/engine.py.224: train: epoch 63 | batch 370/2304 [I 2022-10-27 10:47:04] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:51 [I 2022-10-27 10:47:04] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00279 [I 2022-10-27 10:47:04] ...trap/engines/engine.py.230: loss: 0.19063 [I 2022-10-27 10:47:04] ...optimizers/trijoint.py.151: total_norm: 26.029387 [I 2022-10-27 10:47:09] ...trap/engines/engine.py.224: train: epoch 63 | batch 380/2304 [I 2022-10-27 10:47:09] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:48 [I 2022-10-27 10:47:09] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00290 [I 2022-10-27 10:47:09] ...trap/engines/engine.py.230: loss: 0.13893 [I 2022-10-27 10:47:09] ...optimizers/trijoint.py.151: total_norm: 14.607533 [I 2022-10-27 10:47:15] ...trap/engines/engine.py.224: train: epoch 63 | batch 390/2304 [I 2022-10-27 10:47:15] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:42 [I 2022-10-27 10:47:15] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00311 [I 2022-10-27 10:47:15] ...trap/engines/engine.py.230: loss: 0.12980 [I 2022-10-27 10:47:15] ...optimizers/trijoint.py.151: total_norm: 15.258106 [I 2022-10-27 10:47:20] ...trap/engines/engine.py.224: train: epoch 63 | batch 400/2304 [I 2022-10-27 10:47:20] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:55 [I 2022-10-27 10:47:20] ...trap/engines/engine.py.229: process: 0.57455 | load: 0.00267 [I 2022-10-27 10:47:20] ...trap/engines/engine.py.230: loss: 0.13774 [I 2022-10-27 10:47:20] ...optimizers/trijoint.py.151: total_norm: 34.543453 [I 2022-10-27 10:47:26] ...trap/engines/engine.py.224: train: epoch 63 | batch 410/2304 [I 2022-10-27 10:47:26] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:34 [I 2022-10-27 10:47:26] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00265 [I 2022-10-27 10:47:26] ...trap/engines/engine.py.230: loss: 0.13302 [I 2022-10-27 10:47:26] ...optimizers/trijoint.py.151: total_norm: 14.624333 [I 2022-10-27 10:47:32] ...trap/engines/engine.py.224: train: epoch 63 | batch 420/2304 [I 2022-10-27 10:47:32] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:32 [I 2022-10-27 10:47:32] ...trap/engines/engine.py.229: process: 0.57236 | load: 0.00270 [I 2022-10-27 10:47:32] ...trap/engines/engine.py.230: loss: 0.11874 [I 2022-10-27 10:47:32] ...optimizers/trijoint.py.151: total_norm: 22.421469 [I 2022-10-27 10:47:37] ...trap/engines/engine.py.224: train: epoch 63 | batch 430/2304 [I 2022-10-27 10:47:37] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:26 [I 2022-10-27 10:47:37] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00261 [I 2022-10-27 10:47:37] ...trap/engines/engine.py.230: loss: 0.13530 [I 2022-10-27 10:47:37] ...optimizers/trijoint.py.151: total_norm: 16.574133 [I 2022-10-27 10:47:43] ...trap/engines/engine.py.224: train: epoch 63 | batch 440/2304 [I 2022-10-27 10:47:43] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:24 [I 2022-10-27 10:47:43] ...trap/engines/engine.py.229: process: 0.58070 | load: 0.00261 [I 2022-10-27 10:47:43] ...trap/engines/engine.py.230: loss: 0.11358 [I 2022-10-27 10:47:43] ...optimizers/trijoint.py.151: total_norm: 10.720243 [I 2022-10-27 10:47:48] ...trap/engines/engine.py.224: train: epoch 63 | batch 450/2304 [I 2022-10-27 10:47:48] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:18 [I 2022-10-27 10:47:48] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00267 [I 2022-10-27 10:47:48] ...trap/engines/engine.py.230: loss: 0.10647 [I 2022-10-27 10:47:48] ...optimizers/trijoint.py.151: total_norm: 11.677073 [I 2022-10-27 10:47:54] ...trap/engines/engine.py.224: train: epoch 63 | batch 460/2304 [I 2022-10-27 10:47:54] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:14 [I 2022-10-27 10:47:54] ...trap/engines/engine.py.229: process: 0.57042 | load: 0.00256 [I 2022-10-27 10:47:54] ...trap/engines/engine.py.230: loss: 0.11417 [I 2022-10-27 10:47:54] ...optimizers/trijoint.py.151: total_norm: 15.726454 [I 2022-10-27 10:48:00] ...trap/engines/engine.py.224: train: epoch 63 | batch 470/2304 [I 2022-10-27 10:48:00] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:02 [I 2022-10-27 10:48:00] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00265 [I 2022-10-27 10:48:00] ...trap/engines/engine.py.230: loss: 0.15672 [I 2022-10-27 10:48:00] ...optimizers/trijoint.py.151: total_norm: 11.834767 [I 2022-10-27 10:48:05] ...trap/engines/engine.py.224: train: epoch 63 | batch 480/2304 [I 2022-10-27 10:48:05] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:17:21 [I 2022-10-27 10:48:05] ...trap/engines/engine.py.229: process: 0.60993 | load: 0.00253 [I 2022-10-27 10:48:05] ...trap/engines/engine.py.230: loss: 0.18523 [I 2022-10-27 10:48:05] ...optimizers/trijoint.py.151: total_norm: 13.219841 [I 2022-10-27 10:48:11] ...trap/engines/engine.py.224: train: epoch 63 | batch 490/2304 [I 2022-10-27 10:48:11] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:55 [I 2022-10-27 10:48:11] ...trap/engines/engine.py.229: process: 0.56519 | load: 0.00253 [I 2022-10-27 10:48:11] ...trap/engines/engine.py.230: loss: 0.13230 [I 2022-10-27 10:48:11] ...optimizers/trijoint.py.151: total_norm: 13.318579 [I 2022-10-27 10:48:16] ...trap/engines/engine.py.224: train: epoch 63 | batch 500/2304 [I 2022-10-27 10:48:16] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:49 [I 2022-10-27 10:48:16] ...trap/engines/engine.py.229: process: 0.56418 | load: 0.00258 [I 2022-10-27 10:48:16] ...trap/engines/engine.py.230: loss: 0.11288 [I 2022-10-27 10:48:16] ...optimizers/trijoint.py.151: total_norm: 9.885904 [I 2022-10-27 10:48:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 510/2304 [I 2022-10-27 10:48:22] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:39 [I 2022-10-27 10:48:22] ...trap/engines/engine.py.229: process: 0.55688 | load: 0.00259 [I 2022-10-27 10:48:22] ...trap/engines/engine.py.230: loss: 0.12636 [I 2022-10-27 10:48:22] ...optimizers/trijoint.py.151: total_norm: 17.262154 [I 2022-10-27 10:48:28] ...trap/engines/engine.py.224: train: epoch 63 | batch 520/2304 [I 2022-10-27 10:48:28] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:37 [I 2022-10-27 10:48:28] ...trap/engines/engine.py.229: process: 0.56242 | load: 0.00273 [I 2022-10-27 10:48:28] ...trap/engines/engine.py.230: loss: 0.13904 [I 2022-10-27 10:48:28] ...optimizers/trijoint.py.151: total_norm: 10.428559 [I 2022-10-27 10:48:33] ...trap/engines/engine.py.224: train: epoch 63 | batch 530/2304 [I 2022-10-27 10:48:33] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:31 [I 2022-10-27 10:48:33] ...trap/engines/engine.py.229: process: 0.54380 | load: 0.00250 [I 2022-10-27 10:48:33] ...trap/engines/engine.py.230: loss: 0.15275 [I 2022-10-27 10:48:33] ...optimizers/trijoint.py.151: total_norm: 12.608442 [I 2022-10-27 10:48:39] ...trap/engines/engine.py.224: train: epoch 63 | batch 540/2304 [I 2022-10-27 10:48:39] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:17 [I 2022-10-27 10:48:39] ...trap/engines/engine.py.229: process: 0.55799 | load: 0.00258 [I 2022-10-27 10:48:39] ...trap/engines/engine.py.230: loss: 0.11217 [I 2022-10-27 10:48:39] ...optimizers/trijoint.py.151: total_norm: 12.317895 [I 2022-10-27 10:48:44] ...trap/engines/engine.py.224: train: epoch 63 | batch 550/2304 [I 2022-10-27 10:48:44] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:17 [I 2022-10-27 10:48:44] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00252 [I 2022-10-27 10:48:44] ...trap/engines/engine.py.230: loss: 0.12176 [I 2022-10-27 10:48:44] ...optimizers/trijoint.py.151: total_norm: 14.486503 [I 2022-10-27 10:48:50] ...trap/engines/engine.py.224: train: epoch 63 | batch 560/2304 [I 2022-10-27 10:48:50] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:14 [I 2022-10-27 10:48:50] ...trap/engines/engine.py.229: process: 0.56237 | load: 0.00260 [I 2022-10-27 10:48:50] ...trap/engines/engine.py.230: loss: 0.12856 [I 2022-10-27 10:48:50] ...optimizers/trijoint.py.151: total_norm: 8.995177 [I 2022-10-27 10:48:56] ...trap/engines/engine.py.224: train: epoch 63 | batch 570/2304 [I 2022-10-27 10:48:56] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:03 [I 2022-10-27 10:48:56] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00284 [I 2022-10-27 10:48:56] ...trap/engines/engine.py.230: loss: 0.11292 [I 2022-10-27 10:48:56] ...optimizers/trijoint.py.151: total_norm: 32.003963 [I 2022-10-27 10:49:01] ...trap/engines/engine.py.224: train: epoch 63 | batch 580/2304 [I 2022-10-27 10:49:01] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:03 [I 2022-10-27 10:49:01] ...trap/engines/engine.py.229: process: 0.56134 | load: 0.00286 [I 2022-10-27 10:49:01] ...trap/engines/engine.py.230: loss: 0.15417 [I 2022-10-27 10:49:01] ...optimizers/trijoint.py.151: total_norm: 21.929501 [I 2022-10-27 10:49:07] ...trap/engines/engine.py.224: train: epoch 63 | batch 590/2304 [I 2022-10-27 10:49:07] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:53 [I 2022-10-27 10:49:07] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00262 [I 2022-10-27 10:49:07] ...trap/engines/engine.py.230: loss: 0.13556 [I 2022-10-27 10:49:07] ...optimizers/trijoint.py.151: total_norm: 15.410486 [I 2022-10-27 10:49:12] ...trap/engines/engine.py.224: train: epoch 63 | batch 600/2304 [I 2022-10-27 10:49:12] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:47 [I 2022-10-27 10:49:12] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00253 [I 2022-10-27 10:49:12] ...trap/engines/engine.py.230: loss: 0.11323 [I 2022-10-27 10:49:12] ...optimizers/trijoint.py.151: total_norm: 12.838699 [I 2022-10-27 10:49:18] ...trap/engines/engine.py.224: train: epoch 63 | batch 610/2304 [I 2022-10-27 10:49:18] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:44 [I 2022-10-27 10:49:18] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00249 [I 2022-10-27 10:49:18] ...trap/engines/engine.py.230: loss: 0.14902 [I 2022-10-27 10:49:18] ...optimizers/trijoint.py.151: total_norm: 13.279199 [I 2022-10-27 10:49:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 620/2304 [I 2022-10-27 10:49:23] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:39 [I 2022-10-27 10:49:23] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00247 [I 2022-10-27 10:49:23] ...trap/engines/engine.py.230: loss: 0.17086 [I 2022-10-27 10:49:23] ...optimizers/trijoint.py.151: total_norm: 18.191820 [I 2022-10-27 10:49:29] ...trap/engines/engine.py.224: train: epoch 63 | batch 630/2304 [I 2022-10-27 10:49:29] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:40 [I 2022-10-27 10:49:29] ...trap/engines/engine.py.229: process: 0.57109 | load: 0.00264 [I 2022-10-27 10:49:29] ...trap/engines/engine.py.230: loss: 0.17019 [I 2022-10-27 10:49:29] ...optimizers/trijoint.py.151: total_norm: 28.690397 [I 2022-10-27 10:49:35] ...trap/engines/engine.py.224: train: epoch 63 | batch 640/2304 [I 2022-10-27 10:49:35] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:40 [I 2022-10-27 10:49:35] ...trap/engines/engine.py.229: process: 0.56233 | load: 0.00266 [I 2022-10-27 10:49:35] ...trap/engines/engine.py.230: loss: 0.17301 [I 2022-10-27 10:49:35] ...optimizers/trijoint.py.151: total_norm: 18.376266 [I 2022-10-27 10:49:40] ...trap/engines/engine.py.224: train: epoch 63 | batch 650/2304 [I 2022-10-27 10:49:40] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:20 [I 2022-10-27 10:49:40] ...trap/engines/engine.py.229: process: 0.55088 | load: 0.00246 [I 2022-10-27 10:49:40] ...trap/engines/engine.py.230: loss: 0.12128 [I 2022-10-27 10:49:40] ...optimizers/trijoint.py.151: total_norm: 19.268089 [I 2022-10-27 10:49:46] ...trap/engines/engine.py.224: train: epoch 63 | batch 660/2304 [I 2022-10-27 10:49:46] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:13 [I 2022-10-27 10:49:46] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00252 [I 2022-10-27 10:49:46] ...trap/engines/engine.py.230: loss: 0.18751 [I 2022-10-27 10:49:46] ...optimizers/trijoint.py.151: total_norm: 23.314510 [I 2022-10-27 10:49:51] ...trap/engines/engine.py.224: train: epoch 63 | batch 670/2304 [I 2022-10-27 10:49:51] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:09 [I 2022-10-27 10:49:51] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00270 [I 2022-10-27 10:49:51] ...trap/engines/engine.py.230: loss: 0.22746 [I 2022-10-27 10:49:51] ...optimizers/trijoint.py.151: total_norm: 24.075005 [I 2022-10-27 10:49:57] ...trap/engines/engine.py.224: train: epoch 63 | batch 680/2304 [I 2022-10-27 10:49:57] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:03 [I 2022-10-27 10:49:57] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00245 [I 2022-10-27 10:49:57] ...trap/engines/engine.py.230: loss: 0.15558 [I 2022-10-27 10:49:57] ...optimizers/trijoint.py.151: total_norm: 19.083113 [I 2022-10-27 10:50:03] ...trap/engines/engine.py.224: train: epoch 63 | batch 690/2304 [I 2022-10-27 10:50:03] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:55 [I 2022-10-27 10:50:03] ...trap/engines/engine.py.229: process: 0.54805 | load: 0.00248 [I 2022-10-27 10:50:03] ...trap/engines/engine.py.230: loss: 0.19741 [I 2022-10-27 10:50:03] ...optimizers/trijoint.py.151: total_norm: 16.189175 [I 2022-10-27 10:50:08] ...trap/engines/engine.py.224: train: epoch 63 | batch 700/2304 [I 2022-10-27 10:50:08] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:51 [I 2022-10-27 10:50:08] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00249 [I 2022-10-27 10:50:08] ...trap/engines/engine.py.230: loss: 0.17928 [I 2022-10-27 10:50:08] ...optimizers/trijoint.py.151: total_norm: 19.770615 [I 2022-10-27 10:50:14] ...trap/engines/engine.py.224: train: epoch 63 | batch 710/2304 [I 2022-10-27 10:50:14] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:40 [I 2022-10-27 10:50:14] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00248 [I 2022-10-27 10:50:14] ...trap/engines/engine.py.230: loss: 0.12316 [I 2022-10-27 10:50:14] ...optimizers/trijoint.py.151: total_norm: 14.220982 [I 2022-10-27 10:50:19] ...trap/engines/engine.py.224: train: epoch 63 | batch 720/2304 [I 2022-10-27 10:50:19] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:35 [I 2022-10-27 10:50:19] ...trap/engines/engine.py.229: process: 0.54577 | load: 0.00244 [I 2022-10-27 10:50:19] ...trap/engines/engine.py.230: loss: 0.15444 [I 2022-10-27 10:50:19] ...optimizers/trijoint.py.151: total_norm: 11.372267 [I 2022-10-27 10:50:25] ...trap/engines/engine.py.224: train: epoch 63 | batch 730/2304 [I 2022-10-27 10:50:25] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:38 [I 2022-10-27 10:50:25] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00244 [I 2022-10-27 10:50:25] ...trap/engines/engine.py.230: loss: 0.12630 [I 2022-10-27 10:50:25] ...optimizers/trijoint.py.151: total_norm: 11.122075 [I 2022-10-27 10:50:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 740/2304 [I 2022-10-27 10:50:30] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:29 [I 2022-10-27 10:50:30] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00243 [I 2022-10-27 10:50:30] ...trap/engines/engine.py.230: loss: 0.12052 [I 2022-10-27 10:50:30] ...optimizers/trijoint.py.151: total_norm: 16.689770 [I 2022-10-27 10:50:36] ...trap/engines/engine.py.224: train: epoch 63 | batch 750/2304 [I 2022-10-27 10:50:36] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:21 [I 2022-10-27 10:50:36] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00256 [I 2022-10-27 10:50:36] ...trap/engines/engine.py.230: loss: 0.16028 [I 2022-10-27 10:50:36] ...optimizers/trijoint.py.151: total_norm: 16.345325 [I 2022-10-27 10:50:42] ...trap/engines/engine.py.224: train: epoch 63 | batch 760/2304 [I 2022-10-27 10:50:42] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:22 [I 2022-10-27 10:50:42] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00249 [I 2022-10-27 10:50:42] ...trap/engines/engine.py.230: loss: 0.18505 [I 2022-10-27 10:50:42] ...optimizers/trijoint.py.151: total_norm: 17.774429 [I 2022-10-27 10:50:47] ...trap/engines/engine.py.224: train: epoch 63 | batch 770/2304 [I 2022-10-27 10:50:47] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:14 [I 2022-10-27 10:50:47] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00250 [I 2022-10-27 10:50:47] ...trap/engines/engine.py.230: loss: 0.12741 [I 2022-10-27 10:50:47] ...optimizers/trijoint.py.151: total_norm: 17.777731 [I 2022-10-27 10:50:53] ...trap/engines/engine.py.224: train: epoch 63 | batch 780/2304 [I 2022-10-27 10:50:53] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:05 [I 2022-10-27 10:50:53] ...trap/engines/engine.py.229: process: 0.56610 | load: 0.00251 [I 2022-10-27 10:50:53] ...trap/engines/engine.py.230: loss: 0.16430 [I 2022-10-27 10:50:53] ...optimizers/trijoint.py.151: total_norm: 19.344887 [I 2022-10-27 10:50:58] ...trap/engines/engine.py.224: train: epoch 63 | batch 790/2304 [I 2022-10-27 10:50:58] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:06 [I 2022-10-27 10:50:58] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00259 [I 2022-10-27 10:50:58] ...trap/engines/engine.py.230: loss: 0.10264 [I 2022-10-27 10:50:58] ...optimizers/trijoint.py.151: total_norm: 9.738979 [I 2022-10-27 10:51:04] ...trap/engines/engine.py.224: train: epoch 63 | batch 800/2304 [I 2022-10-27 10:51:04] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:58 [I 2022-10-27 10:51:04] ...trap/engines/engine.py.229: process: 0.56491 | load: 0.00257 [I 2022-10-27 10:51:04] ...trap/engines/engine.py.230: loss: 0.18067 [I 2022-10-27 10:51:04] ...optimizers/trijoint.py.151: total_norm: 18.301136 [I 2022-10-27 10:51:10] ...trap/engines/engine.py.224: train: epoch 63 | batch 810/2304 [I 2022-10-27 10:51:10] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:54 [I 2022-10-27 10:51:10] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00259 [I 2022-10-27 10:51:10] ...trap/engines/engine.py.230: loss: 0.15533 [I 2022-10-27 10:51:10] ...optimizers/trijoint.py.151: total_norm: 16.122721 [I 2022-10-27 10:51:15] ...trap/engines/engine.py.224: train: epoch 63 | batch 820/2304 [I 2022-10-27 10:51:15] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:50 [I 2022-10-27 10:51:15] ...trap/engines/engine.py.229: process: 0.56939 | load: 0.00269 [I 2022-10-27 10:51:15] ...trap/engines/engine.py.230: loss: 0.11330 [I 2022-10-27 10:51:15] ...optimizers/trijoint.py.151: total_norm: 23.103893 [I 2022-10-27 10:51:21] ...trap/engines/engine.py.224: train: epoch 63 | batch 830/2304 [I 2022-10-27 10:51:21] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:42 [I 2022-10-27 10:51:21] ...trap/engines/engine.py.229: process: 0.55042 | load: 0.00254 [I 2022-10-27 10:51:21] ...trap/engines/engine.py.230: loss: 0.19426 [I 2022-10-27 10:51:21] ...optimizers/trijoint.py.151: total_norm: 21.710194 [I 2022-10-27 10:51:26] ...trap/engines/engine.py.224: train: epoch 63 | batch 840/2304 [I 2022-10-27 10:51:26] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:32 [I 2022-10-27 10:51:26] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00254 [I 2022-10-27 10:51:26] ...trap/engines/engine.py.230: loss: 0.12629 [I 2022-10-27 10:51:26] ...optimizers/trijoint.py.151: total_norm: 12.354550 [I 2022-10-27 10:51:32] ...trap/engines/engine.py.224: train: epoch 63 | batch 850/2304 [I 2022-10-27 10:51:32] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:27 [I 2022-10-27 10:51:32] ...trap/engines/engine.py.229: process: 0.54188 | load: 0.00252 [I 2022-10-27 10:51:32] ...trap/engines/engine.py.230: loss: 0.11984 [I 2022-10-27 10:51:32] ...optimizers/trijoint.py.151: total_norm: 13.950129 [I 2022-10-27 10:51:38] ...trap/engines/engine.py.224: train: epoch 63 | batch 860/2304 [I 2022-10-27 10:51:38] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:19 [I 2022-10-27 10:51:38] ...trap/engines/engine.py.229: process: 0.54615 | load: 0.00248 [I 2022-10-27 10:51:38] ...trap/engines/engine.py.230: loss: 0.11132 [I 2022-10-27 10:51:38] ...optimizers/trijoint.py.151: total_norm: 18.630287 [I 2022-10-27 10:51:43] ...trap/engines/engine.py.224: train: epoch 63 | batch 870/2304 [I 2022-10-27 10:51:43] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:14 [I 2022-10-27 10:51:43] ...trap/engines/engine.py.229: process: 0.55594 | load: 0.00253 [I 2022-10-27 10:51:43] ...trap/engines/engine.py.230: loss: 0.16648 [I 2022-10-27 10:51:43] ...optimizers/trijoint.py.151: total_norm: 17.569740 [I 2022-10-27 10:51:49] ...trap/engines/engine.py.224: train: epoch 63 | batch 880/2304 [I 2022-10-27 10:51:49] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:13 [I 2022-10-27 10:51:49] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00244 [I 2022-10-27 10:51:49] ...trap/engines/engine.py.230: loss: 0.11927 [I 2022-10-27 10:51:49] ...optimizers/trijoint.py.151: total_norm: 12.888596 [I 2022-10-27 10:51:54] ...trap/engines/engine.py.224: train: epoch 63 | batch 890/2304 [I 2022-10-27 10:51:54] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:05 [I 2022-10-27 10:51:54] ...trap/engines/engine.py.229: process: 0.55496 | load: 0.00244 [I 2022-10-27 10:51:54] ...trap/engines/engine.py.230: loss: 0.16701 [I 2022-10-27 10:51:54] ...optimizers/trijoint.py.151: total_norm: 16.439360 [I 2022-10-27 10:52:00] ...trap/engines/engine.py.224: train: epoch 63 | batch 900/2304 [I 2022-10-27 10:52:00] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:12:58 [I 2022-10-27 10:52:00] ...trap/engines/engine.py.229: process: 0.54980 | load: 0.00243 [I 2022-10-27 10:52:00] ...trap/engines/engine.py.230: loss: 0.18584 [I 2022-10-27 10:52:00] ...optimizers/trijoint.py.151: total_norm: 45.505287 [I 2022-10-27 10:52:05] ...trap/engines/engine.py.224: train: epoch 63 | batch 910/2304 [I 2022-10-27 10:52:05] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:51 [I 2022-10-27 10:52:05] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00242 [I 2022-10-27 10:52:05] ...trap/engines/engine.py.230: loss: 0.13638 [I 2022-10-27 10:52:05] ...optimizers/trijoint.py.151: total_norm: 17.192871 [I 2022-10-27 10:52:11] ...trap/engines/engine.py.224: train: epoch 63 | batch 920/2304 [I 2022-10-27 10:52:11] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:51 [I 2022-10-27 10:52:11] ...trap/engines/engine.py.229: process: 0.56055 | load: 0.00252 [I 2022-10-27 10:52:11] ...trap/engines/engine.py.230: loss: 0.14761 [I 2022-10-27 10:52:11] ...optimizers/trijoint.py.151: total_norm: 15.749863 [I 2022-10-27 10:52:16] ...trap/engines/engine.py.224: train: epoch 63 | batch 930/2304 [I 2022-10-27 10:52:16] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:47 [I 2022-10-27 10:52:16] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00254 [I 2022-10-27 10:52:16] ...trap/engines/engine.py.230: loss: 0.14966 [I 2022-10-27 10:52:16] ...optimizers/trijoint.py.151: total_norm: 11.820914 [I 2022-10-27 10:52:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 940/2304 [I 2022-10-27 10:52:22] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:34 [I 2022-10-27 10:52:22] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00247 [I 2022-10-27 10:52:22] ...trap/engines/engine.py.230: loss: 0.12147 [I 2022-10-27 10:52:22] ...optimizers/trijoint.py.151: total_norm: 16.354155 [I 2022-10-27 10:52:28] ...trap/engines/engine.py.224: train: epoch 63 | batch 950/2304 [I 2022-10-27 10:52:28] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:30 [I 2022-10-27 10:52:28] ...trap/engines/engine.py.229: process: 0.54173 | load: 0.00252 [I 2022-10-27 10:52:28] ...trap/engines/engine.py.230: loss: 0.14996 [I 2022-10-27 10:52:28] ...optimizers/trijoint.py.151: total_norm: 14.686828 [I 2022-10-27 10:52:33] ...trap/engines/engine.py.224: train: epoch 63 | batch 960/2304 [I 2022-10-27 10:52:33] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:29 [I 2022-10-27 10:52:33] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00251 [I 2022-10-27 10:52:33] ...trap/engines/engine.py.230: loss: 0.14559 [I 2022-10-27 10:52:33] ...optimizers/trijoint.py.151: total_norm: 11.148946 [I 2022-10-27 10:52:39] ...trap/engines/engine.py.224: train: epoch 63 | batch 970/2304 [I 2022-10-27 10:52:39] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:20 [I 2022-10-27 10:52:39] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00262 [I 2022-10-27 10:52:39] ...trap/engines/engine.py.230: loss: 0.12949 [I 2022-10-27 10:52:39] ...optimizers/trijoint.py.151: total_norm: 11.344470 [I 2022-10-27 10:52:44] ...trap/engines/engine.py.224: train: epoch 63 | batch 980/2304 [I 2022-10-27 10:52:44] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:11 [I 2022-10-27 10:52:44] ...trap/engines/engine.py.229: process: 0.55507 | load: 0.00253 [I 2022-10-27 10:52:44] ...trap/engines/engine.py.230: loss: 0.12328 [I 2022-10-27 10:52:44] ...optimizers/trijoint.py.151: total_norm: 32.439919 [I 2022-10-27 10:52:50] ...trap/engines/engine.py.224: train: epoch 63 | batch 990/2304 [I 2022-10-27 10:52:50] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:10 [I 2022-10-27 10:52:50] ...trap/engines/engine.py.229: process: 0.55525 | load: 0.00261 [I 2022-10-27 10:52:50] ...trap/engines/engine.py.230: loss: 0.09393 [I 2022-10-27 10:52:50] ...optimizers/trijoint.py.151: total_norm: 12.713765 [I 2022-10-27 10:52:55] ...trap/engines/engine.py.224: train: epoch 63 | batch 1000/2304 [I 2022-10-27 10:52:55] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:02 [I 2022-10-27 10:52:55] ...trap/engines/engine.py.229: process: 0.55079 | load: 0.00258 [I 2022-10-27 10:52:55] ...trap/engines/engine.py.230: loss: 0.18109 [I 2022-10-27 10:52:55] ...optimizers/trijoint.py.151: total_norm: 20.597094 [I 2022-10-27 10:53:01] ...trap/engines/engine.py.224: train: epoch 63 | batch 1010/2304 [I 2022-10-27 10:53:01] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:11:54 [I 2022-10-27 10:53:01] ...trap/engines/engine.py.229: process: 0.54071 | load: 0.00255 [I 2022-10-27 10:53:01] ...trap/engines/engine.py.230: loss: 0.15010 [I 2022-10-27 10:53:01] ...optimizers/trijoint.py.151: total_norm: 20.752903 [I 2022-10-27 10:53:06] ...trap/engines/engine.py.224: train: epoch 63 | batch 1020/2304 [I 2022-10-27 10:53:06] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:51 [I 2022-10-27 10:53:06] ...trap/engines/engine.py.229: process: 0.54629 | load: 0.00250 [I 2022-10-27 10:53:06] ...trap/engines/engine.py.230: loss: 0.16051 [I 2022-10-27 10:53:06] ...optimizers/trijoint.py.151: total_norm: 14.098874 [I 2022-10-27 10:53:12] ...trap/engines/engine.py.224: train: epoch 63 | batch 1030/2304 [I 2022-10-27 10:53:12] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:51 [I 2022-10-27 10:53:12] ...trap/engines/engine.py.229: process: 0.57187 | load: 0.00247 [I 2022-10-27 10:53:12] ...trap/engines/engine.py.230: loss: 0.09717 [I 2022-10-27 10:53:12] ...optimizers/trijoint.py.151: total_norm: 16.831411 [I 2022-10-27 10:53:18] ...trap/engines/engine.py.224: train: epoch 63 | batch 1040/2304 [I 2022-10-27 10:53:18] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:41 [I 2022-10-27 10:53:18] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00249 [I 2022-10-27 10:53:18] ...trap/engines/engine.py.230: loss: 0.19270 [I 2022-10-27 10:53:18] ...optimizers/trijoint.py.151: total_norm: 9.535398 [I 2022-10-27 10:53:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 1050/2304 [I 2022-10-27 10:53:23] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:38 [I 2022-10-27 10:53:23] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00245 [I 2022-10-27 10:53:23] ...trap/engines/engine.py.230: loss: 0.20828 [I 2022-10-27 10:53:23] ...optimizers/trijoint.py.151: total_norm: 22.701803 [I 2022-10-27 10:53:29] ...trap/engines/engine.py.224: train: epoch 63 | batch 1060/2304 [I 2022-10-27 10:53:29] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:30 [I 2022-10-27 10:53:29] ...trap/engines/engine.py.229: process: 0.55057 | load: 0.00245 [I 2022-10-27 10:53:29] ...trap/engines/engine.py.230: loss: 0.09294 [I 2022-10-27 10:53:29] ...optimizers/trijoint.py.151: total_norm: 13.885484 [I 2022-10-27 10:53:34] ...trap/engines/engine.py.224: train: epoch 63 | batch 1070/2304 [I 2022-10-27 10:53:34] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:25 [I 2022-10-27 10:53:34] ...trap/engines/engine.py.229: process: 0.54151 | load: 0.00242 [I 2022-10-27 10:53:34] ...trap/engines/engine.py.230: loss: 0.11529 [I 2022-10-27 10:53:34] ...optimizers/trijoint.py.151: total_norm: 7.580257 [I 2022-10-27 10:53:40] ...trap/engines/engine.py.224: train: epoch 63 | batch 1080/2304 [I 2022-10-27 10:53:40] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:16 [I 2022-10-27 10:53:40] ...trap/engines/engine.py.229: process: 0.54069 | load: 0.00258 [I 2022-10-27 10:53:40] ...trap/engines/engine.py.230: loss: 0.17748 [I 2022-10-27 10:53:40] ...optimizers/trijoint.py.151: total_norm: 30.828415 [I 2022-10-27 10:53:45] ...trap/engines/engine.py.224: train: epoch 63 | batch 1090/2304 [I 2022-10-27 10:53:45] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:08 [I 2022-10-27 10:53:45] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00249 [I 2022-10-27 10:53:45] ...trap/engines/engine.py.230: loss: 0.14133 [I 2022-10-27 10:53:45] ...optimizers/trijoint.py.151: total_norm: 20.404533 [I 2022-10-27 10:53:51] ...trap/engines/engine.py.224: train: epoch 63 | batch 1100/2304 [I 2022-10-27 10:53:51] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:10 [I 2022-10-27 10:53:51] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00248 [I 2022-10-27 10:53:51] ...trap/engines/engine.py.230: loss: 0.13785 [I 2022-10-27 10:53:51] ...optimizers/trijoint.py.151: total_norm: 17.666691 [I 2022-10-27 10:53:57] ...trap/engines/engine.py.224: train: epoch 63 | batch 1110/2304 [I 2022-10-27 10:53:57] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:10:59 [I 2022-10-27 10:53:57] ...trap/engines/engine.py.229: process: 0.54065 | load: 0.00243 [I 2022-10-27 10:53:57] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-27 10:53:57] ...optimizers/trijoint.py.151: total_norm: 36.570572 [I 2022-10-27 10:54:02] ...trap/engines/engine.py.224: train: epoch 63 | batch 1120/2304 [I 2022-10-27 10:54:02] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:10:56 [I 2022-10-27 10:54:02] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00255 [I 2022-10-27 10:54:02] ...trap/engines/engine.py.230: loss: 0.24372 [I 2022-10-27 10:54:02] ...optimizers/trijoint.py.151: total_norm: 28.230419 [I 2022-10-27 10:54:08] ...trap/engines/engine.py.224: train: epoch 63 | batch 1130/2304 [I 2022-10-27 10:54:08] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:51 [I 2022-10-27 10:54:08] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00254 [I 2022-10-27 10:54:08] ...trap/engines/engine.py.230: loss: 0.18033 [I 2022-10-27 10:54:08] ...optimizers/trijoint.py.151: total_norm: 46.379425 [I 2022-10-27 10:54:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 1140/2304 [I 2022-10-27 10:54:13] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:10:45 [I 2022-10-27 10:54:13] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00250 [I 2022-10-27 10:54:13] ...trap/engines/engine.py.230: loss: 0.22489 [I 2022-10-27 10:54:13] ...optimizers/trijoint.py.151: total_norm: 19.542068 [I 2022-10-27 10:54:19] ...trap/engines/engine.py.224: train: epoch 63 | batch 1150/2304 [I 2022-10-27 10:54:19] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:10:41 [I 2022-10-27 10:54:19] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00250 [I 2022-10-27 10:54:19] ...trap/engines/engine.py.230: loss: 0.19094 [I 2022-10-27 10:54:19] ...optimizers/trijoint.py.151: total_norm: 53.128586 [I 2022-10-27 10:54:24] ...trap/engines/engine.py.224: train: epoch 63 | batch 1160/2304 [I 2022-10-27 10:54:24] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:38 [I 2022-10-27 10:54:24] ...trap/engines/engine.py.229: process: 0.55364 | load: 0.00248 [I 2022-10-27 10:54:24] ...trap/engines/engine.py.230: loss: 0.12454 [I 2022-10-27 10:54:24] ...optimizers/trijoint.py.151: total_norm: 16.493704 [I 2022-10-27 10:54:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 1170/2304 [I 2022-10-27 10:54:30] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:32 [I 2022-10-27 10:54:30] ...trap/engines/engine.py.229: process: 0.55143 | load: 0.00260 [I 2022-10-27 10:54:30] ...trap/engines/engine.py.230: loss: 0.18699 [I 2022-10-27 10:54:30] ...optimizers/trijoint.py.151: total_norm: 16.856598 [I 2022-10-27 10:54:36] ...trap/engines/engine.py.224: train: epoch 63 | batch 1180/2304 [I 2022-10-27 10:54:36] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:25 [I 2022-10-27 10:54:36] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00254 [I 2022-10-27 10:54:36] ...trap/engines/engine.py.230: loss: 0.11813 [I 2022-10-27 10:54:36] ...optimizers/trijoint.py.151: total_norm: 8.449352 [I 2022-10-27 10:54:41] ...trap/engines/engine.py.224: train: epoch 63 | batch 1190/2304 [I 2022-10-27 10:54:41] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:14 [I 2022-10-27 10:54:41] ...trap/engines/engine.py.229: process: 0.54344 | load: 0.00246 [I 2022-10-27 10:54:41] ...trap/engines/engine.py.230: loss: 0.19711 [I 2022-10-27 10:54:41] ...optimizers/trijoint.py.151: total_norm: 16.934959 [I 2022-10-27 10:54:47] ...trap/engines/engine.py.224: train: epoch 63 | batch 1200/2304 [I 2022-10-27 10:54:47] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:14 [I 2022-10-27 10:54:47] ...trap/engines/engine.py.229: process: 0.56500 | load: 0.00246 [I 2022-10-27 10:54:47] ...trap/engines/engine.py.230: loss: 0.11944 [I 2022-10-27 10:54:47] ...optimizers/trijoint.py.151: total_norm: 22.576351 [I 2022-10-27 10:55:10] ...trap/engines/engine.py.224: train: epoch 63 | batch 1210/2304 [I 2022-10-27 10:55:10] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:24:00 [I 2022-10-27 10:55:10] ...trap/engines/engine.py.229: process: 0.54608 | load: 0.00238 [I 2022-10-27 10:55:10] ...trap/engines/engine.py.230: loss: 0.12487 [I 2022-10-27 10:55:10] ...optimizers/trijoint.py.151: total_norm: 18.855396 [I 2022-10-27 10:55:16] ...trap/engines/engine.py.224: train: epoch 63 | batch 1220/2304 [I 2022-10-27 10:55:16] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:11:30 [I 2022-10-27 10:55:16] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00239 [I 2022-10-27 10:55:16] ...trap/engines/engine.py.230: loss: 0.17587 [I 2022-10-27 10:55:16] ...optimizers/trijoint.py.151: total_norm: 24.041372 [I 2022-10-27 10:55:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 1230/2304 [I 2022-10-27 10:55:22] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:10:00 [I 2022-10-27 10:55:22] ...trap/engines/engine.py.229: process: 0.55157 | load: 0.00238 [I 2022-10-27 10:55:22] ...trap/engines/engine.py.230: loss: 0.11980 [I 2022-10-27 10:55:22] ...optimizers/trijoint.py.151: total_norm: 26.159025 [I 2022-10-27 10:55:27] ...trap/engines/engine.py.224: train: epoch 63 | batch 1240/2304 [I 2022-10-27 10:55:27] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:53 [I 2022-10-27 10:55:27] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00238 [I 2022-10-27 10:55:27] ...trap/engines/engine.py.230: loss: 0.18589 [I 2022-10-27 10:55:27] ...optimizers/trijoint.py.151: total_norm: 20.204762 [I 2022-10-27 10:55:33] ...trap/engines/engine.py.224: train: epoch 63 | batch 1250/2304 [I 2022-10-27 10:55:33] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:48 [I 2022-10-27 10:55:33] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00238 [I 2022-10-27 10:55:33] ...trap/engines/engine.py.230: loss: 0.14483 [I 2022-10-27 10:55:33] ...optimizers/trijoint.py.151: total_norm: 20.743343 [I 2022-10-27 10:55:38] ...trap/engines/engine.py.224: train: epoch 63 | batch 1260/2304 [I 2022-10-27 10:55:38] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:40 [I 2022-10-27 10:55:38] ...trap/engines/engine.py.229: process: 0.56312 | load: 0.00242 [I 2022-10-27 10:55:38] ...trap/engines/engine.py.230: loss: 0.16254 [I 2022-10-27 10:55:38] ...optimizers/trijoint.py.151: total_norm: 12.525516 [I 2022-10-27 10:55:44] ...trap/engines/engine.py.224: train: epoch 63 | batch 1270/2304 [I 2022-10-27 10:55:44] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:36 [I 2022-10-27 10:55:44] ...trap/engines/engine.py.229: process: 0.55154 | load: 0.00240 [I 2022-10-27 10:55:44] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-27 10:55:44] ...optimizers/trijoint.py.151: total_norm: 11.685203 [I 2022-10-27 10:55:49] ...trap/engines/engine.py.224: train: epoch 63 | batch 1280/2304 [I 2022-10-27 10:55:49] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:28 [I 2022-10-27 10:55:49] ...trap/engines/engine.py.229: process: 0.56314 | load: 0.00241 [I 2022-10-27 10:55:49] ...trap/engines/engine.py.230: loss: 0.12105 [I 2022-10-27 10:55:49] ...optimizers/trijoint.py.151: total_norm: 13.506669 [I 2022-10-27 10:55:55] ...trap/engines/engine.py.224: train: epoch 63 | batch 1290/2304 [I 2022-10-27 10:55:55] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:22 [I 2022-10-27 10:55:55] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00237 [I 2022-10-27 10:55:55] ...trap/engines/engine.py.230: loss: 0.20419 [I 2022-10-27 10:55:55] ...optimizers/trijoint.py.151: total_norm: 70.724617 [I 2022-10-27 10:56:01] ...trap/engines/engine.py.224: train: epoch 63 | batch 1300/2304 [I 2022-10-27 10:56:01] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:29 [I 2022-10-27 10:56:01] ...trap/engines/engine.py.229: process: 0.58962 | load: 0.00246 [I 2022-10-27 10:56:01] ...trap/engines/engine.py.230: loss: 0.16531 [I 2022-10-27 10:56:01] ...optimizers/trijoint.py.151: total_norm: 11.999692 [I 2022-10-27 10:56:06] ...trap/engines/engine.py.224: train: epoch 63 | batch 1310/2304 [I 2022-10-27 10:56:06] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:09:14 [I 2022-10-27 10:56:06] ...trap/engines/engine.py.229: process: 0.54925 | load: 0.00247 [I 2022-10-27 10:56:06] ...trap/engines/engine.py.230: loss: 0.19392 [I 2022-10-27 10:56:06] ...optimizers/trijoint.py.151: total_norm: 24.843454 [I 2022-10-27 10:56:12] ...trap/engines/engine.py.224: train: epoch 63 | batch 1320/2304 [I 2022-10-27 10:56:12] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:09:19 [I 2022-10-27 10:56:12] ...trap/engines/engine.py.229: process: 0.56342 | load: 0.00241 [I 2022-10-27 10:56:12] ...trap/engines/engine.py.230: loss: 0.10616 [I 2022-10-27 10:56:12] ...optimizers/trijoint.py.151: total_norm: 10.317677 [I 2022-10-27 10:56:18] ...trap/engines/engine.py.224: train: epoch 63 | batch 1330/2304 [I 2022-10-27 10:56:18] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:09:03 [I 2022-10-27 10:56:18] ...trap/engines/engine.py.229: process: 0.54925 | load: 0.00242 [I 2022-10-27 10:56:18] ...trap/engines/engine.py.230: loss: 0.10692 [I 2022-10-27 10:56:18] ...optimizers/trijoint.py.151: total_norm: 7.786250 [I 2022-10-27 10:56:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 1340/2304 [I 2022-10-27 10:56:23] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:56 [I 2022-10-27 10:56:23] ...trap/engines/engine.py.229: process: 0.56559 | load: 0.00260 [I 2022-10-27 10:56:23] ...trap/engines/engine.py.230: loss: 0.12424 [I 2022-10-27 10:56:23] ...optimizers/trijoint.py.151: total_norm: 32.009960 [I 2022-10-27 10:56:29] ...trap/engines/engine.py.224: train: epoch 63 | batch 1350/2304 [I 2022-10-27 10:56:29] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:48 [I 2022-10-27 10:56:29] ...trap/engines/engine.py.229: process: 0.54839 | load: 0.00257 [I 2022-10-27 10:56:29] ...trap/engines/engine.py.230: loss: 0.14609 [I 2022-10-27 10:56:29] ...optimizers/trijoint.py.151: total_norm: 23.111977 [I 2022-10-27 10:56:34] ...trap/engines/engine.py.224: train: epoch 63 | batch 1360/2304 [I 2022-10-27 10:56:34] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:43 [I 2022-10-27 10:56:34] ...trap/engines/engine.py.229: process: 0.56659 | load: 0.00258 [I 2022-10-27 10:56:34] ...trap/engines/engine.py.230: loss: 0.18659 [I 2022-10-27 10:56:34] ...optimizers/trijoint.py.151: total_norm: 23.181644 [I 2022-10-27 10:56:40] ...trap/engines/engine.py.224: train: epoch 63 | batch 1370/2304 [I 2022-10-27 10:56:40] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:42 [I 2022-10-27 10:56:40] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00255 [I 2022-10-27 10:56:40] ...trap/engines/engine.py.230: loss: 0.17703 [I 2022-10-27 10:56:40] ...optimizers/trijoint.py.151: total_norm: 13.950815 [I 2022-10-27 10:56:45] ...trap/engines/engine.py.224: train: epoch 63 | batch 1380/2304 [I 2022-10-27 10:56:45] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:33 [I 2022-10-27 10:56:45] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00270 [I 2022-10-27 10:56:45] ...trap/engines/engine.py.230: loss: 0.17871 [I 2022-10-27 10:56:45] ...optimizers/trijoint.py.151: total_norm: 11.627707 [I 2022-10-27 10:56:51] ...trap/engines/engine.py.224: train: epoch 63 | batch 1390/2304 [I 2022-10-27 10:56:51] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:30 [I 2022-10-27 10:56:51] ...trap/engines/engine.py.229: process: 0.56382 | load: 0.00262 [I 2022-10-27 10:56:51] ...trap/engines/engine.py.230: loss: 0.25710 [I 2022-10-27 10:56:51] ...optimizers/trijoint.py.151: total_norm: 15.036395 [I 2022-10-27 10:56:57] ...trap/engines/engine.py.224: train: epoch 63 | batch 1400/2304 [I 2022-10-27 10:56:57] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:29 [I 2022-10-27 10:56:57] ...trap/engines/engine.py.229: process: 0.55821 | load: 0.00255 [I 2022-10-27 10:56:57] ...trap/engines/engine.py.230: loss: 0.17563 [I 2022-10-27 10:56:57] ...optimizers/trijoint.py.151: total_norm: 31.974863 [I 2022-10-27 10:57:02] ...trap/engines/engine.py.224: train: epoch 63 | batch 1410/2304 [I 2022-10-27 10:57:02] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:16 [I 2022-10-27 10:57:02] ...trap/engines/engine.py.229: process: 0.54793 | load: 0.00252 [I 2022-10-27 10:57:02] ...trap/engines/engine.py.230: loss: 0.10594 [I 2022-10-27 10:57:02] ...optimizers/trijoint.py.151: total_norm: 8.926970 [I 2022-10-27 10:57:08] ...trap/engines/engine.py.224: train: epoch 63 | batch 1420/2304 [I 2022-10-27 10:57:08] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:08:12 [I 2022-10-27 10:57:08] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00246 [I 2022-10-27 10:57:08] ...trap/engines/engine.py.230: loss: 0.17080 [I 2022-10-27 10:57:08] ...optimizers/trijoint.py.151: total_norm: 16.418077 [I 2022-10-27 10:57:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 1430/2304 [I 2022-10-27 10:57:13] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:08:06 [I 2022-10-27 10:57:13] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00260 [I 2022-10-27 10:57:13] ...trap/engines/engine.py.230: loss: 0.11536 [I 2022-10-27 10:57:13] ...optimizers/trijoint.py.151: total_norm: 7.145914 [I 2022-10-27 10:57:19] ...trap/engines/engine.py.224: train: epoch 63 | batch 1440/2304 [I 2022-10-27 10:57:19] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:08:01 [I 2022-10-27 10:57:19] ...trap/engines/engine.py.229: process: 0.56201 | load: 0.00260 [I 2022-10-27 10:57:19] ...trap/engines/engine.py.230: loss: 0.09958 [I 2022-10-27 10:57:19] ...optimizers/trijoint.py.151: total_norm: 22.442305 [I 2022-10-27 10:57:25] ...trap/engines/engine.py.224: train: epoch 63 | batch 1450/2304 [I 2022-10-27 10:57:25] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:57 [I 2022-10-27 10:57:25] ...trap/engines/engine.py.229: process: 0.56931 | load: 0.00258 [I 2022-10-27 10:57:25] ...trap/engines/engine.py.230: loss: 0.16181 [I 2022-10-27 10:57:25] ...optimizers/trijoint.py.151: total_norm: 28.183092 [I 2022-10-27 10:57:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 1460/2304 [I 2022-10-27 10:57:30] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:08:10 [I 2022-10-27 10:57:30] ...trap/engines/engine.py.229: process: 0.57430 | load: 0.00258 [I 2022-10-27 10:57:30] ...trap/engines/engine.py.230: loss: 0.26246 [I 2022-10-27 10:57:30] ...optimizers/trijoint.py.151: total_norm: 11.124144 [I 2022-10-27 10:57:36] ...trap/engines/engine.py.224: train: epoch 63 | batch 1470/2304 [I 2022-10-27 10:57:36] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:45 [I 2022-10-27 10:57:36] ...trap/engines/engine.py.229: process: 0.55657 | load: 0.00258 [I 2022-10-27 10:57:36] ...trap/engines/engine.py.230: loss: 0.21740 [I 2022-10-27 10:57:36] ...optimizers/trijoint.py.151: total_norm: 29.689587 [I 2022-10-27 10:57:42] ...trap/engines/engine.py.224: train: epoch 63 | batch 1480/2304 [I 2022-10-27 10:57:42] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:42 [I 2022-10-27 10:57:42] ...trap/engines/engine.py.229: process: 0.57062 | load: 0.00252 [I 2022-10-27 10:57:42] ...trap/engines/engine.py.230: loss: 0.20447 [I 2022-10-27 10:57:42] ...optimizers/trijoint.py.151: total_norm: 16.640659 [I 2022-10-27 10:57:47] ...trap/engines/engine.py.224: train: epoch 63 | batch 1490/2304 [I 2022-10-27 10:57:47] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:34 [I 2022-10-27 10:57:47] ...trap/engines/engine.py.229: process: 0.55156 | load: 0.00260 [I 2022-10-27 10:57:47] ...trap/engines/engine.py.230: loss: 0.21269 [I 2022-10-27 10:57:47] ...optimizers/trijoint.py.151: total_norm: 21.143454 [I 2022-10-27 10:57:53] ...trap/engines/engine.py.224: train: epoch 63 | batch 1500/2304 [I 2022-10-27 10:57:53] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:27 [I 2022-10-27 10:57:53] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00258 [I 2022-10-27 10:57:53] ...trap/engines/engine.py.230: loss: 0.16834 [I 2022-10-27 10:57:53] ...optimizers/trijoint.py.151: total_norm: 18.239090 [I 2022-10-27 10:57:58] ...trap/engines/engine.py.224: train: epoch 63 | batch 1510/2304 [I 2022-10-27 10:57:58] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:20 [I 2022-10-27 10:57:58] ...trap/engines/engine.py.229: process: 0.55167 | load: 0.00254 [I 2022-10-27 10:57:58] ...trap/engines/engine.py.230: loss: 0.14126 [I 2022-10-27 10:57:58] ...optimizers/trijoint.py.151: total_norm: 8.154018 [I 2022-10-27 10:58:04] ...trap/engines/engine.py.224: train: epoch 63 | batch 1520/2304 [I 2022-10-27 10:58:04] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:15 [I 2022-10-27 10:58:04] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00251 [I 2022-10-27 10:58:04] ...trap/engines/engine.py.230: loss: 0.17793 [I 2022-10-27 10:58:04] ...optimizers/trijoint.py.151: total_norm: 12.559915 [I 2022-10-27 10:58:09] ...trap/engines/engine.py.224: train: epoch 63 | batch 1530/2304 [I 2022-10-27 10:58:09] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:07:08 [I 2022-10-27 10:58:09] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00259 [I 2022-10-27 10:58:09] ...trap/engines/engine.py.230: loss: 0.13989 [I 2022-10-27 10:58:09] ...optimizers/trijoint.py.151: total_norm: 21.407841 [I 2022-10-27 10:58:15] ...trap/engines/engine.py.224: train: epoch 63 | batch 1540/2304 [I 2022-10-27 10:58:15] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:07:05 [I 2022-10-27 10:58:15] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00253 [I 2022-10-27 10:58:15] ...trap/engines/engine.py.230: loss: 0.19050 [I 2022-10-27 10:58:15] ...optimizers/trijoint.py.151: total_norm: 19.886539 [I 2022-10-27 10:58:20] ...trap/engines/engine.py.224: train: epoch 63 | batch 1550/2304 [I 2022-10-27 10:58:20] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:57 [I 2022-10-27 10:58:20] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00255 [I 2022-10-27 10:58:20] ...trap/engines/engine.py.230: loss: 0.11692 [I 2022-10-27 10:58:20] ...optimizers/trijoint.py.151: total_norm: 11.605825 [I 2022-10-27 10:58:26] ...trap/engines/engine.py.224: train: epoch 63 | batch 1560/2304 [I 2022-10-27 10:58:26] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:55 [I 2022-10-27 10:58:26] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00258 [I 2022-10-27 10:58:26] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-27 10:58:26] ...optimizers/trijoint.py.151: total_norm: 43.845921 [I 2022-10-27 10:58:32] ...trap/engines/engine.py.224: train: epoch 63 | batch 1570/2304 [I 2022-10-27 10:58:32] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:46 [I 2022-10-27 10:58:32] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00254 [I 2022-10-27 10:58:32] ...trap/engines/engine.py.230: loss: 0.19206 [I 2022-10-27 10:58:32] ...optimizers/trijoint.py.151: total_norm: 17.166574 [I 2022-10-27 10:58:37] ...trap/engines/engine.py.224: train: epoch 63 | batch 1580/2304 [I 2022-10-27 10:58:37] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:47 [I 2022-10-27 10:58:37] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00253 [I 2022-10-27 10:58:37] ...trap/engines/engine.py.230: loss: 0.16041 [I 2022-10-27 10:58:37] ...optimizers/trijoint.py.151: total_norm: 22.246265 [I 2022-10-27 10:58:43] ...trap/engines/engine.py.224: train: epoch 63 | batch 1590/2304 [I 2022-10-27 10:58:43] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:38 [I 2022-10-27 10:58:43] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00255 [I 2022-10-27 10:58:43] ...trap/engines/engine.py.230: loss: 0.16474 [I 2022-10-27 10:58:43] ...optimizers/trijoint.py.151: total_norm: 33.526012 [I 2022-10-27 10:58:48] ...trap/engines/engine.py.224: train: epoch 63 | batch 1600/2304 [I 2022-10-27 10:58:48] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:33 [I 2022-10-27 10:58:48] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00256 [I 2022-10-27 10:58:48] ...trap/engines/engine.py.230: loss: 0.19560 [I 2022-10-27 10:58:48] ...optimizers/trijoint.py.151: total_norm: 24.883249 [I 2022-10-27 10:58:54] ...trap/engines/engine.py.224: train: epoch 63 | batch 1610/2304 [I 2022-10-27 10:58:54] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:26 [I 2022-10-27 10:58:54] ...trap/engines/engine.py.229: process: 0.54550 | load: 0.00254 [I 2022-10-27 10:58:54] ...trap/engines/engine.py.230: loss: 0.15229 [I 2022-10-27 10:58:54] ...optimizers/trijoint.py.151: total_norm: 19.224022 [I 2022-10-27 10:59:00] ...trap/engines/engine.py.224: train: epoch 63 | batch 1620/2304 [I 2022-10-27 10:59:00] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:18 [I 2022-10-27 10:59:00] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00266 [I 2022-10-27 10:59:00] ...trap/engines/engine.py.230: loss: 0.12100 [I 2022-10-27 10:59:00] ...optimizers/trijoint.py.151: total_norm: 23.107334 [I 2022-10-27 10:59:05] ...trap/engines/engine.py.224: train: epoch 63 | batch 1630/2304 [I 2022-10-27 10:59:05] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:17 [I 2022-10-27 10:59:05] ...trap/engines/engine.py.229: process: 0.56518 | load: 0.00533 [I 2022-10-27 10:59:05] ...trap/engines/engine.py.230: loss: 0.12208 [I 2022-10-27 10:59:05] ...optimizers/trijoint.py.151: total_norm: 12.844160 [I 2022-10-27 10:59:11] ...trap/engines/engine.py.224: train: epoch 63 | batch 1640/2304 [I 2022-10-27 10:59:11] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:09 [I 2022-10-27 10:59:11] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00249 [I 2022-10-27 10:59:11] ...trap/engines/engine.py.230: loss: 0.14692 [I 2022-10-27 10:59:11] ...optimizers/trijoint.py.151: total_norm: 29.656092 [I 2022-10-27 10:59:16] ...trap/engines/engine.py.224: train: epoch 63 | batch 1650/2304 [I 2022-10-27 10:59:16] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:06:05 [I 2022-10-27 10:59:16] ...trap/engines/engine.py.229: process: 0.56591 | load: 0.00256 [I 2022-10-27 10:59:16] ...trap/engines/engine.py.230: loss: 0.21186 [I 2022-10-27 10:59:16] ...optimizers/trijoint.py.151: total_norm: 23.980305 [I 2022-10-27 10:59:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 1660/2304 [I 2022-10-27 10:59:22] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:06:02 [I 2022-10-27 10:59:22] ...trap/engines/engine.py.229: process: 0.56039 | load: 0.00254 [I 2022-10-27 10:59:22] ...trap/engines/engine.py.230: loss: 0.16462 [I 2022-10-27 10:59:22] ...optimizers/trijoint.py.151: total_norm: 29.694311 [I 2022-10-27 10:59:28] ...trap/engines/engine.py.224: train: epoch 63 | batch 1670/2304 [I 2022-10-27 10:59:28] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:52 [I 2022-10-27 10:59:28] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00260 [I 2022-10-27 10:59:28] ...trap/engines/engine.py.230: loss: 0.19264 [I 2022-10-27 10:59:28] ...optimizers/trijoint.py.151: total_norm: 28.291092 [I 2022-10-27 10:59:33] ...trap/engines/engine.py.224: train: epoch 63 | batch 1680/2304 [I 2022-10-27 10:59:33] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:48 [I 2022-10-27 10:59:33] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00256 [I 2022-10-27 10:59:33] ...trap/engines/engine.py.230: loss: 0.15226 [I 2022-10-27 10:59:33] ...optimizers/trijoint.py.151: total_norm: 33.830837 [I 2022-10-27 10:59:39] ...trap/engines/engine.py.224: train: epoch 63 | batch 1690/2304 [I 2022-10-27 10:59:39] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:46 [I 2022-10-27 10:59:39] ...trap/engines/engine.py.229: process: 0.55108 | load: 0.00265 [I 2022-10-27 10:59:39] ...trap/engines/engine.py.230: loss: 0.15264 [I 2022-10-27 10:59:39] ...optimizers/trijoint.py.151: total_norm: 25.863173 [I 2022-10-27 10:59:44] ...trap/engines/engine.py.224: train: epoch 63 | batch 1700/2304 [I 2022-10-27 10:59:44] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:38 [I 2022-10-27 10:59:44] ...trap/engines/engine.py.229: process: 0.57071 | load: 0.00264 [I 2022-10-27 10:59:44] ...trap/engines/engine.py.230: loss: 0.13230 [I 2022-10-27 10:59:44] ...optimizers/trijoint.py.151: total_norm: 11.401165 [I 2022-10-27 10:59:50] ...trap/engines/engine.py.224: train: epoch 63 | batch 1710/2304 [I 2022-10-27 10:59:50] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:34 [I 2022-10-27 10:59:50] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00264 [I 2022-10-27 10:59:50] ...trap/engines/engine.py.230: loss: 0.15194 [I 2022-10-27 10:59:50] ...optimizers/trijoint.py.151: total_norm: 12.747015 [I 2022-10-27 10:59:56] ...trap/engines/engine.py.224: train: epoch 63 | batch 1720/2304 [I 2022-10-27 10:59:56] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:25 [I 2022-10-27 10:59:56] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00261 [I 2022-10-27 10:59:56] ...trap/engines/engine.py.230: loss: 0.14034 [I 2022-10-27 10:59:56] ...optimizers/trijoint.py.151: total_norm: 45.786091 [I 2022-10-27 11:00:01] ...trap/engines/engine.py.224: train: epoch 63 | batch 1730/2304 [I 2022-10-27 11:00:01] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:20 [I 2022-10-27 11:00:01] ...trap/engines/engine.py.229: process: 0.54441 | load: 0.00262 [I 2022-10-27 11:00:01] ...trap/engines/engine.py.230: loss: 0.16744 [I 2022-10-27 11:00:01] ...optimizers/trijoint.py.151: total_norm: 15.415593 [I 2022-10-27 11:00:07] ...trap/engines/engine.py.224: train: epoch 63 | batch 1740/2304 [I 2022-10-27 11:00:07] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:16 [I 2022-10-27 11:00:07] ...trap/engines/engine.py.229: process: 0.57161 | load: 0.00254 [I 2022-10-27 11:00:07] ...trap/engines/engine.py.230: loss: 0.10938 [I 2022-10-27 11:00:07] ...optimizers/trijoint.py.151: total_norm: 7.145494 [I 2022-10-27 11:00:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 1750/2304 [I 2022-10-27 11:00:13] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:05:12 [I 2022-10-27 11:00:13] ...trap/engines/engine.py.229: process: 0.56922 | load: 0.00254 [I 2022-10-27 11:00:13] ...trap/engines/engine.py.230: loss: 0.11106 [I 2022-10-27 11:00:13] ...optimizers/trijoint.py.151: total_norm: 12.948792 [I 2022-10-27 11:00:18] ...trap/engines/engine.py.224: train: epoch 63 | batch 1760/2304 [I 2022-10-27 11:00:18] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:06 [I 2022-10-27 11:00:18] ...trap/engines/engine.py.229: process: 0.57412 | load: 0.00258 [I 2022-10-27 11:00:18] ...trap/engines/engine.py.230: loss: 0.22691 [I 2022-10-27 11:00:18] ...optimizers/trijoint.py.151: total_norm: 32.402092 [I 2022-10-27 11:00:24] ...trap/engines/engine.py.224: train: epoch 63 | batch 1770/2304 [I 2022-10-27 11:00:24] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:59 [I 2022-10-27 11:00:24] ...trap/engines/engine.py.229: process: 0.56567 | load: 0.00255 [I 2022-10-27 11:00:24] ...trap/engines/engine.py.230: loss: 0.14714 [I 2022-10-27 11:00:24] ...optimizers/trijoint.py.151: total_norm: 23.352179 [I 2022-10-27 11:00:29] ...trap/engines/engine.py.224: train: epoch 63 | batch 1780/2304 [I 2022-10-27 11:00:29] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:55 [I 2022-10-27 11:00:29] ...trap/engines/engine.py.229: process: 0.56972 | load: 0.00257 [I 2022-10-27 11:00:29] ...trap/engines/engine.py.230: loss: 0.11716 [I 2022-10-27 11:00:29] ...optimizers/trijoint.py.151: total_norm: 11.758393 [I 2022-10-27 11:00:35] ...trap/engines/engine.py.224: train: epoch 63 | batch 1790/2304 [I 2022-10-27 11:00:35] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:47 [I 2022-10-27 11:00:35] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00259 [I 2022-10-27 11:00:35] ...trap/engines/engine.py.230: loss: 0.16969 [I 2022-10-27 11:00:35] ...optimizers/trijoint.py.151: total_norm: 17.198101 [I 2022-10-27 11:00:41] ...trap/engines/engine.py.224: train: epoch 63 | batch 1800/2304 [I 2022-10-27 11:00:41] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:45 [I 2022-10-27 11:00:41] ...trap/engines/engine.py.229: process: 0.56822 | load: 0.00253 [I 2022-10-27 11:00:41] ...trap/engines/engine.py.230: loss: 0.16811 [I 2022-10-27 11:00:41] ...optimizers/trijoint.py.151: total_norm: 27.306509 [I 2022-10-27 11:00:46] ...trap/engines/engine.py.224: train: epoch 63 | batch 1810/2304 [I 2022-10-27 11:00:46] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:36 [I 2022-10-27 11:00:46] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00257 [I 2022-10-27 11:00:46] ...trap/engines/engine.py.230: loss: 0.13099 [I 2022-10-27 11:00:46] ...optimizers/trijoint.py.151: total_norm: 14.582289 [I 2022-10-27 11:00:52] ...trap/engines/engine.py.224: train: epoch 63 | batch 1820/2304 [I 2022-10-27 11:00:52] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:31 [I 2022-10-27 11:00:52] ...trap/engines/engine.py.229: process: 0.57695 | load: 0.00253 [I 2022-10-27 11:00:52] ...trap/engines/engine.py.230: loss: 0.12942 [I 2022-10-27 11:00:52] ...optimizers/trijoint.py.151: total_norm: 18.482790 [I 2022-10-27 11:00:58] ...trap/engines/engine.py.224: train: epoch 63 | batch 1830/2304 [I 2022-10-27 11:00:58] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:25 [I 2022-10-27 11:00:58] ...trap/engines/engine.py.229: process: 0.55096 | load: 0.00251 [I 2022-10-27 11:00:58] ...trap/engines/engine.py.230: loss: 0.24588 [I 2022-10-27 11:00:58] ...optimizers/trijoint.py.151: total_norm: 18.620085 [I 2022-10-27 11:01:03] ...trap/engines/engine.py.224: train: epoch 63 | batch 1840/2304 [I 2022-10-27 11:01:03] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:20 [I 2022-10-27 11:01:03] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00258 [I 2022-10-27 11:01:03] ...trap/engines/engine.py.230: loss: 0.14003 [I 2022-10-27 11:01:03] ...optimizers/trijoint.py.151: total_norm: 16.687336 [I 2022-10-27 11:01:09] ...trap/engines/engine.py.224: train: epoch 63 | batch 1850/2304 [I 2022-10-27 11:01:09] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:04:12 [I 2022-10-27 11:01:09] ...trap/engines/engine.py.229: process: 0.54671 | load: 0.00257 [I 2022-10-27 11:01:09] ...trap/engines/engine.py.230: loss: 0.17740 [I 2022-10-27 11:01:09] ...optimizers/trijoint.py.151: total_norm: 19.321093 [I 2022-10-27 11:01:14] ...trap/engines/engine.py.224: train: epoch 63 | batch 1860/2304 [I 2022-10-27 11:01:14] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:04:08 [I 2022-10-27 11:01:14] ...trap/engines/engine.py.229: process: 0.57511 | load: 0.00261 [I 2022-10-27 11:01:14] ...trap/engines/engine.py.230: loss: 0.11536 [I 2022-10-27 11:01:14] ...optimizers/trijoint.py.151: total_norm: 13.060379 [I 2022-10-27 11:01:20] ...trap/engines/engine.py.224: train: epoch 63 | batch 1870/2304 [I 2022-10-27 11:01:20] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:04:02 [I 2022-10-27 11:01:20] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00272 [I 2022-10-27 11:01:20] ...trap/engines/engine.py.230: loss: 0.21919 [I 2022-10-27 11:01:20] ...optimizers/trijoint.py.151: total_norm: 21.696718 [I 2022-10-27 11:01:26] ...trap/engines/engine.py.224: train: epoch 63 | batch 1880/2304 [I 2022-10-27 11:01:26] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:57 [I 2022-10-27 11:01:26] ...trap/engines/engine.py.229: process: 0.57711 | load: 0.00255 [I 2022-10-27 11:01:26] ...trap/engines/engine.py.230: loss: 0.17595 [I 2022-10-27 11:01:26] ...optimizers/trijoint.py.151: total_norm: 18.644154 [I 2022-10-27 11:01:31] ...trap/engines/engine.py.224: train: epoch 63 | batch 1890/2304 [I 2022-10-27 11:01:31] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:52 [I 2022-10-27 11:01:31] ...trap/engines/engine.py.229: process: 0.56332 | load: 0.00252 [I 2022-10-27 11:01:31] ...trap/engines/engine.py.230: loss: 0.18197 [I 2022-10-27 11:01:31] ...optimizers/trijoint.py.151: total_norm: 23.198402 [I 2022-10-27 11:01:37] ...trap/engines/engine.py.224: train: epoch 63 | batch 1900/2304 [I 2022-10-27 11:01:37] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:47 [I 2022-10-27 11:01:37] ...trap/engines/engine.py.229: process: 0.57769 | load: 0.00254 [I 2022-10-27 11:01:37] ...trap/engines/engine.py.230: loss: 0.18077 [I 2022-10-27 11:01:37] ...optimizers/trijoint.py.151: total_norm: 35.609764 [I 2022-10-27 11:01:42] ...trap/engines/engine.py.224: train: epoch 63 | batch 1910/2304 [I 2022-10-27 11:01:42] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:41 [I 2022-10-27 11:01:42] ...trap/engines/engine.py.229: process: 0.56576 | load: 0.00260 [I 2022-10-27 11:01:42] ...trap/engines/engine.py.230: loss: 0.12754 [I 2022-10-27 11:01:42] ...optimizers/trijoint.py.151: total_norm: 15.563032 [I 2022-10-27 11:01:48] ...trap/engines/engine.py.224: train: epoch 63 | batch 1920/2304 [I 2022-10-27 11:01:48] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:36 [I 2022-10-27 11:01:48] ...trap/engines/engine.py.229: process: 0.57494 | load: 0.00259 [I 2022-10-27 11:01:48] ...trap/engines/engine.py.230: loss: 0.17406 [I 2022-10-27 11:01:48] ...optimizers/trijoint.py.151: total_norm: 19.646976 [I 2022-10-27 11:01:54] ...trap/engines/engine.py.224: train: epoch 63 | batch 1930/2304 [I 2022-10-27 11:01:54] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:30 [I 2022-10-27 11:01:54] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00264 [I 2022-10-27 11:01:54] ...trap/engines/engine.py.230: loss: 0.16549 [I 2022-10-27 11:01:54] ...optimizers/trijoint.py.151: total_norm: 14.478661 [I 2022-10-27 11:01:59] ...trap/engines/engine.py.224: train: epoch 63 | batch 1940/2304 [I 2022-10-27 11:01:59] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:25 [I 2022-10-27 11:01:59] ...trap/engines/engine.py.229: process: 0.56709 | load: 0.00259 [I 2022-10-27 11:01:59] ...trap/engines/engine.py.230: loss: 0.14951 [I 2022-10-27 11:01:59] ...optimizers/trijoint.py.151: total_norm: 13.127860 [I 2022-10-27 11:02:05] ...trap/engines/engine.py.224: train: epoch 63 | batch 1950/2304 [I 2022-10-27 11:02:05] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:16 [I 2022-10-27 11:02:05] ...trap/engines/engine.py.229: process: 0.54306 | load: 0.00258 [I 2022-10-27 11:02:05] ...trap/engines/engine.py.230: loss: 0.12582 [I 2022-10-27 11:02:05] ...optimizers/trijoint.py.151: total_norm: 19.951561 [I 2022-10-27 11:02:10] ...trap/engines/engine.py.224: train: epoch 63 | batch 1960/2304 [I 2022-10-27 11:02:10] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:03:12 [I 2022-10-27 11:02:10] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00259 [I 2022-10-27 11:02:10] ...trap/engines/engine.py.230: loss: 0.13397 [I 2022-10-27 11:02:10] ...optimizers/trijoint.py.151: total_norm: 28.755333 [I 2022-10-27 11:02:16] ...trap/engines/engine.py.224: train: epoch 63 | batch 1970/2304 [I 2022-10-27 11:02:16] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:03:06 [I 2022-10-27 11:02:16] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00261 [I 2022-10-27 11:02:16] ...trap/engines/engine.py.230: loss: 0.20296 [I 2022-10-27 11:02:16] ...optimizers/trijoint.py.151: total_norm: 26.390255 [I 2022-10-27 11:02:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 1980/2304 [I 2022-10-27 11:02:22] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:03:00 [I 2022-10-27 11:02:22] ...trap/engines/engine.py.229: process: 0.54725 | load: 0.00255 [I 2022-10-27 11:02:22] ...trap/engines/engine.py.230: loss: 0.18434 [I 2022-10-27 11:02:22] ...optimizers/trijoint.py.151: total_norm: 13.057573 [I 2022-10-27 11:02:27] ...trap/engines/engine.py.224: train: epoch 63 | batch 1990/2304 [I 2022-10-27 11:02:27] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:55 [I 2022-10-27 11:02:27] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00276 [I 2022-10-27 11:02:27] ...trap/engines/engine.py.230: loss: 0.14566 [I 2022-10-27 11:02:27] ...optimizers/trijoint.py.151: total_norm: 16.376394 [I 2022-10-27 11:02:33] ...trap/engines/engine.py.224: train: epoch 63 | batch 2000/2304 [I 2022-10-27 11:02:33] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:49 [I 2022-10-27 11:02:33] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00254 [I 2022-10-27 11:02:33] ...trap/engines/engine.py.230: loss: 0.12737 [I 2022-10-27 11:02:33] ...optimizers/trijoint.py.151: total_norm: 9.529314 [I 2022-10-27 11:02:38] ...trap/engines/engine.py.224: train: epoch 63 | batch 2010/2304 [I 2022-10-27 11:02:38] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:42 [I 2022-10-27 11:02:38] ...trap/engines/engine.py.229: process: 0.54389 | load: 0.00255 [I 2022-10-27 11:02:38] ...trap/engines/engine.py.230: loss: 0.13167 [I 2022-10-27 11:02:38] ...optimizers/trijoint.py.151: total_norm: 7.736174 [I 2022-10-27 11:02:44] ...trap/engines/engine.py.224: train: epoch 63 | batch 2020/2304 [I 2022-10-27 11:02:44] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:37 [I 2022-10-27 11:02:44] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00252 [I 2022-10-27 11:02:44] ...trap/engines/engine.py.230: loss: 0.13786 [I 2022-10-27 11:02:44] ...optimizers/trijoint.py.151: total_norm: 23.534359 [I 2022-10-27 11:02:50] ...trap/engines/engine.py.224: train: epoch 63 | batch 2030/2304 [I 2022-10-27 11:02:50] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:31 [I 2022-10-27 11:02:50] ...trap/engines/engine.py.229: process: 0.55597 | load: 0.00252 [I 2022-10-27 11:02:50] ...trap/engines/engine.py.230: loss: 0.13662 [I 2022-10-27 11:02:50] ...optimizers/trijoint.py.151: total_norm: 12.309140 [I 2022-10-27 11:02:55] ...trap/engines/engine.py.224: train: epoch 63 | batch 2040/2304 [I 2022-10-27 11:02:55] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:26 [I 2022-10-27 11:02:55] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00256 [I 2022-10-27 11:02:55] ...trap/engines/engine.py.230: loss: 0.11525 [I 2022-10-27 11:02:55] ...optimizers/trijoint.py.151: total_norm: 18.574835 [I 2022-10-27 11:03:01] ...trap/engines/engine.py.224: train: epoch 63 | batch 2050/2304 [I 2022-10-27 11:03:01] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:21 [I 2022-10-27 11:03:01] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00253 [I 2022-10-27 11:03:01] ...trap/engines/engine.py.230: loss: 0.15518 [I 2022-10-27 11:03:01] ...optimizers/trijoint.py.151: total_norm: 34.809265 [I 2022-10-27 11:03:06] ...trap/engines/engine.py.224: train: epoch 63 | batch 2060/2304 [I 2022-10-27 11:03:06] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:13 [I 2022-10-27 11:03:06] ...trap/engines/engine.py.229: process: 0.54476 | load: 0.00252 [I 2022-10-27 11:03:06] ...trap/engines/engine.py.230: loss: 0.14454 [I 2022-10-27 11:03:06] ...optimizers/trijoint.py.151: total_norm: 11.125364 [I 2022-10-27 11:03:12] ...trap/engines/engine.py.224: train: epoch 63 | batch 2070/2304 [I 2022-10-27 11:03:12] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:09 [I 2022-10-27 11:03:12] ...trap/engines/engine.py.229: process: 0.55411 | load: 0.00266 [I 2022-10-27 11:03:12] ...trap/engines/engine.py.230: loss: 0.19875 [I 2022-10-27 11:03:12] ...optimizers/trijoint.py.151: total_norm: 10.657815 [I 2022-10-27 11:03:17] ...trap/engines/engine.py.224: train: epoch 63 | batch 2080/2304 [I 2022-10-27 11:03:17] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:03 [I 2022-10-27 11:03:17] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00261 [I 2022-10-27 11:03:17] ...trap/engines/engine.py.230: loss: 0.14975 [I 2022-10-27 11:03:17] ...optimizers/trijoint.py.151: total_norm: 8.741260 [I 2022-10-27 11:03:23] ...trap/engines/engine.py.224: train: epoch 63 | batch 2090/2304 [I 2022-10-27 11:03:23] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:58 [I 2022-10-27 11:03:23] ...trap/engines/engine.py.229: process: 0.55049 | load: 0.00253 [I 2022-10-27 11:03:23] ...trap/engines/engine.py.230: loss: 0.11215 [I 2022-10-27 11:03:23] ...optimizers/trijoint.py.151: total_norm: 11.605958 [I 2022-10-27 11:03:28] ...trap/engines/engine.py.224: train: epoch 63 | batch 2100/2304 [I 2022-10-27 11:03:28] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:52 [I 2022-10-27 11:03:28] ...trap/engines/engine.py.229: process: 0.54397 | load: 0.00257 [I 2022-10-27 11:03:28] ...trap/engines/engine.py.230: loss: 0.18931 [I 2022-10-27 11:03:28] ...optimizers/trijoint.py.151: total_norm: 23.181395 [I 2022-10-27 11:03:34] ...trap/engines/engine.py.224: train: epoch 63 | batch 2110/2304 [I 2022-10-27 11:03:34] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:48 [I 2022-10-27 11:03:34] ...trap/engines/engine.py.229: process: 0.57479 | load: 0.00255 [I 2022-10-27 11:03:34] ...trap/engines/engine.py.230: loss: 0.19472 [I 2022-10-27 11:03:34] ...optimizers/trijoint.py.151: total_norm: 16.321028 [I 2022-10-27 11:03:40] ...trap/engines/engine.py.224: train: epoch 63 | batch 2120/2304 [I 2022-10-27 11:03:40] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:42 [I 2022-10-27 11:03:40] ...trap/engines/engine.py.229: process: 0.55320 | load: 0.00258 [I 2022-10-27 11:03:40] ...trap/engines/engine.py.230: loss: 0.15308 [I 2022-10-27 11:03:40] ...optimizers/trijoint.py.151: total_norm: 20.838089 [I 2022-10-27 11:03:45] ...trap/engines/engine.py.224: train: epoch 63 | batch 2130/2304 [I 2022-10-27 11:03:45] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:36 [I 2022-10-27 11:03:45] ...trap/engines/engine.py.229: process: 0.54994 | load: 0.00255 [I 2022-10-27 11:03:45] ...trap/engines/engine.py.230: loss: 0.12675 [I 2022-10-27 11:03:45] ...optimizers/trijoint.py.151: total_norm: 10.815895 [I 2022-10-27 11:03:51] ...trap/engines/engine.py.224: train: epoch 63 | batch 2140/2304 [I 2022-10-27 11:03:51] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:31 [I 2022-10-27 11:03:51] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00254 [I 2022-10-27 11:03:51] ...trap/engines/engine.py.230: loss: 0.11281 [I 2022-10-27 11:03:51] ...optimizers/trijoint.py.151: total_norm: 9.301509 [I 2022-10-27 11:03:56] ...trap/engines/engine.py.224: train: epoch 63 | batch 2150/2304 [I 2022-10-27 11:03:56] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:25 [I 2022-10-27 11:03:56] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00250 [I 2022-10-27 11:03:56] ...trap/engines/engine.py.230: loss: 0.14412 [I 2022-10-27 11:03:56] ...optimizers/trijoint.py.151: total_norm: 15.917002 [I 2022-10-27 11:04:02] ...trap/engines/engine.py.224: train: epoch 63 | batch 2160/2304 [I 2022-10-27 11:04:02] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:19 [I 2022-10-27 11:04:02] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00248 [I 2022-10-27 11:04:02] ...trap/engines/engine.py.230: loss: 0.11218 [I 2022-10-27 11:04:02] ...optimizers/trijoint.py.151: total_norm: 7.465106 [I 2022-10-27 11:04:07] ...trap/engines/engine.py.224: train: epoch 63 | batch 2170/2304 [I 2022-10-27 11:04:07] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:01:14 [I 2022-10-27 11:04:07] ...trap/engines/engine.py.229: process: 0.55050 | load: 0.00258 [I 2022-10-27 11:04:07] ...trap/engines/engine.py.230: loss: 0.24043 [I 2022-10-27 11:04:07] ...optimizers/trijoint.py.151: total_norm: 27.994989 [I 2022-10-27 11:04:13] ...trap/engines/engine.py.224: train: epoch 63 | batch 2180/2304 [I 2022-10-27 11:04:13] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:09 [I 2022-10-27 11:04:13] ...trap/engines/engine.py.229: process: 0.56550 | load: 0.00261 [I 2022-10-27 11:04:13] ...trap/engines/engine.py.230: loss: 0.10479 [I 2022-10-27 11:04:13] ...optimizers/trijoint.py.151: total_norm: 12.176485 [I 2022-10-27 11:04:19] ...trap/engines/engine.py.224: train: epoch 63 | batch 2190/2304 [I 2022-10-27 11:04:19] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:03 [I 2022-10-27 11:04:19] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00260 [I 2022-10-27 11:04:19] ...trap/engines/engine.py.230: loss: 0.28155 [I 2022-10-27 11:04:19] ...optimizers/trijoint.py.151: total_norm: 83.076691 [I 2022-10-27 11:04:24] ...trap/engines/engine.py.224: train: epoch 63 | batch 2200/2304 [I 2022-10-27 11:04:24] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:57 [I 2022-10-27 11:04:24] ...trap/engines/engine.py.229: process: 0.56532 | load: 0.00250 [I 2022-10-27 11:04:24] ...trap/engines/engine.py.230: loss: 0.14021 [I 2022-10-27 11:04:24] ...optimizers/trijoint.py.151: total_norm: 11.081993 [I 2022-10-27 11:04:30] ...trap/engines/engine.py.224: train: epoch 63 | batch 2210/2304 [I 2022-10-27 11:04:30] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:52 [I 2022-10-27 11:04:30] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00261 [I 2022-10-27 11:04:30] ...trap/engines/engine.py.230: loss: 0.15885 [I 2022-10-27 11:04:30] ...optimizers/trijoint.py.151: total_norm: 17.053318 [I 2022-10-27 11:04:35] ...trap/engines/engine.py.224: train: epoch 63 | batch 2220/2304 [I 2022-10-27 11:04:35] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:46 [I 2022-10-27 11:04:35] ...trap/engines/engine.py.229: process: 0.56713 | load: 0.00259 [I 2022-10-27 11:04:35] ...trap/engines/engine.py.230: loss: 0.16638 [I 2022-10-27 11:04:35] ...optimizers/trijoint.py.151: total_norm: 22.238127 [I 2022-10-27 11:04:41] ...trap/engines/engine.py.224: train: epoch 63 | batch 2230/2304 [I 2022-10-27 11:04:41] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:41 [I 2022-10-27 11:04:41] ...trap/engines/engine.py.229: process: 0.55719 | load: 0.00254 [I 2022-10-27 11:04:41] ...trap/engines/engine.py.230: loss: 0.18999 [I 2022-10-27 11:04:41] ...optimizers/trijoint.py.151: total_norm: 22.312920 [I 2022-10-27 11:04:46] ...trap/engines/engine.py.224: train: epoch 63 | batch 2240/2304 [I 2022-10-27 11:04:46] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:35 [I 2022-10-27 11:04:46] ...trap/engines/engine.py.229: process: 0.57195 | load: 0.00256 [I 2022-10-27 11:04:46] ...trap/engines/engine.py.230: loss: 0.23628 [I 2022-10-27 11:04:46] ...optimizers/trijoint.py.151: total_norm: 19.975725 [I 2022-10-27 11:04:52] ...trap/engines/engine.py.224: train: epoch 63 | batch 2250/2304 [I 2022-10-27 11:04:52] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:30 [I 2022-10-27 11:04:52] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00252 [I 2022-10-27 11:04:52] ...trap/engines/engine.py.230: loss: 0.14400 [I 2022-10-27 11:04:52] ...optimizers/trijoint.py.151: total_norm: 15.978503 [I 2022-10-27 11:04:58] ...trap/engines/engine.py.224: train: epoch 63 | batch 2260/2304 [I 2022-10-27 11:04:58] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:24 [I 2022-10-27 11:04:58] ...trap/engines/engine.py.229: process: 0.57471 | load: 0.00262 [I 2022-10-27 11:04:58] ...trap/engines/engine.py.230: loss: 0.16144 [I 2022-10-27 11:04:58] ...optimizers/trijoint.py.151: total_norm: 13.823263 [I 2022-10-27 11:05:03] ...trap/engines/engine.py.224: train: epoch 63 | batch 2270/2304 [I 2022-10-27 11:05:03] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:18 [I 2022-10-27 11:05:03] ...trap/engines/engine.py.229: process: 0.54949 | load: 0.00256 [I 2022-10-27 11:05:03] ...trap/engines/engine.py.230: loss: 0.14646 [I 2022-10-27 11:05:03] ...optimizers/trijoint.py.151: total_norm: 16.351868 [I 2022-10-27 11:05:09] ...trap/engines/engine.py.224: train: epoch 63 | batch 2280/2304 [I 2022-10-27 11:05:09] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:13 [I 2022-10-27 11:05:09] ...trap/engines/engine.py.229: process: 0.58959 | load: 0.00253 [I 2022-10-27 11:05:09] ...trap/engines/engine.py.230: loss: 0.15143 [I 2022-10-27 11:05:09] ...optimizers/trijoint.py.151: total_norm: 31.444557 [I 2022-10-27 11:05:14] ...trap/engines/engine.py.224: train: epoch 63 | batch 2290/2304 [I 2022-10-27 11:05:14] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:07 [I 2022-10-27 11:05:14] ...trap/engines/engine.py.229: process: 0.56260 | load: 0.00257 [I 2022-10-27 11:05:14] ...trap/engines/engine.py.230: loss: 0.10725 [I 2022-10-27 11:05:14] ...optimizers/trijoint.py.151: total_norm: 11.502278 [I 2022-10-27 11:05:20] ...trap/engines/engine.py.224: train: epoch 63 | batch 2300/2304 [I 2022-10-27 11:05:20] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:02 [I 2022-10-27 11:05:20] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00015 [I 2022-10-27 11:05:20] ...trap/engines/engine.py.230: loss: 0.13366 [I 2022-10-27 11:05:20] ...optimizers/trijoint.py.151: total_norm: 18.452166 [I 2022-10-27 11:05:22] ...trap/engines/engine.py.224: train: epoch 63 | batch 2304/2304 [I 2022-10-27 11:05:22] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:00 [I 2022-10-27 11:05:22] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00020 [I 2022-10-27 11:05:22] ...trap/engines/engine.py.230: loss: 0.12253 [I 2022-10-27 11:05:22] ...optimizers/trijoint.py.151: total_norm: 12.056033 [S 2022-10-27 11:05:23] ...trap/engines/engine.py.129: train_epoch.epoch: 63 [S 2022-10-27 11:05:23] ...trap/engines/engine.py.129: train_epoch.loss: 0.1519220832533278 [I 2022-10-27 11:05:23] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 11:05:24] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 63 [I 2022-10-27 11:05:24] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 11:05:26] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 11:05:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 11:05:26] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 11:05:28] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 11:05:29] ...trap/engines/engine.py.321: eval: epoch 63 | batch 0/510 [I 2022-10-27 11:05:29] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:37:48 [I 2022-10-27 11:05:29] ...trap/engines/engine.py.326: process: 4.44811 | load: 3.13394 [I 2022-10-27 11:05:32] ...trap/engines/engine.py.321: eval: epoch 63 | batch 10/510 [I 2022-10-27 11:05:32] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:28 [I 2022-10-27 11:05:32] ...trap/engines/engine.py.326: process: 0.33723 | load: 0.00019 [I 2022-10-27 11:05:35] ...trap/engines/engine.py.321: eval: epoch 63 | batch 20/510 [I 2022-10-27 11:05:35] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:08 [I 2022-10-27 11:05:35] ...trap/engines/engine.py.326: process: 0.32735 | load: 0.00018 [I 2022-10-27 11:05:39] ...trap/engines/engine.py.321: eval: epoch 63 | batch 30/510 [I 2022-10-27 11:05:39] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:42 [I 2022-10-27 11:05:39] ...trap/engines/engine.py.326: process: 0.32567 | load: 0.00016 [I 2022-10-27 11:05:42] ...trap/engines/engine.py.321: eval: epoch 63 | batch 40/510 [I 2022-10-27 11:05:42] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:35 [I 2022-10-27 11:05:42] ...trap/engines/engine.py.326: process: 0.32871 | load: 0.00017 [I 2022-10-27 11:05:45] ...trap/engines/engine.py.321: eval: epoch 63 | batch 50/510 [I 2022-10-27 11:05:45] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:34 [I 2022-10-27 11:05:45] ...trap/engines/engine.py.326: process: 0.35102 | load: 0.00021 [I 2022-10-27 11:05:49] ...trap/engines/engine.py.321: eval: epoch 63 | batch 60/510 [I 2022-10-27 11:05:49] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:32 [I 2022-10-27 11:05:49] ...trap/engines/engine.py.326: process: 0.33432 | load: 0.00020 [I 2022-10-27 11:05:52] ...trap/engines/engine.py.321: eval: epoch 63 | batch 70/510 [I 2022-10-27 11:05:52] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:27 [I 2022-10-27 11:05:52] ...trap/engines/engine.py.326: process: 0.32432 | load: 0.00018 [I 2022-10-27 11:05:55] ...trap/engines/engine.py.321: eval: epoch 63 | batch 80/510 [I 2022-10-27 11:05:55] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:23 [I 2022-10-27 11:05:55] ...trap/engines/engine.py.326: process: 0.32823 | load: 0.00019 [I 2022-10-27 11:05:59] ...trap/engines/engine.py.321: eval: epoch 63 | batch 90/510 [I 2022-10-27 11:05:59] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:28 [I 2022-10-27 11:05:59] ...trap/engines/engine.py.326: process: 0.37710 | load: 0.00016 [I 2022-10-27 11:06:02] ...trap/engines/engine.py.321: eval: epoch 63 | batch 100/510 [I 2022-10-27 11:06:02] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:15 [I 2022-10-27 11:06:02] ...trap/engines/engine.py.326: process: 0.29318 | load: 0.00020 [I 2022-10-27 11:06:05] ...trap/engines/engine.py.321: eval: epoch 63 | batch 110/510 [I 2022-10-27 11:06:05] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 11:06:05] ...trap/engines/engine.py.326: process: 0.28436 | load: 0.00019 [I 2022-10-27 11:06:08] ...trap/engines/engine.py.321: eval: epoch 63 | batch 120/510 [I 2022-10-27 11:06:08] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:57 [I 2022-10-27 11:06:08] ...trap/engines/engine.py.326: process: 0.28850 | load: 0.00015 [I 2022-10-27 11:06:11] ...trap/engines/engine.py.321: eval: epoch 63 | batch 130/510 [I 2022-10-27 11:06:11] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:50 [I 2022-10-27 11:06:11] ...trap/engines/engine.py.326: process: 0.28847 | load: 0.00017 [I 2022-10-27 11:06:14] ...trap/engines/engine.py.321: eval: epoch 63 | batch 140/510 [I 2022-10-27 11:06:14] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:46 [I 2022-10-27 11:06:14] ...trap/engines/engine.py.326: process: 0.29363 | load: 0.00020 [I 2022-10-27 11:06:17] ...trap/engines/engine.py.321: eval: epoch 63 | batch 150/510 [I 2022-10-27 11:06:17] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 11:06:17] ...trap/engines/engine.py.326: process: 0.28717 | load: 0.00019 [I 2022-10-27 11:06:20] ...trap/engines/engine.py.321: eval: epoch 63 | batch 160/510 [I 2022-10-27 11:06:20] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 11:06:20] ...trap/engines/engine.py.326: process: 0.30251 | load: 0.00016 [I 2022-10-27 11:06:23] ...trap/engines/engine.py.321: eval: epoch 63 | batch 170/510 [I 2022-10-27 11:06:23] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:39 [I 2022-10-27 11:06:23] ...trap/engines/engine.py.326: process: 0.30663 | load: 0.00015 [I 2022-10-27 11:06:26] ...trap/engines/engine.py.321: eval: epoch 63 | batch 180/510 [I 2022-10-27 11:06:26] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:37 [I 2022-10-27 11:06:26] ...trap/engines/engine.py.326: process: 0.29030 | load: 0.00014 [I 2022-10-27 11:06:29] ...trap/engines/engine.py.321: eval: epoch 63 | batch 190/510 [I 2022-10-27 11:06:29] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:33 [I 2022-10-27 11:06:29] ...trap/engines/engine.py.326: process: 0.30496 | load: 0.00014 [I 2022-10-27 11:06:31] ...trap/engines/engine.py.321: eval: epoch 63 | batch 200/510 [I 2022-10-27 11:06:31] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 11:06:31] ...trap/engines/engine.py.326: process: 0.29391 | load: 0.00021 [I 2022-10-27 11:06:34] ...trap/engines/engine.py.321: eval: epoch 63 | batch 210/510 [I 2022-10-27 11:06:34] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:25 [I 2022-10-27 11:06:34] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00014 [I 2022-10-27 11:06:37] ...trap/engines/engine.py.321: eval: epoch 63 | batch 220/510 [I 2022-10-27 11:06:37] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 11:06:37] ...trap/engines/engine.py.326: process: 0.28935 | load: 0.00016 [I 2022-10-27 11:06:40] ...trap/engines/engine.py.321: eval: epoch 63 | batch 230/510 [I 2022-10-27 11:06:40] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 11:06:40] ...trap/engines/engine.py.326: process: 0.28958 | load: 0.00019 [I 2022-10-27 11:06:43] ...trap/engines/engine.py.321: eval: epoch 63 | batch 240/510 [I 2022-10-27 11:06:43] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 11:06:43] ...trap/engines/engine.py.326: process: 0.28665 | load: 0.00015 [I 2022-10-27 11:06:46] ...trap/engines/engine.py.321: eval: epoch 63 | batch 250/510 [I 2022-10-27 11:06:46] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 11:06:46] ...trap/engines/engine.py.326: process: 0.28513 | load: 0.00015 [I 2022-10-27 11:06:49] ...trap/engines/engine.py.321: eval: epoch 63 | batch 260/510 [I 2022-10-27 11:06:49] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 11:06:49] ...trap/engines/engine.py.326: process: 0.29510 | load: 0.00022 [I 2022-10-27 11:06:52] ...trap/engines/engine.py.321: eval: epoch 63 | batch 270/510 [I 2022-10-27 11:06:52] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 11:06:52] ...trap/engines/engine.py.326: process: 0.29812 | load: 0.00024 [I 2022-10-27 11:06:55] ...trap/engines/engine.py.321: eval: epoch 63 | batch 280/510 [I 2022-10-27 11:06:55] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:07 [I 2022-10-27 11:06:55] ...trap/engines/engine.py.326: process: 0.29287 | load: 0.00019 [I 2022-10-27 11:06:58] ...trap/engines/engine.py.321: eval: epoch 63 | batch 290/510 [I 2022-10-27 11:06:58] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 11:06:58] ...trap/engines/engine.py.326: process: 0.29311 | load: 0.00015 [I 2022-10-27 11:07:00] ...trap/engines/engine.py.321: eval: epoch 63 | batch 300/510 [I 2022-10-27 11:07:00] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 11:07:00] ...trap/engines/engine.py.326: process: 0.28630 | load: 0.00025 [I 2022-10-27 11:07:03] ...trap/engines/engine.py.321: eval: epoch 63 | batch 310/510 [I 2022-10-27 11:07:03] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 11:07:03] ...trap/engines/engine.py.326: process: 0.28155 | load: 0.00021 [I 2022-10-27 11:07:06] ...trap/engines/engine.py.321: eval: epoch 63 | batch 320/510 [I 2022-10-27 11:07:06] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:55 [I 2022-10-27 11:07:06] ...trap/engines/engine.py.326: process: 0.30225 | load: 0.00014 [I 2022-10-27 11:07:09] ...trap/engines/engine.py.321: eval: epoch 63 | batch 330/510 [I 2022-10-27 11:07:09] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:52 [I 2022-10-27 11:07:09] ...trap/engines/engine.py.326: process: 0.29387 | load: 0.00015 [I 2022-10-27 11:07:12] ...trap/engines/engine.py.321: eval: epoch 63 | batch 340/510 [I 2022-10-27 11:07:12] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 11:07:12] ...trap/engines/engine.py.326: process: 0.28403 | load: 0.00015 [I 2022-10-27 11:07:15] ...trap/engines/engine.py.321: eval: epoch 63 | batch 350/510 [I 2022-10-27 11:07:15] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 11:07:15] ...trap/engines/engine.py.326: process: 0.29449 | load: 0.00020 [I 2022-10-27 11:07:18] ...trap/engines/engine.py.321: eval: epoch 63 | batch 360/510 [I 2022-10-27 11:07:18] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 11:07:18] ...trap/engines/engine.py.326: process: 0.29084 | load: 0.00015 [I 2022-10-27 11:07:21] ...trap/engines/engine.py.321: eval: epoch 63 | batch 370/510 [I 2022-10-27 11:07:21] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 11:07:21] ...trap/engines/engine.py.326: process: 0.28953 | load: 0.00016 [I 2022-10-27 11:07:24] ...trap/engines/engine.py.321: eval: epoch 63 | batch 380/510 [I 2022-10-27 11:07:24] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 11:07:24] ...trap/engines/engine.py.326: process: 0.28580 | load: 0.00018 [I 2022-10-27 11:07:26] ...trap/engines/engine.py.321: eval: epoch 63 | batch 390/510 [I 2022-10-27 11:07:26] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 11:07:26] ...trap/engines/engine.py.326: process: 0.28691 | load: 0.00015 [I 2022-10-27 11:07:29] ...trap/engines/engine.py.321: eval: epoch 63 | batch 400/510 [I 2022-10-27 11:07:29] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:32 [I 2022-10-27 11:07:29] ...trap/engines/engine.py.326: process: 0.28849 | load: 0.00015 [I 2022-10-27 11:07:32] ...trap/engines/engine.py.321: eval: epoch 63 | batch 410/510 [I 2022-10-27 11:07:32] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 11:07:32] ...trap/engines/engine.py.326: process: 0.28915 | load: 0.00016 [I 2022-10-27 11:07:35] ...trap/engines/engine.py.321: eval: epoch 63 | batch 420/510 [I 2022-10-27 11:07:35] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 11:07:35] ...trap/engines/engine.py.326: process: 0.29019 | load: 0.00014 [I 2022-10-27 11:07:38] ...trap/engines/engine.py.321: eval: epoch 63 | batch 430/510 [I 2022-10-27 11:07:38] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 11:07:38] ...trap/engines/engine.py.326: process: 0.28831 | load: 0.00021 [I 2022-10-27 11:07:41] ...trap/engines/engine.py.321: eval: epoch 63 | batch 440/510 [I 2022-10-27 11:07:41] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 11:07:41] ...trap/engines/engine.py.326: process: 0.28984 | load: 0.00019 [I 2022-10-27 11:07:44] ...trap/engines/engine.py.321: eval: epoch 63 | batch 450/510 [I 2022-10-27 11:07:44] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 11:07:44] ...trap/engines/engine.py.326: process: 0.28994 | load: 0.00016 [I 2022-10-27 11:07:47] ...trap/engines/engine.py.321: eval: epoch 63 | batch 460/510 [I 2022-10-27 11:07:47] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 11:07:47] ...trap/engines/engine.py.326: process: 0.29157 | load: 0.00016 [I 2022-10-27 11:07:50] ...trap/engines/engine.py.321: eval: epoch 63 | batch 470/510 [I 2022-10-27 11:07:50] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 11:07:50] ...trap/engines/engine.py.326: process: 0.28990 | load: 0.00016 [I 2022-10-27 11:07:52] ...trap/engines/engine.py.321: eval: epoch 63 | batch 480/510 [I 2022-10-27 11:07:52] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 11:07:52] ...trap/engines/engine.py.326: process: 0.28593 | load: 0.00016 [I 2022-10-27 11:07:55] ...trap/engines/engine.py.321: eval: epoch 63 | batch 490/510 [I 2022-10-27 11:07:55] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 11:07:55] ...trap/engines/engine.py.326: process: 0.28675 | load: 0.00025 [I 2022-10-27 11:07:58] ...trap/engines/engine.py.321: eval: epoch 63 | batch 500/510 [I 2022-10-27 11:07:58] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 11:07:58] ...trap/engines/engine.py.326: process: 0.28922 | load: 0.00015 [I 2022-10-27 11:08:01] ...trap/engines/engine.py.321: eval: epoch 63 | batch 510/510 [I 2022-10-27 11:08:01] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 11:08:01] ...trap/engines/engine.py.326: process: 0.29749 | load: 0.00010 [S 2022-10-27 11:08:02] ...trap/engines/engine.py.132: eval_epoch.epoch: 63 [I 2022-10-27 11:08:02] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.716500 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908500 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939800 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.709800 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904600 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937300 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008789 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008405 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006145 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009968 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006232 [S 2022-10-27 11:08:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004148 [I 2022-10-27 11:08:07] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 11:08:07] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 11:08:08] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 11:08:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 11:08:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 11:08:11] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 11:08:12] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 11:08:18] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 11:08:18] ...trap/engines/engine.py.163: Training model on trainset for epoch 64 [I 2022-10-27 11:08:18] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 11:08:21] ...trap/engines/engine.py.224: train: epoch 64 | batch 0/2304 [I 2022-10-27 11:08:21] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:43 [I 2022-10-27 11:08:21] ...trap/engines/engine.py.229: process: 2.70116 | load: 2.08203 [I 2022-10-27 11:08:21] ...trap/engines/engine.py.230: loss: 0.13006 [I 2022-10-27 11:08:21] ...optimizers/trijoint.py.151: total_norm: 18.470671 [I 2022-10-27 11:08:26] ...trap/engines/engine.py.224: train: epoch 64 | batch 10/2304 [I 2022-10-27 11:08:26] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:02 [I 2022-10-27 11:08:26] ...trap/engines/engine.py.229: process: 0.54921 | load: 0.00307 [I 2022-10-27 11:08:26] ...trap/engines/engine.py.230: loss: 0.18179 [I 2022-10-27 11:08:26] ...optimizers/trijoint.py.151: total_norm: 16.213385 [I 2022-10-27 11:08:32] ...trap/engines/engine.py.224: train: epoch 64 | batch 20/2304 [I 2022-10-27 11:08:32] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:50 [I 2022-10-27 11:08:32] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00348 [I 2022-10-27 11:08:32] ...trap/engines/engine.py.230: loss: 0.21543 [I 2022-10-27 11:08:32] ...optimizers/trijoint.py.151: total_norm: 27.664322 [I 2022-10-27 11:08:38] ...trap/engines/engine.py.224: train: epoch 64 | batch 30/2304 [I 2022-10-27 11:08:38] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:12 [I 2022-10-27 11:08:38] ...trap/engines/engine.py.229: process: 0.54410 | load: 0.00304 [I 2022-10-27 11:08:38] ...trap/engines/engine.py.230: loss: 0.17022 [I 2022-10-27 11:08:38] ...optimizers/trijoint.py.151: total_norm: 16.382477 [I 2022-10-27 11:08:43] ...trap/engines/engine.py.224: train: epoch 64 | batch 40/2304 [I 2022-10-27 11:08:43] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:01 [I 2022-10-27 11:08:43] ...trap/engines/engine.py.229: process: 0.56347 | load: 0.00302 [I 2022-10-27 11:08:43] ...trap/engines/engine.py.230: loss: 0.19653 [I 2022-10-27 11:08:43] ...optimizers/trijoint.py.151: total_norm: 16.433767 [I 2022-10-27 11:08:49] ...trap/engines/engine.py.224: train: epoch 64 | batch 50/2304 [I 2022-10-27 11:08:49] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:45 [I 2022-10-27 11:08:49] ...trap/engines/engine.py.229: process: 0.54857 | load: 0.00264 [I 2022-10-27 11:08:49] ...trap/engines/engine.py.230: loss: 0.11050 [I 2022-10-27 11:08:49] ...optimizers/trijoint.py.151: total_norm: 7.506915 [I 2022-10-27 11:08:54] ...trap/engines/engine.py.224: train: epoch 64 | batch 60/2304 [I 2022-10-27 11:08:54] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:59 [I 2022-10-27 11:08:54] ...trap/engines/engine.py.229: process: 0.57833 | load: 0.00282 [I 2022-10-27 11:08:54] ...trap/engines/engine.py.230: loss: 0.15406 [I 2022-10-27 11:08:54] ...optimizers/trijoint.py.151: total_norm: 23.058655 [I 2022-10-27 11:09:00] ...trap/engines/engine.py.224: train: epoch 64 | batch 70/2304 [I 2022-10-27 11:09:00] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:45 [I 2022-10-27 11:09:00] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00274 [I 2022-10-27 11:09:00] ...trap/engines/engine.py.230: loss: 0.16974 [I 2022-10-27 11:09:00] ...optimizers/trijoint.py.151: total_norm: 26.667345 [I 2022-10-27 11:09:06] ...trap/engines/engine.py.224: train: epoch 64 | batch 80/2304 [I 2022-10-27 11:09:06] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:35 [I 2022-10-27 11:09:06] ...trap/engines/engine.py.229: process: 0.55280 | load: 0.00281 [I 2022-10-27 11:09:06] ...trap/engines/engine.py.230: loss: 0.13777 [I 2022-10-27 11:09:06] ...optimizers/trijoint.py.151: total_norm: 16.723326 [I 2022-10-27 11:09:11] ...trap/engines/engine.py.224: train: epoch 64 | batch 90/2304 [I 2022-10-27 11:09:11] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:42 [I 2022-10-27 11:09:11] ...trap/engines/engine.py.229: process: 0.56991 | load: 0.00267 [I 2022-10-27 11:09:11] ...trap/engines/engine.py.230: loss: 0.12951 [I 2022-10-27 11:09:11] ...optimizers/trijoint.py.151: total_norm: 8.746630 [I 2022-10-27 11:09:17] ...trap/engines/engine.py.224: train: epoch 64 | batch 100/2304 [I 2022-10-27 11:09:17] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:37 [I 2022-10-27 11:09:17] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00292 [I 2022-10-27 11:09:17] ...trap/engines/engine.py.230: loss: 0.20316 [I 2022-10-27 11:09:17] ...optimizers/trijoint.py.151: total_norm: 15.938852 [I 2022-10-27 11:09:22] ...trap/engines/engine.py.224: train: epoch 64 | batch 110/2304 [I 2022-10-27 11:09:22] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:20 [I 2022-10-27 11:09:22] ...trap/engines/engine.py.229: process: 0.55525 | load: 0.00288 [I 2022-10-27 11:09:22] ...trap/engines/engine.py.230: loss: 0.13385 [I 2022-10-27 11:09:22] ...optimizers/trijoint.py.151: total_norm: 10.582187 [I 2022-10-27 11:09:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 120/2304 [I 2022-10-27 11:09:28] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:16 [I 2022-10-27 11:09:28] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00256 [I 2022-10-27 11:09:28] ...trap/engines/engine.py.230: loss: 0.15828 [I 2022-10-27 11:09:28] ...optimizers/trijoint.py.151: total_norm: 20.366117 [I 2022-10-27 11:09:34] ...trap/engines/engine.py.224: train: epoch 64 | batch 130/2304 [I 2022-10-27 11:09:34] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:21:03 [I 2022-10-27 11:09:34] ...trap/engines/engine.py.229: process: 0.57809 | load: 0.00312 [I 2022-10-27 11:09:34] ...trap/engines/engine.py.230: loss: 0.12837 [I 2022-10-27 11:09:34] ...optimizers/trijoint.py.151: total_norm: 12.790790 [I 2022-10-27 11:09:39] ...trap/engines/engine.py.224: train: epoch 64 | batch 140/2304 [I 2022-10-27 11:09:39] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:07 [I 2022-10-27 11:09:39] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00266 [I 2022-10-27 11:09:39] ...trap/engines/engine.py.230: loss: 0.15756 [I 2022-10-27 11:09:39] ...optimizers/trijoint.py.151: total_norm: 16.522442 [I 2022-10-27 11:09:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 150/2304 [I 2022-10-27 11:09:45] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:55 [I 2022-10-27 11:09:45] ...trap/engines/engine.py.229: process: 0.54877 | load: 0.00252 [I 2022-10-27 11:09:45] ...trap/engines/engine.py.230: loss: 0.11263 [I 2022-10-27 11:09:45] ...optimizers/trijoint.py.151: total_norm: 12.492235 [I 2022-10-27 11:09:50] ...trap/engines/engine.py.224: train: epoch 64 | batch 160/2304 [I 2022-10-27 11:09:50] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:48 [I 2022-10-27 11:09:50] ...trap/engines/engine.py.229: process: 0.56293 | load: 0.00269 [I 2022-10-27 11:09:50] ...trap/engines/engine.py.230: loss: 0.12156 [I 2022-10-27 11:09:50] ...optimizers/trijoint.py.151: total_norm: 12.757715 [I 2022-10-27 11:09:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 170/2304 [I 2022-10-27 11:09:56] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:43 [I 2022-10-27 11:09:56] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00264 [I 2022-10-27 11:09:56] ...trap/engines/engine.py.230: loss: 0.11968 [I 2022-10-27 11:09:56] ...optimizers/trijoint.py.151: total_norm: 15.874361 [I 2022-10-27 11:10:02] ...trap/engines/engine.py.224: train: epoch 64 | batch 180/2304 [I 2022-10-27 11:10:02] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:41 [I 2022-10-27 11:10:02] ...trap/engines/engine.py.229: process: 0.55019 | load: 0.00260 [I 2022-10-27 11:10:02] ...trap/engines/engine.py.230: loss: 0.19402 [I 2022-10-27 11:10:02] ...optimizers/trijoint.py.151: total_norm: 20.672667 [I 2022-10-27 11:10:07] ...trap/engines/engine.py.224: train: epoch 64 | batch 190/2304 [I 2022-10-27 11:10:07] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:41 [I 2022-10-27 11:10:07] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00289 [I 2022-10-27 11:10:07] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-27 11:10:07] ...optimizers/trijoint.py.151: total_norm: 50.513836 [I 2022-10-27 11:10:13] ...trap/engines/engine.py.224: train: epoch 64 | batch 200/2304 [I 2022-10-27 11:10:13] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:36 [I 2022-10-27 11:10:13] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00253 [I 2022-10-27 11:10:13] ...trap/engines/engine.py.230: loss: 0.14022 [I 2022-10-27 11:10:13] ...optimizers/trijoint.py.151: total_norm: 23.046883 [I 2022-10-27 11:10:18] ...trap/engines/engine.py.224: train: epoch 64 | batch 210/2304 [I 2022-10-27 11:10:18] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:25 [I 2022-10-27 11:10:18] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00253 [I 2022-10-27 11:10:18] ...trap/engines/engine.py.230: loss: 0.14823 [I 2022-10-27 11:10:18] ...optimizers/trijoint.py.151: total_norm: 28.189631 [I 2022-10-27 11:10:24] ...trap/engines/engine.py.224: train: epoch 64 | batch 220/2304 [I 2022-10-27 11:10:24] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:20 [I 2022-10-27 11:10:24] ...trap/engines/engine.py.229: process: 0.54613 | load: 0.00247 [I 2022-10-27 11:10:24] ...trap/engines/engine.py.230: loss: 0.20681 [I 2022-10-27 11:10:24] ...optimizers/trijoint.py.151: total_norm: 20.817375 [I 2022-10-27 11:10:29] ...trap/engines/engine.py.224: train: epoch 64 | batch 230/2304 [I 2022-10-27 11:10:29] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:12 [I 2022-10-27 11:10:29] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00247 [I 2022-10-27 11:10:29] ...trap/engines/engine.py.230: loss: 0.19840 [I 2022-10-27 11:10:29] ...optimizers/trijoint.py.151: total_norm: 13.112999 [I 2022-10-27 11:10:35] ...trap/engines/engine.py.224: train: epoch 64 | batch 240/2304 [I 2022-10-27 11:10:35] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:04 [I 2022-10-27 11:10:35] ...trap/engines/engine.py.229: process: 0.55028 | load: 0.00248 [I 2022-10-27 11:10:35] ...trap/engines/engine.py.230: loss: 0.14307 [I 2022-10-27 11:10:35] ...optimizers/trijoint.py.151: total_norm: 32.013538 [I 2022-10-27 11:10:41] ...trap/engines/engine.py.224: train: epoch 64 | batch 250/2304 [I 2022-10-27 11:10:41] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:06 [I 2022-10-27 11:10:41] ...trap/engines/engine.py.229: process: 0.54760 | load: 0.00248 [I 2022-10-27 11:10:41] ...trap/engines/engine.py.230: loss: 0.16179 [I 2022-10-27 11:10:41] ...optimizers/trijoint.py.151: total_norm: 20.496595 [I 2022-10-27 11:10:46] ...trap/engines/engine.py.224: train: epoch 64 | batch 260/2304 [I 2022-10-27 11:10:46] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:59 [I 2022-10-27 11:10:46] ...trap/engines/engine.py.229: process: 0.54843 | load: 0.00270 [I 2022-10-27 11:10:46] ...trap/engines/engine.py.230: loss: 0.10485 [I 2022-10-27 11:10:46] ...optimizers/trijoint.py.151: total_norm: 7.585706 [I 2022-10-27 11:10:52] ...trap/engines/engine.py.224: train: epoch 64 | batch 270/2304 [I 2022-10-27 11:10:52] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:58 [I 2022-10-27 11:10:52] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00268 [I 2022-10-27 11:10:52] ...trap/engines/engine.py.230: loss: 0.14593 [I 2022-10-27 11:10:52] ...optimizers/trijoint.py.151: total_norm: 22.999878 [I 2022-10-27 11:10:57] ...trap/engines/engine.py.224: train: epoch 64 | batch 280/2304 [I 2022-10-27 11:10:57] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:42 [I 2022-10-27 11:10:57] ...trap/engines/engine.py.229: process: 0.55986 | load: 0.00252 [I 2022-10-27 11:10:57] ...trap/engines/engine.py.230: loss: 0.25176 [I 2022-10-27 11:10:58] ...optimizers/trijoint.py.151: total_norm: 15.848852 [I 2022-10-27 11:11:03] ...trap/engines/engine.py.224: train: epoch 64 | batch 290/2304 [I 2022-10-27 11:11:03] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:51 [I 2022-10-27 11:11:03] ...trap/engines/engine.py.229: process: 0.56432 | load: 0.00254 [I 2022-10-27 11:11:03] ...trap/engines/engine.py.230: loss: 0.21478 [I 2022-10-27 11:11:03] ...optimizers/trijoint.py.151: total_norm: 32.937012 [I 2022-10-27 11:11:09] ...trap/engines/engine.py.224: train: epoch 64 | batch 300/2304 [I 2022-10-27 11:11:09] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:34 [I 2022-10-27 11:11:09] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00245 [I 2022-10-27 11:11:09] ...trap/engines/engine.py.230: loss: 0.13480 [I 2022-10-27 11:11:09] ...optimizers/trijoint.py.151: total_norm: 12.811136 [I 2022-10-27 11:11:14] ...trap/engines/engine.py.224: train: epoch 64 | batch 310/2304 [I 2022-10-27 11:11:14] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:27 [I 2022-10-27 11:11:14] ...trap/engines/engine.py.229: process: 0.55126 | load: 0.00279 [I 2022-10-27 11:11:14] ...trap/engines/engine.py.230: loss: 0.14207 [I 2022-10-27 11:11:14] ...optimizers/trijoint.py.151: total_norm: 10.578115 [I 2022-10-27 11:11:20] ...trap/engines/engine.py.224: train: epoch 64 | batch 320/2304 [I 2022-10-27 11:11:20] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:18 [I 2022-10-27 11:11:20] ...trap/engines/engine.py.229: process: 0.54782 | load: 0.00266 [I 2022-10-27 11:11:20] ...trap/engines/engine.py.230: loss: 0.16760 [I 2022-10-27 11:11:20] ...optimizers/trijoint.py.151: total_norm: 15.657855 [I 2022-10-27 11:11:25] ...trap/engines/engine.py.224: train: epoch 64 | batch 330/2304 [I 2022-10-27 11:11:25] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:21 [I 2022-10-27 11:11:25] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00256 [I 2022-10-27 11:11:25] ...trap/engines/engine.py.230: loss: 0.14715 [I 2022-10-27 11:11:25] ...optimizers/trijoint.py.151: total_norm: 18.903477 [I 2022-10-27 11:11:31] ...trap/engines/engine.py.224: train: epoch 64 | batch 340/2304 [I 2022-10-27 11:11:31] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:04 [I 2022-10-27 11:11:31] ...trap/engines/engine.py.229: process: 0.54562 | load: 0.00250 [I 2022-10-27 11:11:31] ...trap/engines/engine.py.230: loss: 0.16760 [I 2022-10-27 11:11:31] ...optimizers/trijoint.py.151: total_norm: 23.663137 [I 2022-10-27 11:11:36] ...trap/engines/engine.py.224: train: epoch 64 | batch 350/2304 [I 2022-10-27 11:11:36] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:06 [I 2022-10-27 11:11:36] ...trap/engines/engine.py.229: process: 0.56411 | load: 0.00246 [I 2022-10-27 11:11:36] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-27 11:11:36] ...optimizers/trijoint.py.151: total_norm: 22.952719 [I 2022-10-27 11:11:42] ...trap/engines/engine.py.224: train: epoch 64 | batch 360/2304 [I 2022-10-27 11:11:42] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:07 [I 2022-10-27 11:11:42] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00247 [I 2022-10-27 11:11:42] ...trap/engines/engine.py.230: loss: 0.11753 [I 2022-10-27 11:11:42] ...optimizers/trijoint.py.151: total_norm: 9.016648 [I 2022-10-27 11:11:48] ...trap/engines/engine.py.224: train: epoch 64 | batch 370/2304 [I 2022-10-27 11:11:48] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:54 [I 2022-10-27 11:11:48] ...trap/engines/engine.py.229: process: 0.55737 | load: 0.00258 [I 2022-10-27 11:11:48] ...trap/engines/engine.py.230: loss: 0.15312 [I 2022-10-27 11:11:48] ...optimizers/trijoint.py.151: total_norm: 14.018300 [I 2022-10-27 11:11:53] ...trap/engines/engine.py.224: train: epoch 64 | batch 380/2304 [I 2022-10-27 11:11:53] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:59 [I 2022-10-27 11:11:53] ...trap/engines/engine.py.229: process: 0.56770 | load: 0.00252 [I 2022-10-27 11:11:53] ...trap/engines/engine.py.230: loss: 0.12118 [I 2022-10-27 11:11:53] ...optimizers/trijoint.py.151: total_norm: 13.333916 [I 2022-10-27 11:11:59] ...trap/engines/engine.py.224: train: epoch 64 | batch 390/2304 [I 2022-10-27 11:11:59] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:55 [I 2022-10-27 11:11:59] ...trap/engines/engine.py.229: process: 0.56517 | load: 0.00251 [I 2022-10-27 11:11:59] ...trap/engines/engine.py.230: loss: 0.16880 [I 2022-10-27 11:11:59] ...optimizers/trijoint.py.151: total_norm: 23.305634 [I 2022-10-27 11:12:05] ...trap/engines/engine.py.224: train: epoch 64 | batch 400/2304 [I 2022-10-27 11:12:05] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:48 [I 2022-10-27 11:12:05] ...trap/engines/engine.py.229: process: 0.55676 | load: 0.00256 [I 2022-10-27 11:12:05] ...trap/engines/engine.py.230: loss: 0.21489 [I 2022-10-27 11:12:05] ...optimizers/trijoint.py.151: total_norm: 32.939106 [I 2022-10-27 11:12:10] ...trap/engines/engine.py.224: train: epoch 64 | batch 410/2304 [I 2022-10-27 11:12:10] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:35 [I 2022-10-27 11:12:10] ...trap/engines/engine.py.229: process: 0.54779 | load: 0.00247 [I 2022-10-27 11:12:10] ...trap/engines/engine.py.230: loss: 0.13766 [I 2022-10-27 11:12:10] ...optimizers/trijoint.py.151: total_norm: 11.919721 [I 2022-10-27 11:12:16] ...trap/engines/engine.py.224: train: epoch 64 | batch 420/2304 [I 2022-10-27 11:12:16] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:32 [I 2022-10-27 11:12:16] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00244 [I 2022-10-27 11:12:16] ...trap/engines/engine.py.230: loss: 0.08384 [I 2022-10-27 11:12:16] ...optimizers/trijoint.py.151: total_norm: 11.468128 [I 2022-10-27 11:12:21] ...trap/engines/engine.py.224: train: epoch 64 | batch 430/2304 [I 2022-10-27 11:12:21] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:19 [I 2022-10-27 11:12:21] ...trap/engines/engine.py.229: process: 0.55044 | load: 0.00242 [I 2022-10-27 11:12:21] ...trap/engines/engine.py.230: loss: 0.21102 [I 2022-10-27 11:12:21] ...optimizers/trijoint.py.151: total_norm: 22.794710 [I 2022-10-27 11:12:27] ...trap/engines/engine.py.224: train: epoch 64 | batch 440/2304 [I 2022-10-27 11:12:27] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:16 [I 2022-10-27 11:12:27] ...trap/engines/engine.py.229: process: 0.54838 | load: 0.00240 [I 2022-10-27 11:12:27] ...trap/engines/engine.py.230: loss: 0.17060 [I 2022-10-27 11:12:27] ...optimizers/trijoint.py.151: total_norm: 16.918968 [I 2022-10-27 11:12:32] ...trap/engines/engine.py.224: train: epoch 64 | batch 450/2304 [I 2022-10-27 11:12:32] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:11 [I 2022-10-27 11:12:32] ...trap/engines/engine.py.229: process: 0.55208 | load: 0.00249 [I 2022-10-27 11:12:32] ...trap/engines/engine.py.230: loss: 0.11783 [I 2022-10-27 11:12:32] ...optimizers/trijoint.py.151: total_norm: 16.286648 [I 2022-10-27 11:12:38] ...trap/engines/engine.py.224: train: epoch 64 | batch 460/2304 [I 2022-10-27 11:12:38] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:07 [I 2022-10-27 11:12:38] ...trap/engines/engine.py.229: process: 0.56162 | load: 0.00241 [I 2022-10-27 11:12:38] ...trap/engines/engine.py.230: loss: 0.15982 [I 2022-10-27 11:12:38] ...optimizers/trijoint.py.151: total_norm: 18.168020 [I 2022-10-27 11:12:44] ...trap/engines/engine.py.224: train: epoch 64 | batch 470/2304 [I 2022-10-27 11:12:44] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:02 [I 2022-10-27 11:12:44] ...trap/engines/engine.py.229: process: 0.54892 | load: 0.00248 [I 2022-10-27 11:12:44] ...trap/engines/engine.py.230: loss: 0.16567 [I 2022-10-27 11:12:44] ...optimizers/trijoint.py.151: total_norm: 17.677622 [I 2022-10-27 11:12:49] ...trap/engines/engine.py.224: train: epoch 64 | batch 480/2304 [I 2022-10-27 11:12:49] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:52 [I 2022-10-27 11:12:49] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00252 [I 2022-10-27 11:12:49] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-27 11:12:49] ...optimizers/trijoint.py.151: total_norm: 17.480921 [I 2022-10-27 11:12:55] ...trap/engines/engine.py.224: train: epoch 64 | batch 490/2304 [I 2022-10-27 11:12:55] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:52 [I 2022-10-27 11:12:55] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00250 [I 2022-10-27 11:12:55] ...trap/engines/engine.py.230: loss: 0.23517 [I 2022-10-27 11:12:55] ...optimizers/trijoint.py.151: total_norm: 16.268923 [I 2022-10-27 11:13:00] ...trap/engines/engine.py.224: train: epoch 64 | batch 500/2304 [I 2022-10-27 11:13:00] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:46 [I 2022-10-27 11:13:00] ...trap/engines/engine.py.229: process: 0.56661 | load: 0.00246 [I 2022-10-27 11:13:00] ...trap/engines/engine.py.230: loss: 0.22560 [I 2022-10-27 11:13:00] ...optimizers/trijoint.py.151: total_norm: 20.999622 [I 2022-10-27 11:13:06] ...trap/engines/engine.py.224: train: epoch 64 | batch 510/2304 [I 2022-10-27 11:13:06] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:48 [I 2022-10-27 11:13:06] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00243 [I 2022-10-27 11:13:06] ...trap/engines/engine.py.230: loss: 0.09473 [I 2022-10-27 11:13:06] ...optimizers/trijoint.py.151: total_norm: 9.597382 [I 2022-10-27 11:13:12] ...trap/engines/engine.py.224: train: epoch 64 | batch 520/2304 [I 2022-10-27 11:13:12] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:31 [I 2022-10-27 11:13:12] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00240 [I 2022-10-27 11:13:12] ...trap/engines/engine.py.230: loss: 0.13517 [I 2022-10-27 11:13:12] ...optimizers/trijoint.py.151: total_norm: 54.945171 [I 2022-10-27 11:13:17] ...trap/engines/engine.py.224: train: epoch 64 | batch 530/2304 [I 2022-10-27 11:13:17] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:28 [I 2022-10-27 11:13:17] ...trap/engines/engine.py.229: process: 0.55832 | load: 0.00241 [I 2022-10-27 11:13:17] ...trap/engines/engine.py.230: loss: 0.18724 [I 2022-10-27 11:13:17] ...optimizers/trijoint.py.151: total_norm: 20.720425 [I 2022-10-27 11:13:23] ...trap/engines/engine.py.224: train: epoch 64 | batch 540/2304 [I 2022-10-27 11:13:23] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:21 [I 2022-10-27 11:13:23] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00243 [I 2022-10-27 11:13:23] ...trap/engines/engine.py.230: loss: 0.12946 [I 2022-10-27 11:13:23] ...optimizers/trijoint.py.151: total_norm: 16.034521 [I 2022-10-27 11:13:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 550/2304 [I 2022-10-27 11:13:28] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:17 [I 2022-10-27 11:13:28] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00240 [I 2022-10-27 11:13:28] ...trap/engines/engine.py.230: loss: 0.18596 [I 2022-10-27 11:13:28] ...optimizers/trijoint.py.151: total_norm: 15.281352 [I 2022-10-27 11:13:34] ...trap/engines/engine.py.224: train: epoch 64 | batch 560/2304 [I 2022-10-27 11:13:34] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:17 [I 2022-10-27 11:13:34] ...trap/engines/engine.py.229: process: 0.56564 | load: 0.00242 [I 2022-10-27 11:13:34] ...trap/engines/engine.py.230: loss: 0.16860 [I 2022-10-27 11:13:34] ...optimizers/trijoint.py.151: total_norm: 15.787852 [I 2022-10-27 11:13:39] ...trap/engines/engine.py.224: train: epoch 64 | batch 570/2304 [I 2022-10-27 11:13:39] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:09 [I 2022-10-27 11:13:39] ...trap/engines/engine.py.229: process: 0.57358 | load: 0.00241 [I 2022-10-27 11:13:39] ...trap/engines/engine.py.230: loss: 0.15674 [I 2022-10-27 11:13:39] ...optimizers/trijoint.py.151: total_norm: 40.845428 [I 2022-10-27 11:13:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 580/2304 [I 2022-10-27 11:13:45] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:15:54 [I 2022-10-27 11:13:45] ...trap/engines/engine.py.229: process: 0.54969 | load: 0.00254 [I 2022-10-27 11:13:45] ...trap/engines/engine.py.230: loss: 0.09878 [I 2022-10-27 11:13:45] ...optimizers/trijoint.py.151: total_norm: 13.891503 [I 2022-10-27 11:13:51] ...trap/engines/engine.py.224: train: epoch 64 | batch 590/2304 [I 2022-10-27 11:13:51] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:16:02 [I 2022-10-27 11:13:51] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00247 [I 2022-10-27 11:13:51] ...trap/engines/engine.py.230: loss: 0.14205 [I 2022-10-27 11:13:51] ...optimizers/trijoint.py.151: total_norm: 14.275164 [I 2022-10-27 11:13:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 600/2304 [I 2022-10-27 11:13:56] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:47 [I 2022-10-27 11:13:56] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00244 [I 2022-10-27 11:13:56] ...trap/engines/engine.py.230: loss: 0.15043 [I 2022-10-27 11:13:56] ...optimizers/trijoint.py.151: total_norm: 19.747911 [I 2022-10-27 11:14:02] ...trap/engines/engine.py.224: train: epoch 64 | batch 610/2304 [I 2022-10-27 11:14:02] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:38 [I 2022-10-27 11:14:02] ...trap/engines/engine.py.229: process: 0.55043 | load: 0.00242 [I 2022-10-27 11:14:02] ...trap/engines/engine.py.230: loss: 0.15973 [I 2022-10-27 11:14:02] ...optimizers/trijoint.py.151: total_norm: 29.589314 [I 2022-10-27 11:14:07] ...trap/engines/engine.py.224: train: epoch 64 | batch 620/2304 [I 2022-10-27 11:14:07] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:39 [I 2022-10-27 11:14:07] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00249 [I 2022-10-27 11:14:07] ...trap/engines/engine.py.230: loss: 0.11509 [I 2022-10-27 11:14:07] ...optimizers/trijoint.py.151: total_norm: 14.741303 [I 2022-10-27 11:14:13] ...trap/engines/engine.py.224: train: epoch 64 | batch 630/2304 [I 2022-10-27 11:14:13] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:33 [I 2022-10-27 11:14:13] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00241 [I 2022-10-27 11:14:13] ...trap/engines/engine.py.230: loss: 0.20466 [I 2022-10-27 11:14:13] ...optimizers/trijoint.py.151: total_norm: 19.185072 [I 2022-10-27 11:14:19] ...trap/engines/engine.py.224: train: epoch 64 | batch 640/2304 [I 2022-10-27 11:14:19] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:35 [I 2022-10-27 11:14:19] ...trap/engines/engine.py.229: process: 0.55609 | load: 0.00243 [I 2022-10-27 11:14:19] ...trap/engines/engine.py.230: loss: 0.17866 [I 2022-10-27 11:14:19] ...optimizers/trijoint.py.151: total_norm: 17.056101 [I 2022-10-27 11:14:24] ...trap/engines/engine.py.224: train: epoch 64 | batch 650/2304 [I 2022-10-27 11:14:24] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:25 [I 2022-10-27 11:14:24] ...trap/engines/engine.py.229: process: 0.57044 | load: 0.00246 [I 2022-10-27 11:14:24] ...trap/engines/engine.py.230: loss: 0.15643 [I 2022-10-27 11:14:24] ...optimizers/trijoint.py.151: total_norm: 14.431862 [I 2022-10-27 11:14:30] ...trap/engines/engine.py.224: train: epoch 64 | batch 660/2304 [I 2022-10-27 11:14:30] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:19 [I 2022-10-27 11:14:30] ...trap/engines/engine.py.229: process: 0.56127 | load: 0.00240 [I 2022-10-27 11:14:30] ...trap/engines/engine.py.230: loss: 0.12659 [I 2022-10-27 11:14:30] ...optimizers/trijoint.py.151: total_norm: 9.949201 [I 2022-10-27 11:14:35] ...trap/engines/engine.py.224: train: epoch 64 | batch 670/2304 [I 2022-10-27 11:14:35] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:15 [I 2022-10-27 11:14:35] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00245 [I 2022-10-27 11:14:35] ...trap/engines/engine.py.230: loss: 0.13892 [I 2022-10-27 11:14:35] ...optimizers/trijoint.py.151: total_norm: 13.482187 [I 2022-10-27 11:14:41] ...trap/engines/engine.py.224: train: epoch 64 | batch 680/2304 [I 2022-10-27 11:14:41] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:02 [I 2022-10-27 11:14:41] ...trap/engines/engine.py.229: process: 0.55127 | load: 0.00241 [I 2022-10-27 11:14:41] ...trap/engines/engine.py.230: loss: 0.17199 [I 2022-10-27 11:14:41] ...optimizers/trijoint.py.151: total_norm: 13.485291 [I 2022-10-27 11:14:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 690/2304 [I 2022-10-27 11:14:47] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:57 [I 2022-10-27 11:14:47] ...trap/engines/engine.py.229: process: 0.55246 | load: 0.00242 [I 2022-10-27 11:14:47] ...trap/engines/engine.py.230: loss: 0.16851 [I 2022-10-27 11:14:47] ...optimizers/trijoint.py.151: total_norm: 17.082832 [I 2022-10-27 11:14:52] ...trap/engines/engine.py.224: train: epoch 64 | batch 700/2304 [I 2022-10-27 11:14:52] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:45 [I 2022-10-27 11:14:52] ...trap/engines/engine.py.229: process: 0.53562 | load: 0.00242 [I 2022-10-27 11:14:52] ...trap/engines/engine.py.230: loss: 0.15358 [I 2022-10-27 11:14:52] ...optimizers/trijoint.py.151: total_norm: 10.332044 [I 2022-10-27 11:14:58] ...trap/engines/engine.py.224: train: epoch 64 | batch 710/2304 [I 2022-10-27 11:14:58] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:44 [I 2022-10-27 11:14:58] ...trap/engines/engine.py.229: process: 0.55184 | load: 0.00242 [I 2022-10-27 11:14:58] ...trap/engines/engine.py.230: loss: 0.14575 [I 2022-10-27 11:14:58] ...optimizers/trijoint.py.151: total_norm: 11.506688 [I 2022-10-27 11:15:03] ...trap/engines/engine.py.224: train: epoch 64 | batch 720/2304 [I 2022-10-27 11:15:03] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:42 [I 2022-10-27 11:15:03] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00239 [I 2022-10-27 11:15:03] ...trap/engines/engine.py.230: loss: 0.17332 [I 2022-10-27 11:15:03] ...optimizers/trijoint.py.151: total_norm: 9.907735 [I 2022-10-27 11:15:09] ...trap/engines/engine.py.224: train: epoch 64 | batch 730/2304 [I 2022-10-27 11:15:09] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:41 [I 2022-10-27 11:15:09] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00243 [I 2022-10-27 11:15:09] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-27 11:15:09] ...optimizers/trijoint.py.151: total_norm: 11.805374 [I 2022-10-27 11:15:15] ...trap/engines/engine.py.224: train: epoch 64 | batch 740/2304 [I 2022-10-27 11:15:15] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:41 [I 2022-10-27 11:15:15] ...trap/engines/engine.py.229: process: 0.58286 | load: 0.00249 [I 2022-10-27 11:15:15] ...trap/engines/engine.py.230: loss: 0.14069 [I 2022-10-27 11:15:15] ...optimizers/trijoint.py.151: total_norm: 17.651985 [I 2022-10-27 11:15:20] ...trap/engines/engine.py.224: train: epoch 64 | batch 750/2304 [I 2022-10-27 11:15:20] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:31 [I 2022-10-27 11:15:20] ...trap/engines/engine.py.229: process: 0.56880 | load: 0.00253 [I 2022-10-27 11:15:20] ...trap/engines/engine.py.230: loss: 0.20271 [I 2022-10-27 11:15:20] ...optimizers/trijoint.py.151: total_norm: 24.051544 [I 2022-10-27 11:15:26] ...trap/engines/engine.py.224: train: epoch 64 | batch 760/2304 [I 2022-10-27 11:15:26] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:23 [I 2022-10-27 11:15:26] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00248 [I 2022-10-27 11:15:26] ...trap/engines/engine.py.230: loss: 0.19421 [I 2022-10-27 11:15:26] ...optimizers/trijoint.py.151: total_norm: 46.940552 [I 2022-10-27 11:15:31] ...trap/engines/engine.py.224: train: epoch 64 | batch 770/2304 [I 2022-10-27 11:15:31] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:19 [I 2022-10-27 11:15:31] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00247 [I 2022-10-27 11:15:31] ...trap/engines/engine.py.230: loss: 0.12711 [I 2022-10-27 11:15:31] ...optimizers/trijoint.py.151: total_norm: 15.014000 [I 2022-10-27 11:15:37] ...trap/engines/engine.py.224: train: epoch 64 | batch 780/2304 [I 2022-10-27 11:15:37] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:14 [I 2022-10-27 11:15:37] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00248 [I 2022-10-27 11:15:37] ...trap/engines/engine.py.230: loss: 0.14680 [I 2022-10-27 11:15:37] ...optimizers/trijoint.py.151: total_norm: 26.497168 [I 2022-10-27 11:15:43] ...trap/engines/engine.py.224: train: epoch 64 | batch 790/2304 [I 2022-10-27 11:15:43] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:06 [I 2022-10-27 11:15:43] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00251 [I 2022-10-27 11:15:43] ...trap/engines/engine.py.230: loss: 0.14454 [I 2022-10-27 11:15:43] ...optimizers/trijoint.py.151: total_norm: 21.426010 [I 2022-10-27 11:15:48] ...trap/engines/engine.py.224: train: epoch 64 | batch 800/2304 [I 2022-10-27 11:15:48] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:06 [I 2022-10-27 11:15:48] ...trap/engines/engine.py.229: process: 0.56513 | load: 0.00246 [I 2022-10-27 11:15:48] ...trap/engines/engine.py.230: loss: 0.13817 [I 2022-10-27 11:15:48] ...optimizers/trijoint.py.151: total_norm: 21.909941 [I 2022-10-27 11:15:54] ...trap/engines/engine.py.224: train: epoch 64 | batch 810/2304 [I 2022-10-27 11:15:54] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:58 [I 2022-10-27 11:15:54] ...trap/engines/engine.py.229: process: 0.56451 | load: 0.00242 [I 2022-10-27 11:15:54] ...trap/engines/engine.py.230: loss: 0.18733 [I 2022-10-27 11:15:54] ...optimizers/trijoint.py.151: total_norm: 22.515347 [I 2022-10-27 11:15:59] ...trap/engines/engine.py.224: train: epoch 64 | batch 820/2304 [I 2022-10-27 11:15:59] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:47 [I 2022-10-27 11:15:59] ...trap/engines/engine.py.229: process: 0.54917 | load: 0.00243 [I 2022-10-27 11:15:59] ...trap/engines/engine.py.230: loss: 0.20325 [I 2022-10-27 11:15:59] ...optimizers/trijoint.py.151: total_norm: 23.565712 [I 2022-10-27 11:16:05] ...trap/engines/engine.py.224: train: epoch 64 | batch 830/2304 [I 2022-10-27 11:16:05] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:55 [I 2022-10-27 11:16:05] ...trap/engines/engine.py.229: process: 0.56234 | load: 0.00250 [I 2022-10-27 11:16:05] ...trap/engines/engine.py.230: loss: 0.16834 [I 2022-10-27 11:16:05] ...optimizers/trijoint.py.151: total_norm: 21.687012 [I 2022-10-27 11:16:11] ...trap/engines/engine.py.224: train: epoch 64 | batch 840/2304 [I 2022-10-27 11:16:11] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:39 [I 2022-10-27 11:16:11] ...trap/engines/engine.py.229: process: 0.55075 | load: 0.00249 [I 2022-10-27 11:16:11] ...trap/engines/engine.py.230: loss: 0.11813 [I 2022-10-27 11:16:11] ...optimizers/trijoint.py.151: total_norm: 6.603394 [I 2022-10-27 11:16:16] ...trap/engines/engine.py.224: train: epoch 64 | batch 850/2304 [I 2022-10-27 11:16:16] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:40 [I 2022-10-27 11:16:16] ...trap/engines/engine.py.229: process: 0.56647 | load: 0.00258 [I 2022-10-27 11:16:16] ...trap/engines/engine.py.230: loss: 0.11696 [I 2022-10-27 11:16:16] ...optimizers/trijoint.py.151: total_norm: 13.880082 [I 2022-10-27 11:16:22] ...trap/engines/engine.py.224: train: epoch 64 | batch 860/2304 [I 2022-10-27 11:16:22] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:24 [I 2022-10-27 11:16:22] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00249 [I 2022-10-27 11:16:22] ...trap/engines/engine.py.230: loss: 0.16136 [I 2022-10-27 11:16:22] ...optimizers/trijoint.py.151: total_norm: 11.086866 [I 2022-10-27 11:16:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 870/2304 [I 2022-10-27 11:16:28] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:14 [I 2022-10-27 11:16:28] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00246 [I 2022-10-27 11:16:28] ...trap/engines/engine.py.230: loss: 0.17799 [I 2022-10-27 11:16:28] ...optimizers/trijoint.py.151: total_norm: 22.641947 [I 2022-10-27 11:16:33] ...trap/engines/engine.py.224: train: epoch 64 | batch 880/2304 [I 2022-10-27 11:16:33] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:13 [I 2022-10-27 11:16:33] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00243 [I 2022-10-27 11:16:33] ...trap/engines/engine.py.230: loss: 0.18130 [I 2022-10-27 11:16:33] ...optimizers/trijoint.py.151: total_norm: 19.160736 [I 2022-10-27 11:16:39] ...trap/engines/engine.py.224: train: epoch 64 | batch 890/2304 [I 2022-10-27 11:16:39] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:05 [I 2022-10-27 11:16:39] ...trap/engines/engine.py.229: process: 0.54524 | load: 0.00262 [I 2022-10-27 11:16:39] ...trap/engines/engine.py.230: loss: 0.20270 [I 2022-10-27 11:16:39] ...optimizers/trijoint.py.151: total_norm: 21.619493 [I 2022-10-27 11:16:44] ...trap/engines/engine.py.224: train: epoch 64 | batch 900/2304 [I 2022-10-27 11:16:44] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:03 [I 2022-10-27 11:16:44] ...trap/engines/engine.py.229: process: 0.56820 | load: 0.00244 [I 2022-10-27 11:16:44] ...trap/engines/engine.py.230: loss: 0.17571 [I 2022-10-27 11:16:44] ...optimizers/trijoint.py.151: total_norm: 17.119192 [I 2022-10-27 11:16:50] ...trap/engines/engine.py.224: train: epoch 64 | batch 910/2304 [I 2022-10-27 11:16:50] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:13:01 [I 2022-10-27 11:16:50] ...trap/engines/engine.py.229: process: 0.56665 | load: 0.00245 [I 2022-10-27 11:16:50] ...trap/engines/engine.py.230: loss: 0.23824 [I 2022-10-27 11:16:50] ...optimizers/trijoint.py.151: total_norm: 22.377441 [I 2022-10-27 11:16:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 920/2304 [I 2022-10-27 11:16:56] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:13:07 [I 2022-10-27 11:16:56] ...trap/engines/engine.py.229: process: 0.54663 | load: 0.00245 [I 2022-10-27 11:16:56] ...trap/engines/engine.py.230: loss: 0.13595 [I 2022-10-27 11:16:56] ...optimizers/trijoint.py.151: total_norm: 24.283602 [I 2022-10-27 11:17:01] ...trap/engines/engine.py.224: train: epoch 64 | batch 930/2304 [I 2022-10-27 11:17:01] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:49 [I 2022-10-27 11:17:01] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00243 [I 2022-10-27 11:17:01] ...trap/engines/engine.py.230: loss: 0.11860 [I 2022-10-27 11:17:01] ...optimizers/trijoint.py.151: total_norm: 9.487142 [I 2022-10-27 11:17:07] ...trap/engines/engine.py.224: train: epoch 64 | batch 940/2304 [I 2022-10-27 11:17:07] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:40 [I 2022-10-27 11:17:07] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00242 [I 2022-10-27 11:17:07] ...trap/engines/engine.py.230: loss: 0.15467 [I 2022-10-27 11:17:07] ...optimizers/trijoint.py.151: total_norm: 17.534517 [I 2022-10-27 11:17:12] ...trap/engines/engine.py.224: train: epoch 64 | batch 950/2304 [I 2022-10-27 11:17:12] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:36 [I 2022-10-27 11:17:12] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00242 [I 2022-10-27 11:17:12] ...trap/engines/engine.py.230: loss: 0.15069 [I 2022-10-27 11:17:12] ...optimizers/trijoint.py.151: total_norm: 16.413404 [I 2022-10-27 11:17:18] ...trap/engines/engine.py.224: train: epoch 64 | batch 960/2304 [I 2022-10-27 11:17:18] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:34 [I 2022-10-27 11:17:18] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00250 [I 2022-10-27 11:17:18] ...trap/engines/engine.py.230: loss: 0.11627 [I 2022-10-27 11:17:18] ...optimizers/trijoint.py.151: total_norm: 15.208076 [I 2022-10-27 11:17:24] ...trap/engines/engine.py.224: train: epoch 64 | batch 970/2304 [I 2022-10-27 11:17:24] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:19 [I 2022-10-27 11:17:24] ...trap/engines/engine.py.229: process: 0.56765 | load: 0.00245 [I 2022-10-27 11:17:24] ...trap/engines/engine.py.230: loss: 0.15838 [I 2022-10-27 11:17:24] ...optimizers/trijoint.py.151: total_norm: 19.147558 [I 2022-10-27 11:17:29] ...trap/engines/engine.py.224: train: epoch 64 | batch 980/2304 [I 2022-10-27 11:17:29] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:34 [I 2022-10-27 11:17:29] ...trap/engines/engine.py.229: process: 0.56532 | load: 0.00250 [I 2022-10-27 11:17:29] ...trap/engines/engine.py.230: loss: 0.12474 [I 2022-10-27 11:17:29] ...optimizers/trijoint.py.151: total_norm: 13.549568 [I 2022-10-27 11:17:35] ...trap/engines/engine.py.224: train: epoch 64 | batch 990/2304 [I 2022-10-27 11:17:35] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:17 [I 2022-10-27 11:17:35] ...trap/engines/engine.py.229: process: 0.57021 | load: 0.00251 [I 2022-10-27 11:17:35] ...trap/engines/engine.py.230: loss: 0.10548 [I 2022-10-27 11:17:35] ...optimizers/trijoint.py.151: total_norm: 9.146371 [I 2022-10-27 11:17:41] ...trap/engines/engine.py.224: train: epoch 64 | batch 1000/2304 [I 2022-10-27 11:17:41] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:09 [I 2022-10-27 11:17:41] ...trap/engines/engine.py.229: process: 0.56678 | load: 0.00242 [I 2022-10-27 11:17:41] ...trap/engines/engine.py.230: loss: 0.12978 [I 2022-10-27 11:17:41] ...optimizers/trijoint.py.151: total_norm: 16.083529 [I 2022-10-27 11:17:46] ...trap/engines/engine.py.224: train: epoch 64 | batch 1010/2304 [I 2022-10-27 11:17:46] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:00 [I 2022-10-27 11:17:46] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00242 [I 2022-10-27 11:17:46] ...trap/engines/engine.py.230: loss: 0.13570 [I 2022-10-27 11:17:46] ...optimizers/trijoint.py.151: total_norm: 13.201515 [I 2022-10-27 11:17:52] ...trap/engines/engine.py.224: train: epoch 64 | batch 1020/2304 [I 2022-10-27 11:17:52] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:48 [I 2022-10-27 11:17:52] ...trap/engines/engine.py.229: process: 0.54165 | load: 0.00245 [I 2022-10-27 11:17:52] ...trap/engines/engine.py.230: loss: 0.14326 [I 2022-10-27 11:17:52] ...optimizers/trijoint.py.151: total_norm: 18.636667 [I 2022-10-27 11:17:57] ...trap/engines/engine.py.224: train: epoch 64 | batch 1030/2304 [I 2022-10-27 11:17:57] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:52 [I 2022-10-27 11:17:57] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00253 [I 2022-10-27 11:17:57] ...trap/engines/engine.py.230: loss: 0.17016 [I 2022-10-27 11:17:57] ...optimizers/trijoint.py.151: total_norm: 34.053635 [I 2022-10-27 11:18:03] ...trap/engines/engine.py.224: train: epoch 64 | batch 1040/2304 [I 2022-10-27 11:18:03] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:40 [I 2022-10-27 11:18:03] ...trap/engines/engine.py.229: process: 0.55490 | load: 0.00248 [I 2022-10-27 11:18:03] ...trap/engines/engine.py.230: loss: 0.11398 [I 2022-10-27 11:18:03] ...optimizers/trijoint.py.151: total_norm: 7.104964 [I 2022-10-27 11:18:08] ...trap/engines/engine.py.224: train: epoch 64 | batch 1050/2304 [I 2022-10-27 11:18:08] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:42 [I 2022-10-27 11:18:08] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00243 [I 2022-10-27 11:18:08] ...trap/engines/engine.py.230: loss: 0.17733 [I 2022-10-27 11:18:08] ...optimizers/trijoint.py.151: total_norm: 20.942394 [I 2022-10-27 11:18:14] ...trap/engines/engine.py.224: train: epoch 64 | batch 1060/2304 [I 2022-10-27 11:18:14] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:36 [I 2022-10-27 11:18:14] ...trap/engines/engine.py.229: process: 0.56352 | load: 0.00244 [I 2022-10-27 11:18:14] ...trap/engines/engine.py.230: loss: 0.17512 [I 2022-10-27 11:18:14] ...optimizers/trijoint.py.151: total_norm: 24.486967 [I 2022-10-27 11:18:20] ...trap/engines/engine.py.224: train: epoch 64 | batch 1070/2304 [I 2022-10-27 11:18:20] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:34 [I 2022-10-27 11:18:20] ...trap/engines/engine.py.229: process: 0.56800 | load: 0.00244 [I 2022-10-27 11:18:20] ...trap/engines/engine.py.230: loss: 0.16338 [I 2022-10-27 11:18:20] ...optimizers/trijoint.py.151: total_norm: 12.455402 [I 2022-10-27 11:18:25] ...trap/engines/engine.py.224: train: epoch 64 | batch 1080/2304 [I 2022-10-27 11:18:25] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:23 [I 2022-10-27 11:18:25] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00244 [I 2022-10-27 11:18:25] ...trap/engines/engine.py.230: loss: 0.13556 [I 2022-10-27 11:18:25] ...optimizers/trijoint.py.151: total_norm: 10.273993 [I 2022-10-27 11:18:31] ...trap/engines/engine.py.224: train: epoch 64 | batch 1090/2304 [I 2022-10-27 11:18:31] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:19 [I 2022-10-27 11:18:31] ...trap/engines/engine.py.229: process: 0.56788 | load: 0.00242 [I 2022-10-27 11:18:31] ...trap/engines/engine.py.230: loss: 0.19344 [I 2022-10-27 11:18:31] ...optimizers/trijoint.py.151: total_norm: 66.867310 [I 2022-10-27 11:18:37] ...trap/engines/engine.py.224: train: epoch 64 | batch 1100/2304 [I 2022-10-27 11:18:37] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:15 [I 2022-10-27 11:18:37] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00241 [I 2022-10-27 11:18:37] ...trap/engines/engine.py.230: loss: 0.12423 [I 2022-10-27 11:18:37] ...optimizers/trijoint.py.151: total_norm: 29.249472 [I 2022-10-27 11:18:42] ...trap/engines/engine.py.224: train: epoch 64 | batch 1110/2304 [I 2022-10-27 11:18:42] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:09 [I 2022-10-27 11:18:42] ...trap/engines/engine.py.229: process: 0.56102 | load: 0.00242 [I 2022-10-27 11:18:42] ...trap/engines/engine.py.230: loss: 0.16711 [I 2022-10-27 11:18:42] ...optimizers/trijoint.py.151: total_norm: 16.841597 [I 2022-10-27 11:18:48] ...trap/engines/engine.py.224: train: epoch 64 | batch 1120/2304 [I 2022-10-27 11:18:48] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:01 [I 2022-10-27 11:18:48] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00247 [I 2022-10-27 11:18:48] ...trap/engines/engine.py.230: loss: 0.24235 [I 2022-10-27 11:18:48] ...optimizers/trijoint.py.151: total_norm: 16.480148 [I 2022-10-27 11:18:53] ...trap/engines/engine.py.224: train: epoch 64 | batch 1130/2304 [I 2022-10-27 11:18:53] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:55 [I 2022-10-27 11:18:53] ...trap/engines/engine.py.229: process: 0.56633 | load: 0.00241 [I 2022-10-27 11:18:53] ...trap/engines/engine.py.230: loss: 0.12520 [I 2022-10-27 11:18:53] ...optimizers/trijoint.py.151: total_norm: 15.065847 [I 2022-10-27 11:18:59] ...trap/engines/engine.py.224: train: epoch 64 | batch 1140/2304 [I 2022-10-27 11:18:59] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:48 [I 2022-10-27 11:18:59] ...trap/engines/engine.py.229: process: 0.55432 | load: 0.00243 [I 2022-10-27 11:18:59] ...trap/engines/engine.py.230: loss: 0.15088 [I 2022-10-27 11:18:59] ...optimizers/trijoint.py.151: total_norm: 27.870672 [I 2022-10-27 11:19:05] ...trap/engines/engine.py.224: train: epoch 64 | batch 1150/2304 [I 2022-10-27 11:19:05] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:45 [I 2022-10-27 11:19:05] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00251 [I 2022-10-27 11:19:05] ...trap/engines/engine.py.230: loss: 0.10680 [I 2022-10-27 11:19:05] ...optimizers/trijoint.py.151: total_norm: 14.188437 [I 2022-10-27 11:19:10] ...trap/engines/engine.py.224: train: epoch 64 | batch 1160/2304 [I 2022-10-27 11:19:10] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:39 [I 2022-10-27 11:19:10] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00247 [I 2022-10-27 11:19:10] ...trap/engines/engine.py.230: loss: 0.13550 [I 2022-10-27 11:19:10] ...optimizers/trijoint.py.151: total_norm: 34.045856 [I 2022-10-27 11:19:16] ...trap/engines/engine.py.224: train: epoch 64 | batch 1170/2304 [I 2022-10-27 11:19:16] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:29 [I 2022-10-27 11:19:16] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00249 [I 2022-10-27 11:19:16] ...trap/engines/engine.py.230: loss: 0.26122 [I 2022-10-27 11:19:16] ...optimizers/trijoint.py.151: total_norm: 30.403387 [I 2022-10-27 11:19:21] ...trap/engines/engine.py.224: train: epoch 64 | batch 1180/2304 [I 2022-10-27 11:19:21] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:26 [I 2022-10-27 11:19:21] ...trap/engines/engine.py.229: process: 0.56050 | load: 0.00246 [I 2022-10-27 11:19:21] ...trap/engines/engine.py.230: loss: 0.13548 [I 2022-10-27 11:19:21] ...optimizers/trijoint.py.151: total_norm: 11.427835 [I 2022-10-27 11:19:27] ...trap/engines/engine.py.224: train: epoch 64 | batch 1190/2304 [I 2022-10-27 11:19:27] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:18 [I 2022-10-27 11:19:27] ...trap/engines/engine.py.229: process: 0.57215 | load: 0.00252 [I 2022-10-27 11:19:27] ...trap/engines/engine.py.230: loss: 0.22807 [I 2022-10-27 11:19:27] ...optimizers/trijoint.py.151: total_norm: 11.141628 [I 2022-10-27 11:19:32] ...trap/engines/engine.py.224: train: epoch 64 | batch 1200/2304 [I 2022-10-27 11:19:32] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:13 [I 2022-10-27 11:19:32] ...trap/engines/engine.py.229: process: 0.54403 | load: 0.00243 [I 2022-10-27 11:19:32] ...trap/engines/engine.py.230: loss: 0.13560 [I 2022-10-27 11:19:32] ...optimizers/trijoint.py.151: total_norm: 14.948158 [I 2022-10-27 11:19:38] ...trap/engines/engine.py.224: train: epoch 64 | batch 1210/2304 [I 2022-10-27 11:19:38] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:16 [I 2022-10-27 11:19:38] ...trap/engines/engine.py.229: process: 0.56538 | load: 0.00243 [I 2022-10-27 11:19:38] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-27 11:19:38] ...optimizers/trijoint.py.151: total_norm: 9.299111 [I 2022-10-27 11:19:44] ...trap/engines/engine.py.224: train: epoch 64 | batch 1220/2304 [I 2022-10-27 11:19:44] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:06 [I 2022-10-27 11:19:44] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00243 [I 2022-10-27 11:19:44] ...trap/engines/engine.py.230: loss: 0.12656 [I 2022-10-27 11:19:44] ...optimizers/trijoint.py.151: total_norm: 15.593678 [I 2022-10-27 11:19:49] ...trap/engines/engine.py.224: train: epoch 64 | batch 1230/2304 [I 2022-10-27 11:19:49] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:09:58 [I 2022-10-27 11:19:49] ...trap/engines/engine.py.229: process: 0.54773 | load: 0.00246 [I 2022-10-27 11:19:49] ...trap/engines/engine.py.230: loss: 0.12524 [I 2022-10-27 11:19:49] ...optimizers/trijoint.py.151: total_norm: 19.195599 [I 2022-10-27 11:19:55] ...trap/engines/engine.py.224: train: epoch 64 | batch 1240/2304 [I 2022-10-27 11:19:55] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:53 [I 2022-10-27 11:19:55] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00247 [I 2022-10-27 11:19:55] ...trap/engines/engine.py.230: loss: 0.13867 [I 2022-10-27 11:19:55] ...optimizers/trijoint.py.151: total_norm: 9.540709 [I 2022-10-27 11:20:01] ...trap/engines/engine.py.224: train: epoch 64 | batch 1250/2304 [I 2022-10-27 11:20:01] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:56 [I 2022-10-27 11:20:01] ...trap/engines/engine.py.229: process: 0.59047 | load: 0.00248 [I 2022-10-27 11:20:01] ...trap/engines/engine.py.230: loss: 0.12348 [I 2022-10-27 11:20:01] ...optimizers/trijoint.py.151: total_norm: 17.798903 [I 2022-10-27 11:20:06] ...trap/engines/engine.py.224: train: epoch 64 | batch 1260/2304 [I 2022-10-27 11:20:06] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:42 [I 2022-10-27 11:20:06] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00246 [I 2022-10-27 11:20:06] ...trap/engines/engine.py.230: loss: 0.18257 [I 2022-10-27 11:20:06] ...optimizers/trijoint.py.151: total_norm: 19.373297 [I 2022-10-27 11:20:12] ...trap/engines/engine.py.224: train: epoch 64 | batch 1270/2304 [I 2022-10-27 11:20:12] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:34 [I 2022-10-27 11:20:12] ...trap/engines/engine.py.229: process: 0.54743 | load: 0.00252 [I 2022-10-27 11:20:12] ...trap/engines/engine.py.230: loss: 0.14833 [I 2022-10-27 11:20:12] ...optimizers/trijoint.py.151: total_norm: 26.820526 [I 2022-10-27 11:20:17] ...trap/engines/engine.py.224: train: epoch 64 | batch 1280/2304 [I 2022-10-27 11:20:17] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:28 [I 2022-10-27 11:20:17] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00248 [I 2022-10-27 11:20:17] ...trap/engines/engine.py.230: loss: 0.13526 [I 2022-10-27 11:20:17] ...optimizers/trijoint.py.151: total_norm: 20.743500 [I 2022-10-27 11:20:23] ...trap/engines/engine.py.224: train: epoch 64 | batch 1290/2304 [I 2022-10-27 11:20:23] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:24 [I 2022-10-27 11:20:23] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00244 [I 2022-10-27 11:20:23] ...trap/engines/engine.py.230: loss: 0.12851 [I 2022-10-27 11:20:23] ...optimizers/trijoint.py.151: total_norm: 17.904270 [I 2022-10-27 11:20:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 1300/2304 [I 2022-10-27 11:20:28] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:20 [I 2022-10-27 11:20:28] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00244 [I 2022-10-27 11:20:28] ...trap/engines/engine.py.230: loss: 0.21940 [I 2022-10-27 11:20:28] ...optimizers/trijoint.py.151: total_norm: 28.082323 [I 2022-10-27 11:20:34] ...trap/engines/engine.py.224: train: epoch 64 | batch 1310/2304 [I 2022-10-27 11:20:34] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:14 [I 2022-10-27 11:20:34] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00249 [I 2022-10-27 11:20:34] ...trap/engines/engine.py.230: loss: 0.16204 [I 2022-10-27 11:20:34] ...optimizers/trijoint.py.151: total_norm: 24.045216 [I 2022-10-27 11:20:40] ...trap/engines/engine.py.224: train: epoch 64 | batch 1320/2304 [I 2022-10-27 11:20:40] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:06 [I 2022-10-27 11:20:40] ...trap/engines/engine.py.229: process: 0.55494 | load: 0.00243 [I 2022-10-27 11:20:40] ...trap/engines/engine.py.230: loss: 0.22136 [I 2022-10-27 11:20:40] ...optimizers/trijoint.py.151: total_norm: 17.680866 [I 2022-10-27 11:20:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 1330/2304 [I 2022-10-27 11:20:45] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:06 [I 2022-10-27 11:20:45] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00244 [I 2022-10-27 11:20:45] ...trap/engines/engine.py.230: loss: 0.17525 [I 2022-10-27 11:20:45] ...optimizers/trijoint.py.151: total_norm: 46.587223 [I 2022-10-27 11:20:51] ...trap/engines/engine.py.224: train: epoch 64 | batch 1340/2304 [I 2022-10-27 11:20:51] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:57 [I 2022-10-27 11:20:51] ...trap/engines/engine.py.229: process: 0.54522 | load: 0.00244 [I 2022-10-27 11:20:51] ...trap/engines/engine.py.230: loss: 0.11211 [I 2022-10-27 11:20:51] ...optimizers/trijoint.py.151: total_norm: 10.162225 [I 2022-10-27 11:20:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 1350/2304 [I 2022-10-27 11:20:56] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:48 [I 2022-10-27 11:20:56] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00244 [I 2022-10-27 11:20:56] ...trap/engines/engine.py.230: loss: 0.14931 [I 2022-10-27 11:20:56] ...optimizers/trijoint.py.151: total_norm: 9.834496 [I 2022-10-27 11:21:02] ...trap/engines/engine.py.224: train: epoch 64 | batch 1360/2304 [I 2022-10-27 11:21:02] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:43 [I 2022-10-27 11:21:02] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00245 [I 2022-10-27 11:21:02] ...trap/engines/engine.py.230: loss: 0.14558 [I 2022-10-27 11:21:02] ...optimizers/trijoint.py.151: total_norm: 15.979081 [I 2022-10-27 11:21:08] ...trap/engines/engine.py.224: train: epoch 64 | batch 1370/2304 [I 2022-10-27 11:21:08] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:41 [I 2022-10-27 11:21:08] ...trap/engines/engine.py.229: process: 0.54766 | load: 0.00242 [I 2022-10-27 11:21:08] ...trap/engines/engine.py.230: loss: 0.14521 [I 2022-10-27 11:21:08] ...optimizers/trijoint.py.151: total_norm: 14.258536 [I 2022-10-27 11:21:32] ...trap/engines/engine.py.224: train: epoch 64 | batch 1380/2304 [I 2022-10-27 11:21:32] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:38:49 [I 2022-10-27 11:21:32] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00242 [I 2022-10-27 11:21:32] ...trap/engines/engine.py.230: loss: 0.11390 [I 2022-10-27 11:21:32] ...optimizers/trijoint.py.151: total_norm: 10.767131 [I 2022-10-27 11:21:38] ...trap/engines/engine.py.224: train: epoch 64 | batch 1390/2304 [I 2022-10-27 11:21:38] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:11:41 [I 2022-10-27 11:21:38] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00261 [I 2022-10-27 11:21:38] ...trap/engines/engine.py.230: loss: 0.21708 [I 2022-10-27 11:21:38] ...optimizers/trijoint.py.151: total_norm: 63.517380 [I 2022-10-27 11:21:44] ...trap/engines/engine.py.224: train: epoch 64 | batch 1400/2304 [I 2022-10-27 11:21:44] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:47 [I 2022-10-27 11:21:44] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00241 [I 2022-10-27 11:21:44] ...trap/engines/engine.py.230: loss: 0.13975 [I 2022-10-27 11:21:44] ...optimizers/trijoint.py.151: total_norm: 20.700165 [I 2022-10-27 11:21:49] ...trap/engines/engine.py.224: train: epoch 64 | batch 1410/2304 [I 2022-10-27 11:21:49] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:17 [I 2022-10-27 11:21:49] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00237 [I 2022-10-27 11:21:49] ...trap/engines/engine.py.230: loss: 0.17484 [I 2022-10-27 11:21:49] ...optimizers/trijoint.py.151: total_norm: 19.024422 [I 2022-10-27 11:21:55] ...trap/engines/engine.py.224: train: epoch 64 | batch 1420/2304 [I 2022-10-27 11:21:55] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:11 [I 2022-10-27 11:21:55] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00240 [I 2022-10-27 11:21:55] ...trap/engines/engine.py.230: loss: 0.17677 [I 2022-10-27 11:21:55] ...optimizers/trijoint.py.151: total_norm: 27.910572 [I 2022-10-27 11:22:00] ...trap/engines/engine.py.224: train: epoch 64 | batch 1430/2304 [I 2022-10-27 11:22:00] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:08:07 [I 2022-10-27 11:22:00] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00242 [I 2022-10-27 11:22:00] ...trap/engines/engine.py.230: loss: 0.12144 [I 2022-10-27 11:22:00] ...optimizers/trijoint.py.151: total_norm: 21.251839 [I 2022-10-27 11:22:06] ...trap/engines/engine.py.224: train: epoch 64 | batch 1440/2304 [I 2022-10-27 11:22:06] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:08:02 [I 2022-10-27 11:22:06] ...trap/engines/engine.py.229: process: 0.56473 | load: 0.00243 [I 2022-10-27 11:22:06] ...trap/engines/engine.py.230: loss: 0.17777 [I 2022-10-27 11:22:06] ...optimizers/trijoint.py.151: total_norm: 18.208799 [I 2022-10-27 11:22:11] ...trap/engines/engine.py.224: train: epoch 64 | batch 1450/2304 [I 2022-10-27 11:22:11] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:56 [I 2022-10-27 11:22:11] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00246 [I 2022-10-27 11:22:11] ...trap/engines/engine.py.230: loss: 0.14943 [I 2022-10-27 11:22:11] ...optimizers/trijoint.py.151: total_norm: 25.497498 [I 2022-10-27 11:22:17] ...trap/engines/engine.py.224: train: epoch 64 | batch 1460/2304 [I 2022-10-27 11:22:17] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:53 [I 2022-10-27 11:22:17] ...trap/engines/engine.py.229: process: 0.56366 | load: 0.00254 [I 2022-10-27 11:22:17] ...trap/engines/engine.py.230: loss: 0.12970 [I 2022-10-27 11:22:17] ...optimizers/trijoint.py.151: total_norm: 13.425588 [I 2022-10-27 11:22:23] ...trap/engines/engine.py.224: train: epoch 64 | batch 1470/2304 [I 2022-10-27 11:22:23] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:41 [I 2022-10-27 11:22:23] ...trap/engines/engine.py.229: process: 0.53898 | load: 0.00241 [I 2022-10-27 11:22:23] ...trap/engines/engine.py.230: loss: 0.12125 [I 2022-10-27 11:22:23] ...optimizers/trijoint.py.151: total_norm: 14.279016 [I 2022-10-27 11:22:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 1480/2304 [I 2022-10-27 11:22:28] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:41 [I 2022-10-27 11:22:28] ...trap/engines/engine.py.229: process: 0.54898 | load: 0.00242 [I 2022-10-27 11:22:28] ...trap/engines/engine.py.230: loss: 0.13178 [I 2022-10-27 11:22:28] ...optimizers/trijoint.py.151: total_norm: 14.802682 [I 2022-10-27 11:22:34] ...trap/engines/engine.py.224: train: epoch 64 | batch 1490/2304 [I 2022-10-27 11:22:34] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:31 [I 2022-10-27 11:22:34] ...trap/engines/engine.py.229: process: 0.55221 | load: 0.00252 [I 2022-10-27 11:22:34] ...trap/engines/engine.py.230: loss: 0.10794 [I 2022-10-27 11:22:34] ...optimizers/trijoint.py.151: total_norm: 26.984602 [I 2022-10-27 11:22:39] ...trap/engines/engine.py.224: train: epoch 64 | batch 1500/2304 [I 2022-10-27 11:22:39] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:26 [I 2022-10-27 11:22:39] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00245 [I 2022-10-27 11:22:39] ...trap/engines/engine.py.230: loss: 0.17271 [I 2022-10-27 11:22:39] ...optimizers/trijoint.py.151: total_norm: 17.377285 [I 2022-10-27 11:22:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 1510/2304 [I 2022-10-27 11:22:45] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:21 [I 2022-10-27 11:22:45] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00245 [I 2022-10-27 11:22:45] ...trap/engines/engine.py.230: loss: 0.13121 [I 2022-10-27 11:22:45] ...optimizers/trijoint.py.151: total_norm: 14.782671 [I 2022-10-27 11:22:50] ...trap/engines/engine.py.224: train: epoch 64 | batch 1520/2304 [I 2022-10-27 11:22:50] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:14 [I 2022-10-27 11:22:50] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00240 [I 2022-10-27 11:22:50] ...trap/engines/engine.py.230: loss: 0.17051 [I 2022-10-27 11:22:50] ...optimizers/trijoint.py.151: total_norm: 44.379768 [I 2022-10-27 11:22:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 1530/2304 [I 2022-10-27 11:22:56] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:07:11 [I 2022-10-27 11:22:56] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00243 [I 2022-10-27 11:22:56] ...trap/engines/engine.py.230: loss: 0.13720 [I 2022-10-27 11:22:56] ...optimizers/trijoint.py.151: total_norm: 17.508499 [I 2022-10-27 11:23:02] ...trap/engines/engine.py.224: train: epoch 64 | batch 1540/2304 [I 2022-10-27 11:23:02] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:07:07 [I 2022-10-27 11:23:02] ...trap/engines/engine.py.229: process: 0.56238 | load: 0.00243 [I 2022-10-27 11:23:02] ...trap/engines/engine.py.230: loss: 0.10931 [I 2022-10-27 11:23:02] ...optimizers/trijoint.py.151: total_norm: 13.603026 [I 2022-10-27 11:23:07] ...trap/engines/engine.py.224: train: epoch 64 | batch 1550/2304 [I 2022-10-27 11:23:07] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:07:00 [I 2022-10-27 11:23:07] ...trap/engines/engine.py.229: process: 0.55316 | load: 0.00243 [I 2022-10-27 11:23:07] ...trap/engines/engine.py.230: loss: 0.12182 [I 2022-10-27 11:23:07] ...optimizers/trijoint.py.151: total_norm: 13.452518 [I 2022-10-27 11:23:13] ...trap/engines/engine.py.224: train: epoch 64 | batch 1560/2304 [I 2022-10-27 11:23:13] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:57 [I 2022-10-27 11:23:13] ...trap/engines/engine.py.229: process: 0.56477 | load: 0.00241 [I 2022-10-27 11:23:13] ...trap/engines/engine.py.230: loss: 0.14829 [I 2022-10-27 11:23:13] ...optimizers/trijoint.py.151: total_norm: 29.857725 [I 2022-10-27 11:23:18] ...trap/engines/engine.py.224: train: epoch 64 | batch 1570/2304 [I 2022-10-27 11:23:18] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:46 [I 2022-10-27 11:23:18] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00245 [I 2022-10-27 11:23:18] ...trap/engines/engine.py.230: loss: 0.12757 [I 2022-10-27 11:23:18] ...optimizers/trijoint.py.151: total_norm: 18.370049 [I 2022-10-27 11:23:24] ...trap/engines/engine.py.224: train: epoch 64 | batch 1580/2304 [I 2022-10-27 11:23:24] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:40 [I 2022-10-27 11:23:24] ...trap/engines/engine.py.229: process: 0.53750 | load: 0.00241 [I 2022-10-27 11:23:24] ...trap/engines/engine.py.230: loss: 0.10648 [I 2022-10-27 11:23:24] ...optimizers/trijoint.py.151: total_norm: 10.964817 [I 2022-10-27 11:23:30] ...trap/engines/engine.py.224: train: epoch 64 | batch 1590/2304 [I 2022-10-27 11:23:30] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:37 [I 2022-10-27 11:23:30] ...trap/engines/engine.py.229: process: 0.56557 | load: 0.00242 [I 2022-10-27 11:23:30] ...trap/engines/engine.py.230: loss: 0.18503 [I 2022-10-27 11:23:30] ...optimizers/trijoint.py.151: total_norm: 17.257557 [I 2022-10-27 11:23:35] ...trap/engines/engine.py.224: train: epoch 64 | batch 1600/2304 [I 2022-10-27 11:23:35] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:30 [I 2022-10-27 11:23:35] ...trap/engines/engine.py.229: process: 0.55039 | load: 0.00243 [I 2022-10-27 11:23:35] ...trap/engines/engine.py.230: loss: 0.14383 [I 2022-10-27 11:23:35] ...optimizers/trijoint.py.151: total_norm: 7.761254 [I 2022-10-27 11:23:41] ...trap/engines/engine.py.224: train: epoch 64 | batch 1610/2304 [I 2022-10-27 11:23:41] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:26 [I 2022-10-27 11:23:41] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00243 [I 2022-10-27 11:23:41] ...trap/engines/engine.py.230: loss: 0.11990 [I 2022-10-27 11:23:41] ...optimizers/trijoint.py.151: total_norm: 16.696573 [I 2022-10-27 11:23:46] ...trap/engines/engine.py.224: train: epoch 64 | batch 1620/2304 [I 2022-10-27 11:23:46] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:20 [I 2022-10-27 11:23:46] ...trap/engines/engine.py.229: process: 0.55450 | load: 0.00247 [I 2022-10-27 11:23:46] ...trap/engines/engine.py.230: loss: 0.14054 [I 2022-10-27 11:23:46] ...optimizers/trijoint.py.151: total_norm: 30.980831 [I 2022-10-27 11:23:52] ...trap/engines/engine.py.224: train: epoch 64 | batch 1630/2304 [I 2022-10-27 11:23:52] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:14 [I 2022-10-27 11:23:52] ...trap/engines/engine.py.229: process: 0.54834 | load: 0.00241 [I 2022-10-27 11:23:52] ...trap/engines/engine.py.230: loss: 0.12903 [I 2022-10-27 11:23:52] ...optimizers/trijoint.py.151: total_norm: 10.751377 [I 2022-10-27 11:23:57] ...trap/engines/engine.py.224: train: epoch 64 | batch 1640/2304 [I 2022-10-27 11:23:57] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:06:11 [I 2022-10-27 11:23:57] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00244 [I 2022-10-27 11:23:57] ...trap/engines/engine.py.230: loss: 0.15779 [I 2022-10-27 11:23:57] ...optimizers/trijoint.py.151: total_norm: 20.458990 [I 2022-10-27 11:24:03] ...trap/engines/engine.py.224: train: epoch 64 | batch 1650/2304 [I 2022-10-27 11:24:03] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:06:04 [I 2022-10-27 11:24:03] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00250 [I 2022-10-27 11:24:03] ...trap/engines/engine.py.230: loss: 0.18992 [I 2022-10-27 11:24:03] ...optimizers/trijoint.py.151: total_norm: 13.629675 [I 2022-10-27 11:24:09] ...trap/engines/engine.py.224: train: epoch 64 | batch 1660/2304 [I 2022-10-27 11:24:09] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:06:00 [I 2022-10-27 11:24:09] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00245 [I 2022-10-27 11:24:09] ...trap/engines/engine.py.230: loss: 0.16384 [I 2022-10-27 11:24:09] ...optimizers/trijoint.py.151: total_norm: 20.788687 [I 2022-10-27 11:24:14] ...trap/engines/engine.py.224: train: epoch 64 | batch 1670/2304 [I 2022-10-27 11:24:14] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:53 [I 2022-10-27 11:24:14] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00244 [I 2022-10-27 11:24:14] ...trap/engines/engine.py.230: loss: 0.11075 [I 2022-10-27 11:24:14] ...optimizers/trijoint.py.151: total_norm: 8.344940 [I 2022-10-27 11:24:20] ...trap/engines/engine.py.224: train: epoch 64 | batch 1680/2304 [I 2022-10-27 11:24:20] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:45 [I 2022-10-27 11:24:20] ...trap/engines/engine.py.229: process: 0.55190 | load: 0.00243 [I 2022-10-27 11:24:20] ...trap/engines/engine.py.230: loss: 0.12749 [I 2022-10-27 11:24:20] ...optimizers/trijoint.py.151: total_norm: 14.281461 [I 2022-10-27 11:24:25] ...trap/engines/engine.py.224: train: epoch 64 | batch 1690/2304 [I 2022-10-27 11:24:25] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:40 [I 2022-10-27 11:24:25] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00242 [I 2022-10-27 11:24:25] ...trap/engines/engine.py.230: loss: 0.13790 [I 2022-10-27 11:24:25] ...optimizers/trijoint.py.151: total_norm: 11.410501 [I 2022-10-27 11:24:31] ...trap/engines/engine.py.224: train: epoch 64 | batch 1700/2304 [I 2022-10-27 11:24:31] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:33 [I 2022-10-27 11:24:31] ...trap/engines/engine.py.229: process: 0.54996 | load: 0.00243 [I 2022-10-27 11:24:31] ...trap/engines/engine.py.230: loss: 0.19131 [I 2022-10-27 11:24:31] ...optimizers/trijoint.py.151: total_norm: 23.328365 [I 2022-10-27 11:24:36] ...trap/engines/engine.py.224: train: epoch 64 | batch 1710/2304 [I 2022-10-27 11:24:36] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:29 [I 2022-10-27 11:24:36] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00243 [I 2022-10-27 11:24:36] ...trap/engines/engine.py.230: loss: 0.17407 [I 2022-10-27 11:24:36] ...optimizers/trijoint.py.151: total_norm: 13.327727 [I 2022-10-27 11:24:42] ...trap/engines/engine.py.224: train: epoch 64 | batch 1720/2304 [I 2022-10-27 11:24:42] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:24 [I 2022-10-27 11:24:42] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00243 [I 2022-10-27 11:24:42] ...trap/engines/engine.py.230: loss: 0.24058 [I 2022-10-27 11:24:42] ...optimizers/trijoint.py.151: total_norm: 15.483204 [I 2022-10-27 11:24:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 1730/2304 [I 2022-10-27 11:24:47] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:16 [I 2022-10-27 11:24:47] ...trap/engines/engine.py.229: process: 0.54588 | load: 0.00244 [I 2022-10-27 11:24:47] ...trap/engines/engine.py.230: loss: 0.16228 [I 2022-10-27 11:24:47] ...optimizers/trijoint.py.151: total_norm: 18.769239 [I 2022-10-27 11:24:53] ...trap/engines/engine.py.224: train: epoch 64 | batch 1740/2304 [I 2022-10-27 11:24:53] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:05:13 [I 2022-10-27 11:24:53] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00240 [I 2022-10-27 11:24:53] ...trap/engines/engine.py.230: loss: 0.15480 [I 2022-10-27 11:24:53] ...optimizers/trijoint.py.151: total_norm: 22.406942 [I 2022-10-27 11:24:59] ...trap/engines/engine.py.224: train: epoch 64 | batch 1750/2304 [I 2022-10-27 11:24:59] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:05:06 [I 2022-10-27 11:24:59] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00244 [I 2022-10-27 11:24:59] ...trap/engines/engine.py.230: loss: 0.13457 [I 2022-10-27 11:24:59] ...optimizers/trijoint.py.151: total_norm: 13.977791 [I 2022-10-27 11:25:04] ...trap/engines/engine.py.224: train: epoch 64 | batch 1760/2304 [I 2022-10-27 11:25:04] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:05:00 [I 2022-10-27 11:25:04] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00243 [I 2022-10-27 11:25:04] ...trap/engines/engine.py.230: loss: 0.14027 [I 2022-10-27 11:25:04] ...optimizers/trijoint.py.151: total_norm: 16.895647 [I 2022-10-27 11:25:10] ...trap/engines/engine.py.224: train: epoch 64 | batch 1770/2304 [I 2022-10-27 11:25:10] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:57 [I 2022-10-27 11:25:10] ...trap/engines/engine.py.229: process: 0.55520 | load: 0.00242 [I 2022-10-27 11:25:10] ...trap/engines/engine.py.230: loss: 0.12944 [I 2022-10-27 11:25:10] ...optimizers/trijoint.py.151: total_norm: 14.284059 [I 2022-10-27 11:25:15] ...trap/engines/engine.py.224: train: epoch 64 | batch 1780/2304 [I 2022-10-27 11:25:15] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:51 [I 2022-10-27 11:25:15] ...trap/engines/engine.py.229: process: 0.55510 | load: 0.00243 [I 2022-10-27 11:25:15] ...trap/engines/engine.py.230: loss: 0.14973 [I 2022-10-27 11:25:15] ...optimizers/trijoint.py.151: total_norm: 17.556095 [I 2022-10-27 11:25:21] ...trap/engines/engine.py.224: train: epoch 64 | batch 1790/2304 [I 2022-10-27 11:25:21] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:45 [I 2022-10-27 11:25:21] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00244 [I 2022-10-27 11:25:21] ...trap/engines/engine.py.230: loss: 0.17019 [I 2022-10-27 11:25:21] ...optimizers/trijoint.py.151: total_norm: 19.854761 [I 2022-10-27 11:25:26] ...trap/engines/engine.py.224: train: epoch 64 | batch 1800/2304 [I 2022-10-27 11:25:26] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:39 [I 2022-10-27 11:25:26] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00244 [I 2022-10-27 11:25:26] ...trap/engines/engine.py.230: loss: 0.09814 [I 2022-10-27 11:25:26] ...optimizers/trijoint.py.151: total_norm: 14.816253 [I 2022-10-27 11:25:32] ...trap/engines/engine.py.224: train: epoch 64 | batch 1810/2304 [I 2022-10-27 11:25:32] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:34 [I 2022-10-27 11:25:32] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00240 [I 2022-10-27 11:25:32] ...trap/engines/engine.py.230: loss: 0.08767 [I 2022-10-27 11:25:32] ...optimizers/trijoint.py.151: total_norm: 9.868690 [I 2022-10-27 11:25:38] ...trap/engines/engine.py.224: train: epoch 64 | batch 1820/2304 [I 2022-10-27 11:25:38] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:27 [I 2022-10-27 11:25:38] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00240 [I 2022-10-27 11:25:38] ...trap/engines/engine.py.230: loss: 0.11135 [I 2022-10-27 11:25:38] ...optimizers/trijoint.py.151: total_norm: 8.030164 [I 2022-10-27 11:25:43] ...trap/engines/engine.py.224: train: epoch 64 | batch 1830/2304 [I 2022-10-27 11:25:43] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:24 [I 2022-10-27 11:25:43] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00242 [I 2022-10-27 11:25:43] ...trap/engines/engine.py.230: loss: 0.15529 [I 2022-10-27 11:25:43] ...optimizers/trijoint.py.151: total_norm: 34.189793 [I 2022-10-27 11:25:49] ...trap/engines/engine.py.224: train: epoch 64 | batch 1840/2304 [I 2022-10-27 11:25:49] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:18 [I 2022-10-27 11:25:49] ...trap/engines/engine.py.229: process: 0.55847 | load: 0.00241 [I 2022-10-27 11:25:49] ...trap/engines/engine.py.230: loss: 0.18700 [I 2022-10-27 11:25:49] ...optimizers/trijoint.py.151: total_norm: 20.999079 [I 2022-10-27 11:25:54] ...trap/engines/engine.py.224: train: epoch 64 | batch 1850/2304 [I 2022-10-27 11:25:54] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:04:10 [I 2022-10-27 11:25:54] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00243 [I 2022-10-27 11:25:54] ...trap/engines/engine.py.230: loss: 0.16149 [I 2022-10-27 11:25:54] ...optimizers/trijoint.py.151: total_norm: 14.353962 [I 2022-10-27 11:26:00] ...trap/engines/engine.py.224: train: epoch 64 | batch 1860/2304 [I 2022-10-27 11:26:00] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:04:05 [I 2022-10-27 11:26:00] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00240 [I 2022-10-27 11:26:00] ...trap/engines/engine.py.230: loss: 0.17505 [I 2022-10-27 11:26:00] ...optimizers/trijoint.py.151: total_norm: 15.715703 [I 2022-10-27 11:26:05] ...trap/engines/engine.py.224: train: epoch 64 | batch 1870/2304 [I 2022-10-27 11:26:05] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:04:01 [I 2022-10-27 11:26:05] ...trap/engines/engine.py.229: process: 0.55414 | load: 0.00244 [I 2022-10-27 11:26:05] ...trap/engines/engine.py.230: loss: 0.11862 [I 2022-10-27 11:26:05] ...optimizers/trijoint.py.151: total_norm: 22.002401 [I 2022-10-27 11:26:11] ...trap/engines/engine.py.224: train: epoch 64 | batch 1880/2304 [I 2022-10-27 11:26:11] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:57 [I 2022-10-27 11:26:11] ...trap/engines/engine.py.229: process: 0.56845 | load: 0.00248 [I 2022-10-27 11:26:11] ...trap/engines/engine.py.230: loss: 0.20520 [I 2022-10-27 11:26:11] ...optimizers/trijoint.py.151: total_norm: 31.171698 [I 2022-10-27 11:26:17] ...trap/engines/engine.py.224: train: epoch 64 | batch 1890/2304 [I 2022-10-27 11:26:17] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:04:03 [I 2022-10-27 11:26:17] ...trap/engines/engine.py.229: process: 0.63817 | load: 0.00273 [I 2022-10-27 11:26:17] ...trap/engines/engine.py.230: loss: 0.17588 [I 2022-10-27 11:26:17] ...optimizers/trijoint.py.151: total_norm: 16.446453 [I 2022-10-27 11:26:22] ...trap/engines/engine.py.224: train: epoch 64 | batch 1900/2304 [I 2022-10-27 11:26:22] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:45 [I 2022-10-27 11:26:22] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00242 [I 2022-10-27 11:26:22] ...trap/engines/engine.py.230: loss: 0.19379 [I 2022-10-27 11:26:22] ...optimizers/trijoint.py.151: total_norm: 19.690031 [I 2022-10-27 11:26:28] ...trap/engines/engine.py.224: train: epoch 64 | batch 1910/2304 [I 2022-10-27 11:26:28] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:37 [I 2022-10-27 11:26:28] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00241 [I 2022-10-27 11:26:28] ...trap/engines/engine.py.230: loss: 0.12995 [I 2022-10-27 11:26:28] ...optimizers/trijoint.py.151: total_norm: 18.342365 [I 2022-10-27 11:26:33] ...trap/engines/engine.py.224: train: epoch 64 | batch 1920/2304 [I 2022-10-27 11:26:33] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:32 [I 2022-10-27 11:26:33] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00264 [I 2022-10-27 11:26:33] ...trap/engines/engine.py.230: loss: 0.12431 [I 2022-10-27 11:26:33] ...optimizers/trijoint.py.151: total_norm: 15.279036 [I 2022-10-27 11:26:39] ...trap/engines/engine.py.224: train: epoch 64 | batch 1930/2304 [I 2022-10-27 11:26:39] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:27 [I 2022-10-27 11:26:39] ...trap/engines/engine.py.229: process: 0.56237 | load: 0.00254 [I 2022-10-27 11:26:39] ...trap/engines/engine.py.230: loss: 0.24387 [I 2022-10-27 11:26:39] ...optimizers/trijoint.py.151: total_norm: 28.718639 [I 2022-10-27 11:26:45] ...trap/engines/engine.py.224: train: epoch 64 | batch 1940/2304 [I 2022-10-27 11:26:45] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:21 [I 2022-10-27 11:26:45] ...trap/engines/engine.py.229: process: 0.54871 | load: 0.00247 [I 2022-10-27 11:26:45] ...trap/engines/engine.py.230: loss: 0.11542 [I 2022-10-27 11:26:45] ...optimizers/trijoint.py.151: total_norm: 11.668663 [I 2022-10-27 11:26:50] ...trap/engines/engine.py.224: train: epoch 64 | batch 1950/2304 [I 2022-10-27 11:26:50] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:16 [I 2022-10-27 11:26:50] ...trap/engines/engine.py.229: process: 0.55432 | load: 0.00244 [I 2022-10-27 11:26:50] ...trap/engines/engine.py.230: loss: 0.16690 [I 2022-10-27 11:26:50] ...optimizers/trijoint.py.151: total_norm: 12.818779 [I 2022-10-27 11:26:56] ...trap/engines/engine.py.224: train: epoch 64 | batch 1960/2304 [I 2022-10-27 11:26:56] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:03:18 [I 2022-10-27 11:26:56] ...trap/engines/engine.py.229: process: 0.56918 | load: 0.00266 [I 2022-10-27 11:26:56] ...trap/engines/engine.py.230: loss: 0.11768 [I 2022-10-27 11:26:56] ...optimizers/trijoint.py.151: total_norm: 10.385722 [I 2022-10-27 11:27:01] ...trap/engines/engine.py.224: train: epoch 64 | batch 1970/2304 [I 2022-10-27 11:27:01] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:03:06 [I 2022-10-27 11:27:01] ...trap/engines/engine.py.229: process: 0.55543 | load: 0.00243 [I 2022-10-27 11:27:01] ...trap/engines/engine.py.230: loss: 0.11174 [I 2022-10-27 11:27:01] ...optimizers/trijoint.py.151: total_norm: 7.764069 [I 2022-10-27 11:27:07] ...trap/engines/engine.py.224: train: epoch 64 | batch 1980/2304 [I 2022-10-27 11:27:07] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:59 [I 2022-10-27 11:27:07] ...trap/engines/engine.py.229: process: 0.55461 | load: 0.00435 [I 2022-10-27 11:27:07] ...trap/engines/engine.py.230: loss: 0.24200 [I 2022-10-27 11:27:07] ...optimizers/trijoint.py.151: total_norm: 12.663070 [I 2022-10-27 11:27:12] ...trap/engines/engine.py.224: train: epoch 64 | batch 1990/2304 [I 2022-10-27 11:27:12] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:54 [I 2022-10-27 11:27:12] ...trap/engines/engine.py.229: process: 0.56327 | load: 0.00242 [I 2022-10-27 11:27:12] ...trap/engines/engine.py.230: loss: 0.14237 [I 2022-10-27 11:27:12] ...optimizers/trijoint.py.151: total_norm: 19.974449 [I 2022-10-27 11:27:18] ...trap/engines/engine.py.224: train: epoch 64 | batch 2000/2304 [I 2022-10-27 11:27:18] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:48 [I 2022-10-27 11:27:18] ...trap/engines/engine.py.229: process: 0.54919 | load: 0.00244 [I 2022-10-27 11:27:18] ...trap/engines/engine.py.230: loss: 0.13733 [I 2022-10-27 11:27:18] ...optimizers/trijoint.py.151: total_norm: 11.603312 [I 2022-10-27 11:27:24] ...trap/engines/engine.py.224: train: epoch 64 | batch 2010/2304 [I 2022-10-27 11:27:24] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:43 [I 2022-10-27 11:27:24] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00252 [I 2022-10-27 11:27:24] ...trap/engines/engine.py.230: loss: 0.12536 [I 2022-10-27 11:27:24] ...optimizers/trijoint.py.151: total_norm: 21.880501 [I 2022-10-27 11:27:29] ...trap/engines/engine.py.224: train: epoch 64 | batch 2020/2304 [I 2022-10-27 11:27:29] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:38 [I 2022-10-27 11:27:29] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00247 [I 2022-10-27 11:27:29] ...trap/engines/engine.py.230: loss: 0.23229 [I 2022-10-27 11:27:29] ...optimizers/trijoint.py.151: total_norm: 21.377531 [I 2022-10-27 11:27:35] ...trap/engines/engine.py.224: train: epoch 64 | batch 2030/2304 [I 2022-10-27 11:27:35] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:33 [I 2022-10-27 11:27:35] ...trap/engines/engine.py.229: process: 0.55049 | load: 0.00254 [I 2022-10-27 11:27:35] ...trap/engines/engine.py.230: loss: 0.15430 [I 2022-10-27 11:27:35] ...optimizers/trijoint.py.151: total_norm: 14.436613 [I 2022-10-27 11:27:40] ...trap/engines/engine.py.224: train: epoch 64 | batch 2040/2304 [I 2022-10-27 11:27:40] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:27 [I 2022-10-27 11:27:40] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00259 [I 2022-10-27 11:27:40] ...trap/engines/engine.py.230: loss: 0.11446 [I 2022-10-27 11:27:40] ...optimizers/trijoint.py.151: total_norm: 11.022109 [I 2022-10-27 11:27:46] ...trap/engines/engine.py.224: train: epoch 64 | batch 2050/2304 [I 2022-10-27 11:27:46] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:20 [I 2022-10-27 11:27:46] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00249 [I 2022-10-27 11:27:46] ...trap/engines/engine.py.230: loss: 0.12056 [I 2022-10-27 11:27:46] ...optimizers/trijoint.py.151: total_norm: 16.592997 [I 2022-10-27 11:27:52] ...trap/engines/engine.py.224: train: epoch 64 | batch 2060/2304 [I 2022-10-27 11:27:52] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:15 [I 2022-10-27 11:27:52] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00253 [I 2022-10-27 11:27:52] ...trap/engines/engine.py.230: loss: 0.22047 [I 2022-10-27 11:27:52] ...optimizers/trijoint.py.151: total_norm: 18.818064 [I 2022-10-27 11:27:57] ...trap/engines/engine.py.224: train: epoch 64 | batch 2070/2304 [I 2022-10-27 11:27:57] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:02:08 [I 2022-10-27 11:27:57] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00286 [I 2022-10-27 11:27:57] ...trap/engines/engine.py.230: loss: 0.12596 [I 2022-10-27 11:27:57] ...optimizers/trijoint.py.151: total_norm: 10.649902 [I 2022-10-27 11:28:03] ...trap/engines/engine.py.224: train: epoch 64 | batch 2080/2304 [I 2022-10-27 11:28:03] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:02:03 [I 2022-10-27 11:28:03] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00257 [I 2022-10-27 11:28:03] ...trap/engines/engine.py.230: loss: 0.11214 [I 2022-10-27 11:28:03] ...optimizers/trijoint.py.151: total_norm: 30.607132 [I 2022-10-27 11:28:08] ...trap/engines/engine.py.224: train: epoch 64 | batch 2090/2304 [I 2022-10-27 11:28:08] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:02:02 [I 2022-10-27 11:28:08] ...trap/engines/engine.py.229: process: 0.56680 | load: 0.00282 [I 2022-10-27 11:28:08] ...trap/engines/engine.py.230: loss: 0.19061 [I 2022-10-27 11:28:08] ...optimizers/trijoint.py.151: total_norm: 15.004342 [I 2022-10-27 11:28:14] ...trap/engines/engine.py.224: train: epoch 64 | batch 2100/2304 [I 2022-10-27 11:28:14] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:53 [I 2022-10-27 11:28:14] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00278 [I 2022-10-27 11:28:14] ...trap/engines/engine.py.230: loss: 0.15265 [I 2022-10-27 11:28:14] ...optimizers/trijoint.py.151: total_norm: 24.243633 [I 2022-10-27 11:28:19] ...trap/engines/engine.py.224: train: epoch 64 | batch 2110/2304 [I 2022-10-27 11:28:19] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:47 [I 2022-10-27 11:28:19] ...trap/engines/engine.py.229: process: 0.55521 | load: 0.00267 [I 2022-10-27 11:28:19] ...trap/engines/engine.py.230: loss: 0.13287 [I 2022-10-27 11:28:19] ...optimizers/trijoint.py.151: total_norm: 18.113916 [I 2022-10-27 11:28:25] ...trap/engines/engine.py.224: train: epoch 64 | batch 2120/2304 [I 2022-10-27 11:28:25] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:41 [I 2022-10-27 11:28:25] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00264 [I 2022-10-27 11:28:25] ...trap/engines/engine.py.230: loss: 0.09823 [I 2022-10-27 11:28:25] ...optimizers/trijoint.py.151: total_norm: 9.048979 [I 2022-10-27 11:28:30] ...trap/engines/engine.py.224: train: epoch 64 | batch 2130/2304 [I 2022-10-27 11:28:30] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:36 [I 2022-10-27 11:28:30] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00274 [I 2022-10-27 11:28:30] ...trap/engines/engine.py.230: loss: 0.17858 [I 2022-10-27 11:28:30] ...optimizers/trijoint.py.151: total_norm: 14.627519 [I 2022-10-27 11:28:36] ...trap/engines/engine.py.224: train: epoch 64 | batch 2140/2304 [I 2022-10-27 11:28:36] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:31 [I 2022-10-27 11:28:36] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00262 [I 2022-10-27 11:28:36] ...trap/engines/engine.py.230: loss: 0.17706 [I 2022-10-27 11:28:36] ...optimizers/trijoint.py.151: total_norm: 9.760588 [I 2022-10-27 11:28:42] ...trap/engines/engine.py.224: train: epoch 64 | batch 2150/2304 [I 2022-10-27 11:28:42] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:25 [I 2022-10-27 11:28:42] ...trap/engines/engine.py.229: process: 0.56378 | load: 0.00254 [I 2022-10-27 11:28:42] ...trap/engines/engine.py.230: loss: 0.14003 [I 2022-10-27 11:28:42] ...optimizers/trijoint.py.151: total_norm: 12.746751 [I 2022-10-27 11:28:47] ...trap/engines/engine.py.224: train: epoch 64 | batch 2160/2304 [I 2022-10-27 11:28:47] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:20 [I 2022-10-27 11:28:47] ...trap/engines/engine.py.229: process: 0.55993 | load: 0.00256 [I 2022-10-27 11:28:47] ...trap/engines/engine.py.230: loss: 0.16833 [I 2022-10-27 11:28:47] ...optimizers/trijoint.py.151: total_norm: 16.718630 [I 2022-10-27 11:28:53] ...trap/engines/engine.py.224: train: epoch 64 | batch 2170/2304 [I 2022-10-27 11:28:53] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:01:14 [I 2022-10-27 11:28:53] ...trap/engines/engine.py.229: process: 0.55491 | load: 0.00247 [I 2022-10-27 11:28:53] ...trap/engines/engine.py.230: loss: 0.15549 [I 2022-10-27 11:28:53] ...optimizers/trijoint.py.151: total_norm: 14.087708 [I 2022-10-27 11:28:58] ...trap/engines/engine.py.224: train: epoch 64 | batch 2180/2304 [I 2022-10-27 11:28:58] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:08 [I 2022-10-27 11:28:58] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00252 [I 2022-10-27 11:28:58] ...trap/engines/engine.py.230: loss: 0.18867 [I 2022-10-27 11:28:58] ...optimizers/trijoint.py.151: total_norm: 27.406527 [I 2022-10-27 11:29:04] ...trap/engines/engine.py.224: train: epoch 64 | batch 2190/2304 [I 2022-10-27 11:29:04] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:01:03 [I 2022-10-27 11:29:04] ...trap/engines/engine.py.229: process: 0.55950 | load: 0.00250 [I 2022-10-27 11:29:04] ...trap/engines/engine.py.230: loss: 0.15702 [I 2022-10-27 11:29:04] ...optimizers/trijoint.py.151: total_norm: 20.241978 [I 2022-10-27 11:29:09] ...trap/engines/engine.py.224: train: epoch 64 | batch 2200/2304 [I 2022-10-27 11:29:09] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:57 [I 2022-10-27 11:29:09] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00249 [I 2022-10-27 11:29:09] ...trap/engines/engine.py.230: loss: 0.14040 [I 2022-10-27 11:29:09] ...optimizers/trijoint.py.151: total_norm: 19.321743 [I 2022-10-27 11:29:15] ...trap/engines/engine.py.224: train: epoch 64 | batch 2210/2304 [I 2022-10-27 11:29:15] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:52 [I 2022-10-27 11:29:15] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00242 [I 2022-10-27 11:29:15] ...trap/engines/engine.py.230: loss: 0.13891 [I 2022-10-27 11:29:15] ...optimizers/trijoint.py.151: total_norm: 13.163612 [I 2022-10-27 11:29:21] ...trap/engines/engine.py.224: train: epoch 64 | batch 2220/2304 [I 2022-10-27 11:29:21] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:48 [I 2022-10-27 11:29:21] ...trap/engines/engine.py.229: process: 0.56188 | load: 0.00252 [I 2022-10-27 11:29:21] ...trap/engines/engine.py.230: loss: 0.15891 [I 2022-10-27 11:29:21] ...optimizers/trijoint.py.151: total_norm: 16.408442 [I 2022-10-27 11:29:26] ...trap/engines/engine.py.224: train: epoch 64 | batch 2230/2304 [I 2022-10-27 11:29:26] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:41 [I 2022-10-27 11:29:26] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00249 [I 2022-10-27 11:29:26] ...trap/engines/engine.py.230: loss: 0.18384 [I 2022-10-27 11:29:26] ...optimizers/trijoint.py.151: total_norm: 21.273935 [I 2022-10-27 11:29:32] ...trap/engines/engine.py.224: train: epoch 64 | batch 2240/2304 [I 2022-10-27 11:29:32] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:35 [I 2022-10-27 11:29:32] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00250 [I 2022-10-27 11:29:32] ...trap/engines/engine.py.230: loss: 0.13190 [I 2022-10-27 11:29:32] ...optimizers/trijoint.py.151: total_norm: 8.236096 [I 2022-10-27 11:29:37] ...trap/engines/engine.py.224: train: epoch 64 | batch 2250/2304 [I 2022-10-27 11:29:37] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:29 [I 2022-10-27 11:29:37] ...trap/engines/engine.py.229: process: 0.56478 | load: 0.00254 [I 2022-10-27 11:29:37] ...trap/engines/engine.py.230: loss: 0.15110 [I 2022-10-27 11:29:37] ...optimizers/trijoint.py.151: total_norm: 11.076811 [I 2022-10-27 11:29:43] ...trap/engines/engine.py.224: train: epoch 64 | batch 2260/2304 [I 2022-10-27 11:29:43] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:24 [I 2022-10-27 11:29:43] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00246 [I 2022-10-27 11:29:43] ...trap/engines/engine.py.230: loss: 0.15710 [I 2022-10-27 11:29:43] ...optimizers/trijoint.py.151: total_norm: 12.252074 [I 2022-10-27 11:29:49] ...trap/engines/engine.py.224: train: epoch 64 | batch 2270/2304 [I 2022-10-27 11:29:49] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:18 [I 2022-10-27 11:29:49] ...trap/engines/engine.py.229: process: 0.55942 | load: 0.00249 [I 2022-10-27 11:29:49] ...trap/engines/engine.py.230: loss: 0.12604 [I 2022-10-27 11:29:49] ...optimizers/trijoint.py.151: total_norm: 16.351463 [I 2022-10-27 11:29:54] ...trap/engines/engine.py.224: train: epoch 64 | batch 2280/2304 [I 2022-10-27 11:29:54] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:13 [I 2022-10-27 11:29:54] ...trap/engines/engine.py.229: process: 0.55359 | load: 0.00249 [I 2022-10-27 11:29:54] ...trap/engines/engine.py.230: loss: 0.25354 [I 2022-10-27 11:29:54] ...optimizers/trijoint.py.151: total_norm: 13.239174 [I 2022-10-27 11:30:00] ...trap/engines/engine.py.224: train: epoch 64 | batch 2290/2304 [I 2022-10-27 11:30:00] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:00:07 [I 2022-10-27 11:30:00] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00339 [I 2022-10-27 11:30:00] ...trap/engines/engine.py.230: loss: 0.16820 [I 2022-10-27 11:30:00] ...optimizers/trijoint.py.151: total_norm: 24.382814 [I 2022-10-27 11:30:06] ...trap/engines/engine.py.224: train: epoch 64 | batch 2300/2304 [I 2022-10-27 11:30:06] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:02 [I 2022-10-27 11:30:06] ...trap/engines/engine.py.229: process: 0.54600 | load: 0.00016 [I 2022-10-27 11:30:06] ...trap/engines/engine.py.230: loss: 0.13491 [I 2022-10-27 11:30:06] ...optimizers/trijoint.py.151: total_norm: 15.667723 [I 2022-10-27 11:30:08] ...trap/engines/engine.py.224: train: epoch 64 | batch 2304/2304 [I 2022-10-27 11:30:08] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:00 [I 2022-10-27 11:30:08] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00017 [I 2022-10-27 11:30:08] ...trap/engines/engine.py.230: loss: 0.13986 [I 2022-10-27 11:30:08] ...optimizers/trijoint.py.151: total_norm: 15.735415 [S 2022-10-27 11:30:09] ...trap/engines/engine.py.129: train_epoch.epoch: 64 [S 2022-10-27 11:30:09] ...trap/engines/engine.py.129: train_epoch.loss: 0.15007621848919386 [I 2022-10-27 11:30:09] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 11:30:10] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 64 [I 2022-10-27 11:30:10] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 11:30:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 11:30:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 11:30:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 11:30:14] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 11:30:14] ...trap/engines/engine.py.321: eval: epoch 64 | batch 0/510 [I 2022-10-27 11:30:14] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:37:14 [I 2022-10-27 11:30:14] ...trap/engines/engine.py.326: process: 4.38184 | load: 4.01198 [I 2022-10-27 11:30:18] ...trap/engines/engine.py.321: eval: epoch 64 | batch 10/510 [I 2022-10-27 11:30:18] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:24 [I 2022-10-27 11:30:18] ...trap/engines/engine.py.326: process: 0.34106 | load: 0.00019 [I 2022-10-27 11:30:21] ...trap/engines/engine.py.321: eval: epoch 64 | batch 20/510 [I 2022-10-27 11:30:21] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:05 [I 2022-10-27 11:30:21] ...trap/engines/engine.py.326: process: 0.33448 | load: 0.00020 [I 2022-10-27 11:30:24] ...trap/engines/engine.py.321: eval: epoch 64 | batch 30/510 [I 2022-10-27 11:30:24] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:44 [I 2022-10-27 11:30:24] ...trap/engines/engine.py.326: process: 0.34252 | load: 0.00016 [I 2022-10-27 11:30:28] ...trap/engines/engine.py.321: eval: epoch 64 | batch 40/510 [I 2022-10-27 11:30:28] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:38 [I 2022-10-27 11:30:28] ...trap/engines/engine.py.326: process: 0.34286 | load: 0.00019 [I 2022-10-27 11:30:31] ...trap/engines/engine.py.321: eval: epoch 64 | batch 50/510 [I 2022-10-27 11:30:31] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:35 [I 2022-10-27 11:30:31] ...trap/engines/engine.py.326: process: 0.33243 | load: 0.00017 [I 2022-10-27 11:30:34] ...trap/engines/engine.py.321: eval: epoch 64 | batch 60/510 [I 2022-10-27 11:30:34] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:31 [I 2022-10-27 11:30:34] ...trap/engines/engine.py.326: process: 0.32619 | load: 0.00018 [I 2022-10-27 11:30:38] ...trap/engines/engine.py.321: eval: epoch 64 | batch 70/510 [I 2022-10-27 11:30:38] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:27 [I 2022-10-27 11:30:38] ...trap/engines/engine.py.326: process: 0.34467 | load: 0.00016 [I 2022-10-27 11:30:41] ...trap/engines/engine.py.321: eval: epoch 64 | batch 80/510 [I 2022-10-27 11:30:41] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:24 [I 2022-10-27 11:30:41] ...trap/engines/engine.py.326: process: 0.33934 | load: 0.00018 [I 2022-10-27 11:30:45] ...trap/engines/engine.py.321: eval: epoch 64 | batch 90/510 [I 2022-10-27 11:30:45] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:22 [I 2022-10-27 11:30:45] ...trap/engines/engine.py.326: process: 0.33316 | load: 0.00019 [I 2022-10-27 11:30:48] ...trap/engines/engine.py.321: eval: epoch 64 | batch 100/510 [I 2022-10-27 11:30:48] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:14 [I 2022-10-27 11:30:48] ...trap/engines/engine.py.326: process: 0.28260 | load: 0.00017 [I 2022-10-27 11:30:51] ...trap/engines/engine.py.321: eval: epoch 64 | batch 110/510 [I 2022-10-27 11:30:51] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:57 [I 2022-10-27 11:30:51] ...trap/engines/engine.py.326: process: 0.29544 | load: 0.00014 [I 2022-10-27 11:30:54] ...trap/engines/engine.py.321: eval: epoch 64 | batch 120/510 [I 2022-10-27 11:30:54] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:53 [I 2022-10-27 11:30:54] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00025 [I 2022-10-27 11:30:57] ...trap/engines/engine.py.321: eval: epoch 64 | batch 130/510 [I 2022-10-27 11:30:57] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:50 [I 2022-10-27 11:30:57] ...trap/engines/engine.py.326: process: 0.29271 | load: 0.00022 [I 2022-10-27 11:30:59] ...trap/engines/engine.py.321: eval: epoch 64 | batch 140/510 [I 2022-10-27 11:30:59] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:48 [I 2022-10-27 11:30:59] ...trap/engines/engine.py.326: process: 0.28816 | load: 0.00025 [I 2022-10-27 11:31:02] ...trap/engines/engine.py.321: eval: epoch 64 | batch 150/510 [I 2022-10-27 11:31:02] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:46 [I 2022-10-27 11:31:02] ...trap/engines/engine.py.326: process: 0.29208 | load: 0.00019 [I 2022-10-27 11:31:05] ...trap/engines/engine.py.321: eval: epoch 64 | batch 160/510 [I 2022-10-27 11:31:05] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:43 [I 2022-10-27 11:31:05] ...trap/engines/engine.py.326: process: 0.28838 | load: 0.00025 [I 2022-10-27 11:31:08] ...trap/engines/engine.py.321: eval: epoch 64 | batch 170/510 [I 2022-10-27 11:31:08] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:39 [I 2022-10-27 11:31:08] ...trap/engines/engine.py.326: process: 0.28887 | load: 0.00015 [I 2022-10-27 11:31:11] ...trap/engines/engine.py.321: eval: epoch 64 | batch 180/510 [I 2022-10-27 11:31:11] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 11:31:11] ...trap/engines/engine.py.326: process: 0.28349 | load: 0.00018 [I 2022-10-27 11:31:14] ...trap/engines/engine.py.321: eval: epoch 64 | batch 190/510 [I 2022-10-27 11:31:14] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 11:31:14] ...trap/engines/engine.py.326: process: 0.29311 | load: 0.00015 [I 2022-10-27 11:31:17] ...trap/engines/engine.py.321: eval: epoch 64 | batch 200/510 [I 2022-10-27 11:31:17] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 11:31:17] ...trap/engines/engine.py.326: process: 0.29333 | load: 0.00018 [I 2022-10-27 11:31:20] ...trap/engines/engine.py.321: eval: epoch 64 | batch 210/510 [I 2022-10-27 11:31:20] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:27 [I 2022-10-27 11:31:20] ...trap/engines/engine.py.326: process: 0.29387 | load: 0.00015 [I 2022-10-27 11:31:23] ...trap/engines/engine.py.321: eval: epoch 64 | batch 220/510 [I 2022-10-27 11:31:23] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 11:31:23] ...trap/engines/engine.py.326: process: 0.28667 | load: 0.00029 [I 2022-10-27 11:31:26] ...trap/engines/engine.py.321: eval: epoch 64 | batch 230/510 [I 2022-10-27 11:31:26] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 11:31:26] ...trap/engines/engine.py.326: process: 0.28986 | load: 0.00017 [I 2022-10-27 11:31:29] ...trap/engines/engine.py.321: eval: epoch 64 | batch 240/510 [I 2022-10-27 11:31:29] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 11:31:29] ...trap/engines/engine.py.326: process: 0.28583 | load: 0.00031 [I 2022-10-27 11:31:31] ...trap/engines/engine.py.321: eval: epoch 64 | batch 250/510 [I 2022-10-27 11:31:31] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 11:31:31] ...trap/engines/engine.py.326: process: 0.29453 | load: 0.00023 [I 2022-10-27 11:31:34] ...trap/engines/engine.py.321: eval: epoch 64 | batch 260/510 [I 2022-10-27 11:31:34] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 11:31:34] ...trap/engines/engine.py.326: process: 0.29198 | load: 0.00030 [I 2022-10-27 11:31:37] ...trap/engines/engine.py.321: eval: epoch 64 | batch 270/510 [I 2022-10-27 11:31:37] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 11:31:37] ...trap/engines/engine.py.326: process: 0.29354 | load: 0.00020 [I 2022-10-27 11:31:40] ...trap/engines/engine.py.321: eval: epoch 64 | batch 280/510 [I 2022-10-27 11:31:40] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:07 [I 2022-10-27 11:31:40] ...trap/engines/engine.py.326: process: 0.29423 | load: 0.00028 [I 2022-10-27 11:31:43] ...trap/engines/engine.py.321: eval: epoch 64 | batch 290/510 [I 2022-10-27 11:31:43] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:04 [I 2022-10-27 11:31:43] ...trap/engines/engine.py.326: process: 0.29412 | load: 0.00020 [I 2022-10-27 11:31:46] ...trap/engines/engine.py.321: eval: epoch 64 | batch 300/510 [I 2022-10-27 11:31:46] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 11:31:46] ...trap/engines/engine.py.326: process: 0.28692 | load: 0.00025 [I 2022-10-27 11:31:49] ...trap/engines/engine.py.321: eval: epoch 64 | batch 310/510 [I 2022-10-27 11:31:49] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 11:31:49] ...trap/engines/engine.py.326: process: 0.28240 | load: 0.00019 [I 2022-10-27 11:31:52] ...trap/engines/engine.py.321: eval: epoch 64 | batch 320/510 [I 2022-10-27 11:31:52] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 11:31:52] ...trap/engines/engine.py.326: process: 0.28811 | load: 0.00028 [I 2022-10-27 11:31:55] ...trap/engines/engine.py.321: eval: epoch 64 | batch 330/510 [I 2022-10-27 11:31:55] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:52 [I 2022-10-27 11:31:55] ...trap/engines/engine.py.326: process: 0.28909 | load: 0.00017 [I 2022-10-27 11:31:58] ...trap/engines/engine.py.321: eval: epoch 64 | batch 340/510 [I 2022-10-27 11:31:58] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 11:31:58] ...trap/engines/engine.py.326: process: 0.28587 | load: 0.00028 [I 2022-10-27 11:32:00] ...trap/engines/engine.py.321: eval: epoch 64 | batch 350/510 [I 2022-10-27 11:32:00] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 11:32:00] ...trap/engines/engine.py.326: process: 0.28301 | load: 0.00017 [I 2022-10-27 11:32:03] ...trap/engines/engine.py.321: eval: epoch 64 | batch 360/510 [I 2022-10-27 11:32:03] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 11:32:03] ...trap/engines/engine.py.326: process: 0.28826 | load: 0.00029 [I 2022-10-27 11:32:06] ...trap/engines/engine.py.321: eval: epoch 64 | batch 370/510 [I 2022-10-27 11:32:06] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 11:32:06] ...trap/engines/engine.py.326: process: 0.29027 | load: 0.00019 [I 2022-10-27 11:32:09] ...trap/engines/engine.py.321: eval: epoch 64 | batch 380/510 [I 2022-10-27 11:32:09] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 11:32:09] ...trap/engines/engine.py.326: process: 0.29304 | load: 0.00029 [I 2022-10-27 11:32:12] ...trap/engines/engine.py.321: eval: epoch 64 | batch 390/510 [I 2022-10-27 11:32:12] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 11:32:12] ...trap/engines/engine.py.326: process: 0.29118 | load: 0.00018 [I 2022-10-27 11:32:15] ...trap/engines/engine.py.321: eval: epoch 64 | batch 400/510 [I 2022-10-27 11:32:15] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 11:32:15] ...trap/engines/engine.py.326: process: 0.29096 | load: 0.00028 [I 2022-10-27 11:32:18] ...trap/engines/engine.py.321: eval: epoch 64 | batch 410/510 [I 2022-10-27 11:32:18] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:29 [I 2022-10-27 11:32:18] ...trap/engines/engine.py.326: process: 0.29322 | load: 0.00018 [I 2022-10-27 11:32:21] ...trap/engines/engine.py.321: eval: epoch 64 | batch 420/510 [I 2022-10-27 11:32:21] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:26 [I 2022-10-27 11:32:21] ...trap/engines/engine.py.326: process: 0.28637 | load: 0.00028 [I 2022-10-27 11:32:24] ...trap/engines/engine.py.321: eval: epoch 64 | batch 430/510 [I 2022-10-27 11:32:24] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 11:32:24] ...trap/engines/engine.py.326: process: 0.29697 | load: 0.00024 [I 2022-10-27 11:32:26] ...trap/engines/engine.py.321: eval: epoch 64 | batch 440/510 [I 2022-10-27 11:32:26] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 11:32:26] ...trap/engines/engine.py.326: process: 0.28375 | load: 0.00028 [I 2022-10-27 11:32:29] ...trap/engines/engine.py.321: eval: epoch 64 | batch 450/510 [I 2022-10-27 11:32:29] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 11:32:29] ...trap/engines/engine.py.326: process: 0.29028 | load: 0.00024 [I 2022-10-27 11:32:32] ...trap/engines/engine.py.321: eval: epoch 64 | batch 460/510 [I 2022-10-27 11:32:32] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 11:32:32] ...trap/engines/engine.py.326: process: 0.28726 | load: 0.00030 [I 2022-10-27 11:32:35] ...trap/engines/engine.py.321: eval: epoch 64 | batch 470/510 [I 2022-10-27 11:32:35] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 11:32:35] ...trap/engines/engine.py.326: process: 0.28820 | load: 0.00019 [I 2022-10-27 11:32:38] ...trap/engines/engine.py.321: eval: epoch 64 | batch 480/510 [I 2022-10-27 11:32:38] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 11:32:38] ...trap/engines/engine.py.326: process: 0.29071 | load: 0.00027 [I 2022-10-27 11:32:41] ...trap/engines/engine.py.321: eval: epoch 64 | batch 490/510 [I 2022-10-27 11:32:41] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 11:32:41] ...trap/engines/engine.py.326: process: 0.29181 | load: 0.00021 [I 2022-10-27 11:32:44] ...trap/engines/engine.py.321: eval: epoch 64 | batch 500/510 [I 2022-10-27 11:32:44] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 11:32:44] ...trap/engines/engine.py.326: process: 0.28935 | load: 0.00029 [I 2022-10-27 11:32:47] ...trap/engines/engine.py.321: eval: epoch 64 | batch 510/510 [I 2022-10-27 11:32:47] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 11:32:47] ...trap/engines/engine.py.326: process: 0.28388 | load: 0.00011 [S 2022-10-27 11:32:47] ...trap/engines/engine.py.132: eval_epoch.epoch: 64 [I 2022-10-27 11:32:47] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.710600 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.899800 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935300 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.700700 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.898600 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931400 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014520 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007666 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009706 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010955 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009952 [S 2022-10-27 11:32:52] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009952 [I 2022-10-27 11:32:53] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 11:32:53] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 11:32:53] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 11:32:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 11:32:53] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 11:32:57] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 11:32:58] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 11:33:04] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 11:33:04] ...trap/engines/engine.py.163: Training model on trainset for epoch 65 [I 2022-10-27 11:33:04] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 11:33:07] ...trap/engines/engine.py.224: train: epoch 65 | batch 0/2304 [I 2022-10-27 11:33:07] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:31 [I 2022-10-27 11:33:07] ...trap/engines/engine.py.229: process: 2.66998 | load: 2.04634 [I 2022-10-27 11:33:07] ...trap/engines/engine.py.230: loss: 0.11198 [I 2022-10-27 11:33:07] ...optimizers/trijoint.py.151: total_norm: 15.497741 [I 2022-10-27 11:33:12] ...trap/engines/engine.py.224: train: epoch 65 | batch 10/2304 [I 2022-10-27 11:33:12] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:54 [I 2022-10-27 11:33:12] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00385 [I 2022-10-27 11:33:12] ...trap/engines/engine.py.230: loss: 0.19127 [I 2022-10-27 11:33:12] ...optimizers/trijoint.py.151: total_norm: 18.127605 [I 2022-10-27 11:33:18] ...trap/engines/engine.py.224: train: epoch 65 | batch 20/2304 [I 2022-10-27 11:33:18] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:21:58 [I 2022-10-27 11:33:18] ...trap/engines/engine.py.229: process: 0.54277 | load: 0.00443 [I 2022-10-27 11:33:18] ...trap/engines/engine.py.230: loss: 0.14361 [I 2022-10-27 11:33:18] ...optimizers/trijoint.py.151: total_norm: 11.152414 [I 2022-10-27 11:33:23] ...trap/engines/engine.py.224: train: epoch 65 | batch 30/2304 [I 2022-10-27 11:33:23] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:10 [I 2022-10-27 11:33:23] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00350 [I 2022-10-27 11:33:23] ...trap/engines/engine.py.230: loss: 0.09868 [I 2022-10-27 11:33:23] ...optimizers/trijoint.py.151: total_norm: 11.458100 [I 2022-10-27 11:33:29] ...trap/engines/engine.py.224: train: epoch 65 | batch 40/2304 [I 2022-10-27 11:33:29] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:51 [I 2022-10-27 11:33:29] ...trap/engines/engine.py.229: process: 0.53892 | load: 0.00287 [I 2022-10-27 11:33:29] ...trap/engines/engine.py.230: loss: 0.14491 [I 2022-10-27 11:33:29] ...optimizers/trijoint.py.151: total_norm: 17.192751 [I 2022-10-27 11:33:34] ...trap/engines/engine.py.224: train: epoch 65 | batch 50/2304 [I 2022-10-27 11:33:34] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:49 [I 2022-10-27 11:33:34] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00343 [I 2022-10-27 11:33:34] ...trap/engines/engine.py.230: loss: 0.12749 [I 2022-10-27 11:33:34] ...optimizers/trijoint.py.151: total_norm: 15.203990 [I 2022-10-27 11:33:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 60/2304 [I 2022-10-27 11:33:40] ...trap/engines/engine.py.228: elapsed: 0:00:35 | left: 0:20:33 [I 2022-10-27 11:33:40] ...trap/engines/engine.py.229: process: 0.54749 | load: 0.00299 [I 2022-10-27 11:33:40] ...trap/engines/engine.py.230: loss: 0.17752 [I 2022-10-27 11:33:40] ...optimizers/trijoint.py.151: total_norm: 13.329152 [I 2022-10-27 11:33:45] ...trap/engines/engine.py.224: train: epoch 65 | batch 70/2304 [I 2022-10-27 11:33:45] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:32 [I 2022-10-27 11:33:45] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00250 [I 2022-10-27 11:33:45] ...trap/engines/engine.py.230: loss: 0.20383 [I 2022-10-27 11:33:45] ...optimizers/trijoint.py.151: total_norm: 23.673178 [I 2022-10-27 11:33:51] ...trap/engines/engine.py.224: train: epoch 65 | batch 80/2304 [I 2022-10-27 11:33:51] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:38 [I 2022-10-27 11:33:51] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00295 [I 2022-10-27 11:33:51] ...trap/engines/engine.py.230: loss: 0.15360 [I 2022-10-27 11:33:51] ...optimizers/trijoint.py.151: total_norm: 16.320805 [I 2022-10-27 11:33:57] ...trap/engines/engine.py.224: train: epoch 65 | batch 90/2304 [I 2022-10-27 11:33:57] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:36 [I 2022-10-27 11:33:57] ...trap/engines/engine.py.229: process: 0.54181 | load: 0.00282 [I 2022-10-27 11:33:57] ...trap/engines/engine.py.230: loss: 0.11937 [I 2022-10-27 11:33:57] ...optimizers/trijoint.py.151: total_norm: 16.023230 [I 2022-10-27 11:34:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 100/2304 [I 2022-10-27 11:34:02] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:25 [I 2022-10-27 11:34:02] ...trap/engines/engine.py.229: process: 0.55576 | load: 0.00267 [I 2022-10-27 11:34:02] ...trap/engines/engine.py.230: loss: 0.13478 [I 2022-10-27 11:34:02] ...optimizers/trijoint.py.151: total_norm: 11.764965 [I 2022-10-27 11:34:08] ...trap/engines/engine.py.224: train: epoch 65 | batch 110/2304 [I 2022-10-27 11:34:08] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:20:26 [I 2022-10-27 11:34:08] ...trap/engines/engine.py.229: process: 0.55804 | load: 0.00240 [I 2022-10-27 11:34:08] ...trap/engines/engine.py.230: loss: 0.10311 [I 2022-10-27 11:34:08] ...optimizers/trijoint.py.151: total_norm: 9.078100 [I 2022-10-27 11:34:13] ...trap/engines/engine.py.224: train: epoch 65 | batch 120/2304 [I 2022-10-27 11:34:13] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:06 [I 2022-10-27 11:34:13] ...trap/engines/engine.py.229: process: 0.54570 | load: 0.00243 [I 2022-10-27 11:34:13] ...trap/engines/engine.py.230: loss: 0.12126 [I 2022-10-27 11:34:13] ...optimizers/trijoint.py.151: total_norm: 10.044104 [I 2022-10-27 11:34:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 130/2304 [I 2022-10-27 11:34:19] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:20 [I 2022-10-27 11:34:19] ...trap/engines/engine.py.229: process: 0.55510 | load: 0.00242 [I 2022-10-27 11:34:19] ...trap/engines/engine.py.230: loss: 0.12933 [I 2022-10-27 11:34:19] ...optimizers/trijoint.py.151: total_norm: 22.352797 [I 2022-10-27 11:34:25] ...trap/engines/engine.py.224: train: epoch 65 | batch 140/2304 [I 2022-10-27 11:34:25] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:57 [I 2022-10-27 11:34:25] ...trap/engines/engine.py.229: process: 0.54169 | load: 0.00246 [I 2022-10-27 11:34:25] ...trap/engines/engine.py.230: loss: 0.22363 [I 2022-10-27 11:34:25] ...optimizers/trijoint.py.151: total_norm: 23.771385 [I 2022-10-27 11:34:30] ...trap/engines/engine.py.224: train: epoch 65 | batch 150/2304 [I 2022-10-27 11:34:30] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:49 [I 2022-10-27 11:34:30] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00244 [I 2022-10-27 11:34:30] ...trap/engines/engine.py.230: loss: 0.14505 [I 2022-10-27 11:34:30] ...optimizers/trijoint.py.151: total_norm: 23.195017 [I 2022-10-27 11:34:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 160/2304 [I 2022-10-27 11:34:36] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:58 [I 2022-10-27 11:34:36] ...trap/engines/engine.py.229: process: 0.56972 | load: 0.00242 [I 2022-10-27 11:34:36] ...trap/engines/engine.py.230: loss: 0.13349 [I 2022-10-27 11:34:36] ...optimizers/trijoint.py.151: total_norm: 12.005077 [I 2022-10-27 11:34:41] ...trap/engines/engine.py.224: train: epoch 65 | batch 170/2304 [I 2022-10-27 11:34:41] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:51 [I 2022-10-27 11:34:41] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00258 [I 2022-10-27 11:34:41] ...trap/engines/engine.py.230: loss: 0.15552 [I 2022-10-27 11:34:41] ...optimizers/trijoint.py.151: total_norm: 28.191307 [I 2022-10-27 11:34:47] ...trap/engines/engine.py.224: train: epoch 65 | batch 180/2304 [I 2022-10-27 11:34:47] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:19:47 [I 2022-10-27 11:34:47] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00247 [I 2022-10-27 11:34:47] ...trap/engines/engine.py.230: loss: 0.17601 [I 2022-10-27 11:34:47] ...optimizers/trijoint.py.151: total_norm: 56.099800 [I 2022-10-27 11:34:52] ...trap/engines/engine.py.224: train: epoch 65 | batch 190/2304 [I 2022-10-27 11:34:52] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:23 [I 2022-10-27 11:34:52] ...trap/engines/engine.py.229: process: 0.53408 | load: 0.00260 [I 2022-10-27 11:34:52] ...trap/engines/engine.py.230: loss: 0.17710 [I 2022-10-27 11:34:52] ...optimizers/trijoint.py.151: total_norm: 22.713079 [I 2022-10-27 11:34:58] ...trap/engines/engine.py.224: train: epoch 65 | batch 200/2304 [I 2022-10-27 11:34:58] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:31 [I 2022-10-27 11:34:58] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00252 [I 2022-10-27 11:34:58] ...trap/engines/engine.py.230: loss: 0.20194 [I 2022-10-27 11:34:58] ...optimizers/trijoint.py.151: total_norm: 18.928825 [I 2022-10-27 11:35:04] ...trap/engines/engine.py.224: train: epoch 65 | batch 210/2304 [I 2022-10-27 11:35:04] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:14 [I 2022-10-27 11:35:04] ...trap/engines/engine.py.229: process: 0.55055 | load: 0.00253 [I 2022-10-27 11:35:04] ...trap/engines/engine.py.230: loss: 0.10598 [I 2022-10-27 11:35:04] ...optimizers/trijoint.py.151: total_norm: 7.978315 [I 2022-10-27 11:35:09] ...trap/engines/engine.py.224: train: epoch 65 | batch 220/2304 [I 2022-10-27 11:35:09] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:15 [I 2022-10-27 11:35:09] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00255 [I 2022-10-27 11:35:09] ...trap/engines/engine.py.230: loss: 0.13869 [I 2022-10-27 11:35:09] ...optimizers/trijoint.py.151: total_norm: 11.543857 [I 2022-10-27 11:35:15] ...trap/engines/engine.py.224: train: epoch 65 | batch 230/2304 [I 2022-10-27 11:35:15] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:13 [I 2022-10-27 11:35:15] ...trap/engines/engine.py.229: process: 0.55072 | load: 0.00248 [I 2022-10-27 11:35:15] ...trap/engines/engine.py.230: loss: 0.16704 [I 2022-10-27 11:35:15] ...optimizers/trijoint.py.151: total_norm: 11.585695 [I 2022-10-27 11:35:20] ...trap/engines/engine.py.224: train: epoch 65 | batch 240/2304 [I 2022-10-27 11:35:20] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:08 [I 2022-10-27 11:35:20] ...trap/engines/engine.py.229: process: 0.56218 | load: 0.00266 [I 2022-10-27 11:35:20] ...trap/engines/engine.py.230: loss: 0.17702 [I 2022-10-27 11:35:20] ...optimizers/trijoint.py.151: total_norm: 20.406239 [I 2022-10-27 11:35:26] ...trap/engines/engine.py.224: train: epoch 65 | batch 250/2304 [I 2022-10-27 11:35:26] ...trap/engines/engine.py.228: elapsed: 0:02:21 | left: 0:19:17 [I 2022-10-27 11:35:26] ...trap/engines/engine.py.229: process: 0.56898 | load: 0.00254 [I 2022-10-27 11:35:26] ...trap/engines/engine.py.230: loss: 0.15368 [I 2022-10-27 11:35:26] ...optimizers/trijoint.py.151: total_norm: 53.766129 [I 2022-10-27 11:35:32] ...trap/engines/engine.py.224: train: epoch 65 | batch 260/2304 [I 2022-10-27 11:35:32] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:05 [I 2022-10-27 11:35:32] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00255 [I 2022-10-27 11:35:32] ...trap/engines/engine.py.230: loss: 0.17787 [I 2022-10-27 11:35:32] ...optimizers/trijoint.py.151: total_norm: 27.598059 [I 2022-10-27 11:35:37] ...trap/engines/engine.py.224: train: epoch 65 | batch 270/2304 [I 2022-10-27 11:35:37] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:48 [I 2022-10-27 11:35:37] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00244 [I 2022-10-27 11:35:37] ...trap/engines/engine.py.230: loss: 0.14317 [I 2022-10-27 11:35:37] ...optimizers/trijoint.py.151: total_norm: 27.305128 [I 2022-10-27 11:35:43] ...trap/engines/engine.py.224: train: epoch 65 | batch 280/2304 [I 2022-10-27 11:35:43] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:48 [I 2022-10-27 11:35:43] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00253 [I 2022-10-27 11:35:43] ...trap/engines/engine.py.230: loss: 0.15643 [I 2022-10-27 11:35:43] ...optimizers/trijoint.py.151: total_norm: 52.193810 [I 2022-10-27 11:35:48] ...trap/engines/engine.py.224: train: epoch 65 | batch 290/2304 [I 2022-10-27 11:35:48] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:45 [I 2022-10-27 11:35:48] ...trap/engines/engine.py.229: process: 0.56488 | load: 0.00254 [I 2022-10-27 11:35:48] ...trap/engines/engine.py.230: loss: 0.15354 [I 2022-10-27 11:35:48] ...optimizers/trijoint.py.151: total_norm: 32.491001 [I 2022-10-27 11:35:54] ...trap/engines/engine.py.224: train: epoch 65 | batch 300/2304 [I 2022-10-27 11:35:54] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:18:38 [I 2022-10-27 11:35:54] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00242 [I 2022-10-27 11:35:54] ...trap/engines/engine.py.230: loss: 0.16532 [I 2022-10-27 11:35:54] ...optimizers/trijoint.py.151: total_norm: 18.403875 [I 2022-10-27 11:35:59] ...trap/engines/engine.py.224: train: epoch 65 | batch 310/2304 [I 2022-10-27 11:35:59] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:30 [I 2022-10-27 11:35:59] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00244 [I 2022-10-27 11:35:59] ...trap/engines/engine.py.230: loss: 0.14762 [I 2022-10-27 11:35:59] ...optimizers/trijoint.py.151: total_norm: 17.762342 [I 2022-10-27 11:36:05] ...trap/engines/engine.py.224: train: epoch 65 | batch 320/2304 [I 2022-10-27 11:36:05] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:28 [I 2022-10-27 11:36:05] ...trap/engines/engine.py.229: process: 0.56533 | load: 0.00243 [I 2022-10-27 11:36:05] ...trap/engines/engine.py.230: loss: 0.15407 [I 2022-10-27 11:36:05] ...optimizers/trijoint.py.151: total_norm: 38.191406 [I 2022-10-27 11:36:11] ...trap/engines/engine.py.224: train: epoch 65 | batch 330/2304 [I 2022-10-27 11:36:11] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:37 [I 2022-10-27 11:36:11] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00252 [I 2022-10-27 11:36:11] ...trap/engines/engine.py.230: loss: 0.12747 [I 2022-10-27 11:36:11] ...optimizers/trijoint.py.151: total_norm: 9.225123 [I 2022-10-27 11:36:16] ...trap/engines/engine.py.224: train: epoch 65 | batch 340/2304 [I 2022-10-27 11:36:16] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:33 [I 2022-10-27 11:36:16] ...trap/engines/engine.py.229: process: 0.60025 | load: 0.00261 [I 2022-10-27 11:36:16] ...trap/engines/engine.py.230: loss: 0.12564 [I 2022-10-27 11:36:16] ...optimizers/trijoint.py.151: total_norm: 12.229641 [I 2022-10-27 11:36:22] ...trap/engines/engine.py.224: train: epoch 65 | batch 350/2304 [I 2022-10-27 11:36:22] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:14 [I 2022-10-27 11:36:22] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00246 [I 2022-10-27 11:36:22] ...trap/engines/engine.py.230: loss: 0.17801 [I 2022-10-27 11:36:22] ...optimizers/trijoint.py.151: total_norm: 26.702068 [I 2022-10-27 11:36:28] ...trap/engines/engine.py.224: train: epoch 65 | batch 360/2304 [I 2022-10-27 11:36:28] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:58 [I 2022-10-27 11:36:28] ...trap/engines/engine.py.229: process: 0.55315 | load: 0.00249 [I 2022-10-27 11:36:28] ...trap/engines/engine.py.230: loss: 0.15974 [I 2022-10-27 11:36:28] ...optimizers/trijoint.py.151: total_norm: 11.663345 [I 2022-10-27 11:36:33] ...trap/engines/engine.py.224: train: epoch 65 | batch 370/2304 [I 2022-10-27 11:36:33] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:59 [I 2022-10-27 11:36:33] ...trap/engines/engine.py.229: process: 0.56923 | load: 0.00245 [I 2022-10-27 11:36:33] ...trap/engines/engine.py.230: loss: 0.21089 [I 2022-10-27 11:36:33] ...optimizers/trijoint.py.151: total_norm: 41.125759 [I 2022-10-27 11:36:39] ...trap/engines/engine.py.224: train: epoch 65 | batch 380/2304 [I 2022-10-27 11:36:39] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:49 [I 2022-10-27 11:36:39] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00244 [I 2022-10-27 11:36:39] ...trap/engines/engine.py.230: loss: 0.14454 [I 2022-10-27 11:36:39] ...optimizers/trijoint.py.151: total_norm: 19.019493 [I 2022-10-27 11:36:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 390/2304 [I 2022-10-27 11:36:44] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:44 [I 2022-10-27 11:36:44] ...trap/engines/engine.py.229: process: 0.54883 | load: 0.00242 [I 2022-10-27 11:36:44] ...trap/engines/engine.py.230: loss: 0.12147 [I 2022-10-27 11:36:44] ...optimizers/trijoint.py.151: total_norm: 12.178189 [I 2022-10-27 11:36:50] ...trap/engines/engine.py.224: train: epoch 65 | batch 400/2304 [I 2022-10-27 11:36:50] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:49 [I 2022-10-27 11:36:50] ...trap/engines/engine.py.229: process: 0.56522 | load: 0.00254 [I 2022-10-27 11:36:50] ...trap/engines/engine.py.230: loss: 0.13361 [I 2022-10-27 11:36:50] ...optimizers/trijoint.py.151: total_norm: 11.157413 [I 2022-10-27 11:36:56] ...trap/engines/engine.py.224: train: epoch 65 | batch 410/2304 [I 2022-10-27 11:36:56] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:39 [I 2022-10-27 11:36:56] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00247 [I 2022-10-27 11:36:56] ...trap/engines/engine.py.230: loss: 0.11420 [I 2022-10-27 11:36:56] ...optimizers/trijoint.py.151: total_norm: 7.302754 [I 2022-10-27 11:37:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 420/2304 [I 2022-10-27 11:37:01] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:18:07 [I 2022-10-27 11:37:01] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00251 [I 2022-10-27 11:37:01] ...trap/engines/engine.py.230: loss: 0.12599 [I 2022-10-27 11:37:01] ...optimizers/trijoint.py.151: total_norm: 10.514752 [I 2022-10-27 11:37:07] ...trap/engines/engine.py.224: train: epoch 65 | batch 430/2304 [I 2022-10-27 11:37:07] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:30 [I 2022-10-27 11:37:07] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00262 [I 2022-10-27 11:37:07] ...trap/engines/engine.py.230: loss: 0.14230 [I 2022-10-27 11:37:07] ...optimizers/trijoint.py.151: total_norm: 19.502993 [I 2022-10-27 11:37:13] ...trap/engines/engine.py.224: train: epoch 65 | batch 440/2304 [I 2022-10-27 11:37:13] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:18 [I 2022-10-27 11:37:13] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00244 [I 2022-10-27 11:37:13] ...trap/engines/engine.py.230: loss: 0.10930 [I 2022-10-27 11:37:13] ...optimizers/trijoint.py.151: total_norm: 22.600340 [I 2022-10-27 11:37:18] ...trap/engines/engine.py.224: train: epoch 65 | batch 450/2304 [I 2022-10-27 11:37:18] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:04 [I 2022-10-27 11:37:18] ...trap/engines/engine.py.229: process: 0.55055 | load: 0.00242 [I 2022-10-27 11:37:18] ...trap/engines/engine.py.230: loss: 0.13894 [I 2022-10-27 11:37:18] ...optimizers/trijoint.py.151: total_norm: 17.978195 [I 2022-10-27 11:37:24] ...trap/engines/engine.py.224: train: epoch 65 | batch 460/2304 [I 2022-10-27 11:37:24] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:05 [I 2022-10-27 11:37:24] ...trap/engines/engine.py.229: process: 0.55191 | load: 0.00244 [I 2022-10-27 11:37:24] ...trap/engines/engine.py.230: loss: 0.12012 [I 2022-10-27 11:37:24] ...optimizers/trijoint.py.151: total_norm: 14.115009 [I 2022-10-27 11:37:29] ...trap/engines/engine.py.224: train: epoch 65 | batch 470/2304 [I 2022-10-27 11:37:29] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:57 [I 2022-10-27 11:37:29] ...trap/engines/engine.py.229: process: 0.54641 | load: 0.00240 [I 2022-10-27 11:37:29] ...trap/engines/engine.py.230: loss: 0.09559 [I 2022-10-27 11:37:29] ...optimizers/trijoint.py.151: total_norm: 7.839045 [I 2022-10-27 11:37:35] ...trap/engines/engine.py.224: train: epoch 65 | batch 480/2304 [I 2022-10-27 11:37:35] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:17:23 [I 2022-10-27 11:37:35] ...trap/engines/engine.py.229: process: 0.63185 | load: 0.00244 [I 2022-10-27 11:37:35] ...trap/engines/engine.py.230: loss: 0.15767 [I 2022-10-27 11:37:35] ...optimizers/trijoint.py.151: total_norm: 21.023264 [I 2022-10-27 11:37:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 490/2304 [I 2022-10-27 11:37:40] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:52 [I 2022-10-27 11:37:40] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00239 [I 2022-10-27 11:37:40] ...trap/engines/engine.py.230: loss: 0.10724 [I 2022-10-27 11:37:40] ...optimizers/trijoint.py.151: total_norm: 7.292695 [I 2022-10-27 11:37:46] ...trap/engines/engine.py.224: train: epoch 65 | batch 500/2304 [I 2022-10-27 11:37:46] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:33 [I 2022-10-27 11:37:46] ...trap/engines/engine.py.229: process: 0.53734 | load: 0.00242 [I 2022-10-27 11:37:46] ...trap/engines/engine.py.230: loss: 0.10654 [I 2022-10-27 11:37:46] ...optimizers/trijoint.py.151: total_norm: 7.231315 [I 2022-10-27 11:37:52] ...trap/engines/engine.py.224: train: epoch 65 | batch 510/2304 [I 2022-10-27 11:37:52] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:33 [I 2022-10-27 11:37:52] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00251 [I 2022-10-27 11:37:52] ...trap/engines/engine.py.230: loss: 0.16084 [I 2022-10-27 11:37:52] ...optimizers/trijoint.py.151: total_norm: 31.849495 [I 2022-10-27 11:37:57] ...trap/engines/engine.py.224: train: epoch 65 | batch 520/2304 [I 2022-10-27 11:37:57] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:31 [I 2022-10-27 11:37:57] ...trap/engines/engine.py.229: process: 0.55417 | load: 0.00240 [I 2022-10-27 11:37:57] ...trap/engines/engine.py.230: loss: 0.13309 [I 2022-10-27 11:37:57] ...optimizers/trijoint.py.151: total_norm: 7.900486 [I 2022-10-27 11:38:03] ...trap/engines/engine.py.224: train: epoch 65 | batch 530/2304 [I 2022-10-27 11:38:03] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:26 [I 2022-10-27 11:38:03] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00246 [I 2022-10-27 11:38:03] ...trap/engines/engine.py.230: loss: 0.13805 [I 2022-10-27 11:38:03] ...optimizers/trijoint.py.151: total_norm: 10.483904 [I 2022-10-27 11:38:08] ...trap/engines/engine.py.224: train: epoch 65 | batch 540/2304 [I 2022-10-27 11:38:08] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:20 [I 2022-10-27 11:38:08] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00242 [I 2022-10-27 11:38:08] ...trap/engines/engine.py.230: loss: 0.12744 [I 2022-10-27 11:38:08] ...optimizers/trijoint.py.151: total_norm: 15.076278 [I 2022-10-27 11:38:14] ...trap/engines/engine.py.224: train: epoch 65 | batch 550/2304 [I 2022-10-27 11:38:14] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:17 [I 2022-10-27 11:38:14] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00254 [I 2022-10-27 11:38:14] ...trap/engines/engine.py.230: loss: 0.17385 [I 2022-10-27 11:38:14] ...optimizers/trijoint.py.151: total_norm: 21.692883 [I 2022-10-27 11:38:20] ...trap/engines/engine.py.224: train: epoch 65 | batch 560/2304 [I 2022-10-27 11:38:20] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:11 [I 2022-10-27 11:38:20] ...trap/engines/engine.py.229: process: 0.55105 | load: 0.00250 [I 2022-10-27 11:38:20] ...trap/engines/engine.py.230: loss: 0.13399 [I 2022-10-27 11:38:20] ...optimizers/trijoint.py.151: total_norm: 12.018839 [I 2022-10-27 11:38:25] ...trap/engines/engine.py.224: train: epoch 65 | batch 570/2304 [I 2022-10-27 11:38:25] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:27 [I 2022-10-27 11:38:25] ...trap/engines/engine.py.229: process: 0.58047 | load: 0.00254 [I 2022-10-27 11:38:25] ...trap/engines/engine.py.230: loss: 0.16027 [I 2022-10-27 11:38:25] ...optimizers/trijoint.py.151: total_norm: 18.791819 [I 2022-10-27 11:38:31] ...trap/engines/engine.py.224: train: epoch 65 | batch 580/2304 [I 2022-10-27 11:38:31] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:10 [I 2022-10-27 11:38:31] ...trap/engines/engine.py.229: process: 0.56250 | load: 0.00243 [I 2022-10-27 11:38:31] ...trap/engines/engine.py.230: loss: 0.11609 [I 2022-10-27 11:38:31] ...optimizers/trijoint.py.151: total_norm: 13.592751 [I 2022-10-27 11:38:37] ...trap/engines/engine.py.224: train: epoch 65 | batch 590/2304 [I 2022-10-27 11:38:37] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:16:07 [I 2022-10-27 11:38:37] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00243 [I 2022-10-27 11:38:37] ...trap/engines/engine.py.230: loss: 0.13657 [I 2022-10-27 11:38:37] ...optimizers/trijoint.py.151: total_norm: 8.397191 [I 2022-10-27 11:38:42] ...trap/engines/engine.py.224: train: epoch 65 | batch 600/2304 [I 2022-10-27 11:38:42] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:16:00 [I 2022-10-27 11:38:42] ...trap/engines/engine.py.229: process: 0.55997 | load: 0.00245 [I 2022-10-27 11:38:42] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-27 11:38:42] ...optimizers/trijoint.py.151: total_norm: 18.912693 [I 2022-10-27 11:38:48] ...trap/engines/engine.py.224: train: epoch 65 | batch 610/2304 [I 2022-10-27 11:38:48] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:55 [I 2022-10-27 11:38:48] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00243 [I 2022-10-27 11:38:48] ...trap/engines/engine.py.230: loss: 0.13857 [I 2022-10-27 11:38:48] ...optimizers/trijoint.py.151: total_norm: 17.416363 [I 2022-10-27 11:38:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 620/2304 [I 2022-10-27 11:38:53] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:39 [I 2022-10-27 11:38:53] ...trap/engines/engine.py.229: process: 0.55414 | load: 0.00244 [I 2022-10-27 11:38:53] ...trap/engines/engine.py.230: loss: 0.11930 [I 2022-10-27 11:38:53] ...optimizers/trijoint.py.151: total_norm: 18.961311 [I 2022-10-27 11:38:59] ...trap/engines/engine.py.224: train: epoch 65 | batch 630/2304 [I 2022-10-27 11:38:59] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:24 [I 2022-10-27 11:38:59] ...trap/engines/engine.py.229: process: 0.53951 | load: 0.00243 [I 2022-10-27 11:38:59] ...trap/engines/engine.py.230: loss: 0.16472 [I 2022-10-27 11:38:59] ...optimizers/trijoint.py.151: total_norm: 10.878976 [I 2022-10-27 11:39:05] ...trap/engines/engine.py.224: train: epoch 65 | batch 640/2304 [I 2022-10-27 11:39:05] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:27 [I 2022-10-27 11:39:05] ...trap/engines/engine.py.229: process: 0.54802 | load: 0.00245 [I 2022-10-27 11:39:05] ...trap/engines/engine.py.230: loss: 0.17157 [I 2022-10-27 11:39:05] ...optimizers/trijoint.py.151: total_norm: 31.765985 [I 2022-10-27 11:39:10] ...trap/engines/engine.py.224: train: epoch 65 | batch 650/2304 [I 2022-10-27 11:39:10] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:25 [I 2022-10-27 11:39:10] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00246 [I 2022-10-27 11:39:10] ...trap/engines/engine.py.230: loss: 0.18927 [I 2022-10-27 11:39:10] ...optimizers/trijoint.py.151: total_norm: 13.164315 [I 2022-10-27 11:39:16] ...trap/engines/engine.py.224: train: epoch 65 | batch 660/2304 [I 2022-10-27 11:39:16] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:23 [I 2022-10-27 11:39:16] ...trap/engines/engine.py.229: process: 0.54902 | load: 0.00246 [I 2022-10-27 11:39:16] ...trap/engines/engine.py.230: loss: 0.15166 [I 2022-10-27 11:39:16] ...optimizers/trijoint.py.151: total_norm: 13.967997 [I 2022-10-27 11:39:21] ...trap/engines/engine.py.224: train: epoch 65 | batch 670/2304 [I 2022-10-27 11:39:21] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:16 [I 2022-10-27 11:39:21] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00251 [I 2022-10-27 11:39:21] ...trap/engines/engine.py.230: loss: 0.13748 [I 2022-10-27 11:39:21] ...optimizers/trijoint.py.151: total_norm: 18.998278 [I 2022-10-27 11:39:27] ...trap/engines/engine.py.224: train: epoch 65 | batch 680/2304 [I 2022-10-27 11:39:27] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:03 [I 2022-10-27 11:39:27] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00246 [I 2022-10-27 11:39:27] ...trap/engines/engine.py.230: loss: 0.17781 [I 2022-10-27 11:39:27] ...optimizers/trijoint.py.151: total_norm: 14.275779 [I 2022-10-27 11:39:33] ...trap/engines/engine.py.224: train: epoch 65 | batch 690/2304 [I 2022-10-27 11:39:33] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:03 [I 2022-10-27 11:39:33] ...trap/engines/engine.py.229: process: 0.56382 | load: 0.00247 [I 2022-10-27 11:39:33] ...trap/engines/engine.py.230: loss: 0.13734 [I 2022-10-27 11:39:33] ...optimizers/trijoint.py.151: total_norm: 20.665174 [I 2022-10-27 11:39:38] ...trap/engines/engine.py.224: train: epoch 65 | batch 700/2304 [I 2022-10-27 11:39:38] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:15:34 [I 2022-10-27 11:39:38] ...trap/engines/engine.py.229: process: 0.69220 | load: 0.00265 [I 2022-10-27 11:39:38] ...trap/engines/engine.py.230: loss: 0.18626 [I 2022-10-27 11:39:38] ...optimizers/trijoint.py.151: total_norm: 15.034431 [I 2022-10-27 11:39:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 710/2304 [I 2022-10-27 11:39:44] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:50 [I 2022-10-27 11:39:44] ...trap/engines/engine.py.229: process: 0.56326 | load: 0.00248 [I 2022-10-27 11:39:44] ...trap/engines/engine.py.230: loss: 0.11950 [I 2022-10-27 11:39:44] ...optimizers/trijoint.py.151: total_norm: 13.461740 [I 2022-10-27 11:39:50] ...trap/engines/engine.py.224: train: epoch 65 | batch 720/2304 [I 2022-10-27 11:39:50] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:42 [I 2022-10-27 11:39:50] ...trap/engines/engine.py.229: process: 0.55542 | load: 0.00247 [I 2022-10-27 11:39:50] ...trap/engines/engine.py.230: loss: 0.13725 [I 2022-10-27 11:39:50] ...optimizers/trijoint.py.151: total_norm: 16.497580 [I 2022-10-27 11:39:55] ...trap/engines/engine.py.224: train: epoch 65 | batch 730/2304 [I 2022-10-27 11:39:55] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:38 [I 2022-10-27 11:39:55] ...trap/engines/engine.py.229: process: 0.57467 | load: 0.00242 [I 2022-10-27 11:39:55] ...trap/engines/engine.py.230: loss: 0.12006 [I 2022-10-27 11:39:55] ...optimizers/trijoint.py.151: total_norm: 17.601408 [I 2022-10-27 11:40:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 740/2304 [I 2022-10-27 11:40:01] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:25 [I 2022-10-27 11:40:01] ...trap/engines/engine.py.229: process: 0.54781 | load: 0.00244 [I 2022-10-27 11:40:01] ...trap/engines/engine.py.230: loss: 0.10971 [I 2022-10-27 11:40:01] ...optimizers/trijoint.py.151: total_norm: 12.344912 [I 2022-10-27 11:40:06] ...trap/engines/engine.py.224: train: epoch 65 | batch 750/2304 [I 2022-10-27 11:40:06] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:26 [I 2022-10-27 11:40:06] ...trap/engines/engine.py.229: process: 0.56695 | load: 0.00242 [I 2022-10-27 11:40:06] ...trap/engines/engine.py.230: loss: 0.11725 [I 2022-10-27 11:40:06] ...optimizers/trijoint.py.151: total_norm: 17.070364 [I 2022-10-27 11:40:12] ...trap/engines/engine.py.224: train: epoch 65 | batch 760/2304 [I 2022-10-27 11:40:12] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:19 [I 2022-10-27 11:40:12] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00242 [I 2022-10-27 11:40:12] ...trap/engines/engine.py.230: loss: 0.19165 [I 2022-10-27 11:40:12] ...optimizers/trijoint.py.151: total_norm: 55.989643 [I 2022-10-27 11:40:17] ...trap/engines/engine.py.224: train: epoch 65 | batch 770/2304 [I 2022-10-27 11:40:17] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:06 [I 2022-10-27 11:40:17] ...trap/engines/engine.py.229: process: 0.54132 | load: 0.00240 [I 2022-10-27 11:40:17] ...trap/engines/engine.py.230: loss: 0.08681 [I 2022-10-27 11:40:17] ...optimizers/trijoint.py.151: total_norm: 13.222825 [I 2022-10-27 11:40:23] ...trap/engines/engine.py.224: train: epoch 65 | batch 780/2304 [I 2022-10-27 11:40:23] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:00 [I 2022-10-27 11:40:23] ...trap/engines/engine.py.229: process: 0.54616 | load: 0.00239 [I 2022-10-27 11:40:23] ...trap/engines/engine.py.230: loss: 0.09280 [I 2022-10-27 11:40:23] ...optimizers/trijoint.py.151: total_norm: 10.317299 [I 2022-10-27 11:40:28] ...trap/engines/engine.py.224: train: epoch 65 | batch 790/2304 [I 2022-10-27 11:40:28] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:01 [I 2022-10-27 11:40:28] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00242 [I 2022-10-27 11:40:28] ...trap/engines/engine.py.230: loss: 0.14694 [I 2022-10-27 11:40:28] ...optimizers/trijoint.py.151: total_norm: 11.552406 [I 2022-10-27 11:40:34] ...trap/engines/engine.py.224: train: epoch 65 | batch 800/2304 [I 2022-10-27 11:40:34] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:51 [I 2022-10-27 11:40:34] ...trap/engines/engine.py.229: process: 0.54478 | load: 0.00267 [I 2022-10-27 11:40:34] ...trap/engines/engine.py.230: loss: 0.15465 [I 2022-10-27 11:40:34] ...optimizers/trijoint.py.151: total_norm: 11.218145 [I 2022-10-27 11:40:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 810/2304 [I 2022-10-27 11:40:40] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:52 [I 2022-10-27 11:40:40] ...trap/engines/engine.py.229: process: 0.57028 | load: 0.00261 [I 2022-10-27 11:40:40] ...trap/engines/engine.py.230: loss: 0.18741 [I 2022-10-27 11:40:40] ...optimizers/trijoint.py.151: total_norm: 29.823597 [I 2022-10-27 11:40:45] ...trap/engines/engine.py.224: train: epoch 65 | batch 820/2304 [I 2022-10-27 11:40:45] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:53 [I 2022-10-27 11:40:45] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00246 [I 2022-10-27 11:40:45] ...trap/engines/engine.py.230: loss: 0.15579 [I 2022-10-27 11:40:45] ...optimizers/trijoint.py.151: total_norm: 44.061428 [I 2022-10-27 11:40:51] ...trap/engines/engine.py.224: train: epoch 65 | batch 830/2304 [I 2022-10-27 11:40:51] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:45 [I 2022-10-27 11:40:51] ...trap/engines/engine.py.229: process: 0.56808 | load: 0.00246 [I 2022-10-27 11:40:51] ...trap/engines/engine.py.230: loss: 0.11590 [I 2022-10-27 11:40:51] ...optimizers/trijoint.py.151: total_norm: 13.231012 [I 2022-10-27 11:40:56] ...trap/engines/engine.py.224: train: epoch 65 | batch 840/2304 [I 2022-10-27 11:40:56] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:36 [I 2022-10-27 11:40:56] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00243 [I 2022-10-27 11:40:56] ...trap/engines/engine.py.230: loss: 0.15851 [I 2022-10-27 11:40:56] ...optimizers/trijoint.py.151: total_norm: 40.887390 [I 2022-10-27 11:41:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 850/2304 [I 2022-10-27 11:41:02] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:27 [I 2022-10-27 11:41:02] ...trap/engines/engine.py.229: process: 0.55130 | load: 0.00242 [I 2022-10-27 11:41:02] ...trap/engines/engine.py.230: loss: 0.10896 [I 2022-10-27 11:41:02] ...optimizers/trijoint.py.151: total_norm: 12.837490 [I 2022-10-27 11:41:08] ...trap/engines/engine.py.224: train: epoch 65 | batch 860/2304 [I 2022-10-27 11:41:08] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:19 [I 2022-10-27 11:41:08] ...trap/engines/engine.py.229: process: 0.55893 | load: 0.00242 [I 2022-10-27 11:41:08] ...trap/engines/engine.py.230: loss: 0.09359 [I 2022-10-27 11:41:08] ...optimizers/trijoint.py.151: total_norm: 9.283936 [I 2022-10-27 11:41:13] ...trap/engines/engine.py.224: train: epoch 65 | batch 870/2304 [I 2022-10-27 11:41:13] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:17 [I 2022-10-27 11:41:13] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00241 [I 2022-10-27 11:41:13] ...trap/engines/engine.py.230: loss: 0.15044 [I 2022-10-27 11:41:13] ...optimizers/trijoint.py.151: total_norm: 22.179785 [I 2022-10-27 11:41:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 880/2304 [I 2022-10-27 11:41:19] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:11 [I 2022-10-27 11:41:19] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00242 [I 2022-10-27 11:41:19] ...trap/engines/engine.py.230: loss: 0.11091 [I 2022-10-27 11:41:19] ...optimizers/trijoint.py.151: total_norm: 12.500959 [I 2022-10-27 11:41:24] ...trap/engines/engine.py.224: train: epoch 65 | batch 890/2304 [I 2022-10-27 11:41:24] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:00 [I 2022-10-27 11:41:24] ...trap/engines/engine.py.229: process: 0.54835 | load: 0.00244 [I 2022-10-27 11:41:24] ...trap/engines/engine.py.230: loss: 0.13722 [I 2022-10-27 11:41:24] ...optimizers/trijoint.py.151: total_norm: 14.177244 [I 2022-10-27 11:41:30] ...trap/engines/engine.py.224: train: epoch 65 | batch 900/2304 [I 2022-10-27 11:41:30] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:00 [I 2022-10-27 11:41:30] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00241 [I 2022-10-27 11:41:30] ...trap/engines/engine.py.230: loss: 0.16962 [I 2022-10-27 11:41:30] ...optimizers/trijoint.py.151: total_norm: 9.730685 [I 2022-10-27 11:41:35] ...trap/engines/engine.py.224: train: epoch 65 | batch 910/2304 [I 2022-10-27 11:41:35] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:13:02 [I 2022-10-27 11:41:35] ...trap/engines/engine.py.229: process: 0.56668 | load: 0.00244 [I 2022-10-27 11:41:35] ...trap/engines/engine.py.230: loss: 0.09628 [I 2022-10-27 11:41:35] ...optimizers/trijoint.py.151: total_norm: 11.661469 [I 2022-10-27 11:41:41] ...trap/engines/engine.py.224: train: epoch 65 | batch 920/2304 [I 2022-10-27 11:41:41] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:49 [I 2022-10-27 11:41:41] ...trap/engines/engine.py.229: process: 0.55413 | load: 0.00250 [I 2022-10-27 11:41:41] ...trap/engines/engine.py.230: loss: 0.10491 [I 2022-10-27 11:41:41] ...optimizers/trijoint.py.151: total_norm: 12.002198 [I 2022-10-27 11:41:47] ...trap/engines/engine.py.224: train: epoch 65 | batch 930/2304 [I 2022-10-27 11:41:47] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:43 [I 2022-10-27 11:41:47] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00247 [I 2022-10-27 11:41:47] ...trap/engines/engine.py.230: loss: 0.18691 [I 2022-10-27 11:41:47] ...optimizers/trijoint.py.151: total_norm: 34.331886 [I 2022-10-27 11:41:52] ...trap/engines/engine.py.224: train: epoch 65 | batch 940/2304 [I 2022-10-27 11:41:52] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:39 [I 2022-10-27 11:41:52] ...trap/engines/engine.py.229: process: 0.56238 | load: 0.00243 [I 2022-10-27 11:41:52] ...trap/engines/engine.py.230: loss: 0.13938 [I 2022-10-27 11:41:52] ...optimizers/trijoint.py.151: total_norm: 22.837297 [I 2022-10-27 11:41:58] ...trap/engines/engine.py.224: train: epoch 65 | batch 950/2304 [I 2022-10-27 11:41:58] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:35 [I 2022-10-27 11:41:58] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00244 [I 2022-10-27 11:41:58] ...trap/engines/engine.py.230: loss: 0.13256 [I 2022-10-27 11:41:58] ...optimizers/trijoint.py.151: total_norm: 24.620886 [I 2022-10-27 11:42:03] ...trap/engines/engine.py.224: train: epoch 65 | batch 960/2304 [I 2022-10-27 11:42:03] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:26 [I 2022-10-27 11:42:03] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00241 [I 2022-10-27 11:42:03] ...trap/engines/engine.py.230: loss: 0.14836 [I 2022-10-27 11:42:03] ...optimizers/trijoint.py.151: total_norm: 13.237926 [I 2022-10-27 11:42:09] ...trap/engines/engine.py.224: train: epoch 65 | batch 970/2304 [I 2022-10-27 11:42:09] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:21 [I 2022-10-27 11:42:09] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00243 [I 2022-10-27 11:42:09] ...trap/engines/engine.py.230: loss: 0.15457 [I 2022-10-27 11:42:09] ...optimizers/trijoint.py.151: total_norm: 13.481297 [I 2022-10-27 11:42:15] ...trap/engines/engine.py.224: train: epoch 65 | batch 980/2304 [I 2022-10-27 11:42:15] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:19 [I 2022-10-27 11:42:15] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00241 [I 2022-10-27 11:42:15] ...trap/engines/engine.py.230: loss: 0.15491 [I 2022-10-27 11:42:15] ...optimizers/trijoint.py.151: total_norm: 8.852289 [I 2022-10-27 11:42:20] ...trap/engines/engine.py.224: train: epoch 65 | batch 990/2304 [I 2022-10-27 11:42:20] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:09 [I 2022-10-27 11:42:20] ...trap/engines/engine.py.229: process: 0.54614 | load: 0.00243 [I 2022-10-27 11:42:20] ...trap/engines/engine.py.230: loss: 0.18910 [I 2022-10-27 11:42:20] ...optimizers/trijoint.py.151: total_norm: 20.234146 [I 2022-10-27 11:42:26] ...trap/engines/engine.py.224: train: epoch 65 | batch 1000/2304 [I 2022-10-27 11:42:26] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:05 [I 2022-10-27 11:42:26] ...trap/engines/engine.py.229: process: 0.54886 | load: 0.00240 [I 2022-10-27 11:42:26] ...trap/engines/engine.py.230: loss: 0.11011 [I 2022-10-27 11:42:26] ...optimizers/trijoint.py.151: total_norm: 10.585596 [I 2022-10-27 11:42:31] ...trap/engines/engine.py.224: train: epoch 65 | batch 1010/2304 [I 2022-10-27 11:42:31] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:03 [I 2022-10-27 11:42:31] ...trap/engines/engine.py.229: process: 0.56265 | load: 0.00241 [I 2022-10-27 11:42:31] ...trap/engines/engine.py.230: loss: 0.23917 [I 2022-10-27 11:42:31] ...optimizers/trijoint.py.151: total_norm: 13.692237 [I 2022-10-27 11:42:37] ...trap/engines/engine.py.224: train: epoch 65 | batch 1020/2304 [I 2022-10-27 11:42:37] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:53 [I 2022-10-27 11:42:37] ...trap/engines/engine.py.229: process: 0.55505 | load: 0.00240 [I 2022-10-27 11:42:37] ...trap/engines/engine.py.230: loss: 0.14648 [I 2022-10-27 11:42:37] ...optimizers/trijoint.py.151: total_norm: 23.316736 [I 2022-10-27 11:42:42] ...trap/engines/engine.py.224: train: epoch 65 | batch 1030/2304 [I 2022-10-27 11:42:42] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:51 [I 2022-10-27 11:42:42] ...trap/engines/engine.py.229: process: 0.56367 | load: 0.00248 [I 2022-10-27 11:42:42] ...trap/engines/engine.py.230: loss: 0.16653 [I 2022-10-27 11:42:42] ...optimizers/trijoint.py.151: total_norm: 12.530552 [I 2022-10-27 11:42:48] ...trap/engines/engine.py.224: train: epoch 65 | batch 1040/2304 [I 2022-10-27 11:42:48] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:42 [I 2022-10-27 11:42:48] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00249 [I 2022-10-27 11:42:48] ...trap/engines/engine.py.230: loss: 0.18820 [I 2022-10-27 11:42:48] ...optimizers/trijoint.py.151: total_norm: 24.259647 [I 2022-10-27 11:42:54] ...trap/engines/engine.py.224: train: epoch 65 | batch 1050/2304 [I 2022-10-27 11:42:54] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:37 [I 2022-10-27 11:42:54] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00248 [I 2022-10-27 11:42:54] ...trap/engines/engine.py.230: loss: 0.12724 [I 2022-10-27 11:42:54] ...optimizers/trijoint.py.151: total_norm: 14.382631 [I 2022-10-27 11:42:59] ...trap/engines/engine.py.224: train: epoch 65 | batch 1060/2304 [I 2022-10-27 11:42:59] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:29 [I 2022-10-27 11:42:59] ...trap/engines/engine.py.229: process: 0.55161 | load: 0.00243 [I 2022-10-27 11:42:59] ...trap/engines/engine.py.230: loss: 0.11340 [I 2022-10-27 11:42:59] ...optimizers/trijoint.py.151: total_norm: 15.079882 [I 2022-10-27 11:43:05] ...trap/engines/engine.py.224: train: epoch 65 | batch 1070/2304 [I 2022-10-27 11:43:05] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:22 [I 2022-10-27 11:43:05] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00242 [I 2022-10-27 11:43:05] ...trap/engines/engine.py.230: loss: 0.10988 [I 2022-10-27 11:43:05] ...optimizers/trijoint.py.151: total_norm: 7.018426 [I 2022-10-27 11:43:10] ...trap/engines/engine.py.224: train: epoch 65 | batch 1080/2304 [I 2022-10-27 11:43:10] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:22 [I 2022-10-27 11:43:10] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00241 [I 2022-10-27 11:43:10] ...trap/engines/engine.py.230: loss: 0.09359 [I 2022-10-27 11:43:10] ...optimizers/trijoint.py.151: total_norm: 8.134254 [I 2022-10-27 11:43:16] ...trap/engines/engine.py.224: train: epoch 65 | batch 1090/2304 [I 2022-10-27 11:43:16] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:16 [I 2022-10-27 11:43:16] ...trap/engines/engine.py.229: process: 0.55124 | load: 0.00245 [I 2022-10-27 11:43:16] ...trap/engines/engine.py.230: loss: 0.10143 [I 2022-10-27 11:43:16] ...optimizers/trijoint.py.151: total_norm: 12.558851 [I 2022-10-27 11:43:21] ...trap/engines/engine.py.224: train: epoch 65 | batch 1100/2304 [I 2022-10-27 11:43:21] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:12 [I 2022-10-27 11:43:21] ...trap/engines/engine.py.229: process: 0.55460 | load: 0.00241 [I 2022-10-27 11:43:21] ...trap/engines/engine.py.230: loss: 0.12524 [I 2022-10-27 11:43:21] ...optimizers/trijoint.py.151: total_norm: 9.750414 [I 2022-10-27 11:43:27] ...trap/engines/engine.py.224: train: epoch 65 | batch 1110/2304 [I 2022-10-27 11:43:27] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:00 [I 2022-10-27 11:43:27] ...trap/engines/engine.py.229: process: 0.54841 | load: 0.00243 [I 2022-10-27 11:43:27] ...trap/engines/engine.py.230: loss: 0.22940 [I 2022-10-27 11:43:27] ...optimizers/trijoint.py.151: total_norm: 16.138231 [I 2022-10-27 11:43:33] ...trap/engines/engine.py.224: train: epoch 65 | batch 1120/2304 [I 2022-10-27 11:43:33] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:10:59 [I 2022-10-27 11:43:33] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00244 [I 2022-10-27 11:43:33] ...trap/engines/engine.py.230: loss: 0.16965 [I 2022-10-27 11:43:33] ...optimizers/trijoint.py.151: total_norm: 17.289822 [I 2022-10-27 11:43:38] ...trap/engines/engine.py.224: train: epoch 65 | batch 1130/2304 [I 2022-10-27 11:43:38] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:52 [I 2022-10-27 11:43:38] ...trap/engines/engine.py.229: process: 0.55325 | load: 0.00247 [I 2022-10-27 11:43:38] ...trap/engines/engine.py.230: loss: 0.22746 [I 2022-10-27 11:43:38] ...optimizers/trijoint.py.151: total_norm: 35.597141 [I 2022-10-27 11:43:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 1140/2304 [I 2022-10-27 11:43:44] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:48 [I 2022-10-27 11:43:44] ...trap/engines/engine.py.229: process: 0.56646 | load: 0.00244 [I 2022-10-27 11:43:44] ...trap/engines/engine.py.230: loss: 0.18388 [I 2022-10-27 11:43:44] ...optimizers/trijoint.py.151: total_norm: 17.935999 [I 2022-10-27 11:43:49] ...trap/engines/engine.py.224: train: epoch 65 | batch 1150/2304 [I 2022-10-27 11:43:49] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:44 [I 2022-10-27 11:43:49] ...trap/engines/engine.py.229: process: 0.57284 | load: 0.00244 [I 2022-10-27 11:43:49] ...trap/engines/engine.py.230: loss: 0.14595 [I 2022-10-27 11:43:49] ...optimizers/trijoint.py.151: total_norm: 12.874564 [I 2022-10-27 11:43:55] ...trap/engines/engine.py.224: train: epoch 65 | batch 1160/2304 [I 2022-10-27 11:43:55] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:42 [I 2022-10-27 11:43:55] ...trap/engines/engine.py.229: process: 0.56711 | load: 0.00245 [I 2022-10-27 11:43:55] ...trap/engines/engine.py.230: loss: 0.13790 [I 2022-10-27 11:43:55] ...optimizers/trijoint.py.151: total_norm: 11.891031 [I 2022-10-27 11:44:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 1170/2304 [I 2022-10-27 11:44:01] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:40 [I 2022-10-27 11:44:01] ...trap/engines/engine.py.229: process: 0.56967 | load: 0.00245 [I 2022-10-27 11:44:01] ...trap/engines/engine.py.230: loss: 0.13459 [I 2022-10-27 11:44:01] ...optimizers/trijoint.py.151: total_norm: 15.676398 [I 2022-10-27 11:44:06] ...trap/engines/engine.py.224: train: epoch 65 | batch 1180/2304 [I 2022-10-27 11:44:06] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:26 [I 2022-10-27 11:44:06] ...trap/engines/engine.py.229: process: 0.54596 | load: 0.00244 [I 2022-10-27 11:44:06] ...trap/engines/engine.py.230: loss: 0.15724 [I 2022-10-27 11:44:06] ...optimizers/trijoint.py.151: total_norm: 33.375393 [I 2022-10-27 11:44:12] ...trap/engines/engine.py.224: train: epoch 65 | batch 1190/2304 [I 2022-10-27 11:44:12] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:19 [I 2022-10-27 11:44:12] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00246 [I 2022-10-27 11:44:12] ...trap/engines/engine.py.230: loss: 0.13958 [I 2022-10-27 11:44:12] ...optimizers/trijoint.py.151: total_norm: 9.976785 [I 2022-10-27 11:44:17] ...trap/engines/engine.py.224: train: epoch 65 | batch 1200/2304 [I 2022-10-27 11:44:17] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:13 [I 2022-10-27 11:44:17] ...trap/engines/engine.py.229: process: 0.55241 | load: 0.00247 [I 2022-10-27 11:44:17] ...trap/engines/engine.py.230: loss: 0.10606 [I 2022-10-27 11:44:17] ...optimizers/trijoint.py.151: total_norm: 13.390436 [I 2022-10-27 11:44:23] ...trap/engines/engine.py.224: train: epoch 65 | batch 1210/2304 [I 2022-10-27 11:44:23] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:10 [I 2022-10-27 11:44:23] ...trap/engines/engine.py.229: process: 0.56226 | load: 0.00242 [I 2022-10-27 11:44:23] ...trap/engines/engine.py.230: loss: 0.14889 [I 2022-10-27 11:44:23] ...optimizers/trijoint.py.151: total_norm: 15.896680 [I 2022-10-27 11:44:28] ...trap/engines/engine.py.224: train: epoch 65 | batch 1220/2304 [I 2022-10-27 11:44:28] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:09:59 [I 2022-10-27 11:44:28] ...trap/engines/engine.py.229: process: 0.55158 | load: 0.00243 [I 2022-10-27 11:44:28] ...trap/engines/engine.py.230: loss: 0.12409 [I 2022-10-27 11:44:28] ...optimizers/trijoint.py.151: total_norm: 21.356970 [I 2022-10-27 11:44:34] ...trap/engines/engine.py.224: train: epoch 65 | batch 1230/2304 [I 2022-10-27 11:44:34] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:55 [I 2022-10-27 11:44:34] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00244 [I 2022-10-27 11:44:34] ...trap/engines/engine.py.230: loss: 0.11637 [I 2022-10-27 11:44:34] ...optimizers/trijoint.py.151: total_norm: 21.307838 [I 2022-10-27 11:44:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 1240/2304 [I 2022-10-27 11:44:40] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:51 [I 2022-10-27 11:44:40] ...trap/engines/engine.py.229: process: 0.54877 | load: 0.00243 [I 2022-10-27 11:44:40] ...trap/engines/engine.py.230: loss: 0.10785 [I 2022-10-27 11:44:40] ...optimizers/trijoint.py.151: total_norm: 9.404845 [I 2022-10-27 11:44:45] ...trap/engines/engine.py.224: train: epoch 65 | batch 1250/2304 [I 2022-10-27 11:44:45] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:44 [I 2022-10-27 11:44:45] ...trap/engines/engine.py.229: process: 0.54840 | load: 0.00240 [I 2022-10-27 11:44:45] ...trap/engines/engine.py.230: loss: 0.12339 [I 2022-10-27 11:44:45] ...optimizers/trijoint.py.151: total_norm: 13.204701 [I 2022-10-27 11:44:51] ...trap/engines/engine.py.224: train: epoch 65 | batch 1260/2304 [I 2022-10-27 11:44:51] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:34 [I 2022-10-27 11:44:51] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00242 [I 2022-10-27 11:44:51] ...trap/engines/engine.py.230: loss: 0.17404 [I 2022-10-27 11:44:51] ...optimizers/trijoint.py.151: total_norm: 14.435422 [I 2022-10-27 11:44:56] ...trap/engines/engine.py.224: train: epoch 65 | batch 1270/2304 [I 2022-10-27 11:44:56] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:36 [I 2022-10-27 11:44:56] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00241 [I 2022-10-27 11:44:56] ...trap/engines/engine.py.230: loss: 0.11590 [I 2022-10-27 11:44:56] ...optimizers/trijoint.py.151: total_norm: 17.337170 [I 2022-10-27 11:45:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 1280/2304 [I 2022-10-27 11:45:02] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:27 [I 2022-10-27 11:45:02] ...trap/engines/engine.py.229: process: 0.55102 | load: 0.00242 [I 2022-10-27 11:45:02] ...trap/engines/engine.py.230: loss: 0.13461 [I 2022-10-27 11:45:02] ...optimizers/trijoint.py.151: total_norm: 16.817554 [I 2022-10-27 11:45:07] ...trap/engines/engine.py.224: train: epoch 65 | batch 1290/2304 [I 2022-10-27 11:45:07] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:26 [I 2022-10-27 11:45:07] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00240 [I 2022-10-27 11:45:07] ...trap/engines/engine.py.230: loss: 0.10807 [I 2022-10-27 11:45:07] ...optimizers/trijoint.py.151: total_norm: 9.835809 [I 2022-10-27 11:45:13] ...trap/engines/engine.py.224: train: epoch 65 | batch 1300/2304 [I 2022-10-27 11:45:13] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:16 [I 2022-10-27 11:45:13] ...trap/engines/engine.py.229: process: 0.54182 | load: 0.00241 [I 2022-10-27 11:45:13] ...trap/engines/engine.py.230: loss: 0.14475 [I 2022-10-27 11:45:13] ...optimizers/trijoint.py.151: total_norm: 16.120441 [I 2022-10-27 11:45:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1310/2304 [I 2022-10-27 11:45:19] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:08 [I 2022-10-27 11:45:19] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00243 [I 2022-10-27 11:45:19] ...trap/engines/engine.py.230: loss: 0.16386 [I 2022-10-27 11:45:19] ...optimizers/trijoint.py.151: total_norm: 15.658535 [I 2022-10-27 11:45:24] ...trap/engines/engine.py.224: train: epoch 65 | batch 1320/2304 [I 2022-10-27 11:45:24] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:04 [I 2022-10-27 11:45:24] ...trap/engines/engine.py.229: process: 0.54560 | load: 0.00242 [I 2022-10-27 11:45:24] ...trap/engines/engine.py.230: loss: 0.14992 [I 2022-10-27 11:45:24] ...optimizers/trijoint.py.151: total_norm: 24.135387 [I 2022-10-27 11:45:30] ...trap/engines/engine.py.224: train: epoch 65 | batch 1330/2304 [I 2022-10-27 11:45:30] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:02 [I 2022-10-27 11:45:30] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00243 [I 2022-10-27 11:45:30] ...trap/engines/engine.py.230: loss: 0.13730 [I 2022-10-27 11:45:30] ...optimizers/trijoint.py.151: total_norm: 18.421629 [I 2022-10-27 11:45:35] ...trap/engines/engine.py.224: train: epoch 65 | batch 1340/2304 [I 2022-10-27 11:45:35] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:59 [I 2022-10-27 11:45:35] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00251 [I 2022-10-27 11:45:35] ...trap/engines/engine.py.230: loss: 0.10261 [I 2022-10-27 11:45:35] ...optimizers/trijoint.py.151: total_norm: 13.718973 [I 2022-10-27 11:45:41] ...trap/engines/engine.py.224: train: epoch 65 | batch 1350/2304 [I 2022-10-27 11:45:41] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:50 [I 2022-10-27 11:45:41] ...trap/engines/engine.py.229: process: 0.55102 | load: 0.00248 [I 2022-10-27 11:45:41] ...trap/engines/engine.py.230: loss: 0.09714 [I 2022-10-27 11:45:41] ...optimizers/trijoint.py.151: total_norm: 12.724204 [I 2022-10-27 11:45:46] ...trap/engines/engine.py.224: train: epoch 65 | batch 1360/2304 [I 2022-10-27 11:45:46] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:44 [I 2022-10-27 11:45:46] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00247 [I 2022-10-27 11:45:46] ...trap/engines/engine.py.230: loss: 0.09991 [I 2022-10-27 11:45:46] ...optimizers/trijoint.py.151: total_norm: 20.428629 [I 2022-10-27 11:45:52] ...trap/engines/engine.py.224: train: epoch 65 | batch 1370/2304 [I 2022-10-27 11:45:52] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:37 [I 2022-10-27 11:45:52] ...trap/engines/engine.py.229: process: 0.55201 | load: 0.00245 [I 2022-10-27 11:45:52] ...trap/engines/engine.py.230: loss: 0.10803 [I 2022-10-27 11:45:52] ...optimizers/trijoint.py.151: total_norm: 8.142737 [I 2022-10-27 11:45:58] ...trap/engines/engine.py.224: train: epoch 65 | batch 1380/2304 [I 2022-10-27 11:45:58] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:34 [I 2022-10-27 11:45:58] ...trap/engines/engine.py.229: process: 0.54946 | load: 0.00243 [I 2022-10-27 11:45:58] ...trap/engines/engine.py.230: loss: 0.14403 [I 2022-10-27 11:45:58] ...optimizers/trijoint.py.151: total_norm: 14.583802 [I 2022-10-27 11:46:03] ...trap/engines/engine.py.224: train: epoch 65 | batch 1390/2304 [I 2022-10-27 11:46:03] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:28 [I 2022-10-27 11:46:03] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00244 [I 2022-10-27 11:46:03] ...trap/engines/engine.py.230: loss: 0.10885 [I 2022-10-27 11:46:03] ...optimizers/trijoint.py.151: total_norm: 15.854987 [I 2022-10-27 11:46:09] ...trap/engines/engine.py.224: train: epoch 65 | batch 1400/2304 [I 2022-10-27 11:46:09] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:22 [I 2022-10-27 11:46:09] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00247 [I 2022-10-27 11:46:09] ...trap/engines/engine.py.230: loss: 0.11777 [I 2022-10-27 11:46:09] ...optimizers/trijoint.py.151: total_norm: 18.962994 [I 2022-10-27 11:46:15] ...trap/engines/engine.py.224: train: epoch 65 | batch 1410/2304 [I 2022-10-27 11:46:15] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:31 [I 2022-10-27 11:46:15] ...trap/engines/engine.py.229: process: 0.56582 | load: 0.00244 [I 2022-10-27 11:46:15] ...trap/engines/engine.py.230: loss: 0.21355 [I 2022-10-27 11:46:15] ...optimizers/trijoint.py.151: total_norm: 25.117136 [I 2022-10-27 11:46:20] ...trap/engines/engine.py.224: train: epoch 65 | batch 1420/2304 [I 2022-10-27 11:46:20] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:29 [I 2022-10-27 11:46:20] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00241 [I 2022-10-27 11:46:20] ...trap/engines/engine.py.230: loss: 0.16899 [I 2022-10-27 11:46:20] ...optimizers/trijoint.py.151: total_norm: 21.553469 [I 2022-10-27 11:46:26] ...trap/engines/engine.py.224: train: epoch 65 | batch 1430/2304 [I 2022-10-27 11:46:26] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:09 [I 2022-10-27 11:46:26] ...trap/engines/engine.py.229: process: 0.55501 | load: 0.00242 [I 2022-10-27 11:46:26] ...trap/engines/engine.py.230: loss: 0.13544 [I 2022-10-27 11:46:26] ...optimizers/trijoint.py.151: total_norm: 10.062703 [I 2022-10-27 11:46:32] ...trap/engines/engine.py.224: train: epoch 65 | batch 1440/2304 [I 2022-10-27 11:46:32] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:04 [I 2022-10-27 11:46:32] ...trap/engines/engine.py.229: process: 0.54719 | load: 0.00247 [I 2022-10-27 11:46:32] ...trap/engines/engine.py.230: loss: 0.20317 [I 2022-10-27 11:46:32] ...optimizers/trijoint.py.151: total_norm: 32.940479 [I 2022-10-27 11:46:37] ...trap/engines/engine.py.224: train: epoch 65 | batch 1450/2304 [I 2022-10-27 11:46:37] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:59 [I 2022-10-27 11:46:37] ...trap/engines/engine.py.229: process: 0.56696 | load: 0.00245 [I 2022-10-27 11:46:37] ...trap/engines/engine.py.230: loss: 0.11765 [I 2022-10-27 11:46:37] ...optimizers/trijoint.py.151: total_norm: 20.799192 [I 2022-10-27 11:46:43] ...trap/engines/engine.py.224: train: epoch 65 | batch 1460/2304 [I 2022-10-27 11:46:43] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:50 [I 2022-10-27 11:46:43] ...trap/engines/engine.py.229: process: 0.56375 | load: 0.00248 [I 2022-10-27 11:46:43] ...trap/engines/engine.py.230: loss: 0.14397 [I 2022-10-27 11:46:43] ...optimizers/trijoint.py.151: total_norm: 33.372997 [I 2022-10-27 11:46:48] ...trap/engines/engine.py.224: train: epoch 65 | batch 1470/2304 [I 2022-10-27 11:46:48] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:44 [I 2022-10-27 11:46:48] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00267 [I 2022-10-27 11:46:48] ...trap/engines/engine.py.230: loss: 0.21894 [I 2022-10-27 11:46:48] ...optimizers/trijoint.py.151: total_norm: 16.512575 [I 2022-10-27 11:46:54] ...trap/engines/engine.py.224: train: epoch 65 | batch 1480/2304 [I 2022-10-27 11:46:54] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:40 [I 2022-10-27 11:46:54] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00247 [I 2022-10-27 11:46:54] ...trap/engines/engine.py.230: loss: 0.13522 [I 2022-10-27 11:46:54] ...optimizers/trijoint.py.151: total_norm: 24.042936 [I 2022-10-27 11:47:00] ...trap/engines/engine.py.224: train: epoch 65 | batch 1490/2304 [I 2022-10-27 11:47:00] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:49 [I 2022-10-27 11:47:00] ...trap/engines/engine.py.229: process: 0.62945 | load: 0.00242 [I 2022-10-27 11:47:00] ...trap/engines/engine.py.230: loss: 0.13377 [I 2022-10-27 11:47:00] ...optimizers/trijoint.py.151: total_norm: 18.610126 [I 2022-10-27 11:47:05] ...trap/engines/engine.py.224: train: epoch 65 | batch 1500/2304 [I 2022-10-27 11:47:05] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:27 [I 2022-10-27 11:47:05] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00243 [I 2022-10-27 11:47:05] ...trap/engines/engine.py.230: loss: 0.14505 [I 2022-10-27 11:47:05] ...optimizers/trijoint.py.151: total_norm: 12.059692 [I 2022-10-27 11:47:11] ...trap/engines/engine.py.224: train: epoch 65 | batch 1510/2304 [I 2022-10-27 11:47:11] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:23 [I 2022-10-27 11:47:11] ...trap/engines/engine.py.229: process: 0.56278 | load: 0.00242 [I 2022-10-27 11:47:11] ...trap/engines/engine.py.230: loss: 0.14590 [I 2022-10-27 11:47:11] ...optimizers/trijoint.py.151: total_norm: 10.881056 [I 2022-10-27 11:47:16] ...trap/engines/engine.py.224: train: epoch 65 | batch 1520/2304 [I 2022-10-27 11:47:16] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:18 [I 2022-10-27 11:47:16] ...trap/engines/engine.py.229: process: 0.56826 | load: 0.00242 [I 2022-10-27 11:47:16] ...trap/engines/engine.py.230: loss: 0.12212 [I 2022-10-27 11:47:16] ...optimizers/trijoint.py.151: total_norm: 17.174711 [I 2022-10-27 11:47:22] ...trap/engines/engine.py.224: train: epoch 65 | batch 1530/2304 [I 2022-10-27 11:47:22] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:10 [I 2022-10-27 11:47:22] ...trap/engines/engine.py.229: process: 0.56861 | load: 0.00241 [I 2022-10-27 11:47:22] ...trap/engines/engine.py.230: loss: 0.13689 [I 2022-10-27 11:47:22] ...optimizers/trijoint.py.151: total_norm: 9.829267 [I 2022-10-27 11:47:27] ...trap/engines/engine.py.224: train: epoch 65 | batch 1540/2304 [I 2022-10-27 11:47:27] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:05 [I 2022-10-27 11:47:27] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00243 [I 2022-10-27 11:47:27] ...trap/engines/engine.py.230: loss: 0.17000 [I 2022-10-27 11:47:27] ...optimizers/trijoint.py.151: total_norm: 13.387103 [I 2022-10-27 11:47:33] ...trap/engines/engine.py.224: train: epoch 65 | batch 1550/2304 [I 2022-10-27 11:47:33] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:06:59 [I 2022-10-27 11:47:33] ...trap/engines/engine.py.229: process: 0.55007 | load: 0.00245 [I 2022-10-27 11:47:33] ...trap/engines/engine.py.230: loss: 0.16187 [I 2022-10-27 11:47:33] ...optimizers/trijoint.py.151: total_norm: 20.404957 [I 2022-10-27 11:47:39] ...trap/engines/engine.py.224: train: epoch 65 | batch 1560/2304 [I 2022-10-27 11:47:39] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:06:52 [I 2022-10-27 11:47:39] ...trap/engines/engine.py.229: process: 0.55501 | load: 0.00241 [I 2022-10-27 11:47:39] ...trap/engines/engine.py.230: loss: 0.13173 [I 2022-10-27 11:47:39] ...optimizers/trijoint.py.151: total_norm: 10.154758 [I 2022-10-27 11:47:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 1570/2304 [I 2022-10-27 11:47:44] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:51 [I 2022-10-27 11:47:44] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00249 [I 2022-10-27 11:47:44] ...trap/engines/engine.py.230: loss: 0.11424 [I 2022-10-27 11:47:44] ...optimizers/trijoint.py.151: total_norm: 7.325854 [I 2022-10-27 11:47:50] ...trap/engines/engine.py.224: train: epoch 65 | batch 1580/2304 [I 2022-10-27 11:47:50] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:45 [I 2022-10-27 11:47:50] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00248 [I 2022-10-27 11:47:50] ...trap/engines/engine.py.230: loss: 0.10462 [I 2022-10-27 11:47:50] ...optimizers/trijoint.py.151: total_norm: 13.162129 [I 2022-10-27 11:47:55] ...trap/engines/engine.py.224: train: epoch 65 | batch 1590/2304 [I 2022-10-27 11:47:55] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:38 [I 2022-10-27 11:47:55] ...trap/engines/engine.py.229: process: 0.56342 | load: 0.00245 [I 2022-10-27 11:47:55] ...trap/engines/engine.py.230: loss: 0.10630 [I 2022-10-27 11:47:55] ...optimizers/trijoint.py.151: total_norm: 10.990823 [I 2022-10-27 11:48:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 1600/2304 [I 2022-10-27 11:48:01] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:31 [I 2022-10-27 11:48:01] ...trap/engines/engine.py.229: process: 0.55302 | load: 0.00245 [I 2022-10-27 11:48:01] ...trap/engines/engine.py.230: loss: 0.11780 [I 2022-10-27 11:48:01] ...optimizers/trijoint.py.151: total_norm: 16.232201 [I 2022-10-27 11:48:07] ...trap/engines/engine.py.224: train: epoch 65 | batch 1610/2304 [I 2022-10-27 11:48:07] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:24 [I 2022-10-27 11:48:07] ...trap/engines/engine.py.229: process: 0.56891 | load: 0.00243 [I 2022-10-27 11:48:07] ...trap/engines/engine.py.230: loss: 0.16389 [I 2022-10-27 11:48:07] ...optimizers/trijoint.py.151: total_norm: 27.796722 [I 2022-10-27 11:48:12] ...trap/engines/engine.py.224: train: epoch 65 | batch 1620/2304 [I 2022-10-27 11:48:12] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:19 [I 2022-10-27 11:48:12] ...trap/engines/engine.py.229: process: 0.54647 | load: 0.00242 [I 2022-10-27 11:48:12] ...trap/engines/engine.py.230: loss: 0.20144 [I 2022-10-27 11:48:12] ...optimizers/trijoint.py.151: total_norm: 24.706469 [I 2022-10-27 11:48:18] ...trap/engines/engine.py.224: train: epoch 65 | batch 1630/2304 [I 2022-10-27 11:48:18] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:16 [I 2022-10-27 11:48:18] ...trap/engines/engine.py.229: process: 0.56676 | load: 0.00246 [I 2022-10-27 11:48:18] ...trap/engines/engine.py.230: loss: 0.19381 [I 2022-10-27 11:48:18] ...optimizers/trijoint.py.151: total_norm: 21.183102 [I 2022-10-27 11:48:23] ...trap/engines/engine.py.224: train: epoch 65 | batch 1640/2304 [I 2022-10-27 11:48:23] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:09 [I 2022-10-27 11:48:23] ...trap/engines/engine.py.229: process: 0.54943 | load: 0.00249 [I 2022-10-27 11:48:23] ...trap/engines/engine.py.230: loss: 0.19294 [I 2022-10-27 11:48:23] ...optimizers/trijoint.py.151: total_norm: 20.203735 [I 2022-10-27 11:48:29] ...trap/engines/engine.py.224: train: epoch 65 | batch 1650/2304 [I 2022-10-27 11:48:29] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:02 [I 2022-10-27 11:48:29] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00249 [I 2022-10-27 11:48:29] ...trap/engines/engine.py.230: loss: 0.14433 [I 2022-10-27 11:48:29] ...optimizers/trijoint.py.151: total_norm: 14.521626 [I 2022-10-27 11:48:35] ...trap/engines/engine.py.224: train: epoch 65 | batch 1660/2304 [I 2022-10-27 11:48:35] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:05:59 [I 2022-10-27 11:48:35] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00244 [I 2022-10-27 11:48:35] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-27 11:48:35] ...optimizers/trijoint.py.151: total_norm: 16.130722 [I 2022-10-27 11:48:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 1670/2304 [I 2022-10-27 11:48:40] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:55 [I 2022-10-27 11:48:40] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00248 [I 2022-10-27 11:48:40] ...trap/engines/engine.py.230: loss: 0.19759 [I 2022-10-27 11:48:40] ...optimizers/trijoint.py.151: total_norm: 11.040565 [I 2022-10-27 11:48:46] ...trap/engines/engine.py.224: train: epoch 65 | batch 1680/2304 [I 2022-10-27 11:48:46] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:49 [I 2022-10-27 11:48:46] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00243 [I 2022-10-27 11:48:46] ...trap/engines/engine.py.230: loss: 0.15000 [I 2022-10-27 11:48:46] ...optimizers/trijoint.py.151: total_norm: 23.146475 [I 2022-10-27 11:48:51] ...trap/engines/engine.py.224: train: epoch 65 | batch 1690/2304 [I 2022-10-27 11:48:51] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:39 [I 2022-10-27 11:48:51] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00244 [I 2022-10-27 11:48:51] ...trap/engines/engine.py.230: loss: 0.13548 [I 2022-10-27 11:48:51] ...optimizers/trijoint.py.151: total_norm: 11.206848 [I 2022-10-27 11:48:57] ...trap/engines/engine.py.224: train: epoch 65 | batch 1700/2304 [I 2022-10-27 11:48:57] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:33 [I 2022-10-27 11:48:57] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00240 [I 2022-10-27 11:48:57] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-27 11:48:57] ...optimizers/trijoint.py.151: total_norm: 18.825779 [I 2022-10-27 11:49:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 1710/2304 [I 2022-10-27 11:49:02] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:30 [I 2022-10-27 11:49:02] ...trap/engines/engine.py.229: process: 0.55950 | load: 0.00242 [I 2022-10-27 11:49:02] ...trap/engines/engine.py.230: loss: 0.11493 [I 2022-10-27 11:49:02] ...optimizers/trijoint.py.151: total_norm: 12.231339 [I 2022-10-27 11:49:08] ...trap/engines/engine.py.224: train: epoch 65 | batch 1720/2304 [I 2022-10-27 11:49:08] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:25 [I 2022-10-27 11:49:08] ...trap/engines/engine.py.229: process: 0.55339 | load: 0.00240 [I 2022-10-27 11:49:08] ...trap/engines/engine.py.230: loss: 0.18097 [I 2022-10-27 11:49:08] ...optimizers/trijoint.py.151: total_norm: 7.894999 [I 2022-10-27 11:49:14] ...trap/engines/engine.py.224: train: epoch 65 | batch 1730/2304 [I 2022-10-27 11:49:14] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:18 [I 2022-10-27 11:49:14] ...trap/engines/engine.py.229: process: 0.56325 | load: 0.00241 [I 2022-10-27 11:49:14] ...trap/engines/engine.py.230: loss: 0.14498 [I 2022-10-27 11:49:14] ...optimizers/trijoint.py.151: total_norm: 14.305014 [I 2022-10-27 11:49:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 1740/2304 [I 2022-10-27 11:49:19] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:14 [I 2022-10-27 11:49:19] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00240 [I 2022-10-27 11:49:19] ...trap/engines/engine.py.230: loss: 0.22482 [I 2022-10-27 11:49:19] ...optimizers/trijoint.py.151: total_norm: 33.268417 [I 2022-10-27 11:49:25] ...trap/engines/engine.py.224: train: epoch 65 | batch 1750/2304 [I 2022-10-27 11:49:25] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:07 [I 2022-10-27 11:49:25] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00242 [I 2022-10-27 11:49:25] ...trap/engines/engine.py.230: loss: 0.13597 [I 2022-10-27 11:49:25] ...optimizers/trijoint.py.151: total_norm: 11.364164 [I 2022-10-27 11:49:30] ...trap/engines/engine.py.224: train: epoch 65 | batch 1760/2304 [I 2022-10-27 11:49:30] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:03 [I 2022-10-27 11:49:30] ...trap/engines/engine.py.229: process: 0.56141 | load: 0.00240 [I 2022-10-27 11:49:30] ...trap/engines/engine.py.230: loss: 0.18584 [I 2022-10-27 11:49:30] ...optimizers/trijoint.py.151: total_norm: 24.238293 [I 2022-10-27 11:49:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 1770/2304 [I 2022-10-27 11:49:36] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:04:57 [I 2022-10-27 11:49:36] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00241 [I 2022-10-27 11:49:36] ...trap/engines/engine.py.230: loss: 0.19633 [I 2022-10-27 11:49:36] ...optimizers/trijoint.py.151: total_norm: 16.643116 [I 2022-10-27 11:49:42] ...trap/engines/engine.py.224: train: epoch 65 | batch 1780/2304 [I 2022-10-27 11:49:42] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:04:59 [I 2022-10-27 11:49:42] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00256 [I 2022-10-27 11:49:42] ...trap/engines/engine.py.230: loss: 0.12808 [I 2022-10-27 11:49:42] ...optimizers/trijoint.py.151: total_norm: 34.596786 [I 2022-10-27 11:49:47] ...trap/engines/engine.py.224: train: epoch 65 | batch 1790/2304 [I 2022-10-27 11:49:47] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:46 [I 2022-10-27 11:49:47] ...trap/engines/engine.py.229: process: 0.56324 | load: 0.00247 [I 2022-10-27 11:49:47] ...trap/engines/engine.py.230: loss: 0.15333 [I 2022-10-27 11:49:47] ...optimizers/trijoint.py.151: total_norm: 30.441629 [I 2022-10-27 11:49:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 1800/2304 [I 2022-10-27 11:49:53] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:04:39 [I 2022-10-27 11:49:53] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00245 [I 2022-10-27 11:49:53] ...trap/engines/engine.py.230: loss: 0.16180 [I 2022-10-27 11:49:53] ...optimizers/trijoint.py.151: total_norm: 26.821985 [I 2022-10-27 11:49:58] ...trap/engines/engine.py.224: train: epoch 65 | batch 1810/2304 [I 2022-10-27 11:49:58] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:37 [I 2022-10-27 11:49:58] ...trap/engines/engine.py.229: process: 0.57542 | load: 0.00243 [I 2022-10-27 11:49:58] ...trap/engines/engine.py.230: loss: 0.16532 [I 2022-10-27 11:49:58] ...optimizers/trijoint.py.151: total_norm: 19.188301 [I 2022-10-27 11:50:04] ...trap/engines/engine.py.224: train: epoch 65 | batch 1820/2304 [I 2022-10-27 11:50:04] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:29 [I 2022-10-27 11:50:04] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00251 [I 2022-10-27 11:50:04] ...trap/engines/engine.py.230: loss: 0.13474 [I 2022-10-27 11:50:04] ...optimizers/trijoint.py.151: total_norm: 20.501181 [I 2022-10-27 11:50:09] ...trap/engines/engine.py.224: train: epoch 65 | batch 1830/2304 [I 2022-10-27 11:50:09] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:24 [I 2022-10-27 11:50:09] ...trap/engines/engine.py.229: process: 0.56553 | load: 0.00243 [I 2022-10-27 11:50:09] ...trap/engines/engine.py.230: loss: 0.12675 [I 2022-10-27 11:50:09] ...optimizers/trijoint.py.151: total_norm: 21.526266 [I 2022-10-27 11:50:15] ...trap/engines/engine.py.224: train: epoch 65 | batch 1840/2304 [I 2022-10-27 11:50:15] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:18 [I 2022-10-27 11:50:15] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00242 [I 2022-10-27 11:50:15] ...trap/engines/engine.py.230: loss: 0.12571 [I 2022-10-27 11:50:15] ...optimizers/trijoint.py.151: total_norm: 11.021922 [I 2022-10-27 11:50:21] ...trap/engines/engine.py.224: train: epoch 65 | batch 1850/2304 [I 2022-10-27 11:50:21] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:12 [I 2022-10-27 11:50:21] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00244 [I 2022-10-27 11:50:21] ...trap/engines/engine.py.230: loss: 0.18087 [I 2022-10-27 11:50:21] ...optimizers/trijoint.py.151: total_norm: 17.885929 [I 2022-10-27 11:50:26] ...trap/engines/engine.py.224: train: epoch 65 | batch 1860/2304 [I 2022-10-27 11:50:26] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:07 [I 2022-10-27 11:50:26] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00241 [I 2022-10-27 11:50:26] ...trap/engines/engine.py.230: loss: 0.14642 [I 2022-10-27 11:50:26] ...optimizers/trijoint.py.151: total_norm: 20.913839 [I 2022-10-27 11:50:32] ...trap/engines/engine.py.224: train: epoch 65 | batch 1870/2304 [I 2022-10-27 11:50:32] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:02 [I 2022-10-27 11:50:32] ...trap/engines/engine.py.229: process: 0.56483 | load: 0.00245 [I 2022-10-27 11:50:32] ...trap/engines/engine.py.230: loss: 0.16006 [I 2022-10-27 11:50:32] ...optimizers/trijoint.py.151: total_norm: 32.668510 [I 2022-10-27 11:50:37] ...trap/engines/engine.py.224: train: epoch 65 | batch 1880/2304 [I 2022-10-27 11:50:37] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:03:57 [I 2022-10-27 11:50:37] ...trap/engines/engine.py.229: process: 0.55486 | load: 0.00248 [I 2022-10-27 11:50:37] ...trap/engines/engine.py.230: loss: 0.16287 [I 2022-10-27 11:50:37] ...optimizers/trijoint.py.151: total_norm: 21.693457 [I 2022-10-27 11:50:43] ...trap/engines/engine.py.224: train: epoch 65 | batch 1890/2304 [I 2022-10-27 11:50:43] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:03:51 [I 2022-10-27 11:50:43] ...trap/engines/engine.py.229: process: 0.57460 | load: 0.00244 [I 2022-10-27 11:50:43] ...trap/engines/engine.py.230: loss: 0.18394 [I 2022-10-27 11:50:43] ...optimizers/trijoint.py.151: total_norm: 27.323603 [I 2022-10-27 11:50:49] ...trap/engines/engine.py.224: train: epoch 65 | batch 1900/2304 [I 2022-10-27 11:50:49] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:03:50 [I 2022-10-27 11:50:49] ...trap/engines/engine.py.229: process: 0.62226 | load: 0.00245 [I 2022-10-27 11:50:49] ...trap/engines/engine.py.230: loss: 0.14169 [I 2022-10-27 11:50:49] ...optimizers/trijoint.py.151: total_norm: 27.712183 [I 2022-10-27 11:50:54] ...trap/engines/engine.py.224: train: epoch 65 | batch 1910/2304 [I 2022-10-27 11:50:54] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:38 [I 2022-10-27 11:50:54] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00244 [I 2022-10-27 11:50:54] ...trap/engines/engine.py.230: loss: 0.13451 [I 2022-10-27 11:50:54] ...optimizers/trijoint.py.151: total_norm: 20.851294 [I 2022-10-27 11:51:00] ...trap/engines/engine.py.224: train: epoch 65 | batch 1920/2304 [I 2022-10-27 11:51:00] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:31 [I 2022-10-27 11:51:00] ...trap/engines/engine.py.229: process: 0.53795 | load: 0.00239 [I 2022-10-27 11:51:00] ...trap/engines/engine.py.230: loss: 0.21493 [I 2022-10-27 11:51:00] ...optimizers/trijoint.py.151: total_norm: 18.129183 [I 2022-10-27 11:51:05] ...trap/engines/engine.py.224: train: epoch 65 | batch 1930/2304 [I 2022-10-27 11:51:05] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:25 [I 2022-10-27 11:51:05] ...trap/engines/engine.py.229: process: 0.54524 | load: 0.00241 [I 2022-10-27 11:51:05] ...trap/engines/engine.py.230: loss: 0.12221 [I 2022-10-27 11:51:05] ...optimizers/trijoint.py.151: total_norm: 8.719682 [I 2022-10-27 11:51:11] ...trap/engines/engine.py.224: train: epoch 65 | batch 1940/2304 [I 2022-10-27 11:51:11] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:20 [I 2022-10-27 11:51:11] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00241 [I 2022-10-27 11:51:11] ...trap/engines/engine.py.230: loss: 0.14416 [I 2022-10-27 11:51:11] ...optimizers/trijoint.py.151: total_norm: 24.399446 [I 2022-10-27 11:51:16] ...trap/engines/engine.py.224: train: epoch 65 | batch 1950/2304 [I 2022-10-27 11:51:16] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:16 [I 2022-10-27 11:51:16] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00243 [I 2022-10-27 11:51:16] ...trap/engines/engine.py.230: loss: 0.13858 [I 2022-10-27 11:51:16] ...optimizers/trijoint.py.151: total_norm: 12.236833 [I 2022-10-27 11:51:22] ...trap/engines/engine.py.224: train: epoch 65 | batch 1960/2304 [I 2022-10-27 11:51:22] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:10 [I 2022-10-27 11:51:22] ...trap/engines/engine.py.229: process: 0.54968 | load: 0.00240 [I 2022-10-27 11:51:22] ...trap/engines/engine.py.230: loss: 0.13067 [I 2022-10-27 11:51:22] ...optimizers/trijoint.py.151: total_norm: 8.772884 [I 2022-10-27 11:51:28] ...trap/engines/engine.py.224: train: epoch 65 | batch 1970/2304 [I 2022-10-27 11:51:28] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:04 [I 2022-10-27 11:51:28] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00243 [I 2022-10-27 11:51:28] ...trap/engines/engine.py.230: loss: 0.09457 [I 2022-10-27 11:51:28] ...optimizers/trijoint.py.151: total_norm: 8.572884 [I 2022-10-27 11:51:33] ...trap/engines/engine.py.224: train: epoch 65 | batch 1980/2304 [I 2022-10-27 11:51:33] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:02:59 [I 2022-10-27 11:51:33] ...trap/engines/engine.py.229: process: 0.54548 | load: 0.00240 [I 2022-10-27 11:51:33] ...trap/engines/engine.py.230: loss: 0.17149 [I 2022-10-27 11:51:33] ...optimizers/trijoint.py.151: total_norm: 25.525755 [I 2022-10-27 11:51:39] ...trap/engines/engine.py.224: train: epoch 65 | batch 1990/2304 [I 2022-10-27 11:51:39] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:02:54 [I 2022-10-27 11:51:39] ...trap/engines/engine.py.229: process: 0.56854 | load: 0.00243 [I 2022-10-27 11:51:39] ...trap/engines/engine.py.230: loss: 0.13227 [I 2022-10-27 11:51:39] ...optimizers/trijoint.py.151: total_norm: 24.825218 [I 2022-10-27 11:51:44] ...trap/engines/engine.py.224: train: epoch 65 | batch 2000/2304 [I 2022-10-27 11:51:44] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:02:49 [I 2022-10-27 11:51:44] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00251 [I 2022-10-27 11:51:44] ...trap/engines/engine.py.230: loss: 0.21393 [I 2022-10-27 11:51:44] ...optimizers/trijoint.py.151: total_norm: 30.363430 [I 2022-10-27 11:51:50] ...trap/engines/engine.py.224: train: epoch 65 | batch 2010/2304 [I 2022-10-27 11:51:50] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:02:44 [I 2022-10-27 11:51:50] ...trap/engines/engine.py.229: process: 0.56775 | load: 0.00249 [I 2022-10-27 11:51:50] ...trap/engines/engine.py.230: loss: 0.14437 [I 2022-10-27 11:51:50] ...optimizers/trijoint.py.151: total_norm: 15.579611 [I 2022-10-27 11:51:55] ...trap/engines/engine.py.224: train: epoch 65 | batch 2020/2304 [I 2022-10-27 11:51:55] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:38 [I 2022-10-27 11:51:55] ...trap/engines/engine.py.229: process: 0.56154 | load: 0.00245 [I 2022-10-27 11:51:55] ...trap/engines/engine.py.230: loss: 0.12733 [I 2022-10-27 11:51:55] ...optimizers/trijoint.py.151: total_norm: 15.091285 [I 2022-10-27 11:52:01] ...trap/engines/engine.py.224: train: epoch 65 | batch 2030/2304 [I 2022-10-27 11:52:01] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:31 [I 2022-10-27 11:52:01] ...trap/engines/engine.py.229: process: 0.54745 | load: 0.00242 [I 2022-10-27 11:52:01] ...trap/engines/engine.py.230: loss: 0.12419 [I 2022-10-27 11:52:01] ...optimizers/trijoint.py.151: total_norm: 13.858546 [I 2022-10-27 11:52:07] ...trap/engines/engine.py.224: train: epoch 65 | batch 2040/2304 [I 2022-10-27 11:52:07] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:26 [I 2022-10-27 11:52:07] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00242 [I 2022-10-27 11:52:07] ...trap/engines/engine.py.230: loss: 0.12180 [I 2022-10-27 11:52:07] ...optimizers/trijoint.py.151: total_norm: 18.302225 [I 2022-10-27 11:52:12] ...trap/engines/engine.py.224: train: epoch 65 | batch 2050/2304 [I 2022-10-27 11:52:12] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:21 [I 2022-10-27 11:52:12] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00245 [I 2022-10-27 11:52:12] ...trap/engines/engine.py.230: loss: 0.11991 [I 2022-10-27 11:52:12] ...optimizers/trijoint.py.151: total_norm: 24.088123 [I 2022-10-27 11:52:18] ...trap/engines/engine.py.224: train: epoch 65 | batch 2060/2304 [I 2022-10-27 11:52:18] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:16 [I 2022-10-27 11:52:18] ...trap/engines/engine.py.229: process: 0.56463 | load: 0.00243 [I 2022-10-27 11:52:18] ...trap/engines/engine.py.230: loss: 0.15181 [I 2022-10-27 11:52:18] ...optimizers/trijoint.py.151: total_norm: 10.157974 [I 2022-10-27 11:52:23] ...trap/engines/engine.py.224: train: epoch 65 | batch 2070/2304 [I 2022-10-27 11:52:23] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:09 [I 2022-10-27 11:52:23] ...trap/engines/engine.py.229: process: 0.55371 | load: 0.00242 [I 2022-10-27 11:52:23] ...trap/engines/engine.py.230: loss: 0.20437 [I 2022-10-27 11:52:23] ...optimizers/trijoint.py.151: total_norm: 36.366299 [I 2022-10-27 11:52:29] ...trap/engines/engine.py.224: train: epoch 65 | batch 2080/2304 [I 2022-10-27 11:52:29] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:04 [I 2022-10-27 11:52:29] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00243 [I 2022-10-27 11:52:29] ...trap/engines/engine.py.230: loss: 0.14224 [I 2022-10-27 11:52:29] ...optimizers/trijoint.py.151: total_norm: 16.519876 [I 2022-10-27 11:52:34] ...trap/engines/engine.py.224: train: epoch 65 | batch 2090/2304 [I 2022-10-27 11:52:34] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:02:00 [I 2022-10-27 11:52:34] ...trap/engines/engine.py.229: process: 0.56514 | load: 0.00250 [I 2022-10-27 11:52:34] ...trap/engines/engine.py.230: loss: 0.21144 [I 2022-10-27 11:52:34] ...optimizers/trijoint.py.151: total_norm: 19.819395 [I 2022-10-27 11:52:40] ...trap/engines/engine.py.224: train: epoch 65 | batch 2100/2304 [I 2022-10-27 11:52:40] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:01:54 [I 2022-10-27 11:52:40] ...trap/engines/engine.py.229: process: 0.56951 | load: 0.00249 [I 2022-10-27 11:52:40] ...trap/engines/engine.py.230: loss: 0.12635 [I 2022-10-27 11:52:40] ...optimizers/trijoint.py.151: total_norm: 12.667286 [I 2022-10-27 11:52:46] ...trap/engines/engine.py.224: train: epoch 65 | batch 2110/2304 [I 2022-10-27 11:52:46] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:01:48 [I 2022-10-27 11:52:46] ...trap/engines/engine.py.229: process: 0.57252 | load: 0.00245 [I 2022-10-27 11:52:46] ...trap/engines/engine.py.230: loss: 0.15242 [I 2022-10-27 11:52:46] ...optimizers/trijoint.py.151: total_norm: 21.763668 [I 2022-10-27 11:52:51] ...trap/engines/engine.py.224: train: epoch 65 | batch 2120/2304 [I 2022-10-27 11:52:51] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:01:43 [I 2022-10-27 11:52:51] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00245 [I 2022-10-27 11:52:51] ...trap/engines/engine.py.230: loss: 0.10888 [I 2022-10-27 11:52:51] ...optimizers/trijoint.py.151: total_norm: 20.257290 [I 2022-10-27 11:52:57] ...trap/engines/engine.py.224: train: epoch 65 | batch 2130/2304 [I 2022-10-27 11:52:57] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:37 [I 2022-10-27 11:52:57] ...trap/engines/engine.py.229: process: 0.56420 | load: 0.00246 [I 2022-10-27 11:52:57] ...trap/engines/engine.py.230: loss: 0.15676 [I 2022-10-27 11:52:57] ...optimizers/trijoint.py.151: total_norm: 18.841812 [I 2022-10-27 11:53:02] ...trap/engines/engine.py.224: train: epoch 65 | batch 2140/2304 [I 2022-10-27 11:53:02] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:31 [I 2022-10-27 11:53:02] ...trap/engines/engine.py.229: process: 0.55404 | load: 0.00242 [I 2022-10-27 11:53:02] ...trap/engines/engine.py.230: loss: 0.12490 [I 2022-10-27 11:53:02] ...optimizers/trijoint.py.151: total_norm: 11.861217 [I 2022-10-27 11:53:08] ...trap/engines/engine.py.224: train: epoch 65 | batch 2150/2304 [I 2022-10-27 11:53:08] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:25 [I 2022-10-27 11:53:08] ...trap/engines/engine.py.229: process: 0.56669 | load: 0.00248 [I 2022-10-27 11:53:08] ...trap/engines/engine.py.230: loss: 0.15500 [I 2022-10-27 11:53:08] ...optimizers/trijoint.py.151: total_norm: 9.756707 [I 2022-10-27 11:53:14] ...trap/engines/engine.py.224: train: epoch 65 | batch 2160/2304 [I 2022-10-27 11:53:14] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:20 [I 2022-10-27 11:53:14] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00244 [I 2022-10-27 11:53:14] ...trap/engines/engine.py.230: loss: 0.10664 [I 2022-10-27 11:53:14] ...optimizers/trijoint.py.151: total_norm: 8.213632 [I 2022-10-27 11:53:19] ...trap/engines/engine.py.224: train: epoch 65 | batch 2170/2304 [I 2022-10-27 11:53:19] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:14 [I 2022-10-27 11:53:19] ...trap/engines/engine.py.229: process: 0.56581 | load: 0.00244 [I 2022-10-27 11:53:19] ...trap/engines/engine.py.230: loss: 0.16277 [I 2022-10-27 11:53:19] ...optimizers/trijoint.py.151: total_norm: 14.213573 [I 2022-10-27 11:53:25] ...trap/engines/engine.py.224: train: epoch 65 | batch 2180/2304 [I 2022-10-27 11:53:25] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:08 [I 2022-10-27 11:53:25] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00244 [I 2022-10-27 11:53:25] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-27 11:53:25] ...optimizers/trijoint.py.151: total_norm: 13.565882 [I 2022-10-27 11:53:30] ...trap/engines/engine.py.224: train: epoch 65 | batch 2190/2304 [I 2022-10-27 11:53:30] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:03 [I 2022-10-27 11:53:30] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00243 [I 2022-10-27 11:53:30] ...trap/engines/engine.py.230: loss: 0.12703 [I 2022-10-27 11:53:30] ...optimizers/trijoint.py.151: total_norm: 13.754729 [I 2022-10-27 11:53:36] ...trap/engines/engine.py.224: train: epoch 65 | batch 2200/2304 [I 2022-10-27 11:53:36] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:00:58 [I 2022-10-27 11:53:36] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00243 [I 2022-10-27 11:53:36] ...trap/engines/engine.py.230: loss: 0.13265 [I 2022-10-27 11:53:36] ...optimizers/trijoint.py.151: total_norm: 20.012789 [I 2022-10-27 11:53:41] ...trap/engines/engine.py.224: train: epoch 65 | batch 2210/2304 [I 2022-10-27 11:53:41] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:00:52 [I 2022-10-27 11:53:41] ...trap/engines/engine.py.229: process: 0.56857 | load: 0.00243 [I 2022-10-27 11:53:41] ...trap/engines/engine.py.230: loss: 0.09709 [I 2022-10-27 11:53:41] ...optimizers/trijoint.py.151: total_norm: 15.296785 [I 2022-10-27 11:53:47] ...trap/engines/engine.py.224: train: epoch 65 | batch 2220/2304 [I 2022-10-27 11:53:47] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:00:46 [I 2022-10-27 11:53:47] ...trap/engines/engine.py.229: process: 0.55155 | load: 0.00245 [I 2022-10-27 11:53:47] ...trap/engines/engine.py.230: loss: 0.11008 [I 2022-10-27 11:53:47] ...optimizers/trijoint.py.151: total_norm: 38.626175 [I 2022-10-27 11:53:53] ...trap/engines/engine.py.224: train: epoch 65 | batch 2230/2304 [I 2022-10-27 11:53:53] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:41 [I 2022-10-27 11:53:53] ...trap/engines/engine.py.229: process: 0.55215 | load: 0.00244 [I 2022-10-27 11:53:53] ...trap/engines/engine.py.230: loss: 0.11406 [I 2022-10-27 11:53:53] ...optimizers/trijoint.py.151: total_norm: 20.778725 [I 2022-10-27 11:53:58] ...trap/engines/engine.py.224: train: epoch 65 | batch 2240/2304 [I 2022-10-27 11:53:58] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:35 [I 2022-10-27 11:53:58] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00238 [I 2022-10-27 11:53:58] ...trap/engines/engine.py.230: loss: 0.12422 [I 2022-10-27 11:53:58] ...optimizers/trijoint.py.151: total_norm: 12.007816 [I 2022-10-27 11:54:04] ...trap/engines/engine.py.224: train: epoch 65 | batch 2250/2304 [I 2022-10-27 11:54:04] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:30 [I 2022-10-27 11:54:04] ...trap/engines/engine.py.229: process: 0.56163 | load: 0.00244 [I 2022-10-27 11:54:04] ...trap/engines/engine.py.230: loss: 0.18887 [I 2022-10-27 11:54:04] ...optimizers/trijoint.py.151: total_norm: 16.201963 [I 2022-10-27 11:54:09] ...trap/engines/engine.py.224: train: epoch 65 | batch 2260/2304 [I 2022-10-27 11:54:09] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:24 [I 2022-10-27 11:54:09] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00244 [I 2022-10-27 11:54:09] ...trap/engines/engine.py.230: loss: 0.14530 [I 2022-10-27 11:54:09] ...optimizers/trijoint.py.151: total_norm: 20.768452 [I 2022-10-27 11:54:15] ...trap/engines/engine.py.224: train: epoch 65 | batch 2270/2304 [I 2022-10-27 11:54:15] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:19 [I 2022-10-27 11:54:15] ...trap/engines/engine.py.229: process: 0.56162 | load: 0.00242 [I 2022-10-27 11:54:15] ...trap/engines/engine.py.230: loss: 0.13000 [I 2022-10-27 11:54:15] ...optimizers/trijoint.py.151: total_norm: 56.369888 [I 2022-10-27 11:54:21] ...trap/engines/engine.py.224: train: epoch 65 | batch 2280/2304 [I 2022-10-27 11:54:21] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:13 [I 2022-10-27 11:54:21] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00243 [I 2022-10-27 11:54:21] ...trap/engines/engine.py.230: loss: 0.18202 [I 2022-10-27 11:54:21] ...optimizers/trijoint.py.151: total_norm: 21.737404 [I 2022-10-27 11:54:26] ...trap/engines/engine.py.224: train: epoch 65 | batch 2290/2304 [I 2022-10-27 11:54:26] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:07 [I 2022-10-27 11:54:26] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00246 [I 2022-10-27 11:54:26] ...trap/engines/engine.py.230: loss: 0.15489 [I 2022-10-27 11:54:26] ...optimizers/trijoint.py.151: total_norm: 26.883869 [I 2022-10-27 11:54:32] ...trap/engines/engine.py.224: train: epoch 65 | batch 2300/2304 [I 2022-10-27 11:54:32] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:02 [I 2022-10-27 11:54:32] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00014 [I 2022-10-27 11:54:32] ...trap/engines/engine.py.230: loss: 0.12503 [I 2022-10-27 11:54:32] ...optimizers/trijoint.py.151: total_norm: 12.080432 [I 2022-10-27 11:54:34] ...trap/engines/engine.py.224: train: epoch 65 | batch 2304/2304 [I 2022-10-27 11:54:34] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:00 [I 2022-10-27 11:54:34] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00014 [I 2022-10-27 11:54:34] ...trap/engines/engine.py.230: loss: 0.11751 [I 2022-10-27 11:54:34] ...optimizers/trijoint.py.151: total_norm: 14.175642 [S 2022-10-27 11:54:35] ...trap/engines/engine.py.129: train_epoch.epoch: 65 [S 2022-10-27 11:54:35] ...trap/engines/engine.py.129: train_epoch.loss: 0.14873598335621932 [I 2022-10-27 11:54:35] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 11:54:36] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 65 [I 2022-10-27 11:54:36] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 11:54:38] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 11:54:38] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 11:54:38] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 11:54:40] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 11:54:40] ...trap/engines/engine.py.321: eval: epoch 65 | batch 0/510 [I 2022-10-27 11:54:40] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:36:49 [I 2022-10-27 11:54:40] ...trap/engines/engine.py.326: process: 4.33316 | load: 3.47922 [I 2022-10-27 11:54:44] ...trap/engines/engine.py.321: eval: epoch 65 | batch 10/510 [I 2022-10-27 11:54:44] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:21 [I 2022-10-27 11:54:44] ...trap/engines/engine.py.326: process: 0.32304 | load: 0.00017 [I 2022-10-27 11:54:47] ...trap/engines/engine.py.321: eval: epoch 65 | batch 20/510 [I 2022-10-27 11:54:47] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:06 [I 2022-10-27 11:54:47] ...trap/engines/engine.py.326: process: 0.32815 | load: 0.00023 [I 2022-10-27 11:54:50] ...trap/engines/engine.py.321: eval: epoch 65 | batch 30/510 [I 2022-10-27 11:54:50] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:43 [I 2022-10-27 11:54:50] ...trap/engines/engine.py.326: process: 0.33861 | load: 0.00027 [I 2022-10-27 11:54:54] ...trap/engines/engine.py.321: eval: epoch 65 | batch 40/510 [I 2022-10-27 11:54:54] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:39 [I 2022-10-27 11:54:54] ...trap/engines/engine.py.326: process: 0.32556 | load: 0.00026 [I 2022-10-27 11:54:57] ...trap/engines/engine.py.321: eval: epoch 65 | batch 50/510 [I 2022-10-27 11:54:57] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:35 [I 2022-10-27 11:54:57] ...trap/engines/engine.py.326: process: 0.33164 | load: 0.00022 [I 2022-10-27 11:55:01] ...trap/engines/engine.py.321: eval: epoch 65 | batch 60/510 [I 2022-10-27 11:55:01] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:33 [I 2022-10-27 11:55:01] ...trap/engines/engine.py.326: process: 0.32491 | load: 0.00020 [I 2022-10-27 11:55:04] ...trap/engines/engine.py.321: eval: epoch 65 | batch 70/510 [I 2022-10-27 11:55:04] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:29 [I 2022-10-27 11:55:04] ...trap/engines/engine.py.326: process: 0.33400 | load: 0.00016 [I 2022-10-27 11:55:07] ...trap/engines/engine.py.321: eval: epoch 65 | batch 80/510 [I 2022-10-27 11:55:07] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:23 [I 2022-10-27 11:55:07] ...trap/engines/engine.py.326: process: 0.33657 | load: 0.00020 [I 2022-10-27 11:55:11] ...trap/engines/engine.py.321: eval: epoch 65 | batch 90/510 [I 2022-10-27 11:55:11] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:22 [I 2022-10-27 11:55:11] ...trap/engines/engine.py.326: process: 0.33960 | load: 0.00017 [I 2022-10-27 11:55:14] ...trap/engines/engine.py.321: eval: epoch 65 | batch 100/510 [I 2022-10-27 11:55:14] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:15 [I 2022-10-27 11:55:14] ...trap/engines/engine.py.326: process: 0.28049 | load: 0.00018 [I 2022-10-27 11:55:17] ...trap/engines/engine.py.321: eval: epoch 65 | batch 110/510 [I 2022-10-27 11:55:17] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:56 [I 2022-10-27 11:55:17] ...trap/engines/engine.py.326: process: 0.28279 | load: 0.00024 [I 2022-10-27 11:55:20] ...trap/engines/engine.py.321: eval: epoch 65 | batch 120/510 [I 2022-10-27 11:55:20] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:53 [I 2022-10-27 11:55:20] ...trap/engines/engine.py.326: process: 0.28895 | load: 0.00020 [I 2022-10-27 11:55:23] ...trap/engines/engine.py.321: eval: epoch 65 | batch 130/510 [I 2022-10-27 11:55:23] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:51 [I 2022-10-27 11:55:23] ...trap/engines/engine.py.326: process: 0.30361 | load: 0.00015 [I 2022-10-27 11:55:26] ...trap/engines/engine.py.321: eval: epoch 65 | batch 140/510 [I 2022-10-27 11:55:26] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 11:55:26] ...trap/engines/engine.py.326: process: 0.29492 | load: 0.00014 [I 2022-10-27 11:55:28] ...trap/engines/engine.py.321: eval: epoch 65 | batch 150/510 [I 2022-10-27 11:55:28] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:43 [I 2022-10-27 11:55:28] ...trap/engines/engine.py.326: process: 0.28996 | load: 0.00023 [I 2022-10-27 11:55:31] ...trap/engines/engine.py.321: eval: epoch 65 | batch 160/510 [I 2022-10-27 11:55:31] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 11:55:31] ...trap/engines/engine.py.326: process: 0.28975 | load: 0.00016 [I 2022-10-27 11:55:34] ...trap/engines/engine.py.321: eval: epoch 65 | batch 170/510 [I 2022-10-27 11:55:34] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 11:55:34] ...trap/engines/engine.py.326: process: 0.28775 | load: 0.00018 [I 2022-10-27 11:55:37] ...trap/engines/engine.py.321: eval: epoch 65 | batch 180/510 [I 2022-10-27 11:55:37] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 11:55:37] ...trap/engines/engine.py.326: process: 0.28563 | load: 0.00014 [I 2022-10-27 11:55:40] ...trap/engines/engine.py.321: eval: epoch 65 | batch 190/510 [I 2022-10-27 11:55:40] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:31 [I 2022-10-27 11:55:40] ...trap/engines/engine.py.326: process: 0.28237 | load: 0.00021 [I 2022-10-27 11:55:43] ...trap/engines/engine.py.321: eval: epoch 65 | batch 200/510 [I 2022-10-27 11:55:43] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 11:55:43] ...trap/engines/engine.py.326: process: 0.29171 | load: 0.00023 [I 2022-10-27 11:55:46] ...trap/engines/engine.py.321: eval: epoch 65 | batch 210/510 [I 2022-10-27 11:55:46] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 11:55:46] ...trap/engines/engine.py.326: process: 0.28765 | load: 0.00015 [I 2022-10-27 11:55:49] ...trap/engines/engine.py.321: eval: epoch 65 | batch 220/510 [I 2022-10-27 11:55:49] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 11:55:49] ...trap/engines/engine.py.326: process: 0.29660 | load: 0.00014 [I 2022-10-27 11:55:51] ...trap/engines/engine.py.321: eval: epoch 65 | batch 230/510 [I 2022-10-27 11:55:51] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:21 [I 2022-10-27 11:55:51] ...trap/engines/engine.py.326: process: 0.29506 | load: 0.00019 [I 2022-10-27 11:55:54] ...trap/engines/engine.py.321: eval: epoch 65 | batch 240/510 [I 2022-10-27 11:55:54] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 11:55:54] ...trap/engines/engine.py.326: process: 0.28782 | load: 0.00015 [I 2022-10-27 11:55:57] ...trap/engines/engine.py.321: eval: epoch 65 | batch 250/510 [I 2022-10-27 11:55:57] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 11:55:57] ...trap/engines/engine.py.326: process: 0.28458 | load: 0.00021 [I 2022-10-27 11:56:00] ...trap/engines/engine.py.321: eval: epoch 65 | batch 260/510 [I 2022-10-27 11:56:00] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 11:56:00] ...trap/engines/engine.py.326: process: 0.28554 | load: 0.00022 [I 2022-10-27 11:56:03] ...trap/engines/engine.py.321: eval: epoch 65 | batch 270/510 [I 2022-10-27 11:56:03] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 11:56:03] ...trap/engines/engine.py.326: process: 0.28452 | load: 0.00016 [I 2022-10-27 11:56:06] ...trap/engines/engine.py.321: eval: epoch 65 | batch 280/510 [I 2022-10-27 11:56:06] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:07 [I 2022-10-27 11:56:06] ...trap/engines/engine.py.326: process: 0.29561 | load: 0.00014 [I 2022-10-27 11:56:09] ...trap/engines/engine.py.321: eval: epoch 65 | batch 290/510 [I 2022-10-27 11:56:09] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 11:56:09] ...trap/engines/engine.py.326: process: 0.29425 | load: 0.00014 [I 2022-10-27 11:56:12] ...trap/engines/engine.py.321: eval: epoch 65 | batch 300/510 [I 2022-10-27 11:56:12] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 11:56:12] ...trap/engines/engine.py.326: process: 0.28585 | load: 0.00018 [I 2022-10-27 11:56:15] ...trap/engines/engine.py.321: eval: epoch 65 | batch 310/510 [I 2022-10-27 11:56:15] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:00 [I 2022-10-27 11:56:15] ...trap/engines/engine.py.326: process: 0.31889 | load: 0.00017 [I 2022-10-27 11:56:18] ...trap/engines/engine.py.321: eval: epoch 65 | batch 320/510 [I 2022-10-27 11:56:18] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:55 [I 2022-10-27 11:56:18] ...trap/engines/engine.py.326: process: 0.29008 | load: 0.00015 [I 2022-10-27 11:56:21] ...trap/engines/engine.py.321: eval: epoch 65 | batch 330/510 [I 2022-10-27 11:56:21] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:52 [I 2022-10-27 11:56:21] ...trap/engines/engine.py.326: process: 0.29342 | load: 0.00014 [I 2022-10-27 11:56:23] ...trap/engines/engine.py.321: eval: epoch 65 | batch 340/510 [I 2022-10-27 11:56:23] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-27 11:56:23] ...trap/engines/engine.py.326: process: 0.28906 | load: 0.00026 [I 2022-10-27 11:56:26] ...trap/engines/engine.py.321: eval: epoch 65 | batch 350/510 [I 2022-10-27 11:56:26] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:45 [I 2022-10-27 11:56:26] ...trap/engines/engine.py.326: process: 0.29000 | load: 0.00015 [I 2022-10-27 11:56:29] ...trap/engines/engine.py.321: eval: epoch 65 | batch 360/510 [I 2022-10-27 11:56:29] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 11:56:29] ...trap/engines/engine.py.326: process: 0.28405 | load: 0.00015 [I 2022-10-27 11:56:32] ...trap/engines/engine.py.321: eval: epoch 65 | batch 370/510 [I 2022-10-27 11:56:32] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 11:56:32] ...trap/engines/engine.py.326: process: 0.28634 | load: 0.00015 [I 2022-10-27 11:56:35] ...trap/engines/engine.py.321: eval: epoch 65 | batch 380/510 [I 2022-10-27 11:56:35] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 11:56:35] ...trap/engines/engine.py.326: process: 0.28813 | load: 0.00019 [I 2022-10-27 11:56:38] ...trap/engines/engine.py.321: eval: epoch 65 | batch 390/510 [I 2022-10-27 11:56:38] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 11:56:38] ...trap/engines/engine.py.326: process: 0.29216 | load: 0.00015 [I 2022-10-27 11:56:41] ...trap/engines/engine.py.321: eval: epoch 65 | batch 400/510 [I 2022-10-27 11:56:41] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 11:56:41] ...trap/engines/engine.py.326: process: 0.28971 | load: 0.00015 [I 2022-10-27 11:56:44] ...trap/engines/engine.py.321: eval: epoch 65 | batch 410/510 [I 2022-10-27 11:56:44] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 11:56:44] ...trap/engines/engine.py.326: process: 0.28488 | load: 0.00015 [I 2022-10-27 11:56:46] ...trap/engines/engine.py.321: eval: epoch 65 | batch 420/510 [I 2022-10-27 11:56:46] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 11:56:46] ...trap/engines/engine.py.326: process: 0.28660 | load: 0.00019 [I 2022-10-27 11:56:49] ...trap/engines/engine.py.321: eval: epoch 65 | batch 430/510 [I 2022-10-27 11:56:49] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 11:56:49] ...trap/engines/engine.py.326: process: 0.29196 | load: 0.00015 [I 2022-10-27 11:56:52] ...trap/engines/engine.py.321: eval: epoch 65 | batch 440/510 [I 2022-10-27 11:56:52] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:21 [I 2022-10-27 11:56:52] ...trap/engines/engine.py.326: process: 0.35395 | load: 0.00016 [I 2022-10-27 11:56:55] ...trap/engines/engine.py.321: eval: epoch 65 | batch 450/510 [I 2022-10-27 11:56:55] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 11:56:55] ...trap/engines/engine.py.326: process: 0.29092 | load: 0.00019 [I 2022-10-27 11:56:58] ...trap/engines/engine.py.321: eval: epoch 65 | batch 460/510 [I 2022-10-27 11:56:58] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 11:56:58] ...trap/engines/engine.py.326: process: 0.28555 | load: 0.00016 [I 2022-10-27 11:57:01] ...trap/engines/engine.py.321: eval: epoch 65 | batch 470/510 [I 2022-10-27 11:57:01] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 11:57:01] ...trap/engines/engine.py.326: process: 0.28310 | load: 0.00016 [I 2022-10-27 11:57:04] ...trap/engines/engine.py.321: eval: epoch 65 | batch 480/510 [I 2022-10-27 11:57:04] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 11:57:04] ...trap/engines/engine.py.326: process: 0.28907 | load: 0.00016 [I 2022-10-27 11:57:07] ...trap/engines/engine.py.321: eval: epoch 65 | batch 490/510 [I 2022-10-27 11:57:07] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 11:57:07] ...trap/engines/engine.py.326: process: 0.29055 | load: 0.00021 [I 2022-10-27 11:57:10] ...trap/engines/engine.py.321: eval: epoch 65 | batch 500/510 [I 2022-10-27 11:57:10] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 11:57:10] ...trap/engines/engine.py.326: process: 0.28759 | load: 0.00017 [I 2022-10-27 11:57:12] ...trap/engines/engine.py.321: eval: epoch 65 | batch 510/510 [I 2022-10-27 11:57:12] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 11:57:12] ...trap/engines/engine.py.326: process: 0.28250 | load: 0.00012 [S 2022-10-27 11:57:13] ...trap/engines/engine.py.132: eval_epoch.epoch: 65 [I 2022-10-27 11:57:13] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.718900 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905300 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936700 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.712400 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904200 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936200 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013874 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012410 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010277 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.018720 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009228 [S 2022-10-27 11:57:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009282 [I 2022-10-27 11:57:19] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 11:57:19] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 11:57:19] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 11:57:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 11:57:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 11:57:22] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 11:57:23] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 11:57:29] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 11:57:29] ...trap/engines/engine.py.163: Training model on trainset for epoch 66 [I 2022-10-27 11:57:29] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 11:57:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 0/2304 [I 2022-10-27 11:57:32] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:45 [I 2022-10-27 11:57:32] ...trap/engines/engine.py.229: process: 2.78037 | load: 2.05246 [I 2022-10-27 11:57:32] ...trap/engines/engine.py.230: loss: 0.13246 [I 2022-10-27 11:57:32] ...optimizers/trijoint.py.151: total_norm: 8.598533 [I 2022-10-27 11:57:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 10/2304 [I 2022-10-27 11:57:38] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:21 [I 2022-10-27 11:57:38] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00444 [I 2022-10-27 11:57:38] ...trap/engines/engine.py.230: loss: 0.11530 [I 2022-10-27 11:57:38] ...optimizers/trijoint.py.151: total_norm: 15.124733 [I 2022-10-27 11:57:43] ...trap/engines/engine.py.224: train: epoch 66 | batch 20/2304 [I 2022-10-27 11:57:43] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:04 [I 2022-10-27 11:57:43] ...trap/engines/engine.py.229: process: 0.54198 | load: 0.00388 [I 2022-10-27 11:57:43] ...trap/engines/engine.py.230: loss: 0.13519 [I 2022-10-27 11:57:43] ...optimizers/trijoint.py.151: total_norm: 11.716409 [I 2022-10-27 11:57:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 30/2304 [I 2022-10-27 11:57:49] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:08 [I 2022-10-27 11:57:49] ...trap/engines/engine.py.229: process: 0.54437 | load: 0.00338 [I 2022-10-27 11:57:49] ...trap/engines/engine.py.230: loss: 0.21153 [I 2022-10-27 11:57:49] ...optimizers/trijoint.py.151: total_norm: 26.557829 [I 2022-10-27 11:57:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 40/2304 [I 2022-10-27 11:57:55] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:02 [I 2022-10-27 11:57:55] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00282 [I 2022-10-27 11:57:55] ...trap/engines/engine.py.230: loss: 0.18606 [I 2022-10-27 11:57:55] ...optimizers/trijoint.py.151: total_norm: 20.301394 [I 2022-10-27 11:58:00] ...trap/engines/engine.py.224: train: epoch 66 | batch 50/2304 [I 2022-10-27 11:58:00] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:49 [I 2022-10-27 11:58:00] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00287 [I 2022-10-27 11:58:00] ...trap/engines/engine.py.230: loss: 0.24376 [I 2022-10-27 11:58:00] ...optimizers/trijoint.py.151: total_norm: 30.563728 [I 2022-10-27 11:58:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 60/2304 [I 2022-10-27 11:58:06] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:45 [I 2022-10-27 11:58:06] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00306 [I 2022-10-27 11:58:06] ...trap/engines/engine.py.230: loss: 0.11564 [I 2022-10-27 11:58:06] ...optimizers/trijoint.py.151: total_norm: 11.993148 [I 2022-10-27 11:58:11] ...trap/engines/engine.py.224: train: epoch 66 | batch 70/2304 [I 2022-10-27 11:58:11] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:57 [I 2022-10-27 11:58:11] ...trap/engines/engine.py.229: process: 0.55672 | load: 0.00274 [I 2022-10-27 11:58:11] ...trap/engines/engine.py.230: loss: 0.11053 [I 2022-10-27 11:58:11] ...optimizers/trijoint.py.151: total_norm: 9.977986 [I 2022-10-27 11:58:17] ...trap/engines/engine.py.224: train: epoch 66 | batch 80/2304 [I 2022-10-27 11:58:17] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:43 [I 2022-10-27 11:58:17] ...trap/engines/engine.py.229: process: 0.55790 | load: 0.00282 [I 2022-10-27 11:58:17] ...trap/engines/engine.py.230: loss: 0.14814 [I 2022-10-27 11:58:17] ...optimizers/trijoint.py.151: total_norm: 12.318161 [I 2022-10-27 11:58:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 90/2304 [I 2022-10-27 11:58:22] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:47 [I 2022-10-27 11:58:22] ...trap/engines/engine.py.229: process: 0.56303 | load: 0.00291 [I 2022-10-27 11:58:22] ...trap/engines/engine.py.230: loss: 0.10342 [I 2022-10-27 11:58:22] ...optimizers/trijoint.py.151: total_norm: 14.928274 [I 2022-10-27 11:58:28] ...trap/engines/engine.py.224: train: epoch 66 | batch 100/2304 [I 2022-10-27 11:58:28] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:27 [I 2022-10-27 11:58:28] ...trap/engines/engine.py.229: process: 0.56331 | load: 0.00265 [I 2022-10-27 11:58:28] ...trap/engines/engine.py.230: loss: 0.11777 [I 2022-10-27 11:58:28] ...optimizers/trijoint.py.151: total_norm: 12.622920 [I 2022-10-27 11:58:34] ...trap/engines/engine.py.224: train: epoch 66 | batch 110/2304 [I 2022-10-27 11:58:34] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:22 [I 2022-10-27 11:58:34] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00275 [I 2022-10-27 11:58:34] ...trap/engines/engine.py.230: loss: 0.13134 [I 2022-10-27 11:58:34] ...optimizers/trijoint.py.151: total_norm: 15.081373 [I 2022-10-27 11:58:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 120/2304 [I 2022-10-27 11:58:39] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:18 [I 2022-10-27 11:58:39] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00265 [I 2022-10-27 11:58:39] ...trap/engines/engine.py.230: loss: 0.14671 [I 2022-10-27 11:58:39] ...optimizers/trijoint.py.151: total_norm: 13.371638 [I 2022-10-27 11:58:45] ...trap/engines/engine.py.224: train: epoch 66 | batch 130/2304 [I 2022-10-27 11:58:45] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:23 [I 2022-10-27 11:58:45] ...trap/engines/engine.py.229: process: 0.54340 | load: 0.00264 [I 2022-10-27 11:58:45] ...trap/engines/engine.py.230: loss: 0.18193 [I 2022-10-27 11:58:45] ...optimizers/trijoint.py.151: total_norm: 15.637199 [I 2022-10-27 11:58:51] ...trap/engines/engine.py.224: train: epoch 66 | batch 140/2304 [I 2022-10-27 11:58:51] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:09 [I 2022-10-27 11:58:51] ...trap/engines/engine.py.229: process: 0.55387 | load: 0.00260 [I 2022-10-27 11:58:51] ...trap/engines/engine.py.230: loss: 0.11717 [I 2022-10-27 11:58:51] ...optimizers/trijoint.py.151: total_norm: 11.799973 [I 2022-10-27 11:58:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 150/2304 [I 2022-10-27 11:58:56] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:56 [I 2022-10-27 11:58:56] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00255 [I 2022-10-27 11:58:56] ...trap/engines/engine.py.230: loss: 0.11735 [I 2022-10-27 11:58:56] ...optimizers/trijoint.py.151: total_norm: 9.985126 [I 2022-10-27 11:59:02] ...trap/engines/engine.py.224: train: epoch 66 | batch 160/2304 [I 2022-10-27 11:59:02] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:00 [I 2022-10-27 11:59:02] ...trap/engines/engine.py.229: process: 0.55764 | load: 0.00260 [I 2022-10-27 11:59:02] ...trap/engines/engine.py.230: loss: 0.12202 [I 2022-10-27 11:59:02] ...optimizers/trijoint.py.151: total_norm: 9.454129 [I 2022-10-27 11:59:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 170/2304 [I 2022-10-27 11:59:07] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:55 [I 2022-10-27 11:59:07] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00255 [I 2022-10-27 11:59:07] ...trap/engines/engine.py.230: loss: 0.13537 [I 2022-10-27 11:59:07] ...optimizers/trijoint.py.151: total_norm: 13.134350 [I 2022-10-27 11:59:13] ...trap/engines/engine.py.224: train: epoch 66 | batch 180/2304 [I 2022-10-27 11:59:13] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:54 [I 2022-10-27 11:59:13] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00260 [I 2022-10-27 11:59:13] ...trap/engines/engine.py.230: loss: 0.10923 [I 2022-10-27 11:59:13] ...optimizers/trijoint.py.151: total_norm: 10.107585 [I 2022-10-27 11:59:19] ...trap/engines/engine.py.224: train: epoch 66 | batch 190/2304 [I 2022-10-27 11:59:19] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:45 [I 2022-10-27 11:59:19] ...trap/engines/engine.py.229: process: 0.54725 | load: 0.00266 [I 2022-10-27 11:59:19] ...trap/engines/engine.py.230: loss: 0.16834 [I 2022-10-27 11:59:19] ...optimizers/trijoint.py.151: total_norm: 15.712913 [I 2022-10-27 11:59:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 200/2304 [I 2022-10-27 11:59:24] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:51 [I 2022-10-27 11:59:24] ...trap/engines/engine.py.229: process: 0.57076 | load: 0.00246 [I 2022-10-27 11:59:24] ...trap/engines/engine.py.230: loss: 0.12536 [I 2022-10-27 11:59:24] ...optimizers/trijoint.py.151: total_norm: 18.487564 [I 2022-10-27 11:59:30] ...trap/engines/engine.py.224: train: epoch 66 | batch 210/2304 [I 2022-10-27 11:59:30] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:36 [I 2022-10-27 11:59:30] ...trap/engines/engine.py.229: process: 0.56655 | load: 0.00264 [I 2022-10-27 11:59:30] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-27 11:59:30] ...optimizers/trijoint.py.151: total_norm: 17.418264 [I 2022-10-27 11:59:36] ...trap/engines/engine.py.224: train: epoch 66 | batch 220/2304 [I 2022-10-27 11:59:36] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:23 [I 2022-10-27 11:59:36] ...trap/engines/engine.py.229: process: 0.56024 | load: 0.00255 [I 2022-10-27 11:59:36] ...trap/engines/engine.py.230: loss: 0.11539 [I 2022-10-27 11:59:36] ...optimizers/trijoint.py.151: total_norm: 34.726269 [I 2022-10-27 11:59:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 230/2304 [I 2022-10-27 11:59:41] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:13 [I 2022-10-27 11:59:41] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00241 [I 2022-10-27 11:59:41] ...trap/engines/engine.py.230: loss: 0.15783 [I 2022-10-27 11:59:41] ...optimizers/trijoint.py.151: total_norm: 27.174196 [I 2022-10-27 11:59:47] ...trap/engines/engine.py.224: train: epoch 66 | batch 240/2304 [I 2022-10-27 11:59:47] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:05 [I 2022-10-27 11:59:47] ...trap/engines/engine.py.229: process: 0.54544 | load: 0.00242 [I 2022-10-27 11:59:47] ...trap/engines/engine.py.230: loss: 0.11315 [I 2022-10-27 11:59:47] ...optimizers/trijoint.py.151: total_norm: 15.931718 [I 2022-10-27 11:59:52] ...trap/engines/engine.py.224: train: epoch 66 | batch 250/2304 [I 2022-10-27 11:59:52] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:10 [I 2022-10-27 11:59:52] ...trap/engines/engine.py.229: process: 0.56966 | load: 0.00247 [I 2022-10-27 11:59:52] ...trap/engines/engine.py.230: loss: 0.16775 [I 2022-10-27 11:59:52] ...optimizers/trijoint.py.151: total_norm: 31.803818 [I 2022-10-27 11:59:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 260/2304 [I 2022-10-27 11:59:58] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:59 [I 2022-10-27 11:59:58] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00255 [I 2022-10-27 11:59:58] ...trap/engines/engine.py.230: loss: 0.11261 [I 2022-10-27 11:59:58] ...optimizers/trijoint.py.151: total_norm: 13.115473 [I 2022-10-27 12:00:04] ...trap/engines/engine.py.224: train: epoch 66 | batch 270/2304 [I 2022-10-27 12:00:04] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:19:04 [I 2022-10-27 12:00:04] ...trap/engines/engine.py.229: process: 0.56101 | load: 0.00246 [I 2022-10-27 12:00:04] ...trap/engines/engine.py.230: loss: 0.17376 [I 2022-10-27 12:00:04] ...optimizers/trijoint.py.151: total_norm: 13.769951 [I 2022-10-27 12:00:09] ...trap/engines/engine.py.224: train: epoch 66 | batch 280/2304 [I 2022-10-27 12:00:09] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:59 [I 2022-10-27 12:00:09] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00258 [I 2022-10-27 12:00:09] ...trap/engines/engine.py.230: loss: 0.18190 [I 2022-10-27 12:00:09] ...optimizers/trijoint.py.151: total_norm: 23.978991 [I 2022-10-27 12:00:15] ...trap/engines/engine.py.224: train: epoch 66 | batch 290/2304 [I 2022-10-27 12:00:15] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:43 [I 2022-10-27 12:00:15] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00250 [I 2022-10-27 12:00:15] ...trap/engines/engine.py.230: loss: 0.12947 [I 2022-10-27 12:00:15] ...optimizers/trijoint.py.151: total_norm: 64.076683 [I 2022-10-27 12:00:20] ...trap/engines/engine.py.224: train: epoch 66 | batch 300/2304 [I 2022-10-27 12:00:20] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:35 [I 2022-10-27 12:00:20] ...trap/engines/engine.py.229: process: 0.54547 | load: 0.00245 [I 2022-10-27 12:00:20] ...trap/engines/engine.py.230: loss: 0.12351 [I 2022-10-27 12:00:20] ...optimizers/trijoint.py.151: total_norm: 13.453641 [I 2022-10-27 12:00:26] ...trap/engines/engine.py.224: train: epoch 66 | batch 310/2304 [I 2022-10-27 12:00:26] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:26 [I 2022-10-27 12:00:26] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00262 [I 2022-10-27 12:00:26] ...trap/engines/engine.py.230: loss: 0.11584 [I 2022-10-27 12:00:26] ...optimizers/trijoint.py.151: total_norm: 11.492882 [I 2022-10-27 12:00:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 320/2304 [I 2022-10-27 12:00:32] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:31 [I 2022-10-27 12:00:32] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00243 [I 2022-10-27 12:00:32] ...trap/engines/engine.py.230: loss: 0.12551 [I 2022-10-27 12:00:32] ...optimizers/trijoint.py.151: total_norm: 14.594690 [I 2022-10-27 12:00:37] ...trap/engines/engine.py.224: train: epoch 66 | batch 330/2304 [I 2022-10-27 12:00:37] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:17 [I 2022-10-27 12:00:37] ...trap/engines/engine.py.229: process: 0.54624 | load: 0.00253 [I 2022-10-27 12:00:37] ...trap/engines/engine.py.230: loss: 0.18179 [I 2022-10-27 12:00:37] ...optimizers/trijoint.py.151: total_norm: 18.715425 [I 2022-10-27 12:00:43] ...trap/engines/engine.py.224: train: epoch 66 | batch 340/2304 [I 2022-10-27 12:00:43] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:18 [I 2022-10-27 12:00:43] ...trap/engines/engine.py.229: process: 0.56691 | load: 0.00246 [I 2022-10-27 12:00:43] ...trap/engines/engine.py.230: loss: 0.17017 [I 2022-10-27 12:00:43] ...optimizers/trijoint.py.151: total_norm: 31.710783 [I 2022-10-27 12:00:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 350/2304 [I 2022-10-27 12:00:48] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:13 [I 2022-10-27 12:00:48] ...trap/engines/engine.py.229: process: 0.56155 | load: 0.00243 [I 2022-10-27 12:00:48] ...trap/engines/engine.py.230: loss: 0.19717 [I 2022-10-27 12:00:48] ...optimizers/trijoint.py.151: total_norm: 24.222843 [I 2022-10-27 12:00:54] ...trap/engines/engine.py.224: train: epoch 66 | batch 360/2304 [I 2022-10-27 12:00:54] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:20 [I 2022-10-27 12:00:54] ...trap/engines/engine.py.229: process: 0.55936 | load: 0.00242 [I 2022-10-27 12:00:54] ...trap/engines/engine.py.230: loss: 0.17616 [I 2022-10-27 12:00:54] ...optimizers/trijoint.py.151: total_norm: 24.448269 [I 2022-10-27 12:01:00] ...trap/engines/engine.py.224: train: epoch 66 | batch 370/2304 [I 2022-10-27 12:01:00] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:17:56 [I 2022-10-27 12:01:00] ...trap/engines/engine.py.229: process: 0.55044 | load: 0.00258 [I 2022-10-27 12:01:00] ...trap/engines/engine.py.230: loss: 0.14761 [I 2022-10-27 12:01:00] ...optimizers/trijoint.py.151: total_norm: 27.176905 [I 2022-10-27 12:01:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 380/2304 [I 2022-10-27 12:01:05] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:18:00 [I 2022-10-27 12:01:05] ...trap/engines/engine.py.229: process: 0.56862 | load: 0.00257 [I 2022-10-27 12:01:05] ...trap/engines/engine.py.230: loss: 0.18557 [I 2022-10-27 12:01:05] ...optimizers/trijoint.py.151: total_norm: 17.213118 [I 2022-10-27 12:01:11] ...trap/engines/engine.py.224: train: epoch 66 | batch 390/2304 [I 2022-10-27 12:01:11] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:49 [I 2022-10-27 12:01:11] ...trap/engines/engine.py.229: process: 0.54575 | load: 0.00243 [I 2022-10-27 12:01:11] ...trap/engines/engine.py.230: loss: 0.11162 [I 2022-10-27 12:01:11] ...optimizers/trijoint.py.151: total_norm: 14.371080 [I 2022-10-27 12:01:17] ...trap/engines/engine.py.224: train: epoch 66 | batch 400/2304 [I 2022-10-27 12:01:17] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:40 [I 2022-10-27 12:01:17] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00245 [I 2022-10-27 12:01:17] ...trap/engines/engine.py.230: loss: 0.11771 [I 2022-10-27 12:01:17] ...optimizers/trijoint.py.151: total_norm: 20.819181 [I 2022-10-27 12:01:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 410/2304 [I 2022-10-27 12:01:22] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:38 [I 2022-10-27 12:01:22] ...trap/engines/engine.py.229: process: 0.55662 | load: 0.00242 [I 2022-10-27 12:01:22] ...trap/engines/engine.py.230: loss: 0.20064 [I 2022-10-27 12:01:22] ...optimizers/trijoint.py.151: total_norm: 20.843657 [I 2022-10-27 12:01:28] ...trap/engines/engine.py.224: train: epoch 66 | batch 420/2304 [I 2022-10-27 12:01:28] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:28 [I 2022-10-27 12:01:28] ...trap/engines/engine.py.229: process: 0.54390 | load: 0.00247 [I 2022-10-27 12:01:28] ...trap/engines/engine.py.230: loss: 0.09856 [I 2022-10-27 12:01:28] ...optimizers/trijoint.py.151: total_norm: 7.641510 [I 2022-10-27 12:01:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 430/2304 [I 2022-10-27 12:01:33] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:24 [I 2022-10-27 12:01:33] ...trap/engines/engine.py.229: process: 0.55594 | load: 0.00246 [I 2022-10-27 12:01:33] ...trap/engines/engine.py.230: loss: 0.15786 [I 2022-10-27 12:01:33] ...optimizers/trijoint.py.151: total_norm: 16.353868 [I 2022-10-27 12:01:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 440/2304 [I 2022-10-27 12:01:39] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:22 [I 2022-10-27 12:01:39] ...trap/engines/engine.py.229: process: 0.55014 | load: 0.00240 [I 2022-10-27 12:01:39] ...trap/engines/engine.py.230: loss: 0.11682 [I 2022-10-27 12:01:39] ...optimizers/trijoint.py.151: total_norm: 12.661256 [I 2022-10-27 12:01:45] ...trap/engines/engine.py.224: train: epoch 66 | batch 450/2304 [I 2022-10-27 12:01:45] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:11 [I 2022-10-27 12:01:45] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00241 [I 2022-10-27 12:01:45] ...trap/engines/engine.py.230: loss: 0.29334 [I 2022-10-27 12:01:45] ...optimizers/trijoint.py.151: total_norm: 50.917145 [I 2022-10-27 12:01:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 460/2304 [I 2022-10-27 12:01:50] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:08 [I 2022-10-27 12:01:50] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00243 [I 2022-10-27 12:01:50] ...trap/engines/engine.py.230: loss: 0.11855 [I 2022-10-27 12:01:50] ...optimizers/trijoint.py.151: total_norm: 17.882448 [I 2022-10-27 12:01:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 470/2304 [I 2022-10-27 12:01:56] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:06 [I 2022-10-27 12:01:56] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00242 [I 2022-10-27 12:01:56] ...trap/engines/engine.py.230: loss: 0.13795 [I 2022-10-27 12:01:56] ...optimizers/trijoint.py.151: total_norm: 11.514969 [I 2022-10-27 12:02:01] ...trap/engines/engine.py.224: train: epoch 66 | batch 480/2304 [I 2022-10-27 12:02:01] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:59 [I 2022-10-27 12:02:01] ...trap/engines/engine.py.229: process: 0.54442 | load: 0.00250 [I 2022-10-27 12:02:01] ...trap/engines/engine.py.230: loss: 0.16939 [I 2022-10-27 12:02:01] ...optimizers/trijoint.py.151: total_norm: 14.986820 [I 2022-10-27 12:02:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 490/2304 [I 2022-10-27 12:02:07] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:59 [I 2022-10-27 12:02:07] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00245 [I 2022-10-27 12:02:07] ...trap/engines/engine.py.230: loss: 0.17757 [I 2022-10-27 12:02:07] ...optimizers/trijoint.py.151: total_norm: 12.561773 [I 2022-10-27 12:02:13] ...trap/engines/engine.py.224: train: epoch 66 | batch 500/2304 [I 2022-10-27 12:02:13] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:46 [I 2022-10-27 12:02:13] ...trap/engines/engine.py.229: process: 0.55174 | load: 0.00255 [I 2022-10-27 12:02:13] ...trap/engines/engine.py.230: loss: 0.18280 [I 2022-10-27 12:02:13] ...optimizers/trijoint.py.151: total_norm: 28.622763 [I 2022-10-27 12:02:18] ...trap/engines/engine.py.224: train: epoch 66 | batch 510/2304 [I 2022-10-27 12:02:18] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:44 [I 2022-10-27 12:02:18] ...trap/engines/engine.py.229: process: 0.54316 | load: 0.00242 [I 2022-10-27 12:02:18] ...trap/engines/engine.py.230: loss: 0.17205 [I 2022-10-27 12:02:18] ...optimizers/trijoint.py.151: total_norm: 16.749462 [I 2022-10-27 12:02:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 520/2304 [I 2022-10-27 12:02:24] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:41 [I 2022-10-27 12:02:24] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00246 [I 2022-10-27 12:02:24] ...trap/engines/engine.py.230: loss: 0.14136 [I 2022-10-27 12:02:24] ...optimizers/trijoint.py.151: total_norm: 26.734747 [I 2022-10-27 12:02:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 530/2304 [I 2022-10-27 12:02:29] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:31 [I 2022-10-27 12:02:29] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00248 [I 2022-10-27 12:02:29] ...trap/engines/engine.py.230: loss: 0.14010 [I 2022-10-27 12:02:29] ...optimizers/trijoint.py.151: total_norm: 36.631573 [I 2022-10-27 12:02:35] ...trap/engines/engine.py.224: train: epoch 66 | batch 540/2304 [I 2022-10-27 12:02:35] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:27 [I 2022-10-27 12:02:35] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00250 [I 2022-10-27 12:02:35] ...trap/engines/engine.py.230: loss: 0.13645 [I 2022-10-27 12:02:35] ...optimizers/trijoint.py.151: total_norm: 9.382715 [I 2022-10-27 12:02:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 550/2304 [I 2022-10-27 12:02:41] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:27 [I 2022-10-27 12:02:41] ...trap/engines/engine.py.229: process: 0.56895 | load: 0.00243 [I 2022-10-27 12:02:41] ...trap/engines/engine.py.230: loss: 0.12912 [I 2022-10-27 12:02:41] ...optimizers/trijoint.py.151: total_norm: 26.658493 [I 2022-10-27 12:02:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 560/2304 [I 2022-10-27 12:02:46] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:21 [I 2022-10-27 12:02:46] ...trap/engines/engine.py.229: process: 0.56287 | load: 0.00245 [I 2022-10-27 12:02:46] ...trap/engines/engine.py.230: loss: 0.16909 [I 2022-10-27 12:02:46] ...optimizers/trijoint.py.151: total_norm: 15.277826 [I 2022-10-27 12:02:52] ...trap/engines/engine.py.224: train: epoch 66 | batch 570/2304 [I 2022-10-27 12:02:52] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:11 [I 2022-10-27 12:02:52] ...trap/engines/engine.py.229: process: 0.56352 | load: 0.00242 [I 2022-10-27 12:02:52] ...trap/engines/engine.py.230: loss: 0.13973 [I 2022-10-27 12:02:52] ...optimizers/trijoint.py.151: total_norm: 8.407488 [I 2022-10-27 12:02:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 580/2304 [I 2022-10-27 12:02:57] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:16:06 [I 2022-10-27 12:02:57] ...trap/engines/engine.py.229: process: 0.57079 | load: 0.00247 [I 2022-10-27 12:02:57] ...trap/engines/engine.py.230: loss: 0.13300 [I 2022-10-27 12:02:57] ...optimizers/trijoint.py.151: total_norm: 19.789654 [I 2022-10-27 12:03:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 590/2304 [I 2022-10-27 12:03:03] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:00 [I 2022-10-27 12:03:03] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00241 [I 2022-10-27 12:03:03] ...trap/engines/engine.py.230: loss: 0.12430 [I 2022-10-27 12:03:03] ...optimizers/trijoint.py.151: total_norm: 18.861589 [I 2022-10-27 12:03:09] ...trap/engines/engine.py.224: train: epoch 66 | batch 600/2304 [I 2022-10-27 12:03:09] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:49 [I 2022-10-27 12:03:09] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00250 [I 2022-10-27 12:03:09] ...trap/engines/engine.py.230: loss: 0.16835 [I 2022-10-27 12:03:09] ...optimizers/trijoint.py.151: total_norm: 21.750313 [I 2022-10-27 12:03:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 610/2304 [I 2022-10-27 12:03:14] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:51 [I 2022-10-27 12:03:14] ...trap/engines/engine.py.229: process: 0.56517 | load: 0.00248 [I 2022-10-27 12:03:14] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-27 12:03:14] ...optimizers/trijoint.py.151: total_norm: 17.479523 [I 2022-10-27 12:03:20] ...trap/engines/engine.py.224: train: epoch 66 | batch 620/2304 [I 2022-10-27 12:03:20] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:36 [I 2022-10-27 12:03:20] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00244 [I 2022-10-27 12:03:20] ...trap/engines/engine.py.230: loss: 0.11548 [I 2022-10-27 12:03:20] ...optimizers/trijoint.py.151: total_norm: 14.051849 [I 2022-10-27 12:03:25] ...trap/engines/engine.py.224: train: epoch 66 | batch 630/2304 [I 2022-10-27 12:03:25] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:35 [I 2022-10-27 12:03:25] ...trap/engines/engine.py.229: process: 0.56655 | load: 0.00250 [I 2022-10-27 12:03:25] ...trap/engines/engine.py.230: loss: 0.14479 [I 2022-10-27 12:03:25] ...optimizers/trijoint.py.151: total_norm: 14.447674 [I 2022-10-27 12:03:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 640/2304 [I 2022-10-27 12:03:31] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:38 [I 2022-10-27 12:03:31] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00244 [I 2022-10-27 12:03:31] ...trap/engines/engine.py.230: loss: 0.12692 [I 2022-10-27 12:03:31] ...optimizers/trijoint.py.151: total_norm: 19.774952 [I 2022-10-27 12:03:37] ...trap/engines/engine.py.224: train: epoch 66 | batch 650/2304 [I 2022-10-27 12:03:37] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:31 [I 2022-10-27 12:03:37] ...trap/engines/engine.py.229: process: 0.56761 | load: 0.00243 [I 2022-10-27 12:03:37] ...trap/engines/engine.py.230: loss: 0.16592 [I 2022-10-27 12:03:37] ...optimizers/trijoint.py.151: total_norm: 19.188042 [I 2022-10-27 12:03:42] ...trap/engines/engine.py.224: train: epoch 66 | batch 660/2304 [I 2022-10-27 12:03:42] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:25 [I 2022-10-27 12:03:42] ...trap/engines/engine.py.229: process: 0.55298 | load: 0.00249 [I 2022-10-27 12:03:42] ...trap/engines/engine.py.230: loss: 0.13744 [I 2022-10-27 12:03:42] ...optimizers/trijoint.py.151: total_norm: 11.877023 [I 2022-10-27 12:03:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 670/2304 [I 2022-10-27 12:03:48] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:17 [I 2022-10-27 12:03:48] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00258 [I 2022-10-27 12:03:48] ...trap/engines/engine.py.230: loss: 0.15629 [I 2022-10-27 12:03:48] ...optimizers/trijoint.py.151: total_norm: 11.771064 [I 2022-10-27 12:03:54] ...trap/engines/engine.py.224: train: epoch 66 | batch 680/2304 [I 2022-10-27 12:03:54] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:14:56 [I 2022-10-27 12:03:54] ...trap/engines/engine.py.229: process: 0.54703 | load: 0.00248 [I 2022-10-27 12:03:54] ...trap/engines/engine.py.230: loss: 0.10110 [I 2022-10-27 12:03:54] ...optimizers/trijoint.py.151: total_norm: 21.439129 [I 2022-10-27 12:03:59] ...trap/engines/engine.py.224: train: epoch 66 | batch 690/2304 [I 2022-10-27 12:03:59] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:01 [I 2022-10-27 12:03:59] ...trap/engines/engine.py.229: process: 0.56704 | load: 0.00248 [I 2022-10-27 12:03:59] ...trap/engines/engine.py.230: loss: 0.15103 [I 2022-10-27 12:03:59] ...optimizers/trijoint.py.151: total_norm: 15.274726 [I 2022-10-27 12:04:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 700/2304 [I 2022-10-27 12:04:05] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:14:58 [I 2022-10-27 12:04:05] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00251 [I 2022-10-27 12:04:05] ...trap/engines/engine.py.230: loss: 0.24345 [I 2022-10-27 12:04:05] ...optimizers/trijoint.py.151: total_norm: 21.175400 [I 2022-10-27 12:04:10] ...trap/engines/engine.py.224: train: epoch 66 | batch 710/2304 [I 2022-10-27 12:04:10] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:49 [I 2022-10-27 12:04:10] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00254 [I 2022-10-27 12:04:10] ...trap/engines/engine.py.230: loss: 0.12371 [I 2022-10-27 12:04:10] ...optimizers/trijoint.py.151: total_norm: 11.001472 [I 2022-10-27 12:04:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 720/2304 [I 2022-10-27 12:04:16] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:45 [I 2022-10-27 12:04:16] ...trap/engines/engine.py.229: process: 0.56686 | load: 0.00245 [I 2022-10-27 12:04:16] ...trap/engines/engine.py.230: loss: 0.16297 [I 2022-10-27 12:04:16] ...optimizers/trijoint.py.151: total_norm: 18.968149 [I 2022-10-27 12:04:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 730/2304 [I 2022-10-27 12:04:22] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:14:37 [I 2022-10-27 12:04:22] ...trap/engines/engine.py.229: process: 0.57263 | load: 0.00244 [I 2022-10-27 12:04:22] ...trap/engines/engine.py.230: loss: 0.15337 [I 2022-10-27 12:04:22] ...optimizers/trijoint.py.151: total_norm: 14.794173 [I 2022-10-27 12:04:27] ...trap/engines/engine.py.224: train: epoch 66 | batch 740/2304 [I 2022-10-27 12:04:27] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:35 [I 2022-10-27 12:04:27] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00250 [I 2022-10-27 12:04:27] ...trap/engines/engine.py.230: loss: 0.15018 [I 2022-10-27 12:04:27] ...optimizers/trijoint.py.151: total_norm: 20.873259 [I 2022-10-27 12:04:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 750/2304 [I 2022-10-27 12:04:33] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:30 [I 2022-10-27 12:04:33] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00248 [I 2022-10-27 12:04:33] ...trap/engines/engine.py.230: loss: 0.16769 [I 2022-10-27 12:04:33] ...optimizers/trijoint.py.151: total_norm: 19.262545 [I 2022-10-27 12:04:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 760/2304 [I 2022-10-27 12:04:38] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:27 [I 2022-10-27 12:04:38] ...trap/engines/engine.py.229: process: 0.57070 | load: 0.00247 [I 2022-10-27 12:04:38] ...trap/engines/engine.py.230: loss: 0.13611 [I 2022-10-27 12:04:38] ...optimizers/trijoint.py.151: total_norm: 11.252363 [I 2022-10-27 12:04:44] ...trap/engines/engine.py.224: train: epoch 66 | batch 770/2304 [I 2022-10-27 12:04:44] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:19 [I 2022-10-27 12:04:44] ...trap/engines/engine.py.229: process: 0.56397 | load: 0.00247 [I 2022-10-27 12:04:44] ...trap/engines/engine.py.230: loss: 0.17246 [I 2022-10-27 12:04:44] ...optimizers/trijoint.py.151: total_norm: 24.638870 [I 2022-10-27 12:04:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 780/2304 [I 2022-10-27 12:04:50] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:10 [I 2022-10-27 12:04:50] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00273 [I 2022-10-27 12:04:50] ...trap/engines/engine.py.230: loss: 0.20256 [I 2022-10-27 12:04:50] ...optimizers/trijoint.py.151: total_norm: 25.597349 [I 2022-10-27 12:04:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 790/2304 [I 2022-10-27 12:04:55] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:06 [I 2022-10-27 12:04:55] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00262 [I 2022-10-27 12:04:55] ...trap/engines/engine.py.230: loss: 0.15531 [I 2022-10-27 12:04:55] ...optimizers/trijoint.py.151: total_norm: 27.562836 [I 2022-10-27 12:05:01] ...trap/engines/engine.py.224: train: epoch 66 | batch 800/2304 [I 2022-10-27 12:05:01] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:14:04 [I 2022-10-27 12:05:01] ...trap/engines/engine.py.229: process: 0.56655 | load: 0.00252 [I 2022-10-27 12:05:01] ...trap/engines/engine.py.230: loss: 0.18557 [I 2022-10-27 12:05:01] ...optimizers/trijoint.py.151: total_norm: 14.316566 [I 2022-10-27 12:05:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 810/2304 [I 2022-10-27 12:05:06] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:58 [I 2022-10-27 12:05:06] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00251 [I 2022-10-27 12:05:06] ...trap/engines/engine.py.230: loss: 0.14380 [I 2022-10-27 12:05:06] ...optimizers/trijoint.py.151: total_norm: 15.994200 [I 2022-10-27 12:05:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 820/2304 [I 2022-10-27 12:05:12] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:52 [I 2022-10-27 12:05:12] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00256 [I 2022-10-27 12:05:12] ...trap/engines/engine.py.230: loss: 0.16691 [I 2022-10-27 12:05:12] ...optimizers/trijoint.py.151: total_norm: 19.471745 [I 2022-10-27 12:05:18] ...trap/engines/engine.py.224: train: epoch 66 | batch 830/2304 [I 2022-10-27 12:05:18] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:37 [I 2022-10-27 12:05:18] ...trap/engines/engine.py.229: process: 0.54320 | load: 0.00249 [I 2022-10-27 12:05:18] ...trap/engines/engine.py.230: loss: 0.09789 [I 2022-10-27 12:05:18] ...optimizers/trijoint.py.151: total_norm: 19.837940 [I 2022-10-27 12:05:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 840/2304 [I 2022-10-27 12:05:23] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:38 [I 2022-10-27 12:05:23] ...trap/engines/engine.py.229: process: 0.57014 | load: 0.00249 [I 2022-10-27 12:05:23] ...trap/engines/engine.py.230: loss: 0.17256 [I 2022-10-27 12:05:23] ...optimizers/trijoint.py.151: total_norm: 14.445825 [I 2022-10-27 12:05:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 850/2304 [I 2022-10-27 12:05:29] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:36 [I 2022-10-27 12:05:29] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00246 [I 2022-10-27 12:05:29] ...trap/engines/engine.py.230: loss: 0.13810 [I 2022-10-27 12:05:29] ...optimizers/trijoint.py.151: total_norm: 14.088028 [I 2022-10-27 12:05:35] ...trap/engines/engine.py.224: train: epoch 66 | batch 860/2304 [I 2022-10-27 12:05:35] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:13:39 [I 2022-10-27 12:05:35] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00247 [I 2022-10-27 12:05:35] ...trap/engines/engine.py.230: loss: 0.10758 [I 2022-10-27 12:05:35] ...optimizers/trijoint.py.151: total_norm: 9.603518 [I 2022-10-27 12:05:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 870/2304 [I 2022-10-27 12:05:40] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:32 [I 2022-10-27 12:05:40] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00242 [I 2022-10-27 12:05:40] ...trap/engines/engine.py.230: loss: 0.14708 [I 2022-10-27 12:05:40] ...optimizers/trijoint.py.151: total_norm: 15.613507 [I 2022-10-27 12:05:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 880/2304 [I 2022-10-27 12:05:46] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:27 [I 2022-10-27 12:05:46] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00245 [I 2022-10-27 12:05:46] ...trap/engines/engine.py.230: loss: 0.17727 [I 2022-10-27 12:05:46] ...optimizers/trijoint.py.151: total_norm: 9.747524 [I 2022-10-27 12:06:10] ...trap/engines/engine.py.224: train: epoch 66 | batch 890/2304 [I 2022-10-27 12:06:10] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:58:43 [I 2022-10-27 12:06:10] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00243 [I 2022-10-27 12:06:10] ...trap/engines/engine.py.230: loss: 0.18124 [I 2022-10-27 12:06:10] ...optimizers/trijoint.py.151: total_norm: 16.252262 [I 2022-10-27 12:06:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 900/2304 [I 2022-10-27 12:06:16] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:18:02 [I 2022-10-27 12:06:16] ...trap/engines/engine.py.229: process: 0.55425 | load: 0.00238 [I 2022-10-27 12:06:16] ...trap/engines/engine.py.230: loss: 0.15275 [I 2022-10-27 12:06:16] ...optimizers/trijoint.py.151: total_norm: 15.323709 [I 2022-10-27 12:06:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 910/2304 [I 2022-10-27 12:06:22] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:13:36 [I 2022-10-27 12:06:22] ...trap/engines/engine.py.229: process: 0.57077 | load: 0.00240 [I 2022-10-27 12:06:22] ...trap/engines/engine.py.230: loss: 0.16299 [I 2022-10-27 12:06:22] ...optimizers/trijoint.py.151: total_norm: 49.910534 [I 2022-10-27 12:06:27] ...trap/engines/engine.py.224: train: epoch 66 | batch 920/2304 [I 2022-10-27 12:06:27] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:54 [I 2022-10-27 12:06:27] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00241 [I 2022-10-27 12:06:27] ...trap/engines/engine.py.230: loss: 0.14973 [I 2022-10-27 12:06:27] ...optimizers/trijoint.py.151: total_norm: 14.562557 [I 2022-10-27 12:06:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 930/2304 [I 2022-10-27 12:06:33] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:48 [I 2022-10-27 12:06:33] ...trap/engines/engine.py.229: process: 0.56778 | load: 0.00241 [I 2022-10-27 12:06:33] ...trap/engines/engine.py.230: loss: 0.15499 [I 2022-10-27 12:06:33] ...optimizers/trijoint.py.151: total_norm: 16.727119 [I 2022-10-27 12:06:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 940/2304 [I 2022-10-27 12:06:38] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:44 [I 2022-10-27 12:06:38] ...trap/engines/engine.py.229: process: 0.56102 | load: 0.00286 [I 2022-10-27 12:06:38] ...trap/engines/engine.py.230: loss: 0.14938 [I 2022-10-27 12:06:38] ...optimizers/trijoint.py.151: total_norm: 32.060257 [I 2022-10-27 12:06:44] ...trap/engines/engine.py.224: train: epoch 66 | batch 950/2304 [I 2022-10-27 12:06:44] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:35 [I 2022-10-27 12:06:44] ...trap/engines/engine.py.229: process: 0.55026 | load: 0.00245 [I 2022-10-27 12:06:44] ...trap/engines/engine.py.230: loss: 0.20318 [I 2022-10-27 12:06:44] ...optimizers/trijoint.py.151: total_norm: 11.799481 [I 2022-10-27 12:06:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 960/2304 [I 2022-10-27 12:06:50] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:40 [I 2022-10-27 12:06:50] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00243 [I 2022-10-27 12:06:50] ...trap/engines/engine.py.230: loss: 0.18062 [I 2022-10-27 12:06:50] ...optimizers/trijoint.py.151: total_norm: 29.164061 [I 2022-10-27 12:06:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 970/2304 [I 2022-10-27 12:06:56] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:38 [I 2022-10-27 12:06:56] ...trap/engines/engine.py.229: process: 0.54976 | load: 0.00245 [I 2022-10-27 12:06:56] ...trap/engines/engine.py.230: loss: 0.11080 [I 2022-10-27 12:06:56] ...optimizers/trijoint.py.151: total_norm: 19.009815 [I 2022-10-27 12:07:01] ...trap/engines/engine.py.224: train: epoch 66 | batch 980/2304 [I 2022-10-27 12:07:01] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:23 [I 2022-10-27 12:07:01] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00251 [I 2022-10-27 12:07:01] ...trap/engines/engine.py.230: loss: 0.10779 [I 2022-10-27 12:07:01] ...optimizers/trijoint.py.151: total_norm: 11.607029 [I 2022-10-27 12:07:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 990/2304 [I 2022-10-27 12:07:07] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:12:16 [I 2022-10-27 12:07:07] ...trap/engines/engine.py.229: process: 0.55834 | load: 0.00263 [I 2022-10-27 12:07:07] ...trap/engines/engine.py.230: loss: 0.11252 [I 2022-10-27 12:07:07] ...optimizers/trijoint.py.151: total_norm: 20.047102 [I 2022-10-27 12:07:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 1000/2304 [I 2022-10-27 12:07:12] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:12:14 [I 2022-10-27 12:07:12] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00245 [I 2022-10-27 12:07:12] ...trap/engines/engine.py.230: loss: 0.12484 [I 2022-10-27 12:07:12] ...optimizers/trijoint.py.151: total_norm: 22.393274 [I 2022-10-27 12:07:18] ...trap/engines/engine.py.224: train: epoch 66 | batch 1010/2304 [I 2022-10-27 12:07:18] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:12:04 [I 2022-10-27 12:07:18] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00242 [I 2022-10-27 12:07:18] ...trap/engines/engine.py.230: loss: 0.17408 [I 2022-10-27 12:07:18] ...optimizers/trijoint.py.151: total_norm: 14.566677 [I 2022-10-27 12:07:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 1020/2304 [I 2022-10-27 12:07:24] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:12:03 [I 2022-10-27 12:07:24] ...trap/engines/engine.py.229: process: 0.56405 | load: 0.00239 [I 2022-10-27 12:07:24] ...trap/engines/engine.py.230: loss: 0.17776 [I 2022-10-27 12:07:24] ...optimizers/trijoint.py.151: total_norm: 25.175873 [I 2022-10-27 12:07:29] ...trap/engines/engine.py.224: train: epoch 66 | batch 1030/2304 [I 2022-10-27 12:07:29] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:55 [I 2022-10-27 12:07:29] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00239 [I 2022-10-27 12:07:29] ...trap/engines/engine.py.230: loss: 0.14855 [I 2022-10-27 12:07:29] ...optimizers/trijoint.py.151: total_norm: 40.613960 [I 2022-10-27 12:07:35] ...trap/engines/engine.py.224: train: epoch 66 | batch 1040/2304 [I 2022-10-27 12:07:35] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:45 [I 2022-10-27 12:07:35] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00242 [I 2022-10-27 12:07:35] ...trap/engines/engine.py.230: loss: 0.15223 [I 2022-10-27 12:07:35] ...optimizers/trijoint.py.151: total_norm: 16.453800 [I 2022-10-27 12:07:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 1050/2304 [I 2022-10-27 12:07:40] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:33 [I 2022-10-27 12:07:40] ...trap/engines/engine.py.229: process: 0.54986 | load: 0.00239 [I 2022-10-27 12:07:40] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-27 12:07:40] ...optimizers/trijoint.py.151: total_norm: 23.975815 [I 2022-10-27 12:07:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 1060/2304 [I 2022-10-27 12:07:46] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:31 [I 2022-10-27 12:07:46] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00247 [I 2022-10-27 12:07:46] ...trap/engines/engine.py.230: loss: 0.19782 [I 2022-10-27 12:07:46] ...optimizers/trijoint.py.151: total_norm: 16.241926 [I 2022-10-27 12:07:52] ...trap/engines/engine.py.224: train: epoch 66 | batch 1070/2304 [I 2022-10-27 12:07:52] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:27 [I 2022-10-27 12:07:52] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00243 [I 2022-10-27 12:07:52] ...trap/engines/engine.py.230: loss: 0.17745 [I 2022-10-27 12:07:52] ...optimizers/trijoint.py.151: total_norm: 25.471298 [I 2022-10-27 12:07:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 1080/2304 [I 2022-10-27 12:07:57] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:22 [I 2022-10-27 12:07:57] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00243 [I 2022-10-27 12:07:57] ...trap/engines/engine.py.230: loss: 0.11159 [I 2022-10-27 12:07:57] ...optimizers/trijoint.py.151: total_norm: 10.506587 [I 2022-10-27 12:08:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 1090/2304 [I 2022-10-27 12:08:03] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:11:14 [I 2022-10-27 12:08:03] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00242 [I 2022-10-27 12:08:03] ...trap/engines/engine.py.230: loss: 0.18036 [I 2022-10-27 12:08:03] ...optimizers/trijoint.py.151: total_norm: 19.634468 [I 2022-10-27 12:08:08] ...trap/engines/engine.py.224: train: epoch 66 | batch 1100/2304 [I 2022-10-27 12:08:08] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:11:17 [I 2022-10-27 12:08:08] ...trap/engines/engine.py.229: process: 0.57094 | load: 0.00242 [I 2022-10-27 12:08:08] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-27 12:08:08] ...optimizers/trijoint.py.151: total_norm: 18.105652 [I 2022-10-27 12:08:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 1110/2304 [I 2022-10-27 12:08:14] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:10:58 [I 2022-10-27 12:08:14] ...trap/engines/engine.py.229: process: 0.53720 | load: 0.00242 [I 2022-10-27 12:08:14] ...trap/engines/engine.py.230: loss: 0.17639 [I 2022-10-27 12:08:14] ...optimizers/trijoint.py.151: total_norm: 13.351637 [I 2022-10-27 12:08:20] ...trap/engines/engine.py.224: train: epoch 66 | batch 1120/2304 [I 2022-10-27 12:08:20] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:58 [I 2022-10-27 12:08:20] ...trap/engines/engine.py.229: process: 0.56775 | load: 0.00244 [I 2022-10-27 12:08:20] ...trap/engines/engine.py.230: loss: 0.16973 [I 2022-10-27 12:08:20] ...optimizers/trijoint.py.151: total_norm: 14.755704 [I 2022-10-27 12:08:25] ...trap/engines/engine.py.224: train: epoch 66 | batch 1130/2304 [I 2022-10-27 12:08:25] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:51 [I 2022-10-27 12:08:25] ...trap/engines/engine.py.229: process: 0.54870 | load: 0.00247 [I 2022-10-27 12:08:25] ...trap/engines/engine.py.230: loss: 0.13527 [I 2022-10-27 12:08:25] ...optimizers/trijoint.py.151: total_norm: 11.014203 [I 2022-10-27 12:08:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 1140/2304 [I 2022-10-27 12:08:31] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:50 [I 2022-10-27 12:08:31] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00248 [I 2022-10-27 12:08:31] ...trap/engines/engine.py.230: loss: 0.23159 [I 2022-10-27 12:08:31] ...optimizers/trijoint.py.151: total_norm: 20.690451 [I 2022-10-27 12:08:36] ...trap/engines/engine.py.224: train: epoch 66 | batch 1150/2304 [I 2022-10-27 12:08:36] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:47 [I 2022-10-27 12:08:36] ...trap/engines/engine.py.229: process: 0.56969 | load: 0.00245 [I 2022-10-27 12:08:36] ...trap/engines/engine.py.230: loss: 0.20753 [I 2022-10-27 12:08:36] ...optimizers/trijoint.py.151: total_norm: 16.073683 [I 2022-10-27 12:08:42] ...trap/engines/engine.py.224: train: epoch 66 | batch 1160/2304 [I 2022-10-27 12:08:42] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:47 [I 2022-10-27 12:08:42] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00243 [I 2022-10-27 12:08:42] ...trap/engines/engine.py.230: loss: 0.20603 [I 2022-10-27 12:08:42] ...optimizers/trijoint.py.151: total_norm: 34.135231 [I 2022-10-27 12:08:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 1170/2304 [I 2022-10-27 12:08:48] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:25 [I 2022-10-27 12:08:48] ...trap/engines/engine.py.229: process: 0.53850 | load: 0.00242 [I 2022-10-27 12:08:48] ...trap/engines/engine.py.230: loss: 0.11843 [I 2022-10-27 12:08:48] ...optimizers/trijoint.py.151: total_norm: 7.320420 [I 2022-10-27 12:08:53] ...trap/engines/engine.py.224: train: epoch 66 | batch 1180/2304 [I 2022-10-27 12:08:53] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:26 [I 2022-10-27 12:08:53] ...trap/engines/engine.py.229: process: 0.57645 | load: 0.00243 [I 2022-10-27 12:08:53] ...trap/engines/engine.py.230: loss: 0.15206 [I 2022-10-27 12:08:53] ...optimizers/trijoint.py.151: total_norm: 21.532207 [I 2022-10-27 12:08:59] ...trap/engines/engine.py.224: train: epoch 66 | batch 1190/2304 [I 2022-10-27 12:08:59] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:10:22 [I 2022-10-27 12:08:59] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00241 [I 2022-10-27 12:08:59] ...trap/engines/engine.py.230: loss: 0.11938 [I 2022-10-27 12:08:59] ...optimizers/trijoint.py.151: total_norm: 24.670242 [I 2022-10-27 12:09:04] ...trap/engines/engine.py.224: train: epoch 66 | batch 1200/2304 [I 2022-10-27 12:09:04] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:10:10 [I 2022-10-27 12:09:04] ...trap/engines/engine.py.229: process: 0.54496 | load: 0.00243 [I 2022-10-27 12:09:04] ...trap/engines/engine.py.230: loss: 0.12311 [I 2022-10-27 12:09:04] ...optimizers/trijoint.py.151: total_norm: 16.028917 [I 2022-10-27 12:09:10] ...trap/engines/engine.py.224: train: epoch 66 | batch 1210/2304 [I 2022-10-27 12:09:10] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:10:07 [I 2022-10-27 12:09:10] ...trap/engines/engine.py.229: process: 0.55858 | load: 0.00241 [I 2022-10-27 12:09:10] ...trap/engines/engine.py.230: loss: 0.14237 [I 2022-10-27 12:09:10] ...optimizers/trijoint.py.151: total_norm: 19.234169 [I 2022-10-27 12:09:15] ...trap/engines/engine.py.224: train: epoch 66 | batch 1220/2304 [I 2022-10-27 12:09:15] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:09:58 [I 2022-10-27 12:09:15] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00244 [I 2022-10-27 12:09:15] ...trap/engines/engine.py.230: loss: 0.12127 [I 2022-10-27 12:09:15] ...optimizers/trijoint.py.151: total_norm: 8.714564 [I 2022-10-27 12:09:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 1230/2304 [I 2022-10-27 12:09:21] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:10:07 [I 2022-10-27 12:09:21] ...trap/engines/engine.py.229: process: 0.56484 | load: 0.00244 [I 2022-10-27 12:09:21] ...trap/engines/engine.py.230: loss: 0.13346 [I 2022-10-27 12:09:21] ...optimizers/trijoint.py.151: total_norm: 13.231076 [I 2022-10-27 12:09:27] ...trap/engines/engine.py.224: train: epoch 66 | batch 1240/2304 [I 2022-10-27 12:09:27] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:53 [I 2022-10-27 12:09:27] ...trap/engines/engine.py.229: process: 0.55317 | load: 0.00242 [I 2022-10-27 12:09:27] ...trap/engines/engine.py.230: loss: 0.12172 [I 2022-10-27 12:09:27] ...optimizers/trijoint.py.151: total_norm: 13.392935 [I 2022-10-27 12:09:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 1250/2304 [I 2022-10-27 12:09:32] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:48 [I 2022-10-27 12:09:32] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00241 [I 2022-10-27 12:09:32] ...trap/engines/engine.py.230: loss: 0.15128 [I 2022-10-27 12:09:32] ...optimizers/trijoint.py.151: total_norm: 10.688494 [I 2022-10-27 12:09:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 1260/2304 [I 2022-10-27 12:09:38] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:40 [I 2022-10-27 12:09:38] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00243 [I 2022-10-27 12:09:38] ...trap/engines/engine.py.230: loss: 0.15524 [I 2022-10-27 12:09:38] ...optimizers/trijoint.py.151: total_norm: 24.136078 [I 2022-10-27 12:09:43] ...trap/engines/engine.py.224: train: epoch 66 | batch 1270/2304 [I 2022-10-27 12:09:43] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:34 [I 2022-10-27 12:09:43] ...trap/engines/engine.py.229: process: 0.54929 | load: 0.00243 [I 2022-10-27 12:09:43] ...trap/engines/engine.py.230: loss: 0.18827 [I 2022-10-27 12:09:43] ...optimizers/trijoint.py.151: total_norm: 15.257435 [I 2022-10-27 12:09:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 1280/2304 [I 2022-10-27 12:09:49] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:29 [I 2022-10-27 12:09:49] ...trap/engines/engine.py.229: process: 0.55422 | load: 0.00241 [I 2022-10-27 12:09:49] ...trap/engines/engine.py.230: loss: 0.14524 [I 2022-10-27 12:09:49] ...optimizers/trijoint.py.151: total_norm: 13.401693 [I 2022-10-27 12:09:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 1290/2304 [I 2022-10-27 12:09:55] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:31 [I 2022-10-27 12:09:55] ...trap/engines/engine.py.229: process: 0.54404 | load: 0.00243 [I 2022-10-27 12:09:55] ...trap/engines/engine.py.230: loss: 0.13281 [I 2022-10-27 12:09:55] ...optimizers/trijoint.py.151: total_norm: 23.452942 [I 2022-10-27 12:10:00] ...trap/engines/engine.py.224: train: epoch 66 | batch 1300/2304 [I 2022-10-27 12:10:00] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:09:20 [I 2022-10-27 12:10:00] ...trap/engines/engine.py.229: process: 0.56655 | load: 0.00243 [I 2022-10-27 12:10:00] ...trap/engines/engine.py.230: loss: 0.11732 [I 2022-10-27 12:10:00] ...optimizers/trijoint.py.151: total_norm: 25.756235 [I 2022-10-27 12:10:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 1310/2304 [I 2022-10-27 12:10:06] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:09:14 [I 2022-10-27 12:10:06] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00240 [I 2022-10-27 12:10:06] ...trap/engines/engine.py.230: loss: 0.13097 [I 2022-10-27 12:10:06] ...optimizers/trijoint.py.151: total_norm: 16.251091 [I 2022-10-27 12:10:11] ...trap/engines/engine.py.224: train: epoch 66 | batch 1320/2304 [I 2022-10-27 12:10:11] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:09:10 [I 2022-10-27 12:10:11] ...trap/engines/engine.py.229: process: 0.57081 | load: 0.00241 [I 2022-10-27 12:10:11] ...trap/engines/engine.py.230: loss: 0.12818 [I 2022-10-27 12:10:11] ...optimizers/trijoint.py.151: total_norm: 14.702825 [I 2022-10-27 12:10:17] ...trap/engines/engine.py.224: train: epoch 66 | batch 1330/2304 [I 2022-10-27 12:10:17] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:59 [I 2022-10-27 12:10:17] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00242 [I 2022-10-27 12:10:17] ...trap/engines/engine.py.230: loss: 0.16278 [I 2022-10-27 12:10:17] ...optimizers/trijoint.py.151: total_norm: 17.291014 [I 2022-10-27 12:10:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 1340/2304 [I 2022-10-27 12:10:22] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:56 [I 2022-10-27 12:10:22] ...trap/engines/engine.py.229: process: 0.56612 | load: 0.00242 [I 2022-10-27 12:10:22] ...trap/engines/engine.py.230: loss: 0.13399 [I 2022-10-27 12:10:22] ...optimizers/trijoint.py.151: total_norm: 12.782343 [I 2022-10-27 12:10:28] ...trap/engines/engine.py.224: train: epoch 66 | batch 1350/2304 [I 2022-10-27 12:10:28] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:57 [I 2022-10-27 12:10:28] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00248 [I 2022-10-27 12:10:28] ...trap/engines/engine.py.230: loss: 0.22123 [I 2022-10-27 12:10:28] ...optimizers/trijoint.py.151: total_norm: 37.492714 [I 2022-10-27 12:10:34] ...trap/engines/engine.py.224: train: epoch 66 | batch 1360/2304 [I 2022-10-27 12:10:34] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:45 [I 2022-10-27 12:10:34] ...trap/engines/engine.py.229: process: 0.55268 | load: 0.00241 [I 2022-10-27 12:10:34] ...trap/engines/engine.py.230: loss: 0.12977 [I 2022-10-27 12:10:34] ...optimizers/trijoint.py.151: total_norm: 13.367717 [I 2022-10-27 12:10:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 1370/2304 [I 2022-10-27 12:10:39] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:39 [I 2022-10-27 12:10:39] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00240 [I 2022-10-27 12:10:39] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-27 12:10:39] ...optimizers/trijoint.py.151: total_norm: 9.940943 [I 2022-10-27 12:10:45] ...trap/engines/engine.py.224: train: epoch 66 | batch 1380/2304 [I 2022-10-27 12:10:45] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:31 [I 2022-10-27 12:10:45] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00241 [I 2022-10-27 12:10:45] ...trap/engines/engine.py.230: loss: 0.09453 [I 2022-10-27 12:10:45] ...optimizers/trijoint.py.151: total_norm: 9.256561 [I 2022-10-27 12:10:50] ...trap/engines/engine.py.224: train: epoch 66 | batch 1390/2304 [I 2022-10-27 12:10:50] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:27 [I 2022-10-27 12:10:50] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00240 [I 2022-10-27 12:10:50] ...trap/engines/engine.py.230: loss: 0.13960 [I 2022-10-27 12:10:50] ...optimizers/trijoint.py.151: total_norm: 16.797930 [I 2022-10-27 12:10:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 1400/2304 [I 2022-10-27 12:10:56] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:22 [I 2022-10-27 12:10:56] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00246 [I 2022-10-27 12:10:56] ...trap/engines/engine.py.230: loss: 0.12350 [I 2022-10-27 12:10:56] ...optimizers/trijoint.py.151: total_norm: 19.097300 [I 2022-10-27 12:11:01] ...trap/engines/engine.py.224: train: epoch 66 | batch 1410/2304 [I 2022-10-27 12:11:01] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:18 [I 2022-10-27 12:11:01] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00245 [I 2022-10-27 12:11:01] ...trap/engines/engine.py.230: loss: 0.16449 [I 2022-10-27 12:11:01] ...optimizers/trijoint.py.151: total_norm: 11.240329 [I 2022-10-27 12:11:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 1420/2304 [I 2022-10-27 12:11:07] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:08:24 [I 2022-10-27 12:11:07] ...trap/engines/engine.py.229: process: 0.55892 | load: 0.00242 [I 2022-10-27 12:11:07] ...trap/engines/engine.py.230: loss: 0.17596 [I 2022-10-27 12:11:07] ...optimizers/trijoint.py.151: total_norm: 47.837799 [I 2022-10-27 12:11:13] ...trap/engines/engine.py.224: train: epoch 66 | batch 1430/2304 [I 2022-10-27 12:11:13] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:08:06 [I 2022-10-27 12:11:13] ...trap/engines/engine.py.229: process: 0.54086 | load: 0.00246 [I 2022-10-27 12:11:13] ...trap/engines/engine.py.230: loss: 0.10527 [I 2022-10-27 12:11:13] ...optimizers/trijoint.py.151: total_norm: 12.973269 [I 2022-10-27 12:11:18] ...trap/engines/engine.py.224: train: epoch 66 | batch 1440/2304 [I 2022-10-27 12:11:18] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:08:01 [I 2022-10-27 12:11:18] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00244 [I 2022-10-27 12:11:18] ...trap/engines/engine.py.230: loss: 0.16952 [I 2022-10-27 12:11:18] ...optimizers/trijoint.py.151: total_norm: 13.520190 [I 2022-10-27 12:11:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 1450/2304 [I 2022-10-27 12:11:24] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:52 [I 2022-10-27 12:11:24] ...trap/engines/engine.py.229: process: 0.53952 | load: 0.00240 [I 2022-10-27 12:11:24] ...trap/engines/engine.py.230: loss: 0.10658 [I 2022-10-27 12:11:24] ...optimizers/trijoint.py.151: total_norm: 7.520341 [I 2022-10-27 12:11:30] ...trap/engines/engine.py.224: train: epoch 66 | batch 1460/2304 [I 2022-10-27 12:11:30] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:47 [I 2022-10-27 12:11:30] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00243 [I 2022-10-27 12:11:30] ...trap/engines/engine.py.230: loss: 0.17396 [I 2022-10-27 12:11:30] ...optimizers/trijoint.py.151: total_norm: 11.166989 [I 2022-10-27 12:11:35] ...trap/engines/engine.py.224: train: epoch 66 | batch 1470/2304 [I 2022-10-27 12:11:35] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:42 [I 2022-10-27 12:11:35] ...trap/engines/engine.py.229: process: 0.55194 | load: 0.00241 [I 2022-10-27 12:11:35] ...trap/engines/engine.py.230: loss: 0.10004 [I 2022-10-27 12:11:35] ...optimizers/trijoint.py.151: total_norm: 7.647135 [I 2022-10-27 12:11:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 1480/2304 [I 2022-10-27 12:11:41] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:39 [I 2022-10-27 12:11:41] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00244 [I 2022-10-27 12:11:41] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-27 12:11:41] ...optimizers/trijoint.py.151: total_norm: 9.990774 [I 2022-10-27 12:11:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 1490/2304 [I 2022-10-27 12:11:46] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:31 [I 2022-10-27 12:11:46] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00242 [I 2022-10-27 12:11:46] ...trap/engines/engine.py.230: loss: 0.11268 [I 2022-10-27 12:11:46] ...optimizers/trijoint.py.151: total_norm: 17.309345 [I 2022-10-27 12:11:52] ...trap/engines/engine.py.224: train: epoch 66 | batch 1500/2304 [I 2022-10-27 12:11:52] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:28 [I 2022-10-27 12:11:52] ...trap/engines/engine.py.229: process: 0.56272 | load: 0.00243 [I 2022-10-27 12:11:52] ...trap/engines/engine.py.230: loss: 0.14150 [I 2022-10-27 12:11:52] ...optimizers/trijoint.py.151: total_norm: 14.763515 [I 2022-10-27 12:11:57] ...trap/engines/engine.py.224: train: epoch 66 | batch 1510/2304 [I 2022-10-27 12:11:57] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:20 [I 2022-10-27 12:11:57] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00243 [I 2022-10-27 12:11:57] ...trap/engines/engine.py.230: loss: 0.17641 [I 2022-10-27 12:11:57] ...optimizers/trijoint.py.151: total_norm: 14.275997 [I 2022-10-27 12:12:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 1520/2304 [I 2022-10-27 12:12:03] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:13 [I 2022-10-27 12:12:03] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00242 [I 2022-10-27 12:12:03] ...trap/engines/engine.py.230: loss: 0.19513 [I 2022-10-27 12:12:03] ...optimizers/trijoint.py.151: total_norm: 14.072869 [I 2022-10-27 12:12:09] ...trap/engines/engine.py.224: train: epoch 66 | batch 1530/2304 [I 2022-10-27 12:12:09] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:07:13 [I 2022-10-27 12:12:09] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00247 [I 2022-10-27 12:12:09] ...trap/engines/engine.py.230: loss: 0.11945 [I 2022-10-27 12:12:09] ...optimizers/trijoint.py.151: total_norm: 44.116955 [I 2022-10-27 12:12:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 1540/2304 [I 2022-10-27 12:12:14] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:07:07 [I 2022-10-27 12:12:14] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00246 [I 2022-10-27 12:12:14] ...trap/engines/engine.py.230: loss: 0.16370 [I 2022-10-27 12:12:14] ...optimizers/trijoint.py.151: total_norm: 13.032143 [I 2022-10-27 12:12:20] ...trap/engines/engine.py.224: train: epoch 66 | batch 1550/2304 [I 2022-10-27 12:12:20] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:58 [I 2022-10-27 12:12:20] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00266 [I 2022-10-27 12:12:20] ...trap/engines/engine.py.230: loss: 0.15539 [I 2022-10-27 12:12:20] ...optimizers/trijoint.py.151: total_norm: 16.261890 [I 2022-10-27 12:12:25] ...trap/engines/engine.py.224: train: epoch 66 | batch 1560/2304 [I 2022-10-27 12:12:25] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:51 [I 2022-10-27 12:12:25] ...trap/engines/engine.py.229: process: 0.54358 | load: 0.00245 [I 2022-10-27 12:12:25] ...trap/engines/engine.py.230: loss: 0.13142 [I 2022-10-27 12:12:25] ...optimizers/trijoint.py.151: total_norm: 11.762892 [I 2022-10-27 12:12:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 1570/2304 [I 2022-10-27 12:12:31] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:43 [I 2022-10-27 12:12:31] ...trap/engines/engine.py.229: process: 0.54477 | load: 0.00243 [I 2022-10-27 12:12:31] ...trap/engines/engine.py.230: loss: 0.12559 [I 2022-10-27 12:12:31] ...optimizers/trijoint.py.151: total_norm: 18.730961 [I 2022-10-27 12:12:36] ...trap/engines/engine.py.224: train: epoch 66 | batch 1580/2304 [I 2022-10-27 12:12:36] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:40 [I 2022-10-27 12:12:36] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00242 [I 2022-10-27 12:12:36] ...trap/engines/engine.py.230: loss: 0.15284 [I 2022-10-27 12:12:36] ...optimizers/trijoint.py.151: total_norm: 18.532701 [I 2022-10-27 12:12:42] ...trap/engines/engine.py.224: train: epoch 66 | batch 1590/2304 [I 2022-10-27 12:12:42] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:40 [I 2022-10-27 12:12:42] ...trap/engines/engine.py.229: process: 0.56802 | load: 0.00243 [I 2022-10-27 12:12:42] ...trap/engines/engine.py.230: loss: 0.18971 [I 2022-10-27 12:12:42] ...optimizers/trijoint.py.151: total_norm: 14.436743 [I 2022-10-27 12:12:47] ...trap/engines/engine.py.224: train: epoch 66 | batch 1600/2304 [I 2022-10-27 12:12:47] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:31 [I 2022-10-27 12:12:47] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00242 [I 2022-10-27 12:12:47] ...trap/engines/engine.py.230: loss: 0.11299 [I 2022-10-27 12:12:47] ...optimizers/trijoint.py.151: total_norm: 12.228659 [I 2022-10-27 12:12:53] ...trap/engines/engine.py.224: train: epoch 66 | batch 1610/2304 [I 2022-10-27 12:12:53] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:31 [I 2022-10-27 12:12:53] ...trap/engines/engine.py.229: process: 0.55897 | load: 0.00245 [I 2022-10-27 12:12:53] ...trap/engines/engine.py.230: loss: 0.14940 [I 2022-10-27 12:12:53] ...optimizers/trijoint.py.151: total_norm: 20.862774 [I 2022-10-27 12:12:59] ...trap/engines/engine.py.224: train: epoch 66 | batch 1620/2304 [I 2022-10-27 12:12:59] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:26 [I 2022-10-27 12:12:59] ...trap/engines/engine.py.229: process: 0.57535 | load: 0.00243 [I 2022-10-27 12:12:59] ...trap/engines/engine.py.230: loss: 0.11295 [I 2022-10-27 12:12:59] ...optimizers/trijoint.py.151: total_norm: 16.326891 [I 2022-10-27 12:13:04] ...trap/engines/engine.py.224: train: epoch 66 | batch 1630/2304 [I 2022-10-27 12:13:04] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:06:16 [I 2022-10-27 12:13:04] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00244 [I 2022-10-27 12:13:04] ...trap/engines/engine.py.230: loss: 0.17801 [I 2022-10-27 12:13:04] ...optimizers/trijoint.py.151: total_norm: 20.125782 [I 2022-10-27 12:13:10] ...trap/engines/engine.py.224: train: epoch 66 | batch 1640/2304 [I 2022-10-27 12:13:10] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:06:09 [I 2022-10-27 12:13:10] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00243 [I 2022-10-27 12:13:10] ...trap/engines/engine.py.230: loss: 0.11900 [I 2022-10-27 12:13:10] ...optimizers/trijoint.py.151: total_norm: 10.682299 [I 2022-10-27 12:13:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 1650/2304 [I 2022-10-27 12:13:16] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:06:03 [I 2022-10-27 12:13:16] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00244 [I 2022-10-27 12:13:16] ...trap/engines/engine.py.230: loss: 0.16159 [I 2022-10-27 12:13:16] ...optimizers/trijoint.py.151: total_norm: 16.533001 [I 2022-10-27 12:13:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 1660/2304 [I 2022-10-27 12:13:21] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:59 [I 2022-10-27 12:13:21] ...trap/engines/engine.py.229: process: 0.56584 | load: 0.00244 [I 2022-10-27 12:13:21] ...trap/engines/engine.py.230: loss: 0.13400 [I 2022-10-27 12:13:21] ...optimizers/trijoint.py.151: total_norm: 10.748732 [I 2022-10-27 12:13:27] ...trap/engines/engine.py.224: train: epoch 66 | batch 1670/2304 [I 2022-10-27 12:13:27] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:52 [I 2022-10-27 12:13:27] ...trap/engines/engine.py.229: process: 0.55108 | load: 0.00242 [I 2022-10-27 12:13:27] ...trap/engines/engine.py.230: loss: 0.13813 [I 2022-10-27 12:13:27] ...optimizers/trijoint.py.151: total_norm: 16.424242 [I 2022-10-27 12:13:32] ...trap/engines/engine.py.224: train: epoch 66 | batch 1680/2304 [I 2022-10-27 12:13:32] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:46 [I 2022-10-27 12:13:32] ...trap/engines/engine.py.229: process: 0.55010 | load: 0.00244 [I 2022-10-27 12:13:32] ...trap/engines/engine.py.230: loss: 0.24526 [I 2022-10-27 12:13:32] ...optimizers/trijoint.py.151: total_norm: 15.235467 [I 2022-10-27 12:13:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 1690/2304 [I 2022-10-27 12:13:38] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:41 [I 2022-10-27 12:13:38] ...trap/engines/engine.py.229: process: 0.55077 | load: 0.00240 [I 2022-10-27 12:13:38] ...trap/engines/engine.py.230: loss: 0.16129 [I 2022-10-27 12:13:38] ...optimizers/trijoint.py.151: total_norm: 11.346721 [I 2022-10-27 12:13:43] ...trap/engines/engine.py.224: train: epoch 66 | batch 1700/2304 [I 2022-10-27 12:13:43] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:36 [I 2022-10-27 12:13:43] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00244 [I 2022-10-27 12:13:43] ...trap/engines/engine.py.230: loss: 0.19059 [I 2022-10-27 12:13:43] ...optimizers/trijoint.py.151: total_norm: 17.506861 [I 2022-10-27 12:13:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 1710/2304 [I 2022-10-27 12:13:49] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:32 [I 2022-10-27 12:13:49] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00242 [I 2022-10-27 12:13:49] ...trap/engines/engine.py.230: loss: 0.17856 [I 2022-10-27 12:13:49] ...optimizers/trijoint.py.151: total_norm: 21.098076 [I 2022-10-27 12:13:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 1720/2304 [I 2022-10-27 12:13:55] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:26 [I 2022-10-27 12:13:55] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00244 [I 2022-10-27 12:13:55] ...trap/engines/engine.py.230: loss: 0.12996 [I 2022-10-27 12:13:55] ...optimizers/trijoint.py.151: total_norm: 18.483690 [I 2022-10-27 12:14:00] ...trap/engines/engine.py.224: train: epoch 66 | batch 1730/2304 [I 2022-10-27 12:14:00] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:22 [I 2022-10-27 12:14:00] ...trap/engines/engine.py.229: process: 0.56920 | load: 0.00273 [I 2022-10-27 12:14:00] ...trap/engines/engine.py.230: loss: 0.14232 [I 2022-10-27 12:14:00] ...optimizers/trijoint.py.151: total_norm: 17.822817 [I 2022-10-27 12:14:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 1740/2304 [I 2022-10-27 12:14:06] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:05:14 [I 2022-10-27 12:14:06] ...trap/engines/engine.py.229: process: 0.56109 | load: 0.00249 [I 2022-10-27 12:14:06] ...trap/engines/engine.py.230: loss: 0.21010 [I 2022-10-27 12:14:06] ...optimizers/trijoint.py.151: total_norm: 20.768803 [I 2022-10-27 12:14:11] ...trap/engines/engine.py.224: train: epoch 66 | batch 1750/2304 [I 2022-10-27 12:14:11] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:10 [I 2022-10-27 12:14:11] ...trap/engines/engine.py.229: process: 0.55719 | load: 0.00245 [I 2022-10-27 12:14:11] ...trap/engines/engine.py.230: loss: 0.11513 [I 2022-10-27 12:14:11] ...optimizers/trijoint.py.151: total_norm: 12.357840 [I 2022-10-27 12:14:17] ...trap/engines/engine.py.224: train: epoch 66 | batch 1760/2304 [I 2022-10-27 12:14:17] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:05:02 [I 2022-10-27 12:14:17] ...trap/engines/engine.py.229: process: 0.56062 | load: 0.00246 [I 2022-10-27 12:14:17] ...trap/engines/engine.py.230: loss: 0.14360 [I 2022-10-27 12:14:17] ...optimizers/trijoint.py.151: total_norm: 10.072216 [I 2022-10-27 12:14:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 1770/2304 [I 2022-10-27 12:14:23] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:05:00 [I 2022-10-27 12:14:23] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00243 [I 2022-10-27 12:14:23] ...trap/engines/engine.py.230: loss: 0.17639 [I 2022-10-27 12:14:23] ...optimizers/trijoint.py.151: total_norm: 22.378773 [I 2022-10-27 12:14:28] ...trap/engines/engine.py.224: train: epoch 66 | batch 1780/2304 [I 2022-10-27 12:14:28] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:53 [I 2022-10-27 12:14:28] ...trap/engines/engine.py.229: process: 0.56316 | load: 0.00245 [I 2022-10-27 12:14:28] ...trap/engines/engine.py.230: loss: 0.16146 [I 2022-10-27 12:14:28] ...optimizers/trijoint.py.151: total_norm: 22.016445 [I 2022-10-27 12:14:34] ...trap/engines/engine.py.224: train: epoch 66 | batch 1790/2304 [I 2022-10-27 12:14:34] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:49 [I 2022-10-27 12:14:34] ...trap/engines/engine.py.229: process: 0.57102 | load: 0.00241 [I 2022-10-27 12:14:34] ...trap/engines/engine.py.230: loss: 0.15199 [I 2022-10-27 12:14:34] ...optimizers/trijoint.py.151: total_norm: 27.657450 [I 2022-10-27 12:14:39] ...trap/engines/engine.py.224: train: epoch 66 | batch 1800/2304 [I 2022-10-27 12:14:39] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:43 [I 2022-10-27 12:14:39] ...trap/engines/engine.py.229: process: 0.57278 | load: 0.00246 [I 2022-10-27 12:14:39] ...trap/engines/engine.py.230: loss: 0.16569 [I 2022-10-27 12:14:39] ...optimizers/trijoint.py.151: total_norm: 16.337481 [I 2022-10-27 12:14:45] ...trap/engines/engine.py.224: train: epoch 66 | batch 1810/2304 [I 2022-10-27 12:14:45] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:37 [I 2022-10-27 12:14:45] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00244 [I 2022-10-27 12:14:45] ...trap/engines/engine.py.230: loss: 0.22459 [I 2022-10-27 12:14:45] ...optimizers/trijoint.py.151: total_norm: 10.653614 [I 2022-10-27 12:14:51] ...trap/engines/engine.py.224: train: epoch 66 | batch 1820/2304 [I 2022-10-27 12:14:51] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:30 [I 2022-10-27 12:14:51] ...trap/engines/engine.py.229: process: 0.55277 | load: 0.00244 [I 2022-10-27 12:14:51] ...trap/engines/engine.py.230: loss: 0.09575 [I 2022-10-27 12:14:51] ...optimizers/trijoint.py.151: total_norm: 9.081813 [I 2022-10-27 12:14:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 1830/2304 [I 2022-10-27 12:14:56] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:22 [I 2022-10-27 12:14:56] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00239 [I 2022-10-27 12:14:56] ...trap/engines/engine.py.230: loss: 0.20921 [I 2022-10-27 12:14:56] ...optimizers/trijoint.py.151: total_norm: 18.862452 [I 2022-10-27 12:15:02] ...trap/engines/engine.py.224: train: epoch 66 | batch 1840/2304 [I 2022-10-27 12:15:02] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:18 [I 2022-10-27 12:15:02] ...trap/engines/engine.py.229: process: 0.56789 | load: 0.00241 [I 2022-10-27 12:15:02] ...trap/engines/engine.py.230: loss: 0.15328 [I 2022-10-27 12:15:02] ...optimizers/trijoint.py.151: total_norm: 8.811244 [I 2022-10-27 12:15:07] ...trap/engines/engine.py.224: train: epoch 66 | batch 1850/2304 [I 2022-10-27 12:15:07] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:04:13 [I 2022-10-27 12:15:07] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00244 [I 2022-10-27 12:15:07] ...trap/engines/engine.py.230: loss: 0.11096 [I 2022-10-27 12:15:07] ...optimizers/trijoint.py.151: total_norm: 12.373340 [I 2022-10-27 12:15:13] ...trap/engines/engine.py.224: train: epoch 66 | batch 1860/2304 [I 2022-10-27 12:15:13] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:04:06 [I 2022-10-27 12:15:13] ...trap/engines/engine.py.229: process: 0.54213 | load: 0.00241 [I 2022-10-27 12:15:13] ...trap/engines/engine.py.230: loss: 0.20669 [I 2022-10-27 12:15:13] ...optimizers/trijoint.py.151: total_norm: 29.940739 [I 2022-10-27 12:15:19] ...trap/engines/engine.py.224: train: epoch 66 | batch 1870/2304 [I 2022-10-27 12:15:19] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:04:00 [I 2022-10-27 12:15:19] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00247 [I 2022-10-27 12:15:19] ...trap/engines/engine.py.230: loss: 0.13785 [I 2022-10-27 12:15:19] ...optimizers/trijoint.py.151: total_norm: 15.789793 [I 2022-10-27 12:15:24] ...trap/engines/engine.py.224: train: epoch 66 | batch 1880/2304 [I 2022-10-27 12:15:24] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:54 [I 2022-10-27 12:15:24] ...trap/engines/engine.py.229: process: 0.55319 | load: 0.00246 [I 2022-10-27 12:15:24] ...trap/engines/engine.py.230: loss: 0.13208 [I 2022-10-27 12:15:24] ...optimizers/trijoint.py.151: total_norm: 12.666676 [I 2022-10-27 12:15:30] ...trap/engines/engine.py.224: train: epoch 66 | batch 1890/2304 [I 2022-10-27 12:15:30] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:50 [I 2022-10-27 12:15:30] ...trap/engines/engine.py.229: process: 0.56210 | load: 0.00247 [I 2022-10-27 12:15:30] ...trap/engines/engine.py.230: loss: 0.12124 [I 2022-10-27 12:15:30] ...optimizers/trijoint.py.151: total_norm: 11.408435 [I 2022-10-27 12:15:35] ...trap/engines/engine.py.224: train: epoch 66 | batch 1900/2304 [I 2022-10-27 12:15:35] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:46 [I 2022-10-27 12:15:35] ...trap/engines/engine.py.229: process: 0.56707 | load: 0.00247 [I 2022-10-27 12:15:35] ...trap/engines/engine.py.230: loss: 0.12262 [I 2022-10-27 12:15:35] ...optimizers/trijoint.py.151: total_norm: 14.411889 [I 2022-10-27 12:15:41] ...trap/engines/engine.py.224: train: epoch 66 | batch 1910/2304 [I 2022-10-27 12:15:41] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:38 [I 2022-10-27 12:15:41] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00245 [I 2022-10-27 12:15:41] ...trap/engines/engine.py.230: loss: 0.12093 [I 2022-10-27 12:15:41] ...optimizers/trijoint.py.151: total_norm: 7.867917 [I 2022-10-27 12:15:46] ...trap/engines/engine.py.224: train: epoch 66 | batch 1920/2304 [I 2022-10-27 12:15:46] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:33 [I 2022-10-27 12:15:46] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00243 [I 2022-10-27 12:15:46] ...trap/engines/engine.py.230: loss: 0.14260 [I 2022-10-27 12:15:46] ...optimizers/trijoint.py.151: total_norm: 10.185214 [I 2022-10-27 12:15:52] ...trap/engines/engine.py.224: train: epoch 66 | batch 1930/2304 [I 2022-10-27 12:15:52] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:28 [I 2022-10-27 12:15:52] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00244 [I 2022-10-27 12:15:52] ...trap/engines/engine.py.230: loss: 0.25767 [I 2022-10-27 12:15:52] ...optimizers/trijoint.py.151: total_norm: 28.550621 [I 2022-10-27 12:15:58] ...trap/engines/engine.py.224: train: epoch 66 | batch 1940/2304 [I 2022-10-27 12:15:58] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:21 [I 2022-10-27 12:15:58] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00244 [I 2022-10-27 12:15:58] ...trap/engines/engine.py.230: loss: 0.12346 [I 2022-10-27 12:15:58] ...optimizers/trijoint.py.151: total_norm: 21.283953 [I 2022-10-27 12:16:03] ...trap/engines/engine.py.224: train: epoch 66 | batch 1950/2304 [I 2022-10-27 12:16:03] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:16 [I 2022-10-27 12:16:03] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00244 [I 2022-10-27 12:16:03] ...trap/engines/engine.py.230: loss: 0.18351 [I 2022-10-27 12:16:03] ...optimizers/trijoint.py.151: total_norm: 10.347859 [I 2022-10-27 12:16:09] ...trap/engines/engine.py.224: train: epoch 66 | batch 1960/2304 [I 2022-10-27 12:16:09] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:03:13 [I 2022-10-27 12:16:09] ...trap/engines/engine.py.229: process: 0.56391 | load: 0.00244 [I 2022-10-27 12:16:09] ...trap/engines/engine.py.230: loss: 0.11134 [I 2022-10-27 12:16:09] ...optimizers/trijoint.py.151: total_norm: 8.042966 [I 2022-10-27 12:16:14] ...trap/engines/engine.py.224: train: epoch 66 | batch 1970/2304 [I 2022-10-27 12:16:14] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:03:05 [I 2022-10-27 12:16:14] ...trap/engines/engine.py.229: process: 0.54059 | load: 0.00244 [I 2022-10-27 12:16:14] ...trap/engines/engine.py.230: loss: 0.20698 [I 2022-10-27 12:16:14] ...optimizers/trijoint.py.151: total_norm: 14.670897 [I 2022-10-27 12:16:20] ...trap/engines/engine.py.224: train: epoch 66 | batch 1980/2304 [I 2022-10-27 12:16:20] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:03:04 [I 2022-10-27 12:16:20] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00243 [I 2022-10-27 12:16:20] ...trap/engines/engine.py.230: loss: 0.18644 [I 2022-10-27 12:16:20] ...optimizers/trijoint.py.151: total_norm: 25.092823 [I 2022-10-27 12:16:26] ...trap/engines/engine.py.224: train: epoch 66 | batch 1990/2304 [I 2022-10-27 12:16:26] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:57 [I 2022-10-27 12:16:26] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00245 [I 2022-10-27 12:16:26] ...trap/engines/engine.py.230: loss: 0.13153 [I 2022-10-27 12:16:26] ...optimizers/trijoint.py.151: total_norm: 12.529680 [I 2022-10-27 12:16:31] ...trap/engines/engine.py.224: train: epoch 66 | batch 2000/2304 [I 2022-10-27 12:16:31] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:49 [I 2022-10-27 12:16:31] ...trap/engines/engine.py.229: process: 0.55036 | load: 0.00242 [I 2022-10-27 12:16:31] ...trap/engines/engine.py.230: loss: 0.16867 [I 2022-10-27 12:16:31] ...optimizers/trijoint.py.151: total_norm: 28.092342 [I 2022-10-27 12:16:37] ...trap/engines/engine.py.224: train: epoch 66 | batch 2010/2304 [I 2022-10-27 12:16:37] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:42 [I 2022-10-27 12:16:37] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00242 [I 2022-10-27 12:16:37] ...trap/engines/engine.py.230: loss: 0.18837 [I 2022-10-27 12:16:37] ...optimizers/trijoint.py.151: total_norm: 8.973960 [I 2022-10-27 12:16:42] ...trap/engines/engine.py.224: train: epoch 66 | batch 2020/2304 [I 2022-10-27 12:16:42] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:38 [I 2022-10-27 12:16:42] ...trap/engines/engine.py.229: process: 0.57269 | load: 0.00242 [I 2022-10-27 12:16:42] ...trap/engines/engine.py.230: loss: 0.10834 [I 2022-10-27 12:16:42] ...optimizers/trijoint.py.151: total_norm: 15.551338 [I 2022-10-27 12:16:48] ...trap/engines/engine.py.224: train: epoch 66 | batch 2030/2304 [I 2022-10-27 12:16:48] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:32 [I 2022-10-27 12:16:48] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00244 [I 2022-10-27 12:16:48] ...trap/engines/engine.py.230: loss: 0.13305 [I 2022-10-27 12:16:48] ...optimizers/trijoint.py.151: total_norm: 13.835677 [I 2022-10-27 12:16:54] ...trap/engines/engine.py.224: train: epoch 66 | batch 2040/2304 [I 2022-10-27 12:16:54] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:29 [I 2022-10-27 12:16:54] ...trap/engines/engine.py.229: process: 0.59457 | load: 0.00243 [I 2022-10-27 12:16:54] ...trap/engines/engine.py.230: loss: 0.19738 [I 2022-10-27 12:16:54] ...optimizers/trijoint.py.151: total_norm: 11.167400 [I 2022-10-27 12:16:59] ...trap/engines/engine.py.224: train: epoch 66 | batch 2050/2304 [I 2022-10-27 12:16:59] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:22 [I 2022-10-27 12:16:59] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00244 [I 2022-10-27 12:16:59] ...trap/engines/engine.py.230: loss: 0.12021 [I 2022-10-27 12:16:59] ...optimizers/trijoint.py.151: total_norm: 11.165057 [I 2022-10-27 12:17:05] ...trap/engines/engine.py.224: train: epoch 66 | batch 2060/2304 [I 2022-10-27 12:17:05] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:02:14 [I 2022-10-27 12:17:05] ...trap/engines/engine.py.229: process: 0.54604 | load: 0.00240 [I 2022-10-27 12:17:05] ...trap/engines/engine.py.230: loss: 0.12582 [I 2022-10-27 12:17:05] ...optimizers/trijoint.py.151: total_norm: 13.162992 [I 2022-10-27 12:17:10] ...trap/engines/engine.py.224: train: epoch 66 | batch 2070/2304 [I 2022-10-27 12:17:10] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:02:09 [I 2022-10-27 12:17:10] ...trap/engines/engine.py.229: process: 0.54166 | load: 0.00252 [I 2022-10-27 12:17:10] ...trap/engines/engine.py.230: loss: 0.11947 [I 2022-10-27 12:17:10] ...optimizers/trijoint.py.151: total_norm: 7.177335 [I 2022-10-27 12:17:16] ...trap/engines/engine.py.224: train: epoch 66 | batch 2080/2304 [I 2022-10-27 12:17:16] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:02:04 [I 2022-10-27 12:17:16] ...trap/engines/engine.py.229: process: 0.55507 | load: 0.00248 [I 2022-10-27 12:17:16] ...trap/engines/engine.py.230: loss: 0.13808 [I 2022-10-27 12:17:16] ...optimizers/trijoint.py.151: total_norm: 30.788292 [I 2022-10-27 12:17:22] ...trap/engines/engine.py.224: train: epoch 66 | batch 2090/2304 [I 2022-10-27 12:17:22] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:58 [I 2022-10-27 12:17:22] ...trap/engines/engine.py.229: process: 0.54956 | load: 0.00245 [I 2022-10-27 12:17:22] ...trap/engines/engine.py.230: loss: 0.13537 [I 2022-10-27 12:17:22] ...optimizers/trijoint.py.151: total_norm: 14.365118 [I 2022-10-27 12:17:27] ...trap/engines/engine.py.224: train: epoch 66 | batch 2100/2304 [I 2022-10-27 12:17:27] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:53 [I 2022-10-27 12:17:27] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00244 [I 2022-10-27 12:17:27] ...trap/engines/engine.py.230: loss: 0.15806 [I 2022-10-27 12:17:27] ...optimizers/trijoint.py.151: total_norm: 18.078125 [I 2022-10-27 12:17:33] ...trap/engines/engine.py.224: train: epoch 66 | batch 2110/2304 [I 2022-10-27 12:17:33] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:48 [I 2022-10-27 12:17:33] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00241 [I 2022-10-27 12:17:33] ...trap/engines/engine.py.230: loss: 0.21843 [I 2022-10-27 12:17:33] ...optimizers/trijoint.py.151: total_norm: 13.499331 [I 2022-10-27 12:17:38] ...trap/engines/engine.py.224: train: epoch 66 | batch 2120/2304 [I 2022-10-27 12:17:38] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:42 [I 2022-10-27 12:17:38] ...trap/engines/engine.py.229: process: 0.54751 | load: 0.00244 [I 2022-10-27 12:17:38] ...trap/engines/engine.py.230: loss: 0.13506 [I 2022-10-27 12:17:38] ...optimizers/trijoint.py.151: total_norm: 17.157581 [I 2022-10-27 12:17:44] ...trap/engines/engine.py.224: train: epoch 66 | batch 2130/2304 [I 2022-10-27 12:17:44] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:36 [I 2022-10-27 12:17:44] ...trap/engines/engine.py.229: process: 0.54156 | load: 0.00241 [I 2022-10-27 12:17:44] ...trap/engines/engine.py.230: loss: 0.14289 [I 2022-10-27 12:17:44] ...optimizers/trijoint.py.151: total_norm: 18.310955 [I 2022-10-27 12:17:49] ...trap/engines/engine.py.224: train: epoch 66 | batch 2140/2304 [I 2022-10-27 12:17:49] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:30 [I 2022-10-27 12:17:49] ...trap/engines/engine.py.229: process: 0.54445 | load: 0.00245 [I 2022-10-27 12:17:49] ...trap/engines/engine.py.230: loss: 0.16309 [I 2022-10-27 12:17:49] ...optimizers/trijoint.py.151: total_norm: 27.703173 [I 2022-10-27 12:17:55] ...trap/engines/engine.py.224: train: epoch 66 | batch 2150/2304 [I 2022-10-27 12:17:55] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:25 [I 2022-10-27 12:17:55] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00248 [I 2022-10-27 12:17:55] ...trap/engines/engine.py.230: loss: 0.10701 [I 2022-10-27 12:17:55] ...optimizers/trijoint.py.151: total_norm: 10.280300 [I 2022-10-27 12:18:01] ...trap/engines/engine.py.224: train: epoch 66 | batch 2160/2304 [I 2022-10-27 12:18:01] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:19 [I 2022-10-27 12:18:01] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00241 [I 2022-10-27 12:18:01] ...trap/engines/engine.py.230: loss: 0.15929 [I 2022-10-27 12:18:01] ...optimizers/trijoint.py.151: total_norm: 16.522520 [I 2022-10-27 12:18:06] ...trap/engines/engine.py.224: train: epoch 66 | batch 2170/2304 [I 2022-10-27 12:18:06] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:15 [I 2022-10-27 12:18:06] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00240 [I 2022-10-27 12:18:06] ...trap/engines/engine.py.230: loss: 0.16681 [I 2022-10-27 12:18:06] ...optimizers/trijoint.py.151: total_norm: 12.671722 [I 2022-10-27 12:18:12] ...trap/engines/engine.py.224: train: epoch 66 | batch 2180/2304 [I 2022-10-27 12:18:12] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:09 [I 2022-10-27 12:18:12] ...trap/engines/engine.py.229: process: 0.56875 | load: 0.00245 [I 2022-10-27 12:18:12] ...trap/engines/engine.py.230: loss: 0.11268 [I 2022-10-27 12:18:12] ...optimizers/trijoint.py.151: total_norm: 17.604557 [I 2022-10-27 12:18:17] ...trap/engines/engine.py.224: train: epoch 66 | batch 2190/2304 [I 2022-10-27 12:18:17] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:01:03 [I 2022-10-27 12:18:17] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00242 [I 2022-10-27 12:18:17] ...trap/engines/engine.py.230: loss: 0.12415 [I 2022-10-27 12:18:17] ...optimizers/trijoint.py.151: total_norm: 10.113807 [I 2022-10-27 12:18:23] ...trap/engines/engine.py.224: train: epoch 66 | batch 2200/2304 [I 2022-10-27 12:18:23] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:57 [I 2022-10-27 12:18:23] ...trap/engines/engine.py.229: process: 0.55153 | load: 0.00244 [I 2022-10-27 12:18:23] ...trap/engines/engine.py.230: loss: 0.15027 [I 2022-10-27 12:18:23] ...optimizers/trijoint.py.151: total_norm: 20.863342 [I 2022-10-27 12:18:28] ...trap/engines/engine.py.224: train: epoch 66 | batch 2210/2304 [I 2022-10-27 12:18:28] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:52 [I 2022-10-27 12:18:28] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00241 [I 2022-10-27 12:18:28] ...trap/engines/engine.py.230: loss: 0.12051 [I 2022-10-27 12:18:28] ...optimizers/trijoint.py.151: total_norm: 12.906434 [I 2022-10-27 12:18:34] ...trap/engines/engine.py.224: train: epoch 66 | batch 2220/2304 [I 2022-10-27 12:18:34] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:46 [I 2022-10-27 12:18:34] ...trap/engines/engine.py.229: process: 0.53979 | load: 0.00243 [I 2022-10-27 12:18:34] ...trap/engines/engine.py.230: loss: 0.11372 [I 2022-10-27 12:18:34] ...optimizers/trijoint.py.151: total_norm: 9.997064 [I 2022-10-27 12:18:40] ...trap/engines/engine.py.224: train: epoch 66 | batch 2230/2304 [I 2022-10-27 12:18:40] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:42 [I 2022-10-27 12:18:40] ...trap/engines/engine.py.229: process: 0.63863 | load: 0.00241 [I 2022-10-27 12:18:40] ...trap/engines/engine.py.230: loss: 0.12253 [I 2022-10-27 12:18:40] ...optimizers/trijoint.py.151: total_norm: 14.234688 [I 2022-10-27 12:18:45] ...trap/engines/engine.py.224: train: epoch 66 | batch 2240/2304 [I 2022-10-27 12:18:45] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:35 [I 2022-10-27 12:18:45] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00243 [I 2022-10-27 12:18:45] ...trap/engines/engine.py.230: loss: 0.15187 [I 2022-10-27 12:18:45] ...optimizers/trijoint.py.151: total_norm: 22.234467 [I 2022-10-27 12:18:51] ...trap/engines/engine.py.224: train: epoch 66 | batch 2250/2304 [I 2022-10-27 12:18:51] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:30 [I 2022-10-27 12:18:51] ...trap/engines/engine.py.229: process: 0.56284 | load: 0.00242 [I 2022-10-27 12:18:51] ...trap/engines/engine.py.230: loss: 0.16725 [I 2022-10-27 12:18:51] ...optimizers/trijoint.py.151: total_norm: 17.814737 [I 2022-10-27 12:18:56] ...trap/engines/engine.py.224: train: epoch 66 | batch 2260/2304 [I 2022-10-27 12:18:56] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:24 [I 2022-10-27 12:18:56] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00245 [I 2022-10-27 12:18:56] ...trap/engines/engine.py.230: loss: 0.17656 [I 2022-10-27 12:18:56] ...optimizers/trijoint.py.151: total_norm: 27.880438 [I 2022-10-27 12:19:02] ...trap/engines/engine.py.224: train: epoch 66 | batch 2270/2304 [I 2022-10-27 12:19:02] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:18 [I 2022-10-27 12:19:02] ...trap/engines/engine.py.229: process: 0.54971 | load: 0.00244 [I 2022-10-27 12:19:02] ...trap/engines/engine.py.230: loss: 0.13089 [I 2022-10-27 12:19:02] ...optimizers/trijoint.py.151: total_norm: 13.756641 [I 2022-10-27 12:19:08] ...trap/engines/engine.py.224: train: epoch 66 | batch 2280/2304 [I 2022-10-27 12:19:08] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:13 [I 2022-10-27 12:19:08] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00243 [I 2022-10-27 12:19:08] ...trap/engines/engine.py.230: loss: 0.10768 [I 2022-10-27 12:19:08] ...optimizers/trijoint.py.151: total_norm: 15.062843 [I 2022-10-27 12:19:13] ...trap/engines/engine.py.224: train: epoch 66 | batch 2290/2304 [I 2022-10-27 12:19:13] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:00:07 [I 2022-10-27 12:19:13] ...trap/engines/engine.py.229: process: 0.57395 | load: 0.00242 [I 2022-10-27 12:19:13] ...trap/engines/engine.py.230: loss: 0.11262 [I 2022-10-27 12:19:13] ...optimizers/trijoint.py.151: total_norm: 52.687645 [I 2022-10-27 12:19:19] ...trap/engines/engine.py.224: train: epoch 66 | batch 2300/2304 [I 2022-10-27 12:19:19] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:02 [I 2022-10-27 12:19:19] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00016 [I 2022-10-27 12:19:19] ...trap/engines/engine.py.230: loss: 0.16559 [I 2022-10-27 12:19:19] ...optimizers/trijoint.py.151: total_norm: 20.298796 [I 2022-10-27 12:19:21] ...trap/engines/engine.py.224: train: epoch 66 | batch 2304/2304 [I 2022-10-27 12:19:21] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:00 [I 2022-10-27 12:19:21] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00015 [I 2022-10-27 12:19:21] ...trap/engines/engine.py.230: loss: 0.11688 [I 2022-10-27 12:19:21] ...optimizers/trijoint.py.151: total_norm: 15.996050 [S 2022-10-27 12:19:22] ...trap/engines/engine.py.129: train_epoch.epoch: 66 [S 2022-10-27 12:19:22] ...trap/engines/engine.py.129: train_epoch.loss: 0.14795404094354192 [I 2022-10-27 12:19:22] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 12:19:23] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 66 [I 2022-10-27 12:19:23] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 12:19:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 12:19:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 12:19:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 12:19:27] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 12:19:28] ...trap/engines/engine.py.321: eval: epoch 66 | batch 0/510 [I 2022-10-27 12:19:28] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:36:58 [I 2022-10-27 12:19:28] ...trap/engines/engine.py.326: process: 4.34946 | load: 3.11672 [I 2022-10-27 12:19:31] ...trap/engines/engine.py.321: eval: epoch 66 | batch 10/510 [I 2022-10-27 12:19:31] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:19 [I 2022-10-27 12:19:31] ...trap/engines/engine.py.326: process: 0.32480 | load: 0.00016 [I 2022-10-27 12:19:34] ...trap/engines/engine.py.321: eval: epoch 66 | batch 20/510 [I 2022-10-27 12:19:34] ...trap/engines/engine.py.325: elapsed: 0:00:10 | left: 0:03:05 [I 2022-10-27 12:19:34] ...trap/engines/engine.py.326: process: 0.32410 | load: 0.00020 [I 2022-10-27 12:19:38] ...trap/engines/engine.py.321: eval: epoch 66 | batch 30/510 [I 2022-10-27 12:19:38] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:41 [I 2022-10-27 12:19:38] ...trap/engines/engine.py.326: process: 0.32722 | load: 0.00017 [I 2022-10-27 12:19:41] ...trap/engines/engine.py.321: eval: epoch 66 | batch 40/510 [I 2022-10-27 12:19:41] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:38 [I 2022-10-27 12:19:41] ...trap/engines/engine.py.326: process: 0.34244 | load: 0.00017 [I 2022-10-27 12:19:44] ...trap/engines/engine.py.321: eval: epoch 66 | batch 50/510 [I 2022-10-27 12:19:44] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:33 [I 2022-10-27 12:19:44] ...trap/engines/engine.py.326: process: 0.33324 | load: 0.00017 [I 2022-10-27 12:19:48] ...trap/engines/engine.py.321: eval: epoch 66 | batch 60/510 [I 2022-10-27 12:19:48] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:29 [I 2022-10-27 12:19:48] ...trap/engines/engine.py.326: process: 0.33148 | load: 0.00017 [I 2022-10-27 12:19:51] ...trap/engines/engine.py.321: eval: epoch 66 | batch 70/510 [I 2022-10-27 12:19:51] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:26 [I 2022-10-27 12:19:51] ...trap/engines/engine.py.326: process: 0.32916 | load: 0.00017 [I 2022-10-27 12:19:55] ...trap/engines/engine.py.321: eval: epoch 66 | batch 80/510 [I 2022-10-27 12:19:55] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:24 [I 2022-10-27 12:19:55] ...trap/engines/engine.py.326: process: 0.33449 | load: 0.00020 [I 2022-10-27 12:19:58] ...trap/engines/engine.py.321: eval: epoch 66 | batch 90/510 [I 2022-10-27 12:19:58] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:22 [I 2022-10-27 12:19:58] ...trap/engines/engine.py.326: process: 0.34956 | load: 0.00019 [I 2022-10-27 12:20:01] ...trap/engines/engine.py.321: eval: epoch 66 | batch 100/510 [I 2022-10-27 12:20:01] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:13 [I 2022-10-27 12:20:01] ...trap/engines/engine.py.326: process: 0.28531 | load: 0.00034 [I 2022-10-27 12:20:04] ...trap/engines/engine.py.321: eval: epoch 66 | batch 110/510 [I 2022-10-27 12:20:04] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:56 [I 2022-10-27 12:20:04] ...trap/engines/engine.py.326: process: 0.28257 | load: 0.00030 [I 2022-10-27 12:20:07] ...trap/engines/engine.py.321: eval: epoch 66 | batch 120/510 [I 2022-10-27 12:20:07] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:52 [I 2022-10-27 12:20:07] ...trap/engines/engine.py.326: process: 0.28411 | load: 0.00027 [I 2022-10-27 12:20:10] ...trap/engines/engine.py.321: eval: epoch 66 | batch 130/510 [I 2022-10-27 12:20:10] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:49 [I 2022-10-27 12:20:10] ...trap/engines/engine.py.326: process: 0.30311 | load: 0.00019 [I 2022-10-27 12:20:13] ...trap/engines/engine.py.321: eval: epoch 66 | batch 140/510 [I 2022-10-27 12:20:13] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:48 [I 2022-10-27 12:20:13] ...trap/engines/engine.py.326: process: 0.30494 | load: 0.00017 [I 2022-10-27 12:20:16] ...trap/engines/engine.py.321: eval: epoch 66 | batch 150/510 [I 2022-10-27 12:20:16] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 12:20:16] ...trap/engines/engine.py.326: process: 0.29092 | load: 0.00021 [I 2022-10-27 12:20:19] ...trap/engines/engine.py.321: eval: epoch 66 | batch 160/510 [I 2022-10-27 12:20:19] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 12:20:19] ...trap/engines/engine.py.326: process: 0.28966 | load: 0.00016 [I 2022-10-27 12:20:21] ...trap/engines/engine.py.321: eval: epoch 66 | batch 170/510 [I 2022-10-27 12:20:21] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 12:20:21] ...trap/engines/engine.py.326: process: 0.28557 | load: 0.00019 [I 2022-10-27 12:20:24] ...trap/engines/engine.py.321: eval: epoch 66 | batch 180/510 [I 2022-10-27 12:20:24] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:34 [I 2022-10-27 12:20:24] ...trap/engines/engine.py.326: process: 0.28396 | load: 0.00017 [I 2022-10-27 12:20:27] ...trap/engines/engine.py.321: eval: epoch 66 | batch 190/510 [I 2022-10-27 12:20:27] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:31 [I 2022-10-27 12:20:27] ...trap/engines/engine.py.326: process: 0.28749 | load: 0.00016 [I 2022-10-27 12:20:30] ...trap/engines/engine.py.321: eval: epoch 66 | batch 200/510 [I 2022-10-27 12:20:30] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:29 [I 2022-10-27 12:20:30] ...trap/engines/engine.py.326: process: 0.28742 | load: 0.00016 [I 2022-10-27 12:20:33] ...trap/engines/engine.py.321: eval: epoch 66 | batch 210/510 [I 2022-10-27 12:20:33] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 12:20:33] ...trap/engines/engine.py.326: process: 0.28513 | load: 0.00017 [I 2022-10-27 12:20:36] ...trap/engines/engine.py.321: eval: epoch 66 | batch 220/510 [I 2022-10-27 12:20:36] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:22 [I 2022-10-27 12:20:36] ...trap/engines/engine.py.326: process: 0.28679 | load: 0.00019 [I 2022-10-27 12:20:39] ...trap/engines/engine.py.321: eval: epoch 66 | batch 230/510 [I 2022-10-27 12:20:39] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 12:20:39] ...trap/engines/engine.py.326: process: 0.29089 | load: 0.00019 [I 2022-10-27 12:20:42] ...trap/engines/engine.py.321: eval: epoch 66 | batch 240/510 [I 2022-10-27 12:20:42] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 12:20:42] ...trap/engines/engine.py.326: process: 0.28694 | load: 0.00023 [I 2022-10-27 12:20:44] ...trap/engines/engine.py.321: eval: epoch 66 | batch 250/510 [I 2022-10-27 12:20:44] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 12:20:44] ...trap/engines/engine.py.326: process: 0.29147 | load: 0.00019 [I 2022-10-27 12:20:47] ...trap/engines/engine.py.321: eval: epoch 66 | batch 260/510 [I 2022-10-27 12:20:47] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:11 [I 2022-10-27 12:20:47] ...trap/engines/engine.py.326: process: 0.28024 | load: 0.00024 [I 2022-10-27 12:20:50] ...trap/engines/engine.py.321: eval: epoch 66 | batch 270/510 [I 2022-10-27 12:20:50] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:09 [I 2022-10-27 12:20:50] ...trap/engines/engine.py.326: process: 0.28886 | load: 0.00020 [I 2022-10-27 12:20:53] ...trap/engines/engine.py.321: eval: epoch 66 | batch 280/510 [I 2022-10-27 12:20:53] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:06 [I 2022-10-27 12:20:53] ...trap/engines/engine.py.326: process: 0.28472 | load: 0.00022 [I 2022-10-27 12:20:56] ...trap/engines/engine.py.321: eval: epoch 66 | batch 290/510 [I 2022-10-27 12:20:56] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:03 [I 2022-10-27 12:20:56] ...trap/engines/engine.py.326: process: 0.28479 | load: 0.00017 [I 2022-10-27 12:20:59] ...trap/engines/engine.py.321: eval: epoch 66 | batch 300/510 [I 2022-10-27 12:20:59] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 12:20:59] ...trap/engines/engine.py.326: process: 0.28660 | load: 0.00016 [I 2022-10-27 12:21:02] ...trap/engines/engine.py.321: eval: epoch 66 | batch 310/510 [I 2022-10-27 12:21:02] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 12:21:02] ...trap/engines/engine.py.326: process: 0.28695 | load: 0.00020 [I 2022-10-27 12:21:05] ...trap/engines/engine.py.321: eval: epoch 66 | batch 320/510 [I 2022-10-27 12:21:05] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:56 [I 2022-10-27 12:21:05] ...trap/engines/engine.py.326: process: 0.28833 | load: 0.00025 [I 2022-10-27 12:21:08] ...trap/engines/engine.py.321: eval: epoch 66 | batch 330/510 [I 2022-10-27 12:21:08] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 12:21:08] ...trap/engines/engine.py.326: process: 0.29168 | load: 0.00017 [I 2022-10-27 12:21:10] ...trap/engines/engine.py.321: eval: epoch 66 | batch 340/510 [I 2022-10-27 12:21:10] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-27 12:21:10] ...trap/engines/engine.py.326: process: 0.28856 | load: 0.00017 [I 2022-10-27 12:21:13] ...trap/engines/engine.py.321: eval: epoch 66 | batch 350/510 [I 2022-10-27 12:21:13] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:45 [I 2022-10-27 12:21:13] ...trap/engines/engine.py.326: process: 0.28711 | load: 0.00019 [I 2022-10-27 12:21:16] ...trap/engines/engine.py.321: eval: epoch 66 | batch 360/510 [I 2022-10-27 12:21:16] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:42 [I 2022-10-27 12:21:16] ...trap/engines/engine.py.326: process: 0.28089 | load: 0.00023 [I 2022-10-27 12:21:19] ...trap/engines/engine.py.321: eval: epoch 66 | batch 370/510 [I 2022-10-27 12:21:19] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:40 [I 2022-10-27 12:21:19] ...trap/engines/engine.py.326: process: 0.29107 | load: 0.00020 [I 2022-10-27 12:21:22] ...trap/engines/engine.py.321: eval: epoch 66 | batch 380/510 [I 2022-10-27 12:21:22] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 12:21:22] ...trap/engines/engine.py.326: process: 0.28598 | load: 0.00019 [I 2022-10-27 12:21:25] ...trap/engines/engine.py.321: eval: epoch 66 | batch 390/510 [I 2022-10-27 12:21:25] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 12:21:25] ...trap/engines/engine.py.326: process: 0.28691 | load: 0.00020 [I 2022-10-27 12:21:28] ...trap/engines/engine.py.321: eval: epoch 66 | batch 400/510 [I 2022-10-27 12:21:28] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 12:21:28] ...trap/engines/engine.py.326: process: 0.29520 | load: 0.00019 [I 2022-10-27 12:21:30] ...trap/engines/engine.py.321: eval: epoch 66 | batch 410/510 [I 2022-10-27 12:21:30] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 12:21:30] ...trap/engines/engine.py.326: process: 0.28980 | load: 0.00018 [I 2022-10-27 12:21:33] ...trap/engines/engine.py.321: eval: epoch 66 | batch 420/510 [I 2022-10-27 12:21:33] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:25 [I 2022-10-27 12:21:33] ...trap/engines/engine.py.326: process: 0.28685 | load: 0.00018 [I 2022-10-27 12:21:36] ...trap/engines/engine.py.321: eval: epoch 66 | batch 430/510 [I 2022-10-27 12:21:36] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:23 [I 2022-10-27 12:21:36] ...trap/engines/engine.py.326: process: 0.28588 | load: 0.00017 [I 2022-10-27 12:21:39] ...trap/engines/engine.py.321: eval: epoch 66 | batch 440/510 [I 2022-10-27 12:21:39] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:20 [I 2022-10-27 12:21:39] ...trap/engines/engine.py.326: process: 0.28924 | load: 0.00017 [I 2022-10-27 12:21:42] ...trap/engines/engine.py.321: eval: epoch 66 | batch 450/510 [I 2022-10-27 12:21:42] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:17 [I 2022-10-27 12:21:42] ...trap/engines/engine.py.326: process: 0.29457 | load: 0.00026 [I 2022-10-27 12:21:45] ...trap/engines/engine.py.321: eval: epoch 66 | batch 460/510 [I 2022-10-27 12:21:45] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 12:21:45] ...trap/engines/engine.py.326: process: 0.28785 | load: 0.00018 [I 2022-10-27 12:21:48] ...trap/engines/engine.py.321: eval: epoch 66 | batch 470/510 [I 2022-10-27 12:21:48] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 12:21:48] ...trap/engines/engine.py.326: process: 0.29048 | load: 0.00017 [I 2022-10-27 12:21:51] ...trap/engines/engine.py.321: eval: epoch 66 | batch 480/510 [I 2022-10-27 12:21:51] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 12:21:51] ...trap/engines/engine.py.326: process: 0.29088 | load: 0.00030 [I 2022-10-27 12:21:54] ...trap/engines/engine.py.321: eval: epoch 66 | batch 490/510 [I 2022-10-27 12:21:54] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 12:21:54] ...trap/engines/engine.py.326: process: 0.28928 | load: 0.00014 [I 2022-10-27 12:21:56] ...trap/engines/engine.py.321: eval: epoch 66 | batch 500/510 [I 2022-10-27 12:21:56] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:02 [I 2022-10-27 12:21:56] ...trap/engines/engine.py.326: process: 0.27613 | load: 0.00016 [I 2022-10-27 12:21:59] ...trap/engines/engine.py.321: eval: epoch 66 | batch 510/510 [I 2022-10-27 12:21:59] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:00 [I 2022-10-27 12:21:59] ...trap/engines/engine.py.326: process: 0.28964 | load: 0.00011 [S 2022-10-27 12:22:00] ...trap/engines/engine.py.132: eval_epoch.epoch: 66 [I 2022-10-27 12:22:00] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.726700 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905300 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936200 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.709200 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.901000 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933900 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014388 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007682 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006720 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008795 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007629 [S 2022-10-27 12:22:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007341 [I 2022-10-27 12:22:05] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 12:22:05] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 12:22:06] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 12:22:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 12:22:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 12:22:09] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 12:22:10] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 12:22:17] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 12:22:17] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 12:22:17] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 12:22:20] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 12:22:26] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 12:22:26] ...trap/engines/engine.py.163: Training model on trainset for epoch 67 [I 2022-10-27 12:22:26] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 12:22:28] ...trap/engines/engine.py.224: train: epoch 67 | batch 0/2304 [I 2022-10-27 12:22:28] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:39:37 [I 2022-10-27 12:22:28] ...trap/engines/engine.py.229: process: 2.59444 | load: 1.97532 [I 2022-10-27 12:22:28] ...trap/engines/engine.py.230: loss: 0.08588 [I 2022-10-27 12:22:28] ...optimizers/trijoint.py.151: total_norm: 7.665331 [I 2022-10-27 12:22:34] ...trap/engines/engine.py.224: train: epoch 67 | batch 10/2304 [I 2022-10-27 12:22:34] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:36 [I 2022-10-27 12:22:34] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00444 [I 2022-10-27 12:22:34] ...trap/engines/engine.py.230: loss: 0.12118 [I 2022-10-27 12:22:34] ...optimizers/trijoint.py.151: total_norm: 18.493513 [I 2022-10-27 12:22:39] ...trap/engines/engine.py.224: train: epoch 67 | batch 20/2304 [I 2022-10-27 12:22:39] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:12 [I 2022-10-27 12:22:39] ...trap/engines/engine.py.229: process: 0.56907 | load: 0.00393 [I 2022-10-27 12:22:39] ...trap/engines/engine.py.230: loss: 0.18438 [I 2022-10-27 12:22:39] ...optimizers/trijoint.py.151: total_norm: 15.699064 [I 2022-10-27 12:22:45] ...trap/engines/engine.py.224: train: epoch 67 | batch 30/2304 [I 2022-10-27 12:22:45] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:13 [I 2022-10-27 12:22:45] ...trap/engines/engine.py.229: process: 0.55693 | load: 0.00309 [I 2022-10-27 12:22:45] ...trap/engines/engine.py.230: loss: 0.11070 [I 2022-10-27 12:22:45] ...optimizers/trijoint.py.151: total_norm: 9.773861 [I 2022-10-27 12:22:51] ...trap/engines/engine.py.224: train: epoch 67 | batch 40/2304 [I 2022-10-27 12:22:51] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:52 [I 2022-10-27 12:22:51] ...trap/engines/engine.py.229: process: 0.53817 | load: 0.00299 [I 2022-10-27 12:22:51] ...trap/engines/engine.py.230: loss: 0.19930 [I 2022-10-27 12:22:51] ...optimizers/trijoint.py.151: total_norm: 16.879591 [I 2022-10-27 12:22:56] ...trap/engines/engine.py.224: train: epoch 67 | batch 50/2304 [I 2022-10-27 12:22:56] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:52 [I 2022-10-27 12:22:56] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00260 [I 2022-10-27 12:22:56] ...trap/engines/engine.py.230: loss: 0.12140 [I 2022-10-27 12:22:56] ...optimizers/trijoint.py.151: total_norm: 12.910359 [I 2022-10-27 12:23:02] ...trap/engines/engine.py.224: train: epoch 67 | batch 60/2304 [I 2022-10-27 12:23:02] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:44 [I 2022-10-27 12:23:02] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00238 [I 2022-10-27 12:23:02] ...trap/engines/engine.py.230: loss: 0.10303 [I 2022-10-27 12:23:02] ...optimizers/trijoint.py.151: total_norm: 15.982433 [I 2022-10-27 12:23:07] ...trap/engines/engine.py.224: train: epoch 67 | batch 70/2304 [I 2022-10-27 12:23:07] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:39 [I 2022-10-27 12:23:07] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00280 [I 2022-10-27 12:23:07] ...trap/engines/engine.py.230: loss: 0.16245 [I 2022-10-27 12:23:07] ...optimizers/trijoint.py.151: total_norm: 24.032936 [I 2022-10-27 12:23:13] ...trap/engines/engine.py.224: train: epoch 67 | batch 80/2304 [I 2022-10-27 12:23:13] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:40 [I 2022-10-27 12:23:13] ...trap/engines/engine.py.229: process: 0.55061 | load: 0.00283 [I 2022-10-27 12:23:13] ...trap/engines/engine.py.230: loss: 0.15505 [I 2022-10-27 12:23:13] ...optimizers/trijoint.py.151: total_norm: 30.118092 [I 2022-10-27 12:23:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 90/2304 [I 2022-10-27 12:23:18] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:35 [I 2022-10-27 12:23:18] ...trap/engines/engine.py.229: process: 0.55135 | load: 0.00271 [I 2022-10-27 12:23:18] ...trap/engines/engine.py.230: loss: 0.14617 [I 2022-10-27 12:23:18] ...optimizers/trijoint.py.151: total_norm: 17.668337 [I 2022-10-27 12:23:24] ...trap/engines/engine.py.224: train: epoch 67 | batch 100/2304 [I 2022-10-27 12:23:24] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:30 [I 2022-10-27 12:23:24] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00266 [I 2022-10-27 12:23:24] ...trap/engines/engine.py.230: loss: 0.16719 [I 2022-10-27 12:23:24] ...optimizers/trijoint.py.151: total_norm: 17.640667 [I 2022-10-27 12:23:30] ...trap/engines/engine.py.224: train: epoch 67 | batch 110/2304 [I 2022-10-27 12:23:30] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:20 [I 2022-10-27 12:23:30] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00255 [I 2022-10-27 12:23:30] ...trap/engines/engine.py.230: loss: 0.13084 [I 2022-10-27 12:23:30] ...optimizers/trijoint.py.151: total_norm: 26.832827 [I 2022-10-27 12:23:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 120/2304 [I 2022-10-27 12:23:35] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:19 [I 2022-10-27 12:23:35] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00277 [I 2022-10-27 12:23:35] ...trap/engines/engine.py.230: loss: 0.11818 [I 2022-10-27 12:23:35] ...optimizers/trijoint.py.151: total_norm: 11.425589 [I 2022-10-27 12:23:41] ...trap/engines/engine.py.224: train: epoch 67 | batch 130/2304 [I 2022-10-27 12:23:41] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:08 [I 2022-10-27 12:23:41] ...trap/engines/engine.py.229: process: 0.54558 | load: 0.00253 [I 2022-10-27 12:23:41] ...trap/engines/engine.py.230: loss: 0.13939 [I 2022-10-27 12:23:41] ...optimizers/trijoint.py.151: total_norm: 13.227636 [I 2022-10-27 12:23:46] ...trap/engines/engine.py.224: train: epoch 67 | batch 140/2304 [I 2022-10-27 12:23:46] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:46 [I 2022-10-27 12:23:46] ...trap/engines/engine.py.229: process: 0.53885 | load: 0.00251 [I 2022-10-27 12:23:46] ...trap/engines/engine.py.230: loss: 0.11287 [I 2022-10-27 12:23:46] ...optimizers/trijoint.py.151: total_norm: 7.331462 [I 2022-10-27 12:23:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 150/2304 [I 2022-10-27 12:23:52] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:56 [I 2022-10-27 12:23:52] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00257 [I 2022-10-27 12:23:52] ...trap/engines/engine.py.230: loss: 0.21440 [I 2022-10-27 12:23:52] ...optimizers/trijoint.py.151: total_norm: 18.239407 [I 2022-10-27 12:23:58] ...trap/engines/engine.py.224: train: epoch 67 | batch 160/2304 [I 2022-10-27 12:23:58] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:50 [I 2022-10-27 12:23:58] ...trap/engines/engine.py.229: process: 0.54653 | load: 0.00259 [I 2022-10-27 12:23:58] ...trap/engines/engine.py.230: loss: 0.12187 [I 2022-10-27 12:23:58] ...optimizers/trijoint.py.151: total_norm: 18.126818 [I 2022-10-27 12:24:03] ...trap/engines/engine.py.224: train: epoch 67 | batch 170/2304 [I 2022-10-27 12:24:03] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:44 [I 2022-10-27 12:24:03] ...trap/engines/engine.py.229: process: 0.56377 | load: 0.00253 [I 2022-10-27 12:24:03] ...trap/engines/engine.py.230: loss: 0.10880 [I 2022-10-27 12:24:03] ...optimizers/trijoint.py.151: total_norm: 8.746208 [I 2022-10-27 12:24:09] ...trap/engines/engine.py.224: train: epoch 67 | batch 180/2304 [I 2022-10-27 12:24:09] ...trap/engines/engine.py.228: elapsed: 0:01:42 | left: 0:19:31 [I 2022-10-27 12:24:09] ...trap/engines/engine.py.229: process: 0.54454 | load: 0.00260 [I 2022-10-27 12:24:09] ...trap/engines/engine.py.230: loss: 0.16122 [I 2022-10-27 12:24:09] ...optimizers/trijoint.py.151: total_norm: 17.896730 [I 2022-10-27 12:24:14] ...trap/engines/engine.py.224: train: epoch 67 | batch 190/2304 [I 2022-10-27 12:24:14] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:35 [I 2022-10-27 12:24:14] ...trap/engines/engine.py.229: process: 0.55110 | load: 0.00241 [I 2022-10-27 12:24:14] ...trap/engines/engine.py.230: loss: 0.16320 [I 2022-10-27 12:24:14] ...optimizers/trijoint.py.151: total_norm: 21.018141 [I 2022-10-27 12:24:20] ...trap/engines/engine.py.224: train: epoch 67 | batch 200/2304 [I 2022-10-27 12:24:20] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:37 [I 2022-10-27 12:24:20] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00259 [I 2022-10-27 12:24:20] ...trap/engines/engine.py.230: loss: 0.12286 [I 2022-10-27 12:24:20] ...optimizers/trijoint.py.151: total_norm: 17.974190 [I 2022-10-27 12:24:25] ...trap/engines/engine.py.224: train: epoch 67 | batch 210/2304 [I 2022-10-27 12:24:25] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:27 [I 2022-10-27 12:24:25] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00318 [I 2022-10-27 12:24:25] ...trap/engines/engine.py.230: loss: 0.11921 [I 2022-10-27 12:24:25] ...optimizers/trijoint.py.151: total_norm: 12.950208 [I 2022-10-27 12:24:31] ...trap/engines/engine.py.224: train: epoch 67 | batch 220/2304 [I 2022-10-27 12:24:31] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:26 [I 2022-10-27 12:24:31] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00252 [I 2022-10-27 12:24:31] ...trap/engines/engine.py.230: loss: 0.14328 [I 2022-10-27 12:24:31] ...optimizers/trijoint.py.151: total_norm: 14.021243 [I 2022-10-27 12:24:37] ...trap/engines/engine.py.224: train: epoch 67 | batch 230/2304 [I 2022-10-27 12:24:37] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:22 [I 2022-10-27 12:24:37] ...trap/engines/engine.py.229: process: 0.56680 | load: 0.00246 [I 2022-10-27 12:24:37] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-27 12:24:37] ...optimizers/trijoint.py.151: total_norm: 10.510571 [I 2022-10-27 12:24:42] ...trap/engines/engine.py.224: train: epoch 67 | batch 240/2304 [I 2022-10-27 12:24:42] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:04 [I 2022-10-27 12:24:42] ...trap/engines/engine.py.229: process: 0.54268 | load: 0.00257 [I 2022-10-27 12:24:42] ...trap/engines/engine.py.230: loss: 0.12638 [I 2022-10-27 12:24:42] ...optimizers/trijoint.py.151: total_norm: 12.089505 [I 2022-10-27 12:24:48] ...trap/engines/engine.py.224: train: epoch 67 | batch 250/2304 [I 2022-10-27 12:24:48] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:07 [I 2022-10-27 12:24:48] ...trap/engines/engine.py.229: process: 0.55734 | load: 0.00241 [I 2022-10-27 12:24:48] ...trap/engines/engine.py.230: loss: 0.17011 [I 2022-10-27 12:24:48] ...optimizers/trijoint.py.151: total_norm: 14.110604 [I 2022-10-27 12:24:53] ...trap/engines/engine.py.224: train: epoch 67 | batch 260/2304 [I 2022-10-27 12:24:53] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:03 [I 2022-10-27 12:24:53] ...trap/engines/engine.py.229: process: 0.54548 | load: 0.00246 [I 2022-10-27 12:24:53] ...trap/engines/engine.py.230: loss: 0.13376 [I 2022-10-27 12:24:53] ...optimizers/trijoint.py.151: total_norm: 30.254978 [I 2022-10-27 12:24:59] ...trap/engines/engine.py.224: train: epoch 67 | batch 270/2304 [I 2022-10-27 12:24:59] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:56 [I 2022-10-27 12:24:59] ...trap/engines/engine.py.229: process: 0.55058 | load: 0.00247 [I 2022-10-27 12:24:59] ...trap/engines/engine.py.230: loss: 0.18505 [I 2022-10-27 12:24:59] ...optimizers/trijoint.py.151: total_norm: 23.472065 [I 2022-10-27 12:25:05] ...trap/engines/engine.py.224: train: epoch 67 | batch 280/2304 [I 2022-10-27 12:25:05] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:59 [I 2022-10-27 12:25:05] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00252 [I 2022-10-27 12:25:05] ...trap/engines/engine.py.230: loss: 0.13746 [I 2022-10-27 12:25:05] ...optimizers/trijoint.py.151: total_norm: 12.341596 [I 2022-10-27 12:25:10] ...trap/engines/engine.py.224: train: epoch 67 | batch 290/2304 [I 2022-10-27 12:25:10] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:37 [I 2022-10-27 12:25:10] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00255 [I 2022-10-27 12:25:10] ...trap/engines/engine.py.230: loss: 0.15696 [I 2022-10-27 12:25:10] ...optimizers/trijoint.py.151: total_norm: 25.580282 [I 2022-10-27 12:25:16] ...trap/engines/engine.py.224: train: epoch 67 | batch 300/2304 [I 2022-10-27 12:25:16] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:40 [I 2022-10-27 12:25:16] ...trap/engines/engine.py.229: process: 0.55494 | load: 0.00245 [I 2022-10-27 12:25:16] ...trap/engines/engine.py.230: loss: 0.14627 [I 2022-10-27 12:25:16] ...optimizers/trijoint.py.151: total_norm: 10.587379 [I 2022-10-27 12:25:21] ...trap/engines/engine.py.224: train: epoch 67 | batch 310/2304 [I 2022-10-27 12:25:21] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:36 [I 2022-10-27 12:25:21] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00251 [I 2022-10-27 12:25:21] ...trap/engines/engine.py.230: loss: 0.15474 [I 2022-10-27 12:25:21] ...optimizers/trijoint.py.151: total_norm: 19.975437 [I 2022-10-27 12:25:27] ...trap/engines/engine.py.224: train: epoch 67 | batch 320/2304 [I 2022-10-27 12:25:27] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:31 [I 2022-10-27 12:25:27] ...trap/engines/engine.py.229: process: 0.56603 | load: 0.00242 [I 2022-10-27 12:25:27] ...trap/engines/engine.py.230: loss: 0.17108 [I 2022-10-27 12:25:27] ...optimizers/trijoint.py.151: total_norm: 14.786696 [I 2022-10-27 12:25:33] ...trap/engines/engine.py.224: train: epoch 67 | batch 330/2304 [I 2022-10-27 12:25:33] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:24 [I 2022-10-27 12:25:33] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00243 [I 2022-10-27 12:25:33] ...trap/engines/engine.py.230: loss: 0.12791 [I 2022-10-27 12:25:33] ...optimizers/trijoint.py.151: total_norm: 12.248216 [I 2022-10-27 12:25:38] ...trap/engines/engine.py.224: train: epoch 67 | batch 340/2304 [I 2022-10-27 12:25:38] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:12 [I 2022-10-27 12:25:38] ...trap/engines/engine.py.229: process: 0.56525 | load: 0.00242 [I 2022-10-27 12:25:38] ...trap/engines/engine.py.230: loss: 0.10352 [I 2022-10-27 12:25:38] ...optimizers/trijoint.py.151: total_norm: 9.089940 [I 2022-10-27 12:25:44] ...trap/engines/engine.py.224: train: epoch 67 | batch 350/2304 [I 2022-10-27 12:25:44] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:09 [I 2022-10-27 12:25:44] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00248 [I 2022-10-27 12:25:44] ...trap/engines/engine.py.230: loss: 0.10427 [I 2022-10-27 12:25:44] ...optimizers/trijoint.py.151: total_norm: 10.565823 [I 2022-10-27 12:25:49] ...trap/engines/engine.py.224: train: epoch 67 | batch 360/2304 [I 2022-10-27 12:25:49] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:58 [I 2022-10-27 12:25:49] ...trap/engines/engine.py.229: process: 0.55047 | load: 0.00248 [I 2022-10-27 12:25:49] ...trap/engines/engine.py.230: loss: 0.17806 [I 2022-10-27 12:25:49] ...optimizers/trijoint.py.151: total_norm: 14.153133 [I 2022-10-27 12:25:55] ...trap/engines/engine.py.224: train: epoch 67 | batch 370/2304 [I 2022-10-27 12:25:55] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:56 [I 2022-10-27 12:25:55] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00245 [I 2022-10-27 12:25:55] ...trap/engines/engine.py.230: loss: 0.16015 [I 2022-10-27 12:25:55] ...optimizers/trijoint.py.151: total_norm: 55.220676 [I 2022-10-27 12:26:00] ...trap/engines/engine.py.224: train: epoch 67 | batch 380/2304 [I 2022-10-27 12:26:00] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:48 [I 2022-10-27 12:26:00] ...trap/engines/engine.py.229: process: 0.56149 | load: 0.00245 [I 2022-10-27 12:26:00] ...trap/engines/engine.py.230: loss: 0.16262 [I 2022-10-27 12:26:00] ...optimizers/trijoint.py.151: total_norm: 15.220097 [I 2022-10-27 12:26:06] ...trap/engines/engine.py.224: train: epoch 67 | batch 390/2304 [I 2022-10-27 12:26:06] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:38 [I 2022-10-27 12:26:06] ...trap/engines/engine.py.229: process: 0.54902 | load: 0.00251 [I 2022-10-27 12:26:06] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-27 12:26:06] ...optimizers/trijoint.py.151: total_norm: 7.883981 [I 2022-10-27 12:26:12] ...trap/engines/engine.py.224: train: epoch 67 | batch 400/2304 [I 2022-10-27 12:26:12] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:18:09 [I 2022-10-27 12:26:12] ...trap/engines/engine.py.229: process: 0.57737 | load: 0.00248 [I 2022-10-27 12:26:12] ...trap/engines/engine.py.230: loss: 0.19153 [I 2022-10-27 12:26:12] ...optimizers/trijoint.py.151: total_norm: 32.881794 [I 2022-10-27 12:26:17] ...trap/engines/engine.py.224: train: epoch 67 | batch 410/2304 [I 2022-10-27 12:26:17] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:42 [I 2022-10-27 12:26:17] ...trap/engines/engine.py.229: process: 0.55266 | load: 0.00250 [I 2022-10-27 12:26:17] ...trap/engines/engine.py.230: loss: 0.12142 [I 2022-10-27 12:26:17] ...optimizers/trijoint.py.151: total_norm: 15.876828 [I 2022-10-27 12:26:23] ...trap/engines/engine.py.224: train: epoch 67 | batch 420/2304 [I 2022-10-27 12:26:23] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:36 [I 2022-10-27 12:26:23] ...trap/engines/engine.py.229: process: 0.55807 | load: 0.00245 [I 2022-10-27 12:26:23] ...trap/engines/engine.py.230: loss: 0.12378 [I 2022-10-27 12:26:23] ...optimizers/trijoint.py.151: total_norm: 15.373931 [I 2022-10-27 12:26:29] ...trap/engines/engine.py.224: train: epoch 67 | batch 430/2304 [I 2022-10-27 12:26:29] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:22 [I 2022-10-27 12:26:29] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00248 [I 2022-10-27 12:26:29] ...trap/engines/engine.py.230: loss: 0.15371 [I 2022-10-27 12:26:29] ...optimizers/trijoint.py.151: total_norm: 13.894357 [I 2022-10-27 12:26:34] ...trap/engines/engine.py.224: train: epoch 67 | batch 440/2304 [I 2022-10-27 12:26:34] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:17 [I 2022-10-27 12:26:34] ...trap/engines/engine.py.229: process: 0.56371 | load: 0.00249 [I 2022-10-27 12:26:34] ...trap/engines/engine.py.230: loss: 0.13176 [I 2022-10-27 12:26:34] ...optimizers/trijoint.py.151: total_norm: 22.581902 [I 2022-10-27 12:26:40] ...trap/engines/engine.py.224: train: epoch 67 | batch 450/2304 [I 2022-10-27 12:26:40] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:13 [I 2022-10-27 12:26:40] ...trap/engines/engine.py.229: process: 0.56875 | load: 0.00245 [I 2022-10-27 12:26:40] ...trap/engines/engine.py.230: loss: 0.17478 [I 2022-10-27 12:26:40] ...optimizers/trijoint.py.151: total_norm: 23.553474 [I 2022-10-27 12:26:45] ...trap/engines/engine.py.224: train: epoch 67 | batch 460/2304 [I 2022-10-27 12:26:45] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:08 [I 2022-10-27 12:26:45] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00244 [I 2022-10-27 12:26:45] ...trap/engines/engine.py.230: loss: 0.14100 [I 2022-10-27 12:26:45] ...optimizers/trijoint.py.151: total_norm: 18.612837 [I 2022-10-27 12:26:51] ...trap/engines/engine.py.224: train: epoch 67 | batch 470/2304 [I 2022-10-27 12:26:51] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:07 [I 2022-10-27 12:26:51] ...trap/engines/engine.py.229: process: 0.56871 | load: 0.00252 [I 2022-10-27 12:26:51] ...trap/engines/engine.py.230: loss: 0.15103 [I 2022-10-27 12:26:51] ...optimizers/trijoint.py.151: total_norm: 12.828390 [I 2022-10-27 12:26:56] ...trap/engines/engine.py.224: train: epoch 67 | batch 480/2304 [I 2022-10-27 12:26:56] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:50 [I 2022-10-27 12:26:56] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00244 [I 2022-10-27 12:26:56] ...trap/engines/engine.py.230: loss: 0.12558 [I 2022-10-27 12:26:56] ...optimizers/trijoint.py.151: total_norm: 12.526809 [I 2022-10-27 12:27:02] ...trap/engines/engine.py.224: train: epoch 67 | batch 490/2304 [I 2022-10-27 12:27:02] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:53 [I 2022-10-27 12:27:02] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00242 [I 2022-10-27 12:27:02] ...trap/engines/engine.py.230: loss: 0.15184 [I 2022-10-27 12:27:02] ...optimizers/trijoint.py.151: total_norm: 15.642420 [I 2022-10-27 12:27:08] ...trap/engines/engine.py.224: train: epoch 67 | batch 500/2304 [I 2022-10-27 12:27:08] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:45 [I 2022-10-27 12:27:08] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00238 [I 2022-10-27 12:27:08] ...trap/engines/engine.py.230: loss: 0.20871 [I 2022-10-27 12:27:08] ...optimizers/trijoint.py.151: total_norm: 57.051937 [I 2022-10-27 12:27:13] ...trap/engines/engine.py.224: train: epoch 67 | batch 510/2304 [I 2022-10-27 12:27:13] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:38 [I 2022-10-27 12:27:13] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00243 [I 2022-10-27 12:27:13] ...trap/engines/engine.py.230: loss: 0.13084 [I 2022-10-27 12:27:13] ...optimizers/trijoint.py.151: total_norm: 17.064245 [I 2022-10-27 12:27:19] ...trap/engines/engine.py.224: train: epoch 67 | batch 520/2304 [I 2022-10-27 12:27:19] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:36 [I 2022-10-27 12:27:19] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00249 [I 2022-10-27 12:27:19] ...trap/engines/engine.py.230: loss: 0.13996 [I 2022-10-27 12:27:19] ...optimizers/trijoint.py.151: total_norm: 11.548046 [I 2022-10-27 12:27:24] ...trap/engines/engine.py.224: train: epoch 67 | batch 530/2304 [I 2022-10-27 12:27:24] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:25 [I 2022-10-27 12:27:24] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00246 [I 2022-10-27 12:27:24] ...trap/engines/engine.py.230: loss: 0.16710 [I 2022-10-27 12:27:24] ...optimizers/trijoint.py.151: total_norm: 21.575029 [I 2022-10-27 12:27:30] ...trap/engines/engine.py.224: train: epoch 67 | batch 540/2304 [I 2022-10-27 12:27:30] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:22 [I 2022-10-27 12:27:30] ...trap/engines/engine.py.229: process: 0.56091 | load: 0.00255 [I 2022-10-27 12:27:30] ...trap/engines/engine.py.230: loss: 0.16146 [I 2022-10-27 12:27:30] ...optimizers/trijoint.py.151: total_norm: 23.205246 [I 2022-10-27 12:27:36] ...trap/engines/engine.py.224: train: epoch 67 | batch 550/2304 [I 2022-10-27 12:27:36] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:56 [I 2022-10-27 12:27:36] ...trap/engines/engine.py.229: process: 0.67378 | load: 0.00249 [I 2022-10-27 12:27:36] ...trap/engines/engine.py.230: loss: 0.18242 [I 2022-10-27 12:27:36] ...optimizers/trijoint.py.151: total_norm: 21.128635 [I 2022-10-27 12:27:41] ...trap/engines/engine.py.224: train: epoch 67 | batch 560/2304 [I 2022-10-27 12:27:41] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:18 [I 2022-10-27 12:27:41] ...trap/engines/engine.py.229: process: 0.55819 | load: 0.00245 [I 2022-10-27 12:27:41] ...trap/engines/engine.py.230: loss: 0.13661 [I 2022-10-27 12:27:41] ...optimizers/trijoint.py.151: total_norm: 15.497922 [I 2022-10-27 12:27:47] ...trap/engines/engine.py.224: train: epoch 67 | batch 570/2304 [I 2022-10-27 12:27:47] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:03 [I 2022-10-27 12:27:47] ...trap/engines/engine.py.229: process: 0.55458 | load: 0.00246 [I 2022-10-27 12:27:47] ...trap/engines/engine.py.230: loss: 0.11973 [I 2022-10-27 12:27:47] ...optimizers/trijoint.py.151: total_norm: 12.766320 [I 2022-10-27 12:27:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 580/2304 [I 2022-10-27 12:27:52] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:15:58 [I 2022-10-27 12:27:52] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00245 [I 2022-10-27 12:27:52] ...trap/engines/engine.py.230: loss: 0.19648 [I 2022-10-27 12:27:52] ...optimizers/trijoint.py.151: total_norm: 13.083209 [I 2022-10-27 12:27:58] ...trap/engines/engine.py.224: train: epoch 67 | batch 590/2304 [I 2022-10-27 12:27:58] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:55 [I 2022-10-27 12:27:58] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00257 [I 2022-10-27 12:27:58] ...trap/engines/engine.py.230: loss: 0.08786 [I 2022-10-27 12:27:58] ...optimizers/trijoint.py.151: total_norm: 10.208941 [I 2022-10-27 12:28:04] ...trap/engines/engine.py.224: train: epoch 67 | batch 600/2304 [I 2022-10-27 12:28:04] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:47 [I 2022-10-27 12:28:04] ...trap/engines/engine.py.229: process: 0.55006 | load: 0.00252 [I 2022-10-27 12:28:04] ...trap/engines/engine.py.230: loss: 0.10682 [I 2022-10-27 12:28:04] ...optimizers/trijoint.py.151: total_norm: 11.312728 [I 2022-10-27 12:28:09] ...trap/engines/engine.py.224: train: epoch 67 | batch 610/2304 [I 2022-10-27 12:28:09] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:39 [I 2022-10-27 12:28:09] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00245 [I 2022-10-27 12:28:09] ...trap/engines/engine.py.230: loss: 0.13374 [I 2022-10-27 12:28:09] ...optimizers/trijoint.py.151: total_norm: 10.018022 [I 2022-10-27 12:28:15] ...trap/engines/engine.py.224: train: epoch 67 | batch 620/2304 [I 2022-10-27 12:28:15] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:16:06 [I 2022-10-27 12:28:15] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00250 [I 2022-10-27 12:28:15] ...trap/engines/engine.py.230: loss: 0.16546 [I 2022-10-27 12:28:15] ...optimizers/trijoint.py.151: total_norm: 22.822256 [I 2022-10-27 12:28:20] ...trap/engines/engine.py.224: train: epoch 67 | batch 630/2304 [I 2022-10-27 12:28:20] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:36 [I 2022-10-27 12:28:20] ...trap/engines/engine.py.229: process: 0.55421 | load: 0.00245 [I 2022-10-27 12:28:20] ...trap/engines/engine.py.230: loss: 0.11859 [I 2022-10-27 12:28:20] ...optimizers/trijoint.py.151: total_norm: 12.343822 [I 2022-10-27 12:28:26] ...trap/engines/engine.py.224: train: epoch 67 | batch 640/2304 [I 2022-10-27 12:28:26] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:22 [I 2022-10-27 12:28:26] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00245 [I 2022-10-27 12:28:26] ...trap/engines/engine.py.230: loss: 0.11025 [I 2022-10-27 12:28:26] ...optimizers/trijoint.py.151: total_norm: 13.161215 [I 2022-10-27 12:28:32] ...trap/engines/engine.py.224: train: epoch 67 | batch 650/2304 [I 2022-10-27 12:28:32] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:14 [I 2022-10-27 12:28:32] ...trap/engines/engine.py.229: process: 0.54406 | load: 0.00248 [I 2022-10-27 12:28:32] ...trap/engines/engine.py.230: loss: 0.19259 [I 2022-10-27 12:28:32] ...optimizers/trijoint.py.151: total_norm: 16.394001 [I 2022-10-27 12:28:37] ...trap/engines/engine.py.224: train: epoch 67 | batch 660/2304 [I 2022-10-27 12:28:37] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:11 [I 2022-10-27 12:28:37] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00243 [I 2022-10-27 12:28:37] ...trap/engines/engine.py.230: loss: 0.09518 [I 2022-10-27 12:28:37] ...optimizers/trijoint.py.151: total_norm: 13.374786 [I 2022-10-27 12:28:43] ...trap/engines/engine.py.224: train: epoch 67 | batch 670/2304 [I 2022-10-27 12:28:43] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:01 [I 2022-10-27 12:28:43] ...trap/engines/engine.py.229: process: 0.54279 | load: 0.00246 [I 2022-10-27 12:28:43] ...trap/engines/engine.py.230: loss: 0.12945 [I 2022-10-27 12:28:43] ...optimizers/trijoint.py.151: total_norm: 7.945915 [I 2022-10-27 12:28:48] ...trap/engines/engine.py.224: train: epoch 67 | batch 680/2304 [I 2022-10-27 12:28:48] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:28 [I 2022-10-27 12:28:48] ...trap/engines/engine.py.229: process: 0.56324 | load: 0.00259 [I 2022-10-27 12:28:48] ...trap/engines/engine.py.230: loss: 0.15179 [I 2022-10-27 12:28:48] ...optimizers/trijoint.py.151: total_norm: 8.550707 [I 2022-10-27 12:28:54] ...trap/engines/engine.py.224: train: epoch 67 | batch 690/2304 [I 2022-10-27 12:28:54] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:54 [I 2022-10-27 12:28:54] ...trap/engines/engine.py.229: process: 0.54903 | load: 0.00248 [I 2022-10-27 12:28:54] ...trap/engines/engine.py.230: loss: 0.18488 [I 2022-10-27 12:28:54] ...optimizers/trijoint.py.151: total_norm: 22.633070 [I 2022-10-27 12:29:00] ...trap/engines/engine.py.224: train: epoch 67 | batch 700/2304 [I 2022-10-27 12:29:00] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:51 [I 2022-10-27 12:29:00] ...trap/engines/engine.py.229: process: 0.55343 | load: 0.00244 [I 2022-10-27 12:29:00] ...trap/engines/engine.py.230: loss: 0.14610 [I 2022-10-27 12:29:00] ...optimizers/trijoint.py.151: total_norm: 20.591606 [I 2022-10-27 12:29:05] ...trap/engines/engine.py.224: train: epoch 67 | batch 710/2304 [I 2022-10-27 12:29:05] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:54 [I 2022-10-27 12:29:05] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00240 [I 2022-10-27 12:29:05] ...trap/engines/engine.py.230: loss: 0.14283 [I 2022-10-27 12:29:05] ...optimizers/trijoint.py.151: total_norm: 22.461988 [I 2022-10-27 12:29:11] ...trap/engines/engine.py.224: train: epoch 67 | batch 720/2304 [I 2022-10-27 12:29:11] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:47 [I 2022-10-27 12:29:11] ...trap/engines/engine.py.229: process: 0.56703 | load: 0.00252 [I 2022-10-27 12:29:11] ...trap/engines/engine.py.230: loss: 0.12104 [I 2022-10-27 12:29:11] ...optimizers/trijoint.py.151: total_norm: 11.419855 [I 2022-10-27 12:29:16] ...trap/engines/engine.py.224: train: epoch 67 | batch 730/2304 [I 2022-10-27 12:29:16] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:39 [I 2022-10-27 12:29:16] ...trap/engines/engine.py.229: process: 0.56546 | load: 0.00262 [I 2022-10-27 12:29:16] ...trap/engines/engine.py.230: loss: 0.17393 [I 2022-10-27 12:29:16] ...optimizers/trijoint.py.151: total_norm: 26.176897 [I 2022-10-27 12:29:22] ...trap/engines/engine.py.224: train: epoch 67 | batch 740/2304 [I 2022-10-27 12:29:22] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:29 [I 2022-10-27 12:29:22] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00245 [I 2022-10-27 12:29:22] ...trap/engines/engine.py.230: loss: 0.14551 [I 2022-10-27 12:29:22] ...optimizers/trijoint.py.151: total_norm: 11.594683 [I 2022-10-27 12:29:28] ...trap/engines/engine.py.224: train: epoch 67 | batch 750/2304 [I 2022-10-27 12:29:28] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:39 [I 2022-10-27 12:29:28] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00244 [I 2022-10-27 12:29:28] ...trap/engines/engine.py.230: loss: 0.16185 [I 2022-10-27 12:29:28] ...optimizers/trijoint.py.151: total_norm: 17.436474 [I 2022-10-27 12:29:33] ...trap/engines/engine.py.224: train: epoch 67 | batch 760/2304 [I 2022-10-27 12:29:33] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:18 [I 2022-10-27 12:29:33] ...trap/engines/engine.py.229: process: 0.55417 | load: 0.00245 [I 2022-10-27 12:29:33] ...trap/engines/engine.py.230: loss: 0.11530 [I 2022-10-27 12:29:33] ...optimizers/trijoint.py.151: total_norm: 16.667074 [I 2022-10-27 12:29:39] ...trap/engines/engine.py.224: train: epoch 67 | batch 770/2304 [I 2022-10-27 12:29:39] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:19 [I 2022-10-27 12:29:39] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00245 [I 2022-10-27 12:29:39] ...trap/engines/engine.py.230: loss: 0.15086 [I 2022-10-27 12:29:39] ...optimizers/trijoint.py.151: total_norm: 12.792061 [I 2022-10-27 12:29:44] ...trap/engines/engine.py.224: train: epoch 67 | batch 780/2304 [I 2022-10-27 12:29:44] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:13 [I 2022-10-27 12:29:44] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00253 [I 2022-10-27 12:29:44] ...trap/engines/engine.py.230: loss: 0.11791 [I 2022-10-27 12:29:44] ...optimizers/trijoint.py.151: total_norm: 15.088257 [I 2022-10-27 12:29:50] ...trap/engines/engine.py.224: train: epoch 67 | batch 790/2304 [I 2022-10-27 12:29:50] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:01 [I 2022-10-27 12:29:50] ...trap/engines/engine.py.229: process: 0.54340 | load: 0.00251 [I 2022-10-27 12:29:50] ...trap/engines/engine.py.230: loss: 0.13992 [I 2022-10-27 12:29:50] ...optimizers/trijoint.py.151: total_norm: 33.121040 [I 2022-10-27 12:29:56] ...trap/engines/engine.py.224: train: epoch 67 | batch 800/2304 [I 2022-10-27 12:29:56] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:14:00 [I 2022-10-27 12:29:56] ...trap/engines/engine.py.229: process: 0.56156 | load: 0.00251 [I 2022-10-27 12:29:56] ...trap/engines/engine.py.230: loss: 0.13124 [I 2022-10-27 12:29:56] ...optimizers/trijoint.py.151: total_norm: 21.581745 [I 2022-10-27 12:30:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 810/2304 [I 2022-10-27 12:30:01] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:14:33 [I 2022-10-27 12:30:01] ...trap/engines/engine.py.229: process: 0.69373 | load: 0.00247 [I 2022-10-27 12:30:01] ...trap/engines/engine.py.230: loss: 0.17327 [I 2022-10-27 12:30:01] ...optimizers/trijoint.py.151: total_norm: 13.285919 [I 2022-10-27 12:30:07] ...trap/engines/engine.py.224: train: epoch 67 | batch 820/2304 [I 2022-10-27 12:30:07] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:45 [I 2022-10-27 12:30:07] ...trap/engines/engine.py.229: process: 0.54641 | load: 0.00248 [I 2022-10-27 12:30:07] ...trap/engines/engine.py.230: loss: 0.10358 [I 2022-10-27 12:30:07] ...optimizers/trijoint.py.151: total_norm: 7.328824 [I 2022-10-27 12:30:12] ...trap/engines/engine.py.224: train: epoch 67 | batch 830/2304 [I 2022-10-27 12:30:12] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:37 [I 2022-10-27 12:30:12] ...trap/engines/engine.py.229: process: 0.56905 | load: 0.00245 [I 2022-10-27 12:30:12] ...trap/engines/engine.py.230: loss: 0.11926 [I 2022-10-27 12:30:12] ...optimizers/trijoint.py.151: total_norm: 17.276384 [I 2022-10-27 12:30:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 840/2304 [I 2022-10-27 12:30:18] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:36 [I 2022-10-27 12:30:18] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00268 [I 2022-10-27 12:30:18] ...trap/engines/engine.py.230: loss: 0.15187 [I 2022-10-27 12:30:18] ...optimizers/trijoint.py.151: total_norm: 20.273968 [I 2022-10-27 12:30:23] ...trap/engines/engine.py.224: train: epoch 67 | batch 850/2304 [I 2022-10-27 12:30:23] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:30 [I 2022-10-27 12:30:23] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00246 [I 2022-10-27 12:30:23] ...trap/engines/engine.py.230: loss: 0.11171 [I 2022-10-27 12:30:23] ...optimizers/trijoint.py.151: total_norm: 11.769148 [I 2022-10-27 12:30:29] ...trap/engines/engine.py.224: train: epoch 67 | batch 860/2304 [I 2022-10-27 12:30:29] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:28 [I 2022-10-27 12:30:29] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00244 [I 2022-10-27 12:30:29] ...trap/engines/engine.py.230: loss: 0.10366 [I 2022-10-27 12:30:29] ...optimizers/trijoint.py.151: total_norm: 9.976630 [I 2022-10-27 12:30:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 870/2304 [I 2022-10-27 12:30:35] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:22 [I 2022-10-27 12:30:35] ...trap/engines/engine.py.229: process: 0.55053 | load: 0.00240 [I 2022-10-27 12:30:35] ...trap/engines/engine.py.230: loss: 0.14990 [I 2022-10-27 12:30:35] ...optimizers/trijoint.py.151: total_norm: 13.660282 [I 2022-10-27 12:30:40] ...trap/engines/engine.py.224: train: epoch 67 | batch 880/2304 [I 2022-10-27 12:30:40] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:17 [I 2022-10-27 12:30:40] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00241 [I 2022-10-27 12:30:40] ...trap/engines/engine.py.230: loss: 0.10595 [I 2022-10-27 12:30:40] ...optimizers/trijoint.py.151: total_norm: 9.085592 [I 2022-10-27 12:30:46] ...trap/engines/engine.py.224: train: epoch 67 | batch 890/2304 [I 2022-10-27 12:30:46] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:08 [I 2022-10-27 12:30:46] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00246 [I 2022-10-27 12:30:46] ...trap/engines/engine.py.230: loss: 0.16351 [I 2022-10-27 12:30:46] ...optimizers/trijoint.py.151: total_norm: 19.632463 [I 2022-10-27 12:30:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 900/2304 [I 2022-10-27 12:30:52] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:02 [I 2022-10-27 12:30:52] ...trap/engines/engine.py.229: process: 0.56391 | load: 0.00246 [I 2022-10-27 12:30:52] ...trap/engines/engine.py.230: loss: 0.11396 [I 2022-10-27 12:30:52] ...optimizers/trijoint.py.151: total_norm: 15.907966 [I 2022-10-27 12:30:57] ...trap/engines/engine.py.224: train: epoch 67 | batch 910/2304 [I 2022-10-27 12:30:57] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:57 [I 2022-10-27 12:30:57] ...trap/engines/engine.py.229: process: 0.55799 | load: 0.00243 [I 2022-10-27 12:30:57] ...trap/engines/engine.py.230: loss: 0.15372 [I 2022-10-27 12:30:57] ...optimizers/trijoint.py.151: total_norm: 15.016003 [I 2022-10-27 12:31:03] ...trap/engines/engine.py.224: train: epoch 67 | batch 920/2304 [I 2022-10-27 12:31:03] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:46 [I 2022-10-27 12:31:03] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00246 [I 2022-10-27 12:31:03] ...trap/engines/engine.py.230: loss: 0.16646 [I 2022-10-27 12:31:03] ...optimizers/trijoint.py.151: total_norm: 26.449144 [I 2022-10-27 12:31:08] ...trap/engines/engine.py.224: train: epoch 67 | batch 930/2304 [I 2022-10-27 12:31:08] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:45 [I 2022-10-27 12:31:08] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00247 [I 2022-10-27 12:31:08] ...trap/engines/engine.py.230: loss: 0.15149 [I 2022-10-27 12:31:08] ...optimizers/trijoint.py.151: total_norm: 35.565941 [I 2022-10-27 12:31:14] ...trap/engines/engine.py.224: train: epoch 67 | batch 940/2304 [I 2022-10-27 12:31:14] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:42 [I 2022-10-27 12:31:14] ...trap/engines/engine.py.229: process: 0.54424 | load: 0.00245 [I 2022-10-27 12:31:14] ...trap/engines/engine.py.230: loss: 0.09236 [I 2022-10-27 12:31:14] ...optimizers/trijoint.py.151: total_norm: 10.414451 [I 2022-10-27 12:31:19] ...trap/engines/engine.py.224: train: epoch 67 | batch 950/2304 [I 2022-10-27 12:31:19] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:34 [I 2022-10-27 12:31:19] ...trap/engines/engine.py.229: process: 0.55561 | load: 0.00246 [I 2022-10-27 12:31:19] ...trap/engines/engine.py.230: loss: 0.15104 [I 2022-10-27 12:31:19] ...optimizers/trijoint.py.151: total_norm: 12.879095 [I 2022-10-27 12:31:25] ...trap/engines/engine.py.224: train: epoch 67 | batch 960/2304 [I 2022-10-27 12:31:25] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:24 [I 2022-10-27 12:31:25] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00248 [I 2022-10-27 12:31:25] ...trap/engines/engine.py.230: loss: 0.13664 [I 2022-10-27 12:31:25] ...optimizers/trijoint.py.151: total_norm: 22.170429 [I 2022-10-27 12:31:31] ...trap/engines/engine.py.224: train: epoch 67 | batch 970/2304 [I 2022-10-27 12:31:31] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:19 [I 2022-10-27 12:31:31] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00254 [I 2022-10-27 12:31:31] ...trap/engines/engine.py.230: loss: 0.15665 [I 2022-10-27 12:31:31] ...optimizers/trijoint.py.151: total_norm: 23.172062 [I 2022-10-27 12:31:36] ...trap/engines/engine.py.224: train: epoch 67 | batch 980/2304 [I 2022-10-27 12:31:36] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:14 [I 2022-10-27 12:31:36] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00249 [I 2022-10-27 12:31:36] ...trap/engines/engine.py.230: loss: 0.19427 [I 2022-10-27 12:31:36] ...optimizers/trijoint.py.151: total_norm: 23.390230 [I 2022-10-27 12:31:42] ...trap/engines/engine.py.224: train: epoch 67 | batch 990/2304 [I 2022-10-27 12:31:42] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:13 [I 2022-10-27 12:31:42] ...trap/engines/engine.py.229: process: 0.54846 | load: 0.00245 [I 2022-10-27 12:31:42] ...trap/engines/engine.py.230: loss: 0.15818 [I 2022-10-27 12:31:42] ...optimizers/trijoint.py.151: total_norm: 14.107452 [I 2022-10-27 12:31:47] ...trap/engines/engine.py.224: train: epoch 67 | batch 1000/2304 [I 2022-10-27 12:31:47] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:03 [I 2022-10-27 12:31:47] ...trap/engines/engine.py.229: process: 0.54367 | load: 0.00244 [I 2022-10-27 12:31:47] ...trap/engines/engine.py.230: loss: 0.21037 [I 2022-10-27 12:31:47] ...optimizers/trijoint.py.151: total_norm: 36.622112 [I 2022-10-27 12:31:53] ...trap/engines/engine.py.224: train: epoch 67 | batch 1010/2304 [I 2022-10-27 12:31:53] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:01 [I 2022-10-27 12:31:53] ...trap/engines/engine.py.229: process: 0.55113 | load: 0.00245 [I 2022-10-27 12:31:53] ...trap/engines/engine.py.230: loss: 0.12706 [I 2022-10-27 12:31:53] ...optimizers/trijoint.py.151: total_norm: 12.381408 [I 2022-10-27 12:31:58] ...trap/engines/engine.py.224: train: epoch 67 | batch 1020/2304 [I 2022-10-27 12:31:58] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:58 [I 2022-10-27 12:31:58] ...trap/engines/engine.py.229: process: 0.55092 | load: 0.00246 [I 2022-10-27 12:31:58] ...trap/engines/engine.py.230: loss: 0.09503 [I 2022-10-27 12:31:58] ...optimizers/trijoint.py.151: total_norm: 14.215101 [I 2022-10-27 12:32:04] ...trap/engines/engine.py.224: train: epoch 67 | batch 1030/2304 [I 2022-10-27 12:32:04] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:45 [I 2022-10-27 12:32:04] ...trap/engines/engine.py.229: process: 0.54795 | load: 0.00243 [I 2022-10-27 12:32:04] ...trap/engines/engine.py.230: loss: 0.17290 [I 2022-10-27 12:32:04] ...optimizers/trijoint.py.151: total_norm: 18.184816 [I 2022-10-27 12:32:10] ...trap/engines/engine.py.224: train: epoch 67 | batch 1040/2304 [I 2022-10-27 12:32:10] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:41 [I 2022-10-27 12:32:10] ...trap/engines/engine.py.229: process: 0.55106 | load: 0.00241 [I 2022-10-27 12:32:10] ...trap/engines/engine.py.230: loss: 0.14123 [I 2022-10-27 12:32:10] ...optimizers/trijoint.py.151: total_norm: 11.874657 [I 2022-10-27 12:32:15] ...trap/engines/engine.py.224: train: epoch 67 | batch 1050/2304 [I 2022-10-27 12:32:15] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:36 [I 2022-10-27 12:32:15] ...trap/engines/engine.py.229: process: 0.55212 | load: 0.00246 [I 2022-10-27 12:32:15] ...trap/engines/engine.py.230: loss: 0.12703 [I 2022-10-27 12:32:15] ...optimizers/trijoint.py.151: total_norm: 19.477724 [I 2022-10-27 12:32:21] ...trap/engines/engine.py.224: train: epoch 67 | batch 1060/2304 [I 2022-10-27 12:32:21] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:36 [I 2022-10-27 12:32:21] ...trap/engines/engine.py.229: process: 0.56306 | load: 0.00242 [I 2022-10-27 12:32:21] ...trap/engines/engine.py.230: loss: 0.13176 [I 2022-10-27 12:32:21] ...optimizers/trijoint.py.151: total_norm: 11.809460 [I 2022-10-27 12:32:26] ...trap/engines/engine.py.224: train: epoch 67 | batch 1070/2304 [I 2022-10-27 12:32:26] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:31 [I 2022-10-27 12:32:26] ...trap/engines/engine.py.229: process: 0.57346 | load: 0.00242 [I 2022-10-27 12:32:26] ...trap/engines/engine.py.230: loss: 0.17032 [I 2022-10-27 12:32:26] ...optimizers/trijoint.py.151: total_norm: 31.620005 [I 2022-10-27 12:32:32] ...trap/engines/engine.py.224: train: epoch 67 | batch 1080/2304 [I 2022-10-27 12:32:32] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:22 [I 2022-10-27 12:32:32] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00253 [I 2022-10-27 12:32:32] ...trap/engines/engine.py.230: loss: 0.11693 [I 2022-10-27 12:32:32] ...optimizers/trijoint.py.151: total_norm: 11.629163 [I 2022-10-27 12:32:38] ...trap/engines/engine.py.224: train: epoch 67 | batch 1090/2304 [I 2022-10-27 12:32:38] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:20 [I 2022-10-27 12:32:38] ...trap/engines/engine.py.229: process: 0.56347 | load: 0.00247 [I 2022-10-27 12:32:38] ...trap/engines/engine.py.230: loss: 0.12677 [I 2022-10-27 12:32:38] ...optimizers/trijoint.py.151: total_norm: 13.773961 [I 2022-10-27 12:32:43] ...trap/engines/engine.py.224: train: epoch 67 | batch 1100/2304 [I 2022-10-27 12:32:43] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:10 [I 2022-10-27 12:32:43] ...trap/engines/engine.py.229: process: 0.54784 | load: 0.00247 [I 2022-10-27 12:32:43] ...trap/engines/engine.py.230: loss: 0.13635 [I 2022-10-27 12:32:43] ...optimizers/trijoint.py.151: total_norm: 13.601760 [I 2022-10-27 12:32:49] ...trap/engines/engine.py.224: train: epoch 67 | batch 1110/2304 [I 2022-10-27 12:32:49] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:05 [I 2022-10-27 12:32:49] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00242 [I 2022-10-27 12:32:49] ...trap/engines/engine.py.230: loss: 0.13062 [I 2022-10-27 12:32:49] ...optimizers/trijoint.py.151: total_norm: 9.893462 [I 2022-10-27 12:32:54] ...trap/engines/engine.py.224: train: epoch 67 | batch 1120/2304 [I 2022-10-27 12:32:54] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:10:54 [I 2022-10-27 12:32:54] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00242 [I 2022-10-27 12:32:54] ...trap/engines/engine.py.230: loss: 0.18081 [I 2022-10-27 12:32:54] ...optimizers/trijoint.py.151: total_norm: 15.188617 [I 2022-10-27 12:33:00] ...trap/engines/engine.py.224: train: epoch 67 | batch 1130/2304 [I 2022-10-27 12:33:00] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:54 [I 2022-10-27 12:33:00] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00245 [I 2022-10-27 12:33:00] ...trap/engines/engine.py.230: loss: 0.18056 [I 2022-10-27 12:33:00] ...optimizers/trijoint.py.151: total_norm: 28.908169 [I 2022-10-27 12:33:05] ...trap/engines/engine.py.224: train: epoch 67 | batch 1140/2304 [I 2022-10-27 12:33:05] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:53 [I 2022-10-27 12:33:05] ...trap/engines/engine.py.229: process: 0.56233 | load: 0.00245 [I 2022-10-27 12:33:05] ...trap/engines/engine.py.230: loss: 0.21347 [I 2022-10-27 12:33:05] ...optimizers/trijoint.py.151: total_norm: 14.989492 [I 2022-10-27 12:33:11] ...trap/engines/engine.py.224: train: epoch 67 | batch 1150/2304 [I 2022-10-27 12:33:11] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:45 [I 2022-10-27 12:33:11] ...trap/engines/engine.py.229: process: 0.56757 | load: 0.00244 [I 2022-10-27 12:33:11] ...trap/engines/engine.py.230: loss: 0.18579 [I 2022-10-27 12:33:11] ...optimizers/trijoint.py.151: total_norm: 21.746960 [I 2022-10-27 12:33:17] ...trap/engines/engine.py.224: train: epoch 67 | batch 1160/2304 [I 2022-10-27 12:33:17] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:40 [I 2022-10-27 12:33:17] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00256 [I 2022-10-27 12:33:17] ...trap/engines/engine.py.230: loss: 0.10830 [I 2022-10-27 12:33:17] ...optimizers/trijoint.py.151: total_norm: 12.584876 [I 2022-10-27 12:33:22] ...trap/engines/engine.py.224: train: epoch 67 | batch 1170/2304 [I 2022-10-27 12:33:22] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:38 [I 2022-10-27 12:33:22] ...trap/engines/engine.py.229: process: 0.56666 | load: 0.00247 [I 2022-10-27 12:33:22] ...trap/engines/engine.py.230: loss: 0.12446 [I 2022-10-27 12:33:22] ...optimizers/trijoint.py.151: total_norm: 9.630347 [I 2022-10-27 12:33:28] ...trap/engines/engine.py.224: train: epoch 67 | batch 1180/2304 [I 2022-10-27 12:33:28] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:24 [I 2022-10-27 12:33:28] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00249 [I 2022-10-27 12:33:28] ...trap/engines/engine.py.230: loss: 0.15198 [I 2022-10-27 12:33:28] ...optimizers/trijoint.py.151: total_norm: 18.754549 [I 2022-10-27 12:33:33] ...trap/engines/engine.py.224: train: epoch 67 | batch 1190/2304 [I 2022-10-27 12:33:33] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:16 [I 2022-10-27 12:33:33] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00242 [I 2022-10-27 12:33:33] ...trap/engines/engine.py.230: loss: 0.17724 [I 2022-10-27 12:33:33] ...optimizers/trijoint.py.151: total_norm: 14.053249 [I 2022-10-27 12:33:39] ...trap/engines/engine.py.224: train: epoch 67 | batch 1200/2304 [I 2022-10-27 12:33:39] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:18 [I 2022-10-27 12:33:39] ...trap/engines/engine.py.229: process: 0.56460 | load: 0.00242 [I 2022-10-27 12:33:39] ...trap/engines/engine.py.230: loss: 0.11481 [I 2022-10-27 12:33:39] ...optimizers/trijoint.py.151: total_norm: 16.475760 [I 2022-10-27 12:33:45] ...trap/engines/engine.py.224: train: epoch 67 | batch 1210/2304 [I 2022-10-27 12:33:45] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:08 [I 2022-10-27 12:33:45] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00242 [I 2022-10-27 12:33:45] ...trap/engines/engine.py.230: loss: 0.17110 [I 2022-10-27 12:33:45] ...optimizers/trijoint.py.151: total_norm: 10.986935 [I 2022-10-27 12:33:50] ...trap/engines/engine.py.224: train: epoch 67 | batch 1220/2304 [I 2022-10-27 12:33:50] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:02 [I 2022-10-27 12:33:50] ...trap/engines/engine.py.229: process: 0.55061 | load: 0.00244 [I 2022-10-27 12:33:50] ...trap/engines/engine.py.230: loss: 0.12378 [I 2022-10-27 12:33:50] ...optimizers/trijoint.py.151: total_norm: 10.967463 [I 2022-10-27 12:33:56] ...trap/engines/engine.py.224: train: epoch 67 | batch 1230/2304 [I 2022-10-27 12:33:56] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:57 [I 2022-10-27 12:33:56] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00242 [I 2022-10-27 12:33:56] ...trap/engines/engine.py.230: loss: 0.12403 [I 2022-10-27 12:33:56] ...optimizers/trijoint.py.151: total_norm: 12.677750 [I 2022-10-27 12:34:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 1240/2304 [I 2022-10-27 12:34:01] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:52 [I 2022-10-27 12:34:01] ...trap/engines/engine.py.229: process: 0.55431 | load: 0.00258 [I 2022-10-27 12:34:01] ...trap/engines/engine.py.230: loss: 0.20279 [I 2022-10-27 12:34:01] ...optimizers/trijoint.py.151: total_norm: 21.041744 [I 2022-10-27 12:34:07] ...trap/engines/engine.py.224: train: epoch 67 | batch 1250/2304 [I 2022-10-27 12:34:07] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:46 [I 2022-10-27 12:34:07] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00253 [I 2022-10-27 12:34:07] ...trap/engines/engine.py.230: loss: 0.16360 [I 2022-10-27 12:34:07] ...optimizers/trijoint.py.151: total_norm: 20.886740 [I 2022-10-27 12:34:13] ...trap/engines/engine.py.224: train: epoch 67 | batch 1260/2304 [I 2022-10-27 12:34:13] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:45 [I 2022-10-27 12:34:13] ...trap/engines/engine.py.229: process: 0.56116 | load: 0.00248 [I 2022-10-27 12:34:13] ...trap/engines/engine.py.230: loss: 0.15918 [I 2022-10-27 12:34:13] ...optimizers/trijoint.py.151: total_norm: 17.253010 [I 2022-10-27 12:34:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 1270/2304 [I 2022-10-27 12:34:18] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:37 [I 2022-10-27 12:34:18] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00246 [I 2022-10-27 12:34:18] ...trap/engines/engine.py.230: loss: 0.20383 [I 2022-10-27 12:34:18] ...optimizers/trijoint.py.151: total_norm: 12.495916 [I 2022-10-27 12:34:24] ...trap/engines/engine.py.224: train: epoch 67 | batch 1280/2304 [I 2022-10-27 12:34:24] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:28 [I 2022-10-27 12:34:24] ...trap/engines/engine.py.229: process: 0.54718 | load: 0.00248 [I 2022-10-27 12:34:24] ...trap/engines/engine.py.230: loss: 0.12952 [I 2022-10-27 12:34:24] ...optimizers/trijoint.py.151: total_norm: 13.773285 [I 2022-10-27 12:34:29] ...trap/engines/engine.py.224: train: epoch 67 | batch 1290/2304 [I 2022-10-27 12:34:29] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:26 [I 2022-10-27 12:34:29] ...trap/engines/engine.py.229: process: 0.56091 | load: 0.00240 [I 2022-10-27 12:34:29] ...trap/engines/engine.py.230: loss: 0.10055 [I 2022-10-27 12:34:29] ...optimizers/trijoint.py.151: total_norm: 11.426230 [I 2022-10-27 12:34:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 1300/2304 [I 2022-10-27 12:34:35] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:16 [I 2022-10-27 12:34:35] ...trap/engines/engine.py.229: process: 0.54850 | load: 0.00246 [I 2022-10-27 12:34:35] ...trap/engines/engine.py.230: loss: 0.09773 [I 2022-10-27 12:34:35] ...optimizers/trijoint.py.151: total_norm: 8.348854 [I 2022-10-27 12:34:40] ...trap/engines/engine.py.224: train: epoch 67 | batch 1310/2304 [I 2022-10-27 12:34:40] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:17 [I 2022-10-27 12:34:40] ...trap/engines/engine.py.229: process: 0.57650 | load: 0.00246 [I 2022-10-27 12:34:40] ...trap/engines/engine.py.230: loss: 0.15624 [I 2022-10-27 12:34:40] ...optimizers/trijoint.py.151: total_norm: 18.179983 [I 2022-10-27 12:34:46] ...trap/engines/engine.py.224: train: epoch 67 | batch 1320/2304 [I 2022-10-27 12:34:46] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:10 [I 2022-10-27 12:34:46] ...trap/engines/engine.py.229: process: 0.56613 | load: 0.00246 [I 2022-10-27 12:34:46] ...trap/engines/engine.py.230: loss: 0.13888 [I 2022-10-27 12:34:46] ...optimizers/trijoint.py.151: total_norm: 13.780129 [I 2022-10-27 12:34:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 1330/2304 [I 2022-10-27 12:34:52] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:08:58 [I 2022-10-27 12:34:52] ...trap/engines/engine.py.229: process: 0.55658 | load: 0.00247 [I 2022-10-27 12:34:52] ...trap/engines/engine.py.230: loss: 0.16352 [I 2022-10-27 12:34:52] ...optimizers/trijoint.py.151: total_norm: 28.442997 [I 2022-10-27 12:34:57] ...trap/engines/engine.py.224: train: epoch 67 | batch 1340/2304 [I 2022-10-27 12:34:57] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:57 [I 2022-10-27 12:34:57] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00245 [I 2022-10-27 12:34:57] ...trap/engines/engine.py.230: loss: 0.11187 [I 2022-10-27 12:34:57] ...optimizers/trijoint.py.151: total_norm: 14.789276 [I 2022-10-27 12:35:03] ...trap/engines/engine.py.224: train: epoch 67 | batch 1350/2304 [I 2022-10-27 12:35:03] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:49 [I 2022-10-27 12:35:03] ...trap/engines/engine.py.229: process: 0.55558 | load: 0.00265 [I 2022-10-27 12:35:03] ...trap/engines/engine.py.230: loss: 0.16769 [I 2022-10-27 12:35:03] ...optimizers/trijoint.py.151: total_norm: 23.082764 [I 2022-10-27 12:35:08] ...trap/engines/engine.py.224: train: epoch 67 | batch 1360/2304 [I 2022-10-27 12:35:08] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:47 [I 2022-10-27 12:35:08] ...trap/engines/engine.py.229: process: 0.58219 | load: 0.00248 [I 2022-10-27 12:35:08] ...trap/engines/engine.py.230: loss: 0.15776 [I 2022-10-27 12:35:08] ...optimizers/trijoint.py.151: total_norm: 17.094345 [I 2022-10-27 12:35:14] ...trap/engines/engine.py.224: train: epoch 67 | batch 1370/2304 [I 2022-10-27 12:35:14] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:40 [I 2022-10-27 12:35:14] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00244 [I 2022-10-27 12:35:14] ...trap/engines/engine.py.230: loss: 0.15050 [I 2022-10-27 12:35:14] ...optimizers/trijoint.py.151: total_norm: 20.900921 [I 2022-10-27 12:35:19] ...trap/engines/engine.py.224: train: epoch 67 | batch 1380/2304 [I 2022-10-27 12:35:19] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:33 [I 2022-10-27 12:35:19] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00243 [I 2022-10-27 12:35:19] ...trap/engines/engine.py.230: loss: 0.17889 [I 2022-10-27 12:35:19] ...optimizers/trijoint.py.151: total_norm: 10.782640 [I 2022-10-27 12:35:25] ...trap/engines/engine.py.224: train: epoch 67 | batch 1390/2304 [I 2022-10-27 12:35:25] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:31 [I 2022-10-27 12:35:25] ...trap/engines/engine.py.229: process: 0.56776 | load: 0.00242 [I 2022-10-27 12:35:25] ...trap/engines/engine.py.230: loss: 0.13493 [I 2022-10-27 12:35:25] ...optimizers/trijoint.py.151: total_norm: 23.370474 [I 2022-10-27 12:35:31] ...trap/engines/engine.py.224: train: epoch 67 | batch 1400/2304 [I 2022-10-27 12:35:31] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:24 [I 2022-10-27 12:35:31] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00242 [I 2022-10-27 12:35:31] ...trap/engines/engine.py.230: loss: 0.13946 [I 2022-10-27 12:35:31] ...optimizers/trijoint.py.151: total_norm: 18.715263 [I 2022-10-27 12:35:36] ...trap/engines/engine.py.224: train: epoch 67 | batch 1410/2304 [I 2022-10-27 12:35:36] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:17 [I 2022-10-27 12:35:36] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00248 [I 2022-10-27 12:35:36] ...trap/engines/engine.py.230: loss: 0.15889 [I 2022-10-27 12:35:36] ...optimizers/trijoint.py.151: total_norm: 11.849941 [I 2022-10-27 12:35:42] ...trap/engines/engine.py.224: train: epoch 67 | batch 1420/2304 [I 2022-10-27 12:35:42] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:13 [I 2022-10-27 12:35:42] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00244 [I 2022-10-27 12:35:42] ...trap/engines/engine.py.230: loss: 0.13694 [I 2022-10-27 12:35:42] ...optimizers/trijoint.py.151: total_norm: 65.953712 [I 2022-10-27 12:35:47] ...trap/engines/engine.py.224: train: epoch 67 | batch 1430/2304 [I 2022-10-27 12:35:47] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:05 [I 2022-10-27 12:35:47] ...trap/engines/engine.py.229: process: 0.54842 | load: 0.00244 [I 2022-10-27 12:35:47] ...trap/engines/engine.py.230: loss: 0.19258 [I 2022-10-27 12:35:47] ...optimizers/trijoint.py.151: total_norm: 47.847553 [I 2022-10-27 12:35:53] ...trap/engines/engine.py.224: train: epoch 67 | batch 1440/2304 [I 2022-10-27 12:35:53] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:03 [I 2022-10-27 12:35:53] ...trap/engines/engine.py.229: process: 0.56726 | load: 0.00242 [I 2022-10-27 12:35:53] ...trap/engines/engine.py.230: loss: 0.17050 [I 2022-10-27 12:35:53] ...optimizers/trijoint.py.151: total_norm: 12.937575 [I 2022-10-27 12:35:59] ...trap/engines/engine.py.224: train: epoch 67 | batch 1450/2304 [I 2022-10-27 12:35:59] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:07:54 [I 2022-10-27 12:35:59] ...trap/engines/engine.py.229: process: 0.56426 | load: 0.00243 [I 2022-10-27 12:35:59] ...trap/engines/engine.py.230: loss: 0.11828 [I 2022-10-27 12:35:59] ...optimizers/trijoint.py.151: total_norm: 13.793039 [I 2022-10-27 12:36:04] ...trap/engines/engine.py.224: train: epoch 67 | batch 1460/2304 [I 2022-10-27 12:36:04] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:48 [I 2022-10-27 12:36:04] ...trap/engines/engine.py.229: process: 0.54837 | load: 0.00243 [I 2022-10-27 12:36:04] ...trap/engines/engine.py.230: loss: 0.16313 [I 2022-10-27 12:36:04] ...optimizers/trijoint.py.151: total_norm: 8.658793 [I 2022-10-27 12:36:10] ...trap/engines/engine.py.224: train: epoch 67 | batch 1470/2304 [I 2022-10-27 12:36:10] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:45 [I 2022-10-27 12:36:10] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00244 [I 2022-10-27 12:36:10] ...trap/engines/engine.py.230: loss: 0.10627 [I 2022-10-27 12:36:10] ...optimizers/trijoint.py.151: total_norm: 14.159573 [I 2022-10-27 12:36:15] ...trap/engines/engine.py.224: train: epoch 67 | batch 1480/2304 [I 2022-10-27 12:36:15] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:42 [I 2022-10-27 12:36:15] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00238 [I 2022-10-27 12:36:15] ...trap/engines/engine.py.230: loss: 0.09090 [I 2022-10-27 12:36:15] ...optimizers/trijoint.py.151: total_norm: 11.894689 [I 2022-10-27 12:36:21] ...trap/engines/engine.py.224: train: epoch 67 | batch 1490/2304 [I 2022-10-27 12:36:21] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:37 [I 2022-10-27 12:36:21] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00248 [I 2022-10-27 12:36:21] ...trap/engines/engine.py.230: loss: 0.19192 [I 2022-10-27 12:36:21] ...optimizers/trijoint.py.151: total_norm: 13.575746 [I 2022-10-27 12:36:27] ...trap/engines/engine.py.224: train: epoch 67 | batch 1500/2304 [I 2022-10-27 12:36:27] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:31 [I 2022-10-27 12:36:27] ...trap/engines/engine.py.229: process: 0.56159 | load: 0.00243 [I 2022-10-27 12:36:27] ...trap/engines/engine.py.230: loss: 0.15263 [I 2022-10-27 12:36:27] ...optimizers/trijoint.py.151: total_norm: 31.028400 [I 2022-10-27 12:36:32] ...trap/engines/engine.py.224: train: epoch 67 | batch 1510/2304 [I 2022-10-27 12:36:32] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:19 [I 2022-10-27 12:36:32] ...trap/engines/engine.py.229: process: 0.54569 | load: 0.00249 [I 2022-10-27 12:36:32] ...trap/engines/engine.py.230: loss: 0.13528 [I 2022-10-27 12:36:32] ...optimizers/trijoint.py.151: total_norm: 15.320333 [I 2022-10-27 12:36:38] ...trap/engines/engine.py.224: train: epoch 67 | batch 1520/2304 [I 2022-10-27 12:36:38] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:16 [I 2022-10-27 12:36:38] ...trap/engines/engine.py.229: process: 0.55451 | load: 0.00248 [I 2022-10-27 12:36:38] ...trap/engines/engine.py.230: loss: 0.19904 [I 2022-10-27 12:36:38] ...optimizers/trijoint.py.151: total_norm: 9.801686 [I 2022-10-27 12:36:43] ...trap/engines/engine.py.224: train: epoch 67 | batch 1530/2304 [I 2022-10-27 12:36:43] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:10 [I 2022-10-27 12:36:43] ...trap/engines/engine.py.229: process: 0.55767 | load: 0.00247 [I 2022-10-27 12:36:43] ...trap/engines/engine.py.230: loss: 0.17817 [I 2022-10-27 12:36:43] ...optimizers/trijoint.py.151: total_norm: 15.133518 [I 2022-10-27 12:36:49] ...trap/engines/engine.py.224: train: epoch 67 | batch 1540/2304 [I 2022-10-27 12:36:49] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:05 [I 2022-10-27 12:36:49] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00244 [I 2022-10-27 12:36:49] ...trap/engines/engine.py.230: loss: 0.12821 [I 2022-10-27 12:36:49] ...optimizers/trijoint.py.151: total_norm: 8.980122 [I 2022-10-27 12:36:55] ...trap/engines/engine.py.224: train: epoch 67 | batch 1550/2304 [I 2022-10-27 12:36:55] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:06:59 [I 2022-10-27 12:36:55] ...trap/engines/engine.py.229: process: 0.54845 | load: 0.00244 [I 2022-10-27 12:36:55] ...trap/engines/engine.py.230: loss: 0.19298 [I 2022-10-27 12:36:55] ...optimizers/trijoint.py.151: total_norm: 42.706078 [I 2022-10-27 12:37:00] ...trap/engines/engine.py.224: train: epoch 67 | batch 1560/2304 [I 2022-10-27 12:37:00] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:06:54 [I 2022-10-27 12:37:00] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00243 [I 2022-10-27 12:37:00] ...trap/engines/engine.py.230: loss: 0.18283 [I 2022-10-27 12:37:00] ...optimizers/trijoint.py.151: total_norm: 11.103231 [I 2022-10-27 12:37:06] ...trap/engines/engine.py.224: train: epoch 67 | batch 1570/2304 [I 2022-10-27 12:37:06] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:46 [I 2022-10-27 12:37:06] ...trap/engines/engine.py.229: process: 0.53979 | load: 0.00243 [I 2022-10-27 12:37:06] ...trap/engines/engine.py.230: loss: 0.12957 [I 2022-10-27 12:37:06] ...optimizers/trijoint.py.151: total_norm: 32.061672 [I 2022-10-27 12:37:11] ...trap/engines/engine.py.224: train: epoch 67 | batch 1580/2304 [I 2022-10-27 12:37:11] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:45 [I 2022-10-27 12:37:11] ...trap/engines/engine.py.229: process: 0.56249 | load: 0.00240 [I 2022-10-27 12:37:11] ...trap/engines/engine.py.230: loss: 0.16144 [I 2022-10-27 12:37:11] ...optimizers/trijoint.py.151: total_norm: 11.693327 [I 2022-10-27 12:37:17] ...trap/engines/engine.py.224: train: epoch 67 | batch 1590/2304 [I 2022-10-27 12:37:17] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:38 [I 2022-10-27 12:37:17] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00242 [I 2022-10-27 12:37:17] ...trap/engines/engine.py.230: loss: 0.15355 [I 2022-10-27 12:37:17] ...optimizers/trijoint.py.151: total_norm: 14.547283 [I 2022-10-27 12:37:22] ...trap/engines/engine.py.224: train: epoch 67 | batch 1600/2304 [I 2022-10-27 12:37:22] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:33 [I 2022-10-27 12:37:22] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00240 [I 2022-10-27 12:37:22] ...trap/engines/engine.py.230: loss: 0.14045 [I 2022-10-27 12:37:22] ...optimizers/trijoint.py.151: total_norm: 13.491319 [I 2022-10-27 12:37:28] ...trap/engines/engine.py.224: train: epoch 67 | batch 1610/2304 [I 2022-10-27 12:37:28] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:29 [I 2022-10-27 12:37:28] ...trap/engines/engine.py.229: process: 0.56263 | load: 0.00242 [I 2022-10-27 12:37:28] ...trap/engines/engine.py.230: loss: 0.14111 [I 2022-10-27 12:37:28] ...optimizers/trijoint.py.151: total_norm: 8.723401 [I 2022-10-27 12:37:34] ...trap/engines/engine.py.224: train: epoch 67 | batch 1620/2304 [I 2022-10-27 12:37:34] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:22 [I 2022-10-27 12:37:34] ...trap/engines/engine.py.229: process: 0.56049 | load: 0.00253 [I 2022-10-27 12:37:34] ...trap/engines/engine.py.230: loss: 0.16087 [I 2022-10-27 12:37:34] ...optimizers/trijoint.py.151: total_norm: 17.822546 [I 2022-10-27 12:37:39] ...trap/engines/engine.py.224: train: epoch 67 | batch 1630/2304 [I 2022-10-27 12:37:39] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:16 [I 2022-10-27 12:37:39] ...trap/engines/engine.py.229: process: 0.56462 | load: 0.00245 [I 2022-10-27 12:37:39] ...trap/engines/engine.py.230: loss: 0.14013 [I 2022-10-27 12:37:39] ...optimizers/trijoint.py.151: total_norm: 19.676649 [I 2022-10-27 12:37:45] ...trap/engines/engine.py.224: train: epoch 67 | batch 1640/2304 [I 2022-10-27 12:37:45] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:10 [I 2022-10-27 12:37:45] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00248 [I 2022-10-27 12:37:45] ...trap/engines/engine.py.230: loss: 0.18905 [I 2022-10-27 12:37:45] ...optimizers/trijoint.py.151: total_norm: 30.880077 [I 2022-10-27 12:37:51] ...trap/engines/engine.py.224: train: epoch 67 | batch 1650/2304 [I 2022-10-27 12:37:51] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:05 [I 2022-10-27 12:37:51] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00243 [I 2022-10-27 12:37:51] ...trap/engines/engine.py.230: loss: 0.14566 [I 2022-10-27 12:37:51] ...optimizers/trijoint.py.151: total_norm: 14.170720 [I 2022-10-27 12:38:15] ...trap/engines/engine.py.224: train: epoch 67 | batch 1660/2304 [I 2022-10-27 12:38:15] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:38:29 [I 2022-10-27 12:38:15] ...trap/engines/engine.py.229: process: 0.56547 | load: 0.00244 [I 2022-10-27 12:38:15] ...trap/engines/engine.py.230: loss: 0.15856 [I 2022-10-27 12:38:15] ...optimizers/trijoint.py.151: total_norm: 24.130751 [I 2022-10-27 12:38:21] ...trap/engines/engine.py.224: train: epoch 67 | batch 1670/2304 [I 2022-10-27 12:38:21] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:09:19 [I 2022-10-27 12:38:21] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00238 [I 2022-10-27 12:38:21] ...trap/engines/engine.py.230: loss: 0.14132 [I 2022-10-27 12:38:21] ...optimizers/trijoint.py.151: total_norm: 16.432108 [I 2022-10-27 12:38:26] ...trap/engines/engine.py.224: train: epoch 67 | batch 1680/2304 [I 2022-10-27 12:38:26] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:06:09 [I 2022-10-27 12:38:26] ...trap/engines/engine.py.229: process: 0.55206 | load: 0.00239 [I 2022-10-27 12:38:26] ...trap/engines/engine.py.230: loss: 0.16185 [I 2022-10-27 12:38:26] ...optimizers/trijoint.py.151: total_norm: 29.795385 [I 2022-10-27 12:38:32] ...trap/engines/engine.py.224: train: epoch 67 | batch 1690/2304 [I 2022-10-27 12:38:32] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:45 [I 2022-10-27 12:38:32] ...trap/engines/engine.py.229: process: 0.56442 | load: 0.00239 [I 2022-10-27 12:38:32] ...trap/engines/engine.py.230: loss: 0.10157 [I 2022-10-27 12:38:32] ...optimizers/trijoint.py.151: total_norm: 9.143861 [I 2022-10-27 12:38:37] ...trap/engines/engine.py.224: train: epoch 67 | batch 1700/2304 [I 2022-10-27 12:38:37] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:37 [I 2022-10-27 12:38:37] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00237 [I 2022-10-27 12:38:37] ...trap/engines/engine.py.230: loss: 0.10692 [I 2022-10-27 12:38:37] ...optimizers/trijoint.py.151: total_norm: 6.725646 [I 2022-10-27 12:38:43] ...trap/engines/engine.py.224: train: epoch 67 | batch 1710/2304 [I 2022-10-27 12:38:43] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:27 [I 2022-10-27 12:38:43] ...trap/engines/engine.py.229: process: 0.54438 | load: 0.00239 [I 2022-10-27 12:38:43] ...trap/engines/engine.py.230: loss: 0.23409 [I 2022-10-27 12:38:43] ...optimizers/trijoint.py.151: total_norm: 28.331432 [I 2022-10-27 12:38:49] ...trap/engines/engine.py.224: train: epoch 67 | batch 1720/2304 [I 2022-10-27 12:38:49] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:36 [I 2022-10-27 12:38:49] ...trap/engines/engine.py.229: process: 0.54836 | load: 0.00240 [I 2022-10-27 12:38:49] ...trap/engines/engine.py.230: loss: 0.10350 [I 2022-10-27 12:38:49] ...optimizers/trijoint.py.151: total_norm: 12.589883 [I 2022-10-27 12:38:54] ...trap/engines/engine.py.224: train: epoch 67 | batch 1730/2304 [I 2022-10-27 12:38:54] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:18 [I 2022-10-27 12:38:54] ...trap/engines/engine.py.229: process: 0.54101 | load: 0.00238 [I 2022-10-27 12:38:54] ...trap/engines/engine.py.230: loss: 0.10876 [I 2022-10-27 12:38:54] ...optimizers/trijoint.py.151: total_norm: 7.949423 [I 2022-10-27 12:39:00] ...trap/engines/engine.py.224: train: epoch 67 | batch 1740/2304 [I 2022-10-27 12:39:00] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:05:12 [I 2022-10-27 12:39:00] ...trap/engines/engine.py.229: process: 0.55248 | load: 0.00241 [I 2022-10-27 12:39:00] ...trap/engines/engine.py.230: loss: 0.09209 [I 2022-10-27 12:39:00] ...optimizers/trijoint.py.151: total_norm: 9.308231 [I 2022-10-27 12:39:05] ...trap/engines/engine.py.224: train: epoch 67 | batch 1750/2304 [I 2022-10-27 12:39:05] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:05:09 [I 2022-10-27 12:39:05] ...trap/engines/engine.py.229: process: 0.55734 | load: 0.00245 [I 2022-10-27 12:39:05] ...trap/engines/engine.py.230: loss: 0.16482 [I 2022-10-27 12:39:05] ...optimizers/trijoint.py.151: total_norm: 16.578272 [I 2022-10-27 12:39:11] ...trap/engines/engine.py.224: train: epoch 67 | batch 1760/2304 [I 2022-10-27 12:39:11] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:05:02 [I 2022-10-27 12:39:11] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00245 [I 2022-10-27 12:39:11] ...trap/engines/engine.py.230: loss: 0.19014 [I 2022-10-27 12:39:11] ...optimizers/trijoint.py.151: total_norm: 24.470264 [I 2022-10-27 12:39:17] ...trap/engines/engine.py.224: train: epoch 67 | batch 1770/2304 [I 2022-10-27 12:39:17] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:58 [I 2022-10-27 12:39:17] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00241 [I 2022-10-27 12:39:17] ...trap/engines/engine.py.230: loss: 0.17038 [I 2022-10-27 12:39:17] ...optimizers/trijoint.py.151: total_norm: 27.577623 [I 2022-10-27 12:39:22] ...trap/engines/engine.py.224: train: epoch 67 | batch 1780/2304 [I 2022-10-27 12:39:22] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:52 [I 2022-10-27 12:39:22] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00239 [I 2022-10-27 12:39:22] ...trap/engines/engine.py.230: loss: 0.23029 [I 2022-10-27 12:39:22] ...optimizers/trijoint.py.151: total_norm: 57.143837 [I 2022-10-27 12:39:28] ...trap/engines/engine.py.224: train: epoch 67 | batch 1790/2304 [I 2022-10-27 12:39:28] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:46 [I 2022-10-27 12:39:28] ...trap/engines/engine.py.229: process: 0.55383 | load: 0.00239 [I 2022-10-27 12:39:28] ...trap/engines/engine.py.230: loss: 0.13169 [I 2022-10-27 12:39:28] ...optimizers/trijoint.py.151: total_norm: 23.992811 [I 2022-10-27 12:39:33] ...trap/engines/engine.py.224: train: epoch 67 | batch 1800/2304 [I 2022-10-27 12:39:33] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:40 [I 2022-10-27 12:39:33] ...trap/engines/engine.py.229: process: 0.54512 | load: 0.00240 [I 2022-10-27 12:39:33] ...trap/engines/engine.py.230: loss: 0.13571 [I 2022-10-27 12:39:33] ...optimizers/trijoint.py.151: total_norm: 7.517910 [I 2022-10-27 12:39:39] ...trap/engines/engine.py.224: train: epoch 67 | batch 1810/2304 [I 2022-10-27 12:39:39] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:35 [I 2022-10-27 12:39:39] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00240 [I 2022-10-27 12:39:39] ...trap/engines/engine.py.230: loss: 0.23279 [I 2022-10-27 12:39:39] ...optimizers/trijoint.py.151: total_norm: 12.704679 [I 2022-10-27 12:39:44] ...trap/engines/engine.py.224: train: epoch 67 | batch 1820/2304 [I 2022-10-27 12:39:44] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:29 [I 2022-10-27 12:39:44] ...trap/engines/engine.py.229: process: 0.55534 | load: 0.00238 [I 2022-10-27 12:39:44] ...trap/engines/engine.py.230: loss: 0.15103 [I 2022-10-27 12:39:44] ...optimizers/trijoint.py.151: total_norm: 18.063189 [I 2022-10-27 12:39:50] ...trap/engines/engine.py.224: train: epoch 67 | batch 1830/2304 [I 2022-10-27 12:39:50] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:25 [I 2022-10-27 12:39:50] ...trap/engines/engine.py.229: process: 0.56602 | load: 0.00240 [I 2022-10-27 12:39:50] ...trap/engines/engine.py.230: loss: 0.15017 [I 2022-10-27 12:39:50] ...optimizers/trijoint.py.151: total_norm: 23.150574 [I 2022-10-27 12:39:56] ...trap/engines/engine.py.224: train: epoch 67 | batch 1840/2304 [I 2022-10-27 12:39:56] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:18 [I 2022-10-27 12:39:56] ...trap/engines/engine.py.229: process: 0.56464 | load: 0.00239 [I 2022-10-27 12:39:56] ...trap/engines/engine.py.230: loss: 0.14006 [I 2022-10-27 12:39:56] ...optimizers/trijoint.py.151: total_norm: 15.727944 [I 2022-10-27 12:40:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 1850/2304 [I 2022-10-27 12:40:01] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:14 [I 2022-10-27 12:40:01] ...trap/engines/engine.py.229: process: 0.55185 | load: 0.00237 [I 2022-10-27 12:40:01] ...trap/engines/engine.py.230: loss: 0.16531 [I 2022-10-27 12:40:01] ...optimizers/trijoint.py.151: total_norm: 14.401345 [I 2022-10-27 12:40:07] ...trap/engines/engine.py.224: train: epoch 67 | batch 1860/2304 [I 2022-10-27 12:40:07] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:04:06 [I 2022-10-27 12:40:07] ...trap/engines/engine.py.229: process: 0.54527 | load: 0.00240 [I 2022-10-27 12:40:07] ...trap/engines/engine.py.230: loss: 0.15169 [I 2022-10-27 12:40:07] ...optimizers/trijoint.py.151: total_norm: 18.945759 [I 2022-10-27 12:40:13] ...trap/engines/engine.py.224: train: epoch 67 | batch 1870/2304 [I 2022-10-27 12:40:13] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:04:02 [I 2022-10-27 12:40:13] ...trap/engines/engine.py.229: process: 0.54928 | load: 0.00241 [I 2022-10-27 12:40:13] ...trap/engines/engine.py.230: loss: 0.14727 [I 2022-10-27 12:40:13] ...optimizers/trijoint.py.151: total_norm: 13.675590 [I 2022-10-27 12:40:18] ...trap/engines/engine.py.224: train: epoch 67 | batch 1880/2304 [I 2022-10-27 12:40:18] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:55 [I 2022-10-27 12:40:18] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00237 [I 2022-10-27 12:40:18] ...trap/engines/engine.py.230: loss: 0.15140 [I 2022-10-27 12:40:18] ...optimizers/trijoint.py.151: total_norm: 18.263288 [I 2022-10-27 12:40:24] ...trap/engines/engine.py.224: train: epoch 67 | batch 1890/2304 [I 2022-10-27 12:40:24] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:51 [I 2022-10-27 12:40:24] ...trap/engines/engine.py.229: process: 0.56237 | load: 0.00250 [I 2022-10-27 12:40:24] ...trap/engines/engine.py.230: loss: 0.17063 [I 2022-10-27 12:40:24] ...optimizers/trijoint.py.151: total_norm: 18.171623 [I 2022-10-27 12:40:29] ...trap/engines/engine.py.224: train: epoch 67 | batch 1900/2304 [I 2022-10-27 12:40:29] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:45 [I 2022-10-27 12:40:29] ...trap/engines/engine.py.229: process: 0.56497 | load: 0.00246 [I 2022-10-27 12:40:29] ...trap/engines/engine.py.230: loss: 0.17194 [I 2022-10-27 12:40:29] ...optimizers/trijoint.py.151: total_norm: 16.468145 [I 2022-10-27 12:40:35] ...trap/engines/engine.py.224: train: epoch 67 | batch 1910/2304 [I 2022-10-27 12:40:35] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:40 [I 2022-10-27 12:40:35] ...trap/engines/engine.py.229: process: 0.54919 | load: 0.00250 [I 2022-10-27 12:40:35] ...trap/engines/engine.py.230: loss: 0.20209 [I 2022-10-27 12:40:35] ...optimizers/trijoint.py.151: total_norm: 16.731255 [I 2022-10-27 12:40:40] ...trap/engines/engine.py.224: train: epoch 67 | batch 1920/2304 [I 2022-10-27 12:40:40] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:33 [I 2022-10-27 12:40:40] ...trap/engines/engine.py.229: process: 0.55605 | load: 0.00243 [I 2022-10-27 12:40:40] ...trap/engines/engine.py.230: loss: 0.13335 [I 2022-10-27 12:40:40] ...optimizers/trijoint.py.151: total_norm: 14.815704 [I 2022-10-27 12:40:46] ...trap/engines/engine.py.224: train: epoch 67 | batch 1930/2304 [I 2022-10-27 12:40:46] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:28 [I 2022-10-27 12:40:46] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00243 [I 2022-10-27 12:40:46] ...trap/engines/engine.py.230: loss: 0.14997 [I 2022-10-27 12:40:46] ...optimizers/trijoint.py.151: total_norm: 20.810413 [I 2022-10-27 12:40:52] ...trap/engines/engine.py.224: train: epoch 67 | batch 1940/2304 [I 2022-10-27 12:40:52] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:22 [I 2022-10-27 12:40:52] ...trap/engines/engine.py.229: process: 0.54180 | load: 0.00250 [I 2022-10-27 12:40:52] ...trap/engines/engine.py.230: loss: 0.17768 [I 2022-10-27 12:40:52] ...optimizers/trijoint.py.151: total_norm: 12.586863 [I 2022-10-27 12:40:57] ...trap/engines/engine.py.224: train: epoch 67 | batch 1950/2304 [I 2022-10-27 12:40:57] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:16 [I 2022-10-27 12:40:57] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00245 [I 2022-10-27 12:40:57] ...trap/engines/engine.py.230: loss: 0.15015 [I 2022-10-27 12:40:57] ...optimizers/trijoint.py.151: total_norm: 17.153124 [I 2022-10-27 12:41:03] ...trap/engines/engine.py.224: train: epoch 67 | batch 1960/2304 [I 2022-10-27 12:41:03] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:03:16 [I 2022-10-27 12:41:03] ...trap/engines/engine.py.229: process: 0.54430 | load: 0.00247 [I 2022-10-27 12:41:03] ...trap/engines/engine.py.230: loss: 0.20395 [I 2022-10-27 12:41:03] ...optimizers/trijoint.py.151: total_norm: 8.111409 [I 2022-10-27 12:41:09] ...trap/engines/engine.py.224: train: epoch 67 | batch 1970/2304 [I 2022-10-27 12:41:09] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:03:06 [I 2022-10-27 12:41:09] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00244 [I 2022-10-27 12:41:09] ...trap/engines/engine.py.230: loss: 0.14563 [I 2022-10-27 12:41:09] ...optimizers/trijoint.py.151: total_norm: 17.664270 [I 2022-10-27 12:41:14] ...trap/engines/engine.py.224: train: epoch 67 | batch 1980/2304 [I 2022-10-27 12:41:14] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:59 [I 2022-10-27 12:41:14] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00260 [I 2022-10-27 12:41:14] ...trap/engines/engine.py.230: loss: 0.13281 [I 2022-10-27 12:41:14] ...optimizers/trijoint.py.151: total_norm: 14.907022 [I 2022-10-27 12:41:20] ...trap/engines/engine.py.224: train: epoch 67 | batch 1990/2304 [I 2022-10-27 12:41:20] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:54 [I 2022-10-27 12:41:20] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00252 [I 2022-10-27 12:41:20] ...trap/engines/engine.py.230: loss: 0.13081 [I 2022-10-27 12:41:20] ...optimizers/trijoint.py.151: total_norm: 11.891869 [I 2022-10-27 12:41:25] ...trap/engines/engine.py.224: train: epoch 67 | batch 2000/2304 [I 2022-10-27 12:41:25] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:49 [I 2022-10-27 12:41:25] ...trap/engines/engine.py.229: process: 0.55556 | load: 0.00251 [I 2022-10-27 12:41:25] ...trap/engines/engine.py.230: loss: 0.15549 [I 2022-10-27 12:41:25] ...optimizers/trijoint.py.151: total_norm: 18.735767 [I 2022-10-27 12:41:31] ...trap/engines/engine.py.224: train: epoch 67 | batch 2010/2304 [I 2022-10-27 12:41:31] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:41 [I 2022-10-27 12:41:31] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00244 [I 2022-10-27 12:41:31] ...trap/engines/engine.py.230: loss: 0.21239 [I 2022-10-27 12:41:31] ...optimizers/trijoint.py.151: total_norm: 41.227283 [I 2022-10-27 12:41:36] ...trap/engines/engine.py.224: train: epoch 67 | batch 2020/2304 [I 2022-10-27 12:41:36] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:38 [I 2022-10-27 12:41:36] ...trap/engines/engine.py.229: process: 0.55108 | load: 0.00260 [I 2022-10-27 12:41:36] ...trap/engines/engine.py.230: loss: 0.10055 [I 2022-10-27 12:41:36] ...optimizers/trijoint.py.151: total_norm: 8.366345 [I 2022-10-27 12:41:42] ...trap/engines/engine.py.224: train: epoch 67 | batch 2030/2304 [I 2022-10-27 12:41:42] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:32 [I 2022-10-27 12:41:42] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00245 [I 2022-10-27 12:41:42] ...trap/engines/engine.py.230: loss: 0.10198 [I 2022-10-27 12:41:42] ...optimizers/trijoint.py.151: total_norm: 32.135841 [I 2022-10-27 12:41:47] ...trap/engines/engine.py.224: train: epoch 67 | batch 2040/2304 [I 2022-10-27 12:41:47] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:26 [I 2022-10-27 12:41:47] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00249 [I 2022-10-27 12:41:47] ...trap/engines/engine.py.230: loss: 0.14700 [I 2022-10-27 12:41:47] ...optimizers/trijoint.py.151: total_norm: 14.943394 [I 2022-10-27 12:41:53] ...trap/engines/engine.py.224: train: epoch 67 | batch 2050/2304 [I 2022-10-27 12:41:53] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:21 [I 2022-10-27 12:41:53] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00248 [I 2022-10-27 12:41:53] ...trap/engines/engine.py.230: loss: 0.15017 [I 2022-10-27 12:41:53] ...optimizers/trijoint.py.151: total_norm: 24.069683 [I 2022-10-27 12:41:59] ...trap/engines/engine.py.224: train: epoch 67 | batch 2060/2304 [I 2022-10-27 12:41:59] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:02:16 [I 2022-10-27 12:41:59] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00242 [I 2022-10-27 12:41:59] ...trap/engines/engine.py.230: loss: 0.13413 [I 2022-10-27 12:41:59] ...optimizers/trijoint.py.151: total_norm: 22.559111 [I 2022-10-27 12:42:04] ...trap/engines/engine.py.224: train: epoch 67 | batch 2070/2304 [I 2022-10-27 12:42:04] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:08 [I 2022-10-27 12:42:04] ...trap/engines/engine.py.229: process: 0.54588 | load: 0.00242 [I 2022-10-27 12:42:04] ...trap/engines/engine.py.230: loss: 0.12971 [I 2022-10-27 12:42:04] ...optimizers/trijoint.py.151: total_norm: 25.467478 [I 2022-10-27 12:42:10] ...trap/engines/engine.py.224: train: epoch 67 | batch 2080/2304 [I 2022-10-27 12:42:10] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:02:05 [I 2022-10-27 12:42:10] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00241 [I 2022-10-27 12:42:10] ...trap/engines/engine.py.230: loss: 0.15881 [I 2022-10-27 12:42:10] ...optimizers/trijoint.py.151: total_norm: 14.412364 [I 2022-10-27 12:42:15] ...trap/engines/engine.py.224: train: epoch 67 | batch 2090/2304 [I 2022-10-27 12:42:15] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:58 [I 2022-10-27 12:42:15] ...trap/engines/engine.py.229: process: 0.54602 | load: 0.00241 [I 2022-10-27 12:42:15] ...trap/engines/engine.py.230: loss: 0.11491 [I 2022-10-27 12:42:15] ...optimizers/trijoint.py.151: total_norm: 30.036121 [I 2022-10-27 12:42:21] ...trap/engines/engine.py.224: train: epoch 67 | batch 2100/2304 [I 2022-10-27 12:42:21] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:52 [I 2022-10-27 12:42:21] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00245 [I 2022-10-27 12:42:21] ...trap/engines/engine.py.230: loss: 0.12747 [I 2022-10-27 12:42:21] ...optimizers/trijoint.py.151: total_norm: 14.468754 [I 2022-10-27 12:42:26] ...trap/engines/engine.py.224: train: epoch 67 | batch 2110/2304 [I 2022-10-27 12:42:26] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:48 [I 2022-10-27 12:42:26] ...trap/engines/engine.py.229: process: 0.56299 | load: 0.00240 [I 2022-10-27 12:42:26] ...trap/engines/engine.py.230: loss: 0.14682 [I 2022-10-27 12:42:26] ...optimizers/trijoint.py.151: total_norm: 31.150421 [I 2022-10-27 12:42:32] ...trap/engines/engine.py.224: train: epoch 67 | batch 2120/2304 [I 2022-10-27 12:42:32] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:41 [I 2022-10-27 12:42:32] ...trap/engines/engine.py.229: process: 0.54645 | load: 0.00245 [I 2022-10-27 12:42:32] ...trap/engines/engine.py.230: loss: 0.16095 [I 2022-10-27 12:42:32] ...optimizers/trijoint.py.151: total_norm: 9.580345 [I 2022-10-27 12:42:38] ...trap/engines/engine.py.224: train: epoch 67 | batch 2130/2304 [I 2022-10-27 12:42:38] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:35 [I 2022-10-27 12:42:38] ...trap/engines/engine.py.229: process: 0.54279 | load: 0.00243 [I 2022-10-27 12:42:38] ...trap/engines/engine.py.230: loss: 0.21162 [I 2022-10-27 12:42:38] ...optimizers/trijoint.py.151: total_norm: 48.326176 [I 2022-10-27 12:42:43] ...trap/engines/engine.py.224: train: epoch 67 | batch 2140/2304 [I 2022-10-27 12:42:43] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:30 [I 2022-10-27 12:42:43] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00243 [I 2022-10-27 12:42:43] ...trap/engines/engine.py.230: loss: 0.11767 [I 2022-10-27 12:42:43] ...optimizers/trijoint.py.151: total_norm: 25.363794 [I 2022-10-27 12:42:49] ...trap/engines/engine.py.224: train: epoch 67 | batch 2150/2304 [I 2022-10-27 12:42:49] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:25 [I 2022-10-27 12:42:49] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00257 [I 2022-10-27 12:42:49] ...trap/engines/engine.py.230: loss: 0.12235 [I 2022-10-27 12:42:49] ...optimizers/trijoint.py.151: total_norm: 18.660263 [I 2022-10-27 12:42:54] ...trap/engines/engine.py.224: train: epoch 67 | batch 2160/2304 [I 2022-10-27 12:42:54] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:19 [I 2022-10-27 12:42:54] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00250 [I 2022-10-27 12:42:54] ...trap/engines/engine.py.230: loss: 0.11792 [I 2022-10-27 12:42:54] ...optimizers/trijoint.py.151: total_norm: 13.664378 [I 2022-10-27 12:43:00] ...trap/engines/engine.py.224: train: epoch 67 | batch 2170/2304 [I 2022-10-27 12:43:00] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:01:14 [I 2022-10-27 12:43:00] ...trap/engines/engine.py.229: process: 0.54680 | load: 0.00250 [I 2022-10-27 12:43:00] ...trap/engines/engine.py.230: loss: 0.19685 [I 2022-10-27 12:43:00] ...optimizers/trijoint.py.151: total_norm: 17.528706 [I 2022-10-27 12:43:05] ...trap/engines/engine.py.224: train: epoch 67 | batch 2180/2304 [I 2022-10-27 12:43:05] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:01:08 [I 2022-10-27 12:43:05] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00250 [I 2022-10-27 12:43:05] ...trap/engines/engine.py.230: loss: 0.10739 [I 2022-10-27 12:43:05] ...optimizers/trijoint.py.151: total_norm: 16.864290 [I 2022-10-27 12:43:11] ...trap/engines/engine.py.224: train: epoch 67 | batch 2190/2304 [I 2022-10-27 12:43:11] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:01:02 [I 2022-10-27 12:43:11] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00245 [I 2022-10-27 12:43:11] ...trap/engines/engine.py.230: loss: 0.11213 [I 2022-10-27 12:43:11] ...optimizers/trijoint.py.151: total_norm: 16.174282 [I 2022-10-27 12:43:16] ...trap/engines/engine.py.224: train: epoch 67 | batch 2200/2304 [I 2022-10-27 12:43:16] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:57 [I 2022-10-27 12:43:16] ...trap/engines/engine.py.229: process: 0.56397 | load: 0.00257 [I 2022-10-27 12:43:16] ...trap/engines/engine.py.230: loss: 0.16746 [I 2022-10-27 12:43:16] ...optimizers/trijoint.py.151: total_norm: 20.931267 [I 2022-10-27 12:43:22] ...trap/engines/engine.py.224: train: epoch 67 | batch 2210/2304 [I 2022-10-27 12:43:22] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:52 [I 2022-10-27 12:43:22] ...trap/engines/engine.py.229: process: 0.55258 | load: 0.00256 [I 2022-10-27 12:43:22] ...trap/engines/engine.py.230: loss: 0.11260 [I 2022-10-27 12:43:22] ...optimizers/trijoint.py.151: total_norm: 8.758240 [I 2022-10-27 12:43:28] ...trap/engines/engine.py.224: train: epoch 67 | batch 2220/2304 [I 2022-10-27 12:43:28] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:46 [I 2022-10-27 12:43:28] ...trap/engines/engine.py.229: process: 0.55612 | load: 0.00247 [I 2022-10-27 12:43:28] ...trap/engines/engine.py.230: loss: 0.16862 [I 2022-10-27 12:43:28] ...optimizers/trijoint.py.151: total_norm: 10.421054 [I 2022-10-27 12:43:33] ...trap/engines/engine.py.224: train: epoch 67 | batch 2230/2304 [I 2022-10-27 12:43:33] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:40 [I 2022-10-27 12:43:33] ...trap/engines/engine.py.229: process: 0.54317 | load: 0.00246 [I 2022-10-27 12:43:33] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-27 12:43:33] ...optimizers/trijoint.py.151: total_norm: 12.461122 [I 2022-10-27 12:43:39] ...trap/engines/engine.py.224: train: epoch 67 | batch 2240/2304 [I 2022-10-27 12:43:39] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:35 [I 2022-10-27 12:43:39] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00276 [I 2022-10-27 12:43:39] ...trap/engines/engine.py.230: loss: 0.20117 [I 2022-10-27 12:43:39] ...optimizers/trijoint.py.151: total_norm: 16.714930 [I 2022-10-27 12:43:44] ...trap/engines/engine.py.224: train: epoch 67 | batch 2250/2304 [I 2022-10-27 12:43:44] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:30 [I 2022-10-27 12:43:44] ...trap/engines/engine.py.229: process: 0.54611 | load: 0.00268 [I 2022-10-27 12:43:44] ...trap/engines/engine.py.230: loss: 0.11487 [I 2022-10-27 12:43:44] ...optimizers/trijoint.py.151: total_norm: 20.514261 [I 2022-10-27 12:43:50] ...trap/engines/engine.py.224: train: epoch 67 | batch 2260/2304 [I 2022-10-27 12:43:50] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:24 [I 2022-10-27 12:43:50] ...trap/engines/engine.py.229: process: 0.55145 | load: 0.00277 [I 2022-10-27 12:43:50] ...trap/engines/engine.py.230: loss: 0.15592 [I 2022-10-27 12:43:50] ...optimizers/trijoint.py.151: total_norm: 28.466263 [I 2022-10-27 12:43:55] ...trap/engines/engine.py.224: train: epoch 67 | batch 2270/2304 [I 2022-10-27 12:43:55] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:18 [I 2022-10-27 12:43:55] ...trap/engines/engine.py.229: process: 0.54423 | load: 0.00263 [I 2022-10-27 12:43:55] ...trap/engines/engine.py.230: loss: 0.24395 [I 2022-10-27 12:43:55] ...optimizers/trijoint.py.151: total_norm: 18.667858 [I 2022-10-27 12:44:01] ...trap/engines/engine.py.224: train: epoch 67 | batch 2280/2304 [I 2022-10-27 12:44:01] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:13 [I 2022-10-27 12:44:01] ...trap/engines/engine.py.229: process: 0.54604 | load: 0.00260 [I 2022-10-27 12:44:01] ...trap/engines/engine.py.230: loss: 0.13620 [I 2022-10-27 12:44:01] ...optimizers/trijoint.py.151: total_norm: 14.650647 [I 2022-10-27 12:44:07] ...trap/engines/engine.py.224: train: epoch 67 | batch 2290/2304 [I 2022-10-27 12:44:07] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:07 [I 2022-10-27 12:44:07] ...trap/engines/engine.py.229: process: 0.56569 | load: 0.00257 [I 2022-10-27 12:44:07] ...trap/engines/engine.py.230: loss: 0.18764 [I 2022-10-27 12:44:07] ...optimizers/trijoint.py.151: total_norm: 17.103432 [I 2022-10-27 12:44:12] ...trap/engines/engine.py.224: train: epoch 67 | batch 2300/2304 [I 2022-10-27 12:44:12] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:02 [I 2022-10-27 12:44:12] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00016 [I 2022-10-27 12:44:12] ...trap/engines/engine.py.230: loss: 0.19243 [I 2022-10-27 12:44:12] ...optimizers/trijoint.py.151: total_norm: 27.096882 [I 2022-10-27 12:44:14] ...trap/engines/engine.py.224: train: epoch 67 | batch 2304/2304 [I 2022-10-27 12:44:14] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:00 [I 2022-10-27 12:44:14] ...trap/engines/engine.py.229: process: 0.54720 | load: 0.00017 [I 2022-10-27 12:44:14] ...trap/engines/engine.py.230: loss: 0.21047 [I 2022-10-27 12:44:14] ...optimizers/trijoint.py.151: total_norm: 26.199692 [S 2022-10-27 12:44:16] ...trap/engines/engine.py.129: train_epoch.epoch: 67 [S 2022-10-27 12:44:16] ...trap/engines/engine.py.129: train_epoch.loss: 0.14863422417976851 [I 2022-10-27 12:44:16] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 12:44:17] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 67 [I 2022-10-27 12:44:17] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 12:44:18] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 12:44:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 12:44:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 12:44:21] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 12:44:21] ...trap/engines/engine.py.321: eval: epoch 67 | batch 0/510 [I 2022-10-27 12:44:21] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:38:27 [I 2022-10-27 12:44:21] ...trap/engines/engine.py.326: process: 4.52439 | load: 2.90347 [I 2022-10-27 12:44:24] ...trap/engines/engine.py.321: eval: epoch 67 | batch 10/510 [I 2022-10-27 12:44:24] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:33 [I 2022-10-27 12:44:24] ...trap/engines/engine.py.326: process: 0.34218 | load: 0.00016 [I 2022-10-27 12:44:28] ...trap/engines/engine.py.321: eval: epoch 67 | batch 20/510 [I 2022-10-27 12:44:28] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:06 [I 2022-10-27 12:44:28] ...trap/engines/engine.py.326: process: 0.32994 | load: 0.00022 [I 2022-10-27 12:44:31] ...trap/engines/engine.py.321: eval: epoch 67 | batch 30/510 [I 2022-10-27 12:44:31] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:42 [I 2022-10-27 12:44:31] ...trap/engines/engine.py.326: process: 0.33334 | load: 0.00017 [I 2022-10-27 12:44:34] ...trap/engines/engine.py.321: eval: epoch 67 | batch 40/510 [I 2022-10-27 12:44:34] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:36 [I 2022-10-27 12:44:34] ...trap/engines/engine.py.326: process: 0.32577 | load: 0.00019 [I 2022-10-27 12:44:38] ...trap/engines/engine.py.321: eval: epoch 67 | batch 50/510 [I 2022-10-27 12:44:38] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:32 [I 2022-10-27 12:44:38] ...trap/engines/engine.py.326: process: 0.32840 | load: 0.00016 [I 2022-10-27 12:44:41] ...trap/engines/engine.py.321: eval: epoch 67 | batch 60/510 [I 2022-10-27 12:44:41] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:28 [I 2022-10-27 12:44:41] ...trap/engines/engine.py.326: process: 0.32266 | load: 0.00017 [I 2022-10-27 12:44:44] ...trap/engines/engine.py.321: eval: epoch 67 | batch 70/510 [I 2022-10-27 12:44:44] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:23 [I 2022-10-27 12:44:44] ...trap/engines/engine.py.326: process: 0.32277 | load: 0.00015 [I 2022-10-27 12:44:48] ...trap/engines/engine.py.321: eval: epoch 67 | batch 80/510 [I 2022-10-27 12:44:48] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:24 [I 2022-10-27 12:44:48] ...trap/engines/engine.py.326: process: 0.35706 | load: 0.00018 [I 2022-10-27 12:44:51] ...trap/engines/engine.py.321: eval: epoch 67 | batch 90/510 [I 2022-10-27 12:44:51] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:23 [I 2022-10-27 12:44:51] ...trap/engines/engine.py.326: process: 0.35110 | load: 0.00017 [I 2022-10-27 12:44:54] ...trap/engines/engine.py.321: eval: epoch 67 | batch 100/510 [I 2022-10-27 12:44:54] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:14 [I 2022-10-27 12:44:54] ...trap/engines/engine.py.326: process: 0.29809 | load: 0.00018 [I 2022-10-27 12:44:57] ...trap/engines/engine.py.321: eval: epoch 67 | batch 110/510 [I 2022-10-27 12:44:57] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:57 [I 2022-10-27 12:44:57] ...trap/engines/engine.py.326: process: 0.28085 | load: 0.00014 [I 2022-10-27 12:45:00] ...trap/engines/engine.py.321: eval: epoch 67 | batch 120/510 [I 2022-10-27 12:45:00] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:53 [I 2022-10-27 12:45:00] ...trap/engines/engine.py.326: process: 0.28351 | load: 0.00019 [I 2022-10-27 12:45:03] ...trap/engines/engine.py.321: eval: epoch 67 | batch 130/510 [I 2022-10-27 12:45:03] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:50 [I 2022-10-27 12:45:03] ...trap/engines/engine.py.326: process: 0.29038 | load: 0.00021 [I 2022-10-27 12:45:06] ...trap/engines/engine.py.321: eval: epoch 67 | batch 140/510 [I 2022-10-27 12:45:06] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:46 [I 2022-10-27 12:45:06] ...trap/engines/engine.py.326: process: 0.27938 | load: 0.00016 [I 2022-10-27 12:45:09] ...trap/engines/engine.py.321: eval: epoch 67 | batch 150/510 [I 2022-10-27 12:45:09] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:43 [I 2022-10-27 12:45:09] ...trap/engines/engine.py.326: process: 0.28300 | load: 0.00015 [I 2022-10-27 12:45:12] ...trap/engines/engine.py.321: eval: epoch 67 | batch 160/510 [I 2022-10-27 12:45:12] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 12:45:12] ...trap/engines/engine.py.326: process: 0.28326 | load: 0.00030 [I 2022-10-27 12:45:14] ...trap/engines/engine.py.321: eval: epoch 67 | batch 170/510 [I 2022-10-27 12:45:14] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:37 [I 2022-10-27 12:45:14] ...trap/engines/engine.py.326: process: 0.29270 | load: 0.00024 [I 2022-10-27 12:45:17] ...trap/engines/engine.py.321: eval: epoch 67 | batch 180/510 [I 2022-10-27 12:45:17] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:35 [I 2022-10-27 12:45:17] ...trap/engines/engine.py.326: process: 0.28144 | load: 0.00020 [I 2022-10-27 12:45:20] ...trap/engines/engine.py.321: eval: epoch 67 | batch 190/510 [I 2022-10-27 12:45:20] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:31 [I 2022-10-27 12:45:20] ...trap/engines/engine.py.326: process: 0.28491 | load: 0.00017 [I 2022-10-27 12:45:23] ...trap/engines/engine.py.321: eval: epoch 67 | batch 200/510 [I 2022-10-27 12:45:23] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:43 [I 2022-10-27 12:45:23] ...trap/engines/engine.py.326: process: 0.28726 | load: 0.00017 [I 2022-10-27 12:45:27] ...trap/engines/engine.py.321: eval: epoch 67 | batch 210/510 [I 2022-10-27 12:45:27] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:50 [I 2022-10-27 12:45:27] ...trap/engines/engine.py.326: process: 0.42648 | load: 0.00021 [I 2022-10-27 12:45:30] ...trap/engines/engine.py.321: eval: epoch 67 | batch 220/510 [I 2022-10-27 12:45:30] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:27 [I 2022-10-27 12:45:30] ...trap/engines/engine.py.326: process: 0.28512 | load: 0.00016 [I 2022-10-27 12:45:33] ...trap/engines/engine.py.321: eval: epoch 67 | batch 230/510 [I 2022-10-27 12:45:33] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 12:45:33] ...trap/engines/engine.py.326: process: 0.28259 | load: 0.00022 [I 2022-10-27 12:45:36] ...trap/engines/engine.py.321: eval: epoch 67 | batch 240/510 [I 2022-10-27 12:45:36] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-27 12:45:36] ...trap/engines/engine.py.326: process: 0.28339 | load: 0.00024 [I 2022-10-27 12:45:39] ...trap/engines/engine.py.321: eval: epoch 67 | batch 250/510 [I 2022-10-27 12:45:39] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-27 12:45:39] ...trap/engines/engine.py.326: process: 0.29035 | load: 0.00025 [I 2022-10-27 12:45:42] ...trap/engines/engine.py.321: eval: epoch 67 | batch 260/510 [I 2022-10-27 12:45:42] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:11 [I 2022-10-27 12:45:42] ...trap/engines/engine.py.326: process: 0.28434 | load: 0.00016 [I 2022-10-27 12:45:45] ...trap/engines/engine.py.321: eval: epoch 67 | batch 270/510 [I 2022-10-27 12:45:45] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-27 12:45:45] ...trap/engines/engine.py.326: process: 0.28893 | load: 0.00018 [I 2022-10-27 12:45:47] ...trap/engines/engine.py.321: eval: epoch 67 | batch 280/510 [I 2022-10-27 12:45:47] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 12:45:47] ...trap/engines/engine.py.326: process: 0.28689 | load: 0.00016 [I 2022-10-27 12:45:50] ...trap/engines/engine.py.321: eval: epoch 67 | batch 290/510 [I 2022-10-27 12:45:50] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:04 [I 2022-10-27 12:45:50] ...trap/engines/engine.py.326: process: 0.29356 | load: 0.00024 [I 2022-10-27 12:45:53] ...trap/engines/engine.py.321: eval: epoch 67 | batch 300/510 [I 2022-10-27 12:45:53] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 12:45:53] ...trap/engines/engine.py.326: process: 0.28816 | load: 0.00021 [I 2022-10-27 12:45:56] ...trap/engines/engine.py.321: eval: epoch 67 | batch 310/510 [I 2022-10-27 12:45:56] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:58 [I 2022-10-27 12:45:56] ...trap/engines/engine.py.326: process: 0.30585 | load: 0.00022 [I 2022-10-27 12:45:59] ...trap/engines/engine.py.321: eval: epoch 67 | batch 320/510 [I 2022-10-27 12:45:59] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 12:45:59] ...trap/engines/engine.py.326: process: 0.28475 | load: 0.00017 [I 2022-10-27 12:46:02] ...trap/engines/engine.py.321: eval: epoch 67 | batch 330/510 [I 2022-10-27 12:46:02] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-27 12:46:02] ...trap/engines/engine.py.326: process: 0.28747 | load: 0.00016 [I 2022-10-27 12:46:05] ...trap/engines/engine.py.321: eval: epoch 67 | batch 340/510 [I 2022-10-27 12:46:05] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-27 12:46:05] ...trap/engines/engine.py.326: process: 0.28847 | load: 0.00016 [I 2022-10-27 12:46:08] ...trap/engines/engine.py.321: eval: epoch 67 | batch 350/510 [I 2022-10-27 12:46:08] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 12:46:08] ...trap/engines/engine.py.326: process: 0.28918 | load: 0.00016 [I 2022-10-27 12:46:11] ...trap/engines/engine.py.321: eval: epoch 67 | batch 360/510 [I 2022-10-27 12:46:11] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:44 [I 2022-10-27 12:46:11] ...trap/engines/engine.py.326: process: 0.29596 | load: 0.00017 [I 2022-10-27 12:46:14] ...trap/engines/engine.py.321: eval: epoch 67 | batch 370/510 [I 2022-10-27 12:46:14] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-27 12:46:14] ...trap/engines/engine.py.326: process: 0.28749 | load: 0.00015 [I 2022-10-27 12:46:17] ...trap/engines/engine.py.321: eval: epoch 67 | batch 380/510 [I 2022-10-27 12:46:17] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-27 12:46:17] ...trap/engines/engine.py.326: process: 0.28973 | load: 0.00020 [I 2022-10-27 12:46:19] ...trap/engines/engine.py.321: eval: epoch 67 | batch 390/510 [I 2022-10-27 12:46:19] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 12:46:19] ...trap/engines/engine.py.326: process: 0.28785 | load: 0.00018 [I 2022-10-27 12:46:22] ...trap/engines/engine.py.321: eval: epoch 67 | batch 400/510 [I 2022-10-27 12:46:22] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:32 [I 2022-10-27 12:46:22] ...trap/engines/engine.py.326: process: 0.29163 | load: 0.00020 [I 2022-10-27 12:46:25] ...trap/engines/engine.py.321: eval: epoch 67 | batch 410/510 [I 2022-10-27 12:46:25] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 12:46:25] ...trap/engines/engine.py.326: process: 0.28666 | load: 0.00021 [I 2022-10-27 12:46:28] ...trap/engines/engine.py.321: eval: epoch 67 | batch 420/510 [I 2022-10-27 12:46:28] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-27 12:46:28] ...trap/engines/engine.py.326: process: 0.29474 | load: 0.00019 [I 2022-10-27 12:46:31] ...trap/engines/engine.py.321: eval: epoch 67 | batch 430/510 [I 2022-10-27 12:46:31] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:22 [I 2022-10-27 12:46:31] ...trap/engines/engine.py.326: process: 0.27720 | load: 0.00020 [I 2022-10-27 12:46:34] ...trap/engines/engine.py.321: eval: epoch 67 | batch 440/510 [I 2022-10-27 12:46:34] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 12:46:34] ...trap/engines/engine.py.326: process: 0.28695 | load: 0.00027 [I 2022-10-27 12:46:37] ...trap/engines/engine.py.321: eval: epoch 67 | batch 450/510 [I 2022-10-27 12:46:37] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 12:46:37] ...trap/engines/engine.py.326: process: 0.28894 | load: 0.00017 [I 2022-10-27 12:46:40] ...trap/engines/engine.py.321: eval: epoch 67 | batch 460/510 [I 2022-10-27 12:46:40] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-27 12:46:40] ...trap/engines/engine.py.326: process: 0.28918 | load: 0.00016 [I 2022-10-27 12:46:43] ...trap/engines/engine.py.321: eval: epoch 67 | batch 470/510 [I 2022-10-27 12:46:43] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-27 12:46:43] ...trap/engines/engine.py.326: process: 0.29332 | load: 0.00017 [I 2022-10-27 12:46:46] ...trap/engines/engine.py.321: eval: epoch 67 | batch 480/510 [I 2022-10-27 12:46:46] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 12:46:46] ...trap/engines/engine.py.326: process: 0.28765 | load: 0.00016 [I 2022-10-27 12:46:48] ...trap/engines/engine.py.321: eval: epoch 67 | batch 490/510 [I 2022-10-27 12:46:48] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 12:46:48] ...trap/engines/engine.py.326: process: 0.28718 | load: 0.00016 [I 2022-10-27 12:46:51] ...trap/engines/engine.py.321: eval: epoch 67 | batch 500/510 [I 2022-10-27 12:46:51] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 12:46:51] ...trap/engines/engine.py.326: process: 0.28655 | load: 0.00022 [I 2022-10-27 12:46:54] ...trap/engines/engine.py.321: eval: epoch 67 | batch 510/510 [I 2022-10-27 12:46:54] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 12:46:54] ...trap/engines/engine.py.326: process: 0.28644 | load: 0.00012 [S 2022-10-27 12:46:55] ...trap/engines/engine.py.132: eval_epoch.epoch: 67 [I 2022-10-27 12:46:55] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.715100 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905900 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.938000 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.705500 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.900100 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933500 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014335 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010406 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009110 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016476 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010597 [S 2022-10-27 12:46:59] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008164 [I 2022-10-27 12:47:01] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 12:47:01] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 12:47:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 12:47:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 12:47:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 12:47:04] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 12:47:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 12:47:12] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 12:47:12] ...trap/engines/engine.py.163: Training model on trainset for epoch 68 [I 2022-10-27 12:47:12] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 12:47:14] ...trap/engines/engine.py.224: train: epoch 68 | batch 0/2304 [I 2022-10-27 12:47:14] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:15 [I 2022-10-27 12:47:14] ...trap/engines/engine.py.229: process: 2.68915 | load: 2.07319 [I 2022-10-27 12:47:14] ...trap/engines/engine.py.230: loss: 0.17604 [I 2022-10-27 12:47:14] ...optimizers/trijoint.py.151: total_norm: 11.910071 [I 2022-10-27 12:47:20] ...trap/engines/engine.py.224: train: epoch 68 | batch 10/2304 [I 2022-10-27 12:47:20] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:06 [I 2022-10-27 12:47:20] ...trap/engines/engine.py.229: process: 0.56324 | load: 0.00375 [I 2022-10-27 12:47:20] ...trap/engines/engine.py.230: loss: 0.17379 [I 2022-10-27 12:47:20] ...optimizers/trijoint.py.151: total_norm: 32.627975 [I 2022-10-27 12:47:26] ...trap/engines/engine.py.224: train: epoch 68 | batch 20/2304 [I 2022-10-27 12:47:26] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:37 [I 2022-10-27 12:47:26] ...trap/engines/engine.py.229: process: 0.56738 | load: 0.00422 [I 2022-10-27 12:47:26] ...trap/engines/engine.py.230: loss: 0.24503 [I 2022-10-27 12:47:26] ...optimizers/trijoint.py.151: total_norm: 20.051073 [I 2022-10-27 12:47:31] ...trap/engines/engine.py.224: train: epoch 68 | batch 30/2304 [I 2022-10-27 12:47:31] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:18 [I 2022-10-27 12:47:31] ...trap/engines/engine.py.229: process: 0.56849 | load: 0.00306 [I 2022-10-27 12:47:31] ...trap/engines/engine.py.230: loss: 0.15195 [I 2022-10-27 12:47:31] ...optimizers/trijoint.py.151: total_norm: 14.746260 [I 2022-10-27 12:47:37] ...trap/engines/engine.py.224: train: epoch 68 | batch 40/2304 [I 2022-10-27 12:47:37] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:11 [I 2022-10-27 12:47:37] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00311 [I 2022-10-27 12:47:37] ...trap/engines/engine.py.230: loss: 0.11209 [I 2022-10-27 12:47:37] ...optimizers/trijoint.py.151: total_norm: 11.272738 [I 2022-10-27 12:47:42] ...trap/engines/engine.py.224: train: epoch 68 | batch 50/2304 [I 2022-10-27 12:47:42] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:59 [I 2022-10-27 12:47:42] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00301 [I 2022-10-27 12:47:42] ...trap/engines/engine.py.230: loss: 0.17056 [I 2022-10-27 12:47:42] ...optimizers/trijoint.py.151: total_norm: 16.115789 [I 2022-10-27 12:47:48] ...trap/engines/engine.py.224: train: epoch 68 | batch 60/2304 [I 2022-10-27 12:47:48] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:53 [I 2022-10-27 12:47:48] ...trap/engines/engine.py.229: process: 0.57136 | load: 0.00281 [I 2022-10-27 12:47:48] ...trap/engines/engine.py.230: loss: 0.15394 [I 2022-10-27 12:47:48] ...optimizers/trijoint.py.151: total_norm: 20.003210 [I 2022-10-27 12:47:54] ...trap/engines/engine.py.224: train: epoch 68 | batch 70/2304 [I 2022-10-27 12:47:54] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:37 [I 2022-10-27 12:47:54] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00257 [I 2022-10-27 12:47:54] ...trap/engines/engine.py.230: loss: 0.15785 [I 2022-10-27 12:47:54] ...optimizers/trijoint.py.151: total_norm: 12.135399 [I 2022-10-27 12:47:59] ...trap/engines/engine.py.224: train: epoch 68 | batch 80/2304 [I 2022-10-27 12:47:59] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:39 [I 2022-10-27 12:47:59] ...trap/engines/engine.py.229: process: 0.57342 | load: 0.00302 [I 2022-10-27 12:47:59] ...trap/engines/engine.py.230: loss: 0.15940 [I 2022-10-27 12:47:59] ...optimizers/trijoint.py.151: total_norm: 16.246126 [I 2022-10-27 12:48:05] ...trap/engines/engine.py.224: train: epoch 68 | batch 90/2304 [I 2022-10-27 12:48:05] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:36 [I 2022-10-27 12:48:05] ...trap/engines/engine.py.229: process: 0.56185 | load: 0.00284 [I 2022-10-27 12:48:05] ...trap/engines/engine.py.230: loss: 0.12725 [I 2022-10-27 12:48:05] ...optimizers/trijoint.py.151: total_norm: 25.062147 [I 2022-10-27 12:48:10] ...trap/engines/engine.py.224: train: epoch 68 | batch 100/2304 [I 2022-10-27 12:48:10] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:21:19 [I 2022-10-27 12:48:10] ...trap/engines/engine.py.229: process: 0.56986 | load: 0.00256 [I 2022-10-27 12:48:10] ...trap/engines/engine.py.230: loss: 0.10493 [I 2022-10-27 12:48:10] ...optimizers/trijoint.py.151: total_norm: 15.900105 [I 2022-10-27 12:48:16] ...trap/engines/engine.py.224: train: epoch 68 | batch 110/2304 [I 2022-10-27 12:48:16] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:33 [I 2022-10-27 12:48:16] ...trap/engines/engine.py.229: process: 0.56070 | load: 0.00272 [I 2022-10-27 12:48:16] ...trap/engines/engine.py.230: loss: 0.21805 [I 2022-10-27 12:48:16] ...optimizers/trijoint.py.151: total_norm: 18.055492 [I 2022-10-27 12:48:22] ...trap/engines/engine.py.224: train: epoch 68 | batch 120/2304 [I 2022-10-27 12:48:22] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:12 [I 2022-10-27 12:48:22] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00279 [I 2022-10-27 12:48:22] ...trap/engines/engine.py.230: loss: 0.19347 [I 2022-10-27 12:48:22] ...optimizers/trijoint.py.151: total_norm: 12.602893 [I 2022-10-27 12:48:27] ...trap/engines/engine.py.224: train: epoch 68 | batch 130/2304 [I 2022-10-27 12:48:27] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:10 [I 2022-10-27 12:48:27] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00292 [I 2022-10-27 12:48:27] ...trap/engines/engine.py.230: loss: 0.17140 [I 2022-10-27 12:48:27] ...optimizers/trijoint.py.151: total_norm: 16.002289 [I 2022-10-27 12:48:33] ...trap/engines/engine.py.224: train: epoch 68 | batch 140/2304 [I 2022-10-27 12:48:33] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:19:55 [I 2022-10-27 12:48:33] ...trap/engines/engine.py.229: process: 0.55005 | load: 0.00281 [I 2022-10-27 12:48:33] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-27 12:48:33] ...optimizers/trijoint.py.151: total_norm: 18.118244 [I 2022-10-27 12:48:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 150/2304 [I 2022-10-27 12:48:38] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:52 [I 2022-10-27 12:48:38] ...trap/engines/engine.py.229: process: 0.55096 | load: 0.00256 [I 2022-10-27 12:48:38] ...trap/engines/engine.py.230: loss: 0.10609 [I 2022-10-27 12:48:38] ...optimizers/trijoint.py.151: total_norm: 10.233282 [I 2022-10-27 12:48:44] ...trap/engines/engine.py.224: train: epoch 68 | batch 160/2304 [I 2022-10-27 12:48:44] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:47 [I 2022-10-27 12:48:44] ...trap/engines/engine.py.229: process: 0.54675 | load: 0.00267 [I 2022-10-27 12:48:44] ...trap/engines/engine.py.230: loss: 0.13479 [I 2022-10-27 12:48:44] ...optimizers/trijoint.py.151: total_norm: 13.309690 [I 2022-10-27 12:48:50] ...trap/engines/engine.py.224: train: epoch 68 | batch 170/2304 [I 2022-10-27 12:48:50] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:52 [I 2022-10-27 12:48:50] ...trap/engines/engine.py.229: process: 0.56166 | load: 0.00285 [I 2022-10-27 12:48:50] ...trap/engines/engine.py.230: loss: 0.11997 [I 2022-10-27 12:48:50] ...optimizers/trijoint.py.151: total_norm: 11.010987 [I 2022-10-27 12:48:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 180/2304 [I 2022-10-27 12:48:55] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:34 [I 2022-10-27 12:48:55] ...trap/engines/engine.py.229: process: 0.55006 | load: 0.00278 [I 2022-10-27 12:48:55] ...trap/engines/engine.py.230: loss: 0.11133 [I 2022-10-27 12:48:55] ...optimizers/trijoint.py.151: total_norm: 10.980235 [I 2022-10-27 12:49:01] ...trap/engines/engine.py.224: train: epoch 68 | batch 190/2304 [I 2022-10-27 12:49:01] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:32 [I 2022-10-27 12:49:01] ...trap/engines/engine.py.229: process: 0.55938 | load: 0.00260 [I 2022-10-27 12:49:01] ...trap/engines/engine.py.230: loss: 0.13104 [I 2022-10-27 12:49:01] ...optimizers/trijoint.py.151: total_norm: 19.127552 [I 2022-10-27 12:49:06] ...trap/engines/engine.py.224: train: epoch 68 | batch 200/2304 [I 2022-10-27 12:49:06] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:34 [I 2022-10-27 12:49:06] ...trap/engines/engine.py.229: process: 0.55807 | load: 0.00257 [I 2022-10-27 12:49:06] ...trap/engines/engine.py.230: loss: 0.12667 [I 2022-10-27 12:49:06] ...optimizers/trijoint.py.151: total_norm: 13.379889 [I 2022-10-27 12:49:12] ...trap/engines/engine.py.224: train: epoch 68 | batch 210/2304 [I 2022-10-27 12:49:12] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:22 [I 2022-10-27 12:49:12] ...trap/engines/engine.py.229: process: 0.55518 | load: 0.00305 [I 2022-10-27 12:49:12] ...trap/engines/engine.py.230: loss: 0.13080 [I 2022-10-27 12:49:12] ...optimizers/trijoint.py.151: total_norm: 11.410982 [I 2022-10-27 12:49:17] ...trap/engines/engine.py.224: train: epoch 68 | batch 220/2304 [I 2022-10-27 12:49:17] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:16 [I 2022-10-27 12:49:17] ...trap/engines/engine.py.229: process: 0.54647 | load: 0.00264 [I 2022-10-27 12:49:17] ...trap/engines/engine.py.230: loss: 0.14871 [I 2022-10-27 12:49:17] ...optimizers/trijoint.py.151: total_norm: 18.543657 [I 2022-10-27 12:49:23] ...trap/engines/engine.py.224: train: epoch 68 | batch 230/2304 [I 2022-10-27 12:49:23] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:44 [I 2022-10-27 12:49:23] ...trap/engines/engine.py.229: process: 0.56998 | load: 0.00252 [I 2022-10-27 12:49:23] ...trap/engines/engine.py.230: loss: 0.12051 [I 2022-10-27 12:49:23] ...optimizers/trijoint.py.151: total_norm: 11.420998 [I 2022-10-27 12:49:29] ...trap/engines/engine.py.224: train: epoch 68 | batch 240/2304 [I 2022-10-27 12:49:29] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:13 [I 2022-10-27 12:49:29] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00260 [I 2022-10-27 12:49:29] ...trap/engines/engine.py.230: loss: 0.12370 [I 2022-10-27 12:49:29] ...optimizers/trijoint.py.151: total_norm: 13.865946 [I 2022-10-27 12:49:34] ...trap/engines/engine.py.224: train: epoch 68 | batch 250/2304 [I 2022-10-27 12:49:34] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:06 [I 2022-10-27 12:49:34] ...trap/engines/engine.py.229: process: 0.54426 | load: 0.00244 [I 2022-10-27 12:49:34] ...trap/engines/engine.py.230: loss: 0.11113 [I 2022-10-27 12:49:34] ...optimizers/trijoint.py.151: total_norm: 11.159945 [I 2022-10-27 12:49:40] ...trap/engines/engine.py.224: train: epoch 68 | batch 260/2304 [I 2022-10-27 12:49:40] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:56 [I 2022-10-27 12:49:40] ...trap/engines/engine.py.229: process: 0.55348 | load: 0.00244 [I 2022-10-27 12:49:40] ...trap/engines/engine.py.230: loss: 0.15903 [I 2022-10-27 12:49:40] ...optimizers/trijoint.py.151: total_norm: 17.360512 [I 2022-10-27 12:49:46] ...trap/engines/engine.py.224: train: epoch 68 | batch 270/2304 [I 2022-10-27 12:49:46] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:55 [I 2022-10-27 12:49:46] ...trap/engines/engine.py.229: process: 0.55905 | load: 0.00264 [I 2022-10-27 12:49:46] ...trap/engines/engine.py.230: loss: 0.11391 [I 2022-10-27 12:49:46] ...optimizers/trijoint.py.151: total_norm: 7.840021 [I 2022-10-27 12:49:51] ...trap/engines/engine.py.224: train: epoch 68 | batch 280/2304 [I 2022-10-27 12:49:51] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:46 [I 2022-10-27 12:49:51] ...trap/engines/engine.py.229: process: 0.56249 | load: 0.00250 [I 2022-10-27 12:49:51] ...trap/engines/engine.py.230: loss: 0.17247 [I 2022-10-27 12:49:51] ...optimizers/trijoint.py.151: total_norm: 43.143658 [I 2022-10-27 12:49:57] ...trap/engines/engine.py.224: train: epoch 68 | batch 290/2304 [I 2022-10-27 12:49:57] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:19:14 [I 2022-10-27 12:49:57] ...trap/engines/engine.py.229: process: 0.56416 | load: 0.00252 [I 2022-10-27 12:49:57] ...trap/engines/engine.py.230: loss: 0.10379 [I 2022-10-27 12:49:57] ...optimizers/trijoint.py.151: total_norm: 8.415229 [I 2022-10-27 12:50:02] ...trap/engines/engine.py.224: train: epoch 68 | batch 300/2304 [I 2022-10-27 12:50:02] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:31 [I 2022-10-27 12:50:02] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00247 [I 2022-10-27 12:50:02] ...trap/engines/engine.py.230: loss: 0.20325 [I 2022-10-27 12:50:02] ...optimizers/trijoint.py.151: total_norm: 23.262833 [I 2022-10-27 12:50:08] ...trap/engines/engine.py.224: train: epoch 68 | batch 310/2304 [I 2022-10-27 12:50:08] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:38 [I 2022-10-27 12:50:08] ...trap/engines/engine.py.229: process: 0.56184 | load: 0.00253 [I 2022-10-27 12:50:08] ...trap/engines/engine.py.230: loss: 0.10160 [I 2022-10-27 12:50:08] ...optimizers/trijoint.py.151: total_norm: 11.413617 [I 2022-10-27 12:50:14] ...trap/engines/engine.py.224: train: epoch 68 | batch 320/2304 [I 2022-10-27 12:50:14] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:25 [I 2022-10-27 12:50:14] ...trap/engines/engine.py.229: process: 0.55057 | load: 0.00255 [I 2022-10-27 12:50:14] ...trap/engines/engine.py.230: loss: 0.15387 [I 2022-10-27 12:50:14] ...optimizers/trijoint.py.151: total_norm: 14.768598 [I 2022-10-27 12:50:19] ...trap/engines/engine.py.224: train: epoch 68 | batch 330/2304 [I 2022-10-27 12:50:19] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:28 [I 2022-10-27 12:50:19] ...trap/engines/engine.py.229: process: 0.56192 | load: 0.00253 [I 2022-10-27 12:50:19] ...trap/engines/engine.py.230: loss: 0.14030 [I 2022-10-27 12:50:19] ...optimizers/trijoint.py.151: total_norm: 21.215981 [I 2022-10-27 12:50:25] ...trap/engines/engine.py.224: train: epoch 68 | batch 340/2304 [I 2022-10-27 12:50:25] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:20 [I 2022-10-27 12:50:25] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00249 [I 2022-10-27 12:50:25] ...trap/engines/engine.py.230: loss: 0.12022 [I 2022-10-27 12:50:25] ...optimizers/trijoint.py.151: total_norm: 9.239243 [I 2022-10-27 12:50:30] ...trap/engines/engine.py.224: train: epoch 68 | batch 350/2304 [I 2022-10-27 12:50:30] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:19 [I 2022-10-27 12:50:30] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00266 [I 2022-10-27 12:50:30] ...trap/engines/engine.py.230: loss: 0.15645 [I 2022-10-27 12:50:30] ...optimizers/trijoint.py.151: total_norm: 13.180880 [I 2022-10-27 12:50:36] ...trap/engines/engine.py.224: train: epoch 68 | batch 360/2304 [I 2022-10-27 12:50:36] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:01 [I 2022-10-27 12:50:36] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00291 [I 2022-10-27 12:50:36] ...trap/engines/engine.py.230: loss: 0.09131 [I 2022-10-27 12:50:36] ...optimizers/trijoint.py.151: total_norm: 27.390371 [I 2022-10-27 12:50:42] ...trap/engines/engine.py.224: train: epoch 68 | batch 370/2304 [I 2022-10-27 12:50:42] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:51 [I 2022-10-27 12:50:42] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00292 [I 2022-10-27 12:50:42] ...trap/engines/engine.py.230: loss: 0.12986 [I 2022-10-27 12:50:42] ...optimizers/trijoint.py.151: total_norm: 10.168365 [I 2022-10-27 12:50:47] ...trap/engines/engine.py.224: train: epoch 68 | batch 380/2304 [I 2022-10-27 12:50:47] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:59 [I 2022-10-27 12:50:47] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00274 [I 2022-10-27 12:50:47] ...trap/engines/engine.py.230: loss: 0.12483 [I 2022-10-27 12:50:47] ...optimizers/trijoint.py.151: total_norm: 15.256814 [I 2022-10-27 12:50:53] ...trap/engines/engine.py.224: train: epoch 68 | batch 390/2304 [I 2022-10-27 12:50:53] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:48 [I 2022-10-27 12:50:53] ...trap/engines/engine.py.229: process: 0.56505 | load: 0.00256 [I 2022-10-27 12:50:53] ...trap/engines/engine.py.230: loss: 0.13319 [I 2022-10-27 12:50:53] ...optimizers/trijoint.py.151: total_norm: 26.566542 [I 2022-10-27 12:50:58] ...trap/engines/engine.py.224: train: epoch 68 | batch 400/2304 [I 2022-10-27 12:50:58] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:46 [I 2022-10-27 12:50:58] ...trap/engines/engine.py.229: process: 0.56562 | load: 0.00249 [I 2022-10-27 12:50:58] ...trap/engines/engine.py.230: loss: 0.10907 [I 2022-10-27 12:50:58] ...optimizers/trijoint.py.151: total_norm: 16.693489 [I 2022-10-27 12:51:04] ...trap/engines/engine.py.224: train: epoch 68 | batch 410/2304 [I 2022-10-27 12:51:04] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:45 [I 2022-10-27 12:51:04] ...trap/engines/engine.py.229: process: 0.56959 | load: 0.00275 [I 2022-10-27 12:51:04] ...trap/engines/engine.py.230: loss: 0.10495 [I 2022-10-27 12:51:04] ...optimizers/trijoint.py.151: total_norm: 7.662886 [I 2022-10-27 12:51:10] ...trap/engines/engine.py.224: train: epoch 68 | batch 420/2304 [I 2022-10-27 12:51:10] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:47 [I 2022-10-27 12:51:10] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00279 [I 2022-10-27 12:51:10] ...trap/engines/engine.py.230: loss: 0.14452 [I 2022-10-27 12:51:10] ...optimizers/trijoint.py.151: total_norm: 18.255856 [I 2022-10-27 12:51:15] ...trap/engines/engine.py.224: train: epoch 68 | batch 430/2304 [I 2022-10-27 12:51:15] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:20 [I 2022-10-27 12:51:15] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00253 [I 2022-10-27 12:51:15] ...trap/engines/engine.py.230: loss: 0.13439 [I 2022-10-27 12:51:15] ...optimizers/trijoint.py.151: total_norm: 12.287069 [I 2022-10-27 12:51:21] ...trap/engines/engine.py.224: train: epoch 68 | batch 440/2304 [I 2022-10-27 12:51:21] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:13 [I 2022-10-27 12:51:21] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00249 [I 2022-10-27 12:51:21] ...trap/engines/engine.py.230: loss: 0.16679 [I 2022-10-27 12:51:21] ...optimizers/trijoint.py.151: total_norm: 17.952198 [I 2022-10-27 12:51:26] ...trap/engines/engine.py.224: train: epoch 68 | batch 450/2304 [I 2022-10-27 12:51:26] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:01 [I 2022-10-27 12:51:26] ...trap/engines/engine.py.229: process: 0.54307 | load: 0.00252 [I 2022-10-27 12:51:26] ...trap/engines/engine.py.230: loss: 0.11815 [I 2022-10-27 12:51:26] ...optimizers/trijoint.py.151: total_norm: 19.470802 [I 2022-10-27 12:51:32] ...trap/engines/engine.py.224: train: epoch 68 | batch 460/2304 [I 2022-10-27 12:51:32] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:03 [I 2022-10-27 12:51:32] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00251 [I 2022-10-27 12:51:32] ...trap/engines/engine.py.230: loss: 0.20074 [I 2022-10-27 12:51:32] ...optimizers/trijoint.py.151: total_norm: 16.231918 [I 2022-10-27 12:51:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 470/2304 [I 2022-10-27 12:51:38] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:04 [I 2022-10-27 12:51:38] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00256 [I 2022-10-27 12:51:38] ...trap/engines/engine.py.230: loss: 0.16077 [I 2022-10-27 12:51:38] ...optimizers/trijoint.py.151: total_norm: 12.746541 [I 2022-10-27 12:51:43] ...trap/engines/engine.py.224: train: epoch 68 | batch 480/2304 [I 2022-10-27 12:51:43] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:00 [I 2022-10-27 12:51:43] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00260 [I 2022-10-27 12:51:43] ...trap/engines/engine.py.230: loss: 0.15014 [I 2022-10-27 12:51:43] ...optimizers/trijoint.py.151: total_norm: 14.356809 [I 2022-10-27 12:51:49] ...trap/engines/engine.py.224: train: epoch 68 | batch 490/2304 [I 2022-10-27 12:51:49] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:50 [I 2022-10-27 12:51:49] ...trap/engines/engine.py.229: process: 0.55434 | load: 0.00246 [I 2022-10-27 12:51:49] ...trap/engines/engine.py.230: loss: 0.11492 [I 2022-10-27 12:51:49] ...optimizers/trijoint.py.151: total_norm: 7.922594 [I 2022-10-27 12:51:54] ...trap/engines/engine.py.224: train: epoch 68 | batch 500/2304 [I 2022-10-27 12:51:54] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:41 [I 2022-10-27 12:51:54] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00243 [I 2022-10-27 12:51:54] ...trap/engines/engine.py.230: loss: 0.14451 [I 2022-10-27 12:51:54] ...optimizers/trijoint.py.151: total_norm: 19.711519 [I 2022-10-27 12:52:00] ...trap/engines/engine.py.224: train: epoch 68 | batch 510/2304 [I 2022-10-27 12:52:00] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:41 [I 2022-10-27 12:52:00] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00248 [I 2022-10-27 12:52:00] ...trap/engines/engine.py.230: loss: 0.14298 [I 2022-10-27 12:52:00] ...optimizers/trijoint.py.151: total_norm: 11.789116 [I 2022-10-27 12:52:05] ...trap/engines/engine.py.224: train: epoch 68 | batch 520/2304 [I 2022-10-27 12:52:05] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:31 [I 2022-10-27 12:52:05] ...trap/engines/engine.py.229: process: 0.54824 | load: 0.00246 [I 2022-10-27 12:52:05] ...trap/engines/engine.py.230: loss: 0.15141 [I 2022-10-27 12:52:05] ...optimizers/trijoint.py.151: total_norm: 12.720572 [I 2022-10-27 12:52:11] ...trap/engines/engine.py.224: train: epoch 68 | batch 530/2304 [I 2022-10-27 12:52:11] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:35 [I 2022-10-27 12:52:11] ...trap/engines/engine.py.229: process: 0.58909 | load: 0.00254 [I 2022-10-27 12:52:11] ...trap/engines/engine.py.230: loss: 0.13545 [I 2022-10-27 12:52:11] ...optimizers/trijoint.py.151: total_norm: 15.245340 [I 2022-10-27 12:52:17] ...trap/engines/engine.py.224: train: epoch 68 | batch 540/2304 [I 2022-10-27 12:52:17] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:28 [I 2022-10-27 12:52:17] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00248 [I 2022-10-27 12:52:17] ...trap/engines/engine.py.230: loss: 0.10749 [I 2022-10-27 12:52:17] ...optimizers/trijoint.py.151: total_norm: 11.955653 [I 2022-10-27 12:52:22] ...trap/engines/engine.py.224: train: epoch 68 | batch 550/2304 [I 2022-10-27 12:52:22] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:22 [I 2022-10-27 12:52:22] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00252 [I 2022-10-27 12:52:22] ...trap/engines/engine.py.230: loss: 0.14477 [I 2022-10-27 12:52:22] ...optimizers/trijoint.py.151: total_norm: 14.389714 [I 2022-10-27 12:52:28] ...trap/engines/engine.py.224: train: epoch 68 | batch 560/2304 [I 2022-10-27 12:52:28] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:04 [I 2022-10-27 12:52:28] ...trap/engines/engine.py.229: process: 0.55150 | load: 0.00252 [I 2022-10-27 12:52:28] ...trap/engines/engine.py.230: loss: 0.11400 [I 2022-10-27 12:52:28] ...optimizers/trijoint.py.151: total_norm: 15.178415 [I 2022-10-27 12:52:33] ...trap/engines/engine.py.224: train: epoch 68 | batch 570/2304 [I 2022-10-27 12:52:33] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:07 [I 2022-10-27 12:52:33] ...trap/engines/engine.py.229: process: 0.57405 | load: 0.00248 [I 2022-10-27 12:52:33] ...trap/engines/engine.py.230: loss: 0.15816 [I 2022-10-27 12:52:33] ...optimizers/trijoint.py.151: total_norm: 14.269834 [I 2022-10-27 12:52:39] ...trap/engines/engine.py.224: train: epoch 68 | batch 580/2304 [I 2022-10-27 12:52:39] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:58 [I 2022-10-27 12:52:39] ...trap/engines/engine.py.229: process: 0.55434 | load: 0.00245 [I 2022-10-27 12:52:39] ...trap/engines/engine.py.230: loss: 0.10345 [I 2022-10-27 12:52:39] ...optimizers/trijoint.py.151: total_norm: 13.435942 [I 2022-10-27 12:52:45] ...trap/engines/engine.py.224: train: epoch 68 | batch 590/2304 [I 2022-10-27 12:52:45] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:54 [I 2022-10-27 12:52:45] ...trap/engines/engine.py.229: process: 0.55023 | load: 0.00242 [I 2022-10-27 12:52:45] ...trap/engines/engine.py.230: loss: 0.14306 [I 2022-10-27 12:52:45] ...optimizers/trijoint.py.151: total_norm: 21.349501 [I 2022-10-27 12:52:50] ...trap/engines/engine.py.224: train: epoch 68 | batch 600/2304 [I 2022-10-27 12:52:50] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:42 [I 2022-10-27 12:52:50] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00247 [I 2022-10-27 12:52:50] ...trap/engines/engine.py.230: loss: 0.14541 [I 2022-10-27 12:52:50] ...optimizers/trijoint.py.151: total_norm: 17.949316 [I 2022-10-27 12:52:56] ...trap/engines/engine.py.224: train: epoch 68 | batch 610/2304 [I 2022-10-27 12:52:56] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:46 [I 2022-10-27 12:52:56] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00251 [I 2022-10-27 12:52:56] ...trap/engines/engine.py.230: loss: 0.12417 [I 2022-10-27 12:52:56] ...optimizers/trijoint.py.151: total_norm: 15.519924 [I 2022-10-27 12:53:01] ...trap/engines/engine.py.224: train: epoch 68 | batch 620/2304 [I 2022-10-27 12:53:01] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:37 [I 2022-10-27 12:53:01] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00246 [I 2022-10-27 12:53:01] ...trap/engines/engine.py.230: loss: 0.11949 [I 2022-10-27 12:53:01] ...optimizers/trijoint.py.151: total_norm: 10.374308 [I 2022-10-27 12:53:07] ...trap/engines/engine.py.224: train: epoch 68 | batch 630/2304 [I 2022-10-27 12:53:07] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:37 [I 2022-10-27 12:53:07] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00249 [I 2022-10-27 12:53:07] ...trap/engines/engine.py.230: loss: 0.11763 [I 2022-10-27 12:53:07] ...optimizers/trijoint.py.151: total_norm: 13.063548 [I 2022-10-27 12:53:12] ...trap/engines/engine.py.224: train: epoch 68 | batch 640/2304 [I 2022-10-27 12:53:12] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:23 [I 2022-10-27 12:53:12] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00247 [I 2022-10-27 12:53:12] ...trap/engines/engine.py.230: loss: 0.16284 [I 2022-10-27 12:53:12] ...optimizers/trijoint.py.151: total_norm: 13.179265 [I 2022-10-27 12:53:18] ...trap/engines/engine.py.224: train: epoch 68 | batch 650/2304 [I 2022-10-27 12:53:18] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:18 [I 2022-10-27 12:53:18] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00244 [I 2022-10-27 12:53:18] ...trap/engines/engine.py.230: loss: 0.12283 [I 2022-10-27 12:53:18] ...optimizers/trijoint.py.151: total_norm: 9.050734 [I 2022-10-27 12:53:24] ...trap/engines/engine.py.224: train: epoch 68 | batch 660/2304 [I 2022-10-27 12:53:24] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:20 [I 2022-10-27 12:53:24] ...trap/engines/engine.py.229: process: 0.56713 | load: 0.00242 [I 2022-10-27 12:53:24] ...trap/engines/engine.py.230: loss: 0.10076 [I 2022-10-27 12:53:24] ...optimizers/trijoint.py.151: total_norm: 12.444779 [I 2022-10-27 12:53:29] ...trap/engines/engine.py.224: train: epoch 68 | batch 670/2304 [I 2022-10-27 12:53:29] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:14 [I 2022-10-27 12:53:29] ...trap/engines/engine.py.229: process: 0.55391 | load: 0.00249 [I 2022-10-27 12:53:29] ...trap/engines/engine.py.230: loss: 0.11383 [I 2022-10-27 12:53:29] ...optimizers/trijoint.py.151: total_norm: 9.738821 [I 2022-10-27 12:53:35] ...trap/engines/engine.py.224: train: epoch 68 | batch 680/2304 [I 2022-10-27 12:53:35] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:01 [I 2022-10-27 12:53:35] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00252 [I 2022-10-27 12:53:35] ...trap/engines/engine.py.230: loss: 0.12021 [I 2022-10-27 12:53:35] ...optimizers/trijoint.py.151: total_norm: 17.052732 [I 2022-10-27 12:53:40] ...trap/engines/engine.py.224: train: epoch 68 | batch 690/2304 [I 2022-10-27 12:53:40] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:00 [I 2022-10-27 12:53:40] ...trap/engines/engine.py.229: process: 0.55221 | load: 0.00251 [I 2022-10-27 12:53:40] ...trap/engines/engine.py.230: loss: 0.11048 [I 2022-10-27 12:53:40] ...optimizers/trijoint.py.151: total_norm: 13.138182 [I 2022-10-27 12:53:46] ...trap/engines/engine.py.224: train: epoch 68 | batch 700/2304 [I 2022-10-27 12:53:46] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:51 [I 2022-10-27 12:53:46] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00255 [I 2022-10-27 12:53:46] ...trap/engines/engine.py.230: loss: 0.23786 [I 2022-10-27 12:53:46] ...optimizers/trijoint.py.151: total_norm: 16.371107 [I 2022-10-27 12:53:52] ...trap/engines/engine.py.224: train: epoch 68 | batch 710/2304 [I 2022-10-27 12:53:52] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:55 [I 2022-10-27 12:53:52] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00250 [I 2022-10-27 12:53:52] ...trap/engines/engine.py.230: loss: 0.12400 [I 2022-10-27 12:53:52] ...optimizers/trijoint.py.151: total_norm: 14.383340 [I 2022-10-27 12:53:57] ...trap/engines/engine.py.224: train: epoch 68 | batch 720/2304 [I 2022-10-27 12:53:57] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:48 [I 2022-10-27 12:53:57] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00248 [I 2022-10-27 12:53:57] ...trap/engines/engine.py.230: loss: 0.14461 [I 2022-10-27 12:53:57] ...optimizers/trijoint.py.151: total_norm: 11.105922 [I 2022-10-27 12:54:03] ...trap/engines/engine.py.224: train: epoch 68 | batch 730/2304 [I 2022-10-27 12:54:03] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:42 [I 2022-10-27 12:54:03] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00249 [I 2022-10-27 12:54:03] ...trap/engines/engine.py.230: loss: 0.21073 [I 2022-10-27 12:54:03] ...optimizers/trijoint.py.151: total_norm: 31.714417 [I 2022-10-27 12:54:08] ...trap/engines/engine.py.224: train: epoch 68 | batch 740/2304 [I 2022-10-27 12:54:08] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:33 [I 2022-10-27 12:54:08] ...trap/engines/engine.py.229: process: 0.56417 | load: 0.00247 [I 2022-10-27 12:54:08] ...trap/engines/engine.py.230: loss: 0.16474 [I 2022-10-27 12:54:08] ...optimizers/trijoint.py.151: total_norm: 8.962145 [I 2022-10-27 12:54:14] ...trap/engines/engine.py.224: train: epoch 68 | batch 750/2304 [I 2022-10-27 12:54:14] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:31 [I 2022-10-27 12:54:14] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00249 [I 2022-10-27 12:54:14] ...trap/engines/engine.py.230: loss: 0.17150 [I 2022-10-27 12:54:14] ...optimizers/trijoint.py.151: total_norm: 12.952168 [I 2022-10-27 12:54:20] ...trap/engines/engine.py.224: train: epoch 68 | batch 760/2304 [I 2022-10-27 12:54:20] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:18 [I 2022-10-27 12:54:20] ...trap/engines/engine.py.229: process: 0.54628 | load: 0.00244 [I 2022-10-27 12:54:20] ...trap/engines/engine.py.230: loss: 0.17261 [I 2022-10-27 12:54:20] ...optimizers/trijoint.py.151: total_norm: 21.243322 [I 2022-10-27 12:54:25] ...trap/engines/engine.py.224: train: epoch 68 | batch 770/2304 [I 2022-10-27 12:54:25] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:15 [I 2022-10-27 12:54:25] ...trap/engines/engine.py.229: process: 0.56403 | load: 0.00247 [I 2022-10-27 12:54:25] ...trap/engines/engine.py.230: loss: 0.14088 [I 2022-10-27 12:54:25] ...optimizers/trijoint.py.151: total_norm: 16.414782 [I 2022-10-27 12:54:31] ...trap/engines/engine.py.224: train: epoch 68 | batch 780/2304 [I 2022-10-27 12:54:31] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:06 [I 2022-10-27 12:54:31] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00243 [I 2022-10-27 12:54:31] ...trap/engines/engine.py.230: loss: 0.15519 [I 2022-10-27 12:54:31] ...optimizers/trijoint.py.151: total_norm: 11.548254 [I 2022-10-27 12:54:36] ...trap/engines/engine.py.224: train: epoch 68 | batch 790/2304 [I 2022-10-27 12:54:36] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:04 [I 2022-10-27 12:54:36] ...trap/engines/engine.py.229: process: 0.55309 | load: 0.00245 [I 2022-10-27 12:54:36] ...trap/engines/engine.py.230: loss: 0.17633 [I 2022-10-27 12:54:36] ...optimizers/trijoint.py.151: total_norm: 18.264635 [I 2022-10-27 12:54:42] ...trap/engines/engine.py.224: train: epoch 68 | batch 800/2304 [I 2022-10-27 12:54:42] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:53 [I 2022-10-27 12:54:42] ...trap/engines/engine.py.229: process: 0.54087 | load: 0.00249 [I 2022-10-27 12:54:42] ...trap/engines/engine.py.230: loss: 0.18032 [I 2022-10-27 12:54:42] ...optimizers/trijoint.py.151: total_norm: 15.324636 [I 2022-10-27 12:54:48] ...trap/engines/engine.py.224: train: epoch 68 | batch 810/2304 [I 2022-10-27 12:54:48] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:54 [I 2022-10-27 12:54:48] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00249 [I 2022-10-27 12:54:48] ...trap/engines/engine.py.230: loss: 0.14875 [I 2022-10-27 12:54:48] ...optimizers/trijoint.py.151: total_norm: 18.497120 [I 2022-10-27 12:54:53] ...trap/engines/engine.py.224: train: epoch 68 | batch 820/2304 [I 2022-10-27 12:54:53] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:45 [I 2022-10-27 12:54:53] ...trap/engines/engine.py.229: process: 0.55309 | load: 0.00248 [I 2022-10-27 12:54:53] ...trap/engines/engine.py.230: loss: 0.17454 [I 2022-10-27 12:54:53] ...optimizers/trijoint.py.151: total_norm: 21.472025 [I 2022-10-27 12:54:59] ...trap/engines/engine.py.224: train: epoch 68 | batch 830/2304 [I 2022-10-27 12:54:59] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:36 [I 2022-10-27 12:54:59] ...trap/engines/engine.py.229: process: 0.54371 | load: 0.00246 [I 2022-10-27 12:54:59] ...trap/engines/engine.py.230: loss: 0.12554 [I 2022-10-27 12:54:59] ...optimizers/trijoint.py.151: total_norm: 18.234467 [I 2022-10-27 12:55:04] ...trap/engines/engine.py.224: train: epoch 68 | batch 840/2304 [I 2022-10-27 12:55:04] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:40 [I 2022-10-27 12:55:04] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00257 [I 2022-10-27 12:55:04] ...trap/engines/engine.py.230: loss: 0.14056 [I 2022-10-27 12:55:04] ...optimizers/trijoint.py.151: total_norm: 15.830557 [I 2022-10-27 12:55:10] ...trap/engines/engine.py.224: train: epoch 68 | batch 850/2304 [I 2022-10-27 12:55:10] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:27 [I 2022-10-27 12:55:10] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00250 [I 2022-10-27 12:55:10] ...trap/engines/engine.py.230: loss: 0.16276 [I 2022-10-27 12:55:10] ...optimizers/trijoint.py.151: total_norm: 12.434781 [I 2022-10-27 12:55:15] ...trap/engines/engine.py.224: train: epoch 68 | batch 860/2304 [I 2022-10-27 12:55:15] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:28 [I 2022-10-27 12:55:15] ...trap/engines/engine.py.229: process: 0.56497 | load: 0.00251 [I 2022-10-27 12:55:15] ...trap/engines/engine.py.230: loss: 0.09753 [I 2022-10-27 12:55:15] ...optimizers/trijoint.py.151: total_norm: 14.371718 [I 2022-10-27 12:55:21] ...trap/engines/engine.py.224: train: epoch 68 | batch 870/2304 [I 2022-10-27 12:55:21] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:16 [I 2022-10-27 12:55:21] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00253 [I 2022-10-27 12:55:21] ...trap/engines/engine.py.230: loss: 0.14275 [I 2022-10-27 12:55:21] ...optimizers/trijoint.py.151: total_norm: 12.046323 [I 2022-10-27 12:55:27] ...trap/engines/engine.py.224: train: epoch 68 | batch 880/2304 [I 2022-10-27 12:55:27] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:12 [I 2022-10-27 12:55:27] ...trap/engines/engine.py.229: process: 0.56254 | load: 0.00242 [I 2022-10-27 12:55:27] ...trap/engines/engine.py.230: loss: 0.13128 [I 2022-10-27 12:55:27] ...optimizers/trijoint.py.151: total_norm: 10.172651 [I 2022-10-27 12:55:32] ...trap/engines/engine.py.224: train: epoch 68 | batch 890/2304 [I 2022-10-27 12:55:32] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:05 [I 2022-10-27 12:55:32] ...trap/engines/engine.py.229: process: 0.55494 | load: 0.00244 [I 2022-10-27 12:55:32] ...trap/engines/engine.py.230: loss: 0.22411 [I 2022-10-27 12:55:32] ...optimizers/trijoint.py.151: total_norm: 13.615767 [I 2022-10-27 12:55:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 900/2304 [I 2022-10-27 12:55:38] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:12:59 [I 2022-10-27 12:55:38] ...trap/engines/engine.py.229: process: 0.55320 | load: 0.00242 [I 2022-10-27 12:55:38] ...trap/engines/engine.py.230: loss: 0.14225 [I 2022-10-27 12:55:38] ...optimizers/trijoint.py.151: total_norm: 18.116056 [I 2022-10-27 12:55:43] ...trap/engines/engine.py.224: train: epoch 68 | batch 910/2304 [I 2022-10-27 12:55:43] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:51 [I 2022-10-27 12:55:43] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00242 [I 2022-10-27 12:55:43] ...trap/engines/engine.py.230: loss: 0.14816 [I 2022-10-27 12:55:43] ...optimizers/trijoint.py.151: total_norm: 12.452291 [I 2022-10-27 12:55:49] ...trap/engines/engine.py.224: train: epoch 68 | batch 920/2304 [I 2022-10-27 12:55:49] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:51 [I 2022-10-27 12:55:49] ...trap/engines/engine.py.229: process: 0.55713 | load: 0.00242 [I 2022-10-27 12:55:49] ...trap/engines/engine.py.230: loss: 0.11965 [I 2022-10-27 12:55:49] ...optimizers/trijoint.py.151: total_norm: 18.605658 [I 2022-10-27 12:55:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 930/2304 [I 2022-10-27 12:55:55] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:49 [I 2022-10-27 12:55:55] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00242 [I 2022-10-27 12:55:55] ...trap/engines/engine.py.230: loss: 0.11077 [I 2022-10-27 12:55:55] ...optimizers/trijoint.py.151: total_norm: 14.249238 [I 2022-10-27 12:56:00] ...trap/engines/engine.py.224: train: epoch 68 | batch 940/2304 [I 2022-10-27 12:56:00] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:38 [I 2022-10-27 12:56:00] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00247 [I 2022-10-27 12:56:00] ...trap/engines/engine.py.230: loss: 0.13839 [I 2022-10-27 12:56:00] ...optimizers/trijoint.py.151: total_norm: 16.851374 [I 2022-10-27 12:56:06] ...trap/engines/engine.py.224: train: epoch 68 | batch 950/2304 [I 2022-10-27 12:56:06] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:33 [I 2022-10-27 12:56:06] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00257 [I 2022-10-27 12:56:06] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-27 12:56:06] ...optimizers/trijoint.py.151: total_norm: 23.719372 [I 2022-10-27 12:56:11] ...trap/engines/engine.py.224: train: epoch 68 | batch 960/2304 [I 2022-10-27 12:56:11] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:25 [I 2022-10-27 12:56:11] ...trap/engines/engine.py.229: process: 0.55029 | load: 0.00250 [I 2022-10-27 12:56:11] ...trap/engines/engine.py.230: loss: 0.17584 [I 2022-10-27 12:56:11] ...optimizers/trijoint.py.151: total_norm: 19.176163 [I 2022-10-27 12:56:17] ...trap/engines/engine.py.224: train: epoch 68 | batch 970/2304 [I 2022-10-27 12:56:17] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:32 [I 2022-10-27 12:56:17] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00250 [I 2022-10-27 12:56:17] ...trap/engines/engine.py.230: loss: 0.14465 [I 2022-10-27 12:56:17] ...optimizers/trijoint.py.151: total_norm: 11.914510 [I 2022-10-27 12:56:23] ...trap/engines/engine.py.224: train: epoch 68 | batch 980/2304 [I 2022-10-27 12:56:23] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:21 [I 2022-10-27 12:56:23] ...trap/engines/engine.py.229: process: 0.55900 | load: 0.00244 [I 2022-10-27 12:56:23] ...trap/engines/engine.py.230: loss: 0.16755 [I 2022-10-27 12:56:23] ...optimizers/trijoint.py.151: total_norm: 30.869034 [I 2022-10-27 12:56:28] ...trap/engines/engine.py.224: train: epoch 68 | batch 990/2304 [I 2022-10-27 12:56:28] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:20 [I 2022-10-27 12:56:28] ...trap/engines/engine.py.229: process: 0.54696 | load: 0.00242 [I 2022-10-27 12:56:28] ...trap/engines/engine.py.230: loss: 0.17359 [I 2022-10-27 12:56:28] ...optimizers/trijoint.py.151: total_norm: 16.506300 [I 2022-10-27 12:56:34] ...trap/engines/engine.py.224: train: epoch 68 | batch 1000/2304 [I 2022-10-27 12:56:34] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:06 [I 2022-10-27 12:56:34] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00242 [I 2022-10-27 12:56:34] ...trap/engines/engine.py.230: loss: 0.12058 [I 2022-10-27 12:56:34] ...optimizers/trijoint.py.151: total_norm: 15.157982 [I 2022-10-27 12:56:39] ...trap/engines/engine.py.224: train: epoch 68 | batch 1010/2304 [I 2022-10-27 12:56:39] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:01 [I 2022-10-27 12:56:39] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00247 [I 2022-10-27 12:56:39] ...trap/engines/engine.py.230: loss: 0.14853 [I 2022-10-27 12:56:39] ...optimizers/trijoint.py.151: total_norm: 25.882429 [I 2022-10-27 12:56:45] ...trap/engines/engine.py.224: train: epoch 68 | batch 1020/2304 [I 2022-10-27 12:56:45] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:53 [I 2022-10-27 12:56:45] ...trap/engines/engine.py.229: process: 0.54337 | load: 0.00244 [I 2022-10-27 12:56:45] ...trap/engines/engine.py.230: loss: 0.11485 [I 2022-10-27 12:56:45] ...optimizers/trijoint.py.151: total_norm: 33.518677 [I 2022-10-27 12:56:51] ...trap/engines/engine.py.224: train: epoch 68 | batch 1030/2304 [I 2022-10-27 12:56:51] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:48 [I 2022-10-27 12:56:51] ...trap/engines/engine.py.229: process: 0.54943 | load: 0.00243 [I 2022-10-27 12:56:51] ...trap/engines/engine.py.230: loss: 0.12072 [I 2022-10-27 12:56:51] ...optimizers/trijoint.py.151: total_norm: 15.051615 [I 2022-10-27 12:56:56] ...trap/engines/engine.py.224: train: epoch 68 | batch 1040/2304 [I 2022-10-27 12:56:56] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:44 [I 2022-10-27 12:56:56] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00242 [I 2022-10-27 12:56:56] ...trap/engines/engine.py.230: loss: 0.18609 [I 2022-10-27 12:56:56] ...optimizers/trijoint.py.151: total_norm: 23.476406 [I 2022-10-27 12:57:02] ...trap/engines/engine.py.224: train: epoch 68 | batch 1050/2304 [I 2022-10-27 12:57:02] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:12:06 [I 2022-10-27 12:57:02] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00242 [I 2022-10-27 12:57:02] ...trap/engines/engine.py.230: loss: 0.12120 [I 2022-10-27 12:57:02] ...optimizers/trijoint.py.151: total_norm: 12.200579 [I 2022-10-27 12:57:08] ...trap/engines/engine.py.224: train: epoch 68 | batch 1060/2304 [I 2022-10-27 12:57:08] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:39 [I 2022-10-27 12:57:08] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00241 [I 2022-10-27 12:57:08] ...trap/engines/engine.py.230: loss: 0.24006 [I 2022-10-27 12:57:08] ...optimizers/trijoint.py.151: total_norm: 22.373938 [I 2022-10-27 12:57:13] ...trap/engines/engine.py.224: train: epoch 68 | batch 1070/2304 [I 2022-10-27 12:57:13] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:22 [I 2022-10-27 12:57:13] ...trap/engines/engine.py.229: process: 0.54967 | load: 0.00247 [I 2022-10-27 12:57:13] ...trap/engines/engine.py.230: loss: 0.10110 [I 2022-10-27 12:57:13] ...optimizers/trijoint.py.151: total_norm: 11.616964 [I 2022-10-27 12:57:19] ...trap/engines/engine.py.224: train: epoch 68 | batch 1080/2304 [I 2022-10-27 12:57:19] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:18 [I 2022-10-27 12:57:19] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00246 [I 2022-10-27 12:57:19] ...trap/engines/engine.py.230: loss: 0.12776 [I 2022-10-27 12:57:19] ...optimizers/trijoint.py.151: total_norm: 20.745277 [I 2022-10-27 12:57:24] ...trap/engines/engine.py.224: train: epoch 68 | batch 1090/2304 [I 2022-10-27 12:57:24] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:13 [I 2022-10-27 12:57:24] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00241 [I 2022-10-27 12:57:24] ...trap/engines/engine.py.230: loss: 0.10867 [I 2022-10-27 12:57:24] ...optimizers/trijoint.py.151: total_norm: 8.813143 [I 2022-10-27 12:57:30] ...trap/engines/engine.py.224: train: epoch 68 | batch 1100/2304 [I 2022-10-27 12:57:30] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:13 [I 2022-10-27 12:57:30] ...trap/engines/engine.py.229: process: 0.56880 | load: 0.00240 [I 2022-10-27 12:57:30] ...trap/engines/engine.py.230: loss: 0.14115 [I 2022-10-27 12:57:30] ...optimizers/trijoint.py.151: total_norm: 21.397165 [I 2022-10-27 12:57:35] ...trap/engines/engine.py.224: train: epoch 68 | batch 1110/2304 [I 2022-10-27 12:57:35] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:07 [I 2022-10-27 12:57:35] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00262 [I 2022-10-27 12:57:35] ...trap/engines/engine.py.230: loss: 0.17063 [I 2022-10-27 12:57:35] ...optimizers/trijoint.py.151: total_norm: 12.717702 [I 2022-10-27 12:57:41] ...trap/engines/engine.py.224: train: epoch 68 | batch 1120/2304 [I 2022-10-27 12:57:41] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:03 [I 2022-10-27 12:57:41] ...trap/engines/engine.py.229: process: 0.56340 | load: 0.00252 [I 2022-10-27 12:57:41] ...trap/engines/engine.py.230: loss: 0.10751 [I 2022-10-27 12:57:41] ...optimizers/trijoint.py.151: total_norm: 22.648861 [I 2022-10-27 12:57:47] ...trap/engines/engine.py.224: train: epoch 68 | batch 1130/2304 [I 2022-10-27 12:57:47] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:56 [I 2022-10-27 12:57:47] ...trap/engines/engine.py.229: process: 0.56799 | load: 0.00246 [I 2022-10-27 12:57:47] ...trap/engines/engine.py.230: loss: 0.15391 [I 2022-10-27 12:57:47] ...optimizers/trijoint.py.151: total_norm: 12.284021 [I 2022-10-27 12:57:52] ...trap/engines/engine.py.224: train: epoch 68 | batch 1140/2304 [I 2022-10-27 12:57:52] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:50 [I 2022-10-27 12:57:52] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00245 [I 2022-10-27 12:57:52] ...trap/engines/engine.py.230: loss: 0.17330 [I 2022-10-27 12:57:52] ...optimizers/trijoint.py.151: total_norm: 26.649912 [I 2022-10-27 12:57:58] ...trap/engines/engine.py.224: train: epoch 68 | batch 1150/2304 [I 2022-10-27 12:57:58] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:42 [I 2022-10-27 12:57:58] ...trap/engines/engine.py.229: process: 0.54447 | load: 0.00245 [I 2022-10-27 12:57:58] ...trap/engines/engine.py.230: loss: 0.11663 [I 2022-10-27 12:57:58] ...optimizers/trijoint.py.151: total_norm: 8.306056 [I 2022-10-27 12:58:03] ...trap/engines/engine.py.224: train: epoch 68 | batch 1160/2304 [I 2022-10-27 12:58:03] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:38 [I 2022-10-27 12:58:03] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00248 [I 2022-10-27 12:58:03] ...trap/engines/engine.py.230: loss: 0.18965 [I 2022-10-27 12:58:03] ...optimizers/trijoint.py.151: total_norm: 16.638575 [I 2022-10-27 12:58:09] ...trap/engines/engine.py.224: train: epoch 68 | batch 1170/2304 [I 2022-10-27 12:58:09] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:30 [I 2022-10-27 12:58:09] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00249 [I 2022-10-27 12:58:09] ...trap/engines/engine.py.230: loss: 0.12153 [I 2022-10-27 12:58:09] ...optimizers/trijoint.py.151: total_norm: 11.446288 [I 2022-10-27 12:58:15] ...trap/engines/engine.py.224: train: epoch 68 | batch 1180/2304 [I 2022-10-27 12:58:15] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:25 [I 2022-10-27 12:58:15] ...trap/engines/engine.py.229: process: 0.56662 | load: 0.00248 [I 2022-10-27 12:58:15] ...trap/engines/engine.py.230: loss: 0.18022 [I 2022-10-27 12:58:15] ...optimizers/trijoint.py.151: total_norm: 17.784098 [I 2022-10-27 12:58:20] ...trap/engines/engine.py.224: train: epoch 68 | batch 1190/2304 [I 2022-10-27 12:58:20] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:21 [I 2022-10-27 12:58:20] ...trap/engines/engine.py.229: process: 0.55932 | load: 0.00243 [I 2022-10-27 12:58:20] ...trap/engines/engine.py.230: loss: 0.13785 [I 2022-10-27 12:58:20] ...optimizers/trijoint.py.151: total_norm: 25.854576 [I 2022-10-27 12:58:26] ...trap/engines/engine.py.224: train: epoch 68 | batch 1200/2304 [I 2022-10-27 12:58:26] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:14 [I 2022-10-27 12:58:26] ...trap/engines/engine.py.229: process: 0.55694 | load: 0.00240 [I 2022-10-27 12:58:26] ...trap/engines/engine.py.230: loss: 0.09836 [I 2022-10-27 12:58:26] ...optimizers/trijoint.py.151: total_norm: 16.704433 [I 2022-10-27 12:58:31] ...trap/engines/engine.py.224: train: epoch 68 | batch 1210/2304 [I 2022-10-27 12:58:31] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:17 [I 2022-10-27 12:58:31] ...trap/engines/engine.py.229: process: 0.57824 | load: 0.00243 [I 2022-10-27 12:58:31] ...trap/engines/engine.py.230: loss: 0.15262 [I 2022-10-27 12:58:31] ...optimizers/trijoint.py.151: total_norm: 14.243994 [I 2022-10-27 12:58:37] ...trap/engines/engine.py.224: train: epoch 68 | batch 1220/2304 [I 2022-10-27 12:58:37] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:06 [I 2022-10-27 12:58:37] ...trap/engines/engine.py.229: process: 0.55679 | load: 0.00249 [I 2022-10-27 12:58:37] ...trap/engines/engine.py.230: loss: 0.17548 [I 2022-10-27 12:58:37] ...optimizers/trijoint.py.151: total_norm: 20.073444 [I 2022-10-27 12:58:43] ...trap/engines/engine.py.224: train: epoch 68 | batch 1230/2304 [I 2022-10-27 12:58:43] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:59 [I 2022-10-27 12:58:43] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00248 [I 2022-10-27 12:58:43] ...trap/engines/engine.py.230: loss: 0.12987 [I 2022-10-27 12:58:43] ...optimizers/trijoint.py.151: total_norm: 15.634626 [I 2022-10-27 12:58:48] ...trap/engines/engine.py.224: train: epoch 68 | batch 1240/2304 [I 2022-10-27 12:58:48] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:58 [I 2022-10-27 12:58:48] ...trap/engines/engine.py.229: process: 0.56829 | load: 0.00261 [I 2022-10-27 12:58:48] ...trap/engines/engine.py.230: loss: 0.15292 [I 2022-10-27 12:58:48] ...optimizers/trijoint.py.151: total_norm: 14.472929 [I 2022-10-27 12:58:54] ...trap/engines/engine.py.224: train: epoch 68 | batch 1250/2304 [I 2022-10-27 12:58:54] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:46 [I 2022-10-27 12:58:54] ...trap/engines/engine.py.229: process: 0.56406 | load: 0.00247 [I 2022-10-27 12:58:54] ...trap/engines/engine.py.230: loss: 0.19481 [I 2022-10-27 12:58:54] ...optimizers/trijoint.py.151: total_norm: 29.145481 [I 2022-10-27 12:58:59] ...trap/engines/engine.py.224: train: epoch 68 | batch 1260/2304 [I 2022-10-27 12:58:59] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:37 [I 2022-10-27 12:58:59] ...trap/engines/engine.py.229: process: 0.54831 | load: 0.00250 [I 2022-10-27 12:58:59] ...trap/engines/engine.py.230: loss: 0.19130 [I 2022-10-27 12:58:59] ...optimizers/trijoint.py.151: total_norm: 14.982063 [I 2022-10-27 12:59:05] ...trap/engines/engine.py.224: train: epoch 68 | batch 1270/2304 [I 2022-10-27 12:59:05] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:35 [I 2022-10-27 12:59:05] ...trap/engines/engine.py.229: process: 0.56524 | load: 0.00246 [I 2022-10-27 12:59:05] ...trap/engines/engine.py.230: loss: 0.11317 [I 2022-10-27 12:59:05] ...optimizers/trijoint.py.151: total_norm: 12.050111 [I 2022-10-27 12:59:10] ...trap/engines/engine.py.224: train: epoch 68 | batch 1280/2304 [I 2022-10-27 12:59:10] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:28 [I 2022-10-27 12:59:10] ...trap/engines/engine.py.229: process: 0.54944 | load: 0.00268 [I 2022-10-27 12:59:10] ...trap/engines/engine.py.230: loss: 0.12369 [I 2022-10-27 12:59:10] ...optimizers/trijoint.py.151: total_norm: 15.696248 [I 2022-10-27 12:59:16] ...trap/engines/engine.py.224: train: epoch 68 | batch 1290/2304 [I 2022-10-27 12:59:16] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:27 [I 2022-10-27 12:59:16] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00245 [I 2022-10-27 12:59:16] ...trap/engines/engine.py.230: loss: 0.13228 [I 2022-10-27 12:59:16] ...optimizers/trijoint.py.151: total_norm: 33.072086 [I 2022-10-27 12:59:22] ...trap/engines/engine.py.224: train: epoch 68 | batch 1300/2304 [I 2022-10-27 12:59:22] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:17 [I 2022-10-27 12:59:22] ...trap/engines/engine.py.229: process: 0.54370 | load: 0.00247 [I 2022-10-27 12:59:22] ...trap/engines/engine.py.230: loss: 0.09799 [I 2022-10-27 12:59:22] ...optimizers/trijoint.py.151: total_norm: 10.254186 [I 2022-10-27 12:59:27] ...trap/engines/engine.py.224: train: epoch 68 | batch 1310/2304 [I 2022-10-27 12:59:27] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:11 [I 2022-10-27 12:59:27] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00244 [I 2022-10-27 12:59:27] ...trap/engines/engine.py.230: loss: 0.13624 [I 2022-10-27 12:59:27] ...optimizers/trijoint.py.151: total_norm: 18.229975 [I 2022-10-27 12:59:33] ...trap/engines/engine.py.224: train: epoch 68 | batch 1320/2304 [I 2022-10-27 12:59:33] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:09 [I 2022-10-27 12:59:33] ...trap/engines/engine.py.229: process: 0.55437 | load: 0.00245 [I 2022-10-27 12:59:33] ...trap/engines/engine.py.230: loss: 0.13602 [I 2022-10-27 12:59:33] ...optimizers/trijoint.py.151: total_norm: 24.116739 [I 2022-10-27 12:59:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 1330/2304 [I 2022-10-27 12:59:38] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:03 [I 2022-10-27 12:59:38] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00245 [I 2022-10-27 12:59:38] ...trap/engines/engine.py.230: loss: 0.14055 [I 2022-10-27 12:59:38] ...optimizers/trijoint.py.151: total_norm: 13.426681 [I 2022-10-27 12:59:44] ...trap/engines/engine.py.224: train: epoch 68 | batch 1340/2304 [I 2022-10-27 12:59:44] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:09:09 [I 2022-10-27 12:59:44] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00259 [I 2022-10-27 12:59:44] ...trap/engines/engine.py.230: loss: 0.15963 [I 2022-10-27 12:59:44] ...optimizers/trijoint.py.151: total_norm: 17.284895 [I 2022-10-27 12:59:50] ...trap/engines/engine.py.224: train: epoch 68 | batch 1350/2304 [I 2022-10-27 12:59:50] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:54 [I 2022-10-27 12:59:50] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00242 [I 2022-10-27 12:59:50] ...trap/engines/engine.py.230: loss: 0.22548 [I 2022-10-27 12:59:50] ...optimizers/trijoint.py.151: total_norm: 29.613480 [I 2022-10-27 12:59:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 1360/2304 [I 2022-10-27 12:59:55] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:45 [I 2022-10-27 12:59:55] ...trap/engines/engine.py.229: process: 0.55583 | load: 0.00245 [I 2022-10-27 12:59:55] ...trap/engines/engine.py.230: loss: 0.21655 [I 2022-10-27 12:59:55] ...optimizers/trijoint.py.151: total_norm: 21.160761 [I 2022-10-27 13:00:01] ...trap/engines/engine.py.224: train: epoch 68 | batch 1370/2304 [I 2022-10-27 13:00:01] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:41 [I 2022-10-27 13:00:01] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00240 [I 2022-10-27 13:00:01] ...trap/engines/engine.py.230: loss: 0.11569 [I 2022-10-27 13:00:01] ...optimizers/trijoint.py.151: total_norm: 8.699775 [I 2022-10-27 13:00:06] ...trap/engines/engine.py.224: train: epoch 68 | batch 1380/2304 [I 2022-10-27 13:00:06] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:34 [I 2022-10-27 13:00:06] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00250 [I 2022-10-27 13:00:06] ...trap/engines/engine.py.230: loss: 0.13159 [I 2022-10-27 13:00:06] ...optimizers/trijoint.py.151: total_norm: 19.905231 [I 2022-10-27 13:00:12] ...trap/engines/engine.py.224: train: epoch 68 | batch 1390/2304 [I 2022-10-27 13:00:12] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:29 [I 2022-10-27 13:00:12] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00243 [I 2022-10-27 13:00:12] ...trap/engines/engine.py.230: loss: 0.10055 [I 2022-10-27 13:00:12] ...optimizers/trijoint.py.151: total_norm: 14.428400 [I 2022-10-27 13:00:18] ...trap/engines/engine.py.224: train: epoch 68 | batch 1400/2304 [I 2022-10-27 13:00:18] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:22 [I 2022-10-27 13:00:18] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00244 [I 2022-10-27 13:00:18] ...trap/engines/engine.py.230: loss: 0.12754 [I 2022-10-27 13:00:18] ...optimizers/trijoint.py.151: total_norm: 19.344334 [I 2022-10-27 13:00:23] ...trap/engines/engine.py.224: train: epoch 68 | batch 1410/2304 [I 2022-10-27 13:00:23] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:19 [I 2022-10-27 13:00:23] ...trap/engines/engine.py.229: process: 0.56164 | load: 0.00244 [I 2022-10-27 13:00:23] ...trap/engines/engine.py.230: loss: 0.10124 [I 2022-10-27 13:00:23] ...optimizers/trijoint.py.151: total_norm: 12.637732 [I 2022-10-27 13:00:29] ...trap/engines/engine.py.224: train: epoch 68 | batch 1420/2304 [I 2022-10-27 13:00:29] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:17 [I 2022-10-27 13:00:29] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00244 [I 2022-10-27 13:00:29] ...trap/engines/engine.py.230: loss: 0.20992 [I 2022-10-27 13:00:29] ...optimizers/trijoint.py.151: total_norm: 27.991968 [I 2022-10-27 13:00:34] ...trap/engines/engine.py.224: train: epoch 68 | batch 1430/2304 [I 2022-10-27 13:00:34] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:05 [I 2022-10-27 13:00:34] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00240 [I 2022-10-27 13:00:34] ...trap/engines/engine.py.230: loss: 0.12505 [I 2022-10-27 13:00:34] ...optimizers/trijoint.py.151: total_norm: 14.282835 [I 2022-10-27 13:00:40] ...trap/engines/engine.py.224: train: epoch 68 | batch 1440/2304 [I 2022-10-27 13:00:40] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:12 [I 2022-10-27 13:00:40] ...trap/engines/engine.py.229: process: 0.57373 | load: 0.01712 [I 2022-10-27 13:00:40] ...trap/engines/engine.py.230: loss: 0.10695 [I 2022-10-27 13:00:40] ...optimizers/trijoint.py.151: total_norm: 8.351890 [I 2022-10-27 13:00:46] ...trap/engines/engine.py.224: train: epoch 68 | batch 1450/2304 [I 2022-10-27 13:00:46] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:54 [I 2022-10-27 13:00:46] ...trap/engines/engine.py.229: process: 0.54660 | load: 0.00242 [I 2022-10-27 13:00:46] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-27 13:00:46] ...optimizers/trijoint.py.151: total_norm: 10.212614 [I 2022-10-27 13:00:51] ...trap/engines/engine.py.224: train: epoch 68 | batch 1460/2304 [I 2022-10-27 13:00:51] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:53 [I 2022-10-27 13:00:51] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00241 [I 2022-10-27 13:00:51] ...trap/engines/engine.py.230: loss: 0.11383 [I 2022-10-27 13:00:51] ...optimizers/trijoint.py.151: total_norm: 16.618649 [I 2022-10-27 13:00:57] ...trap/engines/engine.py.224: train: epoch 68 | batch 1470/2304 [I 2022-10-27 13:00:57] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:48 [I 2022-10-27 13:00:57] ...trap/engines/engine.py.229: process: 0.57570 | load: 0.00245 [I 2022-10-27 13:00:57] ...trap/engines/engine.py.230: loss: 0.11356 [I 2022-10-27 13:00:57] ...optimizers/trijoint.py.151: total_norm: 8.146670 [I 2022-10-27 13:01:02] ...trap/engines/engine.py.224: train: epoch 68 | batch 1480/2304 [I 2022-10-27 13:01:02] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:40 [I 2022-10-27 13:01:02] ...trap/engines/engine.py.229: process: 0.54827 | load: 0.00260 [I 2022-10-27 13:01:02] ...trap/engines/engine.py.230: loss: 0.21991 [I 2022-10-27 13:01:02] ...optimizers/trijoint.py.151: total_norm: 30.534700 [I 2022-10-27 13:01:08] ...trap/engines/engine.py.224: train: epoch 68 | batch 1490/2304 [I 2022-10-27 13:01:08] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:35 [I 2022-10-27 13:01:08] ...trap/engines/engine.py.229: process: 0.56843 | load: 0.00247 [I 2022-10-27 13:01:08] ...trap/engines/engine.py.230: loss: 0.11094 [I 2022-10-27 13:01:08] ...optimizers/trijoint.py.151: total_norm: 19.936253 [I 2022-10-27 13:01:14] ...trap/engines/engine.py.224: train: epoch 68 | batch 1500/2304 [I 2022-10-27 13:01:14] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:28 [I 2022-10-27 13:01:14] ...trap/engines/engine.py.229: process: 0.56249 | load: 0.00248 [I 2022-10-27 13:01:14] ...trap/engines/engine.py.230: loss: 0.13687 [I 2022-10-27 13:01:14] ...optimizers/trijoint.py.151: total_norm: 14.462941 [I 2022-10-27 13:01:19] ...trap/engines/engine.py.224: train: epoch 68 | batch 1510/2304 [I 2022-10-27 13:01:19] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:21 [I 2022-10-27 13:01:19] ...trap/engines/engine.py.229: process: 0.55868 | load: 0.00244 [I 2022-10-27 13:01:19] ...trap/engines/engine.py.230: loss: 0.16336 [I 2022-10-27 13:01:19] ...optimizers/trijoint.py.151: total_norm: 24.836239 [I 2022-10-27 13:01:25] ...trap/engines/engine.py.224: train: epoch 68 | batch 1520/2304 [I 2022-10-27 13:01:25] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:17 [I 2022-10-27 13:01:25] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00245 [I 2022-10-27 13:01:25] ...trap/engines/engine.py.230: loss: 0.11300 [I 2022-10-27 13:01:25] ...optimizers/trijoint.py.151: total_norm: 16.390785 [I 2022-10-27 13:01:30] ...trap/engines/engine.py.224: train: epoch 68 | batch 1530/2304 [I 2022-10-27 13:01:30] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:12 [I 2022-10-27 13:01:30] ...trap/engines/engine.py.229: process: 0.56637 | load: 0.00245 [I 2022-10-27 13:01:30] ...trap/engines/engine.py.230: loss: 0.12781 [I 2022-10-27 13:01:30] ...optimizers/trijoint.py.151: total_norm: 13.251365 [I 2022-10-27 13:01:36] ...trap/engines/engine.py.224: train: epoch 68 | batch 1540/2304 [I 2022-10-27 13:01:36] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:07 [I 2022-10-27 13:01:36] ...trap/engines/engine.py.229: process: 0.56229 | load: 0.00243 [I 2022-10-27 13:01:36] ...trap/engines/engine.py.230: loss: 0.13231 [I 2022-10-27 13:01:36] ...optimizers/trijoint.py.151: total_norm: 16.604330 [I 2022-10-27 13:01:42] ...trap/engines/engine.py.224: train: epoch 68 | batch 1550/2304 [I 2022-10-27 13:01:42] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:01 [I 2022-10-27 13:01:42] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00241 [I 2022-10-27 13:01:42] ...trap/engines/engine.py.230: loss: 0.15578 [I 2022-10-27 13:01:42] ...optimizers/trijoint.py.151: total_norm: 11.582384 [I 2022-10-27 13:01:47] ...trap/engines/engine.py.224: train: epoch 68 | batch 1560/2304 [I 2022-10-27 13:01:47] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:55 [I 2022-10-27 13:01:47] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00247 [I 2022-10-27 13:01:47] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-27 13:01:47] ...optimizers/trijoint.py.151: total_norm: 7.971267 [I 2022-10-27 13:01:53] ...trap/engines/engine.py.224: train: epoch 68 | batch 1570/2304 [I 2022-10-27 13:01:53] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:49 [I 2022-10-27 13:01:53] ...trap/engines/engine.py.229: process: 0.56549 | load: 0.00248 [I 2022-10-27 13:01:53] ...trap/engines/engine.py.230: loss: 0.10193 [I 2022-10-27 13:01:53] ...optimizers/trijoint.py.151: total_norm: 9.961570 [I 2022-10-27 13:01:58] ...trap/engines/engine.py.224: train: epoch 68 | batch 1580/2304 [I 2022-10-27 13:01:58] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:42 [I 2022-10-27 13:01:58] ...trap/engines/engine.py.229: process: 0.55115 | load: 0.00243 [I 2022-10-27 13:01:58] ...trap/engines/engine.py.230: loss: 0.25829 [I 2022-10-27 13:01:58] ...optimizers/trijoint.py.151: total_norm: 33.680141 [I 2022-10-27 13:02:04] ...trap/engines/engine.py.224: train: epoch 68 | batch 1590/2304 [I 2022-10-27 13:02:04] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:36 [I 2022-10-27 13:02:04] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00255 [I 2022-10-27 13:02:04] ...trap/engines/engine.py.230: loss: 0.10566 [I 2022-10-27 13:02:04] ...optimizers/trijoint.py.151: total_norm: 12.178542 [I 2022-10-27 13:02:09] ...trap/engines/engine.py.224: train: epoch 68 | batch 1600/2304 [I 2022-10-27 13:02:09] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:29 [I 2022-10-27 13:02:09] ...trap/engines/engine.py.229: process: 0.55219 | load: 0.00246 [I 2022-10-27 13:02:09] ...trap/engines/engine.py.230: loss: 0.10210 [I 2022-10-27 13:02:09] ...optimizers/trijoint.py.151: total_norm: 11.619885 [I 2022-10-27 13:02:15] ...trap/engines/engine.py.224: train: epoch 68 | batch 1610/2304 [I 2022-10-27 13:02:15] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:25 [I 2022-10-27 13:02:15] ...trap/engines/engine.py.229: process: 0.56162 | load: 0.00244 [I 2022-10-27 13:02:15] ...trap/engines/engine.py.230: loss: 0.17374 [I 2022-10-27 13:02:15] ...optimizers/trijoint.py.151: total_norm: 41.463947 [I 2022-10-27 13:02:21] ...trap/engines/engine.py.224: train: epoch 68 | batch 1620/2304 [I 2022-10-27 13:02:21] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:21 [I 2022-10-27 13:02:21] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00242 [I 2022-10-27 13:02:21] ...trap/engines/engine.py.230: loss: 0.12850 [I 2022-10-27 13:02:21] ...optimizers/trijoint.py.151: total_norm: 28.475704 [I 2022-10-27 13:02:26] ...trap/engines/engine.py.224: train: epoch 68 | batch 1630/2304 [I 2022-10-27 13:02:26] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:17 [I 2022-10-27 13:02:26] ...trap/engines/engine.py.229: process: 0.56646 | load: 0.00242 [I 2022-10-27 13:02:26] ...trap/engines/engine.py.230: loss: 0.19565 [I 2022-10-27 13:02:26] ...optimizers/trijoint.py.151: total_norm: 51.915848 [I 2022-10-27 13:02:32] ...trap/engines/engine.py.224: train: epoch 68 | batch 1640/2304 [I 2022-10-27 13:02:32] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:10 [I 2022-10-27 13:02:32] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00247 [I 2022-10-27 13:02:32] ...trap/engines/engine.py.230: loss: 0.14267 [I 2022-10-27 13:02:32] ...optimizers/trijoint.py.151: total_norm: 16.629005 [I 2022-10-27 13:02:37] ...trap/engines/engine.py.224: train: epoch 68 | batch 1650/2304 [I 2022-10-27 13:02:37] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:03 [I 2022-10-27 13:02:37] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00244 [I 2022-10-27 13:02:37] ...trap/engines/engine.py.230: loss: 0.11720 [I 2022-10-27 13:02:37] ...optimizers/trijoint.py.151: total_norm: 12.651967 [I 2022-10-27 13:02:43] ...trap/engines/engine.py.224: train: epoch 68 | batch 1660/2304 [I 2022-10-27 13:02:43] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:01 [I 2022-10-27 13:02:43] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00244 [I 2022-10-27 13:02:43] ...trap/engines/engine.py.230: loss: 0.14057 [I 2022-10-27 13:02:43] ...optimizers/trijoint.py.151: total_norm: 14.210765 [I 2022-10-27 13:02:49] ...trap/engines/engine.py.224: train: epoch 68 | batch 1670/2304 [I 2022-10-27 13:02:49] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:52 [I 2022-10-27 13:02:49] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00243 [I 2022-10-27 13:02:49] ...trap/engines/engine.py.230: loss: 0.14511 [I 2022-10-27 13:02:49] ...optimizers/trijoint.py.151: total_norm: 24.745565 [I 2022-10-27 13:02:54] ...trap/engines/engine.py.224: train: epoch 68 | batch 1680/2304 [I 2022-10-27 13:02:54] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:48 [I 2022-10-27 13:02:54] ...trap/engines/engine.py.229: process: 0.55408 | load: 0.00246 [I 2022-10-27 13:02:54] ...trap/engines/engine.py.230: loss: 0.21204 [I 2022-10-27 13:02:54] ...optimizers/trijoint.py.151: total_norm: 24.848200 [I 2022-10-27 13:03:00] ...trap/engines/engine.py.224: train: epoch 68 | batch 1690/2304 [I 2022-10-27 13:03:00] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:41 [I 2022-10-27 13:03:00] ...trap/engines/engine.py.229: process: 0.55555 | load: 0.00283 [I 2022-10-27 13:03:00] ...trap/engines/engine.py.230: loss: 0.11332 [I 2022-10-27 13:03:00] ...optimizers/trijoint.py.151: total_norm: 10.210456 [I 2022-10-27 13:03:05] ...trap/engines/engine.py.224: train: epoch 68 | batch 1700/2304 [I 2022-10-27 13:03:05] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:35 [I 2022-10-27 13:03:05] ...trap/engines/engine.py.229: process: 0.54661 | load: 0.00247 [I 2022-10-27 13:03:05] ...trap/engines/engine.py.230: loss: 0.16487 [I 2022-10-27 13:03:05] ...optimizers/trijoint.py.151: total_norm: 16.257326 [I 2022-10-27 13:03:11] ...trap/engines/engine.py.224: train: epoch 68 | batch 1710/2304 [I 2022-10-27 13:03:11] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:31 [I 2022-10-27 13:03:11] ...trap/engines/engine.py.229: process: 0.54515 | load: 0.00250 [I 2022-10-27 13:03:11] ...trap/engines/engine.py.230: loss: 0.10287 [I 2022-10-27 13:03:11] ...optimizers/trijoint.py.151: total_norm: 11.280407 [I 2022-10-27 13:03:16] ...trap/engines/engine.py.224: train: epoch 68 | batch 1720/2304 [I 2022-10-27 13:03:16] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:26 [I 2022-10-27 13:03:16] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00249 [I 2022-10-27 13:03:16] ...trap/engines/engine.py.230: loss: 0.19968 [I 2022-10-27 13:03:16] ...optimizers/trijoint.py.151: total_norm: 37.491863 [I 2022-10-27 13:03:22] ...trap/engines/engine.py.224: train: epoch 68 | batch 1730/2304 [I 2022-10-27 13:03:22] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:19 [I 2022-10-27 13:03:22] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00248 [I 2022-10-27 13:03:22] ...trap/engines/engine.py.230: loss: 0.11388 [I 2022-10-27 13:03:22] ...optimizers/trijoint.py.151: total_norm: 7.768980 [I 2022-10-27 13:03:28] ...trap/engines/engine.py.224: train: epoch 68 | batch 1740/2304 [I 2022-10-27 13:03:28] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:16 [I 2022-10-27 13:03:28] ...trap/engines/engine.py.229: process: 0.57188 | load: 0.00252 [I 2022-10-27 13:03:28] ...trap/engines/engine.py.230: loss: 0.11550 [I 2022-10-27 13:03:28] ...optimizers/trijoint.py.151: total_norm: 22.015402 [I 2022-10-27 13:03:33] ...trap/engines/engine.py.224: train: epoch 68 | batch 1750/2304 [I 2022-10-27 13:03:33] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:09 [I 2022-10-27 13:03:33] ...trap/engines/engine.py.229: process: 0.56874 | load: 0.00244 [I 2022-10-27 13:03:33] ...trap/engines/engine.py.230: loss: 0.11141 [I 2022-10-27 13:03:33] ...optimizers/trijoint.py.151: total_norm: 8.078814 [I 2022-10-27 13:03:39] ...trap/engines/engine.py.224: train: epoch 68 | batch 1760/2304 [I 2022-10-27 13:03:39] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:01 [I 2022-10-27 13:03:39] ...trap/engines/engine.py.229: process: 0.55711 | load: 0.00242 [I 2022-10-27 13:03:39] ...trap/engines/engine.py.230: loss: 0.10657 [I 2022-10-27 13:03:39] ...optimizers/trijoint.py.151: total_norm: 14.359747 [I 2022-10-27 13:03:44] ...trap/engines/engine.py.224: train: epoch 68 | batch 1770/2304 [I 2022-10-27 13:03:44] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:04:55 [I 2022-10-27 13:03:44] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00240 [I 2022-10-27 13:03:44] ...trap/engines/engine.py.230: loss: 0.10480 [I 2022-10-27 13:03:44] ...optimizers/trijoint.py.151: total_norm: 10.005802 [I 2022-10-27 13:03:50] ...trap/engines/engine.py.224: train: epoch 68 | batch 1780/2304 [I 2022-10-27 13:03:50] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:51 [I 2022-10-27 13:03:50] ...trap/engines/engine.py.229: process: 0.55054 | load: 0.00244 [I 2022-10-27 13:03:50] ...trap/engines/engine.py.230: loss: 0.18040 [I 2022-10-27 13:03:50] ...optimizers/trijoint.py.151: total_norm: 17.088589 [I 2022-10-27 13:03:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 1790/2304 [I 2022-10-27 13:03:55] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:45 [I 2022-10-27 13:03:55] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00242 [I 2022-10-27 13:03:55] ...trap/engines/engine.py.230: loss: 0.11661 [I 2022-10-27 13:03:55] ...optimizers/trijoint.py.151: total_norm: 15.343780 [I 2022-10-27 13:04:01] ...trap/engines/engine.py.224: train: epoch 68 | batch 1800/2304 [I 2022-10-27 13:04:01] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:38 [I 2022-10-27 13:04:01] ...trap/engines/engine.py.229: process: 0.55529 | load: 0.00243 [I 2022-10-27 13:04:01] ...trap/engines/engine.py.230: loss: 0.13221 [I 2022-10-27 13:04:01] ...optimizers/trijoint.py.151: total_norm: 29.969027 [I 2022-10-27 13:04:07] ...trap/engines/engine.py.224: train: epoch 68 | batch 1810/2304 [I 2022-10-27 13:04:07] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:35 [I 2022-10-27 13:04:07] ...trap/engines/engine.py.229: process: 0.56419 | load: 0.00238 [I 2022-10-27 13:04:07] ...trap/engines/engine.py.230: loss: 0.15048 [I 2022-10-27 13:04:07] ...optimizers/trijoint.py.151: total_norm: 17.227251 [I 2022-10-27 13:04:12] ...trap/engines/engine.py.224: train: epoch 68 | batch 1820/2304 [I 2022-10-27 13:04:12] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:28 [I 2022-10-27 13:04:12] ...trap/engines/engine.py.229: process: 0.55198 | load: 0.00240 [I 2022-10-27 13:04:12] ...trap/engines/engine.py.230: loss: 0.15299 [I 2022-10-27 13:04:12] ...optimizers/trijoint.py.151: total_norm: 19.768463 [I 2022-10-27 13:04:18] ...trap/engines/engine.py.224: train: epoch 68 | batch 1830/2304 [I 2022-10-27 13:04:18] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:21 [I 2022-10-27 13:04:18] ...trap/engines/engine.py.229: process: 0.55312 | load: 0.00252 [I 2022-10-27 13:04:18] ...trap/engines/engine.py.230: loss: 0.12115 [I 2022-10-27 13:04:18] ...optimizers/trijoint.py.151: total_norm: 21.796310 [I 2022-10-27 13:04:23] ...trap/engines/engine.py.224: train: epoch 68 | batch 1840/2304 [I 2022-10-27 13:04:23] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:18 [I 2022-10-27 13:04:23] ...trap/engines/engine.py.229: process: 0.55679 | load: 0.00246 [I 2022-10-27 13:04:23] ...trap/engines/engine.py.230: loss: 0.11065 [I 2022-10-27 13:04:23] ...optimizers/trijoint.py.151: total_norm: 12.616677 [I 2022-10-27 13:04:29] ...trap/engines/engine.py.224: train: epoch 68 | batch 1850/2304 [I 2022-10-27 13:04:29] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:13 [I 2022-10-27 13:04:29] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00242 [I 2022-10-27 13:04:29] ...trap/engines/engine.py.230: loss: 0.12930 [I 2022-10-27 13:04:29] ...optimizers/trijoint.py.151: total_norm: 9.733525 [I 2022-10-27 13:04:34] ...trap/engines/engine.py.224: train: epoch 68 | batch 1860/2304 [I 2022-10-27 13:04:34] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:06 [I 2022-10-27 13:04:34] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00245 [I 2022-10-27 13:04:34] ...trap/engines/engine.py.230: loss: 0.26858 [I 2022-10-27 13:04:34] ...optimizers/trijoint.py.151: total_norm: 18.531683 [I 2022-10-27 13:04:40] ...trap/engines/engine.py.224: train: epoch 68 | batch 1870/2304 [I 2022-10-27 13:04:40] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:02 [I 2022-10-27 13:04:40] ...trap/engines/engine.py.229: process: 0.56559 | load: 0.00243 [I 2022-10-27 13:04:40] ...trap/engines/engine.py.230: loss: 0.10272 [I 2022-10-27 13:04:40] ...optimizers/trijoint.py.151: total_norm: 10.932297 [I 2022-10-27 13:04:46] ...trap/engines/engine.py.224: train: epoch 68 | batch 1880/2304 [I 2022-10-27 13:04:46] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:03:57 [I 2022-10-27 13:04:46] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00242 [I 2022-10-27 13:04:46] ...trap/engines/engine.py.230: loss: 0.22733 [I 2022-10-27 13:04:46] ...optimizers/trijoint.py.151: total_norm: 29.135622 [I 2022-10-27 13:04:51] ...trap/engines/engine.py.224: train: epoch 68 | batch 1890/2304 [I 2022-10-27 13:04:51] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:03:50 [I 2022-10-27 13:04:51] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00244 [I 2022-10-27 13:04:51] ...trap/engines/engine.py.230: loss: 0.18057 [I 2022-10-27 13:04:51] ...optimizers/trijoint.py.151: total_norm: 17.387327 [I 2022-10-27 13:04:57] ...trap/engines/engine.py.224: train: epoch 68 | batch 1900/2304 [I 2022-10-27 13:04:57] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:45 [I 2022-10-27 13:04:57] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00244 [I 2022-10-27 13:04:57] ...trap/engines/engine.py.230: loss: 0.17160 [I 2022-10-27 13:04:57] ...optimizers/trijoint.py.151: total_norm: 12.048721 [I 2022-10-27 13:05:02] ...trap/engines/engine.py.224: train: epoch 68 | batch 1910/2304 [I 2022-10-27 13:05:02] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:39 [I 2022-10-27 13:05:02] ...trap/engines/engine.py.229: process: 0.55770 | load: 0.00242 [I 2022-10-27 13:05:02] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-27 13:05:02] ...optimizers/trijoint.py.151: total_norm: 16.829987 [I 2022-10-27 13:05:08] ...trap/engines/engine.py.224: train: epoch 68 | batch 1920/2304 [I 2022-10-27 13:05:08] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:34 [I 2022-10-27 13:05:08] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00247 [I 2022-10-27 13:05:08] ...trap/engines/engine.py.230: loss: 0.22951 [I 2022-10-27 13:05:08] ...optimizers/trijoint.py.151: total_norm: 36.469810 [I 2022-10-27 13:05:14] ...trap/engines/engine.py.224: train: epoch 68 | batch 1930/2304 [I 2022-10-27 13:05:14] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:29 [I 2022-10-27 13:05:14] ...trap/engines/engine.py.229: process: 0.55010 | load: 0.00245 [I 2022-10-27 13:05:14] ...trap/engines/engine.py.230: loss: 0.12980 [I 2022-10-27 13:05:14] ...optimizers/trijoint.py.151: total_norm: 11.469812 [I 2022-10-27 13:05:19] ...trap/engines/engine.py.224: train: epoch 68 | batch 1940/2304 [I 2022-10-27 13:05:19] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:23 [I 2022-10-27 13:05:19] ...trap/engines/engine.py.229: process: 0.56056 | load: 0.00249 [I 2022-10-27 13:05:19] ...trap/engines/engine.py.230: loss: 0.13947 [I 2022-10-27 13:05:19] ...optimizers/trijoint.py.151: total_norm: 31.028658 [I 2022-10-27 13:05:25] ...trap/engines/engine.py.224: train: epoch 68 | batch 1950/2304 [I 2022-10-27 13:05:25] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:17 [I 2022-10-27 13:05:25] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00246 [I 2022-10-27 13:05:25] ...trap/engines/engine.py.230: loss: 0.17917 [I 2022-10-27 13:05:25] ...optimizers/trijoint.py.151: total_norm: 13.352365 [I 2022-10-27 13:05:30] ...trap/engines/engine.py.224: train: epoch 68 | batch 1960/2304 [I 2022-10-27 13:05:30] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:11 [I 2022-10-27 13:05:30] ...trap/engines/engine.py.229: process: 0.54235 | load: 0.00242 [I 2022-10-27 13:05:30] ...trap/engines/engine.py.230: loss: 0.13763 [I 2022-10-27 13:05:30] ...optimizers/trijoint.py.151: total_norm: 21.837250 [I 2022-10-27 13:05:36] ...trap/engines/engine.py.224: train: epoch 68 | batch 1970/2304 [I 2022-10-27 13:05:36] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:06 [I 2022-10-27 13:05:36] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00245 [I 2022-10-27 13:05:36] ...trap/engines/engine.py.230: loss: 0.12941 [I 2022-10-27 13:05:36] ...optimizers/trijoint.py.151: total_norm: 18.320908 [I 2022-10-27 13:05:42] ...trap/engines/engine.py.224: train: epoch 68 | batch 1980/2304 [I 2022-10-27 13:05:42] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:02:58 [I 2022-10-27 13:05:42] ...trap/engines/engine.py.229: process: 0.54232 | load: 0.00246 [I 2022-10-27 13:05:42] ...trap/engines/engine.py.230: loss: 0.08470 [I 2022-10-27 13:05:42] ...optimizers/trijoint.py.151: total_norm: 16.853409 [I 2022-10-27 13:05:47] ...trap/engines/engine.py.224: train: epoch 68 | batch 1990/2304 [I 2022-10-27 13:05:47] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:55 [I 2022-10-27 13:05:47] ...trap/engines/engine.py.229: process: 0.56959 | load: 0.00242 [I 2022-10-27 13:05:47] ...trap/engines/engine.py.230: loss: 0.13547 [I 2022-10-27 13:05:47] ...optimizers/trijoint.py.151: total_norm: 15.261348 [I 2022-10-27 13:05:53] ...trap/engines/engine.py.224: train: epoch 68 | batch 2000/2304 [I 2022-10-27 13:05:53] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:02:49 [I 2022-10-27 13:05:53] ...trap/engines/engine.py.229: process: 0.54840 | load: 0.00248 [I 2022-10-27 13:05:53] ...trap/engines/engine.py.230: loss: 0.12540 [I 2022-10-27 13:05:53] ...optimizers/trijoint.py.151: total_norm: 12.393972 [I 2022-10-27 13:05:58] ...trap/engines/engine.py.224: train: epoch 68 | batch 2010/2304 [I 2022-10-27 13:05:58] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:44 [I 2022-10-27 13:05:58] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00247 [I 2022-10-27 13:05:58] ...trap/engines/engine.py.230: loss: 0.12080 [I 2022-10-27 13:05:58] ...optimizers/trijoint.py.151: total_norm: 8.410464 [I 2022-10-27 13:06:04] ...trap/engines/engine.py.224: train: epoch 68 | batch 2020/2304 [I 2022-10-27 13:06:04] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:42 [I 2022-10-27 13:06:04] ...trap/engines/engine.py.229: process: 0.58713 | load: 0.00241 [I 2022-10-27 13:06:04] ...trap/engines/engine.py.230: loss: 0.10816 [I 2022-10-27 13:06:04] ...optimizers/trijoint.py.151: total_norm: 9.912218 [I 2022-10-27 13:06:10] ...trap/engines/engine.py.224: train: epoch 68 | batch 2030/2304 [I 2022-10-27 13:06:10] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:33 [I 2022-10-27 13:06:10] ...trap/engines/engine.py.229: process: 0.55071 | load: 0.00241 [I 2022-10-27 13:06:10] ...trap/engines/engine.py.230: loss: 0.12255 [I 2022-10-27 13:06:10] ...optimizers/trijoint.py.151: total_norm: 10.324035 [I 2022-10-27 13:06:15] ...trap/engines/engine.py.224: train: epoch 68 | batch 2040/2304 [I 2022-10-27 13:06:15] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:28 [I 2022-10-27 13:06:15] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00242 [I 2022-10-27 13:06:15] ...trap/engines/engine.py.230: loss: 0.15430 [I 2022-10-27 13:06:15] ...optimizers/trijoint.py.151: total_norm: 13.921987 [I 2022-10-27 13:06:21] ...trap/engines/engine.py.224: train: epoch 68 | batch 2050/2304 [I 2022-10-27 13:06:21] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:21 [I 2022-10-27 13:06:21] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00241 [I 2022-10-27 13:06:21] ...trap/engines/engine.py.230: loss: 0.12050 [I 2022-10-27 13:06:21] ...optimizers/trijoint.py.151: total_norm: 11.125366 [I 2022-10-27 13:06:27] ...trap/engines/engine.py.224: train: epoch 68 | batch 2060/2304 [I 2022-10-27 13:06:27] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:16 [I 2022-10-27 13:06:27] ...trap/engines/engine.py.229: process: 0.56448 | load: 0.00247 [I 2022-10-27 13:06:27] ...trap/engines/engine.py.230: loss: 0.09960 [I 2022-10-27 13:06:27] ...optimizers/trijoint.py.151: total_norm: 11.988297 [I 2022-10-27 13:06:32] ...trap/engines/engine.py.224: train: epoch 68 | batch 2070/2304 [I 2022-10-27 13:06:32] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:10 [I 2022-10-27 13:06:32] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00243 [I 2022-10-27 13:06:32] ...trap/engines/engine.py.230: loss: 0.13296 [I 2022-10-27 13:06:32] ...optimizers/trijoint.py.151: total_norm: 11.512839 [I 2022-10-27 13:06:38] ...trap/engines/engine.py.224: train: epoch 68 | batch 2080/2304 [I 2022-10-27 13:06:38] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:04 [I 2022-10-27 13:06:38] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00239 [I 2022-10-27 13:06:38] ...trap/engines/engine.py.230: loss: 0.17834 [I 2022-10-27 13:06:38] ...optimizers/trijoint.py.151: total_norm: 25.244720 [I 2022-10-27 13:06:43] ...trap/engines/engine.py.224: train: epoch 68 | batch 2090/2304 [I 2022-10-27 13:06:43] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:01:58 [I 2022-10-27 13:06:43] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00242 [I 2022-10-27 13:06:43] ...trap/engines/engine.py.230: loss: 0.11628 [I 2022-10-27 13:06:43] ...optimizers/trijoint.py.151: total_norm: 15.142509 [I 2022-10-27 13:06:49] ...trap/engines/engine.py.224: train: epoch 68 | batch 2100/2304 [I 2022-10-27 13:06:49] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:54 [I 2022-10-27 13:06:49] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00262 [I 2022-10-27 13:06:49] ...trap/engines/engine.py.230: loss: 0.15738 [I 2022-10-27 13:06:49] ...optimizers/trijoint.py.151: total_norm: 20.004786 [I 2022-10-27 13:06:55] ...trap/engines/engine.py.224: train: epoch 68 | batch 2110/2304 [I 2022-10-27 13:06:55] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:01:52 [I 2022-10-27 13:06:55] ...trap/engines/engine.py.229: process: 0.56246 | load: 0.00243 [I 2022-10-27 13:06:55] ...trap/engines/engine.py.230: loss: 0.22708 [I 2022-10-27 13:06:55] ...optimizers/trijoint.py.151: total_norm: 23.218540 [I 2022-10-27 13:07:00] ...trap/engines/engine.py.224: train: epoch 68 | batch 2120/2304 [I 2022-10-27 13:07:00] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:42 [I 2022-10-27 13:07:00] ...trap/engines/engine.py.229: process: 0.56548 | load: 0.00238 [I 2022-10-27 13:07:00] ...trap/engines/engine.py.230: loss: 0.13364 [I 2022-10-27 13:07:00] ...optimizers/trijoint.py.151: total_norm: 21.296635 [I 2022-10-27 13:07:06] ...trap/engines/engine.py.224: train: epoch 68 | batch 2130/2304 [I 2022-10-27 13:07:06] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:37 [I 2022-10-27 13:07:06] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00243 [I 2022-10-27 13:07:06] ...trap/engines/engine.py.230: loss: 0.11610 [I 2022-10-27 13:07:06] ...optimizers/trijoint.py.151: total_norm: 11.276692 [I 2022-10-27 13:07:11] ...trap/engines/engine.py.224: train: epoch 68 | batch 2140/2304 [I 2022-10-27 13:07:11] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:31 [I 2022-10-27 13:07:11] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00247 [I 2022-10-27 13:07:11] ...trap/engines/engine.py.230: loss: 0.12401 [I 2022-10-27 13:07:11] ...optimizers/trijoint.py.151: total_norm: 22.475967 [I 2022-10-27 13:07:17] ...trap/engines/engine.py.224: train: epoch 68 | batch 2150/2304 [I 2022-10-27 13:07:17] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:26 [I 2022-10-27 13:07:17] ...trap/engines/engine.py.229: process: 0.56462 | load: 0.00246 [I 2022-10-27 13:07:17] ...trap/engines/engine.py.230: loss: 0.12354 [I 2022-10-27 13:07:17] ...optimizers/trijoint.py.151: total_norm: 16.597609 [I 2022-10-27 13:07:23] ...trap/engines/engine.py.224: train: epoch 68 | batch 2160/2304 [I 2022-10-27 13:07:23] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:20 [I 2022-10-27 13:07:23] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00244 [I 2022-10-27 13:07:23] ...trap/engines/engine.py.230: loss: 0.18448 [I 2022-10-27 13:07:23] ...optimizers/trijoint.py.151: total_norm: 23.368511 [I 2022-10-27 13:07:28] ...trap/engines/engine.py.224: train: epoch 68 | batch 2170/2304 [I 2022-10-27 13:07:28] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:18 [I 2022-10-27 13:07:28] ...trap/engines/engine.py.229: process: 0.66945 | load: 0.00256 [I 2022-10-27 13:07:28] ...trap/engines/engine.py.230: loss: 0.18139 [I 2022-10-27 13:07:28] ...optimizers/trijoint.py.151: total_norm: 14.458245 [I 2022-10-27 13:07:34] ...trap/engines/engine.py.224: train: epoch 68 | batch 2180/2304 [I 2022-10-27 13:07:34] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:09 [I 2022-10-27 13:07:34] ...trap/engines/engine.py.229: process: 0.54471 | load: 0.00247 [I 2022-10-27 13:07:34] ...trap/engines/engine.py.230: loss: 0.12636 [I 2022-10-27 13:07:34] ...optimizers/trijoint.py.151: total_norm: 14.356356 [I 2022-10-27 13:07:39] ...trap/engines/engine.py.224: train: epoch 68 | batch 2190/2304 [I 2022-10-27 13:07:39] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:03 [I 2022-10-27 13:07:39] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00244 [I 2022-10-27 13:07:39] ...trap/engines/engine.py.230: loss: 0.13252 [I 2022-10-27 13:07:39] ...optimizers/trijoint.py.151: total_norm: 18.368023 [I 2022-10-27 13:07:45] ...trap/engines/engine.py.224: train: epoch 68 | batch 2200/2304 [I 2022-10-27 13:07:45] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:57 [I 2022-10-27 13:07:45] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00247 [I 2022-10-27 13:07:45] ...trap/engines/engine.py.230: loss: 0.09357 [I 2022-10-27 13:07:45] ...optimizers/trijoint.py.151: total_norm: 7.304218 [I 2022-10-27 13:07:51] ...trap/engines/engine.py.224: train: epoch 68 | batch 2210/2304 [I 2022-10-27 13:07:51] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:00:52 [I 2022-10-27 13:07:51] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00242 [I 2022-10-27 13:07:51] ...trap/engines/engine.py.230: loss: 0.17293 [I 2022-10-27 13:07:51] ...optimizers/trijoint.py.151: total_norm: 9.577077 [I 2022-10-27 13:07:56] ...trap/engines/engine.py.224: train: epoch 68 | batch 2220/2304 [I 2022-10-27 13:07:56] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:00:47 [I 2022-10-27 13:07:56] ...trap/engines/engine.py.229: process: 0.57099 | load: 0.00241 [I 2022-10-27 13:07:56] ...trap/engines/engine.py.230: loss: 0.10290 [I 2022-10-27 13:07:56] ...optimizers/trijoint.py.151: total_norm: 8.343553 [I 2022-10-27 13:08:02] ...trap/engines/engine.py.224: train: epoch 68 | batch 2230/2304 [I 2022-10-27 13:08:02] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:41 [I 2022-10-27 13:08:02] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00244 [I 2022-10-27 13:08:02] ...trap/engines/engine.py.230: loss: 0.15916 [I 2022-10-27 13:08:02] ...optimizers/trijoint.py.151: total_norm: 22.719889 [I 2022-10-27 13:08:07] ...trap/engines/engine.py.224: train: epoch 68 | batch 2240/2304 [I 2022-10-27 13:08:07] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:35 [I 2022-10-27 13:08:07] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00245 [I 2022-10-27 13:08:07] ...trap/engines/engine.py.230: loss: 0.15310 [I 2022-10-27 13:08:07] ...optimizers/trijoint.py.151: total_norm: 19.047882 [I 2022-10-27 13:08:13] ...trap/engines/engine.py.224: train: epoch 68 | batch 2250/2304 [I 2022-10-27 13:08:13] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:30 [I 2022-10-27 13:08:13] ...trap/engines/engine.py.229: process: 0.57165 | load: 0.00245 [I 2022-10-27 13:08:13] ...trap/engines/engine.py.230: loss: 0.12361 [I 2022-10-27 13:08:13] ...optimizers/trijoint.py.151: total_norm: 23.304251 [I 2022-10-27 13:08:19] ...trap/engines/engine.py.224: train: epoch 68 | batch 2260/2304 [I 2022-10-27 13:08:19] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:24 [I 2022-10-27 13:08:19] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00244 [I 2022-10-27 13:08:19] ...trap/engines/engine.py.230: loss: 0.13006 [I 2022-10-27 13:08:19] ...optimizers/trijoint.py.151: total_norm: 20.399799 [I 2022-10-27 13:08:24] ...trap/engines/engine.py.224: train: epoch 68 | batch 2270/2304 [I 2022-10-27 13:08:24] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:19 [I 2022-10-27 13:08:24] ...trap/engines/engine.py.229: process: 0.56067 | load: 0.00246 [I 2022-10-27 13:08:24] ...trap/engines/engine.py.230: loss: 0.16780 [I 2022-10-27 13:08:24] ...optimizers/trijoint.py.151: total_norm: 18.809271 [I 2022-10-27 13:08:30] ...trap/engines/engine.py.224: train: epoch 68 | batch 2280/2304 [I 2022-10-27 13:08:30] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:13 [I 2022-10-27 13:08:30] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00247 [I 2022-10-27 13:08:30] ...trap/engines/engine.py.230: loss: 0.17884 [I 2022-10-27 13:08:30] ...optimizers/trijoint.py.151: total_norm: 22.285847 [I 2022-10-27 13:08:35] ...trap/engines/engine.py.224: train: epoch 68 | batch 2290/2304 [I 2022-10-27 13:08:35] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:07 [I 2022-10-27 13:08:35] ...trap/engines/engine.py.229: process: 0.55299 | load: 0.00243 [I 2022-10-27 13:08:35] ...trap/engines/engine.py.230: loss: 0.12936 [I 2022-10-27 13:08:35] ...optimizers/trijoint.py.151: total_norm: 19.304955 [I 2022-10-27 13:08:41] ...trap/engines/engine.py.224: train: epoch 68 | batch 2300/2304 [I 2022-10-27 13:08:41] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:02 [I 2022-10-27 13:08:41] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00015 [I 2022-10-27 13:08:41] ...trap/engines/engine.py.230: loss: 0.11086 [I 2022-10-27 13:08:41] ...optimizers/trijoint.py.151: total_norm: 9.320852 [I 2022-10-27 13:08:43] ...trap/engines/engine.py.224: train: epoch 68 | batch 2304/2304 [I 2022-10-27 13:08:43] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:00 [I 2022-10-27 13:08:43] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00015 [I 2022-10-27 13:08:43] ...trap/engines/engine.py.230: loss: 0.10633 [I 2022-10-27 13:08:43] ...optimizers/trijoint.py.151: total_norm: 16.988100 [S 2022-10-27 13:08:44] ...trap/engines/engine.py.129: train_epoch.epoch: 68 [S 2022-10-27 13:08:44] ...trap/engines/engine.py.129: train_epoch.loss: 0.1467906342928425 [I 2022-10-27 13:08:44] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 13:08:45] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 68 [I 2022-10-27 13:08:45] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 13:08:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 13:08:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 13:08:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 13:08:50] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 13:08:50] ...trap/engines/engine.py.321: eval: epoch 68 | batch 0/510 [I 2022-10-27 13:08:50] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:39:02 [I 2022-10-27 13:08:50] ...trap/engines/engine.py.326: process: 4.59304 | load: 4.19811 [I 2022-10-27 13:08:53] ...trap/engines/engine.py.321: eval: epoch 68 | batch 10/510 [I 2022-10-27 13:08:53] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:35 [I 2022-10-27 13:08:53] ...trap/engines/engine.py.326: process: 0.33605 | load: 0.00018 [I 2022-10-27 13:08:57] ...trap/engines/engine.py.321: eval: epoch 68 | batch 20/510 [I 2022-10-27 13:08:57] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:06 [I 2022-10-27 13:08:57] ...trap/engines/engine.py.326: process: 0.32459 | load: 0.00016 [I 2022-10-27 13:09:00] ...trap/engines/engine.py.321: eval: epoch 68 | batch 30/510 [I 2022-10-27 13:09:00] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:42 [I 2022-10-27 13:09:00] ...trap/engines/engine.py.326: process: 0.32756 | load: 0.00018 [I 2022-10-27 13:09:03] ...trap/engines/engine.py.321: eval: epoch 68 | batch 40/510 [I 2022-10-27 13:09:03] ...trap/engines/engine.py.325: elapsed: 0:00:17 | left: 0:02:36 [I 2022-10-27 13:09:03] ...trap/engines/engine.py.326: process: 0.33045 | load: 0.00018 [I 2022-10-27 13:09:07] ...trap/engines/engine.py.321: eval: epoch 68 | batch 50/510 [I 2022-10-27 13:09:07] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:32 [I 2022-10-27 13:09:07] ...trap/engines/engine.py.326: process: 0.33074 | load: 0.00019 [I 2022-10-27 13:09:10] ...trap/engines/engine.py.321: eval: epoch 68 | batch 60/510 [I 2022-10-27 13:09:10] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:31 [I 2022-10-27 13:09:10] ...trap/engines/engine.py.326: process: 0.32764 | load: 0.00016 [I 2022-10-27 13:09:13] ...trap/engines/engine.py.321: eval: epoch 68 | batch 70/510 [I 2022-10-27 13:09:13] ...trap/engines/engine.py.325: elapsed: 0:00:27 | left: 0:02:27 [I 2022-10-27 13:09:13] ...trap/engines/engine.py.326: process: 0.33284 | load: 0.00021 [I 2022-10-27 13:09:17] ...trap/engines/engine.py.321: eval: epoch 68 | batch 80/510 [I 2022-10-27 13:09:17] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:23 [I 2022-10-27 13:09:17] ...trap/engines/engine.py.326: process: 0.32483 | load: 0.00017 [I 2022-10-27 13:09:20] ...trap/engines/engine.py.321: eval: epoch 68 | batch 90/510 [I 2022-10-27 13:09:20] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:20 [I 2022-10-27 13:09:20] ...trap/engines/engine.py.326: process: 0.32644 | load: 0.00018 [I 2022-10-27 13:09:23] ...trap/engines/engine.py.321: eval: epoch 68 | batch 100/510 [I 2022-10-27 13:09:23] ...trap/engines/engine.py.325: elapsed: 0:00:37 | left: 0:02:14 [I 2022-10-27 13:09:23] ...trap/engines/engine.py.326: process: 0.28753 | load: 0.00016 [I 2022-10-27 13:09:26] ...trap/engines/engine.py.321: eval: epoch 68 | batch 110/510 [I 2022-10-27 13:09:26] ...trap/engines/engine.py.325: elapsed: 0:00:40 | left: 0:01:57 [I 2022-10-27 13:09:26] ...trap/engines/engine.py.326: process: 0.28660 | load: 0.00016 [I 2022-10-27 13:09:29] ...trap/engines/engine.py.321: eval: epoch 68 | batch 120/510 [I 2022-10-27 13:09:29] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:53 [I 2022-10-27 13:09:29] ...trap/engines/engine.py.326: process: 0.28901 | load: 0.00021 [I 2022-10-27 13:09:32] ...trap/engines/engine.py.321: eval: epoch 68 | batch 130/510 [I 2022-10-27 13:09:32] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:52 [I 2022-10-27 13:09:32] ...trap/engines/engine.py.326: process: 0.30826 | load: 0.00016 [I 2022-10-27 13:09:35] ...trap/engines/engine.py.321: eval: epoch 68 | batch 140/510 [I 2022-10-27 13:09:35] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 13:09:35] ...trap/engines/engine.py.326: process: 0.29054 | load: 0.00017 [I 2022-10-27 13:09:38] ...trap/engines/engine.py.321: eval: epoch 68 | batch 150/510 [I 2022-10-27 13:09:38] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 13:09:38] ...trap/engines/engine.py.326: process: 0.28197 | load: 0.00014 [I 2022-10-27 13:09:41] ...trap/engines/engine.py.321: eval: epoch 68 | batch 160/510 [I 2022-10-27 13:09:41] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:46 [I 2022-10-27 13:09:41] ...trap/engines/engine.py.326: process: 0.29260 | load: 0.00017 [I 2022-10-27 13:09:44] ...trap/engines/engine.py.321: eval: epoch 68 | batch 170/510 [I 2022-10-27 13:09:44] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:39 [I 2022-10-27 13:09:44] ...trap/engines/engine.py.326: process: 0.29272 | load: 0.00015 [I 2022-10-27 13:09:47] ...trap/engines/engine.py.321: eval: epoch 68 | batch 180/510 [I 2022-10-27 13:09:47] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 13:09:47] ...trap/engines/engine.py.326: process: 0.28905 | load: 0.00017 [I 2022-10-27 13:09:50] ...trap/engines/engine.py.321: eval: epoch 68 | batch 190/510 [I 2022-10-27 13:09:50] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 13:09:50] ...trap/engines/engine.py.326: process: 0.28748 | load: 0.00019 [I 2022-10-27 13:09:53] ...trap/engines/engine.py.321: eval: epoch 68 | batch 200/510 [I 2022-10-27 13:09:53] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 13:09:53] ...trap/engines/engine.py.326: process: 0.28903 | load: 0.00021 [I 2022-10-27 13:09:55] ...trap/engines/engine.py.321: eval: epoch 68 | batch 210/510 [I 2022-10-27 13:09:55] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:27 [I 2022-10-27 13:09:55] ...trap/engines/engine.py.326: process: 0.28875 | load: 0.00022 [I 2022-10-27 13:09:58] ...trap/engines/engine.py.321: eval: epoch 68 | batch 220/510 [I 2022-10-27 13:09:58] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:25 [I 2022-10-27 13:09:58] ...trap/engines/engine.py.326: process: 0.29486 | load: 0.00019 [I 2022-10-27 13:10:01] ...trap/engines/engine.py.321: eval: epoch 68 | batch 230/510 [I 2022-10-27 13:10:01] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:22 [I 2022-10-27 13:10:01] ...trap/engines/engine.py.326: process: 0.29284 | load: 0.00023 [I 2022-10-27 13:10:04] ...trap/engines/engine.py.321: eval: epoch 68 | batch 240/510 [I 2022-10-27 13:10:04] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 13:10:04] ...trap/engines/engine.py.326: process: 0.29447 | load: 0.00015 [I 2022-10-27 13:10:07] ...trap/engines/engine.py.321: eval: epoch 68 | batch 250/510 [I 2022-10-27 13:10:07] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 13:10:07] ...trap/engines/engine.py.326: process: 0.28767 | load: 0.00021 [I 2022-10-27 13:10:10] ...trap/engines/engine.py.321: eval: epoch 68 | batch 260/510 [I 2022-10-27 13:10:10] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 13:10:10] ...trap/engines/engine.py.326: process: 0.28230 | load: 0.00020 [I 2022-10-27 13:10:13] ...trap/engines/engine.py.321: eval: epoch 68 | batch 270/510 [I 2022-10-27 13:10:13] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:10 [I 2022-10-27 13:10:13] ...trap/engines/engine.py.326: process: 0.28879 | load: 0.00022 [I 2022-10-27 13:10:16] ...trap/engines/engine.py.321: eval: epoch 68 | batch 280/510 [I 2022-10-27 13:10:16] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 13:10:16] ...trap/engines/engine.py.326: process: 0.28521 | load: 0.00015 [I 2022-10-27 13:10:19] ...trap/engines/engine.py.321: eval: epoch 68 | batch 290/510 [I 2022-10-27 13:10:19] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 13:10:19] ...trap/engines/engine.py.326: process: 0.28552 | load: 0.00022 [I 2022-10-27 13:10:22] ...trap/engines/engine.py.321: eval: epoch 68 | batch 300/510 [I 2022-10-27 13:10:22] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 13:10:22] ...trap/engines/engine.py.326: process: 0.28876 | load: 0.00020 [I 2022-10-27 13:10:25] ...trap/engines/engine.py.321: eval: epoch 68 | batch 310/510 [I 2022-10-27 13:10:25] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 13:10:25] ...trap/engines/engine.py.326: process: 0.28897 | load: 0.00020 [I 2022-10-27 13:10:27] ...trap/engines/engine.py.321: eval: epoch 68 | batch 320/510 [I 2022-10-27 13:10:27] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:55 [I 2022-10-27 13:10:27] ...trap/engines/engine.py.326: process: 0.28967 | load: 0.00020 [I 2022-10-27 13:10:30] ...trap/engines/engine.py.321: eval: epoch 68 | batch 330/510 [I 2022-10-27 13:10:30] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-27 13:10:30] ...trap/engines/engine.py.326: process: 0.29244 | load: 0.00015 [I 2022-10-27 13:10:33] ...trap/engines/engine.py.321: eval: epoch 68 | batch 340/510 [I 2022-10-27 13:10:33] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 13:10:33] ...trap/engines/engine.py.326: process: 0.28420 | load: 0.00017 [I 2022-10-27 13:10:36] ...trap/engines/engine.py.321: eval: epoch 68 | batch 350/510 [I 2022-10-27 13:10:36] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 13:10:36] ...trap/engines/engine.py.326: process: 0.28811 | load: 0.00015 [I 2022-10-27 13:10:39] ...trap/engines/engine.py.321: eval: epoch 68 | batch 360/510 [I 2022-10-27 13:10:39] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 13:10:39] ...trap/engines/engine.py.326: process: 0.29500 | load: 0.00018 [I 2022-10-27 13:10:42] ...trap/engines/engine.py.321: eval: epoch 68 | batch 370/510 [I 2022-10-27 13:10:42] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 13:10:42] ...trap/engines/engine.py.326: process: 0.28672 | load: 0.00020 [I 2022-10-27 13:10:45] ...trap/engines/engine.py.321: eval: epoch 68 | batch 380/510 [I 2022-10-27 13:10:45] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 13:10:45] ...trap/engines/engine.py.326: process: 0.28981 | load: 0.00022 [I 2022-10-27 13:10:48] ...trap/engines/engine.py.321: eval: epoch 68 | batch 390/510 [I 2022-10-27 13:10:48] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:35 [I 2022-10-27 13:10:48] ...trap/engines/engine.py.326: process: 0.28985 | load: 0.00023 [I 2022-10-27 13:10:51] ...trap/engines/engine.py.321: eval: epoch 68 | batch 400/510 [I 2022-10-27 13:10:51] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 13:10:51] ...trap/engines/engine.py.326: process: 0.28548 | load: 0.00015 [I 2022-10-27 13:10:54] ...trap/engines/engine.py.321: eval: epoch 68 | batch 410/510 [I 2022-10-27 13:10:54] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:29 [I 2022-10-27 13:10:54] ...trap/engines/engine.py.326: process: 0.29059 | load: 0.00021 [I 2022-10-27 13:10:57] ...trap/engines/engine.py.321: eval: epoch 68 | batch 420/510 [I 2022-10-27 13:10:57] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-27 13:10:57] ...trap/engines/engine.py.326: process: 0.28552 | load: 0.00014 [I 2022-10-27 13:10:59] ...trap/engines/engine.py.321: eval: epoch 68 | batch 430/510 [I 2022-10-27 13:10:59] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 13:10:59] ...trap/engines/engine.py.326: process: 0.28868 | load: 0.00020 [I 2022-10-27 13:11:02] ...trap/engines/engine.py.321: eval: epoch 68 | batch 440/510 [I 2022-10-27 13:11:02] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 13:11:02] ...trap/engines/engine.py.326: process: 0.29280 | load: 0.00015 [I 2022-10-27 13:11:05] ...trap/engines/engine.py.321: eval: epoch 68 | batch 450/510 [I 2022-10-27 13:11:05] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 13:11:05] ...trap/engines/engine.py.326: process: 0.29030 | load: 0.00023 [I 2022-10-27 13:11:08] ...trap/engines/engine.py.321: eval: epoch 68 | batch 460/510 [I 2022-10-27 13:11:08] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 13:11:08] ...trap/engines/engine.py.326: process: 0.28775 | load: 0.00016 [I 2022-10-27 13:11:11] ...trap/engines/engine.py.321: eval: epoch 68 | batch 470/510 [I 2022-10-27 13:11:11] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 13:11:11] ...trap/engines/engine.py.326: process: 0.29186 | load: 0.00021 [I 2022-10-27 13:11:14] ...trap/engines/engine.py.321: eval: epoch 68 | batch 480/510 [I 2022-10-27 13:11:14] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 13:11:14] ...trap/engines/engine.py.326: process: 0.29191 | load: 0.00016 [I 2022-10-27 13:11:17] ...trap/engines/engine.py.321: eval: epoch 68 | batch 490/510 [I 2022-10-27 13:11:17] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 13:11:17] ...trap/engines/engine.py.326: process: 0.28231 | load: 0.00014 [I 2022-10-27 13:11:20] ...trap/engines/engine.py.321: eval: epoch 68 | batch 500/510 [I 2022-10-27 13:11:20] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 13:11:20] ...trap/engines/engine.py.326: process: 0.28048 | load: 0.00016 [I 2022-10-27 13:11:23] ...trap/engines/engine.py.321: eval: epoch 68 | batch 510/510 [I 2022-10-27 13:11:23] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 13:11:23] ...trap/engines/engine.py.326: process: 0.29000 | load: 0.00011 [S 2022-10-27 13:11:23] ...trap/engines/engine.py.132: eval_epoch.epoch: 68 [I 2022-10-27 13:11:23] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 13:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 13:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 13:11:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.712500 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.902700 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936500 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.704600 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.899800 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933600 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013231 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.012705 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011395 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014800 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012773 [S 2022-10-27 13:11:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009902 [I 2022-10-27 13:11:29] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 13:11:29] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 13:11:29] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 13:11:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 13:11:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 13:11:33] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 13:11:34] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 13:11:40] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 13:11:40] ...trap/engines/engine.py.163: Training model on trainset for epoch 69 [I 2022-10-27 13:11:40] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 13:11:42] ...trap/engines/engine.py.224: train: epoch 69 | batch 0/2304 [I 2022-10-27 13:11:42] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:52 [I 2022-10-27 13:11:42] ...trap/engines/engine.py.229: process: 2.70529 | load: 2.07678 [I 2022-10-27 13:11:42] ...trap/engines/engine.py.230: loss: 0.10737 [I 2022-10-27 13:11:42] ...optimizers/trijoint.py.151: total_norm: 12.340737 [I 2022-10-27 13:11:48] ...trap/engines/engine.py.224: train: epoch 69 | batch 10/2304 [I 2022-10-27 13:11:48] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:04 [I 2022-10-27 13:11:48] ...trap/engines/engine.py.229: process: 0.55269 | load: 0.00423 [I 2022-10-27 13:11:48] ...trap/engines/engine.py.230: loss: 0.10111 [I 2022-10-27 13:11:48] ...optimizers/trijoint.py.151: total_norm: 33.377319 [I 2022-10-27 13:11:54] ...trap/engines/engine.py.224: train: epoch 69 | batch 20/2304 [I 2022-10-27 13:11:54] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:08 [I 2022-10-27 13:11:54] ...trap/engines/engine.py.229: process: 0.55030 | load: 0.00294 [I 2022-10-27 13:11:54] ...trap/engines/engine.py.230: loss: 0.11162 [I 2022-10-27 13:11:54] ...optimizers/trijoint.py.151: total_norm: 11.497607 [I 2022-10-27 13:11:59] ...trap/engines/engine.py.224: train: epoch 69 | batch 30/2304 [I 2022-10-27 13:11:59] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:11 [I 2022-10-27 13:11:59] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00285 [I 2022-10-27 13:11:59] ...trap/engines/engine.py.230: loss: 0.11450 [I 2022-10-27 13:11:59] ...optimizers/trijoint.py.151: total_norm: 17.414749 [I 2022-10-27 13:12:05] ...trap/engines/engine.py.224: train: epoch 69 | batch 40/2304 [I 2022-10-27 13:12:05] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:01 [I 2022-10-27 13:12:05] ...trap/engines/engine.py.229: process: 0.55941 | load: 0.00312 [I 2022-10-27 13:12:05] ...trap/engines/engine.py.230: loss: 0.14086 [I 2022-10-27 13:12:05] ...optimizers/trijoint.py.151: total_norm: 9.746889 [I 2022-10-27 13:12:10] ...trap/engines/engine.py.224: train: epoch 69 | batch 50/2304 [I 2022-10-27 13:12:10] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:50 [I 2022-10-27 13:12:10] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00335 [I 2022-10-27 13:12:10] ...trap/engines/engine.py.230: loss: 0.09630 [I 2022-10-27 13:12:10] ...optimizers/trijoint.py.151: total_norm: 15.065040 [I 2022-10-27 13:12:16] ...trap/engines/engine.py.224: train: epoch 69 | batch 60/2304 [I 2022-10-27 13:12:16] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:03 [I 2022-10-27 13:12:16] ...trap/engines/engine.py.229: process: 0.56582 | load: 0.00258 [I 2022-10-27 13:12:16] ...trap/engines/engine.py.230: loss: 0.11910 [I 2022-10-27 13:12:16] ...optimizers/trijoint.py.151: total_norm: 13.570904 [I 2022-10-27 13:12:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 70/2304 [I 2022-10-27 13:12:22] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:53 [I 2022-10-27 13:12:22] ...trap/engines/engine.py.229: process: 0.56673 | load: 0.00276 [I 2022-10-27 13:12:22] ...trap/engines/engine.py.230: loss: 0.10238 [I 2022-10-27 13:12:22] ...optimizers/trijoint.py.151: total_norm: 9.732166 [I 2022-10-27 13:12:27] ...trap/engines/engine.py.224: train: epoch 69 | batch 80/2304 [I 2022-10-27 13:12:27] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:42 [I 2022-10-27 13:12:27] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00278 [I 2022-10-27 13:12:27] ...trap/engines/engine.py.230: loss: 0.11639 [I 2022-10-27 13:12:27] ...optimizers/trijoint.py.151: total_norm: 12.916510 [I 2022-10-27 13:12:33] ...trap/engines/engine.py.224: train: epoch 69 | batch 90/2304 [I 2022-10-27 13:12:33] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:33 [I 2022-10-27 13:12:33] ...trap/engines/engine.py.229: process: 0.54733 | load: 0.00270 [I 2022-10-27 13:12:33] ...trap/engines/engine.py.230: loss: 0.10410 [I 2022-10-27 13:12:33] ...optimizers/trijoint.py.151: total_norm: 9.614758 [I 2022-10-27 13:12:38] ...trap/engines/engine.py.224: train: epoch 69 | batch 100/2304 [I 2022-10-27 13:12:38] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:13 [I 2022-10-27 13:12:38] ...trap/engines/engine.py.229: process: 0.54318 | load: 0.00270 [I 2022-10-27 13:12:38] ...trap/engines/engine.py.230: loss: 0.18181 [I 2022-10-27 13:12:38] ...optimizers/trijoint.py.151: total_norm: 20.678690 [I 2022-10-27 13:12:44] ...trap/engines/engine.py.224: train: epoch 69 | batch 110/2304 [I 2022-10-27 13:12:44] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:24 [I 2022-10-27 13:12:44] ...trap/engines/engine.py.229: process: 0.56773 | load: 0.00260 [I 2022-10-27 13:12:44] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-27 13:12:44] ...optimizers/trijoint.py.151: total_norm: 18.761488 [I 2022-10-27 13:12:50] ...trap/engines/engine.py.224: train: epoch 69 | batch 120/2304 [I 2022-10-27 13:12:50] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:17 [I 2022-10-27 13:12:50] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00267 [I 2022-10-27 13:12:50] ...trap/engines/engine.py.230: loss: 0.10757 [I 2022-10-27 13:12:50] ...optimizers/trijoint.py.151: total_norm: 9.169078 [I 2022-10-27 13:12:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 130/2304 [I 2022-10-27 13:12:55] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:05 [I 2022-10-27 13:12:55] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00254 [I 2022-10-27 13:12:55] ...trap/engines/engine.py.230: loss: 0.12875 [I 2022-10-27 13:12:55] ...optimizers/trijoint.py.151: total_norm: 11.380373 [I 2022-10-27 13:13:01] ...trap/engines/engine.py.224: train: epoch 69 | batch 140/2304 [I 2022-10-27 13:13:01] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:58 [I 2022-10-27 13:13:01] ...trap/engines/engine.py.229: process: 0.54742 | load: 0.00260 [I 2022-10-27 13:13:01] ...trap/engines/engine.py.230: loss: 0.11656 [I 2022-10-27 13:13:01] ...optimizers/trijoint.py.151: total_norm: 11.835680 [I 2022-10-27 13:13:06] ...trap/engines/engine.py.224: train: epoch 69 | batch 150/2304 [I 2022-10-27 13:13:06] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:56 [I 2022-10-27 13:13:06] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00265 [I 2022-10-27 13:13:06] ...trap/engines/engine.py.230: loss: 0.16014 [I 2022-10-27 13:13:06] ...optimizers/trijoint.py.151: total_norm: 10.059242 [I 2022-10-27 13:13:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 160/2304 [I 2022-10-27 13:13:12] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:46 [I 2022-10-27 13:13:12] ...trap/engines/engine.py.229: process: 0.56353 | load: 0.00252 [I 2022-10-27 13:13:12] ...trap/engines/engine.py.230: loss: 0.15690 [I 2022-10-27 13:13:12] ...optimizers/trijoint.py.151: total_norm: 34.955215 [I 2022-10-27 13:13:17] ...trap/engines/engine.py.224: train: epoch 69 | batch 170/2304 [I 2022-10-27 13:13:17] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:44 [I 2022-10-27 13:13:17] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00510 [I 2022-10-27 13:13:17] ...trap/engines/engine.py.230: loss: 0.09844 [I 2022-10-27 13:13:17] ...optimizers/trijoint.py.151: total_norm: 18.605755 [I 2022-10-27 13:13:23] ...trap/engines/engine.py.224: train: epoch 69 | batch 180/2304 [I 2022-10-27 13:13:23] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:44 [I 2022-10-27 13:13:23] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00247 [I 2022-10-27 13:13:23] ...trap/engines/engine.py.230: loss: 0.11375 [I 2022-10-27 13:13:23] ...optimizers/trijoint.py.151: total_norm: 8.335968 [I 2022-10-27 13:13:28] ...trap/engines/engine.py.224: train: epoch 69 | batch 190/2304 [I 2022-10-27 13:13:28] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:34 [I 2022-10-27 13:13:28] ...trap/engines/engine.py.229: process: 0.55605 | load: 0.00242 [I 2022-10-27 13:13:28] ...trap/engines/engine.py.230: loss: 0.12574 [I 2022-10-27 13:13:28] ...optimizers/trijoint.py.151: total_norm: 22.152142 [I 2022-10-27 13:13:34] ...trap/engines/engine.py.224: train: epoch 69 | batch 200/2304 [I 2022-10-27 13:13:34] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:29 [I 2022-10-27 13:13:34] ...trap/engines/engine.py.229: process: 0.54559 | load: 0.00257 [I 2022-10-27 13:13:34] ...trap/engines/engine.py.230: loss: 0.19659 [I 2022-10-27 13:13:34] ...optimizers/trijoint.py.151: total_norm: 11.866407 [I 2022-10-27 13:13:40] ...trap/engines/engine.py.224: train: epoch 69 | batch 210/2304 [I 2022-10-27 13:13:40] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:11 [I 2022-10-27 13:13:40] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00248 [I 2022-10-27 13:13:40] ...trap/engines/engine.py.230: loss: 0.13433 [I 2022-10-27 13:13:40] ...optimizers/trijoint.py.151: total_norm: 10.640943 [I 2022-10-27 13:13:45] ...trap/engines/engine.py.224: train: epoch 69 | batch 220/2304 [I 2022-10-27 13:13:45] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:22 [I 2022-10-27 13:13:45] ...trap/engines/engine.py.229: process: 0.56703 | load: 0.00249 [I 2022-10-27 13:13:45] ...trap/engines/engine.py.230: loss: 0.10662 [I 2022-10-27 13:13:45] ...optimizers/trijoint.py.151: total_norm: 11.816539 [I 2022-10-27 13:13:51] ...trap/engines/engine.py.224: train: epoch 69 | batch 230/2304 [I 2022-10-27 13:13:51] ...trap/engines/engine.py.228: elapsed: 0:02:10 | left: 0:19:09 [I 2022-10-27 13:13:51] ...trap/engines/engine.py.229: process: 0.54779 | load: 0.00243 [I 2022-10-27 13:13:51] ...trap/engines/engine.py.230: loss: 0.21225 [I 2022-10-27 13:13:51] ...optimizers/trijoint.py.151: total_norm: 14.738573 [I 2022-10-27 13:13:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 240/2304 [I 2022-10-27 13:13:56] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:12 [I 2022-10-27 13:13:56] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00249 [I 2022-10-27 13:13:56] ...trap/engines/engine.py.230: loss: 0.17359 [I 2022-10-27 13:13:56] ...optimizers/trijoint.py.151: total_norm: 16.073132 [I 2022-10-27 13:14:02] ...trap/engines/engine.py.224: train: epoch 69 | batch 250/2304 [I 2022-10-27 13:14:02] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:53 [I 2022-10-27 13:14:02] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00257 [I 2022-10-27 13:14:02] ...trap/engines/engine.py.230: loss: 0.20627 [I 2022-10-27 13:14:02] ...optimizers/trijoint.py.151: total_norm: 24.763151 [I 2022-10-27 13:14:07] ...trap/engines/engine.py.224: train: epoch 69 | batch 260/2304 [I 2022-10-27 13:14:07] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:06 [I 2022-10-27 13:14:07] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00245 [I 2022-10-27 13:14:07] ...trap/engines/engine.py.230: loss: 0.15460 [I 2022-10-27 13:14:07] ...optimizers/trijoint.py.151: total_norm: 29.687635 [I 2022-10-27 13:14:13] ...trap/engines/engine.py.224: train: epoch 69 | batch 270/2304 [I 2022-10-27 13:14:13] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:48 [I 2022-10-27 13:14:13] ...trap/engines/engine.py.229: process: 0.54454 | load: 0.00245 [I 2022-10-27 13:14:13] ...trap/engines/engine.py.230: loss: 0.11239 [I 2022-10-27 13:14:13] ...optimizers/trijoint.py.151: total_norm: 9.483606 [I 2022-10-27 13:14:19] ...trap/engines/engine.py.224: train: epoch 69 | batch 280/2304 [I 2022-10-27 13:14:19] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:52 [I 2022-10-27 13:14:19] ...trap/engines/engine.py.229: process: 0.55477 | load: 0.00242 [I 2022-10-27 13:14:19] ...trap/engines/engine.py.230: loss: 0.11346 [I 2022-10-27 13:14:19] ...optimizers/trijoint.py.151: total_norm: 10.375788 [I 2022-10-27 13:14:24] ...trap/engines/engine.py.224: train: epoch 69 | batch 290/2304 [I 2022-10-27 13:14:24] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:19:01 [I 2022-10-27 13:14:24] ...trap/engines/engine.py.229: process: 0.55623 | load: 0.00273 [I 2022-10-27 13:14:24] ...trap/engines/engine.py.230: loss: 0.09265 [I 2022-10-27 13:14:24] ...optimizers/trijoint.py.151: total_norm: 15.739618 [I 2022-10-27 13:14:30] ...trap/engines/engine.py.224: train: epoch 69 | batch 300/2304 [I 2022-10-27 13:14:30] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:43 [I 2022-10-27 13:14:30] ...trap/engines/engine.py.229: process: 0.55307 | load: 0.00254 [I 2022-10-27 13:14:30] ...trap/engines/engine.py.230: loss: 0.11096 [I 2022-10-27 13:14:30] ...optimizers/trijoint.py.151: total_norm: 8.764703 [I 2022-10-27 13:14:35] ...trap/engines/engine.py.224: train: epoch 69 | batch 310/2304 [I 2022-10-27 13:14:35] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:42 [I 2022-10-27 13:14:35] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00265 [I 2022-10-27 13:14:35] ...trap/engines/engine.py.230: loss: 0.11318 [I 2022-10-27 13:14:35] ...optimizers/trijoint.py.151: total_norm: 19.597549 [I 2022-10-27 13:14:41] ...trap/engines/engine.py.224: train: epoch 69 | batch 320/2304 [I 2022-10-27 13:14:41] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:30 [I 2022-10-27 13:14:41] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00270 [I 2022-10-27 13:14:41] ...trap/engines/engine.py.230: loss: 0.17836 [I 2022-10-27 13:14:41] ...optimizers/trijoint.py.151: total_norm: 9.404261 [I 2022-10-27 13:14:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 330/2304 [I 2022-10-27 13:14:47] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:21 [I 2022-10-27 13:14:47] ...trap/engines/engine.py.229: process: 0.56480 | load: 0.00252 [I 2022-10-27 13:14:47] ...trap/engines/engine.py.230: loss: 0.12933 [I 2022-10-27 13:14:47] ...optimizers/trijoint.py.151: total_norm: 18.633968 [I 2022-10-27 13:14:52] ...trap/engines/engine.py.224: train: epoch 69 | batch 340/2304 [I 2022-10-27 13:14:52] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:10 [I 2022-10-27 13:14:52] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00246 [I 2022-10-27 13:14:52] ...trap/engines/engine.py.230: loss: 0.13794 [I 2022-10-27 13:14:52] ...optimizers/trijoint.py.151: total_norm: 16.560303 [I 2022-10-27 13:14:58] ...trap/engines/engine.py.224: train: epoch 69 | batch 350/2304 [I 2022-10-27 13:14:58] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:07 [I 2022-10-27 13:14:58] ...trap/engines/engine.py.229: process: 0.57215 | load: 0.00245 [I 2022-10-27 13:14:58] ...trap/engines/engine.py.230: loss: 0.24608 [I 2022-10-27 13:14:58] ...optimizers/trijoint.py.151: total_norm: 20.276463 [I 2022-10-27 13:15:03] ...trap/engines/engine.py.224: train: epoch 69 | batch 360/2304 [I 2022-10-27 13:15:03] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:56 [I 2022-10-27 13:15:03] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00246 [I 2022-10-27 13:15:03] ...trap/engines/engine.py.230: loss: 0.10997 [I 2022-10-27 13:15:03] ...optimizers/trijoint.py.151: total_norm: 10.278393 [I 2022-10-27 13:15:09] ...trap/engines/engine.py.224: train: epoch 69 | batch 370/2304 [I 2022-10-27 13:15:09] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:59 [I 2022-10-27 13:15:09] ...trap/engines/engine.py.229: process: 0.57168 | load: 0.00252 [I 2022-10-27 13:15:09] ...trap/engines/engine.py.230: loss: 0.13011 [I 2022-10-27 13:15:09] ...optimizers/trijoint.py.151: total_norm: 20.223473 [I 2022-10-27 13:15:15] ...trap/engines/engine.py.224: train: epoch 69 | batch 380/2304 [I 2022-10-27 13:15:15] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:53 [I 2022-10-27 13:15:15] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00245 [I 2022-10-27 13:15:15] ...trap/engines/engine.py.230: loss: 0.16879 [I 2022-10-27 13:15:15] ...optimizers/trijoint.py.151: total_norm: 32.877758 [I 2022-10-27 13:15:20] ...trap/engines/engine.py.224: train: epoch 69 | batch 390/2304 [I 2022-10-27 13:15:20] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:39 [I 2022-10-27 13:15:20] ...trap/engines/engine.py.229: process: 0.55127 | load: 0.00249 [I 2022-10-27 13:15:20] ...trap/engines/engine.py.230: loss: 0.19822 [I 2022-10-27 13:15:20] ...optimizers/trijoint.py.151: total_norm: 17.977261 [I 2022-10-27 13:15:26] ...trap/engines/engine.py.224: train: epoch 69 | batch 400/2304 [I 2022-10-27 13:15:26] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:36 [I 2022-10-27 13:15:26] ...trap/engines/engine.py.229: process: 0.54873 | load: 0.00245 [I 2022-10-27 13:15:26] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-27 13:15:26] ...optimizers/trijoint.py.151: total_norm: 22.896152 [I 2022-10-27 13:15:31] ...trap/engines/engine.py.224: train: epoch 69 | batch 410/2304 [I 2022-10-27 13:15:31] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:37 [I 2022-10-27 13:15:31] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00245 [I 2022-10-27 13:15:31] ...trap/engines/engine.py.230: loss: 0.09960 [I 2022-10-27 13:15:31] ...optimizers/trijoint.py.151: total_norm: 9.158762 [I 2022-10-27 13:15:37] ...trap/engines/engine.py.224: train: epoch 69 | batch 420/2304 [I 2022-10-27 13:15:37] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:14 [I 2022-10-27 13:15:37] ...trap/engines/engine.py.229: process: 0.53865 | load: 0.00245 [I 2022-10-27 13:15:37] ...trap/engines/engine.py.230: loss: 0.18593 [I 2022-10-27 13:15:37] ...optimizers/trijoint.py.151: total_norm: 11.465653 [I 2022-10-27 13:15:42] ...trap/engines/engine.py.224: train: epoch 69 | batch 430/2304 [I 2022-10-27 13:15:42] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:16 [I 2022-10-27 13:15:42] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00242 [I 2022-10-27 13:15:42] ...trap/engines/engine.py.230: loss: 0.17944 [I 2022-10-27 13:15:42] ...optimizers/trijoint.py.151: total_norm: 20.386068 [I 2022-10-27 13:15:48] ...trap/engines/engine.py.224: train: epoch 69 | batch 440/2304 [I 2022-10-27 13:15:48] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:19 [I 2022-10-27 13:15:48] ...trap/engines/engine.py.229: process: 0.56553 | load: 0.00251 [I 2022-10-27 13:15:48] ...trap/engines/engine.py.230: loss: 0.13881 [I 2022-10-27 13:15:48] ...optimizers/trijoint.py.151: total_norm: 11.934132 [I 2022-10-27 13:15:53] ...trap/engines/engine.py.224: train: epoch 69 | batch 450/2304 [I 2022-10-27 13:15:53] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:08 [I 2022-10-27 13:15:53] ...trap/engines/engine.py.229: process: 0.56663 | load: 0.00244 [I 2022-10-27 13:15:53] ...trap/engines/engine.py.230: loss: 0.20454 [I 2022-10-27 13:15:53] ...optimizers/trijoint.py.151: total_norm: 18.360991 [I 2022-10-27 13:15:59] ...trap/engines/engine.py.224: train: epoch 69 | batch 460/2304 [I 2022-10-27 13:15:59] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:16:55 [I 2022-10-27 13:15:59] ...trap/engines/engine.py.229: process: 0.55171 | load: 0.00252 [I 2022-10-27 13:15:59] ...trap/engines/engine.py.230: loss: 0.15237 [I 2022-10-27 13:15:59] ...optimizers/trijoint.py.151: total_norm: 12.833435 [I 2022-10-27 13:16:05] ...trap/engines/engine.py.224: train: epoch 69 | batch 470/2304 [I 2022-10-27 13:16:05] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:16:54 [I 2022-10-27 13:16:05] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00245 [I 2022-10-27 13:16:05] ...trap/engines/engine.py.230: loss: 0.16574 [I 2022-10-27 13:16:05] ...optimizers/trijoint.py.151: total_norm: 36.936497 [I 2022-10-27 13:16:10] ...trap/engines/engine.py.224: train: epoch 69 | batch 480/2304 [I 2022-10-27 13:16:10] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:17:25 [I 2022-10-27 13:16:10] ...trap/engines/engine.py.229: process: 0.61855 | load: 0.00254 [I 2022-10-27 13:16:10] ...trap/engines/engine.py.230: loss: 0.12525 [I 2022-10-27 13:16:10] ...optimizers/trijoint.py.151: total_norm: 18.166260 [I 2022-10-27 13:16:16] ...trap/engines/engine.py.224: train: epoch 69 | batch 490/2304 [I 2022-10-27 13:16:16] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:57 [I 2022-10-27 13:16:16] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00256 [I 2022-10-27 13:16:16] ...trap/engines/engine.py.230: loss: 0.10922 [I 2022-10-27 13:16:16] ...optimizers/trijoint.py.151: total_norm: 6.533944 [I 2022-10-27 13:16:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 500/2304 [I 2022-10-27 13:16:22] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:58 [I 2022-10-27 13:16:22] ...trap/engines/engine.py.229: process: 0.54287 | load: 0.00251 [I 2022-10-27 13:16:22] ...trap/engines/engine.py.230: loss: 0.11656 [I 2022-10-27 13:16:22] ...optimizers/trijoint.py.151: total_norm: 13.628445 [I 2022-10-27 13:16:27] ...trap/engines/engine.py.224: train: epoch 69 | batch 510/2304 [I 2022-10-27 13:16:27] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:37 [I 2022-10-27 13:16:27] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00264 [I 2022-10-27 13:16:27] ...trap/engines/engine.py.230: loss: 0.17280 [I 2022-10-27 13:16:27] ...optimizers/trijoint.py.151: total_norm: 18.018068 [I 2022-10-27 13:16:33] ...trap/engines/engine.py.224: train: epoch 69 | batch 520/2304 [I 2022-10-27 13:16:33] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:16:34 [I 2022-10-27 13:16:33] ...trap/engines/engine.py.229: process: 0.57130 | load: 0.00245 [I 2022-10-27 13:16:33] ...trap/engines/engine.py.230: loss: 0.09244 [I 2022-10-27 13:16:33] ...optimizers/trijoint.py.151: total_norm: 16.662470 [I 2022-10-27 13:16:38] ...trap/engines/engine.py.224: train: epoch 69 | batch 530/2304 [I 2022-10-27 13:16:38] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:31 [I 2022-10-27 13:16:38] ...trap/engines/engine.py.229: process: 0.56509 | load: 0.00241 [I 2022-10-27 13:16:38] ...trap/engines/engine.py.230: loss: 0.13050 [I 2022-10-27 13:16:38] ...optimizers/trijoint.py.151: total_norm: 12.614657 [I 2022-10-27 13:16:44] ...trap/engines/engine.py.224: train: epoch 69 | batch 540/2304 [I 2022-10-27 13:16:44] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:13 [I 2022-10-27 13:16:44] ...trap/engines/engine.py.229: process: 0.54937 | load: 0.00262 [I 2022-10-27 13:16:44] ...trap/engines/engine.py.230: loss: 0.16788 [I 2022-10-27 13:16:44] ...optimizers/trijoint.py.151: total_norm: 21.004599 [I 2022-10-27 13:16:49] ...trap/engines/engine.py.224: train: epoch 69 | batch 550/2304 [I 2022-10-27 13:16:49] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:17 [I 2022-10-27 13:16:49] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00242 [I 2022-10-27 13:16:49] ...trap/engines/engine.py.230: loss: 0.11434 [I 2022-10-27 13:16:49] ...optimizers/trijoint.py.151: total_norm: 15.313509 [I 2022-10-27 13:16:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 560/2304 [I 2022-10-27 13:16:55] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:12 [I 2022-10-27 13:16:55] ...trap/engines/engine.py.229: process: 0.55638 | load: 0.00252 [I 2022-10-27 13:16:55] ...trap/engines/engine.py.230: loss: 0.24986 [I 2022-10-27 13:16:55] ...optimizers/trijoint.py.151: total_norm: 24.139997 [I 2022-10-27 13:17:01] ...trap/engines/engine.py.224: train: epoch 69 | batch 570/2304 [I 2022-10-27 13:17:01] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:16:10 [I 2022-10-27 13:17:01] ...trap/engines/engine.py.229: process: 0.56748 | load: 0.00249 [I 2022-10-27 13:17:01] ...trap/engines/engine.py.230: loss: 0.17653 [I 2022-10-27 13:17:01] ...optimizers/trijoint.py.151: total_norm: 15.551678 [I 2022-10-27 13:17:06] ...trap/engines/engine.py.224: train: epoch 69 | batch 580/2304 [I 2022-10-27 13:17:06] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:05 [I 2022-10-27 13:17:06] ...trap/engines/engine.py.229: process: 0.56039 | load: 0.00246 [I 2022-10-27 13:17:06] ...trap/engines/engine.py.230: loss: 0.15844 [I 2022-10-27 13:17:06] ...optimizers/trijoint.py.151: total_norm: 29.441473 [I 2022-10-27 13:17:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 590/2304 [I 2022-10-27 13:17:12] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:54 [I 2022-10-27 13:17:12] ...trap/engines/engine.py.229: process: 0.55562 | load: 0.00242 [I 2022-10-27 13:17:12] ...trap/engines/engine.py.230: loss: 0.12273 [I 2022-10-27 13:17:12] ...optimizers/trijoint.py.151: total_norm: 16.170736 [I 2022-10-27 13:17:17] ...trap/engines/engine.py.224: train: epoch 69 | batch 600/2304 [I 2022-10-27 13:17:17] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:56 [I 2022-10-27 13:17:17] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00246 [I 2022-10-27 13:17:17] ...trap/engines/engine.py.230: loss: 0.13328 [I 2022-10-27 13:17:17] ...optimizers/trijoint.py.151: total_norm: 11.087722 [I 2022-10-27 13:17:23] ...trap/engines/engine.py.224: train: epoch 69 | batch 610/2304 [I 2022-10-27 13:17:23] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:44 [I 2022-10-27 13:17:23] ...trap/engines/engine.py.229: process: 0.55847 | load: 0.00242 [I 2022-10-27 13:17:23] ...trap/engines/engine.py.230: loss: 0.15135 [I 2022-10-27 13:17:23] ...optimizers/trijoint.py.151: total_norm: 9.769262 [I 2022-10-27 13:17:29] ...trap/engines/engine.py.224: train: epoch 69 | batch 620/2304 [I 2022-10-27 13:17:29] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:33 [I 2022-10-27 13:17:29] ...trap/engines/engine.py.229: process: 0.54855 | load: 0.00256 [I 2022-10-27 13:17:29] ...trap/engines/engine.py.230: loss: 0.12360 [I 2022-10-27 13:17:29] ...optimizers/trijoint.py.151: total_norm: 17.279913 [I 2022-10-27 13:17:34] ...trap/engines/engine.py.224: train: epoch 69 | batch 630/2304 [I 2022-10-27 13:17:34] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:53 [I 2022-10-27 13:17:34] ...trap/engines/engine.py.229: process: 0.61771 | load: 0.00248 [I 2022-10-27 13:17:34] ...trap/engines/engine.py.230: loss: 0.13787 [I 2022-10-27 13:17:34] ...optimizers/trijoint.py.151: total_norm: 17.447905 [I 2022-10-27 13:17:40] ...trap/engines/engine.py.224: train: epoch 69 | batch 640/2304 [I 2022-10-27 13:17:40] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:25 [I 2022-10-27 13:17:40] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00253 [I 2022-10-27 13:17:40] ...trap/engines/engine.py.230: loss: 0.15158 [I 2022-10-27 13:17:40] ...optimizers/trijoint.py.151: total_norm: 18.550756 [I 2022-10-27 13:17:45] ...trap/engines/engine.py.224: train: epoch 69 | batch 650/2304 [I 2022-10-27 13:17:45] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:14 [I 2022-10-27 13:17:45] ...trap/engines/engine.py.229: process: 0.54325 | load: 0.00244 [I 2022-10-27 13:17:45] ...trap/engines/engine.py.230: loss: 0.20111 [I 2022-10-27 13:17:45] ...optimizers/trijoint.py.151: total_norm: 20.329491 [I 2022-10-27 13:17:51] ...trap/engines/engine.py.224: train: epoch 69 | batch 660/2304 [I 2022-10-27 13:17:51] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:19 [I 2022-10-27 13:17:51] ...trap/engines/engine.py.229: process: 0.56438 | load: 0.00266 [I 2022-10-27 13:17:51] ...trap/engines/engine.py.230: loss: 0.15945 [I 2022-10-27 13:17:51] ...optimizers/trijoint.py.151: total_norm: 13.430834 [I 2022-10-27 13:17:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 670/2304 [I 2022-10-27 13:17:56] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:09 [I 2022-10-27 13:17:56] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00246 [I 2022-10-27 13:17:56] ...trap/engines/engine.py.230: loss: 0.17064 [I 2022-10-27 13:17:56] ...optimizers/trijoint.py.151: total_norm: 18.590532 [I 2022-10-27 13:18:02] ...trap/engines/engine.py.224: train: epoch 69 | batch 680/2304 [I 2022-10-27 13:18:02] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:06 [I 2022-10-27 13:18:02] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00242 [I 2022-10-27 13:18:02] ...trap/engines/engine.py.230: loss: 0.11758 [I 2022-10-27 13:18:02] ...optimizers/trijoint.py.151: total_norm: 17.589437 [I 2022-10-27 13:18:08] ...trap/engines/engine.py.224: train: epoch 69 | batch 690/2304 [I 2022-10-27 13:18:08] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:14:59 [I 2022-10-27 13:18:08] ...trap/engines/engine.py.229: process: 0.55094 | load: 0.00243 [I 2022-10-27 13:18:08] ...trap/engines/engine.py.230: loss: 0.13947 [I 2022-10-27 13:18:08] ...optimizers/trijoint.py.151: total_norm: 13.480303 [I 2022-10-27 13:18:13] ...trap/engines/engine.py.224: train: epoch 69 | batch 700/2304 [I 2022-10-27 13:18:13] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:58 [I 2022-10-27 13:18:13] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00243 [I 2022-10-27 13:18:13] ...trap/engines/engine.py.230: loss: 0.18336 [I 2022-10-27 13:18:13] ...optimizers/trijoint.py.151: total_norm: 12.519616 [I 2022-10-27 13:18:19] ...trap/engines/engine.py.224: train: epoch 69 | batch 710/2304 [I 2022-10-27 13:18:19] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:48 [I 2022-10-27 13:18:19] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00243 [I 2022-10-27 13:18:19] ...trap/engines/engine.py.230: loss: 0.20477 [I 2022-10-27 13:18:19] ...optimizers/trijoint.py.151: total_norm: 14.487488 [I 2022-10-27 13:18:24] ...trap/engines/engine.py.224: train: epoch 69 | batch 720/2304 [I 2022-10-27 13:18:24] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:41 [I 2022-10-27 13:18:24] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00245 [I 2022-10-27 13:18:24] ...trap/engines/engine.py.230: loss: 0.13566 [I 2022-10-27 13:18:24] ...optimizers/trijoint.py.151: total_norm: 16.777855 [I 2022-10-27 13:18:30] ...trap/engines/engine.py.224: train: epoch 69 | batch 730/2304 [I 2022-10-27 13:18:30] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:37 [I 2022-10-27 13:18:30] ...trap/engines/engine.py.229: process: 0.56316 | load: 0.00254 [I 2022-10-27 13:18:30] ...trap/engines/engine.py.230: loss: 0.12462 [I 2022-10-27 13:18:30] ...optimizers/trijoint.py.151: total_norm: 14.355226 [I 2022-10-27 13:18:36] ...trap/engines/engine.py.224: train: epoch 69 | batch 740/2304 [I 2022-10-27 13:18:36] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:15:11 [I 2022-10-27 13:18:36] ...trap/engines/engine.py.229: process: 0.61504 | load: 0.00273 [I 2022-10-27 13:18:36] ...trap/engines/engine.py.230: loss: 0.16074 [I 2022-10-27 13:18:36] ...optimizers/trijoint.py.151: total_norm: 29.684361 [I 2022-10-27 13:18:41] ...trap/engines/engine.py.224: train: epoch 69 | batch 750/2304 [I 2022-10-27 13:18:41] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:32 [I 2022-10-27 13:18:41] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00245 [I 2022-10-27 13:18:41] ...trap/engines/engine.py.230: loss: 0.13149 [I 2022-10-27 13:18:41] ...optimizers/trijoint.py.151: total_norm: 10.289155 [I 2022-10-27 13:18:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 760/2304 [I 2022-10-27 13:18:47] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:17 [I 2022-10-27 13:18:47] ...trap/engines/engine.py.229: process: 0.54335 | load: 0.00250 [I 2022-10-27 13:18:47] ...trap/engines/engine.py.230: loss: 0.21366 [I 2022-10-27 13:18:47] ...optimizers/trijoint.py.151: total_norm: 28.369366 [I 2022-10-27 13:18:52] ...trap/engines/engine.py.224: train: epoch 69 | batch 770/2304 [I 2022-10-27 13:18:52] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:18 [I 2022-10-27 13:18:52] ...trap/engines/engine.py.229: process: 0.56996 | load: 0.00241 [I 2022-10-27 13:18:52] ...trap/engines/engine.py.230: loss: 0.15031 [I 2022-10-27 13:18:52] ...optimizers/trijoint.py.151: total_norm: 11.709181 [I 2022-10-27 13:18:58] ...trap/engines/engine.py.224: train: epoch 69 | batch 780/2304 [I 2022-10-27 13:18:58] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:13 [I 2022-10-27 13:18:58] ...trap/engines/engine.py.229: process: 0.55259 | load: 0.00244 [I 2022-10-27 13:18:58] ...trap/engines/engine.py.230: loss: 0.11521 [I 2022-10-27 13:18:58] ...optimizers/trijoint.py.151: total_norm: 13.395892 [I 2022-10-27 13:19:04] ...trap/engines/engine.py.224: train: epoch 69 | batch 790/2304 [I 2022-10-27 13:19:04] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:03 [I 2022-10-27 13:19:04] ...trap/engines/engine.py.229: process: 0.56458 | load: 0.00246 [I 2022-10-27 13:19:04] ...trap/engines/engine.py.230: loss: 0.16114 [I 2022-10-27 13:19:04] ...optimizers/trijoint.py.151: total_norm: 28.399380 [I 2022-10-27 13:19:09] ...trap/engines/engine.py.224: train: epoch 69 | batch 800/2304 [I 2022-10-27 13:19:09] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:55 [I 2022-10-27 13:19:09] ...trap/engines/engine.py.229: process: 0.56796 | load: 0.00246 [I 2022-10-27 13:19:09] ...trap/engines/engine.py.230: loss: 0.13087 [I 2022-10-27 13:19:09] ...optimizers/trijoint.py.151: total_norm: 21.935780 [I 2022-10-27 13:19:15] ...trap/engines/engine.py.224: train: epoch 69 | batch 810/2304 [I 2022-10-27 13:19:15] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:53 [I 2022-10-27 13:19:15] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00249 [I 2022-10-27 13:19:15] ...trap/engines/engine.py.230: loss: 0.13672 [I 2022-10-27 13:19:15] ...optimizers/trijoint.py.151: total_norm: 20.109873 [I 2022-10-27 13:19:20] ...trap/engines/engine.py.224: train: epoch 69 | batch 820/2304 [I 2022-10-27 13:19:20] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:46 [I 2022-10-27 13:19:20] ...trap/engines/engine.py.229: process: 0.56698 | load: 0.00253 [I 2022-10-27 13:19:20] ...trap/engines/engine.py.230: loss: 0.10178 [I 2022-10-27 13:19:20] ...optimizers/trijoint.py.151: total_norm: 14.659195 [I 2022-10-27 13:19:26] ...trap/engines/engine.py.224: train: epoch 69 | batch 830/2304 [I 2022-10-27 13:19:26] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:40 [I 2022-10-27 13:19:26] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00245 [I 2022-10-27 13:19:26] ...trap/engines/engine.py.230: loss: 0.14982 [I 2022-10-27 13:19:26] ...optimizers/trijoint.py.151: total_norm: 11.353949 [I 2022-10-27 13:19:32] ...trap/engines/engine.py.224: train: epoch 69 | batch 840/2304 [I 2022-10-27 13:19:32] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:35 [I 2022-10-27 13:19:32] ...trap/engines/engine.py.229: process: 0.56473 | load: 0.00248 [I 2022-10-27 13:19:32] ...trap/engines/engine.py.230: loss: 0.09034 [I 2022-10-27 13:19:32] ...optimizers/trijoint.py.151: total_norm: 9.171578 [I 2022-10-27 13:19:37] ...trap/engines/engine.py.224: train: epoch 69 | batch 850/2304 [I 2022-10-27 13:19:37] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:36 [I 2022-10-27 13:19:37] ...trap/engines/engine.py.229: process: 0.56663 | load: 0.00244 [I 2022-10-27 13:19:37] ...trap/engines/engine.py.230: loss: 0.12808 [I 2022-10-27 13:19:37] ...optimizers/trijoint.py.151: total_norm: 9.506982 [I 2022-10-27 13:19:43] ...trap/engines/engine.py.224: train: epoch 69 | batch 860/2304 [I 2022-10-27 13:19:43] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:27 [I 2022-10-27 13:19:43] ...trap/engines/engine.py.229: process: 0.54178 | load: 0.00243 [I 2022-10-27 13:19:43] ...trap/engines/engine.py.230: loss: 0.12245 [I 2022-10-27 13:19:43] ...optimizers/trijoint.py.151: total_norm: 15.685473 [I 2022-10-27 13:19:48] ...trap/engines/engine.py.224: train: epoch 69 | batch 870/2304 [I 2022-10-27 13:19:48] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:23 [I 2022-10-27 13:19:48] ...trap/engines/engine.py.229: process: 0.56664 | load: 0.00242 [I 2022-10-27 13:19:48] ...trap/engines/engine.py.230: loss: 0.17364 [I 2022-10-27 13:19:48] ...optimizers/trijoint.py.151: total_norm: 16.253759 [I 2022-10-27 13:19:54] ...trap/engines/engine.py.224: train: epoch 69 | batch 880/2304 [I 2022-10-27 13:19:54] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:12 [I 2022-10-27 13:19:54] ...trap/engines/engine.py.229: process: 0.55206 | load: 0.00240 [I 2022-10-27 13:19:54] ...trap/engines/engine.py.230: loss: 0.13934 [I 2022-10-27 13:19:54] ...optimizers/trijoint.py.151: total_norm: 11.213497 [I 2022-10-27 13:20:00] ...trap/engines/engine.py.224: train: epoch 69 | batch 890/2304 [I 2022-10-27 13:20:00] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:02 [I 2022-10-27 13:20:00] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00250 [I 2022-10-27 13:20:00] ...trap/engines/engine.py.230: loss: 0.16407 [I 2022-10-27 13:20:00] ...optimizers/trijoint.py.151: total_norm: 19.191507 [I 2022-10-27 13:20:05] ...trap/engines/engine.py.224: train: epoch 69 | batch 900/2304 [I 2022-10-27 13:20:05] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:00 [I 2022-10-27 13:20:05] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00245 [I 2022-10-27 13:20:05] ...trap/engines/engine.py.230: loss: 0.11145 [I 2022-10-27 13:20:05] ...optimizers/trijoint.py.151: total_norm: 16.984022 [I 2022-10-27 13:20:11] ...trap/engines/engine.py.224: train: epoch 69 | batch 910/2304 [I 2022-10-27 13:20:11] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:57 [I 2022-10-27 13:20:11] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00244 [I 2022-10-27 13:20:11] ...trap/engines/engine.py.230: loss: 0.10948 [I 2022-10-27 13:20:11] ...optimizers/trijoint.py.151: total_norm: 11.860471 [I 2022-10-27 13:20:16] ...trap/engines/engine.py.224: train: epoch 69 | batch 920/2304 [I 2022-10-27 13:20:16] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:49 [I 2022-10-27 13:20:16] ...trap/engines/engine.py.229: process: 0.54714 | load: 0.00245 [I 2022-10-27 13:20:16] ...trap/engines/engine.py.230: loss: 0.21585 [I 2022-10-27 13:20:16] ...optimizers/trijoint.py.151: total_norm: 32.505711 [I 2022-10-27 13:20:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 930/2304 [I 2022-10-27 13:20:22] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:51 [I 2022-10-27 13:20:22] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00244 [I 2022-10-27 13:20:22] ...trap/engines/engine.py.230: loss: 0.18151 [I 2022-10-27 13:20:22] ...optimizers/trijoint.py.151: total_norm: 30.417650 [I 2022-10-27 13:20:28] ...trap/engines/engine.py.224: train: epoch 69 | batch 940/2304 [I 2022-10-27 13:20:28] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:38 [I 2022-10-27 13:20:28] ...trap/engines/engine.py.229: process: 0.55144 | load: 0.00247 [I 2022-10-27 13:20:28] ...trap/engines/engine.py.230: loss: 0.15529 [I 2022-10-27 13:20:28] ...optimizers/trijoint.py.151: total_norm: 20.661764 [I 2022-10-27 13:20:33] ...trap/engines/engine.py.224: train: epoch 69 | batch 950/2304 [I 2022-10-27 13:20:33] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:35 [I 2022-10-27 13:20:33] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00248 [I 2022-10-27 13:20:33] ...trap/engines/engine.py.230: loss: 0.19208 [I 2022-10-27 13:20:33] ...optimizers/trijoint.py.151: total_norm: 14.428531 [I 2022-10-27 13:20:39] ...trap/engines/engine.py.224: train: epoch 69 | batch 960/2304 [I 2022-10-27 13:20:39] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:27 [I 2022-10-27 13:20:39] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00246 [I 2022-10-27 13:20:39] ...trap/engines/engine.py.230: loss: 0.14306 [I 2022-10-27 13:20:39] ...optimizers/trijoint.py.151: total_norm: 21.084412 [I 2022-10-27 13:20:44] ...trap/engines/engine.py.224: train: epoch 69 | batch 970/2304 [I 2022-10-27 13:20:44] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:24 [I 2022-10-27 13:20:44] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00246 [I 2022-10-27 13:20:44] ...trap/engines/engine.py.230: loss: 0.17727 [I 2022-10-27 13:20:44] ...optimizers/trijoint.py.151: total_norm: 29.252199 [I 2022-10-27 13:20:50] ...trap/engines/engine.py.224: train: epoch 69 | batch 980/2304 [I 2022-10-27 13:20:50] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:42 [I 2022-10-27 13:20:50] ...trap/engines/engine.py.229: process: 0.55100 | load: 0.00242 [I 2022-10-27 13:20:50] ...trap/engines/engine.py.230: loss: 0.16874 [I 2022-10-27 13:20:50] ...optimizers/trijoint.py.151: total_norm: 41.935905 [I 2022-10-27 13:20:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 990/2304 [I 2022-10-27 13:20:55] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:03 [I 2022-10-27 13:20:55] ...trap/engines/engine.py.229: process: 0.54805 | load: 0.00241 [I 2022-10-27 13:20:55] ...trap/engines/engine.py.230: loss: 0.13160 [I 2022-10-27 13:20:55] ...optimizers/trijoint.py.151: total_norm: 10.213661 [I 2022-10-27 13:21:01] ...trap/engines/engine.py.224: train: epoch 69 | batch 1000/2304 [I 2022-10-27 13:21:01] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:02 [I 2022-10-27 13:21:01] ...trap/engines/engine.py.229: process: 0.55720 | load: 0.00244 [I 2022-10-27 13:21:01] ...trap/engines/engine.py.230: loss: 0.13656 [I 2022-10-27 13:21:01] ...optimizers/trijoint.py.151: total_norm: 15.191280 [I 2022-10-27 13:21:07] ...trap/engines/engine.py.224: train: epoch 69 | batch 1010/2304 [I 2022-10-27 13:21:07] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:11:56 [I 2022-10-27 13:21:07] ...trap/engines/engine.py.229: process: 0.54588 | load: 0.00264 [I 2022-10-27 13:21:07] ...trap/engines/engine.py.230: loss: 0.16622 [I 2022-10-27 13:21:07] ...optimizers/trijoint.py.151: total_norm: 13.789802 [I 2022-10-27 13:21:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 1020/2304 [I 2022-10-27 13:21:12] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:54 [I 2022-10-27 13:21:12] ...trap/engines/engine.py.229: process: 0.56599 | load: 0.00242 [I 2022-10-27 13:21:12] ...trap/engines/engine.py.230: loss: 0.11916 [I 2022-10-27 13:21:12] ...optimizers/trijoint.py.151: total_norm: 26.867378 [I 2022-10-27 13:21:18] ...trap/engines/engine.py.224: train: epoch 69 | batch 1030/2304 [I 2022-10-27 13:21:18] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:49 [I 2022-10-27 13:21:18] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00244 [I 2022-10-27 13:21:18] ...trap/engines/engine.py.230: loss: 0.14322 [I 2022-10-27 13:21:18] ...optimizers/trijoint.py.151: total_norm: 9.806668 [I 2022-10-27 13:21:23] ...trap/engines/engine.py.224: train: epoch 69 | batch 1040/2304 [I 2022-10-27 13:21:23] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:43 [I 2022-10-27 13:21:23] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00241 [I 2022-10-27 13:21:23] ...trap/engines/engine.py.230: loss: 0.24418 [I 2022-10-27 13:21:23] ...optimizers/trijoint.py.151: total_norm: 22.856373 [I 2022-10-27 13:21:29] ...trap/engines/engine.py.224: train: epoch 69 | batch 1050/2304 [I 2022-10-27 13:21:29] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:32 [I 2022-10-27 13:21:29] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00242 [I 2022-10-27 13:21:29] ...trap/engines/engine.py.230: loss: 0.25496 [I 2022-10-27 13:21:29] ...optimizers/trijoint.py.151: total_norm: 15.935937 [I 2022-10-27 13:21:34] ...trap/engines/engine.py.224: train: epoch 69 | batch 1060/2304 [I 2022-10-27 13:21:34] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:31 [I 2022-10-27 13:21:34] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00255 [I 2022-10-27 13:21:34] ...trap/engines/engine.py.230: loss: 0.19989 [I 2022-10-27 13:21:34] ...optimizers/trijoint.py.151: total_norm: 17.358328 [I 2022-10-27 13:21:40] ...trap/engines/engine.py.224: train: epoch 69 | batch 1070/2304 [I 2022-10-27 13:21:40] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:22 [I 2022-10-27 13:21:40] ...trap/engines/engine.py.229: process: 0.54654 | load: 0.00249 [I 2022-10-27 13:21:40] ...trap/engines/engine.py.230: loss: 0.14378 [I 2022-10-27 13:21:40] ...optimizers/trijoint.py.151: total_norm: 14.295732 [I 2022-10-27 13:21:46] ...trap/engines/engine.py.224: train: epoch 69 | batch 1080/2304 [I 2022-10-27 13:21:46] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:24 [I 2022-10-27 13:21:46] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00243 [I 2022-10-27 13:21:46] ...trap/engines/engine.py.230: loss: 0.14933 [I 2022-10-27 13:21:46] ...optimizers/trijoint.py.151: total_norm: 18.571465 [I 2022-10-27 13:21:51] ...trap/engines/engine.py.224: train: epoch 69 | batch 1090/2304 [I 2022-10-27 13:21:51] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:21 [I 2022-10-27 13:21:51] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00243 [I 2022-10-27 13:21:51] ...trap/engines/engine.py.230: loss: 0.10863 [I 2022-10-27 13:21:51] ...optimizers/trijoint.py.151: total_norm: 11.933333 [I 2022-10-27 13:21:57] ...trap/engines/engine.py.224: train: epoch 69 | batch 1100/2304 [I 2022-10-27 13:21:57] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:11 [I 2022-10-27 13:21:57] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00243 [I 2022-10-27 13:21:57] ...trap/engines/engine.py.230: loss: 0.13801 [I 2022-10-27 13:21:57] ...optimizers/trijoint.py.151: total_norm: 31.780640 [I 2022-10-27 13:22:02] ...trap/engines/engine.py.224: train: epoch 69 | batch 1110/2304 [I 2022-10-27 13:22:02] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:08 [I 2022-10-27 13:22:02] ...trap/engines/engine.py.229: process: 0.57202 | load: 0.00241 [I 2022-10-27 13:22:02] ...trap/engines/engine.py.230: loss: 0.13267 [I 2022-10-27 13:22:02] ...optimizers/trijoint.py.151: total_norm: 12.470363 [I 2022-10-27 13:22:08] ...trap/engines/engine.py.224: train: epoch 69 | batch 1120/2304 [I 2022-10-27 13:22:08] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:00 [I 2022-10-27 13:22:08] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00254 [I 2022-10-27 13:22:08] ...trap/engines/engine.py.230: loss: 0.18741 [I 2022-10-27 13:22:08] ...optimizers/trijoint.py.151: total_norm: 29.188328 [I 2022-10-27 13:22:14] ...trap/engines/engine.py.224: train: epoch 69 | batch 1130/2304 [I 2022-10-27 13:22:14] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:11:00 [I 2022-10-27 13:22:14] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00244 [I 2022-10-27 13:22:14] ...trap/engines/engine.py.230: loss: 0.12745 [I 2022-10-27 13:22:14] ...optimizers/trijoint.py.151: total_norm: 18.282192 [I 2022-10-27 13:22:19] ...trap/engines/engine.py.224: train: epoch 69 | batch 1140/2304 [I 2022-10-27 13:22:19] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:50 [I 2022-10-27 13:22:19] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00254 [I 2022-10-27 13:22:19] ...trap/engines/engine.py.230: loss: 0.12809 [I 2022-10-27 13:22:19] ...optimizers/trijoint.py.151: total_norm: 22.378403 [I 2022-10-27 13:22:25] ...trap/engines/engine.py.224: train: epoch 69 | batch 1150/2304 [I 2022-10-27 13:22:25] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:43 [I 2022-10-27 13:22:25] ...trap/engines/engine.py.229: process: 0.56036 | load: 0.00247 [I 2022-10-27 13:22:25] ...trap/engines/engine.py.230: loss: 0.13894 [I 2022-10-27 13:22:25] ...optimizers/trijoint.py.151: total_norm: 21.792000 [I 2022-10-27 13:22:30] ...trap/engines/engine.py.224: train: epoch 69 | batch 1160/2304 [I 2022-10-27 13:22:30] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:37 [I 2022-10-27 13:22:30] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00249 [I 2022-10-27 13:22:30] ...trap/engines/engine.py.230: loss: 0.11082 [I 2022-10-27 13:22:30] ...optimizers/trijoint.py.151: total_norm: 15.556046 [I 2022-10-27 13:22:36] ...trap/engines/engine.py.224: train: epoch 69 | batch 1170/2304 [I 2022-10-27 13:22:36] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:31 [I 2022-10-27 13:22:36] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00243 [I 2022-10-27 13:22:36] ...trap/engines/engine.py.230: loss: 0.09522 [I 2022-10-27 13:22:36] ...optimizers/trijoint.py.151: total_norm: 10.082568 [I 2022-10-27 13:22:42] ...trap/engines/engine.py.224: train: epoch 69 | batch 1180/2304 [I 2022-10-27 13:22:42] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:26 [I 2022-10-27 13:22:42] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00250 [I 2022-10-27 13:22:42] ...trap/engines/engine.py.230: loss: 0.19601 [I 2022-10-27 13:22:42] ...optimizers/trijoint.py.151: total_norm: 20.159109 [I 2022-10-27 13:22:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 1190/2304 [I 2022-10-27 13:22:47] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:25 [I 2022-10-27 13:22:47] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00251 [I 2022-10-27 13:22:47] ...trap/engines/engine.py.230: loss: 0.09131 [I 2022-10-27 13:22:47] ...optimizers/trijoint.py.151: total_norm: 8.941396 [I 2022-10-27 13:22:53] ...trap/engines/engine.py.224: train: epoch 69 | batch 1200/2304 [I 2022-10-27 13:22:53] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:13 [I 2022-10-27 13:22:53] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00246 [I 2022-10-27 13:22:53] ...trap/engines/engine.py.230: loss: 0.16457 [I 2022-10-27 13:22:53] ...optimizers/trijoint.py.151: total_norm: 34.927364 [I 2022-10-27 13:22:58] ...trap/engines/engine.py.224: train: epoch 69 | batch 1210/2304 [I 2022-10-27 13:22:58] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:12 [I 2022-10-27 13:22:58] ...trap/engines/engine.py.229: process: 0.57352 | load: 0.00246 [I 2022-10-27 13:22:58] ...trap/engines/engine.py.230: loss: 0.17618 [I 2022-10-27 13:22:58] ...optimizers/trijoint.py.151: total_norm: 27.255672 [I 2022-10-27 13:23:04] ...trap/engines/engine.py.224: train: epoch 69 | batch 1220/2304 [I 2022-10-27 13:23:04] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:09:59 [I 2022-10-27 13:23:04] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00246 [I 2022-10-27 13:23:04] ...trap/engines/engine.py.230: loss: 0.15221 [I 2022-10-27 13:23:04] ...optimizers/trijoint.py.151: total_norm: 16.020458 [I 2022-10-27 13:23:09] ...trap/engines/engine.py.224: train: epoch 69 | batch 1230/2304 [I 2022-10-27 13:23:09] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:09:57 [I 2022-10-27 13:23:09] ...trap/engines/engine.py.229: process: 0.55119 | load: 0.00258 [I 2022-10-27 13:23:09] ...trap/engines/engine.py.230: loss: 0.13966 [I 2022-10-27 13:23:09] ...optimizers/trijoint.py.151: total_norm: 31.604975 [I 2022-10-27 13:23:15] ...trap/engines/engine.py.224: train: epoch 69 | batch 1240/2304 [I 2022-10-27 13:23:15] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:51 [I 2022-10-27 13:23:15] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00245 [I 2022-10-27 13:23:15] ...trap/engines/engine.py.230: loss: 0.17849 [I 2022-10-27 13:23:15] ...optimizers/trijoint.py.151: total_norm: 42.798668 [I 2022-10-27 13:23:21] ...trap/engines/engine.py.224: train: epoch 69 | batch 1250/2304 [I 2022-10-27 13:23:21] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:09:48 [I 2022-10-27 13:23:21] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00245 [I 2022-10-27 13:23:21] ...trap/engines/engine.py.230: loss: 0.12365 [I 2022-10-27 13:23:21] ...optimizers/trijoint.py.151: total_norm: 17.526018 [I 2022-10-27 13:23:26] ...trap/engines/engine.py.224: train: epoch 69 | batch 1260/2304 [I 2022-10-27 13:23:26] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:41 [I 2022-10-27 13:23:26] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00247 [I 2022-10-27 13:23:26] ...trap/engines/engine.py.230: loss: 0.16748 [I 2022-10-27 13:23:26] ...optimizers/trijoint.py.151: total_norm: 23.142847 [I 2022-10-27 13:23:32] ...trap/engines/engine.py.224: train: epoch 69 | batch 1270/2304 [I 2022-10-27 13:23:32] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:35 [I 2022-10-27 13:23:32] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00280 [I 2022-10-27 13:23:32] ...trap/engines/engine.py.230: loss: 0.17910 [I 2022-10-27 13:23:32] ...optimizers/trijoint.py.151: total_norm: 15.768346 [I 2022-10-27 13:23:37] ...trap/engines/engine.py.224: train: epoch 69 | batch 1280/2304 [I 2022-10-27 13:23:37] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:31 [I 2022-10-27 13:23:37] ...trap/engines/engine.py.229: process: 0.53743 | load: 0.00240 [I 2022-10-27 13:23:37] ...trap/engines/engine.py.230: loss: 0.17426 [I 2022-10-27 13:23:37] ...optimizers/trijoint.py.151: total_norm: 18.240646 [I 2022-10-27 13:23:43] ...trap/engines/engine.py.224: train: epoch 69 | batch 1290/2304 [I 2022-10-27 13:23:43] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:26 [I 2022-10-27 13:23:43] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00246 [I 2022-10-27 13:23:43] ...trap/engines/engine.py.230: loss: 0.09998 [I 2022-10-27 13:23:43] ...optimizers/trijoint.py.151: total_norm: 8.023991 [I 2022-10-27 13:23:49] ...trap/engines/engine.py.224: train: epoch 69 | batch 1300/2304 [I 2022-10-27 13:23:49] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:20 [I 2022-10-27 13:23:49] ...trap/engines/engine.py.229: process: 0.55288 | load: 0.00242 [I 2022-10-27 13:23:49] ...trap/engines/engine.py.230: loss: 0.12144 [I 2022-10-27 13:23:49] ...optimizers/trijoint.py.151: total_norm: 16.280554 [I 2022-10-27 13:23:54] ...trap/engines/engine.py.224: train: epoch 69 | batch 1310/2304 [I 2022-10-27 13:23:54] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:12 [I 2022-10-27 13:23:54] ...trap/engines/engine.py.229: process: 0.55431 | load: 0.00242 [I 2022-10-27 13:23:54] ...trap/engines/engine.py.230: loss: 0.14078 [I 2022-10-27 13:23:54] ...optimizers/trijoint.py.151: total_norm: 13.415327 [I 2022-10-27 13:24:00] ...trap/engines/engine.py.224: train: epoch 69 | batch 1320/2304 [I 2022-10-27 13:24:00] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:07 [I 2022-10-27 13:24:00] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00248 [I 2022-10-27 13:24:00] ...trap/engines/engine.py.230: loss: 0.09747 [I 2022-10-27 13:24:00] ...optimizers/trijoint.py.151: total_norm: 11.223050 [I 2022-10-27 13:24:05] ...trap/engines/engine.py.224: train: epoch 69 | batch 1330/2304 [I 2022-10-27 13:24:05] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:06 [I 2022-10-27 13:24:05] ...trap/engines/engine.py.229: process: 0.56308 | load: 0.00267 [I 2022-10-27 13:24:05] ...trap/engines/engine.py.230: loss: 0.20879 [I 2022-10-27 13:24:05] ...optimizers/trijoint.py.151: total_norm: 21.443670 [I 2022-10-27 13:24:11] ...trap/engines/engine.py.224: train: epoch 69 | batch 1340/2304 [I 2022-10-27 13:24:11] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:57 [I 2022-10-27 13:24:11] ...trap/engines/engine.py.229: process: 0.55567 | load: 0.00249 [I 2022-10-27 13:24:11] ...trap/engines/engine.py.230: loss: 0.12414 [I 2022-10-27 13:24:11] ...optimizers/trijoint.py.151: total_norm: 19.578779 [I 2022-10-27 13:24:17] ...trap/engines/engine.py.224: train: epoch 69 | batch 1350/2304 [I 2022-10-27 13:24:17] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:54 [I 2022-10-27 13:24:17] ...trap/engines/engine.py.229: process: 0.57067 | load: 0.00245 [I 2022-10-27 13:24:17] ...trap/engines/engine.py.230: loss: 0.13510 [I 2022-10-27 13:24:17] ...optimizers/trijoint.py.151: total_norm: 12.908891 [I 2022-10-27 13:24:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 1360/2304 [I 2022-10-27 13:24:22] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:45 [I 2022-10-27 13:24:22] ...trap/engines/engine.py.229: process: 0.55568 | load: 0.00244 [I 2022-10-27 13:24:22] ...trap/engines/engine.py.230: loss: 0.11513 [I 2022-10-27 13:24:22] ...optimizers/trijoint.py.151: total_norm: 14.770098 [I 2022-10-27 13:24:28] ...trap/engines/engine.py.224: train: epoch 69 | batch 1370/2304 [I 2022-10-27 13:24:28] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:41 [I 2022-10-27 13:24:28] ...trap/engines/engine.py.229: process: 0.54399 | load: 0.00244 [I 2022-10-27 13:24:28] ...trap/engines/engine.py.230: loss: 0.19680 [I 2022-10-27 13:24:28] ...optimizers/trijoint.py.151: total_norm: 17.388128 [I 2022-10-27 13:24:33] ...trap/engines/engine.py.224: train: epoch 69 | batch 1380/2304 [I 2022-10-27 13:24:33] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:30 [I 2022-10-27 13:24:33] ...trap/engines/engine.py.229: process: 0.54744 | load: 0.00240 [I 2022-10-27 13:24:33] ...trap/engines/engine.py.230: loss: 0.16931 [I 2022-10-27 13:24:33] ...optimizers/trijoint.py.151: total_norm: 25.919556 [I 2022-10-27 13:24:39] ...trap/engines/engine.py.224: train: epoch 69 | batch 1390/2304 [I 2022-10-27 13:24:39] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:28 [I 2022-10-27 13:24:39] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00244 [I 2022-10-27 13:24:39] ...trap/engines/engine.py.230: loss: 0.16137 [I 2022-10-27 13:24:39] ...optimizers/trijoint.py.151: total_norm: 21.790100 [I 2022-10-27 13:24:44] ...trap/engines/engine.py.224: train: epoch 69 | batch 1400/2304 [I 2022-10-27 13:24:44] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:22 [I 2022-10-27 13:24:44] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00241 [I 2022-10-27 13:24:44] ...trap/engines/engine.py.230: loss: 0.13090 [I 2022-10-27 13:24:44] ...optimizers/trijoint.py.151: total_norm: 10.859950 [I 2022-10-27 13:24:50] ...trap/engines/engine.py.224: train: epoch 69 | batch 1410/2304 [I 2022-10-27 13:24:50] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:19 [I 2022-10-27 13:24:50] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00242 [I 2022-10-27 13:24:50] ...trap/engines/engine.py.230: loss: 0.10475 [I 2022-10-27 13:24:50] ...optimizers/trijoint.py.151: total_norm: 10.283121 [I 2022-10-27 13:24:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 1420/2304 [I 2022-10-27 13:24:56] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:15 [I 2022-10-27 13:24:56] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00245 [I 2022-10-27 13:24:56] ...trap/engines/engine.py.230: loss: 0.10488 [I 2022-10-27 13:24:56] ...optimizers/trijoint.py.151: total_norm: 10.906168 [I 2022-10-27 13:25:01] ...trap/engines/engine.py.224: train: epoch 69 | batch 1430/2304 [I 2022-10-27 13:25:01] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:07 [I 2022-10-27 13:25:01] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00243 [I 2022-10-27 13:25:01] ...trap/engines/engine.py.230: loss: 0.11339 [I 2022-10-27 13:25:01] ...optimizers/trijoint.py.151: total_norm: 33.324604 [I 2022-10-27 13:25:07] ...trap/engines/engine.py.224: train: epoch 69 | batch 1440/2304 [I 2022-10-27 13:25:07] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:00 [I 2022-10-27 13:25:07] ...trap/engines/engine.py.229: process: 0.55505 | load: 0.00241 [I 2022-10-27 13:25:07] ...trap/engines/engine.py.230: loss: 0.13933 [I 2022-10-27 13:25:07] ...optimizers/trijoint.py.151: total_norm: 13.189358 [I 2022-10-27 13:25:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 1450/2304 [I 2022-10-27 13:25:12] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:07:54 [I 2022-10-27 13:25:12] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00244 [I 2022-10-27 13:25:12] ...trap/engines/engine.py.230: loss: 0.15243 [I 2022-10-27 13:25:12] ...optimizers/trijoint.py.151: total_norm: 19.506273 [I 2022-10-27 13:25:18] ...trap/engines/engine.py.224: train: epoch 69 | batch 1460/2304 [I 2022-10-27 13:25:18] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:47 [I 2022-10-27 13:25:18] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00243 [I 2022-10-27 13:25:18] ...trap/engines/engine.py.230: loss: 0.17195 [I 2022-10-27 13:25:18] ...optimizers/trijoint.py.151: total_norm: 10.319059 [I 2022-10-27 13:25:23] ...trap/engines/engine.py.224: train: epoch 69 | batch 1470/2304 [I 2022-10-27 13:25:23] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:07:41 [I 2022-10-27 13:25:23] ...trap/engines/engine.py.229: process: 0.54381 | load: 0.00254 [I 2022-10-27 13:25:23] ...trap/engines/engine.py.230: loss: 0.14273 [I 2022-10-27 13:25:23] ...optimizers/trijoint.py.151: total_norm: 20.189865 [I 2022-10-27 13:25:29] ...trap/engines/engine.py.224: train: epoch 69 | batch 1480/2304 [I 2022-10-27 13:25:29] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:39 [I 2022-10-27 13:25:29] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00250 [I 2022-10-27 13:25:29] ...trap/engines/engine.py.230: loss: 0.14521 [I 2022-10-27 13:25:29] ...optimizers/trijoint.py.151: total_norm: 12.001149 [I 2022-10-27 13:25:35] ...trap/engines/engine.py.224: train: epoch 69 | batch 1490/2304 [I 2022-10-27 13:25:35] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:35 [I 2022-10-27 13:25:35] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00249 [I 2022-10-27 13:25:35] ...trap/engines/engine.py.230: loss: 0.13607 [I 2022-10-27 13:25:35] ...optimizers/trijoint.py.151: total_norm: 20.511484 [I 2022-10-27 13:25:40] ...trap/engines/engine.py.224: train: epoch 69 | batch 1500/2304 [I 2022-10-27 13:25:40] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:27 [I 2022-10-27 13:25:40] ...trap/engines/engine.py.229: process: 0.56501 | load: 0.00244 [I 2022-10-27 13:25:40] ...trap/engines/engine.py.230: loss: 0.16543 [I 2022-10-27 13:25:40] ...optimizers/trijoint.py.151: total_norm: 55.384380 [I 2022-10-27 13:25:46] ...trap/engines/engine.py.224: train: epoch 69 | batch 1510/2304 [I 2022-10-27 13:25:46] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:26 [I 2022-10-27 13:25:46] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00245 [I 2022-10-27 13:25:46] ...trap/engines/engine.py.230: loss: 0.11396 [I 2022-10-27 13:25:46] ...optimizers/trijoint.py.151: total_norm: 16.727533 [I 2022-10-27 13:25:51] ...trap/engines/engine.py.224: train: epoch 69 | batch 1520/2304 [I 2022-10-27 13:25:51] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:15 [I 2022-10-27 13:25:51] ...trap/engines/engine.py.229: process: 0.54714 | load: 0.00248 [I 2022-10-27 13:25:51] ...trap/engines/engine.py.230: loss: 0.11542 [I 2022-10-27 13:25:51] ...optimizers/trijoint.py.151: total_norm: 11.471300 [I 2022-10-27 13:25:57] ...trap/engines/engine.py.224: train: epoch 69 | batch 1530/2304 [I 2022-10-27 13:25:57] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:11 [I 2022-10-27 13:25:57] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00244 [I 2022-10-27 13:25:57] ...trap/engines/engine.py.230: loss: 0.12541 [I 2022-10-27 13:25:57] ...optimizers/trijoint.py.151: total_norm: 18.989983 [I 2022-10-27 13:26:03] ...trap/engines/engine.py.224: train: epoch 69 | batch 1540/2304 [I 2022-10-27 13:26:03] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:05 [I 2022-10-27 13:26:03] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00245 [I 2022-10-27 13:26:03] ...trap/engines/engine.py.230: loss: 0.16737 [I 2022-10-27 13:26:03] ...optimizers/trijoint.py.151: total_norm: 13.928629 [I 2022-10-27 13:26:08] ...trap/engines/engine.py.224: train: epoch 69 | batch 1550/2304 [I 2022-10-27 13:26:08] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:01 [I 2022-10-27 13:26:08] ...trap/engines/engine.py.229: process: 0.56546 | load: 0.00252 [I 2022-10-27 13:26:08] ...trap/engines/engine.py.230: loss: 0.17714 [I 2022-10-27 13:26:08] ...optimizers/trijoint.py.151: total_norm: 27.449404 [I 2022-10-27 13:26:14] ...trap/engines/engine.py.224: train: epoch 69 | batch 1560/2304 [I 2022-10-27 13:26:14] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:06:51 [I 2022-10-27 13:26:14] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00250 [I 2022-10-27 13:26:14] ...trap/engines/engine.py.230: loss: 0.10738 [I 2022-10-27 13:26:14] ...optimizers/trijoint.py.151: total_norm: 20.733027 [I 2022-10-27 13:26:19] ...trap/engines/engine.py.224: train: epoch 69 | batch 1570/2304 [I 2022-10-27 13:26:19] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:06:50 [I 2022-10-27 13:26:19] ...trap/engines/engine.py.229: process: 0.54714 | load: 0.00245 [I 2022-10-27 13:26:19] ...trap/engines/engine.py.230: loss: 0.11434 [I 2022-10-27 13:26:19] ...optimizers/trijoint.py.151: total_norm: 12.309097 [I 2022-10-27 13:26:25] ...trap/engines/engine.py.224: train: epoch 69 | batch 1580/2304 [I 2022-10-27 13:26:25] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:53 [I 2022-10-27 13:26:25] ...trap/engines/engine.py.229: process: 0.56155 | load: 0.00245 [I 2022-10-27 13:26:25] ...trap/engines/engine.py.230: loss: 0.13122 [I 2022-10-27 13:26:25] ...optimizers/trijoint.py.151: total_norm: 17.894314 [I 2022-10-27 13:26:31] ...trap/engines/engine.py.224: train: epoch 69 | batch 1590/2304 [I 2022-10-27 13:26:31] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:37 [I 2022-10-27 13:26:31] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00250 [I 2022-10-27 13:26:31] ...trap/engines/engine.py.230: loss: 0.15422 [I 2022-10-27 13:26:31] ...optimizers/trijoint.py.151: total_norm: 9.385995 [I 2022-10-27 13:26:36] ...trap/engines/engine.py.224: train: epoch 69 | batch 1600/2304 [I 2022-10-27 13:26:36] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:31 [I 2022-10-27 13:26:36] ...trap/engines/engine.py.229: process: 0.55583 | load: 0.00243 [I 2022-10-27 13:26:36] ...trap/engines/engine.py.230: loss: 0.17446 [I 2022-10-27 13:26:36] ...optimizers/trijoint.py.151: total_norm: 14.740729 [I 2022-10-27 13:26:42] ...trap/engines/engine.py.224: train: epoch 69 | batch 1610/2304 [I 2022-10-27 13:26:42] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:24 [I 2022-10-27 13:26:42] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00241 [I 2022-10-27 13:26:42] ...trap/engines/engine.py.230: loss: 0.13670 [I 2022-10-27 13:26:42] ...optimizers/trijoint.py.151: total_norm: 17.454618 [I 2022-10-27 13:26:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 1620/2304 [I 2022-10-27 13:26:47] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:20 [I 2022-10-27 13:26:47] ...trap/engines/engine.py.229: process: 0.55361 | load: 0.00244 [I 2022-10-27 13:26:47] ...trap/engines/engine.py.230: loss: 0.18706 [I 2022-10-27 13:26:47] ...optimizers/trijoint.py.151: total_norm: 12.069125 [I 2022-10-27 13:26:53] ...trap/engines/engine.py.224: train: epoch 69 | batch 1630/2304 [I 2022-10-27 13:26:53] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:16 [I 2022-10-27 13:26:53] ...trap/engines/engine.py.229: process: 0.55926 | load: 0.00243 [I 2022-10-27 13:26:53] ...trap/engines/engine.py.230: loss: 0.13601 [I 2022-10-27 13:26:53] ...optimizers/trijoint.py.151: total_norm: 28.745390 [I 2022-10-27 13:26:59] ...trap/engines/engine.py.224: train: epoch 69 | batch 1640/2304 [I 2022-10-27 13:26:59] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:12 [I 2022-10-27 13:26:59] ...trap/engines/engine.py.229: process: 0.56854 | load: 0.00244 [I 2022-10-27 13:26:59] ...trap/engines/engine.py.230: loss: 0.11578 [I 2022-10-27 13:26:59] ...optimizers/trijoint.py.151: total_norm: 14.784038 [I 2022-10-27 13:27:04] ...trap/engines/engine.py.224: train: epoch 69 | batch 1650/2304 [I 2022-10-27 13:27:04] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:04 [I 2022-10-27 13:27:04] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00245 [I 2022-10-27 13:27:04] ...trap/engines/engine.py.230: loss: 0.16353 [I 2022-10-27 13:27:04] ...optimizers/trijoint.py.151: total_norm: 13.331346 [I 2022-10-27 13:27:10] ...trap/engines/engine.py.224: train: epoch 69 | batch 1660/2304 [I 2022-10-27 13:27:10] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:04 [I 2022-10-27 13:27:10] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00252 [I 2022-10-27 13:27:10] ...trap/engines/engine.py.230: loss: 0.10859 [I 2022-10-27 13:27:10] ...optimizers/trijoint.py.151: total_norm: 27.571404 [I 2022-10-27 13:27:16] ...trap/engines/engine.py.224: train: epoch 69 | batch 1670/2304 [I 2022-10-27 13:27:16] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:05:53 [I 2022-10-27 13:27:16] ...trap/engines/engine.py.229: process: 0.56193 | load: 0.00247 [I 2022-10-27 13:27:16] ...trap/engines/engine.py.230: loss: 0.18090 [I 2022-10-27 13:27:16] ...optimizers/trijoint.py.151: total_norm: 30.827148 [I 2022-10-27 13:27:21] ...trap/engines/engine.py.224: train: epoch 69 | batch 1680/2304 [I 2022-10-27 13:27:21] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:48 [I 2022-10-27 13:27:21] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00248 [I 2022-10-27 13:27:21] ...trap/engines/engine.py.230: loss: 0.11827 [I 2022-10-27 13:27:21] ...optimizers/trijoint.py.151: total_norm: 14.756624 [I 2022-10-27 13:27:27] ...trap/engines/engine.py.224: train: epoch 69 | batch 1690/2304 [I 2022-10-27 13:27:27] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:05:43 [I 2022-10-27 13:27:27] ...trap/engines/engine.py.229: process: 0.55171 | load: 0.00246 [I 2022-10-27 13:27:27] ...trap/engines/engine.py.230: loss: 0.13366 [I 2022-10-27 13:27:27] ...optimizers/trijoint.py.151: total_norm: 11.237252 [I 2022-10-27 13:27:32] ...trap/engines/engine.py.224: train: epoch 69 | batch 1700/2304 [I 2022-10-27 13:27:32] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:38 [I 2022-10-27 13:27:32] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00244 [I 2022-10-27 13:27:32] ...trap/engines/engine.py.230: loss: 0.13901 [I 2022-10-27 13:27:32] ...optimizers/trijoint.py.151: total_norm: 14.052777 [I 2022-10-27 13:27:38] ...trap/engines/engine.py.224: train: epoch 69 | batch 1710/2304 [I 2022-10-27 13:27:38] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:29 [I 2022-10-27 13:27:38] ...trap/engines/engine.py.229: process: 0.54218 | load: 0.00243 [I 2022-10-27 13:27:38] ...trap/engines/engine.py.230: loss: 0.14298 [I 2022-10-27 13:27:38] ...optimizers/trijoint.py.151: total_norm: 17.884739 [I 2022-10-27 13:27:44] ...trap/engines/engine.py.224: train: epoch 69 | batch 1720/2304 [I 2022-10-27 13:27:44] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:29 [I 2022-10-27 13:27:44] ...trap/engines/engine.py.229: process: 0.54866 | load: 0.00245 [I 2022-10-27 13:27:44] ...trap/engines/engine.py.230: loss: 0.14052 [I 2022-10-27 13:27:44] ...optimizers/trijoint.py.151: total_norm: 21.555925 [I 2022-10-27 13:27:49] ...trap/engines/engine.py.224: train: epoch 69 | batch 1730/2304 [I 2022-10-27 13:27:49] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:21 [I 2022-10-27 13:27:49] ...trap/engines/engine.py.229: process: 0.56119 | load: 0.00251 [I 2022-10-27 13:27:49] ...trap/engines/engine.py.230: loss: 0.10138 [I 2022-10-27 13:27:49] ...optimizers/trijoint.py.151: total_norm: 11.785246 [I 2022-10-27 13:27:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 1740/2304 [I 2022-10-27 13:27:55] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:15 [I 2022-10-27 13:27:55] ...trap/engines/engine.py.229: process: 0.56490 | load: 0.00248 [I 2022-10-27 13:27:55] ...trap/engines/engine.py.230: loss: 0.10234 [I 2022-10-27 13:27:55] ...optimizers/trijoint.py.151: total_norm: 30.948921 [I 2022-10-27 13:28:00] ...trap/engines/engine.py.224: train: epoch 69 | batch 1750/2304 [I 2022-10-27 13:28:00] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:11 [I 2022-10-27 13:28:00] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00246 [I 2022-10-27 13:28:00] ...trap/engines/engine.py.230: loss: 0.15912 [I 2022-10-27 13:28:00] ...optimizers/trijoint.py.151: total_norm: 15.984959 [I 2022-10-27 13:28:06] ...trap/engines/engine.py.224: train: epoch 69 | batch 1760/2304 [I 2022-10-27 13:28:06] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:01 [I 2022-10-27 13:28:06] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00247 [I 2022-10-27 13:28:06] ...trap/engines/engine.py.230: loss: 0.11770 [I 2022-10-27 13:28:06] ...optimizers/trijoint.py.151: total_norm: 23.192701 [I 2022-10-27 13:28:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 1770/2304 [I 2022-10-27 13:28:12] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:04:56 [I 2022-10-27 13:28:12] ...trap/engines/engine.py.229: process: 0.55237 | load: 0.00242 [I 2022-10-27 13:28:12] ...trap/engines/engine.py.230: loss: 0.14005 [I 2022-10-27 13:28:12] ...optimizers/trijoint.py.151: total_norm: 21.972868 [I 2022-10-27 13:28:17] ...trap/engines/engine.py.224: train: epoch 69 | batch 1780/2304 [I 2022-10-27 13:28:17] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:04:51 [I 2022-10-27 13:28:17] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00244 [I 2022-10-27 13:28:17] ...trap/engines/engine.py.230: loss: 0.13203 [I 2022-10-27 13:28:17] ...optimizers/trijoint.py.151: total_norm: 19.975574 [I 2022-10-27 13:28:23] ...trap/engines/engine.py.224: train: epoch 69 | batch 1790/2304 [I 2022-10-27 13:28:23] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:45 [I 2022-10-27 13:28:23] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00241 [I 2022-10-27 13:28:23] ...trap/engines/engine.py.230: loss: 0.13035 [I 2022-10-27 13:28:23] ...optimizers/trijoint.py.151: total_norm: 22.435640 [I 2022-10-27 13:28:28] ...trap/engines/engine.py.224: train: epoch 69 | batch 1800/2304 [I 2022-10-27 13:28:28] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:04:41 [I 2022-10-27 13:28:28] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00243 [I 2022-10-27 13:28:28] ...trap/engines/engine.py.230: loss: 0.35408 [I 2022-10-27 13:28:28] ...optimizers/trijoint.py.151: total_norm: 29.382490 [I 2022-10-27 13:28:34] ...trap/engines/engine.py.224: train: epoch 69 | batch 1810/2304 [I 2022-10-27 13:28:34] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:38 [I 2022-10-27 13:28:34] ...trap/engines/engine.py.229: process: 0.57048 | load: 0.00261 [I 2022-10-27 13:28:34] ...trap/engines/engine.py.230: loss: 0.16823 [I 2022-10-27 13:28:34] ...optimizers/trijoint.py.151: total_norm: 22.672773 [I 2022-10-27 13:28:40] ...trap/engines/engine.py.224: train: epoch 69 | batch 1820/2304 [I 2022-10-27 13:28:40] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:27 [I 2022-10-27 13:28:40] ...trap/engines/engine.py.229: process: 0.53948 | load: 0.00243 [I 2022-10-27 13:28:40] ...trap/engines/engine.py.230: loss: 0.20982 [I 2022-10-27 13:28:40] ...optimizers/trijoint.py.151: total_norm: 14.930975 [I 2022-10-27 13:28:45] ...trap/engines/engine.py.224: train: epoch 69 | batch 1830/2304 [I 2022-10-27 13:28:45] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:23 [I 2022-10-27 13:28:45] ...trap/engines/engine.py.229: process: 0.56472 | load: 0.00254 [I 2022-10-27 13:28:45] ...trap/engines/engine.py.230: loss: 0.10356 [I 2022-10-27 13:28:45] ...optimizers/trijoint.py.151: total_norm: 17.473660 [I 2022-10-27 13:28:51] ...trap/engines/engine.py.224: train: epoch 69 | batch 1840/2304 [I 2022-10-27 13:28:51] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:17 [I 2022-10-27 13:28:51] ...trap/engines/engine.py.229: process: 0.54379 | load: 0.00245 [I 2022-10-27 13:28:51] ...trap/engines/engine.py.230: loss: 0.13204 [I 2022-10-27 13:28:51] ...optimizers/trijoint.py.151: total_norm: 11.911298 [I 2022-10-27 13:28:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 1850/2304 [I 2022-10-27 13:28:56] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:12 [I 2022-10-27 13:28:56] ...trap/engines/engine.py.229: process: 0.55338 | load: 0.00243 [I 2022-10-27 13:28:56] ...trap/engines/engine.py.230: loss: 0.14879 [I 2022-10-27 13:28:56] ...optimizers/trijoint.py.151: total_norm: 23.156885 [I 2022-10-27 13:29:02] ...trap/engines/engine.py.224: train: epoch 69 | batch 1860/2304 [I 2022-10-27 13:29:02] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:06 [I 2022-10-27 13:29:02] ...trap/engines/engine.py.229: process: 0.55202 | load: 0.00243 [I 2022-10-27 13:29:02] ...trap/engines/engine.py.230: loss: 0.11953 [I 2022-10-27 13:29:02] ...optimizers/trijoint.py.151: total_norm: 28.467751 [I 2022-10-27 13:29:07] ...trap/engines/engine.py.224: train: epoch 69 | batch 1870/2304 [I 2022-10-27 13:29:07] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:01 [I 2022-10-27 13:29:07] ...trap/engines/engine.py.229: process: 0.56339 | load: 0.00243 [I 2022-10-27 13:29:07] ...trap/engines/engine.py.230: loss: 0.11380 [I 2022-10-27 13:29:07] ...optimizers/trijoint.py.151: total_norm: 10.907282 [I 2022-10-27 13:29:13] ...trap/engines/engine.py.224: train: epoch 69 | batch 1880/2304 [I 2022-10-27 13:29:13] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:03:56 [I 2022-10-27 13:29:13] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00241 [I 2022-10-27 13:29:13] ...trap/engines/engine.py.230: loss: 0.13702 [I 2022-10-27 13:29:13] ...optimizers/trijoint.py.151: total_norm: 13.100310 [I 2022-10-27 13:29:19] ...trap/engines/engine.py.224: train: epoch 69 | batch 1890/2304 [I 2022-10-27 13:29:19] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:03:52 [I 2022-10-27 13:29:19] ...trap/engines/engine.py.229: process: 0.57774 | load: 0.00249 [I 2022-10-27 13:29:19] ...trap/engines/engine.py.230: loss: 0.14316 [I 2022-10-27 13:29:19] ...optimizers/trijoint.py.151: total_norm: 16.384741 [I 2022-10-27 13:29:24] ...trap/engines/engine.py.224: train: epoch 69 | batch 1900/2304 [I 2022-10-27 13:29:24] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:03:46 [I 2022-10-27 13:29:24] ...trap/engines/engine.py.229: process: 0.56702 | load: 0.00245 [I 2022-10-27 13:29:24] ...trap/engines/engine.py.230: loss: 0.16584 [I 2022-10-27 13:29:24] ...optimizers/trijoint.py.151: total_norm: 50.535088 [I 2022-10-27 13:29:30] ...trap/engines/engine.py.224: train: epoch 69 | batch 1910/2304 [I 2022-10-27 13:29:30] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:44 [I 2022-10-27 13:29:30] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00245 [I 2022-10-27 13:29:30] ...trap/engines/engine.py.230: loss: 0.17753 [I 2022-10-27 13:29:30] ...optimizers/trijoint.py.151: total_norm: 21.416821 [I 2022-10-27 13:29:35] ...trap/engines/engine.py.224: train: epoch 69 | batch 1920/2304 [I 2022-10-27 13:29:35] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:35 [I 2022-10-27 13:29:35] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00245 [I 2022-10-27 13:29:35] ...trap/engines/engine.py.230: loss: 0.14948 [I 2022-10-27 13:29:35] ...optimizers/trijoint.py.151: total_norm: 15.803103 [I 2022-10-27 13:29:41] ...trap/engines/engine.py.224: train: epoch 69 | batch 1930/2304 [I 2022-10-27 13:29:41] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:27 [I 2022-10-27 13:29:41] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00247 [I 2022-10-27 13:29:41] ...trap/engines/engine.py.230: loss: 0.17329 [I 2022-10-27 13:29:41] ...optimizers/trijoint.py.151: total_norm: 19.059353 [I 2022-10-27 13:29:47] ...trap/engines/engine.py.224: train: epoch 69 | batch 1940/2304 [I 2022-10-27 13:29:47] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:23 [I 2022-10-27 13:29:47] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00243 [I 2022-10-27 13:29:47] ...trap/engines/engine.py.230: loss: 0.17679 [I 2022-10-27 13:29:47] ...optimizers/trijoint.py.151: total_norm: 15.585312 [I 2022-10-27 13:29:52] ...trap/engines/engine.py.224: train: epoch 69 | batch 1950/2304 [I 2022-10-27 13:29:52] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:16 [I 2022-10-27 13:29:52] ...trap/engines/engine.py.229: process: 0.55214 | load: 0.00242 [I 2022-10-27 13:29:52] ...trap/engines/engine.py.230: loss: 0.14670 [I 2022-10-27 13:29:52] ...optimizers/trijoint.py.151: total_norm: 15.906084 [I 2022-10-27 13:29:58] ...trap/engines/engine.py.224: train: epoch 69 | batch 1960/2304 [I 2022-10-27 13:29:58] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:11 [I 2022-10-27 13:29:58] ...trap/engines/engine.py.229: process: 0.54939 | load: 0.00240 [I 2022-10-27 13:29:58] ...trap/engines/engine.py.230: loss: 0.09993 [I 2022-10-27 13:29:58] ...optimizers/trijoint.py.151: total_norm: 10.903139 [I 2022-10-27 13:30:03] ...trap/engines/engine.py.224: train: epoch 69 | batch 1970/2304 [I 2022-10-27 13:30:03] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:13 [I 2022-10-27 13:30:03] ...trap/engines/engine.py.229: process: 0.65828 | load: 0.00261 [I 2022-10-27 13:30:03] ...trap/engines/engine.py.230: loss: 0.18857 [I 2022-10-27 13:30:03] ...optimizers/trijoint.py.151: total_norm: 17.316074 [I 2022-10-27 13:30:09] ...trap/engines/engine.py.224: train: epoch 69 | batch 1980/2304 [I 2022-10-27 13:30:09] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:00 [I 2022-10-27 13:30:09] ...trap/engines/engine.py.229: process: 0.55089 | load: 0.00248 [I 2022-10-27 13:30:09] ...trap/engines/engine.py.230: loss: 0.09315 [I 2022-10-27 13:30:09] ...optimizers/trijoint.py.151: total_norm: 17.890303 [I 2022-10-27 13:30:15] ...trap/engines/engine.py.224: train: epoch 69 | batch 1990/2304 [I 2022-10-27 13:30:15] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:02:54 [I 2022-10-27 13:30:15] ...trap/engines/engine.py.229: process: 0.54947 | load: 0.00243 [I 2022-10-27 13:30:15] ...trap/engines/engine.py.230: loss: 0.16395 [I 2022-10-27 13:30:15] ...optimizers/trijoint.py.151: total_norm: 42.040333 [I 2022-10-27 13:30:20] ...trap/engines/engine.py.224: train: epoch 69 | batch 2000/2304 [I 2022-10-27 13:30:20] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:02:49 [I 2022-10-27 13:30:20] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00246 [I 2022-10-27 13:30:20] ...trap/engines/engine.py.230: loss: 0.11932 [I 2022-10-27 13:30:20] ...optimizers/trijoint.py.151: total_norm: 11.890373 [I 2022-10-27 13:30:26] ...trap/engines/engine.py.224: train: epoch 69 | batch 2010/2304 [I 2022-10-27 13:30:26] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:02:44 [I 2022-10-27 13:30:26] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00260 [I 2022-10-27 13:30:26] ...trap/engines/engine.py.230: loss: 0.12196 [I 2022-10-27 13:30:26] ...optimizers/trijoint.py.151: total_norm: 9.862698 [I 2022-10-27 13:30:31] ...trap/engines/engine.py.224: train: epoch 69 | batch 2020/2304 [I 2022-10-27 13:30:31] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:36 [I 2022-10-27 13:30:31] ...trap/engines/engine.py.229: process: 0.55157 | load: 0.00252 [I 2022-10-27 13:30:31] ...trap/engines/engine.py.230: loss: 0.23915 [I 2022-10-27 13:30:31] ...optimizers/trijoint.py.151: total_norm: 11.954651 [I 2022-10-27 13:30:37] ...trap/engines/engine.py.224: train: epoch 69 | batch 2030/2304 [I 2022-10-27 13:30:37] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:31 [I 2022-10-27 13:30:37] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00254 [I 2022-10-27 13:30:37] ...trap/engines/engine.py.230: loss: 0.21747 [I 2022-10-27 13:30:37] ...optimizers/trijoint.py.151: total_norm: 33.559036 [I 2022-10-27 13:30:42] ...trap/engines/engine.py.224: train: epoch 69 | batch 2040/2304 [I 2022-10-27 13:30:42] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:26 [I 2022-10-27 13:30:42] ...trap/engines/engine.py.229: process: 0.55031 | load: 0.00244 [I 2022-10-27 13:30:42] ...trap/engines/engine.py.230: loss: 0.20578 [I 2022-10-27 13:30:42] ...optimizers/trijoint.py.151: total_norm: 20.720760 [I 2022-10-27 13:30:48] ...trap/engines/engine.py.224: train: epoch 69 | batch 2050/2304 [I 2022-10-27 13:30:48] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:21 [I 2022-10-27 13:30:48] ...trap/engines/engine.py.229: process: 0.54512 | load: 0.00247 [I 2022-10-27 13:30:48] ...trap/engines/engine.py.230: loss: 0.14007 [I 2022-10-27 13:30:48] ...optimizers/trijoint.py.151: total_norm: 18.015295 [I 2022-10-27 13:30:54] ...trap/engines/engine.py.224: train: epoch 69 | batch 2060/2304 [I 2022-10-27 13:30:54] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:16 [I 2022-10-27 13:30:54] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00244 [I 2022-10-27 13:30:54] ...trap/engines/engine.py.230: loss: 0.21283 [I 2022-10-27 13:30:54] ...optimizers/trijoint.py.151: total_norm: 22.988249 [I 2022-10-27 13:30:59] ...trap/engines/engine.py.224: train: epoch 69 | batch 2070/2304 [I 2022-10-27 13:30:59] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:09 [I 2022-10-27 13:30:59] ...trap/engines/engine.py.229: process: 0.55793 | load: 0.00241 [I 2022-10-27 13:30:59] ...trap/engines/engine.py.230: loss: 0.13188 [I 2022-10-27 13:30:59] ...optimizers/trijoint.py.151: total_norm: 14.349775 [I 2022-10-27 13:31:05] ...trap/engines/engine.py.224: train: epoch 69 | batch 2080/2304 [I 2022-10-27 13:31:05] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:04 [I 2022-10-27 13:31:05] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00242 [I 2022-10-27 13:31:05] ...trap/engines/engine.py.230: loss: 0.09842 [I 2022-10-27 13:31:05] ...optimizers/trijoint.py.151: total_norm: 9.732932 [I 2022-10-27 13:31:10] ...trap/engines/engine.py.224: train: epoch 69 | batch 2090/2304 [I 2022-10-27 13:31:10] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:01:58 [I 2022-10-27 13:31:10] ...trap/engines/engine.py.229: process: 0.54597 | load: 0.00241 [I 2022-10-27 13:31:10] ...trap/engines/engine.py.230: loss: 0.10572 [I 2022-10-27 13:31:10] ...optimizers/trijoint.py.151: total_norm: 10.593340 [I 2022-10-27 13:31:16] ...trap/engines/engine.py.224: train: epoch 69 | batch 2100/2304 [I 2022-10-27 13:31:16] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:01:56 [I 2022-10-27 13:31:16] ...trap/engines/engine.py.229: process: 0.54969 | load: 0.00240 [I 2022-10-27 13:31:16] ...trap/engines/engine.py.230: loss: 0.09299 [I 2022-10-27 13:31:16] ...optimizers/trijoint.py.151: total_norm: 13.200072 [I 2022-10-27 13:31:22] ...trap/engines/engine.py.224: train: epoch 69 | batch 2110/2304 [I 2022-10-27 13:31:22] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:01:47 [I 2022-10-27 13:31:22] ...trap/engines/engine.py.229: process: 0.53850 | load: 0.00243 [I 2022-10-27 13:31:22] ...trap/engines/engine.py.230: loss: 0.12549 [I 2022-10-27 13:31:22] ...optimizers/trijoint.py.151: total_norm: 7.566174 [I 2022-10-27 13:31:27] ...trap/engines/engine.py.224: train: epoch 69 | batch 2120/2304 [I 2022-10-27 13:31:27] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:01:41 [I 2022-10-27 13:31:27] ...trap/engines/engine.py.229: process: 0.55547 | load: 0.00248 [I 2022-10-27 13:31:27] ...trap/engines/engine.py.230: loss: 0.14696 [I 2022-10-27 13:31:27] ...optimizers/trijoint.py.151: total_norm: 16.981705 [I 2022-10-27 13:31:33] ...trap/engines/engine.py.224: train: epoch 69 | batch 2130/2304 [I 2022-10-27 13:31:33] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:36 [I 2022-10-27 13:31:33] ...trap/engines/engine.py.229: process: 0.54410 | load: 0.00249 [I 2022-10-27 13:31:33] ...trap/engines/engine.py.230: loss: 0.17366 [I 2022-10-27 13:31:33] ...optimizers/trijoint.py.151: total_norm: 27.750175 [I 2022-10-27 13:31:38] ...trap/engines/engine.py.224: train: epoch 69 | batch 2140/2304 [I 2022-10-27 13:31:38] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:31 [I 2022-10-27 13:31:38] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00244 [I 2022-10-27 13:31:38] ...trap/engines/engine.py.230: loss: 0.12036 [I 2022-10-27 13:31:38] ...optimizers/trijoint.py.151: total_norm: 12.334156 [I 2022-10-27 13:31:44] ...trap/engines/engine.py.224: train: epoch 69 | batch 2150/2304 [I 2022-10-27 13:31:44] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:25 [I 2022-10-27 13:31:44] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00244 [I 2022-10-27 13:31:44] ...trap/engines/engine.py.230: loss: 0.12168 [I 2022-10-27 13:31:44] ...optimizers/trijoint.py.151: total_norm: 37.609024 [I 2022-10-27 13:31:49] ...trap/engines/engine.py.224: train: epoch 69 | batch 2160/2304 [I 2022-10-27 13:31:49] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:19 [I 2022-10-27 13:31:49] ...trap/engines/engine.py.229: process: 0.54355 | load: 0.00244 [I 2022-10-27 13:31:49] ...trap/engines/engine.py.230: loss: 0.11781 [I 2022-10-27 13:31:49] ...optimizers/trijoint.py.151: total_norm: 16.279987 [I 2022-10-27 13:31:55] ...trap/engines/engine.py.224: train: epoch 69 | batch 2170/2304 [I 2022-10-27 13:31:55] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:14 [I 2022-10-27 13:31:55] ...trap/engines/engine.py.229: process: 0.54824 | load: 0.00243 [I 2022-10-27 13:31:55] ...trap/engines/engine.py.230: loss: 0.12128 [I 2022-10-27 13:31:55] ...optimizers/trijoint.py.151: total_norm: 12.948967 [I 2022-10-27 13:32:01] ...trap/engines/engine.py.224: train: epoch 69 | batch 2180/2304 [I 2022-10-27 13:32:01] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:09 [I 2022-10-27 13:32:01] ...trap/engines/engine.py.229: process: 0.55471 | load: 0.00249 [I 2022-10-27 13:32:01] ...trap/engines/engine.py.230: loss: 0.13969 [I 2022-10-27 13:32:01] ...optimizers/trijoint.py.151: total_norm: 14.598560 [I 2022-10-27 13:32:06] ...trap/engines/engine.py.224: train: epoch 69 | batch 2190/2304 [I 2022-10-27 13:32:06] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:03 [I 2022-10-27 13:32:06] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00244 [I 2022-10-27 13:32:06] ...trap/engines/engine.py.230: loss: 0.12059 [I 2022-10-27 13:32:06] ...optimizers/trijoint.py.151: total_norm: 9.695497 [I 2022-10-27 13:32:12] ...trap/engines/engine.py.224: train: epoch 69 | batch 2200/2304 [I 2022-10-27 13:32:12] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:00:57 [I 2022-10-27 13:32:12] ...trap/engines/engine.py.229: process: 0.56487 | load: 0.00243 [I 2022-10-27 13:32:12] ...trap/engines/engine.py.230: loss: 0.13489 [I 2022-10-27 13:32:12] ...optimizers/trijoint.py.151: total_norm: 13.304709 [I 2022-10-27 13:32:17] ...trap/engines/engine.py.224: train: epoch 69 | batch 2210/2304 [I 2022-10-27 13:32:17] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:00:52 [I 2022-10-27 13:32:17] ...trap/engines/engine.py.229: process: 0.56538 | load: 0.00241 [I 2022-10-27 13:32:17] ...trap/engines/engine.py.230: loss: 0.16546 [I 2022-10-27 13:32:17] ...optimizers/trijoint.py.151: total_norm: 20.068464 [I 2022-10-27 13:32:23] ...trap/engines/engine.py.224: train: epoch 69 | batch 2220/2304 [I 2022-10-27 13:32:23] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:00:47 [I 2022-10-27 13:32:23] ...trap/engines/engine.py.229: process: 0.56340 | load: 0.00244 [I 2022-10-27 13:32:23] ...trap/engines/engine.py.230: loss: 0.20363 [I 2022-10-27 13:32:23] ...optimizers/trijoint.py.151: total_norm: 39.867062 [I 2022-10-27 13:32:28] ...trap/engines/engine.py.224: train: epoch 69 | batch 2230/2304 [I 2022-10-27 13:32:28] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:41 [I 2022-10-27 13:32:28] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00242 [I 2022-10-27 13:32:28] ...trap/engines/engine.py.230: loss: 0.11867 [I 2022-10-27 13:32:28] ...optimizers/trijoint.py.151: total_norm: 20.808090 [I 2022-10-27 13:32:34] ...trap/engines/engine.py.224: train: epoch 69 | batch 2240/2304 [I 2022-10-27 13:32:34] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:35 [I 2022-10-27 13:32:34] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00244 [I 2022-10-27 13:32:34] ...trap/engines/engine.py.230: loss: 0.12714 [I 2022-10-27 13:32:34] ...optimizers/trijoint.py.151: total_norm: 15.364117 [I 2022-10-27 13:32:40] ...trap/engines/engine.py.224: train: epoch 69 | batch 2250/2304 [I 2022-10-27 13:32:40] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:29 [I 2022-10-27 13:32:40] ...trap/engines/engine.py.229: process: 0.55480 | load: 0.00256 [I 2022-10-27 13:32:40] ...trap/engines/engine.py.230: loss: 0.20397 [I 2022-10-27 13:32:40] ...optimizers/trijoint.py.151: total_norm: 16.721674 [I 2022-10-27 13:32:45] ...trap/engines/engine.py.224: train: epoch 69 | batch 2260/2304 [I 2022-10-27 13:32:45] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:24 [I 2022-10-27 13:32:45] ...trap/engines/engine.py.229: process: 0.56025 | load: 0.00247 [I 2022-10-27 13:32:45] ...trap/engines/engine.py.230: loss: 0.19004 [I 2022-10-27 13:32:45] ...optimizers/trijoint.py.151: total_norm: 22.975716 [I 2022-10-27 13:32:51] ...trap/engines/engine.py.224: train: epoch 69 | batch 2270/2304 [I 2022-10-27 13:32:51] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:19 [I 2022-10-27 13:32:51] ...trap/engines/engine.py.229: process: 0.56789 | load: 0.00294 [I 2022-10-27 13:32:51] ...trap/engines/engine.py.230: loss: 0.17275 [I 2022-10-27 13:32:51] ...optimizers/trijoint.py.151: total_norm: 31.846355 [I 2022-10-27 13:32:56] ...trap/engines/engine.py.224: train: epoch 69 | batch 2280/2304 [I 2022-10-27 13:32:56] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:13 [I 2022-10-27 13:32:56] ...trap/engines/engine.py.229: process: 0.54036 | load: 0.00243 [I 2022-10-27 13:32:56] ...trap/engines/engine.py.230: loss: 0.13375 [I 2022-10-27 13:32:56] ...optimizers/trijoint.py.151: total_norm: 20.479427 [I 2022-10-27 13:33:02] ...trap/engines/engine.py.224: train: epoch 69 | batch 2290/2304 [I 2022-10-27 13:33:02] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:07 [I 2022-10-27 13:33:02] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00244 [I 2022-10-27 13:33:02] ...trap/engines/engine.py.230: loss: 0.11374 [I 2022-10-27 13:33:02] ...optimizers/trijoint.py.151: total_norm: 8.344377 [I 2022-10-27 13:33:08] ...trap/engines/engine.py.224: train: epoch 69 | batch 2300/2304 [I 2022-10-27 13:33:08] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:02 [I 2022-10-27 13:33:08] ...trap/engines/engine.py.229: process: 0.55048 | load: 0.00015 [I 2022-10-27 13:33:08] ...trap/engines/engine.py.230: loss: 0.11359 [I 2022-10-27 13:33:08] ...optimizers/trijoint.py.151: total_norm: 17.054609 [I 2022-10-27 13:33:10] ...trap/engines/engine.py.224: train: epoch 69 | batch 2304/2304 [I 2022-10-27 13:33:10] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:00 [I 2022-10-27 13:33:10] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00015 [I 2022-10-27 13:33:10] ...trap/engines/engine.py.230: loss: 0.19569 [I 2022-10-27 13:33:10] ...optimizers/trijoint.py.151: total_norm: 13.057689 [S 2022-10-27 13:33:11] ...trap/engines/engine.py.129: train_epoch.epoch: 69 [S 2022-10-27 13:33:11] ...trap/engines/engine.py.129: train_epoch.loss: 0.14696535188502707 [I 2022-10-27 13:33:11] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 13:33:12] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 69 [I 2022-10-27 13:33:12] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 13:33:14] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 13:33:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 13:33:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 13:33:16] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 13:33:16] ...trap/engines/engine.py.321: eval: epoch 69 | batch 0/510 [I 2022-10-27 13:33:16] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:38:30 [I 2022-10-27 13:33:16] ...trap/engines/engine.py.326: process: 4.53010 | load: 4.15329 [I 2022-10-27 13:33:20] ...trap/engines/engine.py.321: eval: epoch 69 | batch 10/510 [I 2022-10-27 13:33:20] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:33 [I 2022-10-27 13:33:20] ...trap/engines/engine.py.326: process: 0.31656 | load: 0.00028 [I 2022-10-27 13:33:23] ...trap/engines/engine.py.321: eval: epoch 69 | batch 20/510 [I 2022-10-27 13:33:23] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:09 [I 2022-10-27 13:33:23] ...trap/engines/engine.py.326: process: 0.34174 | load: 0.00027 [I 2022-10-27 13:33:27] ...trap/engines/engine.py.321: eval: epoch 69 | batch 30/510 [I 2022-10-27 13:33:27] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:46 [I 2022-10-27 13:33:27] ...trap/engines/engine.py.326: process: 0.34002 | load: 0.00021 [I 2022-10-27 13:33:30] ...trap/engines/engine.py.321: eval: epoch 69 | batch 40/510 [I 2022-10-27 13:33:30] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:40 [I 2022-10-27 13:33:30] ...trap/engines/engine.py.326: process: 0.33556 | load: 0.00020 [I 2022-10-27 13:33:33] ...trap/engines/engine.py.321: eval: epoch 69 | batch 50/510 [I 2022-10-27 13:33:33] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:34 [I 2022-10-27 13:33:33] ...trap/engines/engine.py.326: process: 0.32977 | load: 0.00017 [I 2022-10-27 13:33:37] ...trap/engines/engine.py.321: eval: epoch 69 | batch 60/510 [I 2022-10-27 13:33:37] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:31 [I 2022-10-27 13:33:37] ...trap/engines/engine.py.326: process: 0.33769 | load: 0.00016 [I 2022-10-27 13:33:40] ...trap/engines/engine.py.321: eval: epoch 69 | batch 70/510 [I 2022-10-27 13:33:40] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:26 [I 2022-10-27 13:33:40] ...trap/engines/engine.py.326: process: 0.33456 | load: 0.00019 [I 2022-10-27 13:33:43] ...trap/engines/engine.py.321: eval: epoch 69 | batch 80/510 [I 2022-10-27 13:33:43] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:24 [I 2022-10-27 13:33:43] ...trap/engines/engine.py.326: process: 0.33774 | load: 0.00017 [I 2022-10-27 13:33:47] ...trap/engines/engine.py.321: eval: epoch 69 | batch 90/510 [I 2022-10-27 13:33:47] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:20 [I 2022-10-27 13:33:47] ...trap/engines/engine.py.326: process: 0.32512 | load: 0.00018 [I 2022-10-27 13:33:50] ...trap/engines/engine.py.321: eval: epoch 69 | batch 100/510 [I 2022-10-27 13:33:50] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-27 13:33:50] ...trap/engines/engine.py.326: process: 0.29145 | load: 0.00016 [I 2022-10-27 13:33:53] ...trap/engines/engine.py.321: eval: epoch 69 | batch 110/510 [I 2022-10-27 13:33:53] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:58 [I 2022-10-27 13:33:53] ...trap/engines/engine.py.326: process: 0.30622 | load: 0.00015 [I 2022-10-27 13:33:56] ...trap/engines/engine.py.321: eval: epoch 69 | batch 120/510 [I 2022-10-27 13:33:56] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:54 [I 2022-10-27 13:33:56] ...trap/engines/engine.py.326: process: 0.30768 | load: 0.00014 [I 2022-10-27 13:33:59] ...trap/engines/engine.py.321: eval: epoch 69 | batch 130/510 [I 2022-10-27 13:33:59] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:51 [I 2022-10-27 13:33:59] ...trap/engines/engine.py.326: process: 0.30406 | load: 0.00015 [I 2022-10-27 13:34:02] ...trap/engines/engine.py.321: eval: epoch 69 | batch 140/510 [I 2022-10-27 13:34:02] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:46 [I 2022-10-27 13:34:02] ...trap/engines/engine.py.326: process: 0.28528 | load: 0.00014 [I 2022-10-27 13:34:05] ...trap/engines/engine.py.321: eval: epoch 69 | batch 150/510 [I 2022-10-27 13:34:05] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:43 [I 2022-10-27 13:34:05] ...trap/engines/engine.py.326: process: 0.28818 | load: 0.00014 [I 2022-10-27 13:34:07] ...trap/engines/engine.py.321: eval: epoch 69 | batch 160/510 [I 2022-10-27 13:34:07] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 13:34:07] ...trap/engines/engine.py.326: process: 0.29152 | load: 0.00018 [I 2022-10-27 13:34:10] ...trap/engines/engine.py.321: eval: epoch 69 | batch 170/510 [I 2022-10-27 13:34:10] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:37 [I 2022-10-27 13:34:10] ...trap/engines/engine.py.326: process: 0.27599 | load: 0.00014 [I 2022-10-27 13:34:13] ...trap/engines/engine.py.321: eval: epoch 69 | batch 180/510 [I 2022-10-27 13:34:13] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 13:34:13] ...trap/engines/engine.py.326: process: 0.29361 | load: 0.00014 [I 2022-10-27 13:34:16] ...trap/engines/engine.py.321: eval: epoch 69 | batch 190/510 [I 2022-10-27 13:34:16] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:31 [I 2022-10-27 13:34:16] ...trap/engines/engine.py.326: process: 0.28310 | load: 0.00018 [I 2022-10-27 13:34:19] ...trap/engines/engine.py.321: eval: epoch 69 | batch 200/510 [I 2022-10-27 13:34:19] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:28 [I 2022-10-27 13:34:19] ...trap/engines/engine.py.326: process: 0.28436 | load: 0.00015 [I 2022-10-27 13:34:22] ...trap/engines/engine.py.321: eval: epoch 69 | batch 210/510 [I 2022-10-27 13:34:22] ...trap/engines/engine.py.325: elapsed: 0:01:09 | left: 0:01:26 [I 2022-10-27 13:34:22] ...trap/engines/engine.py.326: process: 0.28934 | load: 0.00020 [I 2022-10-27 13:34:25] ...trap/engines/engine.py.321: eval: epoch 69 | batch 220/510 [I 2022-10-27 13:34:25] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:24 [I 2022-10-27 13:34:25] ...trap/engines/engine.py.326: process: 0.29416 | load: 0.00019 [I 2022-10-27 13:34:28] ...trap/engines/engine.py.321: eval: epoch 69 | batch 230/510 [I 2022-10-27 13:34:28] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:21 [I 2022-10-27 13:34:28] ...trap/engines/engine.py.326: process: 0.29324 | load: 0.00032 [I 2022-10-27 13:34:30] ...trap/engines/engine.py.321: eval: epoch 69 | batch 240/510 [I 2022-10-27 13:34:30] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 13:34:30] ...trap/engines/engine.py.326: process: 0.30113 | load: 0.00015 [I 2022-10-27 13:34:33] ...trap/engines/engine.py.321: eval: epoch 69 | batch 250/510 [I 2022-10-27 13:34:33] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 13:34:33] ...trap/engines/engine.py.326: process: 0.28477 | load: 0.00023 [I 2022-10-27 13:34:36] ...trap/engines/engine.py.321: eval: epoch 69 | batch 260/510 [I 2022-10-27 13:34:36] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 13:34:36] ...trap/engines/engine.py.326: process: 0.28924 | load: 0.00016 [I 2022-10-27 13:34:39] ...trap/engines/engine.py.321: eval: epoch 69 | batch 270/510 [I 2022-10-27 13:34:39] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 13:34:39] ...trap/engines/engine.py.326: process: 0.28666 | load: 0.00017 [I 2022-10-27 13:34:42] ...trap/engines/engine.py.321: eval: epoch 69 | batch 280/510 [I 2022-10-27 13:34:42] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 13:34:42] ...trap/engines/engine.py.326: process: 0.29250 | load: 0.00020 [I 2022-10-27 13:34:45] ...trap/engines/engine.py.321: eval: epoch 69 | batch 290/510 [I 2022-10-27 13:34:45] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:02 [I 2022-10-27 13:34:45] ...trap/engines/engine.py.326: process: 0.27806 | load: 0.00018 [I 2022-10-27 13:34:48] ...trap/engines/engine.py.321: eval: epoch 69 | batch 300/510 [I 2022-10-27 13:34:48] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 13:34:48] ...trap/engines/engine.py.326: process: 0.28220 | load: 0.00018 [I 2022-10-27 13:34:51] ...trap/engines/engine.py.321: eval: epoch 69 | batch 310/510 [I 2022-10-27 13:34:51] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 13:34:51] ...trap/engines/engine.py.326: process: 0.28267 | load: 0.00017 [I 2022-10-27 13:34:54] ...trap/engines/engine.py.321: eval: epoch 69 | batch 320/510 [I 2022-10-27 13:34:54] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 13:34:54] ...trap/engines/engine.py.326: process: 0.28752 | load: 0.00015 [I 2022-10-27 13:34:56] ...trap/engines/engine.py.321: eval: epoch 69 | batch 330/510 [I 2022-10-27 13:34:56] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 13:34:56] ...trap/engines/engine.py.326: process: 0.29026 | load: 0.00024 [I 2022-10-27 13:34:59] ...trap/engines/engine.py.321: eval: epoch 69 | batch 340/510 [I 2022-10-27 13:34:59] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-27 13:34:59] ...trap/engines/engine.py.326: process: 0.28975 | load: 0.00018 [I 2022-10-27 13:35:02] ...trap/engines/engine.py.321: eval: epoch 69 | batch 350/510 [I 2022-10-27 13:35:02] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 13:35:02] ...trap/engines/engine.py.326: process: 0.28740 | load: 0.00023 [I 2022-10-27 13:35:05] ...trap/engines/engine.py.321: eval: epoch 69 | batch 360/510 [I 2022-10-27 13:35:05] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 13:35:05] ...trap/engines/engine.py.326: process: 0.28643 | load: 0.00020 [I 2022-10-27 13:35:08] ...trap/engines/engine.py.321: eval: epoch 69 | batch 370/510 [I 2022-10-27 13:35:08] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 13:35:08] ...trap/engines/engine.py.326: process: 0.28985 | load: 0.00050 [I 2022-10-27 13:35:11] ...trap/engines/engine.py.321: eval: epoch 69 | batch 380/510 [I 2022-10-27 13:35:11] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 13:35:11] ...trap/engines/engine.py.326: process: 0.29055 | load: 0.00016 [I 2022-10-27 13:35:14] ...trap/engines/engine.py.321: eval: epoch 69 | batch 390/510 [I 2022-10-27 13:35:14] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:35 [I 2022-10-27 13:35:14] ...trap/engines/engine.py.326: process: 0.29382 | load: 0.00022 [I 2022-10-27 13:35:17] ...trap/engines/engine.py.321: eval: epoch 69 | batch 400/510 [I 2022-10-27 13:35:17] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:32 [I 2022-10-27 13:35:17] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00015 [I 2022-10-27 13:35:20] ...trap/engines/engine.py.321: eval: epoch 69 | batch 410/510 [I 2022-10-27 13:35:20] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 13:35:20] ...trap/engines/engine.py.326: process: 0.28180 | load: 0.00025 [I 2022-10-27 13:35:23] ...trap/engines/engine.py.321: eval: epoch 69 | batch 420/510 [I 2022-10-27 13:35:23] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 13:35:23] ...trap/engines/engine.py.326: process: 0.28625 | load: 0.00015 [I 2022-10-27 13:35:25] ...trap/engines/engine.py.321: eval: epoch 69 | batch 430/510 [I 2022-10-27 13:35:25] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 13:35:25] ...trap/engines/engine.py.326: process: 0.28930 | load: 0.00025 [I 2022-10-27 13:35:28] ...trap/engines/engine.py.321: eval: epoch 69 | batch 440/510 [I 2022-10-27 13:35:28] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 13:35:28] ...trap/engines/engine.py.326: process: 0.28974 | load: 0.00015 [I 2022-10-27 13:35:31] ...trap/engines/engine.py.321: eval: epoch 69 | batch 450/510 [I 2022-10-27 13:35:31] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 13:35:31] ...trap/engines/engine.py.326: process: 0.28374 | load: 0.00015 [I 2022-10-27 13:35:34] ...trap/engines/engine.py.321: eval: epoch 69 | batch 460/510 [I 2022-10-27 13:35:34] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 13:35:34] ...trap/engines/engine.py.326: process: 0.28193 | load: 0.00016 [I 2022-10-27 13:35:37] ...trap/engines/engine.py.321: eval: epoch 69 | batch 470/510 [I 2022-10-27 13:35:37] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 13:35:37] ...trap/engines/engine.py.326: process: 0.28810 | load: 0.00026 [I 2022-10-27 13:35:40] ...trap/engines/engine.py.321: eval: epoch 69 | batch 480/510 [I 2022-10-27 13:35:40] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 13:35:40] ...trap/engines/engine.py.326: process: 0.29032 | load: 0.00015 [I 2022-10-27 13:35:43] ...trap/engines/engine.py.321: eval: epoch 69 | batch 490/510 [I 2022-10-27 13:35:43] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 13:35:43] ...trap/engines/engine.py.326: process: 0.28555 | load: 0.00035 [I 2022-10-27 13:35:46] ...trap/engines/engine.py.321: eval: epoch 69 | batch 500/510 [I 2022-10-27 13:35:46] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 13:35:46] ...trap/engines/engine.py.326: process: 0.28646 | load: 0.00020 [I 2022-10-27 13:35:48] ...trap/engines/engine.py.321: eval: epoch 69 | batch 510/510 [I 2022-10-27 13:35:48] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 13:35:48] ...trap/engines/engine.py.326: process: 0.28833 | load: 0.00022 [S 2022-10-27 13:35:49] ...trap/engines/engine.py.132: eval_epoch.epoch: 69 [I 2022-10-27 13:35:49] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.719100 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.902400 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934700 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.704500 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.899700 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931900 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014508 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009810 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006634 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011404 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006435 [S 2022-10-27 13:35:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007217 [I 2022-10-27 13:35:55] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 13:35:55] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 13:35:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 13:35:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 13:35:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 13:35:59] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 13:36:00] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 13:36:06] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 13:36:06] ...trap/engines/engine.py.163: Training model on trainset for epoch 70 [I 2022-10-27 13:36:06] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 13:36:09] ...trap/engines/engine.py.224: train: epoch 70 | batch 0/2304 [I 2022-10-27 13:36:09] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:50:08 [I 2022-10-27 13:36:09] ...trap/engines/engine.py.229: process: 2.86834 | load: 2.13084 [I 2022-10-27 13:36:09] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-27 13:36:09] ...optimizers/trijoint.py.151: total_norm: 14.264748 [I 2022-10-27 13:36:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 10/2304 [I 2022-10-27 13:36:14] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:31:03 [I 2022-10-27 13:36:14] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00386 [I 2022-10-27 13:36:14] ...trap/engines/engine.py.230: loss: 0.11127 [I 2022-10-27 13:36:14] ...optimizers/trijoint.py.151: total_norm: 18.093025 [I 2022-10-27 13:36:20] ...trap/engines/engine.py.224: train: epoch 70 | batch 20/2304 [I 2022-10-27 13:36:20] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:39 [I 2022-10-27 13:36:20] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00410 [I 2022-10-27 13:36:20] ...trap/engines/engine.py.230: loss: 0.12789 [I 2022-10-27 13:36:20] ...optimizers/trijoint.py.151: total_norm: 15.429980 [I 2022-10-27 13:36:26] ...trap/engines/engine.py.224: train: epoch 70 | batch 30/2304 [I 2022-10-27 13:36:26] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:11 [I 2022-10-27 13:36:26] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00295 [I 2022-10-27 13:36:26] ...trap/engines/engine.py.230: loss: 0.12972 [I 2022-10-27 13:36:26] ...optimizers/trijoint.py.151: total_norm: 19.974863 [I 2022-10-27 13:36:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 40/2304 [I 2022-10-27 13:36:31] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:03 [I 2022-10-27 13:36:31] ...trap/engines/engine.py.229: process: 0.56687 | load: 0.00346 [I 2022-10-27 13:36:31] ...trap/engines/engine.py.230: loss: 0.14109 [I 2022-10-27 13:36:31] ...optimizers/trijoint.py.151: total_norm: 17.307594 [I 2022-10-27 13:36:37] ...trap/engines/engine.py.224: train: epoch 70 | batch 50/2304 [I 2022-10-27 13:36:37] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:52 [I 2022-10-27 13:36:37] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00278 [I 2022-10-27 13:36:37] ...trap/engines/engine.py.230: loss: 0.12611 [I 2022-10-27 13:36:37] ...optimizers/trijoint.py.151: total_norm: 8.400618 [I 2022-10-27 13:36:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 60/2304 [I 2022-10-27 13:36:42] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:45 [I 2022-10-27 13:36:42] ...trap/engines/engine.py.229: process: 0.55656 | load: 0.00269 [I 2022-10-27 13:36:42] ...trap/engines/engine.py.230: loss: 0.12856 [I 2022-10-27 13:36:42] ...optimizers/trijoint.py.151: total_norm: 14.656193 [I 2022-10-27 13:36:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 70/2304 [I 2022-10-27 13:36:48] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:41 [I 2022-10-27 13:36:48] ...trap/engines/engine.py.229: process: 0.54567 | load: 0.00268 [I 2022-10-27 13:36:48] ...trap/engines/engine.py.230: loss: 0.15347 [I 2022-10-27 13:36:48] ...optimizers/trijoint.py.151: total_norm: 16.853701 [I 2022-10-27 13:36:53] ...trap/engines/engine.py.224: train: epoch 70 | batch 80/2304 [I 2022-10-27 13:36:53] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:41 [I 2022-10-27 13:36:53] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00245 [I 2022-10-27 13:36:53] ...trap/engines/engine.py.230: loss: 0.13910 [I 2022-10-27 13:36:53] ...optimizers/trijoint.py.151: total_norm: 17.624516 [I 2022-10-27 13:36:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 90/2304 [I 2022-10-27 13:36:59] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:21:09 [I 2022-10-27 13:36:59] ...trap/engines/engine.py.229: process: 0.56622 | load: 0.00304 [I 2022-10-27 13:36:59] ...trap/engines/engine.py.230: loss: 0.15448 [I 2022-10-27 13:36:59] ...optimizers/trijoint.py.151: total_norm: 16.919825 [I 2022-10-27 13:37:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 100/2304 [I 2022-10-27 13:37:05] ...trap/engines/engine.py.228: elapsed: 0:00:59 | left: 0:20:40 [I 2022-10-27 13:37:05] ...trap/engines/engine.py.229: process: 0.56477 | load: 0.00269 [I 2022-10-27 13:37:05] ...trap/engines/engine.py.230: loss: 0.13892 [I 2022-10-27 13:37:05] ...optimizers/trijoint.py.151: total_norm: 14.563278 [I 2022-10-27 13:37:10] ...trap/engines/engine.py.224: train: epoch 70 | batch 110/2304 [I 2022-10-27 13:37:10] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:38 [I 2022-10-27 13:37:10] ...trap/engines/engine.py.229: process: 0.57105 | load: 0.00268 [I 2022-10-27 13:37:10] ...trap/engines/engine.py.230: loss: 0.14934 [I 2022-10-27 13:37:10] ...optimizers/trijoint.py.151: total_norm: 27.584421 [I 2022-10-27 13:37:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 120/2304 [I 2022-10-27 13:37:16] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:13 [I 2022-10-27 13:37:16] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00291 [I 2022-10-27 13:37:16] ...trap/engines/engine.py.230: loss: 0.08826 [I 2022-10-27 13:37:16] ...optimizers/trijoint.py.151: total_norm: 8.375652 [I 2022-10-27 13:37:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 130/2304 [I 2022-10-27 13:37:22] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:03 [I 2022-10-27 13:37:22] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00263 [I 2022-10-27 13:37:22] ...trap/engines/engine.py.230: loss: 0.10885 [I 2022-10-27 13:37:22] ...optimizers/trijoint.py.151: total_norm: 11.114512 [I 2022-10-27 13:37:27] ...trap/engines/engine.py.224: train: epoch 70 | batch 140/2304 [I 2022-10-27 13:37:27] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:07 [I 2022-10-27 13:37:27] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00254 [I 2022-10-27 13:37:27] ...trap/engines/engine.py.230: loss: 0.08834 [I 2022-10-27 13:37:27] ...optimizers/trijoint.py.151: total_norm: 8.629355 [I 2022-10-27 13:37:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 150/2304 [I 2022-10-27 13:37:33] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:20:03 [I 2022-10-27 13:37:33] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00250 [I 2022-10-27 13:37:33] ...trap/engines/engine.py.230: loss: 0.11380 [I 2022-10-27 13:37:33] ...optimizers/trijoint.py.151: total_norm: 9.785265 [I 2022-10-27 13:37:38] ...trap/engines/engine.py.224: train: epoch 70 | batch 160/2304 [I 2022-10-27 13:37:38] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:14 [I 2022-10-27 13:37:38] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00253 [I 2022-10-27 13:37:38] ...trap/engines/engine.py.230: loss: 0.12928 [I 2022-10-27 13:37:38] ...optimizers/trijoint.py.151: total_norm: 17.840893 [I 2022-10-27 13:37:44] ...trap/engines/engine.py.224: train: epoch 70 | batch 170/2304 [I 2022-10-27 13:37:44] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:48 [I 2022-10-27 13:37:44] ...trap/engines/engine.py.229: process: 0.56463 | load: 0.00239 [I 2022-10-27 13:37:44] ...trap/engines/engine.py.230: loss: 0.16193 [I 2022-10-27 13:37:44] ...optimizers/trijoint.py.151: total_norm: 14.877154 [I 2022-10-27 13:37:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 180/2304 [I 2022-10-27 13:37:50] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:44 [I 2022-10-27 13:37:50] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00251 [I 2022-10-27 13:37:50] ...trap/engines/engine.py.230: loss: 0.22920 [I 2022-10-27 13:37:50] ...optimizers/trijoint.py.151: total_norm: 21.212124 [I 2022-10-27 13:37:55] ...trap/engines/engine.py.224: train: epoch 70 | batch 190/2304 [I 2022-10-27 13:37:55] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:41 [I 2022-10-27 13:37:55] ...trap/engines/engine.py.229: process: 0.56328 | load: 0.00244 [I 2022-10-27 13:37:55] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-27 13:37:55] ...optimizers/trijoint.py.151: total_norm: 14.838858 [I 2022-10-27 13:38:01] ...trap/engines/engine.py.224: train: epoch 70 | batch 200/2304 [I 2022-10-27 13:38:01] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:44 [I 2022-10-27 13:38:01] ...trap/engines/engine.py.229: process: 0.55821 | load: 0.00259 [I 2022-10-27 13:38:01] ...trap/engines/engine.py.230: loss: 0.13021 [I 2022-10-27 13:38:01] ...optimizers/trijoint.py.151: total_norm: 17.571266 [I 2022-10-27 13:38:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 210/2304 [I 2022-10-27 13:38:07] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:32 [I 2022-10-27 13:38:07] ...trap/engines/engine.py.229: process: 0.54352 | load: 0.00253 [I 2022-10-27 13:38:07] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-27 13:38:07] ...optimizers/trijoint.py.151: total_norm: 17.768414 [I 2022-10-27 13:38:12] ...trap/engines/engine.py.224: train: epoch 70 | batch 220/2304 [I 2022-10-27 13:38:12] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:21 [I 2022-10-27 13:38:12] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00241 [I 2022-10-27 13:38:12] ...trap/engines/engine.py.230: loss: 0.19133 [I 2022-10-27 13:38:12] ...optimizers/trijoint.py.151: total_norm: 11.531154 [I 2022-10-27 13:38:18] ...trap/engines/engine.py.224: train: epoch 70 | batch 230/2304 [I 2022-10-27 13:38:18] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:14 [I 2022-10-27 13:38:18] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00255 [I 2022-10-27 13:38:18] ...trap/engines/engine.py.230: loss: 0.12477 [I 2022-10-27 13:38:18] ...optimizers/trijoint.py.151: total_norm: 7.806126 [I 2022-10-27 13:38:23] ...trap/engines/engine.py.224: train: epoch 70 | batch 240/2304 [I 2022-10-27 13:38:23] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:14 [I 2022-10-27 13:38:23] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00250 [I 2022-10-27 13:38:23] ...trap/engines/engine.py.230: loss: 0.15794 [I 2022-10-27 13:38:23] ...optimizers/trijoint.py.151: total_norm: 30.208881 [I 2022-10-27 13:38:29] ...trap/engines/engine.py.224: train: epoch 70 | batch 250/2304 [I 2022-10-27 13:38:29] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:18:56 [I 2022-10-27 13:38:29] ...trap/engines/engine.py.229: process: 0.55014 | load: 0.00252 [I 2022-10-27 13:38:29] ...trap/engines/engine.py.230: loss: 0.22967 [I 2022-10-27 13:38:29] ...optimizers/trijoint.py.151: total_norm: 19.530745 [I 2022-10-27 13:38:34] ...trap/engines/engine.py.224: train: epoch 70 | batch 260/2304 [I 2022-10-27 13:38:34] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:01 [I 2022-10-27 13:38:34] ...trap/engines/engine.py.229: process: 0.55372 | load: 0.00254 [I 2022-10-27 13:38:34] ...trap/engines/engine.py.230: loss: 0.13580 [I 2022-10-27 13:38:34] ...optimizers/trijoint.py.151: total_norm: 15.542329 [I 2022-10-27 13:38:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 270/2304 [I 2022-10-27 13:38:40] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:56 [I 2022-10-27 13:38:40] ...trap/engines/engine.py.229: process: 0.55213 | load: 0.00271 [I 2022-10-27 13:38:40] ...trap/engines/engine.py.230: loss: 0.18077 [I 2022-10-27 13:38:40] ...optimizers/trijoint.py.151: total_norm: 7.922698 [I 2022-10-27 13:38:46] ...trap/engines/engine.py.224: train: epoch 70 | batch 280/2304 [I 2022-10-27 13:38:46] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:19:05 [I 2022-10-27 13:38:46] ...trap/engines/engine.py.229: process: 0.57360 | load: 0.00241 [I 2022-10-27 13:38:46] ...trap/engines/engine.py.230: loss: 0.12207 [I 2022-10-27 13:38:46] ...optimizers/trijoint.py.151: total_norm: 16.115963 [I 2022-10-27 13:38:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 290/2304 [I 2022-10-27 13:38:51] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:46 [I 2022-10-27 13:38:51] ...trap/engines/engine.py.229: process: 0.55104 | load: 0.00255 [I 2022-10-27 13:38:51] ...trap/engines/engine.py.230: loss: 0.21929 [I 2022-10-27 13:38:51] ...optimizers/trijoint.py.151: total_norm: 21.306133 [I 2022-10-27 13:38:57] ...trap/engines/engine.py.224: train: epoch 70 | batch 300/2304 [I 2022-10-27 13:38:57] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:34 [I 2022-10-27 13:38:57] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00244 [I 2022-10-27 13:38:57] ...trap/engines/engine.py.230: loss: 0.15268 [I 2022-10-27 13:38:57] ...optimizers/trijoint.py.151: total_norm: 26.300266 [I 2022-10-27 13:39:03] ...trap/engines/engine.py.224: train: epoch 70 | batch 310/2304 [I 2022-10-27 13:39:03] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:35 [I 2022-10-27 13:39:03] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00270 [I 2022-10-27 13:39:03] ...trap/engines/engine.py.230: loss: 0.10115 [I 2022-10-27 13:39:03] ...optimizers/trijoint.py.151: total_norm: 8.617450 [I 2022-10-27 13:39:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 320/2304 [I 2022-10-27 13:39:08] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:40 [I 2022-10-27 13:39:08] ...trap/engines/engine.py.229: process: 0.56545 | load: 0.00257 [I 2022-10-27 13:39:08] ...trap/engines/engine.py.230: loss: 0.18909 [I 2022-10-27 13:39:08] ...optimizers/trijoint.py.151: total_norm: 34.715599 [I 2022-10-27 13:39:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 330/2304 [I 2022-10-27 13:39:14] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:18:20 [I 2022-10-27 13:39:14] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00255 [I 2022-10-27 13:39:14] ...trap/engines/engine.py.230: loss: 0.13425 [I 2022-10-27 13:39:14] ...optimizers/trijoint.py.151: total_norm: 14.092319 [I 2022-10-27 13:39:19] ...trap/engines/engine.py.224: train: epoch 70 | batch 340/2304 [I 2022-10-27 13:39:19] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:16 [I 2022-10-27 13:39:19] ...trap/engines/engine.py.229: process: 0.55880 | load: 0.00244 [I 2022-10-27 13:39:19] ...trap/engines/engine.py.230: loss: 0.12702 [I 2022-10-27 13:39:19] ...optimizers/trijoint.py.151: total_norm: 12.197738 [I 2022-10-27 13:39:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 350/2304 [I 2022-10-27 13:39:25] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:02 [I 2022-10-27 13:39:25] ...trap/engines/engine.py.229: process: 0.54925 | load: 0.00250 [I 2022-10-27 13:39:25] ...trap/engines/engine.py.230: loss: 0.11454 [I 2022-10-27 13:39:25] ...optimizers/trijoint.py.151: total_norm: 14.584387 [I 2022-10-27 13:39:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 360/2304 [I 2022-10-27 13:39:31] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:58 [I 2022-10-27 13:39:31] ...trap/engines/engine.py.229: process: 0.54644 | load: 0.00258 [I 2022-10-27 13:39:31] ...trap/engines/engine.py.230: loss: 0.15800 [I 2022-10-27 13:39:31] ...optimizers/trijoint.py.151: total_norm: 28.682867 [I 2022-10-27 13:39:36] ...trap/engines/engine.py.224: train: epoch 70 | batch 370/2304 [I 2022-10-27 13:39:36] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:08 [I 2022-10-27 13:39:36] ...trap/engines/engine.py.229: process: 0.57184 | load: 0.00252 [I 2022-10-27 13:39:36] ...trap/engines/engine.py.230: loss: 0.14175 [I 2022-10-27 13:39:36] ...optimizers/trijoint.py.151: total_norm: 17.323486 [I 2022-10-27 13:39:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 380/2304 [I 2022-10-27 13:39:42] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:44 [I 2022-10-27 13:39:42] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00263 [I 2022-10-27 13:39:42] ...trap/engines/engine.py.230: loss: 0.13277 [I 2022-10-27 13:39:42] ...optimizers/trijoint.py.151: total_norm: 14.263647 [I 2022-10-27 13:39:47] ...trap/engines/engine.py.224: train: epoch 70 | batch 390/2304 [I 2022-10-27 13:39:47] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:39 [I 2022-10-27 13:39:47] ...trap/engines/engine.py.229: process: 0.55211 | load: 0.00245 [I 2022-10-27 13:39:47] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-27 13:39:47] ...optimizers/trijoint.py.151: total_norm: 17.089842 [I 2022-10-27 13:39:53] ...trap/engines/engine.py.224: train: epoch 70 | batch 400/2304 [I 2022-10-27 13:39:53] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:48 [I 2022-10-27 13:39:53] ...trap/engines/engine.py.229: process: 0.54625 | load: 0.00244 [I 2022-10-27 13:39:53] ...trap/engines/engine.py.230: loss: 0.14905 [I 2022-10-27 13:39:53] ...optimizers/trijoint.py.151: total_norm: 34.678471 [I 2022-10-27 13:39:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 410/2304 [I 2022-10-27 13:39:59] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:35 [I 2022-10-27 13:39:59] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00253 [I 2022-10-27 13:39:59] ...trap/engines/engine.py.230: loss: 0.14158 [I 2022-10-27 13:39:59] ...optimizers/trijoint.py.151: total_norm: 14.360708 [I 2022-10-27 13:40:04] ...trap/engines/engine.py.224: train: epoch 70 | batch 420/2304 [I 2022-10-27 13:40:04] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:35 [I 2022-10-27 13:40:04] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00249 [I 2022-10-27 13:40:04] ...trap/engines/engine.py.230: loss: 0.10707 [I 2022-10-27 13:40:04] ...optimizers/trijoint.py.151: total_norm: 13.145817 [I 2022-10-27 13:40:10] ...trap/engines/engine.py.224: train: epoch 70 | batch 430/2304 [I 2022-10-27 13:40:10] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:17 [I 2022-10-27 13:40:10] ...trap/engines/engine.py.229: process: 0.55009 | load: 0.00257 [I 2022-10-27 13:40:10] ...trap/engines/engine.py.230: loss: 0.12224 [I 2022-10-27 13:40:10] ...optimizers/trijoint.py.151: total_norm: 11.641527 [I 2022-10-27 13:40:15] ...trap/engines/engine.py.224: train: epoch 70 | batch 440/2304 [I 2022-10-27 13:40:15] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:17 [I 2022-10-27 13:40:15] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00250 [I 2022-10-27 13:40:15] ...trap/engines/engine.py.230: loss: 0.11469 [I 2022-10-27 13:40:15] ...optimizers/trijoint.py.151: total_norm: 11.606075 [I 2022-10-27 13:40:21] ...trap/engines/engine.py.224: train: epoch 70 | batch 450/2304 [I 2022-10-27 13:40:21] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:19 [I 2022-10-27 13:40:21] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00243 [I 2022-10-27 13:40:21] ...trap/engines/engine.py.230: loss: 0.17243 [I 2022-10-27 13:40:21] ...optimizers/trijoint.py.151: total_norm: 18.109505 [I 2022-10-27 13:40:27] ...trap/engines/engine.py.224: train: epoch 70 | batch 460/2304 [I 2022-10-27 13:40:27] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:15 [I 2022-10-27 13:40:27] ...trap/engines/engine.py.229: process: 0.56560 | load: 0.00244 [I 2022-10-27 13:40:27] ...trap/engines/engine.py.230: loss: 0.21517 [I 2022-10-27 13:40:27] ...optimizers/trijoint.py.151: total_norm: 9.701837 [I 2022-10-27 13:40:32] ...trap/engines/engine.py.224: train: epoch 70 | batch 470/2304 [I 2022-10-27 13:40:32] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:12 [I 2022-10-27 13:40:32] ...trap/engines/engine.py.229: process: 0.57389 | load: 0.00243 [I 2022-10-27 13:40:32] ...trap/engines/engine.py.230: loss: 0.13060 [I 2022-10-27 13:40:32] ...optimizers/trijoint.py.151: total_norm: 14.469578 [I 2022-10-27 13:40:38] ...trap/engines/engine.py.224: train: epoch 70 | batch 480/2304 [I 2022-10-27 13:40:38] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:17:01 [I 2022-10-27 13:40:38] ...trap/engines/engine.py.229: process: 0.56272 | load: 0.00243 [I 2022-10-27 13:40:38] ...trap/engines/engine.py.230: loss: 0.15069 [I 2022-10-27 13:40:38] ...optimizers/trijoint.py.151: total_norm: 10.446935 [I 2022-10-27 13:40:43] ...trap/engines/engine.py.224: train: epoch 70 | batch 490/2304 [I 2022-10-27 13:40:43] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:55 [I 2022-10-27 13:40:43] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00257 [I 2022-10-27 13:40:43] ...trap/engines/engine.py.230: loss: 0.13868 [I 2022-10-27 13:40:43] ...optimizers/trijoint.py.151: total_norm: 18.883003 [I 2022-10-27 13:40:49] ...trap/engines/engine.py.224: train: epoch 70 | batch 500/2304 [I 2022-10-27 13:40:49] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:55 [I 2022-10-27 13:40:49] ...trap/engines/engine.py.229: process: 0.57176 | load: 0.00247 [I 2022-10-27 13:40:49] ...trap/engines/engine.py.230: loss: 0.15610 [I 2022-10-27 13:40:49] ...optimizers/trijoint.py.151: total_norm: 21.659086 [I 2022-10-27 13:40:55] ...trap/engines/engine.py.224: train: epoch 70 | batch 510/2304 [I 2022-10-27 13:40:55] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:37 [I 2022-10-27 13:40:55] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00253 [I 2022-10-27 13:40:55] ...trap/engines/engine.py.230: loss: 0.16977 [I 2022-10-27 13:40:55] ...optimizers/trijoint.py.151: total_norm: 21.525316 [I 2022-10-27 13:41:00] ...trap/engines/engine.py.224: train: epoch 70 | batch 520/2304 [I 2022-10-27 13:41:00] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:50 [I 2022-10-27 13:41:00] ...trap/engines/engine.py.229: process: 0.57326 | load: 0.00243 [I 2022-10-27 13:41:00] ...trap/engines/engine.py.230: loss: 0.17100 [I 2022-10-27 13:41:00] ...optimizers/trijoint.py.151: total_norm: 23.708363 [I 2022-10-27 13:41:06] ...trap/engines/engine.py.224: train: epoch 70 | batch 530/2304 [I 2022-10-27 13:41:06] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:33 [I 2022-10-27 13:41:06] ...trap/engines/engine.py.229: process: 0.54508 | load: 0.00239 [I 2022-10-27 13:41:06] ...trap/engines/engine.py.230: loss: 0.11861 [I 2022-10-27 13:41:06] ...optimizers/trijoint.py.151: total_norm: 13.723526 [I 2022-10-27 13:41:12] ...trap/engines/engine.py.224: train: epoch 70 | batch 540/2304 [I 2022-10-27 13:41:12] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:25 [I 2022-10-27 13:41:12] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00254 [I 2022-10-27 13:41:12] ...trap/engines/engine.py.230: loss: 0.13202 [I 2022-10-27 13:41:12] ...optimizers/trijoint.py.151: total_norm: 17.924271 [I 2022-10-27 13:41:17] ...trap/engines/engine.py.224: train: epoch 70 | batch 550/2304 [I 2022-10-27 13:41:17] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:22 [I 2022-10-27 13:41:17] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00245 [I 2022-10-27 13:41:17] ...trap/engines/engine.py.230: loss: 0.13070 [I 2022-10-27 13:41:17] ...optimizers/trijoint.py.151: total_norm: 15.990747 [I 2022-10-27 13:41:23] ...trap/engines/engine.py.224: train: epoch 70 | batch 560/2304 [I 2022-10-27 13:41:23] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:23 [I 2022-10-27 13:41:23] ...trap/engines/engine.py.229: process: 0.57096 | load: 0.00262 [I 2022-10-27 13:41:23] ...trap/engines/engine.py.230: loss: 0.11406 [I 2022-10-27 13:41:23] ...optimizers/trijoint.py.151: total_norm: 10.318989 [I 2022-10-27 13:41:28] ...trap/engines/engine.py.224: train: epoch 70 | batch 570/2304 [I 2022-10-27 13:41:28] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:12 [I 2022-10-27 13:41:28] ...trap/engines/engine.py.229: process: 0.57000 | load: 0.00271 [I 2022-10-27 13:41:28] ...trap/engines/engine.py.230: loss: 0.13876 [I 2022-10-27 13:41:28] ...optimizers/trijoint.py.151: total_norm: 20.362524 [I 2022-10-27 13:41:34] ...trap/engines/engine.py.224: train: epoch 70 | batch 580/2304 [I 2022-10-27 13:41:34] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:16:16 [I 2022-10-27 13:41:34] ...trap/engines/engine.py.229: process: 0.58736 | load: 0.00259 [I 2022-10-27 13:41:34] ...trap/engines/engine.py.230: loss: 0.21046 [I 2022-10-27 13:41:34] ...optimizers/trijoint.py.151: total_norm: 21.138014 [I 2022-10-27 13:41:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 590/2304 [I 2022-10-27 13:41:40] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:04 [I 2022-10-27 13:41:40] ...trap/engines/engine.py.229: process: 0.56775 | load: 0.00259 [I 2022-10-27 13:41:40] ...trap/engines/engine.py.230: loss: 0.12895 [I 2022-10-27 13:41:40] ...optimizers/trijoint.py.151: total_norm: 20.778324 [I 2022-10-27 13:41:45] ...trap/engines/engine.py.224: train: epoch 70 | batch 600/2304 [I 2022-10-27 13:41:45] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:57 [I 2022-10-27 13:41:45] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00255 [I 2022-10-27 13:41:45] ...trap/engines/engine.py.230: loss: 0.11170 [I 2022-10-27 13:41:45] ...optimizers/trijoint.py.151: total_norm: 14.639825 [I 2022-10-27 13:41:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 610/2304 [I 2022-10-27 13:41:51] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:15:35 [I 2022-10-27 13:41:51] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00248 [I 2022-10-27 13:41:51] ...trap/engines/engine.py.230: loss: 0.17445 [I 2022-10-27 13:41:51] ...optimizers/trijoint.py.151: total_norm: 20.660173 [I 2022-10-27 13:41:56] ...trap/engines/engine.py.224: train: epoch 70 | batch 620/2304 [I 2022-10-27 13:41:56] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:40 [I 2022-10-27 13:41:56] ...trap/engines/engine.py.229: process: 0.56449 | load: 0.00244 [I 2022-10-27 13:41:56] ...trap/engines/engine.py.230: loss: 0.11519 [I 2022-10-27 13:41:56] ...optimizers/trijoint.py.151: total_norm: 14.382951 [I 2022-10-27 13:42:02] ...trap/engines/engine.py.224: train: epoch 70 | batch 630/2304 [I 2022-10-27 13:42:02] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:34 [I 2022-10-27 13:42:02] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00248 [I 2022-10-27 13:42:02] ...trap/engines/engine.py.230: loss: 0.13790 [I 2022-10-27 13:42:02] ...optimizers/trijoint.py.151: total_norm: 7.467346 [I 2022-10-27 13:42:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 640/2304 [I 2022-10-27 13:42:08] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:37 [I 2022-10-27 13:42:08] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00241 [I 2022-10-27 13:42:08] ...trap/engines/engine.py.230: loss: 0.10802 [I 2022-10-27 13:42:08] ...optimizers/trijoint.py.151: total_norm: 8.549635 [I 2022-10-27 13:42:13] ...trap/engines/engine.py.224: train: epoch 70 | batch 650/2304 [I 2022-10-27 13:42:13] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:23 [I 2022-10-27 13:42:13] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00243 [I 2022-10-27 13:42:13] ...trap/engines/engine.py.230: loss: 0.16807 [I 2022-10-27 13:42:13] ...optimizers/trijoint.py.151: total_norm: 33.237030 [I 2022-10-27 13:42:19] ...trap/engines/engine.py.224: train: epoch 70 | batch 660/2304 [I 2022-10-27 13:42:19] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:15:15 [I 2022-10-27 13:42:19] ...trap/engines/engine.py.229: process: 0.55019 | load: 0.00263 [I 2022-10-27 13:42:19] ...trap/engines/engine.py.230: loss: 0.15619 [I 2022-10-27 13:42:19] ...optimizers/trijoint.py.151: total_norm: 10.996295 [I 2022-10-27 13:42:24] ...trap/engines/engine.py.224: train: epoch 70 | batch 670/2304 [I 2022-10-27 13:42:24] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:09 [I 2022-10-27 13:42:24] ...trap/engines/engine.py.229: process: 0.56909 | load: 0.00286 [I 2022-10-27 13:42:24] ...trap/engines/engine.py.230: loss: 0.16621 [I 2022-10-27 13:42:24] ...optimizers/trijoint.py.151: total_norm: 19.053333 [I 2022-10-27 13:42:30] ...trap/engines/engine.py.224: train: epoch 70 | batch 680/2304 [I 2022-10-27 13:42:30] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:06 [I 2022-10-27 13:42:30] ...trap/engines/engine.py.229: process: 0.54994 | load: 0.00263 [I 2022-10-27 13:42:30] ...trap/engines/engine.py.230: loss: 0.14564 [I 2022-10-27 13:42:30] ...optimizers/trijoint.py.151: total_norm: 15.012804 [I 2022-10-27 13:42:36] ...trap/engines/engine.py.224: train: epoch 70 | batch 690/2304 [I 2022-10-27 13:42:36] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:02 [I 2022-10-27 13:42:36] ...trap/engines/engine.py.229: process: 0.56734 | load: 0.00251 [I 2022-10-27 13:42:36] ...trap/engines/engine.py.230: loss: 0.15485 [I 2022-10-27 13:42:36] ...optimizers/trijoint.py.151: total_norm: 23.336603 [I 2022-10-27 13:42:41] ...trap/engines/engine.py.224: train: epoch 70 | batch 700/2304 [I 2022-10-27 13:42:41] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:15:05 [I 2022-10-27 13:42:41] ...trap/engines/engine.py.229: process: 0.58028 | load: 0.00245 [I 2022-10-27 13:42:41] ...trap/engines/engine.py.230: loss: 0.16411 [I 2022-10-27 13:42:41] ...optimizers/trijoint.py.151: total_norm: 27.347971 [I 2022-10-27 13:42:47] ...trap/engines/engine.py.224: train: epoch 70 | batch 710/2304 [I 2022-10-27 13:42:47] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:14:48 [I 2022-10-27 13:42:47] ...trap/engines/engine.py.229: process: 0.56805 | load: 0.00242 [I 2022-10-27 13:42:47] ...trap/engines/engine.py.230: loss: 0.13492 [I 2022-10-27 13:42:47] ...optimizers/trijoint.py.151: total_norm: 13.476300 [I 2022-10-27 13:42:52] ...trap/engines/engine.py.224: train: epoch 70 | batch 720/2304 [I 2022-10-27 13:42:52] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:54 [I 2022-10-27 13:42:52] ...trap/engines/engine.py.229: process: 0.56771 | load: 0.00244 [I 2022-10-27 13:42:52] ...trap/engines/engine.py.230: loss: 0.14876 [I 2022-10-27 13:42:52] ...optimizers/trijoint.py.151: total_norm: 23.922409 [I 2022-10-27 13:42:58] ...trap/engines/engine.py.224: train: epoch 70 | batch 730/2304 [I 2022-10-27 13:42:58] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:14:46 [I 2022-10-27 13:42:58] ...trap/engines/engine.py.229: process: 0.55822 | load: 0.00241 [I 2022-10-27 13:42:58] ...trap/engines/engine.py.230: loss: 0.12224 [I 2022-10-27 13:42:58] ...optimizers/trijoint.py.151: total_norm: 18.335968 [I 2022-10-27 13:43:04] ...trap/engines/engine.py.224: train: epoch 70 | batch 740/2304 [I 2022-10-27 13:43:04] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:34 [I 2022-10-27 13:43:04] ...trap/engines/engine.py.229: process: 0.56985 | load: 0.00271 [I 2022-10-27 13:43:04] ...trap/engines/engine.py.230: loss: 0.13930 [I 2022-10-27 13:43:04] ...optimizers/trijoint.py.151: total_norm: 16.431248 [I 2022-10-27 13:43:09] ...trap/engines/engine.py.224: train: epoch 70 | batch 750/2304 [I 2022-10-27 13:43:09] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:26 [I 2022-10-27 13:43:09] ...trap/engines/engine.py.229: process: 0.56712 | load: 0.00266 [I 2022-10-27 13:43:09] ...trap/engines/engine.py.230: loss: 0.14285 [I 2022-10-27 13:43:09] ...optimizers/trijoint.py.151: total_norm: 21.191742 [I 2022-10-27 13:43:15] ...trap/engines/engine.py.224: train: epoch 70 | batch 760/2304 [I 2022-10-27 13:43:15] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:21 [I 2022-10-27 13:43:15] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00252 [I 2022-10-27 13:43:15] ...trap/engines/engine.py.230: loss: 0.14623 [I 2022-10-27 13:43:15] ...optimizers/trijoint.py.151: total_norm: 26.168589 [I 2022-10-27 13:43:20] ...trap/engines/engine.py.224: train: epoch 70 | batch 770/2304 [I 2022-10-27 13:43:20] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:12 [I 2022-10-27 13:43:20] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00248 [I 2022-10-27 13:43:20] ...trap/engines/engine.py.230: loss: 0.11267 [I 2022-10-27 13:43:20] ...optimizers/trijoint.py.151: total_norm: 8.601533 [I 2022-10-27 13:43:26] ...trap/engines/engine.py.224: train: epoch 70 | batch 780/2304 [I 2022-10-27 13:43:26] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:06 [I 2022-10-27 13:43:26] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00253 [I 2022-10-27 13:43:26] ...trap/engines/engine.py.230: loss: 0.12109 [I 2022-10-27 13:43:26] ...optimizers/trijoint.py.151: total_norm: 14.388618 [I 2022-10-27 13:43:32] ...trap/engines/engine.py.224: train: epoch 70 | batch 790/2304 [I 2022-10-27 13:43:32] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:00 [I 2022-10-27 13:43:32] ...trap/engines/engine.py.229: process: 0.55471 | load: 0.00248 [I 2022-10-27 13:43:32] ...trap/engines/engine.py.230: loss: 0.11383 [I 2022-10-27 13:43:32] ...optimizers/trijoint.py.151: total_norm: 17.328482 [I 2022-10-27 13:43:37] ...trap/engines/engine.py.224: train: epoch 70 | batch 800/2304 [I 2022-10-27 13:43:37] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:13:55 [I 2022-10-27 13:43:37] ...trap/engines/engine.py.229: process: 0.55521 | load: 0.00252 [I 2022-10-27 13:43:37] ...trap/engines/engine.py.230: loss: 0.19105 [I 2022-10-27 13:43:37] ...optimizers/trijoint.py.151: total_norm: 14.531351 [I 2022-10-27 13:43:43] ...trap/engines/engine.py.224: train: epoch 70 | batch 810/2304 [I 2022-10-27 13:43:43] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:13:54 [I 2022-10-27 13:43:43] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00251 [I 2022-10-27 13:43:43] ...trap/engines/engine.py.230: loss: 0.13485 [I 2022-10-27 13:43:43] ...optimizers/trijoint.py.151: total_norm: 21.685501 [I 2022-10-27 13:43:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 820/2304 [I 2022-10-27 13:43:48] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:48 [I 2022-10-27 13:43:48] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00240 [I 2022-10-27 13:43:48] ...trap/engines/engine.py.230: loss: 0.19238 [I 2022-10-27 13:43:48] ...optimizers/trijoint.py.151: total_norm: 18.532070 [I 2022-10-27 13:43:54] ...trap/engines/engine.py.224: train: epoch 70 | batch 830/2304 [I 2022-10-27 13:43:54] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:44 [I 2022-10-27 13:43:54] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00246 [I 2022-10-27 13:43:54] ...trap/engines/engine.py.230: loss: 0.12656 [I 2022-10-27 13:43:54] ...optimizers/trijoint.py.151: total_norm: 12.764695 [I 2022-10-27 13:44:00] ...trap/engines/engine.py.224: train: epoch 70 | batch 840/2304 [I 2022-10-27 13:44:00] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:38 [I 2022-10-27 13:44:00] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00243 [I 2022-10-27 13:44:00] ...trap/engines/engine.py.230: loss: 0.17983 [I 2022-10-27 13:44:00] ...optimizers/trijoint.py.151: total_norm: 15.433990 [I 2022-10-27 13:44:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 850/2304 [I 2022-10-27 13:44:05] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:26 [I 2022-10-27 13:44:05] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00249 [I 2022-10-27 13:44:05] ...trap/engines/engine.py.230: loss: 0.13932 [I 2022-10-27 13:44:05] ...optimizers/trijoint.py.151: total_norm: 14.695447 [I 2022-10-27 13:44:11] ...trap/engines/engine.py.224: train: epoch 70 | batch 860/2304 [I 2022-10-27 13:44:11] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:26 [I 2022-10-27 13:44:11] ...trap/engines/engine.py.229: process: 0.56533 | load: 0.00270 [I 2022-10-27 13:44:11] ...trap/engines/engine.py.230: loss: 0.12731 [I 2022-10-27 13:44:11] ...optimizers/trijoint.py.151: total_norm: 17.894196 [I 2022-10-27 13:44:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 870/2304 [I 2022-10-27 13:44:16] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:20 [I 2022-10-27 13:44:16] ...trap/engines/engine.py.229: process: 0.56050 | load: 0.00262 [I 2022-10-27 13:44:16] ...trap/engines/engine.py.230: loss: 0.11529 [I 2022-10-27 13:44:16] ...optimizers/trijoint.py.151: total_norm: 12.356331 [I 2022-10-27 13:44:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 880/2304 [I 2022-10-27 13:44:22] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:16 [I 2022-10-27 13:44:22] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00252 [I 2022-10-27 13:44:22] ...trap/engines/engine.py.230: loss: 0.19384 [I 2022-10-27 13:44:22] ...optimizers/trijoint.py.151: total_norm: 24.079351 [I 2022-10-27 13:44:27] ...trap/engines/engine.py.224: train: epoch 70 | batch 890/2304 [I 2022-10-27 13:44:27] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:08 [I 2022-10-27 13:44:27] ...trap/engines/engine.py.229: process: 0.54299 | load: 0.00252 [I 2022-10-27 13:44:27] ...trap/engines/engine.py.230: loss: 0.16952 [I 2022-10-27 13:44:27] ...optimizers/trijoint.py.151: total_norm: 21.282795 [I 2022-10-27 13:44:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 900/2304 [I 2022-10-27 13:44:33] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:08 [I 2022-10-27 13:44:33] ...trap/engines/engine.py.229: process: 0.57406 | load: 0.00246 [I 2022-10-27 13:44:33] ...trap/engines/engine.py.230: loss: 0.11925 [I 2022-10-27 13:44:33] ...optimizers/trijoint.py.151: total_norm: 37.241894 [I 2022-10-27 13:44:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 910/2304 [I 2022-10-27 13:44:39] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:13:07 [I 2022-10-27 13:44:39] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00244 [I 2022-10-27 13:44:39] ...trap/engines/engine.py.230: loss: 0.12623 [I 2022-10-27 13:44:39] ...optimizers/trijoint.py.151: total_norm: 12.257034 [I 2022-10-27 13:44:44] ...trap/engines/engine.py.224: train: epoch 70 | batch 920/2304 [I 2022-10-27 13:44:44] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:47 [I 2022-10-27 13:44:44] ...trap/engines/engine.py.229: process: 0.54292 | load: 0.00248 [I 2022-10-27 13:44:44] ...trap/engines/engine.py.230: loss: 0.10277 [I 2022-10-27 13:44:44] ...optimizers/trijoint.py.151: total_norm: 6.954538 [I 2022-10-27 13:44:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 930/2304 [I 2022-10-27 13:44:50] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:12:57 [I 2022-10-27 13:44:50] ...trap/engines/engine.py.229: process: 0.55189 | load: 0.00255 [I 2022-10-27 13:44:50] ...trap/engines/engine.py.230: loss: 0.16937 [I 2022-10-27 13:44:50] ...optimizers/trijoint.py.151: total_norm: 36.392853 [I 2022-10-27 13:44:56] ...trap/engines/engine.py.224: train: epoch 70 | batch 940/2304 [I 2022-10-27 13:44:56] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:44 [I 2022-10-27 13:44:56] ...trap/engines/engine.py.229: process: 0.56635 | load: 0.00247 [I 2022-10-27 13:44:56] ...trap/engines/engine.py.230: loss: 0.20652 [I 2022-10-27 13:44:56] ...optimizers/trijoint.py.151: total_norm: 16.365316 [I 2022-10-27 13:45:01] ...trap/engines/engine.py.224: train: epoch 70 | batch 950/2304 [I 2022-10-27 13:45:01] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:40 [I 2022-10-27 13:45:01] ...trap/engines/engine.py.229: process: 0.57318 | load: 0.00245 [I 2022-10-27 13:45:01] ...trap/engines/engine.py.230: loss: 0.10569 [I 2022-10-27 13:45:01] ...optimizers/trijoint.py.151: total_norm: 8.619771 [I 2022-10-27 13:45:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 960/2304 [I 2022-10-27 13:45:07] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:34 [I 2022-10-27 13:45:07] ...trap/engines/engine.py.229: process: 0.56851 | load: 0.00248 [I 2022-10-27 13:45:07] ...trap/engines/engine.py.230: loss: 0.15303 [I 2022-10-27 13:45:07] ...optimizers/trijoint.py.151: total_norm: 15.631107 [I 2022-10-27 13:45:12] ...trap/engines/engine.py.224: train: epoch 70 | batch 970/2304 [I 2022-10-27 13:45:12] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:28 [I 2022-10-27 13:45:12] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00253 [I 2022-10-27 13:45:12] ...trap/engines/engine.py.230: loss: 0.18483 [I 2022-10-27 13:45:12] ...optimizers/trijoint.py.151: total_norm: 19.736361 [I 2022-10-27 13:45:18] ...trap/engines/engine.py.224: train: epoch 70 | batch 980/2304 [I 2022-10-27 13:45:18] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:12:19 [I 2022-10-27 13:45:18] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00247 [I 2022-10-27 13:45:18] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-27 13:45:18] ...optimizers/trijoint.py.151: total_norm: 14.443164 [I 2022-10-27 13:45:24] ...trap/engines/engine.py.224: train: epoch 70 | batch 990/2304 [I 2022-10-27 13:45:24] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:08 [I 2022-10-27 13:45:24] ...trap/engines/engine.py.229: process: 0.54691 | load: 0.00245 [I 2022-10-27 13:45:24] ...trap/engines/engine.py.230: loss: 0.11709 [I 2022-10-27 13:45:24] ...optimizers/trijoint.py.151: total_norm: 22.081694 [I 2022-10-27 13:45:29] ...trap/engines/engine.py.224: train: epoch 70 | batch 1000/2304 [I 2022-10-27 13:45:29] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:08 [I 2022-10-27 13:45:29] ...trap/engines/engine.py.229: process: 0.56324 | load: 0.00244 [I 2022-10-27 13:45:29] ...trap/engines/engine.py.230: loss: 0.22927 [I 2022-10-27 13:45:29] ...optimizers/trijoint.py.151: total_norm: 35.123806 [I 2022-10-27 13:45:35] ...trap/engines/engine.py.224: train: epoch 70 | batch 1010/2304 [I 2022-10-27 13:45:35] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:11:59 [I 2022-10-27 13:45:35] ...trap/engines/engine.py.229: process: 0.54562 | load: 0.00239 [I 2022-10-27 13:45:35] ...trap/engines/engine.py.230: loss: 0.21476 [I 2022-10-27 13:45:35] ...optimizers/trijoint.py.151: total_norm: 22.298632 [I 2022-10-27 13:45:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 1020/2304 [I 2022-10-27 13:45:40] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:12:00 [I 2022-10-27 13:45:40] ...trap/engines/engine.py.229: process: 0.57428 | load: 0.00244 [I 2022-10-27 13:45:40] ...trap/engines/engine.py.230: loss: 0.14328 [I 2022-10-27 13:45:40] ...optimizers/trijoint.py.151: total_norm: 16.414135 [I 2022-10-27 13:45:46] ...trap/engines/engine.py.224: train: epoch 70 | batch 1030/2304 [I 2022-10-27 13:45:46] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:11:49 [I 2022-10-27 13:45:46] ...trap/engines/engine.py.229: process: 0.55543 | load: 0.00265 [I 2022-10-27 13:45:46] ...trap/engines/engine.py.230: loss: 0.10590 [I 2022-10-27 13:45:46] ...optimizers/trijoint.py.151: total_norm: 9.772335 [I 2022-10-27 13:45:52] ...trap/engines/engine.py.224: train: epoch 70 | batch 1040/2304 [I 2022-10-27 13:45:52] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:42 [I 2022-10-27 13:45:52] ...trap/engines/engine.py.229: process: 0.55041 | load: 0.00257 [I 2022-10-27 13:45:52] ...trap/engines/engine.py.230: loss: 0.09563 [I 2022-10-27 13:45:52] ...optimizers/trijoint.py.151: total_norm: 7.816951 [I 2022-10-27 13:45:57] ...trap/engines/engine.py.224: train: epoch 70 | batch 1050/2304 [I 2022-10-27 13:45:57] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:41 [I 2022-10-27 13:45:57] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00246 [I 2022-10-27 13:45:57] ...trap/engines/engine.py.230: loss: 0.11063 [I 2022-10-27 13:45:57] ...optimizers/trijoint.py.151: total_norm: 11.784161 [I 2022-10-27 13:46:03] ...trap/engines/engine.py.224: train: epoch 70 | batch 1060/2304 [I 2022-10-27 13:46:03] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:36 [I 2022-10-27 13:46:03] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00256 [I 2022-10-27 13:46:03] ...trap/engines/engine.py.230: loss: 0.13633 [I 2022-10-27 13:46:03] ...optimizers/trijoint.py.151: total_norm: 12.777292 [I 2022-10-27 13:46:09] ...trap/engines/engine.py.224: train: epoch 70 | batch 1070/2304 [I 2022-10-27 13:46:09] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:37 [I 2022-10-27 13:46:09] ...trap/engines/engine.py.229: process: 0.56286 | load: 0.00253 [I 2022-10-27 13:46:09] ...trap/engines/engine.py.230: loss: 0.19195 [I 2022-10-27 13:46:09] ...optimizers/trijoint.py.151: total_norm: 30.618464 [I 2022-10-27 13:46:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 1080/2304 [I 2022-10-27 13:46:14] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:29 [I 2022-10-27 13:46:14] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00252 [I 2022-10-27 13:46:14] ...trap/engines/engine.py.230: loss: 0.15552 [I 2022-10-27 13:46:14] ...optimizers/trijoint.py.151: total_norm: 16.595131 [I 2022-10-27 13:46:20] ...trap/engines/engine.py.224: train: epoch 70 | batch 1090/2304 [I 2022-10-27 13:46:20] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:24 [I 2022-10-27 13:46:20] ...trap/engines/engine.py.229: process: 0.56279 | load: 0.00253 [I 2022-10-27 13:46:20] ...trap/engines/engine.py.230: loss: 0.18487 [I 2022-10-27 13:46:20] ...optimizers/trijoint.py.151: total_norm: 19.106609 [I 2022-10-27 13:46:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 1100/2304 [I 2022-10-27 13:46:25] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:15 [I 2022-10-27 13:46:25] ...trap/engines/engine.py.229: process: 0.54440 | load: 0.00260 [I 2022-10-27 13:46:25] ...trap/engines/engine.py.230: loss: 0.19170 [I 2022-10-27 13:46:25] ...optimizers/trijoint.py.151: total_norm: 16.989035 [I 2022-10-27 13:46:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 1110/2304 [I 2022-10-27 13:46:31] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:05 [I 2022-10-27 13:46:31] ...trap/engines/engine.py.229: process: 0.54595 | load: 0.00251 [I 2022-10-27 13:46:31] ...trap/engines/engine.py.230: loss: 0.11576 [I 2022-10-27 13:46:31] ...optimizers/trijoint.py.151: total_norm: 10.242632 [I 2022-10-27 13:46:37] ...trap/engines/engine.py.224: train: epoch 70 | batch 1120/2304 [I 2022-10-27 13:46:37] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:05 [I 2022-10-27 13:46:37] ...trap/engines/engine.py.229: process: 0.56839 | load: 0.00253 [I 2022-10-27 13:46:37] ...trap/engines/engine.py.230: loss: 0.16677 [I 2022-10-27 13:46:37] ...optimizers/trijoint.py.151: total_norm: 14.558537 [I 2022-10-27 13:46:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 1130/2304 [I 2022-10-27 13:46:42] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:10:57 [I 2022-10-27 13:46:42] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00252 [I 2022-10-27 13:46:42] ...trap/engines/engine.py.230: loss: 0.11857 [I 2022-10-27 13:46:42] ...optimizers/trijoint.py.151: total_norm: 20.152073 [I 2022-10-27 13:46:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 1140/2304 [I 2022-10-27 13:46:48] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:58 [I 2022-10-27 13:46:48] ...trap/engines/engine.py.229: process: 0.57296 | load: 0.00248 [I 2022-10-27 13:46:48] ...trap/engines/engine.py.230: loss: 0.13044 [I 2022-10-27 13:46:48] ...optimizers/trijoint.py.151: total_norm: 17.105207 [I 2022-10-27 13:46:54] ...trap/engines/engine.py.224: train: epoch 70 | batch 1150/2304 [I 2022-10-27 13:46:54] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:11:03 [I 2022-10-27 13:46:54] ...trap/engines/engine.py.229: process: 0.56044 | load: 0.00246 [I 2022-10-27 13:46:54] ...trap/engines/engine.py.230: loss: 0.13658 [I 2022-10-27 13:46:54] ...optimizers/trijoint.py.151: total_norm: 15.883980 [I 2022-10-27 13:46:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 1160/2304 [I 2022-10-27 13:46:59] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:40 [I 2022-10-27 13:46:59] ...trap/engines/engine.py.229: process: 0.56642 | load: 0.00247 [I 2022-10-27 13:46:59] ...trap/engines/engine.py.230: loss: 0.12354 [I 2022-10-27 13:46:59] ...optimizers/trijoint.py.151: total_norm: 8.492211 [I 2022-10-27 13:47:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 1170/2304 [I 2022-10-27 13:47:05] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:36 [I 2022-10-27 13:47:05] ...trap/engines/engine.py.229: process: 0.56761 | load: 0.00242 [I 2022-10-27 13:47:05] ...trap/engines/engine.py.230: loss: 0.12903 [I 2022-10-27 13:47:05] ...optimizers/trijoint.py.151: total_norm: 17.598623 [I 2022-10-27 13:47:11] ...trap/engines/engine.py.224: train: epoch 70 | batch 1180/2304 [I 2022-10-27 13:47:11] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:33 [I 2022-10-27 13:47:11] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00243 [I 2022-10-27 13:47:11] ...trap/engines/engine.py.230: loss: 0.13306 [I 2022-10-27 13:47:11] ...optimizers/trijoint.py.151: total_norm: 21.938305 [I 2022-10-27 13:47:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 1190/2304 [I 2022-10-27 13:47:16] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:22 [I 2022-10-27 13:47:16] ...trap/engines/engine.py.229: process: 0.57079 | load: 0.00243 [I 2022-10-27 13:47:16] ...trap/engines/engine.py.230: loss: 0.14113 [I 2022-10-27 13:47:16] ...optimizers/trijoint.py.151: total_norm: 11.941624 [I 2022-10-27 13:47:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 1200/2304 [I 2022-10-27 13:47:22] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:16 [I 2022-10-27 13:47:22] ...trap/engines/engine.py.229: process: 0.54869 | load: 0.00247 [I 2022-10-27 13:47:22] ...trap/engines/engine.py.230: loss: 0.11400 [I 2022-10-27 13:47:22] ...optimizers/trijoint.py.151: total_norm: 16.481131 [I 2022-10-27 13:47:27] ...trap/engines/engine.py.224: train: epoch 70 | batch 1210/2304 [I 2022-10-27 13:47:27] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:11 [I 2022-10-27 13:47:27] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00246 [I 2022-10-27 13:47:27] ...trap/engines/engine.py.230: loss: 0.13646 [I 2022-10-27 13:47:27] ...optimizers/trijoint.py.151: total_norm: 30.040306 [I 2022-10-27 13:47:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 1220/2304 [I 2022-10-27 13:47:33] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:10 [I 2022-10-27 13:47:33] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00248 [I 2022-10-27 13:47:33] ...trap/engines/engine.py.230: loss: 0.15620 [I 2022-10-27 13:47:33] ...optimizers/trijoint.py.151: total_norm: 19.222971 [I 2022-10-27 13:47:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 1230/2304 [I 2022-10-27 13:47:39] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:09:59 [I 2022-10-27 13:47:39] ...trap/engines/engine.py.229: process: 0.54584 | load: 0.00242 [I 2022-10-27 13:47:39] ...trap/engines/engine.py.230: loss: 0.09735 [I 2022-10-27 13:47:39] ...optimizers/trijoint.py.151: total_norm: 7.003925 [I 2022-10-27 13:47:44] ...trap/engines/engine.py.224: train: epoch 70 | batch 1240/2304 [I 2022-10-27 13:47:44] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:09:52 [I 2022-10-27 13:47:44] ...trap/engines/engine.py.229: process: 0.54880 | load: 0.00239 [I 2022-10-27 13:47:44] ...trap/engines/engine.py.230: loss: 0.13792 [I 2022-10-27 13:47:44] ...optimizers/trijoint.py.151: total_norm: 14.528049 [I 2022-10-27 13:47:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 1250/2304 [I 2022-10-27 13:47:50] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:09:48 [I 2022-10-27 13:47:50] ...trap/engines/engine.py.229: process: 0.56127 | load: 0.00246 [I 2022-10-27 13:47:50] ...trap/engines/engine.py.230: loss: 0.11837 [I 2022-10-27 13:47:50] ...optimizers/trijoint.py.151: total_norm: 12.280414 [I 2022-10-27 13:47:55] ...trap/engines/engine.py.224: train: epoch 70 | batch 1260/2304 [I 2022-10-27 13:47:55] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:09:45 [I 2022-10-27 13:47:55] ...trap/engines/engine.py.229: process: 0.56127 | load: 0.00284 [I 2022-10-27 13:47:55] ...trap/engines/engine.py.230: loss: 0.13684 [I 2022-10-27 13:47:55] ...optimizers/trijoint.py.151: total_norm: 12.490065 [I 2022-10-27 13:48:01] ...trap/engines/engine.py.224: train: epoch 70 | batch 1270/2304 [I 2022-10-27 13:48:01] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:40 [I 2022-10-27 13:48:01] ...trap/engines/engine.py.229: process: 0.57603 | load: 0.00255 [I 2022-10-27 13:48:01] ...trap/engines/engine.py.230: loss: 0.16180 [I 2022-10-27 13:48:01] ...optimizers/trijoint.py.151: total_norm: 15.810460 [I 2022-10-27 13:48:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 1280/2304 [I 2022-10-27 13:48:07] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:09:46 [I 2022-10-27 13:48:07] ...trap/engines/engine.py.229: process: 0.62902 | load: 0.00245 [I 2022-10-27 13:48:07] ...trap/engines/engine.py.230: loss: 0.17799 [I 2022-10-27 13:48:07] ...optimizers/trijoint.py.151: total_norm: 22.521791 [I 2022-10-27 13:48:12] ...trap/engines/engine.py.224: train: epoch 70 | batch 1290/2304 [I 2022-10-27 13:48:12] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:25 [I 2022-10-27 13:48:12] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00250 [I 2022-10-27 13:48:12] ...trap/engines/engine.py.230: loss: 0.15147 [I 2022-10-27 13:48:12] ...optimizers/trijoint.py.151: total_norm: 16.497534 [I 2022-10-27 13:48:18] ...trap/engines/engine.py.224: train: epoch 70 | batch 1300/2304 [I 2022-10-27 13:48:18] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:23 [I 2022-10-27 13:48:18] ...trap/engines/engine.py.229: process: 0.57189 | load: 0.00242 [I 2022-10-27 13:48:18] ...trap/engines/engine.py.230: loss: 0.11435 [I 2022-10-27 13:48:18] ...optimizers/trijoint.py.151: total_norm: 22.234308 [I 2022-10-27 13:48:23] ...trap/engines/engine.py.224: train: epoch 70 | batch 1310/2304 [I 2022-10-27 13:48:23] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:16 [I 2022-10-27 13:48:23] ...trap/engines/engine.py.229: process: 0.56823 | load: 0.00242 [I 2022-10-27 13:48:23] ...trap/engines/engine.py.230: loss: 0.15090 [I 2022-10-27 13:48:23] ...optimizers/trijoint.py.151: total_norm: 16.869839 [I 2022-10-27 13:48:29] ...trap/engines/engine.py.224: train: epoch 70 | batch 1320/2304 [I 2022-10-27 13:48:29] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:12 [I 2022-10-27 13:48:29] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00245 [I 2022-10-27 13:48:29] ...trap/engines/engine.py.230: loss: 0.16043 [I 2022-10-27 13:48:29] ...optimizers/trijoint.py.151: total_norm: 21.003813 [I 2022-10-27 13:48:35] ...trap/engines/engine.py.224: train: epoch 70 | batch 1330/2304 [I 2022-10-27 13:48:35] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:06 [I 2022-10-27 13:48:35] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00245 [I 2022-10-27 13:48:35] ...trap/engines/engine.py.230: loss: 0.15751 [I 2022-10-27 13:48:35] ...optimizers/trijoint.py.151: total_norm: 21.173269 [I 2022-10-27 13:48:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 1340/2304 [I 2022-10-27 13:48:40] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:09:05 [I 2022-10-27 13:48:40] ...trap/engines/engine.py.229: process: 0.56264 | load: 0.00251 [I 2022-10-27 13:48:40] ...trap/engines/engine.py.230: loss: 0.27354 [I 2022-10-27 13:48:40] ...optimizers/trijoint.py.151: total_norm: 26.537703 [I 2022-10-27 13:48:46] ...trap/engines/engine.py.224: train: epoch 70 | batch 1350/2304 [I 2022-10-27 13:48:46] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:00 [I 2022-10-27 13:48:46] ...trap/engines/engine.py.229: process: 0.58633 | load: 0.00249 [I 2022-10-27 13:48:46] ...trap/engines/engine.py.230: loss: 0.11974 [I 2022-10-27 13:48:46] ...optimizers/trijoint.py.151: total_norm: 9.410341 [I 2022-10-27 13:48:52] ...trap/engines/engine.py.224: train: epoch 70 | batch 1360/2304 [I 2022-10-27 13:48:52] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:08:51 [I 2022-10-27 13:48:52] ...trap/engines/engine.py.229: process: 0.56553 | load: 0.00253 [I 2022-10-27 13:48:52] ...trap/engines/engine.py.230: loss: 0.14263 [I 2022-10-27 13:48:52] ...optimizers/trijoint.py.151: total_norm: 28.235044 [I 2022-10-27 13:48:57] ...trap/engines/engine.py.224: train: epoch 70 | batch 1370/2304 [I 2022-10-27 13:48:57] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:49 [I 2022-10-27 13:48:57] ...trap/engines/engine.py.229: process: 0.57839 | load: 0.00245 [I 2022-10-27 13:48:57] ...trap/engines/engine.py.230: loss: 0.12735 [I 2022-10-27 13:48:57] ...optimizers/trijoint.py.151: total_norm: 14.874430 [I 2022-10-27 13:49:03] ...trap/engines/engine.py.224: train: epoch 70 | batch 1380/2304 [I 2022-10-27 13:49:03] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:37 [I 2022-10-27 13:49:03] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00243 [I 2022-10-27 13:49:03] ...trap/engines/engine.py.230: loss: 0.12550 [I 2022-10-27 13:49:03] ...optimizers/trijoint.py.151: total_norm: 12.460502 [I 2022-10-27 13:49:09] ...trap/engines/engine.py.224: train: epoch 70 | batch 1390/2304 [I 2022-10-27 13:49:09] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:30 [I 2022-10-27 13:49:09] ...trap/engines/engine.py.229: process: 0.54469 | load: 0.00248 [I 2022-10-27 13:49:09] ...trap/engines/engine.py.230: loss: 0.11064 [I 2022-10-27 13:49:09] ...optimizers/trijoint.py.151: total_norm: 18.791368 [I 2022-10-27 13:49:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 1400/2304 [I 2022-10-27 13:49:14] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:25 [I 2022-10-27 13:49:14] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00252 [I 2022-10-27 13:49:14] ...trap/engines/engine.py.230: loss: 0.14295 [I 2022-10-27 13:49:14] ...optimizers/trijoint.py.151: total_norm: 11.774141 [I 2022-10-27 13:49:20] ...trap/engines/engine.py.224: train: epoch 70 | batch 1410/2304 [I 2022-10-27 13:49:20] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:32 [I 2022-10-27 13:49:20] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00242 [I 2022-10-27 13:49:20] ...trap/engines/engine.py.230: loss: 0.17251 [I 2022-10-27 13:49:20] ...optimizers/trijoint.py.151: total_norm: 12.395221 [I 2022-10-27 13:49:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 1420/2304 [I 2022-10-27 13:49:25] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:11 [I 2022-10-27 13:49:25] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00245 [I 2022-10-27 13:49:25] ...trap/engines/engine.py.230: loss: 0.12842 [I 2022-10-27 13:49:25] ...optimizers/trijoint.py.151: total_norm: 19.508909 [I 2022-10-27 13:49:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 1430/2304 [I 2022-10-27 13:49:31] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:07 [I 2022-10-27 13:49:31] ...trap/engines/engine.py.229: process: 0.55669 | load: 0.00250 [I 2022-10-27 13:49:31] ...trap/engines/engine.py.230: loss: 0.13233 [I 2022-10-27 13:49:31] ...optimizers/trijoint.py.151: total_norm: 27.771132 [I 2022-10-27 13:49:37] ...trap/engines/engine.py.224: train: epoch 70 | batch 1440/2304 [I 2022-10-27 13:49:37] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:02 [I 2022-10-27 13:49:37] ...trap/engines/engine.py.229: process: 0.55432 | load: 0.00244 [I 2022-10-27 13:49:37] ...trap/engines/engine.py.230: loss: 0.26450 [I 2022-10-27 13:49:37] ...optimizers/trijoint.py.151: total_norm: 34.447178 [I 2022-10-27 13:49:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 1450/2304 [I 2022-10-27 13:49:42] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:07:57 [I 2022-10-27 13:49:42] ...trap/engines/engine.py.229: process: 0.56226 | load: 0.00248 [I 2022-10-27 13:49:42] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-27 13:49:42] ...optimizers/trijoint.py.151: total_norm: 16.870279 [I 2022-10-27 13:49:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 1460/2304 [I 2022-10-27 13:49:48] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:07:51 [I 2022-10-27 13:49:48] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00252 [I 2022-10-27 13:49:48] ...trap/engines/engine.py.230: loss: 0.14492 [I 2022-10-27 13:49:48] ...optimizers/trijoint.py.151: total_norm: 10.022652 [I 2022-10-27 13:49:54] ...trap/engines/engine.py.224: train: epoch 70 | batch 1470/2304 [I 2022-10-27 13:49:54] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:07:58 [I 2022-10-27 13:49:54] ...trap/engines/engine.py.229: process: 0.55381 | load: 0.00246 [I 2022-10-27 13:49:54] ...trap/engines/engine.py.230: loss: 0.18474 [I 2022-10-27 13:49:54] ...optimizers/trijoint.py.151: total_norm: 23.522921 [I 2022-10-27 13:49:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 1480/2304 [I 2022-10-27 13:49:59] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:44 [I 2022-10-27 13:49:59] ...trap/engines/engine.py.229: process: 0.57875 | load: 0.00247 [I 2022-10-27 13:49:59] ...trap/engines/engine.py.230: loss: 0.14769 [I 2022-10-27 13:49:59] ...optimizers/trijoint.py.151: total_norm: 14.603188 [I 2022-10-27 13:50:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 1490/2304 [I 2022-10-27 13:50:05] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:39 [I 2022-10-27 13:50:05] ...trap/engines/engine.py.229: process: 0.57540 | load: 0.00245 [I 2022-10-27 13:50:05] ...trap/engines/engine.py.230: loss: 0.10580 [I 2022-10-27 13:50:05] ...optimizers/trijoint.py.151: total_norm: 18.093237 [I 2022-10-27 13:50:10] ...trap/engines/engine.py.224: train: epoch 70 | batch 1500/2304 [I 2022-10-27 13:50:10] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:31 [I 2022-10-27 13:50:10] ...trap/engines/engine.py.229: process: 0.56525 | load: 0.00251 [I 2022-10-27 13:50:10] ...trap/engines/engine.py.230: loss: 0.12681 [I 2022-10-27 13:50:10] ...optimizers/trijoint.py.151: total_norm: 9.521250 [I 2022-10-27 13:50:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 1510/2304 [I 2022-10-27 13:50:16] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:22 [I 2022-10-27 13:50:16] ...trap/engines/engine.py.229: process: 0.54791 | load: 0.00255 [I 2022-10-27 13:50:16] ...trap/engines/engine.py.230: loss: 0.12033 [I 2022-10-27 13:50:16] ...optimizers/trijoint.py.151: total_norm: 18.709375 [I 2022-10-27 13:50:22] ...trap/engines/engine.py.224: train: epoch 70 | batch 1520/2304 [I 2022-10-27 13:50:22] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:21 [I 2022-10-27 13:50:22] ...trap/engines/engine.py.229: process: 0.57285 | load: 0.00248 [I 2022-10-27 13:50:22] ...trap/engines/engine.py.230: loss: 0.18807 [I 2022-10-27 13:50:22] ...optimizers/trijoint.py.151: total_norm: 23.260149 [I 2022-10-27 13:50:27] ...trap/engines/engine.py.224: train: epoch 70 | batch 1530/2304 [I 2022-10-27 13:50:27] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:12 [I 2022-10-27 13:50:27] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00244 [I 2022-10-27 13:50:27] ...trap/engines/engine.py.230: loss: 0.12168 [I 2022-10-27 13:50:27] ...optimizers/trijoint.py.151: total_norm: 14.665114 [I 2022-10-27 13:50:33] ...trap/engines/engine.py.224: train: epoch 70 | batch 1540/2304 [I 2022-10-27 13:50:33] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:11 [I 2022-10-27 13:50:33] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00256 [I 2022-10-27 13:50:33] ...trap/engines/engine.py.230: loss: 0.13433 [I 2022-10-27 13:50:33] ...optimizers/trijoint.py.151: total_norm: 16.168890 [I 2022-10-27 13:50:39] ...trap/engines/engine.py.224: train: epoch 70 | batch 1550/2304 [I 2022-10-27 13:50:39] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:06:57 [I 2022-10-27 13:50:39] ...trap/engines/engine.py.229: process: 0.55013 | load: 0.00255 [I 2022-10-27 13:50:39] ...trap/engines/engine.py.230: loss: 0.11322 [I 2022-10-27 13:50:39] ...optimizers/trijoint.py.151: total_norm: 10.979335 [I 2022-10-27 13:50:44] ...trap/engines/engine.py.224: train: epoch 70 | batch 1560/2304 [I 2022-10-27 13:50:44] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:06:55 [I 2022-10-27 13:50:44] ...trap/engines/engine.py.229: process: 0.53524 | load: 0.00253 [I 2022-10-27 13:50:44] ...trap/engines/engine.py.230: loss: 0.13603 [I 2022-10-27 13:50:44] ...optimizers/trijoint.py.151: total_norm: 14.293620 [I 2022-10-27 13:50:50] ...trap/engines/engine.py.224: train: epoch 70 | batch 1570/2304 [I 2022-10-27 13:50:50] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:06:50 [I 2022-10-27 13:50:50] ...trap/engines/engine.py.229: process: 0.57488 | load: 0.00253 [I 2022-10-27 13:50:50] ...trap/engines/engine.py.230: loss: 0.17518 [I 2022-10-27 13:50:50] ...optimizers/trijoint.py.151: total_norm: 30.787161 [I 2022-10-27 13:50:55] ...trap/engines/engine.py.224: train: epoch 70 | batch 1580/2304 [I 2022-10-27 13:50:55] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:06:40 [I 2022-10-27 13:50:55] ...trap/engines/engine.py.229: process: 0.55131 | load: 0.00251 [I 2022-10-27 13:50:55] ...trap/engines/engine.py.230: loss: 0.09687 [I 2022-10-27 13:50:55] ...optimizers/trijoint.py.151: total_norm: 6.220242 [I 2022-10-27 13:51:01] ...trap/engines/engine.py.224: train: epoch 70 | batch 1590/2304 [I 2022-10-27 13:51:01] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:41 [I 2022-10-27 13:51:01] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00240 [I 2022-10-27 13:51:01] ...trap/engines/engine.py.230: loss: 0.18122 [I 2022-10-27 13:51:01] ...optimizers/trijoint.py.151: total_norm: 21.697531 [I 2022-10-27 13:51:07] ...trap/engines/engine.py.224: train: epoch 70 | batch 1600/2304 [I 2022-10-27 13:51:07] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:34 [I 2022-10-27 13:51:07] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00247 [I 2022-10-27 13:51:07] ...trap/engines/engine.py.230: loss: 0.18210 [I 2022-10-27 13:51:07] ...optimizers/trijoint.py.151: total_norm: 42.222027 [I 2022-10-27 13:51:12] ...trap/engines/engine.py.224: train: epoch 70 | batch 1610/2304 [I 2022-10-27 13:51:12] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:26 [I 2022-10-27 13:51:12] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00246 [I 2022-10-27 13:51:12] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-27 13:51:12] ...optimizers/trijoint.py.151: total_norm: 8.512843 [I 2022-10-27 13:51:18] ...trap/engines/engine.py.224: train: epoch 70 | batch 1620/2304 [I 2022-10-27 13:51:18] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:22 [I 2022-10-27 13:51:18] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00244 [I 2022-10-27 13:51:18] ...trap/engines/engine.py.230: loss: 0.19659 [I 2022-10-27 13:51:18] ...optimizers/trijoint.py.151: total_norm: 15.038839 [I 2022-10-27 13:51:23] ...trap/engines/engine.py.224: train: epoch 70 | batch 1630/2304 [I 2022-10-27 13:51:23] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:16 [I 2022-10-27 13:51:23] ...trap/engines/engine.py.229: process: 0.56442 | load: 0.00246 [I 2022-10-27 13:51:23] ...trap/engines/engine.py.230: loss: 0.10398 [I 2022-10-27 13:51:23] ...optimizers/trijoint.py.151: total_norm: 15.389689 [I 2022-10-27 13:51:29] ...trap/engines/engine.py.224: train: epoch 70 | batch 1640/2304 [I 2022-10-27 13:51:29] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:09 [I 2022-10-27 13:51:29] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00243 [I 2022-10-27 13:51:29] ...trap/engines/engine.py.230: loss: 0.12630 [I 2022-10-27 13:51:29] ...optimizers/trijoint.py.151: total_norm: 18.087181 [I 2022-10-27 13:51:35] ...trap/engines/engine.py.224: train: epoch 70 | batch 1650/2304 [I 2022-10-27 13:51:35] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:06 [I 2022-10-27 13:51:35] ...trap/engines/engine.py.229: process: 0.56816 | load: 0.00244 [I 2022-10-27 13:51:35] ...trap/engines/engine.py.230: loss: 0.15673 [I 2022-10-27 13:51:35] ...optimizers/trijoint.py.151: total_norm: 31.723351 [I 2022-10-27 13:51:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 1660/2304 [I 2022-10-27 13:51:40] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:05:59 [I 2022-10-27 13:51:40] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00243 [I 2022-10-27 13:51:40] ...trap/engines/engine.py.230: loss: 0.19852 [I 2022-10-27 13:51:40] ...optimizers/trijoint.py.151: total_norm: 46.100800 [I 2022-10-27 13:51:46] ...trap/engines/engine.py.224: train: epoch 70 | batch 1670/2304 [I 2022-10-27 13:51:46] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:05:53 [I 2022-10-27 13:51:46] ...trap/engines/engine.py.229: process: 0.54743 | load: 0.00256 [I 2022-10-27 13:51:46] ...trap/engines/engine.py.230: loss: 0.12975 [I 2022-10-27 13:51:46] ...optimizers/trijoint.py.151: total_norm: 12.745503 [I 2022-10-27 13:51:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 1680/2304 [I 2022-10-27 13:51:51] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:05:46 [I 2022-10-27 13:51:51] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00251 [I 2022-10-27 13:51:51] ...trap/engines/engine.py.230: loss: 0.11174 [I 2022-10-27 13:51:51] ...optimizers/trijoint.py.151: total_norm: 13.646779 [I 2022-10-27 13:51:57] ...trap/engines/engine.py.224: train: epoch 70 | batch 1690/2304 [I 2022-10-27 13:51:57] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:41 [I 2022-10-27 13:51:57] ...trap/engines/engine.py.229: process: 0.54488 | load: 0.00247 [I 2022-10-27 13:51:57] ...trap/engines/engine.py.230: loss: 0.16048 [I 2022-10-27 13:51:57] ...optimizers/trijoint.py.151: total_norm: 27.592543 [I 2022-10-27 13:52:03] ...trap/engines/engine.py.224: train: epoch 70 | batch 1700/2304 [I 2022-10-27 13:52:03] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:36 [I 2022-10-27 13:52:03] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00251 [I 2022-10-27 13:52:03] ...trap/engines/engine.py.230: loss: 0.15197 [I 2022-10-27 13:52:03] ...optimizers/trijoint.py.151: total_norm: 8.542084 [I 2022-10-27 13:52:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 1710/2304 [I 2022-10-27 13:52:08] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:29 [I 2022-10-27 13:52:08] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00265 [I 2022-10-27 13:52:08] ...trap/engines/engine.py.230: loss: 0.09569 [I 2022-10-27 13:52:08] ...optimizers/trijoint.py.151: total_norm: 9.411921 [I 2022-10-27 13:52:14] ...trap/engines/engine.py.224: train: epoch 70 | batch 1720/2304 [I 2022-10-27 13:52:14] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:26 [I 2022-10-27 13:52:14] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00266 [I 2022-10-27 13:52:14] ...trap/engines/engine.py.230: loss: 0.17774 [I 2022-10-27 13:52:14] ...optimizers/trijoint.py.151: total_norm: 12.859656 [I 2022-10-27 13:52:19] ...trap/engines/engine.py.224: train: epoch 70 | batch 1730/2304 [I 2022-10-27 13:52:19] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:19 [I 2022-10-27 13:52:19] ...trap/engines/engine.py.229: process: 0.57109 | load: 0.00254 [I 2022-10-27 13:52:19] ...trap/engines/engine.py.230: loss: 0.14585 [I 2022-10-27 13:52:19] ...optimizers/trijoint.py.151: total_norm: 13.764442 [I 2022-10-27 13:52:25] ...trap/engines/engine.py.224: train: epoch 70 | batch 1740/2304 [I 2022-10-27 13:52:25] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:13 [I 2022-10-27 13:52:25] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00257 [I 2022-10-27 13:52:25] ...trap/engines/engine.py.230: loss: 0.27967 [I 2022-10-27 13:52:25] ...optimizers/trijoint.py.151: total_norm: 26.459160 [I 2022-10-27 13:52:31] ...trap/engines/engine.py.224: train: epoch 70 | batch 1750/2304 [I 2022-10-27 13:52:31] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:10 [I 2022-10-27 13:52:31] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00264 [I 2022-10-27 13:52:31] ...trap/engines/engine.py.230: loss: 0.14185 [I 2022-10-27 13:52:31] ...optimizers/trijoint.py.151: total_norm: 103.821457 [I 2022-10-27 13:52:36] ...trap/engines/engine.py.224: train: epoch 70 | batch 1760/2304 [I 2022-10-27 13:52:36] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:03 [I 2022-10-27 13:52:36] ...trap/engines/engine.py.229: process: 0.55776 | load: 0.00263 [I 2022-10-27 13:52:36] ...trap/engines/engine.py.230: loss: 0.14485 [I 2022-10-27 13:52:36] ...optimizers/trijoint.py.151: total_norm: 17.984322 [I 2022-10-27 13:52:42] ...trap/engines/engine.py.224: train: epoch 70 | batch 1770/2304 [I 2022-10-27 13:52:42] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:04:56 [I 2022-10-27 13:52:42] ...trap/engines/engine.py.229: process: 0.54746 | load: 0.00258 [I 2022-10-27 13:52:42] ...trap/engines/engine.py.230: loss: 0.10889 [I 2022-10-27 13:52:42] ...optimizers/trijoint.py.151: total_norm: 17.432934 [I 2022-10-27 13:52:47] ...trap/engines/engine.py.224: train: epoch 70 | batch 1780/2304 [I 2022-10-27 13:52:47] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:04:53 [I 2022-10-27 13:52:47] ...trap/engines/engine.py.229: process: 0.54624 | load: 0.00242 [I 2022-10-27 13:52:47] ...trap/engines/engine.py.230: loss: 0.17475 [I 2022-10-27 13:52:47] ...optimizers/trijoint.py.151: total_norm: 20.476580 [I 2022-10-27 13:52:53] ...trap/engines/engine.py.224: train: epoch 70 | batch 1790/2304 [I 2022-10-27 13:52:53] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:49 [I 2022-10-27 13:52:53] ...trap/engines/engine.py.229: process: 0.56622 | load: 0.00254 [I 2022-10-27 13:52:53] ...trap/engines/engine.py.230: loss: 0.14121 [I 2022-10-27 13:52:53] ...optimizers/trijoint.py.151: total_norm: 17.854349 [I 2022-10-27 13:52:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 1800/2304 [I 2022-10-27 13:52:59] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:41 [I 2022-10-27 13:52:59] ...trap/engines/engine.py.229: process: 0.56577 | load: 0.00250 [I 2022-10-27 13:52:59] ...trap/engines/engine.py.230: loss: 0.13636 [I 2022-10-27 13:52:59] ...optimizers/trijoint.py.151: total_norm: 20.738329 [I 2022-10-27 13:53:04] ...trap/engines/engine.py.224: train: epoch 70 | batch 1810/2304 [I 2022-10-27 13:53:04] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:37 [I 2022-10-27 13:53:04] ...trap/engines/engine.py.229: process: 0.55001 | load: 0.00249 [I 2022-10-27 13:53:04] ...trap/engines/engine.py.230: loss: 0.17412 [I 2022-10-27 13:53:04] ...optimizers/trijoint.py.151: total_norm: 13.807148 [I 2022-10-27 13:53:10] ...trap/engines/engine.py.224: train: epoch 70 | batch 1820/2304 [I 2022-10-27 13:53:10] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:31 [I 2022-10-27 13:53:10] ...trap/engines/engine.py.229: process: 0.54982 | load: 0.00246 [I 2022-10-27 13:53:10] ...trap/engines/engine.py.230: loss: 0.19078 [I 2022-10-27 13:53:10] ...optimizers/trijoint.py.151: total_norm: 16.970989 [I 2022-10-27 13:53:15] ...trap/engines/engine.py.224: train: epoch 70 | batch 1830/2304 [I 2022-10-27 13:53:15] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:24 [I 2022-10-27 13:53:15] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00263 [I 2022-10-27 13:53:15] ...trap/engines/engine.py.230: loss: 0.10057 [I 2022-10-27 13:53:15] ...optimizers/trijoint.py.151: total_norm: 12.530131 [I 2022-10-27 13:53:21] ...trap/engines/engine.py.224: train: epoch 70 | batch 1840/2304 [I 2022-10-27 13:53:21] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:18 [I 2022-10-27 13:53:21] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00246 [I 2022-10-27 13:53:21] ...trap/engines/engine.py.230: loss: 0.14706 [I 2022-10-27 13:53:21] ...optimizers/trijoint.py.151: total_norm: 22.924028 [I 2022-10-27 13:53:27] ...trap/engines/engine.py.224: train: epoch 70 | batch 1850/2304 [I 2022-10-27 13:53:27] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:14 [I 2022-10-27 13:53:27] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00256 [I 2022-10-27 13:53:27] ...trap/engines/engine.py.230: loss: 0.15582 [I 2022-10-27 13:53:27] ...optimizers/trijoint.py.151: total_norm: 12.641447 [I 2022-10-27 13:53:32] ...trap/engines/engine.py.224: train: epoch 70 | batch 1860/2304 [I 2022-10-27 13:53:32] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:09 [I 2022-10-27 13:53:32] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00247 [I 2022-10-27 13:53:32] ...trap/engines/engine.py.230: loss: 0.12222 [I 2022-10-27 13:53:32] ...optimizers/trijoint.py.151: total_norm: 8.568665 [I 2022-10-27 13:53:38] ...trap/engines/engine.py.224: train: epoch 70 | batch 1870/2304 [I 2022-10-27 13:53:38] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:04 [I 2022-10-27 13:53:38] ...trap/engines/engine.py.229: process: 0.56164 | load: 0.00250 [I 2022-10-27 13:53:38] ...trap/engines/engine.py.230: loss: 0.09758 [I 2022-10-27 13:53:38] ...optimizers/trijoint.py.151: total_norm: 9.892772 [I 2022-10-27 13:53:44] ...trap/engines/engine.py.224: train: epoch 70 | batch 1880/2304 [I 2022-10-27 13:53:44] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:03:56 [I 2022-10-27 13:53:44] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00244 [I 2022-10-27 13:53:44] ...trap/engines/engine.py.230: loss: 0.09954 [I 2022-10-27 13:53:44] ...optimizers/trijoint.py.151: total_norm: 7.493439 [I 2022-10-27 13:53:49] ...trap/engines/engine.py.224: train: epoch 70 | batch 1890/2304 [I 2022-10-27 13:53:49] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:03:53 [I 2022-10-27 13:53:49] ...trap/engines/engine.py.229: process: 0.56434 | load: 0.00245 [I 2022-10-27 13:53:49] ...trap/engines/engine.py.230: loss: 0.15623 [I 2022-10-27 13:53:49] ...optimizers/trijoint.py.151: total_norm: 18.821609 [I 2022-10-27 13:53:55] ...trap/engines/engine.py.224: train: epoch 70 | batch 1900/2304 [I 2022-10-27 13:53:55] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:03:46 [I 2022-10-27 13:53:55] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00253 [I 2022-10-27 13:53:55] ...trap/engines/engine.py.230: loss: 0.09940 [I 2022-10-27 13:53:55] ...optimizers/trijoint.py.151: total_norm: 16.509150 [I 2022-10-27 13:54:00] ...trap/engines/engine.py.224: train: epoch 70 | batch 1910/2304 [I 2022-10-27 13:54:00] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:40 [I 2022-10-27 13:54:00] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00245 [I 2022-10-27 13:54:00] ...trap/engines/engine.py.230: loss: 0.12804 [I 2022-10-27 13:54:00] ...optimizers/trijoint.py.151: total_norm: 11.453354 [I 2022-10-27 13:54:06] ...trap/engines/engine.py.224: train: epoch 70 | batch 1920/2304 [I 2022-10-27 13:54:06] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:34 [I 2022-10-27 13:54:06] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00250 [I 2022-10-27 13:54:06] ...trap/engines/engine.py.230: loss: 0.18411 [I 2022-10-27 13:54:06] ...optimizers/trijoint.py.151: total_norm: 15.678411 [I 2022-10-27 13:54:12] ...trap/engines/engine.py.224: train: epoch 70 | batch 1930/2304 [I 2022-10-27 13:54:12] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:32 [I 2022-10-27 13:54:12] ...trap/engines/engine.py.229: process: 0.58944 | load: 0.00249 [I 2022-10-27 13:54:12] ...trap/engines/engine.py.230: loss: 0.10890 [I 2022-10-27 13:54:12] ...optimizers/trijoint.py.151: total_norm: 9.358603 [I 2022-10-27 13:54:17] ...trap/engines/engine.py.224: train: epoch 70 | batch 1940/2304 [I 2022-10-27 13:54:17] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:23 [I 2022-10-27 13:54:17] ...trap/engines/engine.py.229: process: 0.55453 | load: 0.00252 [I 2022-10-27 13:54:17] ...trap/engines/engine.py.230: loss: 0.14495 [I 2022-10-27 13:54:17] ...optimizers/trijoint.py.151: total_norm: 17.362785 [I 2022-10-27 13:54:23] ...trap/engines/engine.py.224: train: epoch 70 | batch 1950/2304 [I 2022-10-27 13:54:23] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:17 [I 2022-10-27 13:54:23] ...trap/engines/engine.py.229: process: 0.54930 | load: 0.00242 [I 2022-10-27 13:54:23] ...trap/engines/engine.py.230: loss: 0.20839 [I 2022-10-27 13:54:23] ...optimizers/trijoint.py.151: total_norm: 11.554673 [I 2022-10-27 13:54:28] ...trap/engines/engine.py.224: train: epoch 70 | batch 1960/2304 [I 2022-10-27 13:54:28] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:11 [I 2022-10-27 13:54:28] ...trap/engines/engine.py.229: process: 0.55050 | load: 0.00251 [I 2022-10-27 13:54:28] ...trap/engines/engine.py.230: loss: 0.16305 [I 2022-10-27 13:54:28] ...optimizers/trijoint.py.151: total_norm: 22.005459 [I 2022-10-27 13:54:34] ...trap/engines/engine.py.224: train: epoch 70 | batch 1970/2304 [I 2022-10-27 13:54:34] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:06 [I 2022-10-27 13:54:34] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00257 [I 2022-10-27 13:54:34] ...trap/engines/engine.py.230: loss: 0.16068 [I 2022-10-27 13:54:34] ...optimizers/trijoint.py.151: total_norm: 8.624046 [I 2022-10-27 13:54:40] ...trap/engines/engine.py.224: train: epoch 70 | batch 1980/2304 [I 2022-10-27 13:54:40] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:01 [I 2022-10-27 13:54:40] ...trap/engines/engine.py.229: process: 0.56812 | load: 0.00248 [I 2022-10-27 13:54:40] ...trap/engines/engine.py.230: loss: 0.16211 [I 2022-10-27 13:54:40] ...optimizers/trijoint.py.151: total_norm: 22.400888 [I 2022-10-27 13:54:45] ...trap/engines/engine.py.224: train: epoch 70 | batch 1990/2304 [I 2022-10-27 13:54:45] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:02:55 [I 2022-10-27 13:54:45] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00244 [I 2022-10-27 13:54:45] ...trap/engines/engine.py.230: loss: 0.15181 [I 2022-10-27 13:54:45] ...optimizers/trijoint.py.151: total_norm: 23.763912 [I 2022-10-27 13:54:51] ...trap/engines/engine.py.224: train: epoch 70 | batch 2000/2304 [I 2022-10-27 13:54:51] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:02:50 [I 2022-10-27 13:54:51] ...trap/engines/engine.py.229: process: 0.57070 | load: 0.00243 [I 2022-10-27 13:54:51] ...trap/engines/engine.py.230: loss: 0.14155 [I 2022-10-27 13:54:51] ...optimizers/trijoint.py.151: total_norm: 21.875141 [I 2022-10-27 13:54:56] ...trap/engines/engine.py.224: train: epoch 70 | batch 2010/2304 [I 2022-10-27 13:54:56] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:43 [I 2022-10-27 13:54:56] ...trap/engines/engine.py.229: process: 0.54242 | load: 0.00250 [I 2022-10-27 13:54:56] ...trap/engines/engine.py.230: loss: 0.12397 [I 2022-10-27 13:54:56] ...optimizers/trijoint.py.151: total_norm: 13.120688 [I 2022-10-27 13:55:02] ...trap/engines/engine.py.224: train: epoch 70 | batch 2020/2304 [I 2022-10-27 13:55:02] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:39 [I 2022-10-27 13:55:02] ...trap/engines/engine.py.229: process: 0.55229 | load: 0.00251 [I 2022-10-27 13:55:02] ...trap/engines/engine.py.230: loss: 0.10636 [I 2022-10-27 13:55:02] ...optimizers/trijoint.py.151: total_norm: 13.654387 [I 2022-10-27 13:55:08] ...trap/engines/engine.py.224: train: epoch 70 | batch 2030/2304 [I 2022-10-27 13:55:08] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:32 [I 2022-10-27 13:55:08] ...trap/engines/engine.py.229: process: 0.55234 | load: 0.00243 [I 2022-10-27 13:55:08] ...trap/engines/engine.py.230: loss: 0.18093 [I 2022-10-27 13:55:08] ...optimizers/trijoint.py.151: total_norm: 24.236292 [I 2022-10-27 13:55:13] ...trap/engines/engine.py.224: train: epoch 70 | batch 2040/2304 [I 2022-10-27 13:55:13] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:26 [I 2022-10-27 13:55:13] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00249 [I 2022-10-27 13:55:13] ...trap/engines/engine.py.230: loss: 0.11362 [I 2022-10-27 13:55:13] ...optimizers/trijoint.py.151: total_norm: 14.401518 [I 2022-10-27 13:55:19] ...trap/engines/engine.py.224: train: epoch 70 | batch 2050/2304 [I 2022-10-27 13:55:19] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:21 [I 2022-10-27 13:55:19] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00241 [I 2022-10-27 13:55:19] ...trap/engines/engine.py.230: loss: 0.17799 [I 2022-10-27 13:55:19] ...optimizers/trijoint.py.151: total_norm: 14.795316 [I 2022-10-27 13:55:24] ...trap/engines/engine.py.224: train: epoch 70 | batch 2060/2304 [I 2022-10-27 13:55:24] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:16 [I 2022-10-27 13:55:24] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00245 [I 2022-10-27 13:55:24] ...trap/engines/engine.py.230: loss: 0.14080 [I 2022-10-27 13:55:24] ...optimizers/trijoint.py.151: total_norm: 10.979936 [I 2022-10-27 13:55:30] ...trap/engines/engine.py.224: train: epoch 70 | batch 2070/2304 [I 2022-10-27 13:55:30] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:09 [I 2022-10-27 13:55:30] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00255 [I 2022-10-27 13:55:30] ...trap/engines/engine.py.230: loss: 0.15462 [I 2022-10-27 13:55:30] ...optimizers/trijoint.py.151: total_norm: 25.495874 [I 2022-10-27 13:55:36] ...trap/engines/engine.py.224: train: epoch 70 | batch 2080/2304 [I 2022-10-27 13:55:36] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:04 [I 2022-10-27 13:55:36] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00249 [I 2022-10-27 13:55:36] ...trap/engines/engine.py.230: loss: 0.13292 [I 2022-10-27 13:55:36] ...optimizers/trijoint.py.151: total_norm: 31.951630 [I 2022-10-27 13:55:41] ...trap/engines/engine.py.224: train: epoch 70 | batch 2090/2304 [I 2022-10-27 13:55:41] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:01:59 [I 2022-10-27 13:55:41] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00248 [I 2022-10-27 13:55:41] ...trap/engines/engine.py.230: loss: 0.09635 [I 2022-10-27 13:55:41] ...optimizers/trijoint.py.151: total_norm: 9.252484 [I 2022-10-27 13:55:47] ...trap/engines/engine.py.224: train: epoch 70 | batch 2100/2304 [I 2022-10-27 13:55:47] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:01:53 [I 2022-10-27 13:55:47] ...trap/engines/engine.py.229: process: 0.54792 | load: 0.00244 [I 2022-10-27 13:55:47] ...trap/engines/engine.py.230: loss: 0.20821 [I 2022-10-27 13:55:47] ...optimizers/trijoint.py.151: total_norm: 33.174770 [I 2022-10-27 13:55:52] ...trap/engines/engine.py.224: train: epoch 70 | batch 2110/2304 [I 2022-10-27 13:55:52] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:01:48 [I 2022-10-27 13:55:52] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00246 [I 2022-10-27 13:55:52] ...trap/engines/engine.py.230: loss: 0.12820 [I 2022-10-27 13:55:52] ...optimizers/trijoint.py.151: total_norm: 14.582092 [I 2022-10-27 13:55:58] ...trap/engines/engine.py.224: train: epoch 70 | batch 2120/2304 [I 2022-10-27 13:55:58] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:41 [I 2022-10-27 13:55:58] ...trap/engines/engine.py.229: process: 0.55202 | load: 0.00252 [I 2022-10-27 13:55:58] ...trap/engines/engine.py.230: loss: 0.12587 [I 2022-10-27 13:55:58] ...optimizers/trijoint.py.151: total_norm: 16.136127 [I 2022-10-27 13:56:03] ...trap/engines/engine.py.224: train: epoch 70 | batch 2130/2304 [I 2022-10-27 13:56:03] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:36 [I 2022-10-27 13:56:03] ...trap/engines/engine.py.229: process: 0.54723 | load: 0.00250 [I 2022-10-27 13:56:03] ...trap/engines/engine.py.230: loss: 0.18488 [I 2022-10-27 13:56:03] ...optimizers/trijoint.py.151: total_norm: 15.870504 [I 2022-10-27 13:56:09] ...trap/engines/engine.py.224: train: epoch 70 | batch 2140/2304 [I 2022-10-27 13:56:09] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:31 [I 2022-10-27 13:56:09] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00252 [I 2022-10-27 13:56:09] ...trap/engines/engine.py.230: loss: 0.16706 [I 2022-10-27 13:56:09] ...optimizers/trijoint.py.151: total_norm: 53.381115 [I 2022-10-27 13:56:15] ...trap/engines/engine.py.224: train: epoch 70 | batch 2150/2304 [I 2022-10-27 13:56:15] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:28 [I 2022-10-27 13:56:15] ...trap/engines/engine.py.229: process: 0.59013 | load: 0.00250 [I 2022-10-27 13:56:15] ...trap/engines/engine.py.230: loss: 0.08767 [I 2022-10-27 13:56:15] ...optimizers/trijoint.py.151: total_norm: 9.579177 [I 2022-10-27 13:56:20] ...trap/engines/engine.py.224: train: epoch 70 | batch 2160/2304 [I 2022-10-27 13:56:20] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:20 [I 2022-10-27 13:56:20] ...trap/engines/engine.py.229: process: 0.56805 | load: 0.00239 [I 2022-10-27 13:56:20] ...trap/engines/engine.py.230: loss: 0.14598 [I 2022-10-27 13:56:20] ...optimizers/trijoint.py.151: total_norm: 11.620570 [I 2022-10-27 13:56:26] ...trap/engines/engine.py.224: train: epoch 70 | batch 2170/2304 [I 2022-10-27 13:56:26] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:15 [I 2022-10-27 13:56:26] ...trap/engines/engine.py.229: process: 0.56647 | load: 0.00243 [I 2022-10-27 13:56:26] ...trap/engines/engine.py.230: loss: 0.15033 [I 2022-10-27 13:56:26] ...optimizers/trijoint.py.151: total_norm: 12.046686 [I 2022-10-27 13:56:32] ...trap/engines/engine.py.224: train: epoch 70 | batch 2180/2304 [I 2022-10-27 13:56:32] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:09 [I 2022-10-27 13:56:32] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00249 [I 2022-10-27 13:56:32] ...trap/engines/engine.py.230: loss: 0.16202 [I 2022-10-27 13:56:32] ...optimizers/trijoint.py.151: total_norm: 25.223749 [I 2022-10-27 13:56:37] ...trap/engines/engine.py.224: train: epoch 70 | batch 2190/2304 [I 2022-10-27 13:56:37] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:03 [I 2022-10-27 13:56:37] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00247 [I 2022-10-27 13:56:37] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-27 13:56:37] ...optimizers/trijoint.py.151: total_norm: 11.077179 [I 2022-10-27 13:56:43] ...trap/engines/engine.py.224: train: epoch 70 | batch 2200/2304 [I 2022-10-27 13:56:43] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:00:58 [I 2022-10-27 13:56:43] ...trap/engines/engine.py.229: process: 0.55189 | load: 0.00246 [I 2022-10-27 13:56:43] ...trap/engines/engine.py.230: loss: 0.15195 [I 2022-10-27 13:56:43] ...optimizers/trijoint.py.151: total_norm: 20.553415 [I 2022-10-27 13:56:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 2210/2304 [I 2022-10-27 13:56:48] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:00:52 [I 2022-10-27 13:56:48] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00259 [I 2022-10-27 13:56:48] ...trap/engines/engine.py.230: loss: 0.13565 [I 2022-10-27 13:56:48] ...optimizers/trijoint.py.151: total_norm: 19.521143 [I 2022-10-27 13:56:54] ...trap/engines/engine.py.224: train: epoch 70 | batch 2220/2304 [I 2022-10-27 13:56:54] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:47 [I 2022-10-27 13:56:54] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00243 [I 2022-10-27 13:56:54] ...trap/engines/engine.py.230: loss: 0.09534 [I 2022-10-27 13:56:54] ...optimizers/trijoint.py.151: total_norm: 6.744663 [I 2022-10-27 13:57:00] ...trap/engines/engine.py.224: train: epoch 70 | batch 2230/2304 [I 2022-10-27 13:57:00] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:41 [I 2022-10-27 13:57:00] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00245 [I 2022-10-27 13:57:00] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-27 13:57:00] ...optimizers/trijoint.py.151: total_norm: 10.845695 [I 2022-10-27 13:57:05] ...trap/engines/engine.py.224: train: epoch 70 | batch 2240/2304 [I 2022-10-27 13:57:05] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:35 [I 2022-10-27 13:57:05] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00243 [I 2022-10-27 13:57:05] ...trap/engines/engine.py.230: loss: 0.12328 [I 2022-10-27 13:57:05] ...optimizers/trijoint.py.151: total_norm: 30.681841 [I 2022-10-27 13:57:11] ...trap/engines/engine.py.224: train: epoch 70 | batch 2250/2304 [I 2022-10-27 13:57:11] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:30 [I 2022-10-27 13:57:11] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00247 [I 2022-10-27 13:57:11] ...trap/engines/engine.py.230: loss: 0.14008 [I 2022-10-27 13:57:11] ...optimizers/trijoint.py.151: total_norm: 8.881488 [I 2022-10-27 13:57:16] ...trap/engines/engine.py.224: train: epoch 70 | batch 2260/2304 [I 2022-10-27 13:57:16] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:24 [I 2022-10-27 13:57:16] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00240 [I 2022-10-27 13:57:16] ...trap/engines/engine.py.230: loss: 0.17204 [I 2022-10-27 13:57:16] ...optimizers/trijoint.py.151: total_norm: 32.450859 [I 2022-10-27 13:57:43] ...trap/engines/engine.py.224: train: epoch 70 | batch 2270/2304 [I 2022-10-27 13:57:43] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:01:30 [I 2022-10-27 13:57:43] ...trap/engines/engine.py.229: process: 0.56841 | load: 0.00240 [I 2022-10-27 13:57:43] ...trap/engines/engine.py.230: loss: 0.11238 [I 2022-10-27 13:57:43] ...optimizers/trijoint.py.151: total_norm: 30.347536 [I 2022-10-27 13:57:48] ...trap/engines/engine.py.224: train: epoch 70 | batch 2280/2304 [I 2022-10-27 13:57:48] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:00:18 [I 2022-10-27 13:57:48] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00241 [I 2022-10-27 13:57:48] ...trap/engines/engine.py.230: loss: 0.16190 [I 2022-10-27 13:57:48] ...optimizers/trijoint.py.151: total_norm: 18.850555 [I 2022-10-27 13:57:54] ...trap/engines/engine.py.224: train: epoch 70 | batch 2290/2304 [I 2022-10-27 13:57:54] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:08 [I 2022-10-27 13:57:54] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00245 [I 2022-10-27 13:57:54] ...trap/engines/engine.py.230: loss: 0.14644 [I 2022-10-27 13:57:54] ...optimizers/trijoint.py.151: total_norm: 13.739328 [I 2022-10-27 13:57:59] ...trap/engines/engine.py.224: train: epoch 70 | batch 2300/2304 [I 2022-10-27 13:57:59] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:00:02 [I 2022-10-27 13:57:59] ...trap/engines/engine.py.229: process: 0.55263 | load: 0.00015 [I 2022-10-27 13:57:59] ...trap/engines/engine.py.230: loss: 0.11331 [I 2022-10-27 13:57:59] ...optimizers/trijoint.py.151: total_norm: 8.764747 [I 2022-10-27 13:58:02] ...trap/engines/engine.py.224: train: epoch 70 | batch 2304/2304 [I 2022-10-27 13:58:02] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:00:00 [I 2022-10-27 13:58:02] ...trap/engines/engine.py.229: process: 0.54153 | load: 0.00016 [I 2022-10-27 13:58:02] ...trap/engines/engine.py.230: loss: 0.09349 [I 2022-10-27 13:58:02] ...optimizers/trijoint.py.151: total_norm: 10.246921 [S 2022-10-27 13:58:03] ...trap/engines/engine.py.129: train_epoch.epoch: 70 [S 2022-10-27 13:58:03] ...trap/engines/engine.py.129: train_epoch.loss: 0.14632725598302684 [I 2022-10-27 13:58:03] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 13:58:04] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 70 [I 2022-10-27 13:58:04] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 13:58:06] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 13:58:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 13:58:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 13:58:08] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 13:58:09] ...trap/engines/engine.py.321: eval: epoch 70 | batch 0/510 [I 2022-10-27 13:58:09] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:40:39 [I 2022-10-27 13:58:09] ...trap/engines/engine.py.326: process: 4.78322 | load: 3.10567 [I 2022-10-27 13:58:12] ...trap/engines/engine.py.321: eval: epoch 70 | batch 10/510 [I 2022-10-27 13:58:12] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:46 [I 2022-10-27 13:58:12] ...trap/engines/engine.py.326: process: 0.32412 | load: 0.00018 [I 2022-10-27 13:58:15] ...trap/engines/engine.py.321: eval: epoch 70 | batch 20/510 [I 2022-10-27 13:58:15] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:07 [I 2022-10-27 13:58:15] ...trap/engines/engine.py.326: process: 0.33307 | load: 0.00017 [I 2022-10-27 13:58:19] ...trap/engines/engine.py.321: eval: epoch 70 | batch 30/510 [I 2022-10-27 13:58:19] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:43 [I 2022-10-27 13:58:19] ...trap/engines/engine.py.326: process: 0.33465 | load: 0.00017 [I 2022-10-27 13:58:22] ...trap/engines/engine.py.321: eval: epoch 70 | batch 40/510 [I 2022-10-27 13:58:22] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:39 [I 2022-10-27 13:58:22] ...trap/engines/engine.py.326: process: 0.36227 | load: 0.00019 [I 2022-10-27 13:58:26] ...trap/engines/engine.py.321: eval: epoch 70 | batch 50/510 [I 2022-10-27 13:58:26] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:37 [I 2022-10-27 13:58:26] ...trap/engines/engine.py.326: process: 0.32874 | load: 0.00020 [I 2022-10-27 13:58:29] ...trap/engines/engine.py.321: eval: epoch 70 | batch 60/510 [I 2022-10-27 13:58:29] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:33 [I 2022-10-27 13:58:29] ...trap/engines/engine.py.326: process: 0.36212 | load: 0.00018 [I 2022-10-27 13:58:32] ...trap/engines/engine.py.321: eval: epoch 70 | batch 70/510 [I 2022-10-27 13:58:32] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:30 [I 2022-10-27 13:58:32] ...trap/engines/engine.py.326: process: 0.35615 | load: 0.00021 [I 2022-10-27 13:58:36] ...trap/engines/engine.py.321: eval: epoch 70 | batch 80/510 [I 2022-10-27 13:58:36] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:25 [I 2022-10-27 13:58:36] ...trap/engines/engine.py.326: process: 0.33424 | load: 0.00020 [I 2022-10-27 13:58:39] ...trap/engines/engine.py.321: eval: epoch 70 | batch 90/510 [I 2022-10-27 13:58:39] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-27 13:58:39] ...trap/engines/engine.py.326: process: 0.32769 | load: 0.00020 [I 2022-10-27 13:58:42] ...trap/engines/engine.py.321: eval: epoch 70 | batch 100/510 [I 2022-10-27 13:58:42] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-27 13:58:42] ...trap/engines/engine.py.326: process: 0.29111 | load: 0.00019 [I 2022-10-27 13:58:45] ...trap/engines/engine.py.321: eval: epoch 70 | batch 110/510 [I 2022-10-27 13:58:45] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:02:03 [I 2022-10-27 13:58:45] ...trap/engines/engine.py.326: process: 0.29620 | load: 0.00025 [I 2022-10-27 13:58:48] ...trap/engines/engine.py.321: eval: epoch 70 | batch 120/510 [I 2022-10-27 13:58:48] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:55 [I 2022-10-27 13:58:48] ...trap/engines/engine.py.326: process: 0.30710 | load: 0.00022 [I 2022-10-27 13:58:51] ...trap/engines/engine.py.321: eval: epoch 70 | batch 130/510 [I 2022-10-27 13:58:51] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:53 [I 2022-10-27 13:58:51] ...trap/engines/engine.py.326: process: 0.31490 | load: 0.00024 [I 2022-10-27 13:58:54] ...trap/engines/engine.py.321: eval: epoch 70 | batch 140/510 [I 2022-10-27 13:58:54] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:48 [I 2022-10-27 13:58:54] ...trap/engines/engine.py.326: process: 0.28015 | load: 0.00018 [I 2022-10-27 13:58:57] ...trap/engines/engine.py.321: eval: epoch 70 | batch 150/510 [I 2022-10-27 13:58:57] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:44 [I 2022-10-27 13:58:57] ...trap/engines/engine.py.326: process: 0.28378 | load: 0.00018 [I 2022-10-27 13:59:00] ...trap/engines/engine.py.321: eval: epoch 70 | batch 160/510 [I 2022-10-27 13:59:00] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-27 13:59:00] ...trap/engines/engine.py.326: process: 0.28908 | load: 0.00016 [I 2022-10-27 13:59:03] ...trap/engines/engine.py.321: eval: epoch 70 | batch 170/510 [I 2022-10-27 13:59:03] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-27 13:59:03] ...trap/engines/engine.py.326: process: 0.29276 | load: 0.00021 [I 2022-10-27 13:59:06] ...trap/engines/engine.py.321: eval: epoch 70 | batch 180/510 [I 2022-10-27 13:59:06] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 13:59:06] ...trap/engines/engine.py.326: process: 0.28686 | load: 0.00017 [I 2022-10-27 13:59:09] ...trap/engines/engine.py.321: eval: epoch 70 | batch 190/510 [I 2022-10-27 13:59:09] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:34 [I 2022-10-27 13:59:09] ...trap/engines/engine.py.326: process: 0.30146 | load: 0.00018 [I 2022-10-27 13:59:12] ...trap/engines/engine.py.321: eval: epoch 70 | batch 200/510 [I 2022-10-27 13:59:12] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:31 [I 2022-10-27 13:59:12] ...trap/engines/engine.py.326: process: 0.31039 | load: 0.00020 [I 2022-10-27 13:59:15] ...trap/engines/engine.py.321: eval: epoch 70 | batch 210/510 [I 2022-10-27 13:59:15] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:29 [I 2022-10-27 13:59:15] ...trap/engines/engine.py.326: process: 0.31575 | load: 0.00015 [I 2022-10-27 13:59:18] ...trap/engines/engine.py.321: eval: epoch 70 | batch 220/510 [I 2022-10-27 13:59:18] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:25 [I 2022-10-27 13:59:18] ...trap/engines/engine.py.326: process: 0.29718 | load: 0.00022 [I 2022-10-27 13:59:21] ...trap/engines/engine.py.321: eval: epoch 70 | batch 230/510 [I 2022-10-27 13:59:21] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:21 [I 2022-10-27 13:59:21] ...trap/engines/engine.py.326: process: 0.29149 | load: 0.00017 [I 2022-10-27 13:59:23] ...trap/engines/engine.py.321: eval: epoch 70 | batch 240/510 [I 2022-10-27 13:59:23] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-27 13:59:23] ...trap/engines/engine.py.326: process: 0.29558 | load: 0.00018 [I 2022-10-27 13:59:26] ...trap/engines/engine.py.321: eval: epoch 70 | batch 250/510 [I 2022-10-27 13:59:26] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-27 13:59:26] ...trap/engines/engine.py.326: process: 0.28752 | load: 0.00022 [I 2022-10-27 13:59:29] ...trap/engines/engine.py.321: eval: epoch 70 | batch 260/510 [I 2022-10-27 13:59:29] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-27 13:59:29] ...trap/engines/engine.py.326: process: 0.29225 | load: 0.00029 [I 2022-10-27 13:59:32] ...trap/engines/engine.py.321: eval: epoch 70 | batch 270/510 [I 2022-10-27 13:59:32] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-27 13:59:32] ...trap/engines/engine.py.326: process: 0.28509 | load: 0.00023 [I 2022-10-27 13:59:35] ...trap/engines/engine.py.321: eval: epoch 70 | batch 280/510 [I 2022-10-27 13:59:35] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-27 13:59:35] ...trap/engines/engine.py.326: process: 0.29485 | load: 0.00026 [I 2022-10-27 13:59:38] ...trap/engines/engine.py.321: eval: epoch 70 | batch 290/510 [I 2022-10-27 13:59:38] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:04 [I 2022-10-27 13:59:38] ...trap/engines/engine.py.326: process: 0.30030 | load: 0.00015 [I 2022-10-27 13:59:41] ...trap/engines/engine.py.321: eval: epoch 70 | batch 300/510 [I 2022-10-27 13:59:41] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:02 [I 2022-10-27 13:59:41] ...trap/engines/engine.py.326: process: 0.30164 | load: 0.00025 [I 2022-10-27 13:59:44] ...trap/engines/engine.py.321: eval: epoch 70 | batch 310/510 [I 2022-10-27 13:59:44] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:58 [I 2022-10-27 13:59:44] ...trap/engines/engine.py.326: process: 0.28639 | load: 0.00016 [I 2022-10-27 13:59:47] ...trap/engines/engine.py.321: eval: epoch 70 | batch 320/510 [I 2022-10-27 13:59:47] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 13:59:47] ...trap/engines/engine.py.326: process: 0.29268 | load: 0.00020 [I 2022-10-27 13:59:50] ...trap/engines/engine.py.321: eval: epoch 70 | batch 330/510 [I 2022-10-27 13:59:50] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-27 13:59:50] ...trap/engines/engine.py.326: process: 0.28767 | load: 0.00015 [I 2022-10-27 13:59:53] ...trap/engines/engine.py.321: eval: epoch 70 | batch 340/510 [I 2022-10-27 13:59:53] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:51 [I 2022-10-27 13:59:53] ...trap/engines/engine.py.326: process: 0.36025 | load: 0.00032 [I 2022-10-27 13:59:55] ...trap/engines/engine.py.321: eval: epoch 70 | batch 350/510 [I 2022-10-27 13:59:55] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 13:59:55] ...trap/engines/engine.py.326: process: 0.28699 | load: 0.00020 [I 2022-10-27 13:59:58] ...trap/engines/engine.py.321: eval: epoch 70 | batch 360/510 [I 2022-10-27 13:59:58] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-27 13:59:58] ...trap/engines/engine.py.326: process: 0.28529 | load: 0.00021 [I 2022-10-27 14:00:01] ...trap/engines/engine.py.321: eval: epoch 70 | batch 370/510 [I 2022-10-27 14:00:01] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:39 [I 2022-10-27 14:00:01] ...trap/engines/engine.py.326: process: 0.28292 | load: 0.00022 [I 2022-10-27 14:00:04] ...trap/engines/engine.py.321: eval: epoch 70 | batch 380/510 [I 2022-10-27 14:00:04] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-27 14:00:04] ...trap/engines/engine.py.326: process: 0.30114 | load: 0.00020 [I 2022-10-27 14:00:07] ...trap/engines/engine.py.321: eval: epoch 70 | batch 390/510 [I 2022-10-27 14:00:07] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:35 [I 2022-10-27 14:00:07] ...trap/engines/engine.py.326: process: 0.29853 | load: 0.00020 [I 2022-10-27 14:00:10] ...trap/engines/engine.py.321: eval: epoch 70 | batch 400/510 [I 2022-10-27 14:00:10] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:32 [I 2022-10-27 14:00:10] ...trap/engines/engine.py.326: process: 0.29411 | load: 0.00022 [I 2022-10-27 14:00:13] ...trap/engines/engine.py.321: eval: epoch 70 | batch 410/510 [I 2022-10-27 14:00:13] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 14:00:13] ...trap/engines/engine.py.326: process: 0.28885 | load: 0.00020 [I 2022-10-27 14:00:16] ...trap/engines/engine.py.321: eval: epoch 70 | batch 420/510 [I 2022-10-27 14:00:16] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-27 14:00:16] ...trap/engines/engine.py.326: process: 0.30338 | load: 0.00025 [I 2022-10-27 14:00:19] ...trap/engines/engine.py.321: eval: epoch 70 | batch 430/510 [I 2022-10-27 14:00:19] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 14:00:19] ...trap/engines/engine.py.326: process: 0.28809 | load: 0.00021 [I 2022-10-27 14:00:22] ...trap/engines/engine.py.321: eval: epoch 70 | batch 440/510 [I 2022-10-27 14:00:22] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 14:00:22] ...trap/engines/engine.py.326: process: 0.28940 | load: 0.00020 [I 2022-10-27 14:00:24] ...trap/engines/engine.py.321: eval: epoch 70 | batch 450/510 [I 2022-10-27 14:00:24] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 14:00:24] ...trap/engines/engine.py.326: process: 0.28346 | load: 0.00020 [I 2022-10-27 14:00:27] ...trap/engines/engine.py.321: eval: epoch 70 | batch 460/510 [I 2022-10-27 14:00:27] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-27 14:00:27] ...trap/engines/engine.py.326: process: 0.28855 | load: 0.00016 [I 2022-10-27 14:00:30] ...trap/engines/engine.py.321: eval: epoch 70 | batch 470/510 [I 2022-10-27 14:00:30] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-27 14:00:30] ...trap/engines/engine.py.326: process: 0.28973 | load: 0.00021 [I 2022-10-27 14:00:33] ...trap/engines/engine.py.321: eval: epoch 70 | batch 480/510 [I 2022-10-27 14:00:33] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-27 14:00:33] ...trap/engines/engine.py.326: process: 0.28481 | load: 0.00022 [I 2022-10-27 14:00:36] ...trap/engines/engine.py.321: eval: epoch 70 | batch 490/510 [I 2022-10-27 14:00:36] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 14:00:36] ...trap/engines/engine.py.326: process: 0.28597 | load: 0.00032 [I 2022-10-27 14:00:39] ...trap/engines/engine.py.321: eval: epoch 70 | batch 500/510 [I 2022-10-27 14:00:39] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 14:00:39] ...trap/engines/engine.py.326: process: 0.28336 | load: 0.00016 [I 2022-10-27 14:00:42] ...trap/engines/engine.py.321: eval: epoch 70 | batch 510/510 [I 2022-10-27 14:00:42] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 14:00:42] ...trap/engines/engine.py.326: process: 0.28890 | load: 0.00020 [S 2022-10-27 14:00:42] ...trap/engines/engine.py.132: eval_epoch.epoch: 70 [I 2022-10-27 14:00:42] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.712800 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.896200 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.931800 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.703100 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.895300 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.927400 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014538 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009282 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006896 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011844 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010335 [S 2022-10-27 14:00:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006666 [I 2022-10-27 14:00:48] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 14:00:48] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 14:00:48] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 14:00:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 14:00:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 14:00:52] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 14:00:53] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 14:00:59] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 14:00:59] ...trap/engines/engine.py.163: Training model on trainset for epoch 71 [I 2022-10-27 14:00:59] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 14:01:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 0/2304 [I 2022-10-27 14:01:02] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:49:19 [I 2022-10-27 14:01:02] ...trap/engines/engine.py.229: process: 2.84713 | load: 2.19036 [I 2022-10-27 14:01:02] ...trap/engines/engine.py.230: loss: 0.19636 [I 2022-10-27 14:01:02] ...optimizers/trijoint.py.151: total_norm: 19.835592 [I 2022-10-27 14:01:08] ...trap/engines/engine.py.224: train: epoch 71 | batch 10/2304 [I 2022-10-27 14:01:08] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:48 [I 2022-10-27 14:01:08] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00457 [I 2022-10-27 14:01:08] ...trap/engines/engine.py.230: loss: 0.13185 [I 2022-10-27 14:01:08] ...optimizers/trijoint.py.151: total_norm: 11.833702 [I 2022-10-27 14:01:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 20/2304 [I 2022-10-27 14:01:13] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:09 [I 2022-10-27 14:01:13] ...trap/engines/engine.py.229: process: 0.54625 | load: 0.00315 [I 2022-10-27 14:01:13] ...trap/engines/engine.py.230: loss: 0.13798 [I 2022-10-27 14:01:13] ...optimizers/trijoint.py.151: total_norm: 17.820335 [I 2022-10-27 14:01:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 30/2304 [I 2022-10-27 14:01:19] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-27 14:01:19] ...trap/engines/engine.py.229: process: 0.56701 | load: 0.00328 [I 2022-10-27 14:01:19] ...trap/engines/engine.py.230: loss: 0.14058 [I 2022-10-27 14:01:19] ...optimizers/trijoint.py.151: total_norm: 9.206429 [I 2022-10-27 14:01:24] ...trap/engines/engine.py.224: train: epoch 71 | batch 40/2304 [I 2022-10-27 14:01:24] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:00 [I 2022-10-27 14:01:24] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00345 [I 2022-10-27 14:01:24] ...trap/engines/engine.py.230: loss: 0.15390 [I 2022-10-27 14:01:24] ...optimizers/trijoint.py.151: total_norm: 13.785099 [I 2022-10-27 14:01:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 50/2304 [I 2022-10-27 14:01:30] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:55 [I 2022-10-27 14:01:30] ...trap/engines/engine.py.229: process: 0.55628 | load: 0.00310 [I 2022-10-27 14:01:30] ...trap/engines/engine.py.230: loss: 0.09955 [I 2022-10-27 14:01:30] ...optimizers/trijoint.py.151: total_norm: 9.853799 [I 2022-10-27 14:01:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 60/2304 [I 2022-10-27 14:01:36] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:51 [I 2022-10-27 14:01:36] ...trap/engines/engine.py.229: process: 0.54867 | load: 0.00284 [I 2022-10-27 14:01:36] ...trap/engines/engine.py.230: loss: 0.32136 [I 2022-10-27 14:01:36] ...optimizers/trijoint.py.151: total_norm: 36.659027 [I 2022-10-27 14:01:41] ...trap/engines/engine.py.224: train: epoch 71 | batch 70/2304 [I 2022-10-27 14:01:41] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:46 [I 2022-10-27 14:01:41] ...trap/engines/engine.py.229: process: 0.55485 | load: 0.00291 [I 2022-10-27 14:01:41] ...trap/engines/engine.py.230: loss: 0.13603 [I 2022-10-27 14:01:41] ...optimizers/trijoint.py.151: total_norm: 12.226470 [I 2022-10-27 14:01:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 80/2304 [I 2022-10-27 14:01:47] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:40 [I 2022-10-27 14:01:47] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00262 [I 2022-10-27 14:01:47] ...trap/engines/engine.py.230: loss: 0.09598 [I 2022-10-27 14:01:47] ...optimizers/trijoint.py.151: total_norm: 8.338696 [I 2022-10-27 14:01:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 90/2304 [I 2022-10-27 14:01:52] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:28 [I 2022-10-27 14:01:52] ...trap/engines/engine.py.229: process: 0.55437 | load: 0.00270 [I 2022-10-27 14:01:52] ...trap/engines/engine.py.230: loss: 0.09768 [I 2022-10-27 14:01:52] ...optimizers/trijoint.py.151: total_norm: 9.940766 [I 2022-10-27 14:01:58] ...trap/engines/engine.py.224: train: epoch 71 | batch 100/2304 [I 2022-10-27 14:01:58] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:12 [I 2022-10-27 14:01:58] ...trap/engines/engine.py.229: process: 0.53746 | load: 0.00282 [I 2022-10-27 14:01:58] ...trap/engines/engine.py.230: loss: 0.12183 [I 2022-10-27 14:01:58] ...optimizers/trijoint.py.151: total_norm: 8.571198 [I 2022-10-27 14:02:03] ...trap/engines/engine.py.224: train: epoch 71 | batch 110/2304 [I 2022-10-27 14:02:03] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:24 [I 2022-10-27 14:02:03] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00271 [I 2022-10-27 14:02:03] ...trap/engines/engine.py.230: loss: 0.10212 [I 2022-10-27 14:02:03] ...optimizers/trijoint.py.151: total_norm: 11.556072 [I 2022-10-27 14:02:09] ...trap/engines/engine.py.224: train: epoch 71 | batch 120/2304 [I 2022-10-27 14:02:09] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:12 [I 2022-10-27 14:02:09] ...trap/engines/engine.py.229: process: 0.54970 | load: 0.00268 [I 2022-10-27 14:02:09] ...trap/engines/engine.py.230: loss: 0.18963 [I 2022-10-27 14:02:09] ...optimizers/trijoint.py.151: total_norm: 15.333866 [I 2022-10-27 14:02:15] ...trap/engines/engine.py.224: train: epoch 71 | batch 130/2304 [I 2022-10-27 14:02:15] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:12 [I 2022-10-27 14:02:15] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00287 [I 2022-10-27 14:02:15] ...trap/engines/engine.py.230: loss: 0.09783 [I 2022-10-27 14:02:15] ...optimizers/trijoint.py.151: total_norm: 11.598317 [I 2022-10-27 14:02:20] ...trap/engines/engine.py.224: train: epoch 71 | batch 140/2304 [I 2022-10-27 14:02:20] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:05 [I 2022-10-27 14:02:20] ...trap/engines/engine.py.229: process: 0.55509 | load: 0.00277 [I 2022-10-27 14:02:20] ...trap/engines/engine.py.230: loss: 0.20561 [I 2022-10-27 14:02:20] ...optimizers/trijoint.py.151: total_norm: 21.664158 [I 2022-10-27 14:02:26] ...trap/engines/engine.py.224: train: epoch 71 | batch 150/2304 [I 2022-10-27 14:02:26] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:12 [I 2022-10-27 14:02:26] ...trap/engines/engine.py.229: process: 0.56644 | load: 0.00281 [I 2022-10-27 14:02:26] ...trap/engines/engine.py.230: loss: 0.12243 [I 2022-10-27 14:02:26] ...optimizers/trijoint.py.151: total_norm: 8.100934 [I 2022-10-27 14:02:32] ...trap/engines/engine.py.224: train: epoch 71 | batch 160/2304 [I 2022-10-27 14:02:32] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:09 [I 2022-10-27 14:02:32] ...trap/engines/engine.py.229: process: 0.56744 | load: 0.00266 [I 2022-10-27 14:02:32] ...trap/engines/engine.py.230: loss: 0.14719 [I 2022-10-27 14:02:32] ...optimizers/trijoint.py.151: total_norm: 14.711848 [I 2022-10-27 14:02:37] ...trap/engines/engine.py.224: train: epoch 71 | batch 170/2304 [I 2022-10-27 14:02:37] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:41 [I 2022-10-27 14:02:37] ...trap/engines/engine.py.229: process: 0.55144 | load: 0.00256 [I 2022-10-27 14:02:37] ...trap/engines/engine.py.230: loss: 0.11315 [I 2022-10-27 14:02:37] ...optimizers/trijoint.py.151: total_norm: 22.183390 [I 2022-10-27 14:02:43] ...trap/engines/engine.py.224: train: epoch 71 | batch 180/2304 [I 2022-10-27 14:02:43] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:39 [I 2022-10-27 14:02:43] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00260 [I 2022-10-27 14:02:43] ...trap/engines/engine.py.230: loss: 0.11699 [I 2022-10-27 14:02:43] ...optimizers/trijoint.py.151: total_norm: 10.899722 [I 2022-10-27 14:02:48] ...trap/engines/engine.py.224: train: epoch 71 | batch 190/2304 [I 2022-10-27 14:02:48] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:30 [I 2022-10-27 14:02:48] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00258 [I 2022-10-27 14:02:48] ...trap/engines/engine.py.230: loss: 0.14610 [I 2022-10-27 14:02:48] ...optimizers/trijoint.py.151: total_norm: 9.726201 [I 2022-10-27 14:02:54] ...trap/engines/engine.py.224: train: epoch 71 | batch 200/2304 [I 2022-10-27 14:02:54] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:27 [I 2022-10-27 14:02:54] ...trap/engines/engine.py.229: process: 0.54076 | load: 0.00258 [I 2022-10-27 14:02:54] ...trap/engines/engine.py.230: loss: 0.11779 [I 2022-10-27 14:02:54] ...optimizers/trijoint.py.151: total_norm: 9.013471 [I 2022-10-27 14:02:59] ...trap/engines/engine.py.224: train: epoch 71 | batch 210/2304 [I 2022-10-27 14:02:59] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:30 [I 2022-10-27 14:02:59] ...trap/engines/engine.py.229: process: 0.55451 | load: 0.00246 [I 2022-10-27 14:02:59] ...trap/engines/engine.py.230: loss: 0.16722 [I 2022-10-27 14:02:59] ...optimizers/trijoint.py.151: total_norm: 24.030659 [I 2022-10-27 14:03:05] ...trap/engines/engine.py.224: train: epoch 71 | batch 220/2304 [I 2022-10-27 14:03:05] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:27 [I 2022-10-27 14:03:05] ...trap/engines/engine.py.229: process: 0.55547 | load: 0.00264 [I 2022-10-27 14:03:05] ...trap/engines/engine.py.230: loss: 0.17907 [I 2022-10-27 14:03:05] ...optimizers/trijoint.py.151: total_norm: 28.031462 [I 2022-10-27 14:03:11] ...trap/engines/engine.py.224: train: epoch 71 | batch 230/2304 [I 2022-10-27 14:03:11] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:22 [I 2022-10-27 14:03:11] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00253 [I 2022-10-27 14:03:11] ...trap/engines/engine.py.230: loss: 0.13368 [I 2022-10-27 14:03:11] ...optimizers/trijoint.py.151: total_norm: 14.467875 [I 2022-10-27 14:03:16] ...trap/engines/engine.py.224: train: epoch 71 | batch 240/2304 [I 2022-10-27 14:03:16] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:15 [I 2022-10-27 14:03:16] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00249 [I 2022-10-27 14:03:16] ...trap/engines/engine.py.230: loss: 0.10846 [I 2022-10-27 14:03:16] ...optimizers/trijoint.py.151: total_norm: 30.060698 [I 2022-10-27 14:03:22] ...trap/engines/engine.py.224: train: epoch 71 | batch 250/2304 [I 2022-10-27 14:03:22] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:03 [I 2022-10-27 14:03:22] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00259 [I 2022-10-27 14:03:22] ...trap/engines/engine.py.230: loss: 0.13875 [I 2022-10-27 14:03:22] ...optimizers/trijoint.py.151: total_norm: 16.595644 [I 2022-10-27 14:03:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 260/2304 [I 2022-10-27 14:03:27] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:54 [I 2022-10-27 14:03:27] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00298 [I 2022-10-27 14:03:27] ...trap/engines/engine.py.230: loss: 0.14996 [I 2022-10-27 14:03:27] ...optimizers/trijoint.py.151: total_norm: 14.522410 [I 2022-10-27 14:03:33] ...trap/engines/engine.py.224: train: epoch 71 | batch 270/2304 [I 2022-10-27 14:03:33] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:54 [I 2022-10-27 14:03:33] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00256 [I 2022-10-27 14:03:33] ...trap/engines/engine.py.230: loss: 0.12245 [I 2022-10-27 14:03:33] ...optimizers/trijoint.py.151: total_norm: 13.404753 [I 2022-10-27 14:03:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 280/2304 [I 2022-10-27 14:03:38] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:41 [I 2022-10-27 14:03:38] ...trap/engines/engine.py.229: process: 0.54910 | load: 0.00260 [I 2022-10-27 14:03:39] ...trap/engines/engine.py.230: loss: 0.16929 [I 2022-10-27 14:03:39] ...optimizers/trijoint.py.151: total_norm: 20.555410 [I 2022-10-27 14:03:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 290/2304 [I 2022-10-27 14:03:44] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:44 [I 2022-10-27 14:03:44] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00250 [I 2022-10-27 14:03:44] ...trap/engines/engine.py.230: loss: 0.10391 [I 2022-10-27 14:03:44] ...optimizers/trijoint.py.151: total_norm: 13.840513 [I 2022-10-27 14:03:50] ...trap/engines/engine.py.224: train: epoch 71 | batch 300/2304 [I 2022-10-27 14:03:50] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:32 [I 2022-10-27 14:03:50] ...trap/engines/engine.py.229: process: 0.55637 | load: 0.00250 [I 2022-10-27 14:03:50] ...trap/engines/engine.py.230: loss: 0.14472 [I 2022-10-27 14:03:50] ...optimizers/trijoint.py.151: total_norm: 13.092642 [I 2022-10-27 14:03:55] ...trap/engines/engine.py.224: train: epoch 71 | batch 310/2304 [I 2022-10-27 14:03:55] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:39 [I 2022-10-27 14:03:55] ...trap/engines/engine.py.229: process: 0.54946 | load: 0.00250 [I 2022-10-27 14:03:55] ...trap/engines/engine.py.230: loss: 0.23620 [I 2022-10-27 14:03:55] ...optimizers/trijoint.py.151: total_norm: 18.119505 [I 2022-10-27 14:04:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 320/2304 [I 2022-10-27 14:04:01] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:19 [I 2022-10-27 14:04:01] ...trap/engines/engine.py.229: process: 0.54683 | load: 0.00253 [I 2022-10-27 14:04:01] ...trap/engines/engine.py.230: loss: 0.12152 [I 2022-10-27 14:04:01] ...optimizers/trijoint.py.151: total_norm: 13.842795 [I 2022-10-27 14:04:06] ...trap/engines/engine.py.224: train: epoch 71 | batch 330/2304 [I 2022-10-27 14:04:06] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:17 [I 2022-10-27 14:04:06] ...trap/engines/engine.py.229: process: 0.56693 | load: 0.00247 [I 2022-10-27 14:04:06] ...trap/engines/engine.py.230: loss: 0.17796 [I 2022-10-27 14:04:06] ...optimizers/trijoint.py.151: total_norm: 23.369711 [I 2022-10-27 14:04:12] ...trap/engines/engine.py.224: train: epoch 71 | batch 340/2304 [I 2022-10-27 14:04:12] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:21 [I 2022-10-27 14:04:12] ...trap/engines/engine.py.229: process: 0.56450 | load: 0.00245 [I 2022-10-27 14:04:12] ...trap/engines/engine.py.230: loss: 0.10623 [I 2022-10-27 14:04:12] ...optimizers/trijoint.py.151: total_norm: 16.003582 [I 2022-10-27 14:04:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 350/2304 [I 2022-10-27 14:04:18] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:07 [I 2022-10-27 14:04:18] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00253 [I 2022-10-27 14:04:18] ...trap/engines/engine.py.230: loss: 0.18292 [I 2022-10-27 14:04:18] ...optimizers/trijoint.py.151: total_norm: 20.766275 [I 2022-10-27 14:04:23] ...trap/engines/engine.py.224: train: epoch 71 | batch 360/2304 [I 2022-10-27 14:04:23] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:05 [I 2022-10-27 14:04:23] ...trap/engines/engine.py.229: process: 0.56824 | load: 0.00246 [I 2022-10-27 14:04:23] ...trap/engines/engine.py.230: loss: 0.19777 [I 2022-10-27 14:04:23] ...optimizers/trijoint.py.151: total_norm: 36.592312 [I 2022-10-27 14:04:29] ...trap/engines/engine.py.224: train: epoch 71 | batch 370/2304 [I 2022-10-27 14:04:29] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:50 [I 2022-10-27 14:04:29] ...trap/engines/engine.py.229: process: 0.54224 | load: 0.00254 [I 2022-10-27 14:04:29] ...trap/engines/engine.py.230: loss: 0.10149 [I 2022-10-27 14:04:29] ...optimizers/trijoint.py.151: total_norm: 14.195893 [I 2022-10-27 14:04:34] ...trap/engines/engine.py.224: train: epoch 71 | batch 380/2304 [I 2022-10-27 14:04:34] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:50 [I 2022-10-27 14:04:34] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00253 [I 2022-10-27 14:04:34] ...trap/engines/engine.py.230: loss: 0.12772 [I 2022-10-27 14:04:34] ...optimizers/trijoint.py.151: total_norm: 20.295460 [I 2022-10-27 14:04:40] ...trap/engines/engine.py.224: train: epoch 71 | batch 390/2304 [I 2022-10-27 14:04:40] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:45 [I 2022-10-27 14:04:40] ...trap/engines/engine.py.229: process: 0.55110 | load: 0.00247 [I 2022-10-27 14:04:40] ...trap/engines/engine.py.230: loss: 0.13824 [I 2022-10-27 14:04:40] ...optimizers/trijoint.py.151: total_norm: 11.316405 [I 2022-10-27 14:04:46] ...trap/engines/engine.py.224: train: epoch 71 | batch 400/2304 [I 2022-10-27 14:04:46] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:46 [I 2022-10-27 14:04:46] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00245 [I 2022-10-27 14:04:46] ...trap/engines/engine.py.230: loss: 0.13265 [I 2022-10-27 14:04:46] ...optimizers/trijoint.py.151: total_norm: 17.892498 [I 2022-10-27 14:04:51] ...trap/engines/engine.py.224: train: epoch 71 | batch 410/2304 [I 2022-10-27 14:04:51] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:38 [I 2022-10-27 14:04:51] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00248 [I 2022-10-27 14:04:51] ...trap/engines/engine.py.230: loss: 0.15139 [I 2022-10-27 14:04:51] ...optimizers/trijoint.py.151: total_norm: 14.443180 [I 2022-10-27 14:04:57] ...trap/engines/engine.py.224: train: epoch 71 | batch 420/2304 [I 2022-10-27 14:04:57] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:34 [I 2022-10-27 14:04:57] ...trap/engines/engine.py.229: process: 0.56657 | load: 0.00245 [I 2022-10-27 14:04:57] ...trap/engines/engine.py.230: loss: 0.22270 [I 2022-10-27 14:04:57] ...optimizers/trijoint.py.151: total_norm: 13.150960 [I 2022-10-27 14:05:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 430/2304 [I 2022-10-27 14:05:02] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:22 [I 2022-10-27 14:05:02] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00250 [I 2022-10-27 14:05:02] ...trap/engines/engine.py.230: loss: 0.18736 [I 2022-10-27 14:05:02] ...optimizers/trijoint.py.151: total_norm: 14.686856 [I 2022-10-27 14:05:08] ...trap/engines/engine.py.224: train: epoch 71 | batch 440/2304 [I 2022-10-27 14:05:08] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:18 [I 2022-10-27 14:05:08] ...trap/engines/engine.py.229: process: 0.56338 | load: 0.00246 [I 2022-10-27 14:05:08] ...trap/engines/engine.py.230: loss: 0.19206 [I 2022-10-27 14:05:08] ...optimizers/trijoint.py.151: total_norm: 9.884585 [I 2022-10-27 14:05:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 450/2304 [I 2022-10-27 14:05:13] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:12 [I 2022-10-27 14:05:13] ...trap/engines/engine.py.229: process: 0.56198 | load: 0.00244 [I 2022-10-27 14:05:13] ...trap/engines/engine.py.230: loss: 0.15451 [I 2022-10-27 14:05:13] ...optimizers/trijoint.py.151: total_norm: 27.316500 [I 2022-10-27 14:05:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 460/2304 [I 2022-10-27 14:05:19] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:16:57 [I 2022-10-27 14:05:19] ...trap/engines/engine.py.229: process: 0.54901 | load: 0.00242 [I 2022-10-27 14:05:19] ...trap/engines/engine.py.230: loss: 0.14310 [I 2022-10-27 14:05:19] ...optimizers/trijoint.py.151: total_norm: 17.100302 [I 2022-10-27 14:05:25] ...trap/engines/engine.py.224: train: epoch 71 | batch 470/2304 [I 2022-10-27 14:05:25] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:03 [I 2022-10-27 14:05:25] ...trap/engines/engine.py.229: process: 0.56475 | load: 0.00246 [I 2022-10-27 14:05:25] ...trap/engines/engine.py.230: loss: 0.12164 [I 2022-10-27 14:05:25] ...optimizers/trijoint.py.151: total_norm: 20.917456 [I 2022-10-27 14:05:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 480/2304 [I 2022-10-27 14:05:30] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:47 [I 2022-10-27 14:05:30] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00246 [I 2022-10-27 14:05:30] ...trap/engines/engine.py.230: loss: 0.14021 [I 2022-10-27 14:05:30] ...optimizers/trijoint.py.151: total_norm: 30.509983 [I 2022-10-27 14:05:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 490/2304 [I 2022-10-27 14:05:36] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:43 [I 2022-10-27 14:05:36] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00245 [I 2022-10-27 14:05:36] ...trap/engines/engine.py.230: loss: 0.14213 [I 2022-10-27 14:05:36] ...optimizers/trijoint.py.151: total_norm: 11.574147 [I 2022-10-27 14:05:41] ...trap/engines/engine.py.224: train: epoch 71 | batch 500/2304 [I 2022-10-27 14:05:41] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:45 [I 2022-10-27 14:05:41] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00251 [I 2022-10-27 14:05:41] ...trap/engines/engine.py.230: loss: 0.12768 [I 2022-10-27 14:05:41] ...optimizers/trijoint.py.151: total_norm: 8.879692 [I 2022-10-27 14:05:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 510/2304 [I 2022-10-27 14:05:47] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:34 [I 2022-10-27 14:05:47] ...trap/engines/engine.py.229: process: 0.54467 | load: 0.00250 [I 2022-10-27 14:05:47] ...trap/engines/engine.py.230: loss: 0.11863 [I 2022-10-27 14:05:47] ...optimizers/trijoint.py.151: total_norm: 9.127974 [I 2022-10-27 14:05:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 520/2304 [I 2022-10-27 14:05:52] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:40 [I 2022-10-27 14:05:52] ...trap/engines/engine.py.229: process: 0.57359 | load: 0.00257 [I 2022-10-27 14:05:52] ...trap/engines/engine.py.230: loss: 0.13567 [I 2022-10-27 14:05:52] ...optimizers/trijoint.py.151: total_norm: 45.153202 [I 2022-10-27 14:05:58] ...trap/engines/engine.py.224: train: epoch 71 | batch 530/2304 [I 2022-10-27 14:05:58] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:33 [I 2022-10-27 14:05:58] ...trap/engines/engine.py.229: process: 0.55001 | load: 0.00247 [I 2022-10-27 14:05:58] ...trap/engines/engine.py.230: loss: 0.18950 [I 2022-10-27 14:05:58] ...optimizers/trijoint.py.151: total_norm: 16.568480 [I 2022-10-27 14:06:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 540/2304 [I 2022-10-27 14:06:04] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:22 [I 2022-10-27 14:06:04] ...trap/engines/engine.py.229: process: 0.54799 | load: 0.00245 [I 2022-10-27 14:06:04] ...trap/engines/engine.py.230: loss: 0.16357 [I 2022-10-27 14:06:04] ...optimizers/trijoint.py.151: total_norm: 29.521458 [I 2022-10-27 14:06:09] ...trap/engines/engine.py.224: train: epoch 71 | batch 550/2304 [I 2022-10-27 14:06:09] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:15 [I 2022-10-27 14:06:09] ...trap/engines/engine.py.229: process: 0.54935 | load: 0.00246 [I 2022-10-27 14:06:09] ...trap/engines/engine.py.230: loss: 0.17207 [I 2022-10-27 14:06:09] ...optimizers/trijoint.py.151: total_norm: 20.759386 [I 2022-10-27 14:06:15] ...trap/engines/engine.py.224: train: epoch 71 | batch 560/2304 [I 2022-10-27 14:06:15] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:30 [I 2022-10-27 14:06:15] ...trap/engines/engine.py.229: process: 0.56968 | load: 0.00244 [I 2022-10-27 14:06:15] ...trap/engines/engine.py.230: loss: 0.11846 [I 2022-10-27 14:06:15] ...optimizers/trijoint.py.151: total_norm: 10.575613 [I 2022-10-27 14:06:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 570/2304 [I 2022-10-27 14:06:21] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:09 [I 2022-10-27 14:06:21] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00243 [I 2022-10-27 14:06:21] ...trap/engines/engine.py.230: loss: 0.15426 [I 2022-10-27 14:06:21] ...optimizers/trijoint.py.151: total_norm: 14.997692 [I 2022-10-27 14:06:26] ...trap/engines/engine.py.224: train: epoch 71 | batch 580/2304 [I 2022-10-27 14:06:26] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:08 [I 2022-10-27 14:06:26] ...trap/engines/engine.py.229: process: 0.54735 | load: 0.00242 [I 2022-10-27 14:06:26] ...trap/engines/engine.py.230: loss: 0.22065 [I 2022-10-27 14:06:26] ...optimizers/trijoint.py.151: total_norm: 26.640139 [I 2022-10-27 14:06:32] ...trap/engines/engine.py.224: train: epoch 71 | batch 590/2304 [I 2022-10-27 14:06:32] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:57 [I 2022-10-27 14:06:32] ...trap/engines/engine.py.229: process: 0.55366 | load: 0.00241 [I 2022-10-27 14:06:32] ...trap/engines/engine.py.230: loss: 0.12389 [I 2022-10-27 14:06:32] ...optimizers/trijoint.py.151: total_norm: 12.971850 [I 2022-10-27 14:06:37] ...trap/engines/engine.py.224: train: epoch 71 | batch 600/2304 [I 2022-10-27 14:06:37] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:49 [I 2022-10-27 14:06:37] ...trap/engines/engine.py.229: process: 0.54482 | load: 0.00250 [I 2022-10-27 14:06:37] ...trap/engines/engine.py.230: loss: 0.16579 [I 2022-10-27 14:06:37] ...optimizers/trijoint.py.151: total_norm: 34.199883 [I 2022-10-27 14:06:43] ...trap/engines/engine.py.224: train: epoch 71 | batch 610/2304 [I 2022-10-27 14:06:43] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:46 [I 2022-10-27 14:06:43] ...trap/engines/engine.py.229: process: 0.56087 | load: 0.00247 [I 2022-10-27 14:06:43] ...trap/engines/engine.py.230: loss: 0.08773 [I 2022-10-27 14:06:43] ...optimizers/trijoint.py.151: total_norm: 9.387331 [I 2022-10-27 14:06:49] ...trap/engines/engine.py.224: train: epoch 71 | batch 620/2304 [I 2022-10-27 14:06:49] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:38 [I 2022-10-27 14:06:49] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00245 [I 2022-10-27 14:06:49] ...trap/engines/engine.py.230: loss: 0.10092 [I 2022-10-27 14:06:49] ...optimizers/trijoint.py.151: total_norm: 9.797571 [I 2022-10-27 14:06:54] ...trap/engines/engine.py.224: train: epoch 71 | batch 630/2304 [I 2022-10-27 14:06:54] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:32 [I 2022-10-27 14:06:54] ...trap/engines/engine.py.229: process: 0.55734 | load: 0.00246 [I 2022-10-27 14:06:54] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-27 14:06:54] ...optimizers/trijoint.py.151: total_norm: 23.418667 [I 2022-10-27 14:07:00] ...trap/engines/engine.py.224: train: epoch 71 | batch 640/2304 [I 2022-10-27 14:07:00] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:28 [I 2022-10-27 14:07:00] ...trap/engines/engine.py.229: process: 0.56086 | load: 0.00243 [I 2022-10-27 14:07:00] ...trap/engines/engine.py.230: loss: 0.16795 [I 2022-10-27 14:07:00] ...optimizers/trijoint.py.151: total_norm: 13.629338 [I 2022-10-27 14:07:05] ...trap/engines/engine.py.224: train: epoch 71 | batch 650/2304 [I 2022-10-27 14:07:05] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:37 [I 2022-10-27 14:07:05] ...trap/engines/engine.py.229: process: 0.56550 | load: 0.00242 [I 2022-10-27 14:07:05] ...trap/engines/engine.py.230: loss: 0.12120 [I 2022-10-27 14:07:05] ...optimizers/trijoint.py.151: total_norm: 10.285161 [I 2022-10-27 14:07:11] ...trap/engines/engine.py.224: train: epoch 71 | batch 660/2304 [I 2022-10-27 14:07:11] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:16 [I 2022-10-27 14:07:11] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00243 [I 2022-10-27 14:07:11] ...trap/engines/engine.py.230: loss: 0.11697 [I 2022-10-27 14:07:11] ...optimizers/trijoint.py.151: total_norm: 13.331287 [I 2022-10-27 14:07:17] ...trap/engines/engine.py.224: train: epoch 71 | batch 670/2304 [I 2022-10-27 14:07:17] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:10 [I 2022-10-27 14:07:17] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00246 [I 2022-10-27 14:07:17] ...trap/engines/engine.py.230: loss: 0.20433 [I 2022-10-27 14:07:17] ...optimizers/trijoint.py.151: total_norm: 18.134466 [I 2022-10-27 14:07:22] ...trap/engines/engine.py.224: train: epoch 71 | batch 680/2304 [I 2022-10-27 14:07:22] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:06 [I 2022-10-27 14:07:22] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00246 [I 2022-10-27 14:07:22] ...trap/engines/engine.py.230: loss: 0.09419 [I 2022-10-27 14:07:22] ...optimizers/trijoint.py.151: total_norm: 11.625157 [I 2022-10-27 14:07:28] ...trap/engines/engine.py.224: train: epoch 71 | batch 690/2304 [I 2022-10-27 14:07:28] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:03 [I 2022-10-27 14:07:28] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00248 [I 2022-10-27 14:07:28] ...trap/engines/engine.py.230: loss: 0.20280 [I 2022-10-27 14:07:28] ...optimizers/trijoint.py.151: total_norm: 22.131262 [I 2022-10-27 14:07:33] ...trap/engines/engine.py.224: train: epoch 71 | batch 700/2304 [I 2022-10-27 14:07:33] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:53 [I 2022-10-27 14:07:33] ...trap/engines/engine.py.229: process: 0.55384 | load: 0.00247 [I 2022-10-27 14:07:33] ...trap/engines/engine.py.230: loss: 0.13748 [I 2022-10-27 14:07:33] ...optimizers/trijoint.py.151: total_norm: 13.114072 [I 2022-10-27 14:07:39] ...trap/engines/engine.py.224: train: epoch 71 | batch 710/2304 [I 2022-10-27 14:07:39] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:54 [I 2022-10-27 14:07:39] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00248 [I 2022-10-27 14:07:39] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-27 14:07:39] ...optimizers/trijoint.py.151: total_norm: 13.588039 [I 2022-10-27 14:07:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 720/2304 [I 2022-10-27 14:07:45] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:50 [I 2022-10-27 14:07:45] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00241 [I 2022-10-27 14:07:45] ...trap/engines/engine.py.230: loss: 0.17151 [I 2022-10-27 14:07:45] ...optimizers/trijoint.py.151: total_norm: 11.152431 [I 2022-10-27 14:07:50] ...trap/engines/engine.py.224: train: epoch 71 | batch 730/2304 [I 2022-10-27 14:07:50] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:39 [I 2022-10-27 14:07:50] ...trap/engines/engine.py.229: process: 0.55902 | load: 0.00246 [I 2022-10-27 14:07:50] ...trap/engines/engine.py.230: loss: 0.16747 [I 2022-10-27 14:07:50] ...optimizers/trijoint.py.151: total_norm: 17.584324 [I 2022-10-27 14:07:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 740/2304 [I 2022-10-27 14:07:56] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:26 [I 2022-10-27 14:07:56] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00244 [I 2022-10-27 14:07:56] ...trap/engines/engine.py.230: loss: 0.12868 [I 2022-10-27 14:07:56] ...optimizers/trijoint.py.151: total_norm: 20.430576 [I 2022-10-27 14:08:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 750/2304 [I 2022-10-27 14:08:01] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:26 [I 2022-10-27 14:08:01] ...trap/engines/engine.py.229: process: 0.56682 | load: 0.00241 [I 2022-10-27 14:08:01] ...trap/engines/engine.py.230: loss: 0.15382 [I 2022-10-27 14:08:01] ...optimizers/trijoint.py.151: total_norm: 11.232422 [I 2022-10-27 14:08:07] ...trap/engines/engine.py.224: train: epoch 71 | batch 760/2304 [I 2022-10-27 14:08:07] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:21 [I 2022-10-27 14:08:07] ...trap/engines/engine.py.229: process: 0.55406 | load: 0.00241 [I 2022-10-27 14:08:07] ...trap/engines/engine.py.230: loss: 0.22611 [I 2022-10-27 14:08:07] ...optimizers/trijoint.py.151: total_norm: 23.394112 [I 2022-10-27 14:08:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 770/2304 [I 2022-10-27 14:08:13] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:14 [I 2022-10-27 14:08:13] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00244 [I 2022-10-27 14:08:13] ...trap/engines/engine.py.230: loss: 0.15169 [I 2022-10-27 14:08:13] ...optimizers/trijoint.py.151: total_norm: 14.880922 [I 2022-10-27 14:08:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 780/2304 [I 2022-10-27 14:08:18] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:17 [I 2022-10-27 14:08:18] ...trap/engines/engine.py.229: process: 0.55594 | load: 0.00241 [I 2022-10-27 14:08:18] ...trap/engines/engine.py.230: loss: 0.10219 [I 2022-10-27 14:08:18] ...optimizers/trijoint.py.151: total_norm: 15.601369 [I 2022-10-27 14:08:24] ...trap/engines/engine.py.224: train: epoch 71 | batch 790/2304 [I 2022-10-27 14:08:24] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:01 [I 2022-10-27 14:08:24] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00252 [I 2022-10-27 14:08:24] ...trap/engines/engine.py.230: loss: 0.13702 [I 2022-10-27 14:08:24] ...optimizers/trijoint.py.151: total_norm: 27.219009 [I 2022-10-27 14:08:29] ...trap/engines/engine.py.224: train: epoch 71 | batch 800/2304 [I 2022-10-27 14:08:29] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:53 [I 2022-10-27 14:08:29] ...trap/engines/engine.py.229: process: 0.54985 | load: 0.00248 [I 2022-10-27 14:08:29] ...trap/engines/engine.py.230: loss: 0.16576 [I 2022-10-27 14:08:29] ...optimizers/trijoint.py.151: total_norm: 18.504457 [I 2022-10-27 14:08:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 810/2304 [I 2022-10-27 14:08:35] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:52 [I 2022-10-27 14:08:35] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00244 [I 2022-10-27 14:08:35] ...trap/engines/engine.py.230: loss: 0.14522 [I 2022-10-27 14:08:35] ...optimizers/trijoint.py.151: total_norm: 8.828959 [I 2022-10-27 14:08:41] ...trap/engines/engine.py.224: train: epoch 71 | batch 820/2304 [I 2022-10-27 14:08:41] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:48 [I 2022-10-27 14:08:41] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00245 [I 2022-10-27 14:08:41] ...trap/engines/engine.py.230: loss: 0.15091 [I 2022-10-27 14:08:41] ...optimizers/trijoint.py.151: total_norm: 15.856261 [I 2022-10-27 14:08:46] ...trap/engines/engine.py.224: train: epoch 71 | batch 830/2304 [I 2022-10-27 14:08:46] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:41 [I 2022-10-27 14:08:46] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00241 [I 2022-10-27 14:08:46] ...trap/engines/engine.py.230: loss: 0.15901 [I 2022-10-27 14:08:46] ...optimizers/trijoint.py.151: total_norm: 41.636616 [I 2022-10-27 14:08:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 840/2304 [I 2022-10-27 14:08:52] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:51 [I 2022-10-27 14:08:52] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00249 [I 2022-10-27 14:08:52] ...trap/engines/engine.py.230: loss: 0.13079 [I 2022-10-27 14:08:52] ...optimizers/trijoint.py.151: total_norm: 20.821487 [I 2022-10-27 14:08:58] ...trap/engines/engine.py.224: train: epoch 71 | batch 850/2304 [I 2022-10-27 14:08:58] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:33 [I 2022-10-27 14:08:58] ...trap/engines/engine.py.229: process: 0.55547 | load: 0.00248 [I 2022-10-27 14:08:58] ...trap/engines/engine.py.230: loss: 0.20693 [I 2022-10-27 14:08:58] ...optimizers/trijoint.py.151: total_norm: 22.265274 [I 2022-10-27 14:09:03] ...trap/engines/engine.py.224: train: epoch 71 | batch 860/2304 [I 2022-10-27 14:09:03] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:29 [I 2022-10-27 14:09:03] ...trap/engines/engine.py.229: process: 0.54697 | load: 0.00245 [I 2022-10-27 14:09:03] ...trap/engines/engine.py.230: loss: 0.15480 [I 2022-10-27 14:09:03] ...optimizers/trijoint.py.151: total_norm: 11.960655 [I 2022-10-27 14:09:09] ...trap/engines/engine.py.224: train: epoch 71 | batch 870/2304 [I 2022-10-27 14:09:09] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:19 [I 2022-10-27 14:09:09] ...trap/engines/engine.py.229: process: 0.55411 | load: 0.00247 [I 2022-10-27 14:09:09] ...trap/engines/engine.py.230: loss: 0.13794 [I 2022-10-27 14:09:09] ...optimizers/trijoint.py.151: total_norm: 15.490067 [I 2022-10-27 14:09:14] ...trap/engines/engine.py.224: train: epoch 71 | batch 880/2304 [I 2022-10-27 14:09:14] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:20 [I 2022-10-27 14:09:14] ...trap/engines/engine.py.229: process: 0.55684 | load: 0.00247 [I 2022-10-27 14:09:14] ...trap/engines/engine.py.230: loss: 0.18585 [I 2022-10-27 14:09:14] ...optimizers/trijoint.py.151: total_norm: 33.466686 [I 2022-10-27 14:09:20] ...trap/engines/engine.py.224: train: epoch 71 | batch 890/2304 [I 2022-10-27 14:09:20] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:10 [I 2022-10-27 14:09:20] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00243 [I 2022-10-27 14:09:20] ...trap/engines/engine.py.230: loss: 0.09653 [I 2022-10-27 14:09:20] ...optimizers/trijoint.py.151: total_norm: 7.496614 [I 2022-10-27 14:09:26] ...trap/engines/engine.py.224: train: epoch 71 | batch 900/2304 [I 2022-10-27 14:09:26] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:34 [I 2022-10-27 14:09:26] ...trap/engines/engine.py.229: process: 0.54461 | load: 0.00247 [I 2022-10-27 14:09:26] ...trap/engines/engine.py.230: loss: 0.13966 [I 2022-10-27 14:09:26] ...optimizers/trijoint.py.151: total_norm: 11.756536 [I 2022-10-27 14:09:31] ...trap/engines/engine.py.224: train: epoch 71 | batch 910/2304 [I 2022-10-27 14:09:31] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:59 [I 2022-10-27 14:09:31] ...trap/engines/engine.py.229: process: 0.56048 | load: 0.00242 [I 2022-10-27 14:09:31] ...trap/engines/engine.py.230: loss: 0.09315 [I 2022-10-27 14:09:31] ...optimizers/trijoint.py.151: total_norm: 10.706734 [I 2022-10-27 14:09:37] ...trap/engines/engine.py.224: train: epoch 71 | batch 920/2304 [I 2022-10-27 14:09:37] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:53 [I 2022-10-27 14:09:37] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00241 [I 2022-10-27 14:09:37] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-27 14:09:37] ...optimizers/trijoint.py.151: total_norm: 13.120920 [I 2022-10-27 14:09:43] ...trap/engines/engine.py.224: train: epoch 71 | batch 930/2304 [I 2022-10-27 14:09:43] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:52 [I 2022-10-27 14:09:43] ...trap/engines/engine.py.229: process: 0.56297 | load: 0.00244 [I 2022-10-27 14:09:43] ...trap/engines/engine.py.230: loss: 0.14349 [I 2022-10-27 14:09:43] ...optimizers/trijoint.py.151: total_norm: 16.541733 [I 2022-10-27 14:09:48] ...trap/engines/engine.py.224: train: epoch 71 | batch 940/2304 [I 2022-10-27 14:09:48] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:39 [I 2022-10-27 14:09:48] ...trap/engines/engine.py.229: process: 0.54383 | load: 0.00251 [I 2022-10-27 14:09:48] ...trap/engines/engine.py.230: loss: 0.17037 [I 2022-10-27 14:09:48] ...optimizers/trijoint.py.151: total_norm: 28.506168 [I 2022-10-27 14:09:54] ...trap/engines/engine.py.224: train: epoch 71 | batch 950/2304 [I 2022-10-27 14:09:54] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:34 [I 2022-10-27 14:09:54] ...trap/engines/engine.py.229: process: 0.55830 | load: 0.00249 [I 2022-10-27 14:09:54] ...trap/engines/engine.py.230: loss: 0.12106 [I 2022-10-27 14:09:54] ...optimizers/trijoint.py.151: total_norm: 18.384554 [I 2022-10-27 14:09:59] ...trap/engines/engine.py.224: train: epoch 71 | batch 960/2304 [I 2022-10-27 14:09:59] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:29 [I 2022-10-27 14:09:59] ...trap/engines/engine.py.229: process: 0.56311 | load: 0.00246 [I 2022-10-27 14:09:59] ...trap/engines/engine.py.230: loss: 0.11430 [I 2022-10-27 14:09:59] ...optimizers/trijoint.py.151: total_norm: 9.394501 [I 2022-10-27 14:10:05] ...trap/engines/engine.py.224: train: epoch 71 | batch 970/2304 [I 2022-10-27 14:10:05] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:29 [I 2022-10-27 14:10:05] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00244 [I 2022-10-27 14:10:05] ...trap/engines/engine.py.230: loss: 0.09112 [I 2022-10-27 14:10:05] ...optimizers/trijoint.py.151: total_norm: 9.959721 [I 2022-10-27 14:10:11] ...trap/engines/engine.py.224: train: epoch 71 | batch 980/2304 [I 2022-10-27 14:10:11] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:16 [I 2022-10-27 14:10:11] ...trap/engines/engine.py.229: process: 0.54920 | load: 0.00247 [I 2022-10-27 14:10:11] ...trap/engines/engine.py.230: loss: 0.10070 [I 2022-10-27 14:10:11] ...optimizers/trijoint.py.151: total_norm: 9.652861 [I 2022-10-27 14:10:16] ...trap/engines/engine.py.224: train: epoch 71 | batch 990/2304 [I 2022-10-27 14:10:16] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:16 [I 2022-10-27 14:10:16] ...trap/engines/engine.py.229: process: 0.56711 | load: 0.00241 [I 2022-10-27 14:10:16] ...trap/engines/engine.py.230: loss: 0.19234 [I 2022-10-27 14:10:16] ...optimizers/trijoint.py.151: total_norm: 26.948908 [I 2022-10-27 14:10:22] ...trap/engines/engine.py.224: train: epoch 71 | batch 1000/2304 [I 2022-10-27 14:10:22] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:11 [I 2022-10-27 14:10:22] ...trap/engines/engine.py.229: process: 0.55126 | load: 0.00264 [I 2022-10-27 14:10:22] ...trap/engines/engine.py.230: loss: 0.16427 [I 2022-10-27 14:10:22] ...optimizers/trijoint.py.151: total_norm: 70.929672 [I 2022-10-27 14:10:28] ...trap/engines/engine.py.224: train: epoch 71 | batch 1010/2304 [I 2022-10-27 14:10:28] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:08 [I 2022-10-27 14:10:28] ...trap/engines/engine.py.229: process: 0.54938 | load: 0.00249 [I 2022-10-27 14:10:28] ...trap/engines/engine.py.230: loss: 0.12560 [I 2022-10-27 14:10:28] ...optimizers/trijoint.py.151: total_norm: 16.967653 [I 2022-10-27 14:10:33] ...trap/engines/engine.py.224: train: epoch 71 | batch 1020/2304 [I 2022-10-27 14:10:33] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:59 [I 2022-10-27 14:10:33] ...trap/engines/engine.py.229: process: 0.56633 | load: 0.00246 [I 2022-10-27 14:10:33] ...trap/engines/engine.py.230: loss: 0.11860 [I 2022-10-27 14:10:33] ...optimizers/trijoint.py.151: total_norm: 9.233936 [I 2022-10-27 14:10:39] ...trap/engines/engine.py.224: train: epoch 71 | batch 1030/2304 [I 2022-10-27 14:10:39] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:12:21 [I 2022-10-27 14:10:39] ...trap/engines/engine.py.229: process: 0.67287 | load: 0.00246 [I 2022-10-27 14:10:39] ...trap/engines/engine.py.230: loss: 0.15396 [I 2022-10-27 14:10:39] ...optimizers/trijoint.py.151: total_norm: 18.514734 [I 2022-10-27 14:10:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 1040/2304 [I 2022-10-27 14:10:44] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:50 [I 2022-10-27 14:10:44] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00246 [I 2022-10-27 14:10:44] ...trap/engines/engine.py.230: loss: 0.18414 [I 2022-10-27 14:10:44] ...optimizers/trijoint.py.151: total_norm: 11.265678 [I 2022-10-27 14:10:50] ...trap/engines/engine.py.224: train: epoch 71 | batch 1050/2304 [I 2022-10-27 14:10:50] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:42 [I 2022-10-27 14:10:50] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00243 [I 2022-10-27 14:10:50] ...trap/engines/engine.py.230: loss: 0.12042 [I 2022-10-27 14:10:50] ...optimizers/trijoint.py.151: total_norm: 10.962334 [I 2022-10-27 14:10:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 1060/2304 [I 2022-10-27 14:10:56] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:35 [I 2022-10-27 14:10:56] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00243 [I 2022-10-27 14:10:56] ...trap/engines/engine.py.230: loss: 0.13185 [I 2022-10-27 14:10:56] ...optimizers/trijoint.py.151: total_norm: 11.141768 [I 2022-10-27 14:11:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 1070/2304 [I 2022-10-27 14:11:01] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:30 [I 2022-10-27 14:11:01] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00513 [I 2022-10-27 14:11:01] ...trap/engines/engine.py.230: loss: 0.12819 [I 2022-10-27 14:11:01] ...optimizers/trijoint.py.151: total_norm: 29.170433 [I 2022-10-27 14:11:07] ...trap/engines/engine.py.224: train: epoch 71 | batch 1080/2304 [I 2022-10-27 14:11:07] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:25 [I 2022-10-27 14:11:07] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00243 [I 2022-10-27 14:11:07] ...trap/engines/engine.py.230: loss: 0.14724 [I 2022-10-27 14:11:07] ...optimizers/trijoint.py.151: total_norm: 10.911271 [I 2022-10-27 14:11:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1090/2304 [I 2022-10-27 14:11:13] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:19 [I 2022-10-27 14:11:13] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00249 [I 2022-10-27 14:11:13] ...trap/engines/engine.py.230: loss: 0.20966 [I 2022-10-27 14:11:13] ...optimizers/trijoint.py.151: total_norm: 13.841228 [I 2022-10-27 14:11:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 1100/2304 [I 2022-10-27 14:11:18] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:15 [I 2022-10-27 14:11:18] ...trap/engines/engine.py.229: process: 0.54284 | load: 0.00248 [I 2022-10-27 14:11:18] ...trap/engines/engine.py.230: loss: 0.17887 [I 2022-10-27 14:11:18] ...optimizers/trijoint.py.151: total_norm: 18.834774 [I 2022-10-27 14:11:24] ...trap/engines/engine.py.224: train: epoch 71 | batch 1110/2304 [I 2022-10-27 14:11:24] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:07 [I 2022-10-27 14:11:24] ...trap/engines/engine.py.229: process: 0.54552 | load: 0.00261 [I 2022-10-27 14:11:24] ...trap/engines/engine.py.230: loss: 0.13318 [I 2022-10-27 14:11:24] ...optimizers/trijoint.py.151: total_norm: 25.720068 [I 2022-10-27 14:11:29] ...trap/engines/engine.py.224: train: epoch 71 | batch 1120/2304 [I 2022-10-27 14:11:29] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:01 [I 2022-10-27 14:11:29] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00256 [I 2022-10-27 14:11:29] ...trap/engines/engine.py.230: loss: 0.11366 [I 2022-10-27 14:11:29] ...optimizers/trijoint.py.151: total_norm: 8.145009 [I 2022-10-27 14:11:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 1130/2304 [I 2022-10-27 14:11:35] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:56 [I 2022-10-27 14:11:35] ...trap/engines/engine.py.229: process: 0.56477 | load: 0.00252 [I 2022-10-27 14:11:35] ...trap/engines/engine.py.230: loss: 0.23311 [I 2022-10-27 14:11:35] ...optimizers/trijoint.py.151: total_norm: 65.589333 [I 2022-10-27 14:11:41] ...trap/engines/engine.py.224: train: epoch 71 | batch 1140/2304 [I 2022-10-27 14:11:41] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:10:46 [I 2022-10-27 14:11:41] ...trap/engines/engine.py.229: process: 0.55067 | load: 0.00262 [I 2022-10-27 14:11:41] ...trap/engines/engine.py.230: loss: 0.13922 [I 2022-10-27 14:11:41] ...optimizers/trijoint.py.151: total_norm: 17.553694 [I 2022-10-27 14:11:46] ...trap/engines/engine.py.224: train: epoch 71 | batch 1150/2304 [I 2022-10-27 14:11:46] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:47 [I 2022-10-27 14:11:46] ...trap/engines/engine.py.229: process: 0.56829 | load: 0.00251 [I 2022-10-27 14:11:46] ...trap/engines/engine.py.230: loss: 0.15941 [I 2022-10-27 14:11:46] ...optimizers/trijoint.py.151: total_norm: 26.705423 [I 2022-10-27 14:11:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 1160/2304 [I 2022-10-27 14:11:52] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:36 [I 2022-10-27 14:11:52] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00249 [I 2022-10-27 14:11:52] ...trap/engines/engine.py.230: loss: 0.12176 [I 2022-10-27 14:11:52] ...optimizers/trijoint.py.151: total_norm: 12.302699 [I 2022-10-27 14:11:57] ...trap/engines/engine.py.224: train: epoch 71 | batch 1170/2304 [I 2022-10-27 14:11:57] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:31 [I 2022-10-27 14:11:57] ...trap/engines/engine.py.229: process: 0.56452 | load: 0.00271 [I 2022-10-27 14:11:57] ...trap/engines/engine.py.230: loss: 0.14815 [I 2022-10-27 14:11:57] ...optimizers/trijoint.py.151: total_norm: 21.736652 [I 2022-10-27 14:12:03] ...trap/engines/engine.py.224: train: epoch 71 | batch 1180/2304 [I 2022-10-27 14:12:03] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:25 [I 2022-10-27 14:12:03] ...trap/engines/engine.py.229: process: 0.56372 | load: 0.00243 [I 2022-10-27 14:12:03] ...trap/engines/engine.py.230: loss: 0.08396 [I 2022-10-27 14:12:03] ...optimizers/trijoint.py.151: total_norm: 11.063345 [I 2022-10-27 14:12:09] ...trap/engines/engine.py.224: train: epoch 71 | batch 1190/2304 [I 2022-10-27 14:12:09] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:16 [I 2022-10-27 14:12:09] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00241 [I 2022-10-27 14:12:09] ...trap/engines/engine.py.230: loss: 0.14329 [I 2022-10-27 14:12:09] ...optimizers/trijoint.py.151: total_norm: 24.931202 [I 2022-10-27 14:12:14] ...trap/engines/engine.py.224: train: epoch 71 | batch 1200/2304 [I 2022-10-27 14:12:14] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:18 [I 2022-10-27 14:12:14] ...trap/engines/engine.py.229: process: 0.56933 | load: 0.00250 [I 2022-10-27 14:12:14] ...trap/engines/engine.py.230: loss: 0.15728 [I 2022-10-27 14:12:14] ...optimizers/trijoint.py.151: total_norm: 28.786512 [I 2022-10-27 14:12:20] ...trap/engines/engine.py.224: train: epoch 71 | batch 1210/2304 [I 2022-10-27 14:12:20] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:08 [I 2022-10-27 14:12:20] ...trap/engines/engine.py.229: process: 0.54365 | load: 0.00253 [I 2022-10-27 14:12:20] ...trap/engines/engine.py.230: loss: 0.18985 [I 2022-10-27 14:12:20] ...optimizers/trijoint.py.151: total_norm: 8.614495 [I 2022-10-27 14:12:25] ...trap/engines/engine.py.224: train: epoch 71 | batch 1220/2304 [I 2022-10-27 14:12:25] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:01 [I 2022-10-27 14:12:25] ...trap/engines/engine.py.229: process: 0.55288 | load: 0.00254 [I 2022-10-27 14:12:25] ...trap/engines/engine.py.230: loss: 0.14006 [I 2022-10-27 14:12:25] ...optimizers/trijoint.py.151: total_norm: 16.955719 [I 2022-10-27 14:12:31] ...trap/engines/engine.py.224: train: epoch 71 | batch 1230/2304 [I 2022-10-27 14:12:31] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:09:58 [I 2022-10-27 14:12:31] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00246 [I 2022-10-27 14:12:31] ...trap/engines/engine.py.230: loss: 0.20359 [I 2022-10-27 14:12:31] ...optimizers/trijoint.py.151: total_norm: 31.415899 [I 2022-10-27 14:12:36] ...trap/engines/engine.py.224: train: epoch 71 | batch 1240/2304 [I 2022-10-27 14:12:36] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:54 [I 2022-10-27 14:12:36] ...trap/engines/engine.py.229: process: 0.56897 | load: 0.00245 [I 2022-10-27 14:12:36] ...trap/engines/engine.py.230: loss: 0.15713 [I 2022-10-27 14:12:36] ...optimizers/trijoint.py.151: total_norm: 33.501251 [I 2022-10-27 14:12:42] ...trap/engines/engine.py.224: train: epoch 71 | batch 1250/2304 [I 2022-10-27 14:12:42] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:43 [I 2022-10-27 14:12:42] ...trap/engines/engine.py.229: process: 0.55016 | load: 0.00244 [I 2022-10-27 14:12:42] ...trap/engines/engine.py.230: loss: 0.19269 [I 2022-10-27 14:12:42] ...optimizers/trijoint.py.151: total_norm: 13.659845 [I 2022-10-27 14:12:48] ...trap/engines/engine.py.224: train: epoch 71 | batch 1260/2304 [I 2022-10-27 14:12:48] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:44 [I 2022-10-27 14:12:48] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00245 [I 2022-10-27 14:12:48] ...trap/engines/engine.py.230: loss: 0.15516 [I 2022-10-27 14:12:48] ...optimizers/trijoint.py.151: total_norm: 16.339539 [I 2022-10-27 14:12:53] ...trap/engines/engine.py.224: train: epoch 71 | batch 1270/2304 [I 2022-10-27 14:12:53] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:35 [I 2022-10-27 14:12:53] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00249 [I 2022-10-27 14:12:53] ...trap/engines/engine.py.230: loss: 0.20729 [I 2022-10-27 14:12:53] ...optimizers/trijoint.py.151: total_norm: 22.098261 [I 2022-10-27 14:12:59] ...trap/engines/engine.py.224: train: epoch 71 | batch 1280/2304 [I 2022-10-27 14:12:59] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:31 [I 2022-10-27 14:12:59] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00252 [I 2022-10-27 14:12:59] ...trap/engines/engine.py.230: loss: 0.13825 [I 2022-10-27 14:12:59] ...optimizers/trijoint.py.151: total_norm: 24.387617 [I 2022-10-27 14:13:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 1290/2304 [I 2022-10-27 14:13:04] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:25 [I 2022-10-27 14:13:04] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00242 [I 2022-10-27 14:13:04] ...trap/engines/engine.py.230: loss: 0.12758 [I 2022-10-27 14:13:04] ...optimizers/trijoint.py.151: total_norm: 9.069410 [I 2022-10-27 14:13:10] ...trap/engines/engine.py.224: train: epoch 71 | batch 1300/2304 [I 2022-10-27 14:13:10] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:22 [I 2022-10-27 14:13:10] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00241 [I 2022-10-27 14:13:10] ...trap/engines/engine.py.230: loss: 0.11760 [I 2022-10-27 14:13:10] ...optimizers/trijoint.py.151: total_norm: 10.792784 [I 2022-10-27 14:13:16] ...trap/engines/engine.py.224: train: epoch 71 | batch 1310/2304 [I 2022-10-27 14:13:16] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:13 [I 2022-10-27 14:13:16] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00243 [I 2022-10-27 14:13:16] ...trap/engines/engine.py.230: loss: 0.12240 [I 2022-10-27 14:13:16] ...optimizers/trijoint.py.151: total_norm: 12.826210 [I 2022-10-27 14:13:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1320/2304 [I 2022-10-27 14:13:21] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:05 [I 2022-10-27 14:13:21] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00242 [I 2022-10-27 14:13:21] ...trap/engines/engine.py.230: loss: 0.20482 [I 2022-10-27 14:13:21] ...optimizers/trijoint.py.151: total_norm: 41.048435 [I 2022-10-27 14:13:27] ...trap/engines/engine.py.224: train: epoch 71 | batch 1330/2304 [I 2022-10-27 14:13:27] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:00 [I 2022-10-27 14:13:27] ...trap/engines/engine.py.229: process: 0.54551 | load: 0.00244 [I 2022-10-27 14:13:27] ...trap/engines/engine.py.230: loss: 0.15321 [I 2022-10-27 14:13:27] ...optimizers/trijoint.py.151: total_norm: 10.475178 [I 2022-10-27 14:13:32] ...trap/engines/engine.py.224: train: epoch 71 | batch 1340/2304 [I 2022-10-27 14:13:32] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:08:59 [I 2022-10-27 14:13:32] ...trap/engines/engine.py.229: process: 0.56301 | load: 0.00257 [I 2022-10-27 14:13:32] ...trap/engines/engine.py.230: loss: 0.11048 [I 2022-10-27 14:13:32] ...optimizers/trijoint.py.151: total_norm: 17.040672 [I 2022-10-27 14:13:38] ...trap/engines/engine.py.224: train: epoch 71 | batch 1350/2304 [I 2022-10-27 14:13:38] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:51 [I 2022-10-27 14:13:38] ...trap/engines/engine.py.229: process: 0.55076 | load: 0.00250 [I 2022-10-27 14:13:38] ...trap/engines/engine.py.230: loss: 0.13159 [I 2022-10-27 14:13:38] ...optimizers/trijoint.py.151: total_norm: 21.270540 [I 2022-10-27 14:13:43] ...trap/engines/engine.py.224: train: epoch 71 | batch 1360/2304 [I 2022-10-27 14:13:43] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:43 [I 2022-10-27 14:13:43] ...trap/engines/engine.py.229: process: 0.55734 | load: 0.00242 [I 2022-10-27 14:13:43] ...trap/engines/engine.py.230: loss: 0.13753 [I 2022-10-27 14:13:43] ...optimizers/trijoint.py.151: total_norm: 13.159738 [I 2022-10-27 14:13:49] ...trap/engines/engine.py.224: train: epoch 71 | batch 1370/2304 [I 2022-10-27 14:13:49] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:41 [I 2022-10-27 14:13:49] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00244 [I 2022-10-27 14:13:49] ...trap/engines/engine.py.230: loss: 0.19734 [I 2022-10-27 14:13:49] ...optimizers/trijoint.py.151: total_norm: 21.494848 [I 2022-10-27 14:13:55] ...trap/engines/engine.py.224: train: epoch 71 | batch 1380/2304 [I 2022-10-27 14:13:55] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:37 [I 2022-10-27 14:13:55] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00248 [I 2022-10-27 14:13:55] ...trap/engines/engine.py.230: loss: 0.12913 [I 2022-10-27 14:13:55] ...optimizers/trijoint.py.151: total_norm: 16.455563 [I 2022-10-27 14:14:00] ...trap/engines/engine.py.224: train: epoch 71 | batch 1390/2304 [I 2022-10-27 14:14:00] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:28 [I 2022-10-27 14:14:00] ...trap/engines/engine.py.229: process: 0.54876 | load: 0.00244 [I 2022-10-27 14:14:00] ...trap/engines/engine.py.230: loss: 0.17293 [I 2022-10-27 14:14:00] ...optimizers/trijoint.py.151: total_norm: 13.283531 [I 2022-10-27 14:14:06] ...trap/engines/engine.py.224: train: epoch 71 | batch 1400/2304 [I 2022-10-27 14:14:06] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:25 [I 2022-10-27 14:14:06] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00246 [I 2022-10-27 14:14:06] ...trap/engines/engine.py.230: loss: 0.11402 [I 2022-10-27 14:14:06] ...optimizers/trijoint.py.151: total_norm: 14.158762 [I 2022-10-27 14:14:11] ...trap/engines/engine.py.224: train: epoch 71 | batch 1410/2304 [I 2022-10-27 14:14:11] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:19 [I 2022-10-27 14:14:11] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00250 [I 2022-10-27 14:14:11] ...trap/engines/engine.py.230: loss: 0.12363 [I 2022-10-27 14:14:11] ...optimizers/trijoint.py.151: total_norm: 13.299791 [I 2022-10-27 14:14:17] ...trap/engines/engine.py.224: train: epoch 71 | batch 1420/2304 [I 2022-10-27 14:14:17] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:17 [I 2022-10-27 14:14:17] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00248 [I 2022-10-27 14:14:17] ...trap/engines/engine.py.230: loss: 0.14863 [I 2022-10-27 14:14:17] ...optimizers/trijoint.py.151: total_norm: 16.339022 [I 2022-10-27 14:14:23] ...trap/engines/engine.py.224: train: epoch 71 | batch 1430/2304 [I 2022-10-27 14:14:23] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:10 [I 2022-10-27 14:14:23] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00251 [I 2022-10-27 14:14:23] ...trap/engines/engine.py.230: loss: 0.13056 [I 2022-10-27 14:14:23] ...optimizers/trijoint.py.151: total_norm: 17.963234 [I 2022-10-27 14:14:28] ...trap/engines/engine.py.224: train: epoch 71 | batch 1440/2304 [I 2022-10-27 14:14:28] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:08:06 [I 2022-10-27 14:14:28] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00246 [I 2022-10-27 14:14:28] ...trap/engines/engine.py.230: loss: 0.15853 [I 2022-10-27 14:14:28] ...optimizers/trijoint.py.151: total_norm: 16.865564 [I 2022-10-27 14:14:34] ...trap/engines/engine.py.224: train: epoch 71 | batch 1450/2304 [I 2022-10-27 14:14:34] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:08:00 [I 2022-10-27 14:14:34] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00244 [I 2022-10-27 14:14:34] ...trap/engines/engine.py.230: loss: 0.10481 [I 2022-10-27 14:14:34] ...optimizers/trijoint.py.151: total_norm: 12.283932 [I 2022-10-27 14:14:40] ...trap/engines/engine.py.224: train: epoch 71 | batch 1460/2304 [I 2022-10-27 14:14:40] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:54 [I 2022-10-27 14:14:40] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00249 [I 2022-10-27 14:14:40] ...trap/engines/engine.py.230: loss: 0.14384 [I 2022-10-27 14:14:40] ...optimizers/trijoint.py.151: total_norm: 17.976665 [I 2022-10-27 14:14:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 1470/2304 [I 2022-10-27 14:14:45] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:45 [I 2022-10-27 14:14:45] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00245 [I 2022-10-27 14:14:45] ...trap/engines/engine.py.230: loss: 0.14450 [I 2022-10-27 14:14:45] ...optimizers/trijoint.py.151: total_norm: 20.891804 [I 2022-10-27 14:14:51] ...trap/engines/engine.py.224: train: epoch 71 | batch 1480/2304 [I 2022-10-27 14:14:51] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:43 [I 2022-10-27 14:14:51] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00244 [I 2022-10-27 14:14:51] ...trap/engines/engine.py.230: loss: 0.10590 [I 2022-10-27 14:14:51] ...optimizers/trijoint.py.151: total_norm: 11.871497 [I 2022-10-27 14:14:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 1490/2304 [I 2022-10-27 14:14:56] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:37 [I 2022-10-27 14:14:56] ...trap/engines/engine.py.229: process: 0.57043 | load: 0.00262 [I 2022-10-27 14:14:56] ...trap/engines/engine.py.230: loss: 0.12707 [I 2022-10-27 14:14:56] ...optimizers/trijoint.py.151: total_norm: 16.939703 [I 2022-10-27 14:15:02] ...trap/engines/engine.py.224: train: epoch 71 | batch 1500/2304 [I 2022-10-27 14:15:02] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:26 [I 2022-10-27 14:15:02] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00252 [I 2022-10-27 14:15:02] ...trap/engines/engine.py.230: loss: 0.15684 [I 2022-10-27 14:15:02] ...optimizers/trijoint.py.151: total_norm: 15.134989 [I 2022-10-27 14:15:08] ...trap/engines/engine.py.224: train: epoch 71 | batch 1510/2304 [I 2022-10-27 14:15:08] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:21 [I 2022-10-27 14:15:08] ...trap/engines/engine.py.229: process: 0.55534 | load: 0.00254 [I 2022-10-27 14:15:08] ...trap/engines/engine.py.230: loss: 0.12061 [I 2022-10-27 14:15:08] ...optimizers/trijoint.py.151: total_norm: 13.287562 [I 2022-10-27 14:15:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1520/2304 [I 2022-10-27 14:15:13] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:14 [I 2022-10-27 14:15:13] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00243 [I 2022-10-27 14:15:13] ...trap/engines/engine.py.230: loss: 0.20896 [I 2022-10-27 14:15:13] ...optimizers/trijoint.py.151: total_norm: 11.414039 [I 2022-10-27 14:15:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 1530/2304 [I 2022-10-27 14:15:19] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:09 [I 2022-10-27 14:15:19] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00244 [I 2022-10-27 14:15:19] ...trap/engines/engine.py.230: loss: 0.11554 [I 2022-10-27 14:15:19] ...optimizers/trijoint.py.151: total_norm: 14.875670 [I 2022-10-27 14:15:24] ...trap/engines/engine.py.224: train: epoch 71 | batch 1540/2304 [I 2022-10-27 14:15:24] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:00 [I 2022-10-27 14:15:24] ...trap/engines/engine.py.229: process: 0.54433 | load: 0.00242 [I 2022-10-27 14:15:24] ...trap/engines/engine.py.230: loss: 0.21870 [I 2022-10-27 14:15:24] ...optimizers/trijoint.py.151: total_norm: 18.087130 [I 2022-10-27 14:15:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 1550/2304 [I 2022-10-27 14:15:30] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:06:56 [I 2022-10-27 14:15:30] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00246 [I 2022-10-27 14:15:30] ...trap/engines/engine.py.230: loss: 0.12656 [I 2022-10-27 14:15:30] ...optimizers/trijoint.py.151: total_norm: 26.371790 [I 2022-10-27 14:15:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 1560/2304 [I 2022-10-27 14:15:35] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:54 [I 2022-10-27 14:15:35] ...trap/engines/engine.py.229: process: 0.55472 | load: 0.00244 [I 2022-10-27 14:15:35] ...trap/engines/engine.py.230: loss: 0.18478 [I 2022-10-27 14:15:35] ...optimizers/trijoint.py.151: total_norm: 20.005104 [I 2022-10-27 14:15:41] ...trap/engines/engine.py.224: train: epoch 71 | batch 1570/2304 [I 2022-10-27 14:15:41] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:50 [I 2022-10-27 14:15:41] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00251 [I 2022-10-27 14:15:41] ...trap/engines/engine.py.230: loss: 0.14051 [I 2022-10-27 14:15:41] ...optimizers/trijoint.py.151: total_norm: 13.721971 [I 2022-10-27 14:15:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 1580/2304 [I 2022-10-27 14:15:47] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:43 [I 2022-10-27 14:15:47] ...trap/engines/engine.py.229: process: 0.55234 | load: 0.00248 [I 2022-10-27 14:15:47] ...trap/engines/engine.py.230: loss: 0.13341 [I 2022-10-27 14:15:47] ...optimizers/trijoint.py.151: total_norm: 19.390858 [I 2022-10-27 14:15:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 1590/2304 [I 2022-10-27 14:15:52] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:41 [I 2022-10-27 14:15:52] ...trap/engines/engine.py.229: process: 0.56184 | load: 0.00248 [I 2022-10-27 14:15:52] ...trap/engines/engine.py.230: loss: 0.12422 [I 2022-10-27 14:15:52] ...optimizers/trijoint.py.151: total_norm: 15.297667 [I 2022-10-27 14:15:58] ...trap/engines/engine.py.224: train: epoch 71 | batch 1600/2304 [I 2022-10-27 14:15:58] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:36 [I 2022-10-27 14:15:58] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00243 [I 2022-10-27 14:15:58] ...trap/engines/engine.py.230: loss: 0.10820 [I 2022-10-27 14:15:58] ...optimizers/trijoint.py.151: total_norm: 20.987946 [I 2022-10-27 14:16:04] ...trap/engines/engine.py.224: train: epoch 71 | batch 1610/2304 [I 2022-10-27 14:16:04] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:28 [I 2022-10-27 14:16:04] ...trap/engines/engine.py.229: process: 0.54406 | load: 0.00248 [I 2022-10-27 14:16:04] ...trap/engines/engine.py.230: loss: 0.16213 [I 2022-10-27 14:16:04] ...optimizers/trijoint.py.151: total_norm: 23.566093 [I 2022-10-27 14:16:09] ...trap/engines/engine.py.224: train: epoch 71 | batch 1620/2304 [I 2022-10-27 14:16:09] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:28 [I 2022-10-27 14:16:09] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00241 [I 2022-10-27 14:16:09] ...trap/engines/engine.py.230: loss: 0.10995 [I 2022-10-27 14:16:09] ...optimizers/trijoint.py.151: total_norm: 9.208596 [I 2022-10-27 14:16:15] ...trap/engines/engine.py.224: train: epoch 71 | batch 1630/2304 [I 2022-10-27 14:16:15] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:20 [I 2022-10-27 14:16:15] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00245 [I 2022-10-27 14:16:15] ...trap/engines/engine.py.230: loss: 0.10707 [I 2022-10-27 14:16:15] ...optimizers/trijoint.py.151: total_norm: 11.151394 [I 2022-10-27 14:16:21] ...trap/engines/engine.py.224: train: epoch 71 | batch 1640/2304 [I 2022-10-27 14:16:21] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:23 [I 2022-10-27 14:16:21] ...trap/engines/engine.py.229: process: 0.55471 | load: 0.00246 [I 2022-10-27 14:16:21] ...trap/engines/engine.py.230: loss: 0.13580 [I 2022-10-27 14:16:21] ...optimizers/trijoint.py.151: total_norm: 9.261297 [I 2022-10-27 14:16:26] ...trap/engines/engine.py.224: train: epoch 71 | batch 1650/2304 [I 2022-10-27 14:16:26] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:05 [I 2022-10-27 14:16:26] ...trap/engines/engine.py.229: process: 0.56639 | load: 0.00269 [I 2022-10-27 14:16:26] ...trap/engines/engine.py.230: loss: 0.14911 [I 2022-10-27 14:16:26] ...optimizers/trijoint.py.151: total_norm: 30.076862 [I 2022-10-27 14:16:32] ...trap/engines/engine.py.224: train: epoch 71 | batch 1660/2304 [I 2022-10-27 14:16:32] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:05:58 [I 2022-10-27 14:16:32] ...trap/engines/engine.py.229: process: 0.55104 | load: 0.00254 [I 2022-10-27 14:16:32] ...trap/engines/engine.py.230: loss: 0.16741 [I 2022-10-27 14:16:32] ...optimizers/trijoint.py.151: total_norm: 19.467852 [I 2022-10-27 14:16:37] ...trap/engines/engine.py.224: train: epoch 71 | batch 1670/2304 [I 2022-10-27 14:16:37] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:53 [I 2022-10-27 14:16:37] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00247 [I 2022-10-27 14:16:37] ...trap/engines/engine.py.230: loss: 0.15199 [I 2022-10-27 14:16:37] ...optimizers/trijoint.py.151: total_norm: 20.314425 [I 2022-10-27 14:16:43] ...trap/engines/engine.py.224: train: epoch 71 | batch 1680/2304 [I 2022-10-27 14:16:43] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:46 [I 2022-10-27 14:16:43] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00244 [I 2022-10-27 14:16:43] ...trap/engines/engine.py.230: loss: 0.19262 [I 2022-10-27 14:16:43] ...optimizers/trijoint.py.151: total_norm: 17.879230 [I 2022-10-27 14:16:49] ...trap/engines/engine.py.224: train: epoch 71 | batch 1690/2304 [I 2022-10-27 14:16:49] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:40 [I 2022-10-27 14:16:49] ...trap/engines/engine.py.229: process: 0.54684 | load: 0.00246 [I 2022-10-27 14:16:49] ...trap/engines/engine.py.230: loss: 0.14014 [I 2022-10-27 14:16:49] ...optimizers/trijoint.py.151: total_norm: 17.898739 [I 2022-10-27 14:16:54] ...trap/engines/engine.py.224: train: epoch 71 | batch 1700/2304 [I 2022-10-27 14:16:54] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:36 [I 2022-10-27 14:16:54] ...trap/engines/engine.py.229: process: 0.54466 | load: 0.00243 [I 2022-10-27 14:16:54] ...trap/engines/engine.py.230: loss: 0.14205 [I 2022-10-27 14:16:54] ...optimizers/trijoint.py.151: total_norm: 21.041882 [I 2022-10-27 14:17:00] ...trap/engines/engine.py.224: train: epoch 71 | batch 1710/2304 [I 2022-10-27 14:17:00] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:33 [I 2022-10-27 14:17:00] ...trap/engines/engine.py.229: process: 0.56635 | load: 0.00249 [I 2022-10-27 14:17:00] ...trap/engines/engine.py.230: loss: 0.17633 [I 2022-10-27 14:17:00] ...optimizers/trijoint.py.151: total_norm: 45.101604 [I 2022-10-27 14:17:05] ...trap/engines/engine.py.224: train: epoch 71 | batch 1720/2304 [I 2022-10-27 14:17:05] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:27 [I 2022-10-27 14:17:05] ...trap/engines/engine.py.229: process: 0.54628 | load: 0.00246 [I 2022-10-27 14:17:05] ...trap/engines/engine.py.230: loss: 0.15229 [I 2022-10-27 14:17:05] ...optimizers/trijoint.py.151: total_norm: 12.732400 [I 2022-10-27 14:17:11] ...trap/engines/engine.py.224: train: epoch 71 | batch 1730/2304 [I 2022-10-27 14:17:11] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:21 [I 2022-10-27 14:17:11] ...trap/engines/engine.py.229: process: 0.56751 | load: 0.00245 [I 2022-10-27 14:17:11] ...trap/engines/engine.py.230: loss: 0.12042 [I 2022-10-27 14:17:11] ...optimizers/trijoint.py.151: total_norm: 15.974658 [I 2022-10-27 14:17:17] ...trap/engines/engine.py.224: train: epoch 71 | batch 1740/2304 [I 2022-10-27 14:17:17] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:14 [I 2022-10-27 14:17:17] ...trap/engines/engine.py.229: process: 0.54974 | load: 0.00252 [I 2022-10-27 14:17:17] ...trap/engines/engine.py.230: loss: 0.16422 [I 2022-10-27 14:17:17] ...optimizers/trijoint.py.151: total_norm: 6.763823 [I 2022-10-27 14:17:22] ...trap/engines/engine.py.224: train: epoch 71 | batch 1750/2304 [I 2022-10-27 14:17:22] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:08 [I 2022-10-27 14:17:22] ...trap/engines/engine.py.229: process: 0.56149 | load: 0.00251 [I 2022-10-27 14:17:22] ...trap/engines/engine.py.230: loss: 0.15616 [I 2022-10-27 14:17:22] ...optimizers/trijoint.py.151: total_norm: 24.622715 [I 2022-10-27 14:17:28] ...trap/engines/engine.py.224: train: epoch 71 | batch 1760/2304 [I 2022-10-27 14:17:28] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:03 [I 2022-10-27 14:17:28] ...trap/engines/engine.py.229: process: 0.56647 | load: 0.00246 [I 2022-10-27 14:17:28] ...trap/engines/engine.py.230: loss: 0.13439 [I 2022-10-27 14:17:28] ...optimizers/trijoint.py.151: total_norm: 10.444092 [I 2022-10-27 14:17:33] ...trap/engines/engine.py.224: train: epoch 71 | batch 1770/2304 [I 2022-10-27 14:17:33] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:05:03 [I 2022-10-27 14:17:33] ...trap/engines/engine.py.229: process: 0.56697 | load: 0.00244 [I 2022-10-27 14:17:33] ...trap/engines/engine.py.230: loss: 0.16230 [I 2022-10-27 14:17:33] ...optimizers/trijoint.py.151: total_norm: 11.551844 [I 2022-10-27 14:17:39] ...trap/engines/engine.py.224: train: epoch 71 | batch 1780/2304 [I 2022-10-27 14:17:39] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:53 [I 2022-10-27 14:17:39] ...trap/engines/engine.py.229: process: 0.56814 | load: 0.00245 [I 2022-10-27 14:17:39] ...trap/engines/engine.py.230: loss: 0.13004 [I 2022-10-27 14:17:39] ...optimizers/trijoint.py.151: total_norm: 28.174713 [I 2022-10-27 14:17:45] ...trap/engines/engine.py.224: train: epoch 71 | batch 1790/2304 [I 2022-10-27 14:17:45] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:48 [I 2022-10-27 14:17:45] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00241 [I 2022-10-27 14:17:45] ...trap/engines/engine.py.230: loss: 0.16076 [I 2022-10-27 14:17:45] ...optimizers/trijoint.py.151: total_norm: 17.770861 [I 2022-10-27 14:17:50] ...trap/engines/engine.py.224: train: epoch 71 | batch 1800/2304 [I 2022-10-27 14:17:50] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:42 [I 2022-10-27 14:17:50] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00242 [I 2022-10-27 14:17:50] ...trap/engines/engine.py.230: loss: 0.17730 [I 2022-10-27 14:17:50] ...optimizers/trijoint.py.151: total_norm: 10.027296 [I 2022-10-27 14:17:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 1810/2304 [I 2022-10-27 14:17:56] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:35 [I 2022-10-27 14:17:56] ...trap/engines/engine.py.229: process: 0.54168 | load: 0.00244 [I 2022-10-27 14:17:56] ...trap/engines/engine.py.230: loss: 0.14644 [I 2022-10-27 14:17:56] ...optimizers/trijoint.py.151: total_norm: 33.665890 [I 2022-10-27 14:18:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 1820/2304 [I 2022-10-27 14:18:01] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:30 [I 2022-10-27 14:18:01] ...trap/engines/engine.py.229: process: 0.54421 | load: 0.00244 [I 2022-10-27 14:18:01] ...trap/engines/engine.py.230: loss: 0.12485 [I 2022-10-27 14:18:01] ...optimizers/trijoint.py.151: total_norm: 16.627062 [I 2022-10-27 14:18:07] ...trap/engines/engine.py.224: train: epoch 71 | batch 1830/2304 [I 2022-10-27 14:18:07] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:23 [I 2022-10-27 14:18:07] ...trap/engines/engine.py.229: process: 0.54524 | load: 0.00244 [I 2022-10-27 14:18:07] ...trap/engines/engine.py.230: loss: 0.18419 [I 2022-10-27 14:18:07] ...optimizers/trijoint.py.151: total_norm: 11.805455 [I 2022-10-27 14:18:13] ...trap/engines/engine.py.224: train: epoch 71 | batch 1840/2304 [I 2022-10-27 14:18:13] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:19 [I 2022-10-27 14:18:13] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00255 [I 2022-10-27 14:18:13] ...trap/engines/engine.py.230: loss: 0.27317 [I 2022-10-27 14:18:13] ...optimizers/trijoint.py.151: total_norm: 18.362497 [I 2022-10-27 14:18:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 1850/2304 [I 2022-10-27 14:18:18] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:16 [I 2022-10-27 14:18:18] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00247 [I 2022-10-27 14:18:18] ...trap/engines/engine.py.230: loss: 0.13233 [I 2022-10-27 14:18:18] ...optimizers/trijoint.py.151: total_norm: 15.945320 [I 2022-10-27 14:18:24] ...trap/engines/engine.py.224: train: epoch 71 | batch 1860/2304 [I 2022-10-27 14:18:24] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:07 [I 2022-10-27 14:18:24] ...trap/engines/engine.py.229: process: 0.54783 | load: 0.00249 [I 2022-10-27 14:18:24] ...trap/engines/engine.py.230: loss: 0.12576 [I 2022-10-27 14:18:24] ...optimizers/trijoint.py.151: total_norm: 21.877485 [I 2022-10-27 14:18:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 1870/2304 [I 2022-10-27 14:18:30] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:03 [I 2022-10-27 14:18:30] ...trap/engines/engine.py.229: process: 0.55482 | load: 0.00243 [I 2022-10-27 14:18:30] ...trap/engines/engine.py.230: loss: 0.11180 [I 2022-10-27 14:18:30] ...optimizers/trijoint.py.151: total_norm: 18.377533 [I 2022-10-27 14:18:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 1880/2304 [I 2022-10-27 14:18:35] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:04:04 [I 2022-10-27 14:18:35] ...trap/engines/engine.py.229: process: 0.64543 | load: 0.00244 [I 2022-10-27 14:18:35] ...trap/engines/engine.py.230: loss: 0.14593 [I 2022-10-27 14:18:35] ...optimizers/trijoint.py.151: total_norm: 28.996595 [I 2022-10-27 14:18:41] ...trap/engines/engine.py.224: train: epoch 71 | batch 1890/2304 [I 2022-10-27 14:18:41] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:53 [I 2022-10-27 14:18:41] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00245 [I 2022-10-27 14:18:41] ...trap/engines/engine.py.230: loss: 0.14641 [I 2022-10-27 14:18:41] ...optimizers/trijoint.py.151: total_norm: 7.683990 [I 2022-10-27 14:18:47] ...trap/engines/engine.py.224: train: epoch 71 | batch 1900/2304 [I 2022-10-27 14:18:47] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:03:48 [I 2022-10-27 14:18:47] ...trap/engines/engine.py.229: process: 0.57676 | load: 0.00253 [I 2022-10-27 14:18:47] ...trap/engines/engine.py.230: loss: 0.10926 [I 2022-10-27 14:18:47] ...optimizers/trijoint.py.151: total_norm: 11.680694 [I 2022-10-27 14:18:52] ...trap/engines/engine.py.224: train: epoch 71 | batch 1910/2304 [I 2022-10-27 14:18:52] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:42 [I 2022-10-27 14:18:52] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00255 [I 2022-10-27 14:18:52] ...trap/engines/engine.py.230: loss: 0.19971 [I 2022-10-27 14:18:52] ...optimizers/trijoint.py.151: total_norm: 32.975254 [I 2022-10-27 14:18:58] ...trap/engines/engine.py.224: train: epoch 71 | batch 1920/2304 [I 2022-10-27 14:18:58] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:34 [I 2022-10-27 14:18:58] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00247 [I 2022-10-27 14:18:58] ...trap/engines/engine.py.230: loss: 0.15117 [I 2022-10-27 14:18:58] ...optimizers/trijoint.py.151: total_norm: 26.171185 [I 2022-10-27 14:19:03] ...trap/engines/engine.py.224: train: epoch 71 | batch 1930/2304 [I 2022-10-27 14:19:03] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:29 [I 2022-10-27 14:19:03] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00246 [I 2022-10-27 14:19:03] ...trap/engines/engine.py.230: loss: 0.11108 [I 2022-10-27 14:19:03] ...optimizers/trijoint.py.151: total_norm: 22.114069 [I 2022-10-27 14:19:09] ...trap/engines/engine.py.224: train: epoch 71 | batch 1940/2304 [I 2022-10-27 14:19:09] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:23 [I 2022-10-27 14:19:09] ...trap/engines/engine.py.229: process: 0.55268 | load: 0.00247 [I 2022-10-27 14:19:09] ...trap/engines/engine.py.230: loss: 0.13754 [I 2022-10-27 14:19:09] ...optimizers/trijoint.py.151: total_norm: 14.493674 [I 2022-10-27 14:19:15] ...trap/engines/engine.py.224: train: epoch 71 | batch 1950/2304 [I 2022-10-27 14:19:15] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:17 [I 2022-10-27 14:19:15] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00243 [I 2022-10-27 14:19:15] ...trap/engines/engine.py.230: loss: 0.16126 [I 2022-10-27 14:19:15] ...optimizers/trijoint.py.151: total_norm: 16.803503 [I 2022-10-27 14:19:20] ...trap/engines/engine.py.224: train: epoch 71 | batch 1960/2304 [I 2022-10-27 14:19:20] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:10 [I 2022-10-27 14:19:20] ...trap/engines/engine.py.229: process: 0.55126 | load: 0.00246 [I 2022-10-27 14:19:20] ...trap/engines/engine.py.230: loss: 0.10871 [I 2022-10-27 14:19:20] ...optimizers/trijoint.py.151: total_norm: 15.820794 [I 2022-10-27 14:19:26] ...trap/engines/engine.py.224: train: epoch 71 | batch 1970/2304 [I 2022-10-27 14:19:26] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:05 [I 2022-10-27 14:19:26] ...trap/engines/engine.py.229: process: 0.55016 | load: 0.00241 [I 2022-10-27 14:19:26] ...trap/engines/engine.py.230: loss: 0.12933 [I 2022-10-27 14:19:26] ...optimizers/trijoint.py.151: total_norm: 9.227147 [I 2022-10-27 14:19:31] ...trap/engines/engine.py.224: train: epoch 71 | batch 1980/2304 [I 2022-10-27 14:19:31] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:01 [I 2022-10-27 14:19:31] ...trap/engines/engine.py.229: process: 0.56243 | load: 0.00245 [I 2022-10-27 14:19:31] ...trap/engines/engine.py.230: loss: 0.09714 [I 2022-10-27 14:19:31] ...optimizers/trijoint.py.151: total_norm: 10.490325 [I 2022-10-27 14:19:37] ...trap/engines/engine.py.224: train: epoch 71 | batch 1990/2304 [I 2022-10-27 14:19:37] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:02:55 [I 2022-10-27 14:19:37] ...trap/engines/engine.py.229: process: 0.55319 | load: 0.00242 [I 2022-10-27 14:19:37] ...trap/engines/engine.py.230: loss: 0.12995 [I 2022-10-27 14:19:37] ...optimizers/trijoint.py.151: total_norm: 13.955757 [I 2022-10-27 14:19:43] ...trap/engines/engine.py.224: train: epoch 71 | batch 2000/2304 [I 2022-10-27 14:19:43] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:02:55 [I 2022-10-27 14:19:43] ...trap/engines/engine.py.229: process: 0.56357 | load: 0.00247 [I 2022-10-27 14:19:43] ...trap/engines/engine.py.230: loss: 0.11475 [I 2022-10-27 14:19:43] ...optimizers/trijoint.py.151: total_norm: 18.582628 [I 2022-10-27 14:19:48] ...trap/engines/engine.py.224: train: epoch 71 | batch 2010/2304 [I 2022-10-27 14:19:48] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:02:45 [I 2022-10-27 14:19:48] ...trap/engines/engine.py.229: process: 0.55767 | load: 0.00250 [I 2022-10-27 14:19:48] ...trap/engines/engine.py.230: loss: 0.17125 [I 2022-10-27 14:19:48] ...optimizers/trijoint.py.151: total_norm: 24.987526 [I 2022-10-27 14:19:54] ...trap/engines/engine.py.224: train: epoch 71 | batch 2020/2304 [I 2022-10-27 14:19:54] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:40 [I 2022-10-27 14:19:54] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00248 [I 2022-10-27 14:19:54] ...trap/engines/engine.py.230: loss: 0.14378 [I 2022-10-27 14:19:54] ...optimizers/trijoint.py.151: total_norm: 13.138109 [I 2022-10-27 14:20:00] ...trap/engines/engine.py.224: train: epoch 71 | batch 2030/2304 [I 2022-10-27 14:20:00] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:32 [I 2022-10-27 14:20:00] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00246 [I 2022-10-27 14:20:00] ...trap/engines/engine.py.230: loss: 0.14307 [I 2022-10-27 14:20:00] ...optimizers/trijoint.py.151: total_norm: 37.555428 [I 2022-10-27 14:20:05] ...trap/engines/engine.py.224: train: epoch 71 | batch 2040/2304 [I 2022-10-27 14:20:05] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:28 [I 2022-10-27 14:20:05] ...trap/engines/engine.py.229: process: 0.57068 | load: 0.00245 [I 2022-10-27 14:20:05] ...trap/engines/engine.py.230: loss: 0.12858 [I 2022-10-27 14:20:05] ...optimizers/trijoint.py.151: total_norm: 12.195585 [I 2022-10-27 14:20:11] ...trap/engines/engine.py.224: train: epoch 71 | batch 2050/2304 [I 2022-10-27 14:20:11] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:21 [I 2022-10-27 14:20:11] ...trap/engines/engine.py.229: process: 0.55711 | load: 0.00245 [I 2022-10-27 14:20:11] ...trap/engines/engine.py.230: loss: 0.09614 [I 2022-10-27 14:20:11] ...optimizers/trijoint.py.151: total_norm: 9.805228 [I 2022-10-27 14:20:16] ...trap/engines/engine.py.224: train: epoch 71 | batch 2060/2304 [I 2022-10-27 14:20:16] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:15 [I 2022-10-27 14:20:16] ...trap/engines/engine.py.229: process: 0.55066 | load: 0.00241 [I 2022-10-27 14:20:16] ...trap/engines/engine.py.230: loss: 0.14554 [I 2022-10-27 14:20:16] ...optimizers/trijoint.py.151: total_norm: 32.135399 [I 2022-10-27 14:20:22] ...trap/engines/engine.py.224: train: epoch 71 | batch 2070/2304 [I 2022-10-27 14:20:22] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:10 [I 2022-10-27 14:20:22] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00244 [I 2022-10-27 14:20:22] ...trap/engines/engine.py.230: loss: 0.10968 [I 2022-10-27 14:20:22] ...optimizers/trijoint.py.151: total_norm: 14.088675 [I 2022-10-27 14:20:28] ...trap/engines/engine.py.224: train: epoch 71 | batch 2080/2304 [I 2022-10-27 14:20:28] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:04 [I 2022-10-27 14:20:28] ...trap/engines/engine.py.229: process: 0.55174 | load: 0.00245 [I 2022-10-27 14:20:28] ...trap/engines/engine.py.230: loss: 0.17007 [I 2022-10-27 14:20:28] ...optimizers/trijoint.py.151: total_norm: 17.639765 [I 2022-10-27 14:20:33] ...trap/engines/engine.py.224: train: epoch 71 | batch 2090/2304 [I 2022-10-27 14:20:33] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:01:59 [I 2022-10-27 14:20:33] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00249 [I 2022-10-27 14:20:33] ...trap/engines/engine.py.230: loss: 0.13351 [I 2022-10-27 14:20:33] ...optimizers/trijoint.py.151: total_norm: 49.029373 [I 2022-10-27 14:20:39] ...trap/engines/engine.py.224: train: epoch 71 | batch 2100/2304 [I 2022-10-27 14:20:39] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:01:54 [I 2022-10-27 14:20:39] ...trap/engines/engine.py.229: process: 0.56293 | load: 0.00245 [I 2022-10-27 14:20:39] ...trap/engines/engine.py.230: loss: 0.15224 [I 2022-10-27 14:20:39] ...optimizers/trijoint.py.151: total_norm: 22.257320 [I 2022-10-27 14:20:44] ...trap/engines/engine.py.224: train: epoch 71 | batch 2110/2304 [I 2022-10-27 14:20:44] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:01:48 [I 2022-10-27 14:20:44] ...trap/engines/engine.py.229: process: 0.55605 | load: 0.00243 [I 2022-10-27 14:20:44] ...trap/engines/engine.py.230: loss: 0.14472 [I 2022-10-27 14:20:44] ...optimizers/trijoint.py.151: total_norm: 32.493210 [I 2022-10-27 14:20:50] ...trap/engines/engine.py.224: train: epoch 71 | batch 2120/2304 [I 2022-10-27 14:20:50] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:43 [I 2022-10-27 14:20:50] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00241 [I 2022-10-27 14:20:50] ...trap/engines/engine.py.230: loss: 0.13174 [I 2022-10-27 14:20:50] ...optimizers/trijoint.py.151: total_norm: 18.115564 [I 2022-10-27 14:20:56] ...trap/engines/engine.py.224: train: epoch 71 | batch 2130/2304 [I 2022-10-27 14:20:56] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:37 [I 2022-10-27 14:20:56] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00245 [I 2022-10-27 14:20:56] ...trap/engines/engine.py.230: loss: 0.13465 [I 2022-10-27 14:20:56] ...optimizers/trijoint.py.151: total_norm: 27.199478 [I 2022-10-27 14:21:01] ...trap/engines/engine.py.224: train: epoch 71 | batch 2140/2304 [I 2022-10-27 14:21:01] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:31 [I 2022-10-27 14:21:01] ...trap/engines/engine.py.229: process: 0.56330 | load: 0.00243 [I 2022-10-27 14:21:01] ...trap/engines/engine.py.230: loss: 0.12288 [I 2022-10-27 14:21:01] ...optimizers/trijoint.py.151: total_norm: 12.790546 [I 2022-10-27 14:21:07] ...trap/engines/engine.py.224: train: epoch 71 | batch 2150/2304 [I 2022-10-27 14:21:07] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:25 [I 2022-10-27 14:21:07] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00243 [I 2022-10-27 14:21:07] ...trap/engines/engine.py.230: loss: 0.10894 [I 2022-10-27 14:21:07] ...optimizers/trijoint.py.151: total_norm: 10.253244 [I 2022-10-27 14:21:12] ...trap/engines/engine.py.224: train: epoch 71 | batch 2160/2304 [I 2022-10-27 14:21:12] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:20 [I 2022-10-27 14:21:12] ...trap/engines/engine.py.229: process: 0.56419 | load: 0.00253 [I 2022-10-27 14:21:12] ...trap/engines/engine.py.230: loss: 0.18632 [I 2022-10-27 14:21:12] ...optimizers/trijoint.py.151: total_norm: 26.078690 [I 2022-10-27 14:21:18] ...trap/engines/engine.py.224: train: epoch 71 | batch 2170/2304 [I 2022-10-27 14:21:18] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:14 [I 2022-10-27 14:21:18] ...trap/engines/engine.py.229: process: 0.54767 | load: 0.00241 [I 2022-10-27 14:21:18] ...trap/engines/engine.py.230: loss: 0.21465 [I 2022-10-27 14:21:18] ...optimizers/trijoint.py.151: total_norm: 11.716410 [I 2022-10-27 14:21:23] ...trap/engines/engine.py.224: train: epoch 71 | batch 2180/2304 [I 2022-10-27 14:21:23] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:08 [I 2022-10-27 14:21:23] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00245 [I 2022-10-27 14:21:23] ...trap/engines/engine.py.230: loss: 0.19094 [I 2022-10-27 14:21:23] ...optimizers/trijoint.py.151: total_norm: 37.356709 [I 2022-10-27 14:21:29] ...trap/engines/engine.py.224: train: epoch 71 | batch 2190/2304 [I 2022-10-27 14:21:29] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:03 [I 2022-10-27 14:21:29] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00252 [I 2022-10-27 14:21:29] ...trap/engines/engine.py.230: loss: 0.11152 [I 2022-10-27 14:21:29] ...optimizers/trijoint.py.151: total_norm: 11.172902 [I 2022-10-27 14:21:35] ...trap/engines/engine.py.224: train: epoch 71 | batch 2200/2304 [I 2022-10-27 14:21:35] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:00:57 [I 2022-10-27 14:21:35] ...trap/engines/engine.py.229: process: 0.54939 | load: 0.00253 [I 2022-10-27 14:21:35] ...trap/engines/engine.py.230: loss: 0.14774 [I 2022-10-27 14:21:35] ...optimizers/trijoint.py.151: total_norm: 18.749863 [I 2022-10-27 14:21:40] ...trap/engines/engine.py.224: train: epoch 71 | batch 2210/2304 [I 2022-10-27 14:21:40] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:00:52 [I 2022-10-27 14:21:40] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00243 [I 2022-10-27 14:21:40] ...trap/engines/engine.py.230: loss: 0.17642 [I 2022-10-27 14:21:40] ...optimizers/trijoint.py.151: total_norm: 28.024618 [I 2022-10-27 14:21:46] ...trap/engines/engine.py.224: train: epoch 71 | batch 2220/2304 [I 2022-10-27 14:21:46] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:00:46 [I 2022-10-27 14:21:46] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00241 [I 2022-10-27 14:21:46] ...trap/engines/engine.py.230: loss: 0.19206 [I 2022-10-27 14:21:46] ...optimizers/trijoint.py.151: total_norm: 23.504744 [I 2022-10-27 14:21:51] ...trap/engines/engine.py.224: train: epoch 71 | batch 2230/2304 [I 2022-10-27 14:21:51] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:41 [I 2022-10-27 14:21:51] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00243 [I 2022-10-27 14:21:51] ...trap/engines/engine.py.230: loss: 0.12437 [I 2022-10-27 14:21:51] ...optimizers/trijoint.py.151: total_norm: 12.867850 [I 2022-10-27 14:21:57] ...trap/engines/engine.py.224: train: epoch 71 | batch 2240/2304 [I 2022-10-27 14:21:57] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:35 [I 2022-10-27 14:21:57] ...trap/engines/engine.py.229: process: 0.55378 | load: 0.00243 [I 2022-10-27 14:21:57] ...trap/engines/engine.py.230: loss: 0.18364 [I 2022-10-27 14:21:57] ...optimizers/trijoint.py.151: total_norm: 21.547991 [I 2022-10-27 14:22:03] ...trap/engines/engine.py.224: train: epoch 71 | batch 2250/2304 [I 2022-10-27 14:22:03] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:30 [I 2022-10-27 14:22:03] ...trap/engines/engine.py.229: process: 0.55485 | load: 0.00242 [I 2022-10-27 14:22:03] ...trap/engines/engine.py.230: loss: 0.12253 [I 2022-10-27 14:22:03] ...optimizers/trijoint.py.151: total_norm: 21.210653 [I 2022-10-27 14:22:08] ...trap/engines/engine.py.224: train: epoch 71 | batch 2260/2304 [I 2022-10-27 14:22:08] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:24 [I 2022-10-27 14:22:08] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00249 [I 2022-10-27 14:22:08] ...trap/engines/engine.py.230: loss: 0.11078 [I 2022-10-27 14:22:08] ...optimizers/trijoint.py.151: total_norm: 11.389126 [I 2022-10-27 14:22:14] ...trap/engines/engine.py.224: train: epoch 71 | batch 2270/2304 [I 2022-10-27 14:22:14] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:19 [I 2022-10-27 14:22:14] ...trap/engines/engine.py.229: process: 0.57885 | load: 0.00247 [I 2022-10-27 14:22:14] ...trap/engines/engine.py.230: loss: 0.12918 [I 2022-10-27 14:22:14] ...optimizers/trijoint.py.151: total_norm: 11.242622 [I 2022-10-27 14:22:19] ...trap/engines/engine.py.224: train: epoch 71 | batch 2280/2304 [I 2022-10-27 14:22:19] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:13 [I 2022-10-27 14:22:19] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00267 [I 2022-10-27 14:22:19] ...trap/engines/engine.py.230: loss: 0.15849 [I 2022-10-27 14:22:19] ...optimizers/trijoint.py.151: total_norm: 21.813852 [I 2022-10-27 14:22:25] ...trap/engines/engine.py.224: train: epoch 71 | batch 2290/2304 [I 2022-10-27 14:22:25] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:07 [I 2022-10-27 14:22:25] ...trap/engines/engine.py.229: process: 0.55231 | load: 0.00250 [I 2022-10-27 14:22:25] ...trap/engines/engine.py.230: loss: 0.14260 [I 2022-10-27 14:22:25] ...optimizers/trijoint.py.151: total_norm: 15.667076 [I 2022-10-27 14:22:30] ...trap/engines/engine.py.224: train: epoch 71 | batch 2300/2304 [I 2022-10-27 14:22:30] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:02 [I 2022-10-27 14:22:30] ...trap/engines/engine.py.229: process: 0.54777 | load: 0.00016 [I 2022-10-27 14:22:30] ...trap/engines/engine.py.230: loss: 0.16428 [I 2022-10-27 14:22:30] ...optimizers/trijoint.py.151: total_norm: 13.784111 [I 2022-10-27 14:22:33] ...trap/engines/engine.py.224: train: epoch 71 | batch 2304/2304 [I 2022-10-27 14:22:33] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:00 [I 2022-10-27 14:22:33] ...trap/engines/engine.py.229: process: 0.54122 | load: 0.00015 [I 2022-10-27 14:22:33] ...trap/engines/engine.py.230: loss: 0.17970 [I 2022-10-27 14:22:33] ...optimizers/trijoint.py.151: total_norm: 15.216301 [S 2022-10-27 14:22:34] ...trap/engines/engine.py.129: train_epoch.epoch: 71 [S 2022-10-27 14:22:34] ...trap/engines/engine.py.129: train_epoch.loss: 0.1434365162137015 [I 2022-10-27 14:22:34] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 14:22:35] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 71 [I 2022-10-27 14:22:35] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 14:22:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 14:22:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 14:22:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 14:22:39] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 14:22:40] ...trap/engines/engine.py.321: eval: epoch 71 | batch 0/510 [I 2022-10-27 14:22:40] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:40:38 [I 2022-10-27 14:22:40] ...trap/engines/engine.py.326: process: 4.78197 | load: 3.06129 [I 2022-10-27 14:22:43] ...trap/engines/engine.py.321: eval: epoch 71 | batch 10/510 [I 2022-10-27 14:22:43] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:43 [I 2022-10-27 14:22:43] ...trap/engines/engine.py.326: process: 0.32908 | load: 0.00022 [I 2022-10-27 14:22:46] ...trap/engines/engine.py.321: eval: epoch 71 | batch 20/510 [I 2022-10-27 14:22:46] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:09 [I 2022-10-27 14:22:46] ...trap/engines/engine.py.326: process: 0.32444 | load: 0.00018 [I 2022-10-27 14:22:50] ...trap/engines/engine.py.321: eval: epoch 71 | batch 30/510 [I 2022-10-27 14:22:50] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:42 [I 2022-10-27 14:22:50] ...trap/engines/engine.py.326: process: 0.32666 | load: 0.00017 [I 2022-10-27 14:22:53] ...trap/engines/engine.py.321: eval: epoch 71 | batch 40/510 [I 2022-10-27 14:22:53] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:36 [I 2022-10-27 14:22:53] ...trap/engines/engine.py.326: process: 0.32865 | load: 0.00018 [I 2022-10-27 14:22:56] ...trap/engines/engine.py.321: eval: epoch 71 | batch 50/510 [I 2022-10-27 14:22:56] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:34 [I 2022-10-27 14:22:56] ...trap/engines/engine.py.326: process: 0.33109 | load: 0.00020 [I 2022-10-27 14:23:00] ...trap/engines/engine.py.321: eval: epoch 71 | batch 60/510 [I 2022-10-27 14:23:00] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:34 [I 2022-10-27 14:23:00] ...trap/engines/engine.py.326: process: 0.35558 | load: 0.00022 [I 2022-10-27 14:23:03] ...trap/engines/engine.py.321: eval: epoch 71 | batch 70/510 [I 2022-10-27 14:23:03] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:28 [I 2022-10-27 14:23:03] ...trap/engines/engine.py.326: process: 0.33242 | load: 0.00022 [I 2022-10-27 14:23:07] ...trap/engines/engine.py.321: eval: epoch 71 | batch 80/510 [I 2022-10-27 14:23:07] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:26 [I 2022-10-27 14:23:07] ...trap/engines/engine.py.326: process: 0.31864 | load: 0.00017 [I 2022-10-27 14:23:10] ...trap/engines/engine.py.321: eval: epoch 71 | batch 90/510 [I 2022-10-27 14:23:10] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:21 [I 2022-10-27 14:23:10] ...trap/engines/engine.py.326: process: 0.32179 | load: 0.00022 [I 2022-10-27 14:23:13] ...trap/engines/engine.py.321: eval: epoch 71 | batch 100/510 [I 2022-10-27 14:23:13] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-27 14:23:13] ...trap/engines/engine.py.326: process: 0.28863 | load: 0.00017 [I 2022-10-27 14:23:16] ...trap/engines/engine.py.321: eval: epoch 71 | batch 110/510 [I 2022-10-27 14:23:16] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-27 14:23:16] ...trap/engines/engine.py.326: process: 0.28457 | load: 0.00013 [I 2022-10-27 14:23:19] ...trap/engines/engine.py.321: eval: epoch 71 | batch 120/510 [I 2022-10-27 14:23:19] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 14:23:19] ...trap/engines/engine.py.326: process: 0.28931 | load: 0.00016 [I 2022-10-27 14:23:22] ...trap/engines/engine.py.321: eval: epoch 71 | batch 130/510 [I 2022-10-27 14:23:22] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:48 [I 2022-10-27 14:23:22] ...trap/engines/engine.py.326: process: 0.28405 | load: 0.00015 [I 2022-10-27 14:23:25] ...trap/engines/engine.py.321: eval: epoch 71 | batch 140/510 [I 2022-10-27 14:23:25] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:46 [I 2022-10-27 14:23:25] ...trap/engines/engine.py.326: process: 0.29157 | load: 0.00014 [I 2022-10-27 14:23:28] ...trap/engines/engine.py.321: eval: epoch 71 | batch 150/510 [I 2022-10-27 14:23:28] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 14:23:28] ...trap/engines/engine.py.326: process: 0.30118 | load: 0.00014 [I 2022-10-27 14:23:31] ...trap/engines/engine.py.321: eval: epoch 71 | batch 160/510 [I 2022-10-27 14:23:31] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 14:23:31] ...trap/engines/engine.py.326: process: 0.28341 | load: 0.00014 [I 2022-10-27 14:23:33] ...trap/engines/engine.py.321: eval: epoch 71 | batch 170/510 [I 2022-10-27 14:23:33] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 14:23:33] ...trap/engines/engine.py.326: process: 0.28385 | load: 0.00014 [I 2022-10-27 14:23:36] ...trap/engines/engine.py.321: eval: epoch 71 | batch 180/510 [I 2022-10-27 14:23:36] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:36 [I 2022-10-27 14:23:36] ...trap/engines/engine.py.326: process: 0.30483 | load: 0.00014 [I 2022-10-27 14:23:39] ...trap/engines/engine.py.321: eval: epoch 71 | batch 190/510 [I 2022-10-27 14:23:39] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 14:23:39] ...trap/engines/engine.py.326: process: 0.28448 | load: 0.00031 [I 2022-10-27 14:23:42] ...trap/engines/engine.py.321: eval: epoch 71 | batch 200/510 [I 2022-10-27 14:23:42] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 14:23:42] ...trap/engines/engine.py.326: process: 0.29103 | load: 0.00017 [I 2022-10-27 14:23:45] ...trap/engines/engine.py.321: eval: epoch 71 | batch 210/510 [I 2022-10-27 14:23:45] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:25 [I 2022-10-27 14:23:45] ...trap/engines/engine.py.326: process: 0.28042 | load: 0.00014 [I 2022-10-27 14:23:48] ...trap/engines/engine.py.321: eval: epoch 71 | batch 220/510 [I 2022-10-27 14:23:48] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 14:23:48] ...trap/engines/engine.py.326: process: 0.29216 | load: 0.00015 [I 2022-10-27 14:23:51] ...trap/engines/engine.py.321: eval: epoch 71 | batch 230/510 [I 2022-10-27 14:23:51] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:19 [I 2022-10-27 14:23:51] ...trap/engines/engine.py.326: process: 0.28023 | load: 0.00015 [I 2022-10-27 14:23:54] ...trap/engines/engine.py.321: eval: epoch 71 | batch 240/510 [I 2022-10-27 14:23:54] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 14:23:54] ...trap/engines/engine.py.326: process: 0.28957 | load: 0.00014 [I 2022-10-27 14:23:56] ...trap/engines/engine.py.321: eval: epoch 71 | batch 250/510 [I 2022-10-27 14:23:56] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 14:23:56] ...trap/engines/engine.py.326: process: 0.28850 | load: 0.00022 [I 2022-10-27 14:23:59] ...trap/engines/engine.py.321: eval: epoch 71 | batch 260/510 [I 2022-10-27 14:23:59] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:11 [I 2022-10-27 14:23:59] ...trap/engines/engine.py.326: process: 0.28546 | load: 0.00015 [I 2022-10-27 14:24:02] ...trap/engines/engine.py.321: eval: epoch 71 | batch 270/510 [I 2022-10-27 14:24:02] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 14:24:02] ...trap/engines/engine.py.326: process: 0.28898 | load: 0.00021 [I 2022-10-27 14:24:05] ...trap/engines/engine.py.321: eval: epoch 71 | batch 280/510 [I 2022-10-27 14:24:05] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 14:24:05] ...trap/engines/engine.py.326: process: 0.29690 | load: 0.00015 [I 2022-10-27 14:24:08] ...trap/engines/engine.py.321: eval: epoch 71 | batch 290/510 [I 2022-10-27 14:24:08] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 14:24:08] ...trap/engines/engine.py.326: process: 0.28579 | load: 0.00015 [I 2022-10-27 14:24:11] ...trap/engines/engine.py.321: eval: epoch 71 | batch 300/510 [I 2022-10-27 14:24:11] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:00 [I 2022-10-27 14:24:11] ...trap/engines/engine.py.326: process: 0.29623 | load: 0.00016 [I 2022-10-27 14:24:14] ...trap/engines/engine.py.321: eval: epoch 71 | batch 310/510 [I 2022-10-27 14:24:14] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 14:24:14] ...trap/engines/engine.py.326: process: 0.28719 | load: 0.00015 [I 2022-10-27 14:24:17] ...trap/engines/engine.py.321: eval: epoch 71 | batch 320/510 [I 2022-10-27 14:24:17] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 14:24:17] ...trap/engines/engine.py.326: process: 0.28367 | load: 0.00015 [I 2022-10-27 14:24:19] ...trap/engines/engine.py.321: eval: epoch 71 | batch 330/510 [I 2022-10-27 14:24:19] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 14:24:19] ...trap/engines/engine.py.326: process: 0.28390 | load: 0.00014 [I 2022-10-27 14:24:22] ...trap/engines/engine.py.321: eval: epoch 71 | batch 340/510 [I 2022-10-27 14:24:22] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-27 14:24:22] ...trap/engines/engine.py.326: process: 0.28378 | load: 0.00015 [I 2022-10-27 14:24:25] ...trap/engines/engine.py.321: eval: epoch 71 | batch 350/510 [I 2022-10-27 14:24:25] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:45 [I 2022-10-27 14:24:25] ...trap/engines/engine.py.326: process: 0.28546 | load: 0.00016 [I 2022-10-27 14:24:28] ...trap/engines/engine.py.321: eval: epoch 71 | batch 360/510 [I 2022-10-27 14:24:28] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 14:24:28] ...trap/engines/engine.py.326: process: 0.28960 | load: 0.00018 [I 2022-10-27 14:24:31] ...trap/engines/engine.py.321: eval: epoch 71 | batch 370/510 [I 2022-10-27 14:24:31] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 14:24:31] ...trap/engines/engine.py.326: process: 0.28373 | load: 0.00015 [I 2022-10-27 14:24:34] ...trap/engines/engine.py.321: eval: epoch 71 | batch 380/510 [I 2022-10-27 14:24:34] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:37 [I 2022-10-27 14:24:34] ...trap/engines/engine.py.326: process: 0.29273 | load: 0.00015 [I 2022-10-27 14:24:37] ...trap/engines/engine.py.321: eval: epoch 71 | batch 390/510 [I 2022-10-27 14:24:37] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 14:24:37] ...trap/engines/engine.py.326: process: 0.30805 | load: 0.00015 [I 2022-10-27 14:24:40] ...trap/engines/engine.py.321: eval: epoch 71 | batch 400/510 [I 2022-10-27 14:24:40] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 14:24:40] ...trap/engines/engine.py.326: process: 0.28924 | load: 0.00015 [I 2022-10-27 14:24:42] ...trap/engines/engine.py.321: eval: epoch 71 | batch 410/510 [I 2022-10-27 14:24:42] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 14:24:42] ...trap/engines/engine.py.326: process: 0.28328 | load: 0.00015 [I 2022-10-27 14:24:45] ...trap/engines/engine.py.321: eval: epoch 71 | batch 420/510 [I 2022-10-27 14:24:45] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 14:24:45] ...trap/engines/engine.py.326: process: 0.28611 | load: 0.00015 [I 2022-10-27 14:24:48] ...trap/engines/engine.py.321: eval: epoch 71 | batch 430/510 [I 2022-10-27 14:24:48] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:22 [I 2022-10-27 14:24:48] ...trap/engines/engine.py.326: process: 0.28354 | load: 0.00015 [I 2022-10-27 14:24:51] ...trap/engines/engine.py.321: eval: epoch 71 | batch 440/510 [I 2022-10-27 14:24:51] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 14:24:51] ...trap/engines/engine.py.326: process: 0.28897 | load: 0.00015 [I 2022-10-27 14:24:54] ...trap/engines/engine.py.321: eval: epoch 71 | batch 450/510 [I 2022-10-27 14:24:54] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 14:24:54] ...trap/engines/engine.py.326: process: 0.30110 | load: 0.00016 [I 2022-10-27 14:24:57] ...trap/engines/engine.py.321: eval: epoch 71 | batch 460/510 [I 2022-10-27 14:24:57] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:14 [I 2022-10-27 14:24:57] ...trap/engines/engine.py.326: process: 0.28596 | load: 0.00025 [I 2022-10-27 14:25:00] ...trap/engines/engine.py.321: eval: epoch 71 | batch 470/510 [I 2022-10-27 14:25:00] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 14:25:00] ...trap/engines/engine.py.326: process: 0.29080 | load: 0.00015 [I 2022-10-27 14:25:03] ...trap/engines/engine.py.321: eval: epoch 71 | batch 480/510 [I 2022-10-27 14:25:03] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 14:25:03] ...trap/engines/engine.py.326: process: 0.28939 | load: 0.00015 [I 2022-10-27 14:25:05] ...trap/engines/engine.py.321: eval: epoch 71 | batch 490/510 [I 2022-10-27 14:25:05] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 14:25:05] ...trap/engines/engine.py.326: process: 0.29076 | load: 0.00015 [I 2022-10-27 14:25:08] ...trap/engines/engine.py.321: eval: epoch 71 | batch 500/510 [I 2022-10-27 14:25:08] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 14:25:08] ...trap/engines/engine.py.326: process: 0.29015 | load: 0.00015 [I 2022-10-27 14:25:11] ...trap/engines/engine.py.321: eval: epoch 71 | batch 510/510 [I 2022-10-27 14:25:11] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 14:25:11] ...trap/engines/engine.py.326: process: 0.28766 | load: 0.00015 [S 2022-10-27 14:25:12] ...trap/engines/engine.py.132: eval_epoch.epoch: 71 [I 2022-10-27 14:25:12] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.719200 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.902400 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933000 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.715000 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.896400 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.928300 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011044 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010052 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007294 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008025 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006989 [S 2022-10-27 14:25:16] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009274 [I 2022-10-27 14:25:18] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 14:25:18] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 14:25:18] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 14:25:18] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 14:25:18] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 14:25:21] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 14:25:22] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 14:25:28] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 14:25:28] ...trap/engines/engine.py.163: Training model on trainset for epoch 72 [I 2022-10-27 14:25:28] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 14:25:31] ...trap/engines/engine.py.224: train: epoch 72 | batch 0/2304 [I 2022-10-27 14:25:31] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:49 [I 2022-10-27 14:25:31] ...trap/engines/engine.py.229: process: 2.80809 | load: 2.14746 [I 2022-10-27 14:25:31] ...trap/engines/engine.py.230: loss: 0.14602 [I 2022-10-27 14:25:31] ...optimizers/trijoint.py.151: total_norm: 19.936575 [I 2022-10-27 14:25:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 10/2304 [I 2022-10-27 14:25:37] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:27 [I 2022-10-27 14:25:37] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00399 [I 2022-10-27 14:25:37] ...trap/engines/engine.py.230: loss: 0.16098 [I 2022-10-27 14:25:37] ...optimizers/trijoint.py.151: total_norm: 15.814410 [I 2022-10-27 14:25:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 20/2304 [I 2022-10-27 14:25:42] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:05 [I 2022-10-27 14:25:42] ...trap/engines/engine.py.229: process: 0.55378 | load: 0.00410 [I 2022-10-27 14:25:42] ...trap/engines/engine.py.230: loss: 0.12686 [I 2022-10-27 14:25:42] ...optimizers/trijoint.py.151: total_norm: 9.530760 [I 2022-10-27 14:25:48] ...trap/engines/engine.py.224: train: epoch 72 | batch 30/2304 [I 2022-10-27 14:25:48] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:10 [I 2022-10-27 14:25:48] ...trap/engines/engine.py.229: process: 0.56414 | load: 0.00310 [I 2022-10-27 14:25:48] ...trap/engines/engine.py.230: loss: 0.11047 [I 2022-10-27 14:25:48] ...optimizers/trijoint.py.151: total_norm: 11.766249 [I 2022-10-27 14:25:53] ...trap/engines/engine.py.224: train: epoch 72 | batch 40/2304 [I 2022-10-27 14:25:53] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:54 [I 2022-10-27 14:25:53] ...trap/engines/engine.py.229: process: 0.53879 | load: 0.00303 [I 2022-10-27 14:25:53] ...trap/engines/engine.py.230: loss: 0.11116 [I 2022-10-27 14:25:53] ...optimizers/trijoint.py.151: total_norm: 6.370225 [I 2022-10-27 14:25:59] ...trap/engines/engine.py.224: train: epoch 72 | batch 50/2304 [I 2022-10-27 14:25:59] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:26 [I 2022-10-27 14:25:59] ...trap/engines/engine.py.229: process: 0.55150 | load: 0.00287 [I 2022-10-27 14:25:59] ...trap/engines/engine.py.230: loss: 0.14210 [I 2022-10-27 14:25:59] ...optimizers/trijoint.py.151: total_norm: 20.265982 [I 2022-10-27 14:26:05] ...trap/engines/engine.py.224: train: epoch 72 | batch 60/2304 [I 2022-10-27 14:26:05] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:48 [I 2022-10-27 14:26:05] ...trap/engines/engine.py.229: process: 0.55459 | load: 0.00308 [I 2022-10-27 14:26:05] ...trap/engines/engine.py.230: loss: 0.11540 [I 2022-10-27 14:26:05] ...optimizers/trijoint.py.151: total_norm: 12.983936 [I 2022-10-27 14:26:10] ...trap/engines/engine.py.224: train: epoch 72 | batch 70/2304 [I 2022-10-27 14:26:10] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:21:08 [I 2022-10-27 14:26:10] ...trap/engines/engine.py.229: process: 0.57407 | load: 0.00269 [I 2022-10-27 14:26:10] ...trap/engines/engine.py.230: loss: 0.11064 [I 2022-10-27 14:26:10] ...optimizers/trijoint.py.151: total_norm: 32.906414 [I 2022-10-27 14:26:16] ...trap/engines/engine.py.224: train: epoch 72 | batch 80/2304 [I 2022-10-27 14:26:16] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:45 [I 2022-10-27 14:26:16] ...trap/engines/engine.py.229: process: 0.56398 | load: 0.00285 [I 2022-10-27 14:26:16] ...trap/engines/engine.py.230: loss: 0.10808 [I 2022-10-27 14:26:16] ...optimizers/trijoint.py.151: total_norm: 12.228388 [I 2022-10-27 14:26:22] ...trap/engines/engine.py.224: train: epoch 72 | batch 90/2304 [I 2022-10-27 14:26:22] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:38 [I 2022-10-27 14:26:22] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00247 [I 2022-10-27 14:26:22] ...trap/engines/engine.py.230: loss: 0.15369 [I 2022-10-27 14:26:22] ...optimizers/trijoint.py.151: total_norm: 17.987541 [I 2022-10-27 14:26:27] ...trap/engines/engine.py.224: train: epoch 72 | batch 100/2304 [I 2022-10-27 14:26:27] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:20 [I 2022-10-27 14:26:27] ...trap/engines/engine.py.229: process: 0.54492 | load: 0.00296 [I 2022-10-27 14:26:27] ...trap/engines/engine.py.230: loss: 0.11887 [I 2022-10-27 14:26:27] ...optimizers/trijoint.py.151: total_norm: 11.289282 [I 2022-10-27 14:26:33] ...trap/engines/engine.py.224: train: epoch 72 | batch 110/2304 [I 2022-10-27 14:26:33] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:21 [I 2022-10-27 14:26:33] ...trap/engines/engine.py.229: process: 0.54905 | load: 0.00245 [I 2022-10-27 14:26:33] ...trap/engines/engine.py.230: loss: 0.13863 [I 2022-10-27 14:26:33] ...optimizers/trijoint.py.151: total_norm: 29.125824 [I 2022-10-27 14:26:38] ...trap/engines/engine.py.224: train: epoch 72 | batch 120/2304 [I 2022-10-27 14:26:38] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:14 [I 2022-10-27 14:26:38] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00258 [I 2022-10-27 14:26:38] ...trap/engines/engine.py.230: loss: 0.10490 [I 2022-10-27 14:26:38] ...optimizers/trijoint.py.151: total_norm: 12.024334 [I 2022-10-27 14:26:44] ...trap/engines/engine.py.224: train: epoch 72 | batch 130/2304 [I 2022-10-27 14:26:44] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:10 [I 2022-10-27 14:26:44] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00254 [I 2022-10-27 14:26:44] ...trap/engines/engine.py.230: loss: 0.12827 [I 2022-10-27 14:26:44] ...optimizers/trijoint.py.151: total_norm: 19.514271 [I 2022-10-27 14:26:50] ...trap/engines/engine.py.224: train: epoch 72 | batch 140/2304 [I 2022-10-27 14:26:50] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:02 [I 2022-10-27 14:26:50] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00250 [I 2022-10-27 14:26:50] ...trap/engines/engine.py.230: loss: 0.13254 [I 2022-10-27 14:26:50] ...optimizers/trijoint.py.151: total_norm: 21.742994 [I 2022-10-27 14:26:55] ...trap/engines/engine.py.224: train: epoch 72 | batch 150/2304 [I 2022-10-27 14:26:55] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:00 [I 2022-10-27 14:26:55] ...trap/engines/engine.py.229: process: 0.54948 | load: 0.00263 [I 2022-10-27 14:26:55] ...trap/engines/engine.py.230: loss: 0.10430 [I 2022-10-27 14:26:55] ...optimizers/trijoint.py.151: total_norm: 13.915027 [I 2022-10-27 14:27:01] ...trap/engines/engine.py.224: train: epoch 72 | batch 160/2304 [I 2022-10-27 14:27:01] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:54 [I 2022-10-27 14:27:01] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00291 [I 2022-10-27 14:27:01] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-27 14:27:01] ...optimizers/trijoint.py.151: total_norm: 13.731759 [I 2022-10-27 14:27:06] ...trap/engines/engine.py.224: train: epoch 72 | batch 170/2304 [I 2022-10-27 14:27:06] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:49 [I 2022-10-27 14:27:06] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00249 [I 2022-10-27 14:27:06] ...trap/engines/engine.py.230: loss: 0.12592 [I 2022-10-27 14:27:06] ...optimizers/trijoint.py.151: total_norm: 11.323920 [I 2022-10-27 14:27:12] ...trap/engines/engine.py.224: train: epoch 72 | batch 180/2304 [I 2022-10-27 14:27:12] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:37 [I 2022-10-27 14:27:12] ...trap/engines/engine.py.229: process: 0.55490 | load: 0.00269 [I 2022-10-27 14:27:12] ...trap/engines/engine.py.230: loss: 0.12451 [I 2022-10-27 14:27:12] ...optimizers/trijoint.py.151: total_norm: 8.474523 [I 2022-10-27 14:27:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 190/2304 [I 2022-10-27 14:27:17] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:34 [I 2022-10-27 14:27:17] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00262 [I 2022-10-27 14:27:17] ...trap/engines/engine.py.230: loss: 0.09729 [I 2022-10-27 14:27:18] ...optimizers/trijoint.py.151: total_norm: 10.260522 [I 2022-10-27 14:27:23] ...trap/engines/engine.py.224: train: epoch 72 | batch 200/2304 [I 2022-10-27 14:27:23] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:29 [I 2022-10-27 14:27:23] ...trap/engines/engine.py.229: process: 0.55639 | load: 0.00245 [I 2022-10-27 14:27:23] ...trap/engines/engine.py.230: loss: 0.11467 [I 2022-10-27 14:27:23] ...optimizers/trijoint.py.151: total_norm: 7.280328 [I 2022-10-27 14:27:29] ...trap/engines/engine.py.224: train: epoch 72 | batch 210/2304 [I 2022-10-27 14:27:29] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:20:07 [I 2022-10-27 14:27:29] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00264 [I 2022-10-27 14:27:29] ...trap/engines/engine.py.230: loss: 0.10904 [I 2022-10-27 14:27:29] ...optimizers/trijoint.py.151: total_norm: 16.673334 [I 2022-10-27 14:27:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 220/2304 [I 2022-10-27 14:27:34] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:15 [I 2022-10-27 14:27:34] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00257 [I 2022-10-27 14:27:34] ...trap/engines/engine.py.230: loss: 0.15152 [I 2022-10-27 14:27:34] ...optimizers/trijoint.py.151: total_norm: 17.825266 [I 2022-10-27 14:27:40] ...trap/engines/engine.py.224: train: epoch 72 | batch 230/2304 [I 2022-10-27 14:27:40] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:17 [I 2022-10-27 14:27:40] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00265 [I 2022-10-27 14:27:40] ...trap/engines/engine.py.230: loss: 0.11354 [I 2022-10-27 14:27:40] ...optimizers/trijoint.py.151: total_norm: 15.967019 [I 2022-10-27 14:27:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 240/2304 [I 2022-10-27 14:27:45] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:10 [I 2022-10-27 14:27:45] ...trap/engines/engine.py.229: process: 0.56440 | load: 0.00274 [I 2022-10-27 14:27:45] ...trap/engines/engine.py.230: loss: 0.16252 [I 2022-10-27 14:27:45] ...optimizers/trijoint.py.151: total_norm: 16.174070 [I 2022-10-27 14:27:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 250/2304 [I 2022-10-27 14:27:51] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:00 [I 2022-10-27 14:27:51] ...trap/engines/engine.py.229: process: 0.55001 | load: 0.00262 [I 2022-10-27 14:27:51] ...trap/engines/engine.py.230: loss: 0.10935 [I 2022-10-27 14:27:51] ...optimizers/trijoint.py.151: total_norm: 9.430015 [I 2022-10-27 14:27:57] ...trap/engines/engine.py.224: train: epoch 72 | batch 260/2304 [I 2022-10-27 14:27:57] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:02 [I 2022-10-27 14:27:57] ...trap/engines/engine.py.229: process: 0.55932 | load: 0.00272 [I 2022-10-27 14:27:57] ...trap/engines/engine.py.230: loss: 0.17438 [I 2022-10-27 14:27:57] ...optimizers/trijoint.py.151: total_norm: 17.906578 [I 2022-10-27 14:28:02] ...trap/engines/engine.py.224: train: epoch 72 | batch 270/2304 [I 2022-10-27 14:28:02] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:52 [I 2022-10-27 14:28:02] ...trap/engines/engine.py.229: process: 0.54778 | load: 0.00260 [I 2022-10-27 14:28:02] ...trap/engines/engine.py.230: loss: 0.11829 [I 2022-10-27 14:28:02] ...optimizers/trijoint.py.151: total_norm: 26.601480 [I 2022-10-27 14:28:08] ...trap/engines/engine.py.224: train: epoch 72 | batch 280/2304 [I 2022-10-27 14:28:08] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:57 [I 2022-10-27 14:28:08] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00253 [I 2022-10-27 14:28:08] ...trap/engines/engine.py.230: loss: 0.12939 [I 2022-10-27 14:28:08] ...optimizers/trijoint.py.151: total_norm: 20.072748 [I 2022-10-27 14:28:13] ...trap/engines/engine.py.224: train: epoch 72 | batch 290/2304 [I 2022-10-27 14:28:13] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:49 [I 2022-10-27 14:28:13] ...trap/engines/engine.py.229: process: 0.55375 | load: 0.00258 [I 2022-10-27 14:28:13] ...trap/engines/engine.py.230: loss: 0.12144 [I 2022-10-27 14:28:13] ...optimizers/trijoint.py.151: total_norm: 12.132847 [I 2022-10-27 14:28:19] ...trap/engines/engine.py.224: train: epoch 72 | batch 300/2304 [I 2022-10-27 14:28:19] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:47 [I 2022-10-27 14:28:19] ...trap/engines/engine.py.229: process: 0.56300 | load: 0.00256 [I 2022-10-27 14:28:19] ...trap/engines/engine.py.230: loss: 0.14115 [I 2022-10-27 14:28:19] ...optimizers/trijoint.py.151: total_norm: 19.022512 [I 2022-10-27 14:28:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 310/2304 [I 2022-10-27 14:28:25] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:47 [I 2022-10-27 14:28:25] ...trap/engines/engine.py.229: process: 0.56071 | load: 0.00247 [I 2022-10-27 14:28:25] ...trap/engines/engine.py.230: loss: 0.15523 [I 2022-10-27 14:28:25] ...optimizers/trijoint.py.151: total_norm: 22.544868 [I 2022-10-27 14:28:30] ...trap/engines/engine.py.224: train: epoch 72 | batch 320/2304 [I 2022-10-27 14:28:30] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:35 [I 2022-10-27 14:28:30] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00261 [I 2022-10-27 14:28:30] ...trap/engines/engine.py.230: loss: 0.11567 [I 2022-10-27 14:28:30] ...optimizers/trijoint.py.151: total_norm: 20.663712 [I 2022-10-27 14:28:36] ...trap/engines/engine.py.224: train: epoch 72 | batch 330/2304 [I 2022-10-27 14:28:36] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:17 [I 2022-10-27 14:28:36] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00258 [I 2022-10-27 14:28:36] ...trap/engines/engine.py.230: loss: 0.11612 [I 2022-10-27 14:28:36] ...optimizers/trijoint.py.151: total_norm: 17.554340 [I 2022-10-27 14:28:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 340/2304 [I 2022-10-27 14:28:42] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:12 [I 2022-10-27 14:28:42] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00245 [I 2022-10-27 14:28:42] ...trap/engines/engine.py.230: loss: 0.08371 [I 2022-10-27 14:28:42] ...optimizers/trijoint.py.151: total_norm: 10.739572 [I 2022-10-27 14:28:47] ...trap/engines/engine.py.224: train: epoch 72 | batch 350/2304 [I 2022-10-27 14:28:47] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:06 [I 2022-10-27 14:28:47] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00245 [I 2022-10-27 14:28:47] ...trap/engines/engine.py.230: loss: 0.11625 [I 2022-10-27 14:28:47] ...optimizers/trijoint.py.151: total_norm: 13.635798 [I 2022-10-27 14:28:53] ...trap/engines/engine.py.224: train: epoch 72 | batch 360/2304 [I 2022-10-27 14:28:53] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:00 [I 2022-10-27 14:28:53] ...trap/engines/engine.py.229: process: 0.54586 | load: 0.00247 [I 2022-10-27 14:28:53] ...trap/engines/engine.py.230: loss: 0.21061 [I 2022-10-27 14:28:53] ...optimizers/trijoint.py.151: total_norm: 17.359140 [I 2022-10-27 14:28:58] ...trap/engines/engine.py.224: train: epoch 72 | batch 370/2304 [I 2022-10-27 14:28:58] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:47 [I 2022-10-27 14:28:58] ...trap/engines/engine.py.229: process: 0.54851 | load: 0.00255 [I 2022-10-27 14:28:58] ...trap/engines/engine.py.230: loss: 0.11778 [I 2022-10-27 14:28:58] ...optimizers/trijoint.py.151: total_norm: 19.429197 [I 2022-10-27 14:29:04] ...trap/engines/engine.py.224: train: epoch 72 | batch 380/2304 [I 2022-10-27 14:29:04] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:47 [I 2022-10-27 14:29:04] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00245 [I 2022-10-27 14:29:04] ...trap/engines/engine.py.230: loss: 0.25582 [I 2022-10-27 14:29:04] ...optimizers/trijoint.py.151: total_norm: 31.257456 [I 2022-10-27 14:29:09] ...trap/engines/engine.py.224: train: epoch 72 | batch 390/2304 [I 2022-10-27 14:29:09] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:43 [I 2022-10-27 14:29:09] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00276 [I 2022-10-27 14:29:09] ...trap/engines/engine.py.230: loss: 0.17001 [I 2022-10-27 14:29:09] ...optimizers/trijoint.py.151: total_norm: 23.236546 [I 2022-10-27 14:29:15] ...trap/engines/engine.py.224: train: epoch 72 | batch 400/2304 [I 2022-10-27 14:29:15] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:45 [I 2022-10-27 14:29:15] ...trap/engines/engine.py.229: process: 0.56942 | load: 0.00245 [I 2022-10-27 14:29:15] ...trap/engines/engine.py.230: loss: 0.08916 [I 2022-10-27 14:29:15] ...optimizers/trijoint.py.151: total_norm: 9.186153 [I 2022-10-27 14:29:21] ...trap/engines/engine.py.224: train: epoch 72 | batch 410/2304 [I 2022-10-27 14:29:21] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:33 [I 2022-10-27 14:29:21] ...trap/engines/engine.py.229: process: 0.55265 | load: 0.00247 [I 2022-10-27 14:29:21] ...trap/engines/engine.py.230: loss: 0.18956 [I 2022-10-27 14:29:21] ...optimizers/trijoint.py.151: total_norm: 22.921900 [I 2022-10-27 14:29:26] ...trap/engines/engine.py.224: train: epoch 72 | batch 420/2304 [I 2022-10-27 14:29:26] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:30 [I 2022-10-27 14:29:26] ...trap/engines/engine.py.229: process: 0.55275 | load: 0.00253 [I 2022-10-27 14:29:26] ...trap/engines/engine.py.230: loss: 0.09612 [I 2022-10-27 14:29:26] ...optimizers/trijoint.py.151: total_norm: 8.682782 [I 2022-10-27 14:29:32] ...trap/engines/engine.py.224: train: epoch 72 | batch 430/2304 [I 2022-10-27 14:29:32] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:35 [I 2022-10-27 14:29:32] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00246 [I 2022-10-27 14:29:32] ...trap/engines/engine.py.230: loss: 0.24398 [I 2022-10-27 14:29:32] ...optimizers/trijoint.py.151: total_norm: 22.243315 [I 2022-10-27 14:29:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 440/2304 [I 2022-10-27 14:29:37] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:18 [I 2022-10-27 14:29:37] ...trap/engines/engine.py.229: process: 0.55472 | load: 0.00244 [I 2022-10-27 14:29:37] ...trap/engines/engine.py.230: loss: 0.14894 [I 2022-10-27 14:29:37] ...optimizers/trijoint.py.151: total_norm: 20.507092 [I 2022-10-27 14:29:43] ...trap/engines/engine.py.224: train: epoch 72 | batch 450/2304 [I 2022-10-27 14:29:43] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:10 [I 2022-10-27 14:29:43] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00243 [I 2022-10-27 14:29:43] ...trap/engines/engine.py.230: loss: 0.14523 [I 2022-10-27 14:29:43] ...optimizers/trijoint.py.151: total_norm: 29.142963 [I 2022-10-27 14:29:49] ...trap/engines/engine.py.224: train: epoch 72 | batch 460/2304 [I 2022-10-27 14:29:49] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:05 [I 2022-10-27 14:29:49] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00242 [I 2022-10-27 14:29:49] ...trap/engines/engine.py.230: loss: 0.09676 [I 2022-10-27 14:29:49] ...optimizers/trijoint.py.151: total_norm: 17.133808 [I 2022-10-27 14:29:54] ...trap/engines/engine.py.224: train: epoch 72 | batch 470/2304 [I 2022-10-27 14:29:54] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:56 [I 2022-10-27 14:29:54] ...trap/engines/engine.py.229: process: 0.54518 | load: 0.00242 [I 2022-10-27 14:29:54] ...trap/engines/engine.py.230: loss: 0.15210 [I 2022-10-27 14:29:54] ...optimizers/trijoint.py.151: total_norm: 26.102091 [I 2022-10-27 14:30:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 480/2304 [I 2022-10-27 14:30:00] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:00 [I 2022-10-27 14:30:00] ...trap/engines/engine.py.229: process: 0.56707 | load: 0.00268 [I 2022-10-27 14:30:00] ...trap/engines/engine.py.230: loss: 0.15543 [I 2022-10-27 14:30:00] ...optimizers/trijoint.py.151: total_norm: 20.777954 [I 2022-10-27 14:30:05] ...trap/engines/engine.py.224: train: epoch 72 | batch 490/2304 [I 2022-10-27 14:30:05] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:55 [I 2022-10-27 14:30:05] ...trap/engines/engine.py.229: process: 0.55186 | load: 0.00243 [I 2022-10-27 14:30:05] ...trap/engines/engine.py.230: loss: 0.11951 [I 2022-10-27 14:30:05] ...optimizers/trijoint.py.151: total_norm: 15.068746 [I 2022-10-27 14:30:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 500/2304 [I 2022-10-27 14:30:11] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:47 [I 2022-10-27 14:30:11] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00252 [I 2022-10-27 14:30:11] ...trap/engines/engine.py.230: loss: 0.17111 [I 2022-10-27 14:30:11] ...optimizers/trijoint.py.151: total_norm: 10.370170 [I 2022-10-27 14:30:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 510/2304 [I 2022-10-27 14:30:17] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:40 [I 2022-10-27 14:30:17] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00245 [I 2022-10-27 14:30:17] ...trap/engines/engine.py.230: loss: 0.13960 [I 2022-10-27 14:30:17] ...optimizers/trijoint.py.151: total_norm: 12.964313 [I 2022-10-27 14:30:22] ...trap/engines/engine.py.224: train: epoch 72 | batch 520/2304 [I 2022-10-27 14:30:22] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:34 [I 2022-10-27 14:30:22] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00243 [I 2022-10-27 14:30:22] ...trap/engines/engine.py.230: loss: 0.11839 [I 2022-10-27 14:30:22] ...optimizers/trijoint.py.151: total_norm: 19.483053 [I 2022-10-27 14:30:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 530/2304 [I 2022-10-27 14:30:28] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:50 [I 2022-10-27 14:30:28] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00242 [I 2022-10-27 14:30:28] ...trap/engines/engine.py.230: loss: 0.12122 [I 2022-10-27 14:30:28] ...optimizers/trijoint.py.151: total_norm: 10.194744 [I 2022-10-27 14:30:33] ...trap/engines/engine.py.224: train: epoch 72 | batch 540/2304 [I 2022-10-27 14:30:33] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:21 [I 2022-10-27 14:30:33] ...trap/engines/engine.py.229: process: 0.55193 | load: 0.00268 [I 2022-10-27 14:30:33] ...trap/engines/engine.py.230: loss: 0.18857 [I 2022-10-27 14:30:33] ...optimizers/trijoint.py.151: total_norm: 13.419911 [I 2022-10-27 14:30:39] ...trap/engines/engine.py.224: train: epoch 72 | batch 550/2304 [I 2022-10-27 14:30:39] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:13 [I 2022-10-27 14:30:39] ...trap/engines/engine.py.229: process: 0.55127 | load: 0.00259 [I 2022-10-27 14:30:39] ...trap/engines/engine.py.230: loss: 0.16585 [I 2022-10-27 14:30:39] ...optimizers/trijoint.py.151: total_norm: 9.469866 [I 2022-10-27 14:30:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 560/2304 [I 2022-10-27 14:30:45] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:10 [I 2022-10-27 14:30:45] ...trap/engines/engine.py.229: process: 0.55436 | load: 0.00257 [I 2022-10-27 14:30:45] ...trap/engines/engine.py.230: loss: 0.10895 [I 2022-10-27 14:30:45] ...optimizers/trijoint.py.151: total_norm: 11.642032 [I 2022-10-27 14:30:50] ...trap/engines/engine.py.224: train: epoch 72 | batch 570/2304 [I 2022-10-27 14:30:50] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:07 [I 2022-10-27 14:30:50] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00254 [I 2022-10-27 14:30:50] ...trap/engines/engine.py.230: loss: 0.12898 [I 2022-10-27 14:30:50] ...optimizers/trijoint.py.151: total_norm: 34.594353 [I 2022-10-27 14:30:56] ...trap/engines/engine.py.224: train: epoch 72 | batch 580/2304 [I 2022-10-27 14:30:56] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:01 [I 2022-10-27 14:30:56] ...trap/engines/engine.py.229: process: 0.56534 | load: 0.00251 [I 2022-10-27 14:30:56] ...trap/engines/engine.py.230: loss: 0.12016 [I 2022-10-27 14:30:56] ...optimizers/trijoint.py.151: total_norm: 18.032026 [I 2022-10-27 14:31:01] ...trap/engines/engine.py.224: train: epoch 72 | batch 590/2304 [I 2022-10-27 14:31:01] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:53 [I 2022-10-27 14:31:01] ...trap/engines/engine.py.229: process: 0.55321 | load: 0.00246 [I 2022-10-27 14:31:01] ...trap/engines/engine.py.230: loss: 0.14736 [I 2022-10-27 14:31:01] ...optimizers/trijoint.py.151: total_norm: 13.296216 [I 2022-10-27 14:31:07] ...trap/engines/engine.py.224: train: epoch 72 | batch 600/2304 [I 2022-10-27 14:31:07] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:56 [I 2022-10-27 14:31:07] ...trap/engines/engine.py.229: process: 0.55657 | load: 0.00247 [I 2022-10-27 14:31:07] ...trap/engines/engine.py.230: loss: 0.14097 [I 2022-10-27 14:31:07] ...optimizers/trijoint.py.151: total_norm: 9.744474 [I 2022-10-27 14:31:13] ...trap/engines/engine.py.224: train: epoch 72 | batch 610/2304 [I 2022-10-27 14:31:13] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:45 [I 2022-10-27 14:31:13] ...trap/engines/engine.py.229: process: 0.54877 | load: 0.00245 [I 2022-10-27 14:31:13] ...trap/engines/engine.py.230: loss: 0.16011 [I 2022-10-27 14:31:13] ...optimizers/trijoint.py.151: total_norm: 20.088274 [I 2022-10-27 14:31:18] ...trap/engines/engine.py.224: train: epoch 72 | batch 620/2304 [I 2022-10-27 14:31:18] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:42 [I 2022-10-27 14:31:18] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00245 [I 2022-10-27 14:31:18] ...trap/engines/engine.py.230: loss: 0.10101 [I 2022-10-27 14:31:18] ...optimizers/trijoint.py.151: total_norm: 10.972230 [I 2022-10-27 14:31:24] ...trap/engines/engine.py.224: train: epoch 72 | batch 630/2304 [I 2022-10-27 14:31:24] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:31 [I 2022-10-27 14:31:24] ...trap/engines/engine.py.229: process: 0.54747 | load: 0.00253 [I 2022-10-27 14:31:24] ...trap/engines/engine.py.230: loss: 0.10108 [I 2022-10-27 14:31:24] ...optimizers/trijoint.py.151: total_norm: 14.250418 [I 2022-10-27 14:31:29] ...trap/engines/engine.py.224: train: epoch 72 | batch 640/2304 [I 2022-10-27 14:31:29] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:24 [I 2022-10-27 14:31:29] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00247 [I 2022-10-27 14:31:29] ...trap/engines/engine.py.230: loss: 0.14896 [I 2022-10-27 14:31:29] ...optimizers/trijoint.py.151: total_norm: 14.751541 [I 2022-10-27 14:31:35] ...trap/engines/engine.py.224: train: epoch 72 | batch 650/2304 [I 2022-10-27 14:31:35] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:22 [I 2022-10-27 14:31:35] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00251 [I 2022-10-27 14:31:35] ...trap/engines/engine.py.230: loss: 0.10637 [I 2022-10-27 14:31:35] ...optimizers/trijoint.py.151: total_norm: 10.389254 [I 2022-10-27 14:31:40] ...trap/engines/engine.py.224: train: epoch 72 | batch 660/2304 [I 2022-10-27 14:31:40] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:20 [I 2022-10-27 14:31:40] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00244 [I 2022-10-27 14:31:40] ...trap/engines/engine.py.230: loss: 0.14969 [I 2022-10-27 14:31:40] ...optimizers/trijoint.py.151: total_norm: 9.030169 [I 2022-10-27 14:31:46] ...trap/engines/engine.py.224: train: epoch 72 | batch 670/2304 [I 2022-10-27 14:31:46] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:09 [I 2022-10-27 14:31:46] ...trap/engines/engine.py.229: process: 0.54698 | load: 0.00244 [I 2022-10-27 14:31:46] ...trap/engines/engine.py.230: loss: 0.17337 [I 2022-10-27 14:31:46] ...optimizers/trijoint.py.151: total_norm: 14.735257 [I 2022-10-27 14:31:52] ...trap/engines/engine.py.224: train: epoch 72 | batch 680/2304 [I 2022-10-27 14:31:52] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:14:58 [I 2022-10-27 14:31:52] ...trap/engines/engine.py.229: process: 0.54985 | load: 0.00245 [I 2022-10-27 14:31:52] ...trap/engines/engine.py.230: loss: 0.11656 [I 2022-10-27 14:31:52] ...optimizers/trijoint.py.151: total_norm: 21.641171 [I 2022-10-27 14:31:57] ...trap/engines/engine.py.224: train: epoch 72 | batch 690/2304 [I 2022-10-27 14:31:57] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:59 [I 2022-10-27 14:31:57] ...trap/engines/engine.py.229: process: 0.55035 | load: 0.00245 [I 2022-10-27 14:31:57] ...trap/engines/engine.py.230: loss: 0.12565 [I 2022-10-27 14:31:57] ...optimizers/trijoint.py.151: total_norm: 21.363997 [I 2022-10-27 14:32:03] ...trap/engines/engine.py.224: train: epoch 72 | batch 700/2304 [I 2022-10-27 14:32:03] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:50 [I 2022-10-27 14:32:03] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00247 [I 2022-10-27 14:32:03] ...trap/engines/engine.py.230: loss: 0.20726 [I 2022-10-27 14:32:03] ...optimizers/trijoint.py.151: total_norm: 23.399761 [I 2022-10-27 14:32:08] ...trap/engines/engine.py.224: train: epoch 72 | batch 710/2304 [I 2022-10-27 14:32:08] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:49 [I 2022-10-27 14:32:08] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00245 [I 2022-10-27 14:32:08] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-27 14:32:08] ...optimizers/trijoint.py.151: total_norm: 9.449668 [I 2022-10-27 14:32:14] ...trap/engines/engine.py.224: train: epoch 72 | batch 720/2304 [I 2022-10-27 14:32:14] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:42 [I 2022-10-27 14:32:14] ...trap/engines/engine.py.229: process: 0.55822 | load: 0.00241 [I 2022-10-27 14:32:14] ...trap/engines/engine.py.230: loss: 0.16675 [I 2022-10-27 14:32:14] ...optimizers/trijoint.py.151: total_norm: 15.910727 [I 2022-10-27 14:32:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 730/2304 [I 2022-10-27 14:32:20] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:36 [I 2022-10-27 14:32:20] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00242 [I 2022-10-27 14:32:20] ...trap/engines/engine.py.230: loss: 0.12918 [I 2022-10-27 14:32:20] ...optimizers/trijoint.py.151: total_norm: 29.081259 [I 2022-10-27 14:32:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 740/2304 [I 2022-10-27 14:32:25] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:30 [I 2022-10-27 14:32:25] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00250 [I 2022-10-27 14:32:25] ...trap/engines/engine.py.230: loss: 0.12558 [I 2022-10-27 14:32:25] ...optimizers/trijoint.py.151: total_norm: 30.922779 [I 2022-10-27 14:32:31] ...trap/engines/engine.py.224: train: epoch 72 | batch 750/2304 [I 2022-10-27 14:32:31] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:25 [I 2022-10-27 14:32:31] ...trap/engines/engine.py.229: process: 0.55623 | load: 0.00247 [I 2022-10-27 14:32:31] ...trap/engines/engine.py.230: loss: 0.20432 [I 2022-10-27 14:32:31] ...optimizers/trijoint.py.151: total_norm: 10.536510 [I 2022-10-27 14:32:36] ...trap/engines/engine.py.224: train: epoch 72 | batch 760/2304 [I 2022-10-27 14:32:36] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:26 [I 2022-10-27 14:32:36] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00245 [I 2022-10-27 14:32:36] ...trap/engines/engine.py.230: loss: 0.12236 [I 2022-10-27 14:32:36] ...optimizers/trijoint.py.151: total_norm: 11.956415 [I 2022-10-27 14:32:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 770/2304 [I 2022-10-27 14:32:42] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:20 [I 2022-10-27 14:32:42] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00250 [I 2022-10-27 14:32:42] ...trap/engines/engine.py.230: loss: 0.14732 [I 2022-10-27 14:32:42] ...optimizers/trijoint.py.151: total_norm: 21.156754 [I 2022-10-27 14:32:48] ...trap/engines/engine.py.224: train: epoch 72 | batch 780/2304 [I 2022-10-27 14:32:48] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:11 [I 2022-10-27 14:32:48] ...trap/engines/engine.py.229: process: 0.56134 | load: 0.00248 [I 2022-10-27 14:32:48] ...trap/engines/engine.py.230: loss: 0.13261 [I 2022-10-27 14:32:48] ...optimizers/trijoint.py.151: total_norm: 22.587914 [I 2022-10-27 14:32:53] ...trap/engines/engine.py.224: train: epoch 72 | batch 790/2304 [I 2022-10-27 14:32:53] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:10 [I 2022-10-27 14:32:53] ...trap/engines/engine.py.229: process: 0.56815 | load: 0.00254 [I 2022-10-27 14:32:53] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-27 14:32:53] ...optimizers/trijoint.py.151: total_norm: 14.251518 [I 2022-10-27 14:32:59] ...trap/engines/engine.py.224: train: epoch 72 | batch 800/2304 [I 2022-10-27 14:32:59] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:02 [I 2022-10-27 14:32:59] ...trap/engines/engine.py.229: process: 0.56704 | load: 0.00245 [I 2022-10-27 14:32:59] ...trap/engines/engine.py.230: loss: 0.10392 [I 2022-10-27 14:32:59] ...optimizers/trijoint.py.151: total_norm: 11.419236 [I 2022-10-27 14:33:04] ...trap/engines/engine.py.224: train: epoch 72 | batch 810/2304 [I 2022-10-27 14:33:04] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:58 [I 2022-10-27 14:33:04] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00251 [I 2022-10-27 14:33:04] ...trap/engines/engine.py.230: loss: 0.15634 [I 2022-10-27 14:33:04] ...optimizers/trijoint.py.151: total_norm: 18.720791 [I 2022-10-27 14:33:10] ...trap/engines/engine.py.224: train: epoch 72 | batch 820/2304 [I 2022-10-27 14:33:10] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:41 [I 2022-10-27 14:33:10] ...trap/engines/engine.py.229: process: 0.54397 | load: 0.00255 [I 2022-10-27 14:33:10] ...trap/engines/engine.py.230: loss: 0.12343 [I 2022-10-27 14:33:10] ...optimizers/trijoint.py.151: total_norm: 16.930239 [I 2022-10-27 14:33:16] ...trap/engines/engine.py.224: train: epoch 72 | batch 830/2304 [I 2022-10-27 14:33:16] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:41 [I 2022-10-27 14:33:16] ...trap/engines/engine.py.229: process: 0.55466 | load: 0.00249 [I 2022-10-27 14:33:16] ...trap/engines/engine.py.230: loss: 0.14938 [I 2022-10-27 14:33:16] ...optimizers/trijoint.py.151: total_norm: 12.948701 [I 2022-10-27 14:33:21] ...trap/engines/engine.py.224: train: epoch 72 | batch 840/2304 [I 2022-10-27 14:33:21] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:40 [I 2022-10-27 14:33:21] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00249 [I 2022-10-27 14:33:21] ...trap/engines/engine.py.230: loss: 0.11599 [I 2022-10-27 14:33:21] ...optimizers/trijoint.py.151: total_norm: 16.177771 [I 2022-10-27 14:33:27] ...trap/engines/engine.py.224: train: epoch 72 | batch 850/2304 [I 2022-10-27 14:33:27] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:30 [I 2022-10-27 14:33:27] ...trap/engines/engine.py.229: process: 0.54689 | load: 0.00246 [I 2022-10-27 14:33:27] ...trap/engines/engine.py.230: loss: 0.17918 [I 2022-10-27 14:33:27] ...optimizers/trijoint.py.151: total_norm: 11.078146 [I 2022-10-27 14:33:32] ...trap/engines/engine.py.224: train: epoch 72 | batch 860/2304 [I 2022-10-27 14:33:32] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:23 [I 2022-10-27 14:33:32] ...trap/engines/engine.py.229: process: 0.55945 | load: 0.00248 [I 2022-10-27 14:33:32] ...trap/engines/engine.py.230: loss: 0.12050 [I 2022-10-27 14:33:32] ...optimizers/trijoint.py.151: total_norm: 21.861004 [I 2022-10-27 14:33:38] ...trap/engines/engine.py.224: train: epoch 72 | batch 870/2304 [I 2022-10-27 14:33:38] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:13 [I 2022-10-27 14:33:38] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00244 [I 2022-10-27 14:33:38] ...trap/engines/engine.py.230: loss: 0.14429 [I 2022-10-27 14:33:38] ...optimizers/trijoint.py.151: total_norm: 26.281521 [I 2022-10-27 14:33:43] ...trap/engines/engine.py.224: train: epoch 72 | batch 880/2304 [I 2022-10-27 14:33:43] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:18 [I 2022-10-27 14:33:43] ...trap/engines/engine.py.229: process: 0.56433 | load: 0.00243 [I 2022-10-27 14:33:43] ...trap/engines/engine.py.230: loss: 0.11385 [I 2022-10-27 14:33:43] ...optimizers/trijoint.py.151: total_norm: 14.459742 [I 2022-10-27 14:33:49] ...trap/engines/engine.py.224: train: epoch 72 | batch 890/2304 [I 2022-10-27 14:33:49] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:02 [I 2022-10-27 14:33:49] ...trap/engines/engine.py.229: process: 0.54057 | load: 0.00245 [I 2022-10-27 14:33:49] ...trap/engines/engine.py.230: loss: 0.12201 [I 2022-10-27 14:33:49] ...optimizers/trijoint.py.151: total_norm: 23.814739 [I 2022-10-27 14:33:55] ...trap/engines/engine.py.224: train: epoch 72 | batch 900/2304 [I 2022-10-27 14:33:55] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:06 [I 2022-10-27 14:33:55] ...trap/engines/engine.py.229: process: 0.56392 | load: 0.00247 [I 2022-10-27 14:33:55] ...trap/engines/engine.py.230: loss: 0.14360 [I 2022-10-27 14:33:55] ...optimizers/trijoint.py.151: total_norm: 22.732893 [I 2022-10-27 14:34:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 910/2304 [I 2022-10-27 14:34:00] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:52 [I 2022-10-27 14:34:00] ...trap/engines/engine.py.229: process: 0.55892 | load: 0.00249 [I 2022-10-27 14:34:00] ...trap/engines/engine.py.230: loss: 0.12898 [I 2022-10-27 14:34:00] ...optimizers/trijoint.py.151: total_norm: 20.840082 [I 2022-10-27 14:34:06] ...trap/engines/engine.py.224: train: epoch 72 | batch 920/2304 [I 2022-10-27 14:34:06] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:55 [I 2022-10-27 14:34:06] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00249 [I 2022-10-27 14:34:06] ...trap/engines/engine.py.230: loss: 0.18218 [I 2022-10-27 14:34:06] ...optimizers/trijoint.py.151: total_norm: 9.492706 [I 2022-10-27 14:34:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 930/2304 [I 2022-10-27 14:34:11] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:41 [I 2022-10-27 14:34:11] ...trap/engines/engine.py.229: process: 0.54389 | load: 0.00245 [I 2022-10-27 14:34:11] ...trap/engines/engine.py.230: loss: 0.10908 [I 2022-10-27 14:34:11] ...optimizers/trijoint.py.151: total_norm: 6.969893 [I 2022-10-27 14:34:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 940/2304 [I 2022-10-27 14:34:17] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:39 [I 2022-10-27 14:34:17] ...trap/engines/engine.py.229: process: 0.56391 | load: 0.00247 [I 2022-10-27 14:34:17] ...trap/engines/engine.py.230: loss: 0.15961 [I 2022-10-27 14:34:17] ...optimizers/trijoint.py.151: total_norm: 20.027142 [I 2022-10-27 14:34:23] ...trap/engines/engine.py.224: train: epoch 72 | batch 950/2304 [I 2022-10-27 14:34:23] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:35 [I 2022-10-27 14:34:23] ...trap/engines/engine.py.229: process: 0.55719 | load: 0.00244 [I 2022-10-27 14:34:23] ...trap/engines/engine.py.230: loss: 0.10431 [I 2022-10-27 14:34:23] ...optimizers/trijoint.py.151: total_norm: 11.468128 [I 2022-10-27 14:34:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 960/2304 [I 2022-10-27 14:34:28] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:33 [I 2022-10-27 14:34:28] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00255 [I 2022-10-27 14:34:28] ...trap/engines/engine.py.230: loss: 0.13502 [I 2022-10-27 14:34:28] ...optimizers/trijoint.py.151: total_norm: 12.609533 [I 2022-10-27 14:34:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 970/2304 [I 2022-10-27 14:34:34] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:27 [I 2022-10-27 14:34:34] ...trap/engines/engine.py.229: process: 0.56719 | load: 0.00252 [I 2022-10-27 14:34:34] ...trap/engines/engine.py.230: loss: 0.21325 [I 2022-10-27 14:34:34] ...optimizers/trijoint.py.151: total_norm: 14.625536 [I 2022-10-27 14:34:39] ...trap/engines/engine.py.224: train: epoch 72 | batch 980/2304 [I 2022-10-27 14:34:39] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:15 [I 2022-10-27 14:34:39] ...trap/engines/engine.py.229: process: 0.54760 | load: 0.00250 [I 2022-10-27 14:34:39] ...trap/engines/engine.py.230: loss: 0.13275 [I 2022-10-27 14:34:39] ...optimizers/trijoint.py.151: total_norm: 14.507984 [I 2022-10-27 14:34:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 990/2304 [I 2022-10-27 14:34:45] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:10 [I 2022-10-27 14:34:45] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00248 [I 2022-10-27 14:34:45] ...trap/engines/engine.py.230: loss: 0.16203 [I 2022-10-27 14:34:45] ...optimizers/trijoint.py.151: total_norm: 18.928358 [I 2022-10-27 14:34:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 1000/2304 [I 2022-10-27 14:34:51] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:04 [I 2022-10-27 14:34:51] ...trap/engines/engine.py.229: process: 0.54566 | load: 0.00243 [I 2022-10-27 14:34:51] ...trap/engines/engine.py.230: loss: 0.13210 [I 2022-10-27 14:34:51] ...optimizers/trijoint.py.151: total_norm: 29.426060 [I 2022-10-27 14:34:56] ...trap/engines/engine.py.224: train: epoch 72 | batch 1010/2304 [I 2022-10-27 14:34:56] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:01 [I 2022-10-27 14:34:56] ...trap/engines/engine.py.229: process: 0.56270 | load: 0.00247 [I 2022-10-27 14:34:56] ...trap/engines/engine.py.230: loss: 0.10003 [I 2022-10-27 14:34:56] ...optimizers/trijoint.py.151: total_norm: 12.269445 [I 2022-10-27 14:35:02] ...trap/engines/engine.py.224: train: epoch 72 | batch 1020/2304 [I 2022-10-27 14:35:02] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:57 [I 2022-10-27 14:35:02] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00240 [I 2022-10-27 14:35:02] ...trap/engines/engine.py.230: loss: 0.12872 [I 2022-10-27 14:35:02] ...optimizers/trijoint.py.151: total_norm: 8.341638 [I 2022-10-27 14:35:07] ...trap/engines/engine.py.224: train: epoch 72 | batch 1030/2304 [I 2022-10-27 14:35:07] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:49 [I 2022-10-27 14:35:07] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00245 [I 2022-10-27 14:35:07] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-27 14:35:07] ...optimizers/trijoint.py.151: total_norm: 14.553872 [I 2022-10-27 14:35:13] ...trap/engines/engine.py.224: train: epoch 72 | batch 1040/2304 [I 2022-10-27 14:35:13] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:46 [I 2022-10-27 14:35:13] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00244 [I 2022-10-27 14:35:13] ...trap/engines/engine.py.230: loss: 0.23468 [I 2022-10-27 14:35:13] ...optimizers/trijoint.py.151: total_norm: 19.116262 [I 2022-10-27 14:35:18] ...trap/engines/engine.py.224: train: epoch 72 | batch 1050/2304 [I 2022-10-27 14:35:18] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:44 [I 2022-10-27 14:35:18] ...trap/engines/engine.py.229: process: 0.56913 | load: 0.00251 [I 2022-10-27 14:35:18] ...trap/engines/engine.py.230: loss: 0.18964 [I 2022-10-27 14:35:18] ...optimizers/trijoint.py.151: total_norm: 38.590809 [I 2022-10-27 14:35:24] ...trap/engines/engine.py.224: train: epoch 72 | batch 1060/2304 [I 2022-10-27 14:35:24] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:35 [I 2022-10-27 14:35:24] ...trap/engines/engine.py.229: process: 0.56702 | load: 0.00248 [I 2022-10-27 14:35:24] ...trap/engines/engine.py.230: loss: 0.09252 [I 2022-10-27 14:35:24] ...optimizers/trijoint.py.151: total_norm: 15.170285 [I 2022-10-27 14:35:30] ...trap/engines/engine.py.224: train: epoch 72 | batch 1070/2304 [I 2022-10-27 14:35:30] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:22 [I 2022-10-27 14:35:30] ...trap/engines/engine.py.229: process: 0.54127 | load: 0.00244 [I 2022-10-27 14:35:30] ...trap/engines/engine.py.230: loss: 0.24380 [I 2022-10-27 14:35:30] ...optimizers/trijoint.py.151: total_norm: 64.076019 [I 2022-10-27 14:35:35] ...trap/engines/engine.py.224: train: epoch 72 | batch 1080/2304 [I 2022-10-27 14:35:35] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:12 [I 2022-10-27 14:35:35] ...trap/engines/engine.py.229: process: 0.54987 | load: 0.00248 [I 2022-10-27 14:35:35] ...trap/engines/engine.py.230: loss: 0.11460 [I 2022-10-27 14:35:35] ...optimizers/trijoint.py.151: total_norm: 9.166929 [I 2022-10-27 14:35:41] ...trap/engines/engine.py.224: train: epoch 72 | batch 1090/2304 [I 2022-10-27 14:35:41] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:10 [I 2022-10-27 14:35:41] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00242 [I 2022-10-27 14:35:41] ...trap/engines/engine.py.230: loss: 0.10964 [I 2022-10-27 14:35:41] ...optimizers/trijoint.py.151: total_norm: 17.715385 [I 2022-10-27 14:35:46] ...trap/engines/engine.py.224: train: epoch 72 | batch 1100/2304 [I 2022-10-27 14:35:46] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:10 [I 2022-10-27 14:35:46] ...trap/engines/engine.py.229: process: 0.56293 | load: 0.00242 [I 2022-10-27 14:35:46] ...trap/engines/engine.py.230: loss: 0.17675 [I 2022-10-27 14:35:46] ...optimizers/trijoint.py.151: total_norm: 18.890507 [I 2022-10-27 14:35:52] ...trap/engines/engine.py.224: train: epoch 72 | batch 1110/2304 [I 2022-10-27 14:35:52] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:09 [I 2022-10-27 14:35:52] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00250 [I 2022-10-27 14:35:52] ...trap/engines/engine.py.230: loss: 0.16687 [I 2022-10-27 14:35:52] ...optimizers/trijoint.py.151: total_norm: 16.337078 [I 2022-10-27 14:35:57] ...trap/engines/engine.py.224: train: epoch 72 | batch 1120/2304 [I 2022-10-27 14:35:57] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:04 [I 2022-10-27 14:35:57] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00247 [I 2022-10-27 14:35:57] ...trap/engines/engine.py.230: loss: 0.15820 [I 2022-10-27 14:35:57] ...optimizers/trijoint.py.151: total_norm: 11.417630 [I 2022-10-27 14:36:03] ...trap/engines/engine.py.224: train: epoch 72 | batch 1130/2304 [I 2022-10-27 14:36:03] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:51 [I 2022-10-27 14:36:03] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00248 [I 2022-10-27 14:36:03] ...trap/engines/engine.py.230: loss: 0.14588 [I 2022-10-27 14:36:03] ...optimizers/trijoint.py.151: total_norm: 32.939232 [I 2022-10-27 14:36:09] ...trap/engines/engine.py.224: train: epoch 72 | batch 1140/2304 [I 2022-10-27 14:36:09] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:56 [I 2022-10-27 14:36:09] ...trap/engines/engine.py.229: process: 0.55809 | load: 0.00247 [I 2022-10-27 14:36:09] ...trap/engines/engine.py.230: loss: 0.22821 [I 2022-10-27 14:36:09] ...optimizers/trijoint.py.151: total_norm: 19.343941 [I 2022-10-27 14:36:14] ...trap/engines/engine.py.224: train: epoch 72 | batch 1150/2304 [I 2022-10-27 14:36:14] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:45 [I 2022-10-27 14:36:14] ...trap/engines/engine.py.229: process: 0.54784 | load: 0.00242 [I 2022-10-27 14:36:14] ...trap/engines/engine.py.230: loss: 0.13209 [I 2022-10-27 14:36:14] ...optimizers/trijoint.py.151: total_norm: 24.018450 [I 2022-10-27 14:36:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 1160/2304 [I 2022-10-27 14:36:20] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:43 [I 2022-10-27 14:36:20] ...trap/engines/engine.py.229: process: 0.56724 | load: 0.00241 [I 2022-10-27 14:36:20] ...trap/engines/engine.py.230: loss: 0.13205 [I 2022-10-27 14:36:20] ...optimizers/trijoint.py.151: total_norm: 10.509931 [I 2022-10-27 14:36:26] ...trap/engines/engine.py.224: train: epoch 72 | batch 1170/2304 [I 2022-10-27 14:36:26] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:31 [I 2022-10-27 14:36:26] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00242 [I 2022-10-27 14:36:26] ...trap/engines/engine.py.230: loss: 0.09110 [I 2022-10-27 14:36:26] ...optimizers/trijoint.py.151: total_norm: 9.164029 [I 2022-10-27 14:36:31] ...trap/engines/engine.py.224: train: epoch 72 | batch 1180/2304 [I 2022-10-27 14:36:31] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:26 [I 2022-10-27 14:36:31] ...trap/engines/engine.py.229: process: 0.56153 | load: 0.00245 [I 2022-10-27 14:36:31] ...trap/engines/engine.py.230: loss: 0.11239 [I 2022-10-27 14:36:31] ...optimizers/trijoint.py.151: total_norm: 11.983116 [I 2022-10-27 14:36:37] ...trap/engines/engine.py.224: train: epoch 72 | batch 1190/2304 [I 2022-10-27 14:36:37] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:23 [I 2022-10-27 14:36:37] ...trap/engines/engine.py.229: process: 0.56648 | load: 0.00241 [I 2022-10-27 14:36:37] ...trap/engines/engine.py.230: loss: 0.11192 [I 2022-10-27 14:36:37] ...optimizers/trijoint.py.151: total_norm: 50.934364 [I 2022-10-27 14:36:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 1200/2304 [I 2022-10-27 14:36:42] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:15 [I 2022-10-27 14:36:42] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00253 [I 2022-10-27 14:36:42] ...trap/engines/engine.py.230: loss: 0.11158 [I 2022-10-27 14:36:42] ...optimizers/trijoint.py.151: total_norm: 17.627203 [I 2022-10-27 14:36:48] ...trap/engines/engine.py.224: train: epoch 72 | batch 1210/2304 [I 2022-10-27 14:36:48] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:12 [I 2022-10-27 14:36:48] ...trap/engines/engine.py.229: process: 0.56193 | load: 0.00249 [I 2022-10-27 14:36:48] ...trap/engines/engine.py.230: loss: 0.13059 [I 2022-10-27 14:36:48] ...optimizers/trijoint.py.151: total_norm: 13.113004 [I 2022-10-27 14:36:54] ...trap/engines/engine.py.224: train: epoch 72 | batch 1220/2304 [I 2022-10-27 14:36:54] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:09 [I 2022-10-27 14:36:54] ...trap/engines/engine.py.229: process: 0.56256 | load: 0.00241 [I 2022-10-27 14:36:54] ...trap/engines/engine.py.230: loss: 0.15084 [I 2022-10-27 14:36:54] ...optimizers/trijoint.py.151: total_norm: 27.598166 [I 2022-10-27 14:36:59] ...trap/engines/engine.py.224: train: epoch 72 | batch 1230/2304 [I 2022-10-27 14:36:59] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:00 [I 2022-10-27 14:36:59] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00242 [I 2022-10-27 14:36:59] ...trap/engines/engine.py.230: loss: 0.13435 [I 2022-10-27 14:36:59] ...optimizers/trijoint.py.151: total_norm: 14.039821 [I 2022-10-27 14:37:05] ...trap/engines/engine.py.224: train: epoch 72 | batch 1240/2304 [I 2022-10-27 14:37:05] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:53 [I 2022-10-27 14:37:05] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00248 [I 2022-10-27 14:37:05] ...trap/engines/engine.py.230: loss: 0.23659 [I 2022-10-27 14:37:05] ...optimizers/trijoint.py.151: total_norm: 13.472275 [I 2022-10-27 14:37:10] ...trap/engines/engine.py.224: train: epoch 72 | batch 1250/2304 [I 2022-10-27 14:37:10] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:43 [I 2022-10-27 14:37:10] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00249 [I 2022-10-27 14:37:10] ...trap/engines/engine.py.230: loss: 0.24605 [I 2022-10-27 14:37:10] ...optimizers/trijoint.py.151: total_norm: 19.762899 [I 2022-10-27 14:37:16] ...trap/engines/engine.py.224: train: epoch 72 | batch 1260/2304 [I 2022-10-27 14:37:16] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:40 [I 2022-10-27 14:37:16] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00245 [I 2022-10-27 14:37:16] ...trap/engines/engine.py.230: loss: 0.12429 [I 2022-10-27 14:37:16] ...optimizers/trijoint.py.151: total_norm: 16.135380 [I 2022-10-27 14:37:22] ...trap/engines/engine.py.224: train: epoch 72 | batch 1270/2304 [I 2022-10-27 14:37:22] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:37 [I 2022-10-27 14:37:22] ...trap/engines/engine.py.229: process: 0.56665 | load: 0.00253 [I 2022-10-27 14:37:22] ...trap/engines/engine.py.230: loss: 0.10546 [I 2022-10-27 14:37:22] ...optimizers/trijoint.py.151: total_norm: 8.764372 [I 2022-10-27 14:37:27] ...trap/engines/engine.py.224: train: epoch 72 | batch 1280/2304 [I 2022-10-27 14:37:27] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:33 [I 2022-10-27 14:37:27] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00253 [I 2022-10-27 14:37:27] ...trap/engines/engine.py.230: loss: 0.16639 [I 2022-10-27 14:37:27] ...optimizers/trijoint.py.151: total_norm: 10.165679 [I 2022-10-27 14:37:33] ...trap/engines/engine.py.224: train: epoch 72 | batch 1290/2304 [I 2022-10-27 14:37:33] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:35 [I 2022-10-27 14:37:33] ...trap/engines/engine.py.229: process: 0.57565 | load: 0.00244 [I 2022-10-27 14:37:33] ...trap/engines/engine.py.230: loss: 0.09655 [I 2022-10-27 14:37:33] ...optimizers/trijoint.py.151: total_norm: 14.012347 [I 2022-10-27 14:37:38] ...trap/engines/engine.py.224: train: epoch 72 | batch 1300/2304 [I 2022-10-27 14:37:38] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:20 [I 2022-10-27 14:37:38] ...trap/engines/engine.py.229: process: 0.55095 | load: 0.00243 [I 2022-10-27 14:37:38] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-27 14:37:38] ...optimizers/trijoint.py.151: total_norm: 12.852696 [I 2022-10-27 14:37:44] ...trap/engines/engine.py.224: train: epoch 72 | batch 1310/2304 [I 2022-10-27 14:37:44] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:15 [I 2022-10-27 14:37:44] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00244 [I 2022-10-27 14:37:44] ...trap/engines/engine.py.230: loss: 0.11937 [I 2022-10-27 14:37:44] ...optimizers/trijoint.py.151: total_norm: 10.352018 [I 2022-10-27 14:37:50] ...trap/engines/engine.py.224: train: epoch 72 | batch 1320/2304 [I 2022-10-27 14:37:50] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:07 [I 2022-10-27 14:37:50] ...trap/engines/engine.py.229: process: 0.55684 | load: 0.00244 [I 2022-10-27 14:37:50] ...trap/engines/engine.py.230: loss: 0.13986 [I 2022-10-27 14:37:50] ...optimizers/trijoint.py.151: total_norm: 12.030096 [I 2022-10-27 14:37:55] ...trap/engines/engine.py.224: train: epoch 72 | batch 1330/2304 [I 2022-10-27 14:37:55] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:02 [I 2022-10-27 14:37:55] ...trap/engines/engine.py.229: process: 0.55126 | load: 0.00246 [I 2022-10-27 14:37:55] ...trap/engines/engine.py.230: loss: 0.09225 [I 2022-10-27 14:37:55] ...optimizers/trijoint.py.151: total_norm: 9.812889 [I 2022-10-27 14:38:01] ...trap/engines/engine.py.224: train: epoch 72 | batch 1340/2304 [I 2022-10-27 14:38:01] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:58 [I 2022-10-27 14:38:01] ...trap/engines/engine.py.229: process: 0.56270 | load: 0.00242 [I 2022-10-27 14:38:01] ...trap/engines/engine.py.230: loss: 0.18423 [I 2022-10-27 14:38:01] ...optimizers/trijoint.py.151: total_norm: 19.550081 [I 2022-10-27 14:38:07] ...trap/engines/engine.py.224: train: epoch 72 | batch 1350/2304 [I 2022-10-27 14:38:07] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:09:19 [I 2022-10-27 14:38:07] ...trap/engines/engine.py.229: process: 0.68235 | load: 0.00975 [I 2022-10-27 14:38:07] ...trap/engines/engine.py.230: loss: 0.12938 [I 2022-10-27 14:38:07] ...optimizers/trijoint.py.151: total_norm: 14.275833 [I 2022-10-27 14:38:12] ...trap/engines/engine.py.224: train: epoch 72 | batch 1360/2304 [I 2022-10-27 14:38:12] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:49 [I 2022-10-27 14:38:12] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00241 [I 2022-10-27 14:38:12] ...trap/engines/engine.py.230: loss: 0.12859 [I 2022-10-27 14:38:12] ...optimizers/trijoint.py.151: total_norm: 26.794363 [I 2022-10-27 14:38:18] ...trap/engines/engine.py.224: train: epoch 72 | batch 1370/2304 [I 2022-10-27 14:38:18] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:36 [I 2022-10-27 14:38:18] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00242 [I 2022-10-27 14:38:18] ...trap/engines/engine.py.230: loss: 0.12208 [I 2022-10-27 14:38:18] ...optimizers/trijoint.py.151: total_norm: 31.177807 [I 2022-10-27 14:38:23] ...trap/engines/engine.py.224: train: epoch 72 | batch 1380/2304 [I 2022-10-27 14:38:23] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:34 [I 2022-10-27 14:38:23] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00249 [I 2022-10-27 14:38:23] ...trap/engines/engine.py.230: loss: 0.15393 [I 2022-10-27 14:38:23] ...optimizers/trijoint.py.151: total_norm: 12.776519 [I 2022-10-27 14:38:29] ...trap/engines/engine.py.224: train: epoch 72 | batch 1390/2304 [I 2022-10-27 14:38:29] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:27 [I 2022-10-27 14:38:29] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00247 [I 2022-10-27 14:38:29] ...trap/engines/engine.py.230: loss: 0.16384 [I 2022-10-27 14:38:29] ...optimizers/trijoint.py.151: total_norm: 19.818588 [I 2022-10-27 14:38:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 1400/2304 [I 2022-10-27 14:38:34] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:22 [I 2022-10-27 14:38:34] ...trap/engines/engine.py.229: process: 0.56274 | load: 0.00245 [I 2022-10-27 14:38:34] ...trap/engines/engine.py.230: loss: 0.12739 [I 2022-10-27 14:38:34] ...optimizers/trijoint.py.151: total_norm: 16.118301 [I 2022-10-27 14:38:40] ...trap/engines/engine.py.224: train: epoch 72 | batch 1410/2304 [I 2022-10-27 14:38:40] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:14 [I 2022-10-27 14:38:40] ...trap/engines/engine.py.229: process: 0.54183 | load: 0.00243 [I 2022-10-27 14:38:40] ...trap/engines/engine.py.230: loss: 0.10929 [I 2022-10-27 14:38:40] ...optimizers/trijoint.py.151: total_norm: 7.862811 [I 2022-10-27 14:38:46] ...trap/engines/engine.py.224: train: epoch 72 | batch 1420/2304 [I 2022-10-27 14:38:46] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:18 [I 2022-10-27 14:38:46] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00246 [I 2022-10-27 14:38:46] ...trap/engines/engine.py.230: loss: 0.24834 [I 2022-10-27 14:38:46] ...optimizers/trijoint.py.151: total_norm: 27.970272 [I 2022-10-27 14:38:51] ...trap/engines/engine.py.224: train: epoch 72 | batch 1430/2304 [I 2022-10-27 14:38:51] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:06 [I 2022-10-27 14:38:51] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00267 [I 2022-10-27 14:38:51] ...trap/engines/engine.py.230: loss: 0.10177 [I 2022-10-27 14:38:51] ...optimizers/trijoint.py.151: total_norm: 13.612914 [I 2022-10-27 14:38:57] ...trap/engines/engine.py.224: train: epoch 72 | batch 1440/2304 [I 2022-10-27 14:38:57] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:02 [I 2022-10-27 14:38:57] ...trap/engines/engine.py.229: process: 0.56423 | load: 0.00251 [I 2022-10-27 14:38:57] ...trap/engines/engine.py.230: loss: 0.10680 [I 2022-10-27 14:38:57] ...optimizers/trijoint.py.151: total_norm: 22.913584 [I 2022-10-27 14:39:02] ...trap/engines/engine.py.224: train: epoch 72 | batch 1450/2304 [I 2022-10-27 14:39:02] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:56 [I 2022-10-27 14:39:02] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00247 [I 2022-10-27 14:39:02] ...trap/engines/engine.py.230: loss: 0.18044 [I 2022-10-27 14:39:02] ...optimizers/trijoint.py.151: total_norm: 16.873674 [I 2022-10-27 14:39:08] ...trap/engines/engine.py.224: train: epoch 72 | batch 1460/2304 [I 2022-10-27 14:39:08] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:54 [I 2022-10-27 14:39:08] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00249 [I 2022-10-27 14:39:08] ...trap/engines/engine.py.230: loss: 0.11863 [I 2022-10-27 14:39:08] ...optimizers/trijoint.py.151: total_norm: 9.152528 [I 2022-10-27 14:39:14] ...trap/engines/engine.py.224: train: epoch 72 | batch 1470/2304 [I 2022-10-27 14:39:14] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:43 [I 2022-10-27 14:39:14] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00247 [I 2022-10-27 14:39:14] ...trap/engines/engine.py.230: loss: 0.14528 [I 2022-10-27 14:39:14] ...optimizers/trijoint.py.151: total_norm: 12.988496 [I 2022-10-27 14:39:19] ...trap/engines/engine.py.224: train: epoch 72 | batch 1480/2304 [I 2022-10-27 14:39:19] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:59 [I 2022-10-27 14:39:19] ...trap/engines/engine.py.229: process: 0.68943 | load: 0.00247 [I 2022-10-27 14:39:19] ...trap/engines/engine.py.230: loss: 0.18457 [I 2022-10-27 14:39:19] ...optimizers/trijoint.py.151: total_norm: 25.254850 [I 2022-10-27 14:39:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 1490/2304 [I 2022-10-27 14:39:25] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:35 [I 2022-10-27 14:39:25] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00243 [I 2022-10-27 14:39:25] ...trap/engines/engine.py.230: loss: 0.12026 [I 2022-10-27 14:39:25] ...optimizers/trijoint.py.151: total_norm: 15.392705 [I 2022-10-27 14:39:30] ...trap/engines/engine.py.224: train: epoch 72 | batch 1500/2304 [I 2022-10-27 14:39:30] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:28 [I 2022-10-27 14:39:30] ...trap/engines/engine.py.229: process: 0.56568 | load: 0.00241 [I 2022-10-27 14:39:30] ...trap/engines/engine.py.230: loss: 0.11298 [I 2022-10-27 14:39:30] ...optimizers/trijoint.py.151: total_norm: 17.009439 [I 2022-10-27 14:39:36] ...trap/engines/engine.py.224: train: epoch 72 | batch 1510/2304 [I 2022-10-27 14:39:36] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:23 [I 2022-10-27 14:39:36] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00244 [I 2022-10-27 14:39:36] ...trap/engines/engine.py.230: loss: 0.16123 [I 2022-10-27 14:39:36] ...optimizers/trijoint.py.151: total_norm: 20.169195 [I 2022-10-27 14:39:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 1520/2304 [I 2022-10-27 14:39:42] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:17 [I 2022-10-27 14:39:42] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00243 [I 2022-10-27 14:39:42] ...trap/engines/engine.py.230: loss: 0.16678 [I 2022-10-27 14:39:42] ...optimizers/trijoint.py.151: total_norm: 51.002010 [I 2022-10-27 14:39:47] ...trap/engines/engine.py.224: train: epoch 72 | batch 1530/2304 [I 2022-10-27 14:39:47] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:11 [I 2022-10-27 14:39:47] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00244 [I 2022-10-27 14:39:47] ...trap/engines/engine.py.230: loss: 0.11359 [I 2022-10-27 14:39:47] ...optimizers/trijoint.py.151: total_norm: 9.542472 [I 2022-10-27 14:39:53] ...trap/engines/engine.py.224: train: epoch 72 | batch 1540/2304 [I 2022-10-27 14:39:53] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:05 [I 2022-10-27 14:39:53] ...trap/engines/engine.py.229: process: 0.56164 | load: 0.00243 [I 2022-10-27 14:39:53] ...trap/engines/engine.py.230: loss: 0.15354 [I 2022-10-27 14:39:53] ...optimizers/trijoint.py.151: total_norm: 24.364061 [I 2022-10-27 14:39:58] ...trap/engines/engine.py.224: train: epoch 72 | batch 1550/2304 [I 2022-10-27 14:39:58] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:02 [I 2022-10-27 14:39:58] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00244 [I 2022-10-27 14:39:58] ...trap/engines/engine.py.230: loss: 0.11851 [I 2022-10-27 14:39:58] ...optimizers/trijoint.py.151: total_norm: 22.265734 [I 2022-10-27 14:40:04] ...trap/engines/engine.py.224: train: epoch 72 | batch 1560/2304 [I 2022-10-27 14:40:04] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:56 [I 2022-10-27 14:40:04] ...trap/engines/engine.py.229: process: 0.56162 | load: 0.00284 [I 2022-10-27 14:40:04] ...trap/engines/engine.py.230: loss: 0.15625 [I 2022-10-27 14:40:04] ...optimizers/trijoint.py.151: total_norm: 14.907370 [I 2022-10-27 14:40:10] ...trap/engines/engine.py.224: train: epoch 72 | batch 1570/2304 [I 2022-10-27 14:40:10] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:50 [I 2022-10-27 14:40:10] ...trap/engines/engine.py.229: process: 0.55544 | load: 0.00248 [I 2022-10-27 14:40:10] ...trap/engines/engine.py.230: loss: 0.10892 [I 2022-10-27 14:40:10] ...optimizers/trijoint.py.151: total_norm: 14.227511 [I 2022-10-27 14:40:15] ...trap/engines/engine.py.224: train: epoch 72 | batch 1580/2304 [I 2022-10-27 14:40:15] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:46 [I 2022-10-27 14:40:15] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00245 [I 2022-10-27 14:40:15] ...trap/engines/engine.py.230: loss: 0.09398 [I 2022-10-27 14:40:15] ...optimizers/trijoint.py.151: total_norm: 10.900229 [I 2022-10-27 14:40:21] ...trap/engines/engine.py.224: train: epoch 72 | batch 1590/2304 [I 2022-10-27 14:40:21] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:40 [I 2022-10-27 14:40:21] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00247 [I 2022-10-27 14:40:21] ...trap/engines/engine.py.230: loss: 0.08737 [I 2022-10-27 14:40:21] ...optimizers/trijoint.py.151: total_norm: 8.744741 [I 2022-10-27 14:40:27] ...trap/engines/engine.py.224: train: epoch 72 | batch 1600/2304 [I 2022-10-27 14:40:27] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:34 [I 2022-10-27 14:40:27] ...trap/engines/engine.py.229: process: 0.55496 | load: 0.00244 [I 2022-10-27 14:40:27] ...trap/engines/engine.py.230: loss: 0.18193 [I 2022-10-27 14:40:27] ...optimizers/trijoint.py.151: total_norm: 27.976370 [I 2022-10-27 14:40:32] ...trap/engines/engine.py.224: train: epoch 72 | batch 1610/2304 [I 2022-10-27 14:40:32] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:29 [I 2022-10-27 14:40:32] ...trap/engines/engine.py.229: process: 0.55072 | load: 0.00244 [I 2022-10-27 14:40:32] ...trap/engines/engine.py.230: loss: 0.13765 [I 2022-10-27 14:40:32] ...optimizers/trijoint.py.151: total_norm: 19.435791 [I 2022-10-27 14:40:38] ...trap/engines/engine.py.224: train: epoch 72 | batch 1620/2304 [I 2022-10-27 14:40:38] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:23 [I 2022-10-27 14:40:38] ...trap/engines/engine.py.229: process: 0.55248 | load: 0.00241 [I 2022-10-27 14:40:38] ...trap/engines/engine.py.230: loss: 0.13225 [I 2022-10-27 14:40:38] ...optimizers/trijoint.py.151: total_norm: 20.729420 [I 2022-10-27 14:40:43] ...trap/engines/engine.py.224: train: epoch 72 | batch 1630/2304 [I 2022-10-27 14:40:43] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:14 [I 2022-10-27 14:40:43] ...trap/engines/engine.py.229: process: 0.55092 | load: 0.00257 [I 2022-10-27 14:40:43] ...trap/engines/engine.py.230: loss: 0.13380 [I 2022-10-27 14:40:43] ...optimizers/trijoint.py.151: total_norm: 20.605825 [I 2022-10-27 14:40:49] ...trap/engines/engine.py.224: train: epoch 72 | batch 1640/2304 [I 2022-10-27 14:40:49] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:12 [I 2022-10-27 14:40:49] ...trap/engines/engine.py.229: process: 0.55218 | load: 0.00253 [I 2022-10-27 14:40:49] ...trap/engines/engine.py.230: loss: 0.11504 [I 2022-10-27 14:40:49] ...optimizers/trijoint.py.151: total_norm: 10.367787 [I 2022-10-27 14:40:55] ...trap/engines/engine.py.224: train: epoch 72 | batch 1650/2304 [I 2022-10-27 14:40:55] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:06 [I 2022-10-27 14:40:55] ...trap/engines/engine.py.229: process: 0.56624 | load: 0.00248 [I 2022-10-27 14:40:55] ...trap/engines/engine.py.230: loss: 0.19345 [I 2022-10-27 14:40:55] ...optimizers/trijoint.py.151: total_norm: 20.078272 [I 2022-10-27 14:41:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 1660/2304 [I 2022-10-27 14:41:00] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:59 [I 2022-10-27 14:41:00] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00247 [I 2022-10-27 14:41:00] ...trap/engines/engine.py.230: loss: 0.10829 [I 2022-10-27 14:41:00] ...optimizers/trijoint.py.151: total_norm: 11.929397 [I 2022-10-27 14:41:06] ...trap/engines/engine.py.224: train: epoch 72 | batch 1670/2304 [I 2022-10-27 14:41:06] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:06:01 [I 2022-10-27 14:41:06] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00248 [I 2022-10-27 14:41:06] ...trap/engines/engine.py.230: loss: 0.11000 [I 2022-10-27 14:41:06] ...optimizers/trijoint.py.151: total_norm: 10.320164 [I 2022-10-27 14:41:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 1680/2304 [I 2022-10-27 14:41:11] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:48 [I 2022-10-27 14:41:11] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00244 [I 2022-10-27 14:41:11] ...trap/engines/engine.py.230: loss: 0.11511 [I 2022-10-27 14:41:11] ...optimizers/trijoint.py.151: total_norm: 17.029669 [I 2022-10-27 14:41:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 1690/2304 [I 2022-10-27 14:41:17] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:38 [I 2022-10-27 14:41:17] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00248 [I 2022-10-27 14:41:17] ...trap/engines/engine.py.230: loss: 0.13112 [I 2022-10-27 14:41:17] ...optimizers/trijoint.py.151: total_norm: 28.918709 [I 2022-10-27 14:41:23] ...trap/engines/engine.py.224: train: epoch 72 | batch 1700/2304 [I 2022-10-27 14:41:23] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:36 [I 2022-10-27 14:41:23] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00246 [I 2022-10-27 14:41:23] ...trap/engines/engine.py.230: loss: 0.11789 [I 2022-10-27 14:41:23] ...optimizers/trijoint.py.151: total_norm: 13.326840 [I 2022-10-27 14:41:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 1710/2304 [I 2022-10-27 14:41:28] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:29 [I 2022-10-27 14:41:28] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00246 [I 2022-10-27 14:41:28] ...trap/engines/engine.py.230: loss: 0.07902 [I 2022-10-27 14:41:28] ...optimizers/trijoint.py.151: total_norm: 10.253637 [I 2022-10-27 14:41:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 1720/2304 [I 2022-10-27 14:41:34] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:25 [I 2022-10-27 14:41:34] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00243 [I 2022-10-27 14:41:34] ...trap/engines/engine.py.230: loss: 0.11925 [I 2022-10-27 14:41:34] ...optimizers/trijoint.py.151: total_norm: 12.013190 [I 2022-10-27 14:41:39] ...trap/engines/engine.py.224: train: epoch 72 | batch 1730/2304 [I 2022-10-27 14:41:39] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:20 [I 2022-10-27 14:41:39] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00241 [I 2022-10-27 14:41:39] ...trap/engines/engine.py.230: loss: 0.12056 [I 2022-10-27 14:41:39] ...optimizers/trijoint.py.151: total_norm: 14.983516 [I 2022-10-27 14:41:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 1740/2304 [I 2022-10-27 14:41:45] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:11 [I 2022-10-27 14:41:45] ...trap/engines/engine.py.229: process: 0.54375 | load: 0.00246 [I 2022-10-27 14:41:45] ...trap/engines/engine.py.230: loss: 0.10342 [I 2022-10-27 14:41:45] ...optimizers/trijoint.py.151: total_norm: 20.417864 [I 2022-10-27 14:41:50] ...trap/engines/engine.py.224: train: epoch 72 | batch 1750/2304 [I 2022-10-27 14:41:50] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:08 [I 2022-10-27 14:41:50] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00242 [I 2022-10-27 14:41:50] ...trap/engines/engine.py.230: loss: 0.12307 [I 2022-10-27 14:41:50] ...optimizers/trijoint.py.151: total_norm: 21.118425 [I 2022-10-27 14:41:56] ...trap/engines/engine.py.224: train: epoch 72 | batch 1760/2304 [I 2022-10-27 14:41:56] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:03 [I 2022-10-27 14:41:56] ...trap/engines/engine.py.229: process: 0.55185 | load: 0.00246 [I 2022-10-27 14:41:56] ...trap/engines/engine.py.230: loss: 0.19944 [I 2022-10-27 14:41:56] ...optimizers/trijoint.py.151: total_norm: 23.470472 [I 2022-10-27 14:42:02] ...trap/engines/engine.py.224: train: epoch 72 | batch 1770/2304 [I 2022-10-27 14:42:02] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:04:56 [I 2022-10-27 14:42:02] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00245 [I 2022-10-27 14:42:02] ...trap/engines/engine.py.230: loss: 0.18527 [I 2022-10-27 14:42:02] ...optimizers/trijoint.py.151: total_norm: 26.900709 [I 2022-10-27 14:42:07] ...trap/engines/engine.py.224: train: epoch 72 | batch 1780/2304 [I 2022-10-27 14:42:07] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:50 [I 2022-10-27 14:42:07] ...trap/engines/engine.py.229: process: 0.54313 | load: 0.00248 [I 2022-10-27 14:42:07] ...trap/engines/engine.py.230: loss: 0.17443 [I 2022-10-27 14:42:07] ...optimizers/trijoint.py.151: total_norm: 11.781280 [I 2022-10-27 14:42:13] ...trap/engines/engine.py.224: train: epoch 72 | batch 1790/2304 [I 2022-10-27 14:42:13] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:45 [I 2022-10-27 14:42:13] ...trap/engines/engine.py.229: process: 0.55835 | load: 0.00240 [I 2022-10-27 14:42:13] ...trap/engines/engine.py.230: loss: 0.11044 [I 2022-10-27 14:42:13] ...optimizers/trijoint.py.151: total_norm: 11.921457 [I 2022-10-27 14:42:18] ...trap/engines/engine.py.224: train: epoch 72 | batch 1800/2304 [I 2022-10-27 14:42:18] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:43 [I 2022-10-27 14:42:18] ...trap/engines/engine.py.229: process: 0.56854 | load: 0.00243 [I 2022-10-27 14:42:18] ...trap/engines/engine.py.230: loss: 0.12901 [I 2022-10-27 14:42:18] ...optimizers/trijoint.py.151: total_norm: 17.536493 [I 2022-10-27 14:42:24] ...trap/engines/engine.py.224: train: epoch 72 | batch 1810/2304 [I 2022-10-27 14:42:24] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:35 [I 2022-10-27 14:42:24] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00252 [I 2022-10-27 14:42:24] ...trap/engines/engine.py.230: loss: 0.20300 [I 2022-10-27 14:42:24] ...optimizers/trijoint.py.151: total_norm: 17.232067 [I 2022-10-27 14:42:30] ...trap/engines/engine.py.224: train: epoch 72 | batch 1820/2304 [I 2022-10-27 14:42:30] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:32 [I 2022-10-27 14:42:30] ...trap/engines/engine.py.229: process: 0.56543 | load: 0.00271 [I 2022-10-27 14:42:30] ...trap/engines/engine.py.230: loss: 0.12910 [I 2022-10-27 14:42:30] ...optimizers/trijoint.py.151: total_norm: 16.825541 [I 2022-10-27 14:42:35] ...trap/engines/engine.py.224: train: epoch 72 | batch 1830/2304 [I 2022-10-27 14:42:35] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:25 [I 2022-10-27 14:42:35] ...trap/engines/engine.py.229: process: 0.55642 | load: 0.00246 [I 2022-10-27 14:42:35] ...trap/engines/engine.py.230: loss: 0.14213 [I 2022-10-27 14:42:35] ...optimizers/trijoint.py.151: total_norm: 13.943599 [I 2022-10-27 14:42:41] ...trap/engines/engine.py.224: train: epoch 72 | batch 1840/2304 [I 2022-10-27 14:42:41] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:19 [I 2022-10-27 14:42:41] ...trap/engines/engine.py.229: process: 0.54045 | load: 0.00247 [I 2022-10-27 14:42:41] ...trap/engines/engine.py.230: loss: 0.17468 [I 2022-10-27 14:42:41] ...optimizers/trijoint.py.151: total_norm: 19.275503 [I 2022-10-27 14:42:46] ...trap/engines/engine.py.224: train: epoch 72 | batch 1850/2304 [I 2022-10-27 14:42:46] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:11 [I 2022-10-27 14:42:46] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00243 [I 2022-10-27 14:42:46] ...trap/engines/engine.py.230: loss: 0.10991 [I 2022-10-27 14:42:46] ...optimizers/trijoint.py.151: total_norm: 8.873712 [I 2022-10-27 14:42:52] ...trap/engines/engine.py.224: train: epoch 72 | batch 1860/2304 [I 2022-10-27 14:42:52] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:05 [I 2022-10-27 14:42:52] ...trap/engines/engine.py.229: process: 0.54842 | load: 0.00242 [I 2022-10-27 14:42:52] ...trap/engines/engine.py.230: loss: 0.13688 [I 2022-10-27 14:42:52] ...optimizers/trijoint.py.151: total_norm: 10.339245 [I 2022-10-27 14:42:57] ...trap/engines/engine.py.224: train: epoch 72 | batch 1870/2304 [I 2022-10-27 14:42:57] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:03:59 [I 2022-10-27 14:42:57] ...trap/engines/engine.py.229: process: 0.55040 | load: 0.00244 [I 2022-10-27 14:42:57] ...trap/engines/engine.py.230: loss: 0.21903 [I 2022-10-27 14:42:57] ...optimizers/trijoint.py.151: total_norm: 29.086506 [I 2022-10-27 14:43:03] ...trap/engines/engine.py.224: train: epoch 72 | batch 1880/2304 [I 2022-10-27 14:43:03] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:56 [I 2022-10-27 14:43:03] ...trap/engines/engine.py.229: process: 0.56272 | load: 0.00244 [I 2022-10-27 14:43:03] ...trap/engines/engine.py.230: loss: 0.16166 [I 2022-10-27 14:43:03] ...optimizers/trijoint.py.151: total_norm: 59.941643 [I 2022-10-27 14:43:09] ...trap/engines/engine.py.224: train: epoch 72 | batch 1890/2304 [I 2022-10-27 14:43:09] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:03:49 [I 2022-10-27 14:43:09] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00241 [I 2022-10-27 14:43:09] ...trap/engines/engine.py.230: loss: 0.14699 [I 2022-10-27 14:43:09] ...optimizers/trijoint.py.151: total_norm: 29.866468 [I 2022-10-27 14:43:14] ...trap/engines/engine.py.224: train: epoch 72 | batch 1900/2304 [I 2022-10-27 14:43:14] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:45 [I 2022-10-27 14:43:14] ...trap/engines/engine.py.229: process: 0.56260 | load: 0.00249 [I 2022-10-27 14:43:14] ...trap/engines/engine.py.230: loss: 0.14681 [I 2022-10-27 14:43:14] ...optimizers/trijoint.py.151: total_norm: 17.731228 [I 2022-10-27 14:43:20] ...trap/engines/engine.py.224: train: epoch 72 | batch 1910/2304 [I 2022-10-27 14:43:20] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:39 [I 2022-10-27 14:43:20] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00258 [I 2022-10-27 14:43:20] ...trap/engines/engine.py.230: loss: 0.11995 [I 2022-10-27 14:43:20] ...optimizers/trijoint.py.151: total_norm: 10.096832 [I 2022-10-27 14:43:25] ...trap/engines/engine.py.224: train: epoch 72 | batch 1920/2304 [I 2022-10-27 14:43:25] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:34 [I 2022-10-27 14:43:25] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00250 [I 2022-10-27 14:43:25] ...trap/engines/engine.py.230: loss: 0.09878 [I 2022-10-27 14:43:25] ...optimizers/trijoint.py.151: total_norm: 9.364141 [I 2022-10-27 14:43:31] ...trap/engines/engine.py.224: train: epoch 72 | batch 1930/2304 [I 2022-10-27 14:43:31] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:27 [I 2022-10-27 14:43:31] ...trap/engines/engine.py.229: process: 0.55491 | load: 0.00243 [I 2022-10-27 14:43:31] ...trap/engines/engine.py.230: loss: 0.12398 [I 2022-10-27 14:43:31] ...optimizers/trijoint.py.151: total_norm: 20.189606 [I 2022-10-27 14:43:36] ...trap/engines/engine.py.224: train: epoch 72 | batch 1940/2304 [I 2022-10-27 14:43:36] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:23 [I 2022-10-27 14:43:36] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00247 [I 2022-10-27 14:43:36] ...trap/engines/engine.py.230: loss: 0.11105 [I 2022-10-27 14:43:36] ...optimizers/trijoint.py.151: total_norm: 16.535460 [I 2022-10-27 14:43:42] ...trap/engines/engine.py.224: train: epoch 72 | batch 1950/2304 [I 2022-10-27 14:43:42] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:17 [I 2022-10-27 14:43:42] ...trap/engines/engine.py.229: process: 0.54592 | load: 0.00245 [I 2022-10-27 14:43:42] ...trap/engines/engine.py.230: loss: 0.10703 [I 2022-10-27 14:43:42] ...optimizers/trijoint.py.151: total_norm: 14.634973 [I 2022-10-27 14:43:48] ...trap/engines/engine.py.224: train: epoch 72 | batch 1960/2304 [I 2022-10-27 14:43:48] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:11 [I 2022-10-27 14:43:48] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00245 [I 2022-10-27 14:43:48] ...trap/engines/engine.py.230: loss: 0.12969 [I 2022-10-27 14:43:48] ...optimizers/trijoint.py.151: total_norm: 25.449492 [I 2022-10-27 14:43:53] ...trap/engines/engine.py.224: train: epoch 72 | batch 1970/2304 [I 2022-10-27 14:43:53] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:05 [I 2022-10-27 14:43:53] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00245 [I 2022-10-27 14:43:53] ...trap/engines/engine.py.230: loss: 0.16302 [I 2022-10-27 14:43:53] ...optimizers/trijoint.py.151: total_norm: 10.725724 [I 2022-10-27 14:43:59] ...trap/engines/engine.py.224: train: epoch 72 | batch 1980/2304 [I 2022-10-27 14:43:59] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:02 [I 2022-10-27 14:43:59] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00254 [I 2022-10-27 14:43:59] ...trap/engines/engine.py.230: loss: 0.12054 [I 2022-10-27 14:43:59] ...optimizers/trijoint.py.151: total_norm: 19.147547 [I 2022-10-27 14:44:04] ...trap/engines/engine.py.224: train: epoch 72 | batch 1990/2304 [I 2022-10-27 14:44:04] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:55 [I 2022-10-27 14:44:04] ...trap/engines/engine.py.229: process: 0.57048 | load: 0.00248 [I 2022-10-27 14:44:04] ...trap/engines/engine.py.230: loss: 0.09954 [I 2022-10-27 14:44:04] ...optimizers/trijoint.py.151: total_norm: 17.843172 [I 2022-10-27 14:44:10] ...trap/engines/engine.py.224: train: epoch 72 | batch 2000/2304 [I 2022-10-27 14:44:10] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:02:49 [I 2022-10-27 14:44:10] ...trap/engines/engine.py.229: process: 0.56924 | load: 0.00253 [I 2022-10-27 14:44:10] ...trap/engines/engine.py.230: loss: 0.11371 [I 2022-10-27 14:44:10] ...optimizers/trijoint.py.151: total_norm: 14.899680 [I 2022-10-27 14:44:16] ...trap/engines/engine.py.224: train: epoch 72 | batch 2010/2304 [I 2022-10-27 14:44:16] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:02:43 [I 2022-10-27 14:44:16] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00243 [I 2022-10-27 14:44:16] ...trap/engines/engine.py.230: loss: 0.14247 [I 2022-10-27 14:44:16] ...optimizers/trijoint.py.151: total_norm: 27.775372 [I 2022-10-27 14:44:21] ...trap/engines/engine.py.224: train: epoch 72 | batch 2020/2304 [I 2022-10-27 14:44:21] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:38 [I 2022-10-27 14:44:21] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00248 [I 2022-10-27 14:44:21] ...trap/engines/engine.py.230: loss: 0.15160 [I 2022-10-27 14:44:21] ...optimizers/trijoint.py.151: total_norm: 24.215075 [I 2022-10-27 14:44:27] ...trap/engines/engine.py.224: train: epoch 72 | batch 2030/2304 [I 2022-10-27 14:44:27] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:32 [I 2022-10-27 14:44:27] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00243 [I 2022-10-27 14:44:27] ...trap/engines/engine.py.230: loss: 0.11740 [I 2022-10-27 14:44:27] ...optimizers/trijoint.py.151: total_norm: 11.466443 [I 2022-10-27 14:44:32] ...trap/engines/engine.py.224: train: epoch 72 | batch 2040/2304 [I 2022-10-27 14:44:32] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:27 [I 2022-10-27 14:44:32] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00249 [I 2022-10-27 14:44:32] ...trap/engines/engine.py.230: loss: 0.20252 [I 2022-10-27 14:44:32] ...optimizers/trijoint.py.151: total_norm: 21.526041 [I 2022-10-27 14:44:38] ...trap/engines/engine.py.224: train: epoch 72 | batch 2050/2304 [I 2022-10-27 14:44:38] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:21 [I 2022-10-27 14:44:38] ...trap/engines/engine.py.229: process: 0.54930 | load: 0.00246 [I 2022-10-27 14:44:38] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-27 14:44:38] ...optimizers/trijoint.py.151: total_norm: 23.027405 [I 2022-10-27 14:44:43] ...trap/engines/engine.py.224: train: epoch 72 | batch 2060/2304 [I 2022-10-27 14:44:43] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:16 [I 2022-10-27 14:44:43] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00250 [I 2022-10-27 14:44:43] ...trap/engines/engine.py.230: loss: 0.11341 [I 2022-10-27 14:44:43] ...optimizers/trijoint.py.151: total_norm: 9.146138 [I 2022-10-27 14:44:49] ...trap/engines/engine.py.224: train: epoch 72 | batch 2070/2304 [I 2022-10-27 14:44:49] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:11 [I 2022-10-27 14:44:49] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00240 [I 2022-10-27 14:44:49] ...trap/engines/engine.py.230: loss: 0.11105 [I 2022-10-27 14:44:49] ...optimizers/trijoint.py.151: total_norm: 12.337070 [I 2022-10-27 14:44:55] ...trap/engines/engine.py.224: train: epoch 72 | batch 2080/2304 [I 2022-10-27 14:44:55] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:05 [I 2022-10-27 14:44:55] ...trap/engines/engine.py.229: process: 0.56837 | load: 0.00243 [I 2022-10-27 14:44:55] ...trap/engines/engine.py.230: loss: 0.17456 [I 2022-10-27 14:44:55] ...optimizers/trijoint.py.151: total_norm: 35.171200 [I 2022-10-27 14:45:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 2090/2304 [I 2022-10-27 14:45:00] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:01:58 [I 2022-10-27 14:45:00] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00253 [I 2022-10-27 14:45:00] ...trap/engines/engine.py.230: loss: 0.10770 [I 2022-10-27 14:45:00] ...optimizers/trijoint.py.151: total_norm: 11.513594 [I 2022-10-27 14:45:06] ...trap/engines/engine.py.224: train: epoch 72 | batch 2100/2304 [I 2022-10-27 14:45:06] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:53 [I 2022-10-27 14:45:06] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00249 [I 2022-10-27 14:45:06] ...trap/engines/engine.py.230: loss: 0.11744 [I 2022-10-27 14:45:06] ...optimizers/trijoint.py.151: total_norm: 13.106810 [I 2022-10-27 14:45:11] ...trap/engines/engine.py.224: train: epoch 72 | batch 2110/2304 [I 2022-10-27 14:45:11] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:01:49 [I 2022-10-27 14:45:11] ...trap/engines/engine.py.229: process: 0.57172 | load: 0.00250 [I 2022-10-27 14:45:11] ...trap/engines/engine.py.230: loss: 0.15874 [I 2022-10-27 14:45:11] ...optimizers/trijoint.py.151: total_norm: 13.982237 [I 2022-10-27 14:45:17] ...trap/engines/engine.py.224: train: epoch 72 | batch 2120/2304 [I 2022-10-27 14:45:17] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:42 [I 2022-10-27 14:45:17] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00248 [I 2022-10-27 14:45:17] ...trap/engines/engine.py.230: loss: 0.13600 [I 2022-10-27 14:45:17] ...optimizers/trijoint.py.151: total_norm: 9.937192 [I 2022-10-27 14:45:23] ...trap/engines/engine.py.224: train: epoch 72 | batch 2130/2304 [I 2022-10-27 14:45:23] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:37 [I 2022-10-27 14:45:23] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00249 [I 2022-10-27 14:45:23] ...trap/engines/engine.py.230: loss: 0.12824 [I 2022-10-27 14:45:23] ...optimizers/trijoint.py.151: total_norm: 14.402916 [I 2022-10-27 14:45:28] ...trap/engines/engine.py.224: train: epoch 72 | batch 2140/2304 [I 2022-10-27 14:45:28] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:31 [I 2022-10-27 14:45:28] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00245 [I 2022-10-27 14:45:28] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-27 14:45:28] ...optimizers/trijoint.py.151: total_norm: 11.270130 [I 2022-10-27 14:45:34] ...trap/engines/engine.py.224: train: epoch 72 | batch 2150/2304 [I 2022-10-27 14:45:34] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:25 [I 2022-10-27 14:45:34] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00241 [I 2022-10-27 14:45:34] ...trap/engines/engine.py.230: loss: 0.16854 [I 2022-10-27 14:45:34] ...optimizers/trijoint.py.151: total_norm: 10.106874 [I 2022-10-27 14:45:39] ...trap/engines/engine.py.224: train: epoch 72 | batch 2160/2304 [I 2022-10-27 14:45:39] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:20 [I 2022-10-27 14:45:39] ...trap/engines/engine.py.229: process: 0.56636 | load: 0.00243 [I 2022-10-27 14:45:39] ...trap/engines/engine.py.230: loss: 0.18636 [I 2022-10-27 14:45:39] ...optimizers/trijoint.py.151: total_norm: 84.269852 [I 2022-10-27 14:45:45] ...trap/engines/engine.py.224: train: epoch 72 | batch 2170/2304 [I 2022-10-27 14:45:45] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:15 [I 2022-10-27 14:45:45] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00249 [I 2022-10-27 14:45:45] ...trap/engines/engine.py.230: loss: 0.12666 [I 2022-10-27 14:45:45] ...optimizers/trijoint.py.151: total_norm: 15.170041 [I 2022-10-27 14:45:50] ...trap/engines/engine.py.224: train: epoch 72 | batch 2180/2304 [I 2022-10-27 14:45:50] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:08 [I 2022-10-27 14:45:50] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00252 [I 2022-10-27 14:45:50] ...trap/engines/engine.py.230: loss: 0.20793 [I 2022-10-27 14:45:50] ...optimizers/trijoint.py.151: total_norm: 26.489721 [I 2022-10-27 14:45:56] ...trap/engines/engine.py.224: train: epoch 72 | batch 2190/2304 [I 2022-10-27 14:45:56] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:03 [I 2022-10-27 14:45:56] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00245 [I 2022-10-27 14:45:56] ...trap/engines/engine.py.230: loss: 0.12333 [I 2022-10-27 14:45:56] ...optimizers/trijoint.py.151: total_norm: 8.508574 [I 2022-10-27 14:46:02] ...trap/engines/engine.py.224: train: epoch 72 | batch 2200/2304 [I 2022-10-27 14:46:02] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:58 [I 2022-10-27 14:46:02] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00244 [I 2022-10-27 14:46:02] ...trap/engines/engine.py.230: loss: 0.11270 [I 2022-10-27 14:46:02] ...optimizers/trijoint.py.151: total_norm: 10.954340 [I 2022-10-27 14:46:07] ...trap/engines/engine.py.224: train: epoch 72 | batch 2210/2304 [I 2022-10-27 14:46:07] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:00:52 [I 2022-10-27 14:46:07] ...trap/engines/engine.py.229: process: 0.55471 | load: 0.00246 [I 2022-10-27 14:46:07] ...trap/engines/engine.py.230: loss: 0.10325 [I 2022-10-27 14:46:07] ...optimizers/trijoint.py.151: total_norm: 9.568426 [I 2022-10-27 14:46:13] ...trap/engines/engine.py.224: train: epoch 72 | batch 2220/2304 [I 2022-10-27 14:46:13] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:00:47 [I 2022-10-27 14:46:13] ...trap/engines/engine.py.229: process: 0.56207 | load: 0.00244 [I 2022-10-27 14:46:13] ...trap/engines/engine.py.230: loss: 0.17956 [I 2022-10-27 14:46:13] ...optimizers/trijoint.py.151: total_norm: 27.964489 [I 2022-10-27 14:46:18] ...trap/engines/engine.py.224: train: epoch 72 | batch 2230/2304 [I 2022-10-27 14:46:18] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:41 [I 2022-10-27 14:46:18] ...trap/engines/engine.py.229: process: 0.54703 | load: 0.00242 [I 2022-10-27 14:46:18] ...trap/engines/engine.py.230: loss: 0.11548 [I 2022-10-27 14:46:18] ...optimizers/trijoint.py.151: total_norm: 18.810810 [I 2022-10-27 14:46:24] ...trap/engines/engine.py.224: train: epoch 72 | batch 2240/2304 [I 2022-10-27 14:46:24] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:35 [I 2022-10-27 14:46:24] ...trap/engines/engine.py.229: process: 0.54528 | load: 0.00245 [I 2022-10-27 14:46:24] ...trap/engines/engine.py.230: loss: 0.10411 [I 2022-10-27 14:46:24] ...optimizers/trijoint.py.151: total_norm: 8.890726 [I 2022-10-27 14:46:30] ...trap/engines/engine.py.224: train: epoch 72 | batch 2250/2304 [I 2022-10-27 14:46:30] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:30 [I 2022-10-27 14:46:30] ...trap/engines/engine.py.229: process: 0.56436 | load: 0.00240 [I 2022-10-27 14:46:30] ...trap/engines/engine.py.230: loss: 0.10574 [I 2022-10-27 14:46:30] ...optimizers/trijoint.py.151: total_norm: 14.719146 [I 2022-10-27 14:46:35] ...trap/engines/engine.py.224: train: epoch 72 | batch 2260/2304 [I 2022-10-27 14:46:35] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:24 [I 2022-10-27 14:46:35] ...trap/engines/engine.py.229: process: 0.54882 | load: 0.00241 [I 2022-10-27 14:46:35] ...trap/engines/engine.py.230: loss: 0.11017 [I 2022-10-27 14:46:35] ...optimizers/trijoint.py.151: total_norm: 13.524283 [I 2022-10-27 14:46:41] ...trap/engines/engine.py.224: train: epoch 72 | batch 2270/2304 [I 2022-10-27 14:46:41] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:18 [I 2022-10-27 14:46:41] ...trap/engines/engine.py.229: process: 0.54390 | load: 0.00246 [I 2022-10-27 14:46:41] ...trap/engines/engine.py.230: loss: 0.12540 [I 2022-10-27 14:46:41] ...optimizers/trijoint.py.151: total_norm: 22.227316 [I 2022-10-27 14:46:47] ...trap/engines/engine.py.224: train: epoch 72 | batch 2280/2304 [I 2022-10-27 14:46:47] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:13 [I 2022-10-27 14:46:47] ...trap/engines/engine.py.229: process: 0.56972 | load: 0.00249 [I 2022-10-27 14:46:47] ...trap/engines/engine.py.230: loss: 0.09770 [I 2022-10-27 14:46:47] ...optimizers/trijoint.py.151: total_norm: 9.566143 [I 2022-10-27 14:46:52] ...trap/engines/engine.py.224: train: epoch 72 | batch 2290/2304 [I 2022-10-27 14:46:52] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:07 [I 2022-10-27 14:46:52] ...trap/engines/engine.py.229: process: 0.55230 | load: 0.00245 [I 2022-10-27 14:46:52] ...trap/engines/engine.py.230: loss: 0.08958 [I 2022-10-27 14:46:52] ...optimizers/trijoint.py.151: total_norm: 11.286597 [I 2022-10-27 14:46:58] ...trap/engines/engine.py.224: train: epoch 72 | batch 2300/2304 [I 2022-10-27 14:46:58] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:02 [I 2022-10-27 14:46:58] ...trap/engines/engine.py.229: process: 0.61533 | load: 0.00016 [I 2022-10-27 14:46:58] ...trap/engines/engine.py.230: loss: 0.19122 [I 2022-10-27 14:46:58] ...optimizers/trijoint.py.151: total_norm: 23.499273 [I 2022-10-27 14:47:00] ...trap/engines/engine.py.224: train: epoch 72 | batch 2304/2304 [I 2022-10-27 14:47:00] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:00 [I 2022-10-27 14:47:00] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00015 [I 2022-10-27 14:47:00] ...trap/engines/engine.py.230: loss: 0.15239 [I 2022-10-27 14:47:00] ...optimizers/trijoint.py.151: total_norm: 20.491568 [S 2022-10-27 14:47:01] ...trap/engines/engine.py.129: train_epoch.epoch: 72 [S 2022-10-27 14:47:01] ...trap/engines/engine.py.129: train_epoch.loss: 0.14280288077874712 [I 2022-10-27 14:47:01] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 14:47:02] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 72 [I 2022-10-27 14:47:02] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 14:47:04] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 14:47:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 14:47:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 14:47:07] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 14:47:07] ...trap/engines/engine.py.321: eval: epoch 72 | batch 0/510 [I 2022-10-27 14:47:07] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:39:49 [I 2022-10-27 14:47:07] ...trap/engines/engine.py.326: process: 4.68539 | load: 3.63555 [I 2022-10-27 14:47:10] ...trap/engines/engine.py.321: eval: epoch 72 | batch 10/510 [I 2022-10-27 14:47:10] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:41 [I 2022-10-27 14:47:10] ...trap/engines/engine.py.326: process: 0.32652 | load: 0.00020 [I 2022-10-27 14:47:14] ...trap/engines/engine.py.321: eval: epoch 72 | batch 20/510 [I 2022-10-27 14:47:14] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:11 [I 2022-10-27 14:47:14] ...trap/engines/engine.py.326: process: 0.32447 | load: 0.00018 [I 2022-10-27 14:47:17] ...trap/engines/engine.py.321: eval: epoch 72 | batch 30/510 [I 2022-10-27 14:47:17] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:45 [I 2022-10-27 14:47:17] ...trap/engines/engine.py.326: process: 0.33362 | load: 0.00018 [I 2022-10-27 14:47:21] ...trap/engines/engine.py.321: eval: epoch 72 | batch 40/510 [I 2022-10-27 14:47:21] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:36 [I 2022-10-27 14:47:21] ...trap/engines/engine.py.326: process: 0.33824 | load: 0.00017 [I 2022-10-27 14:47:24] ...trap/engines/engine.py.321: eval: epoch 72 | batch 50/510 [I 2022-10-27 14:47:24] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:34 [I 2022-10-27 14:47:24] ...trap/engines/engine.py.326: process: 0.33881 | load: 0.00041 [I 2022-10-27 14:47:27] ...trap/engines/engine.py.321: eval: epoch 72 | batch 60/510 [I 2022-10-27 14:47:27] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:30 [I 2022-10-27 14:47:27] ...trap/engines/engine.py.326: process: 0.33047 | load: 0.00023 [I 2022-10-27 14:47:31] ...trap/engines/engine.py.321: eval: epoch 72 | batch 70/510 [I 2022-10-27 14:47:31] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:29 [I 2022-10-27 14:47:31] ...trap/engines/engine.py.326: process: 0.32881 | load: 0.00022 [I 2022-10-27 14:47:34] ...trap/engines/engine.py.321: eval: epoch 72 | batch 80/510 [I 2022-10-27 14:47:34] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:26 [I 2022-10-27 14:47:34] ...trap/engines/engine.py.326: process: 0.32928 | load: 0.00020 [I 2022-10-27 14:47:38] ...trap/engines/engine.py.321: eval: epoch 72 | batch 90/510 [I 2022-10-27 14:47:38] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:25 [I 2022-10-27 14:47:38] ...trap/engines/engine.py.326: process: 0.36892 | load: 0.00019 [I 2022-10-27 14:47:41] ...trap/engines/engine.py.321: eval: epoch 72 | batch 100/510 [I 2022-10-27 14:47:41] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:15 [I 2022-10-27 14:47:41] ...trap/engines/engine.py.326: process: 0.28677 | load: 0.00020 [I 2022-10-27 14:47:44] ...trap/engines/engine.py.321: eval: epoch 72 | batch 110/510 [I 2022-10-27 14:47:44] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-27 14:47:44] ...trap/engines/engine.py.326: process: 0.28184 | load: 0.00015 [I 2022-10-27 14:47:47] ...trap/engines/engine.py.321: eval: epoch 72 | batch 120/510 [I 2022-10-27 14:47:47] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 14:47:47] ...trap/engines/engine.py.326: process: 0.28980 | load: 0.00022 [I 2022-10-27 14:47:50] ...trap/engines/engine.py.321: eval: epoch 72 | batch 130/510 [I 2022-10-27 14:47:50] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-27 14:47:50] ...trap/engines/engine.py.326: process: 0.28668 | load: 0.00021 [I 2022-10-27 14:47:52] ...trap/engines/engine.py.321: eval: epoch 72 | batch 140/510 [I 2022-10-27 14:47:52] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-27 14:47:52] ...trap/engines/engine.py.326: process: 0.29019 | load: 0.00015 [I 2022-10-27 14:47:55] ...trap/engines/engine.py.321: eval: epoch 72 | batch 150/510 [I 2022-10-27 14:47:55] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:43 [I 2022-10-27 14:47:55] ...trap/engines/engine.py.326: process: 0.27998 | load: 0.00016 [I 2022-10-27 14:47:58] ...trap/engines/engine.py.321: eval: epoch 72 | batch 160/510 [I 2022-10-27 14:47:58] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 14:47:58] ...trap/engines/engine.py.326: process: 0.28696 | load: 0.00018 [I 2022-10-27 14:48:01] ...trap/engines/engine.py.321: eval: epoch 72 | batch 170/510 [I 2022-10-27 14:48:01] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 14:48:01] ...trap/engines/engine.py.326: process: 0.28214 | load: 0.00025 [I 2022-10-27 14:48:04] ...trap/engines/engine.py.321: eval: epoch 72 | batch 180/510 [I 2022-10-27 14:48:04] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:34 [I 2022-10-27 14:48:04] ...trap/engines/engine.py.326: process: 0.28985 | load: 0.00017 [I 2022-10-27 14:48:07] ...trap/engines/engine.py.321: eval: epoch 72 | batch 190/510 [I 2022-10-27 14:48:07] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:34 [I 2022-10-27 14:48:07] ...trap/engines/engine.py.326: process: 0.30727 | load: 0.00016 [I 2022-10-27 14:48:10] ...trap/engines/engine.py.321: eval: epoch 72 | batch 200/510 [I 2022-10-27 14:48:10] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:31 [I 2022-10-27 14:48:10] ...trap/engines/engine.py.326: process: 0.28700 | load: 0.00024 [I 2022-10-27 14:48:13] ...trap/engines/engine.py.321: eval: epoch 72 | batch 210/510 [I 2022-10-27 14:48:13] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:27 [I 2022-10-27 14:48:13] ...trap/engines/engine.py.326: process: 0.29018 | load: 0.00023 [I 2022-10-27 14:48:16] ...trap/engines/engine.py.321: eval: epoch 72 | batch 220/510 [I 2022-10-27 14:48:16] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 14:48:16] ...trap/engines/engine.py.326: process: 0.28631 | load: 0.00021 [I 2022-10-27 14:48:19] ...trap/engines/engine.py.321: eval: epoch 72 | batch 230/510 [I 2022-10-27 14:48:19] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 14:48:19] ...trap/engines/engine.py.326: process: 0.28477 | load: 0.00018 [I 2022-10-27 14:48:21] ...trap/engines/engine.py.321: eval: epoch 72 | batch 240/510 [I 2022-10-27 14:48:21] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-27 14:48:21] ...trap/engines/engine.py.326: process: 0.28606 | load: 0.00022 [I 2022-10-27 14:48:24] ...trap/engines/engine.py.321: eval: epoch 72 | batch 250/510 [I 2022-10-27 14:48:24] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-27 14:48:24] ...trap/engines/engine.py.326: process: 0.28727 | load: 0.00017 [I 2022-10-27 14:48:27] ...trap/engines/engine.py.321: eval: epoch 72 | batch 260/510 [I 2022-10-27 14:48:27] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:11 [I 2022-10-27 14:48:27] ...trap/engines/engine.py.326: process: 0.28626 | load: 0.00023 [I 2022-10-27 14:48:30] ...trap/engines/engine.py.321: eval: epoch 72 | batch 270/510 [I 2022-10-27 14:48:30] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 14:48:30] ...trap/engines/engine.py.326: process: 0.28488 | load: 0.00018 [I 2022-10-27 14:48:33] ...trap/engines/engine.py.321: eval: epoch 72 | batch 280/510 [I 2022-10-27 14:48:33] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 14:48:33] ...trap/engines/engine.py.326: process: 0.28925 | load: 0.00016 [I 2022-10-27 14:48:36] ...trap/engines/engine.py.321: eval: epoch 72 | batch 290/510 [I 2022-10-27 14:48:36] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 14:48:36] ...trap/engines/engine.py.326: process: 0.28998 | load: 0.00019 [I 2022-10-27 14:48:39] ...trap/engines/engine.py.321: eval: epoch 72 | batch 300/510 [I 2022-10-27 14:48:39] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 14:48:39] ...trap/engines/engine.py.326: process: 0.29066 | load: 0.00023 [I 2022-10-27 14:48:42] ...trap/engines/engine.py.321: eval: epoch 72 | batch 310/510 [I 2022-10-27 14:48:42] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:59 [I 2022-10-27 14:48:42] ...trap/engines/engine.py.326: process: 0.28564 | load: 0.00017 [I 2022-10-27 14:48:45] ...trap/engines/engine.py.321: eval: epoch 72 | batch 320/510 [I 2022-10-27 14:48:45] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 14:48:45] ...trap/engines/engine.py.326: process: 0.29085 | load: 0.00020 [I 2022-10-27 14:48:47] ...trap/engines/engine.py.321: eval: epoch 72 | batch 330/510 [I 2022-10-27 14:48:47] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-27 14:48:47] ...trap/engines/engine.py.326: process: 0.28838 | load: 0.00015 [I 2022-10-27 14:48:50] ...trap/engines/engine.py.321: eval: epoch 72 | batch 340/510 [I 2022-10-27 14:48:50] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-27 14:48:50] ...trap/engines/engine.py.326: process: 0.28695 | load: 0.00018 [I 2022-10-27 14:48:53] ...trap/engines/engine.py.321: eval: epoch 72 | batch 350/510 [I 2022-10-27 14:48:53] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 14:48:53] ...trap/engines/engine.py.326: process: 0.28363 | load: 0.00017 [I 2022-10-27 14:48:56] ...trap/engines/engine.py.321: eval: epoch 72 | batch 360/510 [I 2022-10-27 14:48:56] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 14:48:56] ...trap/engines/engine.py.326: process: 0.28962 | load: 0.00017 [I 2022-10-27 14:48:59] ...trap/engines/engine.py.321: eval: epoch 72 | batch 370/510 [I 2022-10-27 14:48:59] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 14:48:59] ...trap/engines/engine.py.326: process: 0.28936 | load: 0.00017 [I 2022-10-27 14:49:02] ...trap/engines/engine.py.321: eval: epoch 72 | batch 380/510 [I 2022-10-27 14:49:02] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 14:49:02] ...trap/engines/engine.py.326: process: 0.29175 | load: 0.00016 [I 2022-10-27 14:49:05] ...trap/engines/engine.py.321: eval: epoch 72 | batch 390/510 [I 2022-10-27 14:49:05] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 14:49:05] ...trap/engines/engine.py.326: process: 0.28302 | load: 0.00016 [I 2022-10-27 14:49:08] ...trap/engines/engine.py.321: eval: epoch 72 | batch 400/510 [I 2022-10-27 14:49:08] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 14:49:08] ...trap/engines/engine.py.326: process: 0.28247 | load: 0.00016 [I 2022-10-27 14:49:11] ...trap/engines/engine.py.321: eval: epoch 72 | batch 410/510 [I 2022-10-27 14:49:11] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 14:49:11] ...trap/engines/engine.py.326: process: 0.29097 | load: 0.00017 [I 2022-10-27 14:49:13] ...trap/engines/engine.py.321: eval: epoch 72 | batch 420/510 [I 2022-10-27 14:49:13] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 14:49:13] ...trap/engines/engine.py.326: process: 0.29061 | load: 0.00017 [I 2022-10-27 14:49:16] ...trap/engines/engine.py.321: eval: epoch 72 | batch 430/510 [I 2022-10-27 14:49:16] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:22 [I 2022-10-27 14:49:16] ...trap/engines/engine.py.326: process: 0.28348 | load: 0.00015 [I 2022-10-27 14:49:19] ...trap/engines/engine.py.321: eval: epoch 72 | batch 440/510 [I 2022-10-27 14:49:19] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 14:49:19] ...trap/engines/engine.py.326: process: 0.28341 | load: 0.00021 [I 2022-10-27 14:49:22] ...trap/engines/engine.py.321: eval: epoch 72 | batch 450/510 [I 2022-10-27 14:49:22] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 14:49:22] ...trap/engines/engine.py.326: process: 0.28202 | load: 0.00018 [I 2022-10-27 14:49:25] ...trap/engines/engine.py.321: eval: epoch 72 | batch 460/510 [I 2022-10-27 14:49:25] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 14:49:25] ...trap/engines/engine.py.326: process: 0.31558 | load: 0.00026 [I 2022-10-27 14:49:28] ...trap/engines/engine.py.321: eval: epoch 72 | batch 470/510 [I 2022-10-27 14:49:28] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 14:49:28] ...trap/engines/engine.py.326: process: 0.28996 | load: 0.00023 [I 2022-10-27 14:49:31] ...trap/engines/engine.py.321: eval: epoch 72 | batch 480/510 [I 2022-10-27 14:49:31] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 14:49:31] ...trap/engines/engine.py.326: process: 0.29593 | load: 0.00019 [I 2022-10-27 14:49:34] ...trap/engines/engine.py.321: eval: epoch 72 | batch 490/510 [I 2022-10-27 14:49:34] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 14:49:34] ...trap/engines/engine.py.326: process: 0.28417 | load: 0.00017 [I 2022-10-27 14:49:37] ...trap/engines/engine.py.321: eval: epoch 72 | batch 500/510 [I 2022-10-27 14:49:37] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 14:49:37] ...trap/engines/engine.py.326: process: 0.29564 | load: 0.00019 [I 2022-10-27 14:49:40] ...trap/engines/engine.py.321: eval: epoch 72 | batch 510/510 [I 2022-10-27 14:49:40] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 14:49:40] ...trap/engines/engine.py.326: process: 0.28639 | load: 0.00011 [S 2022-10-27 14:49:40] ...trap/engines/engine.py.132: eval_epoch.epoch: 72 [I 2022-10-27 14:49:40] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.721700 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905700 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937900 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.717200 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905000 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935600 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017961 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009819 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007231 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015039 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010296 [S 2022-10-27 14:49:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009340 [I 2022-10-27 14:49:46] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 14:49:46] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 14:49:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 14:49:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 14:49:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 14:49:50] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 14:49:51] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 14:49:58] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 14:49:58] ...trap/engines/engine.py.163: Training model on trainset for epoch 73 [I 2022-10-27 14:49:58] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 14:50:00] ...trap/engines/engine.py.224: train: epoch 73 | batch 0/2304 [I 2022-10-27 14:50:00] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:45:40 [I 2022-10-27 14:50:00] ...trap/engines/engine.py.229: process: 2.75204 | load: 2.10603 [I 2022-10-27 14:50:00] ...trap/engines/engine.py.230: loss: 0.15483 [I 2022-10-27 14:50:00] ...optimizers/trijoint.py.151: total_norm: 26.657305 [I 2022-10-27 14:50:06] ...trap/engines/engine.py.224: train: epoch 73 | batch 10/2304 [I 2022-10-27 14:50:06] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:28 [I 2022-10-27 14:50:06] ...trap/engines/engine.py.229: process: 0.57701 | load: 0.00393 [I 2022-10-27 14:50:06] ...trap/engines/engine.py.230: loss: 0.11976 [I 2022-10-27 14:50:06] ...optimizers/trijoint.py.151: total_norm: 16.711851 [I 2022-10-27 14:50:12] ...trap/engines/engine.py.224: train: epoch 73 | batch 20/2304 [I 2022-10-27 14:50:12] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:22 [I 2022-10-27 14:50:12] ...trap/engines/engine.py.229: process: 0.56677 | load: 0.00374 [I 2022-10-27 14:50:12] ...trap/engines/engine.py.230: loss: 0.10044 [I 2022-10-27 14:50:12] ...optimizers/trijoint.py.151: total_norm: 12.375103 [I 2022-10-27 14:50:17] ...trap/engines/engine.py.224: train: epoch 73 | batch 30/2304 [I 2022-10-27 14:50:17] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:31 [I 2022-10-27 14:50:17] ...trap/engines/engine.py.229: process: 0.56828 | load: 0.00333 [I 2022-10-27 14:50:17] ...trap/engines/engine.py.230: loss: 0.20717 [I 2022-10-27 14:50:17] ...optimizers/trijoint.py.151: total_norm: 24.153860 [I 2022-10-27 14:50:23] ...trap/engines/engine.py.224: train: epoch 73 | batch 40/2304 [I 2022-10-27 14:50:23] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:20 [I 2022-10-27 14:50:23] ...trap/engines/engine.py.229: process: 0.55844 | load: 0.00294 [I 2022-10-27 14:50:23] ...trap/engines/engine.py.230: loss: 0.15519 [I 2022-10-27 14:50:23] ...optimizers/trijoint.py.151: total_norm: 21.714607 [I 2022-10-27 14:50:28] ...trap/engines/engine.py.224: train: epoch 73 | batch 50/2304 [I 2022-10-27 14:50:28] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:13 [I 2022-10-27 14:50:28] ...trap/engines/engine.py.229: process: 0.57746 | load: 0.00323 [I 2022-10-27 14:50:28] ...trap/engines/engine.py.230: loss: 0.10276 [I 2022-10-27 14:50:28] ...optimizers/trijoint.py.151: total_norm: 11.441751 [I 2022-10-27 14:50:34] ...trap/engines/engine.py.224: train: epoch 73 | batch 60/2304 [I 2022-10-27 14:50:34] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:02 [I 2022-10-27 14:50:34] ...trap/engines/engine.py.229: process: 0.57788 | load: 0.00282 [I 2022-10-27 14:50:34] ...trap/engines/engine.py.230: loss: 0.16132 [I 2022-10-27 14:50:34] ...optimizers/trijoint.py.151: total_norm: 19.144224 [I 2022-10-27 14:50:40] ...trap/engines/engine.py.224: train: epoch 73 | batch 70/2304 [I 2022-10-27 14:50:40] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:47 [I 2022-10-27 14:50:40] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00288 [I 2022-10-27 14:50:40] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-27 14:50:40] ...optimizers/trijoint.py.151: total_norm: 21.592770 [I 2022-10-27 14:50:45] ...trap/engines/engine.py.224: train: epoch 73 | batch 80/2304 [I 2022-10-27 14:50:45] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:38 [I 2022-10-27 14:50:45] ...trap/engines/engine.py.229: process: 0.55369 | load: 0.00282 [I 2022-10-27 14:50:45] ...trap/engines/engine.py.230: loss: 0.12577 [I 2022-10-27 14:50:45] ...optimizers/trijoint.py.151: total_norm: 27.532251 [I 2022-10-27 14:50:51] ...trap/engines/engine.py.224: train: epoch 73 | batch 90/2304 [I 2022-10-27 14:50:51] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:21:11 [I 2022-10-27 14:50:51] ...trap/engines/engine.py.229: process: 0.62322 | load: 0.00278 [I 2022-10-27 14:50:51] ...trap/engines/engine.py.230: loss: 0.19809 [I 2022-10-27 14:50:51] ...optimizers/trijoint.py.151: total_norm: 22.168547 [I 2022-10-27 14:50:57] ...trap/engines/engine.py.224: train: epoch 73 | batch 100/2304 [I 2022-10-27 14:50:57] ...trap/engines/engine.py.228: elapsed: 0:00:59 | left: 0:20:37 [I 2022-10-27 14:50:57] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00249 [I 2022-10-27 14:50:57] ...trap/engines/engine.py.230: loss: 0.18833 [I 2022-10-27 14:50:57] ...optimizers/trijoint.py.151: total_norm: 20.599373 [I 2022-10-27 14:51:02] ...trap/engines/engine.py.224: train: epoch 73 | batch 110/2304 [I 2022-10-27 14:51:02] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:24 [I 2022-10-27 14:51:02] ...trap/engines/engine.py.229: process: 0.56368 | load: 0.00258 [I 2022-10-27 14:51:02] ...trap/engines/engine.py.230: loss: 0.12992 [I 2022-10-27 14:51:02] ...optimizers/trijoint.py.151: total_norm: 15.575282 [I 2022-10-27 14:51:08] ...trap/engines/engine.py.224: train: epoch 73 | batch 120/2304 [I 2022-10-27 14:51:08] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:16 [I 2022-10-27 14:51:08] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00271 [I 2022-10-27 14:51:08] ...trap/engines/engine.py.230: loss: 0.12710 [I 2022-10-27 14:51:08] ...optimizers/trijoint.py.151: total_norm: 20.069330 [I 2022-10-27 14:51:13] ...trap/engines/engine.py.224: train: epoch 73 | batch 130/2304 [I 2022-10-27 14:51:13] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:16 [I 2022-10-27 14:51:13] ...trap/engines/engine.py.229: process: 0.55058 | load: 0.00258 [I 2022-10-27 14:51:13] ...trap/engines/engine.py.230: loss: 0.10683 [I 2022-10-27 14:51:13] ...optimizers/trijoint.py.151: total_norm: 7.837101 [I 2022-10-27 14:51:19] ...trap/engines/engine.py.224: train: epoch 73 | batch 140/2304 [I 2022-10-27 14:51:19] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:07 [I 2022-10-27 14:51:19] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00281 [I 2022-10-27 14:51:19] ...trap/engines/engine.py.230: loss: 0.14237 [I 2022-10-27 14:51:19] ...optimizers/trijoint.py.151: total_norm: 25.046129 [I 2022-10-27 14:51:25] ...trap/engines/engine.py.224: train: epoch 73 | batch 150/2304 [I 2022-10-27 14:51:25] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:19:54 [I 2022-10-27 14:51:25] ...trap/engines/engine.py.229: process: 0.54779 | load: 0.00248 [I 2022-10-27 14:51:25] ...trap/engines/engine.py.230: loss: 0.10825 [I 2022-10-27 14:51:25] ...optimizers/trijoint.py.151: total_norm: 9.214973 [I 2022-10-27 14:51:30] ...trap/engines/engine.py.224: train: epoch 73 | batch 160/2304 [I 2022-10-27 14:51:30] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:50 [I 2022-10-27 14:51:30] ...trap/engines/engine.py.229: process: 0.55684 | load: 0.00255 [I 2022-10-27 14:51:30] ...trap/engines/engine.py.230: loss: 0.16036 [I 2022-10-27 14:51:30] ...optimizers/trijoint.py.151: total_norm: 18.949188 [I 2022-10-27 14:51:36] ...trap/engines/engine.py.224: train: epoch 73 | batch 170/2304 [I 2022-10-27 14:51:36] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:51 [I 2022-10-27 14:51:36] ...trap/engines/engine.py.229: process: 0.54963 | load: 0.00264 [I 2022-10-27 14:51:36] ...trap/engines/engine.py.230: loss: 0.11354 [I 2022-10-27 14:51:36] ...optimizers/trijoint.py.151: total_norm: 13.920839 [I 2022-10-27 14:51:41] ...trap/engines/engine.py.224: train: epoch 73 | batch 180/2304 [I 2022-10-27 14:51:41] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:48 [I 2022-10-27 14:51:41] ...trap/engines/engine.py.229: process: 0.56094 | load: 0.00255 [I 2022-10-27 14:51:41] ...trap/engines/engine.py.230: loss: 0.11432 [I 2022-10-27 14:51:41] ...optimizers/trijoint.py.151: total_norm: 16.176401 [I 2022-10-27 14:51:47] ...trap/engines/engine.py.224: train: epoch 73 | batch 190/2304 [I 2022-10-27 14:51:47] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:39 [I 2022-10-27 14:51:47] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00254 [I 2022-10-27 14:51:47] ...trap/engines/engine.py.230: loss: 0.17452 [I 2022-10-27 14:51:47] ...optimizers/trijoint.py.151: total_norm: 10.233273 [I 2022-10-27 14:51:53] ...trap/engines/engine.py.224: train: epoch 73 | batch 200/2304 [I 2022-10-27 14:51:53] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:41 [I 2022-10-27 14:51:53] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00257 [I 2022-10-27 14:51:53] ...trap/engines/engine.py.230: loss: 0.10805 [I 2022-10-27 14:51:53] ...optimizers/trijoint.py.151: total_norm: 11.093575 [I 2022-10-27 14:51:58] ...trap/engines/engine.py.224: train: epoch 73 | batch 210/2304 [I 2022-10-27 14:51:58] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:23 [I 2022-10-27 14:51:58] ...trap/engines/engine.py.229: process: 0.55348 | load: 0.00278 [I 2022-10-27 14:51:58] ...trap/engines/engine.py.230: loss: 0.15682 [I 2022-10-27 14:51:58] ...optimizers/trijoint.py.151: total_norm: 24.265343 [I 2022-10-27 14:52:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 220/2304 [I 2022-10-27 14:52:04] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:24 [I 2022-10-27 14:52:04] ...trap/engines/engine.py.229: process: 0.57435 | load: 0.00253 [I 2022-10-27 14:52:04] ...trap/engines/engine.py.230: loss: 0.13460 [I 2022-10-27 14:52:04] ...optimizers/trijoint.py.151: total_norm: 12.208153 [I 2022-10-27 14:52:09] ...trap/engines/engine.py.224: train: epoch 73 | batch 230/2304 [I 2022-10-27 14:52:09] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:13 [I 2022-10-27 14:52:09] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00246 [I 2022-10-27 14:52:09] ...trap/engines/engine.py.230: loss: 0.17367 [I 2022-10-27 14:52:09] ...optimizers/trijoint.py.151: total_norm: 26.011393 [I 2022-10-27 14:52:15] ...trap/engines/engine.py.224: train: epoch 73 | batch 240/2304 [I 2022-10-27 14:52:15] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:07 [I 2022-10-27 14:52:15] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00244 [I 2022-10-27 14:52:15] ...trap/engines/engine.py.230: loss: 0.13192 [I 2022-10-27 14:52:15] ...optimizers/trijoint.py.151: total_norm: 14.093805 [I 2022-10-27 14:52:20] ...trap/engines/engine.py.224: train: epoch 73 | batch 250/2304 [I 2022-10-27 14:52:20] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:12 [I 2022-10-27 14:52:20] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00261 [I 2022-10-27 14:52:20] ...trap/engines/engine.py.230: loss: 0.15996 [I 2022-10-27 14:52:20] ...optimizers/trijoint.py.151: total_norm: 13.189301 [I 2022-10-27 14:52:26] ...trap/engines/engine.py.224: train: epoch 73 | batch 260/2304 [I 2022-10-27 14:52:26] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:06 [I 2022-10-27 14:52:26] ...trap/engines/engine.py.229: process: 0.55428 | load: 0.00257 [I 2022-10-27 14:52:26] ...trap/engines/engine.py.230: loss: 0.13968 [I 2022-10-27 14:52:26] ...optimizers/trijoint.py.151: total_norm: 17.653688 [I 2022-10-27 14:52:32] ...trap/engines/engine.py.224: train: epoch 73 | batch 270/2304 [I 2022-10-27 14:52:32] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:55 [I 2022-10-27 14:52:32] ...trap/engines/engine.py.229: process: 0.56895 | load: 0.00246 [I 2022-10-27 14:52:32] ...trap/engines/engine.py.230: loss: 0.21327 [I 2022-10-27 14:52:32] ...optimizers/trijoint.py.151: total_norm: 39.853916 [I 2022-10-27 14:52:37] ...trap/engines/engine.py.224: train: epoch 73 | batch 280/2304 [I 2022-10-27 14:52:37] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:53 [I 2022-10-27 14:52:37] ...trap/engines/engine.py.229: process: 0.56705 | load: 0.00245 [I 2022-10-27 14:52:37] ...trap/engines/engine.py.230: loss: 0.21395 [I 2022-10-27 14:52:37] ...optimizers/trijoint.py.151: total_norm: 44.067188 [I 2022-10-27 14:52:43] ...trap/engines/engine.py.224: train: epoch 73 | batch 290/2304 [I 2022-10-27 14:52:43] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:40 [I 2022-10-27 14:52:43] ...trap/engines/engine.py.229: process: 0.54615 | load: 0.00262 [I 2022-10-27 14:52:43] ...trap/engines/engine.py.230: loss: 0.22201 [I 2022-10-27 14:52:43] ...optimizers/trijoint.py.151: total_norm: 20.255350 [I 2022-10-27 14:52:49] ...trap/engines/engine.py.224: train: epoch 73 | batch 300/2304 [I 2022-10-27 14:52:49] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:39 [I 2022-10-27 14:52:49] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00249 [I 2022-10-27 14:52:49] ...trap/engines/engine.py.230: loss: 0.15023 [I 2022-10-27 14:52:49] ...optimizers/trijoint.py.151: total_norm: 19.353922 [I 2022-10-27 14:52:54] ...trap/engines/engine.py.224: train: epoch 73 | batch 310/2304 [I 2022-10-27 14:52:54] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:29 [I 2022-10-27 14:52:54] ...trap/engines/engine.py.229: process: 0.56845 | load: 0.00246 [I 2022-10-27 14:52:54] ...trap/engines/engine.py.230: loss: 0.14412 [I 2022-10-27 14:52:54] ...optimizers/trijoint.py.151: total_norm: 71.330605 [I 2022-10-27 14:53:00] ...trap/engines/engine.py.224: train: epoch 73 | batch 320/2304 [I 2022-10-27 14:53:00] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:27 [I 2022-10-27 14:53:00] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00252 [I 2022-10-27 14:53:00] ...trap/engines/engine.py.230: loss: 0.20745 [I 2022-10-27 14:53:00] ...optimizers/trijoint.py.151: total_norm: 18.283836 [I 2022-10-27 14:53:05] ...trap/engines/engine.py.224: train: epoch 73 | batch 330/2304 [I 2022-10-27 14:53:05] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:24 [I 2022-10-27 14:53:05] ...trap/engines/engine.py.229: process: 0.56917 | load: 0.00248 [I 2022-10-27 14:53:05] ...trap/engines/engine.py.230: loss: 0.15664 [I 2022-10-27 14:53:05] ...optimizers/trijoint.py.151: total_norm: 16.299019 [I 2022-10-27 14:53:11] ...trap/engines/engine.py.224: train: epoch 73 | batch 340/2304 [I 2022-10-27 14:53:11] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:09 [I 2022-10-27 14:53:11] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00271 [I 2022-10-27 14:53:11] ...trap/engines/engine.py.230: loss: 0.14969 [I 2022-10-27 14:53:11] ...optimizers/trijoint.py.151: total_norm: 17.476749 [I 2022-10-27 14:53:16] ...trap/engines/engine.py.224: train: epoch 73 | batch 350/2304 [I 2022-10-27 14:53:16] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:10 [I 2022-10-27 14:53:16] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00249 [I 2022-10-27 14:53:16] ...trap/engines/engine.py.230: loss: 0.12419 [I 2022-10-27 14:53:16] ...optimizers/trijoint.py.151: total_norm: 14.653181 [I 2022-10-27 14:53:22] ...trap/engines/engine.py.224: train: epoch 73 | batch 360/2304 [I 2022-10-27 14:53:22] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:02 [I 2022-10-27 14:53:22] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00257 [I 2022-10-27 14:53:22] ...trap/engines/engine.py.230: loss: 0.21497 [I 2022-10-27 14:53:22] ...optimizers/trijoint.py.151: total_norm: 16.448009 [I 2022-10-27 14:53:28] ...trap/engines/engine.py.224: train: epoch 73 | batch 370/2304 [I 2022-10-27 14:53:28] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:01 [I 2022-10-27 14:53:28] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00247 [I 2022-10-27 14:53:28] ...trap/engines/engine.py.230: loss: 0.11868 [I 2022-10-27 14:53:28] ...optimizers/trijoint.py.151: total_norm: 13.009295 [I 2022-10-27 14:53:33] ...trap/engines/engine.py.224: train: epoch 73 | batch 380/2304 [I 2022-10-27 14:53:33] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:55 [I 2022-10-27 14:53:33] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00245 [I 2022-10-27 14:53:33] ...trap/engines/engine.py.230: loss: 0.10299 [I 2022-10-27 14:53:33] ...optimizers/trijoint.py.151: total_norm: 8.823496 [I 2022-10-27 14:53:39] ...trap/engines/engine.py.224: train: epoch 73 | batch 390/2304 [I 2022-10-27 14:53:39] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:52 [I 2022-10-27 14:53:39] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00242 [I 2022-10-27 14:53:39] ...trap/engines/engine.py.230: loss: 0.15390 [I 2022-10-27 14:53:39] ...optimizers/trijoint.py.151: total_norm: 23.201008 [I 2022-10-27 14:53:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 400/2304 [I 2022-10-27 14:53:44] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:41 [I 2022-10-27 14:53:44] ...trap/engines/engine.py.229: process: 0.56109 | load: 0.00241 [I 2022-10-27 14:53:44] ...trap/engines/engine.py.230: loss: 0.15800 [I 2022-10-27 14:53:44] ...optimizers/trijoint.py.151: total_norm: 8.407733 [I 2022-10-27 14:53:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 410/2304 [I 2022-10-27 14:53:50] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:41 [I 2022-10-27 14:53:50] ...trap/engines/engine.py.229: process: 0.55568 | load: 0.00245 [I 2022-10-27 14:53:50] ...trap/engines/engine.py.230: loss: 0.11432 [I 2022-10-27 14:53:50] ...optimizers/trijoint.py.151: total_norm: 9.580930 [I 2022-10-27 14:53:56] ...trap/engines/engine.py.224: train: epoch 73 | batch 420/2304 [I 2022-10-27 14:53:56] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:28 [I 2022-10-27 14:53:56] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00245 [I 2022-10-27 14:53:56] ...trap/engines/engine.py.230: loss: 0.13438 [I 2022-10-27 14:53:56] ...optimizers/trijoint.py.151: total_norm: 20.210928 [I 2022-10-27 14:54:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 430/2304 [I 2022-10-27 14:54:01] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:22 [I 2022-10-27 14:54:01] ...trap/engines/engine.py.229: process: 0.54724 | load: 0.00250 [I 2022-10-27 14:54:01] ...trap/engines/engine.py.230: loss: 0.16649 [I 2022-10-27 14:54:01] ...optimizers/trijoint.py.151: total_norm: 11.869954 [I 2022-10-27 14:54:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 440/2304 [I 2022-10-27 14:54:07] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:28 [I 2022-10-27 14:54:07] ...trap/engines/engine.py.229: process: 0.57416 | load: 0.00247 [I 2022-10-27 14:54:07] ...trap/engines/engine.py.230: loss: 0.11658 [I 2022-10-27 14:54:07] ...optimizers/trijoint.py.151: total_norm: 8.991758 [I 2022-10-27 14:54:12] ...trap/engines/engine.py.224: train: epoch 73 | batch 450/2304 [I 2022-10-27 14:54:12] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:18 [I 2022-10-27 14:54:12] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00243 [I 2022-10-27 14:54:12] ...trap/engines/engine.py.230: loss: 0.12877 [I 2022-10-27 14:54:12] ...optimizers/trijoint.py.151: total_norm: 16.631229 [I 2022-10-27 14:54:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 460/2304 [I 2022-10-27 14:54:18] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:15 [I 2022-10-27 14:54:18] ...trap/engines/engine.py.229: process: 0.56774 | load: 0.00245 [I 2022-10-27 14:54:18] ...trap/engines/engine.py.230: loss: 0.21572 [I 2022-10-27 14:54:18] ...optimizers/trijoint.py.151: total_norm: 19.117800 [I 2022-10-27 14:54:43] ...trap/engines/engine.py.224: train: epoch 73 | batch 470/2304 [I 2022-10-27 14:54:43] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 1:17:14 [I 2022-10-27 14:54:43] ...trap/engines/engine.py.229: process: 0.55657 | load: 0.00240 [I 2022-10-27 14:54:43] ...trap/engines/engine.py.230: loss: 0.15943 [I 2022-10-27 14:54:43] ...optimizers/trijoint.py.151: total_norm: 33.077572 [I 2022-10-27 14:54:49] ...trap/engines/engine.py.224: train: epoch 73 | batch 480/2304 [I 2022-10-27 14:54:49] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:23:31 [I 2022-10-27 14:54:49] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00242 [I 2022-10-27 14:54:49] ...trap/engines/engine.py.230: loss: 0.15288 [I 2022-10-27 14:54:49] ...optimizers/trijoint.py.151: total_norm: 14.782931 [I 2022-10-27 14:54:54] ...trap/engines/engine.py.224: train: epoch 73 | batch 490/2304 [I 2022-10-27 14:54:54] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:17:30 [I 2022-10-27 14:54:54] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00235 [I 2022-10-27 14:54:54] ...trap/engines/engine.py.230: loss: 0.13210 [I 2022-10-27 14:54:54] ...optimizers/trijoint.py.151: total_norm: 30.447380 [I 2022-10-27 14:55:00] ...trap/engines/engine.py.224: train: epoch 73 | batch 500/2304 [I 2022-10-27 14:55:00] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:16:48 [I 2022-10-27 14:55:00] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00236 [I 2022-10-27 14:55:00] ...trap/engines/engine.py.230: loss: 0.11119 [I 2022-10-27 14:55:00] ...optimizers/trijoint.py.151: total_norm: 14.258583 [I 2022-10-27 14:55:05] ...trap/engines/engine.py.224: train: epoch 73 | batch 510/2304 [I 2022-10-27 14:55:05] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:16:44 [I 2022-10-27 14:55:05] ...trap/engines/engine.py.229: process: 0.56427 | load: 0.00241 [I 2022-10-27 14:55:05] ...trap/engines/engine.py.230: loss: 0.14388 [I 2022-10-27 14:55:05] ...optimizers/trijoint.py.151: total_norm: 26.319658 [I 2022-10-27 14:55:11] ...trap/engines/engine.py.224: train: epoch 73 | batch 520/2304 [I 2022-10-27 14:55:11] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:16:34 [I 2022-10-27 14:55:11] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00240 [I 2022-10-27 14:55:11] ...trap/engines/engine.py.230: loss: 0.12115 [I 2022-10-27 14:55:11] ...optimizers/trijoint.py.151: total_norm: 14.500587 [I 2022-10-27 14:55:17] ...trap/engines/engine.py.224: train: epoch 73 | batch 530/2304 [I 2022-10-27 14:55:17] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:16:33 [I 2022-10-27 14:55:17] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00242 [I 2022-10-27 14:55:17] ...trap/engines/engine.py.230: loss: 0.15554 [I 2022-10-27 14:55:17] ...optimizers/trijoint.py.151: total_norm: 33.672825 [I 2022-10-27 14:55:22] ...trap/engines/engine.py.224: train: epoch 73 | batch 540/2304 [I 2022-10-27 14:55:22] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:16:21 [I 2022-10-27 14:55:22] ...trap/engines/engine.py.229: process: 0.55228 | load: 0.00239 [I 2022-10-27 14:55:22] ...trap/engines/engine.py.230: loss: 0.14393 [I 2022-10-27 14:55:22] ...optimizers/trijoint.py.151: total_norm: 15.307605 [I 2022-10-27 14:55:28] ...trap/engines/engine.py.224: train: epoch 73 | batch 550/2304 [I 2022-10-27 14:55:28] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:16:19 [I 2022-10-27 14:55:28] ...trap/engines/engine.py.229: process: 0.56501 | load: 0.00238 [I 2022-10-27 14:55:28] ...trap/engines/engine.py.230: loss: 0.12291 [I 2022-10-27 14:55:28] ...optimizers/trijoint.py.151: total_norm: 14.455654 [I 2022-10-27 14:55:33] ...trap/engines/engine.py.224: train: epoch 73 | batch 560/2304 [I 2022-10-27 14:55:33] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:16:20 [I 2022-10-27 14:55:33] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00242 [I 2022-10-27 14:55:33] ...trap/engines/engine.py.230: loss: 0.16518 [I 2022-10-27 14:55:33] ...optimizers/trijoint.py.151: total_norm: 25.872000 [I 2022-10-27 14:55:39] ...trap/engines/engine.py.224: train: epoch 73 | batch 570/2304 [I 2022-10-27 14:55:39] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:16:03 [I 2022-10-27 14:55:39] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00240 [I 2022-10-27 14:55:39] ...trap/engines/engine.py.230: loss: 0.12284 [I 2022-10-27 14:55:39] ...optimizers/trijoint.py.151: total_norm: 27.384449 [I 2022-10-27 14:55:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 580/2304 [I 2022-10-27 14:55:44] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:15:59 [I 2022-10-27 14:55:44] ...trap/engines/engine.py.229: process: 0.55301 | load: 0.00237 [I 2022-10-27 14:55:44] ...trap/engines/engine.py.230: loss: 0.18682 [I 2022-10-27 14:55:44] ...optimizers/trijoint.py.151: total_norm: 22.759144 [I 2022-10-27 14:55:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 590/2304 [I 2022-10-27 14:55:50] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:15:54 [I 2022-10-27 14:55:50] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00240 [I 2022-10-27 14:55:50] ...trap/engines/engine.py.230: loss: 0.12939 [I 2022-10-27 14:55:50] ...optimizers/trijoint.py.151: total_norm: 33.929569 [I 2022-10-27 14:55:56] ...trap/engines/engine.py.224: train: epoch 73 | batch 600/2304 [I 2022-10-27 14:55:56] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:50 [I 2022-10-27 14:55:56] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00244 [I 2022-10-27 14:55:56] ...trap/engines/engine.py.230: loss: 0.11448 [I 2022-10-27 14:55:56] ...optimizers/trijoint.py.151: total_norm: 13.858383 [I 2022-10-27 14:56:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 610/2304 [I 2022-10-27 14:56:01] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:15:51 [I 2022-10-27 14:56:01] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00236 [I 2022-10-27 14:56:01] ...trap/engines/engine.py.230: loss: 0.13458 [I 2022-10-27 14:56:01] ...optimizers/trijoint.py.151: total_norm: 27.973875 [I 2022-10-27 14:56:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 620/2304 [I 2022-10-27 14:56:07] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:44 [I 2022-10-27 14:56:07] ...trap/engines/engine.py.229: process: 0.56642 | load: 0.00245 [I 2022-10-27 14:56:07] ...trap/engines/engine.py.230: loss: 0.14211 [I 2022-10-27 14:56:07] ...optimizers/trijoint.py.151: total_norm: 12.642002 [I 2022-10-27 14:56:12] ...trap/engines/engine.py.224: train: epoch 73 | batch 630/2304 [I 2022-10-27 14:56:12] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:15:36 [I 2022-10-27 14:56:12] ...trap/engines/engine.py.229: process: 0.55444 | load: 0.00242 [I 2022-10-27 14:56:12] ...trap/engines/engine.py.230: loss: 0.10028 [I 2022-10-27 14:56:12] ...optimizers/trijoint.py.151: total_norm: 12.655010 [I 2022-10-27 14:56:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 640/2304 [I 2022-10-27 14:56:18] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:44 [I 2022-10-27 14:56:18] ...trap/engines/engine.py.229: process: 0.56854 | load: 0.00243 [I 2022-10-27 14:56:18] ...trap/engines/engine.py.230: loss: 0.15674 [I 2022-10-27 14:56:18] ...optimizers/trijoint.py.151: total_norm: 14.777826 [I 2022-10-27 14:56:24] ...trap/engines/engine.py.224: train: epoch 73 | batch 650/2304 [I 2022-10-27 14:56:24] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:26 [I 2022-10-27 14:56:24] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00242 [I 2022-10-27 14:56:24] ...trap/engines/engine.py.230: loss: 0.13441 [I 2022-10-27 14:56:24] ...optimizers/trijoint.py.151: total_norm: 22.820457 [I 2022-10-27 14:56:30] ...trap/engines/engine.py.224: train: epoch 73 | batch 660/2304 [I 2022-10-27 14:56:30] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:15:36 [I 2022-10-27 14:56:30] ...trap/engines/engine.py.229: process: 0.54248 | load: 0.00243 [I 2022-10-27 14:56:30] ...trap/engines/engine.py.230: loss: 0.15869 [I 2022-10-27 14:56:30] ...optimizers/trijoint.py.151: total_norm: 15.563814 [I 2022-10-27 14:56:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 670/2304 [I 2022-10-27 14:56:35] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:10 [I 2022-10-27 14:56:35] ...trap/engines/engine.py.229: process: 0.55117 | load: 0.00242 [I 2022-10-27 14:56:35] ...trap/engines/engine.py.230: loss: 0.11079 [I 2022-10-27 14:56:35] ...optimizers/trijoint.py.151: total_norm: 13.912423 [I 2022-10-27 14:56:41] ...trap/engines/engine.py.224: train: epoch 73 | batch 680/2304 [I 2022-10-27 14:56:41] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:14:58 [I 2022-10-27 14:56:41] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00248 [I 2022-10-27 14:56:41] ...trap/engines/engine.py.230: loss: 0.11992 [I 2022-10-27 14:56:41] ...optimizers/trijoint.py.151: total_norm: 13.549923 [I 2022-10-27 14:56:46] ...trap/engines/engine.py.224: train: epoch 73 | batch 690/2304 [I 2022-10-27 14:56:46] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:15:00 [I 2022-10-27 14:56:46] ...trap/engines/engine.py.229: process: 0.56329 | load: 0.00246 [I 2022-10-27 14:56:46] ...trap/engines/engine.py.230: loss: 0.16959 [I 2022-10-27 14:56:46] ...optimizers/trijoint.py.151: total_norm: 26.861141 [I 2022-10-27 14:56:52] ...trap/engines/engine.py.224: train: epoch 73 | batch 700/2304 [I 2022-10-27 14:56:52] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:14:54 [I 2022-10-27 14:56:52] ...trap/engines/engine.py.229: process: 0.55807 | load: 0.00244 [I 2022-10-27 14:56:52] ...trap/engines/engine.py.230: loss: 0.07643 [I 2022-10-27 14:56:52] ...optimizers/trijoint.py.151: total_norm: 11.218185 [I 2022-10-27 14:56:57] ...trap/engines/engine.py.224: train: epoch 73 | batch 710/2304 [I 2022-10-27 14:56:57] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:14:50 [I 2022-10-27 14:56:57] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00247 [I 2022-10-27 14:56:57] ...trap/engines/engine.py.230: loss: 0.16156 [I 2022-10-27 14:56:57] ...optimizers/trijoint.py.151: total_norm: 15.286013 [I 2022-10-27 14:57:03] ...trap/engines/engine.py.224: train: epoch 73 | batch 720/2304 [I 2022-10-27 14:57:03] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:47 [I 2022-10-27 14:57:03] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00244 [I 2022-10-27 14:57:03] ...trap/engines/engine.py.230: loss: 0.12574 [I 2022-10-27 14:57:03] ...optimizers/trijoint.py.151: total_norm: 50.874195 [I 2022-10-27 14:57:09] ...trap/engines/engine.py.224: train: epoch 73 | batch 730/2304 [I 2022-10-27 14:57:09] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:46 [I 2022-10-27 14:57:09] ...trap/engines/engine.py.229: process: 0.56793 | load: 0.00244 [I 2022-10-27 14:57:09] ...trap/engines/engine.py.230: loss: 0.11435 [I 2022-10-27 14:57:09] ...optimizers/trijoint.py.151: total_norm: 11.988093 [I 2022-10-27 14:57:14] ...trap/engines/engine.py.224: train: epoch 73 | batch 740/2304 [I 2022-10-27 14:57:14] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:39 [I 2022-10-27 14:57:14] ...trap/engines/engine.py.229: process: 0.56674 | load: 0.00250 [I 2022-10-27 14:57:14] ...trap/engines/engine.py.230: loss: 0.13284 [I 2022-10-27 14:57:14] ...optimizers/trijoint.py.151: total_norm: 13.529024 [I 2022-10-27 14:57:20] ...trap/engines/engine.py.224: train: epoch 73 | batch 750/2304 [I 2022-10-27 14:57:20] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:32 [I 2022-10-27 14:57:20] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00242 [I 2022-10-27 14:57:20] ...trap/engines/engine.py.230: loss: 0.13105 [I 2022-10-27 14:57:20] ...optimizers/trijoint.py.151: total_norm: 22.827957 [I 2022-10-27 14:57:25] ...trap/engines/engine.py.224: train: epoch 73 | batch 760/2304 [I 2022-10-27 14:57:25] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:14:18 [I 2022-10-27 14:57:25] ...trap/engines/engine.py.229: process: 0.54899 | load: 0.00243 [I 2022-10-27 14:57:25] ...trap/engines/engine.py.230: loss: 0.17114 [I 2022-10-27 14:57:25] ...optimizers/trijoint.py.151: total_norm: 23.489023 [I 2022-10-27 14:57:31] ...trap/engines/engine.py.224: train: epoch 73 | batch 770/2304 [I 2022-10-27 14:57:31] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:14:16 [I 2022-10-27 14:57:31] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00244 [I 2022-10-27 14:57:31] ...trap/engines/engine.py.230: loss: 0.18739 [I 2022-10-27 14:57:31] ...optimizers/trijoint.py.151: total_norm: 18.036060 [I 2022-10-27 14:57:37] ...trap/engines/engine.py.224: train: epoch 73 | batch 780/2304 [I 2022-10-27 14:57:37] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:12 [I 2022-10-27 14:57:37] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00240 [I 2022-10-27 14:57:37] ...trap/engines/engine.py.230: loss: 0.12631 [I 2022-10-27 14:57:37] ...optimizers/trijoint.py.151: total_norm: 12.815633 [I 2022-10-27 14:57:42] ...trap/engines/engine.py.224: train: epoch 73 | batch 790/2304 [I 2022-10-27 14:57:42] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:14:11 [I 2022-10-27 14:57:42] ...trap/engines/engine.py.229: process: 0.56440 | load: 0.00272 [I 2022-10-27 14:57:42] ...trap/engines/engine.py.230: loss: 0.10635 [I 2022-10-27 14:57:42] ...optimizers/trijoint.py.151: total_norm: 26.105171 [I 2022-10-27 14:57:48] ...trap/engines/engine.py.224: train: epoch 73 | batch 800/2304 [I 2022-10-27 14:57:48] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:14:05 [I 2022-10-27 14:57:48] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00251 [I 2022-10-27 14:57:48] ...trap/engines/engine.py.230: loss: 0.17289 [I 2022-10-27 14:57:48] ...optimizers/trijoint.py.151: total_norm: 19.715460 [I 2022-10-27 14:57:54] ...trap/engines/engine.py.224: train: epoch 73 | batch 810/2304 [I 2022-10-27 14:57:54] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:56 [I 2022-10-27 14:57:54] ...trap/engines/engine.py.229: process: 0.57168 | load: 0.00252 [I 2022-10-27 14:57:54] ...trap/engines/engine.py.230: loss: 0.20502 [I 2022-10-27 14:57:54] ...optimizers/trijoint.py.151: total_norm: 19.215897 [I 2022-10-27 14:57:59] ...trap/engines/engine.py.224: train: epoch 73 | batch 820/2304 [I 2022-10-27 14:57:59] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:13:47 [I 2022-10-27 14:57:59] ...trap/engines/engine.py.229: process: 0.56554 | load: 0.00247 [I 2022-10-27 14:57:59] ...trap/engines/engine.py.230: loss: 0.12007 [I 2022-10-27 14:57:59] ...optimizers/trijoint.py.151: total_norm: 13.041931 [I 2022-10-27 14:58:05] ...trap/engines/engine.py.224: train: epoch 73 | batch 830/2304 [I 2022-10-27 14:58:05] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:39 [I 2022-10-27 14:58:05] ...trap/engines/engine.py.229: process: 0.55171 | load: 0.00253 [I 2022-10-27 14:58:05] ...trap/engines/engine.py.230: loss: 0.11076 [I 2022-10-27 14:58:05] ...optimizers/trijoint.py.151: total_norm: 11.243494 [I 2022-10-27 14:58:10] ...trap/engines/engine.py.224: train: epoch 73 | batch 840/2304 [I 2022-10-27 14:58:10] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:13:29 [I 2022-10-27 14:58:10] ...trap/engines/engine.py.229: process: 0.55208 | load: 0.00246 [I 2022-10-27 14:58:10] ...trap/engines/engine.py.230: loss: 0.10956 [I 2022-10-27 14:58:10] ...optimizers/trijoint.py.151: total_norm: 20.888271 [I 2022-10-27 14:58:16] ...trap/engines/engine.py.224: train: epoch 73 | batch 850/2304 [I 2022-10-27 14:58:16] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:31 [I 2022-10-27 14:58:16] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00257 [I 2022-10-27 14:58:16] ...trap/engines/engine.py.230: loss: 0.20576 [I 2022-10-27 14:58:16] ...optimizers/trijoint.py.151: total_norm: 12.188520 [I 2022-10-27 14:58:22] ...trap/engines/engine.py.224: train: epoch 73 | batch 860/2304 [I 2022-10-27 14:58:22] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:42 [I 2022-10-27 14:58:22] ...trap/engines/engine.py.229: process: 0.56510 | load: 0.00259 [I 2022-10-27 14:58:22] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-27 14:58:22] ...optimizers/trijoint.py.151: total_norm: 30.630972 [I 2022-10-27 14:58:27] ...trap/engines/engine.py.224: train: epoch 73 | batch 870/2304 [I 2022-10-27 14:58:27] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:13:21 [I 2022-10-27 14:58:27] ...trap/engines/engine.py.229: process: 0.56005 | load: 0.00248 [I 2022-10-27 14:58:27] ...trap/engines/engine.py.230: loss: 0.14520 [I 2022-10-27 14:58:27] ...optimizers/trijoint.py.151: total_norm: 14.035447 [I 2022-10-27 14:58:33] ...trap/engines/engine.py.224: train: epoch 73 | batch 880/2304 [I 2022-10-27 14:58:33] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:14 [I 2022-10-27 14:58:33] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00246 [I 2022-10-27 14:58:33] ...trap/engines/engine.py.230: loss: 0.12556 [I 2022-10-27 14:58:33] ...optimizers/trijoint.py.151: total_norm: 13.498195 [I 2022-10-27 14:58:38] ...trap/engines/engine.py.224: train: epoch 73 | batch 890/2304 [I 2022-10-27 14:58:38] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:13:07 [I 2022-10-27 14:58:38] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00247 [I 2022-10-27 14:58:38] ...trap/engines/engine.py.230: loss: 0.15778 [I 2022-10-27 14:58:38] ...optimizers/trijoint.py.151: total_norm: 8.920338 [I 2022-10-27 14:58:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 900/2304 [I 2022-10-27 14:58:44] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:13:03 [I 2022-10-27 14:58:44] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00258 [I 2022-10-27 14:58:44] ...trap/engines/engine.py.230: loss: 0.16032 [I 2022-10-27 14:58:44] ...optimizers/trijoint.py.151: total_norm: 24.868933 [I 2022-10-27 14:58:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 910/2304 [I 2022-10-27 14:58:50] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:57 [I 2022-10-27 14:58:50] ...trap/engines/engine.py.229: process: 0.55865 | load: 0.00247 [I 2022-10-27 14:58:50] ...trap/engines/engine.py.230: loss: 0.15857 [I 2022-10-27 14:58:50] ...optimizers/trijoint.py.151: total_norm: 13.577394 [I 2022-10-27 14:58:55] ...trap/engines/engine.py.224: train: epoch 73 | batch 920/2304 [I 2022-10-27 14:58:55] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:13:17 [I 2022-10-27 14:58:55] ...trap/engines/engine.py.229: process: 0.56210 | load: 0.00242 [I 2022-10-27 14:58:55] ...trap/engines/engine.py.230: loss: 0.17573 [I 2022-10-27 14:58:55] ...optimizers/trijoint.py.151: total_norm: 15.485951 [I 2022-10-27 14:59:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 930/2304 [I 2022-10-27 14:59:01] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:46 [I 2022-10-27 14:59:01] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00246 [I 2022-10-27 14:59:01] ...trap/engines/engine.py.230: loss: 0.15072 [I 2022-10-27 14:59:01] ...optimizers/trijoint.py.151: total_norm: 13.791631 [I 2022-10-27 14:59:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 940/2304 [I 2022-10-27 14:59:07] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:46 [I 2022-10-27 14:59:07] ...trap/engines/engine.py.229: process: 0.56428 | load: 0.00249 [I 2022-10-27 14:59:07] ...trap/engines/engine.py.230: loss: 0.13525 [I 2022-10-27 14:59:07] ...optimizers/trijoint.py.151: total_norm: 9.292259 [I 2022-10-27 14:59:12] ...trap/engines/engine.py.224: train: epoch 73 | batch 950/2304 [I 2022-10-27 14:59:12] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:31 [I 2022-10-27 14:59:12] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00242 [I 2022-10-27 14:59:12] ...trap/engines/engine.py.230: loss: 0.12352 [I 2022-10-27 14:59:12] ...optimizers/trijoint.py.151: total_norm: 14.312524 [I 2022-10-27 14:59:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 960/2304 [I 2022-10-27 14:59:18] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:24 [I 2022-10-27 14:59:18] ...trap/engines/engine.py.229: process: 0.54323 | load: 0.00261 [I 2022-10-27 14:59:18] ...trap/engines/engine.py.230: loss: 0.22360 [I 2022-10-27 14:59:18] ...optimizers/trijoint.py.151: total_norm: 11.489565 [I 2022-10-27 14:59:23] ...trap/engines/engine.py.224: train: epoch 73 | batch 970/2304 [I 2022-10-27 14:59:23] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:12:25 [I 2022-10-27 14:59:23] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00252 [I 2022-10-27 14:59:23] ...trap/engines/engine.py.230: loss: 0.16438 [I 2022-10-27 14:59:23] ...optimizers/trijoint.py.151: total_norm: 11.944996 [I 2022-10-27 14:59:29] ...trap/engines/engine.py.224: train: epoch 73 | batch 980/2304 [I 2022-10-27 14:59:29] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:15 [I 2022-10-27 14:59:29] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00251 [I 2022-10-27 14:59:29] ...trap/engines/engine.py.230: loss: 0.09395 [I 2022-10-27 14:59:29] ...optimizers/trijoint.py.151: total_norm: 11.178842 [I 2022-10-27 14:59:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 990/2304 [I 2022-10-27 14:59:35] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:12:21 [I 2022-10-27 14:59:35] ...trap/engines/engine.py.229: process: 0.56221 | load: 0.00246 [I 2022-10-27 14:59:35] ...trap/engines/engine.py.230: loss: 0.11563 [I 2022-10-27 14:59:35] ...optimizers/trijoint.py.151: total_norm: 11.484860 [I 2022-10-27 14:59:40] ...trap/engines/engine.py.224: train: epoch 73 | batch 1000/2304 [I 2022-10-27 14:59:40] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:03 [I 2022-10-27 14:59:40] ...trap/engines/engine.py.229: process: 0.54325 | load: 0.00244 [I 2022-10-27 14:59:40] ...trap/engines/engine.py.230: loss: 0.11482 [I 2022-10-27 14:59:40] ...optimizers/trijoint.py.151: total_norm: 11.372036 [I 2022-10-27 14:59:46] ...trap/engines/engine.py.224: train: epoch 73 | batch 1010/2304 [I 2022-10-27 14:59:46] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:12:00 [I 2022-10-27 14:59:46] ...trap/engines/engine.py.229: process: 0.56503 | load: 0.00264 [I 2022-10-27 14:59:46] ...trap/engines/engine.py.230: loss: 0.10553 [I 2022-10-27 14:59:46] ...optimizers/trijoint.py.151: total_norm: 13.074456 [I 2022-10-27 14:59:51] ...trap/engines/engine.py.224: train: epoch 73 | batch 1020/2304 [I 2022-10-27 14:59:51] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:57 [I 2022-10-27 14:59:51] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00248 [I 2022-10-27 14:59:51] ...trap/engines/engine.py.230: loss: 0.09999 [I 2022-10-27 14:59:51] ...optimizers/trijoint.py.151: total_norm: 10.948920 [I 2022-10-27 14:59:57] ...trap/engines/engine.py.224: train: epoch 73 | batch 1030/2304 [I 2022-10-27 14:59:57] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:48 [I 2022-10-27 14:59:57] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00265 [I 2022-10-27 14:59:57] ...trap/engines/engine.py.230: loss: 0.18230 [I 2022-10-27 14:59:57] ...optimizers/trijoint.py.151: total_norm: 14.435624 [I 2022-10-27 15:00:03] ...trap/engines/engine.py.224: train: epoch 73 | batch 1040/2304 [I 2022-10-27 15:00:03] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:45 [I 2022-10-27 15:00:03] ...trap/engines/engine.py.229: process: 0.54731 | load: 0.00250 [I 2022-10-27 15:00:03] ...trap/engines/engine.py.230: loss: 0.18804 [I 2022-10-27 15:00:03] ...optimizers/trijoint.py.151: total_norm: 44.763046 [I 2022-10-27 15:00:08] ...trap/engines/engine.py.224: train: epoch 73 | batch 1050/2304 [I 2022-10-27 15:00:08] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:52 [I 2022-10-27 15:00:08] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00246 [I 2022-10-27 15:00:08] ...trap/engines/engine.py.230: loss: 0.12788 [I 2022-10-27 15:00:08] ...optimizers/trijoint.py.151: total_norm: 17.895510 [I 2022-10-27 15:00:14] ...trap/engines/engine.py.224: train: epoch 73 | batch 1060/2304 [I 2022-10-27 15:00:14] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:35 [I 2022-10-27 15:00:14] ...trap/engines/engine.py.229: process: 0.56408 | load: 0.00241 [I 2022-10-27 15:00:14] ...trap/engines/engine.py.230: loss: 0.13571 [I 2022-10-27 15:00:14] ...optimizers/trijoint.py.151: total_norm: 18.081715 [I 2022-10-27 15:00:19] ...trap/engines/engine.py.224: train: epoch 73 | batch 1070/2304 [I 2022-10-27 15:00:19] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:26 [I 2022-10-27 15:00:19] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00247 [I 2022-10-27 15:00:19] ...trap/engines/engine.py.230: loss: 0.13558 [I 2022-10-27 15:00:19] ...optimizers/trijoint.py.151: total_norm: 11.980155 [I 2022-10-27 15:00:25] ...trap/engines/engine.py.224: train: epoch 73 | batch 1080/2304 [I 2022-10-27 15:00:25] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:20 [I 2022-10-27 15:00:25] ...trap/engines/engine.py.229: process: 0.56096 | load: 0.00243 [I 2022-10-27 15:00:25] ...trap/engines/engine.py.230: loss: 0.09682 [I 2022-10-27 15:00:25] ...optimizers/trijoint.py.151: total_norm: 13.679981 [I 2022-10-27 15:00:31] ...trap/engines/engine.py.224: train: epoch 73 | batch 1090/2304 [I 2022-10-27 15:00:31] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:12 [I 2022-10-27 15:00:31] ...trap/engines/engine.py.229: process: 0.54969 | load: 0.00242 [I 2022-10-27 15:00:31] ...trap/engines/engine.py.230: loss: 0.21343 [I 2022-10-27 15:00:31] ...optimizers/trijoint.py.151: total_norm: 32.602428 [I 2022-10-27 15:00:36] ...trap/engines/engine.py.224: train: epoch 73 | batch 1100/2304 [I 2022-10-27 15:00:36] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:11:09 [I 2022-10-27 15:00:36] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00243 [I 2022-10-27 15:00:36] ...trap/engines/engine.py.230: loss: 0.14585 [I 2022-10-27 15:00:36] ...optimizers/trijoint.py.151: total_norm: 12.977716 [I 2022-10-27 15:00:42] ...trap/engines/engine.py.224: train: epoch 73 | batch 1110/2304 [I 2022-10-27 15:00:42] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:11:02 [I 2022-10-27 15:00:42] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00241 [I 2022-10-27 15:00:42] ...trap/engines/engine.py.230: loss: 0.17701 [I 2022-10-27 15:00:42] ...optimizers/trijoint.py.151: total_norm: 19.673000 [I 2022-10-27 15:00:47] ...trap/engines/engine.py.224: train: epoch 73 | batch 1120/2304 [I 2022-10-27 15:00:47] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:10:57 [I 2022-10-27 15:00:47] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00244 [I 2022-10-27 15:00:47] ...trap/engines/engine.py.230: loss: 0.15910 [I 2022-10-27 15:00:47] ...optimizers/trijoint.py.151: total_norm: 21.942551 [I 2022-10-27 15:00:53] ...trap/engines/engine.py.224: train: epoch 73 | batch 1130/2304 [I 2022-10-27 15:00:53] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:50 [I 2022-10-27 15:00:53] ...trap/engines/engine.py.229: process: 0.55096 | load: 0.00252 [I 2022-10-27 15:00:53] ...trap/engines/engine.py.230: loss: 0.17699 [I 2022-10-27 15:00:53] ...optimizers/trijoint.py.151: total_norm: 24.423626 [I 2022-10-27 15:00:58] ...trap/engines/engine.py.224: train: epoch 73 | batch 1140/2304 [I 2022-10-27 15:00:58] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:44 [I 2022-10-27 15:00:58] ...trap/engines/engine.py.229: process: 0.55954 | load: 0.00248 [I 2022-10-27 15:00:58] ...trap/engines/engine.py.230: loss: 0.13994 [I 2022-10-27 15:00:58] ...optimizers/trijoint.py.151: total_norm: 18.327925 [I 2022-10-27 15:01:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 1150/2304 [I 2022-10-27 15:01:04] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:39 [I 2022-10-27 15:01:04] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00247 [I 2022-10-27 15:01:04] ...trap/engines/engine.py.230: loss: 0.12276 [I 2022-10-27 15:01:04] ...optimizers/trijoint.py.151: total_norm: 14.653065 [I 2022-10-27 15:01:09] ...trap/engines/engine.py.224: train: epoch 73 | batch 1160/2304 [I 2022-10-27 15:01:09] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:38 [I 2022-10-27 15:01:09] ...trap/engines/engine.py.229: process: 0.55159 | load: 0.00244 [I 2022-10-27 15:01:09] ...trap/engines/engine.py.230: loss: 0.16825 [I 2022-10-27 15:01:09] ...optimizers/trijoint.py.151: total_norm: 17.123447 [I 2022-10-27 15:01:15] ...trap/engines/engine.py.224: train: epoch 73 | batch 1170/2304 [I 2022-10-27 15:01:15] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:29 [I 2022-10-27 15:01:15] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00244 [I 2022-10-27 15:01:15] ...trap/engines/engine.py.230: loss: 0.20444 [I 2022-10-27 15:01:15] ...optimizers/trijoint.py.151: total_norm: 22.721622 [I 2022-10-27 15:01:21] ...trap/engines/engine.py.224: train: epoch 73 | batch 1180/2304 [I 2022-10-27 15:01:21] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:32 [I 2022-10-27 15:01:21] ...trap/engines/engine.py.229: process: 0.55114 | load: 0.00243 [I 2022-10-27 15:01:21] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-27 15:01:21] ...optimizers/trijoint.py.151: total_norm: 14.542192 [I 2022-10-27 15:01:26] ...trap/engines/engine.py.224: train: epoch 73 | batch 1190/2304 [I 2022-10-27 15:01:26] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:21 [I 2022-10-27 15:01:26] ...trap/engines/engine.py.229: process: 0.55186 | load: 0.00244 [I 2022-10-27 15:01:26] ...trap/engines/engine.py.230: loss: 0.16141 [I 2022-10-27 15:01:26] ...optimizers/trijoint.py.151: total_norm: 22.832863 [I 2022-10-27 15:01:32] ...trap/engines/engine.py.224: train: epoch 73 | batch 1200/2304 [I 2022-10-27 15:01:32] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:10:09 [I 2022-10-27 15:01:32] ...trap/engines/engine.py.229: process: 0.54678 | load: 0.00251 [I 2022-10-27 15:01:32] ...trap/engines/engine.py.230: loss: 0.23164 [I 2022-10-27 15:01:32] ...optimizers/trijoint.py.151: total_norm: 31.065046 [I 2022-10-27 15:01:37] ...trap/engines/engine.py.224: train: epoch 73 | batch 1210/2304 [I 2022-10-27 15:01:37] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:10:07 [I 2022-10-27 15:01:37] ...trap/engines/engine.py.229: process: 0.55272 | load: 0.00246 [I 2022-10-27 15:01:37] ...trap/engines/engine.py.230: loss: 0.12438 [I 2022-10-27 15:01:37] ...optimizers/trijoint.py.151: total_norm: 25.632921 [I 2022-10-27 15:01:43] ...trap/engines/engine.py.224: train: epoch 73 | batch 1220/2304 [I 2022-10-27 15:01:43] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:10:01 [I 2022-10-27 15:01:43] ...trap/engines/engine.py.229: process: 0.54950 | load: 0.00245 [I 2022-10-27 15:01:43] ...trap/engines/engine.py.230: loss: 0.11576 [I 2022-10-27 15:01:43] ...optimizers/trijoint.py.151: total_norm: 16.963747 [I 2022-10-27 15:01:49] ...trap/engines/engine.py.224: train: epoch 73 | batch 1230/2304 [I 2022-10-27 15:01:49] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:57 [I 2022-10-27 15:01:49] ...trap/engines/engine.py.229: process: 0.54498 | load: 0.00246 [I 2022-10-27 15:01:49] ...trap/engines/engine.py.230: loss: 0.13770 [I 2022-10-27 15:01:49] ...optimizers/trijoint.py.151: total_norm: 23.083153 [I 2022-10-27 15:01:54] ...trap/engines/engine.py.224: train: epoch 73 | batch 1240/2304 [I 2022-10-27 15:01:54] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:50 [I 2022-10-27 15:01:54] ...trap/engines/engine.py.229: process: 0.55176 | load: 0.00242 [I 2022-10-27 15:01:54] ...trap/engines/engine.py.230: loss: 0.18155 [I 2022-10-27 15:01:54] ...optimizers/trijoint.py.151: total_norm: 16.372210 [I 2022-10-27 15:02:00] ...trap/engines/engine.py.224: train: epoch 73 | batch 1250/2304 [I 2022-10-27 15:02:00] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:47 [I 2022-10-27 15:02:00] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00243 [I 2022-10-27 15:02:00] ...trap/engines/engine.py.230: loss: 0.12045 [I 2022-10-27 15:02:00] ...optimizers/trijoint.py.151: total_norm: 12.799715 [I 2022-10-27 15:02:05] ...trap/engines/engine.py.224: train: epoch 73 | batch 1260/2304 [I 2022-10-27 15:02:05] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:41 [I 2022-10-27 15:02:05] ...trap/engines/engine.py.229: process: 0.55873 | load: 0.00243 [I 2022-10-27 15:02:05] ...trap/engines/engine.py.230: loss: 0.10276 [I 2022-10-27 15:02:05] ...optimizers/trijoint.py.151: total_norm: 16.676044 [I 2022-10-27 15:02:11] ...trap/engines/engine.py.224: train: epoch 73 | batch 1270/2304 [I 2022-10-27 15:02:11] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:34 [I 2022-10-27 15:02:11] ...trap/engines/engine.py.229: process: 0.55128 | load: 0.00259 [I 2022-10-27 15:02:11] ...trap/engines/engine.py.230: loss: 0.15935 [I 2022-10-27 15:02:11] ...optimizers/trijoint.py.151: total_norm: 16.621590 [I 2022-10-27 15:02:17] ...trap/engines/engine.py.224: train: epoch 73 | batch 1280/2304 [I 2022-10-27 15:02:17] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:30 [I 2022-10-27 15:02:17] ...trap/engines/engine.py.229: process: 0.56265 | load: 0.00244 [I 2022-10-27 15:02:17] ...trap/engines/engine.py.230: loss: 0.16413 [I 2022-10-27 15:02:17] ...optimizers/trijoint.py.151: total_norm: 24.293386 [I 2022-10-27 15:02:22] ...trap/engines/engine.py.224: train: epoch 73 | batch 1290/2304 [I 2022-10-27 15:02:22] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:22 [I 2022-10-27 15:02:22] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00245 [I 2022-10-27 15:02:22] ...trap/engines/engine.py.230: loss: 0.10822 [I 2022-10-27 15:02:22] ...optimizers/trijoint.py.151: total_norm: 42.320065 [I 2022-10-27 15:02:28] ...trap/engines/engine.py.224: train: epoch 73 | batch 1300/2304 [I 2022-10-27 15:02:28] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:09:21 [I 2022-10-27 15:02:28] ...trap/engines/engine.py.229: process: 0.56562 | load: 0.00241 [I 2022-10-27 15:02:28] ...trap/engines/engine.py.230: loss: 0.09455 [I 2022-10-27 15:02:28] ...optimizers/trijoint.py.151: total_norm: 7.733213 [I 2022-10-27 15:02:33] ...trap/engines/engine.py.224: train: epoch 73 | batch 1310/2304 [I 2022-10-27 15:02:33] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:11 [I 2022-10-27 15:02:33] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00241 [I 2022-10-27 15:02:33] ...trap/engines/engine.py.230: loss: 0.14945 [I 2022-10-27 15:02:33] ...optimizers/trijoint.py.151: total_norm: 14.427659 [I 2022-10-27 15:02:39] ...trap/engines/engine.py.224: train: epoch 73 | batch 1320/2304 [I 2022-10-27 15:02:39] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:09:07 [I 2022-10-27 15:02:39] ...trap/engines/engine.py.229: process: 0.55264 | load: 0.00253 [I 2022-10-27 15:02:39] ...trap/engines/engine.py.230: loss: 0.15981 [I 2022-10-27 15:02:39] ...optimizers/trijoint.py.151: total_norm: 22.306417 [I 2022-10-27 15:02:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 1330/2304 [I 2022-10-27 15:02:44] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:09:00 [I 2022-10-27 15:02:44] ...trap/engines/engine.py.229: process: 0.54409 | load: 0.00249 [I 2022-10-27 15:02:44] ...trap/engines/engine.py.230: loss: 0.17962 [I 2022-10-27 15:02:44] ...optimizers/trijoint.py.151: total_norm: 14.157392 [I 2022-10-27 15:02:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 1340/2304 [I 2022-10-27 15:02:50] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:09:00 [I 2022-10-27 15:02:50] ...trap/engines/engine.py.229: process: 0.56704 | load: 0.00242 [I 2022-10-27 15:02:50] ...trap/engines/engine.py.230: loss: 0.11573 [I 2022-10-27 15:02:50] ...optimizers/trijoint.py.151: total_norm: 10.659521 [I 2022-10-27 15:02:56] ...trap/engines/engine.py.224: train: epoch 73 | batch 1350/2304 [I 2022-10-27 15:02:56] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:53 [I 2022-10-27 15:02:56] ...trap/engines/engine.py.229: process: 0.56327 | load: 0.00242 [I 2022-10-27 15:02:56] ...trap/engines/engine.py.230: loss: 0.23549 [I 2022-10-27 15:02:56] ...optimizers/trijoint.py.151: total_norm: 14.792902 [I 2022-10-27 15:03:01] ...trap/engines/engine.py.224: train: epoch 73 | batch 1360/2304 [I 2022-10-27 15:03:01] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:46 [I 2022-10-27 15:03:01] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00244 [I 2022-10-27 15:03:01] ...trap/engines/engine.py.230: loss: 0.10376 [I 2022-10-27 15:03:01] ...optimizers/trijoint.py.151: total_norm: 14.108355 [I 2022-10-27 15:03:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 1370/2304 [I 2022-10-27 15:03:07] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:42 [I 2022-10-27 15:03:07] ...trap/engines/engine.py.229: process: 0.55776 | load: 0.00244 [I 2022-10-27 15:03:07] ...trap/engines/engine.py.230: loss: 0.12126 [I 2022-10-27 15:03:07] ...optimizers/trijoint.py.151: total_norm: 36.067661 [I 2022-10-27 15:03:12] ...trap/engines/engine.py.224: train: epoch 73 | batch 1380/2304 [I 2022-10-27 15:03:12] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:31 [I 2022-10-27 15:03:12] ...trap/engines/engine.py.229: process: 0.55450 | load: 0.00245 [I 2022-10-27 15:03:12] ...trap/engines/engine.py.230: loss: 0.13492 [I 2022-10-27 15:03:12] ...optimizers/trijoint.py.151: total_norm: 12.399944 [I 2022-10-27 15:03:18] ...trap/engines/engine.py.224: train: epoch 73 | batch 1390/2304 [I 2022-10-27 15:03:18] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:28 [I 2022-10-27 15:03:18] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00246 [I 2022-10-27 15:03:18] ...trap/engines/engine.py.230: loss: 0.20667 [I 2022-10-27 15:03:18] ...optimizers/trijoint.py.151: total_norm: 22.409960 [I 2022-10-27 15:03:24] ...trap/engines/engine.py.224: train: epoch 73 | batch 1400/2304 [I 2022-10-27 15:03:24] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:22 [I 2022-10-27 15:03:24] ...trap/engines/engine.py.229: process: 0.56249 | load: 0.00247 [I 2022-10-27 15:03:24] ...trap/engines/engine.py.230: loss: 0.16335 [I 2022-10-27 15:03:24] ...optimizers/trijoint.py.151: total_norm: 29.230556 [I 2022-10-27 15:03:29] ...trap/engines/engine.py.224: train: epoch 73 | batch 1410/2304 [I 2022-10-27 15:03:29] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:20 [I 2022-10-27 15:03:29] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00247 [I 2022-10-27 15:03:29] ...trap/engines/engine.py.230: loss: 0.11977 [I 2022-10-27 15:03:29] ...optimizers/trijoint.py.151: total_norm: 11.597281 [I 2022-10-27 15:03:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 1420/2304 [I 2022-10-27 15:03:35] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:08:18 [I 2022-10-27 15:03:35] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00252 [I 2022-10-27 15:03:35] ...trap/engines/engine.py.230: loss: 0.20071 [I 2022-10-27 15:03:35] ...optimizers/trijoint.py.151: total_norm: 11.973932 [I 2022-10-27 15:03:40] ...trap/engines/engine.py.224: train: epoch 73 | batch 1430/2304 [I 2022-10-27 15:03:40] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:08:08 [I 2022-10-27 15:03:40] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00245 [I 2022-10-27 15:03:40] ...trap/engines/engine.py.230: loss: 0.11246 [I 2022-10-27 15:03:40] ...optimizers/trijoint.py.151: total_norm: 10.031571 [I 2022-10-27 15:03:46] ...trap/engines/engine.py.224: train: epoch 73 | batch 1440/2304 [I 2022-10-27 15:03:46] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:08:02 [I 2022-10-27 15:03:46] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00244 [I 2022-10-27 15:03:46] ...trap/engines/engine.py.230: loss: 0.15324 [I 2022-10-27 15:03:46] ...optimizers/trijoint.py.151: total_norm: 23.479588 [I 2022-10-27 15:03:52] ...trap/engines/engine.py.224: train: epoch 73 | batch 1450/2304 [I 2022-10-27 15:03:52] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:54 [I 2022-10-27 15:03:52] ...trap/engines/engine.py.229: process: 0.56874 | load: 0.00247 [I 2022-10-27 15:03:52] ...trap/engines/engine.py.230: loss: 0.13675 [I 2022-10-27 15:03:52] ...optimizers/trijoint.py.151: total_norm: 22.017588 [I 2022-10-27 15:03:57] ...trap/engines/engine.py.224: train: epoch 73 | batch 1460/2304 [I 2022-10-27 15:03:57] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:51 [I 2022-10-27 15:03:57] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00243 [I 2022-10-27 15:03:57] ...trap/engines/engine.py.230: loss: 0.18324 [I 2022-10-27 15:03:57] ...optimizers/trijoint.py.151: total_norm: 29.095198 [I 2022-10-27 15:04:03] ...trap/engines/engine.py.224: train: epoch 73 | batch 1470/2304 [I 2022-10-27 15:04:03] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:44 [I 2022-10-27 15:04:03] ...trap/engines/engine.py.229: process: 0.55129 | load: 0.00248 [I 2022-10-27 15:04:03] ...trap/engines/engine.py.230: loss: 0.15622 [I 2022-10-27 15:04:03] ...optimizers/trijoint.py.151: total_norm: 9.015534 [I 2022-10-27 15:04:08] ...trap/engines/engine.py.224: train: epoch 73 | batch 1480/2304 [I 2022-10-27 15:04:08] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:39 [I 2022-10-27 15:04:08] ...trap/engines/engine.py.229: process: 0.54721 | load: 0.00241 [I 2022-10-27 15:04:08] ...trap/engines/engine.py.230: loss: 0.10835 [I 2022-10-27 15:04:08] ...optimizers/trijoint.py.151: total_norm: 10.484294 [I 2022-10-27 15:04:14] ...trap/engines/engine.py.224: train: epoch 73 | batch 1490/2304 [I 2022-10-27 15:04:14] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:31 [I 2022-10-27 15:04:14] ...trap/engines/engine.py.229: process: 0.54228 | load: 0.00245 [I 2022-10-27 15:04:14] ...trap/engines/engine.py.230: loss: 0.13702 [I 2022-10-27 15:04:14] ...optimizers/trijoint.py.151: total_norm: 17.327423 [I 2022-10-27 15:04:20] ...trap/engines/engine.py.224: train: epoch 73 | batch 1500/2304 [I 2022-10-27 15:04:20] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:29 [I 2022-10-27 15:04:20] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00245 [I 2022-10-27 15:04:20] ...trap/engines/engine.py.230: loss: 0.10204 [I 2022-10-27 15:04:20] ...optimizers/trijoint.py.151: total_norm: 10.483243 [I 2022-10-27 15:04:25] ...trap/engines/engine.py.224: train: epoch 73 | batch 1510/2304 [I 2022-10-27 15:04:25] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:21 [I 2022-10-27 15:04:25] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00245 [I 2022-10-27 15:04:25] ...trap/engines/engine.py.230: loss: 0.11179 [I 2022-10-27 15:04:25] ...optimizers/trijoint.py.151: total_norm: 11.032220 [I 2022-10-27 15:04:31] ...trap/engines/engine.py.224: train: epoch 73 | batch 1520/2304 [I 2022-10-27 15:04:31] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:15 [I 2022-10-27 15:04:31] ...trap/engines/engine.py.229: process: 0.55362 | load: 0.00241 [I 2022-10-27 15:04:31] ...trap/engines/engine.py.230: loss: 0.14397 [I 2022-10-27 15:04:31] ...optimizers/trijoint.py.151: total_norm: 12.131577 [I 2022-10-27 15:04:36] ...trap/engines/engine.py.224: train: epoch 73 | batch 1530/2304 [I 2022-10-27 15:04:36] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:07:11 [I 2022-10-27 15:04:36] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00245 [I 2022-10-27 15:04:36] ...trap/engines/engine.py.230: loss: 0.15021 [I 2022-10-27 15:04:36] ...optimizers/trijoint.py.151: total_norm: 23.341591 [I 2022-10-27 15:04:42] ...trap/engines/engine.py.224: train: epoch 73 | batch 1540/2304 [I 2022-10-27 15:04:42] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:07:06 [I 2022-10-27 15:04:42] ...trap/engines/engine.py.229: process: 0.56569 | load: 0.00250 [I 2022-10-27 15:04:42] ...trap/engines/engine.py.230: loss: 0.20712 [I 2022-10-27 15:04:42] ...optimizers/trijoint.py.151: total_norm: 26.151140 [I 2022-10-27 15:04:47] ...trap/engines/engine.py.224: train: epoch 73 | batch 1550/2304 [I 2022-10-27 15:04:47] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:07:03 [I 2022-10-27 15:04:47] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00243 [I 2022-10-27 15:04:47] ...trap/engines/engine.py.230: loss: 0.16471 [I 2022-10-27 15:04:47] ...optimizers/trijoint.py.151: total_norm: 13.199480 [I 2022-10-27 15:04:53] ...trap/engines/engine.py.224: train: epoch 73 | batch 1560/2304 [I 2022-10-27 15:04:53] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:53 [I 2022-10-27 15:04:53] ...trap/engines/engine.py.229: process: 0.55571 | load: 0.00244 [I 2022-10-27 15:04:53] ...trap/engines/engine.py.230: loss: 0.14044 [I 2022-10-27 15:04:53] ...optimizers/trijoint.py.151: total_norm: 11.981819 [I 2022-10-27 15:04:59] ...trap/engines/engine.py.224: train: epoch 73 | batch 1570/2304 [I 2022-10-27 15:04:59] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:50 [I 2022-10-27 15:04:59] ...trap/engines/engine.py.229: process: 0.56912 | load: 0.00241 [I 2022-10-27 15:04:59] ...trap/engines/engine.py.230: loss: 0.09368 [I 2022-10-27 15:04:59] ...optimizers/trijoint.py.151: total_norm: 11.046550 [I 2022-10-27 15:05:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 1580/2304 [I 2022-10-27 15:05:04] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:45 [I 2022-10-27 15:05:04] ...trap/engines/engine.py.229: process: 0.56372 | load: 0.00244 [I 2022-10-27 15:05:04] ...trap/engines/engine.py.230: loss: 0.12716 [I 2022-10-27 15:05:04] ...optimizers/trijoint.py.151: total_norm: 39.084923 [I 2022-10-27 15:05:10] ...trap/engines/engine.py.224: train: epoch 73 | batch 1590/2304 [I 2022-10-27 15:05:10] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:38 [I 2022-10-27 15:05:10] ...trap/engines/engine.py.229: process: 0.56384 | load: 0.00240 [I 2022-10-27 15:05:10] ...trap/engines/engine.py.230: loss: 0.17131 [I 2022-10-27 15:05:10] ...optimizers/trijoint.py.151: total_norm: 12.905959 [I 2022-10-27 15:05:16] ...trap/engines/engine.py.224: train: epoch 73 | batch 1600/2304 [I 2022-10-27 15:05:16] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:37 [I 2022-10-27 15:05:16] ...trap/engines/engine.py.229: process: 0.56780 | load: 0.00246 [I 2022-10-27 15:05:16] ...trap/engines/engine.py.230: loss: 0.11386 [I 2022-10-27 15:05:16] ...optimizers/trijoint.py.151: total_norm: 14.830142 [I 2022-10-27 15:05:21] ...trap/engines/engine.py.224: train: epoch 73 | batch 1610/2304 [I 2022-10-27 15:05:21] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:31 [I 2022-10-27 15:05:21] ...trap/engines/engine.py.229: process: 0.57524 | load: 0.00249 [I 2022-10-27 15:05:21] ...trap/engines/engine.py.230: loss: 0.11246 [I 2022-10-27 15:05:21] ...optimizers/trijoint.py.151: total_norm: 22.018631 [I 2022-10-27 15:05:27] ...trap/engines/engine.py.224: train: epoch 73 | batch 1620/2304 [I 2022-10-27 15:05:27] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:25 [I 2022-10-27 15:05:27] ...trap/engines/engine.py.229: process: 0.57030 | load: 0.00248 [I 2022-10-27 15:05:27] ...trap/engines/engine.py.230: loss: 0.11243 [I 2022-10-27 15:05:27] ...optimizers/trijoint.py.151: total_norm: 8.221939 [I 2022-10-27 15:05:32] ...trap/engines/engine.py.224: train: epoch 73 | batch 1630/2304 [I 2022-10-27 15:05:32] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:06:18 [I 2022-10-27 15:05:32] ...trap/engines/engine.py.229: process: 0.56390 | load: 0.00245 [I 2022-10-27 15:05:32] ...trap/engines/engine.py.230: loss: 0.12269 [I 2022-10-27 15:05:32] ...optimizers/trijoint.py.151: total_norm: 13.596017 [I 2022-10-27 15:05:38] ...trap/engines/engine.py.224: train: epoch 73 | batch 1640/2304 [I 2022-10-27 15:05:38] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:06:09 [I 2022-10-27 15:05:38] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00245 [I 2022-10-27 15:05:38] ...trap/engines/engine.py.230: loss: 0.16186 [I 2022-10-27 15:05:38] ...optimizers/trijoint.py.151: total_norm: 22.345385 [I 2022-10-27 15:05:44] ...trap/engines/engine.py.224: train: epoch 73 | batch 1650/2304 [I 2022-10-27 15:05:44] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:06:04 [I 2022-10-27 15:05:44] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00245 [I 2022-10-27 15:05:44] ...trap/engines/engine.py.230: loss: 0.24119 [I 2022-10-27 15:05:44] ...optimizers/trijoint.py.151: total_norm: 21.810240 [I 2022-10-27 15:05:49] ...trap/engines/engine.py.224: train: epoch 73 | batch 1660/2304 [I 2022-10-27 15:05:49] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:06:01 [I 2022-10-27 15:05:49] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00245 [I 2022-10-27 15:05:49] ...trap/engines/engine.py.230: loss: 0.13375 [I 2022-10-27 15:05:49] ...optimizers/trijoint.py.151: total_norm: 17.131355 [I 2022-10-27 15:05:55] ...trap/engines/engine.py.224: train: epoch 73 | batch 1670/2304 [I 2022-10-27 15:05:55] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:52 [I 2022-10-27 15:05:55] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00244 [I 2022-10-27 15:05:55] ...trap/engines/engine.py.230: loss: 0.16073 [I 2022-10-27 15:05:55] ...optimizers/trijoint.py.151: total_norm: 45.904362 [I 2022-10-27 15:06:00] ...trap/engines/engine.py.224: train: epoch 73 | batch 1680/2304 [I 2022-10-27 15:06:00] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:48 [I 2022-10-27 15:06:00] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00247 [I 2022-10-27 15:06:00] ...trap/engines/engine.py.230: loss: 0.13462 [I 2022-10-27 15:06:00] ...optimizers/trijoint.py.151: total_norm: 12.360825 [I 2022-10-27 15:06:06] ...trap/engines/engine.py.224: train: epoch 73 | batch 1690/2304 [I 2022-10-27 15:06:06] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:42 [I 2022-10-27 15:06:06] ...trap/engines/engine.py.229: process: 0.56511 | load: 0.00248 [I 2022-10-27 15:06:06] ...trap/engines/engine.py.230: loss: 0.10142 [I 2022-10-27 15:06:06] ...optimizers/trijoint.py.151: total_norm: 10.458089 [I 2022-10-27 15:06:12] ...trap/engines/engine.py.224: train: epoch 73 | batch 1700/2304 [I 2022-10-27 15:06:12] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:40 [I 2022-10-27 15:06:12] ...trap/engines/engine.py.229: process: 0.56330 | load: 0.00248 [I 2022-10-27 15:06:12] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-27 15:06:12] ...optimizers/trijoint.py.151: total_norm: 13.062760 [I 2022-10-27 15:06:17] ...trap/engines/engine.py.224: train: epoch 73 | batch 1710/2304 [I 2022-10-27 15:06:17] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:43 [I 2022-10-27 15:06:17] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00250 [I 2022-10-27 15:06:17] ...trap/engines/engine.py.230: loss: 0.12854 [I 2022-10-27 15:06:17] ...optimizers/trijoint.py.151: total_norm: 17.953327 [I 2022-10-27 15:06:23] ...trap/engines/engine.py.224: train: epoch 73 | batch 1720/2304 [I 2022-10-27 15:06:23] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:25 [I 2022-10-27 15:06:23] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00251 [I 2022-10-27 15:06:23] ...trap/engines/engine.py.230: loss: 0.11389 [I 2022-10-27 15:06:23] ...optimizers/trijoint.py.151: total_norm: 16.787733 [I 2022-10-27 15:06:29] ...trap/engines/engine.py.224: train: epoch 73 | batch 1730/2304 [I 2022-10-27 15:06:29] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:05:21 [I 2022-10-27 15:06:29] ...trap/engines/engine.py.229: process: 0.57650 | load: 0.00250 [I 2022-10-27 15:06:29] ...trap/engines/engine.py.230: loss: 0.11508 [I 2022-10-27 15:06:29] ...optimizers/trijoint.py.151: total_norm: 10.377781 [I 2022-10-27 15:06:34] ...trap/engines/engine.py.224: train: epoch 73 | batch 1740/2304 [I 2022-10-27 15:06:34] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:05:16 [I 2022-10-27 15:06:34] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00246 [I 2022-10-27 15:06:34] ...trap/engines/engine.py.230: loss: 0.16558 [I 2022-10-27 15:06:34] ...optimizers/trijoint.py.151: total_norm: 23.038649 [I 2022-10-27 15:06:40] ...trap/engines/engine.py.224: train: epoch 73 | batch 1750/2304 [I 2022-10-27 15:06:40] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:05:10 [I 2022-10-27 15:06:40] ...trap/engines/engine.py.229: process: 0.56546 | load: 0.00246 [I 2022-10-27 15:06:40] ...trap/engines/engine.py.230: loss: 0.16450 [I 2022-10-27 15:06:40] ...optimizers/trijoint.py.151: total_norm: 15.963784 [I 2022-10-27 15:06:45] ...trap/engines/engine.py.224: train: epoch 73 | batch 1760/2304 [I 2022-10-27 15:06:45] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:05:03 [I 2022-10-27 15:06:45] ...trap/engines/engine.py.229: process: 0.55513 | load: 0.00247 [I 2022-10-27 15:06:45] ...trap/engines/engine.py.230: loss: 0.13851 [I 2022-10-27 15:06:45] ...optimizers/trijoint.py.151: total_norm: 18.556139 [I 2022-10-27 15:06:51] ...trap/engines/engine.py.224: train: epoch 73 | batch 1770/2304 [I 2022-10-27 15:06:51] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:58 [I 2022-10-27 15:06:51] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00247 [I 2022-10-27 15:06:51] ...trap/engines/engine.py.230: loss: 0.15177 [I 2022-10-27 15:06:51] ...optimizers/trijoint.py.151: total_norm: 48.657780 [I 2022-10-27 15:06:57] ...trap/engines/engine.py.224: train: epoch 73 | batch 1780/2304 [I 2022-10-27 15:06:57] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:53 [I 2022-10-27 15:06:57] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00250 [I 2022-10-27 15:06:57] ...trap/engines/engine.py.230: loss: 0.13541 [I 2022-10-27 15:06:57] ...optimizers/trijoint.py.151: total_norm: 15.858061 [I 2022-10-27 15:07:02] ...trap/engines/engine.py.224: train: epoch 73 | batch 1790/2304 [I 2022-10-27 15:07:02] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:46 [I 2022-10-27 15:07:02] ...trap/engines/engine.py.229: process: 0.53934 | load: 0.00246 [I 2022-10-27 15:07:02] ...trap/engines/engine.py.230: loss: 0.16213 [I 2022-10-27 15:07:02] ...optimizers/trijoint.py.151: total_norm: 19.808222 [I 2022-10-27 15:07:08] ...trap/engines/engine.py.224: train: epoch 73 | batch 1800/2304 [I 2022-10-27 15:07:08] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:43 [I 2022-10-27 15:07:08] ...trap/engines/engine.py.229: process: 0.56352 | load: 0.00247 [I 2022-10-27 15:07:08] ...trap/engines/engine.py.230: loss: 0.18598 [I 2022-10-27 15:07:08] ...optimizers/trijoint.py.151: total_norm: 33.421608 [I 2022-10-27 15:07:13] ...trap/engines/engine.py.224: train: epoch 73 | batch 1810/2304 [I 2022-10-27 15:07:13] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:34 [I 2022-10-27 15:07:13] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00248 [I 2022-10-27 15:07:13] ...trap/engines/engine.py.230: loss: 0.12451 [I 2022-10-27 15:07:13] ...optimizers/trijoint.py.151: total_norm: 27.200413 [I 2022-10-27 15:07:19] ...trap/engines/engine.py.224: train: epoch 73 | batch 1820/2304 [I 2022-10-27 15:07:19] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:32 [I 2022-10-27 15:07:19] ...trap/engines/engine.py.229: process: 0.56300 | load: 0.00246 [I 2022-10-27 15:07:19] ...trap/engines/engine.py.230: loss: 0.15971 [I 2022-10-27 15:07:19] ...optimizers/trijoint.py.151: total_norm: 15.754457 [I 2022-10-27 15:07:25] ...trap/engines/engine.py.224: train: epoch 73 | batch 1830/2304 [I 2022-10-27 15:07:25] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:24 [I 2022-10-27 15:07:25] ...trap/engines/engine.py.229: process: 0.58315 | load: 0.00245 [I 2022-10-27 15:07:25] ...trap/engines/engine.py.230: loss: 0.12997 [I 2022-10-27 15:07:25] ...optimizers/trijoint.py.151: total_norm: 14.577580 [I 2022-10-27 15:07:30] ...trap/engines/engine.py.224: train: epoch 73 | batch 1840/2304 [I 2022-10-27 15:07:30] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:19 [I 2022-10-27 15:07:30] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00246 [I 2022-10-27 15:07:30] ...trap/engines/engine.py.230: loss: 0.12440 [I 2022-10-27 15:07:30] ...optimizers/trijoint.py.151: total_norm: 20.196810 [I 2022-10-27 15:07:36] ...trap/engines/engine.py.224: train: epoch 73 | batch 1850/2304 [I 2022-10-27 15:07:36] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:04:16 [I 2022-10-27 15:07:36] ...trap/engines/engine.py.229: process: 0.56492 | load: 0.00244 [I 2022-10-27 15:07:36] ...trap/engines/engine.py.230: loss: 0.15922 [I 2022-10-27 15:07:36] ...optimizers/trijoint.py.151: total_norm: 8.723053 [I 2022-10-27 15:07:42] ...trap/engines/engine.py.224: train: epoch 73 | batch 1860/2304 [I 2022-10-27 15:07:42] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:04:09 [I 2022-10-27 15:07:42] ...trap/engines/engine.py.229: process: 0.55764 | load: 0.00248 [I 2022-10-27 15:07:42] ...trap/engines/engine.py.230: loss: 0.17187 [I 2022-10-27 15:07:42] ...optimizers/trijoint.py.151: total_norm: 27.942823 [I 2022-10-27 15:07:47] ...trap/engines/engine.py.224: train: epoch 73 | batch 1870/2304 [I 2022-10-27 15:07:47] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:03:59 [I 2022-10-27 15:07:47] ...trap/engines/engine.py.229: process: 0.54283 | load: 0.00250 [I 2022-10-27 15:07:47] ...trap/engines/engine.py.230: loss: 0.12769 [I 2022-10-27 15:07:47] ...optimizers/trijoint.py.151: total_norm: 19.014919 [I 2022-10-27 15:07:53] ...trap/engines/engine.py.224: train: epoch 73 | batch 1880/2304 [I 2022-10-27 15:07:53] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:55 [I 2022-10-27 15:07:53] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00248 [I 2022-10-27 15:07:53] ...trap/engines/engine.py.230: loss: 0.11857 [I 2022-10-27 15:07:53] ...optimizers/trijoint.py.151: total_norm: 18.384844 [I 2022-10-27 15:07:58] ...trap/engines/engine.py.224: train: epoch 73 | batch 1890/2304 [I 2022-10-27 15:07:58] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:50 [I 2022-10-27 15:07:58] ...trap/engines/engine.py.229: process: 0.56332 | load: 0.00246 [I 2022-10-27 15:07:58] ...trap/engines/engine.py.230: loss: 0.13425 [I 2022-10-27 15:07:58] ...optimizers/trijoint.py.151: total_norm: 26.433752 [I 2022-10-27 15:08:04] ...trap/engines/engine.py.224: train: epoch 73 | batch 1900/2304 [I 2022-10-27 15:08:04] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:44 [I 2022-10-27 15:08:04] ...trap/engines/engine.py.229: process: 0.55556 | load: 0.00243 [I 2022-10-27 15:08:04] ...trap/engines/engine.py.230: loss: 0.13056 [I 2022-10-27 15:08:04] ...optimizers/trijoint.py.151: total_norm: 13.828326 [I 2022-10-27 15:08:10] ...trap/engines/engine.py.224: train: epoch 73 | batch 1910/2304 [I 2022-10-27 15:08:10] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:40 [I 2022-10-27 15:08:10] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00246 [I 2022-10-27 15:08:10] ...trap/engines/engine.py.230: loss: 0.18844 [I 2022-10-27 15:08:10] ...optimizers/trijoint.py.151: total_norm: 17.514160 [I 2022-10-27 15:08:15] ...trap/engines/engine.py.224: train: epoch 73 | batch 1920/2304 [I 2022-10-27 15:08:15] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:33 [I 2022-10-27 15:08:15] ...trap/engines/engine.py.229: process: 0.55216 | load: 0.00243 [I 2022-10-27 15:08:15] ...trap/engines/engine.py.230: loss: 0.10953 [I 2022-10-27 15:08:15] ...optimizers/trijoint.py.151: total_norm: 21.430853 [I 2022-10-27 15:08:21] ...trap/engines/engine.py.224: train: epoch 73 | batch 1930/2304 [I 2022-10-27 15:08:21] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:27 [I 2022-10-27 15:08:21] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00246 [I 2022-10-27 15:08:21] ...trap/engines/engine.py.230: loss: 0.17087 [I 2022-10-27 15:08:21] ...optimizers/trijoint.py.151: total_norm: 39.462589 [I 2022-10-27 15:08:26] ...trap/engines/engine.py.224: train: epoch 73 | batch 1940/2304 [I 2022-10-27 15:08:26] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:24 [I 2022-10-27 15:08:26] ...trap/engines/engine.py.229: process: 0.57938 | load: 0.00248 [I 2022-10-27 15:08:26] ...trap/engines/engine.py.230: loss: 0.12414 [I 2022-10-27 15:08:26] ...optimizers/trijoint.py.151: total_norm: 40.995811 [I 2022-10-27 15:08:32] ...trap/engines/engine.py.224: train: epoch 73 | batch 1950/2304 [I 2022-10-27 15:08:32] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:03:15 [I 2022-10-27 15:08:32] ...trap/engines/engine.py.229: process: 0.55260 | load: 0.00251 [I 2022-10-27 15:08:32] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-27 15:08:32] ...optimizers/trijoint.py.151: total_norm: 14.561554 [I 2022-10-27 15:08:37] ...trap/engines/engine.py.224: train: epoch 73 | batch 1960/2304 [I 2022-10-27 15:08:37] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:03:11 [I 2022-10-27 15:08:37] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00243 [I 2022-10-27 15:08:37] ...trap/engines/engine.py.230: loss: 0.13967 [I 2022-10-27 15:08:37] ...optimizers/trijoint.py.151: total_norm: 12.213512 [I 2022-10-27 15:08:43] ...trap/engines/engine.py.224: train: epoch 73 | batch 1970/2304 [I 2022-10-27 15:08:43] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:03:12 [I 2022-10-27 15:08:43] ...trap/engines/engine.py.229: process: 0.63972 | load: 0.00245 [I 2022-10-27 15:08:43] ...trap/engines/engine.py.230: loss: 0.12883 [I 2022-10-27 15:08:43] ...optimizers/trijoint.py.151: total_norm: 16.941885 [I 2022-10-27 15:08:49] ...trap/engines/engine.py.224: train: epoch 73 | batch 1980/2304 [I 2022-10-27 15:08:49] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:03:02 [I 2022-10-27 15:08:49] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00245 [I 2022-10-27 15:08:49] ...trap/engines/engine.py.230: loss: 0.15566 [I 2022-10-27 15:08:49] ...optimizers/trijoint.py.151: total_norm: 10.543269 [I 2022-10-27 15:08:54] ...trap/engines/engine.py.224: train: epoch 73 | batch 1990/2304 [I 2022-10-27 15:08:54] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:56 [I 2022-10-27 15:08:54] ...trap/engines/engine.py.229: process: 0.56478 | load: 0.00245 [I 2022-10-27 15:08:54] ...trap/engines/engine.py.230: loss: 0.14995 [I 2022-10-27 15:08:54] ...optimizers/trijoint.py.151: total_norm: 13.163550 [I 2022-10-27 15:09:00] ...trap/engines/engine.py.224: train: epoch 73 | batch 2000/2304 [I 2022-10-27 15:09:00] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:49 [I 2022-10-27 15:09:00] ...trap/engines/engine.py.229: process: 0.54905 | load: 0.00247 [I 2022-10-27 15:09:00] ...trap/engines/engine.py.230: loss: 0.12971 [I 2022-10-27 15:09:00] ...optimizers/trijoint.py.151: total_norm: 15.621385 [I 2022-10-27 15:09:06] ...trap/engines/engine.py.224: train: epoch 73 | batch 2010/2304 [I 2022-10-27 15:09:06] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:44 [I 2022-10-27 15:09:06] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00244 [I 2022-10-27 15:09:06] ...trap/engines/engine.py.230: loss: 0.09499 [I 2022-10-27 15:09:06] ...optimizers/trijoint.py.151: total_norm: 10.294071 [I 2022-10-27 15:09:11] ...trap/engines/engine.py.224: train: epoch 73 | batch 2020/2304 [I 2022-10-27 15:09:11] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:38 [I 2022-10-27 15:09:11] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00243 [I 2022-10-27 15:09:11] ...trap/engines/engine.py.230: loss: 0.15622 [I 2022-10-27 15:09:11] ...optimizers/trijoint.py.151: total_norm: 17.966198 [I 2022-10-27 15:09:17] ...trap/engines/engine.py.224: train: epoch 73 | batch 2030/2304 [I 2022-10-27 15:09:17] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:33 [I 2022-10-27 15:09:17] ...trap/engines/engine.py.229: process: 0.56448 | load: 0.00243 [I 2022-10-27 15:09:17] ...trap/engines/engine.py.230: loss: 0.13658 [I 2022-10-27 15:09:17] ...optimizers/trijoint.py.151: total_norm: 14.489616 [I 2022-10-27 15:09:22] ...trap/engines/engine.py.224: train: epoch 73 | batch 2040/2304 [I 2022-10-27 15:09:22] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:27 [I 2022-10-27 15:09:22] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00242 [I 2022-10-27 15:09:22] ...trap/engines/engine.py.230: loss: 0.18834 [I 2022-10-27 15:09:22] ...optimizers/trijoint.py.151: total_norm: 24.109865 [I 2022-10-27 15:09:28] ...trap/engines/engine.py.224: train: epoch 73 | batch 2050/2304 [I 2022-10-27 15:09:28] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:02:21 [I 2022-10-27 15:09:28] ...trap/engines/engine.py.229: process: 0.54636 | load: 0.00255 [I 2022-10-27 15:09:28] ...trap/engines/engine.py.230: loss: 0.10223 [I 2022-10-27 15:09:28] ...optimizers/trijoint.py.151: total_norm: 10.935040 [I 2022-10-27 15:09:34] ...trap/engines/engine.py.224: train: epoch 73 | batch 2060/2304 [I 2022-10-27 15:09:34] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:02:15 [I 2022-10-27 15:09:34] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00246 [I 2022-10-27 15:09:34] ...trap/engines/engine.py.230: loss: 0.18783 [I 2022-10-27 15:09:34] ...optimizers/trijoint.py.151: total_norm: 11.206829 [I 2022-10-27 15:09:39] ...trap/engines/engine.py.224: train: epoch 73 | batch 2070/2304 [I 2022-10-27 15:09:39] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:02:11 [I 2022-10-27 15:09:39] ...trap/engines/engine.py.229: process: 0.57189 | load: 0.00251 [I 2022-10-27 15:09:39] ...trap/engines/engine.py.230: loss: 0.12484 [I 2022-10-27 15:09:39] ...optimizers/trijoint.py.151: total_norm: 13.763951 [I 2022-10-27 15:09:45] ...trap/engines/engine.py.224: train: epoch 73 | batch 2080/2304 [I 2022-10-27 15:09:45] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:02:04 [I 2022-10-27 15:09:45] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00243 [I 2022-10-27 15:09:45] ...trap/engines/engine.py.230: loss: 0.09913 [I 2022-10-27 15:09:45] ...optimizers/trijoint.py.151: total_norm: 10.846123 [I 2022-10-27 15:09:51] ...trap/engines/engine.py.224: train: epoch 73 | batch 2090/2304 [I 2022-10-27 15:09:51] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:59 [I 2022-10-27 15:09:51] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00243 [I 2022-10-27 15:09:51] ...trap/engines/engine.py.230: loss: 0.16253 [I 2022-10-27 15:09:51] ...optimizers/trijoint.py.151: total_norm: 13.155235 [I 2022-10-27 15:09:56] ...trap/engines/engine.py.224: train: epoch 73 | batch 2100/2304 [I 2022-10-27 15:09:56] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:54 [I 2022-10-27 15:09:56] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00247 [I 2022-10-27 15:09:56] ...trap/engines/engine.py.230: loss: 0.12408 [I 2022-10-27 15:09:56] ...optimizers/trijoint.py.151: total_norm: 22.937092 [I 2022-10-27 15:10:02] ...trap/engines/engine.py.224: train: epoch 73 | batch 2110/2304 [I 2022-10-27 15:10:02] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:47 [I 2022-10-27 15:10:02] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00243 [I 2022-10-27 15:10:02] ...trap/engines/engine.py.230: loss: 0.14192 [I 2022-10-27 15:10:02] ...optimizers/trijoint.py.151: total_norm: 14.080064 [I 2022-10-27 15:10:07] ...trap/engines/engine.py.224: train: epoch 73 | batch 2120/2304 [I 2022-10-27 15:10:07] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:43 [I 2022-10-27 15:10:07] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00251 [I 2022-10-27 15:10:07] ...trap/engines/engine.py.230: loss: 0.12344 [I 2022-10-27 15:10:07] ...optimizers/trijoint.py.151: total_norm: 25.796192 [I 2022-10-27 15:10:13] ...trap/engines/engine.py.224: train: epoch 73 | batch 2130/2304 [I 2022-10-27 15:10:13] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:37 [I 2022-10-27 15:10:13] ...trap/engines/engine.py.229: process: 0.56580 | load: 0.00251 [I 2022-10-27 15:10:13] ...trap/engines/engine.py.230: loss: 0.15251 [I 2022-10-27 15:10:13] ...optimizers/trijoint.py.151: total_norm: 18.604359 [I 2022-10-27 15:10:19] ...trap/engines/engine.py.224: train: epoch 73 | batch 2140/2304 [I 2022-10-27 15:10:19] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:31 [I 2022-10-27 15:10:19] ...trap/engines/engine.py.229: process: 0.55184 | load: 0.00243 [I 2022-10-27 15:10:19] ...trap/engines/engine.py.230: loss: 0.15643 [I 2022-10-27 15:10:19] ...optimizers/trijoint.py.151: total_norm: 22.731640 [I 2022-10-27 15:10:24] ...trap/engines/engine.py.224: train: epoch 73 | batch 2150/2304 [I 2022-10-27 15:10:24] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:26 [I 2022-10-27 15:10:24] ...trap/engines/engine.py.229: process: 0.56387 | load: 0.00249 [I 2022-10-27 15:10:24] ...trap/engines/engine.py.230: loss: 0.15706 [I 2022-10-27 15:10:24] ...optimizers/trijoint.py.151: total_norm: 25.973095 [I 2022-10-27 15:10:30] ...trap/engines/engine.py.224: train: epoch 73 | batch 2160/2304 [I 2022-10-27 15:10:30] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:01:19 [I 2022-10-27 15:10:30] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00243 [I 2022-10-27 15:10:30] ...trap/engines/engine.py.230: loss: 0.14330 [I 2022-10-27 15:10:30] ...optimizers/trijoint.py.151: total_norm: 10.118533 [I 2022-10-27 15:10:35] ...trap/engines/engine.py.224: train: epoch 73 | batch 2170/2304 [I 2022-10-27 15:10:35] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:01:15 [I 2022-10-27 15:10:35] ...trap/engines/engine.py.229: process: 0.56415 | load: 0.00243 [I 2022-10-27 15:10:35] ...trap/engines/engine.py.230: loss: 0.13389 [I 2022-10-27 15:10:35] ...optimizers/trijoint.py.151: total_norm: 11.308560 [I 2022-10-27 15:10:41] ...trap/engines/engine.py.224: train: epoch 73 | batch 2180/2304 [I 2022-10-27 15:10:41] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:01:08 [I 2022-10-27 15:10:41] ...trap/engines/engine.py.229: process: 0.54683 | load: 0.00245 [I 2022-10-27 15:10:41] ...trap/engines/engine.py.230: loss: 0.12020 [I 2022-10-27 15:10:41] ...optimizers/trijoint.py.151: total_norm: 15.957766 [I 2022-10-27 15:10:46] ...trap/engines/engine.py.224: train: epoch 73 | batch 2190/2304 [I 2022-10-27 15:10:46] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:01:03 [I 2022-10-27 15:10:46] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00243 [I 2022-10-27 15:10:46] ...trap/engines/engine.py.230: loss: 0.15136 [I 2022-10-27 15:10:46] ...optimizers/trijoint.py.151: total_norm: 16.682995 [I 2022-10-27 15:10:52] ...trap/engines/engine.py.224: train: epoch 73 | batch 2200/2304 [I 2022-10-27 15:10:52] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:59 [I 2022-10-27 15:10:52] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00244 [I 2022-10-27 15:10:52] ...trap/engines/engine.py.230: loss: 0.15314 [I 2022-10-27 15:10:52] ...optimizers/trijoint.py.151: total_norm: 14.724335 [I 2022-10-27 15:10:58] ...trap/engines/engine.py.224: train: epoch 73 | batch 2210/2304 [I 2022-10-27 15:10:58] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:52 [I 2022-10-27 15:10:58] ...trap/engines/engine.py.229: process: 0.56409 | load: 0.00250 [I 2022-10-27 15:10:58] ...trap/engines/engine.py.230: loss: 0.15718 [I 2022-10-27 15:10:58] ...optimizers/trijoint.py.151: total_norm: 18.180109 [I 2022-10-27 15:11:03] ...trap/engines/engine.py.224: train: epoch 73 | batch 2220/2304 [I 2022-10-27 15:11:03] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:47 [I 2022-10-27 15:11:03] ...trap/engines/engine.py.229: process: 0.56487 | load: 0.00245 [I 2022-10-27 15:11:03] ...trap/engines/engine.py.230: loss: 0.21277 [I 2022-10-27 15:11:03] ...optimizers/trijoint.py.151: total_norm: 19.091722 [I 2022-10-27 15:11:09] ...trap/engines/engine.py.224: train: epoch 73 | batch 2230/2304 [I 2022-10-27 15:11:09] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:41 [I 2022-10-27 15:11:09] ...trap/engines/engine.py.229: process: 0.55526 | load: 0.00244 [I 2022-10-27 15:11:09] ...trap/engines/engine.py.230: loss: 0.13015 [I 2022-10-27 15:11:09] ...optimizers/trijoint.py.151: total_norm: 21.978331 [I 2022-10-27 15:11:15] ...trap/engines/engine.py.224: train: epoch 73 | batch 2240/2304 [I 2022-10-27 15:11:15] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:35 [I 2022-10-27 15:11:15] ...trap/engines/engine.py.229: process: 0.56549 | load: 0.00244 [I 2022-10-27 15:11:15] ...trap/engines/engine.py.230: loss: 0.14704 [I 2022-10-27 15:11:15] ...optimizers/trijoint.py.151: total_norm: 16.552763 [I 2022-10-27 15:11:20] ...trap/engines/engine.py.224: train: epoch 73 | batch 2250/2304 [I 2022-10-27 15:11:20] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:30 [I 2022-10-27 15:11:20] ...trap/engines/engine.py.229: process: 0.55436 | load: 0.00263 [I 2022-10-27 15:11:20] ...trap/engines/engine.py.230: loss: 0.18688 [I 2022-10-27 15:11:20] ...optimizers/trijoint.py.151: total_norm: 12.712525 [I 2022-10-27 15:11:26] ...trap/engines/engine.py.224: train: epoch 73 | batch 2260/2304 [I 2022-10-27 15:11:26] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:24 [I 2022-10-27 15:11:26] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00244 [I 2022-10-27 15:11:26] ...trap/engines/engine.py.230: loss: 0.10914 [I 2022-10-27 15:11:26] ...optimizers/trijoint.py.151: total_norm: 15.732435 [I 2022-10-27 15:11:31] ...trap/engines/engine.py.224: train: epoch 73 | batch 2270/2304 [I 2022-10-27 15:11:31] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:18 [I 2022-10-27 15:11:31] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00255 [I 2022-10-27 15:11:31] ...trap/engines/engine.py.230: loss: 0.20193 [I 2022-10-27 15:11:31] ...optimizers/trijoint.py.151: total_norm: 15.952903 [I 2022-10-27 15:11:37] ...trap/engines/engine.py.224: train: epoch 73 | batch 2280/2304 [I 2022-10-27 15:11:37] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:13 [I 2022-10-27 15:11:37] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00244 [I 2022-10-27 15:11:37] ...trap/engines/engine.py.230: loss: 0.11192 [I 2022-10-27 15:11:37] ...optimizers/trijoint.py.151: total_norm: 12.135270 [I 2022-10-27 15:11:43] ...trap/engines/engine.py.224: train: epoch 73 | batch 2290/2304 [I 2022-10-27 15:11:43] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:07 [I 2022-10-27 15:11:43] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00249 [I 2022-10-27 15:11:43] ...trap/engines/engine.py.230: loss: 0.10697 [I 2022-10-27 15:11:43] ...optimizers/trijoint.py.151: total_norm: 9.735527 [I 2022-10-27 15:11:48] ...trap/engines/engine.py.224: train: epoch 73 | batch 2300/2304 [I 2022-10-27 15:11:48] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:02 [I 2022-10-27 15:11:48] ...trap/engines/engine.py.229: process: 0.55436 | load: 0.00015 [I 2022-10-27 15:11:48] ...trap/engines/engine.py.230: loss: 0.15014 [I 2022-10-27 15:11:48] ...optimizers/trijoint.py.151: total_norm: 26.148226 [I 2022-10-27 15:11:50] ...trap/engines/engine.py.224: train: epoch 73 | batch 2304/2304 [I 2022-10-27 15:11:50] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:00:00 [I 2022-10-27 15:11:50] ...trap/engines/engine.py.229: process: 0.54808 | load: 0.00024 [I 2022-10-27 15:11:50] ...trap/engines/engine.py.230: loss: 0.20619 [I 2022-10-27 15:11:50] ...optimizers/trijoint.py.151: total_norm: 12.408263 [S 2022-10-27 15:11:52] ...trap/engines/engine.py.129: train_epoch.epoch: 73 [S 2022-10-27 15:11:52] ...trap/engines/engine.py.129: train_epoch.loss: 0.14236865238551205 [I 2022-10-27 15:11:52] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 15:11:53] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 73 [I 2022-10-27 15:11:53] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 15:11:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 15:11:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 15:11:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 15:11:57] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 15:11:57] ...trap/engines/engine.py.321: eval: epoch 73 | batch 0/510 [I 2022-10-27 15:11:57] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:40:03 [I 2022-10-27 15:11:57] ...trap/engines/engine.py.326: process: 4.71188 | load: 3.57100 [I 2022-10-27 15:12:01] ...trap/engines/engine.py.321: eval: epoch 73 | batch 10/510 [I 2022-10-27 15:12:01] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:40 [I 2022-10-27 15:12:01] ...trap/engines/engine.py.326: process: 0.33333 | load: 0.00018 [I 2022-10-27 15:12:04] ...trap/engines/engine.py.321: eval: epoch 73 | batch 20/510 [I 2022-10-27 15:12:04] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:08 [I 2022-10-27 15:12:04] ...trap/engines/engine.py.326: process: 0.33227 | load: 0.00023 [I 2022-10-27 15:12:07] ...trap/engines/engine.py.321: eval: epoch 73 | batch 30/510 [I 2022-10-27 15:12:07] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:45 [I 2022-10-27 15:12:07] ...trap/engines/engine.py.326: process: 0.37152 | load: 0.00058 [I 2022-10-27 15:12:11] ...trap/engines/engine.py.321: eval: epoch 73 | batch 40/510 [I 2022-10-27 15:12:11] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:37 [I 2022-10-27 15:12:11] ...trap/engines/engine.py.326: process: 0.32331 | load: 0.00032 [I 2022-10-27 15:12:14] ...trap/engines/engine.py.321: eval: epoch 73 | batch 50/510 [I 2022-10-27 15:12:14] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:36 [I 2022-10-27 15:12:14] ...trap/engines/engine.py.326: process: 0.35866 | load: 0.00020 [I 2022-10-27 15:12:18] ...trap/engines/engine.py.321: eval: epoch 73 | batch 60/510 [I 2022-10-27 15:12:18] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:33 [I 2022-10-27 15:12:18] ...trap/engines/engine.py.326: process: 0.36213 | load: 0.00019 [I 2022-10-27 15:12:21] ...trap/engines/engine.py.321: eval: epoch 73 | batch 70/510 [I 2022-10-27 15:12:21] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:26 [I 2022-10-27 15:12:21] ...trap/engines/engine.py.326: process: 0.33639 | load: 0.00017 [I 2022-10-27 15:12:24] ...trap/engines/engine.py.321: eval: epoch 73 | batch 80/510 [I 2022-10-27 15:12:24] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:25 [I 2022-10-27 15:12:24] ...trap/engines/engine.py.326: process: 0.34440 | load: 0.00016 [I 2022-10-27 15:12:28] ...trap/engines/engine.py.321: eval: epoch 73 | batch 90/510 [I 2022-10-27 15:12:28] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:21 [I 2022-10-27 15:12:28] ...trap/engines/engine.py.326: process: 0.35974 | load: 0.00018 [I 2022-10-27 15:12:31] ...trap/engines/engine.py.321: eval: epoch 73 | batch 100/510 [I 2022-10-27 15:12:31] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:12 [I 2022-10-27 15:12:31] ...trap/engines/engine.py.326: process: 0.28440 | load: 0.00020 [I 2022-10-27 15:12:34] ...trap/engines/engine.py.321: eval: epoch 73 | batch 110/510 [I 2022-10-27 15:12:34] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:55 [I 2022-10-27 15:12:34] ...trap/engines/engine.py.326: process: 0.28778 | load: 0.00019 [I 2022-10-27 15:12:37] ...trap/engines/engine.py.321: eval: epoch 73 | batch 120/510 [I 2022-10-27 15:12:37] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:51 [I 2022-10-27 15:12:37] ...trap/engines/engine.py.326: process: 0.28466 | load: 0.00015 [I 2022-10-27 15:12:39] ...trap/engines/engine.py.321: eval: epoch 73 | batch 130/510 [I 2022-10-27 15:12:39] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:50 [I 2022-10-27 15:12:39] ...trap/engines/engine.py.326: process: 0.29017 | load: 0.00026 [I 2022-10-27 15:12:42] ...trap/engines/engine.py.321: eval: epoch 73 | batch 140/510 [I 2022-10-27 15:12:42] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 15:12:42] ...trap/engines/engine.py.326: process: 0.29088 | load: 0.00017 [I 2022-10-27 15:12:45] ...trap/engines/engine.py.321: eval: epoch 73 | batch 150/510 [I 2022-10-27 15:12:45] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 15:12:45] ...trap/engines/engine.py.326: process: 0.29011 | load: 0.00015 [I 2022-10-27 15:12:48] ...trap/engines/engine.py.321: eval: epoch 73 | batch 160/510 [I 2022-10-27 15:12:48] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 15:12:48] ...trap/engines/engine.py.326: process: 0.29623 | load: 0.00014 [I 2022-10-27 15:12:51] ...trap/engines/engine.py.321: eval: epoch 73 | batch 170/510 [I 2022-10-27 15:12:51] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:39 [I 2022-10-27 15:12:51] ...trap/engines/engine.py.326: process: 0.29408 | load: 0.00014 [I 2022-10-27 15:12:54] ...trap/engines/engine.py.321: eval: epoch 73 | batch 180/510 [I 2022-10-27 15:12:54] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 15:12:54] ...trap/engines/engine.py.326: process: 0.28455 | load: 0.00014 [I 2022-10-27 15:12:57] ...trap/engines/engine.py.321: eval: epoch 73 | batch 190/510 [I 2022-10-27 15:12:57] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 15:12:57] ...trap/engines/engine.py.326: process: 0.29202 | load: 0.00015 [I 2022-10-27 15:13:00] ...trap/engines/engine.py.321: eval: epoch 73 | batch 200/510 [I 2022-10-27 15:13:00] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 15:13:00] ...trap/engines/engine.py.326: process: 0.28638 | load: 0.00015 [I 2022-10-27 15:13:03] ...trap/engines/engine.py.321: eval: epoch 73 | batch 210/510 [I 2022-10-27 15:13:03] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 15:13:03] ...trap/engines/engine.py.326: process: 0.29011 | load: 0.00019 [I 2022-10-27 15:13:06] ...trap/engines/engine.py.321: eval: epoch 73 | batch 220/510 [I 2022-10-27 15:13:06] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 15:13:06] ...trap/engines/engine.py.326: process: 0.29118 | load: 0.00018 [I 2022-10-27 15:13:08] ...trap/engines/engine.py.321: eval: epoch 73 | batch 230/510 [I 2022-10-27 15:13:08] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 15:13:08] ...trap/engines/engine.py.326: process: 0.28422 | load: 0.00017 [I 2022-10-27 15:13:11] ...trap/engines/engine.py.321: eval: epoch 73 | batch 240/510 [I 2022-10-27 15:13:11] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 15:13:11] ...trap/engines/engine.py.326: process: 0.28593 | load: 0.00015 [I 2022-10-27 15:13:14] ...trap/engines/engine.py.321: eval: epoch 73 | batch 250/510 [I 2022-10-27 15:13:14] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 15:13:14] ...trap/engines/engine.py.326: process: 0.28703 | load: 0.00014 [I 2022-10-27 15:13:17] ...trap/engines/engine.py.321: eval: epoch 73 | batch 260/510 [I 2022-10-27 15:13:17] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 15:13:17] ...trap/engines/engine.py.326: process: 0.29134 | load: 0.00016 [I 2022-10-27 15:13:20] ...trap/engines/engine.py.321: eval: epoch 73 | batch 270/510 [I 2022-10-27 15:13:20] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 15:13:20] ...trap/engines/engine.py.326: process: 0.28875 | load: 0.00030 [I 2022-10-27 15:13:23] ...trap/engines/engine.py.321: eval: epoch 73 | batch 280/510 [I 2022-10-27 15:13:23] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 15:13:23] ...trap/engines/engine.py.326: process: 0.28135 | load: 0.00018 [I 2022-10-27 15:13:26] ...trap/engines/engine.py.321: eval: epoch 73 | batch 290/510 [I 2022-10-27 15:13:26] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 15:13:26] ...trap/engines/engine.py.326: process: 0.29019 | load: 0.00015 [I 2022-10-27 15:13:29] ...trap/engines/engine.py.321: eval: epoch 73 | batch 300/510 [I 2022-10-27 15:13:29] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 15:13:29] ...trap/engines/engine.py.326: process: 0.28913 | load: 0.00016 [I 2022-10-27 15:13:32] ...trap/engines/engine.py.321: eval: epoch 73 | batch 310/510 [I 2022-10-27 15:13:32] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:58 [I 2022-10-27 15:13:32] ...trap/engines/engine.py.326: process: 0.28872 | load: 0.00015 [I 2022-10-27 15:13:35] ...trap/engines/engine.py.321: eval: epoch 73 | batch 320/510 [I 2022-10-27 15:13:35] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 15:13:35] ...trap/engines/engine.py.326: process: 0.28325 | load: 0.00016 [I 2022-10-27 15:13:37] ...trap/engines/engine.py.321: eval: epoch 73 | batch 330/510 [I 2022-10-27 15:13:37] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 15:13:37] ...trap/engines/engine.py.326: process: 0.28752 | load: 0.00016 [I 2022-10-27 15:13:40] ...trap/engines/engine.py.321: eval: epoch 73 | batch 340/510 [I 2022-10-27 15:13:40] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 15:13:40] ...trap/engines/engine.py.326: process: 0.29340 | load: 0.00021 [I 2022-10-27 15:13:43] ...trap/engines/engine.py.321: eval: epoch 73 | batch 350/510 [I 2022-10-27 15:13:43] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 15:13:43] ...trap/engines/engine.py.326: process: 0.29439 | load: 0.00021 [I 2022-10-27 15:13:46] ...trap/engines/engine.py.321: eval: epoch 73 | batch 360/510 [I 2022-10-27 15:13:46] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 15:13:46] ...trap/engines/engine.py.326: process: 0.28772 | load: 0.00015 [I 2022-10-27 15:13:49] ...trap/engines/engine.py.321: eval: epoch 73 | batch 370/510 [I 2022-10-27 15:13:49] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 15:13:49] ...trap/engines/engine.py.326: process: 0.28769 | load: 0.00014 [I 2022-10-27 15:13:52] ...trap/engines/engine.py.321: eval: epoch 73 | batch 380/510 [I 2022-10-27 15:13:52] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 15:13:52] ...trap/engines/engine.py.326: process: 0.29228 | load: 0.00015 [I 2022-10-27 15:13:55] ...trap/engines/engine.py.321: eval: epoch 73 | batch 390/510 [I 2022-10-27 15:13:55] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 15:13:55] ...trap/engines/engine.py.326: process: 0.27859 | load: 0.00016 [I 2022-10-27 15:13:58] ...trap/engines/engine.py.321: eval: epoch 73 | batch 400/510 [I 2022-10-27 15:13:58] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 15:13:58] ...trap/engines/engine.py.326: process: 0.28868 | load: 0.00020 [I 2022-10-27 15:14:01] ...trap/engines/engine.py.321: eval: epoch 73 | batch 410/510 [I 2022-10-27 15:14:01] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 15:14:01] ...trap/engines/engine.py.326: process: 0.29350 | load: 0.00045 [I 2022-10-27 15:14:04] ...trap/engines/engine.py.321: eval: epoch 73 | batch 420/510 [I 2022-10-27 15:14:04] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 15:14:04] ...trap/engines/engine.py.326: process: 0.28493 | load: 0.00020 [I 2022-10-27 15:14:06] ...trap/engines/engine.py.321: eval: epoch 73 | batch 430/510 [I 2022-10-27 15:14:06] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 15:14:06] ...trap/engines/engine.py.326: process: 0.29160 | load: 0.00015 [I 2022-10-27 15:14:09] ...trap/engines/engine.py.321: eval: epoch 73 | batch 440/510 [I 2022-10-27 15:14:09] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 15:14:09] ...trap/engines/engine.py.326: process: 0.29040 | load: 0.00015 [I 2022-10-27 15:14:12] ...trap/engines/engine.py.321: eval: epoch 73 | batch 450/510 [I 2022-10-27 15:14:12] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 15:14:12] ...trap/engines/engine.py.326: process: 0.29399 | load: 0.00017 [I 2022-10-27 15:14:15] ...trap/engines/engine.py.321: eval: epoch 73 | batch 460/510 [I 2022-10-27 15:14:15] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 15:14:15] ...trap/engines/engine.py.326: process: 0.30954 | load: 0.00016 [I 2022-10-27 15:14:18] ...trap/engines/engine.py.321: eval: epoch 73 | batch 470/510 [I 2022-10-27 15:14:18] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 15:14:18] ...trap/engines/engine.py.326: process: 0.29270 | load: 0.00016 [I 2022-10-27 15:14:21] ...trap/engines/engine.py.321: eval: epoch 73 | batch 480/510 [I 2022-10-27 15:14:21] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 15:14:21] ...trap/engines/engine.py.326: process: 0.28713 | load: 0.00016 [I 2022-10-27 15:14:24] ...trap/engines/engine.py.321: eval: epoch 73 | batch 490/510 [I 2022-10-27 15:14:24] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 15:14:24] ...trap/engines/engine.py.326: process: 0.29227 | load: 0.00017 [I 2022-10-27 15:14:27] ...trap/engines/engine.py.321: eval: epoch 73 | batch 500/510 [I 2022-10-27 15:14:27] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 15:14:27] ...trap/engines/engine.py.326: process: 0.29212 | load: 0.00016 [I 2022-10-27 15:14:30] ...trap/engines/engine.py.321: eval: epoch 73 | batch 510/510 [I 2022-10-27 15:14:30] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 15:14:30] ...trap/engines/engine.py.326: process: 0.28506 | load: 0.00016 [S 2022-10-27 15:14:30] ...trap/engines/engine.py.132: eval_epoch.epoch: 73 [I 2022-10-27 15:14:30] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.727100 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907000 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935900 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.714700 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904300 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934800 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012308 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007510 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007569 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013878 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008521 [S 2022-10-27 15:14:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006911 [I 2022-10-27 15:14:36] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 15:14:36] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 15:14:36] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 15:14:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 15:14:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 15:14:40] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 15:14:41] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 15:14:47] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 15:14:47] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 15:14:47] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 15:14:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 15:14:56] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 15:14:56] ...trap/engines/engine.py.163: Training model on trainset for epoch 74 [I 2022-10-27 15:14:56] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 15:14:59] ...trap/engines/engine.py.224: train: epoch 74 | batch 0/2304 [I 2022-10-27 15:14:59] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:39:41 [I 2022-10-27 15:14:59] ...trap/engines/engine.py.229: process: 2.59595 | load: 1.97761 [I 2022-10-27 15:14:59] ...trap/engines/engine.py.230: loss: 0.13888 [I 2022-10-27 15:14:59] ...optimizers/trijoint.py.151: total_norm: 39.391472 [I 2022-10-27 15:15:05] ...trap/engines/engine.py.224: train: epoch 74 | batch 10/2304 [I 2022-10-27 15:15:05] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:38 [I 2022-10-27 15:15:05] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00515 [I 2022-10-27 15:15:05] ...trap/engines/engine.py.230: loss: 0.14052 [I 2022-10-27 15:15:05] ...optimizers/trijoint.py.151: total_norm: 23.375380 [I 2022-10-27 15:15:10] ...trap/engines/engine.py.224: train: epoch 74 | batch 20/2304 [I 2022-10-27 15:15:10] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:11 [I 2022-10-27 15:15:10] ...trap/engines/engine.py.229: process: 0.54907 | load: 0.00342 [I 2022-10-27 15:15:10] ...trap/engines/engine.py.230: loss: 0.15045 [I 2022-10-27 15:15:10] ...optimizers/trijoint.py.151: total_norm: 9.600224 [I 2022-10-27 15:15:16] ...trap/engines/engine.py.224: train: epoch 74 | batch 30/2304 [I 2022-10-27 15:15:16] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:12 [I 2022-10-27 15:15:16] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00357 [I 2022-10-27 15:15:16] ...trap/engines/engine.py.230: loss: 0.15274 [I 2022-10-27 15:15:16] ...optimizers/trijoint.py.151: total_norm: 20.113455 [I 2022-10-27 15:15:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 40/2304 [I 2022-10-27 15:15:21] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:58 [I 2022-10-27 15:15:21] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00293 [I 2022-10-27 15:15:21] ...trap/engines/engine.py.230: loss: 0.19187 [I 2022-10-27 15:15:21] ...optimizers/trijoint.py.151: total_norm: 26.076447 [I 2022-10-27 15:15:27] ...trap/engines/engine.py.224: train: epoch 74 | batch 50/2304 [I 2022-10-27 15:15:27] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:04 [I 2022-10-27 15:15:27] ...trap/engines/engine.py.229: process: 0.58340 | load: 0.00310 [I 2022-10-27 15:15:27] ...trap/engines/engine.py.230: loss: 0.13601 [I 2022-10-27 15:15:27] ...optimizers/trijoint.py.151: total_norm: 12.487665 [I 2022-10-27 15:15:33] ...trap/engines/engine.py.224: train: epoch 74 | batch 60/2304 [I 2022-10-27 15:15:33] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:39 [I 2022-10-27 15:15:33] ...trap/engines/engine.py.229: process: 0.54980 | load: 0.00270 [I 2022-10-27 15:15:33] ...trap/engines/engine.py.230: loss: 0.14940 [I 2022-10-27 15:15:33] ...optimizers/trijoint.py.151: total_norm: 16.542110 [I 2022-10-27 15:15:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 70/2304 [I 2022-10-27 15:15:38] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:48 [I 2022-10-27 15:15:38] ...trap/engines/engine.py.229: process: 0.56713 | load: 0.00293 [I 2022-10-27 15:15:38] ...trap/engines/engine.py.230: loss: 0.12109 [I 2022-10-27 15:15:38] ...optimizers/trijoint.py.151: total_norm: 9.712646 [I 2022-10-27 15:15:44] ...trap/engines/engine.py.224: train: epoch 74 | batch 80/2304 [I 2022-10-27 15:15:44] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:45 [I 2022-10-27 15:15:44] ...trap/engines/engine.py.229: process: 0.56417 | load: 0.00264 [I 2022-10-27 15:15:44] ...trap/engines/engine.py.230: loss: 0.13231 [I 2022-10-27 15:15:44] ...optimizers/trijoint.py.151: total_norm: 17.373558 [I 2022-10-27 15:15:49] ...trap/engines/engine.py.224: train: epoch 74 | batch 90/2304 [I 2022-10-27 15:15:49] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:29 [I 2022-10-27 15:15:49] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00265 [I 2022-10-27 15:15:49] ...trap/engines/engine.py.230: loss: 0.09775 [I 2022-10-27 15:15:49] ...optimizers/trijoint.py.151: total_norm: 11.096838 [I 2022-10-27 15:15:55] ...trap/engines/engine.py.224: train: epoch 74 | batch 100/2304 [I 2022-10-27 15:15:55] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:21 [I 2022-10-27 15:15:55] ...trap/engines/engine.py.229: process: 0.54991 | load: 0.00275 [I 2022-10-27 15:15:55] ...trap/engines/engine.py.230: loss: 0.16853 [I 2022-10-27 15:15:55] ...optimizers/trijoint.py.151: total_norm: 10.956418 [I 2022-10-27 15:16:00] ...trap/engines/engine.py.224: train: epoch 74 | batch 110/2304 [I 2022-10-27 15:16:00] ...trap/engines/engine.py.228: elapsed: 0:01:03 | left: 0:20:09 [I 2022-10-27 15:16:00] ...trap/engines/engine.py.229: process: 0.53770 | load: 0.00259 [I 2022-10-27 15:16:00] ...trap/engines/engine.py.230: loss: 0.10399 [I 2022-10-27 15:16:00] ...optimizers/trijoint.py.151: total_norm: 6.970579 [I 2022-10-27 15:16:06] ...trap/engines/engine.py.224: train: epoch 74 | batch 120/2304 [I 2022-10-27 15:16:06] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:16 [I 2022-10-27 15:16:06] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00272 [I 2022-10-27 15:16:06] ...trap/engines/engine.py.230: loss: 0.15495 [I 2022-10-27 15:16:06] ...optimizers/trijoint.py.151: total_norm: 19.306326 [I 2022-10-27 15:16:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 130/2304 [I 2022-10-27 15:16:12] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:12 [I 2022-10-27 15:16:12] ...trap/engines/engine.py.229: process: 0.55842 | load: 0.00244 [I 2022-10-27 15:16:12] ...trap/engines/engine.py.230: loss: 0.13491 [I 2022-10-27 15:16:12] ...optimizers/trijoint.py.151: total_norm: 13.084213 [I 2022-10-27 15:16:17] ...trap/engines/engine.py.224: train: epoch 74 | batch 140/2304 [I 2022-10-27 15:16:17] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:01 [I 2022-10-27 15:16:17] ...trap/engines/engine.py.229: process: 0.54904 | load: 0.00257 [I 2022-10-27 15:16:17] ...trap/engines/engine.py.230: loss: 0.14749 [I 2022-10-27 15:16:17] ...optimizers/trijoint.py.151: total_norm: 21.053686 [I 2022-10-27 15:16:23] ...trap/engines/engine.py.224: train: epoch 74 | batch 150/2304 [I 2022-10-27 15:16:23] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:11 [I 2022-10-27 15:16:23] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00260 [I 2022-10-27 15:16:23] ...trap/engines/engine.py.230: loss: 0.11794 [I 2022-10-27 15:16:23] ...optimizers/trijoint.py.151: total_norm: 13.446974 [I 2022-10-27 15:16:28] ...trap/engines/engine.py.224: train: epoch 74 | batch 160/2304 [I 2022-10-27 15:16:28] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:59 [I 2022-10-27 15:16:28] ...trap/engines/engine.py.229: process: 0.55842 | load: 0.00256 [I 2022-10-27 15:16:28] ...trap/engines/engine.py.230: loss: 0.15866 [I 2022-10-27 15:16:28] ...optimizers/trijoint.py.151: total_norm: 26.107517 [I 2022-10-27 15:16:34] ...trap/engines/engine.py.224: train: epoch 74 | batch 170/2304 [I 2022-10-27 15:16:34] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:20:02 [I 2022-10-27 15:16:34] ...trap/engines/engine.py.229: process: 0.56809 | load: 0.00263 [I 2022-10-27 15:16:34] ...trap/engines/engine.py.230: loss: 0.10667 [I 2022-10-27 15:16:34] ...optimizers/trijoint.py.151: total_norm: 11.186002 [I 2022-10-27 15:16:40] ...trap/engines/engine.py.224: train: epoch 74 | batch 180/2304 [I 2022-10-27 15:16:40] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:38 [I 2022-10-27 15:16:40] ...trap/engines/engine.py.229: process: 0.55555 | load: 0.00262 [I 2022-10-27 15:16:40] ...trap/engines/engine.py.230: loss: 0.15053 [I 2022-10-27 15:16:40] ...optimizers/trijoint.py.151: total_norm: 21.127659 [I 2022-10-27 15:16:45] ...trap/engines/engine.py.224: train: epoch 74 | batch 190/2304 [I 2022-10-27 15:16:45] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:31 [I 2022-10-27 15:16:45] ...trap/engines/engine.py.229: process: 0.54154 | load: 0.00275 [I 2022-10-27 15:16:45] ...trap/engines/engine.py.230: loss: 0.22403 [I 2022-10-27 15:16:45] ...optimizers/trijoint.py.151: total_norm: 30.801888 [I 2022-10-27 15:16:51] ...trap/engines/engine.py.224: train: epoch 74 | batch 200/2304 [I 2022-10-27 15:16:51] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:38 [I 2022-10-27 15:16:51] ...trap/engines/engine.py.229: process: 0.56772 | load: 0.00247 [I 2022-10-27 15:16:51] ...trap/engines/engine.py.230: loss: 0.11206 [I 2022-10-27 15:16:51] ...optimizers/trijoint.py.151: total_norm: 16.357666 [I 2022-10-27 15:16:56] ...trap/engines/engine.py.224: train: epoch 74 | batch 210/2304 [I 2022-10-27 15:16:56] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:25 [I 2022-10-27 15:16:56] ...trap/engines/engine.py.229: process: 0.54869 | load: 0.00264 [I 2022-10-27 15:16:56] ...trap/engines/engine.py.230: loss: 0.09375 [I 2022-10-27 15:16:56] ...optimizers/trijoint.py.151: total_norm: 9.606027 [I 2022-10-27 15:17:02] ...trap/engines/engine.py.224: train: epoch 74 | batch 220/2304 [I 2022-10-27 15:17:02] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:20 [I 2022-10-27 15:17:02] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00241 [I 2022-10-27 15:17:02] ...trap/engines/engine.py.230: loss: 0.13195 [I 2022-10-27 15:17:02] ...optimizers/trijoint.py.151: total_norm: 18.232624 [I 2022-10-27 15:17:07] ...trap/engines/engine.py.224: train: epoch 74 | batch 230/2304 [I 2022-10-27 15:17:07] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:12 [I 2022-10-27 15:17:07] ...trap/engines/engine.py.229: process: 0.56831 | load: 0.00585 [I 2022-10-27 15:17:07] ...trap/engines/engine.py.230: loss: 0.19685 [I 2022-10-27 15:17:07] ...optimizers/trijoint.py.151: total_norm: 23.700157 [I 2022-10-27 15:17:13] ...trap/engines/engine.py.224: train: epoch 74 | batch 240/2304 [I 2022-10-27 15:17:13] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:14 [I 2022-10-27 15:17:13] ...trap/engines/engine.py.229: process: 0.56530 | load: 0.00265 [I 2022-10-27 15:17:13] ...trap/engines/engine.py.230: loss: 0.21684 [I 2022-10-27 15:17:13] ...optimizers/trijoint.py.151: total_norm: 21.428411 [I 2022-10-27 15:17:19] ...trap/engines/engine.py.224: train: epoch 74 | batch 250/2304 [I 2022-10-27 15:17:19] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:11 [I 2022-10-27 15:17:19] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00244 [I 2022-10-27 15:17:19] ...trap/engines/engine.py.230: loss: 0.13142 [I 2022-10-27 15:17:19] ...optimizers/trijoint.py.151: total_norm: 16.051346 [I 2022-10-27 15:17:24] ...trap/engines/engine.py.224: train: epoch 74 | batch 260/2304 [I 2022-10-27 15:17:24] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:00 [I 2022-10-27 15:17:24] ...trap/engines/engine.py.229: process: 0.55863 | load: 0.00252 [I 2022-10-27 15:17:24] ...trap/engines/engine.py.230: loss: 0.17647 [I 2022-10-27 15:17:24] ...optimizers/trijoint.py.151: total_norm: 26.469471 [I 2022-10-27 15:17:30] ...trap/engines/engine.py.224: train: epoch 74 | batch 270/2304 [I 2022-10-27 15:17:30] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:48 [I 2022-10-27 15:17:30] ...trap/engines/engine.py.229: process: 0.55609 | load: 0.00280 [I 2022-10-27 15:17:30] ...trap/engines/engine.py.230: loss: 0.10543 [I 2022-10-27 15:17:30] ...optimizers/trijoint.py.151: total_norm: 10.507544 [I 2022-10-27 15:17:35] ...trap/engines/engine.py.224: train: epoch 74 | batch 280/2304 [I 2022-10-27 15:17:35] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:57 [I 2022-10-27 15:17:35] ...trap/engines/engine.py.229: process: 0.56832 | load: 0.00260 [I 2022-10-27 15:17:35] ...trap/engines/engine.py.230: loss: 0.16767 [I 2022-10-27 15:17:35] ...optimizers/trijoint.py.151: total_norm: 35.567101 [I 2022-10-27 15:17:41] ...trap/engines/engine.py.224: train: epoch 74 | batch 290/2304 [I 2022-10-27 15:17:41] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:47 [I 2022-10-27 15:17:41] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00259 [I 2022-10-27 15:17:41] ...trap/engines/engine.py.230: loss: 0.14542 [I 2022-10-27 15:17:41] ...optimizers/trijoint.py.151: total_norm: 12.070747 [I 2022-10-27 15:17:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 300/2304 [I 2022-10-27 15:17:47] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:41 [I 2022-10-27 15:17:47] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00252 [I 2022-10-27 15:17:47] ...trap/engines/engine.py.230: loss: 0.10912 [I 2022-10-27 15:17:47] ...optimizers/trijoint.py.151: total_norm: 16.283710 [I 2022-10-27 15:17:52] ...trap/engines/engine.py.224: train: epoch 74 | batch 310/2304 [I 2022-10-27 15:17:52] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:19:05 [I 2022-10-27 15:17:52] ...trap/engines/engine.py.229: process: 0.55880 | load: 0.00302 [I 2022-10-27 15:17:52] ...trap/engines/engine.py.230: loss: 0.11760 [I 2022-10-27 15:17:52] ...optimizers/trijoint.py.151: total_norm: 17.972893 [I 2022-10-27 15:17:58] ...trap/engines/engine.py.224: train: epoch 74 | batch 320/2304 [I 2022-10-27 15:17:58] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:22 [I 2022-10-27 15:17:58] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00248 [I 2022-10-27 15:17:58] ...trap/engines/engine.py.230: loss: 0.12276 [I 2022-10-27 15:17:58] ...optimizers/trijoint.py.151: total_norm: 19.227188 [I 2022-10-27 15:18:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 330/2304 [I 2022-10-27 15:18:04] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:15 [I 2022-10-27 15:18:04] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00254 [I 2022-10-27 15:18:04] ...trap/engines/engine.py.230: loss: 0.15104 [I 2022-10-27 15:18:04] ...optimizers/trijoint.py.151: total_norm: 14.521042 [I 2022-10-27 15:18:09] ...trap/engines/engine.py.224: train: epoch 74 | batch 340/2304 [I 2022-10-27 15:18:09] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:12 [I 2022-10-27 15:18:09] ...trap/engines/engine.py.229: process: 0.56311 | load: 0.00253 [I 2022-10-27 15:18:09] ...trap/engines/engine.py.230: loss: 0.15802 [I 2022-10-27 15:18:09] ...optimizers/trijoint.py.151: total_norm: 9.493821 [I 2022-10-27 15:18:15] ...trap/engines/engine.py.224: train: epoch 74 | batch 350/2304 [I 2022-10-27 15:18:15] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:10 [I 2022-10-27 15:18:15] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00260 [I 2022-10-27 15:18:15] ...trap/engines/engine.py.230: loss: 0.15445 [I 2022-10-27 15:18:15] ...optimizers/trijoint.py.151: total_norm: 33.319988 [I 2022-10-27 15:18:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 360/2304 [I 2022-10-27 15:18:20] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:04 [I 2022-10-27 15:18:20] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00250 [I 2022-10-27 15:18:20] ...trap/engines/engine.py.230: loss: 0.13261 [I 2022-10-27 15:18:20] ...optimizers/trijoint.py.151: total_norm: 24.408464 [I 2022-10-27 15:18:26] ...trap/engines/engine.py.224: train: epoch 74 | batch 370/2304 [I 2022-10-27 15:18:26] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:56 [I 2022-10-27 15:18:26] ...trap/engines/engine.py.229: process: 0.55461 | load: 0.00246 [I 2022-10-27 15:18:26] ...trap/engines/engine.py.230: loss: 0.17024 [I 2022-10-27 15:18:26] ...optimizers/trijoint.py.151: total_norm: 9.122508 [I 2022-10-27 15:18:32] ...trap/engines/engine.py.224: train: epoch 74 | batch 380/2304 [I 2022-10-27 15:18:32] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:54 [I 2022-10-27 15:18:32] ...trap/engines/engine.py.229: process: 0.56975 | load: 0.00250 [I 2022-10-27 15:18:32] ...trap/engines/engine.py.230: loss: 0.09886 [I 2022-10-27 15:18:32] ...optimizers/trijoint.py.151: total_norm: 20.053024 [I 2022-10-27 15:18:37] ...trap/engines/engine.py.224: train: epoch 74 | batch 390/2304 [I 2022-10-27 15:18:37] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:46 [I 2022-10-27 15:18:37] ...trap/engines/engine.py.229: process: 0.56134 | load: 0.00246 [I 2022-10-27 15:18:37] ...trap/engines/engine.py.230: loss: 0.13177 [I 2022-10-27 15:18:37] ...optimizers/trijoint.py.151: total_norm: 13.689931 [I 2022-10-27 15:19:03] ...trap/engines/engine.py.224: train: epoch 74 | batch 400/2304 [I 2022-10-27 15:19:03] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 1:23:06 [I 2022-10-27 15:19:03] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00239 [I 2022-10-27 15:19:03] ...trap/engines/engine.py.230: loss: 0.13010 [I 2022-10-27 15:19:03] ...optimizers/trijoint.py.151: total_norm: 13.302679 [I 2022-10-27 15:19:08] ...trap/engines/engine.py.224: train: epoch 74 | batch 410/2304 [I 2022-10-27 15:19:08] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:24:42 [I 2022-10-27 15:19:08] ...trap/engines/engine.py.229: process: 0.56624 | load: 0.00239 [I 2022-10-27 15:19:08] ...trap/engines/engine.py.230: loss: 0.15543 [I 2022-10-27 15:19:08] ...optimizers/trijoint.py.151: total_norm: 26.629875 [I 2022-10-27 15:19:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 420/2304 [I 2022-10-27 15:19:14] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:18:10 [I 2022-10-27 15:19:14] ...trap/engines/engine.py.229: process: 0.54424 | load: 0.00246 [I 2022-10-27 15:19:14] ...trap/engines/engine.py.230: loss: 0.16275 [I 2022-10-27 15:19:14] ...optimizers/trijoint.py.151: total_norm: 21.286299 [I 2022-10-27 15:19:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 430/2304 [I 2022-10-27 15:19:20] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:17:23 [I 2022-10-27 15:19:20] ...trap/engines/engine.py.229: process: 0.54662 | load: 0.00261 [I 2022-10-27 15:19:20] ...trap/engines/engine.py.230: loss: 0.13967 [I 2022-10-27 15:19:20] ...optimizers/trijoint.py.151: total_norm: 12.161613 [I 2022-10-27 15:19:25] ...trap/engines/engine.py.224: train: epoch 74 | batch 440/2304 [I 2022-10-27 15:19:25] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:17:25 [I 2022-10-27 15:19:25] ...trap/engines/engine.py.229: process: 0.56597 | load: 0.00243 [I 2022-10-27 15:19:25] ...trap/engines/engine.py.230: loss: 0.13902 [I 2022-10-27 15:19:25] ...optimizers/trijoint.py.151: total_norm: 26.637495 [I 2022-10-27 15:19:31] ...trap/engines/engine.py.224: train: epoch 74 | batch 450/2304 [I 2022-10-27 15:19:31] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:17:13 [I 2022-10-27 15:19:31] ...trap/engines/engine.py.229: process: 0.54234 | load: 0.00238 [I 2022-10-27 15:19:31] ...trap/engines/engine.py.230: loss: 0.11558 [I 2022-10-27 15:19:31] ...optimizers/trijoint.py.151: total_norm: 13.570224 [I 2022-10-27 15:19:36] ...trap/engines/engine.py.224: train: epoch 74 | batch 460/2304 [I 2022-10-27 15:19:36] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:17:04 [I 2022-10-27 15:19:36] ...trap/engines/engine.py.229: process: 0.54656 | load: 0.00241 [I 2022-10-27 15:19:36] ...trap/engines/engine.py.230: loss: 0.14824 [I 2022-10-27 15:19:36] ...optimizers/trijoint.py.151: total_norm: 20.038460 [I 2022-10-27 15:19:42] ...trap/engines/engine.py.224: train: epoch 74 | batch 470/2304 [I 2022-10-27 15:19:42] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:17:07 [I 2022-10-27 15:19:42] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00240 [I 2022-10-27 15:19:42] ...trap/engines/engine.py.230: loss: 0.24478 [I 2022-10-27 15:19:42] ...optimizers/trijoint.py.151: total_norm: 41.635166 [I 2022-10-27 15:19:48] ...trap/engines/engine.py.224: train: epoch 74 | batch 480/2304 [I 2022-10-27 15:19:48] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:16:52 [I 2022-10-27 15:19:48] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00252 [I 2022-10-27 15:19:48] ...trap/engines/engine.py.230: loss: 0.16313 [I 2022-10-27 15:19:48] ...optimizers/trijoint.py.151: total_norm: 15.781358 [I 2022-10-27 15:19:53] ...trap/engines/engine.py.224: train: epoch 74 | batch 490/2304 [I 2022-10-27 15:19:53] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:16:54 [I 2022-10-27 15:19:53] ...trap/engines/engine.py.229: process: 0.57027 | load: 0.00245 [I 2022-10-27 15:19:53] ...trap/engines/engine.py.230: loss: 0.20076 [I 2022-10-27 15:19:53] ...optimizers/trijoint.py.151: total_norm: 12.718853 [I 2022-10-27 15:19:59] ...trap/engines/engine.py.224: train: epoch 74 | batch 500/2304 [I 2022-10-27 15:19:59] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:17:02 [I 2022-10-27 15:19:59] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00243 [I 2022-10-27 15:19:59] ...trap/engines/engine.py.230: loss: 0.13933 [I 2022-10-27 15:19:59] ...optimizers/trijoint.py.151: total_norm: 17.565540 [I 2022-10-27 15:20:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 510/2304 [I 2022-10-27 15:20:04] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:16:39 [I 2022-10-27 15:20:04] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00248 [I 2022-10-27 15:20:04] ...trap/engines/engine.py.230: loss: 0.13687 [I 2022-10-27 15:20:04] ...optimizers/trijoint.py.151: total_norm: 19.308006 [I 2022-10-27 15:20:10] ...trap/engines/engine.py.224: train: epoch 74 | batch 520/2304 [I 2022-10-27 15:20:10] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:16:36 [I 2022-10-27 15:20:10] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00251 [I 2022-10-27 15:20:10] ...trap/engines/engine.py.230: loss: 0.13877 [I 2022-10-27 15:20:10] ...optimizers/trijoint.py.151: total_norm: 23.032082 [I 2022-10-27 15:20:16] ...trap/engines/engine.py.224: train: epoch 74 | batch 530/2304 [I 2022-10-27 15:20:16] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:16:39 [I 2022-10-27 15:20:16] ...trap/engines/engine.py.229: process: 0.57526 | load: 0.00241 [I 2022-10-27 15:20:16] ...trap/engines/engine.py.230: loss: 0.14215 [I 2022-10-27 15:20:16] ...optimizers/trijoint.py.151: total_norm: 27.440807 [I 2022-10-27 15:20:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 540/2304 [I 2022-10-27 15:20:21] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:16:25 [I 2022-10-27 15:20:21] ...trap/engines/engine.py.229: process: 0.55118 | load: 0.00250 [I 2022-10-27 15:20:21] ...trap/engines/engine.py.230: loss: 0.18539 [I 2022-10-27 15:20:21] ...optimizers/trijoint.py.151: total_norm: 32.195534 [I 2022-10-27 15:20:27] ...trap/engines/engine.py.224: train: epoch 74 | batch 550/2304 [I 2022-10-27 15:20:27] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:16:23 [I 2022-10-27 15:20:27] ...trap/engines/engine.py.229: process: 0.55881 | load: 0.00248 [I 2022-10-27 15:20:27] ...trap/engines/engine.py.230: loss: 0.14705 [I 2022-10-27 15:20:27] ...optimizers/trijoint.py.151: total_norm: 20.537788 [I 2022-10-27 15:20:32] ...trap/engines/engine.py.224: train: epoch 74 | batch 560/2304 [I 2022-10-27 15:20:32] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:16:14 [I 2022-10-27 15:20:32] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00244 [I 2022-10-27 15:20:32] ...trap/engines/engine.py.230: loss: 0.11896 [I 2022-10-27 15:20:32] ...optimizers/trijoint.py.151: total_norm: 20.193274 [I 2022-10-27 15:20:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 570/2304 [I 2022-10-27 15:20:38] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:16:03 [I 2022-10-27 15:20:38] ...trap/engines/engine.py.229: process: 0.54937 | load: 0.00251 [I 2022-10-27 15:20:38] ...trap/engines/engine.py.230: loss: 0.15382 [I 2022-10-27 15:20:38] ...optimizers/trijoint.py.151: total_norm: 16.959377 [I 2022-10-27 15:20:44] ...trap/engines/engine.py.224: train: epoch 74 | batch 580/2304 [I 2022-10-27 15:20:44] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:15:57 [I 2022-10-27 15:20:44] ...trap/engines/engine.py.229: process: 0.55262 | load: 0.00241 [I 2022-10-27 15:20:44] ...trap/engines/engine.py.230: loss: 0.21812 [I 2022-10-27 15:20:44] ...optimizers/trijoint.py.151: total_norm: 34.149403 [I 2022-10-27 15:20:49] ...trap/engines/engine.py.224: train: epoch 74 | batch 590/2304 [I 2022-10-27 15:20:49] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:15:52 [I 2022-10-27 15:20:49] ...trap/engines/engine.py.229: process: 0.55419 | load: 0.00243 [I 2022-10-27 15:20:49] ...trap/engines/engine.py.230: loss: 0.12984 [I 2022-10-27 15:20:49] ...optimizers/trijoint.py.151: total_norm: 40.697632 [I 2022-10-27 15:20:55] ...trap/engines/engine.py.224: train: epoch 74 | batch 600/2304 [I 2022-10-27 15:20:55] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:52 [I 2022-10-27 15:20:55] ...trap/engines/engine.py.229: process: 0.56420 | load: 0.00242 [I 2022-10-27 15:20:55] ...trap/engines/engine.py.230: loss: 0.14044 [I 2022-10-27 15:20:55] ...optimizers/trijoint.py.151: total_norm: 20.453779 [I 2022-10-27 15:21:00] ...trap/engines/engine.py.224: train: epoch 74 | batch 610/2304 [I 2022-10-27 15:21:00] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:15:46 [I 2022-10-27 15:21:00] ...trap/engines/engine.py.229: process: 0.56550 | load: 0.00277 [I 2022-10-27 15:21:00] ...trap/engines/engine.py.230: loss: 0.18957 [I 2022-10-27 15:21:00] ...optimizers/trijoint.py.151: total_norm: 19.130335 [I 2022-10-27 15:21:06] ...trap/engines/engine.py.224: train: epoch 74 | batch 620/2304 [I 2022-10-27 15:21:06] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:47 [I 2022-10-27 15:21:06] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00262 [I 2022-10-27 15:21:06] ...trap/engines/engine.py.230: loss: 0.10551 [I 2022-10-27 15:21:06] ...optimizers/trijoint.py.151: total_norm: 20.047770 [I 2022-10-27 15:21:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 630/2304 [I 2022-10-27 15:21:12] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:37 [I 2022-10-27 15:21:12] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00248 [I 2022-10-27 15:21:12] ...trap/engines/engine.py.230: loss: 0.21533 [I 2022-10-27 15:21:12] ...optimizers/trijoint.py.151: total_norm: 13.571514 [I 2022-10-27 15:21:17] ...trap/engines/engine.py.224: train: epoch 74 | batch 640/2304 [I 2022-10-27 15:21:17] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:22 [I 2022-10-27 15:21:17] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00257 [I 2022-10-27 15:21:17] ...trap/engines/engine.py.230: loss: 0.10705 [I 2022-10-27 15:21:17] ...optimizers/trijoint.py.151: total_norm: 10.483027 [I 2022-10-27 15:21:23] ...trap/engines/engine.py.224: train: epoch 74 | batch 650/2304 [I 2022-10-27 15:21:23] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:26 [I 2022-10-27 15:21:23] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00250 [I 2022-10-27 15:21:23] ...trap/engines/engine.py.230: loss: 0.14478 [I 2022-10-27 15:21:23] ...optimizers/trijoint.py.151: total_norm: 31.185581 [I 2022-10-27 15:21:29] ...trap/engines/engine.py.224: train: epoch 74 | batch 660/2304 [I 2022-10-27 15:21:29] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:15:12 [I 2022-10-27 15:21:29] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00250 [I 2022-10-27 15:21:29] ...trap/engines/engine.py.230: loss: 0.13318 [I 2022-10-27 15:21:29] ...optimizers/trijoint.py.151: total_norm: 19.060211 [I 2022-10-27 15:21:34] ...trap/engines/engine.py.224: train: epoch 74 | batch 670/2304 [I 2022-10-27 15:21:34] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:14 [I 2022-10-27 15:21:34] ...trap/engines/engine.py.229: process: 0.56186 | load: 0.00245 [I 2022-10-27 15:21:34] ...trap/engines/engine.py.230: loss: 0.15211 [I 2022-10-27 15:21:34] ...optimizers/trijoint.py.151: total_norm: 13.258406 [I 2022-10-27 15:21:40] ...trap/engines/engine.py.224: train: epoch 74 | batch 680/2304 [I 2022-10-27 15:21:40] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:14 [I 2022-10-27 15:21:40] ...trap/engines/engine.py.229: process: 0.56943 | load: 0.00253 [I 2022-10-27 15:21:40] ...trap/engines/engine.py.230: loss: 0.13474 [I 2022-10-27 15:21:40] ...optimizers/trijoint.py.151: total_norm: 11.031123 [I 2022-10-27 15:21:45] ...trap/engines/engine.py.224: train: epoch 74 | batch 690/2304 [I 2022-10-27 15:21:45] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:15:02 [I 2022-10-27 15:21:45] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00253 [I 2022-10-27 15:21:45] ...trap/engines/engine.py.230: loss: 0.11473 [I 2022-10-27 15:21:45] ...optimizers/trijoint.py.151: total_norm: 19.265261 [I 2022-10-27 15:21:51] ...trap/engines/engine.py.224: train: epoch 74 | batch 700/2304 [I 2022-10-27 15:21:51] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:14:43 [I 2022-10-27 15:21:51] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00243 [I 2022-10-27 15:21:51] ...trap/engines/engine.py.230: loss: 0.11221 [I 2022-10-27 15:21:51] ...optimizers/trijoint.py.151: total_norm: 14.942183 [I 2022-10-27 15:21:56] ...trap/engines/engine.py.224: train: epoch 74 | batch 710/2304 [I 2022-10-27 15:21:56] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:45 [I 2022-10-27 15:21:56] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00248 [I 2022-10-27 15:21:56] ...trap/engines/engine.py.230: loss: 0.16491 [I 2022-10-27 15:21:56] ...optimizers/trijoint.py.151: total_norm: 16.438128 [I 2022-10-27 15:22:02] ...trap/engines/engine.py.224: train: epoch 74 | batch 720/2304 [I 2022-10-27 15:22:02] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:45 [I 2022-10-27 15:22:02] ...trap/engines/engine.py.229: process: 0.56584 | load: 0.00241 [I 2022-10-27 15:22:02] ...trap/engines/engine.py.230: loss: 0.15909 [I 2022-10-27 15:22:02] ...optimizers/trijoint.py.151: total_norm: 45.271786 [I 2022-10-27 15:22:08] ...trap/engines/engine.py.224: train: epoch 74 | batch 730/2304 [I 2022-10-27 15:22:08] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:38 [I 2022-10-27 15:22:08] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00246 [I 2022-10-27 15:22:08] ...trap/engines/engine.py.230: loss: 0.23387 [I 2022-10-27 15:22:08] ...optimizers/trijoint.py.151: total_norm: 27.105616 [I 2022-10-27 15:22:13] ...trap/engines/engine.py.224: train: epoch 74 | batch 740/2304 [I 2022-10-27 15:22:13] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:27 [I 2022-10-27 15:22:13] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00240 [I 2022-10-27 15:22:13] ...trap/engines/engine.py.230: loss: 0.09771 [I 2022-10-27 15:22:13] ...optimizers/trijoint.py.151: total_norm: 26.885935 [I 2022-10-27 15:22:19] ...trap/engines/engine.py.224: train: epoch 74 | batch 750/2304 [I 2022-10-27 15:22:19] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:31 [I 2022-10-27 15:22:19] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00243 [I 2022-10-27 15:22:19] ...trap/engines/engine.py.230: loss: 0.11882 [I 2022-10-27 15:22:19] ...optimizers/trijoint.py.151: total_norm: 6.783885 [I 2022-10-27 15:22:24] ...trap/engines/engine.py.224: train: epoch 74 | batch 760/2304 [I 2022-10-27 15:22:24] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:14:22 [I 2022-10-27 15:22:24] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00244 [I 2022-10-27 15:22:24] ...trap/engines/engine.py.230: loss: 0.18666 [I 2022-10-27 15:22:24] ...optimizers/trijoint.py.151: total_norm: 27.682472 [I 2022-10-27 15:22:30] ...trap/engines/engine.py.224: train: epoch 74 | batch 770/2304 [I 2022-10-27 15:22:30] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:14:10 [I 2022-10-27 15:22:30] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00252 [I 2022-10-27 15:22:30] ...trap/engines/engine.py.230: loss: 0.09056 [I 2022-10-27 15:22:30] ...optimizers/trijoint.py.151: total_norm: 14.250069 [I 2022-10-27 15:22:36] ...trap/engines/engine.py.224: train: epoch 74 | batch 780/2304 [I 2022-10-27 15:22:36] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:11 [I 2022-10-27 15:22:36] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00243 [I 2022-10-27 15:22:36] ...trap/engines/engine.py.230: loss: 0.16405 [I 2022-10-27 15:22:36] ...optimizers/trijoint.py.151: total_norm: 22.307978 [I 2022-10-27 15:22:41] ...trap/engines/engine.py.224: train: epoch 74 | batch 790/2304 [I 2022-10-27 15:22:41] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:14:08 [I 2022-10-27 15:22:41] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00244 [I 2022-10-27 15:22:41] ...trap/engines/engine.py.230: loss: 0.24772 [I 2022-10-27 15:22:41] ...optimizers/trijoint.py.151: total_norm: 42.187027 [I 2022-10-27 15:22:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 800/2304 [I 2022-10-27 15:22:47] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:14:00 [I 2022-10-27 15:22:47] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00260 [I 2022-10-27 15:22:47] ...trap/engines/engine.py.230: loss: 0.11140 [I 2022-10-27 15:22:47] ...optimizers/trijoint.py.151: total_norm: 9.292410 [I 2022-10-27 15:22:52] ...trap/engines/engine.py.224: train: epoch 74 | batch 810/2304 [I 2022-10-27 15:22:52] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:13:56 [I 2022-10-27 15:22:52] ...trap/engines/engine.py.229: process: 0.56717 | load: 0.00248 [I 2022-10-27 15:22:52] ...trap/engines/engine.py.230: loss: 0.12636 [I 2022-10-27 15:22:52] ...optimizers/trijoint.py.151: total_norm: 15.280477 [I 2022-10-27 15:22:58] ...trap/engines/engine.py.224: train: epoch 74 | batch 820/2304 [I 2022-10-27 15:22:58] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:13:49 [I 2022-10-27 15:22:58] ...trap/engines/engine.py.229: process: 0.55168 | load: 0.00249 [I 2022-10-27 15:22:58] ...trap/engines/engine.py.230: loss: 0.10292 [I 2022-10-27 15:22:58] ...optimizers/trijoint.py.151: total_norm: 10.069458 [I 2022-10-27 15:23:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 830/2304 [I 2022-10-27 15:23:04] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:48 [I 2022-10-27 15:23:04] ...trap/engines/engine.py.229: process: 0.57414 | load: 0.00252 [I 2022-10-27 15:23:04] ...trap/engines/engine.py.230: loss: 0.12029 [I 2022-10-27 15:23:04] ...optimizers/trijoint.py.151: total_norm: 22.737474 [I 2022-10-27 15:23:09] ...trap/engines/engine.py.224: train: epoch 74 | batch 840/2304 [I 2022-10-27 15:23:09] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:13:34 [I 2022-10-27 15:23:09] ...trap/engines/engine.py.229: process: 0.55214 | load: 0.00248 [I 2022-10-27 15:23:09] ...trap/engines/engine.py.230: loss: 0.17676 [I 2022-10-27 15:23:09] ...optimizers/trijoint.py.151: total_norm: 29.098351 [I 2022-10-27 15:23:15] ...trap/engines/engine.py.224: train: epoch 74 | batch 850/2304 [I 2022-10-27 15:23:15] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:31 [I 2022-10-27 15:23:15] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00249 [I 2022-10-27 15:23:15] ...trap/engines/engine.py.230: loss: 0.16856 [I 2022-10-27 15:23:15] ...optimizers/trijoint.py.151: total_norm: 12.675478 [I 2022-10-27 15:23:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 860/2304 [I 2022-10-27 15:23:20] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:13:25 [I 2022-10-27 15:23:20] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00245 [I 2022-10-27 15:23:20] ...trap/engines/engine.py.230: loss: 0.12006 [I 2022-10-27 15:23:20] ...optimizers/trijoint.py.151: total_norm: 14.118401 [I 2022-10-27 15:23:26] ...trap/engines/engine.py.224: train: epoch 74 | batch 870/2304 [I 2022-10-27 15:23:26] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:13:18 [I 2022-10-27 15:23:26] ...trap/engines/engine.py.229: process: 0.55658 | load: 0.00245 [I 2022-10-27 15:23:26] ...trap/engines/engine.py.230: loss: 0.10934 [I 2022-10-27 15:23:26] ...optimizers/trijoint.py.151: total_norm: 15.855390 [I 2022-10-27 15:23:32] ...trap/engines/engine.py.224: train: epoch 74 | batch 880/2304 [I 2022-10-27 15:23:32] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:16 [I 2022-10-27 15:23:32] ...trap/engines/engine.py.229: process: 0.56431 | load: 0.00245 [I 2022-10-27 15:23:32] ...trap/engines/engine.py.230: loss: 0.13998 [I 2022-10-27 15:23:32] ...optimizers/trijoint.py.151: total_norm: 13.289682 [I 2022-10-27 15:23:37] ...trap/engines/engine.py.224: train: epoch 74 | batch 890/2304 [I 2022-10-27 15:23:37] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:13:06 [I 2022-10-27 15:23:37] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00243 [I 2022-10-27 15:23:37] ...trap/engines/engine.py.230: loss: 0.16857 [I 2022-10-27 15:23:37] ...optimizers/trijoint.py.151: total_norm: 27.658875 [I 2022-10-27 15:23:43] ...trap/engines/engine.py.224: train: epoch 74 | batch 900/2304 [I 2022-10-27 15:23:43] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:13:02 [I 2022-10-27 15:23:43] ...trap/engines/engine.py.229: process: 0.56164 | load: 0.00251 [I 2022-10-27 15:23:43] ...trap/engines/engine.py.230: loss: 0.09431 [I 2022-10-27 15:23:43] ...optimizers/trijoint.py.151: total_norm: 9.208309 [I 2022-10-27 15:23:48] ...trap/engines/engine.py.224: train: epoch 74 | batch 910/2304 [I 2022-10-27 15:23:48] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:12:55 [I 2022-10-27 15:23:48] ...trap/engines/engine.py.229: process: 0.54328 | load: 0.00246 [I 2022-10-27 15:23:48] ...trap/engines/engine.py.230: loss: 0.16194 [I 2022-10-27 15:23:48] ...optimizers/trijoint.py.151: total_norm: 20.428038 [I 2022-10-27 15:23:54] ...trap/engines/engine.py.224: train: epoch 74 | batch 920/2304 [I 2022-10-27 15:23:54] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:56 [I 2022-10-27 15:23:54] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00249 [I 2022-10-27 15:23:54] ...trap/engines/engine.py.230: loss: 0.13383 [I 2022-10-27 15:23:54] ...optimizers/trijoint.py.151: total_norm: 21.645998 [I 2022-10-27 15:24:00] ...trap/engines/engine.py.224: train: epoch 74 | batch 930/2304 [I 2022-10-27 15:24:00] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:45 [I 2022-10-27 15:24:00] ...trap/engines/engine.py.229: process: 0.56766 | load: 0.00250 [I 2022-10-27 15:24:00] ...trap/engines/engine.py.230: loss: 0.14169 [I 2022-10-27 15:24:00] ...optimizers/trijoint.py.151: total_norm: 25.329819 [I 2022-10-27 15:24:05] ...trap/engines/engine.py.224: train: epoch 74 | batch 940/2304 [I 2022-10-27 15:24:05] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:12:42 [I 2022-10-27 15:24:05] ...trap/engines/engine.py.229: process: 0.55968 | load: 0.00247 [I 2022-10-27 15:24:05] ...trap/engines/engine.py.230: loss: 0.13943 [I 2022-10-27 15:24:05] ...optimizers/trijoint.py.151: total_norm: 17.522966 [I 2022-10-27 15:24:11] ...trap/engines/engine.py.224: train: epoch 74 | batch 950/2304 [I 2022-10-27 15:24:11] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:37 [I 2022-10-27 15:24:11] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00243 [I 2022-10-27 15:24:11] ...trap/engines/engine.py.230: loss: 0.09348 [I 2022-10-27 15:24:11] ...optimizers/trijoint.py.151: total_norm: 15.514870 [I 2022-10-27 15:24:16] ...trap/engines/engine.py.224: train: epoch 74 | batch 960/2304 [I 2022-10-27 15:24:16] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:12:28 [I 2022-10-27 15:24:16] ...trap/engines/engine.py.229: process: 0.55466 | load: 0.00244 [I 2022-10-27 15:24:16] ...trap/engines/engine.py.230: loss: 0.11631 [I 2022-10-27 15:24:16] ...optimizers/trijoint.py.151: total_norm: 11.645223 [I 2022-10-27 15:24:22] ...trap/engines/engine.py.224: train: epoch 74 | batch 970/2304 [I 2022-10-27 15:24:22] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:12:35 [I 2022-10-27 15:24:22] ...trap/engines/engine.py.229: process: 0.57573 | load: 0.00246 [I 2022-10-27 15:24:22] ...trap/engines/engine.py.230: loss: 0.14275 [I 2022-10-27 15:24:22] ...optimizers/trijoint.py.151: total_norm: 17.396830 [I 2022-10-27 15:24:28] ...trap/engines/engine.py.224: train: epoch 74 | batch 980/2304 [I 2022-10-27 15:24:28] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:21 [I 2022-10-27 15:24:28] ...trap/engines/engine.py.229: process: 0.56614 | load: 0.00242 [I 2022-10-27 15:24:28] ...trap/engines/engine.py.230: loss: 0.15673 [I 2022-10-27 15:24:28] ...optimizers/trijoint.py.151: total_norm: 26.302820 [I 2022-10-27 15:24:33] ...trap/engines/engine.py.224: train: epoch 74 | batch 990/2304 [I 2022-10-27 15:24:33] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:12:12 [I 2022-10-27 15:24:33] ...trap/engines/engine.py.229: process: 0.54729 | load: 0.00247 [I 2022-10-27 15:24:33] ...trap/engines/engine.py.230: loss: 0.15393 [I 2022-10-27 15:24:33] ...optimizers/trijoint.py.151: total_norm: 13.899608 [I 2022-10-27 15:24:39] ...trap/engines/engine.py.224: train: epoch 74 | batch 1000/2304 [I 2022-10-27 15:24:39] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:07 [I 2022-10-27 15:24:39] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00242 [I 2022-10-27 15:24:39] ...trap/engines/engine.py.230: loss: 0.15248 [I 2022-10-27 15:24:39] ...optimizers/trijoint.py.151: total_norm: 31.289736 [I 2022-10-27 15:24:44] ...trap/engines/engine.py.224: train: epoch 74 | batch 1010/2304 [I 2022-10-27 15:24:44] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:12:02 [I 2022-10-27 15:24:44] ...trap/engines/engine.py.229: process: 0.56845 | load: 0.00242 [I 2022-10-27 15:24:44] ...trap/engines/engine.py.230: loss: 0.13180 [I 2022-10-27 15:24:44] ...optimizers/trijoint.py.151: total_norm: 16.104771 [I 2022-10-27 15:24:50] ...trap/engines/engine.py.224: train: epoch 74 | batch 1020/2304 [I 2022-10-27 15:24:50] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:59 [I 2022-10-27 15:24:50] ...trap/engines/engine.py.229: process: 0.56292 | load: 0.00244 [I 2022-10-27 15:24:50] ...trap/engines/engine.py.230: loss: 0.14612 [I 2022-10-27 15:24:50] ...optimizers/trijoint.py.151: total_norm: 9.828703 [I 2022-10-27 15:24:56] ...trap/engines/engine.py.224: train: epoch 74 | batch 1030/2304 [I 2022-10-27 15:24:56] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:56 [I 2022-10-27 15:24:56] ...trap/engines/engine.py.229: process: 0.56707 | load: 0.00256 [I 2022-10-27 15:24:56] ...trap/engines/engine.py.230: loss: 0.15803 [I 2022-10-27 15:24:56] ...optimizers/trijoint.py.151: total_norm: 15.876088 [I 2022-10-27 15:25:01] ...trap/engines/engine.py.224: train: epoch 74 | batch 1040/2304 [I 2022-10-27 15:25:01] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:47 [I 2022-10-27 15:25:01] ...trap/engines/engine.py.229: process: 0.54834 | load: 0.00250 [I 2022-10-27 15:25:01] ...trap/engines/engine.py.230: loss: 0.15459 [I 2022-10-27 15:25:01] ...optimizers/trijoint.py.151: total_norm: 12.192112 [I 2022-10-27 15:25:07] ...trap/engines/engine.py.224: train: epoch 74 | batch 1050/2304 [I 2022-10-27 15:25:07] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:39 [I 2022-10-27 15:25:07] ...trap/engines/engine.py.229: process: 0.57518 | load: 0.00248 [I 2022-10-27 15:25:07] ...trap/engines/engine.py.230: loss: 0.14055 [I 2022-10-27 15:25:07] ...optimizers/trijoint.py.151: total_norm: 13.330931 [I 2022-10-27 15:25:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 1060/2304 [I 2022-10-27 15:25:12] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:31 [I 2022-10-27 15:25:12] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00248 [I 2022-10-27 15:25:12] ...trap/engines/engine.py.230: loss: 0.11591 [I 2022-10-27 15:25:12] ...optimizers/trijoint.py.151: total_norm: 11.819852 [I 2022-10-27 15:25:18] ...trap/engines/engine.py.224: train: epoch 74 | batch 1070/2304 [I 2022-10-27 15:25:18] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:32 [I 2022-10-27 15:25:18] ...trap/engines/engine.py.229: process: 0.56490 | load: 0.00247 [I 2022-10-27 15:25:18] ...trap/engines/engine.py.230: loss: 0.17846 [I 2022-10-27 15:25:18] ...optimizers/trijoint.py.151: total_norm: 25.452343 [I 2022-10-27 15:25:24] ...trap/engines/engine.py.224: train: epoch 74 | batch 1080/2304 [I 2022-10-27 15:25:24] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:23 [I 2022-10-27 15:25:24] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00246 [I 2022-10-27 15:25:24] ...trap/engines/engine.py.230: loss: 0.15290 [I 2022-10-27 15:25:24] ...optimizers/trijoint.py.151: total_norm: 28.267084 [I 2022-10-27 15:25:29] ...trap/engines/engine.py.224: train: epoch 74 | batch 1090/2304 [I 2022-10-27 15:25:29] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:18 [I 2022-10-27 15:25:29] ...trap/engines/engine.py.229: process: 0.54832 | load: 0.00248 [I 2022-10-27 15:25:29] ...trap/engines/engine.py.230: loss: 0.20384 [I 2022-10-27 15:25:29] ...optimizers/trijoint.py.151: total_norm: 19.872396 [I 2022-10-27 15:25:35] ...trap/engines/engine.py.224: train: epoch 74 | batch 1100/2304 [I 2022-10-27 15:25:35] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:11:13 [I 2022-10-27 15:25:35] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00248 [I 2022-10-27 15:25:35] ...trap/engines/engine.py.230: loss: 0.15835 [I 2022-10-27 15:25:35] ...optimizers/trijoint.py.151: total_norm: 19.802496 [I 2022-10-27 15:25:41] ...trap/engines/engine.py.224: train: epoch 74 | batch 1110/2304 [I 2022-10-27 15:25:41] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:11:12 [I 2022-10-27 15:25:41] ...trap/engines/engine.py.229: process: 0.56556 | load: 0.00251 [I 2022-10-27 15:25:41] ...trap/engines/engine.py.230: loss: 0.17382 [I 2022-10-27 15:25:41] ...optimizers/trijoint.py.151: total_norm: 24.977661 [I 2022-10-27 15:25:46] ...trap/engines/engine.py.224: train: epoch 74 | batch 1120/2304 [I 2022-10-27 15:25:46] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:10:58 [I 2022-10-27 15:25:46] ...trap/engines/engine.py.229: process: 0.55381 | load: 0.00248 [I 2022-10-27 15:25:46] ...trap/engines/engine.py.230: loss: 0.13726 [I 2022-10-27 15:25:46] ...optimizers/trijoint.py.151: total_norm: 54.481037 [I 2022-10-27 15:25:52] ...trap/engines/engine.py.224: train: epoch 74 | batch 1130/2304 [I 2022-10-27 15:25:52] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:57 [I 2022-10-27 15:25:52] ...trap/engines/engine.py.229: process: 0.56653 | load: 0.00249 [I 2022-10-27 15:25:52] ...trap/engines/engine.py.230: loss: 0.09816 [I 2022-10-27 15:25:52] ...optimizers/trijoint.py.151: total_norm: 11.450655 [I 2022-10-27 15:25:57] ...trap/engines/engine.py.224: train: epoch 74 | batch 1140/2304 [I 2022-10-27 15:25:57] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:50 [I 2022-10-27 15:25:57] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00247 [I 2022-10-27 15:25:57] ...trap/engines/engine.py.230: loss: 0.15887 [I 2022-10-27 15:25:57] ...optimizers/trijoint.py.151: total_norm: 17.434578 [I 2022-10-27 15:26:03] ...trap/engines/engine.py.224: train: epoch 74 | batch 1150/2304 [I 2022-10-27 15:26:03] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:40 [I 2022-10-27 15:26:03] ...trap/engines/engine.py.229: process: 0.55305 | load: 0.00247 [I 2022-10-27 15:26:03] ...trap/engines/engine.py.230: loss: 0.19574 [I 2022-10-27 15:26:03] ...optimizers/trijoint.py.151: total_norm: 16.844091 [I 2022-10-27 15:26:09] ...trap/engines/engine.py.224: train: epoch 74 | batch 1160/2304 [I 2022-10-27 15:26:09] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:39 [I 2022-10-27 15:26:09] ...trap/engines/engine.py.229: process: 0.55411 | load: 0.00246 [I 2022-10-27 15:26:09] ...trap/engines/engine.py.230: loss: 0.12607 [I 2022-10-27 15:26:09] ...optimizers/trijoint.py.151: total_norm: 13.405184 [I 2022-10-27 15:26:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 1170/2304 [I 2022-10-27 15:26:14] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:46 [I 2022-10-27 15:26:14] ...trap/engines/engine.py.229: process: 0.59602 | load: 0.00243 [I 2022-10-27 15:26:14] ...trap/engines/engine.py.230: loss: 0.11661 [I 2022-10-27 15:26:14] ...optimizers/trijoint.py.151: total_norm: 13.637094 [I 2022-10-27 15:26:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 1180/2304 [I 2022-10-27 15:26:20] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:29 [I 2022-10-27 15:26:20] ...trap/engines/engine.py.229: process: 0.56377 | load: 0.00243 [I 2022-10-27 15:26:20] ...trap/engines/engine.py.230: loss: 0.10409 [I 2022-10-27 15:26:20] ...optimizers/trijoint.py.151: total_norm: 16.965868 [I 2022-10-27 15:26:26] ...trap/engines/engine.py.224: train: epoch 74 | batch 1190/2304 [I 2022-10-27 15:26:26] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:10:41 [I 2022-10-27 15:26:26] ...trap/engines/engine.py.229: process: 0.55734 | load: 0.00264 [I 2022-10-27 15:26:26] ...trap/engines/engine.py.230: loss: 0.19181 [I 2022-10-27 15:26:26] ...optimizers/trijoint.py.151: total_norm: 25.182020 [I 2022-10-27 15:26:31] ...trap/engines/engine.py.224: train: epoch 74 | batch 1200/2304 [I 2022-10-27 15:26:31] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:10:19 [I 2022-10-27 15:26:31] ...trap/engines/engine.py.229: process: 0.56433 | load: 0.00279 [I 2022-10-27 15:26:31] ...trap/engines/engine.py.230: loss: 0.12075 [I 2022-10-27 15:26:31] ...optimizers/trijoint.py.151: total_norm: 15.980492 [I 2022-10-27 15:26:37] ...trap/engines/engine.py.224: train: epoch 74 | batch 1210/2304 [I 2022-10-27 15:26:37] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:10:06 [I 2022-10-27 15:26:37] ...trap/engines/engine.py.229: process: 0.54814 | load: 0.00247 [I 2022-10-27 15:26:37] ...trap/engines/engine.py.230: loss: 0.12311 [I 2022-10-27 15:26:37] ...optimizers/trijoint.py.151: total_norm: 21.197340 [I 2022-10-27 15:26:42] ...trap/engines/engine.py.224: train: epoch 74 | batch 1220/2304 [I 2022-10-27 15:26:42] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:10:07 [I 2022-10-27 15:26:42] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00250 [I 2022-10-27 15:26:42] ...trap/engines/engine.py.230: loss: 0.24299 [I 2022-10-27 15:26:42] ...optimizers/trijoint.py.151: total_norm: 16.471546 [I 2022-10-27 15:26:48] ...trap/engines/engine.py.224: train: epoch 74 | batch 1230/2304 [I 2022-10-27 15:26:48] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:10:06 [I 2022-10-27 15:26:48] ...trap/engines/engine.py.229: process: 0.56782 | load: 0.00246 [I 2022-10-27 15:26:48] ...trap/engines/engine.py.230: loss: 0.11118 [I 2022-10-27 15:26:48] ...optimizers/trijoint.py.151: total_norm: 9.399873 [I 2022-10-27 15:26:54] ...trap/engines/engine.py.224: train: epoch 74 | batch 1240/2304 [I 2022-10-27 15:26:54] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:52 [I 2022-10-27 15:26:54] ...trap/engines/engine.py.229: process: 0.55010 | load: 0.00242 [I 2022-10-27 15:26:54] ...trap/engines/engine.py.230: loss: 0.11129 [I 2022-10-27 15:26:54] ...optimizers/trijoint.py.151: total_norm: 24.658878 [I 2022-10-27 15:26:59] ...trap/engines/engine.py.224: train: epoch 74 | batch 1250/2304 [I 2022-10-27 15:26:59] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:46 [I 2022-10-27 15:26:59] ...trap/engines/engine.py.229: process: 0.54728 | load: 0.00247 [I 2022-10-27 15:26:59] ...trap/engines/engine.py.230: loss: 0.08872 [I 2022-10-27 15:26:59] ...optimizers/trijoint.py.151: total_norm: 14.812806 [I 2022-10-27 15:27:05] ...trap/engines/engine.py.224: train: epoch 74 | batch 1260/2304 [I 2022-10-27 15:27:05] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:51 [I 2022-10-27 15:27:05] ...trap/engines/engine.py.229: process: 0.56279 | load: 0.00249 [I 2022-10-27 15:27:05] ...trap/engines/engine.py.230: loss: 0.12412 [I 2022-10-27 15:27:05] ...optimizers/trijoint.py.151: total_norm: 17.133127 [I 2022-10-27 15:27:11] ...trap/engines/engine.py.224: train: epoch 74 | batch 1270/2304 [I 2022-10-27 15:27:11] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:46 [I 2022-10-27 15:27:11] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00251 [I 2022-10-27 15:27:11] ...trap/engines/engine.py.230: loss: 0.11481 [I 2022-10-27 15:27:11] ...optimizers/trijoint.py.151: total_norm: 12.172218 [I 2022-10-27 15:27:16] ...trap/engines/engine.py.224: train: epoch 74 | batch 1280/2304 [I 2022-10-27 15:27:16] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:36 [I 2022-10-27 15:27:16] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00249 [I 2022-10-27 15:27:16] ...trap/engines/engine.py.230: loss: 0.12865 [I 2022-10-27 15:27:16] ...optimizers/trijoint.py.151: total_norm: 50.765232 [I 2022-10-27 15:27:22] ...trap/engines/engine.py.224: train: epoch 74 | batch 1290/2304 [I 2022-10-27 15:27:22] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:25 [I 2022-10-27 15:27:22] ...trap/engines/engine.py.229: process: 0.57189 | load: 0.00246 [I 2022-10-27 15:27:22] ...trap/engines/engine.py.230: loss: 0.15127 [I 2022-10-27 15:27:22] ...optimizers/trijoint.py.151: total_norm: 11.996450 [I 2022-10-27 15:27:27] ...trap/engines/engine.py.224: train: epoch 74 | batch 1300/2304 [I 2022-10-27 15:27:27] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:09:26 [I 2022-10-27 15:27:27] ...trap/engines/engine.py.229: process: 0.56233 | load: 0.00246 [I 2022-10-27 15:27:27] ...trap/engines/engine.py.230: loss: 0.12927 [I 2022-10-27 15:27:27] ...optimizers/trijoint.py.151: total_norm: 19.897303 [I 2022-10-27 15:27:33] ...trap/engines/engine.py.224: train: epoch 74 | batch 1310/2304 [I 2022-10-27 15:27:33] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:09:12 [I 2022-10-27 15:27:33] ...trap/engines/engine.py.229: process: 0.55562 | load: 0.00247 [I 2022-10-27 15:27:33] ...trap/engines/engine.py.230: loss: 0.16232 [I 2022-10-27 15:27:33] ...optimizers/trijoint.py.151: total_norm: 39.364410 [I 2022-10-27 15:27:39] ...trap/engines/engine.py.224: train: epoch 74 | batch 1320/2304 [I 2022-10-27 15:27:39] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:09:07 [I 2022-10-27 15:27:39] ...trap/engines/engine.py.229: process: 0.56718 | load: 0.00244 [I 2022-10-27 15:27:39] ...trap/engines/engine.py.230: loss: 0.14805 [I 2022-10-27 15:27:39] ...optimizers/trijoint.py.151: total_norm: 27.509283 [I 2022-10-27 15:27:44] ...trap/engines/engine.py.224: train: epoch 74 | batch 1330/2304 [I 2022-10-27 15:27:44] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:09:13 [I 2022-10-27 15:27:44] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00243 [I 2022-10-27 15:27:44] ...trap/engines/engine.py.230: loss: 0.19006 [I 2022-10-27 15:27:44] ...optimizers/trijoint.py.151: total_norm: 17.201128 [I 2022-10-27 15:27:50] ...trap/engines/engine.py.224: train: epoch 74 | batch 1340/2304 [I 2022-10-27 15:27:50] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:09:03 [I 2022-10-27 15:27:50] ...trap/engines/engine.py.229: process: 0.57033 | load: 0.00246 [I 2022-10-27 15:27:50] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-27 15:27:50] ...optimizers/trijoint.py.151: total_norm: 10.360686 [I 2022-10-27 15:27:56] ...trap/engines/engine.py.224: train: epoch 74 | batch 1350/2304 [I 2022-10-27 15:27:56] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:55 [I 2022-10-27 15:27:56] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00242 [I 2022-10-27 15:27:56] ...trap/engines/engine.py.230: loss: 0.10238 [I 2022-10-27 15:27:56] ...optimizers/trijoint.py.151: total_norm: 11.241386 [I 2022-10-27 15:28:01] ...trap/engines/engine.py.224: train: epoch 74 | batch 1360/2304 [I 2022-10-27 15:28:01] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:49 [I 2022-10-27 15:28:01] ...trap/engines/engine.py.229: process: 0.56574 | load: 0.00245 [I 2022-10-27 15:28:01] ...trap/engines/engine.py.230: loss: 0.14660 [I 2022-10-27 15:28:01] ...optimizers/trijoint.py.151: total_norm: 35.070412 [I 2022-10-27 15:28:07] ...trap/engines/engine.py.224: train: epoch 74 | batch 1370/2304 [I 2022-10-27 15:28:07] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:42 [I 2022-10-27 15:28:07] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00242 [I 2022-10-27 15:28:07] ...trap/engines/engine.py.230: loss: 0.17227 [I 2022-10-27 15:28:07] ...optimizers/trijoint.py.151: total_norm: 22.068045 [I 2022-10-27 15:28:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 1380/2304 [I 2022-10-27 15:28:12] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:40 [I 2022-10-27 15:28:12] ...trap/engines/engine.py.229: process: 0.56603 | load: 0.00244 [I 2022-10-27 15:28:12] ...trap/engines/engine.py.230: loss: 0.13470 [I 2022-10-27 15:28:12] ...optimizers/trijoint.py.151: total_norm: 13.946702 [I 2022-10-27 15:28:18] ...trap/engines/engine.py.224: train: epoch 74 | batch 1390/2304 [I 2022-10-27 15:28:18] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:36 [I 2022-10-27 15:28:18] ...trap/engines/engine.py.229: process: 0.56337 | load: 0.00243 [I 2022-10-27 15:28:18] ...trap/engines/engine.py.230: loss: 0.14895 [I 2022-10-27 15:28:18] ...optimizers/trijoint.py.151: total_norm: 26.271545 [I 2022-10-27 15:28:24] ...trap/engines/engine.py.224: train: epoch 74 | batch 1400/2304 [I 2022-10-27 15:28:24] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:25 [I 2022-10-27 15:28:24] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00253 [I 2022-10-27 15:28:24] ...trap/engines/engine.py.230: loss: 0.09831 [I 2022-10-27 15:28:24] ...optimizers/trijoint.py.151: total_norm: 12.884407 [I 2022-10-27 15:28:29] ...trap/engines/engine.py.224: train: epoch 74 | batch 1410/2304 [I 2022-10-27 15:28:29] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:08:17 [I 2022-10-27 15:28:29] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00246 [I 2022-10-27 15:28:29] ...trap/engines/engine.py.230: loss: 0.11939 [I 2022-10-27 15:28:29] ...optimizers/trijoint.py.151: total_norm: 16.846985 [I 2022-10-27 15:28:35] ...trap/engines/engine.py.224: train: epoch 74 | batch 1420/2304 [I 2022-10-27 15:28:35] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:08:14 [I 2022-10-27 15:28:35] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00244 [I 2022-10-27 15:28:35] ...trap/engines/engine.py.230: loss: 0.16158 [I 2022-10-27 15:28:35] ...optimizers/trijoint.py.151: total_norm: 25.834211 [I 2022-10-27 15:28:41] ...trap/engines/engine.py.224: train: epoch 74 | batch 1430/2304 [I 2022-10-27 15:28:41] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:08:09 [I 2022-10-27 15:28:41] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00245 [I 2022-10-27 15:28:41] ...trap/engines/engine.py.230: loss: 0.15570 [I 2022-10-27 15:28:41] ...optimizers/trijoint.py.151: total_norm: 13.830734 [I 2022-10-27 15:28:46] ...trap/engines/engine.py.224: train: epoch 74 | batch 1440/2304 [I 2022-10-27 15:28:46] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:08:02 [I 2022-10-27 15:28:46] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00246 [I 2022-10-27 15:28:46] ...trap/engines/engine.py.230: loss: 0.12120 [I 2022-10-27 15:28:46] ...optimizers/trijoint.py.151: total_norm: 11.006298 [I 2022-10-27 15:28:52] ...trap/engines/engine.py.224: train: epoch 74 | batch 1450/2304 [I 2022-10-27 15:28:52] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:08:06 [I 2022-10-27 15:28:52] ...trap/engines/engine.py.229: process: 0.56042 | load: 0.00245 [I 2022-10-27 15:28:52] ...trap/engines/engine.py.230: loss: 0.13191 [I 2022-10-27 15:28:52] ...optimizers/trijoint.py.151: total_norm: 14.204057 [I 2022-10-27 15:28:57] ...trap/engines/engine.py.224: train: epoch 74 | batch 1460/2304 [I 2022-10-27 15:28:57] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:54 [I 2022-10-27 15:28:57] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00251 [I 2022-10-27 15:28:57] ...trap/engines/engine.py.230: loss: 0.13081 [I 2022-10-27 15:28:57] ...optimizers/trijoint.py.151: total_norm: 7.978411 [I 2022-10-27 15:29:03] ...trap/engines/engine.py.224: train: epoch 74 | batch 1470/2304 [I 2022-10-27 15:29:03] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:50 [I 2022-10-27 15:29:03] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00250 [I 2022-10-27 15:29:03] ...trap/engines/engine.py.230: loss: 0.15172 [I 2022-10-27 15:29:03] ...optimizers/trijoint.py.151: total_norm: 15.679730 [I 2022-10-27 15:29:09] ...trap/engines/engine.py.224: train: epoch 74 | batch 1480/2304 [I 2022-10-27 15:29:09] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:42 [I 2022-10-27 15:29:09] ...trap/engines/engine.py.229: process: 0.56004 | load: 0.00249 [I 2022-10-27 15:29:09] ...trap/engines/engine.py.230: loss: 0.13375 [I 2022-10-27 15:29:09] ...optimizers/trijoint.py.151: total_norm: 12.668429 [I 2022-10-27 15:29:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 1490/2304 [I 2022-10-27 15:29:14] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:32 [I 2022-10-27 15:29:14] ...trap/engines/engine.py.229: process: 0.55316 | load: 0.00250 [I 2022-10-27 15:29:14] ...trap/engines/engine.py.230: loss: 0.12778 [I 2022-10-27 15:29:14] ...optimizers/trijoint.py.151: total_norm: 41.292297 [I 2022-10-27 15:29:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 1500/2304 [I 2022-10-27 15:29:20] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:26 [I 2022-10-27 15:29:20] ...trap/engines/engine.py.229: process: 0.56184 | load: 0.00246 [I 2022-10-27 15:29:20] ...trap/engines/engine.py.230: loss: 0.13972 [I 2022-10-27 15:29:20] ...optimizers/trijoint.py.151: total_norm: 15.935205 [I 2022-10-27 15:29:26] ...trap/engines/engine.py.224: train: epoch 74 | batch 1510/2304 [I 2022-10-27 15:29:26] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:34 [I 2022-10-27 15:29:26] ...trap/engines/engine.py.229: process: 0.62194 | load: 0.00243 [I 2022-10-27 15:29:26] ...trap/engines/engine.py.230: loss: 0.10433 [I 2022-10-27 15:29:26] ...optimizers/trijoint.py.151: total_norm: 23.766420 [I 2022-10-27 15:29:31] ...trap/engines/engine.py.224: train: epoch 74 | batch 1520/2304 [I 2022-10-27 15:29:31] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:07:20 [I 2022-10-27 15:29:31] ...trap/engines/engine.py.229: process: 0.56258 | load: 0.00242 [I 2022-10-27 15:29:31] ...trap/engines/engine.py.230: loss: 0.16249 [I 2022-10-27 15:29:31] ...optimizers/trijoint.py.151: total_norm: 15.510945 [I 2022-10-27 15:29:37] ...trap/engines/engine.py.224: train: epoch 74 | batch 1530/2304 [I 2022-10-27 15:29:37] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:07:11 [I 2022-10-27 15:29:37] ...trap/engines/engine.py.229: process: 0.55137 | load: 0.00243 [I 2022-10-27 15:29:37] ...trap/engines/engine.py.230: loss: 0.10357 [I 2022-10-27 15:29:37] ...optimizers/trijoint.py.151: total_norm: 11.777715 [I 2022-10-27 15:29:42] ...trap/engines/engine.py.224: train: epoch 74 | batch 1540/2304 [I 2022-10-27 15:29:42] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:07:07 [I 2022-10-27 15:29:42] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00245 [I 2022-10-27 15:29:42] ...trap/engines/engine.py.230: loss: 0.13470 [I 2022-10-27 15:29:42] ...optimizers/trijoint.py.151: total_norm: 13.061624 [I 2022-10-27 15:29:48] ...trap/engines/engine.py.224: train: epoch 74 | batch 1550/2304 [I 2022-10-27 15:29:48] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:07:01 [I 2022-10-27 15:29:48] ...trap/engines/engine.py.229: process: 0.55137 | load: 0.00254 [I 2022-10-27 15:29:48] ...trap/engines/engine.py.230: loss: 0.11304 [I 2022-10-27 15:29:48] ...optimizers/trijoint.py.151: total_norm: 22.077991 [I 2022-10-27 15:29:54] ...trap/engines/engine.py.224: train: epoch 74 | batch 1560/2304 [I 2022-10-27 15:29:54] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:54 [I 2022-10-27 15:29:54] ...trap/engines/engine.py.229: process: 0.56723 | load: 0.00245 [I 2022-10-27 15:29:54] ...trap/engines/engine.py.230: loss: 0.12547 [I 2022-10-27 15:29:54] ...optimizers/trijoint.py.151: total_norm: 23.239632 [I 2022-10-27 15:29:59] ...trap/engines/engine.py.224: train: epoch 74 | batch 1570/2304 [I 2022-10-27 15:29:59] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:51 [I 2022-10-27 15:29:59] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00251 [I 2022-10-27 15:29:59] ...trap/engines/engine.py.230: loss: 0.10938 [I 2022-10-27 15:29:59] ...optimizers/trijoint.py.151: total_norm: 30.132660 [I 2022-10-27 15:30:05] ...trap/engines/engine.py.224: train: epoch 74 | batch 1580/2304 [I 2022-10-27 15:30:05] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:50 [I 2022-10-27 15:30:05] ...trap/engines/engine.py.229: process: 0.56791 | load: 0.00248 [I 2022-10-27 15:30:05] ...trap/engines/engine.py.230: loss: 0.11031 [I 2022-10-27 15:30:05] ...optimizers/trijoint.py.151: total_norm: 34.033791 [I 2022-10-27 15:30:11] ...trap/engines/engine.py.224: train: epoch 74 | batch 1590/2304 [I 2022-10-27 15:30:11] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:39 [I 2022-10-27 15:30:11] ...trap/engines/engine.py.229: process: 0.56409 | load: 0.00248 [I 2022-10-27 15:30:11] ...trap/engines/engine.py.230: loss: 0.13022 [I 2022-10-27 15:30:11] ...optimizers/trijoint.py.151: total_norm: 15.820674 [I 2022-10-27 15:30:16] ...trap/engines/engine.py.224: train: epoch 74 | batch 1600/2304 [I 2022-10-27 15:30:16] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:33 [I 2022-10-27 15:30:16] ...trap/engines/engine.py.229: process: 0.55484 | load: 0.00246 [I 2022-10-27 15:30:16] ...trap/engines/engine.py.230: loss: 0.16701 [I 2022-10-27 15:30:16] ...optimizers/trijoint.py.151: total_norm: 19.892769 [I 2022-10-27 15:30:22] ...trap/engines/engine.py.224: train: epoch 74 | batch 1610/2304 [I 2022-10-27 15:30:22] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:27 [I 2022-10-27 15:30:22] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00249 [I 2022-10-27 15:30:22] ...trap/engines/engine.py.230: loss: 0.13691 [I 2022-10-27 15:30:22] ...optimizers/trijoint.py.151: total_norm: 8.383570 [I 2022-10-27 15:30:27] ...trap/engines/engine.py.224: train: epoch 74 | batch 1620/2304 [I 2022-10-27 15:30:27] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:21 [I 2022-10-27 15:30:27] ...trap/engines/engine.py.229: process: 0.56408 | load: 0.00243 [I 2022-10-27 15:30:27] ...trap/engines/engine.py.230: loss: 0.12748 [I 2022-10-27 15:30:27] ...optimizers/trijoint.py.151: total_norm: 12.109143 [I 2022-10-27 15:30:33] ...trap/engines/engine.py.224: train: epoch 74 | batch 1630/2304 [I 2022-10-27 15:30:33] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:17 [I 2022-10-27 15:30:33] ...trap/engines/engine.py.229: process: 0.56609 | load: 0.00254 [I 2022-10-27 15:30:33] ...trap/engines/engine.py.230: loss: 0.10575 [I 2022-10-27 15:30:33] ...optimizers/trijoint.py.151: total_norm: 28.170443 [I 2022-10-27 15:30:39] ...trap/engines/engine.py.224: train: epoch 74 | batch 1640/2304 [I 2022-10-27 15:30:39] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:06:18 [I 2022-10-27 15:30:39] ...trap/engines/engine.py.229: process: 0.56779 | load: 0.00253 [I 2022-10-27 15:30:39] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-27 15:30:39] ...optimizers/trijoint.py.151: total_norm: 13.649956 [I 2022-10-27 15:30:44] ...trap/engines/engine.py.224: train: epoch 74 | batch 1650/2304 [I 2022-10-27 15:30:44] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:06:08 [I 2022-10-27 15:30:44] ...trap/engines/engine.py.229: process: 0.56896 | load: 0.00245 [I 2022-10-27 15:30:44] ...trap/engines/engine.py.230: loss: 0.17651 [I 2022-10-27 15:30:44] ...optimizers/trijoint.py.151: total_norm: 22.912903 [I 2022-10-27 15:30:50] ...trap/engines/engine.py.224: train: epoch 74 | batch 1660/2304 [I 2022-10-27 15:30:50] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:06:00 [I 2022-10-27 15:30:50] ...trap/engines/engine.py.229: process: 0.55638 | load: 0.00246 [I 2022-10-27 15:30:50] ...trap/engines/engine.py.230: loss: 0.08650 [I 2022-10-27 15:30:50] ...optimizers/trijoint.py.151: total_norm: 8.204190 [I 2022-10-27 15:30:56] ...trap/engines/engine.py.224: train: epoch 74 | batch 1670/2304 [I 2022-10-27 15:30:56] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:56 [I 2022-10-27 15:30:56] ...trap/engines/engine.py.229: process: 0.57479 | load: 0.00242 [I 2022-10-27 15:30:56] ...trap/engines/engine.py.230: loss: 0.15972 [I 2022-10-27 15:30:56] ...optimizers/trijoint.py.151: total_norm: 17.238127 [I 2022-10-27 15:31:01] ...trap/engines/engine.py.224: train: epoch 74 | batch 1680/2304 [I 2022-10-27 15:31:01] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:45 [I 2022-10-27 15:31:01] ...trap/engines/engine.py.229: process: 0.54072 | load: 0.00251 [I 2022-10-27 15:31:01] ...trap/engines/engine.py.230: loss: 0.11164 [I 2022-10-27 15:31:01] ...optimizers/trijoint.py.151: total_norm: 7.912131 [I 2022-10-27 15:31:07] ...trap/engines/engine.py.224: train: epoch 74 | batch 1690/2304 [I 2022-10-27 15:31:07] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:41 [I 2022-10-27 15:31:07] ...trap/engines/engine.py.229: process: 0.55350 | load: 0.00247 [I 2022-10-27 15:31:07] ...trap/engines/engine.py.230: loss: 0.13234 [I 2022-10-27 15:31:07] ...optimizers/trijoint.py.151: total_norm: 15.891894 [I 2022-10-27 15:31:12] ...trap/engines/engine.py.224: train: epoch 74 | batch 1700/2304 [I 2022-10-27 15:31:12] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:35 [I 2022-10-27 15:31:12] ...trap/engines/engine.py.229: process: 0.55202 | load: 0.00246 [I 2022-10-27 15:31:12] ...trap/engines/engine.py.230: loss: 0.13755 [I 2022-10-27 15:31:12] ...optimizers/trijoint.py.151: total_norm: 10.522104 [I 2022-10-27 15:31:18] ...trap/engines/engine.py.224: train: epoch 74 | batch 1710/2304 [I 2022-10-27 15:31:18] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:34 [I 2022-10-27 15:31:18] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00245 [I 2022-10-27 15:31:18] ...trap/engines/engine.py.230: loss: 0.13748 [I 2022-10-27 15:31:18] ...optimizers/trijoint.py.151: total_norm: 20.682657 [I 2022-10-27 15:31:24] ...trap/engines/engine.py.224: train: epoch 74 | batch 1720/2304 [I 2022-10-27 15:31:24] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:26 [I 2022-10-27 15:31:24] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00242 [I 2022-10-27 15:31:24] ...trap/engines/engine.py.230: loss: 0.13923 [I 2022-10-27 15:31:24] ...optimizers/trijoint.py.151: total_norm: 18.438412 [I 2022-10-27 15:31:29] ...trap/engines/engine.py.224: train: epoch 74 | batch 1730/2304 [I 2022-10-27 15:31:29] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:19 [I 2022-10-27 15:31:29] ...trap/engines/engine.py.229: process: 0.54955 | load: 0.00243 [I 2022-10-27 15:31:29] ...trap/engines/engine.py.230: loss: 0.12982 [I 2022-10-27 15:31:29] ...optimizers/trijoint.py.151: total_norm: 16.588230 [I 2022-10-27 15:31:35] ...trap/engines/engine.py.224: train: epoch 74 | batch 1740/2304 [I 2022-10-27 15:31:35] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:05:12 [I 2022-10-27 15:31:35] ...trap/engines/engine.py.229: process: 0.55217 | load: 0.00246 [I 2022-10-27 15:31:35] ...trap/engines/engine.py.230: loss: 0.09266 [I 2022-10-27 15:31:35] ...optimizers/trijoint.py.151: total_norm: 9.718163 [I 2022-10-27 15:31:40] ...trap/engines/engine.py.224: train: epoch 74 | batch 1750/2304 [I 2022-10-27 15:31:40] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:08 [I 2022-10-27 15:31:40] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00244 [I 2022-10-27 15:31:40] ...trap/engines/engine.py.230: loss: 0.11936 [I 2022-10-27 15:31:40] ...optimizers/trijoint.py.151: total_norm: 17.526237 [I 2022-10-27 15:31:46] ...trap/engines/engine.py.224: train: epoch 74 | batch 1760/2304 [I 2022-10-27 15:31:46] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:05:01 [I 2022-10-27 15:31:46] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00242 [I 2022-10-27 15:31:46] ...trap/engines/engine.py.230: loss: 0.13269 [I 2022-10-27 15:31:46] ...optimizers/trijoint.py.151: total_norm: 16.464863 [I 2022-10-27 15:31:51] ...trap/engines/engine.py.224: train: epoch 74 | batch 1770/2304 [I 2022-10-27 15:31:51] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:57 [I 2022-10-27 15:31:51] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00244 [I 2022-10-27 15:31:51] ...trap/engines/engine.py.230: loss: 0.15742 [I 2022-10-27 15:31:51] ...optimizers/trijoint.py.151: total_norm: 50.538582 [I 2022-10-27 15:31:57] ...trap/engines/engine.py.224: train: epoch 74 | batch 1780/2304 [I 2022-10-27 15:31:57] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:53 [I 2022-10-27 15:31:57] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00243 [I 2022-10-27 15:31:57] ...trap/engines/engine.py.230: loss: 0.11273 [I 2022-10-27 15:31:57] ...optimizers/trijoint.py.151: total_norm: 9.940876 [I 2022-10-27 15:32:03] ...trap/engines/engine.py.224: train: epoch 74 | batch 1790/2304 [I 2022-10-27 15:32:03] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:49 [I 2022-10-27 15:32:03] ...trap/engines/engine.py.229: process: 0.57848 | load: 0.00253 [I 2022-10-27 15:32:03] ...trap/engines/engine.py.230: loss: 0.10864 [I 2022-10-27 15:32:03] ...optimizers/trijoint.py.151: total_norm: 14.113067 [I 2022-10-27 15:32:08] ...trap/engines/engine.py.224: train: epoch 74 | batch 1800/2304 [I 2022-10-27 15:32:08] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:43 [I 2022-10-27 15:32:08] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00247 [I 2022-10-27 15:32:08] ...trap/engines/engine.py.230: loss: 0.13071 [I 2022-10-27 15:32:08] ...optimizers/trijoint.py.151: total_norm: 14.570783 [I 2022-10-27 15:32:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 1810/2304 [I 2022-10-27 15:32:14] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:35 [I 2022-10-27 15:32:14] ...trap/engines/engine.py.229: process: 0.55193 | load: 0.00248 [I 2022-10-27 15:32:14] ...trap/engines/engine.py.230: loss: 0.17499 [I 2022-10-27 15:32:14] ...optimizers/trijoint.py.151: total_norm: 15.148488 [I 2022-10-27 15:32:19] ...trap/engines/engine.py.224: train: epoch 74 | batch 1820/2304 [I 2022-10-27 15:32:19] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:29 [I 2022-10-27 15:32:19] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00247 [I 2022-10-27 15:32:19] ...trap/engines/engine.py.230: loss: 0.11715 [I 2022-10-27 15:32:19] ...optimizers/trijoint.py.151: total_norm: 14.671956 [I 2022-10-27 15:32:25] ...trap/engines/engine.py.224: train: epoch 74 | batch 1830/2304 [I 2022-10-27 15:32:25] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:24 [I 2022-10-27 15:32:25] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00244 [I 2022-10-27 15:32:25] ...trap/engines/engine.py.230: loss: 0.13579 [I 2022-10-27 15:32:25] ...optimizers/trijoint.py.151: total_norm: 19.724628 [I 2022-10-27 15:32:31] ...trap/engines/engine.py.224: train: epoch 74 | batch 1840/2304 [I 2022-10-27 15:32:31] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:04:20 [I 2022-10-27 15:32:31] ...trap/engines/engine.py.229: process: 0.56525 | load: 0.00246 [I 2022-10-27 15:32:31] ...trap/engines/engine.py.230: loss: 0.16340 [I 2022-10-27 15:32:31] ...optimizers/trijoint.py.151: total_norm: 23.375198 [I 2022-10-27 15:32:36] ...trap/engines/engine.py.224: train: epoch 74 | batch 1850/2304 [I 2022-10-27 15:32:36] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:04:12 [I 2022-10-27 15:32:36] ...trap/engines/engine.py.229: process: 0.56552 | load: 0.00242 [I 2022-10-27 15:32:36] ...trap/engines/engine.py.230: loss: 0.11266 [I 2022-10-27 15:32:36] ...optimizers/trijoint.py.151: total_norm: 9.434759 [I 2022-10-27 15:32:42] ...trap/engines/engine.py.224: train: epoch 74 | batch 1860/2304 [I 2022-10-27 15:32:42] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:04:08 [I 2022-10-27 15:32:42] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00243 [I 2022-10-27 15:32:42] ...trap/engines/engine.py.230: loss: 0.16174 [I 2022-10-27 15:32:42] ...optimizers/trijoint.py.151: total_norm: 28.659008 [I 2022-10-27 15:32:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 1870/2304 [I 2022-10-27 15:32:47] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:04:03 [I 2022-10-27 15:32:47] ...trap/engines/engine.py.229: process: 0.57235 | load: 0.00244 [I 2022-10-27 15:32:47] ...trap/engines/engine.py.230: loss: 0.18205 [I 2022-10-27 15:32:47] ...optimizers/trijoint.py.151: total_norm: 21.347223 [I 2022-10-27 15:32:53] ...trap/engines/engine.py.224: train: epoch 74 | batch 1880/2304 [I 2022-10-27 15:32:53] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:58 [I 2022-10-27 15:32:53] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00252 [I 2022-10-27 15:32:53] ...trap/engines/engine.py.230: loss: 0.18838 [I 2022-10-27 15:32:53] ...optimizers/trijoint.py.151: total_norm: 40.002106 [I 2022-10-27 15:32:59] ...trap/engines/engine.py.224: train: epoch 74 | batch 1890/2304 [I 2022-10-27 15:32:59] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:50 [I 2022-10-27 15:32:59] ...trap/engines/engine.py.229: process: 0.55061 | load: 0.00245 [I 2022-10-27 15:32:59] ...trap/engines/engine.py.230: loss: 0.12425 [I 2022-10-27 15:32:59] ...optimizers/trijoint.py.151: total_norm: 9.372865 [I 2022-10-27 15:33:04] ...trap/engines/engine.py.224: train: epoch 74 | batch 1900/2304 [I 2022-10-27 15:33:04] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:45 [I 2022-10-27 15:33:04] ...trap/engines/engine.py.229: process: 0.56567 | load: 0.00249 [I 2022-10-27 15:33:04] ...trap/engines/engine.py.230: loss: 0.11346 [I 2022-10-27 15:33:04] ...optimizers/trijoint.py.151: total_norm: 15.545971 [I 2022-10-27 15:33:10] ...trap/engines/engine.py.224: train: epoch 74 | batch 1910/2304 [I 2022-10-27 15:33:10] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:39 [I 2022-10-27 15:33:10] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00243 [I 2022-10-27 15:33:10] ...trap/engines/engine.py.230: loss: 0.13612 [I 2022-10-27 15:33:10] ...optimizers/trijoint.py.151: total_norm: 15.149373 [I 2022-10-27 15:33:15] ...trap/engines/engine.py.224: train: epoch 74 | batch 1920/2304 [I 2022-10-27 15:33:15] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:35 [I 2022-10-27 15:33:15] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00243 [I 2022-10-27 15:33:15] ...trap/engines/engine.py.230: loss: 0.08768 [I 2022-10-27 15:33:15] ...optimizers/trijoint.py.151: total_norm: 10.592155 [I 2022-10-27 15:33:21] ...trap/engines/engine.py.224: train: epoch 74 | batch 1930/2304 [I 2022-10-27 15:33:21] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:31 [I 2022-10-27 15:33:21] ...trap/engines/engine.py.229: process: 0.57429 | load: 0.00247 [I 2022-10-27 15:33:21] ...trap/engines/engine.py.230: loss: 0.19907 [I 2022-10-27 15:33:21] ...optimizers/trijoint.py.151: total_norm: 18.261312 [I 2022-10-27 15:33:27] ...trap/engines/engine.py.224: train: epoch 74 | batch 1940/2304 [I 2022-10-27 15:33:27] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:24 [I 2022-10-27 15:33:27] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00246 [I 2022-10-27 15:33:27] ...trap/engines/engine.py.230: loss: 0.09028 [I 2022-10-27 15:33:27] ...optimizers/trijoint.py.151: total_norm: 19.085434 [I 2022-10-27 15:33:32] ...trap/engines/engine.py.224: train: epoch 74 | batch 1950/2304 [I 2022-10-27 15:33:32] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:18 [I 2022-10-27 15:33:32] ...trap/engines/engine.py.229: process: 0.57322 | load: 0.00244 [I 2022-10-27 15:33:32] ...trap/engines/engine.py.230: loss: 0.16075 [I 2022-10-27 15:33:32] ...optimizers/trijoint.py.151: total_norm: 17.053114 [I 2022-10-27 15:33:38] ...trap/engines/engine.py.224: train: epoch 74 | batch 1960/2304 [I 2022-10-27 15:33:38] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:03:11 [I 2022-10-27 15:33:38] ...trap/engines/engine.py.229: process: 0.55027 | load: 0.00246 [I 2022-10-27 15:33:38] ...trap/engines/engine.py.230: loss: 0.25294 [I 2022-10-27 15:33:38] ...optimizers/trijoint.py.151: total_norm: 21.680422 [I 2022-10-27 15:33:44] ...trap/engines/engine.py.224: train: epoch 74 | batch 1970/2304 [I 2022-10-27 15:33:44] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:03:06 [I 2022-10-27 15:33:44] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00243 [I 2022-10-27 15:33:44] ...trap/engines/engine.py.230: loss: 0.10149 [I 2022-10-27 15:33:44] ...optimizers/trijoint.py.151: total_norm: 11.581189 [I 2022-10-27 15:33:49] ...trap/engines/engine.py.224: train: epoch 74 | batch 1980/2304 [I 2022-10-27 15:33:49] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:59 [I 2022-10-27 15:33:49] ...trap/engines/engine.py.229: process: 0.54524 | load: 0.00245 [I 2022-10-27 15:33:49] ...trap/engines/engine.py.230: loss: 0.18712 [I 2022-10-27 15:33:49] ...optimizers/trijoint.py.151: total_norm: 31.627344 [I 2022-10-27 15:33:55] ...trap/engines/engine.py.224: train: epoch 74 | batch 1990/2304 [I 2022-10-27 15:33:55] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:54 [I 2022-10-27 15:33:55] ...trap/engines/engine.py.229: process: 0.54696 | load: 0.00242 [I 2022-10-27 15:33:55] ...trap/engines/engine.py.230: loss: 0.11544 [I 2022-10-27 15:33:55] ...optimizers/trijoint.py.151: total_norm: 11.339310 [I 2022-10-27 15:34:00] ...trap/engines/engine.py.224: train: epoch 74 | batch 2000/2304 [I 2022-10-27 15:34:00] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:49 [I 2022-10-27 15:34:00] ...trap/engines/engine.py.229: process: 0.55482 | load: 0.00247 [I 2022-10-27 15:34:00] ...trap/engines/engine.py.230: loss: 0.11166 [I 2022-10-27 15:34:00] ...optimizers/trijoint.py.151: total_norm: 12.491371 [I 2022-10-27 15:34:06] ...trap/engines/engine.py.224: train: epoch 74 | batch 2010/2304 [I 2022-10-27 15:34:06] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:43 [I 2022-10-27 15:34:06] ...trap/engines/engine.py.229: process: 0.56461 | load: 0.00246 [I 2022-10-27 15:34:06] ...trap/engines/engine.py.230: loss: 0.15805 [I 2022-10-27 15:34:06] ...optimizers/trijoint.py.151: total_norm: 18.259405 [I 2022-10-27 15:34:11] ...trap/engines/engine.py.224: train: epoch 74 | batch 2020/2304 [I 2022-10-27 15:34:11] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:38 [I 2022-10-27 15:34:11] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00253 [I 2022-10-27 15:34:11] ...trap/engines/engine.py.230: loss: 0.12533 [I 2022-10-27 15:34:11] ...optimizers/trijoint.py.151: total_norm: 18.278572 [I 2022-10-27 15:34:17] ...trap/engines/engine.py.224: train: epoch 74 | batch 2030/2304 [I 2022-10-27 15:34:17] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:32 [I 2022-10-27 15:34:17] ...trap/engines/engine.py.229: process: 0.54581 | load: 0.00248 [I 2022-10-27 15:34:17] ...trap/engines/engine.py.230: loss: 0.09500 [I 2022-10-27 15:34:17] ...optimizers/trijoint.py.151: total_norm: 10.393137 [I 2022-10-27 15:34:23] ...trap/engines/engine.py.224: train: epoch 74 | batch 2040/2304 [I 2022-10-27 15:34:23] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:27 [I 2022-10-27 15:34:23] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00247 [I 2022-10-27 15:34:23] ...trap/engines/engine.py.230: loss: 0.15275 [I 2022-10-27 15:34:23] ...optimizers/trijoint.py.151: total_norm: 22.664459 [I 2022-10-27 15:34:28] ...trap/engines/engine.py.224: train: epoch 74 | batch 2050/2304 [I 2022-10-27 15:34:28] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:21 [I 2022-10-27 15:34:28] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00245 [I 2022-10-27 15:34:28] ...trap/engines/engine.py.230: loss: 0.14339 [I 2022-10-27 15:34:28] ...optimizers/trijoint.py.151: total_norm: 20.547899 [I 2022-10-27 15:34:34] ...trap/engines/engine.py.224: train: epoch 74 | batch 2060/2304 [I 2022-10-27 15:34:34] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:16 [I 2022-10-27 15:34:34] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00243 [I 2022-10-27 15:34:34] ...trap/engines/engine.py.230: loss: 0.14201 [I 2022-10-27 15:34:34] ...optimizers/trijoint.py.151: total_norm: 9.607843 [I 2022-10-27 15:34:39] ...trap/engines/engine.py.224: train: epoch 74 | batch 2070/2304 [I 2022-10-27 15:34:39] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:02:11 [I 2022-10-27 15:34:39] ...trap/engines/engine.py.229: process: 0.57099 | load: 0.00245 [I 2022-10-27 15:34:39] ...trap/engines/engine.py.230: loss: 0.11944 [I 2022-10-27 15:34:39] ...optimizers/trijoint.py.151: total_norm: 22.474630 [I 2022-10-27 15:34:45] ...trap/engines/engine.py.224: train: epoch 74 | batch 2080/2304 [I 2022-10-27 15:34:45] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:02:04 [I 2022-10-27 15:34:45] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00244 [I 2022-10-27 15:34:45] ...trap/engines/engine.py.230: loss: 0.11339 [I 2022-10-27 15:34:45] ...optimizers/trijoint.py.151: total_norm: 17.652832 [I 2022-10-27 15:34:51] ...trap/engines/engine.py.224: train: epoch 74 | batch 2090/2304 [I 2022-10-27 15:34:51] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:59 [I 2022-10-27 15:34:51] ...trap/engines/engine.py.229: process: 0.54714 | load: 0.00242 [I 2022-10-27 15:34:51] ...trap/engines/engine.py.230: loss: 0.19225 [I 2022-10-27 15:34:51] ...optimizers/trijoint.py.151: total_norm: 17.293089 [I 2022-10-27 15:34:56] ...trap/engines/engine.py.224: train: epoch 74 | batch 2100/2304 [I 2022-10-27 15:34:56] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:53 [I 2022-10-27 15:34:56] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00247 [I 2022-10-27 15:34:56] ...trap/engines/engine.py.230: loss: 0.12555 [I 2022-10-27 15:34:56] ...optimizers/trijoint.py.151: total_norm: 15.329079 [I 2022-10-27 15:35:02] ...trap/engines/engine.py.224: train: epoch 74 | batch 2110/2304 [I 2022-10-27 15:35:02] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:48 [I 2022-10-27 15:35:02] ...trap/engines/engine.py.229: process: 0.55845 | load: 0.00252 [I 2022-10-27 15:35:02] ...trap/engines/engine.py.230: loss: 0.12220 [I 2022-10-27 15:35:02] ...optimizers/trijoint.py.151: total_norm: 21.629896 [I 2022-10-27 15:35:07] ...trap/engines/engine.py.224: train: epoch 74 | batch 2120/2304 [I 2022-10-27 15:35:07] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:43 [I 2022-10-27 15:35:07] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00256 [I 2022-10-27 15:35:07] ...trap/engines/engine.py.230: loss: 0.19301 [I 2022-10-27 15:35:07] ...optimizers/trijoint.py.151: total_norm: 34.917236 [I 2022-10-27 15:35:13] ...trap/engines/engine.py.224: train: epoch 74 | batch 2130/2304 [I 2022-10-27 15:35:13] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:37 [I 2022-10-27 15:35:13] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00256 [I 2022-10-27 15:35:13] ...trap/engines/engine.py.230: loss: 0.18515 [I 2022-10-27 15:35:13] ...optimizers/trijoint.py.151: total_norm: 16.230883 [I 2022-10-27 15:35:19] ...trap/engines/engine.py.224: train: epoch 74 | batch 2140/2304 [I 2022-10-27 15:35:19] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:31 [I 2022-10-27 15:35:19] ...trap/engines/engine.py.229: process: 0.54840 | load: 0.00254 [I 2022-10-27 15:35:19] ...trap/engines/engine.py.230: loss: 0.13150 [I 2022-10-27 15:35:19] ...optimizers/trijoint.py.151: total_norm: 26.911678 [I 2022-10-27 15:35:24] ...trap/engines/engine.py.224: train: epoch 74 | batch 2150/2304 [I 2022-10-27 15:35:24] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:26 [I 2022-10-27 15:35:24] ...trap/engines/engine.py.229: process: 0.56250 | load: 0.00241 [I 2022-10-27 15:35:24] ...trap/engines/engine.py.230: loss: 0.17835 [I 2022-10-27 15:35:24] ...optimizers/trijoint.py.151: total_norm: 33.047497 [I 2022-10-27 15:35:30] ...trap/engines/engine.py.224: train: epoch 74 | batch 2160/2304 [I 2022-10-27 15:35:30] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:20 [I 2022-10-27 15:35:30] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00246 [I 2022-10-27 15:35:30] ...trap/engines/engine.py.230: loss: 0.21858 [I 2022-10-27 15:35:30] ...optimizers/trijoint.py.151: total_norm: 19.864912 [I 2022-10-27 15:35:35] ...trap/engines/engine.py.224: train: epoch 74 | batch 2170/2304 [I 2022-10-27 15:35:35] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:01:15 [I 2022-10-27 15:35:35] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00242 [I 2022-10-27 15:35:35] ...trap/engines/engine.py.230: loss: 0.10519 [I 2022-10-27 15:35:35] ...optimizers/trijoint.py.151: total_norm: 24.267181 [I 2022-10-27 15:35:41] ...trap/engines/engine.py.224: train: epoch 74 | batch 2180/2304 [I 2022-10-27 15:35:41] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:09 [I 2022-10-27 15:35:41] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00244 [I 2022-10-27 15:35:41] ...trap/engines/engine.py.230: loss: 0.10531 [I 2022-10-27 15:35:41] ...optimizers/trijoint.py.151: total_norm: 15.813506 [I 2022-10-27 15:35:47] ...trap/engines/engine.py.224: train: epoch 74 | batch 2190/2304 [I 2022-10-27 15:35:47] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:01:03 [I 2022-10-27 15:35:47] ...trap/engines/engine.py.229: process: 0.54252 | load: 0.00245 [I 2022-10-27 15:35:47] ...trap/engines/engine.py.230: loss: 0.13170 [I 2022-10-27 15:35:47] ...optimizers/trijoint.py.151: total_norm: 14.403568 [I 2022-10-27 15:35:52] ...trap/engines/engine.py.224: train: epoch 74 | batch 2200/2304 [I 2022-10-27 15:35:52] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:58 [I 2022-10-27 15:35:52] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00263 [I 2022-10-27 15:35:52] ...trap/engines/engine.py.230: loss: 0.20051 [I 2022-10-27 15:35:52] ...optimizers/trijoint.py.151: total_norm: 14.777578 [I 2022-10-27 15:35:58] ...trap/engines/engine.py.224: train: epoch 74 | batch 2210/2304 [I 2022-10-27 15:35:58] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:52 [I 2022-10-27 15:35:58] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00251 [I 2022-10-27 15:35:58] ...trap/engines/engine.py.230: loss: 0.14460 [I 2022-10-27 15:35:58] ...optimizers/trijoint.py.151: total_norm: 13.330715 [I 2022-10-27 15:36:03] ...trap/engines/engine.py.224: train: epoch 74 | batch 2220/2304 [I 2022-10-27 15:36:03] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:46 [I 2022-10-27 15:36:03] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00245 [I 2022-10-27 15:36:03] ...trap/engines/engine.py.230: loss: 0.10763 [I 2022-10-27 15:36:03] ...optimizers/trijoint.py.151: total_norm: 10.528389 [I 2022-10-27 15:36:09] ...trap/engines/engine.py.224: train: epoch 74 | batch 2230/2304 [I 2022-10-27 15:36:09] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:40 [I 2022-10-27 15:36:09] ...trap/engines/engine.py.229: process: 0.54793 | load: 0.00246 [I 2022-10-27 15:36:09] ...trap/engines/engine.py.230: loss: 0.16815 [I 2022-10-27 15:36:09] ...optimizers/trijoint.py.151: total_norm: 12.522216 [I 2022-10-27 15:36:14] ...trap/engines/engine.py.224: train: epoch 74 | batch 2240/2304 [I 2022-10-27 15:36:14] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:35 [I 2022-10-27 15:36:14] ...trap/engines/engine.py.229: process: 0.54791 | load: 0.00245 [I 2022-10-27 15:36:14] ...trap/engines/engine.py.230: loss: 0.12709 [I 2022-10-27 15:36:14] ...optimizers/trijoint.py.151: total_norm: 15.128763 [I 2022-10-27 15:36:20] ...trap/engines/engine.py.224: train: epoch 74 | batch 2250/2304 [I 2022-10-27 15:36:20] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:30 [I 2022-10-27 15:36:20] ...trap/engines/engine.py.229: process: 0.55493 | load: 0.00246 [I 2022-10-27 15:36:20] ...trap/engines/engine.py.230: loss: 0.12681 [I 2022-10-27 15:36:20] ...optimizers/trijoint.py.151: total_norm: 9.843730 [I 2022-10-27 15:36:26] ...trap/engines/engine.py.224: train: epoch 74 | batch 2260/2304 [I 2022-10-27 15:36:26] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:24 [I 2022-10-27 15:36:26] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00243 [I 2022-10-27 15:36:26] ...trap/engines/engine.py.230: loss: 0.14036 [I 2022-10-27 15:36:26] ...optimizers/trijoint.py.151: total_norm: 23.217508 [I 2022-10-27 15:36:32] ...trap/engines/engine.py.224: train: epoch 74 | batch 2270/2304 [I 2022-10-27 15:36:32] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:19 [I 2022-10-27 15:36:32] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00249 [I 2022-10-27 15:36:32] ...trap/engines/engine.py.230: loss: 0.14921 [I 2022-10-27 15:36:32] ...optimizers/trijoint.py.151: total_norm: 15.834450 [I 2022-10-27 15:36:37] ...trap/engines/engine.py.224: train: epoch 74 | batch 2280/2304 [I 2022-10-27 15:36:37] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:13 [I 2022-10-27 15:36:37] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00254 [I 2022-10-27 15:36:37] ...trap/engines/engine.py.230: loss: 0.20202 [I 2022-10-27 15:36:37] ...optimizers/trijoint.py.151: total_norm: 28.679811 [I 2022-10-27 15:36:43] ...trap/engines/engine.py.224: train: epoch 74 | batch 2290/2304 [I 2022-10-27 15:36:43] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:07 [I 2022-10-27 15:36:43] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00248 [I 2022-10-27 15:36:43] ...trap/engines/engine.py.230: loss: 0.13130 [I 2022-10-27 15:36:43] ...optimizers/trijoint.py.151: total_norm: 16.861116 [I 2022-10-27 15:36:48] ...trap/engines/engine.py.224: train: epoch 74 | batch 2300/2304 [I 2022-10-27 15:36:48] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:02 [I 2022-10-27 15:36:48] ...trap/engines/engine.py.229: process: 0.55124 | load: 0.00015 [I 2022-10-27 15:36:48] ...trap/engines/engine.py.230: loss: 0.15255 [I 2022-10-27 15:36:48] ...optimizers/trijoint.py.151: total_norm: 17.596380 [I 2022-10-27 15:36:51] ...trap/engines/engine.py.224: train: epoch 74 | batch 2304/2304 [I 2022-10-27 15:36:51] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:00:00 [I 2022-10-27 15:36:51] ...trap/engines/engine.py.229: process: 0.56729 | load: 0.00020 [I 2022-10-27 15:36:51] ...trap/engines/engine.py.230: loss: 0.14563 [I 2022-10-27 15:36:51] ...optimizers/trijoint.py.151: total_norm: 22.351274 [S 2022-10-27 15:36:52] ...trap/engines/engine.py.129: train_epoch.epoch: 74 [S 2022-10-27 15:36:52] ...trap/engines/engine.py.129: train_epoch.loss: 0.14107410859260022 [I 2022-10-27 15:36:52] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 15:36:53] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 74 [I 2022-10-27 15:36:53] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 15:36:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 15:36:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 15:36:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 15:36:57] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 15:36:58] ...trap/engines/engine.py.321: eval: epoch 74 | batch 0/510 [I 2022-10-27 15:36:58] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:40:57 [I 2022-10-27 15:36:58] ...trap/engines/engine.py.326: process: 4.81931 | load: 4.44103 [I 2022-10-27 15:37:01] ...trap/engines/engine.py.321: eval: epoch 74 | batch 10/510 [I 2022-10-27 15:37:01] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:46 [I 2022-10-27 15:37:01] ...trap/engines/engine.py.326: process: 0.32077 | load: 0.00016 [I 2022-10-27 15:37:04] ...trap/engines/engine.py.321: eval: epoch 74 | batch 20/510 [I 2022-10-27 15:37:04] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:08 [I 2022-10-27 15:37:04] ...trap/engines/engine.py.326: process: 0.33550 | load: 0.00020 [I 2022-10-27 15:37:08] ...trap/engines/engine.py.321: eval: epoch 74 | batch 30/510 [I 2022-10-27 15:37:08] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:45 [I 2022-10-27 15:37:08] ...trap/engines/engine.py.326: process: 0.34025 | load: 0.00018 [I 2022-10-27 15:37:11] ...trap/engines/engine.py.321: eval: epoch 74 | batch 40/510 [I 2022-10-27 15:37:11] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:37 [I 2022-10-27 15:37:11] ...trap/engines/engine.py.326: process: 0.32506 | load: 0.00016 [I 2022-10-27 15:37:14] ...trap/engines/engine.py.321: eval: epoch 74 | batch 50/510 [I 2022-10-27 15:37:14] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:33 [I 2022-10-27 15:37:14] ...trap/engines/engine.py.326: process: 0.33592 | load: 0.00027 [I 2022-10-27 15:37:18] ...trap/engines/engine.py.321: eval: epoch 74 | batch 60/510 [I 2022-10-27 15:37:18] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:29 [I 2022-10-27 15:37:18] ...trap/engines/engine.py.326: process: 0.32263 | load: 0.00018 [I 2022-10-27 15:37:21] ...trap/engines/engine.py.321: eval: epoch 74 | batch 70/510 [I 2022-10-27 15:37:21] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:27 [I 2022-10-27 15:37:21] ...trap/engines/engine.py.326: process: 0.32590 | load: 0.00016 [I 2022-10-27 15:37:24] ...trap/engines/engine.py.321: eval: epoch 74 | batch 80/510 [I 2022-10-27 15:37:24] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:22 [I 2022-10-27 15:37:24] ...trap/engines/engine.py.326: process: 0.31982 | load: 0.00016 [I 2022-10-27 15:37:28] ...trap/engines/engine.py.321: eval: epoch 74 | batch 90/510 [I 2022-10-27 15:37:28] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:19 [I 2022-10-27 15:37:28] ...trap/engines/engine.py.326: process: 0.32726 | load: 0.00026 [I 2022-10-27 15:37:31] ...trap/engines/engine.py.321: eval: epoch 74 | batch 100/510 [I 2022-10-27 15:37:31] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:15 [I 2022-10-27 15:37:31] ...trap/engines/engine.py.326: process: 0.28826 | load: 0.00018 [I 2022-10-27 15:37:34] ...trap/engines/engine.py.321: eval: epoch 74 | batch 110/510 [I 2022-10-27 15:37:34] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 15:37:34] ...trap/engines/engine.py.326: process: 0.28758 | load: 0.00022 [I 2022-10-27 15:37:37] ...trap/engines/engine.py.321: eval: epoch 74 | batch 120/510 [I 2022-10-27 15:37:37] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 15:37:37] ...trap/engines/engine.py.326: process: 0.28532 | load: 0.00019 [I 2022-10-27 15:37:40] ...trap/engines/engine.py.321: eval: epoch 74 | batch 130/510 [I 2022-10-27 15:37:40] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:49 [I 2022-10-27 15:37:40] ...trap/engines/engine.py.326: process: 0.28264 | load: 0.00024 [I 2022-10-27 15:37:43] ...trap/engines/engine.py.321: eval: epoch 74 | batch 140/510 [I 2022-10-27 15:37:43] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:45 [I 2022-10-27 15:37:43] ...trap/engines/engine.py.326: process: 0.28171 | load: 0.00014 [I 2022-10-27 15:37:46] ...trap/engines/engine.py.321: eval: epoch 74 | batch 150/510 [I 2022-10-27 15:37:46] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:45 [I 2022-10-27 15:37:46] ...trap/engines/engine.py.326: process: 0.28821 | load: 0.00025 [I 2022-10-27 15:37:49] ...trap/engines/engine.py.321: eval: epoch 74 | batch 160/510 [I 2022-10-27 15:37:49] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 15:37:49] ...trap/engines/engine.py.326: process: 0.28861 | load: 0.00015 [I 2022-10-27 15:37:51] ...trap/engines/engine.py.321: eval: epoch 74 | batch 170/510 [I 2022-10-27 15:37:51] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 15:37:51] ...trap/engines/engine.py.326: process: 0.29031 | load: 0.00014 [I 2022-10-27 15:37:54] ...trap/engines/engine.py.321: eval: epoch 74 | batch 180/510 [I 2022-10-27 15:37:54] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 15:37:54] ...trap/engines/engine.py.326: process: 0.28523 | load: 0.00023 [I 2022-10-27 15:37:57] ...trap/engines/engine.py.321: eval: epoch 74 | batch 190/510 [I 2022-10-27 15:37:57] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 15:37:57] ...trap/engines/engine.py.326: process: 0.28974 | load: 0.00015 [I 2022-10-27 15:38:00] ...trap/engines/engine.py.321: eval: epoch 74 | batch 200/510 [I 2022-10-27 15:38:00] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 15:38:00] ...trap/engines/engine.py.326: process: 0.28988 | load: 0.00026 [I 2022-10-27 15:38:03] ...trap/engines/engine.py.321: eval: epoch 74 | batch 210/510 [I 2022-10-27 15:38:03] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 15:38:03] ...trap/engines/engine.py.326: process: 0.29138 | load: 0.00019 [I 2022-10-27 15:38:06] ...trap/engines/engine.py.321: eval: epoch 74 | batch 220/510 [I 2022-10-27 15:38:06] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 15:38:06] ...trap/engines/engine.py.326: process: 0.28792 | load: 0.00024 [I 2022-10-27 15:38:09] ...trap/engines/engine.py.321: eval: epoch 74 | batch 230/510 [I 2022-10-27 15:38:09] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 15:38:09] ...trap/engines/engine.py.326: process: 0.28636 | load: 0.00026 [I 2022-10-27 15:38:12] ...trap/engines/engine.py.321: eval: epoch 74 | batch 240/510 [I 2022-10-27 15:38:12] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 15:38:12] ...trap/engines/engine.py.326: process: 0.28404 | load: 0.00024 [I 2022-10-27 15:38:14] ...trap/engines/engine.py.321: eval: epoch 74 | batch 250/510 [I 2022-10-27 15:38:14] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 15:38:14] ...trap/engines/engine.py.326: process: 0.28464 | load: 0.00021 [I 2022-10-27 15:38:17] ...trap/engines/engine.py.321: eval: epoch 74 | batch 260/510 [I 2022-10-27 15:38:17] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 15:38:17] ...trap/engines/engine.py.326: process: 0.28979 | load: 0.00026 [I 2022-10-27 15:38:20] ...trap/engines/engine.py.321: eval: epoch 74 | batch 270/510 [I 2022-10-27 15:38:20] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 15:38:20] ...trap/engines/engine.py.326: process: 0.29014 | load: 0.00026 [I 2022-10-27 15:38:23] ...trap/engines/engine.py.321: eval: epoch 74 | batch 280/510 [I 2022-10-27 15:38:23] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 15:38:23] ...trap/engines/engine.py.326: process: 0.29027 | load: 0.00023 [I 2022-10-27 15:38:26] ...trap/engines/engine.py.321: eval: epoch 74 | batch 290/510 [I 2022-10-27 15:38:26] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 15:38:26] ...trap/engines/engine.py.326: process: 0.28574 | load: 0.00024 [I 2022-10-27 15:38:29] ...trap/engines/engine.py.321: eval: epoch 74 | batch 300/510 [I 2022-10-27 15:38:29] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 15:38:29] ...trap/engines/engine.py.326: process: 0.28912 | load: 0.00022 [I 2022-10-27 15:38:32] ...trap/engines/engine.py.321: eval: epoch 74 | batch 310/510 [I 2022-10-27 15:38:32] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 15:38:32] ...trap/engines/engine.py.326: process: 0.28883 | load: 0.00023 [I 2022-10-27 15:38:35] ...trap/engines/engine.py.321: eval: epoch 74 | batch 320/510 [I 2022-10-27 15:38:35] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 15:38:35] ...trap/engines/engine.py.326: process: 0.28918 | load: 0.00024 [I 2022-10-27 15:38:38] ...trap/engines/engine.py.321: eval: epoch 74 | batch 330/510 [I 2022-10-27 15:38:38] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 15:38:38] ...trap/engines/engine.py.326: process: 0.28355 | load: 0.00024 [I 2022-10-27 15:38:40] ...trap/engines/engine.py.321: eval: epoch 74 | batch 340/510 [I 2022-10-27 15:38:40] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 15:38:40] ...trap/engines/engine.py.326: process: 0.29265 | load: 0.00025 [I 2022-10-27 15:38:43] ...trap/engines/engine.py.321: eval: epoch 74 | batch 350/510 [I 2022-10-27 15:38:43] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 15:38:43] ...trap/engines/engine.py.326: process: 0.28766 | load: 0.00024 [I 2022-10-27 15:38:46] ...trap/engines/engine.py.321: eval: epoch 74 | batch 360/510 [I 2022-10-27 15:38:46] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 15:38:46] ...trap/engines/engine.py.326: process: 0.28224 | load: 0.00024 [I 2022-10-27 15:38:49] ...trap/engines/engine.py.321: eval: epoch 74 | batch 370/510 [I 2022-10-27 15:38:49] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 15:38:49] ...trap/engines/engine.py.326: process: 0.28670 | load: 0.00015 [I 2022-10-27 15:38:52] ...trap/engines/engine.py.321: eval: epoch 74 | batch 380/510 [I 2022-10-27 15:38:52] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:38 [I 2022-10-27 15:38:52] ...trap/engines/engine.py.326: process: 0.28632 | load: 0.00018 [I 2022-10-27 15:38:55] ...trap/engines/engine.py.321: eval: epoch 74 | batch 390/510 [I 2022-10-27 15:38:55] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 15:38:55] ...trap/engines/engine.py.326: process: 0.28798 | load: 0.00016 [I 2022-10-27 15:38:58] ...trap/engines/engine.py.321: eval: epoch 74 | batch 400/510 [I 2022-10-27 15:38:58] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 15:38:58] ...trap/engines/engine.py.326: process: 0.29007 | load: 0.00016 [I 2022-10-27 15:39:01] ...trap/engines/engine.py.321: eval: epoch 74 | batch 410/510 [I 2022-10-27 15:39:01] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 15:39:01] ...trap/engines/engine.py.326: process: 0.28900 | load: 0.00015 [I 2022-10-27 15:39:04] ...trap/engines/engine.py.321: eval: epoch 74 | batch 420/510 [I 2022-10-27 15:39:04] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 15:39:04] ...trap/engines/engine.py.326: process: 0.28534 | load: 0.00016 [I 2022-10-27 15:39:07] ...trap/engines/engine.py.321: eval: epoch 74 | batch 430/510 [I 2022-10-27 15:39:07] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 15:39:07] ...trap/engines/engine.py.326: process: 0.28870 | load: 0.00018 [I 2022-10-27 15:39:09] ...trap/engines/engine.py.321: eval: epoch 74 | batch 440/510 [I 2022-10-27 15:39:09] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 15:39:09] ...trap/engines/engine.py.326: process: 0.28681 | load: 0.00020 [I 2022-10-27 15:39:12] ...trap/engines/engine.py.321: eval: epoch 74 | batch 450/510 [I 2022-10-27 15:39:12] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 15:39:12] ...trap/engines/engine.py.326: process: 0.28876 | load: 0.00024 [I 2022-10-27 15:39:15] ...trap/engines/engine.py.321: eval: epoch 74 | batch 460/510 [I 2022-10-27 15:39:15] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 15:39:15] ...trap/engines/engine.py.326: process: 0.28940 | load: 0.00022 [I 2022-10-27 15:39:18] ...trap/engines/engine.py.321: eval: epoch 74 | batch 470/510 [I 2022-10-27 15:39:18] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 15:39:18] ...trap/engines/engine.py.326: process: 0.28154 | load: 0.00017 [I 2022-10-27 15:39:21] ...trap/engines/engine.py.321: eval: epoch 74 | batch 480/510 [I 2022-10-27 15:39:21] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 15:39:21] ...trap/engines/engine.py.326: process: 0.28082 | load: 0.00024 [I 2022-10-27 15:39:24] ...trap/engines/engine.py.321: eval: epoch 74 | batch 490/510 [I 2022-10-27 15:39:24] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 15:39:24] ...trap/engines/engine.py.326: process: 0.29330 | load: 0.00017 [I 2022-10-27 15:39:27] ...trap/engines/engine.py.321: eval: epoch 74 | batch 500/510 [I 2022-10-27 15:39:27] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 15:39:27] ...trap/engines/engine.py.326: process: 0.28495 | load: 0.00021 [I 2022-10-27 15:39:30] ...trap/engines/engine.py.321: eval: epoch 74 | batch 510/510 [I 2022-10-27 15:39:30] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 15:39:30] ...trap/engines/engine.py.326: process: 0.28956 | load: 0.00017 [S 2022-10-27 15:39:30] ...trap/engines/engine.py.132: eval_epoch.epoch: 74 [I 2022-10-27 15:39:30] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.729100 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905100 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936400 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.707400 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904300 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934100 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014775 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008479 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006576 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014610 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008649 [S 2022-10-27 15:39:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006284 [I 2022-10-27 15:39:36] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 15:39:36] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 15:39:36] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 15:39:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 15:39:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 15:39:40] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 15:39:41] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 15:39:47] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 15:39:47] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 15:39:47] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 15:39:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 15:39:55] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 15:39:55] ...trap/engines/engine.py.163: Training model on trainset for epoch 75 [I 2022-10-27 15:39:55] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 15:39:58] ...trap/engines/engine.py.224: train: epoch 75 | batch 0/2304 [I 2022-10-27 15:39:58] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:36 [I 2022-10-27 15:39:58] ...trap/engines/engine.py.229: process: 2.64612 | load: 2.02188 [I 2022-10-27 15:39:58] ...trap/engines/engine.py.230: loss: 0.16230 [I 2022-10-27 15:39:58] ...optimizers/trijoint.py.151: total_norm: 28.507082 [I 2022-10-27 15:40:03] ...trap/engines/engine.py.224: train: epoch 75 | batch 10/2304 [I 2022-10-27 15:40:03] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:46 [I 2022-10-27 15:40:03] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00451 [I 2022-10-27 15:40:03] ...trap/engines/engine.py.230: loss: 0.12158 [I 2022-10-27 15:40:03] ...optimizers/trijoint.py.151: total_norm: 16.460249 [I 2022-10-27 15:40:09] ...trap/engines/engine.py.224: train: epoch 75 | batch 20/2304 [I 2022-10-27 15:40:09] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:07 [I 2022-10-27 15:40:09] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00314 [I 2022-10-27 15:40:09] ...trap/engines/engine.py.230: loss: 0.17416 [I 2022-10-27 15:40:09] ...optimizers/trijoint.py.151: total_norm: 33.318745 [I 2022-10-27 15:40:15] ...trap/engines/engine.py.224: train: epoch 75 | batch 30/2304 [I 2022-10-27 15:40:15] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:18 [I 2022-10-27 15:40:15] ...trap/engines/engine.py.229: process: 0.56503 | load: 0.00427 [I 2022-10-27 15:40:15] ...trap/engines/engine.py.230: loss: 0.20974 [I 2022-10-27 15:40:15] ...optimizers/trijoint.py.151: total_norm: 23.039003 [I 2022-10-27 15:40:20] ...trap/engines/engine.py.224: train: epoch 75 | batch 40/2304 [I 2022-10-27 15:40:20] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:21:10 [I 2022-10-27 15:40:20] ...trap/engines/engine.py.229: process: 0.56410 | load: 0.00280 [I 2022-10-27 15:40:20] ...trap/engines/engine.py.230: loss: 0.15263 [I 2022-10-27 15:40:20] ...optimizers/trijoint.py.151: total_norm: 12.357790 [I 2022-10-27 15:40:26] ...trap/engines/engine.py.224: train: epoch 75 | batch 50/2304 [I 2022-10-27 15:40:26] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:54 [I 2022-10-27 15:40:26] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00347 [I 2022-10-27 15:40:26] ...trap/engines/engine.py.230: loss: 0.10790 [I 2022-10-27 15:40:26] ...optimizers/trijoint.py.151: total_norm: 13.443284 [I 2022-10-27 15:40:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 60/2304 [I 2022-10-27 15:40:31] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:00 [I 2022-10-27 15:40:31] ...trap/engines/engine.py.229: process: 0.56652 | load: 0.00277 [I 2022-10-27 15:40:31] ...trap/engines/engine.py.230: loss: 0.11968 [I 2022-10-27 15:40:31] ...optimizers/trijoint.py.151: total_norm: 8.144188 [I 2022-10-27 15:40:37] ...trap/engines/engine.py.224: train: epoch 75 | batch 70/2304 [I 2022-10-27 15:40:37] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:42 [I 2022-10-27 15:40:37] ...trap/engines/engine.py.229: process: 0.54586 | load: 0.00265 [I 2022-10-27 15:40:37] ...trap/engines/engine.py.230: loss: 0.17845 [I 2022-10-27 15:40:37] ...optimizers/trijoint.py.151: total_norm: 15.450776 [I 2022-10-27 15:40:43] ...trap/engines/engine.py.224: train: epoch 75 | batch 80/2304 [I 2022-10-27 15:40:43] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:38 [I 2022-10-27 15:40:43] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00282 [I 2022-10-27 15:40:43] ...trap/engines/engine.py.230: loss: 0.23067 [I 2022-10-27 15:40:43] ...optimizers/trijoint.py.151: total_norm: 30.007753 [I 2022-10-27 15:40:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 90/2304 [I 2022-10-27 15:40:48] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:32 [I 2022-10-27 15:40:48] ...trap/engines/engine.py.229: process: 0.53911 | load: 0.00290 [I 2022-10-27 15:40:48] ...trap/engines/engine.py.230: loss: 0.18304 [I 2022-10-27 15:40:48] ...optimizers/trijoint.py.151: total_norm: 39.200653 [I 2022-10-27 15:40:54] ...trap/engines/engine.py.224: train: epoch 75 | batch 100/2304 [I 2022-10-27 15:40:54] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:20 [I 2022-10-27 15:40:54] ...trap/engines/engine.py.229: process: 0.54646 | load: 0.00263 [I 2022-10-27 15:40:54] ...trap/engines/engine.py.230: loss: 0.13984 [I 2022-10-27 15:40:54] ...optimizers/trijoint.py.151: total_norm: 21.050024 [I 2022-10-27 15:40:59] ...trap/engines/engine.py.224: train: epoch 75 | batch 110/2304 [I 2022-10-27 15:40:59] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:47 [I 2022-10-27 15:40:59] ...trap/engines/engine.py.229: process: 0.55484 | load: 0.00286 [I 2022-10-27 15:40:59] ...trap/engines/engine.py.230: loss: 0.12893 [I 2022-10-27 15:40:59] ...optimizers/trijoint.py.151: total_norm: 10.653267 [I 2022-10-27 15:41:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 120/2304 [I 2022-10-27 15:41:05] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:28 [I 2022-10-27 15:41:05] ...trap/engines/engine.py.229: process: 0.54724 | load: 0.00266 [I 2022-10-27 15:41:05] ...trap/engines/engine.py.230: loss: 0.13424 [I 2022-10-27 15:41:05] ...optimizers/trijoint.py.151: total_norm: 16.241194 [I 2022-10-27 15:41:11] ...trap/engines/engine.py.224: train: epoch 75 | batch 130/2304 [I 2022-10-27 15:41:11] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:12 [I 2022-10-27 15:41:11] ...trap/engines/engine.py.229: process: 0.57614 | load: 0.00301 [I 2022-10-27 15:41:11] ...trap/engines/engine.py.230: loss: 0.10189 [I 2022-10-27 15:41:11] ...optimizers/trijoint.py.151: total_norm: 10.413583 [I 2022-10-27 15:41:16] ...trap/engines/engine.py.224: train: epoch 75 | batch 140/2304 [I 2022-10-27 15:41:16] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:10 [I 2022-10-27 15:41:16] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00301 [I 2022-10-27 15:41:16] ...trap/engines/engine.py.230: loss: 0.09713 [I 2022-10-27 15:41:16] ...optimizers/trijoint.py.151: total_norm: 9.742659 [I 2022-10-27 15:41:22] ...trap/engines/engine.py.224: train: epoch 75 | batch 150/2304 [I 2022-10-27 15:41:22] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:15 [I 2022-10-27 15:41:22] ...trap/engines/engine.py.229: process: 0.56893 | load: 0.00277 [I 2022-10-27 15:41:22] ...trap/engines/engine.py.230: loss: 0.24108 [I 2022-10-27 15:41:22] ...optimizers/trijoint.py.151: total_norm: 27.291893 [I 2022-10-27 15:41:28] ...trap/engines/engine.py.224: train: epoch 75 | batch 160/2304 [I 2022-10-27 15:41:28] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:59 [I 2022-10-27 15:41:28] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00271 [I 2022-10-27 15:41:28] ...trap/engines/engine.py.230: loss: 0.12759 [I 2022-10-27 15:41:28] ...optimizers/trijoint.py.151: total_norm: 20.391094 [I 2022-10-27 15:41:33] ...trap/engines/engine.py.224: train: epoch 75 | batch 170/2304 [I 2022-10-27 15:41:33] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:56 [I 2022-10-27 15:41:33] ...trap/engines/engine.py.229: process: 0.56740 | load: 0.00272 [I 2022-10-27 15:41:33] ...trap/engines/engine.py.230: loss: 0.17653 [I 2022-10-27 15:41:33] ...optimizers/trijoint.py.151: total_norm: 23.386564 [I 2022-10-27 15:41:39] ...trap/engines/engine.py.224: train: epoch 75 | batch 180/2304 [I 2022-10-27 15:41:39] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:51 [I 2022-10-27 15:41:39] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00256 [I 2022-10-27 15:41:39] ...trap/engines/engine.py.230: loss: 0.18296 [I 2022-10-27 15:41:39] ...optimizers/trijoint.py.151: total_norm: 13.551035 [I 2022-10-27 15:41:45] ...trap/engines/engine.py.224: train: epoch 75 | batch 190/2304 [I 2022-10-27 15:41:45] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:46 [I 2022-10-27 15:41:45] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00265 [I 2022-10-27 15:41:45] ...trap/engines/engine.py.230: loss: 0.09957 [I 2022-10-27 15:41:45] ...optimizers/trijoint.py.151: total_norm: 10.543433 [I 2022-10-27 15:41:50] ...trap/engines/engine.py.224: train: epoch 75 | batch 200/2304 [I 2022-10-27 15:41:50] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:27 [I 2022-10-27 15:41:50] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00297 [I 2022-10-27 15:41:50] ...trap/engines/engine.py.230: loss: 0.12638 [I 2022-10-27 15:41:50] ...optimizers/trijoint.py.151: total_norm: 10.263480 [I 2022-10-27 15:41:56] ...trap/engines/engine.py.224: train: epoch 75 | batch 210/2304 [I 2022-10-27 15:41:56] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:25 [I 2022-10-27 15:41:56] ...trap/engines/engine.py.229: process: 0.54361 | load: 0.00286 [I 2022-10-27 15:41:56] ...trap/engines/engine.py.230: loss: 0.17113 [I 2022-10-27 15:41:56] ...optimizers/trijoint.py.151: total_norm: 12.363647 [I 2022-10-27 15:42:01] ...trap/engines/engine.py.224: train: epoch 75 | batch 220/2304 [I 2022-10-27 15:42:01] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:17 [I 2022-10-27 15:42:01] ...trap/engines/engine.py.229: process: 0.56598 | load: 0.00278 [I 2022-10-27 15:42:01] ...trap/engines/engine.py.230: loss: 0.13508 [I 2022-10-27 15:42:01] ...optimizers/trijoint.py.151: total_norm: 17.842941 [I 2022-10-27 15:42:07] ...trap/engines/engine.py.224: train: epoch 75 | batch 230/2304 [I 2022-10-27 15:42:07] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:10 [I 2022-10-27 15:42:07] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00270 [I 2022-10-27 15:42:07] ...trap/engines/engine.py.230: loss: 0.09281 [I 2022-10-27 15:42:07] ...optimizers/trijoint.py.151: total_norm: 10.078961 [I 2022-10-27 15:42:12] ...trap/engines/engine.py.224: train: epoch 75 | batch 240/2304 [I 2022-10-27 15:42:12] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:05 [I 2022-10-27 15:42:12] ...trap/engines/engine.py.229: process: 0.54804 | load: 0.00306 [I 2022-10-27 15:42:12] ...trap/engines/engine.py.230: loss: 0.11446 [I 2022-10-27 15:42:12] ...optimizers/trijoint.py.151: total_norm: 47.450310 [I 2022-10-27 15:42:18] ...trap/engines/engine.py.224: train: epoch 75 | batch 250/2304 [I 2022-10-27 15:42:18] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:00 [I 2022-10-27 15:42:18] ...trap/engines/engine.py.229: process: 0.55375 | load: 0.00274 [I 2022-10-27 15:42:18] ...trap/engines/engine.py.230: loss: 0.13506 [I 2022-10-27 15:42:18] ...optimizers/trijoint.py.151: total_norm: 16.620741 [I 2022-10-27 15:42:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 260/2304 [I 2022-10-27 15:42:23] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:01 [I 2022-10-27 15:42:23] ...trap/engines/engine.py.229: process: 0.55892 | load: 0.00284 [I 2022-10-27 15:42:23] ...trap/engines/engine.py.230: loss: 0.11733 [I 2022-10-27 15:42:23] ...optimizers/trijoint.py.151: total_norm: 14.818083 [I 2022-10-27 15:42:29] ...trap/engines/engine.py.224: train: epoch 75 | batch 270/2304 [I 2022-10-27 15:42:29] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:47 [I 2022-10-27 15:42:29] ...trap/engines/engine.py.229: process: 0.53950 | load: 0.00253 [I 2022-10-27 15:42:29] ...trap/engines/engine.py.230: loss: 0.16638 [I 2022-10-27 15:42:29] ...optimizers/trijoint.py.151: total_norm: 24.677235 [I 2022-10-27 15:42:35] ...trap/engines/engine.py.224: train: epoch 75 | batch 280/2304 [I 2022-10-27 15:42:35] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:54 [I 2022-10-27 15:42:35] ...trap/engines/engine.py.229: process: 0.56819 | load: 0.00256 [I 2022-10-27 15:42:35] ...trap/engines/engine.py.230: loss: 0.12862 [I 2022-10-27 15:42:35] ...optimizers/trijoint.py.151: total_norm: 25.857388 [I 2022-10-27 15:42:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 290/2304 [I 2022-10-27 15:42:40] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:49 [I 2022-10-27 15:42:40] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00264 [I 2022-10-27 15:42:40] ...trap/engines/engine.py.230: loss: 0.17596 [I 2022-10-27 15:42:40] ...optimizers/trijoint.py.151: total_norm: 12.505360 [I 2022-10-27 15:42:46] ...trap/engines/engine.py.224: train: epoch 75 | batch 300/2304 [I 2022-10-27 15:42:46] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:50 [I 2022-10-27 15:42:46] ...trap/engines/engine.py.229: process: 0.58197 | load: 0.00262 [I 2022-10-27 15:42:46] ...trap/engines/engine.py.230: loss: 0.10944 [I 2022-10-27 15:42:46] ...optimizers/trijoint.py.151: total_norm: 12.714776 [I 2022-10-27 15:42:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 310/2304 [I 2022-10-27 15:42:51] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:35 [I 2022-10-27 15:42:51] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00248 [I 2022-10-27 15:42:51] ...trap/engines/engine.py.230: loss: 0.15171 [I 2022-10-27 15:42:51] ...optimizers/trijoint.py.151: total_norm: 16.905603 [I 2022-10-27 15:42:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 320/2304 [I 2022-10-27 15:42:57] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:42 [I 2022-10-27 15:42:57] ...trap/engines/engine.py.229: process: 0.58123 | load: 0.00256 [I 2022-10-27 15:42:57] ...trap/engines/engine.py.230: loss: 0.18186 [I 2022-10-27 15:42:57] ...optimizers/trijoint.py.151: total_norm: 32.228374 [I 2022-10-27 15:43:03] ...trap/engines/engine.py.224: train: epoch 75 | batch 330/2304 [I 2022-10-27 15:43:03] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:34 [I 2022-10-27 15:43:03] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00252 [I 2022-10-27 15:43:03] ...trap/engines/engine.py.230: loss: 0.15417 [I 2022-10-27 15:43:03] ...optimizers/trijoint.py.151: total_norm: 30.633444 [I 2022-10-27 15:43:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 340/2304 [I 2022-10-27 15:43:08] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:24 [I 2022-10-27 15:43:08] ...trap/engines/engine.py.229: process: 0.57045 | load: 0.00531 [I 2022-10-27 15:43:08] ...trap/engines/engine.py.230: loss: 0.14454 [I 2022-10-27 15:43:08] ...optimizers/trijoint.py.151: total_norm: 21.601271 [I 2022-10-27 15:43:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 350/2304 [I 2022-10-27 15:43:14] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:04 [I 2022-10-27 15:43:14] ...trap/engines/engine.py.229: process: 0.53762 | load: 0.00243 [I 2022-10-27 15:43:14] ...trap/engines/engine.py.230: loss: 0.16389 [I 2022-10-27 15:43:14] ...optimizers/trijoint.py.151: total_norm: 10.813810 [I 2022-10-27 15:43:20] ...trap/engines/engine.py.224: train: epoch 75 | batch 360/2304 [I 2022-10-27 15:43:20] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:06 [I 2022-10-27 15:43:20] ...trap/engines/engine.py.229: process: 0.56264 | load: 0.00242 [I 2022-10-27 15:43:20] ...trap/engines/engine.py.230: loss: 0.13407 [I 2022-10-27 15:43:20] ...optimizers/trijoint.py.151: total_norm: 16.740555 [I 2022-10-27 15:43:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 370/2304 [I 2022-10-27 15:43:25] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:50 [I 2022-10-27 15:43:25] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00247 [I 2022-10-27 15:43:25] ...trap/engines/engine.py.230: loss: 0.12518 [I 2022-10-27 15:43:25] ...optimizers/trijoint.py.151: total_norm: 12.776438 [I 2022-10-27 15:43:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 380/2304 [I 2022-10-27 15:43:31] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:59 [I 2022-10-27 15:43:31] ...trap/engines/engine.py.229: process: 0.57238 | load: 0.00243 [I 2022-10-27 15:43:31] ...trap/engines/engine.py.230: loss: 0.11131 [I 2022-10-27 15:43:31] ...optimizers/trijoint.py.151: total_norm: 16.458839 [I 2022-10-27 15:43:36] ...trap/engines/engine.py.224: train: epoch 75 | batch 390/2304 [I 2022-10-27 15:43:36] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:18:05 [I 2022-10-27 15:43:36] ...trap/engines/engine.py.229: process: 0.56319 | load: 0.00242 [I 2022-10-27 15:43:36] ...trap/engines/engine.py.230: loss: 0.13047 [I 2022-10-27 15:43:36] ...optimizers/trijoint.py.151: total_norm: 8.765868 [I 2022-10-27 15:43:42] ...trap/engines/engine.py.224: train: epoch 75 | batch 400/2304 [I 2022-10-27 15:43:42] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:51 [I 2022-10-27 15:43:42] ...trap/engines/engine.py.229: process: 0.57705 | load: 0.00248 [I 2022-10-27 15:43:42] ...trap/engines/engine.py.230: loss: 0.14936 [I 2022-10-27 15:43:42] ...optimizers/trijoint.py.151: total_norm: 17.587296 [I 2022-10-27 15:43:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 410/2304 [I 2022-10-27 15:43:48] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:44 [I 2022-10-27 15:43:48] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00267 [I 2022-10-27 15:43:48] ...trap/engines/engine.py.230: loss: 0.21547 [I 2022-10-27 15:43:48] ...optimizers/trijoint.py.151: total_norm: 26.169359 [I 2022-10-27 15:43:53] ...trap/engines/engine.py.224: train: epoch 75 | batch 420/2304 [I 2022-10-27 15:43:53] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:45 [I 2022-10-27 15:43:53] ...trap/engines/engine.py.229: process: 0.57257 | load: 0.00266 [I 2022-10-27 15:43:53] ...trap/engines/engine.py.230: loss: 0.10206 [I 2022-10-27 15:43:53] ...optimizers/trijoint.py.151: total_norm: 16.213427 [I 2022-10-27 15:43:59] ...trap/engines/engine.py.224: train: epoch 75 | batch 430/2304 [I 2022-10-27 15:43:59] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:30 [I 2022-10-27 15:43:59] ...trap/engines/engine.py.229: process: 0.55575 | load: 0.00248 [I 2022-10-27 15:43:59] ...trap/engines/engine.py.230: loss: 0.11862 [I 2022-10-27 15:43:59] ...optimizers/trijoint.py.151: total_norm: 9.359010 [I 2022-10-27 15:44:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 440/2304 [I 2022-10-27 15:44:05] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:25 [I 2022-10-27 15:44:05] ...trap/engines/engine.py.229: process: 0.56303 | load: 0.00243 [I 2022-10-27 15:44:05] ...trap/engines/engine.py.230: loss: 0.12582 [I 2022-10-27 15:44:05] ...optimizers/trijoint.py.151: total_norm: 13.010183 [I 2022-10-27 15:44:10] ...trap/engines/engine.py.224: train: epoch 75 | batch 450/2304 [I 2022-10-27 15:44:10] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:21 [I 2022-10-27 15:44:10] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00259 [I 2022-10-27 15:44:10] ...trap/engines/engine.py.230: loss: 0.20917 [I 2022-10-27 15:44:10] ...optimizers/trijoint.py.151: total_norm: 11.754328 [I 2022-10-27 15:44:16] ...trap/engines/engine.py.224: train: epoch 75 | batch 460/2304 [I 2022-10-27 15:44:16] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:16 [I 2022-10-27 15:44:16] ...trap/engines/engine.py.229: process: 0.56613 | load: 0.00248 [I 2022-10-27 15:44:16] ...trap/engines/engine.py.230: loss: 0.10132 [I 2022-10-27 15:44:16] ...optimizers/trijoint.py.151: total_norm: 14.248412 [I 2022-10-27 15:44:21] ...trap/engines/engine.py.224: train: epoch 75 | batch 470/2304 [I 2022-10-27 15:44:21] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:12 [I 2022-10-27 15:44:21] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00256 [I 2022-10-27 15:44:21] ...trap/engines/engine.py.230: loss: 0.17380 [I 2022-10-27 15:44:21] ...optimizers/trijoint.py.151: total_norm: 15.949262 [I 2022-10-27 15:44:27] ...trap/engines/engine.py.224: train: epoch 75 | batch 480/2304 [I 2022-10-27 15:44:27] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:59 [I 2022-10-27 15:44:27] ...trap/engines/engine.py.229: process: 0.56332 | load: 0.00251 [I 2022-10-27 15:44:27] ...trap/engines/engine.py.230: loss: 0.15419 [I 2022-10-27 15:44:27] ...optimizers/trijoint.py.151: total_norm: 15.814312 [I 2022-10-27 15:44:33] ...trap/engines/engine.py.224: train: epoch 75 | batch 490/2304 [I 2022-10-27 15:44:33] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:55 [I 2022-10-27 15:44:33] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00247 [I 2022-10-27 15:44:33] ...trap/engines/engine.py.230: loss: 0.14543 [I 2022-10-27 15:44:33] ...optimizers/trijoint.py.151: total_norm: 13.842668 [I 2022-10-27 15:44:38] ...trap/engines/engine.py.224: train: epoch 75 | batch 500/2304 [I 2022-10-27 15:44:38] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:46 [I 2022-10-27 15:44:38] ...trap/engines/engine.py.229: process: 0.56746 | load: 0.00263 [I 2022-10-27 15:44:38] ...trap/engines/engine.py.230: loss: 0.10107 [I 2022-10-27 15:44:38] ...optimizers/trijoint.py.151: total_norm: 16.380104 [I 2022-10-27 15:44:44] ...trap/engines/engine.py.224: train: epoch 75 | batch 510/2304 [I 2022-10-27 15:44:44] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:32 [I 2022-10-27 15:44:44] ...trap/engines/engine.py.229: process: 0.54498 | load: 0.00247 [I 2022-10-27 15:44:44] ...trap/engines/engine.py.230: loss: 0.20534 [I 2022-10-27 15:44:44] ...optimizers/trijoint.py.151: total_norm: 29.416462 [I 2022-10-27 15:44:49] ...trap/engines/engine.py.224: train: epoch 75 | batch 520/2304 [I 2022-10-27 15:44:49] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:39 [I 2022-10-27 15:44:49] ...trap/engines/engine.py.229: process: 0.57108 | load: 0.00248 [I 2022-10-27 15:44:49] ...trap/engines/engine.py.230: loss: 0.17927 [I 2022-10-27 15:44:49] ...optimizers/trijoint.py.151: total_norm: 36.993805 [I 2022-10-27 15:44:55] ...trap/engines/engine.py.224: train: epoch 75 | batch 530/2304 [I 2022-10-27 15:44:55] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:34 [I 2022-10-27 15:44:55] ...trap/engines/engine.py.229: process: 0.55434 | load: 0.00245 [I 2022-10-27 15:44:55] ...trap/engines/engine.py.230: loss: 0.11404 [I 2022-10-27 15:44:55] ...optimizers/trijoint.py.151: total_norm: 14.365825 [I 2022-10-27 15:45:01] ...trap/engines/engine.py.224: train: epoch 75 | batch 540/2304 [I 2022-10-27 15:45:01] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:27 [I 2022-10-27 15:45:01] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00250 [I 2022-10-27 15:45:01] ...trap/engines/engine.py.230: loss: 0.14872 [I 2022-10-27 15:45:01] ...optimizers/trijoint.py.151: total_norm: 12.337683 [I 2022-10-27 15:45:06] ...trap/engines/engine.py.224: train: epoch 75 | batch 550/2304 [I 2022-10-27 15:45:06] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:20 [I 2022-10-27 15:45:06] ...trap/engines/engine.py.229: process: 0.55347 | load: 0.00251 [I 2022-10-27 15:45:06] ...trap/engines/engine.py.230: loss: 0.14942 [I 2022-10-27 15:45:06] ...optimizers/trijoint.py.151: total_norm: 19.423454 [I 2022-10-27 15:45:12] ...trap/engines/engine.py.224: train: epoch 75 | batch 560/2304 [I 2022-10-27 15:45:12] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:19 [I 2022-10-27 15:45:12] ...trap/engines/engine.py.229: process: 0.57137 | load: 0.00256 [I 2022-10-27 15:45:12] ...trap/engines/engine.py.230: loss: 0.12015 [I 2022-10-27 15:45:12] ...optimizers/trijoint.py.151: total_norm: 17.171728 [I 2022-10-27 15:45:18] ...trap/engines/engine.py.224: train: epoch 75 | batch 570/2304 [I 2022-10-27 15:45:18] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:11 [I 2022-10-27 15:45:18] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00251 [I 2022-10-27 15:45:18] ...trap/engines/engine.py.230: loss: 0.13076 [I 2022-10-27 15:45:18] ...optimizers/trijoint.py.151: total_norm: 14.254858 [I 2022-10-27 15:45:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 580/2304 [I 2022-10-27 15:45:23] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:09 [I 2022-10-27 15:45:23] ...trap/engines/engine.py.229: process: 0.56742 | load: 0.00252 [I 2022-10-27 15:45:23] ...trap/engines/engine.py.230: loss: 0.11865 [I 2022-10-27 15:45:23] ...optimizers/trijoint.py.151: total_norm: 10.315904 [I 2022-10-27 15:45:29] ...trap/engines/engine.py.224: train: epoch 75 | batch 590/2304 [I 2022-10-27 15:45:29] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:58 [I 2022-10-27 15:45:29] ...trap/engines/engine.py.229: process: 0.56457 | load: 0.00249 [I 2022-10-27 15:45:29] ...trap/engines/engine.py.230: loss: 0.11141 [I 2022-10-27 15:45:29] ...optimizers/trijoint.py.151: total_norm: 10.309814 [I 2022-10-27 15:45:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 600/2304 [I 2022-10-27 15:45:34] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:45 [I 2022-10-27 15:45:34] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00243 [I 2022-10-27 15:45:34] ...trap/engines/engine.py.230: loss: 0.08023 [I 2022-10-27 15:45:34] ...optimizers/trijoint.py.151: total_norm: 12.960420 [I 2022-10-27 15:45:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 610/2304 [I 2022-10-27 15:45:40] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:49 [I 2022-10-27 15:45:40] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00251 [I 2022-10-27 15:45:40] ...trap/engines/engine.py.230: loss: 0.17105 [I 2022-10-27 15:45:40] ...optimizers/trijoint.py.151: total_norm: 58.259342 [I 2022-10-27 15:45:46] ...trap/engines/engine.py.224: train: epoch 75 | batch 620/2304 [I 2022-10-27 15:45:46] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:43 [I 2022-10-27 15:45:46] ...trap/engines/engine.py.229: process: 0.56511 | load: 0.00241 [I 2022-10-27 15:45:46] ...trap/engines/engine.py.230: loss: 0.11583 [I 2022-10-27 15:45:46] ...optimizers/trijoint.py.151: total_norm: 14.151895 [I 2022-10-27 15:45:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 630/2304 [I 2022-10-27 15:45:51] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:29 [I 2022-10-27 15:45:51] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00246 [I 2022-10-27 15:45:51] ...trap/engines/engine.py.230: loss: 0.12647 [I 2022-10-27 15:45:51] ...optimizers/trijoint.py.151: total_norm: 12.155210 [I 2022-10-27 15:45:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 640/2304 [I 2022-10-27 15:45:57] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:24 [I 2022-10-27 15:45:57] ...trap/engines/engine.py.229: process: 0.54910 | load: 0.00245 [I 2022-10-27 15:45:57] ...trap/engines/engine.py.230: loss: 0.13484 [I 2022-10-27 15:45:57] ...optimizers/trijoint.py.151: total_norm: 11.149859 [I 2022-10-27 15:46:02] ...trap/engines/engine.py.224: train: epoch 75 | batch 650/2304 [I 2022-10-27 15:46:02] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:16 [I 2022-10-27 15:46:02] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00248 [I 2022-10-27 15:46:02] ...trap/engines/engine.py.230: loss: 0.11109 [I 2022-10-27 15:46:02] ...optimizers/trijoint.py.151: total_norm: 10.690278 [I 2022-10-27 15:46:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 660/2304 [I 2022-10-27 15:46:08] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:17 [I 2022-10-27 15:46:08] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00247 [I 2022-10-27 15:46:08] ...trap/engines/engine.py.230: loss: 0.13339 [I 2022-10-27 15:46:08] ...optimizers/trijoint.py.151: total_norm: 11.991394 [I 2022-10-27 15:46:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 670/2304 [I 2022-10-27 15:46:14] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:35 [I 2022-10-27 15:46:14] ...trap/engines/engine.py.229: process: 0.56803 | load: 0.00258 [I 2022-10-27 15:46:14] ...trap/engines/engine.py.230: loss: 0.13215 [I 2022-10-27 15:46:14] ...optimizers/trijoint.py.151: total_norm: 23.790318 [I 2022-10-27 15:46:19] ...trap/engines/engine.py.224: train: epoch 75 | batch 680/2304 [I 2022-10-27 15:46:19] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:06 [I 2022-10-27 15:46:19] ...trap/engines/engine.py.229: process: 0.54900 | load: 0.00247 [I 2022-10-27 15:46:19] ...trap/engines/engine.py.230: loss: 0.14588 [I 2022-10-27 15:46:19] ...optimizers/trijoint.py.151: total_norm: 7.611679 [I 2022-10-27 15:46:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 690/2304 [I 2022-10-27 15:46:25] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:06 [I 2022-10-27 15:46:25] ...trap/engines/engine.py.229: process: 0.55220 | load: 0.00243 [I 2022-10-27 15:46:25] ...trap/engines/engine.py.230: loss: 0.17904 [I 2022-10-27 15:46:25] ...optimizers/trijoint.py.151: total_norm: 15.690688 [I 2022-10-27 15:46:30] ...trap/engines/engine.py.224: train: epoch 75 | batch 700/2304 [I 2022-10-27 15:46:30] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:15:02 [I 2022-10-27 15:46:30] ...trap/engines/engine.py.229: process: 0.58808 | load: 0.00247 [I 2022-10-27 15:46:30] ...trap/engines/engine.py.230: loss: 0.17021 [I 2022-10-27 15:46:30] ...optimizers/trijoint.py.151: total_norm: 13.776729 [I 2022-10-27 15:46:36] ...trap/engines/engine.py.224: train: epoch 75 | batch 710/2304 [I 2022-10-27 15:46:36] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:51 [I 2022-10-27 15:46:36] ...trap/engines/engine.py.229: process: 0.55578 | load: 0.00254 [I 2022-10-27 15:46:36] ...trap/engines/engine.py.230: loss: 0.16980 [I 2022-10-27 15:46:36] ...optimizers/trijoint.py.151: total_norm: 40.643970 [I 2022-10-27 15:46:42] ...trap/engines/engine.py.224: train: epoch 75 | batch 720/2304 [I 2022-10-27 15:46:42] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:38 [I 2022-10-27 15:46:42] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00254 [I 2022-10-27 15:46:42] ...trap/engines/engine.py.230: loss: 0.14218 [I 2022-10-27 15:46:42] ...optimizers/trijoint.py.151: total_norm: 19.723963 [I 2022-10-27 15:46:47] ...trap/engines/engine.py.224: train: epoch 75 | batch 730/2304 [I 2022-10-27 15:46:47] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:33 [I 2022-10-27 15:46:47] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00250 [I 2022-10-27 15:46:47] ...trap/engines/engine.py.230: loss: 0.10916 [I 2022-10-27 15:46:47] ...optimizers/trijoint.py.151: total_norm: 14.988868 [I 2022-10-27 15:46:53] ...trap/engines/engine.py.224: train: epoch 75 | batch 740/2304 [I 2022-10-27 15:46:53] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:22 [I 2022-10-27 15:46:53] ...trap/engines/engine.py.229: process: 0.54372 | load: 0.00253 [I 2022-10-27 15:46:53] ...trap/engines/engine.py.230: loss: 0.12262 [I 2022-10-27 15:46:53] ...optimizers/trijoint.py.151: total_norm: 16.478081 [I 2022-10-27 15:46:58] ...trap/engines/engine.py.224: train: epoch 75 | batch 750/2304 [I 2022-10-27 15:46:58] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:22 [I 2022-10-27 15:46:58] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00258 [I 2022-10-27 15:46:58] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-27 15:46:58] ...optimizers/trijoint.py.151: total_norm: 19.211250 [I 2022-10-27 15:47:04] ...trap/engines/engine.py.224: train: epoch 75 | batch 760/2304 [I 2022-10-27 15:47:04] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:23 [I 2022-10-27 15:47:04] ...trap/engines/engine.py.229: process: 0.54830 | load: 0.00253 [I 2022-10-27 15:47:04] ...trap/engines/engine.py.230: loss: 0.12059 [I 2022-10-27 15:47:04] ...optimizers/trijoint.py.151: total_norm: 37.485889 [I 2022-10-27 15:47:10] ...trap/engines/engine.py.224: train: epoch 75 | batch 770/2304 [I 2022-10-27 15:47:10] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:10 [I 2022-10-27 15:47:10] ...trap/engines/engine.py.229: process: 0.55873 | load: 0.00254 [I 2022-10-27 15:47:10] ...trap/engines/engine.py.230: loss: 0.13480 [I 2022-10-27 15:47:10] ...optimizers/trijoint.py.151: total_norm: 13.222754 [I 2022-10-27 15:47:15] ...trap/engines/engine.py.224: train: epoch 75 | batch 780/2304 [I 2022-10-27 15:47:15] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:12 [I 2022-10-27 15:47:15] ...trap/engines/engine.py.229: process: 0.56844 | load: 0.00249 [I 2022-10-27 15:47:15] ...trap/engines/engine.py.230: loss: 0.11085 [I 2022-10-27 15:47:15] ...optimizers/trijoint.py.151: total_norm: 19.657999 [I 2022-10-27 15:47:21] ...trap/engines/engine.py.224: train: epoch 75 | batch 790/2304 [I 2022-10-27 15:47:21] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:13:59 [I 2022-10-27 15:47:21] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00253 [I 2022-10-27 15:47:21] ...trap/engines/engine.py.230: loss: 0.18072 [I 2022-10-27 15:47:21] ...optimizers/trijoint.py.151: total_norm: 28.015371 [I 2022-10-27 15:47:26] ...trap/engines/engine.py.224: train: epoch 75 | batch 800/2304 [I 2022-10-27 15:47:26] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:13:58 [I 2022-10-27 15:47:26] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00249 [I 2022-10-27 15:47:26] ...trap/engines/engine.py.230: loss: 0.13946 [I 2022-10-27 15:47:26] ...optimizers/trijoint.py.151: total_norm: 16.161629 [I 2022-10-27 15:47:32] ...trap/engines/engine.py.224: train: epoch 75 | batch 810/2304 [I 2022-10-27 15:47:32] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:47 [I 2022-10-27 15:47:32] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00265 [I 2022-10-27 15:47:32] ...trap/engines/engine.py.230: loss: 0.10326 [I 2022-10-27 15:47:32] ...optimizers/trijoint.py.151: total_norm: 8.687138 [I 2022-10-27 15:47:37] ...trap/engines/engine.py.224: train: epoch 75 | batch 820/2304 [I 2022-10-27 15:47:37] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:44 [I 2022-10-27 15:47:37] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00244 [I 2022-10-27 15:47:37] ...trap/engines/engine.py.230: loss: 0.12029 [I 2022-10-27 15:47:37] ...optimizers/trijoint.py.151: total_norm: 13.886554 [I 2022-10-27 15:47:43] ...trap/engines/engine.py.224: train: epoch 75 | batch 830/2304 [I 2022-10-27 15:47:43] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:43 [I 2022-10-27 15:47:43] ...trap/engines/engine.py.229: process: 0.54305 | load: 0.00246 [I 2022-10-27 15:47:43] ...trap/engines/engine.py.230: loss: 0.18380 [I 2022-10-27 15:47:43] ...optimizers/trijoint.py.151: total_norm: 24.657543 [I 2022-10-27 15:47:49] ...trap/engines/engine.py.224: train: epoch 75 | batch 840/2304 [I 2022-10-27 15:47:49] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:32 [I 2022-10-27 15:47:49] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00268 [I 2022-10-27 15:47:49] ...trap/engines/engine.py.230: loss: 0.12957 [I 2022-10-27 15:47:49] ...optimizers/trijoint.py.151: total_norm: 10.274140 [I 2022-10-27 15:47:54] ...trap/engines/engine.py.224: train: epoch 75 | batch 850/2304 [I 2022-10-27 15:47:54] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:27 [I 2022-10-27 15:47:54] ...trap/engines/engine.py.229: process: 0.56258 | load: 0.00246 [I 2022-10-27 15:47:54] ...trap/engines/engine.py.230: loss: 0.14330 [I 2022-10-27 15:47:54] ...optimizers/trijoint.py.151: total_norm: 11.402595 [I 2022-10-27 15:48:00] ...trap/engines/engine.py.224: train: epoch 75 | batch 860/2304 [I 2022-10-27 15:48:00] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:23 [I 2022-10-27 15:48:00] ...trap/engines/engine.py.229: process: 0.55218 | load: 0.00240 [I 2022-10-27 15:48:00] ...trap/engines/engine.py.230: loss: 0.16991 [I 2022-10-27 15:48:00] ...optimizers/trijoint.py.151: total_norm: 37.310974 [I 2022-10-27 15:48:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 870/2304 [I 2022-10-27 15:48:05] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:16 [I 2022-10-27 15:48:05] ...trap/engines/engine.py.229: process: 0.56515 | load: 0.00246 [I 2022-10-27 15:48:05] ...trap/engines/engine.py.230: loss: 0.20459 [I 2022-10-27 15:48:05] ...optimizers/trijoint.py.151: total_norm: 26.991545 [I 2022-10-27 15:48:11] ...trap/engines/engine.py.224: train: epoch 75 | batch 880/2304 [I 2022-10-27 15:48:11] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:12 [I 2022-10-27 15:48:11] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00259 [I 2022-10-27 15:48:11] ...trap/engines/engine.py.230: loss: 0.17276 [I 2022-10-27 15:48:11] ...optimizers/trijoint.py.151: total_norm: 27.944895 [I 2022-10-27 15:48:17] ...trap/engines/engine.py.224: train: epoch 75 | batch 890/2304 [I 2022-10-27 15:48:17] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:14 [I 2022-10-27 15:48:17] ...trap/engines/engine.py.229: process: 0.55006 | load: 0.00254 [I 2022-10-27 15:48:17] ...trap/engines/engine.py.230: loss: 0.11433 [I 2022-10-27 15:48:17] ...optimizers/trijoint.py.151: total_norm: 16.387278 [I 2022-10-27 15:48:22] ...trap/engines/engine.py.224: train: epoch 75 | batch 900/2304 [I 2022-10-27 15:48:22] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:12:59 [I 2022-10-27 15:48:22] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00254 [I 2022-10-27 15:48:22] ...trap/engines/engine.py.230: loss: 0.18261 [I 2022-10-27 15:48:22] ...optimizers/trijoint.py.151: total_norm: 10.815463 [I 2022-10-27 15:48:28] ...trap/engines/engine.py.224: train: epoch 75 | batch 910/2304 [I 2022-10-27 15:48:28] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:54 [I 2022-10-27 15:48:28] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00247 [I 2022-10-27 15:48:28] ...trap/engines/engine.py.230: loss: 0.15934 [I 2022-10-27 15:48:28] ...optimizers/trijoint.py.151: total_norm: 17.875769 [I 2022-10-27 15:48:33] ...trap/engines/engine.py.224: train: epoch 75 | batch 920/2304 [I 2022-10-27 15:48:33] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:45 [I 2022-10-27 15:48:33] ...trap/engines/engine.py.229: process: 0.54355 | load: 0.00246 [I 2022-10-27 15:48:33] ...trap/engines/engine.py.230: loss: 0.14316 [I 2022-10-27 15:48:33] ...optimizers/trijoint.py.151: total_norm: 17.032063 [I 2022-10-27 15:48:39] ...trap/engines/engine.py.224: train: epoch 75 | batch 930/2304 [I 2022-10-27 15:48:39] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:42 [I 2022-10-27 15:48:39] ...trap/engines/engine.py.229: process: 0.55176 | load: 0.00251 [I 2022-10-27 15:48:39] ...trap/engines/engine.py.230: loss: 0.09165 [I 2022-10-27 15:48:39] ...optimizers/trijoint.py.151: total_norm: 13.323933 [I 2022-10-27 15:48:44] ...trap/engines/engine.py.224: train: epoch 75 | batch 940/2304 [I 2022-10-27 15:48:44] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:38 [I 2022-10-27 15:48:44] ...trap/engines/engine.py.229: process: 0.56590 | load: 0.00253 [I 2022-10-27 15:48:44] ...trap/engines/engine.py.230: loss: 0.17276 [I 2022-10-27 15:48:44] ...optimizers/trijoint.py.151: total_norm: 15.322537 [I 2022-10-27 15:48:50] ...trap/engines/engine.py.224: train: epoch 75 | batch 950/2304 [I 2022-10-27 15:48:50] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:13:05 [I 2022-10-27 15:48:50] ...trap/engines/engine.py.229: process: 0.67772 | load: 0.00253 [I 2022-10-27 15:48:50] ...trap/engines/engine.py.230: loss: 0.20902 [I 2022-10-27 15:48:50] ...optimizers/trijoint.py.151: total_norm: 22.181183 [I 2022-10-27 15:48:56] ...trap/engines/engine.py.224: train: epoch 75 | batch 960/2304 [I 2022-10-27 15:48:56] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:28 [I 2022-10-27 15:48:56] ...trap/engines/engine.py.229: process: 0.55609 | load: 0.00256 [I 2022-10-27 15:48:56] ...trap/engines/engine.py.230: loss: 0.15098 [I 2022-10-27 15:48:56] ...optimizers/trijoint.py.151: total_norm: 23.349003 [I 2022-10-27 15:49:01] ...trap/engines/engine.py.224: train: epoch 75 | batch 970/2304 [I 2022-10-27 15:49:01] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:18 [I 2022-10-27 15:49:01] ...trap/engines/engine.py.229: process: 0.56160 | load: 0.00243 [I 2022-10-27 15:49:01] ...trap/engines/engine.py.230: loss: 0.17257 [I 2022-10-27 15:49:01] ...optimizers/trijoint.py.151: total_norm: 17.056398 [I 2022-10-27 15:49:07] ...trap/engines/engine.py.224: train: epoch 75 | batch 980/2304 [I 2022-10-27 15:49:07] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:12 [I 2022-10-27 15:49:07] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00248 [I 2022-10-27 15:49:07] ...trap/engines/engine.py.230: loss: 0.16136 [I 2022-10-27 15:49:07] ...optimizers/trijoint.py.151: total_norm: 7.946907 [I 2022-10-27 15:49:12] ...trap/engines/engine.py.224: train: epoch 75 | batch 990/2304 [I 2022-10-27 15:49:12] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:05 [I 2022-10-27 15:49:12] ...trap/engines/engine.py.229: process: 0.54902 | load: 0.00248 [I 2022-10-27 15:49:12] ...trap/engines/engine.py.230: loss: 0.12929 [I 2022-10-27 15:49:12] ...optimizers/trijoint.py.151: total_norm: 22.351173 [I 2022-10-27 15:49:18] ...trap/engines/engine.py.224: train: epoch 75 | batch 1000/2304 [I 2022-10-27 15:49:18] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:05 [I 2022-10-27 15:49:18] ...trap/engines/engine.py.229: process: 0.54206 | load: 0.00250 [I 2022-10-27 15:49:18] ...trap/engines/engine.py.230: loss: 0.13153 [I 2022-10-27 15:49:18] ...optimizers/trijoint.py.151: total_norm: 51.657730 [I 2022-10-27 15:49:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 1010/2304 [I 2022-10-27 15:49:23] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:02 [I 2022-10-27 15:49:23] ...trap/engines/engine.py.229: process: 0.55609 | load: 0.00257 [I 2022-10-27 15:49:23] ...trap/engines/engine.py.230: loss: 0.11963 [I 2022-10-27 15:49:23] ...optimizers/trijoint.py.151: total_norm: 23.117176 [I 2022-10-27 15:49:29] ...trap/engines/engine.py.224: train: epoch 75 | batch 1020/2304 [I 2022-10-27 15:49:29] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:12:03 [I 2022-10-27 15:49:29] ...trap/engines/engine.py.229: process: 0.56455 | load: 0.00247 [I 2022-10-27 15:49:29] ...trap/engines/engine.py.230: loss: 0.10903 [I 2022-10-27 15:49:29] ...optimizers/trijoint.py.151: total_norm: 13.152544 [I 2022-10-27 15:49:35] ...trap/engines/engine.py.224: train: epoch 75 | batch 1030/2304 [I 2022-10-27 15:49:35] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:51 [I 2022-10-27 15:49:35] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00255 [I 2022-10-27 15:49:35] ...trap/engines/engine.py.230: loss: 0.12127 [I 2022-10-27 15:49:35] ...optimizers/trijoint.py.151: total_norm: 22.052341 [I 2022-10-27 15:49:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 1040/2304 [I 2022-10-27 15:49:40] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:45 [I 2022-10-27 15:49:40] ...trap/engines/engine.py.229: process: 0.55932 | load: 0.00252 [I 2022-10-27 15:49:40] ...trap/engines/engine.py.230: loss: 0.14694 [I 2022-10-27 15:49:40] ...optimizers/trijoint.py.151: total_norm: 19.662092 [I 2022-10-27 15:49:46] ...trap/engines/engine.py.224: train: epoch 75 | batch 1050/2304 [I 2022-10-27 15:49:46] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:36 [I 2022-10-27 15:49:46] ...trap/engines/engine.py.229: process: 0.55977 | load: 0.00251 [I 2022-10-27 15:49:46] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-27 15:49:46] ...optimizers/trijoint.py.151: total_norm: 12.905479 [I 2022-10-27 15:49:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 1060/2304 [I 2022-10-27 15:49:51] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:28 [I 2022-10-27 15:49:51] ...trap/engines/engine.py.229: process: 0.54533 | load: 0.00248 [I 2022-10-27 15:49:51] ...trap/engines/engine.py.230: loss: 0.21055 [I 2022-10-27 15:49:51] ...optimizers/trijoint.py.151: total_norm: 22.719656 [I 2022-10-27 15:49:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 1070/2304 [I 2022-10-27 15:49:57] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:24 [I 2022-10-27 15:49:57] ...trap/engines/engine.py.229: process: 0.54673 | load: 0.00244 [I 2022-10-27 15:49:57] ...trap/engines/engine.py.230: loss: 0.15631 [I 2022-10-27 15:49:57] ...optimizers/trijoint.py.151: total_norm: 27.548414 [I 2022-10-27 15:50:03] ...trap/engines/engine.py.224: train: epoch 75 | batch 1080/2304 [I 2022-10-27 15:50:03] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:44 [I 2022-10-27 15:50:03] ...trap/engines/engine.py.229: process: 0.57096 | load: 0.00242 [I 2022-10-27 15:50:03] ...trap/engines/engine.py.230: loss: 0.10307 [I 2022-10-27 15:50:03] ...optimizers/trijoint.py.151: total_norm: 22.655807 [I 2022-10-27 15:50:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 1090/2304 [I 2022-10-27 15:50:08] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:16 [I 2022-10-27 15:50:08] ...trap/engines/engine.py.229: process: 0.55054 | load: 0.00251 [I 2022-10-27 15:50:08] ...trap/engines/engine.py.230: loss: 0.12735 [I 2022-10-27 15:50:08] ...optimizers/trijoint.py.151: total_norm: 19.167311 [I 2022-10-27 15:50:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 1100/2304 [I 2022-10-27 15:50:14] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:08 [I 2022-10-27 15:50:14] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00243 [I 2022-10-27 15:50:14] ...trap/engines/engine.py.230: loss: 0.11271 [I 2022-10-27 15:50:14] ...optimizers/trijoint.py.151: total_norm: 10.918194 [I 2022-10-27 15:50:19] ...trap/engines/engine.py.224: train: epoch 75 | batch 1110/2304 [I 2022-10-27 15:50:19] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:06 [I 2022-10-27 15:50:19] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00246 [I 2022-10-27 15:50:19] ...trap/engines/engine.py.230: loss: 0.24860 [I 2022-10-27 15:50:19] ...optimizers/trijoint.py.151: total_norm: 31.215330 [I 2022-10-27 15:50:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 1120/2304 [I 2022-10-27 15:50:25] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:10:59 [I 2022-10-27 15:50:25] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00243 [I 2022-10-27 15:50:25] ...trap/engines/engine.py.230: loss: 0.10153 [I 2022-10-27 15:50:25] ...optimizers/trijoint.py.151: total_norm: 19.228951 [I 2022-10-27 15:50:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 1130/2304 [I 2022-10-27 15:50:31] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:46 [I 2022-10-27 15:50:31] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00251 [I 2022-10-27 15:50:31] ...trap/engines/engine.py.230: loss: 0.12356 [I 2022-10-27 15:50:31] ...optimizers/trijoint.py.151: total_norm: 15.408551 [I 2022-10-27 15:50:36] ...trap/engines/engine.py.224: train: epoch 75 | batch 1140/2304 [I 2022-10-27 15:50:36] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:50 [I 2022-10-27 15:50:36] ...trap/engines/engine.py.229: process: 0.57128 | load: 0.00242 [I 2022-10-27 15:50:36] ...trap/engines/engine.py.230: loss: 0.17642 [I 2022-10-27 15:50:36] ...optimizers/trijoint.py.151: total_norm: 27.288929 [I 2022-10-27 15:50:42] ...trap/engines/engine.py.224: train: epoch 75 | batch 1150/2304 [I 2022-10-27 15:50:42] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:48 [I 2022-10-27 15:50:42] ...trap/engines/engine.py.229: process: 0.55587 | load: 0.00252 [I 2022-10-27 15:50:42] ...trap/engines/engine.py.230: loss: 0.10854 [I 2022-10-27 15:50:42] ...optimizers/trijoint.py.151: total_norm: 23.619049 [I 2022-10-27 15:50:47] ...trap/engines/engine.py.224: train: epoch 75 | batch 1160/2304 [I 2022-10-27 15:50:47] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:32 [I 2022-10-27 15:50:47] ...trap/engines/engine.py.229: process: 0.54861 | load: 0.00250 [I 2022-10-27 15:50:47] ...trap/engines/engine.py.230: loss: 0.10454 [I 2022-10-27 15:50:47] ...optimizers/trijoint.py.151: total_norm: 9.287484 [I 2022-10-27 15:50:53] ...trap/engines/engine.py.224: train: epoch 75 | batch 1170/2304 [I 2022-10-27 15:50:53] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:27 [I 2022-10-27 15:50:53] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00250 [I 2022-10-27 15:50:53] ...trap/engines/engine.py.230: loss: 0.19299 [I 2022-10-27 15:50:53] ...optimizers/trijoint.py.151: total_norm: 42.563946 [I 2022-10-27 15:50:59] ...trap/engines/engine.py.224: train: epoch 75 | batch 1180/2304 [I 2022-10-27 15:50:59] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:24 [I 2022-10-27 15:50:59] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00244 [I 2022-10-27 15:50:59] ...trap/engines/engine.py.230: loss: 0.14183 [I 2022-10-27 15:50:59] ...optimizers/trijoint.py.151: total_norm: 18.490732 [I 2022-10-27 15:51:04] ...trap/engines/engine.py.224: train: epoch 75 | batch 1190/2304 [I 2022-10-27 15:51:04] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:22 [I 2022-10-27 15:51:04] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00257 [I 2022-10-27 15:51:04] ...trap/engines/engine.py.230: loss: 0.13801 [I 2022-10-27 15:51:04] ...optimizers/trijoint.py.151: total_norm: 8.488158 [I 2022-10-27 15:51:10] ...trap/engines/engine.py.224: train: epoch 75 | batch 1200/2304 [I 2022-10-27 15:51:10] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:13 [I 2022-10-27 15:51:10] ...trap/engines/engine.py.229: process: 0.55997 | load: 0.00259 [I 2022-10-27 15:51:10] ...trap/engines/engine.py.230: loss: 0.08981 [I 2022-10-27 15:51:10] ...optimizers/trijoint.py.151: total_norm: 9.656590 [I 2022-10-27 15:51:15] ...trap/engines/engine.py.224: train: epoch 75 | batch 1210/2304 [I 2022-10-27 15:51:15] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:06 [I 2022-10-27 15:51:15] ...trap/engines/engine.py.229: process: 0.53046 | load: 0.00257 [I 2022-10-27 15:51:15] ...trap/engines/engine.py.230: loss: 0.16563 [I 2022-10-27 15:51:15] ...optimizers/trijoint.py.151: total_norm: 18.897026 [I 2022-10-27 15:51:21] ...trap/engines/engine.py.224: train: epoch 75 | batch 1220/2304 [I 2022-10-27 15:51:21] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:00 [I 2022-10-27 15:51:21] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00254 [I 2022-10-27 15:51:21] ...trap/engines/engine.py.230: loss: 0.22276 [I 2022-10-27 15:51:21] ...optimizers/trijoint.py.151: total_norm: 20.455847 [I 2022-10-27 15:51:26] ...trap/engines/engine.py.224: train: epoch 75 | batch 1230/2304 [I 2022-10-27 15:51:26] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:09:54 [I 2022-10-27 15:51:26] ...trap/engines/engine.py.229: process: 0.54789 | load: 0.00251 [I 2022-10-27 15:51:26] ...trap/engines/engine.py.230: loss: 0.22879 [I 2022-10-27 15:51:26] ...optimizers/trijoint.py.151: total_norm: 16.240829 [I 2022-10-27 15:51:32] ...trap/engines/engine.py.224: train: epoch 75 | batch 1240/2304 [I 2022-10-27 15:51:32] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:55 [I 2022-10-27 15:51:32] ...trap/engines/engine.py.229: process: 0.56618 | load: 0.00252 [I 2022-10-27 15:51:32] ...trap/engines/engine.py.230: loss: 0.21177 [I 2022-10-27 15:51:32] ...optimizers/trijoint.py.151: total_norm: 27.731861 [I 2022-10-27 15:51:38] ...trap/engines/engine.py.224: train: epoch 75 | batch 1250/2304 [I 2022-10-27 15:51:38] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:43 [I 2022-10-27 15:51:38] ...trap/engines/engine.py.229: process: 0.54444 | load: 0.00244 [I 2022-10-27 15:51:38] ...trap/engines/engine.py.230: loss: 0.13879 [I 2022-10-27 15:51:38] ...optimizers/trijoint.py.151: total_norm: 11.530971 [I 2022-10-27 15:51:43] ...trap/engines/engine.py.224: train: epoch 75 | batch 1260/2304 [I 2022-10-27 15:51:43] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:38 [I 2022-10-27 15:51:43] ...trap/engines/engine.py.229: process: 0.56141 | load: 0.00247 [I 2022-10-27 15:51:43] ...trap/engines/engine.py.230: loss: 0.13365 [I 2022-10-27 15:51:43] ...optimizers/trijoint.py.151: total_norm: 25.465378 [I 2022-10-27 15:51:49] ...trap/engines/engine.py.224: train: epoch 75 | batch 1270/2304 [I 2022-10-27 15:51:49] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:42 [I 2022-10-27 15:51:49] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00254 [I 2022-10-27 15:51:49] ...trap/engines/engine.py.230: loss: 0.20059 [I 2022-10-27 15:51:49] ...optimizers/trijoint.py.151: total_norm: 13.505991 [I 2022-10-27 15:51:54] ...trap/engines/engine.py.224: train: epoch 75 | batch 1280/2304 [I 2022-10-27 15:51:54] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:43 [I 2022-10-27 15:51:54] ...trap/engines/engine.py.229: process: 0.58302 | load: 0.00265 [I 2022-10-27 15:51:54] ...trap/engines/engine.py.230: loss: 0.15319 [I 2022-10-27 15:51:54] ...optimizers/trijoint.py.151: total_norm: 30.520948 [I 2022-10-27 15:52:00] ...trap/engines/engine.py.224: train: epoch 75 | batch 1290/2304 [I 2022-10-27 15:52:00] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:29 [I 2022-10-27 15:52:00] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00257 [I 2022-10-27 15:52:00] ...trap/engines/engine.py.230: loss: 0.10697 [I 2022-10-27 15:52:00] ...optimizers/trijoint.py.151: total_norm: 13.057163 [I 2022-10-27 15:52:06] ...trap/engines/engine.py.224: train: epoch 75 | batch 1300/2304 [I 2022-10-27 15:52:06] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:18 [I 2022-10-27 15:52:06] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00250 [I 2022-10-27 15:52:06] ...trap/engines/engine.py.230: loss: 0.14134 [I 2022-10-27 15:52:06] ...optimizers/trijoint.py.151: total_norm: 13.822128 [I 2022-10-27 15:52:11] ...trap/engines/engine.py.224: train: epoch 75 | batch 1310/2304 [I 2022-10-27 15:52:11] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:14 [I 2022-10-27 15:52:11] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00253 [I 2022-10-27 15:52:11] ...trap/engines/engine.py.230: loss: 0.16881 [I 2022-10-27 15:52:11] ...optimizers/trijoint.py.151: total_norm: 14.572351 [I 2022-10-27 15:52:17] ...trap/engines/engine.py.224: train: epoch 75 | batch 1320/2304 [I 2022-10-27 15:52:17] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:08 [I 2022-10-27 15:52:17] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00251 [I 2022-10-27 15:52:17] ...trap/engines/engine.py.230: loss: 0.09621 [I 2022-10-27 15:52:17] ...optimizers/trijoint.py.151: total_norm: 13.415654 [I 2022-10-27 15:52:22] ...trap/engines/engine.py.224: train: epoch 75 | batch 1330/2304 [I 2022-10-27 15:52:22] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:04 [I 2022-10-27 15:52:22] ...trap/engines/engine.py.229: process: 0.55505 | load: 0.00246 [I 2022-10-27 15:52:22] ...trap/engines/engine.py.230: loss: 0.13611 [I 2022-10-27 15:52:22] ...optimizers/trijoint.py.151: total_norm: 17.085499 [I 2022-10-27 15:52:28] ...trap/engines/engine.py.224: train: epoch 75 | batch 1340/2304 [I 2022-10-27 15:52:28] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:53 [I 2022-10-27 15:52:28] ...trap/engines/engine.py.229: process: 0.54672 | load: 0.00244 [I 2022-10-27 15:52:28] ...trap/engines/engine.py.230: loss: 0.14987 [I 2022-10-27 15:52:28] ...optimizers/trijoint.py.151: total_norm: 8.635037 [I 2022-10-27 15:52:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 1350/2304 [I 2022-10-27 15:52:34] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:43 [I 2022-10-27 15:52:34] ...trap/engines/engine.py.229: process: 0.54263 | load: 0.00246 [I 2022-10-27 15:52:34] ...trap/engines/engine.py.230: loss: 0.14897 [I 2022-10-27 15:52:34] ...optimizers/trijoint.py.151: total_norm: 30.243683 [I 2022-10-27 15:52:39] ...trap/engines/engine.py.224: train: epoch 75 | batch 1360/2304 [I 2022-10-27 15:52:39] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:41 [I 2022-10-27 15:52:39] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00241 [I 2022-10-27 15:52:39] ...trap/engines/engine.py.230: loss: 0.12970 [I 2022-10-27 15:52:39] ...optimizers/trijoint.py.151: total_norm: 7.599781 [I 2022-10-27 15:52:45] ...trap/engines/engine.py.224: train: epoch 75 | batch 1370/2304 [I 2022-10-27 15:52:45] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:40 [I 2022-10-27 15:52:45] ...trap/engines/engine.py.229: process: 0.56413 | load: 0.00248 [I 2022-10-27 15:52:45] ...trap/engines/engine.py.230: loss: 0.14066 [I 2022-10-27 15:52:45] ...optimizers/trijoint.py.151: total_norm: 17.372431 [I 2022-10-27 15:52:50] ...trap/engines/engine.py.224: train: epoch 75 | batch 1380/2304 [I 2022-10-27 15:52:50] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:33 [I 2022-10-27 15:52:50] ...trap/engines/engine.py.229: process: 0.55542 | load: 0.00241 [I 2022-10-27 15:52:50] ...trap/engines/engine.py.230: loss: 0.11877 [I 2022-10-27 15:52:50] ...optimizers/trijoint.py.151: total_norm: 9.861293 [I 2022-10-27 15:52:56] ...trap/engines/engine.py.224: train: epoch 75 | batch 1390/2304 [I 2022-10-27 15:52:56] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:23 [I 2022-10-27 15:52:56] ...trap/engines/engine.py.229: process: 0.54792 | load: 0.00247 [I 2022-10-27 15:52:56] ...trap/engines/engine.py.230: loss: 0.10560 [I 2022-10-27 15:52:56] ...optimizers/trijoint.py.151: total_norm: 7.614270 [I 2022-10-27 15:53:01] ...trap/engines/engine.py.224: train: epoch 75 | batch 1400/2304 [I 2022-10-27 15:53:01] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:17 [I 2022-10-27 15:53:01] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00240 [I 2022-10-27 15:53:01] ...trap/engines/engine.py.230: loss: 0.11273 [I 2022-10-27 15:53:01] ...optimizers/trijoint.py.151: total_norm: 21.929626 [I 2022-10-27 15:53:07] ...trap/engines/engine.py.224: train: epoch 75 | batch 1410/2304 [I 2022-10-27 15:53:07] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:16 [I 2022-10-27 15:53:07] ...trap/engines/engine.py.229: process: 0.55832 | load: 0.00256 [I 2022-10-27 15:53:07] ...trap/engines/engine.py.230: loss: 0.11548 [I 2022-10-27 15:53:07] ...optimizers/trijoint.py.151: total_norm: 30.535528 [I 2022-10-27 15:53:12] ...trap/engines/engine.py.224: train: epoch 75 | batch 1420/2304 [I 2022-10-27 15:53:12] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:09 [I 2022-10-27 15:53:12] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00247 [I 2022-10-27 15:53:12] ...trap/engines/engine.py.230: loss: 0.14332 [I 2022-10-27 15:53:12] ...optimizers/trijoint.py.151: total_norm: 11.137966 [I 2022-10-27 15:53:18] ...trap/engines/engine.py.224: train: epoch 75 | batch 1430/2304 [I 2022-10-27 15:53:18] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:04 [I 2022-10-27 15:53:18] ...trap/engines/engine.py.229: process: 0.55269 | load: 0.00248 [I 2022-10-27 15:53:18] ...trap/engines/engine.py.230: loss: 0.13406 [I 2022-10-27 15:53:18] ...optimizers/trijoint.py.151: total_norm: 15.636894 [I 2022-10-27 15:53:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 1440/2304 [I 2022-10-27 15:53:23] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:07:58 [I 2022-10-27 15:53:23] ...trap/engines/engine.py.229: process: 0.54517 | load: 0.00248 [I 2022-10-27 15:53:23] ...trap/engines/engine.py.230: loss: 0.08845 [I 2022-10-27 15:53:23] ...optimizers/trijoint.py.151: total_norm: 17.552273 [I 2022-10-27 15:53:29] ...trap/engines/engine.py.224: train: epoch 75 | batch 1450/2304 [I 2022-10-27 15:53:29] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:52 [I 2022-10-27 15:53:29] ...trap/engines/engine.py.229: process: 0.54333 | load: 0.00246 [I 2022-10-27 15:53:29] ...trap/engines/engine.py.230: loss: 0.09945 [I 2022-10-27 15:53:29] ...optimizers/trijoint.py.151: total_norm: 7.640799 [I 2022-10-27 15:53:35] ...trap/engines/engine.py.224: train: epoch 75 | batch 1460/2304 [I 2022-10-27 15:53:35] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:45 [I 2022-10-27 15:53:35] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00254 [I 2022-10-27 15:53:35] ...trap/engines/engine.py.230: loss: 0.12869 [I 2022-10-27 15:53:35] ...optimizers/trijoint.py.151: total_norm: 19.632833 [I 2022-10-27 15:53:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 1470/2304 [I 2022-10-27 15:53:40] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:46 [I 2022-10-27 15:53:40] ...trap/engines/engine.py.229: process: 0.55507 | load: 0.00271 [I 2022-10-27 15:53:40] ...trap/engines/engine.py.230: loss: 0.12749 [I 2022-10-27 15:53:40] ...optimizers/trijoint.py.151: total_norm: 32.562099 [I 2022-10-27 15:53:46] ...trap/engines/engine.py.224: train: epoch 75 | batch 1480/2304 [I 2022-10-27 15:53:46] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:37 [I 2022-10-27 15:53:46] ...trap/engines/engine.py.229: process: 0.53978 | load: 0.00255 [I 2022-10-27 15:53:46] ...trap/engines/engine.py.230: loss: 0.11516 [I 2022-10-27 15:53:46] ...optimizers/trijoint.py.151: total_norm: 16.241838 [I 2022-10-27 15:53:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 1490/2304 [I 2022-10-27 15:53:51] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:31 [I 2022-10-27 15:53:51] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00249 [I 2022-10-27 15:53:51] ...trap/engines/engine.py.230: loss: 0.11034 [I 2022-10-27 15:53:51] ...optimizers/trijoint.py.151: total_norm: 24.757177 [I 2022-10-27 15:53:57] ...trap/engines/engine.py.224: train: epoch 75 | batch 1500/2304 [I 2022-10-27 15:53:57] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:25 [I 2022-10-27 15:53:57] ...trap/engines/engine.py.229: process: 0.55114 | load: 0.00250 [I 2022-10-27 15:53:57] ...trap/engines/engine.py.230: loss: 0.19500 [I 2022-10-27 15:53:57] ...optimizers/trijoint.py.151: total_norm: 18.771206 [I 2022-10-27 15:54:02] ...trap/engines/engine.py.224: train: epoch 75 | batch 1510/2304 [I 2022-10-27 15:54:02] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:22 [I 2022-10-27 15:54:02] ...trap/engines/engine.py.229: process: 0.56843 | load: 0.00247 [I 2022-10-27 15:54:02] ...trap/engines/engine.py.230: loss: 0.09553 [I 2022-10-27 15:54:02] ...optimizers/trijoint.py.151: total_norm: 12.329713 [I 2022-10-27 15:54:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 1520/2304 [I 2022-10-27 15:54:08] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:15 [I 2022-10-27 15:54:08] ...trap/engines/engine.py.229: process: 0.55154 | load: 0.00250 [I 2022-10-27 15:54:08] ...trap/engines/engine.py.230: loss: 0.13313 [I 2022-10-27 15:54:08] ...optimizers/trijoint.py.151: total_norm: 14.558444 [I 2022-10-27 15:54:14] ...trap/engines/engine.py.224: train: epoch 75 | batch 1530/2304 [I 2022-10-27 15:54:14] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:08 [I 2022-10-27 15:54:14] ...trap/engines/engine.py.229: process: 0.55029 | load: 0.00251 [I 2022-10-27 15:54:14] ...trap/engines/engine.py.230: loss: 0.18060 [I 2022-10-27 15:54:14] ...optimizers/trijoint.py.151: total_norm: 49.530048 [I 2022-10-27 15:54:19] ...trap/engines/engine.py.224: train: epoch 75 | batch 1540/2304 [I 2022-10-27 15:54:19] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:04 [I 2022-10-27 15:54:19] ...trap/engines/engine.py.229: process: 0.55526 | load: 0.00270 [I 2022-10-27 15:54:19] ...trap/engines/engine.py.230: loss: 0.14855 [I 2022-10-27 15:54:19] ...optimizers/trijoint.py.151: total_norm: 13.744546 [I 2022-10-27 15:54:25] ...trap/engines/engine.py.224: train: epoch 75 | batch 1550/2304 [I 2022-10-27 15:54:25] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:00 [I 2022-10-27 15:54:25] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00253 [I 2022-10-27 15:54:25] ...trap/engines/engine.py.230: loss: 0.14250 [I 2022-10-27 15:54:25] ...optimizers/trijoint.py.151: total_norm: 10.533840 [I 2022-10-27 15:54:30] ...trap/engines/engine.py.224: train: epoch 75 | batch 1560/2304 [I 2022-10-27 15:54:30] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:54 [I 2022-10-27 15:54:30] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00248 [I 2022-10-27 15:54:30] ...trap/engines/engine.py.230: loss: 0.12429 [I 2022-10-27 15:54:30] ...optimizers/trijoint.py.151: total_norm: 15.930275 [I 2022-10-27 15:54:36] ...trap/engines/engine.py.224: train: epoch 75 | batch 1570/2304 [I 2022-10-27 15:54:36] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:50 [I 2022-10-27 15:54:36] ...trap/engines/engine.py.229: process: 0.55993 | load: 0.00247 [I 2022-10-27 15:54:36] ...trap/engines/engine.py.230: loss: 0.11720 [I 2022-10-27 15:54:36] ...optimizers/trijoint.py.151: total_norm: 11.290055 [I 2022-10-27 15:54:41] ...trap/engines/engine.py.224: train: epoch 75 | batch 1580/2304 [I 2022-10-27 15:54:41] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:45 [I 2022-10-27 15:54:41] ...trap/engines/engine.py.229: process: 0.56502 | load: 0.00250 [I 2022-10-27 15:54:41] ...trap/engines/engine.py.230: loss: 0.18437 [I 2022-10-27 15:54:41] ...optimizers/trijoint.py.151: total_norm: 18.345043 [I 2022-10-27 15:54:47] ...trap/engines/engine.py.224: train: epoch 75 | batch 1590/2304 [I 2022-10-27 15:54:47] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:34 [I 2022-10-27 15:54:47] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00249 [I 2022-10-27 15:54:47] ...trap/engines/engine.py.230: loss: 0.17303 [I 2022-10-27 15:54:47] ...optimizers/trijoint.py.151: total_norm: 26.525625 [I 2022-10-27 15:54:53] ...trap/engines/engine.py.224: train: epoch 75 | batch 1600/2304 [I 2022-10-27 15:54:53] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:29 [I 2022-10-27 15:54:53] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00249 [I 2022-10-27 15:54:53] ...trap/engines/engine.py.230: loss: 0.14016 [I 2022-10-27 15:54:53] ...optimizers/trijoint.py.151: total_norm: 14.516627 [I 2022-10-27 15:54:58] ...trap/engines/engine.py.224: train: epoch 75 | batch 1610/2304 [I 2022-10-27 15:54:58] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:26 [I 2022-10-27 15:54:58] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00247 [I 2022-10-27 15:54:58] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-27 15:54:58] ...optimizers/trijoint.py.151: total_norm: 19.724762 [I 2022-10-27 15:55:04] ...trap/engines/engine.py.224: train: epoch 75 | batch 1620/2304 [I 2022-10-27 15:55:04] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:20 [I 2022-10-27 15:55:04] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00244 [I 2022-10-27 15:55:04] ...trap/engines/engine.py.230: loss: 0.10974 [I 2022-10-27 15:55:04] ...optimizers/trijoint.py.151: total_norm: 23.087729 [I 2022-10-27 15:55:09] ...trap/engines/engine.py.224: train: epoch 75 | batch 1630/2304 [I 2022-10-27 15:55:09] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:16 [I 2022-10-27 15:55:09] ...trap/engines/engine.py.229: process: 0.54456 | load: 0.00262 [I 2022-10-27 15:55:09] ...trap/engines/engine.py.230: loss: 0.18396 [I 2022-10-27 15:55:09] ...optimizers/trijoint.py.151: total_norm: 22.356123 [I 2022-10-27 15:55:15] ...trap/engines/engine.py.224: train: epoch 75 | batch 1640/2304 [I 2022-10-27 15:55:15] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:05 [I 2022-10-27 15:55:15] ...trap/engines/engine.py.229: process: 0.54114 | load: 0.00253 [I 2022-10-27 15:55:15] ...trap/engines/engine.py.230: loss: 0.26137 [I 2022-10-27 15:55:15] ...optimizers/trijoint.py.151: total_norm: 39.345623 [I 2022-10-27 15:55:20] ...trap/engines/engine.py.224: train: epoch 75 | batch 1650/2304 [I 2022-10-27 15:55:20] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:01 [I 2022-10-27 15:55:20] ...trap/engines/engine.py.229: process: 0.54824 | load: 0.00255 [I 2022-10-27 15:55:20] ...trap/engines/engine.py.230: loss: 0.13670 [I 2022-10-27 15:55:20] ...optimizers/trijoint.py.151: total_norm: 21.964098 [I 2022-10-27 15:55:26] ...trap/engines/engine.py.224: train: epoch 75 | batch 1660/2304 [I 2022-10-27 15:55:26] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:02 [I 2022-10-27 15:55:26] ...trap/engines/engine.py.229: process: 0.57246 | load: 0.00244 [I 2022-10-27 15:55:26] ...trap/engines/engine.py.230: loss: 0.12948 [I 2022-10-27 15:55:26] ...optimizers/trijoint.py.151: total_norm: 21.440842 [I 2022-10-27 15:55:32] ...trap/engines/engine.py.224: train: epoch 75 | batch 1670/2304 [I 2022-10-27 15:55:32] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:53 [I 2022-10-27 15:55:32] ...trap/engines/engine.py.229: process: 0.55115 | load: 0.00258 [I 2022-10-27 15:55:32] ...trap/engines/engine.py.230: loss: 0.10460 [I 2022-10-27 15:55:32] ...optimizers/trijoint.py.151: total_norm: 14.782098 [I 2022-10-27 15:55:37] ...trap/engines/engine.py.224: train: epoch 75 | batch 1680/2304 [I 2022-10-27 15:55:37] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:49 [I 2022-10-27 15:55:37] ...trap/engines/engine.py.229: process: 0.56192 | load: 0.00255 [I 2022-10-27 15:55:37] ...trap/engines/engine.py.230: loss: 0.14692 [I 2022-10-27 15:55:37] ...optimizers/trijoint.py.151: total_norm: 25.481703 [I 2022-10-27 15:55:43] ...trap/engines/engine.py.224: train: epoch 75 | batch 1690/2304 [I 2022-10-27 15:55:43] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:42 [I 2022-10-27 15:55:43] ...trap/engines/engine.py.229: process: 0.55001 | load: 0.00247 [I 2022-10-27 15:55:43] ...trap/engines/engine.py.230: loss: 0.18962 [I 2022-10-27 15:55:43] ...optimizers/trijoint.py.151: total_norm: 22.721138 [I 2022-10-27 15:55:49] ...trap/engines/engine.py.224: train: epoch 75 | batch 1700/2304 [I 2022-10-27 15:55:49] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:38 [I 2022-10-27 15:55:49] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00255 [I 2022-10-27 15:55:49] ...trap/engines/engine.py.230: loss: 0.12272 [I 2022-10-27 15:55:49] ...optimizers/trijoint.py.151: total_norm: 9.205269 [I 2022-10-27 15:55:54] ...trap/engines/engine.py.224: train: epoch 75 | batch 1710/2304 [I 2022-10-27 15:55:54] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:35 [I 2022-10-27 15:55:54] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00258 [I 2022-10-27 15:55:54] ...trap/engines/engine.py.230: loss: 0.11878 [I 2022-10-27 15:55:54] ...optimizers/trijoint.py.151: total_norm: 23.288519 [I 2022-10-27 15:56:00] ...trap/engines/engine.py.224: train: epoch 75 | batch 1720/2304 [I 2022-10-27 15:56:00] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:28 [I 2022-10-27 15:56:00] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00251 [I 2022-10-27 15:56:00] ...trap/engines/engine.py.230: loss: 0.12450 [I 2022-10-27 15:56:00] ...optimizers/trijoint.py.151: total_norm: 26.349457 [I 2022-10-27 15:56:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 1730/2304 [I 2022-10-27 15:56:05] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:21 [I 2022-10-27 15:56:05] ...trap/engines/engine.py.229: process: 0.56526 | load: 0.00254 [I 2022-10-27 15:56:05] ...trap/engines/engine.py.230: loss: 0.16657 [I 2022-10-27 15:56:05] ...optimizers/trijoint.py.151: total_norm: 28.889662 [I 2022-10-27 15:56:11] ...trap/engines/engine.py.224: train: epoch 75 | batch 1740/2304 [I 2022-10-27 15:56:11] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:17 [I 2022-10-27 15:56:11] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00251 [I 2022-10-27 15:56:11] ...trap/engines/engine.py.230: loss: 0.11556 [I 2022-10-27 15:56:11] ...optimizers/trijoint.py.151: total_norm: 17.823900 [I 2022-10-27 15:56:17] ...trap/engines/engine.py.224: train: epoch 75 | batch 1750/2304 [I 2022-10-27 15:56:17] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:18 [I 2022-10-27 15:56:17] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00251 [I 2022-10-27 15:56:17] ...trap/engines/engine.py.230: loss: 0.14302 [I 2022-10-27 15:56:17] ...optimizers/trijoint.py.151: total_norm: 49.551697 [I 2022-10-27 15:56:22] ...trap/engines/engine.py.224: train: epoch 75 | batch 1760/2304 [I 2022-10-27 15:56:22] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:05 [I 2022-10-27 15:56:22] ...trap/engines/engine.py.229: process: 0.56641 | load: 0.00248 [I 2022-10-27 15:56:22] ...trap/engines/engine.py.230: loss: 0.15611 [I 2022-10-27 15:56:22] ...optimizers/trijoint.py.151: total_norm: 24.021944 [I 2022-10-27 15:56:28] ...trap/engines/engine.py.224: train: epoch 75 | batch 1770/2304 [I 2022-10-27 15:56:28] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:00 [I 2022-10-27 15:56:28] ...trap/engines/engine.py.229: process: 0.56795 | load: 0.00244 [I 2022-10-27 15:56:28] ...trap/engines/engine.py.230: loss: 0.16439 [I 2022-10-27 15:56:28] ...optimizers/trijoint.py.151: total_norm: 17.130350 [I 2022-10-27 15:56:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 1780/2304 [I 2022-10-27 15:56:34] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:51 [I 2022-10-27 15:56:34] ...trap/engines/engine.py.229: process: 0.56885 | load: 0.00250 [I 2022-10-27 15:56:34] ...trap/engines/engine.py.230: loss: 0.15318 [I 2022-10-27 15:56:34] ...optimizers/trijoint.py.151: total_norm: 14.856153 [I 2022-10-27 15:56:39] ...trap/engines/engine.py.224: train: epoch 75 | batch 1790/2304 [I 2022-10-27 15:56:39] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:46 [I 2022-10-27 15:56:39] ...trap/engines/engine.py.229: process: 0.55900 | load: 0.00243 [I 2022-10-27 15:56:39] ...trap/engines/engine.py.230: loss: 0.11555 [I 2022-10-27 15:56:39] ...optimizers/trijoint.py.151: total_norm: 18.540392 [I 2022-10-27 15:56:45] ...trap/engines/engine.py.224: train: epoch 75 | batch 1800/2304 [I 2022-10-27 15:56:45] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:45 [I 2022-10-27 15:56:45] ...trap/engines/engine.py.229: process: 0.58384 | load: 0.00249 [I 2022-10-27 15:56:45] ...trap/engines/engine.py.230: loss: 0.22168 [I 2022-10-27 15:56:45] ...optimizers/trijoint.py.151: total_norm: 20.103140 [I 2022-10-27 15:56:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 1810/2304 [I 2022-10-27 15:56:51] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:36 [I 2022-10-27 15:56:51] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00248 [I 2022-10-27 15:56:51] ...trap/engines/engine.py.230: loss: 0.14416 [I 2022-10-27 15:56:51] ...optimizers/trijoint.py.151: total_norm: 80.225403 [I 2022-10-27 15:56:56] ...trap/engines/engine.py.224: train: epoch 75 | batch 1820/2304 [I 2022-10-27 15:56:56] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:32 [I 2022-10-27 15:56:56] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00254 [I 2022-10-27 15:56:56] ...trap/engines/engine.py.230: loss: 0.13215 [I 2022-10-27 15:56:56] ...optimizers/trijoint.py.151: total_norm: 6.149388 [I 2022-10-27 15:57:02] ...trap/engines/engine.py.224: train: epoch 75 | batch 1830/2304 [I 2022-10-27 15:57:02] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:24 [I 2022-10-27 15:57:02] ...trap/engines/engine.py.229: process: 0.56024 | load: 0.00246 [I 2022-10-27 15:57:02] ...trap/engines/engine.py.230: loss: 0.12237 [I 2022-10-27 15:57:02] ...optimizers/trijoint.py.151: total_norm: 14.556417 [I 2022-10-27 15:57:07] ...trap/engines/engine.py.224: train: epoch 75 | batch 1840/2304 [I 2022-10-27 15:57:07] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:18 [I 2022-10-27 15:57:07] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00245 [I 2022-10-27 15:57:07] ...trap/engines/engine.py.230: loss: 0.15279 [I 2022-10-27 15:57:07] ...optimizers/trijoint.py.151: total_norm: 16.571537 [I 2022-10-27 15:57:13] ...trap/engines/engine.py.224: train: epoch 75 | batch 1850/2304 [I 2022-10-27 15:57:13] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:14 [I 2022-10-27 15:57:13] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00246 [I 2022-10-27 15:57:13] ...trap/engines/engine.py.230: loss: 0.18480 [I 2022-10-27 15:57:13] ...optimizers/trijoint.py.151: total_norm: 15.661601 [I 2022-10-27 15:57:19] ...trap/engines/engine.py.224: train: epoch 75 | batch 1860/2304 [I 2022-10-27 15:57:19] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:06 [I 2022-10-27 15:57:19] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00245 [I 2022-10-27 15:57:19] ...trap/engines/engine.py.230: loss: 0.16234 [I 2022-10-27 15:57:19] ...optimizers/trijoint.py.151: total_norm: 31.027639 [I 2022-10-27 15:57:24] ...trap/engines/engine.py.224: train: epoch 75 | batch 1870/2304 [I 2022-10-27 15:57:24] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:00 [I 2022-10-27 15:57:24] ...trap/engines/engine.py.229: process: 0.55298 | load: 0.00258 [I 2022-10-27 15:57:24] ...trap/engines/engine.py.230: loss: 0.10911 [I 2022-10-27 15:57:24] ...optimizers/trijoint.py.151: total_norm: 14.384438 [I 2022-10-27 15:57:30] ...trap/engines/engine.py.224: train: epoch 75 | batch 1880/2304 [I 2022-10-27 15:57:30] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:56 [I 2022-10-27 15:57:30] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00257 [I 2022-10-27 15:57:30] ...trap/engines/engine.py.230: loss: 0.14208 [I 2022-10-27 15:57:30] ...optimizers/trijoint.py.151: total_norm: 10.311388 [I 2022-10-27 15:57:35] ...trap/engines/engine.py.224: train: epoch 75 | batch 1890/2304 [I 2022-10-27 15:57:35] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:03:54 [I 2022-10-27 15:57:35] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00253 [I 2022-10-27 15:57:35] ...trap/engines/engine.py.230: loss: 0.15267 [I 2022-10-27 15:57:35] ...optimizers/trijoint.py.151: total_norm: 27.278126 [I 2022-10-27 15:57:41] ...trap/engines/engine.py.224: train: epoch 75 | batch 1900/2304 [I 2022-10-27 15:57:41] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:47 [I 2022-10-27 15:57:41] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00259 [I 2022-10-27 15:57:41] ...trap/engines/engine.py.230: loss: 0.15556 [I 2022-10-27 15:57:41] ...optimizers/trijoint.py.151: total_norm: 30.202143 [I 2022-10-27 15:57:47] ...trap/engines/engine.py.224: train: epoch 75 | batch 1910/2304 [I 2022-10-27 15:57:47] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:39 [I 2022-10-27 15:57:47] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00258 [I 2022-10-27 15:57:47] ...trap/engines/engine.py.230: loss: 0.19044 [I 2022-10-27 15:57:47] ...optimizers/trijoint.py.151: total_norm: 16.622778 [I 2022-10-27 15:57:52] ...trap/engines/engine.py.224: train: epoch 75 | batch 1920/2304 [I 2022-10-27 15:57:52] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:34 [I 2022-10-27 15:57:52] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00253 [I 2022-10-27 15:57:52] ...trap/engines/engine.py.230: loss: 0.14056 [I 2022-10-27 15:57:52] ...optimizers/trijoint.py.151: total_norm: 10.115221 [I 2022-10-27 15:57:58] ...trap/engines/engine.py.224: train: epoch 75 | batch 1930/2304 [I 2022-10-27 15:57:58] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:29 [I 2022-10-27 15:57:58] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00249 [I 2022-10-27 15:57:58] ...trap/engines/engine.py.230: loss: 0.16083 [I 2022-10-27 15:57:58] ...optimizers/trijoint.py.151: total_norm: 12.250794 [I 2022-10-27 15:58:03] ...trap/engines/engine.py.224: train: epoch 75 | batch 1940/2304 [I 2022-10-27 15:58:03] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:23 [I 2022-10-27 15:58:03] ...trap/engines/engine.py.229: process: 0.55219 | load: 0.00252 [I 2022-10-27 15:58:03] ...trap/engines/engine.py.230: loss: 0.12744 [I 2022-10-27 15:58:03] ...optimizers/trijoint.py.151: total_norm: 18.330908 [I 2022-10-27 15:58:09] ...trap/engines/engine.py.224: train: epoch 75 | batch 1950/2304 [I 2022-10-27 15:58:09] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:17 [I 2022-10-27 15:58:09] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00250 [I 2022-10-27 15:58:09] ...trap/engines/engine.py.230: loss: 0.12716 [I 2022-10-27 15:58:09] ...optimizers/trijoint.py.151: total_norm: 19.098669 [I 2022-10-27 15:58:15] ...trap/engines/engine.py.224: train: epoch 75 | batch 1960/2304 [I 2022-10-27 15:58:15] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:12 [I 2022-10-27 15:58:15] ...trap/engines/engine.py.229: process: 0.55150 | load: 0.00249 [I 2022-10-27 15:58:15] ...trap/engines/engine.py.230: loss: 0.09308 [I 2022-10-27 15:58:15] ...optimizers/trijoint.py.151: total_norm: 19.374060 [I 2022-10-27 15:58:20] ...trap/engines/engine.py.224: train: epoch 75 | batch 1970/2304 [I 2022-10-27 15:58:20] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:06 [I 2022-10-27 15:58:20] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00249 [I 2022-10-27 15:58:20] ...trap/engines/engine.py.230: loss: 0.17771 [I 2022-10-27 15:58:20] ...optimizers/trijoint.py.151: total_norm: 17.934221 [I 2022-10-27 15:58:26] ...trap/engines/engine.py.224: train: epoch 75 | batch 1980/2304 [I 2022-10-27 15:58:26] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:00 [I 2022-10-27 15:58:26] ...trap/engines/engine.py.229: process: 0.56279 | load: 0.00247 [I 2022-10-27 15:58:26] ...trap/engines/engine.py.230: loss: 0.16643 [I 2022-10-27 15:58:26] ...optimizers/trijoint.py.151: total_norm: 19.202866 [I 2022-10-27 15:58:31] ...trap/engines/engine.py.224: train: epoch 75 | batch 1990/2304 [I 2022-10-27 15:58:31] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:02:55 [I 2022-10-27 15:58:31] ...trap/engines/engine.py.229: process: 0.55316 | load: 0.00244 [I 2022-10-27 15:58:31] ...trap/engines/engine.py.230: loss: 0.18837 [I 2022-10-27 15:58:31] ...optimizers/trijoint.py.151: total_norm: 15.546324 [I 2022-10-27 15:58:37] ...trap/engines/engine.py.224: train: epoch 75 | batch 2000/2304 [I 2022-10-27 15:58:37] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:02:48 [I 2022-10-27 15:58:37] ...trap/engines/engine.py.229: process: 0.54673 | load: 0.00243 [I 2022-10-27 15:58:37] ...trap/engines/engine.py.230: loss: 0.11826 [I 2022-10-27 15:58:37] ...optimizers/trijoint.py.151: total_norm: 13.963681 [I 2022-10-27 15:58:43] ...trap/engines/engine.py.224: train: epoch 75 | batch 2010/2304 [I 2022-10-27 15:58:43] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:02:46 [I 2022-10-27 15:58:43] ...trap/engines/engine.py.229: process: 0.56485 | load: 0.00253 [I 2022-10-27 15:58:43] ...trap/engines/engine.py.230: loss: 0.14133 [I 2022-10-27 15:58:43] ...optimizers/trijoint.py.151: total_norm: 14.725195 [I 2022-10-27 15:58:48] ...trap/engines/engine.py.224: train: epoch 75 | batch 2020/2304 [I 2022-10-27 15:58:48] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:38 [I 2022-10-27 15:58:48] ...trap/engines/engine.py.229: process: 0.54987 | load: 0.00254 [I 2022-10-27 15:58:48] ...trap/engines/engine.py.230: loss: 0.11123 [I 2022-10-27 15:58:48] ...optimizers/trijoint.py.151: total_norm: 20.972103 [I 2022-10-27 15:58:54] ...trap/engines/engine.py.224: train: epoch 75 | batch 2030/2304 [I 2022-10-27 15:58:54] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:32 [I 2022-10-27 15:58:54] ...trap/engines/engine.py.229: process: 0.56378 | load: 0.00252 [I 2022-10-27 15:58:54] ...trap/engines/engine.py.230: loss: 0.14708 [I 2022-10-27 15:58:54] ...optimizers/trijoint.py.151: total_norm: 19.913752 [I 2022-10-27 15:58:59] ...trap/engines/engine.py.224: train: epoch 75 | batch 2040/2304 [I 2022-10-27 15:58:59] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:26 [I 2022-10-27 15:58:59] ...trap/engines/engine.py.229: process: 0.55346 | load: 0.00243 [I 2022-10-27 15:58:59] ...trap/engines/engine.py.230: loss: 0.14608 [I 2022-10-27 15:58:59] ...optimizers/trijoint.py.151: total_norm: 15.163536 [I 2022-10-27 15:59:05] ...trap/engines/engine.py.224: train: epoch 75 | batch 2050/2304 [I 2022-10-27 15:59:05] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:22 [I 2022-10-27 15:59:05] ...trap/engines/engine.py.229: process: 0.56426 | load: 0.00255 [I 2022-10-27 15:59:05] ...trap/engines/engine.py.230: loss: 0.13024 [I 2022-10-27 15:59:05] ...optimizers/trijoint.py.151: total_norm: 19.540197 [I 2022-10-27 15:59:11] ...trap/engines/engine.py.224: train: epoch 75 | batch 2060/2304 [I 2022-10-27 15:59:11] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:16 [I 2022-10-27 15:59:11] ...trap/engines/engine.py.229: process: 0.56301 | load: 0.00271 [I 2022-10-27 15:59:11] ...trap/engines/engine.py.230: loss: 0.08384 [I 2022-10-27 15:59:11] ...optimizers/trijoint.py.151: total_norm: 9.147405 [I 2022-10-27 15:59:16] ...trap/engines/engine.py.224: train: epoch 75 | batch 2070/2304 [I 2022-10-27 15:59:16] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:10 [I 2022-10-27 15:59:16] ...trap/engines/engine.py.229: process: 0.56545 | load: 0.00255 [I 2022-10-27 15:59:16] ...trap/engines/engine.py.230: loss: 0.17699 [I 2022-10-27 15:59:16] ...optimizers/trijoint.py.151: total_norm: 11.646077 [I 2022-10-27 15:59:22] ...trap/engines/engine.py.224: train: epoch 75 | batch 2080/2304 [I 2022-10-27 15:59:22] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:04 [I 2022-10-27 15:59:22] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00252 [I 2022-10-27 15:59:22] ...trap/engines/engine.py.230: loss: 0.14349 [I 2022-10-27 15:59:22] ...optimizers/trijoint.py.151: total_norm: 28.650873 [I 2022-10-27 15:59:27] ...trap/engines/engine.py.224: train: epoch 75 | batch 2090/2304 [I 2022-10-27 15:59:27] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:01:57 [I 2022-10-27 15:59:27] ...trap/engines/engine.py.229: process: 0.55428 | load: 0.00249 [I 2022-10-27 15:59:27] ...trap/engines/engine.py.230: loss: 0.12292 [I 2022-10-27 15:59:27] ...optimizers/trijoint.py.151: total_norm: 23.150774 [I 2022-10-27 15:59:33] ...trap/engines/engine.py.224: train: epoch 75 | batch 2100/2304 [I 2022-10-27 15:59:33] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:52 [I 2022-10-27 15:59:33] ...trap/engines/engine.py.229: process: 0.54842 | load: 0.00247 [I 2022-10-27 15:59:33] ...trap/engines/engine.py.230: loss: 0.09929 [I 2022-10-27 15:59:33] ...optimizers/trijoint.py.151: total_norm: 28.454298 [I 2022-10-27 15:59:38] ...trap/engines/engine.py.224: train: epoch 75 | batch 2110/2304 [I 2022-10-27 15:59:38] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:01:47 [I 2022-10-27 15:59:38] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00257 [I 2022-10-27 15:59:38] ...trap/engines/engine.py.230: loss: 0.16639 [I 2022-10-27 15:59:38] ...optimizers/trijoint.py.151: total_norm: 12.116591 [I 2022-10-27 15:59:44] ...trap/engines/engine.py.224: train: epoch 75 | batch 2120/2304 [I 2022-10-27 15:59:44] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:44 [I 2022-10-27 15:59:44] ...trap/engines/engine.py.229: process: 0.54157 | load: 0.00269 [I 2022-10-27 15:59:44] ...trap/engines/engine.py.230: loss: 0.18419 [I 2022-10-27 15:59:44] ...optimizers/trijoint.py.151: total_norm: 29.618502 [I 2022-10-27 15:59:50] ...trap/engines/engine.py.224: train: epoch 75 | batch 2130/2304 [I 2022-10-27 15:59:50] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:37 [I 2022-10-27 15:59:50] ...trap/engines/engine.py.229: process: 0.54925 | load: 0.00256 [I 2022-10-27 15:59:50] ...trap/engines/engine.py.230: loss: 0.14816 [I 2022-10-27 15:59:50] ...optimizers/trijoint.py.151: total_norm: 10.248446 [I 2022-10-27 15:59:55] ...trap/engines/engine.py.224: train: epoch 75 | batch 2140/2304 [I 2022-10-27 15:59:55] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:30 [I 2022-10-27 15:59:55] ...trap/engines/engine.py.229: process: 0.54336 | load: 0.00247 [I 2022-10-27 15:59:55] ...trap/engines/engine.py.230: loss: 0.17521 [I 2022-10-27 15:59:55] ...optimizers/trijoint.py.151: total_norm: 12.366611 [I 2022-10-27 16:00:01] ...trap/engines/engine.py.224: train: epoch 75 | batch 2150/2304 [I 2022-10-27 16:00:01] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:25 [I 2022-10-27 16:00:01] ...trap/engines/engine.py.229: process: 0.55040 | load: 0.00250 [I 2022-10-27 16:00:01] ...trap/engines/engine.py.230: loss: 0.24939 [I 2022-10-27 16:00:01] ...optimizers/trijoint.py.151: total_norm: 20.332586 [I 2022-10-27 16:00:06] ...trap/engines/engine.py.224: train: epoch 75 | batch 2160/2304 [I 2022-10-27 16:00:06] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:19 [I 2022-10-27 16:00:06] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00250 [I 2022-10-27 16:00:06] ...trap/engines/engine.py.230: loss: 0.11357 [I 2022-10-27 16:00:06] ...optimizers/trijoint.py.151: total_norm: 12.934544 [I 2022-10-27 16:00:12] ...trap/engines/engine.py.224: train: epoch 75 | batch 2170/2304 [I 2022-10-27 16:00:12] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:14 [I 2022-10-27 16:00:12] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00241 [I 2022-10-27 16:00:12] ...trap/engines/engine.py.230: loss: 0.14194 [I 2022-10-27 16:00:12] ...optimizers/trijoint.py.151: total_norm: 10.287088 [I 2022-10-27 16:00:17] ...trap/engines/engine.py.224: train: epoch 75 | batch 2180/2304 [I 2022-10-27 16:00:17] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:08 [I 2022-10-27 16:00:17] ...trap/engines/engine.py.229: process: 0.54856 | load: 0.00244 [I 2022-10-27 16:00:17] ...trap/engines/engine.py.230: loss: 0.16367 [I 2022-10-27 16:00:17] ...optimizers/trijoint.py.151: total_norm: 24.338722 [I 2022-10-27 16:00:23] ...trap/engines/engine.py.224: train: epoch 75 | batch 2190/2304 [I 2022-10-27 16:00:23] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:03 [I 2022-10-27 16:00:23] ...trap/engines/engine.py.229: process: 0.55030 | load: 0.00246 [I 2022-10-27 16:00:23] ...trap/engines/engine.py.230: loss: 0.13436 [I 2022-10-27 16:00:23] ...optimizers/trijoint.py.151: total_norm: 36.083218 [I 2022-10-27 16:00:29] ...trap/engines/engine.py.224: train: epoch 75 | batch 2200/2304 [I 2022-10-27 16:00:29] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:57 [I 2022-10-27 16:00:29] ...trap/engines/engine.py.229: process: 0.55695 | load: 0.00242 [I 2022-10-27 16:00:29] ...trap/engines/engine.py.230: loss: 0.17628 [I 2022-10-27 16:00:29] ...optimizers/trijoint.py.151: total_norm: 13.992385 [I 2022-10-27 16:00:34] ...trap/engines/engine.py.224: train: epoch 75 | batch 2210/2304 [I 2022-10-27 16:00:34] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:00:52 [I 2022-10-27 16:00:34] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00248 [I 2022-10-27 16:00:34] ...trap/engines/engine.py.230: loss: 0.18425 [I 2022-10-27 16:00:34] ...optimizers/trijoint.py.151: total_norm: 16.076355 [I 2022-10-27 16:00:40] ...trap/engines/engine.py.224: train: epoch 75 | batch 2220/2304 [I 2022-10-27 16:00:40] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:00:46 [I 2022-10-27 16:00:40] ...trap/engines/engine.py.229: process: 0.56342 | load: 0.00240 [I 2022-10-27 16:00:40] ...trap/engines/engine.py.230: loss: 0.10910 [I 2022-10-27 16:00:40] ...optimizers/trijoint.py.151: total_norm: 8.828488 [I 2022-10-27 16:00:45] ...trap/engines/engine.py.224: train: epoch 75 | batch 2230/2304 [I 2022-10-27 16:00:45] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:41 [I 2022-10-27 16:00:45] ...trap/engines/engine.py.229: process: 0.56154 | load: 0.00242 [I 2022-10-27 16:00:45] ...trap/engines/engine.py.230: loss: 0.10433 [I 2022-10-27 16:00:45] ...optimizers/trijoint.py.151: total_norm: 20.402933 [I 2022-10-27 16:00:51] ...trap/engines/engine.py.224: train: epoch 75 | batch 2240/2304 [I 2022-10-27 16:00:51] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:35 [I 2022-10-27 16:00:51] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00246 [I 2022-10-27 16:00:51] ...trap/engines/engine.py.230: loss: 0.11297 [I 2022-10-27 16:00:51] ...optimizers/trijoint.py.151: total_norm: 13.460069 [I 2022-10-27 16:00:56] ...trap/engines/engine.py.224: train: epoch 75 | batch 2250/2304 [I 2022-10-27 16:00:56] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:30 [I 2022-10-27 16:00:56] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00245 [I 2022-10-27 16:00:56] ...trap/engines/engine.py.230: loss: 0.10174 [I 2022-10-27 16:00:56] ...optimizers/trijoint.py.151: total_norm: 9.161349 [I 2022-10-27 16:01:02] ...trap/engines/engine.py.224: train: epoch 75 | batch 2260/2304 [I 2022-10-27 16:01:02] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:24 [I 2022-10-27 16:01:02] ...trap/engines/engine.py.229: process: 0.54918 | load: 0.00246 [I 2022-10-27 16:01:02] ...trap/engines/engine.py.230: loss: 0.11196 [I 2022-10-27 16:01:02] ...optimizers/trijoint.py.151: total_norm: 19.845278 [I 2022-10-27 16:01:08] ...trap/engines/engine.py.224: train: epoch 75 | batch 2270/2304 [I 2022-10-27 16:01:08] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:19 [I 2022-10-27 16:01:08] ...trap/engines/engine.py.229: process: 0.56428 | load: 0.00250 [I 2022-10-27 16:01:08] ...trap/engines/engine.py.230: loss: 0.11534 [I 2022-10-27 16:01:08] ...optimizers/trijoint.py.151: total_norm: 22.034346 [I 2022-10-27 16:01:13] ...trap/engines/engine.py.224: train: epoch 75 | batch 2280/2304 [I 2022-10-27 16:01:13] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:13 [I 2022-10-27 16:01:13] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00245 [I 2022-10-27 16:01:13] ...trap/engines/engine.py.230: loss: 0.11446 [I 2022-10-27 16:01:13] ...optimizers/trijoint.py.151: total_norm: 13.953311 [I 2022-10-27 16:01:19] ...trap/engines/engine.py.224: train: epoch 75 | batch 2290/2304 [I 2022-10-27 16:01:19] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:07 [I 2022-10-27 16:01:19] ...trap/engines/engine.py.229: process: 0.56263 | load: 0.00250 [I 2022-10-27 16:01:19] ...trap/engines/engine.py.230: loss: 0.15124 [I 2022-10-27 16:01:19] ...optimizers/trijoint.py.151: total_norm: 18.905859 [I 2022-10-27 16:01:24] ...trap/engines/engine.py.224: train: epoch 75 | batch 2300/2304 [I 2022-10-27 16:01:24] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:02 [I 2022-10-27 16:01:24] ...trap/engines/engine.py.229: process: 0.54253 | load: 0.00016 [I 2022-10-27 16:01:24] ...trap/engines/engine.py.230: loss: 0.10358 [I 2022-10-27 16:01:24] ...optimizers/trijoint.py.151: total_norm: 12.639971 [I 2022-10-27 16:01:26] ...trap/engines/engine.py.224: train: epoch 75 | batch 2304/2304 [I 2022-10-27 16:01:26] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:00 [I 2022-10-27 16:01:26] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00016 [I 2022-10-27 16:01:26] ...trap/engines/engine.py.230: loss: 0.21064 [I 2022-10-27 16:01:26] ...optimizers/trijoint.py.151: total_norm: 26.022995 [S 2022-10-27 16:01:28] ...trap/engines/engine.py.129: train_epoch.epoch: 75 [S 2022-10-27 16:01:28] ...trap/engines/engine.py.129: train_epoch.loss: 0.14121842160659343 [I 2022-10-27 16:01:28] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 16:01:29] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 75 [I 2022-10-27 16:01:29] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 16:01:31] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 16:01:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 16:01:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 16:01:33] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 16:01:33] ...trap/engines/engine.py.321: eval: epoch 75 | batch 0/510 [I 2022-10-27 16:01:33] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:39:59 [I 2022-10-27 16:01:33] ...trap/engines/engine.py.326: process: 4.70415 | load: 2.58799 [I 2022-10-27 16:01:37] ...trap/engines/engine.py.321: eval: epoch 75 | batch 10/510 [I 2022-10-27 16:01:37] ...trap/engines/engine.py.325: elapsed: 0:00:07 | left: 0:06:39 [I 2022-10-27 16:01:37] ...trap/engines/engine.py.326: process: 0.32465 | load: 0.00017 [I 2022-10-27 16:01:40] ...trap/engines/engine.py.321: eval: epoch 75 | batch 20/510 [I 2022-10-27 16:01:40] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:06 [I 2022-10-27 16:01:40] ...trap/engines/engine.py.326: process: 0.32974 | load: 0.00019 [I 2022-10-27 16:01:43] ...trap/engines/engine.py.321: eval: epoch 75 | batch 30/510 [I 2022-10-27 16:01:43] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:46 [I 2022-10-27 16:01:43] ...trap/engines/engine.py.326: process: 0.35737 | load: 0.00023 [I 2022-10-27 16:01:47] ...trap/engines/engine.py.321: eval: epoch 75 | batch 40/510 [I 2022-10-27 16:01:47] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:38 [I 2022-10-27 16:01:47] ...trap/engines/engine.py.326: process: 0.32871 | load: 0.00031 [I 2022-10-27 16:01:50] ...trap/engines/engine.py.321: eval: epoch 75 | batch 50/510 [I 2022-10-27 16:01:50] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:35 [I 2022-10-27 16:01:50] ...trap/engines/engine.py.326: process: 0.32937 | load: 0.00021 [I 2022-10-27 16:01:54] ...trap/engines/engine.py.321: eval: epoch 75 | batch 60/510 [I 2022-10-27 16:01:54] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:37 [I 2022-10-27 16:01:54] ...trap/engines/engine.py.326: process: 0.40409 | load: 0.00018 [I 2022-10-27 16:01:57] ...trap/engines/engine.py.321: eval: epoch 75 | batch 70/510 [I 2022-10-27 16:01:57] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:27 [I 2022-10-27 16:01:57] ...trap/engines/engine.py.326: process: 0.34263 | load: 0.00017 [I 2022-10-27 16:02:00] ...trap/engines/engine.py.321: eval: epoch 75 | batch 80/510 [I 2022-10-27 16:02:00] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:25 [I 2022-10-27 16:02:00] ...trap/engines/engine.py.326: process: 0.34893 | load: 0.00034 [I 2022-10-27 16:02:04] ...trap/engines/engine.py.321: eval: epoch 75 | batch 90/510 [I 2022-10-27 16:02:04] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:21 [I 2022-10-27 16:02:04] ...trap/engines/engine.py.326: process: 0.35800 | load: 0.00018 [I 2022-10-27 16:02:07] ...trap/engines/engine.py.321: eval: epoch 75 | batch 100/510 [I 2022-10-27 16:02:07] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:12 [I 2022-10-27 16:02:07] ...trap/engines/engine.py.326: process: 0.28713 | load: 0.00025 [I 2022-10-27 16:02:10] ...trap/engines/engine.py.321: eval: epoch 75 | batch 110/510 [I 2022-10-27 16:02:10] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 16:02:10] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00016 [I 2022-10-27 16:02:13] ...trap/engines/engine.py.321: eval: epoch 75 | batch 120/510 [I 2022-10-27 16:02:13] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:53 [I 2022-10-27 16:02:13] ...trap/engines/engine.py.326: process: 0.29056 | load: 0.00021 [I 2022-10-27 16:02:16] ...trap/engines/engine.py.321: eval: epoch 75 | batch 130/510 [I 2022-10-27 16:02:16] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:51 [I 2022-10-27 16:02:16] ...trap/engines/engine.py.326: process: 0.31143 | load: 0.00014 [I 2022-10-27 16:02:19] ...trap/engines/engine.py.321: eval: epoch 75 | batch 140/510 [I 2022-10-27 16:02:19] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:47 [I 2022-10-27 16:02:19] ...trap/engines/engine.py.326: process: 0.29136 | load: 0.00015 [I 2022-10-27 16:02:21] ...trap/engines/engine.py.321: eval: epoch 75 | batch 150/510 [I 2022-10-27 16:02:21] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 16:02:21] ...trap/engines/engine.py.326: process: 0.28256 | load: 0.00017 [I 2022-10-27 16:02:24] ...trap/engines/engine.py.321: eval: epoch 75 | batch 160/510 [I 2022-10-27 16:02:24] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 16:02:24] ...trap/engines/engine.py.326: process: 0.29106 | load: 0.00015 [I 2022-10-27 16:02:27] ...trap/engines/engine.py.321: eval: epoch 75 | batch 170/510 [I 2022-10-27 16:02:27] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:37 [I 2022-10-27 16:02:27] ...trap/engines/engine.py.326: process: 0.28797 | load: 0.00015 [I 2022-10-27 16:02:30] ...trap/engines/engine.py.321: eval: epoch 75 | batch 180/510 [I 2022-10-27 16:02:30] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:34 [I 2022-10-27 16:02:30] ...trap/engines/engine.py.326: process: 0.28711 | load: 0.00019 [I 2022-10-27 16:02:33] ...trap/engines/engine.py.321: eval: epoch 75 | batch 190/510 [I 2022-10-27 16:02:33] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:31 [I 2022-10-27 16:02:33] ...trap/engines/engine.py.326: process: 0.28420 | load: 0.00015 [I 2022-10-27 16:02:36] ...trap/engines/engine.py.321: eval: epoch 75 | batch 200/510 [I 2022-10-27 16:02:36] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 16:02:36] ...trap/engines/engine.py.326: process: 0.29039 | load: 0.00017 [I 2022-10-27 16:02:39] ...trap/engines/engine.py.321: eval: epoch 75 | batch 210/510 [I 2022-10-27 16:02:39] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 16:02:39] ...trap/engines/engine.py.326: process: 0.29171 | load: 0.00015 [I 2022-10-27 16:02:42] ...trap/engines/engine.py.321: eval: epoch 75 | batch 220/510 [I 2022-10-27 16:02:42] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 16:02:42] ...trap/engines/engine.py.326: process: 0.29072 | load: 0.00017 [I 2022-10-27 16:02:44] ...trap/engines/engine.py.321: eval: epoch 75 | batch 230/510 [I 2022-10-27 16:02:44] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:20 [I 2022-10-27 16:02:44] ...trap/engines/engine.py.326: process: 0.28502 | load: 0.00019 [I 2022-10-27 16:02:47] ...trap/engines/engine.py.321: eval: epoch 75 | batch 240/510 [I 2022-10-27 16:02:47] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 16:02:47] ...trap/engines/engine.py.326: process: 0.29718 | load: 0.00016 [I 2022-10-27 16:02:50] ...trap/engines/engine.py.321: eval: epoch 75 | batch 250/510 [I 2022-10-27 16:02:50] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 16:02:50] ...trap/engines/engine.py.326: process: 0.27887 | load: 0.00027 [I 2022-10-27 16:02:53] ...trap/engines/engine.py.321: eval: epoch 75 | batch 260/510 [I 2022-10-27 16:02:53] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:11 [I 2022-10-27 16:02:53] ...trap/engines/engine.py.326: process: 0.28810 | load: 0.00016 [I 2022-10-27 16:02:56] ...trap/engines/engine.py.321: eval: epoch 75 | batch 270/510 [I 2022-10-27 16:02:56] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 16:02:56] ...trap/engines/engine.py.326: process: 0.28528 | load: 0.00022 [I 2022-10-27 16:02:59] ...trap/engines/engine.py.321: eval: epoch 75 | batch 280/510 [I 2022-10-27 16:02:59] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:07 [I 2022-10-27 16:02:59] ...trap/engines/engine.py.326: process: 0.29152 | load: 0.00018 [I 2022-10-27 16:03:02] ...trap/engines/engine.py.321: eval: epoch 75 | batch 290/510 [I 2022-10-27 16:03:02] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:04 [I 2022-10-27 16:03:02] ...trap/engines/engine.py.326: process: 0.28720 | load: 0.00021 [I 2022-10-27 16:03:05] ...trap/engines/engine.py.321: eval: epoch 75 | batch 300/510 [I 2022-10-27 16:03:05] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:01 [I 2022-10-27 16:03:05] ...trap/engines/engine.py.326: process: 0.30681 | load: 0.00015 [I 2022-10-27 16:03:08] ...trap/engines/engine.py.321: eval: epoch 75 | batch 310/510 [I 2022-10-27 16:03:08] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 16:03:08] ...trap/engines/engine.py.326: process: 0.29114 | load: 0.00016 [I 2022-10-27 16:03:10] ...trap/engines/engine.py.321: eval: epoch 75 | batch 320/510 [I 2022-10-27 16:03:10] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:55 [I 2022-10-27 16:03:10] ...trap/engines/engine.py.326: process: 0.30402 | load: 0.00015 [I 2022-10-27 16:03:13] ...trap/engines/engine.py.321: eval: epoch 75 | batch 330/510 [I 2022-10-27 16:03:13] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 16:03:13] ...trap/engines/engine.py.326: process: 0.28826 | load: 0.00015 [I 2022-10-27 16:03:16] ...trap/engines/engine.py.321: eval: epoch 75 | batch 340/510 [I 2022-10-27 16:03:16] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-27 16:03:16] ...trap/engines/engine.py.326: process: 0.29011 | load: 0.00014 [I 2022-10-27 16:03:19] ...trap/engines/engine.py.321: eval: epoch 75 | batch 350/510 [I 2022-10-27 16:03:19] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:45 [I 2022-10-27 16:03:19] ...trap/engines/engine.py.326: process: 0.28640 | load: 0.00016 [I 2022-10-27 16:03:22] ...trap/engines/engine.py.321: eval: epoch 75 | batch 360/510 [I 2022-10-27 16:03:22] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 16:03:22] ...trap/engines/engine.py.326: process: 0.28914 | load: 0.00016 [I 2022-10-27 16:03:25] ...trap/engines/engine.py.321: eval: epoch 75 | batch 370/510 [I 2022-10-27 16:03:25] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 16:03:25] ...trap/engines/engine.py.326: process: 0.28016 | load: 0.00022 [I 2022-10-27 16:03:28] ...trap/engines/engine.py.321: eval: epoch 75 | batch 380/510 [I 2022-10-27 16:03:28] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 16:03:28] ...trap/engines/engine.py.326: process: 0.28723 | load: 0.00018 [I 2022-10-27 16:03:31] ...trap/engines/engine.py.321: eval: epoch 75 | batch 390/510 [I 2022-10-27 16:03:31] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 16:03:31] ...trap/engines/engine.py.326: process: 0.28546 | load: 0.00022 [I 2022-10-27 16:03:34] ...trap/engines/engine.py.321: eval: epoch 75 | batch 400/510 [I 2022-10-27 16:03:34] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 16:03:34] ...trap/engines/engine.py.326: process: 0.29011 | load: 0.00020 [I 2022-10-27 16:03:36] ...trap/engines/engine.py.321: eval: epoch 75 | batch 410/510 [I 2022-10-27 16:03:36] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 16:03:36] ...trap/engines/engine.py.326: process: 0.28357 | load: 0.00019 [I 2022-10-27 16:03:39] ...trap/engines/engine.py.321: eval: epoch 75 | batch 420/510 [I 2022-10-27 16:03:39] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:26 [I 2022-10-27 16:03:39] ...trap/engines/engine.py.326: process: 0.29589 | load: 0.00015 [I 2022-10-27 16:03:42] ...trap/engines/engine.py.321: eval: epoch 75 | batch 430/510 [I 2022-10-27 16:03:42] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 16:03:42] ...trap/engines/engine.py.326: process: 0.28476 | load: 0.00018 [I 2022-10-27 16:03:45] ...trap/engines/engine.py.321: eval: epoch 75 | batch 440/510 [I 2022-10-27 16:03:45] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 16:03:45] ...trap/engines/engine.py.326: process: 0.28375 | load: 0.00015 [I 2022-10-27 16:03:48] ...trap/engines/engine.py.321: eval: epoch 75 | batch 450/510 [I 2022-10-27 16:03:48] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 16:03:48] ...trap/engines/engine.py.326: process: 0.28113 | load: 0.00022 [I 2022-10-27 16:03:51] ...trap/engines/engine.py.321: eval: epoch 75 | batch 460/510 [I 2022-10-27 16:03:51] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 16:03:51] ...trap/engines/engine.py.326: process: 0.28888 | load: 0.00017 [I 2022-10-27 16:03:54] ...trap/engines/engine.py.321: eval: epoch 75 | batch 470/510 [I 2022-10-27 16:03:54] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 16:03:54] ...trap/engines/engine.py.326: process: 0.28244 | load: 0.00020 [I 2022-10-27 16:03:57] ...trap/engines/engine.py.321: eval: epoch 75 | batch 480/510 [I 2022-10-27 16:03:57] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 16:03:57] ...trap/engines/engine.py.326: process: 0.28640 | load: 0.00018 [I 2022-10-27 16:03:59] ...trap/engines/engine.py.321: eval: epoch 75 | batch 490/510 [I 2022-10-27 16:03:59] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 16:03:59] ...trap/engines/engine.py.326: process: 0.28903 | load: 0.00019 [I 2022-10-27 16:04:02] ...trap/engines/engine.py.321: eval: epoch 75 | batch 500/510 [I 2022-10-27 16:04:02] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 16:04:02] ...trap/engines/engine.py.326: process: 0.30949 | load: 0.00016 [I 2022-10-27 16:04:05] ...trap/engines/engine.py.321: eval: epoch 75 | batch 510/510 [I 2022-10-27 16:04:05] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 16:04:05] ...trap/engines/engine.py.326: process: 0.28846 | load: 0.00017 [S 2022-10-27 16:04:06] ...trap/engines/engine.py.132: eval_epoch.epoch: 75 [I 2022-10-27 16:04:06] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.728400 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907400 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940400 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.718300 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905800 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937100 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010200 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006437 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004176 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010891 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007897 [S 2022-10-27 16:04:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006610 [I 2022-10-27 16:04:12] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 16:04:12] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 16:04:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 16:04:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 16:04:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 16:04:16] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 16:04:17] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 16:04:22] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 16:04:22] ...trap/engines/engine.py.163: Training model on trainset for epoch 76 [I 2022-10-27 16:04:23] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 16:04:25] ...trap/engines/engine.py.224: train: epoch 76 | batch 0/2304 [I 2022-10-27 16:04:25] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:16 [I 2022-10-27 16:04:25] ...trap/engines/engine.py.229: process: 2.68961 | load: 2.04053 [I 2022-10-27 16:04:25] ...trap/engines/engine.py.230: loss: 0.11189 [I 2022-10-27 16:04:25] ...optimizers/trijoint.py.151: total_norm: 16.997356 [I 2022-10-27 16:04:31] ...trap/engines/engine.py.224: train: epoch 76 | batch 10/2304 [I 2022-10-27 16:04:31] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:58 [I 2022-10-27 16:04:31] ...trap/engines/engine.py.229: process: 0.55271 | load: 0.00363 [I 2022-10-27 16:04:31] ...trap/engines/engine.py.230: loss: 0.25403 [I 2022-10-27 16:04:31] ...optimizers/trijoint.py.151: total_norm: 11.452413 [I 2022-10-27 16:04:36] ...trap/engines/engine.py.224: train: epoch 76 | batch 20/2304 [I 2022-10-27 16:04:36] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:12 [I 2022-10-27 16:04:36] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00395 [I 2022-10-27 16:04:36] ...trap/engines/engine.py.230: loss: 0.13001 [I 2022-10-27 16:04:36] ...optimizers/trijoint.py.151: total_norm: 22.337807 [I 2022-10-27 16:04:42] ...trap/engines/engine.py.224: train: epoch 76 | batch 30/2304 [I 2022-10-27 16:04:42] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:16 [I 2022-10-27 16:04:42] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00292 [I 2022-10-27 16:04:42] ...trap/engines/engine.py.230: loss: 0.11645 [I 2022-10-27 16:04:42] ...optimizers/trijoint.py.151: total_norm: 15.252386 [I 2022-10-27 16:04:48] ...trap/engines/engine.py.224: train: epoch 76 | batch 40/2304 [I 2022-10-27 16:04:48] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:05 [I 2022-10-27 16:04:48] ...trap/engines/engine.py.229: process: 0.56548 | load: 0.00283 [I 2022-10-27 16:04:48] ...trap/engines/engine.py.230: loss: 0.08923 [I 2022-10-27 16:04:48] ...optimizers/trijoint.py.151: total_norm: 10.266542 [I 2022-10-27 16:04:53] ...trap/engines/engine.py.224: train: epoch 76 | batch 50/2304 [I 2022-10-27 16:04:53] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:59 [I 2022-10-27 16:04:53] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00314 [I 2022-10-27 16:04:53] ...trap/engines/engine.py.230: loss: 0.13952 [I 2022-10-27 16:04:53] ...optimizers/trijoint.py.151: total_norm: 9.671803 [I 2022-10-27 16:05:19] ...trap/engines/engine.py.224: train: epoch 76 | batch 60/2304 [I 2022-10-27 16:05:19] ...trap/engines/engine.py.228: elapsed: 0:00:56 | left: 0:46:44 [I 2022-10-27 16:05:19] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00272 [I 2022-10-27 16:05:19] ...trap/engines/engine.py.230: loss: 0.09432 [I 2022-10-27 16:05:19] ...optimizers/trijoint.py.151: total_norm: 24.374466 [I 2022-10-27 16:05:25] ...trap/engines/engine.py.224: train: epoch 76 | batch 70/2304 [I 2022-10-27 16:05:25] ...trap/engines/engine.py.228: elapsed: 0:01:02 | left: 0:23:31 [I 2022-10-27 16:05:25] ...trap/engines/engine.py.229: process: 0.55829 | load: 0.00286 [I 2022-10-27 16:05:25] ...trap/engines/engine.py.230: loss: 0.15809 [I 2022-10-27 16:05:25] ...optimizers/trijoint.py.151: total_norm: 13.366530 [I 2022-10-27 16:05:30] ...trap/engines/engine.py.224: train: epoch 76 | batch 80/2304 [I 2022-10-27 16:05:30] ...trap/engines/engine.py.228: elapsed: 0:01:07 | left: 0:20:44 [I 2022-10-27 16:05:30] ...trap/engines/engine.py.229: process: 0.54487 | load: 0.00270 [I 2022-10-27 16:05:30] ...trap/engines/engine.py.230: loss: 0.13311 [I 2022-10-27 16:05:30] ...optimizers/trijoint.py.151: total_norm: 21.045828 [I 2022-10-27 16:05:36] ...trap/engines/engine.py.224: train: epoch 76 | batch 90/2304 [I 2022-10-27 16:05:36] ...trap/engines/engine.py.228: elapsed: 0:01:13 | left: 0:20:26 [I 2022-10-27 16:05:36] ...trap/engines/engine.py.229: process: 0.54478 | load: 0.00274 [I 2022-10-27 16:05:36] ...trap/engines/engine.py.230: loss: 0.15142 [I 2022-10-27 16:05:36] ...optimizers/trijoint.py.151: total_norm: 7.086596 [I 2022-10-27 16:05:42] ...trap/engines/engine.py.224: train: epoch 76 | batch 100/2304 [I 2022-10-27 16:05:42] ...trap/engines/engine.py.228: elapsed: 0:01:19 | left: 0:20:27 [I 2022-10-27 16:05:42] ...trap/engines/engine.py.229: process: 0.55076 | load: 0.00271 [I 2022-10-27 16:05:42] ...trap/engines/engine.py.230: loss: 0.12110 [I 2022-10-27 16:05:42] ...optimizers/trijoint.py.151: total_norm: 16.517370 [I 2022-10-27 16:05:47] ...trap/engines/engine.py.224: train: epoch 76 | batch 110/2304 [I 2022-10-27 16:05:47] ...trap/engines/engine.py.228: elapsed: 0:01:24 | left: 0:20:22 [I 2022-10-27 16:05:47] ...trap/engines/engine.py.229: process: 0.56380 | load: 0.00259 [I 2022-10-27 16:05:47] ...trap/engines/engine.py.230: loss: 0.18664 [I 2022-10-27 16:05:47] ...optimizers/trijoint.py.151: total_norm: 20.860910 [I 2022-10-27 16:05:53] ...trap/engines/engine.py.224: train: epoch 76 | batch 120/2304 [I 2022-10-27 16:05:53] ...trap/engines/engine.py.228: elapsed: 0:01:30 | left: 0:20:19 [I 2022-10-27 16:05:53] ...trap/engines/engine.py.229: process: 0.56037 | load: 0.00261 [I 2022-10-27 16:05:53] ...trap/engines/engine.py.230: loss: 0.16807 [I 2022-10-27 16:05:53] ...optimizers/trijoint.py.151: total_norm: 13.534680 [I 2022-10-27 16:05:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 130/2304 [I 2022-10-27 16:05:58] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:20:11 [I 2022-10-27 16:05:58] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00239 [I 2022-10-27 16:05:58] ...trap/engines/engine.py.230: loss: 0.12848 [I 2022-10-27 16:05:58] ...optimizers/trijoint.py.151: total_norm: 24.063578 [I 2022-10-27 16:06:04] ...trap/engines/engine.py.224: train: epoch 76 | batch 140/2304 [I 2022-10-27 16:06:04] ...trap/engines/engine.py.228: elapsed: 0:01:41 | left: 0:20:08 [I 2022-10-27 16:06:04] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00258 [I 2022-10-27 16:06:04] ...trap/engines/engine.py.230: loss: 0.14664 [I 2022-10-27 16:06:04] ...optimizers/trijoint.py.151: total_norm: 15.285470 [I 2022-10-27 16:06:10] ...trap/engines/engine.py.224: train: epoch 76 | batch 150/2304 [I 2022-10-27 16:06:10] ...trap/engines/engine.py.228: elapsed: 0:01:47 | left: 0:20:04 [I 2022-10-27 16:06:10] ...trap/engines/engine.py.229: process: 0.56626 | load: 0.00238 [I 2022-10-27 16:06:10] ...trap/engines/engine.py.230: loss: 0.10367 [I 2022-10-27 16:06:10] ...optimizers/trijoint.py.151: total_norm: 11.304385 [I 2022-10-27 16:06:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 160/2304 [I 2022-10-27 16:06:15] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:19:53 [I 2022-10-27 16:06:15] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00241 [I 2022-10-27 16:06:15] ...trap/engines/engine.py.230: loss: 0.11857 [I 2022-10-27 16:06:15] ...optimizers/trijoint.py.151: total_norm: 23.500402 [I 2022-10-27 16:06:21] ...trap/engines/engine.py.224: train: epoch 76 | batch 170/2304 [I 2022-10-27 16:06:21] ...trap/engines/engine.py.228: elapsed: 0:01:58 | left: 0:20:05 [I 2022-10-27 16:06:21] ...trap/engines/engine.py.229: process: 0.58966 | load: 0.00246 [I 2022-10-27 16:06:21] ...trap/engines/engine.py.230: loss: 0.12115 [I 2022-10-27 16:06:21] ...optimizers/trijoint.py.151: total_norm: 10.395295 [I 2022-10-27 16:06:26] ...trap/engines/engine.py.224: train: epoch 76 | batch 180/2304 [I 2022-10-27 16:06:26] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:19:51 [I 2022-10-27 16:06:26] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00257 [I 2022-10-27 16:06:26] ...trap/engines/engine.py.230: loss: 0.13300 [I 2022-10-27 16:06:26] ...optimizers/trijoint.py.151: total_norm: 11.312757 [I 2022-10-27 16:06:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 190/2304 [I 2022-10-27 16:06:32] ...trap/engines/engine.py.228: elapsed: 0:02:09 | left: 0:19:46 [I 2022-10-27 16:06:32] ...trap/engines/engine.py.229: process: 0.56951 | load: 0.00245 [I 2022-10-27 16:06:32] ...trap/engines/engine.py.230: loss: 0.15561 [I 2022-10-27 16:06:32] ...optimizers/trijoint.py.151: total_norm: 13.999709 [I 2022-10-27 16:06:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 200/2304 [I 2022-10-27 16:06:38] ...trap/engines/engine.py.228: elapsed: 0:02:15 | left: 0:19:35 [I 2022-10-27 16:06:38] ...trap/engines/engine.py.229: process: 0.55054 | load: 0.00276 [I 2022-10-27 16:06:38] ...trap/engines/engine.py.230: loss: 0.20256 [I 2022-10-27 16:06:38] ...optimizers/trijoint.py.151: total_norm: 21.874701 [I 2022-10-27 16:06:43] ...trap/engines/engine.py.224: train: epoch 76 | batch 210/2304 [I 2022-10-27 16:06:43] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:19:28 [I 2022-10-27 16:06:43] ...trap/engines/engine.py.229: process: 0.55638 | load: 0.00244 [I 2022-10-27 16:06:43] ...trap/engines/engine.py.230: loss: 0.10711 [I 2022-10-27 16:06:43] ...optimizers/trijoint.py.151: total_norm: 8.126124 [I 2022-10-27 16:06:49] ...trap/engines/engine.py.224: train: epoch 76 | batch 220/2304 [I 2022-10-27 16:06:49] ...trap/engines/engine.py.228: elapsed: 0:02:26 | left: 0:19:19 [I 2022-10-27 16:06:49] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00244 [I 2022-10-27 16:06:49] ...trap/engines/engine.py.230: loss: 0.12510 [I 2022-10-27 16:06:49] ...optimizers/trijoint.py.151: total_norm: 14.725193 [I 2022-10-27 16:06:54] ...trap/engines/engine.py.224: train: epoch 76 | batch 230/2304 [I 2022-10-27 16:06:54] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:19:11 [I 2022-10-27 16:06:54] ...trap/engines/engine.py.229: process: 0.54687 | load: 0.00252 [I 2022-10-27 16:06:54] ...trap/engines/engine.py.230: loss: 0.15450 [I 2022-10-27 16:06:54] ...optimizers/trijoint.py.151: total_norm: 14.253461 [I 2022-10-27 16:07:00] ...trap/engines/engine.py.224: train: epoch 76 | batch 240/2304 [I 2022-10-27 16:07:00] ...trap/engines/engine.py.228: elapsed: 0:02:37 | left: 0:19:13 [I 2022-10-27 16:07:00] ...trap/engines/engine.py.229: process: 0.55089 | load: 0.00242 [I 2022-10-27 16:07:00] ...trap/engines/engine.py.230: loss: 0.16396 [I 2022-10-27 16:07:00] ...optimizers/trijoint.py.151: total_norm: 10.370923 [I 2022-10-27 16:07:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 250/2304 [I 2022-10-27 16:07:06] ...trap/engines/engine.py.228: elapsed: 0:02:43 | left: 0:19:05 [I 2022-10-27 16:07:06] ...trap/engines/engine.py.229: process: 0.54958 | load: 0.00264 [I 2022-10-27 16:07:06] ...trap/engines/engine.py.230: loss: 0.11844 [I 2022-10-27 16:07:06] ...optimizers/trijoint.py.151: total_norm: 23.861282 [I 2022-10-27 16:07:11] ...trap/engines/engine.py.224: train: epoch 76 | batch 260/2304 [I 2022-10-27 16:07:11] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:19:45 [I 2022-10-27 16:07:11] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00260 [I 2022-10-27 16:07:11] ...trap/engines/engine.py.230: loss: 0.22980 [I 2022-10-27 16:07:11] ...optimizers/trijoint.py.151: total_norm: 24.407366 [I 2022-10-27 16:07:17] ...trap/engines/engine.py.224: train: epoch 76 | batch 270/2304 [I 2022-10-27 16:07:17] ...trap/engines/engine.py.228: elapsed: 0:02:54 | left: 0:18:59 [I 2022-10-27 16:07:17] ...trap/engines/engine.py.229: process: 0.55065 | load: 0.00256 [I 2022-10-27 16:07:17] ...trap/engines/engine.py.230: loss: 0.10332 [I 2022-10-27 16:07:17] ...optimizers/trijoint.py.151: total_norm: 17.295616 [I 2022-10-27 16:07:23] ...trap/engines/engine.py.224: train: epoch 76 | batch 280/2304 [I 2022-10-27 16:07:23] ...trap/engines/engine.py.228: elapsed: 0:03:00 | left: 0:18:56 [I 2022-10-27 16:07:23] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00258 [I 2022-10-27 16:07:23] ...trap/engines/engine.py.230: loss: 0.13384 [I 2022-10-27 16:07:23] ...optimizers/trijoint.py.151: total_norm: 12.570989 [I 2022-10-27 16:07:28] ...trap/engines/engine.py.224: train: epoch 76 | batch 290/2304 [I 2022-10-27 16:07:28] ...trap/engines/engine.py.228: elapsed: 0:03:05 | left: 0:18:38 [I 2022-10-27 16:07:28] ...trap/engines/engine.py.229: process: 0.56542 | load: 0.00246 [I 2022-10-27 16:07:28] ...trap/engines/engine.py.230: loss: 0.15862 [I 2022-10-27 16:07:28] ...optimizers/trijoint.py.151: total_norm: 26.565136 [I 2022-10-27 16:07:34] ...trap/engines/engine.py.224: train: epoch 76 | batch 300/2304 [I 2022-10-27 16:07:34] ...trap/engines/engine.py.228: elapsed: 0:03:11 | left: 0:18:32 [I 2022-10-27 16:07:34] ...trap/engines/engine.py.229: process: 0.56159 | load: 0.00244 [I 2022-10-27 16:07:34] ...trap/engines/engine.py.230: loss: 0.11867 [I 2022-10-27 16:07:34] ...optimizers/trijoint.py.151: total_norm: 7.837202 [I 2022-10-27 16:07:39] ...trap/engines/engine.py.224: train: epoch 76 | batch 310/2304 [I 2022-10-27 16:07:39] ...trap/engines/engine.py.228: elapsed: 0:03:16 | left: 0:18:28 [I 2022-10-27 16:07:39] ...trap/engines/engine.py.229: process: 0.54740 | load: 0.00244 [I 2022-10-27 16:07:39] ...trap/engines/engine.py.230: loss: 0.12879 [I 2022-10-27 16:07:39] ...optimizers/trijoint.py.151: total_norm: 16.759081 [I 2022-10-27 16:07:45] ...trap/engines/engine.py.224: train: epoch 76 | batch 320/2304 [I 2022-10-27 16:07:45] ...trap/engines/engine.py.228: elapsed: 0:03:22 | left: 0:18:25 [I 2022-10-27 16:07:45] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00246 [I 2022-10-27 16:07:45] ...trap/engines/engine.py.230: loss: 0.16630 [I 2022-10-27 16:07:45] ...optimizers/trijoint.py.151: total_norm: 14.876730 [I 2022-10-27 16:07:51] ...trap/engines/engine.py.224: train: epoch 76 | batch 330/2304 [I 2022-10-27 16:07:51] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:18:22 [I 2022-10-27 16:07:51] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00246 [I 2022-10-27 16:07:51] ...trap/engines/engine.py.230: loss: 0.13317 [I 2022-10-27 16:07:51] ...optimizers/trijoint.py.151: total_norm: 15.753235 [I 2022-10-27 16:07:56] ...trap/engines/engine.py.224: train: epoch 76 | batch 340/2304 [I 2022-10-27 16:07:56] ...trap/engines/engine.py.228: elapsed: 0:03:33 | left: 0:18:11 [I 2022-10-27 16:07:56] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00252 [I 2022-10-27 16:07:56] ...trap/engines/engine.py.230: loss: 0.13252 [I 2022-10-27 16:07:56] ...optimizers/trijoint.py.151: total_norm: 14.791718 [I 2022-10-27 16:08:02] ...trap/engines/engine.py.224: train: epoch 76 | batch 350/2304 [I 2022-10-27 16:08:02] ...trap/engines/engine.py.228: elapsed: 0:03:39 | left: 0:18:08 [I 2022-10-27 16:08:02] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00245 [I 2022-10-27 16:08:02] ...trap/engines/engine.py.230: loss: 0.09031 [I 2022-10-27 16:08:02] ...optimizers/trijoint.py.151: total_norm: 10.125047 [I 2022-10-27 16:08:07] ...trap/engines/engine.py.224: train: epoch 76 | batch 360/2304 [I 2022-10-27 16:08:07] ...trap/engines/engine.py.228: elapsed: 0:03:44 | left: 0:18:07 [I 2022-10-27 16:08:07] ...trap/engines/engine.py.229: process: 0.56005 | load: 0.00244 [I 2022-10-27 16:08:07] ...trap/engines/engine.py.230: loss: 0.13492 [I 2022-10-27 16:08:07] ...optimizers/trijoint.py.151: total_norm: 13.987697 [I 2022-10-27 16:08:13] ...trap/engines/engine.py.224: train: epoch 76 | batch 370/2304 [I 2022-10-27 16:08:13] ...trap/engines/engine.py.228: elapsed: 0:03:50 | left: 0:17:50 [I 2022-10-27 16:08:13] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00244 [I 2022-10-27 16:08:13] ...trap/engines/engine.py.230: loss: 0.13401 [I 2022-10-27 16:08:13] ...optimizers/trijoint.py.151: total_norm: 23.062084 [I 2022-10-27 16:08:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 380/2304 [I 2022-10-27 16:08:18] ...trap/engines/engine.py.228: elapsed: 0:03:55 | left: 0:17:52 [I 2022-10-27 16:08:18] ...trap/engines/engine.py.229: process: 0.55359 | load: 0.00255 [I 2022-10-27 16:08:18] ...trap/engines/engine.py.230: loss: 0.16474 [I 2022-10-27 16:08:18] ...optimizers/trijoint.py.151: total_norm: 13.573168 [I 2022-10-27 16:08:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 390/2304 [I 2022-10-27 16:08:24] ...trap/engines/engine.py.228: elapsed: 0:04:01 | left: 0:17:45 [I 2022-10-27 16:08:24] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00248 [I 2022-10-27 16:08:24] ...trap/engines/engine.py.230: loss: 0.12715 [I 2022-10-27 16:08:24] ...optimizers/trijoint.py.151: total_norm: 12.444260 [I 2022-10-27 16:08:30] ...trap/engines/engine.py.224: train: epoch 76 | batch 400/2304 [I 2022-10-27 16:08:30] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:17:34 [I 2022-10-27 16:08:30] ...trap/engines/engine.py.229: process: 0.54803 | load: 0.00248 [I 2022-10-27 16:08:30] ...trap/engines/engine.py.230: loss: 0.11791 [I 2022-10-27 16:08:30] ...optimizers/trijoint.py.151: total_norm: 16.191460 [I 2022-10-27 16:08:35] ...trap/engines/engine.py.224: train: epoch 76 | batch 410/2304 [I 2022-10-27 16:08:35] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:17:34 [I 2022-10-27 16:08:35] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00243 [I 2022-10-27 16:08:35] ...trap/engines/engine.py.230: loss: 0.14533 [I 2022-10-27 16:08:35] ...optimizers/trijoint.py.151: total_norm: 30.046658 [I 2022-10-27 16:08:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 420/2304 [I 2022-10-27 16:08:41] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:17:21 [I 2022-10-27 16:08:41] ...trap/engines/engine.py.229: process: 0.54931 | load: 0.00243 [I 2022-10-27 16:08:41] ...trap/engines/engine.py.230: loss: 0.12794 [I 2022-10-27 16:08:41] ...optimizers/trijoint.py.151: total_norm: 13.683662 [I 2022-10-27 16:08:46] ...trap/engines/engine.py.224: train: epoch 76 | batch 430/2304 [I 2022-10-27 16:08:46] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:17:45 [I 2022-10-27 16:08:46] ...trap/engines/engine.py.229: process: 0.62619 | load: 0.00243 [I 2022-10-27 16:08:46] ...trap/engines/engine.py.230: loss: 0.13336 [I 2022-10-27 16:08:46] ...optimizers/trijoint.py.151: total_norm: 12.643097 [I 2022-10-27 16:08:52] ...trap/engines/engine.py.224: train: epoch 76 | batch 440/2304 [I 2022-10-27 16:08:52] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:17:21 [I 2022-10-27 16:08:52] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00242 [I 2022-10-27 16:08:52] ...trap/engines/engine.py.230: loss: 0.12668 [I 2022-10-27 16:08:52] ...optimizers/trijoint.py.151: total_norm: 22.642466 [I 2022-10-27 16:08:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 450/2304 [I 2022-10-27 16:08:58] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:17:10 [I 2022-10-27 16:08:58] ...trap/engines/engine.py.229: process: 0.55336 | load: 0.00242 [I 2022-10-27 16:08:58] ...trap/engines/engine.py.230: loss: 0.13191 [I 2022-10-27 16:08:58] ...optimizers/trijoint.py.151: total_norm: 12.411911 [I 2022-10-27 16:09:03] ...trap/engines/engine.py.224: train: epoch 76 | batch 460/2304 [I 2022-10-27 16:09:03] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:17:01 [I 2022-10-27 16:09:03] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00256 [I 2022-10-27 16:09:03] ...trap/engines/engine.py.230: loss: 0.11381 [I 2022-10-27 16:09:03] ...optimizers/trijoint.py.151: total_norm: 11.917826 [I 2022-10-27 16:09:09] ...trap/engines/engine.py.224: train: epoch 76 | batch 470/2304 [I 2022-10-27 16:09:09] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:17:08 [I 2022-10-27 16:09:09] ...trap/engines/engine.py.229: process: 0.57219 | load: 0.00246 [I 2022-10-27 16:09:09] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-27 16:09:09] ...optimizers/trijoint.py.151: total_norm: 18.939837 [I 2022-10-27 16:09:14] ...trap/engines/engine.py.224: train: epoch 76 | batch 480/2304 [I 2022-10-27 16:09:14] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:17:01 [I 2022-10-27 16:09:14] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00242 [I 2022-10-27 16:09:14] ...trap/engines/engine.py.230: loss: 0.10322 [I 2022-10-27 16:09:14] ...optimizers/trijoint.py.151: total_norm: 9.369406 [I 2022-10-27 16:09:20] ...trap/engines/engine.py.224: train: epoch 76 | batch 490/2304 [I 2022-10-27 16:09:20] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:16:54 [I 2022-10-27 16:09:20] ...trap/engines/engine.py.229: process: 0.55676 | load: 0.00251 [I 2022-10-27 16:09:20] ...trap/engines/engine.py.230: loss: 0.10871 [I 2022-10-27 16:09:20] ...optimizers/trijoint.py.151: total_norm: 18.248627 [I 2022-10-27 16:09:26] ...trap/engines/engine.py.224: train: epoch 76 | batch 500/2304 [I 2022-10-27 16:09:26] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:16:42 [I 2022-10-27 16:09:26] ...trap/engines/engine.py.229: process: 0.56073 | load: 0.00251 [I 2022-10-27 16:09:26] ...trap/engines/engine.py.230: loss: 0.12320 [I 2022-10-27 16:09:26] ...optimizers/trijoint.py.151: total_norm: 10.761016 [I 2022-10-27 16:09:31] ...trap/engines/engine.py.224: train: epoch 76 | batch 510/2304 [I 2022-10-27 16:09:31] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:16:45 [I 2022-10-27 16:09:31] ...trap/engines/engine.py.229: process: 0.56857 | load: 0.00246 [I 2022-10-27 16:09:31] ...trap/engines/engine.py.230: loss: 0.10504 [I 2022-10-27 16:09:31] ...optimizers/trijoint.py.151: total_norm: 16.096556 [I 2022-10-27 16:09:37] ...trap/engines/engine.py.224: train: epoch 76 | batch 520/2304 [I 2022-10-27 16:09:37] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:16:34 [I 2022-10-27 16:09:37] ...trap/engines/engine.py.229: process: 0.56582 | load: 0.00244 [I 2022-10-27 16:09:37] ...trap/engines/engine.py.230: loss: 0.10573 [I 2022-10-27 16:09:37] ...optimizers/trijoint.py.151: total_norm: 8.757809 [I 2022-10-27 16:09:42] ...trap/engines/engine.py.224: train: epoch 76 | batch 530/2304 [I 2022-10-27 16:09:42] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:16:27 [I 2022-10-27 16:09:42] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00245 [I 2022-10-27 16:09:42] ...trap/engines/engine.py.230: loss: 0.10292 [I 2022-10-27 16:09:42] ...optimizers/trijoint.py.151: total_norm: 21.239910 [I 2022-10-27 16:09:48] ...trap/engines/engine.py.224: train: epoch 76 | batch 540/2304 [I 2022-10-27 16:09:48] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:17:05 [I 2022-10-27 16:09:48] ...trap/engines/engine.py.229: process: 0.67454 | load: 0.00247 [I 2022-10-27 16:09:48] ...trap/engines/engine.py.230: loss: 0.10957 [I 2022-10-27 16:09:48] ...optimizers/trijoint.py.151: total_norm: 9.435625 [I 2022-10-27 16:09:54] ...trap/engines/engine.py.224: train: epoch 76 | batch 550/2304 [I 2022-10-27 16:09:54] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:16:19 [I 2022-10-27 16:09:54] ...trap/engines/engine.py.229: process: 0.54955 | load: 0.00242 [I 2022-10-27 16:09:54] ...trap/engines/engine.py.230: loss: 0.17233 [I 2022-10-27 16:09:54] ...optimizers/trijoint.py.151: total_norm: 14.290763 [I 2022-10-27 16:09:59] ...trap/engines/engine.py.224: train: epoch 76 | batch 560/2304 [I 2022-10-27 16:09:59] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:16:18 [I 2022-10-27 16:09:59] ...trap/engines/engine.py.229: process: 0.56268 | load: 0.00272 [I 2022-10-27 16:09:59] ...trap/engines/engine.py.230: loss: 0.11489 [I 2022-10-27 16:09:59] ...optimizers/trijoint.py.151: total_norm: 10.230008 [I 2022-10-27 16:10:05] ...trap/engines/engine.py.224: train: epoch 76 | batch 570/2304 [I 2022-10-27 16:10:05] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:16:08 [I 2022-10-27 16:10:05] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00252 [I 2022-10-27 16:10:05] ...trap/engines/engine.py.230: loss: 0.15095 [I 2022-10-27 16:10:05] ...optimizers/trijoint.py.151: total_norm: 13.823953 [I 2022-10-27 16:10:10] ...trap/engines/engine.py.224: train: epoch 76 | batch 580/2304 [I 2022-10-27 16:10:10] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:16:01 [I 2022-10-27 16:10:10] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00253 [I 2022-10-27 16:10:10] ...trap/engines/engine.py.230: loss: 0.09686 [I 2022-10-27 16:10:10] ...optimizers/trijoint.py.151: total_norm: 11.352834 [I 2022-10-27 16:10:16] ...trap/engines/engine.py.224: train: epoch 76 | batch 590/2304 [I 2022-10-27 16:10:16] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:15:56 [I 2022-10-27 16:10:16] ...trap/engines/engine.py.229: process: 0.54840 | load: 0.00246 [I 2022-10-27 16:10:16] ...trap/engines/engine.py.230: loss: 0.10464 [I 2022-10-27 16:10:16] ...optimizers/trijoint.py.151: total_norm: 11.244932 [I 2022-10-27 16:10:22] ...trap/engines/engine.py.224: train: epoch 76 | batch 600/2304 [I 2022-10-27 16:10:22] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:57 [I 2022-10-27 16:10:22] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00249 [I 2022-10-27 16:10:22] ...trap/engines/engine.py.230: loss: 0.09945 [I 2022-10-27 16:10:22] ...optimizers/trijoint.py.151: total_norm: 21.415390 [I 2022-10-27 16:10:27] ...trap/engines/engine.py.224: train: epoch 76 | batch 610/2304 [I 2022-10-27 16:10:27] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:44 [I 2022-10-27 16:10:27] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00246 [I 2022-10-27 16:10:27] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-27 16:10:27] ...optimizers/trijoint.py.151: total_norm: 10.943332 [I 2022-10-27 16:10:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 620/2304 [I 2022-10-27 16:10:33] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:15:37 [I 2022-10-27 16:10:33] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00243 [I 2022-10-27 16:10:33] ...trap/engines/engine.py.230: loss: 0.12241 [I 2022-10-27 16:10:33] ...optimizers/trijoint.py.151: total_norm: 22.100794 [I 2022-10-27 16:10:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 630/2304 [I 2022-10-27 16:10:38] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:32 [I 2022-10-27 16:10:38] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00248 [I 2022-10-27 16:10:38] ...trap/engines/engine.py.230: loss: 0.13214 [I 2022-10-27 16:10:38] ...optimizers/trijoint.py.151: total_norm: 11.023672 [I 2022-10-27 16:10:44] ...trap/engines/engine.py.224: train: epoch 76 | batch 640/2304 [I 2022-10-27 16:10:44] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:15:26 [I 2022-10-27 16:10:44] ...trap/engines/engine.py.229: process: 0.55351 | load: 0.00248 [I 2022-10-27 16:10:44] ...trap/engines/engine.py.230: loss: 0.16371 [I 2022-10-27 16:10:44] ...optimizers/trijoint.py.151: total_norm: 12.845931 [I 2022-10-27 16:10:50] ...trap/engines/engine.py.224: train: epoch 76 | batch 650/2304 [I 2022-10-27 16:10:50] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:15:22 [I 2022-10-27 16:10:50] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00243 [I 2022-10-27 16:10:50] ...trap/engines/engine.py.230: loss: 0.11061 [I 2022-10-27 16:10:50] ...optimizers/trijoint.py.151: total_norm: 8.095437 [I 2022-10-27 16:10:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 660/2304 [I 2022-10-27 16:10:55] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:15:42 [I 2022-10-27 16:10:55] ...trap/engines/engine.py.229: process: 0.57348 | load: 0.00252 [I 2022-10-27 16:10:55] ...trap/engines/engine.py.230: loss: 0.26612 [I 2022-10-27 16:10:55] ...optimizers/trijoint.py.151: total_norm: 20.452858 [I 2022-10-27 16:11:01] ...trap/engines/engine.py.224: train: epoch 76 | batch 670/2304 [I 2022-10-27 16:11:01] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:15:06 [I 2022-10-27 16:11:01] ...trap/engines/engine.py.229: process: 0.55639 | load: 0.00242 [I 2022-10-27 16:11:01] ...trap/engines/engine.py.230: loss: 0.14757 [I 2022-10-27 16:11:01] ...optimizers/trijoint.py.151: total_norm: 16.466303 [I 2022-10-27 16:11:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 680/2304 [I 2022-10-27 16:11:06] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:04 [I 2022-10-27 16:11:06] ...trap/engines/engine.py.229: process: 0.55121 | load: 0.00240 [I 2022-10-27 16:11:06] ...trap/engines/engine.py.230: loss: 0.12086 [I 2022-10-27 16:11:06] ...optimizers/trijoint.py.151: total_norm: 20.917868 [I 2022-10-27 16:11:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 690/2304 [I 2022-10-27 16:11:12] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:14:59 [I 2022-10-27 16:11:12] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00251 [I 2022-10-27 16:11:12] ...trap/engines/engine.py.230: loss: 0.20552 [I 2022-10-27 16:11:12] ...optimizers/trijoint.py.151: total_norm: 23.984011 [I 2022-10-27 16:11:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 700/2304 [I 2022-10-27 16:11:18] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:53 [I 2022-10-27 16:11:18] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00242 [I 2022-10-27 16:11:18] ...trap/engines/engine.py.230: loss: 0.09192 [I 2022-10-27 16:11:18] ...optimizers/trijoint.py.151: total_norm: 12.154128 [I 2022-10-27 16:11:23] ...trap/engines/engine.py.224: train: epoch 76 | batch 710/2304 [I 2022-10-27 16:11:23] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:45 [I 2022-10-27 16:11:23] ...trap/engines/engine.py.229: process: 0.55524 | load: 0.00251 [I 2022-10-27 16:11:23] ...trap/engines/engine.py.230: loss: 0.10898 [I 2022-10-27 16:11:23] ...optimizers/trijoint.py.151: total_norm: 10.707448 [I 2022-10-27 16:11:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 720/2304 [I 2022-10-27 16:11:29] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:42 [I 2022-10-27 16:11:29] ...trap/engines/engine.py.229: process: 0.54299 | load: 0.00251 [I 2022-10-27 16:11:29] ...trap/engines/engine.py.230: loss: 0.12379 [I 2022-10-27 16:11:29] ...optimizers/trijoint.py.151: total_norm: 26.244394 [I 2022-10-27 16:11:34] ...trap/engines/engine.py.224: train: epoch 76 | batch 730/2304 [I 2022-10-27 16:11:34] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:32 [I 2022-10-27 16:11:34] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00247 [I 2022-10-27 16:11:34] ...trap/engines/engine.py.230: loss: 0.12709 [I 2022-10-27 16:11:34] ...optimizers/trijoint.py.151: total_norm: 27.505629 [I 2022-10-27 16:11:40] ...trap/engines/engine.py.224: train: epoch 76 | batch 740/2304 [I 2022-10-27 16:11:40] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:14:30 [I 2022-10-27 16:11:40] ...trap/engines/engine.py.229: process: 0.54794 | load: 0.00249 [I 2022-10-27 16:11:40] ...trap/engines/engine.py.230: loss: 0.10455 [I 2022-10-27 16:11:40] ...optimizers/trijoint.py.151: total_norm: 15.710719 [I 2022-10-27 16:11:45] ...trap/engines/engine.py.224: train: epoch 76 | batch 750/2304 [I 2022-10-27 16:11:45] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:20 [I 2022-10-27 16:11:45] ...trap/engines/engine.py.229: process: 0.54597 | load: 0.00245 [I 2022-10-27 16:11:45] ...trap/engines/engine.py.230: loss: 0.13215 [I 2022-10-27 16:11:45] ...optimizers/trijoint.py.151: total_norm: 17.811089 [I 2022-10-27 16:11:51] ...trap/engines/engine.py.224: train: epoch 76 | batch 760/2304 [I 2022-10-27 16:11:51] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:21 [I 2022-10-27 16:11:51] ...trap/engines/engine.py.229: process: 0.55513 | load: 0.00243 [I 2022-10-27 16:11:51] ...trap/engines/engine.py.230: loss: 0.11050 [I 2022-10-27 16:11:51] ...optimizers/trijoint.py.151: total_norm: 11.224613 [I 2022-10-27 16:11:57] ...trap/engines/engine.py.224: train: epoch 76 | batch 770/2304 [I 2022-10-27 16:11:57] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:14:17 [I 2022-10-27 16:11:57] ...trap/engines/engine.py.229: process: 0.56229 | load: 0.00247 [I 2022-10-27 16:11:57] ...trap/engines/engine.py.230: loss: 0.10423 [I 2022-10-27 16:11:57] ...optimizers/trijoint.py.151: total_norm: 9.135878 [I 2022-10-27 16:12:02] ...trap/engines/engine.py.224: train: epoch 76 | batch 780/2304 [I 2022-10-27 16:12:02] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:15 [I 2022-10-27 16:12:02] ...trap/engines/engine.py.229: process: 0.57118 | load: 0.00247 [I 2022-10-27 16:12:02] ...trap/engines/engine.py.230: loss: 0.19971 [I 2022-10-27 16:12:02] ...optimizers/trijoint.py.151: total_norm: 34.668324 [I 2022-10-27 16:12:08] ...trap/engines/engine.py.224: train: epoch 76 | batch 790/2304 [I 2022-10-27 16:12:08] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:14:02 [I 2022-10-27 16:12:08] ...trap/engines/engine.py.229: process: 0.55248 | load: 0.00260 [I 2022-10-27 16:12:08] ...trap/engines/engine.py.230: loss: 0.14986 [I 2022-10-27 16:12:08] ...optimizers/trijoint.py.151: total_norm: 13.774879 [I 2022-10-27 16:12:13] ...trap/engines/engine.py.224: train: epoch 76 | batch 800/2304 [I 2022-10-27 16:12:13] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:13:58 [I 2022-10-27 16:12:13] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00256 [I 2022-10-27 16:12:13] ...trap/engines/engine.py.230: loss: 0.18603 [I 2022-10-27 16:12:13] ...optimizers/trijoint.py.151: total_norm: 14.616356 [I 2022-10-27 16:12:19] ...trap/engines/engine.py.224: train: epoch 76 | batch 810/2304 [I 2022-10-27 16:12:19] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:50 [I 2022-10-27 16:12:19] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00254 [I 2022-10-27 16:12:19] ...trap/engines/engine.py.230: loss: 0.11670 [I 2022-10-27 16:12:19] ...optimizers/trijoint.py.151: total_norm: 47.909264 [I 2022-10-27 16:12:25] ...trap/engines/engine.py.224: train: epoch 76 | batch 820/2304 [I 2022-10-27 16:12:25] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:43 [I 2022-10-27 16:12:25] ...trap/engines/engine.py.229: process: 0.55301 | load: 0.00253 [I 2022-10-27 16:12:25] ...trap/engines/engine.py.230: loss: 0.11870 [I 2022-10-27 16:12:25] ...optimizers/trijoint.py.151: total_norm: 24.359192 [I 2022-10-27 16:12:30] ...trap/engines/engine.py.224: train: epoch 76 | batch 830/2304 [I 2022-10-27 16:12:30] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:36 [I 2022-10-27 16:12:30] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00245 [I 2022-10-27 16:12:30] ...trap/engines/engine.py.230: loss: 0.15477 [I 2022-10-27 16:12:30] ...optimizers/trijoint.py.151: total_norm: 18.618542 [I 2022-10-27 16:12:36] ...trap/engines/engine.py.224: train: epoch 76 | batch 840/2304 [I 2022-10-27 16:12:36] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:37 [I 2022-10-27 16:12:36] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00249 [I 2022-10-27 16:12:36] ...trap/engines/engine.py.230: loss: 0.13022 [I 2022-10-27 16:12:36] ...optimizers/trijoint.py.151: total_norm: 14.007727 [I 2022-10-27 16:12:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 850/2304 [I 2022-10-27 16:12:41] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:33 [I 2022-10-27 16:12:41] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00246 [I 2022-10-27 16:12:41] ...trap/engines/engine.py.230: loss: 0.13314 [I 2022-10-27 16:12:41] ...optimizers/trijoint.py.151: total_norm: 10.472167 [I 2022-10-27 16:12:47] ...trap/engines/engine.py.224: train: epoch 76 | batch 860/2304 [I 2022-10-27 16:12:47] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:25 [I 2022-10-27 16:12:47] ...trap/engines/engine.py.229: process: 0.56476 | load: 0.00242 [I 2022-10-27 16:12:47] ...trap/engines/engine.py.230: loss: 0.12708 [I 2022-10-27 16:12:47] ...optimizers/trijoint.py.151: total_norm: 16.043472 [I 2022-10-27 16:12:52] ...trap/engines/engine.py.224: train: epoch 76 | batch 870/2304 [I 2022-10-27 16:12:52] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:13:20 [I 2022-10-27 16:12:52] ...trap/engines/engine.py.229: process: 0.56508 | load: 0.00241 [I 2022-10-27 16:12:52] ...trap/engines/engine.py.230: loss: 0.14215 [I 2022-10-27 16:12:52] ...optimizers/trijoint.py.151: total_norm: 30.623827 [I 2022-10-27 16:12:58] ...trap/engines/engine.py.224: train: epoch 76 | batch 880/2304 [I 2022-10-27 16:12:58] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:18 [I 2022-10-27 16:12:58] ...trap/engines/engine.py.229: process: 0.56518 | load: 0.00245 [I 2022-10-27 16:12:58] ...trap/engines/engine.py.230: loss: 0.13138 [I 2022-10-27 16:12:58] ...optimizers/trijoint.py.151: total_norm: 11.886091 [I 2022-10-27 16:13:04] ...trap/engines/engine.py.224: train: epoch 76 | batch 890/2304 [I 2022-10-27 16:13:04] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:13:03 [I 2022-10-27 16:13:04] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00245 [I 2022-10-27 16:13:04] ...trap/engines/engine.py.230: loss: 0.11694 [I 2022-10-27 16:13:04] ...optimizers/trijoint.py.151: total_norm: 12.817613 [I 2022-10-27 16:13:09] ...trap/engines/engine.py.224: train: epoch 76 | batch 900/2304 [I 2022-10-27 16:13:09] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:13:12 [I 2022-10-27 16:13:09] ...trap/engines/engine.py.229: process: 0.56511 | load: 0.00245 [I 2022-10-27 16:13:09] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-27 16:13:09] ...optimizers/trijoint.py.151: total_norm: 20.987684 [I 2022-10-27 16:13:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 910/2304 [I 2022-10-27 16:13:15] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:56 [I 2022-10-27 16:13:15] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00257 [I 2022-10-27 16:13:15] ...trap/engines/engine.py.230: loss: 0.19169 [I 2022-10-27 16:13:15] ...optimizers/trijoint.py.151: total_norm: 18.371872 [I 2022-10-27 16:13:20] ...trap/engines/engine.py.224: train: epoch 76 | batch 920/2304 [I 2022-10-27 16:13:20] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:52 [I 2022-10-27 16:13:20] ...trap/engines/engine.py.229: process: 0.55280 | load: 0.00253 [I 2022-10-27 16:13:20] ...trap/engines/engine.py.230: loss: 0.25163 [I 2022-10-27 16:13:20] ...optimizers/trijoint.py.151: total_norm: 23.544943 [I 2022-10-27 16:13:26] ...trap/engines/engine.py.224: train: epoch 76 | batch 930/2304 [I 2022-10-27 16:13:26] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:43 [I 2022-10-27 16:13:26] ...trap/engines/engine.py.229: process: 0.54509 | load: 0.00250 [I 2022-10-27 16:13:26] ...trap/engines/engine.py.230: loss: 0.10254 [I 2022-10-27 16:13:26] ...optimizers/trijoint.py.151: total_norm: 15.008213 [I 2022-10-27 16:13:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 940/2304 [I 2022-10-27 16:13:32] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:47 [I 2022-10-27 16:13:32] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00244 [I 2022-10-27 16:13:32] ...trap/engines/engine.py.230: loss: 0.23882 [I 2022-10-27 16:13:32] ...optimizers/trijoint.py.151: total_norm: 17.968664 [I 2022-10-27 16:13:37] ...trap/engines/engine.py.224: train: epoch 76 | batch 950/2304 [I 2022-10-27 16:13:37] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:35 [I 2022-10-27 16:13:37] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00250 [I 2022-10-27 16:13:37] ...trap/engines/engine.py.230: loss: 0.11770 [I 2022-10-27 16:13:37] ...optimizers/trijoint.py.151: total_norm: 16.343061 [I 2022-10-27 16:13:43] ...trap/engines/engine.py.224: train: epoch 76 | batch 960/2304 [I 2022-10-27 16:13:43] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:27 [I 2022-10-27 16:13:43] ...trap/engines/engine.py.229: process: 0.53999 | load: 0.00247 [I 2022-10-27 16:13:43] ...trap/engines/engine.py.230: loss: 0.17937 [I 2022-10-27 16:13:43] ...optimizers/trijoint.py.151: total_norm: 36.791107 [I 2022-10-27 16:13:48] ...trap/engines/engine.py.224: train: epoch 76 | batch 970/2304 [I 2022-10-27 16:13:48] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:12:22 [I 2022-10-27 16:13:48] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00245 [I 2022-10-27 16:13:48] ...trap/engines/engine.py.230: loss: 0.15071 [I 2022-10-27 16:13:48] ...optimizers/trijoint.py.151: total_norm: 17.071808 [I 2022-10-27 16:13:54] ...trap/engines/engine.py.224: train: epoch 76 | batch 980/2304 [I 2022-10-27 16:13:54] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:19 [I 2022-10-27 16:13:54] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00255 [I 2022-10-27 16:13:54] ...trap/engines/engine.py.230: loss: 0.16185 [I 2022-10-27 16:13:54] ...optimizers/trijoint.py.151: total_norm: 22.478024 [I 2022-10-27 16:14:00] ...trap/engines/engine.py.224: train: epoch 76 | batch 990/2304 [I 2022-10-27 16:14:00] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:12:12 [I 2022-10-27 16:14:00] ...trap/engines/engine.py.229: process: 0.55262 | load: 0.00247 [I 2022-10-27 16:14:00] ...trap/engines/engine.py.230: loss: 0.13214 [I 2022-10-27 16:14:00] ...optimizers/trijoint.py.151: total_norm: 16.326204 [I 2022-10-27 16:14:05] ...trap/engines/engine.py.224: train: epoch 76 | batch 1000/2304 [I 2022-10-27 16:14:05] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:05 [I 2022-10-27 16:14:05] ...trap/engines/engine.py.229: process: 0.56212 | load: 0.00254 [I 2022-10-27 16:14:05] ...trap/engines/engine.py.230: loss: 0.11609 [I 2022-10-27 16:14:05] ...optimizers/trijoint.py.151: total_norm: 17.519518 [I 2022-10-27 16:14:11] ...trap/engines/engine.py.224: train: epoch 76 | batch 1010/2304 [I 2022-10-27 16:14:11] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:12:02 [I 2022-10-27 16:14:11] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00247 [I 2022-10-27 16:14:11] ...trap/engines/engine.py.230: loss: 0.12588 [I 2022-10-27 16:14:11] ...optimizers/trijoint.py.151: total_norm: 17.278524 [I 2022-10-27 16:14:16] ...trap/engines/engine.py.224: train: epoch 76 | batch 1020/2304 [I 2022-10-27 16:14:16] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:58 [I 2022-10-27 16:14:16] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00245 [I 2022-10-27 16:14:16] ...trap/engines/engine.py.230: loss: 0.10848 [I 2022-10-27 16:14:16] ...optimizers/trijoint.py.151: total_norm: 8.997573 [I 2022-10-27 16:14:22] ...trap/engines/engine.py.224: train: epoch 76 | batch 1030/2304 [I 2022-10-27 16:14:22] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:49 [I 2022-10-27 16:14:22] ...trap/engines/engine.py.229: process: 0.55315 | load: 0.00292 [I 2022-10-27 16:14:22] ...trap/engines/engine.py.230: loss: 0.10451 [I 2022-10-27 16:14:22] ...optimizers/trijoint.py.151: total_norm: 9.143900 [I 2022-10-27 16:14:28] ...trap/engines/engine.py.224: train: epoch 76 | batch 1040/2304 [I 2022-10-27 16:14:28] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:43 [I 2022-10-27 16:14:28] ...trap/engines/engine.py.229: process: 0.54984 | load: 0.00244 [I 2022-10-27 16:14:28] ...trap/engines/engine.py.230: loss: 0.11262 [I 2022-10-27 16:14:28] ...optimizers/trijoint.py.151: total_norm: 17.617002 [I 2022-10-27 16:14:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 1050/2304 [I 2022-10-27 16:14:33] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:37 [I 2022-10-27 16:14:33] ...trap/engines/engine.py.229: process: 0.54507 | load: 0.00245 [I 2022-10-27 16:14:33] ...trap/engines/engine.py.230: loss: 0.10467 [I 2022-10-27 16:14:33] ...optimizers/trijoint.py.151: total_norm: 11.717087 [I 2022-10-27 16:14:39] ...trap/engines/engine.py.224: train: epoch 76 | batch 1060/2304 [I 2022-10-27 16:14:39] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:33 [I 2022-10-27 16:14:39] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00247 [I 2022-10-27 16:14:39] ...trap/engines/engine.py.230: loss: 0.13949 [I 2022-10-27 16:14:39] ...optimizers/trijoint.py.151: total_norm: 9.380594 [I 2022-10-27 16:14:44] ...trap/engines/engine.py.224: train: epoch 76 | batch 1070/2304 [I 2022-10-27 16:14:44] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:25 [I 2022-10-27 16:14:44] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00244 [I 2022-10-27 16:14:44] ...trap/engines/engine.py.230: loss: 0.10560 [I 2022-10-27 16:14:44] ...optimizers/trijoint.py.151: total_norm: 31.349031 [I 2022-10-27 16:14:50] ...trap/engines/engine.py.224: train: epoch 76 | batch 1080/2304 [I 2022-10-27 16:14:50] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:26 [I 2022-10-27 16:14:50] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00240 [I 2022-10-27 16:14:50] ...trap/engines/engine.py.230: loss: 0.14728 [I 2022-10-27 16:14:50] ...optimizers/trijoint.py.151: total_norm: 11.963905 [I 2022-10-27 16:14:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 1090/2304 [I 2022-10-27 16:14:55] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:12 [I 2022-10-27 16:14:55] ...trap/engines/engine.py.229: process: 0.54086 | load: 0.00243 [I 2022-10-27 16:14:55] ...trap/engines/engine.py.230: loss: 0.17616 [I 2022-10-27 16:14:55] ...optimizers/trijoint.py.151: total_norm: 16.330240 [I 2022-10-27 16:15:01] ...trap/engines/engine.py.224: train: epoch 76 | batch 1100/2304 [I 2022-10-27 16:15:01] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:11:14 [I 2022-10-27 16:15:01] ...trap/engines/engine.py.229: process: 0.55353 | load: 0.00245 [I 2022-10-27 16:15:01] ...trap/engines/engine.py.230: loss: 0.10379 [I 2022-10-27 16:15:01] ...optimizers/trijoint.py.151: total_norm: 10.819759 [I 2022-10-27 16:15:07] ...trap/engines/engine.py.224: train: epoch 76 | batch 1110/2304 [I 2022-10-27 16:15:07] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:11:04 [I 2022-10-27 16:15:07] ...trap/engines/engine.py.229: process: 0.55639 | load: 0.00244 [I 2022-10-27 16:15:07] ...trap/engines/engine.py.230: loss: 0.16115 [I 2022-10-27 16:15:07] ...optimizers/trijoint.py.151: total_norm: 15.015632 [I 2022-10-27 16:15:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 1120/2304 [I 2022-10-27 16:15:12] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:11:01 [I 2022-10-27 16:15:12] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00245 [I 2022-10-27 16:15:12] ...trap/engines/engine.py.230: loss: 0.11062 [I 2022-10-27 16:15:12] ...optimizers/trijoint.py.151: total_norm: 12.981034 [I 2022-10-27 16:15:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 1130/2304 [I 2022-10-27 16:15:18] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:53 [I 2022-10-27 16:15:18] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00255 [I 2022-10-27 16:15:18] ...trap/engines/engine.py.230: loss: 0.11866 [I 2022-10-27 16:15:18] ...optimizers/trijoint.py.151: total_norm: 16.904341 [I 2022-10-27 16:15:23] ...trap/engines/engine.py.224: train: epoch 76 | batch 1140/2304 [I 2022-10-27 16:15:23] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:50 [I 2022-10-27 16:15:23] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00246 [I 2022-10-27 16:15:23] ...trap/engines/engine.py.230: loss: 0.11186 [I 2022-10-27 16:15:23] ...optimizers/trijoint.py.151: total_norm: 6.937365 [I 2022-10-27 16:15:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 1150/2304 [I 2022-10-27 16:15:29] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:45 [I 2022-10-27 16:15:29] ...trap/engines/engine.py.229: process: 0.55892 | load: 0.00257 [I 2022-10-27 16:15:29] ...trap/engines/engine.py.230: loss: 0.16092 [I 2022-10-27 16:15:29] ...optimizers/trijoint.py.151: total_norm: 40.985741 [I 2022-10-27 16:15:35] ...trap/engines/engine.py.224: train: epoch 76 | batch 1160/2304 [I 2022-10-27 16:15:35] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:38 [I 2022-10-27 16:15:35] ...trap/engines/engine.py.229: process: 0.54586 | load: 0.00247 [I 2022-10-27 16:15:35] ...trap/engines/engine.py.230: loss: 0.14164 [I 2022-10-27 16:15:35] ...optimizers/trijoint.py.151: total_norm: 19.929173 [I 2022-10-27 16:15:40] ...trap/engines/engine.py.224: train: epoch 76 | batch 1170/2304 [I 2022-10-27 16:15:40] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:36 [I 2022-10-27 16:15:40] ...trap/engines/engine.py.229: process: 0.56510 | load: 0.00246 [I 2022-10-27 16:15:40] ...trap/engines/engine.py.230: loss: 0.15094 [I 2022-10-27 16:15:40] ...optimizers/trijoint.py.151: total_norm: 20.784029 [I 2022-10-27 16:15:46] ...trap/engines/engine.py.224: train: epoch 76 | batch 1180/2304 [I 2022-10-27 16:15:46] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:29 [I 2022-10-27 16:15:46] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00242 [I 2022-10-27 16:15:46] ...trap/engines/engine.py.230: loss: 0.19069 [I 2022-10-27 16:15:46] ...optimizers/trijoint.py.151: total_norm: 12.980124 [I 2022-10-27 16:15:52] ...trap/engines/engine.py.224: train: epoch 76 | batch 1190/2304 [I 2022-10-27 16:15:52] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:10:18 [I 2022-10-27 16:15:52] ...trap/engines/engine.py.229: process: 0.54344 | load: 0.00244 [I 2022-10-27 16:15:52] ...trap/engines/engine.py.230: loss: 0.20272 [I 2022-10-27 16:15:52] ...optimizers/trijoint.py.151: total_norm: 14.553299 [I 2022-10-27 16:15:57] ...trap/engines/engine.py.224: train: epoch 76 | batch 1200/2304 [I 2022-10-27 16:15:57] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:10:16 [I 2022-10-27 16:15:57] ...trap/engines/engine.py.229: process: 0.55371 | load: 0.00245 [I 2022-10-27 16:15:57] ...trap/engines/engine.py.230: loss: 0.12891 [I 2022-10-27 16:15:57] ...optimizers/trijoint.py.151: total_norm: 17.699722 [I 2022-10-27 16:16:03] ...trap/engines/engine.py.224: train: epoch 76 | batch 1210/2304 [I 2022-10-27 16:16:03] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:10:08 [I 2022-10-27 16:16:03] ...trap/engines/engine.py.229: process: 0.54805 | load: 0.00244 [I 2022-10-27 16:16:03] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-27 16:16:03] ...optimizers/trijoint.py.151: total_norm: 9.290383 [I 2022-10-27 16:16:08] ...trap/engines/engine.py.224: train: epoch 76 | batch 1220/2304 [I 2022-10-27 16:16:08] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:10:14 [I 2022-10-27 16:16:08] ...trap/engines/engine.py.229: process: 0.58421 | load: 0.02184 [I 2022-10-27 16:16:08] ...trap/engines/engine.py.230: loss: 0.22930 [I 2022-10-27 16:16:08] ...optimizers/trijoint.py.151: total_norm: 18.376259 [I 2022-10-27 16:16:14] ...trap/engines/engine.py.224: train: epoch 76 | batch 1230/2304 [I 2022-10-27 16:16:14] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:10:08 [I 2022-10-27 16:16:14] ...trap/engines/engine.py.229: process: 0.57683 | load: 0.00256 [I 2022-10-27 16:16:14] ...trap/engines/engine.py.230: loss: 0.10571 [I 2022-10-27 16:16:14] ...optimizers/trijoint.py.151: total_norm: 8.690564 [I 2022-10-27 16:16:20] ...trap/engines/engine.py.224: train: epoch 76 | batch 1240/2304 [I 2022-10-27 16:16:20] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:54 [I 2022-10-27 16:16:20] ...trap/engines/engine.py.229: process: 0.54339 | load: 0.00248 [I 2022-10-27 16:16:20] ...trap/engines/engine.py.230: loss: 0.12746 [I 2022-10-27 16:16:20] ...optimizers/trijoint.py.151: total_norm: 8.861246 [I 2022-10-27 16:16:25] ...trap/engines/engine.py.224: train: epoch 76 | batch 1250/2304 [I 2022-10-27 16:16:25] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:57 [I 2022-10-27 16:16:25] ...trap/engines/engine.py.229: process: 0.55103 | load: 0.00245 [I 2022-10-27 16:16:25] ...trap/engines/engine.py.230: loss: 0.12115 [I 2022-10-27 16:16:25] ...optimizers/trijoint.py.151: total_norm: 8.760329 [I 2022-10-27 16:16:31] ...trap/engines/engine.py.224: train: epoch 76 | batch 1260/2304 [I 2022-10-27 16:16:31] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:54 [I 2022-10-27 16:16:31] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00251 [I 2022-10-27 16:16:31] ...trap/engines/engine.py.230: loss: 0.15904 [I 2022-10-27 16:16:31] ...optimizers/trijoint.py.151: total_norm: 16.196241 [I 2022-10-27 16:16:37] ...trap/engines/engine.py.224: train: epoch 76 | batch 1270/2304 [I 2022-10-27 16:16:37] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:40 [I 2022-10-27 16:16:37] ...trap/engines/engine.py.229: process: 0.57004 | load: 0.00244 [I 2022-10-27 16:16:37] ...trap/engines/engine.py.230: loss: 0.12384 [I 2022-10-27 16:16:37] ...optimizers/trijoint.py.151: total_norm: 21.024887 [I 2022-10-27 16:16:42] ...trap/engines/engine.py.224: train: epoch 76 | batch 1280/2304 [I 2022-10-27 16:16:42] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:33 [I 2022-10-27 16:16:42] ...trap/engines/engine.py.229: process: 0.54210 | load: 0.00247 [I 2022-10-27 16:16:42] ...trap/engines/engine.py.230: loss: 0.15657 [I 2022-10-27 16:16:42] ...optimizers/trijoint.py.151: total_norm: 14.476493 [I 2022-10-27 16:16:48] ...trap/engines/engine.py.224: train: epoch 76 | batch 1290/2304 [I 2022-10-27 16:16:48] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:29 [I 2022-10-27 16:16:48] ...trap/engines/engine.py.229: process: 0.57169 | load: 0.00242 [I 2022-10-27 16:16:48] ...trap/engines/engine.py.230: loss: 0.12228 [I 2022-10-27 16:16:48] ...optimizers/trijoint.py.151: total_norm: 19.579603 [I 2022-10-27 16:16:54] ...trap/engines/engine.py.224: train: epoch 76 | batch 1300/2304 [I 2022-10-27 16:16:54] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:09:19 [I 2022-10-27 16:16:54] ...trap/engines/engine.py.229: process: 0.54219 | load: 0.00245 [I 2022-10-27 16:16:54] ...trap/engines/engine.py.230: loss: 0.15229 [I 2022-10-27 16:16:54] ...optimizers/trijoint.py.151: total_norm: 33.821434 [I 2022-10-27 16:16:59] ...trap/engines/engine.py.224: train: epoch 76 | batch 1310/2304 [I 2022-10-27 16:16:59] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:09:15 [I 2022-10-27 16:16:59] ...trap/engines/engine.py.229: process: 0.56322 | load: 0.00249 [I 2022-10-27 16:16:59] ...trap/engines/engine.py.230: loss: 0.21116 [I 2022-10-27 16:16:59] ...optimizers/trijoint.py.151: total_norm: 22.617914 [I 2022-10-27 16:17:05] ...trap/engines/engine.py.224: train: epoch 76 | batch 1320/2304 [I 2022-10-27 16:17:05] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:09:09 [I 2022-10-27 16:17:05] ...trap/engines/engine.py.229: process: 0.54933 | load: 0.00246 [I 2022-10-27 16:17:05] ...trap/engines/engine.py.230: loss: 0.15803 [I 2022-10-27 16:17:05] ...optimizers/trijoint.py.151: total_norm: 14.273596 [I 2022-10-27 16:17:10] ...trap/engines/engine.py.224: train: epoch 76 | batch 1330/2304 [I 2022-10-27 16:17:10] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:09:02 [I 2022-10-27 16:17:10] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00255 [I 2022-10-27 16:17:10] ...trap/engines/engine.py.230: loss: 0.14254 [I 2022-10-27 16:17:10] ...optimizers/trijoint.py.151: total_norm: 19.580919 [I 2022-10-27 16:17:16] ...trap/engines/engine.py.224: train: epoch 76 | batch 1340/2304 [I 2022-10-27 16:17:16] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:09:08 [I 2022-10-27 16:17:16] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00249 [I 2022-10-27 16:17:16] ...trap/engines/engine.py.230: loss: 0.22692 [I 2022-10-27 16:17:16] ...optimizers/trijoint.py.151: total_norm: 10.129457 [I 2022-10-27 16:17:22] ...trap/engines/engine.py.224: train: epoch 76 | batch 1350/2304 [I 2022-10-27 16:17:22] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:51 [I 2022-10-27 16:17:22] ...trap/engines/engine.py.229: process: 0.54796 | load: 0.00248 [I 2022-10-27 16:17:22] ...trap/engines/engine.py.230: loss: 0.15173 [I 2022-10-27 16:17:22] ...optimizers/trijoint.py.151: total_norm: 22.226824 [I 2022-10-27 16:17:27] ...trap/engines/engine.py.224: train: epoch 76 | batch 1360/2304 [I 2022-10-27 16:17:27] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:48 [I 2022-10-27 16:17:27] ...trap/engines/engine.py.229: process: 0.54826 | load: 0.00248 [I 2022-10-27 16:17:27] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-27 16:17:27] ...optimizers/trijoint.py.151: total_norm: 11.247105 [I 2022-10-27 16:17:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 1370/2304 [I 2022-10-27 16:17:33] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:43 [I 2022-10-27 16:17:33] ...trap/engines/engine.py.229: process: 0.56004 | load: 0.00241 [I 2022-10-27 16:17:33] ...trap/engines/engine.py.230: loss: 0.15645 [I 2022-10-27 16:17:33] ...optimizers/trijoint.py.151: total_norm: 18.091276 [I 2022-10-27 16:17:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 1380/2304 [I 2022-10-27 16:17:38] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:36 [I 2022-10-27 16:17:38] ...trap/engines/engine.py.229: process: 0.54488 | load: 0.00249 [I 2022-10-27 16:17:38] ...trap/engines/engine.py.230: loss: 0.15779 [I 2022-10-27 16:17:38] ...optimizers/trijoint.py.151: total_norm: 24.335278 [I 2022-10-27 16:17:44] ...trap/engines/engine.py.224: train: epoch 76 | batch 1390/2304 [I 2022-10-27 16:17:44] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:32 [I 2022-10-27 16:17:44] ...trap/engines/engine.py.229: process: 0.55623 | load: 0.00250 [I 2022-10-27 16:17:44] ...trap/engines/engine.py.230: loss: 0.12289 [I 2022-10-27 16:17:44] ...optimizers/trijoint.py.151: total_norm: 25.818897 [I 2022-10-27 16:17:50] ...trap/engines/engine.py.224: train: epoch 76 | batch 1400/2304 [I 2022-10-27 16:17:50] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:33 [I 2022-10-27 16:17:50] ...trap/engines/engine.py.229: process: 0.60738 | load: 0.00242 [I 2022-10-27 16:17:50] ...trap/engines/engine.py.230: loss: 0.14554 [I 2022-10-27 16:17:50] ...optimizers/trijoint.py.151: total_norm: 9.840441 [I 2022-10-27 16:17:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 1410/2304 [I 2022-10-27 16:17:55] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:08:25 [I 2022-10-27 16:17:55] ...trap/engines/engine.py.229: process: 0.55888 | load: 0.00248 [I 2022-10-27 16:17:55] ...trap/engines/engine.py.230: loss: 0.11401 [I 2022-10-27 16:17:55] ...optimizers/trijoint.py.151: total_norm: 13.488205 [I 2022-10-27 16:18:01] ...trap/engines/engine.py.224: train: epoch 76 | batch 1420/2304 [I 2022-10-27 16:18:01] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:08:11 [I 2022-10-27 16:18:01] ...trap/engines/engine.py.229: process: 0.54638 | load: 0.00248 [I 2022-10-27 16:18:01] ...trap/engines/engine.py.230: loss: 0.11572 [I 2022-10-27 16:18:01] ...optimizers/trijoint.py.151: total_norm: 16.717436 [I 2022-10-27 16:18:07] ...trap/engines/engine.py.224: train: epoch 76 | batch 1430/2304 [I 2022-10-27 16:18:07] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:08:07 [I 2022-10-27 16:18:07] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00249 [I 2022-10-27 16:18:07] ...trap/engines/engine.py.230: loss: 0.14235 [I 2022-10-27 16:18:07] ...optimizers/trijoint.py.151: total_norm: 16.866768 [I 2022-10-27 16:18:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 1440/2304 [I 2022-10-27 16:18:12] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:08:02 [I 2022-10-27 16:18:12] ...trap/engines/engine.py.229: process: 0.56134 | load: 0.00244 [I 2022-10-27 16:18:12] ...trap/engines/engine.py.230: loss: 0.09342 [I 2022-10-27 16:18:12] ...optimizers/trijoint.py.151: total_norm: 16.506517 [I 2022-10-27 16:18:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 1450/2304 [I 2022-10-27 16:18:18] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:56 [I 2022-10-27 16:18:18] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00246 [I 2022-10-27 16:18:18] ...trap/engines/engine.py.230: loss: 0.09123 [I 2022-10-27 16:18:18] ...optimizers/trijoint.py.151: total_norm: 12.783964 [I 2022-10-27 16:18:23] ...trap/engines/engine.py.224: train: epoch 76 | batch 1460/2304 [I 2022-10-27 16:18:23] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:51 [I 2022-10-27 16:18:23] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00245 [I 2022-10-27 16:18:23] ...trap/engines/engine.py.230: loss: 0.14929 [I 2022-10-27 16:18:23] ...optimizers/trijoint.py.151: total_norm: 25.896290 [I 2022-10-27 16:18:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 1470/2304 [I 2022-10-27 16:18:29] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:44 [I 2022-10-27 16:18:29] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00246 [I 2022-10-27 16:18:29] ...trap/engines/engine.py.230: loss: 0.13769 [I 2022-10-27 16:18:29] ...optimizers/trijoint.py.151: total_norm: 20.631474 [I 2022-10-27 16:18:35] ...trap/engines/engine.py.224: train: epoch 76 | batch 1480/2304 [I 2022-10-27 16:18:35] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:41 [I 2022-10-27 16:18:35] ...trap/engines/engine.py.229: process: 0.55943 | load: 0.00247 [I 2022-10-27 16:18:35] ...trap/engines/engine.py.230: loss: 0.15212 [I 2022-10-27 16:18:35] ...optimizers/trijoint.py.151: total_norm: 21.417154 [I 2022-10-27 16:18:40] ...trap/engines/engine.py.224: train: epoch 76 | batch 1490/2304 [I 2022-10-27 16:18:40] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:32 [I 2022-10-27 16:18:40] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00242 [I 2022-10-27 16:18:40] ...trap/engines/engine.py.230: loss: 0.10994 [I 2022-10-27 16:18:40] ...optimizers/trijoint.py.151: total_norm: 25.193613 [I 2022-10-27 16:18:46] ...trap/engines/engine.py.224: train: epoch 76 | batch 1500/2304 [I 2022-10-27 16:18:46] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:40 [I 2022-10-27 16:18:46] ...trap/engines/engine.py.229: process: 0.57815 | load: 0.00272 [I 2022-10-27 16:18:46] ...trap/engines/engine.py.230: loss: 0.13109 [I 2022-10-27 16:18:46] ...optimizers/trijoint.py.151: total_norm: 20.433262 [I 2022-10-27 16:18:51] ...trap/engines/engine.py.224: train: epoch 76 | batch 1510/2304 [I 2022-10-27 16:18:51] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:22 [I 2022-10-27 16:18:51] ...trap/engines/engine.py.229: process: 0.55213 | load: 0.00241 [I 2022-10-27 16:18:51] ...trap/engines/engine.py.230: loss: 0.14046 [I 2022-10-27 16:18:51] ...optimizers/trijoint.py.151: total_norm: 20.860821 [I 2022-10-27 16:18:57] ...trap/engines/engine.py.224: train: epoch 76 | batch 1520/2304 [I 2022-10-27 16:18:57] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:07:15 [I 2022-10-27 16:18:57] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00250 [I 2022-10-27 16:18:57] ...trap/engines/engine.py.230: loss: 0.10185 [I 2022-10-27 16:18:57] ...optimizers/trijoint.py.151: total_norm: 30.073534 [I 2022-10-27 16:19:03] ...trap/engines/engine.py.224: train: epoch 76 | batch 1530/2304 [I 2022-10-27 16:19:03] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:07:11 [I 2022-10-27 16:19:03] ...trap/engines/engine.py.229: process: 0.55510 | load: 0.00244 [I 2022-10-27 16:19:03] ...trap/engines/engine.py.230: loss: 0.12514 [I 2022-10-27 16:19:03] ...optimizers/trijoint.py.151: total_norm: 21.074957 [I 2022-10-27 16:19:08] ...trap/engines/engine.py.224: train: epoch 76 | batch 1540/2304 [I 2022-10-27 16:19:08] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:07:07 [I 2022-10-27 16:19:08] ...trap/engines/engine.py.229: process: 0.55375 | load: 0.00255 [I 2022-10-27 16:19:08] ...trap/engines/engine.py.230: loss: 0.14095 [I 2022-10-27 16:19:08] ...optimizers/trijoint.py.151: total_norm: 16.201376 [I 2022-10-27 16:19:14] ...trap/engines/engine.py.224: train: epoch 76 | batch 1550/2304 [I 2022-10-27 16:19:14] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:07:01 [I 2022-10-27 16:19:14] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00252 [I 2022-10-27 16:19:14] ...trap/engines/engine.py.230: loss: 0.13379 [I 2022-10-27 16:19:14] ...optimizers/trijoint.py.151: total_norm: 10.618338 [I 2022-10-27 16:19:19] ...trap/engines/engine.py.224: train: epoch 76 | batch 1560/2304 [I 2022-10-27 16:19:19] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:54 [I 2022-10-27 16:19:19] ...trap/engines/engine.py.229: process: 0.55936 | load: 0.00245 [I 2022-10-27 16:19:19] ...trap/engines/engine.py.230: loss: 0.12982 [I 2022-10-27 16:19:19] ...optimizers/trijoint.py.151: total_norm: 11.685208 [I 2022-10-27 16:19:25] ...trap/engines/engine.py.224: train: epoch 76 | batch 1570/2304 [I 2022-10-27 16:19:25] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:51 [I 2022-10-27 16:19:25] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00244 [I 2022-10-27 16:19:25] ...trap/engines/engine.py.230: loss: 0.15349 [I 2022-10-27 16:19:25] ...optimizers/trijoint.py.151: total_norm: 21.045481 [I 2022-10-27 16:19:31] ...trap/engines/engine.py.224: train: epoch 76 | batch 1580/2304 [I 2022-10-27 16:19:31] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:45 [I 2022-10-27 16:19:31] ...trap/engines/engine.py.229: process: 0.55897 | load: 0.00245 [I 2022-10-27 16:19:31] ...trap/engines/engine.py.230: loss: 0.15172 [I 2022-10-27 16:19:31] ...optimizers/trijoint.py.151: total_norm: 27.241030 [I 2022-10-27 16:19:36] ...trap/engines/engine.py.224: train: epoch 76 | batch 1590/2304 [I 2022-10-27 16:19:36] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:40 [I 2022-10-27 16:19:36] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00249 [I 2022-10-27 16:19:36] ...trap/engines/engine.py.230: loss: 0.12413 [I 2022-10-27 16:19:36] ...optimizers/trijoint.py.151: total_norm: 15.095980 [I 2022-10-27 16:19:42] ...trap/engines/engine.py.224: train: epoch 76 | batch 1600/2304 [I 2022-10-27 16:19:42] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:33 [I 2022-10-27 16:19:42] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00244 [I 2022-10-27 16:19:42] ...trap/engines/engine.py.230: loss: 0.10528 [I 2022-10-27 16:19:42] ...optimizers/trijoint.py.151: total_norm: 9.560665 [I 2022-10-27 16:19:47] ...trap/engines/engine.py.224: train: epoch 76 | batch 1610/2304 [I 2022-10-27 16:19:47] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:26 [I 2022-10-27 16:19:47] ...trap/engines/engine.py.229: process: 0.56405 | load: 0.00253 [I 2022-10-27 16:19:47] ...trap/engines/engine.py.230: loss: 0.20609 [I 2022-10-27 16:19:47] ...optimizers/trijoint.py.151: total_norm: 14.673060 [I 2022-10-27 16:19:53] ...trap/engines/engine.py.224: train: epoch 76 | batch 1620/2304 [I 2022-10-27 16:19:53] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:21 [I 2022-10-27 16:19:53] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00245 [I 2022-10-27 16:19:53] ...trap/engines/engine.py.230: loss: 0.16547 [I 2022-10-27 16:19:53] ...optimizers/trijoint.py.151: total_norm: 16.546801 [I 2022-10-27 16:19:59] ...trap/engines/engine.py.224: train: epoch 76 | batch 1630/2304 [I 2022-10-27 16:19:59] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:21 [I 2022-10-27 16:19:59] ...trap/engines/engine.py.229: process: 0.58616 | load: 0.00246 [I 2022-10-27 16:19:59] ...trap/engines/engine.py.230: loss: 0.17408 [I 2022-10-27 16:19:59] ...optimizers/trijoint.py.151: total_norm: 16.800024 [I 2022-10-27 16:20:04] ...trap/engines/engine.py.224: train: epoch 76 | batch 1640/2304 [I 2022-10-27 16:20:04] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:06:08 [I 2022-10-27 16:20:04] ...trap/engines/engine.py.229: process: 0.56489 | load: 0.00246 [I 2022-10-27 16:20:04] ...trap/engines/engine.py.230: loss: 0.12115 [I 2022-10-27 16:20:04] ...optimizers/trijoint.py.151: total_norm: 13.670664 [I 2022-10-27 16:20:10] ...trap/engines/engine.py.224: train: epoch 76 | batch 1650/2304 [I 2022-10-27 16:20:10] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:06:03 [I 2022-10-27 16:20:10] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00244 [I 2022-10-27 16:20:10] ...trap/engines/engine.py.230: loss: 0.17250 [I 2022-10-27 16:20:10] ...optimizers/trijoint.py.151: total_norm: 8.944058 [I 2022-10-27 16:20:15] ...trap/engines/engine.py.224: train: epoch 76 | batch 1660/2304 [I 2022-10-27 16:20:15] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:57 [I 2022-10-27 16:20:15] ...trap/engines/engine.py.229: process: 0.54817 | load: 0.00243 [I 2022-10-27 16:20:15] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-27 16:20:15] ...optimizers/trijoint.py.151: total_norm: 21.018873 [I 2022-10-27 16:20:21] ...trap/engines/engine.py.224: train: epoch 76 | batch 1670/2304 [I 2022-10-27 16:20:21] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:52 [I 2022-10-27 16:20:21] ...trap/engines/engine.py.229: process: 0.55575 | load: 0.00244 [I 2022-10-27 16:20:21] ...trap/engines/engine.py.230: loss: 0.14192 [I 2022-10-27 16:20:21] ...optimizers/trijoint.py.151: total_norm: 27.386171 [I 2022-10-27 16:20:26] ...trap/engines/engine.py.224: train: epoch 76 | batch 1680/2304 [I 2022-10-27 16:20:26] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:43 [I 2022-10-27 16:20:26] ...trap/engines/engine.py.229: process: 0.54186 | load: 0.00244 [I 2022-10-27 16:20:26] ...trap/engines/engine.py.230: loss: 0.10951 [I 2022-10-27 16:20:26] ...optimizers/trijoint.py.151: total_norm: 10.944317 [I 2022-10-27 16:20:32] ...trap/engines/engine.py.224: train: epoch 76 | batch 1690/2304 [I 2022-10-27 16:20:32] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:40 [I 2022-10-27 16:20:32] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00243 [I 2022-10-27 16:20:32] ...trap/engines/engine.py.230: loss: 0.15095 [I 2022-10-27 16:20:32] ...optimizers/trijoint.py.151: total_norm: 17.154842 [I 2022-10-27 16:20:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 1700/2304 [I 2022-10-27 16:20:38] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:36 [I 2022-10-27 16:20:38] ...trap/engines/engine.py.229: process: 0.56709 | load: 0.00244 [I 2022-10-27 16:20:38] ...trap/engines/engine.py.230: loss: 0.09754 [I 2022-10-27 16:20:38] ...optimizers/trijoint.py.151: total_norm: 12.247383 [I 2022-10-27 16:20:43] ...trap/engines/engine.py.224: train: epoch 76 | batch 1710/2304 [I 2022-10-27 16:20:43] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:28 [I 2022-10-27 16:20:43] ...trap/engines/engine.py.229: process: 0.54301 | load: 0.00244 [I 2022-10-27 16:20:43] ...trap/engines/engine.py.230: loss: 0.14828 [I 2022-10-27 16:20:43] ...optimizers/trijoint.py.151: total_norm: 18.261496 [I 2022-10-27 16:20:49] ...trap/engines/engine.py.224: train: epoch 76 | batch 1720/2304 [I 2022-10-27 16:20:49] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:23 [I 2022-10-27 16:20:49] ...trap/engines/engine.py.229: process: 0.55160 | load: 0.00243 [I 2022-10-27 16:20:49] ...trap/engines/engine.py.230: loss: 0.08853 [I 2022-10-27 16:20:49] ...optimizers/trijoint.py.151: total_norm: 10.025762 [I 2022-10-27 16:20:54] ...trap/engines/engine.py.224: train: epoch 76 | batch 1730/2304 [I 2022-10-27 16:20:54] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:05:28 [I 2022-10-27 16:20:54] ...trap/engines/engine.py.229: process: 0.64027 | load: 0.00242 [I 2022-10-27 16:20:54] ...trap/engines/engine.py.230: loss: 0.18345 [I 2022-10-27 16:20:54] ...optimizers/trijoint.py.151: total_norm: 15.049172 [I 2022-10-27 16:21:00] ...trap/engines/engine.py.224: train: epoch 76 | batch 1740/2304 [I 2022-10-27 16:21:00] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:05:15 [I 2022-10-27 16:21:00] ...trap/engines/engine.py.229: process: 0.56023 | load: 0.00241 [I 2022-10-27 16:21:00] ...trap/engines/engine.py.230: loss: 0.17510 [I 2022-10-27 16:21:00] ...optimizers/trijoint.py.151: total_norm: 24.147152 [I 2022-10-27 16:21:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 1750/2304 [I 2022-10-27 16:21:06] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:07 [I 2022-10-27 16:21:06] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00251 [I 2022-10-27 16:21:06] ...trap/engines/engine.py.230: loss: 0.11261 [I 2022-10-27 16:21:06] ...optimizers/trijoint.py.151: total_norm: 13.536827 [I 2022-10-27 16:21:11] ...trap/engines/engine.py.224: train: epoch 76 | batch 1760/2304 [I 2022-10-27 16:21:11] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:05:03 [I 2022-10-27 16:21:11] ...trap/engines/engine.py.229: process: 0.57655 | load: 0.00247 [I 2022-10-27 16:21:11] ...trap/engines/engine.py.230: loss: 0.13136 [I 2022-10-27 16:21:11] ...optimizers/trijoint.py.151: total_norm: 11.141276 [I 2022-10-27 16:21:17] ...trap/engines/engine.py.224: train: epoch 76 | batch 1770/2304 [I 2022-10-27 16:21:17] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:55 [I 2022-10-27 16:21:17] ...trap/engines/engine.py.229: process: 0.54672 | load: 0.00244 [I 2022-10-27 16:21:17] ...trap/engines/engine.py.230: loss: 0.08869 [I 2022-10-27 16:21:17] ...optimizers/trijoint.py.151: total_norm: 8.102350 [I 2022-10-27 16:21:22] ...trap/engines/engine.py.224: train: epoch 76 | batch 1780/2304 [I 2022-10-27 16:21:22] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:52 [I 2022-10-27 16:21:22] ...trap/engines/engine.py.229: process: 0.55104 | load: 0.00243 [I 2022-10-27 16:21:22] ...trap/engines/engine.py.230: loss: 0.12306 [I 2022-10-27 16:21:22] ...optimizers/trijoint.py.151: total_norm: 13.781892 [I 2022-10-27 16:21:28] ...trap/engines/engine.py.224: train: epoch 76 | batch 1790/2304 [I 2022-10-27 16:21:28] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:46 [I 2022-10-27 16:21:28] ...trap/engines/engine.py.229: process: 0.56838 | load: 0.00246 [I 2022-10-27 16:21:28] ...trap/engines/engine.py.230: loss: 0.12287 [I 2022-10-27 16:21:28] ...optimizers/trijoint.py.151: total_norm: 14.619169 [I 2022-10-27 16:21:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 1800/2304 [I 2022-10-27 16:21:33] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:37 [I 2022-10-27 16:21:33] ...trap/engines/engine.py.229: process: 0.54717 | load: 0.00251 [I 2022-10-27 16:21:33] ...trap/engines/engine.py.230: loss: 0.19667 [I 2022-10-27 16:21:33] ...optimizers/trijoint.py.151: total_norm: 32.384857 [I 2022-10-27 16:21:39] ...trap/engines/engine.py.224: train: epoch 76 | batch 1810/2304 [I 2022-10-27 16:21:39] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:35 [I 2022-10-27 16:21:39] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00244 [I 2022-10-27 16:21:39] ...trap/engines/engine.py.230: loss: 0.11744 [I 2022-10-27 16:21:39] ...optimizers/trijoint.py.151: total_norm: 15.825643 [I 2022-10-27 16:21:45] ...trap/engines/engine.py.224: train: epoch 76 | batch 1820/2304 [I 2022-10-27 16:21:45] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:30 [I 2022-10-27 16:21:45] ...trap/engines/engine.py.229: process: 0.55321 | load: 0.00246 [I 2022-10-27 16:21:45] ...trap/engines/engine.py.230: loss: 0.19718 [I 2022-10-27 16:21:45] ...optimizers/trijoint.py.151: total_norm: 58.980568 [I 2022-10-27 16:21:50] ...trap/engines/engine.py.224: train: epoch 76 | batch 1830/2304 [I 2022-10-27 16:21:50] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:23 [I 2022-10-27 16:21:50] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00259 [I 2022-10-27 16:21:50] ...trap/engines/engine.py.230: loss: 0.11485 [I 2022-10-27 16:21:50] ...optimizers/trijoint.py.151: total_norm: 12.019117 [I 2022-10-27 16:21:56] ...trap/engines/engine.py.224: train: epoch 76 | batch 1840/2304 [I 2022-10-27 16:21:56] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:04:19 [I 2022-10-27 16:21:56] ...trap/engines/engine.py.229: process: 0.56822 | load: 0.00246 [I 2022-10-27 16:21:56] ...trap/engines/engine.py.230: loss: 0.15460 [I 2022-10-27 16:21:56] ...optimizers/trijoint.py.151: total_norm: 17.372843 [I 2022-10-27 16:22:01] ...trap/engines/engine.py.224: train: epoch 76 | batch 1850/2304 [I 2022-10-27 16:22:01] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:04:12 [I 2022-10-27 16:22:01] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00250 [I 2022-10-27 16:22:01] ...trap/engines/engine.py.230: loss: 0.11186 [I 2022-10-27 16:22:01] ...optimizers/trijoint.py.151: total_norm: 15.662015 [I 2022-10-27 16:22:07] ...trap/engines/engine.py.224: train: epoch 76 | batch 1860/2304 [I 2022-10-27 16:22:07] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:04:08 [I 2022-10-27 16:22:07] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00244 [I 2022-10-27 16:22:07] ...trap/engines/engine.py.230: loss: 0.09153 [I 2022-10-27 16:22:07] ...optimizers/trijoint.py.151: total_norm: 10.173429 [I 2022-10-27 16:22:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 1870/2304 [I 2022-10-27 16:22:12] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:04:03 [I 2022-10-27 16:22:12] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00247 [I 2022-10-27 16:22:12] ...trap/engines/engine.py.230: loss: 0.22430 [I 2022-10-27 16:22:12] ...optimizers/trijoint.py.151: total_norm: 25.251266 [I 2022-10-27 16:22:18] ...trap/engines/engine.py.224: train: epoch 76 | batch 1880/2304 [I 2022-10-27 16:22:18] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:57 [I 2022-10-27 16:22:18] ...trap/engines/engine.py.229: process: 0.55868 | load: 0.00245 [I 2022-10-27 16:22:18] ...trap/engines/engine.py.230: loss: 0.10071 [I 2022-10-27 16:22:18] ...optimizers/trijoint.py.151: total_norm: 19.666283 [I 2022-10-27 16:22:24] ...trap/engines/engine.py.224: train: epoch 76 | batch 1890/2304 [I 2022-10-27 16:22:24] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:50 [I 2022-10-27 16:22:24] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00247 [I 2022-10-27 16:22:24] ...trap/engines/engine.py.230: loss: 0.14172 [I 2022-10-27 16:22:24] ...optimizers/trijoint.py.151: total_norm: 16.523460 [I 2022-10-27 16:22:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 1900/2304 [I 2022-10-27 16:22:29] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:45 [I 2022-10-27 16:22:29] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00244 [I 2022-10-27 16:22:29] ...trap/engines/engine.py.230: loss: 0.10788 [I 2022-10-27 16:22:29] ...optimizers/trijoint.py.151: total_norm: 27.358038 [I 2022-10-27 16:22:35] ...trap/engines/engine.py.224: train: epoch 76 | batch 1910/2304 [I 2022-10-27 16:22:35] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:40 [I 2022-10-27 16:22:35] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00246 [I 2022-10-27 16:22:35] ...trap/engines/engine.py.230: loss: 0.15397 [I 2022-10-27 16:22:35] ...optimizers/trijoint.py.151: total_norm: 17.055737 [I 2022-10-27 16:22:41] ...trap/engines/engine.py.224: train: epoch 76 | batch 1920/2304 [I 2022-10-27 16:22:41] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:35 [I 2022-10-27 16:22:41] ...trap/engines/engine.py.229: process: 0.56823 | load: 0.00243 [I 2022-10-27 16:22:41] ...trap/engines/engine.py.230: loss: 0.14941 [I 2022-10-27 16:22:41] ...optimizers/trijoint.py.151: total_norm: 23.251965 [I 2022-10-27 16:22:46] ...trap/engines/engine.py.224: train: epoch 76 | batch 1930/2304 [I 2022-10-27 16:22:46] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:30 [I 2022-10-27 16:22:46] ...trap/engines/engine.py.229: process: 0.56750 | load: 0.00241 [I 2022-10-27 16:22:46] ...trap/engines/engine.py.230: loss: 0.14070 [I 2022-10-27 16:22:46] ...optimizers/trijoint.py.151: total_norm: 23.438433 [I 2022-10-27 16:22:52] ...trap/engines/engine.py.224: train: epoch 76 | batch 1940/2304 [I 2022-10-27 16:22:52] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:25 [I 2022-10-27 16:22:52] ...trap/engines/engine.py.229: process: 0.56918 | load: 0.00261 [I 2022-10-27 16:22:52] ...trap/engines/engine.py.230: loss: 0.13907 [I 2022-10-27 16:22:52] ...optimizers/trijoint.py.151: total_norm: 14.158102 [I 2022-10-27 16:22:57] ...trap/engines/engine.py.224: train: epoch 76 | batch 1950/2304 [I 2022-10-27 16:22:57] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:03:17 [I 2022-10-27 16:22:57] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00250 [I 2022-10-27 16:22:57] ...trap/engines/engine.py.230: loss: 0.10134 [I 2022-10-27 16:22:57] ...optimizers/trijoint.py.151: total_norm: 12.240036 [I 2022-10-27 16:23:03] ...trap/engines/engine.py.224: train: epoch 76 | batch 1960/2304 [I 2022-10-27 16:23:03] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:03:12 [I 2022-10-27 16:23:03] ...trap/engines/engine.py.229: process: 0.54840 | load: 0.00249 [I 2022-10-27 16:23:03] ...trap/engines/engine.py.230: loss: 0.11726 [I 2022-10-27 16:23:03] ...optimizers/trijoint.py.151: total_norm: 10.179523 [I 2022-10-27 16:23:09] ...trap/engines/engine.py.224: train: epoch 76 | batch 1970/2304 [I 2022-10-27 16:23:09] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:03:06 [I 2022-10-27 16:23:09] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00251 [I 2022-10-27 16:23:09] ...trap/engines/engine.py.230: loss: 0.14501 [I 2022-10-27 16:23:09] ...optimizers/trijoint.py.151: total_norm: 14.274764 [I 2022-10-27 16:23:14] ...trap/engines/engine.py.224: train: epoch 76 | batch 1980/2304 [I 2022-10-27 16:23:14] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:03:02 [I 2022-10-27 16:23:14] ...trap/engines/engine.py.229: process: 0.57013 | load: 0.00245 [I 2022-10-27 16:23:14] ...trap/engines/engine.py.230: loss: 0.11833 [I 2022-10-27 16:23:14] ...optimizers/trijoint.py.151: total_norm: 10.255802 [I 2022-10-27 16:23:20] ...trap/engines/engine.py.224: train: epoch 76 | batch 1990/2304 [I 2022-10-27 16:23:20] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:55 [I 2022-10-27 16:23:20] ...trap/engines/engine.py.229: process: 0.55703 | load: 0.00247 [I 2022-10-27 16:23:20] ...trap/engines/engine.py.230: loss: 0.15327 [I 2022-10-27 16:23:20] ...optimizers/trijoint.py.151: total_norm: 10.111831 [I 2022-10-27 16:23:25] ...trap/engines/engine.py.224: train: epoch 76 | batch 2000/2304 [I 2022-10-27 16:23:25] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:49 [I 2022-10-27 16:23:25] ...trap/engines/engine.py.229: process: 0.55233 | load: 0.00246 [I 2022-10-27 16:23:25] ...trap/engines/engine.py.230: loss: 0.17745 [I 2022-10-27 16:23:25] ...optimizers/trijoint.py.151: total_norm: 37.956676 [I 2022-10-27 16:23:31] ...trap/engines/engine.py.224: train: epoch 76 | batch 2010/2304 [I 2022-10-27 16:23:31] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:45 [I 2022-10-27 16:23:31] ...trap/engines/engine.py.229: process: 0.56976 | load: 0.00250 [I 2022-10-27 16:23:31] ...trap/engines/engine.py.230: loss: 0.12602 [I 2022-10-27 16:23:31] ...optimizers/trijoint.py.151: total_norm: 16.152664 [I 2022-10-27 16:23:37] ...trap/engines/engine.py.224: train: epoch 76 | batch 2020/2304 [I 2022-10-27 16:23:37] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:41 [I 2022-10-27 16:23:37] ...trap/engines/engine.py.229: process: 0.57896 | load: 0.00252 [I 2022-10-27 16:23:37] ...trap/engines/engine.py.230: loss: 0.19366 [I 2022-10-27 16:23:37] ...optimizers/trijoint.py.151: total_norm: 26.679018 [I 2022-10-27 16:23:42] ...trap/engines/engine.py.224: train: epoch 76 | batch 2030/2304 [I 2022-10-27 16:23:42] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:34 [I 2022-10-27 16:23:42] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00255 [I 2022-10-27 16:23:42] ...trap/engines/engine.py.230: loss: 0.17694 [I 2022-10-27 16:23:42] ...optimizers/trijoint.py.151: total_norm: 12.523839 [I 2022-10-27 16:23:48] ...trap/engines/engine.py.224: train: epoch 76 | batch 2040/2304 [I 2022-10-27 16:23:48] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:28 [I 2022-10-27 16:23:48] ...trap/engines/engine.py.229: process: 0.56283 | load: 0.00259 [I 2022-10-27 16:23:48] ...trap/engines/engine.py.230: loss: 0.17387 [I 2022-10-27 16:23:48] ...optimizers/trijoint.py.151: total_norm: 14.423795 [I 2022-10-27 16:23:54] ...trap/engines/engine.py.224: train: epoch 76 | batch 2050/2304 [I 2022-10-27 16:23:54] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:20 [I 2022-10-27 16:23:54] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00254 [I 2022-10-27 16:23:54] ...trap/engines/engine.py.230: loss: 0.09714 [I 2022-10-27 16:23:54] ...optimizers/trijoint.py.151: total_norm: 15.059958 [I 2022-10-27 16:23:59] ...trap/engines/engine.py.224: train: epoch 76 | batch 2060/2304 [I 2022-10-27 16:23:59] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:02:15 [I 2022-10-27 16:23:59] ...trap/engines/engine.py.229: process: 0.54521 | load: 0.00251 [I 2022-10-27 16:23:59] ...trap/engines/engine.py.230: loss: 0.08990 [I 2022-10-27 16:23:59] ...optimizers/trijoint.py.151: total_norm: 8.008080 [I 2022-10-27 16:24:05] ...trap/engines/engine.py.224: train: epoch 76 | batch 2070/2304 [I 2022-10-27 16:24:05] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:11 [I 2022-10-27 16:24:05] ...trap/engines/engine.py.229: process: 0.57160 | load: 0.00256 [I 2022-10-27 16:24:05] ...trap/engines/engine.py.230: loss: 0.14242 [I 2022-10-27 16:24:05] ...optimizers/trijoint.py.151: total_norm: 28.923519 [I 2022-10-27 16:24:10] ...trap/engines/engine.py.224: train: epoch 76 | batch 2080/2304 [I 2022-10-27 16:24:10] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:02:04 [I 2022-10-27 16:24:10] ...trap/engines/engine.py.229: process: 0.54966 | load: 0.00256 [I 2022-10-27 16:24:10] ...trap/engines/engine.py.230: loss: 0.17622 [I 2022-10-27 16:24:10] ...optimizers/trijoint.py.151: total_norm: 37.111229 [I 2022-10-27 16:24:16] ...trap/engines/engine.py.224: train: epoch 76 | batch 2090/2304 [I 2022-10-27 16:24:16] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:59 [I 2022-10-27 16:24:16] ...trap/engines/engine.py.229: process: 0.55108 | load: 0.00258 [I 2022-10-27 16:24:16] ...trap/engines/engine.py.230: loss: 0.14253 [I 2022-10-27 16:24:16] ...optimizers/trijoint.py.151: total_norm: 14.693112 [I 2022-10-27 16:24:21] ...trap/engines/engine.py.224: train: epoch 76 | batch 2100/2304 [I 2022-10-27 16:24:21] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:53 [I 2022-10-27 16:24:21] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00257 [I 2022-10-27 16:24:21] ...trap/engines/engine.py.230: loss: 0.10116 [I 2022-10-27 16:24:21] ...optimizers/trijoint.py.151: total_norm: 21.045824 [I 2022-10-27 16:24:27] ...trap/engines/engine.py.224: train: epoch 76 | batch 2110/2304 [I 2022-10-27 16:24:27] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:48 [I 2022-10-27 16:24:27] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00247 [I 2022-10-27 16:24:27] ...trap/engines/engine.py.230: loss: 0.11113 [I 2022-10-27 16:24:27] ...optimizers/trijoint.py.151: total_norm: 23.216127 [I 2022-10-27 16:24:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 2120/2304 [I 2022-10-27 16:24:33] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:42 [I 2022-10-27 16:24:33] ...trap/engines/engine.py.229: process: 0.55799 | load: 0.00252 [I 2022-10-27 16:24:33] ...trap/engines/engine.py.230: loss: 0.08963 [I 2022-10-27 16:24:33] ...optimizers/trijoint.py.151: total_norm: 9.321822 [I 2022-10-27 16:24:38] ...trap/engines/engine.py.224: train: epoch 76 | batch 2130/2304 [I 2022-10-27 16:24:38] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:36 [I 2022-10-27 16:24:38] ...trap/engines/engine.py.229: process: 0.55070 | load: 0.00251 [I 2022-10-27 16:24:38] ...trap/engines/engine.py.230: loss: 0.14232 [I 2022-10-27 16:24:38] ...optimizers/trijoint.py.151: total_norm: 34.850197 [I 2022-10-27 16:24:44] ...trap/engines/engine.py.224: train: epoch 76 | batch 2140/2304 [I 2022-10-27 16:24:44] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:31 [I 2022-10-27 16:24:44] ...trap/engines/engine.py.229: process: 0.56313 | load: 0.00273 [I 2022-10-27 16:24:44] ...trap/engines/engine.py.230: loss: 0.18944 [I 2022-10-27 16:24:44] ...optimizers/trijoint.py.151: total_norm: 89.821213 [I 2022-10-27 16:24:49] ...trap/engines/engine.py.224: train: epoch 76 | batch 2150/2304 [I 2022-10-27 16:24:49] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:25 [I 2022-10-27 16:24:49] ...trap/engines/engine.py.229: process: 0.56552 | load: 0.00258 [I 2022-10-27 16:24:49] ...trap/engines/engine.py.230: loss: 0.09591 [I 2022-10-27 16:24:49] ...optimizers/trijoint.py.151: total_norm: 8.761796 [I 2022-10-27 16:24:55] ...trap/engines/engine.py.224: train: epoch 76 | batch 2160/2304 [I 2022-10-27 16:24:55] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:01:21 [I 2022-10-27 16:24:55] ...trap/engines/engine.py.229: process: 0.56950 | load: 0.00260 [I 2022-10-27 16:24:55] ...trap/engines/engine.py.230: loss: 0.15699 [I 2022-10-27 16:24:55] ...optimizers/trijoint.py.151: total_norm: 22.268944 [I 2022-10-27 16:25:01] ...trap/engines/engine.py.224: train: epoch 76 | batch 2170/2304 [I 2022-10-27 16:25:01] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:14 [I 2022-10-27 16:25:01] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00247 [I 2022-10-27 16:25:01] ...trap/engines/engine.py.230: loss: 0.17496 [I 2022-10-27 16:25:01] ...optimizers/trijoint.py.151: total_norm: 25.411716 [I 2022-10-27 16:25:06] ...trap/engines/engine.py.224: train: epoch 76 | batch 2180/2304 [I 2022-10-27 16:25:06] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:01:09 [I 2022-10-27 16:25:06] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00244 [I 2022-10-27 16:25:06] ...trap/engines/engine.py.230: loss: 0.22824 [I 2022-10-27 16:25:06] ...optimizers/trijoint.py.151: total_norm: 21.636482 [I 2022-10-27 16:25:12] ...trap/engines/engine.py.224: train: epoch 76 | batch 2190/2304 [I 2022-10-27 16:25:12] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:01:03 [I 2022-10-27 16:25:12] ...trap/engines/engine.py.229: process: 0.56643 | load: 0.00257 [I 2022-10-27 16:25:12] ...trap/engines/engine.py.230: loss: 0.15581 [I 2022-10-27 16:25:12] ...optimizers/trijoint.py.151: total_norm: 23.639071 [I 2022-10-27 16:25:17] ...trap/engines/engine.py.224: train: epoch 76 | batch 2200/2304 [I 2022-10-27 16:25:17] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:58 [I 2022-10-27 16:25:17] ...trap/engines/engine.py.229: process: 0.56399 | load: 0.00260 [I 2022-10-27 16:25:17] ...trap/engines/engine.py.230: loss: 0.10511 [I 2022-10-27 16:25:17] ...optimizers/trijoint.py.151: total_norm: 13.391945 [I 2022-10-27 16:25:23] ...trap/engines/engine.py.224: train: epoch 76 | batch 2210/2304 [I 2022-10-27 16:25:23] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:52 [I 2022-10-27 16:25:23] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00275 [I 2022-10-27 16:25:23] ...trap/engines/engine.py.230: loss: 0.27026 [I 2022-10-27 16:25:23] ...optimizers/trijoint.py.151: total_norm: 32.174236 [I 2022-10-27 16:25:29] ...trap/engines/engine.py.224: train: epoch 76 | batch 2220/2304 [I 2022-10-27 16:25:29] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:46 [I 2022-10-27 16:25:29] ...trap/engines/engine.py.229: process: 0.55267 | load: 0.00268 [I 2022-10-27 16:25:29] ...trap/engines/engine.py.230: loss: 0.13199 [I 2022-10-27 16:25:29] ...optimizers/trijoint.py.151: total_norm: 14.107430 [I 2022-10-27 16:25:34] ...trap/engines/engine.py.224: train: epoch 76 | batch 2230/2304 [I 2022-10-27 16:25:34] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:41 [I 2022-10-27 16:25:34] ...trap/engines/engine.py.229: process: 0.56656 | load: 0.00269 [I 2022-10-27 16:25:34] ...trap/engines/engine.py.230: loss: 0.18910 [I 2022-10-27 16:25:34] ...optimizers/trijoint.py.151: total_norm: 18.144489 [I 2022-10-27 16:25:40] ...trap/engines/engine.py.224: train: epoch 76 | batch 2240/2304 [I 2022-10-27 16:25:40] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:35 [I 2022-10-27 16:25:40] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00261 [I 2022-10-27 16:25:40] ...trap/engines/engine.py.230: loss: 0.15444 [I 2022-10-27 16:25:40] ...optimizers/trijoint.py.151: total_norm: 14.303361 [I 2022-10-27 16:25:45] ...trap/engines/engine.py.224: train: epoch 76 | batch 2250/2304 [I 2022-10-27 16:25:45] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:30 [I 2022-10-27 16:25:45] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00263 [I 2022-10-27 16:25:45] ...trap/engines/engine.py.230: loss: 0.11565 [I 2022-10-27 16:25:45] ...optimizers/trijoint.py.151: total_norm: 10.492938 [I 2022-10-27 16:26:08] ...trap/engines/engine.py.224: train: epoch 76 | batch 2260/2304 [I 2022-10-27 16:26:08] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:02:00 [I 2022-10-27 16:26:08] ...trap/engines/engine.py.229: process: 0.54792 | load: 0.00237 [I 2022-10-27 16:26:08] ...trap/engines/engine.py.230: loss: 0.10513 [I 2022-10-27 16:26:08] ...optimizers/trijoint.py.151: total_norm: 8.990859 [I 2022-10-27 16:26:14] ...trap/engines/engine.py.224: train: epoch 76 | batch 2270/2304 [I 2022-10-27 16:26:14] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:26 [I 2022-10-27 16:26:14] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00304 [I 2022-10-27 16:26:14] ...trap/engines/engine.py.230: loss: 0.09913 [I 2022-10-27 16:26:14] ...optimizers/trijoint.py.151: total_norm: 7.056764 [I 2022-10-27 16:26:19] ...trap/engines/engine.py.224: train: epoch 76 | batch 2280/2304 [I 2022-10-27 16:26:19] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:00:14 [I 2022-10-27 16:26:19] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00269 [I 2022-10-27 16:26:19] ...trap/engines/engine.py.230: loss: 0.09812 [I 2022-10-27 16:26:19] ...optimizers/trijoint.py.151: total_norm: 17.058155 [I 2022-10-27 16:26:25] ...trap/engines/engine.py.224: train: epoch 76 | batch 2290/2304 [I 2022-10-27 16:26:25] ...trap/engines/engine.py.228: elapsed: 0:22:02 | left: 0:00:07 [I 2022-10-27 16:26:25] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00253 [I 2022-10-27 16:26:25] ...trap/engines/engine.py.230: loss: 0.20504 [I 2022-10-27 16:26:25] ...optimizers/trijoint.py.151: total_norm: 25.506626 [I 2022-10-27 16:26:31] ...trap/engines/engine.py.224: train: epoch 76 | batch 2300/2304 [I 2022-10-27 16:26:31] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:00:02 [I 2022-10-27 16:26:31] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00015 [I 2022-10-27 16:26:31] ...trap/engines/engine.py.230: loss: 0.10660 [I 2022-10-27 16:26:31] ...optimizers/trijoint.py.151: total_norm: 10.345687 [I 2022-10-27 16:26:33] ...trap/engines/engine.py.224: train: epoch 76 | batch 2304/2304 [I 2022-10-27 16:26:33] ...trap/engines/engine.py.228: elapsed: 0:22:10 | left: 0:00:00 [I 2022-10-27 16:26:33] ...trap/engines/engine.py.229: process: 0.56005 | load: 0.00016 [I 2022-10-27 16:26:33] ...trap/engines/engine.py.230: loss: 0.18294 [I 2022-10-27 16:26:33] ...optimizers/trijoint.py.151: total_norm: 21.653265 [S 2022-10-27 16:26:34] ...trap/engines/engine.py.129: train_epoch.epoch: 76 [S 2022-10-27 16:26:34] ...trap/engines/engine.py.129: train_epoch.loss: 0.1394457552929504 [I 2022-10-27 16:26:34] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 16:26:35] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 76 [I 2022-10-27 16:26:35] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 16:26:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 16:26:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 16:26:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 16:26:40] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 16:26:40] ...trap/engines/engine.py.321: eval: epoch 76 | batch 0/510 [I 2022-10-27 16:26:40] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:40:39 [I 2022-10-27 16:26:40] ...trap/engines/engine.py.326: process: 4.78301 | load: 3.59819 [I 2022-10-27 16:26:43] ...trap/engines/engine.py.321: eval: epoch 76 | batch 10/510 [I 2022-10-27 16:26:43] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:46 [I 2022-10-27 16:26:43] ...trap/engines/engine.py.326: process: 0.31765 | load: 0.00020 [I 2022-10-27 16:26:47] ...trap/engines/engine.py.321: eval: epoch 76 | batch 20/510 [I 2022-10-27 16:26:47] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:09 [I 2022-10-27 16:26:47] ...trap/engines/engine.py.326: process: 0.32867 | load: 0.00027 [I 2022-10-27 16:26:50] ...trap/engines/engine.py.321: eval: epoch 76 | batch 30/510 [I 2022-10-27 16:26:50] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:43 [I 2022-10-27 16:26:50] ...trap/engines/engine.py.326: process: 0.33494 | load: 0.00017 [I 2022-10-27 16:26:53] ...trap/engines/engine.py.321: eval: epoch 76 | batch 40/510 [I 2022-10-27 16:26:53] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:36 [I 2022-10-27 16:26:53] ...trap/engines/engine.py.326: process: 0.32203 | load: 0.00020 [I 2022-10-27 16:26:57] ...trap/engines/engine.py.321: eval: epoch 76 | batch 50/510 [I 2022-10-27 16:26:57] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:33 [I 2022-10-27 16:26:57] ...trap/engines/engine.py.326: process: 0.33534 | load: 0.00016 [I 2022-10-27 16:27:00] ...trap/engines/engine.py.321: eval: epoch 76 | batch 60/510 [I 2022-10-27 16:27:00] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:33 [I 2022-10-27 16:27:00] ...trap/engines/engine.py.326: process: 0.35864 | load: 0.00016 [I 2022-10-27 16:27:04] ...trap/engines/engine.py.321: eval: epoch 76 | batch 70/510 [I 2022-10-27 16:27:04] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:29 [I 2022-10-27 16:27:04] ...trap/engines/engine.py.326: process: 0.34643 | load: 0.00019 [I 2022-10-27 16:27:07] ...trap/engines/engine.py.321: eval: epoch 76 | batch 80/510 [I 2022-10-27 16:27:07] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:22 [I 2022-10-27 16:27:07] ...trap/engines/engine.py.326: process: 0.32869 | load: 0.00019 [I 2022-10-27 16:27:10] ...trap/engines/engine.py.321: eval: epoch 76 | batch 90/510 [I 2022-10-27 16:27:10] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:18 [I 2022-10-27 16:27:10] ...trap/engines/engine.py.326: process: 0.32694 | load: 0.00017 [I 2022-10-27 16:27:14] ...trap/engines/engine.py.321: eval: epoch 76 | batch 100/510 [I 2022-10-27 16:27:14] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:20 [I 2022-10-27 16:27:14] ...trap/engines/engine.py.326: process: 0.32721 | load: 0.00018 [I 2022-10-27 16:27:16] ...trap/engines/engine.py.321: eval: epoch 76 | batch 110/510 [I 2022-10-27 16:27:16] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:59 [I 2022-10-27 16:27:16] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00023 [I 2022-10-27 16:27:19] ...trap/engines/engine.py.321: eval: epoch 76 | batch 120/510 [I 2022-10-27 16:27:19] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:54 [I 2022-10-27 16:27:19] ...trap/engines/engine.py.326: process: 0.29229 | load: 0.00019 [I 2022-10-27 16:27:22] ...trap/engines/engine.py.321: eval: epoch 76 | batch 130/510 [I 2022-10-27 16:27:22] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:52 [I 2022-10-27 16:27:22] ...trap/engines/engine.py.326: process: 0.31160 | load: 0.00014 [I 2022-10-27 16:27:25] ...trap/engines/engine.py.321: eval: epoch 76 | batch 140/510 [I 2022-10-27 16:27:25] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-27 16:27:25] ...trap/engines/engine.py.326: process: 0.29296 | load: 0.00019 [I 2022-10-27 16:27:28] ...trap/engines/engine.py.321: eval: epoch 76 | batch 150/510 [I 2022-10-27 16:27:28] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:44 [I 2022-10-27 16:27:28] ...trap/engines/engine.py.326: process: 0.28955 | load: 0.00022 [I 2022-10-27 16:27:31] ...trap/engines/engine.py.321: eval: epoch 76 | batch 160/510 [I 2022-10-27 16:27:31] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 16:27:31] ...trap/engines/engine.py.326: process: 0.28206 | load: 0.00019 [I 2022-10-27 16:27:34] ...trap/engines/engine.py.321: eval: epoch 76 | batch 170/510 [I 2022-10-27 16:27:34] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 16:27:34] ...trap/engines/engine.py.326: process: 0.29006 | load: 0.00023 [I 2022-10-27 16:27:37] ...trap/engines/engine.py.321: eval: epoch 76 | batch 180/510 [I 2022-10-27 16:27:37] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 16:27:37] ...trap/engines/engine.py.326: process: 0.28480 | load: 0.00018 [I 2022-10-27 16:27:40] ...trap/engines/engine.py.321: eval: epoch 76 | batch 190/510 [I 2022-10-27 16:27:40] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 16:27:40] ...trap/engines/engine.py.326: process: 0.28864 | load: 0.00020 [I 2022-10-27 16:27:43] ...trap/engines/engine.py.321: eval: epoch 76 | batch 200/510 [I 2022-10-27 16:27:43] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:30 [I 2022-10-27 16:27:43] ...trap/engines/engine.py.326: process: 0.28484 | load: 0.00021 [I 2022-10-27 16:27:46] ...trap/engines/engine.py.321: eval: epoch 76 | batch 210/510 [I 2022-10-27 16:27:46] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:28 [I 2022-10-27 16:27:46] ...trap/engines/engine.py.326: process: 0.30695 | load: 0.00017 [I 2022-10-27 16:27:49] ...trap/engines/engine.py.321: eval: epoch 76 | batch 220/510 [I 2022-10-27 16:27:49] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 16:27:49] ...trap/engines/engine.py.326: process: 0.28337 | load: 0.00017 [I 2022-10-27 16:27:52] ...trap/engines/engine.py.321: eval: epoch 76 | batch 230/510 [I 2022-10-27 16:27:52] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:23 [I 2022-10-27 16:27:52] ...trap/engines/engine.py.326: process: 0.33166 | load: 0.00018 [I 2022-10-27 16:27:54] ...trap/engines/engine.py.321: eval: epoch 76 | batch 240/510 [I 2022-10-27 16:27:54] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-27 16:27:54] ...trap/engines/engine.py.326: process: 0.27971 | load: 0.00017 [I 2022-10-27 16:27:57] ...trap/engines/engine.py.321: eval: epoch 76 | batch 250/510 [I 2022-10-27 16:27:57] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-27 16:27:57] ...trap/engines/engine.py.326: process: 0.28676 | load: 0.00019 [I 2022-10-27 16:28:00] ...trap/engines/engine.py.321: eval: epoch 76 | batch 260/510 [I 2022-10-27 16:28:00] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 16:28:00] ...trap/engines/engine.py.326: process: 0.29190 | load: 0.00017 [I 2022-10-27 16:28:03] ...trap/engines/engine.py.321: eval: epoch 76 | batch 270/510 [I 2022-10-27 16:28:03] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 16:28:03] ...trap/engines/engine.py.326: process: 0.29020 | load: 0.00016 [I 2022-10-27 16:28:06] ...trap/engines/engine.py.321: eval: epoch 76 | batch 280/510 [I 2022-10-27 16:28:06] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 16:28:06] ...trap/engines/engine.py.326: process: 0.28787 | load: 0.00018 [I 2022-10-27 16:28:09] ...trap/engines/engine.py.321: eval: epoch 76 | batch 290/510 [I 2022-10-27 16:28:09] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 16:28:09] ...trap/engines/engine.py.326: process: 0.28991 | load: 0.00016 [I 2022-10-27 16:28:12] ...trap/engines/engine.py.321: eval: epoch 76 | batch 300/510 [I 2022-10-27 16:28:12] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 16:28:12] ...trap/engines/engine.py.326: process: 0.29038 | load: 0.00019 [I 2022-10-27 16:28:15] ...trap/engines/engine.py.321: eval: epoch 76 | batch 310/510 [I 2022-10-27 16:28:15] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:58 [I 2022-10-27 16:28:15] ...trap/engines/engine.py.326: process: 0.28788 | load: 0.00015 [I 2022-10-27 16:28:18] ...trap/engines/engine.py.321: eval: epoch 76 | batch 320/510 [I 2022-10-27 16:28:18] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:55 [I 2022-10-27 16:28:18] ...trap/engines/engine.py.326: process: 0.28738 | load: 0.00022 [I 2022-10-27 16:28:21] ...trap/engines/engine.py.321: eval: epoch 76 | batch 330/510 [I 2022-10-27 16:28:21] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-27 16:28:21] ...trap/engines/engine.py.326: process: 0.29369 | load: 0.00017 [I 2022-10-27 16:28:23] ...trap/engines/engine.py.321: eval: epoch 76 | batch 340/510 [I 2022-10-27 16:28:23] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-27 16:28:23] ...trap/engines/engine.py.326: process: 0.28853 | load: 0.00015 [I 2022-10-27 16:28:26] ...trap/engines/engine.py.321: eval: epoch 76 | batch 350/510 [I 2022-10-27 16:28:26] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 16:28:26] ...trap/engines/engine.py.326: process: 0.29138 | load: 0.00015 [I 2022-10-27 16:28:29] ...trap/engines/engine.py.321: eval: epoch 76 | batch 360/510 [I 2022-10-27 16:28:29] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-27 16:28:29] ...trap/engines/engine.py.326: process: 0.28276 | load: 0.00018 [I 2022-10-27 16:28:32] ...trap/engines/engine.py.321: eval: epoch 76 | batch 370/510 [I 2022-10-27 16:28:32] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 16:28:32] ...trap/engines/engine.py.326: process: 0.29252 | load: 0.00018 [I 2022-10-27 16:28:35] ...trap/engines/engine.py.321: eval: epoch 76 | batch 380/510 [I 2022-10-27 16:28:35] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 16:28:35] ...trap/engines/engine.py.326: process: 0.28394 | load: 0.00016 [I 2022-10-27 16:28:38] ...trap/engines/engine.py.321: eval: epoch 76 | batch 390/510 [I 2022-10-27 16:28:38] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 16:28:38] ...trap/engines/engine.py.326: process: 0.28898 | load: 0.00015 [I 2022-10-27 16:28:41] ...trap/engines/engine.py.321: eval: epoch 76 | batch 400/510 [I 2022-10-27 16:28:41] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 16:28:41] ...trap/engines/engine.py.326: process: 0.28859 | load: 0.00016 [I 2022-10-27 16:28:44] ...trap/engines/engine.py.321: eval: epoch 76 | batch 410/510 [I 2022-10-27 16:28:44] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 16:28:44] ...trap/engines/engine.py.326: process: 0.29067 | load: 0.00015 [I 2022-10-27 16:28:47] ...trap/engines/engine.py.321: eval: epoch 76 | batch 420/510 [I 2022-10-27 16:28:47] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-27 16:28:47] ...trap/engines/engine.py.326: process: 0.28857 | load: 0.00020 [I 2022-10-27 16:28:50] ...trap/engines/engine.py.321: eval: epoch 76 | batch 430/510 [I 2022-10-27 16:28:50] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 16:28:50] ...trap/engines/engine.py.326: process: 0.28873 | load: 0.00019 [I 2022-10-27 16:28:52] ...trap/engines/engine.py.321: eval: epoch 76 | batch 440/510 [I 2022-10-27 16:28:52] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 16:28:52] ...trap/engines/engine.py.326: process: 0.28405 | load: 0.00020 [I 2022-10-27 16:28:55] ...trap/engines/engine.py.321: eval: epoch 76 | batch 450/510 [I 2022-10-27 16:28:55] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 16:28:55] ...trap/engines/engine.py.326: process: 0.29159 | load: 0.00021 [I 2022-10-27 16:28:58] ...trap/engines/engine.py.321: eval: epoch 76 | batch 460/510 [I 2022-10-27 16:28:58] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-27 16:28:58] ...trap/engines/engine.py.326: process: 0.28685 | load: 0.00016 [I 2022-10-27 16:29:01] ...trap/engines/engine.py.321: eval: epoch 76 | batch 470/510 [I 2022-10-27 16:29:01] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-27 16:29:01] ...trap/engines/engine.py.326: process: 0.29120 | load: 0.00017 [I 2022-10-27 16:29:04] ...trap/engines/engine.py.321: eval: epoch 76 | batch 480/510 [I 2022-10-27 16:29:04] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 16:29:04] ...trap/engines/engine.py.326: process: 0.28877 | load: 0.00016 [I 2022-10-27 16:29:07] ...trap/engines/engine.py.321: eval: epoch 76 | batch 490/510 [I 2022-10-27 16:29:07] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 16:29:07] ...trap/engines/engine.py.326: process: 0.29482 | load: 0.00015 [I 2022-10-27 16:29:10] ...trap/engines/engine.py.321: eval: epoch 76 | batch 500/510 [I 2022-10-27 16:29:10] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 16:29:10] ...trap/engines/engine.py.326: process: 0.28918 | load: 0.00016 [I 2022-10-27 16:29:13] ...trap/engines/engine.py.321: eval: epoch 76 | batch 510/510 [I 2022-10-27 16:29:13] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 16:29:13] ...trap/engines/engine.py.326: process: 0.28395 | load: 0.00012 [S 2022-10-27 16:29:14] ...trap/engines/engine.py.132: eval_epoch.epoch: 76 [I 2022-10-27 16:29:14] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.726700 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.910900 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940300 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.723700 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909500 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.938400 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014561 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008642 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008533 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014086 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009657 [S 2022-10-27 16:29:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007486 [I 2022-10-27 16:29:19] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 16:29:19] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 16:29:20] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 16:29:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 16:29:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 16:29:23] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 16:29:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 16:29:30] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 16:29:30] ...trap/engines/engine.py.163: Training model on trainset for epoch 77 [I 2022-10-27 16:29:30] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 16:29:33] ...trap/engines/engine.py.224: train: epoch 77 | batch 0/2304 [I 2022-10-27 16:29:33] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:49:09 [I 2022-10-27 16:29:33] ...trap/engines/engine.py.229: process: 2.84253 | load: 2.20539 [I 2022-10-27 16:29:33] ...trap/engines/engine.py.230: loss: 0.12022 [I 2022-10-27 16:29:33] ...optimizers/trijoint.py.151: total_norm: 15.234001 [I 2022-10-27 16:29:39] ...trap/engines/engine.py.224: train: epoch 77 | batch 10/2304 [I 2022-10-27 16:29:39] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:41 [I 2022-10-27 16:29:39] ...trap/engines/engine.py.229: process: 0.54742 | load: 0.00443 [I 2022-10-27 16:29:39] ...trap/engines/engine.py.230: loss: 0.14994 [I 2022-10-27 16:29:39] ...optimizers/trijoint.py.151: total_norm: 9.868947 [I 2022-10-27 16:29:44] ...trap/engines/engine.py.224: train: epoch 77 | batch 20/2304 [I 2022-10-27 16:29:44] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:05 [I 2022-10-27 16:29:44] ...trap/engines/engine.py.229: process: 0.56894 | load: 0.00380 [I 2022-10-27 16:29:44] ...trap/engines/engine.py.230: loss: 0.10350 [I 2022-10-27 16:29:44] ...optimizers/trijoint.py.151: total_norm: 12.046154 [I 2022-10-27 16:29:50] ...trap/engines/engine.py.224: train: epoch 77 | batch 30/2304 [I 2022-10-27 16:29:50] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-27 16:29:50] ...trap/engines/engine.py.229: process: 0.56351 | load: 0.00313 [I 2022-10-27 16:29:50] ...trap/engines/engine.py.230: loss: 0.23725 [I 2022-10-27 16:29:50] ...optimizers/trijoint.py.151: total_norm: 13.259532 [I 2022-10-27 16:29:55] ...trap/engines/engine.py.224: train: epoch 77 | batch 40/2304 [I 2022-10-27 16:29:55] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:12 [I 2022-10-27 16:29:55] ...trap/engines/engine.py.229: process: 0.56138 | load: 0.00324 [I 2022-10-27 16:29:55] ...trap/engines/engine.py.230: loss: 0.13669 [I 2022-10-27 16:29:55] ...optimizers/trijoint.py.151: total_norm: 25.604036 [I 2022-10-27 16:30:01] ...trap/engines/engine.py.224: train: epoch 77 | batch 50/2304 [I 2022-10-27 16:30:01] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:36 [I 2022-10-27 16:30:01] ...trap/engines/engine.py.229: process: 0.61193 | load: 0.00288 [I 2022-10-27 16:30:01] ...trap/engines/engine.py.230: loss: 0.10080 [I 2022-10-27 16:30:01] ...optimizers/trijoint.py.151: total_norm: 10.242032 [I 2022-10-27 16:30:07] ...trap/engines/engine.py.224: train: epoch 77 | batch 60/2304 [I 2022-10-27 16:30:07] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:04 [I 2022-10-27 16:30:07] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00340 [I 2022-10-27 16:30:07] ...trap/engines/engine.py.230: loss: 0.16975 [I 2022-10-27 16:30:07] ...optimizers/trijoint.py.151: total_norm: 19.354189 [I 2022-10-27 16:30:12] ...trap/engines/engine.py.224: train: epoch 77 | batch 70/2304 [I 2022-10-27 16:30:12] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:47 [I 2022-10-27 16:30:12] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00281 [I 2022-10-27 16:30:12] ...trap/engines/engine.py.230: loss: 0.15238 [I 2022-10-27 16:30:12] ...optimizers/trijoint.py.151: total_norm: 20.092087 [I 2022-10-27 16:30:18] ...trap/engines/engine.py.224: train: epoch 77 | batch 80/2304 [I 2022-10-27 16:30:18] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:36 [I 2022-10-27 16:30:18] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00280 [I 2022-10-27 16:30:18] ...trap/engines/engine.py.230: loss: 0.20140 [I 2022-10-27 16:30:18] ...optimizers/trijoint.py.151: total_norm: 52.457813 [I 2022-10-27 16:30:24] ...trap/engines/engine.py.224: train: epoch 77 | batch 90/2304 [I 2022-10-27 16:30:24] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:31 [I 2022-10-27 16:30:24] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00269 [I 2022-10-27 16:30:24] ...trap/engines/engine.py.230: loss: 0.18384 [I 2022-10-27 16:30:24] ...optimizers/trijoint.py.151: total_norm: 15.766106 [I 2022-10-27 16:30:29] ...trap/engines/engine.py.224: train: epoch 77 | batch 100/2304 [I 2022-10-27 16:30:29] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:43 [I 2022-10-27 16:30:29] ...trap/engines/engine.py.229: process: 0.55936 | load: 0.00286 [I 2022-10-27 16:30:29] ...trap/engines/engine.py.230: loss: 0.13051 [I 2022-10-27 16:30:29] ...optimizers/trijoint.py.151: total_norm: 11.979301 [I 2022-10-27 16:30:35] ...trap/engines/engine.py.224: train: epoch 77 | batch 110/2304 [I 2022-10-27 16:30:35] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:32 [I 2022-10-27 16:30:35] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00256 [I 2022-10-27 16:30:35] ...trap/engines/engine.py.230: loss: 0.12761 [I 2022-10-27 16:30:35] ...optimizers/trijoint.py.151: total_norm: 16.751034 [I 2022-10-27 16:30:40] ...trap/engines/engine.py.224: train: epoch 77 | batch 120/2304 [I 2022-10-27 16:30:40] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:27 [I 2022-10-27 16:30:40] ...trap/engines/engine.py.229: process: 0.55458 | load: 0.00274 [I 2022-10-27 16:30:40] ...trap/engines/engine.py.230: loss: 0.10805 [I 2022-10-27 16:30:40] ...optimizers/trijoint.py.151: total_norm: 8.706579 [I 2022-10-27 16:30:46] ...trap/engines/engine.py.224: train: epoch 77 | batch 130/2304 [I 2022-10-27 16:30:46] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:11 [I 2022-10-27 16:30:46] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00276 [I 2022-10-27 16:30:46] ...trap/engines/engine.py.230: loss: 0.14586 [I 2022-10-27 16:30:46] ...optimizers/trijoint.py.151: total_norm: 13.330873 [I 2022-10-27 16:30:52] ...trap/engines/engine.py.224: train: epoch 77 | batch 140/2304 [I 2022-10-27 16:30:52] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:03 [I 2022-10-27 16:30:52] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00253 [I 2022-10-27 16:30:52] ...trap/engines/engine.py.230: loss: 0.10881 [I 2022-10-27 16:30:52] ...optimizers/trijoint.py.151: total_norm: 9.811823 [I 2022-10-27 16:30:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 150/2304 [I 2022-10-27 16:30:57] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:03 [I 2022-10-27 16:30:57] ...trap/engines/engine.py.229: process: 0.55609 | load: 0.00262 [I 2022-10-27 16:30:57] ...trap/engines/engine.py.230: loss: 0.08094 [I 2022-10-27 16:30:57] ...optimizers/trijoint.py.151: total_norm: 8.106205 [I 2022-10-27 16:31:03] ...trap/engines/engine.py.224: train: epoch 77 | batch 160/2304 [I 2022-10-27 16:31:03] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:57 [I 2022-10-27 16:31:03] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00275 [I 2022-10-27 16:31:03] ...trap/engines/engine.py.230: loss: 0.11186 [I 2022-10-27 16:31:03] ...optimizers/trijoint.py.151: total_norm: 10.491980 [I 2022-10-27 16:31:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 170/2304 [I 2022-10-27 16:31:08] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:20:02 [I 2022-10-27 16:31:08] ...trap/engines/engine.py.229: process: 0.56049 | load: 0.00252 [I 2022-10-27 16:31:08] ...trap/engines/engine.py.230: loss: 0.13936 [I 2022-10-27 16:31:08] ...optimizers/trijoint.py.151: total_norm: 15.404216 [I 2022-10-27 16:31:14] ...trap/engines/engine.py.224: train: epoch 77 | batch 180/2304 [I 2022-10-27 16:31:14] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:45 [I 2022-10-27 16:31:14] ...trap/engines/engine.py.229: process: 0.55001 | load: 0.00267 [I 2022-10-27 16:31:14] ...trap/engines/engine.py.230: loss: 0.10847 [I 2022-10-27 16:31:14] ...optimizers/trijoint.py.151: total_norm: 13.145284 [I 2022-10-27 16:31:20] ...trap/engines/engine.py.224: train: epoch 77 | batch 190/2304 [I 2022-10-27 16:31:20] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:24 [I 2022-10-27 16:31:20] ...trap/engines/engine.py.229: process: 0.55212 | load: 0.00249 [I 2022-10-27 16:31:20] ...trap/engines/engine.py.230: loss: 0.11825 [I 2022-10-27 16:31:20] ...optimizers/trijoint.py.151: total_norm: 16.247980 [I 2022-10-27 16:31:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 200/2304 [I 2022-10-27 16:31:25] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:26 [I 2022-10-27 16:31:25] ...trap/engines/engine.py.229: process: 0.56238 | load: 0.00246 [I 2022-10-27 16:31:25] ...trap/engines/engine.py.230: loss: 0.15896 [I 2022-10-27 16:31:25] ...optimizers/trijoint.py.151: total_norm: 12.531392 [I 2022-10-27 16:31:31] ...trap/engines/engine.py.224: train: epoch 77 | batch 210/2304 [I 2022-10-27 16:31:31] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:20 [I 2022-10-27 16:31:31] ...trap/engines/engine.py.229: process: 0.54700 | load: 0.00249 [I 2022-10-27 16:31:31] ...trap/engines/engine.py.230: loss: 0.10988 [I 2022-10-27 16:31:31] ...optimizers/trijoint.py.151: total_norm: 14.856652 [I 2022-10-27 16:31:36] ...trap/engines/engine.py.224: train: epoch 77 | batch 220/2304 [I 2022-10-27 16:31:36] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:10 [I 2022-10-27 16:31:36] ...trap/engines/engine.py.229: process: 0.55215 | load: 0.00266 [I 2022-10-27 16:31:36] ...trap/engines/engine.py.230: loss: 0.15217 [I 2022-10-27 16:31:36] ...optimizers/trijoint.py.151: total_norm: 10.306186 [I 2022-10-27 16:31:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 230/2304 [I 2022-10-27 16:31:42] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:10 [I 2022-10-27 16:31:42] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00258 [I 2022-10-27 16:31:42] ...trap/engines/engine.py.230: loss: 0.12903 [I 2022-10-27 16:31:42] ...optimizers/trijoint.py.151: total_norm: 15.276838 [I 2022-10-27 16:31:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 240/2304 [I 2022-10-27 16:31:47] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:01 [I 2022-10-27 16:31:47] ...trap/engines/engine.py.229: process: 0.54657 | load: 0.00246 [I 2022-10-27 16:31:47] ...trap/engines/engine.py.230: loss: 0.18288 [I 2022-10-27 16:31:47] ...optimizers/trijoint.py.151: total_norm: 33.596058 [I 2022-10-27 16:31:53] ...trap/engines/engine.py.224: train: epoch 77 | batch 250/2304 [I 2022-10-27 16:31:53] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:48 [I 2022-10-27 16:31:53] ...trap/engines/engine.py.229: process: 0.54558 | load: 0.00255 [I 2022-10-27 16:31:53] ...trap/engines/engine.py.230: loss: 0.15208 [I 2022-10-27 16:31:53] ...optimizers/trijoint.py.151: total_norm: 8.437499 [I 2022-10-27 16:31:58] ...trap/engines/engine.py.224: train: epoch 77 | batch 260/2304 [I 2022-10-27 16:31:58] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:55 [I 2022-10-27 16:31:58] ...trap/engines/engine.py.229: process: 0.55138 | load: 0.00243 [I 2022-10-27 16:31:58] ...trap/engines/engine.py.230: loss: 0.14972 [I 2022-10-27 16:31:58] ...optimizers/trijoint.py.151: total_norm: 9.384151 [I 2022-10-27 16:32:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 270/2304 [I 2022-10-27 16:32:04] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:57 [I 2022-10-27 16:32:04] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00257 [I 2022-10-27 16:32:04] ...trap/engines/engine.py.230: loss: 0.09948 [I 2022-10-27 16:32:04] ...optimizers/trijoint.py.151: total_norm: 10.334719 [I 2022-10-27 16:32:10] ...trap/engines/engine.py.224: train: epoch 77 | batch 280/2304 [I 2022-10-27 16:32:10] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:40 [I 2022-10-27 16:32:10] ...trap/engines/engine.py.229: process: 0.54147 | load: 0.00253 [I 2022-10-27 16:32:10] ...trap/engines/engine.py.230: loss: 0.13457 [I 2022-10-27 16:32:10] ...optimizers/trijoint.py.151: total_norm: 24.621614 [I 2022-10-27 16:32:15] ...trap/engines/engine.py.224: train: epoch 77 | batch 290/2304 [I 2022-10-27 16:32:15] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:40 [I 2022-10-27 16:32:15] ...trap/engines/engine.py.229: process: 0.57478 | load: 0.00270 [I 2022-10-27 16:32:15] ...trap/engines/engine.py.230: loss: 0.13740 [I 2022-10-27 16:32:15] ...optimizers/trijoint.py.151: total_norm: 38.994503 [I 2022-10-27 16:32:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 300/2304 [I 2022-10-27 16:32:21] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:37 [I 2022-10-27 16:32:21] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00250 [I 2022-10-27 16:32:21] ...trap/engines/engine.py.230: loss: 0.16636 [I 2022-10-27 16:32:21] ...optimizers/trijoint.py.151: total_norm: 16.266363 [I 2022-10-27 16:32:26] ...trap/engines/engine.py.224: train: epoch 77 | batch 310/2304 [I 2022-10-27 16:32:26] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:30 [I 2022-10-27 16:32:26] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00255 [I 2022-10-27 16:32:26] ...trap/engines/engine.py.230: loss: 0.12711 [I 2022-10-27 16:32:26] ...optimizers/trijoint.py.151: total_norm: 22.223824 [I 2022-10-27 16:32:32] ...trap/engines/engine.py.224: train: epoch 77 | batch 320/2304 [I 2022-10-27 16:32:32] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:24 [I 2022-10-27 16:32:32] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00247 [I 2022-10-27 16:32:32] ...trap/engines/engine.py.230: loss: 0.16237 [I 2022-10-27 16:32:32] ...optimizers/trijoint.py.151: total_norm: 11.942700 [I 2022-10-27 16:32:37] ...trap/engines/engine.py.224: train: epoch 77 | batch 330/2304 [I 2022-10-27 16:32:37] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:20 [I 2022-10-27 16:32:37] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00247 [I 2022-10-27 16:32:37] ...trap/engines/engine.py.230: loss: 0.09962 [I 2022-10-27 16:32:37] ...optimizers/trijoint.py.151: total_norm: 9.064244 [I 2022-10-27 16:32:43] ...trap/engines/engine.py.224: train: epoch 77 | batch 340/2304 [I 2022-10-27 16:32:43] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:20 [I 2022-10-27 16:32:43] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00247 [I 2022-10-27 16:32:43] ...trap/engines/engine.py.230: loss: 0.12506 [I 2022-10-27 16:32:43] ...optimizers/trijoint.py.151: total_norm: 9.586921 [I 2022-10-27 16:32:49] ...trap/engines/engine.py.224: train: epoch 77 | batch 350/2304 [I 2022-10-27 16:32:49] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:18 [I 2022-10-27 16:32:49] ...trap/engines/engine.py.229: process: 0.56417 | load: 0.00245 [I 2022-10-27 16:32:49] ...trap/engines/engine.py.230: loss: 0.20142 [I 2022-10-27 16:32:49] ...optimizers/trijoint.py.151: total_norm: 18.622505 [I 2022-10-27 16:32:54] ...trap/engines/engine.py.224: train: epoch 77 | batch 360/2304 [I 2022-10-27 16:32:54] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:06 [I 2022-10-27 16:32:54] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00242 [I 2022-10-27 16:32:54] ...trap/engines/engine.py.230: loss: 0.13155 [I 2022-10-27 16:32:54] ...optimizers/trijoint.py.151: total_norm: 23.436968 [I 2022-10-27 16:33:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 370/2304 [I 2022-10-27 16:33:00] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:12 [I 2022-10-27 16:33:00] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00260 [I 2022-10-27 16:33:00] ...trap/engines/engine.py.230: loss: 0.10802 [I 2022-10-27 16:33:00] ...optimizers/trijoint.py.151: total_norm: 15.464946 [I 2022-10-27 16:33:06] ...trap/engines/engine.py.224: train: epoch 77 | batch 380/2304 [I 2022-10-27 16:33:06] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:56 [I 2022-10-27 16:33:06] ...trap/engines/engine.py.229: process: 0.56214 | load: 0.00250 [I 2022-10-27 16:33:06] ...trap/engines/engine.py.230: loss: 0.13789 [I 2022-10-27 16:33:06] ...optimizers/trijoint.py.151: total_norm: 11.111555 [I 2022-10-27 16:33:11] ...trap/engines/engine.py.224: train: epoch 77 | batch 390/2304 [I 2022-10-27 16:33:11] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:54 [I 2022-10-27 16:33:11] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00244 [I 2022-10-27 16:33:11] ...trap/engines/engine.py.230: loss: 0.11297 [I 2022-10-27 16:33:11] ...optimizers/trijoint.py.151: total_norm: 13.082093 [I 2022-10-27 16:33:17] ...trap/engines/engine.py.224: train: epoch 77 | batch 400/2304 [I 2022-10-27 16:33:17] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:38 [I 2022-10-27 16:33:17] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00245 [I 2022-10-27 16:33:17] ...trap/engines/engine.py.230: loss: 0.10904 [I 2022-10-27 16:33:17] ...optimizers/trijoint.py.151: total_norm: 8.947794 [I 2022-10-27 16:33:22] ...trap/engines/engine.py.224: train: epoch 77 | batch 410/2304 [I 2022-10-27 16:33:22] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:26 [I 2022-10-27 16:33:22] ...trap/engines/engine.py.229: process: 0.54622 | load: 0.00245 [I 2022-10-27 16:33:22] ...trap/engines/engine.py.230: loss: 0.18811 [I 2022-10-27 16:33:22] ...optimizers/trijoint.py.151: total_norm: 21.166716 [I 2022-10-27 16:33:28] ...trap/engines/engine.py.224: train: epoch 77 | batch 420/2304 [I 2022-10-27 16:33:28] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:22 [I 2022-10-27 16:33:28] ...trap/engines/engine.py.229: process: 0.54735 | load: 0.00241 [I 2022-10-27 16:33:28] ...trap/engines/engine.py.230: loss: 0.10519 [I 2022-10-27 16:33:28] ...optimizers/trijoint.py.151: total_norm: 10.854897 [I 2022-10-27 16:33:33] ...trap/engines/engine.py.224: train: epoch 77 | batch 430/2304 [I 2022-10-27 16:33:33] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:27 [I 2022-10-27 16:33:33] ...trap/engines/engine.py.229: process: 0.55696 | load: 0.00251 [I 2022-10-27 16:33:33] ...trap/engines/engine.py.230: loss: 0.14185 [I 2022-10-27 16:33:33] ...optimizers/trijoint.py.151: total_norm: 16.957676 [I 2022-10-27 16:33:39] ...trap/engines/engine.py.224: train: epoch 77 | batch 440/2304 [I 2022-10-27 16:33:39] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:27 [I 2022-10-27 16:33:39] ...trap/engines/engine.py.229: process: 0.54733 | load: 0.00253 [I 2022-10-27 16:33:39] ...trap/engines/engine.py.230: loss: 0.11226 [I 2022-10-27 16:33:39] ...optimizers/trijoint.py.151: total_norm: 12.384326 [I 2022-10-27 16:33:45] ...trap/engines/engine.py.224: train: epoch 77 | batch 450/2304 [I 2022-10-27 16:33:45] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:06 [I 2022-10-27 16:33:45] ...trap/engines/engine.py.229: process: 0.54764 | load: 0.00251 [I 2022-10-27 16:33:45] ...trap/engines/engine.py.230: loss: 0.16255 [I 2022-10-27 16:33:45] ...optimizers/trijoint.py.151: total_norm: 53.763367 [I 2022-10-27 16:33:50] ...trap/engines/engine.py.224: train: epoch 77 | batch 460/2304 [I 2022-10-27 16:33:50] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:04 [I 2022-10-27 16:33:50] ...trap/engines/engine.py.229: process: 0.55767 | load: 0.00251 [I 2022-10-27 16:33:50] ...trap/engines/engine.py.230: loss: 0.10168 [I 2022-10-27 16:33:50] ...optimizers/trijoint.py.151: total_norm: 8.764250 [I 2022-10-27 16:33:56] ...trap/engines/engine.py.224: train: epoch 77 | batch 470/2304 [I 2022-10-27 16:33:56] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:59 [I 2022-10-27 16:33:56] ...trap/engines/engine.py.229: process: 0.56335 | load: 0.00255 [I 2022-10-27 16:33:56] ...trap/engines/engine.py.230: loss: 0.16091 [I 2022-10-27 16:33:56] ...optimizers/trijoint.py.151: total_norm: 24.307894 [I 2022-10-27 16:34:01] ...trap/engines/engine.py.224: train: epoch 77 | batch 480/2304 [I 2022-10-27 16:34:01] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:53 [I 2022-10-27 16:34:01] ...trap/engines/engine.py.229: process: 0.55848 | load: 0.00244 [I 2022-10-27 16:34:01] ...trap/engines/engine.py.230: loss: 0.14364 [I 2022-10-27 16:34:01] ...optimizers/trijoint.py.151: total_norm: 13.594086 [I 2022-10-27 16:34:07] ...trap/engines/engine.py.224: train: epoch 77 | batch 490/2304 [I 2022-10-27 16:34:07] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:48 [I 2022-10-27 16:34:07] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00241 [I 2022-10-27 16:34:07] ...trap/engines/engine.py.230: loss: 0.11071 [I 2022-10-27 16:34:07] ...optimizers/trijoint.py.151: total_norm: 29.259155 [I 2022-10-27 16:34:12] ...trap/engines/engine.py.224: train: epoch 77 | batch 500/2304 [I 2022-10-27 16:34:12] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:39 [I 2022-10-27 16:34:12] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00246 [I 2022-10-27 16:34:12] ...trap/engines/engine.py.230: loss: 0.08734 [I 2022-10-27 16:34:12] ...optimizers/trijoint.py.151: total_norm: 20.635056 [I 2022-10-27 16:34:18] ...trap/engines/engine.py.224: train: epoch 77 | batch 510/2304 [I 2022-10-27 16:34:18] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:37 [I 2022-10-27 16:34:18] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00254 [I 2022-10-27 16:34:18] ...trap/engines/engine.py.230: loss: 0.18254 [I 2022-10-27 16:34:18] ...optimizers/trijoint.py.151: total_norm: 12.396977 [I 2022-10-27 16:34:24] ...trap/engines/engine.py.224: train: epoch 77 | batch 520/2304 [I 2022-10-27 16:34:24] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:36 [I 2022-10-27 16:34:24] ...trap/engines/engine.py.229: process: 0.56836 | load: 0.00248 [I 2022-10-27 16:34:24] ...trap/engines/engine.py.230: loss: 0.12525 [I 2022-10-27 16:34:24] ...optimizers/trijoint.py.151: total_norm: 8.598969 [I 2022-10-27 16:34:29] ...trap/engines/engine.py.224: train: epoch 77 | batch 530/2304 [I 2022-10-27 16:34:29] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:23 [I 2022-10-27 16:34:29] ...trap/engines/engine.py.229: process: 0.55770 | load: 0.00251 [I 2022-10-27 16:34:29] ...trap/engines/engine.py.230: loss: 0.10850 [I 2022-10-27 16:34:29] ...optimizers/trijoint.py.151: total_norm: 10.833793 [I 2022-10-27 16:34:35] ...trap/engines/engine.py.224: train: epoch 77 | batch 540/2304 [I 2022-10-27 16:34:35] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:23 [I 2022-10-27 16:34:35] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00246 [I 2022-10-27 16:34:35] ...trap/engines/engine.py.230: loss: 0.12438 [I 2022-10-27 16:34:35] ...optimizers/trijoint.py.151: total_norm: 9.301351 [I 2022-10-27 16:34:40] ...trap/engines/engine.py.224: train: epoch 77 | batch 550/2304 [I 2022-10-27 16:34:40] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:10 [I 2022-10-27 16:34:40] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00269 [I 2022-10-27 16:34:40] ...trap/engines/engine.py.230: loss: 0.16710 [I 2022-10-27 16:34:40] ...optimizers/trijoint.py.151: total_norm: 40.122036 [I 2022-10-27 16:34:46] ...trap/engines/engine.py.224: train: epoch 77 | batch 560/2304 [I 2022-10-27 16:34:46] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:13 [I 2022-10-27 16:34:46] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00281 [I 2022-10-27 16:34:46] ...trap/engines/engine.py.230: loss: 0.12577 [I 2022-10-27 16:34:46] ...optimizers/trijoint.py.151: total_norm: 16.471418 [I 2022-10-27 16:34:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 570/2304 [I 2022-10-27 16:34:51] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:10 [I 2022-10-27 16:34:51] ...trap/engines/engine.py.229: process: 0.56224 | load: 0.00245 [I 2022-10-27 16:34:51] ...trap/engines/engine.py.230: loss: 0.14273 [I 2022-10-27 16:34:51] ...optimizers/trijoint.py.151: total_norm: 14.175329 [I 2022-10-27 16:34:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 580/2304 [I 2022-10-27 16:34:57] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:04 [I 2022-10-27 16:34:57] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00246 [I 2022-10-27 16:34:57] ...trap/engines/engine.py.230: loss: 0.11904 [I 2022-10-27 16:34:57] ...optimizers/trijoint.py.151: total_norm: 39.656342 [I 2022-10-27 16:35:03] ...trap/engines/engine.py.224: train: epoch 77 | batch 590/2304 [I 2022-10-27 16:35:03] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:51 [I 2022-10-27 16:35:03] ...trap/engines/engine.py.229: process: 0.55221 | load: 0.00261 [I 2022-10-27 16:35:03] ...trap/engines/engine.py.230: loss: 0.15083 [I 2022-10-27 16:35:03] ...optimizers/trijoint.py.151: total_norm: 20.435263 [I 2022-10-27 16:35:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 600/2304 [I 2022-10-27 16:35:08] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:48 [I 2022-10-27 16:35:08] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00243 [I 2022-10-27 16:35:08] ...trap/engines/engine.py.230: loss: 0.12557 [I 2022-10-27 16:35:08] ...optimizers/trijoint.py.151: total_norm: 11.822042 [I 2022-10-27 16:35:14] ...trap/engines/engine.py.224: train: epoch 77 | batch 610/2304 [I 2022-10-27 16:35:14] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:38 [I 2022-10-27 16:35:14] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00244 [I 2022-10-27 16:35:14] ...trap/engines/engine.py.230: loss: 0.11189 [I 2022-10-27 16:35:14] ...optimizers/trijoint.py.151: total_norm: 13.712176 [I 2022-10-27 16:35:19] ...trap/engines/engine.py.224: train: epoch 77 | batch 620/2304 [I 2022-10-27 16:35:19] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:34 [I 2022-10-27 16:35:19] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00244 [I 2022-10-27 16:35:19] ...trap/engines/engine.py.230: loss: 0.14725 [I 2022-10-27 16:35:19] ...optimizers/trijoint.py.151: total_norm: 21.351597 [I 2022-10-27 16:35:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 630/2304 [I 2022-10-27 16:35:25] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:34 [I 2022-10-27 16:35:25] ...trap/engines/engine.py.229: process: 0.56274 | load: 0.00255 [I 2022-10-27 16:35:25] ...trap/engines/engine.py.230: loss: 0.14631 [I 2022-10-27 16:35:25] ...optimizers/trijoint.py.151: total_norm: 44.115074 [I 2022-10-27 16:35:31] ...trap/engines/engine.py.224: train: epoch 77 | batch 640/2304 [I 2022-10-27 16:35:31] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:28 [I 2022-10-27 16:35:31] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00249 [I 2022-10-27 16:35:31] ...trap/engines/engine.py.230: loss: 0.13838 [I 2022-10-27 16:35:31] ...optimizers/trijoint.py.151: total_norm: 22.127195 [I 2022-10-27 16:35:36] ...trap/engines/engine.py.224: train: epoch 77 | batch 650/2304 [I 2022-10-27 16:35:36] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:16 [I 2022-10-27 16:35:36] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00246 [I 2022-10-27 16:35:36] ...trap/engines/engine.py.230: loss: 0.10653 [I 2022-10-27 16:35:36] ...optimizers/trijoint.py.151: total_norm: 20.477207 [I 2022-10-27 16:35:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 660/2304 [I 2022-10-27 16:35:42] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:12 [I 2022-10-27 16:35:42] ...trap/engines/engine.py.229: process: 0.54855 | load: 0.00246 [I 2022-10-27 16:35:42] ...trap/engines/engine.py.230: loss: 0.13582 [I 2022-10-27 16:35:42] ...optimizers/trijoint.py.151: total_norm: 14.398471 [I 2022-10-27 16:35:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 670/2304 [I 2022-10-27 16:35:47] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:04 [I 2022-10-27 16:35:47] ...trap/engines/engine.py.229: process: 0.56412 | load: 0.00244 [I 2022-10-27 16:35:47] ...trap/engines/engine.py.230: loss: 0.10216 [I 2022-10-27 16:35:47] ...optimizers/trijoint.py.151: total_norm: 13.304993 [I 2022-10-27 16:35:53] ...trap/engines/engine.py.224: train: epoch 77 | batch 680/2304 [I 2022-10-27 16:35:53] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:01 [I 2022-10-27 16:35:53] ...trap/engines/engine.py.229: process: 0.55028 | load: 0.00247 [I 2022-10-27 16:35:53] ...trap/engines/engine.py.230: loss: 0.18260 [I 2022-10-27 16:35:53] ...optimizers/trijoint.py.151: total_norm: 41.492195 [I 2022-10-27 16:35:58] ...trap/engines/engine.py.224: train: epoch 77 | batch 690/2304 [I 2022-10-27 16:35:58] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:04 [I 2022-10-27 16:35:58] ...trap/engines/engine.py.229: process: 0.56997 | load: 0.00243 [I 2022-10-27 16:35:58] ...trap/engines/engine.py.230: loss: 0.09959 [I 2022-10-27 16:35:58] ...optimizers/trijoint.py.151: total_norm: 16.384182 [I 2022-10-27 16:36:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 700/2304 [I 2022-10-27 16:36:04] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:54 [I 2022-10-27 16:36:04] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00269 [I 2022-10-27 16:36:04] ...trap/engines/engine.py.230: loss: 0.15072 [I 2022-10-27 16:36:04] ...optimizers/trijoint.py.151: total_norm: 28.324570 [I 2022-10-27 16:36:09] ...trap/engines/engine.py.224: train: epoch 77 | batch 710/2304 [I 2022-10-27 16:36:09] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:47 [I 2022-10-27 16:36:09] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00249 [I 2022-10-27 16:36:09] ...trap/engines/engine.py.230: loss: 0.09831 [I 2022-10-27 16:36:09] ...optimizers/trijoint.py.151: total_norm: 10.798450 [I 2022-10-27 16:36:15] ...trap/engines/engine.py.224: train: epoch 77 | batch 720/2304 [I 2022-10-27 16:36:15] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:52 [I 2022-10-27 16:36:15] ...trap/engines/engine.py.229: process: 0.55490 | load: 0.00271 [I 2022-10-27 16:36:15] ...trap/engines/engine.py.230: loss: 0.14933 [I 2022-10-27 16:36:15] ...optimizers/trijoint.py.151: total_norm: 21.255770 [I 2022-10-27 16:36:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 730/2304 [I 2022-10-27 16:36:21] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:15:03 [I 2022-10-27 16:36:21] ...trap/engines/engine.py.229: process: 0.60255 | load: 0.00297 [I 2022-10-27 16:36:21] ...trap/engines/engine.py.230: loss: 0.13911 [I 2022-10-27 16:36:21] ...optimizers/trijoint.py.151: total_norm: 13.438700 [I 2022-10-27 16:36:26] ...trap/engines/engine.py.224: train: epoch 77 | batch 740/2304 [I 2022-10-27 16:36:26] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:32 [I 2022-10-27 16:36:26] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00246 [I 2022-10-27 16:36:26] ...trap/engines/engine.py.230: loss: 0.12598 [I 2022-10-27 16:36:26] ...optimizers/trijoint.py.151: total_norm: 13.420693 [I 2022-10-27 16:36:32] ...trap/engines/engine.py.224: train: epoch 77 | batch 750/2304 [I 2022-10-27 16:36:32] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:24 [I 2022-10-27 16:36:32] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00243 [I 2022-10-27 16:36:32] ...trap/engines/engine.py.230: loss: 0.14409 [I 2022-10-27 16:36:32] ...optimizers/trijoint.py.151: total_norm: 38.549717 [I 2022-10-27 16:36:37] ...trap/engines/engine.py.224: train: epoch 77 | batch 760/2304 [I 2022-10-27 16:36:37] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:14 [I 2022-10-27 16:36:37] ...trap/engines/engine.py.229: process: 0.54736 | load: 0.00244 [I 2022-10-27 16:36:37] ...trap/engines/engine.py.230: loss: 0.11724 [I 2022-10-27 16:36:37] ...optimizers/trijoint.py.151: total_norm: 40.934822 [I 2022-10-27 16:36:43] ...trap/engines/engine.py.224: train: epoch 77 | batch 770/2304 [I 2022-10-27 16:36:43] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:08 [I 2022-10-27 16:36:43] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00243 [I 2022-10-27 16:36:43] ...trap/engines/engine.py.230: loss: 0.23440 [I 2022-10-27 16:36:43] ...optimizers/trijoint.py.151: total_norm: 22.664701 [I 2022-10-27 16:36:49] ...trap/engines/engine.py.224: train: epoch 77 | batch 780/2304 [I 2022-10-27 16:36:49] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:03 [I 2022-10-27 16:36:49] ...trap/engines/engine.py.229: process: 0.55110 | load: 0.00244 [I 2022-10-27 16:36:49] ...trap/engines/engine.py.230: loss: 0.17109 [I 2022-10-27 16:36:49] ...optimizers/trijoint.py.151: total_norm: 24.208120 [I 2022-10-27 16:36:54] ...trap/engines/engine.py.224: train: epoch 77 | batch 790/2304 [I 2022-10-27 16:36:54] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:00 [I 2022-10-27 16:36:54] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00242 [I 2022-10-27 16:36:54] ...trap/engines/engine.py.230: loss: 0.10473 [I 2022-10-27 16:36:54] ...optimizers/trijoint.py.151: total_norm: 8.790492 [I 2022-10-27 16:37:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 800/2304 [I 2022-10-27 16:37:00] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:58 [I 2022-10-27 16:37:00] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00242 [I 2022-10-27 16:37:00] ...trap/engines/engine.py.230: loss: 0.15682 [I 2022-10-27 16:37:00] ...optimizers/trijoint.py.151: total_norm: 27.828979 [I 2022-10-27 16:37:05] ...trap/engines/engine.py.224: train: epoch 77 | batch 810/2304 [I 2022-10-27 16:37:05] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:58 [I 2022-10-27 16:37:05] ...trap/engines/engine.py.229: process: 0.56423 | load: 0.00242 [I 2022-10-27 16:37:05] ...trap/engines/engine.py.230: loss: 0.18514 [I 2022-10-27 16:37:05] ...optimizers/trijoint.py.151: total_norm: 20.132576 [I 2022-10-27 16:37:11] ...trap/engines/engine.py.224: train: epoch 77 | batch 820/2304 [I 2022-10-27 16:37:11] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:46 [I 2022-10-27 16:37:11] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00244 [I 2022-10-27 16:37:11] ...trap/engines/engine.py.230: loss: 0.11338 [I 2022-10-27 16:37:11] ...optimizers/trijoint.py.151: total_norm: 18.279036 [I 2022-10-27 16:37:17] ...trap/engines/engine.py.224: train: epoch 77 | batch 830/2304 [I 2022-10-27 16:37:17] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:35 [I 2022-10-27 16:37:17] ...trap/engines/engine.py.229: process: 0.55315 | load: 0.00250 [I 2022-10-27 16:37:17] ...trap/engines/engine.py.230: loss: 0.14719 [I 2022-10-27 16:37:17] ...optimizers/trijoint.py.151: total_norm: 20.030352 [I 2022-10-27 16:37:22] ...trap/engines/engine.py.224: train: epoch 77 | batch 840/2304 [I 2022-10-27 16:37:22] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:33 [I 2022-10-27 16:37:22] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00248 [I 2022-10-27 16:37:22] ...trap/engines/engine.py.230: loss: 0.17033 [I 2022-10-27 16:37:22] ...optimizers/trijoint.py.151: total_norm: 25.135900 [I 2022-10-27 16:37:28] ...trap/engines/engine.py.224: train: epoch 77 | batch 850/2304 [I 2022-10-27 16:37:28] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:26 [I 2022-10-27 16:37:28] ...trap/engines/engine.py.229: process: 0.54577 | load: 0.00245 [I 2022-10-27 16:37:28] ...trap/engines/engine.py.230: loss: 0.09041 [I 2022-10-27 16:37:28] ...optimizers/trijoint.py.151: total_norm: 7.256674 [I 2022-10-27 16:37:33] ...trap/engines/engine.py.224: train: epoch 77 | batch 860/2304 [I 2022-10-27 16:37:33] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:17 [I 2022-10-27 16:37:33] ...trap/engines/engine.py.229: process: 0.54511 | load: 0.00244 [I 2022-10-27 16:37:33] ...trap/engines/engine.py.230: loss: 0.10798 [I 2022-10-27 16:37:33] ...optimizers/trijoint.py.151: total_norm: 10.257992 [I 2022-10-27 16:37:39] ...trap/engines/engine.py.224: train: epoch 77 | batch 870/2304 [I 2022-10-27 16:37:39] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:10 [I 2022-10-27 16:37:39] ...trap/engines/engine.py.229: process: 0.55831 | load: 0.00246 [I 2022-10-27 16:37:39] ...trap/engines/engine.py.230: loss: 0.14937 [I 2022-10-27 16:37:39] ...optimizers/trijoint.py.151: total_norm: 19.964104 [I 2022-10-27 16:37:44] ...trap/engines/engine.py.224: train: epoch 77 | batch 880/2304 [I 2022-10-27 16:37:44] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:18 [I 2022-10-27 16:37:44] ...trap/engines/engine.py.229: process: 0.55505 | load: 0.00247 [I 2022-10-27 16:37:44] ...trap/engines/engine.py.230: loss: 0.10560 [I 2022-10-27 16:37:44] ...optimizers/trijoint.py.151: total_norm: 18.460150 [I 2022-10-27 16:37:50] ...trap/engines/engine.py.224: train: epoch 77 | batch 890/2304 [I 2022-10-27 16:37:50] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:02 [I 2022-10-27 16:37:50] ...trap/engines/engine.py.229: process: 0.54572 | load: 0.00250 [I 2022-10-27 16:37:50] ...trap/engines/engine.py.230: loss: 0.11357 [I 2022-10-27 16:37:50] ...optimizers/trijoint.py.151: total_norm: 22.097033 [I 2022-10-27 16:37:56] ...trap/engines/engine.py.224: train: epoch 77 | batch 900/2304 [I 2022-10-27 16:37:56] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:05 [I 2022-10-27 16:37:56] ...trap/engines/engine.py.229: process: 0.54751 | load: 0.00250 [I 2022-10-27 16:37:56] ...trap/engines/engine.py.230: loss: 0.12289 [I 2022-10-27 16:37:56] ...optimizers/trijoint.py.151: total_norm: 15.104118 [I 2022-10-27 16:38:01] ...trap/engines/engine.py.224: train: epoch 77 | batch 910/2304 [I 2022-10-27 16:38:01] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:56 [I 2022-10-27 16:38:01] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00252 [I 2022-10-27 16:38:01] ...trap/engines/engine.py.230: loss: 0.11742 [I 2022-10-27 16:38:01] ...optimizers/trijoint.py.151: total_norm: 9.950334 [I 2022-10-27 16:38:07] ...trap/engines/engine.py.224: train: epoch 77 | batch 920/2304 [I 2022-10-27 16:38:07] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:56 [I 2022-10-27 16:38:07] ...trap/engines/engine.py.229: process: 0.56874 | load: 0.00243 [I 2022-10-27 16:38:07] ...trap/engines/engine.py.230: loss: 0.17098 [I 2022-10-27 16:38:07] ...optimizers/trijoint.py.151: total_norm: 48.046658 [I 2022-10-27 16:38:12] ...trap/engines/engine.py.224: train: epoch 77 | batch 930/2304 [I 2022-10-27 16:38:12] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:43 [I 2022-10-27 16:38:12] ...trap/engines/engine.py.229: process: 0.54924 | load: 0.00245 [I 2022-10-27 16:38:12] ...trap/engines/engine.py.230: loss: 0.14813 [I 2022-10-27 16:38:12] ...optimizers/trijoint.py.151: total_norm: 16.709026 [I 2022-10-27 16:38:18] ...trap/engines/engine.py.224: train: epoch 77 | batch 940/2304 [I 2022-10-27 16:38:18] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:34 [I 2022-10-27 16:38:18] ...trap/engines/engine.py.229: process: 0.56308 | load: 0.00244 [I 2022-10-27 16:38:18] ...trap/engines/engine.py.230: loss: 0.10456 [I 2022-10-27 16:38:18] ...optimizers/trijoint.py.151: total_norm: 12.600319 [I 2022-10-27 16:38:23] ...trap/engines/engine.py.224: train: epoch 77 | batch 950/2304 [I 2022-10-27 16:38:23] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:35 [I 2022-10-27 16:38:23] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00246 [I 2022-10-27 16:38:23] ...trap/engines/engine.py.230: loss: 0.21936 [I 2022-10-27 16:38:23] ...optimizers/trijoint.py.151: total_norm: 53.099289 [I 2022-10-27 16:38:29] ...trap/engines/engine.py.224: train: epoch 77 | batch 960/2304 [I 2022-10-27 16:38:29] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:31 [I 2022-10-27 16:38:29] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00245 [I 2022-10-27 16:38:29] ...trap/engines/engine.py.230: loss: 0.20278 [I 2022-10-27 16:38:29] ...optimizers/trijoint.py.151: total_norm: 17.101303 [I 2022-10-27 16:38:35] ...trap/engines/engine.py.224: train: epoch 77 | batch 970/2304 [I 2022-10-27 16:38:35] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:20 [I 2022-10-27 16:38:35] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00242 [I 2022-10-27 16:38:35] ...trap/engines/engine.py.230: loss: 0.14982 [I 2022-10-27 16:38:35] ...optimizers/trijoint.py.151: total_norm: 23.503147 [I 2022-10-27 16:38:40] ...trap/engines/engine.py.224: train: epoch 77 | batch 980/2304 [I 2022-10-27 16:38:40] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:13 [I 2022-10-27 16:38:40] ...trap/engines/engine.py.229: process: 0.54624 | load: 0.00242 [I 2022-10-27 16:38:40] ...trap/engines/engine.py.230: loss: 0.10724 [I 2022-10-27 16:38:40] ...optimizers/trijoint.py.151: total_norm: 10.347700 [I 2022-10-27 16:38:46] ...trap/engines/engine.py.224: train: epoch 77 | batch 990/2304 [I 2022-10-27 16:38:46] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:17 [I 2022-10-27 16:38:46] ...trap/engines/engine.py.229: process: 0.55297 | load: 0.00247 [I 2022-10-27 16:38:46] ...trap/engines/engine.py.230: loss: 0.12477 [I 2022-10-27 16:38:46] ...optimizers/trijoint.py.151: total_norm: 22.844006 [I 2022-10-27 16:38:52] ...trap/engines/engine.py.224: train: epoch 77 | batch 1000/2304 [I 2022-10-27 16:38:52] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:09 [I 2022-10-27 16:38:52] ...trap/engines/engine.py.229: process: 0.53938 | load: 0.00250 [I 2022-10-27 16:38:52] ...trap/engines/engine.py.230: loss: 0.15145 [I 2022-10-27 16:38:52] ...optimizers/trijoint.py.151: total_norm: 15.332922 [I 2022-10-27 16:38:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 1010/2304 [I 2022-10-27 16:38:57] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:05 [I 2022-10-27 16:38:57] ...trap/engines/engine.py.229: process: 0.56406 | load: 0.00246 [I 2022-10-27 16:38:57] ...trap/engines/engine.py.230: loss: 0.12202 [I 2022-10-27 16:38:57] ...optimizers/trijoint.py.151: total_norm: 20.905989 [I 2022-10-27 16:39:03] ...trap/engines/engine.py.224: train: epoch 77 | batch 1020/2304 [I 2022-10-27 16:39:03] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:56 [I 2022-10-27 16:39:03] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00244 [I 2022-10-27 16:39:03] ...trap/engines/engine.py.230: loss: 0.09309 [I 2022-10-27 16:39:03] ...optimizers/trijoint.py.151: total_norm: 9.030542 [I 2022-10-27 16:39:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 1030/2304 [I 2022-10-27 16:39:08] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:49 [I 2022-10-27 16:39:08] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00276 [I 2022-10-27 16:39:08] ...trap/engines/engine.py.230: loss: 0.14018 [I 2022-10-27 16:39:08] ...optimizers/trijoint.py.151: total_norm: 17.421288 [I 2022-10-27 16:39:14] ...trap/engines/engine.py.224: train: epoch 77 | batch 1040/2304 [I 2022-10-27 16:39:14] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:43 [I 2022-10-27 16:39:14] ...trap/engines/engine.py.229: process: 0.54968 | load: 0.00242 [I 2022-10-27 16:39:14] ...trap/engines/engine.py.230: loss: 0.11379 [I 2022-10-27 16:39:14] ...optimizers/trijoint.py.151: total_norm: 15.253384 [I 2022-10-27 16:39:19] ...trap/engines/engine.py.224: train: epoch 77 | batch 1050/2304 [I 2022-10-27 16:39:19] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:37 [I 2022-10-27 16:39:19] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00243 [I 2022-10-27 16:39:19] ...trap/engines/engine.py.230: loss: 0.19088 [I 2022-10-27 16:39:19] ...optimizers/trijoint.py.151: total_norm: 12.150771 [I 2022-10-27 16:39:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 1060/2304 [I 2022-10-27 16:39:25] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:30 [I 2022-10-27 16:39:25] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00244 [I 2022-10-27 16:39:25] ...trap/engines/engine.py.230: loss: 0.11376 [I 2022-10-27 16:39:25] ...optimizers/trijoint.py.151: total_norm: 9.872311 [I 2022-10-27 16:39:31] ...trap/engines/engine.py.224: train: epoch 77 | batch 1070/2304 [I 2022-10-27 16:39:31] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:22 [I 2022-10-27 16:39:31] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00242 [I 2022-10-27 16:39:31] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-27 16:39:31] ...optimizers/trijoint.py.151: total_norm: 37.286243 [I 2022-10-27 16:39:36] ...trap/engines/engine.py.224: train: epoch 77 | batch 1080/2304 [I 2022-10-27 16:39:36] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:20 [I 2022-10-27 16:39:36] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00242 [I 2022-10-27 16:39:36] ...trap/engines/engine.py.230: loss: 0.14142 [I 2022-10-27 16:39:36] ...optimizers/trijoint.py.151: total_norm: 13.799237 [I 2022-10-27 16:39:42] ...trap/engines/engine.py.224: train: epoch 77 | batch 1090/2304 [I 2022-10-27 16:39:42] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:13 [I 2022-10-27 16:39:42] ...trap/engines/engine.py.229: process: 0.56141 | load: 0.00244 [I 2022-10-27 16:39:42] ...trap/engines/engine.py.230: loss: 0.12214 [I 2022-10-27 16:39:42] ...optimizers/trijoint.py.151: total_norm: 8.715775 [I 2022-10-27 16:39:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 1100/2304 [I 2022-10-27 16:39:47] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:10 [I 2022-10-27 16:39:47] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00242 [I 2022-10-27 16:39:47] ...trap/engines/engine.py.230: loss: 0.12572 [I 2022-10-27 16:39:47] ...optimizers/trijoint.py.151: total_norm: 21.289856 [I 2022-10-27 16:39:53] ...trap/engines/engine.py.224: train: epoch 77 | batch 1110/2304 [I 2022-10-27 16:39:53] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:08 [I 2022-10-27 16:39:53] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00247 [I 2022-10-27 16:39:53] ...trap/engines/engine.py.230: loss: 0.14946 [I 2022-10-27 16:39:53] ...optimizers/trijoint.py.151: total_norm: 11.610672 [I 2022-10-27 16:39:59] ...trap/engines/engine.py.224: train: epoch 77 | batch 1120/2304 [I 2022-10-27 16:39:59] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:01 [I 2022-10-27 16:39:59] ...trap/engines/engine.py.229: process: 0.55828 | load: 0.00245 [I 2022-10-27 16:39:59] ...trap/engines/engine.py.230: loss: 0.17619 [I 2022-10-27 16:39:59] ...optimizers/trijoint.py.151: total_norm: 11.666916 [I 2022-10-27 16:40:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 1130/2304 [I 2022-10-27 16:40:04] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:49 [I 2022-10-27 16:40:04] ...trap/engines/engine.py.229: process: 0.54871 | load: 0.00245 [I 2022-10-27 16:40:04] ...trap/engines/engine.py.230: loss: 0.15187 [I 2022-10-27 16:40:04] ...optimizers/trijoint.py.151: total_norm: 62.367207 [I 2022-10-27 16:40:10] ...trap/engines/engine.py.224: train: epoch 77 | batch 1140/2304 [I 2022-10-27 16:40:10] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:44 [I 2022-10-27 16:40:10] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00243 [I 2022-10-27 16:40:10] ...trap/engines/engine.py.230: loss: 0.12115 [I 2022-10-27 16:40:10] ...optimizers/trijoint.py.151: total_norm: 15.828326 [I 2022-10-27 16:40:15] ...trap/engines/engine.py.224: train: epoch 77 | batch 1150/2304 [I 2022-10-27 16:40:15] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:10:41 [I 2022-10-27 16:40:15] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00243 [I 2022-10-27 16:40:15] ...trap/engines/engine.py.230: loss: 0.13834 [I 2022-10-27 16:40:15] ...optimizers/trijoint.py.151: total_norm: 14.802365 [I 2022-10-27 16:40:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 1160/2304 [I 2022-10-27 16:40:21] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:35 [I 2022-10-27 16:40:21] ...trap/engines/engine.py.229: process: 0.54346 | load: 0.00245 [I 2022-10-27 16:40:21] ...trap/engines/engine.py.230: loss: 0.15982 [I 2022-10-27 16:40:21] ...optimizers/trijoint.py.151: total_norm: 23.297316 [I 2022-10-27 16:40:26] ...trap/engines/engine.py.224: train: epoch 77 | batch 1170/2304 [I 2022-10-27 16:40:26] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:26 [I 2022-10-27 16:40:26] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00242 [I 2022-10-27 16:40:26] ...trap/engines/engine.py.230: loss: 0.10748 [I 2022-10-27 16:40:26] ...optimizers/trijoint.py.151: total_norm: 12.780355 [I 2022-10-27 16:40:32] ...trap/engines/engine.py.224: train: epoch 77 | batch 1180/2304 [I 2022-10-27 16:40:32] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:26 [I 2022-10-27 16:40:32] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00264 [I 2022-10-27 16:40:32] ...trap/engines/engine.py.230: loss: 0.13363 [I 2022-10-27 16:40:32] ...optimizers/trijoint.py.151: total_norm: 17.509256 [I 2022-10-27 16:40:37] ...trap/engines/engine.py.224: train: epoch 77 | batch 1190/2304 [I 2022-10-27 16:40:37] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:16 [I 2022-10-27 16:40:37] ...trap/engines/engine.py.229: process: 0.54726 | load: 0.00243 [I 2022-10-27 16:40:37] ...trap/engines/engine.py.230: loss: 0.14096 [I 2022-10-27 16:40:37] ...optimizers/trijoint.py.151: total_norm: 11.170357 [I 2022-10-27 16:40:43] ...trap/engines/engine.py.224: train: epoch 77 | batch 1200/2304 [I 2022-10-27 16:40:43] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:13 [I 2022-10-27 16:40:43] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00241 [I 2022-10-27 16:40:43] ...trap/engines/engine.py.230: loss: 0.10144 [I 2022-10-27 16:40:43] ...optimizers/trijoint.py.151: total_norm: 14.536001 [I 2022-10-27 16:40:49] ...trap/engines/engine.py.224: train: epoch 77 | batch 1210/2304 [I 2022-10-27 16:40:49] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:03 [I 2022-10-27 16:40:49] ...trap/engines/engine.py.229: process: 0.54508 | load: 0.00241 [I 2022-10-27 16:40:49] ...trap/engines/engine.py.230: loss: 0.09036 [I 2022-10-27 16:40:49] ...optimizers/trijoint.py.151: total_norm: 7.533720 [I 2022-10-27 16:40:54] ...trap/engines/engine.py.224: train: epoch 77 | batch 1220/2304 [I 2022-10-27 16:40:54] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:09:57 [I 2022-10-27 16:40:54] ...trap/engines/engine.py.229: process: 0.54892 | load: 0.00247 [I 2022-10-27 16:40:54] ...trap/engines/engine.py.230: loss: 0.11040 [I 2022-10-27 16:40:54] ...optimizers/trijoint.py.151: total_norm: 8.519029 [I 2022-10-27 16:41:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 1230/2304 [I 2022-10-27 16:41:00] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:09:57 [I 2022-10-27 16:41:00] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00246 [I 2022-10-27 16:41:00] ...trap/engines/engine.py.230: loss: 0.12004 [I 2022-10-27 16:41:00] ...optimizers/trijoint.py.151: total_norm: 22.026213 [I 2022-10-27 16:41:05] ...trap/engines/engine.py.224: train: epoch 77 | batch 1240/2304 [I 2022-10-27 16:41:05] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:53 [I 2022-10-27 16:41:05] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00244 [I 2022-10-27 16:41:05] ...trap/engines/engine.py.230: loss: 0.12257 [I 2022-10-27 16:41:05] ...optimizers/trijoint.py.151: total_norm: 17.105928 [I 2022-10-27 16:41:11] ...trap/engines/engine.py.224: train: epoch 77 | batch 1250/2304 [I 2022-10-27 16:41:11] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:09:42 [I 2022-10-27 16:41:11] ...trap/engines/engine.py.229: process: 0.54405 | load: 0.00242 [I 2022-10-27 16:41:11] ...trap/engines/engine.py.230: loss: 0.09602 [I 2022-10-27 16:41:11] ...optimizers/trijoint.py.151: total_norm: 10.369014 [I 2022-10-27 16:41:16] ...trap/engines/engine.py.224: train: epoch 77 | batch 1260/2304 [I 2022-10-27 16:41:16] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:42 [I 2022-10-27 16:41:16] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00243 [I 2022-10-27 16:41:16] ...trap/engines/engine.py.230: loss: 0.12534 [I 2022-10-27 16:41:16] ...optimizers/trijoint.py.151: total_norm: 17.496819 [I 2022-10-27 16:41:22] ...trap/engines/engine.py.224: train: epoch 77 | batch 1270/2304 [I 2022-10-27 16:41:22] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:40 [I 2022-10-27 16:41:22] ...trap/engines/engine.py.229: process: 0.56335 | load: 0.00243 [I 2022-10-27 16:41:22] ...trap/engines/engine.py.230: loss: 0.17844 [I 2022-10-27 16:41:22] ...optimizers/trijoint.py.151: total_norm: 36.621662 [I 2022-10-27 16:41:28] ...trap/engines/engine.py.224: train: epoch 77 | batch 1280/2304 [I 2022-10-27 16:41:28] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:27 [I 2022-10-27 16:41:28] ...trap/engines/engine.py.229: process: 0.55079 | load: 0.00243 [I 2022-10-27 16:41:28] ...trap/engines/engine.py.230: loss: 0.16594 [I 2022-10-27 16:41:28] ...optimizers/trijoint.py.151: total_norm: 59.212105 [I 2022-10-27 16:41:33] ...trap/engines/engine.py.224: train: epoch 77 | batch 1290/2304 [I 2022-10-27 16:41:33] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:18 [I 2022-10-27 16:41:33] ...trap/engines/engine.py.229: process: 0.54388 | load: 0.00247 [I 2022-10-27 16:41:33] ...trap/engines/engine.py.230: loss: 0.19520 [I 2022-10-27 16:41:33] ...optimizers/trijoint.py.151: total_norm: 32.396084 [I 2022-10-27 16:41:39] ...trap/engines/engine.py.224: train: epoch 77 | batch 1300/2304 [I 2022-10-27 16:41:39] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:20 [I 2022-10-27 16:41:39] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00243 [I 2022-10-27 16:41:39] ...trap/engines/engine.py.230: loss: 0.12437 [I 2022-10-27 16:41:39] ...optimizers/trijoint.py.151: total_norm: 23.431095 [I 2022-10-27 16:41:44] ...trap/engines/engine.py.224: train: epoch 77 | batch 1310/2304 [I 2022-10-27 16:41:44] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:13 [I 2022-10-27 16:41:44] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00242 [I 2022-10-27 16:41:44] ...trap/engines/engine.py.230: loss: 0.10986 [I 2022-10-27 16:41:44] ...optimizers/trijoint.py.151: total_norm: 79.556686 [I 2022-10-27 16:41:50] ...trap/engines/engine.py.224: train: epoch 77 | batch 1320/2304 [I 2022-10-27 16:41:50] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:08 [I 2022-10-27 16:41:50] ...trap/engines/engine.py.229: process: 0.57466 | load: 0.00247 [I 2022-10-27 16:41:50] ...trap/engines/engine.py.230: loss: 0.12725 [I 2022-10-27 16:41:50] ...optimizers/trijoint.py.151: total_norm: 14.713876 [I 2022-10-27 16:41:55] ...trap/engines/engine.py.224: train: epoch 77 | batch 1330/2304 [I 2022-10-27 16:41:55] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:00 [I 2022-10-27 16:41:55] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00248 [I 2022-10-27 16:41:55] ...trap/engines/engine.py.230: loss: 0.13160 [I 2022-10-27 16:41:55] ...optimizers/trijoint.py.151: total_norm: 23.294493 [I 2022-10-27 16:42:01] ...trap/engines/engine.py.224: train: epoch 77 | batch 1340/2304 [I 2022-10-27 16:42:01] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:09:01 [I 2022-10-27 16:42:01] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00254 [I 2022-10-27 16:42:01] ...trap/engines/engine.py.230: loss: 0.11315 [I 2022-10-27 16:42:01] ...optimizers/trijoint.py.151: total_norm: 13.064936 [I 2022-10-27 16:42:07] ...trap/engines/engine.py.224: train: epoch 77 | batch 1350/2304 [I 2022-10-27 16:42:07] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:48 [I 2022-10-27 16:42:07] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00246 [I 2022-10-27 16:42:07] ...trap/engines/engine.py.230: loss: 0.17899 [I 2022-10-27 16:42:07] ...optimizers/trijoint.py.151: total_norm: 22.244307 [I 2022-10-27 16:42:12] ...trap/engines/engine.py.224: train: epoch 77 | batch 1360/2304 [I 2022-10-27 16:42:12] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:42 [I 2022-10-27 16:42:12] ...trap/engines/engine.py.229: process: 0.55198 | load: 0.00243 [I 2022-10-27 16:42:12] ...trap/engines/engine.py.230: loss: 0.17439 [I 2022-10-27 16:42:12] ...optimizers/trijoint.py.151: total_norm: 17.882902 [I 2022-10-27 16:42:18] ...trap/engines/engine.py.224: train: epoch 77 | batch 1370/2304 [I 2022-10-27 16:42:18] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:39 [I 2022-10-27 16:42:18] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00242 [I 2022-10-27 16:42:18] ...trap/engines/engine.py.230: loss: 0.17760 [I 2022-10-27 16:42:18] ...optimizers/trijoint.py.151: total_norm: 15.053767 [I 2022-10-27 16:42:23] ...trap/engines/engine.py.224: train: epoch 77 | batch 1380/2304 [I 2022-10-27 16:42:23] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:34 [I 2022-10-27 16:42:23] ...trap/engines/engine.py.229: process: 0.57020 | load: 0.00244 [I 2022-10-27 16:42:23] ...trap/engines/engine.py.230: loss: 0.24280 [I 2022-10-27 16:42:23] ...optimizers/trijoint.py.151: total_norm: 23.495285 [I 2022-10-27 16:42:29] ...trap/engines/engine.py.224: train: epoch 77 | batch 1390/2304 [I 2022-10-27 16:42:29] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:31 [I 2022-10-27 16:42:29] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00246 [I 2022-10-27 16:42:29] ...trap/engines/engine.py.230: loss: 0.12419 [I 2022-10-27 16:42:29] ...optimizers/trijoint.py.151: total_norm: 13.109414 [I 2022-10-27 16:42:35] ...trap/engines/engine.py.224: train: epoch 77 | batch 1400/2304 [I 2022-10-27 16:42:35] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:21 [I 2022-10-27 16:42:35] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00244 [I 2022-10-27 16:42:35] ...trap/engines/engine.py.230: loss: 0.14465 [I 2022-10-27 16:42:35] ...optimizers/trijoint.py.151: total_norm: 21.667824 [I 2022-10-27 16:42:40] ...trap/engines/engine.py.224: train: epoch 77 | batch 1410/2304 [I 2022-10-27 16:42:40] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:17 [I 2022-10-27 16:42:40] ...trap/engines/engine.py.229: process: 0.56326 | load: 0.00242 [I 2022-10-27 16:42:40] ...trap/engines/engine.py.230: loss: 0.11938 [I 2022-10-27 16:42:40] ...optimizers/trijoint.py.151: total_norm: 16.525650 [I 2022-10-27 16:42:46] ...trap/engines/engine.py.224: train: epoch 77 | batch 1420/2304 [I 2022-10-27 16:42:46] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:11 [I 2022-10-27 16:42:46] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00242 [I 2022-10-27 16:42:46] ...trap/engines/engine.py.230: loss: 0.11198 [I 2022-10-27 16:42:46] ...optimizers/trijoint.py.151: total_norm: 9.099976 [I 2022-10-27 16:42:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 1430/2304 [I 2022-10-27 16:42:51] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:05 [I 2022-10-27 16:42:51] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00239 [I 2022-10-27 16:42:51] ...trap/engines/engine.py.230: loss: 0.14285 [I 2022-10-27 16:42:51] ...optimizers/trijoint.py.151: total_norm: 16.897404 [I 2022-10-27 16:42:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 1440/2304 [I 2022-10-27 16:42:57] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:07:59 [I 2022-10-27 16:42:57] ...trap/engines/engine.py.229: process: 0.54747 | load: 0.00242 [I 2022-10-27 16:42:57] ...trap/engines/engine.py.230: loss: 0.11519 [I 2022-10-27 16:42:57] ...optimizers/trijoint.py.151: total_norm: 16.297943 [I 2022-10-27 16:43:02] ...trap/engines/engine.py.224: train: epoch 77 | batch 1450/2304 [I 2022-10-27 16:43:02] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:07:53 [I 2022-10-27 16:43:02] ...trap/engines/engine.py.229: process: 0.55064 | load: 0.00242 [I 2022-10-27 16:43:02] ...trap/engines/engine.py.230: loss: 0.12905 [I 2022-10-27 16:43:02] ...optimizers/trijoint.py.151: total_norm: 19.093956 [I 2022-10-27 16:43:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 1460/2304 [I 2022-10-27 16:43:08] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:07:50 [I 2022-10-27 16:43:08] ...trap/engines/engine.py.229: process: 0.56119 | load: 0.00240 [I 2022-10-27 16:43:08] ...trap/engines/engine.py.230: loss: 0.13895 [I 2022-10-27 16:43:08] ...optimizers/trijoint.py.151: total_norm: 10.887547 [I 2022-10-27 16:43:14] ...trap/engines/engine.py.224: train: epoch 77 | batch 1470/2304 [I 2022-10-27 16:43:14] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:07:45 [I 2022-10-27 16:43:14] ...trap/engines/engine.py.229: process: 0.56264 | load: 0.00244 [I 2022-10-27 16:43:14] ...trap/engines/engine.py.230: loss: 0.10170 [I 2022-10-27 16:43:14] ...optimizers/trijoint.py.151: total_norm: 14.538101 [I 2022-10-27 16:43:19] ...trap/engines/engine.py.224: train: epoch 77 | batch 1480/2304 [I 2022-10-27 16:43:19] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:36 [I 2022-10-27 16:43:19] ...trap/engines/engine.py.229: process: 0.54553 | load: 0.00251 [I 2022-10-27 16:43:19] ...trap/engines/engine.py.230: loss: 0.09118 [I 2022-10-27 16:43:19] ...optimizers/trijoint.py.151: total_norm: 14.124255 [I 2022-10-27 16:43:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 1490/2304 [I 2022-10-27 16:43:25] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:31 [I 2022-10-27 16:43:25] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00247 [I 2022-10-27 16:43:25] ...trap/engines/engine.py.230: loss: 0.15646 [I 2022-10-27 16:43:25] ...optimizers/trijoint.py.151: total_norm: 15.745708 [I 2022-10-27 16:43:30] ...trap/engines/engine.py.224: train: epoch 77 | batch 1500/2304 [I 2022-10-27 16:43:30] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:26 [I 2022-10-27 16:43:30] ...trap/engines/engine.py.229: process: 0.54443 | load: 0.00246 [I 2022-10-27 16:43:30] ...trap/engines/engine.py.230: loss: 0.22466 [I 2022-10-27 16:43:30] ...optimizers/trijoint.py.151: total_norm: 20.752640 [I 2022-10-27 16:43:36] ...trap/engines/engine.py.224: train: epoch 77 | batch 1510/2304 [I 2022-10-27 16:43:36] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:20 [I 2022-10-27 16:43:36] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00242 [I 2022-10-27 16:43:36] ...trap/engines/engine.py.230: loss: 0.17987 [I 2022-10-27 16:43:36] ...optimizers/trijoint.py.151: total_norm: 18.036613 [I 2022-10-27 16:43:41] ...trap/engines/engine.py.224: train: epoch 77 | batch 1520/2304 [I 2022-10-27 16:43:41] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:16 [I 2022-10-27 16:43:41] ...trap/engines/engine.py.229: process: 0.56178 | load: 0.00241 [I 2022-10-27 16:43:41] ...trap/engines/engine.py.230: loss: 0.14268 [I 2022-10-27 16:43:41] ...optimizers/trijoint.py.151: total_norm: 16.559168 [I 2022-10-27 16:43:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 1530/2304 [I 2022-10-27 16:43:47] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:11 [I 2022-10-27 16:43:47] ...trap/engines/engine.py.229: process: 0.55050 | load: 0.00243 [I 2022-10-27 16:43:47] ...trap/engines/engine.py.230: loss: 0.19949 [I 2022-10-27 16:43:47] ...optimizers/trijoint.py.151: total_norm: 31.635611 [I 2022-10-27 16:43:53] ...trap/engines/engine.py.224: train: epoch 77 | batch 1540/2304 [I 2022-10-27 16:43:53] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:06 [I 2022-10-27 16:43:53] ...trap/engines/engine.py.229: process: 0.55911 | load: 0.00243 [I 2022-10-27 16:43:53] ...trap/engines/engine.py.230: loss: 0.11631 [I 2022-10-27 16:43:53] ...optimizers/trijoint.py.151: total_norm: 12.671935 [I 2022-10-27 16:43:58] ...trap/engines/engine.py.224: train: epoch 77 | batch 1550/2304 [I 2022-10-27 16:43:58] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:06:59 [I 2022-10-27 16:43:58] ...trap/engines/engine.py.229: process: 0.55486 | load: 0.00248 [I 2022-10-27 16:43:58] ...trap/engines/engine.py.230: loss: 0.13025 [I 2022-10-27 16:43:58] ...optimizers/trijoint.py.151: total_norm: 12.062222 [I 2022-10-27 16:44:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 1560/2304 [I 2022-10-27 16:44:04] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:06:53 [I 2022-10-27 16:44:04] ...trap/engines/engine.py.229: process: 0.54510 | load: 0.00245 [I 2022-10-27 16:44:04] ...trap/engines/engine.py.230: loss: 0.19196 [I 2022-10-27 16:44:04] ...optimizers/trijoint.py.151: total_norm: 14.725140 [I 2022-10-27 16:44:09] ...trap/engines/engine.py.224: train: epoch 77 | batch 1570/2304 [I 2022-10-27 16:44:09] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:06:47 [I 2022-10-27 16:44:09] ...trap/engines/engine.py.229: process: 0.54286 | load: 0.00247 [I 2022-10-27 16:44:09] ...trap/engines/engine.py.230: loss: 0.11863 [I 2022-10-27 16:44:09] ...optimizers/trijoint.py.151: total_norm: 12.878042 [I 2022-10-27 16:44:15] ...trap/engines/engine.py.224: train: epoch 77 | batch 1580/2304 [I 2022-10-27 16:44:15] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:06:42 [I 2022-10-27 16:44:15] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00242 [I 2022-10-27 16:44:15] ...trap/engines/engine.py.230: loss: 0.09951 [I 2022-10-27 16:44:15] ...optimizers/trijoint.py.151: total_norm: 10.353652 [I 2022-10-27 16:44:20] ...trap/engines/engine.py.224: train: epoch 77 | batch 1590/2304 [I 2022-10-27 16:44:20] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:37 [I 2022-10-27 16:44:20] ...trap/engines/engine.py.229: process: 0.55703 | load: 0.00248 [I 2022-10-27 16:44:20] ...trap/engines/engine.py.230: loss: 0.07924 [I 2022-10-27 16:44:20] ...optimizers/trijoint.py.151: total_norm: 17.468861 [I 2022-10-27 16:44:26] ...trap/engines/engine.py.224: train: epoch 77 | batch 1600/2304 [I 2022-10-27 16:44:26] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:34 [I 2022-10-27 16:44:26] ...trap/engines/engine.py.229: process: 0.55474 | load: 0.00246 [I 2022-10-27 16:44:26] ...trap/engines/engine.py.230: loss: 0.20511 [I 2022-10-27 16:44:26] ...optimizers/trijoint.py.151: total_norm: 22.865759 [I 2022-10-27 16:44:32] ...trap/engines/engine.py.224: train: epoch 77 | batch 1610/2304 [I 2022-10-27 16:44:32] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:28 [I 2022-10-27 16:44:32] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00250 [I 2022-10-27 16:44:32] ...trap/engines/engine.py.230: loss: 0.10839 [I 2022-10-27 16:44:32] ...optimizers/trijoint.py.151: total_norm: 8.990479 [I 2022-10-27 16:44:37] ...trap/engines/engine.py.224: train: epoch 77 | batch 1620/2304 [I 2022-10-27 16:44:37] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:18 [I 2022-10-27 16:44:37] ...trap/engines/engine.py.229: process: 0.55432 | load: 0.00246 [I 2022-10-27 16:44:37] ...trap/engines/engine.py.230: loss: 0.19176 [I 2022-10-27 16:44:37] ...optimizers/trijoint.py.151: total_norm: 25.601091 [I 2022-10-27 16:44:43] ...trap/engines/engine.py.224: train: epoch 77 | batch 1630/2304 [I 2022-10-27 16:44:43] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:14 [I 2022-10-27 16:44:43] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00243 [I 2022-10-27 16:44:43] ...trap/engines/engine.py.230: loss: 0.08383 [I 2022-10-27 16:44:43] ...optimizers/trijoint.py.151: total_norm: 9.270116 [I 2022-10-27 16:44:48] ...trap/engines/engine.py.224: train: epoch 77 | batch 1640/2304 [I 2022-10-27 16:44:48] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:08 [I 2022-10-27 16:44:48] ...trap/engines/engine.py.229: process: 0.54821 | load: 0.00243 [I 2022-10-27 16:44:48] ...trap/engines/engine.py.230: loss: 0.15523 [I 2022-10-27 16:44:48] ...optimizers/trijoint.py.151: total_norm: 35.800404 [I 2022-10-27 16:44:54] ...trap/engines/engine.py.224: train: epoch 77 | batch 1650/2304 [I 2022-10-27 16:44:54] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:04 [I 2022-10-27 16:44:54] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00243 [I 2022-10-27 16:44:54] ...trap/engines/engine.py.230: loss: 0.15389 [I 2022-10-27 16:44:54] ...optimizers/trijoint.py.151: total_norm: 17.094931 [I 2022-10-27 16:45:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 1660/2304 [I 2022-10-27 16:45:00] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:05:58 [I 2022-10-27 16:45:00] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00244 [I 2022-10-27 16:45:00] ...trap/engines/engine.py.230: loss: 0.14928 [I 2022-10-27 16:45:00] ...optimizers/trijoint.py.151: total_norm: 21.956116 [I 2022-10-27 16:45:05] ...trap/engines/engine.py.224: train: epoch 77 | batch 1670/2304 [I 2022-10-27 16:45:05] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:05:51 [I 2022-10-27 16:45:05] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00241 [I 2022-10-27 16:45:05] ...trap/engines/engine.py.230: loss: 0.16573 [I 2022-10-27 16:45:05] ...optimizers/trijoint.py.151: total_norm: 34.094517 [I 2022-10-27 16:45:11] ...trap/engines/engine.py.224: train: epoch 77 | batch 1680/2304 [I 2022-10-27 16:45:11] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:05:45 [I 2022-10-27 16:45:11] ...trap/engines/engine.py.229: process: 0.54950 | load: 0.00246 [I 2022-10-27 16:45:11] ...trap/engines/engine.py.230: loss: 0.16012 [I 2022-10-27 16:45:11] ...optimizers/trijoint.py.151: total_norm: 18.071869 [I 2022-10-27 16:45:16] ...trap/engines/engine.py.224: train: epoch 77 | batch 1690/2304 [I 2022-10-27 16:45:16] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:05:40 [I 2022-10-27 16:45:16] ...trap/engines/engine.py.229: process: 0.55050 | load: 0.00244 [I 2022-10-27 16:45:16] ...trap/engines/engine.py.230: loss: 0.10967 [I 2022-10-27 16:45:16] ...optimizers/trijoint.py.151: total_norm: 9.391383 [I 2022-10-27 16:45:22] ...trap/engines/engine.py.224: train: epoch 77 | batch 1700/2304 [I 2022-10-27 16:45:22] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:34 [I 2022-10-27 16:45:22] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00242 [I 2022-10-27 16:45:22] ...trap/engines/engine.py.230: loss: 0.17792 [I 2022-10-27 16:45:22] ...optimizers/trijoint.py.151: total_norm: 26.215044 [I 2022-10-27 16:45:27] ...trap/engines/engine.py.224: train: epoch 77 | batch 1710/2304 [I 2022-10-27 16:45:27] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:28 [I 2022-10-27 16:45:27] ...trap/engines/engine.py.229: process: 0.54019 | load: 0.00239 [I 2022-10-27 16:45:27] ...trap/engines/engine.py.230: loss: 0.11639 [I 2022-10-27 16:45:27] ...optimizers/trijoint.py.151: total_norm: 11.328769 [I 2022-10-27 16:45:33] ...trap/engines/engine.py.224: train: epoch 77 | batch 1720/2304 [I 2022-10-27 16:45:33] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:26 [I 2022-10-27 16:45:33] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00242 [I 2022-10-27 16:45:33] ...trap/engines/engine.py.230: loss: 0.12543 [I 2022-10-27 16:45:33] ...optimizers/trijoint.py.151: total_norm: 31.517019 [I 2022-10-27 16:45:38] ...trap/engines/engine.py.224: train: epoch 77 | batch 1730/2304 [I 2022-10-27 16:45:38] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:20 [I 2022-10-27 16:45:38] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00243 [I 2022-10-27 16:45:38] ...trap/engines/engine.py.230: loss: 0.12078 [I 2022-10-27 16:45:38] ...optimizers/trijoint.py.151: total_norm: 12.138155 [I 2022-10-27 16:45:44] ...trap/engines/engine.py.224: train: epoch 77 | batch 1740/2304 [I 2022-10-27 16:45:44] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:14 [I 2022-10-27 16:45:44] ...trap/engines/engine.py.229: process: 0.55764 | load: 0.00271 [I 2022-10-27 16:45:44] ...trap/engines/engine.py.230: loss: 0.17099 [I 2022-10-27 16:45:44] ...optimizers/trijoint.py.151: total_norm: 33.005207 [I 2022-10-27 16:45:50] ...trap/engines/engine.py.224: train: epoch 77 | batch 1750/2304 [I 2022-10-27 16:45:50] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:10 [I 2022-10-27 16:45:50] ...trap/engines/engine.py.229: process: 0.57291 | load: 0.00248 [I 2022-10-27 16:45:50] ...trap/engines/engine.py.230: loss: 0.11824 [I 2022-10-27 16:45:50] ...optimizers/trijoint.py.151: total_norm: 18.923384 [I 2022-10-27 16:45:55] ...trap/engines/engine.py.224: train: epoch 77 | batch 1760/2304 [I 2022-10-27 16:45:55] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:02 [I 2022-10-27 16:45:55] ...trap/engines/engine.py.229: process: 0.55178 | load: 0.00248 [I 2022-10-27 16:45:55] ...trap/engines/engine.py.230: loss: 0.16952 [I 2022-10-27 16:45:55] ...optimizers/trijoint.py.151: total_norm: 41.125782 [I 2022-10-27 16:46:01] ...trap/engines/engine.py.224: train: epoch 77 | batch 1770/2304 [I 2022-10-27 16:46:01] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:04:58 [I 2022-10-27 16:46:01] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00248 [I 2022-10-27 16:46:01] ...trap/engines/engine.py.230: loss: 0.11850 [I 2022-10-27 16:46:01] ...optimizers/trijoint.py.151: total_norm: 8.959986 [I 2022-10-27 16:46:06] ...trap/engines/engine.py.224: train: epoch 77 | batch 1780/2304 [I 2022-10-27 16:46:06] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:04:52 [I 2022-10-27 16:46:06] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00244 [I 2022-10-27 16:46:06] ...trap/engines/engine.py.230: loss: 0.11400 [I 2022-10-27 16:46:06] ...optimizers/trijoint.py.151: total_norm: 15.306861 [I 2022-10-27 16:46:12] ...trap/engines/engine.py.224: train: epoch 77 | batch 1790/2304 [I 2022-10-27 16:46:12] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:04:45 [I 2022-10-27 16:46:12] ...trap/engines/engine.py.229: process: 0.54370 | load: 0.00240 [I 2022-10-27 16:46:12] ...trap/engines/engine.py.230: loss: 0.11921 [I 2022-10-27 16:46:12] ...optimizers/trijoint.py.151: total_norm: 40.491383 [I 2022-10-27 16:46:18] ...trap/engines/engine.py.224: train: epoch 77 | batch 1800/2304 [I 2022-10-27 16:46:18] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:38 [I 2022-10-27 16:46:18] ...trap/engines/engine.py.229: process: 0.55343 | load: 0.00248 [I 2022-10-27 16:46:18] ...trap/engines/engine.py.230: loss: 0.14377 [I 2022-10-27 16:46:18] ...optimizers/trijoint.py.151: total_norm: 26.886345 [I 2022-10-27 16:46:23] ...trap/engines/engine.py.224: train: epoch 77 | batch 1810/2304 [I 2022-10-27 16:46:23] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:38 [I 2022-10-27 16:46:23] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00241 [I 2022-10-27 16:46:23] ...trap/engines/engine.py.230: loss: 0.15489 [I 2022-10-27 16:46:23] ...optimizers/trijoint.py.151: total_norm: 19.565289 [I 2022-10-27 16:46:29] ...trap/engines/engine.py.224: train: epoch 77 | batch 1820/2304 [I 2022-10-27 16:46:29] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:28 [I 2022-10-27 16:46:29] ...trap/engines/engine.py.229: process: 0.54588 | load: 0.00244 [I 2022-10-27 16:46:29] ...trap/engines/engine.py.230: loss: 0.11116 [I 2022-10-27 16:46:29] ...optimizers/trijoint.py.151: total_norm: 12.876715 [I 2022-10-27 16:46:34] ...trap/engines/engine.py.224: train: epoch 77 | batch 1830/2304 [I 2022-10-27 16:46:34] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:23 [I 2022-10-27 16:46:34] ...trap/engines/engine.py.229: process: 0.54587 | load: 0.00243 [I 2022-10-27 16:46:34] ...trap/engines/engine.py.230: loss: 0.15968 [I 2022-10-27 16:46:34] ...optimizers/trijoint.py.151: total_norm: 19.617489 [I 2022-10-27 16:46:40] ...trap/engines/engine.py.224: train: epoch 77 | batch 1840/2304 [I 2022-10-27 16:46:40] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:17 [I 2022-10-27 16:46:40] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00242 [I 2022-10-27 16:46:40] ...trap/engines/engine.py.230: loss: 0.12702 [I 2022-10-27 16:46:40] ...optimizers/trijoint.py.151: total_norm: 13.043099 [I 2022-10-27 16:46:46] ...trap/engines/engine.py.224: train: epoch 77 | batch 1850/2304 [I 2022-10-27 16:46:46] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:11 [I 2022-10-27 16:46:46] ...trap/engines/engine.py.229: process: 0.56048 | load: 0.00245 [I 2022-10-27 16:46:46] ...trap/engines/engine.py.230: loss: 0.15590 [I 2022-10-27 16:46:46] ...optimizers/trijoint.py.151: total_norm: 11.150280 [I 2022-10-27 16:46:51] ...trap/engines/engine.py.224: train: epoch 77 | batch 1860/2304 [I 2022-10-27 16:46:51] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:08 [I 2022-10-27 16:46:51] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00242 [I 2022-10-27 16:46:51] ...trap/engines/engine.py.230: loss: 0.18644 [I 2022-10-27 16:46:51] ...optimizers/trijoint.py.151: total_norm: 11.211977 [I 2022-10-27 16:46:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 1870/2304 [I 2022-10-27 16:46:57] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:00 [I 2022-10-27 16:46:57] ...trap/engines/engine.py.229: process: 0.55185 | load: 0.00243 [I 2022-10-27 16:46:57] ...trap/engines/engine.py.230: loss: 0.12667 [I 2022-10-27 16:46:57] ...optimizers/trijoint.py.151: total_norm: 16.196426 [I 2022-10-27 16:47:02] ...trap/engines/engine.py.224: train: epoch 77 | batch 1880/2304 [I 2022-10-27 16:47:02] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:03:57 [I 2022-10-27 16:47:02] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00243 [I 2022-10-27 16:47:02] ...trap/engines/engine.py.230: loss: 0.10190 [I 2022-10-27 16:47:02] ...optimizers/trijoint.py.151: total_norm: 9.660833 [I 2022-10-27 16:47:08] ...trap/engines/engine.py.224: train: epoch 77 | batch 1890/2304 [I 2022-10-27 16:47:08] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:03:56 [I 2022-10-27 16:47:08] ...trap/engines/engine.py.229: process: 0.61250 | load: 0.00247 [I 2022-10-27 16:47:08] ...trap/engines/engine.py.230: loss: 0.10843 [I 2022-10-27 16:47:08] ...optimizers/trijoint.py.151: total_norm: 14.864481 [I 2022-10-27 16:47:14] ...trap/engines/engine.py.224: train: epoch 77 | batch 1900/2304 [I 2022-10-27 16:47:14] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:03:46 [I 2022-10-27 16:47:14] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00246 [I 2022-10-27 16:47:14] ...trap/engines/engine.py.230: loss: 0.08469 [I 2022-10-27 16:47:14] ...optimizers/trijoint.py.151: total_norm: 11.697739 [I 2022-10-27 16:47:19] ...trap/engines/engine.py.224: train: epoch 77 | batch 1910/2304 [I 2022-10-27 16:47:19] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:03:39 [I 2022-10-27 16:47:19] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00240 [I 2022-10-27 16:47:19] ...trap/engines/engine.py.230: loss: 0.14352 [I 2022-10-27 16:47:19] ...optimizers/trijoint.py.151: total_norm: 18.461849 [I 2022-10-27 16:47:25] ...trap/engines/engine.py.224: train: epoch 77 | batch 1920/2304 [I 2022-10-27 16:47:25] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:33 [I 2022-10-27 16:47:25] ...trap/engines/engine.py.229: process: 0.54572 | load: 0.00241 [I 2022-10-27 16:47:25] ...trap/engines/engine.py.230: loss: 0.11891 [I 2022-10-27 16:47:25] ...optimizers/trijoint.py.151: total_norm: 11.977316 [I 2022-10-27 16:47:30] ...trap/engines/engine.py.224: train: epoch 77 | batch 1930/2304 [I 2022-10-27 16:47:30] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:27 [I 2022-10-27 16:47:30] ...trap/engines/engine.py.229: process: 0.55510 | load: 0.00245 [I 2022-10-27 16:47:30] ...trap/engines/engine.py.230: loss: 0.12744 [I 2022-10-27 16:47:30] ...optimizers/trijoint.py.151: total_norm: 9.631784 [I 2022-10-27 16:47:36] ...trap/engines/engine.py.224: train: epoch 77 | batch 1940/2304 [I 2022-10-27 16:47:36] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:22 [I 2022-10-27 16:47:36] ...trap/engines/engine.py.229: process: 0.55609 | load: 0.00242 [I 2022-10-27 16:47:36] ...trap/engines/engine.py.230: loss: 0.19247 [I 2022-10-27 16:47:36] ...optimizers/trijoint.py.151: total_norm: 19.993921 [I 2022-10-27 16:47:41] ...trap/engines/engine.py.224: train: epoch 77 | batch 1950/2304 [I 2022-10-27 16:47:41] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:17 [I 2022-10-27 16:47:41] ...trap/engines/engine.py.229: process: 0.56557 | load: 0.00242 [I 2022-10-27 16:47:41] ...trap/engines/engine.py.230: loss: 0.10586 [I 2022-10-27 16:47:41] ...optimizers/trijoint.py.151: total_norm: 11.349082 [I 2022-10-27 16:47:47] ...trap/engines/engine.py.224: train: epoch 77 | batch 1960/2304 [I 2022-10-27 16:47:47] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:13 [I 2022-10-27 16:47:47] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00242 [I 2022-10-27 16:47:47] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-27 16:47:47] ...optimizers/trijoint.py.151: total_norm: 11.825365 [I 2022-10-27 16:47:53] ...trap/engines/engine.py.224: train: epoch 77 | batch 1970/2304 [I 2022-10-27 16:47:53] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:07 [I 2022-10-27 16:47:53] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00242 [I 2022-10-27 16:47:53] ...trap/engines/engine.py.230: loss: 0.12034 [I 2022-10-27 16:47:53] ...optimizers/trijoint.py.151: total_norm: 14.878079 [I 2022-10-27 16:47:58] ...trap/engines/engine.py.224: train: epoch 77 | batch 1980/2304 [I 2022-10-27 16:47:58] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:00 [I 2022-10-27 16:47:58] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00244 [I 2022-10-27 16:47:58] ...trap/engines/engine.py.230: loss: 0.11648 [I 2022-10-27 16:47:58] ...optimizers/trijoint.py.151: total_norm: 17.638580 [I 2022-10-27 16:48:04] ...trap/engines/engine.py.224: train: epoch 77 | batch 1990/2304 [I 2022-10-27 16:48:04] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:02:54 [I 2022-10-27 16:48:04] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00241 [I 2022-10-27 16:48:04] ...trap/engines/engine.py.230: loss: 0.21547 [I 2022-10-27 16:48:04] ...optimizers/trijoint.py.151: total_norm: 32.148746 [I 2022-10-27 16:48:09] ...trap/engines/engine.py.224: train: epoch 77 | batch 2000/2304 [I 2022-10-27 16:48:09] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:02:49 [I 2022-10-27 16:48:09] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00240 [I 2022-10-27 16:48:09] ...trap/engines/engine.py.230: loss: 0.11167 [I 2022-10-27 16:48:09] ...optimizers/trijoint.py.151: total_norm: 13.236640 [I 2022-10-27 16:48:15] ...trap/engines/engine.py.224: train: epoch 77 | batch 2010/2304 [I 2022-10-27 16:48:15] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:02:43 [I 2022-10-27 16:48:15] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00251 [I 2022-10-27 16:48:15] ...trap/engines/engine.py.230: loss: 0.10325 [I 2022-10-27 16:48:15] ...optimizers/trijoint.py.151: total_norm: 9.542077 [I 2022-10-27 16:48:21] ...trap/engines/engine.py.224: train: epoch 77 | batch 2020/2304 [I 2022-10-27 16:48:21] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:38 [I 2022-10-27 16:48:21] ...trap/engines/engine.py.229: process: 0.56722 | load: 0.00241 [I 2022-10-27 16:48:21] ...trap/engines/engine.py.230: loss: 0.16268 [I 2022-10-27 16:48:21] ...optimizers/trijoint.py.151: total_norm: 19.014996 [I 2022-10-27 16:48:26] ...trap/engines/engine.py.224: train: epoch 77 | batch 2030/2304 [I 2022-10-27 16:48:26] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:32 [I 2022-10-27 16:48:26] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00245 [I 2022-10-27 16:48:26] ...trap/engines/engine.py.230: loss: 0.13994 [I 2022-10-27 16:48:26] ...optimizers/trijoint.py.151: total_norm: 20.041788 [I 2022-10-27 16:48:32] ...trap/engines/engine.py.224: train: epoch 77 | batch 2040/2304 [I 2022-10-27 16:48:32] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:27 [I 2022-10-27 16:48:32] ...trap/engines/engine.py.229: process: 0.56138 | load: 0.00241 [I 2022-10-27 16:48:32] ...trap/engines/engine.py.230: loss: 0.14510 [I 2022-10-27 16:48:32] ...optimizers/trijoint.py.151: total_norm: 16.861862 [I 2022-10-27 16:48:37] ...trap/engines/engine.py.224: train: epoch 77 | batch 2050/2304 [I 2022-10-27 16:48:37] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:21 [I 2022-10-27 16:48:37] ...trap/engines/engine.py.229: process: 0.56687 | load: 0.00251 [I 2022-10-27 16:48:37] ...trap/engines/engine.py.230: loss: 0.11720 [I 2022-10-27 16:48:37] ...optimizers/trijoint.py.151: total_norm: 10.645138 [I 2022-10-27 16:48:43] ...trap/engines/engine.py.224: train: epoch 77 | batch 2060/2304 [I 2022-10-27 16:48:43] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:16 [I 2022-10-27 16:48:43] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00245 [I 2022-10-27 16:48:43] ...trap/engines/engine.py.230: loss: 0.09219 [I 2022-10-27 16:48:43] ...optimizers/trijoint.py.151: total_norm: 7.091825 [I 2022-10-27 16:48:49] ...trap/engines/engine.py.224: train: epoch 77 | batch 2070/2304 [I 2022-10-27 16:48:49] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:10 [I 2022-10-27 16:48:49] ...trap/engines/engine.py.229: process: 0.54749 | load: 0.00241 [I 2022-10-27 16:48:49] ...trap/engines/engine.py.230: loss: 0.12316 [I 2022-10-27 16:48:49] ...optimizers/trijoint.py.151: total_norm: 9.280126 [I 2022-10-27 16:48:54] ...trap/engines/engine.py.224: train: epoch 77 | batch 2080/2304 [I 2022-10-27 16:48:54] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:06 [I 2022-10-27 16:48:54] ...trap/engines/engine.py.229: process: 0.54922 | load: 0.00243 [I 2022-10-27 16:48:54] ...trap/engines/engine.py.230: loss: 0.12089 [I 2022-10-27 16:48:54] ...optimizers/trijoint.py.151: total_norm: 15.550769 [I 2022-10-27 16:49:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 2090/2304 [I 2022-10-27 16:49:00] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:01:58 [I 2022-10-27 16:49:00] ...trap/engines/engine.py.229: process: 0.54032 | load: 0.00239 [I 2022-10-27 16:49:00] ...trap/engines/engine.py.230: loss: 0.16403 [I 2022-10-27 16:49:00] ...optimizers/trijoint.py.151: total_norm: 26.490540 [I 2022-10-27 16:49:05] ...trap/engines/engine.py.224: train: epoch 77 | batch 2100/2304 [I 2022-10-27 16:49:05] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:01:53 [I 2022-10-27 16:49:05] ...trap/engines/engine.py.229: process: 0.56656 | load: 0.00241 [I 2022-10-27 16:49:05] ...trap/engines/engine.py.230: loss: 0.16021 [I 2022-10-27 16:49:05] ...optimizers/trijoint.py.151: total_norm: 16.975910 [I 2022-10-27 16:49:11] ...trap/engines/engine.py.224: train: epoch 77 | batch 2110/2304 [I 2022-10-27 16:49:11] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:01:47 [I 2022-10-27 16:49:11] ...trap/engines/engine.py.229: process: 0.55496 | load: 0.00242 [I 2022-10-27 16:49:11] ...trap/engines/engine.py.230: loss: 0.09436 [I 2022-10-27 16:49:11] ...optimizers/trijoint.py.151: total_norm: 9.333030 [I 2022-10-27 16:49:16] ...trap/engines/engine.py.224: train: epoch 77 | batch 2120/2304 [I 2022-10-27 16:49:16] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:01:43 [I 2022-10-27 16:49:16] ...trap/engines/engine.py.229: process: 0.55727 | load: 0.00259 [I 2022-10-27 16:49:16] ...trap/engines/engine.py.230: loss: 0.11814 [I 2022-10-27 16:49:16] ...optimizers/trijoint.py.151: total_norm: 12.373789 [I 2022-10-27 16:49:22] ...trap/engines/engine.py.224: train: epoch 77 | batch 2130/2304 [I 2022-10-27 16:49:22] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:37 [I 2022-10-27 16:49:22] ...trap/engines/engine.py.229: process: 0.56298 | load: 0.00243 [I 2022-10-27 16:49:22] ...trap/engines/engine.py.230: loss: 0.15856 [I 2022-10-27 16:49:22] ...optimizers/trijoint.py.151: total_norm: 39.789856 [I 2022-10-27 16:49:28] ...trap/engines/engine.py.224: train: epoch 77 | batch 2140/2304 [I 2022-10-27 16:49:28] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:31 [I 2022-10-27 16:49:28] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00242 [I 2022-10-27 16:49:28] ...trap/engines/engine.py.230: loss: 0.12356 [I 2022-10-27 16:49:28] ...optimizers/trijoint.py.151: total_norm: 21.829344 [I 2022-10-27 16:49:33] ...trap/engines/engine.py.224: train: epoch 77 | batch 2150/2304 [I 2022-10-27 16:49:33] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:25 [I 2022-10-27 16:49:33] ...trap/engines/engine.py.229: process: 0.53935 | load: 0.00244 [I 2022-10-27 16:49:33] ...trap/engines/engine.py.230: loss: 0.09720 [I 2022-10-27 16:49:33] ...optimizers/trijoint.py.151: total_norm: 7.212577 [I 2022-10-27 16:49:39] ...trap/engines/engine.py.224: train: epoch 77 | batch 2160/2304 [I 2022-10-27 16:49:39] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:20 [I 2022-10-27 16:49:39] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00247 [I 2022-10-27 16:49:39] ...trap/engines/engine.py.230: loss: 0.10360 [I 2022-10-27 16:49:39] ...optimizers/trijoint.py.151: total_norm: 12.148376 [I 2022-10-27 16:49:44] ...trap/engines/engine.py.224: train: epoch 77 | batch 2170/2304 [I 2022-10-27 16:49:44] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:14 [I 2022-10-27 16:49:44] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00250 [I 2022-10-27 16:49:44] ...trap/engines/engine.py.230: loss: 0.16001 [I 2022-10-27 16:49:44] ...optimizers/trijoint.py.151: total_norm: 19.363024 [I 2022-10-27 16:49:50] ...trap/engines/engine.py.224: train: epoch 77 | batch 2180/2304 [I 2022-10-27 16:49:50] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:09 [I 2022-10-27 16:49:50] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00244 [I 2022-10-27 16:49:50] ...trap/engines/engine.py.230: loss: 0.11513 [I 2022-10-27 16:49:50] ...optimizers/trijoint.py.151: total_norm: 14.899332 [I 2022-10-27 16:49:56] ...trap/engines/engine.py.224: train: epoch 77 | batch 2190/2304 [I 2022-10-27 16:49:56] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:05 [I 2022-10-27 16:49:56] ...trap/engines/engine.py.229: process: 0.56739 | load: 0.00243 [I 2022-10-27 16:49:56] ...trap/engines/engine.py.230: loss: 0.09706 [I 2022-10-27 16:49:56] ...optimizers/trijoint.py.151: total_norm: 16.862108 [I 2022-10-27 16:50:01] ...trap/engines/engine.py.224: train: epoch 77 | batch 2200/2304 [I 2022-10-27 16:50:01] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:00:58 [I 2022-10-27 16:50:01] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00244 [I 2022-10-27 16:50:01] ...trap/engines/engine.py.230: loss: 0.10349 [I 2022-10-27 16:50:01] ...optimizers/trijoint.py.151: total_norm: 28.588478 [I 2022-10-27 16:50:07] ...trap/engines/engine.py.224: train: epoch 77 | batch 2210/2304 [I 2022-10-27 16:50:07] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:00:52 [I 2022-10-27 16:50:07] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00245 [I 2022-10-27 16:50:07] ...trap/engines/engine.py.230: loss: 0.13339 [I 2022-10-27 16:50:07] ...optimizers/trijoint.py.151: total_norm: 26.246422 [I 2022-10-27 16:50:13] ...trap/engines/engine.py.224: train: epoch 77 | batch 2220/2304 [I 2022-10-27 16:50:13] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:00:46 [I 2022-10-27 16:50:13] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00244 [I 2022-10-27 16:50:13] ...trap/engines/engine.py.230: loss: 0.11247 [I 2022-10-27 16:50:13] ...optimizers/trijoint.py.151: total_norm: 16.893660 [I 2022-10-27 16:50:18] ...trap/engines/engine.py.224: train: epoch 77 | batch 2230/2304 [I 2022-10-27 16:50:18] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:00:41 [I 2022-10-27 16:50:18] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00243 [I 2022-10-27 16:50:18] ...trap/engines/engine.py.230: loss: 0.08601 [I 2022-10-27 16:50:18] ...optimizers/trijoint.py.151: total_norm: 9.432956 [I 2022-10-27 16:50:24] ...trap/engines/engine.py.224: train: epoch 77 | batch 2240/2304 [I 2022-10-27 16:50:24] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:35 [I 2022-10-27 16:50:24] ...trap/engines/engine.py.229: process: 0.54420 | load: 0.00241 [I 2022-10-27 16:50:24] ...trap/engines/engine.py.230: loss: 0.13705 [I 2022-10-27 16:50:24] ...optimizers/trijoint.py.151: total_norm: 21.890337 [I 2022-10-27 16:50:29] ...trap/engines/engine.py.224: train: epoch 77 | batch 2250/2304 [I 2022-10-27 16:50:29] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:30 [I 2022-10-27 16:50:29] ...trap/engines/engine.py.229: process: 0.56578 | load: 0.00243 [I 2022-10-27 16:50:29] ...trap/engines/engine.py.230: loss: 0.18463 [I 2022-10-27 16:50:29] ...optimizers/trijoint.py.151: total_norm: 21.078693 [I 2022-10-27 16:50:35] ...trap/engines/engine.py.224: train: epoch 77 | batch 2260/2304 [I 2022-10-27 16:50:35] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:24 [I 2022-10-27 16:50:35] ...trap/engines/engine.py.229: process: 0.55168 | load: 0.00243 [I 2022-10-27 16:50:35] ...trap/engines/engine.py.230: loss: 0.13445 [I 2022-10-27 16:50:35] ...optimizers/trijoint.py.151: total_norm: 24.233292 [I 2022-10-27 16:50:41] ...trap/engines/engine.py.224: train: epoch 77 | batch 2270/2304 [I 2022-10-27 16:50:41] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:18 [I 2022-10-27 16:50:41] ...trap/engines/engine.py.229: process: 0.54702 | load: 0.00244 [I 2022-10-27 16:50:41] ...trap/engines/engine.py.230: loss: 0.13353 [I 2022-10-27 16:50:41] ...optimizers/trijoint.py.151: total_norm: 62.434666 [I 2022-10-27 16:50:46] ...trap/engines/engine.py.224: train: epoch 77 | batch 2280/2304 [I 2022-10-27 16:50:46] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:13 [I 2022-10-27 16:50:46] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00239 [I 2022-10-27 16:50:46] ...trap/engines/engine.py.230: loss: 0.12749 [I 2022-10-27 16:50:46] ...optimizers/trijoint.py.151: total_norm: 12.692579 [I 2022-10-27 16:50:52] ...trap/engines/engine.py.224: train: epoch 77 | batch 2290/2304 [I 2022-10-27 16:50:52] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:07 [I 2022-10-27 16:50:52] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00250 [I 2022-10-27 16:50:52] ...trap/engines/engine.py.230: loss: 0.14702 [I 2022-10-27 16:50:52] ...optimizers/trijoint.py.151: total_norm: 20.092215 [I 2022-10-27 16:50:57] ...trap/engines/engine.py.224: train: epoch 77 | batch 2300/2304 [I 2022-10-27 16:50:57] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:02 [I 2022-10-27 16:50:57] ...trap/engines/engine.py.229: process: 0.54492 | load: 0.00015 [I 2022-10-27 16:50:57] ...trap/engines/engine.py.230: loss: 0.12340 [I 2022-10-27 16:50:57] ...optimizers/trijoint.py.151: total_norm: 11.292326 [I 2022-10-27 16:51:00] ...trap/engines/engine.py.224: train: epoch 77 | batch 2304/2304 [I 2022-10-27 16:51:00] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:00 [I 2022-10-27 16:51:00] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00014 [I 2022-10-27 16:51:00] ...trap/engines/engine.py.230: loss: 0.11029 [I 2022-10-27 16:51:00] ...optimizers/trijoint.py.151: total_norm: 16.429276 [S 2022-10-27 16:51:01] ...trap/engines/engine.py.129: train_epoch.epoch: 77 [S 2022-10-27 16:51:01] ...trap/engines/engine.py.129: train_epoch.loss: 0.13899244477006722 [I 2022-10-27 16:51:01] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 16:51:02] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 77 [I 2022-10-27 16:51:02] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 16:51:04] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 16:51:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 16:51:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 16:51:06] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 16:51:07] ...trap/engines/engine.py.321: eval: epoch 77 | batch 0/510 [I 2022-10-27 16:51:07] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:41:56 [I 2022-10-27 16:51:07] ...trap/engines/engine.py.326: process: 4.93478 | load: 2.55316 [I 2022-10-27 16:51:10] ...trap/engines/engine.py.321: eval: epoch 77 | batch 10/510 [I 2022-10-27 16:51:10] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:51 [I 2022-10-27 16:51:10] ...trap/engines/engine.py.326: process: 0.32296 | load: 0.00022 [I 2022-10-27 16:51:13] ...trap/engines/engine.py.321: eval: epoch 77 | batch 20/510 [I 2022-10-27 16:51:13] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:08 [I 2022-10-27 16:51:13] ...trap/engines/engine.py.326: process: 0.33209 | load: 0.00023 [I 2022-10-27 16:51:17] ...trap/engines/engine.py.321: eval: epoch 77 | batch 30/510 [I 2022-10-27 16:51:17] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:42 [I 2022-10-27 16:51:17] ...trap/engines/engine.py.326: process: 0.33422 | load: 0.00021 [I 2022-10-27 16:51:20] ...trap/engines/engine.py.321: eval: epoch 77 | batch 40/510 [I 2022-10-27 16:51:20] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:38 [I 2022-10-27 16:51:20] ...trap/engines/engine.py.326: process: 0.33893 | load: 0.00024 [I 2022-10-27 16:51:23] ...trap/engines/engine.py.321: eval: epoch 77 | batch 50/510 [I 2022-10-27 16:51:23] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:36 [I 2022-10-27 16:51:23] ...trap/engines/engine.py.326: process: 0.34219 | load: 0.00026 [I 2022-10-27 16:51:27] ...trap/engines/engine.py.321: eval: epoch 77 | batch 60/510 [I 2022-10-27 16:51:27] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:31 [I 2022-10-27 16:51:27] ...trap/engines/engine.py.326: process: 0.33078 | load: 0.00017 [I 2022-10-27 16:51:30] ...trap/engines/engine.py.321: eval: epoch 77 | batch 70/510 [I 2022-10-27 16:51:30] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:29 [I 2022-10-27 16:51:30] ...trap/engines/engine.py.326: process: 0.33478 | load: 0.00015 [I 2022-10-27 16:51:34] ...trap/engines/engine.py.321: eval: epoch 77 | batch 80/510 [I 2022-10-27 16:51:34] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:23 [I 2022-10-27 16:51:34] ...trap/engines/engine.py.326: process: 0.33656 | load: 0.00016 [I 2022-10-27 16:51:37] ...trap/engines/engine.py.321: eval: epoch 77 | batch 90/510 [I 2022-10-27 16:51:37] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:20 [I 2022-10-27 16:51:37] ...trap/engines/engine.py.326: process: 0.33395 | load: 0.00016 [I 2022-10-27 16:51:40] ...trap/engines/engine.py.321: eval: epoch 77 | batch 100/510 [I 2022-10-27 16:51:40] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:14 [I 2022-10-27 16:51:40] ...trap/engines/engine.py.326: process: 0.28676 | load: 0.00018 [I 2022-10-27 16:51:43] ...trap/engines/engine.py.321: eval: epoch 77 | batch 110/510 [I 2022-10-27 16:51:43] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 16:51:43] ...trap/engines/engine.py.326: process: 0.28821 | load: 0.00020 [I 2022-10-27 16:51:46] ...trap/engines/engine.py.321: eval: epoch 77 | batch 120/510 [I 2022-10-27 16:51:46] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:54 [I 2022-10-27 16:51:46] ...trap/engines/engine.py.326: process: 0.30931 | load: 0.00018 [I 2022-10-27 16:51:49] ...trap/engines/engine.py.321: eval: epoch 77 | batch 130/510 [I 2022-10-27 16:51:49] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:50 [I 2022-10-27 16:51:49] ...trap/engines/engine.py.326: process: 0.29435 | load: 0.00018 [I 2022-10-27 16:51:52] ...trap/engines/engine.py.321: eval: epoch 77 | batch 140/510 [I 2022-10-27 16:51:52] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:48 [I 2022-10-27 16:51:52] ...trap/engines/engine.py.326: process: 0.30360 | load: 0.00015 [I 2022-10-27 16:51:55] ...trap/engines/engine.py.321: eval: epoch 77 | batch 150/510 [I 2022-10-27 16:51:55] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 16:51:55] ...trap/engines/engine.py.326: process: 0.29105 | load: 0.00014 [I 2022-10-27 16:51:58] ...trap/engines/engine.py.321: eval: epoch 77 | batch 160/510 [I 2022-10-27 16:51:58] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 16:51:58] ...trap/engines/engine.py.326: process: 0.28476 | load: 0.00016 [I 2022-10-27 16:52:01] ...trap/engines/engine.py.321: eval: epoch 77 | batch 170/510 [I 2022-10-27 16:52:01] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 16:52:01] ...trap/engines/engine.py.326: process: 0.28904 | load: 0.00014 [I 2022-10-27 16:52:03] ...trap/engines/engine.py.321: eval: epoch 77 | batch 180/510 [I 2022-10-27 16:52:03] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 16:52:03] ...trap/engines/engine.py.326: process: 0.28857 | load: 0.00013 [I 2022-10-27 16:52:06] ...trap/engines/engine.py.321: eval: epoch 77 | batch 190/510 [I 2022-10-27 16:52:06] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 16:52:06] ...trap/engines/engine.py.326: process: 0.29366 | load: 0.00019 [I 2022-10-27 16:52:09] ...trap/engines/engine.py.321: eval: epoch 77 | batch 200/510 [I 2022-10-27 16:52:09] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 16:52:09] ...trap/engines/engine.py.326: process: 0.29201 | load: 0.00015 [I 2022-10-27 16:52:12] ...trap/engines/engine.py.321: eval: epoch 77 | batch 210/510 [I 2022-10-27 16:52:12] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 16:52:12] ...trap/engines/engine.py.326: process: 0.28731 | load: 0.00024 [I 2022-10-27 16:52:15] ...trap/engines/engine.py.321: eval: epoch 77 | batch 220/510 [I 2022-10-27 16:52:15] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 16:52:15] ...trap/engines/engine.py.326: process: 0.28712 | load: 0.00017 [I 2022-10-27 16:52:18] ...trap/engines/engine.py.321: eval: epoch 77 | batch 230/510 [I 2022-10-27 16:52:18] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:21 [I 2022-10-27 16:52:18] ...trap/engines/engine.py.326: process: 0.29792 | load: 0.00026 [I 2022-10-27 16:52:21] ...trap/engines/engine.py.321: eval: epoch 77 | batch 240/510 [I 2022-10-27 16:52:21] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-27 16:52:21] ...trap/engines/engine.py.326: process: 0.28742 | load: 0.00019 [I 2022-10-27 16:52:24] ...trap/engines/engine.py.321: eval: epoch 77 | batch 250/510 [I 2022-10-27 16:52:24] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 16:52:24] ...trap/engines/engine.py.326: process: 0.28957 | load: 0.00025 [I 2022-10-27 16:52:27] ...trap/engines/engine.py.321: eval: epoch 77 | batch 260/510 [I 2022-10-27 16:52:27] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 16:52:27] ...trap/engines/engine.py.326: process: 0.29001 | load: 0.00018 [I 2022-10-27 16:52:30] ...trap/engines/engine.py.321: eval: epoch 77 | batch 270/510 [I 2022-10-27 16:52:30] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 16:52:30] ...trap/engines/engine.py.326: process: 0.29352 | load: 0.00025 [I 2022-10-27 16:52:32] ...trap/engines/engine.py.321: eval: epoch 77 | batch 280/510 [I 2022-10-27 16:52:32] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 16:52:32] ...trap/engines/engine.py.326: process: 0.28591 | load: 0.00019 [I 2022-10-27 16:52:35] ...trap/engines/engine.py.321: eval: epoch 77 | batch 290/510 [I 2022-10-27 16:52:35] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 16:52:35] ...trap/engines/engine.py.326: process: 0.28545 | load: 0.00025 [I 2022-10-27 16:52:38] ...trap/engines/engine.py.321: eval: epoch 77 | batch 300/510 [I 2022-10-27 16:52:38] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 16:52:38] ...trap/engines/engine.py.326: process: 0.28350 | load: 0.00020 [I 2022-10-27 16:52:41] ...trap/engines/engine.py.321: eval: epoch 77 | batch 310/510 [I 2022-10-27 16:52:41] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 16:52:41] ...trap/engines/engine.py.326: process: 0.28616 | load: 0.00020 [I 2022-10-27 16:52:44] ...trap/engines/engine.py.321: eval: epoch 77 | batch 320/510 [I 2022-10-27 16:52:44] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:55 [I 2022-10-27 16:52:44] ...trap/engines/engine.py.326: process: 0.29007 | load: 0.00018 [I 2022-10-27 16:52:47] ...trap/engines/engine.py.321: eval: epoch 77 | batch 330/510 [I 2022-10-27 16:52:47] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-27 16:52:47] ...trap/engines/engine.py.326: process: 0.29125 | load: 0.00024 [I 2022-10-27 16:52:50] ...trap/engines/engine.py.321: eval: epoch 77 | batch 340/510 [I 2022-10-27 16:52:50] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-27 16:52:50] ...trap/engines/engine.py.326: process: 0.28739 | load: 0.00014 [I 2022-10-27 16:52:53] ...trap/engines/engine.py.321: eval: epoch 77 | batch 350/510 [I 2022-10-27 16:52:53] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 16:52:53] ...trap/engines/engine.py.326: process: 0.28638 | load: 0.00026 [I 2022-10-27 16:52:56] ...trap/engines/engine.py.321: eval: epoch 77 | batch 360/510 [I 2022-10-27 16:52:56] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 16:52:56] ...trap/engines/engine.py.326: process: 0.28853 | load: 0.00016 [I 2022-10-27 16:52:58] ...trap/engines/engine.py.321: eval: epoch 77 | batch 370/510 [I 2022-10-27 16:52:58] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 16:52:58] ...trap/engines/engine.py.326: process: 0.29281 | load: 0.00023 [I 2022-10-27 16:53:01] ...trap/engines/engine.py.321: eval: epoch 77 | batch 380/510 [I 2022-10-27 16:53:01] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 16:53:01] ...trap/engines/engine.py.326: process: 0.28650 | load: 0.00017 [I 2022-10-27 16:53:04] ...trap/engines/engine.py.321: eval: epoch 77 | batch 390/510 [I 2022-10-27 16:53:04] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 16:53:04] ...trap/engines/engine.py.326: process: 0.28838 | load: 0.00024 [I 2022-10-27 16:53:07] ...trap/engines/engine.py.321: eval: epoch 77 | batch 400/510 [I 2022-10-27 16:53:07] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 16:53:07] ...trap/engines/engine.py.326: process: 0.28736 | load: 0.00017 [I 2022-10-27 16:53:10] ...trap/engines/engine.py.321: eval: epoch 77 | batch 410/510 [I 2022-10-27 16:53:10] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 16:53:10] ...trap/engines/engine.py.326: process: 0.28735 | load: 0.00024 [I 2022-10-27 16:53:13] ...trap/engines/engine.py.321: eval: epoch 77 | batch 420/510 [I 2022-10-27 16:53:13] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-27 16:53:13] ...trap/engines/engine.py.326: process: 0.28790 | load: 0.00020 [I 2022-10-27 16:53:16] ...trap/engines/engine.py.321: eval: epoch 77 | batch 430/510 [I 2022-10-27 16:53:16] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 16:53:16] ...trap/engines/engine.py.326: process: 0.28707 | load: 0.00024 [I 2022-10-27 16:53:19] ...trap/engines/engine.py.321: eval: epoch 77 | batch 440/510 [I 2022-10-27 16:53:19] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 16:53:19] ...trap/engines/engine.py.326: process: 0.28926 | load: 0.00018 [I 2022-10-27 16:53:22] ...trap/engines/engine.py.321: eval: epoch 77 | batch 450/510 [I 2022-10-27 16:53:22] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 16:53:22] ...trap/engines/engine.py.326: process: 0.28595 | load: 0.00026 [I 2022-10-27 16:53:24] ...trap/engines/engine.py.321: eval: epoch 77 | batch 460/510 [I 2022-10-27 16:53:24] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 16:53:24] ...trap/engines/engine.py.326: process: 0.28340 | load: 0.00020 [I 2022-10-27 16:53:27] ...trap/engines/engine.py.321: eval: epoch 77 | batch 470/510 [I 2022-10-27 16:53:27] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 16:53:27] ...trap/engines/engine.py.326: process: 0.29175 | load: 0.00026 [I 2022-10-27 16:53:30] ...trap/engines/engine.py.321: eval: epoch 77 | batch 480/510 [I 2022-10-27 16:53:30] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 16:53:30] ...trap/engines/engine.py.326: process: 0.29029 | load: 0.00016 [I 2022-10-27 16:53:33] ...trap/engines/engine.py.321: eval: epoch 77 | batch 490/510 [I 2022-10-27 16:53:33] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 16:53:33] ...trap/engines/engine.py.326: process: 0.29432 | load: 0.00019 [I 2022-10-27 16:53:36] ...trap/engines/engine.py.321: eval: epoch 77 | batch 500/510 [I 2022-10-27 16:53:36] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 16:53:36] ...trap/engines/engine.py.326: process: 0.28845 | load: 0.00017 [I 2022-10-27 16:53:39] ...trap/engines/engine.py.321: eval: epoch 77 | batch 510/510 [I 2022-10-27 16:53:39] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 16:53:39] ...trap/engines/engine.py.326: process: 0.28885 | load: 0.00011 [S 2022-10-27 16:53:40] ...trap/engines/engine.py.132: eval_epoch.epoch: 77 [I 2022-10-27 16:53:40] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.728500 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.911700 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940800 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.719000 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907100 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937900 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012159 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007976 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007587 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014339 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007595 [S 2022-10-27 16:53:44] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007489 [I 2022-10-27 16:53:45] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 16:53:45] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 16:53:46] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 16:53:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 16:53:46] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 16:53:49] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 16:53:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 16:53:56] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 16:53:56] ...trap/engines/engine.py.163: Training model on trainset for epoch 78 [I 2022-10-27 16:53:56] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 16:53:59] ...trap/engines/engine.py.224: train: epoch 78 | batch 0/2304 [I 2022-10-27 16:53:59] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:29 [I 2022-10-27 16:53:59] ...trap/engines/engine.py.229: process: 2.69526 | load: 2.06470 [I 2022-10-27 16:53:59] ...trap/engines/engine.py.230: loss: 0.15184 [I 2022-10-27 16:53:59] ...optimizers/trijoint.py.151: total_norm: 14.219327 [I 2022-10-27 16:54:04] ...trap/engines/engine.py.224: train: epoch 78 | batch 10/2304 [I 2022-10-27 16:54:04] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:16 [I 2022-10-27 16:54:04] ...trap/engines/engine.py.229: process: 0.57768 | load: 0.00357 [I 2022-10-27 16:54:04] ...trap/engines/engine.py.230: loss: 0.11982 [I 2022-10-27 16:54:04] ...optimizers/trijoint.py.151: total_norm: 11.445869 [I 2022-10-27 16:54:10] ...trap/engines/engine.py.224: train: epoch 78 | batch 20/2304 [I 2022-10-27 16:54:10] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:09 [I 2022-10-27 16:54:10] ...trap/engines/engine.py.229: process: 0.55093 | load: 0.00358 [I 2022-10-27 16:54:10] ...trap/engines/engine.py.230: loss: 0.09987 [I 2022-10-27 16:54:10] ...optimizers/trijoint.py.151: total_norm: 9.963228 [I 2022-10-27 16:54:15] ...trap/engines/engine.py.224: train: epoch 78 | batch 30/2304 [I 2022-10-27 16:54:15] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:21 [I 2022-10-27 16:54:15] ...trap/engines/engine.py.229: process: 0.56412 | load: 0.00333 [I 2022-10-27 16:54:15] ...trap/engines/engine.py.230: loss: 0.13194 [I 2022-10-27 16:54:15] ...optimizers/trijoint.py.151: total_norm: 16.417572 [I 2022-10-27 16:54:21] ...trap/engines/engine.py.224: train: epoch 78 | batch 40/2304 [I 2022-10-27 16:54:21] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:59 [I 2022-10-27 16:54:21] ...trap/engines/engine.py.229: process: 0.56440 | load: 0.00297 [I 2022-10-27 16:54:21] ...trap/engines/engine.py.230: loss: 0.09752 [I 2022-10-27 16:54:21] ...optimizers/trijoint.py.151: total_norm: 10.762367 [I 2022-10-27 16:54:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 50/2304 [I 2022-10-27 16:54:27] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:54 [I 2022-10-27 16:54:27] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00269 [I 2022-10-27 16:54:27] ...trap/engines/engine.py.230: loss: 0.16438 [I 2022-10-27 16:54:27] ...optimizers/trijoint.py.151: total_norm: 16.395878 [I 2022-10-27 16:54:32] ...trap/engines/engine.py.224: train: epoch 78 | batch 60/2304 [I 2022-10-27 16:54:32] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:46 [I 2022-10-27 16:54:32] ...trap/engines/engine.py.229: process: 0.54849 | load: 0.00325 [I 2022-10-27 16:54:32] ...trap/engines/engine.py.230: loss: 0.09562 [I 2022-10-27 16:54:32] ...optimizers/trijoint.py.151: total_norm: 8.930766 [I 2022-10-27 16:54:38] ...trap/engines/engine.py.224: train: epoch 78 | batch 70/2304 [I 2022-10-27 16:54:38] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:42 [I 2022-10-27 16:54:38] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00271 [I 2022-10-27 16:54:38] ...trap/engines/engine.py.230: loss: 0.14214 [I 2022-10-27 16:54:38] ...optimizers/trijoint.py.151: total_norm: 23.463087 [I 2022-10-27 16:54:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 80/2304 [I 2022-10-27 16:54:43] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:35 [I 2022-10-27 16:54:43] ...trap/engines/engine.py.229: process: 0.54612 | load: 0.00271 [I 2022-10-27 16:54:43] ...trap/engines/engine.py.230: loss: 0.13549 [I 2022-10-27 16:54:43] ...optimizers/trijoint.py.151: total_norm: 17.144114 [I 2022-10-27 16:54:49] ...trap/engines/engine.py.224: train: epoch 78 | batch 90/2304 [I 2022-10-27 16:54:49] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:32 [I 2022-10-27 16:54:49] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00276 [I 2022-10-27 16:54:49] ...trap/engines/engine.py.230: loss: 0.15043 [I 2022-10-27 16:54:49] ...optimizers/trijoint.py.151: total_norm: 15.632138 [I 2022-10-27 16:54:54] ...trap/engines/engine.py.224: train: epoch 78 | batch 100/2304 [I 2022-10-27 16:54:54] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:15 [I 2022-10-27 16:54:54] ...trap/engines/engine.py.229: process: 0.54799 | load: 0.00266 [I 2022-10-27 16:54:54] ...trap/engines/engine.py.230: loss: 0.18339 [I 2022-10-27 16:54:54] ...optimizers/trijoint.py.151: total_norm: 31.886744 [I 2022-10-27 16:55:00] ...trap/engines/engine.py.224: train: epoch 78 | batch 110/2304 [I 2022-10-27 16:55:00] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:41 [I 2022-10-27 16:55:00] ...trap/engines/engine.py.229: process: 0.56706 | load: 0.00327 [I 2022-10-27 16:55:00] ...trap/engines/engine.py.230: loss: 0.16534 [I 2022-10-27 16:55:00] ...optimizers/trijoint.py.151: total_norm: 28.907663 [I 2022-10-27 16:55:06] ...trap/engines/engine.py.224: train: epoch 78 | batch 120/2304 [I 2022-10-27 16:55:06] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:19 [I 2022-10-27 16:55:06] ...trap/engines/engine.py.229: process: 0.55694 | load: 0.00316 [I 2022-10-27 16:55:06] ...trap/engines/engine.py.230: loss: 0.14784 [I 2022-10-27 16:55:06] ...optimizers/trijoint.py.151: total_norm: 26.099911 [I 2022-10-27 16:55:11] ...trap/engines/engine.py.224: train: epoch 78 | batch 130/2304 [I 2022-10-27 16:55:11] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:05 [I 2022-10-27 16:55:11] ...trap/engines/engine.py.229: process: 0.54815 | load: 0.00327 [I 2022-10-27 16:55:11] ...trap/engines/engine.py.230: loss: 0.14532 [I 2022-10-27 16:55:11] ...optimizers/trijoint.py.151: total_norm: 25.581680 [I 2022-10-27 16:55:17] ...trap/engines/engine.py.224: train: epoch 78 | batch 140/2304 [I 2022-10-27 16:55:17] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:01 [I 2022-10-27 16:55:17] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00267 [I 2022-10-27 16:55:17] ...trap/engines/engine.py.230: loss: 0.13464 [I 2022-10-27 16:55:17] ...optimizers/trijoint.py.151: total_norm: 16.329454 [I 2022-10-27 16:55:22] ...trap/engines/engine.py.224: train: epoch 78 | batch 150/2304 [I 2022-10-27 16:55:22] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:51 [I 2022-10-27 16:55:22] ...trap/engines/engine.py.229: process: 0.55924 | load: 0.00263 [I 2022-10-27 16:55:22] ...trap/engines/engine.py.230: loss: 0.17370 [I 2022-10-27 16:55:22] ...optimizers/trijoint.py.151: total_norm: 28.021494 [I 2022-10-27 16:55:28] ...trap/engines/engine.py.224: train: epoch 78 | batch 160/2304 [I 2022-10-27 16:55:28] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:59 [I 2022-10-27 16:55:28] ...trap/engines/engine.py.229: process: 0.56393 | load: 0.00269 [I 2022-10-27 16:55:28] ...trap/engines/engine.py.230: loss: 0.21752 [I 2022-10-27 16:55:28] ...optimizers/trijoint.py.151: total_norm: 26.923038 [I 2022-10-27 16:55:33] ...trap/engines/engine.py.224: train: epoch 78 | batch 170/2304 [I 2022-10-27 16:55:33] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:59 [I 2022-10-27 16:55:33] ...trap/engines/engine.py.229: process: 0.56520 | load: 0.00257 [I 2022-10-27 16:55:33] ...trap/engines/engine.py.230: loss: 0.10615 [I 2022-10-27 16:55:33] ...optimizers/trijoint.py.151: total_norm: 11.375090 [I 2022-10-27 16:55:39] ...trap/engines/engine.py.224: train: epoch 78 | batch 180/2304 [I 2022-10-27 16:55:39] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:53 [I 2022-10-27 16:55:39] ...trap/engines/engine.py.229: process: 0.55924 | load: 0.00269 [I 2022-10-27 16:55:39] ...trap/engines/engine.py.230: loss: 0.11121 [I 2022-10-27 16:55:39] ...optimizers/trijoint.py.151: total_norm: 11.910315 [I 2022-10-27 16:55:45] ...trap/engines/engine.py.224: train: epoch 78 | batch 190/2304 [I 2022-10-27 16:55:45] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:38 [I 2022-10-27 16:55:45] ...trap/engines/engine.py.229: process: 0.55639 | load: 0.00265 [I 2022-10-27 16:55:45] ...trap/engines/engine.py.230: loss: 0.17417 [I 2022-10-27 16:55:45] ...optimizers/trijoint.py.151: total_norm: 27.625542 [I 2022-10-27 16:55:50] ...trap/engines/engine.py.224: train: epoch 78 | batch 200/2304 [I 2022-10-27 16:55:50] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:35 [I 2022-10-27 16:55:50] ...trap/engines/engine.py.229: process: 0.56790 | load: 0.00248 [I 2022-10-27 16:55:50] ...trap/engines/engine.py.230: loss: 0.22789 [I 2022-10-27 16:55:50] ...optimizers/trijoint.py.151: total_norm: 29.368805 [I 2022-10-27 16:55:56] ...trap/engines/engine.py.224: train: epoch 78 | batch 210/2304 [I 2022-10-27 16:55:56] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:28 [I 2022-10-27 16:55:56] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00267 [I 2022-10-27 16:55:56] ...trap/engines/engine.py.230: loss: 0.14334 [I 2022-10-27 16:55:56] ...optimizers/trijoint.py.151: total_norm: 18.476820 [I 2022-10-27 16:56:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 220/2304 [I 2022-10-27 16:56:01] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:20 [I 2022-10-27 16:56:01] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00281 [I 2022-10-27 16:56:01] ...trap/engines/engine.py.230: loss: 0.11035 [I 2022-10-27 16:56:01] ...optimizers/trijoint.py.151: total_norm: 21.737967 [I 2022-10-27 16:56:07] ...trap/engines/engine.py.224: train: epoch 78 | batch 230/2304 [I 2022-10-27 16:56:07] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:25 [I 2022-10-27 16:56:07] ...trap/engines/engine.py.229: process: 0.58020 | load: 0.00309 [I 2022-10-27 16:56:07] ...trap/engines/engine.py.230: loss: 0.19337 [I 2022-10-27 16:56:07] ...optimizers/trijoint.py.151: total_norm: 17.117878 [I 2022-10-27 16:56:13] ...trap/engines/engine.py.224: train: epoch 78 | batch 240/2304 [I 2022-10-27 16:56:13] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:03 [I 2022-10-27 16:56:13] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00254 [I 2022-10-27 16:56:13] ...trap/engines/engine.py.230: loss: 0.21174 [I 2022-10-27 16:56:13] ...optimizers/trijoint.py.151: total_norm: 20.853704 [I 2022-10-27 16:56:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 250/2304 [I 2022-10-27 16:56:18] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:57 [I 2022-10-27 16:56:18] ...trap/engines/engine.py.229: process: 0.55243 | load: 0.00252 [I 2022-10-27 16:56:18] ...trap/engines/engine.py.230: loss: 0.12634 [I 2022-10-27 16:56:18] ...optimizers/trijoint.py.151: total_norm: 17.392977 [I 2022-10-27 16:56:24] ...trap/engines/engine.py.224: train: epoch 78 | batch 260/2304 [I 2022-10-27 16:56:24] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:22 [I 2022-10-27 16:56:24] ...trap/engines/engine.py.229: process: 0.56154 | load: 0.00244 [I 2022-10-27 16:56:24] ...trap/engines/engine.py.230: loss: 0.17728 [I 2022-10-27 16:56:24] ...optimizers/trijoint.py.151: total_norm: 21.276075 [I 2022-10-27 16:56:29] ...trap/engines/engine.py.224: train: epoch 78 | batch 270/2304 [I 2022-10-27 16:56:29] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:00 [I 2022-10-27 16:56:29] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00246 [I 2022-10-27 16:56:29] ...trap/engines/engine.py.230: loss: 0.12900 [I 2022-10-27 16:56:29] ...optimizers/trijoint.py.151: total_norm: 12.672123 [I 2022-10-27 16:56:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 280/2304 [I 2022-10-27 16:56:35] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:52 [I 2022-10-27 16:56:35] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00244 [I 2022-10-27 16:56:35] ...trap/engines/engine.py.230: loss: 0.12590 [I 2022-10-27 16:56:35] ...optimizers/trijoint.py.151: total_norm: 17.244066 [I 2022-10-27 16:56:41] ...trap/engines/engine.py.224: train: epoch 78 | batch 290/2304 [I 2022-10-27 16:56:41] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:48 [I 2022-10-27 16:56:41] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00258 [I 2022-10-27 16:56:41] ...trap/engines/engine.py.230: loss: 0.16633 [I 2022-10-27 16:56:41] ...optimizers/trijoint.py.151: total_norm: 13.111224 [I 2022-10-27 16:56:46] ...trap/engines/engine.py.224: train: epoch 78 | batch 300/2304 [I 2022-10-27 16:56:46] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:43 [I 2022-10-27 16:56:46] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00282 [I 2022-10-27 16:56:46] ...trap/engines/engine.py.230: loss: 0.10150 [I 2022-10-27 16:56:46] ...optimizers/trijoint.py.151: total_norm: 11.445199 [I 2022-10-27 16:56:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 310/2304 [I 2022-10-27 16:56:52] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:38 [I 2022-10-27 16:56:52] ...trap/engines/engine.py.229: process: 0.56958 | load: 0.00245 [I 2022-10-27 16:56:52] ...trap/engines/engine.py.230: loss: 0.12550 [I 2022-10-27 16:56:52] ...optimizers/trijoint.py.151: total_norm: 19.929426 [I 2022-10-27 16:56:58] ...trap/engines/engine.py.224: train: epoch 78 | batch 320/2304 [I 2022-10-27 16:56:58] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:37 [I 2022-10-27 16:56:58] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00246 [I 2022-10-27 16:56:58] ...trap/engines/engine.py.230: loss: 0.13869 [I 2022-10-27 16:56:58] ...optimizers/trijoint.py.151: total_norm: 11.546323 [I 2022-10-27 16:57:03] ...trap/engines/engine.py.224: train: epoch 78 | batch 330/2304 [I 2022-10-27 16:57:03] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:23 [I 2022-10-27 16:57:03] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00255 [I 2022-10-27 16:57:03] ...trap/engines/engine.py.230: loss: 0.11277 [I 2022-10-27 16:57:03] ...optimizers/trijoint.py.151: total_norm: 8.286603 [I 2022-10-27 16:57:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 340/2304 [I 2022-10-27 16:57:09] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:09 [I 2022-10-27 16:57:09] ...trap/engines/engine.py.229: process: 0.55059 | load: 0.00249 [I 2022-10-27 16:57:09] ...trap/engines/engine.py.230: loss: 0.21131 [I 2022-10-27 16:57:09] ...optimizers/trijoint.py.151: total_norm: 52.630146 [I 2022-10-27 16:57:14] ...trap/engines/engine.py.224: train: epoch 78 | batch 350/2304 [I 2022-10-27 16:57:14] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:26 [I 2022-10-27 16:57:14] ...trap/engines/engine.py.229: process: 0.56572 | load: 0.00252 [I 2022-10-27 16:57:14] ...trap/engines/engine.py.230: loss: 0.27335 [I 2022-10-27 16:57:14] ...optimizers/trijoint.py.151: total_norm: 13.596368 [I 2022-10-27 16:57:20] ...trap/engines/engine.py.224: train: epoch 78 | batch 360/2304 [I 2022-10-27 16:57:20] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:08 [I 2022-10-27 16:57:20] ...trap/engines/engine.py.229: process: 0.56330 | load: 0.00248 [I 2022-10-27 16:57:20] ...trap/engines/engine.py.230: loss: 0.29187 [I 2022-10-27 16:57:20] ...optimizers/trijoint.py.151: total_norm: 17.143402 [I 2022-10-27 16:57:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 370/2304 [I 2022-10-27 16:57:26] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:02 [I 2022-10-27 16:57:26] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00247 [I 2022-10-27 16:57:26] ...trap/engines/engine.py.230: loss: 0.14917 [I 2022-10-27 16:57:26] ...optimizers/trijoint.py.151: total_norm: 12.960938 [I 2022-10-27 16:57:31] ...trap/engines/engine.py.224: train: epoch 78 | batch 380/2304 [I 2022-10-27 16:57:31] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:50 [I 2022-10-27 16:57:31] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00245 [I 2022-10-27 16:57:31] ...trap/engines/engine.py.230: loss: 0.13533 [I 2022-10-27 16:57:31] ...optimizers/trijoint.py.151: total_norm: 16.016628 [I 2022-10-27 16:57:37] ...trap/engines/engine.py.224: train: epoch 78 | batch 390/2304 [I 2022-10-27 16:57:37] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:44 [I 2022-10-27 16:57:37] ...trap/engines/engine.py.229: process: 0.55460 | load: 0.00246 [I 2022-10-27 16:57:37] ...trap/engines/engine.py.230: loss: 0.13890 [I 2022-10-27 16:57:37] ...optimizers/trijoint.py.151: total_norm: 10.148318 [I 2022-10-27 16:57:42] ...trap/engines/engine.py.224: train: epoch 78 | batch 400/2304 [I 2022-10-27 16:57:42] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:44 [I 2022-10-27 16:57:42] ...trap/engines/engine.py.229: process: 0.56214 | load: 0.00249 [I 2022-10-27 16:57:42] ...trap/engines/engine.py.230: loss: 0.11563 [I 2022-10-27 16:57:42] ...optimizers/trijoint.py.151: total_norm: 19.056684 [I 2022-10-27 16:57:48] ...trap/engines/engine.py.224: train: epoch 78 | batch 410/2304 [I 2022-10-27 16:57:48] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:29 [I 2022-10-27 16:57:48] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00251 [I 2022-10-27 16:57:48] ...trap/engines/engine.py.230: loss: 0.15404 [I 2022-10-27 16:57:48] ...optimizers/trijoint.py.151: total_norm: 11.515425 [I 2022-10-27 16:57:54] ...trap/engines/engine.py.224: train: epoch 78 | batch 420/2304 [I 2022-10-27 16:57:54] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:44 [I 2022-10-27 16:57:54] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00246 [I 2022-10-27 16:57:54] ...trap/engines/engine.py.230: loss: 0.11438 [I 2022-10-27 16:57:54] ...optimizers/trijoint.py.151: total_norm: 15.913837 [I 2022-10-27 16:57:59] ...trap/engines/engine.py.224: train: epoch 78 | batch 430/2304 [I 2022-10-27 16:57:59] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:27 [I 2022-10-27 16:57:59] ...trap/engines/engine.py.229: process: 0.56311 | load: 0.00292 [I 2022-10-27 16:57:59] ...trap/engines/engine.py.230: loss: 0.16997 [I 2022-10-27 16:57:59] ...optimizers/trijoint.py.151: total_norm: 32.573929 [I 2022-10-27 16:58:05] ...trap/engines/engine.py.224: train: epoch 78 | batch 440/2304 [I 2022-10-27 16:58:05] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:17 [I 2022-10-27 16:58:05] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00247 [I 2022-10-27 16:58:05] ...trap/engines/engine.py.230: loss: 0.10405 [I 2022-10-27 16:58:05] ...optimizers/trijoint.py.151: total_norm: 9.150279 [I 2022-10-27 16:58:10] ...trap/engines/engine.py.224: train: epoch 78 | batch 450/2304 [I 2022-10-27 16:58:10] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:15 [I 2022-10-27 16:58:10] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00251 [I 2022-10-27 16:58:10] ...trap/engines/engine.py.230: loss: 0.14787 [I 2022-10-27 16:58:10] ...optimizers/trijoint.py.151: total_norm: 15.262315 [I 2022-10-27 16:58:16] ...trap/engines/engine.py.224: train: epoch 78 | batch 460/2304 [I 2022-10-27 16:58:16] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:04 [I 2022-10-27 16:58:16] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00244 [I 2022-10-27 16:58:16] ...trap/engines/engine.py.230: loss: 0.21483 [I 2022-10-27 16:58:16] ...optimizers/trijoint.py.151: total_norm: 17.509348 [I 2022-10-27 16:58:22] ...trap/engines/engine.py.224: train: epoch 78 | batch 470/2304 [I 2022-10-27 16:58:22] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:56 [I 2022-10-27 16:58:22] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00242 [I 2022-10-27 16:58:22] ...trap/engines/engine.py.230: loss: 0.10182 [I 2022-10-27 16:58:22] ...optimizers/trijoint.py.151: total_norm: 20.087345 [I 2022-10-27 16:58:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 480/2304 [I 2022-10-27 16:58:27] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:46 [I 2022-10-27 16:58:27] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00243 [I 2022-10-27 16:58:27] ...trap/engines/engine.py.230: loss: 0.19650 [I 2022-10-27 16:58:27] ...optimizers/trijoint.py.151: total_norm: 77.359230 [I 2022-10-27 16:58:33] ...trap/engines/engine.py.224: train: epoch 78 | batch 490/2304 [I 2022-10-27 16:58:33] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:51 [I 2022-10-27 16:58:33] ...trap/engines/engine.py.229: process: 0.54719 | load: 0.00281 [I 2022-10-27 16:58:33] ...trap/engines/engine.py.230: loss: 0.15100 [I 2022-10-27 16:58:33] ...optimizers/trijoint.py.151: total_norm: 10.930902 [I 2022-10-27 16:58:38] ...trap/engines/engine.py.224: train: epoch 78 | batch 500/2304 [I 2022-10-27 16:58:38] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:46 [I 2022-10-27 16:58:38] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00242 [I 2022-10-27 16:58:38] ...trap/engines/engine.py.230: loss: 0.13347 [I 2022-10-27 16:58:38] ...optimizers/trijoint.py.151: total_norm: 23.632463 [I 2022-10-27 16:58:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 510/2304 [I 2022-10-27 16:58:44] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:38 [I 2022-10-27 16:58:44] ...trap/engines/engine.py.229: process: 0.54568 | load: 0.00251 [I 2022-10-27 16:58:44] ...trap/engines/engine.py.230: loss: 0.09364 [I 2022-10-27 16:58:44] ...optimizers/trijoint.py.151: total_norm: 7.944331 [I 2022-10-27 16:58:49] ...trap/engines/engine.py.224: train: epoch 78 | batch 520/2304 [I 2022-10-27 16:58:49] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:33 [I 2022-10-27 16:58:49] ...trap/engines/engine.py.229: process: 0.56471 | load: 0.00255 [I 2022-10-27 16:58:49] ...trap/engines/engine.py.230: loss: 0.22297 [I 2022-10-27 16:58:49] ...optimizers/trijoint.py.151: total_norm: 20.061407 [I 2022-10-27 16:58:55] ...trap/engines/engine.py.224: train: epoch 78 | batch 530/2304 [I 2022-10-27 16:58:55] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:33 [I 2022-10-27 16:58:55] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00245 [I 2022-10-27 16:58:55] ...trap/engines/engine.py.230: loss: 0.13490 [I 2022-10-27 16:58:55] ...optimizers/trijoint.py.151: total_norm: 9.826340 [I 2022-10-27 16:59:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 540/2304 [I 2022-10-27 16:59:01] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:28 [I 2022-10-27 16:59:01] ...trap/engines/engine.py.229: process: 0.56888 | load: 0.00245 [I 2022-10-27 16:59:01] ...trap/engines/engine.py.230: loss: 0.11808 [I 2022-10-27 16:59:01] ...optimizers/trijoint.py.151: total_norm: 10.843242 [I 2022-10-27 16:59:06] ...trap/engines/engine.py.224: train: epoch 78 | batch 550/2304 [I 2022-10-27 16:59:06] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:26 [I 2022-10-27 16:59:06] ...trap/engines/engine.py.229: process: 0.56703 | load: 0.00245 [I 2022-10-27 16:59:06] ...trap/engines/engine.py.230: loss: 0.13565 [I 2022-10-27 16:59:06] ...optimizers/trijoint.py.151: total_norm: 22.900173 [I 2022-10-27 16:59:12] ...trap/engines/engine.py.224: train: epoch 78 | batch 560/2304 [I 2022-10-27 16:59:12] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:09 [I 2022-10-27 16:59:12] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00246 [I 2022-10-27 16:59:12] ...trap/engines/engine.py.230: loss: 0.15972 [I 2022-10-27 16:59:12] ...optimizers/trijoint.py.151: total_norm: 18.015923 [I 2022-10-27 16:59:17] ...trap/engines/engine.py.224: train: epoch 78 | batch 570/2304 [I 2022-10-27 16:59:17] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:03 [I 2022-10-27 16:59:17] ...trap/engines/engine.py.229: process: 0.54847 | load: 0.00244 [I 2022-10-27 16:59:17] ...trap/engines/engine.py.230: loss: 0.07319 [I 2022-10-27 16:59:17] ...optimizers/trijoint.py.151: total_norm: 12.924529 [I 2022-10-27 16:59:23] ...trap/engines/engine.py.224: train: epoch 78 | batch 580/2304 [I 2022-10-27 16:59:23] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:00 [I 2022-10-27 16:59:23] ...trap/engines/engine.py.229: process: 0.55434 | load: 0.00251 [I 2022-10-27 16:59:23] ...trap/engines/engine.py.230: loss: 0.14926 [I 2022-10-27 16:59:23] ...optimizers/trijoint.py.151: total_norm: 50.332703 [I 2022-10-27 16:59:29] ...trap/engines/engine.py.224: train: epoch 78 | batch 590/2304 [I 2022-10-27 16:59:29] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:59 [I 2022-10-27 16:59:29] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00247 [I 2022-10-27 16:59:29] ...trap/engines/engine.py.230: loss: 0.16549 [I 2022-10-27 16:59:29] ...optimizers/trijoint.py.151: total_norm: 8.472914 [I 2022-10-27 16:59:34] ...trap/engines/engine.py.224: train: epoch 78 | batch 600/2304 [I 2022-10-27 16:59:34] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:50 [I 2022-10-27 16:59:34] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00246 [I 2022-10-27 16:59:34] ...trap/engines/engine.py.230: loss: 0.30565 [I 2022-10-27 16:59:34] ...optimizers/trijoint.py.151: total_norm: 36.575981 [I 2022-10-27 16:59:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 610/2304 [I 2022-10-27 16:59:40] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:46 [I 2022-10-27 16:59:40] ...trap/engines/engine.py.229: process: 0.55272 | load: 0.00246 [I 2022-10-27 16:59:40] ...trap/engines/engine.py.230: loss: 0.10916 [I 2022-10-27 16:59:40] ...optimizers/trijoint.py.151: total_norm: 22.075167 [I 2022-10-27 16:59:45] ...trap/engines/engine.py.224: train: epoch 78 | batch 620/2304 [I 2022-10-27 16:59:45] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:36 [I 2022-10-27 16:59:45] ...trap/engines/engine.py.229: process: 0.54637 | load: 0.00247 [I 2022-10-27 16:59:45] ...trap/engines/engine.py.230: loss: 0.15518 [I 2022-10-27 16:59:45] ...optimizers/trijoint.py.151: total_norm: 15.520049 [I 2022-10-27 16:59:51] ...trap/engines/engine.py.224: train: epoch 78 | batch 630/2304 [I 2022-10-27 16:59:51] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:29 [I 2022-10-27 16:59:51] ...trap/engines/engine.py.229: process: 0.55126 | load: 0.00242 [I 2022-10-27 16:59:51] ...trap/engines/engine.py.230: loss: 0.11043 [I 2022-10-27 16:59:51] ...optimizers/trijoint.py.151: total_norm: 20.025898 [I 2022-10-27 16:59:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 640/2304 [I 2022-10-27 16:59:57] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:49 [I 2022-10-27 16:59:57] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00243 [I 2022-10-27 16:59:57] ...trap/engines/engine.py.230: loss: 0.14517 [I 2022-10-27 16:59:57] ...optimizers/trijoint.py.151: total_norm: 18.107550 [I 2022-10-27 17:00:02] ...trap/engines/engine.py.224: train: epoch 78 | batch 650/2304 [I 2022-10-27 17:00:02] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:23 [I 2022-10-27 17:00:02] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00241 [I 2022-10-27 17:00:02] ...trap/engines/engine.py.230: loss: 0.13258 [I 2022-10-27 17:00:02] ...optimizers/trijoint.py.151: total_norm: 21.686792 [I 2022-10-27 17:00:08] ...trap/engines/engine.py.224: train: epoch 78 | batch 660/2304 [I 2022-10-27 17:00:08] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:24 [I 2022-10-27 17:00:08] ...trap/engines/engine.py.229: process: 0.55942 | load: 0.00243 [I 2022-10-27 17:00:08] ...trap/engines/engine.py.230: loss: 0.17175 [I 2022-10-27 17:00:08] ...optimizers/trijoint.py.151: total_norm: 9.899224 [I 2022-10-27 17:00:13] ...trap/engines/engine.py.224: train: epoch 78 | batch 670/2304 [I 2022-10-27 17:00:13] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:08 [I 2022-10-27 17:00:13] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00251 [I 2022-10-27 17:00:13] ...trap/engines/engine.py.230: loss: 0.10965 [I 2022-10-27 17:00:13] ...optimizers/trijoint.py.151: total_norm: 12.939528 [I 2022-10-27 17:00:19] ...trap/engines/engine.py.224: train: epoch 78 | batch 680/2304 [I 2022-10-27 17:00:19] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:14:58 [I 2022-10-27 17:00:19] ...trap/engines/engine.py.229: process: 0.55233 | load: 0.00253 [I 2022-10-27 17:00:19] ...trap/engines/engine.py.230: loss: 0.08913 [I 2022-10-27 17:00:19] ...optimizers/trijoint.py.151: total_norm: 12.514897 [I 2022-10-27 17:00:25] ...trap/engines/engine.py.224: train: epoch 78 | batch 690/2304 [I 2022-10-27 17:00:25] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:58 [I 2022-10-27 17:00:25] ...trap/engines/engine.py.229: process: 0.55337 | load: 0.00246 [I 2022-10-27 17:00:25] ...trap/engines/engine.py.230: loss: 0.11270 [I 2022-10-27 17:00:25] ...optimizers/trijoint.py.151: total_norm: 8.700752 [I 2022-10-27 17:00:30] ...trap/engines/engine.py.224: train: epoch 78 | batch 700/2304 [I 2022-10-27 17:00:30] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:52 [I 2022-10-27 17:00:30] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00249 [I 2022-10-27 17:00:30] ...trap/engines/engine.py.230: loss: 0.10559 [I 2022-10-27 17:00:30] ...optimizers/trijoint.py.151: total_norm: 15.877556 [I 2022-10-27 17:00:36] ...trap/engines/engine.py.224: train: epoch 78 | batch 710/2304 [I 2022-10-27 17:00:36] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:43 [I 2022-10-27 17:00:36] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00258 [I 2022-10-27 17:00:36] ...trap/engines/engine.py.230: loss: 0.09602 [I 2022-10-27 17:00:36] ...optimizers/trijoint.py.151: total_norm: 8.719006 [I 2022-10-27 17:00:41] ...trap/engines/engine.py.224: train: epoch 78 | batch 720/2304 [I 2022-10-27 17:00:41] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:37 [I 2022-10-27 17:00:41] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00253 [I 2022-10-27 17:00:41] ...trap/engines/engine.py.230: loss: 0.19978 [I 2022-10-27 17:00:41] ...optimizers/trijoint.py.151: total_norm: 12.677113 [I 2022-10-27 17:00:47] ...trap/engines/engine.py.224: train: epoch 78 | batch 730/2304 [I 2022-10-27 17:00:47] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:41 [I 2022-10-27 17:00:47] ...trap/engines/engine.py.229: process: 0.56643 | load: 0.00257 [I 2022-10-27 17:00:47] ...trap/engines/engine.py.230: loss: 0.20192 [I 2022-10-27 17:00:47] ...optimizers/trijoint.py.151: total_norm: 19.245749 [I 2022-10-27 17:00:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 740/2304 [I 2022-10-27 17:00:52] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:39 [I 2022-10-27 17:00:52] ...trap/engines/engine.py.229: process: 0.55105 | load: 0.00268 [I 2022-10-27 17:00:52] ...trap/engines/engine.py.230: loss: 0.12484 [I 2022-10-27 17:00:52] ...optimizers/trijoint.py.151: total_norm: 14.155343 [I 2022-10-27 17:00:58] ...trap/engines/engine.py.224: train: epoch 78 | batch 750/2304 [I 2022-10-27 17:00:58] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:24 [I 2022-10-27 17:00:58] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00254 [I 2022-10-27 17:00:58] ...trap/engines/engine.py.230: loss: 0.15226 [I 2022-10-27 17:00:58] ...optimizers/trijoint.py.151: total_norm: 15.264300 [I 2022-10-27 17:01:04] ...trap/engines/engine.py.224: train: epoch 78 | batch 760/2304 [I 2022-10-27 17:01:04] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:32 [I 2022-10-27 17:01:04] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00270 [I 2022-10-27 17:01:04] ...trap/engines/engine.py.230: loss: 0.15072 [I 2022-10-27 17:01:04] ...optimizers/trijoint.py.151: total_norm: 13.591962 [I 2022-10-27 17:01:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 770/2304 [I 2022-10-27 17:01:09] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:11 [I 2022-10-27 17:01:09] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00248 [I 2022-10-27 17:01:09] ...trap/engines/engine.py.230: loss: 0.12183 [I 2022-10-27 17:01:09] ...optimizers/trijoint.py.151: total_norm: 19.714069 [I 2022-10-27 17:01:15] ...trap/engines/engine.py.224: train: epoch 78 | batch 780/2304 [I 2022-10-27 17:01:15] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:06 [I 2022-10-27 17:01:15] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00266 [I 2022-10-27 17:01:15] ...trap/engines/engine.py.230: loss: 0.14504 [I 2022-10-27 17:01:15] ...optimizers/trijoint.py.151: total_norm: 20.078074 [I 2022-10-27 17:01:20] ...trap/engines/engine.py.224: train: epoch 78 | batch 790/2304 [I 2022-10-27 17:01:20] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:13:55 [I 2022-10-27 17:01:20] ...trap/engines/engine.py.229: process: 0.54132 | load: 0.00244 [I 2022-10-27 17:01:20] ...trap/engines/engine.py.230: loss: 0.14404 [I 2022-10-27 17:01:20] ...optimizers/trijoint.py.151: total_norm: 20.043884 [I 2022-10-27 17:01:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 800/2304 [I 2022-10-27 17:01:26] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:52 [I 2022-10-27 17:01:26] ...trap/engines/engine.py.229: process: 0.55058 | load: 0.00248 [I 2022-10-27 17:01:26] ...trap/engines/engine.py.230: loss: 0.20065 [I 2022-10-27 17:01:26] ...optimizers/trijoint.py.151: total_norm: 12.182396 [I 2022-10-27 17:01:31] ...trap/engines/engine.py.224: train: epoch 78 | batch 810/2304 [I 2022-10-27 17:01:31] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:47 [I 2022-10-27 17:01:31] ...trap/engines/engine.py.229: process: 0.55073 | load: 0.00241 [I 2022-10-27 17:01:31] ...trap/engines/engine.py.230: loss: 0.09974 [I 2022-10-27 17:01:31] ...optimizers/trijoint.py.151: total_norm: 7.410059 [I 2022-10-27 17:01:37] ...trap/engines/engine.py.224: train: epoch 78 | batch 820/2304 [I 2022-10-27 17:01:37] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:42 [I 2022-10-27 17:01:37] ...trap/engines/engine.py.229: process: 0.55259 | load: 0.00246 [I 2022-10-27 17:01:37] ...trap/engines/engine.py.230: loss: 0.16747 [I 2022-10-27 17:01:37] ...optimizers/trijoint.py.151: total_norm: 18.075317 [I 2022-10-27 17:01:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 830/2304 [I 2022-10-27 17:01:43] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:35 [I 2022-10-27 17:01:43] ...trap/engines/engine.py.229: process: 0.55114 | load: 0.00259 [I 2022-10-27 17:01:43] ...trap/engines/engine.py.230: loss: 0.14515 [I 2022-10-27 17:01:43] ...optimizers/trijoint.py.151: total_norm: 13.045934 [I 2022-10-27 17:01:48] ...trap/engines/engine.py.224: train: epoch 78 | batch 840/2304 [I 2022-10-27 17:01:48] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:29 [I 2022-10-27 17:01:48] ...trap/engines/engine.py.229: process: 0.55400 | load: 0.00255 [I 2022-10-27 17:01:48] ...trap/engines/engine.py.230: loss: 0.10487 [I 2022-10-27 17:01:48] ...optimizers/trijoint.py.151: total_norm: 10.236753 [I 2022-10-27 17:01:54] ...trap/engines/engine.py.224: train: epoch 78 | batch 850/2304 [I 2022-10-27 17:01:54] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:32 [I 2022-10-27 17:01:54] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00245 [I 2022-10-27 17:01:54] ...trap/engines/engine.py.230: loss: 0.14474 [I 2022-10-27 17:01:54] ...optimizers/trijoint.py.151: total_norm: 13.197285 [I 2022-10-27 17:01:59] ...trap/engines/engine.py.224: train: epoch 78 | batch 860/2304 [I 2022-10-27 17:01:59] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:26 [I 2022-10-27 17:01:59] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00244 [I 2022-10-27 17:01:59] ...trap/engines/engine.py.230: loss: 0.14799 [I 2022-10-27 17:01:59] ...optimizers/trijoint.py.151: total_norm: 33.599213 [I 2022-10-27 17:02:05] ...trap/engines/engine.py.224: train: epoch 78 | batch 870/2304 [I 2022-10-27 17:02:05] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:22 [I 2022-10-27 17:02:05] ...trap/engines/engine.py.229: process: 0.56329 | load: 0.00244 [I 2022-10-27 17:02:05] ...trap/engines/engine.py.230: loss: 0.12611 [I 2022-10-27 17:02:05] ...optimizers/trijoint.py.151: total_norm: 56.874496 [I 2022-10-27 17:02:11] ...trap/engines/engine.py.224: train: epoch 78 | batch 880/2304 [I 2022-10-27 17:02:11] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:16 [I 2022-10-27 17:02:11] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00244 [I 2022-10-27 17:02:11] ...trap/engines/engine.py.230: loss: 0.16025 [I 2022-10-27 17:02:11] ...optimizers/trijoint.py.151: total_norm: 13.394079 [I 2022-10-27 17:02:16] ...trap/engines/engine.py.224: train: epoch 78 | batch 890/2304 [I 2022-10-27 17:02:16] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:12 [I 2022-10-27 17:02:16] ...trap/engines/engine.py.229: process: 0.56499 | load: 0.00246 [I 2022-10-27 17:02:16] ...trap/engines/engine.py.230: loss: 0.24746 [I 2022-10-27 17:02:16] ...optimizers/trijoint.py.151: total_norm: 14.960269 [I 2022-10-27 17:02:22] ...trap/engines/engine.py.224: train: epoch 78 | batch 900/2304 [I 2022-10-27 17:02:22] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:03 [I 2022-10-27 17:02:22] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00243 [I 2022-10-27 17:02:22] ...trap/engines/engine.py.230: loss: 0.11941 [I 2022-10-27 17:02:22] ...optimizers/trijoint.py.151: total_norm: 45.051460 [I 2022-10-27 17:02:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 910/2304 [I 2022-10-27 17:02:27] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:56 [I 2022-10-27 17:02:27] ...trap/engines/engine.py.229: process: 0.54680 | load: 0.00248 [I 2022-10-27 17:02:27] ...trap/engines/engine.py.230: loss: 0.19746 [I 2022-10-27 17:02:27] ...optimizers/trijoint.py.151: total_norm: 13.001425 [I 2022-10-27 17:02:33] ...trap/engines/engine.py.224: train: epoch 78 | batch 920/2304 [I 2022-10-27 17:02:33] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:55 [I 2022-10-27 17:02:33] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00248 [I 2022-10-27 17:02:33] ...trap/engines/engine.py.230: loss: 0.10132 [I 2022-10-27 17:02:33] ...optimizers/trijoint.py.151: total_norm: 21.298393 [I 2022-10-27 17:02:38] ...trap/engines/engine.py.224: train: epoch 78 | batch 930/2304 [I 2022-10-27 17:02:38] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:42 [I 2022-10-27 17:02:38] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00245 [I 2022-10-27 17:02:38] ...trap/engines/engine.py.230: loss: 0.13342 [I 2022-10-27 17:02:38] ...optimizers/trijoint.py.151: total_norm: 14.334099 [I 2022-10-27 17:02:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 940/2304 [I 2022-10-27 17:02:44] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:38 [I 2022-10-27 17:02:44] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00245 [I 2022-10-27 17:02:44] ...trap/engines/engine.py.230: loss: 0.18318 [I 2022-10-27 17:02:44] ...optimizers/trijoint.py.151: total_norm: 30.274927 [I 2022-10-27 17:02:50] ...trap/engines/engine.py.224: train: epoch 78 | batch 950/2304 [I 2022-10-27 17:02:50] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:31 [I 2022-10-27 17:02:50] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00245 [I 2022-10-27 17:02:50] ...trap/engines/engine.py.230: loss: 0.16787 [I 2022-10-27 17:02:50] ...optimizers/trijoint.py.151: total_norm: 12.527246 [I 2022-10-27 17:02:55] ...trap/engines/engine.py.224: train: epoch 78 | batch 960/2304 [I 2022-10-27 17:02:55] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:26 [I 2022-10-27 17:02:55] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00243 [I 2022-10-27 17:02:55] ...trap/engines/engine.py.230: loss: 0.13727 [I 2022-10-27 17:02:55] ...optimizers/trijoint.py.151: total_norm: 12.979113 [I 2022-10-27 17:03:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 970/2304 [I 2022-10-27 17:03:01] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:26 [I 2022-10-27 17:03:01] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00241 [I 2022-10-27 17:03:01] ...trap/engines/engine.py.230: loss: 0.10186 [I 2022-10-27 17:03:01] ...optimizers/trijoint.py.151: total_norm: 12.431168 [I 2022-10-27 17:03:06] ...trap/engines/engine.py.224: train: epoch 78 | batch 980/2304 [I 2022-10-27 17:03:06] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:16 [I 2022-10-27 17:03:06] ...trap/engines/engine.py.229: process: 0.54891 | load: 0.00329 [I 2022-10-27 17:03:06] ...trap/engines/engine.py.230: loss: 0.11417 [I 2022-10-27 17:03:06] ...optimizers/trijoint.py.151: total_norm: 9.221694 [I 2022-10-27 17:03:12] ...trap/engines/engine.py.224: train: epoch 78 | batch 990/2304 [I 2022-10-27 17:03:12] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:14 [I 2022-10-27 17:03:12] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00246 [I 2022-10-27 17:03:12] ...trap/engines/engine.py.230: loss: 0.15498 [I 2022-10-27 17:03:12] ...optimizers/trijoint.py.151: total_norm: 16.290373 [I 2022-10-27 17:03:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 1000/2304 [I 2022-10-27 17:03:18] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:07 [I 2022-10-27 17:03:18] ...trap/engines/engine.py.229: process: 0.56115 | load: 0.00256 [I 2022-10-27 17:03:18] ...trap/engines/engine.py.230: loss: 0.10990 [I 2022-10-27 17:03:18] ...optimizers/trijoint.py.151: total_norm: 16.449039 [I 2022-10-27 17:03:23] ...trap/engines/engine.py.224: train: epoch 78 | batch 1010/2304 [I 2022-10-27 17:03:23] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:00 [I 2022-10-27 17:03:23] ...trap/engines/engine.py.229: process: 0.56312 | load: 0.00248 [I 2022-10-27 17:03:23] ...trap/engines/engine.py.230: loss: 0.12141 [I 2022-10-27 17:03:23] ...optimizers/trijoint.py.151: total_norm: 9.953220 [I 2022-10-27 17:03:29] ...trap/engines/engine.py.224: train: epoch 78 | batch 1020/2304 [I 2022-10-27 17:03:29] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:56 [I 2022-10-27 17:03:29] ...trap/engines/engine.py.229: process: 0.56298 | load: 0.00251 [I 2022-10-27 17:03:29] ...trap/engines/engine.py.230: loss: 0.10736 [I 2022-10-27 17:03:29] ...optimizers/trijoint.py.151: total_norm: 8.358392 [I 2022-10-27 17:03:34] ...trap/engines/engine.py.224: train: epoch 78 | batch 1030/2304 [I 2022-10-27 17:03:34] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:47 [I 2022-10-27 17:03:34] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00252 [I 2022-10-27 17:03:34] ...trap/engines/engine.py.230: loss: 0.13391 [I 2022-10-27 17:03:34] ...optimizers/trijoint.py.151: total_norm: 13.864944 [I 2022-10-27 17:03:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 1040/2304 [I 2022-10-27 17:03:40] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:42 [I 2022-10-27 17:03:40] ...trap/engines/engine.py.229: process: 0.55070 | load: 0.00245 [I 2022-10-27 17:03:40] ...trap/engines/engine.py.230: loss: 0.11660 [I 2022-10-27 17:03:40] ...optimizers/trijoint.py.151: total_norm: 10.919671 [I 2022-10-27 17:03:45] ...trap/engines/engine.py.224: train: epoch 78 | batch 1050/2304 [I 2022-10-27 17:03:45] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:41 [I 2022-10-27 17:03:45] ...trap/engines/engine.py.229: process: 0.56554 | load: 0.00262 [I 2022-10-27 17:03:45] ...trap/engines/engine.py.230: loss: 0.23994 [I 2022-10-27 17:03:45] ...optimizers/trijoint.py.151: total_norm: 21.999443 [I 2022-10-27 17:03:51] ...trap/engines/engine.py.224: train: epoch 78 | batch 1060/2304 [I 2022-10-27 17:03:51] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:34 [I 2022-10-27 17:03:51] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00241 [I 2022-10-27 17:03:51] ...trap/engines/engine.py.230: loss: 0.09994 [I 2022-10-27 17:03:51] ...optimizers/trijoint.py.151: total_norm: 11.180059 [I 2022-10-27 17:03:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 1070/2304 [I 2022-10-27 17:03:57] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:22 [I 2022-10-27 17:03:57] ...trap/engines/engine.py.229: process: 0.54379 | load: 0.00247 [I 2022-10-27 17:03:57] ...trap/engines/engine.py.230: loss: 0.13800 [I 2022-10-27 17:03:57] ...optimizers/trijoint.py.151: total_norm: 18.370529 [I 2022-10-27 17:04:02] ...trap/engines/engine.py.224: train: epoch 78 | batch 1080/2304 [I 2022-10-27 17:04:02] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:22 [I 2022-10-27 17:04:02] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00245 [I 2022-10-27 17:04:02] ...trap/engines/engine.py.230: loss: 0.14531 [I 2022-10-27 17:04:02] ...optimizers/trijoint.py.151: total_norm: 9.376461 [I 2022-10-27 17:04:08] ...trap/engines/engine.py.224: train: epoch 78 | batch 1090/2304 [I 2022-10-27 17:04:08] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:12 [I 2022-10-27 17:04:08] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00244 [I 2022-10-27 17:04:08] ...trap/engines/engine.py.230: loss: 0.21410 [I 2022-10-27 17:04:08] ...optimizers/trijoint.py.151: total_norm: 33.482948 [I 2022-10-27 17:04:13] ...trap/engines/engine.py.224: train: epoch 78 | batch 1100/2304 [I 2022-10-27 17:04:13] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:13 [I 2022-10-27 17:04:13] ...trap/engines/engine.py.229: process: 0.56234 | load: 0.00246 [I 2022-10-27 17:04:13] ...trap/engines/engine.py.230: loss: 0.15034 [I 2022-10-27 17:04:13] ...optimizers/trijoint.py.151: total_norm: 18.927317 [I 2022-10-27 17:04:19] ...trap/engines/engine.py.224: train: epoch 78 | batch 1110/2304 [I 2022-10-27 17:04:19] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:14 [I 2022-10-27 17:04:19] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00256 [I 2022-10-27 17:04:19] ...trap/engines/engine.py.230: loss: 0.16344 [I 2022-10-27 17:04:19] ...optimizers/trijoint.py.151: total_norm: 17.556686 [I 2022-10-27 17:04:25] ...trap/engines/engine.py.224: train: epoch 78 | batch 1120/2304 [I 2022-10-27 17:04:25] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:01 [I 2022-10-27 17:04:25] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00245 [I 2022-10-27 17:04:25] ...trap/engines/engine.py.230: loss: 0.10728 [I 2022-10-27 17:04:25] ...optimizers/trijoint.py.151: total_norm: 8.700990 [I 2022-10-27 17:04:30] ...trap/engines/engine.py.224: train: epoch 78 | batch 1130/2304 [I 2022-10-27 17:04:30] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:55 [I 2022-10-27 17:04:30] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00247 [I 2022-10-27 17:04:30] ...trap/engines/engine.py.230: loss: 0.10295 [I 2022-10-27 17:04:30] ...optimizers/trijoint.py.151: total_norm: 18.718342 [I 2022-10-27 17:04:36] ...trap/engines/engine.py.224: train: epoch 78 | batch 1140/2304 [I 2022-10-27 17:04:36] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:47 [I 2022-10-27 17:04:36] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00243 [I 2022-10-27 17:04:36] ...trap/engines/engine.py.230: loss: 0.14283 [I 2022-10-27 17:04:36] ...optimizers/trijoint.py.151: total_norm: 29.642401 [I 2022-10-27 17:04:41] ...trap/engines/engine.py.224: train: epoch 78 | batch 1150/2304 [I 2022-10-27 17:04:41] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:42 [I 2022-10-27 17:04:41] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00243 [I 2022-10-27 17:04:41] ...trap/engines/engine.py.230: loss: 0.12335 [I 2022-10-27 17:04:41] ...optimizers/trijoint.py.151: total_norm: 16.301863 [I 2022-10-27 17:04:47] ...trap/engines/engine.py.224: train: epoch 78 | batch 1160/2304 [I 2022-10-27 17:04:47] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:38 [I 2022-10-27 17:04:47] ...trap/engines/engine.py.229: process: 0.57092 | load: 0.00244 [I 2022-10-27 17:04:47] ...trap/engines/engine.py.230: loss: 0.14159 [I 2022-10-27 17:04:47] ...optimizers/trijoint.py.151: total_norm: 19.122189 [I 2022-10-27 17:04:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 1170/2304 [I 2022-10-27 17:04:52] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:35 [I 2022-10-27 17:04:52] ...trap/engines/engine.py.229: process: 0.56212 | load: 0.00260 [I 2022-10-27 17:04:52] ...trap/engines/engine.py.230: loss: 0.13473 [I 2022-10-27 17:04:52] ...optimizers/trijoint.py.151: total_norm: 29.556522 [I 2022-10-27 17:04:58] ...trap/engines/engine.py.224: train: epoch 78 | batch 1180/2304 [I 2022-10-27 17:04:58] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:22 [I 2022-10-27 17:04:58] ...trap/engines/engine.py.229: process: 0.54921 | load: 0.00246 [I 2022-10-27 17:04:58] ...trap/engines/engine.py.230: loss: 0.16547 [I 2022-10-27 17:04:58] ...optimizers/trijoint.py.151: total_norm: 22.440321 [I 2022-10-27 17:05:04] ...trap/engines/engine.py.224: train: epoch 78 | batch 1190/2304 [I 2022-10-27 17:05:04] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:24 [I 2022-10-27 17:05:04] ...trap/engines/engine.py.229: process: 0.56741 | load: 0.00281 [I 2022-10-27 17:05:04] ...trap/engines/engine.py.230: loss: 0.12836 [I 2022-10-27 17:05:04] ...optimizers/trijoint.py.151: total_norm: 16.033091 [I 2022-10-27 17:05:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 1200/2304 [I 2022-10-27 17:05:09] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:11 [I 2022-10-27 17:05:09] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00243 [I 2022-10-27 17:05:09] ...trap/engines/engine.py.230: loss: 0.11249 [I 2022-10-27 17:05:09] ...optimizers/trijoint.py.151: total_norm: 13.827762 [I 2022-10-27 17:05:15] ...trap/engines/engine.py.224: train: epoch 78 | batch 1210/2304 [I 2022-10-27 17:05:15] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:10 [I 2022-10-27 17:05:15] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00253 [I 2022-10-27 17:05:15] ...trap/engines/engine.py.230: loss: 0.14207 [I 2022-10-27 17:05:15] ...optimizers/trijoint.py.151: total_norm: 13.371694 [I 2022-10-27 17:05:20] ...trap/engines/engine.py.224: train: epoch 78 | batch 1220/2304 [I 2022-10-27 17:05:20] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:06 [I 2022-10-27 17:05:20] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00247 [I 2022-10-27 17:05:20] ...trap/engines/engine.py.230: loss: 0.14853 [I 2022-10-27 17:05:20] ...optimizers/trijoint.py.151: total_norm: 11.170877 [I 2022-10-27 17:05:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 1230/2304 [I 2022-10-27 17:05:26] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:56 [I 2022-10-27 17:05:26] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00241 [I 2022-10-27 17:05:26] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-27 17:05:26] ...optimizers/trijoint.py.151: total_norm: 18.151661 [I 2022-10-27 17:05:31] ...trap/engines/engine.py.224: train: epoch 78 | batch 1240/2304 [I 2022-10-27 17:05:31] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:54 [I 2022-10-27 17:05:31] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00246 [I 2022-10-27 17:05:31] ...trap/engines/engine.py.230: loss: 0.17447 [I 2022-10-27 17:05:31] ...optimizers/trijoint.py.151: total_norm: 34.717842 [I 2022-10-27 17:05:37] ...trap/engines/engine.py.224: train: epoch 78 | batch 1250/2304 [I 2022-10-27 17:05:37] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:49 [I 2022-10-27 17:05:37] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00247 [I 2022-10-27 17:05:37] ...trap/engines/engine.py.230: loss: 0.19565 [I 2022-10-27 17:05:37] ...optimizers/trijoint.py.151: total_norm: 9.240129 [I 2022-10-27 17:05:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 1260/2304 [I 2022-10-27 17:05:43] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:40 [I 2022-10-27 17:05:43] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00247 [I 2022-10-27 17:05:43] ...trap/engines/engine.py.230: loss: 0.25724 [I 2022-10-27 17:05:43] ...optimizers/trijoint.py.151: total_norm: 31.046667 [I 2022-10-27 17:05:48] ...trap/engines/engine.py.224: train: epoch 78 | batch 1270/2304 [I 2022-10-27 17:05:48] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:36 [I 2022-10-27 17:05:48] ...trap/engines/engine.py.229: process: 0.56303 | load: 0.00253 [I 2022-10-27 17:05:48] ...trap/engines/engine.py.230: loss: 0.15118 [I 2022-10-27 17:05:48] ...optimizers/trijoint.py.151: total_norm: 11.877832 [I 2022-10-27 17:05:54] ...trap/engines/engine.py.224: train: epoch 78 | batch 1280/2304 [I 2022-10-27 17:05:54] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:29 [I 2022-10-27 17:05:54] ...trap/engines/engine.py.229: process: 0.55831 | load: 0.00250 [I 2022-10-27 17:05:54] ...trap/engines/engine.py.230: loss: 0.12047 [I 2022-10-27 17:05:54] ...optimizers/trijoint.py.151: total_norm: 29.399658 [I 2022-10-27 17:05:59] ...trap/engines/engine.py.224: train: epoch 78 | batch 1290/2304 [I 2022-10-27 17:05:59] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:25 [I 2022-10-27 17:05:59] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00255 [I 2022-10-27 17:05:59] ...trap/engines/engine.py.230: loss: 0.13512 [I 2022-10-27 17:05:59] ...optimizers/trijoint.py.151: total_norm: 13.425075 [I 2022-10-27 17:06:05] ...trap/engines/engine.py.224: train: epoch 78 | batch 1300/2304 [I 2022-10-27 17:06:05] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:14 [I 2022-10-27 17:06:05] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00247 [I 2022-10-27 17:06:05] ...trap/engines/engine.py.230: loss: 0.09745 [I 2022-10-27 17:06:05] ...optimizers/trijoint.py.151: total_norm: 12.966347 [I 2022-10-27 17:06:11] ...trap/engines/engine.py.224: train: epoch 78 | batch 1310/2304 [I 2022-10-27 17:06:11] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:15 [I 2022-10-27 17:06:11] ...trap/engines/engine.py.229: process: 0.56941 | load: 0.00245 [I 2022-10-27 17:06:11] ...trap/engines/engine.py.230: loss: 0.13489 [I 2022-10-27 17:06:11] ...optimizers/trijoint.py.151: total_norm: 8.394457 [I 2022-10-27 17:06:16] ...trap/engines/engine.py.224: train: epoch 78 | batch 1320/2304 [I 2022-10-27 17:06:16] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:05 [I 2022-10-27 17:06:16] ...trap/engines/engine.py.229: process: 0.55544 | load: 0.00245 [I 2022-10-27 17:06:16] ...trap/engines/engine.py.230: loss: 0.21271 [I 2022-10-27 17:06:16] ...optimizers/trijoint.py.151: total_norm: 31.912657 [I 2022-10-27 17:06:22] ...trap/engines/engine.py.224: train: epoch 78 | batch 1330/2304 [I 2022-10-27 17:06:22] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:16 [I 2022-10-27 17:06:22] ...trap/engines/engine.py.229: process: 0.57187 | load: 0.00243 [I 2022-10-27 17:06:22] ...trap/engines/engine.py.230: loss: 0.11080 [I 2022-10-27 17:06:22] ...optimizers/trijoint.py.151: total_norm: 13.413973 [I 2022-10-27 17:06:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 1340/2304 [I 2022-10-27 17:06:27] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:09:08 [I 2022-10-27 17:06:27] ...trap/engines/engine.py.229: process: 0.61289 | load: 0.00245 [I 2022-10-27 17:06:27] ...trap/engines/engine.py.230: loss: 0.17256 [I 2022-10-27 17:06:27] ...optimizers/trijoint.py.151: total_norm: 17.550856 [I 2022-10-27 17:06:33] ...trap/engines/engine.py.224: train: epoch 78 | batch 1350/2304 [I 2022-10-27 17:06:33] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:55 [I 2022-10-27 17:06:33] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00243 [I 2022-10-27 17:06:33] ...trap/engines/engine.py.230: loss: 0.10469 [I 2022-10-27 17:06:33] ...optimizers/trijoint.py.151: total_norm: 7.625627 [I 2022-10-27 17:06:39] ...trap/engines/engine.py.224: train: epoch 78 | batch 1360/2304 [I 2022-10-27 17:06:39] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:45 [I 2022-10-27 17:06:39] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00244 [I 2022-10-27 17:06:39] ...trap/engines/engine.py.230: loss: 0.19355 [I 2022-10-27 17:06:39] ...optimizers/trijoint.py.151: total_norm: 24.240196 [I 2022-10-27 17:06:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 1370/2304 [I 2022-10-27 17:06:44] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:40 [I 2022-10-27 17:06:44] ...trap/engines/engine.py.229: process: 0.55571 | load: 0.00241 [I 2022-10-27 17:06:44] ...trap/engines/engine.py.230: loss: 0.11795 [I 2022-10-27 17:06:44] ...optimizers/trijoint.py.151: total_norm: 15.043672 [I 2022-10-27 17:06:50] ...trap/engines/engine.py.224: train: epoch 78 | batch 1380/2304 [I 2022-10-27 17:06:50] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:35 [I 2022-10-27 17:06:50] ...trap/engines/engine.py.229: process: 0.56296 | load: 0.00242 [I 2022-10-27 17:06:50] ...trap/engines/engine.py.230: loss: 0.15735 [I 2022-10-27 17:06:50] ...optimizers/trijoint.py.151: total_norm: 17.694109 [I 2022-10-27 17:06:55] ...trap/engines/engine.py.224: train: epoch 78 | batch 1390/2304 [I 2022-10-27 17:06:55] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:34 [I 2022-10-27 17:06:55] ...trap/engines/engine.py.229: process: 0.57521 | load: 0.00241 [I 2022-10-27 17:06:55] ...trap/engines/engine.py.230: loss: 0.24064 [I 2022-10-27 17:06:55] ...optimizers/trijoint.py.151: total_norm: 23.515221 [I 2022-10-27 17:07:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 1400/2304 [I 2022-10-27 17:07:01] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:26 [I 2022-10-27 17:07:01] ...trap/engines/engine.py.229: process: 0.56448 | load: 0.00252 [I 2022-10-27 17:07:01] ...trap/engines/engine.py.230: loss: 0.15384 [I 2022-10-27 17:07:01] ...optimizers/trijoint.py.151: total_norm: 15.899172 [I 2022-10-27 17:07:07] ...trap/engines/engine.py.224: train: epoch 78 | batch 1410/2304 [I 2022-10-27 17:07:07] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:20 [I 2022-10-27 17:07:07] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00246 [I 2022-10-27 17:07:07] ...trap/engines/engine.py.230: loss: 0.10353 [I 2022-10-27 17:07:07] ...optimizers/trijoint.py.151: total_norm: 13.343701 [I 2022-10-27 17:07:12] ...trap/engines/engine.py.224: train: epoch 78 | batch 1420/2304 [I 2022-10-27 17:07:12] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:30 [I 2022-10-27 17:07:12] ...trap/engines/engine.py.229: process: 0.56290 | load: 0.00246 [I 2022-10-27 17:07:12] ...trap/engines/engine.py.230: loss: 0.12759 [I 2022-10-27 17:07:12] ...optimizers/trijoint.py.151: total_norm: 16.108969 [I 2022-10-27 17:07:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 1430/2304 [I 2022-10-27 17:07:18] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:05 [I 2022-10-27 17:07:18] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00248 [I 2022-10-27 17:07:18] ...trap/engines/engine.py.230: loss: 0.17532 [I 2022-10-27 17:07:18] ...optimizers/trijoint.py.151: total_norm: 17.951820 [I 2022-10-27 17:07:24] ...trap/engines/engine.py.224: train: epoch 78 | batch 1440/2304 [I 2022-10-27 17:07:24] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:03 [I 2022-10-27 17:07:24] ...trap/engines/engine.py.229: process: 0.57015 | load: 0.00246 [I 2022-10-27 17:07:24] ...trap/engines/engine.py.230: loss: 0.13803 [I 2022-10-27 17:07:24] ...optimizers/trijoint.py.151: total_norm: 12.888726 [I 2022-10-27 17:07:29] ...trap/engines/engine.py.224: train: epoch 78 | batch 1450/2304 [I 2022-10-27 17:07:29] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:57 [I 2022-10-27 17:07:29] ...trap/engines/engine.py.229: process: 0.56820 | load: 0.00245 [I 2022-10-27 17:07:29] ...trap/engines/engine.py.230: loss: 0.11313 [I 2022-10-27 17:07:29] ...optimizers/trijoint.py.151: total_norm: 11.544529 [I 2022-10-27 17:07:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 1460/2304 [I 2022-10-27 17:07:35] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:48 [I 2022-10-27 17:07:35] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00245 [I 2022-10-27 17:07:35] ...trap/engines/engine.py.230: loss: 0.09120 [I 2022-10-27 17:07:35] ...optimizers/trijoint.py.151: total_norm: 10.351290 [I 2022-10-27 17:07:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 1470/2304 [I 2022-10-27 17:07:40] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:45 [I 2022-10-27 17:07:40] ...trap/engines/engine.py.229: process: 0.55471 | load: 0.00250 [I 2022-10-27 17:07:40] ...trap/engines/engine.py.230: loss: 0.13350 [I 2022-10-27 17:07:40] ...optimizers/trijoint.py.151: total_norm: 15.611820 [I 2022-10-27 17:07:46] ...trap/engines/engine.py.224: train: epoch 78 | batch 1480/2304 [I 2022-10-27 17:07:46] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:56 [I 2022-10-27 17:07:46] ...trap/engines/engine.py.229: process: 0.64945 | load: 0.00247 [I 2022-10-27 17:07:46] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-27 17:07:46] ...optimizers/trijoint.py.151: total_norm: 9.592043 [I 2022-10-27 17:07:52] ...trap/engines/engine.py.224: train: epoch 78 | batch 1490/2304 [I 2022-10-27 17:07:52] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:37 [I 2022-10-27 17:07:52] ...trap/engines/engine.py.229: process: 0.56386 | load: 0.00247 [I 2022-10-27 17:07:52] ...trap/engines/engine.py.230: loss: 0.11434 [I 2022-10-27 17:07:52] ...optimizers/trijoint.py.151: total_norm: 21.536676 [I 2022-10-27 17:07:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 1500/2304 [I 2022-10-27 17:07:57] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:26 [I 2022-10-27 17:07:57] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00248 [I 2022-10-27 17:07:57] ...trap/engines/engine.py.230: loss: 0.14975 [I 2022-10-27 17:07:57] ...optimizers/trijoint.py.151: total_norm: 16.243336 [I 2022-10-27 17:08:03] ...trap/engines/engine.py.224: train: epoch 78 | batch 1510/2304 [I 2022-10-27 17:08:03] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:23 [I 2022-10-27 17:08:03] ...trap/engines/engine.py.229: process: 0.55437 | load: 0.00245 [I 2022-10-27 17:08:03] ...trap/engines/engine.py.230: loss: 0.14129 [I 2022-10-27 17:08:03] ...optimizers/trijoint.py.151: total_norm: 13.980081 [I 2022-10-27 17:08:08] ...trap/engines/engine.py.224: train: epoch 78 | batch 1520/2304 [I 2022-10-27 17:08:08] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:17 [I 2022-10-27 17:08:08] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00246 [I 2022-10-27 17:08:08] ...trap/engines/engine.py.230: loss: 0.10786 [I 2022-10-27 17:08:08] ...optimizers/trijoint.py.151: total_norm: 10.356066 [I 2022-10-27 17:08:14] ...trap/engines/engine.py.224: train: epoch 78 | batch 1530/2304 [I 2022-10-27 17:08:14] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:11 [I 2022-10-27 17:08:14] ...trap/engines/engine.py.229: process: 0.56049 | load: 0.00283 [I 2022-10-27 17:08:14] ...trap/engines/engine.py.230: loss: 0.15793 [I 2022-10-27 17:08:14] ...optimizers/trijoint.py.151: total_norm: 19.976065 [I 2022-10-27 17:08:20] ...trap/engines/engine.py.224: train: epoch 78 | batch 1540/2304 [I 2022-10-27 17:08:20] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:08 [I 2022-10-27 17:08:20] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00243 [I 2022-10-27 17:08:20] ...trap/engines/engine.py.230: loss: 0.18318 [I 2022-10-27 17:08:20] ...optimizers/trijoint.py.151: total_norm: 23.120127 [I 2022-10-27 17:08:25] ...trap/engines/engine.py.224: train: epoch 78 | batch 1550/2304 [I 2022-10-27 17:08:25] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:05 [I 2022-10-27 17:08:25] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00243 [I 2022-10-27 17:08:25] ...trap/engines/engine.py.230: loss: 0.12906 [I 2022-10-27 17:08:25] ...optimizers/trijoint.py.151: total_norm: 31.405830 [I 2022-10-27 17:08:31] ...trap/engines/engine.py.224: train: epoch 78 | batch 1560/2304 [I 2022-10-27 17:08:31] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:56 [I 2022-10-27 17:08:31] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00245 [I 2022-10-27 17:08:31] ...trap/engines/engine.py.230: loss: 0.11157 [I 2022-10-27 17:08:31] ...optimizers/trijoint.py.151: total_norm: 11.124185 [I 2022-10-27 17:08:37] ...trap/engines/engine.py.224: train: epoch 78 | batch 1570/2304 [I 2022-10-27 17:08:37] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:50 [I 2022-10-27 17:08:37] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00247 [I 2022-10-27 17:08:37] ...trap/engines/engine.py.230: loss: 0.13972 [I 2022-10-27 17:08:37] ...optimizers/trijoint.py.151: total_norm: 16.532461 [I 2022-10-27 17:08:42] ...trap/engines/engine.py.224: train: epoch 78 | batch 1580/2304 [I 2022-10-27 17:08:42] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:43 [I 2022-10-27 17:08:42] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00247 [I 2022-10-27 17:08:42] ...trap/engines/engine.py.230: loss: 0.19290 [I 2022-10-27 17:08:42] ...optimizers/trijoint.py.151: total_norm: 23.520973 [I 2022-10-27 17:08:48] ...trap/engines/engine.py.224: train: epoch 78 | batch 1590/2304 [I 2022-10-27 17:08:48] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:39 [I 2022-10-27 17:08:48] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00253 [I 2022-10-27 17:08:48] ...trap/engines/engine.py.230: loss: 0.18477 [I 2022-10-27 17:08:48] ...optimizers/trijoint.py.151: total_norm: 35.949398 [I 2022-10-27 17:08:53] ...trap/engines/engine.py.224: train: epoch 78 | batch 1600/2304 [I 2022-10-27 17:08:53] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:32 [I 2022-10-27 17:08:53] ...trap/engines/engine.py.229: process: 0.56591 | load: 0.00247 [I 2022-10-27 17:08:53] ...trap/engines/engine.py.230: loss: 0.11300 [I 2022-10-27 17:08:53] ...optimizers/trijoint.py.151: total_norm: 13.072783 [I 2022-10-27 17:08:59] ...trap/engines/engine.py.224: train: epoch 78 | batch 1610/2304 [I 2022-10-27 17:08:59] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:35 [I 2022-10-27 17:08:59] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00246 [I 2022-10-27 17:08:59] ...trap/engines/engine.py.230: loss: 0.15531 [I 2022-10-27 17:08:59] ...optimizers/trijoint.py.151: total_norm: 15.943428 [I 2022-10-27 17:09:05] ...trap/engines/engine.py.224: train: epoch 78 | batch 1620/2304 [I 2022-10-27 17:09:05] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:21 [I 2022-10-27 17:09:05] ...trap/engines/engine.py.229: process: 0.56361 | load: 0.00244 [I 2022-10-27 17:09:05] ...trap/engines/engine.py.230: loss: 0.15278 [I 2022-10-27 17:09:05] ...optimizers/trijoint.py.151: total_norm: 24.906914 [I 2022-10-27 17:09:10] ...trap/engines/engine.py.224: train: epoch 78 | batch 1630/2304 [I 2022-10-27 17:09:10] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:17 [I 2022-10-27 17:09:10] ...trap/engines/engine.py.229: process: 0.56044 | load: 0.00245 [I 2022-10-27 17:09:10] ...trap/engines/engine.py.230: loss: 0.14826 [I 2022-10-27 17:09:10] ...optimizers/trijoint.py.151: total_norm: 23.160587 [I 2022-10-27 17:09:16] ...trap/engines/engine.py.224: train: epoch 78 | batch 1640/2304 [I 2022-10-27 17:09:16] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:10 [I 2022-10-27 17:09:16] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00244 [I 2022-10-27 17:09:16] ...trap/engines/engine.py.230: loss: 0.11973 [I 2022-10-27 17:09:16] ...optimizers/trijoint.py.151: total_norm: 19.108503 [I 2022-10-27 17:09:21] ...trap/engines/engine.py.224: train: epoch 78 | batch 1650/2304 [I 2022-10-27 17:09:21] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:06 [I 2022-10-27 17:09:21] ...trap/engines/engine.py.229: process: 0.56823 | load: 0.00245 [I 2022-10-27 17:09:21] ...trap/engines/engine.py.230: loss: 0.15622 [I 2022-10-27 17:09:21] ...optimizers/trijoint.py.151: total_norm: 16.235085 [I 2022-10-27 17:09:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 1660/2304 [I 2022-10-27 17:09:27] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:59 [I 2022-10-27 17:09:27] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00247 [I 2022-10-27 17:09:27] ...trap/engines/engine.py.230: loss: 0.16164 [I 2022-10-27 17:09:27] ...optimizers/trijoint.py.151: total_norm: 14.466026 [I 2022-10-27 17:09:33] ...trap/engines/engine.py.224: train: epoch 78 | batch 1670/2304 [I 2022-10-27 17:09:33] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:55 [I 2022-10-27 17:09:33] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00249 [I 2022-10-27 17:09:33] ...trap/engines/engine.py.230: loss: 0.15133 [I 2022-10-27 17:09:33] ...optimizers/trijoint.py.151: total_norm: 19.126966 [I 2022-10-27 17:09:38] ...trap/engines/engine.py.224: train: epoch 78 | batch 1680/2304 [I 2022-10-27 17:09:38] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:53 [I 2022-10-27 17:09:38] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00273 [I 2022-10-27 17:09:38] ...trap/engines/engine.py.230: loss: 0.15759 [I 2022-10-27 17:09:38] ...optimizers/trijoint.py.151: total_norm: 22.971151 [I 2022-10-27 17:09:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 1690/2304 [I 2022-10-27 17:09:44] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:42 [I 2022-10-27 17:09:44] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00249 [I 2022-10-27 17:09:44] ...trap/engines/engine.py.230: loss: 0.13285 [I 2022-10-27 17:09:44] ...optimizers/trijoint.py.151: total_norm: 22.187355 [I 2022-10-27 17:09:49] ...trap/engines/engine.py.224: train: epoch 78 | batch 1700/2304 [I 2022-10-27 17:09:49] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:36 [I 2022-10-27 17:09:49] ...trap/engines/engine.py.229: process: 0.56518 | load: 0.00280 [I 2022-10-27 17:09:49] ...trap/engines/engine.py.230: loss: 0.10143 [I 2022-10-27 17:09:49] ...optimizers/trijoint.py.151: total_norm: 15.022963 [I 2022-10-27 17:09:55] ...trap/engines/engine.py.224: train: epoch 78 | batch 1710/2304 [I 2022-10-27 17:09:55] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:32 [I 2022-10-27 17:09:55] ...trap/engines/engine.py.229: process: 0.55819 | load: 0.00291 [I 2022-10-27 17:09:55] ...trap/engines/engine.py.230: loss: 0.15905 [I 2022-10-27 17:09:55] ...optimizers/trijoint.py.151: total_norm: 14.248076 [I 2022-10-27 17:10:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 1720/2304 [I 2022-10-27 17:10:01] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:28 [I 2022-10-27 17:10:01] ...trap/engines/engine.py.229: process: 0.56913 | load: 0.00265 [I 2022-10-27 17:10:01] ...trap/engines/engine.py.230: loss: 0.16190 [I 2022-10-27 17:10:01] ...optimizers/trijoint.py.151: total_norm: 16.247192 [I 2022-10-27 17:10:06] ...trap/engines/engine.py.224: train: epoch 78 | batch 1730/2304 [I 2022-10-27 17:10:06] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:19 [I 2022-10-27 17:10:06] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00289 [I 2022-10-27 17:10:06] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-27 17:10:06] ...optimizers/trijoint.py.151: total_norm: 17.470360 [I 2022-10-27 17:10:12] ...trap/engines/engine.py.224: train: epoch 78 | batch 1740/2304 [I 2022-10-27 17:10:12] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:22 [I 2022-10-27 17:10:12] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00245 [I 2022-10-27 17:10:12] ...trap/engines/engine.py.230: loss: 0.15748 [I 2022-10-27 17:10:12] ...optimizers/trijoint.py.151: total_norm: 34.504917 [I 2022-10-27 17:10:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 1750/2304 [I 2022-10-27 17:10:18] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:12 [I 2022-10-27 17:10:18] ...trap/engines/engine.py.229: process: 0.57136 | load: 0.00243 [I 2022-10-27 17:10:18] ...trap/engines/engine.py.230: loss: 0.15420 [I 2022-10-27 17:10:18] ...optimizers/trijoint.py.151: total_norm: 21.451107 [I 2022-10-27 17:10:23] ...trap/engines/engine.py.224: train: epoch 78 | batch 1760/2304 [I 2022-10-27 17:10:23] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:04 [I 2022-10-27 17:10:23] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00244 [I 2022-10-27 17:10:23] ...trap/engines/engine.py.230: loss: 0.10811 [I 2022-10-27 17:10:23] ...optimizers/trijoint.py.151: total_norm: 13.876411 [I 2022-10-27 17:10:29] ...trap/engines/engine.py.224: train: epoch 78 | batch 1770/2304 [I 2022-10-27 17:10:29] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:04:58 [I 2022-10-27 17:10:29] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00245 [I 2022-10-27 17:10:29] ...trap/engines/engine.py.230: loss: 0.16988 [I 2022-10-27 17:10:29] ...optimizers/trijoint.py.151: total_norm: 30.314095 [I 2022-10-27 17:10:34] ...trap/engines/engine.py.224: train: epoch 78 | batch 1780/2304 [I 2022-10-27 17:10:34] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:51 [I 2022-10-27 17:10:34] ...trap/engines/engine.py.229: process: 0.55596 | load: 0.00261 [I 2022-10-27 17:10:34] ...trap/engines/engine.py.230: loss: 0.18876 [I 2022-10-27 17:10:34] ...optimizers/trijoint.py.151: total_norm: 16.075287 [I 2022-10-27 17:10:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 1790/2304 [I 2022-10-27 17:10:40] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:45 [I 2022-10-27 17:10:40] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00249 [I 2022-10-27 17:10:40] ...trap/engines/engine.py.230: loss: 0.18661 [I 2022-10-27 17:10:40] ...optimizers/trijoint.py.151: total_norm: 19.326689 [I 2022-10-27 17:10:46] ...trap/engines/engine.py.224: train: epoch 78 | batch 1800/2304 [I 2022-10-27 17:10:46] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:47 [I 2022-10-27 17:10:46] ...trap/engines/engine.py.229: process: 0.54987 | load: 0.00272 [I 2022-10-27 17:10:46] ...trap/engines/engine.py.230: loss: 0.16625 [I 2022-10-27 17:10:46] ...optimizers/trijoint.py.151: total_norm: 13.525193 [I 2022-10-27 17:10:51] ...trap/engines/engine.py.224: train: epoch 78 | batch 1810/2304 [I 2022-10-27 17:10:51] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:37 [I 2022-10-27 17:10:51] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00246 [I 2022-10-27 17:10:51] ...trap/engines/engine.py.230: loss: 0.24533 [I 2022-10-27 17:10:51] ...optimizers/trijoint.py.151: total_norm: 19.939289 [I 2022-10-27 17:10:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 1820/2304 [I 2022-10-27 17:10:57] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:31 [I 2022-10-27 17:10:57] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00252 [I 2022-10-27 17:10:57] ...trap/engines/engine.py.230: loss: 0.11425 [I 2022-10-27 17:10:57] ...optimizers/trijoint.py.151: total_norm: 13.870410 [I 2022-10-27 17:11:02] ...trap/engines/engine.py.224: train: epoch 78 | batch 1830/2304 [I 2022-10-27 17:11:02] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:23 [I 2022-10-27 17:11:02] ...trap/engines/engine.py.229: process: 0.56260 | load: 0.00247 [I 2022-10-27 17:11:02] ...trap/engines/engine.py.230: loss: 0.17147 [I 2022-10-27 17:11:02] ...optimizers/trijoint.py.151: total_norm: 13.864697 [I 2022-10-27 17:11:08] ...trap/engines/engine.py.224: train: epoch 78 | batch 1840/2304 [I 2022-10-27 17:11:08] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:16 [I 2022-10-27 17:11:08] ...trap/engines/engine.py.229: process: 0.54483 | load: 0.00243 [I 2022-10-27 17:11:08] ...trap/engines/engine.py.230: loss: 0.18862 [I 2022-10-27 17:11:08] ...optimizers/trijoint.py.151: total_norm: 26.077133 [I 2022-10-27 17:11:13] ...trap/engines/engine.py.224: train: epoch 78 | batch 1850/2304 [I 2022-10-27 17:11:13] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:10 [I 2022-10-27 17:11:13] ...trap/engines/engine.py.229: process: 0.55152 | load: 0.00243 [I 2022-10-27 17:11:13] ...trap/engines/engine.py.230: loss: 0.09279 [I 2022-10-27 17:11:13] ...optimizers/trijoint.py.151: total_norm: 13.526653 [I 2022-10-27 17:11:19] ...trap/engines/engine.py.224: train: epoch 78 | batch 1860/2304 [I 2022-10-27 17:11:19] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:05 [I 2022-10-27 17:11:19] ...trap/engines/engine.py.229: process: 0.54735 | load: 0.00248 [I 2022-10-27 17:11:19] ...trap/engines/engine.py.230: loss: 0.12083 [I 2022-10-27 17:11:19] ...optimizers/trijoint.py.151: total_norm: 9.102285 [I 2022-10-27 17:11:25] ...trap/engines/engine.py.224: train: epoch 78 | batch 1870/2304 [I 2022-10-27 17:11:25] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:05 [I 2022-10-27 17:11:25] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00247 [I 2022-10-27 17:11:25] ...trap/engines/engine.py.230: loss: 0.12834 [I 2022-10-27 17:11:25] ...optimizers/trijoint.py.151: total_norm: 9.580738 [I 2022-10-27 17:11:30] ...trap/engines/engine.py.224: train: epoch 78 | batch 1880/2304 [I 2022-10-27 17:11:30] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:56 [I 2022-10-27 17:11:30] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00262 [I 2022-10-27 17:11:30] ...trap/engines/engine.py.230: loss: 0.18015 [I 2022-10-27 17:11:30] ...optimizers/trijoint.py.151: total_norm: 26.898363 [I 2022-10-27 17:11:36] ...trap/engines/engine.py.224: train: epoch 78 | batch 1890/2304 [I 2022-10-27 17:11:36] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:03:50 [I 2022-10-27 17:11:36] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00251 [I 2022-10-27 17:11:36] ...trap/engines/engine.py.230: loss: 0.13919 [I 2022-10-27 17:11:36] ...optimizers/trijoint.py.151: total_norm: 15.831319 [I 2022-10-27 17:11:42] ...trap/engines/engine.py.224: train: epoch 78 | batch 1900/2304 [I 2022-10-27 17:11:42] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:45 [I 2022-10-27 17:11:42] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00293 [I 2022-10-27 17:11:42] ...trap/engines/engine.py.230: loss: 0.16343 [I 2022-10-27 17:11:42] ...optimizers/trijoint.py.151: total_norm: 27.567717 [I 2022-10-27 17:11:47] ...trap/engines/engine.py.224: train: epoch 78 | batch 1910/2304 [I 2022-10-27 17:11:47] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:40 [I 2022-10-27 17:11:47] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00249 [I 2022-10-27 17:11:47] ...trap/engines/engine.py.230: loss: 0.09877 [I 2022-10-27 17:11:47] ...optimizers/trijoint.py.151: total_norm: 8.627448 [I 2022-10-27 17:11:53] ...trap/engines/engine.py.224: train: epoch 78 | batch 1920/2304 [I 2022-10-27 17:11:53] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:34 [I 2022-10-27 17:11:53] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00245 [I 2022-10-27 17:11:53] ...trap/engines/engine.py.230: loss: 0.17029 [I 2022-10-27 17:11:53] ...optimizers/trijoint.py.151: total_norm: 25.841526 [I 2022-10-27 17:11:58] ...trap/engines/engine.py.224: train: epoch 78 | batch 1930/2304 [I 2022-10-27 17:11:58] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:28 [I 2022-10-27 17:11:58] ...trap/engines/engine.py.229: process: 0.56835 | load: 0.00246 [I 2022-10-27 17:11:58] ...trap/engines/engine.py.230: loss: 0.11574 [I 2022-10-27 17:11:58] ...optimizers/trijoint.py.151: total_norm: 20.200851 [I 2022-10-27 17:12:04] ...trap/engines/engine.py.224: train: epoch 78 | batch 1940/2304 [I 2022-10-27 17:12:04] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:22 [I 2022-10-27 17:12:04] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00248 [I 2022-10-27 17:12:04] ...trap/engines/engine.py.230: loss: 0.12033 [I 2022-10-27 17:12:04] ...optimizers/trijoint.py.151: total_norm: 20.786243 [I 2022-10-27 17:12:09] ...trap/engines/engine.py.224: train: epoch 78 | batch 1950/2304 [I 2022-10-27 17:12:09] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:17 [I 2022-10-27 17:12:09] ...trap/engines/engine.py.229: process: 0.54537 | load: 0.00286 [I 2022-10-27 17:12:09] ...trap/engines/engine.py.230: loss: 0.14481 [I 2022-10-27 17:12:09] ...optimizers/trijoint.py.151: total_norm: 36.779438 [I 2022-10-27 17:12:15] ...trap/engines/engine.py.224: train: epoch 78 | batch 1960/2304 [I 2022-10-27 17:12:15] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:10 [I 2022-10-27 17:12:15] ...trap/engines/engine.py.229: process: 0.54889 | load: 0.00247 [I 2022-10-27 17:12:15] ...trap/engines/engine.py.230: loss: 0.17002 [I 2022-10-27 17:12:15] ...optimizers/trijoint.py.151: total_norm: 16.351877 [I 2022-10-27 17:12:21] ...trap/engines/engine.py.224: train: epoch 78 | batch 1970/2304 [I 2022-10-27 17:12:21] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:06 [I 2022-10-27 17:12:21] ...trap/engines/engine.py.229: process: 0.56771 | load: 0.00246 [I 2022-10-27 17:12:21] ...trap/engines/engine.py.230: loss: 0.16474 [I 2022-10-27 17:12:21] ...optimizers/trijoint.py.151: total_norm: 19.309931 [I 2022-10-27 17:12:26] ...trap/engines/engine.py.224: train: epoch 78 | batch 1980/2304 [I 2022-10-27 17:12:26] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:00 [I 2022-10-27 17:12:26] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00283 [I 2022-10-27 17:12:26] ...trap/engines/engine.py.230: loss: 0.13858 [I 2022-10-27 17:12:26] ...optimizers/trijoint.py.151: total_norm: 51.211342 [I 2022-10-27 17:12:32] ...trap/engines/engine.py.224: train: epoch 78 | batch 1990/2304 [I 2022-10-27 17:12:32] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:55 [I 2022-10-27 17:12:32] ...trap/engines/engine.py.229: process: 0.55513 | load: 0.00255 [I 2022-10-27 17:12:32] ...trap/engines/engine.py.230: loss: 0.11574 [I 2022-10-27 17:12:32] ...optimizers/trijoint.py.151: total_norm: 11.915255 [I 2022-10-27 17:12:37] ...trap/engines/engine.py.224: train: epoch 78 | batch 2000/2304 [I 2022-10-27 17:12:37] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:02:49 [I 2022-10-27 17:12:37] ...trap/engines/engine.py.229: process: 0.56834 | load: 0.00247 [I 2022-10-27 17:12:37] ...trap/engines/engine.py.230: loss: 0.12039 [I 2022-10-27 17:12:37] ...optimizers/trijoint.py.151: total_norm: 10.563310 [I 2022-10-27 17:12:43] ...trap/engines/engine.py.224: train: epoch 78 | batch 2010/2304 [I 2022-10-27 17:12:43] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:02:42 [I 2022-10-27 17:12:43] ...trap/engines/engine.py.229: process: 0.56109 | load: 0.00244 [I 2022-10-27 17:12:43] ...trap/engines/engine.py.230: loss: 0.15692 [I 2022-10-27 17:12:43] ...optimizers/trijoint.py.151: total_norm: 15.202632 [I 2022-10-27 17:12:48] ...trap/engines/engine.py.224: train: epoch 78 | batch 2020/2304 [I 2022-10-27 17:12:48] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:37 [I 2022-10-27 17:12:48] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00245 [I 2022-10-27 17:12:48] ...trap/engines/engine.py.230: loss: 0.12672 [I 2022-10-27 17:12:48] ...optimizers/trijoint.py.151: total_norm: 25.153315 [I 2022-10-27 17:12:54] ...trap/engines/engine.py.224: train: epoch 78 | batch 2030/2304 [I 2022-10-27 17:12:54] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:32 [I 2022-10-27 17:12:54] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00247 [I 2022-10-27 17:12:54] ...trap/engines/engine.py.230: loss: 0.18796 [I 2022-10-27 17:12:54] ...optimizers/trijoint.py.151: total_norm: 20.542973 [I 2022-10-27 17:13:00] ...trap/engines/engine.py.224: train: epoch 78 | batch 2040/2304 [I 2022-10-27 17:13:00] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:27 [I 2022-10-27 17:13:00] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00249 [I 2022-10-27 17:13:00] ...trap/engines/engine.py.230: loss: 0.16567 [I 2022-10-27 17:13:00] ...optimizers/trijoint.py.151: total_norm: 17.120474 [I 2022-10-27 17:13:05] ...trap/engines/engine.py.224: train: epoch 78 | batch 2050/2304 [I 2022-10-27 17:13:05] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:21 [I 2022-10-27 17:13:05] ...trap/engines/engine.py.229: process: 0.56334 | load: 0.00246 [I 2022-10-27 17:13:05] ...trap/engines/engine.py.230: loss: 0.13998 [I 2022-10-27 17:13:05] ...optimizers/trijoint.py.151: total_norm: 25.900307 [I 2022-10-27 17:13:11] ...trap/engines/engine.py.224: train: epoch 78 | batch 2060/2304 [I 2022-10-27 17:13:11] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:15 [I 2022-10-27 17:13:11] ...trap/engines/engine.py.229: process: 0.55488 | load: 0.00246 [I 2022-10-27 17:13:11] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-27 17:13:11] ...optimizers/trijoint.py.151: total_norm: 27.086033 [I 2022-10-27 17:13:16] ...trap/engines/engine.py.224: train: epoch 78 | batch 2070/2304 [I 2022-10-27 17:13:16] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:09 [I 2022-10-27 17:13:16] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00248 [I 2022-10-27 17:13:16] ...trap/engines/engine.py.230: loss: 0.11504 [I 2022-10-27 17:13:16] ...optimizers/trijoint.py.151: total_norm: 12.306721 [I 2022-10-27 17:13:22] ...trap/engines/engine.py.224: train: epoch 78 | batch 2080/2304 [I 2022-10-27 17:13:22] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:04 [I 2022-10-27 17:13:22] ...trap/engines/engine.py.229: process: 0.56957 | load: 0.00252 [I 2022-10-27 17:13:22] ...trap/engines/engine.py.230: loss: 0.10523 [I 2022-10-27 17:13:22] ...optimizers/trijoint.py.151: total_norm: 13.016435 [I 2022-10-27 17:13:28] ...trap/engines/engine.py.224: train: epoch 78 | batch 2090/2304 [I 2022-10-27 17:13:28] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:00 [I 2022-10-27 17:13:28] ...trap/engines/engine.py.229: process: 0.56647 | load: 0.00248 [I 2022-10-27 17:13:28] ...trap/engines/engine.py.230: loss: 0.11553 [I 2022-10-27 17:13:28] ...optimizers/trijoint.py.151: total_norm: 18.827824 [I 2022-10-27 17:13:33] ...trap/engines/engine.py.224: train: epoch 78 | batch 2100/2304 [I 2022-10-27 17:13:33] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:54 [I 2022-10-27 17:13:33] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00252 [I 2022-10-27 17:13:33] ...trap/engines/engine.py.230: loss: 0.15582 [I 2022-10-27 17:13:33] ...optimizers/trijoint.py.151: total_norm: 31.771130 [I 2022-10-27 17:13:39] ...trap/engines/engine.py.224: train: epoch 78 | batch 2110/2304 [I 2022-10-27 17:13:39] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:01:48 [I 2022-10-27 17:13:39] ...trap/engines/engine.py.229: process: 0.56393 | load: 0.00247 [I 2022-10-27 17:13:39] ...trap/engines/engine.py.230: loss: 0.13675 [I 2022-10-27 17:13:39] ...optimizers/trijoint.py.151: total_norm: 12.870425 [I 2022-10-27 17:13:44] ...trap/engines/engine.py.224: train: epoch 78 | batch 2120/2304 [I 2022-10-27 17:13:44] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:42 [I 2022-10-27 17:13:44] ...trap/engines/engine.py.229: process: 0.56272 | load: 0.00247 [I 2022-10-27 17:13:44] ...trap/engines/engine.py.230: loss: 0.15625 [I 2022-10-27 17:13:44] ...optimizers/trijoint.py.151: total_norm: 21.950850 [I 2022-10-27 17:13:50] ...trap/engines/engine.py.224: train: epoch 78 | batch 2130/2304 [I 2022-10-27 17:13:50] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:37 [I 2022-10-27 17:13:50] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00249 [I 2022-10-27 17:13:50] ...trap/engines/engine.py.230: loss: 0.12020 [I 2022-10-27 17:13:50] ...optimizers/trijoint.py.151: total_norm: 10.562474 [I 2022-10-27 17:13:55] ...trap/engines/engine.py.224: train: epoch 78 | batch 2140/2304 [I 2022-10-27 17:13:55] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:30 [I 2022-10-27 17:13:55] ...trap/engines/engine.py.229: process: 0.55169 | load: 0.00248 [I 2022-10-27 17:13:55] ...trap/engines/engine.py.230: loss: 0.10807 [I 2022-10-27 17:13:55] ...optimizers/trijoint.py.151: total_norm: 21.022696 [I 2022-10-27 17:14:01] ...trap/engines/engine.py.224: train: epoch 78 | batch 2150/2304 [I 2022-10-27 17:14:01] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:25 [I 2022-10-27 17:14:01] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00240 [I 2022-10-27 17:14:01] ...trap/engines/engine.py.230: loss: 0.10060 [I 2022-10-27 17:14:01] ...optimizers/trijoint.py.151: total_norm: 8.250721 [I 2022-10-27 17:14:07] ...trap/engines/engine.py.224: train: epoch 78 | batch 2160/2304 [I 2022-10-27 17:14:07] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:20 [I 2022-10-27 17:14:07] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00246 [I 2022-10-27 17:14:07] ...trap/engines/engine.py.230: loss: 0.11732 [I 2022-10-27 17:14:07] ...optimizers/trijoint.py.151: total_norm: 10.131346 [I 2022-10-27 17:14:12] ...trap/engines/engine.py.224: train: epoch 78 | batch 2170/2304 [I 2022-10-27 17:14:12] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:14 [I 2022-10-27 17:14:12] ...trap/engines/engine.py.229: process: 0.55068 | load: 0.00241 [I 2022-10-27 17:14:12] ...trap/engines/engine.py.230: loss: 0.13068 [I 2022-10-27 17:14:12] ...optimizers/trijoint.py.151: total_norm: 16.895521 [I 2022-10-27 17:14:18] ...trap/engines/engine.py.224: train: epoch 78 | batch 2180/2304 [I 2022-10-27 17:14:18] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:09 [I 2022-10-27 17:14:18] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00240 [I 2022-10-27 17:14:18] ...trap/engines/engine.py.230: loss: 0.14338 [I 2022-10-27 17:14:18] ...optimizers/trijoint.py.151: total_norm: 20.998503 [I 2022-10-27 17:14:23] ...trap/engines/engine.py.224: train: epoch 78 | batch 2190/2304 [I 2022-10-27 17:14:23] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:03 [I 2022-10-27 17:14:23] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00282 [I 2022-10-27 17:14:23] ...trap/engines/engine.py.230: loss: 0.18733 [I 2022-10-27 17:14:23] ...optimizers/trijoint.py.151: total_norm: 28.935261 [I 2022-10-27 17:14:29] ...trap/engines/engine.py.224: train: epoch 78 | batch 2200/2304 [I 2022-10-27 17:14:29] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:57 [I 2022-10-27 17:14:29] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00240 [I 2022-10-27 17:14:29] ...trap/engines/engine.py.230: loss: 0.11659 [I 2022-10-27 17:14:29] ...optimizers/trijoint.py.151: total_norm: 16.085634 [I 2022-10-27 17:14:35] ...trap/engines/engine.py.224: train: epoch 78 | batch 2210/2304 [I 2022-10-27 17:14:35] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:00:52 [I 2022-10-27 17:14:35] ...trap/engines/engine.py.229: process: 0.56421 | load: 0.00242 [I 2022-10-27 17:14:35] ...trap/engines/engine.py.230: loss: 0.17478 [I 2022-10-27 17:14:35] ...optimizers/trijoint.py.151: total_norm: 16.581310 [I 2022-10-27 17:14:40] ...trap/engines/engine.py.224: train: epoch 78 | batch 2220/2304 [I 2022-10-27 17:14:40] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:00:46 [I 2022-10-27 17:14:40] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00248 [I 2022-10-27 17:14:40] ...trap/engines/engine.py.230: loss: 0.16763 [I 2022-10-27 17:14:40] ...optimizers/trijoint.py.151: total_norm: 9.568581 [I 2022-10-27 17:14:46] ...trap/engines/engine.py.224: train: epoch 78 | batch 2230/2304 [I 2022-10-27 17:14:46] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:41 [I 2022-10-27 17:14:46] ...trap/engines/engine.py.229: process: 0.55043 | load: 0.00242 [I 2022-10-27 17:14:46] ...trap/engines/engine.py.230: loss: 0.10220 [I 2022-10-27 17:14:46] ...optimizers/trijoint.py.151: total_norm: 16.813299 [I 2022-10-27 17:14:51] ...trap/engines/engine.py.224: train: epoch 78 | batch 2240/2304 [I 2022-10-27 17:14:51] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:35 [I 2022-10-27 17:14:51] ...trap/engines/engine.py.229: process: 0.55099 | load: 0.00252 [I 2022-10-27 17:14:51] ...trap/engines/engine.py.230: loss: 0.09772 [I 2022-10-27 17:14:51] ...optimizers/trijoint.py.151: total_norm: 12.858835 [I 2022-10-27 17:14:57] ...trap/engines/engine.py.224: train: epoch 78 | batch 2250/2304 [I 2022-10-27 17:14:57] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:30 [I 2022-10-27 17:14:57] ...trap/engines/engine.py.229: process: 0.55158 | load: 0.00247 [I 2022-10-27 17:14:57] ...trap/engines/engine.py.230: loss: 0.14748 [I 2022-10-27 17:14:57] ...optimizers/trijoint.py.151: total_norm: 14.738463 [I 2022-10-27 17:15:02] ...trap/engines/engine.py.224: train: epoch 78 | batch 2260/2304 [I 2022-10-27 17:15:02] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:24 [I 2022-10-27 17:15:02] ...trap/engines/engine.py.229: process: 0.54580 | load: 0.00247 [I 2022-10-27 17:15:02] ...trap/engines/engine.py.230: loss: 0.10929 [I 2022-10-27 17:15:02] ...optimizers/trijoint.py.151: total_norm: 28.862770 [I 2022-10-27 17:15:08] ...trap/engines/engine.py.224: train: epoch 78 | batch 2270/2304 [I 2022-10-27 17:15:08] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:18 [I 2022-10-27 17:15:08] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00245 [I 2022-10-27 17:15:08] ...trap/engines/engine.py.230: loss: 0.15972 [I 2022-10-27 17:15:08] ...optimizers/trijoint.py.151: total_norm: 8.921001 [I 2022-10-27 17:15:14] ...trap/engines/engine.py.224: train: epoch 78 | batch 2280/2304 [I 2022-10-27 17:15:14] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:13 [I 2022-10-27 17:15:14] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00243 [I 2022-10-27 17:15:14] ...trap/engines/engine.py.230: loss: 0.20258 [I 2022-10-27 17:15:14] ...optimizers/trijoint.py.151: total_norm: 46.905956 [I 2022-10-27 17:15:19] ...trap/engines/engine.py.224: train: epoch 78 | batch 2290/2304 [I 2022-10-27 17:15:19] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:07 [I 2022-10-27 17:15:19] ...trap/engines/engine.py.229: process: 0.56713 | load: 0.00243 [I 2022-10-27 17:15:19] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-27 17:15:19] ...optimizers/trijoint.py.151: total_norm: 18.701931 [I 2022-10-27 17:15:25] ...trap/engines/engine.py.224: train: epoch 78 | batch 2300/2304 [I 2022-10-27 17:15:25] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:02 [I 2022-10-27 17:15:25] ...trap/engines/engine.py.229: process: 0.55381 | load: 0.00016 [I 2022-10-27 17:15:25] ...trap/engines/engine.py.230: loss: 0.11426 [I 2022-10-27 17:15:25] ...optimizers/trijoint.py.151: total_norm: 9.490181 [I 2022-10-27 17:15:27] ...trap/engines/engine.py.224: train: epoch 78 | batch 2304/2304 [I 2022-10-27 17:15:27] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:00 [I 2022-10-27 17:15:27] ...trap/engines/engine.py.229: process: 0.55100 | load: 0.00014 [I 2022-10-27 17:15:27] ...trap/engines/engine.py.230: loss: 0.12195 [I 2022-10-27 17:15:27] ...optimizers/trijoint.py.151: total_norm: 10.509529 [S 2022-10-27 17:15:28] ...trap/engines/engine.py.129: train_epoch.epoch: 78 [S 2022-10-27 17:15:28] ...trap/engines/engine.py.129: train_epoch.loss: 0.13911182882341025 [I 2022-10-27 17:15:28] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 17:15:29] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 78 [I 2022-10-27 17:15:29] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 17:15:31] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 17:15:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 17:15:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 17:15:34] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 17:15:34] ...trap/engines/engine.py.321: eval: epoch 78 | batch 0/510 [I 2022-10-27 17:15:34] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:41:06 [I 2022-10-27 17:15:34] ...trap/engines/engine.py.326: process: 4.83650 | load: 4.31997 [I 2022-10-27 17:15:38] ...trap/engines/engine.py.321: eval: epoch 78 | batch 10/510 [I 2022-10-27 17:15:38] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:49 [I 2022-10-27 17:15:38] ...trap/engines/engine.py.326: process: 0.34707 | load: 0.00020 [I 2022-10-27 17:15:41] ...trap/engines/engine.py.321: eval: epoch 78 | batch 20/510 [I 2022-10-27 17:15:41] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:08 [I 2022-10-27 17:15:41] ...trap/engines/engine.py.326: process: 0.31957 | load: 0.00022 [I 2022-10-27 17:15:44] ...trap/engines/engine.py.321: eval: epoch 78 | batch 30/510 [I 2022-10-27 17:15:44] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:44 [I 2022-10-27 17:15:44] ...trap/engines/engine.py.326: process: 0.31816 | load: 0.00018 [I 2022-10-27 17:15:48] ...trap/engines/engine.py.321: eval: epoch 78 | batch 40/510 [I 2022-10-27 17:15:48] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:37 [I 2022-10-27 17:15:48] ...trap/engines/engine.py.326: process: 0.32340 | load: 0.00017 [I 2022-10-27 17:15:51] ...trap/engines/engine.py.321: eval: epoch 78 | batch 50/510 [I 2022-10-27 17:15:51] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:34 [I 2022-10-27 17:15:51] ...trap/engines/engine.py.326: process: 0.33331 | load: 0.00022 [I 2022-10-27 17:15:54] ...trap/engines/engine.py.321: eval: epoch 78 | batch 60/510 [I 2022-10-27 17:15:54] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:31 [I 2022-10-27 17:15:54] ...trap/engines/engine.py.326: process: 0.32639 | load: 0.00018 [I 2022-10-27 17:15:58] ...trap/engines/engine.py.321: eval: epoch 78 | batch 70/510 [I 2022-10-27 17:15:58] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:25 [I 2022-10-27 17:15:58] ...trap/engines/engine.py.326: process: 0.32832 | load: 0.00022 [I 2022-10-27 17:16:01] ...trap/engines/engine.py.321: eval: epoch 78 | batch 80/510 [I 2022-10-27 17:16:01] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:22 [I 2022-10-27 17:16:01] ...trap/engines/engine.py.326: process: 0.32859 | load: 0.00021 [I 2022-10-27 17:16:04] ...trap/engines/engine.py.321: eval: epoch 78 | batch 90/510 [I 2022-10-27 17:16:04] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-27 17:16:04] ...trap/engines/engine.py.326: process: 0.32648 | load: 0.00021 [I 2022-10-27 17:16:08] ...trap/engines/engine.py.321: eval: epoch 78 | batch 100/510 [I 2022-10-27 17:16:08] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-27 17:16:08] ...trap/engines/engine.py.326: process: 0.29835 | load: 0.00021 [I 2022-10-27 17:16:11] ...trap/engines/engine.py.321: eval: epoch 78 | batch 110/510 [I 2022-10-27 17:16:11] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 17:16:11] ...trap/engines/engine.py.326: process: 0.29855 | load: 0.00020 [I 2022-10-27 17:16:13] ...trap/engines/engine.py.321: eval: epoch 78 | batch 120/510 [I 2022-10-27 17:16:13] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:54 [I 2022-10-27 17:16:13] ...trap/engines/engine.py.326: process: 0.30271 | load: 0.00018 [I 2022-10-27 17:16:16] ...trap/engines/engine.py.321: eval: epoch 78 | batch 130/510 [I 2022-10-27 17:16:16] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:55 [I 2022-10-27 17:16:16] ...trap/engines/engine.py.326: process: 0.30600 | load: 0.00019 [I 2022-10-27 17:16:19] ...trap/engines/engine.py.321: eval: epoch 78 | batch 140/510 [I 2022-10-27 17:16:19] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:49 [I 2022-10-27 17:16:19] ...trap/engines/engine.py.326: process: 0.29090 | load: 0.00023 [I 2022-10-27 17:16:22] ...trap/engines/engine.py.321: eval: epoch 78 | batch 150/510 [I 2022-10-27 17:16:22] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 17:16:22] ...trap/engines/engine.py.326: process: 0.28678 | load: 0.00031 [I 2022-10-27 17:16:25] ...trap/engines/engine.py.321: eval: epoch 78 | batch 160/510 [I 2022-10-27 17:16:25] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:48 [I 2022-10-27 17:16:25] ...trap/engines/engine.py.326: process: 0.36143 | load: 0.00024 [I 2022-10-27 17:16:28] ...trap/engines/engine.py.321: eval: epoch 78 | batch 170/510 [I 2022-10-27 17:16:28] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:40 [I 2022-10-27 17:16:28] ...trap/engines/engine.py.326: process: 0.29045 | load: 0.00025 [I 2022-10-27 17:16:31] ...trap/engines/engine.py.321: eval: epoch 78 | batch 180/510 [I 2022-10-27 17:16:31] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:36 [I 2022-10-27 17:16:31] ...trap/engines/engine.py.326: process: 0.30218 | load: 0.00024 [I 2022-10-27 17:16:34] ...trap/engines/engine.py.321: eval: epoch 78 | batch 190/510 [I 2022-10-27 17:16:34] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 17:16:34] ...trap/engines/engine.py.326: process: 0.28678 | load: 0.00024 [I 2022-10-27 17:16:37] ...trap/engines/engine.py.321: eval: epoch 78 | batch 200/510 [I 2022-10-27 17:16:37] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:30 [I 2022-10-27 17:16:37] ...trap/engines/engine.py.326: process: 0.30047 | load: 0.00022 [I 2022-10-27 17:16:40] ...trap/engines/engine.py.321: eval: epoch 78 | batch 210/510 [I 2022-10-27 17:16:40] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 17:16:40] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00022 [I 2022-10-27 17:16:43] ...trap/engines/engine.py.321: eval: epoch 78 | batch 220/510 [I 2022-10-27 17:16:43] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 17:16:43] ...trap/engines/engine.py.326: process: 0.28679 | load: 0.00019 [I 2022-10-27 17:16:46] ...trap/engines/engine.py.321: eval: epoch 78 | batch 230/510 [I 2022-10-27 17:16:46] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:21 [I 2022-10-27 17:16:46] ...trap/engines/engine.py.326: process: 0.29142 | load: 0.00024 [I 2022-10-27 17:16:49] ...trap/engines/engine.py.321: eval: epoch 78 | batch 240/510 [I 2022-10-27 17:16:49] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-27 17:16:49] ...trap/engines/engine.py.326: process: 0.29340 | load: 0.00029 [I 2022-10-27 17:16:51] ...trap/engines/engine.py.321: eval: epoch 78 | batch 250/510 [I 2022-10-27 17:16:51] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-27 17:16:51] ...trap/engines/engine.py.326: process: 0.28828 | load: 0.00027 [I 2022-10-27 17:16:54] ...trap/engines/engine.py.321: eval: epoch 78 | batch 260/510 [I 2022-10-27 17:16:54] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-27 17:16:54] ...trap/engines/engine.py.326: process: 0.29035 | load: 0.00028 [I 2022-10-27 17:16:57] ...trap/engines/engine.py.321: eval: epoch 78 | batch 270/510 [I 2022-10-27 17:16:57] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 17:16:57] ...trap/engines/engine.py.326: process: 0.29234 | load: 0.00029 [I 2022-10-27 17:17:00] ...trap/engines/engine.py.321: eval: epoch 78 | batch 280/510 [I 2022-10-27 17:17:00] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 17:17:00] ...trap/engines/engine.py.326: process: 0.28909 | load: 0.00026 [I 2022-10-27 17:17:03] ...trap/engines/engine.py.321: eval: epoch 78 | batch 290/510 [I 2022-10-27 17:17:03] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 17:17:03] ...trap/engines/engine.py.326: process: 0.28384 | load: 0.00018 [I 2022-10-27 17:17:06] ...trap/engines/engine.py.321: eval: epoch 78 | batch 300/510 [I 2022-10-27 17:17:06] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:03 [I 2022-10-27 17:17:06] ...trap/engines/engine.py.326: process: 0.29024 | load: 0.00019 [I 2022-10-27 17:17:09] ...trap/engines/engine.py.321: eval: epoch 78 | batch 310/510 [I 2022-10-27 17:17:09] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:58 [I 2022-10-27 17:17:09] ...trap/engines/engine.py.326: process: 0.28598 | load: 0.00019 [I 2022-10-27 17:17:12] ...trap/engines/engine.py.321: eval: epoch 78 | batch 320/510 [I 2022-10-27 17:17:12] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 17:17:12] ...trap/engines/engine.py.326: process: 0.29147 | load: 0.00026 [I 2022-10-27 17:17:15] ...trap/engines/engine.py.321: eval: epoch 78 | batch 330/510 [I 2022-10-27 17:17:15] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-27 17:17:15] ...trap/engines/engine.py.326: process: 0.28409 | load: 0.00021 [I 2022-10-27 17:17:18] ...trap/engines/engine.py.321: eval: epoch 78 | batch 340/510 [I 2022-10-27 17:17:18] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-27 17:17:18] ...trap/engines/engine.py.326: process: 0.29366 | load: 0.00028 [I 2022-10-27 17:17:21] ...trap/engines/engine.py.321: eval: epoch 78 | batch 350/510 [I 2022-10-27 17:17:21] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 17:17:21] ...trap/engines/engine.py.326: process: 0.29226 | load: 0.00020 [I 2022-10-27 17:17:23] ...trap/engines/engine.py.321: eval: epoch 78 | batch 360/510 [I 2022-10-27 17:17:23] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-27 17:17:23] ...trap/engines/engine.py.326: process: 0.29006 | load: 0.00021 [I 2022-10-27 17:17:26] ...trap/engines/engine.py.321: eval: epoch 78 | batch 370/510 [I 2022-10-27 17:17:26] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 17:17:26] ...trap/engines/engine.py.326: process: 0.28701 | load: 0.00026 [I 2022-10-27 17:17:29] ...trap/engines/engine.py.321: eval: epoch 78 | batch 380/510 [I 2022-10-27 17:17:29] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 17:17:29] ...trap/engines/engine.py.326: process: 0.28855 | load: 0.00020 [I 2022-10-27 17:17:32] ...trap/engines/engine.py.321: eval: epoch 78 | batch 390/510 [I 2022-10-27 17:17:32] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 17:17:32] ...trap/engines/engine.py.326: process: 0.29428 | load: 0.00029 [I 2022-10-27 17:17:35] ...trap/engines/engine.py.321: eval: epoch 78 | batch 400/510 [I 2022-10-27 17:17:35] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 17:17:35] ...trap/engines/engine.py.326: process: 0.29094 | load: 0.00026 [I 2022-10-27 17:17:38] ...trap/engines/engine.py.321: eval: epoch 78 | batch 410/510 [I 2022-10-27 17:17:38] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 17:17:38] ...trap/engines/engine.py.326: process: 0.29073 | load: 0.00023 [I 2022-10-27 17:17:41] ...trap/engines/engine.py.321: eval: epoch 78 | batch 420/510 [I 2022-10-27 17:17:41] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-27 17:17:41] ...trap/engines/engine.py.326: process: 0.29174 | load: 0.00020 [I 2022-10-27 17:17:44] ...trap/engines/engine.py.321: eval: epoch 78 | batch 430/510 [I 2022-10-27 17:17:44] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 17:17:44] ...trap/engines/engine.py.326: process: 0.28260 | load: 0.00025 [I 2022-10-27 17:17:47] ...trap/engines/engine.py.321: eval: epoch 78 | batch 440/510 [I 2022-10-27 17:17:47] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 17:17:47] ...trap/engines/engine.py.326: process: 0.28180 | load: 0.00023 [I 2022-10-27 17:17:50] ...trap/engines/engine.py.321: eval: epoch 78 | batch 450/510 [I 2022-10-27 17:17:50] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 17:17:50] ...trap/engines/engine.py.326: process: 0.28906 | load: 0.00023 [I 2022-10-27 17:17:52] ...trap/engines/engine.py.321: eval: epoch 78 | batch 460/510 [I 2022-10-27 17:17:52] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-27 17:17:52] ...trap/engines/engine.py.326: process: 0.28272 | load: 0.00024 [I 2022-10-27 17:17:55] ...trap/engines/engine.py.321: eval: epoch 78 | batch 470/510 [I 2022-10-27 17:17:55] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 17:17:55] ...trap/engines/engine.py.326: process: 0.29073 | load: 0.00030 [I 2022-10-27 17:17:58] ...trap/engines/engine.py.321: eval: epoch 78 | batch 480/510 [I 2022-10-27 17:17:58] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 17:17:58] ...trap/engines/engine.py.326: process: 0.29048 | load: 0.00023 [I 2022-10-27 17:18:01] ...trap/engines/engine.py.321: eval: epoch 78 | batch 490/510 [I 2022-10-27 17:18:01] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 17:18:01] ...trap/engines/engine.py.326: process: 0.28911 | load: 0.00027 [I 2022-10-27 17:18:04] ...trap/engines/engine.py.321: eval: epoch 78 | batch 500/510 [I 2022-10-27 17:18:04] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 17:18:04] ...trap/engines/engine.py.326: process: 0.28679 | load: 0.00018 [I 2022-10-27 17:18:07] ...trap/engines/engine.py.321: eval: epoch 78 | batch 510/510 [I 2022-10-27 17:18:07] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 17:18:07] ...trap/engines/engine.py.326: process: 0.27937 | load: 0.00010 [S 2022-10-27 17:18:08] ...trap/engines/engine.py.132: eval_epoch.epoch: 78 [I 2022-10-27 17:18:08] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.724400 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906800 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935000 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.713200 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.900100 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932400 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014389 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007359 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.003464 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012221 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006640 [S 2022-10-27 17:18:12] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005200 [I 2022-10-27 17:18:13] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 17:18:13] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 17:18:14] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 17:18:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 17:18:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 17:18:17] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 17:18:18] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 17:18:24] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 17:18:24] ...trap/engines/engine.py.163: Training model on trainset for epoch 79 [I 2022-10-27 17:18:24] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 17:18:27] ...trap/engines/engine.py.224: train: epoch 79 | batch 0/2304 [I 2022-10-27 17:18:27] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:42 [I 2022-10-27 17:18:27] ...trap/engines/engine.py.229: process: 2.80493 | load: 2.14360 [I 2022-10-27 17:18:27] ...trap/engines/engine.py.230: loss: 0.14955 [I 2022-10-27 17:18:27] ...optimizers/trijoint.py.151: total_norm: 14.062525 [I 2022-10-27 17:18:33] ...trap/engines/engine.py.224: train: epoch 79 | batch 10/2304 [I 2022-10-27 17:18:33] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:42 [I 2022-10-27 17:18:33] ...trap/engines/engine.py.229: process: 0.57871 | load: 0.00395 [I 2022-10-27 17:18:33] ...trap/engines/engine.py.230: loss: 0.12997 [I 2022-10-27 17:18:33] ...optimizers/trijoint.py.151: total_norm: 30.328856 [I 2022-10-27 17:18:38] ...trap/engines/engine.py.224: train: epoch 79 | batch 20/2304 [I 2022-10-27 17:18:38] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:16 [I 2022-10-27 17:18:38] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00320 [I 2022-10-27 17:18:38] ...trap/engines/engine.py.230: loss: 0.10865 [I 2022-10-27 17:18:38] ...optimizers/trijoint.py.151: total_norm: 10.749468 [I 2022-10-27 17:18:44] ...trap/engines/engine.py.224: train: epoch 79 | batch 30/2304 [I 2022-10-27 17:18:44] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:12 [I 2022-10-27 17:18:44] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00339 [I 2022-10-27 17:18:44] ...trap/engines/engine.py.230: loss: 0.18807 [I 2022-10-27 17:18:44] ...optimizers/trijoint.py.151: total_norm: 19.734974 [I 2022-10-27 17:18:50] ...trap/engines/engine.py.224: train: epoch 79 | batch 40/2304 [I 2022-10-27 17:18:50] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:00 [I 2022-10-27 17:18:50] ...trap/engines/engine.py.229: process: 0.55512 | load: 0.00285 [I 2022-10-27 17:18:50] ...trap/engines/engine.py.230: loss: 0.11939 [I 2022-10-27 17:18:50] ...optimizers/trijoint.py.151: total_norm: 25.775169 [I 2022-10-27 17:18:55] ...trap/engines/engine.py.224: train: epoch 79 | batch 50/2304 [I 2022-10-27 17:18:55] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:01 [I 2022-10-27 17:18:55] ...trap/engines/engine.py.229: process: 0.55526 | load: 0.00289 [I 2022-10-27 17:18:55] ...trap/engines/engine.py.230: loss: 0.18501 [I 2022-10-27 17:18:55] ...optimizers/trijoint.py.151: total_norm: 31.913675 [I 2022-10-27 17:19:01] ...trap/engines/engine.py.224: train: epoch 79 | batch 60/2304 [I 2022-10-27 17:19:01] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:43 [I 2022-10-27 17:19:01] ...trap/engines/engine.py.229: process: 0.54710 | load: 0.00278 [I 2022-10-27 17:19:01] ...trap/engines/engine.py.230: loss: 0.11537 [I 2022-10-27 17:19:01] ...optimizers/trijoint.py.151: total_norm: 18.519693 [I 2022-10-27 17:19:06] ...trap/engines/engine.py.224: train: epoch 79 | batch 70/2304 [I 2022-10-27 17:19:06] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:51 [I 2022-10-27 17:19:06] ...trap/engines/engine.py.229: process: 0.56220 | load: 0.00294 [I 2022-10-27 17:19:06] ...trap/engines/engine.py.230: loss: 0.18442 [I 2022-10-27 17:19:06] ...optimizers/trijoint.py.151: total_norm: 21.612947 [I 2022-10-27 17:19:12] ...trap/engines/engine.py.224: train: epoch 79 | batch 80/2304 [I 2022-10-27 17:19:12] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:35 [I 2022-10-27 17:19:12] ...trap/engines/engine.py.229: process: 0.55183 | load: 0.00302 [I 2022-10-27 17:19:12] ...trap/engines/engine.py.230: loss: 0.12668 [I 2022-10-27 17:19:12] ...optimizers/trijoint.py.151: total_norm: 20.012568 [I 2022-10-27 17:19:18] ...trap/engines/engine.py.224: train: epoch 79 | batch 90/2304 [I 2022-10-27 17:19:18] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:46 [I 2022-10-27 17:19:18] ...trap/engines/engine.py.229: process: 0.58636 | load: 0.00265 [I 2022-10-27 17:19:18] ...trap/engines/engine.py.230: loss: 0.15472 [I 2022-10-27 17:19:18] ...optimizers/trijoint.py.151: total_norm: 16.128353 [I 2022-10-27 17:19:23] ...trap/engines/engine.py.224: train: epoch 79 | batch 100/2304 [I 2022-10-27 17:19:23] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:35 [I 2022-10-27 17:19:23] ...trap/engines/engine.py.229: process: 0.56165 | load: 0.00268 [I 2022-10-27 17:19:23] ...trap/engines/engine.py.230: loss: 0.16334 [I 2022-10-27 17:19:23] ...optimizers/trijoint.py.151: total_norm: 27.094788 [I 2022-10-27 17:19:29] ...trap/engines/engine.py.224: train: epoch 79 | batch 110/2304 [I 2022-10-27 17:19:29] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:18 [I 2022-10-27 17:19:29] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00273 [I 2022-10-27 17:19:29] ...trap/engines/engine.py.230: loss: 0.09383 [I 2022-10-27 17:19:29] ...optimizers/trijoint.py.151: total_norm: 10.143951 [I 2022-10-27 17:19:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 120/2304 [I 2022-10-27 17:19:34] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:19 [I 2022-10-27 17:19:34] ...trap/engines/engine.py.229: process: 0.54620 | load: 0.00247 [I 2022-10-27 17:19:34] ...trap/engines/engine.py.230: loss: 0.19005 [I 2022-10-27 17:19:34] ...optimizers/trijoint.py.151: total_norm: 40.338036 [I 2022-10-27 17:19:40] ...trap/engines/engine.py.224: train: epoch 79 | batch 130/2304 [I 2022-10-27 17:19:40] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:01 [I 2022-10-27 17:19:40] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00252 [I 2022-10-27 17:19:40] ...trap/engines/engine.py.230: loss: 0.12892 [I 2022-10-27 17:19:40] ...optimizers/trijoint.py.151: total_norm: 37.078793 [I 2022-10-27 17:19:45] ...trap/engines/engine.py.224: train: epoch 79 | batch 140/2304 [I 2022-10-27 17:19:45] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:05 [I 2022-10-27 17:19:45] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00269 [I 2022-10-27 17:19:45] ...trap/engines/engine.py.230: loss: 0.13793 [I 2022-10-27 17:19:45] ...optimizers/trijoint.py.151: total_norm: 12.084265 [I 2022-10-27 17:19:51] ...trap/engines/engine.py.224: train: epoch 79 | batch 150/2304 [I 2022-10-27 17:19:51] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:03 [I 2022-10-27 17:19:51] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00250 [I 2022-10-27 17:19:51] ...trap/engines/engine.py.230: loss: 0.15213 [I 2022-10-27 17:19:51] ...optimizers/trijoint.py.151: total_norm: 8.444418 [I 2022-10-27 17:19:57] ...trap/engines/engine.py.224: train: epoch 79 | batch 160/2304 [I 2022-10-27 17:19:57] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:08 [I 2022-10-27 17:19:57] ...trap/engines/engine.py.229: process: 0.56387 | load: 0.00252 [I 2022-10-27 17:19:57] ...trap/engines/engine.py.230: loss: 0.11592 [I 2022-10-27 17:19:57] ...optimizers/trijoint.py.151: total_norm: 18.914221 [I 2022-10-27 17:20:02] ...trap/engines/engine.py.224: train: epoch 79 | batch 170/2304 [I 2022-10-27 17:20:02] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:50 [I 2022-10-27 17:20:02] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00250 [I 2022-10-27 17:20:02] ...trap/engines/engine.py.230: loss: 0.10349 [I 2022-10-27 17:20:02] ...optimizers/trijoint.py.151: total_norm: 8.746203 [I 2022-10-27 17:20:08] ...trap/engines/engine.py.224: train: epoch 79 | batch 180/2304 [I 2022-10-27 17:20:08] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:51 [I 2022-10-27 17:20:08] ...trap/engines/engine.py.229: process: 0.55258 | load: 0.00247 [I 2022-10-27 17:20:08] ...trap/engines/engine.py.230: loss: 0.12430 [I 2022-10-27 17:20:08] ...optimizers/trijoint.py.151: total_norm: 17.364243 [I 2022-10-27 17:20:14] ...trap/engines/engine.py.224: train: epoch 79 | batch 190/2304 [I 2022-10-27 17:20:14] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:41 [I 2022-10-27 17:20:14] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00252 [I 2022-10-27 17:20:14] ...trap/engines/engine.py.230: loss: 0.10593 [I 2022-10-27 17:20:14] ...optimizers/trijoint.py.151: total_norm: 28.797640 [I 2022-10-27 17:20:19] ...trap/engines/engine.py.224: train: epoch 79 | batch 200/2304 [I 2022-10-27 17:20:19] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:25 [I 2022-10-27 17:20:19] ...trap/engines/engine.py.229: process: 0.54608 | load: 0.00252 [I 2022-10-27 17:20:19] ...trap/engines/engine.py.230: loss: 0.16341 [I 2022-10-27 17:20:19] ...optimizers/trijoint.py.151: total_norm: 16.664787 [I 2022-10-27 17:20:25] ...trap/engines/engine.py.224: train: epoch 79 | batch 210/2304 [I 2022-10-27 17:20:25] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:25 [I 2022-10-27 17:20:25] ...trap/engines/engine.py.229: process: 0.55824 | load: 0.00269 [I 2022-10-27 17:20:25] ...trap/engines/engine.py.230: loss: 0.10461 [I 2022-10-27 17:20:25] ...optimizers/trijoint.py.151: total_norm: 13.562145 [I 2022-10-27 17:20:30] ...trap/engines/engine.py.224: train: epoch 79 | batch 220/2304 [I 2022-10-27 17:20:30] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:17 [I 2022-10-27 17:20:30] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00244 [I 2022-10-27 17:20:30] ...trap/engines/engine.py.230: loss: 0.11326 [I 2022-10-27 17:20:30] ...optimizers/trijoint.py.151: total_norm: 14.308036 [I 2022-10-27 17:20:36] ...trap/engines/engine.py.224: train: epoch 79 | batch 230/2304 [I 2022-10-27 17:20:36] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:23 [I 2022-10-27 17:20:36] ...trap/engines/engine.py.229: process: 0.54592 | load: 0.00260 [I 2022-10-27 17:20:36] ...trap/engines/engine.py.230: loss: 0.12073 [I 2022-10-27 17:20:36] ...optimizers/trijoint.py.151: total_norm: 16.857508 [I 2022-10-27 17:20:42] ...trap/engines/engine.py.224: train: epoch 79 | batch 240/2304 [I 2022-10-27 17:20:42] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:48 [I 2022-10-27 17:20:42] ...trap/engines/engine.py.229: process: 0.54445 | load: 0.00247 [I 2022-10-27 17:20:42] ...trap/engines/engine.py.230: loss: 0.13465 [I 2022-10-27 17:20:42] ...optimizers/trijoint.py.151: total_norm: 11.719284 [I 2022-10-27 17:20:47] ...trap/engines/engine.py.224: train: epoch 79 | batch 250/2304 [I 2022-10-27 17:20:47] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:30 [I 2022-10-27 17:20:47] ...trap/engines/engine.py.229: process: 0.58741 | load: 0.00256 [I 2022-10-27 17:20:47] ...trap/engines/engine.py.230: loss: 0.10909 [I 2022-10-27 17:20:47] ...optimizers/trijoint.py.151: total_norm: 10.724409 [I 2022-10-27 17:20:53] ...trap/engines/engine.py.224: train: epoch 79 | batch 260/2304 [I 2022-10-27 17:20:53] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:58 [I 2022-10-27 17:20:53] ...trap/engines/engine.py.229: process: 0.54871 | load: 0.00250 [I 2022-10-27 17:20:53] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-27 17:20:53] ...optimizers/trijoint.py.151: total_norm: 11.851353 [I 2022-10-27 17:20:59] ...trap/engines/engine.py.224: train: epoch 79 | batch 270/2304 [I 2022-10-27 17:20:59] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:59 [I 2022-10-27 17:20:59] ...trap/engines/engine.py.229: process: 0.55325 | load: 0.00255 [I 2022-10-27 17:20:59] ...trap/engines/engine.py.230: loss: 0.10527 [I 2022-10-27 17:20:59] ...optimizers/trijoint.py.151: total_norm: 16.365105 [I 2022-10-27 17:21:04] ...trap/engines/engine.py.224: train: epoch 79 | batch 280/2304 [I 2022-10-27 17:21:04] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:53 [I 2022-10-27 17:21:04] ...trap/engines/engine.py.229: process: 0.55941 | load: 0.00249 [I 2022-10-27 17:21:04] ...trap/engines/engine.py.230: loss: 0.13811 [I 2022-10-27 17:21:04] ...optimizers/trijoint.py.151: total_norm: 12.806285 [I 2022-10-27 17:21:10] ...trap/engines/engine.py.224: train: epoch 79 | batch 290/2304 [I 2022-10-27 17:21:10] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:36 [I 2022-10-27 17:21:10] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00244 [I 2022-10-27 17:21:10] ...trap/engines/engine.py.230: loss: 0.12724 [I 2022-10-27 17:21:10] ...optimizers/trijoint.py.151: total_norm: 9.584120 [I 2022-10-27 17:21:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 300/2304 [I 2022-10-27 17:21:15] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:42 [I 2022-10-27 17:21:15] ...trap/engines/engine.py.229: process: 0.55756 | load: 0.00244 [I 2022-10-27 17:21:15] ...trap/engines/engine.py.230: loss: 0.17006 [I 2022-10-27 17:21:15] ...optimizers/trijoint.py.151: total_norm: 17.328276 [I 2022-10-27 17:21:21] ...trap/engines/engine.py.224: train: epoch 79 | batch 310/2304 [I 2022-10-27 17:21:21] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:46 [I 2022-10-27 17:21:21] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00244 [I 2022-10-27 17:21:21] ...trap/engines/engine.py.230: loss: 0.11873 [I 2022-10-27 17:21:21] ...optimizers/trijoint.py.151: total_norm: 15.720468 [I 2022-10-27 17:21:27] ...trap/engines/engine.py.224: train: epoch 79 | batch 320/2304 [I 2022-10-27 17:21:27] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:30 [I 2022-10-27 17:21:27] ...trap/engines/engine.py.229: process: 0.55638 | load: 0.00273 [I 2022-10-27 17:21:27] ...trap/engines/engine.py.230: loss: 0.12202 [I 2022-10-27 17:21:27] ...optimizers/trijoint.py.151: total_norm: 18.161785 [I 2022-10-27 17:21:32] ...trap/engines/engine.py.224: train: epoch 79 | batch 330/2304 [I 2022-10-27 17:21:32] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:23 [I 2022-10-27 17:21:32] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00260 [I 2022-10-27 17:21:32] ...trap/engines/engine.py.230: loss: 0.11420 [I 2022-10-27 17:21:32] ...optimizers/trijoint.py.151: total_norm: 12.917533 [I 2022-10-27 17:21:38] ...trap/engines/engine.py.224: train: epoch 79 | batch 340/2304 [I 2022-10-27 17:21:38] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:07 [I 2022-10-27 17:21:38] ...trap/engines/engine.py.229: process: 0.55713 | load: 0.00247 [I 2022-10-27 17:21:38] ...trap/engines/engine.py.230: loss: 0.13688 [I 2022-10-27 17:21:38] ...optimizers/trijoint.py.151: total_norm: 25.011850 [I 2022-10-27 17:21:43] ...trap/engines/engine.py.224: train: epoch 79 | batch 350/2304 [I 2022-10-27 17:21:43] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:13 [I 2022-10-27 17:21:43] ...trap/engines/engine.py.229: process: 0.56641 | load: 0.00287 [I 2022-10-27 17:21:43] ...trap/engines/engine.py.230: loss: 0.13545 [I 2022-10-27 17:21:43] ...optimizers/trijoint.py.151: total_norm: 10.850134 [I 2022-10-27 17:21:49] ...trap/engines/engine.py.224: train: epoch 79 | batch 360/2304 [I 2022-10-27 17:21:49] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:04 [I 2022-10-27 17:21:49] ...trap/engines/engine.py.229: process: 0.56573 | load: 0.00255 [I 2022-10-27 17:21:49] ...trap/engines/engine.py.230: loss: 0.15420 [I 2022-10-27 17:21:49] ...optimizers/trijoint.py.151: total_norm: 15.385681 [I 2022-10-27 17:21:55] ...trap/engines/engine.py.224: train: epoch 79 | batch 370/2304 [I 2022-10-27 17:21:55] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:17:58 [I 2022-10-27 17:21:55] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00243 [I 2022-10-27 17:21:55] ...trap/engines/engine.py.230: loss: 0.08744 [I 2022-10-27 17:21:55] ...optimizers/trijoint.py.151: total_norm: 10.961470 [I 2022-10-27 17:22:00] ...trap/engines/engine.py.224: train: epoch 79 | batch 380/2304 [I 2022-10-27 17:22:00] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:57 [I 2022-10-27 17:22:00] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00244 [I 2022-10-27 17:22:00] ...trap/engines/engine.py.230: loss: 0.18830 [I 2022-10-27 17:22:00] ...optimizers/trijoint.py.151: total_norm: 14.421181 [I 2022-10-27 17:22:06] ...trap/engines/engine.py.224: train: epoch 79 | batch 390/2304 [I 2022-10-27 17:22:06] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:54 [I 2022-10-27 17:22:06] ...trap/engines/engine.py.229: process: 0.57619 | load: 0.00268 [I 2022-10-27 17:22:06] ...trap/engines/engine.py.230: loss: 0.09103 [I 2022-10-27 17:22:06] ...optimizers/trijoint.py.151: total_norm: 10.001096 [I 2022-10-27 17:22:11] ...trap/engines/engine.py.224: train: epoch 79 | batch 400/2304 [I 2022-10-27 17:22:11] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:36 [I 2022-10-27 17:22:11] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00257 [I 2022-10-27 17:22:11] ...trap/engines/engine.py.230: loss: 0.12719 [I 2022-10-27 17:22:11] ...optimizers/trijoint.py.151: total_norm: 14.417566 [I 2022-10-27 17:22:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 410/2304 [I 2022-10-27 17:22:17] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:28 [I 2022-10-27 17:22:17] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00243 [I 2022-10-27 17:22:17] ...trap/engines/engine.py.230: loss: 0.21381 [I 2022-10-27 17:22:17] ...optimizers/trijoint.py.151: total_norm: 58.836376 [I 2022-10-27 17:22:22] ...trap/engines/engine.py.224: train: epoch 79 | batch 420/2304 [I 2022-10-27 17:22:22] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:26 [I 2022-10-27 17:22:22] ...trap/engines/engine.py.229: process: 0.54615 | load: 0.00250 [I 2022-10-27 17:22:22] ...trap/engines/engine.py.230: loss: 0.17070 [I 2022-10-27 17:22:22] ...optimizers/trijoint.py.151: total_norm: 27.387550 [I 2022-10-27 17:22:28] ...trap/engines/engine.py.224: train: epoch 79 | batch 430/2304 [I 2022-10-27 17:22:28] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:23 [I 2022-10-27 17:22:28] ...trap/engines/engine.py.229: process: 0.54714 | load: 0.00247 [I 2022-10-27 17:22:28] ...trap/engines/engine.py.230: loss: 0.10576 [I 2022-10-27 17:22:28] ...optimizers/trijoint.py.151: total_norm: 14.465025 [I 2022-10-27 17:22:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 440/2304 [I 2022-10-27 17:22:34] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:18 [I 2022-10-27 17:22:34] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00242 [I 2022-10-27 17:22:34] ...trap/engines/engine.py.230: loss: 0.10852 [I 2022-10-27 17:22:34] ...optimizers/trijoint.py.151: total_norm: 16.580484 [I 2022-10-27 17:22:39] ...trap/engines/engine.py.224: train: epoch 79 | batch 450/2304 [I 2022-10-27 17:22:39] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:16 [I 2022-10-27 17:22:39] ...trap/engines/engine.py.229: process: 0.55283 | load: 0.00249 [I 2022-10-27 17:22:39] ...trap/engines/engine.py.230: loss: 0.17533 [I 2022-10-27 17:22:39] ...optimizers/trijoint.py.151: total_norm: 16.663486 [I 2022-10-27 17:22:45] ...trap/engines/engine.py.224: train: epoch 79 | batch 460/2304 [I 2022-10-27 17:22:45] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:08 [I 2022-10-27 17:22:45] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00254 [I 2022-10-27 17:22:45] ...trap/engines/engine.py.230: loss: 0.10701 [I 2022-10-27 17:22:45] ...optimizers/trijoint.py.151: total_norm: 21.584972 [I 2022-10-27 17:22:50] ...trap/engines/engine.py.224: train: epoch 79 | batch 470/2304 [I 2022-10-27 17:22:50] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:04 [I 2022-10-27 17:22:50] ...trap/engines/engine.py.229: process: 0.57051 | load: 0.00243 [I 2022-10-27 17:22:50] ...trap/engines/engine.py.230: loss: 0.15696 [I 2022-10-27 17:22:50] ...optimizers/trijoint.py.151: total_norm: 10.751242 [I 2022-10-27 17:22:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 480/2304 [I 2022-10-27 17:22:56] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:57 [I 2022-10-27 17:22:56] ...trap/engines/engine.py.229: process: 0.56260 | load: 0.00244 [I 2022-10-27 17:22:56] ...trap/engines/engine.py.230: loss: 0.10683 [I 2022-10-27 17:22:56] ...optimizers/trijoint.py.151: total_norm: 13.220077 [I 2022-10-27 17:23:02] ...trap/engines/engine.py.224: train: epoch 79 | batch 490/2304 [I 2022-10-27 17:23:02] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:46 [I 2022-10-27 17:23:02] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00243 [I 2022-10-27 17:23:02] ...trap/engines/engine.py.230: loss: 0.20789 [I 2022-10-27 17:23:02] ...optimizers/trijoint.py.151: total_norm: 21.346975 [I 2022-10-27 17:23:07] ...trap/engines/engine.py.224: train: epoch 79 | batch 500/2304 [I 2022-10-27 17:23:07] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:38 [I 2022-10-27 17:23:07] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00245 [I 2022-10-27 17:23:07] ...trap/engines/engine.py.230: loss: 0.11502 [I 2022-10-27 17:23:07] ...optimizers/trijoint.py.151: total_norm: 10.586361 [I 2022-10-27 17:23:13] ...trap/engines/engine.py.224: train: epoch 79 | batch 510/2304 [I 2022-10-27 17:23:13] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:44 [I 2022-10-27 17:23:13] ...trap/engines/engine.py.229: process: 0.56501 | load: 0.00252 [I 2022-10-27 17:23:13] ...trap/engines/engine.py.230: loss: 0.15972 [I 2022-10-27 17:23:13] ...optimizers/trijoint.py.151: total_norm: 19.099705 [I 2022-10-27 17:23:18] ...trap/engines/engine.py.224: train: epoch 79 | batch 520/2304 [I 2022-10-27 17:23:18] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:31 [I 2022-10-27 17:23:18] ...trap/engines/engine.py.229: process: 0.55218 | load: 0.00240 [I 2022-10-27 17:23:18] ...trap/engines/engine.py.230: loss: 0.13523 [I 2022-10-27 17:23:18] ...optimizers/trijoint.py.151: total_norm: 11.952256 [I 2022-10-27 17:23:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 530/2304 [I 2022-10-27 17:23:24] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:27 [I 2022-10-27 17:23:24] ...trap/engines/engine.py.229: process: 0.56934 | load: 0.00246 [I 2022-10-27 17:23:24] ...trap/engines/engine.py.230: loss: 0.12279 [I 2022-10-27 17:23:24] ...optimizers/trijoint.py.151: total_norm: 23.371058 [I 2022-10-27 17:23:29] ...trap/engines/engine.py.224: train: epoch 79 | batch 540/2304 [I 2022-10-27 17:23:29] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:16 [I 2022-10-27 17:23:29] ...trap/engines/engine.py.229: process: 0.54015 | load: 0.00244 [I 2022-10-27 17:23:29] ...trap/engines/engine.py.230: loss: 0.12566 [I 2022-10-27 17:23:29] ...optimizers/trijoint.py.151: total_norm: 14.819243 [I 2022-10-27 17:23:35] ...trap/engines/engine.py.224: train: epoch 79 | batch 550/2304 [I 2022-10-27 17:23:35] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:09 [I 2022-10-27 17:23:35] ...trap/engines/engine.py.229: process: 0.55298 | load: 0.00251 [I 2022-10-27 17:23:35] ...trap/engines/engine.py.230: loss: 0.12700 [I 2022-10-27 17:23:35] ...optimizers/trijoint.py.151: total_norm: 39.556732 [I 2022-10-27 17:23:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 560/2304 [I 2022-10-27 17:23:41] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:09 [I 2022-10-27 17:23:41] ...trap/engines/engine.py.229: process: 0.55293 | load: 0.00256 [I 2022-10-27 17:23:41] ...trap/engines/engine.py.230: loss: 0.20076 [I 2022-10-27 17:23:41] ...optimizers/trijoint.py.151: total_norm: 30.490063 [I 2022-10-27 17:23:46] ...trap/engines/engine.py.224: train: epoch 79 | batch 570/2304 [I 2022-10-27 17:23:46] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:14 [I 2022-10-27 17:23:46] ...trap/engines/engine.py.229: process: 0.56730 | load: 0.00246 [I 2022-10-27 17:23:46] ...trap/engines/engine.py.230: loss: 0.13689 [I 2022-10-27 17:23:46] ...optimizers/trijoint.py.151: total_norm: 17.198742 [I 2022-10-27 17:23:52] ...trap/engines/engine.py.224: train: epoch 79 | batch 580/2304 [I 2022-10-27 17:23:52] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:06 [I 2022-10-27 17:23:52] ...trap/engines/engine.py.229: process: 0.56165 | load: 0.00271 [I 2022-10-27 17:23:52] ...trap/engines/engine.py.230: loss: 0.10557 [I 2022-10-27 17:23:52] ...optimizers/trijoint.py.151: total_norm: 24.229965 [I 2022-10-27 17:23:57] ...trap/engines/engine.py.224: train: epoch 79 | batch 590/2304 [I 2022-10-27 17:23:57] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:56 [I 2022-10-27 17:23:57] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00242 [I 2022-10-27 17:23:57] ...trap/engines/engine.py.230: loss: 0.11639 [I 2022-10-27 17:23:57] ...optimizers/trijoint.py.151: total_norm: 13.862487 [I 2022-10-27 17:24:03] ...trap/engines/engine.py.224: train: epoch 79 | batch 600/2304 [I 2022-10-27 17:24:03] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:46 [I 2022-10-27 17:24:03] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00243 [I 2022-10-27 17:24:03] ...trap/engines/engine.py.230: loss: 0.11195 [I 2022-10-27 17:24:03] ...optimizers/trijoint.py.151: total_norm: 11.409187 [I 2022-10-27 17:24:08] ...trap/engines/engine.py.224: train: epoch 79 | batch 610/2304 [I 2022-10-27 17:24:08] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:41 [I 2022-10-27 17:24:08] ...trap/engines/engine.py.229: process: 0.54751 | load: 0.00242 [I 2022-10-27 17:24:08] ...trap/engines/engine.py.230: loss: 0.13181 [I 2022-10-27 17:24:08] ...optimizers/trijoint.py.151: total_norm: 30.279530 [I 2022-10-27 17:24:14] ...trap/engines/engine.py.224: train: epoch 79 | batch 620/2304 [I 2022-10-27 17:24:14] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:37 [I 2022-10-27 17:24:14] ...trap/engines/engine.py.229: process: 0.55098 | load: 0.00243 [I 2022-10-27 17:24:14] ...trap/engines/engine.py.230: loss: 0.11572 [I 2022-10-27 17:24:14] ...optimizers/trijoint.py.151: total_norm: 14.931011 [I 2022-10-27 17:24:20] ...trap/engines/engine.py.224: train: epoch 79 | batch 630/2304 [I 2022-10-27 17:24:20] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:28 [I 2022-10-27 17:24:20] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00242 [I 2022-10-27 17:24:20] ...trap/engines/engine.py.230: loss: 0.17270 [I 2022-10-27 17:24:20] ...optimizers/trijoint.py.151: total_norm: 11.480310 [I 2022-10-27 17:24:25] ...trap/engines/engine.py.224: train: epoch 79 | batch 640/2304 [I 2022-10-27 17:24:25] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:26 [I 2022-10-27 17:24:25] ...trap/engines/engine.py.229: process: 0.55493 | load: 0.00247 [I 2022-10-27 17:24:25] ...trap/engines/engine.py.230: loss: 0.16137 [I 2022-10-27 17:24:25] ...optimizers/trijoint.py.151: total_norm: 16.864811 [I 2022-10-27 17:24:31] ...trap/engines/engine.py.224: train: epoch 79 | batch 650/2304 [I 2022-10-27 17:24:31] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:23 [I 2022-10-27 17:24:31] ...trap/engines/engine.py.229: process: 0.58036 | load: 0.00250 [I 2022-10-27 17:24:31] ...trap/engines/engine.py.230: loss: 0.16233 [I 2022-10-27 17:24:31] ...optimizers/trijoint.py.151: total_norm: 11.104097 [I 2022-10-27 17:24:36] ...trap/engines/engine.py.224: train: epoch 79 | batch 660/2304 [I 2022-10-27 17:24:36] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:11 [I 2022-10-27 17:24:36] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00263 [I 2022-10-27 17:24:36] ...trap/engines/engine.py.230: loss: 0.13591 [I 2022-10-27 17:24:36] ...optimizers/trijoint.py.151: total_norm: 24.845839 [I 2022-10-27 17:24:42] ...trap/engines/engine.py.224: train: epoch 79 | batch 670/2304 [I 2022-10-27 17:24:42] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:08 [I 2022-10-27 17:24:42] ...trap/engines/engine.py.229: process: 0.56505 | load: 0.00244 [I 2022-10-27 17:24:42] ...trap/engines/engine.py.230: loss: 0.14526 [I 2022-10-27 17:24:42] ...optimizers/trijoint.py.151: total_norm: 13.701074 [I 2022-10-27 17:24:47] ...trap/engines/engine.py.224: train: epoch 79 | batch 680/2304 [I 2022-10-27 17:24:47] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:08 [I 2022-10-27 17:24:47] ...trap/engines/engine.py.229: process: 0.56420 | load: 0.00244 [I 2022-10-27 17:24:47] ...trap/engines/engine.py.230: loss: 0.13742 [I 2022-10-27 17:24:47] ...optimizers/trijoint.py.151: total_norm: 18.218494 [I 2022-10-27 17:24:53] ...trap/engines/engine.py.224: train: epoch 79 | batch 690/2304 [I 2022-10-27 17:24:53] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:04 [I 2022-10-27 17:24:53] ...trap/engines/engine.py.229: process: 0.57204 | load: 0.00239 [I 2022-10-27 17:24:53] ...trap/engines/engine.py.230: loss: 0.12915 [I 2022-10-27 17:24:53] ...optimizers/trijoint.py.151: total_norm: 15.077746 [I 2022-10-27 17:24:59] ...trap/engines/engine.py.224: train: epoch 79 | batch 700/2304 [I 2022-10-27 17:24:59] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:53 [I 2022-10-27 17:24:59] ...trap/engines/engine.py.229: process: 0.55442 | load: 0.00240 [I 2022-10-27 17:24:59] ...trap/engines/engine.py.230: loss: 0.23420 [I 2022-10-27 17:24:59] ...optimizers/trijoint.py.151: total_norm: 16.322544 [I 2022-10-27 17:25:04] ...trap/engines/engine.py.224: train: epoch 79 | batch 710/2304 [I 2022-10-27 17:25:04] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:50 [I 2022-10-27 17:25:04] ...trap/engines/engine.py.229: process: 0.56571 | load: 0.00242 [I 2022-10-27 17:25:04] ...trap/engines/engine.py.230: loss: 0.16769 [I 2022-10-27 17:25:04] ...optimizers/trijoint.py.151: total_norm: 29.255659 [I 2022-10-27 17:25:10] ...trap/engines/engine.py.224: train: epoch 79 | batch 720/2304 [I 2022-10-27 17:25:10] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:51 [I 2022-10-27 17:25:10] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00241 [I 2022-10-27 17:25:10] ...trap/engines/engine.py.230: loss: 0.09233 [I 2022-10-27 17:25:10] ...optimizers/trijoint.py.151: total_norm: 15.853819 [I 2022-10-27 17:25:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 730/2304 [I 2022-10-27 17:25:15] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:35 [I 2022-10-27 17:25:15] ...trap/engines/engine.py.229: process: 0.55020 | load: 0.00243 [I 2022-10-27 17:25:15] ...trap/engines/engine.py.230: loss: 0.09977 [I 2022-10-27 17:25:15] ...optimizers/trijoint.py.151: total_norm: 10.096388 [I 2022-10-27 17:25:21] ...trap/engines/engine.py.224: train: epoch 79 | batch 740/2304 [I 2022-10-27 17:25:21] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:30 [I 2022-10-27 17:25:21] ...trap/engines/engine.py.229: process: 0.56352 | load: 0.00241 [I 2022-10-27 17:25:21] ...trap/engines/engine.py.230: loss: 0.15100 [I 2022-10-27 17:25:21] ...optimizers/trijoint.py.151: total_norm: 26.354124 [I 2022-10-27 17:25:27] ...trap/engines/engine.py.224: train: epoch 79 | batch 750/2304 [I 2022-10-27 17:25:27] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:19 [I 2022-10-27 17:25:27] ...trap/engines/engine.py.229: process: 0.54680 | load: 0.00243 [I 2022-10-27 17:25:27] ...trap/engines/engine.py.230: loss: 0.18805 [I 2022-10-27 17:25:27] ...optimizers/trijoint.py.151: total_norm: 22.862444 [I 2022-10-27 17:25:32] ...trap/engines/engine.py.224: train: epoch 79 | batch 760/2304 [I 2022-10-27 17:25:32] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:22 [I 2022-10-27 17:25:32] ...trap/engines/engine.py.229: process: 0.56998 | load: 0.00240 [I 2022-10-27 17:25:32] ...trap/engines/engine.py.230: loss: 0.15255 [I 2022-10-27 17:25:32] ...optimizers/trijoint.py.151: total_norm: 22.320902 [I 2022-10-27 17:25:38] ...trap/engines/engine.py.224: train: epoch 79 | batch 770/2304 [I 2022-10-27 17:25:38] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:14 [I 2022-10-27 17:25:38] ...trap/engines/engine.py.229: process: 0.55040 | load: 0.00247 [I 2022-10-27 17:25:38] ...trap/engines/engine.py.230: loss: 0.12490 [I 2022-10-27 17:25:38] ...optimizers/trijoint.py.151: total_norm: 16.140265 [I 2022-10-27 17:25:43] ...trap/engines/engine.py.224: train: epoch 79 | batch 780/2304 [I 2022-10-27 17:25:43] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:07 [I 2022-10-27 17:25:43] ...trap/engines/engine.py.229: process: 0.54374 | load: 0.00245 [I 2022-10-27 17:25:43] ...trap/engines/engine.py.230: loss: 0.17782 [I 2022-10-27 17:25:43] ...optimizers/trijoint.py.151: total_norm: 36.038452 [I 2022-10-27 17:25:49] ...trap/engines/engine.py.224: train: epoch 79 | batch 790/2304 [I 2022-10-27 17:25:49] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:13:57 [I 2022-10-27 17:25:49] ...trap/engines/engine.py.229: process: 0.54799 | load: 0.00241 [I 2022-10-27 17:25:49] ...trap/engines/engine.py.230: loss: 0.11285 [I 2022-10-27 17:25:49] ...optimizers/trijoint.py.151: total_norm: 10.493585 [I 2022-10-27 17:25:54] ...trap/engines/engine.py.224: train: epoch 79 | batch 800/2304 [I 2022-10-27 17:25:54] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:47 [I 2022-10-27 17:25:54] ...trap/engines/engine.py.229: process: 0.54438 | load: 0.00240 [I 2022-10-27 17:25:54] ...trap/engines/engine.py.230: loss: 0.10616 [I 2022-10-27 17:25:54] ...optimizers/trijoint.py.151: total_norm: 7.534935 [I 2022-10-27 17:26:00] ...trap/engines/engine.py.224: train: epoch 79 | batch 810/2304 [I 2022-10-27 17:26:00] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:50 [I 2022-10-27 17:26:00] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00243 [I 2022-10-27 17:26:00] ...trap/engines/engine.py.230: loss: 0.17485 [I 2022-10-27 17:26:00] ...optimizers/trijoint.py.151: total_norm: 18.634760 [I 2022-10-27 17:26:06] ...trap/engines/engine.py.224: train: epoch 79 | batch 820/2304 [I 2022-10-27 17:26:06] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:46 [I 2022-10-27 17:26:06] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00243 [I 2022-10-27 17:26:06] ...trap/engines/engine.py.230: loss: 0.14911 [I 2022-10-27 17:26:06] ...optimizers/trijoint.py.151: total_norm: 20.788260 [I 2022-10-27 17:26:11] ...trap/engines/engine.py.224: train: epoch 79 | batch 830/2304 [I 2022-10-27 17:26:11] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:43 [I 2022-10-27 17:26:11] ...trap/engines/engine.py.229: process: 0.56688 | load: 0.00243 [I 2022-10-27 17:26:11] ...trap/engines/engine.py.230: loss: 0.15970 [I 2022-10-27 17:26:11] ...optimizers/trijoint.py.151: total_norm: 17.500975 [I 2022-10-27 17:26:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 840/2304 [I 2022-10-27 17:26:17] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:38 [I 2022-10-27 17:26:17] ...trap/engines/engine.py.229: process: 0.55688 | load: 0.00244 [I 2022-10-27 17:26:17] ...trap/engines/engine.py.230: loss: 0.11642 [I 2022-10-27 17:26:17] ...optimizers/trijoint.py.151: total_norm: 11.626807 [I 2022-10-27 17:26:23] ...trap/engines/engine.py.224: train: epoch 79 | batch 850/2304 [I 2022-10-27 17:26:23] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:14:05 [I 2022-10-27 17:26:23] ...trap/engines/engine.py.229: process: 0.57865 | load: 0.00248 [I 2022-10-27 17:26:23] ...trap/engines/engine.py.230: loss: 0.17823 [I 2022-10-27 17:26:23] ...optimizers/trijoint.py.151: total_norm: 31.975615 [I 2022-10-27 17:26:28] ...trap/engines/engine.py.224: train: epoch 79 | batch 860/2304 [I 2022-10-27 17:26:28] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:32 [I 2022-10-27 17:26:28] ...trap/engines/engine.py.229: process: 0.55506 | load: 0.00246 [I 2022-10-27 17:26:28] ...trap/engines/engine.py.230: loss: 0.13421 [I 2022-10-27 17:26:28] ...optimizers/trijoint.py.151: total_norm: 15.861631 [I 2022-10-27 17:26:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 870/2304 [I 2022-10-27 17:26:34] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:25 [I 2022-10-27 17:26:34] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00262 [I 2022-10-27 17:26:34] ...trap/engines/engine.py.230: loss: 0.10331 [I 2022-10-27 17:26:34] ...optimizers/trijoint.py.151: total_norm: 9.024348 [I 2022-10-27 17:26:39] ...trap/engines/engine.py.224: train: epoch 79 | batch 880/2304 [I 2022-10-27 17:26:39] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:10 [I 2022-10-27 17:26:39] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00240 [I 2022-10-27 17:26:39] ...trap/engines/engine.py.230: loss: 0.15661 [I 2022-10-27 17:26:39] ...optimizers/trijoint.py.151: total_norm: 20.307764 [I 2022-10-27 17:26:45] ...trap/engines/engine.py.224: train: epoch 79 | batch 890/2304 [I 2022-10-27 17:26:45] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:08 [I 2022-10-27 17:26:45] ...trap/engines/engine.py.229: process: 0.56245 | load: 0.00243 [I 2022-10-27 17:26:45] ...trap/engines/engine.py.230: loss: 0.16362 [I 2022-10-27 17:26:45] ...optimizers/trijoint.py.151: total_norm: 16.566742 [I 2022-10-27 17:26:51] ...trap/engines/engine.py.224: train: epoch 79 | batch 900/2304 [I 2022-10-27 17:26:51] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:12:53 [I 2022-10-27 17:26:51] ...trap/engines/engine.py.229: process: 0.53823 | load: 0.00242 [I 2022-10-27 17:26:51] ...trap/engines/engine.py.230: loss: 0.21820 [I 2022-10-27 17:26:51] ...optimizers/trijoint.py.151: total_norm: 23.948841 [I 2022-10-27 17:26:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 910/2304 [I 2022-10-27 17:26:56] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:49 [I 2022-10-27 17:26:56] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00242 [I 2022-10-27 17:26:56] ...trap/engines/engine.py.230: loss: 0.14562 [I 2022-10-27 17:26:56] ...optimizers/trijoint.py.151: total_norm: 13.278833 [I 2022-10-27 17:27:02] ...trap/engines/engine.py.224: train: epoch 79 | batch 920/2304 [I 2022-10-27 17:27:02] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:43 [I 2022-10-27 17:27:02] ...trap/engines/engine.py.229: process: 0.54819 | load: 0.00242 [I 2022-10-27 17:27:02] ...trap/engines/engine.py.230: loss: 0.15519 [I 2022-10-27 17:27:02] ...optimizers/trijoint.py.151: total_norm: 37.810917 [I 2022-10-27 17:27:07] ...trap/engines/engine.py.224: train: epoch 79 | batch 930/2304 [I 2022-10-27 17:27:07] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:43 [I 2022-10-27 17:27:07] ...trap/engines/engine.py.229: process: 0.55642 | load: 0.00241 [I 2022-10-27 17:27:07] ...trap/engines/engine.py.230: loss: 0.09775 [I 2022-10-27 17:27:07] ...optimizers/trijoint.py.151: total_norm: 12.619967 [I 2022-10-27 17:27:13] ...trap/engines/engine.py.224: train: epoch 79 | batch 940/2304 [I 2022-10-27 17:27:13] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:45 [I 2022-10-27 17:27:13] ...trap/engines/engine.py.229: process: 0.56890 | load: 0.00245 [I 2022-10-27 17:27:13] ...trap/engines/engine.py.230: loss: 0.13496 [I 2022-10-27 17:27:13] ...optimizers/trijoint.py.151: total_norm: 14.271016 [I 2022-10-27 17:27:18] ...trap/engines/engine.py.224: train: epoch 79 | batch 950/2304 [I 2022-10-27 17:27:18] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:35 [I 2022-10-27 17:27:18] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00240 [I 2022-10-27 17:27:18] ...trap/engines/engine.py.230: loss: 0.11492 [I 2022-10-27 17:27:18] ...optimizers/trijoint.py.151: total_norm: 10.100142 [I 2022-10-27 17:27:24] ...trap/engines/engine.py.224: train: epoch 79 | batch 960/2304 [I 2022-10-27 17:27:24] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:27 [I 2022-10-27 17:27:24] ...trap/engines/engine.py.229: process: 0.54994 | load: 0.00241 [I 2022-10-27 17:27:24] ...trap/engines/engine.py.230: loss: 0.13469 [I 2022-10-27 17:27:24] ...optimizers/trijoint.py.151: total_norm: 14.101192 [I 2022-10-27 17:27:30] ...trap/engines/engine.py.224: train: epoch 79 | batch 970/2304 [I 2022-10-27 17:27:30] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:23 [I 2022-10-27 17:27:30] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00249 [I 2022-10-27 17:27:30] ...trap/engines/engine.py.230: loss: 0.11976 [I 2022-10-27 17:27:30] ...optimizers/trijoint.py.151: total_norm: 18.280363 [I 2022-10-27 17:27:35] ...trap/engines/engine.py.224: train: epoch 79 | batch 980/2304 [I 2022-10-27 17:27:35] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:20 [I 2022-10-27 17:27:35] ...trap/engines/engine.py.229: process: 0.56637 | load: 0.00249 [I 2022-10-27 17:27:35] ...trap/engines/engine.py.230: loss: 0.12700 [I 2022-10-27 17:27:35] ...optimizers/trijoint.py.151: total_norm: 27.548819 [I 2022-10-27 17:27:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 990/2304 [I 2022-10-27 17:27:41] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:13 [I 2022-10-27 17:27:41] ...trap/engines/engine.py.229: process: 0.57144 | load: 0.00240 [I 2022-10-27 17:27:41] ...trap/engines/engine.py.230: loss: 0.11601 [I 2022-10-27 17:27:41] ...optimizers/trijoint.py.151: total_norm: 9.288253 [I 2022-10-27 17:27:46] ...trap/engines/engine.py.224: train: epoch 79 | batch 1000/2304 [I 2022-10-27 17:27:46] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:00 [I 2022-10-27 17:27:46] ...trap/engines/engine.py.229: process: 0.54219 | load: 0.00242 [I 2022-10-27 17:27:46] ...trap/engines/engine.py.230: loss: 0.10546 [I 2022-10-27 17:27:46] ...optimizers/trijoint.py.151: total_norm: 8.283662 [I 2022-10-27 17:27:52] ...trap/engines/engine.py.224: train: epoch 79 | batch 1010/2304 [I 2022-10-27 17:27:52] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:09 [I 2022-10-27 17:27:52] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00244 [I 2022-10-27 17:27:52] ...trap/engines/engine.py.230: loss: 0.10188 [I 2022-10-27 17:27:52] ...optimizers/trijoint.py.151: total_norm: 31.159117 [I 2022-10-27 17:27:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 1020/2304 [I 2022-10-27 17:27:58] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:12:00 [I 2022-10-27 17:27:58] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00242 [I 2022-10-27 17:27:58] ...trap/engines/engine.py.230: loss: 0.10448 [I 2022-10-27 17:27:58] ...optimizers/trijoint.py.151: total_norm: 15.293177 [I 2022-10-27 17:28:03] ...trap/engines/engine.py.224: train: epoch 79 | batch 1030/2304 [I 2022-10-27 17:28:03] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:49 [I 2022-10-27 17:28:03] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00242 [I 2022-10-27 17:28:03] ...trap/engines/engine.py.230: loss: 0.15520 [I 2022-10-27 17:28:03] ...optimizers/trijoint.py.151: total_norm: 22.897772 [I 2022-10-27 17:28:09] ...trap/engines/engine.py.224: train: epoch 79 | batch 1040/2304 [I 2022-10-27 17:28:09] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:40 [I 2022-10-27 17:28:09] ...trap/engines/engine.py.229: process: 0.53871 | load: 0.00248 [I 2022-10-27 17:28:09] ...trap/engines/engine.py.230: loss: 0.14945 [I 2022-10-27 17:28:09] ...optimizers/trijoint.py.151: total_norm: 25.737040 [I 2022-10-27 17:28:14] ...trap/engines/engine.py.224: train: epoch 79 | batch 1050/2304 [I 2022-10-27 17:28:14] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:33 [I 2022-10-27 17:28:14] ...trap/engines/engine.py.229: process: 0.54674 | load: 0.00248 [I 2022-10-27 17:28:14] ...trap/engines/engine.py.230: loss: 0.11821 [I 2022-10-27 17:28:14] ...optimizers/trijoint.py.151: total_norm: 15.686929 [I 2022-10-27 17:28:20] ...trap/engines/engine.py.224: train: epoch 79 | batch 1060/2304 [I 2022-10-27 17:28:20] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:30 [I 2022-10-27 17:28:20] ...trap/engines/engine.py.229: process: 0.55417 | load: 0.00244 [I 2022-10-27 17:28:20] ...trap/engines/engine.py.230: loss: 0.10860 [I 2022-10-27 17:28:20] ...optimizers/trijoint.py.151: total_norm: 14.124807 [I 2022-10-27 17:28:25] ...trap/engines/engine.py.224: train: epoch 79 | batch 1070/2304 [I 2022-10-27 17:28:25] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:21 [I 2022-10-27 17:28:25] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00261 [I 2022-10-27 17:28:25] ...trap/engines/engine.py.230: loss: 0.13447 [I 2022-10-27 17:28:25] ...optimizers/trijoint.py.151: total_norm: 23.841583 [I 2022-10-27 17:28:31] ...trap/engines/engine.py.224: train: epoch 79 | batch 1080/2304 [I 2022-10-27 17:28:31] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:20 [I 2022-10-27 17:28:31] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00240 [I 2022-10-27 17:28:31] ...trap/engines/engine.py.230: loss: 0.13389 [I 2022-10-27 17:28:31] ...optimizers/trijoint.py.151: total_norm: 25.433825 [I 2022-10-27 17:28:37] ...trap/engines/engine.py.224: train: epoch 79 | batch 1090/2304 [I 2022-10-27 17:28:37] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:11 [I 2022-10-27 17:28:37] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00241 [I 2022-10-27 17:28:37] ...trap/engines/engine.py.230: loss: 0.11705 [I 2022-10-27 17:28:37] ...optimizers/trijoint.py.151: total_norm: 17.508942 [I 2022-10-27 17:28:42] ...trap/engines/engine.py.224: train: epoch 79 | batch 1100/2304 [I 2022-10-27 17:28:42] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:06 [I 2022-10-27 17:28:42] ...trap/engines/engine.py.229: process: 0.54939 | load: 0.00244 [I 2022-10-27 17:28:42] ...trap/engines/engine.py.230: loss: 0.11475 [I 2022-10-27 17:28:42] ...optimizers/trijoint.py.151: total_norm: 20.755821 [I 2022-10-27 17:28:48] ...trap/engines/engine.py.224: train: epoch 79 | batch 1110/2304 [I 2022-10-27 17:28:48] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:16 [I 2022-10-27 17:28:48] ...trap/engines/engine.py.229: process: 0.55045 | load: 0.00244 [I 2022-10-27 17:28:48] ...trap/engines/engine.py.230: loss: 0.18553 [I 2022-10-27 17:28:48] ...optimizers/trijoint.py.151: total_norm: 26.858032 [I 2022-10-27 17:28:53] ...trap/engines/engine.py.224: train: epoch 79 | batch 1120/2304 [I 2022-10-27 17:28:53] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:01 [I 2022-10-27 17:28:53] ...trap/engines/engine.py.229: process: 0.54897 | load: 0.00242 [I 2022-10-27 17:28:53] ...trap/engines/engine.py.230: loss: 0.17942 [I 2022-10-27 17:28:53] ...optimizers/trijoint.py.151: total_norm: 22.082502 [I 2022-10-27 17:28:59] ...trap/engines/engine.py.224: train: epoch 79 | batch 1130/2304 [I 2022-10-27 17:28:59] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:50 [I 2022-10-27 17:28:59] ...trap/engines/engine.py.229: process: 0.55043 | load: 0.00242 [I 2022-10-27 17:28:59] ...trap/engines/engine.py.230: loss: 0.13974 [I 2022-10-27 17:28:59] ...optimizers/trijoint.py.151: total_norm: 19.305943 [I 2022-10-27 17:29:05] ...trap/engines/engine.py.224: train: epoch 79 | batch 1140/2304 [I 2022-10-27 17:29:05] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:47 [I 2022-10-27 17:29:05] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00241 [I 2022-10-27 17:29:05] ...trap/engines/engine.py.230: loss: 0.09237 [I 2022-10-27 17:29:05] ...optimizers/trijoint.py.151: total_norm: 11.178725 [I 2022-10-27 17:29:10] ...trap/engines/engine.py.224: train: epoch 79 | batch 1150/2304 [I 2022-10-27 17:29:10] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:42 [I 2022-10-27 17:29:10] ...trap/engines/engine.py.229: process: 0.55845 | load: 0.00239 [I 2022-10-27 17:29:10] ...trap/engines/engine.py.230: loss: 0.10949 [I 2022-10-27 17:29:10] ...optimizers/trijoint.py.151: total_norm: 12.918937 [I 2022-10-27 17:29:16] ...trap/engines/engine.py.224: train: epoch 79 | batch 1160/2304 [I 2022-10-27 17:29:16] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:36 [I 2022-10-27 17:29:16] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00245 [I 2022-10-27 17:29:16] ...trap/engines/engine.py.230: loss: 0.18197 [I 2022-10-27 17:29:16] ...optimizers/trijoint.py.151: total_norm: 27.199657 [I 2022-10-27 17:29:21] ...trap/engines/engine.py.224: train: epoch 79 | batch 1170/2304 [I 2022-10-27 17:29:21] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:33 [I 2022-10-27 17:29:21] ...trap/engines/engine.py.229: process: 0.57261 | load: 0.00542 [I 2022-10-27 17:29:21] ...trap/engines/engine.py.230: loss: 0.16335 [I 2022-10-27 17:29:21] ...optimizers/trijoint.py.151: total_norm: 21.979679 [I 2022-10-27 17:29:27] ...trap/engines/engine.py.224: train: epoch 79 | batch 1180/2304 [I 2022-10-27 17:29:27] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:21 [I 2022-10-27 17:29:27] ...trap/engines/engine.py.229: process: 0.54630 | load: 0.00250 [I 2022-10-27 17:29:27] ...trap/engines/engine.py.230: loss: 0.11462 [I 2022-10-27 17:29:27] ...optimizers/trijoint.py.151: total_norm: 15.815133 [I 2022-10-27 17:29:32] ...trap/engines/engine.py.224: train: epoch 79 | batch 1190/2304 [I 2022-10-27 17:29:32] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:21 [I 2022-10-27 17:29:32] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00244 [I 2022-10-27 17:29:32] ...trap/engines/engine.py.230: loss: 0.12150 [I 2022-10-27 17:29:32] ...optimizers/trijoint.py.151: total_norm: 22.765993 [I 2022-10-27 17:29:38] ...trap/engines/engine.py.224: train: epoch 79 | batch 1200/2304 [I 2022-10-27 17:29:38] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:18 [I 2022-10-27 17:29:38] ...trap/engines/engine.py.229: process: 0.55524 | load: 0.00261 [I 2022-10-27 17:29:38] ...trap/engines/engine.py.230: loss: 0.14405 [I 2022-10-27 17:29:38] ...optimizers/trijoint.py.151: total_norm: 32.652264 [I 2022-10-27 17:29:44] ...trap/engines/engine.py.224: train: epoch 79 | batch 1210/2304 [I 2022-10-27 17:29:44] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:03 [I 2022-10-27 17:29:44] ...trap/engines/engine.py.229: process: 0.54707 | load: 0.00238 [I 2022-10-27 17:29:44] ...trap/engines/engine.py.230: loss: 0.11096 [I 2022-10-27 17:29:44] ...optimizers/trijoint.py.151: total_norm: 16.290735 [I 2022-10-27 17:29:49] ...trap/engines/engine.py.224: train: epoch 79 | batch 1220/2304 [I 2022-10-27 17:29:49] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:23 [I 2022-10-27 17:29:49] ...trap/engines/engine.py.229: process: 0.66073 | load: 0.00242 [I 2022-10-27 17:29:49] ...trap/engines/engine.py.230: loss: 0.12501 [I 2022-10-27 17:29:49] ...optimizers/trijoint.py.151: total_norm: 12.329724 [I 2022-10-27 17:29:55] ...trap/engines/engine.py.224: train: epoch 79 | batch 1230/2304 [I 2022-10-27 17:29:55] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:02 [I 2022-10-27 17:29:55] ...trap/engines/engine.py.229: process: 0.55351 | load: 0.00251 [I 2022-10-27 17:29:55] ...trap/engines/engine.py.230: loss: 0.18342 [I 2022-10-27 17:29:55] ...optimizers/trijoint.py.151: total_norm: 12.355396 [I 2022-10-27 17:30:00] ...trap/engines/engine.py.224: train: epoch 79 | batch 1240/2304 [I 2022-10-27 17:30:00] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:52 [I 2022-10-27 17:30:00] ...trap/engines/engine.py.229: process: 0.55829 | load: 0.00243 [I 2022-10-27 17:30:00] ...trap/engines/engine.py.230: loss: 0.11311 [I 2022-10-27 17:30:00] ...optimizers/trijoint.py.151: total_norm: 16.880581 [I 2022-10-27 17:30:06] ...trap/engines/engine.py.224: train: epoch 79 | batch 1250/2304 [I 2022-10-27 17:30:06] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:49 [I 2022-10-27 17:30:06] ...trap/engines/engine.py.229: process: 0.57307 | load: 0.00241 [I 2022-10-27 17:30:06] ...trap/engines/engine.py.230: loss: 0.10933 [I 2022-10-27 17:30:06] ...optimizers/trijoint.py.151: total_norm: 20.801325 [I 2022-10-27 17:30:12] ...trap/engines/engine.py.224: train: epoch 79 | batch 1260/2304 [I 2022-10-27 17:30:12] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:45 [I 2022-10-27 17:30:12] ...trap/engines/engine.py.229: process: 0.56767 | load: 0.00250 [I 2022-10-27 17:30:12] ...trap/engines/engine.py.230: loss: 0.09721 [I 2022-10-27 17:30:12] ...optimizers/trijoint.py.151: total_norm: 18.475052 [I 2022-10-27 17:30:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 1270/2304 [I 2022-10-27 17:30:17] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:33 [I 2022-10-27 17:30:17] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00245 [I 2022-10-27 17:30:17] ...trap/engines/engine.py.230: loss: 0.18999 [I 2022-10-27 17:30:17] ...optimizers/trijoint.py.151: total_norm: 49.449558 [I 2022-10-27 17:30:42] ...trap/engines/engine.py.224: train: epoch 79 | batch 1280/2304 [I 2022-10-27 17:30:42] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 1:02:40 [I 2022-10-27 17:30:42] ...trap/engines/engine.py.229: process: 0.56581 | load: 0.00240 [I 2022-10-27 17:30:42] ...trap/engines/engine.py.230: loss: 0.14202 [I 2022-10-27 17:30:42] ...optimizers/trijoint.py.151: total_norm: 12.093978 [I 2022-10-27 17:30:48] ...trap/engines/engine.py.224: train: epoch 79 | batch 1290/2304 [I 2022-10-27 17:30:48] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:15:03 [I 2022-10-27 17:30:48] ...trap/engines/engine.py.229: process: 0.56795 | load: 0.00238 [I 2022-10-27 17:30:48] ...trap/engines/engine.py.230: loss: 0.13662 [I 2022-10-27 17:30:48] ...optimizers/trijoint.py.151: total_norm: 9.332205 [I 2022-10-27 17:30:53] ...trap/engines/engine.py.224: train: epoch 79 | batch 1300/2304 [I 2022-10-27 17:30:53] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:53 [I 2022-10-27 17:30:53] ...trap/engines/engine.py.229: process: 0.54686 | load: 0.00237 [I 2022-10-27 17:30:53] ...trap/engines/engine.py.230: loss: 0.15273 [I 2022-10-27 17:30:53] ...optimizers/trijoint.py.151: total_norm: 26.978189 [I 2022-10-27 17:30:59] ...trap/engines/engine.py.224: train: epoch 79 | batch 1310/2304 [I 2022-10-27 17:30:59] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:09:19 [I 2022-10-27 17:30:59] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00235 [I 2022-10-27 17:30:59] ...trap/engines/engine.py.230: loss: 0.09146 [I 2022-10-27 17:30:59] ...optimizers/trijoint.py.151: total_norm: 9.932632 [I 2022-10-27 17:31:05] ...trap/engines/engine.py.224: train: epoch 79 | batch 1320/2304 [I 2022-10-27 17:31:05] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:06 [I 2022-10-27 17:31:05] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00239 [I 2022-10-27 17:31:05] ...trap/engines/engine.py.230: loss: 0.10261 [I 2022-10-27 17:31:05] ...optimizers/trijoint.py.151: total_norm: 11.079803 [I 2022-10-27 17:31:10] ...trap/engines/engine.py.224: train: epoch 79 | batch 1330/2304 [I 2022-10-27 17:31:10] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:09:04 [I 2022-10-27 17:31:10] ...trap/engines/engine.py.229: process: 0.56183 | load: 0.00236 [I 2022-10-27 17:31:10] ...trap/engines/engine.py.230: loss: 0.17640 [I 2022-10-27 17:31:10] ...optimizers/trijoint.py.151: total_norm: 20.867062 [I 2022-10-27 17:31:16] ...trap/engines/engine.py.224: train: epoch 79 | batch 1340/2304 [I 2022-10-27 17:31:16] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:53 [I 2022-10-27 17:31:16] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00237 [I 2022-10-27 17:31:16] ...trap/engines/engine.py.230: loss: 0.15176 [I 2022-10-27 17:31:16] ...optimizers/trijoint.py.151: total_norm: 21.253986 [I 2022-10-27 17:31:21] ...trap/engines/engine.py.224: train: epoch 79 | batch 1350/2304 [I 2022-10-27 17:31:21] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:51 [I 2022-10-27 17:31:21] ...trap/engines/engine.py.229: process: 0.56324 | load: 0.00238 [I 2022-10-27 17:31:21] ...trap/engines/engine.py.230: loss: 0.21770 [I 2022-10-27 17:31:21] ...optimizers/trijoint.py.151: total_norm: 26.567131 [I 2022-10-27 17:31:27] ...trap/engines/engine.py.224: train: epoch 79 | batch 1360/2304 [I 2022-10-27 17:31:27] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:42 [I 2022-10-27 17:31:27] ...trap/engines/engine.py.229: process: 0.55031 | load: 0.00242 [I 2022-10-27 17:31:27] ...trap/engines/engine.py.230: loss: 0.10760 [I 2022-10-27 17:31:27] ...optimizers/trijoint.py.151: total_norm: 10.472713 [I 2022-10-27 17:31:32] ...trap/engines/engine.py.224: train: epoch 79 | batch 1370/2304 [I 2022-10-27 17:31:32] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:41 [I 2022-10-27 17:31:32] ...trap/engines/engine.py.229: process: 0.56819 | load: 0.00239 [I 2022-10-27 17:31:32] ...trap/engines/engine.py.230: loss: 0.15518 [I 2022-10-27 17:31:32] ...optimizers/trijoint.py.151: total_norm: 22.650244 [I 2022-10-27 17:31:38] ...trap/engines/engine.py.224: train: epoch 79 | batch 1380/2304 [I 2022-10-27 17:31:38] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:34 [I 2022-10-27 17:31:38] ...trap/engines/engine.py.229: process: 0.54922 | load: 0.00239 [I 2022-10-27 17:31:38] ...trap/engines/engine.py.230: loss: 0.12917 [I 2022-10-27 17:31:38] ...optimizers/trijoint.py.151: total_norm: 15.596267 [I 2022-10-27 17:31:44] ...trap/engines/engine.py.224: train: epoch 79 | batch 1390/2304 [I 2022-10-27 17:31:44] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:30 [I 2022-10-27 17:31:44] ...trap/engines/engine.py.229: process: 0.56080 | load: 0.00239 [I 2022-10-27 17:31:44] ...trap/engines/engine.py.230: loss: 0.18141 [I 2022-10-27 17:31:44] ...optimizers/trijoint.py.151: total_norm: 15.659269 [I 2022-10-27 17:31:49] ...trap/engines/engine.py.224: train: epoch 79 | batch 1400/2304 [I 2022-10-27 17:31:49] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:23 [I 2022-10-27 17:31:49] ...trap/engines/engine.py.229: process: 0.55817 | load: 0.00246 [I 2022-10-27 17:31:49] ...trap/engines/engine.py.230: loss: 0.14819 [I 2022-10-27 17:31:49] ...optimizers/trijoint.py.151: total_norm: 23.086836 [I 2022-10-27 17:31:55] ...trap/engines/engine.py.224: train: epoch 79 | batch 1410/2304 [I 2022-10-27 17:31:55] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:17 [I 2022-10-27 17:31:55] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00241 [I 2022-10-27 17:31:55] ...trap/engines/engine.py.230: loss: 0.12399 [I 2022-10-27 17:31:55] ...optimizers/trijoint.py.151: total_norm: 11.917910 [I 2022-10-27 17:32:00] ...trap/engines/engine.py.224: train: epoch 79 | batch 1420/2304 [I 2022-10-27 17:32:00] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:16 [I 2022-10-27 17:32:00] ...trap/engines/engine.py.229: process: 0.55897 | load: 0.00241 [I 2022-10-27 17:32:00] ...trap/engines/engine.py.230: loss: 0.23460 [I 2022-10-27 17:32:00] ...optimizers/trijoint.py.151: total_norm: 30.201248 [I 2022-10-27 17:32:06] ...trap/engines/engine.py.224: train: epoch 79 | batch 1430/2304 [I 2022-10-27 17:32:06] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:08:09 [I 2022-10-27 17:32:06] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00241 [I 2022-10-27 17:32:06] ...trap/engines/engine.py.230: loss: 0.20685 [I 2022-10-27 17:32:06] ...optimizers/trijoint.py.151: total_norm: 24.810556 [I 2022-10-27 17:32:12] ...trap/engines/engine.py.224: train: epoch 79 | batch 1440/2304 [I 2022-10-27 17:32:12] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:08:04 [I 2022-10-27 17:32:12] ...trap/engines/engine.py.229: process: 0.57142 | load: 0.00242 [I 2022-10-27 17:32:12] ...trap/engines/engine.py.230: loss: 0.10462 [I 2022-10-27 17:32:12] ...optimizers/trijoint.py.151: total_norm: 13.572777 [I 2022-10-27 17:32:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 1450/2304 [I 2022-10-27 17:32:17] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:59 [I 2022-10-27 17:32:17] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00245 [I 2022-10-27 17:32:17] ...trap/engines/engine.py.230: loss: 0.11043 [I 2022-10-27 17:32:17] ...optimizers/trijoint.py.151: total_norm: 11.563708 [I 2022-10-27 17:32:23] ...trap/engines/engine.py.224: train: epoch 79 | batch 1460/2304 [I 2022-10-27 17:32:23] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:58 [I 2022-10-27 17:32:23] ...trap/engines/engine.py.229: process: 0.58104 | load: 0.00244 [I 2022-10-27 17:32:23] ...trap/engines/engine.py.230: loss: 0.17568 [I 2022-10-27 17:32:23] ...optimizers/trijoint.py.151: total_norm: 25.189127 [I 2022-10-27 17:32:29] ...trap/engines/engine.py.224: train: epoch 79 | batch 1470/2304 [I 2022-10-27 17:32:29] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:54 [I 2022-10-27 17:32:29] ...trap/engines/engine.py.229: process: 0.56179 | load: 0.00246 [I 2022-10-27 17:32:29] ...trap/engines/engine.py.230: loss: 0.13960 [I 2022-10-27 17:32:29] ...optimizers/trijoint.py.151: total_norm: 19.578650 [I 2022-10-27 17:32:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 1480/2304 [I 2022-10-27 17:32:34] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:47 [I 2022-10-27 17:32:34] ...trap/engines/engine.py.229: process: 0.57511 | load: 0.00251 [I 2022-10-27 17:32:34] ...trap/engines/engine.py.230: loss: 0.14052 [I 2022-10-27 17:32:34] ...optimizers/trijoint.py.151: total_norm: 10.530926 [I 2022-10-27 17:32:40] ...trap/engines/engine.py.224: train: epoch 79 | batch 1490/2304 [I 2022-10-27 17:32:40] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:33 [I 2022-10-27 17:32:40] ...trap/engines/engine.py.229: process: 0.54215 | load: 0.00249 [I 2022-10-27 17:32:40] ...trap/engines/engine.py.230: loss: 0.10491 [I 2022-10-27 17:32:40] ...optimizers/trijoint.py.151: total_norm: 10.804042 [I 2022-10-27 17:32:46] ...trap/engines/engine.py.224: train: epoch 79 | batch 1500/2304 [I 2022-10-27 17:32:46] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:30 [I 2022-10-27 17:32:46] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00245 [I 2022-10-27 17:32:46] ...trap/engines/engine.py.230: loss: 0.13553 [I 2022-10-27 17:32:46] ...optimizers/trijoint.py.151: total_norm: 19.188065 [I 2022-10-27 17:32:51] ...trap/engines/engine.py.224: train: epoch 79 | batch 1510/2304 [I 2022-10-27 17:32:51] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:22 [I 2022-10-27 17:32:51] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00245 [I 2022-10-27 17:32:51] ...trap/engines/engine.py.230: loss: 0.10600 [I 2022-10-27 17:32:51] ...optimizers/trijoint.py.151: total_norm: 13.409884 [I 2022-10-27 17:32:57] ...trap/engines/engine.py.224: train: epoch 79 | batch 1520/2304 [I 2022-10-27 17:32:57] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:23 [I 2022-10-27 17:32:57] ...trap/engines/engine.py.229: process: 0.58630 | load: 0.00246 [I 2022-10-27 17:32:57] ...trap/engines/engine.py.230: loss: 0.09570 [I 2022-10-27 17:32:57] ...optimizers/trijoint.py.151: total_norm: 8.778413 [I 2022-10-27 17:33:02] ...trap/engines/engine.py.224: train: epoch 79 | batch 1530/2304 [I 2022-10-27 17:33:02] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:07:16 [I 2022-10-27 17:33:02] ...trap/engines/engine.py.229: process: 0.55190 | load: 0.00246 [I 2022-10-27 17:33:02] ...trap/engines/engine.py.230: loss: 0.12448 [I 2022-10-27 17:33:02] ...optimizers/trijoint.py.151: total_norm: 6.858130 [I 2022-10-27 17:33:08] ...trap/engines/engine.py.224: train: epoch 79 | batch 1540/2304 [I 2022-10-27 17:33:08] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:07:12 [I 2022-10-27 17:33:08] ...trap/engines/engine.py.229: process: 0.56397 | load: 0.00245 [I 2022-10-27 17:33:08] ...trap/engines/engine.py.230: loss: 0.13857 [I 2022-10-27 17:33:08] ...optimizers/trijoint.py.151: total_norm: 20.896275 [I 2022-10-27 17:33:14] ...trap/engines/engine.py.224: train: epoch 79 | batch 1550/2304 [I 2022-10-27 17:33:14] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:07:02 [I 2022-10-27 17:33:14] ...trap/engines/engine.py.229: process: 0.54893 | load: 0.00247 [I 2022-10-27 17:33:14] ...trap/engines/engine.py.230: loss: 0.14678 [I 2022-10-27 17:33:14] ...optimizers/trijoint.py.151: total_norm: 14.280115 [I 2022-10-27 17:33:19] ...trap/engines/engine.py.224: train: epoch 79 | batch 1560/2304 [I 2022-10-27 17:33:19] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:59 [I 2022-10-27 17:33:19] ...trap/engines/engine.py.229: process: 0.56334 | load: 0.00245 [I 2022-10-27 17:33:19] ...trap/engines/engine.py.230: loss: 0.19633 [I 2022-10-27 17:33:19] ...optimizers/trijoint.py.151: total_norm: 33.774719 [I 2022-10-27 17:33:25] ...trap/engines/engine.py.224: train: epoch 79 | batch 1570/2304 [I 2022-10-27 17:33:25] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:53 [I 2022-10-27 17:33:25] ...trap/engines/engine.py.229: process: 0.53909 | load: 0.00246 [I 2022-10-27 17:33:25] ...trap/engines/engine.py.230: loss: 0.18434 [I 2022-10-27 17:33:25] ...optimizers/trijoint.py.151: total_norm: 24.462395 [I 2022-10-27 17:33:31] ...trap/engines/engine.py.224: train: epoch 79 | batch 1580/2304 [I 2022-10-27 17:33:31] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:50 [I 2022-10-27 17:33:31] ...trap/engines/engine.py.229: process: 0.57457 | load: 0.00246 [I 2022-10-27 17:33:31] ...trap/engines/engine.py.230: loss: 0.11256 [I 2022-10-27 17:33:31] ...optimizers/trijoint.py.151: total_norm: 15.297386 [I 2022-10-27 17:33:36] ...trap/engines/engine.py.224: train: epoch 79 | batch 1590/2304 [I 2022-10-27 17:33:36] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:43 [I 2022-10-27 17:33:36] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00247 [I 2022-10-27 17:33:36] ...trap/engines/engine.py.230: loss: 0.19991 [I 2022-10-27 17:33:36] ...optimizers/trijoint.py.151: total_norm: 12.930182 [I 2022-10-27 17:33:42] ...trap/engines/engine.py.224: train: epoch 79 | batch 1600/2304 [I 2022-10-27 17:33:42] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:40 [I 2022-10-27 17:33:42] ...trap/engines/engine.py.229: process: 0.57313 | load: 0.00247 [I 2022-10-27 17:33:42] ...trap/engines/engine.py.230: loss: 0.14705 [I 2022-10-27 17:33:42] ...optimizers/trijoint.py.151: total_norm: 22.662910 [I 2022-10-27 17:33:48] ...trap/engines/engine.py.224: train: epoch 79 | batch 1610/2304 [I 2022-10-27 17:33:48] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:29 [I 2022-10-27 17:33:48] ...trap/engines/engine.py.229: process: 0.54580 | load: 0.00245 [I 2022-10-27 17:33:48] ...trap/engines/engine.py.230: loss: 0.12770 [I 2022-10-27 17:33:48] ...optimizers/trijoint.py.151: total_norm: 7.337026 [I 2022-10-27 17:33:53] ...trap/engines/engine.py.224: train: epoch 79 | batch 1620/2304 [I 2022-10-27 17:33:53] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:19 [I 2022-10-27 17:33:53] ...trap/engines/engine.py.229: process: 0.55539 | load: 0.00242 [I 2022-10-27 17:33:53] ...trap/engines/engine.py.230: loss: 0.10654 [I 2022-10-27 17:33:53] ...optimizers/trijoint.py.151: total_norm: 7.637664 [I 2022-10-27 17:33:59] ...trap/engines/engine.py.224: train: epoch 79 | batch 1630/2304 [I 2022-10-27 17:33:59] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:06:16 [I 2022-10-27 17:33:59] ...trap/engines/engine.py.229: process: 0.54913 | load: 0.00245 [I 2022-10-27 17:33:59] ...trap/engines/engine.py.230: loss: 0.10266 [I 2022-10-27 17:33:59] ...optimizers/trijoint.py.151: total_norm: 20.164471 [I 2022-10-27 17:34:05] ...trap/engines/engine.py.224: train: epoch 79 | batch 1640/2304 [I 2022-10-27 17:34:05] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:06:11 [I 2022-10-27 17:34:05] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00244 [I 2022-10-27 17:34:05] ...trap/engines/engine.py.230: loss: 0.15166 [I 2022-10-27 17:34:05] ...optimizers/trijoint.py.151: total_norm: 12.588910 [I 2022-10-27 17:34:10] ...trap/engines/engine.py.224: train: epoch 79 | batch 1650/2304 [I 2022-10-27 17:34:10] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:06:09 [I 2022-10-27 17:34:10] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00258 [I 2022-10-27 17:34:10] ...trap/engines/engine.py.230: loss: 0.15872 [I 2022-10-27 17:34:10] ...optimizers/trijoint.py.151: total_norm: 16.983114 [I 2022-10-27 17:34:16] ...trap/engines/engine.py.224: train: epoch 79 | batch 1660/2304 [I 2022-10-27 17:34:16] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:06:00 [I 2022-10-27 17:34:16] ...trap/engines/engine.py.229: process: 0.55138 | load: 0.00266 [I 2022-10-27 17:34:16] ...trap/engines/engine.py.230: loss: 0.09433 [I 2022-10-27 17:34:16] ...optimizers/trijoint.py.151: total_norm: 14.979792 [I 2022-10-27 17:34:21] ...trap/engines/engine.py.224: train: epoch 79 | batch 1670/2304 [I 2022-10-27 17:34:21] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:53 [I 2022-10-27 17:34:21] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00250 [I 2022-10-27 17:34:21] ...trap/engines/engine.py.230: loss: 0.21204 [I 2022-10-27 17:34:21] ...optimizers/trijoint.py.151: total_norm: 11.044968 [I 2022-10-27 17:34:27] ...trap/engines/engine.py.224: train: epoch 79 | batch 1680/2304 [I 2022-10-27 17:34:27] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:46 [I 2022-10-27 17:34:27] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00254 [I 2022-10-27 17:34:27] ...trap/engines/engine.py.230: loss: 0.11551 [I 2022-10-27 17:34:27] ...optimizers/trijoint.py.151: total_norm: 21.652319 [I 2022-10-27 17:34:33] ...trap/engines/engine.py.224: train: epoch 79 | batch 1690/2304 [I 2022-10-27 17:34:33] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:42 [I 2022-10-27 17:34:33] ...trap/engines/engine.py.229: process: 0.55039 | load: 0.00251 [I 2022-10-27 17:34:33] ...trap/engines/engine.py.230: loss: 0.23736 [I 2022-10-27 17:34:33] ...optimizers/trijoint.py.151: total_norm: 18.495955 [I 2022-10-27 17:34:38] ...trap/engines/engine.py.224: train: epoch 79 | batch 1700/2304 [I 2022-10-27 17:34:38] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:37 [I 2022-10-27 17:34:38] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00253 [I 2022-10-27 17:34:38] ...trap/engines/engine.py.230: loss: 0.14438 [I 2022-10-27 17:34:38] ...optimizers/trijoint.py.151: total_norm: 14.761595 [I 2022-10-27 17:34:44] ...trap/engines/engine.py.224: train: epoch 79 | batch 1710/2304 [I 2022-10-27 17:34:44] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:31 [I 2022-10-27 17:34:44] ...trap/engines/engine.py.229: process: 0.54105 | load: 0.00252 [I 2022-10-27 17:34:44] ...trap/engines/engine.py.230: loss: 0.10084 [I 2022-10-27 17:34:44] ...optimizers/trijoint.py.151: total_norm: 10.050462 [I 2022-10-27 17:34:49] ...trap/engines/engine.py.224: train: epoch 79 | batch 1720/2304 [I 2022-10-27 17:34:49] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:25 [I 2022-10-27 17:34:49] ...trap/engines/engine.py.229: process: 0.55954 | load: 0.00253 [I 2022-10-27 17:34:49] ...trap/engines/engine.py.230: loss: 0.11925 [I 2022-10-27 17:34:49] ...optimizers/trijoint.py.151: total_norm: 48.090389 [I 2022-10-27 17:34:55] ...trap/engines/engine.py.224: train: epoch 79 | batch 1730/2304 [I 2022-10-27 17:34:55] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:22 [I 2022-10-27 17:34:55] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00286 [I 2022-10-27 17:34:55] ...trap/engines/engine.py.230: loss: 0.15445 [I 2022-10-27 17:34:55] ...optimizers/trijoint.py.151: total_norm: 10.187346 [I 2022-10-27 17:35:01] ...trap/engines/engine.py.224: train: epoch 79 | batch 1740/2304 [I 2022-10-27 17:35:01] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:05:15 [I 2022-10-27 17:35:01] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00271 [I 2022-10-27 17:35:01] ...trap/engines/engine.py.230: loss: 0.12684 [I 2022-10-27 17:35:01] ...optimizers/trijoint.py.151: total_norm: 9.902142 [I 2022-10-27 17:35:06] ...trap/engines/engine.py.224: train: epoch 79 | batch 1750/2304 [I 2022-10-27 17:35:06] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:10 [I 2022-10-27 17:35:06] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00275 [I 2022-10-27 17:35:06] ...trap/engines/engine.py.230: loss: 0.17859 [I 2022-10-27 17:35:06] ...optimizers/trijoint.py.151: total_norm: 18.689394 [I 2022-10-27 17:35:12] ...trap/engines/engine.py.224: train: epoch 79 | batch 1760/2304 [I 2022-10-27 17:35:12] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:05:03 [I 2022-10-27 17:35:12] ...trap/engines/engine.py.229: process: 0.55556 | load: 0.00265 [I 2022-10-27 17:35:12] ...trap/engines/engine.py.230: loss: 0.18814 [I 2022-10-27 17:35:12] ...optimizers/trijoint.py.151: total_norm: 11.514414 [I 2022-10-27 17:35:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 1770/2304 [I 2022-10-27 17:35:17] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:57 [I 2022-10-27 17:35:17] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00259 [I 2022-10-27 17:35:17] ...trap/engines/engine.py.230: loss: 0.11262 [I 2022-10-27 17:35:17] ...optimizers/trijoint.py.151: total_norm: 9.684772 [I 2022-10-27 17:35:23] ...trap/engines/engine.py.224: train: epoch 79 | batch 1780/2304 [I 2022-10-27 17:35:23] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:51 [I 2022-10-27 17:35:23] ...trap/engines/engine.py.229: process: 0.54931 | load: 0.00255 [I 2022-10-27 17:35:23] ...trap/engines/engine.py.230: loss: 0.19134 [I 2022-10-27 17:35:23] ...optimizers/trijoint.py.151: total_norm: 19.297525 [I 2022-10-27 17:35:29] ...trap/engines/engine.py.224: train: epoch 79 | batch 1790/2304 [I 2022-10-27 17:35:29] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:42 [I 2022-10-27 17:35:29] ...trap/engines/engine.py.229: process: 0.54544 | load: 0.00255 [I 2022-10-27 17:35:29] ...trap/engines/engine.py.230: loss: 0.09923 [I 2022-10-27 17:35:29] ...optimizers/trijoint.py.151: total_norm: 14.410404 [I 2022-10-27 17:35:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 1800/2304 [I 2022-10-27 17:35:34] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:40 [I 2022-10-27 17:35:34] ...trap/engines/engine.py.229: process: 0.56156 | load: 0.00254 [I 2022-10-27 17:35:34] ...trap/engines/engine.py.230: loss: 0.13151 [I 2022-10-27 17:35:34] ...optimizers/trijoint.py.151: total_norm: 19.849941 [I 2022-10-27 17:35:40] ...trap/engines/engine.py.224: train: epoch 79 | batch 1810/2304 [I 2022-10-27 17:35:40] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:34 [I 2022-10-27 17:35:40] ...trap/engines/engine.py.229: process: 0.55881 | load: 0.00248 [I 2022-10-27 17:35:40] ...trap/engines/engine.py.230: loss: 0.11894 [I 2022-10-27 17:35:40] ...optimizers/trijoint.py.151: total_norm: 16.311163 [I 2022-10-27 17:35:45] ...trap/engines/engine.py.224: train: epoch 79 | batch 1820/2304 [I 2022-10-27 17:35:45] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:28 [I 2022-10-27 17:35:45] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00248 [I 2022-10-27 17:35:45] ...trap/engines/engine.py.230: loss: 0.13412 [I 2022-10-27 17:35:45] ...optimizers/trijoint.py.151: total_norm: 18.559456 [I 2022-10-27 17:35:51] ...trap/engines/engine.py.224: train: epoch 79 | batch 1830/2304 [I 2022-10-27 17:35:51] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:24 [I 2022-10-27 17:35:51] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00274 [I 2022-10-27 17:35:51] ...trap/engines/engine.py.230: loss: 0.10474 [I 2022-10-27 17:35:51] ...optimizers/trijoint.py.151: total_norm: 11.112378 [I 2022-10-27 17:35:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 1840/2304 [I 2022-10-27 17:35:56] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:19 [I 2022-10-27 17:35:56] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00308 [I 2022-10-27 17:35:56] ...trap/engines/engine.py.230: loss: 0.12404 [I 2022-10-27 17:35:56] ...optimizers/trijoint.py.151: total_norm: 10.907107 [I 2022-10-27 17:36:02] ...trap/engines/engine.py.224: train: epoch 79 | batch 1850/2304 [I 2022-10-27 17:36:02] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:04:14 [I 2022-10-27 17:36:02] ...trap/engines/engine.py.229: process: 0.55713 | load: 0.00280 [I 2022-10-27 17:36:02] ...trap/engines/engine.py.230: loss: 0.13402 [I 2022-10-27 17:36:02] ...optimizers/trijoint.py.151: total_norm: 29.911938 [I 2022-10-27 17:36:08] ...trap/engines/engine.py.224: train: epoch 79 | batch 1860/2304 [I 2022-10-27 17:36:08] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:04:11 [I 2022-10-27 17:36:08] ...trap/engines/engine.py.229: process: 0.56868 | load: 0.00289 [I 2022-10-27 17:36:08] ...trap/engines/engine.py.230: loss: 0.17225 [I 2022-10-27 17:36:08] ...optimizers/trijoint.py.151: total_norm: 21.298164 [I 2022-10-27 17:36:13] ...trap/engines/engine.py.224: train: epoch 79 | batch 1870/2304 [I 2022-10-27 17:36:13] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:04:03 [I 2022-10-27 17:36:13] ...trap/engines/engine.py.229: process: 0.56633 | load: 0.00252 [I 2022-10-27 17:36:13] ...trap/engines/engine.py.230: loss: 0.13145 [I 2022-10-27 17:36:13] ...optimizers/trijoint.py.151: total_norm: 12.715988 [I 2022-10-27 17:36:19] ...trap/engines/engine.py.224: train: epoch 79 | batch 1880/2304 [I 2022-10-27 17:36:19] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:56 [I 2022-10-27 17:36:19] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00255 [I 2022-10-27 17:36:19] ...trap/engines/engine.py.230: loss: 0.15713 [I 2022-10-27 17:36:19] ...optimizers/trijoint.py.151: total_norm: 14.639658 [I 2022-10-27 17:36:25] ...trap/engines/engine.py.224: train: epoch 79 | batch 1890/2304 [I 2022-10-27 17:36:25] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:54 [I 2022-10-27 17:36:25] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00256 [I 2022-10-27 17:36:25] ...trap/engines/engine.py.230: loss: 0.10536 [I 2022-10-27 17:36:25] ...optimizers/trijoint.py.151: total_norm: 11.058966 [I 2022-10-27 17:36:30] ...trap/engines/engine.py.224: train: epoch 79 | batch 1900/2304 [I 2022-10-27 17:36:30] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:44 [I 2022-10-27 17:36:30] ...trap/engines/engine.py.229: process: 0.55824 | load: 0.00253 [I 2022-10-27 17:36:30] ...trap/engines/engine.py.230: loss: 0.11808 [I 2022-10-27 17:36:30] ...optimizers/trijoint.py.151: total_norm: 12.749681 [I 2022-10-27 17:36:36] ...trap/engines/engine.py.224: train: epoch 79 | batch 1910/2304 [I 2022-10-27 17:36:36] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:40 [I 2022-10-27 17:36:36] ...trap/engines/engine.py.229: process: 0.53981 | load: 0.00255 [I 2022-10-27 17:36:36] ...trap/engines/engine.py.230: loss: 0.12752 [I 2022-10-27 17:36:36] ...optimizers/trijoint.py.151: total_norm: 23.869911 [I 2022-10-27 17:36:42] ...trap/engines/engine.py.224: train: epoch 79 | batch 1920/2304 [I 2022-10-27 17:36:42] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:38 [I 2022-10-27 17:36:42] ...trap/engines/engine.py.229: process: 0.56760 | load: 0.00272 [I 2022-10-27 17:36:42] ...trap/engines/engine.py.230: loss: 0.14635 [I 2022-10-27 17:36:42] ...optimizers/trijoint.py.151: total_norm: 14.579551 [I 2022-10-27 17:36:47] ...trap/engines/engine.py.224: train: epoch 79 | batch 1930/2304 [I 2022-10-27 17:36:47] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:30 [I 2022-10-27 17:36:47] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00248 [I 2022-10-27 17:36:47] ...trap/engines/engine.py.230: loss: 0.11705 [I 2022-10-27 17:36:47] ...optimizers/trijoint.py.151: total_norm: 14.035491 [I 2022-10-27 17:36:53] ...trap/engines/engine.py.224: train: epoch 79 | batch 1940/2304 [I 2022-10-27 17:36:53] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:21 [I 2022-10-27 17:36:53] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00250 [I 2022-10-27 17:36:53] ...trap/engines/engine.py.230: loss: 0.14637 [I 2022-10-27 17:36:53] ...optimizers/trijoint.py.151: total_norm: 11.726543 [I 2022-10-27 17:36:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 1950/2304 [I 2022-10-27 17:36:58] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:03:19 [I 2022-10-27 17:36:58] ...trap/engines/engine.py.229: process: 0.55265 | load: 0.00254 [I 2022-10-27 17:36:58] ...trap/engines/engine.py.230: loss: 0.12333 [I 2022-10-27 17:36:58] ...optimizers/trijoint.py.151: total_norm: 10.085869 [I 2022-10-27 17:37:04] ...trap/engines/engine.py.224: train: epoch 79 | batch 1960/2304 [I 2022-10-27 17:37:04] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:03:13 [I 2022-10-27 17:37:04] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00247 [I 2022-10-27 17:37:04] ...trap/engines/engine.py.230: loss: 0.09851 [I 2022-10-27 17:37:04] ...optimizers/trijoint.py.151: total_norm: 9.051982 [I 2022-10-27 17:37:10] ...trap/engines/engine.py.224: train: epoch 79 | batch 1970/2304 [I 2022-10-27 17:37:10] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:03:05 [I 2022-10-27 17:37:10] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00255 [I 2022-10-27 17:37:10] ...trap/engines/engine.py.230: loss: 0.09937 [I 2022-10-27 17:37:10] ...optimizers/trijoint.py.151: total_norm: 15.007243 [I 2022-10-27 17:37:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 1980/2304 [I 2022-10-27 17:37:15] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:03:02 [I 2022-10-27 17:37:15] ...trap/engines/engine.py.229: process: 0.56664 | load: 0.00247 [I 2022-10-27 17:37:15] ...trap/engines/engine.py.230: loss: 0.08513 [I 2022-10-27 17:37:15] ...optimizers/trijoint.py.151: total_norm: 9.684646 [I 2022-10-27 17:37:21] ...trap/engines/engine.py.224: train: epoch 79 | batch 1990/2304 [I 2022-10-27 17:37:21] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:53 [I 2022-10-27 17:37:21] ...trap/engines/engine.py.229: process: 0.53760 | load: 0.00243 [I 2022-10-27 17:37:21] ...trap/engines/engine.py.230: loss: 0.11027 [I 2022-10-27 17:37:21] ...optimizers/trijoint.py.151: total_norm: 9.988313 [I 2022-10-27 17:37:26] ...trap/engines/engine.py.224: train: epoch 79 | batch 2000/2304 [I 2022-10-27 17:37:26] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:47 [I 2022-10-27 17:37:26] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00261 [I 2022-10-27 17:37:26] ...trap/engines/engine.py.230: loss: 0.24013 [I 2022-10-27 17:37:26] ...optimizers/trijoint.py.151: total_norm: 39.197140 [I 2022-10-27 17:37:32] ...trap/engines/engine.py.224: train: epoch 79 | batch 2010/2304 [I 2022-10-27 17:37:32] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:43 [I 2022-10-27 17:37:32] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00249 [I 2022-10-27 17:37:32] ...trap/engines/engine.py.230: loss: 0.19731 [I 2022-10-27 17:37:32] ...optimizers/trijoint.py.151: total_norm: 30.306211 [I 2022-10-27 17:37:37] ...trap/engines/engine.py.224: train: epoch 79 | batch 2020/2304 [I 2022-10-27 17:37:37] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:37 [I 2022-10-27 17:37:37] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00249 [I 2022-10-27 17:37:37] ...trap/engines/engine.py.230: loss: 0.11726 [I 2022-10-27 17:37:37] ...optimizers/trijoint.py.151: total_norm: 18.280384 [I 2022-10-27 17:37:43] ...trap/engines/engine.py.224: train: epoch 79 | batch 2030/2304 [I 2022-10-27 17:37:43] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:33 [I 2022-10-27 17:37:43] ...trap/engines/engine.py.229: process: 0.56581 | load: 0.00305 [I 2022-10-27 17:37:43] ...trap/engines/engine.py.230: loss: 0.21042 [I 2022-10-27 17:37:43] ...optimizers/trijoint.py.151: total_norm: 21.702040 [I 2022-10-27 17:37:49] ...trap/engines/engine.py.224: train: epoch 79 | batch 2040/2304 [I 2022-10-27 17:37:49] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:29 [I 2022-10-27 17:37:49] ...trap/engines/engine.py.229: process: 0.59966 | load: 0.00254 [I 2022-10-27 17:37:49] ...trap/engines/engine.py.230: loss: 0.12729 [I 2022-10-27 17:37:49] ...optimizers/trijoint.py.151: total_norm: 13.607539 [I 2022-10-27 17:37:54] ...trap/engines/engine.py.224: train: epoch 79 | batch 2050/2304 [I 2022-10-27 17:37:54] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:02:22 [I 2022-10-27 17:37:54] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00253 [I 2022-10-27 17:37:54] ...trap/engines/engine.py.230: loss: 0.12431 [I 2022-10-27 17:37:54] ...optimizers/trijoint.py.151: total_norm: 17.904434 [I 2022-10-27 17:38:00] ...trap/engines/engine.py.224: train: epoch 79 | batch 2060/2304 [I 2022-10-27 17:38:00] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:02:16 [I 2022-10-27 17:38:00] ...trap/engines/engine.py.229: process: 0.55008 | load: 0.00254 [I 2022-10-27 17:38:00] ...trap/engines/engine.py.230: loss: 0.16058 [I 2022-10-27 17:38:00] ...optimizers/trijoint.py.151: total_norm: 14.723633 [I 2022-10-27 17:38:06] ...trap/engines/engine.py.224: train: epoch 79 | batch 2070/2304 [I 2022-10-27 17:38:06] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:02:12 [I 2022-10-27 17:38:06] ...trap/engines/engine.py.229: process: 0.55381 | load: 0.00250 [I 2022-10-27 17:38:06] ...trap/engines/engine.py.230: loss: 0.23617 [I 2022-10-27 17:38:06] ...optimizers/trijoint.py.151: total_norm: 24.172438 [I 2022-10-27 17:38:11] ...trap/engines/engine.py.224: train: epoch 79 | batch 2080/2304 [I 2022-10-27 17:38:11] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:02:07 [I 2022-10-27 17:38:11] ...trap/engines/engine.py.229: process: 0.57756 | load: 0.00257 [I 2022-10-27 17:38:11] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-27 17:38:11] ...optimizers/trijoint.py.151: total_norm: 20.230440 [I 2022-10-27 17:38:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 2090/2304 [I 2022-10-27 17:38:17] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:02:00 [I 2022-10-27 17:38:17] ...trap/engines/engine.py.229: process: 0.55047 | load: 0.00254 [I 2022-10-27 17:38:17] ...trap/engines/engine.py.230: loss: 0.11154 [I 2022-10-27 17:38:17] ...optimizers/trijoint.py.151: total_norm: 12.552046 [I 2022-10-27 17:38:23] ...trap/engines/engine.py.224: train: epoch 79 | batch 2100/2304 [I 2022-10-27 17:38:23] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:54 [I 2022-10-27 17:38:23] ...trap/engines/engine.py.229: process: 0.56760 | load: 0.00253 [I 2022-10-27 17:38:23] ...trap/engines/engine.py.230: loss: 0.11134 [I 2022-10-27 17:38:23] ...optimizers/trijoint.py.151: total_norm: 24.469616 [I 2022-10-27 17:38:28] ...trap/engines/engine.py.224: train: epoch 79 | batch 2110/2304 [I 2022-10-27 17:38:28] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:48 [I 2022-10-27 17:38:28] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00250 [I 2022-10-27 17:38:28] ...trap/engines/engine.py.230: loss: 0.12734 [I 2022-10-27 17:38:28] ...optimizers/trijoint.py.151: total_norm: 22.215805 [I 2022-10-27 17:38:34] ...trap/engines/engine.py.224: train: epoch 79 | batch 2120/2304 [I 2022-10-27 17:38:34] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:42 [I 2022-10-27 17:38:34] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00248 [I 2022-10-27 17:38:34] ...trap/engines/engine.py.230: loss: 0.12771 [I 2022-10-27 17:38:34] ...optimizers/trijoint.py.151: total_norm: 14.583114 [I 2022-10-27 17:38:39] ...trap/engines/engine.py.224: train: epoch 79 | batch 2130/2304 [I 2022-10-27 17:38:39] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:36 [I 2022-10-27 17:38:39] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00250 [I 2022-10-27 17:38:39] ...trap/engines/engine.py.230: loss: 0.16771 [I 2022-10-27 17:38:39] ...optimizers/trijoint.py.151: total_norm: 7.792945 [I 2022-10-27 17:38:45] ...trap/engines/engine.py.224: train: epoch 79 | batch 2140/2304 [I 2022-10-27 17:38:45] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:31 [I 2022-10-27 17:38:45] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00241 [I 2022-10-27 17:38:45] ...trap/engines/engine.py.230: loss: 0.13069 [I 2022-10-27 17:38:45] ...optimizers/trijoint.py.151: total_norm: 7.867976 [I 2022-10-27 17:38:51] ...trap/engines/engine.py.224: train: epoch 79 | batch 2150/2304 [I 2022-10-27 17:38:51] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:25 [I 2022-10-27 17:38:51] ...trap/engines/engine.py.229: process: 0.56138 | load: 0.00246 [I 2022-10-27 17:38:51] ...trap/engines/engine.py.230: loss: 0.14497 [I 2022-10-27 17:38:51] ...optimizers/trijoint.py.151: total_norm: 11.567678 [I 2022-10-27 17:38:56] ...trap/engines/engine.py.224: train: epoch 79 | batch 2160/2304 [I 2022-10-27 17:38:56] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:01:22 [I 2022-10-27 17:38:56] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00246 [I 2022-10-27 17:38:56] ...trap/engines/engine.py.230: loss: 0.11993 [I 2022-10-27 17:38:56] ...optimizers/trijoint.py.151: total_norm: 17.175417 [I 2022-10-27 17:39:02] ...trap/engines/engine.py.224: train: epoch 79 | batch 2170/2304 [I 2022-10-27 17:39:02] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:01:15 [I 2022-10-27 17:39:02] ...trap/engines/engine.py.229: process: 0.54150 | load: 0.00251 [I 2022-10-27 17:39:02] ...trap/engines/engine.py.230: loss: 0.11884 [I 2022-10-27 17:39:02] ...optimizers/trijoint.py.151: total_norm: 16.453691 [I 2022-10-27 17:39:08] ...trap/engines/engine.py.224: train: epoch 79 | batch 2180/2304 [I 2022-10-27 17:39:08] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:01:09 [I 2022-10-27 17:39:08] ...trap/engines/engine.py.229: process: 0.58014 | load: 0.00251 [I 2022-10-27 17:39:08] ...trap/engines/engine.py.230: loss: 0.11182 [I 2022-10-27 17:39:08] ...optimizers/trijoint.py.151: total_norm: 20.320278 [I 2022-10-27 17:39:13] ...trap/engines/engine.py.224: train: epoch 79 | batch 2190/2304 [I 2022-10-27 17:39:13] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:01:04 [I 2022-10-27 17:39:13] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00254 [I 2022-10-27 17:39:13] ...trap/engines/engine.py.230: loss: 0.09875 [I 2022-10-27 17:39:13] ...optimizers/trijoint.py.151: total_norm: 16.193039 [I 2022-10-27 17:39:19] ...trap/engines/engine.py.224: train: epoch 79 | batch 2200/2304 [I 2022-10-27 17:39:19] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:58 [I 2022-10-27 17:39:19] ...trap/engines/engine.py.229: process: 0.56912 | load: 0.00245 [I 2022-10-27 17:39:19] ...trap/engines/engine.py.230: loss: 0.12285 [I 2022-10-27 17:39:19] ...optimizers/trijoint.py.151: total_norm: 13.103382 [I 2022-10-27 17:39:25] ...trap/engines/engine.py.224: train: epoch 79 | batch 2210/2304 [I 2022-10-27 17:39:25] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:52 [I 2022-10-27 17:39:25] ...trap/engines/engine.py.229: process: 0.56352 | load: 0.00255 [I 2022-10-27 17:39:25] ...trap/engines/engine.py.230: loss: 0.12095 [I 2022-10-27 17:39:25] ...optimizers/trijoint.py.151: total_norm: 13.112513 [I 2022-10-27 17:39:30] ...trap/engines/engine.py.224: train: epoch 79 | batch 2220/2304 [I 2022-10-27 17:39:30] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:46 [I 2022-10-27 17:39:30] ...trap/engines/engine.py.229: process: 0.56153 | load: 0.00266 [I 2022-10-27 17:39:30] ...trap/engines/engine.py.230: loss: 0.16843 [I 2022-10-27 17:39:30] ...optimizers/trijoint.py.151: total_norm: 14.942154 [I 2022-10-27 17:39:36] ...trap/engines/engine.py.224: train: epoch 79 | batch 2230/2304 [I 2022-10-27 17:39:36] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:41 [I 2022-10-27 17:39:36] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00251 [I 2022-10-27 17:39:36] ...trap/engines/engine.py.230: loss: 0.14067 [I 2022-10-27 17:39:36] ...optimizers/trijoint.py.151: total_norm: 19.338017 [I 2022-10-27 17:39:41] ...trap/engines/engine.py.224: train: epoch 79 | batch 2240/2304 [I 2022-10-27 17:39:41] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:35 [I 2022-10-27 17:39:41] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00246 [I 2022-10-27 17:39:41] ...trap/engines/engine.py.230: loss: 0.12131 [I 2022-10-27 17:39:41] ...optimizers/trijoint.py.151: total_norm: 15.044205 [I 2022-10-27 17:39:47] ...trap/engines/engine.py.224: train: epoch 79 | batch 2250/2304 [I 2022-10-27 17:39:47] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:30 [I 2022-10-27 17:39:47] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00247 [I 2022-10-27 17:39:47] ...trap/engines/engine.py.230: loss: 0.09125 [I 2022-10-27 17:39:47] ...optimizers/trijoint.py.151: total_norm: 17.638109 [I 2022-10-27 17:39:52] ...trap/engines/engine.py.224: train: epoch 79 | batch 2260/2304 [I 2022-10-27 17:39:52] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:24 [I 2022-10-27 17:39:52] ...trap/engines/engine.py.229: process: 0.56727 | load: 0.00259 [I 2022-10-27 17:39:52] ...trap/engines/engine.py.230: loss: 0.17000 [I 2022-10-27 17:39:52] ...optimizers/trijoint.py.151: total_norm: 13.596626 [I 2022-10-27 17:39:58] ...trap/engines/engine.py.224: train: epoch 79 | batch 2270/2304 [I 2022-10-27 17:39:58] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:19 [I 2022-10-27 17:39:58] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00251 [I 2022-10-27 17:39:58] ...trap/engines/engine.py.230: loss: 0.13399 [I 2022-10-27 17:39:58] ...optimizers/trijoint.py.151: total_norm: 13.153386 [I 2022-10-27 17:40:04] ...trap/engines/engine.py.224: train: epoch 79 | batch 2280/2304 [I 2022-10-27 17:40:04] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:13 [I 2022-10-27 17:40:04] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00278 [I 2022-10-27 17:40:04] ...trap/engines/engine.py.230: loss: 0.13137 [I 2022-10-27 17:40:04] ...optimizers/trijoint.py.151: total_norm: 18.533010 [I 2022-10-27 17:40:09] ...trap/engines/engine.py.224: train: epoch 79 | batch 2290/2304 [I 2022-10-27 17:40:09] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:07 [I 2022-10-27 17:40:09] ...trap/engines/engine.py.229: process: 0.56719 | load: 0.00252 [I 2022-10-27 17:40:09] ...trap/engines/engine.py.230: loss: 0.12228 [I 2022-10-27 17:40:09] ...optimizers/trijoint.py.151: total_norm: 11.780579 [I 2022-10-27 17:40:15] ...trap/engines/engine.py.224: train: epoch 79 | batch 2300/2304 [I 2022-10-27 17:40:15] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:02 [I 2022-10-27 17:40:15] ...trap/engines/engine.py.229: process: 0.54525 | load: 0.00015 [I 2022-10-27 17:40:15] ...trap/engines/engine.py.230: loss: 0.12744 [I 2022-10-27 17:40:15] ...optimizers/trijoint.py.151: total_norm: 12.281067 [I 2022-10-27 17:40:17] ...trap/engines/engine.py.224: train: epoch 79 | batch 2304/2304 [I 2022-10-27 17:40:17] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:00:00 [I 2022-10-27 17:40:17] ...trap/engines/engine.py.229: process: 0.55119 | load: 0.00016 [I 2022-10-27 17:40:17] ...trap/engines/engine.py.230: loss: 0.13072 [I 2022-10-27 17:40:17] ...optimizers/trijoint.py.151: total_norm: 19.481386 [S 2022-10-27 17:40:18] ...trap/engines/engine.py.129: train_epoch.epoch: 79 [S 2022-10-27 17:40:18] ...trap/engines/engine.py.129: train_epoch.loss: 0.13736144056677044 [I 2022-10-27 17:40:18] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 17:40:20] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 79 [I 2022-10-27 17:40:20] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 17:40:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 17:40:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 17:40:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 17:40:24] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 17:40:25] ...trap/engines/engine.py.321: eval: epoch 79 | batch 0/510 [I 2022-10-27 17:40:25] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:42:25 [I 2022-10-27 17:40:25] ...trap/engines/engine.py.326: process: 4.99177 | load: 3.58406 [I 2022-10-27 17:40:28] ...trap/engines/engine.py.321: eval: epoch 79 | batch 10/510 [I 2022-10-27 17:40:28] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:57 [I 2022-10-27 17:40:28] ...trap/engines/engine.py.326: process: 0.34527 | load: 0.00018 [I 2022-10-27 17:40:31] ...trap/engines/engine.py.321: eval: epoch 79 | batch 20/510 [I 2022-10-27 17:40:31] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:08 [I 2022-10-27 17:40:31] ...trap/engines/engine.py.326: process: 0.34211 | load: 0.00018 [I 2022-10-27 17:40:34] ...trap/engines/engine.py.321: eval: epoch 79 | batch 30/510 [I 2022-10-27 17:40:34] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:41 [I 2022-10-27 17:40:34] ...trap/engines/engine.py.326: process: 0.33016 | load: 0.00018 [I 2022-10-27 17:40:38] ...trap/engines/engine.py.321: eval: epoch 79 | batch 40/510 [I 2022-10-27 17:40:38] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:38 [I 2022-10-27 17:40:38] ...trap/engines/engine.py.326: process: 0.33110 | load: 0.00020 [I 2022-10-27 17:40:41] ...trap/engines/engine.py.321: eval: epoch 79 | batch 50/510 [I 2022-10-27 17:40:41] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:32 [I 2022-10-27 17:40:41] ...trap/engines/engine.py.326: process: 0.32594 | load: 0.00032 [I 2022-10-27 17:40:45] ...trap/engines/engine.py.321: eval: epoch 79 | batch 60/510 [I 2022-10-27 17:40:45] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:30 [I 2022-10-27 17:40:45] ...trap/engines/engine.py.326: process: 0.32599 | load: 0.00016 [I 2022-10-27 17:40:48] ...trap/engines/engine.py.321: eval: epoch 79 | batch 70/510 [I 2022-10-27 17:40:48] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:30 [I 2022-10-27 17:40:48] ...trap/engines/engine.py.326: process: 0.35941 | load: 0.00018 [I 2022-10-27 17:40:51] ...trap/engines/engine.py.321: eval: epoch 79 | batch 80/510 [I 2022-10-27 17:40:51] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:26 [I 2022-10-27 17:40:51] ...trap/engines/engine.py.326: process: 0.32922 | load: 0.00018 [I 2022-10-27 17:40:55] ...trap/engines/engine.py.321: eval: epoch 79 | batch 90/510 [I 2022-10-27 17:40:55] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-27 17:40:55] ...trap/engines/engine.py.326: process: 0.32903 | load: 0.00017 [I 2022-10-27 17:40:58] ...trap/engines/engine.py.321: eval: epoch 79 | batch 100/510 [I 2022-10-27 17:40:58] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-27 17:40:58] ...trap/engines/engine.py.326: process: 0.27999 | load: 0.00016 [I 2022-10-27 17:41:01] ...trap/engines/engine.py.321: eval: epoch 79 | batch 110/510 [I 2022-10-27 17:41:01] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-27 17:41:01] ...trap/engines/engine.py.326: process: 0.28732 | load: 0.00014 [I 2022-10-27 17:41:04] ...trap/engines/engine.py.321: eval: epoch 79 | batch 120/510 [I 2022-10-27 17:41:04] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 17:41:04] ...trap/engines/engine.py.326: process: 0.29070 | load: 0.00015 [I 2022-10-27 17:41:07] ...trap/engines/engine.py.321: eval: epoch 79 | batch 130/510 [I 2022-10-27 17:41:07] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:48 [I 2022-10-27 17:41:07] ...trap/engines/engine.py.326: process: 0.28133 | load: 0.00015 [I 2022-10-27 17:41:10] ...trap/engines/engine.py.321: eval: epoch 79 | batch 140/510 [I 2022-10-27 17:41:10] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:46 [I 2022-10-27 17:41:10] ...trap/engines/engine.py.326: process: 0.28339 | load: 0.00023 [I 2022-10-27 17:41:12] ...trap/engines/engine.py.321: eval: epoch 79 | batch 150/510 [I 2022-10-27 17:41:12] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 17:41:12] ...trap/engines/engine.py.326: process: 0.28546 | load: 0.00015 [I 2022-10-27 17:41:15] ...trap/engines/engine.py.321: eval: epoch 79 | batch 160/510 [I 2022-10-27 17:41:15] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 17:41:15] ...trap/engines/engine.py.326: process: 0.28208 | load: 0.00015 [I 2022-10-27 17:41:18] ...trap/engines/engine.py.321: eval: epoch 79 | batch 170/510 [I 2022-10-27 17:41:18] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:39 [I 2022-10-27 17:41:18] ...trap/engines/engine.py.326: process: 0.29822 | load: 0.00014 [I 2022-10-27 17:41:21] ...trap/engines/engine.py.321: eval: epoch 79 | batch 180/510 [I 2022-10-27 17:41:21] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 17:41:21] ...trap/engines/engine.py.326: process: 0.29911 | load: 0.00014 [I 2022-10-27 17:41:24] ...trap/engines/engine.py.321: eval: epoch 79 | batch 190/510 [I 2022-10-27 17:41:24] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:33 [I 2022-10-27 17:41:24] ...trap/engines/engine.py.326: process: 0.30728 | load: 0.00015 [I 2022-10-27 17:41:27] ...trap/engines/engine.py.321: eval: epoch 79 | batch 200/510 [I 2022-10-27 17:41:27] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 17:41:27] ...trap/engines/engine.py.326: process: 0.29145 | load: 0.00016 [I 2022-10-27 17:41:30] ...trap/engines/engine.py.321: eval: epoch 79 | batch 210/510 [I 2022-10-27 17:41:30] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 17:41:30] ...trap/engines/engine.py.326: process: 0.29017 | load: 0.00016 [I 2022-10-27 17:41:33] ...trap/engines/engine.py.321: eval: epoch 79 | batch 220/510 [I 2022-10-27 17:41:33] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 17:41:33] ...trap/engines/engine.py.326: process: 0.28787 | load: 0.00015 [I 2022-10-27 17:41:36] ...trap/engines/engine.py.321: eval: epoch 79 | batch 230/510 [I 2022-10-27 17:41:36] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 17:41:36] ...trap/engines/engine.py.326: process: 0.28572 | load: 0.00015 [I 2022-10-27 17:41:39] ...trap/engines/engine.py.321: eval: epoch 79 | batch 240/510 [I 2022-10-27 17:41:39] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 17:41:39] ...trap/engines/engine.py.326: process: 0.28797 | load: 0.00015 [I 2022-10-27 17:41:41] ...trap/engines/engine.py.321: eval: epoch 79 | batch 250/510 [I 2022-10-27 17:41:41] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 17:41:41] ...trap/engines/engine.py.326: process: 0.28930 | load: 0.00018 [I 2022-10-27 17:41:44] ...trap/engines/engine.py.321: eval: epoch 79 | batch 260/510 [I 2022-10-27 17:41:44] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:11 [I 2022-10-27 17:41:44] ...trap/engines/engine.py.326: process: 0.28195 | load: 0.00015 [I 2022-10-27 17:41:47] ...trap/engines/engine.py.321: eval: epoch 79 | batch 270/510 [I 2022-10-27 17:41:47] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:08 [I 2022-10-27 17:41:47] ...trap/engines/engine.py.326: process: 0.28659 | load: 0.00016 [I 2022-10-27 17:41:50] ...trap/engines/engine.py.321: eval: epoch 79 | batch 280/510 [I 2022-10-27 17:41:50] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:05 [I 2022-10-27 17:41:50] ...trap/engines/engine.py.326: process: 0.28727 | load: 0.00015 [I 2022-10-27 17:41:53] ...trap/engines/engine.py.321: eval: epoch 79 | batch 290/510 [I 2022-10-27 17:41:53] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 17:41:53] ...trap/engines/engine.py.326: process: 0.28659 | load: 0.00015 [I 2022-10-27 17:41:56] ...trap/engines/engine.py.321: eval: epoch 79 | batch 300/510 [I 2022-10-27 17:41:56] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 17:41:56] ...trap/engines/engine.py.326: process: 0.28302 | load: 0.00018 [I 2022-10-27 17:41:59] ...trap/engines/engine.py.321: eval: epoch 79 | batch 310/510 [I 2022-10-27 17:41:59] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 17:41:59] ...trap/engines/engine.py.326: process: 0.28550 | load: 0.00019 [I 2022-10-27 17:42:01] ...trap/engines/engine.py.321: eval: epoch 79 | batch 320/510 [I 2022-10-27 17:42:01] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 17:42:01] ...trap/engines/engine.py.326: process: 0.28719 | load: 0.00016 [I 2022-10-27 17:42:04] ...trap/engines/engine.py.321: eval: epoch 79 | batch 330/510 [I 2022-10-27 17:42:04] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 17:42:04] ...trap/engines/engine.py.326: process: 0.28522 | load: 0.00015 [I 2022-10-27 17:42:07] ...trap/engines/engine.py.321: eval: epoch 79 | batch 340/510 [I 2022-10-27 17:42:07] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-27 17:42:07] ...trap/engines/engine.py.326: process: 0.28312 | load: 0.00015 [I 2022-10-27 17:42:10] ...trap/engines/engine.py.321: eval: epoch 79 | batch 350/510 [I 2022-10-27 17:42:10] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:45 [I 2022-10-27 17:42:10] ...trap/engines/engine.py.326: process: 0.28600 | load: 0.00019 [I 2022-10-27 17:42:13] ...trap/engines/engine.py.321: eval: epoch 79 | batch 360/510 [I 2022-10-27 17:42:13] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:42 [I 2022-10-27 17:42:13] ...trap/engines/engine.py.326: process: 0.28785 | load: 0.00015 [I 2022-10-27 17:42:16] ...trap/engines/engine.py.321: eval: epoch 79 | batch 370/510 [I 2022-10-27 17:42:16] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 17:42:16] ...trap/engines/engine.py.326: process: 0.28827 | load: 0.00015 [I 2022-10-27 17:42:19] ...trap/engines/engine.py.321: eval: epoch 79 | batch 380/510 [I 2022-10-27 17:42:19] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 17:42:19] ...trap/engines/engine.py.326: process: 0.29152 | load: 0.00018 [I 2022-10-27 17:42:22] ...trap/engines/engine.py.321: eval: epoch 79 | batch 390/510 [I 2022-10-27 17:42:22] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 17:42:22] ...trap/engines/engine.py.326: process: 0.28731 | load: 0.00018 [I 2022-10-27 17:42:24] ...trap/engines/engine.py.321: eval: epoch 79 | batch 400/510 [I 2022-10-27 17:42:24] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 17:42:24] ...trap/engines/engine.py.326: process: 0.28860 | load: 0.00034 [I 2022-10-27 17:42:27] ...trap/engines/engine.py.321: eval: epoch 79 | batch 410/510 [I 2022-10-27 17:42:27] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 17:42:27] ...trap/engines/engine.py.326: process: 0.28893 | load: 0.00021 [I 2022-10-27 17:42:30] ...trap/engines/engine.py.321: eval: epoch 79 | batch 420/510 [I 2022-10-27 17:42:30] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 17:42:30] ...trap/engines/engine.py.326: process: 0.28535 | load: 0.00018 [I 2022-10-27 17:42:33] ...trap/engines/engine.py.321: eval: epoch 79 | batch 430/510 [I 2022-10-27 17:42:33] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:22 [I 2022-10-27 17:42:33] ...trap/engines/engine.py.326: process: 0.28899 | load: 0.00024 [I 2022-10-27 17:42:36] ...trap/engines/engine.py.321: eval: epoch 79 | batch 440/510 [I 2022-10-27 17:42:36] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 17:42:36] ...trap/engines/engine.py.326: process: 0.28289 | load: 0.00018 [I 2022-10-27 17:42:39] ...trap/engines/engine.py.321: eval: epoch 79 | batch 450/510 [I 2022-10-27 17:42:39] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 17:42:39] ...trap/engines/engine.py.326: process: 0.28273 | load: 0.00019 [I 2022-10-27 17:42:42] ...trap/engines/engine.py.321: eval: epoch 79 | batch 460/510 [I 2022-10-27 17:42:42] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 17:42:42] ...trap/engines/engine.py.326: process: 0.28800 | load: 0.00021 [I 2022-10-27 17:42:45] ...trap/engines/engine.py.321: eval: epoch 79 | batch 470/510 [I 2022-10-27 17:42:45] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 17:42:45] ...trap/engines/engine.py.326: process: 0.28804 | load: 0.00020 [I 2022-10-27 17:42:48] ...trap/engines/engine.py.321: eval: epoch 79 | batch 480/510 [I 2022-10-27 17:42:48] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 17:42:48] ...trap/engines/engine.py.326: process: 0.29409 | load: 0.00025 [I 2022-10-27 17:42:50] ...trap/engines/engine.py.321: eval: epoch 79 | batch 490/510 [I 2022-10-27 17:42:50] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 17:42:50] ...trap/engines/engine.py.326: process: 0.28552 | load: 0.00014 [I 2022-10-27 17:42:53] ...trap/engines/engine.py.321: eval: epoch 79 | batch 500/510 [I 2022-10-27 17:42:53] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 17:42:53] ...trap/engines/engine.py.326: process: 0.28616 | load: 0.00018 [I 2022-10-27 17:42:56] ...trap/engines/engine.py.321: eval: epoch 79 | batch 510/510 [I 2022-10-27 17:42:56] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 17:42:56] ...trap/engines/engine.py.326: process: 0.28824 | load: 0.00015 [S 2022-10-27 17:42:57] ...trap/engines/engine.py.132: eval_epoch.epoch: 79 [I 2022-10-27 17:42:57] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.720800 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.903300 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933300 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.716500 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.901500 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931700 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016690 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008776 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007772 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015914 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009091 [S 2022-10-27 17:43:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007537 [I 2022-10-27 17:43:03] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 17:43:03] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 17:43:03] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 17:43:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 17:43:03] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 17:43:07] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 17:43:08] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 17:43:14] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 17:43:14] ...trap/engines/engine.py.163: Training model on trainset for epoch 80 [I 2022-10-27 17:43:14] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 17:43:17] ...trap/engines/engine.py.224: train: epoch 80 | batch 0/2304 [I 2022-10-27 17:43:17] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:26 [I 2022-10-27 17:43:17] ...trap/engines/engine.py.229: process: 2.71984 | load: 2.06005 [I 2022-10-27 17:43:17] ...trap/engines/engine.py.230: loss: 0.13524 [I 2022-10-27 17:43:17] ...optimizers/trijoint.py.151: total_norm: 14.060567 [I 2022-10-27 17:43:23] ...trap/engines/engine.py.224: train: epoch 80 | batch 10/2304 [I 2022-10-27 17:43:23] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:19 [I 2022-10-27 17:43:23] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00404 [I 2022-10-27 17:43:23] ...trap/engines/engine.py.230: loss: 0.13928 [I 2022-10-27 17:43:23] ...optimizers/trijoint.py.151: total_norm: 15.674787 [I 2022-10-27 17:43:28] ...trap/engines/engine.py.224: train: epoch 80 | batch 20/2304 [I 2022-10-27 17:43:28] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:07 [I 2022-10-27 17:43:28] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00371 [I 2022-10-27 17:43:28] ...trap/engines/engine.py.230: loss: 0.10779 [I 2022-10-27 17:43:28] ...optimizers/trijoint.py.151: total_norm: 10.410252 [I 2022-10-27 17:43:34] ...trap/engines/engine.py.224: train: epoch 80 | batch 30/2304 [I 2022-10-27 17:43:34] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-27 17:43:34] ...trap/engines/engine.py.229: process: 0.56600 | load: 0.00287 [I 2022-10-27 17:43:34] ...trap/engines/engine.py.230: loss: 0.14659 [I 2022-10-27 17:43:34] ...optimizers/trijoint.py.151: total_norm: 18.263016 [I 2022-10-27 17:43:40] ...trap/engines/engine.py.224: train: epoch 80 | batch 40/2304 [I 2022-10-27 17:43:40] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:06 [I 2022-10-27 17:43:40] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00279 [I 2022-10-27 17:43:40] ...trap/engines/engine.py.230: loss: 0.11793 [I 2022-10-27 17:43:40] ...optimizers/trijoint.py.151: total_norm: 15.763934 [I 2022-10-27 17:43:45] ...trap/engines/engine.py.224: train: epoch 80 | batch 50/2304 [I 2022-10-27 17:43:45] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:59 [I 2022-10-27 17:43:45] ...trap/engines/engine.py.229: process: 0.56541 | load: 0.00323 [I 2022-10-27 17:43:45] ...trap/engines/engine.py.230: loss: 0.20289 [I 2022-10-27 17:43:45] ...optimizers/trijoint.py.151: total_norm: 24.857298 [I 2022-10-27 17:43:51] ...trap/engines/engine.py.224: train: epoch 80 | batch 60/2304 [I 2022-10-27 17:43:51] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:49 [I 2022-10-27 17:43:51] ...trap/engines/engine.py.229: process: 0.55552 | load: 0.00288 [I 2022-10-27 17:43:51] ...trap/engines/engine.py.230: loss: 0.09182 [I 2022-10-27 17:43:51] ...optimizers/trijoint.py.151: total_norm: 16.991720 [I 2022-10-27 17:43:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 70/2304 [I 2022-10-27 17:43:56] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:48 [I 2022-10-27 17:43:56] ...trap/engines/engine.py.229: process: 0.55658 | load: 0.00285 [I 2022-10-27 17:43:56] ...trap/engines/engine.py.230: loss: 0.10112 [I 2022-10-27 17:43:56] ...optimizers/trijoint.py.151: total_norm: 14.627878 [I 2022-10-27 17:44:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 80/2304 [I 2022-10-27 17:44:02] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:36 [I 2022-10-27 17:44:02] ...trap/engines/engine.py.229: process: 0.55359 | load: 0.00312 [I 2022-10-27 17:44:02] ...trap/engines/engine.py.230: loss: 0.12448 [I 2022-10-27 17:44:02] ...optimizers/trijoint.py.151: total_norm: 22.505747 [I 2022-10-27 17:44:08] ...trap/engines/engine.py.224: train: epoch 80 | batch 90/2304 [I 2022-10-27 17:44:08] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:43 [I 2022-10-27 17:44:08] ...trap/engines/engine.py.229: process: 0.57934 | load: 0.00361 [I 2022-10-27 17:44:08] ...trap/engines/engine.py.230: loss: 0.22908 [I 2022-10-27 17:44:08] ...optimizers/trijoint.py.151: total_norm: 20.512960 [I 2022-10-27 17:44:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 100/2304 [I 2022-10-27 17:44:13] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:36 [I 2022-10-27 17:44:13] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00275 [I 2022-10-27 17:44:13] ...trap/engines/engine.py.230: loss: 0.10374 [I 2022-10-27 17:44:13] ...optimizers/trijoint.py.151: total_norm: 12.344356 [I 2022-10-27 17:44:19] ...trap/engines/engine.py.224: train: epoch 80 | batch 110/2304 [I 2022-10-27 17:44:19] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:30 [I 2022-10-27 17:44:19] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00297 [I 2022-10-27 17:44:19] ...trap/engines/engine.py.230: loss: 0.11561 [I 2022-10-27 17:44:19] ...optimizers/trijoint.py.151: total_norm: 10.871273 [I 2022-10-27 17:44:24] ...trap/engines/engine.py.224: train: epoch 80 | batch 120/2304 [I 2022-10-27 17:44:24] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:22 [I 2022-10-27 17:44:24] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00257 [I 2022-10-27 17:44:24] ...trap/engines/engine.py.230: loss: 0.11907 [I 2022-10-27 17:44:24] ...optimizers/trijoint.py.151: total_norm: 14.283646 [I 2022-10-27 17:44:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 130/2304 [I 2022-10-27 17:44:30] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:12 [I 2022-10-27 17:44:30] ...trap/engines/engine.py.229: process: 0.54923 | load: 0.00261 [I 2022-10-27 17:44:30] ...trap/engines/engine.py.230: loss: 0.16290 [I 2022-10-27 17:44:30] ...optimizers/trijoint.py.151: total_norm: 18.101961 [I 2022-10-27 17:44:36] ...trap/engines/engine.py.224: train: epoch 80 | batch 140/2304 [I 2022-10-27 17:44:36] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:08 [I 2022-10-27 17:44:36] ...trap/engines/engine.py.229: process: 0.57067 | load: 0.00275 [I 2022-10-27 17:44:36] ...trap/engines/engine.py.230: loss: 0.18916 [I 2022-10-27 17:44:36] ...optimizers/trijoint.py.151: total_norm: 39.107525 [I 2022-10-27 17:44:41] ...trap/engines/engine.py.224: train: epoch 80 | batch 150/2304 [I 2022-10-27 17:44:41] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:05 [I 2022-10-27 17:44:41] ...trap/engines/engine.py.229: process: 0.56856 | load: 0.00272 [I 2022-10-27 17:44:41] ...trap/engines/engine.py.230: loss: 0.15852 [I 2022-10-27 17:44:41] ...optimizers/trijoint.py.151: total_norm: 13.430353 [I 2022-10-27 17:44:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 160/2304 [I 2022-10-27 17:44:47] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:53 [I 2022-10-27 17:44:47] ...trap/engines/engine.py.229: process: 0.56609 | load: 0.00253 [I 2022-10-27 17:44:47] ...trap/engines/engine.py.230: loss: 0.10821 [I 2022-10-27 17:44:47] ...optimizers/trijoint.py.151: total_norm: 14.715537 [I 2022-10-27 17:44:52] ...trap/engines/engine.py.224: train: epoch 80 | batch 170/2304 [I 2022-10-27 17:44:52] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:58 [I 2022-10-27 17:44:52] ...trap/engines/engine.py.229: process: 0.55758 | load: 0.00279 [I 2022-10-27 17:44:52] ...trap/engines/engine.py.230: loss: 0.12328 [I 2022-10-27 17:44:52] ...optimizers/trijoint.py.151: total_norm: 21.314619 [I 2022-10-27 17:44:58] ...trap/engines/engine.py.224: train: epoch 80 | batch 180/2304 [I 2022-10-27 17:44:58] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:52 [I 2022-10-27 17:44:58] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00276 [I 2022-10-27 17:44:58] ...trap/engines/engine.py.230: loss: 0.17540 [I 2022-10-27 17:44:58] ...optimizers/trijoint.py.151: total_norm: 26.769344 [I 2022-10-27 17:45:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 190/2304 [I 2022-10-27 17:45:04] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:39 [I 2022-10-27 17:45:04] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00260 [I 2022-10-27 17:45:04] ...trap/engines/engine.py.230: loss: 0.13885 [I 2022-10-27 17:45:04] ...optimizers/trijoint.py.151: total_norm: 12.024926 [I 2022-10-27 17:45:09] ...trap/engines/engine.py.224: train: epoch 80 | batch 200/2304 [I 2022-10-27 17:45:09] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:36 [I 2022-10-27 17:45:09] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00246 [I 2022-10-27 17:45:09] ...trap/engines/engine.py.230: loss: 0.12768 [I 2022-10-27 17:45:09] ...optimizers/trijoint.py.151: total_norm: 16.531143 [I 2022-10-27 17:45:15] ...trap/engines/engine.py.224: train: epoch 80 | batch 210/2304 [I 2022-10-27 17:45:15] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:39 [I 2022-10-27 17:45:15] ...trap/engines/engine.py.229: process: 0.57332 | load: 0.00290 [I 2022-10-27 17:45:15] ...trap/engines/engine.py.230: loss: 0.10971 [I 2022-10-27 17:45:15] ...optimizers/trijoint.py.151: total_norm: 13.576874 [I 2022-10-27 17:45:20] ...trap/engines/engine.py.224: train: epoch 80 | batch 220/2304 [I 2022-10-27 17:45:20] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:15 [I 2022-10-27 17:45:20] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00268 [I 2022-10-27 17:45:20] ...trap/engines/engine.py.230: loss: 0.13056 [I 2022-10-27 17:45:20] ...optimizers/trijoint.py.151: total_norm: 12.981217 [I 2022-10-27 17:45:26] ...trap/engines/engine.py.224: train: epoch 80 | batch 230/2304 [I 2022-10-27 17:45:26] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:12 [I 2022-10-27 17:45:26] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00259 [I 2022-10-27 17:45:26] ...trap/engines/engine.py.230: loss: 0.18628 [I 2022-10-27 17:45:26] ...optimizers/trijoint.py.151: total_norm: 21.041067 [I 2022-10-27 17:45:32] ...trap/engines/engine.py.224: train: epoch 80 | batch 240/2304 [I 2022-10-27 17:45:32] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:10 [I 2022-10-27 17:45:32] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00246 [I 2022-10-27 17:45:32] ...trap/engines/engine.py.230: loss: 0.13175 [I 2022-10-27 17:45:32] ...optimizers/trijoint.py.151: total_norm: 15.372817 [I 2022-10-27 17:45:37] ...trap/engines/engine.py.224: train: epoch 80 | batch 250/2304 [I 2022-10-27 17:45:37] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:04 [I 2022-10-27 17:45:37] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00255 [I 2022-10-27 17:45:37] ...trap/engines/engine.py.230: loss: 0.12503 [I 2022-10-27 17:45:37] ...optimizers/trijoint.py.151: total_norm: 10.352657 [I 2022-10-27 17:45:43] ...trap/engines/engine.py.224: train: epoch 80 | batch 260/2304 [I 2022-10-27 17:45:43] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:56 [I 2022-10-27 17:45:43] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00248 [I 2022-10-27 17:45:43] ...trap/engines/engine.py.230: loss: 0.11714 [I 2022-10-27 17:45:43] ...optimizers/trijoint.py.151: total_norm: 19.888220 [I 2022-10-27 17:45:48] ...trap/engines/engine.py.224: train: epoch 80 | batch 270/2304 [I 2022-10-27 17:45:48] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:04 [I 2022-10-27 17:45:48] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00251 [I 2022-10-27 17:45:48] ...trap/engines/engine.py.230: loss: 0.14326 [I 2022-10-27 17:45:48] ...optimizers/trijoint.py.151: total_norm: 12.916070 [I 2022-10-27 17:45:54] ...trap/engines/engine.py.224: train: epoch 80 | batch 280/2304 [I 2022-10-27 17:45:54] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:50 [I 2022-10-27 17:45:54] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00252 [I 2022-10-27 17:45:54] ...trap/engines/engine.py.230: loss: 0.17711 [I 2022-10-27 17:45:54] ...optimizers/trijoint.py.151: total_norm: 12.294659 [I 2022-10-27 17:46:00] ...trap/engines/engine.py.224: train: epoch 80 | batch 290/2304 [I 2022-10-27 17:46:00] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:52 [I 2022-10-27 17:46:00] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00247 [I 2022-10-27 17:46:00] ...trap/engines/engine.py.230: loss: 0.08811 [I 2022-10-27 17:46:00] ...optimizers/trijoint.py.151: total_norm: 9.842034 [I 2022-10-27 17:46:05] ...trap/engines/engine.py.224: train: epoch 80 | batch 300/2304 [I 2022-10-27 17:46:05] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:53 [I 2022-10-27 17:46:05] ...trap/engines/engine.py.229: process: 0.56270 | load: 0.00251 [I 2022-10-27 17:46:05] ...trap/engines/engine.py.230: loss: 0.07238 [I 2022-10-27 17:46:05] ...optimizers/trijoint.py.151: total_norm: 9.683237 [I 2022-10-27 17:46:11] ...trap/engines/engine.py.224: train: epoch 80 | batch 310/2304 [I 2022-10-27 17:46:11] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:41 [I 2022-10-27 17:46:11] ...trap/engines/engine.py.229: process: 0.56478 | load: 0.00276 [I 2022-10-27 17:46:11] ...trap/engines/engine.py.230: loss: 0.14902 [I 2022-10-27 17:46:11] ...optimizers/trijoint.py.151: total_norm: 13.662736 [I 2022-10-27 17:46:17] ...trap/engines/engine.py.224: train: epoch 80 | batch 320/2304 [I 2022-10-27 17:46:17] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:51 [I 2022-10-27 17:46:17] ...trap/engines/engine.py.229: process: 0.58784 | load: 0.00262 [I 2022-10-27 17:46:17] ...trap/engines/engine.py.230: loss: 0.20297 [I 2022-10-27 17:46:17] ...optimizers/trijoint.py.151: total_norm: 19.913006 [I 2022-10-27 17:46:22] ...trap/engines/engine.py.224: train: epoch 80 | batch 330/2304 [I 2022-10-27 17:46:22] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:29 [I 2022-10-27 17:46:22] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00251 [I 2022-10-27 17:46:22] ...trap/engines/engine.py.230: loss: 0.11391 [I 2022-10-27 17:46:22] ...optimizers/trijoint.py.151: total_norm: 12.015033 [I 2022-10-27 17:46:28] ...trap/engines/engine.py.224: train: epoch 80 | batch 340/2304 [I 2022-10-27 17:46:28] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:56 [I 2022-10-27 17:46:28] ...trap/engines/engine.py.229: process: 0.55341 | load: 0.00256 [I 2022-10-27 17:46:28] ...trap/engines/engine.py.230: loss: 0.12202 [I 2022-10-27 17:46:28] ...optimizers/trijoint.py.151: total_norm: 14.541092 [I 2022-10-27 17:46:34] ...trap/engines/engine.py.224: train: epoch 80 | batch 350/2304 [I 2022-10-27 17:46:34] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:16 [I 2022-10-27 17:46:34] ...trap/engines/engine.py.229: process: 0.54351 | load: 0.00259 [I 2022-10-27 17:46:34] ...trap/engines/engine.py.230: loss: 0.10547 [I 2022-10-27 17:46:34] ...optimizers/trijoint.py.151: total_norm: 8.318441 [I 2022-10-27 17:46:39] ...trap/engines/engine.py.224: train: epoch 80 | batch 360/2304 [I 2022-10-27 17:46:39] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:20 [I 2022-10-27 17:46:39] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00286 [I 2022-10-27 17:46:39] ...trap/engines/engine.py.230: loss: 0.21972 [I 2022-10-27 17:46:39] ...optimizers/trijoint.py.151: total_norm: 11.835464 [I 2022-10-27 17:46:45] ...trap/engines/engine.py.224: train: epoch 80 | batch 370/2304 [I 2022-10-27 17:46:45] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:15 [I 2022-10-27 17:46:45] ...trap/engines/engine.py.229: process: 0.57426 | load: 0.00280 [I 2022-10-27 17:46:45] ...trap/engines/engine.py.230: loss: 0.13395 [I 2022-10-27 17:46:45] ...optimizers/trijoint.py.151: total_norm: 13.099662 [I 2022-10-27 17:46:51] ...trap/engines/engine.py.224: train: epoch 80 | batch 380/2304 [I 2022-10-27 17:46:51] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:17:48 [I 2022-10-27 17:46:51] ...trap/engines/engine.py.229: process: 0.55284 | load: 0.00251 [I 2022-10-27 17:46:51] ...trap/engines/engine.py.230: loss: 0.13085 [I 2022-10-27 17:46:51] ...optimizers/trijoint.py.151: total_norm: 10.989931 [I 2022-10-27 17:46:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 390/2304 [I 2022-10-27 17:46:56] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:44 [I 2022-10-27 17:46:56] ...trap/engines/engine.py.229: process: 0.56049 | load: 0.00252 [I 2022-10-27 17:46:56] ...trap/engines/engine.py.230: loss: 0.18204 [I 2022-10-27 17:46:56] ...optimizers/trijoint.py.151: total_norm: 19.150455 [I 2022-10-27 17:47:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 400/2304 [I 2022-10-27 17:47:02] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:35 [I 2022-10-27 17:47:02] ...trap/engines/engine.py.229: process: 0.56166 | load: 0.00267 [I 2022-10-27 17:47:02] ...trap/engines/engine.py.230: loss: 0.11486 [I 2022-10-27 17:47:02] ...optimizers/trijoint.py.151: total_norm: 8.885717 [I 2022-10-27 17:47:07] ...trap/engines/engine.py.224: train: epoch 80 | batch 410/2304 [I 2022-10-27 17:47:07] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:35 [I 2022-10-27 17:47:07] ...trap/engines/engine.py.229: process: 0.55414 | load: 0.00250 [I 2022-10-27 17:47:07] ...trap/engines/engine.py.230: loss: 0.13375 [I 2022-10-27 17:47:07] ...optimizers/trijoint.py.151: total_norm: 14.099985 [I 2022-10-27 17:47:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 420/2304 [I 2022-10-27 17:47:13] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:27 [I 2022-10-27 17:47:13] ...trap/engines/engine.py.229: process: 0.56678 | load: 0.00247 [I 2022-10-27 17:47:13] ...trap/engines/engine.py.230: loss: 0.11010 [I 2022-10-27 17:47:13] ...optimizers/trijoint.py.151: total_norm: 9.031621 [I 2022-10-27 17:47:19] ...trap/engines/engine.py.224: train: epoch 80 | batch 430/2304 [I 2022-10-27 17:47:19] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:35 [I 2022-10-27 17:47:19] ...trap/engines/engine.py.229: process: 0.56894 | load: 0.00256 [I 2022-10-27 17:47:19] ...trap/engines/engine.py.230: loss: 0.10635 [I 2022-10-27 17:47:19] ...optimizers/trijoint.py.151: total_norm: 10.950852 [I 2022-10-27 17:47:24] ...trap/engines/engine.py.224: train: epoch 80 | batch 440/2304 [I 2022-10-27 17:47:24] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:29 [I 2022-10-27 17:47:24] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00249 [I 2022-10-27 17:47:24] ...trap/engines/engine.py.230: loss: 0.11297 [I 2022-10-27 17:47:24] ...optimizers/trijoint.py.151: total_norm: 10.106306 [I 2022-10-27 17:47:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 450/2304 [I 2022-10-27 17:47:30] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:25 [I 2022-10-27 17:47:30] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00250 [I 2022-10-27 17:47:30] ...trap/engines/engine.py.230: loss: 0.10909 [I 2022-10-27 17:47:30] ...optimizers/trijoint.py.151: total_norm: 8.649693 [I 2022-10-27 17:47:35] ...trap/engines/engine.py.224: train: epoch 80 | batch 460/2304 [I 2022-10-27 17:47:35] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:17:13 [I 2022-10-27 17:47:35] ...trap/engines/engine.py.229: process: 0.57079 | load: 0.00252 [I 2022-10-27 17:47:35] ...trap/engines/engine.py.230: loss: 0.10201 [I 2022-10-27 17:47:35] ...optimizers/trijoint.py.151: total_norm: 12.807849 [I 2022-10-27 17:47:41] ...trap/engines/engine.py.224: train: epoch 80 | batch 470/2304 [I 2022-10-27 17:47:41] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:16:56 [I 2022-10-27 17:47:41] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00248 [I 2022-10-27 17:47:41] ...trap/engines/engine.py.230: loss: 0.11379 [I 2022-10-27 17:47:41] ...optimizers/trijoint.py.151: total_norm: 22.133629 [I 2022-10-27 17:47:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 480/2304 [I 2022-10-27 17:47:47] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:16:59 [I 2022-10-27 17:47:47] ...trap/engines/engine.py.229: process: 0.56365 | load: 0.00244 [I 2022-10-27 17:47:47] ...trap/engines/engine.py.230: loss: 0.10077 [I 2022-10-27 17:47:47] ...optimizers/trijoint.py.151: total_norm: 13.749247 [I 2022-10-27 17:47:52] ...trap/engines/engine.py.224: train: epoch 80 | batch 490/2304 [I 2022-10-27 17:47:52] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:48 [I 2022-10-27 17:47:52] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00255 [I 2022-10-27 17:47:52] ...trap/engines/engine.py.230: loss: 0.12241 [I 2022-10-27 17:47:52] ...optimizers/trijoint.py.151: total_norm: 20.739508 [I 2022-10-27 17:47:58] ...trap/engines/engine.py.224: train: epoch 80 | batch 500/2304 [I 2022-10-27 17:47:58] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:57 [I 2022-10-27 17:47:58] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00256 [I 2022-10-27 17:47:58] ...trap/engines/engine.py.230: loss: 0.11914 [I 2022-10-27 17:47:58] ...optimizers/trijoint.py.151: total_norm: 14.022663 [I 2022-10-27 17:48:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 510/2304 [I 2022-10-27 17:48:04] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:53 [I 2022-10-27 17:48:04] ...trap/engines/engine.py.229: process: 0.56274 | load: 0.00249 [I 2022-10-27 17:48:04] ...trap/engines/engine.py.230: loss: 0.14035 [I 2022-10-27 17:48:04] ...optimizers/trijoint.py.151: total_norm: 18.679386 [I 2022-10-27 17:48:09] ...trap/engines/engine.py.224: train: epoch 80 | batch 520/2304 [I 2022-10-27 17:48:09] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:42 [I 2022-10-27 17:48:09] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00270 [I 2022-10-27 17:48:09] ...trap/engines/engine.py.230: loss: 0.12438 [I 2022-10-27 17:48:09] ...optimizers/trijoint.py.151: total_norm: 47.839214 [I 2022-10-27 17:48:15] ...trap/engines/engine.py.224: train: epoch 80 | batch 530/2304 [I 2022-10-27 17:48:15] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:33 [I 2022-10-27 17:48:15] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00244 [I 2022-10-27 17:48:15] ...trap/engines/engine.py.230: loss: 0.10472 [I 2022-10-27 17:48:15] ...optimizers/trijoint.py.151: total_norm: 28.899925 [I 2022-10-27 17:48:20] ...trap/engines/engine.py.224: train: epoch 80 | batch 540/2304 [I 2022-10-27 17:48:20] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:16:42 [I 2022-10-27 17:48:20] ...trap/engines/engine.py.229: process: 0.57941 | load: 0.00247 [I 2022-10-27 17:48:20] ...trap/engines/engine.py.230: loss: 0.09931 [I 2022-10-27 17:48:20] ...optimizers/trijoint.py.151: total_norm: 12.871603 [I 2022-10-27 17:48:26] ...trap/engines/engine.py.224: train: epoch 80 | batch 550/2304 [I 2022-10-27 17:48:26] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:25 [I 2022-10-27 17:48:26] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00241 [I 2022-10-27 17:48:26] ...trap/engines/engine.py.230: loss: 0.11852 [I 2022-10-27 17:48:26] ...optimizers/trijoint.py.151: total_norm: 9.174463 [I 2022-10-27 17:48:32] ...trap/engines/engine.py.224: train: epoch 80 | batch 560/2304 [I 2022-10-27 17:48:32] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:11 [I 2022-10-27 17:48:32] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00244 [I 2022-10-27 17:48:32] ...trap/engines/engine.py.230: loss: 0.17823 [I 2022-10-27 17:48:32] ...optimizers/trijoint.py.151: total_norm: 13.371053 [I 2022-10-27 17:48:37] ...trap/engines/engine.py.224: train: epoch 80 | batch 570/2304 [I 2022-10-27 17:48:37] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:15:58 [I 2022-10-27 17:48:37] ...trap/engines/engine.py.229: process: 0.55028 | load: 0.00248 [I 2022-10-27 17:48:37] ...trap/engines/engine.py.230: loss: 0.13392 [I 2022-10-27 17:48:37] ...optimizers/trijoint.py.151: total_norm: 11.222956 [I 2022-10-27 17:48:43] ...trap/engines/engine.py.224: train: epoch 80 | batch 580/2304 [I 2022-10-27 17:48:43] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:16:02 [I 2022-10-27 17:48:43] ...trap/engines/engine.py.229: process: 0.56539 | load: 0.00248 [I 2022-10-27 17:48:43] ...trap/engines/engine.py.230: loss: 0.11566 [I 2022-10-27 17:48:43] ...optimizers/trijoint.py.151: total_norm: 16.819044 [I 2022-10-27 17:48:48] ...trap/engines/engine.py.224: train: epoch 80 | batch 590/2304 [I 2022-10-27 17:48:48] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:48 [I 2022-10-27 17:48:48] ...trap/engines/engine.py.229: process: 0.54705 | load: 0.00251 [I 2022-10-27 17:48:48] ...trap/engines/engine.py.230: loss: 0.17501 [I 2022-10-27 17:48:48] ...optimizers/trijoint.py.151: total_norm: 28.756079 [I 2022-10-27 17:48:54] ...trap/engines/engine.py.224: train: epoch 80 | batch 600/2304 [I 2022-10-27 17:48:54] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:16:24 [I 2022-10-27 17:48:54] ...trap/engines/engine.py.229: process: 0.64578 | load: 0.00248 [I 2022-10-27 17:48:54] ...trap/engines/engine.py.230: loss: 0.15274 [I 2022-10-27 17:48:54] ...optimizers/trijoint.py.151: total_norm: 23.697483 [I 2022-10-27 17:49:00] ...trap/engines/engine.py.224: train: epoch 80 | batch 610/2304 [I 2022-10-27 17:49:00] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:15:47 [I 2022-10-27 17:49:00] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00240 [I 2022-10-27 17:49:00] ...trap/engines/engine.py.230: loss: 0.13854 [I 2022-10-27 17:49:00] ...optimizers/trijoint.py.151: total_norm: 18.330639 [I 2022-10-27 17:49:05] ...trap/engines/engine.py.224: train: epoch 80 | batch 620/2304 [I 2022-10-27 17:49:05] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:39 [I 2022-10-27 17:49:05] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00245 [I 2022-10-27 17:49:05] ...trap/engines/engine.py.230: loss: 0.09359 [I 2022-10-27 17:49:05] ...optimizers/trijoint.py.151: total_norm: 7.927225 [I 2022-10-27 17:49:11] ...trap/engines/engine.py.224: train: epoch 80 | batch 630/2304 [I 2022-10-27 17:49:11] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:29 [I 2022-10-27 17:49:11] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00244 [I 2022-10-27 17:49:11] ...trap/engines/engine.py.230: loss: 0.14332 [I 2022-10-27 17:49:11] ...optimizers/trijoint.py.151: total_norm: 14.243024 [I 2022-10-27 17:49:16] ...trap/engines/engine.py.224: train: epoch 80 | batch 640/2304 [I 2022-10-27 17:49:16] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:17 [I 2022-10-27 17:49:16] ...trap/engines/engine.py.229: process: 0.54558 | load: 0.00243 [I 2022-10-27 17:49:16] ...trap/engines/engine.py.230: loss: 0.14826 [I 2022-10-27 17:49:16] ...optimizers/trijoint.py.151: total_norm: 20.949020 [I 2022-10-27 17:49:22] ...trap/engines/engine.py.224: train: epoch 80 | batch 650/2304 [I 2022-10-27 17:49:22] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:10 [I 2022-10-27 17:49:22] ...trap/engines/engine.py.229: process: 0.55095 | load: 0.00245 [I 2022-10-27 17:49:22] ...trap/engines/engine.py.230: loss: 0.08599 [I 2022-10-27 17:49:22] ...optimizers/trijoint.py.151: total_norm: 8.549629 [I 2022-10-27 17:49:27] ...trap/engines/engine.py.224: train: epoch 80 | batch 660/2304 [I 2022-10-27 17:49:27] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:15:10 [I 2022-10-27 17:49:27] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00246 [I 2022-10-27 17:49:27] ...trap/engines/engine.py.230: loss: 0.11606 [I 2022-10-27 17:49:27] ...optimizers/trijoint.py.151: total_norm: 10.990543 [I 2022-10-27 17:49:33] ...trap/engines/engine.py.224: train: epoch 80 | batch 670/2304 [I 2022-10-27 17:49:33] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:15 [I 2022-10-27 17:49:33] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00251 [I 2022-10-27 17:49:33] ...trap/engines/engine.py.230: loss: 0.14255 [I 2022-10-27 17:49:33] ...optimizers/trijoint.py.151: total_norm: 9.277341 [I 2022-10-27 17:49:39] ...trap/engines/engine.py.224: train: epoch 80 | batch 680/2304 [I 2022-10-27 17:49:39] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:04 [I 2022-10-27 17:49:39] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00242 [I 2022-10-27 17:49:39] ...trap/engines/engine.py.230: loss: 0.16070 [I 2022-10-27 17:49:39] ...optimizers/trijoint.py.151: total_norm: 9.444233 [I 2022-10-27 17:49:44] ...trap/engines/engine.py.224: train: epoch 80 | batch 690/2304 [I 2022-10-27 17:49:44] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:01 [I 2022-10-27 17:49:44] ...trap/engines/engine.py.229: process: 0.56484 | load: 0.00244 [I 2022-10-27 17:49:44] ...trap/engines/engine.py.230: loss: 0.09345 [I 2022-10-27 17:49:44] ...optimizers/trijoint.py.151: total_norm: 21.326006 [I 2022-10-27 17:49:50] ...trap/engines/engine.py.224: train: epoch 80 | batch 700/2304 [I 2022-10-27 17:49:50] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:14:50 [I 2022-10-27 17:49:50] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00245 [I 2022-10-27 17:49:50] ...trap/engines/engine.py.230: loss: 0.10355 [I 2022-10-27 17:49:50] ...optimizers/trijoint.py.151: total_norm: 11.695367 [I 2022-10-27 17:49:55] ...trap/engines/engine.py.224: train: epoch 80 | batch 710/2304 [I 2022-10-27 17:49:55] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:52 [I 2022-10-27 17:49:55] ...trap/engines/engine.py.229: process: 0.56470 | load: 0.00245 [I 2022-10-27 17:49:55] ...trap/engines/engine.py.230: loss: 0.14578 [I 2022-10-27 17:49:55] ...optimizers/trijoint.py.151: total_norm: 9.562420 [I 2022-10-27 17:50:01] ...trap/engines/engine.py.224: train: epoch 80 | batch 720/2304 [I 2022-10-27 17:50:01] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:47 [I 2022-10-27 17:50:01] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00244 [I 2022-10-27 17:50:01] ...trap/engines/engine.py.230: loss: 0.19070 [I 2022-10-27 17:50:01] ...optimizers/trijoint.py.151: total_norm: 62.335991 [I 2022-10-27 17:50:07] ...trap/engines/engine.py.224: train: epoch 80 | batch 730/2304 [I 2022-10-27 17:50:07] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:14:44 [I 2022-10-27 17:50:07] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00246 [I 2022-10-27 17:50:07] ...trap/engines/engine.py.230: loss: 0.10695 [I 2022-10-27 17:50:07] ...optimizers/trijoint.py.151: total_norm: 18.609823 [I 2022-10-27 17:50:12] ...trap/engines/engine.py.224: train: epoch 80 | batch 740/2304 [I 2022-10-27 17:50:12] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:33 [I 2022-10-27 17:50:12] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00241 [I 2022-10-27 17:50:12] ...trap/engines/engine.py.230: loss: 0.09986 [I 2022-10-27 17:50:12] ...optimizers/trijoint.py.151: total_norm: 11.640425 [I 2022-10-27 17:50:18] ...trap/engines/engine.py.224: train: epoch 80 | batch 750/2304 [I 2022-10-27 17:50:18] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:32 [I 2022-10-27 17:50:18] ...trap/engines/engine.py.229: process: 0.57630 | load: 0.00248 [I 2022-10-27 17:50:18] ...trap/engines/engine.py.230: loss: 0.15296 [I 2022-10-27 17:50:18] ...optimizers/trijoint.py.151: total_norm: 22.562880 [I 2022-10-27 17:50:23] ...trap/engines/engine.py.224: train: epoch 80 | batch 760/2304 [I 2022-10-27 17:50:23] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:18 [I 2022-10-27 17:50:23] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00240 [I 2022-10-27 17:50:23] ...trap/engines/engine.py.230: loss: 0.14847 [I 2022-10-27 17:50:23] ...optimizers/trijoint.py.151: total_norm: 30.367260 [I 2022-10-27 17:50:29] ...trap/engines/engine.py.224: train: epoch 80 | batch 770/2304 [I 2022-10-27 17:50:29] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:10 [I 2022-10-27 17:50:29] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00242 [I 2022-10-27 17:50:29] ...trap/engines/engine.py.230: loss: 0.15368 [I 2022-10-27 17:50:29] ...optimizers/trijoint.py.151: total_norm: 23.670504 [I 2022-10-27 17:50:35] ...trap/engines/engine.py.224: train: epoch 80 | batch 780/2304 [I 2022-10-27 17:50:35] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:06 [I 2022-10-27 17:50:35] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00248 [I 2022-10-27 17:50:35] ...trap/engines/engine.py.230: loss: 0.11679 [I 2022-10-27 17:50:35] ...optimizers/trijoint.py.151: total_norm: 27.714205 [I 2022-10-27 17:50:40] ...trap/engines/engine.py.224: train: epoch 80 | batch 790/2304 [I 2022-10-27 17:50:40] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:01 [I 2022-10-27 17:50:40] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00254 [I 2022-10-27 17:50:40] ...trap/engines/engine.py.230: loss: 0.10353 [I 2022-10-27 17:50:40] ...optimizers/trijoint.py.151: total_norm: 9.285030 [I 2022-10-27 17:50:46] ...trap/engines/engine.py.224: train: epoch 80 | batch 800/2304 [I 2022-10-27 17:50:46] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:13:58 [I 2022-10-27 17:50:46] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00244 [I 2022-10-27 17:50:46] ...trap/engines/engine.py.230: loss: 0.14496 [I 2022-10-27 17:50:46] ...optimizers/trijoint.py.151: total_norm: 15.530875 [I 2022-10-27 17:50:51] ...trap/engines/engine.py.224: train: epoch 80 | batch 810/2304 [I 2022-10-27 17:50:51] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:50 [I 2022-10-27 17:50:51] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00249 [I 2022-10-27 17:50:51] ...trap/engines/engine.py.230: loss: 0.11375 [I 2022-10-27 17:50:51] ...optimizers/trijoint.py.151: total_norm: 14.604585 [I 2022-10-27 17:50:57] ...trap/engines/engine.py.224: train: epoch 80 | batch 820/2304 [I 2022-10-27 17:50:57] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:48 [I 2022-10-27 17:50:57] ...trap/engines/engine.py.229: process: 0.56409 | load: 0.00247 [I 2022-10-27 17:50:57] ...trap/engines/engine.py.230: loss: 0.17135 [I 2022-10-27 17:50:57] ...optimizers/trijoint.py.151: total_norm: 18.998644 [I 2022-10-27 17:51:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 830/2304 [I 2022-10-27 17:51:02] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:38 [I 2022-10-27 17:51:02] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00241 [I 2022-10-27 17:51:02] ...trap/engines/engine.py.230: loss: 0.16639 [I 2022-10-27 17:51:02] ...optimizers/trijoint.py.151: total_norm: 19.660233 [I 2022-10-27 17:51:08] ...trap/engines/engine.py.224: train: epoch 80 | batch 840/2304 [I 2022-10-27 17:51:08] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:36 [I 2022-10-27 17:51:08] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00242 [I 2022-10-27 17:51:08] ...trap/engines/engine.py.230: loss: 0.13310 [I 2022-10-27 17:51:08] ...optimizers/trijoint.py.151: total_norm: 22.099842 [I 2022-10-27 17:51:14] ...trap/engines/engine.py.224: train: epoch 80 | batch 850/2304 [I 2022-10-27 17:51:14] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:47 [I 2022-10-27 17:51:14] ...trap/engines/engine.py.229: process: 0.54346 | load: 0.00245 [I 2022-10-27 17:51:14] ...trap/engines/engine.py.230: loss: 0.11741 [I 2022-10-27 17:51:14] ...optimizers/trijoint.py.151: total_norm: 12.447953 [I 2022-10-27 17:51:19] ...trap/engines/engine.py.224: train: epoch 80 | batch 860/2304 [I 2022-10-27 17:51:19] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:30 [I 2022-10-27 17:51:19] ...trap/engines/engine.py.229: process: 0.56612 | load: 0.00250 [I 2022-10-27 17:51:19] ...trap/engines/engine.py.230: loss: 0.17468 [I 2022-10-27 17:51:19] ...optimizers/trijoint.py.151: total_norm: 46.326263 [I 2022-10-27 17:51:25] ...trap/engines/engine.py.224: train: epoch 80 | batch 870/2304 [I 2022-10-27 17:51:25] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:19 [I 2022-10-27 17:51:25] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00246 [I 2022-10-27 17:51:25] ...trap/engines/engine.py.230: loss: 0.19526 [I 2022-10-27 17:51:25] ...optimizers/trijoint.py.151: total_norm: 21.852810 [I 2022-10-27 17:51:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 880/2304 [I 2022-10-27 17:51:30] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:09 [I 2022-10-27 17:51:30] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00247 [I 2022-10-27 17:51:30] ...trap/engines/engine.py.230: loss: 0.10425 [I 2022-10-27 17:51:30] ...optimizers/trijoint.py.151: total_norm: 8.195510 [I 2022-10-27 17:51:36] ...trap/engines/engine.py.224: train: epoch 80 | batch 890/2304 [I 2022-10-27 17:51:36] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:07 [I 2022-10-27 17:51:36] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00246 [I 2022-10-27 17:51:36] ...trap/engines/engine.py.230: loss: 0.15486 [I 2022-10-27 17:51:36] ...optimizers/trijoint.py.151: total_norm: 19.109882 [I 2022-10-27 17:51:42] ...trap/engines/engine.py.224: train: epoch 80 | batch 900/2304 [I 2022-10-27 17:51:42] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:04 [I 2022-10-27 17:51:42] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00260 [I 2022-10-27 17:51:42] ...trap/engines/engine.py.230: loss: 0.10415 [I 2022-10-27 17:51:42] ...optimizers/trijoint.py.151: total_norm: 14.919501 [I 2022-10-27 17:51:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 910/2304 [I 2022-10-27 17:51:47] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:57 [I 2022-10-27 17:51:47] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00250 [I 2022-10-27 17:51:47] ...trap/engines/engine.py.230: loss: 0.11805 [I 2022-10-27 17:51:47] ...optimizers/trijoint.py.151: total_norm: 16.490740 [I 2022-10-27 17:51:53] ...trap/engines/engine.py.224: train: epoch 80 | batch 920/2304 [I 2022-10-27 17:51:53] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:49 [I 2022-10-27 17:51:53] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00243 [I 2022-10-27 17:51:53] ...trap/engines/engine.py.230: loss: 0.18032 [I 2022-10-27 17:51:53] ...optimizers/trijoint.py.151: total_norm: 25.931936 [I 2022-10-27 17:51:58] ...trap/engines/engine.py.224: train: epoch 80 | batch 930/2304 [I 2022-10-27 17:51:58] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:46 [I 2022-10-27 17:51:58] ...trap/engines/engine.py.229: process: 0.55624 | load: 0.00245 [I 2022-10-27 17:51:58] ...trap/engines/engine.py.230: loss: 0.16125 [I 2022-10-27 17:51:58] ...optimizers/trijoint.py.151: total_norm: 36.382164 [I 2022-10-27 17:52:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 940/2304 [I 2022-10-27 17:52:04] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:41 [I 2022-10-27 17:52:04] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00243 [I 2022-10-27 17:52:04] ...trap/engines/engine.py.230: loss: 0.12956 [I 2022-10-27 17:52:04] ...optimizers/trijoint.py.151: total_norm: 14.944632 [I 2022-10-27 17:52:09] ...trap/engines/engine.py.224: train: epoch 80 | batch 950/2304 [I 2022-10-27 17:52:09] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:30 [I 2022-10-27 17:52:09] ...trap/engines/engine.py.229: process: 0.55696 | load: 0.00239 [I 2022-10-27 17:52:09] ...trap/engines/engine.py.230: loss: 0.17226 [I 2022-10-27 17:52:09] ...optimizers/trijoint.py.151: total_norm: 30.565952 [I 2022-10-27 17:52:15] ...trap/engines/engine.py.224: train: epoch 80 | batch 960/2304 [I 2022-10-27 17:52:15] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:25 [I 2022-10-27 17:52:15] ...trap/engines/engine.py.229: process: 0.56947 | load: 0.00245 [I 2022-10-27 17:52:15] ...trap/engines/engine.py.230: loss: 0.11452 [I 2022-10-27 17:52:15] ...optimizers/trijoint.py.151: total_norm: 36.426472 [I 2022-10-27 17:52:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 970/2304 [I 2022-10-27 17:52:21] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:21 [I 2022-10-27 17:52:21] ...trap/engines/engine.py.229: process: 0.56358 | load: 0.00242 [I 2022-10-27 17:52:21] ...trap/engines/engine.py.230: loss: 0.09817 [I 2022-10-27 17:52:21] ...optimizers/trijoint.py.151: total_norm: 9.220387 [I 2022-10-27 17:52:26] ...trap/engines/engine.py.224: train: epoch 80 | batch 980/2304 [I 2022-10-27 17:52:26] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:16 [I 2022-10-27 17:52:26] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00241 [I 2022-10-27 17:52:26] ...trap/engines/engine.py.230: loss: 0.15620 [I 2022-10-27 17:52:26] ...optimizers/trijoint.py.151: total_norm: 16.653963 [I 2022-10-27 17:52:32] ...trap/engines/engine.py.224: train: epoch 80 | batch 990/2304 [I 2022-10-27 17:52:32] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:17 [I 2022-10-27 17:52:32] ...trap/engines/engine.py.229: process: 0.56838 | load: 0.00250 [I 2022-10-27 17:52:32] ...trap/engines/engine.py.230: loss: 0.10254 [I 2022-10-27 17:52:32] ...optimizers/trijoint.py.151: total_norm: 12.232009 [I 2022-10-27 17:52:37] ...trap/engines/engine.py.224: train: epoch 80 | batch 1000/2304 [I 2022-10-27 17:52:37] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:02 [I 2022-10-27 17:52:37] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00245 [I 2022-10-27 17:52:37] ...trap/engines/engine.py.230: loss: 0.17112 [I 2022-10-27 17:52:37] ...optimizers/trijoint.py.151: total_norm: 17.654711 [I 2022-10-27 17:52:43] ...trap/engines/engine.py.224: train: epoch 80 | batch 1010/2304 [I 2022-10-27 17:52:43] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:01 [I 2022-10-27 17:52:43] ...trap/engines/engine.py.229: process: 0.56456 | load: 0.00248 [I 2022-10-27 17:52:43] ...trap/engines/engine.py.230: loss: 0.21033 [I 2022-10-27 17:52:43] ...optimizers/trijoint.py.151: total_norm: 8.844210 [I 2022-10-27 17:52:48] ...trap/engines/engine.py.224: train: epoch 80 | batch 1020/2304 [I 2022-10-27 17:52:48] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:51 [I 2022-10-27 17:52:48] ...trap/engines/engine.py.229: process: 0.55093 | load: 0.00244 [I 2022-10-27 17:52:48] ...trap/engines/engine.py.230: loss: 0.19549 [I 2022-10-27 17:52:48] ...optimizers/trijoint.py.151: total_norm: 75.869362 [I 2022-10-27 17:52:54] ...trap/engines/engine.py.224: train: epoch 80 | batch 1030/2304 [I 2022-10-27 17:52:54] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:49 [I 2022-10-27 17:52:54] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00248 [I 2022-10-27 17:52:54] ...trap/engines/engine.py.230: loss: 0.14102 [I 2022-10-27 17:52:54] ...optimizers/trijoint.py.151: total_norm: 30.989534 [I 2022-10-27 17:53:00] ...trap/engines/engine.py.224: train: epoch 80 | batch 1040/2304 [I 2022-10-27 17:53:00] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:46 [I 2022-10-27 17:53:00] ...trap/engines/engine.py.229: process: 0.56382 | load: 0.00242 [I 2022-10-27 17:53:00] ...trap/engines/engine.py.230: loss: 0.10463 [I 2022-10-27 17:53:00] ...optimizers/trijoint.py.151: total_norm: 11.059025 [I 2022-10-27 17:53:05] ...trap/engines/engine.py.224: train: epoch 80 | batch 1050/2304 [I 2022-10-27 17:53:05] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:34 [I 2022-10-27 17:53:05] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00243 [I 2022-10-27 17:53:05] ...trap/engines/engine.py.230: loss: 0.07588 [I 2022-10-27 17:53:05] ...optimizers/trijoint.py.151: total_norm: 16.723110 [I 2022-10-27 17:53:11] ...trap/engines/engine.py.224: train: epoch 80 | batch 1060/2304 [I 2022-10-27 17:53:11] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:30 [I 2022-10-27 17:53:11] ...trap/engines/engine.py.229: process: 0.56094 | load: 0.00250 [I 2022-10-27 17:53:11] ...trap/engines/engine.py.230: loss: 0.12601 [I 2022-10-27 17:53:11] ...optimizers/trijoint.py.151: total_norm: 13.699023 [I 2022-10-27 17:53:16] ...trap/engines/engine.py.224: train: epoch 80 | batch 1070/2304 [I 2022-10-27 17:53:16] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:25 [I 2022-10-27 17:53:16] ...trap/engines/engine.py.229: process: 0.56573 | load: 0.00245 [I 2022-10-27 17:53:16] ...trap/engines/engine.py.230: loss: 0.14276 [I 2022-10-27 17:53:16] ...optimizers/trijoint.py.151: total_norm: 21.250744 [I 2022-10-27 17:53:22] ...trap/engines/engine.py.224: train: epoch 80 | batch 1080/2304 [I 2022-10-27 17:53:22] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:24 [I 2022-10-27 17:53:22] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00246 [I 2022-10-27 17:53:22] ...trap/engines/engine.py.230: loss: 0.16276 [I 2022-10-27 17:53:22] ...optimizers/trijoint.py.151: total_norm: 21.615297 [I 2022-10-27 17:53:27] ...trap/engines/engine.py.224: train: epoch 80 | batch 1090/2304 [I 2022-10-27 17:53:27] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:13 [I 2022-10-27 17:53:27] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00243 [I 2022-10-27 17:53:27] ...trap/engines/engine.py.230: loss: 0.15767 [I 2022-10-27 17:53:27] ...optimizers/trijoint.py.151: total_norm: 17.579062 [I 2022-10-27 17:53:33] ...trap/engines/engine.py.224: train: epoch 80 | batch 1100/2304 [I 2022-10-27 17:53:33] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:06 [I 2022-10-27 17:53:33] ...trap/engines/engine.py.229: process: 0.54501 | load: 0.00241 [I 2022-10-27 17:53:33] ...trap/engines/engine.py.230: loss: 0.10654 [I 2022-10-27 17:53:33] ...optimizers/trijoint.py.151: total_norm: 10.457348 [I 2022-10-27 17:53:39] ...trap/engines/engine.py.224: train: epoch 80 | batch 1110/2304 [I 2022-10-27 17:53:39] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:04 [I 2022-10-27 17:53:39] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00243 [I 2022-10-27 17:53:39] ...trap/engines/engine.py.230: loss: 0.11884 [I 2022-10-27 17:53:39] ...optimizers/trijoint.py.151: total_norm: 22.721317 [I 2022-10-27 17:53:44] ...trap/engines/engine.py.224: train: epoch 80 | batch 1120/2304 [I 2022-10-27 17:53:44] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:10:59 [I 2022-10-27 17:53:44] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00241 [I 2022-10-27 17:53:44] ...trap/engines/engine.py.230: loss: 0.19562 [I 2022-10-27 17:53:44] ...optimizers/trijoint.py.151: total_norm: 65.273247 [I 2022-10-27 17:53:50] ...trap/engines/engine.py.224: train: epoch 80 | batch 1130/2304 [I 2022-10-27 17:53:50] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:51 [I 2022-10-27 17:53:50] ...trap/engines/engine.py.229: process: 0.56899 | load: 0.00244 [I 2022-10-27 17:53:50] ...trap/engines/engine.py.230: loss: 0.12853 [I 2022-10-27 17:53:50] ...optimizers/trijoint.py.151: total_norm: 22.870647 [I 2022-10-27 17:53:55] ...trap/engines/engine.py.224: train: epoch 80 | batch 1140/2304 [I 2022-10-27 17:53:55] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:48 [I 2022-10-27 17:53:55] ...trap/engines/engine.py.229: process: 0.55543 | load: 0.00242 [I 2022-10-27 17:53:55] ...trap/engines/engine.py.230: loss: 0.12040 [I 2022-10-27 17:53:55] ...optimizers/trijoint.py.151: total_norm: 9.692950 [I 2022-10-27 17:54:01] ...trap/engines/engine.py.224: train: epoch 80 | batch 1150/2304 [I 2022-10-27 17:54:01] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:46 [I 2022-10-27 17:54:01] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00242 [I 2022-10-27 17:54:01] ...trap/engines/engine.py.230: loss: 0.09264 [I 2022-10-27 17:54:01] ...optimizers/trijoint.py.151: total_norm: 16.794687 [I 2022-10-27 17:54:06] ...trap/engines/engine.py.224: train: epoch 80 | batch 1160/2304 [I 2022-10-27 17:54:06] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:39 [I 2022-10-27 17:54:06] ...trap/engines/engine.py.229: process: 0.56490 | load: 0.00239 [I 2022-10-27 17:54:06] ...trap/engines/engine.py.230: loss: 0.11806 [I 2022-10-27 17:54:06] ...optimizers/trijoint.py.151: total_norm: 9.000713 [I 2022-10-27 17:54:12] ...trap/engines/engine.py.224: train: epoch 80 | batch 1170/2304 [I 2022-10-27 17:54:12] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:30 [I 2022-10-27 17:54:12] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00239 [I 2022-10-27 17:54:12] ...trap/engines/engine.py.230: loss: 0.15393 [I 2022-10-27 17:54:12] ...optimizers/trijoint.py.151: total_norm: 12.998509 [I 2022-10-27 17:54:18] ...trap/engines/engine.py.224: train: epoch 80 | batch 1180/2304 [I 2022-10-27 17:54:18] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:26 [I 2022-10-27 17:54:18] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00240 [I 2022-10-27 17:54:18] ...trap/engines/engine.py.230: loss: 0.10887 [I 2022-10-27 17:54:18] ...optimizers/trijoint.py.151: total_norm: 17.563498 [I 2022-10-27 17:54:23] ...trap/engines/engine.py.224: train: epoch 80 | batch 1190/2304 [I 2022-10-27 17:54:23] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:16 [I 2022-10-27 17:54:23] ...trap/engines/engine.py.229: process: 0.54948 | load: 0.00240 [I 2022-10-27 17:54:23] ...trap/engines/engine.py.230: loss: 0.17720 [I 2022-10-27 17:54:23] ...optimizers/trijoint.py.151: total_norm: 18.485516 [I 2022-10-27 17:54:29] ...trap/engines/engine.py.224: train: epoch 80 | batch 1200/2304 [I 2022-10-27 17:54:29] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:11 [I 2022-10-27 17:54:29] ...trap/engines/engine.py.229: process: 0.54779 | load: 0.00250 [I 2022-10-27 17:54:29] ...trap/engines/engine.py.230: loss: 0.12478 [I 2022-10-27 17:54:29] ...optimizers/trijoint.py.151: total_norm: 19.426319 [I 2022-10-27 17:54:34] ...trap/engines/engine.py.224: train: epoch 80 | batch 1210/2304 [I 2022-10-27 17:54:34] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:12 [I 2022-10-27 17:54:34] ...trap/engines/engine.py.229: process: 0.56888 | load: 0.00244 [I 2022-10-27 17:54:34] ...trap/engines/engine.py.230: loss: 0.16072 [I 2022-10-27 17:54:34] ...optimizers/trijoint.py.151: total_norm: 16.854441 [I 2022-10-27 17:54:40] ...trap/engines/engine.py.224: train: epoch 80 | batch 1220/2304 [I 2022-10-27 17:54:40] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:03 [I 2022-10-27 17:54:40] ...trap/engines/engine.py.229: process: 0.55284 | load: 0.00244 [I 2022-10-27 17:54:40] ...trap/engines/engine.py.230: loss: 0.18805 [I 2022-10-27 17:54:40] ...optimizers/trijoint.py.151: total_norm: 10.825738 [I 2022-10-27 17:54:45] ...trap/engines/engine.py.224: train: epoch 80 | batch 1230/2304 [I 2022-10-27 17:54:45] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:10:01 [I 2022-10-27 17:54:45] ...trap/engines/engine.py.229: process: 0.56062 | load: 0.00243 [I 2022-10-27 17:54:45] ...trap/engines/engine.py.230: loss: 0.13284 [I 2022-10-27 17:54:45] ...optimizers/trijoint.py.151: total_norm: 14.162821 [I 2022-10-27 17:54:51] ...trap/engines/engine.py.224: train: epoch 80 | batch 1240/2304 [I 2022-10-27 17:54:51] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:55 [I 2022-10-27 17:54:51] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00245 [I 2022-10-27 17:54:51] ...trap/engines/engine.py.230: loss: 0.19171 [I 2022-10-27 17:54:51] ...optimizers/trijoint.py.151: total_norm: 14.995195 [I 2022-10-27 17:54:57] ...trap/engines/engine.py.224: train: epoch 80 | batch 1250/2304 [I 2022-10-27 17:54:57] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:45 [I 2022-10-27 17:54:57] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00242 [I 2022-10-27 17:54:57] ...trap/engines/engine.py.230: loss: 0.12353 [I 2022-10-27 17:54:57] ...optimizers/trijoint.py.151: total_norm: 21.107649 [I 2022-10-27 17:55:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 1260/2304 [I 2022-10-27 17:55:02] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:44 [I 2022-10-27 17:55:02] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00243 [I 2022-10-27 17:55:02] ...trap/engines/engine.py.230: loss: 0.12852 [I 2022-10-27 17:55:02] ...optimizers/trijoint.py.151: total_norm: 26.235981 [I 2022-10-27 17:55:08] ...trap/engines/engine.py.224: train: epoch 80 | batch 1270/2304 [I 2022-10-27 17:55:08] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:36 [I 2022-10-27 17:55:08] ...trap/engines/engine.py.229: process: 0.56025 | load: 0.00252 [I 2022-10-27 17:55:08] ...trap/engines/engine.py.230: loss: 0.15278 [I 2022-10-27 17:55:08] ...optimizers/trijoint.py.151: total_norm: 23.836332 [I 2022-10-27 17:55:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 1280/2304 [I 2022-10-27 17:55:13] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:34 [I 2022-10-27 17:55:13] ...trap/engines/engine.py.229: process: 0.56390 | load: 0.00252 [I 2022-10-27 17:55:13] ...trap/engines/engine.py.230: loss: 0.13138 [I 2022-10-27 17:55:13] ...optimizers/trijoint.py.151: total_norm: 14.515250 [I 2022-10-27 17:55:19] ...trap/engines/engine.py.224: train: epoch 80 | batch 1290/2304 [I 2022-10-27 17:55:19] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:27 [I 2022-10-27 17:55:19] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00245 [I 2022-10-27 17:55:19] ...trap/engines/engine.py.230: loss: 0.16411 [I 2022-10-27 17:55:19] ...optimizers/trijoint.py.151: total_norm: 30.218513 [I 2022-10-27 17:55:25] ...trap/engines/engine.py.224: train: epoch 80 | batch 1300/2304 [I 2022-10-27 17:55:25] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:18 [I 2022-10-27 17:55:25] ...trap/engines/engine.py.229: process: 0.55158 | load: 0.00243 [I 2022-10-27 17:55:25] ...trap/engines/engine.py.230: loss: 0.13424 [I 2022-10-27 17:55:25] ...optimizers/trijoint.py.151: total_norm: 15.005808 [I 2022-10-27 17:55:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 1310/2304 [I 2022-10-27 17:55:30] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:12 [I 2022-10-27 17:55:30] ...trap/engines/engine.py.229: process: 0.55218 | load: 0.00245 [I 2022-10-27 17:55:30] ...trap/engines/engine.py.230: loss: 0.11502 [I 2022-10-27 17:55:30] ...optimizers/trijoint.py.151: total_norm: 13.882942 [I 2022-10-27 17:55:36] ...trap/engines/engine.py.224: train: epoch 80 | batch 1320/2304 [I 2022-10-27 17:55:36] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:06 [I 2022-10-27 17:55:36] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00245 [I 2022-10-27 17:55:36] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-27 17:55:36] ...optimizers/trijoint.py.151: total_norm: 17.045237 [I 2022-10-27 17:55:41] ...trap/engines/engine.py.224: train: epoch 80 | batch 1330/2304 [I 2022-10-27 17:55:41] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:04 [I 2022-10-27 17:55:41] ...trap/engines/engine.py.229: process: 0.55158 | load: 0.00244 [I 2022-10-27 17:55:41] ...trap/engines/engine.py.230: loss: 0.11025 [I 2022-10-27 17:55:41] ...optimizers/trijoint.py.151: total_norm: 13.724767 [I 2022-10-27 17:55:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 1340/2304 [I 2022-10-27 17:55:47] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:58 [I 2022-10-27 17:55:47] ...trap/engines/engine.py.229: process: 0.55137 | load: 0.00239 [I 2022-10-27 17:55:47] ...trap/engines/engine.py.230: loss: 0.08875 [I 2022-10-27 17:55:47] ...optimizers/trijoint.py.151: total_norm: 6.877240 [I 2022-10-27 17:55:52] ...trap/engines/engine.py.224: train: epoch 80 | batch 1350/2304 [I 2022-10-27 17:55:52] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:53 [I 2022-10-27 17:55:52] ...trap/engines/engine.py.229: process: 0.55642 | load: 0.00242 [I 2022-10-27 17:55:52] ...trap/engines/engine.py.230: loss: 0.17459 [I 2022-10-27 17:55:52] ...optimizers/trijoint.py.151: total_norm: 17.269724 [I 2022-10-27 17:55:58] ...trap/engines/engine.py.224: train: epoch 80 | batch 1360/2304 [I 2022-10-27 17:55:58] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:47 [I 2022-10-27 17:55:58] ...trap/engines/engine.py.229: process: 0.55115 | load: 0.00244 [I 2022-10-27 17:55:58] ...trap/engines/engine.py.230: loss: 0.12167 [I 2022-10-27 17:55:58] ...optimizers/trijoint.py.151: total_norm: 72.664574 [I 2022-10-27 17:56:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 1370/2304 [I 2022-10-27 17:56:04] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:41 [I 2022-10-27 17:56:04] ...trap/engines/engine.py.229: process: 0.54818 | load: 0.00251 [I 2022-10-27 17:56:04] ...trap/engines/engine.py.230: loss: 0.13367 [I 2022-10-27 17:56:04] ...optimizers/trijoint.py.151: total_norm: 15.207778 [I 2022-10-27 17:56:09] ...trap/engines/engine.py.224: train: epoch 80 | batch 1380/2304 [I 2022-10-27 17:56:09] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:40 [I 2022-10-27 17:56:09] ...trap/engines/engine.py.229: process: 0.56964 | load: 0.00267 [I 2022-10-27 17:56:09] ...trap/engines/engine.py.230: loss: 0.15549 [I 2022-10-27 17:56:09] ...optimizers/trijoint.py.151: total_norm: 19.900539 [I 2022-10-27 17:56:15] ...trap/engines/engine.py.224: train: epoch 80 | batch 1390/2304 [I 2022-10-27 17:56:15] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:33 [I 2022-10-27 17:56:15] ...trap/engines/engine.py.229: process: 0.56274 | load: 0.00246 [I 2022-10-27 17:56:15] ...trap/engines/engine.py.230: loss: 0.12748 [I 2022-10-27 17:56:15] ...optimizers/trijoint.py.151: total_norm: 18.131231 [I 2022-10-27 17:56:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 1400/2304 [I 2022-10-27 17:56:21] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:26 [I 2022-10-27 17:56:21] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00251 [I 2022-10-27 17:56:21] ...trap/engines/engine.py.230: loss: 0.13348 [I 2022-10-27 17:56:21] ...optimizers/trijoint.py.151: total_norm: 18.333916 [I 2022-10-27 17:56:26] ...trap/engines/engine.py.224: train: epoch 80 | batch 1410/2304 [I 2022-10-27 17:56:26] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:23 [I 2022-10-27 17:56:26] ...trap/engines/engine.py.229: process: 0.57000 | load: 0.00246 [I 2022-10-27 17:56:26] ...trap/engines/engine.py.230: loss: 0.11120 [I 2022-10-27 17:56:26] ...optimizers/trijoint.py.151: total_norm: 14.878525 [I 2022-10-27 17:56:32] ...trap/engines/engine.py.224: train: epoch 80 | batch 1420/2304 [I 2022-10-27 17:56:32] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:31 [I 2022-10-27 17:56:32] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00250 [I 2022-10-27 17:56:32] ...trap/engines/engine.py.230: loss: 0.09074 [I 2022-10-27 17:56:32] ...optimizers/trijoint.py.151: total_norm: 13.102383 [I 2022-10-27 17:56:38] ...trap/engines/engine.py.224: train: epoch 80 | batch 1430/2304 [I 2022-10-27 17:56:38] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:13 [I 2022-10-27 17:56:38] ...trap/engines/engine.py.229: process: 0.57375 | load: 0.00244 [I 2022-10-27 17:56:38] ...trap/engines/engine.py.230: loss: 0.16779 [I 2022-10-27 17:56:38] ...optimizers/trijoint.py.151: total_norm: 38.802711 [I 2022-10-27 17:56:43] ...trap/engines/engine.py.224: train: epoch 80 | batch 1440/2304 [I 2022-10-27 17:56:43] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:04 [I 2022-10-27 17:56:43] ...trap/engines/engine.py.229: process: 0.57542 | load: 0.00248 [I 2022-10-27 17:56:43] ...trap/engines/engine.py.230: loss: 0.10734 [I 2022-10-27 17:56:43] ...optimizers/trijoint.py.151: total_norm: 8.406656 [I 2022-10-27 17:56:49] ...trap/engines/engine.py.224: train: epoch 80 | batch 1450/2304 [I 2022-10-27 17:56:49] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:07:56 [I 2022-10-27 17:56:49] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00246 [I 2022-10-27 17:56:49] ...trap/engines/engine.py.230: loss: 0.10597 [I 2022-10-27 17:56:49] ...optimizers/trijoint.py.151: total_norm: 16.361647 [I 2022-10-27 17:56:54] ...trap/engines/engine.py.224: train: epoch 80 | batch 1460/2304 [I 2022-10-27 17:56:54] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:49 [I 2022-10-27 17:56:54] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00246 [I 2022-10-27 17:56:54] ...trap/engines/engine.py.230: loss: 0.16610 [I 2022-10-27 17:56:54] ...optimizers/trijoint.py.151: total_norm: 12.164835 [I 2022-10-27 17:57:00] ...trap/engines/engine.py.224: train: epoch 80 | batch 1470/2304 [I 2022-10-27 17:57:00] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:47 [I 2022-10-27 17:57:00] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00244 [I 2022-10-27 17:57:00] ...trap/engines/engine.py.230: loss: 0.12643 [I 2022-10-27 17:57:00] ...optimizers/trijoint.py.151: total_norm: 18.515362 [I 2022-10-27 17:57:06] ...trap/engines/engine.py.224: train: epoch 80 | batch 1480/2304 [I 2022-10-27 17:57:06] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:41 [I 2022-10-27 17:57:06] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00243 [I 2022-10-27 17:57:06] ...trap/engines/engine.py.230: loss: 0.10615 [I 2022-10-27 17:57:06] ...optimizers/trijoint.py.151: total_norm: 15.167404 [I 2022-10-27 17:57:11] ...trap/engines/engine.py.224: train: epoch 80 | batch 1490/2304 [I 2022-10-27 17:57:11] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:33 [I 2022-10-27 17:57:11] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00243 [I 2022-10-27 17:57:11] ...trap/engines/engine.py.230: loss: 0.17202 [I 2022-10-27 17:57:11] ...optimizers/trijoint.py.151: total_norm: 16.126402 [I 2022-10-27 17:57:17] ...trap/engines/engine.py.224: train: epoch 80 | batch 1500/2304 [I 2022-10-27 17:57:17] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:32 [I 2022-10-27 17:57:17] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00248 [I 2022-10-27 17:57:17] ...trap/engines/engine.py.230: loss: 0.12260 [I 2022-10-27 17:57:17] ...optimizers/trijoint.py.151: total_norm: 12.415264 [I 2022-10-27 17:57:23] ...trap/engines/engine.py.224: train: epoch 80 | batch 1510/2304 [I 2022-10-27 17:57:23] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:24 [I 2022-10-27 17:57:23] ...trap/engines/engine.py.229: process: 0.57278 | load: 0.00244 [I 2022-10-27 17:57:23] ...trap/engines/engine.py.230: loss: 0.20586 [I 2022-10-27 17:57:23] ...optimizers/trijoint.py.151: total_norm: 18.356806 [I 2022-10-27 17:57:28] ...trap/engines/engine.py.224: train: epoch 80 | batch 1520/2304 [I 2022-10-27 17:57:28] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:16 [I 2022-10-27 17:57:28] ...trap/engines/engine.py.229: process: 0.55149 | load: 0.00242 [I 2022-10-27 17:57:28] ...trap/engines/engine.py.230: loss: 0.11818 [I 2022-10-27 17:57:28] ...optimizers/trijoint.py.151: total_norm: 13.000209 [I 2022-10-27 17:57:34] ...trap/engines/engine.py.224: train: epoch 80 | batch 1530/2304 [I 2022-10-27 17:57:34] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:10 [I 2022-10-27 17:57:34] ...trap/engines/engine.py.229: process: 0.55118 | load: 0.00263 [I 2022-10-27 17:57:34] ...trap/engines/engine.py.230: loss: 0.20039 [I 2022-10-27 17:57:34] ...optimizers/trijoint.py.151: total_norm: 12.417389 [I 2022-10-27 17:57:39] ...trap/engines/engine.py.224: train: epoch 80 | batch 1540/2304 [I 2022-10-27 17:57:39] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:06 [I 2022-10-27 17:57:39] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00245 [I 2022-10-27 17:57:39] ...trap/engines/engine.py.230: loss: 0.16889 [I 2022-10-27 17:57:39] ...optimizers/trijoint.py.151: total_norm: 23.643318 [I 2022-10-27 17:57:45] ...trap/engines/engine.py.224: train: epoch 80 | batch 1550/2304 [I 2022-10-27 17:57:45] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:06:59 [I 2022-10-27 17:57:45] ...trap/engines/engine.py.229: process: 0.55190 | load: 0.00241 [I 2022-10-27 17:57:45] ...trap/engines/engine.py.230: loss: 0.09911 [I 2022-10-27 17:57:45] ...optimizers/trijoint.py.151: total_norm: 9.932197 [I 2022-10-27 17:57:51] ...trap/engines/engine.py.224: train: epoch 80 | batch 1560/2304 [I 2022-10-27 17:57:51] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:07:11 [I 2022-10-27 17:57:51] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00242 [I 2022-10-27 17:57:51] ...trap/engines/engine.py.230: loss: 0.11569 [I 2022-10-27 17:57:51] ...optimizers/trijoint.py.151: total_norm: 17.097954 [I 2022-10-27 17:57:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 1570/2304 [I 2022-10-27 17:57:56] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:54 [I 2022-10-27 17:57:56] ...trap/engines/engine.py.229: process: 0.57398 | load: 0.00249 [I 2022-10-27 17:57:56] ...trap/engines/engine.py.230: loss: 0.20687 [I 2022-10-27 17:57:56] ...optimizers/trijoint.py.151: total_norm: 16.104116 [I 2022-10-27 17:58:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 1580/2304 [I 2022-10-27 17:58:02] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:44 [I 2022-10-27 17:58:02] ...trap/engines/engine.py.229: process: 0.56194 | load: 0.00246 [I 2022-10-27 17:58:02] ...trap/engines/engine.py.230: loss: 0.10965 [I 2022-10-27 17:58:02] ...optimizers/trijoint.py.151: total_norm: 8.181136 [I 2022-10-27 17:58:07] ...trap/engines/engine.py.224: train: epoch 80 | batch 1590/2304 [I 2022-10-27 17:58:07] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:39 [I 2022-10-27 17:58:07] ...trap/engines/engine.py.229: process: 0.56391 | load: 0.00244 [I 2022-10-27 17:58:07] ...trap/engines/engine.py.230: loss: 0.14427 [I 2022-10-27 17:58:07] ...optimizers/trijoint.py.151: total_norm: 14.281754 [I 2022-10-27 17:58:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 1600/2304 [I 2022-10-27 17:58:13] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:30 [I 2022-10-27 17:58:13] ...trap/engines/engine.py.229: process: 0.54567 | load: 0.00246 [I 2022-10-27 17:58:13] ...trap/engines/engine.py.230: loss: 0.14754 [I 2022-10-27 17:58:13] ...optimizers/trijoint.py.151: total_norm: 11.999744 [I 2022-10-27 17:58:19] ...trap/engines/engine.py.224: train: epoch 80 | batch 1610/2304 [I 2022-10-27 17:58:19] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:27 [I 2022-10-27 17:58:19] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00267 [I 2022-10-27 17:58:19] ...trap/engines/engine.py.230: loss: 0.12241 [I 2022-10-27 17:58:19] ...optimizers/trijoint.py.151: total_norm: 18.537298 [I 2022-10-27 17:58:24] ...trap/engines/engine.py.224: train: epoch 80 | batch 1620/2304 [I 2022-10-27 17:58:24] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:23 [I 2022-10-27 17:58:24] ...trap/engines/engine.py.229: process: 0.56926 | load: 0.00243 [I 2022-10-27 17:58:24] ...trap/engines/engine.py.230: loss: 0.16026 [I 2022-10-27 17:58:24] ...optimizers/trijoint.py.151: total_norm: 16.803606 [I 2022-10-27 17:58:30] ...trap/engines/engine.py.224: train: epoch 80 | batch 1630/2304 [I 2022-10-27 17:58:30] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:16 [I 2022-10-27 17:58:30] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00243 [I 2022-10-27 17:58:30] ...trap/engines/engine.py.230: loss: 0.12738 [I 2022-10-27 17:58:30] ...optimizers/trijoint.py.151: total_norm: 39.692314 [I 2022-10-27 17:58:35] ...trap/engines/engine.py.224: train: epoch 80 | batch 1640/2304 [I 2022-10-27 17:58:35] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:09 [I 2022-10-27 17:58:35] ...trap/engines/engine.py.229: process: 0.55803 | load: 0.00245 [I 2022-10-27 17:58:35] ...trap/engines/engine.py.230: loss: 0.20099 [I 2022-10-27 17:58:35] ...optimizers/trijoint.py.151: total_norm: 15.491843 [I 2022-10-27 17:58:41] ...trap/engines/engine.py.224: train: epoch 80 | batch 1650/2304 [I 2022-10-27 17:58:41] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:05 [I 2022-10-27 17:58:41] ...trap/engines/engine.py.229: process: 0.54819 | load: 0.00244 [I 2022-10-27 17:58:41] ...trap/engines/engine.py.230: loss: 0.10826 [I 2022-10-27 17:58:41] ...optimizers/trijoint.py.151: total_norm: 7.820652 [I 2022-10-27 17:58:47] ...trap/engines/engine.py.224: train: epoch 80 | batch 1660/2304 [I 2022-10-27 17:58:47] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:06:12 [I 2022-10-27 17:58:47] ...trap/engines/engine.py.229: process: 0.56823 | load: 0.00250 [I 2022-10-27 17:58:47] ...trap/engines/engine.py.230: loss: 0.08837 [I 2022-10-27 17:58:47] ...optimizers/trijoint.py.151: total_norm: 19.069342 [I 2022-10-27 17:58:52] ...trap/engines/engine.py.224: train: epoch 80 | batch 1670/2304 [I 2022-10-27 17:58:52] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:57 [I 2022-10-27 17:58:52] ...trap/engines/engine.py.229: process: 0.56413 | load: 0.00249 [I 2022-10-27 17:58:52] ...trap/engines/engine.py.230: loss: 0.10301 [I 2022-10-27 17:58:52] ...optimizers/trijoint.py.151: total_norm: 10.595502 [I 2022-10-27 17:58:58] ...trap/engines/engine.py.224: train: epoch 80 | batch 1680/2304 [I 2022-10-27 17:58:58] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:48 [I 2022-10-27 17:58:58] ...trap/engines/engine.py.229: process: 0.56044 | load: 0.00244 [I 2022-10-27 17:58:58] ...trap/engines/engine.py.230: loss: 0.20596 [I 2022-10-27 17:58:58] ...optimizers/trijoint.py.151: total_norm: 20.488977 [I 2022-10-27 17:59:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 1690/2304 [I 2022-10-27 17:59:04] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:42 [I 2022-10-27 17:59:04] ...trap/engines/engine.py.229: process: 0.54651 | load: 0.00246 [I 2022-10-27 17:59:04] ...trap/engines/engine.py.230: loss: 0.15250 [I 2022-10-27 17:59:04] ...optimizers/trijoint.py.151: total_norm: 16.184889 [I 2022-10-27 17:59:09] ...trap/engines/engine.py.224: train: epoch 80 | batch 1700/2304 [I 2022-10-27 17:59:09] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:37 [I 2022-10-27 17:59:09] ...trap/engines/engine.py.229: process: 0.54663 | load: 0.00244 [I 2022-10-27 17:59:09] ...trap/engines/engine.py.230: loss: 0.09398 [I 2022-10-27 17:59:09] ...optimizers/trijoint.py.151: total_norm: 21.673161 [I 2022-10-27 17:59:15] ...trap/engines/engine.py.224: train: epoch 80 | batch 1710/2304 [I 2022-10-27 17:59:15] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:32 [I 2022-10-27 17:59:15] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00243 [I 2022-10-27 17:59:15] ...trap/engines/engine.py.230: loss: 0.16232 [I 2022-10-27 17:59:15] ...optimizers/trijoint.py.151: total_norm: 16.504175 [I 2022-10-27 17:59:21] ...trap/engines/engine.py.224: train: epoch 80 | batch 1720/2304 [I 2022-10-27 17:59:21] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:27 [I 2022-10-27 17:59:21] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00244 [I 2022-10-27 17:59:21] ...trap/engines/engine.py.230: loss: 0.11128 [I 2022-10-27 17:59:21] ...optimizers/trijoint.py.151: total_norm: 11.028282 [I 2022-10-27 17:59:26] ...trap/engines/engine.py.224: train: epoch 80 | batch 1730/2304 [I 2022-10-27 17:59:26] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:20 [I 2022-10-27 17:59:26] ...trap/engines/engine.py.229: process: 0.54921 | load: 0.00243 [I 2022-10-27 17:59:26] ...trap/engines/engine.py.230: loss: 0.10314 [I 2022-10-27 17:59:26] ...optimizers/trijoint.py.151: total_norm: 7.653763 [I 2022-10-27 17:59:32] ...trap/engines/engine.py.224: train: epoch 80 | batch 1740/2304 [I 2022-10-27 17:59:32] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:15 [I 2022-10-27 17:59:32] ...trap/engines/engine.py.229: process: 0.56416 | load: 0.00243 [I 2022-10-27 17:59:32] ...trap/engines/engine.py.230: loss: 0.09257 [I 2022-10-27 17:59:32] ...optimizers/trijoint.py.151: total_norm: 19.060255 [I 2022-10-27 17:59:37] ...trap/engines/engine.py.224: train: epoch 80 | batch 1750/2304 [I 2022-10-27 17:59:37] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:07 [I 2022-10-27 17:59:37] ...trap/engines/engine.py.229: process: 0.54663 | load: 0.00243 [I 2022-10-27 17:59:37] ...trap/engines/engine.py.230: loss: 0.10612 [I 2022-10-27 17:59:37] ...optimizers/trijoint.py.151: total_norm: 13.823222 [I 2022-10-27 17:59:43] ...trap/engines/engine.py.224: train: epoch 80 | batch 1760/2304 [I 2022-10-27 17:59:43] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:03 [I 2022-10-27 17:59:43] ...trap/engines/engine.py.229: process: 0.54207 | load: 0.00251 [I 2022-10-27 17:59:43] ...trap/engines/engine.py.230: loss: 0.09997 [I 2022-10-27 17:59:43] ...optimizers/trijoint.py.151: total_norm: 7.987964 [I 2022-10-27 17:59:49] ...trap/engines/engine.py.224: train: epoch 80 | batch 1770/2304 [I 2022-10-27 17:59:49] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:05:02 [I 2022-10-27 17:59:49] ...trap/engines/engine.py.229: process: 0.56348 | load: 0.00251 [I 2022-10-27 17:59:49] ...trap/engines/engine.py.230: loss: 0.12014 [I 2022-10-27 17:59:49] ...optimizers/trijoint.py.151: total_norm: 19.722784 [I 2022-10-27 17:59:54] ...trap/engines/engine.py.224: train: epoch 80 | batch 1780/2304 [I 2022-10-27 17:59:54] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:50 [I 2022-10-27 17:59:54] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00248 [I 2022-10-27 17:59:54] ...trap/engines/engine.py.230: loss: 0.11333 [I 2022-10-27 17:59:54] ...optimizers/trijoint.py.151: total_norm: 11.529685 [I 2022-10-27 18:00:00] ...trap/engines/engine.py.224: train: epoch 80 | batch 1790/2304 [I 2022-10-27 18:00:00] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:45 [I 2022-10-27 18:00:00] ...trap/engines/engine.py.229: process: 0.54909 | load: 0.00243 [I 2022-10-27 18:00:00] ...trap/engines/engine.py.230: loss: 0.11039 [I 2022-10-27 18:00:00] ...optimizers/trijoint.py.151: total_norm: 13.981291 [I 2022-10-27 18:00:05] ...trap/engines/engine.py.224: train: epoch 80 | batch 1800/2304 [I 2022-10-27 18:00:05] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:41 [I 2022-10-27 18:00:05] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00245 [I 2022-10-27 18:00:05] ...trap/engines/engine.py.230: loss: 0.09874 [I 2022-10-27 18:00:05] ...optimizers/trijoint.py.151: total_norm: 15.046768 [I 2022-10-27 18:00:11] ...trap/engines/engine.py.224: train: epoch 80 | batch 1810/2304 [I 2022-10-27 18:00:11] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:36 [I 2022-10-27 18:00:11] ...trap/engines/engine.py.229: process: 0.56377 | load: 0.00242 [I 2022-10-27 18:00:11] ...trap/engines/engine.py.230: loss: 0.14113 [I 2022-10-27 18:00:11] ...optimizers/trijoint.py.151: total_norm: 24.980164 [I 2022-10-27 18:00:16] ...trap/engines/engine.py.224: train: epoch 80 | batch 1820/2304 [I 2022-10-27 18:00:16] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:30 [I 2022-10-27 18:00:16] ...trap/engines/engine.py.229: process: 0.55488 | load: 0.00244 [I 2022-10-27 18:00:16] ...trap/engines/engine.py.230: loss: 0.16164 [I 2022-10-27 18:00:16] ...optimizers/trijoint.py.151: total_norm: 18.992985 [I 2022-10-27 18:00:22] ...trap/engines/engine.py.224: train: epoch 80 | batch 1830/2304 [I 2022-10-27 18:00:22] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:25 [I 2022-10-27 18:00:22] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00253 [I 2022-10-27 18:00:22] ...trap/engines/engine.py.230: loss: 0.13133 [I 2022-10-27 18:00:22] ...optimizers/trijoint.py.151: total_norm: 12.833244 [I 2022-10-27 18:00:28] ...trap/engines/engine.py.224: train: epoch 80 | batch 1840/2304 [I 2022-10-27 18:00:28] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:20 [I 2022-10-27 18:00:28] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00247 [I 2022-10-27 18:00:28] ...trap/engines/engine.py.230: loss: 0.17206 [I 2022-10-27 18:00:28] ...optimizers/trijoint.py.151: total_norm: 14.737708 [I 2022-10-27 18:00:33] ...trap/engines/engine.py.224: train: epoch 80 | batch 1850/2304 [I 2022-10-27 18:00:33] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:14 [I 2022-10-27 18:00:33] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00247 [I 2022-10-27 18:00:33] ...trap/engines/engine.py.230: loss: 0.11588 [I 2022-10-27 18:00:33] ...optimizers/trijoint.py.151: total_norm: 17.783560 [I 2022-10-27 18:00:39] ...trap/engines/engine.py.224: train: epoch 80 | batch 1860/2304 [I 2022-10-27 18:00:39] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:07 [I 2022-10-27 18:00:39] ...trap/engines/engine.py.229: process: 0.54756 | load: 0.00245 [I 2022-10-27 18:00:39] ...trap/engines/engine.py.230: loss: 0.18983 [I 2022-10-27 18:00:39] ...optimizers/trijoint.py.151: total_norm: 11.451139 [I 2022-10-27 18:00:44] ...trap/engines/engine.py.224: train: epoch 80 | batch 1870/2304 [I 2022-10-27 18:00:44] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:00 [I 2022-10-27 18:00:44] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00241 [I 2022-10-27 18:00:44] ...trap/engines/engine.py.230: loss: 0.13006 [I 2022-10-27 18:00:44] ...optimizers/trijoint.py.151: total_norm: 18.771648 [I 2022-10-27 18:00:50] ...trap/engines/engine.py.224: train: epoch 80 | batch 1880/2304 [I 2022-10-27 18:00:50] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:03 [I 2022-10-27 18:00:50] ...trap/engines/engine.py.229: process: 0.55423 | load: 0.00244 [I 2022-10-27 18:00:50] ...trap/engines/engine.py.230: loss: 0.11768 [I 2022-10-27 18:00:50] ...optimizers/trijoint.py.151: total_norm: 16.193640 [I 2022-10-27 18:00:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 1890/2304 [I 2022-10-27 18:00:56] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:50 [I 2022-10-27 18:00:56] ...trap/engines/engine.py.229: process: 0.54719 | load: 0.00242 [I 2022-10-27 18:00:56] ...trap/engines/engine.py.230: loss: 0.08640 [I 2022-10-27 18:00:56] ...optimizers/trijoint.py.151: total_norm: 11.262555 [I 2022-10-27 18:01:01] ...trap/engines/engine.py.224: train: epoch 80 | batch 1900/2304 [I 2022-10-27 18:01:01] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:46 [I 2022-10-27 18:01:01] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00242 [I 2022-10-27 18:01:01] ...trap/engines/engine.py.230: loss: 0.12882 [I 2022-10-27 18:01:01] ...optimizers/trijoint.py.151: total_norm: 34.121834 [I 2022-10-27 18:01:07] ...trap/engines/engine.py.224: train: epoch 80 | batch 1910/2304 [I 2022-10-27 18:01:07] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:39 [I 2022-10-27 18:01:07] ...trap/engines/engine.py.229: process: 0.54937 | load: 0.00243 [I 2022-10-27 18:01:07] ...trap/engines/engine.py.230: loss: 0.11952 [I 2022-10-27 18:01:07] ...optimizers/trijoint.py.151: total_norm: 11.511176 [I 2022-10-27 18:01:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 1920/2304 [I 2022-10-27 18:01:13] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:34 [I 2022-10-27 18:01:13] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00241 [I 2022-10-27 18:01:13] ...trap/engines/engine.py.230: loss: 0.11671 [I 2022-10-27 18:01:13] ...optimizers/trijoint.py.151: total_norm: 10.283201 [I 2022-10-27 18:01:18] ...trap/engines/engine.py.224: train: epoch 80 | batch 1930/2304 [I 2022-10-27 18:01:18] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:29 [I 2022-10-27 18:01:18] ...trap/engines/engine.py.229: process: 0.56458 | load: 0.00242 [I 2022-10-27 18:01:18] ...trap/engines/engine.py.230: loss: 0.11541 [I 2022-10-27 18:01:18] ...optimizers/trijoint.py.151: total_norm: 18.085276 [I 2022-10-27 18:01:24] ...trap/engines/engine.py.224: train: epoch 80 | batch 1940/2304 [I 2022-10-27 18:01:24] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:21 [I 2022-10-27 18:01:24] ...trap/engines/engine.py.229: process: 0.54503 | load: 0.00240 [I 2022-10-27 18:01:24] ...trap/engines/engine.py.230: loss: 0.11157 [I 2022-10-27 18:01:24] ...optimizers/trijoint.py.151: total_norm: 13.503727 [I 2022-10-27 18:01:29] ...trap/engines/engine.py.224: train: epoch 80 | batch 1950/2304 [I 2022-10-27 18:01:29] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:17 [I 2022-10-27 18:01:29] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00240 [I 2022-10-27 18:01:29] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-27 18:01:29] ...optimizers/trijoint.py.151: total_norm: 19.262075 [I 2022-10-27 18:01:35] ...trap/engines/engine.py.224: train: epoch 80 | batch 1960/2304 [I 2022-10-27 18:01:35] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:13 [I 2022-10-27 18:01:35] ...trap/engines/engine.py.229: process: 0.56536 | load: 0.00243 [I 2022-10-27 18:01:35] ...trap/engines/engine.py.230: loss: 0.13897 [I 2022-10-27 18:01:35] ...optimizers/trijoint.py.151: total_norm: 36.851559 [I 2022-10-27 18:01:41] ...trap/engines/engine.py.224: train: epoch 80 | batch 1970/2304 [I 2022-10-27 18:01:41] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:05 [I 2022-10-27 18:01:41] ...trap/engines/engine.py.229: process: 0.54661 | load: 0.00241 [I 2022-10-27 18:01:41] ...trap/engines/engine.py.230: loss: 0.18923 [I 2022-10-27 18:01:41] ...optimizers/trijoint.py.151: total_norm: 20.071840 [I 2022-10-27 18:01:46] ...trap/engines/engine.py.224: train: epoch 80 | batch 1980/2304 [I 2022-10-27 18:01:46] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:00 [I 2022-10-27 18:01:46] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00241 [I 2022-10-27 18:01:46] ...trap/engines/engine.py.230: loss: 0.12823 [I 2022-10-27 18:01:46] ...optimizers/trijoint.py.151: total_norm: 67.846840 [I 2022-10-27 18:01:52] ...trap/engines/engine.py.224: train: epoch 80 | batch 1990/2304 [I 2022-10-27 18:01:52] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:02:57 [I 2022-10-27 18:01:52] ...trap/engines/engine.py.229: process: 0.56710 | load: 0.00255 [I 2022-10-27 18:01:52] ...trap/engines/engine.py.230: loss: 0.14942 [I 2022-10-27 18:01:52] ...optimizers/trijoint.py.151: total_norm: 20.163406 [I 2022-10-27 18:01:57] ...trap/engines/engine.py.224: train: epoch 80 | batch 2000/2304 [I 2022-10-27 18:01:57] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:48 [I 2022-10-27 18:01:57] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00245 [I 2022-10-27 18:01:57] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-27 18:01:57] ...optimizers/trijoint.py.151: total_norm: 10.491709 [I 2022-10-27 18:02:03] ...trap/engines/engine.py.224: train: epoch 80 | batch 2010/2304 [I 2022-10-27 18:02:03] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:43 [I 2022-10-27 18:02:03] ...trap/engines/engine.py.229: process: 0.54723 | load: 0.00244 [I 2022-10-27 18:02:03] ...trap/engines/engine.py.230: loss: 0.11779 [I 2022-10-27 18:02:03] ...optimizers/trijoint.py.151: total_norm: 16.555199 [I 2022-10-27 18:02:09] ...trap/engines/engine.py.224: train: epoch 80 | batch 2020/2304 [I 2022-10-27 18:02:09] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:38 [I 2022-10-27 18:02:09] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00245 [I 2022-10-27 18:02:09] ...trap/engines/engine.py.230: loss: 0.18918 [I 2022-10-27 18:02:09] ...optimizers/trijoint.py.151: total_norm: 40.402664 [I 2022-10-27 18:02:14] ...trap/engines/engine.py.224: train: epoch 80 | batch 2030/2304 [I 2022-10-27 18:02:14] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:30 [I 2022-10-27 18:02:14] ...trap/engines/engine.py.229: process: 0.54246 | load: 0.00243 [I 2022-10-27 18:02:14] ...trap/engines/engine.py.230: loss: 0.10339 [I 2022-10-27 18:02:14] ...optimizers/trijoint.py.151: total_norm: 14.653552 [I 2022-10-27 18:02:20] ...trap/engines/engine.py.224: train: epoch 80 | batch 2040/2304 [I 2022-10-27 18:02:20] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:26 [I 2022-10-27 18:02:20] ...trap/engines/engine.py.229: process: 0.54740 | load: 0.00245 [I 2022-10-27 18:02:20] ...trap/engines/engine.py.230: loss: 0.14271 [I 2022-10-27 18:02:20] ...optimizers/trijoint.py.151: total_norm: 12.799101 [I 2022-10-27 18:02:25] ...trap/engines/engine.py.224: train: epoch 80 | batch 2050/2304 [I 2022-10-27 18:02:25] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:21 [I 2022-10-27 18:02:25] ...trap/engines/engine.py.229: process: 0.55383 | load: 0.00264 [I 2022-10-27 18:02:25] ...trap/engines/engine.py.230: loss: 0.13322 [I 2022-10-27 18:02:25] ...optimizers/trijoint.py.151: total_norm: 20.132437 [I 2022-10-27 18:02:31] ...trap/engines/engine.py.224: train: epoch 80 | batch 2060/2304 [I 2022-10-27 18:02:31] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:15 [I 2022-10-27 18:02:31] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00244 [I 2022-10-27 18:02:31] ...trap/engines/engine.py.230: loss: 0.10978 [I 2022-10-27 18:02:31] ...optimizers/trijoint.py.151: total_norm: 14.396068 [I 2022-10-27 18:02:36] ...trap/engines/engine.py.224: train: epoch 80 | batch 2070/2304 [I 2022-10-27 18:02:36] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:09 [I 2022-10-27 18:02:36] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00241 [I 2022-10-27 18:02:36] ...trap/engines/engine.py.230: loss: 0.12548 [I 2022-10-27 18:02:36] ...optimizers/trijoint.py.151: total_norm: 32.266701 [I 2022-10-27 18:02:42] ...trap/engines/engine.py.224: train: epoch 80 | batch 2080/2304 [I 2022-10-27 18:02:42] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:05 [I 2022-10-27 18:02:42] ...trap/engines/engine.py.229: process: 0.56750 | load: 0.00246 [I 2022-10-27 18:02:42] ...trap/engines/engine.py.230: loss: 0.11828 [I 2022-10-27 18:02:42] ...optimizers/trijoint.py.151: total_norm: 10.486360 [I 2022-10-27 18:02:48] ...trap/engines/engine.py.224: train: epoch 80 | batch 2090/2304 [I 2022-10-27 18:02:48] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:00 [I 2022-10-27 18:02:48] ...trap/engines/engine.py.229: process: 0.56753 | load: 0.00250 [I 2022-10-27 18:02:48] ...trap/engines/engine.py.230: loss: 0.23527 [I 2022-10-27 18:02:48] ...optimizers/trijoint.py.151: total_norm: 21.531849 [I 2022-10-27 18:02:53] ...trap/engines/engine.py.224: train: epoch 80 | batch 2100/2304 [I 2022-10-27 18:02:53] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:01:53 [I 2022-10-27 18:02:53] ...trap/engines/engine.py.229: process: 0.54995 | load: 0.00248 [I 2022-10-27 18:02:53] ...trap/engines/engine.py.230: loss: 0.12325 [I 2022-10-27 18:02:53] ...optimizers/trijoint.py.151: total_norm: 10.081382 [I 2022-10-27 18:02:59] ...trap/engines/engine.py.224: train: epoch 80 | batch 2110/2304 [I 2022-10-27 18:02:59] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:48 [I 2022-10-27 18:02:59] ...trap/engines/engine.py.229: process: 0.56503 | load: 0.00246 [I 2022-10-27 18:02:59] ...trap/engines/engine.py.230: loss: 0.10032 [I 2022-10-27 18:02:59] ...optimizers/trijoint.py.151: total_norm: 12.372131 [I 2022-10-27 18:03:04] ...trap/engines/engine.py.224: train: epoch 80 | batch 2120/2304 [I 2022-10-27 18:03:04] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:41 [I 2022-10-27 18:03:04] ...trap/engines/engine.py.229: process: 0.54564 | load: 0.00245 [I 2022-10-27 18:03:04] ...trap/engines/engine.py.230: loss: 0.12283 [I 2022-10-27 18:03:04] ...optimizers/trijoint.py.151: total_norm: 14.339013 [I 2022-10-27 18:03:10] ...trap/engines/engine.py.224: train: epoch 80 | batch 2130/2304 [I 2022-10-27 18:03:10] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:36 [I 2022-10-27 18:03:10] ...trap/engines/engine.py.229: process: 0.54273 | load: 0.00243 [I 2022-10-27 18:03:10] ...trap/engines/engine.py.230: loss: 0.13035 [I 2022-10-27 18:03:10] ...optimizers/trijoint.py.151: total_norm: 27.621391 [I 2022-10-27 18:03:15] ...trap/engines/engine.py.224: train: epoch 80 | batch 2140/2304 [I 2022-10-27 18:03:15] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:31 [I 2022-10-27 18:03:15] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00241 [I 2022-10-27 18:03:15] ...trap/engines/engine.py.230: loss: 0.10645 [I 2022-10-27 18:03:15] ...optimizers/trijoint.py.151: total_norm: 20.475605 [I 2022-10-27 18:03:39] ...trap/engines/engine.py.224: train: epoch 80 | batch 2150/2304 [I 2022-10-27 18:03:39] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:06:13 [I 2022-10-27 18:03:39] ...trap/engines/engine.py.229: process: 0.58624 | load: 0.00236 [I 2022-10-27 18:03:39] ...trap/engines/engine.py.230: loss: 0.19023 [I 2022-10-27 18:03:39] ...optimizers/trijoint.py.151: total_norm: 23.358734 [I 2022-10-27 18:03:45] ...trap/engines/engine.py.224: train: epoch 80 | batch 2160/2304 [I 2022-10-27 18:03:45] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:01:48 [I 2022-10-27 18:03:45] ...trap/engines/engine.py.229: process: 0.54437 | load: 0.00236 [I 2022-10-27 18:03:45] ...trap/engines/engine.py.230: loss: 0.12056 [I 2022-10-27 18:03:45] ...optimizers/trijoint.py.151: total_norm: 23.874157 [I 2022-10-27 18:03:50] ...trap/engines/engine.py.224: train: epoch 80 | batch 2170/2304 [I 2022-10-27 18:03:50] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:17 [I 2022-10-27 18:03:50] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00237 [I 2022-10-27 18:03:50] ...trap/engines/engine.py.230: loss: 0.14177 [I 2022-10-27 18:03:50] ...optimizers/trijoint.py.151: total_norm: 21.395037 [I 2022-10-27 18:03:56] ...trap/engines/engine.py.224: train: epoch 80 | batch 2180/2304 [I 2022-10-27 18:03:56] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:01:09 [I 2022-10-27 18:03:56] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00248 [I 2022-10-27 18:03:56] ...trap/engines/engine.py.230: loss: 0.16584 [I 2022-10-27 18:03:56] ...optimizers/trijoint.py.151: total_norm: 30.676233 [I 2022-10-27 18:04:02] ...trap/engines/engine.py.224: train: epoch 80 | batch 2190/2304 [I 2022-10-27 18:04:02] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:01:03 [I 2022-10-27 18:04:02] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00244 [I 2022-10-27 18:04:02] ...trap/engines/engine.py.230: loss: 0.17235 [I 2022-10-27 18:04:02] ...optimizers/trijoint.py.151: total_norm: 13.283071 [I 2022-10-27 18:04:07] ...trap/engines/engine.py.224: train: epoch 80 | batch 2200/2304 [I 2022-10-27 18:04:07] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:57 [I 2022-10-27 18:04:07] ...trap/engines/engine.py.229: process: 0.54539 | load: 0.00241 [I 2022-10-27 18:04:07] ...trap/engines/engine.py.230: loss: 0.14576 [I 2022-10-27 18:04:07] ...optimizers/trijoint.py.151: total_norm: 13.177501 [I 2022-10-27 18:04:13] ...trap/engines/engine.py.224: train: epoch 80 | batch 2210/2304 [I 2022-10-27 18:04:13] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:52 [I 2022-10-27 18:04:13] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00237 [I 2022-10-27 18:04:13] ...trap/engines/engine.py.230: loss: 0.14959 [I 2022-10-27 18:04:13] ...optimizers/trijoint.py.151: total_norm: 16.015610 [I 2022-10-27 18:04:18] ...trap/engines/engine.py.224: train: epoch 80 | batch 2220/2304 [I 2022-10-27 18:04:18] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:46 [I 2022-10-27 18:04:18] ...trap/engines/engine.py.229: process: 0.55243 | load: 0.00243 [I 2022-10-27 18:04:18] ...trap/engines/engine.py.230: loss: 0.12885 [I 2022-10-27 18:04:18] ...optimizers/trijoint.py.151: total_norm: 17.903086 [I 2022-10-27 18:04:24] ...trap/engines/engine.py.224: train: epoch 80 | batch 2230/2304 [I 2022-10-27 18:04:24] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:41 [I 2022-10-27 18:04:24] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00244 [I 2022-10-27 18:04:24] ...trap/engines/engine.py.230: loss: 0.17598 [I 2022-10-27 18:04:24] ...optimizers/trijoint.py.151: total_norm: 11.542959 [I 2022-10-27 18:04:29] ...trap/engines/engine.py.224: train: epoch 80 | batch 2240/2304 [I 2022-10-27 18:04:29] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:35 [I 2022-10-27 18:04:29] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00244 [I 2022-10-27 18:04:29] ...trap/engines/engine.py.230: loss: 0.10986 [I 2022-10-27 18:04:29] ...optimizers/trijoint.py.151: total_norm: 7.864625 [I 2022-10-27 18:04:35] ...trap/engines/engine.py.224: train: epoch 80 | batch 2250/2304 [I 2022-10-27 18:04:35] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:30 [I 2022-10-27 18:04:35] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00255 [I 2022-10-27 18:04:35] ...trap/engines/engine.py.230: loss: 0.11902 [I 2022-10-27 18:04:35] ...optimizers/trijoint.py.151: total_norm: 10.189923 [I 2022-10-27 18:04:41] ...trap/engines/engine.py.224: train: epoch 80 | batch 2260/2304 [I 2022-10-27 18:04:41] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:24 [I 2022-10-27 18:04:41] ...trap/engines/engine.py.229: process: 0.57420 | load: 0.00246 [I 2022-10-27 18:04:41] ...trap/engines/engine.py.230: loss: 0.09572 [I 2022-10-27 18:04:41] ...optimizers/trijoint.py.151: total_norm: 8.938498 [I 2022-10-27 18:04:46] ...trap/engines/engine.py.224: train: epoch 80 | batch 2270/2304 [I 2022-10-27 18:04:46] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:19 [I 2022-10-27 18:04:46] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00246 [I 2022-10-27 18:04:46] ...trap/engines/engine.py.230: loss: 0.19469 [I 2022-10-27 18:04:46] ...optimizers/trijoint.py.151: total_norm: 16.301474 [I 2022-10-27 18:04:52] ...trap/engines/engine.py.224: train: epoch 80 | batch 2280/2304 [I 2022-10-27 18:04:52] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:00:13 [I 2022-10-27 18:04:52] ...trap/engines/engine.py.229: process: 0.56539 | load: 0.00245 [I 2022-10-27 18:04:52] ...trap/engines/engine.py.230: loss: 0.11027 [I 2022-10-27 18:04:52] ...optimizers/trijoint.py.151: total_norm: 14.785731 [I 2022-10-27 18:04:58] ...trap/engines/engine.py.224: train: epoch 80 | batch 2290/2304 [I 2022-10-27 18:04:58] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:00:07 [I 2022-10-27 18:04:58] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00242 [I 2022-10-27 18:04:58] ...trap/engines/engine.py.230: loss: 0.14777 [I 2022-10-27 18:04:58] ...optimizers/trijoint.py.151: total_norm: 24.267334 [I 2022-10-27 18:05:03] ...trap/engines/engine.py.224: train: epoch 80 | batch 2300/2304 [I 2022-10-27 18:05:03] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:02 [I 2022-10-27 18:05:03] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00016 [I 2022-10-27 18:05:03] ...trap/engines/engine.py.230: loss: 0.11683 [I 2022-10-27 18:05:03] ...optimizers/trijoint.py.151: total_norm: 19.510260 [I 2022-10-27 18:05:05] ...trap/engines/engine.py.224: train: epoch 80 | batch 2304/2304 [I 2022-10-27 18:05:05] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:00 [I 2022-10-27 18:05:05] ...trap/engines/engine.py.229: process: 0.55258 | load: 0.00017 [I 2022-10-27 18:05:05] ...trap/engines/engine.py.230: loss: 0.15413 [I 2022-10-27 18:05:05] ...optimizers/trijoint.py.151: total_norm: 11.129098 [S 2022-10-27 18:05:07] ...trap/engines/engine.py.129: train_epoch.epoch: 80 [S 2022-10-27 18:05:07] ...trap/engines/engine.py.129: train_epoch.loss: 0.1371983825481636 [I 2022-10-27 18:05:07] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 18:05:08] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 80 [I 2022-10-27 18:05:08] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 18:05:10] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 18:05:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 18:05:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 18:05:12] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 18:05:13] ...trap/engines/engine.py.321: eval: epoch 80 | batch 0/510 [I 2022-10-27 18:05:13] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:41:56 [I 2022-10-27 18:05:13] ...trap/engines/engine.py.326: process: 4.93379 | load: 2.93318 [I 2022-10-27 18:05:16] ...trap/engines/engine.py.321: eval: epoch 80 | batch 10/510 [I 2022-10-27 18:05:16] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:55 [I 2022-10-27 18:05:16] ...trap/engines/engine.py.326: process: 0.33586 | load: 0.00017 [I 2022-10-27 18:05:19] ...trap/engines/engine.py.321: eval: epoch 80 | batch 20/510 [I 2022-10-27 18:05:19] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:07 [I 2022-10-27 18:05:19] ...trap/engines/engine.py.326: process: 0.32877 | load: 0.00020 [I 2022-10-27 18:05:23] ...trap/engines/engine.py.321: eval: epoch 80 | batch 30/510 [I 2022-10-27 18:05:23] ...trap/engines/engine.py.325: elapsed: 0:00:14 | left: 0:02:40 [I 2022-10-27 18:05:23] ...trap/engines/engine.py.326: process: 0.32681 | load: 0.00017 [I 2022-10-27 18:05:26] ...trap/engines/engine.py.321: eval: epoch 80 | batch 40/510 [I 2022-10-27 18:05:26] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:34 [I 2022-10-27 18:05:26] ...trap/engines/engine.py.326: process: 0.32184 | load: 0.00018 [I 2022-10-27 18:05:29] ...trap/engines/engine.py.321: eval: epoch 80 | batch 50/510 [I 2022-10-27 18:05:29] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:31 [I 2022-10-27 18:05:29] ...trap/engines/engine.py.326: process: 0.32992 | load: 0.00016 [I 2022-10-27 18:05:33] ...trap/engines/engine.py.321: eval: epoch 80 | batch 60/510 [I 2022-10-27 18:05:33] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:29 [I 2022-10-27 18:05:33] ...trap/engines/engine.py.326: process: 0.33072 | load: 0.00018 [I 2022-10-27 18:05:36] ...trap/engines/engine.py.321: eval: epoch 80 | batch 70/510 [I 2022-10-27 18:05:36] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:26 [I 2022-10-27 18:05:36] ...trap/engines/engine.py.326: process: 0.32025 | load: 0.00017 [I 2022-10-27 18:05:39] ...trap/engines/engine.py.321: eval: epoch 80 | batch 80/510 [I 2022-10-27 18:05:39] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:21 [I 2022-10-27 18:05:39] ...trap/engines/engine.py.326: process: 0.31946 | load: 0.00018 [I 2022-10-27 18:05:43] ...trap/engines/engine.py.321: eval: epoch 80 | batch 90/510 [I 2022-10-27 18:05:43] ...trap/engines/engine.py.325: elapsed: 0:00:34 | left: 0:02:21 [I 2022-10-27 18:05:43] ...trap/engines/engine.py.326: process: 0.32933 | load: 0.00019 [I 2022-10-27 18:05:46] ...trap/engines/engine.py.321: eval: epoch 80 | batch 100/510 [I 2022-10-27 18:05:46] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-27 18:05:46] ...trap/engines/engine.py.326: process: 0.28751 | load: 0.00025 [I 2022-10-27 18:05:49] ...trap/engines/engine.py.321: eval: epoch 80 | batch 110/510 [I 2022-10-27 18:05:49] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-27 18:05:49] ...trap/engines/engine.py.326: process: 0.28610 | load: 0.00014 [I 2022-10-27 18:05:52] ...trap/engines/engine.py.321: eval: epoch 80 | batch 120/510 [I 2022-10-27 18:05:52] ...trap/engines/engine.py.325: elapsed: 0:00:43 | left: 0:01:54 [I 2022-10-27 18:05:52] ...trap/engines/engine.py.326: process: 0.30989 | load: 0.00015 [I 2022-10-27 18:05:55] ...trap/engines/engine.py.321: eval: epoch 80 | batch 130/510 [I 2022-10-27 18:05:55] ...trap/engines/engine.py.325: elapsed: 0:00:46 | left: 0:01:49 [I 2022-10-27 18:05:55] ...trap/engines/engine.py.326: process: 0.28391 | load: 0.00020 [I 2022-10-27 18:05:58] ...trap/engines/engine.py.321: eval: epoch 80 | batch 140/510 [I 2022-10-27 18:05:58] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:48 [I 2022-10-27 18:05:58] ...trap/engines/engine.py.326: process: 0.28713 | load: 0.00024 [I 2022-10-27 18:06:00] ...trap/engines/engine.py.321: eval: epoch 80 | batch 150/510 [I 2022-10-27 18:06:00] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 18:06:00] ...trap/engines/engine.py.326: process: 0.28385 | load: 0.00019 [I 2022-10-27 18:06:03] ...trap/engines/engine.py.321: eval: epoch 80 | batch 160/510 [I 2022-10-27 18:06:03] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 18:06:03] ...trap/engines/engine.py.326: process: 0.28242 | load: 0.00015 [I 2022-10-27 18:06:06] ...trap/engines/engine.py.321: eval: epoch 80 | batch 170/510 [I 2022-10-27 18:06:06] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 18:06:06] ...trap/engines/engine.py.326: process: 0.28798 | load: 0.00018 [I 2022-10-27 18:06:09] ...trap/engines/engine.py.321: eval: epoch 80 | batch 180/510 [I 2022-10-27 18:06:09] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 18:06:09] ...trap/engines/engine.py.326: process: 0.28815 | load: 0.00024 [I 2022-10-27 18:06:12] ...trap/engines/engine.py.321: eval: epoch 80 | batch 190/510 [I 2022-10-27 18:06:12] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 18:06:12] ...trap/engines/engine.py.326: process: 0.29525 | load: 0.00014 [I 2022-10-27 18:06:15] ...trap/engines/engine.py.321: eval: epoch 80 | batch 200/510 [I 2022-10-27 18:06:15] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:28 [I 2022-10-27 18:06:15] ...trap/engines/engine.py.326: process: 0.28134 | load: 0.00015 [I 2022-10-27 18:06:18] ...trap/engines/engine.py.321: eval: epoch 80 | batch 210/510 [I 2022-10-27 18:06:18] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 18:06:18] ...trap/engines/engine.py.326: process: 0.28458 | load: 0.00015 [I 2022-10-27 18:06:21] ...trap/engines/engine.py.321: eval: epoch 80 | batch 220/510 [I 2022-10-27 18:06:21] ...trap/engines/engine.py.325: elapsed: 0:01:12 | left: 0:01:23 [I 2022-10-27 18:06:21] ...trap/engines/engine.py.326: process: 0.29037 | load: 0.00021 [I 2022-10-27 18:06:24] ...trap/engines/engine.py.321: eval: epoch 80 | batch 230/510 [I 2022-10-27 18:06:24] ...trap/engines/engine.py.325: elapsed: 0:01:15 | left: 0:01:22 [I 2022-10-27 18:06:24] ...trap/engines/engine.py.326: process: 0.29016 | load: 0.00015 [I 2022-10-27 18:06:27] ...trap/engines/engine.py.321: eval: epoch 80 | batch 240/510 [I 2022-10-27 18:06:27] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 18:06:27] ...trap/engines/engine.py.326: process: 0.28612 | load: 0.00019 [I 2022-10-27 18:06:29] ...trap/engines/engine.py.321: eval: epoch 80 | batch 250/510 [I 2022-10-27 18:06:29] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 18:06:29] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00015 [I 2022-10-27 18:06:32] ...trap/engines/engine.py.321: eval: epoch 80 | batch 260/510 [I 2022-10-27 18:06:32] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:15 [I 2022-10-27 18:06:32] ...trap/engines/engine.py.326: process: 0.28911 | load: 0.00016 [I 2022-10-27 18:06:35] ...trap/engines/engine.py.321: eval: epoch 80 | batch 270/510 [I 2022-10-27 18:06:35] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 18:06:35] ...trap/engines/engine.py.326: process: 0.29181 | load: 0.00016 [I 2022-10-27 18:06:38] ...trap/engines/engine.py.321: eval: epoch 80 | batch 280/510 [I 2022-10-27 18:06:38] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 18:06:38] ...trap/engines/engine.py.326: process: 0.29019 | load: 0.00022 [I 2022-10-27 18:06:41] ...trap/engines/engine.py.321: eval: epoch 80 | batch 290/510 [I 2022-10-27 18:06:41] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 18:06:41] ...trap/engines/engine.py.326: process: 0.28594 | load: 0.00015 [I 2022-10-27 18:06:44] ...trap/engines/engine.py.321: eval: epoch 80 | batch 300/510 [I 2022-10-27 18:06:44] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 18:06:44] ...trap/engines/engine.py.326: process: 0.29464 | load: 0.00019 [I 2022-10-27 18:06:47] ...trap/engines/engine.py.321: eval: epoch 80 | batch 310/510 [I 2022-10-27 18:06:47] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 18:06:47] ...trap/engines/engine.py.326: process: 0.28567 | load: 0.00016 [I 2022-10-27 18:06:50] ...trap/engines/engine.py.321: eval: epoch 80 | batch 320/510 [I 2022-10-27 18:06:50] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 18:06:50] ...trap/engines/engine.py.326: process: 0.29092 | load: 0.00020 [I 2022-10-27 18:06:53] ...trap/engines/engine.py.321: eval: epoch 80 | batch 330/510 [I 2022-10-27 18:06:53] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 18:06:53] ...trap/engines/engine.py.326: process: 0.29028 | load: 0.00023 [I 2022-10-27 18:06:55] ...trap/engines/engine.py.321: eval: epoch 80 | batch 340/510 [I 2022-10-27 18:06:55] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 18:06:55] ...trap/engines/engine.py.326: process: 0.28887 | load: 0.00023 [I 2022-10-27 18:06:58] ...trap/engines/engine.py.321: eval: epoch 80 | batch 350/510 [I 2022-10-27 18:06:58] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 18:06:58] ...trap/engines/engine.py.326: process: 0.28705 | load: 0.00020 [I 2022-10-27 18:07:01] ...trap/engines/engine.py.321: eval: epoch 80 | batch 360/510 [I 2022-10-27 18:07:01] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 18:07:01] ...trap/engines/engine.py.326: process: 0.29116 | load: 0.00019 [I 2022-10-27 18:07:04] ...trap/engines/engine.py.321: eval: epoch 80 | batch 370/510 [I 2022-10-27 18:07:04] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 18:07:04] ...trap/engines/engine.py.326: process: 0.28568 | load: 0.00019 [I 2022-10-27 18:07:07] ...trap/engines/engine.py.321: eval: epoch 80 | batch 380/510 [I 2022-10-27 18:07:07] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:38 [I 2022-10-27 18:07:07] ...trap/engines/engine.py.326: process: 0.28754 | load: 0.00021 [I 2022-10-27 18:07:10] ...trap/engines/engine.py.321: eval: epoch 80 | batch 390/510 [I 2022-10-27 18:07:10] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 18:07:10] ...trap/engines/engine.py.326: process: 0.28319 | load: 0.00015 [I 2022-10-27 18:07:13] ...trap/engines/engine.py.321: eval: epoch 80 | batch 400/510 [I 2022-10-27 18:07:13] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 18:07:13] ...trap/engines/engine.py.326: process: 0.29007 | load: 0.00027 [I 2022-10-27 18:07:16] ...trap/engines/engine.py.321: eval: epoch 80 | batch 410/510 [I 2022-10-27 18:07:16] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:29 [I 2022-10-27 18:07:16] ...trap/engines/engine.py.326: process: 0.29104 | load: 0.00015 [I 2022-10-27 18:07:19] ...trap/engines/engine.py.321: eval: epoch 80 | batch 420/510 [I 2022-10-27 18:07:19] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-27 18:07:19] ...trap/engines/engine.py.326: process: 0.28744 | load: 0.00015 [I 2022-10-27 18:07:22] ...trap/engines/engine.py.321: eval: epoch 80 | batch 430/510 [I 2022-10-27 18:07:22] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 18:07:22] ...trap/engines/engine.py.326: process: 0.29192 | load: 0.00016 [I 2022-10-27 18:07:24] ...trap/engines/engine.py.321: eval: epoch 80 | batch 440/510 [I 2022-10-27 18:07:24] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 18:07:24] ...trap/engines/engine.py.326: process: 0.29080 | load: 0.00017 [I 2022-10-27 18:07:27] ...trap/engines/engine.py.321: eval: epoch 80 | batch 450/510 [I 2022-10-27 18:07:27] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 18:07:27] ...trap/engines/engine.py.326: process: 0.28912 | load: 0.00017 [I 2022-10-27 18:07:30] ...trap/engines/engine.py.321: eval: epoch 80 | batch 460/510 [I 2022-10-27 18:07:30] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 18:07:30] ...trap/engines/engine.py.326: process: 0.28888 | load: 0.00017 [I 2022-10-27 18:07:33] ...trap/engines/engine.py.321: eval: epoch 80 | batch 470/510 [I 2022-10-27 18:07:33] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 18:07:33] ...trap/engines/engine.py.326: process: 0.28886 | load: 0.00020 [I 2022-10-27 18:07:36] ...trap/engines/engine.py.321: eval: epoch 80 | batch 480/510 [I 2022-10-27 18:07:36] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 18:07:36] ...trap/engines/engine.py.326: process: 0.29491 | load: 0.00019 [I 2022-10-27 18:07:39] ...trap/engines/engine.py.321: eval: epoch 80 | batch 490/510 [I 2022-10-27 18:07:39] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 18:07:39] ...trap/engines/engine.py.326: process: 0.28489 | load: 0.00021 [I 2022-10-27 18:07:42] ...trap/engines/engine.py.321: eval: epoch 80 | batch 500/510 [I 2022-10-27 18:07:42] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 18:07:42] ...trap/engines/engine.py.326: process: 0.29292 | load: 0.00039 [I 2022-10-27 18:07:45] ...trap/engines/engine.py.321: eval: epoch 80 | batch 510/510 [I 2022-10-27 18:07:45] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 18:07:45] ...trap/engines/engine.py.326: process: 0.28495 | load: 0.00011 [S 2022-10-27 18:07:45] ...trap/engines/engine.py.132: eval_epoch.epoch: 80 [I 2022-10-27 18:07:45] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.726700 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907000 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.939000 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.716000 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.903300 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936800 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012570 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011874 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010050 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012385 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011472 [S 2022-10-27 18:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009673 [I 2022-10-27 18:07:51] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 18:07:51] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 18:07:52] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 18:07:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 18:07:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 18:07:56] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 18:07:56] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 18:08:02] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 18:08:02] ...trap/engines/engine.py.163: Training model on trainset for epoch 81 [I 2022-10-27 18:08:02] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 18:08:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 0/2304 [I 2022-10-27 18:08:05] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:46 [I 2022-10-27 18:08:05] ...trap/engines/engine.py.229: process: 2.65047 | load: 1.99068 [I 2022-10-27 18:08:05] ...trap/engines/engine.py.230: loss: 0.14007 [I 2022-10-27 18:08:05] ...optimizers/trijoint.py.151: total_norm: 14.284714 [I 2022-10-27 18:08:11] ...trap/engines/engine.py.224: train: epoch 81 | batch 10/2304 [I 2022-10-27 18:08:11] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:47 [I 2022-10-27 18:08:11] ...trap/engines/engine.py.229: process: 0.55043 | load: 0.00354 [I 2022-10-27 18:08:11] ...trap/engines/engine.py.230: loss: 0.12429 [I 2022-10-27 18:08:11] ...optimizers/trijoint.py.151: total_norm: 17.851059 [I 2022-10-27 18:08:16] ...trap/engines/engine.py.224: train: epoch 81 | batch 20/2304 [I 2022-10-27 18:08:16] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:12 [I 2022-10-27 18:08:16] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00357 [I 2022-10-27 18:08:16] ...trap/engines/engine.py.230: loss: 0.21078 [I 2022-10-27 18:08:16] ...optimizers/trijoint.py.151: total_norm: 21.833139 [I 2022-10-27 18:08:22] ...trap/engines/engine.py.224: train: epoch 81 | batch 30/2304 [I 2022-10-27 18:08:22] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:17 [I 2022-10-27 18:08:22] ...trap/engines/engine.py.229: process: 0.55851 | load: 0.00349 [I 2022-10-27 18:08:22] ...trap/engines/engine.py.230: loss: 0.12381 [I 2022-10-27 18:08:22] ...optimizers/trijoint.py.151: total_norm: 11.655913 [I 2022-10-27 18:08:27] ...trap/engines/engine.py.224: train: epoch 81 | batch 40/2304 [I 2022-10-27 18:08:27] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:27 [I 2022-10-27 18:08:27] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00337 [I 2022-10-27 18:08:27] ...trap/engines/engine.py.230: loss: 0.11311 [I 2022-10-27 18:08:27] ...optimizers/trijoint.py.151: total_norm: 14.012767 [I 2022-10-27 18:08:33] ...trap/engines/engine.py.224: train: epoch 81 | batch 50/2304 [I 2022-10-27 18:08:33] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:51 [I 2022-10-27 18:08:33] ...trap/engines/engine.py.229: process: 0.55175 | load: 0.00296 [I 2022-10-27 18:08:33] ...trap/engines/engine.py.230: loss: 0.10646 [I 2022-10-27 18:08:33] ...optimizers/trijoint.py.151: total_norm: 11.298786 [I 2022-10-27 18:08:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 60/2304 [I 2022-10-27 18:08:39] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:39 [I 2022-10-27 18:08:39] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00309 [I 2022-10-27 18:08:39] ...trap/engines/engine.py.230: loss: 0.14508 [I 2022-10-27 18:08:39] ...optimizers/trijoint.py.151: total_norm: 20.710266 [I 2022-10-27 18:08:44] ...trap/engines/engine.py.224: train: epoch 81 | batch 70/2304 [I 2022-10-27 18:08:44] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:34 [I 2022-10-27 18:08:44] ...trap/engines/engine.py.229: process: 0.54759 | load: 0.00303 [I 2022-10-27 18:08:44] ...trap/engines/engine.py.230: loss: 0.10679 [I 2022-10-27 18:08:44] ...optimizers/trijoint.py.151: total_norm: 7.996462 [I 2022-10-27 18:08:50] ...trap/engines/engine.py.224: train: epoch 81 | batch 80/2304 [I 2022-10-27 18:08:50] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:33 [I 2022-10-27 18:08:50] ...trap/engines/engine.py.229: process: 0.55046 | load: 0.00249 [I 2022-10-27 18:08:50] ...trap/engines/engine.py.230: loss: 0.12246 [I 2022-10-27 18:08:50] ...optimizers/trijoint.py.151: total_norm: 14.927109 [I 2022-10-27 18:08:55] ...trap/engines/engine.py.224: train: epoch 81 | batch 90/2304 [I 2022-10-27 18:08:55] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:29 [I 2022-10-27 18:08:55] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00267 [I 2022-10-27 18:08:55] ...trap/engines/engine.py.230: loss: 0.18083 [I 2022-10-27 18:08:55] ...optimizers/trijoint.py.151: total_norm: 24.804888 [I 2022-10-27 18:09:01] ...trap/engines/engine.py.224: train: epoch 81 | batch 100/2304 [I 2022-10-27 18:09:01] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:29 [I 2022-10-27 18:09:01] ...trap/engines/engine.py.229: process: 0.55325 | load: 0.00256 [I 2022-10-27 18:09:01] ...trap/engines/engine.py.230: loss: 0.14580 [I 2022-10-27 18:09:01] ...optimizers/trijoint.py.151: total_norm: 13.273913 [I 2022-10-27 18:09:07] ...trap/engines/engine.py.224: train: epoch 81 | batch 110/2304 [I 2022-10-27 18:09:07] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:23 [I 2022-10-27 18:09:07] ...trap/engines/engine.py.229: process: 0.54080 | load: 0.00266 [I 2022-10-27 18:09:07] ...trap/engines/engine.py.230: loss: 0.10423 [I 2022-10-27 18:09:07] ...optimizers/trijoint.py.151: total_norm: 9.056592 [I 2022-10-27 18:09:12] ...trap/engines/engine.py.224: train: epoch 81 | batch 120/2304 [I 2022-10-27 18:09:12] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:21 [I 2022-10-27 18:09:12] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00293 [I 2022-10-27 18:09:12] ...trap/engines/engine.py.230: loss: 0.10513 [I 2022-10-27 18:09:12] ...optimizers/trijoint.py.151: total_norm: 12.669455 [I 2022-10-27 18:09:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 130/2304 [I 2022-10-27 18:09:18] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:11 [I 2022-10-27 18:09:18] ...trap/engines/engine.py.229: process: 0.56296 | load: 0.00273 [I 2022-10-27 18:09:18] ...trap/engines/engine.py.230: loss: 0.11886 [I 2022-10-27 18:09:18] ...optimizers/trijoint.py.151: total_norm: 15.789172 [I 2022-10-27 18:09:23] ...trap/engines/engine.py.224: train: epoch 81 | batch 140/2304 [I 2022-10-27 18:09:23] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:01 [I 2022-10-27 18:09:23] ...trap/engines/engine.py.229: process: 0.55211 | load: 0.00277 [I 2022-10-27 18:09:23] ...trap/engines/engine.py.230: loss: 0.11484 [I 2022-10-27 18:09:23] ...optimizers/trijoint.py.151: total_norm: 17.073666 [I 2022-10-27 18:09:29] ...trap/engines/engine.py.224: train: epoch 81 | batch 150/2304 [I 2022-10-27 18:09:29] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:01 [I 2022-10-27 18:09:29] ...trap/engines/engine.py.229: process: 0.57111 | load: 0.00265 [I 2022-10-27 18:09:29] ...trap/engines/engine.py.230: loss: 0.22250 [I 2022-10-27 18:09:29] ...optimizers/trijoint.py.151: total_norm: 10.726873 [I 2022-10-27 18:09:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 160/2304 [I 2022-10-27 18:09:34] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:55 [I 2022-10-27 18:09:34] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00241 [I 2022-10-27 18:09:34] ...trap/engines/engine.py.230: loss: 0.11243 [I 2022-10-27 18:09:34] ...optimizers/trijoint.py.151: total_norm: 11.092561 [I 2022-10-27 18:09:40] ...trap/engines/engine.py.224: train: epoch 81 | batch 170/2304 [I 2022-10-27 18:09:40] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:50 [I 2022-10-27 18:09:40] ...trap/engines/engine.py.229: process: 0.54485 | load: 0.00263 [I 2022-10-27 18:09:40] ...trap/engines/engine.py.230: loss: 0.11076 [I 2022-10-27 18:09:40] ...optimizers/trijoint.py.151: total_norm: 30.912231 [I 2022-10-27 18:09:46] ...trap/engines/engine.py.224: train: epoch 81 | batch 180/2304 [I 2022-10-27 18:09:46] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:39 [I 2022-10-27 18:09:46] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00251 [I 2022-10-27 18:09:46] ...trap/engines/engine.py.230: loss: 0.14844 [I 2022-10-27 18:09:46] ...optimizers/trijoint.py.151: total_norm: 18.308968 [I 2022-10-27 18:09:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 190/2304 [I 2022-10-27 18:09:51] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:32 [I 2022-10-27 18:09:51] ...trap/engines/engine.py.229: process: 0.55391 | load: 0.00265 [I 2022-10-27 18:09:51] ...trap/engines/engine.py.230: loss: 0.11493 [I 2022-10-27 18:09:51] ...optimizers/trijoint.py.151: total_norm: 11.240256 [I 2022-10-27 18:09:57] ...trap/engines/engine.py.224: train: epoch 81 | batch 200/2304 [I 2022-10-27 18:09:57] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:30 [I 2022-10-27 18:09:57] ...trap/engines/engine.py.229: process: 0.55672 | load: 0.00255 [I 2022-10-27 18:09:57] ...trap/engines/engine.py.230: loss: 0.09577 [I 2022-10-27 18:09:57] ...optimizers/trijoint.py.151: total_norm: 10.783785 [I 2022-10-27 18:10:02] ...trap/engines/engine.py.224: train: epoch 81 | batch 210/2304 [I 2022-10-27 18:10:02] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:29 [I 2022-10-27 18:10:02] ...trap/engines/engine.py.229: process: 0.56764 | load: 0.00241 [I 2022-10-27 18:10:02] ...trap/engines/engine.py.230: loss: 0.11219 [I 2022-10-27 18:10:02] ...optimizers/trijoint.py.151: total_norm: 12.306091 [I 2022-10-27 18:10:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 220/2304 [I 2022-10-27 18:10:08] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:15 [I 2022-10-27 18:10:08] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00248 [I 2022-10-27 18:10:08] ...trap/engines/engine.py.230: loss: 0.12170 [I 2022-10-27 18:10:08] ...optimizers/trijoint.py.151: total_norm: 15.488154 [I 2022-10-27 18:10:13] ...trap/engines/engine.py.224: train: epoch 81 | batch 230/2304 [I 2022-10-27 18:10:13] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:40 [I 2022-10-27 18:10:13] ...trap/engines/engine.py.229: process: 0.56907 | load: 0.00257 [I 2022-10-27 18:10:13] ...trap/engines/engine.py.230: loss: 0.10164 [I 2022-10-27 18:10:13] ...optimizers/trijoint.py.151: total_norm: 10.558360 [I 2022-10-27 18:10:19] ...trap/engines/engine.py.224: train: epoch 81 | batch 240/2304 [I 2022-10-27 18:10:19] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:07 [I 2022-10-27 18:10:19] ...trap/engines/engine.py.229: process: 0.54547 | load: 0.00247 [I 2022-10-27 18:10:19] ...trap/engines/engine.py.230: loss: 0.15111 [I 2022-10-27 18:10:19] ...optimizers/trijoint.py.151: total_norm: 33.995712 [I 2022-10-27 18:10:25] ...trap/engines/engine.py.224: train: epoch 81 | batch 250/2304 [I 2022-10-27 18:10:25] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:13 [I 2022-10-27 18:10:25] ...trap/engines/engine.py.229: process: 0.56988 | load: 0.00253 [I 2022-10-27 18:10:25] ...trap/engines/engine.py.230: loss: 0.13060 [I 2022-10-27 18:10:25] ...optimizers/trijoint.py.151: total_norm: 13.069059 [I 2022-10-27 18:10:30] ...trap/engines/engine.py.224: train: epoch 81 | batch 260/2304 [I 2022-10-27 18:10:30] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:59 [I 2022-10-27 18:10:30] ...trap/engines/engine.py.229: process: 0.55241 | load: 0.00243 [I 2022-10-27 18:10:30] ...trap/engines/engine.py.230: loss: 0.09654 [I 2022-10-27 18:10:30] ...optimizers/trijoint.py.151: total_norm: 9.525921 [I 2022-10-27 18:10:36] ...trap/engines/engine.py.224: train: epoch 81 | batch 270/2304 [I 2022-10-27 18:10:36] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:56 [I 2022-10-27 18:10:36] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00255 [I 2022-10-27 18:10:36] ...trap/engines/engine.py.230: loss: 0.13984 [I 2022-10-27 18:10:36] ...optimizers/trijoint.py.151: total_norm: 20.571630 [I 2022-10-27 18:10:41] ...trap/engines/engine.py.224: train: epoch 81 | batch 280/2304 [I 2022-10-27 18:10:41] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:44 [I 2022-10-27 18:10:41] ...trap/engines/engine.py.229: process: 0.54909 | load: 0.00256 [I 2022-10-27 18:10:41] ...trap/engines/engine.py.230: loss: 0.11400 [I 2022-10-27 18:10:41] ...optimizers/trijoint.py.151: total_norm: 10.694064 [I 2022-10-27 18:10:47] ...trap/engines/engine.py.224: train: epoch 81 | batch 290/2304 [I 2022-10-27 18:10:47] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:43 [I 2022-10-27 18:10:47] ...trap/engines/engine.py.229: process: 0.56701 | load: 0.00248 [I 2022-10-27 18:10:47] ...trap/engines/engine.py.230: loss: 0.19052 [I 2022-10-27 18:10:47] ...optimizers/trijoint.py.151: total_norm: 21.019901 [I 2022-10-27 18:10:53] ...trap/engines/engine.py.224: train: epoch 81 | batch 300/2304 [I 2022-10-27 18:10:53] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:50 [I 2022-10-27 18:10:53] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00248 [I 2022-10-27 18:10:53] ...trap/engines/engine.py.230: loss: 0.13690 [I 2022-10-27 18:10:53] ...optimizers/trijoint.py.151: total_norm: 20.724699 [I 2022-10-27 18:10:58] ...trap/engines/engine.py.224: train: epoch 81 | batch 310/2304 [I 2022-10-27 18:10:58] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:38 [I 2022-10-27 18:10:58] ...trap/engines/engine.py.229: process: 0.57791 | load: 0.00245 [I 2022-10-27 18:10:58] ...trap/engines/engine.py.230: loss: 0.12505 [I 2022-10-27 18:10:58] ...optimizers/trijoint.py.151: total_norm: 15.121140 [I 2022-10-27 18:11:04] ...trap/engines/engine.py.224: train: epoch 81 | batch 320/2304 [I 2022-10-27 18:11:04] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:21 [I 2022-10-27 18:11:04] ...trap/engines/engine.py.229: process: 0.55211 | load: 0.00254 [I 2022-10-27 18:11:04] ...trap/engines/engine.py.230: loss: 0.15423 [I 2022-10-27 18:11:04] ...optimizers/trijoint.py.151: total_norm: 37.519440 [I 2022-10-27 18:11:09] ...trap/engines/engine.py.224: train: epoch 81 | batch 330/2304 [I 2022-10-27 18:11:09] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:14 [I 2022-10-27 18:11:09] ...trap/engines/engine.py.229: process: 0.54838 | load: 0.00250 [I 2022-10-27 18:11:09] ...trap/engines/engine.py.230: loss: 0.12253 [I 2022-10-27 18:11:09] ...optimizers/trijoint.py.151: total_norm: 9.863400 [I 2022-10-27 18:11:15] ...trap/engines/engine.py.224: train: epoch 81 | batch 340/2304 [I 2022-10-27 18:11:15] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:16 [I 2022-10-27 18:11:15] ...trap/engines/engine.py.229: process: 0.57223 | load: 0.00246 [I 2022-10-27 18:11:15] ...trap/engines/engine.py.230: loss: 0.08388 [I 2022-10-27 18:11:15] ...optimizers/trijoint.py.151: total_norm: 11.946827 [I 2022-10-27 18:11:21] ...trap/engines/engine.py.224: train: epoch 81 | batch 350/2304 [I 2022-10-27 18:11:21] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:17 [I 2022-10-27 18:11:21] ...trap/engines/engine.py.229: process: 0.57224 | load: 0.00243 [I 2022-10-27 18:11:21] ...trap/engines/engine.py.230: loss: 0.17485 [I 2022-10-27 18:11:21] ...optimizers/trijoint.py.151: total_norm: 33.207832 [I 2022-10-27 18:11:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 360/2304 [I 2022-10-27 18:11:26] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:12 [I 2022-10-27 18:11:26] ...trap/engines/engine.py.229: process: 0.56812 | load: 0.00242 [I 2022-10-27 18:11:26] ...trap/engines/engine.py.230: loss: 0.15853 [I 2022-10-27 18:11:26] ...optimizers/trijoint.py.151: total_norm: 86.476196 [I 2022-10-27 18:11:32] ...trap/engines/engine.py.224: train: epoch 81 | batch 370/2304 [I 2022-10-27 18:11:32] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:59 [I 2022-10-27 18:11:32] ...trap/engines/engine.py.229: process: 0.55466 | load: 0.00244 [I 2022-10-27 18:11:32] ...trap/engines/engine.py.230: loss: 0.15103 [I 2022-10-27 18:11:32] ...optimizers/trijoint.py.151: total_norm: 15.780354 [I 2022-10-27 18:11:38] ...trap/engines/engine.py.224: train: epoch 81 | batch 380/2304 [I 2022-10-27 18:11:38] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:49 [I 2022-10-27 18:11:38] ...trap/engines/engine.py.229: process: 0.54650 | load: 0.00243 [I 2022-10-27 18:11:38] ...trap/engines/engine.py.230: loss: 0.14668 [I 2022-10-27 18:11:38] ...optimizers/trijoint.py.151: total_norm: 15.257472 [I 2022-10-27 18:11:43] ...trap/engines/engine.py.224: train: epoch 81 | batch 390/2304 [I 2022-10-27 18:11:43] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:42 [I 2022-10-27 18:11:43] ...trap/engines/engine.py.229: process: 0.54982 | load: 0.00254 [I 2022-10-27 18:11:43] ...trap/engines/engine.py.230: loss: 0.12022 [I 2022-10-27 18:11:43] ...optimizers/trijoint.py.151: total_norm: 24.047951 [I 2022-10-27 18:11:49] ...trap/engines/engine.py.224: train: epoch 81 | batch 400/2304 [I 2022-10-27 18:11:49] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:38 [I 2022-10-27 18:11:49] ...trap/engines/engine.py.229: process: 0.54795 | load: 0.00240 [I 2022-10-27 18:11:49] ...trap/engines/engine.py.230: loss: 0.09975 [I 2022-10-27 18:11:49] ...optimizers/trijoint.py.151: total_norm: 14.904236 [I 2022-10-27 18:11:54] ...trap/engines/engine.py.224: train: epoch 81 | batch 410/2304 [I 2022-10-27 18:11:54] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:32 [I 2022-10-27 18:11:54] ...trap/engines/engine.py.229: process: 0.56373 | load: 0.00249 [I 2022-10-27 18:11:54] ...trap/engines/engine.py.230: loss: 0.14601 [I 2022-10-27 18:11:54] ...optimizers/trijoint.py.151: total_norm: 14.627083 [I 2022-10-27 18:12:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 420/2304 [I 2022-10-27 18:12:00] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:23 [I 2022-10-27 18:12:00] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00245 [I 2022-10-27 18:12:00] ...trap/engines/engine.py.230: loss: 0.10593 [I 2022-10-27 18:12:00] ...optimizers/trijoint.py.151: total_norm: 10.566868 [I 2022-10-27 18:12:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 430/2304 [I 2022-10-27 18:12:05] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:18 [I 2022-10-27 18:12:05] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00244 [I 2022-10-27 18:12:05] ...trap/engines/engine.py.230: loss: 0.18913 [I 2022-10-27 18:12:05] ...optimizers/trijoint.py.151: total_norm: 41.014130 [I 2022-10-27 18:12:11] ...trap/engines/engine.py.224: train: epoch 81 | batch 440/2304 [I 2022-10-27 18:12:11] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:22 [I 2022-10-27 18:12:11] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00253 [I 2022-10-27 18:12:11] ...trap/engines/engine.py.230: loss: 0.24939 [I 2022-10-27 18:12:11] ...optimizers/trijoint.py.151: total_norm: 9.246296 [I 2022-10-27 18:12:16] ...trap/engines/engine.py.224: train: epoch 81 | batch 450/2304 [I 2022-10-27 18:12:16] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:16 [I 2022-10-27 18:12:16] ...trap/engines/engine.py.229: process: 0.56361 | load: 0.00250 [I 2022-10-27 18:12:16] ...trap/engines/engine.py.230: loss: 0.19422 [I 2022-10-27 18:12:16] ...optimizers/trijoint.py.151: total_norm: 23.061150 [I 2022-10-27 18:12:22] ...trap/engines/engine.py.224: train: epoch 81 | batch 460/2304 [I 2022-10-27 18:12:22] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:07 [I 2022-10-27 18:12:22] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00245 [I 2022-10-27 18:12:22] ...trap/engines/engine.py.230: loss: 0.13892 [I 2022-10-27 18:12:22] ...optimizers/trijoint.py.151: total_norm: 33.126194 [I 2022-10-27 18:12:28] ...trap/engines/engine.py.224: train: epoch 81 | batch 470/2304 [I 2022-10-27 18:12:28] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:57 [I 2022-10-27 18:12:28] ...trap/engines/engine.py.229: process: 0.55272 | load: 0.00284 [I 2022-10-27 18:12:28] ...trap/engines/engine.py.230: loss: 0.10147 [I 2022-10-27 18:12:28] ...optimizers/trijoint.py.151: total_norm: 11.606230 [I 2022-10-27 18:12:33] ...trap/engines/engine.py.224: train: epoch 81 | batch 480/2304 [I 2022-10-27 18:12:33] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:17:03 [I 2022-10-27 18:12:33] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00253 [I 2022-10-27 18:12:33] ...trap/engines/engine.py.230: loss: 0.10197 [I 2022-10-27 18:12:33] ...optimizers/trijoint.py.151: total_norm: 15.704077 [I 2022-10-27 18:12:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 490/2304 [I 2022-10-27 18:12:39] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:46 [I 2022-10-27 18:12:39] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00245 [I 2022-10-27 18:12:39] ...trap/engines/engine.py.230: loss: 0.14219 [I 2022-10-27 18:12:39] ...optimizers/trijoint.py.151: total_norm: 16.206774 [I 2022-10-27 18:12:44] ...trap/engines/engine.py.224: train: epoch 81 | batch 500/2304 [I 2022-10-27 18:12:44] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:35 [I 2022-10-27 18:12:44] ...trap/engines/engine.py.229: process: 0.53697 | load: 0.00242 [I 2022-10-27 18:12:44] ...trap/engines/engine.py.230: loss: 0.11502 [I 2022-10-27 18:12:44] ...optimizers/trijoint.py.151: total_norm: 10.537631 [I 2022-10-27 18:12:50] ...trap/engines/engine.py.224: train: epoch 81 | batch 510/2304 [I 2022-10-27 18:12:50] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:35 [I 2022-10-27 18:12:50] ...trap/engines/engine.py.229: process: 0.56641 | load: 0.00248 [I 2022-10-27 18:12:50] ...trap/engines/engine.py.230: loss: 0.21137 [I 2022-10-27 18:12:50] ...optimizers/trijoint.py.151: total_norm: 15.518067 [I 2022-10-27 18:12:56] ...trap/engines/engine.py.224: train: epoch 81 | batch 520/2304 [I 2022-10-27 18:12:56] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:30 [I 2022-10-27 18:12:56] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00245 [I 2022-10-27 18:12:56] ...trap/engines/engine.py.230: loss: 0.10448 [I 2022-10-27 18:12:56] ...optimizers/trijoint.py.151: total_norm: 9.567683 [I 2022-10-27 18:13:01] ...trap/engines/engine.py.224: train: epoch 81 | batch 530/2304 [I 2022-10-27 18:13:01] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:29 [I 2022-10-27 18:13:01] ...trap/engines/engine.py.229: process: 0.57878 | load: 0.00245 [I 2022-10-27 18:13:01] ...trap/engines/engine.py.230: loss: 0.13313 [I 2022-10-27 18:13:01] ...optimizers/trijoint.py.151: total_norm: 21.688761 [I 2022-10-27 18:13:07] ...trap/engines/engine.py.224: train: epoch 81 | batch 540/2304 [I 2022-10-27 18:13:07] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:27 [I 2022-10-27 18:13:07] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00241 [I 2022-10-27 18:13:07] ...trap/engines/engine.py.230: loss: 0.11242 [I 2022-10-27 18:13:07] ...optimizers/trijoint.py.151: total_norm: 18.598820 [I 2022-10-27 18:13:12] ...trap/engines/engine.py.224: train: epoch 81 | batch 550/2304 [I 2022-10-27 18:13:12] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:27 [I 2022-10-27 18:13:12] ...trap/engines/engine.py.229: process: 0.55880 | load: 0.00240 [I 2022-10-27 18:13:12] ...trap/engines/engine.py.230: loss: 0.17510 [I 2022-10-27 18:13:12] ...optimizers/trijoint.py.151: total_norm: 19.045113 [I 2022-10-27 18:13:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 560/2304 [I 2022-10-27 18:13:18] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:14 [I 2022-10-27 18:13:18] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00246 [I 2022-10-27 18:13:18] ...trap/engines/engine.py.230: loss: 0.15527 [I 2022-10-27 18:13:18] ...optimizers/trijoint.py.151: total_norm: 22.333988 [I 2022-10-27 18:13:24] ...trap/engines/engine.py.224: train: epoch 81 | batch 570/2304 [I 2022-10-27 18:13:24] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:07 [I 2022-10-27 18:13:24] ...trap/engines/engine.py.229: process: 0.57027 | load: 0.00246 [I 2022-10-27 18:13:24] ...trap/engines/engine.py.230: loss: 0.14741 [I 2022-10-27 18:13:24] ...optimizers/trijoint.py.151: total_norm: 20.776627 [I 2022-10-27 18:13:29] ...trap/engines/engine.py.224: train: epoch 81 | batch 580/2304 [I 2022-10-27 18:13:29] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:05 [I 2022-10-27 18:13:29] ...trap/engines/engine.py.229: process: 0.54828 | load: 0.00242 [I 2022-10-27 18:13:29] ...trap/engines/engine.py.230: loss: 0.12031 [I 2022-10-27 18:13:29] ...optimizers/trijoint.py.151: total_norm: 18.013784 [I 2022-10-27 18:13:35] ...trap/engines/engine.py.224: train: epoch 81 | batch 590/2304 [I 2022-10-27 18:13:35] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:16:14 [I 2022-10-27 18:13:35] ...trap/engines/engine.py.229: process: 0.58491 | load: 0.00242 [I 2022-10-27 18:13:35] ...trap/engines/engine.py.230: loss: 0.09732 [I 2022-10-27 18:13:35] ...optimizers/trijoint.py.151: total_norm: 9.683295 [I 2022-10-27 18:13:40] ...trap/engines/engine.py.224: train: epoch 81 | batch 600/2304 [I 2022-10-27 18:13:40] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:54 [I 2022-10-27 18:13:40] ...trap/engines/engine.py.229: process: 0.55265 | load: 0.00249 [I 2022-10-27 18:13:40] ...trap/engines/engine.py.230: loss: 0.13142 [I 2022-10-27 18:13:40] ...optimizers/trijoint.py.151: total_norm: 18.362730 [I 2022-10-27 18:13:46] ...trap/engines/engine.py.224: train: epoch 81 | batch 610/2304 [I 2022-10-27 18:13:46] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:43 [I 2022-10-27 18:13:46] ...trap/engines/engine.py.229: process: 0.55056 | load: 0.00242 [I 2022-10-27 18:13:46] ...trap/engines/engine.py.230: loss: 0.21123 [I 2022-10-27 18:13:46] ...optimizers/trijoint.py.151: total_norm: 36.008423 [I 2022-10-27 18:13:52] ...trap/engines/engine.py.224: train: epoch 81 | batch 620/2304 [I 2022-10-27 18:13:52] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:40 [I 2022-10-27 18:13:52] ...trap/engines/engine.py.229: process: 0.55767 | load: 0.00246 [I 2022-10-27 18:13:52] ...trap/engines/engine.py.230: loss: 0.11752 [I 2022-10-27 18:13:52] ...optimizers/trijoint.py.151: total_norm: 13.399404 [I 2022-10-27 18:13:57] ...trap/engines/engine.py.224: train: epoch 81 | batch 630/2304 [I 2022-10-27 18:13:57] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:40 [I 2022-10-27 18:13:57] ...trap/engines/engine.py.229: process: 0.58034 | load: 0.00244 [I 2022-10-27 18:13:57] ...trap/engines/engine.py.230: loss: 0.12228 [I 2022-10-27 18:13:57] ...optimizers/trijoint.py.151: total_norm: 36.855873 [I 2022-10-27 18:14:03] ...trap/engines/engine.py.224: train: epoch 81 | batch 640/2304 [I 2022-10-27 18:14:03] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:32 [I 2022-10-27 18:14:03] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00244 [I 2022-10-27 18:14:03] ...trap/engines/engine.py.230: loss: 0.14249 [I 2022-10-27 18:14:03] ...optimizers/trijoint.py.151: total_norm: 16.499197 [I 2022-10-27 18:14:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 650/2304 [I 2022-10-27 18:14:08] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:23 [I 2022-10-27 18:14:08] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00246 [I 2022-10-27 18:14:08] ...trap/engines/engine.py.230: loss: 0.12681 [I 2022-10-27 18:14:08] ...optimizers/trijoint.py.151: total_norm: 16.373705 [I 2022-10-27 18:14:14] ...trap/engines/engine.py.224: train: epoch 81 | batch 660/2304 [I 2022-10-27 18:14:14] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:13 [I 2022-10-27 18:14:14] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00249 [I 2022-10-27 18:14:14] ...trap/engines/engine.py.230: loss: 0.18627 [I 2022-10-27 18:14:14] ...optimizers/trijoint.py.151: total_norm: 17.307697 [I 2022-10-27 18:14:19] ...trap/engines/engine.py.224: train: epoch 81 | batch 670/2304 [I 2022-10-27 18:14:19] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:04 [I 2022-10-27 18:14:19] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00244 [I 2022-10-27 18:14:19] ...trap/engines/engine.py.230: loss: 0.08663 [I 2022-10-27 18:14:19] ...optimizers/trijoint.py.151: total_norm: 8.261269 [I 2022-10-27 18:14:25] ...trap/engines/engine.py.224: train: epoch 81 | batch 680/2304 [I 2022-10-27 18:14:25] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:14:56 [I 2022-10-27 18:14:25] ...trap/engines/engine.py.229: process: 0.54457 | load: 0.00262 [I 2022-10-27 18:14:25] ...trap/engines/engine.py.230: loss: 0.11212 [I 2022-10-27 18:14:25] ...optimizers/trijoint.py.151: total_norm: 11.566566 [I 2022-10-27 18:14:31] ...trap/engines/engine.py.224: train: epoch 81 | batch 690/2304 [I 2022-10-27 18:14:31] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:58 [I 2022-10-27 18:14:31] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00247 [I 2022-10-27 18:14:31] ...trap/engines/engine.py.230: loss: 0.10706 [I 2022-10-27 18:14:31] ...optimizers/trijoint.py.151: total_norm: 12.116530 [I 2022-10-27 18:14:36] ...trap/engines/engine.py.224: train: epoch 81 | batch 700/2304 [I 2022-10-27 18:14:36] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:56 [I 2022-10-27 18:14:36] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00247 [I 2022-10-27 18:14:36] ...trap/engines/engine.py.230: loss: 0.10777 [I 2022-10-27 18:14:36] ...optimizers/trijoint.py.151: total_norm: 10.008581 [I 2022-10-27 18:14:42] ...trap/engines/engine.py.224: train: epoch 81 | batch 710/2304 [I 2022-10-27 18:14:42] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:48 [I 2022-10-27 18:14:42] ...trap/engines/engine.py.229: process: 0.56432 | load: 0.00247 [I 2022-10-27 18:14:42] ...trap/engines/engine.py.230: loss: 0.14696 [I 2022-10-27 18:14:42] ...optimizers/trijoint.py.151: total_norm: 22.448301 [I 2022-10-27 18:14:47] ...trap/engines/engine.py.224: train: epoch 81 | batch 720/2304 [I 2022-10-27 18:14:47] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:40 [I 2022-10-27 18:14:47] ...trap/engines/engine.py.229: process: 0.55845 | load: 0.00242 [I 2022-10-27 18:14:47] ...trap/engines/engine.py.230: loss: 0.12951 [I 2022-10-27 18:14:47] ...optimizers/trijoint.py.151: total_norm: 20.104498 [I 2022-10-27 18:14:53] ...trap/engines/engine.py.224: train: epoch 81 | batch 730/2304 [I 2022-10-27 18:14:53] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:37 [I 2022-10-27 18:14:53] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00244 [I 2022-10-27 18:14:53] ...trap/engines/engine.py.230: loss: 0.17813 [I 2022-10-27 18:14:53] ...optimizers/trijoint.py.151: total_norm: 34.628445 [I 2022-10-27 18:14:58] ...trap/engines/engine.py.224: train: epoch 81 | batch 740/2304 [I 2022-10-27 18:14:58] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:25 [I 2022-10-27 18:14:58] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00243 [I 2022-10-27 18:14:58] ...trap/engines/engine.py.230: loss: 0.14001 [I 2022-10-27 18:14:58] ...optimizers/trijoint.py.151: total_norm: 11.580462 [I 2022-10-27 18:15:04] ...trap/engines/engine.py.224: train: epoch 81 | batch 750/2304 [I 2022-10-27 18:15:04] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:17 [I 2022-10-27 18:15:04] ...trap/engines/engine.py.229: process: 0.55354 | load: 0.00242 [I 2022-10-27 18:15:04] ...trap/engines/engine.py.230: loss: 0.13635 [I 2022-10-27 18:15:04] ...optimizers/trijoint.py.151: total_norm: 14.892561 [I 2022-10-27 18:15:10] ...trap/engines/engine.py.224: train: epoch 81 | batch 760/2304 [I 2022-10-27 18:15:10] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:16 [I 2022-10-27 18:15:10] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00241 [I 2022-10-27 18:15:10] ...trap/engines/engine.py.230: loss: 0.15223 [I 2022-10-27 18:15:10] ...optimizers/trijoint.py.151: total_norm: 15.891836 [I 2022-10-27 18:15:15] ...trap/engines/engine.py.224: train: epoch 81 | batch 770/2304 [I 2022-10-27 18:15:15] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:05 [I 2022-10-27 18:15:15] ...trap/engines/engine.py.229: process: 0.54807 | load: 0.00242 [I 2022-10-27 18:15:15] ...trap/engines/engine.py.230: loss: 0.08999 [I 2022-10-27 18:15:15] ...optimizers/trijoint.py.151: total_norm: 7.770707 [I 2022-10-27 18:15:21] ...trap/engines/engine.py.224: train: epoch 81 | batch 780/2304 [I 2022-10-27 18:15:21] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:07 [I 2022-10-27 18:15:21] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00250 [I 2022-10-27 18:15:21] ...trap/engines/engine.py.230: loss: 0.13814 [I 2022-10-27 18:15:21] ...optimizers/trijoint.py.151: total_norm: 21.246763 [I 2022-10-27 18:15:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 790/2304 [I 2022-10-27 18:15:26] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:03 [I 2022-10-27 18:15:26] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00245 [I 2022-10-27 18:15:26] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-27 18:15:26] ...optimizers/trijoint.py.151: total_norm: 16.832291 [I 2022-10-27 18:15:32] ...trap/engines/engine.py.224: train: epoch 81 | batch 800/2304 [I 2022-10-27 18:15:32] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:54 [I 2022-10-27 18:15:32] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00240 [I 2022-10-27 18:15:32] ...trap/engines/engine.py.230: loss: 0.13187 [I 2022-10-27 18:15:32] ...optimizers/trijoint.py.151: total_norm: 84.193741 [I 2022-10-27 18:15:37] ...trap/engines/engine.py.224: train: epoch 81 | batch 810/2304 [I 2022-10-27 18:15:37] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:47 [I 2022-10-27 18:15:37] ...trap/engines/engine.py.229: process: 0.54604 | load: 0.00242 [I 2022-10-27 18:15:37] ...trap/engines/engine.py.230: loss: 0.11503 [I 2022-10-27 18:15:37] ...optimizers/trijoint.py.151: total_norm: 14.759769 [I 2022-10-27 18:15:43] ...trap/engines/engine.py.224: train: epoch 81 | batch 820/2304 [I 2022-10-27 18:15:43] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:38 [I 2022-10-27 18:15:43] ...trap/engines/engine.py.229: process: 0.54641 | load: 0.00241 [I 2022-10-27 18:15:43] ...trap/engines/engine.py.230: loss: 0.17268 [I 2022-10-27 18:15:43] ...optimizers/trijoint.py.151: total_norm: 14.725703 [I 2022-10-27 18:15:48] ...trap/engines/engine.py.224: train: epoch 81 | batch 830/2304 [I 2022-10-27 18:15:48] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:38 [I 2022-10-27 18:15:48] ...trap/engines/engine.py.229: process: 0.55596 | load: 0.00246 [I 2022-10-27 18:15:48] ...trap/engines/engine.py.230: loss: 0.18315 [I 2022-10-27 18:15:48] ...optimizers/trijoint.py.151: total_norm: 29.713001 [I 2022-10-27 18:15:54] ...trap/engines/engine.py.224: train: epoch 81 | batch 840/2304 [I 2022-10-27 18:15:54] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:34 [I 2022-10-27 18:15:54] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00242 [I 2022-10-27 18:15:54] ...trap/engines/engine.py.230: loss: 0.15650 [I 2022-10-27 18:15:54] ...optimizers/trijoint.py.151: total_norm: 54.209190 [I 2022-10-27 18:16:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 850/2304 [I 2022-10-27 18:16:00] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:37 [I 2022-10-27 18:16:00] ...trap/engines/engine.py.229: process: 0.57522 | load: 0.00242 [I 2022-10-27 18:16:00] ...trap/engines/engine.py.230: loss: 0.09416 [I 2022-10-27 18:16:00] ...optimizers/trijoint.py.151: total_norm: 17.484612 [I 2022-10-27 18:16:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 860/2304 [I 2022-10-27 18:16:05] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:28 [I 2022-10-27 18:16:05] ...trap/engines/engine.py.229: process: 0.56635 | load: 0.00250 [I 2022-10-27 18:16:05] ...trap/engines/engine.py.230: loss: 0.11391 [I 2022-10-27 18:16:05] ...optimizers/trijoint.py.151: total_norm: 16.402855 [I 2022-10-27 18:16:11] ...trap/engines/engine.py.224: train: epoch 81 | batch 870/2304 [I 2022-10-27 18:16:11] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:27 [I 2022-10-27 18:16:11] ...trap/engines/engine.py.229: process: 0.56571 | load: 0.00249 [I 2022-10-27 18:16:11] ...trap/engines/engine.py.230: loss: 0.11478 [I 2022-10-27 18:16:11] ...optimizers/trijoint.py.151: total_norm: 10.917243 [I 2022-10-27 18:16:16] ...trap/engines/engine.py.224: train: epoch 81 | batch 880/2304 [I 2022-10-27 18:16:16] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:10 [I 2022-10-27 18:16:16] ...trap/engines/engine.py.229: process: 0.54503 | load: 0.00245 [I 2022-10-27 18:16:16] ...trap/engines/engine.py.230: loss: 0.11244 [I 2022-10-27 18:16:16] ...optimizers/trijoint.py.151: total_norm: 14.316735 [I 2022-10-27 18:16:22] ...trap/engines/engine.py.224: train: epoch 81 | batch 890/2304 [I 2022-10-27 18:16:22] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:10 [I 2022-10-27 18:16:22] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00243 [I 2022-10-27 18:16:22] ...trap/engines/engine.py.230: loss: 0.15882 [I 2022-10-27 18:16:22] ...optimizers/trijoint.py.151: total_norm: 11.689946 [I 2022-10-27 18:16:28] ...trap/engines/engine.py.224: train: epoch 81 | batch 900/2304 [I 2022-10-27 18:16:28] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:14 [I 2022-10-27 18:16:28] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00250 [I 2022-10-27 18:16:28] ...trap/engines/engine.py.230: loss: 0.15121 [I 2022-10-27 18:16:28] ...optimizers/trijoint.py.151: total_norm: 37.136814 [I 2022-10-27 18:16:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 910/2304 [I 2022-10-27 18:16:34] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:13:27 [I 2022-10-27 18:16:34] ...trap/engines/engine.py.229: process: 0.64639 | load: 0.00242 [I 2022-10-27 18:16:34] ...trap/engines/engine.py.230: loss: 0.10086 [I 2022-10-27 18:16:34] ...optimizers/trijoint.py.151: total_norm: 12.525076 [I 2022-10-27 18:16:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 920/2304 [I 2022-10-27 18:16:39] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:52 [I 2022-10-27 18:16:39] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00242 [I 2022-10-27 18:16:39] ...trap/engines/engine.py.230: loss: 0.14521 [I 2022-10-27 18:16:39] ...optimizers/trijoint.py.151: total_norm: 16.875628 [I 2022-10-27 18:16:45] ...trap/engines/engine.py.224: train: epoch 81 | batch 930/2304 [I 2022-10-27 18:16:45] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:45 [I 2022-10-27 18:16:45] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00242 [I 2022-10-27 18:16:45] ...trap/engines/engine.py.230: loss: 0.11058 [I 2022-10-27 18:16:45] ...optimizers/trijoint.py.151: total_norm: 10.393262 [I 2022-10-27 18:16:50] ...trap/engines/engine.py.224: train: epoch 81 | batch 940/2304 [I 2022-10-27 18:16:50] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:40 [I 2022-10-27 18:16:50] ...trap/engines/engine.py.229: process: 0.55100 | load: 0.00245 [I 2022-10-27 18:16:50] ...trap/engines/engine.py.230: loss: 0.15694 [I 2022-10-27 18:16:50] ...optimizers/trijoint.py.151: total_norm: 14.413611 [I 2022-10-27 18:16:56] ...trap/engines/engine.py.224: train: epoch 81 | batch 950/2304 [I 2022-10-27 18:16:56] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:34 [I 2022-10-27 18:16:56] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00242 [I 2022-10-27 18:16:56] ...trap/engines/engine.py.230: loss: 0.15207 [I 2022-10-27 18:16:56] ...optimizers/trijoint.py.151: total_norm: 6.816055 [I 2022-10-27 18:17:01] ...trap/engines/engine.py.224: train: epoch 81 | batch 960/2304 [I 2022-10-27 18:17:01] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:21 [I 2022-10-27 18:17:01] ...trap/engines/engine.py.229: process: 0.54446 | load: 0.00244 [I 2022-10-27 18:17:01] ...trap/engines/engine.py.230: loss: 0.08684 [I 2022-10-27 18:17:01] ...optimizers/trijoint.py.151: total_norm: 11.515383 [I 2022-10-27 18:17:07] ...trap/engines/engine.py.224: train: epoch 81 | batch 970/2304 [I 2022-10-27 18:17:07] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:21 [I 2022-10-27 18:17:07] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00244 [I 2022-10-27 18:17:07] ...trap/engines/engine.py.230: loss: 0.14701 [I 2022-10-27 18:17:07] ...optimizers/trijoint.py.151: total_norm: 26.305874 [I 2022-10-27 18:17:13] ...trap/engines/engine.py.224: train: epoch 81 | batch 980/2304 [I 2022-10-27 18:17:13] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:34 [I 2022-10-27 18:17:13] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00248 [I 2022-10-27 18:17:13] ...trap/engines/engine.py.230: loss: 0.10350 [I 2022-10-27 18:17:13] ...optimizers/trijoint.py.151: total_norm: 10.510563 [I 2022-10-27 18:17:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 990/2304 [I 2022-10-27 18:17:18] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:08 [I 2022-10-27 18:17:18] ...trap/engines/engine.py.229: process: 0.55124 | load: 0.00248 [I 2022-10-27 18:17:18] ...trap/engines/engine.py.230: loss: 0.12871 [I 2022-10-27 18:17:18] ...optimizers/trijoint.py.151: total_norm: 18.226702 [I 2022-10-27 18:17:24] ...trap/engines/engine.py.224: train: epoch 81 | batch 1000/2304 [I 2022-10-27 18:17:24] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:09 [I 2022-10-27 18:17:24] ...trap/engines/engine.py.229: process: 0.56783 | load: 0.00243 [I 2022-10-27 18:17:24] ...trap/engines/engine.py.230: loss: 0.11790 [I 2022-10-27 18:17:24] ...optimizers/trijoint.py.151: total_norm: 15.008976 [I 2022-10-27 18:17:29] ...trap/engines/engine.py.224: train: epoch 81 | batch 1010/2304 [I 2022-10-27 18:17:29] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:01 [I 2022-10-27 18:17:29] ...trap/engines/engine.py.229: process: 0.54944 | load: 0.00245 [I 2022-10-27 18:17:29] ...trap/engines/engine.py.230: loss: 0.21566 [I 2022-10-27 18:17:29] ...optimizers/trijoint.py.151: total_norm: 24.405785 [I 2022-10-27 18:17:35] ...trap/engines/engine.py.224: train: epoch 81 | batch 1020/2304 [I 2022-10-27 18:17:35] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:12:00 [I 2022-10-27 18:17:35] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00251 [I 2022-10-27 18:17:35] ...trap/engines/engine.py.230: loss: 0.13567 [I 2022-10-27 18:17:35] ...optimizers/trijoint.py.151: total_norm: 17.037170 [I 2022-10-27 18:17:41] ...trap/engines/engine.py.224: train: epoch 81 | batch 1030/2304 [I 2022-10-27 18:17:41] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:52 [I 2022-10-27 18:17:41] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00250 [I 2022-10-27 18:17:41] ...trap/engines/engine.py.230: loss: 0.15371 [I 2022-10-27 18:17:41] ...optimizers/trijoint.py.151: total_norm: 12.185871 [I 2022-10-27 18:17:46] ...trap/engines/engine.py.224: train: epoch 81 | batch 1040/2304 [I 2022-10-27 18:17:46] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:46 [I 2022-10-27 18:17:46] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00247 [I 2022-10-27 18:17:46] ...trap/engines/engine.py.230: loss: 0.11759 [I 2022-10-27 18:17:46] ...optimizers/trijoint.py.151: total_norm: 69.427734 [I 2022-10-27 18:17:52] ...trap/engines/engine.py.224: train: epoch 81 | batch 1050/2304 [I 2022-10-27 18:17:52] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:56 [I 2022-10-27 18:17:52] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00252 [I 2022-10-27 18:17:52] ...trap/engines/engine.py.230: loss: 0.13088 [I 2022-10-27 18:17:52] ...optimizers/trijoint.py.151: total_norm: 21.289515 [I 2022-10-27 18:17:58] ...trap/engines/engine.py.224: train: epoch 81 | batch 1060/2304 [I 2022-10-27 18:17:58] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:39 [I 2022-10-27 18:17:58] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00250 [I 2022-10-27 18:17:58] ...trap/engines/engine.py.230: loss: 0.11397 [I 2022-10-27 18:17:58] ...optimizers/trijoint.py.151: total_norm: 12.337371 [I 2022-10-27 18:18:03] ...trap/engines/engine.py.224: train: epoch 81 | batch 1070/2304 [I 2022-10-27 18:18:03] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:26 [I 2022-10-27 18:18:03] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00246 [I 2022-10-27 18:18:03] ...trap/engines/engine.py.230: loss: 0.14424 [I 2022-10-27 18:18:03] ...optimizers/trijoint.py.151: total_norm: 14.102863 [I 2022-10-27 18:18:09] ...trap/engines/engine.py.224: train: epoch 81 | batch 1080/2304 [I 2022-10-27 18:18:09] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:23 [I 2022-10-27 18:18:09] ...trap/engines/engine.py.229: process: 0.56558 | load: 0.00243 [I 2022-10-27 18:18:09] ...trap/engines/engine.py.230: loss: 0.11305 [I 2022-10-27 18:18:09] ...optimizers/trijoint.py.151: total_norm: 17.670324 [I 2022-10-27 18:18:14] ...trap/engines/engine.py.224: train: epoch 81 | batch 1090/2304 [I 2022-10-27 18:18:14] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:15 [I 2022-10-27 18:18:14] ...trap/engines/engine.py.229: process: 0.54984 | load: 0.00244 [I 2022-10-27 18:18:14] ...trap/engines/engine.py.230: loss: 0.12411 [I 2022-10-27 18:18:14] ...optimizers/trijoint.py.151: total_norm: 12.116665 [I 2022-10-27 18:18:20] ...trap/engines/engine.py.224: train: epoch 81 | batch 1100/2304 [I 2022-10-27 18:18:20] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:06 [I 2022-10-27 18:18:20] ...trap/engines/engine.py.229: process: 0.54808 | load: 0.00244 [I 2022-10-27 18:18:20] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-27 18:18:20] ...optimizers/trijoint.py.151: total_norm: 15.200105 [I 2022-10-27 18:18:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 1110/2304 [I 2022-10-27 18:18:26] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:11 [I 2022-10-27 18:18:26] ...trap/engines/engine.py.229: process: 0.55127 | load: 0.00246 [I 2022-10-27 18:18:26] ...trap/engines/engine.py.230: loss: 0.12527 [I 2022-10-27 18:18:26] ...optimizers/trijoint.py.151: total_norm: 13.727590 [I 2022-10-27 18:18:31] ...trap/engines/engine.py.224: train: epoch 81 | batch 1120/2304 [I 2022-10-27 18:18:31] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:02 [I 2022-10-27 18:18:31] ...trap/engines/engine.py.229: process: 0.56062 | load: 0.00241 [I 2022-10-27 18:18:31] ...trap/engines/engine.py.230: loss: 0.11017 [I 2022-10-27 18:18:31] ...optimizers/trijoint.py.151: total_norm: 16.281906 [I 2022-10-27 18:18:37] ...trap/engines/engine.py.224: train: epoch 81 | batch 1130/2304 [I 2022-10-27 18:18:37] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:53 [I 2022-10-27 18:18:37] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00247 [I 2022-10-27 18:18:37] ...trap/engines/engine.py.230: loss: 0.09537 [I 2022-10-27 18:18:37] ...optimizers/trijoint.py.151: total_norm: 11.970305 [I 2022-10-27 18:18:42] ...trap/engines/engine.py.224: train: epoch 81 | batch 1140/2304 [I 2022-10-27 18:18:42] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:48 [I 2022-10-27 18:18:42] ...trap/engines/engine.py.229: process: 0.56112 | load: 0.00240 [I 2022-10-27 18:18:42] ...trap/engines/engine.py.230: loss: 0.08389 [I 2022-10-27 18:18:42] ...optimizers/trijoint.py.151: total_norm: 11.530974 [I 2022-10-27 18:18:48] ...trap/engines/engine.py.224: train: epoch 81 | batch 1150/2304 [I 2022-10-27 18:18:48] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:45 [I 2022-10-27 18:18:48] ...trap/engines/engine.py.229: process: 0.56601 | load: 0.00252 [I 2022-10-27 18:18:48] ...trap/engines/engine.py.230: loss: 0.16523 [I 2022-10-27 18:18:48] ...optimizers/trijoint.py.151: total_norm: 28.977322 [I 2022-10-27 18:18:53] ...trap/engines/engine.py.224: train: epoch 81 | batch 1160/2304 [I 2022-10-27 18:18:53] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:39 [I 2022-10-27 18:18:53] ...trap/engines/engine.py.229: process: 0.56662 | load: 0.00250 [I 2022-10-27 18:18:53] ...trap/engines/engine.py.230: loss: 0.10786 [I 2022-10-27 18:18:53] ...optimizers/trijoint.py.151: total_norm: 15.963520 [I 2022-10-27 18:18:59] ...trap/engines/engine.py.224: train: epoch 81 | batch 1170/2304 [I 2022-10-27 18:18:59] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:35 [I 2022-10-27 18:18:59] ...trap/engines/engine.py.229: process: 0.56482 | load: 0.00245 [I 2022-10-27 18:18:59] ...trap/engines/engine.py.230: loss: 0.09757 [I 2022-10-27 18:18:59] ...optimizers/trijoint.py.151: total_norm: 15.114576 [I 2022-10-27 18:19:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 1180/2304 [I 2022-10-27 18:19:05] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:28 [I 2022-10-27 18:19:05] ...trap/engines/engine.py.229: process: 0.56917 | load: 0.00242 [I 2022-10-27 18:19:05] ...trap/engines/engine.py.230: loss: 0.11260 [I 2022-10-27 18:19:05] ...optimizers/trijoint.py.151: total_norm: 19.300266 [I 2022-10-27 18:19:10] ...trap/engines/engine.py.224: train: epoch 81 | batch 1190/2304 [I 2022-10-27 18:19:10] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:22 [I 2022-10-27 18:19:10] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00245 [I 2022-10-27 18:19:10] ...trap/engines/engine.py.230: loss: 0.14136 [I 2022-10-27 18:19:10] ...optimizers/trijoint.py.151: total_norm: 44.703587 [I 2022-10-27 18:19:16] ...trap/engines/engine.py.224: train: epoch 81 | batch 1200/2304 [I 2022-10-27 18:19:16] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:19 [I 2022-10-27 18:19:16] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00245 [I 2022-10-27 18:19:16] ...trap/engines/engine.py.230: loss: 0.15992 [I 2022-10-27 18:19:16] ...optimizers/trijoint.py.151: total_norm: 17.786549 [I 2022-10-27 18:19:22] ...trap/engines/engine.py.224: train: epoch 81 | batch 1210/2304 [I 2022-10-27 18:19:22] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:13 [I 2022-10-27 18:19:22] ...trap/engines/engine.py.229: process: 0.56486 | load: 0.00250 [I 2022-10-27 18:19:22] ...trap/engines/engine.py.230: loss: 0.16378 [I 2022-10-27 18:19:22] ...optimizers/trijoint.py.151: total_norm: 25.791800 [I 2022-10-27 18:19:27] ...trap/engines/engine.py.224: train: epoch 81 | batch 1220/2304 [I 2022-10-27 18:19:27] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:09 [I 2022-10-27 18:19:27] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00252 [I 2022-10-27 18:19:27] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-27 18:19:27] ...optimizers/trijoint.py.151: total_norm: 23.215086 [I 2022-10-27 18:19:33] ...trap/engines/engine.py.224: train: epoch 81 | batch 1230/2304 [I 2022-10-27 18:19:33] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:01 [I 2022-10-27 18:19:33] ...trap/engines/engine.py.229: process: 0.57679 | load: 0.00248 [I 2022-10-27 18:19:33] ...trap/engines/engine.py.230: loss: 0.19027 [I 2022-10-27 18:19:33] ...optimizers/trijoint.py.151: total_norm: 49.228214 [I 2022-10-27 18:19:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 1240/2304 [I 2022-10-27 18:19:39] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:59 [I 2022-10-27 18:19:39] ...trap/engines/engine.py.229: process: 0.54544 | load: 0.00247 [I 2022-10-27 18:19:39] ...trap/engines/engine.py.230: loss: 0.13744 [I 2022-10-27 18:19:39] ...optimizers/trijoint.py.151: total_norm: 12.438551 [I 2022-10-27 18:19:44] ...trap/engines/engine.py.224: train: epoch 81 | batch 1250/2304 [I 2022-10-27 18:19:44] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:49 [I 2022-10-27 18:19:44] ...trap/engines/engine.py.229: process: 0.56368 | load: 0.00248 [I 2022-10-27 18:19:44] ...trap/engines/engine.py.230: loss: 0.16085 [I 2022-10-27 18:19:44] ...optimizers/trijoint.py.151: total_norm: 13.262008 [I 2022-10-27 18:19:50] ...trap/engines/engine.py.224: train: epoch 81 | batch 1260/2304 [I 2022-10-27 18:19:50] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:36 [I 2022-10-27 18:19:50] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00245 [I 2022-10-27 18:19:50] ...trap/engines/engine.py.230: loss: 0.21214 [I 2022-10-27 18:19:50] ...optimizers/trijoint.py.151: total_norm: 28.094204 [I 2022-10-27 18:19:55] ...trap/engines/engine.py.224: train: epoch 81 | batch 1270/2304 [I 2022-10-27 18:19:55] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:38 [I 2022-10-27 18:19:55] ...trap/engines/engine.py.229: process: 0.56718 | load: 0.00240 [I 2022-10-27 18:19:55] ...trap/engines/engine.py.230: loss: 0.09970 [I 2022-10-27 18:19:55] ...optimizers/trijoint.py.151: total_norm: 12.778707 [I 2022-10-27 18:20:01] ...trap/engines/engine.py.224: train: epoch 81 | batch 1280/2304 [I 2022-10-27 18:20:01] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:31 [I 2022-10-27 18:20:01] ...trap/engines/engine.py.229: process: 0.55611 | load: 0.00241 [I 2022-10-27 18:20:01] ...trap/engines/engine.py.230: loss: 0.11447 [I 2022-10-27 18:20:01] ...optimizers/trijoint.py.151: total_norm: 11.026543 [I 2022-10-27 18:20:06] ...trap/engines/engine.py.224: train: epoch 81 | batch 1290/2304 [I 2022-10-27 18:20:06] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:22 [I 2022-10-27 18:20:06] ...trap/engines/engine.py.229: process: 0.55066 | load: 0.00245 [I 2022-10-27 18:20:06] ...trap/engines/engine.py.230: loss: 0.15924 [I 2022-10-27 18:20:06] ...optimizers/trijoint.py.151: total_norm: 13.228368 [I 2022-10-27 18:20:12] ...trap/engines/engine.py.224: train: epoch 81 | batch 1300/2304 [I 2022-10-27 18:20:12] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:29 [I 2022-10-27 18:20:12] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00250 [I 2022-10-27 18:20:12] ...trap/engines/engine.py.230: loss: 0.13857 [I 2022-10-27 18:20:12] ...optimizers/trijoint.py.151: total_norm: 83.065544 [I 2022-10-27 18:20:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 1310/2304 [I 2022-10-27 18:20:18] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:17 [I 2022-10-27 18:20:18] ...trap/engines/engine.py.229: process: 0.56378 | load: 0.00247 [I 2022-10-27 18:20:18] ...trap/engines/engine.py.230: loss: 0.12950 [I 2022-10-27 18:20:18] ...optimizers/trijoint.py.151: total_norm: 14.339979 [I 2022-10-27 18:20:23] ...trap/engines/engine.py.224: train: epoch 81 | batch 1320/2304 [I 2022-10-27 18:20:23] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:06 [I 2022-10-27 18:20:23] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00241 [I 2022-10-27 18:20:23] ...trap/engines/engine.py.230: loss: 0.14913 [I 2022-10-27 18:20:23] ...optimizers/trijoint.py.151: total_norm: 14.113942 [I 2022-10-27 18:20:29] ...trap/engines/engine.py.224: train: epoch 81 | batch 1330/2304 [I 2022-10-27 18:20:29] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:00 [I 2022-10-27 18:20:29] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00244 [I 2022-10-27 18:20:29] ...trap/engines/engine.py.230: loss: 0.12647 [I 2022-10-27 18:20:29] ...optimizers/trijoint.py.151: total_norm: 9.917342 [I 2022-10-27 18:20:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 1340/2304 [I 2022-10-27 18:20:34] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:54 [I 2022-10-27 18:20:34] ...trap/engines/engine.py.229: process: 0.55104 | load: 0.00244 [I 2022-10-27 18:20:34] ...trap/engines/engine.py.230: loss: 0.10564 [I 2022-10-27 18:20:34] ...optimizers/trijoint.py.151: total_norm: 16.519527 [I 2022-10-27 18:20:40] ...trap/engines/engine.py.224: train: epoch 81 | batch 1350/2304 [I 2022-10-27 18:20:40] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:48 [I 2022-10-27 18:20:40] ...trap/engines/engine.py.229: process: 0.55583 | load: 0.00246 [I 2022-10-27 18:20:40] ...trap/engines/engine.py.230: loss: 0.09711 [I 2022-10-27 18:20:40] ...optimizers/trijoint.py.151: total_norm: 10.027884 [I 2022-10-27 18:20:46] ...trap/engines/engine.py.224: train: epoch 81 | batch 1360/2304 [I 2022-10-27 18:20:46] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:47 [I 2022-10-27 18:20:46] ...trap/engines/engine.py.229: process: 0.56334 | load: 0.00244 [I 2022-10-27 18:20:46] ...trap/engines/engine.py.230: loss: 0.11825 [I 2022-10-27 18:20:46] ...optimizers/trijoint.py.151: total_norm: 18.882330 [I 2022-10-27 18:20:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 1370/2304 [I 2022-10-27 18:20:51] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:47 [I 2022-10-27 18:20:51] ...trap/engines/engine.py.229: process: 0.56181 | load: 0.00251 [I 2022-10-27 18:20:51] ...trap/engines/engine.py.230: loss: 0.12805 [I 2022-10-27 18:20:51] ...optimizers/trijoint.py.151: total_norm: 12.674149 [I 2022-10-27 18:20:57] ...trap/engines/engine.py.224: train: epoch 81 | batch 1380/2304 [I 2022-10-27 18:20:57] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:33 [I 2022-10-27 18:20:57] ...trap/engines/engine.py.229: process: 0.54505 | load: 0.00246 [I 2022-10-27 18:20:57] ...trap/engines/engine.py.230: loss: 0.13192 [I 2022-10-27 18:20:57] ...optimizers/trijoint.py.151: total_norm: 13.115069 [I 2022-10-27 18:21:02] ...trap/engines/engine.py.224: train: epoch 81 | batch 1390/2304 [I 2022-10-27 18:21:02] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:28 [I 2022-10-27 18:21:02] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00249 [I 2022-10-27 18:21:02] ...trap/engines/engine.py.230: loss: 0.16238 [I 2022-10-27 18:21:02] ...optimizers/trijoint.py.151: total_norm: 24.472067 [I 2022-10-27 18:21:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 1400/2304 [I 2022-10-27 18:21:08] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:22 [I 2022-10-27 18:21:08] ...trap/engines/engine.py.229: process: 0.56382 | load: 0.00245 [I 2022-10-27 18:21:08] ...trap/engines/engine.py.230: loss: 0.09722 [I 2022-10-27 18:21:08] ...optimizers/trijoint.py.151: total_norm: 13.645232 [I 2022-10-27 18:21:14] ...trap/engines/engine.py.224: train: epoch 81 | batch 1410/2304 [I 2022-10-27 18:21:14] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:14 [I 2022-10-27 18:21:14] ...trap/engines/engine.py.229: process: 0.55068 | load: 0.00245 [I 2022-10-27 18:21:14] ...trap/engines/engine.py.230: loss: 0.19205 [I 2022-10-27 18:21:14] ...optimizers/trijoint.py.151: total_norm: 35.201168 [I 2022-10-27 18:21:19] ...trap/engines/engine.py.224: train: epoch 81 | batch 1420/2304 [I 2022-10-27 18:21:19] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:11 [I 2022-10-27 18:21:19] ...trap/engines/engine.py.229: process: 0.54914 | load: 0.00242 [I 2022-10-27 18:21:19] ...trap/engines/engine.py.230: loss: 0.13127 [I 2022-10-27 18:21:19] ...optimizers/trijoint.py.151: total_norm: 50.575901 [I 2022-10-27 18:21:25] ...trap/engines/engine.py.224: train: epoch 81 | batch 1430/2304 [I 2022-10-27 18:21:25] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:14 [I 2022-10-27 18:21:25] ...trap/engines/engine.py.229: process: 0.55518 | load: 0.00241 [I 2022-10-27 18:21:25] ...trap/engines/engine.py.230: loss: 0.13373 [I 2022-10-27 18:21:25] ...optimizers/trijoint.py.151: total_norm: 20.752287 [I 2022-10-27 18:21:30] ...trap/engines/engine.py.224: train: epoch 81 | batch 1440/2304 [I 2022-10-27 18:21:30] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:05 [I 2022-10-27 18:21:30] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00241 [I 2022-10-27 18:21:30] ...trap/engines/engine.py.230: loss: 0.10614 [I 2022-10-27 18:21:30] ...optimizers/trijoint.py.151: total_norm: 14.355060 [I 2022-10-27 18:21:36] ...trap/engines/engine.py.224: train: epoch 81 | batch 1450/2304 [I 2022-10-27 18:21:36] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:54 [I 2022-10-27 18:21:36] ...trap/engines/engine.py.229: process: 0.54401 | load: 0.00244 [I 2022-10-27 18:21:36] ...trap/engines/engine.py.230: loss: 0.10967 [I 2022-10-27 18:21:36] ...optimizers/trijoint.py.151: total_norm: 30.834150 [I 2022-10-27 18:21:42] ...trap/engines/engine.py.224: train: epoch 81 | batch 1460/2304 [I 2022-10-27 18:21:42] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:47 [I 2022-10-27 18:21:42] ...trap/engines/engine.py.229: process: 0.55129 | load: 0.00245 [I 2022-10-27 18:21:42] ...trap/engines/engine.py.230: loss: 0.15287 [I 2022-10-27 18:21:42] ...optimizers/trijoint.py.151: total_norm: 25.226990 [I 2022-10-27 18:21:47] ...trap/engines/engine.py.224: train: epoch 81 | batch 1470/2304 [I 2022-10-27 18:21:47] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:42 [I 2022-10-27 18:21:47] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00245 [I 2022-10-27 18:21:47] ...trap/engines/engine.py.230: loss: 0.13534 [I 2022-10-27 18:21:47] ...optimizers/trijoint.py.151: total_norm: 22.005707 [I 2022-10-27 18:21:53] ...trap/engines/engine.py.224: train: epoch 81 | batch 1480/2304 [I 2022-10-27 18:21:53] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:37 [I 2022-10-27 18:21:53] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00259 [I 2022-10-27 18:21:53] ...trap/engines/engine.py.230: loss: 0.12140 [I 2022-10-27 18:21:53] ...optimizers/trijoint.py.151: total_norm: 11.739286 [I 2022-10-27 18:21:58] ...trap/engines/engine.py.224: train: epoch 81 | batch 1490/2304 [I 2022-10-27 18:21:58] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:30 [I 2022-10-27 18:21:58] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00248 [I 2022-10-27 18:21:58] ...trap/engines/engine.py.230: loss: 0.15946 [I 2022-10-27 18:21:58] ...optimizers/trijoint.py.151: total_norm: 22.307961 [I 2022-10-27 18:22:04] ...trap/engines/engine.py.224: train: epoch 81 | batch 1500/2304 [I 2022-10-27 18:22:04] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:26 [I 2022-10-27 18:22:04] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00247 [I 2022-10-27 18:22:04] ...trap/engines/engine.py.230: loss: 0.12883 [I 2022-10-27 18:22:04] ...optimizers/trijoint.py.151: total_norm: 11.767231 [I 2022-10-27 18:22:09] ...trap/engines/engine.py.224: train: epoch 81 | batch 1510/2304 [I 2022-10-27 18:22:09] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:19 [I 2022-10-27 18:22:09] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00244 [I 2022-10-27 18:22:09] ...trap/engines/engine.py.230: loss: 0.11094 [I 2022-10-27 18:22:09] ...optimizers/trijoint.py.151: total_norm: 9.345032 [I 2022-10-27 18:22:15] ...trap/engines/engine.py.224: train: epoch 81 | batch 1520/2304 [I 2022-10-27 18:22:15] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:17 [I 2022-10-27 18:22:15] ...trap/engines/engine.py.229: process: 0.55309 | load: 0.00245 [I 2022-10-27 18:22:15] ...trap/engines/engine.py.230: loss: 0.11698 [I 2022-10-27 18:22:15] ...optimizers/trijoint.py.151: total_norm: 12.019042 [I 2022-10-27 18:22:21] ...trap/engines/engine.py.224: train: epoch 81 | batch 1530/2304 [I 2022-10-27 18:22:21] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:08 [I 2022-10-27 18:22:21] ...trap/engines/engine.py.229: process: 0.55880 | load: 0.00247 [I 2022-10-27 18:22:21] ...trap/engines/engine.py.230: loss: 0.14433 [I 2022-10-27 18:22:21] ...optimizers/trijoint.py.151: total_norm: 13.086126 [I 2022-10-27 18:22:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 1540/2304 [I 2022-10-27 18:22:26] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:03 [I 2022-10-27 18:22:26] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00244 [I 2022-10-27 18:22:26] ...trap/engines/engine.py.230: loss: 0.13918 [I 2022-10-27 18:22:26] ...optimizers/trijoint.py.151: total_norm: 18.251844 [I 2022-10-27 18:22:32] ...trap/engines/engine.py.224: train: epoch 81 | batch 1550/2304 [I 2022-10-27 18:22:32] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:06:58 [I 2022-10-27 18:22:32] ...trap/engines/engine.py.229: process: 0.56464 | load: 0.00245 [I 2022-10-27 18:22:32] ...trap/engines/engine.py.230: loss: 0.11290 [I 2022-10-27 18:22:32] ...optimizers/trijoint.py.151: total_norm: 10.368813 [I 2022-10-27 18:22:37] ...trap/engines/engine.py.224: train: epoch 81 | batch 1560/2304 [I 2022-10-27 18:22:37] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:06:52 [I 2022-10-27 18:22:37] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00246 [I 2022-10-27 18:22:37] ...trap/engines/engine.py.230: loss: 0.11328 [I 2022-10-27 18:22:37] ...optimizers/trijoint.py.151: total_norm: 18.674847 [I 2022-10-27 18:22:43] ...trap/engines/engine.py.224: train: epoch 81 | batch 1570/2304 [I 2022-10-27 18:22:43] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:48 [I 2022-10-27 18:22:43] ...trap/engines/engine.py.229: process: 0.55924 | load: 0.00246 [I 2022-10-27 18:22:43] ...trap/engines/engine.py.230: loss: 0.14286 [I 2022-10-27 18:22:43] ...optimizers/trijoint.py.151: total_norm: 14.822715 [I 2022-10-27 18:22:48] ...trap/engines/engine.py.224: train: epoch 81 | batch 1580/2304 [I 2022-10-27 18:22:48] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:46 [I 2022-10-27 18:22:48] ...trap/engines/engine.py.229: process: 0.57308 | load: 0.00245 [I 2022-10-27 18:22:48] ...trap/engines/engine.py.230: loss: 0.09280 [I 2022-10-27 18:22:48] ...optimizers/trijoint.py.151: total_norm: 12.113687 [I 2022-10-27 18:22:54] ...trap/engines/engine.py.224: train: epoch 81 | batch 1590/2304 [I 2022-10-27 18:22:54] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:34 [I 2022-10-27 18:22:54] ...trap/engines/engine.py.229: process: 0.55101 | load: 0.00245 [I 2022-10-27 18:22:54] ...trap/engines/engine.py.230: loss: 0.11221 [I 2022-10-27 18:22:54] ...optimizers/trijoint.py.151: total_norm: 12.271723 [I 2022-10-27 18:22:59] ...trap/engines/engine.py.224: train: epoch 81 | batch 1600/2304 [I 2022-10-27 18:22:59] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:29 [I 2022-10-27 18:22:59] ...trap/engines/engine.py.229: process: 0.54939 | load: 0.00245 [I 2022-10-27 18:22:59] ...trap/engines/engine.py.230: loss: 0.12748 [I 2022-10-27 18:22:59] ...optimizers/trijoint.py.151: total_norm: 27.541847 [I 2022-10-27 18:23:05] ...trap/engines/engine.py.224: train: epoch 81 | batch 1610/2304 [I 2022-10-27 18:23:05] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:26 [I 2022-10-27 18:23:05] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00241 [I 2022-10-27 18:23:05] ...trap/engines/engine.py.230: loss: 0.12205 [I 2022-10-27 18:23:05] ...optimizers/trijoint.py.151: total_norm: 8.747271 [I 2022-10-27 18:23:11] ...trap/engines/engine.py.224: train: epoch 81 | batch 1620/2304 [I 2022-10-27 18:23:11] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:21 [I 2022-10-27 18:23:11] ...trap/engines/engine.py.229: process: 0.56426 | load: 0.00292 [I 2022-10-27 18:23:11] ...trap/engines/engine.py.230: loss: 0.09909 [I 2022-10-27 18:23:11] ...optimizers/trijoint.py.151: total_norm: 12.251481 [I 2022-10-27 18:23:16] ...trap/engines/engine.py.224: train: epoch 81 | batch 1630/2304 [I 2022-10-27 18:23:16] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:15 [I 2022-10-27 18:23:16] ...trap/engines/engine.py.229: process: 0.54575 | load: 0.00242 [I 2022-10-27 18:23:16] ...trap/engines/engine.py.230: loss: 0.12287 [I 2022-10-27 18:23:16] ...optimizers/trijoint.py.151: total_norm: 16.297462 [I 2022-10-27 18:23:22] ...trap/engines/engine.py.224: train: epoch 81 | batch 1640/2304 [I 2022-10-27 18:23:22] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:11 [I 2022-10-27 18:23:22] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00262 [I 2022-10-27 18:23:22] ...trap/engines/engine.py.230: loss: 0.11634 [I 2022-10-27 18:23:22] ...optimizers/trijoint.py.151: total_norm: 13.825393 [I 2022-10-27 18:23:27] ...trap/engines/engine.py.224: train: epoch 81 | batch 1650/2304 [I 2022-10-27 18:23:27] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:03 [I 2022-10-27 18:23:27] ...trap/engines/engine.py.229: process: 0.54635 | load: 0.00238 [I 2022-10-27 18:23:27] ...trap/engines/engine.py.230: loss: 0.13189 [I 2022-10-27 18:23:27] ...optimizers/trijoint.py.151: total_norm: 20.093008 [I 2022-10-27 18:23:33] ...trap/engines/engine.py.224: train: epoch 81 | batch 1660/2304 [I 2022-10-27 18:23:33] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:05:58 [I 2022-10-27 18:23:33] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00262 [I 2022-10-27 18:23:33] ...trap/engines/engine.py.230: loss: 0.18853 [I 2022-10-27 18:23:33] ...optimizers/trijoint.py.151: total_norm: 23.445293 [I 2022-10-27 18:23:38] ...trap/engines/engine.py.224: train: epoch 81 | batch 1670/2304 [I 2022-10-27 18:23:38] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:53 [I 2022-10-27 18:23:38] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00240 [I 2022-10-27 18:23:38] ...trap/engines/engine.py.230: loss: 0.20928 [I 2022-10-27 18:23:38] ...optimizers/trijoint.py.151: total_norm: 36.291290 [I 2022-10-27 18:23:44] ...trap/engines/engine.py.224: train: epoch 81 | batch 1680/2304 [I 2022-10-27 18:23:44] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:48 [I 2022-10-27 18:23:44] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00243 [I 2022-10-27 18:23:44] ...trap/engines/engine.py.230: loss: 0.09514 [I 2022-10-27 18:23:44] ...optimizers/trijoint.py.151: total_norm: 14.253161 [I 2022-10-27 18:23:50] ...trap/engines/engine.py.224: train: epoch 81 | batch 1690/2304 [I 2022-10-27 18:23:50] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:42 [I 2022-10-27 18:23:50] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00250 [I 2022-10-27 18:23:50] ...trap/engines/engine.py.230: loss: 0.17462 [I 2022-10-27 18:23:50] ...optimizers/trijoint.py.151: total_norm: 27.800188 [I 2022-10-27 18:23:55] ...trap/engines/engine.py.224: train: epoch 81 | batch 1700/2304 [I 2022-10-27 18:23:55] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:37 [I 2022-10-27 18:23:55] ...trap/engines/engine.py.229: process: 0.56313 | load: 0.00247 [I 2022-10-27 18:23:55] ...trap/engines/engine.py.230: loss: 0.17062 [I 2022-10-27 18:23:55] ...optimizers/trijoint.py.151: total_norm: 16.691378 [I 2022-10-27 18:24:01] ...trap/engines/engine.py.224: train: epoch 81 | batch 1710/2304 [I 2022-10-27 18:24:01] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:32 [I 2022-10-27 18:24:01] ...trap/engines/engine.py.229: process: 0.56798 | load: 0.00246 [I 2022-10-27 18:24:01] ...trap/engines/engine.py.230: loss: 0.18606 [I 2022-10-27 18:24:01] ...optimizers/trijoint.py.151: total_norm: 14.823278 [I 2022-10-27 18:24:06] ...trap/engines/engine.py.224: train: epoch 81 | batch 1720/2304 [I 2022-10-27 18:24:06] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:21 [I 2022-10-27 18:24:06] ...trap/engines/engine.py.229: process: 0.54883 | load: 0.00244 [I 2022-10-27 18:24:06] ...trap/engines/engine.py.230: loss: 0.10573 [I 2022-10-27 18:24:06] ...optimizers/trijoint.py.151: total_norm: 7.773461 [I 2022-10-27 18:24:12] ...trap/engines/engine.py.224: train: epoch 81 | batch 1730/2304 [I 2022-10-27 18:24:12] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:18 [I 2022-10-27 18:24:12] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00242 [I 2022-10-27 18:24:12] ...trap/engines/engine.py.230: loss: 0.16322 [I 2022-10-27 18:24:12] ...optimizers/trijoint.py.151: total_norm: 21.327789 [I 2022-10-27 18:24:18] ...trap/engines/engine.py.224: train: epoch 81 | batch 1740/2304 [I 2022-10-27 18:24:18] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:14 [I 2022-10-27 18:24:18] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00242 [I 2022-10-27 18:24:18] ...trap/engines/engine.py.230: loss: 0.14175 [I 2022-10-27 18:24:18] ...optimizers/trijoint.py.151: total_norm: 15.388084 [I 2022-10-27 18:24:23] ...trap/engines/engine.py.224: train: epoch 81 | batch 1750/2304 [I 2022-10-27 18:24:23] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:08 [I 2022-10-27 18:24:23] ...trap/engines/engine.py.229: process: 0.56513 | load: 0.00243 [I 2022-10-27 18:24:23] ...trap/engines/engine.py.230: loss: 0.11978 [I 2022-10-27 18:24:23] ...optimizers/trijoint.py.151: total_norm: 18.333788 [I 2022-10-27 18:24:29] ...trap/engines/engine.py.224: train: epoch 81 | batch 1760/2304 [I 2022-10-27 18:24:29] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:02 [I 2022-10-27 18:24:29] ...trap/engines/engine.py.229: process: 0.55844 | load: 0.00246 [I 2022-10-27 18:24:29] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-27 18:24:29] ...optimizers/trijoint.py.151: total_norm: 12.286595 [I 2022-10-27 18:24:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 1770/2304 [I 2022-10-27 18:24:34] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:04:57 [I 2022-10-27 18:24:34] ...trap/engines/engine.py.229: process: 0.54862 | load: 0.00247 [I 2022-10-27 18:24:34] ...trap/engines/engine.py.230: loss: 0.12584 [I 2022-10-27 18:24:34] ...optimizers/trijoint.py.151: total_norm: 8.093678 [I 2022-10-27 18:24:40] ...trap/engines/engine.py.224: train: epoch 81 | batch 1780/2304 [I 2022-10-27 18:24:40] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:04:53 [I 2022-10-27 18:24:40] ...trap/engines/engine.py.229: process: 0.56669 | load: 0.00274 [I 2022-10-27 18:24:40] ...trap/engines/engine.py.230: loss: 0.11402 [I 2022-10-27 18:24:40] ...optimizers/trijoint.py.151: total_norm: 23.103201 [I 2022-10-27 18:24:46] ...trap/engines/engine.py.224: train: epoch 81 | batch 1790/2304 [I 2022-10-27 18:24:46] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:48 [I 2022-10-27 18:24:46] ...trap/engines/engine.py.229: process: 0.56877 | load: 0.00248 [I 2022-10-27 18:24:46] ...trap/engines/engine.py.230: loss: 0.09648 [I 2022-10-27 18:24:46] ...optimizers/trijoint.py.151: total_norm: 10.612160 [I 2022-10-27 18:24:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 1800/2304 [I 2022-10-27 18:24:51] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:04:42 [I 2022-10-27 18:24:51] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00245 [I 2022-10-27 18:24:51] ...trap/engines/engine.py.230: loss: 0.08460 [I 2022-10-27 18:24:51] ...optimizers/trijoint.py.151: total_norm: 13.191226 [I 2022-10-27 18:24:57] ...trap/engines/engine.py.224: train: epoch 81 | batch 1810/2304 [I 2022-10-27 18:24:57] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:35 [I 2022-10-27 18:24:57] ...trap/engines/engine.py.229: process: 0.56998 | load: 0.00247 [I 2022-10-27 18:24:57] ...trap/engines/engine.py.230: loss: 0.11421 [I 2022-10-27 18:24:57] ...optimizers/trijoint.py.151: total_norm: 21.204027 [I 2022-10-27 18:25:02] ...trap/engines/engine.py.224: train: epoch 81 | batch 1820/2304 [I 2022-10-27 18:25:02] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:28 [I 2022-10-27 18:25:02] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00243 [I 2022-10-27 18:25:02] ...trap/engines/engine.py.230: loss: 0.11616 [I 2022-10-27 18:25:02] ...optimizers/trijoint.py.151: total_norm: 9.755261 [I 2022-10-27 18:25:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 1830/2304 [I 2022-10-27 18:25:08] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:25 [I 2022-10-27 18:25:08] ...trap/engines/engine.py.229: process: 0.56665 | load: 0.00247 [I 2022-10-27 18:25:08] ...trap/engines/engine.py.230: loss: 0.09549 [I 2022-10-27 18:25:08] ...optimizers/trijoint.py.151: total_norm: 13.169996 [I 2022-10-27 18:25:14] ...trap/engines/engine.py.224: train: epoch 81 | batch 1840/2304 [I 2022-10-27 18:25:14] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:20 [I 2022-10-27 18:25:14] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00243 [I 2022-10-27 18:25:14] ...trap/engines/engine.py.230: loss: 0.11671 [I 2022-10-27 18:25:14] ...optimizers/trijoint.py.151: total_norm: 13.657553 [I 2022-10-27 18:25:19] ...trap/engines/engine.py.224: train: epoch 81 | batch 1850/2304 [I 2022-10-27 18:25:19] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:14 [I 2022-10-27 18:25:19] ...trap/engines/engine.py.229: process: 0.57539 | load: 0.00247 [I 2022-10-27 18:25:19] ...trap/engines/engine.py.230: loss: 0.10454 [I 2022-10-27 18:25:19] ...optimizers/trijoint.py.151: total_norm: 14.877684 [I 2022-10-27 18:25:25] ...trap/engines/engine.py.224: train: epoch 81 | batch 1860/2304 [I 2022-10-27 18:25:25] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:07 [I 2022-10-27 18:25:25] ...trap/engines/engine.py.229: process: 0.54865 | load: 0.00237 [I 2022-10-27 18:25:25] ...trap/engines/engine.py.230: loss: 0.13236 [I 2022-10-27 18:25:25] ...optimizers/trijoint.py.151: total_norm: 11.690906 [I 2022-10-27 18:25:30] ...trap/engines/engine.py.224: train: epoch 81 | batch 1870/2304 [I 2022-10-27 18:25:30] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:02 [I 2022-10-27 18:25:30] ...trap/engines/engine.py.229: process: 0.57016 | load: 0.00253 [I 2022-10-27 18:25:30] ...trap/engines/engine.py.230: loss: 0.08881 [I 2022-10-27 18:25:30] ...optimizers/trijoint.py.151: total_norm: 14.973788 [I 2022-10-27 18:25:36] ...trap/engines/engine.py.224: train: epoch 81 | batch 1880/2304 [I 2022-10-27 18:25:36] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:03:56 [I 2022-10-27 18:25:36] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00246 [I 2022-10-27 18:25:36] ...trap/engines/engine.py.230: loss: 0.17060 [I 2022-10-27 18:25:36] ...optimizers/trijoint.py.151: total_norm: 24.714512 [I 2022-10-27 18:25:41] ...trap/engines/engine.py.224: train: epoch 81 | batch 1890/2304 [I 2022-10-27 18:25:41] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:03:50 [I 2022-10-27 18:25:41] ...trap/engines/engine.py.229: process: 0.55979 | load: 0.00245 [I 2022-10-27 18:25:41] ...trap/engines/engine.py.230: loss: 0.17943 [I 2022-10-27 18:25:41] ...optimizers/trijoint.py.151: total_norm: 38.305710 [I 2022-10-27 18:25:47] ...trap/engines/engine.py.224: train: epoch 81 | batch 1900/2304 [I 2022-10-27 18:25:47] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:03:46 [I 2022-10-27 18:25:47] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00245 [I 2022-10-27 18:25:47] ...trap/engines/engine.py.230: loss: 0.14365 [I 2022-10-27 18:25:47] ...optimizers/trijoint.py.151: total_norm: 57.888783 [I 2022-10-27 18:25:53] ...trap/engines/engine.py.224: train: epoch 81 | batch 1910/2304 [I 2022-10-27 18:25:53] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:39 [I 2022-10-27 18:25:53] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00247 [I 2022-10-27 18:25:53] ...trap/engines/engine.py.230: loss: 0.12998 [I 2022-10-27 18:25:53] ...optimizers/trijoint.py.151: total_norm: 16.784954 [I 2022-10-27 18:25:58] ...trap/engines/engine.py.224: train: epoch 81 | batch 1920/2304 [I 2022-10-27 18:25:58] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:33 [I 2022-10-27 18:25:58] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00243 [I 2022-10-27 18:25:58] ...trap/engines/engine.py.230: loss: 0.15901 [I 2022-10-27 18:25:58] ...optimizers/trijoint.py.151: total_norm: 15.137622 [I 2022-10-27 18:26:04] ...trap/engines/engine.py.224: train: epoch 81 | batch 1930/2304 [I 2022-10-27 18:26:04] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:30 [I 2022-10-27 18:26:04] ...trap/engines/engine.py.229: process: 0.57520 | load: 0.00250 [I 2022-10-27 18:26:04] ...trap/engines/engine.py.230: loss: 0.21891 [I 2022-10-27 18:26:04] ...optimizers/trijoint.py.151: total_norm: 17.099566 [I 2022-10-27 18:26:09] ...trap/engines/engine.py.224: train: epoch 81 | batch 1940/2304 [I 2022-10-27 18:26:09] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:24 [I 2022-10-27 18:26:09] ...trap/engines/engine.py.229: process: 0.56790 | load: 0.00251 [I 2022-10-27 18:26:09] ...trap/engines/engine.py.230: loss: 0.12032 [I 2022-10-27 18:26:09] ...optimizers/trijoint.py.151: total_norm: 27.255339 [I 2022-10-27 18:26:15] ...trap/engines/engine.py.224: train: epoch 81 | batch 1950/2304 [I 2022-10-27 18:26:15] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:16 [I 2022-10-27 18:26:15] ...trap/engines/engine.py.229: process: 0.56728 | load: 0.00244 [I 2022-10-27 18:26:15] ...trap/engines/engine.py.230: loss: 0.10455 [I 2022-10-27 18:26:15] ...optimizers/trijoint.py.151: total_norm: 16.926981 [I 2022-10-27 18:26:21] ...trap/engines/engine.py.224: train: epoch 81 | batch 1960/2304 [I 2022-10-27 18:26:21] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:11 [I 2022-10-27 18:26:21] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00247 [I 2022-10-27 18:26:21] ...trap/engines/engine.py.230: loss: 0.13428 [I 2022-10-27 18:26:21] ...optimizers/trijoint.py.151: total_norm: 16.343250 [I 2022-10-27 18:26:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 1970/2304 [I 2022-10-27 18:26:26] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:08 [I 2022-10-27 18:26:26] ...trap/engines/engine.py.229: process: 0.57681 | load: 0.00246 [I 2022-10-27 18:26:26] ...trap/engines/engine.py.230: loss: 0.10527 [I 2022-10-27 18:26:26] ...optimizers/trijoint.py.151: total_norm: 12.678383 [I 2022-10-27 18:26:32] ...trap/engines/engine.py.224: train: epoch 81 | batch 1980/2304 [I 2022-10-27 18:26:32] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:02 [I 2022-10-27 18:26:32] ...trap/engines/engine.py.229: process: 0.56449 | load: 0.00241 [I 2022-10-27 18:26:32] ...trap/engines/engine.py.230: loss: 0.11451 [I 2022-10-27 18:26:32] ...optimizers/trijoint.py.151: total_norm: 43.447392 [I 2022-10-27 18:26:38] ...trap/engines/engine.py.224: train: epoch 81 | batch 1990/2304 [I 2022-10-27 18:26:38] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:58 [I 2022-10-27 18:26:38] ...trap/engines/engine.py.229: process: 0.58501 | load: 0.00252 [I 2022-10-27 18:26:38] ...trap/engines/engine.py.230: loss: 0.15696 [I 2022-10-27 18:26:38] ...optimizers/trijoint.py.151: total_norm: 36.848244 [I 2022-10-27 18:26:43] ...trap/engines/engine.py.224: train: epoch 81 | batch 2000/2304 [I 2022-10-27 18:26:43] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:02:49 [I 2022-10-27 18:26:43] ...trap/engines/engine.py.229: process: 0.54325 | load: 0.00243 [I 2022-10-27 18:26:43] ...trap/engines/engine.py.230: loss: 0.13936 [I 2022-10-27 18:26:43] ...optimizers/trijoint.py.151: total_norm: 13.349954 [I 2022-10-27 18:26:49] ...trap/engines/engine.py.224: train: epoch 81 | batch 2010/2304 [I 2022-10-27 18:26:49] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:44 [I 2022-10-27 18:26:49] ...trap/engines/engine.py.229: process: 0.56884 | load: 0.00246 [I 2022-10-27 18:26:49] ...trap/engines/engine.py.230: loss: 0.16451 [I 2022-10-27 18:26:49] ...optimizers/trijoint.py.151: total_norm: 9.702552 [I 2022-10-27 18:26:54] ...trap/engines/engine.py.224: train: epoch 81 | batch 2020/2304 [I 2022-10-27 18:26:54] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:38 [I 2022-10-27 18:26:54] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00253 [I 2022-10-27 18:26:54] ...trap/engines/engine.py.230: loss: 0.28007 [I 2022-10-27 18:26:54] ...optimizers/trijoint.py.151: total_norm: 22.029911 [I 2022-10-27 18:27:00] ...trap/engines/engine.py.224: train: epoch 81 | batch 2030/2304 [I 2022-10-27 18:27:00] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:32 [I 2022-10-27 18:27:00] ...trap/engines/engine.py.229: process: 0.56498 | load: 0.00250 [I 2022-10-27 18:27:00] ...trap/engines/engine.py.230: loss: 0.22241 [I 2022-10-27 18:27:00] ...optimizers/trijoint.py.151: total_norm: 20.722555 [I 2022-10-27 18:27:06] ...trap/engines/engine.py.224: train: epoch 81 | batch 2040/2304 [I 2022-10-27 18:27:06] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:26 [I 2022-10-27 18:27:06] ...trap/engines/engine.py.229: process: 0.54465 | load: 0.00244 [I 2022-10-27 18:27:06] ...trap/engines/engine.py.230: loss: 0.13262 [I 2022-10-27 18:27:06] ...optimizers/trijoint.py.151: total_norm: 7.945984 [I 2022-10-27 18:27:11] ...trap/engines/engine.py.224: train: epoch 81 | batch 2050/2304 [I 2022-10-27 18:27:11] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:24 [I 2022-10-27 18:27:11] ...trap/engines/engine.py.229: process: 0.56765 | load: 0.00245 [I 2022-10-27 18:27:11] ...trap/engines/engine.py.230: loss: 0.11445 [I 2022-10-27 18:27:11] ...optimizers/trijoint.py.151: total_norm: 16.531937 [I 2022-10-27 18:27:17] ...trap/engines/engine.py.224: train: epoch 81 | batch 2060/2304 [I 2022-10-27 18:27:17] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:16 [I 2022-10-27 18:27:17] ...trap/engines/engine.py.229: process: 0.55057 | load: 0.00248 [I 2022-10-27 18:27:17] ...trap/engines/engine.py.230: loss: 0.10008 [I 2022-10-27 18:27:17] ...optimizers/trijoint.py.151: total_norm: 10.317423 [I 2022-10-27 18:27:23] ...trap/engines/engine.py.224: train: epoch 81 | batch 2070/2304 [I 2022-10-27 18:27:23] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:11 [I 2022-10-27 18:27:23] ...trap/engines/engine.py.229: process: 0.56819 | load: 0.00245 [I 2022-10-27 18:27:23] ...trap/engines/engine.py.230: loss: 0.12422 [I 2022-10-27 18:27:23] ...optimizers/trijoint.py.151: total_norm: 45.676849 [I 2022-10-27 18:27:28] ...trap/engines/engine.py.224: train: epoch 81 | batch 2080/2304 [I 2022-10-27 18:27:28] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:05 [I 2022-10-27 18:27:28] ...trap/engines/engine.py.229: process: 0.55512 | load: 0.00242 [I 2022-10-27 18:27:28] ...trap/engines/engine.py.230: loss: 0.12576 [I 2022-10-27 18:27:28] ...optimizers/trijoint.py.151: total_norm: 15.279770 [I 2022-10-27 18:27:34] ...trap/engines/engine.py.224: train: epoch 81 | batch 2090/2304 [I 2022-10-27 18:27:34] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:01:59 [I 2022-10-27 18:27:34] ...trap/engines/engine.py.229: process: 0.57726 | load: 0.00244 [I 2022-10-27 18:27:34] ...trap/engines/engine.py.230: loss: 0.13618 [I 2022-10-27 18:27:34] ...optimizers/trijoint.py.151: total_norm: 20.006531 [I 2022-10-27 18:27:39] ...trap/engines/engine.py.224: train: epoch 81 | batch 2100/2304 [I 2022-10-27 18:27:39] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:53 [I 2022-10-27 18:27:39] ...trap/engines/engine.py.229: process: 0.55954 | load: 0.00245 [I 2022-10-27 18:27:39] ...trap/engines/engine.py.230: loss: 0.15896 [I 2022-10-27 18:27:39] ...optimizers/trijoint.py.151: total_norm: 14.224683 [I 2022-10-27 18:27:45] ...trap/engines/engine.py.224: train: epoch 81 | batch 2110/2304 [I 2022-10-27 18:27:45] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:01:49 [I 2022-10-27 18:27:45] ...trap/engines/engine.py.229: process: 0.58036 | load: 0.00248 [I 2022-10-27 18:27:45] ...trap/engines/engine.py.230: loss: 0.22045 [I 2022-10-27 18:27:45] ...optimizers/trijoint.py.151: total_norm: 31.427546 [I 2022-10-27 18:27:51] ...trap/engines/engine.py.224: train: epoch 81 | batch 2120/2304 [I 2022-10-27 18:27:51] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:43 [I 2022-10-27 18:27:51] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00244 [I 2022-10-27 18:27:51] ...trap/engines/engine.py.230: loss: 0.15021 [I 2022-10-27 18:27:51] ...optimizers/trijoint.py.151: total_norm: 13.487779 [I 2022-10-27 18:27:56] ...trap/engines/engine.py.224: train: epoch 81 | batch 2130/2304 [I 2022-10-27 18:27:56] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:38 [I 2022-10-27 18:27:56] ...trap/engines/engine.py.229: process: 0.57547 | load: 0.00250 [I 2022-10-27 18:27:56] ...trap/engines/engine.py.230: loss: 0.11931 [I 2022-10-27 18:27:56] ...optimizers/trijoint.py.151: total_norm: 34.391304 [I 2022-10-27 18:28:02] ...trap/engines/engine.py.224: train: epoch 81 | batch 2140/2304 [I 2022-10-27 18:28:02] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:31 [I 2022-10-27 18:28:02] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00246 [I 2022-10-27 18:28:02] ...trap/engines/engine.py.230: loss: 0.14801 [I 2022-10-27 18:28:02] ...optimizers/trijoint.py.151: total_norm: 17.619339 [I 2022-10-27 18:28:08] ...trap/engines/engine.py.224: train: epoch 81 | batch 2150/2304 [I 2022-10-27 18:28:08] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:26 [I 2022-10-27 18:28:08] ...trap/engines/engine.py.229: process: 0.59174 | load: 0.00245 [I 2022-10-27 18:28:08] ...trap/engines/engine.py.230: loss: 0.09588 [I 2022-10-27 18:28:08] ...optimizers/trijoint.py.151: total_norm: 9.488948 [I 2022-10-27 18:28:13] ...trap/engines/engine.py.224: train: epoch 81 | batch 2160/2304 [I 2022-10-27 18:28:13] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:20 [I 2022-10-27 18:28:13] ...trap/engines/engine.py.229: process: 0.56375 | load: 0.00242 [I 2022-10-27 18:28:13] ...trap/engines/engine.py.230: loss: 0.17089 [I 2022-10-27 18:28:13] ...optimizers/trijoint.py.151: total_norm: 14.613379 [I 2022-10-27 18:28:19] ...trap/engines/engine.py.224: train: epoch 81 | batch 2170/2304 [I 2022-10-27 18:28:19] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:14 [I 2022-10-27 18:28:19] ...trap/engines/engine.py.229: process: 0.55400 | load: 0.00243 [I 2022-10-27 18:28:19] ...trap/engines/engine.py.230: loss: 0.21374 [I 2022-10-27 18:28:19] ...optimizers/trijoint.py.151: total_norm: 20.250435 [I 2022-10-27 18:28:24] ...trap/engines/engine.py.224: train: epoch 81 | batch 2180/2304 [I 2022-10-27 18:28:24] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:08 [I 2022-10-27 18:28:24] ...trap/engines/engine.py.229: process: 0.54521 | load: 0.00245 [I 2022-10-27 18:28:24] ...trap/engines/engine.py.230: loss: 0.10533 [I 2022-10-27 18:28:24] ...optimizers/trijoint.py.151: total_norm: 14.699615 [I 2022-10-27 18:28:30] ...trap/engines/engine.py.224: train: epoch 81 | batch 2190/2304 [I 2022-10-27 18:28:30] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:03 [I 2022-10-27 18:28:30] ...trap/engines/engine.py.229: process: 0.54156 | load: 0.00243 [I 2022-10-27 18:28:30] ...trap/engines/engine.py.230: loss: 0.12450 [I 2022-10-27 18:28:30] ...optimizers/trijoint.py.151: total_norm: 15.522331 [I 2022-10-27 18:28:35] ...trap/engines/engine.py.224: train: epoch 81 | batch 2200/2304 [I 2022-10-27 18:28:35] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:57 [I 2022-10-27 18:28:35] ...trap/engines/engine.py.229: process: 0.55923 | load: 0.00242 [I 2022-10-27 18:28:35] ...trap/engines/engine.py.230: loss: 0.11106 [I 2022-10-27 18:28:35] ...optimizers/trijoint.py.151: total_norm: 11.597647 [I 2022-10-27 18:28:41] ...trap/engines/engine.py.224: train: epoch 81 | batch 2210/2304 [I 2022-10-27 18:28:41] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:00:51 [I 2022-10-27 18:28:41] ...trap/engines/engine.py.229: process: 0.54694 | load: 0.00241 [I 2022-10-27 18:28:41] ...trap/engines/engine.py.230: loss: 0.12931 [I 2022-10-27 18:28:41] ...optimizers/trijoint.py.151: total_norm: 7.684389 [I 2022-10-27 18:28:47] ...trap/engines/engine.py.224: train: epoch 81 | batch 2220/2304 [I 2022-10-27 18:28:47] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:00:47 [I 2022-10-27 18:28:47] ...trap/engines/engine.py.229: process: 0.55162 | load: 0.00240 [I 2022-10-27 18:28:47] ...trap/engines/engine.py.230: loss: 0.19133 [I 2022-10-27 18:28:47] ...optimizers/trijoint.py.151: total_norm: 15.144191 [I 2022-10-27 18:28:52] ...trap/engines/engine.py.224: train: epoch 81 | batch 2230/2304 [I 2022-10-27 18:28:52] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:41 [I 2022-10-27 18:28:52] ...trap/engines/engine.py.229: process: 0.55520 | load: 0.00254 [I 2022-10-27 18:28:52] ...trap/engines/engine.py.230: loss: 0.21294 [I 2022-10-27 18:28:52] ...optimizers/trijoint.py.151: total_norm: 24.752563 [I 2022-10-27 18:28:58] ...trap/engines/engine.py.224: train: epoch 81 | batch 2240/2304 [I 2022-10-27 18:28:58] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:35 [I 2022-10-27 18:28:58] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00249 [I 2022-10-27 18:28:58] ...trap/engines/engine.py.230: loss: 0.18468 [I 2022-10-27 18:28:58] ...optimizers/trijoint.py.151: total_norm: 22.665342 [I 2022-10-27 18:29:03] ...trap/engines/engine.py.224: train: epoch 81 | batch 2250/2304 [I 2022-10-27 18:29:03] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:30 [I 2022-10-27 18:29:03] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00245 [I 2022-10-27 18:29:03] ...trap/engines/engine.py.230: loss: 0.15163 [I 2022-10-27 18:29:03] ...optimizers/trijoint.py.151: total_norm: 12.960290 [I 2022-10-27 18:29:09] ...trap/engines/engine.py.224: train: epoch 81 | batch 2260/2304 [I 2022-10-27 18:29:09] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:24 [I 2022-10-27 18:29:09] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00239 [I 2022-10-27 18:29:09] ...trap/engines/engine.py.230: loss: 0.14951 [I 2022-10-27 18:29:09] ...optimizers/trijoint.py.151: total_norm: 10.278214 [I 2022-10-27 18:29:15] ...trap/engines/engine.py.224: train: epoch 81 | batch 2270/2304 [I 2022-10-27 18:29:15] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:18 [I 2022-10-27 18:29:15] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00245 [I 2022-10-27 18:29:15] ...trap/engines/engine.py.230: loss: 0.09989 [I 2022-10-27 18:29:15] ...optimizers/trijoint.py.151: total_norm: 16.457211 [I 2022-10-27 18:29:20] ...trap/engines/engine.py.224: train: epoch 81 | batch 2280/2304 [I 2022-10-27 18:29:20] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:13 [I 2022-10-27 18:29:20] ...trap/engines/engine.py.229: process: 0.56181 | load: 0.00246 [I 2022-10-27 18:29:20] ...trap/engines/engine.py.230: loss: 0.09952 [I 2022-10-27 18:29:20] ...optimizers/trijoint.py.151: total_norm: 14.212938 [I 2022-10-27 18:29:26] ...trap/engines/engine.py.224: train: epoch 81 | batch 2290/2304 [I 2022-10-27 18:29:26] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:07 [I 2022-10-27 18:29:26] ...trap/engines/engine.py.229: process: 0.56752 | load: 0.00242 [I 2022-10-27 18:29:26] ...trap/engines/engine.py.230: loss: 0.14619 [I 2022-10-27 18:29:26] ...optimizers/trijoint.py.151: total_norm: 18.996565 [I 2022-10-27 18:29:31] ...trap/engines/engine.py.224: train: epoch 81 | batch 2300/2304 [I 2022-10-27 18:29:31] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:02 [I 2022-10-27 18:29:31] ...trap/engines/engine.py.229: process: 0.54400 | load: 0.00015 [I 2022-10-27 18:29:31] ...trap/engines/engine.py.230: loss: 0.09461 [I 2022-10-27 18:29:31] ...optimizers/trijoint.py.151: total_norm: 19.388536 [I 2022-10-27 18:29:33] ...trap/engines/engine.py.224: train: epoch 81 | batch 2304/2304 [I 2022-10-27 18:29:33] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:00 [I 2022-10-27 18:29:33] ...trap/engines/engine.py.229: process: 0.54766 | load: 0.00014 [I 2022-10-27 18:29:33] ...trap/engines/engine.py.230: loss: 0.11719 [I 2022-10-27 18:29:33] ...optimizers/trijoint.py.151: total_norm: 13.245823 [S 2022-10-27 18:29:35] ...trap/engines/engine.py.129: train_epoch.epoch: 81 [S 2022-10-27 18:29:35] ...trap/engines/engine.py.129: train_epoch.loss: 0.13625306136049573 [I 2022-10-27 18:29:35] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 18:29:36] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 81 [I 2022-10-27 18:29:36] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 18:29:38] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 18:29:38] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 18:29:38] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 18:29:40] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 18:29:41] ...trap/engines/engine.py.321: eval: epoch 81 | batch 0/510 [I 2022-10-27 18:29:41] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:41:16 [I 2022-10-27 18:29:41] ...trap/engines/engine.py.326: process: 4.85588 | load: 2.89985 [I 2022-10-27 18:29:44] ...trap/engines/engine.py.321: eval: epoch 81 | batch 10/510 [I 2022-10-27 18:29:44] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:57 [I 2022-10-27 18:29:44] ...trap/engines/engine.py.326: process: 0.32441 | load: 0.00017 [I 2022-10-27 18:29:47] ...trap/engines/engine.py.321: eval: epoch 81 | batch 20/510 [I 2022-10-27 18:29:47] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:09 [I 2022-10-27 18:29:47] ...trap/engines/engine.py.326: process: 0.32814 | load: 0.00018 [I 2022-10-27 18:29:51] ...trap/engines/engine.py.321: eval: epoch 81 | batch 30/510 [I 2022-10-27 18:29:51] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:45 [I 2022-10-27 18:29:51] ...trap/engines/engine.py.326: process: 0.34190 | load: 0.00017 [I 2022-10-27 18:29:54] ...trap/engines/engine.py.321: eval: epoch 81 | batch 40/510 [I 2022-10-27 18:29:54] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:39 [I 2022-10-27 18:29:54] ...trap/engines/engine.py.326: process: 0.32398 | load: 0.00017 [I 2022-10-27 18:29:58] ...trap/engines/engine.py.321: eval: epoch 81 | batch 50/510 [I 2022-10-27 18:29:58] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:32 [I 2022-10-27 18:29:58] ...trap/engines/engine.py.326: process: 0.33371 | load: 0.00018 [I 2022-10-27 18:30:01] ...trap/engines/engine.py.321: eval: epoch 81 | batch 60/510 [I 2022-10-27 18:30:01] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:29 [I 2022-10-27 18:30:01] ...trap/engines/engine.py.326: process: 0.33222 | load: 0.00017 [I 2022-10-27 18:30:04] ...trap/engines/engine.py.321: eval: epoch 81 | batch 70/510 [I 2022-10-27 18:30:04] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:27 [I 2022-10-27 18:30:04] ...trap/engines/engine.py.326: process: 0.32697 | load: 0.00016 [I 2022-10-27 18:30:08] ...trap/engines/engine.py.321: eval: epoch 81 | batch 80/510 [I 2022-10-27 18:30:08] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:22 [I 2022-10-27 18:30:08] ...trap/engines/engine.py.326: process: 0.32536 | load: 0.00016 [I 2022-10-27 18:30:11] ...trap/engines/engine.py.321: eval: epoch 81 | batch 90/510 [I 2022-10-27 18:30:11] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-27 18:30:11] ...trap/engines/engine.py.326: process: 0.32542 | load: 0.00021 [I 2022-10-27 18:30:14] ...trap/engines/engine.py.321: eval: epoch 81 | batch 100/510 [I 2022-10-27 18:30:14] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:15 [I 2022-10-27 18:30:14] ...trap/engines/engine.py.326: process: 0.28632 | load: 0.00023 [I 2022-10-27 18:30:17] ...trap/engines/engine.py.321: eval: epoch 81 | batch 110/510 [I 2022-10-27 18:30:17] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:58 [I 2022-10-27 18:30:17] ...trap/engines/engine.py.326: process: 0.30777 | load: 0.00015 [I 2022-10-27 18:30:20] ...trap/engines/engine.py.321: eval: epoch 81 | batch 120/510 [I 2022-10-27 18:30:20] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:53 [I 2022-10-27 18:30:20] ...trap/engines/engine.py.326: process: 0.28990 | load: 0.00018 [I 2022-10-27 18:30:23] ...trap/engines/engine.py.321: eval: epoch 81 | batch 130/510 [I 2022-10-27 18:30:23] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:50 [I 2022-10-27 18:30:23] ...trap/engines/engine.py.326: process: 0.28051 | load: 0.00019 [I 2022-10-27 18:30:26] ...trap/engines/engine.py.321: eval: epoch 81 | batch 140/510 [I 2022-10-27 18:30:26] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-27 18:30:26] ...trap/engines/engine.py.326: process: 0.28621 | load: 0.00018 [I 2022-10-27 18:30:29] ...trap/engines/engine.py.321: eval: epoch 81 | batch 150/510 [I 2022-10-27 18:30:29] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:44 [I 2022-10-27 18:30:29] ...trap/engines/engine.py.326: process: 0.28791 | load: 0.00018 [I 2022-10-27 18:30:32] ...trap/engines/engine.py.321: eval: epoch 81 | batch 160/510 [I 2022-10-27 18:30:32] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:42 [I 2022-10-27 18:30:32] ...trap/engines/engine.py.326: process: 0.29378 | load: 0.00018 [I 2022-10-27 18:30:35] ...trap/engines/engine.py.321: eval: epoch 81 | batch 170/510 [I 2022-10-27 18:30:35] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:37 [I 2022-10-27 18:30:35] ...trap/engines/engine.py.326: process: 0.27772 | load: 0.00014 [I 2022-10-27 18:30:37] ...trap/engines/engine.py.321: eval: epoch 81 | batch 180/510 [I 2022-10-27 18:30:37] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 18:30:37] ...trap/engines/engine.py.326: process: 0.29447 | load: 0.00016 [I 2022-10-27 18:30:40] ...trap/engines/engine.py.321: eval: epoch 81 | batch 190/510 [I 2022-10-27 18:30:40] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 18:30:40] ...trap/engines/engine.py.326: process: 0.28669 | load: 0.00019 [I 2022-10-27 18:30:43] ...trap/engines/engine.py.321: eval: epoch 81 | batch 200/510 [I 2022-10-27 18:30:43] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 18:30:43] ...trap/engines/engine.py.326: process: 0.28861 | load: 0.00023 [I 2022-10-27 18:30:46] ...trap/engines/engine.py.321: eval: epoch 81 | batch 210/510 [I 2022-10-27 18:30:46] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:27 [I 2022-10-27 18:30:46] ...trap/engines/engine.py.326: process: 0.29720 | load: 0.00017 [I 2022-10-27 18:30:49] ...trap/engines/engine.py.321: eval: epoch 81 | batch 220/510 [I 2022-10-27 18:30:49] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:24 [I 2022-10-27 18:30:49] ...trap/engines/engine.py.326: process: 0.29577 | load: 0.00016 [I 2022-10-27 18:30:52] ...trap/engines/engine.py.321: eval: epoch 81 | batch 230/510 [I 2022-10-27 18:30:52] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:22 [I 2022-10-27 18:30:52] ...trap/engines/engine.py.326: process: 0.28785 | load: 0.00027 [I 2022-10-27 18:30:55] ...trap/engines/engine.py.321: eval: epoch 81 | batch 240/510 [I 2022-10-27 18:30:55] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:21 [I 2022-10-27 18:30:55] ...trap/engines/engine.py.326: process: 0.28895 | load: 0.00023 [I 2022-10-27 18:30:58] ...trap/engines/engine.py.321: eval: epoch 81 | batch 250/510 [I 2022-10-27 18:30:58] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:16 [I 2022-10-27 18:30:58] ...trap/engines/engine.py.326: process: 0.28939 | load: 0.00018 [I 2022-10-27 18:31:01] ...trap/engines/engine.py.321: eval: epoch 81 | batch 260/510 [I 2022-10-27 18:31:01] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-27 18:31:01] ...trap/engines/engine.py.326: process: 0.28805 | load: 0.00017 [I 2022-10-27 18:31:04] ...trap/engines/engine.py.321: eval: epoch 81 | batch 270/510 [I 2022-10-27 18:31:04] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:08 [I 2022-10-27 18:31:04] ...trap/engines/engine.py.326: process: 0.28004 | load: 0.00023 [I 2022-10-27 18:31:07] ...trap/engines/engine.py.321: eval: epoch 81 | batch 280/510 [I 2022-10-27 18:31:07] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 18:31:07] ...trap/engines/engine.py.326: process: 0.29541 | load: 0.00023 [I 2022-10-27 18:31:10] ...trap/engines/engine.py.321: eval: epoch 81 | batch 290/510 [I 2022-10-27 18:31:10] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 18:31:10] ...trap/engines/engine.py.326: process: 0.28509 | load: 0.00041 [I 2022-10-27 18:31:12] ...trap/engines/engine.py.321: eval: epoch 81 | batch 300/510 [I 2022-10-27 18:31:12] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 18:31:12] ...trap/engines/engine.py.326: process: 0.29470 | load: 0.00030 [I 2022-10-27 18:31:15] ...trap/engines/engine.py.321: eval: epoch 81 | batch 310/510 [I 2022-10-27 18:31:15] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 18:31:15] ...trap/engines/engine.py.326: process: 0.28771 | load: 0.00016 [I 2022-10-27 18:31:18] ...trap/engines/engine.py.321: eval: epoch 81 | batch 320/510 [I 2022-10-27 18:31:18] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 18:31:18] ...trap/engines/engine.py.326: process: 0.29152 | load: 0.00025 [I 2022-10-27 18:31:21] ...trap/engines/engine.py.321: eval: epoch 81 | batch 330/510 [I 2022-10-27 18:31:21] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-27 18:31:21] ...trap/engines/engine.py.326: process: 0.28625 | load: 0.00016 [I 2022-10-27 18:31:24] ...trap/engines/engine.py.321: eval: epoch 81 | batch 340/510 [I 2022-10-27 18:31:24] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-27 18:31:24] ...trap/engines/engine.py.326: process: 0.28068 | load: 0.00016 [I 2022-10-27 18:31:27] ...trap/engines/engine.py.321: eval: epoch 81 | batch 350/510 [I 2022-10-27 18:31:27] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 18:31:27] ...trap/engines/engine.py.326: process: 0.28537 | load: 0.00015 [I 2022-10-27 18:31:30] ...trap/engines/engine.py.321: eval: epoch 81 | batch 360/510 [I 2022-10-27 18:31:30] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-27 18:31:30] ...trap/engines/engine.py.326: process: 0.28467 | load: 0.00019 [I 2022-10-27 18:31:33] ...trap/engines/engine.py.321: eval: epoch 81 | batch 370/510 [I 2022-10-27 18:31:33] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 18:31:33] ...trap/engines/engine.py.326: process: 0.28602 | load: 0.00019 [I 2022-10-27 18:31:36] ...trap/engines/engine.py.321: eval: epoch 81 | batch 380/510 [I 2022-10-27 18:31:36] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 18:31:36] ...trap/engines/engine.py.326: process: 0.28881 | load: 0.00015 [I 2022-10-27 18:31:38] ...trap/engines/engine.py.321: eval: epoch 81 | batch 390/510 [I 2022-10-27 18:31:38] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 18:31:38] ...trap/engines/engine.py.326: process: 0.28555 | load: 0.00016 [I 2022-10-27 18:31:41] ...trap/engines/engine.py.321: eval: epoch 81 | batch 400/510 [I 2022-10-27 18:31:41] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 18:31:41] ...trap/engines/engine.py.326: process: 0.28287 | load: 0.00016 [I 2022-10-27 18:31:44] ...trap/engines/engine.py.321: eval: epoch 81 | batch 410/510 [I 2022-10-27 18:31:44] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 18:31:44] ...trap/engines/engine.py.326: process: 0.28803 | load: 0.00015 [I 2022-10-27 18:31:47] ...trap/engines/engine.py.321: eval: epoch 81 | batch 420/510 [I 2022-10-27 18:31:47] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 18:31:47] ...trap/engines/engine.py.326: process: 0.29074 | load: 0.00019 [I 2022-10-27 18:31:50] ...trap/engines/engine.py.321: eval: epoch 81 | batch 430/510 [I 2022-10-27 18:31:50] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:22 [I 2022-10-27 18:31:50] ...trap/engines/engine.py.326: process: 0.28645 | load: 0.00016 [I 2022-10-27 18:31:53] ...trap/engines/engine.py.321: eval: epoch 81 | batch 440/510 [I 2022-10-27 18:31:53] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 18:31:53] ...trap/engines/engine.py.326: process: 0.28804 | load: 0.00015 [I 2022-10-27 18:31:56] ...trap/engines/engine.py.321: eval: epoch 81 | batch 450/510 [I 2022-10-27 18:31:56] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 18:31:56] ...trap/engines/engine.py.326: process: 0.28978 | load: 0.00016 [I 2022-10-27 18:31:59] ...trap/engines/engine.py.321: eval: epoch 81 | batch 460/510 [I 2022-10-27 18:31:59] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 18:31:59] ...trap/engines/engine.py.326: process: 0.28759 | load: 0.00021 [I 2022-10-27 18:32:01] ...trap/engines/engine.py.321: eval: epoch 81 | batch 470/510 [I 2022-10-27 18:32:01] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 18:32:01] ...trap/engines/engine.py.326: process: 0.28834 | load: 0.00019 [I 2022-10-27 18:32:04] ...trap/engines/engine.py.321: eval: epoch 81 | batch 480/510 [I 2022-10-27 18:32:04] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 18:32:04] ...trap/engines/engine.py.326: process: 0.28832 | load: 0.00021 [I 2022-10-27 18:32:07] ...trap/engines/engine.py.321: eval: epoch 81 | batch 490/510 [I 2022-10-27 18:32:07] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 18:32:07] ...trap/engines/engine.py.326: process: 0.28712 | load: 0.00024 [I 2022-10-27 18:32:10] ...trap/engines/engine.py.321: eval: epoch 81 | batch 500/510 [I 2022-10-27 18:32:10] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 18:32:10] ...trap/engines/engine.py.326: process: 0.28876 | load: 0.00015 [I 2022-10-27 18:32:13] ...trap/engines/engine.py.321: eval: epoch 81 | batch 510/510 [I 2022-10-27 18:32:13] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 18:32:13] ...trap/engines/engine.py.326: process: 0.28317 | load: 0.00013 [S 2022-10-27 18:32:14] ...trap/engines/engine.py.132: eval_epoch.epoch: 81 [I 2022-10-27 18:32:14] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.724700 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.900900 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.932300 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.717600 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.900300 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931500 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016000 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009617 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008149 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012635 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010706 [S 2022-10-27 18:32:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008925 [I 2022-10-27 18:32:19] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 18:32:19] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 18:32:20] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 18:32:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 18:32:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 18:32:24] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 18:32:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 18:32:30] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 18:32:30] ...trap/engines/engine.py.163: Training model on trainset for epoch 82 [I 2022-10-27 18:32:30] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 18:32:33] ...trap/engines/engine.py.224: train: epoch 82 | batch 0/2304 [I 2022-10-27 18:32:33] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:40:41 [I 2022-10-27 18:32:33] ...trap/engines/engine.py.229: process: 2.62215 | load: 1.98468 [I 2022-10-27 18:32:33] ...trap/engines/engine.py.230: loss: 0.12556 [I 2022-10-27 18:32:33] ...optimizers/trijoint.py.151: total_norm: 11.414777 [I 2022-10-27 18:32:38] ...trap/engines/engine.py.224: train: epoch 82 | batch 10/2304 [I 2022-10-27 18:32:38] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:45 [I 2022-10-27 18:32:38] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00406 [I 2022-10-27 18:32:38] ...trap/engines/engine.py.230: loss: 0.12671 [I 2022-10-27 18:32:38] ...optimizers/trijoint.py.151: total_norm: 16.327454 [I 2022-10-27 18:32:44] ...trap/engines/engine.py.224: train: epoch 82 | batch 20/2304 [I 2022-10-27 18:32:44] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:07 [I 2022-10-27 18:32:44] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00396 [I 2022-10-27 18:32:44] ...trap/engines/engine.py.230: loss: 0.09239 [I 2022-10-27 18:32:44] ...optimizers/trijoint.py.151: total_norm: 7.321639 [I 2022-10-27 18:32:50] ...trap/engines/engine.py.224: train: epoch 82 | batch 30/2304 [I 2022-10-27 18:32:50] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:18 [I 2022-10-27 18:32:50] ...trap/engines/engine.py.229: process: 0.57242 | load: 0.00385 [I 2022-10-27 18:32:50] ...trap/engines/engine.py.230: loss: 0.16857 [I 2022-10-27 18:32:50] ...optimizers/trijoint.py.151: total_norm: 20.352676 [I 2022-10-27 18:32:55] ...trap/engines/engine.py.224: train: epoch 82 | batch 40/2304 [I 2022-10-27 18:32:55] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:16 [I 2022-10-27 18:32:55] ...trap/engines/engine.py.229: process: 0.57207 | load: 0.00291 [I 2022-10-27 18:32:55] ...trap/engines/engine.py.230: loss: 0.15567 [I 2022-10-27 18:32:55] ...optimizers/trijoint.py.151: total_norm: 34.487823 [I 2022-10-27 18:33:01] ...trap/engines/engine.py.224: train: epoch 82 | batch 50/2304 [I 2022-10-27 18:33:01] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:15 [I 2022-10-27 18:33:01] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00348 [I 2022-10-27 18:33:01] ...trap/engines/engine.py.230: loss: 0.19669 [I 2022-10-27 18:33:01] ...optimizers/trijoint.py.151: total_norm: 20.974115 [I 2022-10-27 18:33:06] ...trap/engines/engine.py.224: train: epoch 82 | batch 60/2304 [I 2022-10-27 18:33:06] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:44 [I 2022-10-27 18:33:06] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00292 [I 2022-10-27 18:33:06] ...trap/engines/engine.py.230: loss: 0.13777 [I 2022-10-27 18:33:06] ...optimizers/trijoint.py.151: total_norm: 9.634234 [I 2022-10-27 18:33:12] ...trap/engines/engine.py.224: train: epoch 82 | batch 70/2304 [I 2022-10-27 18:33:12] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:21:01 [I 2022-10-27 18:33:12] ...trap/engines/engine.py.229: process: 0.57438 | load: 0.00261 [I 2022-10-27 18:33:12] ...trap/engines/engine.py.230: loss: 0.15239 [I 2022-10-27 18:33:12] ...optimizers/trijoint.py.151: total_norm: 23.597380 [I 2022-10-27 18:33:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 80/2304 [I 2022-10-27 18:33:18] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:41 [I 2022-10-27 18:33:18] ...trap/engines/engine.py.229: process: 0.55177 | load: 0.00289 [I 2022-10-27 18:33:18] ...trap/engines/engine.py.230: loss: 0.15449 [I 2022-10-27 18:33:18] ...optimizers/trijoint.py.151: total_norm: 14.578006 [I 2022-10-27 18:33:23] ...trap/engines/engine.py.224: train: epoch 82 | batch 90/2304 [I 2022-10-27 18:33:23] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:41 [I 2022-10-27 18:33:23] ...trap/engines/engine.py.229: process: 0.56372 | load: 0.00261 [I 2022-10-27 18:33:23] ...trap/engines/engine.py.230: loss: 0.12622 [I 2022-10-27 18:33:23] ...optimizers/trijoint.py.151: total_norm: 14.208178 [I 2022-10-27 18:33:29] ...trap/engines/engine.py.224: train: epoch 82 | batch 100/2304 [I 2022-10-27 18:33:29] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:20 [I 2022-10-27 18:33:29] ...trap/engines/engine.py.229: process: 0.55118 | load: 0.00272 [I 2022-10-27 18:33:29] ...trap/engines/engine.py.230: loss: 0.26611 [I 2022-10-27 18:33:29] ...optimizers/trijoint.py.151: total_norm: 19.424368 [I 2022-10-27 18:33:34] ...trap/engines/engine.py.224: train: epoch 82 | batch 110/2304 [I 2022-10-27 18:33:34] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:28 [I 2022-10-27 18:33:34] ...trap/engines/engine.py.229: process: 0.57260 | load: 0.00270 [I 2022-10-27 18:33:34] ...trap/engines/engine.py.230: loss: 0.10248 [I 2022-10-27 18:33:34] ...optimizers/trijoint.py.151: total_norm: 11.281222 [I 2022-10-27 18:33:40] ...trap/engines/engine.py.224: train: epoch 82 | batch 120/2304 [I 2022-10-27 18:33:40] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:11 [I 2022-10-27 18:33:40] ...trap/engines/engine.py.229: process: 0.55039 | load: 0.00352 [I 2022-10-27 18:33:40] ...trap/engines/engine.py.230: loss: 0.10939 [I 2022-10-27 18:33:40] ...optimizers/trijoint.py.151: total_norm: 16.738726 [I 2022-10-27 18:33:46] ...trap/engines/engine.py.224: train: epoch 82 | batch 130/2304 [I 2022-10-27 18:33:46] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:06 [I 2022-10-27 18:33:46] ...trap/engines/engine.py.229: process: 0.55065 | load: 0.00261 [I 2022-10-27 18:33:46] ...trap/engines/engine.py.230: loss: 0.17203 [I 2022-10-27 18:33:46] ...optimizers/trijoint.py.151: total_norm: 18.553654 [I 2022-10-27 18:33:51] ...trap/engines/engine.py.224: train: epoch 82 | batch 140/2304 [I 2022-10-27 18:33:51] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:06 [I 2022-10-27 18:33:51] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00280 [I 2022-10-27 18:33:51] ...trap/engines/engine.py.230: loss: 0.10353 [I 2022-10-27 18:33:51] ...optimizers/trijoint.py.151: total_norm: 9.495938 [I 2022-10-27 18:33:57] ...trap/engines/engine.py.224: train: epoch 82 | batch 150/2304 [I 2022-10-27 18:33:57] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:00 [I 2022-10-27 18:33:57] ...trap/engines/engine.py.229: process: 0.56343 | load: 0.00261 [I 2022-10-27 18:33:57] ...trap/engines/engine.py.230: loss: 0.13061 [I 2022-10-27 18:33:57] ...optimizers/trijoint.py.151: total_norm: 19.035501 [I 2022-10-27 18:34:02] ...trap/engines/engine.py.224: train: epoch 82 | batch 160/2304 [I 2022-10-27 18:34:02] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:59 [I 2022-10-27 18:34:02] ...trap/engines/engine.py.229: process: 0.56182 | load: 0.00260 [I 2022-10-27 18:34:02] ...trap/engines/engine.py.230: loss: 0.10479 [I 2022-10-27 18:34:02] ...optimizers/trijoint.py.151: total_norm: 10.542739 [I 2022-10-27 18:34:08] ...trap/engines/engine.py.224: train: epoch 82 | batch 170/2304 [I 2022-10-27 18:34:08] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:39 [I 2022-10-27 18:34:08] ...trap/engines/engine.py.229: process: 0.54174 | load: 0.00241 [I 2022-10-27 18:34:08] ...trap/engines/engine.py.230: loss: 0.10091 [I 2022-10-27 18:34:08] ...optimizers/trijoint.py.151: total_norm: 6.652479 [I 2022-10-27 18:34:13] ...trap/engines/engine.py.224: train: epoch 82 | batch 180/2304 [I 2022-10-27 18:34:13] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:43 [I 2022-10-27 18:34:13] ...trap/engines/engine.py.229: process: 0.57880 | load: 0.00256 [I 2022-10-27 18:34:13] ...trap/engines/engine.py.230: loss: 0.13369 [I 2022-10-27 18:34:13] ...optimizers/trijoint.py.151: total_norm: 12.177410 [I 2022-10-27 18:34:19] ...trap/engines/engine.py.224: train: epoch 82 | batch 190/2304 [I 2022-10-27 18:34:19] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:39 [I 2022-10-27 18:34:19] ...trap/engines/engine.py.229: process: 0.55436 | load: 0.00258 [I 2022-10-27 18:34:19] ...trap/engines/engine.py.230: loss: 0.12668 [I 2022-10-27 18:34:19] ...optimizers/trijoint.py.151: total_norm: 17.298733 [I 2022-10-27 18:34:25] ...trap/engines/engine.py.224: train: epoch 82 | batch 200/2304 [I 2022-10-27 18:34:25] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:32 [I 2022-10-27 18:34:25] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00250 [I 2022-10-27 18:34:25] ...trap/engines/engine.py.230: loss: 0.12020 [I 2022-10-27 18:34:25] ...optimizers/trijoint.py.151: total_norm: 19.362421 [I 2022-10-27 18:34:30] ...trap/engines/engine.py.224: train: epoch 82 | batch 210/2304 [I 2022-10-27 18:34:30] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:27 [I 2022-10-27 18:34:30] ...trap/engines/engine.py.229: process: 0.57107 | load: 0.00253 [I 2022-10-27 18:34:30] ...trap/engines/engine.py.230: loss: 0.14842 [I 2022-10-27 18:34:30] ...optimizers/trijoint.py.151: total_norm: 13.139929 [I 2022-10-27 18:34:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 220/2304 [I 2022-10-27 18:34:36] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:13 [I 2022-10-27 18:34:36] ...trap/engines/engine.py.229: process: 0.55205 | load: 0.00262 [I 2022-10-27 18:34:36] ...trap/engines/engine.py.230: loss: 0.16720 [I 2022-10-27 18:34:36] ...optimizers/trijoint.py.151: total_norm: 17.169910 [I 2022-10-27 18:34:41] ...trap/engines/engine.py.224: train: epoch 82 | batch 230/2304 [I 2022-10-27 18:34:41] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:16 [I 2022-10-27 18:34:41] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00241 [I 2022-10-27 18:34:41] ...trap/engines/engine.py.230: loss: 0.11615 [I 2022-10-27 18:34:41] ...optimizers/trijoint.py.151: total_norm: 17.318571 [I 2022-10-27 18:34:47] ...trap/engines/engine.py.224: train: epoch 82 | batch 240/2304 [I 2022-10-27 18:34:47] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:25 [I 2022-10-27 18:34:47] ...trap/engines/engine.py.229: process: 0.56980 | load: 0.00247 [I 2022-10-27 18:34:47] ...trap/engines/engine.py.230: loss: 0.20450 [I 2022-10-27 18:34:47] ...optimizers/trijoint.py.151: total_norm: 19.468058 [I 2022-10-27 18:34:52] ...trap/engines/engine.py.224: train: epoch 82 | batch 250/2304 [I 2022-10-27 18:34:52] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:07 [I 2022-10-27 18:34:52] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00252 [I 2022-10-27 18:34:52] ...trap/engines/engine.py.230: loss: 0.22712 [I 2022-10-27 18:34:52] ...optimizers/trijoint.py.151: total_norm: 19.500713 [I 2022-10-27 18:34:58] ...trap/engines/engine.py.224: train: epoch 82 | batch 260/2304 [I 2022-10-27 18:34:58] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:01 [I 2022-10-27 18:34:58] ...trap/engines/engine.py.229: process: 0.56745 | load: 0.00249 [I 2022-10-27 18:34:58] ...trap/engines/engine.py.230: loss: 0.15435 [I 2022-10-27 18:34:58] ...optimizers/trijoint.py.151: total_norm: 14.195195 [I 2022-10-27 18:35:04] ...trap/engines/engine.py.224: train: epoch 82 | batch 270/2304 [I 2022-10-27 18:35:04] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:54 [I 2022-10-27 18:35:04] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00251 [I 2022-10-27 18:35:04] ...trap/engines/engine.py.230: loss: 0.11529 [I 2022-10-27 18:35:04] ...optimizers/trijoint.py.151: total_norm: 12.902761 [I 2022-10-27 18:35:09] ...trap/engines/engine.py.224: train: epoch 82 | batch 280/2304 [I 2022-10-27 18:35:09] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:50 [I 2022-10-27 18:35:09] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00247 [I 2022-10-27 18:35:09] ...trap/engines/engine.py.230: loss: 0.08734 [I 2022-10-27 18:35:09] ...optimizers/trijoint.py.151: total_norm: 14.775017 [I 2022-10-27 18:35:15] ...trap/engines/engine.py.224: train: epoch 82 | batch 290/2304 [I 2022-10-27 18:35:15] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:39 [I 2022-10-27 18:35:15] ...trap/engines/engine.py.229: process: 0.56332 | load: 0.00254 [I 2022-10-27 18:35:15] ...trap/engines/engine.py.230: loss: 0.08895 [I 2022-10-27 18:35:15] ...optimizers/trijoint.py.151: total_norm: 9.605169 [I 2022-10-27 18:35:20] ...trap/engines/engine.py.224: train: epoch 82 | batch 300/2304 [I 2022-10-27 18:35:20] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:36 [I 2022-10-27 18:35:20] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00246 [I 2022-10-27 18:35:20] ...trap/engines/engine.py.230: loss: 0.15819 [I 2022-10-27 18:35:20] ...optimizers/trijoint.py.151: total_norm: 20.215305 [I 2022-10-27 18:35:26] ...trap/engines/engine.py.224: train: epoch 82 | batch 310/2304 [I 2022-10-27 18:35:26] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:44 [I 2022-10-27 18:35:26] ...trap/engines/engine.py.229: process: 0.57238 | load: 0.00261 [I 2022-10-27 18:35:26] ...trap/engines/engine.py.230: loss: 0.12154 [I 2022-10-27 18:35:26] ...optimizers/trijoint.py.151: total_norm: 18.706905 [I 2022-10-27 18:35:32] ...trap/engines/engine.py.224: train: epoch 82 | batch 320/2304 [I 2022-10-27 18:35:32] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:31 [I 2022-10-27 18:35:32] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00266 [I 2022-10-27 18:35:32] ...trap/engines/engine.py.230: loss: 0.10617 [I 2022-10-27 18:35:32] ...optimizers/trijoint.py.151: total_norm: 20.262175 [I 2022-10-27 18:35:37] ...trap/engines/engine.py.224: train: epoch 82 | batch 330/2304 [I 2022-10-27 18:35:37] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:21 [I 2022-10-27 18:35:37] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00279 [I 2022-10-27 18:35:37] ...trap/engines/engine.py.230: loss: 0.13988 [I 2022-10-27 18:35:37] ...optimizers/trijoint.py.151: total_norm: 12.957644 [I 2022-10-27 18:35:43] ...trap/engines/engine.py.224: train: epoch 82 | batch 340/2304 [I 2022-10-27 18:35:43] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:18 [I 2022-10-27 18:35:43] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00275 [I 2022-10-27 18:35:43] ...trap/engines/engine.py.230: loss: 0.16120 [I 2022-10-27 18:35:43] ...optimizers/trijoint.py.151: total_norm: 28.541862 [I 2022-10-27 18:35:48] ...trap/engines/engine.py.224: train: epoch 82 | batch 350/2304 [I 2022-10-27 18:35:48] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:00 [I 2022-10-27 18:35:48] ...trap/engines/engine.py.229: process: 0.54996 | load: 0.00266 [I 2022-10-27 18:35:48] ...trap/engines/engine.py.230: loss: 0.16509 [I 2022-10-27 18:35:48] ...optimizers/trijoint.py.151: total_norm: 25.238216 [I 2022-10-27 18:35:54] ...trap/engines/engine.py.224: train: epoch 82 | batch 360/2304 [I 2022-10-27 18:35:54] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:08 [I 2022-10-27 18:35:54] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00271 [I 2022-10-27 18:35:54] ...trap/engines/engine.py.230: loss: 0.09449 [I 2022-10-27 18:35:54] ...optimizers/trijoint.py.151: total_norm: 12.095572 [I 2022-10-27 18:35:59] ...trap/engines/engine.py.224: train: epoch 82 | batch 370/2304 [I 2022-10-27 18:35:59] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:49 [I 2022-10-27 18:35:59] ...trap/engines/engine.py.229: process: 0.54712 | load: 0.00248 [I 2022-10-27 18:35:59] ...trap/engines/engine.py.230: loss: 0.12320 [I 2022-10-27 18:35:59] ...optimizers/trijoint.py.151: total_norm: 10.238222 [I 2022-10-27 18:36:05] ...trap/engines/engine.py.224: train: epoch 82 | batch 380/2304 [I 2022-10-27 18:36:05] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:50 [I 2022-10-27 18:36:05] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00243 [I 2022-10-27 18:36:05] ...trap/engines/engine.py.230: loss: 0.16851 [I 2022-10-27 18:36:05] ...optimizers/trijoint.py.151: total_norm: 13.628244 [I 2022-10-27 18:36:11] ...trap/engines/engine.py.224: train: epoch 82 | batch 390/2304 [I 2022-10-27 18:36:11] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:52 [I 2022-10-27 18:36:11] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00262 [I 2022-10-27 18:36:11] ...trap/engines/engine.py.230: loss: 0.10373 [I 2022-10-27 18:36:11] ...optimizers/trijoint.py.151: total_norm: 9.075853 [I 2022-10-27 18:36:16] ...trap/engines/engine.py.224: train: epoch 82 | batch 400/2304 [I 2022-10-27 18:36:16] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:42 [I 2022-10-27 18:36:16] ...trap/engines/engine.py.229: process: 0.54903 | load: 0.00247 [I 2022-10-27 18:36:16] ...trap/engines/engine.py.230: loss: 0.10685 [I 2022-10-27 18:36:16] ...optimizers/trijoint.py.151: total_norm: 8.859465 [I 2022-10-27 18:36:22] ...trap/engines/engine.py.224: train: epoch 82 | batch 410/2304 [I 2022-10-27 18:36:22] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:18:07 [I 2022-10-27 18:36:22] ...trap/engines/engine.py.229: process: 0.56585 | load: 0.00258 [I 2022-10-27 18:36:22] ...trap/engines/engine.py.230: loss: 0.09168 [I 2022-10-27 18:36:22] ...optimizers/trijoint.py.151: total_norm: 10.763005 [I 2022-10-27 18:36:28] ...trap/engines/engine.py.224: train: epoch 82 | batch 420/2304 [I 2022-10-27 18:36:28] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:27 [I 2022-10-27 18:36:28] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00250 [I 2022-10-27 18:36:28] ...trap/engines/engine.py.230: loss: 0.17927 [I 2022-10-27 18:36:28] ...optimizers/trijoint.py.151: total_norm: 9.631213 [I 2022-10-27 18:36:33] ...trap/engines/engine.py.224: train: epoch 82 | batch 430/2304 [I 2022-10-27 18:36:33] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:49 [I 2022-10-27 18:36:33] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00240 [I 2022-10-27 18:36:33] ...trap/engines/engine.py.230: loss: 0.11687 [I 2022-10-27 18:36:33] ...optimizers/trijoint.py.151: total_norm: 11.009953 [I 2022-10-27 18:36:39] ...trap/engines/engine.py.224: train: epoch 82 | batch 440/2304 [I 2022-10-27 18:36:39] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:24 [I 2022-10-27 18:36:39] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00246 [I 2022-10-27 18:36:39] ...trap/engines/engine.py.230: loss: 0.12759 [I 2022-10-27 18:36:39] ...optimizers/trijoint.py.151: total_norm: 28.242458 [I 2022-10-27 18:36:44] ...trap/engines/engine.py.224: train: epoch 82 | batch 450/2304 [I 2022-10-27 18:36:44] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:11 [I 2022-10-27 18:36:44] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00242 [I 2022-10-27 18:36:44] ...trap/engines/engine.py.230: loss: 0.12704 [I 2022-10-27 18:36:44] ...optimizers/trijoint.py.151: total_norm: 11.774274 [I 2022-10-27 18:36:50] ...trap/engines/engine.py.224: train: epoch 82 | batch 460/2304 [I 2022-10-27 18:36:50] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:03 [I 2022-10-27 18:36:50] ...trap/engines/engine.py.229: process: 0.54526 | load: 0.00244 [I 2022-10-27 18:36:50] ...trap/engines/engine.py.230: loss: 0.17084 [I 2022-10-27 18:36:50] ...optimizers/trijoint.py.151: total_norm: 11.871396 [I 2022-10-27 18:36:56] ...trap/engines/engine.py.224: train: epoch 82 | batch 470/2304 [I 2022-10-27 18:36:56] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:06 [I 2022-10-27 18:36:56] ...trap/engines/engine.py.229: process: 0.56292 | load: 0.00256 [I 2022-10-27 18:36:56] ...trap/engines/engine.py.230: loss: 0.22007 [I 2022-10-27 18:36:56] ...optimizers/trijoint.py.151: total_norm: 36.923820 [I 2022-10-27 18:37:01] ...trap/engines/engine.py.224: train: epoch 82 | batch 480/2304 [I 2022-10-27 18:37:01] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:16:58 [I 2022-10-27 18:37:01] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00247 [I 2022-10-27 18:37:01] ...trap/engines/engine.py.230: loss: 0.28109 [I 2022-10-27 18:37:01] ...optimizers/trijoint.py.151: total_norm: 23.735571 [I 2022-10-27 18:37:07] ...trap/engines/engine.py.224: train: epoch 82 | batch 490/2304 [I 2022-10-27 18:37:07] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:44 [I 2022-10-27 18:37:07] ...trap/engines/engine.py.229: process: 0.55395 | load: 0.00247 [I 2022-10-27 18:37:07] ...trap/engines/engine.py.230: loss: 0.14980 [I 2022-10-27 18:37:07] ...optimizers/trijoint.py.151: total_norm: 16.682894 [I 2022-10-27 18:37:12] ...trap/engines/engine.py.224: train: epoch 82 | batch 500/2304 [I 2022-10-27 18:37:12] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:55 [I 2022-10-27 18:37:12] ...trap/engines/engine.py.229: process: 0.54905 | load: 0.00243 [I 2022-10-27 18:37:12] ...trap/engines/engine.py.230: loss: 0.18678 [I 2022-10-27 18:37:12] ...optimizers/trijoint.py.151: total_norm: 29.445683 [I 2022-10-27 18:37:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 510/2304 [I 2022-10-27 18:37:18] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:32 [I 2022-10-27 18:37:18] ...trap/engines/engine.py.229: process: 0.55169 | load: 0.00246 [I 2022-10-27 18:37:18] ...trap/engines/engine.py.230: loss: 0.11947 [I 2022-10-27 18:37:18] ...optimizers/trijoint.py.151: total_norm: 18.085241 [I 2022-10-27 18:37:23] ...trap/engines/engine.py.224: train: epoch 82 | batch 520/2304 [I 2022-10-27 18:37:23] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:35 [I 2022-10-27 18:37:23] ...trap/engines/engine.py.229: process: 0.54683 | load: 0.00240 [I 2022-10-27 18:37:23] ...trap/engines/engine.py.230: loss: 0.19137 [I 2022-10-27 18:37:23] ...optimizers/trijoint.py.151: total_norm: 23.614653 [I 2022-10-27 18:37:29] ...trap/engines/engine.py.224: train: epoch 82 | batch 530/2304 [I 2022-10-27 18:37:29] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:24 [I 2022-10-27 18:37:29] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00246 [I 2022-10-27 18:37:29] ...trap/engines/engine.py.230: loss: 0.11343 [I 2022-10-27 18:37:29] ...optimizers/trijoint.py.151: total_norm: 11.904170 [I 2022-10-27 18:37:35] ...trap/engines/engine.py.224: train: epoch 82 | batch 540/2304 [I 2022-10-27 18:37:35] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:21 [I 2022-10-27 18:37:35] ...trap/engines/engine.py.229: process: 0.54416 | load: 0.00254 [I 2022-10-27 18:37:35] ...trap/engines/engine.py.230: loss: 0.09489 [I 2022-10-27 18:37:35] ...optimizers/trijoint.py.151: total_norm: 17.120718 [I 2022-10-27 18:37:40] ...trap/engines/engine.py.224: train: epoch 82 | batch 550/2304 [I 2022-10-27 18:37:40] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:24 [I 2022-10-27 18:37:40] ...trap/engines/engine.py.229: process: 0.54796 | load: 0.00298 [I 2022-10-27 18:37:40] ...trap/engines/engine.py.230: loss: 0.11590 [I 2022-10-27 18:37:40] ...optimizers/trijoint.py.151: total_norm: 11.039416 [I 2022-10-27 18:37:46] ...trap/engines/engine.py.224: train: epoch 82 | batch 560/2304 [I 2022-10-27 18:37:46] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:49 [I 2022-10-27 18:37:46] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00248 [I 2022-10-27 18:37:46] ...trap/engines/engine.py.230: loss: 0.10471 [I 2022-10-27 18:37:46] ...optimizers/trijoint.py.151: total_norm: 21.204676 [I 2022-10-27 18:37:51] ...trap/engines/engine.py.224: train: epoch 82 | batch 570/2304 [I 2022-10-27 18:37:51] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:03 [I 2022-10-27 18:37:51] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00245 [I 2022-10-27 18:37:51] ...trap/engines/engine.py.230: loss: 0.14067 [I 2022-10-27 18:37:51] ...optimizers/trijoint.py.151: total_norm: 14.791635 [I 2022-10-27 18:37:57] ...trap/engines/engine.py.224: train: epoch 82 | batch 580/2304 [I 2022-10-27 18:37:57] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:15:57 [I 2022-10-27 18:37:57] ...trap/engines/engine.py.229: process: 0.55266 | load: 0.00250 [I 2022-10-27 18:37:57] ...trap/engines/engine.py.230: loss: 0.11354 [I 2022-10-27 18:37:57] ...optimizers/trijoint.py.151: total_norm: 16.759214 [I 2022-10-27 18:38:03] ...trap/engines/engine.py.224: train: epoch 82 | batch 590/2304 [I 2022-10-27 18:38:03] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:54 [I 2022-10-27 18:38:03] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00282 [I 2022-10-27 18:38:03] ...trap/engines/engine.py.230: loss: 0.19101 [I 2022-10-27 18:38:03] ...optimizers/trijoint.py.151: total_norm: 27.937971 [I 2022-10-27 18:38:08] ...trap/engines/engine.py.224: train: epoch 82 | batch 600/2304 [I 2022-10-27 18:38:08] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:52 [I 2022-10-27 18:38:08] ...trap/engines/engine.py.229: process: 0.56765 | load: 0.00245 [I 2022-10-27 18:38:08] ...trap/engines/engine.py.230: loss: 0.10904 [I 2022-10-27 18:38:08] ...optimizers/trijoint.py.151: total_norm: 8.626385 [I 2022-10-27 18:38:14] ...trap/engines/engine.py.224: train: epoch 82 | batch 610/2304 [I 2022-10-27 18:38:14] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:40 [I 2022-10-27 18:38:14] ...trap/engines/engine.py.229: process: 0.55160 | load: 0.00244 [I 2022-10-27 18:38:14] ...trap/engines/engine.py.230: loss: 0.19357 [I 2022-10-27 18:38:14] ...optimizers/trijoint.py.151: total_norm: 22.478477 [I 2022-10-27 18:38:19] ...trap/engines/engine.py.224: train: epoch 82 | batch 620/2304 [I 2022-10-27 18:38:19] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:44 [I 2022-10-27 18:38:19] ...trap/engines/engine.py.229: process: 0.57162 | load: 0.00262 [I 2022-10-27 18:38:19] ...trap/engines/engine.py.230: loss: 0.13035 [I 2022-10-27 18:38:19] ...optimizers/trijoint.py.151: total_norm: 19.984297 [I 2022-10-27 18:38:25] ...trap/engines/engine.py.224: train: epoch 82 | batch 630/2304 [I 2022-10-27 18:38:25] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:30 [I 2022-10-27 18:38:25] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00253 [I 2022-10-27 18:38:25] ...trap/engines/engine.py.230: loss: 0.13458 [I 2022-10-27 18:38:25] ...optimizers/trijoint.py.151: total_norm: 15.653126 [I 2022-10-27 18:38:31] ...trap/engines/engine.py.224: train: epoch 82 | batch 640/2304 [I 2022-10-27 18:38:31] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:25 [I 2022-10-27 18:38:31] ...trap/engines/engine.py.229: process: 0.54454 | load: 0.00245 [I 2022-10-27 18:38:31] ...trap/engines/engine.py.230: loss: 0.18953 [I 2022-10-27 18:38:31] ...optimizers/trijoint.py.151: total_norm: 27.917997 [I 2022-10-27 18:38:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 650/2304 [I 2022-10-27 18:38:36] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:25 [I 2022-10-27 18:38:36] ...trap/engines/engine.py.229: process: 0.56893 | load: 0.00247 [I 2022-10-27 18:38:36] ...trap/engines/engine.py.230: loss: 0.13437 [I 2022-10-27 18:38:36] ...optimizers/trijoint.py.151: total_norm: 19.132580 [I 2022-10-27 18:38:42] ...trap/engines/engine.py.224: train: epoch 82 | batch 660/2304 [I 2022-10-27 18:38:42] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:08 [I 2022-10-27 18:38:42] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00242 [I 2022-10-27 18:38:42] ...trap/engines/engine.py.230: loss: 0.15408 [I 2022-10-27 18:38:42] ...optimizers/trijoint.py.151: total_norm: 15.344616 [I 2022-10-27 18:38:47] ...trap/engines/engine.py.224: train: epoch 82 | batch 670/2304 [I 2022-10-27 18:38:47] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:08 [I 2022-10-27 18:38:47] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00248 [I 2022-10-27 18:38:47] ...trap/engines/engine.py.230: loss: 0.12790 [I 2022-10-27 18:38:47] ...optimizers/trijoint.py.151: total_norm: 17.495501 [I 2022-10-27 18:38:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 680/2304 [I 2022-10-27 18:38:53] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:13 [I 2022-10-27 18:38:53] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00248 [I 2022-10-27 18:38:53] ...trap/engines/engine.py.230: loss: 0.13599 [I 2022-10-27 18:38:53] ...optimizers/trijoint.py.151: total_norm: 10.459579 [I 2022-10-27 18:38:59] ...trap/engines/engine.py.224: train: epoch 82 | batch 690/2304 [I 2022-10-27 18:38:59] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:05 [I 2022-10-27 18:38:59] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00244 [I 2022-10-27 18:38:59] ...trap/engines/engine.py.230: loss: 0.12594 [I 2022-10-27 18:38:59] ...optimizers/trijoint.py.151: total_norm: 15.335604 [I 2022-10-27 18:39:04] ...trap/engines/engine.py.224: train: epoch 82 | batch 700/2304 [I 2022-10-27 18:39:04] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:15:00 [I 2022-10-27 18:39:04] ...trap/engines/engine.py.229: process: 0.55567 | load: 0.00253 [I 2022-10-27 18:39:04] ...trap/engines/engine.py.230: loss: 0.16661 [I 2022-10-27 18:39:04] ...optimizers/trijoint.py.151: total_norm: 17.420038 [I 2022-10-27 18:39:10] ...trap/engines/engine.py.224: train: epoch 82 | batch 710/2304 [I 2022-10-27 18:39:10] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:57 [I 2022-10-27 18:39:10] ...trap/engines/engine.py.229: process: 0.58232 | load: 0.00244 [I 2022-10-27 18:39:10] ...trap/engines/engine.py.230: loss: 0.13295 [I 2022-10-27 18:39:10] ...optimizers/trijoint.py.151: total_norm: 29.295122 [I 2022-10-27 18:39:15] ...trap/engines/engine.py.224: train: epoch 82 | batch 720/2304 [I 2022-10-27 18:39:15] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:42 [I 2022-10-27 18:39:15] ...trap/engines/engine.py.229: process: 0.55642 | load: 0.00245 [I 2022-10-27 18:39:15] ...trap/engines/engine.py.230: loss: 0.17172 [I 2022-10-27 18:39:15] ...optimizers/trijoint.py.151: total_norm: 18.522142 [I 2022-10-27 18:39:21] ...trap/engines/engine.py.224: train: epoch 82 | batch 730/2304 [I 2022-10-27 18:39:21] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:34 [I 2022-10-27 18:39:21] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00246 [I 2022-10-27 18:39:21] ...trap/engines/engine.py.230: loss: 0.13296 [I 2022-10-27 18:39:21] ...optimizers/trijoint.py.151: total_norm: 29.896730 [I 2022-10-27 18:39:26] ...trap/engines/engine.py.224: train: epoch 82 | batch 740/2304 [I 2022-10-27 18:39:26] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:31 [I 2022-10-27 18:39:26] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00246 [I 2022-10-27 18:39:26] ...trap/engines/engine.py.230: loss: 0.19102 [I 2022-10-27 18:39:26] ...optimizers/trijoint.py.151: total_norm: 12.328499 [I 2022-10-27 18:39:32] ...trap/engines/engine.py.224: train: epoch 82 | batch 750/2304 [I 2022-10-27 18:39:32] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:26 [I 2022-10-27 18:39:32] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00242 [I 2022-10-27 18:39:32] ...trap/engines/engine.py.230: loss: 0.11647 [I 2022-10-27 18:39:32] ...optimizers/trijoint.py.151: total_norm: 11.748555 [I 2022-10-27 18:39:38] ...trap/engines/engine.py.224: train: epoch 82 | batch 760/2304 [I 2022-10-27 18:39:38] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:19 [I 2022-10-27 18:39:38] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00242 [I 2022-10-27 18:39:38] ...trap/engines/engine.py.230: loss: 0.11204 [I 2022-10-27 18:39:38] ...optimizers/trijoint.py.151: total_norm: 15.844025 [I 2022-10-27 18:39:43] ...trap/engines/engine.py.224: train: epoch 82 | batch 770/2304 [I 2022-10-27 18:39:43] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:19 [I 2022-10-27 18:39:43] ...trap/engines/engine.py.229: process: 0.56495 | load: 0.00243 [I 2022-10-27 18:39:43] ...trap/engines/engine.py.230: loss: 0.16895 [I 2022-10-27 18:39:43] ...optimizers/trijoint.py.151: total_norm: 21.536694 [I 2022-10-27 18:39:49] ...trap/engines/engine.py.224: train: epoch 82 | batch 780/2304 [I 2022-10-27 18:39:49] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:01 [I 2022-10-27 18:39:49] ...trap/engines/engine.py.229: process: 0.55234 | load: 0.00245 [I 2022-10-27 18:39:49] ...trap/engines/engine.py.230: loss: 0.11091 [I 2022-10-27 18:39:49] ...optimizers/trijoint.py.151: total_norm: 13.416842 [I 2022-10-27 18:39:54] ...trap/engines/engine.py.224: train: epoch 82 | batch 790/2304 [I 2022-10-27 18:39:54] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:12 [I 2022-10-27 18:39:54] ...trap/engines/engine.py.229: process: 0.54712 | load: 0.00239 [I 2022-10-27 18:39:54] ...trap/engines/engine.py.230: loss: 0.09967 [I 2022-10-27 18:39:54] ...optimizers/trijoint.py.151: total_norm: 6.634287 [I 2022-10-27 18:40:00] ...trap/engines/engine.py.224: train: epoch 82 | batch 800/2304 [I 2022-10-27 18:40:00] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:14:02 [I 2022-10-27 18:40:00] ...trap/engines/engine.py.229: process: 0.55283 | load: 0.00240 [I 2022-10-27 18:40:00] ...trap/engines/engine.py.230: loss: 0.12191 [I 2022-10-27 18:40:00] ...optimizers/trijoint.py.151: total_norm: 25.348444 [I 2022-10-27 18:40:06] ...trap/engines/engine.py.224: train: epoch 82 | batch 810/2304 [I 2022-10-27 18:40:06] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:55 [I 2022-10-27 18:40:06] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00254 [I 2022-10-27 18:40:06] ...trap/engines/engine.py.230: loss: 0.16162 [I 2022-10-27 18:40:06] ...optimizers/trijoint.py.151: total_norm: 19.631039 [I 2022-10-27 18:40:11] ...trap/engines/engine.py.224: train: epoch 82 | batch 820/2304 [I 2022-10-27 18:40:11] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:46 [I 2022-10-27 18:40:11] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00247 [I 2022-10-27 18:40:11] ...trap/engines/engine.py.230: loss: 0.12214 [I 2022-10-27 18:40:11] ...optimizers/trijoint.py.151: total_norm: 19.807724 [I 2022-10-27 18:40:17] ...trap/engines/engine.py.224: train: epoch 82 | batch 830/2304 [I 2022-10-27 18:40:17] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:37 [I 2022-10-27 18:40:17] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00246 [I 2022-10-27 18:40:17] ...trap/engines/engine.py.230: loss: 0.11523 [I 2022-10-27 18:40:17] ...optimizers/trijoint.py.151: total_norm: 25.369455 [I 2022-10-27 18:40:22] ...trap/engines/engine.py.224: train: epoch 82 | batch 840/2304 [I 2022-10-27 18:40:22] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:36 [I 2022-10-27 18:40:22] ...trap/engines/engine.py.229: process: 0.55639 | load: 0.00247 [I 2022-10-27 18:40:22] ...trap/engines/engine.py.230: loss: 0.15544 [I 2022-10-27 18:40:22] ...optimizers/trijoint.py.151: total_norm: 14.910522 [I 2022-10-27 18:40:28] ...trap/engines/engine.py.224: train: epoch 82 | batch 850/2304 [I 2022-10-27 18:40:28] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:25 [I 2022-10-27 18:40:28] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00250 [I 2022-10-27 18:40:28] ...trap/engines/engine.py.230: loss: 0.21290 [I 2022-10-27 18:40:28] ...optimizers/trijoint.py.151: total_norm: 31.717468 [I 2022-10-27 18:40:34] ...trap/engines/engine.py.224: train: epoch 82 | batch 860/2304 [I 2022-10-27 18:40:34] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:21 [I 2022-10-27 18:40:34] ...trap/engines/engine.py.229: process: 0.54603 | load: 0.00244 [I 2022-10-27 18:40:34] ...trap/engines/engine.py.230: loss: 0.14193 [I 2022-10-27 18:40:34] ...optimizers/trijoint.py.151: total_norm: 8.777206 [I 2022-10-27 18:40:39] ...trap/engines/engine.py.224: train: epoch 82 | batch 870/2304 [I 2022-10-27 18:40:39] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:20 [I 2022-10-27 18:40:39] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00239 [I 2022-10-27 18:40:39] ...trap/engines/engine.py.230: loss: 0.16471 [I 2022-10-27 18:40:39] ...optimizers/trijoint.py.151: total_norm: 17.544333 [I 2022-10-27 18:40:45] ...trap/engines/engine.py.224: train: epoch 82 | batch 880/2304 [I 2022-10-27 18:40:45] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:08 [I 2022-10-27 18:40:45] ...trap/engines/engine.py.229: process: 0.53936 | load: 0.00256 [I 2022-10-27 18:40:45] ...trap/engines/engine.py.230: loss: 0.15095 [I 2022-10-27 18:40:45] ...optimizers/trijoint.py.151: total_norm: 30.985556 [I 2022-10-27 18:40:50] ...trap/engines/engine.py.224: train: epoch 82 | batch 890/2304 [I 2022-10-27 18:40:50] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:03 [I 2022-10-27 18:40:50] ...trap/engines/engine.py.229: process: 0.54772 | load: 0.00249 [I 2022-10-27 18:40:50] ...trap/engines/engine.py.230: loss: 0.13528 [I 2022-10-27 18:40:50] ...optimizers/trijoint.py.151: total_norm: 16.405973 [I 2022-10-27 18:40:56] ...trap/engines/engine.py.224: train: epoch 82 | batch 900/2304 [I 2022-10-27 18:40:56] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:12 [I 2022-10-27 18:40:56] ...trap/engines/engine.py.229: process: 0.55434 | load: 0.00247 [I 2022-10-27 18:40:56] ...trap/engines/engine.py.230: loss: 0.12703 [I 2022-10-27 18:40:56] ...optimizers/trijoint.py.151: total_norm: 16.049112 [I 2022-10-27 18:41:02] ...trap/engines/engine.py.224: train: epoch 82 | batch 910/2304 [I 2022-10-27 18:41:02] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:55 [I 2022-10-27 18:41:02] ...trap/engines/engine.py.229: process: 0.56245 | load: 0.00248 [I 2022-10-27 18:41:02] ...trap/engines/engine.py.230: loss: 0.17720 [I 2022-10-27 18:41:02] ...optimizers/trijoint.py.151: total_norm: 14.332365 [I 2022-10-27 18:41:07] ...trap/engines/engine.py.224: train: epoch 82 | batch 920/2304 [I 2022-10-27 18:41:07] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:48 [I 2022-10-27 18:41:07] ...trap/engines/engine.py.229: process: 0.54936 | load: 0.00247 [I 2022-10-27 18:41:07] ...trap/engines/engine.py.230: loss: 0.14210 [I 2022-10-27 18:41:07] ...optimizers/trijoint.py.151: total_norm: 25.707315 [I 2022-10-27 18:41:13] ...trap/engines/engine.py.224: train: epoch 82 | batch 930/2304 [I 2022-10-27 18:41:13] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:45 [I 2022-10-27 18:41:13] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00245 [I 2022-10-27 18:41:13] ...trap/engines/engine.py.230: loss: 0.16120 [I 2022-10-27 18:41:13] ...optimizers/trijoint.py.151: total_norm: 39.609867 [I 2022-10-27 18:41:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 940/2304 [I 2022-10-27 18:41:18] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:35 [I 2022-10-27 18:41:18] ...trap/engines/engine.py.229: process: 0.55214 | load: 0.00246 [I 2022-10-27 18:41:18] ...trap/engines/engine.py.230: loss: 0.07908 [I 2022-10-27 18:41:18] ...optimizers/trijoint.py.151: total_norm: 12.209501 [I 2022-10-27 18:41:24] ...trap/engines/engine.py.224: train: epoch 82 | batch 950/2304 [I 2022-10-27 18:41:24] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:31 [I 2022-10-27 18:41:24] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00240 [I 2022-10-27 18:41:24] ...trap/engines/engine.py.230: loss: 0.08956 [I 2022-10-27 18:41:24] ...optimizers/trijoint.py.151: total_norm: 18.518707 [I 2022-10-27 18:41:29] ...trap/engines/engine.py.224: train: epoch 82 | batch 960/2304 [I 2022-10-27 18:41:29] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:34 [I 2022-10-27 18:41:29] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00242 [I 2022-10-27 18:41:29] ...trap/engines/engine.py.230: loss: 0.10728 [I 2022-10-27 18:41:29] ...optimizers/trijoint.py.151: total_norm: 29.589588 [I 2022-10-27 18:41:35] ...trap/engines/engine.py.224: train: epoch 82 | batch 970/2304 [I 2022-10-27 18:41:35] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:24 [I 2022-10-27 18:41:35] ...trap/engines/engine.py.229: process: 0.55529 | load: 0.00247 [I 2022-10-27 18:41:35] ...trap/engines/engine.py.230: loss: 0.12372 [I 2022-10-27 18:41:35] ...optimizers/trijoint.py.151: total_norm: 13.111227 [I 2022-10-27 18:41:41] ...trap/engines/engine.py.224: train: epoch 82 | batch 980/2304 [I 2022-10-27 18:41:41] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:14 [I 2022-10-27 18:41:41] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00241 [I 2022-10-27 18:41:41] ...trap/engines/engine.py.230: loss: 0.13736 [I 2022-10-27 18:41:41] ...optimizers/trijoint.py.151: total_norm: 24.029472 [I 2022-10-27 18:41:46] ...trap/engines/engine.py.224: train: epoch 82 | batch 990/2304 [I 2022-10-27 18:41:46] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:11 [I 2022-10-27 18:41:46] ...trap/engines/engine.py.229: process: 0.55360 | load: 0.00244 [I 2022-10-27 18:41:46] ...trap/engines/engine.py.230: loss: 0.11136 [I 2022-10-27 18:41:46] ...optimizers/trijoint.py.151: total_norm: 11.102191 [I 2022-10-27 18:41:52] ...trap/engines/engine.py.224: train: epoch 82 | batch 1000/2304 [I 2022-10-27 18:41:52] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:04 [I 2022-10-27 18:41:52] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00261 [I 2022-10-27 18:41:52] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-27 18:41:52] ...optimizers/trijoint.py.151: total_norm: 17.610193 [I 2022-10-27 18:41:57] ...trap/engines/engine.py.224: train: epoch 82 | batch 1010/2304 [I 2022-10-27 18:41:57] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:03 [I 2022-10-27 18:41:57] ...trap/engines/engine.py.229: process: 0.56829 | load: 0.00252 [I 2022-10-27 18:41:57] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-27 18:41:57] ...optimizers/trijoint.py.151: total_norm: 14.774871 [I 2022-10-27 18:42:03] ...trap/engines/engine.py.224: train: epoch 82 | batch 1020/2304 [I 2022-10-27 18:42:03] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:54 [I 2022-10-27 18:42:03] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00248 [I 2022-10-27 18:42:03] ...trap/engines/engine.py.230: loss: 0.10530 [I 2022-10-27 18:42:03] ...optimizers/trijoint.py.151: total_norm: 7.490114 [I 2022-10-27 18:42:09] ...trap/engines/engine.py.224: train: epoch 82 | batch 1030/2304 [I 2022-10-27 18:42:09] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:45 [I 2022-10-27 18:42:09] ...trap/engines/engine.py.229: process: 0.55098 | load: 0.00243 [I 2022-10-27 18:42:09] ...trap/engines/engine.py.230: loss: 0.10895 [I 2022-10-27 18:42:09] ...optimizers/trijoint.py.151: total_norm: 14.784324 [I 2022-10-27 18:42:14] ...trap/engines/engine.py.224: train: epoch 82 | batch 1040/2304 [I 2022-10-27 18:42:14] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:47 [I 2022-10-27 18:42:14] ...trap/engines/engine.py.229: process: 0.56568 | load: 0.00239 [I 2022-10-27 18:42:14] ...trap/engines/engine.py.230: loss: 0.11199 [I 2022-10-27 18:42:14] ...optimizers/trijoint.py.151: total_norm: 13.860323 [I 2022-10-27 18:42:20] ...trap/engines/engine.py.224: train: epoch 82 | batch 1050/2304 [I 2022-10-27 18:42:20] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:40 [I 2022-10-27 18:42:20] ...trap/engines/engine.py.229: process: 0.55103 | load: 0.00245 [I 2022-10-27 18:42:20] ...trap/engines/engine.py.230: loss: 0.09759 [I 2022-10-27 18:42:20] ...optimizers/trijoint.py.151: total_norm: 11.365618 [I 2022-10-27 18:42:25] ...trap/engines/engine.py.224: train: epoch 82 | batch 1060/2304 [I 2022-10-27 18:42:25] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:29 [I 2022-10-27 18:42:25] ...trap/engines/engine.py.229: process: 0.54167 | load: 0.00246 [I 2022-10-27 18:42:25] ...trap/engines/engine.py.230: loss: 0.14139 [I 2022-10-27 18:42:25] ...optimizers/trijoint.py.151: total_norm: 24.892769 [I 2022-10-27 18:42:31] ...trap/engines/engine.py.224: train: epoch 82 | batch 1070/2304 [I 2022-10-27 18:42:31] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:23 [I 2022-10-27 18:42:31] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00241 [I 2022-10-27 18:42:31] ...trap/engines/engine.py.230: loss: 0.09642 [I 2022-10-27 18:42:31] ...optimizers/trijoint.py.151: total_norm: 12.465208 [I 2022-10-27 18:42:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 1080/2304 [I 2022-10-27 18:42:36] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:23 [I 2022-10-27 18:42:36] ...trap/engines/engine.py.229: process: 0.56479 | load: 0.00242 [I 2022-10-27 18:42:36] ...trap/engines/engine.py.230: loss: 0.14885 [I 2022-10-27 18:42:36] ...optimizers/trijoint.py.151: total_norm: 13.463551 [I 2022-10-27 18:42:42] ...trap/engines/engine.py.224: train: epoch 82 | batch 1090/2304 [I 2022-10-27 18:42:42] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:13 [I 2022-10-27 18:42:42] ...trap/engines/engine.py.229: process: 0.55102 | load: 0.00247 [I 2022-10-27 18:42:42] ...trap/engines/engine.py.230: loss: 0.14102 [I 2022-10-27 18:42:42] ...optimizers/trijoint.py.151: total_norm: 16.119476 [I 2022-10-27 18:42:48] ...trap/engines/engine.py.224: train: epoch 82 | batch 1100/2304 [I 2022-10-27 18:42:48] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:16 [I 2022-10-27 18:42:48] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00260 [I 2022-10-27 18:42:48] ...trap/engines/engine.py.230: loss: 0.10663 [I 2022-10-27 18:42:48] ...optimizers/trijoint.py.151: total_norm: 10.525364 [I 2022-10-27 18:42:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 1110/2304 [I 2022-10-27 18:42:53] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:03 [I 2022-10-27 18:42:53] ...trap/engines/engine.py.229: process: 0.54925 | load: 0.00262 [I 2022-10-27 18:42:53] ...trap/engines/engine.py.230: loss: 0.15327 [I 2022-10-27 18:42:53] ...optimizers/trijoint.py.151: total_norm: 13.167722 [I 2022-10-27 18:42:59] ...trap/engines/engine.py.224: train: epoch 82 | batch 1120/2304 [I 2022-10-27 18:42:59] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:00 [I 2022-10-27 18:42:59] ...trap/engines/engine.py.229: process: 0.56290 | load: 0.00249 [I 2022-10-27 18:42:59] ...trap/engines/engine.py.230: loss: 0.11446 [I 2022-10-27 18:42:59] ...optimizers/trijoint.py.151: total_norm: 11.896455 [I 2022-10-27 18:43:04] ...trap/engines/engine.py.224: train: epoch 82 | batch 1130/2304 [I 2022-10-27 18:43:04] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:55 [I 2022-10-27 18:43:04] ...trap/engines/engine.py.229: process: 0.55830 | load: 0.00249 [I 2022-10-27 18:43:04] ...trap/engines/engine.py.230: loss: 0.09853 [I 2022-10-27 18:43:04] ...optimizers/trijoint.py.151: total_norm: 10.444023 [I 2022-10-27 18:43:10] ...trap/engines/engine.py.224: train: epoch 82 | batch 1140/2304 [I 2022-10-27 18:43:10] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:52 [I 2022-10-27 18:43:10] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00250 [I 2022-10-27 18:43:10] ...trap/engines/engine.py.230: loss: 0.09622 [I 2022-10-27 18:43:10] ...optimizers/trijoint.py.151: total_norm: 9.819644 [I 2022-10-27 18:43:16] ...trap/engines/engine.py.224: train: epoch 82 | batch 1150/2304 [I 2022-10-27 18:43:16] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:46 [I 2022-10-27 18:43:16] ...trap/engines/engine.py.229: process: 0.57725 | load: 0.00244 [I 2022-10-27 18:43:16] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-27 18:43:16] ...optimizers/trijoint.py.151: total_norm: 11.069978 [I 2022-10-27 18:43:21] ...trap/engines/engine.py.224: train: epoch 82 | batch 1160/2304 [I 2022-10-27 18:43:21] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:42 [I 2022-10-27 18:43:21] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00247 [I 2022-10-27 18:43:21] ...trap/engines/engine.py.230: loss: 0.11995 [I 2022-10-27 18:43:21] ...optimizers/trijoint.py.151: total_norm: 11.495552 [I 2022-10-27 18:43:27] ...trap/engines/engine.py.224: train: epoch 82 | batch 1170/2304 [I 2022-10-27 18:43:27] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:32 [I 2022-10-27 18:43:27] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00251 [I 2022-10-27 18:43:27] ...trap/engines/engine.py.230: loss: 0.13396 [I 2022-10-27 18:43:27] ...optimizers/trijoint.py.151: total_norm: 18.632442 [I 2022-10-27 18:43:32] ...trap/engines/engine.py.224: train: epoch 82 | batch 1180/2304 [I 2022-10-27 18:43:32] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:29 [I 2022-10-27 18:43:32] ...trap/engines/engine.py.229: process: 0.56374 | load: 0.00246 [I 2022-10-27 18:43:32] ...trap/engines/engine.py.230: loss: 0.09124 [I 2022-10-27 18:43:32] ...optimizers/trijoint.py.151: total_norm: 10.396400 [I 2022-10-27 18:43:38] ...trap/engines/engine.py.224: train: epoch 82 | batch 1190/2304 [I 2022-10-27 18:43:38] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:24 [I 2022-10-27 18:43:38] ...trap/engines/engine.py.229: process: 0.55950 | load: 0.00243 [I 2022-10-27 18:43:38] ...trap/engines/engine.py.230: loss: 0.10592 [I 2022-10-27 18:43:38] ...optimizers/trijoint.py.151: total_norm: 29.315275 [I 2022-10-27 18:43:44] ...trap/engines/engine.py.224: train: epoch 82 | batch 1200/2304 [I 2022-10-27 18:43:44] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:20 [I 2022-10-27 18:43:44] ...trap/engines/engine.py.229: process: 0.55943 | load: 0.00243 [I 2022-10-27 18:43:44] ...trap/engines/engine.py.230: loss: 0.18517 [I 2022-10-27 18:43:44] ...optimizers/trijoint.py.151: total_norm: 29.548899 [I 2022-10-27 18:43:49] ...trap/engines/engine.py.224: train: epoch 82 | batch 1210/2304 [I 2022-10-27 18:43:49] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:13 [I 2022-10-27 18:43:49] ...trap/engines/engine.py.229: process: 0.56348 | load: 0.00248 [I 2022-10-27 18:43:49] ...trap/engines/engine.py.230: loss: 0.10551 [I 2022-10-27 18:43:49] ...optimizers/trijoint.py.151: total_norm: 15.121561 [I 2022-10-27 18:43:55] ...trap/engines/engine.py.224: train: epoch 82 | batch 1220/2304 [I 2022-10-27 18:43:55] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:02 [I 2022-10-27 18:43:55] ...trap/engines/engine.py.229: process: 0.55583 | load: 0.00249 [I 2022-10-27 18:43:55] ...trap/engines/engine.py.230: loss: 0.09618 [I 2022-10-27 18:43:55] ...optimizers/trijoint.py.151: total_norm: 9.906339 [I 2022-10-27 18:44:00] ...trap/engines/engine.py.224: train: epoch 82 | batch 1230/2304 [I 2022-10-27 18:44:00] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:58 [I 2022-10-27 18:44:00] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00245 [I 2022-10-27 18:44:00] ...trap/engines/engine.py.230: loss: 0.11822 [I 2022-10-27 18:44:00] ...optimizers/trijoint.py.151: total_norm: 39.372742 [I 2022-10-27 18:44:06] ...trap/engines/engine.py.224: train: epoch 82 | batch 1240/2304 [I 2022-10-27 18:44:06] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:54 [I 2022-10-27 18:44:06] ...trap/engines/engine.py.229: process: 0.55423 | load: 0.00251 [I 2022-10-27 18:44:06] ...trap/engines/engine.py.230: loss: 0.10943 [I 2022-10-27 18:44:06] ...optimizers/trijoint.py.151: total_norm: 10.611200 [I 2022-10-27 18:44:12] ...trap/engines/engine.py.224: train: epoch 82 | batch 1250/2304 [I 2022-10-27 18:44:12] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:48 [I 2022-10-27 18:44:12] ...trap/engines/engine.py.229: process: 0.56843 | load: 0.00249 [I 2022-10-27 18:44:12] ...trap/engines/engine.py.230: loss: 0.12566 [I 2022-10-27 18:44:12] ...optimizers/trijoint.py.151: total_norm: 19.635860 [I 2022-10-27 18:44:17] ...trap/engines/engine.py.224: train: epoch 82 | batch 1260/2304 [I 2022-10-27 18:44:17] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:43 [I 2022-10-27 18:44:17] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00247 [I 2022-10-27 18:44:17] ...trap/engines/engine.py.230: loss: 0.17034 [I 2022-10-27 18:44:17] ...optimizers/trijoint.py.151: total_norm: 24.698139 [I 2022-10-27 18:44:23] ...trap/engines/engine.py.224: train: epoch 82 | batch 1270/2304 [I 2022-10-27 18:44:23] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:35 [I 2022-10-27 18:44:23] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00247 [I 2022-10-27 18:44:23] ...trap/engines/engine.py.230: loss: 0.10468 [I 2022-10-27 18:44:23] ...optimizers/trijoint.py.151: total_norm: 11.777941 [I 2022-10-27 18:44:28] ...trap/engines/engine.py.224: train: epoch 82 | batch 1280/2304 [I 2022-10-27 18:44:28] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:28 [I 2022-10-27 18:44:28] ...trap/engines/engine.py.229: process: 0.54717 | load: 0.00253 [I 2022-10-27 18:44:28] ...trap/engines/engine.py.230: loss: 0.12040 [I 2022-10-27 18:44:28] ...optimizers/trijoint.py.151: total_norm: 12.235746 [I 2022-10-27 18:44:34] ...trap/engines/engine.py.224: train: epoch 82 | batch 1290/2304 [I 2022-10-27 18:44:34] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:26 [I 2022-10-27 18:44:34] ...trap/engines/engine.py.229: process: 0.54811 | load: 0.00269 [I 2022-10-27 18:44:34] ...trap/engines/engine.py.230: loss: 0.10077 [I 2022-10-27 18:44:34] ...optimizers/trijoint.py.151: total_norm: 26.998421 [I 2022-10-27 18:44:39] ...trap/engines/engine.py.224: train: epoch 82 | batch 1300/2304 [I 2022-10-27 18:44:39] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:18 [I 2022-10-27 18:44:39] ...trap/engines/engine.py.229: process: 0.56678 | load: 0.00242 [I 2022-10-27 18:44:39] ...trap/engines/engine.py.230: loss: 0.18065 [I 2022-10-27 18:44:39] ...optimizers/trijoint.py.151: total_norm: 23.445189 [I 2022-10-27 18:44:45] ...trap/engines/engine.py.224: train: epoch 82 | batch 1310/2304 [I 2022-10-27 18:44:45] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:16 [I 2022-10-27 18:44:45] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00248 [I 2022-10-27 18:44:45] ...trap/engines/engine.py.230: loss: 0.13463 [I 2022-10-27 18:44:45] ...optimizers/trijoint.py.151: total_norm: 21.574894 [I 2022-10-27 18:44:51] ...trap/engines/engine.py.224: train: epoch 82 | batch 1320/2304 [I 2022-10-27 18:44:51] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:12 [I 2022-10-27 18:44:51] ...trap/engines/engine.py.229: process: 0.56820 | load: 0.00243 [I 2022-10-27 18:44:51] ...trap/engines/engine.py.230: loss: 0.16651 [I 2022-10-27 18:44:51] ...optimizers/trijoint.py.151: total_norm: 8.330606 [I 2022-10-27 18:44:56] ...trap/engines/engine.py.224: train: epoch 82 | batch 1330/2304 [I 2022-10-27 18:44:56] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:03 [I 2022-10-27 18:44:56] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00243 [I 2022-10-27 18:44:56] ...trap/engines/engine.py.230: loss: 0.13802 [I 2022-10-27 18:44:56] ...optimizers/trijoint.py.151: total_norm: 10.234467 [I 2022-10-27 18:45:02] ...trap/engines/engine.py.224: train: epoch 82 | batch 1340/2304 [I 2022-10-27 18:45:02] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:56 [I 2022-10-27 18:45:02] ...trap/engines/engine.py.229: process: 0.55526 | load: 0.00244 [I 2022-10-27 18:45:02] ...trap/engines/engine.py.230: loss: 0.12881 [I 2022-10-27 18:45:02] ...optimizers/trijoint.py.151: total_norm: 14.350577 [I 2022-10-27 18:45:07] ...trap/engines/engine.py.224: train: epoch 82 | batch 1350/2304 [I 2022-10-27 18:45:07] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:52 [I 2022-10-27 18:45:07] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00242 [I 2022-10-27 18:45:07] ...trap/engines/engine.py.230: loss: 0.13209 [I 2022-10-27 18:45:07] ...optimizers/trijoint.py.151: total_norm: 21.050329 [I 2022-10-27 18:45:13] ...trap/engines/engine.py.224: train: epoch 82 | batch 1360/2304 [I 2022-10-27 18:45:13] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:45 [I 2022-10-27 18:45:13] ...trap/engines/engine.py.229: process: 0.54963 | load: 0.00264 [I 2022-10-27 18:45:13] ...trap/engines/engine.py.230: loss: 0.11981 [I 2022-10-27 18:45:13] ...optimizers/trijoint.py.151: total_norm: 21.642096 [I 2022-10-27 18:45:19] ...trap/engines/engine.py.224: train: epoch 82 | batch 1370/2304 [I 2022-10-27 18:45:19] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:39 [I 2022-10-27 18:45:19] ...trap/engines/engine.py.229: process: 0.55452 | load: 0.00242 [I 2022-10-27 18:45:19] ...trap/engines/engine.py.230: loss: 0.14824 [I 2022-10-27 18:45:19] ...optimizers/trijoint.py.151: total_norm: 14.007837 [I 2022-10-27 18:45:25] ...trap/engines/engine.py.224: train: epoch 82 | batch 1380/2304 [I 2022-10-27 18:45:25] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:10:22 [I 2022-10-27 18:45:25] ...trap/engines/engine.py.229: process: 0.75231 | load: 0.00267 [I 2022-10-27 18:45:25] ...trap/engines/engine.py.230: loss: 0.15677 [I 2022-10-27 18:45:25] ...optimizers/trijoint.py.151: total_norm: 23.171188 [I 2022-10-27 18:45:31] ...trap/engines/engine.py.224: train: epoch 82 | batch 1390/2304 [I 2022-10-27 18:45:31] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:48 [I 2022-10-27 18:45:31] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00241 [I 2022-10-27 18:45:31] ...trap/engines/engine.py.230: loss: 0.13300 [I 2022-10-27 18:45:31] ...optimizers/trijoint.py.151: total_norm: 16.307438 [I 2022-10-27 18:45:37] ...trap/engines/engine.py.224: train: epoch 82 | batch 1400/2304 [I 2022-10-27 18:45:37] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:28 [I 2022-10-27 18:45:37] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00253 [I 2022-10-27 18:45:37] ...trap/engines/engine.py.230: loss: 0.19176 [I 2022-10-27 18:45:37] ...optimizers/trijoint.py.151: total_norm: 12.012701 [I 2022-10-27 18:45:42] ...trap/engines/engine.py.224: train: epoch 82 | batch 1410/2304 [I 2022-10-27 18:45:42] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:22 [I 2022-10-27 18:45:42] ...trap/engines/engine.py.229: process: 0.56188 | load: 0.00258 [I 2022-10-27 18:45:42] ...trap/engines/engine.py.230: loss: 0.16208 [I 2022-10-27 18:45:42] ...optimizers/trijoint.py.151: total_norm: 14.634433 [I 2022-10-27 18:45:48] ...trap/engines/engine.py.224: train: epoch 82 | batch 1420/2304 [I 2022-10-27 18:45:48] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:15 [I 2022-10-27 18:45:48] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00244 [I 2022-10-27 18:45:48] ...trap/engines/engine.py.230: loss: 0.11044 [I 2022-10-27 18:45:48] ...optimizers/trijoint.py.151: total_norm: 18.759655 [I 2022-10-27 18:45:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 1430/2304 [I 2022-10-27 18:45:53] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:03 [I 2022-10-27 18:45:53] ...trap/engines/engine.py.229: process: 0.54536 | load: 0.00248 [I 2022-10-27 18:45:53] ...trap/engines/engine.py.230: loss: 0.11050 [I 2022-10-27 18:45:53] ...optimizers/trijoint.py.151: total_norm: 15.487661 [I 2022-10-27 18:46:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 1440/2304 [I 2022-10-27 18:46:18] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:30:40 [I 2022-10-27 18:46:18] ...trap/engines/engine.py.229: process: 0.54544 | load: 0.00239 [I 2022-10-27 18:46:18] ...trap/engines/engine.py.230: loss: 0.10937 [I 2022-10-27 18:46:18] ...optimizers/trijoint.py.151: total_norm: 24.716948 [I 2022-10-27 18:46:24] ...trap/engines/engine.py.224: train: epoch 82 | batch 1450/2304 [I 2022-10-27 18:46:24] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:10:19 [I 2022-10-27 18:46:24] ...trap/engines/engine.py.229: process: 0.56358 | load: 0.00235 [I 2022-10-27 18:46:24] ...trap/engines/engine.py.230: loss: 0.15237 [I 2022-10-27 18:46:24] ...optimizers/trijoint.py.151: total_norm: 16.933023 [I 2022-10-27 18:46:30] ...trap/engines/engine.py.224: train: epoch 82 | batch 1460/2304 [I 2022-10-27 18:46:30] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:08:11 [I 2022-10-27 18:46:30] ...trap/engines/engine.py.229: process: 0.57283 | load: 0.00237 [I 2022-10-27 18:46:30] ...trap/engines/engine.py.230: loss: 0.14046 [I 2022-10-27 18:46:30] ...optimizers/trijoint.py.151: total_norm: 17.574251 [I 2022-10-27 18:46:35] ...trap/engines/engine.py.224: train: epoch 82 | batch 1470/2304 [I 2022-10-27 18:46:35] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:51 [I 2022-10-27 18:46:35] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00237 [I 2022-10-27 18:46:35] ...trap/engines/engine.py.230: loss: 0.19720 [I 2022-10-27 18:46:35] ...optimizers/trijoint.py.151: total_norm: 25.381977 [I 2022-10-27 18:46:41] ...trap/engines/engine.py.224: train: epoch 82 | batch 1480/2304 [I 2022-10-27 18:46:41] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:43 [I 2022-10-27 18:46:41] ...trap/engines/engine.py.229: process: 0.56641 | load: 0.00239 [I 2022-10-27 18:46:41] ...trap/engines/engine.py.230: loss: 0.14818 [I 2022-10-27 18:46:41] ...optimizers/trijoint.py.151: total_norm: 9.168504 [I 2022-10-27 18:46:46] ...trap/engines/engine.py.224: train: epoch 82 | batch 1490/2304 [I 2022-10-27 18:46:46] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:37 [I 2022-10-27 18:46:46] ...trap/engines/engine.py.229: process: 0.56781 | load: 0.00239 [I 2022-10-27 18:46:46] ...trap/engines/engine.py.230: loss: 0.16774 [I 2022-10-27 18:46:46] ...optimizers/trijoint.py.151: total_norm: 27.633900 [I 2022-10-27 18:46:52] ...trap/engines/engine.py.224: train: epoch 82 | batch 1500/2304 [I 2022-10-27 18:46:52] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:27 [I 2022-10-27 18:46:52] ...trap/engines/engine.py.229: process: 0.55157 | load: 0.00241 [I 2022-10-27 18:46:52] ...trap/engines/engine.py.230: loss: 0.14585 [I 2022-10-27 18:46:52] ...optimizers/trijoint.py.151: total_norm: 38.985287 [I 2022-10-27 18:46:58] ...trap/engines/engine.py.224: train: epoch 82 | batch 1510/2304 [I 2022-10-27 18:46:58] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:22 [I 2022-10-27 18:46:58] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00240 [I 2022-10-27 18:46:58] ...trap/engines/engine.py.230: loss: 0.17053 [I 2022-10-27 18:46:58] ...optimizers/trijoint.py.151: total_norm: 19.701509 [I 2022-10-27 18:47:03] ...trap/engines/engine.py.224: train: epoch 82 | batch 1520/2304 [I 2022-10-27 18:47:03] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:17 [I 2022-10-27 18:47:03] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00238 [I 2022-10-27 18:47:03] ...trap/engines/engine.py.230: loss: 0.11717 [I 2022-10-27 18:47:03] ...optimizers/trijoint.py.151: total_norm: 10.830737 [I 2022-10-27 18:47:09] ...trap/engines/engine.py.224: train: epoch 82 | batch 1530/2304 [I 2022-10-27 18:47:09] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:07:12 [I 2022-10-27 18:47:09] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00241 [I 2022-10-27 18:47:09] ...trap/engines/engine.py.230: loss: 0.08466 [I 2022-10-27 18:47:09] ...optimizers/trijoint.py.151: total_norm: 12.399059 [I 2022-10-27 18:47:14] ...trap/engines/engine.py.224: train: epoch 82 | batch 1540/2304 [I 2022-10-27 18:47:14] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:07:08 [I 2022-10-27 18:47:14] ...trap/engines/engine.py.229: process: 0.56184 | load: 0.00238 [I 2022-10-27 18:47:14] ...trap/engines/engine.py.230: loss: 0.09456 [I 2022-10-27 18:47:14] ...optimizers/trijoint.py.151: total_norm: 8.974796 [I 2022-10-27 18:47:20] ...trap/engines/engine.py.224: train: epoch 82 | batch 1550/2304 [I 2022-10-27 18:47:20] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:07:03 [I 2022-10-27 18:47:20] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00264 [I 2022-10-27 18:47:20] ...trap/engines/engine.py.230: loss: 0.14862 [I 2022-10-27 18:47:20] ...optimizers/trijoint.py.151: total_norm: 33.755844 [I 2022-10-27 18:47:26] ...trap/engines/engine.py.224: train: epoch 82 | batch 1560/2304 [I 2022-10-27 18:47:26] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:57 [I 2022-10-27 18:47:26] ...trap/engines/engine.py.229: process: 0.57255 | load: 0.00239 [I 2022-10-27 18:47:26] ...trap/engines/engine.py.230: loss: 0.09149 [I 2022-10-27 18:47:26] ...optimizers/trijoint.py.151: total_norm: 10.131645 [I 2022-10-27 18:47:31] ...trap/engines/engine.py.224: train: epoch 82 | batch 1570/2304 [I 2022-10-27 18:47:31] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:48 [I 2022-10-27 18:47:31] ...trap/engines/engine.py.229: process: 0.54389 | load: 0.00242 [I 2022-10-27 18:47:31] ...trap/engines/engine.py.230: loss: 0.10903 [I 2022-10-27 18:47:31] ...optimizers/trijoint.py.151: total_norm: 9.907615 [I 2022-10-27 18:47:37] ...trap/engines/engine.py.224: train: epoch 82 | batch 1580/2304 [I 2022-10-27 18:47:37] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:47 [I 2022-10-27 18:47:37] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00241 [I 2022-10-27 18:47:37] ...trap/engines/engine.py.230: loss: 0.09464 [I 2022-10-27 18:47:37] ...optimizers/trijoint.py.151: total_norm: 29.944553 [I 2022-10-27 18:47:43] ...trap/engines/engine.py.224: train: epoch 82 | batch 1590/2304 [I 2022-10-27 18:47:43] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:36 [I 2022-10-27 18:47:43] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00238 [I 2022-10-27 18:47:43] ...trap/engines/engine.py.230: loss: 0.19464 [I 2022-10-27 18:47:43] ...optimizers/trijoint.py.151: total_norm: 16.229870 [I 2022-10-27 18:47:48] ...trap/engines/engine.py.224: train: epoch 82 | batch 1600/2304 [I 2022-10-27 18:47:48] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:31 [I 2022-10-27 18:47:48] ...trap/engines/engine.py.229: process: 0.55224 | load: 0.00241 [I 2022-10-27 18:47:48] ...trap/engines/engine.py.230: loss: 0.14516 [I 2022-10-27 18:47:48] ...optimizers/trijoint.py.151: total_norm: 13.522213 [I 2022-10-27 18:47:54] ...trap/engines/engine.py.224: train: epoch 82 | batch 1610/2304 [I 2022-10-27 18:47:54] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:31 [I 2022-10-27 18:47:54] ...trap/engines/engine.py.229: process: 0.57009 | load: 0.00244 [I 2022-10-27 18:47:54] ...trap/engines/engine.py.230: loss: 0.10978 [I 2022-10-27 18:47:54] ...optimizers/trijoint.py.151: total_norm: 13.907860 [I 2022-10-27 18:47:59] ...trap/engines/engine.py.224: train: epoch 82 | batch 1620/2304 [I 2022-10-27 18:47:59] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:21 [I 2022-10-27 18:47:59] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00238 [I 2022-10-27 18:47:59] ...trap/engines/engine.py.230: loss: 0.11002 [I 2022-10-27 18:47:59] ...optimizers/trijoint.py.151: total_norm: 13.573849 [I 2022-10-27 18:48:05] ...trap/engines/engine.py.224: train: epoch 82 | batch 1630/2304 [I 2022-10-27 18:48:05] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:06:17 [I 2022-10-27 18:48:05] ...trap/engines/engine.py.229: process: 0.56945 | load: 0.00236 [I 2022-10-27 18:48:05] ...trap/engines/engine.py.230: loss: 0.16974 [I 2022-10-27 18:48:05] ...optimizers/trijoint.py.151: total_norm: 15.724916 [I 2022-10-27 18:48:11] ...trap/engines/engine.py.224: train: epoch 82 | batch 1640/2304 [I 2022-10-27 18:48:11] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:06:08 [I 2022-10-27 18:48:11] ...trap/engines/engine.py.229: process: 0.54996 | load: 0.00236 [I 2022-10-27 18:48:11] ...trap/engines/engine.py.230: loss: 0.13159 [I 2022-10-27 18:48:11] ...optimizers/trijoint.py.151: total_norm: 21.155729 [I 2022-10-27 18:48:16] ...trap/engines/engine.py.224: train: epoch 82 | batch 1650/2304 [I 2022-10-27 18:48:16] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:06:04 [I 2022-10-27 18:48:16] ...trap/engines/engine.py.229: process: 0.55656 | load: 0.00235 [I 2022-10-27 18:48:16] ...trap/engines/engine.py.230: loss: 0.20278 [I 2022-10-27 18:48:16] ...optimizers/trijoint.py.151: total_norm: 16.778046 [I 2022-10-27 18:48:22] ...trap/engines/engine.py.224: train: epoch 82 | batch 1660/2304 [I 2022-10-27 18:48:22] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:59 [I 2022-10-27 18:48:22] ...trap/engines/engine.py.229: process: 0.56173 | load: 0.00240 [I 2022-10-27 18:48:22] ...trap/engines/engine.py.230: loss: 0.15172 [I 2022-10-27 18:48:22] ...optimizers/trijoint.py.151: total_norm: 23.070072 [I 2022-10-27 18:48:27] ...trap/engines/engine.py.224: train: epoch 82 | batch 1670/2304 [I 2022-10-27 18:48:27] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:59 [I 2022-10-27 18:48:27] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00249 [I 2022-10-27 18:48:27] ...trap/engines/engine.py.230: loss: 0.12341 [I 2022-10-27 18:48:27] ...optimizers/trijoint.py.151: total_norm: 12.951587 [I 2022-10-27 18:48:33] ...trap/engines/engine.py.224: train: epoch 82 | batch 1680/2304 [I 2022-10-27 18:48:33] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:45 [I 2022-10-27 18:48:33] ...trap/engines/engine.py.229: process: 0.55693 | load: 0.00245 [I 2022-10-27 18:48:33] ...trap/engines/engine.py.230: loss: 0.11793 [I 2022-10-27 18:48:33] ...optimizers/trijoint.py.151: total_norm: 21.991009 [I 2022-10-27 18:48:38] ...trap/engines/engine.py.224: train: epoch 82 | batch 1690/2304 [I 2022-10-27 18:48:38] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:41 [I 2022-10-27 18:48:38] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00240 [I 2022-10-27 18:48:38] ...trap/engines/engine.py.230: loss: 0.13306 [I 2022-10-27 18:48:38] ...optimizers/trijoint.py.151: total_norm: 9.056281 [I 2022-10-27 18:48:44] ...trap/engines/engine.py.224: train: epoch 82 | batch 1700/2304 [I 2022-10-27 18:48:44] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:35 [I 2022-10-27 18:48:44] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00244 [I 2022-10-27 18:48:44] ...trap/engines/engine.py.230: loss: 0.11992 [I 2022-10-27 18:48:44] ...optimizers/trijoint.py.151: total_norm: 13.532819 [I 2022-10-27 18:48:50] ...trap/engines/engine.py.224: train: epoch 82 | batch 1710/2304 [I 2022-10-27 18:48:50] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:31 [I 2022-10-27 18:48:50] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00241 [I 2022-10-27 18:48:50] ...trap/engines/engine.py.230: loss: 0.12032 [I 2022-10-27 18:48:50] ...optimizers/trijoint.py.151: total_norm: 8.490247 [I 2022-10-27 18:48:55] ...trap/engines/engine.py.224: train: epoch 82 | batch 1720/2304 [I 2022-10-27 18:48:55] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:24 [I 2022-10-27 18:48:55] ...trap/engines/engine.py.229: process: 0.56894 | load: 0.00241 [I 2022-10-27 18:48:55] ...trap/engines/engine.py.230: loss: 0.28673 [I 2022-10-27 18:48:55] ...optimizers/trijoint.py.151: total_norm: 26.359112 [I 2022-10-27 18:49:01] ...trap/engines/engine.py.224: train: epoch 82 | batch 1730/2304 [I 2022-10-27 18:49:01] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:19 [I 2022-10-27 18:49:01] ...trap/engines/engine.py.229: process: 0.54763 | load: 0.00244 [I 2022-10-27 18:49:01] ...trap/engines/engine.py.230: loss: 0.12312 [I 2022-10-27 18:49:01] ...optimizers/trijoint.py.151: total_norm: 22.550505 [I 2022-10-27 18:49:06] ...trap/engines/engine.py.224: train: epoch 82 | batch 1740/2304 [I 2022-10-27 18:49:06] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:05:15 [I 2022-10-27 18:49:06] ...trap/engines/engine.py.229: process: 0.55307 | load: 0.00244 [I 2022-10-27 18:49:06] ...trap/engines/engine.py.230: loss: 0.11100 [I 2022-10-27 18:49:06] ...optimizers/trijoint.py.151: total_norm: 21.327562 [I 2022-10-27 18:49:12] ...trap/engines/engine.py.224: train: epoch 82 | batch 1750/2304 [I 2022-10-27 18:49:12] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:08 [I 2022-10-27 18:49:12] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00244 [I 2022-10-27 18:49:12] ...trap/engines/engine.py.230: loss: 0.14880 [I 2022-10-27 18:49:12] ...optimizers/trijoint.py.151: total_norm: 15.598646 [I 2022-10-27 18:49:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 1760/2304 [I 2022-10-27 18:49:18] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:05:01 [I 2022-10-27 18:49:18] ...trap/engines/engine.py.229: process: 0.55145 | load: 0.00252 [I 2022-10-27 18:49:18] ...trap/engines/engine.py.230: loss: 0.17117 [I 2022-10-27 18:49:18] ...optimizers/trijoint.py.151: total_norm: 8.512607 [I 2022-10-27 18:49:23] ...trap/engines/engine.py.224: train: epoch 82 | batch 1770/2304 [I 2022-10-27 18:49:23] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:05:05 [I 2022-10-27 18:49:23] ...trap/engines/engine.py.229: process: 0.57495 | load: 0.00245 [I 2022-10-27 18:49:23] ...trap/engines/engine.py.230: loss: 0.12285 [I 2022-10-27 18:49:23] ...optimizers/trijoint.py.151: total_norm: 20.373850 [I 2022-10-27 18:49:29] ...trap/engines/engine.py.224: train: epoch 82 | batch 1780/2304 [I 2022-10-27 18:49:29] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:57 [I 2022-10-27 18:49:29] ...trap/engines/engine.py.229: process: 0.56769 | load: 0.00263 [I 2022-10-27 18:49:29] ...trap/engines/engine.py.230: loss: 0.13627 [I 2022-10-27 18:49:29] ...optimizers/trijoint.py.151: total_norm: 12.731741 [I 2022-10-27 18:49:35] ...trap/engines/engine.py.224: train: epoch 82 | batch 1790/2304 [I 2022-10-27 18:49:35] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:52 [I 2022-10-27 18:49:35] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00244 [I 2022-10-27 18:49:35] ...trap/engines/engine.py.230: loss: 0.14768 [I 2022-10-27 18:49:35] ...optimizers/trijoint.py.151: total_norm: 28.591305 [I 2022-10-27 18:49:40] ...trap/engines/engine.py.224: train: epoch 82 | batch 1800/2304 [I 2022-10-27 18:49:40] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:48 [I 2022-10-27 18:49:40] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00247 [I 2022-10-27 18:49:40] ...trap/engines/engine.py.230: loss: 0.12753 [I 2022-10-27 18:49:40] ...optimizers/trijoint.py.151: total_norm: 10.173656 [I 2022-10-27 18:49:46] ...trap/engines/engine.py.224: train: epoch 82 | batch 1810/2304 [I 2022-10-27 18:49:46] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:36 [I 2022-10-27 18:49:46] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00248 [I 2022-10-27 18:49:46] ...trap/engines/engine.py.230: loss: 0.08693 [I 2022-10-27 18:49:46] ...optimizers/trijoint.py.151: total_norm: 14.520819 [I 2022-10-27 18:49:52] ...trap/engines/engine.py.224: train: epoch 82 | batch 1820/2304 [I 2022-10-27 18:49:52] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:29 [I 2022-10-27 18:49:52] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00242 [I 2022-10-27 18:49:52] ...trap/engines/engine.py.230: loss: 0.13640 [I 2022-10-27 18:49:52] ...optimizers/trijoint.py.151: total_norm: 9.585594 [I 2022-10-27 18:49:57] ...trap/engines/engine.py.224: train: epoch 82 | batch 1830/2304 [I 2022-10-27 18:49:57] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:24 [I 2022-10-27 18:49:57] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00244 [I 2022-10-27 18:49:57] ...trap/engines/engine.py.230: loss: 0.13255 [I 2022-10-27 18:49:57] ...optimizers/trijoint.py.151: total_norm: 15.701228 [I 2022-10-27 18:50:03] ...trap/engines/engine.py.224: train: epoch 82 | batch 1840/2304 [I 2022-10-27 18:50:03] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:18 [I 2022-10-27 18:50:03] ...trap/engines/engine.py.229: process: 0.57136 | load: 0.00245 [I 2022-10-27 18:50:03] ...trap/engines/engine.py.230: loss: 0.10350 [I 2022-10-27 18:50:03] ...optimizers/trijoint.py.151: total_norm: 11.522898 [I 2022-10-27 18:50:08] ...trap/engines/engine.py.224: train: epoch 82 | batch 1850/2304 [I 2022-10-27 18:50:08] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:04:13 [I 2022-10-27 18:50:08] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00241 [I 2022-10-27 18:50:08] ...trap/engines/engine.py.230: loss: 0.14898 [I 2022-10-27 18:50:08] ...optimizers/trijoint.py.151: total_norm: 18.215336 [I 2022-10-27 18:50:14] ...trap/engines/engine.py.224: train: epoch 82 | batch 1860/2304 [I 2022-10-27 18:50:14] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:04:07 [I 2022-10-27 18:50:14] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00244 [I 2022-10-27 18:50:14] ...trap/engines/engine.py.230: loss: 0.20233 [I 2022-10-27 18:50:14] ...optimizers/trijoint.py.151: total_norm: 21.901026 [I 2022-10-27 18:50:20] ...trap/engines/engine.py.224: train: epoch 82 | batch 1870/2304 [I 2022-10-27 18:50:20] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:04:09 [I 2022-10-27 18:50:20] ...trap/engines/engine.py.229: process: 0.56443 | load: 0.00266 [I 2022-10-27 18:50:20] ...trap/engines/engine.py.230: loss: 0.11814 [I 2022-10-27 18:50:20] ...optimizers/trijoint.py.151: total_norm: 18.591373 [I 2022-10-27 18:50:26] ...trap/engines/engine.py.224: train: epoch 82 | batch 1880/2304 [I 2022-10-27 18:50:26] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:04:00 [I 2022-10-27 18:50:26] ...trap/engines/engine.py.229: process: 0.56322 | load: 0.00242 [I 2022-10-27 18:50:26] ...trap/engines/engine.py.230: loss: 0.10122 [I 2022-10-27 18:50:26] ...optimizers/trijoint.py.151: total_norm: 14.090290 [I 2022-10-27 18:50:31] ...trap/engines/engine.py.224: train: epoch 82 | batch 1890/2304 [I 2022-10-27 18:50:31] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:54 [I 2022-10-27 18:50:31] ...trap/engines/engine.py.229: process: 0.56458 | load: 0.00241 [I 2022-10-27 18:50:31] ...trap/engines/engine.py.230: loss: 0.13534 [I 2022-10-27 18:50:31] ...optimizers/trijoint.py.151: total_norm: 10.044702 [I 2022-10-27 18:50:37] ...trap/engines/engine.py.224: train: epoch 82 | batch 1900/2304 [I 2022-10-27 18:50:37] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:46 [I 2022-10-27 18:50:37] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00243 [I 2022-10-27 18:50:37] ...trap/engines/engine.py.230: loss: 0.17566 [I 2022-10-27 18:50:37] ...optimizers/trijoint.py.151: total_norm: 14.754892 [I 2022-10-27 18:50:42] ...trap/engines/engine.py.224: train: epoch 82 | batch 1910/2304 [I 2022-10-27 18:50:42] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:39 [I 2022-10-27 18:50:42] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00243 [I 2022-10-27 18:50:42] ...trap/engines/engine.py.230: loss: 0.19435 [I 2022-10-27 18:50:42] ...optimizers/trijoint.py.151: total_norm: 60.151344 [I 2022-10-27 18:50:48] ...trap/engines/engine.py.224: train: epoch 82 | batch 1920/2304 [I 2022-10-27 18:50:48] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:36 [I 2022-10-27 18:50:48] ...trap/engines/engine.py.229: process: 0.57543 | load: 0.00244 [I 2022-10-27 18:50:48] ...trap/engines/engine.py.230: loss: 0.10076 [I 2022-10-27 18:50:48] ...optimizers/trijoint.py.151: total_norm: 16.624908 [I 2022-10-27 18:50:54] ...trap/engines/engine.py.224: train: epoch 82 | batch 1930/2304 [I 2022-10-27 18:50:54] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:38 [I 2022-10-27 18:50:54] ...trap/engines/engine.py.229: process: 0.57726 | load: 0.00243 [I 2022-10-27 18:50:54] ...trap/engines/engine.py.230: loss: 0.10047 [I 2022-10-27 18:50:54] ...optimizers/trijoint.py.151: total_norm: 15.331072 [I 2022-10-27 18:51:00] ...trap/engines/engine.py.224: train: epoch 82 | batch 1940/2304 [I 2022-10-27 18:51:00] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:28 [I 2022-10-27 18:51:00] ...trap/engines/engine.py.229: process: 0.56905 | load: 0.00243 [I 2022-10-27 18:51:00] ...trap/engines/engine.py.230: loss: 0.12930 [I 2022-10-27 18:51:00] ...optimizers/trijoint.py.151: total_norm: 28.025383 [I 2022-10-27 18:51:05] ...trap/engines/engine.py.224: train: epoch 82 | batch 1950/2304 [I 2022-10-27 18:51:05] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:21 [I 2022-10-27 18:51:05] ...trap/engines/engine.py.229: process: 0.57257 | load: 0.00246 [I 2022-10-27 18:51:05] ...trap/engines/engine.py.230: loss: 0.14854 [I 2022-10-27 18:51:05] ...optimizers/trijoint.py.151: total_norm: 27.913197 [I 2022-10-27 18:51:11] ...trap/engines/engine.py.224: train: epoch 82 | batch 1960/2304 [I 2022-10-27 18:51:11] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:03:17 [I 2022-10-27 18:51:11] ...trap/engines/engine.py.229: process: 0.57081 | load: 0.00258 [I 2022-10-27 18:51:11] ...trap/engines/engine.py.230: loss: 0.18208 [I 2022-10-27 18:51:11] ...optimizers/trijoint.py.151: total_norm: 18.347191 [I 2022-10-27 18:51:17] ...trap/engines/engine.py.224: train: epoch 82 | batch 1970/2304 [I 2022-10-27 18:51:17] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:03:06 [I 2022-10-27 18:51:17] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00249 [I 2022-10-27 18:51:17] ...trap/engines/engine.py.230: loss: 0.12958 [I 2022-10-27 18:51:17] ...optimizers/trijoint.py.151: total_norm: 30.182125 [I 2022-10-27 18:51:22] ...trap/engines/engine.py.224: train: epoch 82 | batch 1980/2304 [I 2022-10-27 18:51:22] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:03:00 [I 2022-10-27 18:51:22] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00254 [I 2022-10-27 18:51:22] ...trap/engines/engine.py.230: loss: 0.13192 [I 2022-10-27 18:51:22] ...optimizers/trijoint.py.151: total_norm: 17.027979 [I 2022-10-27 18:51:28] ...trap/engines/engine.py.224: train: epoch 82 | batch 1990/2304 [I 2022-10-27 18:51:28] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:56 [I 2022-10-27 18:51:28] ...trap/engines/engine.py.229: process: 0.54715 | load: 0.00252 [I 2022-10-27 18:51:28] ...trap/engines/engine.py.230: loss: 0.07359 [I 2022-10-27 18:51:28] ...optimizers/trijoint.py.151: total_norm: 10.013112 [I 2022-10-27 18:51:34] ...trap/engines/engine.py.224: train: epoch 82 | batch 2000/2304 [I 2022-10-27 18:51:34] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:49 [I 2022-10-27 18:51:34] ...trap/engines/engine.py.229: process: 0.55192 | load: 0.00248 [I 2022-10-27 18:51:34] ...trap/engines/engine.py.230: loss: 0.17072 [I 2022-10-27 18:51:34] ...optimizers/trijoint.py.151: total_norm: 12.680413 [I 2022-10-27 18:51:39] ...trap/engines/engine.py.224: train: epoch 82 | batch 2010/2304 [I 2022-10-27 18:51:39] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:44 [I 2022-10-27 18:51:39] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00258 [I 2022-10-27 18:51:39] ...trap/engines/engine.py.230: loss: 0.14161 [I 2022-10-27 18:51:39] ...optimizers/trijoint.py.151: total_norm: 8.802387 [I 2022-10-27 18:51:45] ...trap/engines/engine.py.224: train: epoch 82 | batch 2020/2304 [I 2022-10-27 18:51:45] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:38 [I 2022-10-27 18:51:45] ...trap/engines/engine.py.229: process: 0.55817 | load: 0.00264 [I 2022-10-27 18:51:45] ...trap/engines/engine.py.230: loss: 0.13734 [I 2022-10-27 18:51:45] ...optimizers/trijoint.py.151: total_norm: 30.284851 [I 2022-10-27 18:51:50] ...trap/engines/engine.py.224: train: epoch 82 | batch 2030/2304 [I 2022-10-27 18:51:50] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:32 [I 2022-10-27 18:51:50] ...trap/engines/engine.py.229: process: 0.56185 | load: 0.00253 [I 2022-10-27 18:51:50] ...trap/engines/engine.py.230: loss: 0.10506 [I 2022-10-27 18:51:50] ...optimizers/trijoint.py.151: total_norm: 9.995728 [I 2022-10-27 18:51:56] ...trap/engines/engine.py.224: train: epoch 82 | batch 2040/2304 [I 2022-10-27 18:51:56] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:27 [I 2022-10-27 18:51:56] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00296 [I 2022-10-27 18:51:56] ...trap/engines/engine.py.230: loss: 0.11626 [I 2022-10-27 18:51:56] ...optimizers/trijoint.py.151: total_norm: 11.127917 [I 2022-10-27 18:52:01] ...trap/engines/engine.py.224: train: epoch 82 | batch 2050/2304 [I 2022-10-27 18:52:01] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:20 [I 2022-10-27 18:52:01] ...trap/engines/engine.py.229: process: 0.54685 | load: 0.00290 [I 2022-10-27 18:52:01] ...trap/engines/engine.py.230: loss: 0.16923 [I 2022-10-27 18:52:01] ...optimizers/trijoint.py.151: total_norm: 19.535397 [I 2022-10-27 18:52:07] ...trap/engines/engine.py.224: train: epoch 82 | batch 2060/2304 [I 2022-10-27 18:52:07] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:02:16 [I 2022-10-27 18:52:07] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00262 [I 2022-10-27 18:52:07] ...trap/engines/engine.py.230: loss: 0.16108 [I 2022-10-27 18:52:07] ...optimizers/trijoint.py.151: total_norm: 14.878556 [I 2022-10-27 18:52:13] ...trap/engines/engine.py.224: train: epoch 82 | batch 2070/2304 [I 2022-10-27 18:52:13] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:10 [I 2022-10-27 18:52:13] ...trap/engines/engine.py.229: process: 0.56411 | load: 0.00266 [I 2022-10-27 18:52:13] ...trap/engines/engine.py.230: loss: 0.19682 [I 2022-10-27 18:52:13] ...optimizers/trijoint.py.151: total_norm: 21.044146 [I 2022-10-27 18:52:18] ...trap/engines/engine.py.224: train: epoch 82 | batch 2080/2304 [I 2022-10-27 18:52:18] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:02:04 [I 2022-10-27 18:52:18] ...trap/engines/engine.py.229: process: 0.53978 | load: 0.00267 [I 2022-10-27 18:52:18] ...trap/engines/engine.py.230: loss: 0.15515 [I 2022-10-27 18:52:18] ...optimizers/trijoint.py.151: total_norm: 10.422565 [I 2022-10-27 18:52:24] ...trap/engines/engine.py.224: train: epoch 82 | batch 2090/2304 [I 2022-10-27 18:52:24] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:58 [I 2022-10-27 18:52:24] ...trap/engines/engine.py.229: process: 0.55102 | load: 0.00257 [I 2022-10-27 18:52:24] ...trap/engines/engine.py.230: loss: 0.18004 [I 2022-10-27 18:52:24] ...optimizers/trijoint.py.151: total_norm: 13.621234 [I 2022-10-27 18:52:29] ...trap/engines/engine.py.224: train: epoch 82 | batch 2100/2304 [I 2022-10-27 18:52:29] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:53 [I 2022-10-27 18:52:29] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00253 [I 2022-10-27 18:52:29] ...trap/engines/engine.py.230: loss: 0.12581 [I 2022-10-27 18:52:29] ...optimizers/trijoint.py.151: total_norm: 19.709827 [I 2022-10-27 18:52:35] ...trap/engines/engine.py.224: train: epoch 82 | batch 2110/2304 [I 2022-10-27 18:52:35] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:48 [I 2022-10-27 18:52:35] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00253 [I 2022-10-27 18:52:35] ...trap/engines/engine.py.230: loss: 0.13236 [I 2022-10-27 18:52:35] ...optimizers/trijoint.py.151: total_norm: 12.477937 [I 2022-10-27 18:52:40] ...trap/engines/engine.py.224: train: epoch 82 | batch 2120/2304 [I 2022-10-27 18:52:40] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:43 [I 2022-10-27 18:52:40] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00242 [I 2022-10-27 18:52:40] ...trap/engines/engine.py.230: loss: 0.10829 [I 2022-10-27 18:52:40] ...optimizers/trijoint.py.151: total_norm: 12.152053 [I 2022-10-27 18:52:46] ...trap/engines/engine.py.224: train: epoch 82 | batch 2130/2304 [I 2022-10-27 18:52:46] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:37 [I 2022-10-27 18:52:46] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00251 [I 2022-10-27 18:52:46] ...trap/engines/engine.py.230: loss: 0.12424 [I 2022-10-27 18:52:46] ...optimizers/trijoint.py.151: total_norm: 21.335184 [I 2022-10-27 18:52:52] ...trap/engines/engine.py.224: train: epoch 82 | batch 2140/2304 [I 2022-10-27 18:52:52] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:31 [I 2022-10-27 18:52:52] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00241 [I 2022-10-27 18:52:52] ...trap/engines/engine.py.230: loss: 0.11231 [I 2022-10-27 18:52:52] ...optimizers/trijoint.py.151: total_norm: 11.754105 [I 2022-10-27 18:52:57] ...trap/engines/engine.py.224: train: epoch 82 | batch 2150/2304 [I 2022-10-27 18:52:57] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:25 [I 2022-10-27 18:52:57] ...trap/engines/engine.py.229: process: 0.54771 | load: 0.00245 [I 2022-10-27 18:52:57] ...trap/engines/engine.py.230: loss: 0.13492 [I 2022-10-27 18:52:57] ...optimizers/trijoint.py.151: total_norm: 13.808764 [I 2022-10-27 18:53:03] ...trap/engines/engine.py.224: train: epoch 82 | batch 2160/2304 [I 2022-10-27 18:53:03] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:01:19 [I 2022-10-27 18:53:03] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00248 [I 2022-10-27 18:53:03] ...trap/engines/engine.py.230: loss: 0.11352 [I 2022-10-27 18:53:03] ...optimizers/trijoint.py.151: total_norm: 16.573032 [I 2022-10-27 18:53:08] ...trap/engines/engine.py.224: train: epoch 82 | batch 2170/2304 [I 2022-10-27 18:53:08] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:14 [I 2022-10-27 18:53:08] ...trap/engines/engine.py.229: process: 0.56036 | load: 0.00242 [I 2022-10-27 18:53:08] ...trap/engines/engine.py.230: loss: 0.09091 [I 2022-10-27 18:53:08] ...optimizers/trijoint.py.151: total_norm: 12.232864 [I 2022-10-27 18:53:14] ...trap/engines/engine.py.224: train: epoch 82 | batch 2180/2304 [I 2022-10-27 18:53:14] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:01:09 [I 2022-10-27 18:53:14] ...trap/engines/engine.py.229: process: 0.55800 | load: 0.00243 [I 2022-10-27 18:53:14] ...trap/engines/engine.py.230: loss: 0.14029 [I 2022-10-27 18:53:14] ...optimizers/trijoint.py.151: total_norm: 16.803852 [I 2022-10-27 18:53:20] ...trap/engines/engine.py.224: train: epoch 82 | batch 2190/2304 [I 2022-10-27 18:53:20] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:01:03 [I 2022-10-27 18:53:20] ...trap/engines/engine.py.229: process: 0.55597 | load: 0.00248 [I 2022-10-27 18:53:20] ...trap/engines/engine.py.230: loss: 0.09851 [I 2022-10-27 18:53:20] ...optimizers/trijoint.py.151: total_norm: 9.910559 [I 2022-10-27 18:53:25] ...trap/engines/engine.py.224: train: epoch 82 | batch 2200/2304 [I 2022-10-27 18:53:25] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:58 [I 2022-10-27 18:53:25] ...trap/engines/engine.py.229: process: 0.56462 | load: 0.00337 [I 2022-10-27 18:53:25] ...trap/engines/engine.py.230: loss: 0.21735 [I 2022-10-27 18:53:25] ...optimizers/trijoint.py.151: total_norm: 21.810759 [I 2022-10-27 18:53:31] ...trap/engines/engine.py.224: train: epoch 82 | batch 2210/2304 [I 2022-10-27 18:53:31] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:52 [I 2022-10-27 18:53:31] ...trap/engines/engine.py.229: process: 0.55059 | load: 0.00281 [I 2022-10-27 18:53:31] ...trap/engines/engine.py.230: loss: 0.10248 [I 2022-10-27 18:53:31] ...optimizers/trijoint.py.151: total_norm: 8.920412 [I 2022-10-27 18:53:36] ...trap/engines/engine.py.224: train: epoch 82 | batch 2220/2304 [I 2022-10-27 18:53:36] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:46 [I 2022-10-27 18:53:36] ...trap/engines/engine.py.229: process: 0.54545 | load: 0.00266 [I 2022-10-27 18:53:36] ...trap/engines/engine.py.230: loss: 0.10419 [I 2022-10-27 18:53:36] ...optimizers/trijoint.py.151: total_norm: 8.486423 [I 2022-10-27 18:53:42] ...trap/engines/engine.py.224: train: epoch 82 | batch 2230/2304 [I 2022-10-27 18:53:42] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:41 [I 2022-10-27 18:53:42] ...trap/engines/engine.py.229: process: 0.55453 | load: 0.00258 [I 2022-10-27 18:53:42] ...trap/engines/engine.py.230: loss: 0.10248 [I 2022-10-27 18:53:42] ...optimizers/trijoint.py.151: total_norm: 9.854678 [I 2022-10-27 18:53:47] ...trap/engines/engine.py.224: train: epoch 82 | batch 2240/2304 [I 2022-10-27 18:53:47] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:35 [I 2022-10-27 18:53:47] ...trap/engines/engine.py.229: process: 0.56473 | load: 0.00249 [I 2022-10-27 18:53:47] ...trap/engines/engine.py.230: loss: 0.11199 [I 2022-10-27 18:53:47] ...optimizers/trijoint.py.151: total_norm: 16.082909 [I 2022-10-27 18:53:53] ...trap/engines/engine.py.224: train: epoch 82 | batch 2250/2304 [I 2022-10-27 18:53:53] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:30 [I 2022-10-27 18:53:53] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00252 [I 2022-10-27 18:53:53] ...trap/engines/engine.py.230: loss: 0.12105 [I 2022-10-27 18:53:53] ...optimizers/trijoint.py.151: total_norm: 12.113167 [I 2022-10-27 18:53:59] ...trap/engines/engine.py.224: train: epoch 82 | batch 2260/2304 [I 2022-10-27 18:53:59] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:24 [I 2022-10-27 18:53:59] ...trap/engines/engine.py.229: process: 0.56102 | load: 0.00250 [I 2022-10-27 18:53:59] ...trap/engines/engine.py.230: loss: 0.14647 [I 2022-10-27 18:53:59] ...optimizers/trijoint.py.151: total_norm: 29.240683 [I 2022-10-27 18:54:04] ...trap/engines/engine.py.224: train: epoch 82 | batch 2270/2304 [I 2022-10-27 18:54:04] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:19 [I 2022-10-27 18:54:04] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00250 [I 2022-10-27 18:54:04] ...trap/engines/engine.py.230: loss: 0.11929 [I 2022-10-27 18:54:04] ...optimizers/trijoint.py.151: total_norm: 12.688454 [I 2022-10-27 18:54:10] ...trap/engines/engine.py.224: train: epoch 82 | batch 2280/2304 [I 2022-10-27 18:54:10] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:13 [I 2022-10-27 18:54:10] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00249 [I 2022-10-27 18:54:10] ...trap/engines/engine.py.230: loss: 0.10625 [I 2022-10-27 18:54:10] ...optimizers/trijoint.py.151: total_norm: 20.889389 [I 2022-10-27 18:54:15] ...trap/engines/engine.py.224: train: epoch 82 | batch 2290/2304 [I 2022-10-27 18:54:15] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:07 [I 2022-10-27 18:54:15] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00241 [I 2022-10-27 18:54:15] ...trap/engines/engine.py.230: loss: 0.12801 [I 2022-10-27 18:54:15] ...optimizers/trijoint.py.151: total_norm: 12.686052 [I 2022-10-27 18:54:21] ...trap/engines/engine.py.224: train: epoch 82 | batch 2300/2304 [I 2022-10-27 18:54:21] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:02 [I 2022-10-27 18:54:21] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00015 [I 2022-10-27 18:54:21] ...trap/engines/engine.py.230: loss: 0.12231 [I 2022-10-27 18:54:21] ...optimizers/trijoint.py.151: total_norm: 26.068850 [I 2022-10-27 18:54:23] ...trap/engines/engine.py.224: train: epoch 82 | batch 2304/2304 [I 2022-10-27 18:54:23] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:00:00 [I 2022-10-27 18:54:23] ...trap/engines/engine.py.229: process: 0.54849 | load: 0.00017 [I 2022-10-27 18:54:23] ...trap/engines/engine.py.230: loss: 0.21039 [I 2022-10-27 18:54:23] ...optimizers/trijoint.py.151: total_norm: 15.717840 [S 2022-10-27 18:54:24] ...trap/engines/engine.py.129: train_epoch.epoch: 82 [S 2022-10-27 18:54:24] ...trap/engines/engine.py.129: train_epoch.loss: 0.13642560378395813 [I 2022-10-27 18:54:24] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 18:54:26] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 82 [I 2022-10-27 18:54:26] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 18:54:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 18:54:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 18:54:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 18:54:30] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 18:54:31] ...trap/engines/engine.py.321: eval: epoch 82 | batch 0/510 [I 2022-10-27 18:54:31] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:42:57 [I 2022-10-27 18:54:31] ...trap/engines/engine.py.326: process: 5.05447 | load: 3.58836 [I 2022-10-27 18:54:34] ...trap/engines/engine.py.321: eval: epoch 82 | batch 10/510 [I 2022-10-27 18:54:34] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:00 [I 2022-10-27 18:54:34] ...trap/engines/engine.py.326: process: 0.34253 | load: 0.00016 [I 2022-10-27 18:54:37] ...trap/engines/engine.py.321: eval: epoch 82 | batch 20/510 [I 2022-10-27 18:54:37] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:08 [I 2022-10-27 18:54:37] ...trap/engines/engine.py.326: process: 0.33134 | load: 0.00019 [I 2022-10-27 18:54:41] ...trap/engines/engine.py.321: eval: epoch 82 | batch 30/510 [I 2022-10-27 18:54:41] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:41 [I 2022-10-27 18:54:41] ...trap/engines/engine.py.326: process: 0.32369 | load: 0.00017 [I 2022-10-27 18:54:44] ...trap/engines/engine.py.321: eval: epoch 82 | batch 40/510 [I 2022-10-27 18:54:44] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:34 [I 2022-10-27 18:54:44] ...trap/engines/engine.py.326: process: 0.32888 | load: 0.00017 [I 2022-10-27 18:54:47] ...trap/engines/engine.py.321: eval: epoch 82 | batch 50/510 [I 2022-10-27 18:54:47] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:32 [I 2022-10-27 18:54:47] ...trap/engines/engine.py.326: process: 0.32801 | load: 0.00017 [I 2022-10-27 18:54:51] ...trap/engines/engine.py.321: eval: epoch 82 | batch 60/510 [I 2022-10-27 18:54:51] ...trap/engines/engine.py.325: elapsed: 0:00:24 | left: 0:02:30 [I 2022-10-27 18:54:51] ...trap/engines/engine.py.326: process: 0.33673 | load: 0.00018 [I 2022-10-27 18:54:54] ...trap/engines/engine.py.321: eval: epoch 82 | batch 70/510 [I 2022-10-27 18:54:54] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:28 [I 2022-10-27 18:54:54] ...trap/engines/engine.py.326: process: 0.34449 | load: 0.00018 [I 2022-10-27 18:54:57] ...trap/engines/engine.py.321: eval: epoch 82 | batch 80/510 [I 2022-10-27 18:54:57] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:24 [I 2022-10-27 18:54:57] ...trap/engines/engine.py.326: process: 0.33190 | load: 0.00017 [I 2022-10-27 18:55:01] ...trap/engines/engine.py.321: eval: epoch 82 | batch 90/510 [I 2022-10-27 18:55:01] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:22 [I 2022-10-27 18:55:01] ...trap/engines/engine.py.326: process: 0.32056 | load: 0.00020 [I 2022-10-27 18:55:04] ...trap/engines/engine.py.321: eval: epoch 82 | batch 100/510 [I 2022-10-27 18:55:04] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:12 [I 2022-10-27 18:55:04] ...trap/engines/engine.py.326: process: 0.28328 | load: 0.00021 [I 2022-10-27 18:55:07] ...trap/engines/engine.py.321: eval: epoch 82 | batch 110/510 [I 2022-10-27 18:55:07] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-27 18:55:07] ...trap/engines/engine.py.326: process: 0.29232 | load: 0.00014 [I 2022-10-27 18:55:10] ...trap/engines/engine.py.321: eval: epoch 82 | batch 120/510 [I 2022-10-27 18:55:10] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 18:55:10] ...trap/engines/engine.py.326: process: 0.28925 | load: 0.00014 [I 2022-10-27 18:55:13] ...trap/engines/engine.py.321: eval: epoch 82 | batch 130/510 [I 2022-10-27 18:55:13] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-27 18:55:13] ...trap/engines/engine.py.326: process: 0.28967 | load: 0.00018 [I 2022-10-27 18:55:16] ...trap/engines/engine.py.321: eval: epoch 82 | batch 140/510 [I 2022-10-27 18:55:16] ...trap/engines/engine.py.325: elapsed: 0:00:49 | left: 0:01:46 [I 2022-10-27 18:55:16] ...trap/engines/engine.py.326: process: 0.28550 | load: 0.00017 [I 2022-10-27 18:55:19] ...trap/engines/engine.py.321: eval: epoch 82 | batch 150/510 [I 2022-10-27 18:55:19] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:43 [I 2022-10-27 18:55:19] ...trap/engines/engine.py.326: process: 0.29059 | load: 0.00022 [I 2022-10-27 18:55:21] ...trap/engines/engine.py.321: eval: epoch 82 | batch 160/510 [I 2022-10-27 18:55:21] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:41 [I 2022-10-27 18:55:21] ...trap/engines/engine.py.326: process: 0.29232 | load: 0.00015 [I 2022-10-27 18:55:24] ...trap/engines/engine.py.321: eval: epoch 82 | batch 170/510 [I 2022-10-27 18:55:24] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 18:55:24] ...trap/engines/engine.py.326: process: 0.29071 | load: 0.00026 [I 2022-10-27 18:55:27] ...trap/engines/engine.py.321: eval: epoch 82 | batch 180/510 [I 2022-10-27 18:55:27] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:36 [I 2022-10-27 18:55:27] ...trap/engines/engine.py.326: process: 0.29301 | load: 0.00017 [I 2022-10-27 18:55:30] ...trap/engines/engine.py.321: eval: epoch 82 | batch 190/510 [I 2022-10-27 18:55:30] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:33 [I 2022-10-27 18:55:30] ...trap/engines/engine.py.326: process: 0.29268 | load: 0.00016 [I 2022-10-27 18:55:33] ...trap/engines/engine.py.321: eval: epoch 82 | batch 200/510 [I 2022-10-27 18:55:33] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:30 [I 2022-10-27 18:55:33] ...trap/engines/engine.py.326: process: 0.28795 | load: 0.00015 [I 2022-10-27 18:55:36] ...trap/engines/engine.py.321: eval: epoch 82 | batch 210/510 [I 2022-10-27 18:55:36] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:27 [I 2022-10-27 18:55:36] ...trap/engines/engine.py.326: process: 0.27883 | load: 0.00015 [I 2022-10-27 18:55:39] ...trap/engines/engine.py.321: eval: epoch 82 | batch 220/510 [I 2022-10-27 18:55:39] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 18:55:39] ...trap/engines/engine.py.326: process: 0.28515 | load: 0.00015 [I 2022-10-27 18:55:42] ...trap/engines/engine.py.321: eval: epoch 82 | batch 230/510 [I 2022-10-27 18:55:42] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 18:55:42] ...trap/engines/engine.py.326: process: 0.29136 | load: 0.00014 [I 2022-10-27 18:55:45] ...trap/engines/engine.py.321: eval: epoch 82 | batch 240/510 [I 2022-10-27 18:55:45] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:18 [I 2022-10-27 18:55:45] ...trap/engines/engine.py.326: process: 0.29087 | load: 0.00019 [I 2022-10-27 18:55:47] ...trap/engines/engine.py.321: eval: epoch 82 | batch 250/510 [I 2022-10-27 18:55:47] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:15 [I 2022-10-27 18:55:47] ...trap/engines/engine.py.326: process: 0.29097 | load: 0.00016 [I 2022-10-27 18:55:50] ...trap/engines/engine.py.321: eval: epoch 82 | batch 260/510 [I 2022-10-27 18:55:50] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 18:55:50] ...trap/engines/engine.py.326: process: 0.28741 | load: 0.00021 [I 2022-10-27 18:55:53] ...trap/engines/engine.py.321: eval: epoch 82 | batch 270/510 [I 2022-10-27 18:55:53] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 18:55:53] ...trap/engines/engine.py.326: process: 0.29893 | load: 0.00017 [I 2022-10-27 18:55:56] ...trap/engines/engine.py.321: eval: epoch 82 | batch 280/510 [I 2022-10-27 18:55:56] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:07 [I 2022-10-27 18:55:56] ...trap/engines/engine.py.326: process: 0.29345 | load: 0.00023 [I 2022-10-27 18:55:59] ...trap/engines/engine.py.321: eval: epoch 82 | batch 290/510 [I 2022-10-27 18:55:59] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:04 [I 2022-10-27 18:55:59] ...trap/engines/engine.py.326: process: 0.29406 | load: 0.00016 [I 2022-10-27 18:56:02] ...trap/engines/engine.py.321: eval: epoch 82 | batch 300/510 [I 2022-10-27 18:56:02] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 18:56:02] ...trap/engines/engine.py.326: process: 0.28384 | load: 0.00021 [I 2022-10-27 18:56:05] ...trap/engines/engine.py.321: eval: epoch 82 | batch 310/510 [I 2022-10-27 18:56:05] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:58 [I 2022-10-27 18:56:05] ...trap/engines/engine.py.326: process: 0.29167 | load: 0.00021 [I 2022-10-27 18:56:08] ...trap/engines/engine.py.321: eval: epoch 82 | batch 320/510 [I 2022-10-27 18:56:08] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 18:56:08] ...trap/engines/engine.py.326: process: 0.28078 | load: 0.00015 [I 2022-10-27 18:56:11] ...trap/engines/engine.py.321: eval: epoch 82 | batch 330/510 [I 2022-10-27 18:56:11] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-27 18:56:11] ...trap/engines/engine.py.326: process: 0.29591 | load: 0.00015 [I 2022-10-27 18:56:14] ...trap/engines/engine.py.321: eval: epoch 82 | batch 340/510 [I 2022-10-27 18:56:14] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 18:56:14] ...trap/engines/engine.py.326: process: 0.30555 | load: 0.00016 [I 2022-10-27 18:56:17] ...trap/engines/engine.py.321: eval: epoch 82 | batch 350/510 [I 2022-10-27 18:56:17] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:48 [I 2022-10-27 18:56:17] ...trap/engines/engine.py.326: process: 0.31510 | load: 0.00016 [I 2022-10-27 18:56:20] ...trap/engines/engine.py.321: eval: epoch 82 | batch 360/510 [I 2022-10-27 18:56:20] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:44 [I 2022-10-27 18:56:20] ...trap/engines/engine.py.326: process: 0.28842 | load: 0.00021 [I 2022-10-27 18:56:22] ...trap/engines/engine.py.321: eval: epoch 82 | batch 370/510 [I 2022-10-27 18:56:22] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 18:56:22] ...trap/engines/engine.py.326: process: 0.29270 | load: 0.00022 [I 2022-10-27 18:56:25] ...trap/engines/engine.py.321: eval: epoch 82 | batch 380/510 [I 2022-10-27 18:56:25] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:40 [I 2022-10-27 18:56:25] ...trap/engines/engine.py.326: process: 0.29291 | load: 0.00017 [I 2022-10-27 18:56:28] ...trap/engines/engine.py.321: eval: epoch 82 | batch 390/510 [I 2022-10-27 18:56:28] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 18:56:28] ...trap/engines/engine.py.326: process: 0.29255 | load: 0.00015 [I 2022-10-27 18:56:31] ...trap/engines/engine.py.321: eval: epoch 82 | batch 400/510 [I 2022-10-27 18:56:31] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 18:56:31] ...trap/engines/engine.py.326: process: 0.28524 | load: 0.00015 [I 2022-10-27 18:56:34] ...trap/engines/engine.py.321: eval: epoch 82 | batch 410/510 [I 2022-10-27 18:56:34] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 18:56:34] ...trap/engines/engine.py.326: process: 0.29123 | load: 0.00016 [I 2022-10-27 18:56:37] ...trap/engines/engine.py.321: eval: epoch 82 | batch 420/510 [I 2022-10-27 18:56:37] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 18:56:37] ...trap/engines/engine.py.326: process: 0.28557 | load: 0.00015 [I 2022-10-27 18:56:40] ...trap/engines/engine.py.321: eval: epoch 82 | batch 430/510 [I 2022-10-27 18:56:40] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 18:56:40] ...trap/engines/engine.py.326: process: 0.28875 | load: 0.00016 [I 2022-10-27 18:56:43] ...trap/engines/engine.py.321: eval: epoch 82 | batch 440/510 [I 2022-10-27 18:56:43] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 18:56:43] ...trap/engines/engine.py.326: process: 0.29154 | load: 0.00015 [I 2022-10-27 18:56:46] ...trap/engines/engine.py.321: eval: epoch 82 | batch 450/510 [I 2022-10-27 18:56:46] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 18:56:46] ...trap/engines/engine.py.326: process: 0.30008 | load: 0.00016 [I 2022-10-27 18:56:49] ...trap/engines/engine.py.321: eval: epoch 82 | batch 460/510 [I 2022-10-27 18:56:49] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 18:56:49] ...trap/engines/engine.py.326: process: 0.28745 | load: 0.00025 [I 2022-10-27 18:56:51] ...trap/engines/engine.py.321: eval: epoch 82 | batch 470/510 [I 2022-10-27 18:56:51] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 18:56:51] ...trap/engines/engine.py.326: process: 0.29520 | load: 0.00018 [I 2022-10-27 18:56:54] ...trap/engines/engine.py.321: eval: epoch 82 | batch 480/510 [I 2022-10-27 18:56:54] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 18:56:54] ...trap/engines/engine.py.326: process: 0.29460 | load: 0.00015 [I 2022-10-27 18:56:57] ...trap/engines/engine.py.321: eval: epoch 82 | batch 490/510 [I 2022-10-27 18:56:57] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 18:56:57] ...trap/engines/engine.py.326: process: 0.29189 | load: 0.00021 [I 2022-10-27 18:57:00] ...trap/engines/engine.py.321: eval: epoch 82 | batch 500/510 [I 2022-10-27 18:57:00] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:03 [I 2022-10-27 18:57:00] ...trap/engines/engine.py.326: process: 0.32905 | load: 0.00016 [I 2022-10-27 18:57:03] ...trap/engines/engine.py.321: eval: epoch 82 | batch 510/510 [I 2022-10-27 18:57:03] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 18:57:03] ...trap/engines/engine.py.326: process: 0.28552 | load: 0.00012 [S 2022-10-27 18:57:04] ...trap/engines/engine.py.132: eval_epoch.epoch: 82 [I 2022-10-27 18:57:04] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.722600 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.901500 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.932500 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.714300 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.899900 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933200 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016954 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007606 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007928 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013929 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008780 [S 2022-10-27 18:57:08] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006493 [I 2022-10-27 18:57:09] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 18:57:09] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 18:57:10] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 18:57:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 18:57:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 18:57:14] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 18:57:15] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 18:57:21] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 18:57:21] ...trap/engines/engine.py.163: Training model on trainset for epoch 83 [I 2022-10-27 18:57:21] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 18:57:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 0/2304 [I 2022-10-27 18:57:23] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:55 [I 2022-10-27 18:57:23] ...trap/engines/engine.py.229: process: 2.78468 | load: 2.16219 [I 2022-10-27 18:57:23] ...trap/engines/engine.py.230: loss: 0.18061 [I 2022-10-27 18:57:23] ...optimizers/trijoint.py.151: total_norm: 12.734687 [I 2022-10-27 18:57:29] ...trap/engines/engine.py.224: train: epoch 83 | batch 10/2304 [I 2022-10-27 18:57:29] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:29 [I 2022-10-27 18:57:29] ...trap/engines/engine.py.229: process: 0.53647 | load: 0.00421 [I 2022-10-27 18:57:29] ...trap/engines/engine.py.230: loss: 0.12650 [I 2022-10-27 18:57:29] ...optimizers/trijoint.py.151: total_norm: 32.445023 [I 2022-10-27 18:57:35] ...trap/engines/engine.py.224: train: epoch 83 | batch 20/2304 [I 2022-10-27 18:57:35] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:19 [I 2022-10-27 18:57:35] ...trap/engines/engine.py.229: process: 0.57995 | load: 0.00322 [I 2022-10-27 18:57:35] ...trap/engines/engine.py.230: loss: 0.16331 [I 2022-10-27 18:57:35] ...optimizers/trijoint.py.151: total_norm: 17.135277 [I 2022-10-27 18:57:40] ...trap/engines/engine.py.224: train: epoch 83 | batch 30/2304 [I 2022-10-27 18:57:40] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:08 [I 2022-10-27 18:57:40] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00338 [I 2022-10-27 18:57:40] ...trap/engines/engine.py.230: loss: 0.14789 [I 2022-10-27 18:57:40] ...optimizers/trijoint.py.151: total_norm: 31.079159 [I 2022-10-27 18:57:46] ...trap/engines/engine.py.224: train: epoch 83 | batch 40/2304 [I 2022-10-27 18:57:46] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:18 [I 2022-10-27 18:57:46] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00313 [I 2022-10-27 18:57:46] ...trap/engines/engine.py.230: loss: 0.14129 [I 2022-10-27 18:57:46] ...optimizers/trijoint.py.151: total_norm: 11.968596 [I 2022-10-27 18:57:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 50/2304 [I 2022-10-27 18:57:51] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:56 [I 2022-10-27 18:57:51] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00307 [I 2022-10-27 18:57:51] ...trap/engines/engine.py.230: loss: 0.13978 [I 2022-10-27 18:57:51] ...optimizers/trijoint.py.151: total_norm: 15.083054 [I 2022-10-27 18:57:57] ...trap/engines/engine.py.224: train: epoch 83 | batch 60/2304 [I 2022-10-27 18:57:57] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:48 [I 2022-10-27 18:57:57] ...trap/engines/engine.py.229: process: 0.56222 | load: 0.00272 [I 2022-10-27 18:57:57] ...trap/engines/engine.py.230: loss: 0.11270 [I 2022-10-27 18:57:57] ...optimizers/trijoint.py.151: total_norm: 9.431700 [I 2022-10-27 18:58:03] ...trap/engines/engine.py.224: train: epoch 83 | batch 70/2304 [I 2022-10-27 18:58:03] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:46 [I 2022-10-27 18:58:03] ...trap/engines/engine.py.229: process: 0.56486 | load: 0.00324 [I 2022-10-27 18:58:03] ...trap/engines/engine.py.230: loss: 0.15573 [I 2022-10-27 18:58:03] ...optimizers/trijoint.py.151: total_norm: 103.428291 [I 2022-10-27 18:58:08] ...trap/engines/engine.py.224: train: epoch 83 | batch 80/2304 [I 2022-10-27 18:58:08] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:43 [I 2022-10-27 18:58:08] ...trap/engines/engine.py.229: process: 0.56357 | load: 0.00296 [I 2022-10-27 18:58:08] ...trap/engines/engine.py.230: loss: 0.12997 [I 2022-10-27 18:58:08] ...optimizers/trijoint.py.151: total_norm: 15.542783 [I 2022-10-27 18:58:14] ...trap/engines/engine.py.224: train: epoch 83 | batch 90/2304 [I 2022-10-27 18:58:14] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:31 [I 2022-10-27 18:58:14] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00270 [I 2022-10-27 18:58:14] ...trap/engines/engine.py.230: loss: 0.11086 [I 2022-10-27 18:58:14] ...optimizers/trijoint.py.151: total_norm: 7.820940 [I 2022-10-27 18:58:19] ...trap/engines/engine.py.224: train: epoch 83 | batch 100/2304 [I 2022-10-27 18:58:19] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:43 [I 2022-10-27 18:58:19] ...trap/engines/engine.py.229: process: 0.57124 | load: 0.00286 [I 2022-10-27 18:58:19] ...trap/engines/engine.py.230: loss: 0.08661 [I 2022-10-27 18:58:19] ...optimizers/trijoint.py.151: total_norm: 8.353284 [I 2022-10-27 18:58:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 110/2304 [I 2022-10-27 18:58:25] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:30 [I 2022-10-27 18:58:25] ...trap/engines/engine.py.229: process: 0.56375 | load: 0.00260 [I 2022-10-27 18:58:25] ...trap/engines/engine.py.230: loss: 0.09803 [I 2022-10-27 18:58:25] ...optimizers/trijoint.py.151: total_norm: 11.741834 [I 2022-10-27 18:58:31] ...trap/engines/engine.py.224: train: epoch 83 | batch 120/2304 [I 2022-10-27 18:58:31] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:46 [I 2022-10-27 18:58:31] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00259 [I 2022-10-27 18:58:31] ...trap/engines/engine.py.230: loss: 0.09582 [I 2022-10-27 18:58:31] ...optimizers/trijoint.py.151: total_norm: 11.584708 [I 2022-10-27 18:58:36] ...trap/engines/engine.py.224: train: epoch 83 | batch 130/2304 [I 2022-10-27 18:58:36] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:22 [I 2022-10-27 18:58:36] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00248 [I 2022-10-27 18:58:36] ...trap/engines/engine.py.230: loss: 0.09625 [I 2022-10-27 18:58:36] ...optimizers/trijoint.py.151: total_norm: 12.686774 [I 2022-10-27 18:58:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 140/2304 [I 2022-10-27 18:58:42] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:07 [I 2022-10-27 18:58:42] ...trap/engines/engine.py.229: process: 0.55183 | load: 0.00291 [I 2022-10-27 18:58:42] ...trap/engines/engine.py.230: loss: 0.17676 [I 2022-10-27 18:58:42] ...optimizers/trijoint.py.151: total_norm: 21.624790 [I 2022-10-27 18:58:48] ...trap/engines/engine.py.224: train: epoch 83 | batch 150/2304 [I 2022-10-27 18:58:48] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:08 [I 2022-10-27 18:58:48] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00259 [I 2022-10-27 18:58:48] ...trap/engines/engine.py.230: loss: 0.16757 [I 2022-10-27 18:58:48] ...optimizers/trijoint.py.151: total_norm: 16.896612 [I 2022-10-27 18:58:53] ...trap/engines/engine.py.224: train: epoch 83 | batch 160/2304 [I 2022-10-27 18:58:53] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:53 [I 2022-10-27 18:58:53] ...trap/engines/engine.py.229: process: 0.54731 | load: 0.00270 [I 2022-10-27 18:58:53] ...trap/engines/engine.py.230: loss: 0.10426 [I 2022-10-27 18:58:53] ...optimizers/trijoint.py.151: total_norm: 8.567616 [I 2022-10-27 18:58:59] ...trap/engines/engine.py.224: train: epoch 83 | batch 170/2304 [I 2022-10-27 18:58:59] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:46 [I 2022-10-27 18:58:59] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00254 [I 2022-10-27 18:58:59] ...trap/engines/engine.py.230: loss: 0.09180 [I 2022-10-27 18:58:59] ...optimizers/trijoint.py.151: total_norm: 9.380517 [I 2022-10-27 18:59:04] ...trap/engines/engine.py.224: train: epoch 83 | batch 180/2304 [I 2022-10-27 18:59:04] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:39 [I 2022-10-27 18:59:04] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00252 [I 2022-10-27 18:59:04] ...trap/engines/engine.py.230: loss: 0.18671 [I 2022-10-27 18:59:04] ...optimizers/trijoint.py.151: total_norm: 19.706934 [I 2022-10-27 18:59:10] ...trap/engines/engine.py.224: train: epoch 83 | batch 190/2304 [I 2022-10-27 18:59:10] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:39 [I 2022-10-27 18:59:10] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00251 [I 2022-10-27 18:59:10] ...trap/engines/engine.py.230: loss: 0.23080 [I 2022-10-27 18:59:10] ...optimizers/trijoint.py.151: total_norm: 35.041061 [I 2022-10-27 18:59:15] ...trap/engines/engine.py.224: train: epoch 83 | batch 200/2304 [I 2022-10-27 18:59:15] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:40 [I 2022-10-27 18:59:15] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00256 [I 2022-10-27 18:59:15] ...trap/engines/engine.py.230: loss: 0.16349 [I 2022-10-27 18:59:15] ...optimizers/trijoint.py.151: total_norm: 13.921952 [I 2022-10-27 18:59:21] ...trap/engines/engine.py.224: train: epoch 83 | batch 210/2304 [I 2022-10-27 18:59:21] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:36 [I 2022-10-27 18:59:21] ...trap/engines/engine.py.229: process: 0.56988 | load: 0.00244 [I 2022-10-27 18:59:21] ...trap/engines/engine.py.230: loss: 0.14796 [I 2022-10-27 18:59:21] ...optimizers/trijoint.py.151: total_norm: 11.963697 [I 2022-10-27 18:59:27] ...trap/engines/engine.py.224: train: epoch 83 | batch 220/2304 [I 2022-10-27 18:59:27] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:21 [I 2022-10-27 18:59:27] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00273 [I 2022-10-27 18:59:27] ...trap/engines/engine.py.230: loss: 0.15152 [I 2022-10-27 18:59:27] ...optimizers/trijoint.py.151: total_norm: 9.156322 [I 2022-10-27 18:59:32] ...trap/engines/engine.py.224: train: epoch 83 | batch 230/2304 [I 2022-10-27 18:59:32] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:36 [I 2022-10-27 18:59:32] ...trap/engines/engine.py.229: process: 0.60805 | load: 0.00253 [I 2022-10-27 18:59:32] ...trap/engines/engine.py.230: loss: 0.09072 [I 2022-10-27 18:59:32] ...optimizers/trijoint.py.151: total_norm: 7.501293 [I 2022-10-27 18:59:38] ...trap/engines/engine.py.224: train: epoch 83 | batch 240/2304 [I 2022-10-27 18:59:38] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:12 [I 2022-10-27 18:59:38] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00259 [I 2022-10-27 18:59:38] ...trap/engines/engine.py.230: loss: 0.14872 [I 2022-10-27 18:59:38] ...optimizers/trijoint.py.151: total_norm: 17.369003 [I 2022-10-27 18:59:44] ...trap/engines/engine.py.224: train: epoch 83 | batch 250/2304 [I 2022-10-27 18:59:44] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:01 [I 2022-10-27 18:59:44] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00252 [I 2022-10-27 18:59:44] ...trap/engines/engine.py.230: loss: 0.18278 [I 2022-10-27 18:59:44] ...optimizers/trijoint.py.151: total_norm: 15.876892 [I 2022-10-27 18:59:49] ...trap/engines/engine.py.224: train: epoch 83 | batch 260/2304 [I 2022-10-27 18:59:49] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:59 [I 2022-10-27 18:59:49] ...trap/engines/engine.py.229: process: 0.54755 | load: 0.00261 [I 2022-10-27 18:59:49] ...trap/engines/engine.py.230: loss: 0.27146 [I 2022-10-27 18:59:49] ...optimizers/trijoint.py.151: total_norm: 17.772184 [I 2022-10-27 18:59:55] ...trap/engines/engine.py.224: train: epoch 83 | batch 270/2304 [I 2022-10-27 18:59:55] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:55 [I 2022-10-27 18:59:55] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00259 [I 2022-10-27 18:59:55] ...trap/engines/engine.py.230: loss: 0.11686 [I 2022-10-27 18:59:55] ...optimizers/trijoint.py.151: total_norm: 11.439934 [I 2022-10-27 19:00:00] ...trap/engines/engine.py.224: train: epoch 83 | batch 280/2304 [I 2022-10-27 19:00:00] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:45 [I 2022-10-27 19:00:00] ...trap/engines/engine.py.229: process: 0.54451 | load: 0.00275 [I 2022-10-27 19:00:00] ...trap/engines/engine.py.230: loss: 0.17439 [I 2022-10-27 19:00:00] ...optimizers/trijoint.py.151: total_norm: 20.728083 [I 2022-10-27 19:00:06] ...trap/engines/engine.py.224: train: epoch 83 | batch 290/2304 [I 2022-10-27 19:00:06] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:44 [I 2022-10-27 19:00:06] ...trap/engines/engine.py.229: process: 0.55217 | load: 0.00250 [I 2022-10-27 19:00:06] ...trap/engines/engine.py.230: loss: 0.17300 [I 2022-10-27 19:00:06] ...optimizers/trijoint.py.151: total_norm: 13.282161 [I 2022-10-27 19:00:12] ...trap/engines/engine.py.224: train: epoch 83 | batch 300/2304 [I 2022-10-27 19:00:12] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:48 [I 2022-10-27 19:00:12] ...trap/engines/engine.py.229: process: 0.56299 | load: 0.00255 [I 2022-10-27 19:00:12] ...trap/engines/engine.py.230: loss: 0.12345 [I 2022-10-27 19:00:12] ...optimizers/trijoint.py.151: total_norm: 8.207767 [I 2022-10-27 19:00:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 310/2304 [I 2022-10-27 19:00:17] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:35 [I 2022-10-27 19:00:17] ...trap/engines/engine.py.229: process: 0.56859 | load: 0.00271 [I 2022-10-27 19:00:17] ...trap/engines/engine.py.230: loss: 0.15005 [I 2022-10-27 19:00:17] ...optimizers/trijoint.py.151: total_norm: 20.010290 [I 2022-10-27 19:00:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 320/2304 [I 2022-10-27 19:00:23] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:33 [I 2022-10-27 19:00:23] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00261 [I 2022-10-27 19:00:23] ...trap/engines/engine.py.230: loss: 0.14825 [I 2022-10-27 19:00:23] ...optimizers/trijoint.py.151: total_norm: 29.330400 [I 2022-10-27 19:00:28] ...trap/engines/engine.py.224: train: epoch 83 | batch 330/2304 [I 2022-10-27 19:00:28] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:21 [I 2022-10-27 19:00:28] ...trap/engines/engine.py.229: process: 0.55121 | load: 0.00262 [I 2022-10-27 19:00:28] ...trap/engines/engine.py.230: loss: 0.19887 [I 2022-10-27 19:00:28] ...optimizers/trijoint.py.151: total_norm: 23.972986 [I 2022-10-27 19:00:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 340/2304 [I 2022-10-27 19:00:34] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:08 [I 2022-10-27 19:00:34] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00259 [I 2022-10-27 19:00:34] ...trap/engines/engine.py.230: loss: 0.09335 [I 2022-10-27 19:00:34] ...optimizers/trijoint.py.151: total_norm: 20.057476 [I 2022-10-27 19:00:40] ...trap/engines/engine.py.224: train: epoch 83 | batch 350/2304 [I 2022-10-27 19:00:40] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:02 [I 2022-10-27 19:00:40] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00247 [I 2022-10-27 19:00:40] ...trap/engines/engine.py.230: loss: 0.12701 [I 2022-10-27 19:00:40] ...optimizers/trijoint.py.151: total_norm: 15.800367 [I 2022-10-27 19:00:45] ...trap/engines/engine.py.224: train: epoch 83 | batch 360/2304 [I 2022-10-27 19:00:45] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:05 [I 2022-10-27 19:00:45] ...trap/engines/engine.py.229: process: 0.55512 | load: 0.00248 [I 2022-10-27 19:00:45] ...trap/engines/engine.py.230: loss: 0.09137 [I 2022-10-27 19:00:45] ...optimizers/trijoint.py.151: total_norm: 11.928894 [I 2022-10-27 19:00:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 370/2304 [I 2022-10-27 19:00:51] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:17:56 [I 2022-10-27 19:00:51] ...trap/engines/engine.py.229: process: 0.54432 | load: 0.00250 [I 2022-10-27 19:00:51] ...trap/engines/engine.py.230: loss: 0.13123 [I 2022-10-27 19:00:51] ...optimizers/trijoint.py.151: total_norm: 20.519535 [I 2022-10-27 19:00:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 380/2304 [I 2022-10-27 19:00:56] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:56 [I 2022-10-27 19:00:56] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00245 [I 2022-10-27 19:00:56] ...trap/engines/engine.py.230: loss: 0.14916 [I 2022-10-27 19:00:56] ...optimizers/trijoint.py.151: total_norm: 29.009134 [I 2022-10-27 19:01:02] ...trap/engines/engine.py.224: train: epoch 83 | batch 390/2304 [I 2022-10-27 19:01:02] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:37 [I 2022-10-27 19:01:02] ...trap/engines/engine.py.229: process: 0.55376 | load: 0.00243 [I 2022-10-27 19:01:02] ...trap/engines/engine.py.230: loss: 0.15019 [I 2022-10-27 19:01:02] ...optimizers/trijoint.py.151: total_norm: 85.488785 [I 2022-10-27 19:01:07] ...trap/engines/engine.py.224: train: epoch 83 | batch 400/2304 [I 2022-10-27 19:01:07] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:30 [I 2022-10-27 19:01:07] ...trap/engines/engine.py.229: process: 0.54313 | load: 0.00242 [I 2022-10-27 19:01:07] ...trap/engines/engine.py.230: loss: 0.12192 [I 2022-10-27 19:01:07] ...optimizers/trijoint.py.151: total_norm: 14.698772 [I 2022-10-27 19:01:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 410/2304 [I 2022-10-27 19:01:13] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:21 [I 2022-10-27 19:01:13] ...trap/engines/engine.py.229: process: 0.54712 | load: 0.00244 [I 2022-10-27 19:01:13] ...trap/engines/engine.py.230: loss: 0.09101 [I 2022-10-27 19:01:13] ...optimizers/trijoint.py.151: total_norm: 13.861827 [I 2022-10-27 19:01:19] ...trap/engines/engine.py.224: train: epoch 83 | batch 420/2304 [I 2022-10-27 19:01:19] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:59 [I 2022-10-27 19:01:19] ...trap/engines/engine.py.229: process: 0.54940 | load: 0.00241 [I 2022-10-27 19:01:19] ...trap/engines/engine.py.230: loss: 0.10387 [I 2022-10-27 19:01:19] ...optimizers/trijoint.py.151: total_norm: 13.599552 [I 2022-10-27 19:01:24] ...trap/engines/engine.py.224: train: epoch 83 | batch 430/2304 [I 2022-10-27 19:01:24] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:28 [I 2022-10-27 19:01:24] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00243 [I 2022-10-27 19:01:24] ...trap/engines/engine.py.230: loss: 0.11497 [I 2022-10-27 19:01:24] ...optimizers/trijoint.py.151: total_norm: 9.388685 [I 2022-10-27 19:01:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 440/2304 [I 2022-10-27 19:01:30] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:16 [I 2022-10-27 19:01:30] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00241 [I 2022-10-27 19:01:30] ...trap/engines/engine.py.230: loss: 0.11112 [I 2022-10-27 19:01:30] ...optimizers/trijoint.py.151: total_norm: 19.529455 [I 2022-10-27 19:01:35] ...trap/engines/engine.py.224: train: epoch 83 | batch 450/2304 [I 2022-10-27 19:01:35] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:09 [I 2022-10-27 19:01:35] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00250 [I 2022-10-27 19:01:35] ...trap/engines/engine.py.230: loss: 0.10218 [I 2022-10-27 19:01:35] ...optimizers/trijoint.py.151: total_norm: 14.487078 [I 2022-10-27 19:01:41] ...trap/engines/engine.py.224: train: epoch 83 | batch 460/2304 [I 2022-10-27 19:01:41] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:08 [I 2022-10-27 19:01:41] ...trap/engines/engine.py.229: process: 0.57376 | load: 0.00240 [I 2022-10-27 19:01:41] ...trap/engines/engine.py.230: loss: 0.10379 [I 2022-10-27 19:01:41] ...optimizers/trijoint.py.151: total_norm: 9.436569 [I 2022-10-27 19:01:46] ...trap/engines/engine.py.224: train: epoch 83 | batch 470/2304 [I 2022-10-27 19:01:46] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:55 [I 2022-10-27 19:01:46] ...trap/engines/engine.py.229: process: 0.54780 | load: 0.00252 [I 2022-10-27 19:01:46] ...trap/engines/engine.py.230: loss: 0.14218 [I 2022-10-27 19:01:46] ...optimizers/trijoint.py.151: total_norm: 17.349792 [I 2022-10-27 19:01:52] ...trap/engines/engine.py.224: train: epoch 83 | batch 480/2304 [I 2022-10-27 19:01:52] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:49 [I 2022-10-27 19:01:52] ...trap/engines/engine.py.229: process: 0.54782 | load: 0.00280 [I 2022-10-27 19:01:52] ...trap/engines/engine.py.230: loss: 0.10653 [I 2022-10-27 19:01:52] ...optimizers/trijoint.py.151: total_norm: 32.034752 [I 2022-10-27 19:01:58] ...trap/engines/engine.py.224: train: epoch 83 | batch 490/2304 [I 2022-10-27 19:01:58] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:54 [I 2022-10-27 19:01:58] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00247 [I 2022-10-27 19:01:58] ...trap/engines/engine.py.230: loss: 0.15088 [I 2022-10-27 19:01:58] ...optimizers/trijoint.py.151: total_norm: 12.667647 [I 2022-10-27 19:02:03] ...trap/engines/engine.py.224: train: epoch 83 | batch 500/2304 [I 2022-10-27 19:02:03] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:44 [I 2022-10-27 19:02:03] ...trap/engines/engine.py.229: process: 0.54849 | load: 0.00247 [I 2022-10-27 19:02:03] ...trap/engines/engine.py.230: loss: 0.20018 [I 2022-10-27 19:02:03] ...optimizers/trijoint.py.151: total_norm: 27.128283 [I 2022-10-27 19:02:09] ...trap/engines/engine.py.224: train: epoch 83 | batch 510/2304 [I 2022-10-27 19:02:09] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:41 [I 2022-10-27 19:02:09] ...trap/engines/engine.py.229: process: 0.56931 | load: 0.00242 [I 2022-10-27 19:02:09] ...trap/engines/engine.py.230: loss: 0.14128 [I 2022-10-27 19:02:09] ...optimizers/trijoint.py.151: total_norm: 15.033454 [I 2022-10-27 19:02:14] ...trap/engines/engine.py.224: train: epoch 83 | batch 520/2304 [I 2022-10-27 19:02:14] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:31 [I 2022-10-27 19:02:14] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00243 [I 2022-10-27 19:02:14] ...trap/engines/engine.py.230: loss: 0.16214 [I 2022-10-27 19:02:14] ...optimizers/trijoint.py.151: total_norm: 55.436085 [I 2022-10-27 19:02:20] ...trap/engines/engine.py.224: train: epoch 83 | batch 530/2304 [I 2022-10-27 19:02:20] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:31 [I 2022-10-27 19:02:20] ...trap/engines/engine.py.229: process: 0.56761 | load: 0.00254 [I 2022-10-27 19:02:20] ...trap/engines/engine.py.230: loss: 0.10929 [I 2022-10-27 19:02:20] ...optimizers/trijoint.py.151: total_norm: 17.640015 [I 2022-10-27 19:02:26] ...trap/engines/engine.py.224: train: epoch 83 | batch 540/2304 [I 2022-10-27 19:02:26] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:34 [I 2022-10-27 19:02:26] ...trap/engines/engine.py.229: process: 0.55126 | load: 0.00252 [I 2022-10-27 19:02:26] ...trap/engines/engine.py.230: loss: 0.16677 [I 2022-10-27 19:02:26] ...optimizers/trijoint.py.151: total_norm: 13.487401 [I 2022-10-27 19:02:31] ...trap/engines/engine.py.224: train: epoch 83 | batch 550/2304 [I 2022-10-27 19:02:31] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:25 [I 2022-10-27 19:02:31] ...trap/engines/engine.py.229: process: 0.56193 | load: 0.00248 [I 2022-10-27 19:02:31] ...trap/engines/engine.py.230: loss: 0.15641 [I 2022-10-27 19:02:31] ...optimizers/trijoint.py.151: total_norm: 21.052137 [I 2022-10-27 19:02:37] ...trap/engines/engine.py.224: train: epoch 83 | batch 560/2304 [I 2022-10-27 19:02:37] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:09 [I 2022-10-27 19:02:37] ...trap/engines/engine.py.229: process: 0.54702 | load: 0.00250 [I 2022-10-27 19:02:37] ...trap/engines/engine.py.230: loss: 0.17936 [I 2022-10-27 19:02:37] ...optimizers/trijoint.py.151: total_norm: 20.453672 [I 2022-10-27 19:02:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 570/2304 [I 2022-10-27 19:02:42] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:07 [I 2022-10-27 19:02:42] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00242 [I 2022-10-27 19:02:42] ...trap/engines/engine.py.230: loss: 0.16601 [I 2022-10-27 19:02:42] ...optimizers/trijoint.py.151: total_norm: 12.185360 [I 2022-10-27 19:02:48] ...trap/engines/engine.py.224: train: epoch 83 | batch 580/2304 [I 2022-10-27 19:02:48] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:59 [I 2022-10-27 19:02:48] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00245 [I 2022-10-27 19:02:48] ...trap/engines/engine.py.230: loss: 0.10403 [I 2022-10-27 19:02:48] ...optimizers/trijoint.py.151: total_norm: 13.047598 [I 2022-10-27 19:02:54] ...trap/engines/engine.py.224: train: epoch 83 | batch 590/2304 [I 2022-10-27 19:02:54] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:55 [I 2022-10-27 19:02:54] ...trap/engines/engine.py.229: process: 0.56459 | load: 0.00240 [I 2022-10-27 19:02:54] ...trap/engines/engine.py.230: loss: 0.12057 [I 2022-10-27 19:02:54] ...optimizers/trijoint.py.151: total_norm: 46.414749 [I 2022-10-27 19:02:59] ...trap/engines/engine.py.224: train: epoch 83 | batch 600/2304 [I 2022-10-27 19:02:59] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:50 [I 2022-10-27 19:02:59] ...trap/engines/engine.py.229: process: 0.56433 | load: 0.00248 [I 2022-10-27 19:02:59] ...trap/engines/engine.py.230: loss: 0.10066 [I 2022-10-27 19:02:59] ...optimizers/trijoint.py.151: total_norm: 27.600470 [I 2022-10-27 19:03:05] ...trap/engines/engine.py.224: train: epoch 83 | batch 610/2304 [I 2022-10-27 19:03:05] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:43 [I 2022-10-27 19:03:05] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00265 [I 2022-10-27 19:03:05] ...trap/engines/engine.py.230: loss: 0.11177 [I 2022-10-27 19:03:05] ...optimizers/trijoint.py.151: total_norm: 10.594502 [I 2022-10-27 19:03:10] ...trap/engines/engine.py.224: train: epoch 83 | batch 620/2304 [I 2022-10-27 19:03:10] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:33 [I 2022-10-27 19:03:10] ...trap/engines/engine.py.229: process: 0.54304 | load: 0.00243 [I 2022-10-27 19:03:10] ...trap/engines/engine.py.230: loss: 0.09552 [I 2022-10-27 19:03:10] ...optimizers/trijoint.py.151: total_norm: 7.906368 [I 2022-10-27 19:03:16] ...trap/engines/engine.py.224: train: epoch 83 | batch 630/2304 [I 2022-10-27 19:03:16] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:30 [I 2022-10-27 19:03:16] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00254 [I 2022-10-27 19:03:16] ...trap/engines/engine.py.230: loss: 0.17590 [I 2022-10-27 19:03:16] ...optimizers/trijoint.py.151: total_norm: 33.559097 [I 2022-10-27 19:03:21] ...trap/engines/engine.py.224: train: epoch 83 | batch 640/2304 [I 2022-10-27 19:03:21] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:27 [I 2022-10-27 19:03:21] ...trap/engines/engine.py.229: process: 0.55873 | load: 0.00248 [I 2022-10-27 19:03:21] ...trap/engines/engine.py.230: loss: 0.15666 [I 2022-10-27 19:03:21] ...optimizers/trijoint.py.151: total_norm: 27.068399 [I 2022-10-27 19:03:27] ...trap/engines/engine.py.224: train: epoch 83 | batch 650/2304 [I 2022-10-27 19:03:27] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:23 [I 2022-10-27 19:03:27] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00245 [I 2022-10-27 19:03:27] ...trap/engines/engine.py.230: loss: 0.12854 [I 2022-10-27 19:03:27] ...optimizers/trijoint.py.151: total_norm: 11.426013 [I 2022-10-27 19:03:33] ...trap/engines/engine.py.224: train: epoch 83 | batch 660/2304 [I 2022-10-27 19:03:33] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:15 [I 2022-10-27 19:03:33] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00245 [I 2022-10-27 19:03:33] ...trap/engines/engine.py.230: loss: 0.10023 [I 2022-10-27 19:03:33] ...optimizers/trijoint.py.151: total_norm: 16.243893 [I 2022-10-27 19:03:38] ...trap/engines/engine.py.224: train: epoch 83 | batch 670/2304 [I 2022-10-27 19:03:38] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:08 [I 2022-10-27 19:03:38] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00243 [I 2022-10-27 19:03:38] ...trap/engines/engine.py.230: loss: 0.18359 [I 2022-10-27 19:03:38] ...optimizers/trijoint.py.151: total_norm: 19.042671 [I 2022-10-27 19:03:44] ...trap/engines/engine.py.224: train: epoch 83 | batch 680/2304 [I 2022-10-27 19:03:44] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:14:57 [I 2022-10-27 19:03:44] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00244 [I 2022-10-27 19:03:44] ...trap/engines/engine.py.230: loss: 0.11580 [I 2022-10-27 19:03:44] ...optimizers/trijoint.py.151: total_norm: 12.630244 [I 2022-10-27 19:03:49] ...trap/engines/engine.py.224: train: epoch 83 | batch 690/2304 [I 2022-10-27 19:03:49] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:01 [I 2022-10-27 19:03:49] ...trap/engines/engine.py.229: process: 0.55257 | load: 0.00250 [I 2022-10-27 19:03:49] ...trap/engines/engine.py.230: loss: 0.09036 [I 2022-10-27 19:03:49] ...optimizers/trijoint.py.151: total_norm: 14.406511 [I 2022-10-27 19:03:55] ...trap/engines/engine.py.224: train: epoch 83 | batch 700/2304 [I 2022-10-27 19:03:55] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:54 [I 2022-10-27 19:03:55] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00250 [I 2022-10-27 19:03:55] ...trap/engines/engine.py.230: loss: 0.13324 [I 2022-10-27 19:03:55] ...optimizers/trijoint.py.151: total_norm: 25.924023 [I 2022-10-27 19:04:00] ...trap/engines/engine.py.224: train: epoch 83 | batch 710/2304 [I 2022-10-27 19:04:00] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:49 [I 2022-10-27 19:04:00] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00244 [I 2022-10-27 19:04:00] ...trap/engines/engine.py.230: loss: 0.11239 [I 2022-10-27 19:04:00] ...optimizers/trijoint.py.151: total_norm: 11.451632 [I 2022-10-27 19:04:06] ...trap/engines/engine.py.224: train: epoch 83 | batch 720/2304 [I 2022-10-27 19:04:06] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:43 [I 2022-10-27 19:04:06] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00245 [I 2022-10-27 19:04:06] ...trap/engines/engine.py.230: loss: 0.20404 [I 2022-10-27 19:04:06] ...optimizers/trijoint.py.151: total_norm: 68.068909 [I 2022-10-27 19:04:12] ...trap/engines/engine.py.224: train: epoch 83 | batch 730/2304 [I 2022-10-27 19:04:12] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:31 [I 2022-10-27 19:04:12] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00246 [I 2022-10-27 19:04:12] ...trap/engines/engine.py.230: loss: 0.22723 [I 2022-10-27 19:04:12] ...optimizers/trijoint.py.151: total_norm: 19.255760 [I 2022-10-27 19:04:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 740/2304 [I 2022-10-27 19:04:17] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:29 [I 2022-10-27 19:04:17] ...trap/engines/engine.py.229: process: 0.56644 | load: 0.00242 [I 2022-10-27 19:04:17] ...trap/engines/engine.py.230: loss: 0.10610 [I 2022-10-27 19:04:17] ...optimizers/trijoint.py.151: total_norm: 8.697838 [I 2022-10-27 19:04:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 750/2304 [I 2022-10-27 19:04:23] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:25 [I 2022-10-27 19:04:23] ...trap/engines/engine.py.229: process: 0.55694 | load: 0.00244 [I 2022-10-27 19:04:23] ...trap/engines/engine.py.230: loss: 0.09244 [I 2022-10-27 19:04:23] ...optimizers/trijoint.py.151: total_norm: 8.919939 [I 2022-10-27 19:04:28] ...trap/engines/engine.py.224: train: epoch 83 | batch 760/2304 [I 2022-10-27 19:04:28] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:21 [I 2022-10-27 19:04:28] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00244 [I 2022-10-27 19:04:28] ...trap/engines/engine.py.230: loss: 0.10933 [I 2022-10-27 19:04:28] ...optimizers/trijoint.py.151: total_norm: 14.335024 [I 2022-10-27 19:04:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 770/2304 [I 2022-10-27 19:04:34] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:12 [I 2022-10-27 19:04:34] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00245 [I 2022-10-27 19:04:34] ...trap/engines/engine.py.230: loss: 0.12785 [I 2022-10-27 19:04:34] ...optimizers/trijoint.py.151: total_norm: 16.235128 [I 2022-10-27 19:04:40] ...trap/engines/engine.py.224: train: epoch 83 | batch 780/2304 [I 2022-10-27 19:04:40] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:10 [I 2022-10-27 19:04:40] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00241 [I 2022-10-27 19:04:40] ...trap/engines/engine.py.230: loss: 0.12526 [I 2022-10-27 19:04:40] ...optimizers/trijoint.py.151: total_norm: 15.450394 [I 2022-10-27 19:04:45] ...trap/engines/engine.py.224: train: epoch 83 | batch 790/2304 [I 2022-10-27 19:04:45] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:07 [I 2022-10-27 19:04:45] ...trap/engines/engine.py.229: process: 0.57167 | load: 0.00246 [I 2022-10-27 19:04:45] ...trap/engines/engine.py.230: loss: 0.11157 [I 2022-10-27 19:04:45] ...optimizers/trijoint.py.151: total_norm: 14.671091 [I 2022-10-27 19:04:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 800/2304 [I 2022-10-27 19:04:51] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:59 [I 2022-10-27 19:04:51] ...trap/engines/engine.py.229: process: 0.55494 | load: 0.00249 [I 2022-10-27 19:04:51] ...trap/engines/engine.py.230: loss: 0.14475 [I 2022-10-27 19:04:51] ...optimizers/trijoint.py.151: total_norm: 17.380184 [I 2022-10-27 19:04:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 810/2304 [I 2022-10-27 19:04:56] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:49 [I 2022-10-27 19:04:56] ...trap/engines/engine.py.229: process: 0.55129 | load: 0.00247 [I 2022-10-27 19:04:56] ...trap/engines/engine.py.230: loss: 0.15596 [I 2022-10-27 19:04:56] ...optimizers/trijoint.py.151: total_norm: 25.263840 [I 2022-10-27 19:05:02] ...trap/engines/engine.py.224: train: epoch 83 | batch 820/2304 [I 2022-10-27 19:05:02] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:50 [I 2022-10-27 19:05:02] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00243 [I 2022-10-27 19:05:02] ...trap/engines/engine.py.230: loss: 0.17220 [I 2022-10-27 19:05:02] ...optimizers/trijoint.py.151: total_norm: 12.126275 [I 2022-10-27 19:05:07] ...trap/engines/engine.py.224: train: epoch 83 | batch 830/2304 [I 2022-10-27 19:05:07] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:42 [I 2022-10-27 19:05:07] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00243 [I 2022-10-27 19:05:07] ...trap/engines/engine.py.230: loss: 0.09758 [I 2022-10-27 19:05:07] ...optimizers/trijoint.py.151: total_norm: 8.360696 [I 2022-10-27 19:05:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 840/2304 [I 2022-10-27 19:05:13] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:31 [I 2022-10-27 19:05:13] ...trap/engines/engine.py.229: process: 0.55211 | load: 0.00243 [I 2022-10-27 19:05:13] ...trap/engines/engine.py.230: loss: 0.13208 [I 2022-10-27 19:05:13] ...optimizers/trijoint.py.151: total_norm: 14.208482 [I 2022-10-27 19:05:19] ...trap/engines/engine.py.224: train: epoch 83 | batch 850/2304 [I 2022-10-27 19:05:19] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:29 [I 2022-10-27 19:05:19] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00244 [I 2022-10-27 19:05:19] ...trap/engines/engine.py.230: loss: 0.12742 [I 2022-10-27 19:05:19] ...optimizers/trijoint.py.151: total_norm: 12.867908 [I 2022-10-27 19:05:24] ...trap/engines/engine.py.224: train: epoch 83 | batch 860/2304 [I 2022-10-27 19:05:24] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:26 [I 2022-10-27 19:05:24] ...trap/engines/engine.py.229: process: 0.56126 | load: 0.00243 [I 2022-10-27 19:05:24] ...trap/engines/engine.py.230: loss: 0.10628 [I 2022-10-27 19:05:24] ...optimizers/trijoint.py.151: total_norm: 13.948133 [I 2022-10-27 19:05:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 870/2304 [I 2022-10-27 19:05:30] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:18 [I 2022-10-27 19:05:30] ...trap/engines/engine.py.229: process: 0.56729 | load: 0.00243 [I 2022-10-27 19:05:30] ...trap/engines/engine.py.230: loss: 0.12549 [I 2022-10-27 19:05:30] ...optimizers/trijoint.py.151: total_norm: 19.085878 [I 2022-10-27 19:05:35] ...trap/engines/engine.py.224: train: epoch 83 | batch 880/2304 [I 2022-10-27 19:05:35] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:14 [I 2022-10-27 19:05:35] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00248 [I 2022-10-27 19:05:35] ...trap/engines/engine.py.230: loss: 0.21578 [I 2022-10-27 19:05:35] ...optimizers/trijoint.py.151: total_norm: 22.571259 [I 2022-10-27 19:05:41] ...trap/engines/engine.py.224: train: epoch 83 | batch 890/2304 [I 2022-10-27 19:05:41] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:14 [I 2022-10-27 19:05:41] ...trap/engines/engine.py.229: process: 0.57388 | load: 0.00270 [I 2022-10-27 19:05:41] ...trap/engines/engine.py.230: loss: 0.10407 [I 2022-10-27 19:05:41] ...optimizers/trijoint.py.151: total_norm: 10.416232 [I 2022-10-27 19:05:47] ...trap/engines/engine.py.224: train: epoch 83 | batch 900/2304 [I 2022-10-27 19:05:47] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:04 [I 2022-10-27 19:05:47] ...trap/engines/engine.py.229: process: 0.57138 | load: 0.00249 [I 2022-10-27 19:05:47] ...trap/engines/engine.py.230: loss: 0.12968 [I 2022-10-27 19:05:47] ...optimizers/trijoint.py.151: total_norm: 8.119530 [I 2022-10-27 19:05:52] ...trap/engines/engine.py.224: train: epoch 83 | batch 910/2304 [I 2022-10-27 19:05:52] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:59 [I 2022-10-27 19:05:52] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00248 [I 2022-10-27 19:05:52] ...trap/engines/engine.py.230: loss: 0.12443 [I 2022-10-27 19:05:52] ...optimizers/trijoint.py.151: total_norm: 15.644959 [I 2022-10-27 19:05:58] ...trap/engines/engine.py.224: train: epoch 83 | batch 920/2304 [I 2022-10-27 19:05:58] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:51 [I 2022-10-27 19:05:58] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00245 [I 2022-10-27 19:05:58] ...trap/engines/engine.py.230: loss: 0.11222 [I 2022-10-27 19:05:58] ...optimizers/trijoint.py.151: total_norm: 13.835147 [I 2022-10-27 19:06:03] ...trap/engines/engine.py.224: train: epoch 83 | batch 930/2304 [I 2022-10-27 19:06:03] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:36 [I 2022-10-27 19:06:03] ...trap/engines/engine.py.229: process: 0.55521 | load: 0.00246 [I 2022-10-27 19:06:03] ...trap/engines/engine.py.230: loss: 0.15050 [I 2022-10-27 19:06:03] ...optimizers/trijoint.py.151: total_norm: 15.010302 [I 2022-10-27 19:06:09] ...trap/engines/engine.py.224: train: epoch 83 | batch 940/2304 [I 2022-10-27 19:06:09] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:34 [I 2022-10-27 19:06:09] ...trap/engines/engine.py.229: process: 0.54880 | load: 0.00244 [I 2022-10-27 19:06:09] ...trap/engines/engine.py.230: loss: 0.16120 [I 2022-10-27 19:06:09] ...optimizers/trijoint.py.151: total_norm: 19.665749 [I 2022-10-27 19:06:14] ...trap/engines/engine.py.224: train: epoch 83 | batch 950/2304 [I 2022-10-27 19:06:14] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:34 [I 2022-10-27 19:06:14] ...trap/engines/engine.py.229: process: 0.58646 | load: 0.00244 [I 2022-10-27 19:06:14] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-27 19:06:14] ...optimizers/trijoint.py.151: total_norm: 21.106224 [I 2022-10-27 19:06:20] ...trap/engines/engine.py.224: train: epoch 83 | batch 960/2304 [I 2022-10-27 19:06:20] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:29 [I 2022-10-27 19:06:20] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00249 [I 2022-10-27 19:06:20] ...trap/engines/engine.py.230: loss: 0.15230 [I 2022-10-27 19:06:20] ...optimizers/trijoint.py.151: total_norm: 13.981491 [I 2022-10-27 19:06:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 970/2304 [I 2022-10-27 19:06:25] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:18 [I 2022-10-27 19:06:25] ...trap/engines/engine.py.229: process: 0.54320 | load: 0.00243 [I 2022-10-27 19:06:25] ...trap/engines/engine.py.230: loss: 0.13380 [I 2022-10-27 19:06:25] ...optimizers/trijoint.py.151: total_norm: 24.635479 [I 2022-10-27 19:06:31] ...trap/engines/engine.py.224: train: epoch 83 | batch 980/2304 [I 2022-10-27 19:06:31] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:38 [I 2022-10-27 19:06:31] ...trap/engines/engine.py.229: process: 0.59553 | load: 0.00242 [I 2022-10-27 19:06:31] ...trap/engines/engine.py.230: loss: 0.11547 [I 2022-10-27 19:06:31] ...optimizers/trijoint.py.151: total_norm: 21.342152 [I 2022-10-27 19:06:37] ...trap/engines/engine.py.224: train: epoch 83 | batch 990/2304 [I 2022-10-27 19:06:37] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:16 [I 2022-10-27 19:06:37] ...trap/engines/engine.py.229: process: 0.56434 | load: 0.00249 [I 2022-10-27 19:06:37] ...trap/engines/engine.py.230: loss: 0.08405 [I 2022-10-27 19:06:37] ...optimizers/trijoint.py.151: total_norm: 9.689545 [I 2022-10-27 19:06:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 1000/2304 [I 2022-10-27 19:06:42] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:32 [I 2022-10-27 19:06:42] ...trap/engines/engine.py.229: process: 0.54529 | load: 0.00243 [I 2022-10-27 19:06:42] ...trap/engines/engine.py.230: loss: 0.13050 [I 2022-10-27 19:06:42] ...optimizers/trijoint.py.151: total_norm: 14.487816 [I 2022-10-27 19:06:48] ...trap/engines/engine.py.224: train: epoch 83 | batch 1010/2304 [I 2022-10-27 19:06:48] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:05 [I 2022-10-27 19:06:48] ...trap/engines/engine.py.229: process: 0.55156 | load: 0.00252 [I 2022-10-27 19:06:48] ...trap/engines/engine.py.230: loss: 0.12269 [I 2022-10-27 19:06:48] ...optimizers/trijoint.py.151: total_norm: 15.132872 [I 2022-10-27 19:06:54] ...trap/engines/engine.py.224: train: epoch 83 | batch 1020/2304 [I 2022-10-27 19:06:54] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:57 [I 2022-10-27 19:06:54] ...trap/engines/engine.py.229: process: 0.56386 | load: 0.00249 [I 2022-10-27 19:06:54] ...trap/engines/engine.py.230: loss: 0.14506 [I 2022-10-27 19:06:54] ...optimizers/trijoint.py.151: total_norm: 22.145489 [I 2022-10-27 19:06:59] ...trap/engines/engine.py.224: train: epoch 83 | batch 1030/2304 [I 2022-10-27 19:06:59] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:49 [I 2022-10-27 19:06:59] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00246 [I 2022-10-27 19:06:59] ...trap/engines/engine.py.230: loss: 0.15779 [I 2022-10-27 19:06:59] ...optimizers/trijoint.py.151: total_norm: 16.989594 [I 2022-10-27 19:07:05] ...trap/engines/engine.py.224: train: epoch 83 | batch 1040/2304 [I 2022-10-27 19:07:05] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:43 [I 2022-10-27 19:07:05] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00242 [I 2022-10-27 19:07:05] ...trap/engines/engine.py.230: loss: 0.10926 [I 2022-10-27 19:07:05] ...optimizers/trijoint.py.151: total_norm: 14.516057 [I 2022-10-27 19:07:10] ...trap/engines/engine.py.224: train: epoch 83 | batch 1050/2304 [I 2022-10-27 19:07:10] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:39 [I 2022-10-27 19:07:10] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00243 [I 2022-10-27 19:07:10] ...trap/engines/engine.py.230: loss: 0.08708 [I 2022-10-27 19:07:10] ...optimizers/trijoint.py.151: total_norm: 14.603324 [I 2022-10-27 19:07:16] ...trap/engines/engine.py.224: train: epoch 83 | batch 1060/2304 [I 2022-10-27 19:07:16] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:32 [I 2022-10-27 19:07:16] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00243 [I 2022-10-27 19:07:16] ...trap/engines/engine.py.230: loss: 0.16862 [I 2022-10-27 19:07:16] ...optimizers/trijoint.py.151: total_norm: 36.203350 [I 2022-10-27 19:07:22] ...trap/engines/engine.py.224: train: epoch 83 | batch 1070/2304 [I 2022-10-27 19:07:22] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:49 [I 2022-10-27 19:07:22] ...trap/engines/engine.py.229: process: 0.56653 | load: 0.00241 [I 2022-10-27 19:07:22] ...trap/engines/engine.py.230: loss: 0.09384 [I 2022-10-27 19:07:22] ...optimizers/trijoint.py.151: total_norm: 13.905842 [I 2022-10-27 19:07:27] ...trap/engines/engine.py.224: train: epoch 83 | batch 1080/2304 [I 2022-10-27 19:07:27] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:27 [I 2022-10-27 19:07:27] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00251 [I 2022-10-27 19:07:27] ...trap/engines/engine.py.230: loss: 0.24044 [I 2022-10-27 19:07:27] ...optimizers/trijoint.py.151: total_norm: 57.539066 [I 2022-10-27 19:07:33] ...trap/engines/engine.py.224: train: epoch 83 | batch 1090/2304 [I 2022-10-27 19:07:33] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:18 [I 2022-10-27 19:07:33] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00246 [I 2022-10-27 19:07:33] ...trap/engines/engine.py.230: loss: 0.13231 [I 2022-10-27 19:07:33] ...optimizers/trijoint.py.151: total_norm: 25.438364 [I 2022-10-27 19:07:38] ...trap/engines/engine.py.224: train: epoch 83 | batch 1100/2304 [I 2022-10-27 19:07:38] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:07 [I 2022-10-27 19:07:38] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00250 [I 2022-10-27 19:07:38] ...trap/engines/engine.py.230: loss: 0.13035 [I 2022-10-27 19:07:38] ...optimizers/trijoint.py.151: total_norm: 14.422269 [I 2022-10-27 19:07:44] ...trap/engines/engine.py.224: train: epoch 83 | batch 1110/2304 [I 2022-10-27 19:07:44] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:06 [I 2022-10-27 19:07:44] ...trap/engines/engine.py.229: process: 0.54897 | load: 0.00240 [I 2022-10-27 19:07:44] ...trap/engines/engine.py.230: loss: 0.10552 [I 2022-10-27 19:07:44] ...optimizers/trijoint.py.151: total_norm: 7.549413 [I 2022-10-27 19:07:50] ...trap/engines/engine.py.224: train: epoch 83 | batch 1120/2304 [I 2022-10-27 19:07:50] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:10:55 [I 2022-10-27 19:07:50] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00243 [I 2022-10-27 19:07:50] ...trap/engines/engine.py.230: loss: 0.12396 [I 2022-10-27 19:07:50] ...optimizers/trijoint.py.151: total_norm: 8.741469 [I 2022-10-27 19:07:55] ...trap/engines/engine.py.224: train: epoch 83 | batch 1130/2304 [I 2022-10-27 19:07:55] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:54 [I 2022-10-27 19:07:55] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00243 [I 2022-10-27 19:07:55] ...trap/engines/engine.py.230: loss: 0.10760 [I 2022-10-27 19:07:55] ...optimizers/trijoint.py.151: total_norm: 8.171463 [I 2022-10-27 19:08:01] ...trap/engines/engine.py.224: train: epoch 83 | batch 1140/2304 [I 2022-10-27 19:08:01] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:54 [I 2022-10-27 19:08:01] ...trap/engines/engine.py.229: process: 0.56155 | load: 0.00242 [I 2022-10-27 19:08:01] ...trap/engines/engine.py.230: loss: 0.18102 [I 2022-10-27 19:08:01] ...optimizers/trijoint.py.151: total_norm: 33.715160 [I 2022-10-27 19:08:06] ...trap/engines/engine.py.224: train: epoch 83 | batch 1150/2304 [I 2022-10-27 19:08:06] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:42 [I 2022-10-27 19:08:06] ...trap/engines/engine.py.229: process: 0.55676 | load: 0.00244 [I 2022-10-27 19:08:06] ...trap/engines/engine.py.230: loss: 0.14765 [I 2022-10-27 19:08:06] ...optimizers/trijoint.py.151: total_norm: 11.410216 [I 2022-10-27 19:08:12] ...trap/engines/engine.py.224: train: epoch 83 | batch 1160/2304 [I 2022-10-27 19:08:12] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:34 [I 2022-10-27 19:08:12] ...trap/engines/engine.py.229: process: 0.54838 | load: 0.00244 [I 2022-10-27 19:08:12] ...trap/engines/engine.py.230: loss: 0.15476 [I 2022-10-27 19:08:12] ...optimizers/trijoint.py.151: total_norm: 12.472355 [I 2022-10-27 19:08:18] ...trap/engines/engine.py.224: train: epoch 83 | batch 1170/2304 [I 2022-10-27 19:08:18] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:33 [I 2022-10-27 19:08:18] ...trap/engines/engine.py.229: process: 0.56002 | load: 0.00241 [I 2022-10-27 19:08:18] ...trap/engines/engine.py.230: loss: 0.09020 [I 2022-10-27 19:08:18] ...optimizers/trijoint.py.151: total_norm: 15.062384 [I 2022-10-27 19:08:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 1180/2304 [I 2022-10-27 19:08:23] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:20 [I 2022-10-27 19:08:23] ...trap/engines/engine.py.229: process: 0.54872 | load: 0.00248 [I 2022-10-27 19:08:23] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-27 19:08:23] ...optimizers/trijoint.py.151: total_norm: 13.323582 [I 2022-10-27 19:08:29] ...trap/engines/engine.py.224: train: epoch 83 | batch 1190/2304 [I 2022-10-27 19:08:29] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:19 [I 2022-10-27 19:08:29] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00245 [I 2022-10-27 19:08:29] ...trap/engines/engine.py.230: loss: 0.11538 [I 2022-10-27 19:08:29] ...optimizers/trijoint.py.151: total_norm: 13.540841 [I 2022-10-27 19:08:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 1200/2304 [I 2022-10-27 19:08:34] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:16 [I 2022-10-27 19:08:34] ...trap/engines/engine.py.229: process: 0.54869 | load: 0.00241 [I 2022-10-27 19:08:34] ...trap/engines/engine.py.230: loss: 0.10689 [I 2022-10-27 19:08:34] ...optimizers/trijoint.py.151: total_norm: 15.894105 [I 2022-10-27 19:08:40] ...trap/engines/engine.py.224: train: epoch 83 | batch 1210/2304 [I 2022-10-27 19:08:40] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:08 [I 2022-10-27 19:08:40] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00251 [I 2022-10-27 19:08:40] ...trap/engines/engine.py.230: loss: 0.10036 [I 2022-10-27 19:08:40] ...optimizers/trijoint.py.151: total_norm: 13.323029 [I 2022-10-27 19:08:45] ...trap/engines/engine.py.224: train: epoch 83 | batch 1220/2304 [I 2022-10-27 19:08:45] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:05 [I 2022-10-27 19:08:45] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00245 [I 2022-10-27 19:08:45] ...trap/engines/engine.py.230: loss: 0.11461 [I 2022-10-27 19:08:45] ...optimizers/trijoint.py.151: total_norm: 9.846021 [I 2022-10-27 19:08:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 1230/2304 [I 2022-10-27 19:08:51] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:53 [I 2022-10-27 19:08:51] ...trap/engines/engine.py.229: process: 0.54929 | load: 0.00246 [I 2022-10-27 19:08:51] ...trap/engines/engine.py.230: loss: 0.12009 [I 2022-10-27 19:08:51] ...optimizers/trijoint.py.151: total_norm: 11.688385 [I 2022-10-27 19:08:57] ...trap/engines/engine.py.224: train: epoch 83 | batch 1240/2304 [I 2022-10-27 19:08:57] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:50 [I 2022-10-27 19:08:57] ...trap/engines/engine.py.229: process: 0.54611 | load: 0.00244 [I 2022-10-27 19:08:57] ...trap/engines/engine.py.230: loss: 0.15080 [I 2022-10-27 19:08:57] ...optimizers/trijoint.py.151: total_norm: 22.894772 [I 2022-10-27 19:09:02] ...trap/engines/engine.py.224: train: epoch 83 | batch 1250/2304 [I 2022-10-27 19:09:02] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:51 [I 2022-10-27 19:09:02] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00244 [I 2022-10-27 19:09:02] ...trap/engines/engine.py.230: loss: 0.09097 [I 2022-10-27 19:09:02] ...optimizers/trijoint.py.151: total_norm: 23.576654 [I 2022-10-27 19:09:08] ...trap/engines/engine.py.224: train: epoch 83 | batch 1260/2304 [I 2022-10-27 19:09:08] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:40 [I 2022-10-27 19:09:08] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00243 [I 2022-10-27 19:09:08] ...trap/engines/engine.py.230: loss: 0.19945 [I 2022-10-27 19:09:08] ...optimizers/trijoint.py.151: total_norm: 17.172913 [I 2022-10-27 19:09:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 1270/2304 [I 2022-10-27 19:09:13] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:35 [I 2022-10-27 19:09:13] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00244 [I 2022-10-27 19:09:13] ...trap/engines/engine.py.230: loss: 0.12060 [I 2022-10-27 19:09:13] ...optimizers/trijoint.py.151: total_norm: 17.270086 [I 2022-10-27 19:09:19] ...trap/engines/engine.py.224: train: epoch 83 | batch 1280/2304 [I 2022-10-27 19:09:19] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:31 [I 2022-10-27 19:09:19] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00245 [I 2022-10-27 19:09:19] ...trap/engines/engine.py.230: loss: 0.12444 [I 2022-10-27 19:09:19] ...optimizers/trijoint.py.151: total_norm: 13.538586 [I 2022-10-27 19:09:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 1290/2304 [I 2022-10-27 19:09:25] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:21 [I 2022-10-27 19:09:25] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00244 [I 2022-10-27 19:09:25] ...trap/engines/engine.py.230: loss: 0.13249 [I 2022-10-27 19:09:25] ...optimizers/trijoint.py.151: total_norm: 10.753282 [I 2022-10-27 19:09:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 1300/2304 [I 2022-10-27 19:09:30] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:19 [I 2022-10-27 19:09:30] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00251 [I 2022-10-27 19:09:30] ...trap/engines/engine.py.230: loss: 0.10181 [I 2022-10-27 19:09:30] ...optimizers/trijoint.py.151: total_norm: 16.174112 [I 2022-10-27 19:09:36] ...trap/engines/engine.py.224: train: epoch 83 | batch 1310/2304 [I 2022-10-27 19:09:36] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:11 [I 2022-10-27 19:09:36] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00249 [I 2022-10-27 19:09:36] ...trap/engines/engine.py.230: loss: 0.20988 [I 2022-10-27 19:09:36] ...optimizers/trijoint.py.151: total_norm: 37.835251 [I 2022-10-27 19:09:41] ...trap/engines/engine.py.224: train: epoch 83 | batch 1320/2304 [I 2022-10-27 19:09:41] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:05 [I 2022-10-27 19:09:41] ...trap/engines/engine.py.229: process: 0.54288 | load: 0.00245 [I 2022-10-27 19:09:41] ...trap/engines/engine.py.230: loss: 0.12411 [I 2022-10-27 19:09:41] ...optimizers/trijoint.py.151: total_norm: 27.015669 [I 2022-10-27 19:09:47] ...trap/engines/engine.py.224: train: epoch 83 | batch 1330/2304 [I 2022-10-27 19:09:47] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:00 [I 2022-10-27 19:09:47] ...trap/engines/engine.py.229: process: 0.54915 | load: 0.00249 [I 2022-10-27 19:09:47] ...trap/engines/engine.py.230: loss: 0.09497 [I 2022-10-27 19:09:47] ...optimizers/trijoint.py.151: total_norm: 10.413125 [I 2022-10-27 19:09:52] ...trap/engines/engine.py.224: train: epoch 83 | batch 1340/2304 [I 2022-10-27 19:09:52] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:56 [I 2022-10-27 19:09:52] ...trap/engines/engine.py.229: process: 0.54511 | load: 0.00243 [I 2022-10-27 19:09:52] ...trap/engines/engine.py.230: loss: 0.13594 [I 2022-10-27 19:09:52] ...optimizers/trijoint.py.151: total_norm: 10.473303 [I 2022-10-27 19:09:58] ...trap/engines/engine.py.224: train: epoch 83 | batch 1350/2304 [I 2022-10-27 19:09:58] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:54 [I 2022-10-27 19:09:58] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00246 [I 2022-10-27 19:09:58] ...trap/engines/engine.py.230: loss: 0.12076 [I 2022-10-27 19:09:58] ...optimizers/trijoint.py.151: total_norm: 46.360973 [I 2022-10-27 19:10:04] ...trap/engines/engine.py.224: train: epoch 83 | batch 1360/2304 [I 2022-10-27 19:10:04] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:40 [I 2022-10-27 19:10:04] ...trap/engines/engine.py.229: process: 0.54861 | load: 0.00244 [I 2022-10-27 19:10:04] ...trap/engines/engine.py.230: loss: 0.10581 [I 2022-10-27 19:10:04] ...optimizers/trijoint.py.151: total_norm: 9.166708 [I 2022-10-27 19:10:09] ...trap/engines/engine.py.224: train: epoch 83 | batch 1370/2304 [I 2022-10-27 19:10:09] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:34 [I 2022-10-27 19:10:09] ...trap/engines/engine.py.229: process: 0.54601 | load: 0.00240 [I 2022-10-27 19:10:09] ...trap/engines/engine.py.230: loss: 0.13696 [I 2022-10-27 19:10:09] ...optimizers/trijoint.py.151: total_norm: 12.529661 [I 2022-10-27 19:10:15] ...trap/engines/engine.py.224: train: epoch 83 | batch 1380/2304 [I 2022-10-27 19:10:15] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:33 [I 2022-10-27 19:10:15] ...trap/engines/engine.py.229: process: 0.55790 | load: 0.00240 [I 2022-10-27 19:10:15] ...trap/engines/engine.py.230: loss: 0.10922 [I 2022-10-27 19:10:15] ...optimizers/trijoint.py.151: total_norm: 21.590725 [I 2022-10-27 19:10:20] ...trap/engines/engine.py.224: train: epoch 83 | batch 1390/2304 [I 2022-10-27 19:10:20] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:26 [I 2022-10-27 19:10:20] ...trap/engines/engine.py.229: process: 0.54885 | load: 0.00243 [I 2022-10-27 19:10:20] ...trap/engines/engine.py.230: loss: 0.17280 [I 2022-10-27 19:10:20] ...optimizers/trijoint.py.151: total_norm: 16.152533 [I 2022-10-27 19:10:26] ...trap/engines/engine.py.224: train: epoch 83 | batch 1400/2304 [I 2022-10-27 19:10:26] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:21 [I 2022-10-27 19:10:26] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00245 [I 2022-10-27 19:10:26] ...trap/engines/engine.py.230: loss: 0.16942 [I 2022-10-27 19:10:26] ...optimizers/trijoint.py.151: total_norm: 19.440193 [I 2022-10-27 19:10:31] ...trap/engines/engine.py.224: train: epoch 83 | batch 1410/2304 [I 2022-10-27 19:10:31] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:17 [I 2022-10-27 19:10:31] ...trap/engines/engine.py.229: process: 0.56256 | load: 0.00247 [I 2022-10-27 19:10:31] ...trap/engines/engine.py.230: loss: 0.10180 [I 2022-10-27 19:10:31] ...optimizers/trijoint.py.151: total_norm: 18.572304 [I 2022-10-27 19:10:37] ...trap/engines/engine.py.224: train: epoch 83 | batch 1420/2304 [I 2022-10-27 19:10:37] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:14 [I 2022-10-27 19:10:37] ...trap/engines/engine.py.229: process: 0.57123 | load: 0.00249 [I 2022-10-27 19:10:37] ...trap/engines/engine.py.230: loss: 0.12333 [I 2022-10-27 19:10:37] ...optimizers/trijoint.py.151: total_norm: 11.868911 [I 2022-10-27 19:10:43] ...trap/engines/engine.py.224: train: epoch 83 | batch 1430/2304 [I 2022-10-27 19:10:43] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:06 [I 2022-10-27 19:10:43] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00247 [I 2022-10-27 19:10:43] ...trap/engines/engine.py.230: loss: 0.13546 [I 2022-10-27 19:10:43] ...optimizers/trijoint.py.151: total_norm: 18.556927 [I 2022-10-27 19:10:48] ...trap/engines/engine.py.224: train: epoch 83 | batch 1440/2304 [I 2022-10-27 19:10:48] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:07:57 [I 2022-10-27 19:10:48] ...trap/engines/engine.py.229: process: 0.53996 | load: 0.00244 [I 2022-10-27 19:10:48] ...trap/engines/engine.py.230: loss: 0.12698 [I 2022-10-27 19:10:48] ...optimizers/trijoint.py.151: total_norm: 12.998872 [I 2022-10-27 19:10:54] ...trap/engines/engine.py.224: train: epoch 83 | batch 1450/2304 [I 2022-10-27 19:10:54] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:54 [I 2022-10-27 19:10:54] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00285 [I 2022-10-27 19:10:54] ...trap/engines/engine.py.230: loss: 0.13668 [I 2022-10-27 19:10:54] ...optimizers/trijoint.py.151: total_norm: 17.082630 [I 2022-10-27 19:10:59] ...trap/engines/engine.py.224: train: epoch 83 | batch 1460/2304 [I 2022-10-27 19:10:59] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:46 [I 2022-10-27 19:10:59] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00247 [I 2022-10-27 19:10:59] ...trap/engines/engine.py.230: loss: 0.16404 [I 2022-10-27 19:10:59] ...optimizers/trijoint.py.151: total_norm: 10.795974 [I 2022-10-27 19:11:05] ...trap/engines/engine.py.224: train: epoch 83 | batch 1470/2304 [I 2022-10-27 19:11:05] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:51 [I 2022-10-27 19:11:05] ...trap/engines/engine.py.229: process: 0.56591 | load: 0.00244 [I 2022-10-27 19:11:05] ...trap/engines/engine.py.230: loss: 0.13531 [I 2022-10-27 19:11:05] ...optimizers/trijoint.py.151: total_norm: 12.824113 [I 2022-10-27 19:11:10] ...trap/engines/engine.py.224: train: epoch 83 | batch 1480/2304 [I 2022-10-27 19:11:10] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:33 [I 2022-10-27 19:11:10] ...trap/engines/engine.py.229: process: 0.55181 | load: 0.00243 [I 2022-10-27 19:11:10] ...trap/engines/engine.py.230: loss: 0.11749 [I 2022-10-27 19:11:10] ...optimizers/trijoint.py.151: total_norm: 8.825587 [I 2022-10-27 19:11:16] ...trap/engines/engine.py.224: train: epoch 83 | batch 1490/2304 [I 2022-10-27 19:11:16] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:30 [I 2022-10-27 19:11:16] ...trap/engines/engine.py.229: process: 0.55099 | load: 0.00260 [I 2022-10-27 19:11:16] ...trap/engines/engine.py.230: loss: 0.12128 [I 2022-10-27 19:11:16] ...optimizers/trijoint.py.151: total_norm: 12.340969 [I 2022-10-27 19:11:22] ...trap/engines/engine.py.224: train: epoch 83 | batch 1500/2304 [I 2022-10-27 19:11:22] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:29 [I 2022-10-27 19:11:22] ...trap/engines/engine.py.229: process: 0.54845 | load: 0.00262 [I 2022-10-27 19:11:22] ...trap/engines/engine.py.230: loss: 0.12338 [I 2022-10-27 19:11:22] ...optimizers/trijoint.py.151: total_norm: 10.866585 [I 2022-10-27 19:11:27] ...trap/engines/engine.py.224: train: epoch 83 | batch 1510/2304 [I 2022-10-27 19:11:27] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:22 [I 2022-10-27 19:11:27] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00247 [I 2022-10-27 19:11:27] ...trap/engines/engine.py.230: loss: 0.09603 [I 2022-10-27 19:11:27] ...optimizers/trijoint.py.151: total_norm: 8.395921 [I 2022-10-27 19:11:33] ...trap/engines/engine.py.224: train: epoch 83 | batch 1520/2304 [I 2022-10-27 19:11:33] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:12 [I 2022-10-27 19:11:33] ...trap/engines/engine.py.229: process: 0.54507 | load: 0.00251 [I 2022-10-27 19:11:33] ...trap/engines/engine.py.230: loss: 0.12021 [I 2022-10-27 19:11:33] ...optimizers/trijoint.py.151: total_norm: 10.963655 [I 2022-10-27 19:11:38] ...trap/engines/engine.py.224: train: epoch 83 | batch 1530/2304 [I 2022-10-27 19:11:38] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:10 [I 2022-10-27 19:11:38] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00249 [I 2022-10-27 19:11:38] ...trap/engines/engine.py.230: loss: 0.10818 [I 2022-10-27 19:11:38] ...optimizers/trijoint.py.151: total_norm: 10.209797 [I 2022-10-27 19:11:44] ...trap/engines/engine.py.224: train: epoch 83 | batch 1540/2304 [I 2022-10-27 19:11:44] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:03 [I 2022-10-27 19:11:44] ...trap/engines/engine.py.229: process: 0.54073 | load: 0.00244 [I 2022-10-27 19:11:44] ...trap/engines/engine.py.230: loss: 0.10930 [I 2022-10-27 19:11:44] ...optimizers/trijoint.py.151: total_norm: 9.511393 [I 2022-10-27 19:11:49] ...trap/engines/engine.py.224: train: epoch 83 | batch 1550/2304 [I 2022-10-27 19:11:49] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:06:55 [I 2022-10-27 19:11:49] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00246 [I 2022-10-27 19:11:49] ...trap/engines/engine.py.230: loss: 0.08592 [I 2022-10-27 19:11:49] ...optimizers/trijoint.py.151: total_norm: 10.741172 [I 2022-10-27 19:11:55] ...trap/engines/engine.py.224: train: epoch 83 | batch 1560/2304 [I 2022-10-27 19:11:55] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:06:53 [I 2022-10-27 19:11:55] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00247 [I 2022-10-27 19:11:55] ...trap/engines/engine.py.230: loss: 0.13925 [I 2022-10-27 19:11:55] ...optimizers/trijoint.py.151: total_norm: 10.349663 [I 2022-10-27 19:12:00] ...trap/engines/engine.py.224: train: epoch 83 | batch 1570/2304 [I 2022-10-27 19:12:00] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:06:47 [I 2022-10-27 19:12:00] ...trap/engines/engine.py.229: process: 0.55312 | load: 0.00243 [I 2022-10-27 19:12:00] ...trap/engines/engine.py.230: loss: 0.10783 [I 2022-10-27 19:12:00] ...optimizers/trijoint.py.151: total_norm: 17.032640 [I 2022-10-27 19:12:06] ...trap/engines/engine.py.224: train: epoch 83 | batch 1580/2304 [I 2022-10-27 19:12:06] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:41 [I 2022-10-27 19:12:06] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00247 [I 2022-10-27 19:12:06] ...trap/engines/engine.py.230: loss: 0.10789 [I 2022-10-27 19:12:06] ...optimizers/trijoint.py.151: total_norm: 14.222014 [I 2022-10-27 19:12:12] ...trap/engines/engine.py.224: train: epoch 83 | batch 1590/2304 [I 2022-10-27 19:12:12] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:36 [I 2022-10-27 19:12:12] ...trap/engines/engine.py.229: process: 0.55507 | load: 0.00241 [I 2022-10-27 19:12:12] ...trap/engines/engine.py.230: loss: 0.18964 [I 2022-10-27 19:12:12] ...optimizers/trijoint.py.151: total_norm: 71.790627 [I 2022-10-27 19:12:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 1600/2304 [I 2022-10-27 19:12:17] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:31 [I 2022-10-27 19:12:17] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00241 [I 2022-10-27 19:12:17] ...trap/engines/engine.py.230: loss: 0.10273 [I 2022-10-27 19:12:17] ...optimizers/trijoint.py.151: total_norm: 10.674034 [I 2022-10-27 19:12:23] ...trap/engines/engine.py.224: train: epoch 83 | batch 1610/2304 [I 2022-10-27 19:12:23] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:27 [I 2022-10-27 19:12:23] ...trap/engines/engine.py.229: process: 0.56460 | load: 0.00244 [I 2022-10-27 19:12:23] ...trap/engines/engine.py.230: loss: 0.09136 [I 2022-10-27 19:12:23] ...optimizers/trijoint.py.151: total_norm: 9.188198 [I 2022-10-27 19:12:28] ...trap/engines/engine.py.224: train: epoch 83 | batch 1620/2304 [I 2022-10-27 19:12:28] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:23 [I 2022-10-27 19:12:28] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00254 [I 2022-10-27 19:12:28] ...trap/engines/engine.py.230: loss: 0.11941 [I 2022-10-27 19:12:28] ...optimizers/trijoint.py.151: total_norm: 8.658347 [I 2022-10-27 19:12:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 1630/2304 [I 2022-10-27 19:12:34] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:12 [I 2022-10-27 19:12:34] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00250 [I 2022-10-27 19:12:34] ...trap/engines/engine.py.230: loss: 0.22737 [I 2022-10-27 19:12:34] ...optimizers/trijoint.py.151: total_norm: 36.819530 [I 2022-10-27 19:12:40] ...trap/engines/engine.py.224: train: epoch 83 | batch 1640/2304 [I 2022-10-27 19:12:40] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:09 [I 2022-10-27 19:12:40] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00246 [I 2022-10-27 19:12:40] ...trap/engines/engine.py.230: loss: 0.15213 [I 2022-10-27 19:12:40] ...optimizers/trijoint.py.151: total_norm: 9.790087 [I 2022-10-27 19:12:45] ...trap/engines/engine.py.224: train: epoch 83 | batch 1650/2304 [I 2022-10-27 19:12:45] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:04 [I 2022-10-27 19:12:45] ...trap/engines/engine.py.229: process: 0.55628 | load: 0.00244 [I 2022-10-27 19:12:45] ...trap/engines/engine.py.230: loss: 0.19064 [I 2022-10-27 19:12:45] ...optimizers/trijoint.py.151: total_norm: 25.184233 [I 2022-10-27 19:12:51] ...trap/engines/engine.py.224: train: epoch 83 | batch 1660/2304 [I 2022-10-27 19:12:51] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:05:57 [I 2022-10-27 19:12:51] ...trap/engines/engine.py.229: process: 0.55381 | load: 0.00244 [I 2022-10-27 19:12:51] ...trap/engines/engine.py.230: loss: 0.11830 [I 2022-10-27 19:12:51] ...optimizers/trijoint.py.151: total_norm: 15.976735 [I 2022-10-27 19:12:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 1670/2304 [I 2022-10-27 19:12:56] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:05:52 [I 2022-10-27 19:12:56] ...trap/engines/engine.py.229: process: 0.55342 | load: 0.00246 [I 2022-10-27 19:12:56] ...trap/engines/engine.py.230: loss: 0.13051 [I 2022-10-27 19:12:56] ...optimizers/trijoint.py.151: total_norm: 22.239166 [I 2022-10-27 19:13:02] ...trap/engines/engine.py.224: train: epoch 83 | batch 1680/2304 [I 2022-10-27 19:13:02] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:48 [I 2022-10-27 19:13:02] ...trap/engines/engine.py.229: process: 0.56359 | load: 0.00245 [I 2022-10-27 19:13:02] ...trap/engines/engine.py.230: loss: 0.16876 [I 2022-10-27 19:13:02] ...optimizers/trijoint.py.151: total_norm: 10.570683 [I 2022-10-27 19:13:07] ...trap/engines/engine.py.224: train: epoch 83 | batch 1690/2304 [I 2022-10-27 19:13:07] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:05:41 [I 2022-10-27 19:13:07] ...trap/engines/engine.py.229: process: 0.56741 | load: 0.00241 [I 2022-10-27 19:13:07] ...trap/engines/engine.py.230: loss: 0.15290 [I 2022-10-27 19:13:07] ...optimizers/trijoint.py.151: total_norm: 15.894227 [I 2022-10-27 19:13:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 1700/2304 [I 2022-10-27 19:13:13] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:32 [I 2022-10-27 19:13:13] ...trap/engines/engine.py.229: process: 0.53784 | load: 0.00245 [I 2022-10-27 19:13:13] ...trap/engines/engine.py.230: loss: 0.09648 [I 2022-10-27 19:13:13] ...optimizers/trijoint.py.151: total_norm: 18.917103 [I 2022-10-27 19:13:19] ...trap/engines/engine.py.224: train: epoch 83 | batch 1710/2304 [I 2022-10-27 19:13:19] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:29 [I 2022-10-27 19:13:19] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00246 [I 2022-10-27 19:13:19] ...trap/engines/engine.py.230: loss: 0.12555 [I 2022-10-27 19:13:19] ...optimizers/trijoint.py.151: total_norm: 20.879971 [I 2022-10-27 19:13:24] ...trap/engines/engine.py.224: train: epoch 83 | batch 1720/2304 [I 2022-10-27 19:13:24] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:23 [I 2022-10-27 19:13:24] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00246 [I 2022-10-27 19:13:24] ...trap/engines/engine.py.230: loss: 0.08270 [I 2022-10-27 19:13:24] ...optimizers/trijoint.py.151: total_norm: 29.444721 [I 2022-10-27 19:13:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 1730/2304 [I 2022-10-27 19:13:30] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:17 [I 2022-10-27 19:13:30] ...trap/engines/engine.py.229: process: 0.54433 | load: 0.00248 [I 2022-10-27 19:13:30] ...trap/engines/engine.py.230: loss: 0.14920 [I 2022-10-27 19:13:30] ...optimizers/trijoint.py.151: total_norm: 17.630596 [I 2022-10-27 19:13:35] ...trap/engines/engine.py.224: train: epoch 83 | batch 1740/2304 [I 2022-10-27 19:13:35] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:13 [I 2022-10-27 19:13:35] ...trap/engines/engine.py.229: process: 0.55297 | load: 0.00247 [I 2022-10-27 19:13:35] ...trap/engines/engine.py.230: loss: 0.11751 [I 2022-10-27 19:13:35] ...optimizers/trijoint.py.151: total_norm: 19.137350 [I 2022-10-27 19:13:41] ...trap/engines/engine.py.224: train: epoch 83 | batch 1750/2304 [I 2022-10-27 19:13:41] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:07 [I 2022-10-27 19:13:41] ...trap/engines/engine.py.229: process: 0.55257 | load: 0.00246 [I 2022-10-27 19:13:41] ...trap/engines/engine.py.230: loss: 0.21443 [I 2022-10-27 19:13:41] ...optimizers/trijoint.py.151: total_norm: 39.858868 [I 2022-10-27 19:13:46] ...trap/engines/engine.py.224: train: epoch 83 | batch 1760/2304 [I 2022-10-27 19:13:46] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:02 [I 2022-10-27 19:13:46] ...trap/engines/engine.py.229: process: 0.54173 | load: 0.00247 [I 2022-10-27 19:13:46] ...trap/engines/engine.py.230: loss: 0.10353 [I 2022-10-27 19:13:46] ...optimizers/trijoint.py.151: total_norm: 31.219631 [I 2022-10-27 19:13:52] ...trap/engines/engine.py.224: train: epoch 83 | batch 1770/2304 [I 2022-10-27 19:13:52] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:04:54 [I 2022-10-27 19:13:52] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00244 [I 2022-10-27 19:13:52] ...trap/engines/engine.py.230: loss: 0.16555 [I 2022-10-27 19:13:52] ...optimizers/trijoint.py.151: total_norm: 11.032306 [I 2022-10-27 19:13:57] ...trap/engines/engine.py.224: train: epoch 83 | batch 1780/2304 [I 2022-10-27 19:13:57] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:04:50 [I 2022-10-27 19:13:57] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00243 [I 2022-10-27 19:13:57] ...trap/engines/engine.py.230: loss: 0.12001 [I 2022-10-27 19:13:57] ...optimizers/trijoint.py.151: total_norm: 21.418598 [I 2022-10-27 19:14:03] ...trap/engines/engine.py.224: train: epoch 83 | batch 1790/2304 [I 2022-10-27 19:14:03] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:04:45 [I 2022-10-27 19:14:03] ...trap/engines/engine.py.229: process: 0.55513 | load: 0.00242 [I 2022-10-27 19:14:03] ...trap/engines/engine.py.230: loss: 0.11972 [I 2022-10-27 19:14:03] ...optimizers/trijoint.py.151: total_norm: 13.132891 [I 2022-10-27 19:14:09] ...trap/engines/engine.py.224: train: epoch 83 | batch 1800/2304 [I 2022-10-27 19:14:09] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:40 [I 2022-10-27 19:14:09] ...trap/engines/engine.py.229: process: 0.56591 | load: 0.00243 [I 2022-10-27 19:14:09] ...trap/engines/engine.py.230: loss: 0.11359 [I 2022-10-27 19:14:09] ...optimizers/trijoint.py.151: total_norm: 24.046444 [I 2022-10-27 19:14:14] ...trap/engines/engine.py.224: train: epoch 83 | batch 1810/2304 [I 2022-10-27 19:14:14] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:35 [I 2022-10-27 19:14:14] ...trap/engines/engine.py.229: process: 0.56163 | load: 0.00245 [I 2022-10-27 19:14:14] ...trap/engines/engine.py.230: loss: 0.15983 [I 2022-10-27 19:14:14] ...optimizers/trijoint.py.151: total_norm: 14.574942 [I 2022-10-27 19:14:20] ...trap/engines/engine.py.224: train: epoch 83 | batch 1820/2304 [I 2022-10-27 19:14:20] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:27 [I 2022-10-27 19:14:20] ...trap/engines/engine.py.229: process: 0.55831 | load: 0.00243 [I 2022-10-27 19:14:20] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-27 19:14:20] ...optimizers/trijoint.py.151: total_norm: 11.440643 [I 2022-10-27 19:14:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 1830/2304 [I 2022-10-27 19:14:25] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:24 [I 2022-10-27 19:14:25] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00239 [I 2022-10-27 19:14:25] ...trap/engines/engine.py.230: loss: 0.14173 [I 2022-10-27 19:14:25] ...optimizers/trijoint.py.151: total_norm: 14.488540 [I 2022-10-27 19:14:31] ...trap/engines/engine.py.224: train: epoch 83 | batch 1840/2304 [I 2022-10-27 19:14:31] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:17 [I 2022-10-27 19:14:31] ...trap/engines/engine.py.229: process: 0.54940 | load: 0.00242 [I 2022-10-27 19:14:31] ...trap/engines/engine.py.230: loss: 0.11307 [I 2022-10-27 19:14:31] ...optimizers/trijoint.py.151: total_norm: 9.045612 [I 2022-10-27 19:14:36] ...trap/engines/engine.py.224: train: epoch 83 | batch 1850/2304 [I 2022-10-27 19:14:36] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:11 [I 2022-10-27 19:14:36] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00243 [I 2022-10-27 19:14:36] ...trap/engines/engine.py.230: loss: 0.12546 [I 2022-10-27 19:14:36] ...optimizers/trijoint.py.151: total_norm: 27.251589 [I 2022-10-27 19:14:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 1860/2304 [I 2022-10-27 19:14:42] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:07 [I 2022-10-27 19:14:42] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00242 [I 2022-10-27 19:14:42] ...trap/engines/engine.py.230: loss: 0.10107 [I 2022-10-27 19:14:42] ...optimizers/trijoint.py.151: total_norm: 11.540770 [I 2022-10-27 19:14:48] ...trap/engines/engine.py.224: train: epoch 83 | batch 1870/2304 [I 2022-10-27 19:14:48] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:00 [I 2022-10-27 19:14:48] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00243 [I 2022-10-27 19:14:48] ...trap/engines/engine.py.230: loss: 0.14710 [I 2022-10-27 19:14:48] ...optimizers/trijoint.py.151: total_norm: 19.534597 [I 2022-10-27 19:14:53] ...trap/engines/engine.py.224: train: epoch 83 | batch 1880/2304 [I 2022-10-27 19:14:53] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:03:55 [I 2022-10-27 19:14:53] ...trap/engines/engine.py.229: process: 0.55204 | load: 0.00244 [I 2022-10-27 19:14:53] ...trap/engines/engine.py.230: loss: 0.21041 [I 2022-10-27 19:14:53] ...optimizers/trijoint.py.151: total_norm: 27.721128 [I 2022-10-27 19:14:59] ...trap/engines/engine.py.224: train: epoch 83 | batch 1890/2304 [I 2022-10-27 19:14:59] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:03:50 [I 2022-10-27 19:14:59] ...trap/engines/engine.py.229: process: 0.54765 | load: 0.00250 [I 2022-10-27 19:14:59] ...trap/engines/engine.py.230: loss: 0.14420 [I 2022-10-27 19:14:59] ...optimizers/trijoint.py.151: total_norm: 32.779186 [I 2022-10-27 19:15:04] ...trap/engines/engine.py.224: train: epoch 83 | batch 1900/2304 [I 2022-10-27 19:15:04] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:03:45 [I 2022-10-27 19:15:04] ...trap/engines/engine.py.229: process: 0.55980 | load: 0.00246 [I 2022-10-27 19:15:04] ...trap/engines/engine.py.230: loss: 0.12708 [I 2022-10-27 19:15:04] ...optimizers/trijoint.py.151: total_norm: 12.753687 [I 2022-10-27 19:15:10] ...trap/engines/engine.py.224: train: epoch 83 | batch 1910/2304 [I 2022-10-27 19:15:10] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:03:38 [I 2022-10-27 19:15:10] ...trap/engines/engine.py.229: process: 0.55942 | load: 0.00244 [I 2022-10-27 19:15:10] ...trap/engines/engine.py.230: loss: 0.09282 [I 2022-10-27 19:15:10] ...optimizers/trijoint.py.151: total_norm: 9.837460 [I 2022-10-27 19:15:15] ...trap/engines/engine.py.224: train: epoch 83 | batch 1920/2304 [I 2022-10-27 19:15:15] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:33 [I 2022-10-27 19:15:15] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00243 [I 2022-10-27 19:15:15] ...trap/engines/engine.py.230: loss: 0.12257 [I 2022-10-27 19:15:15] ...optimizers/trijoint.py.151: total_norm: 48.192104 [I 2022-10-27 19:15:21] ...trap/engines/engine.py.224: train: epoch 83 | batch 1930/2304 [I 2022-10-27 19:15:21] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:27 [I 2022-10-27 19:15:21] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00244 [I 2022-10-27 19:15:21] ...trap/engines/engine.py.230: loss: 0.12877 [I 2022-10-27 19:15:21] ...optimizers/trijoint.py.151: total_norm: 27.639662 [I 2022-10-27 19:15:27] ...trap/engines/engine.py.224: train: epoch 83 | batch 1940/2304 [I 2022-10-27 19:15:27] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:22 [I 2022-10-27 19:15:27] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00244 [I 2022-10-27 19:15:27] ...trap/engines/engine.py.230: loss: 0.15115 [I 2022-10-27 19:15:27] ...optimizers/trijoint.py.151: total_norm: 21.023035 [I 2022-10-27 19:15:32] ...trap/engines/engine.py.224: train: epoch 83 | batch 1950/2304 [I 2022-10-27 19:15:32] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:17 [I 2022-10-27 19:15:32] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00243 [I 2022-10-27 19:15:32] ...trap/engines/engine.py.230: loss: 0.14161 [I 2022-10-27 19:15:32] ...optimizers/trijoint.py.151: total_norm: 12.378581 [I 2022-10-27 19:15:38] ...trap/engines/engine.py.224: train: epoch 83 | batch 1960/2304 [I 2022-10-27 19:15:38] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:12 [I 2022-10-27 19:15:38] ...trap/engines/engine.py.229: process: 0.57265 | load: 0.00248 [I 2022-10-27 19:15:38] ...trap/engines/engine.py.230: loss: 0.11488 [I 2022-10-27 19:15:38] ...optimizers/trijoint.py.151: total_norm: 15.648285 [I 2022-10-27 19:15:43] ...trap/engines/engine.py.224: train: epoch 83 | batch 1970/2304 [I 2022-10-27 19:15:43] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:06 [I 2022-10-27 19:15:43] ...trap/engines/engine.py.229: process: 0.54974 | load: 0.00250 [I 2022-10-27 19:15:43] ...trap/engines/engine.py.230: loss: 0.15753 [I 2022-10-27 19:15:43] ...optimizers/trijoint.py.151: total_norm: 18.246618 [I 2022-10-27 19:15:49] ...trap/engines/engine.py.224: train: epoch 83 | batch 1980/2304 [I 2022-10-27 19:15:49] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:02:59 [I 2022-10-27 19:15:49] ...trap/engines/engine.py.229: process: 0.54703 | load: 0.00245 [I 2022-10-27 19:15:49] ...trap/engines/engine.py.230: loss: 0.11270 [I 2022-10-27 19:15:49] ...optimizers/trijoint.py.151: total_norm: 19.274162 [I 2022-10-27 19:15:54] ...trap/engines/engine.py.224: train: epoch 83 | batch 1990/2304 [I 2022-10-27 19:15:54] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:02:53 [I 2022-10-27 19:15:54] ...trap/engines/engine.py.229: process: 0.56219 | load: 0.00247 [I 2022-10-27 19:15:54] ...trap/engines/engine.py.230: loss: 0.11213 [I 2022-10-27 19:15:54] ...optimizers/trijoint.py.151: total_norm: 12.791243 [I 2022-10-27 19:16:00] ...trap/engines/engine.py.224: train: epoch 83 | batch 2000/2304 [I 2022-10-27 19:16:00] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:02:48 [I 2022-10-27 19:16:00] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00244 [I 2022-10-27 19:16:00] ...trap/engines/engine.py.230: loss: 0.12237 [I 2022-10-27 19:16:00] ...optimizers/trijoint.py.151: total_norm: 11.376091 [I 2022-10-27 19:16:06] ...trap/engines/engine.py.224: train: epoch 83 | batch 2010/2304 [I 2022-10-27 19:16:06] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:02:43 [I 2022-10-27 19:16:06] ...trap/engines/engine.py.229: process: 0.55561 | load: 0.00241 [I 2022-10-27 19:16:06] ...trap/engines/engine.py.230: loss: 0.08806 [I 2022-10-27 19:16:06] ...optimizers/trijoint.py.151: total_norm: 10.679850 [I 2022-10-27 19:16:11] ...trap/engines/engine.py.224: train: epoch 83 | batch 2020/2304 [I 2022-10-27 19:16:11] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:36 [I 2022-10-27 19:16:11] ...trap/engines/engine.py.229: process: 0.54847 | load: 0.00245 [I 2022-10-27 19:16:11] ...trap/engines/engine.py.230: loss: 0.14996 [I 2022-10-27 19:16:11] ...optimizers/trijoint.py.151: total_norm: 19.179482 [I 2022-10-27 19:16:17] ...trap/engines/engine.py.224: train: epoch 83 | batch 2030/2304 [I 2022-10-27 19:16:17] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:30 [I 2022-10-27 19:16:17] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00242 [I 2022-10-27 19:16:17] ...trap/engines/engine.py.230: loss: 0.17109 [I 2022-10-27 19:16:17] ...optimizers/trijoint.py.151: total_norm: 13.301043 [I 2022-10-27 19:16:22] ...trap/engines/engine.py.224: train: epoch 83 | batch 2040/2304 [I 2022-10-27 19:16:22] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:27 [I 2022-10-27 19:16:22] ...trap/engines/engine.py.229: process: 0.56373 | load: 0.00260 [I 2022-10-27 19:16:22] ...trap/engines/engine.py.230: loss: 0.13308 [I 2022-10-27 19:16:22] ...optimizers/trijoint.py.151: total_norm: 24.077948 [I 2022-10-27 19:16:28] ...trap/engines/engine.py.224: train: epoch 83 | batch 2050/2304 [I 2022-10-27 19:16:28] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:26 [I 2022-10-27 19:16:28] ...trap/engines/engine.py.229: process: 0.57423 | load: 0.00242 [I 2022-10-27 19:16:28] ...trap/engines/engine.py.230: loss: 0.10133 [I 2022-10-27 19:16:28] ...optimizers/trijoint.py.151: total_norm: 11.315544 [I 2022-10-27 19:16:34] ...trap/engines/engine.py.224: train: epoch 83 | batch 2060/2304 [I 2022-10-27 19:16:34] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:17 [I 2022-10-27 19:16:34] ...trap/engines/engine.py.229: process: 0.56622 | load: 0.00253 [I 2022-10-27 19:16:34] ...trap/engines/engine.py.230: loss: 0.08709 [I 2022-10-27 19:16:34] ...optimizers/trijoint.py.151: total_norm: 22.882925 [I 2022-10-27 19:16:39] ...trap/engines/engine.py.224: train: epoch 83 | batch 2070/2304 [I 2022-10-27 19:16:39] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:12 [I 2022-10-27 19:16:39] ...trap/engines/engine.py.229: process: 0.56178 | load: 0.00246 [I 2022-10-27 19:16:39] ...trap/engines/engine.py.230: loss: 0.14736 [I 2022-10-27 19:16:39] ...optimizers/trijoint.py.151: total_norm: 20.300198 [I 2022-10-27 19:16:45] ...trap/engines/engine.py.224: train: epoch 83 | batch 2080/2304 [I 2022-10-27 19:16:45] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:05 [I 2022-10-27 19:16:45] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00245 [I 2022-10-27 19:16:45] ...trap/engines/engine.py.230: loss: 0.15015 [I 2022-10-27 19:16:45] ...optimizers/trijoint.py.151: total_norm: 28.211414 [I 2022-10-27 19:16:50] ...trap/engines/engine.py.224: train: epoch 83 | batch 2090/2304 [I 2022-10-27 19:16:50] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:01:59 [I 2022-10-27 19:16:50] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00242 [I 2022-10-27 19:16:50] ...trap/engines/engine.py.230: loss: 0.10966 [I 2022-10-27 19:16:50] ...optimizers/trijoint.py.151: total_norm: 15.293458 [I 2022-10-27 19:16:56] ...trap/engines/engine.py.224: train: epoch 83 | batch 2100/2304 [I 2022-10-27 19:16:56] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:01:54 [I 2022-10-27 19:16:56] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00244 [I 2022-10-27 19:16:56] ...trap/engines/engine.py.230: loss: 0.12027 [I 2022-10-27 19:16:56] ...optimizers/trijoint.py.151: total_norm: 12.673016 [I 2022-10-27 19:17:02] ...trap/engines/engine.py.224: train: epoch 83 | batch 2110/2304 [I 2022-10-27 19:17:02] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:01:48 [I 2022-10-27 19:17:02] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00248 [I 2022-10-27 19:17:02] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-27 19:17:02] ...optimizers/trijoint.py.151: total_norm: 26.481327 [I 2022-10-27 19:17:07] ...trap/engines/engine.py.224: train: epoch 83 | batch 2120/2304 [I 2022-10-27 19:17:07] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:01:42 [I 2022-10-27 19:17:07] ...trap/engines/engine.py.229: process: 0.56380 | load: 0.00243 [I 2022-10-27 19:17:07] ...trap/engines/engine.py.230: loss: 0.21861 [I 2022-10-27 19:17:07] ...optimizers/trijoint.py.151: total_norm: 43.443611 [I 2022-10-27 19:17:13] ...trap/engines/engine.py.224: train: epoch 83 | batch 2130/2304 [I 2022-10-27 19:17:13] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:36 [I 2022-10-27 19:17:13] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00241 [I 2022-10-27 19:17:13] ...trap/engines/engine.py.230: loss: 0.16045 [I 2022-10-27 19:17:13] ...optimizers/trijoint.py.151: total_norm: 12.154576 [I 2022-10-27 19:17:18] ...trap/engines/engine.py.224: train: epoch 83 | batch 2140/2304 [I 2022-10-27 19:17:18] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:30 [I 2022-10-27 19:17:18] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00247 [I 2022-10-27 19:17:18] ...trap/engines/engine.py.230: loss: 0.13572 [I 2022-10-27 19:17:18] ...optimizers/trijoint.py.151: total_norm: 12.984118 [I 2022-10-27 19:17:24] ...trap/engines/engine.py.224: train: epoch 83 | batch 2150/2304 [I 2022-10-27 19:17:24] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:27 [I 2022-10-27 19:17:24] ...trap/engines/engine.py.229: process: 0.56526 | load: 0.00247 [I 2022-10-27 19:17:24] ...trap/engines/engine.py.230: loss: 0.18703 [I 2022-10-27 19:17:24] ...optimizers/trijoint.py.151: total_norm: 27.393408 [I 2022-10-27 19:17:30] ...trap/engines/engine.py.224: train: epoch 83 | batch 2160/2304 [I 2022-10-27 19:17:30] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:20 [I 2022-10-27 19:17:30] ...trap/engines/engine.py.229: process: 0.56087 | load: 0.00246 [I 2022-10-27 19:17:30] ...trap/engines/engine.py.230: loss: 0.15225 [I 2022-10-27 19:17:30] ...optimizers/trijoint.py.151: total_norm: 14.634246 [I 2022-10-27 19:17:35] ...trap/engines/engine.py.224: train: epoch 83 | batch 2170/2304 [I 2022-10-27 19:17:35] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:14 [I 2022-10-27 19:17:35] ...trap/engines/engine.py.229: process: 0.54127 | load: 0.00249 [I 2022-10-27 19:17:35] ...trap/engines/engine.py.230: loss: 0.14560 [I 2022-10-27 19:17:35] ...optimizers/trijoint.py.151: total_norm: 11.334550 [I 2022-10-27 19:17:41] ...trap/engines/engine.py.224: train: epoch 83 | batch 2180/2304 [I 2022-10-27 19:17:41] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:09 [I 2022-10-27 19:17:41] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00245 [I 2022-10-27 19:17:41] ...trap/engines/engine.py.230: loss: 0.14871 [I 2022-10-27 19:17:41] ...optimizers/trijoint.py.151: total_norm: 35.001091 [I 2022-10-27 19:17:46] ...trap/engines/engine.py.224: train: epoch 83 | batch 2190/2304 [I 2022-10-27 19:17:46] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:03 [I 2022-10-27 19:17:46] ...trap/engines/engine.py.229: process: 0.56504 | load: 0.00246 [I 2022-10-27 19:17:46] ...trap/engines/engine.py.230: loss: 0.10456 [I 2022-10-27 19:17:46] ...optimizers/trijoint.py.151: total_norm: 12.617782 [I 2022-10-27 19:17:52] ...trap/engines/engine.py.224: train: epoch 83 | batch 2200/2304 [I 2022-10-27 19:17:52] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:00:57 [I 2022-10-27 19:17:52] ...trap/engines/engine.py.229: process: 0.55587 | load: 0.00245 [I 2022-10-27 19:17:52] ...trap/engines/engine.py.230: loss: 0.17718 [I 2022-10-27 19:17:52] ...optimizers/trijoint.py.151: total_norm: 21.895206 [I 2022-10-27 19:17:58] ...trap/engines/engine.py.224: train: epoch 83 | batch 2210/2304 [I 2022-10-27 19:17:58] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:00:53 [I 2022-10-27 19:17:58] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00247 [I 2022-10-27 19:17:58] ...trap/engines/engine.py.230: loss: 0.11457 [I 2022-10-27 19:17:58] ...optimizers/trijoint.py.151: total_norm: 14.447545 [I 2022-10-27 19:18:03] ...trap/engines/engine.py.224: train: epoch 83 | batch 2220/2304 [I 2022-10-27 19:18:03] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:00:46 [I 2022-10-27 19:18:03] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00240 [I 2022-10-27 19:18:03] ...trap/engines/engine.py.230: loss: 0.17638 [I 2022-10-27 19:18:03] ...optimizers/trijoint.py.151: total_norm: 15.464368 [I 2022-10-27 19:18:09] ...trap/engines/engine.py.224: train: epoch 83 | batch 2230/2304 [I 2022-10-27 19:18:09] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:41 [I 2022-10-27 19:18:09] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00242 [I 2022-10-27 19:18:09] ...trap/engines/engine.py.230: loss: 0.09455 [I 2022-10-27 19:18:09] ...optimizers/trijoint.py.151: total_norm: 14.731542 [I 2022-10-27 19:18:14] ...trap/engines/engine.py.224: train: epoch 83 | batch 2240/2304 [I 2022-10-27 19:18:14] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:35 [I 2022-10-27 19:18:14] ...trap/engines/engine.py.229: process: 0.56462 | load: 0.00242 [I 2022-10-27 19:18:14] ...trap/engines/engine.py.230: loss: 0.11114 [I 2022-10-27 19:18:14] ...optimizers/trijoint.py.151: total_norm: 20.456709 [I 2022-10-27 19:18:20] ...trap/engines/engine.py.224: train: epoch 83 | batch 2250/2304 [I 2022-10-27 19:18:20] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:29 [I 2022-10-27 19:18:20] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00242 [I 2022-10-27 19:18:20] ...trap/engines/engine.py.230: loss: 0.12930 [I 2022-10-27 19:18:20] ...optimizers/trijoint.py.151: total_norm: 14.260703 [I 2022-10-27 19:18:25] ...trap/engines/engine.py.224: train: epoch 83 | batch 2260/2304 [I 2022-10-27 19:18:25] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:24 [I 2022-10-27 19:18:25] ...trap/engines/engine.py.229: process: 0.56566 | load: 0.00241 [I 2022-10-27 19:18:25] ...trap/engines/engine.py.230: loss: 0.14495 [I 2022-10-27 19:18:25] ...optimizers/trijoint.py.151: total_norm: 10.935343 [I 2022-10-27 19:18:31] ...trap/engines/engine.py.224: train: epoch 83 | batch 2270/2304 [I 2022-10-27 19:18:31] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:18 [I 2022-10-27 19:18:31] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00249 [I 2022-10-27 19:18:31] ...trap/engines/engine.py.230: loss: 0.14317 [I 2022-10-27 19:18:31] ...optimizers/trijoint.py.151: total_norm: 24.424545 [I 2022-10-27 19:18:37] ...trap/engines/engine.py.224: train: epoch 83 | batch 2280/2304 [I 2022-10-27 19:18:37] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:13 [I 2022-10-27 19:18:37] ...trap/engines/engine.py.229: process: 0.55822 | load: 0.00244 [I 2022-10-27 19:18:37] ...trap/engines/engine.py.230: loss: 0.13349 [I 2022-10-27 19:18:37] ...optimizers/trijoint.py.151: total_norm: 70.438370 [I 2022-10-27 19:18:42] ...trap/engines/engine.py.224: train: epoch 83 | batch 2290/2304 [I 2022-10-27 19:18:42] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:07 [I 2022-10-27 19:18:42] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00245 [I 2022-10-27 19:18:42] ...trap/engines/engine.py.230: loss: 0.16791 [I 2022-10-27 19:18:42] ...optimizers/trijoint.py.151: total_norm: 14.107594 [I 2022-10-27 19:18:48] ...trap/engines/engine.py.224: train: epoch 83 | batch 2300/2304 [I 2022-10-27 19:18:48] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:02 [I 2022-10-27 19:18:48] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00015 [I 2022-10-27 19:18:48] ...trap/engines/engine.py.230: loss: 0.11190 [I 2022-10-27 19:18:48] ...optimizers/trijoint.py.151: total_norm: 20.468027 [I 2022-10-27 19:18:50] ...trap/engines/engine.py.224: train: epoch 83 | batch 2304/2304 [I 2022-10-27 19:18:50] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:00 [I 2022-10-27 19:18:50] ...trap/engines/engine.py.229: process: 0.54070 | load: 0.00015 [I 2022-10-27 19:18:50] ...trap/engines/engine.py.230: loss: 0.10392 [I 2022-10-27 19:18:50] ...optimizers/trijoint.py.151: total_norm: 12.155768 [S 2022-10-27 19:18:51] ...trap/engines/engine.py.129: train_epoch.epoch: 83 [S 2022-10-27 19:18:51] ...trap/engines/engine.py.129: train_epoch.loss: 0.13481527759234974 [I 2022-10-27 19:18:51] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 19:18:52] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 83 [I 2022-10-27 19:18:52] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 19:18:54] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 19:18:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 19:18:54] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 19:18:57] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 19:18:57] ...trap/engines/engine.py.321: eval: epoch 83 | batch 0/510 [I 2022-10-27 19:18:57] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:44:07 [I 2022-10-27 19:18:57] ...trap/engines/engine.py.326: process: 5.19131 | load: 3.72141 [I 2022-10-27 19:19:01] ...trap/engines/engine.py.321: eval: epoch 83 | batch 10/510 [I 2022-10-27 19:19:01] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:08 [I 2022-10-27 19:19:01] ...trap/engines/engine.py.326: process: 0.32465 | load: 0.00020 [I 2022-10-27 19:19:04] ...trap/engines/engine.py.321: eval: epoch 83 | batch 20/510 [I 2022-10-27 19:19:04] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:10 [I 2022-10-27 19:19:04] ...trap/engines/engine.py.326: process: 0.33545 | load: 0.00019 [I 2022-10-27 19:19:07] ...trap/engines/engine.py.321: eval: epoch 83 | batch 30/510 [I 2022-10-27 19:19:07] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:45 [I 2022-10-27 19:19:07] ...trap/engines/engine.py.326: process: 0.32528 | load: 0.00019 [I 2022-10-27 19:19:11] ...trap/engines/engine.py.321: eval: epoch 83 | batch 40/510 [I 2022-10-27 19:19:11] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:41 [I 2022-10-27 19:19:11] ...trap/engines/engine.py.326: process: 0.35122 | load: 0.00017 [I 2022-10-27 19:19:14] ...trap/engines/engine.py.321: eval: epoch 83 | batch 50/510 [I 2022-10-27 19:19:14] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:32 [I 2022-10-27 19:19:14] ...trap/engines/engine.py.326: process: 0.32953 | load: 0.00017 [I 2022-10-27 19:19:18] ...trap/engines/engine.py.321: eval: epoch 83 | batch 60/510 [I 2022-10-27 19:19:18] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:30 [I 2022-10-27 19:19:18] ...trap/engines/engine.py.326: process: 0.33407 | load: 0.00018 [I 2022-10-27 19:19:21] ...trap/engines/engine.py.321: eval: epoch 83 | batch 70/510 [I 2022-10-27 19:19:21] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:26 [I 2022-10-27 19:19:21] ...trap/engines/engine.py.326: process: 0.31876 | load: 0.00018 [I 2022-10-27 19:19:24] ...trap/engines/engine.py.321: eval: epoch 83 | batch 80/510 [I 2022-10-27 19:19:24] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:21 [I 2022-10-27 19:19:24] ...trap/engines/engine.py.326: process: 0.32654 | load: 0.00018 [I 2022-10-27 19:19:28] ...trap/engines/engine.py.321: eval: epoch 83 | batch 90/510 [I 2022-10-27 19:19:28] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:20 [I 2022-10-27 19:19:28] ...trap/engines/engine.py.326: process: 0.33882 | load: 0.00016 [I 2022-10-27 19:19:31] ...trap/engines/engine.py.321: eval: epoch 83 | batch 100/510 [I 2022-10-27 19:19:31] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:12 [I 2022-10-27 19:19:31] ...trap/engines/engine.py.326: process: 0.29125 | load: 0.00020 [I 2022-10-27 19:19:34] ...trap/engines/engine.py.321: eval: epoch 83 | batch 110/510 [I 2022-10-27 19:19:34] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 19:19:34] ...trap/engines/engine.py.326: process: 0.28984 | load: 0.00018 [I 2022-10-27 19:19:37] ...trap/engines/engine.py.321: eval: epoch 83 | batch 120/510 [I 2022-10-27 19:19:37] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:53 [I 2022-10-27 19:19:37] ...trap/engines/engine.py.326: process: 0.28366 | load: 0.00021 [I 2022-10-27 19:19:40] ...trap/engines/engine.py.321: eval: epoch 83 | batch 130/510 [I 2022-10-27 19:19:40] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:50 [I 2022-10-27 19:19:40] ...trap/engines/engine.py.326: process: 0.28942 | load: 0.00021 [I 2022-10-27 19:19:42] ...trap/engines/engine.py.321: eval: epoch 83 | batch 140/510 [I 2022-10-27 19:19:42] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-27 19:19:42] ...trap/engines/engine.py.326: process: 0.29321 | load: 0.00016 [I 2022-10-27 19:19:45] ...trap/engines/engine.py.321: eval: epoch 83 | batch 150/510 [I 2022-10-27 19:19:45] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-27 19:19:45] ...trap/engines/engine.py.326: process: 0.29308 | load: 0.00016 [I 2022-10-27 19:19:48] ...trap/engines/engine.py.321: eval: epoch 83 | batch 160/510 [I 2022-10-27 19:19:48] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:42 [I 2022-10-27 19:19:48] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00017 [I 2022-10-27 19:19:51] ...trap/engines/engine.py.321: eval: epoch 83 | batch 170/510 [I 2022-10-27 19:19:51] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:39 [I 2022-10-27 19:19:51] ...trap/engines/engine.py.326: process: 0.30269 | load: 0.00017 [I 2022-10-27 19:19:54] ...trap/engines/engine.py.321: eval: epoch 83 | batch 180/510 [I 2022-10-27 19:19:54] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 19:19:54] ...trap/engines/engine.py.326: process: 0.28479 | load: 0.00016 [I 2022-10-27 19:19:57] ...trap/engines/engine.py.321: eval: epoch 83 | batch 190/510 [I 2022-10-27 19:19:57] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:33 [I 2022-10-27 19:19:57] ...trap/engines/engine.py.326: process: 0.29334 | load: 0.00020 [I 2022-10-27 19:20:00] ...trap/engines/engine.py.321: eval: epoch 83 | batch 200/510 [I 2022-10-27 19:20:00] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:30 [I 2022-10-27 19:20:00] ...trap/engines/engine.py.326: process: 0.29086 | load: 0.00015 [I 2022-10-27 19:20:03] ...trap/engines/engine.py.321: eval: epoch 83 | batch 210/510 [I 2022-10-27 19:20:03] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 19:20:03] ...trap/engines/engine.py.326: process: 0.28761 | load: 0.00018 [I 2022-10-27 19:20:06] ...trap/engines/engine.py.321: eval: epoch 83 | batch 220/510 [I 2022-10-27 19:20:06] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:26 [I 2022-10-27 19:20:06] ...trap/engines/engine.py.326: process: 0.28889 | load: 0.00016 [I 2022-10-27 19:20:09] ...trap/engines/engine.py.321: eval: epoch 83 | batch 230/510 [I 2022-10-27 19:20:09] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 19:20:09] ...trap/engines/engine.py.326: process: 0.28912 | load: 0.00016 [I 2022-10-27 19:20:11] ...trap/engines/engine.py.321: eval: epoch 83 | batch 240/510 [I 2022-10-27 19:20:11] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-27 19:20:11] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00018 [I 2022-10-27 19:20:14] ...trap/engines/engine.py.321: eval: epoch 83 | batch 250/510 [I 2022-10-27 19:20:14] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-27 19:20:14] ...trap/engines/engine.py.326: process: 0.28332 | load: 0.00018 [I 2022-10-27 19:20:17] ...trap/engines/engine.py.321: eval: epoch 83 | batch 260/510 [I 2022-10-27 19:20:17] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:11 [I 2022-10-27 19:20:17] ...trap/engines/engine.py.326: process: 0.28447 | load: 0.00022 [I 2022-10-27 19:20:20] ...trap/engines/engine.py.321: eval: epoch 83 | batch 270/510 [I 2022-10-27 19:20:20] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 19:20:20] ...trap/engines/engine.py.326: process: 0.28795 | load: 0.00025 [I 2022-10-27 19:20:23] ...trap/engines/engine.py.321: eval: epoch 83 | batch 280/510 [I 2022-10-27 19:20:23] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:07 [I 2022-10-27 19:20:23] ...trap/engines/engine.py.326: process: 0.29411 | load: 0.00017 [I 2022-10-27 19:20:26] ...trap/engines/engine.py.321: eval: epoch 83 | batch 290/510 [I 2022-10-27 19:20:26] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 19:20:26] ...trap/engines/engine.py.326: process: 0.28200 | load: 0.00017 [I 2022-10-27 19:20:29] ...trap/engines/engine.py.321: eval: epoch 83 | batch 300/510 [I 2022-10-27 19:20:29] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 19:20:29] ...trap/engines/engine.py.326: process: 0.28754 | load: 0.00015 [I 2022-10-27 19:20:32] ...trap/engines/engine.py.321: eval: epoch 83 | batch 310/510 [I 2022-10-27 19:20:32] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 19:20:32] ...trap/engines/engine.py.326: process: 0.28924 | load: 0.00016 [I 2022-10-27 19:20:35] ...trap/engines/engine.py.321: eval: epoch 83 | batch 320/510 [I 2022-10-27 19:20:35] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 19:20:35] ...trap/engines/engine.py.326: process: 0.28606 | load: 0.00017 [I 2022-10-27 19:20:37] ...trap/engines/engine.py.321: eval: epoch 83 | batch 330/510 [I 2022-10-27 19:20:37] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-27 19:20:37] ...trap/engines/engine.py.326: process: 0.28742 | load: 0.00017 [I 2022-10-27 19:20:40] ...trap/engines/engine.py.321: eval: epoch 83 | batch 340/510 [I 2022-10-27 19:20:40] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-27 19:20:40] ...trap/engines/engine.py.326: process: 0.28440 | load: 0.00023 [I 2022-10-27 19:20:43] ...trap/engines/engine.py.321: eval: epoch 83 | batch 350/510 [I 2022-10-27 19:20:43] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 19:20:43] ...trap/engines/engine.py.326: process: 0.29320 | load: 0.00017 [I 2022-10-27 19:20:46] ...trap/engines/engine.py.321: eval: epoch 83 | batch 360/510 [I 2022-10-27 19:20:46] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 19:20:46] ...trap/engines/engine.py.326: process: 0.29145 | load: 0.00017 [I 2022-10-27 19:20:49] ...trap/engines/engine.py.321: eval: epoch 83 | batch 370/510 [I 2022-10-27 19:20:49] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 19:20:49] ...trap/engines/engine.py.326: process: 0.28289 | load: 0.00023 [I 2022-10-27 19:20:52] ...trap/engines/engine.py.321: eval: epoch 83 | batch 380/510 [I 2022-10-27 19:20:52] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 19:20:52] ...trap/engines/engine.py.326: process: 0.28581 | load: 0.00017 [I 2022-10-27 19:20:55] ...trap/engines/engine.py.321: eval: epoch 83 | batch 390/510 [I 2022-10-27 19:20:55] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 19:20:55] ...trap/engines/engine.py.326: process: 0.28842 | load: 0.00014 [I 2022-10-27 19:20:58] ...trap/engines/engine.py.321: eval: epoch 83 | batch 400/510 [I 2022-10-27 19:20:58] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 19:20:58] ...trap/engines/engine.py.326: process: 0.29161 | load: 0.00018 [I 2022-10-27 19:21:00] ...trap/engines/engine.py.321: eval: epoch 83 | batch 410/510 [I 2022-10-27 19:21:01] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 19:21:01] ...trap/engines/engine.py.326: process: 0.28763 | load: 0.00015 [I 2022-10-27 19:21:03] ...trap/engines/engine.py.321: eval: epoch 83 | batch 420/510 [I 2022-10-27 19:21:03] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 19:21:03] ...trap/engines/engine.py.326: process: 0.28824 | load: 0.00016 [I 2022-10-27 19:21:06] ...trap/engines/engine.py.321: eval: epoch 83 | batch 430/510 [I 2022-10-27 19:21:06] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 19:21:06] ...trap/engines/engine.py.326: process: 0.29180 | load: 0.00018 [I 2022-10-27 19:21:09] ...trap/engines/engine.py.321: eval: epoch 83 | batch 440/510 [I 2022-10-27 19:21:09] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 19:21:09] ...trap/engines/engine.py.326: process: 0.29261 | load: 0.00016 [I 2022-10-27 19:21:12] ...trap/engines/engine.py.321: eval: epoch 83 | batch 450/510 [I 2022-10-27 19:21:12] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 19:21:12] ...trap/engines/engine.py.326: process: 0.28873 | load: 0.00026 [I 2022-10-27 19:21:15] ...trap/engines/engine.py.321: eval: epoch 83 | batch 460/510 [I 2022-10-27 19:21:15] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 19:21:15] ...trap/engines/engine.py.326: process: 0.28788 | load: 0.00019 [I 2022-10-27 19:21:18] ...trap/engines/engine.py.321: eval: epoch 83 | batch 470/510 [I 2022-10-27 19:21:18] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 19:21:18] ...trap/engines/engine.py.326: process: 0.28701 | load: 0.00018 [I 2022-10-27 19:21:21] ...trap/engines/engine.py.321: eval: epoch 83 | batch 480/510 [I 2022-10-27 19:21:21] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 19:21:21] ...trap/engines/engine.py.326: process: 0.29077 | load: 0.00019 [I 2022-10-27 19:21:24] ...trap/engines/engine.py.321: eval: epoch 83 | batch 490/510 [I 2022-10-27 19:21:24] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 19:21:24] ...trap/engines/engine.py.326: process: 0.27805 | load: 0.00015 [I 2022-10-27 19:21:26] ...trap/engines/engine.py.321: eval: epoch 83 | batch 500/510 [I 2022-10-27 19:21:26] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 19:21:26] ...trap/engines/engine.py.326: process: 0.28353 | load: 0.00015 [I 2022-10-27 19:21:29] ...trap/engines/engine.py.321: eval: epoch 83 | batch 510/510 [I 2022-10-27 19:21:29] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 19:21:29] ...trap/engines/engine.py.326: process: 0.28944 | load: 0.00011 [S 2022-10-27 19:21:30] ...trap/engines/engine.py.132: eval_epoch.epoch: 83 [I 2022-10-27 19:21:30] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.730000 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905800 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936200 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.725400 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.902900 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932800 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016917 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010998 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009816 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016645 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012103 [S 2022-10-27 19:21:34] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.011125 [I 2022-10-27 19:21:36] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 19:21:36] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 19:21:36] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 19:21:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 19:21:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 19:21:40] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 19:21:41] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 19:21:47] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 19:21:47] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 19:21:47] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 19:21:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 19:21:56] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 19:21:56] ...trap/engines/engine.py.163: Training model on trainset for epoch 84 [I 2022-10-27 19:21:56] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 19:21:59] ...trap/engines/engine.py.224: train: epoch 84 | batch 0/2304 [I 2022-10-27 19:21:59] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:07 [I 2022-10-27 19:21:59] ...trap/engines/engine.py.229: process: 2.71170 | load: 2.08892 [I 2022-10-27 19:21:59] ...trap/engines/engine.py.230: loss: 0.13044 [I 2022-10-27 19:21:59] ...optimizers/trijoint.py.151: total_norm: 12.336547 [I 2022-10-27 19:22:04] ...trap/engines/engine.py.224: train: epoch 84 | batch 10/2304 [I 2022-10-27 19:22:04] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:03 [I 2022-10-27 19:22:04] ...trap/engines/engine.py.229: process: 0.54814 | load: 0.00452 [I 2022-10-27 19:22:04] ...trap/engines/engine.py.230: loss: 0.09385 [I 2022-10-27 19:22:04] ...optimizers/trijoint.py.151: total_norm: 10.368544 [I 2022-10-27 19:22:10] ...trap/engines/engine.py.224: train: epoch 84 | batch 20/2304 [I 2022-10-27 19:22:10] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:05 [I 2022-10-27 19:22:10] ...trap/engines/engine.py.229: process: 0.55656 | load: 0.00367 [I 2022-10-27 19:22:10] ...trap/engines/engine.py.230: loss: 0.10089 [I 2022-10-27 19:22:10] ...optimizers/trijoint.py.151: total_norm: 12.297927 [I 2022-10-27 19:22:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 30/2304 [I 2022-10-27 19:22:15] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:21 [I 2022-10-27 19:22:15] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00301 [I 2022-10-27 19:22:15] ...trap/engines/engine.py.230: loss: 0.10194 [I 2022-10-27 19:22:15] ...optimizers/trijoint.py.151: total_norm: 14.629025 [I 2022-10-27 19:22:21] ...trap/engines/engine.py.224: train: epoch 84 | batch 40/2304 [I 2022-10-27 19:22:21] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:03 [I 2022-10-27 19:22:21] ...trap/engines/engine.py.229: process: 0.56201 | load: 0.00270 [I 2022-10-27 19:22:21] ...trap/engines/engine.py.230: loss: 0.19846 [I 2022-10-27 19:22:21] ...optimizers/trijoint.py.151: total_norm: 20.380152 [I 2022-10-27 19:22:27] ...trap/engines/engine.py.224: train: epoch 84 | batch 50/2304 [I 2022-10-27 19:22:27] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:48 [I 2022-10-27 19:22:27] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00278 [I 2022-10-27 19:22:27] ...trap/engines/engine.py.230: loss: 0.10237 [I 2022-10-27 19:22:27] ...optimizers/trijoint.py.151: total_norm: 12.895685 [I 2022-10-27 19:22:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 60/2304 [I 2022-10-27 19:22:32] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:47 [I 2022-10-27 19:22:32] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00287 [I 2022-10-27 19:22:32] ...trap/engines/engine.py.230: loss: 0.10892 [I 2022-10-27 19:22:32] ...optimizers/trijoint.py.151: total_norm: 33.396637 [I 2022-10-27 19:22:38] ...trap/engines/engine.py.224: train: epoch 84 | batch 70/2304 [I 2022-10-27 19:22:38] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:38 [I 2022-10-27 19:22:38] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00258 [I 2022-10-27 19:22:38] ...trap/engines/engine.py.230: loss: 0.15405 [I 2022-10-27 19:22:38] ...optimizers/trijoint.py.151: total_norm: 15.862041 [I 2022-10-27 19:22:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 80/2304 [I 2022-10-27 19:22:43] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:47 [I 2022-10-27 19:22:43] ...trap/engines/engine.py.229: process: 0.55828 | load: 0.00292 [I 2022-10-27 19:22:43] ...trap/engines/engine.py.230: loss: 0.10473 [I 2022-10-27 19:22:43] ...optimizers/trijoint.py.151: total_norm: 12.615613 [I 2022-10-27 19:22:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 90/2304 [I 2022-10-27 19:22:49] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:34 [I 2022-10-27 19:22:49] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00268 [I 2022-10-27 19:22:49] ...trap/engines/engine.py.230: loss: 0.18776 [I 2022-10-27 19:22:49] ...optimizers/trijoint.py.151: total_norm: 19.825865 [I 2022-10-27 19:22:55] ...trap/engines/engine.py.224: train: epoch 84 | batch 100/2304 [I 2022-10-27 19:22:55] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:32 [I 2022-10-27 19:22:55] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00253 [I 2022-10-27 19:22:55] ...trap/engines/engine.py.230: loss: 0.14974 [I 2022-10-27 19:22:55] ...optimizers/trijoint.py.151: total_norm: 24.598246 [I 2022-10-27 19:23:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 110/2304 [I 2022-10-27 19:23:00] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:23 [I 2022-10-27 19:23:00] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00253 [I 2022-10-27 19:23:00] ...trap/engines/engine.py.230: loss: 0.15818 [I 2022-10-27 19:23:00] ...optimizers/trijoint.py.151: total_norm: 13.620920 [I 2022-10-27 19:23:06] ...trap/engines/engine.py.224: train: epoch 84 | batch 120/2304 [I 2022-10-27 19:23:06] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:11 [I 2022-10-27 19:23:06] ...trap/engines/engine.py.229: process: 0.54712 | load: 0.00276 [I 2022-10-27 19:23:06] ...trap/engines/engine.py.230: loss: 0.15857 [I 2022-10-27 19:23:06] ...optimizers/trijoint.py.151: total_norm: 19.283123 [I 2022-10-27 19:23:11] ...trap/engines/engine.py.224: train: epoch 84 | batch 130/2304 [I 2022-10-27 19:23:11] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:19:57 [I 2022-10-27 19:23:11] ...trap/engines/engine.py.229: process: 0.54072 | load: 0.00246 [I 2022-10-27 19:23:11] ...trap/engines/engine.py.230: loss: 0.14034 [I 2022-10-27 19:23:11] ...optimizers/trijoint.py.151: total_norm: 23.280304 [I 2022-10-27 19:23:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 140/2304 [I 2022-10-27 19:23:17] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:19:57 [I 2022-10-27 19:23:17] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00253 [I 2022-10-27 19:23:17] ...trap/engines/engine.py.230: loss: 0.10354 [I 2022-10-27 19:23:17] ...optimizers/trijoint.py.151: total_norm: 10.884590 [I 2022-10-27 19:23:22] ...trap/engines/engine.py.224: train: epoch 84 | batch 150/2304 [I 2022-10-27 19:23:22] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:08 [I 2022-10-27 19:23:22] ...trap/engines/engine.py.229: process: 0.56287 | load: 0.00257 [I 2022-10-27 19:23:22] ...trap/engines/engine.py.230: loss: 0.15561 [I 2022-10-27 19:23:22] ...optimizers/trijoint.py.151: total_norm: 17.526905 [I 2022-10-27 19:23:28] ...trap/engines/engine.py.224: train: epoch 84 | batch 160/2304 [I 2022-10-27 19:23:28] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:19:49 [I 2022-10-27 19:23:28] ...trap/engines/engine.py.229: process: 0.54566 | load: 0.00269 [I 2022-10-27 19:23:28] ...trap/engines/engine.py.230: loss: 0.14819 [I 2022-10-27 19:23:28] ...optimizers/trijoint.py.151: total_norm: 49.893143 [I 2022-10-27 19:23:34] ...trap/engines/engine.py.224: train: epoch 84 | batch 170/2304 [I 2022-10-27 19:23:34] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:42 [I 2022-10-27 19:23:34] ...trap/engines/engine.py.229: process: 0.54859 | load: 0.00247 [I 2022-10-27 19:23:34] ...trap/engines/engine.py.230: loss: 0.11930 [I 2022-10-27 19:23:34] ...optimizers/trijoint.py.151: total_norm: 11.235618 [I 2022-10-27 19:23:39] ...trap/engines/engine.py.224: train: epoch 84 | batch 180/2304 [I 2022-10-27 19:23:39] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:51 [I 2022-10-27 19:23:39] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00266 [I 2022-10-27 19:23:39] ...trap/engines/engine.py.230: loss: 0.16887 [I 2022-10-27 19:23:39] ...optimizers/trijoint.py.151: total_norm: 23.990435 [I 2022-10-27 19:23:45] ...trap/engines/engine.py.224: train: epoch 84 | batch 190/2304 [I 2022-10-27 19:23:45] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:43 [I 2022-10-27 19:23:45] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00249 [I 2022-10-27 19:23:45] ...trap/engines/engine.py.230: loss: 0.14255 [I 2022-10-27 19:23:45] ...optimizers/trijoint.py.151: total_norm: 17.216772 [I 2022-10-27 19:23:50] ...trap/engines/engine.py.224: train: epoch 84 | batch 200/2304 [I 2022-10-27 19:23:50] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:37 [I 2022-10-27 19:23:50] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00268 [I 2022-10-27 19:23:50] ...trap/engines/engine.py.230: loss: 0.12434 [I 2022-10-27 19:23:50] ...optimizers/trijoint.py.151: total_norm: 15.509103 [I 2022-10-27 19:23:56] ...trap/engines/engine.py.224: train: epoch 84 | batch 210/2304 [I 2022-10-27 19:23:56] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:37 [I 2022-10-27 19:23:56] ...trap/engines/engine.py.229: process: 0.56827 | load: 0.00245 [I 2022-10-27 19:23:56] ...trap/engines/engine.py.230: loss: 0.18070 [I 2022-10-27 19:23:56] ...optimizers/trijoint.py.151: total_norm: 25.215622 [I 2022-10-27 19:24:02] ...trap/engines/engine.py.224: train: epoch 84 | batch 220/2304 [I 2022-10-27 19:24:02] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:22 [I 2022-10-27 19:24:02] ...trap/engines/engine.py.229: process: 0.55031 | load: 0.00281 [I 2022-10-27 19:24:02] ...trap/engines/engine.py.230: loss: 0.11765 [I 2022-10-27 19:24:02] ...optimizers/trijoint.py.151: total_norm: 25.594646 [I 2022-10-27 19:24:07] ...trap/engines/engine.py.224: train: epoch 84 | batch 230/2304 [I 2022-10-27 19:24:07] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:07 [I 2022-10-27 19:24:07] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00251 [I 2022-10-27 19:24:07] ...trap/engines/engine.py.230: loss: 0.10805 [I 2022-10-27 19:24:07] ...optimizers/trijoint.py.151: total_norm: 10.334122 [I 2022-10-27 19:24:13] ...trap/engines/engine.py.224: train: epoch 84 | batch 240/2304 [I 2022-10-27 19:24:13] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:02 [I 2022-10-27 19:24:13] ...trap/engines/engine.py.229: process: 0.54635 | load: 0.00257 [I 2022-10-27 19:24:13] ...trap/engines/engine.py.230: loss: 0.17259 [I 2022-10-27 19:24:13] ...optimizers/trijoint.py.151: total_norm: 18.619797 [I 2022-10-27 19:24:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 250/2304 [I 2022-10-27 19:24:18] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:01 [I 2022-10-27 19:24:18] ...trap/engines/engine.py.229: process: 0.55212 | load: 0.00248 [I 2022-10-27 19:24:18] ...trap/engines/engine.py.230: loss: 0.11869 [I 2022-10-27 19:24:18] ...optimizers/trijoint.py.151: total_norm: 13.454471 [I 2022-10-27 19:24:24] ...trap/engines/engine.py.224: train: epoch 84 | batch 260/2304 [I 2022-10-27 19:24:24] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:58 [I 2022-10-27 19:24:24] ...trap/engines/engine.py.229: process: 0.57147 | load: 0.00247 [I 2022-10-27 19:24:24] ...trap/engines/engine.py.230: loss: 0.09403 [I 2022-10-27 19:24:24] ...optimizers/trijoint.py.151: total_norm: 41.546257 [I 2022-10-27 19:24:29] ...trap/engines/engine.py.224: train: epoch 84 | batch 270/2304 [I 2022-10-27 19:24:29] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:55 [I 2022-10-27 19:24:29] ...trap/engines/engine.py.229: process: 0.56515 | load: 0.00257 [I 2022-10-27 19:24:29] ...trap/engines/engine.py.230: loss: 0.16751 [I 2022-10-27 19:24:29] ...optimizers/trijoint.py.151: total_norm: 14.536672 [I 2022-10-27 19:24:35] ...trap/engines/engine.py.224: train: epoch 84 | batch 280/2304 [I 2022-10-27 19:24:35] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:50 [I 2022-10-27 19:24:35] ...trap/engines/engine.py.229: process: 0.56225 | load: 0.00251 [I 2022-10-27 19:24:35] ...trap/engines/engine.py.230: loss: 0.14887 [I 2022-10-27 19:24:35] ...optimizers/trijoint.py.151: total_norm: 13.181385 [I 2022-10-27 19:24:41] ...trap/engines/engine.py.224: train: epoch 84 | batch 290/2304 [I 2022-10-27 19:24:41] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:45 [I 2022-10-27 19:24:41] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00249 [I 2022-10-27 19:24:41] ...trap/engines/engine.py.230: loss: 0.12356 [I 2022-10-27 19:24:41] ...optimizers/trijoint.py.151: total_norm: 13.954309 [I 2022-10-27 19:24:46] ...trap/engines/engine.py.224: train: epoch 84 | batch 300/2304 [I 2022-10-27 19:24:46] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:37 [I 2022-10-27 19:24:46] ...trap/engines/engine.py.229: process: 0.54843 | load: 0.00252 [I 2022-10-27 19:24:46] ...trap/engines/engine.py.230: loss: 0.12711 [I 2022-10-27 19:24:46] ...optimizers/trijoint.py.151: total_norm: 24.046757 [I 2022-10-27 19:24:52] ...trap/engines/engine.py.224: train: epoch 84 | batch 310/2304 [I 2022-10-27 19:24:52] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:31 [I 2022-10-27 19:24:52] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00253 [I 2022-10-27 19:24:52] ...trap/engines/engine.py.230: loss: 0.11827 [I 2022-10-27 19:24:52] ...optimizers/trijoint.py.151: total_norm: 15.610696 [I 2022-10-27 19:24:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 320/2304 [I 2022-10-27 19:24:57] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:31 [I 2022-10-27 19:24:57] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00251 [I 2022-10-27 19:24:57] ...trap/engines/engine.py.230: loss: 0.11124 [I 2022-10-27 19:24:57] ...optimizers/trijoint.py.151: total_norm: 15.386130 [I 2022-10-27 19:25:03] ...trap/engines/engine.py.224: train: epoch 84 | batch 330/2304 [I 2022-10-27 19:25:03] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:18 [I 2022-10-27 19:25:03] ...trap/engines/engine.py.229: process: 0.55246 | load: 0.00244 [I 2022-10-27 19:25:03] ...trap/engines/engine.py.230: loss: 0.12658 [I 2022-10-27 19:25:03] ...optimizers/trijoint.py.151: total_norm: 18.064478 [I 2022-10-27 19:25:09] ...trap/engines/engine.py.224: train: epoch 84 | batch 340/2304 [I 2022-10-27 19:25:09] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:17 [I 2022-10-27 19:25:09] ...trap/engines/engine.py.229: process: 0.55623 | load: 0.00260 [I 2022-10-27 19:25:09] ...trap/engines/engine.py.230: loss: 0.10862 [I 2022-10-27 19:25:09] ...optimizers/trijoint.py.151: total_norm: 11.144731 [I 2022-10-27 19:25:14] ...trap/engines/engine.py.224: train: epoch 84 | batch 350/2304 [I 2022-10-27 19:25:14] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:05 [I 2022-10-27 19:25:14] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00246 [I 2022-10-27 19:25:14] ...trap/engines/engine.py.230: loss: 0.11402 [I 2022-10-27 19:25:14] ...optimizers/trijoint.py.151: total_norm: 15.758620 [I 2022-10-27 19:25:20] ...trap/engines/engine.py.224: train: epoch 84 | batch 360/2304 [I 2022-10-27 19:25:20] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:59 [I 2022-10-27 19:25:20] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00258 [I 2022-10-27 19:25:20] ...trap/engines/engine.py.230: loss: 0.08758 [I 2022-10-27 19:25:20] ...optimizers/trijoint.py.151: total_norm: 15.141987 [I 2022-10-27 19:25:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 370/2304 [I 2022-10-27 19:25:25] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:57 [I 2022-10-27 19:25:25] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00248 [I 2022-10-27 19:25:25] ...trap/engines/engine.py.230: loss: 0.14288 [I 2022-10-27 19:25:25] ...optimizers/trijoint.py.151: total_norm: 11.764053 [I 2022-10-27 19:25:31] ...trap/engines/engine.py.224: train: epoch 84 | batch 380/2304 [I 2022-10-27 19:25:31] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:18:02 [I 2022-10-27 19:25:31] ...trap/engines/engine.py.229: process: 0.56350 | load: 0.00247 [I 2022-10-27 19:25:31] ...trap/engines/engine.py.230: loss: 0.10244 [I 2022-10-27 19:25:31] ...optimizers/trijoint.py.151: total_norm: 12.866597 [I 2022-10-27 19:25:37] ...trap/engines/engine.py.224: train: epoch 84 | batch 390/2304 [I 2022-10-27 19:25:37] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:47 [I 2022-10-27 19:25:37] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00245 [I 2022-10-27 19:25:37] ...trap/engines/engine.py.230: loss: 0.16408 [I 2022-10-27 19:25:37] ...optimizers/trijoint.py.151: total_norm: 46.604053 [I 2022-10-27 19:25:42] ...trap/engines/engine.py.224: train: epoch 84 | batch 400/2304 [I 2022-10-27 19:25:42] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:36 [I 2022-10-27 19:25:42] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00255 [I 2022-10-27 19:25:42] ...trap/engines/engine.py.230: loss: 0.14779 [I 2022-10-27 19:25:42] ...optimizers/trijoint.py.151: total_norm: 11.356644 [I 2022-10-27 19:25:48] ...trap/engines/engine.py.224: train: epoch 84 | batch 410/2304 [I 2022-10-27 19:25:48] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:36 [I 2022-10-27 19:25:48] ...trap/engines/engine.py.229: process: 0.55941 | load: 0.00243 [I 2022-10-27 19:25:48] ...trap/engines/engine.py.230: loss: 0.10331 [I 2022-10-27 19:25:48] ...optimizers/trijoint.py.151: total_norm: 11.621138 [I 2022-10-27 19:25:53] ...trap/engines/engine.py.224: train: epoch 84 | batch 420/2304 [I 2022-10-27 19:25:53] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:31 [I 2022-10-27 19:25:53] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00245 [I 2022-10-27 19:25:53] ...trap/engines/engine.py.230: loss: 0.11503 [I 2022-10-27 19:25:53] ...optimizers/trijoint.py.151: total_norm: 8.315055 [I 2022-10-27 19:25:59] ...trap/engines/engine.py.224: train: epoch 84 | batch 430/2304 [I 2022-10-27 19:25:59] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:27 [I 2022-10-27 19:25:59] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00241 [I 2022-10-27 19:25:59] ...trap/engines/engine.py.230: loss: 0.12347 [I 2022-10-27 19:25:59] ...optimizers/trijoint.py.151: total_norm: 10.487529 [I 2022-10-27 19:26:05] ...trap/engines/engine.py.224: train: epoch 84 | batch 440/2304 [I 2022-10-27 19:26:05] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:17 [I 2022-10-27 19:26:05] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00252 [I 2022-10-27 19:26:05] ...trap/engines/engine.py.230: loss: 0.17498 [I 2022-10-27 19:26:05] ...optimizers/trijoint.py.151: total_norm: 13.386869 [I 2022-10-27 19:26:10] ...trap/engines/engine.py.224: train: epoch 84 | batch 450/2304 [I 2022-10-27 19:26:10] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:19 [I 2022-10-27 19:26:10] ...trap/engines/engine.py.229: process: 0.56393 | load: 0.00248 [I 2022-10-27 19:26:10] ...trap/engines/engine.py.230: loss: 0.10579 [I 2022-10-27 19:26:10] ...optimizers/trijoint.py.151: total_norm: 17.964159 [I 2022-10-27 19:26:16] ...trap/engines/engine.py.224: train: epoch 84 | batch 460/2304 [I 2022-10-27 19:26:16] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:05 [I 2022-10-27 19:26:16] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00258 [I 2022-10-27 19:26:16] ...trap/engines/engine.py.230: loss: 0.13511 [I 2022-10-27 19:26:16] ...optimizers/trijoint.py.151: total_norm: 13.019288 [I 2022-10-27 19:26:21] ...trap/engines/engine.py.224: train: epoch 84 | batch 470/2304 [I 2022-10-27 19:26:21] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:39 [I 2022-10-27 19:26:21] ...trap/engines/engine.py.229: process: 0.57674 | load: 0.00244 [I 2022-10-27 19:26:21] ...trap/engines/engine.py.230: loss: 0.08929 [I 2022-10-27 19:26:21] ...optimizers/trijoint.py.151: total_norm: 9.337518 [I 2022-10-27 19:26:27] ...trap/engines/engine.py.224: train: epoch 84 | batch 480/2304 [I 2022-10-27 19:26:27] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:08 [I 2022-10-27 19:26:27] ...trap/engines/engine.py.229: process: 0.56353 | load: 0.00251 [I 2022-10-27 19:26:27] ...trap/engines/engine.py.230: loss: 0.19955 [I 2022-10-27 19:26:27] ...optimizers/trijoint.py.151: total_norm: 16.518585 [I 2022-10-27 19:26:33] ...trap/engines/engine.py.224: train: epoch 84 | batch 490/2304 [I 2022-10-27 19:26:33] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:56 [I 2022-10-27 19:26:33] ...trap/engines/engine.py.229: process: 0.56353 | load: 0.00246 [I 2022-10-27 19:26:33] ...trap/engines/engine.py.230: loss: 0.15012 [I 2022-10-27 19:26:33] ...optimizers/trijoint.py.151: total_norm: 14.794490 [I 2022-10-27 19:26:38] ...trap/engines/engine.py.224: train: epoch 84 | batch 500/2304 [I 2022-10-27 19:26:38] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:38 [I 2022-10-27 19:26:38] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00244 [I 2022-10-27 19:26:38] ...trap/engines/engine.py.230: loss: 0.14007 [I 2022-10-27 19:26:38] ...optimizers/trijoint.py.151: total_norm: 24.683271 [I 2022-10-27 19:26:44] ...trap/engines/engine.py.224: train: epoch 84 | batch 510/2304 [I 2022-10-27 19:26:44] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:45 [I 2022-10-27 19:26:44] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00252 [I 2022-10-27 19:26:44] ...trap/engines/engine.py.230: loss: 0.10764 [I 2022-10-27 19:26:44] ...optimizers/trijoint.py.151: total_norm: 8.269964 [I 2022-10-27 19:26:50] ...trap/engines/engine.py.224: train: epoch 84 | batch 520/2304 [I 2022-10-27 19:26:50] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:38 [I 2022-10-27 19:26:50] ...trap/engines/engine.py.229: process: 0.55829 | load: 0.00249 [I 2022-10-27 19:26:50] ...trap/engines/engine.py.230: loss: 0.10612 [I 2022-10-27 19:26:50] ...optimizers/trijoint.py.151: total_norm: 9.993543 [I 2022-10-27 19:26:55] ...trap/engines/engine.py.224: train: epoch 84 | batch 530/2304 [I 2022-10-27 19:26:55] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:30 [I 2022-10-27 19:26:55] ...trap/engines/engine.py.229: process: 0.54686 | load: 0.00248 [I 2022-10-27 19:26:55] ...trap/engines/engine.py.230: loss: 0.11895 [I 2022-10-27 19:26:55] ...optimizers/trijoint.py.151: total_norm: 10.599129 [I 2022-10-27 19:27:01] ...trap/engines/engine.py.224: train: epoch 84 | batch 540/2304 [I 2022-10-27 19:27:01] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:17 [I 2022-10-27 19:27:01] ...trap/engines/engine.py.229: process: 0.55471 | load: 0.00243 [I 2022-10-27 19:27:01] ...trap/engines/engine.py.230: loss: 0.21557 [I 2022-10-27 19:27:01] ...optimizers/trijoint.py.151: total_norm: 15.274690 [I 2022-10-27 19:27:06] ...trap/engines/engine.py.224: train: epoch 84 | batch 550/2304 [I 2022-10-27 19:27:06] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:12 [I 2022-10-27 19:27:06] ...trap/engines/engine.py.229: process: 0.53597 | load: 0.00242 [I 2022-10-27 19:27:06] ...trap/engines/engine.py.230: loss: 0.10635 [I 2022-10-27 19:27:06] ...optimizers/trijoint.py.151: total_norm: 15.376831 [I 2022-10-27 19:27:12] ...trap/engines/engine.py.224: train: epoch 84 | batch 560/2304 [I 2022-10-27 19:27:12] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:10 [I 2022-10-27 19:27:12] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00242 [I 2022-10-27 19:27:12] ...trap/engines/engine.py.230: loss: 0.17534 [I 2022-10-27 19:27:12] ...optimizers/trijoint.py.151: total_norm: 12.391213 [I 2022-10-27 19:27:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 570/2304 [I 2022-10-27 19:27:18] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:27 [I 2022-10-27 19:27:18] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00242 [I 2022-10-27 19:27:18] ...trap/engines/engine.py.230: loss: 0.13232 [I 2022-10-27 19:27:18] ...optimizers/trijoint.py.151: total_norm: 11.340358 [I 2022-10-27 19:27:23] ...trap/engines/engine.py.224: train: epoch 84 | batch 580/2304 [I 2022-10-27 19:27:23] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:04 [I 2022-10-27 19:27:23] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00252 [I 2022-10-27 19:27:23] ...trap/engines/engine.py.230: loss: 0.09151 [I 2022-10-27 19:27:23] ...optimizers/trijoint.py.151: total_norm: 8.759482 [I 2022-10-27 19:27:29] ...trap/engines/engine.py.224: train: epoch 84 | batch 590/2304 [I 2022-10-27 19:27:29] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:58 [I 2022-10-27 19:27:29] ...trap/engines/engine.py.229: process: 0.56535 | load: 0.00246 [I 2022-10-27 19:27:29] ...trap/engines/engine.py.230: loss: 0.09911 [I 2022-10-27 19:27:29] ...optimizers/trijoint.py.151: total_norm: 13.820759 [I 2022-10-27 19:27:34] ...trap/engines/engine.py.224: train: epoch 84 | batch 600/2304 [I 2022-10-27 19:27:34] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:43 [I 2022-10-27 19:27:34] ...trap/engines/engine.py.229: process: 0.53941 | load: 0.00242 [I 2022-10-27 19:27:34] ...trap/engines/engine.py.230: loss: 0.10883 [I 2022-10-27 19:27:34] ...optimizers/trijoint.py.151: total_norm: 13.443120 [I 2022-10-27 19:27:40] ...trap/engines/engine.py.224: train: epoch 84 | batch 610/2304 [I 2022-10-27 19:27:40] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:41 [I 2022-10-27 19:27:40] ...trap/engines/engine.py.229: process: 0.56342 | load: 0.00242 [I 2022-10-27 19:27:40] ...trap/engines/engine.py.230: loss: 0.12550 [I 2022-10-27 19:27:40] ...optimizers/trijoint.py.151: total_norm: 12.047136 [I 2022-10-27 19:27:45] ...trap/engines/engine.py.224: train: epoch 84 | batch 620/2304 [I 2022-10-27 19:27:45] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:37 [I 2022-10-27 19:27:45] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00248 [I 2022-10-27 19:27:45] ...trap/engines/engine.py.230: loss: 0.13123 [I 2022-10-27 19:27:46] ...optimizers/trijoint.py.151: total_norm: 18.994957 [I 2022-10-27 19:27:51] ...trap/engines/engine.py.224: train: epoch 84 | batch 630/2304 [I 2022-10-27 19:27:51] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:16:05 [I 2022-10-27 19:27:51] ...trap/engines/engine.py.229: process: 0.56945 | load: 0.00246 [I 2022-10-27 19:27:51] ...trap/engines/engine.py.230: loss: 0.18471 [I 2022-10-27 19:27:51] ...optimizers/trijoint.py.151: total_norm: 20.199169 [I 2022-10-27 19:27:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 640/2304 [I 2022-10-27 19:27:57] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:31 [I 2022-10-27 19:27:57] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00246 [I 2022-10-27 19:27:57] ...trap/engines/engine.py.230: loss: 0.14682 [I 2022-10-27 19:27:57] ...optimizers/trijoint.py.151: total_norm: 11.258559 [I 2022-10-27 19:28:02] ...trap/engines/engine.py.224: train: epoch 84 | batch 650/2304 [I 2022-10-27 19:28:02] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:20 [I 2022-10-27 19:28:02] ...trap/engines/engine.py.229: process: 0.55491 | load: 0.00245 [I 2022-10-27 19:28:02] ...trap/engines/engine.py.230: loss: 0.21194 [I 2022-10-27 19:28:02] ...optimizers/trijoint.py.151: total_norm: 12.916974 [I 2022-10-27 19:28:08] ...trap/engines/engine.py.224: train: epoch 84 | batch 660/2304 [I 2022-10-27 19:28:08] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:15 [I 2022-10-27 19:28:08] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00245 [I 2022-10-27 19:28:08] ...trap/engines/engine.py.230: loss: 0.16946 [I 2022-10-27 19:28:08] ...optimizers/trijoint.py.151: total_norm: 25.987597 [I 2022-10-27 19:28:14] ...trap/engines/engine.py.224: train: epoch 84 | batch 670/2304 [I 2022-10-27 19:28:14] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:13 [I 2022-10-27 19:28:14] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00245 [I 2022-10-27 19:28:14] ...trap/engines/engine.py.230: loss: 0.16357 [I 2022-10-27 19:28:14] ...optimizers/trijoint.py.151: total_norm: 31.172512 [I 2022-10-27 19:28:19] ...trap/engines/engine.py.224: train: epoch 84 | batch 680/2304 [I 2022-10-27 19:28:19] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:02 [I 2022-10-27 19:28:19] ...trap/engines/engine.py.229: process: 0.53957 | load: 0.00247 [I 2022-10-27 19:28:19] ...trap/engines/engine.py.230: loss: 0.09738 [I 2022-10-27 19:28:19] ...optimizers/trijoint.py.151: total_norm: 9.764438 [I 2022-10-27 19:28:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 690/2304 [I 2022-10-27 19:28:25] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:04 [I 2022-10-27 19:28:25] ...trap/engines/engine.py.229: process: 0.56561 | load: 0.00243 [I 2022-10-27 19:28:25] ...trap/engines/engine.py.230: loss: 0.11817 [I 2022-10-27 19:28:25] ...optimizers/trijoint.py.151: total_norm: 18.951826 [I 2022-10-27 19:28:30] ...trap/engines/engine.py.224: train: epoch 84 | batch 700/2304 [I 2022-10-27 19:28:30] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:59 [I 2022-10-27 19:28:30] ...trap/engines/engine.py.229: process: 0.57545 | load: 0.00247 [I 2022-10-27 19:28:30] ...trap/engines/engine.py.230: loss: 0.16421 [I 2022-10-27 19:28:30] ...optimizers/trijoint.py.151: total_norm: 31.501940 [I 2022-10-27 19:28:36] ...trap/engines/engine.py.224: train: epoch 84 | batch 710/2304 [I 2022-10-27 19:28:36] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:43 [I 2022-10-27 19:28:36] ...trap/engines/engine.py.229: process: 0.55041 | load: 0.00246 [I 2022-10-27 19:28:36] ...trap/engines/engine.py.230: loss: 0.10664 [I 2022-10-27 19:28:36] ...optimizers/trijoint.py.151: total_norm: 13.463587 [I 2022-10-27 19:28:42] ...trap/engines/engine.py.224: train: epoch 84 | batch 720/2304 [I 2022-10-27 19:28:42] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:43 [I 2022-10-27 19:28:42] ...trap/engines/engine.py.229: process: 0.55234 | load: 0.00245 [I 2022-10-27 19:28:42] ...trap/engines/engine.py.230: loss: 0.10950 [I 2022-10-27 19:28:42] ...optimizers/trijoint.py.151: total_norm: 51.173210 [I 2022-10-27 19:28:47] ...trap/engines/engine.py.224: train: epoch 84 | batch 730/2304 [I 2022-10-27 19:28:47] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:33 [I 2022-10-27 19:28:47] ...trap/engines/engine.py.229: process: 0.55219 | load: 0.00246 [I 2022-10-27 19:28:47] ...trap/engines/engine.py.230: loss: 0.11837 [I 2022-10-27 19:28:47] ...optimizers/trijoint.py.151: total_norm: 12.019790 [I 2022-10-27 19:28:53] ...trap/engines/engine.py.224: train: epoch 84 | batch 740/2304 [I 2022-10-27 19:28:53] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:53 [I 2022-10-27 19:28:53] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00244 [I 2022-10-27 19:28:53] ...trap/engines/engine.py.230: loss: 0.11494 [I 2022-10-27 19:28:53] ...optimizers/trijoint.py.151: total_norm: 11.449228 [I 2022-10-27 19:28:58] ...trap/engines/engine.py.224: train: epoch 84 | batch 750/2304 [I 2022-10-27 19:28:58] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:26 [I 2022-10-27 19:28:58] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00243 [I 2022-10-27 19:28:58] ...trap/engines/engine.py.230: loss: 0.16645 [I 2022-10-27 19:28:58] ...optimizers/trijoint.py.151: total_norm: 12.307463 [I 2022-10-27 19:29:04] ...trap/engines/engine.py.224: train: epoch 84 | batch 760/2304 [I 2022-10-27 19:29:04] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:22 [I 2022-10-27 19:29:04] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00241 [I 2022-10-27 19:29:04] ...trap/engines/engine.py.230: loss: 0.10212 [I 2022-10-27 19:29:04] ...optimizers/trijoint.py.151: total_norm: 16.521296 [I 2022-10-27 19:29:10] ...trap/engines/engine.py.224: train: epoch 84 | batch 770/2304 [I 2022-10-27 19:29:10] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:15 [I 2022-10-27 19:29:10] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00243 [I 2022-10-27 19:29:10] ...trap/engines/engine.py.230: loss: 0.16172 [I 2022-10-27 19:29:10] ...optimizers/trijoint.py.151: total_norm: 16.782444 [I 2022-10-27 19:29:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 780/2304 [I 2022-10-27 19:29:15] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:06 [I 2022-10-27 19:29:15] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00241 [I 2022-10-27 19:29:15] ...trap/engines/engine.py.230: loss: 0.18049 [I 2022-10-27 19:29:15] ...optimizers/trijoint.py.151: total_norm: 21.327106 [I 2022-10-27 19:29:21] ...trap/engines/engine.py.224: train: epoch 84 | batch 790/2304 [I 2022-10-27 19:29:21] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:13:59 [I 2022-10-27 19:29:21] ...trap/engines/engine.py.229: process: 0.54995 | load: 0.00240 [I 2022-10-27 19:29:21] ...trap/engines/engine.py.230: loss: 0.12335 [I 2022-10-27 19:29:21] ...optimizers/trijoint.py.151: total_norm: 19.529699 [I 2022-10-27 19:29:26] ...trap/engines/engine.py.224: train: epoch 84 | batch 800/2304 [I 2022-10-27 19:29:26] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:57 [I 2022-10-27 19:29:26] ...trap/engines/engine.py.229: process: 0.55344 | load: 0.00238 [I 2022-10-27 19:29:26] ...trap/engines/engine.py.230: loss: 0.11929 [I 2022-10-27 19:29:26] ...optimizers/trijoint.py.151: total_norm: 95.890594 [I 2022-10-27 19:29:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 810/2304 [I 2022-10-27 19:29:32] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:47 [I 2022-10-27 19:29:32] ...trap/engines/engine.py.229: process: 0.54009 | load: 0.00244 [I 2022-10-27 19:29:32] ...trap/engines/engine.py.230: loss: 0.15607 [I 2022-10-27 19:29:32] ...optimizers/trijoint.py.151: total_norm: 30.167858 [I 2022-10-27 19:29:37] ...trap/engines/engine.py.224: train: epoch 84 | batch 820/2304 [I 2022-10-27 19:29:37] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:44 [I 2022-10-27 19:29:37] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00243 [I 2022-10-27 19:29:37] ...trap/engines/engine.py.230: loss: 0.09479 [I 2022-10-27 19:29:37] ...optimizers/trijoint.py.151: total_norm: 10.580729 [I 2022-10-27 19:29:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 830/2304 [I 2022-10-27 19:29:43] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:41 [I 2022-10-27 19:29:43] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00241 [I 2022-10-27 19:29:43] ...trap/engines/engine.py.230: loss: 0.16501 [I 2022-10-27 19:29:43] ...optimizers/trijoint.py.151: total_norm: 25.119164 [I 2022-10-27 19:29:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 840/2304 [I 2022-10-27 19:29:49] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:31 [I 2022-10-27 19:29:49] ...trap/engines/engine.py.229: process: 0.55289 | load: 0.00241 [I 2022-10-27 19:29:49] ...trap/engines/engine.py.230: loss: 0.12143 [I 2022-10-27 19:29:49] ...optimizers/trijoint.py.151: total_norm: 18.644407 [I 2022-10-27 19:29:54] ...trap/engines/engine.py.224: train: epoch 84 | batch 850/2304 [I 2022-10-27 19:29:54] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:27 [I 2022-10-27 19:29:54] ...trap/engines/engine.py.229: process: 0.55505 | load: 0.00242 [I 2022-10-27 19:29:54] ...trap/engines/engine.py.230: loss: 0.14510 [I 2022-10-27 19:29:54] ...optimizers/trijoint.py.151: total_norm: 19.957851 [I 2022-10-27 19:30:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 860/2304 [I 2022-10-27 19:30:00] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:31 [I 2022-10-27 19:30:00] ...trap/engines/engine.py.229: process: 0.54945 | load: 0.00251 [I 2022-10-27 19:30:00] ...trap/engines/engine.py.230: loss: 0.21593 [I 2022-10-27 19:30:00] ...optimizers/trijoint.py.151: total_norm: 20.530193 [I 2022-10-27 19:30:05] ...trap/engines/engine.py.224: train: epoch 84 | batch 870/2304 [I 2022-10-27 19:30:05] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:19 [I 2022-10-27 19:30:05] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00248 [I 2022-10-27 19:30:05] ...trap/engines/engine.py.230: loss: 0.10526 [I 2022-10-27 19:30:05] ...optimizers/trijoint.py.151: total_norm: 19.062950 [I 2022-10-27 19:30:11] ...trap/engines/engine.py.224: train: epoch 84 | batch 880/2304 [I 2022-10-27 19:30:11] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:10 [I 2022-10-27 19:30:11] ...trap/engines/engine.py.229: process: 0.54522 | load: 0.00246 [I 2022-10-27 19:30:11] ...trap/engines/engine.py.230: loss: 0.13856 [I 2022-10-27 19:30:11] ...optimizers/trijoint.py.151: total_norm: 31.812210 [I 2022-10-27 19:30:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 890/2304 [I 2022-10-27 19:30:17] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:10 [I 2022-10-27 19:30:17] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00243 [I 2022-10-27 19:30:17] ...trap/engines/engine.py.230: loss: 0.11492 [I 2022-10-27 19:30:17] ...optimizers/trijoint.py.151: total_norm: 35.234257 [I 2022-10-27 19:30:22] ...trap/engines/engine.py.224: train: epoch 84 | batch 900/2304 [I 2022-10-27 19:30:22] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:04 [I 2022-10-27 19:30:22] ...trap/engines/engine.py.229: process: 0.56744 | load: 0.00242 [I 2022-10-27 19:30:22] ...trap/engines/engine.py.230: loss: 0.24783 [I 2022-10-27 19:30:22] ...optimizers/trijoint.py.151: total_norm: 28.214260 [I 2022-10-27 19:30:28] ...trap/engines/engine.py.224: train: epoch 84 | batch 910/2304 [I 2022-10-27 19:30:28] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:57 [I 2022-10-27 19:30:28] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00244 [I 2022-10-27 19:30:28] ...trap/engines/engine.py.230: loss: 0.12303 [I 2022-10-27 19:30:28] ...optimizers/trijoint.py.151: total_norm: 17.054335 [I 2022-10-27 19:30:33] ...trap/engines/engine.py.224: train: epoch 84 | batch 920/2304 [I 2022-10-27 19:30:33] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:48 [I 2022-10-27 19:30:33] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00240 [I 2022-10-27 19:30:33] ...trap/engines/engine.py.230: loss: 0.12295 [I 2022-10-27 19:30:33] ...optimizers/trijoint.py.151: total_norm: 24.102150 [I 2022-10-27 19:30:39] ...trap/engines/engine.py.224: train: epoch 84 | batch 930/2304 [I 2022-10-27 19:30:39] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:43 [I 2022-10-27 19:30:39] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00242 [I 2022-10-27 19:30:39] ...trap/engines/engine.py.230: loss: 0.10249 [I 2022-10-27 19:30:39] ...optimizers/trijoint.py.151: total_norm: 15.297694 [I 2022-10-27 19:30:44] ...trap/engines/engine.py.224: train: epoch 84 | batch 940/2304 [I 2022-10-27 19:30:44] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:35 [I 2022-10-27 19:30:44] ...trap/engines/engine.py.229: process: 0.55062 | load: 0.00241 [I 2022-10-27 19:30:44] ...trap/engines/engine.py.230: loss: 0.12615 [I 2022-10-27 19:30:44] ...optimizers/trijoint.py.151: total_norm: 34.837334 [I 2022-10-27 19:30:50] ...trap/engines/engine.py.224: train: epoch 84 | batch 950/2304 [I 2022-10-27 19:30:50] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:32 [I 2022-10-27 19:30:50] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00249 [I 2022-10-27 19:30:50] ...trap/engines/engine.py.230: loss: 0.11104 [I 2022-10-27 19:30:50] ...optimizers/trijoint.py.151: total_norm: 16.185184 [I 2022-10-27 19:30:56] ...trap/engines/engine.py.224: train: epoch 84 | batch 960/2304 [I 2022-10-27 19:30:56] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:27 [I 2022-10-27 19:30:56] ...trap/engines/engine.py.229: process: 0.55301 | load: 0.00246 [I 2022-10-27 19:30:56] ...trap/engines/engine.py.230: loss: 0.10623 [I 2022-10-27 19:30:56] ...optimizers/trijoint.py.151: total_norm: 16.064362 [I 2022-10-27 19:31:01] ...trap/engines/engine.py.224: train: epoch 84 | batch 970/2304 [I 2022-10-27 19:31:01] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:49 [I 2022-10-27 19:31:01] ...trap/engines/engine.py.229: process: 0.64080 | load: 0.00246 [I 2022-10-27 19:31:01] ...trap/engines/engine.py.230: loss: 0.09166 [I 2022-10-27 19:31:01] ...optimizers/trijoint.py.151: total_norm: 14.294236 [I 2022-10-27 19:31:07] ...trap/engines/engine.py.224: train: epoch 84 | batch 980/2304 [I 2022-10-27 19:31:07] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:24 [I 2022-10-27 19:31:07] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00248 [I 2022-10-27 19:31:07] ...trap/engines/engine.py.230: loss: 0.12728 [I 2022-10-27 19:31:07] ...optimizers/trijoint.py.151: total_norm: 13.586613 [I 2022-10-27 19:31:13] ...trap/engines/engine.py.224: train: epoch 84 | batch 990/2304 [I 2022-10-27 19:31:13] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:14 [I 2022-10-27 19:31:13] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00241 [I 2022-10-27 19:31:13] ...trap/engines/engine.py.230: loss: 0.19607 [I 2022-10-27 19:31:13] ...optimizers/trijoint.py.151: total_norm: 22.781530 [I 2022-10-27 19:31:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 1000/2304 [I 2022-10-27 19:31:18] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:12 [I 2022-10-27 19:31:18] ...trap/engines/engine.py.229: process: 0.56908 | load: 0.00245 [I 2022-10-27 19:31:18] ...trap/engines/engine.py.230: loss: 0.14644 [I 2022-10-27 19:31:18] ...optimizers/trijoint.py.151: total_norm: 20.051769 [I 2022-10-27 19:31:24] ...trap/engines/engine.py.224: train: epoch 84 | batch 1010/2304 [I 2022-10-27 19:31:24] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:02 [I 2022-10-27 19:31:24] ...trap/engines/engine.py.229: process: 0.55177 | load: 0.00249 [I 2022-10-27 19:31:24] ...trap/engines/engine.py.230: loss: 0.11792 [I 2022-10-27 19:31:24] ...optimizers/trijoint.py.151: total_norm: 14.337495 [I 2022-10-27 19:31:29] ...trap/engines/engine.py.224: train: epoch 84 | batch 1020/2304 [I 2022-10-27 19:31:29] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:52 [I 2022-10-27 19:31:29] ...trap/engines/engine.py.229: process: 0.55098 | load: 0.00241 [I 2022-10-27 19:31:29] ...trap/engines/engine.py.230: loss: 0.14439 [I 2022-10-27 19:31:29] ...optimizers/trijoint.py.151: total_norm: 9.675323 [I 2022-10-27 19:31:35] ...trap/engines/engine.py.224: train: epoch 84 | batch 1030/2304 [I 2022-10-27 19:31:35] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:47 [I 2022-10-27 19:31:35] ...trap/engines/engine.py.229: process: 0.54475 | load: 0.00244 [I 2022-10-27 19:31:35] ...trap/engines/engine.py.230: loss: 0.20825 [I 2022-10-27 19:31:35] ...optimizers/trijoint.py.151: total_norm: 19.936813 [I 2022-10-27 19:31:40] ...trap/engines/engine.py.224: train: epoch 84 | batch 1040/2304 [I 2022-10-27 19:31:40] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:40 [I 2022-10-27 19:31:40] ...trap/engines/engine.py.229: process: 0.54554 | load: 0.00242 [I 2022-10-27 19:31:40] ...trap/engines/engine.py.230: loss: 0.10069 [I 2022-10-27 19:31:40] ...optimizers/trijoint.py.151: total_norm: 17.773052 [I 2022-10-27 19:31:46] ...trap/engines/engine.py.224: train: epoch 84 | batch 1050/2304 [I 2022-10-27 19:31:46] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:33 [I 2022-10-27 19:31:46] ...trap/engines/engine.py.229: process: 0.53626 | load: 0.00243 [I 2022-10-27 19:31:46] ...trap/engines/engine.py.230: loss: 0.12631 [I 2022-10-27 19:31:46] ...optimizers/trijoint.py.151: total_norm: 8.845029 [I 2022-10-27 19:31:52] ...trap/engines/engine.py.224: train: epoch 84 | batch 1060/2304 [I 2022-10-27 19:31:52] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:29 [I 2022-10-27 19:31:52] ...trap/engines/engine.py.229: process: 0.54668 | load: 0.00242 [I 2022-10-27 19:31:52] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-27 19:31:52] ...optimizers/trijoint.py.151: total_norm: 11.861430 [I 2022-10-27 19:31:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 1070/2304 [I 2022-10-27 19:31:57] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:26 [I 2022-10-27 19:31:57] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00243 [I 2022-10-27 19:31:57] ...trap/engines/engine.py.230: loss: 0.16673 [I 2022-10-27 19:31:57] ...optimizers/trijoint.py.151: total_norm: 29.902330 [I 2022-10-27 19:32:03] ...trap/engines/engine.py.224: train: epoch 84 | batch 1080/2304 [I 2022-10-27 19:32:03] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:19 [I 2022-10-27 19:32:03] ...trap/engines/engine.py.229: process: 0.55832 | load: 0.00241 [I 2022-10-27 19:32:03] ...trap/engines/engine.py.230: loss: 0.11370 [I 2022-10-27 19:32:03] ...optimizers/trijoint.py.151: total_norm: 13.974308 [I 2022-10-27 19:32:08] ...trap/engines/engine.py.224: train: epoch 84 | batch 1090/2304 [I 2022-10-27 19:32:08] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:16 [I 2022-10-27 19:32:08] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00260 [I 2022-10-27 19:32:08] ...trap/engines/engine.py.230: loss: 0.16922 [I 2022-10-27 19:32:08] ...optimizers/trijoint.py.151: total_norm: 40.124451 [I 2022-10-27 19:32:14] ...trap/engines/engine.py.224: train: epoch 84 | batch 1100/2304 [I 2022-10-27 19:32:14] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:11 [I 2022-10-27 19:32:14] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00250 [I 2022-10-27 19:32:14] ...trap/engines/engine.py.230: loss: 0.12115 [I 2022-10-27 19:32:14] ...optimizers/trijoint.py.151: total_norm: 17.871450 [I 2022-10-27 19:32:19] ...trap/engines/engine.py.224: train: epoch 84 | batch 1110/2304 [I 2022-10-27 19:32:19] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:03 [I 2022-10-27 19:32:19] ...trap/engines/engine.py.229: process: 0.55339 | load: 0.00252 [I 2022-10-27 19:32:19] ...trap/engines/engine.py.230: loss: 0.09920 [I 2022-10-27 19:32:19] ...optimizers/trijoint.py.151: total_norm: 7.962140 [I 2022-10-27 19:32:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 1120/2304 [I 2022-10-27 19:32:25] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:10:57 [I 2022-10-27 19:32:25] ...trap/engines/engine.py.229: process: 0.54536 | load: 0.00245 [I 2022-10-27 19:32:25] ...trap/engines/engine.py.230: loss: 0.19125 [I 2022-10-27 19:32:25] ...optimizers/trijoint.py.151: total_norm: 28.676682 [I 2022-10-27 19:32:31] ...trap/engines/engine.py.224: train: epoch 84 | batch 1130/2304 [I 2022-10-27 19:32:31] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:52 [I 2022-10-27 19:32:31] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00244 [I 2022-10-27 19:32:31] ...trap/engines/engine.py.230: loss: 0.20772 [I 2022-10-27 19:32:31] ...optimizers/trijoint.py.151: total_norm: 27.855341 [I 2022-10-27 19:32:36] ...trap/engines/engine.py.224: train: epoch 84 | batch 1140/2304 [I 2022-10-27 19:32:36] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:46 [I 2022-10-27 19:32:36] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00242 [I 2022-10-27 19:32:36] ...trap/engines/engine.py.230: loss: 0.17251 [I 2022-10-27 19:32:36] ...optimizers/trijoint.py.151: total_norm: 21.009853 [I 2022-10-27 19:32:42] ...trap/engines/engine.py.224: train: epoch 84 | batch 1150/2304 [I 2022-10-27 19:32:42] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:43 [I 2022-10-27 19:32:42] ...trap/engines/engine.py.229: process: 0.56045 | load: 0.00243 [I 2022-10-27 19:32:42] ...trap/engines/engine.py.230: loss: 0.10882 [I 2022-10-27 19:32:42] ...optimizers/trijoint.py.151: total_norm: 41.127956 [I 2022-10-27 19:32:47] ...trap/engines/engine.py.224: train: epoch 84 | batch 1160/2304 [I 2022-10-27 19:32:47] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:36 [I 2022-10-27 19:32:47] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00242 [I 2022-10-27 19:32:47] ...trap/engines/engine.py.230: loss: 0.13016 [I 2022-10-27 19:32:47] ...optimizers/trijoint.py.151: total_norm: 23.939734 [I 2022-10-27 19:32:53] ...trap/engines/engine.py.224: train: epoch 84 | batch 1170/2304 [I 2022-10-27 19:32:53] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:29 [I 2022-10-27 19:32:53] ...trap/engines/engine.py.229: process: 0.54868 | load: 0.00244 [I 2022-10-27 19:32:53] ...trap/engines/engine.py.230: loss: 0.11360 [I 2022-10-27 19:32:53] ...optimizers/trijoint.py.151: total_norm: 12.970901 [I 2022-10-27 19:32:58] ...trap/engines/engine.py.224: train: epoch 84 | batch 1180/2304 [I 2022-10-27 19:32:58] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:24 [I 2022-10-27 19:32:58] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00243 [I 2022-10-27 19:32:58] ...trap/engines/engine.py.230: loss: 0.11333 [I 2022-10-27 19:32:58] ...optimizers/trijoint.py.151: total_norm: 19.571754 [I 2022-10-27 19:33:04] ...trap/engines/engine.py.224: train: epoch 84 | batch 1190/2304 [I 2022-10-27 19:33:04] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:19 [I 2022-10-27 19:33:04] ...trap/engines/engine.py.229: process: 0.55814 | load: 0.00250 [I 2022-10-27 19:33:04] ...trap/engines/engine.py.230: loss: 0.16450 [I 2022-10-27 19:33:04] ...optimizers/trijoint.py.151: total_norm: 23.283436 [I 2022-10-27 19:33:10] ...trap/engines/engine.py.224: train: epoch 84 | batch 1200/2304 [I 2022-10-27 19:33:10] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:14 [I 2022-10-27 19:33:10] ...trap/engines/engine.py.229: process: 0.54958 | load: 0.00247 [I 2022-10-27 19:33:10] ...trap/engines/engine.py.230: loss: 0.16400 [I 2022-10-27 19:33:10] ...optimizers/trijoint.py.151: total_norm: 37.263252 [I 2022-10-27 19:33:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 1210/2304 [I 2022-10-27 19:33:15] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:05 [I 2022-10-27 19:33:15] ...trap/engines/engine.py.229: process: 0.55005 | load: 0.00246 [I 2022-10-27 19:33:15] ...trap/engines/engine.py.230: loss: 0.13078 [I 2022-10-27 19:33:15] ...optimizers/trijoint.py.151: total_norm: 21.571308 [I 2022-10-27 19:33:21] ...trap/engines/engine.py.224: train: epoch 84 | batch 1220/2304 [I 2022-10-27 19:33:21] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:04 [I 2022-10-27 19:33:21] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00246 [I 2022-10-27 19:33:21] ...trap/engines/engine.py.230: loss: 0.07687 [I 2022-10-27 19:33:21] ...optimizers/trijoint.py.151: total_norm: 10.804407 [I 2022-10-27 19:33:26] ...trap/engines/engine.py.224: train: epoch 84 | batch 1230/2304 [I 2022-10-27 19:33:26] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:57 [I 2022-10-27 19:33:26] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00244 [I 2022-10-27 19:33:26] ...trap/engines/engine.py.230: loss: 0.13385 [I 2022-10-27 19:33:26] ...optimizers/trijoint.py.151: total_norm: 30.691019 [I 2022-10-27 19:33:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 1240/2304 [I 2022-10-27 19:33:32] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:49 [I 2022-10-27 19:33:32] ...trap/engines/engine.py.229: process: 0.54484 | load: 0.00242 [I 2022-10-27 19:33:32] ...trap/engines/engine.py.230: loss: 0.11043 [I 2022-10-27 19:33:32] ...optimizers/trijoint.py.151: total_norm: 15.413611 [I 2022-10-27 19:33:37] ...trap/engines/engine.py.224: train: epoch 84 | batch 1250/2304 [I 2022-10-27 19:33:37] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:44 [I 2022-10-27 19:33:37] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00247 [I 2022-10-27 19:33:37] ...trap/engines/engine.py.230: loss: 0.15000 [I 2022-10-27 19:33:37] ...optimizers/trijoint.py.151: total_norm: 22.755089 [I 2022-10-27 19:33:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 1260/2304 [I 2022-10-27 19:33:43] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:41 [I 2022-10-27 19:33:43] ...trap/engines/engine.py.229: process: 0.54602 | load: 0.00240 [I 2022-10-27 19:33:43] ...trap/engines/engine.py.230: loss: 0.21832 [I 2022-10-27 19:33:43] ...optimizers/trijoint.py.151: total_norm: 13.322044 [I 2022-10-27 19:33:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 1270/2304 [I 2022-10-27 19:33:49] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:37 [I 2022-10-27 19:33:49] ...trap/engines/engine.py.229: process: 0.56392 | load: 0.00239 [I 2022-10-27 19:33:49] ...trap/engines/engine.py.230: loss: 0.18526 [I 2022-10-27 19:33:49] ...optimizers/trijoint.py.151: total_norm: 29.890057 [I 2022-10-27 19:33:54] ...trap/engines/engine.py.224: train: epoch 84 | batch 1280/2304 [I 2022-10-27 19:33:54] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:28 [I 2022-10-27 19:33:54] ...trap/engines/engine.py.229: process: 0.55727 | load: 0.00244 [I 2022-10-27 19:33:54] ...trap/engines/engine.py.230: loss: 0.13510 [I 2022-10-27 19:33:54] ...optimizers/trijoint.py.151: total_norm: 12.439852 [I 2022-10-27 19:34:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 1290/2304 [I 2022-10-27 19:34:00] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:23 [I 2022-10-27 19:34:00] ...trap/engines/engine.py.229: process: 0.55053 | load: 0.00244 [I 2022-10-27 19:34:00] ...trap/engines/engine.py.230: loss: 0.10807 [I 2022-10-27 19:34:00] ...optimizers/trijoint.py.151: total_norm: 12.896278 [I 2022-10-27 19:34:05] ...trap/engines/engine.py.224: train: epoch 84 | batch 1300/2304 [I 2022-10-27 19:34:05] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:14 [I 2022-10-27 19:34:05] ...trap/engines/engine.py.229: process: 0.56369 | load: 0.00242 [I 2022-10-27 19:34:05] ...trap/engines/engine.py.230: loss: 0.17187 [I 2022-10-27 19:34:05] ...optimizers/trijoint.py.151: total_norm: 16.139032 [I 2022-10-27 19:34:11] ...trap/engines/engine.py.224: train: epoch 84 | batch 1310/2304 [I 2022-10-27 19:34:11] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:15 [I 2022-10-27 19:34:11] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00243 [I 2022-10-27 19:34:11] ...trap/engines/engine.py.230: loss: 0.12465 [I 2022-10-27 19:34:11] ...optimizers/trijoint.py.151: total_norm: 13.946609 [I 2022-10-27 19:34:16] ...trap/engines/engine.py.224: train: epoch 84 | batch 1320/2304 [I 2022-10-27 19:34:16] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:07 [I 2022-10-27 19:34:16] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00252 [I 2022-10-27 19:34:16] ...trap/engines/engine.py.230: loss: 0.10476 [I 2022-10-27 19:34:16] ...optimizers/trijoint.py.151: total_norm: 10.849008 [I 2022-10-27 19:34:22] ...trap/engines/engine.py.224: train: epoch 84 | batch 1330/2304 [I 2022-10-27 19:34:22] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:06 [I 2022-10-27 19:34:22] ...trap/engines/engine.py.229: process: 0.57480 | load: 0.00251 [I 2022-10-27 19:34:22] ...trap/engines/engine.py.230: loss: 0.12548 [I 2022-10-27 19:34:22] ...optimizers/trijoint.py.151: total_norm: 35.581497 [I 2022-10-27 19:34:28] ...trap/engines/engine.py.224: train: epoch 84 | batch 1340/2304 [I 2022-10-27 19:34:28] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:55 [I 2022-10-27 19:34:28] ...trap/engines/engine.py.229: process: 0.54852 | load: 0.00244 [I 2022-10-27 19:34:28] ...trap/engines/engine.py.230: loss: 0.13775 [I 2022-10-27 19:34:28] ...optimizers/trijoint.py.151: total_norm: 12.832788 [I 2022-10-27 19:34:33] ...trap/engines/engine.py.224: train: epoch 84 | batch 1350/2304 [I 2022-10-27 19:34:33] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:49 [I 2022-10-27 19:34:33] ...trap/engines/engine.py.229: process: 0.55404 | load: 0.00241 [I 2022-10-27 19:34:33] ...trap/engines/engine.py.230: loss: 0.13185 [I 2022-10-27 19:34:33] ...optimizers/trijoint.py.151: total_norm: 17.510134 [I 2022-10-27 19:34:39] ...trap/engines/engine.py.224: train: epoch 84 | batch 1360/2304 [I 2022-10-27 19:34:39] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:46 [I 2022-10-27 19:34:39] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00243 [I 2022-10-27 19:34:39] ...trap/engines/engine.py.230: loss: 0.11614 [I 2022-10-27 19:34:39] ...optimizers/trijoint.py.151: total_norm: 8.760495 [I 2022-10-27 19:34:44] ...trap/engines/engine.py.224: train: epoch 84 | batch 1370/2304 [I 2022-10-27 19:34:44] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:40 [I 2022-10-27 19:34:44] ...trap/engines/engine.py.229: process: 0.56355 | load: 0.00249 [I 2022-10-27 19:34:44] ...trap/engines/engine.py.230: loss: 0.09357 [I 2022-10-27 19:34:44] ...optimizers/trijoint.py.151: total_norm: 8.553310 [I 2022-10-27 19:34:50] ...trap/engines/engine.py.224: train: epoch 84 | batch 1380/2304 [I 2022-10-27 19:34:50] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:33 [I 2022-10-27 19:34:50] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00244 [I 2022-10-27 19:34:50] ...trap/engines/engine.py.230: loss: 0.19630 [I 2022-10-27 19:34:50] ...optimizers/trijoint.py.151: total_norm: 42.487240 [I 2022-10-27 19:34:55] ...trap/engines/engine.py.224: train: epoch 84 | batch 1390/2304 [I 2022-10-27 19:34:55] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:25 [I 2022-10-27 19:34:55] ...trap/engines/engine.py.229: process: 0.54888 | load: 0.00245 [I 2022-10-27 19:34:55] ...trap/engines/engine.py.230: loss: 0.17095 [I 2022-10-27 19:34:55] ...optimizers/trijoint.py.151: total_norm: 17.591793 [I 2022-10-27 19:35:01] ...trap/engines/engine.py.224: train: epoch 84 | batch 1400/2304 [I 2022-10-27 19:35:01] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:23 [I 2022-10-27 19:35:01] ...trap/engines/engine.py.229: process: 0.55269 | load: 0.00242 [I 2022-10-27 19:35:01] ...trap/engines/engine.py.230: loss: 0.20095 [I 2022-10-27 19:35:01] ...optimizers/trijoint.py.151: total_norm: 23.411226 [I 2022-10-27 19:35:07] ...trap/engines/engine.py.224: train: epoch 84 | batch 1410/2304 [I 2022-10-27 19:35:07] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:16 [I 2022-10-27 19:35:07] ...trap/engines/engine.py.229: process: 0.56185 | load: 0.00247 [I 2022-10-27 19:35:07] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-27 19:35:07] ...optimizers/trijoint.py.151: total_norm: 39.239628 [I 2022-10-27 19:35:12] ...trap/engines/engine.py.224: train: epoch 84 | batch 1420/2304 [I 2022-10-27 19:35:12] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:12 [I 2022-10-27 19:35:12] ...trap/engines/engine.py.229: process: 0.54561 | load: 0.00246 [I 2022-10-27 19:35:12] ...trap/engines/engine.py.230: loss: 0.12861 [I 2022-10-27 19:35:12] ...optimizers/trijoint.py.151: total_norm: 25.484503 [I 2022-10-27 19:35:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 1430/2304 [I 2022-10-27 19:35:18] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:07 [I 2022-10-27 19:35:18] ...trap/engines/engine.py.229: process: 0.55923 | load: 0.00247 [I 2022-10-27 19:35:18] ...trap/engines/engine.py.230: loss: 0.12092 [I 2022-10-27 19:35:18] ...optimizers/trijoint.py.151: total_norm: 14.048373 [I 2022-10-27 19:35:23] ...trap/engines/engine.py.224: train: epoch 84 | batch 1440/2304 [I 2022-10-27 19:35:23] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:00 [I 2022-10-27 19:35:23] ...trap/engines/engine.py.229: process: 0.54760 | load: 0.00248 [I 2022-10-27 19:35:23] ...trap/engines/engine.py.230: loss: 0.12953 [I 2022-10-27 19:35:23] ...optimizers/trijoint.py.151: total_norm: 13.956911 [I 2022-10-27 19:35:29] ...trap/engines/engine.py.224: train: epoch 84 | batch 1450/2304 [I 2022-10-27 19:35:29] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:07:52 [I 2022-10-27 19:35:29] ...trap/engines/engine.py.229: process: 0.54922 | load: 0.00247 [I 2022-10-27 19:35:29] ...trap/engines/engine.py.230: loss: 0.10503 [I 2022-10-27 19:35:29] ...optimizers/trijoint.py.151: total_norm: 6.837365 [I 2022-10-27 19:35:35] ...trap/engines/engine.py.224: train: epoch 84 | batch 1460/2304 [I 2022-10-27 19:35:35] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:07:52 [I 2022-10-27 19:35:35] ...trap/engines/engine.py.229: process: 0.56607 | load: 0.00244 [I 2022-10-27 19:35:35] ...trap/engines/engine.py.230: loss: 0.12673 [I 2022-10-27 19:35:35] ...optimizers/trijoint.py.151: total_norm: 14.641890 [I 2022-10-27 19:35:40] ...trap/engines/engine.py.224: train: epoch 84 | batch 1470/2304 [I 2022-10-27 19:35:40] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:42 [I 2022-10-27 19:35:40] ...trap/engines/engine.py.229: process: 0.54581 | load: 0.00241 [I 2022-10-27 19:35:40] ...trap/engines/engine.py.230: loss: 0.15345 [I 2022-10-27 19:35:40] ...optimizers/trijoint.py.151: total_norm: 16.211046 [I 2022-10-27 19:35:46] ...trap/engines/engine.py.224: train: epoch 84 | batch 1480/2304 [I 2022-10-27 19:35:46] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:36 [I 2022-10-27 19:35:46] ...trap/engines/engine.py.229: process: 0.55058 | load: 0.00244 [I 2022-10-27 19:35:46] ...trap/engines/engine.py.230: loss: 0.11403 [I 2022-10-27 19:35:46] ...optimizers/trijoint.py.151: total_norm: 19.572905 [I 2022-10-27 19:35:51] ...trap/engines/engine.py.224: train: epoch 84 | batch 1490/2304 [I 2022-10-27 19:35:51] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:31 [I 2022-10-27 19:35:51] ...trap/engines/engine.py.229: process: 0.54988 | load: 0.00243 [I 2022-10-27 19:35:51] ...trap/engines/engine.py.230: loss: 0.16489 [I 2022-10-27 19:35:51] ...optimizers/trijoint.py.151: total_norm: 20.438951 [I 2022-10-27 19:35:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 1500/2304 [I 2022-10-27 19:35:57] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:24 [I 2022-10-27 19:35:57] ...trap/engines/engine.py.229: process: 0.53592 | load: 0.00244 [I 2022-10-27 19:35:57] ...trap/engines/engine.py.230: loss: 0.11327 [I 2022-10-27 19:35:57] ...optimizers/trijoint.py.151: total_norm: 14.196210 [I 2022-10-27 19:36:02] ...trap/engines/engine.py.224: train: epoch 84 | batch 1510/2304 [I 2022-10-27 19:36:02] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:23 [I 2022-10-27 19:36:02] ...trap/engines/engine.py.229: process: 0.56699 | load: 0.00242 [I 2022-10-27 19:36:02] ...trap/engines/engine.py.230: loss: 0.12336 [I 2022-10-27 19:36:02] ...optimizers/trijoint.py.151: total_norm: 8.853743 [I 2022-10-27 19:36:08] ...trap/engines/engine.py.224: train: epoch 84 | batch 1520/2304 [I 2022-10-27 19:36:08] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:14 [I 2022-10-27 19:36:08] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00242 [I 2022-10-27 19:36:08] ...trap/engines/engine.py.230: loss: 0.11733 [I 2022-10-27 19:36:08] ...optimizers/trijoint.py.151: total_norm: 13.727750 [I 2022-10-27 19:36:13] ...trap/engines/engine.py.224: train: epoch 84 | batch 1530/2304 [I 2022-10-27 19:36:13] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:10 [I 2022-10-27 19:36:13] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00240 [I 2022-10-27 19:36:13] ...trap/engines/engine.py.230: loss: 0.22644 [I 2022-10-27 19:36:13] ...optimizers/trijoint.py.151: total_norm: 23.266825 [I 2022-10-27 19:36:19] ...trap/engines/engine.py.224: train: epoch 84 | batch 1540/2304 [I 2022-10-27 19:36:19] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:07 [I 2022-10-27 19:36:19] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00242 [I 2022-10-27 19:36:19] ...trap/engines/engine.py.230: loss: 0.11546 [I 2022-10-27 19:36:19] ...optimizers/trijoint.py.151: total_norm: 26.432941 [I 2022-10-27 19:36:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 1550/2304 [I 2022-10-27 19:36:25] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:03 [I 2022-10-27 19:36:25] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00241 [I 2022-10-27 19:36:25] ...trap/engines/engine.py.230: loss: 0.12768 [I 2022-10-27 19:36:25] ...optimizers/trijoint.py.151: total_norm: 15.965376 [I 2022-10-27 19:36:30] ...trap/engines/engine.py.224: train: epoch 84 | batch 1560/2304 [I 2022-10-27 19:36:30] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:07:04 [I 2022-10-27 19:36:30] ...trap/engines/engine.py.229: process: 0.55501 | load: 0.00244 [I 2022-10-27 19:36:30] ...trap/engines/engine.py.230: loss: 0.22496 [I 2022-10-27 19:36:30] ...optimizers/trijoint.py.151: total_norm: 28.693249 [I 2022-10-27 19:36:36] ...trap/engines/engine.py.224: train: epoch 84 | batch 1570/2304 [I 2022-10-27 19:36:36] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:51 [I 2022-10-27 19:36:36] ...trap/engines/engine.py.229: process: 0.56505 | load: 0.00239 [I 2022-10-27 19:36:36] ...trap/engines/engine.py.230: loss: 0.10713 [I 2022-10-27 19:36:36] ...optimizers/trijoint.py.151: total_norm: 11.571403 [I 2022-10-27 19:36:42] ...trap/engines/engine.py.224: train: epoch 84 | batch 1580/2304 [I 2022-10-27 19:36:42] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:44 [I 2022-10-27 19:36:42] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00242 [I 2022-10-27 19:36:42] ...trap/engines/engine.py.230: loss: 0.15087 [I 2022-10-27 19:36:42] ...optimizers/trijoint.py.151: total_norm: 16.789181 [I 2022-10-27 19:36:47] ...trap/engines/engine.py.224: train: epoch 84 | batch 1590/2304 [I 2022-10-27 19:36:47] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:38 [I 2022-10-27 19:36:47] ...trap/engines/engine.py.229: process: 0.56212 | load: 0.00248 [I 2022-10-27 19:36:47] ...trap/engines/engine.py.230: loss: 0.15033 [I 2022-10-27 19:36:47] ...optimizers/trijoint.py.151: total_norm: 22.563461 [I 2022-10-27 19:36:53] ...trap/engines/engine.py.224: train: epoch 84 | batch 1600/2304 [I 2022-10-27 19:36:53] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:34 [I 2022-10-27 19:36:53] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00246 [I 2022-10-27 19:36:53] ...trap/engines/engine.py.230: loss: 0.17592 [I 2022-10-27 19:36:53] ...optimizers/trijoint.py.151: total_norm: 19.083218 [I 2022-10-27 19:36:58] ...trap/engines/engine.py.224: train: epoch 84 | batch 1610/2304 [I 2022-10-27 19:36:58] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:27 [I 2022-10-27 19:36:58] ...trap/engines/engine.py.229: process: 0.55711 | load: 0.00247 [I 2022-10-27 19:36:58] ...trap/engines/engine.py.230: loss: 0.12657 [I 2022-10-27 19:36:58] ...optimizers/trijoint.py.151: total_norm: 73.628540 [I 2022-10-27 19:37:04] ...trap/engines/engine.py.224: train: epoch 84 | batch 1620/2304 [I 2022-10-27 19:37:04] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:19 [I 2022-10-27 19:37:04] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00242 [I 2022-10-27 19:37:04] ...trap/engines/engine.py.230: loss: 0.12457 [I 2022-10-27 19:37:04] ...optimizers/trijoint.py.151: total_norm: 9.016294 [I 2022-10-27 19:37:10] ...trap/engines/engine.py.224: train: epoch 84 | batch 1630/2304 [I 2022-10-27 19:37:10] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:14 [I 2022-10-27 19:37:10] ...trap/engines/engine.py.229: process: 0.55672 | load: 0.00244 [I 2022-10-27 19:37:10] ...trap/engines/engine.py.230: loss: 0.09741 [I 2022-10-27 19:37:10] ...optimizers/trijoint.py.151: total_norm: 20.789597 [I 2022-10-27 19:37:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 1640/2304 [I 2022-10-27 19:37:15] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:19 [I 2022-10-27 19:37:15] ...trap/engines/engine.py.229: process: 0.56178 | load: 0.00245 [I 2022-10-27 19:37:15] ...trap/engines/engine.py.230: loss: 0.13154 [I 2022-10-27 19:37:15] ...optimizers/trijoint.py.151: total_norm: 14.334495 [I 2022-10-27 19:37:21] ...trap/engines/engine.py.224: train: epoch 84 | batch 1650/2304 [I 2022-10-27 19:37:21] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:03 [I 2022-10-27 19:37:21] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00243 [I 2022-10-27 19:37:21] ...trap/engines/engine.py.230: loss: 0.11723 [I 2022-10-27 19:37:21] ...optimizers/trijoint.py.151: total_norm: 11.501698 [I 2022-10-27 19:37:26] ...trap/engines/engine.py.224: train: epoch 84 | batch 1660/2304 [I 2022-10-27 19:37:26] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:05:56 [I 2022-10-27 19:37:26] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00247 [I 2022-10-27 19:37:26] ...trap/engines/engine.py.230: loss: 0.11568 [I 2022-10-27 19:37:26] ...optimizers/trijoint.py.151: total_norm: 9.775865 [I 2022-10-27 19:37:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 1670/2304 [I 2022-10-27 19:37:32] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:05:52 [I 2022-10-27 19:37:32] ...trap/engines/engine.py.229: process: 0.56601 | load: 0.00243 [I 2022-10-27 19:37:32] ...trap/engines/engine.py.230: loss: 0.23704 [I 2022-10-27 19:37:32] ...optimizers/trijoint.py.151: total_norm: 30.654539 [I 2022-10-27 19:37:38] ...trap/engines/engine.py.224: train: epoch 84 | batch 1680/2304 [I 2022-10-27 19:37:38] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:49 [I 2022-10-27 19:37:38] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00244 [I 2022-10-27 19:37:38] ...trap/engines/engine.py.230: loss: 0.11825 [I 2022-10-27 19:37:38] ...optimizers/trijoint.py.151: total_norm: 16.670601 [I 2022-10-27 19:37:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 1690/2304 [I 2022-10-27 19:37:43] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:42 [I 2022-10-27 19:37:43] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00259 [I 2022-10-27 19:37:43] ...trap/engines/engine.py.230: loss: 0.14661 [I 2022-10-27 19:37:43] ...optimizers/trijoint.py.151: total_norm: 23.318501 [I 2022-10-27 19:37:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 1700/2304 [I 2022-10-27 19:37:49] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:40 [I 2022-10-27 19:37:49] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00250 [I 2022-10-27 19:37:49] ...trap/engines/engine.py.230: loss: 0.11509 [I 2022-10-27 19:37:49] ...optimizers/trijoint.py.151: total_norm: 11.844923 [I 2022-10-27 19:37:54] ...trap/engines/engine.py.224: train: epoch 84 | batch 1710/2304 [I 2022-10-27 19:37:54] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:31 [I 2022-10-27 19:37:54] ...trap/engines/engine.py.229: process: 0.55425 | load: 0.00245 [I 2022-10-27 19:37:54] ...trap/engines/engine.py.230: loss: 0.12311 [I 2022-10-27 19:37:54] ...optimizers/trijoint.py.151: total_norm: 23.325394 [I 2022-10-27 19:38:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 1720/2304 [I 2022-10-27 19:38:00] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:26 [I 2022-10-27 19:38:00] ...trap/engines/engine.py.229: process: 0.56399 | load: 0.00246 [I 2022-10-27 19:38:00] ...trap/engines/engine.py.230: loss: 0.12074 [I 2022-10-27 19:38:00] ...optimizers/trijoint.py.151: total_norm: 10.191382 [I 2022-10-27 19:38:06] ...trap/engines/engine.py.224: train: epoch 84 | batch 1730/2304 [I 2022-10-27 19:38:06] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:20 [I 2022-10-27 19:38:06] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00243 [I 2022-10-27 19:38:06] ...trap/engines/engine.py.230: loss: 0.10501 [I 2022-10-27 19:38:06] ...optimizers/trijoint.py.151: total_norm: 18.006058 [I 2022-10-27 19:38:11] ...trap/engines/engine.py.224: train: epoch 84 | batch 1740/2304 [I 2022-10-27 19:38:11] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:13 [I 2022-10-27 19:38:11] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00242 [I 2022-10-27 19:38:11] ...trap/engines/engine.py.230: loss: 0.11173 [I 2022-10-27 19:38:11] ...optimizers/trijoint.py.151: total_norm: 12.160003 [I 2022-10-27 19:38:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 1750/2304 [I 2022-10-27 19:38:17] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:08 [I 2022-10-27 19:38:17] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00247 [I 2022-10-27 19:38:17] ...trap/engines/engine.py.230: loss: 0.13063 [I 2022-10-27 19:38:17] ...optimizers/trijoint.py.151: total_norm: 17.492855 [I 2022-10-27 19:38:22] ...trap/engines/engine.py.224: train: epoch 84 | batch 1760/2304 [I 2022-10-27 19:38:22] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:02 [I 2022-10-27 19:38:22] ...trap/engines/engine.py.229: process: 0.54857 | load: 0.00245 [I 2022-10-27 19:38:22] ...trap/engines/engine.py.230: loss: 0.10269 [I 2022-10-27 19:38:22] ...optimizers/trijoint.py.151: total_norm: 11.607987 [I 2022-10-27 19:38:28] ...trap/engines/engine.py.224: train: epoch 84 | batch 1770/2304 [I 2022-10-27 19:38:28] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:04:55 [I 2022-10-27 19:38:28] ...trap/engines/engine.py.229: process: 0.55571 | load: 0.00243 [I 2022-10-27 19:38:28] ...trap/engines/engine.py.230: loss: 0.15386 [I 2022-10-27 19:38:28] ...optimizers/trijoint.py.151: total_norm: 24.442636 [I 2022-10-27 19:38:33] ...trap/engines/engine.py.224: train: epoch 84 | batch 1780/2304 [I 2022-10-27 19:38:33] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:04:51 [I 2022-10-27 19:38:33] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00258 [I 2022-10-27 19:38:33] ...trap/engines/engine.py.230: loss: 0.08493 [I 2022-10-27 19:38:33] ...optimizers/trijoint.py.151: total_norm: 10.129991 [I 2022-10-27 19:38:39] ...trap/engines/engine.py.224: train: epoch 84 | batch 1790/2304 [I 2022-10-27 19:38:39] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:45 [I 2022-10-27 19:38:39] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00242 [I 2022-10-27 19:38:39] ...trap/engines/engine.py.230: loss: 0.18017 [I 2022-10-27 19:38:39] ...optimizers/trijoint.py.151: total_norm: 16.166409 [I 2022-10-27 19:38:45] ...trap/engines/engine.py.224: train: epoch 84 | batch 1800/2304 [I 2022-10-27 19:38:45] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:04:39 [I 2022-10-27 19:38:45] ...trap/engines/engine.py.229: process: 0.55293 | load: 0.00242 [I 2022-10-27 19:38:45] ...trap/engines/engine.py.230: loss: 0.13921 [I 2022-10-27 19:38:45] ...optimizers/trijoint.py.151: total_norm: 11.128242 [I 2022-10-27 19:38:50] ...trap/engines/engine.py.224: train: epoch 84 | batch 1810/2304 [I 2022-10-27 19:38:50] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:36 [I 2022-10-27 19:38:50] ...trap/engines/engine.py.229: process: 0.55847 | load: 0.00241 [I 2022-10-27 19:38:50] ...trap/engines/engine.py.230: loss: 0.11511 [I 2022-10-27 19:38:50] ...optimizers/trijoint.py.151: total_norm: 13.233183 [I 2022-10-27 19:38:56] ...trap/engines/engine.py.224: train: epoch 84 | batch 1820/2304 [I 2022-10-27 19:38:56] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:35 [I 2022-10-27 19:38:56] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00242 [I 2022-10-27 19:38:56] ...trap/engines/engine.py.230: loss: 0.19912 [I 2022-10-27 19:38:56] ...optimizers/trijoint.py.151: total_norm: 16.804794 [I 2022-10-27 19:39:01] ...trap/engines/engine.py.224: train: epoch 84 | batch 1830/2304 [I 2022-10-27 19:39:01] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:24 [I 2022-10-27 19:39:01] ...trap/engines/engine.py.229: process: 0.55466 | load: 0.00239 [I 2022-10-27 19:39:01] ...trap/engines/engine.py.230: loss: 0.16156 [I 2022-10-27 19:39:01] ...optimizers/trijoint.py.151: total_norm: 12.879218 [I 2022-10-27 19:39:07] ...trap/engines/engine.py.224: train: epoch 84 | batch 1840/2304 [I 2022-10-27 19:39:07] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:17 [I 2022-10-27 19:39:07] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00242 [I 2022-10-27 19:39:07] ...trap/engines/engine.py.230: loss: 0.11021 [I 2022-10-27 19:39:07] ...optimizers/trijoint.py.151: total_norm: 23.310175 [I 2022-10-27 19:39:13] ...trap/engines/engine.py.224: train: epoch 84 | batch 1850/2304 [I 2022-10-27 19:39:13] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:11 [I 2022-10-27 19:39:13] ...trap/engines/engine.py.229: process: 0.54609 | load: 0.00248 [I 2022-10-27 19:39:13] ...trap/engines/engine.py.230: loss: 0.13306 [I 2022-10-27 19:39:13] ...optimizers/trijoint.py.151: total_norm: 14.048983 [I 2022-10-27 19:39:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 1860/2304 [I 2022-10-27 19:39:18] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:07 [I 2022-10-27 19:39:18] ...trap/engines/engine.py.229: process: 0.55492 | load: 0.00243 [I 2022-10-27 19:39:18] ...trap/engines/engine.py.230: loss: 0.20891 [I 2022-10-27 19:39:18] ...optimizers/trijoint.py.151: total_norm: 14.710043 [I 2022-10-27 19:39:24] ...trap/engines/engine.py.224: train: epoch 84 | batch 1870/2304 [I 2022-10-27 19:39:24] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:02 [I 2022-10-27 19:39:24] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00248 [I 2022-10-27 19:39:24] ...trap/engines/engine.py.230: loss: 0.12314 [I 2022-10-27 19:39:24] ...optimizers/trijoint.py.151: total_norm: 18.464331 [I 2022-10-27 19:39:29] ...trap/engines/engine.py.224: train: epoch 84 | batch 1880/2304 [I 2022-10-27 19:39:29] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:03:55 [I 2022-10-27 19:39:29] ...trap/engines/engine.py.229: process: 0.54026 | load: 0.00242 [I 2022-10-27 19:39:29] ...trap/engines/engine.py.230: loss: 0.15885 [I 2022-10-27 19:39:29] ...optimizers/trijoint.py.151: total_norm: 15.929458 [I 2022-10-27 19:39:35] ...trap/engines/engine.py.224: train: epoch 84 | batch 1890/2304 [I 2022-10-27 19:39:35] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:03:52 [I 2022-10-27 19:39:35] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00239 [I 2022-10-27 19:39:35] ...trap/engines/engine.py.230: loss: 0.13302 [I 2022-10-27 19:39:35] ...optimizers/trijoint.py.151: total_norm: 17.348076 [I 2022-10-27 19:39:40] ...trap/engines/engine.py.224: train: epoch 84 | batch 1900/2304 [I 2022-10-27 19:39:40] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:03:44 [I 2022-10-27 19:39:40] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00244 [I 2022-10-27 19:39:40] ...trap/engines/engine.py.230: loss: 0.16188 [I 2022-10-27 19:39:40] ...optimizers/trijoint.py.151: total_norm: 16.110472 [I 2022-10-27 19:39:46] ...trap/engines/engine.py.224: train: epoch 84 | batch 1910/2304 [I 2022-10-27 19:39:46] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:39 [I 2022-10-27 19:39:46] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00245 [I 2022-10-27 19:39:46] ...trap/engines/engine.py.230: loss: 0.09571 [I 2022-10-27 19:39:46] ...optimizers/trijoint.py.151: total_norm: 8.720580 [I 2022-10-27 19:39:52] ...trap/engines/engine.py.224: train: epoch 84 | batch 1920/2304 [I 2022-10-27 19:39:52] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:34 [I 2022-10-27 19:39:52] ...trap/engines/engine.py.229: process: 0.56212 | load: 0.00238 [I 2022-10-27 19:39:52] ...trap/engines/engine.py.230: loss: 0.18306 [I 2022-10-27 19:39:52] ...optimizers/trijoint.py.151: total_norm: 22.380198 [I 2022-10-27 19:39:57] ...trap/engines/engine.py.224: train: epoch 84 | batch 1930/2304 [I 2022-10-27 19:39:57] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:26 [I 2022-10-27 19:39:57] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00242 [I 2022-10-27 19:39:57] ...trap/engines/engine.py.230: loss: 0.17204 [I 2022-10-27 19:39:57] ...optimizers/trijoint.py.151: total_norm: 35.980713 [I 2022-10-27 19:40:03] ...trap/engines/engine.py.224: train: epoch 84 | batch 1940/2304 [I 2022-10-27 19:40:03] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:28 [I 2022-10-27 19:40:03] ...trap/engines/engine.py.229: process: 0.54691 | load: 0.00250 [I 2022-10-27 19:40:03] ...trap/engines/engine.py.230: loss: 0.11163 [I 2022-10-27 19:40:03] ...optimizers/trijoint.py.151: total_norm: 15.311493 [I 2022-10-27 19:40:09] ...trap/engines/engine.py.224: train: epoch 84 | batch 1950/2304 [I 2022-10-27 19:40:09] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:20 [I 2022-10-27 19:40:09] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00251 [I 2022-10-27 19:40:09] ...trap/engines/engine.py.230: loss: 0.17947 [I 2022-10-27 19:40:09] ...optimizers/trijoint.py.151: total_norm: 23.864567 [I 2022-10-27 19:40:14] ...trap/engines/engine.py.224: train: epoch 84 | batch 1960/2304 [I 2022-10-27 19:40:14] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:12 [I 2022-10-27 19:40:14] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00247 [I 2022-10-27 19:40:14] ...trap/engines/engine.py.230: loss: 0.15704 [I 2022-10-27 19:40:14] ...optimizers/trijoint.py.151: total_norm: 15.020548 [I 2022-10-27 19:40:20] ...trap/engines/engine.py.224: train: epoch 84 | batch 1970/2304 [I 2022-10-27 19:40:20] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:07 [I 2022-10-27 19:40:20] ...trap/engines/engine.py.229: process: 0.57563 | load: 0.00242 [I 2022-10-27 19:40:20] ...trap/engines/engine.py.230: loss: 0.14157 [I 2022-10-27 19:40:20] ...optimizers/trijoint.py.151: total_norm: 19.754086 [I 2022-10-27 19:40:25] ...trap/engines/engine.py.224: train: epoch 84 | batch 1980/2304 [I 2022-10-27 19:40:25] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:00 [I 2022-10-27 19:40:25] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00246 [I 2022-10-27 19:40:25] ...trap/engines/engine.py.230: loss: 0.11052 [I 2022-10-27 19:40:25] ...optimizers/trijoint.py.151: total_norm: 11.207769 [I 2022-10-27 19:40:31] ...trap/engines/engine.py.224: train: epoch 84 | batch 1990/2304 [I 2022-10-27 19:40:31] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:02:54 [I 2022-10-27 19:40:31] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00246 [I 2022-10-27 19:40:31] ...trap/engines/engine.py.230: loss: 0.22329 [I 2022-10-27 19:40:31] ...optimizers/trijoint.py.151: total_norm: 12.132205 [I 2022-10-27 19:40:36] ...trap/engines/engine.py.224: train: epoch 84 | batch 2000/2304 [I 2022-10-27 19:40:36] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:02:48 [I 2022-10-27 19:40:36] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00245 [I 2022-10-27 19:40:36] ...trap/engines/engine.py.230: loss: 0.14752 [I 2022-10-27 19:40:36] ...optimizers/trijoint.py.151: total_norm: 12.339987 [I 2022-10-27 19:40:42] ...trap/engines/engine.py.224: train: epoch 84 | batch 2010/2304 [I 2022-10-27 19:40:42] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:02:44 [I 2022-10-27 19:40:42] ...trap/engines/engine.py.229: process: 0.56463 | load: 0.00244 [I 2022-10-27 19:40:42] ...trap/engines/engine.py.230: loss: 0.17622 [I 2022-10-27 19:40:42] ...optimizers/trijoint.py.151: total_norm: 32.095730 [I 2022-10-27 19:40:48] ...trap/engines/engine.py.224: train: epoch 84 | batch 2020/2304 [I 2022-10-27 19:40:48] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:37 [I 2022-10-27 19:40:48] ...trap/engines/engine.py.229: process: 0.55013 | load: 0.00242 [I 2022-10-27 19:40:48] ...trap/engines/engine.py.230: loss: 0.10331 [I 2022-10-27 19:40:48] ...optimizers/trijoint.py.151: total_norm: 11.861750 [I 2022-10-27 19:40:53] ...trap/engines/engine.py.224: train: epoch 84 | batch 2030/2304 [I 2022-10-27 19:40:53] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:32 [I 2022-10-27 19:40:53] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00243 [I 2022-10-27 19:40:53] ...trap/engines/engine.py.230: loss: 0.16274 [I 2022-10-27 19:40:53] ...optimizers/trijoint.py.151: total_norm: 11.262010 [I 2022-10-27 19:40:59] ...trap/engines/engine.py.224: train: epoch 84 | batch 2040/2304 [I 2022-10-27 19:40:59] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:27 [I 2022-10-27 19:40:59] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00248 [I 2022-10-27 19:40:59] ...trap/engines/engine.py.230: loss: 0.18950 [I 2022-10-27 19:40:59] ...optimizers/trijoint.py.151: total_norm: 28.922882 [I 2022-10-27 19:41:04] ...trap/engines/engine.py.224: train: epoch 84 | batch 2050/2304 [I 2022-10-27 19:41:04] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:20 [I 2022-10-27 19:41:04] ...trap/engines/engine.py.229: process: 0.55524 | load: 0.00242 [I 2022-10-27 19:41:04] ...trap/engines/engine.py.230: loss: 0.12909 [I 2022-10-27 19:41:04] ...optimizers/trijoint.py.151: total_norm: 13.212124 [I 2022-10-27 19:41:10] ...trap/engines/engine.py.224: train: epoch 84 | batch 2060/2304 [I 2022-10-27 19:41:10] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:16 [I 2022-10-27 19:41:10] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00249 [I 2022-10-27 19:41:10] ...trap/engines/engine.py.230: loss: 0.10013 [I 2022-10-27 19:41:10] ...optimizers/trijoint.py.151: total_norm: 12.393687 [I 2022-10-27 19:41:15] ...trap/engines/engine.py.224: train: epoch 84 | batch 2070/2304 [I 2022-10-27 19:41:15] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:09 [I 2022-10-27 19:41:15] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00243 [I 2022-10-27 19:41:15] ...trap/engines/engine.py.230: loss: 0.09418 [I 2022-10-27 19:41:15] ...optimizers/trijoint.py.151: total_norm: 7.670455 [I 2022-10-27 19:41:21] ...trap/engines/engine.py.224: train: epoch 84 | batch 2080/2304 [I 2022-10-27 19:41:21] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:04 [I 2022-10-27 19:41:21] ...trap/engines/engine.py.229: process: 0.56087 | load: 0.00244 [I 2022-10-27 19:41:21] ...trap/engines/engine.py.230: loss: 0.09613 [I 2022-10-27 19:41:21] ...optimizers/trijoint.py.151: total_norm: 10.256752 [I 2022-10-27 19:41:27] ...trap/engines/engine.py.224: train: epoch 84 | batch 2090/2304 [I 2022-10-27 19:41:27] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:01:59 [I 2022-10-27 19:41:27] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00252 [I 2022-10-27 19:41:27] ...trap/engines/engine.py.230: loss: 0.15356 [I 2022-10-27 19:41:27] ...optimizers/trijoint.py.151: total_norm: 13.031519 [I 2022-10-27 19:41:32] ...trap/engines/engine.py.224: train: epoch 84 | batch 2100/2304 [I 2022-10-27 19:41:32] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:01:53 [I 2022-10-27 19:41:32] ...trap/engines/engine.py.229: process: 0.55360 | load: 0.00248 [I 2022-10-27 19:41:32] ...trap/engines/engine.py.230: loss: 0.09929 [I 2022-10-27 19:41:32] ...optimizers/trijoint.py.151: total_norm: 19.664782 [I 2022-10-27 19:41:38] ...trap/engines/engine.py.224: train: epoch 84 | batch 2110/2304 [I 2022-10-27 19:41:38] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:01:47 [I 2022-10-27 19:41:38] ...trap/engines/engine.py.229: process: 0.54676 | load: 0.00245 [I 2022-10-27 19:41:38] ...trap/engines/engine.py.230: loss: 0.13245 [I 2022-10-27 19:41:38] ...optimizers/trijoint.py.151: total_norm: 26.738035 [I 2022-10-27 19:41:43] ...trap/engines/engine.py.224: train: epoch 84 | batch 2120/2304 [I 2022-10-27 19:41:43] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:01:42 [I 2022-10-27 19:41:43] ...trap/engines/engine.py.229: process: 0.55819 | load: 0.00244 [I 2022-10-27 19:41:43] ...trap/engines/engine.py.230: loss: 0.09089 [I 2022-10-27 19:41:43] ...optimizers/trijoint.py.151: total_norm: 13.403473 [I 2022-10-27 19:41:49] ...trap/engines/engine.py.224: train: epoch 84 | batch 2130/2304 [I 2022-10-27 19:41:49] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:37 [I 2022-10-27 19:41:49] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00243 [I 2022-10-27 19:41:49] ...trap/engines/engine.py.230: loss: 0.15306 [I 2022-10-27 19:41:49] ...optimizers/trijoint.py.151: total_norm: 48.874172 [I 2022-10-27 19:41:55] ...trap/engines/engine.py.224: train: epoch 84 | batch 2140/2304 [I 2022-10-27 19:41:55] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:30 [I 2022-10-27 19:41:55] ...trap/engines/engine.py.229: process: 0.54392 | load: 0.00242 [I 2022-10-27 19:41:55] ...trap/engines/engine.py.230: loss: 0.11320 [I 2022-10-27 19:41:55] ...optimizers/trijoint.py.151: total_norm: 24.374105 [I 2022-10-27 19:42:00] ...trap/engines/engine.py.224: train: epoch 84 | batch 2150/2304 [I 2022-10-27 19:42:00] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:24 [I 2022-10-27 19:42:00] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00246 [I 2022-10-27 19:42:00] ...trap/engines/engine.py.230: loss: 0.09696 [I 2022-10-27 19:42:00] ...optimizers/trijoint.py.151: total_norm: 15.089443 [I 2022-10-27 19:42:06] ...trap/engines/engine.py.224: train: epoch 84 | batch 2160/2304 [I 2022-10-27 19:42:06] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:20 [I 2022-10-27 19:42:06] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00244 [I 2022-10-27 19:42:06] ...trap/engines/engine.py.230: loss: 0.18290 [I 2022-10-27 19:42:06] ...optimizers/trijoint.py.151: total_norm: 20.455912 [I 2022-10-27 19:42:11] ...trap/engines/engine.py.224: train: epoch 84 | batch 2170/2304 [I 2022-10-27 19:42:11] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:14 [I 2022-10-27 19:42:11] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00245 [I 2022-10-27 19:42:11] ...trap/engines/engine.py.230: loss: 0.08440 [I 2022-10-27 19:42:11] ...optimizers/trijoint.py.151: total_norm: 10.570243 [I 2022-10-27 19:42:17] ...trap/engines/engine.py.224: train: epoch 84 | batch 2180/2304 [I 2022-10-27 19:42:17] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:09 [I 2022-10-27 19:42:17] ...trap/engines/engine.py.229: process: 0.54741 | load: 0.00249 [I 2022-10-27 19:42:17] ...trap/engines/engine.py.230: loss: 0.09869 [I 2022-10-27 19:42:17] ...optimizers/trijoint.py.151: total_norm: 12.614021 [I 2022-10-27 19:42:22] ...trap/engines/engine.py.224: train: epoch 84 | batch 2190/2304 [I 2022-10-27 19:42:22] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:03 [I 2022-10-27 19:42:22] ...trap/engines/engine.py.229: process: 0.56713 | load: 0.00244 [I 2022-10-27 19:42:22] ...trap/engines/engine.py.230: loss: 0.11157 [I 2022-10-27 19:42:22] ...optimizers/trijoint.py.151: total_norm: 16.587387 [I 2022-10-27 19:42:28] ...trap/engines/engine.py.224: train: epoch 84 | batch 2200/2304 [I 2022-10-27 19:42:28] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:00:58 [I 2022-10-27 19:42:28] ...trap/engines/engine.py.229: process: 0.56762 | load: 0.00244 [I 2022-10-27 19:42:28] ...trap/engines/engine.py.230: loss: 0.23280 [I 2022-10-27 19:42:28] ...optimizers/trijoint.py.151: total_norm: 58.404869 [I 2022-10-27 19:42:33] ...trap/engines/engine.py.224: train: epoch 84 | batch 2210/2304 [I 2022-10-27 19:42:33] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:00:52 [I 2022-10-27 19:42:33] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00246 [I 2022-10-27 19:42:33] ...trap/engines/engine.py.230: loss: 0.10148 [I 2022-10-27 19:42:33] ...optimizers/trijoint.py.151: total_norm: 9.656331 [I 2022-10-27 19:42:39] ...trap/engines/engine.py.224: train: epoch 84 | batch 2220/2304 [I 2022-10-27 19:42:39] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:00:47 [I 2022-10-27 19:42:39] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00246 [I 2022-10-27 19:42:39] ...trap/engines/engine.py.230: loss: 0.11556 [I 2022-10-27 19:42:39] ...optimizers/trijoint.py.151: total_norm: 11.309526 [I 2022-10-27 19:42:45] ...trap/engines/engine.py.224: train: epoch 84 | batch 2230/2304 [I 2022-10-27 19:42:45] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:41 [I 2022-10-27 19:42:45] ...trap/engines/engine.py.229: process: 0.56403 | load: 0.00241 [I 2022-10-27 19:42:45] ...trap/engines/engine.py.230: loss: 0.12101 [I 2022-10-27 19:42:45] ...optimizers/trijoint.py.151: total_norm: 9.212163 [I 2022-10-27 19:42:50] ...trap/engines/engine.py.224: train: epoch 84 | batch 2240/2304 [I 2022-10-27 19:42:50] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:35 [I 2022-10-27 19:42:50] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00246 [I 2022-10-27 19:42:50] ...trap/engines/engine.py.230: loss: 0.14606 [I 2022-10-27 19:42:50] ...optimizers/trijoint.py.151: total_norm: 22.687105 [I 2022-10-27 19:42:56] ...trap/engines/engine.py.224: train: epoch 84 | batch 2250/2304 [I 2022-10-27 19:42:56] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:30 [I 2022-10-27 19:42:56] ...trap/engines/engine.py.229: process: 0.55658 | load: 0.00242 [I 2022-10-27 19:42:56] ...trap/engines/engine.py.230: loss: 0.12294 [I 2022-10-27 19:42:56] ...optimizers/trijoint.py.151: total_norm: 13.323413 [I 2022-10-27 19:43:01] ...trap/engines/engine.py.224: train: epoch 84 | batch 2260/2304 [I 2022-10-27 19:43:01] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:24 [I 2022-10-27 19:43:01] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00245 [I 2022-10-27 19:43:01] ...trap/engines/engine.py.230: loss: 0.13738 [I 2022-10-27 19:43:01] ...optimizers/trijoint.py.151: total_norm: 28.270895 [I 2022-10-27 19:43:07] ...trap/engines/engine.py.224: train: epoch 84 | batch 2270/2304 [I 2022-10-27 19:43:07] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:18 [I 2022-10-27 19:43:07] ...trap/engines/engine.py.229: process: 0.56415 | load: 0.00241 [I 2022-10-27 19:43:07] ...trap/engines/engine.py.230: loss: 0.11437 [I 2022-10-27 19:43:07] ...optimizers/trijoint.py.151: total_norm: 13.626535 [I 2022-10-27 19:43:13] ...trap/engines/engine.py.224: train: epoch 84 | batch 2280/2304 [I 2022-10-27 19:43:13] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:13 [I 2022-10-27 19:43:13] ...trap/engines/engine.py.229: process: 0.54449 | load: 0.00244 [I 2022-10-27 19:43:13] ...trap/engines/engine.py.230: loss: 0.13577 [I 2022-10-27 19:43:13] ...optimizers/trijoint.py.151: total_norm: 12.182268 [I 2022-10-27 19:43:18] ...trap/engines/engine.py.224: train: epoch 84 | batch 2290/2304 [I 2022-10-27 19:43:18] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:07 [I 2022-10-27 19:43:18] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00251 [I 2022-10-27 19:43:18] ...trap/engines/engine.py.230: loss: 0.12016 [I 2022-10-27 19:43:18] ...optimizers/trijoint.py.151: total_norm: 56.017483 [I 2022-10-27 19:43:24] ...trap/engines/engine.py.224: train: epoch 84 | batch 2300/2304 [I 2022-10-27 19:43:24] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:02 [I 2022-10-27 19:43:24] ...trap/engines/engine.py.229: process: 0.55720 | load: 0.00014 [I 2022-10-27 19:43:24] ...trap/engines/engine.py.230: loss: 0.14810 [I 2022-10-27 19:43:24] ...optimizers/trijoint.py.151: total_norm: 9.039718 [I 2022-10-27 19:43:26] ...trap/engines/engine.py.224: train: epoch 84 | batch 2304/2304 [I 2022-10-27 19:43:26] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:00 [I 2022-10-27 19:43:26] ...trap/engines/engine.py.229: process: 0.55169 | load: 0.00014 [I 2022-10-27 19:43:26] ...trap/engines/engine.py.230: loss: 0.10725 [I 2022-10-27 19:43:26] ...optimizers/trijoint.py.151: total_norm: 23.440956 [S 2022-10-27 19:43:27] ...trap/engines/engine.py.129: train_epoch.epoch: 84 [S 2022-10-27 19:43:27] ...trap/engines/engine.py.129: train_epoch.loss: 0.13469354446953133 [I 2022-10-27 19:43:27] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 19:43:28] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 84 [I 2022-10-27 19:43:28] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 19:43:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 19:43:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 19:43:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 19:43:33] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 19:43:33] ...trap/engines/engine.py.321: eval: epoch 84 | batch 0/510 [I 2022-10-27 19:43:33] ...trap/engines/engine.py.325: elapsed: 0:00:04 | left: 0:42:19 [I 2022-10-27 19:43:33] ...trap/engines/engine.py.326: process: 4.97971 | load: 2.96231 [I 2022-10-27 19:43:37] ...trap/engines/engine.py.321: eval: epoch 84 | batch 10/510 [I 2022-10-27 19:43:37] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:06:54 [I 2022-10-27 19:43:37] ...trap/engines/engine.py.326: process: 0.32831 | load: 0.00020 [I 2022-10-27 19:43:40] ...trap/engines/engine.py.321: eval: epoch 84 | batch 20/510 [I 2022-10-27 19:43:40] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:12 [I 2022-10-27 19:43:40] ...trap/engines/engine.py.326: process: 0.35763 | load: 0.00019 [I 2022-10-27 19:43:43] ...trap/engines/engine.py.321: eval: epoch 84 | batch 30/510 [I 2022-10-27 19:43:43] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:47 [I 2022-10-27 19:43:43] ...trap/engines/engine.py.326: process: 0.35385 | load: 0.00019 [I 2022-10-27 19:43:47] ...trap/engines/engine.py.321: eval: epoch 84 | batch 40/510 [I 2022-10-27 19:43:47] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:40 [I 2022-10-27 19:43:47] ...trap/engines/engine.py.326: process: 0.32892 | load: 0.00019 [I 2022-10-27 19:43:50] ...trap/engines/engine.py.321: eval: epoch 84 | batch 50/510 [I 2022-10-27 19:43:50] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:35 [I 2022-10-27 19:43:50] ...trap/engines/engine.py.326: process: 0.33881 | load: 0.00022 [I 2022-10-27 19:43:54] ...trap/engines/engine.py.321: eval: epoch 84 | batch 60/510 [I 2022-10-27 19:43:54] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:30 [I 2022-10-27 19:43:54] ...trap/engines/engine.py.326: process: 0.33155 | load: 0.00017 [I 2022-10-27 19:43:57] ...trap/engines/engine.py.321: eval: epoch 84 | batch 70/510 [I 2022-10-27 19:43:57] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:29 [I 2022-10-27 19:43:57] ...trap/engines/engine.py.326: process: 0.34813 | load: 0.00023 [I 2022-10-27 19:44:00] ...trap/engines/engine.py.321: eval: epoch 84 | batch 80/510 [I 2022-10-27 19:44:00] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:23 [I 2022-10-27 19:44:00] ...trap/engines/engine.py.326: process: 0.33576 | load: 0.00017 [I 2022-10-27 19:44:04] ...trap/engines/engine.py.321: eval: epoch 84 | batch 90/510 [I 2022-10-27 19:44:04] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-27 19:44:04] ...trap/engines/engine.py.326: process: 0.33076 | load: 0.00029 [I 2022-10-27 19:44:07] ...trap/engines/engine.py.321: eval: epoch 84 | batch 100/510 [I 2022-10-27 19:44:07] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:15 [I 2022-10-27 19:44:07] ...trap/engines/engine.py.326: process: 0.28576 | load: 0.00018 [I 2022-10-27 19:44:10] ...trap/engines/engine.py.321: eval: epoch 84 | batch 110/510 [I 2022-10-27 19:44:10] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 19:44:10] ...trap/engines/engine.py.326: process: 0.28590 | load: 0.00017 [I 2022-10-27 19:44:13] ...trap/engines/engine.py.321: eval: epoch 84 | batch 120/510 [I 2022-10-27 19:44:13] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 19:44:13] ...trap/engines/engine.py.326: process: 0.28758 | load: 0.00015 [I 2022-10-27 19:44:16] ...trap/engines/engine.py.321: eval: epoch 84 | batch 130/510 [I 2022-10-27 19:44:16] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:51 [I 2022-10-27 19:44:16] ...trap/engines/engine.py.326: process: 0.30672 | load: 0.00014 [I 2022-10-27 19:44:19] ...trap/engines/engine.py.321: eval: epoch 84 | batch 140/510 [I 2022-10-27 19:44:19] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-27 19:44:19] ...trap/engines/engine.py.326: process: 0.28226 | load: 0.00020 [I 2022-10-27 19:44:22] ...trap/engines/engine.py.321: eval: epoch 84 | batch 150/510 [I 2022-10-27 19:44:22] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:44 [I 2022-10-27 19:44:22] ...trap/engines/engine.py.326: process: 0.28236 | load: 0.00022 [I 2022-10-27 19:44:24] ...trap/engines/engine.py.321: eval: epoch 84 | batch 160/510 [I 2022-10-27 19:44:24] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-27 19:44:24] ...trap/engines/engine.py.326: process: 0.29433 | load: 0.00014 [I 2022-10-27 19:44:27] ...trap/engines/engine.py.321: eval: epoch 84 | batch 170/510 [I 2022-10-27 19:44:27] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 19:44:27] ...trap/engines/engine.py.326: process: 0.28989 | load: 0.00015 [I 2022-10-27 19:44:30] ...trap/engines/engine.py.321: eval: epoch 84 | batch 180/510 [I 2022-10-27 19:44:30] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:34 [I 2022-10-27 19:44:30] ...trap/engines/engine.py.326: process: 0.28613 | load: 0.00022 [I 2022-10-27 19:44:33] ...trap/engines/engine.py.321: eval: epoch 84 | batch 190/510 [I 2022-10-27 19:44:33] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 19:44:33] ...trap/engines/engine.py.326: process: 0.28513 | load: 0.00018 [I 2022-10-27 19:44:36] ...trap/engines/engine.py.321: eval: epoch 84 | batch 200/510 [I 2022-10-27 19:44:36] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 19:44:36] ...trap/engines/engine.py.326: process: 0.28757 | load: 0.00021 [I 2022-10-27 19:44:39] ...trap/engines/engine.py.321: eval: epoch 84 | batch 210/510 [I 2022-10-27 19:44:39] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 19:44:39] ...trap/engines/engine.py.326: process: 0.28381 | load: 0.00020 [I 2022-10-27 19:44:42] ...trap/engines/engine.py.321: eval: epoch 84 | batch 220/510 [I 2022-10-27 19:44:42] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 19:44:42] ...trap/engines/engine.py.326: process: 0.28296 | load: 0.00019 [I 2022-10-27 19:44:45] ...trap/engines/engine.py.321: eval: epoch 84 | batch 230/510 [I 2022-10-27 19:44:45] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 19:44:45] ...trap/engines/engine.py.326: process: 0.28433 | load: 0.00022 [I 2022-10-27 19:44:47] ...trap/engines/engine.py.321: eval: epoch 84 | batch 240/510 [I 2022-10-27 19:44:47] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-27 19:44:47] ...trap/engines/engine.py.326: process: 0.29084 | load: 0.00025 [I 2022-10-27 19:44:50] ...trap/engines/engine.py.321: eval: epoch 84 | batch 250/510 [I 2022-10-27 19:44:50] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 19:44:50] ...trap/engines/engine.py.326: process: 0.28814 | load: 0.00026 [I 2022-10-27 19:44:53] ...trap/engines/engine.py.321: eval: epoch 84 | batch 260/510 [I 2022-10-27 19:44:53] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 19:44:53] ...trap/engines/engine.py.326: process: 0.29006 | load: 0.00018 [I 2022-10-27 19:44:56] ...trap/engines/engine.py.321: eval: epoch 84 | batch 270/510 [I 2022-10-27 19:44:56] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:08 [I 2022-10-27 19:44:56] ...trap/engines/engine.py.326: process: 0.28198 | load: 0.00018 [I 2022-10-27 19:44:59] ...trap/engines/engine.py.321: eval: epoch 84 | batch 280/510 [I 2022-10-27 19:44:59] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 19:44:59] ...trap/engines/engine.py.326: process: 0.28928 | load: 0.00016 [I 2022-10-27 19:45:02] ...trap/engines/engine.py.321: eval: epoch 84 | batch 290/510 [I 2022-10-27 19:45:02] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 19:45:02] ...trap/engines/engine.py.326: process: 0.29204 | load: 0.00016 [I 2022-10-27 19:45:05] ...trap/engines/engine.py.321: eval: epoch 84 | batch 300/510 [I 2022-10-27 19:45:05] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 19:45:05] ...trap/engines/engine.py.326: process: 0.29353 | load: 0.00016 [I 2022-10-27 19:45:08] ...trap/engines/engine.py.321: eval: epoch 84 | batch 310/510 [I 2022-10-27 19:45:08] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 19:45:08] ...trap/engines/engine.py.326: process: 0.28382 | load: 0.00017 [I 2022-10-27 19:45:11] ...trap/engines/engine.py.321: eval: epoch 84 | batch 320/510 [I 2022-10-27 19:45:11] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:55 [I 2022-10-27 19:45:11] ...trap/engines/engine.py.326: process: 0.28810 | load: 0.00016 [I 2022-10-27 19:45:13] ...trap/engines/engine.py.321: eval: epoch 84 | batch 330/510 [I 2022-10-27 19:45:13] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-27 19:45:13] ...trap/engines/engine.py.326: process: 0.28685 | load: 0.00016 [I 2022-10-27 19:45:16] ...trap/engines/engine.py.321: eval: epoch 84 | batch 340/510 [I 2022-10-27 19:45:16] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:49 [I 2022-10-27 19:45:16] ...trap/engines/engine.py.326: process: 0.27917 | load: 0.00017 [I 2022-10-27 19:45:19] ...trap/engines/engine.py.321: eval: epoch 84 | batch 350/510 [I 2022-10-27 19:45:19] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-27 19:45:19] ...trap/engines/engine.py.326: process: 0.28376 | load: 0.00017 [I 2022-10-27 19:45:22] ...trap/engines/engine.py.321: eval: epoch 84 | batch 360/510 [I 2022-10-27 19:45:22] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 19:45:22] ...trap/engines/engine.py.326: process: 0.28692 | load: 0.00015 [I 2022-10-27 19:45:25] ...trap/engines/engine.py.321: eval: epoch 84 | batch 370/510 [I 2022-10-27 19:45:25] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 19:45:25] ...trap/engines/engine.py.326: process: 0.29442 | load: 0.00015 [I 2022-10-27 19:45:28] ...trap/engines/engine.py.321: eval: epoch 84 | batch 380/510 [I 2022-10-27 19:45:28] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 19:45:28] ...trap/engines/engine.py.326: process: 0.29122 | load: 0.00015 [I 2022-10-27 19:45:31] ...trap/engines/engine.py.321: eval: epoch 84 | batch 390/510 [I 2022-10-27 19:45:31] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 19:45:31] ...trap/engines/engine.py.326: process: 0.28258 | load: 0.00016 [I 2022-10-27 19:45:34] ...trap/engines/engine.py.321: eval: epoch 84 | batch 400/510 [I 2022-10-27 19:45:34] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 19:45:34] ...trap/engines/engine.py.326: process: 0.29211 | load: 0.00016 [I 2022-10-27 19:45:37] ...trap/engines/engine.py.321: eval: epoch 84 | batch 410/510 [I 2022-10-27 19:45:37] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 19:45:37] ...trap/engines/engine.py.326: process: 0.28720 | load: 0.00017 [I 2022-10-27 19:45:39] ...trap/engines/engine.py.321: eval: epoch 84 | batch 420/510 [I 2022-10-27 19:45:39] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 19:45:39] ...trap/engines/engine.py.326: process: 0.29032 | load: 0.00016 [I 2022-10-27 19:45:42] ...trap/engines/engine.py.321: eval: epoch 84 | batch 430/510 [I 2022-10-27 19:45:42] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 19:45:42] ...trap/engines/engine.py.326: process: 0.28800 | load: 0.00016 [I 2022-10-27 19:45:45] ...trap/engines/engine.py.321: eval: epoch 84 | batch 440/510 [I 2022-10-27 19:45:45] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 19:45:45] ...trap/engines/engine.py.326: process: 0.29457 | load: 0.00021 [I 2022-10-27 19:45:48] ...trap/engines/engine.py.321: eval: epoch 84 | batch 450/510 [I 2022-10-27 19:45:48] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 19:45:48] ...trap/engines/engine.py.326: process: 0.28633 | load: 0.00018 [I 2022-10-27 19:45:51] ...trap/engines/engine.py.321: eval: epoch 84 | batch 460/510 [I 2022-10-27 19:45:51] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 19:45:51] ...trap/engines/engine.py.326: process: 0.28608 | load: 0.00017 [I 2022-10-27 19:45:54] ...trap/engines/engine.py.321: eval: epoch 84 | batch 470/510 [I 2022-10-27 19:45:54] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 19:45:54] ...trap/engines/engine.py.326: process: 0.28527 | load: 0.00018 [I 2022-10-27 19:45:57] ...trap/engines/engine.py.321: eval: epoch 84 | batch 480/510 [I 2022-10-27 19:45:57] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 19:45:57] ...trap/engines/engine.py.326: process: 0.29213 | load: 0.00017 [I 2022-10-27 19:46:00] ...trap/engines/engine.py.321: eval: epoch 84 | batch 490/510 [I 2022-10-27 19:46:00] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 19:46:00] ...trap/engines/engine.py.326: process: 0.28947 | load: 0.00020 [I 2022-10-27 19:46:03] ...trap/engines/engine.py.321: eval: epoch 84 | batch 500/510 [I 2022-10-27 19:46:03] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 19:46:03] ...trap/engines/engine.py.326: process: 0.28187 | load: 0.00017 [I 2022-10-27 19:46:05] ...trap/engines/engine.py.321: eval: epoch 84 | batch 510/510 [I 2022-10-27 19:46:05] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 19:46:05] ...trap/engines/engine.py.326: process: 0.28659 | load: 0.00010 [S 2022-10-27 19:46:06] ...trap/engines/engine.py.132: eval_epoch.epoch: 84 [I 2022-10-27 19:46:06] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.734700 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909700 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940300 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.730500 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907500 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937500 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.004584 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005460 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005883 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008016 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007420 [S 2022-10-27 19:46:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007256 [I 2022-10-27 19:46:12] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 19:46:12] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 19:46:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 19:46:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 19:46:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 19:46:16] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 19:46:17] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 19:46:23] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 19:46:23] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 19:46:23] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 19:46:26] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 19:46:32] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 19:46:32] ...trap/engines/engine.py.163: Training model on trainset for epoch 85 [I 2022-10-27 19:46:32] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 19:46:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 0/2304 [I 2022-10-27 19:46:35] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:43 [I 2022-10-27 19:46:35] ...trap/engines/engine.py.229: process: 2.64927 | load: 2.02393 [I 2022-10-27 19:46:35] ...trap/engines/engine.py.230: loss: 0.09083 [I 2022-10-27 19:46:35] ...optimizers/trijoint.py.151: total_norm: 12.283008 [I 2022-10-27 19:46:40] ...trap/engines/engine.py.224: train: epoch 85 | batch 10/2304 [I 2022-10-27 19:46:40] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:15 [I 2022-10-27 19:46:40] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00538 [I 2022-10-27 19:46:40] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-27 19:46:40] ...optimizers/trijoint.py.151: total_norm: 11.881690 [I 2022-10-27 19:46:46] ...trap/engines/engine.py.224: train: epoch 85 | batch 20/2304 [I 2022-10-27 19:46:46] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:06 [I 2022-10-27 19:46:46] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00320 [I 2022-10-27 19:46:46] ...trap/engines/engine.py.230: loss: 0.11052 [I 2022-10-27 19:46:46] ...optimizers/trijoint.py.151: total_norm: 16.652834 [I 2022-10-27 19:46:51] ...trap/engines/engine.py.224: train: epoch 85 | batch 30/2304 [I 2022-10-27 19:46:51] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:14 [I 2022-10-27 19:46:51] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00316 [I 2022-10-27 19:46:51] ...trap/engines/engine.py.230: loss: 0.10198 [I 2022-10-27 19:46:51] ...optimizers/trijoint.py.151: total_norm: 11.274767 [I 2022-10-27 19:46:57] ...trap/engines/engine.py.224: train: epoch 85 | batch 40/2304 [I 2022-10-27 19:46:57] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:54 [I 2022-10-27 19:46:57] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00341 [I 2022-10-27 19:46:57] ...trap/engines/engine.py.230: loss: 0.09944 [I 2022-10-27 19:46:57] ...optimizers/trijoint.py.151: total_norm: 10.652524 [I 2022-10-27 19:47:03] ...trap/engines/engine.py.224: train: epoch 85 | batch 50/2304 [I 2022-10-27 19:47:03] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:01 [I 2022-10-27 19:47:03] ...trap/engines/engine.py.229: process: 0.55425 | load: 0.00300 [I 2022-10-27 19:47:03] ...trap/engines/engine.py.230: loss: 0.14321 [I 2022-10-27 19:47:03] ...optimizers/trijoint.py.151: total_norm: 15.553162 [I 2022-10-27 19:47:08] ...trap/engines/engine.py.224: train: epoch 85 | batch 60/2304 [I 2022-10-27 19:47:08] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:49 [I 2022-10-27 19:47:08] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00284 [I 2022-10-27 19:47:08] ...trap/engines/engine.py.230: loss: 0.12707 [I 2022-10-27 19:47:08] ...optimizers/trijoint.py.151: total_norm: 16.482815 [I 2022-10-27 19:47:14] ...trap/engines/engine.py.224: train: epoch 85 | batch 70/2304 [I 2022-10-27 19:47:14] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:38 [I 2022-10-27 19:47:14] ...trap/engines/engine.py.229: process: 0.54822 | load: 0.00258 [I 2022-10-27 19:47:14] ...trap/engines/engine.py.230: loss: 0.09836 [I 2022-10-27 19:47:14] ...optimizers/trijoint.py.151: total_norm: 21.631325 [I 2022-10-27 19:47:19] ...trap/engines/engine.py.224: train: epoch 85 | batch 80/2304 [I 2022-10-27 19:47:19] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:41 [I 2022-10-27 19:47:19] ...trap/engines/engine.py.229: process: 0.54559 | load: 0.00278 [I 2022-10-27 19:47:19] ...trap/engines/engine.py.230: loss: 0.12928 [I 2022-10-27 19:47:19] ...optimizers/trijoint.py.151: total_norm: 12.573659 [I 2022-10-27 19:47:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 90/2304 [I 2022-10-27 19:47:25] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:50 [I 2022-10-27 19:47:25] ...trap/engines/engine.py.229: process: 0.57036 | load: 0.00255 [I 2022-10-27 19:47:25] ...trap/engines/engine.py.230: loss: 0.08706 [I 2022-10-27 19:47:25] ...optimizers/trijoint.py.151: total_norm: 8.943203 [I 2022-10-27 19:47:31] ...trap/engines/engine.py.224: train: epoch 85 | batch 100/2304 [I 2022-10-27 19:47:31] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:38 [I 2022-10-27 19:47:31] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00278 [I 2022-10-27 19:47:31] ...trap/engines/engine.py.230: loss: 0.13592 [I 2022-10-27 19:47:31] ...optimizers/trijoint.py.151: total_norm: 57.666752 [I 2022-10-27 19:47:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 110/2304 [I 2022-10-27 19:47:36] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:28 [I 2022-10-27 19:47:36] ...trap/engines/engine.py.229: process: 0.55888 | load: 0.00266 [I 2022-10-27 19:47:36] ...trap/engines/engine.py.230: loss: 0.12425 [I 2022-10-27 19:47:36] ...optimizers/trijoint.py.151: total_norm: 9.295633 [I 2022-10-27 19:47:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 120/2304 [I 2022-10-27 19:47:42] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:22 [I 2022-10-27 19:47:42] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00269 [I 2022-10-27 19:47:42] ...trap/engines/engine.py.230: loss: 0.23238 [I 2022-10-27 19:47:42] ...optimizers/trijoint.py.151: total_norm: 29.781845 [I 2022-10-27 19:47:48] ...trap/engines/engine.py.224: train: epoch 85 | batch 130/2304 [I 2022-10-27 19:47:48] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:13 [I 2022-10-27 19:47:48] ...trap/engines/engine.py.229: process: 0.54996 | load: 0.00254 [I 2022-10-27 19:47:48] ...trap/engines/engine.py.230: loss: 0.09251 [I 2022-10-27 19:47:48] ...optimizers/trijoint.py.151: total_norm: 11.111799 [I 2022-10-27 19:47:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 140/2304 [I 2022-10-27 19:47:53] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:05 [I 2022-10-27 19:47:53] ...trap/engines/engine.py.229: process: 0.55484 | load: 0.00252 [I 2022-10-27 19:47:53] ...trap/engines/engine.py.230: loss: 0.11749 [I 2022-10-27 19:47:53] ...optimizers/trijoint.py.151: total_norm: 8.918824 [I 2022-10-27 19:47:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 150/2304 [I 2022-10-27 19:47:59] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:21 [I 2022-10-27 19:47:59] ...trap/engines/engine.py.229: process: 0.55695 | load: 0.00249 [I 2022-10-27 19:47:59] ...trap/engines/engine.py.230: loss: 0.11270 [I 2022-10-27 19:47:59] ...optimizers/trijoint.py.151: total_norm: 14.934967 [I 2022-10-27 19:48:05] ...trap/engines/engine.py.224: train: epoch 85 | batch 160/2304 [I 2022-10-27 19:48:05] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:14 [I 2022-10-27 19:48:05] ...trap/engines/engine.py.229: process: 0.57863 | load: 0.00252 [I 2022-10-27 19:48:05] ...trap/engines/engine.py.230: loss: 0.17183 [I 2022-10-27 19:48:05] ...optimizers/trijoint.py.151: total_norm: 17.020573 [I 2022-10-27 19:48:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 170/2304 [I 2022-10-27 19:48:10] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:20:16 [I 2022-10-27 19:48:10] ...trap/engines/engine.py.229: process: 0.57549 | load: 0.00285 [I 2022-10-27 19:48:10] ...trap/engines/engine.py.230: loss: 0.11685 [I 2022-10-27 19:48:10] ...optimizers/trijoint.py.151: total_norm: 16.291199 [I 2022-10-27 19:48:16] ...trap/engines/engine.py.224: train: epoch 85 | batch 180/2304 [I 2022-10-27 19:48:16] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:52 [I 2022-10-27 19:48:16] ...trap/engines/engine.py.229: process: 0.56263 | load: 0.00252 [I 2022-10-27 19:48:16] ...trap/engines/engine.py.230: loss: 0.14960 [I 2022-10-27 19:48:16] ...optimizers/trijoint.py.151: total_norm: 19.420530 [I 2022-10-27 19:48:21] ...trap/engines/engine.py.224: train: epoch 85 | batch 190/2304 [I 2022-10-27 19:48:21] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:45 [I 2022-10-27 19:48:21] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00254 [I 2022-10-27 19:48:21] ...trap/engines/engine.py.230: loss: 0.11232 [I 2022-10-27 19:48:21] ...optimizers/trijoint.py.151: total_norm: 11.311966 [I 2022-10-27 19:48:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 200/2304 [I 2022-10-27 19:48:27] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:25 [I 2022-10-27 19:48:27] ...trap/engines/engine.py.229: process: 0.54839 | load: 0.00250 [I 2022-10-27 19:48:27] ...trap/engines/engine.py.230: loss: 0.18915 [I 2022-10-27 19:48:27] ...optimizers/trijoint.py.151: total_norm: 16.097801 [I 2022-10-27 19:48:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 210/2304 [I 2022-10-27 19:48:33] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:22 [I 2022-10-27 19:48:33] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00259 [I 2022-10-27 19:48:33] ...trap/engines/engine.py.230: loss: 0.11626 [I 2022-10-27 19:48:33] ...optimizers/trijoint.py.151: total_norm: 15.803978 [I 2022-10-27 19:48:38] ...trap/engines/engine.py.224: train: epoch 85 | batch 220/2304 [I 2022-10-27 19:48:38] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:28 [I 2022-10-27 19:48:38] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00249 [I 2022-10-27 19:48:38] ...trap/engines/engine.py.230: loss: 0.11497 [I 2022-10-27 19:48:38] ...optimizers/trijoint.py.151: total_norm: 12.899860 [I 2022-10-27 19:48:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 230/2304 [I 2022-10-27 19:48:44] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:22 [I 2022-10-27 19:48:44] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00245 [I 2022-10-27 19:48:44] ...trap/engines/engine.py.230: loss: 0.10165 [I 2022-10-27 19:48:44] ...optimizers/trijoint.py.151: total_norm: 17.733643 [I 2022-10-27 19:48:49] ...trap/engines/engine.py.224: train: epoch 85 | batch 240/2304 [I 2022-10-27 19:48:49] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:13 [I 2022-10-27 19:48:49] ...trap/engines/engine.py.229: process: 0.56377 | load: 0.00244 [I 2022-10-27 19:48:49] ...trap/engines/engine.py.230: loss: 0.09476 [I 2022-10-27 19:48:49] ...optimizers/trijoint.py.151: total_norm: 10.191720 [I 2022-10-27 19:48:55] ...trap/engines/engine.py.224: train: epoch 85 | batch 250/2304 [I 2022-10-27 19:48:55] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:07 [I 2022-10-27 19:48:55] ...trap/engines/engine.py.229: process: 0.57244 | load: 0.00261 [I 2022-10-27 19:48:55] ...trap/engines/engine.py.230: loss: 0.10400 [I 2022-10-27 19:48:55] ...optimizers/trijoint.py.151: total_norm: 12.332686 [I 2022-10-27 19:49:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 260/2304 [I 2022-10-27 19:49:01] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:18 [I 2022-10-27 19:49:01] ...trap/engines/engine.py.229: process: 0.54944 | load: 0.00250 [I 2022-10-27 19:49:01] ...trap/engines/engine.py.230: loss: 0.10775 [I 2022-10-27 19:49:01] ...optimizers/trijoint.py.151: total_norm: 12.255783 [I 2022-10-27 19:49:06] ...trap/engines/engine.py.224: train: epoch 85 | batch 270/2304 [I 2022-10-27 19:49:06] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:49 [I 2022-10-27 19:49:06] ...trap/engines/engine.py.229: process: 0.55528 | load: 0.00243 [I 2022-10-27 19:49:06] ...trap/engines/engine.py.230: loss: 0.12682 [I 2022-10-27 19:49:06] ...optimizers/trijoint.py.151: total_norm: 14.415068 [I 2022-10-27 19:49:12] ...trap/engines/engine.py.224: train: epoch 85 | batch 280/2304 [I 2022-10-27 19:49:12] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:43 [I 2022-10-27 19:49:12] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00244 [I 2022-10-27 19:49:12] ...trap/engines/engine.py.230: loss: 0.13909 [I 2022-10-27 19:49:12] ...optimizers/trijoint.py.151: total_norm: 40.889793 [I 2022-10-27 19:49:17] ...trap/engines/engine.py.224: train: epoch 85 | batch 290/2304 [I 2022-10-27 19:49:17] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:39 [I 2022-10-27 19:49:17] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00243 [I 2022-10-27 19:49:17] ...trap/engines/engine.py.230: loss: 0.11008 [I 2022-10-27 19:49:17] ...optimizers/trijoint.py.151: total_norm: 8.798479 [I 2022-10-27 19:49:23] ...trap/engines/engine.py.224: train: epoch 85 | batch 300/2304 [I 2022-10-27 19:49:23] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:37 [I 2022-10-27 19:49:23] ...trap/engines/engine.py.229: process: 0.55453 | load: 0.00243 [I 2022-10-27 19:49:23] ...trap/engines/engine.py.230: loss: 0.15441 [I 2022-10-27 19:49:23] ...optimizers/trijoint.py.151: total_norm: 17.677744 [I 2022-10-27 19:49:29] ...trap/engines/engine.py.224: train: epoch 85 | batch 310/2304 [I 2022-10-27 19:49:29] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:30 [I 2022-10-27 19:49:29] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00254 [I 2022-10-27 19:49:29] ...trap/engines/engine.py.230: loss: 0.11699 [I 2022-10-27 19:49:29] ...optimizers/trijoint.py.151: total_norm: 23.319580 [I 2022-10-27 19:49:34] ...trap/engines/engine.py.224: train: epoch 85 | batch 320/2304 [I 2022-10-27 19:49:34] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:33 [I 2022-10-27 19:49:34] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00262 [I 2022-10-27 19:49:34] ...trap/engines/engine.py.230: loss: 0.09999 [I 2022-10-27 19:49:34] ...optimizers/trijoint.py.151: total_norm: 7.335999 [I 2022-10-27 19:49:40] ...trap/engines/engine.py.224: train: epoch 85 | batch 330/2304 [I 2022-10-27 19:49:40] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:15 [I 2022-10-27 19:49:40] ...trap/engines/engine.py.229: process: 0.54601 | load: 0.00247 [I 2022-10-27 19:49:40] ...trap/engines/engine.py.230: loss: 0.13453 [I 2022-10-27 19:49:40] ...optimizers/trijoint.py.151: total_norm: 15.834150 [I 2022-10-27 19:49:45] ...trap/engines/engine.py.224: train: epoch 85 | batch 340/2304 [I 2022-10-27 19:49:45] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:23 [I 2022-10-27 19:49:45] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00281 [I 2022-10-27 19:49:45] ...trap/engines/engine.py.230: loss: 0.20621 [I 2022-10-27 19:49:45] ...optimizers/trijoint.py.151: total_norm: 15.554248 [I 2022-10-27 19:49:51] ...trap/engines/engine.py.224: train: epoch 85 | batch 350/2304 [I 2022-10-27 19:49:51] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:17 [I 2022-10-27 19:49:51] ...trap/engines/engine.py.229: process: 0.57176 | load: 0.00243 [I 2022-10-27 19:49:51] ...trap/engines/engine.py.230: loss: 0.12051 [I 2022-10-27 19:49:51] ...optimizers/trijoint.py.151: total_norm: 12.246730 [I 2022-10-27 19:49:57] ...trap/engines/engine.py.224: train: epoch 85 | batch 360/2304 [I 2022-10-27 19:49:57] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:06 [I 2022-10-27 19:49:57] ...trap/engines/engine.py.229: process: 0.54783 | load: 0.00241 [I 2022-10-27 19:49:57] ...trap/engines/engine.py.230: loss: 0.12340 [I 2022-10-27 19:49:57] ...optimizers/trijoint.py.151: total_norm: 14.877887 [I 2022-10-27 19:50:02] ...trap/engines/engine.py.224: train: epoch 85 | batch 370/2304 [I 2022-10-27 19:50:02] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:09 [I 2022-10-27 19:50:02] ...trap/engines/engine.py.229: process: 0.56723 | load: 0.00242 [I 2022-10-27 19:50:02] ...trap/engines/engine.py.230: loss: 0.10663 [I 2022-10-27 19:50:02] ...optimizers/trijoint.py.151: total_norm: 8.744864 [I 2022-10-27 19:50:08] ...trap/engines/engine.py.224: train: epoch 85 | batch 380/2304 [I 2022-10-27 19:50:08] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:18:22 [I 2022-10-27 19:50:08] ...trap/engines/engine.py.229: process: 0.63412 | load: 0.00241 [I 2022-10-27 19:50:08] ...trap/engines/engine.py.230: loss: 0.11210 [I 2022-10-27 19:50:08] ...optimizers/trijoint.py.151: total_norm: 8.651876 [I 2022-10-27 19:50:13] ...trap/engines/engine.py.224: train: epoch 85 | batch 390/2304 [I 2022-10-27 19:50:13] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:58 [I 2022-10-27 19:50:13] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00250 [I 2022-10-27 19:50:13] ...trap/engines/engine.py.230: loss: 0.12407 [I 2022-10-27 19:50:13] ...optimizers/trijoint.py.151: total_norm: 13.941714 [I 2022-10-27 19:50:39] ...trap/engines/engine.py.224: train: epoch 85 | batch 400/2304 [I 2022-10-27 19:50:39] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:58:24 [I 2022-10-27 19:50:39] ...trap/engines/engine.py.229: process: 0.54803 | load: 0.00238 [I 2022-10-27 19:50:39] ...trap/engines/engine.py.230: loss: 0.17131 [I 2022-10-27 19:50:39] ...optimizers/trijoint.py.151: total_norm: 10.858315 [I 2022-10-27 19:50:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 410/2304 [I 2022-10-27 19:50:44] ...trap/engines/engine.py.228: elapsed: 0:04:12 | left: 0:22:00 [I 2022-10-27 19:50:44] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00246 [I 2022-10-27 19:50:44] ...trap/engines/engine.py.230: loss: 0.11898 [I 2022-10-27 19:50:44] ...optimizers/trijoint.py.151: total_norm: 16.758963 [I 2022-10-27 19:50:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 420/2304 [I 2022-10-27 19:50:50] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:18:03 [I 2022-10-27 19:50:50] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00257 [I 2022-10-27 19:50:50] ...trap/engines/engine.py.230: loss: 0.12662 [I 2022-10-27 19:50:50] ...optimizers/trijoint.py.151: total_norm: 17.752625 [I 2022-10-27 19:50:55] ...trap/engines/engine.py.224: train: epoch 85 | batch 430/2304 [I 2022-10-27 19:50:55] ...trap/engines/engine.py.228: elapsed: 0:04:23 | left: 0:17:33 [I 2022-10-27 19:50:55] ...trap/engines/engine.py.229: process: 0.56243 | load: 0.00243 [I 2022-10-27 19:50:55] ...trap/engines/engine.py.230: loss: 0.15373 [I 2022-10-27 19:50:55] ...optimizers/trijoint.py.151: total_norm: 17.140123 [I 2022-10-27 19:51:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 440/2304 [I 2022-10-27 19:51:01] ...trap/engines/engine.py.228: elapsed: 0:04:29 | left: 0:17:23 [I 2022-10-27 19:51:01] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00252 [I 2022-10-27 19:51:01] ...trap/engines/engine.py.230: loss: 0.10510 [I 2022-10-27 19:51:01] ...optimizers/trijoint.py.151: total_norm: 14.205822 [I 2022-10-27 19:51:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 450/2304 [I 2022-10-27 19:51:07] ...trap/engines/engine.py.228: elapsed: 0:04:34 | left: 0:17:12 [I 2022-10-27 19:51:07] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00249 [I 2022-10-27 19:51:07] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-27 19:51:07] ...optimizers/trijoint.py.151: total_norm: 25.581612 [I 2022-10-27 19:51:12] ...trap/engines/engine.py.224: train: epoch 85 | batch 460/2304 [I 2022-10-27 19:51:12] ...trap/engines/engine.py.228: elapsed: 0:04:40 | left: 0:17:05 [I 2022-10-27 19:51:12] ...trap/engines/engine.py.229: process: 0.55611 | load: 0.00239 [I 2022-10-27 19:51:12] ...trap/engines/engine.py.230: loss: 0.13276 [I 2022-10-27 19:51:12] ...optimizers/trijoint.py.151: total_norm: 20.043007 [I 2022-10-27 19:51:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 470/2304 [I 2022-10-27 19:51:18] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:17:02 [I 2022-10-27 19:51:18] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00243 [I 2022-10-27 19:51:18] ...trap/engines/engine.py.230: loss: 0.14641 [I 2022-10-27 19:51:18] ...optimizers/trijoint.py.151: total_norm: 29.271282 [I 2022-10-27 19:51:24] ...trap/engines/engine.py.224: train: epoch 85 | batch 480/2304 [I 2022-10-27 19:51:24] ...trap/engines/engine.py.228: elapsed: 0:04:51 | left: 0:17:10 [I 2022-10-27 19:51:24] ...trap/engines/engine.py.229: process: 0.55534 | load: 0.00241 [I 2022-10-27 19:51:24] ...trap/engines/engine.py.230: loss: 0.13478 [I 2022-10-27 19:51:24] ...optimizers/trijoint.py.151: total_norm: 24.211338 [I 2022-10-27 19:51:29] ...trap/engines/engine.py.224: train: epoch 85 | batch 490/2304 [I 2022-10-27 19:51:29] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:17:00 [I 2022-10-27 19:51:29] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00241 [I 2022-10-27 19:51:29] ...trap/engines/engine.py.230: loss: 0.13854 [I 2022-10-27 19:51:29] ...optimizers/trijoint.py.151: total_norm: 11.936937 [I 2022-10-27 19:51:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 500/2304 [I 2022-10-27 19:51:35] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:16:50 [I 2022-10-27 19:51:35] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00239 [I 2022-10-27 19:51:35] ...trap/engines/engine.py.230: loss: 0.19754 [I 2022-10-27 19:51:35] ...optimizers/trijoint.py.151: total_norm: 8.644384 [I 2022-10-27 19:51:40] ...trap/engines/engine.py.224: train: epoch 85 | batch 510/2304 [I 2022-10-27 19:51:40] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:16:43 [I 2022-10-27 19:51:40] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00243 [I 2022-10-27 19:51:40] ...trap/engines/engine.py.230: loss: 0.14631 [I 2022-10-27 19:51:40] ...optimizers/trijoint.py.151: total_norm: 26.011759 [I 2022-10-27 19:51:46] ...trap/engines/engine.py.224: train: epoch 85 | batch 520/2304 [I 2022-10-27 19:51:46] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:16:33 [I 2022-10-27 19:51:46] ...trap/engines/engine.py.229: process: 0.55562 | load: 0.00244 [I 2022-10-27 19:51:46] ...trap/engines/engine.py.230: loss: 0.14224 [I 2022-10-27 19:51:46] ...optimizers/trijoint.py.151: total_norm: 12.249129 [I 2022-10-27 19:51:52] ...trap/engines/engine.py.224: train: epoch 85 | batch 530/2304 [I 2022-10-27 19:51:52] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:16:24 [I 2022-10-27 19:51:52] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00243 [I 2022-10-27 19:51:52] ...trap/engines/engine.py.230: loss: 0.16721 [I 2022-10-27 19:51:52] ...optimizers/trijoint.py.151: total_norm: 18.830528 [I 2022-10-27 19:51:57] ...trap/engines/engine.py.224: train: epoch 85 | batch 540/2304 [I 2022-10-27 19:51:57] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:16:21 [I 2022-10-27 19:51:57] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00241 [I 2022-10-27 19:51:57] ...trap/engines/engine.py.230: loss: 0.13205 [I 2022-10-27 19:51:57] ...optimizers/trijoint.py.151: total_norm: 12.092105 [I 2022-10-27 19:52:03] ...trap/engines/engine.py.224: train: epoch 85 | batch 550/2304 [I 2022-10-27 19:52:03] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:16:19 [I 2022-10-27 19:52:03] ...trap/engines/engine.py.229: process: 0.56953 | load: 0.00244 [I 2022-10-27 19:52:03] ...trap/engines/engine.py.230: loss: 0.18452 [I 2022-10-27 19:52:03] ...optimizers/trijoint.py.151: total_norm: 32.991573 [I 2022-10-27 19:52:08] ...trap/engines/engine.py.224: train: epoch 85 | batch 560/2304 [I 2022-10-27 19:52:08] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:16:17 [I 2022-10-27 19:52:08] ...trap/engines/engine.py.229: process: 0.56325 | load: 0.00244 [I 2022-10-27 19:52:08] ...trap/engines/engine.py.230: loss: 0.08229 [I 2022-10-27 19:52:08] ...optimizers/trijoint.py.151: total_norm: 10.718617 [I 2022-10-27 19:52:14] ...trap/engines/engine.py.224: train: epoch 85 | batch 570/2304 [I 2022-10-27 19:52:14] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:16:08 [I 2022-10-27 19:52:14] ...trap/engines/engine.py.229: process: 0.56353 | load: 0.00239 [I 2022-10-27 19:52:14] ...trap/engines/engine.py.230: loss: 0.12798 [I 2022-10-27 19:52:14] ...optimizers/trijoint.py.151: total_norm: 19.707348 [I 2022-10-27 19:52:19] ...trap/engines/engine.py.224: train: epoch 85 | batch 580/2304 [I 2022-10-27 19:52:19] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:15:54 [I 2022-10-27 19:52:19] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00240 [I 2022-10-27 19:52:19] ...trap/engines/engine.py.230: loss: 0.10698 [I 2022-10-27 19:52:19] ...optimizers/trijoint.py.151: total_norm: 8.106416 [I 2022-10-27 19:52:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 590/2304 [I 2022-10-27 19:52:25] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:16:01 [I 2022-10-27 19:52:25] ...trap/engines/engine.py.229: process: 0.56890 | load: 0.00243 [I 2022-10-27 19:52:25] ...trap/engines/engine.py.230: loss: 0.13134 [I 2022-10-27 19:52:25] ...optimizers/trijoint.py.151: total_norm: 16.261059 [I 2022-10-27 19:52:31] ...trap/engines/engine.py.224: train: epoch 85 | batch 600/2304 [I 2022-10-27 19:52:31] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:51 [I 2022-10-27 19:52:31] ...trap/engines/engine.py.229: process: 0.56680 | load: 0.00246 [I 2022-10-27 19:52:31] ...trap/engines/engine.py.230: loss: 0.10432 [I 2022-10-27 19:52:31] ...optimizers/trijoint.py.151: total_norm: 14.063295 [I 2022-10-27 19:52:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 610/2304 [I 2022-10-27 19:52:36] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:44 [I 2022-10-27 19:52:36] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00244 [I 2022-10-27 19:52:36] ...trap/engines/engine.py.230: loss: 0.16502 [I 2022-10-27 19:52:36] ...optimizers/trijoint.py.151: total_norm: 57.003357 [I 2022-10-27 19:52:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 620/2304 [I 2022-10-27 19:52:42] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:45 [I 2022-10-27 19:52:42] ...trap/engines/engine.py.229: process: 0.56687 | load: 0.00240 [I 2022-10-27 19:52:42] ...trap/engines/engine.py.230: loss: 0.13958 [I 2022-10-27 19:52:42] ...optimizers/trijoint.py.151: total_norm: 9.963683 [I 2022-10-27 19:52:47] ...trap/engines/engine.py.224: train: epoch 85 | batch 630/2304 [I 2022-10-27 19:52:47] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:32 [I 2022-10-27 19:52:47] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00242 [I 2022-10-27 19:52:47] ...trap/engines/engine.py.230: loss: 0.16552 [I 2022-10-27 19:52:47] ...optimizers/trijoint.py.151: total_norm: 11.283742 [I 2022-10-27 19:52:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 640/2304 [I 2022-10-27 19:52:53] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:28 [I 2022-10-27 19:52:53] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00242 [I 2022-10-27 19:52:53] ...trap/engines/engine.py.230: loss: 0.13503 [I 2022-10-27 19:52:53] ...optimizers/trijoint.py.151: total_norm: 24.966204 [I 2022-10-27 19:52:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 650/2304 [I 2022-10-27 19:52:59] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:18 [I 2022-10-27 19:52:59] ...trap/engines/engine.py.229: process: 0.54767 | load: 0.00240 [I 2022-10-27 19:52:59] ...trap/engines/engine.py.230: loss: 0.09077 [I 2022-10-27 19:52:59] ...optimizers/trijoint.py.151: total_norm: 7.932222 [I 2022-10-27 19:53:04] ...trap/engines/engine.py.224: train: epoch 85 | batch 660/2304 [I 2022-10-27 19:53:04] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:15:14 [I 2022-10-27 19:53:04] ...trap/engines/engine.py.229: process: 0.55587 | load: 0.00240 [I 2022-10-27 19:53:04] ...trap/engines/engine.py.230: loss: 0.15493 [I 2022-10-27 19:53:04] ...optimizers/trijoint.py.151: total_norm: 12.693315 [I 2022-10-27 19:53:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 670/2304 [I 2022-10-27 19:53:10] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:12 [I 2022-10-27 19:53:10] ...trap/engines/engine.py.229: process: 0.56890 | load: 0.00281 [I 2022-10-27 19:53:10] ...trap/engines/engine.py.230: loss: 0.14006 [I 2022-10-27 19:53:10] ...optimizers/trijoint.py.151: total_norm: 19.485954 [I 2022-10-27 19:53:15] ...trap/engines/engine.py.224: train: epoch 85 | batch 680/2304 [I 2022-10-27 19:53:15] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:04 [I 2022-10-27 19:53:15] ...trap/engines/engine.py.229: process: 0.55302 | load: 0.00241 [I 2022-10-27 19:53:15] ...trap/engines/engine.py.230: loss: 0.11022 [I 2022-10-27 19:53:15] ...optimizers/trijoint.py.151: total_norm: 19.503969 [I 2022-10-27 19:53:21] ...trap/engines/engine.py.224: train: epoch 85 | batch 690/2304 [I 2022-10-27 19:53:21] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:14:59 [I 2022-10-27 19:53:21] ...trap/engines/engine.py.229: process: 0.56524 | load: 0.00250 [I 2022-10-27 19:53:21] ...trap/engines/engine.py.230: loss: 0.09090 [I 2022-10-27 19:53:21] ...optimizers/trijoint.py.151: total_norm: 12.935269 [I 2022-10-27 19:53:26] ...trap/engines/engine.py.224: train: epoch 85 | batch 700/2304 [I 2022-10-27 19:53:26] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:14:56 [I 2022-10-27 19:53:26] ...trap/engines/engine.py.229: process: 0.56479 | load: 0.00246 [I 2022-10-27 19:53:26] ...trap/engines/engine.py.230: loss: 0.17449 [I 2022-10-27 19:53:26] ...optimizers/trijoint.py.151: total_norm: 18.782343 [I 2022-10-27 19:53:32] ...trap/engines/engine.py.224: train: epoch 85 | batch 710/2304 [I 2022-10-27 19:53:32] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:49 [I 2022-10-27 19:53:32] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00244 [I 2022-10-27 19:53:32] ...trap/engines/engine.py.230: loss: 0.07901 [I 2022-10-27 19:53:32] ...optimizers/trijoint.py.151: total_norm: 10.163215 [I 2022-10-27 19:53:38] ...trap/engines/engine.py.224: train: epoch 85 | batch 720/2304 [I 2022-10-27 19:53:38] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:47 [I 2022-10-27 19:53:38] ...trap/engines/engine.py.229: process: 0.56194 | load: 0.00241 [I 2022-10-27 19:53:38] ...trap/engines/engine.py.230: loss: 0.13733 [I 2022-10-27 19:53:38] ...optimizers/trijoint.py.151: total_norm: 22.127975 [I 2022-10-27 19:53:43] ...trap/engines/engine.py.224: train: epoch 85 | batch 730/2304 [I 2022-10-27 19:53:43] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:41 [I 2022-10-27 19:53:43] ...trap/engines/engine.py.229: process: 0.56467 | load: 0.00243 [I 2022-10-27 19:53:43] ...trap/engines/engine.py.230: loss: 0.11436 [I 2022-10-27 19:53:43] ...optimizers/trijoint.py.151: total_norm: 23.713181 [I 2022-10-27 19:53:49] ...trap/engines/engine.py.224: train: epoch 85 | batch 740/2304 [I 2022-10-27 19:53:49] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:34 [I 2022-10-27 19:53:49] ...trap/engines/engine.py.229: process: 0.56198 | load: 0.00245 [I 2022-10-27 19:53:49] ...trap/engines/engine.py.230: loss: 0.09635 [I 2022-10-27 19:53:49] ...optimizers/trijoint.py.151: total_norm: 13.748567 [I 2022-10-27 19:53:54] ...trap/engines/engine.py.224: train: epoch 85 | batch 750/2304 [I 2022-10-27 19:53:54] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:22 [I 2022-10-27 19:53:54] ...trap/engines/engine.py.229: process: 0.56499 | load: 0.00247 [I 2022-10-27 19:53:54] ...trap/engines/engine.py.230: loss: 0.13256 [I 2022-10-27 19:53:54] ...optimizers/trijoint.py.151: total_norm: 15.744812 [I 2022-10-27 19:54:00] ...trap/engines/engine.py.224: train: epoch 85 | batch 760/2304 [I 2022-10-27 19:54:00] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:21 [I 2022-10-27 19:54:00] ...trap/engines/engine.py.229: process: 0.56826 | load: 0.00243 [I 2022-10-27 19:54:00] ...trap/engines/engine.py.230: loss: 0.23118 [I 2022-10-27 19:54:00] ...optimizers/trijoint.py.151: total_norm: 29.895491 [I 2022-10-27 19:54:06] ...trap/engines/engine.py.224: train: epoch 85 | batch 770/2304 [I 2022-10-27 19:54:06] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:14:14 [I 2022-10-27 19:54:06] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00246 [I 2022-10-27 19:54:06] ...trap/engines/engine.py.230: loss: 0.20581 [I 2022-10-27 19:54:06] ...optimizers/trijoint.py.151: total_norm: 22.062775 [I 2022-10-27 19:54:11] ...trap/engines/engine.py.224: train: epoch 85 | batch 780/2304 [I 2022-10-27 19:54:11] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:05 [I 2022-10-27 19:54:11] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00249 [I 2022-10-27 19:54:11] ...trap/engines/engine.py.230: loss: 0.15936 [I 2022-10-27 19:54:11] ...optimizers/trijoint.py.151: total_norm: 25.079166 [I 2022-10-27 19:54:17] ...trap/engines/engine.py.224: train: epoch 85 | batch 790/2304 [I 2022-10-27 19:54:17] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:13:52 [I 2022-10-27 19:54:17] ...trap/engines/engine.py.229: process: 0.53706 | load: 0.00246 [I 2022-10-27 19:54:17] ...trap/engines/engine.py.230: loss: 0.11529 [I 2022-10-27 19:54:17] ...optimizers/trijoint.py.151: total_norm: 15.364027 [I 2022-10-27 19:54:22] ...trap/engines/engine.py.224: train: epoch 85 | batch 800/2304 [I 2022-10-27 19:54:22] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:13:56 [I 2022-10-27 19:54:22] ...trap/engines/engine.py.229: process: 0.54216 | load: 0.00242 [I 2022-10-27 19:54:22] ...trap/engines/engine.py.230: loss: 0.10780 [I 2022-10-27 19:54:22] ...optimizers/trijoint.py.151: total_norm: 12.647152 [I 2022-10-27 19:54:28] ...trap/engines/engine.py.224: train: epoch 85 | batch 810/2304 [I 2022-10-27 19:54:28] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:13:50 [I 2022-10-27 19:54:28] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00244 [I 2022-10-27 19:54:28] ...trap/engines/engine.py.230: loss: 0.15884 [I 2022-10-27 19:54:28] ...optimizers/trijoint.py.151: total_norm: 46.945721 [I 2022-10-27 19:54:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 820/2304 [I 2022-10-27 19:54:33] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:13:44 [I 2022-10-27 19:54:33] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00242 [I 2022-10-27 19:54:33] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-27 19:54:33] ...optimizers/trijoint.py.151: total_norm: 18.093956 [I 2022-10-27 19:54:39] ...trap/engines/engine.py.224: train: epoch 85 | batch 830/2304 [I 2022-10-27 19:54:39] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:13:42 [I 2022-10-27 19:54:39] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00266 [I 2022-10-27 19:54:39] ...trap/engines/engine.py.230: loss: 0.12923 [I 2022-10-27 19:54:39] ...optimizers/trijoint.py.151: total_norm: 18.090927 [I 2022-10-27 19:54:45] ...trap/engines/engine.py.224: train: epoch 85 | batch 840/2304 [I 2022-10-27 19:54:45] ...trap/engines/engine.py.228: elapsed: 0:08:12 | left: 0:13:39 [I 2022-10-27 19:54:45] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00257 [I 2022-10-27 19:54:45] ...trap/engines/engine.py.230: loss: 0.13398 [I 2022-10-27 19:54:45] ...optimizers/trijoint.py.151: total_norm: 15.695210 [I 2022-10-27 19:54:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 850/2304 [I 2022-10-27 19:54:50] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:34 [I 2022-10-27 19:54:50] ...trap/engines/engine.py.229: process: 0.56835 | load: 0.00251 [I 2022-10-27 19:54:50] ...trap/engines/engine.py.230: loss: 0.14584 [I 2022-10-27 19:54:50] ...optimizers/trijoint.py.151: total_norm: 13.915157 [I 2022-10-27 19:54:56] ...trap/engines/engine.py.224: train: epoch 85 | batch 860/2304 [I 2022-10-27 19:54:56] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:13:21 [I 2022-10-27 19:54:56] ...trap/engines/engine.py.229: process: 0.54925 | load: 0.00251 [I 2022-10-27 19:54:56] ...trap/engines/engine.py.230: loss: 0.10283 [I 2022-10-27 19:54:56] ...optimizers/trijoint.py.151: total_norm: 14.892601 [I 2022-10-27 19:55:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 870/2304 [I 2022-10-27 19:55:01] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:13:22 [I 2022-10-27 19:55:01] ...trap/engines/engine.py.229: process: 0.56987 | load: 0.00249 [I 2022-10-27 19:55:01] ...trap/engines/engine.py.230: loss: 0.13345 [I 2022-10-27 19:55:01] ...optimizers/trijoint.py.151: total_norm: 18.326769 [I 2022-10-27 19:55:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 880/2304 [I 2022-10-27 19:55:07] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:13:06 [I 2022-10-27 19:55:07] ...trap/engines/engine.py.229: process: 0.53785 | load: 0.00270 [I 2022-10-27 19:55:07] ...trap/engines/engine.py.230: loss: 0.21039 [I 2022-10-27 19:55:07] ...optimizers/trijoint.py.151: total_norm: 32.725224 [I 2022-10-27 19:55:12] ...trap/engines/engine.py.224: train: epoch 85 | batch 890/2304 [I 2022-10-27 19:55:12] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:12:59 [I 2022-10-27 19:55:12] ...trap/engines/engine.py.229: process: 0.56101 | load: 0.00241 [I 2022-10-27 19:55:12] ...trap/engines/engine.py.230: loss: 0.14367 [I 2022-10-27 19:55:12] ...optimizers/trijoint.py.151: total_norm: 19.135813 [I 2022-10-27 19:55:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 900/2304 [I 2022-10-27 19:55:18] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:12:59 [I 2022-10-27 19:55:18] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00248 [I 2022-10-27 19:55:18] ...trap/engines/engine.py.230: loss: 0.12987 [I 2022-10-27 19:55:18] ...optimizers/trijoint.py.151: total_norm: 34.004330 [I 2022-10-27 19:55:24] ...trap/engines/engine.py.224: train: epoch 85 | batch 910/2304 [I 2022-10-27 19:55:24] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:12:59 [I 2022-10-27 19:55:24] ...trap/engines/engine.py.229: process: 0.56395 | load: 0.00246 [I 2022-10-27 19:55:24] ...trap/engines/engine.py.230: loss: 0.12471 [I 2022-10-27 19:55:24] ...optimizers/trijoint.py.151: total_norm: 25.860319 [I 2022-10-27 19:55:29] ...trap/engines/engine.py.224: train: epoch 85 | batch 920/2304 [I 2022-10-27 19:55:29] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:48 [I 2022-10-27 19:55:29] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00242 [I 2022-10-27 19:55:29] ...trap/engines/engine.py.230: loss: 0.14470 [I 2022-10-27 19:55:29] ...optimizers/trijoint.py.151: total_norm: 44.154099 [I 2022-10-27 19:55:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 930/2304 [I 2022-10-27 19:55:35] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:12:40 [I 2022-10-27 19:55:35] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00250 [I 2022-10-27 19:55:35] ...trap/engines/engine.py.230: loss: 0.14722 [I 2022-10-27 19:55:35] ...optimizers/trijoint.py.151: total_norm: 12.817770 [I 2022-10-27 19:55:40] ...trap/engines/engine.py.224: train: epoch 85 | batch 940/2304 [I 2022-10-27 19:55:40] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:12:34 [I 2022-10-27 19:55:40] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00243 [I 2022-10-27 19:55:40] ...trap/engines/engine.py.230: loss: 0.11477 [I 2022-10-27 19:55:40] ...optimizers/trijoint.py.151: total_norm: 10.983828 [I 2022-10-27 19:55:46] ...trap/engines/engine.py.224: train: epoch 85 | batch 950/2304 [I 2022-10-27 19:55:46] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:29 [I 2022-10-27 19:55:46] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00242 [I 2022-10-27 19:55:46] ...trap/engines/engine.py.230: loss: 0.15252 [I 2022-10-27 19:55:46] ...optimizers/trijoint.py.151: total_norm: 16.858465 [I 2022-10-27 19:55:51] ...trap/engines/engine.py.224: train: epoch 85 | batch 960/2304 [I 2022-10-27 19:55:51] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:12:31 [I 2022-10-27 19:55:51] ...trap/engines/engine.py.229: process: 0.56374 | load: 0.00242 [I 2022-10-27 19:55:51] ...trap/engines/engine.py.230: loss: 0.11296 [I 2022-10-27 19:55:51] ...optimizers/trijoint.py.151: total_norm: 39.083679 [I 2022-10-27 19:55:57] ...trap/engines/engine.py.224: train: epoch 85 | batch 970/2304 [I 2022-10-27 19:55:57] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:24 [I 2022-10-27 19:55:57] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00248 [I 2022-10-27 19:55:57] ...trap/engines/engine.py.230: loss: 0.12976 [I 2022-10-27 19:55:57] ...optimizers/trijoint.py.151: total_norm: 13.620896 [I 2022-10-27 19:56:03] ...trap/engines/engine.py.224: train: epoch 85 | batch 980/2304 [I 2022-10-27 19:56:03] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:13 [I 2022-10-27 19:56:03] ...trap/engines/engine.py.229: process: 0.54800 | load: 0.00245 [I 2022-10-27 19:56:03] ...trap/engines/engine.py.230: loss: 0.19463 [I 2022-10-27 19:56:03] ...optimizers/trijoint.py.151: total_norm: 18.139633 [I 2022-10-27 19:56:08] ...trap/engines/engine.py.224: train: epoch 85 | batch 990/2304 [I 2022-10-27 19:56:08] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:12:12 [I 2022-10-27 19:56:08] ...trap/engines/engine.py.229: process: 0.54932 | load: 0.00241 [I 2022-10-27 19:56:08] ...trap/engines/engine.py.230: loss: 0.11246 [I 2022-10-27 19:56:08] ...optimizers/trijoint.py.151: total_norm: 7.835809 [I 2022-10-27 19:56:14] ...trap/engines/engine.py.224: train: epoch 85 | batch 1000/2304 [I 2022-10-27 19:56:14] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:12:02 [I 2022-10-27 19:56:14] ...trap/engines/engine.py.229: process: 0.55790 | load: 0.00242 [I 2022-10-27 19:56:14] ...trap/engines/engine.py.230: loss: 0.10994 [I 2022-10-27 19:56:14] ...optimizers/trijoint.py.151: total_norm: 34.697166 [I 2022-10-27 19:56:19] ...trap/engines/engine.py.224: train: epoch 85 | batch 1010/2304 [I 2022-10-27 19:56:19] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:11:59 [I 2022-10-27 19:56:19] ...trap/engines/engine.py.229: process: 0.55737 | load: 0.00244 [I 2022-10-27 19:56:19] ...trap/engines/engine.py.230: loss: 0.18330 [I 2022-10-27 19:56:19] ...optimizers/trijoint.py.151: total_norm: 10.545705 [I 2022-10-27 19:56:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 1020/2304 [I 2022-10-27 19:56:25] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:12:21 [I 2022-10-27 19:56:25] ...trap/engines/engine.py.229: process: 0.57210 | load: 0.00264 [I 2022-10-27 19:56:25] ...trap/engines/engine.py.230: loss: 0.10350 [I 2022-10-27 19:56:25] ...optimizers/trijoint.py.151: total_norm: 10.374245 [I 2022-10-27 19:56:31] ...trap/engines/engine.py.224: train: epoch 85 | batch 1030/2304 [I 2022-10-27 19:56:31] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:52 [I 2022-10-27 19:56:31] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00242 [I 2022-10-27 19:56:31] ...trap/engines/engine.py.230: loss: 0.18018 [I 2022-10-27 19:56:31] ...optimizers/trijoint.py.151: total_norm: 12.585101 [I 2022-10-27 19:56:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 1040/2304 [I 2022-10-27 19:56:36] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:48 [I 2022-10-27 19:56:36] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00241 [I 2022-10-27 19:56:36] ...trap/engines/engine.py.230: loss: 0.17994 [I 2022-10-27 19:56:36] ...optimizers/trijoint.py.151: total_norm: 34.815704 [I 2022-10-27 19:56:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 1050/2304 [I 2022-10-27 19:56:42] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:41 [I 2022-10-27 19:56:42] ...trap/engines/engine.py.229: process: 0.56593 | load: 0.00247 [I 2022-10-27 19:56:42] ...trap/engines/engine.py.230: loss: 0.17085 [I 2022-10-27 19:56:42] ...optimizers/trijoint.py.151: total_norm: 14.007168 [I 2022-10-27 19:56:47] ...trap/engines/engine.py.224: train: epoch 85 | batch 1060/2304 [I 2022-10-27 19:56:47] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:34 [I 2022-10-27 19:56:47] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00241 [I 2022-10-27 19:56:47] ...trap/engines/engine.py.230: loss: 0.11945 [I 2022-10-27 19:56:47] ...optimizers/trijoint.py.151: total_norm: 12.860559 [I 2022-10-27 19:56:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 1070/2304 [I 2022-10-27 19:56:53] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:25 [I 2022-10-27 19:56:53] ...trap/engines/engine.py.229: process: 0.55431 | load: 0.00241 [I 2022-10-27 19:56:53] ...trap/engines/engine.py.230: loss: 0.24246 [I 2022-10-27 19:56:53] ...optimizers/trijoint.py.151: total_norm: 37.747124 [I 2022-10-27 19:56:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 1080/2304 [I 2022-10-27 19:56:59] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:16 [I 2022-10-27 19:56:59] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00242 [I 2022-10-27 19:56:59] ...trap/engines/engine.py.230: loss: 0.10088 [I 2022-10-27 19:56:59] ...optimizers/trijoint.py.151: total_norm: 9.441702 [I 2022-10-27 19:57:04] ...trap/engines/engine.py.224: train: epoch 85 | batch 1090/2304 [I 2022-10-27 19:57:04] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:16 [I 2022-10-27 19:57:04] ...trap/engines/engine.py.229: process: 0.56286 | load: 0.00246 [I 2022-10-27 19:57:04] ...trap/engines/engine.py.230: loss: 0.14550 [I 2022-10-27 19:57:04] ...optimizers/trijoint.py.151: total_norm: 20.168083 [I 2022-10-27 19:57:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 1100/2304 [I 2022-10-27 19:57:10] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:13 [I 2022-10-27 19:57:10] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00241 [I 2022-10-27 19:57:10] ...trap/engines/engine.py.230: loss: 0.12991 [I 2022-10-27 19:57:10] ...optimizers/trijoint.py.151: total_norm: 16.941994 [I 2022-10-27 19:57:15] ...trap/engines/engine.py.224: train: epoch 85 | batch 1110/2304 [I 2022-10-27 19:57:15] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:11:14 [I 2022-10-27 19:57:15] ...trap/engines/engine.py.229: process: 0.55632 | load: 0.00246 [I 2022-10-27 19:57:15] ...trap/engines/engine.py.230: loss: 0.14926 [I 2022-10-27 19:57:15] ...optimizers/trijoint.py.151: total_norm: 20.278244 [I 2022-10-27 19:57:21] ...trap/engines/engine.py.224: train: epoch 85 | batch 1120/2304 [I 2022-10-27 19:57:21] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:10:58 [I 2022-10-27 19:57:21] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00252 [I 2022-10-27 19:57:21] ...trap/engines/engine.py.230: loss: 0.10069 [I 2022-10-27 19:57:21] ...optimizers/trijoint.py.151: total_norm: 9.850271 [I 2022-10-27 19:57:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 1130/2304 [I 2022-10-27 19:57:27] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:52 [I 2022-10-27 19:57:27] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00249 [I 2022-10-27 19:57:27] ...trap/engines/engine.py.230: loss: 0.08533 [I 2022-10-27 19:57:27] ...optimizers/trijoint.py.151: total_norm: 8.725876 [I 2022-10-27 19:57:32] ...trap/engines/engine.py.224: train: epoch 85 | batch 1140/2304 [I 2022-10-27 19:57:32] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:45 [I 2022-10-27 19:57:32] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00244 [I 2022-10-27 19:57:32] ...trap/engines/engine.py.230: loss: 0.12754 [I 2022-10-27 19:57:32] ...optimizers/trijoint.py.151: total_norm: 8.662745 [I 2022-10-27 19:57:38] ...trap/engines/engine.py.224: train: epoch 85 | batch 1150/2304 [I 2022-10-27 19:57:38] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:43 [I 2022-10-27 19:57:38] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00248 [I 2022-10-27 19:57:38] ...trap/engines/engine.py.230: loss: 0.10519 [I 2022-10-27 19:57:38] ...optimizers/trijoint.py.151: total_norm: 13.326474 [I 2022-10-27 19:57:43] ...trap/engines/engine.py.224: train: epoch 85 | batch 1160/2304 [I 2022-10-27 19:57:43] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:37 [I 2022-10-27 19:57:43] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00245 [I 2022-10-27 19:57:43] ...trap/engines/engine.py.230: loss: 0.11696 [I 2022-10-27 19:57:43] ...optimizers/trijoint.py.151: total_norm: 14.861839 [I 2022-10-27 19:57:49] ...trap/engines/engine.py.224: train: epoch 85 | batch 1170/2304 [I 2022-10-27 19:57:49] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:27 [I 2022-10-27 19:57:49] ...trap/engines/engine.py.229: process: 0.54818 | load: 0.00245 [I 2022-10-27 19:57:49] ...trap/engines/engine.py.230: loss: 0.16192 [I 2022-10-27 19:57:49] ...optimizers/trijoint.py.151: total_norm: 14.248823 [I 2022-10-27 19:57:55] ...trap/engines/engine.py.224: train: epoch 85 | batch 1180/2304 [I 2022-10-27 19:57:55] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:31 [I 2022-10-27 19:57:55] ...trap/engines/engine.py.229: process: 0.56245 | load: 0.00245 [I 2022-10-27 19:57:55] ...trap/engines/engine.py.230: loss: 0.13637 [I 2022-10-27 19:57:55] ...optimizers/trijoint.py.151: total_norm: 30.128494 [I 2022-10-27 19:58:00] ...trap/engines/engine.py.224: train: epoch 85 | batch 1190/2304 [I 2022-10-27 19:58:00] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:21 [I 2022-10-27 19:58:00] ...trap/engines/engine.py.229: process: 0.56642 | load: 0.00243 [I 2022-10-27 19:58:00] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-27 19:58:00] ...optimizers/trijoint.py.151: total_norm: 12.885127 [I 2022-10-27 19:58:06] ...trap/engines/engine.py.224: train: epoch 85 | batch 1200/2304 [I 2022-10-27 19:58:06] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:09 [I 2022-10-27 19:58:06] ...trap/engines/engine.py.229: process: 0.54022 | load: 0.00245 [I 2022-10-27 19:58:06] ...trap/engines/engine.py.230: loss: 0.13888 [I 2022-10-27 19:58:06] ...optimizers/trijoint.py.151: total_norm: 10.969837 [I 2022-10-27 19:58:11] ...trap/engines/engine.py.224: train: epoch 85 | batch 1210/2304 [I 2022-10-27 19:58:11] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:10:10 [I 2022-10-27 19:58:11] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00252 [I 2022-10-27 19:58:11] ...trap/engines/engine.py.230: loss: 0.08883 [I 2022-10-27 19:58:11] ...optimizers/trijoint.py.151: total_norm: 11.153304 [I 2022-10-27 19:58:17] ...trap/engines/engine.py.224: train: epoch 85 | batch 1220/2304 [I 2022-10-27 19:58:17] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:10:03 [I 2022-10-27 19:58:17] ...trap/engines/engine.py.229: process: 0.56326 | load: 0.00241 [I 2022-10-27 19:58:17] ...trap/engines/engine.py.230: loss: 0.12404 [I 2022-10-27 19:58:17] ...optimizers/trijoint.py.151: total_norm: 27.465273 [I 2022-10-27 19:58:22] ...trap/engines/engine.py.224: train: epoch 85 | batch 1230/2304 [I 2022-10-27 19:58:22] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:10:00 [I 2022-10-27 19:58:22] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00256 [I 2022-10-27 19:58:22] ...trap/engines/engine.py.230: loss: 0.17675 [I 2022-10-27 19:58:22] ...optimizers/trijoint.py.151: total_norm: 16.974989 [I 2022-10-27 19:58:28] ...trap/engines/engine.py.224: train: epoch 85 | batch 1240/2304 [I 2022-10-27 19:58:28] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:50 [I 2022-10-27 19:58:28] ...trap/engines/engine.py.229: process: 0.54551 | load: 0.00248 [I 2022-10-27 19:58:28] ...trap/engines/engine.py.230: loss: 0.08304 [I 2022-10-27 19:58:28] ...optimizers/trijoint.py.151: total_norm: 9.084658 [I 2022-10-27 19:58:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 1250/2304 [I 2022-10-27 19:58:33] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:44 [I 2022-10-27 19:58:33] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00254 [I 2022-10-27 19:58:33] ...trap/engines/engine.py.230: loss: 0.11269 [I 2022-10-27 19:58:33] ...optimizers/trijoint.py.151: total_norm: 18.438019 [I 2022-10-27 19:58:39] ...trap/engines/engine.py.224: train: epoch 85 | batch 1260/2304 [I 2022-10-27 19:58:39] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:41 [I 2022-10-27 19:58:39] ...trap/engines/engine.py.229: process: 0.57101 | load: 0.00288 [I 2022-10-27 19:58:39] ...trap/engines/engine.py.230: loss: 0.15783 [I 2022-10-27 19:58:39] ...optimizers/trijoint.py.151: total_norm: 14.972213 [I 2022-10-27 19:58:45] ...trap/engines/engine.py.224: train: epoch 85 | batch 1270/2304 [I 2022-10-27 19:58:45] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:30 [I 2022-10-27 19:58:45] ...trap/engines/engine.py.229: process: 0.54442 | load: 0.00253 [I 2022-10-27 19:58:45] ...trap/engines/engine.py.230: loss: 0.12409 [I 2022-10-27 19:58:45] ...optimizers/trijoint.py.151: total_norm: 17.177353 [I 2022-10-27 19:58:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 1280/2304 [I 2022-10-27 19:58:50] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:24 [I 2022-10-27 19:58:50] ...trap/engines/engine.py.229: process: 0.53739 | load: 0.00246 [I 2022-10-27 19:58:50] ...trap/engines/engine.py.230: loss: 0.09072 [I 2022-10-27 19:58:50] ...optimizers/trijoint.py.151: total_norm: 9.167721 [I 2022-10-27 19:58:56] ...trap/engines/engine.py.224: train: epoch 85 | batch 1290/2304 [I 2022-10-27 19:58:56] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:27 [I 2022-10-27 19:58:56] ...trap/engines/engine.py.229: process: 0.56523 | load: 0.00251 [I 2022-10-27 19:58:56] ...trap/engines/engine.py.230: loss: 0.09291 [I 2022-10-27 19:58:56] ...optimizers/trijoint.py.151: total_norm: 12.176273 [I 2022-10-27 19:59:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 1300/2304 [I 2022-10-27 19:59:01] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:18 [I 2022-10-27 19:59:01] ...trap/engines/engine.py.229: process: 0.55854 | load: 0.00248 [I 2022-10-27 19:59:01] ...trap/engines/engine.py.230: loss: 0.21028 [I 2022-10-27 19:59:01] ...optimizers/trijoint.py.151: total_norm: 31.919174 [I 2022-10-27 19:59:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 1310/2304 [I 2022-10-27 19:59:07] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:09:15 [I 2022-10-27 19:59:07] ...trap/engines/engine.py.229: process: 0.57271 | load: 0.00252 [I 2022-10-27 19:59:07] ...trap/engines/engine.py.230: loss: 0.09879 [I 2022-10-27 19:59:07] ...optimizers/trijoint.py.151: total_norm: 10.798050 [I 2022-10-27 19:59:12] ...trap/engines/engine.py.224: train: epoch 85 | batch 1320/2304 [I 2022-10-27 19:59:12] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:04 [I 2022-10-27 19:59:12] ...trap/engines/engine.py.229: process: 0.55104 | load: 0.00246 [I 2022-10-27 19:59:12] ...trap/engines/engine.py.230: loss: 0.10130 [I 2022-10-27 19:59:12] ...optimizers/trijoint.py.151: total_norm: 16.367685 [I 2022-10-27 19:59:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 1330/2304 [I 2022-10-27 19:59:18] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:08:58 [I 2022-10-27 19:59:18] ...trap/engines/engine.py.229: process: 0.55181 | load: 0.00264 [I 2022-10-27 19:59:18] ...trap/engines/engine.py.230: loss: 0.13381 [I 2022-10-27 19:59:18] ...optimizers/trijoint.py.151: total_norm: 41.392876 [I 2022-10-27 19:59:24] ...trap/engines/engine.py.224: train: epoch 85 | batch 1340/2304 [I 2022-10-27 19:59:24] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:55 [I 2022-10-27 19:59:24] ...trap/engines/engine.py.229: process: 0.56342 | load: 0.00266 [I 2022-10-27 19:59:24] ...trap/engines/engine.py.230: loss: 0.12137 [I 2022-10-27 19:59:24] ...optimizers/trijoint.py.151: total_norm: 19.779974 [I 2022-10-27 19:59:29] ...trap/engines/engine.py.224: train: epoch 85 | batch 1350/2304 [I 2022-10-27 19:59:29] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:53 [I 2022-10-27 19:59:29] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00252 [I 2022-10-27 19:59:29] ...trap/engines/engine.py.230: loss: 0.14355 [I 2022-10-27 19:59:29] ...optimizers/trijoint.py.151: total_norm: 21.584894 [I 2022-10-27 19:59:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 1360/2304 [I 2022-10-27 19:59:35] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:44 [I 2022-10-27 19:59:35] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00249 [I 2022-10-27 19:59:35] ...trap/engines/engine.py.230: loss: 0.10252 [I 2022-10-27 19:59:35] ...optimizers/trijoint.py.151: total_norm: 11.023956 [I 2022-10-27 19:59:40] ...trap/engines/engine.py.224: train: epoch 85 | batch 1370/2304 [I 2022-10-27 19:59:40] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:39 [I 2022-10-27 19:59:40] ...trap/engines/engine.py.229: process: 0.56348 | load: 0.00252 [I 2022-10-27 19:59:40] ...trap/engines/engine.py.230: loss: 0.12222 [I 2022-10-27 19:59:40] ...optimizers/trijoint.py.151: total_norm: 14.260425 [I 2022-10-27 19:59:46] ...trap/engines/engine.py.224: train: epoch 85 | batch 1380/2304 [I 2022-10-27 19:59:46] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:36 [I 2022-10-27 19:59:46] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00249 [I 2022-10-27 19:59:46] ...trap/engines/engine.py.230: loss: 0.10246 [I 2022-10-27 19:59:46] ...optimizers/trijoint.py.151: total_norm: 11.666607 [I 2022-10-27 19:59:52] ...trap/engines/engine.py.224: train: epoch 85 | batch 1390/2304 [I 2022-10-27 19:59:52] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:35 [I 2022-10-27 19:59:52] ...trap/engines/engine.py.229: process: 0.54171 | load: 0.00248 [I 2022-10-27 19:59:52] ...trap/engines/engine.py.230: loss: 0.10775 [I 2022-10-27 19:59:52] ...optimizers/trijoint.py.151: total_norm: 14.669889 [I 2022-10-27 19:59:57] ...trap/engines/engine.py.224: train: epoch 85 | batch 1400/2304 [I 2022-10-27 19:59:57] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:25 [I 2022-10-27 19:59:57] ...trap/engines/engine.py.229: process: 0.55828 | load: 0.00258 [I 2022-10-27 19:59:57] ...trap/engines/engine.py.230: loss: 0.16529 [I 2022-10-27 19:59:57] ...optimizers/trijoint.py.151: total_norm: 15.024262 [I 2022-10-27 20:00:03] ...trap/engines/engine.py.224: train: epoch 85 | batch 1410/2304 [I 2022-10-27 20:00:03] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:19 [I 2022-10-27 20:00:03] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00243 [I 2022-10-27 20:00:03] ...trap/engines/engine.py.230: loss: 0.11203 [I 2022-10-27 20:00:03] ...optimizers/trijoint.py.151: total_norm: 18.323864 [I 2022-10-27 20:00:08] ...trap/engines/engine.py.224: train: epoch 85 | batch 1420/2304 [I 2022-10-27 20:00:08] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:10 [I 2022-10-27 20:00:08] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00253 [I 2022-10-27 20:00:08] ...trap/engines/engine.py.230: loss: 0.13238 [I 2022-10-27 20:00:08] ...optimizers/trijoint.py.151: total_norm: 16.898499 [I 2022-10-27 20:00:14] ...trap/engines/engine.py.224: train: epoch 85 | batch 1430/2304 [I 2022-10-27 20:00:14] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:08:04 [I 2022-10-27 20:00:14] ...trap/engines/engine.py.229: process: 0.56772 | load: 0.00247 [I 2022-10-27 20:00:14] ...trap/engines/engine.py.230: loss: 0.12074 [I 2022-10-27 20:00:14] ...optimizers/trijoint.py.151: total_norm: 18.699154 [I 2022-10-27 20:00:19] ...trap/engines/engine.py.224: train: epoch 85 | batch 1440/2304 [I 2022-10-27 20:00:19] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:08:00 [I 2022-10-27 20:00:19] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00249 [I 2022-10-27 20:00:19] ...trap/engines/engine.py.230: loss: 0.09538 [I 2022-10-27 20:00:19] ...optimizers/trijoint.py.151: total_norm: 9.647267 [I 2022-10-27 20:00:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 1450/2304 [I 2022-10-27 20:00:25] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:52 [I 2022-10-27 20:00:25] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00266 [I 2022-10-27 20:00:25] ...trap/engines/engine.py.230: loss: 0.12734 [I 2022-10-27 20:00:25] ...optimizers/trijoint.py.151: total_norm: 19.610538 [I 2022-10-27 20:00:30] ...trap/engines/engine.py.224: train: epoch 85 | batch 1460/2304 [I 2022-10-27 20:00:30] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:46 [I 2022-10-27 20:00:30] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00248 [I 2022-10-27 20:00:30] ...trap/engines/engine.py.230: loss: 0.09705 [I 2022-10-27 20:00:30] ...optimizers/trijoint.py.151: total_norm: 23.900490 [I 2022-10-27 20:00:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 1470/2304 [I 2022-10-27 20:00:36] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:41 [I 2022-10-27 20:00:36] ...trap/engines/engine.py.229: process: 0.54781 | load: 0.00246 [I 2022-10-27 20:00:36] ...trap/engines/engine.py.230: loss: 0.11067 [I 2022-10-27 20:00:36] ...optimizers/trijoint.py.151: total_norm: 8.481398 [I 2022-10-27 20:00:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 1480/2304 [I 2022-10-27 20:00:42] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:37 [I 2022-10-27 20:00:42] ...trap/engines/engine.py.229: process: 0.57025 | load: 0.00242 [I 2022-10-27 20:00:42] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-27 20:00:42] ...optimizers/trijoint.py.151: total_norm: 9.450138 [I 2022-10-27 20:00:47] ...trap/engines/engine.py.224: train: epoch 85 | batch 1490/2304 [I 2022-10-27 20:00:47] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:31 [I 2022-10-27 20:00:47] ...trap/engines/engine.py.229: process: 0.55159 | load: 0.00243 [I 2022-10-27 20:00:47] ...trap/engines/engine.py.230: loss: 0.09447 [I 2022-10-27 20:00:47] ...optimizers/trijoint.py.151: total_norm: 11.811764 [I 2022-10-27 20:00:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 1500/2304 [I 2022-10-27 20:00:53] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:26 [I 2022-10-27 20:00:53] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00245 [I 2022-10-27 20:00:53] ...trap/engines/engine.py.230: loss: 0.11569 [I 2022-10-27 20:00:53] ...optimizers/trijoint.py.151: total_norm: 11.733806 [I 2022-10-27 20:00:58] ...trap/engines/engine.py.224: train: epoch 85 | batch 1510/2304 [I 2022-10-27 20:00:58] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:19 [I 2022-10-27 20:00:58] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00249 [I 2022-10-27 20:00:58] ...trap/engines/engine.py.230: loss: 0.14551 [I 2022-10-27 20:00:58] ...optimizers/trijoint.py.151: total_norm: 16.728188 [I 2022-10-27 20:01:04] ...trap/engines/engine.py.224: train: epoch 85 | batch 1520/2304 [I 2022-10-27 20:01:04] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:07:12 [I 2022-10-27 20:01:04] ...trap/engines/engine.py.229: process: 0.54836 | load: 0.00242 [I 2022-10-27 20:01:04] ...trap/engines/engine.py.230: loss: 0.11831 [I 2022-10-27 20:01:04] ...optimizers/trijoint.py.151: total_norm: 14.207319 [I 2022-10-27 20:01:09] ...trap/engines/engine.py.224: train: epoch 85 | batch 1530/2304 [I 2022-10-27 20:01:09] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:07:09 [I 2022-10-27 20:01:09] ...trap/engines/engine.py.229: process: 0.54438 | load: 0.00241 [I 2022-10-27 20:01:09] ...trap/engines/engine.py.230: loss: 0.17304 [I 2022-10-27 20:01:09] ...optimizers/trijoint.py.151: total_norm: 20.091736 [I 2022-10-27 20:01:15] ...trap/engines/engine.py.224: train: epoch 85 | batch 1540/2304 [I 2022-10-27 20:01:15] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:07:08 [I 2022-10-27 20:01:15] ...trap/engines/engine.py.229: process: 0.56946 | load: 0.00242 [I 2022-10-27 20:01:15] ...trap/engines/engine.py.230: loss: 0.10849 [I 2022-10-27 20:01:15] ...optimizers/trijoint.py.151: total_norm: 13.320641 [I 2022-10-27 20:01:21] ...trap/engines/engine.py.224: train: epoch 85 | batch 1550/2304 [I 2022-10-27 20:01:21] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:07:02 [I 2022-10-27 20:01:21] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00248 [I 2022-10-27 20:01:21] ...trap/engines/engine.py.230: loss: 0.10096 [I 2022-10-27 20:01:21] ...optimizers/trijoint.py.151: total_norm: 18.026741 [I 2022-10-27 20:01:26] ...trap/engines/engine.py.224: train: epoch 85 | batch 1560/2304 [I 2022-10-27 20:01:26] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:07:01 [I 2022-10-27 20:01:26] ...trap/engines/engine.py.229: process: 0.56818 | load: 0.00241 [I 2022-10-27 20:01:26] ...trap/engines/engine.py.230: loss: 0.11534 [I 2022-10-27 20:01:26] ...optimizers/trijoint.py.151: total_norm: 17.752365 [I 2022-10-27 20:01:32] ...trap/engines/engine.py.224: train: epoch 85 | batch 1570/2304 [I 2022-10-27 20:01:32] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:51 [I 2022-10-27 20:01:32] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00244 [I 2022-10-27 20:01:32] ...trap/engines/engine.py.230: loss: 0.17154 [I 2022-10-27 20:01:32] ...optimizers/trijoint.py.151: total_norm: 21.441589 [I 2022-10-27 20:01:37] ...trap/engines/engine.py.224: train: epoch 85 | batch 1580/2304 [I 2022-10-27 20:01:37] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:43 [I 2022-10-27 20:01:37] ...trap/engines/engine.py.229: process: 0.55893 | load: 0.00244 [I 2022-10-27 20:01:37] ...trap/engines/engine.py.230: loss: 0.13762 [I 2022-10-27 20:01:37] ...optimizers/trijoint.py.151: total_norm: 15.308037 [I 2022-10-27 20:01:43] ...trap/engines/engine.py.224: train: epoch 85 | batch 1590/2304 [I 2022-10-27 20:01:43] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:37 [I 2022-10-27 20:01:43] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00245 [I 2022-10-27 20:01:43] ...trap/engines/engine.py.230: loss: 0.12764 [I 2022-10-27 20:01:43] ...optimizers/trijoint.py.151: total_norm: 20.677069 [I 2022-10-27 20:01:48] ...trap/engines/engine.py.224: train: epoch 85 | batch 1600/2304 [I 2022-10-27 20:01:48] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:29 [I 2022-10-27 20:01:48] ...trap/engines/engine.py.229: process: 0.55169 | load: 0.00245 [I 2022-10-27 20:01:48] ...trap/engines/engine.py.230: loss: 0.18778 [I 2022-10-27 20:01:48] ...optimizers/trijoint.py.151: total_norm: 15.390985 [I 2022-10-27 20:01:54] ...trap/engines/engine.py.224: train: epoch 85 | batch 1610/2304 [I 2022-10-27 20:01:54] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:26 [I 2022-10-27 20:01:54] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00243 [I 2022-10-27 20:01:54] ...trap/engines/engine.py.230: loss: 0.14631 [I 2022-10-27 20:01:54] ...optimizers/trijoint.py.151: total_norm: 63.405685 [I 2022-10-27 20:02:00] ...trap/engines/engine.py.224: train: epoch 85 | batch 1620/2304 [I 2022-10-27 20:02:00] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:19 [I 2022-10-27 20:02:00] ...trap/engines/engine.py.229: process: 0.54870 | load: 0.00243 [I 2022-10-27 20:02:00] ...trap/engines/engine.py.230: loss: 0.09856 [I 2022-10-27 20:02:00] ...optimizers/trijoint.py.151: total_norm: 11.576104 [I 2022-10-27 20:02:05] ...trap/engines/engine.py.224: train: epoch 85 | batch 1630/2304 [I 2022-10-27 20:02:05] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:12 [I 2022-10-27 20:02:05] ...trap/engines/engine.py.229: process: 0.55102 | load: 0.00244 [I 2022-10-27 20:02:05] ...trap/engines/engine.py.230: loss: 0.10846 [I 2022-10-27 20:02:05] ...optimizers/trijoint.py.151: total_norm: 9.802176 [I 2022-10-27 20:02:11] ...trap/engines/engine.py.224: train: epoch 85 | batch 1640/2304 [I 2022-10-27 20:02:11] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:06:09 [I 2022-10-27 20:02:11] ...trap/engines/engine.py.229: process: 0.55117 | load: 0.00240 [I 2022-10-27 20:02:11] ...trap/engines/engine.py.230: loss: 0.11433 [I 2022-10-27 20:02:11] ...optimizers/trijoint.py.151: total_norm: 7.486860 [I 2022-10-27 20:02:16] ...trap/engines/engine.py.224: train: epoch 85 | batch 1650/2304 [I 2022-10-27 20:02:16] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:06:02 [I 2022-10-27 20:02:16] ...trap/engines/engine.py.229: process: 0.55343 | load: 0.00241 [I 2022-10-27 20:02:16] ...trap/engines/engine.py.230: loss: 0.10772 [I 2022-10-27 20:02:16] ...optimizers/trijoint.py.151: total_norm: 73.972031 [I 2022-10-27 20:02:22] ...trap/engines/engine.py.224: train: epoch 85 | batch 1660/2304 [I 2022-10-27 20:02:22] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:55 [I 2022-10-27 20:02:22] ...trap/engines/engine.py.229: process: 0.54032 | load: 0.00245 [I 2022-10-27 20:02:22] ...trap/engines/engine.py.230: loss: 0.13508 [I 2022-10-27 20:02:22] ...optimizers/trijoint.py.151: total_norm: 18.669184 [I 2022-10-27 20:02:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 1670/2304 [I 2022-10-27 20:02:27] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:51 [I 2022-10-27 20:02:27] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00243 [I 2022-10-27 20:02:27] ...trap/engines/engine.py.230: loss: 0.09876 [I 2022-10-27 20:02:27] ...optimizers/trijoint.py.151: total_norm: 7.833185 [I 2022-10-27 20:02:33] ...trap/engines/engine.py.224: train: epoch 85 | batch 1680/2304 [I 2022-10-27 20:02:33] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:44 [I 2022-10-27 20:02:33] ...trap/engines/engine.py.229: process: 0.54854 | load: 0.00241 [I 2022-10-27 20:02:33] ...trap/engines/engine.py.230: loss: 0.18128 [I 2022-10-27 20:02:33] ...optimizers/trijoint.py.151: total_norm: 27.927320 [I 2022-10-27 20:02:38] ...trap/engines/engine.py.224: train: epoch 85 | batch 1690/2304 [I 2022-10-27 20:02:38] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:39 [I 2022-10-27 20:02:38] ...trap/engines/engine.py.229: process: 0.54446 | load: 0.00301 [I 2022-10-27 20:02:38] ...trap/engines/engine.py.230: loss: 0.11781 [I 2022-10-27 20:02:38] ...optimizers/trijoint.py.151: total_norm: 7.178408 [I 2022-10-27 20:02:44] ...trap/engines/engine.py.224: train: epoch 85 | batch 1700/2304 [I 2022-10-27 20:02:44] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:33 [I 2022-10-27 20:02:44] ...trap/engines/engine.py.229: process: 0.55181 | load: 0.00269 [I 2022-10-27 20:02:44] ...trap/engines/engine.py.230: loss: 0.18206 [I 2022-10-27 20:02:44] ...optimizers/trijoint.py.151: total_norm: 27.352781 [I 2022-10-27 20:02:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 1710/2304 [I 2022-10-27 20:02:50] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:29 [I 2022-10-27 20:02:50] ...trap/engines/engine.py.229: process: 0.54451 | load: 0.00259 [I 2022-10-27 20:02:50] ...trap/engines/engine.py.230: loss: 0.14792 [I 2022-10-27 20:02:50] ...optimizers/trijoint.py.151: total_norm: 17.887840 [I 2022-10-27 20:02:55] ...trap/engines/engine.py.224: train: epoch 85 | batch 1720/2304 [I 2022-10-27 20:02:55] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:21 [I 2022-10-27 20:02:55] ...trap/engines/engine.py.229: process: 0.53911 | load: 0.00250 [I 2022-10-27 20:02:55] ...trap/engines/engine.py.230: loss: 0.11146 [I 2022-10-27 20:02:55] ...optimizers/trijoint.py.151: total_norm: 13.730042 [I 2022-10-27 20:03:01] ...trap/engines/engine.py.224: train: epoch 85 | batch 1730/2304 [I 2022-10-27 20:03:01] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:16 [I 2022-10-27 20:03:01] ...trap/engines/engine.py.229: process: 0.54116 | load: 0.00268 [I 2022-10-27 20:03:01] ...trap/engines/engine.py.230: loss: 0.11449 [I 2022-10-27 20:03:01] ...optimizers/trijoint.py.151: total_norm: 13.277788 [I 2022-10-27 20:03:06] ...trap/engines/engine.py.224: train: epoch 85 | batch 1740/2304 [I 2022-10-27 20:03:06] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:05:14 [I 2022-10-27 20:03:06] ...trap/engines/engine.py.229: process: 0.56807 | load: 0.00249 [I 2022-10-27 20:03:06] ...trap/engines/engine.py.230: loss: 0.10164 [I 2022-10-27 20:03:06] ...optimizers/trijoint.py.151: total_norm: 10.403135 [I 2022-10-27 20:03:12] ...trap/engines/engine.py.224: train: epoch 85 | batch 1750/2304 [I 2022-10-27 20:03:12] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:05:07 [I 2022-10-27 20:03:12] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00243 [I 2022-10-27 20:03:12] ...trap/engines/engine.py.230: loss: 0.18389 [I 2022-10-27 20:03:12] ...optimizers/trijoint.py.151: total_norm: 13.365408 [I 2022-10-27 20:03:17] ...trap/engines/engine.py.224: train: epoch 85 | batch 1760/2304 [I 2022-10-27 20:03:17] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:05:02 [I 2022-10-27 20:03:17] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00249 [I 2022-10-27 20:03:17] ...trap/engines/engine.py.230: loss: 0.14404 [I 2022-10-27 20:03:17] ...optimizers/trijoint.py.151: total_norm: 16.189402 [I 2022-10-27 20:03:23] ...trap/engines/engine.py.224: train: epoch 85 | batch 1770/2304 [I 2022-10-27 20:03:23] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:57 [I 2022-10-27 20:03:23] ...trap/engines/engine.py.229: process: 0.55687 | load: 0.00242 [I 2022-10-27 20:03:23] ...trap/engines/engine.py.230: loss: 0.21048 [I 2022-10-27 20:03:23] ...optimizers/trijoint.py.151: total_norm: 17.607752 [I 2022-10-27 20:03:29] ...trap/engines/engine.py.224: train: epoch 85 | batch 1780/2304 [I 2022-10-27 20:03:29] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:51 [I 2022-10-27 20:03:29] ...trap/engines/engine.py.229: process: 0.56165 | load: 0.00261 [I 2022-10-27 20:03:29] ...trap/engines/engine.py.230: loss: 0.08139 [I 2022-10-27 20:03:29] ...optimizers/trijoint.py.151: total_norm: 11.821524 [I 2022-10-27 20:03:34] ...trap/engines/engine.py.224: train: epoch 85 | batch 1790/2304 [I 2022-10-27 20:03:34] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:44 [I 2022-10-27 20:03:34] ...trap/engines/engine.py.229: process: 0.54086 | load: 0.00254 [I 2022-10-27 20:03:34] ...trap/engines/engine.py.230: loss: 0.11270 [I 2022-10-27 20:03:34] ...optimizers/trijoint.py.151: total_norm: 7.267278 [I 2022-10-27 20:03:40] ...trap/engines/engine.py.224: train: epoch 85 | batch 1800/2304 [I 2022-10-27 20:03:40] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:38 [I 2022-10-27 20:03:40] ...trap/engines/engine.py.229: process: 0.54215 | load: 0.00251 [I 2022-10-27 20:03:40] ...trap/engines/engine.py.230: loss: 0.16064 [I 2022-10-27 20:03:40] ...optimizers/trijoint.py.151: total_norm: 18.365683 [I 2022-10-27 20:03:45] ...trap/engines/engine.py.224: train: epoch 85 | batch 1810/2304 [I 2022-10-27 20:03:45] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:33 [I 2022-10-27 20:03:45] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00248 [I 2022-10-27 20:03:45] ...trap/engines/engine.py.230: loss: 0.10359 [I 2022-10-27 20:03:45] ...optimizers/trijoint.py.151: total_norm: 8.790455 [I 2022-10-27 20:03:51] ...trap/engines/engine.py.224: train: epoch 85 | batch 1820/2304 [I 2022-10-27 20:03:51] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:25 [I 2022-10-27 20:03:51] ...trap/engines/engine.py.229: process: 0.54402 | load: 0.00252 [I 2022-10-27 20:03:51] ...trap/engines/engine.py.230: loss: 0.15884 [I 2022-10-27 20:03:51] ...optimizers/trijoint.py.151: total_norm: 11.324519 [I 2022-10-27 20:03:56] ...trap/engines/engine.py.224: train: epoch 85 | batch 1830/2304 [I 2022-10-27 20:03:56] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:22 [I 2022-10-27 20:03:56] ...trap/engines/engine.py.229: process: 0.54646 | load: 0.00251 [I 2022-10-27 20:03:56] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-27 20:03:56] ...optimizers/trijoint.py.151: total_norm: 22.217854 [I 2022-10-27 20:04:02] ...trap/engines/engine.py.224: train: epoch 85 | batch 1840/2304 [I 2022-10-27 20:04:02] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:17 [I 2022-10-27 20:04:02] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00251 [I 2022-10-27 20:04:02] ...trap/engines/engine.py.230: loss: 0.11304 [I 2022-10-27 20:04:02] ...optimizers/trijoint.py.151: total_norm: 14.784907 [I 2022-10-27 20:04:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 1850/2304 [I 2022-10-27 20:04:07] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:12 [I 2022-10-27 20:04:07] ...trap/engines/engine.py.229: process: 0.55724 | load: 0.00245 [I 2022-10-27 20:04:07] ...trap/engines/engine.py.230: loss: 0.14210 [I 2022-10-27 20:04:07] ...optimizers/trijoint.py.151: total_norm: 15.267505 [I 2022-10-27 20:04:13] ...trap/engines/engine.py.224: train: epoch 85 | batch 1860/2304 [I 2022-10-27 20:04:13] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:04:07 [I 2022-10-27 20:04:13] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00247 [I 2022-10-27 20:04:13] ...trap/engines/engine.py.230: loss: 0.10546 [I 2022-10-27 20:04:13] ...optimizers/trijoint.py.151: total_norm: 21.865055 [I 2022-10-27 20:04:19] ...trap/engines/engine.py.224: train: epoch 85 | batch 1870/2304 [I 2022-10-27 20:04:19] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:04:02 [I 2022-10-27 20:04:19] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00248 [I 2022-10-27 20:04:19] ...trap/engines/engine.py.230: loss: 0.13069 [I 2022-10-27 20:04:19] ...optimizers/trijoint.py.151: total_norm: 25.683849 [I 2022-10-27 20:04:24] ...trap/engines/engine.py.224: train: epoch 85 | batch 1880/2304 [I 2022-10-27 20:04:24] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:55 [I 2022-10-27 20:04:24] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00258 [I 2022-10-27 20:04:24] ...trap/engines/engine.py.230: loss: 0.13842 [I 2022-10-27 20:04:24] ...optimizers/trijoint.py.151: total_norm: 12.857001 [I 2022-10-27 20:04:30] ...trap/engines/engine.py.224: train: epoch 85 | batch 1890/2304 [I 2022-10-27 20:04:30] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:48 [I 2022-10-27 20:04:30] ...trap/engines/engine.py.229: process: 0.54915 | load: 0.00248 [I 2022-10-27 20:04:30] ...trap/engines/engine.py.230: loss: 0.11169 [I 2022-10-27 20:04:30] ...optimizers/trijoint.py.151: total_norm: 13.632572 [I 2022-10-27 20:04:35] ...trap/engines/engine.py.224: train: epoch 85 | batch 1900/2304 [I 2022-10-27 20:04:35] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:44 [I 2022-10-27 20:04:35] ...trap/engines/engine.py.229: process: 0.56141 | load: 0.00251 [I 2022-10-27 20:04:35] ...trap/engines/engine.py.230: loss: 0.08352 [I 2022-10-27 20:04:35] ...optimizers/trijoint.py.151: total_norm: 14.407873 [I 2022-10-27 20:04:41] ...trap/engines/engine.py.224: train: epoch 85 | batch 1910/2304 [I 2022-10-27 20:04:41] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:37 [I 2022-10-27 20:04:41] ...trap/engines/engine.py.229: process: 0.55001 | load: 0.00248 [I 2022-10-27 20:04:41] ...trap/engines/engine.py.230: loss: 0.17018 [I 2022-10-27 20:04:41] ...optimizers/trijoint.py.151: total_norm: 17.802868 [I 2022-10-27 20:04:46] ...trap/engines/engine.py.224: train: epoch 85 | batch 1920/2304 [I 2022-10-27 20:04:46] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:33 [I 2022-10-27 20:04:46] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00243 [I 2022-10-27 20:04:46] ...trap/engines/engine.py.230: loss: 0.19615 [I 2022-10-27 20:04:46] ...optimizers/trijoint.py.151: total_norm: 24.886087 [I 2022-10-27 20:04:52] ...trap/engines/engine.py.224: train: epoch 85 | batch 1930/2304 [I 2022-10-27 20:04:52] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:27 [I 2022-10-27 20:04:52] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00244 [I 2022-10-27 20:04:52] ...trap/engines/engine.py.230: loss: 0.14212 [I 2022-10-27 20:04:52] ...optimizers/trijoint.py.151: total_norm: 14.857890 [I 2022-10-27 20:04:58] ...trap/engines/engine.py.224: train: epoch 85 | batch 1940/2304 [I 2022-10-27 20:04:58] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:21 [I 2022-10-27 20:04:58] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00247 [I 2022-10-27 20:04:58] ...trap/engines/engine.py.230: loss: 0.16073 [I 2022-10-27 20:04:58] ...optimizers/trijoint.py.151: total_norm: 16.257656 [I 2022-10-27 20:05:03] ...trap/engines/engine.py.224: train: epoch 85 | batch 1950/2304 [I 2022-10-27 20:05:03] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:14 [I 2022-10-27 20:05:03] ...trap/engines/engine.py.229: process: 0.55211 | load: 0.00264 [I 2022-10-27 20:05:03] ...trap/engines/engine.py.230: loss: 0.13998 [I 2022-10-27 20:05:03] ...optimizers/trijoint.py.151: total_norm: 27.544876 [I 2022-10-27 20:05:09] ...trap/engines/engine.py.224: train: epoch 85 | batch 1960/2304 [I 2022-10-27 20:05:09] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:03:11 [I 2022-10-27 20:05:09] ...trap/engines/engine.py.229: process: 0.55605 | load: 0.00253 [I 2022-10-27 20:05:09] ...trap/engines/engine.py.230: loss: 0.10884 [I 2022-10-27 20:05:09] ...optimizers/trijoint.py.151: total_norm: 13.466826 [I 2022-10-27 20:05:14] ...trap/engines/engine.py.224: train: epoch 85 | batch 1970/2304 [I 2022-10-27 20:05:14] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:03:05 [I 2022-10-27 20:05:14] ...trap/engines/engine.py.229: process: 0.56872 | load: 0.00254 [I 2022-10-27 20:05:14] ...trap/engines/engine.py.230: loss: 0.13856 [I 2022-10-27 20:05:14] ...optimizers/trijoint.py.151: total_norm: 20.785103 [I 2022-10-27 20:05:20] ...trap/engines/engine.py.224: train: epoch 85 | batch 1980/2304 [I 2022-10-27 20:05:20] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:02:59 [I 2022-10-27 20:05:20] ...trap/engines/engine.py.229: process: 0.54861 | load: 0.00250 [I 2022-10-27 20:05:20] ...trap/engines/engine.py.230: loss: 0.17782 [I 2022-10-27 20:05:20] ...optimizers/trijoint.py.151: total_norm: 12.142954 [I 2022-10-27 20:05:25] ...trap/engines/engine.py.224: train: epoch 85 | batch 1990/2304 [I 2022-10-27 20:05:25] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:55 [I 2022-10-27 20:05:25] ...trap/engines/engine.py.229: process: 0.55684 | load: 0.00248 [I 2022-10-27 20:05:25] ...trap/engines/engine.py.230: loss: 0.14051 [I 2022-10-27 20:05:25] ...optimizers/trijoint.py.151: total_norm: 18.627262 [I 2022-10-27 20:05:31] ...trap/engines/engine.py.224: train: epoch 85 | batch 2000/2304 [I 2022-10-27 20:05:31] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:47 [I 2022-10-27 20:05:31] ...trap/engines/engine.py.229: process: 0.54813 | load: 0.00247 [I 2022-10-27 20:05:31] ...trap/engines/engine.py.230: loss: 0.11453 [I 2022-10-27 20:05:31] ...optimizers/trijoint.py.151: total_norm: 14.884668 [I 2022-10-27 20:05:36] ...trap/engines/engine.py.224: train: epoch 85 | batch 2010/2304 [I 2022-10-27 20:05:36] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:42 [I 2022-10-27 20:05:36] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00250 [I 2022-10-27 20:05:36] ...trap/engines/engine.py.230: loss: 0.15963 [I 2022-10-27 20:05:36] ...optimizers/trijoint.py.151: total_norm: 20.689188 [I 2022-10-27 20:05:42] ...trap/engines/engine.py.224: train: epoch 85 | batch 2020/2304 [I 2022-10-27 20:05:42] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:37 [I 2022-10-27 20:05:42] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00243 [I 2022-10-27 20:05:42] ...trap/engines/engine.py.230: loss: 0.11479 [I 2022-10-27 20:05:42] ...optimizers/trijoint.py.151: total_norm: 11.995507 [I 2022-10-27 20:05:48] ...trap/engines/engine.py.224: train: epoch 85 | batch 2030/2304 [I 2022-10-27 20:05:48] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:33 [I 2022-10-27 20:05:48] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00250 [I 2022-10-27 20:05:48] ...trap/engines/engine.py.230: loss: 0.11374 [I 2022-10-27 20:05:48] ...optimizers/trijoint.py.151: total_norm: 27.853874 [I 2022-10-27 20:05:53] ...trap/engines/engine.py.224: train: epoch 85 | batch 2040/2304 [I 2022-10-27 20:05:53] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:26 [I 2022-10-27 20:05:53] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00246 [I 2022-10-27 20:05:53] ...trap/engines/engine.py.230: loss: 0.18442 [I 2022-10-27 20:05:53] ...optimizers/trijoint.py.151: total_norm: 16.629564 [I 2022-10-27 20:05:59] ...trap/engines/engine.py.224: train: epoch 85 | batch 2050/2304 [I 2022-10-27 20:05:59] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:20 [I 2022-10-27 20:05:59] ...trap/engines/engine.py.229: process: 0.55120 | load: 0.00255 [I 2022-10-27 20:05:59] ...trap/engines/engine.py.230: loss: 0.13458 [I 2022-10-27 20:05:59] ...optimizers/trijoint.py.151: total_norm: 16.320013 [I 2022-10-27 20:06:04] ...trap/engines/engine.py.224: train: epoch 85 | batch 2060/2304 [I 2022-10-27 20:06:04] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:02:16 [I 2022-10-27 20:06:04] ...trap/engines/engine.py.229: process: 0.56339 | load: 0.00246 [I 2022-10-27 20:06:04] ...trap/engines/engine.py.230: loss: 0.15125 [I 2022-10-27 20:06:04] ...optimizers/trijoint.py.151: total_norm: 14.826715 [I 2022-10-27 20:06:10] ...trap/engines/engine.py.224: train: epoch 85 | batch 2070/2304 [I 2022-10-27 20:06:10] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:09 [I 2022-10-27 20:06:10] ...trap/engines/engine.py.229: process: 0.55419 | load: 0.00241 [I 2022-10-27 20:06:10] ...trap/engines/engine.py.230: loss: 0.10022 [I 2022-10-27 20:06:10] ...optimizers/trijoint.py.151: total_norm: 14.110640 [I 2022-10-27 20:06:15] ...trap/engines/engine.py.224: train: epoch 85 | batch 2080/2304 [I 2022-10-27 20:06:15] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:02:03 [I 2022-10-27 20:06:15] ...trap/engines/engine.py.229: process: 0.53982 | load: 0.00242 [I 2022-10-27 20:06:15] ...trap/engines/engine.py.230: loss: 0.12242 [I 2022-10-27 20:06:15] ...optimizers/trijoint.py.151: total_norm: 14.113790 [I 2022-10-27 20:06:21] ...trap/engines/engine.py.224: train: epoch 85 | batch 2090/2304 [I 2022-10-27 20:06:21] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:59 [I 2022-10-27 20:06:21] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00249 [I 2022-10-27 20:06:21] ...trap/engines/engine.py.230: loss: 0.13911 [I 2022-10-27 20:06:21] ...optimizers/trijoint.py.151: total_norm: 10.401849 [I 2022-10-27 20:06:27] ...trap/engines/engine.py.224: train: epoch 85 | batch 2100/2304 [I 2022-10-27 20:06:27] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:54 [I 2022-10-27 20:06:27] ...trap/engines/engine.py.229: process: 0.57842 | load: 0.00245 [I 2022-10-27 20:06:27] ...trap/engines/engine.py.230: loss: 0.11143 [I 2022-10-27 20:06:27] ...optimizers/trijoint.py.151: total_norm: 8.800857 [I 2022-10-27 20:06:32] ...trap/engines/engine.py.224: train: epoch 85 | batch 2110/2304 [I 2022-10-27 20:06:32] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:48 [I 2022-10-27 20:06:32] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00245 [I 2022-10-27 20:06:32] ...trap/engines/engine.py.230: loss: 0.12807 [I 2022-10-27 20:06:32] ...optimizers/trijoint.py.151: total_norm: 11.135818 [I 2022-10-27 20:06:38] ...trap/engines/engine.py.224: train: epoch 85 | batch 2120/2304 [I 2022-10-27 20:06:38] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:42 [I 2022-10-27 20:06:38] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00247 [I 2022-10-27 20:06:38] ...trap/engines/engine.py.230: loss: 0.09303 [I 2022-10-27 20:06:38] ...optimizers/trijoint.py.151: total_norm: 12.344364 [I 2022-10-27 20:06:43] ...trap/engines/engine.py.224: train: epoch 85 | batch 2130/2304 [I 2022-10-27 20:06:43] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:37 [I 2022-10-27 20:06:43] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00245 [I 2022-10-27 20:06:43] ...trap/engines/engine.py.230: loss: 0.12014 [I 2022-10-27 20:06:43] ...optimizers/trijoint.py.151: total_norm: 9.571813 [I 2022-10-27 20:06:49] ...trap/engines/engine.py.224: train: epoch 85 | batch 2140/2304 [I 2022-10-27 20:06:49] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:30 [I 2022-10-27 20:06:49] ...trap/engines/engine.py.229: process: 0.54599 | load: 0.00246 [I 2022-10-27 20:06:49] ...trap/engines/engine.py.230: loss: 0.13557 [I 2022-10-27 20:06:49] ...optimizers/trijoint.py.151: total_norm: 45.614204 [I 2022-10-27 20:06:55] ...trap/engines/engine.py.224: train: epoch 85 | batch 2150/2304 [I 2022-10-27 20:06:55] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:25 [I 2022-10-27 20:06:55] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00242 [I 2022-10-27 20:06:55] ...trap/engines/engine.py.230: loss: 0.11197 [I 2022-10-27 20:06:55] ...optimizers/trijoint.py.151: total_norm: 10.329817 [I 2022-10-27 20:07:00] ...trap/engines/engine.py.224: train: epoch 85 | batch 2160/2304 [I 2022-10-27 20:07:00] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:19 [I 2022-10-27 20:07:00] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00241 [I 2022-10-27 20:07:00] ...trap/engines/engine.py.230: loss: 0.09659 [I 2022-10-27 20:07:00] ...optimizers/trijoint.py.151: total_norm: 16.577370 [I 2022-10-27 20:07:06] ...trap/engines/engine.py.224: train: epoch 85 | batch 2170/2304 [I 2022-10-27 20:07:06] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:14 [I 2022-10-27 20:07:06] ...trap/engines/engine.py.229: process: 0.54919 | load: 0.00245 [I 2022-10-27 20:07:06] ...trap/engines/engine.py.230: loss: 0.24245 [I 2022-10-27 20:07:06] ...optimizers/trijoint.py.151: total_norm: 14.487580 [I 2022-10-27 20:07:11] ...trap/engines/engine.py.224: train: epoch 85 | batch 2180/2304 [I 2022-10-27 20:07:11] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:01:08 [I 2022-10-27 20:07:11] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00248 [I 2022-10-27 20:07:11] ...trap/engines/engine.py.230: loss: 0.15504 [I 2022-10-27 20:07:11] ...optimizers/trijoint.py.151: total_norm: 15.091269 [I 2022-10-27 20:07:17] ...trap/engines/engine.py.224: train: epoch 85 | batch 2190/2304 [I 2022-10-27 20:07:17] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:05 [I 2022-10-27 20:07:17] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00245 [I 2022-10-27 20:07:17] ...trap/engines/engine.py.230: loss: 0.10644 [I 2022-10-27 20:07:17] ...optimizers/trijoint.py.151: total_norm: 24.902344 [I 2022-10-27 20:07:23] ...trap/engines/engine.py.224: train: epoch 85 | batch 2200/2304 [I 2022-10-27 20:07:23] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:58 [I 2022-10-27 20:07:23] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00243 [I 2022-10-27 20:07:23] ...trap/engines/engine.py.230: loss: 0.09882 [I 2022-10-27 20:07:23] ...optimizers/trijoint.py.151: total_norm: 12.954432 [I 2022-10-27 20:07:28] ...trap/engines/engine.py.224: train: epoch 85 | batch 2210/2304 [I 2022-10-27 20:07:28] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:52 [I 2022-10-27 20:07:28] ...trap/engines/engine.py.229: process: 0.56361 | load: 0.00241 [I 2022-10-27 20:07:28] ...trap/engines/engine.py.230: loss: 0.15023 [I 2022-10-27 20:07:28] ...optimizers/trijoint.py.151: total_norm: 20.861864 [I 2022-10-27 20:07:34] ...trap/engines/engine.py.224: train: epoch 85 | batch 2220/2304 [I 2022-10-27 20:07:34] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:46 [I 2022-10-27 20:07:34] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00247 [I 2022-10-27 20:07:34] ...trap/engines/engine.py.230: loss: 0.11438 [I 2022-10-27 20:07:34] ...optimizers/trijoint.py.151: total_norm: 12.411146 [I 2022-10-27 20:07:39] ...trap/engines/engine.py.224: train: epoch 85 | batch 2230/2304 [I 2022-10-27 20:07:39] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:41 [I 2022-10-27 20:07:39] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00240 [I 2022-10-27 20:07:39] ...trap/engines/engine.py.230: loss: 0.12928 [I 2022-10-27 20:07:39] ...optimizers/trijoint.py.151: total_norm: 25.736439 [I 2022-10-27 20:07:45] ...trap/engines/engine.py.224: train: epoch 85 | batch 2240/2304 [I 2022-10-27 20:07:45] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:35 [I 2022-10-27 20:07:45] ...trap/engines/engine.py.229: process: 0.54551 | load: 0.00248 [I 2022-10-27 20:07:45] ...trap/engines/engine.py.230: loss: 0.16860 [I 2022-10-27 20:07:45] ...optimizers/trijoint.py.151: total_norm: 12.991899 [I 2022-10-27 20:07:50] ...trap/engines/engine.py.224: train: epoch 85 | batch 2250/2304 [I 2022-10-27 20:07:50] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:29 [I 2022-10-27 20:07:50] ...trap/engines/engine.py.229: process: 0.55499 | load: 0.00244 [I 2022-10-27 20:07:50] ...trap/engines/engine.py.230: loss: 0.12603 [I 2022-10-27 20:07:50] ...optimizers/trijoint.py.151: total_norm: 12.778590 [I 2022-10-27 20:07:56] ...trap/engines/engine.py.224: train: epoch 85 | batch 2260/2304 [I 2022-10-27 20:07:56] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:24 [I 2022-10-27 20:07:56] ...trap/engines/engine.py.229: process: 0.55174 | load: 0.00243 [I 2022-10-27 20:07:56] ...trap/engines/engine.py.230: loss: 0.11427 [I 2022-10-27 20:07:56] ...optimizers/trijoint.py.151: total_norm: 14.232550 [I 2022-10-27 20:08:02] ...trap/engines/engine.py.224: train: epoch 85 | batch 2270/2304 [I 2022-10-27 20:08:02] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:18 [I 2022-10-27 20:08:02] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00242 [I 2022-10-27 20:08:02] ...trap/engines/engine.py.230: loss: 0.09985 [I 2022-10-27 20:08:02] ...optimizers/trijoint.py.151: total_norm: 18.686132 [I 2022-10-27 20:08:07] ...trap/engines/engine.py.224: train: epoch 85 | batch 2280/2304 [I 2022-10-27 20:08:07] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:13 [I 2022-10-27 20:08:07] ...trap/engines/engine.py.229: process: 0.54937 | load: 0.00239 [I 2022-10-27 20:08:07] ...trap/engines/engine.py.230: loss: 0.19745 [I 2022-10-27 20:08:07] ...optimizers/trijoint.py.151: total_norm: 27.949066 [I 2022-10-27 20:08:13] ...trap/engines/engine.py.224: train: epoch 85 | batch 2290/2304 [I 2022-10-27 20:08:13] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:07 [I 2022-10-27 20:08:13] ...trap/engines/engine.py.229: process: 0.55561 | load: 0.00245 [I 2022-10-27 20:08:13] ...trap/engines/engine.py.230: loss: 0.16225 [I 2022-10-27 20:08:13] ...optimizers/trijoint.py.151: total_norm: 19.554594 [I 2022-10-27 20:08:18] ...trap/engines/engine.py.224: train: epoch 85 | batch 2300/2304 [I 2022-10-27 20:08:18] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:02 [I 2022-10-27 20:08:18] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00015 [I 2022-10-27 20:08:18] ...trap/engines/engine.py.230: loss: 0.13579 [I 2022-10-27 20:08:18] ...optimizers/trijoint.py.151: total_norm: 20.863735 [I 2022-10-27 20:08:20] ...trap/engines/engine.py.224: train: epoch 85 | batch 2304/2304 [I 2022-10-27 20:08:20] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:00 [I 2022-10-27 20:08:20] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00016 [I 2022-10-27 20:08:20] ...trap/engines/engine.py.230: loss: 0.12902 [I 2022-10-27 20:08:20] ...optimizers/trijoint.py.151: total_norm: 9.683906 [S 2022-10-27 20:08:22] ...trap/engines/engine.py.129: train_epoch.epoch: 85 [S 2022-10-27 20:08:22] ...trap/engines/engine.py.129: train_epoch.loss: 0.13348929565601908 [I 2022-10-27 20:08:22] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 20:08:23] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 85 [I 2022-10-27 20:08:23] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 20:08:25] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 20:08:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 20:08:25] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 20:08:28] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 20:08:28] ...trap/engines/engine.py.321: eval: epoch 85 | batch 0/510 [I 2022-10-27 20:08:28] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:43:36 [I 2022-10-27 20:08:28] ...trap/engines/engine.py.326: process: 5.12993 | load: 3.23454 [I 2022-10-27 20:08:31] ...trap/engines/engine.py.321: eval: epoch 85 | batch 10/510 [I 2022-10-27 20:08:31] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:02 [I 2022-10-27 20:08:31] ...trap/engines/engine.py.326: process: 0.32323 | load: 0.00019 [I 2022-10-27 20:08:35] ...trap/engines/engine.py.321: eval: epoch 85 | batch 20/510 [I 2022-10-27 20:08:35] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:09 [I 2022-10-27 20:08:35] ...trap/engines/engine.py.326: process: 0.33221 | load: 0.00018 [I 2022-10-27 20:08:38] ...trap/engines/engine.py.321: eval: epoch 85 | batch 30/510 [I 2022-10-27 20:08:38] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:42 [I 2022-10-27 20:08:38] ...trap/engines/engine.py.326: process: 0.33609 | load: 0.00024 [I 2022-10-27 20:08:41] ...trap/engines/engine.py.321: eval: epoch 85 | batch 40/510 [I 2022-10-27 20:08:41] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:36 [I 2022-10-27 20:08:41] ...trap/engines/engine.py.326: process: 0.34303 | load: 0.00026 [I 2022-10-27 20:08:45] ...trap/engines/engine.py.321: eval: epoch 85 | batch 50/510 [I 2022-10-27 20:08:45] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:33 [I 2022-10-27 20:08:45] ...trap/engines/engine.py.326: process: 0.34151 | load: 0.00023 [I 2022-10-27 20:08:48] ...trap/engines/engine.py.321: eval: epoch 85 | batch 60/510 [I 2022-10-27 20:08:48] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:38 [I 2022-10-27 20:08:48] ...trap/engines/engine.py.326: process: 0.41621 | load: 0.00024 [I 2022-10-27 20:08:51] ...trap/engines/engine.py.321: eval: epoch 85 | batch 70/510 [I 2022-10-27 20:08:51] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:27 [I 2022-10-27 20:08:51] ...trap/engines/engine.py.326: process: 0.34309 | load: 0.00017 [I 2022-10-27 20:08:55] ...trap/engines/engine.py.321: eval: epoch 85 | batch 80/510 [I 2022-10-27 20:08:55] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:23 [I 2022-10-27 20:08:55] ...trap/engines/engine.py.326: process: 0.32823 | load: 0.00018 [I 2022-10-27 20:08:58] ...trap/engines/engine.py.321: eval: epoch 85 | batch 90/510 [I 2022-10-27 20:08:58] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:21 [I 2022-10-27 20:08:58] ...trap/engines/engine.py.326: process: 0.35049 | load: 0.00018 [I 2022-10-27 20:09:01] ...trap/engines/engine.py.321: eval: epoch 85 | batch 100/510 [I 2022-10-27 20:09:01] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-27 20:09:01] ...trap/engines/engine.py.326: process: 0.28500 | load: 0.00023 [I 2022-10-27 20:09:04] ...trap/engines/engine.py.321: eval: epoch 85 | batch 110/510 [I 2022-10-27 20:09:04] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 20:09:04] ...trap/engines/engine.py.326: process: 0.29554 | load: 0.00015 [I 2022-10-27 20:09:07] ...trap/engines/engine.py.321: eval: epoch 85 | batch 120/510 [I 2022-10-27 20:09:07] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 20:09:07] ...trap/engines/engine.py.326: process: 0.28805 | load: 0.00015 [I 2022-10-27 20:09:10] ...trap/engines/engine.py.321: eval: epoch 85 | batch 130/510 [I 2022-10-27 20:09:10] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:51 [I 2022-10-27 20:09:10] ...trap/engines/engine.py.326: process: 0.30151 | load: 0.00018 [I 2022-10-27 20:09:13] ...trap/engines/engine.py.321: eval: epoch 85 | batch 140/510 [I 2022-10-27 20:09:13] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-27 20:09:13] ...trap/engines/engine.py.326: process: 0.28427 | load: 0.00017 [I 2022-10-27 20:09:16] ...trap/engines/engine.py.321: eval: epoch 85 | batch 150/510 [I 2022-10-27 20:09:16] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:45 [I 2022-10-27 20:09:16] ...trap/engines/engine.py.326: process: 0.29134 | load: 0.00017 [I 2022-10-27 20:09:19] ...trap/engines/engine.py.321: eval: epoch 85 | batch 160/510 [I 2022-10-27 20:09:19] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:42 [I 2022-10-27 20:09:19] ...trap/engines/engine.py.326: process: 0.30184 | load: 0.00018 [I 2022-10-27 20:09:22] ...trap/engines/engine.py.321: eval: epoch 85 | batch 170/510 [I 2022-10-27 20:09:22] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:39 [I 2022-10-27 20:09:22] ...trap/engines/engine.py.326: process: 0.28403 | load: 0.00017 [I 2022-10-27 20:09:25] ...trap/engines/engine.py.321: eval: epoch 85 | batch 180/510 [I 2022-10-27 20:09:25] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:36 [I 2022-10-27 20:09:25] ...trap/engines/engine.py.326: process: 0.29125 | load: 0.00018 [I 2022-10-27 20:09:28] ...trap/engines/engine.py.321: eval: epoch 85 | batch 190/510 [I 2022-10-27 20:09:28] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:33 [I 2022-10-27 20:09:28] ...trap/engines/engine.py.326: process: 0.29009 | load: 0.00022 [I 2022-10-27 20:09:31] ...trap/engines/engine.py.321: eval: epoch 85 | batch 200/510 [I 2022-10-27 20:09:31] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:30 [I 2022-10-27 20:09:31] ...trap/engines/engine.py.326: process: 0.28721 | load: 0.00015 [I 2022-10-27 20:09:33] ...trap/engines/engine.py.321: eval: epoch 85 | batch 210/510 [I 2022-10-27 20:09:33] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 20:09:33] ...trap/engines/engine.py.326: process: 0.28155 | load: 0.00018 [I 2022-10-27 20:09:36] ...trap/engines/engine.py.321: eval: epoch 85 | batch 220/510 [I 2022-10-27 20:09:36] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 20:09:36] ...trap/engines/engine.py.326: process: 0.28282 | load: 0.00016 [I 2022-10-27 20:09:39] ...trap/engines/engine.py.321: eval: epoch 85 | batch 230/510 [I 2022-10-27 20:09:39] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 20:09:39] ...trap/engines/engine.py.326: process: 0.28553 | load: 0.00017 [I 2022-10-27 20:09:42] ...trap/engines/engine.py.321: eval: epoch 85 | batch 240/510 [I 2022-10-27 20:09:42] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-27 20:09:42] ...trap/engines/engine.py.326: process: 0.29029 | load: 0.00017 [I 2022-10-27 20:09:45] ...trap/engines/engine.py.321: eval: epoch 85 | batch 250/510 [I 2022-10-27 20:09:45] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-27 20:09:45] ...trap/engines/engine.py.326: process: 0.28498 | load: 0.00018 [I 2022-10-27 20:09:48] ...trap/engines/engine.py.321: eval: epoch 85 | batch 260/510 [I 2022-10-27 20:09:48] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 20:09:48] ...trap/engines/engine.py.326: process: 0.29261 | load: 0.00017 [I 2022-10-27 20:09:51] ...trap/engines/engine.py.321: eval: epoch 85 | batch 270/510 [I 2022-10-27 20:09:51] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:10 [I 2022-10-27 20:09:51] ...trap/engines/engine.py.326: process: 0.28884 | load: 0.00018 [I 2022-10-27 20:09:54] ...trap/engines/engine.py.321: eval: epoch 85 | batch 280/510 [I 2022-10-27 20:09:54] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 20:09:54] ...trap/engines/engine.py.326: process: 0.28515 | load: 0.00018 [I 2022-10-27 20:09:57] ...trap/engines/engine.py.321: eval: epoch 85 | batch 290/510 [I 2022-10-27 20:09:57] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 20:09:57] ...trap/engines/engine.py.326: process: 0.28582 | load: 0.00017 [I 2022-10-27 20:10:00] ...trap/engines/engine.py.321: eval: epoch 85 | batch 300/510 [I 2022-10-27 20:10:00] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 20:10:00] ...trap/engines/engine.py.326: process: 0.29589 | load: 0.00015 [I 2022-10-27 20:10:02] ...trap/engines/engine.py.321: eval: epoch 85 | batch 310/510 [I 2022-10-27 20:10:02] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:58 [I 2022-10-27 20:10:02] ...trap/engines/engine.py.326: process: 0.29490 | load: 0.00017 [I 2022-10-27 20:10:05] ...trap/engines/engine.py.321: eval: epoch 85 | batch 320/510 [I 2022-10-27 20:10:05] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 20:10:05] ...trap/engines/engine.py.326: process: 0.28518 | load: 0.00017 [I 2022-10-27 20:10:08] ...trap/engines/engine.py.321: eval: epoch 85 | batch 330/510 [I 2022-10-27 20:10:08] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-27 20:10:08] ...trap/engines/engine.py.326: process: 0.28979 | load: 0.00017 [I 2022-10-27 20:10:11] ...trap/engines/engine.py.321: eval: epoch 85 | batch 340/510 [I 2022-10-27 20:10:11] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-27 20:10:11] ...trap/engines/engine.py.326: process: 0.29014 | load: 0.00018 [I 2022-10-27 20:10:14] ...trap/engines/engine.py.321: eval: epoch 85 | batch 350/510 [I 2022-10-27 20:10:14] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 20:10:14] ...trap/engines/engine.py.326: process: 0.29051 | load: 0.00020 [I 2022-10-27 20:10:17] ...trap/engines/engine.py.321: eval: epoch 85 | batch 360/510 [I 2022-10-27 20:10:17] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 20:10:17] ...trap/engines/engine.py.326: process: 0.28818 | load: 0.00021 [I 2022-10-27 20:10:20] ...trap/engines/engine.py.321: eval: epoch 85 | batch 370/510 [I 2022-10-27 20:10:20] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 20:10:20] ...trap/engines/engine.py.326: process: 0.29663 | load: 0.00016 [I 2022-10-27 20:10:23] ...trap/engines/engine.py.321: eval: epoch 85 | batch 380/510 [I 2022-10-27 20:10:23] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 20:10:23] ...trap/engines/engine.py.326: process: 0.29497 | load: 0.00015 [I 2022-10-27 20:10:26] ...trap/engines/engine.py.321: eval: epoch 85 | batch 390/510 [I 2022-10-27 20:10:26] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 20:10:26] ...trap/engines/engine.py.326: process: 0.29286 | load: 0.00015 [I 2022-10-27 20:10:28] ...trap/engines/engine.py.321: eval: epoch 85 | batch 400/510 [I 2022-10-27 20:10:28] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 20:10:28] ...trap/engines/engine.py.326: process: 0.28876 | load: 0.00017 [I 2022-10-27 20:10:31] ...trap/engines/engine.py.321: eval: epoch 85 | batch 410/510 [I 2022-10-27 20:10:31] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 20:10:31] ...trap/engines/engine.py.326: process: 0.28820 | load: 0.00017 [I 2022-10-27 20:10:34] ...trap/engines/engine.py.321: eval: epoch 85 | batch 420/510 [I 2022-10-27 20:10:34] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-27 20:10:34] ...trap/engines/engine.py.326: process: 0.29032 | load: 0.00017 [I 2022-10-27 20:10:37] ...trap/engines/engine.py.321: eval: epoch 85 | batch 430/510 [I 2022-10-27 20:10:37] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 20:10:37] ...trap/engines/engine.py.326: process: 0.29396 | load: 0.00017 [I 2022-10-27 20:10:40] ...trap/engines/engine.py.321: eval: epoch 85 | batch 440/510 [I 2022-10-27 20:10:40] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 20:10:40] ...trap/engines/engine.py.326: process: 0.28988 | load: 0.00017 [I 2022-10-27 20:10:43] ...trap/engines/engine.py.321: eval: epoch 85 | batch 450/510 [I 2022-10-27 20:10:43] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 20:10:43] ...trap/engines/engine.py.326: process: 0.28856 | load: 0.00019 [I 2022-10-27 20:10:46] ...trap/engines/engine.py.321: eval: epoch 85 | batch 460/510 [I 2022-10-27 20:10:46] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 20:10:46] ...trap/engines/engine.py.326: process: 0.28936 | load: 0.00020 [I 2022-10-27 20:10:49] ...trap/engines/engine.py.321: eval: epoch 85 | batch 470/510 [I 2022-10-27 20:10:49] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 20:10:49] ...trap/engines/engine.py.326: process: 0.29338 | load: 0.00020 [I 2022-10-27 20:10:52] ...trap/engines/engine.py.321: eval: epoch 85 | batch 480/510 [I 2022-10-27 20:10:52] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 20:10:52] ...trap/engines/engine.py.326: process: 0.28159 | load: 0.00018 [I 2022-10-27 20:10:54] ...trap/engines/engine.py.321: eval: epoch 85 | batch 490/510 [I 2022-10-27 20:10:54] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 20:10:54] ...trap/engines/engine.py.326: process: 0.28731 | load: 0.00015 [I 2022-10-27 20:10:57] ...trap/engines/engine.py.321: eval: epoch 85 | batch 500/510 [I 2022-10-27 20:10:57] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 20:10:57] ...trap/engines/engine.py.326: process: 0.29342 | load: 0.00018 [I 2022-10-27 20:11:00] ...trap/engines/engine.py.321: eval: epoch 85 | batch 510/510 [I 2022-10-27 20:11:00] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 20:11:00] ...trap/engines/engine.py.326: process: 0.28310 | load: 0.00012 [S 2022-10-27 20:11:01] ...trap/engines/engine.py.132: eval_epoch.epoch: 85 [I 2022-10-27 20:11:01] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.732900 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906500 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934000 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.717500 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904100 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934500 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010492 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007580 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007057 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009489 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.003506 [S 2022-10-27 20:11:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005500 [I 2022-10-27 20:11:07] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 20:11:07] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 20:11:07] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 20:11:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 20:11:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 20:11:11] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 20:11:12] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 20:11:18] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 20:11:18] ...trap/engines/engine.py.163: Training model on trainset for epoch 86 [I 2022-10-27 20:11:18] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 20:11:21] ...trap/engines/engine.py.224: train: epoch 86 | batch 0/2304 [I 2022-10-27 20:11:21] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:50 [I 2022-10-27 20:11:21] ...trap/engines/engine.py.229: process: 2.78215 | load: 2.14480 [I 2022-10-27 20:11:21] ...trap/engines/engine.py.230: loss: 0.10635 [I 2022-10-27 20:11:21] ...optimizers/trijoint.py.151: total_norm: 11.437962 [I 2022-10-27 20:11:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 10/2304 [I 2022-10-27 20:11:27] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:21 [I 2022-10-27 20:11:27] ...trap/engines/engine.py.229: process: 0.54361 | load: 0.00343 [I 2022-10-27 20:11:27] ...trap/engines/engine.py.230: loss: 0.11677 [I 2022-10-27 20:11:27] ...optimizers/trijoint.py.151: total_norm: 11.519066 [I 2022-10-27 20:11:32] ...trap/engines/engine.py.224: train: epoch 86 | batch 20/2304 [I 2022-10-27 20:11:32] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:18 [I 2022-10-27 20:11:32] ...trap/engines/engine.py.229: process: 0.56316 | load: 0.00385 [I 2022-10-27 20:11:32] ...trap/engines/engine.py.230: loss: 0.12895 [I 2022-10-27 20:11:32] ...optimizers/trijoint.py.151: total_norm: 23.119844 [I 2022-10-27 20:11:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 30/2304 [I 2022-10-27 20:11:38] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:19 [I 2022-10-27 20:11:38] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00333 [I 2022-10-27 20:11:38] ...trap/engines/engine.py.230: loss: 0.14880 [I 2022-10-27 20:11:38] ...optimizers/trijoint.py.151: total_norm: 17.341724 [I 2022-10-27 20:11:43] ...trap/engines/engine.py.224: train: epoch 86 | batch 40/2304 [I 2022-10-27 20:11:43] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:56 [I 2022-10-27 20:11:43] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00311 [I 2022-10-27 20:11:43] ...trap/engines/engine.py.230: loss: 0.15563 [I 2022-10-27 20:11:43] ...optimizers/trijoint.py.151: total_norm: 12.283220 [I 2022-10-27 20:11:49] ...trap/engines/engine.py.224: train: epoch 86 | batch 50/2304 [I 2022-10-27 20:11:49] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:03 [I 2022-10-27 20:11:49] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00279 [I 2022-10-27 20:11:49] ...trap/engines/engine.py.230: loss: 0.15128 [I 2022-10-27 20:11:49] ...optimizers/trijoint.py.151: total_norm: 18.053699 [I 2022-10-27 20:11:55] ...trap/engines/engine.py.224: train: epoch 86 | batch 60/2304 [I 2022-10-27 20:11:55] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:55 [I 2022-10-27 20:11:55] ...trap/engines/engine.py.229: process: 0.56579 | load: 0.00291 [I 2022-10-27 20:11:55] ...trap/engines/engine.py.230: loss: 0.16055 [I 2022-10-27 20:11:55] ...optimizers/trijoint.py.151: total_norm: 15.005155 [I 2022-10-27 20:12:00] ...trap/engines/engine.py.224: train: epoch 86 | batch 70/2304 [I 2022-10-27 20:12:00] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:58 [I 2022-10-27 20:12:00] ...trap/engines/engine.py.229: process: 0.55834 | load: 0.00257 [I 2022-10-27 20:12:00] ...trap/engines/engine.py.230: loss: 0.15357 [I 2022-10-27 20:12:00] ...optimizers/trijoint.py.151: total_norm: 19.083012 [I 2022-10-27 20:12:06] ...trap/engines/engine.py.224: train: epoch 86 | batch 80/2304 [I 2022-10-27 20:12:06] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:50 [I 2022-10-27 20:12:06] ...trap/engines/engine.py.229: process: 0.56814 | load: 0.00255 [I 2022-10-27 20:12:06] ...trap/engines/engine.py.230: loss: 0.12312 [I 2022-10-27 20:12:06] ...optimizers/trijoint.py.151: total_norm: 15.496298 [I 2022-10-27 20:12:11] ...trap/engines/engine.py.224: train: epoch 86 | batch 90/2304 [I 2022-10-27 20:12:11] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:40 [I 2022-10-27 20:12:11] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00262 [I 2022-10-27 20:12:11] ...trap/engines/engine.py.230: loss: 0.14193 [I 2022-10-27 20:12:11] ...optimizers/trijoint.py.151: total_norm: 20.834936 [I 2022-10-27 20:12:17] ...trap/engines/engine.py.224: train: epoch 86 | batch 100/2304 [I 2022-10-27 20:12:17] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:33 [I 2022-10-27 20:12:17] ...trap/engines/engine.py.229: process: 0.56037 | load: 0.00266 [I 2022-10-27 20:12:17] ...trap/engines/engine.py.230: loss: 0.11292 [I 2022-10-27 20:12:17] ...optimizers/trijoint.py.151: total_norm: 8.794839 [I 2022-10-27 20:12:23] ...trap/engines/engine.py.224: train: epoch 86 | batch 110/2304 [I 2022-10-27 20:12:23] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:22 [I 2022-10-27 20:12:23] ...trap/engines/engine.py.229: process: 0.55768 | load: 0.00273 [I 2022-10-27 20:12:23] ...trap/engines/engine.py.230: loss: 0.12492 [I 2022-10-27 20:12:23] ...optimizers/trijoint.py.151: total_norm: 13.697968 [I 2022-10-27 20:12:28] ...trap/engines/engine.py.224: train: epoch 86 | batch 120/2304 [I 2022-10-27 20:12:28] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:17 [I 2022-10-27 20:12:28] ...trap/engines/engine.py.229: process: 0.54532 | load: 0.00262 [I 2022-10-27 20:12:28] ...trap/engines/engine.py.230: loss: 0.14852 [I 2022-10-27 20:12:28] ...optimizers/trijoint.py.151: total_norm: 15.217567 [I 2022-10-27 20:12:34] ...trap/engines/engine.py.224: train: epoch 86 | batch 130/2304 [I 2022-10-27 20:12:34] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:10 [I 2022-10-27 20:12:34] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00242 [I 2022-10-27 20:12:34] ...trap/engines/engine.py.230: loss: 0.20706 [I 2022-10-27 20:12:34] ...optimizers/trijoint.py.151: total_norm: 12.154975 [I 2022-10-27 20:12:39] ...trap/engines/engine.py.224: train: epoch 86 | batch 140/2304 [I 2022-10-27 20:12:39] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:09 [I 2022-10-27 20:12:39] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00249 [I 2022-10-27 20:12:39] ...trap/engines/engine.py.230: loss: 0.21414 [I 2022-10-27 20:12:39] ...optimizers/trijoint.py.151: total_norm: 16.711731 [I 2022-10-27 20:12:45] ...trap/engines/engine.py.224: train: epoch 86 | batch 150/2304 [I 2022-10-27 20:12:45] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:58 [I 2022-10-27 20:12:45] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00255 [I 2022-10-27 20:12:45] ...trap/engines/engine.py.230: loss: 0.11705 [I 2022-10-27 20:12:45] ...optimizers/trijoint.py.151: total_norm: 9.938231 [I 2022-10-27 20:12:51] ...trap/engines/engine.py.224: train: epoch 86 | batch 160/2304 [I 2022-10-27 20:12:51] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:00 [I 2022-10-27 20:12:51] ...trap/engines/engine.py.229: process: 0.55277 | load: 0.00261 [I 2022-10-27 20:12:51] ...trap/engines/engine.py.230: loss: 0.12231 [I 2022-10-27 20:12:51] ...optimizers/trijoint.py.151: total_norm: 13.234364 [I 2022-10-27 20:12:56] ...trap/engines/engine.py.224: train: epoch 86 | batch 170/2304 [I 2022-10-27 20:12:56] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:44 [I 2022-10-27 20:12:56] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00296 [I 2022-10-27 20:12:56] ...trap/engines/engine.py.230: loss: 0.18760 [I 2022-10-27 20:12:56] ...optimizers/trijoint.py.151: total_norm: 13.103988 [I 2022-10-27 20:13:02] ...trap/engines/engine.py.224: train: epoch 86 | batch 180/2304 [I 2022-10-27 20:13:02] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:47 [I 2022-10-27 20:13:02] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00275 [I 2022-10-27 20:13:02] ...trap/engines/engine.py.230: loss: 0.14158 [I 2022-10-27 20:13:02] ...optimizers/trijoint.py.151: total_norm: 16.622383 [I 2022-10-27 20:13:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 190/2304 [I 2022-10-27 20:13:07] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:46 [I 2022-10-27 20:13:07] ...trap/engines/engine.py.229: process: 0.56889 | load: 0.00274 [I 2022-10-27 20:13:07] ...trap/engines/engine.py.230: loss: 0.09714 [I 2022-10-27 20:13:07] ...optimizers/trijoint.py.151: total_norm: 9.401209 [I 2022-10-27 20:13:13] ...trap/engines/engine.py.224: train: epoch 86 | batch 200/2304 [I 2022-10-27 20:13:13] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:25 [I 2022-10-27 20:13:13] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00276 [I 2022-10-27 20:13:13] ...trap/engines/engine.py.230: loss: 0.17874 [I 2022-10-27 20:13:13] ...optimizers/trijoint.py.151: total_norm: 16.770964 [I 2022-10-27 20:13:18] ...trap/engines/engine.py.224: train: epoch 86 | batch 210/2304 [I 2022-10-27 20:13:18] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:30 [I 2022-10-27 20:13:18] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00267 [I 2022-10-27 20:13:18] ...trap/engines/engine.py.230: loss: 0.12556 [I 2022-10-27 20:13:18] ...optimizers/trijoint.py.151: total_norm: 8.957894 [I 2022-10-27 20:13:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 220/2304 [I 2022-10-27 20:13:24] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:23 [I 2022-10-27 20:13:24] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00262 [I 2022-10-27 20:13:24] ...trap/engines/engine.py.230: loss: 0.16989 [I 2022-10-27 20:13:24] ...optimizers/trijoint.py.151: total_norm: 35.751373 [I 2022-10-27 20:13:30] ...trap/engines/engine.py.224: train: epoch 86 | batch 230/2304 [I 2022-10-27 20:13:30] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:20 [I 2022-10-27 20:13:30] ...trap/engines/engine.py.229: process: 0.56215 | load: 0.00253 [I 2022-10-27 20:13:30] ...trap/engines/engine.py.230: loss: 0.13661 [I 2022-10-27 20:13:30] ...optimizers/trijoint.py.151: total_norm: 13.427159 [I 2022-10-27 20:13:35] ...trap/engines/engine.py.224: train: epoch 86 | batch 240/2304 [I 2022-10-27 20:13:35] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:16 [I 2022-10-27 20:13:35] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00264 [I 2022-10-27 20:13:35] ...trap/engines/engine.py.230: loss: 0.13417 [I 2022-10-27 20:13:35] ...optimizers/trijoint.py.151: total_norm: 10.824993 [I 2022-10-27 20:13:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 250/2304 [I 2022-10-27 20:13:41] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:58 [I 2022-10-27 20:13:41] ...trap/engines/engine.py.229: process: 0.55216 | load: 0.00269 [I 2022-10-27 20:13:41] ...trap/engines/engine.py.230: loss: 0.14454 [I 2022-10-27 20:13:41] ...optimizers/trijoint.py.151: total_norm: 33.446228 [I 2022-10-27 20:13:46] ...trap/engines/engine.py.224: train: epoch 86 | batch 260/2304 [I 2022-10-27 20:13:46] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:08 [I 2022-10-27 20:13:46] ...trap/engines/engine.py.229: process: 0.56168 | load: 0.00247 [I 2022-10-27 20:13:46] ...trap/engines/engine.py.230: loss: 0.11293 [I 2022-10-27 20:13:46] ...optimizers/trijoint.py.151: total_norm: 19.754738 [I 2022-10-27 20:13:52] ...trap/engines/engine.py.224: train: epoch 86 | batch 270/2304 [I 2022-10-27 20:13:52] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:59 [I 2022-10-27 20:13:52] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00249 [I 2022-10-27 20:13:52] ...trap/engines/engine.py.230: loss: 0.11292 [I 2022-10-27 20:13:52] ...optimizers/trijoint.py.151: total_norm: 18.949455 [I 2022-10-27 20:13:58] ...trap/engines/engine.py.224: train: epoch 86 | batch 280/2304 [I 2022-10-27 20:13:58] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:45 [I 2022-10-27 20:13:58] ...trap/engines/engine.py.229: process: 0.56625 | load: 0.00251 [I 2022-10-27 20:13:58] ...trap/engines/engine.py.230: loss: 0.09857 [I 2022-10-27 20:13:58] ...optimizers/trijoint.py.151: total_norm: 12.925157 [I 2022-10-27 20:14:03] ...trap/engines/engine.py.224: train: epoch 86 | batch 290/2304 [I 2022-10-27 20:14:03] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:48 [I 2022-10-27 20:14:03] ...trap/engines/engine.py.229: process: 0.57064 | load: 0.00256 [I 2022-10-27 20:14:03] ...trap/engines/engine.py.230: loss: 0.18593 [I 2022-10-27 20:14:03] ...optimizers/trijoint.py.151: total_norm: 24.422047 [I 2022-10-27 20:14:09] ...trap/engines/engine.py.224: train: epoch 86 | batch 300/2304 [I 2022-10-27 20:14:09] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:37 [I 2022-10-27 20:14:09] ...trap/engines/engine.py.229: process: 0.56463 | load: 0.00258 [I 2022-10-27 20:14:09] ...trap/engines/engine.py.230: loss: 0.10543 [I 2022-10-27 20:14:09] ...optimizers/trijoint.py.151: total_norm: 12.127702 [I 2022-10-27 20:14:14] ...trap/engines/engine.py.224: train: epoch 86 | batch 310/2304 [I 2022-10-27 20:14:14] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:47 [I 2022-10-27 20:14:14] ...trap/engines/engine.py.229: process: 0.56952 | load: 0.00248 [I 2022-10-27 20:14:14] ...trap/engines/engine.py.230: loss: 0.09359 [I 2022-10-27 20:14:14] ...optimizers/trijoint.py.151: total_norm: 15.222510 [I 2022-10-27 20:14:20] ...trap/engines/engine.py.224: train: epoch 86 | batch 320/2304 [I 2022-10-27 20:14:20] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:33 [I 2022-10-27 20:14:20] ...trap/engines/engine.py.229: process: 0.56883 | load: 0.00262 [I 2022-10-27 20:14:20] ...trap/engines/engine.py.230: loss: 0.19460 [I 2022-10-27 20:14:20] ...optimizers/trijoint.py.151: total_norm: 16.983856 [I 2022-10-27 20:14:26] ...trap/engines/engine.py.224: train: epoch 86 | batch 330/2304 [I 2022-10-27 20:14:26] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:19 [I 2022-10-27 20:14:26] ...trap/engines/engine.py.229: process: 0.55098 | load: 0.00298 [I 2022-10-27 20:14:26] ...trap/engines/engine.py.230: loss: 0.12803 [I 2022-10-27 20:14:26] ...optimizers/trijoint.py.151: total_norm: 15.102787 [I 2022-10-27 20:14:31] ...trap/engines/engine.py.224: train: epoch 86 | batch 340/2304 [I 2022-10-27 20:14:31] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:21 [I 2022-10-27 20:14:31] ...trap/engines/engine.py.229: process: 0.56245 | load: 0.00323 [I 2022-10-27 20:14:31] ...trap/engines/engine.py.230: loss: 0.09436 [I 2022-10-27 20:14:31] ...optimizers/trijoint.py.151: total_norm: 9.427760 [I 2022-10-27 20:14:37] ...trap/engines/engine.py.224: train: epoch 86 | batch 350/2304 [I 2022-10-27 20:14:37] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:13 [I 2022-10-27 20:14:37] ...trap/engines/engine.py.229: process: 0.54766 | load: 0.00248 [I 2022-10-27 20:14:37] ...trap/engines/engine.py.230: loss: 0.14022 [I 2022-10-27 20:14:37] ...optimizers/trijoint.py.151: total_norm: 28.311636 [I 2022-10-27 20:14:42] ...trap/engines/engine.py.224: train: epoch 86 | batch 360/2304 [I 2022-10-27 20:14:42] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:10 [I 2022-10-27 20:14:42] ...trap/engines/engine.py.229: process: 0.56221 | load: 0.00250 [I 2022-10-27 20:14:42] ...trap/engines/engine.py.230: loss: 0.10862 [I 2022-10-27 20:14:42] ...optimizers/trijoint.py.151: total_norm: 9.202250 [I 2022-10-27 20:14:48] ...trap/engines/engine.py.224: train: epoch 86 | batch 370/2304 [I 2022-10-27 20:14:48] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:56 [I 2022-10-27 20:14:48] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00245 [I 2022-10-27 20:14:48] ...trap/engines/engine.py.230: loss: 0.07158 [I 2022-10-27 20:14:48] ...optimizers/trijoint.py.151: total_norm: 11.935616 [I 2022-10-27 20:14:54] ...trap/engines/engine.py.224: train: epoch 86 | batch 380/2304 [I 2022-10-27 20:14:54] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:18:01 [I 2022-10-27 20:14:54] ...trap/engines/engine.py.229: process: 0.56782 | load: 0.00246 [I 2022-10-27 20:14:54] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-27 20:14:54] ...optimizers/trijoint.py.151: total_norm: 9.436546 [I 2022-10-27 20:14:59] ...trap/engines/engine.py.224: train: epoch 86 | batch 390/2304 [I 2022-10-27 20:14:59] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:54 [I 2022-10-27 20:14:59] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00255 [I 2022-10-27 20:14:59] ...trap/engines/engine.py.230: loss: 0.11373 [I 2022-10-27 20:14:59] ...optimizers/trijoint.py.151: total_norm: 20.246058 [I 2022-10-27 20:15:05] ...trap/engines/engine.py.224: train: epoch 86 | batch 400/2304 [I 2022-10-27 20:15:05] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:37 [I 2022-10-27 20:15:05] ...trap/engines/engine.py.229: process: 0.53803 | load: 0.00252 [I 2022-10-27 20:15:05] ...trap/engines/engine.py.230: loss: 0.15465 [I 2022-10-27 20:15:05] ...optimizers/trijoint.py.151: total_norm: 17.607782 [I 2022-10-27 20:15:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 410/2304 [I 2022-10-27 20:15:10] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:31 [I 2022-10-27 20:15:10] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00244 [I 2022-10-27 20:15:10] ...trap/engines/engine.py.230: loss: 0.07188 [I 2022-10-27 20:15:10] ...optimizers/trijoint.py.151: total_norm: 16.581930 [I 2022-10-27 20:15:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 420/2304 [I 2022-10-27 20:15:16] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:32 [I 2022-10-27 20:15:16] ...trap/engines/engine.py.229: process: 0.56571 | load: 0.00244 [I 2022-10-27 20:15:16] ...trap/engines/engine.py.230: loss: 0.12170 [I 2022-10-27 20:15:16] ...optimizers/trijoint.py.151: total_norm: 16.302214 [I 2022-10-27 20:15:22] ...trap/engines/engine.py.224: train: epoch 86 | batch 430/2304 [I 2022-10-27 20:15:22] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:27 [I 2022-10-27 20:15:22] ...trap/engines/engine.py.229: process: 0.56625 | load: 0.00244 [I 2022-10-27 20:15:22] ...trap/engines/engine.py.230: loss: 0.17296 [I 2022-10-27 20:15:22] ...optimizers/trijoint.py.151: total_norm: 18.030846 [I 2022-10-27 20:15:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 440/2304 [I 2022-10-27 20:15:27] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:25 [I 2022-10-27 20:15:27] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00241 [I 2022-10-27 20:15:27] ...trap/engines/engine.py.230: loss: 0.15172 [I 2022-10-27 20:15:27] ...optimizers/trijoint.py.151: total_norm: 31.685520 [I 2022-10-27 20:15:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 450/2304 [I 2022-10-27 20:15:33] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:19 [I 2022-10-27 20:15:33] ...trap/engines/engine.py.229: process: 0.56453 | load: 0.00248 [I 2022-10-27 20:15:33] ...trap/engines/engine.py.230: loss: 0.11796 [I 2022-10-27 20:15:33] ...optimizers/trijoint.py.151: total_norm: 20.693356 [I 2022-10-27 20:15:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 460/2304 [I 2022-10-27 20:15:38] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:06 [I 2022-10-27 20:15:38] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00242 [I 2022-10-27 20:15:38] ...trap/engines/engine.py.230: loss: 0.12634 [I 2022-10-27 20:15:38] ...optimizers/trijoint.py.151: total_norm: 13.956765 [I 2022-10-27 20:15:44] ...trap/engines/engine.py.224: train: epoch 86 | batch 470/2304 [I 2022-10-27 20:15:44] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:03 [I 2022-10-27 20:15:44] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00254 [I 2022-10-27 20:15:44] ...trap/engines/engine.py.230: loss: 0.12888 [I 2022-10-27 20:15:44] ...optimizers/trijoint.py.151: total_norm: 29.919870 [I 2022-10-27 20:15:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 480/2304 [I 2022-10-27 20:15:50] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:51 [I 2022-10-27 20:15:50] ...trap/engines/engine.py.229: process: 0.55363 | load: 0.00247 [I 2022-10-27 20:15:50] ...trap/engines/engine.py.230: loss: 0.12920 [I 2022-10-27 20:15:50] ...optimizers/trijoint.py.151: total_norm: 23.579054 [I 2022-10-27 20:15:55] ...trap/engines/engine.py.224: train: epoch 86 | batch 490/2304 [I 2022-10-27 20:15:55] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:58 [I 2022-10-27 20:15:55] ...trap/engines/engine.py.229: process: 0.57653 | load: 0.00255 [I 2022-10-27 20:15:55] ...trap/engines/engine.py.230: loss: 0.12428 [I 2022-10-27 20:15:55] ...optimizers/trijoint.py.151: total_norm: 15.220493 [I 2022-10-27 20:16:01] ...trap/engines/engine.py.224: train: epoch 86 | batch 500/2304 [I 2022-10-27 20:16:01] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:41 [I 2022-10-27 20:16:01] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00246 [I 2022-10-27 20:16:01] ...trap/engines/engine.py.230: loss: 0.09890 [I 2022-10-27 20:16:01] ...optimizers/trijoint.py.151: total_norm: 10.003569 [I 2022-10-27 20:16:06] ...trap/engines/engine.py.224: train: epoch 86 | batch 510/2304 [I 2022-10-27 20:16:06] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:35 [I 2022-10-27 20:16:06] ...trap/engines/engine.py.229: process: 0.54760 | load: 0.00244 [I 2022-10-27 20:16:06] ...trap/engines/engine.py.230: loss: 0.19713 [I 2022-10-27 20:16:06] ...optimizers/trijoint.py.151: total_norm: 23.020330 [I 2022-10-27 20:16:12] ...trap/engines/engine.py.224: train: epoch 86 | batch 520/2304 [I 2022-10-27 20:16:12] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:39 [I 2022-10-27 20:16:12] ...trap/engines/engine.py.229: process: 0.55003 | load: 0.00243 [I 2022-10-27 20:16:12] ...trap/engines/engine.py.230: loss: 0.18381 [I 2022-10-27 20:16:12] ...optimizers/trijoint.py.151: total_norm: 11.404467 [I 2022-10-27 20:16:18] ...trap/engines/engine.py.224: train: epoch 86 | batch 530/2304 [I 2022-10-27 20:16:18] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:33 [I 2022-10-27 20:16:18] ...trap/engines/engine.py.229: process: 0.55945 | load: 0.00245 [I 2022-10-27 20:16:18] ...trap/engines/engine.py.230: loss: 0.15199 [I 2022-10-27 20:16:18] ...optimizers/trijoint.py.151: total_norm: 20.405577 [I 2022-10-27 20:16:23] ...trap/engines/engine.py.224: train: epoch 86 | batch 540/2304 [I 2022-10-27 20:16:23] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:39 [I 2022-10-27 20:16:23] ...trap/engines/engine.py.229: process: 0.58500 | load: 0.00267 [I 2022-10-27 20:16:23] ...trap/engines/engine.py.230: loss: 0.16555 [I 2022-10-27 20:16:23] ...optimizers/trijoint.py.151: total_norm: 26.264376 [I 2022-10-27 20:16:29] ...trap/engines/engine.py.224: train: epoch 86 | batch 550/2304 [I 2022-10-27 20:16:29] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:26 [I 2022-10-27 20:16:29] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00244 [I 2022-10-27 20:16:29] ...trap/engines/engine.py.230: loss: 0.18572 [I 2022-10-27 20:16:29] ...optimizers/trijoint.py.151: total_norm: 18.944862 [I 2022-10-27 20:16:35] ...trap/engines/engine.py.224: train: epoch 86 | batch 560/2304 [I 2022-10-27 20:16:35] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:52 [I 2022-10-27 20:16:35] ...trap/engines/engine.py.229: process: 0.56685 | load: 0.00245 [I 2022-10-27 20:16:35] ...trap/engines/engine.py.230: loss: 0.15698 [I 2022-10-27 20:16:35] ...optimizers/trijoint.py.151: total_norm: 24.143490 [I 2022-10-27 20:16:40] ...trap/engines/engine.py.224: train: epoch 86 | batch 570/2304 [I 2022-10-27 20:16:40] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:17 [I 2022-10-27 20:16:40] ...trap/engines/engine.py.229: process: 0.56331 | load: 0.00253 [I 2022-10-27 20:16:40] ...trap/engines/engine.py.230: loss: 0.13698 [I 2022-10-27 20:16:40] ...optimizers/trijoint.py.151: total_norm: 13.848384 [I 2022-10-27 20:16:46] ...trap/engines/engine.py.224: train: epoch 86 | batch 580/2304 [I 2022-10-27 20:16:46] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:53 [I 2022-10-27 20:16:46] ...trap/engines/engine.py.229: process: 0.55369 | load: 0.00260 [I 2022-10-27 20:16:46] ...trap/engines/engine.py.230: loss: 0.13892 [I 2022-10-27 20:16:46] ...optimizers/trijoint.py.151: total_norm: 19.355064 [I 2022-10-27 20:16:51] ...trap/engines/engine.py.224: train: epoch 86 | batch 590/2304 [I 2022-10-27 20:16:51] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:55 [I 2022-10-27 20:16:51] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00267 [I 2022-10-27 20:16:51] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-27 20:16:51] ...optimizers/trijoint.py.151: total_norm: 19.600113 [I 2022-10-27 20:16:57] ...trap/engines/engine.py.224: train: epoch 86 | batch 600/2304 [I 2022-10-27 20:16:57] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:55 [I 2022-10-27 20:16:57] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00255 [I 2022-10-27 20:16:57] ...trap/engines/engine.py.230: loss: 0.11460 [I 2022-10-27 20:16:57] ...optimizers/trijoint.py.151: total_norm: 17.747507 [I 2022-10-27 20:17:03] ...trap/engines/engine.py.224: train: epoch 86 | batch 610/2304 [I 2022-10-27 20:17:03] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:47 [I 2022-10-27 20:17:03] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00266 [I 2022-10-27 20:17:03] ...trap/engines/engine.py.230: loss: 0.15436 [I 2022-10-27 20:17:03] ...optimizers/trijoint.py.151: total_norm: 44.008446 [I 2022-10-27 20:17:08] ...trap/engines/engine.py.224: train: epoch 86 | batch 620/2304 [I 2022-10-27 20:17:08] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:32 [I 2022-10-27 20:17:08] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00248 [I 2022-10-27 20:17:08] ...trap/engines/engine.py.230: loss: 0.09208 [I 2022-10-27 20:17:08] ...optimizers/trijoint.py.151: total_norm: 12.218505 [I 2022-10-27 20:17:14] ...trap/engines/engine.py.224: train: epoch 86 | batch 630/2304 [I 2022-10-27 20:17:14] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:59 [I 2022-10-27 20:17:14] ...trap/engines/engine.py.229: process: 0.57057 | load: 0.00255 [I 2022-10-27 20:17:14] ...trap/engines/engine.py.230: loss: 0.11885 [I 2022-10-27 20:17:14] ...optimizers/trijoint.py.151: total_norm: 11.020920 [I 2022-10-27 20:17:20] ...trap/engines/engine.py.224: train: epoch 86 | batch 640/2304 [I 2022-10-27 20:17:20] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:32 [I 2022-10-27 20:17:20] ...trap/engines/engine.py.229: process: 0.55348 | load: 0.00243 [I 2022-10-27 20:17:20] ...trap/engines/engine.py.230: loss: 0.09924 [I 2022-10-27 20:17:20] ...optimizers/trijoint.py.151: total_norm: 11.950275 [I 2022-10-27 20:17:25] ...trap/engines/engine.py.224: train: epoch 86 | batch 650/2304 [I 2022-10-27 20:17:25] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:25 [I 2022-10-27 20:17:25] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00250 [I 2022-10-27 20:17:25] ...trap/engines/engine.py.230: loss: 0.15872 [I 2022-10-27 20:17:25] ...optimizers/trijoint.py.151: total_norm: 27.161545 [I 2022-10-27 20:17:31] ...trap/engines/engine.py.224: train: epoch 86 | batch 660/2304 [I 2022-10-27 20:17:31] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:14 [I 2022-10-27 20:17:31] ...trap/engines/engine.py.229: process: 0.55453 | load: 0.00249 [I 2022-10-27 20:17:31] ...trap/engines/engine.py.230: loss: 0.11816 [I 2022-10-27 20:17:31] ...optimizers/trijoint.py.151: total_norm: 15.016147 [I 2022-10-27 20:17:36] ...trap/engines/engine.py.224: train: epoch 86 | batch 670/2304 [I 2022-10-27 20:17:36] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:07 [I 2022-10-27 20:17:36] ...trap/engines/engine.py.229: process: 0.54831 | load: 0.00253 [I 2022-10-27 20:17:36] ...trap/engines/engine.py.230: loss: 0.09508 [I 2022-10-27 20:17:36] ...optimizers/trijoint.py.151: total_norm: 11.270379 [I 2022-10-27 20:17:42] ...trap/engines/engine.py.224: train: epoch 86 | batch 680/2304 [I 2022-10-27 20:17:42] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:06 [I 2022-10-27 20:17:42] ...trap/engines/engine.py.229: process: 0.56479 | load: 0.00241 [I 2022-10-27 20:17:42] ...trap/engines/engine.py.230: loss: 0.10171 [I 2022-10-27 20:17:42] ...optimizers/trijoint.py.151: total_norm: 17.367981 [I 2022-10-27 20:17:48] ...trap/engines/engine.py.224: train: epoch 86 | batch 690/2304 [I 2022-10-27 20:17:48] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:07 [I 2022-10-27 20:17:48] ...trap/engines/engine.py.229: process: 0.56692 | load: 0.00249 [I 2022-10-27 20:17:48] ...trap/engines/engine.py.230: loss: 0.17291 [I 2022-10-27 20:17:48] ...optimizers/trijoint.py.151: total_norm: 24.543509 [I 2022-10-27 20:17:53] ...trap/engines/engine.py.224: train: epoch 86 | batch 700/2304 [I 2022-10-27 20:17:53] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:15:04 [I 2022-10-27 20:17:53] ...trap/engines/engine.py.229: process: 0.56375 | load: 0.00245 [I 2022-10-27 20:17:53] ...trap/engines/engine.py.230: loss: 0.20528 [I 2022-10-27 20:17:53] ...optimizers/trijoint.py.151: total_norm: 15.454953 [I 2022-10-27 20:17:59] ...trap/engines/engine.py.224: train: epoch 86 | batch 710/2304 [I 2022-10-27 20:17:59] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:48 [I 2022-10-27 20:17:59] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00246 [I 2022-10-27 20:17:59] ...trap/engines/engine.py.230: loss: 0.12086 [I 2022-10-27 20:17:59] ...optimizers/trijoint.py.151: total_norm: 13.191137 [I 2022-10-27 20:18:05] ...trap/engines/engine.py.224: train: epoch 86 | batch 720/2304 [I 2022-10-27 20:18:05] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:42 [I 2022-10-27 20:18:05] ...trap/engines/engine.py.229: process: 0.55217 | load: 0.00246 [I 2022-10-27 20:18:05] ...trap/engines/engine.py.230: loss: 0.15891 [I 2022-10-27 20:18:05] ...optimizers/trijoint.py.151: total_norm: 16.870247 [I 2022-10-27 20:18:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 730/2304 [I 2022-10-27 20:18:10] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:35 [I 2022-10-27 20:18:10] ...trap/engines/engine.py.229: process: 0.56473 | load: 0.00242 [I 2022-10-27 20:18:10] ...trap/engines/engine.py.230: loss: 0.08531 [I 2022-10-27 20:18:10] ...optimizers/trijoint.py.151: total_norm: 9.658399 [I 2022-10-27 20:18:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 740/2304 [I 2022-10-27 20:18:16] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:33 [I 2022-10-27 20:18:16] ...trap/engines/engine.py.229: process: 0.56153 | load: 0.00244 [I 2022-10-27 20:18:16] ...trap/engines/engine.py.230: loss: 0.08745 [I 2022-10-27 20:18:16] ...optimizers/trijoint.py.151: total_norm: 18.007854 [I 2022-10-27 20:18:21] ...trap/engines/engine.py.224: train: epoch 86 | batch 750/2304 [I 2022-10-27 20:18:21] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:26 [I 2022-10-27 20:18:21] ...trap/engines/engine.py.229: process: 0.55069 | load: 0.00241 [I 2022-10-27 20:18:21] ...trap/engines/engine.py.230: loss: 0.12156 [I 2022-10-27 20:18:21] ...optimizers/trijoint.py.151: total_norm: 16.965450 [I 2022-10-27 20:18:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 760/2304 [I 2022-10-27 20:18:27] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:36 [I 2022-10-27 20:18:27] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00247 [I 2022-10-27 20:18:27] ...trap/engines/engine.py.230: loss: 0.19221 [I 2022-10-27 20:18:27] ...optimizers/trijoint.py.151: total_norm: 17.741636 [I 2022-10-27 20:18:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 770/2304 [I 2022-10-27 20:18:33] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:19 [I 2022-10-27 20:18:33] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00244 [I 2022-10-27 20:18:33] ...trap/engines/engine.py.230: loss: 0.12129 [I 2022-10-27 20:18:33] ...optimizers/trijoint.py.151: total_norm: 15.420630 [I 2022-10-27 20:18:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 780/2304 [I 2022-10-27 20:18:38] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:10 [I 2022-10-27 20:18:38] ...trap/engines/engine.py.229: process: 0.55596 | load: 0.00242 [I 2022-10-27 20:18:38] ...trap/engines/engine.py.230: loss: 0.10445 [I 2022-10-27 20:18:38] ...optimizers/trijoint.py.151: total_norm: 11.022391 [I 2022-10-27 20:18:44] ...trap/engines/engine.py.224: train: epoch 86 | batch 790/2304 [I 2022-10-27 20:18:44] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:05 [I 2022-10-27 20:18:44] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00242 [I 2022-10-27 20:18:44] ...trap/engines/engine.py.230: loss: 0.08615 [I 2022-10-27 20:18:44] ...optimizers/trijoint.py.151: total_norm: 11.571886 [I 2022-10-27 20:18:49] ...trap/engines/engine.py.224: train: epoch 86 | batch 800/2304 [I 2022-10-27 20:18:49] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:13:58 [I 2022-10-27 20:18:49] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00246 [I 2022-10-27 20:18:49] ...trap/engines/engine.py.230: loss: 0.11916 [I 2022-10-27 20:18:49] ...optimizers/trijoint.py.151: total_norm: 26.216358 [I 2022-10-27 20:18:55] ...trap/engines/engine.py.224: train: epoch 86 | batch 810/2304 [I 2022-10-27 20:18:55] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:55 [I 2022-10-27 20:18:55] ...trap/engines/engine.py.229: process: 0.56614 | load: 0.00244 [I 2022-10-27 20:18:55] ...trap/engines/engine.py.230: loss: 0.13215 [I 2022-10-27 20:18:55] ...optimizers/trijoint.py.151: total_norm: 14.606362 [I 2022-10-27 20:19:01] ...trap/engines/engine.py.224: train: epoch 86 | batch 820/2304 [I 2022-10-27 20:19:01] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:52 [I 2022-10-27 20:19:01] ...trap/engines/engine.py.229: process: 0.56696 | load: 0.00251 [I 2022-10-27 20:19:01] ...trap/engines/engine.py.230: loss: 0.13960 [I 2022-10-27 20:19:01] ...optimizers/trijoint.py.151: total_norm: 12.628065 [I 2022-10-27 20:19:06] ...trap/engines/engine.py.224: train: epoch 86 | batch 830/2304 [I 2022-10-27 20:19:06] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:42 [I 2022-10-27 20:19:06] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00253 [I 2022-10-27 20:19:06] ...trap/engines/engine.py.230: loss: 0.12030 [I 2022-10-27 20:19:06] ...optimizers/trijoint.py.151: total_norm: 25.061567 [I 2022-10-27 20:19:12] ...trap/engines/engine.py.224: train: epoch 86 | batch 840/2304 [I 2022-10-27 20:19:12] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:44 [I 2022-10-27 20:19:12] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00251 [I 2022-10-27 20:19:12] ...trap/engines/engine.py.230: loss: 0.12772 [I 2022-10-27 20:19:12] ...optimizers/trijoint.py.151: total_norm: 18.317013 [I 2022-10-27 20:19:17] ...trap/engines/engine.py.224: train: epoch 86 | batch 850/2304 [I 2022-10-27 20:19:17] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:30 [I 2022-10-27 20:19:17] ...trap/engines/engine.py.229: process: 0.56807 | load: 0.00248 [I 2022-10-27 20:19:17] ...trap/engines/engine.py.230: loss: 0.11239 [I 2022-10-27 20:19:17] ...optimizers/trijoint.py.151: total_norm: 15.562408 [I 2022-10-27 20:19:23] ...trap/engines/engine.py.224: train: epoch 86 | batch 860/2304 [I 2022-10-27 20:19:23] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:31 [I 2022-10-27 20:19:23] ...trap/engines/engine.py.229: process: 0.56592 | load: 0.00251 [I 2022-10-27 20:19:23] ...trap/engines/engine.py.230: loss: 0.15618 [I 2022-10-27 20:19:23] ...optimizers/trijoint.py.151: total_norm: 19.134163 [I 2022-10-27 20:19:29] ...trap/engines/engine.py.224: train: epoch 86 | batch 870/2304 [I 2022-10-27 20:19:29] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:22 [I 2022-10-27 20:19:29] ...trap/engines/engine.py.229: process: 0.56532 | load: 0.00246 [I 2022-10-27 20:19:29] ...trap/engines/engine.py.230: loss: 0.17823 [I 2022-10-27 20:19:29] ...optimizers/trijoint.py.151: total_norm: 19.244576 [I 2022-10-27 20:19:34] ...trap/engines/engine.py.224: train: epoch 86 | batch 880/2304 [I 2022-10-27 20:19:34] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:19 [I 2022-10-27 20:19:34] ...trap/engines/engine.py.229: process: 0.55433 | load: 0.00241 [I 2022-10-27 20:19:34] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-27 20:19:34] ...optimizers/trijoint.py.151: total_norm: 13.248667 [I 2022-10-27 20:19:40] ...trap/engines/engine.py.224: train: epoch 86 | batch 890/2304 [I 2022-10-27 20:19:40] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:19 [I 2022-10-27 20:19:40] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00243 [I 2022-10-27 20:19:40] ...trap/engines/engine.py.230: loss: 0.08923 [I 2022-10-27 20:19:40] ...optimizers/trijoint.py.151: total_norm: 9.979628 [I 2022-10-27 20:19:46] ...trap/engines/engine.py.224: train: epoch 86 | batch 900/2304 [I 2022-10-27 20:19:46] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:09 [I 2022-10-27 20:19:46] ...trap/engines/engine.py.229: process: 0.56866 | load: 0.00244 [I 2022-10-27 20:19:46] ...trap/engines/engine.py.230: loss: 0.12032 [I 2022-10-27 20:19:46] ...optimizers/trijoint.py.151: total_norm: 12.575724 [I 2022-10-27 20:19:51] ...trap/engines/engine.py.224: train: epoch 86 | batch 910/2304 [I 2022-10-27 20:19:51] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:12:53 [I 2022-10-27 20:19:51] ...trap/engines/engine.py.229: process: 0.54517 | load: 0.00243 [I 2022-10-27 20:19:51] ...trap/engines/engine.py.230: loss: 0.15599 [I 2022-10-27 20:19:51] ...optimizers/trijoint.py.151: total_norm: 18.581985 [I 2022-10-27 20:19:57] ...trap/engines/engine.py.224: train: epoch 86 | batch 920/2304 [I 2022-10-27 20:19:57] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:54 [I 2022-10-27 20:19:57] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00242 [I 2022-10-27 20:19:57] ...trap/engines/engine.py.230: loss: 0.10299 [I 2022-10-27 20:19:57] ...optimizers/trijoint.py.151: total_norm: 8.734335 [I 2022-10-27 20:20:02] ...trap/engines/engine.py.224: train: epoch 86 | batch 930/2304 [I 2022-10-27 20:20:02] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:12:41 [I 2022-10-27 20:20:02] ...trap/engines/engine.py.229: process: 0.55695 | load: 0.00247 [I 2022-10-27 20:20:02] ...trap/engines/engine.py.230: loss: 0.09620 [I 2022-10-27 20:20:02] ...optimizers/trijoint.py.151: total_norm: 14.888645 [I 2022-10-27 20:20:08] ...trap/engines/engine.py.224: train: epoch 86 | batch 940/2304 [I 2022-10-27 20:20:08] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:42 [I 2022-10-27 20:20:08] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00248 [I 2022-10-27 20:20:08] ...trap/engines/engine.py.230: loss: 0.14896 [I 2022-10-27 20:20:08] ...optimizers/trijoint.py.151: total_norm: 16.671253 [I 2022-10-27 20:20:14] ...trap/engines/engine.py.224: train: epoch 86 | batch 950/2304 [I 2022-10-27 20:20:14] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:58 [I 2022-10-27 20:20:14] ...trap/engines/engine.py.229: process: 0.63856 | load: 0.00250 [I 2022-10-27 20:20:14] ...trap/engines/engine.py.230: loss: 0.14872 [I 2022-10-27 20:20:14] ...optimizers/trijoint.py.151: total_norm: 53.223049 [I 2022-10-27 20:20:19] ...trap/engines/engine.py.224: train: epoch 86 | batch 960/2304 [I 2022-10-27 20:20:19] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:41 [I 2022-10-27 20:20:19] ...trap/engines/engine.py.229: process: 0.56796 | load: 0.00245 [I 2022-10-27 20:20:19] ...trap/engines/engine.py.230: loss: 0.10559 [I 2022-10-27 20:20:19] ...optimizers/trijoint.py.151: total_norm: 22.876522 [I 2022-10-27 20:20:25] ...trap/engines/engine.py.224: train: epoch 86 | batch 970/2304 [I 2022-10-27 20:20:25] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:34 [I 2022-10-27 20:20:25] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00246 [I 2022-10-27 20:20:25] ...trap/engines/engine.py.230: loss: 0.11785 [I 2022-10-27 20:20:25] ...optimizers/trijoint.py.151: total_norm: 15.456925 [I 2022-10-27 20:20:31] ...trap/engines/engine.py.224: train: epoch 86 | batch 980/2304 [I 2022-10-27 20:20:31] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:12:25 [I 2022-10-27 20:20:31] ...trap/engines/engine.py.229: process: 0.56218 | load: 0.00243 [I 2022-10-27 20:20:31] ...trap/engines/engine.py.230: loss: 0.12589 [I 2022-10-27 20:20:31] ...optimizers/trijoint.py.151: total_norm: 30.400822 [I 2022-10-27 20:20:36] ...trap/engines/engine.py.224: train: epoch 86 | batch 990/2304 [I 2022-10-27 20:20:36] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:17 [I 2022-10-27 20:20:36] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00251 [I 2022-10-27 20:20:36] ...trap/engines/engine.py.230: loss: 0.10774 [I 2022-10-27 20:20:36] ...optimizers/trijoint.py.151: total_norm: 10.433684 [I 2022-10-27 20:20:42] ...trap/engines/engine.py.224: train: epoch 86 | batch 1000/2304 [I 2022-10-27 20:20:42] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:12 [I 2022-10-27 20:20:42] ...trap/engines/engine.py.229: process: 0.54589 | load: 0.00247 [I 2022-10-27 20:20:42] ...trap/engines/engine.py.230: loss: 0.14400 [I 2022-10-27 20:20:42] ...optimizers/trijoint.py.151: total_norm: 26.887867 [I 2022-10-27 20:20:47] ...trap/engines/engine.py.224: train: epoch 86 | batch 1010/2304 [I 2022-10-27 20:20:47] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:12:02 [I 2022-10-27 20:20:47] ...trap/engines/engine.py.229: process: 0.55204 | load: 0.00245 [I 2022-10-27 20:20:47] ...trap/engines/engine.py.230: loss: 0.09616 [I 2022-10-27 20:20:47] ...optimizers/trijoint.py.151: total_norm: 7.415101 [I 2022-10-27 20:20:53] ...trap/engines/engine.py.224: train: epoch 86 | batch 1020/2304 [I 2022-10-27 20:20:53] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:11:59 [I 2022-10-27 20:20:53] ...trap/engines/engine.py.229: process: 0.55687 | load: 0.00247 [I 2022-10-27 20:20:53] ...trap/engines/engine.py.230: loss: 0.18513 [I 2022-10-27 20:20:53] ...optimizers/trijoint.py.151: total_norm: 25.276712 [I 2022-10-27 20:20:59] ...trap/engines/engine.py.224: train: epoch 86 | batch 1030/2304 [I 2022-10-27 20:20:59] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:11:53 [I 2022-10-27 20:20:59] ...trap/engines/engine.py.229: process: 0.54961 | load: 0.00243 [I 2022-10-27 20:20:59] ...trap/engines/engine.py.230: loss: 0.10897 [I 2022-10-27 20:20:59] ...optimizers/trijoint.py.151: total_norm: 11.371946 [I 2022-10-27 20:21:04] ...trap/engines/engine.py.224: train: epoch 86 | batch 1040/2304 [I 2022-10-27 20:21:04] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:11:50 [I 2022-10-27 20:21:04] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00241 [I 2022-10-27 20:21:04] ...trap/engines/engine.py.230: loss: 0.22262 [I 2022-10-27 20:21:04] ...optimizers/trijoint.py.151: total_norm: 26.623484 [I 2022-10-27 20:21:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 1050/2304 [I 2022-10-27 20:21:10] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:35 [I 2022-10-27 20:21:10] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00257 [I 2022-10-27 20:21:10] ...trap/engines/engine.py.230: loss: 0.10772 [I 2022-10-27 20:21:10] ...optimizers/trijoint.py.151: total_norm: 26.857750 [I 2022-10-27 20:21:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 1060/2304 [I 2022-10-27 20:21:16] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:32 [I 2022-10-27 20:21:16] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00247 [I 2022-10-27 20:21:16] ...trap/engines/engine.py.230: loss: 0.14311 [I 2022-10-27 20:21:16] ...optimizers/trijoint.py.151: total_norm: 9.895745 [I 2022-10-27 20:21:21] ...trap/engines/engine.py.224: train: epoch 86 | batch 1070/2304 [I 2022-10-27 20:21:21] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:25 [I 2022-10-27 20:21:21] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00251 [I 2022-10-27 20:21:21] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-27 20:21:21] ...optimizers/trijoint.py.151: total_norm: 13.537524 [I 2022-10-27 20:21:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 1080/2304 [I 2022-10-27 20:21:27] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:38 [I 2022-10-27 20:21:27] ...trap/engines/engine.py.229: process: 0.57784 | load: 0.00248 [I 2022-10-27 20:21:27] ...trap/engines/engine.py.230: loss: 0.13926 [I 2022-10-27 20:21:27] ...optimizers/trijoint.py.151: total_norm: 18.964338 [I 2022-10-27 20:21:32] ...trap/engines/engine.py.224: train: epoch 86 | batch 1090/2304 [I 2022-10-27 20:21:32] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:19 [I 2022-10-27 20:21:32] ...trap/engines/engine.py.229: process: 0.55319 | load: 0.00247 [I 2022-10-27 20:21:32] ...trap/engines/engine.py.230: loss: 0.13197 [I 2022-10-27 20:21:32] ...optimizers/trijoint.py.151: total_norm: 24.043846 [I 2022-10-27 20:21:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 1100/2304 [I 2022-10-27 20:21:38] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:12 [I 2022-10-27 20:21:38] ...trap/engines/engine.py.229: process: 0.55450 | load: 0.00247 [I 2022-10-27 20:21:38] ...trap/engines/engine.py.230: loss: 0.08638 [I 2022-10-27 20:21:38] ...optimizers/trijoint.py.151: total_norm: 11.873219 [I 2022-10-27 20:21:44] ...trap/engines/engine.py.224: train: epoch 86 | batch 1110/2304 [I 2022-10-27 20:21:44] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:00 [I 2022-10-27 20:21:44] ...trap/engines/engine.py.229: process: 0.54924 | load: 0.00251 [I 2022-10-27 20:21:44] ...trap/engines/engine.py.230: loss: 0.09539 [I 2022-10-27 20:21:44] ...optimizers/trijoint.py.151: total_norm: 16.563519 [I 2022-10-27 20:21:49] ...trap/engines/engine.py.224: train: epoch 86 | batch 1120/2304 [I 2022-10-27 20:21:49] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:04 [I 2022-10-27 20:21:49] ...trap/engines/engine.py.229: process: 0.56545 | load: 0.00247 [I 2022-10-27 20:21:49] ...trap/engines/engine.py.230: loss: 0.16504 [I 2022-10-27 20:21:49] ...optimizers/trijoint.py.151: total_norm: 35.476818 [I 2022-10-27 20:21:55] ...trap/engines/engine.py.224: train: epoch 86 | batch 1130/2304 [I 2022-10-27 20:21:55] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:10:52 [I 2022-10-27 20:21:55] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00251 [I 2022-10-27 20:21:55] ...trap/engines/engine.py.230: loss: 0.16047 [I 2022-10-27 20:21:55] ...optimizers/trijoint.py.151: total_norm: 14.759553 [I 2022-10-27 20:22:00] ...trap/engines/engine.py.224: train: epoch 86 | batch 1140/2304 [I 2022-10-27 20:22:00] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:53 [I 2022-10-27 20:22:00] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00251 [I 2022-10-27 20:22:00] ...trap/engines/engine.py.230: loss: 0.16076 [I 2022-10-27 20:22:00] ...optimizers/trijoint.py.151: total_norm: 22.607204 [I 2022-10-27 20:22:06] ...trap/engines/engine.py.224: train: epoch 86 | batch 1150/2304 [I 2022-10-27 20:22:06] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:38 [I 2022-10-27 20:22:06] ...trap/engines/engine.py.229: process: 0.54981 | load: 0.00246 [I 2022-10-27 20:22:06] ...trap/engines/engine.py.230: loss: 0.16651 [I 2022-10-27 20:22:06] ...optimizers/trijoint.py.151: total_norm: 56.860725 [I 2022-10-27 20:22:12] ...trap/engines/engine.py.224: train: epoch 86 | batch 1160/2304 [I 2022-10-27 20:22:12] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:33 [I 2022-10-27 20:22:12] ...trap/engines/engine.py.229: process: 0.55635 | load: 0.00260 [I 2022-10-27 20:22:12] ...trap/engines/engine.py.230: loss: 0.15658 [I 2022-10-27 20:22:12] ...optimizers/trijoint.py.151: total_norm: 7.857677 [I 2022-10-27 20:22:17] ...trap/engines/engine.py.224: train: epoch 86 | batch 1170/2304 [I 2022-10-27 20:22:17] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:30 [I 2022-10-27 20:22:17] ...trap/engines/engine.py.229: process: 0.55069 | load: 0.00249 [I 2022-10-27 20:22:17] ...trap/engines/engine.py.230: loss: 0.13318 [I 2022-10-27 20:22:17] ...optimizers/trijoint.py.151: total_norm: 11.031569 [I 2022-10-27 20:22:23] ...trap/engines/engine.py.224: train: epoch 86 | batch 1180/2304 [I 2022-10-27 20:22:23] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:23 [I 2022-10-27 20:22:23] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00245 [I 2022-10-27 20:22:23] ...trap/engines/engine.py.230: loss: 0.15277 [I 2022-10-27 20:22:23] ...optimizers/trijoint.py.151: total_norm: 18.718077 [I 2022-10-27 20:22:28] ...trap/engines/engine.py.224: train: epoch 86 | batch 1190/2304 [I 2022-10-27 20:22:28] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:19 [I 2022-10-27 20:22:28] ...trap/engines/engine.py.229: process: 0.56789 | load: 0.00244 [I 2022-10-27 20:22:28] ...trap/engines/engine.py.230: loss: 0.09773 [I 2022-10-27 20:22:28] ...optimizers/trijoint.py.151: total_norm: 13.798272 [I 2022-10-27 20:22:34] ...trap/engines/engine.py.224: train: epoch 86 | batch 1200/2304 [I 2022-10-27 20:22:34] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:15 [I 2022-10-27 20:22:34] ...trap/engines/engine.py.229: process: 0.53518 | load: 0.00246 [I 2022-10-27 20:22:34] ...trap/engines/engine.py.230: loss: 0.13811 [I 2022-10-27 20:22:34] ...optimizers/trijoint.py.151: total_norm: 17.540323 [I 2022-10-27 20:22:39] ...trap/engines/engine.py.224: train: epoch 86 | batch 1210/2304 [I 2022-10-27 20:22:39] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:08 [I 2022-10-27 20:22:39] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00247 [I 2022-10-27 20:22:39] ...trap/engines/engine.py.230: loss: 0.13140 [I 2022-10-27 20:22:39] ...optimizers/trijoint.py.151: total_norm: 15.598253 [I 2022-10-27 20:22:45] ...trap/engines/engine.py.224: train: epoch 86 | batch 1220/2304 [I 2022-10-27 20:22:45] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:04 [I 2022-10-27 20:22:45] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00244 [I 2022-10-27 20:22:45] ...trap/engines/engine.py.230: loss: 0.16909 [I 2022-10-27 20:22:45] ...optimizers/trijoint.py.151: total_norm: 16.541645 [I 2022-10-27 20:22:51] ...trap/engines/engine.py.224: train: epoch 86 | batch 1230/2304 [I 2022-10-27 20:22:51] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:09:57 [I 2022-10-27 20:22:51] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00250 [I 2022-10-27 20:22:51] ...trap/engines/engine.py.230: loss: 0.19407 [I 2022-10-27 20:22:51] ...optimizers/trijoint.py.151: total_norm: 32.862015 [I 2022-10-27 20:22:56] ...trap/engines/engine.py.224: train: epoch 86 | batch 1240/2304 [I 2022-10-27 20:22:56] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:53 [I 2022-10-27 20:22:56] ...trap/engines/engine.py.229: process: 0.56905 | load: 0.00247 [I 2022-10-27 20:22:56] ...trap/engines/engine.py.230: loss: 0.12483 [I 2022-10-27 20:22:56] ...optimizers/trijoint.py.151: total_norm: 25.687439 [I 2022-10-27 20:23:02] ...trap/engines/engine.py.224: train: epoch 86 | batch 1250/2304 [I 2022-10-27 20:23:02] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:47 [I 2022-10-27 20:23:02] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00252 [I 2022-10-27 20:23:02] ...trap/engines/engine.py.230: loss: 0.13076 [I 2022-10-27 20:23:02] ...optimizers/trijoint.py.151: total_norm: 11.972959 [I 2022-10-27 20:23:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 1260/2304 [I 2022-10-27 20:23:07] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:09:38 [I 2022-10-27 20:23:07] ...trap/engines/engine.py.229: process: 0.55618 | load: 0.00254 [I 2022-10-27 20:23:07] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-27 20:23:07] ...optimizers/trijoint.py.151: total_norm: 15.142409 [I 2022-10-27 20:23:13] ...trap/engines/engine.py.224: train: epoch 86 | batch 1270/2304 [I 2022-10-27 20:23:13] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:32 [I 2022-10-27 20:23:13] ...trap/engines/engine.py.229: process: 0.54905 | load: 0.00248 [I 2022-10-27 20:23:13] ...trap/engines/engine.py.230: loss: 0.11066 [I 2022-10-27 20:23:13] ...optimizers/trijoint.py.151: total_norm: 20.813786 [I 2022-10-27 20:23:18] ...trap/engines/engine.py.224: train: epoch 86 | batch 1280/2304 [I 2022-10-27 20:23:18] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:09:27 [I 2022-10-27 20:23:18] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00269 [I 2022-10-27 20:23:18] ...trap/engines/engine.py.230: loss: 0.10110 [I 2022-10-27 20:23:18] ...optimizers/trijoint.py.151: total_norm: 22.674128 [I 2022-10-27 20:23:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 1290/2304 [I 2022-10-27 20:23:24] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:21 [I 2022-10-27 20:23:24] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00247 [I 2022-10-27 20:23:24] ...trap/engines/engine.py.230: loss: 0.11426 [I 2022-10-27 20:23:24] ...optimizers/trijoint.py.151: total_norm: 12.063571 [I 2022-10-27 20:23:29] ...trap/engines/engine.py.224: train: epoch 86 | batch 1300/2304 [I 2022-10-27 20:23:29] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:16 [I 2022-10-27 20:23:29] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00253 [I 2022-10-27 20:23:29] ...trap/engines/engine.py.230: loss: 0.16554 [I 2022-10-27 20:23:29] ...optimizers/trijoint.py.151: total_norm: 9.796470 [I 2022-10-27 20:23:35] ...trap/engines/engine.py.224: train: epoch 86 | batch 1310/2304 [I 2022-10-27 20:23:35] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:14 [I 2022-10-27 20:23:35] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00243 [I 2022-10-27 20:23:35] ...trap/engines/engine.py.230: loss: 0.14289 [I 2022-10-27 20:23:35] ...optimizers/trijoint.py.151: total_norm: 13.326835 [I 2022-10-27 20:23:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 1320/2304 [I 2022-10-27 20:23:41] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:07 [I 2022-10-27 20:23:41] ...trap/engines/engine.py.229: process: 0.54831 | load: 0.00252 [I 2022-10-27 20:23:41] ...trap/engines/engine.py.230: loss: 0.13955 [I 2022-10-27 20:23:41] ...optimizers/trijoint.py.151: total_norm: 30.357857 [I 2022-10-27 20:23:46] ...trap/engines/engine.py.224: train: epoch 86 | batch 1330/2304 [I 2022-10-27 20:23:46] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:09:08 [I 2022-10-27 20:23:46] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00266 [I 2022-10-27 20:23:46] ...trap/engines/engine.py.230: loss: 0.15512 [I 2022-10-27 20:23:46] ...optimizers/trijoint.py.151: total_norm: 15.496332 [I 2022-10-27 20:23:52] ...trap/engines/engine.py.224: train: epoch 86 | batch 1340/2304 [I 2022-10-27 20:23:52] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:08:59 [I 2022-10-27 20:23:52] ...trap/engines/engine.py.229: process: 0.55486 | load: 0.00244 [I 2022-10-27 20:23:52] ...trap/engines/engine.py.230: loss: 0.16612 [I 2022-10-27 20:23:52] ...optimizers/trijoint.py.151: total_norm: 15.207556 [I 2022-10-27 20:23:57] ...trap/engines/engine.py.224: train: epoch 86 | batch 1350/2304 [I 2022-10-27 20:23:57] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:08:53 [I 2022-10-27 20:23:57] ...trap/engines/engine.py.229: process: 0.57421 | load: 0.00245 [I 2022-10-27 20:23:57] ...trap/engines/engine.py.230: loss: 0.16330 [I 2022-10-27 20:23:57] ...optimizers/trijoint.py.151: total_norm: 16.375805 [I 2022-10-27 20:24:03] ...trap/engines/engine.py.224: train: epoch 86 | batch 1360/2304 [I 2022-10-27 20:24:03] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:46 [I 2022-10-27 20:24:03] ...trap/engines/engine.py.229: process: 0.55821 | load: 0.00245 [I 2022-10-27 20:24:03] ...trap/engines/engine.py.230: loss: 0.15236 [I 2022-10-27 20:24:03] ...optimizers/trijoint.py.151: total_norm: 16.577545 [I 2022-10-27 20:24:09] ...trap/engines/engine.py.224: train: epoch 86 | batch 1370/2304 [I 2022-10-27 20:24:09] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:36 [I 2022-10-27 20:24:09] ...trap/engines/engine.py.229: process: 0.54834 | load: 0.00241 [I 2022-10-27 20:24:09] ...trap/engines/engine.py.230: loss: 0.14633 [I 2022-10-27 20:24:09] ...optimizers/trijoint.py.151: total_norm: 7.649235 [I 2022-10-27 20:24:14] ...trap/engines/engine.py.224: train: epoch 86 | batch 1380/2304 [I 2022-10-27 20:24:14] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:31 [I 2022-10-27 20:24:14] ...trap/engines/engine.py.229: process: 0.55230 | load: 0.00241 [I 2022-10-27 20:24:14] ...trap/engines/engine.py.230: loss: 0.13942 [I 2022-10-27 20:24:14] ...optimizers/trijoint.py.151: total_norm: 16.776793 [I 2022-10-27 20:24:20] ...trap/engines/engine.py.224: train: epoch 86 | batch 1390/2304 [I 2022-10-27 20:24:20] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:28 [I 2022-10-27 20:24:20] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00246 [I 2022-10-27 20:24:20] ...trap/engines/engine.py.230: loss: 0.12050 [I 2022-10-27 20:24:20] ...optimizers/trijoint.py.151: total_norm: 12.693211 [I 2022-10-27 20:24:25] ...trap/engines/engine.py.224: train: epoch 86 | batch 1400/2304 [I 2022-10-27 20:24:25] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:08:25 [I 2022-10-27 20:24:25] ...trap/engines/engine.py.229: process: 0.55756 | load: 0.00245 [I 2022-10-27 20:24:25] ...trap/engines/engine.py.230: loss: 0.14285 [I 2022-10-27 20:24:25] ...optimizers/trijoint.py.151: total_norm: 10.360498 [I 2022-10-27 20:24:31] ...trap/engines/engine.py.224: train: epoch 86 | batch 1410/2304 [I 2022-10-27 20:24:31] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:18 [I 2022-10-27 20:24:31] ...trap/engines/engine.py.229: process: 0.55926 | load: 0.00241 [I 2022-10-27 20:24:31] ...trap/engines/engine.py.230: loss: 0.11686 [I 2022-10-27 20:24:31] ...optimizers/trijoint.py.151: total_norm: 15.784893 [I 2022-10-27 20:24:37] ...trap/engines/engine.py.224: train: epoch 86 | batch 1420/2304 [I 2022-10-27 20:24:37] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:13 [I 2022-10-27 20:24:37] ...trap/engines/engine.py.229: process: 0.56440 | load: 0.00251 [I 2022-10-27 20:24:37] ...trap/engines/engine.py.230: loss: 0.15373 [I 2022-10-27 20:24:37] ...optimizers/trijoint.py.151: total_norm: 30.149822 [I 2022-10-27 20:24:42] ...trap/engines/engine.py.224: train: epoch 86 | batch 1430/2304 [I 2022-10-27 20:24:42] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:09 [I 2022-10-27 20:24:42] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00248 [I 2022-10-27 20:24:42] ...trap/engines/engine.py.230: loss: 0.10606 [I 2022-10-27 20:24:42] ...optimizers/trijoint.py.151: total_norm: 15.599669 [I 2022-10-27 20:24:48] ...trap/engines/engine.py.224: train: epoch 86 | batch 1440/2304 [I 2022-10-27 20:24:48] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:07:57 [I 2022-10-27 20:24:48] ...trap/engines/engine.py.229: process: 0.54622 | load: 0.00246 [I 2022-10-27 20:24:48] ...trap/engines/engine.py.230: loss: 0.11008 [I 2022-10-27 20:24:48] ...optimizers/trijoint.py.151: total_norm: 14.114100 [I 2022-10-27 20:24:53] ...trap/engines/engine.py.224: train: epoch 86 | batch 1450/2304 [I 2022-10-27 20:24:53] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:07:55 [I 2022-10-27 20:24:53] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00246 [I 2022-10-27 20:24:53] ...trap/engines/engine.py.230: loss: 0.18245 [I 2022-10-27 20:24:53] ...optimizers/trijoint.py.151: total_norm: 21.804295 [I 2022-10-27 20:24:59] ...trap/engines/engine.py.224: train: epoch 86 | batch 1460/2304 [I 2022-10-27 20:24:59] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:47 [I 2022-10-27 20:24:59] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00244 [I 2022-10-27 20:24:59] ...trap/engines/engine.py.230: loss: 0.12846 [I 2022-10-27 20:24:59] ...optimizers/trijoint.py.151: total_norm: 8.899223 [I 2022-10-27 20:25:04] ...trap/engines/engine.py.224: train: epoch 86 | batch 1470/2304 [I 2022-10-27 20:25:04] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:44 [I 2022-10-27 20:25:04] ...trap/engines/engine.py.229: process: 0.55144 | load: 0.00247 [I 2022-10-27 20:25:04] ...trap/engines/engine.py.230: loss: 0.12596 [I 2022-10-27 20:25:04] ...optimizers/trijoint.py.151: total_norm: 20.513634 [I 2022-10-27 20:25:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 1480/2304 [I 2022-10-27 20:25:10] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:36 [I 2022-10-27 20:25:10] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00242 [I 2022-10-27 20:25:10] ...trap/engines/engine.py.230: loss: 0.14506 [I 2022-10-27 20:25:10] ...optimizers/trijoint.py.151: total_norm: 14.180759 [I 2022-10-27 20:25:15] ...trap/engines/engine.py.224: train: epoch 86 | batch 1490/2304 [I 2022-10-27 20:25:15] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:33 [I 2022-10-27 20:25:15] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00244 [I 2022-10-27 20:25:15] ...trap/engines/engine.py.230: loss: 0.12226 [I 2022-10-27 20:25:15] ...optimizers/trijoint.py.151: total_norm: 25.690630 [I 2022-10-27 20:25:21] ...trap/engines/engine.py.224: train: epoch 86 | batch 1500/2304 [I 2022-10-27 20:25:21] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:33 [I 2022-10-27 20:25:21] ...trap/engines/engine.py.229: process: 0.58592 | load: 0.00248 [I 2022-10-27 20:25:21] ...trap/engines/engine.py.230: loss: 0.11286 [I 2022-10-27 20:25:21] ...optimizers/trijoint.py.151: total_norm: 12.320650 [I 2022-10-27 20:25:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 1510/2304 [I 2022-10-27 20:25:27] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:25 [I 2022-10-27 20:25:27] ...trap/engines/engine.py.229: process: 0.56583 | load: 0.00251 [I 2022-10-27 20:25:27] ...trap/engines/engine.py.230: loss: 0.13713 [I 2022-10-27 20:25:27] ...optimizers/trijoint.py.151: total_norm: 15.587175 [I 2022-10-27 20:25:32] ...trap/engines/engine.py.224: train: epoch 86 | batch 1520/2304 [I 2022-10-27 20:25:32] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:07:17 [I 2022-10-27 20:25:32] ...trap/engines/engine.py.229: process: 0.56583 | load: 0.00245 [I 2022-10-27 20:25:32] ...trap/engines/engine.py.230: loss: 0.18612 [I 2022-10-27 20:25:32] ...optimizers/trijoint.py.151: total_norm: 30.550632 [I 2022-10-27 20:25:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 1530/2304 [I 2022-10-27 20:25:38] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:08 [I 2022-10-27 20:25:38] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00245 [I 2022-10-27 20:25:38] ...trap/engines/engine.py.230: loss: 0.14030 [I 2022-10-27 20:25:38] ...optimizers/trijoint.py.151: total_norm: 15.935870 [I 2022-10-27 20:25:43] ...trap/engines/engine.py.224: train: epoch 86 | batch 1540/2304 [I 2022-10-27 20:25:43] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:03 [I 2022-10-27 20:25:43] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00245 [I 2022-10-27 20:25:43] ...trap/engines/engine.py.230: loss: 0.13615 [I 2022-10-27 20:25:43] ...optimizers/trijoint.py.151: total_norm: 15.295578 [I 2022-10-27 20:25:49] ...trap/engines/engine.py.224: train: epoch 86 | batch 1550/2304 [I 2022-10-27 20:25:49] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:06:56 [I 2022-10-27 20:25:49] ...trap/engines/engine.py.229: process: 0.55039 | load: 0.00243 [I 2022-10-27 20:25:49] ...trap/engines/engine.py.230: loss: 0.11240 [I 2022-10-27 20:25:49] ...optimizers/trijoint.py.151: total_norm: 16.529942 [I 2022-10-27 20:25:55] ...trap/engines/engine.py.224: train: epoch 86 | batch 1560/2304 [I 2022-10-27 20:25:55] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:53 [I 2022-10-27 20:25:55] ...trap/engines/engine.py.229: process: 0.55280 | load: 0.00242 [I 2022-10-27 20:25:55] ...trap/engines/engine.py.230: loss: 0.11201 [I 2022-10-27 20:25:55] ...optimizers/trijoint.py.151: total_norm: 20.963690 [I 2022-10-27 20:26:00] ...trap/engines/engine.py.224: train: epoch 86 | batch 1570/2304 [I 2022-10-27 20:26:00] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:45 [I 2022-10-27 20:26:00] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00243 [I 2022-10-27 20:26:00] ...trap/engines/engine.py.230: loss: 0.12758 [I 2022-10-27 20:26:00] ...optimizers/trijoint.py.151: total_norm: 36.564644 [I 2022-10-27 20:26:06] ...trap/engines/engine.py.224: train: epoch 86 | batch 1580/2304 [I 2022-10-27 20:26:06] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:41 [I 2022-10-27 20:26:06] ...trap/engines/engine.py.229: process: 0.54133 | load: 0.00240 [I 2022-10-27 20:26:06] ...trap/engines/engine.py.230: loss: 0.10882 [I 2022-10-27 20:26:06] ...optimizers/trijoint.py.151: total_norm: 30.969030 [I 2022-10-27 20:26:11] ...trap/engines/engine.py.224: train: epoch 86 | batch 1590/2304 [I 2022-10-27 20:26:11] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:36 [I 2022-10-27 20:26:11] ...trap/engines/engine.py.229: process: 0.55768 | load: 0.00240 [I 2022-10-27 20:26:11] ...trap/engines/engine.py.230: loss: 0.09383 [I 2022-10-27 20:26:11] ...optimizers/trijoint.py.151: total_norm: 9.541195 [I 2022-10-27 20:26:17] ...trap/engines/engine.py.224: train: epoch 86 | batch 1600/2304 [I 2022-10-27 20:26:17] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:30 [I 2022-10-27 20:26:17] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00245 [I 2022-10-27 20:26:17] ...trap/engines/engine.py.230: loss: 0.12747 [I 2022-10-27 20:26:17] ...optimizers/trijoint.py.151: total_norm: 13.200832 [I 2022-10-27 20:26:22] ...trap/engines/engine.py.224: train: epoch 86 | batch 1610/2304 [I 2022-10-27 20:26:22] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:43 [I 2022-10-27 20:26:22] ...trap/engines/engine.py.229: process: 0.61280 | load: 0.00253 [I 2022-10-27 20:26:22] ...trap/engines/engine.py.230: loss: 0.10186 [I 2022-10-27 20:26:22] ...optimizers/trijoint.py.151: total_norm: 9.390038 [I 2022-10-27 20:26:28] ...trap/engines/engine.py.224: train: epoch 86 | batch 1620/2304 [I 2022-10-27 20:26:28] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:21 [I 2022-10-27 20:26:28] ...trap/engines/engine.py.229: process: 0.54442 | load: 0.00248 [I 2022-10-27 20:26:28] ...trap/engines/engine.py.230: loss: 0.11319 [I 2022-10-27 20:26:28] ...optimizers/trijoint.py.151: total_norm: 8.085449 [I 2022-10-27 20:26:34] ...trap/engines/engine.py.224: train: epoch 86 | batch 1630/2304 [I 2022-10-27 20:26:34] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:27 [I 2022-10-27 20:26:34] ...trap/engines/engine.py.229: process: 0.62926 | load: 0.00253 [I 2022-10-27 20:26:34] ...trap/engines/engine.py.230: loss: 0.09615 [I 2022-10-27 20:26:34] ...optimizers/trijoint.py.151: total_norm: 8.646914 [I 2022-10-27 20:26:39] ...trap/engines/engine.py.224: train: epoch 86 | batch 1640/2304 [I 2022-10-27 20:26:39] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:12 [I 2022-10-27 20:26:39] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00243 [I 2022-10-27 20:26:39] ...trap/engines/engine.py.230: loss: 0.17527 [I 2022-10-27 20:26:39] ...optimizers/trijoint.py.151: total_norm: 15.171271 [I 2022-10-27 20:26:45] ...trap/engines/engine.py.224: train: epoch 86 | batch 1650/2304 [I 2022-10-27 20:26:45] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:05 [I 2022-10-27 20:26:45] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00245 [I 2022-10-27 20:26:45] ...trap/engines/engine.py.230: loss: 0.11715 [I 2022-10-27 20:26:45] ...optimizers/trijoint.py.151: total_norm: 13.331938 [I 2022-10-27 20:26:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 1660/2304 [I 2022-10-27 20:26:50] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:05:56 [I 2022-10-27 20:26:50] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00250 [I 2022-10-27 20:26:50] ...trap/engines/engine.py.230: loss: 0.17428 [I 2022-10-27 20:26:50] ...optimizers/trijoint.py.151: total_norm: 19.724236 [I 2022-10-27 20:26:56] ...trap/engines/engine.py.224: train: epoch 86 | batch 1670/2304 [I 2022-10-27 20:26:56] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:05:51 [I 2022-10-27 20:26:56] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00246 [I 2022-10-27 20:26:56] ...trap/engines/engine.py.230: loss: 0.15241 [I 2022-10-27 20:26:56] ...optimizers/trijoint.py.151: total_norm: 23.677383 [I 2022-10-27 20:27:02] ...trap/engines/engine.py.224: train: epoch 86 | batch 1680/2304 [I 2022-10-27 20:27:02] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:42 [I 2022-10-27 20:27:02] ...trap/engines/engine.py.229: process: 0.52909 | load: 0.00250 [I 2022-10-27 20:27:02] ...trap/engines/engine.py.230: loss: 0.13391 [I 2022-10-27 20:27:02] ...optimizers/trijoint.py.151: total_norm: 15.711723 [I 2022-10-27 20:27:07] ...trap/engines/engine.py.224: train: epoch 86 | batch 1690/2304 [I 2022-10-27 20:27:07] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:40 [I 2022-10-27 20:27:07] ...trap/engines/engine.py.229: process: 0.56243 | load: 0.00248 [I 2022-10-27 20:27:07] ...trap/engines/engine.py.230: loss: 0.10993 [I 2022-10-27 20:27:07] ...optimizers/trijoint.py.151: total_norm: 16.484842 [I 2022-10-27 20:27:13] ...trap/engines/engine.py.224: train: epoch 86 | batch 1700/2304 [I 2022-10-27 20:27:13] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:41 [I 2022-10-27 20:27:13] ...trap/engines/engine.py.229: process: 0.54956 | load: 0.00262 [I 2022-10-27 20:27:13] ...trap/engines/engine.py.230: loss: 0.23757 [I 2022-10-27 20:27:13] ...optimizers/trijoint.py.151: total_norm: 17.254625 [I 2022-10-27 20:27:18] ...trap/engines/engine.py.224: train: epoch 86 | batch 1710/2304 [I 2022-10-27 20:27:18] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:30 [I 2022-10-27 20:27:18] ...trap/engines/engine.py.229: process: 0.55381 | load: 0.00248 [I 2022-10-27 20:27:18] ...trap/engines/engine.py.230: loss: 0.12530 [I 2022-10-27 20:27:18] ...optimizers/trijoint.py.151: total_norm: 7.861689 [I 2022-10-27 20:27:24] ...trap/engines/engine.py.224: train: epoch 86 | batch 1720/2304 [I 2022-10-27 20:27:24] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:24 [I 2022-10-27 20:27:24] ...trap/engines/engine.py.229: process: 0.56994 | load: 0.00245 [I 2022-10-27 20:27:24] ...trap/engines/engine.py.230: loss: 0.10624 [I 2022-10-27 20:27:24] ...optimizers/trijoint.py.151: total_norm: 16.588806 [I 2022-10-27 20:27:29] ...trap/engines/engine.py.224: train: epoch 86 | batch 1730/2304 [I 2022-10-27 20:27:29] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:18 [I 2022-10-27 20:27:29] ...trap/engines/engine.py.229: process: 0.54766 | load: 0.00242 [I 2022-10-27 20:27:29] ...trap/engines/engine.py.230: loss: 0.13864 [I 2022-10-27 20:27:29] ...optimizers/trijoint.py.151: total_norm: 24.955482 [I 2022-10-27 20:27:35] ...trap/engines/engine.py.224: train: epoch 86 | batch 1740/2304 [I 2022-10-27 20:27:35] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:14 [I 2022-10-27 20:27:35] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00244 [I 2022-10-27 20:27:35] ...trap/engines/engine.py.230: loss: 0.14391 [I 2022-10-27 20:27:35] ...optimizers/trijoint.py.151: total_norm: 17.579977 [I 2022-10-27 20:27:41] ...trap/engines/engine.py.224: train: epoch 86 | batch 1750/2304 [I 2022-10-27 20:27:41] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:08 [I 2022-10-27 20:27:41] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00242 [I 2022-10-27 20:27:41] ...trap/engines/engine.py.230: loss: 0.09145 [I 2022-10-27 20:27:41] ...optimizers/trijoint.py.151: total_norm: 7.991108 [I 2022-10-27 20:27:46] ...trap/engines/engine.py.224: train: epoch 86 | batch 1760/2304 [I 2022-10-27 20:27:46] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:03 [I 2022-10-27 20:27:46] ...trap/engines/engine.py.229: process: 0.54850 | load: 0.00244 [I 2022-10-27 20:27:46] ...trap/engines/engine.py.230: loss: 0.09980 [I 2022-10-27 20:27:46] ...optimizers/trijoint.py.151: total_norm: 16.074080 [I 2022-10-27 20:27:52] ...trap/engines/engine.py.224: train: epoch 86 | batch 1770/2304 [I 2022-10-27 20:27:52] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:05:02 [I 2022-10-27 20:27:52] ...trap/engines/engine.py.229: process: 0.57130 | load: 0.00240 [I 2022-10-27 20:27:52] ...trap/engines/engine.py.230: loss: 0.10127 [I 2022-10-27 20:27:52] ...optimizers/trijoint.py.151: total_norm: 21.665535 [I 2022-10-27 20:27:57] ...trap/engines/engine.py.224: train: epoch 86 | batch 1780/2304 [I 2022-10-27 20:27:57] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:52 [I 2022-10-27 20:27:57] ...trap/engines/engine.py.229: process: 0.54200 | load: 0.00242 [I 2022-10-27 20:27:57] ...trap/engines/engine.py.230: loss: 0.11412 [I 2022-10-27 20:27:57] ...optimizers/trijoint.py.151: total_norm: 36.929916 [I 2022-10-27 20:28:03] ...trap/engines/engine.py.224: train: epoch 86 | batch 1790/2304 [I 2022-10-27 20:28:03] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:47 [I 2022-10-27 20:28:03] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00242 [I 2022-10-27 20:28:03] ...trap/engines/engine.py.230: loss: 0.16537 [I 2022-10-27 20:28:03] ...optimizers/trijoint.py.151: total_norm: 17.622883 [I 2022-10-27 20:28:09] ...trap/engines/engine.py.224: train: epoch 86 | batch 1800/2304 [I 2022-10-27 20:28:09] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:40 [I 2022-10-27 20:28:09] ...trap/engines/engine.py.229: process: 0.55062 | load: 0.00251 [I 2022-10-27 20:28:09] ...trap/engines/engine.py.230: loss: 0.13146 [I 2022-10-27 20:28:09] ...optimizers/trijoint.py.151: total_norm: 12.000878 [I 2022-10-27 20:28:14] ...trap/engines/engine.py.224: train: epoch 86 | batch 1810/2304 [I 2022-10-27 20:28:14] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:34 [I 2022-10-27 20:28:14] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00251 [I 2022-10-27 20:28:14] ...trap/engines/engine.py.230: loss: 0.09423 [I 2022-10-27 20:28:14] ...optimizers/trijoint.py.151: total_norm: 9.432150 [I 2022-10-27 20:28:20] ...trap/engines/engine.py.224: train: epoch 86 | batch 1820/2304 [I 2022-10-27 20:28:20] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:30 [I 2022-10-27 20:28:20] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00243 [I 2022-10-27 20:28:20] ...trap/engines/engine.py.230: loss: 0.11536 [I 2022-10-27 20:28:20] ...optimizers/trijoint.py.151: total_norm: 12.725905 [I 2022-10-27 20:28:25] ...trap/engines/engine.py.224: train: epoch 86 | batch 1830/2304 [I 2022-10-27 20:28:25] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:24 [I 2022-10-27 20:28:25] ...trap/engines/engine.py.229: process: 0.56622 | load: 0.00242 [I 2022-10-27 20:28:25] ...trap/engines/engine.py.230: loss: 0.14070 [I 2022-10-27 20:28:25] ...optimizers/trijoint.py.151: total_norm: 17.244444 [I 2022-10-27 20:28:31] ...trap/engines/engine.py.224: train: epoch 86 | batch 1840/2304 [I 2022-10-27 20:28:31] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:18 [I 2022-10-27 20:28:31] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00243 [I 2022-10-27 20:28:31] ...trap/engines/engine.py.230: loss: 0.09278 [I 2022-10-27 20:28:31] ...optimizers/trijoint.py.151: total_norm: 13.012547 [I 2022-10-27 20:28:37] ...trap/engines/engine.py.224: train: epoch 86 | batch 1850/2304 [I 2022-10-27 20:28:37] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:13 [I 2022-10-27 20:28:37] ...trap/engines/engine.py.229: process: 0.56776 | load: 0.00245 [I 2022-10-27 20:28:37] ...trap/engines/engine.py.230: loss: 0.14767 [I 2022-10-27 20:28:37] ...optimizers/trijoint.py.151: total_norm: 44.713047 [I 2022-10-27 20:28:42] ...trap/engines/engine.py.224: train: epoch 86 | batch 1860/2304 [I 2022-10-27 20:28:42] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:08 [I 2022-10-27 20:28:42] ...trap/engines/engine.py.229: process: 0.56182 | load: 0.00245 [I 2022-10-27 20:28:42] ...trap/engines/engine.py.230: loss: 0.10253 [I 2022-10-27 20:28:42] ...optimizers/trijoint.py.151: total_norm: 10.062729 [I 2022-10-27 20:28:48] ...trap/engines/engine.py.224: train: epoch 86 | batch 1870/2304 [I 2022-10-27 20:28:48] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:05 [I 2022-10-27 20:28:48] ...trap/engines/engine.py.229: process: 0.58231 | load: 0.00252 [I 2022-10-27 20:28:48] ...trap/engines/engine.py.230: loss: 0.11682 [I 2022-10-27 20:28:48] ...optimizers/trijoint.py.151: total_norm: 14.559703 [I 2022-10-27 20:28:54] ...trap/engines/engine.py.224: train: epoch 86 | batch 1880/2304 [I 2022-10-27 20:28:54] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:00 [I 2022-10-27 20:28:54] ...trap/engines/engine.py.229: process: 0.57192 | load: 0.00254 [I 2022-10-27 20:28:54] ...trap/engines/engine.py.230: loss: 0.25327 [I 2022-10-27 20:28:54] ...optimizers/trijoint.py.151: total_norm: 15.981457 [I 2022-10-27 20:28:59] ...trap/engines/engine.py.224: train: epoch 86 | batch 1890/2304 [I 2022-10-27 20:28:59] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:03:51 [I 2022-10-27 20:28:59] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00251 [I 2022-10-27 20:28:59] ...trap/engines/engine.py.230: loss: 0.10051 [I 2022-10-27 20:28:59] ...optimizers/trijoint.py.151: total_norm: 16.858292 [I 2022-10-27 20:29:05] ...trap/engines/engine.py.224: train: epoch 86 | batch 1900/2304 [I 2022-10-27 20:29:05] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:49 [I 2022-10-27 20:29:05] ...trap/engines/engine.py.229: process: 0.56597 | load: 0.00246 [I 2022-10-27 20:29:05] ...trap/engines/engine.py.230: loss: 0.11019 [I 2022-10-27 20:29:05] ...optimizers/trijoint.py.151: total_norm: 21.798586 [I 2022-10-27 20:29:11] ...trap/engines/engine.py.224: train: epoch 86 | batch 1910/2304 [I 2022-10-27 20:29:11] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:39 [I 2022-10-27 20:29:11] ...trap/engines/engine.py.229: process: 0.54530 | load: 0.00242 [I 2022-10-27 20:29:11] ...trap/engines/engine.py.230: loss: 0.18766 [I 2022-10-27 20:29:11] ...optimizers/trijoint.py.151: total_norm: 23.608522 [I 2022-10-27 20:29:16] ...trap/engines/engine.py.224: train: epoch 86 | batch 1920/2304 [I 2022-10-27 20:29:16] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:36 [I 2022-10-27 20:29:16] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00246 [I 2022-10-27 20:29:16] ...trap/engines/engine.py.230: loss: 0.11037 [I 2022-10-27 20:29:16] ...optimizers/trijoint.py.151: total_norm: 12.363874 [I 2022-10-27 20:29:22] ...trap/engines/engine.py.224: train: epoch 86 | batch 1930/2304 [I 2022-10-27 20:29:22] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:27 [I 2022-10-27 20:29:22] ...trap/engines/engine.py.229: process: 0.55531 | load: 0.00246 [I 2022-10-27 20:29:22] ...trap/engines/engine.py.230: loss: 0.12904 [I 2022-10-27 20:29:22] ...optimizers/trijoint.py.151: total_norm: 12.235249 [I 2022-10-27 20:29:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 1940/2304 [I 2022-10-27 20:29:27] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:23 [I 2022-10-27 20:29:27] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00242 [I 2022-10-27 20:29:27] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-27 20:29:27] ...optimizers/trijoint.py.151: total_norm: 10.335077 [I 2022-10-27 20:29:33] ...trap/engines/engine.py.224: train: epoch 86 | batch 1950/2304 [I 2022-10-27 20:29:33] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:16 [I 2022-10-27 20:29:33] ...trap/engines/engine.py.229: process: 0.55520 | load: 0.00247 [I 2022-10-27 20:29:33] ...trap/engines/engine.py.230: loss: 0.14421 [I 2022-10-27 20:29:33] ...optimizers/trijoint.py.151: total_norm: 41.031139 [I 2022-10-27 20:29:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 1960/2304 [I 2022-10-27 20:29:38] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:11 [I 2022-10-27 20:29:38] ...trap/engines/engine.py.229: process: 0.55488 | load: 0.00244 [I 2022-10-27 20:29:38] ...trap/engines/engine.py.230: loss: 0.14607 [I 2022-10-27 20:29:38] ...optimizers/trijoint.py.151: total_norm: 26.396265 [I 2022-10-27 20:29:44] ...trap/engines/engine.py.224: train: epoch 86 | batch 1970/2304 [I 2022-10-27 20:29:44] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:06 [I 2022-10-27 20:29:44] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00243 [I 2022-10-27 20:29:44] ...trap/engines/engine.py.230: loss: 0.12263 [I 2022-10-27 20:29:44] ...optimizers/trijoint.py.151: total_norm: 18.103586 [I 2022-10-27 20:29:50] ...trap/engines/engine.py.224: train: epoch 86 | batch 1980/2304 [I 2022-10-27 20:29:50] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:00 [I 2022-10-27 20:29:50] ...trap/engines/engine.py.229: process: 0.54415 | load: 0.00251 [I 2022-10-27 20:29:50] ...trap/engines/engine.py.230: loss: 0.10315 [I 2022-10-27 20:29:50] ...optimizers/trijoint.py.151: total_norm: 10.625447 [I 2022-10-27 20:29:55] ...trap/engines/engine.py.224: train: epoch 86 | batch 1990/2304 [I 2022-10-27 20:29:55] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:02:55 [I 2022-10-27 20:29:55] ...trap/engines/engine.py.229: process: 0.56312 | load: 0.00248 [I 2022-10-27 20:29:55] ...trap/engines/engine.py.230: loss: 0.11679 [I 2022-10-27 20:29:55] ...optimizers/trijoint.py.151: total_norm: 11.098237 [I 2022-10-27 20:30:01] ...trap/engines/engine.py.224: train: epoch 86 | batch 2000/2304 [I 2022-10-27 20:30:01] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:49 [I 2022-10-27 20:30:01] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00244 [I 2022-10-27 20:30:01] ...trap/engines/engine.py.230: loss: 0.10949 [I 2022-10-27 20:30:01] ...optimizers/trijoint.py.151: total_norm: 34.015282 [I 2022-10-27 20:30:06] ...trap/engines/engine.py.224: train: epoch 86 | batch 2010/2304 [I 2022-10-27 20:30:06] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:43 [I 2022-10-27 20:30:06] ...trap/engines/engine.py.229: process: 0.55803 | load: 0.00242 [I 2022-10-27 20:30:06] ...trap/engines/engine.py.230: loss: 0.11666 [I 2022-10-27 20:30:06] ...optimizers/trijoint.py.151: total_norm: 11.108983 [I 2022-10-27 20:30:12] ...trap/engines/engine.py.224: train: epoch 86 | batch 2020/2304 [I 2022-10-27 20:30:12] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:39 [I 2022-10-27 20:30:12] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00245 [I 2022-10-27 20:30:12] ...trap/engines/engine.py.230: loss: 0.10673 [I 2022-10-27 20:30:12] ...optimizers/trijoint.py.151: total_norm: 10.564367 [I 2022-10-27 20:30:18] ...trap/engines/engine.py.224: train: epoch 86 | batch 2030/2304 [I 2022-10-27 20:30:18] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:33 [I 2022-10-27 20:30:18] ...trap/engines/engine.py.229: process: 0.56883 | load: 0.00244 [I 2022-10-27 20:30:18] ...trap/engines/engine.py.230: loss: 0.12636 [I 2022-10-27 20:30:18] ...optimizers/trijoint.py.151: total_norm: 39.443851 [I 2022-10-27 20:30:23] ...trap/engines/engine.py.224: train: epoch 86 | batch 2040/2304 [I 2022-10-27 20:30:23] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:26 [I 2022-10-27 20:30:23] ...trap/engines/engine.py.229: process: 0.55248 | load: 0.00247 [I 2022-10-27 20:30:23] ...trap/engines/engine.py.230: loss: 0.11453 [I 2022-10-27 20:30:23] ...optimizers/trijoint.py.151: total_norm: 17.783527 [I 2022-10-27 20:30:29] ...trap/engines/engine.py.224: train: epoch 86 | batch 2050/2304 [I 2022-10-27 20:30:29] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:21 [I 2022-10-27 20:30:29] ...trap/engines/engine.py.229: process: 0.56319 | load: 0.00243 [I 2022-10-27 20:30:29] ...trap/engines/engine.py.230: loss: 0.15840 [I 2022-10-27 20:30:29] ...optimizers/trijoint.py.151: total_norm: 48.458054 [I 2022-10-27 20:30:34] ...trap/engines/engine.py.224: train: epoch 86 | batch 2060/2304 [I 2022-10-27 20:30:34] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:15 [I 2022-10-27 20:30:34] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00244 [I 2022-10-27 20:30:34] ...trap/engines/engine.py.230: loss: 0.12435 [I 2022-10-27 20:30:34] ...optimizers/trijoint.py.151: total_norm: 21.976570 [I 2022-10-27 20:30:40] ...trap/engines/engine.py.224: train: epoch 86 | batch 2070/2304 [I 2022-10-27 20:30:40] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:10 [I 2022-10-27 20:30:40] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00250 [I 2022-10-27 20:30:40] ...trap/engines/engine.py.230: loss: 0.07899 [I 2022-10-27 20:30:40] ...optimizers/trijoint.py.151: total_norm: 7.841739 [I 2022-10-27 20:30:46] ...trap/engines/engine.py.224: train: epoch 86 | batch 2080/2304 [I 2022-10-27 20:30:46] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:05 [I 2022-10-27 20:30:46] ...trap/engines/engine.py.229: process: 0.58116 | load: 0.00246 [I 2022-10-27 20:30:46] ...trap/engines/engine.py.230: loss: 0.17480 [I 2022-10-27 20:30:46] ...optimizers/trijoint.py.151: total_norm: 51.673050 [I 2022-10-27 20:30:51] ...trap/engines/engine.py.224: train: epoch 86 | batch 2090/2304 [I 2022-10-27 20:30:51] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:01 [I 2022-10-27 20:30:51] ...trap/engines/engine.py.229: process: 0.57308 | load: 0.00288 [I 2022-10-27 20:30:51] ...trap/engines/engine.py.230: loss: 0.12548 [I 2022-10-27 20:30:51] ...optimizers/trijoint.py.151: total_norm: 19.118231 [I 2022-10-27 20:30:57] ...trap/engines/engine.py.224: train: epoch 86 | batch 2100/2304 [I 2022-10-27 20:30:57] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:01:54 [I 2022-10-27 20:30:57] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00256 [I 2022-10-27 20:30:57] ...trap/engines/engine.py.230: loss: 0.12893 [I 2022-10-27 20:30:57] ...optimizers/trijoint.py.151: total_norm: 15.180528 [I 2022-10-27 20:31:02] ...trap/engines/engine.py.224: train: epoch 86 | batch 2110/2304 [I 2022-10-27 20:31:02] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:49 [I 2022-10-27 20:31:02] ...trap/engines/engine.py.229: process: 0.56708 | load: 0.00278 [I 2022-10-27 20:31:02] ...trap/engines/engine.py.230: loss: 0.12362 [I 2022-10-27 20:31:02] ...optimizers/trijoint.py.151: total_norm: 12.611404 [I 2022-10-27 20:31:08] ...trap/engines/engine.py.224: train: epoch 86 | batch 2120/2304 [I 2022-10-27 20:31:08] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:44 [I 2022-10-27 20:31:08] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00246 [I 2022-10-27 20:31:08] ...trap/engines/engine.py.230: loss: 0.15266 [I 2022-10-27 20:31:08] ...optimizers/trijoint.py.151: total_norm: 10.993606 [I 2022-10-27 20:31:14] ...trap/engines/engine.py.224: train: epoch 86 | batch 2130/2304 [I 2022-10-27 20:31:14] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:38 [I 2022-10-27 20:31:14] ...trap/engines/engine.py.229: process: 0.58201 | load: 0.00516 [I 2022-10-27 20:31:14] ...trap/engines/engine.py.230: loss: 0.13627 [I 2022-10-27 20:31:14] ...optimizers/trijoint.py.151: total_norm: 42.899673 [I 2022-10-27 20:31:19] ...trap/engines/engine.py.224: train: epoch 86 | batch 2140/2304 [I 2022-10-27 20:31:19] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:32 [I 2022-10-27 20:31:19] ...trap/engines/engine.py.229: process: 0.54300 | load: 0.00247 [I 2022-10-27 20:31:19] ...trap/engines/engine.py.230: loss: 0.14732 [I 2022-10-27 20:31:19] ...optimizers/trijoint.py.151: total_norm: 32.283268 [I 2022-10-27 20:31:25] ...trap/engines/engine.py.224: train: epoch 86 | batch 2150/2304 [I 2022-10-27 20:31:25] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:26 [I 2022-10-27 20:31:25] ...trap/engines/engine.py.229: process: 0.56947 | load: 0.00245 [I 2022-10-27 20:31:25] ...trap/engines/engine.py.230: loss: 0.14018 [I 2022-10-27 20:31:25] ...optimizers/trijoint.py.151: total_norm: 17.070127 [I 2022-10-27 20:31:31] ...trap/engines/engine.py.224: train: epoch 86 | batch 2160/2304 [I 2022-10-27 20:31:31] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:20 [I 2022-10-27 20:31:31] ...trap/engines/engine.py.229: process: 0.54131 | load: 0.00246 [I 2022-10-27 20:31:31] ...trap/engines/engine.py.230: loss: 0.14274 [I 2022-10-27 20:31:31] ...optimizers/trijoint.py.151: total_norm: 32.096386 [I 2022-10-27 20:31:36] ...trap/engines/engine.py.224: train: epoch 86 | batch 2170/2304 [I 2022-10-27 20:31:36] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:15 [I 2022-10-27 20:31:36] ...trap/engines/engine.py.229: process: 0.57454 | load: 0.00243 [I 2022-10-27 20:31:36] ...trap/engines/engine.py.230: loss: 0.21727 [I 2022-10-27 20:31:36] ...optimizers/trijoint.py.151: total_norm: 21.428076 [I 2022-10-27 20:31:42] ...trap/engines/engine.py.224: train: epoch 86 | batch 2180/2304 [I 2022-10-27 20:31:42] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:09 [I 2022-10-27 20:31:42] ...trap/engines/engine.py.229: process: 0.55583 | load: 0.00276 [I 2022-10-27 20:31:42] ...trap/engines/engine.py.230: loss: 0.12157 [I 2022-10-27 20:31:42] ...optimizers/trijoint.py.151: total_norm: 17.704407 [I 2022-10-27 20:31:47] ...trap/engines/engine.py.224: train: epoch 86 | batch 2190/2304 [I 2022-10-27 20:31:47] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:03 [I 2022-10-27 20:31:47] ...trap/engines/engine.py.229: process: 0.55092 | load: 0.00245 [I 2022-10-27 20:31:47] ...trap/engines/engine.py.230: loss: 0.17024 [I 2022-10-27 20:31:47] ...optimizers/trijoint.py.151: total_norm: 13.592727 [I 2022-10-27 20:31:53] ...trap/engines/engine.py.224: train: epoch 86 | batch 2200/2304 [I 2022-10-27 20:31:53] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:00:57 [I 2022-10-27 20:31:53] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00244 [I 2022-10-27 20:31:53] ...trap/engines/engine.py.230: loss: 0.20258 [I 2022-10-27 20:31:53] ...optimizers/trijoint.py.151: total_norm: 19.416382 [I 2022-10-27 20:31:59] ...trap/engines/engine.py.224: train: epoch 86 | batch 2210/2304 [I 2022-10-27 20:31:59] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:00:52 [I 2022-10-27 20:31:59] ...trap/engines/engine.py.229: process: 0.55793 | load: 0.00241 [I 2022-10-27 20:31:59] ...trap/engines/engine.py.230: loss: 0.13670 [I 2022-10-27 20:31:59] ...optimizers/trijoint.py.151: total_norm: 23.745539 [I 2022-10-27 20:32:04] ...trap/engines/engine.py.224: train: epoch 86 | batch 2220/2304 [I 2022-10-27 20:32:04] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:00:46 [I 2022-10-27 20:32:04] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00244 [I 2022-10-27 20:32:04] ...trap/engines/engine.py.230: loss: 0.13380 [I 2022-10-27 20:32:04] ...optimizers/trijoint.py.151: total_norm: 8.443933 [I 2022-10-27 20:32:10] ...trap/engines/engine.py.224: train: epoch 86 | batch 2230/2304 [I 2022-10-27 20:32:10] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:41 [I 2022-10-27 20:32:10] ...trap/engines/engine.py.229: process: 0.57067 | load: 0.00241 [I 2022-10-27 20:32:10] ...trap/engines/engine.py.230: loss: 0.14603 [I 2022-10-27 20:32:10] ...optimizers/trijoint.py.151: total_norm: 15.197745 [I 2022-10-27 20:32:15] ...trap/engines/engine.py.224: train: epoch 86 | batch 2240/2304 [I 2022-10-27 20:32:15] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:35 [I 2022-10-27 20:32:15] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00249 [I 2022-10-27 20:32:15] ...trap/engines/engine.py.230: loss: 0.17617 [I 2022-10-27 20:32:15] ...optimizers/trijoint.py.151: total_norm: 70.927246 [I 2022-10-27 20:32:21] ...trap/engines/engine.py.224: train: epoch 86 | batch 2250/2304 [I 2022-10-27 20:32:21] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:30 [I 2022-10-27 20:32:21] ...trap/engines/engine.py.229: process: 0.54927 | load: 0.00255 [I 2022-10-27 20:32:21] ...trap/engines/engine.py.230: loss: 0.12113 [I 2022-10-27 20:32:21] ...optimizers/trijoint.py.151: total_norm: 12.598200 [I 2022-10-27 20:32:27] ...trap/engines/engine.py.224: train: epoch 86 | batch 2260/2304 [I 2022-10-27 20:32:27] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:24 [I 2022-10-27 20:32:27] ...trap/engines/engine.py.229: process: 0.55267 | load: 0.00281 [I 2022-10-27 20:32:27] ...trap/engines/engine.py.230: loss: 0.12360 [I 2022-10-27 20:32:27] ...optimizers/trijoint.py.151: total_norm: 24.280973 [I 2022-10-27 20:32:32] ...trap/engines/engine.py.224: train: epoch 86 | batch 2270/2304 [I 2022-10-27 20:32:32] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:18 [I 2022-10-27 20:32:32] ...trap/engines/engine.py.229: process: 0.54290 | load: 0.00242 [I 2022-10-27 20:32:32] ...trap/engines/engine.py.230: loss: 0.14623 [I 2022-10-27 20:32:32] ...optimizers/trijoint.py.151: total_norm: 20.195105 [I 2022-10-27 20:32:38] ...trap/engines/engine.py.224: train: epoch 86 | batch 2280/2304 [I 2022-10-27 20:32:38] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:13 [I 2022-10-27 20:32:38] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00248 [I 2022-10-27 20:32:38] ...trap/engines/engine.py.230: loss: 0.09017 [I 2022-10-27 20:32:38] ...optimizers/trijoint.py.151: total_norm: 15.213221 [I 2022-10-27 20:32:43] ...trap/engines/engine.py.224: train: epoch 86 | batch 2290/2304 [I 2022-10-27 20:32:43] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:07 [I 2022-10-27 20:32:43] ...trap/engines/engine.py.229: process: 0.55703 | load: 0.00246 [I 2022-10-27 20:32:43] ...trap/engines/engine.py.230: loss: 0.13289 [I 2022-10-27 20:32:43] ...optimizers/trijoint.py.151: total_norm: 11.690618 [I 2022-10-27 20:32:49] ...trap/engines/engine.py.224: train: epoch 86 | batch 2300/2304 [I 2022-10-27 20:32:49] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:02 [I 2022-10-27 20:32:49] ...trap/engines/engine.py.229: process: 0.54653 | load: 0.00015 [I 2022-10-27 20:32:49] ...trap/engines/engine.py.230: loss: 0.15374 [I 2022-10-27 20:32:49] ...optimizers/trijoint.py.151: total_norm: 15.436792 [I 2022-10-27 20:32:51] ...trap/engines/engine.py.224: train: epoch 86 | batch 2304/2304 [I 2022-10-27 20:32:51] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:00 [I 2022-10-27 20:32:51] ...trap/engines/engine.py.229: process: 0.55679 | load: 0.00016 [I 2022-10-27 20:32:51] ...trap/engines/engine.py.230: loss: 0.10594 [I 2022-10-27 20:32:51] ...optimizers/trijoint.py.151: total_norm: 9.151864 [S 2022-10-27 20:32:52] ...trap/engines/engine.py.129: train_epoch.epoch: 86 [S 2022-10-27 20:32:52] ...trap/engines/engine.py.129: train_epoch.loss: 0.13319250632586552 [I 2022-10-27 20:32:52] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 20:32:54] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 86 [I 2022-10-27 20:32:54] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 20:32:56] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 20:32:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 20:32:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 20:32:58] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 20:32:59] ...trap/engines/engine.py.321: eval: epoch 86 | batch 0/510 [I 2022-10-27 20:32:59] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:43:42 [I 2022-10-27 20:32:59] ...trap/engines/engine.py.326: process: 5.14307 | load: 3.55253 [I 2022-10-27 20:33:02] ...trap/engines/engine.py.321: eval: epoch 86 | batch 10/510 [I 2022-10-27 20:33:02] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:09 [I 2022-10-27 20:33:02] ...trap/engines/engine.py.326: process: 0.35133 | load: 0.00017 [I 2022-10-27 20:33:05] ...trap/engines/engine.py.321: eval: epoch 86 | batch 20/510 [I 2022-10-27 20:33:05] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:11 [I 2022-10-27 20:33:05] ...trap/engines/engine.py.326: process: 0.33231 | load: 0.00023 [I 2022-10-27 20:33:09] ...trap/engines/engine.py.321: eval: epoch 86 | batch 30/510 [I 2022-10-27 20:33:09] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:41 [I 2022-10-27 20:33:09] ...trap/engines/engine.py.326: process: 0.32876 | load: 0.00018 [I 2022-10-27 20:33:12] ...trap/engines/engine.py.321: eval: epoch 86 | batch 40/510 [I 2022-10-27 20:33:12] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:39 [I 2022-10-27 20:33:12] ...trap/engines/engine.py.326: process: 0.36110 | load: 0.00017 [I 2022-10-27 20:33:15] ...trap/engines/engine.py.321: eval: epoch 86 | batch 50/510 [I 2022-10-27 20:33:15] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:34 [I 2022-10-27 20:33:15] ...trap/engines/engine.py.326: process: 0.35073 | load: 0.00016 [I 2022-10-27 20:33:19] ...trap/engines/engine.py.321: eval: epoch 86 | batch 60/510 [I 2022-10-27 20:33:19] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:30 [I 2022-10-27 20:33:19] ...trap/engines/engine.py.326: process: 0.33045 | load: 0.00017 [I 2022-10-27 20:33:22] ...trap/engines/engine.py.321: eval: epoch 86 | batch 70/510 [I 2022-10-27 20:33:22] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:27 [I 2022-10-27 20:33:22] ...trap/engines/engine.py.326: process: 0.34295 | load: 0.00016 [I 2022-10-27 20:33:26] ...trap/engines/engine.py.321: eval: epoch 86 | batch 80/510 [I 2022-10-27 20:33:26] ...trap/engines/engine.py.325: elapsed: 0:00:31 | left: 0:02:21 [I 2022-10-27 20:33:26] ...trap/engines/engine.py.326: process: 0.32470 | load: 0.00017 [I 2022-10-27 20:33:29] ...trap/engines/engine.py.321: eval: epoch 86 | batch 90/510 [I 2022-10-27 20:33:29] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-27 20:33:29] ...trap/engines/engine.py.326: process: 0.33148 | load: 0.00016 [I 2022-10-27 20:33:32] ...trap/engines/engine.py.321: eval: epoch 86 | batch 100/510 [I 2022-10-27 20:33:32] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-27 20:33:32] ...trap/engines/engine.py.326: process: 0.28821 | load: 0.00017 [I 2022-10-27 20:33:35] ...trap/engines/engine.py.321: eval: epoch 86 | batch 110/510 [I 2022-10-27 20:33:35] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-27 20:33:35] ...trap/engines/engine.py.326: process: 0.29278 | load: 0.00022 [I 2022-10-27 20:33:38] ...trap/engines/engine.py.321: eval: epoch 86 | batch 120/510 [I 2022-10-27 20:33:38] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 20:33:38] ...trap/engines/engine.py.326: process: 0.28655 | load: 0.00015 [I 2022-10-27 20:33:41] ...trap/engines/engine.py.321: eval: epoch 86 | batch 130/510 [I 2022-10-27 20:33:41] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:48 [I 2022-10-27 20:33:41] ...trap/engines/engine.py.326: process: 0.28702 | load: 0.00024 [I 2022-10-27 20:33:44] ...trap/engines/engine.py.321: eval: epoch 86 | batch 140/510 [I 2022-10-27 20:33:44] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-27 20:33:44] ...trap/engines/engine.py.326: process: 0.29125 | load: 0.00014 [I 2022-10-27 20:33:47] ...trap/engines/engine.py.321: eval: epoch 86 | batch 150/510 [I 2022-10-27 20:33:47] ...trap/engines/engine.py.325: elapsed: 0:00:52 | left: 0:01:43 [I 2022-10-27 20:33:47] ...trap/engines/engine.py.326: process: 0.29040 | load: 0.00023 [I 2022-10-27 20:33:49] ...trap/engines/engine.py.321: eval: epoch 86 | batch 160/510 [I 2022-10-27 20:33:49] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 20:33:49] ...trap/engines/engine.py.326: process: 0.28859 | load: 0.00015 [I 2022-10-27 20:33:52] ...trap/engines/engine.py.321: eval: epoch 86 | batch 170/510 [I 2022-10-27 20:33:52] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 20:33:52] ...trap/engines/engine.py.326: process: 0.29015 | load: 0.00014 [I 2022-10-27 20:33:55] ...trap/engines/engine.py.321: eval: epoch 86 | batch 180/510 [I 2022-10-27 20:33:55] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:34 [I 2022-10-27 20:33:55] ...trap/engines/engine.py.326: process: 0.28229 | load: 0.00017 [I 2022-10-27 20:33:58] ...trap/engines/engine.py.321: eval: epoch 86 | batch 190/510 [I 2022-10-27 20:33:58] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 20:33:58] ...trap/engines/engine.py.326: process: 0.28708 | load: 0.00025 [I 2022-10-27 20:34:01] ...trap/engines/engine.py.321: eval: epoch 86 | batch 200/510 [I 2022-10-27 20:34:01] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 20:34:01] ...trap/engines/engine.py.326: process: 0.28596 | load: 0.00019 [I 2022-10-27 20:34:04] ...trap/engines/engine.py.321: eval: epoch 86 | batch 210/510 [I 2022-10-27 20:34:04] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 20:34:04] ...trap/engines/engine.py.326: process: 0.29014 | load: 0.00023 [I 2022-10-27 20:34:07] ...trap/engines/engine.py.321: eval: epoch 86 | batch 220/510 [I 2022-10-27 20:34:07] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 20:34:07] ...trap/engines/engine.py.326: process: 0.28350 | load: 0.00014 [I 2022-10-27 20:34:10] ...trap/engines/engine.py.321: eval: epoch 86 | batch 230/510 [I 2022-10-27 20:34:10] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 20:34:10] ...trap/engines/engine.py.326: process: 0.28782 | load: 0.00015 [I 2022-10-27 20:34:12] ...trap/engines/engine.py.321: eval: epoch 86 | batch 240/510 [I 2022-10-27 20:34:12] ...trap/engines/engine.py.325: elapsed: 0:01:18 | left: 0:01:17 [I 2022-10-27 20:34:12] ...trap/engines/engine.py.326: process: 0.28828 | load: 0.00015 [I 2022-10-27 20:34:15] ...trap/engines/engine.py.321: eval: epoch 86 | batch 250/510 [I 2022-10-27 20:34:15] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 20:34:15] ...trap/engines/engine.py.326: process: 0.28734 | load: 0.00015 [I 2022-10-27 20:34:18] ...trap/engines/engine.py.321: eval: epoch 86 | batch 260/510 [I 2022-10-27 20:34:18] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:11 [I 2022-10-27 20:34:18] ...trap/engines/engine.py.326: process: 0.28735 | load: 0.00016 [I 2022-10-27 20:34:21] ...trap/engines/engine.py.321: eval: epoch 86 | batch 270/510 [I 2022-10-27 20:34:21] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 20:34:21] ...trap/engines/engine.py.326: process: 0.28872 | load: 0.00016 [I 2022-10-27 20:34:24] ...trap/engines/engine.py.321: eval: epoch 86 | batch 280/510 [I 2022-10-27 20:34:24] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 20:34:24] ...trap/engines/engine.py.326: process: 0.28709 | load: 0.00015 [I 2022-10-27 20:34:27] ...trap/engines/engine.py.321: eval: epoch 86 | batch 290/510 [I 2022-10-27 20:34:27] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 20:34:27] ...trap/engines/engine.py.326: process: 0.28826 | load: 0.00016 [I 2022-10-27 20:34:30] ...trap/engines/engine.py.321: eval: epoch 86 | batch 300/510 [I 2022-10-27 20:34:30] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:00:59 [I 2022-10-27 20:34:30] ...trap/engines/engine.py.326: process: 0.28135 | load: 0.00014 [I 2022-10-27 20:34:32] ...trap/engines/engine.py.321: eval: epoch 86 | batch 310/510 [I 2022-10-27 20:34:32] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:00:57 [I 2022-10-27 20:34:32] ...trap/engines/engine.py.326: process: 0.28724 | load: 0.00015 [I 2022-10-27 20:34:35] ...trap/engines/engine.py.321: eval: epoch 86 | batch 320/510 [I 2022-10-27 20:34:35] ...trap/engines/engine.py.325: elapsed: 0:01:41 | left: 0:00:54 [I 2022-10-27 20:34:35] ...trap/engines/engine.py.326: process: 0.29081 | load: 0.00015 [I 2022-10-27 20:34:38] ...trap/engines/engine.py.321: eval: epoch 86 | batch 330/510 [I 2022-10-27 20:34:38] ...trap/engines/engine.py.325: elapsed: 0:01:44 | left: 0:00:51 [I 2022-10-27 20:34:38] ...trap/engines/engine.py.326: process: 0.28971 | load: 0.00015 [I 2022-10-27 20:34:41] ...trap/engines/engine.py.321: eval: epoch 86 | batch 340/510 [I 2022-10-27 20:34:41] ...trap/engines/engine.py.325: elapsed: 0:01:47 | left: 0:00:48 [I 2022-10-27 20:34:41] ...trap/engines/engine.py.326: process: 0.28743 | load: 0.00020 [I 2022-10-27 20:34:44] ...trap/engines/engine.py.321: eval: epoch 86 | batch 350/510 [I 2022-10-27 20:34:44] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:45 [I 2022-10-27 20:34:44] ...trap/engines/engine.py.326: process: 0.28112 | load: 0.00016 [I 2022-10-27 20:34:47] ...trap/engines/engine.py.321: eval: epoch 86 | batch 360/510 [I 2022-10-27 20:34:47] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 20:34:47] ...trap/engines/engine.py.326: process: 0.28232 | load: 0.00019 [I 2022-10-27 20:34:50] ...trap/engines/engine.py.321: eval: epoch 86 | batch 370/510 [I 2022-10-27 20:34:50] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 20:34:50] ...trap/engines/engine.py.326: process: 0.28930 | load: 0.00015 [I 2022-10-27 20:34:53] ...trap/engines/engine.py.321: eval: epoch 86 | batch 380/510 [I 2022-10-27 20:34:53] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 20:34:53] ...trap/engines/engine.py.326: process: 0.28828 | load: 0.00015 [I 2022-10-27 20:34:55] ...trap/engines/engine.py.321: eval: epoch 86 | batch 390/510 [I 2022-10-27 20:34:55] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:34 [I 2022-10-27 20:34:55] ...trap/engines/engine.py.326: process: 0.28147 | load: 0.00015 [I 2022-10-27 20:34:58] ...trap/engines/engine.py.321: eval: epoch 86 | batch 400/510 [I 2022-10-27 20:34:58] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:31 [I 2022-10-27 20:34:58] ...trap/engines/engine.py.326: process: 0.29046 | load: 0.00019 [I 2022-10-27 20:35:01] ...trap/engines/engine.py.321: eval: epoch 86 | batch 410/510 [I 2022-10-27 20:35:01] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:28 [I 2022-10-27 20:35:01] ...trap/engines/engine.py.326: process: 0.28586 | load: 0.00014 [I 2022-10-27 20:35:04] ...trap/engines/engine.py.321: eval: epoch 86 | batch 420/510 [I 2022-10-27 20:35:04] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:26 [I 2022-10-27 20:35:04] ...trap/engines/engine.py.326: process: 0.29069 | load: 0.00015 [I 2022-10-27 20:35:07] ...trap/engines/engine.py.321: eval: epoch 86 | batch 430/510 [I 2022-10-27 20:35:07] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 20:35:07] ...trap/engines/engine.py.326: process: 0.28992 | load: 0.00015 [I 2022-10-27 20:35:10] ...trap/engines/engine.py.321: eval: epoch 86 | batch 440/510 [I 2022-10-27 20:35:10] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 20:35:10] ...trap/engines/engine.py.326: process: 0.28933 | load: 0.00015 [I 2022-10-27 20:35:13] ...trap/engines/engine.py.321: eval: epoch 86 | batch 450/510 [I 2022-10-27 20:35:13] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 20:35:13] ...trap/engines/engine.py.326: process: 0.28936 | load: 0.00016 [I 2022-10-27 20:35:16] ...trap/engines/engine.py.321: eval: epoch 86 | batch 460/510 [I 2022-10-27 20:35:16] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 20:35:16] ...trap/engines/engine.py.326: process: 0.28355 | load: 0.00017 [I 2022-10-27 20:35:18] ...trap/engines/engine.py.321: eval: epoch 86 | batch 470/510 [I 2022-10-27 20:35:18] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:11 [I 2022-10-27 20:35:18] ...trap/engines/engine.py.326: process: 0.28451 | load: 0.00016 [I 2022-10-27 20:35:21] ...trap/engines/engine.py.321: eval: epoch 86 | batch 480/510 [I 2022-10-27 20:35:21] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:08 [I 2022-10-27 20:35:21] ...trap/engines/engine.py.326: process: 0.28643 | load: 0.00017 [I 2022-10-27 20:35:24] ...trap/engines/engine.py.321: eval: epoch 86 | batch 490/510 [I 2022-10-27 20:35:24] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:05 [I 2022-10-27 20:35:24] ...trap/engines/engine.py.326: process: 0.28892 | load: 0.00020 [I 2022-10-27 20:35:27] ...trap/engines/engine.py.321: eval: epoch 86 | batch 500/510 [I 2022-10-27 20:35:27] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-27 20:35:27] ...trap/engines/engine.py.326: process: 0.28846 | load: 0.00016 [I 2022-10-27 20:35:30] ...trap/engines/engine.py.321: eval: epoch 86 | batch 510/510 [I 2022-10-27 20:35:30] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 20:35:30] ...trap/engines/engine.py.326: process: 0.28770 | load: 0.00011 [S 2022-10-27 20:35:31] ...trap/engines/engine.py.132: eval_epoch.epoch: 86 [I 2022-10-27 20:35:31] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.730100 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906400 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934400 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.723300 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904200 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934000 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017196 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008958 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009287 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015453 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008908 [S 2022-10-27 20:35:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008741 [I 2022-10-27 20:35:37] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 20:35:37] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 20:35:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 20:35:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 20:35:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 20:35:41] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 20:35:42] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 20:35:48] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 20:35:48] ...trap/engines/engine.py.163: Training model on trainset for epoch 87 [I 2022-10-27 20:35:48] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 20:35:51] ...trap/engines/engine.py.224: train: epoch 87 | batch 0/2304 [I 2022-10-27 20:35:51] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:15 [I 2022-10-27 20:35:51] ...trap/engines/engine.py.229: process: 2.71518 | load: 2.06434 [I 2022-10-27 20:35:51] ...trap/engines/engine.py.230: loss: 0.15496 [I 2022-10-27 20:35:51] ...optimizers/trijoint.py.151: total_norm: 15.141533 [I 2022-10-27 20:35:56] ...trap/engines/engine.py.224: train: epoch 87 | batch 10/2304 [I 2022-10-27 20:35:56] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:12 [I 2022-10-27 20:35:56] ...trap/engines/engine.py.229: process: 0.56575 | load: 0.00482 [I 2022-10-27 20:35:56] ...trap/engines/engine.py.230: loss: 0.15843 [I 2022-10-27 20:35:56] ...optimizers/trijoint.py.151: total_norm: 32.283432 [I 2022-10-27 20:36:02] ...trap/engines/engine.py.224: train: epoch 87 | batch 20/2304 [I 2022-10-27 20:36:02] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:02 [I 2022-10-27 20:36:02] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00337 [I 2022-10-27 20:36:02] ...trap/engines/engine.py.230: loss: 0.10124 [I 2022-10-27 20:36:02] ...optimizers/trijoint.py.151: total_norm: 7.110264 [I 2022-10-27 20:36:08] ...trap/engines/engine.py.224: train: epoch 87 | batch 30/2304 [I 2022-10-27 20:36:08] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:05 [I 2022-10-27 20:36:08] ...trap/engines/engine.py.229: process: 0.54615 | load: 0.00374 [I 2022-10-27 20:36:08] ...trap/engines/engine.py.230: loss: 0.10938 [I 2022-10-27 20:36:08] ...optimizers/trijoint.py.151: total_norm: 32.061684 [I 2022-10-27 20:36:13] ...trap/engines/engine.py.224: train: epoch 87 | batch 40/2304 [I 2022-10-27 20:36:13] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:55 [I 2022-10-27 20:36:13] ...trap/engines/engine.py.229: process: 0.54852 | load: 0.00305 [I 2022-10-27 20:36:13] ...trap/engines/engine.py.230: loss: 0.12020 [I 2022-10-27 20:36:13] ...optimizers/trijoint.py.151: total_norm: 25.004423 [I 2022-10-27 20:36:19] ...trap/engines/engine.py.224: train: epoch 87 | batch 50/2304 [I 2022-10-27 20:36:19] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:03 [I 2022-10-27 20:36:19] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00288 [I 2022-10-27 20:36:19] ...trap/engines/engine.py.230: loss: 0.10696 [I 2022-10-27 20:36:19] ...optimizers/trijoint.py.151: total_norm: 28.258520 [I 2022-10-27 20:36:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 60/2304 [I 2022-10-27 20:36:24] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:17 [I 2022-10-27 20:36:24] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00270 [I 2022-10-27 20:36:24] ...trap/engines/engine.py.230: loss: 0.16110 [I 2022-10-27 20:36:24] ...optimizers/trijoint.py.151: total_norm: 16.941057 [I 2022-10-27 20:36:30] ...trap/engines/engine.py.224: train: epoch 87 | batch 70/2304 [I 2022-10-27 20:36:30] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:54 [I 2022-10-27 20:36:30] ...trap/engines/engine.py.229: process: 0.55911 | load: 0.00277 [I 2022-10-27 20:36:30] ...trap/engines/engine.py.230: loss: 0.17853 [I 2022-10-27 20:36:30] ...optimizers/trijoint.py.151: total_norm: 22.908249 [I 2022-10-27 20:36:36] ...trap/engines/engine.py.224: train: epoch 87 | batch 80/2304 [I 2022-10-27 20:36:36] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:53 [I 2022-10-27 20:36:36] ...trap/engines/engine.py.229: process: 0.54450 | load: 0.00263 [I 2022-10-27 20:36:36] ...trap/engines/engine.py.230: loss: 0.15590 [I 2022-10-27 20:36:36] ...optimizers/trijoint.py.151: total_norm: 34.714474 [I 2022-10-27 20:36:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 90/2304 [I 2022-10-27 20:36:41] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:38 [I 2022-10-27 20:36:41] ...trap/engines/engine.py.229: process: 0.56138 | load: 0.00276 [I 2022-10-27 20:36:41] ...trap/engines/engine.py.230: loss: 0.11235 [I 2022-10-27 20:36:41] ...optimizers/trijoint.py.151: total_norm: 21.592913 [I 2022-10-27 20:36:47] ...trap/engines/engine.py.224: train: epoch 87 | batch 100/2304 [I 2022-10-27 20:36:47] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:30 [I 2022-10-27 20:36:47] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00251 [I 2022-10-27 20:36:47] ...trap/engines/engine.py.230: loss: 0.11527 [I 2022-10-27 20:36:47] ...optimizers/trijoint.py.151: total_norm: 32.975628 [I 2022-10-27 20:36:53] ...trap/engines/engine.py.224: train: epoch 87 | batch 110/2304 [I 2022-10-27 20:36:53] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:29 [I 2022-10-27 20:36:53] ...trap/engines/engine.py.229: process: 0.55499 | load: 0.00276 [I 2022-10-27 20:36:53] ...trap/engines/engine.py.230: loss: 0.11847 [I 2022-10-27 20:36:53] ...optimizers/trijoint.py.151: total_norm: 10.829119 [I 2022-10-27 20:36:58] ...trap/engines/engine.py.224: train: epoch 87 | batch 120/2304 [I 2022-10-27 20:36:58] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:27 [I 2022-10-27 20:36:58] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00251 [I 2022-10-27 20:36:58] ...trap/engines/engine.py.230: loss: 0.13349 [I 2022-10-27 20:36:58] ...optimizers/trijoint.py.151: total_norm: 18.417473 [I 2022-10-27 20:37:04] ...trap/engines/engine.py.224: train: epoch 87 | batch 130/2304 [I 2022-10-27 20:37:04] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:14 [I 2022-10-27 20:37:04] ...trap/engines/engine.py.229: process: 0.56497 | load: 0.00273 [I 2022-10-27 20:37:04] ...trap/engines/engine.py.230: loss: 0.10109 [I 2022-10-27 20:37:04] ...optimizers/trijoint.py.151: total_norm: 11.435970 [I 2022-10-27 20:37:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 140/2304 [I 2022-10-27 20:37:09] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:06 [I 2022-10-27 20:37:09] ...trap/engines/engine.py.229: process: 0.56037 | load: 0.00256 [I 2022-10-27 20:37:09] ...trap/engines/engine.py.230: loss: 0.14799 [I 2022-10-27 20:37:09] ...optimizers/trijoint.py.151: total_norm: 10.032014 [I 2022-10-27 20:37:15] ...trap/engines/engine.py.224: train: epoch 87 | batch 150/2304 [I 2022-10-27 20:37:15] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:11 [I 2022-10-27 20:37:15] ...trap/engines/engine.py.229: process: 0.56469 | load: 0.00262 [I 2022-10-27 20:37:15] ...trap/engines/engine.py.230: loss: 0.14643 [I 2022-10-27 20:37:15] ...optimizers/trijoint.py.151: total_norm: 15.775983 [I 2022-10-27 20:37:21] ...trap/engines/engine.py.224: train: epoch 87 | batch 160/2304 [I 2022-10-27 20:37:21] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:59 [I 2022-10-27 20:37:21] ...trap/engines/engine.py.229: process: 0.55768 | load: 0.00257 [I 2022-10-27 20:37:21] ...trap/engines/engine.py.230: loss: 0.19918 [I 2022-10-27 20:37:21] ...optimizers/trijoint.py.151: total_norm: 21.624249 [I 2022-10-27 20:37:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 170/2304 [I 2022-10-27 20:37:26] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:48 [I 2022-10-27 20:37:26] ...trap/engines/engine.py.229: process: 0.55040 | load: 0.00246 [I 2022-10-27 20:37:26] ...trap/engines/engine.py.230: loss: 0.11004 [I 2022-10-27 20:37:26] ...optimizers/trijoint.py.151: total_norm: 14.500112 [I 2022-10-27 20:37:32] ...trap/engines/engine.py.224: train: epoch 87 | batch 180/2304 [I 2022-10-27 20:37:32] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:45 [I 2022-10-27 20:37:32] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00251 [I 2022-10-27 20:37:32] ...trap/engines/engine.py.230: loss: 0.10264 [I 2022-10-27 20:37:32] ...optimizers/trijoint.py.151: total_norm: 16.696339 [I 2022-10-27 20:37:37] ...trap/engines/engine.py.224: train: epoch 87 | batch 190/2304 [I 2022-10-27 20:37:37] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:31 [I 2022-10-27 20:37:37] ...trap/engines/engine.py.229: process: 0.54699 | load: 0.00246 [I 2022-10-27 20:37:37] ...trap/engines/engine.py.230: loss: 0.11422 [I 2022-10-27 20:37:37] ...optimizers/trijoint.py.151: total_norm: 27.561264 [I 2022-10-27 20:37:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 200/2304 [I 2022-10-27 20:37:43] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:28 [I 2022-10-27 20:37:43] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00255 [I 2022-10-27 20:37:43] ...trap/engines/engine.py.230: loss: 0.14291 [I 2022-10-27 20:37:43] ...optimizers/trijoint.py.151: total_norm: 15.651053 [I 2022-10-27 20:37:49] ...trap/engines/engine.py.224: train: epoch 87 | batch 210/2304 [I 2022-10-27 20:37:49] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:25 [I 2022-10-27 20:37:49] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00269 [I 2022-10-27 20:37:49] ...trap/engines/engine.py.230: loss: 0.15190 [I 2022-10-27 20:37:49] ...optimizers/trijoint.py.151: total_norm: 20.955631 [I 2022-10-27 20:37:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 220/2304 [I 2022-10-27 20:37:54] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:22 [I 2022-10-27 20:37:54] ...trap/engines/engine.py.229: process: 0.55233 | load: 0.00244 [I 2022-10-27 20:37:54] ...trap/engines/engine.py.230: loss: 0.10680 [I 2022-10-27 20:37:54] ...optimizers/trijoint.py.151: total_norm: 9.227523 [I 2022-10-27 20:38:00] ...trap/engines/engine.py.224: train: epoch 87 | batch 230/2304 [I 2022-10-27 20:38:00] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:26 [I 2022-10-27 20:38:00] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00253 [I 2022-10-27 20:38:00] ...trap/engines/engine.py.230: loss: 0.12842 [I 2022-10-27 20:38:00] ...optimizers/trijoint.py.151: total_norm: 14.976821 [I 2022-10-27 20:38:05] ...trap/engines/engine.py.224: train: epoch 87 | batch 240/2304 [I 2022-10-27 20:38:05] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:12 [I 2022-10-27 20:38:05] ...trap/engines/engine.py.229: process: 0.56655 | load: 0.00265 [I 2022-10-27 20:38:05] ...trap/engines/engine.py.230: loss: 0.14305 [I 2022-10-27 20:38:05] ...optimizers/trijoint.py.151: total_norm: 15.734221 [I 2022-10-27 20:38:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 250/2304 [I 2022-10-27 20:38:11] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:06 [I 2022-10-27 20:38:11] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00253 [I 2022-10-27 20:38:11] ...trap/engines/engine.py.230: loss: 0.11719 [I 2022-10-27 20:38:11] ...optimizers/trijoint.py.151: total_norm: 9.403567 [I 2022-10-27 20:38:17] ...trap/engines/engine.py.224: train: epoch 87 | batch 260/2304 [I 2022-10-27 20:38:17] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:54 [I 2022-10-27 20:38:17] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00257 [I 2022-10-27 20:38:17] ...trap/engines/engine.py.230: loss: 0.14876 [I 2022-10-27 20:38:17] ...optimizers/trijoint.py.151: total_norm: 23.829081 [I 2022-10-27 20:38:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 270/2304 [I 2022-10-27 20:38:22] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:19:41 [I 2022-10-27 20:38:22] ...trap/engines/engine.py.229: process: 0.57231 | load: 0.01097 [I 2022-10-27 20:38:22] ...trap/engines/engine.py.230: loss: 0.12976 [I 2022-10-27 20:38:22] ...optimizers/trijoint.py.151: total_norm: 12.663220 [I 2022-10-27 20:38:28] ...trap/engines/engine.py.224: train: epoch 87 | batch 280/2304 [I 2022-10-27 20:38:28] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:50 [I 2022-10-27 20:38:28] ...trap/engines/engine.py.229: process: 0.55230 | load: 0.00247 [I 2022-10-27 20:38:28] ...trap/engines/engine.py.230: loss: 0.09472 [I 2022-10-27 20:38:28] ...optimizers/trijoint.py.151: total_norm: 42.913124 [I 2022-10-27 20:38:34] ...trap/engines/engine.py.224: train: epoch 87 | batch 290/2304 [I 2022-10-27 20:38:34] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:43 [I 2022-10-27 20:38:34] ...trap/engines/engine.py.229: process: 0.54947 | load: 0.00251 [I 2022-10-27 20:38:34] ...trap/engines/engine.py.230: loss: 0.09838 [I 2022-10-27 20:38:34] ...optimizers/trijoint.py.151: total_norm: 9.562440 [I 2022-10-27 20:38:39] ...trap/engines/engine.py.224: train: epoch 87 | batch 300/2304 [I 2022-10-27 20:38:39] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:39 [I 2022-10-27 20:38:39] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00248 [I 2022-10-27 20:38:39] ...trap/engines/engine.py.230: loss: 0.12172 [I 2022-10-27 20:38:39] ...optimizers/trijoint.py.151: total_norm: 13.956182 [I 2022-10-27 20:38:45] ...trap/engines/engine.py.224: train: epoch 87 | batch 310/2304 [I 2022-10-27 20:38:45] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:32 [I 2022-10-27 20:38:45] ...trap/engines/engine.py.229: process: 0.56737 | load: 0.00241 [I 2022-10-27 20:38:45] ...trap/engines/engine.py.230: loss: 0.17966 [I 2022-10-27 20:38:45] ...optimizers/trijoint.py.151: total_norm: 19.248749 [I 2022-10-27 20:38:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 320/2304 [I 2022-10-27 20:38:50] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:46 [I 2022-10-27 20:38:50] ...trap/engines/engine.py.229: process: 0.58532 | load: 0.00252 [I 2022-10-27 20:38:50] ...trap/engines/engine.py.230: loss: 0.13268 [I 2022-10-27 20:38:50] ...optimizers/trijoint.py.151: total_norm: 13.404936 [I 2022-10-27 20:38:56] ...trap/engines/engine.py.224: train: epoch 87 | batch 330/2304 [I 2022-10-27 20:38:56] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:28 [I 2022-10-27 20:38:56] ...trap/engines/engine.py.229: process: 0.56233 | load: 0.00257 [I 2022-10-27 20:38:56] ...trap/engines/engine.py.230: loss: 0.09848 [I 2022-10-27 20:38:56] ...optimizers/trijoint.py.151: total_norm: 17.516312 [I 2022-10-27 20:39:02] ...trap/engines/engine.py.224: train: epoch 87 | batch 340/2304 [I 2022-10-27 20:39:02] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:57 [I 2022-10-27 20:39:02] ...trap/engines/engine.py.229: process: 0.56442 | load: 0.00254 [I 2022-10-27 20:39:02] ...trap/engines/engine.py.230: loss: 0.10680 [I 2022-10-27 20:39:02] ...optimizers/trijoint.py.151: total_norm: 23.144098 [I 2022-10-27 20:39:08] ...trap/engines/engine.py.224: train: epoch 87 | batch 350/2304 [I 2022-10-27 20:39:08] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:24 [I 2022-10-27 20:39:08] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00247 [I 2022-10-27 20:39:08] ...trap/engines/engine.py.230: loss: 0.13061 [I 2022-10-27 20:39:08] ...optimizers/trijoint.py.151: total_norm: 13.467902 [I 2022-10-27 20:39:13] ...trap/engines/engine.py.224: train: epoch 87 | batch 360/2304 [I 2022-10-27 20:39:13] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:18:10 [I 2022-10-27 20:39:13] ...trap/engines/engine.py.229: process: 0.55525 | load: 0.00243 [I 2022-10-27 20:39:13] ...trap/engines/engine.py.230: loss: 0.11561 [I 2022-10-27 20:39:13] ...optimizers/trijoint.py.151: total_norm: 16.165697 [I 2022-10-27 20:39:19] ...trap/engines/engine.py.224: train: epoch 87 | batch 370/2304 [I 2022-10-27 20:39:19] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:06 [I 2022-10-27 20:39:19] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00250 [I 2022-10-27 20:39:19] ...trap/engines/engine.py.230: loss: 0.12788 [I 2022-10-27 20:39:19] ...optimizers/trijoint.py.151: total_norm: 18.660236 [I 2022-10-27 20:39:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 380/2304 [I 2022-10-27 20:39:24] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:18:06 [I 2022-10-27 20:39:24] ...trap/engines/engine.py.229: process: 0.56513 | load: 0.00256 [I 2022-10-27 20:39:24] ...trap/engines/engine.py.230: loss: 0.14075 [I 2022-10-27 20:39:24] ...optimizers/trijoint.py.151: total_norm: 14.828626 [I 2022-10-27 20:39:30] ...trap/engines/engine.py.224: train: epoch 87 | batch 390/2304 [I 2022-10-27 20:39:30] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:46 [I 2022-10-27 20:39:30] ...trap/engines/engine.py.229: process: 0.56449 | load: 0.00247 [I 2022-10-27 20:39:30] ...trap/engines/engine.py.230: loss: 0.11539 [I 2022-10-27 20:39:30] ...optimizers/trijoint.py.151: total_norm: 12.593266 [I 2022-10-27 20:39:36] ...trap/engines/engine.py.224: train: epoch 87 | batch 400/2304 [I 2022-10-27 20:39:36] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:18:15 [I 2022-10-27 20:39:36] ...trap/engines/engine.py.229: process: 0.64657 | load: 0.00245 [I 2022-10-27 20:39:36] ...trap/engines/engine.py.230: loss: 0.13623 [I 2022-10-27 20:39:36] ...optimizers/trijoint.py.151: total_norm: 14.902979 [I 2022-10-27 20:39:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 410/2304 [I 2022-10-27 20:39:41] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:17:51 [I 2022-10-27 20:39:41] ...trap/engines/engine.py.229: process: 0.56951 | load: 0.00243 [I 2022-10-27 20:39:41] ...trap/engines/engine.py.230: loss: 0.13023 [I 2022-10-27 20:39:41] ...optimizers/trijoint.py.151: total_norm: 15.622231 [I 2022-10-27 20:39:47] ...trap/engines/engine.py.224: train: epoch 87 | batch 420/2304 [I 2022-10-27 20:39:47] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:37 [I 2022-10-27 20:39:47] ...trap/engines/engine.py.229: process: 0.57184 | load: 0.00250 [I 2022-10-27 20:39:47] ...trap/engines/engine.py.230: loss: 0.20777 [I 2022-10-27 20:39:47] ...optimizers/trijoint.py.151: total_norm: 123.728500 [I 2022-10-27 20:39:53] ...trap/engines/engine.py.224: train: epoch 87 | batch 430/2304 [I 2022-10-27 20:39:53] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:21 [I 2022-10-27 20:39:53] ...trap/engines/engine.py.229: process: 0.54846 | load: 0.00245 [I 2022-10-27 20:39:53] ...trap/engines/engine.py.230: loss: 0.17027 [I 2022-10-27 20:39:53] ...optimizers/trijoint.py.151: total_norm: 37.646877 [I 2022-10-27 20:39:58] ...trap/engines/engine.py.224: train: epoch 87 | batch 440/2304 [I 2022-10-27 20:39:58] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:17:20 [I 2022-10-27 20:39:58] ...trap/engines/engine.py.229: process: 0.56393 | load: 0.00245 [I 2022-10-27 20:39:58] ...trap/engines/engine.py.230: loss: 0.21059 [I 2022-10-27 20:39:58] ...optimizers/trijoint.py.151: total_norm: 10.177381 [I 2022-10-27 20:40:04] ...trap/engines/engine.py.224: train: epoch 87 | batch 450/2304 [I 2022-10-27 20:40:04] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:11 [I 2022-10-27 20:40:04] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00246 [I 2022-10-27 20:40:04] ...trap/engines/engine.py.230: loss: 0.14072 [I 2022-10-27 20:40:04] ...optimizers/trijoint.py.151: total_norm: 12.921065 [I 2022-10-27 20:40:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 460/2304 [I 2022-10-27 20:40:09] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:17:13 [I 2022-10-27 20:40:09] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00246 [I 2022-10-27 20:40:09] ...trap/engines/engine.py.230: loss: 0.17292 [I 2022-10-27 20:40:09] ...optimizers/trijoint.py.151: total_norm: 11.086314 [I 2022-10-27 20:40:15] ...trap/engines/engine.py.224: train: epoch 87 | batch 470/2304 [I 2022-10-27 20:40:15] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:23 [I 2022-10-27 20:40:15] ...trap/engines/engine.py.229: process: 0.55241 | load: 0.00242 [I 2022-10-27 20:40:15] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-27 20:40:15] ...optimizers/trijoint.py.151: total_norm: 12.929723 [I 2022-10-27 20:40:21] ...trap/engines/engine.py.224: train: epoch 87 | batch 480/2304 [I 2022-10-27 20:40:21] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:16:59 [I 2022-10-27 20:40:21] ...trap/engines/engine.py.229: process: 0.55143 | load: 0.00243 [I 2022-10-27 20:40:21] ...trap/engines/engine.py.230: loss: 0.11346 [I 2022-10-27 20:40:21] ...optimizers/trijoint.py.151: total_norm: 14.871031 [I 2022-10-27 20:40:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 490/2304 [I 2022-10-27 20:40:26] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:16:58 [I 2022-10-27 20:40:26] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00273 [I 2022-10-27 20:40:26] ...trap/engines/engine.py.230: loss: 0.11531 [I 2022-10-27 20:40:26] ...optimizers/trijoint.py.151: total_norm: 10.267639 [I 2022-10-27 20:40:32] ...trap/engines/engine.py.224: train: epoch 87 | batch 500/2304 [I 2022-10-27 20:40:32] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:49 [I 2022-10-27 20:40:32] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00251 [I 2022-10-27 20:40:32] ...trap/engines/engine.py.230: loss: 0.14139 [I 2022-10-27 20:40:32] ...optimizers/trijoint.py.151: total_norm: 24.269997 [I 2022-10-27 20:40:38] ...trap/engines/engine.py.224: train: epoch 87 | batch 510/2304 [I 2022-10-27 20:40:38] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:45 [I 2022-10-27 20:40:38] ...trap/engines/engine.py.229: process: 0.55101 | load: 0.00246 [I 2022-10-27 20:40:38] ...trap/engines/engine.py.230: loss: 0.14178 [I 2022-10-27 20:40:38] ...optimizers/trijoint.py.151: total_norm: 35.989487 [I 2022-10-27 20:40:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 520/2304 [I 2022-10-27 20:40:43] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:16:35 [I 2022-10-27 20:40:43] ...trap/engines/engine.py.229: process: 0.54674 | load: 0.00243 [I 2022-10-27 20:40:43] ...trap/engines/engine.py.230: loss: 0.08901 [I 2022-10-27 20:40:43] ...optimizers/trijoint.py.151: total_norm: 10.437461 [I 2022-10-27 20:40:49] ...trap/engines/engine.py.224: train: epoch 87 | batch 530/2304 [I 2022-10-27 20:40:49] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:49 [I 2022-10-27 20:40:49] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00246 [I 2022-10-27 20:40:49] ...trap/engines/engine.py.230: loss: 0.08667 [I 2022-10-27 20:40:49] ...optimizers/trijoint.py.151: total_norm: 10.915462 [I 2022-10-27 20:40:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 540/2304 [I 2022-10-27 20:40:54] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:16:21 [I 2022-10-27 20:40:54] ...trap/engines/engine.py.229: process: 0.54966 | load: 0.00244 [I 2022-10-27 20:40:54] ...trap/engines/engine.py.230: loss: 0.11249 [I 2022-10-27 20:40:54] ...optimizers/trijoint.py.151: total_norm: 11.232758 [I 2022-10-27 20:41:00] ...trap/engines/engine.py.224: train: epoch 87 | batch 550/2304 [I 2022-10-27 20:41:00] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:16 [I 2022-10-27 20:41:00] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00243 [I 2022-10-27 20:41:00] ...trap/engines/engine.py.230: loss: 0.11708 [I 2022-10-27 20:41:00] ...optimizers/trijoint.py.151: total_norm: 21.328064 [I 2022-10-27 20:41:06] ...trap/engines/engine.py.224: train: epoch 87 | batch 560/2304 [I 2022-10-27 20:41:06] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:18 [I 2022-10-27 20:41:06] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00249 [I 2022-10-27 20:41:06] ...trap/engines/engine.py.230: loss: 0.12753 [I 2022-10-27 20:41:06] ...optimizers/trijoint.py.151: total_norm: 16.349745 [I 2022-10-27 20:41:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 570/2304 [I 2022-10-27 20:41:11] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:16:10 [I 2022-10-27 20:41:11] ...trap/engines/engine.py.229: process: 0.55703 | load: 0.00243 [I 2022-10-27 20:41:11] ...trap/engines/engine.py.230: loss: 0.11909 [I 2022-10-27 20:41:11] ...optimizers/trijoint.py.151: total_norm: 9.135267 [I 2022-10-27 20:41:17] ...trap/engines/engine.py.224: train: epoch 87 | batch 580/2304 [I 2022-10-27 20:41:17] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:16:05 [I 2022-10-27 20:41:17] ...trap/engines/engine.py.229: process: 0.57131 | load: 0.00246 [I 2022-10-27 20:41:17] ...trap/engines/engine.py.230: loss: 0.14368 [I 2022-10-27 20:41:17] ...optimizers/trijoint.py.151: total_norm: 81.616730 [I 2022-10-27 20:41:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 590/2304 [I 2022-10-27 20:41:22] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:15:51 [I 2022-10-27 20:41:22] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00243 [I 2022-10-27 20:41:22] ...trap/engines/engine.py.230: loss: 0.10663 [I 2022-10-27 20:41:22] ...optimizers/trijoint.py.151: total_norm: 20.687222 [I 2022-10-27 20:41:28] ...trap/engines/engine.py.224: train: epoch 87 | batch 600/2304 [I 2022-10-27 20:41:28] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:16:08 [I 2022-10-27 20:41:28] ...trap/engines/engine.py.229: process: 0.56771 | load: 0.00243 [I 2022-10-27 20:41:28] ...trap/engines/engine.py.230: loss: 0.12948 [I 2022-10-27 20:41:28] ...optimizers/trijoint.py.151: total_norm: 17.391823 [I 2022-10-27 20:41:34] ...trap/engines/engine.py.224: train: epoch 87 | batch 610/2304 [I 2022-10-27 20:41:34] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:15:46 [I 2022-10-27 20:41:34] ...trap/engines/engine.py.229: process: 0.55602 | load: 0.00244 [I 2022-10-27 20:41:34] ...trap/engines/engine.py.230: loss: 0.10010 [I 2022-10-27 20:41:34] ...optimizers/trijoint.py.151: total_norm: 8.998250 [I 2022-10-27 20:41:39] ...trap/engines/engine.py.224: train: epoch 87 | batch 620/2304 [I 2022-10-27 20:41:39] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:15:41 [I 2022-10-27 20:41:39] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00243 [I 2022-10-27 20:41:39] ...trap/engines/engine.py.230: loss: 0.09982 [I 2022-10-27 20:41:39] ...optimizers/trijoint.py.151: total_norm: 16.050276 [I 2022-10-27 20:41:45] ...trap/engines/engine.py.224: train: epoch 87 | batch 630/2304 [I 2022-10-27 20:41:45] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:43 [I 2022-10-27 20:41:45] ...trap/engines/engine.py.229: process: 0.56687 | load: 0.00242 [I 2022-10-27 20:41:45] ...trap/engines/engine.py.230: loss: 0.09970 [I 2022-10-27 20:41:45] ...optimizers/trijoint.py.151: total_norm: 20.101152 [I 2022-10-27 20:41:51] ...trap/engines/engine.py.224: train: epoch 87 | batch 640/2304 [I 2022-10-27 20:41:51] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:15:30 [I 2022-10-27 20:41:51] ...trap/engines/engine.py.229: process: 0.55354 | load: 0.00256 [I 2022-10-27 20:41:51] ...trap/engines/engine.py.230: loss: 0.07767 [I 2022-10-27 20:41:51] ...optimizers/trijoint.py.151: total_norm: 9.182152 [I 2022-10-27 20:41:56] ...trap/engines/engine.py.224: train: epoch 87 | batch 650/2304 [I 2022-10-27 20:41:56] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:15:25 [I 2022-10-27 20:41:56] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00263 [I 2022-10-27 20:41:56] ...trap/engines/engine.py.230: loss: 0.14872 [I 2022-10-27 20:41:56] ...optimizers/trijoint.py.151: total_norm: 26.513319 [I 2022-10-27 20:42:02] ...trap/engines/engine.py.224: train: epoch 87 | batch 660/2304 [I 2022-10-27 20:42:02] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:15:25 [I 2022-10-27 20:42:02] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00246 [I 2022-10-27 20:42:02] ...trap/engines/engine.py.230: loss: 0.11177 [I 2022-10-27 20:42:02] ...optimizers/trijoint.py.151: total_norm: 10.892520 [I 2022-10-27 20:42:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 670/2304 [I 2022-10-27 20:42:07] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:15:10 [I 2022-10-27 20:42:07] ...trap/engines/engine.py.229: process: 0.55341 | load: 0.00248 [I 2022-10-27 20:42:07] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-27 20:42:07] ...optimizers/trijoint.py.151: total_norm: 9.747416 [I 2022-10-27 20:42:13] ...trap/engines/engine.py.224: train: epoch 87 | batch 680/2304 [I 2022-10-27 20:42:13] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:15:10 [I 2022-10-27 20:42:13] ...trap/engines/engine.py.229: process: 0.54974 | load: 0.00245 [I 2022-10-27 20:42:13] ...trap/engines/engine.py.230: loss: 0.08482 [I 2022-10-27 20:42:13] ...optimizers/trijoint.py.151: total_norm: 11.476268 [I 2022-10-27 20:42:19] ...trap/engines/engine.py.224: train: epoch 87 | batch 690/2304 [I 2022-10-27 20:42:19] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:15:09 [I 2022-10-27 20:42:19] ...trap/engines/engine.py.229: process: 0.56446 | load: 0.00247 [I 2022-10-27 20:42:19] ...trap/engines/engine.py.230: loss: 0.14238 [I 2022-10-27 20:42:19] ...optimizers/trijoint.py.151: total_norm: 11.482377 [I 2022-10-27 20:42:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 700/2304 [I 2022-10-27 20:42:24] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:15:05 [I 2022-10-27 20:42:24] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00252 [I 2022-10-27 20:42:24] ...trap/engines/engine.py.230: loss: 0.11809 [I 2022-10-27 20:42:24] ...optimizers/trijoint.py.151: total_norm: 34.492931 [I 2022-10-27 20:42:30] ...trap/engines/engine.py.224: train: epoch 87 | batch 710/2304 [I 2022-10-27 20:42:30] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:14:56 [I 2022-10-27 20:42:30] ...trap/engines/engine.py.229: process: 0.54905 | load: 0.00273 [I 2022-10-27 20:42:30] ...trap/engines/engine.py.230: loss: 0.10471 [I 2022-10-27 20:42:30] ...optimizers/trijoint.py.151: total_norm: 7.723438 [I 2022-10-27 20:42:36] ...trap/engines/engine.py.224: train: epoch 87 | batch 720/2304 [I 2022-10-27 20:42:36] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:14:43 [I 2022-10-27 20:42:36] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00249 [I 2022-10-27 20:42:36] ...trap/engines/engine.py.230: loss: 0.12186 [I 2022-10-27 20:42:36] ...optimizers/trijoint.py.151: total_norm: 10.991336 [I 2022-10-27 20:42:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 730/2304 [I 2022-10-27 20:42:41] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:14:42 [I 2022-10-27 20:42:41] ...trap/engines/engine.py.229: process: 0.55895 | load: 0.00244 [I 2022-10-27 20:42:41] ...trap/engines/engine.py.230: loss: 0.09740 [I 2022-10-27 20:42:41] ...optimizers/trijoint.py.151: total_norm: 7.922103 [I 2022-10-27 20:42:47] ...trap/engines/engine.py.224: train: epoch 87 | batch 740/2304 [I 2022-10-27 20:42:47] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:14:43 [I 2022-10-27 20:42:47] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00247 [I 2022-10-27 20:42:47] ...trap/engines/engine.py.230: loss: 0.17691 [I 2022-10-27 20:42:47] ...optimizers/trijoint.py.151: total_norm: 16.639475 [I 2022-10-27 20:42:53] ...trap/engines/engine.py.224: train: epoch 87 | batch 750/2304 [I 2022-10-27 20:42:53] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:14:40 [I 2022-10-27 20:42:53] ...trap/engines/engine.py.229: process: 0.56159 | load: 0.00243 [I 2022-10-27 20:42:53] ...trap/engines/engine.py.230: loss: 0.13380 [I 2022-10-27 20:42:53] ...optimizers/trijoint.py.151: total_norm: 9.387271 [I 2022-10-27 20:42:58] ...trap/engines/engine.py.224: train: epoch 87 | batch 760/2304 [I 2022-10-27 20:42:58] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:14:32 [I 2022-10-27 20:42:58] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00242 [I 2022-10-27 20:42:58] ...trap/engines/engine.py.230: loss: 0.11136 [I 2022-10-27 20:42:58] ...optimizers/trijoint.py.151: total_norm: 10.560950 [I 2022-10-27 20:43:04] ...trap/engines/engine.py.224: train: epoch 87 | batch 770/2304 [I 2022-10-27 20:43:04] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:14:21 [I 2022-10-27 20:43:04] ...trap/engines/engine.py.229: process: 0.56023 | load: 0.00243 [I 2022-10-27 20:43:04] ...trap/engines/engine.py.230: loss: 0.13025 [I 2022-10-27 20:43:04] ...optimizers/trijoint.py.151: total_norm: 15.441581 [I 2022-10-27 20:43:10] ...trap/engines/engine.py.224: train: epoch 87 | batch 780/2304 [I 2022-10-27 20:43:10] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:14:21 [I 2022-10-27 20:43:10] ...trap/engines/engine.py.229: process: 0.56886 | load: 0.00241 [I 2022-10-27 20:43:10] ...trap/engines/engine.py.230: loss: 0.10917 [I 2022-10-27 20:43:10] ...optimizers/trijoint.py.151: total_norm: 10.726436 [I 2022-10-27 20:43:15] ...trap/engines/engine.py.224: train: epoch 87 | batch 790/2304 [I 2022-10-27 20:43:15] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:14:18 [I 2022-10-27 20:43:15] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00242 [I 2022-10-27 20:43:15] ...trap/engines/engine.py.230: loss: 0.19630 [I 2022-10-27 20:43:15] ...optimizers/trijoint.py.151: total_norm: 16.122612 [I 2022-10-27 20:43:21] ...trap/engines/engine.py.224: train: epoch 87 | batch 800/2304 [I 2022-10-27 20:43:21] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:14:10 [I 2022-10-27 20:43:21] ...trap/engines/engine.py.229: process: 0.56845 | load: 0.00242 [I 2022-10-27 20:43:21] ...trap/engines/engine.py.230: loss: 0.13418 [I 2022-10-27 20:43:21] ...optimizers/trijoint.py.151: total_norm: 12.383265 [I 2022-10-27 20:43:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 810/2304 [I 2022-10-27 20:43:26] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:13:54 [I 2022-10-27 20:43:26] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00252 [I 2022-10-27 20:43:26] ...trap/engines/engine.py.230: loss: 0.12623 [I 2022-10-27 20:43:26] ...optimizers/trijoint.py.151: total_norm: 9.744691 [I 2022-10-27 20:43:32] ...trap/engines/engine.py.224: train: epoch 87 | batch 820/2304 [I 2022-10-27 20:43:32] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:13:51 [I 2022-10-27 20:43:32] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00249 [I 2022-10-27 20:43:32] ...trap/engines/engine.py.230: loss: 0.10310 [I 2022-10-27 20:43:32] ...optimizers/trijoint.py.151: total_norm: 19.344456 [I 2022-10-27 20:43:38] ...trap/engines/engine.py.224: train: epoch 87 | batch 830/2304 [I 2022-10-27 20:43:38] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:13:46 [I 2022-10-27 20:43:38] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00256 [I 2022-10-27 20:43:38] ...trap/engines/engine.py.230: loss: 0.14659 [I 2022-10-27 20:43:38] ...optimizers/trijoint.py.151: total_norm: 14.014653 [I 2022-10-27 20:43:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 840/2304 [I 2022-10-27 20:43:43] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:13:28 [I 2022-10-27 20:43:43] ...trap/engines/engine.py.229: process: 0.52994 | load: 0.00245 [I 2022-10-27 20:43:43] ...trap/engines/engine.py.230: loss: 0.19423 [I 2022-10-27 20:43:43] ...optimizers/trijoint.py.151: total_norm: 6.768914 [I 2022-10-27 20:43:49] ...trap/engines/engine.py.224: train: epoch 87 | batch 850/2304 [I 2022-10-27 20:43:49] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:13:32 [I 2022-10-27 20:43:49] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00242 [I 2022-10-27 20:43:49] ...trap/engines/engine.py.230: loss: 0.20250 [I 2022-10-27 20:43:49] ...optimizers/trijoint.py.151: total_norm: 26.873095 [I 2022-10-27 20:43:55] ...trap/engines/engine.py.224: train: epoch 87 | batch 860/2304 [I 2022-10-27 20:43:55] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:13:29 [I 2022-10-27 20:43:55] ...trap/engines/engine.py.229: process: 0.56567 | load: 0.00245 [I 2022-10-27 20:43:55] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-27 20:43:55] ...optimizers/trijoint.py.151: total_norm: 15.666631 [I 2022-10-27 20:44:00] ...trap/engines/engine.py.224: train: epoch 87 | batch 870/2304 [I 2022-10-27 20:44:00] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:21 [I 2022-10-27 20:44:00] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00246 [I 2022-10-27 20:44:00] ...trap/engines/engine.py.230: loss: 0.09810 [I 2022-10-27 20:44:00] ...optimizers/trijoint.py.151: total_norm: 13.392606 [I 2022-10-27 20:44:06] ...trap/engines/engine.py.224: train: epoch 87 | batch 880/2304 [I 2022-10-27 20:44:06] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:13:16 [I 2022-10-27 20:44:06] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00264 [I 2022-10-27 20:44:06] ...trap/engines/engine.py.230: loss: 0.17559 [I 2022-10-27 20:44:06] ...optimizers/trijoint.py.151: total_norm: 26.973728 [I 2022-10-27 20:44:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 890/2304 [I 2022-10-27 20:44:11] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:13:21 [I 2022-10-27 20:44:11] ...trap/engines/engine.py.229: process: 0.58044 | load: 0.00253 [I 2022-10-27 20:44:11] ...trap/engines/engine.py.230: loss: 0.14959 [I 2022-10-27 20:44:11] ...optimizers/trijoint.py.151: total_norm: 18.356228 [I 2022-10-27 20:44:17] ...trap/engines/engine.py.224: train: epoch 87 | batch 900/2304 [I 2022-10-27 20:44:17] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:07 [I 2022-10-27 20:44:17] ...trap/engines/engine.py.229: process: 0.55432 | load: 0.00250 [I 2022-10-27 20:44:17] ...trap/engines/engine.py.230: loss: 0.11805 [I 2022-10-27 20:44:17] ...optimizers/trijoint.py.151: total_norm: 20.857126 [I 2022-10-27 20:44:23] ...trap/engines/engine.py.224: train: epoch 87 | batch 910/2304 [I 2022-10-27 20:44:23] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:12:54 [I 2022-10-27 20:44:23] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00247 [I 2022-10-27 20:44:23] ...trap/engines/engine.py.230: loss: 0.15721 [I 2022-10-27 20:44:23] ...optimizers/trijoint.py.151: total_norm: 14.914905 [I 2022-10-27 20:44:28] ...trap/engines/engine.py.224: train: epoch 87 | batch 920/2304 [I 2022-10-27 20:44:28] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:12:52 [I 2022-10-27 20:44:28] ...trap/engines/engine.py.229: process: 0.55507 | load: 0.00244 [I 2022-10-27 20:44:28] ...trap/engines/engine.py.230: loss: 0.17140 [I 2022-10-27 20:44:28] ...optimizers/trijoint.py.151: total_norm: 21.783476 [I 2022-10-27 20:44:34] ...trap/engines/engine.py.224: train: epoch 87 | batch 930/2304 [I 2022-10-27 20:44:34] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:12:46 [I 2022-10-27 20:44:34] ...trap/engines/engine.py.229: process: 0.56649 | load: 0.00247 [I 2022-10-27 20:44:34] ...trap/engines/engine.py.230: loss: 0.11869 [I 2022-10-27 20:44:34] ...optimizers/trijoint.py.151: total_norm: 32.027039 [I 2022-10-27 20:44:39] ...trap/engines/engine.py.224: train: epoch 87 | batch 940/2304 [I 2022-10-27 20:44:39] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:12:40 [I 2022-10-27 20:44:39] ...trap/engines/engine.py.229: process: 0.56163 | load: 0.00242 [I 2022-10-27 20:44:39] ...trap/engines/engine.py.230: loss: 0.09817 [I 2022-10-27 20:44:39] ...optimizers/trijoint.py.151: total_norm: 10.932716 [I 2022-10-27 20:44:45] ...trap/engines/engine.py.224: train: epoch 87 | batch 950/2304 [I 2022-10-27 20:44:45] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:12:35 [I 2022-10-27 20:44:45] ...trap/engines/engine.py.229: process: 0.56067 | load: 0.00242 [I 2022-10-27 20:44:45] ...trap/engines/engine.py.230: loss: 0.13862 [I 2022-10-27 20:44:45] ...optimizers/trijoint.py.151: total_norm: 23.398512 [I 2022-10-27 20:44:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 960/2304 [I 2022-10-27 20:44:50] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:12:25 [I 2022-10-27 20:44:50] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00241 [I 2022-10-27 20:44:50] ...trap/engines/engine.py.230: loss: 0.12359 [I 2022-10-27 20:44:50] ...optimizers/trijoint.py.151: total_norm: 11.651223 [I 2022-10-27 20:44:56] ...trap/engines/engine.py.224: train: epoch 87 | batch 970/2304 [I 2022-10-27 20:44:56] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:21 [I 2022-10-27 20:44:56] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00244 [I 2022-10-27 20:44:56] ...trap/engines/engine.py.230: loss: 0.13735 [I 2022-10-27 20:44:56] ...optimizers/trijoint.py.151: total_norm: 24.442617 [I 2022-10-27 20:45:02] ...trap/engines/engine.py.224: train: epoch 87 | batch 980/2304 [I 2022-10-27 20:45:02] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:22 [I 2022-10-27 20:45:02] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00240 [I 2022-10-27 20:45:02] ...trap/engines/engine.py.230: loss: 0.10938 [I 2022-10-27 20:45:02] ...optimizers/trijoint.py.151: total_norm: 14.965642 [I 2022-10-27 20:45:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 990/2304 [I 2022-10-27 20:45:07] ...trap/engines/engine.py.228: elapsed: 0:09:19 | left: 0:12:16 [I 2022-10-27 20:45:07] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00242 [I 2022-10-27 20:45:07] ...trap/engines/engine.py.230: loss: 0.10328 [I 2022-10-27 20:45:07] ...optimizers/trijoint.py.151: total_norm: 12.792105 [I 2022-10-27 20:45:13] ...trap/engines/engine.py.224: train: epoch 87 | batch 1000/2304 [I 2022-10-27 20:45:13] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:06 [I 2022-10-27 20:45:13] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00244 [I 2022-10-27 20:45:13] ...trap/engines/engine.py.230: loss: 0.08694 [I 2022-10-27 20:45:13] ...optimizers/trijoint.py.151: total_norm: 12.918222 [I 2022-10-27 20:45:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 1010/2304 [I 2022-10-27 20:45:18] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:01 [I 2022-10-27 20:45:18] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00249 [I 2022-10-27 20:45:18] ...trap/engines/engine.py.230: loss: 0.13893 [I 2022-10-27 20:45:18] ...optimizers/trijoint.py.151: total_norm: 11.462749 [I 2022-10-27 20:45:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 1020/2304 [I 2022-10-27 20:45:24] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:12:02 [I 2022-10-27 20:45:24] ...trap/engines/engine.py.229: process: 0.56665 | load: 0.00248 [I 2022-10-27 20:45:24] ...trap/engines/engine.py.230: loss: 0.12245 [I 2022-10-27 20:45:24] ...optimizers/trijoint.py.151: total_norm: 13.063850 [I 2022-10-27 20:45:30] ...trap/engines/engine.py.224: train: epoch 87 | batch 1030/2304 [I 2022-10-27 20:45:30] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:11:55 [I 2022-10-27 20:45:30] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00246 [I 2022-10-27 20:45:30] ...trap/engines/engine.py.230: loss: 0.12048 [I 2022-10-27 20:45:30] ...optimizers/trijoint.py.151: total_norm: 10.514993 [I 2022-10-27 20:45:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 1040/2304 [I 2022-10-27 20:45:35] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:11:46 [I 2022-10-27 20:45:35] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00243 [I 2022-10-27 20:45:35] ...trap/engines/engine.py.230: loss: 0.11051 [I 2022-10-27 20:45:35] ...optimizers/trijoint.py.151: total_norm: 10.659439 [I 2022-10-27 20:45:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 1050/2304 [I 2022-10-27 20:45:41] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:42 [I 2022-10-27 20:45:41] ...trap/engines/engine.py.229: process: 0.56261 | load: 0.00245 [I 2022-10-27 20:45:41] ...trap/engines/engine.py.230: loss: 0.11914 [I 2022-10-27 20:45:41] ...optimizers/trijoint.py.151: total_norm: 8.771772 [I 2022-10-27 20:45:46] ...trap/engines/engine.py.224: train: epoch 87 | batch 1060/2304 [I 2022-10-27 20:45:46] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:35 [I 2022-10-27 20:45:46] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00244 [I 2022-10-27 20:45:46] ...trap/engines/engine.py.230: loss: 0.11510 [I 2022-10-27 20:45:46] ...optimizers/trijoint.py.151: total_norm: 16.038635 [I 2022-10-27 20:45:52] ...trap/engines/engine.py.224: train: epoch 87 | batch 1070/2304 [I 2022-10-27 20:45:52] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:28 [I 2022-10-27 20:45:52] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00242 [I 2022-10-27 20:45:52] ...trap/engines/engine.py.230: loss: 0.16462 [I 2022-10-27 20:45:52] ...optimizers/trijoint.py.151: total_norm: 32.643044 [I 2022-10-27 20:45:58] ...trap/engines/engine.py.224: train: epoch 87 | batch 1080/2304 [I 2022-10-27 20:45:58] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:21 [I 2022-10-27 20:45:58] ...trap/engines/engine.py.229: process: 0.54955 | load: 0.00243 [I 2022-10-27 20:45:58] ...trap/engines/engine.py.230: loss: 0.12175 [I 2022-10-27 20:45:58] ...optimizers/trijoint.py.151: total_norm: 11.186090 [I 2022-10-27 20:46:03] ...trap/engines/engine.py.224: train: epoch 87 | batch 1090/2304 [I 2022-10-27 20:46:03] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:20 [I 2022-10-27 20:46:03] ...trap/engines/engine.py.229: process: 0.56982 | load: 0.00252 [I 2022-10-27 20:46:03] ...trap/engines/engine.py.230: loss: 0.14015 [I 2022-10-27 20:46:03] ...optimizers/trijoint.py.151: total_norm: 29.464697 [I 2022-10-27 20:46:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 1100/2304 [I 2022-10-27 20:46:09] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:13 [I 2022-10-27 20:46:09] ...trap/engines/engine.py.229: process: 0.56732 | load: 0.00255 [I 2022-10-27 20:46:09] ...trap/engines/engine.py.230: loss: 0.15291 [I 2022-10-27 20:46:09] ...optimizers/trijoint.py.151: total_norm: 112.890152 [I 2022-10-27 20:46:14] ...trap/engines/engine.py.224: train: epoch 87 | batch 1110/2304 [I 2022-10-27 20:46:14] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:02 [I 2022-10-27 20:46:14] ...trap/engines/engine.py.229: process: 0.55576 | load: 0.00244 [I 2022-10-27 20:46:14] ...trap/engines/engine.py.230: loss: 0.09496 [I 2022-10-27 20:46:14] ...optimizers/trijoint.py.151: total_norm: 12.760568 [I 2022-10-27 20:46:20] ...trap/engines/engine.py.224: train: epoch 87 | batch 1120/2304 [I 2022-10-27 20:46:20] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:04 [I 2022-10-27 20:46:20] ...trap/engines/engine.py.229: process: 0.57442 | load: 0.00244 [I 2022-10-27 20:46:20] ...trap/engines/engine.py.230: loss: 0.13149 [I 2022-10-27 20:46:20] ...optimizers/trijoint.py.151: total_norm: 19.161459 [I 2022-10-27 20:46:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 1130/2304 [I 2022-10-27 20:46:26] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:16 [I 2022-10-27 20:46:26] ...trap/engines/engine.py.229: process: 0.57876 | load: 0.00244 [I 2022-10-27 20:46:26] ...trap/engines/engine.py.230: loss: 0.12882 [I 2022-10-27 20:46:26] ...optimizers/trijoint.py.151: total_norm: 13.283062 [I 2022-10-27 20:46:31] ...trap/engines/engine.py.224: train: epoch 87 | batch 1140/2304 [I 2022-10-27 20:46:31] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:10:56 [I 2022-10-27 20:46:31] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00247 [I 2022-10-27 20:46:31] ...trap/engines/engine.py.230: loss: 0.13983 [I 2022-10-27 20:46:31] ...optimizers/trijoint.py.151: total_norm: 10.572476 [I 2022-10-27 20:46:37] ...trap/engines/engine.py.224: train: epoch 87 | batch 1150/2304 [I 2022-10-27 20:46:37] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:55 [I 2022-10-27 20:46:37] ...trap/engines/engine.py.229: process: 0.56508 | load: 0.00243 [I 2022-10-27 20:46:37] ...trap/engines/engine.py.230: loss: 0.18004 [I 2022-10-27 20:46:37] ...optimizers/trijoint.py.151: total_norm: 40.623856 [I 2022-10-27 20:46:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 1160/2304 [I 2022-10-27 20:46:43] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:38 [I 2022-10-27 20:46:43] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00248 [I 2022-10-27 20:46:43] ...trap/engines/engine.py.230: loss: 0.10082 [I 2022-10-27 20:46:43] ...optimizers/trijoint.py.151: total_norm: 10.450010 [I 2022-10-27 20:46:48] ...trap/engines/engine.py.224: train: epoch 87 | batch 1170/2304 [I 2022-10-27 20:46:48] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:27 [I 2022-10-27 20:46:48] ...trap/engines/engine.py.229: process: 0.54899 | load: 0.00246 [I 2022-10-27 20:46:48] ...trap/engines/engine.py.230: loss: 0.17396 [I 2022-10-27 20:46:48] ...optimizers/trijoint.py.151: total_norm: 17.748455 [I 2022-10-27 20:46:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 1180/2304 [I 2022-10-27 20:46:54] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:23 [I 2022-10-27 20:46:54] ...trap/engines/engine.py.229: process: 0.55047 | load: 0.00240 [I 2022-10-27 20:46:54] ...trap/engines/engine.py.230: loss: 0.09513 [I 2022-10-27 20:46:54] ...optimizers/trijoint.py.151: total_norm: 12.207639 [I 2022-10-27 20:46:59] ...trap/engines/engine.py.224: train: epoch 87 | batch 1190/2304 [I 2022-10-27 20:46:59] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:22 [I 2022-10-27 20:46:59] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00244 [I 2022-10-27 20:46:59] ...trap/engines/engine.py.230: loss: 0.13842 [I 2022-10-27 20:46:59] ...optimizers/trijoint.py.151: total_norm: 16.720337 [I 2022-10-27 20:47:05] ...trap/engines/engine.py.224: train: epoch 87 | batch 1200/2304 [I 2022-10-27 20:47:05] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:15 [I 2022-10-27 20:47:05] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00243 [I 2022-10-27 20:47:05] ...trap/engines/engine.py.230: loss: 0.09171 [I 2022-10-27 20:47:05] ...optimizers/trijoint.py.151: total_norm: 9.878090 [I 2022-10-27 20:47:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 1210/2304 [I 2022-10-27 20:47:11] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:11 [I 2022-10-27 20:47:11] ...trap/engines/engine.py.229: process: 0.56618 | load: 0.00243 [I 2022-10-27 20:47:11] ...trap/engines/engine.py.230: loss: 0.15041 [I 2022-10-27 20:47:11] ...optimizers/trijoint.py.151: total_norm: 17.663437 [I 2022-10-27 20:47:16] ...trap/engines/engine.py.224: train: epoch 87 | batch 1220/2304 [I 2022-10-27 20:47:16] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:09 [I 2022-10-27 20:47:16] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00241 [I 2022-10-27 20:47:16] ...trap/engines/engine.py.230: loss: 0.16266 [I 2022-10-27 20:47:16] ...optimizers/trijoint.py.151: total_norm: 8.356612 [I 2022-10-27 20:47:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 1230/2304 [I 2022-10-27 20:47:22] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:09:54 [I 2022-10-27 20:47:22] ...trap/engines/engine.py.229: process: 0.55090 | load: 0.00253 [I 2022-10-27 20:47:22] ...trap/engines/engine.py.230: loss: 0.14844 [I 2022-10-27 20:47:22] ...optimizers/trijoint.py.151: total_norm: 9.892163 [I 2022-10-27 20:47:27] ...trap/engines/engine.py.224: train: epoch 87 | batch 1240/2304 [I 2022-10-27 20:47:27] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:09:50 [I 2022-10-27 20:47:27] ...trap/engines/engine.py.229: process: 0.54694 | load: 0.00246 [I 2022-10-27 20:47:27] ...trap/engines/engine.py.230: loss: 0.10012 [I 2022-10-27 20:47:27] ...optimizers/trijoint.py.151: total_norm: 9.797270 [I 2022-10-27 20:47:33] ...trap/engines/engine.py.224: train: epoch 87 | batch 1250/2304 [I 2022-10-27 20:47:33] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:09:47 [I 2022-10-27 20:47:33] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00246 [I 2022-10-27 20:47:33] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-27 20:47:33] ...optimizers/trijoint.py.151: total_norm: 17.131252 [I 2022-10-27 20:47:39] ...trap/engines/engine.py.224: train: epoch 87 | batch 1260/2304 [I 2022-10-27 20:47:39] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:46 [I 2022-10-27 20:47:39] ...trap/engines/engine.py.229: process: 0.56859 | load: 0.00243 [I 2022-10-27 20:47:39] ...trap/engines/engine.py.230: loss: 0.15693 [I 2022-10-27 20:47:39] ...optimizers/trijoint.py.151: total_norm: 28.630304 [I 2022-10-27 20:47:44] ...trap/engines/engine.py.224: train: epoch 87 | batch 1270/2304 [I 2022-10-27 20:47:44] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:36 [I 2022-10-27 20:47:44] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00244 [I 2022-10-27 20:47:44] ...trap/engines/engine.py.230: loss: 0.09582 [I 2022-10-27 20:47:44] ...optimizers/trijoint.py.151: total_norm: 12.656080 [I 2022-10-27 20:47:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 1280/2304 [I 2022-10-27 20:47:50] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:35 [I 2022-10-27 20:47:50] ...trap/engines/engine.py.229: process: 0.54555 | load: 0.00245 [I 2022-10-27 20:47:50] ...trap/engines/engine.py.230: loss: 0.13913 [I 2022-10-27 20:47:50] ...optimizers/trijoint.py.151: total_norm: 17.744091 [I 2022-10-27 20:47:55] ...trap/engines/engine.py.224: train: epoch 87 | batch 1290/2304 [I 2022-10-27 20:47:55] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:28 [I 2022-10-27 20:47:55] ...trap/engines/engine.py.229: process: 0.56650 | load: 0.00243 [I 2022-10-27 20:47:55] ...trap/engines/engine.py.230: loss: 0.11308 [I 2022-10-27 20:47:55] ...optimizers/trijoint.py.151: total_norm: 18.241533 [I 2022-10-27 20:48:01] ...trap/engines/engine.py.224: train: epoch 87 | batch 1300/2304 [I 2022-10-27 20:48:01] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:22 [I 2022-10-27 20:48:01] ...trap/engines/engine.py.229: process: 0.56198 | load: 0.00247 [I 2022-10-27 20:48:01] ...trap/engines/engine.py.230: loss: 0.09607 [I 2022-10-27 20:48:01] ...optimizers/trijoint.py.151: total_norm: 8.912166 [I 2022-10-27 20:48:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 1310/2304 [I 2022-10-27 20:48:07] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:18 [I 2022-10-27 20:48:07] ...trap/engines/engine.py.229: process: 0.57693 | load: 0.00246 [I 2022-10-27 20:48:07] ...trap/engines/engine.py.230: loss: 0.13343 [I 2022-10-27 20:48:07] ...optimizers/trijoint.py.151: total_norm: 23.183281 [I 2022-10-27 20:48:12] ...trap/engines/engine.py.224: train: epoch 87 | batch 1320/2304 [I 2022-10-27 20:48:12] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:10 [I 2022-10-27 20:48:12] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00246 [I 2022-10-27 20:48:12] ...trap/engines/engine.py.230: loss: 0.10524 [I 2022-10-27 20:48:12] ...optimizers/trijoint.py.151: total_norm: 12.302958 [I 2022-10-27 20:48:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 1330/2304 [I 2022-10-27 20:48:18] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:00 [I 2022-10-27 20:48:18] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00245 [I 2022-10-27 20:48:18] ...trap/engines/engine.py.230: loss: 0.13386 [I 2022-10-27 20:48:18] ...optimizers/trijoint.py.151: total_norm: 30.298239 [I 2022-10-27 20:48:23] ...trap/engines/engine.py.224: train: epoch 87 | batch 1340/2304 [I 2022-10-27 20:48:23] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:08:59 [I 2022-10-27 20:48:23] ...trap/engines/engine.py.229: process: 0.55612 | load: 0.00242 [I 2022-10-27 20:48:23] ...trap/engines/engine.py.230: loss: 0.09941 [I 2022-10-27 20:48:23] ...optimizers/trijoint.py.151: total_norm: 16.614485 [I 2022-10-27 20:48:29] ...trap/engines/engine.py.224: train: epoch 87 | batch 1350/2304 [I 2022-10-27 20:48:29] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:08:54 [I 2022-10-27 20:48:29] ...trap/engines/engine.py.229: process: 0.56864 | load: 0.00245 [I 2022-10-27 20:48:29] ...trap/engines/engine.py.230: loss: 0.13665 [I 2022-10-27 20:48:29] ...optimizers/trijoint.py.151: total_norm: 30.240702 [I 2022-10-27 20:48:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 1360/2304 [I 2022-10-27 20:48:35] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:08:45 [I 2022-10-27 20:48:35] ...trap/engines/engine.py.229: process: 0.54686 | load: 0.00244 [I 2022-10-27 20:48:35] ...trap/engines/engine.py.230: loss: 0.16290 [I 2022-10-27 20:48:35] ...optimizers/trijoint.py.151: total_norm: 28.818178 [I 2022-10-27 20:49:00] ...trap/engines/engine.py.224: train: epoch 87 | batch 1370/2304 [I 2022-10-27 20:49:00] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:16:53 [I 2022-10-27 20:49:00] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00245 [I 2022-10-27 20:49:00] ...trap/engines/engine.py.230: loss: 0.11801 [I 2022-10-27 20:49:00] ...optimizers/trijoint.py.151: total_norm: 30.020285 [I 2022-10-27 20:49:05] ...trap/engines/engine.py.224: train: epoch 87 | batch 1380/2304 [I 2022-10-27 20:49:05] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:09:27 [I 2022-10-27 20:49:05] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00245 [I 2022-10-27 20:49:05] ...trap/engines/engine.py.230: loss: 0.11690 [I 2022-10-27 20:49:05] ...optimizers/trijoint.py.151: total_norm: 15.846240 [I 2022-10-27 20:49:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 1390/2304 [I 2022-10-27 20:49:11] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:35 [I 2022-10-27 20:49:11] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00248 [I 2022-10-27 20:49:11] ...trap/engines/engine.py.230: loss: 0.12580 [I 2022-10-27 20:49:11] ...optimizers/trijoint.py.151: total_norm: 12.575006 [I 2022-10-27 20:49:17] ...trap/engines/engine.py.224: train: epoch 87 | batch 1400/2304 [I 2022-10-27 20:49:17] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:26 [I 2022-10-27 20:49:17] ...trap/engines/engine.py.229: process: 0.56554 | load: 0.00253 [I 2022-10-27 20:49:17] ...trap/engines/engine.py.230: loss: 0.10105 [I 2022-10-27 20:49:17] ...optimizers/trijoint.py.151: total_norm: 14.561035 [I 2022-10-27 20:49:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 1410/2304 [I 2022-10-27 20:49:22] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:08:21 [I 2022-10-27 20:49:22] ...trap/engines/engine.py.229: process: 0.56500 | load: 0.00261 [I 2022-10-27 20:49:22] ...trap/engines/engine.py.230: loss: 0.14508 [I 2022-10-27 20:49:22] ...optimizers/trijoint.py.151: total_norm: 20.271414 [I 2022-10-27 20:49:28] ...trap/engines/engine.py.224: train: epoch 87 | batch 1420/2304 [I 2022-10-27 20:49:28] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:08:13 [I 2022-10-27 20:49:28] ...trap/engines/engine.py.229: process: 0.55356 | load: 0.00262 [I 2022-10-27 20:49:28] ...trap/engines/engine.py.230: loss: 0.14045 [I 2022-10-27 20:49:28] ...optimizers/trijoint.py.151: total_norm: 45.662384 [I 2022-10-27 20:49:33] ...trap/engines/engine.py.224: train: epoch 87 | batch 1430/2304 [I 2022-10-27 20:49:33] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:08:08 [I 2022-10-27 20:49:33] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00250 [I 2022-10-27 20:49:33] ...trap/engines/engine.py.230: loss: 0.17654 [I 2022-10-27 20:49:33] ...optimizers/trijoint.py.151: total_norm: 21.248713 [I 2022-10-27 20:49:39] ...trap/engines/engine.py.224: train: epoch 87 | batch 1440/2304 [I 2022-10-27 20:49:39] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:08:00 [I 2022-10-27 20:49:39] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00256 [I 2022-10-27 20:49:39] ...trap/engines/engine.py.230: loss: 0.10103 [I 2022-10-27 20:49:39] ...optimizers/trijoint.py.151: total_norm: 22.401979 [I 2022-10-27 20:49:44] ...trap/engines/engine.py.224: train: epoch 87 | batch 1450/2304 [I 2022-10-27 20:49:44] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:56 [I 2022-10-27 20:49:44] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00261 [I 2022-10-27 20:49:44] ...trap/engines/engine.py.230: loss: 0.14127 [I 2022-10-27 20:49:44] ...optimizers/trijoint.py.151: total_norm: 73.142670 [I 2022-10-27 20:49:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 1460/2304 [I 2022-10-27 20:49:50] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:52 [I 2022-10-27 20:49:50] ...trap/engines/engine.py.229: process: 0.56336 | load: 0.00266 [I 2022-10-27 20:49:50] ...trap/engines/engine.py.230: loss: 0.10723 [I 2022-10-27 20:49:50] ...optimizers/trijoint.py.151: total_norm: 16.373173 [I 2022-10-27 20:49:56] ...trap/engines/engine.py.224: train: epoch 87 | batch 1470/2304 [I 2022-10-27 20:49:56] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:45 [I 2022-10-27 20:49:56] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00260 [I 2022-10-27 20:49:56] ...trap/engines/engine.py.230: loss: 0.10709 [I 2022-10-27 20:49:56] ...optimizers/trijoint.py.151: total_norm: 15.804235 [I 2022-10-27 20:50:01] ...trap/engines/engine.py.224: train: epoch 87 | batch 1480/2304 [I 2022-10-27 20:50:01] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:34 [I 2022-10-27 20:50:01] ...trap/engines/engine.py.229: process: 0.54912 | load: 0.00263 [I 2022-10-27 20:50:01] ...trap/engines/engine.py.230: loss: 0.14351 [I 2022-10-27 20:50:01] ...optimizers/trijoint.py.151: total_norm: 11.561485 [I 2022-10-27 20:50:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 1490/2304 [I 2022-10-27 20:50:07] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:35 [I 2022-10-27 20:50:07] ...trap/engines/engine.py.229: process: 0.56070 | load: 0.00259 [I 2022-10-27 20:50:07] ...trap/engines/engine.py.230: loss: 0.09660 [I 2022-10-27 20:50:07] ...optimizers/trijoint.py.151: total_norm: 20.593956 [I 2022-10-27 20:50:13] ...trap/engines/engine.py.224: train: epoch 87 | batch 1500/2304 [I 2022-10-27 20:50:13] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:26 [I 2022-10-27 20:50:13] ...trap/engines/engine.py.229: process: 0.54923 | load: 0.00259 [I 2022-10-27 20:50:13] ...trap/engines/engine.py.230: loss: 0.17624 [I 2022-10-27 20:50:13] ...optimizers/trijoint.py.151: total_norm: 25.996502 [I 2022-10-27 20:50:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 1510/2304 [I 2022-10-27 20:50:18] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:25 [I 2022-10-27 20:50:18] ...trap/engines/engine.py.229: process: 0.55793 | load: 0.00259 [I 2022-10-27 20:50:18] ...trap/engines/engine.py.230: loss: 0.10932 [I 2022-10-27 20:50:18] ...optimizers/trijoint.py.151: total_norm: 63.401642 [I 2022-10-27 20:50:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 1520/2304 [I 2022-10-27 20:50:24] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:07:16 [I 2022-10-27 20:50:24] ...trap/engines/engine.py.229: process: 0.54333 | load: 0.00260 [I 2022-10-27 20:50:24] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-27 20:50:24] ...optimizers/trijoint.py.151: total_norm: 15.339537 [I 2022-10-27 20:50:29] ...trap/engines/engine.py.224: train: epoch 87 | batch 1530/2304 [I 2022-10-27 20:50:29] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:07:10 [I 2022-10-27 20:50:29] ...trap/engines/engine.py.229: process: 0.54481 | load: 0.00263 [I 2022-10-27 20:50:29] ...trap/engines/engine.py.230: loss: 0.13747 [I 2022-10-27 20:50:29] ...optimizers/trijoint.py.151: total_norm: 11.355204 [I 2022-10-27 20:50:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 1540/2304 [I 2022-10-27 20:50:35] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:07:05 [I 2022-10-27 20:50:35] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00259 [I 2022-10-27 20:50:35] ...trap/engines/engine.py.230: loss: 0.14401 [I 2022-10-27 20:50:35] ...optimizers/trijoint.py.151: total_norm: 14.401737 [I 2022-10-27 20:50:40] ...trap/engines/engine.py.224: train: epoch 87 | batch 1550/2304 [I 2022-10-27 20:50:40] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:07:02 [I 2022-10-27 20:50:40] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00274 [I 2022-10-27 20:50:40] ...trap/engines/engine.py.230: loss: 0.16739 [I 2022-10-27 20:50:40] ...optimizers/trijoint.py.151: total_norm: 23.909357 [I 2022-10-27 20:50:46] ...trap/engines/engine.py.224: train: epoch 87 | batch 1560/2304 [I 2022-10-27 20:50:46] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:56 [I 2022-10-27 20:50:46] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00258 [I 2022-10-27 20:50:46] ...trap/engines/engine.py.230: loss: 0.09538 [I 2022-10-27 20:50:46] ...optimizers/trijoint.py.151: total_norm: 13.946040 [I 2022-10-27 20:50:52] ...trap/engines/engine.py.224: train: epoch 87 | batch 1570/2304 [I 2022-10-27 20:50:52] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:50 [I 2022-10-27 20:50:52] ...trap/engines/engine.py.229: process: 0.56325 | load: 0.00252 [I 2022-10-27 20:50:52] ...trap/engines/engine.py.230: loss: 0.11204 [I 2022-10-27 20:50:52] ...optimizers/trijoint.py.151: total_norm: 21.743200 [I 2022-10-27 20:50:57] ...trap/engines/engine.py.224: train: epoch 87 | batch 1580/2304 [I 2022-10-27 20:50:57] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:53 [I 2022-10-27 20:50:57] ...trap/engines/engine.py.229: process: 0.61419 | load: 0.00274 [I 2022-10-27 20:50:57] ...trap/engines/engine.py.230: loss: 0.13018 [I 2022-10-27 20:50:57] ...optimizers/trijoint.py.151: total_norm: 15.570375 [I 2022-10-27 20:51:03] ...trap/engines/engine.py.224: train: epoch 87 | batch 1590/2304 [I 2022-10-27 20:51:03] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:43 [I 2022-10-27 20:51:03] ...trap/engines/engine.py.229: process: 0.56588 | load: 0.00255 [I 2022-10-27 20:51:03] ...trap/engines/engine.py.230: loss: 0.10909 [I 2022-10-27 20:51:03] ...optimizers/trijoint.py.151: total_norm: 8.440063 [I 2022-10-27 20:51:09] ...trap/engines/engine.py.224: train: epoch 87 | batch 1600/2304 [I 2022-10-27 20:51:09] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:33 [I 2022-10-27 20:51:09] ...trap/engines/engine.py.229: process: 0.54446 | load: 0.00254 [I 2022-10-27 20:51:09] ...trap/engines/engine.py.230: loss: 0.13876 [I 2022-10-27 20:51:09] ...optimizers/trijoint.py.151: total_norm: 9.867377 [I 2022-10-27 20:51:14] ...trap/engines/engine.py.224: train: epoch 87 | batch 1610/2304 [I 2022-10-27 20:51:14] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:27 [I 2022-10-27 20:51:14] ...trap/engines/engine.py.229: process: 0.56261 | load: 0.00266 [I 2022-10-27 20:51:14] ...trap/engines/engine.py.230: loss: 0.12881 [I 2022-10-27 20:51:14] ...optimizers/trijoint.py.151: total_norm: 26.440979 [I 2022-10-27 20:51:20] ...trap/engines/engine.py.224: train: epoch 87 | batch 1620/2304 [I 2022-10-27 20:51:20] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:21 [I 2022-10-27 20:51:20] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00260 [I 2022-10-27 20:51:20] ...trap/engines/engine.py.230: loss: 0.14063 [I 2022-10-27 20:51:20] ...optimizers/trijoint.py.151: total_norm: 16.670364 [I 2022-10-27 20:51:25] ...trap/engines/engine.py.224: train: epoch 87 | batch 1630/2304 [I 2022-10-27 20:51:25] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:06:15 [I 2022-10-27 20:51:25] ...trap/engines/engine.py.229: process: 0.55611 | load: 0.00262 [I 2022-10-27 20:51:25] ...trap/engines/engine.py.230: loss: 0.09768 [I 2022-10-27 20:51:25] ...optimizers/trijoint.py.151: total_norm: 14.390645 [I 2022-10-27 20:51:31] ...trap/engines/engine.py.224: train: epoch 87 | batch 1640/2304 [I 2022-10-27 20:51:31] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:06:13 [I 2022-10-27 20:51:31] ...trap/engines/engine.py.229: process: 0.56577 | load: 0.00259 [I 2022-10-27 20:51:31] ...trap/engines/engine.py.230: loss: 0.17551 [I 2022-10-27 20:51:31] ...optimizers/trijoint.py.151: total_norm: 14.142315 [I 2022-10-27 20:51:37] ...trap/engines/engine.py.224: train: epoch 87 | batch 1650/2304 [I 2022-10-27 20:51:37] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:06:06 [I 2022-10-27 20:51:37] ...trap/engines/engine.py.229: process: 0.56340 | load: 0.00254 [I 2022-10-27 20:51:37] ...trap/engines/engine.py.230: loss: 0.10589 [I 2022-10-27 20:51:37] ...optimizers/trijoint.py.151: total_norm: 13.222636 [I 2022-10-27 20:51:42] ...trap/engines/engine.py.224: train: epoch 87 | batch 1660/2304 [I 2022-10-27 20:51:42] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:06:01 [I 2022-10-27 20:51:42] ...trap/engines/engine.py.229: process: 0.55312 | load: 0.00257 [I 2022-10-27 20:51:42] ...trap/engines/engine.py.230: loss: 0.09355 [I 2022-10-27 20:51:42] ...optimizers/trijoint.py.151: total_norm: 11.246465 [I 2022-10-27 20:51:48] ...trap/engines/engine.py.224: train: epoch 87 | batch 1670/2304 [I 2022-10-27 20:51:48] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:57 [I 2022-10-27 20:51:48] ...trap/engines/engine.py.229: process: 0.57226 | load: 0.00258 [I 2022-10-27 20:51:48] ...trap/engines/engine.py.230: loss: 0.13346 [I 2022-10-27 20:51:48] ...optimizers/trijoint.py.151: total_norm: 27.169563 [I 2022-10-27 20:51:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 1680/2304 [I 2022-10-27 20:51:54] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:50 [I 2022-10-27 20:51:54] ...trap/engines/engine.py.229: process: 0.57969 | load: 0.00261 [I 2022-10-27 20:51:54] ...trap/engines/engine.py.230: loss: 0.10856 [I 2022-10-27 20:51:54] ...optimizers/trijoint.py.151: total_norm: 11.673889 [I 2022-10-27 20:51:59] ...trap/engines/engine.py.224: train: epoch 87 | batch 1690/2304 [I 2022-10-27 20:51:59] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:41 [I 2022-10-27 20:51:59] ...trap/engines/engine.py.229: process: 0.54526 | load: 0.00255 [I 2022-10-27 20:51:59] ...trap/engines/engine.py.230: loss: 0.14700 [I 2022-10-27 20:51:59] ...optimizers/trijoint.py.151: total_norm: 19.022234 [I 2022-10-27 20:52:05] ...trap/engines/engine.py.224: train: epoch 87 | batch 1700/2304 [I 2022-10-27 20:52:05] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:35 [I 2022-10-27 20:52:05] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00259 [I 2022-10-27 20:52:05] ...trap/engines/engine.py.230: loss: 0.10689 [I 2022-10-27 20:52:05] ...optimizers/trijoint.py.151: total_norm: 13.943065 [I 2022-10-27 20:52:10] ...trap/engines/engine.py.224: train: epoch 87 | batch 1710/2304 [I 2022-10-27 20:52:10] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:29 [I 2022-10-27 20:52:10] ...trap/engines/engine.py.229: process: 0.56226 | load: 0.00255 [I 2022-10-27 20:52:10] ...trap/engines/engine.py.230: loss: 0.11515 [I 2022-10-27 20:52:10] ...optimizers/trijoint.py.151: total_norm: 30.097334 [I 2022-10-27 20:52:16] ...trap/engines/engine.py.224: train: epoch 87 | batch 1720/2304 [I 2022-10-27 20:52:16] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:27 [I 2022-10-27 20:52:16] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00257 [I 2022-10-27 20:52:16] ...trap/engines/engine.py.230: loss: 0.15459 [I 2022-10-27 20:52:16] ...optimizers/trijoint.py.151: total_norm: 27.064470 [I 2022-10-27 20:52:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 1730/2304 [I 2022-10-27 20:52:22] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:05:23 [I 2022-10-27 20:52:22] ...trap/engines/engine.py.229: process: 0.56849 | load: 0.00256 [I 2022-10-27 20:52:22] ...trap/engines/engine.py.230: loss: 0.11668 [I 2022-10-27 20:52:22] ...optimizers/trijoint.py.151: total_norm: 11.395910 [I 2022-10-27 20:52:27] ...trap/engines/engine.py.224: train: epoch 87 | batch 1740/2304 [I 2022-10-27 20:52:27] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:05:16 [I 2022-10-27 20:52:27] ...trap/engines/engine.py.229: process: 0.56968 | load: 0.00252 [I 2022-10-27 20:52:27] ...trap/engines/engine.py.230: loss: 0.14522 [I 2022-10-27 20:52:27] ...optimizers/trijoint.py.151: total_norm: 31.389318 [I 2022-10-27 20:52:33] ...trap/engines/engine.py.224: train: epoch 87 | batch 1750/2304 [I 2022-10-27 20:52:33] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:05:10 [I 2022-10-27 20:52:33] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00256 [I 2022-10-27 20:52:33] ...trap/engines/engine.py.230: loss: 0.16369 [I 2022-10-27 20:52:33] ...optimizers/trijoint.py.151: total_norm: 12.354445 [I 2022-10-27 20:52:38] ...trap/engines/engine.py.224: train: epoch 87 | batch 1760/2304 [I 2022-10-27 20:52:38] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:05:04 [I 2022-10-27 20:52:38] ...trap/engines/engine.py.229: process: 0.55724 | load: 0.00254 [I 2022-10-27 20:52:38] ...trap/engines/engine.py.230: loss: 0.09923 [I 2022-10-27 20:52:38] ...optimizers/trijoint.py.151: total_norm: 10.805311 [I 2022-10-27 20:52:44] ...trap/engines/engine.py.224: train: epoch 87 | batch 1770/2304 [I 2022-10-27 20:52:44] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:05:00 [I 2022-10-27 20:52:44] ...trap/engines/engine.py.229: process: 0.57118 | load: 0.00260 [I 2022-10-27 20:52:44] ...trap/engines/engine.py.230: loss: 0.10773 [I 2022-10-27 20:52:44] ...optimizers/trijoint.py.151: total_norm: 10.773149 [I 2022-10-27 20:52:50] ...trap/engines/engine.py.224: train: epoch 87 | batch 1780/2304 [I 2022-10-27 20:52:50] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:52 [I 2022-10-27 20:52:50] ...trap/engines/engine.py.229: process: 0.55485 | load: 0.00255 [I 2022-10-27 20:52:50] ...trap/engines/engine.py.230: loss: 0.17552 [I 2022-10-27 20:52:50] ...optimizers/trijoint.py.151: total_norm: 42.243237 [I 2022-10-27 20:52:55] ...trap/engines/engine.py.224: train: epoch 87 | batch 1790/2304 [I 2022-10-27 20:52:55] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:50 [I 2022-10-27 20:52:55] ...trap/engines/engine.py.229: process: 0.56838 | load: 0.00253 [I 2022-10-27 20:52:55] ...trap/engines/engine.py.230: loss: 0.20041 [I 2022-10-27 20:52:55] ...optimizers/trijoint.py.151: total_norm: 25.027102 [I 2022-10-27 20:53:01] ...trap/engines/engine.py.224: train: epoch 87 | batch 1800/2304 [I 2022-10-27 20:53:01] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:43 [I 2022-10-27 20:53:01] ...trap/engines/engine.py.229: process: 0.57819 | load: 0.00315 [I 2022-10-27 20:53:01] ...trap/engines/engine.py.230: loss: 0.11310 [I 2022-10-27 20:53:01] ...optimizers/trijoint.py.151: total_norm: 13.118431 [I 2022-10-27 20:53:06] ...trap/engines/engine.py.224: train: epoch 87 | batch 1810/2304 [I 2022-10-27 20:53:06] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:36 [I 2022-10-27 20:53:06] ...trap/engines/engine.py.229: process: 0.56327 | load: 0.00246 [I 2022-10-27 20:53:06] ...trap/engines/engine.py.230: loss: 0.14930 [I 2022-10-27 20:53:06] ...optimizers/trijoint.py.151: total_norm: 19.572985 [I 2022-10-27 20:53:12] ...trap/engines/engine.py.224: train: epoch 87 | batch 1820/2304 [I 2022-10-27 20:53:12] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:31 [I 2022-10-27 20:53:12] ...trap/engines/engine.py.229: process: 0.55999 | load: 0.00244 [I 2022-10-27 20:53:12] ...trap/engines/engine.py.230: loss: 0.12005 [I 2022-10-27 20:53:12] ...optimizers/trijoint.py.151: total_norm: 15.917275 [I 2022-10-27 20:53:18] ...trap/engines/engine.py.224: train: epoch 87 | batch 1830/2304 [I 2022-10-27 20:53:18] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:24 [I 2022-10-27 20:53:18] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00245 [I 2022-10-27 20:53:18] ...trap/engines/engine.py.230: loss: 0.16621 [I 2022-10-27 20:53:18] ...optimizers/trijoint.py.151: total_norm: 26.032877 [I 2022-10-27 20:53:23] ...trap/engines/engine.py.224: train: epoch 87 | batch 1840/2304 [I 2022-10-27 20:53:23] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:19 [I 2022-10-27 20:53:23] ...trap/engines/engine.py.229: process: 0.56256 | load: 0.00244 [I 2022-10-27 20:53:23] ...trap/engines/engine.py.230: loss: 0.09930 [I 2022-10-27 20:53:23] ...optimizers/trijoint.py.151: total_norm: 11.535526 [I 2022-10-27 20:53:29] ...trap/engines/engine.py.224: train: epoch 87 | batch 1850/2304 [I 2022-10-27 20:53:29] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:04:13 [I 2022-10-27 20:53:29] ...trap/engines/engine.py.229: process: 0.56138 | load: 0.00241 [I 2022-10-27 20:53:29] ...trap/engines/engine.py.230: loss: 0.09467 [I 2022-10-27 20:53:29] ...optimizers/trijoint.py.151: total_norm: 7.806684 [I 2022-10-27 20:53:34] ...trap/engines/engine.py.224: train: epoch 87 | batch 1860/2304 [I 2022-10-27 20:53:34] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:04:07 [I 2022-10-27 20:53:34] ...trap/engines/engine.py.229: process: 0.54830 | load: 0.00244 [I 2022-10-27 20:53:34] ...trap/engines/engine.py.230: loss: 0.13081 [I 2022-10-27 20:53:34] ...optimizers/trijoint.py.151: total_norm: 26.073177 [I 2022-10-27 20:53:40] ...trap/engines/engine.py.224: train: epoch 87 | batch 1870/2304 [I 2022-10-27 20:53:40] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:04:02 [I 2022-10-27 20:53:40] ...trap/engines/engine.py.229: process: 0.56024 | load: 0.00243 [I 2022-10-27 20:53:40] ...trap/engines/engine.py.230: loss: 0.10889 [I 2022-10-27 20:53:40] ...optimizers/trijoint.py.151: total_norm: 11.432356 [I 2022-10-27 20:53:46] ...trap/engines/engine.py.224: train: epoch 87 | batch 1880/2304 [I 2022-10-27 20:53:46] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:58 [I 2022-10-27 20:53:46] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00243 [I 2022-10-27 20:53:46] ...trap/engines/engine.py.230: loss: 0.12902 [I 2022-10-27 20:53:46] ...optimizers/trijoint.py.151: total_norm: 23.161390 [I 2022-10-27 20:53:51] ...trap/engines/engine.py.224: train: epoch 87 | batch 1890/2304 [I 2022-10-27 20:53:51] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:51 [I 2022-10-27 20:53:51] ...trap/engines/engine.py.229: process: 0.56115 | load: 0.00243 [I 2022-10-27 20:53:51] ...trap/engines/engine.py.230: loss: 0.18152 [I 2022-10-27 20:53:51] ...optimizers/trijoint.py.151: total_norm: 20.533888 [I 2022-10-27 20:53:57] ...trap/engines/engine.py.224: train: epoch 87 | batch 1900/2304 [I 2022-10-27 20:53:57] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:44 [I 2022-10-27 20:53:57] ...trap/engines/engine.py.229: process: 0.54258 | load: 0.00256 [I 2022-10-27 20:53:57] ...trap/engines/engine.py.230: loss: 0.17345 [I 2022-10-27 20:53:57] ...optimizers/trijoint.py.151: total_norm: 20.219746 [I 2022-10-27 20:54:02] ...trap/engines/engine.py.224: train: epoch 87 | batch 1910/2304 [I 2022-10-27 20:54:02] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:41 [I 2022-10-27 20:54:02] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00253 [I 2022-10-27 20:54:02] ...trap/engines/engine.py.230: loss: 0.11111 [I 2022-10-27 20:54:02] ...optimizers/trijoint.py.151: total_norm: 18.612907 [I 2022-10-27 20:54:08] ...trap/engines/engine.py.224: train: epoch 87 | batch 1920/2304 [I 2022-10-27 20:54:08] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:34 [I 2022-10-27 20:54:08] ...trap/engines/engine.py.229: process: 0.55189 | load: 0.00247 [I 2022-10-27 20:54:08] ...trap/engines/engine.py.230: loss: 0.11579 [I 2022-10-27 20:54:08] ...optimizers/trijoint.py.151: total_norm: 17.114967 [I 2022-10-27 20:54:14] ...trap/engines/engine.py.224: train: epoch 87 | batch 1930/2304 [I 2022-10-27 20:54:14] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:30 [I 2022-10-27 20:54:14] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00246 [I 2022-10-27 20:54:14] ...trap/engines/engine.py.230: loss: 0.16179 [I 2022-10-27 20:54:14] ...optimizers/trijoint.py.151: total_norm: 19.279158 [I 2022-10-27 20:54:19] ...trap/engines/engine.py.224: train: epoch 87 | batch 1940/2304 [I 2022-10-27 20:54:19] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:24 [I 2022-10-27 20:54:19] ...trap/engines/engine.py.229: process: 0.57144 | load: 0.00250 [I 2022-10-27 20:54:19] ...trap/engines/engine.py.230: loss: 0.16175 [I 2022-10-27 20:54:19] ...optimizers/trijoint.py.151: total_norm: 14.506753 [I 2022-10-27 20:54:25] ...trap/engines/engine.py.224: train: epoch 87 | batch 1950/2304 [I 2022-10-27 20:54:25] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:03:17 [I 2022-10-27 20:54:25] ...trap/engines/engine.py.229: process: 0.54889 | load: 0.00245 [I 2022-10-27 20:54:25] ...trap/engines/engine.py.230: loss: 0.15241 [I 2022-10-27 20:54:25] ...optimizers/trijoint.py.151: total_norm: 20.706875 [I 2022-10-27 20:54:30] ...trap/engines/engine.py.224: train: epoch 87 | batch 1960/2304 [I 2022-10-27 20:54:30] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:03:11 [I 2022-10-27 20:54:30] ...trap/engines/engine.py.229: process: 0.54865 | load: 0.00243 [I 2022-10-27 20:54:30] ...trap/engines/engine.py.230: loss: 0.14923 [I 2022-10-27 20:54:30] ...optimizers/trijoint.py.151: total_norm: 10.195211 [I 2022-10-27 20:54:36] ...trap/engines/engine.py.224: train: epoch 87 | batch 1970/2304 [I 2022-10-27 20:54:36] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:03:05 [I 2022-10-27 20:54:36] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00243 [I 2022-10-27 20:54:36] ...trap/engines/engine.py.230: loss: 0.11581 [I 2022-10-27 20:54:36] ...optimizers/trijoint.py.151: total_norm: 11.902706 [I 2022-10-27 20:54:42] ...trap/engines/engine.py.224: train: epoch 87 | batch 1980/2304 [I 2022-10-27 20:54:42] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:03:01 [I 2022-10-27 20:54:42] ...trap/engines/engine.py.229: process: 0.57061 | load: 0.00243 [I 2022-10-27 20:54:42] ...trap/engines/engine.py.230: loss: 0.15608 [I 2022-10-27 20:54:42] ...optimizers/trijoint.py.151: total_norm: 15.837532 [I 2022-10-27 20:54:47] ...trap/engines/engine.py.224: train: epoch 87 | batch 1990/2304 [I 2022-10-27 20:54:47] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:55 [I 2022-10-27 20:54:47] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00242 [I 2022-10-27 20:54:47] ...trap/engines/engine.py.230: loss: 0.11411 [I 2022-10-27 20:54:47] ...optimizers/trijoint.py.151: total_norm: 16.172295 [I 2022-10-27 20:54:53] ...trap/engines/engine.py.224: train: epoch 87 | batch 2000/2304 [I 2022-10-27 20:54:53] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:48 [I 2022-10-27 20:54:53] ...trap/engines/engine.py.229: process: 0.54152 | load: 0.00241 [I 2022-10-27 20:54:53] ...trap/engines/engine.py.230: loss: 0.12575 [I 2022-10-27 20:54:53] ...optimizers/trijoint.py.151: total_norm: 19.060156 [I 2022-10-27 20:54:58] ...trap/engines/engine.py.224: train: epoch 87 | batch 2010/2304 [I 2022-10-27 20:54:58] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:44 [I 2022-10-27 20:54:58] ...trap/engines/engine.py.229: process: 0.56407 | load: 0.00241 [I 2022-10-27 20:54:58] ...trap/engines/engine.py.230: loss: 0.16038 [I 2022-10-27 20:54:58] ...optimizers/trijoint.py.151: total_norm: 9.946543 [I 2022-10-27 20:55:04] ...trap/engines/engine.py.224: train: epoch 87 | batch 2020/2304 [I 2022-10-27 20:55:04] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:37 [I 2022-10-27 20:55:04] ...trap/engines/engine.py.229: process: 0.53848 | load: 0.00241 [I 2022-10-27 20:55:04] ...trap/engines/engine.py.230: loss: 0.12952 [I 2022-10-27 20:55:04] ...optimizers/trijoint.py.151: total_norm: 7.509332 [I 2022-10-27 20:55:10] ...trap/engines/engine.py.224: train: epoch 87 | batch 2030/2304 [I 2022-10-27 20:55:10] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:31 [I 2022-10-27 20:55:10] ...trap/engines/engine.py.229: process: 0.55726 | load: 0.00245 [I 2022-10-27 20:55:10] ...trap/engines/engine.py.230: loss: 0.13981 [I 2022-10-27 20:55:10] ...optimizers/trijoint.py.151: total_norm: 19.358644 [I 2022-10-27 20:55:15] ...trap/engines/engine.py.224: train: epoch 87 | batch 2040/2304 [I 2022-10-27 20:55:15] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:27 [I 2022-10-27 20:55:15] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00241 [I 2022-10-27 20:55:15] ...trap/engines/engine.py.230: loss: 0.13476 [I 2022-10-27 20:55:15] ...optimizers/trijoint.py.151: total_norm: 21.467318 [I 2022-10-27 20:55:21] ...trap/engines/engine.py.224: train: epoch 87 | batch 2050/2304 [I 2022-10-27 20:55:21] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:02:22 [I 2022-10-27 20:55:21] ...trap/engines/engine.py.229: process: 0.54910 | load: 0.00240 [I 2022-10-27 20:55:21] ...trap/engines/engine.py.230: loss: 0.12865 [I 2022-10-27 20:55:21] ...optimizers/trijoint.py.151: total_norm: 21.343809 [I 2022-10-27 20:55:26] ...trap/engines/engine.py.224: train: epoch 87 | batch 2060/2304 [I 2022-10-27 20:55:26] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:15 [I 2022-10-27 20:55:26] ...trap/engines/engine.py.229: process: 0.54893 | load: 0.00244 [I 2022-10-27 20:55:26] ...trap/engines/engine.py.230: loss: 0.13488 [I 2022-10-27 20:55:26] ...optimizers/trijoint.py.151: total_norm: 9.314306 [I 2022-10-27 20:55:32] ...trap/engines/engine.py.224: train: epoch 87 | batch 2070/2304 [I 2022-10-27 20:55:32] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:02:11 [I 2022-10-27 20:55:32] ...trap/engines/engine.py.229: process: 0.56970 | load: 0.00250 [I 2022-10-27 20:55:32] ...trap/engines/engine.py.230: loss: 0.10487 [I 2022-10-27 20:55:32] ...optimizers/trijoint.py.151: total_norm: 16.690088 [I 2022-10-27 20:55:37] ...trap/engines/engine.py.224: train: epoch 87 | batch 2080/2304 [I 2022-10-27 20:55:37] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:02:03 [I 2022-10-27 20:55:37] ...trap/engines/engine.py.229: process: 0.55410 | load: 0.00248 [I 2022-10-27 20:55:37] ...trap/engines/engine.py.230: loss: 0.11258 [I 2022-10-27 20:55:37] ...optimizers/trijoint.py.151: total_norm: 15.110513 [I 2022-10-27 20:55:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 2090/2304 [I 2022-10-27 20:55:43] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:59 [I 2022-10-27 20:55:43] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00246 [I 2022-10-27 20:55:43] ...trap/engines/engine.py.230: loss: 0.13393 [I 2022-10-27 20:55:43] ...optimizers/trijoint.py.151: total_norm: 16.443300 [I 2022-10-27 20:55:49] ...trap/engines/engine.py.224: train: epoch 87 | batch 2100/2304 [I 2022-10-27 20:55:49] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:54 [I 2022-10-27 20:55:49] ...trap/engines/engine.py.229: process: 0.56165 | load: 0.00289 [I 2022-10-27 20:55:49] ...trap/engines/engine.py.230: loss: 0.13154 [I 2022-10-27 20:55:49] ...optimizers/trijoint.py.151: total_norm: 22.384693 [I 2022-10-27 20:55:54] ...trap/engines/engine.py.224: train: epoch 87 | batch 2110/2304 [I 2022-10-27 20:55:54] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:48 [I 2022-10-27 20:55:54] ...trap/engines/engine.py.229: process: 0.55268 | load: 0.00255 [I 2022-10-27 20:55:54] ...trap/engines/engine.py.230: loss: 0.10696 [I 2022-10-27 20:55:54] ...optimizers/trijoint.py.151: total_norm: 19.633957 [I 2022-10-27 20:56:00] ...trap/engines/engine.py.224: train: epoch 87 | batch 2120/2304 [I 2022-10-27 20:56:00] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:42 [I 2022-10-27 20:56:00] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00256 [I 2022-10-27 20:56:00] ...trap/engines/engine.py.230: loss: 0.16098 [I 2022-10-27 20:56:00] ...optimizers/trijoint.py.151: total_norm: 9.420501 [I 2022-10-27 20:56:06] ...trap/engines/engine.py.224: train: epoch 87 | batch 2130/2304 [I 2022-10-27 20:56:06] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:37 [I 2022-10-27 20:56:06] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00255 [I 2022-10-27 20:56:06] ...trap/engines/engine.py.230: loss: 0.11047 [I 2022-10-27 20:56:06] ...optimizers/trijoint.py.151: total_norm: 9.099570 [I 2022-10-27 20:56:11] ...trap/engines/engine.py.224: train: epoch 87 | batch 2140/2304 [I 2022-10-27 20:56:11] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:31 [I 2022-10-27 20:56:11] ...trap/engines/engine.py.229: process: 0.54959 | load: 0.00256 [I 2022-10-27 20:56:11] ...trap/engines/engine.py.230: loss: 0.18816 [I 2022-10-27 20:56:11] ...optimizers/trijoint.py.151: total_norm: 13.301241 [I 2022-10-27 20:56:17] ...trap/engines/engine.py.224: train: epoch 87 | batch 2150/2304 [I 2022-10-27 20:56:17] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:27 [I 2022-10-27 20:56:17] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00261 [I 2022-10-27 20:56:17] ...trap/engines/engine.py.230: loss: 0.12884 [I 2022-10-27 20:56:17] ...optimizers/trijoint.py.151: total_norm: 22.726118 [I 2022-10-27 20:56:22] ...trap/engines/engine.py.224: train: epoch 87 | batch 2160/2304 [I 2022-10-27 20:56:22] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:01:21 [I 2022-10-27 20:56:22] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00258 [I 2022-10-27 20:56:22] ...trap/engines/engine.py.230: loss: 0.12572 [I 2022-10-27 20:56:22] ...optimizers/trijoint.py.151: total_norm: 24.795687 [I 2022-10-27 20:56:28] ...trap/engines/engine.py.224: train: epoch 87 | batch 2170/2304 [I 2022-10-27 20:56:28] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:16 [I 2022-10-27 20:56:28] ...trap/engines/engine.py.229: process: 0.56280 | load: 0.00262 [I 2022-10-27 20:56:28] ...trap/engines/engine.py.230: loss: 0.08433 [I 2022-10-27 20:56:28] ...optimizers/trijoint.py.151: total_norm: 23.534142 [I 2022-10-27 20:56:34] ...trap/engines/engine.py.224: train: epoch 87 | batch 2180/2304 [I 2022-10-27 20:56:34] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:01:10 [I 2022-10-27 20:56:34] ...trap/engines/engine.py.229: process: 0.56656 | load: 0.00257 [I 2022-10-27 20:56:34] ...trap/engines/engine.py.230: loss: 0.09899 [I 2022-10-27 20:56:34] ...optimizers/trijoint.py.151: total_norm: 28.998409 [I 2022-10-27 20:56:40] ...trap/engines/engine.py.224: train: epoch 87 | batch 2190/2304 [I 2022-10-27 20:56:40] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:01:04 [I 2022-10-27 20:56:40] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00252 [I 2022-10-27 20:56:40] ...trap/engines/engine.py.230: loss: 0.15210 [I 2022-10-27 20:56:40] ...optimizers/trijoint.py.151: total_norm: 18.376503 [I 2022-10-27 20:56:45] ...trap/engines/engine.py.224: train: epoch 87 | batch 2200/2304 [I 2022-10-27 20:56:45] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:58 [I 2022-10-27 20:56:45] ...trap/engines/engine.py.229: process: 0.56338 | load: 0.00249 [I 2022-10-27 20:56:45] ...trap/engines/engine.py.230: loss: 0.14824 [I 2022-10-27 20:56:45] ...optimizers/trijoint.py.151: total_norm: 11.476945 [I 2022-10-27 20:56:51] ...trap/engines/engine.py.224: train: epoch 87 | batch 2210/2304 [I 2022-10-27 20:56:51] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:52 [I 2022-10-27 20:56:51] ...trap/engines/engine.py.229: process: 0.56194 | load: 0.00255 [I 2022-10-27 20:56:51] ...trap/engines/engine.py.230: loss: 0.14676 [I 2022-10-27 20:56:51] ...optimizers/trijoint.py.151: total_norm: 11.289027 [I 2022-10-27 20:56:56] ...trap/engines/engine.py.224: train: epoch 87 | batch 2220/2304 [I 2022-10-27 20:56:56] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:46 [I 2022-10-27 20:56:56] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00247 [I 2022-10-27 20:56:56] ...trap/engines/engine.py.230: loss: 0.16016 [I 2022-10-27 20:56:56] ...optimizers/trijoint.py.151: total_norm: 22.554350 [I 2022-10-27 20:57:02] ...trap/engines/engine.py.224: train: epoch 87 | batch 2230/2304 [I 2022-10-27 20:57:02] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:41 [I 2022-10-27 20:57:02] ...trap/engines/engine.py.229: process: 0.55602 | load: 0.00243 [I 2022-10-27 20:57:02] ...trap/engines/engine.py.230: loss: 0.25800 [I 2022-10-27 20:57:02] ...optimizers/trijoint.py.151: total_norm: 18.783285 [I 2022-10-27 20:57:07] ...trap/engines/engine.py.224: train: epoch 87 | batch 2240/2304 [I 2022-10-27 20:57:07] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:35 [I 2022-10-27 20:57:07] ...trap/engines/engine.py.229: process: 0.54641 | load: 0.00248 [I 2022-10-27 20:57:07] ...trap/engines/engine.py.230: loss: 0.10371 [I 2022-10-27 20:57:07] ...optimizers/trijoint.py.151: total_norm: 12.690610 [I 2022-10-27 20:57:13] ...trap/engines/engine.py.224: train: epoch 87 | batch 2250/2304 [I 2022-10-27 20:57:13] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:30 [I 2022-10-27 20:57:13] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00253 [I 2022-10-27 20:57:13] ...trap/engines/engine.py.230: loss: 0.10702 [I 2022-10-27 20:57:13] ...optimizers/trijoint.py.151: total_norm: 13.814880 [I 2022-10-27 20:57:19] ...trap/engines/engine.py.224: train: epoch 87 | batch 2260/2304 [I 2022-10-27 20:57:19] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:24 [I 2022-10-27 20:57:19] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00247 [I 2022-10-27 20:57:19] ...trap/engines/engine.py.230: loss: 0.12925 [I 2022-10-27 20:57:19] ...optimizers/trijoint.py.151: total_norm: 27.793621 [I 2022-10-27 20:57:24] ...trap/engines/engine.py.224: train: epoch 87 | batch 2270/2304 [I 2022-10-27 20:57:24] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:18 [I 2022-10-27 20:57:24] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00250 [I 2022-10-27 20:57:24] ...trap/engines/engine.py.230: loss: 0.08901 [I 2022-10-27 20:57:24] ...optimizers/trijoint.py.151: total_norm: 18.882698 [I 2022-10-27 20:57:30] ...trap/engines/engine.py.224: train: epoch 87 | batch 2280/2304 [I 2022-10-27 20:57:30] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:00:13 [I 2022-10-27 20:57:30] ...trap/engines/engine.py.229: process: 0.55317 | load: 0.00240 [I 2022-10-27 20:57:30] ...trap/engines/engine.py.230: loss: 0.09542 [I 2022-10-27 20:57:30] ...optimizers/trijoint.py.151: total_norm: 7.831693 [I 2022-10-27 20:57:35] ...trap/engines/engine.py.224: train: epoch 87 | batch 2290/2304 [I 2022-10-27 20:57:35] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:07 [I 2022-10-27 20:57:35] ...trap/engines/engine.py.229: process: 0.55263 | load: 0.00245 [I 2022-10-27 20:57:35] ...trap/engines/engine.py.230: loss: 0.12418 [I 2022-10-27 20:57:35] ...optimizers/trijoint.py.151: total_norm: 13.141957 [I 2022-10-27 20:57:41] ...trap/engines/engine.py.224: train: epoch 87 | batch 2300/2304 [I 2022-10-27 20:57:41] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:00:02 [I 2022-10-27 20:57:41] ...trap/engines/engine.py.229: process: 0.54233 | load: 0.00016 [I 2022-10-27 20:57:41] ...trap/engines/engine.py.230: loss: 0.11263 [I 2022-10-27 20:57:41] ...optimizers/trijoint.py.151: total_norm: 19.493277 [I 2022-10-27 20:57:43] ...trap/engines/engine.py.224: train: epoch 87 | batch 2304/2304 [I 2022-10-27 20:57:43] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:00:00 [I 2022-10-27 20:57:43] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00018 [I 2022-10-27 20:57:43] ...trap/engines/engine.py.230: loss: 0.18254 [I 2022-10-27 20:57:43] ...optimizers/trijoint.py.151: total_norm: 20.930868 [S 2022-10-27 20:57:44] ...trap/engines/engine.py.129: train_epoch.epoch: 87 [S 2022-10-27 20:57:44] ...trap/engines/engine.py.129: train_epoch.loss: 0.13283648286297625 [I 2022-10-27 20:57:44] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 20:57:46] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 87 [I 2022-10-27 20:57:46] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 20:57:48] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 20:57:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 20:57:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 20:57:51] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 20:57:51] ...trap/engines/engine.py.321: eval: epoch 87 | batch 0/510 [I 2022-10-27 20:57:51] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:44:12 [I 2022-10-27 20:57:51] ...trap/engines/engine.py.326: process: 5.20171 | load: 3.88554 [I 2022-10-27 20:57:54] ...trap/engines/engine.py.321: eval: epoch 87 | batch 10/510 [I 2022-10-27 20:57:54] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:16 [I 2022-10-27 20:57:54] ...trap/engines/engine.py.326: process: 0.38239 | load: 0.00019 [I 2022-10-27 20:57:58] ...trap/engines/engine.py.321: eval: epoch 87 | batch 20/510 [I 2022-10-27 20:57:58] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:11 [I 2022-10-27 20:57:58] ...trap/engines/engine.py.326: process: 0.33566 | load: 0.00031 [I 2022-10-27 20:58:01] ...trap/engines/engine.py.321: eval: epoch 87 | batch 30/510 [I 2022-10-27 20:58:01] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:42 [I 2022-10-27 20:58:01] ...trap/engines/engine.py.326: process: 0.32945 | load: 0.00018 [I 2022-10-27 20:58:04] ...trap/engines/engine.py.321: eval: epoch 87 | batch 40/510 [I 2022-10-27 20:58:04] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:38 [I 2022-10-27 20:58:04] ...trap/engines/engine.py.326: process: 0.36020 | load: 0.00039 [I 2022-10-27 20:58:08] ...trap/engines/engine.py.321: eval: epoch 87 | batch 50/510 [I 2022-10-27 20:58:08] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:34 [I 2022-10-27 20:58:08] ...trap/engines/engine.py.326: process: 0.33543 | load: 0.00027 [I 2022-10-27 20:58:11] ...trap/engines/engine.py.321: eval: epoch 87 | batch 60/510 [I 2022-10-27 20:58:11] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:30 [I 2022-10-27 20:58:11] ...trap/engines/engine.py.326: process: 0.33523 | load: 0.00047 [I 2022-10-27 20:58:14] ...trap/engines/engine.py.321: eval: epoch 87 | batch 70/510 [I 2022-10-27 20:58:14] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:29 [I 2022-10-27 20:58:14] ...trap/engines/engine.py.326: process: 0.35317 | load: 0.00017 [I 2022-10-27 20:58:18] ...trap/engines/engine.py.321: eval: epoch 87 | batch 80/510 [I 2022-10-27 20:58:18] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:26 [I 2022-10-27 20:58:18] ...trap/engines/engine.py.326: process: 0.35964 | load: 0.00020 [I 2022-10-27 20:58:21] ...trap/engines/engine.py.321: eval: epoch 87 | batch 90/510 [I 2022-10-27 20:58:21] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:21 [I 2022-10-27 20:58:21] ...trap/engines/engine.py.326: process: 0.33267 | load: 0.00018 [I 2022-10-27 20:58:25] ...trap/engines/engine.py.321: eval: epoch 87 | batch 100/510 [I 2022-10-27 20:58:25] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:14 [I 2022-10-27 20:58:25] ...trap/engines/engine.py.326: process: 0.29274 | load: 0.00019 [I 2022-10-27 20:58:27] ...trap/engines/engine.py.321: eval: epoch 87 | batch 110/510 [I 2022-10-27 20:58:27] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-27 20:58:27] ...trap/engines/engine.py.326: process: 0.28686 | load: 0.00015 [I 2022-10-27 20:58:30] ...trap/engines/engine.py.321: eval: epoch 87 | batch 120/510 [I 2022-10-27 20:58:30] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 20:58:30] ...trap/engines/engine.py.326: process: 0.29012 | load: 0.00020 [I 2022-10-27 20:58:33] ...trap/engines/engine.py.321: eval: epoch 87 | batch 130/510 [I 2022-10-27 20:58:33] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:48 [I 2022-10-27 20:58:33] ...trap/engines/engine.py.326: process: 0.28146 | load: 0.00016 [I 2022-10-27 20:58:36] ...trap/engines/engine.py.321: eval: epoch 87 | batch 140/510 [I 2022-10-27 20:58:36] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:45 [I 2022-10-27 20:58:36] ...trap/engines/engine.py.326: process: 0.29104 | load: 0.00017 [I 2022-10-27 20:58:39] ...trap/engines/engine.py.321: eval: epoch 87 | batch 150/510 [I 2022-10-27 20:58:39] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-27 20:58:39] ...trap/engines/engine.py.326: process: 0.28595 | load: 0.00016 [I 2022-10-27 20:58:42] ...trap/engines/engine.py.321: eval: epoch 87 | batch 160/510 [I 2022-10-27 20:58:42] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-27 20:58:42] ...trap/engines/engine.py.326: process: 0.29027 | load: 0.00027 [I 2022-10-27 20:58:45] ...trap/engines/engine.py.321: eval: epoch 87 | batch 170/510 [I 2022-10-27 20:58:45] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 20:58:45] ...trap/engines/engine.py.326: process: 0.28505 | load: 0.00016 [I 2022-10-27 20:58:48] ...trap/engines/engine.py.321: eval: epoch 87 | batch 180/510 [I 2022-10-27 20:58:48] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:36 [I 2022-10-27 20:58:48] ...trap/engines/engine.py.326: process: 0.29418 | load: 0.00017 [I 2022-10-27 20:58:51] ...trap/engines/engine.py.321: eval: epoch 87 | batch 190/510 [I 2022-10-27 20:58:51] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:34 [I 2022-10-27 20:58:51] ...trap/engines/engine.py.326: process: 0.28925 | load: 0.00022 [I 2022-10-27 20:58:54] ...trap/engines/engine.py.321: eval: epoch 87 | batch 200/510 [I 2022-10-27 20:58:54] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:30 [I 2022-10-27 20:58:54] ...trap/engines/engine.py.326: process: 0.29096 | load: 0.00021 [I 2022-10-27 20:58:56] ...trap/engines/engine.py.321: eval: epoch 87 | batch 210/510 [I 2022-10-27 20:58:56] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 20:58:56] ...trap/engines/engine.py.326: process: 0.28625 | load: 0.00021 [I 2022-10-27 20:58:59] ...trap/engines/engine.py.321: eval: epoch 87 | batch 220/510 [I 2022-10-27 20:58:59] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:25 [I 2022-10-27 20:58:59] ...trap/engines/engine.py.326: process: 0.28896 | load: 0.00023 [I 2022-10-27 20:59:02] ...trap/engines/engine.py.321: eval: epoch 87 | batch 230/510 [I 2022-10-27 20:59:02] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 20:59:02] ...trap/engines/engine.py.326: process: 0.28772 | load: 0.00016 [I 2022-10-27 20:59:05] ...trap/engines/engine.py.321: eval: epoch 87 | batch 240/510 [I 2022-10-27 20:59:05] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-27 20:59:05] ...trap/engines/engine.py.326: process: 0.29474 | load: 0.00016 [I 2022-10-27 20:59:08] ...trap/engines/engine.py.321: eval: epoch 87 | batch 250/510 [I 2022-10-27 20:59:08] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-27 20:59:08] ...trap/engines/engine.py.326: process: 0.29010 | load: 0.00015 [I 2022-10-27 20:59:11] ...trap/engines/engine.py.321: eval: epoch 87 | batch 260/510 [I 2022-10-27 20:59:11] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-27 20:59:11] ...trap/engines/engine.py.326: process: 0.29274 | load: 0.00021 [I 2022-10-27 20:59:14] ...trap/engines/engine.py.321: eval: epoch 87 | batch 270/510 [I 2022-10-27 20:59:14] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-27 20:59:14] ...trap/engines/engine.py.326: process: 0.28451 | load: 0.00016 [I 2022-10-27 20:59:17] ...trap/engines/engine.py.321: eval: epoch 87 | batch 280/510 [I 2022-10-27 20:59:17] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 20:59:17] ...trap/engines/engine.py.326: process: 0.29059 | load: 0.00017 [I 2022-10-27 20:59:20] ...trap/engines/engine.py.321: eval: epoch 87 | batch 290/510 [I 2022-10-27 20:59:20] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 20:59:20] ...trap/engines/engine.py.326: process: 0.28670 | load: 0.00031 [I 2022-10-27 20:59:23] ...trap/engines/engine.py.321: eval: epoch 87 | batch 300/510 [I 2022-10-27 20:59:23] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 20:59:23] ...trap/engines/engine.py.326: process: 0.27968 | load: 0.00023 [I 2022-10-27 20:59:25] ...trap/engines/engine.py.321: eval: epoch 87 | batch 310/510 [I 2022-10-27 20:59:25] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 20:59:25] ...trap/engines/engine.py.326: process: 0.28885 | load: 0.00016 [I 2022-10-27 20:59:28] ...trap/engines/engine.py.321: eval: epoch 87 | batch 320/510 [I 2022-10-27 20:59:28] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 20:59:28] ...trap/engines/engine.py.326: process: 0.29642 | load: 0.00020 [I 2022-10-27 20:59:31] ...trap/engines/engine.py.321: eval: epoch 87 | batch 330/510 [I 2022-10-27 20:59:31] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-27 20:59:31] ...trap/engines/engine.py.326: process: 0.28368 | load: 0.00021 [I 2022-10-27 20:59:34] ...trap/engines/engine.py.321: eval: epoch 87 | batch 340/510 [I 2022-10-27 20:59:34] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-27 20:59:34] ...trap/engines/engine.py.326: process: 0.28463 | load: 0.00020 [I 2022-10-27 20:59:37] ...trap/engines/engine.py.321: eval: epoch 87 | batch 350/510 [I 2022-10-27 20:59:37] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:45 [I 2022-10-27 20:59:37] ...trap/engines/engine.py.326: process: 0.28203 | load: 0.00017 [I 2022-10-27 20:59:40] ...trap/engines/engine.py.321: eval: epoch 87 | batch 360/510 [I 2022-10-27 20:59:40] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 20:59:40] ...trap/engines/engine.py.326: process: 0.28941 | load: 0.00020 [I 2022-10-27 20:59:43] ...trap/engines/engine.py.321: eval: epoch 87 | batch 370/510 [I 2022-10-27 20:59:43] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 20:59:43] ...trap/engines/engine.py.326: process: 0.28910 | load: 0.00020 [I 2022-10-27 20:59:46] ...trap/engines/engine.py.321: eval: epoch 87 | batch 380/510 [I 2022-10-27 20:59:46] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 20:59:46] ...trap/engines/engine.py.326: process: 0.29003 | load: 0.00016 [I 2022-10-27 20:59:49] ...trap/engines/engine.py.321: eval: epoch 87 | batch 390/510 [I 2022-10-27 20:59:49] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:35 [I 2022-10-27 20:59:49] ...trap/engines/engine.py.326: process: 0.30876 | load: 0.00018 [I 2022-10-27 20:59:51] ...trap/engines/engine.py.321: eval: epoch 87 | batch 400/510 [I 2022-10-27 20:59:51] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:32 [I 2022-10-27 20:59:51] ...trap/engines/engine.py.326: process: 0.28963 | load: 0.00021 [I 2022-10-27 20:59:54] ...trap/engines/engine.py.321: eval: epoch 87 | batch 410/510 [I 2022-10-27 20:59:54] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:29 [I 2022-10-27 20:59:54] ...trap/engines/engine.py.326: process: 0.29189 | load: 0.00018 [I 2022-10-27 20:59:57] ...trap/engines/engine.py.321: eval: epoch 87 | batch 420/510 [I 2022-10-27 20:59:57] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 20:59:57] ...trap/engines/engine.py.326: process: 0.28673 | load: 0.00021 [I 2022-10-27 21:00:00] ...trap/engines/engine.py.321: eval: epoch 87 | batch 430/510 [I 2022-10-27 21:00:00] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 21:00:00] ...trap/engines/engine.py.326: process: 0.28996 | load: 0.00021 [I 2022-10-27 21:00:03] ...trap/engines/engine.py.321: eval: epoch 87 | batch 440/510 [I 2022-10-27 21:00:03] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 21:00:03] ...trap/engines/engine.py.326: process: 0.29243 | load: 0.00021 [I 2022-10-27 21:00:06] ...trap/engines/engine.py.321: eval: epoch 87 | batch 450/510 [I 2022-10-27 21:00:06] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 21:00:06] ...trap/engines/engine.py.326: process: 0.28928 | load: 0.00024 [I 2022-10-27 21:00:09] ...trap/engines/engine.py.321: eval: epoch 87 | batch 460/510 [I 2022-10-27 21:00:09] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-27 21:00:09] ...trap/engines/engine.py.326: process: 0.28668 | load: 0.00016 [I 2022-10-27 21:00:12] ...trap/engines/engine.py.321: eval: epoch 87 | batch 470/510 [I 2022-10-27 21:00:12] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 21:00:12] ...trap/engines/engine.py.326: process: 0.29347 | load: 0.00021 [I 2022-10-27 21:00:15] ...trap/engines/engine.py.321: eval: epoch 87 | batch 480/510 [I 2022-10-27 21:00:15] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 21:00:15] ...trap/engines/engine.py.326: process: 0.28801 | load: 0.00019 [I 2022-10-27 21:00:18] ...trap/engines/engine.py.321: eval: epoch 87 | batch 490/510 [I 2022-10-27 21:00:18] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 21:00:18] ...trap/engines/engine.py.326: process: 0.28421 | load: 0.00016 [I 2022-10-27 21:00:20] ...trap/engines/engine.py.321: eval: epoch 87 | batch 500/510 [I 2022-10-27 21:00:20] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 21:00:20] ...trap/engines/engine.py.326: process: 0.29091 | load: 0.00020 [I 2022-10-27 21:00:23] ...trap/engines/engine.py.321: eval: epoch 87 | batch 510/510 [I 2022-10-27 21:00:23] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 21:00:23] ...trap/engines/engine.py.326: process: 0.28842 | load: 0.00013 [S 2022-10-27 21:00:24] ...trap/engines/engine.py.132: eval_epoch.epoch: 87 [I 2022-10-27 21:00:24] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.724500 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905300 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936000 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.724000 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907600 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934000 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015660 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007550 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005177 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015086 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008065 [S 2022-10-27 21:00:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006033 [I 2022-10-27 21:00:30] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 21:00:30] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 21:00:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 21:00:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 21:00:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 21:00:35] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 21:00:35] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 21:00:41] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 21:00:41] ...trap/engines/engine.py.163: Training model on trainset for epoch 88 [I 2022-10-27 21:00:41] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 21:00:44] ...trap/engines/engine.py.224: train: epoch 88 | batch 0/2304 [I 2022-10-27 21:00:44] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:47 [I 2022-10-27 21:00:44] ...trap/engines/engine.py.229: process: 2.72914 | load: 2.07565 [I 2022-10-27 21:00:44] ...trap/engines/engine.py.230: loss: 0.09128 [I 2022-10-27 21:00:44] ...optimizers/trijoint.py.151: total_norm: 10.756516 [I 2022-10-27 21:00:50] ...trap/engines/engine.py.224: train: epoch 88 | batch 10/2304 [I 2022-10-27 21:00:50] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:59 [I 2022-10-27 21:00:50] ...trap/engines/engine.py.229: process: 0.54334 | load: 0.00350 [I 2022-10-27 21:00:50] ...trap/engines/engine.py.230: loss: 0.08058 [I 2022-10-27 21:00:50] ...optimizers/trijoint.py.151: total_norm: 7.998112 [I 2022-10-27 21:00:55] ...trap/engines/engine.py.224: train: epoch 88 | batch 20/2304 [I 2022-10-27 21:00:55] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:05 [I 2022-10-27 21:00:55] ...trap/engines/engine.py.229: process: 0.54381 | load: 0.00304 [I 2022-10-27 21:00:55] ...trap/engines/engine.py.230: loss: 0.11876 [I 2022-10-27 21:00:55] ...optimizers/trijoint.py.151: total_norm: 17.226627 [I 2022-10-27 21:01:01] ...trap/engines/engine.py.224: train: epoch 88 | batch 30/2304 [I 2022-10-27 21:01:01] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:03 [I 2022-10-27 21:01:01] ...trap/engines/engine.py.229: process: 0.55269 | load: 0.00297 [I 2022-10-27 21:01:01] ...trap/engines/engine.py.230: loss: 0.14710 [I 2022-10-27 21:01:01] ...optimizers/trijoint.py.151: total_norm: 14.540867 [I 2022-10-27 21:01:06] ...trap/engines/engine.py.224: train: epoch 88 | batch 40/2304 [I 2022-10-27 21:01:06] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:50 [I 2022-10-27 21:01:06] ...trap/engines/engine.py.229: process: 0.54324 | load: 0.00334 [I 2022-10-27 21:01:06] ...trap/engines/engine.py.230: loss: 0.15439 [I 2022-10-27 21:01:06] ...optimizers/trijoint.py.151: total_norm: 40.868347 [I 2022-10-27 21:01:12] ...trap/engines/engine.py.224: train: epoch 88 | batch 50/2304 [I 2022-10-27 21:01:12] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:57 [I 2022-10-27 21:01:12] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00289 [I 2022-10-27 21:01:12] ...trap/engines/engine.py.230: loss: 0.15687 [I 2022-10-27 21:01:12] ...optimizers/trijoint.py.151: total_norm: 14.330999 [I 2022-10-27 21:01:18] ...trap/engines/engine.py.224: train: epoch 88 | batch 60/2304 [I 2022-10-27 21:01:18] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:50 [I 2022-10-27 21:01:18] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00299 [I 2022-10-27 21:01:18] ...trap/engines/engine.py.230: loss: 0.13561 [I 2022-10-27 21:01:18] ...optimizers/trijoint.py.151: total_norm: 27.670351 [I 2022-10-27 21:01:23] ...trap/engines/engine.py.224: train: epoch 88 | batch 70/2304 [I 2022-10-27 21:01:23] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:46 [I 2022-10-27 21:01:23] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00266 [I 2022-10-27 21:01:23] ...trap/engines/engine.py.230: loss: 0.11279 [I 2022-10-27 21:01:23] ...optimizers/trijoint.py.151: total_norm: 10.305191 [I 2022-10-27 21:01:29] ...trap/engines/engine.py.224: train: epoch 88 | batch 80/2304 [I 2022-10-27 21:01:29] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:27 [I 2022-10-27 21:01:29] ...trap/engines/engine.py.229: process: 0.54364 | load: 0.00276 [I 2022-10-27 21:01:29] ...trap/engines/engine.py.230: loss: 0.20061 [I 2022-10-27 21:01:29] ...optimizers/trijoint.py.151: total_norm: 23.012777 [I 2022-10-27 21:01:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 90/2304 [I 2022-10-27 21:01:34] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:57 [I 2022-10-27 21:01:34] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00240 [I 2022-10-27 21:01:34] ...trap/engines/engine.py.230: loss: 0.10167 [I 2022-10-27 21:01:34] ...optimizers/trijoint.py.151: total_norm: 12.630082 [I 2022-10-27 21:01:40] ...trap/engines/engine.py.224: train: epoch 88 | batch 100/2304 [I 2022-10-27 21:01:40] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:25 [I 2022-10-27 21:01:40] ...trap/engines/engine.py.229: process: 0.54510 | load: 0.00275 [I 2022-10-27 21:01:40] ...trap/engines/engine.py.230: loss: 0.12869 [I 2022-10-27 21:01:40] ...optimizers/trijoint.py.151: total_norm: 12.386351 [I 2022-10-27 21:01:46] ...trap/engines/engine.py.224: train: epoch 88 | batch 110/2304 [I 2022-10-27 21:01:46] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:19 [I 2022-10-27 21:01:46] ...trap/engines/engine.py.229: process: 0.55359 | load: 0.00285 [I 2022-10-27 21:01:46] ...trap/engines/engine.py.230: loss: 0.21896 [I 2022-10-27 21:01:46] ...optimizers/trijoint.py.151: total_norm: 26.468334 [I 2022-10-27 21:01:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 120/2304 [I 2022-10-27 21:01:51] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:13 [I 2022-10-27 21:01:51] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00277 [I 2022-10-27 21:01:51] ...trap/engines/engine.py.230: loss: 0.30811 [I 2022-10-27 21:01:51] ...optimizers/trijoint.py.151: total_norm: 17.177599 [I 2022-10-27 21:01:57] ...trap/engines/engine.py.224: train: epoch 88 | batch 130/2304 [I 2022-10-27 21:01:57] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:07 [I 2022-10-27 21:01:57] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00258 [I 2022-10-27 21:01:57] ...trap/engines/engine.py.230: loss: 0.10579 [I 2022-10-27 21:01:57] ...optimizers/trijoint.py.151: total_norm: 11.925627 [I 2022-10-27 21:02:02] ...trap/engines/engine.py.224: train: epoch 88 | batch 140/2304 [I 2022-10-27 21:02:02] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:08 [I 2022-10-27 21:02:02] ...trap/engines/engine.py.229: process: 0.56633 | load: 0.00258 [I 2022-10-27 21:02:02] ...trap/engines/engine.py.230: loss: 0.12686 [I 2022-10-27 21:02:02] ...optimizers/trijoint.py.151: total_norm: 15.533615 [I 2022-10-27 21:02:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 150/2304 [I 2022-10-27 21:02:08] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:57 [I 2022-10-27 21:02:08] ...trap/engines/engine.py.229: process: 0.54464 | load: 0.00268 [I 2022-10-27 21:02:08] ...trap/engines/engine.py.230: loss: 0.10578 [I 2022-10-27 21:02:08] ...optimizers/trijoint.py.151: total_norm: 13.801522 [I 2022-10-27 21:02:13] ...trap/engines/engine.py.224: train: epoch 88 | batch 160/2304 [I 2022-10-27 21:02:13] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:53 [I 2022-10-27 21:02:13] ...trap/engines/engine.py.229: process: 0.54449 | load: 0.00243 [I 2022-10-27 21:02:13] ...trap/engines/engine.py.230: loss: 0.11840 [I 2022-10-27 21:02:13] ...optimizers/trijoint.py.151: total_norm: 29.266422 [I 2022-10-27 21:02:19] ...trap/engines/engine.py.224: train: epoch 88 | batch 170/2304 [I 2022-10-27 21:02:19] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:52 [I 2022-10-27 21:02:19] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00242 [I 2022-10-27 21:02:19] ...trap/engines/engine.py.230: loss: 0.16739 [I 2022-10-27 21:02:19] ...optimizers/trijoint.py.151: total_norm: 20.615122 [I 2022-10-27 21:02:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 180/2304 [I 2022-10-27 21:02:25] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:34 [I 2022-10-27 21:02:25] ...trap/engines/engine.py.229: process: 0.55167 | load: 0.00242 [I 2022-10-27 21:02:25] ...trap/engines/engine.py.230: loss: 0.22836 [I 2022-10-27 21:02:25] ...optimizers/trijoint.py.151: total_norm: 16.147673 [I 2022-10-27 21:02:30] ...trap/engines/engine.py.224: train: epoch 88 | batch 190/2304 [I 2022-10-27 21:02:30] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:40 [I 2022-10-27 21:02:30] ...trap/engines/engine.py.229: process: 0.55609 | load: 0.00252 [I 2022-10-27 21:02:30] ...trap/engines/engine.py.230: loss: 0.11736 [I 2022-10-27 21:02:30] ...optimizers/trijoint.py.151: total_norm: 13.253337 [I 2022-10-27 21:02:36] ...trap/engines/engine.py.224: train: epoch 88 | batch 200/2304 [I 2022-10-27 21:02:36] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:31 [I 2022-10-27 21:02:36] ...trap/engines/engine.py.229: process: 0.54473 | load: 0.00247 [I 2022-10-27 21:02:36] ...trap/engines/engine.py.230: loss: 0.09598 [I 2022-10-27 21:02:36] ...optimizers/trijoint.py.151: total_norm: 12.019263 [I 2022-10-27 21:02:41] ...trap/engines/engine.py.224: train: epoch 88 | batch 210/2304 [I 2022-10-27 21:02:41] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:13 [I 2022-10-27 21:02:41] ...trap/engines/engine.py.229: process: 0.55229 | load: 0.00254 [I 2022-10-27 21:02:41] ...trap/engines/engine.py.230: loss: 0.16260 [I 2022-10-27 21:02:41] ...optimizers/trijoint.py.151: total_norm: 18.325010 [I 2022-10-27 21:02:47] ...trap/engines/engine.py.224: train: epoch 88 | batch 220/2304 [I 2022-10-27 21:02:47] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:21 [I 2022-10-27 21:02:47] ...trap/engines/engine.py.229: process: 0.55122 | load: 0.00241 [I 2022-10-27 21:02:47] ...trap/engines/engine.py.230: loss: 0.12094 [I 2022-10-27 21:02:47] ...optimizers/trijoint.py.151: total_norm: 14.117179 [I 2022-10-27 21:02:53] ...trap/engines/engine.py.224: train: epoch 88 | batch 230/2304 [I 2022-10-27 21:02:53] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:24 [I 2022-10-27 21:02:53] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00242 [I 2022-10-27 21:02:53] ...trap/engines/engine.py.230: loss: 0.11915 [I 2022-10-27 21:02:53] ...optimizers/trijoint.py.151: total_norm: 25.096083 [I 2022-10-27 21:02:58] ...trap/engines/engine.py.224: train: epoch 88 | batch 240/2304 [I 2022-10-27 21:02:58] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:10 [I 2022-10-27 21:02:58] ...trap/engines/engine.py.229: process: 0.54571 | load: 0.00249 [I 2022-10-27 21:02:58] ...trap/engines/engine.py.230: loss: 0.14377 [I 2022-10-27 21:02:58] ...optimizers/trijoint.py.151: total_norm: 31.773428 [I 2022-10-27 21:03:04] ...trap/engines/engine.py.224: train: epoch 88 | batch 250/2304 [I 2022-10-27 21:03:04] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:03 [I 2022-10-27 21:03:04] ...trap/engines/engine.py.229: process: 0.54280 | load: 0.00260 [I 2022-10-27 21:03:04] ...trap/engines/engine.py.230: loss: 0.13579 [I 2022-10-27 21:03:04] ...optimizers/trijoint.py.151: total_norm: 11.089816 [I 2022-10-27 21:03:09] ...trap/engines/engine.py.224: train: epoch 88 | batch 260/2304 [I 2022-10-27 21:03:09] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:03 [I 2022-10-27 21:03:09] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00246 [I 2022-10-27 21:03:09] ...trap/engines/engine.py.230: loss: 0.24814 [I 2022-10-27 21:03:09] ...optimizers/trijoint.py.151: total_norm: 33.726238 [I 2022-10-27 21:03:15] ...trap/engines/engine.py.224: train: epoch 88 | batch 270/2304 [I 2022-10-27 21:03:15] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:52 [I 2022-10-27 21:03:15] ...trap/engines/engine.py.229: process: 0.55684 | load: 0.00258 [I 2022-10-27 21:03:15] ...trap/engines/engine.py.230: loss: 0.11139 [I 2022-10-27 21:03:15] ...optimizers/trijoint.py.151: total_norm: 10.841674 [I 2022-10-27 21:03:21] ...trap/engines/engine.py.224: train: epoch 88 | batch 280/2304 [I 2022-10-27 21:03:21] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:57 [I 2022-10-27 21:03:21] ...trap/engines/engine.py.229: process: 0.55099 | load: 0.00244 [I 2022-10-27 21:03:21] ...trap/engines/engine.py.230: loss: 0.09103 [I 2022-10-27 21:03:21] ...optimizers/trijoint.py.151: total_norm: 8.943156 [I 2022-10-27 21:03:26] ...trap/engines/engine.py.224: train: epoch 88 | batch 290/2304 [I 2022-10-27 21:03:26] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:42 [I 2022-10-27 21:03:26] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00243 [I 2022-10-27 21:03:26] ...trap/engines/engine.py.230: loss: 0.09913 [I 2022-10-27 21:03:26] ...optimizers/trijoint.py.151: total_norm: 10.507099 [I 2022-10-27 21:03:32] ...trap/engines/engine.py.224: train: epoch 88 | batch 300/2304 [I 2022-10-27 21:03:32] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:37 [I 2022-10-27 21:03:32] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00245 [I 2022-10-27 21:03:32] ...trap/engines/engine.py.230: loss: 0.17368 [I 2022-10-27 21:03:32] ...optimizers/trijoint.py.151: total_norm: 13.437946 [I 2022-10-27 21:03:37] ...trap/engines/engine.py.224: train: epoch 88 | batch 310/2304 [I 2022-10-27 21:03:37] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:32 [I 2022-10-27 21:03:37] ...trap/engines/engine.py.229: process: 0.55612 | load: 0.00259 [I 2022-10-27 21:03:37] ...trap/engines/engine.py.230: loss: 0.14805 [I 2022-10-27 21:03:37] ...optimizers/trijoint.py.151: total_norm: 10.631135 [I 2022-10-27 21:03:43] ...trap/engines/engine.py.224: train: epoch 88 | batch 320/2304 [I 2022-10-27 21:03:43] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:18 [I 2022-10-27 21:03:43] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00247 [I 2022-10-27 21:03:43] ...trap/engines/engine.py.230: loss: 0.19752 [I 2022-10-27 21:03:43] ...optimizers/trijoint.py.151: total_norm: 12.516416 [I 2022-10-27 21:03:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 330/2304 [I 2022-10-27 21:03:48] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:14 [I 2022-10-27 21:03:48] ...trap/engines/engine.py.229: process: 0.54889 | load: 0.00248 [I 2022-10-27 21:03:48] ...trap/engines/engine.py.230: loss: 0.12621 [I 2022-10-27 21:03:48] ...optimizers/trijoint.py.151: total_norm: 15.179945 [I 2022-10-27 21:03:54] ...trap/engines/engine.py.224: train: epoch 88 | batch 340/2304 [I 2022-10-27 21:03:54] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:17 [I 2022-10-27 21:03:54] ...trap/engines/engine.py.229: process: 0.56294 | load: 0.00251 [I 2022-10-27 21:03:54] ...trap/engines/engine.py.230: loss: 0.15689 [I 2022-10-27 21:03:54] ...optimizers/trijoint.py.151: total_norm: 15.729397 [I 2022-10-27 21:04:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 350/2304 [I 2022-10-27 21:04:00] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:11 [I 2022-10-27 21:04:00] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00246 [I 2022-10-27 21:04:00] ...trap/engines/engine.py.230: loss: 0.13040 [I 2022-10-27 21:04:00] ...optimizers/trijoint.py.151: total_norm: 30.688726 [I 2022-10-27 21:04:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 360/2304 [I 2022-10-27 21:04:05] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:05 [I 2022-10-27 21:04:05] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00245 [I 2022-10-27 21:04:05] ...trap/engines/engine.py.230: loss: 0.11511 [I 2022-10-27 21:04:05] ...optimizers/trijoint.py.151: total_norm: 11.962173 [I 2022-10-27 21:04:11] ...trap/engines/engine.py.224: train: epoch 88 | batch 370/2304 [I 2022-10-27 21:04:11] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:17 [I 2022-10-27 21:04:11] ...trap/engines/engine.py.229: process: 0.57298 | load: 0.00263 [I 2022-10-27 21:04:11] ...trap/engines/engine.py.230: loss: 0.12023 [I 2022-10-27 21:04:11] ...optimizers/trijoint.py.151: total_norm: 9.517370 [I 2022-10-27 21:04:16] ...trap/engines/engine.py.224: train: epoch 88 | batch 380/2304 [I 2022-10-27 21:04:16] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:56 [I 2022-10-27 21:04:16] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00257 [I 2022-10-27 21:04:16] ...trap/engines/engine.py.230: loss: 0.16574 [I 2022-10-27 21:04:16] ...optimizers/trijoint.py.151: total_norm: 21.180986 [I 2022-10-27 21:04:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 390/2304 [I 2022-10-27 21:04:22] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:39 [I 2022-10-27 21:04:22] ...trap/engines/engine.py.229: process: 0.54588 | load: 0.00253 [I 2022-10-27 21:04:22] ...trap/engines/engine.py.230: loss: 0.08500 [I 2022-10-27 21:04:22] ...optimizers/trijoint.py.151: total_norm: 9.965944 [I 2022-10-27 21:04:28] ...trap/engines/engine.py.224: train: epoch 88 | batch 400/2304 [I 2022-10-27 21:04:28] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:37 [I 2022-10-27 21:04:28] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00257 [I 2022-10-27 21:04:28] ...trap/engines/engine.py.230: loss: 0.10218 [I 2022-10-27 21:04:28] ...optimizers/trijoint.py.151: total_norm: 12.076719 [I 2022-10-27 21:04:33] ...trap/engines/engine.py.224: train: epoch 88 | batch 410/2304 [I 2022-10-27 21:04:33] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:28 [I 2022-10-27 21:04:33] ...trap/engines/engine.py.229: process: 0.55063 | load: 0.00246 [I 2022-10-27 21:04:33] ...trap/engines/engine.py.230: loss: 0.14195 [I 2022-10-27 21:04:33] ...optimizers/trijoint.py.151: total_norm: 21.304155 [I 2022-10-27 21:04:39] ...trap/engines/engine.py.224: train: epoch 88 | batch 420/2304 [I 2022-10-27 21:04:39] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:30 [I 2022-10-27 21:04:39] ...trap/engines/engine.py.229: process: 0.56354 | load: 0.00294 [I 2022-10-27 21:04:39] ...trap/engines/engine.py.230: loss: 0.13372 [I 2022-10-27 21:04:39] ...optimizers/trijoint.py.151: total_norm: 13.124391 [I 2022-10-27 21:04:44] ...trap/engines/engine.py.224: train: epoch 88 | batch 430/2304 [I 2022-10-27 21:04:44] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:17 [I 2022-10-27 21:04:44] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00245 [I 2022-10-27 21:04:44] ...trap/engines/engine.py.230: loss: 0.12719 [I 2022-10-27 21:04:44] ...optimizers/trijoint.py.151: total_norm: 18.555706 [I 2022-10-27 21:04:50] ...trap/engines/engine.py.224: train: epoch 88 | batch 440/2304 [I 2022-10-27 21:04:50] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:17 [I 2022-10-27 21:04:50] ...trap/engines/engine.py.229: process: 0.54565 | load: 0.00246 [I 2022-10-27 21:04:50] ...trap/engines/engine.py.230: loss: 0.21844 [I 2022-10-27 21:04:50] ...optimizers/trijoint.py.151: total_norm: 27.060457 [I 2022-10-27 21:04:55] ...trap/engines/engine.py.224: train: epoch 88 | batch 450/2304 [I 2022-10-27 21:04:55] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:06 [I 2022-10-27 21:04:55] ...trap/engines/engine.py.229: process: 0.54389 | load: 0.00268 [I 2022-10-27 21:04:55] ...trap/engines/engine.py.230: loss: 0.09992 [I 2022-10-27 21:04:55] ...optimizers/trijoint.py.151: total_norm: 9.447457 [I 2022-10-27 21:05:01] ...trap/engines/engine.py.224: train: epoch 88 | batch 460/2304 [I 2022-10-27 21:05:01] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:08 [I 2022-10-27 21:05:01] ...trap/engines/engine.py.229: process: 0.55817 | load: 0.00273 [I 2022-10-27 21:05:01] ...trap/engines/engine.py.230: loss: 0.15929 [I 2022-10-27 21:05:01] ...optimizers/trijoint.py.151: total_norm: 17.887138 [I 2022-10-27 21:05:07] ...trap/engines/engine.py.224: train: epoch 88 | batch 470/2304 [I 2022-10-27 21:05:07] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:03 [I 2022-10-27 21:05:07] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00262 [I 2022-10-27 21:05:07] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-27 21:05:07] ...optimizers/trijoint.py.151: total_norm: 13.204722 [I 2022-10-27 21:05:12] ...trap/engines/engine.py.224: train: epoch 88 | batch 480/2304 [I 2022-10-27 21:05:12] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:17:00 [I 2022-10-27 21:05:12] ...trap/engines/engine.py.229: process: 0.56359 | load: 0.00257 [I 2022-10-27 21:05:12] ...trap/engines/engine.py.230: loss: 0.12083 [I 2022-10-27 21:05:12] ...optimizers/trijoint.py.151: total_norm: 18.978338 [I 2022-10-27 21:05:18] ...trap/engines/engine.py.224: train: epoch 88 | batch 490/2304 [I 2022-10-27 21:05:18] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:55 [I 2022-10-27 21:05:18] ...trap/engines/engine.py.229: process: 0.56661 | load: 0.00256 [I 2022-10-27 21:05:18] ...trap/engines/engine.py.230: loss: 0.09225 [I 2022-10-27 21:05:18] ...optimizers/trijoint.py.151: total_norm: 11.653417 [I 2022-10-27 21:05:23] ...trap/engines/engine.py.224: train: epoch 88 | batch 500/2304 [I 2022-10-27 21:05:23] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:44 [I 2022-10-27 21:05:23] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00250 [I 2022-10-27 21:05:23] ...trap/engines/engine.py.230: loss: 0.10348 [I 2022-10-27 21:05:23] ...optimizers/trijoint.py.151: total_norm: 12.411769 [I 2022-10-27 21:05:29] ...trap/engines/engine.py.224: train: epoch 88 | batch 510/2304 [I 2022-10-27 21:05:29] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:41 [I 2022-10-27 21:05:29] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00245 [I 2022-10-27 21:05:29] ...trap/engines/engine.py.230: loss: 0.09358 [I 2022-10-27 21:05:29] ...optimizers/trijoint.py.151: total_norm: 10.537218 [I 2022-10-27 21:05:35] ...trap/engines/engine.py.224: train: epoch 88 | batch 520/2304 [I 2022-10-27 21:05:35] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:27 [I 2022-10-27 21:05:35] ...trap/engines/engine.py.229: process: 0.54183 | load: 0.00242 [I 2022-10-27 21:05:35] ...trap/engines/engine.py.230: loss: 0.10393 [I 2022-10-27 21:05:35] ...optimizers/trijoint.py.151: total_norm: 13.105203 [I 2022-10-27 21:05:40] ...trap/engines/engine.py.224: train: epoch 88 | batch 530/2304 [I 2022-10-27 21:05:40] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:27 [I 2022-10-27 21:05:40] ...trap/engines/engine.py.229: process: 0.55903 | load: 0.00250 [I 2022-10-27 21:05:40] ...trap/engines/engine.py.230: loss: 0.16780 [I 2022-10-27 21:05:40] ...optimizers/trijoint.py.151: total_norm: 19.528954 [I 2022-10-27 21:05:46] ...trap/engines/engine.py.224: train: epoch 88 | batch 540/2304 [I 2022-10-27 21:05:46] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:20 [I 2022-10-27 21:05:46] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00266 [I 2022-10-27 21:05:46] ...trap/engines/engine.py.230: loss: 0.10407 [I 2022-10-27 21:05:46] ...optimizers/trijoint.py.151: total_norm: 10.704602 [I 2022-10-27 21:05:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 550/2304 [I 2022-10-27 21:05:51] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:06 [I 2022-10-27 21:05:51] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00256 [I 2022-10-27 21:05:51] ...trap/engines/engine.py.230: loss: 0.10709 [I 2022-10-27 21:05:51] ...optimizers/trijoint.py.151: total_norm: 8.347798 [I 2022-10-27 21:05:57] ...trap/engines/engine.py.224: train: epoch 88 | batch 560/2304 [I 2022-10-27 21:05:57] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:12 [I 2022-10-27 21:05:57] ...trap/engines/engine.py.229: process: 0.56615 | load: 0.00256 [I 2022-10-27 21:05:57] ...trap/engines/engine.py.230: loss: 0.19861 [I 2022-10-27 21:05:57] ...optimizers/trijoint.py.151: total_norm: 21.419010 [I 2022-10-27 21:06:02] ...trap/engines/engine.py.224: train: epoch 88 | batch 570/2304 [I 2022-10-27 21:06:02] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:14 [I 2022-10-27 21:06:02] ...trap/engines/engine.py.229: process: 0.56894 | load: 0.00249 [I 2022-10-27 21:06:02] ...trap/engines/engine.py.230: loss: 0.14334 [I 2022-10-27 21:06:02] ...optimizers/trijoint.py.151: total_norm: 19.879152 [I 2022-10-27 21:06:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 580/2304 [I 2022-10-27 21:06:08] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:15:58 [I 2022-10-27 21:06:08] ...trap/engines/engine.py.229: process: 0.54563 | load: 0.00262 [I 2022-10-27 21:06:08] ...trap/engines/engine.py.230: loss: 0.14029 [I 2022-10-27 21:06:08] ...optimizers/trijoint.py.151: total_norm: 7.997877 [I 2022-10-27 21:06:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 590/2304 [I 2022-10-27 21:06:14] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:53 [I 2022-10-27 21:06:14] ...trap/engines/engine.py.229: process: 0.55672 | load: 0.00252 [I 2022-10-27 21:06:14] ...trap/engines/engine.py.230: loss: 0.14282 [I 2022-10-27 21:06:14] ...optimizers/trijoint.py.151: total_norm: 22.710079 [I 2022-10-27 21:06:19] ...trap/engines/engine.py.224: train: epoch 88 | batch 600/2304 [I 2022-10-27 21:06:19] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:57 [I 2022-10-27 21:06:19] ...trap/engines/engine.py.229: process: 0.56097 | load: 0.00276 [I 2022-10-27 21:06:19] ...trap/engines/engine.py.230: loss: 0.15040 [I 2022-10-27 21:06:19] ...optimizers/trijoint.py.151: total_norm: 20.001553 [I 2022-10-27 21:06:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 610/2304 [I 2022-10-27 21:06:25] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:16:10 [I 2022-10-27 21:06:25] ...trap/engines/engine.py.229: process: 0.61454 | load: 0.00284 [I 2022-10-27 21:06:25] ...trap/engines/engine.py.230: loss: 0.15304 [I 2022-10-27 21:06:25] ...optimizers/trijoint.py.151: total_norm: 19.965204 [I 2022-10-27 21:06:30] ...trap/engines/engine.py.224: train: epoch 88 | batch 620/2304 [I 2022-10-27 21:06:30] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:43 [I 2022-10-27 21:06:30] ...trap/engines/engine.py.229: process: 0.56297 | load: 0.00256 [I 2022-10-27 21:06:30] ...trap/engines/engine.py.230: loss: 0.11296 [I 2022-10-27 21:06:30] ...optimizers/trijoint.py.151: total_norm: 14.190640 [I 2022-10-27 21:06:36] ...trap/engines/engine.py.224: train: epoch 88 | batch 630/2304 [I 2022-10-27 21:06:36] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:32 [I 2022-10-27 21:06:36] ...trap/engines/engine.py.229: process: 0.55393 | load: 0.00278 [I 2022-10-27 21:06:36] ...trap/engines/engine.py.230: loss: 0.16119 [I 2022-10-27 21:06:36] ...optimizers/trijoint.py.151: total_norm: 12.284509 [I 2022-10-27 21:06:42] ...trap/engines/engine.py.224: train: epoch 88 | batch 640/2304 [I 2022-10-27 21:06:42] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:20 [I 2022-10-27 21:06:42] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00251 [I 2022-10-27 21:06:42] ...trap/engines/engine.py.230: loss: 0.12032 [I 2022-10-27 21:06:42] ...optimizers/trijoint.py.151: total_norm: 10.179006 [I 2022-10-27 21:06:47] ...trap/engines/engine.py.224: train: epoch 88 | batch 650/2304 [I 2022-10-27 21:06:47] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:19 [I 2022-10-27 21:06:47] ...trap/engines/engine.py.229: process: 0.55295 | load: 0.00243 [I 2022-10-27 21:06:47] ...trap/engines/engine.py.230: loss: 0.11014 [I 2022-10-27 21:06:47] ...optimizers/trijoint.py.151: total_norm: 16.270542 [I 2022-10-27 21:06:53] ...trap/engines/engine.py.224: train: epoch 88 | batch 660/2304 [I 2022-10-27 21:06:53] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:10 [I 2022-10-27 21:06:53] ...trap/engines/engine.py.229: process: 0.55391 | load: 0.00252 [I 2022-10-27 21:06:53] ...trap/engines/engine.py.230: loss: 0.15203 [I 2022-10-27 21:06:53] ...optimizers/trijoint.py.151: total_norm: 21.388178 [I 2022-10-27 21:06:58] ...trap/engines/engine.py.224: train: epoch 88 | batch 670/2304 [I 2022-10-27 21:06:58] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:09 [I 2022-10-27 21:06:58] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00256 [I 2022-10-27 21:06:58] ...trap/engines/engine.py.230: loss: 0.10000 [I 2022-10-27 21:06:58] ...optimizers/trijoint.py.151: total_norm: 20.322559 [I 2022-10-27 21:07:04] ...trap/engines/engine.py.224: train: epoch 88 | batch 680/2304 [I 2022-10-27 21:07:04] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:05 [I 2022-10-27 21:07:04] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00247 [I 2022-10-27 21:07:04] ...trap/engines/engine.py.230: loss: 0.09635 [I 2022-10-27 21:07:04] ...optimizers/trijoint.py.151: total_norm: 8.866717 [I 2022-10-27 21:07:09] ...trap/engines/engine.py.224: train: epoch 88 | batch 690/2304 [I 2022-10-27 21:07:09] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:52 [I 2022-10-27 21:07:09] ...trap/engines/engine.py.229: process: 0.54889 | load: 0.00262 [I 2022-10-27 21:07:09] ...trap/engines/engine.py.230: loss: 0.10316 [I 2022-10-27 21:07:09] ...optimizers/trijoint.py.151: total_norm: 10.202211 [I 2022-10-27 21:07:15] ...trap/engines/engine.py.224: train: epoch 88 | batch 700/2304 [I 2022-10-27 21:07:15] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:51 [I 2022-10-27 21:07:15] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00255 [I 2022-10-27 21:07:15] ...trap/engines/engine.py.230: loss: 0.17959 [I 2022-10-27 21:07:15] ...optimizers/trijoint.py.151: total_norm: 29.838116 [I 2022-10-27 21:07:21] ...trap/engines/engine.py.224: train: epoch 88 | batch 710/2304 [I 2022-10-27 21:07:21] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:15:36 [I 2022-10-27 21:07:21] ...trap/engines/engine.py.229: process: 0.68359 | load: 0.00248 [I 2022-10-27 21:07:21] ...trap/engines/engine.py.230: loss: 0.14635 [I 2022-10-27 21:07:21] ...optimizers/trijoint.py.151: total_norm: 12.754260 [I 2022-10-27 21:07:26] ...trap/engines/engine.py.224: train: epoch 88 | batch 720/2304 [I 2022-10-27 21:07:26] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:46 [I 2022-10-27 21:07:26] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00252 [I 2022-10-27 21:07:26] ...trap/engines/engine.py.230: loss: 0.10316 [I 2022-10-27 21:07:26] ...optimizers/trijoint.py.151: total_norm: 8.827690 [I 2022-10-27 21:07:32] ...trap/engines/engine.py.224: train: epoch 88 | batch 730/2304 [I 2022-10-27 21:07:32] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:40 [I 2022-10-27 21:07:32] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00249 [I 2022-10-27 21:07:32] ...trap/engines/engine.py.230: loss: 0.11479 [I 2022-10-27 21:07:32] ...optimizers/trijoint.py.151: total_norm: 13.276236 [I 2022-10-27 21:07:38] ...trap/engines/engine.py.224: train: epoch 88 | batch 740/2304 [I 2022-10-27 21:07:38] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:34 [I 2022-10-27 21:07:38] ...trap/engines/engine.py.229: process: 0.56218 | load: 0.00245 [I 2022-10-27 21:07:38] ...trap/engines/engine.py.230: loss: 0.09686 [I 2022-10-27 21:07:38] ...optimizers/trijoint.py.151: total_norm: 18.289984 [I 2022-10-27 21:07:43] ...trap/engines/engine.py.224: train: epoch 88 | batch 750/2304 [I 2022-10-27 21:07:43] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:37 [I 2022-10-27 21:07:43] ...trap/engines/engine.py.229: process: 0.57036 | load: 0.00246 [I 2022-10-27 21:07:43] ...trap/engines/engine.py.230: loss: 0.22052 [I 2022-10-27 21:07:43] ...optimizers/trijoint.py.151: total_norm: 43.028309 [I 2022-10-27 21:07:49] ...trap/engines/engine.py.224: train: epoch 88 | batch 760/2304 [I 2022-10-27 21:07:49] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:27 [I 2022-10-27 21:07:49] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00249 [I 2022-10-27 21:07:49] ...trap/engines/engine.py.230: loss: 0.10455 [I 2022-10-27 21:07:49] ...optimizers/trijoint.py.151: total_norm: 12.420238 [I 2022-10-27 21:07:55] ...trap/engines/engine.py.224: train: epoch 88 | batch 770/2304 [I 2022-10-27 21:07:55] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:15:06 [I 2022-10-27 21:07:55] ...trap/engines/engine.py.229: process: 0.65853 | load: 0.00266 [I 2022-10-27 21:07:55] ...trap/engines/engine.py.230: loss: 0.17440 [I 2022-10-27 21:07:55] ...optimizers/trijoint.py.151: total_norm: 13.197386 [I 2022-10-27 21:08:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 780/2304 [I 2022-10-27 21:08:00] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:18 [I 2022-10-27 21:08:00] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00248 [I 2022-10-27 21:08:00] ...trap/engines/engine.py.230: loss: 0.10950 [I 2022-10-27 21:08:00] ...optimizers/trijoint.py.151: total_norm: 13.617648 [I 2022-10-27 21:08:06] ...trap/engines/engine.py.224: train: epoch 88 | batch 790/2304 [I 2022-10-27 21:08:06] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:07 [I 2022-10-27 21:08:06] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00251 [I 2022-10-27 21:08:06] ...trap/engines/engine.py.230: loss: 0.12999 [I 2022-10-27 21:08:06] ...optimizers/trijoint.py.151: total_norm: 17.789417 [I 2022-10-27 21:08:11] ...trap/engines/engine.py.224: train: epoch 88 | batch 800/2304 [I 2022-10-27 21:08:11] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:57 [I 2022-10-27 21:08:11] ...trap/engines/engine.py.229: process: 0.56004 | load: 0.00247 [I 2022-10-27 21:08:11] ...trap/engines/engine.py.230: loss: 0.09103 [I 2022-10-27 21:08:11] ...optimizers/trijoint.py.151: total_norm: 9.456778 [I 2022-10-27 21:08:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 810/2304 [I 2022-10-27 21:08:17] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:53 [I 2022-10-27 21:08:17] ...trap/engines/engine.py.229: process: 0.55642 | load: 0.00251 [I 2022-10-27 21:08:17] ...trap/engines/engine.py.230: loss: 0.16479 [I 2022-10-27 21:08:17] ...optimizers/trijoint.py.151: total_norm: 28.860556 [I 2022-10-27 21:08:23] ...trap/engines/engine.py.224: train: epoch 88 | batch 820/2304 [I 2022-10-27 21:08:23] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:43 [I 2022-10-27 21:08:23] ...trap/engines/engine.py.229: process: 0.54933 | load: 0.00250 [I 2022-10-27 21:08:23] ...trap/engines/engine.py.230: loss: 0.11265 [I 2022-10-27 21:08:23] ...optimizers/trijoint.py.151: total_norm: 7.974028 [I 2022-10-27 21:08:28] ...trap/engines/engine.py.224: train: epoch 88 | batch 830/2304 [I 2022-10-27 21:08:28] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:33 [I 2022-10-27 21:08:28] ...trap/engines/engine.py.229: process: 0.55005 | load: 0.00243 [I 2022-10-27 21:08:28] ...trap/engines/engine.py.230: loss: 0.15955 [I 2022-10-27 21:08:28] ...optimizers/trijoint.py.151: total_norm: 15.653271 [I 2022-10-27 21:08:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 840/2304 [I 2022-10-27 21:08:34] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:37 [I 2022-10-27 21:08:34] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00241 [I 2022-10-27 21:08:34] ...trap/engines/engine.py.230: loss: 0.14679 [I 2022-10-27 21:08:34] ...optimizers/trijoint.py.151: total_norm: 23.589643 [I 2022-10-27 21:08:39] ...trap/engines/engine.py.224: train: epoch 88 | batch 850/2304 [I 2022-10-27 21:08:39] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:34 [I 2022-10-27 21:08:39] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00248 [I 2022-10-27 21:08:39] ...trap/engines/engine.py.230: loss: 0.12561 [I 2022-10-27 21:08:39] ...optimizers/trijoint.py.151: total_norm: 21.132359 [I 2022-10-27 21:08:45] ...trap/engines/engine.py.224: train: epoch 88 | batch 860/2304 [I 2022-10-27 21:08:45] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:20 [I 2022-10-27 21:08:45] ...trap/engines/engine.py.229: process: 0.54216 | load: 0.00243 [I 2022-10-27 21:08:45] ...trap/engines/engine.py.230: loss: 0.12474 [I 2022-10-27 21:08:45] ...optimizers/trijoint.py.151: total_norm: 7.677775 [I 2022-10-27 21:08:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 870/2304 [I 2022-10-27 21:08:51] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:25 [I 2022-10-27 21:08:51] ...trap/engines/engine.py.229: process: 0.56717 | load: 0.00259 [I 2022-10-27 21:08:51] ...trap/engines/engine.py.230: loss: 0.13502 [I 2022-10-27 21:08:51] ...optimizers/trijoint.py.151: total_norm: 20.009232 [I 2022-10-27 21:08:56] ...trap/engines/engine.py.224: train: epoch 88 | batch 880/2304 [I 2022-10-27 21:08:56] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:13 [I 2022-10-27 21:08:56] ...trap/engines/engine.py.229: process: 0.55950 | load: 0.00243 [I 2022-10-27 21:08:56] ...trap/engines/engine.py.230: loss: 0.12244 [I 2022-10-27 21:08:56] ...optimizers/trijoint.py.151: total_norm: 18.276596 [I 2022-10-27 21:09:02] ...trap/engines/engine.py.224: train: epoch 88 | batch 890/2304 [I 2022-10-27 21:09:02] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:32 [I 2022-10-27 21:09:02] ...trap/engines/engine.py.229: process: 0.56558 | load: 0.00259 [I 2022-10-27 21:09:02] ...trap/engines/engine.py.230: loss: 0.12818 [I 2022-10-27 21:09:02] ...optimizers/trijoint.py.151: total_norm: 12.106508 [I 2022-10-27 21:09:07] ...trap/engines/engine.py.224: train: epoch 88 | batch 900/2304 [I 2022-10-27 21:09:07] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:04 [I 2022-10-27 21:09:07] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00253 [I 2022-10-27 21:09:07] ...trap/engines/engine.py.230: loss: 0.15104 [I 2022-10-27 21:09:07] ...optimizers/trijoint.py.151: total_norm: 13.202077 [I 2022-10-27 21:09:13] ...trap/engines/engine.py.224: train: epoch 88 | batch 910/2304 [I 2022-10-27 21:09:13] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:52 [I 2022-10-27 21:09:13] ...trap/engines/engine.py.229: process: 0.55486 | load: 0.00246 [I 2022-10-27 21:09:13] ...trap/engines/engine.py.230: loss: 0.07482 [I 2022-10-27 21:09:13] ...optimizers/trijoint.py.151: total_norm: 10.896709 [I 2022-10-27 21:09:19] ...trap/engines/engine.py.224: train: epoch 88 | batch 920/2304 [I 2022-10-27 21:09:19] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:49 [I 2022-10-27 21:09:19] ...trap/engines/engine.py.229: process: 0.54876 | load: 0.00242 [I 2022-10-27 21:09:19] ...trap/engines/engine.py.230: loss: 0.12963 [I 2022-10-27 21:09:19] ...optimizers/trijoint.py.151: total_norm: 11.213362 [I 2022-10-27 21:09:24] ...trap/engines/engine.py.224: train: epoch 88 | batch 930/2304 [I 2022-10-27 21:09:24] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:53 [I 2022-10-27 21:09:24] ...trap/engines/engine.py.229: process: 0.54855 | load: 0.00247 [I 2022-10-27 21:09:24] ...trap/engines/engine.py.230: loss: 0.12256 [I 2022-10-27 21:09:24] ...optimizers/trijoint.py.151: total_norm: 22.300261 [I 2022-10-27 21:09:30] ...trap/engines/engine.py.224: train: epoch 88 | batch 940/2304 [I 2022-10-27 21:09:30] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:46 [I 2022-10-27 21:09:30] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00252 [I 2022-10-27 21:09:30] ...trap/engines/engine.py.230: loss: 0.16246 [I 2022-10-27 21:09:30] ...optimizers/trijoint.py.151: total_norm: 17.432714 [I 2022-10-27 21:09:36] ...trap/engines/engine.py.224: train: epoch 88 | batch 950/2304 [I 2022-10-27 21:09:36] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:36 [I 2022-10-27 21:09:36] ...trap/engines/engine.py.229: process: 0.56468 | load: 0.00245 [I 2022-10-27 21:09:36] ...trap/engines/engine.py.230: loss: 0.16805 [I 2022-10-27 21:09:36] ...optimizers/trijoint.py.151: total_norm: 24.324560 [I 2022-10-27 21:09:41] ...trap/engines/engine.py.224: train: epoch 88 | batch 960/2304 [I 2022-10-27 21:09:41] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:33 [I 2022-10-27 21:09:41] ...trap/engines/engine.py.229: process: 0.56229 | load: 0.00255 [I 2022-10-27 21:09:41] ...trap/engines/engine.py.230: loss: 0.11534 [I 2022-10-27 21:09:41] ...optimizers/trijoint.py.151: total_norm: 16.179903 [I 2022-10-27 21:09:47] ...trap/engines/engine.py.224: train: epoch 88 | batch 970/2304 [I 2022-10-27 21:09:47] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:23 [I 2022-10-27 21:09:47] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00295 [I 2022-10-27 21:09:47] ...trap/engines/engine.py.230: loss: 0.09949 [I 2022-10-27 21:09:47] ...optimizers/trijoint.py.151: total_norm: 38.201855 [I 2022-10-27 21:09:52] ...trap/engines/engine.py.224: train: epoch 88 | batch 980/2304 [I 2022-10-27 21:09:52] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:33 [I 2022-10-27 21:09:52] ...trap/engines/engine.py.229: process: 0.57211 | load: 0.00262 [I 2022-10-27 21:09:52] ...trap/engines/engine.py.230: loss: 0.15764 [I 2022-10-27 21:09:52] ...optimizers/trijoint.py.151: total_norm: 30.839102 [I 2022-10-27 21:09:58] ...trap/engines/engine.py.224: train: epoch 88 | batch 990/2304 [I 2022-10-27 21:09:58] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:18 [I 2022-10-27 21:09:58] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00254 [I 2022-10-27 21:09:58] ...trap/engines/engine.py.230: loss: 0.21108 [I 2022-10-27 21:09:58] ...optimizers/trijoint.py.151: total_norm: 30.237463 [I 2022-10-27 21:10:04] ...trap/engines/engine.py.224: train: epoch 88 | batch 1000/2304 [I 2022-10-27 21:10:04] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:10 [I 2022-10-27 21:10:04] ...trap/engines/engine.py.229: process: 0.55375 | load: 0.00243 [I 2022-10-27 21:10:04] ...trap/engines/engine.py.230: loss: 0.19903 [I 2022-10-27 21:10:04] ...optimizers/trijoint.py.151: total_norm: 16.804911 [I 2022-10-27 21:10:09] ...trap/engines/engine.py.224: train: epoch 88 | batch 1010/2304 [I 2022-10-27 21:10:09] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:08 [I 2022-10-27 21:10:09] ...trap/engines/engine.py.229: process: 0.54945 | load: 0.00250 [I 2022-10-27 21:10:09] ...trap/engines/engine.py.230: loss: 0.11821 [I 2022-10-27 21:10:09] ...optimizers/trijoint.py.151: total_norm: 11.253060 [I 2022-10-27 21:10:15] ...trap/engines/engine.py.224: train: epoch 88 | batch 1020/2304 [I 2022-10-27 21:10:15] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:54 [I 2022-10-27 21:10:15] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00250 [I 2022-10-27 21:10:15] ...trap/engines/engine.py.230: loss: 0.14794 [I 2022-10-27 21:10:15] ...optimizers/trijoint.py.151: total_norm: 22.515739 [I 2022-10-27 21:10:20] ...trap/engines/engine.py.224: train: epoch 88 | batch 1030/2304 [I 2022-10-27 21:10:20] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:48 [I 2022-10-27 21:10:20] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00244 [I 2022-10-27 21:10:20] ...trap/engines/engine.py.230: loss: 0.09482 [I 2022-10-27 21:10:20] ...optimizers/trijoint.py.151: total_norm: 11.455728 [I 2022-10-27 21:10:26] ...trap/engines/engine.py.224: train: epoch 88 | batch 1040/2304 [I 2022-10-27 21:10:26] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:43 [I 2022-10-27 21:10:26] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00250 [I 2022-10-27 21:10:26] ...trap/engines/engine.py.230: loss: 0.08649 [I 2022-10-27 21:10:26] ...optimizers/trijoint.py.151: total_norm: 13.054133 [I 2022-10-27 21:10:32] ...trap/engines/engine.py.224: train: epoch 88 | batch 1050/2304 [I 2022-10-27 21:10:32] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:42 [I 2022-10-27 21:10:32] ...trap/engines/engine.py.229: process: 0.56978 | load: 0.00251 [I 2022-10-27 21:10:32] ...trap/engines/engine.py.230: loss: 0.11603 [I 2022-10-27 21:10:32] ...optimizers/trijoint.py.151: total_norm: 11.349874 [I 2022-10-27 21:10:37] ...trap/engines/engine.py.224: train: epoch 88 | batch 1060/2304 [I 2022-10-27 21:10:37] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:32 [I 2022-10-27 21:10:37] ...trap/engines/engine.py.229: process: 0.55720 | load: 0.00242 [I 2022-10-27 21:10:37] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-27 21:10:37] ...optimizers/trijoint.py.151: total_norm: 17.747969 [I 2022-10-27 21:10:43] ...trap/engines/engine.py.224: train: epoch 88 | batch 1070/2304 [I 2022-10-27 21:10:43] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:31 [I 2022-10-27 21:10:43] ...trap/engines/engine.py.229: process: 0.57068 | load: 0.00243 [I 2022-10-27 21:10:43] ...trap/engines/engine.py.230: loss: 0.09366 [I 2022-10-27 21:10:43] ...optimizers/trijoint.py.151: total_norm: 12.113954 [I 2022-10-27 21:10:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 1080/2304 [I 2022-10-27 21:10:48] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:19 [I 2022-10-27 21:10:48] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00249 [I 2022-10-27 21:10:48] ...trap/engines/engine.py.230: loss: 0.12933 [I 2022-10-27 21:10:48] ...optimizers/trijoint.py.151: total_norm: 19.939352 [I 2022-10-27 21:10:54] ...trap/engines/engine.py.224: train: epoch 88 | batch 1090/2304 [I 2022-10-27 21:10:54] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:17 [I 2022-10-27 21:10:54] ...trap/engines/engine.py.229: process: 0.56361 | load: 0.00247 [I 2022-10-27 21:10:54] ...trap/engines/engine.py.230: loss: 0.12322 [I 2022-10-27 21:10:54] ...optimizers/trijoint.py.151: total_norm: 17.128321 [I 2022-10-27 21:11:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 1100/2304 [I 2022-10-27 21:11:00] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:13 [I 2022-10-27 21:11:00] ...trap/engines/engine.py.229: process: 0.56112 | load: 0.00246 [I 2022-10-27 21:11:00] ...trap/engines/engine.py.230: loss: 0.10654 [I 2022-10-27 21:11:00] ...optimizers/trijoint.py.151: total_norm: 9.104223 [I 2022-10-27 21:11:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 1110/2304 [I 2022-10-27 21:11:05] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:11 [I 2022-10-27 21:11:05] ...trap/engines/engine.py.229: process: 0.58112 | load: 0.00243 [I 2022-10-27 21:11:05] ...trap/engines/engine.py.230: loss: 0.18618 [I 2022-10-27 21:11:05] ...optimizers/trijoint.py.151: total_norm: 13.815816 [I 2022-10-27 21:11:11] ...trap/engines/engine.py.224: train: epoch 88 | batch 1120/2304 [I 2022-10-27 21:11:11] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:14 [I 2022-10-27 21:11:11] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00257 [I 2022-10-27 21:11:11] ...trap/engines/engine.py.230: loss: 0.11991 [I 2022-10-27 21:11:11] ...optimizers/trijoint.py.151: total_norm: 14.999314 [I 2022-10-27 21:11:16] ...trap/engines/engine.py.224: train: epoch 88 | batch 1130/2304 [I 2022-10-27 21:11:16] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:55 [I 2022-10-27 21:11:16] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00247 [I 2022-10-27 21:11:16] ...trap/engines/engine.py.230: loss: 0.09797 [I 2022-10-27 21:11:16] ...optimizers/trijoint.py.151: total_norm: 12.179312 [I 2022-10-27 21:11:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 1140/2304 [I 2022-10-27 21:11:22] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:48 [I 2022-10-27 21:11:22] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00249 [I 2022-10-27 21:11:22] ...trap/engines/engine.py.230: loss: 0.13897 [I 2022-10-27 21:11:22] ...optimizers/trijoint.py.151: total_norm: 12.485609 [I 2022-10-27 21:11:28] ...trap/engines/engine.py.224: train: epoch 88 | batch 1150/2304 [I 2022-10-27 21:11:28] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:43 [I 2022-10-27 21:11:28] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00250 [I 2022-10-27 21:11:28] ...trap/engines/engine.py.230: loss: 0.13021 [I 2022-10-27 21:11:28] ...optimizers/trijoint.py.151: total_norm: 10.028144 [I 2022-10-27 21:11:33] ...trap/engines/engine.py.224: train: epoch 88 | batch 1160/2304 [I 2022-10-27 21:11:33] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:37 [I 2022-10-27 21:11:33] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00249 [I 2022-10-27 21:11:33] ...trap/engines/engine.py.230: loss: 0.09618 [I 2022-10-27 21:11:33] ...optimizers/trijoint.py.151: total_norm: 23.755974 [I 2022-10-27 21:11:39] ...trap/engines/engine.py.224: train: epoch 88 | batch 1170/2304 [I 2022-10-27 21:11:39] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:33 [I 2022-10-27 21:11:39] ...trap/engines/engine.py.229: process: 0.55680 | load: 0.00250 [I 2022-10-27 21:11:39] ...trap/engines/engine.py.230: loss: 0.12212 [I 2022-10-27 21:11:39] ...optimizers/trijoint.py.151: total_norm: 35.342804 [I 2022-10-27 21:11:44] ...trap/engines/engine.py.224: train: epoch 88 | batch 1180/2304 [I 2022-10-27 21:11:44] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:30 [I 2022-10-27 21:11:44] ...trap/engines/engine.py.229: process: 0.57407 | load: 0.00250 [I 2022-10-27 21:11:44] ...trap/engines/engine.py.230: loss: 0.20937 [I 2022-10-27 21:11:44] ...optimizers/trijoint.py.151: total_norm: 27.749313 [I 2022-10-27 21:11:50] ...trap/engines/engine.py.224: train: epoch 88 | batch 1190/2304 [I 2022-10-27 21:11:50] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:22 [I 2022-10-27 21:11:50] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00269 [I 2022-10-27 21:11:50] ...trap/engines/engine.py.230: loss: 0.10700 [I 2022-10-27 21:11:50] ...optimizers/trijoint.py.151: total_norm: 14.777905 [I 2022-10-27 21:11:56] ...trap/engines/engine.py.224: train: epoch 88 | batch 1200/2304 [I 2022-10-27 21:11:56] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:14 [I 2022-10-27 21:11:56] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00242 [I 2022-10-27 21:11:56] ...trap/engines/engine.py.230: loss: 0.11961 [I 2022-10-27 21:11:56] ...optimizers/trijoint.py.151: total_norm: 21.248337 [I 2022-10-27 21:12:01] ...trap/engines/engine.py.224: train: epoch 88 | batch 1210/2304 [I 2022-10-27 21:12:01] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:06 [I 2022-10-27 21:12:01] ...trap/engines/engine.py.229: process: 0.55676 | load: 0.00247 [I 2022-10-27 21:12:01] ...trap/engines/engine.py.230: loss: 0.16555 [I 2022-10-27 21:12:01] ...optimizers/trijoint.py.151: total_norm: 20.182732 [I 2022-10-27 21:12:07] ...trap/engines/engine.py.224: train: epoch 88 | batch 1220/2304 [I 2022-10-27 21:12:07] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:10 [I 2022-10-27 21:12:07] ...trap/engines/engine.py.229: process: 0.56282 | load: 0.00248 [I 2022-10-27 21:12:07] ...trap/engines/engine.py.230: loss: 0.12955 [I 2022-10-27 21:12:07] ...optimizers/trijoint.py.151: total_norm: 21.453951 [I 2022-10-27 21:12:12] ...trap/engines/engine.py.224: train: epoch 88 | batch 1230/2304 [I 2022-10-27 21:12:12] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:09:59 [I 2022-10-27 21:12:12] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00245 [I 2022-10-27 21:12:12] ...trap/engines/engine.py.230: loss: 0.13394 [I 2022-10-27 21:12:12] ...optimizers/trijoint.py.151: total_norm: 12.610420 [I 2022-10-27 21:12:18] ...trap/engines/engine.py.224: train: epoch 88 | batch 1240/2304 [I 2022-10-27 21:12:18] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:53 [I 2022-10-27 21:12:18] ...trap/engines/engine.py.229: process: 0.56163 | load: 0.00246 [I 2022-10-27 21:12:18] ...trap/engines/engine.py.230: loss: 0.14357 [I 2022-10-27 21:12:18] ...optimizers/trijoint.py.151: total_norm: 15.228233 [I 2022-10-27 21:12:24] ...trap/engines/engine.py.224: train: epoch 88 | batch 1250/2304 [I 2022-10-27 21:12:24] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:46 [I 2022-10-27 21:12:24] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00245 [I 2022-10-27 21:12:24] ...trap/engines/engine.py.230: loss: 0.12514 [I 2022-10-27 21:12:24] ...optimizers/trijoint.py.151: total_norm: 27.778563 [I 2022-10-27 21:12:29] ...trap/engines/engine.py.224: train: epoch 88 | batch 1260/2304 [I 2022-10-27 21:12:29] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:42 [I 2022-10-27 21:12:29] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00280 [I 2022-10-27 21:12:29] ...trap/engines/engine.py.230: loss: 0.16440 [I 2022-10-27 21:12:29] ...optimizers/trijoint.py.151: total_norm: 34.666473 [I 2022-10-27 21:12:35] ...trap/engines/engine.py.224: train: epoch 88 | batch 1270/2304 [I 2022-10-27 21:12:35] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:34 [I 2022-10-27 21:12:35] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00258 [I 2022-10-27 21:12:35] ...trap/engines/engine.py.230: loss: 0.12144 [I 2022-10-27 21:12:35] ...optimizers/trijoint.py.151: total_norm: 14.113369 [I 2022-10-27 21:12:40] ...trap/engines/engine.py.224: train: epoch 88 | batch 1280/2304 [I 2022-10-27 21:12:40] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:31 [I 2022-10-27 21:12:40] ...trap/engines/engine.py.229: process: 0.57072 | load: 0.00256 [I 2022-10-27 21:12:40] ...trap/engines/engine.py.230: loss: 0.13023 [I 2022-10-27 21:12:40] ...optimizers/trijoint.py.151: total_norm: 15.525324 [I 2022-10-27 21:12:46] ...trap/engines/engine.py.224: train: epoch 88 | batch 1290/2304 [I 2022-10-27 21:12:46] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:19 [I 2022-10-27 21:12:46] ...trap/engines/engine.py.229: process: 0.55132 | load: 0.00249 [I 2022-10-27 21:12:46] ...trap/engines/engine.py.230: loss: 0.15055 [I 2022-10-27 21:12:46] ...optimizers/trijoint.py.151: total_norm: 12.603270 [I 2022-10-27 21:12:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 1300/2304 [I 2022-10-27 21:12:51] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:15 [I 2022-10-27 21:12:51] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00248 [I 2022-10-27 21:12:51] ...trap/engines/engine.py.230: loss: 0.16473 [I 2022-10-27 21:12:51] ...optimizers/trijoint.py.151: total_norm: 11.843205 [I 2022-10-27 21:12:57] ...trap/engines/engine.py.224: train: epoch 88 | batch 1310/2304 [I 2022-10-27 21:12:57] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:17 [I 2022-10-27 21:12:57] ...trap/engines/engine.py.229: process: 0.56825 | load: 0.00250 [I 2022-10-27 21:12:57] ...trap/engines/engine.py.230: loss: 0.12153 [I 2022-10-27 21:12:57] ...optimizers/trijoint.py.151: total_norm: 22.652180 [I 2022-10-27 21:13:03] ...trap/engines/engine.py.224: train: epoch 88 | batch 1320/2304 [I 2022-10-27 21:13:03] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:02 [I 2022-10-27 21:13:03] ...trap/engines/engine.py.229: process: 0.54997 | load: 0.00255 [I 2022-10-27 21:13:03] ...trap/engines/engine.py.230: loss: 0.16983 [I 2022-10-27 21:13:03] ...optimizers/trijoint.py.151: total_norm: 30.940559 [I 2022-10-27 21:13:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 1330/2304 [I 2022-10-27 21:13:08] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:04 [I 2022-10-27 21:13:08] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00255 [I 2022-10-27 21:13:08] ...trap/engines/engine.py.230: loss: 0.21132 [I 2022-10-27 21:13:08] ...optimizers/trijoint.py.151: total_norm: 20.282785 [I 2022-10-27 21:13:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 1340/2304 [I 2022-10-27 21:13:14] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:59 [I 2022-10-27 21:13:14] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00261 [I 2022-10-27 21:13:14] ...trap/engines/engine.py.230: loss: 0.13460 [I 2022-10-27 21:13:14] ...optimizers/trijoint.py.151: total_norm: 21.848974 [I 2022-10-27 21:13:19] ...trap/engines/engine.py.224: train: epoch 88 | batch 1350/2304 [I 2022-10-27 21:13:19] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:49 [I 2022-10-27 21:13:19] ...trap/engines/engine.py.229: process: 0.54841 | load: 0.00245 [I 2022-10-27 21:13:19] ...trap/engines/engine.py.230: loss: 0.12733 [I 2022-10-27 21:13:19] ...optimizers/trijoint.py.151: total_norm: 24.323847 [I 2022-10-27 21:13:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 1360/2304 [I 2022-10-27 21:13:25] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:44 [I 2022-10-27 21:13:25] ...trap/engines/engine.py.229: process: 0.53978 | load: 0.00247 [I 2022-10-27 21:13:25] ...trap/engines/engine.py.230: loss: 0.11013 [I 2022-10-27 21:13:25] ...optimizers/trijoint.py.151: total_norm: 13.145664 [I 2022-10-27 21:13:30] ...trap/engines/engine.py.224: train: epoch 88 | batch 1370/2304 [I 2022-10-27 21:13:30] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:39 [I 2022-10-27 21:13:30] ...trap/engines/engine.py.229: process: 0.56283 | load: 0.00251 [I 2022-10-27 21:13:30] ...trap/engines/engine.py.230: loss: 0.11883 [I 2022-10-27 21:13:30] ...optimizers/trijoint.py.151: total_norm: 10.574506 [I 2022-10-27 21:13:36] ...trap/engines/engine.py.224: train: epoch 88 | batch 1380/2304 [I 2022-10-27 21:13:36] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:37 [I 2022-10-27 21:13:36] ...trap/engines/engine.py.229: process: 0.55905 | load: 0.00245 [I 2022-10-27 21:13:36] ...trap/engines/engine.py.230: loss: 0.17199 [I 2022-10-27 21:13:36] ...optimizers/trijoint.py.151: total_norm: 36.427677 [I 2022-10-27 21:13:42] ...trap/engines/engine.py.224: train: epoch 88 | batch 1390/2304 [I 2022-10-27 21:13:42] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:30 [I 2022-10-27 21:13:42] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00252 [I 2022-10-27 21:13:42] ...trap/engines/engine.py.230: loss: 0.11758 [I 2022-10-27 21:13:42] ...optimizers/trijoint.py.151: total_norm: 20.842094 [I 2022-10-27 21:13:47] ...trap/engines/engine.py.224: train: epoch 88 | batch 1400/2304 [I 2022-10-27 21:13:47] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:25 [I 2022-10-27 21:13:47] ...trap/engines/engine.py.229: process: 0.57373 | load: 0.00244 [I 2022-10-27 21:13:47] ...trap/engines/engine.py.230: loss: 0.18977 [I 2022-10-27 21:13:47] ...optimizers/trijoint.py.151: total_norm: 12.832524 [I 2022-10-27 21:13:53] ...trap/engines/engine.py.224: train: epoch 88 | batch 1410/2304 [I 2022-10-27 21:13:53] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:21 [I 2022-10-27 21:13:53] ...trap/engines/engine.py.229: process: 0.55662 | load: 0.00244 [I 2022-10-27 21:13:53] ...trap/engines/engine.py.230: loss: 0.11807 [I 2022-10-27 21:13:53] ...optimizers/trijoint.py.151: total_norm: 11.751467 [I 2022-10-27 21:13:58] ...trap/engines/engine.py.224: train: epoch 88 | batch 1420/2304 [I 2022-10-27 21:13:58] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:14 [I 2022-10-27 21:13:58] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00249 [I 2022-10-27 21:13:58] ...trap/engines/engine.py.230: loss: 0.14087 [I 2022-10-27 21:13:58] ...optimizers/trijoint.py.151: total_norm: 15.442699 [I 2022-10-27 21:14:04] ...trap/engines/engine.py.224: train: epoch 88 | batch 1430/2304 [I 2022-10-27 21:14:04] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:08 [I 2022-10-27 21:14:04] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00252 [I 2022-10-27 21:14:04] ...trap/engines/engine.py.230: loss: 0.17906 [I 2022-10-27 21:14:04] ...optimizers/trijoint.py.151: total_norm: 15.313526 [I 2022-10-27 21:14:10] ...trap/engines/engine.py.224: train: epoch 88 | batch 1440/2304 [I 2022-10-27 21:14:10] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:03 [I 2022-10-27 21:14:10] ...trap/engines/engine.py.229: process: 0.56324 | load: 0.00282 [I 2022-10-27 21:14:10] ...trap/engines/engine.py.230: loss: 0.19474 [I 2022-10-27 21:14:10] ...optimizers/trijoint.py.151: total_norm: 23.718546 [I 2022-10-27 21:14:15] ...trap/engines/engine.py.224: train: epoch 88 | batch 1450/2304 [I 2022-10-27 21:14:15] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:56 [I 2022-10-27 21:14:15] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00268 [I 2022-10-27 21:14:15] ...trap/engines/engine.py.230: loss: 0.13688 [I 2022-10-27 21:14:15] ...optimizers/trijoint.py.151: total_norm: 28.101326 [I 2022-10-27 21:14:21] ...trap/engines/engine.py.224: train: epoch 88 | batch 1460/2304 [I 2022-10-27 21:14:21] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:52 [I 2022-10-27 21:14:21] ...trap/engines/engine.py.229: process: 0.56986 | load: 0.00260 [I 2022-10-27 21:14:21] ...trap/engines/engine.py.230: loss: 0.08743 [I 2022-10-27 21:14:21] ...optimizers/trijoint.py.151: total_norm: 12.121965 [I 2022-10-27 21:14:26] ...trap/engines/engine.py.224: train: epoch 88 | batch 1470/2304 [I 2022-10-27 21:14:26] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:47 [I 2022-10-27 21:14:26] ...trap/engines/engine.py.229: process: 0.56316 | load: 0.00249 [I 2022-10-27 21:14:26] ...trap/engines/engine.py.230: loss: 0.15285 [I 2022-10-27 21:14:26] ...optimizers/trijoint.py.151: total_norm: 17.105387 [I 2022-10-27 21:14:32] ...trap/engines/engine.py.224: train: epoch 88 | batch 1480/2304 [I 2022-10-27 21:14:32] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:36 [I 2022-10-27 21:14:32] ...trap/engines/engine.py.229: process: 0.55130 | load: 0.00249 [I 2022-10-27 21:14:32] ...trap/engines/engine.py.230: loss: 0.09885 [I 2022-10-27 21:14:32] ...optimizers/trijoint.py.151: total_norm: 14.103961 [I 2022-10-27 21:14:38] ...trap/engines/engine.py.224: train: epoch 88 | batch 1490/2304 [I 2022-10-27 21:14:38] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:29 [I 2022-10-27 21:14:38] ...trap/engines/engine.py.229: process: 0.54202 | load: 0.00248 [I 2022-10-27 21:14:38] ...trap/engines/engine.py.230: loss: 0.15483 [I 2022-10-27 21:14:38] ...optimizers/trijoint.py.151: total_norm: 21.725460 [I 2022-10-27 21:14:43] ...trap/engines/engine.py.224: train: epoch 88 | batch 1500/2304 [I 2022-10-27 21:14:43] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:27 [I 2022-10-27 21:14:43] ...trap/engines/engine.py.229: process: 0.55555 | load: 0.00246 [I 2022-10-27 21:14:43] ...trap/engines/engine.py.230: loss: 0.10045 [I 2022-10-27 21:14:43] ...optimizers/trijoint.py.151: total_norm: 10.985077 [I 2022-10-27 21:14:49] ...trap/engines/engine.py.224: train: epoch 88 | batch 1510/2304 [I 2022-10-27 21:14:49] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:16 [I 2022-10-27 21:14:49] ...trap/engines/engine.py.229: process: 0.54153 | load: 0.00245 [I 2022-10-27 21:14:49] ...trap/engines/engine.py.230: loss: 0.11040 [I 2022-10-27 21:14:49] ...optimizers/trijoint.py.151: total_norm: 7.668770 [I 2022-10-27 21:14:54] ...trap/engines/engine.py.224: train: epoch 88 | batch 1520/2304 [I 2022-10-27 21:14:54] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:15 [I 2022-10-27 21:14:54] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00246 [I 2022-10-27 21:14:54] ...trap/engines/engine.py.230: loss: 0.15788 [I 2022-10-27 21:14:54] ...optimizers/trijoint.py.151: total_norm: 12.634206 [I 2022-10-27 21:15:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 1530/2304 [I 2022-10-27 21:15:00] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:09 [I 2022-10-27 21:15:00] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00253 [I 2022-10-27 21:15:00] ...trap/engines/engine.py.230: loss: 0.12380 [I 2022-10-27 21:15:00] ...optimizers/trijoint.py.151: total_norm: 14.027087 [I 2022-10-27 21:15:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 1540/2304 [I 2022-10-27 21:15:05] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:07 [I 2022-10-27 21:15:05] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00254 [I 2022-10-27 21:15:05] ...trap/engines/engine.py.230: loss: 0.11251 [I 2022-10-27 21:15:05] ...optimizers/trijoint.py.151: total_norm: 11.497167 [I 2022-10-27 21:15:11] ...trap/engines/engine.py.224: train: epoch 88 | batch 1550/2304 [I 2022-10-27 21:15:11] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:08 [I 2022-10-27 21:15:11] ...trap/engines/engine.py.229: process: 0.58100 | load: 0.00249 [I 2022-10-27 21:15:11] ...trap/engines/engine.py.230: loss: 0.15485 [I 2022-10-27 21:15:11] ...optimizers/trijoint.py.151: total_norm: 11.737235 [I 2022-10-27 21:15:17] ...trap/engines/engine.py.224: train: epoch 88 | batch 1560/2304 [I 2022-10-27 21:15:17] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:52 [I 2022-10-27 21:15:17] ...trap/engines/engine.py.229: process: 0.54962 | load: 0.00247 [I 2022-10-27 21:15:17] ...trap/engines/engine.py.230: loss: 0.13023 [I 2022-10-27 21:15:17] ...optimizers/trijoint.py.151: total_norm: 14.147105 [I 2022-10-27 21:15:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 1570/2304 [I 2022-10-27 21:15:22] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:50 [I 2022-10-27 21:15:22] ...trap/engines/engine.py.229: process: 0.56517 | load: 0.00246 [I 2022-10-27 21:15:22] ...trap/engines/engine.py.230: loss: 0.14369 [I 2022-10-27 21:15:22] ...optimizers/trijoint.py.151: total_norm: 23.780960 [I 2022-10-27 21:15:28] ...trap/engines/engine.py.224: train: epoch 88 | batch 1580/2304 [I 2022-10-27 21:15:28] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:42 [I 2022-10-27 21:15:28] ...trap/engines/engine.py.229: process: 0.55269 | load: 0.00247 [I 2022-10-27 21:15:28] ...trap/engines/engine.py.230: loss: 0.11520 [I 2022-10-27 21:15:28] ...optimizers/trijoint.py.151: total_norm: 19.699429 [I 2022-10-27 21:15:33] ...trap/engines/engine.py.224: train: epoch 88 | batch 1590/2304 [I 2022-10-27 21:15:33] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:36 [I 2022-10-27 21:15:33] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00247 [I 2022-10-27 21:15:33] ...trap/engines/engine.py.230: loss: 0.09778 [I 2022-10-27 21:15:33] ...optimizers/trijoint.py.151: total_norm: 68.885841 [I 2022-10-27 21:15:39] ...trap/engines/engine.py.224: train: epoch 88 | batch 1600/2304 [I 2022-10-27 21:15:39] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:31 [I 2022-10-27 21:15:39] ...trap/engines/engine.py.229: process: 0.54719 | load: 0.00245 [I 2022-10-27 21:15:39] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-27 21:15:39] ...optimizers/trijoint.py.151: total_norm: 24.039429 [I 2022-10-27 21:15:45] ...trap/engines/engine.py.224: train: epoch 88 | batch 1610/2304 [I 2022-10-27 21:15:45] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:28 [I 2022-10-27 21:15:45] ...trap/engines/engine.py.229: process: 0.56349 | load: 0.00239 [I 2022-10-27 21:15:45] ...trap/engines/engine.py.230: loss: 0.10329 [I 2022-10-27 21:15:45] ...optimizers/trijoint.py.151: total_norm: 55.044773 [I 2022-10-27 21:15:50] ...trap/engines/engine.py.224: train: epoch 88 | batch 1620/2304 [I 2022-10-27 21:15:50] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:22 [I 2022-10-27 21:15:50] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00238 [I 2022-10-27 21:15:50] ...trap/engines/engine.py.230: loss: 0.10305 [I 2022-10-27 21:15:50] ...optimizers/trijoint.py.151: total_norm: 7.518834 [I 2022-10-27 21:15:56] ...trap/engines/engine.py.224: train: epoch 88 | batch 1630/2304 [I 2022-10-27 21:15:56] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:20 [I 2022-10-27 21:15:56] ...trap/engines/engine.py.229: process: 0.56576 | load: 0.00243 [I 2022-10-27 21:15:56] ...trap/engines/engine.py.230: loss: 0.16094 [I 2022-10-27 21:15:56] ...optimizers/trijoint.py.151: total_norm: 11.827118 [I 2022-10-27 21:16:01] ...trap/engines/engine.py.224: train: epoch 88 | batch 1640/2304 [I 2022-10-27 21:16:01] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:15 [I 2022-10-27 21:16:01] ...trap/engines/engine.py.229: process: 0.57131 | load: 0.00240 [I 2022-10-27 21:16:01] ...trap/engines/engine.py.230: loss: 0.16306 [I 2022-10-27 21:16:01] ...optimizers/trijoint.py.151: total_norm: 36.958618 [I 2022-10-27 21:16:07] ...trap/engines/engine.py.224: train: epoch 88 | batch 1650/2304 [I 2022-10-27 21:16:07] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:03 [I 2022-10-27 21:16:07] ...trap/engines/engine.py.229: process: 0.54637 | load: 0.00244 [I 2022-10-27 21:16:07] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-27 21:16:07] ...optimizers/trijoint.py.151: total_norm: 13.334823 [I 2022-10-27 21:16:13] ...trap/engines/engine.py.224: train: epoch 88 | batch 1660/2304 [I 2022-10-27 21:16:13] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:59 [I 2022-10-27 21:16:13] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00254 [I 2022-10-27 21:16:13] ...trap/engines/engine.py.230: loss: 0.13387 [I 2022-10-27 21:16:13] ...optimizers/trijoint.py.151: total_norm: 27.992168 [I 2022-10-27 21:16:18] ...trap/engines/engine.py.224: train: epoch 88 | batch 1670/2304 [I 2022-10-27 21:16:18] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:52 [I 2022-10-27 21:16:18] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00253 [I 2022-10-27 21:16:18] ...trap/engines/engine.py.230: loss: 0.09540 [I 2022-10-27 21:16:18] ...optimizers/trijoint.py.151: total_norm: 51.218094 [I 2022-10-27 21:16:24] ...trap/engines/engine.py.224: train: epoch 88 | batch 1680/2304 [I 2022-10-27 21:16:24] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:59 [I 2022-10-27 21:16:24] ...trap/engines/engine.py.229: process: 0.60974 | load: 0.00258 [I 2022-10-27 21:16:24] ...trap/engines/engine.py.230: loss: 0.10707 [I 2022-10-27 21:16:24] ...optimizers/trijoint.py.151: total_norm: 16.690340 [I 2022-10-27 21:16:30] ...trap/engines/engine.py.224: train: epoch 88 | batch 1690/2304 [I 2022-10-27 21:16:30] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:47 [I 2022-10-27 21:16:30] ...trap/engines/engine.py.229: process: 0.55067 | load: 0.00245 [I 2022-10-27 21:16:30] ...trap/engines/engine.py.230: loss: 0.08177 [I 2022-10-27 21:16:30] ...optimizers/trijoint.py.151: total_norm: 9.789320 [I 2022-10-27 21:16:35] ...trap/engines/engine.py.224: train: epoch 88 | batch 1700/2304 [I 2022-10-27 21:16:35] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:51 [I 2022-10-27 21:16:35] ...trap/engines/engine.py.229: process: 0.57212 | load: 0.00274 [I 2022-10-27 21:16:35] ...trap/engines/engine.py.230: loss: 0.09930 [I 2022-10-27 21:16:35] ...optimizers/trijoint.py.151: total_norm: 11.538998 [I 2022-10-27 21:16:41] ...trap/engines/engine.py.224: train: epoch 88 | batch 1710/2304 [I 2022-10-27 21:16:41] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:35 [I 2022-10-27 21:16:41] ...trap/engines/engine.py.229: process: 0.56592 | load: 0.00250 [I 2022-10-27 21:16:41] ...trap/engines/engine.py.230: loss: 0.09289 [I 2022-10-27 21:16:41] ...optimizers/trijoint.py.151: total_norm: 15.112003 [I 2022-10-27 21:16:47] ...trap/engines/engine.py.224: train: epoch 88 | batch 1720/2304 [I 2022-10-27 21:16:47] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:31 [I 2022-10-27 21:16:47] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00256 [I 2022-10-27 21:16:47] ...trap/engines/engine.py.230: loss: 0.11742 [I 2022-10-27 21:16:47] ...optimizers/trijoint.py.151: total_norm: 24.497398 [I 2022-10-27 21:16:52] ...trap/engines/engine.py.224: train: epoch 88 | batch 1730/2304 [I 2022-10-27 21:16:52] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:22 [I 2022-10-27 21:16:52] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00246 [I 2022-10-27 21:16:52] ...trap/engines/engine.py.230: loss: 0.14003 [I 2022-10-27 21:16:52] ...optimizers/trijoint.py.151: total_norm: 17.553783 [I 2022-10-27 21:16:58] ...trap/engines/engine.py.224: train: epoch 88 | batch 1740/2304 [I 2022-10-27 21:16:58] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:16 [I 2022-10-27 21:16:58] ...trap/engines/engine.py.229: process: 0.56810 | load: 0.00245 [I 2022-10-27 21:16:58] ...trap/engines/engine.py.230: loss: 0.13880 [I 2022-10-27 21:16:58] ...optimizers/trijoint.py.151: total_norm: 18.211504 [I 2022-10-27 21:17:04] ...trap/engines/engine.py.224: train: epoch 88 | batch 1750/2304 [I 2022-10-27 21:17:04] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:10 [I 2022-10-27 21:17:04] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00243 [I 2022-10-27 21:17:04] ...trap/engines/engine.py.230: loss: 0.14440 [I 2022-10-27 21:17:04] ...optimizers/trijoint.py.151: total_norm: 16.744612 [I 2022-10-27 21:17:09] ...trap/engines/engine.py.224: train: epoch 88 | batch 1760/2304 [I 2022-10-27 21:17:09] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:03 [I 2022-10-27 21:17:09] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00241 [I 2022-10-27 21:17:09] ...trap/engines/engine.py.230: loss: 0.10554 [I 2022-10-27 21:17:09] ...optimizers/trijoint.py.151: total_norm: 16.334013 [I 2022-10-27 21:17:15] ...trap/engines/engine.py.224: train: epoch 88 | batch 1770/2304 [I 2022-10-27 21:17:15] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:05:03 [I 2022-10-27 21:17:15] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00241 [I 2022-10-27 21:17:15] ...trap/engines/engine.py.230: loss: 0.20535 [I 2022-10-27 21:17:15] ...optimizers/trijoint.py.151: total_norm: 21.416544 [I 2022-10-27 21:17:20] ...trap/engines/engine.py.224: train: epoch 88 | batch 1780/2304 [I 2022-10-27 21:17:20] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:51 [I 2022-10-27 21:17:20] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00242 [I 2022-10-27 21:17:20] ...trap/engines/engine.py.230: loss: 0.18681 [I 2022-10-27 21:17:20] ...optimizers/trijoint.py.151: total_norm: 15.024643 [I 2022-10-27 21:17:26] ...trap/engines/engine.py.224: train: epoch 88 | batch 1790/2304 [I 2022-10-27 21:17:26] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:45 [I 2022-10-27 21:17:26] ...trap/engines/engine.py.229: process: 0.54756 | load: 0.00252 [I 2022-10-27 21:17:26] ...trap/engines/engine.py.230: loss: 0.10525 [I 2022-10-27 21:17:26] ...optimizers/trijoint.py.151: total_norm: 14.987455 [I 2022-10-27 21:17:32] ...trap/engines/engine.py.224: train: epoch 88 | batch 1800/2304 [I 2022-10-27 21:17:32] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:39 [I 2022-10-27 21:17:32] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00250 [I 2022-10-27 21:17:32] ...trap/engines/engine.py.230: loss: 0.13732 [I 2022-10-27 21:17:32] ...optimizers/trijoint.py.151: total_norm: 9.725430 [I 2022-10-27 21:17:37] ...trap/engines/engine.py.224: train: epoch 88 | batch 1810/2304 [I 2022-10-27 21:17:37] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:34 [I 2022-10-27 21:17:37] ...trap/engines/engine.py.229: process: 0.54945 | load: 0.00242 [I 2022-10-27 21:17:37] ...trap/engines/engine.py.230: loss: 0.10764 [I 2022-10-27 21:17:37] ...optimizers/trijoint.py.151: total_norm: 18.392714 [I 2022-10-27 21:17:43] ...trap/engines/engine.py.224: train: epoch 88 | batch 1820/2304 [I 2022-10-27 21:17:43] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:28 [I 2022-10-27 21:17:43] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00250 [I 2022-10-27 21:17:43] ...trap/engines/engine.py.230: loss: 0.13197 [I 2022-10-27 21:17:43] ...optimizers/trijoint.py.151: total_norm: 14.438918 [I 2022-10-27 21:17:48] ...trap/engines/engine.py.224: train: epoch 88 | batch 1830/2304 [I 2022-10-27 21:17:48] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:22 [I 2022-10-27 21:17:48] ...trap/engines/engine.py.229: process: 0.56663 | load: 0.00246 [I 2022-10-27 21:17:48] ...trap/engines/engine.py.230: loss: 0.18197 [I 2022-10-27 21:17:48] ...optimizers/trijoint.py.151: total_norm: 22.187010 [I 2022-10-27 21:17:54] ...trap/engines/engine.py.224: train: epoch 88 | batch 1840/2304 [I 2022-10-27 21:17:54] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:20 [I 2022-10-27 21:17:54] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00249 [I 2022-10-27 21:17:54] ...trap/engines/engine.py.230: loss: 0.09859 [I 2022-10-27 21:17:54] ...optimizers/trijoint.py.151: total_norm: 12.602823 [I 2022-10-27 21:18:00] ...trap/engines/engine.py.224: train: epoch 88 | batch 1850/2304 [I 2022-10-27 21:18:00] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:12 [I 2022-10-27 21:18:00] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00251 [I 2022-10-27 21:18:00] ...trap/engines/engine.py.230: loss: 0.13424 [I 2022-10-27 21:18:00] ...optimizers/trijoint.py.151: total_norm: 13.637679 [I 2022-10-27 21:18:05] ...trap/engines/engine.py.224: train: epoch 88 | batch 1860/2304 [I 2022-10-27 21:18:05] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:05 [I 2022-10-27 21:18:05] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00252 [I 2022-10-27 21:18:05] ...trap/engines/engine.py.230: loss: 0.13164 [I 2022-10-27 21:18:05] ...optimizers/trijoint.py.151: total_norm: 24.658699 [I 2022-10-27 21:18:11] ...trap/engines/engine.py.224: train: epoch 88 | batch 1870/2304 [I 2022-10-27 21:18:11] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:00 [I 2022-10-27 21:18:11] ...trap/engines/engine.py.229: process: 0.55561 | load: 0.00250 [I 2022-10-27 21:18:11] ...trap/engines/engine.py.230: loss: 0.12239 [I 2022-10-27 21:18:11] ...optimizers/trijoint.py.151: total_norm: 14.054207 [I 2022-10-27 21:18:16] ...trap/engines/engine.py.224: train: epoch 88 | batch 1880/2304 [I 2022-10-27 21:18:16] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:56 [I 2022-10-27 21:18:16] ...trap/engines/engine.py.229: process: 0.55865 | load: 0.00245 [I 2022-10-27 21:18:16] ...trap/engines/engine.py.230: loss: 0.10995 [I 2022-10-27 21:18:16] ...optimizers/trijoint.py.151: total_norm: 10.265794 [I 2022-10-27 21:18:22] ...trap/engines/engine.py.224: train: epoch 88 | batch 1890/2304 [I 2022-10-27 21:18:22] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:03:50 [I 2022-10-27 21:18:22] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00245 [I 2022-10-27 21:18:22] ...trap/engines/engine.py.230: loss: 0.18170 [I 2022-10-27 21:18:22] ...optimizers/trijoint.py.151: total_norm: 11.824764 [I 2022-10-27 21:18:27] ...trap/engines/engine.py.224: train: epoch 88 | batch 1900/2304 [I 2022-10-27 21:18:27] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:44 [I 2022-10-27 21:18:27] ...trap/engines/engine.py.229: process: 0.55054 | load: 0.00249 [I 2022-10-27 21:18:27] ...trap/engines/engine.py.230: loss: 0.10634 [I 2022-10-27 21:18:27] ...optimizers/trijoint.py.151: total_norm: 9.605313 [I 2022-10-27 21:18:33] ...trap/engines/engine.py.224: train: epoch 88 | batch 1910/2304 [I 2022-10-27 21:18:33] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:39 [I 2022-10-27 21:18:33] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00248 [I 2022-10-27 21:18:33] ...trap/engines/engine.py.230: loss: 0.12275 [I 2022-10-27 21:18:33] ...optimizers/trijoint.py.151: total_norm: 31.270731 [I 2022-10-27 21:18:38] ...trap/engines/engine.py.224: train: epoch 88 | batch 1920/2304 [I 2022-10-27 21:18:38] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:33 [I 2022-10-27 21:18:38] ...trap/engines/engine.py.229: process: 0.55020 | load: 0.00243 [I 2022-10-27 21:18:38] ...trap/engines/engine.py.230: loss: 0.13186 [I 2022-10-27 21:18:38] ...optimizers/trijoint.py.151: total_norm: 13.660614 [I 2022-10-27 21:18:44] ...trap/engines/engine.py.224: train: epoch 88 | batch 1930/2304 [I 2022-10-27 21:18:44] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:27 [I 2022-10-27 21:18:44] ...trap/engines/engine.py.229: process: 0.54187 | load: 0.00246 [I 2022-10-27 21:18:44] ...trap/engines/engine.py.230: loss: 0.09261 [I 2022-10-27 21:18:44] ...optimizers/trijoint.py.151: total_norm: 19.434460 [I 2022-10-27 21:18:50] ...trap/engines/engine.py.224: train: epoch 88 | batch 1940/2304 [I 2022-10-27 21:18:50] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:21 [I 2022-10-27 21:18:50] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00243 [I 2022-10-27 21:18:50] ...trap/engines/engine.py.230: loss: 0.14182 [I 2022-10-27 21:18:50] ...optimizers/trijoint.py.151: total_norm: 17.139685 [I 2022-10-27 21:18:55] ...trap/engines/engine.py.224: train: epoch 88 | batch 1950/2304 [I 2022-10-27 21:18:55] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:17 [I 2022-10-27 21:18:55] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00243 [I 2022-10-27 21:18:55] ...trap/engines/engine.py.230: loss: 0.10925 [I 2022-10-27 21:18:55] ...optimizers/trijoint.py.151: total_norm: 49.307159 [I 2022-10-27 21:19:01] ...trap/engines/engine.py.224: train: epoch 88 | batch 1960/2304 [I 2022-10-27 21:19:01] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:15 [I 2022-10-27 21:19:01] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00240 [I 2022-10-27 21:19:01] ...trap/engines/engine.py.230: loss: 0.09145 [I 2022-10-27 21:19:01] ...optimizers/trijoint.py.151: total_norm: 12.593877 [I 2022-10-27 21:19:06] ...trap/engines/engine.py.224: train: epoch 88 | batch 1970/2304 [I 2022-10-27 21:19:06] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:06 [I 2022-10-27 21:19:06] ...trap/engines/engine.py.229: process: 0.56673 | load: 0.00240 [I 2022-10-27 21:19:06] ...trap/engines/engine.py.230: loss: 0.16406 [I 2022-10-27 21:19:06] ...optimizers/trijoint.py.151: total_norm: 24.453976 [I 2022-10-27 21:19:12] ...trap/engines/engine.py.224: train: epoch 88 | batch 1980/2304 [I 2022-10-27 21:19:12] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:02:59 [I 2022-10-27 21:19:12] ...trap/engines/engine.py.229: process: 0.53611 | load: 0.00242 [I 2022-10-27 21:19:12] ...trap/engines/engine.py.230: loss: 0.13673 [I 2022-10-27 21:19:12] ...optimizers/trijoint.py.151: total_norm: 17.161440 [I 2022-10-27 21:19:18] ...trap/engines/engine.py.224: train: epoch 88 | batch 1990/2304 [I 2022-10-27 21:19:18] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:02:56 [I 2022-10-27 21:19:18] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00243 [I 2022-10-27 21:19:18] ...trap/engines/engine.py.230: loss: 0.15752 [I 2022-10-27 21:19:18] ...optimizers/trijoint.py.151: total_norm: 13.549559 [I 2022-10-27 21:19:23] ...trap/engines/engine.py.224: train: epoch 88 | batch 2000/2304 [I 2022-10-27 21:19:23] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:02:50 [I 2022-10-27 21:19:23] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00254 [I 2022-10-27 21:19:23] ...trap/engines/engine.py.230: loss: 0.10518 [I 2022-10-27 21:19:23] ...optimizers/trijoint.py.151: total_norm: 10.954216 [I 2022-10-27 21:19:29] ...trap/engines/engine.py.224: train: epoch 88 | batch 2010/2304 [I 2022-10-27 21:19:29] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:02:43 [I 2022-10-27 21:19:29] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00245 [I 2022-10-27 21:19:29] ...trap/engines/engine.py.230: loss: 0.12321 [I 2022-10-27 21:19:29] ...optimizers/trijoint.py.151: total_norm: 10.442621 [I 2022-10-27 21:19:34] ...trap/engines/engine.py.224: train: epoch 88 | batch 2020/2304 [I 2022-10-27 21:19:34] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:38 [I 2022-10-27 21:19:34] ...trap/engines/engine.py.229: process: 0.56712 | load: 0.00242 [I 2022-10-27 21:19:34] ...trap/engines/engine.py.230: loss: 0.19657 [I 2022-10-27 21:19:34] ...optimizers/trijoint.py.151: total_norm: 23.736273 [I 2022-10-27 21:19:40] ...trap/engines/engine.py.224: train: epoch 88 | batch 2030/2304 [I 2022-10-27 21:19:40] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:32 [I 2022-10-27 21:19:40] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00243 [I 2022-10-27 21:19:40] ...trap/engines/engine.py.230: loss: 0.11275 [I 2022-10-27 21:19:40] ...optimizers/trijoint.py.151: total_norm: 13.920120 [I 2022-10-27 21:19:46] ...trap/engines/engine.py.224: train: epoch 88 | batch 2040/2304 [I 2022-10-27 21:19:46] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:26 [I 2022-10-27 21:19:46] ...trap/engines/engine.py.229: process: 0.56318 | load: 0.00241 [I 2022-10-27 21:19:46] ...trap/engines/engine.py.230: loss: 0.16575 [I 2022-10-27 21:19:46] ...optimizers/trijoint.py.151: total_norm: 15.452127 [I 2022-10-27 21:19:51] ...trap/engines/engine.py.224: train: epoch 88 | batch 2050/2304 [I 2022-10-27 21:19:51] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:20 [I 2022-10-27 21:19:51] ...trap/engines/engine.py.229: process: 0.54357 | load: 0.00243 [I 2022-10-27 21:19:51] ...trap/engines/engine.py.230: loss: 0.12934 [I 2022-10-27 21:19:51] ...optimizers/trijoint.py.151: total_norm: 16.131821 [I 2022-10-27 21:19:57] ...trap/engines/engine.py.224: train: epoch 88 | batch 2060/2304 [I 2022-10-27 21:19:57] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:15 [I 2022-10-27 21:19:57] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00247 [I 2022-10-27 21:19:57] ...trap/engines/engine.py.230: loss: 0.13417 [I 2022-10-27 21:19:57] ...optimizers/trijoint.py.151: total_norm: 38.898975 [I 2022-10-27 21:20:03] ...trap/engines/engine.py.224: train: epoch 88 | batch 2070/2304 [I 2022-10-27 21:20:03] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:14 [I 2022-10-27 21:20:03] ...trap/engines/engine.py.229: process: 0.57524 | load: 0.00253 [I 2022-10-27 21:20:03] ...trap/engines/engine.py.230: loss: 0.18995 [I 2022-10-27 21:20:03] ...optimizers/trijoint.py.151: total_norm: 13.839415 [I 2022-10-27 21:20:08] ...trap/engines/engine.py.224: train: epoch 88 | batch 2080/2304 [I 2022-10-27 21:20:08] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:06 [I 2022-10-27 21:20:08] ...trap/engines/engine.py.229: process: 0.54620 | load: 0.00247 [I 2022-10-27 21:20:08] ...trap/engines/engine.py.230: loss: 0.13811 [I 2022-10-27 21:20:08] ...optimizers/trijoint.py.151: total_norm: 29.325815 [I 2022-10-27 21:20:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 2090/2304 [I 2022-10-27 21:20:14] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:02:00 [I 2022-10-27 21:20:14] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00257 [I 2022-10-27 21:20:14] ...trap/engines/engine.py.230: loss: 0.09504 [I 2022-10-27 21:20:14] ...optimizers/trijoint.py.151: total_norm: 12.062783 [I 2022-10-27 21:20:19] ...trap/engines/engine.py.224: train: epoch 88 | batch 2100/2304 [I 2022-10-27 21:20:19] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:01:54 [I 2022-10-27 21:20:19] ...trap/engines/engine.py.229: process: 0.55938 | load: 0.00249 [I 2022-10-27 21:20:19] ...trap/engines/engine.py.230: loss: 0.15399 [I 2022-10-27 21:20:19] ...optimizers/trijoint.py.151: total_norm: 9.825819 [I 2022-10-27 21:20:25] ...trap/engines/engine.py.224: train: epoch 88 | batch 2110/2304 [I 2022-10-27 21:20:25] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:01:48 [I 2022-10-27 21:20:25] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00248 [I 2022-10-27 21:20:25] ...trap/engines/engine.py.230: loss: 0.09115 [I 2022-10-27 21:20:25] ...optimizers/trijoint.py.151: total_norm: 11.381361 [I 2022-10-27 21:20:31] ...trap/engines/engine.py.224: train: epoch 88 | batch 2120/2304 [I 2022-10-27 21:20:31] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:43 [I 2022-10-27 21:20:31] ...trap/engines/engine.py.229: process: 0.55490 | load: 0.00248 [I 2022-10-27 21:20:31] ...trap/engines/engine.py.230: loss: 0.12156 [I 2022-10-27 21:20:31] ...optimizers/trijoint.py.151: total_norm: 32.780098 [I 2022-10-27 21:20:36] ...trap/engines/engine.py.224: train: epoch 88 | batch 2130/2304 [I 2022-10-27 21:20:36] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:36 [I 2022-10-27 21:20:36] ...trap/engines/engine.py.229: process: 0.55350 | load: 0.00246 [I 2022-10-27 21:20:36] ...trap/engines/engine.py.230: loss: 0.11196 [I 2022-10-27 21:20:36] ...optimizers/trijoint.py.151: total_norm: 25.434286 [I 2022-10-27 21:20:42] ...trap/engines/engine.py.224: train: epoch 88 | batch 2140/2304 [I 2022-10-27 21:20:42] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:30 [I 2022-10-27 21:20:42] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00245 [I 2022-10-27 21:20:42] ...trap/engines/engine.py.230: loss: 0.08975 [I 2022-10-27 21:20:42] ...optimizers/trijoint.py.151: total_norm: 15.956481 [I 2022-10-27 21:20:47] ...trap/engines/engine.py.224: train: epoch 88 | batch 2150/2304 [I 2022-10-27 21:20:47] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:25 [I 2022-10-27 21:20:47] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00243 [I 2022-10-27 21:20:47] ...trap/engines/engine.py.230: loss: 0.22426 [I 2022-10-27 21:20:47] ...optimizers/trijoint.py.151: total_norm: 14.033778 [I 2022-10-27 21:20:53] ...trap/engines/engine.py.224: train: epoch 88 | batch 2160/2304 [I 2022-10-27 21:20:53] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:20 [I 2022-10-27 21:20:53] ...trap/engines/engine.py.229: process: 0.55030 | load: 0.00244 [I 2022-10-27 21:20:53] ...trap/engines/engine.py.230: loss: 0.08883 [I 2022-10-27 21:20:53] ...optimizers/trijoint.py.151: total_norm: 10.952396 [I 2022-10-27 21:20:59] ...trap/engines/engine.py.224: train: epoch 88 | batch 2170/2304 [I 2022-10-27 21:20:59] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:14 [I 2022-10-27 21:20:59] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00241 [I 2022-10-27 21:20:59] ...trap/engines/engine.py.230: loss: 0.13702 [I 2022-10-27 21:20:59] ...optimizers/trijoint.py.151: total_norm: 24.509283 [I 2022-10-27 21:21:04] ...trap/engines/engine.py.224: train: epoch 88 | batch 2180/2304 [I 2022-10-27 21:21:04] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:09 [I 2022-10-27 21:21:04] ...trap/engines/engine.py.229: process: 0.56753 | load: 0.00245 [I 2022-10-27 21:21:04] ...trap/engines/engine.py.230: loss: 0.13925 [I 2022-10-27 21:21:04] ...optimizers/trijoint.py.151: total_norm: 18.477121 [I 2022-10-27 21:21:10] ...trap/engines/engine.py.224: train: epoch 88 | batch 2190/2304 [I 2022-10-27 21:21:10] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:04 [I 2022-10-27 21:21:10] ...trap/engines/engine.py.229: process: 0.56153 | load: 0.00249 [I 2022-10-27 21:21:10] ...trap/engines/engine.py.230: loss: 0.13231 [I 2022-10-27 21:21:10] ...optimizers/trijoint.py.151: total_norm: 28.980740 [I 2022-10-27 21:21:15] ...trap/engines/engine.py.224: train: epoch 88 | batch 2200/2304 [I 2022-10-27 21:21:15] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:00:58 [I 2022-10-27 21:21:15] ...trap/engines/engine.py.229: process: 0.55419 | load: 0.00249 [I 2022-10-27 21:21:15] ...trap/engines/engine.py.230: loss: 0.12178 [I 2022-10-27 21:21:15] ...optimizers/trijoint.py.151: total_norm: 22.384640 [I 2022-10-27 21:21:21] ...trap/engines/engine.py.224: train: epoch 88 | batch 2210/2304 [I 2022-10-27 21:21:21] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:00:52 [I 2022-10-27 21:21:21] ...trap/engines/engine.py.229: process: 0.55814 | load: 0.00247 [I 2022-10-27 21:21:21] ...trap/engines/engine.py.230: loss: 0.23167 [I 2022-10-27 21:21:21] ...optimizers/trijoint.py.151: total_norm: 15.521350 [I 2022-10-27 21:21:27] ...trap/engines/engine.py.224: train: epoch 88 | batch 2220/2304 [I 2022-10-27 21:21:27] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:00:47 [I 2022-10-27 21:21:27] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00243 [I 2022-10-27 21:21:27] ...trap/engines/engine.py.230: loss: 0.12619 [I 2022-10-27 21:21:27] ...optimizers/trijoint.py.151: total_norm: 7.810167 [I 2022-10-27 21:21:32] ...trap/engines/engine.py.224: train: epoch 88 | batch 2230/2304 [I 2022-10-27 21:21:32] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:41 [I 2022-10-27 21:21:32] ...trap/engines/engine.py.229: process: 0.55924 | load: 0.00248 [I 2022-10-27 21:21:32] ...trap/engines/engine.py.230: loss: 0.14061 [I 2022-10-27 21:21:32] ...optimizers/trijoint.py.151: total_norm: 10.023129 [I 2022-10-27 21:21:38] ...trap/engines/engine.py.224: train: epoch 88 | batch 2240/2304 [I 2022-10-27 21:21:38] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:35 [I 2022-10-27 21:21:38] ...trap/engines/engine.py.229: process: 0.55215 | load: 0.00252 [I 2022-10-27 21:21:38] ...trap/engines/engine.py.230: loss: 0.12928 [I 2022-10-27 21:21:38] ...optimizers/trijoint.py.151: total_norm: 20.410940 [I 2022-10-27 21:21:44] ...trap/engines/engine.py.224: train: epoch 88 | batch 2250/2304 [I 2022-10-27 21:21:44] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:30 [I 2022-10-27 21:21:44] ...trap/engines/engine.py.229: process: 0.55576 | load: 0.00252 [I 2022-10-27 21:21:44] ...trap/engines/engine.py.230: loss: 0.11036 [I 2022-10-27 21:21:44] ...optimizers/trijoint.py.151: total_norm: 17.875244 [I 2022-10-27 21:21:49] ...trap/engines/engine.py.224: train: epoch 88 | batch 2260/2304 [I 2022-10-27 21:21:49] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:24 [I 2022-10-27 21:21:49] ...trap/engines/engine.py.229: process: 0.54424 | load: 0.00248 [I 2022-10-27 21:21:49] ...trap/engines/engine.py.230: loss: 0.11349 [I 2022-10-27 21:21:49] ...optimizers/trijoint.py.151: total_norm: 7.331264 [I 2022-10-27 21:21:55] ...trap/engines/engine.py.224: train: epoch 88 | batch 2270/2304 [I 2022-10-27 21:21:55] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:19 [I 2022-10-27 21:21:55] ...trap/engines/engine.py.229: process: 0.56612 | load: 0.00248 [I 2022-10-27 21:21:55] ...trap/engines/engine.py.230: loss: 0.14653 [I 2022-10-27 21:21:55] ...optimizers/trijoint.py.151: total_norm: 92.231934 [I 2022-10-27 21:22:01] ...trap/engines/engine.py.224: train: epoch 88 | batch 2280/2304 [I 2022-10-27 21:22:01] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:13 [I 2022-10-27 21:22:01] ...trap/engines/engine.py.229: process: 0.54467 | load: 0.00243 [I 2022-10-27 21:22:01] ...trap/engines/engine.py.230: loss: 0.11333 [I 2022-10-27 21:22:01] ...optimizers/trijoint.py.151: total_norm: 11.215419 [I 2022-10-27 21:22:06] ...trap/engines/engine.py.224: train: epoch 88 | batch 2290/2304 [I 2022-10-27 21:22:06] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:07 [I 2022-10-27 21:22:06] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00247 [I 2022-10-27 21:22:06] ...trap/engines/engine.py.230: loss: 0.10899 [I 2022-10-27 21:22:06] ...optimizers/trijoint.py.151: total_norm: 12.474968 [I 2022-10-27 21:22:12] ...trap/engines/engine.py.224: train: epoch 88 | batch 2300/2304 [I 2022-10-27 21:22:12] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:02 [I 2022-10-27 21:22:12] ...trap/engines/engine.py.229: process: 0.55977 | load: 0.00015 [I 2022-10-27 21:22:12] ...trap/engines/engine.py.230: loss: 0.09052 [I 2022-10-27 21:22:12] ...optimizers/trijoint.py.151: total_norm: 26.947628 [I 2022-10-27 21:22:14] ...trap/engines/engine.py.224: train: epoch 88 | batch 2304/2304 [I 2022-10-27 21:22:14] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:00 [I 2022-10-27 21:22:14] ...trap/engines/engine.py.229: process: 0.54997 | load: 0.00016 [I 2022-10-27 21:22:14] ...trap/engines/engine.py.230: loss: 0.10399 [I 2022-10-27 21:22:14] ...optimizers/trijoint.py.151: total_norm: 19.579538 [S 2022-10-27 21:22:15] ...trap/engines/engine.py.129: train_epoch.epoch: 88 [S 2022-10-27 21:22:15] ...trap/engines/engine.py.129: train_epoch.loss: 0.13286612765649394 [I 2022-10-27 21:22:15] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 21:22:17] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 88 [I 2022-10-27 21:22:17] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 21:22:19] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 21:22:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 21:22:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 21:22:22] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 21:22:22] ...trap/engines/engine.py.321: eval: epoch 88 | batch 0/510 [I 2022-10-27 21:22:22] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:44:25 [I 2022-10-27 21:22:22] ...trap/engines/engine.py.326: process: 5.22704 | load: 3.02834 [I 2022-10-27 21:22:25] ...trap/engines/engine.py.321: eval: epoch 88 | batch 10/510 [I 2022-10-27 21:22:25] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:10 [I 2022-10-27 21:22:25] ...trap/engines/engine.py.326: process: 0.33401 | load: 0.00019 [I 2022-10-27 21:22:29] ...trap/engines/engine.py.321: eval: epoch 88 | batch 20/510 [I 2022-10-27 21:22:29] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:11 [I 2022-10-27 21:22:29] ...trap/engines/engine.py.326: process: 0.33023 | load: 0.00021 [I 2022-10-27 21:22:32] ...trap/engines/engine.py.321: eval: epoch 88 | batch 30/510 [I 2022-10-27 21:22:32] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:45 [I 2022-10-27 21:22:32] ...trap/engines/engine.py.326: process: 0.33044 | load: 0.00027 [I 2022-10-27 21:22:35] ...trap/engines/engine.py.321: eval: epoch 88 | batch 40/510 [I 2022-10-27 21:22:35] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:39 [I 2022-10-27 21:22:35] ...trap/engines/engine.py.326: process: 0.35720 | load: 0.00019 [I 2022-10-27 21:22:39] ...trap/engines/engine.py.321: eval: epoch 88 | batch 50/510 [I 2022-10-27 21:22:39] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:35 [I 2022-10-27 21:22:39] ...trap/engines/engine.py.326: process: 0.32357 | load: 0.00020 [I 2022-10-27 21:22:42] ...trap/engines/engine.py.321: eval: epoch 88 | batch 60/510 [I 2022-10-27 21:22:42] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:30 [I 2022-10-27 21:22:42] ...trap/engines/engine.py.326: process: 0.32247 | load: 0.00018 [I 2022-10-27 21:22:45] ...trap/engines/engine.py.321: eval: epoch 88 | batch 70/510 [I 2022-10-27 21:22:45] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:27 [I 2022-10-27 21:22:45] ...trap/engines/engine.py.326: process: 0.33926 | load: 0.00019 [I 2022-10-27 21:22:49] ...trap/engines/engine.py.321: eval: epoch 88 | batch 80/510 [I 2022-10-27 21:22:49] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:22 [I 2022-10-27 21:22:49] ...trap/engines/engine.py.326: process: 0.32827 | load: 0.00017 [I 2022-10-27 21:22:52] ...trap/engines/engine.py.321: eval: epoch 88 | batch 90/510 [I 2022-10-27 21:22:52] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:20 [I 2022-10-27 21:22:52] ...trap/engines/engine.py.326: process: 0.33664 | load: 0.00018 [I 2022-10-27 21:22:55] ...trap/engines/engine.py.321: eval: epoch 88 | batch 100/510 [I 2022-10-27 21:22:55] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:14 [I 2022-10-27 21:22:55] ...trap/engines/engine.py.326: process: 0.27993 | load: 0.00022 [I 2022-10-27 21:22:58] ...trap/engines/engine.py.321: eval: epoch 88 | batch 110/510 [I 2022-10-27 21:22:58] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 21:22:58] ...trap/engines/engine.py.326: process: 0.28448 | load: 0.00017 [I 2022-10-27 21:23:01] ...trap/engines/engine.py.321: eval: epoch 88 | batch 120/510 [I 2022-10-27 21:23:01] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:53 [I 2022-10-27 21:23:01] ...trap/engines/engine.py.326: process: 0.28768 | load: 0.00022 [I 2022-10-27 21:23:04] ...trap/engines/engine.py.321: eval: epoch 88 | batch 130/510 [I 2022-10-27 21:23:04] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:50 [I 2022-10-27 21:23:04] ...trap/engines/engine.py.326: process: 0.28597 | load: 0.00015 [I 2022-10-27 21:23:07] ...trap/engines/engine.py.321: eval: epoch 88 | batch 140/510 [I 2022-10-27 21:23:07] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-27 21:23:07] ...trap/engines/engine.py.326: process: 0.28707 | load: 0.00023 [I 2022-10-27 21:23:10] ...trap/engines/engine.py.321: eval: epoch 88 | batch 150/510 [I 2022-10-27 21:23:10] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-27 21:23:10] ...trap/engines/engine.py.326: process: 0.29088 | load: 0.00016 [I 2022-10-27 21:23:13] ...trap/engines/engine.py.321: eval: epoch 88 | batch 160/510 [I 2022-10-27 21:23:13] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-27 21:23:13] ...trap/engines/engine.py.326: process: 0.28486 | load: 0.00015 [I 2022-10-27 21:23:16] ...trap/engines/engine.py.321: eval: epoch 88 | batch 170/510 [I 2022-10-27 21:23:16] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-27 21:23:16] ...trap/engines/engine.py.326: process: 0.28922 | load: 0.00015 [I 2022-10-27 21:23:19] ...trap/engines/engine.py.321: eval: epoch 88 | batch 180/510 [I 2022-10-27 21:23:19] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 21:23:19] ...trap/engines/engine.py.326: process: 0.28772 | load: 0.00014 [I 2022-10-27 21:23:22] ...trap/engines/engine.py.321: eval: epoch 88 | batch 190/510 [I 2022-10-27 21:23:22] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 21:23:22] ...trap/engines/engine.py.326: process: 0.29091 | load: 0.00018 [I 2022-10-27 21:23:24] ...trap/engines/engine.py.321: eval: epoch 88 | batch 200/510 [I 2022-10-27 21:23:24] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 21:23:24] ...trap/engines/engine.py.326: process: 0.29228 | load: 0.00015 [I 2022-10-27 21:23:27] ...trap/engines/engine.py.321: eval: epoch 88 | batch 210/510 [I 2022-10-27 21:23:27] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:27 [I 2022-10-27 21:23:27] ...trap/engines/engine.py.326: process: 0.29887 | load: 0.00017 [I 2022-10-27 21:23:30] ...trap/engines/engine.py.321: eval: epoch 88 | batch 220/510 [I 2022-10-27 21:23:30] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 21:23:30] ...trap/engines/engine.py.326: process: 0.28060 | load: 0.00017 [I 2022-10-27 21:23:33] ...trap/engines/engine.py.321: eval: epoch 88 | batch 230/510 [I 2022-10-27 21:23:33] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 21:23:33] ...trap/engines/engine.py.326: process: 0.28883 | load: 0.00015 [I 2022-10-27 21:23:36] ...trap/engines/engine.py.321: eval: epoch 88 | batch 240/510 [I 2022-10-27 21:23:36] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-27 21:23:36] ...trap/engines/engine.py.326: process: 0.28236 | load: 0.00015 [I 2022-10-27 21:23:39] ...trap/engines/engine.py.321: eval: epoch 88 | batch 250/510 [I 2022-10-27 21:23:39] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-27 21:23:39] ...trap/engines/engine.py.326: process: 0.28712 | load: 0.00016 [I 2022-10-27 21:23:42] ...trap/engines/engine.py.321: eval: epoch 88 | batch 260/510 [I 2022-10-27 21:23:42] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-27 21:23:42] ...trap/engines/engine.py.326: process: 0.28691 | load: 0.00015 [I 2022-10-27 21:23:45] ...trap/engines/engine.py.321: eval: epoch 88 | batch 270/510 [I 2022-10-27 21:23:45] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-27 21:23:45] ...trap/engines/engine.py.326: process: 0.28479 | load: 0.00019 [I 2022-10-27 21:23:48] ...trap/engines/engine.py.321: eval: epoch 88 | batch 280/510 [I 2022-10-27 21:23:48] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 21:23:48] ...trap/engines/engine.py.326: process: 0.28568 | load: 0.00021 [I 2022-10-27 21:23:50] ...trap/engines/engine.py.321: eval: epoch 88 | batch 290/510 [I 2022-10-27 21:23:50] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:02 [I 2022-10-27 21:23:50] ...trap/engines/engine.py.326: process: 0.28288 | load: 0.00015 [I 2022-10-27 21:23:53] ...trap/engines/engine.py.321: eval: epoch 88 | batch 300/510 [I 2022-10-27 21:23:53] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 21:23:53] ...trap/engines/engine.py.326: process: 0.28337 | load: 0.00015 [I 2022-10-27 21:23:56] ...trap/engines/engine.py.321: eval: epoch 88 | batch 310/510 [I 2022-10-27 21:23:56] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 21:23:56] ...trap/engines/engine.py.326: process: 0.28637 | load: 0.00015 [I 2022-10-27 21:23:59] ...trap/engines/engine.py.321: eval: epoch 88 | batch 320/510 [I 2022-10-27 21:23:59] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 21:23:59] ...trap/engines/engine.py.326: process: 0.28083 | load: 0.00015 [I 2022-10-27 21:24:02] ...trap/engines/engine.py.321: eval: epoch 88 | batch 330/510 [I 2022-10-27 21:24:02] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-27 21:24:02] ...trap/engines/engine.py.326: process: 0.29353 | load: 0.00015 [I 2022-10-27 21:24:05] ...trap/engines/engine.py.321: eval: epoch 88 | batch 340/510 [I 2022-10-27 21:24:05] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-27 21:24:05] ...trap/engines/engine.py.326: process: 0.28305 | load: 0.00014 [I 2022-10-27 21:24:08] ...trap/engines/engine.py.321: eval: epoch 88 | batch 350/510 [I 2022-10-27 21:24:08] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:45 [I 2022-10-27 21:24:08] ...trap/engines/engine.py.326: process: 0.28616 | load: 0.00015 [I 2022-10-27 21:24:11] ...trap/engines/engine.py.321: eval: epoch 88 | batch 360/510 [I 2022-10-27 21:24:11] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 21:24:11] ...trap/engines/engine.py.326: process: 0.28550 | load: 0.00015 [I 2022-10-27 21:24:13] ...trap/engines/engine.py.321: eval: epoch 88 | batch 370/510 [I 2022-10-27 21:24:13] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 21:24:13] ...trap/engines/engine.py.326: process: 0.28859 | load: 0.00015 [I 2022-10-27 21:24:16] ...trap/engines/engine.py.321: eval: epoch 88 | batch 380/510 [I 2022-10-27 21:24:16] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 21:24:16] ...trap/engines/engine.py.326: process: 0.28848 | load: 0.00016 [I 2022-10-27 21:24:19] ...trap/engines/engine.py.321: eval: epoch 88 | batch 390/510 [I 2022-10-27 21:24:19] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 21:24:19] ...trap/engines/engine.py.326: process: 0.28909 | load: 0.00016 [I 2022-10-27 21:24:22] ...trap/engines/engine.py.321: eval: epoch 88 | batch 400/510 [I 2022-10-27 21:24:22] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 21:24:22] ...trap/engines/engine.py.326: process: 0.28903 | load: 0.00016 [I 2022-10-27 21:24:25] ...trap/engines/engine.py.321: eval: epoch 88 | batch 410/510 [I 2022-10-27 21:24:25] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 21:24:25] ...trap/engines/engine.py.326: process: 0.28860 | load: 0.00031 [I 2022-10-27 21:24:28] ...trap/engines/engine.py.321: eval: epoch 88 | batch 420/510 [I 2022-10-27 21:24:28] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 21:24:28] ...trap/engines/engine.py.326: process: 0.28158 | load: 0.00015 [I 2022-10-27 21:24:31] ...trap/engines/engine.py.321: eval: epoch 88 | batch 430/510 [I 2022-10-27 21:24:31] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:23 [I 2022-10-27 21:24:31] ...trap/engines/engine.py.326: process: 0.28676 | load: 0.00015 [I 2022-10-27 21:24:34] ...trap/engines/engine.py.321: eval: epoch 88 | batch 440/510 [I 2022-10-27 21:24:34] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-27 21:24:34] ...trap/engines/engine.py.326: process: 0.29110 | load: 0.00015 [I 2022-10-27 21:24:36] ...trap/engines/engine.py.321: eval: epoch 88 | batch 450/510 [I 2022-10-27 21:24:36] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 21:24:36] ...trap/engines/engine.py.326: process: 0.29066 | load: 0.00016 [I 2022-10-27 21:24:39] ...trap/engines/engine.py.321: eval: epoch 88 | batch 460/510 [I 2022-10-27 21:24:39] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 21:24:39] ...trap/engines/engine.py.326: process: 0.28819 | load: 0.00016 [I 2022-10-27 21:24:42] ...trap/engines/engine.py.321: eval: epoch 88 | batch 470/510 [I 2022-10-27 21:24:42] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 21:24:42] ...trap/engines/engine.py.326: process: 0.28278 | load: 0.00015 [I 2022-10-27 21:24:45] ...trap/engines/engine.py.321: eval: epoch 88 | batch 480/510 [I 2022-10-27 21:24:45] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 21:24:45] ...trap/engines/engine.py.326: process: 0.28639 | load: 0.00019 [I 2022-10-27 21:24:48] ...trap/engines/engine.py.321: eval: epoch 88 | batch 490/510 [I 2022-10-27 21:24:48] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 21:24:48] ...trap/engines/engine.py.326: process: 0.28930 | load: 0.00015 [I 2022-10-27 21:24:51] ...trap/engines/engine.py.321: eval: epoch 88 | batch 500/510 [I 2022-10-27 21:24:51] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 21:24:51] ...trap/engines/engine.py.326: process: 0.28871 | load: 0.00016 [I 2022-10-27 21:24:54] ...trap/engines/engine.py.321: eval: epoch 88 | batch 510/510 [I 2022-10-27 21:24:54] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-27 21:24:54] ...trap/engines/engine.py.326: process: 0.28256 | load: 0.00015 [S 2022-10-27 21:24:54] ...trap/engines/engine.py.132: eval_epoch.epoch: 88 [I 2022-10-27 21:24:54] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.732300 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908500 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937500 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.723300 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909100 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935200 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013350 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.003170 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.003008 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009930 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.003330 [S 2022-10-27 21:24:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005115 [I 2022-10-27 21:25:00] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 21:25:00] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 21:25:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 21:25:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 21:25:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 21:25:05] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 21:25:06] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 21:25:11] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 21:25:11] ...trap/engines/engine.py.163: Training model on trainset for epoch 89 [I 2022-10-27 21:25:11] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 21:25:14] ...trap/engines/engine.py.224: train: epoch 89 | batch 0/2304 [I 2022-10-27 21:25:14] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:18 [I 2022-10-27 21:25:14] ...trap/engines/engine.py.229: process: 2.63830 | load: 2.02918 [I 2022-10-27 21:25:14] ...trap/engines/engine.py.230: loss: 0.12747 [I 2022-10-27 21:25:14] ...optimizers/trijoint.py.151: total_norm: 7.903976 [I 2022-10-27 21:25:20] ...trap/engines/engine.py.224: train: epoch 89 | batch 10/2304 [I 2022-10-27 21:25:20] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:00 [I 2022-10-27 21:25:20] ...trap/engines/engine.py.229: process: 0.56637 | load: 0.00448 [I 2022-10-27 21:25:20] ...trap/engines/engine.py.230: loss: 0.21858 [I 2022-10-27 21:25:20] ...optimizers/trijoint.py.151: total_norm: 28.252497 [I 2022-10-27 21:25:25] ...trap/engines/engine.py.224: train: epoch 89 | batch 20/2304 [I 2022-10-27 21:25:25] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:12 [I 2022-10-27 21:25:25] ...trap/engines/engine.py.229: process: 0.57819 | load: 0.00347 [I 2022-10-27 21:25:25] ...trap/engines/engine.py.230: loss: 0.12507 [I 2022-10-27 21:25:25] ...optimizers/trijoint.py.151: total_norm: 15.945539 [I 2022-10-27 21:25:31] ...trap/engines/engine.py.224: train: epoch 89 | batch 30/2304 [I 2022-10-27 21:25:31] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:14 [I 2022-10-27 21:25:31] ...trap/engines/engine.py.229: process: 0.55212 | load: 0.00335 [I 2022-10-27 21:25:31] ...trap/engines/engine.py.230: loss: 0.13830 [I 2022-10-27 21:25:31] ...optimizers/trijoint.py.151: total_norm: 15.931566 [I 2022-10-27 21:25:36] ...trap/engines/engine.py.224: train: epoch 89 | batch 40/2304 [I 2022-10-27 21:25:36] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:55 [I 2022-10-27 21:25:36] ...trap/engines/engine.py.229: process: 0.55205 | load: 0.00288 [I 2022-10-27 21:25:36] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-27 21:25:36] ...optimizers/trijoint.py.151: total_norm: 15.275564 [I 2022-10-27 21:25:42] ...trap/engines/engine.py.224: train: epoch 89 | batch 50/2304 [I 2022-10-27 21:25:42] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:51 [I 2022-10-27 21:25:42] ...trap/engines/engine.py.229: process: 0.56753 | load: 0.00295 [I 2022-10-27 21:25:42] ...trap/engines/engine.py.230: loss: 0.08999 [I 2022-10-27 21:25:42] ...optimizers/trijoint.py.151: total_norm: 9.712193 [I 2022-10-27 21:25:47] ...trap/engines/engine.py.224: train: epoch 89 | batch 60/2304 [I 2022-10-27 21:25:47] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:49 [I 2022-10-27 21:25:47] ...trap/engines/engine.py.229: process: 0.56807 | load: 0.00264 [I 2022-10-27 21:25:47] ...trap/engines/engine.py.230: loss: 0.09945 [I 2022-10-27 21:25:47] ...optimizers/trijoint.py.151: total_norm: 17.230307 [I 2022-10-27 21:25:53] ...trap/engines/engine.py.224: train: epoch 89 | batch 70/2304 [I 2022-10-27 21:25:53] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:54 [I 2022-10-27 21:25:53] ...trap/engines/engine.py.229: process: 0.56936 | load: 0.00298 [I 2022-10-27 21:25:53] ...trap/engines/engine.py.230: loss: 0.15378 [I 2022-10-27 21:25:53] ...optimizers/trijoint.py.151: total_norm: 36.828659 [I 2022-10-27 21:25:59] ...trap/engines/engine.py.224: train: epoch 89 | batch 80/2304 [I 2022-10-27 21:25:59] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:47 [I 2022-10-27 21:25:59] ...trap/engines/engine.py.229: process: 0.56387 | load: 0.00255 [I 2022-10-27 21:25:59] ...trap/engines/engine.py.230: loss: 0.13653 [I 2022-10-27 21:25:59] ...optimizers/trijoint.py.151: total_norm: 27.963274 [I 2022-10-27 21:26:04] ...trap/engines/engine.py.224: train: epoch 89 | batch 90/2304 [I 2022-10-27 21:26:04] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:40 [I 2022-10-27 21:26:04] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00287 [I 2022-10-27 21:26:04] ...trap/engines/engine.py.230: loss: 0.10054 [I 2022-10-27 21:26:04] ...optimizers/trijoint.py.151: total_norm: 17.159098 [I 2022-10-27 21:26:10] ...trap/engines/engine.py.224: train: epoch 89 | batch 100/2304 [I 2022-10-27 21:26:10] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:50 [I 2022-10-27 21:26:10] ...trap/engines/engine.py.229: process: 0.59379 | load: 0.00295 [I 2022-10-27 21:26:10] ...trap/engines/engine.py.230: loss: 0.15677 [I 2022-10-27 21:26:10] ...optimizers/trijoint.py.151: total_norm: 27.157162 [I 2022-10-27 21:26:15] ...trap/engines/engine.py.224: train: epoch 89 | batch 110/2304 [I 2022-10-27 21:26:15] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:24 [I 2022-10-27 21:26:15] ...trap/engines/engine.py.229: process: 0.56464 | load: 0.00272 [I 2022-10-27 21:26:15] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-27 21:26:15] ...optimizers/trijoint.py.151: total_norm: 10.321896 [I 2022-10-27 21:26:21] ...trap/engines/engine.py.224: train: epoch 89 | batch 120/2304 [I 2022-10-27 21:26:21] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:17 [I 2022-10-27 21:26:21] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00264 [I 2022-10-27 21:26:21] ...trap/engines/engine.py.230: loss: 0.13377 [I 2022-10-27 21:26:21] ...optimizers/trijoint.py.151: total_norm: 19.085039 [I 2022-10-27 21:26:27] ...trap/engines/engine.py.224: train: epoch 89 | batch 130/2304 [I 2022-10-27 21:26:27] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:32 [I 2022-10-27 21:26:27] ...trap/engines/engine.py.229: process: 0.54160 | load: 0.00252 [I 2022-10-27 21:26:27] ...trap/engines/engine.py.230: loss: 0.16901 [I 2022-10-27 21:26:27] ...optimizers/trijoint.py.151: total_norm: 10.935638 [I 2022-10-27 21:26:32] ...trap/engines/engine.py.224: train: epoch 89 | batch 140/2304 [I 2022-10-27 21:26:32] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:11 [I 2022-10-27 21:26:32] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00254 [I 2022-10-27 21:26:32] ...trap/engines/engine.py.230: loss: 0.11119 [I 2022-10-27 21:26:32] ...optimizers/trijoint.py.151: total_norm: 12.273990 [I 2022-10-27 21:26:38] ...trap/engines/engine.py.224: train: epoch 89 | batch 150/2304 [I 2022-10-27 21:26:38] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:36 [I 2022-10-27 21:26:38] ...trap/engines/engine.py.229: process: 0.54956 | load: 0.00261 [I 2022-10-27 21:26:38] ...trap/engines/engine.py.230: loss: 0.12316 [I 2022-10-27 21:26:38] ...optimizers/trijoint.py.151: total_norm: 61.264446 [I 2022-10-27 21:26:44] ...trap/engines/engine.py.224: train: epoch 89 | batch 160/2304 [I 2022-10-27 21:26:44] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:00 [I 2022-10-27 21:26:44] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00266 [I 2022-10-27 21:26:44] ...trap/engines/engine.py.230: loss: 0.12186 [I 2022-10-27 21:26:44] ...optimizers/trijoint.py.151: total_norm: 19.680719 [I 2022-10-27 21:26:49] ...trap/engines/engine.py.224: train: epoch 89 | batch 170/2304 [I 2022-10-27 21:26:49] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:42 [I 2022-10-27 21:26:49] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00256 [I 2022-10-27 21:26:49] ...trap/engines/engine.py.230: loss: 0.14127 [I 2022-10-27 21:26:49] ...optimizers/trijoint.py.151: total_norm: 12.728820 [I 2022-10-27 21:26:55] ...trap/engines/engine.py.224: train: epoch 89 | batch 180/2304 [I 2022-10-27 21:26:55] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:41 [I 2022-10-27 21:26:55] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00243 [I 2022-10-27 21:26:55] ...trap/engines/engine.py.230: loss: 0.11857 [I 2022-10-27 21:26:55] ...optimizers/trijoint.py.151: total_norm: 16.218479 [I 2022-10-27 21:27:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 190/2304 [I 2022-10-27 21:27:00] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:33 [I 2022-10-27 21:27:00] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00244 [I 2022-10-27 21:27:00] ...trap/engines/engine.py.230: loss: 0.13480 [I 2022-10-27 21:27:00] ...optimizers/trijoint.py.151: total_norm: 13.953341 [I 2022-10-27 21:27:06] ...trap/engines/engine.py.224: train: epoch 89 | batch 200/2304 [I 2022-10-27 21:27:06] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:46 [I 2022-10-27 21:27:06] ...trap/engines/engine.py.229: process: 0.56870 | load: 0.00262 [I 2022-10-27 21:27:06] ...trap/engines/engine.py.230: loss: 0.10943 [I 2022-10-27 21:27:06] ...optimizers/trijoint.py.151: total_norm: 13.178189 [I 2022-10-27 21:27:12] ...trap/engines/engine.py.224: train: epoch 89 | batch 210/2304 [I 2022-10-27 21:27:12] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:29 [I 2022-10-27 21:27:12] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00247 [I 2022-10-27 21:27:12] ...trap/engines/engine.py.230: loss: 0.14711 [I 2022-10-27 21:27:12] ...optimizers/trijoint.py.151: total_norm: 39.301517 [I 2022-10-27 21:27:17] ...trap/engines/engine.py.224: train: epoch 89 | batch 220/2304 [I 2022-10-27 21:27:17] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:22 [I 2022-10-27 21:27:17] ...trap/engines/engine.py.229: process: 0.56477 | load: 0.00250 [I 2022-10-27 21:27:17] ...trap/engines/engine.py.230: loss: 0.13264 [I 2022-10-27 21:27:17] ...optimizers/trijoint.py.151: total_norm: 30.725582 [I 2022-10-27 21:27:23] ...trap/engines/engine.py.224: train: epoch 89 | batch 230/2304 [I 2022-10-27 21:27:23] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:31 [I 2022-10-27 21:27:23] ...trap/engines/engine.py.229: process: 0.57067 | load: 0.00249 [I 2022-10-27 21:27:23] ...trap/engines/engine.py.230: loss: 0.12949 [I 2022-10-27 21:27:23] ...optimizers/trijoint.py.151: total_norm: 11.286709 [I 2022-10-27 21:27:28] ...trap/engines/engine.py.224: train: epoch 89 | batch 240/2304 [I 2022-10-27 21:27:28] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:10 [I 2022-10-27 21:27:28] ...trap/engines/engine.py.229: process: 0.54960 | load: 0.00250 [I 2022-10-27 21:27:28] ...trap/engines/engine.py.230: loss: 0.11810 [I 2022-10-27 21:27:28] ...optimizers/trijoint.py.151: total_norm: 14.228122 [I 2022-10-27 21:27:34] ...trap/engines/engine.py.224: train: epoch 89 | batch 250/2304 [I 2022-10-27 21:27:34] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:08 [I 2022-10-27 21:27:34] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00261 [I 2022-10-27 21:27:34] ...trap/engines/engine.py.230: loss: 0.12172 [I 2022-10-27 21:27:34] ...optimizers/trijoint.py.151: total_norm: 9.697783 [I 2022-10-27 21:27:40] ...trap/engines/engine.py.224: train: epoch 89 | batch 260/2304 [I 2022-10-27 21:27:40] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:02 [I 2022-10-27 21:27:40] ...trap/engines/engine.py.229: process: 0.56258 | load: 0.00247 [I 2022-10-27 21:27:40] ...trap/engines/engine.py.230: loss: 0.11529 [I 2022-10-27 21:27:40] ...optimizers/trijoint.py.151: total_norm: 14.490110 [I 2022-10-27 21:27:45] ...trap/engines/engine.py.224: train: epoch 89 | batch 270/2304 [I 2022-10-27 21:27:45] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:56 [I 2022-10-27 21:27:45] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00260 [I 2022-10-27 21:27:45] ...trap/engines/engine.py.230: loss: 0.12249 [I 2022-10-27 21:27:45] ...optimizers/trijoint.py.151: total_norm: 16.827150 [I 2022-10-27 21:27:51] ...trap/engines/engine.py.224: train: epoch 89 | batch 280/2304 [I 2022-10-27 21:27:51] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:46 [I 2022-10-27 21:27:51] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00256 [I 2022-10-27 21:27:51] ...trap/engines/engine.py.230: loss: 0.19033 [I 2022-10-27 21:27:51] ...optimizers/trijoint.py.151: total_norm: 18.212416 [I 2022-10-27 21:27:56] ...trap/engines/engine.py.224: train: epoch 89 | batch 290/2304 [I 2022-10-27 21:27:56] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:56 [I 2022-10-27 21:27:56] ...trap/engines/engine.py.229: process: 0.56622 | load: 0.00248 [I 2022-10-27 21:27:56] ...trap/engines/engine.py.230: loss: 0.19016 [I 2022-10-27 21:27:56] ...optimizers/trijoint.py.151: total_norm: 11.968327 [I 2022-10-27 21:28:02] ...trap/engines/engine.py.224: train: epoch 89 | batch 300/2304 [I 2022-10-27 21:28:02] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:41 [I 2022-10-27 21:28:02] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00249 [I 2022-10-27 21:28:02] ...trap/engines/engine.py.230: loss: 0.11140 [I 2022-10-27 21:28:02] ...optimizers/trijoint.py.151: total_norm: 24.905863 [I 2022-10-27 21:28:08] ...trap/engines/engine.py.224: train: epoch 89 | batch 310/2304 [I 2022-10-27 21:28:08] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:30 [I 2022-10-27 21:28:08] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00245 [I 2022-10-27 21:28:08] ...trap/engines/engine.py.230: loss: 0.11156 [I 2022-10-27 21:28:08] ...optimizers/trijoint.py.151: total_norm: 16.858332 [I 2022-10-27 21:28:13] ...trap/engines/engine.py.224: train: epoch 89 | batch 320/2304 [I 2022-10-27 21:28:13] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:25 [I 2022-10-27 21:28:13] ...trap/engines/engine.py.229: process: 0.56097 | load: 0.00245 [I 2022-10-27 21:28:13] ...trap/engines/engine.py.230: loss: 0.10115 [I 2022-10-27 21:28:13] ...optimizers/trijoint.py.151: total_norm: 9.194539 [I 2022-10-27 21:28:19] ...trap/engines/engine.py.224: train: epoch 89 | batch 330/2304 [I 2022-10-27 21:28:19] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:24 [I 2022-10-27 21:28:19] ...trap/engines/engine.py.229: process: 0.56467 | load: 0.00274 [I 2022-10-27 21:28:19] ...trap/engines/engine.py.230: loss: 0.14567 [I 2022-10-27 21:28:19] ...optimizers/trijoint.py.151: total_norm: 15.629642 [I 2022-10-27 21:28:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 340/2304 [I 2022-10-27 21:28:24] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:12 [I 2022-10-27 21:28:24] ...trap/engines/engine.py.229: process: 0.54812 | load: 0.00248 [I 2022-10-27 21:28:24] ...trap/engines/engine.py.230: loss: 0.12062 [I 2022-10-27 21:28:24] ...optimizers/trijoint.py.151: total_norm: 9.101257 [I 2022-10-27 21:28:30] ...trap/engines/engine.py.224: train: epoch 89 | batch 350/2304 [I 2022-10-27 21:28:30] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:08 [I 2022-10-27 21:28:30] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00251 [I 2022-10-27 21:28:30] ...trap/engines/engine.py.230: loss: 0.15316 [I 2022-10-27 21:28:30] ...optimizers/trijoint.py.151: total_norm: 13.595540 [I 2022-10-27 21:28:36] ...trap/engines/engine.py.224: train: epoch 89 | batch 360/2304 [I 2022-10-27 21:28:36] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:58 [I 2022-10-27 21:28:36] ...trap/engines/engine.py.229: process: 0.55764 | load: 0.00241 [I 2022-10-27 21:28:36] ...trap/engines/engine.py.230: loss: 0.12723 [I 2022-10-27 21:28:36] ...optimizers/trijoint.py.151: total_norm: 13.342063 [I 2022-10-27 21:28:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 370/2304 [I 2022-10-27 21:28:41] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:58 [I 2022-10-27 21:28:41] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00245 [I 2022-10-27 21:28:41] ...trap/engines/engine.py.230: loss: 0.10408 [I 2022-10-27 21:28:41] ...optimizers/trijoint.py.151: total_norm: 16.526901 [I 2022-10-27 21:28:47] ...trap/engines/engine.py.224: train: epoch 89 | batch 380/2304 [I 2022-10-27 21:28:47] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:18:10 [I 2022-10-27 21:28:47] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00245 [I 2022-10-27 21:28:47] ...trap/engines/engine.py.230: loss: 0.09965 [I 2022-10-27 21:28:47] ...optimizers/trijoint.py.151: total_norm: 10.145001 [I 2022-10-27 21:28:52] ...trap/engines/engine.py.224: train: epoch 89 | batch 390/2304 [I 2022-10-27 21:28:52] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:48 [I 2022-10-27 21:28:52] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00248 [I 2022-10-27 21:28:52] ...trap/engines/engine.py.230: loss: 0.19107 [I 2022-10-27 21:28:52] ...optimizers/trijoint.py.151: total_norm: 24.142017 [I 2022-10-27 21:28:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 400/2304 [I 2022-10-27 21:28:58] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:45 [I 2022-10-27 21:28:58] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00242 [I 2022-10-27 21:28:58] ...trap/engines/engine.py.230: loss: 0.12982 [I 2022-10-27 21:28:58] ...optimizers/trijoint.py.151: total_norm: 22.907965 [I 2022-10-27 21:29:04] ...trap/engines/engine.py.224: train: epoch 89 | batch 410/2304 [I 2022-10-27 21:29:04] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:34 [I 2022-10-27 21:29:04] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00253 [I 2022-10-27 21:29:04] ...trap/engines/engine.py.230: loss: 0.10346 [I 2022-10-27 21:29:04] ...optimizers/trijoint.py.151: total_norm: 31.292284 [I 2022-10-27 21:29:09] ...trap/engines/engine.py.224: train: epoch 89 | batch 420/2304 [I 2022-10-27 21:29:09] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:24 [I 2022-10-27 21:29:09] ...trap/engines/engine.py.229: process: 0.54364 | load: 0.00245 [I 2022-10-27 21:29:09] ...trap/engines/engine.py.230: loss: 0.14090 [I 2022-10-27 21:29:09] ...optimizers/trijoint.py.151: total_norm: 16.689919 [I 2022-10-27 21:29:15] ...trap/engines/engine.py.224: train: epoch 89 | batch 430/2304 [I 2022-10-27 21:29:15] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:23 [I 2022-10-27 21:29:15] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00245 [I 2022-10-27 21:29:15] ...trap/engines/engine.py.230: loss: 0.13006 [I 2022-10-27 21:29:15] ...optimizers/trijoint.py.151: total_norm: 21.325897 [I 2022-10-27 21:29:20] ...trap/engines/engine.py.224: train: epoch 89 | batch 440/2304 [I 2022-10-27 21:29:20] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:28 [I 2022-10-27 21:29:20] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00241 [I 2022-10-27 21:29:20] ...trap/engines/engine.py.230: loss: 0.13010 [I 2022-10-27 21:29:20] ...optimizers/trijoint.py.151: total_norm: 16.751232 [I 2022-10-27 21:29:26] ...trap/engines/engine.py.224: train: epoch 89 | batch 450/2304 [I 2022-10-27 21:29:26] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:22 [I 2022-10-27 21:29:26] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00243 [I 2022-10-27 21:29:26] ...trap/engines/engine.py.230: loss: 0.13471 [I 2022-10-27 21:29:26] ...optimizers/trijoint.py.151: total_norm: 24.928286 [I 2022-10-27 21:29:32] ...trap/engines/engine.py.224: train: epoch 89 | batch 460/2304 [I 2022-10-27 21:29:32] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:09 [I 2022-10-27 21:29:32] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00242 [I 2022-10-27 21:29:32] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-27 21:29:32] ...optimizers/trijoint.py.151: total_norm: 20.464743 [I 2022-10-27 21:29:37] ...trap/engines/engine.py.224: train: epoch 89 | batch 470/2304 [I 2022-10-27 21:29:37] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:02 [I 2022-10-27 21:29:37] ...trap/engines/engine.py.229: process: 0.56250 | load: 0.00241 [I 2022-10-27 21:29:37] ...trap/engines/engine.py.230: loss: 0.08191 [I 2022-10-27 21:29:37] ...optimizers/trijoint.py.151: total_norm: 14.729136 [I 2022-10-27 21:29:43] ...trap/engines/engine.py.224: train: epoch 89 | batch 480/2304 [I 2022-10-27 21:29:43] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:03 [I 2022-10-27 21:29:43] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00244 [I 2022-10-27 21:29:43] ...trap/engines/engine.py.230: loss: 0.13572 [I 2022-10-27 21:29:43] ...optimizers/trijoint.py.151: total_norm: 16.653908 [I 2022-10-27 21:29:49] ...trap/engines/engine.py.224: train: epoch 89 | batch 490/2304 [I 2022-10-27 21:29:49] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:17:12 [I 2022-10-27 21:29:49] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00252 [I 2022-10-27 21:29:49] ...trap/engines/engine.py.230: loss: 0.11654 [I 2022-10-27 21:29:49] ...optimizers/trijoint.py.151: total_norm: 18.520329 [I 2022-10-27 21:29:54] ...trap/engines/engine.py.224: train: epoch 89 | batch 500/2304 [I 2022-10-27 21:29:54] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:42 [I 2022-10-27 21:29:54] ...trap/engines/engine.py.229: process: 0.55410 | load: 0.00245 [I 2022-10-27 21:29:54] ...trap/engines/engine.py.230: loss: 0.13581 [I 2022-10-27 21:29:54] ...optimizers/trijoint.py.151: total_norm: 15.766259 [I 2022-10-27 21:30:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 510/2304 [I 2022-10-27 21:30:00] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:35 [I 2022-10-27 21:30:00] ...trap/engines/engine.py.229: process: 0.55734 | load: 0.00243 [I 2022-10-27 21:30:00] ...trap/engines/engine.py.230: loss: 0.14621 [I 2022-10-27 21:30:00] ...optimizers/trijoint.py.151: total_norm: 18.352884 [I 2022-10-27 21:30:05] ...trap/engines/engine.py.224: train: epoch 89 | batch 520/2304 [I 2022-10-27 21:30:05] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:32 [I 2022-10-27 21:30:05] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00244 [I 2022-10-27 21:30:05] ...trap/engines/engine.py.230: loss: 0.12290 [I 2022-10-27 21:30:05] ...optimizers/trijoint.py.151: total_norm: 12.764700 [I 2022-10-27 21:30:11] ...trap/engines/engine.py.224: train: epoch 89 | batch 530/2304 [I 2022-10-27 21:30:11] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:27 [I 2022-10-27 21:30:11] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00244 [I 2022-10-27 21:30:11] ...trap/engines/engine.py.230: loss: 0.10515 [I 2022-10-27 21:30:11] ...optimizers/trijoint.py.151: total_norm: 14.414261 [I 2022-10-27 21:30:16] ...trap/engines/engine.py.224: train: epoch 89 | batch 540/2304 [I 2022-10-27 21:30:16] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:23 [I 2022-10-27 21:30:16] ...trap/engines/engine.py.229: process: 0.56770 | load: 0.00241 [I 2022-10-27 21:30:16] ...trap/engines/engine.py.230: loss: 0.11546 [I 2022-10-27 21:30:16] ...optimizers/trijoint.py.151: total_norm: 17.744194 [I 2022-10-27 21:30:22] ...trap/engines/engine.py.224: train: epoch 89 | batch 550/2304 [I 2022-10-27 21:30:22] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:22 [I 2022-10-27 21:30:22] ...trap/engines/engine.py.229: process: 0.56792 | load: 0.00246 [I 2022-10-27 21:30:22] ...trap/engines/engine.py.230: loss: 0.12470 [I 2022-10-27 21:30:22] ...optimizers/trijoint.py.151: total_norm: 20.370789 [I 2022-10-27 21:30:28] ...trap/engines/engine.py.224: train: epoch 89 | batch 560/2304 [I 2022-10-27 21:30:28] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:01 [I 2022-10-27 21:30:28] ...trap/engines/engine.py.229: process: 0.54129 | load: 0.00247 [I 2022-10-27 21:30:28] ...trap/engines/engine.py.230: loss: 0.09802 [I 2022-10-27 21:30:28] ...optimizers/trijoint.py.151: total_norm: 15.395514 [I 2022-10-27 21:30:33] ...trap/engines/engine.py.224: train: epoch 89 | batch 570/2304 [I 2022-10-27 21:30:33] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:04 [I 2022-10-27 21:30:33] ...trap/engines/engine.py.229: process: 0.55277 | load: 0.00245 [I 2022-10-27 21:30:33] ...trap/engines/engine.py.230: loss: 0.09577 [I 2022-10-27 21:30:33] ...optimizers/trijoint.py.151: total_norm: 11.328642 [I 2022-10-27 21:30:39] ...trap/engines/engine.py.224: train: epoch 89 | batch 580/2304 [I 2022-10-27 21:30:39] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:58 [I 2022-10-27 21:30:39] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00241 [I 2022-10-27 21:30:39] ...trap/engines/engine.py.230: loss: 0.12666 [I 2022-10-27 21:30:39] ...optimizers/trijoint.py.151: total_norm: 13.385603 [I 2022-10-27 21:30:44] ...trap/engines/engine.py.224: train: epoch 89 | batch 590/2304 [I 2022-10-27 21:30:44] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:56 [I 2022-10-27 21:30:44] ...trap/engines/engine.py.229: process: 0.56067 | load: 0.00262 [I 2022-10-27 21:30:44] ...trap/engines/engine.py.230: loss: 0.12102 [I 2022-10-27 21:30:44] ...optimizers/trijoint.py.151: total_norm: 16.663851 [I 2022-10-27 21:30:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 600/2304 [I 2022-10-27 21:30:50] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:55 [I 2022-10-27 21:30:50] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00243 [I 2022-10-27 21:30:50] ...trap/engines/engine.py.230: loss: 0.13145 [I 2022-10-27 21:30:50] ...optimizers/trijoint.py.151: total_norm: 47.044933 [I 2022-10-27 21:30:56] ...trap/engines/engine.py.224: train: epoch 89 | batch 610/2304 [I 2022-10-27 21:30:56] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:16:12 [I 2022-10-27 21:30:56] ...trap/engines/engine.py.229: process: 0.63166 | load: 0.00245 [I 2022-10-27 21:30:56] ...trap/engines/engine.py.230: loss: 0.10398 [I 2022-10-27 21:30:56] ...optimizers/trijoint.py.151: total_norm: 11.456412 [I 2022-10-27 21:31:01] ...trap/engines/engine.py.224: train: epoch 89 | batch 620/2304 [I 2022-10-27 21:31:01] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:38 [I 2022-10-27 21:31:01] ...trap/engines/engine.py.229: process: 0.55129 | load: 0.00250 [I 2022-10-27 21:31:01] ...trap/engines/engine.py.230: loss: 0.13338 [I 2022-10-27 21:31:01] ...optimizers/trijoint.py.151: total_norm: 22.899996 [I 2022-10-27 21:31:07] ...trap/engines/engine.py.224: train: epoch 89 | batch 630/2304 [I 2022-10-27 21:31:07] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:32 [I 2022-10-27 21:31:07] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00248 [I 2022-10-27 21:31:07] ...trap/engines/engine.py.230: loss: 0.10459 [I 2022-10-27 21:31:07] ...optimizers/trijoint.py.151: total_norm: 15.205510 [I 2022-10-27 21:31:12] ...trap/engines/engine.py.224: train: epoch 89 | batch 640/2304 [I 2022-10-27 21:31:12] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:24 [I 2022-10-27 21:31:12] ...trap/engines/engine.py.229: process: 0.56218 | load: 0.00245 [I 2022-10-27 21:31:12] ...trap/engines/engine.py.230: loss: 0.14299 [I 2022-10-27 21:31:12] ...optimizers/trijoint.py.151: total_norm: 15.075384 [I 2022-10-27 21:31:18] ...trap/engines/engine.py.224: train: epoch 89 | batch 650/2304 [I 2022-10-27 21:31:18] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:20 [I 2022-10-27 21:31:18] ...trap/engines/engine.py.229: process: 0.56814 | load: 0.00245 [I 2022-10-27 21:31:18] ...trap/engines/engine.py.230: loss: 0.11546 [I 2022-10-27 21:31:18] ...optimizers/trijoint.py.151: total_norm: 22.514263 [I 2022-10-27 21:31:23] ...trap/engines/engine.py.224: train: epoch 89 | batch 660/2304 [I 2022-10-27 21:31:23] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:13 [I 2022-10-27 21:31:23] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00248 [I 2022-10-27 21:31:23] ...trap/engines/engine.py.230: loss: 0.14850 [I 2022-10-27 21:31:23] ...optimizers/trijoint.py.151: total_norm: 11.526619 [I 2022-10-27 21:31:29] ...trap/engines/engine.py.224: train: epoch 89 | batch 670/2304 [I 2022-10-27 21:31:29] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:08 [I 2022-10-27 21:31:29] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00253 [I 2022-10-27 21:31:29] ...trap/engines/engine.py.230: loss: 0.07990 [I 2022-10-27 21:31:29] ...optimizers/trijoint.py.151: total_norm: 10.526392 [I 2022-10-27 21:31:35] ...trap/engines/engine.py.224: train: epoch 89 | batch 680/2304 [I 2022-10-27 21:31:35] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:31 [I 2022-10-27 21:31:35] ...trap/engines/engine.py.229: process: 0.61805 | load: 0.00253 [I 2022-10-27 21:31:35] ...trap/engines/engine.py.230: loss: 0.11843 [I 2022-10-27 21:31:35] ...optimizers/trijoint.py.151: total_norm: 16.253796 [I 2022-10-27 21:31:40] ...trap/engines/engine.py.224: train: epoch 89 | batch 690/2304 [I 2022-10-27 21:31:40] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:04 [I 2022-10-27 21:31:40] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00244 [I 2022-10-27 21:31:40] ...trap/engines/engine.py.230: loss: 0.11999 [I 2022-10-27 21:31:40] ...optimizers/trijoint.py.151: total_norm: 19.180624 [I 2022-10-27 21:31:46] ...trap/engines/engine.py.224: train: epoch 89 | batch 700/2304 [I 2022-10-27 21:31:46] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:48 [I 2022-10-27 21:31:46] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00268 [I 2022-10-27 21:31:46] ...trap/engines/engine.py.230: loss: 0.13025 [I 2022-10-27 21:31:46] ...optimizers/trijoint.py.151: total_norm: 25.492342 [I 2022-10-27 21:31:51] ...trap/engines/engine.py.224: train: epoch 89 | batch 710/2304 [I 2022-10-27 21:31:51] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:46 [I 2022-10-27 21:31:51] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00243 [I 2022-10-27 21:31:51] ...trap/engines/engine.py.230: loss: 0.09776 [I 2022-10-27 21:31:51] ...optimizers/trijoint.py.151: total_norm: 11.104936 [I 2022-10-27 21:31:57] ...trap/engines/engine.py.224: train: epoch 89 | batch 720/2304 [I 2022-10-27 21:31:57] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:39 [I 2022-10-27 21:31:57] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00245 [I 2022-10-27 21:31:57] ...trap/engines/engine.py.230: loss: 0.13482 [I 2022-10-27 21:31:57] ...optimizers/trijoint.py.151: total_norm: 10.306959 [I 2022-10-27 21:32:02] ...trap/engines/engine.py.224: train: epoch 89 | batch 730/2304 [I 2022-10-27 21:32:02] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:28 [I 2022-10-27 21:32:02] ...trap/engines/engine.py.229: process: 0.53814 | load: 0.00244 [I 2022-10-27 21:32:02] ...trap/engines/engine.py.230: loss: 0.11966 [I 2022-10-27 21:32:02] ...optimizers/trijoint.py.151: total_norm: 11.119843 [I 2022-10-27 21:32:08] ...trap/engines/engine.py.224: train: epoch 89 | batch 740/2304 [I 2022-10-27 21:32:08] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:29 [I 2022-10-27 21:32:08] ...trap/engines/engine.py.229: process: 0.56857 | load: 0.00249 [I 2022-10-27 21:32:08] ...trap/engines/engine.py.230: loss: 0.09367 [I 2022-10-27 21:32:08] ...optimizers/trijoint.py.151: total_norm: 11.509641 [I 2022-10-27 21:32:14] ...trap/engines/engine.py.224: train: epoch 89 | batch 750/2304 [I 2022-10-27 21:32:14] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:15 [I 2022-10-27 21:32:14] ...trap/engines/engine.py.229: process: 0.54355 | load: 0.00251 [I 2022-10-27 21:32:14] ...trap/engines/engine.py.230: loss: 0.13430 [I 2022-10-27 21:32:14] ...optimizers/trijoint.py.151: total_norm: 24.851927 [I 2022-10-27 21:32:19] ...trap/engines/engine.py.224: train: epoch 89 | batch 760/2304 [I 2022-10-27 21:32:19] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:19 [I 2022-10-27 21:32:19] ...trap/engines/engine.py.229: process: 0.56410 | load: 0.00252 [I 2022-10-27 21:32:19] ...trap/engines/engine.py.230: loss: 0.17235 [I 2022-10-27 21:32:19] ...optimizers/trijoint.py.151: total_norm: 15.168593 [I 2022-10-27 21:32:25] ...trap/engines/engine.py.224: train: epoch 89 | batch 770/2304 [I 2022-10-27 21:32:25] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:16 [I 2022-10-27 21:32:25] ...trap/engines/engine.py.229: process: 0.56179 | load: 0.00247 [I 2022-10-27 21:32:25] ...trap/engines/engine.py.230: loss: 0.12934 [I 2022-10-27 21:32:25] ...optimizers/trijoint.py.151: total_norm: 17.667498 [I 2022-10-27 21:32:30] ...trap/engines/engine.py.224: train: epoch 89 | batch 780/2304 [I 2022-10-27 21:32:30] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:11 [I 2022-10-27 21:32:30] ...trap/engines/engine.py.229: process: 0.56370 | load: 0.00244 [I 2022-10-27 21:32:30] ...trap/engines/engine.py.230: loss: 0.10018 [I 2022-10-27 21:32:30] ...optimizers/trijoint.py.151: total_norm: 8.954518 [I 2022-10-27 21:32:36] ...trap/engines/engine.py.224: train: epoch 89 | batch 790/2304 [I 2022-10-27 21:32:36] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:13:56 [I 2022-10-27 21:32:36] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00241 [I 2022-10-27 21:32:36] ...trap/engines/engine.py.230: loss: 0.10636 [I 2022-10-27 21:32:36] ...optimizers/trijoint.py.151: total_norm: 27.770866 [I 2022-10-27 21:32:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 800/2304 [I 2022-10-27 21:32:41] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:02 [I 2022-10-27 21:32:41] ...trap/engines/engine.py.229: process: 0.56857 | load: 0.00262 [I 2022-10-27 21:32:41] ...trap/engines/engine.py.230: loss: 0.09455 [I 2022-10-27 21:32:41] ...optimizers/trijoint.py.151: total_norm: 14.499839 [I 2022-10-27 21:32:47] ...trap/engines/engine.py.224: train: epoch 89 | batch 810/2304 [I 2022-10-27 21:32:47] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:52 [I 2022-10-27 21:32:47] ...trap/engines/engine.py.229: process: 0.56383 | load: 0.00250 [I 2022-10-27 21:32:47] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-27 21:32:47] ...optimizers/trijoint.py.151: total_norm: 17.353128 [I 2022-10-27 21:32:53] ...trap/engines/engine.py.224: train: epoch 89 | batch 820/2304 [I 2022-10-27 21:32:53] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:45 [I 2022-10-27 21:32:53] ...trap/engines/engine.py.229: process: 0.56242 | load: 0.00249 [I 2022-10-27 21:32:53] ...trap/engines/engine.py.230: loss: 0.09862 [I 2022-10-27 21:32:53] ...optimizers/trijoint.py.151: total_norm: 13.490825 [I 2022-10-27 21:32:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 830/2304 [I 2022-10-27 21:32:58] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:43 [I 2022-10-27 21:32:58] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00266 [I 2022-10-27 21:32:58] ...trap/engines/engine.py.230: loss: 0.13438 [I 2022-10-27 21:32:58] ...optimizers/trijoint.py.151: total_norm: 38.222618 [I 2022-10-27 21:33:04] ...trap/engines/engine.py.224: train: epoch 89 | batch 840/2304 [I 2022-10-27 21:33:04] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:36 [I 2022-10-27 21:33:04] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00251 [I 2022-10-27 21:33:04] ...trap/engines/engine.py.230: loss: 0.13777 [I 2022-10-27 21:33:04] ...optimizers/trijoint.py.151: total_norm: 26.995314 [I 2022-10-27 21:33:09] ...trap/engines/engine.py.224: train: epoch 89 | batch 850/2304 [I 2022-10-27 21:33:09] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:30 [I 2022-10-27 21:33:09] ...trap/engines/engine.py.229: process: 0.55835 | load: 0.00247 [I 2022-10-27 21:33:09] ...trap/engines/engine.py.230: loss: 0.13084 [I 2022-10-27 21:33:09] ...optimizers/trijoint.py.151: total_norm: 29.849417 [I 2022-10-27 21:33:15] ...trap/engines/engine.py.224: train: epoch 89 | batch 860/2304 [I 2022-10-27 21:33:15] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:27 [I 2022-10-27 21:33:15] ...trap/engines/engine.py.229: process: 0.56358 | load: 0.00249 [I 2022-10-27 21:33:15] ...trap/engines/engine.py.230: loss: 0.17385 [I 2022-10-27 21:33:15] ...optimizers/trijoint.py.151: total_norm: 19.355968 [I 2022-10-27 21:33:20] ...trap/engines/engine.py.224: train: epoch 89 | batch 870/2304 [I 2022-10-27 21:33:20] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:14 [I 2022-10-27 21:33:21] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00243 [I 2022-10-27 21:33:21] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-27 21:33:21] ...optimizers/trijoint.py.151: total_norm: 9.932205 [I 2022-10-27 21:33:26] ...trap/engines/engine.py.224: train: epoch 89 | batch 880/2304 [I 2022-10-27 21:33:26] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:07 [I 2022-10-27 21:33:26] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00241 [I 2022-10-27 21:33:26] ...trap/engines/engine.py.230: loss: 0.11911 [I 2022-10-27 21:33:26] ...optimizers/trijoint.py.151: total_norm: 9.834038 [I 2022-10-27 21:33:32] ...trap/engines/engine.py.224: train: epoch 89 | batch 890/2304 [I 2022-10-27 21:33:32] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:07 [I 2022-10-27 21:33:32] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00259 [I 2022-10-27 21:33:32] ...trap/engines/engine.py.230: loss: 0.11762 [I 2022-10-27 21:33:32] ...optimizers/trijoint.py.151: total_norm: 21.452106 [I 2022-10-27 21:33:37] ...trap/engines/engine.py.224: train: epoch 89 | batch 900/2304 [I 2022-10-27 21:33:37] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:12:55 [I 2022-10-27 21:33:37] ...trap/engines/engine.py.229: process: 0.55114 | load: 0.00241 [I 2022-10-27 21:33:37] ...trap/engines/engine.py.230: loss: 0.11578 [I 2022-10-27 21:33:37] ...optimizers/trijoint.py.151: total_norm: 16.960175 [I 2022-10-27 21:33:43] ...trap/engines/engine.py.224: train: epoch 89 | batch 910/2304 [I 2022-10-27 21:33:43] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:51 [I 2022-10-27 21:33:43] ...trap/engines/engine.py.229: process: 0.55658 | load: 0.00248 [I 2022-10-27 21:33:43] ...trap/engines/engine.py.230: loss: 0.14827 [I 2022-10-27 21:33:43] ...optimizers/trijoint.py.151: total_norm: 12.538854 [I 2022-10-27 21:33:48] ...trap/engines/engine.py.224: train: epoch 89 | batch 920/2304 [I 2022-10-27 21:33:48] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:49 [I 2022-10-27 21:33:48] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00246 [I 2022-10-27 21:33:48] ...trap/engines/engine.py.230: loss: 0.22630 [I 2022-10-27 21:33:48] ...optimizers/trijoint.py.151: total_norm: 34.566895 [I 2022-10-27 21:33:54] ...trap/engines/engine.py.224: train: epoch 89 | batch 930/2304 [I 2022-10-27 21:33:54] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:49 [I 2022-10-27 21:33:54] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00246 [I 2022-10-27 21:33:54] ...trap/engines/engine.py.230: loss: 0.12418 [I 2022-10-27 21:33:54] ...optimizers/trijoint.py.151: total_norm: 27.973330 [I 2022-10-27 21:34:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 940/2304 [I 2022-10-27 21:34:00] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:43 [I 2022-10-27 21:34:00] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00239 [I 2022-10-27 21:34:00] ...trap/engines/engine.py.230: loss: 0.09894 [I 2022-10-27 21:34:00] ...optimizers/trijoint.py.151: total_norm: 12.094034 [I 2022-10-27 21:34:05] ...trap/engines/engine.py.224: train: epoch 89 | batch 950/2304 [I 2022-10-27 21:34:05] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:36 [I 2022-10-27 21:34:05] ...trap/engines/engine.py.229: process: 0.56610 | load: 0.00244 [I 2022-10-27 21:34:05] ...trap/engines/engine.py.230: loss: 0.12811 [I 2022-10-27 21:34:05] ...optimizers/trijoint.py.151: total_norm: 10.465461 [I 2022-10-27 21:34:11] ...trap/engines/engine.py.224: train: epoch 89 | batch 960/2304 [I 2022-10-27 21:34:11] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:28 [I 2022-10-27 21:34:11] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00243 [I 2022-10-27 21:34:11] ...trap/engines/engine.py.230: loss: 0.14618 [I 2022-10-27 21:34:11] ...optimizers/trijoint.py.151: total_norm: 18.473406 [I 2022-10-27 21:34:16] ...trap/engines/engine.py.224: train: epoch 89 | batch 970/2304 [I 2022-10-27 21:34:16] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:24 [I 2022-10-27 21:34:16] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00241 [I 2022-10-27 21:34:16] ...trap/engines/engine.py.230: loss: 0.10371 [I 2022-10-27 21:34:16] ...optimizers/trijoint.py.151: total_norm: 11.260855 [I 2022-10-27 21:34:22] ...trap/engines/engine.py.224: train: epoch 89 | batch 980/2304 [I 2022-10-27 21:34:22] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:26 [I 2022-10-27 21:34:22] ...trap/engines/engine.py.229: process: 0.57377 | load: 0.00249 [I 2022-10-27 21:34:22] ...trap/engines/engine.py.230: loss: 0.12010 [I 2022-10-27 21:34:22] ...optimizers/trijoint.py.151: total_norm: 9.598986 [I 2022-10-27 21:34:28] ...trap/engines/engine.py.224: train: epoch 89 | batch 990/2304 [I 2022-10-27 21:34:28] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:18 [I 2022-10-27 21:34:28] ...trap/engines/engine.py.229: process: 0.56384 | load: 0.00248 [I 2022-10-27 21:34:28] ...trap/engines/engine.py.230: loss: 0.15979 [I 2022-10-27 21:34:28] ...optimizers/trijoint.py.151: total_norm: 29.607883 [I 2022-10-27 21:34:33] ...trap/engines/engine.py.224: train: epoch 89 | batch 1000/2304 [I 2022-10-27 21:34:33] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:08 [I 2022-10-27 21:34:33] ...trap/engines/engine.py.229: process: 0.56184 | load: 0.00255 [I 2022-10-27 21:34:33] ...trap/engines/engine.py.230: loss: 0.14393 [I 2022-10-27 21:34:33] ...optimizers/trijoint.py.151: total_norm: 16.414513 [I 2022-10-27 21:34:39] ...trap/engines/engine.py.224: train: epoch 89 | batch 1010/2304 [I 2022-10-27 21:34:39] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:08 [I 2022-10-27 21:34:39] ...trap/engines/engine.py.229: process: 0.56617 | load: 0.00243 [I 2022-10-27 21:34:39] ...trap/engines/engine.py.230: loss: 0.10795 [I 2022-10-27 21:34:39] ...optimizers/trijoint.py.151: total_norm: 16.878263 [I 2022-10-27 21:34:44] ...trap/engines/engine.py.224: train: epoch 89 | batch 1020/2304 [I 2022-10-27 21:34:44] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:55 [I 2022-10-27 21:34:44] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00248 [I 2022-10-27 21:34:44] ...trap/engines/engine.py.230: loss: 0.16937 [I 2022-10-27 21:34:44] ...optimizers/trijoint.py.151: total_norm: 18.445826 [I 2022-10-27 21:34:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 1030/2304 [I 2022-10-27 21:34:50] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:51 [I 2022-10-27 21:34:50] ...trap/engines/engine.py.229: process: 0.56566 | load: 0.00246 [I 2022-10-27 21:34:50] ...trap/engines/engine.py.230: loss: 0.12415 [I 2022-10-27 21:34:50] ...optimizers/trijoint.py.151: total_norm: 9.627109 [I 2022-10-27 21:34:55] ...trap/engines/engine.py.224: train: epoch 89 | batch 1040/2304 [I 2022-10-27 21:34:55] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:42 [I 2022-10-27 21:34:55] ...trap/engines/engine.py.229: process: 0.54515 | load: 0.00248 [I 2022-10-27 21:34:55] ...trap/engines/engine.py.230: loss: 0.12100 [I 2022-10-27 21:34:55] ...optimizers/trijoint.py.151: total_norm: 18.530685 [I 2022-10-27 21:35:01] ...trap/engines/engine.py.224: train: epoch 89 | batch 1050/2304 [I 2022-10-27 21:35:01] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:35 [I 2022-10-27 21:35:01] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00245 [I 2022-10-27 21:35:01] ...trap/engines/engine.py.230: loss: 0.13573 [I 2022-10-27 21:35:01] ...optimizers/trijoint.py.151: total_norm: 22.777065 [I 2022-10-27 21:35:07] ...trap/engines/engine.py.224: train: epoch 89 | batch 1060/2304 [I 2022-10-27 21:35:07] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:30 [I 2022-10-27 21:35:07] ...trap/engines/engine.py.229: process: 0.54921 | load: 0.00239 [I 2022-10-27 21:35:07] ...trap/engines/engine.py.230: loss: 0.09645 [I 2022-10-27 21:35:07] ...optimizers/trijoint.py.151: total_norm: 13.125755 [I 2022-10-27 21:35:12] ...trap/engines/engine.py.224: train: epoch 89 | batch 1070/2304 [I 2022-10-27 21:35:12] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:30 [I 2022-10-27 21:35:12] ...trap/engines/engine.py.229: process: 0.57247 | load: 0.00261 [I 2022-10-27 21:35:12] ...trap/engines/engine.py.230: loss: 0.16900 [I 2022-10-27 21:35:12] ...optimizers/trijoint.py.151: total_norm: 38.888855 [I 2022-10-27 21:35:18] ...trap/engines/engine.py.224: train: epoch 89 | batch 1080/2304 [I 2022-10-27 21:35:18] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:25 [I 2022-10-27 21:35:18] ...trap/engines/engine.py.229: process: 0.56005 | load: 0.00246 [I 2022-10-27 21:35:18] ...trap/engines/engine.py.230: loss: 0.07418 [I 2022-10-27 21:35:18] ...optimizers/trijoint.py.151: total_norm: 12.294590 [I 2022-10-27 21:35:23] ...trap/engines/engine.py.224: train: epoch 89 | batch 1090/2304 [I 2022-10-27 21:35:23] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:21 [I 2022-10-27 21:35:23] ...trap/engines/engine.py.229: process: 0.56465 | load: 0.00251 [I 2022-10-27 21:35:23] ...trap/engines/engine.py.230: loss: 0.15158 [I 2022-10-27 21:35:23] ...optimizers/trijoint.py.151: total_norm: 43.299480 [I 2022-10-27 21:35:29] ...trap/engines/engine.py.224: train: epoch 89 | batch 1100/2304 [I 2022-10-27 21:35:29] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:12 [I 2022-10-27 21:35:29] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00243 [I 2022-10-27 21:35:29] ...trap/engines/engine.py.230: loss: 0.16295 [I 2022-10-27 21:35:29] ...optimizers/trijoint.py.151: total_norm: 34.742012 [I 2022-10-27 21:35:35] ...trap/engines/engine.py.224: train: epoch 89 | batch 1110/2304 [I 2022-10-27 21:35:35] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:03 [I 2022-10-27 21:35:35] ...trap/engines/engine.py.229: process: 0.55556 | load: 0.00246 [I 2022-10-27 21:35:35] ...trap/engines/engine.py.230: loss: 0.14116 [I 2022-10-27 21:35:35] ...optimizers/trijoint.py.151: total_norm: 17.180939 [I 2022-10-27 21:35:40] ...trap/engines/engine.py.224: train: epoch 89 | batch 1120/2304 [I 2022-10-27 21:35:40] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:10:54 [I 2022-10-27 21:35:40] ...trap/engines/engine.py.229: process: 0.54528 | load: 0.00249 [I 2022-10-27 21:35:40] ...trap/engines/engine.py.230: loss: 0.08773 [I 2022-10-27 21:35:40] ...optimizers/trijoint.py.151: total_norm: 13.550600 [I 2022-10-27 21:35:46] ...trap/engines/engine.py.224: train: epoch 89 | batch 1130/2304 [I 2022-10-27 21:35:46] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:54 [I 2022-10-27 21:35:46] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00243 [I 2022-10-27 21:35:46] ...trap/engines/engine.py.230: loss: 0.08478 [I 2022-10-27 21:35:46] ...optimizers/trijoint.py.151: total_norm: 11.977833 [I 2022-10-27 21:35:51] ...trap/engines/engine.py.224: train: epoch 89 | batch 1140/2304 [I 2022-10-27 21:35:51] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:50 [I 2022-10-27 21:35:51] ...trap/engines/engine.py.229: process: 0.56256 | load: 0.00246 [I 2022-10-27 21:35:51] ...trap/engines/engine.py.230: loss: 0.15084 [I 2022-10-27 21:35:51] ...optimizers/trijoint.py.151: total_norm: 14.982816 [I 2022-10-27 21:35:57] ...trap/engines/engine.py.224: train: epoch 89 | batch 1150/2304 [I 2022-10-27 21:35:57] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:46 [I 2022-10-27 21:35:57] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00251 [I 2022-10-27 21:35:57] ...trap/engines/engine.py.230: loss: 0.19317 [I 2022-10-27 21:35:57] ...optimizers/trijoint.py.151: total_norm: 37.624084 [I 2022-10-27 21:36:02] ...trap/engines/engine.py.224: train: epoch 89 | batch 1160/2304 [I 2022-10-27 21:36:02] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:38 [I 2022-10-27 21:36:02] ...trap/engines/engine.py.229: process: 0.55103 | load: 0.00245 [I 2022-10-27 21:36:02] ...trap/engines/engine.py.230: loss: 0.16262 [I 2022-10-27 21:36:02] ...optimizers/trijoint.py.151: total_norm: 24.852123 [I 2022-10-27 21:36:08] ...trap/engines/engine.py.224: train: epoch 89 | batch 1170/2304 [I 2022-10-27 21:36:08] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:27 [I 2022-10-27 21:36:08] ...trap/engines/engine.py.229: process: 0.55111 | load: 0.00242 [I 2022-10-27 21:36:08] ...trap/engines/engine.py.230: loss: 0.13542 [I 2022-10-27 21:36:08] ...optimizers/trijoint.py.151: total_norm: 17.995058 [I 2022-10-27 21:36:14] ...trap/engines/engine.py.224: train: epoch 89 | batch 1180/2304 [I 2022-10-27 21:36:14] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:21 [I 2022-10-27 21:36:14] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00243 [I 2022-10-27 21:36:14] ...trap/engines/engine.py.230: loss: 0.15289 [I 2022-10-27 21:36:14] ...optimizers/trijoint.py.151: total_norm: 18.076075 [I 2022-10-27 21:36:19] ...trap/engines/engine.py.224: train: epoch 89 | batch 1190/2304 [I 2022-10-27 21:36:19] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:20 [I 2022-10-27 21:36:19] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00242 [I 2022-10-27 21:36:19] ...trap/engines/engine.py.230: loss: 0.13215 [I 2022-10-27 21:36:19] ...optimizers/trijoint.py.151: total_norm: 14.567139 [I 2022-10-27 21:36:25] ...trap/engines/engine.py.224: train: epoch 89 | batch 1200/2304 [I 2022-10-27 21:36:25] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:23 [I 2022-10-27 21:36:25] ...trap/engines/engine.py.229: process: 0.59210 | load: 0.00241 [I 2022-10-27 21:36:25] ...trap/engines/engine.py.230: loss: 0.09769 [I 2022-10-27 21:36:25] ...optimizers/trijoint.py.151: total_norm: 16.531559 [I 2022-10-27 21:36:30] ...trap/engines/engine.py.224: train: epoch 89 | batch 1210/2304 [I 2022-10-27 21:36:30] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:18 [I 2022-10-27 21:36:30] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00260 [I 2022-10-27 21:36:30] ...trap/engines/engine.py.230: loss: 0.11572 [I 2022-10-27 21:36:30] ...optimizers/trijoint.py.151: total_norm: 10.264095 [I 2022-10-27 21:36:36] ...trap/engines/engine.py.224: train: epoch 89 | batch 1220/2304 [I 2022-10-27 21:36:36] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:26 [I 2022-10-27 21:36:36] ...trap/engines/engine.py.229: process: 0.63783 | load: 0.00245 [I 2022-10-27 21:36:36] ...trap/engines/engine.py.230: loss: 0.15906 [I 2022-10-27 21:36:36] ...optimizers/trijoint.py.151: total_norm: 19.639500 [I 2022-10-27 21:36:42] ...trap/engines/engine.py.224: train: epoch 89 | batch 1230/2304 [I 2022-10-27 21:36:42] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:59 [I 2022-10-27 21:36:42] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00244 [I 2022-10-27 21:36:42] ...trap/engines/engine.py.230: loss: 0.07812 [I 2022-10-27 21:36:42] ...optimizers/trijoint.py.151: total_norm: 19.069168 [I 2022-10-27 21:36:47] ...trap/engines/engine.py.224: train: epoch 89 | batch 1240/2304 [I 2022-10-27 21:36:47] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:54 [I 2022-10-27 21:36:47] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00250 [I 2022-10-27 21:36:47] ...trap/engines/engine.py.230: loss: 0.13058 [I 2022-10-27 21:36:47] ...optimizers/trijoint.py.151: total_norm: 35.962513 [I 2022-10-27 21:36:53] ...trap/engines/engine.py.224: train: epoch 89 | batch 1250/2304 [I 2022-10-27 21:36:53] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:45 [I 2022-10-27 21:36:53] ...trap/engines/engine.py.229: process: 0.55220 | load: 0.00249 [I 2022-10-27 21:36:53] ...trap/engines/engine.py.230: loss: 0.10770 [I 2022-10-27 21:36:53] ...optimizers/trijoint.py.151: total_norm: 11.021083 [I 2022-10-27 21:36:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 1260/2304 [I 2022-10-27 21:36:58] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:44 [I 2022-10-27 21:36:58] ...trap/engines/engine.py.229: process: 0.56416 | load: 0.00247 [I 2022-10-27 21:36:58] ...trap/engines/engine.py.230: loss: 0.11047 [I 2022-10-27 21:36:58] ...optimizers/trijoint.py.151: total_norm: 11.162351 [I 2022-10-27 21:37:04] ...trap/engines/engine.py.224: train: epoch 89 | batch 1270/2304 [I 2022-10-27 21:37:04] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:41 [I 2022-10-27 21:37:04] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00246 [I 2022-10-27 21:37:04] ...trap/engines/engine.py.230: loss: 0.13702 [I 2022-10-27 21:37:04] ...optimizers/trijoint.py.151: total_norm: 10.550619 [I 2022-10-27 21:37:10] ...trap/engines/engine.py.224: train: epoch 89 | batch 1280/2304 [I 2022-10-27 21:37:10] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:31 [I 2022-10-27 21:37:10] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00247 [I 2022-10-27 21:37:10] ...trap/engines/engine.py.230: loss: 0.10769 [I 2022-10-27 21:37:10] ...optimizers/trijoint.py.151: total_norm: 11.413831 [I 2022-10-27 21:37:15] ...trap/engines/engine.py.224: train: epoch 89 | batch 1290/2304 [I 2022-10-27 21:37:15] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:32 [I 2022-10-27 21:37:15] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00246 [I 2022-10-27 21:37:15] ...trap/engines/engine.py.230: loss: 0.10338 [I 2022-10-27 21:37:15] ...optimizers/trijoint.py.151: total_norm: 11.581460 [I 2022-10-27 21:37:21] ...trap/engines/engine.py.224: train: epoch 89 | batch 1300/2304 [I 2022-10-27 21:37:21] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:23 [I 2022-10-27 21:37:21] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00249 [I 2022-10-27 21:37:21] ...trap/engines/engine.py.230: loss: 0.15112 [I 2022-10-27 21:37:21] ...optimizers/trijoint.py.151: total_norm: 37.491268 [I 2022-10-27 21:37:27] ...trap/engines/engine.py.224: train: epoch 89 | batch 1310/2304 [I 2022-10-27 21:37:27] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:11 [I 2022-10-27 21:37:27] ...trap/engines/engine.py.229: process: 0.54894 | load: 0.00251 [I 2022-10-27 21:37:27] ...trap/engines/engine.py.230: loss: 0.15153 [I 2022-10-27 21:37:27] ...optimizers/trijoint.py.151: total_norm: 12.761447 [I 2022-10-27 21:37:32] ...trap/engines/engine.py.224: train: epoch 89 | batch 1320/2304 [I 2022-10-27 21:37:32] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:04 [I 2022-10-27 21:37:32] ...trap/engines/engine.py.229: process: 0.56149 | load: 0.00244 [I 2022-10-27 21:37:32] ...trap/engines/engine.py.230: loss: 0.09935 [I 2022-10-27 21:37:32] ...optimizers/trijoint.py.151: total_norm: 18.321547 [I 2022-10-27 21:37:38] ...trap/engines/engine.py.224: train: epoch 89 | batch 1330/2304 [I 2022-10-27 21:37:38] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:04 [I 2022-10-27 21:37:38] ...trap/engines/engine.py.229: process: 0.56875 | load: 0.00243 [I 2022-10-27 21:37:38] ...trap/engines/engine.py.230: loss: 0.10400 [I 2022-10-27 21:37:38] ...optimizers/trijoint.py.151: total_norm: 13.803312 [I 2022-10-27 21:37:43] ...trap/engines/engine.py.224: train: epoch 89 | batch 1340/2304 [I 2022-10-27 21:37:43] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:55 [I 2022-10-27 21:37:43] ...trap/engines/engine.py.229: process: 0.56399 | load: 0.00244 [I 2022-10-27 21:37:43] ...trap/engines/engine.py.230: loss: 0.22624 [I 2022-10-27 21:37:43] ...optimizers/trijoint.py.151: total_norm: 17.513544 [I 2022-10-27 21:37:49] ...trap/engines/engine.py.224: train: epoch 89 | batch 1350/2304 [I 2022-10-27 21:37:49] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:48 [I 2022-10-27 21:37:49] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00239 [I 2022-10-27 21:37:49] ...trap/engines/engine.py.230: loss: 0.18433 [I 2022-10-27 21:37:49] ...optimizers/trijoint.py.151: total_norm: 18.137392 [I 2022-10-27 21:37:54] ...trap/engines/engine.py.224: train: epoch 89 | batch 1360/2304 [I 2022-10-27 21:37:54] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:46 [I 2022-10-27 21:37:54] ...trap/engines/engine.py.229: process: 0.55997 | load: 0.00244 [I 2022-10-27 21:37:54] ...trap/engines/engine.py.230: loss: 0.09018 [I 2022-10-27 21:37:54] ...optimizers/trijoint.py.151: total_norm: 14.102863 [I 2022-10-27 21:38:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 1370/2304 [I 2022-10-27 21:38:00] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:37 [I 2022-10-27 21:38:00] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00237 [I 2022-10-27 21:38:00] ...trap/engines/engine.py.230: loss: 0.09255 [I 2022-10-27 21:38:00] ...optimizers/trijoint.py.151: total_norm: 12.515361 [I 2022-10-27 21:38:06] ...trap/engines/engine.py.224: train: epoch 89 | batch 1380/2304 [I 2022-10-27 21:38:06] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:32 [I 2022-10-27 21:38:06] ...trap/engines/engine.py.229: process: 0.55351 | load: 0.00243 [I 2022-10-27 21:38:06] ...trap/engines/engine.py.230: loss: 0.15272 [I 2022-10-27 21:38:06] ...optimizers/trijoint.py.151: total_norm: 22.543640 [I 2022-10-27 21:38:11] ...trap/engines/engine.py.224: train: epoch 89 | batch 1390/2304 [I 2022-10-27 21:38:11] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:27 [I 2022-10-27 21:38:11] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00240 [I 2022-10-27 21:38:11] ...trap/engines/engine.py.230: loss: 0.12683 [I 2022-10-27 21:38:11] ...optimizers/trijoint.py.151: total_norm: 14.256028 [I 2022-10-27 21:38:17] ...trap/engines/engine.py.224: train: epoch 89 | batch 1400/2304 [I 2022-10-27 21:38:17] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:24 [I 2022-10-27 21:38:17] ...trap/engines/engine.py.229: process: 0.56480 | load: 0.00243 [I 2022-10-27 21:38:17] ...trap/engines/engine.py.230: loss: 0.12023 [I 2022-10-27 21:38:17] ...optimizers/trijoint.py.151: total_norm: 17.654951 [I 2022-10-27 21:38:22] ...trap/engines/engine.py.224: train: epoch 89 | batch 1410/2304 [I 2022-10-27 21:38:22] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:17 [I 2022-10-27 21:38:22] ...trap/engines/engine.py.229: process: 0.55038 | load: 0.00245 [I 2022-10-27 21:38:22] ...trap/engines/engine.py.230: loss: 0.11365 [I 2022-10-27 21:38:22] ...optimizers/trijoint.py.151: total_norm: 26.382870 [I 2022-10-27 21:38:28] ...trap/engines/engine.py.224: train: epoch 89 | batch 1420/2304 [I 2022-10-27 21:38:28] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:24 [I 2022-10-27 21:38:28] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00246 [I 2022-10-27 21:38:28] ...trap/engines/engine.py.230: loss: 0.10165 [I 2022-10-27 21:38:28] ...optimizers/trijoint.py.151: total_norm: 11.280811 [I 2022-10-27 21:38:34] ...trap/engines/engine.py.224: train: epoch 89 | batch 1430/2304 [I 2022-10-27 21:38:34] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:07 [I 2022-10-27 21:38:34] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00243 [I 2022-10-27 21:38:34] ...trap/engines/engine.py.230: loss: 0.14389 [I 2022-10-27 21:38:34] ...optimizers/trijoint.py.151: total_norm: 22.828217 [I 2022-10-27 21:38:39] ...trap/engines/engine.py.224: train: epoch 89 | batch 1440/2304 [I 2022-10-27 21:38:39] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:07:59 [I 2022-10-27 21:38:39] ...trap/engines/engine.py.229: process: 0.54697 | load: 0.00238 [I 2022-10-27 21:38:39] ...trap/engines/engine.py.230: loss: 0.11464 [I 2022-10-27 21:38:39] ...optimizers/trijoint.py.151: total_norm: 29.333775 [I 2022-10-27 21:38:45] ...trap/engines/engine.py.224: train: epoch 89 | batch 1450/2304 [I 2022-10-27 21:38:45] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:55 [I 2022-10-27 21:38:45] ...trap/engines/engine.py.229: process: 0.54191 | load: 0.00243 [I 2022-10-27 21:38:45] ...trap/engines/engine.py.230: loss: 0.11121 [I 2022-10-27 21:38:45] ...optimizers/trijoint.py.151: total_norm: 16.442015 [I 2022-10-27 21:38:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 1460/2304 [I 2022-10-27 21:38:50] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:51 [I 2022-10-27 21:38:50] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00242 [I 2022-10-27 21:38:50] ...trap/engines/engine.py.230: loss: 0.09658 [I 2022-10-27 21:38:50] ...optimizers/trijoint.py.151: total_norm: 9.952054 [I 2022-10-27 21:38:56] ...trap/engines/engine.py.224: train: epoch 89 | batch 1470/2304 [I 2022-10-27 21:38:56] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:07:44 [I 2022-10-27 21:38:56] ...trap/engines/engine.py.229: process: 0.55724 | load: 0.00246 [I 2022-10-27 21:38:56] ...trap/engines/engine.py.230: loss: 0.14420 [I 2022-10-27 21:38:56] ...optimizers/trijoint.py.151: total_norm: 16.113226 [I 2022-10-27 21:39:02] ...trap/engines/engine.py.224: train: epoch 89 | batch 1480/2304 [I 2022-10-27 21:39:02] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:41 [I 2022-10-27 21:39:02] ...trap/engines/engine.py.229: process: 0.55023 | load: 0.00246 [I 2022-10-27 21:39:02] ...trap/engines/engine.py.230: loss: 0.16899 [I 2022-10-27 21:39:02] ...optimizers/trijoint.py.151: total_norm: 17.192995 [I 2022-10-27 21:39:07] ...trap/engines/engine.py.224: train: epoch 89 | batch 1490/2304 [I 2022-10-27 21:39:07] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:33 [I 2022-10-27 21:39:07] ...trap/engines/engine.py.229: process: 0.55321 | load: 0.00243 [I 2022-10-27 21:39:07] ...trap/engines/engine.py.230: loss: 0.10679 [I 2022-10-27 21:39:07] ...optimizers/trijoint.py.151: total_norm: 7.419697 [I 2022-10-27 21:39:13] ...trap/engines/engine.py.224: train: epoch 89 | batch 1500/2304 [I 2022-10-27 21:39:13] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:25 [I 2022-10-27 21:39:13] ...trap/engines/engine.py.229: process: 0.55191 | load: 0.00248 [I 2022-10-27 21:39:13] ...trap/engines/engine.py.230: loss: 0.13375 [I 2022-10-27 21:39:13] ...optimizers/trijoint.py.151: total_norm: 18.434311 [I 2022-10-27 21:39:18] ...trap/engines/engine.py.224: train: epoch 89 | batch 1510/2304 [I 2022-10-27 21:39:18] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:21 [I 2022-10-27 21:39:18] ...trap/engines/engine.py.229: process: 0.55623 | load: 0.00244 [I 2022-10-27 21:39:18] ...trap/engines/engine.py.230: loss: 0.11480 [I 2022-10-27 21:39:18] ...optimizers/trijoint.py.151: total_norm: 16.688673 [I 2022-10-27 21:39:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 1520/2304 [I 2022-10-27 21:39:24] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:17 [I 2022-10-27 21:39:24] ...trap/engines/engine.py.229: process: 0.55482 | load: 0.00244 [I 2022-10-27 21:39:24] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-27 21:39:24] ...optimizers/trijoint.py.151: total_norm: 22.913860 [I 2022-10-27 21:39:29] ...trap/engines/engine.py.224: train: epoch 89 | batch 1530/2304 [I 2022-10-27 21:39:29] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:11 [I 2022-10-27 21:39:29] ...trap/engines/engine.py.229: process: 0.56556 | load: 0.00241 [I 2022-10-27 21:39:29] ...trap/engines/engine.py.230: loss: 0.25667 [I 2022-10-27 21:39:29] ...optimizers/trijoint.py.151: total_norm: 18.713966 [I 2022-10-27 21:39:35] ...trap/engines/engine.py.224: train: epoch 89 | batch 1540/2304 [I 2022-10-27 21:39:35] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:19 [I 2022-10-27 21:39:35] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00244 [I 2022-10-27 21:39:35] ...trap/engines/engine.py.230: loss: 0.09469 [I 2022-10-27 21:39:35] ...optimizers/trijoint.py.151: total_norm: 20.869505 [I 2022-10-27 21:39:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 1550/2304 [I 2022-10-27 21:39:41] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:00 [I 2022-10-27 21:39:41] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00238 [I 2022-10-27 21:39:41] ...trap/engines/engine.py.230: loss: 0.14901 [I 2022-10-27 21:39:41] ...optimizers/trijoint.py.151: total_norm: 17.478466 [I 2022-10-27 21:39:46] ...trap/engines/engine.py.224: train: epoch 89 | batch 1560/2304 [I 2022-10-27 21:39:46] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:55 [I 2022-10-27 21:39:46] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00247 [I 2022-10-27 21:39:46] ...trap/engines/engine.py.230: loss: 0.16309 [I 2022-10-27 21:39:46] ...optimizers/trijoint.py.151: total_norm: 18.750483 [I 2022-10-27 21:39:52] ...trap/engines/engine.py.224: train: epoch 89 | batch 1570/2304 [I 2022-10-27 21:39:52] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:51 [I 2022-10-27 21:39:52] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00240 [I 2022-10-27 21:39:52] ...trap/engines/engine.py.230: loss: 0.08242 [I 2022-10-27 21:39:52] ...optimizers/trijoint.py.151: total_norm: 11.358975 [I 2022-10-27 21:39:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 1580/2304 [I 2022-10-27 21:39:58] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:44 [I 2022-10-27 21:39:58] ...trap/engines/engine.py.229: process: 0.57203 | load: 0.00242 [I 2022-10-27 21:39:58] ...trap/engines/engine.py.230: loss: 0.17017 [I 2022-10-27 21:39:58] ...optimizers/trijoint.py.151: total_norm: 32.651974 [I 2022-10-27 21:40:03] ...trap/engines/engine.py.224: train: epoch 89 | batch 1590/2304 [I 2022-10-27 21:40:03] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:06:37 [I 2022-10-27 21:40:03] ...trap/engines/engine.py.229: process: 0.56242 | load: 0.00242 [I 2022-10-27 21:40:03] ...trap/engines/engine.py.230: loss: 0.10245 [I 2022-10-27 21:40:03] ...optimizers/trijoint.py.151: total_norm: 11.389456 [I 2022-10-27 21:40:09] ...trap/engines/engine.py.224: train: epoch 89 | batch 1600/2304 [I 2022-10-27 21:40:09] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:31 [I 2022-10-27 21:40:09] ...trap/engines/engine.py.229: process: 0.55295 | load: 0.00246 [I 2022-10-27 21:40:09] ...trap/engines/engine.py.230: loss: 0.12823 [I 2022-10-27 21:40:09] ...optimizers/trijoint.py.151: total_norm: 19.030954 [I 2022-10-27 21:40:14] ...trap/engines/engine.py.224: train: epoch 89 | batch 1610/2304 [I 2022-10-27 21:40:14] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:33 [I 2022-10-27 21:40:14] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00248 [I 2022-10-27 21:40:14] ...trap/engines/engine.py.230: loss: 0.08467 [I 2022-10-27 21:40:14] ...optimizers/trijoint.py.151: total_norm: 13.078435 [I 2022-10-27 21:40:20] ...trap/engines/engine.py.224: train: epoch 89 | batch 1620/2304 [I 2022-10-27 21:40:20] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:22 [I 2022-10-27 21:40:20] ...trap/engines/engine.py.229: process: 0.56306 | load: 0.00247 [I 2022-10-27 21:40:20] ...trap/engines/engine.py.230: loss: 0.14383 [I 2022-10-27 21:40:20] ...optimizers/trijoint.py.151: total_norm: 38.899292 [I 2022-10-27 21:40:26] ...trap/engines/engine.py.224: train: epoch 89 | batch 1630/2304 [I 2022-10-27 21:40:26] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:14 [I 2022-10-27 21:40:26] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00244 [I 2022-10-27 21:40:26] ...trap/engines/engine.py.230: loss: 0.13063 [I 2022-10-27 21:40:26] ...optimizers/trijoint.py.151: total_norm: 14.703588 [I 2022-10-27 21:40:31] ...trap/engines/engine.py.224: train: epoch 89 | batch 1640/2304 [I 2022-10-27 21:40:31] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:07 [I 2022-10-27 21:40:31] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00255 [I 2022-10-27 21:40:31] ...trap/engines/engine.py.230: loss: 0.12388 [I 2022-10-27 21:40:31] ...optimizers/trijoint.py.151: total_norm: 20.347595 [I 2022-10-27 21:40:37] ...trap/engines/engine.py.224: train: epoch 89 | batch 1650/2304 [I 2022-10-27 21:40:37] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:03 [I 2022-10-27 21:40:37] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00243 [I 2022-10-27 21:40:37] ...trap/engines/engine.py.230: loss: 0.10666 [I 2022-10-27 21:40:37] ...optimizers/trijoint.py.151: total_norm: 21.573626 [I 2022-10-27 21:40:42] ...trap/engines/engine.py.224: train: epoch 89 | batch 1660/2304 [I 2022-10-27 21:40:42] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:05:56 [I 2022-10-27 21:40:42] ...trap/engines/engine.py.229: process: 0.55432 | load: 0.00247 [I 2022-10-27 21:40:42] ...trap/engines/engine.py.230: loss: 0.15862 [I 2022-10-27 21:40:42] ...optimizers/trijoint.py.151: total_norm: 12.569676 [I 2022-10-27 21:40:48] ...trap/engines/engine.py.224: train: epoch 89 | batch 1670/2304 [I 2022-10-27 21:40:48] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:53 [I 2022-10-27 21:40:48] ...trap/engines/engine.py.229: process: 0.54973 | load: 0.00239 [I 2022-10-27 21:40:48] ...trap/engines/engine.py.230: loss: 0.10134 [I 2022-10-27 21:40:48] ...optimizers/trijoint.py.151: total_norm: 9.480404 [I 2022-10-27 21:40:54] ...trap/engines/engine.py.224: train: epoch 89 | batch 1680/2304 [I 2022-10-27 21:40:54] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:52 [I 2022-10-27 21:40:54] ...trap/engines/engine.py.229: process: 0.55150 | load: 0.00253 [I 2022-10-27 21:40:54] ...trap/engines/engine.py.230: loss: 0.09558 [I 2022-10-27 21:40:54] ...optimizers/trijoint.py.151: total_norm: 9.336104 [I 2022-10-27 21:40:59] ...trap/engines/engine.py.224: train: epoch 89 | batch 1690/2304 [I 2022-10-27 21:40:59] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:43 [I 2022-10-27 21:40:59] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00249 [I 2022-10-27 21:40:59] ...trap/engines/engine.py.230: loss: 0.15134 [I 2022-10-27 21:40:59] ...optimizers/trijoint.py.151: total_norm: 25.531399 [I 2022-10-27 21:41:05] ...trap/engines/engine.py.224: train: epoch 89 | batch 1700/2304 [I 2022-10-27 21:41:05] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:39 [I 2022-10-27 21:41:05] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00246 [I 2022-10-27 21:41:05] ...trap/engines/engine.py.230: loss: 0.17532 [I 2022-10-27 21:41:05] ...optimizers/trijoint.py.151: total_norm: 14.293430 [I 2022-10-27 21:41:10] ...trap/engines/engine.py.224: train: epoch 89 | batch 1710/2304 [I 2022-10-27 21:41:10] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:32 [I 2022-10-27 21:41:10] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00248 [I 2022-10-27 21:41:10] ...trap/engines/engine.py.230: loss: 0.15199 [I 2022-10-27 21:41:10] ...optimizers/trijoint.py.151: total_norm: 13.906893 [I 2022-10-27 21:41:16] ...trap/engines/engine.py.224: train: epoch 89 | batch 1720/2304 [I 2022-10-27 21:41:16] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:24 [I 2022-10-27 21:41:16] ...trap/engines/engine.py.229: process: 0.54178 | load: 0.00242 [I 2022-10-27 21:41:16] ...trap/engines/engine.py.230: loss: 0.13908 [I 2022-10-27 21:41:16] ...optimizers/trijoint.py.151: total_norm: 12.480085 [I 2022-10-27 21:41:21] ...trap/engines/engine.py.224: train: epoch 89 | batch 1730/2304 [I 2022-10-27 21:41:21] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:20 [I 2022-10-27 21:41:21] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00242 [I 2022-10-27 21:41:21] ...trap/engines/engine.py.230: loss: 0.13435 [I 2022-10-27 21:41:21] ...optimizers/trijoint.py.151: total_norm: 16.667753 [I 2022-10-27 21:41:27] ...trap/engines/engine.py.224: train: epoch 89 | batch 1740/2304 [I 2022-10-27 21:41:27] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:20 [I 2022-10-27 21:41:27] ...trap/engines/engine.py.229: process: 0.56462 | load: 0.00243 [I 2022-10-27 21:41:27] ...trap/engines/engine.py.230: loss: 0.09189 [I 2022-10-27 21:41:27] ...optimizers/trijoint.py.151: total_norm: 10.439425 [I 2022-10-27 21:41:33] ...trap/engines/engine.py.224: train: epoch 89 | batch 1750/2304 [I 2022-10-27 21:41:33] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:10 [I 2022-10-27 21:41:33] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00246 [I 2022-10-27 21:41:33] ...trap/engines/engine.py.230: loss: 0.09262 [I 2022-10-27 21:41:33] ...optimizers/trijoint.py.151: total_norm: 13.167983 [I 2022-10-27 21:41:38] ...trap/engines/engine.py.224: train: epoch 89 | batch 1760/2304 [I 2022-10-27 21:41:38] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:03 [I 2022-10-27 21:41:38] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00242 [I 2022-10-27 21:41:38] ...trap/engines/engine.py.230: loss: 0.16319 [I 2022-10-27 21:41:38] ...optimizers/trijoint.py.151: total_norm: 21.782860 [I 2022-10-27 21:41:44] ...trap/engines/engine.py.224: train: epoch 89 | batch 1770/2304 [I 2022-10-27 21:41:44] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:04:55 [I 2022-10-27 21:41:44] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00241 [I 2022-10-27 21:41:44] ...trap/engines/engine.py.230: loss: 0.09040 [I 2022-10-27 21:41:44] ...optimizers/trijoint.py.151: total_norm: 14.672954 [I 2022-10-27 21:41:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 1780/2304 [I 2022-10-27 21:41:50] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:51 [I 2022-10-27 21:41:50] ...trap/engines/engine.py.229: process: 0.56822 | load: 0.00242 [I 2022-10-27 21:41:50] ...trap/engines/engine.py.230: loss: 0.15832 [I 2022-10-27 21:41:50] ...optimizers/trijoint.py.151: total_norm: 19.183895 [I 2022-10-27 21:41:55] ...trap/engines/engine.py.224: train: epoch 89 | batch 1790/2304 [I 2022-10-27 21:41:55] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:04:45 [I 2022-10-27 21:41:55] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00243 [I 2022-10-27 21:41:55] ...trap/engines/engine.py.230: loss: 0.13509 [I 2022-10-27 21:41:55] ...optimizers/trijoint.py.151: total_norm: 21.332670 [I 2022-10-27 21:42:01] ...trap/engines/engine.py.224: train: epoch 89 | batch 1800/2304 [I 2022-10-27 21:42:01] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:40 [I 2022-10-27 21:42:01] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00281 [I 2022-10-27 21:42:01] ...trap/engines/engine.py.230: loss: 0.14569 [I 2022-10-27 21:42:01] ...optimizers/trijoint.py.151: total_norm: 16.185089 [I 2022-10-27 21:42:06] ...trap/engines/engine.py.224: train: epoch 89 | batch 1810/2304 [I 2022-10-27 21:42:06] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:35 [I 2022-10-27 21:42:06] ...trap/engines/engine.py.229: process: 0.54511 | load: 0.00260 [I 2022-10-27 21:42:06] ...trap/engines/engine.py.230: loss: 0.11078 [I 2022-10-27 21:42:06] ...optimizers/trijoint.py.151: total_norm: 10.921573 [I 2022-10-27 21:42:12] ...trap/engines/engine.py.224: train: epoch 89 | batch 1820/2304 [I 2022-10-27 21:42:12] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:28 [I 2022-10-27 21:42:12] ...trap/engines/engine.py.229: process: 0.54937 | load: 0.00251 [I 2022-10-27 21:42:12] ...trap/engines/engine.py.230: loss: 0.10522 [I 2022-10-27 21:42:12] ...optimizers/trijoint.py.151: total_norm: 14.039621 [I 2022-10-27 21:42:17] ...trap/engines/engine.py.224: train: epoch 89 | batch 1830/2304 [I 2022-10-27 21:42:17] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:22 [I 2022-10-27 21:42:17] ...trap/engines/engine.py.229: process: 0.54035 | load: 0.00249 [I 2022-10-27 21:42:17] ...trap/engines/engine.py.230: loss: 0.09220 [I 2022-10-27 21:42:17] ...optimizers/trijoint.py.151: total_norm: 13.058912 [I 2022-10-27 21:42:23] ...trap/engines/engine.py.224: train: epoch 89 | batch 1840/2304 [I 2022-10-27 21:42:23] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:15 [I 2022-10-27 21:42:23] ...trap/engines/engine.py.229: process: 0.54281 | load: 0.00247 [I 2022-10-27 21:42:23] ...trap/engines/engine.py.230: loss: 0.09720 [I 2022-10-27 21:42:23] ...optimizers/trijoint.py.151: total_norm: 13.912975 [I 2022-10-27 21:42:28] ...trap/engines/engine.py.224: train: epoch 89 | batch 1850/2304 [I 2022-10-27 21:42:28] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:11 [I 2022-10-27 21:42:28] ...trap/engines/engine.py.229: process: 0.54820 | load: 0.00242 [I 2022-10-27 21:42:28] ...trap/engines/engine.py.230: loss: 0.08551 [I 2022-10-27 21:42:28] ...optimizers/trijoint.py.151: total_norm: 11.884874 [I 2022-10-27 21:42:34] ...trap/engines/engine.py.224: train: epoch 89 | batch 1860/2304 [I 2022-10-27 21:42:34] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:07 [I 2022-10-27 21:42:34] ...trap/engines/engine.py.229: process: 0.56290 | load: 0.00243 [I 2022-10-27 21:42:34] ...trap/engines/engine.py.230: loss: 0.08243 [I 2022-10-27 21:42:34] ...optimizers/trijoint.py.151: total_norm: 7.982818 [I 2022-10-27 21:42:40] ...trap/engines/engine.py.224: train: epoch 89 | batch 1870/2304 [I 2022-10-27 21:42:40] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:04 [I 2022-10-27 21:42:40] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00241 [I 2022-10-27 21:42:40] ...trap/engines/engine.py.230: loss: 0.12094 [I 2022-10-27 21:42:40] ...optimizers/trijoint.py.151: total_norm: 11.198941 [I 2022-10-27 21:42:45] ...trap/engines/engine.py.224: train: epoch 89 | batch 1880/2304 [I 2022-10-27 21:42:45] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:59 [I 2022-10-27 21:42:45] ...trap/engines/engine.py.229: process: 0.57049 | load: 0.00253 [I 2022-10-27 21:42:45] ...trap/engines/engine.py.230: loss: 0.15832 [I 2022-10-27 21:42:45] ...optimizers/trijoint.py.151: total_norm: 29.130356 [I 2022-10-27 21:42:51] ...trap/engines/engine.py.224: train: epoch 89 | batch 1890/2304 [I 2022-10-27 21:42:51] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:03:52 [I 2022-10-27 21:42:51] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00245 [I 2022-10-27 21:42:51] ...trap/engines/engine.py.230: loss: 0.13754 [I 2022-10-27 21:42:51] ...optimizers/trijoint.py.151: total_norm: 12.207047 [I 2022-10-27 21:42:57] ...trap/engines/engine.py.224: train: epoch 89 | batch 1900/2304 [I 2022-10-27 21:42:57] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:03:46 [I 2022-10-27 21:42:57] ...trap/engines/engine.py.229: process: 0.56757 | load: 0.00250 [I 2022-10-27 21:42:57] ...trap/engines/engine.py.230: loss: 0.09785 [I 2022-10-27 21:42:57] ...optimizers/trijoint.py.151: total_norm: 7.444243 [I 2022-10-27 21:43:02] ...trap/engines/engine.py.224: train: epoch 89 | batch 1910/2304 [I 2022-10-27 21:43:02] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:38 [I 2022-10-27 21:43:02] ...trap/engines/engine.py.229: process: 0.54923 | load: 0.00244 [I 2022-10-27 21:43:02] ...trap/engines/engine.py.230: loss: 0.09838 [I 2022-10-27 21:43:02] ...optimizers/trijoint.py.151: total_norm: 10.116141 [I 2022-10-27 21:43:08] ...trap/engines/engine.py.224: train: epoch 89 | batch 1920/2304 [I 2022-10-27 21:43:08] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:33 [I 2022-10-27 21:43:08] ...trap/engines/engine.py.229: process: 0.54865 | load: 0.00249 [I 2022-10-27 21:43:08] ...trap/engines/engine.py.230: loss: 0.16926 [I 2022-10-27 21:43:08] ...optimizers/trijoint.py.151: total_norm: 8.024975 [I 2022-10-27 21:43:13] ...trap/engines/engine.py.224: train: epoch 89 | batch 1930/2304 [I 2022-10-27 21:43:13] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:29 [I 2022-10-27 21:43:13] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00242 [I 2022-10-27 21:43:13] ...trap/engines/engine.py.230: loss: 0.16265 [I 2022-10-27 21:43:13] ...optimizers/trijoint.py.151: total_norm: 17.496889 [I 2022-10-27 21:43:19] ...trap/engines/engine.py.224: train: epoch 89 | batch 1940/2304 [I 2022-10-27 21:43:19] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:23 [I 2022-10-27 21:43:19] ...trap/engines/engine.py.229: process: 0.56166 | load: 0.00249 [I 2022-10-27 21:43:19] ...trap/engines/engine.py.230: loss: 0.10361 [I 2022-10-27 21:43:19] ...optimizers/trijoint.py.151: total_norm: 16.152102 [I 2022-10-27 21:43:25] ...trap/engines/engine.py.224: train: epoch 89 | batch 1950/2304 [I 2022-10-27 21:43:25] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:18 [I 2022-10-27 21:43:25] ...trap/engines/engine.py.229: process: 0.55868 | load: 0.00241 [I 2022-10-27 21:43:25] ...trap/engines/engine.py.230: loss: 0.09842 [I 2022-10-27 21:43:25] ...optimizers/trijoint.py.151: total_norm: 25.208433 [I 2022-10-27 21:43:30] ...trap/engines/engine.py.224: train: epoch 89 | batch 1960/2304 [I 2022-10-27 21:43:30] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:12 [I 2022-10-27 21:43:30] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00261 [I 2022-10-27 21:43:30] ...trap/engines/engine.py.230: loss: 0.14885 [I 2022-10-27 21:43:30] ...optimizers/trijoint.py.151: total_norm: 10.135762 [I 2022-10-27 21:43:36] ...trap/engines/engine.py.224: train: epoch 89 | batch 1970/2304 [I 2022-10-27 21:43:36] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:06 [I 2022-10-27 21:43:36] ...trap/engines/engine.py.229: process: 0.55602 | load: 0.00242 [I 2022-10-27 21:43:36] ...trap/engines/engine.py.230: loss: 0.14854 [I 2022-10-27 21:43:36] ...optimizers/trijoint.py.151: total_norm: 15.539806 [I 2022-10-27 21:43:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 1980/2304 [I 2022-10-27 21:43:41] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:02:59 [I 2022-10-27 21:43:41] ...trap/engines/engine.py.229: process: 0.55082 | load: 0.00243 [I 2022-10-27 21:43:41] ...trap/engines/engine.py.230: loss: 0.12897 [I 2022-10-27 21:43:41] ...optimizers/trijoint.py.151: total_norm: 33.581535 [I 2022-10-27 21:43:47] ...trap/engines/engine.py.224: train: epoch 89 | batch 1990/2304 [I 2022-10-27 21:43:47] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:02:56 [I 2022-10-27 21:43:47] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00240 [I 2022-10-27 21:43:47] ...trap/engines/engine.py.230: loss: 0.11106 [I 2022-10-27 21:43:47] ...optimizers/trijoint.py.151: total_norm: 17.376692 [I 2022-10-27 21:43:52] ...trap/engines/engine.py.224: train: epoch 89 | batch 2000/2304 [I 2022-10-27 21:43:52] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:02:49 [I 2022-10-27 21:43:52] ...trap/engines/engine.py.229: process: 0.56634 | load: 0.00242 [I 2022-10-27 21:43:52] ...trap/engines/engine.py.230: loss: 0.12549 [I 2022-10-27 21:43:52] ...optimizers/trijoint.py.151: total_norm: 10.171034 [I 2022-10-27 21:43:58] ...trap/engines/engine.py.224: train: epoch 89 | batch 2010/2304 [I 2022-10-27 21:43:58] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:02:43 [I 2022-10-27 21:43:58] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00249 [I 2022-10-27 21:43:58] ...trap/engines/engine.py.230: loss: 0.15954 [I 2022-10-27 21:43:58] ...optimizers/trijoint.py.151: total_norm: 32.215717 [I 2022-10-27 21:44:04] ...trap/engines/engine.py.224: train: epoch 89 | batch 2020/2304 [I 2022-10-27 21:44:04] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:38 [I 2022-10-27 21:44:04] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00247 [I 2022-10-27 21:44:04] ...trap/engines/engine.py.230: loss: 0.07487 [I 2022-10-27 21:44:04] ...optimizers/trijoint.py.151: total_norm: 13.765435 [I 2022-10-27 21:44:09] ...trap/engines/engine.py.224: train: epoch 89 | batch 2030/2304 [I 2022-10-27 21:44:09] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:33 [I 2022-10-27 21:44:09] ...trap/engines/engine.py.229: process: 0.56935 | load: 0.00254 [I 2022-10-27 21:44:09] ...trap/engines/engine.py.230: loss: 0.22416 [I 2022-10-27 21:44:09] ...optimizers/trijoint.py.151: total_norm: 14.711578 [I 2022-10-27 21:44:15] ...trap/engines/engine.py.224: train: epoch 89 | batch 2040/2304 [I 2022-10-27 21:44:15] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:27 [I 2022-10-27 21:44:15] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00245 [I 2022-10-27 21:44:15] ...trap/engines/engine.py.230: loss: 0.14071 [I 2022-10-27 21:44:15] ...optimizers/trijoint.py.151: total_norm: 8.605663 [I 2022-10-27 21:44:20] ...trap/engines/engine.py.224: train: epoch 89 | batch 2050/2304 [I 2022-10-27 21:44:20] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:21 [I 2022-10-27 21:44:20] ...trap/engines/engine.py.229: process: 0.56699 | load: 0.00244 [I 2022-10-27 21:44:20] ...trap/engines/engine.py.230: loss: 0.12199 [I 2022-10-27 21:44:20] ...optimizers/trijoint.py.151: total_norm: 8.093856 [I 2022-10-27 21:44:26] ...trap/engines/engine.py.224: train: epoch 89 | batch 2060/2304 [I 2022-10-27 21:44:26] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:15 [I 2022-10-27 21:44:26] ...trap/engines/engine.py.229: process: 0.55081 | load: 0.00245 [I 2022-10-27 21:44:26] ...trap/engines/engine.py.230: loss: 0.14566 [I 2022-10-27 21:44:26] ...optimizers/trijoint.py.151: total_norm: 31.543789 [I 2022-10-27 21:44:32] ...trap/engines/engine.py.224: train: epoch 89 | batch 2070/2304 [I 2022-10-27 21:44:32] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:10 [I 2022-10-27 21:44:32] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00244 [I 2022-10-27 21:44:32] ...trap/engines/engine.py.230: loss: 0.09514 [I 2022-10-27 21:44:32] ...optimizers/trijoint.py.151: total_norm: 10.054337 [I 2022-10-27 21:44:37] ...trap/engines/engine.py.224: train: epoch 89 | batch 2080/2304 [I 2022-10-27 21:44:37] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:04 [I 2022-10-27 21:44:37] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00246 [I 2022-10-27 21:44:37] ...trap/engines/engine.py.230: loss: 0.13283 [I 2022-10-27 21:44:37] ...optimizers/trijoint.py.151: total_norm: 18.866446 [I 2022-10-27 21:44:43] ...trap/engines/engine.py.224: train: epoch 89 | batch 2090/2304 [I 2022-10-27 21:44:43] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:01:59 [I 2022-10-27 21:44:43] ...trap/engines/engine.py.229: process: 0.55480 | load: 0.00244 [I 2022-10-27 21:44:43] ...trap/engines/engine.py.230: loss: 0.09970 [I 2022-10-27 21:44:43] ...optimizers/trijoint.py.151: total_norm: 15.672823 [I 2022-10-27 21:44:48] ...trap/engines/engine.py.224: train: epoch 89 | batch 2100/2304 [I 2022-10-27 21:44:48] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:54 [I 2022-10-27 21:44:48] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00250 [I 2022-10-27 21:44:48] ...trap/engines/engine.py.230: loss: 0.14535 [I 2022-10-27 21:44:48] ...optimizers/trijoint.py.151: total_norm: 14.509513 [I 2022-10-27 21:44:54] ...trap/engines/engine.py.224: train: epoch 89 | batch 2110/2304 [I 2022-10-27 21:44:54] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:01:47 [I 2022-10-27 21:44:54] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00248 [I 2022-10-27 21:44:54] ...trap/engines/engine.py.230: loss: 0.15391 [I 2022-10-27 21:44:54] ...optimizers/trijoint.py.151: total_norm: 20.834507 [I 2022-10-27 21:45:00] ...trap/engines/engine.py.224: train: epoch 89 | batch 2120/2304 [I 2022-10-27 21:45:00] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:01:43 [I 2022-10-27 21:45:00] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00254 [I 2022-10-27 21:45:00] ...trap/engines/engine.py.230: loss: 0.11437 [I 2022-10-27 21:45:00] ...optimizers/trijoint.py.151: total_norm: 11.360951 [I 2022-10-27 21:45:05] ...trap/engines/engine.py.224: train: epoch 89 | batch 2130/2304 [I 2022-10-27 21:45:05] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:36 [I 2022-10-27 21:45:05] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00253 [I 2022-10-27 21:45:05] ...trap/engines/engine.py.230: loss: 0.10855 [I 2022-10-27 21:45:05] ...optimizers/trijoint.py.151: total_norm: 19.990337 [I 2022-10-27 21:45:11] ...trap/engines/engine.py.224: train: epoch 89 | batch 2140/2304 [I 2022-10-27 21:45:11] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:31 [I 2022-10-27 21:45:11] ...trap/engines/engine.py.229: process: 0.56192 | load: 0.00250 [I 2022-10-27 21:45:11] ...trap/engines/engine.py.230: loss: 0.09775 [I 2022-10-27 21:45:11] ...optimizers/trijoint.py.151: total_norm: 9.805277 [I 2022-10-27 21:45:16] ...trap/engines/engine.py.224: train: epoch 89 | batch 2150/2304 [I 2022-10-27 21:45:16] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:26 [I 2022-10-27 21:45:16] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00248 [I 2022-10-27 21:45:16] ...trap/engines/engine.py.230: loss: 0.12968 [I 2022-10-27 21:45:16] ...optimizers/trijoint.py.151: total_norm: 21.442240 [I 2022-10-27 21:45:22] ...trap/engines/engine.py.224: train: epoch 89 | batch 2160/2304 [I 2022-10-27 21:45:22] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:20 [I 2022-10-27 21:45:22] ...trap/engines/engine.py.229: process: 0.56592 | load: 0.00245 [I 2022-10-27 21:45:22] ...trap/engines/engine.py.230: loss: 0.09238 [I 2022-10-27 21:45:22] ...optimizers/trijoint.py.151: total_norm: 47.468628 [I 2022-10-27 21:45:28] ...trap/engines/engine.py.224: train: epoch 89 | batch 2170/2304 [I 2022-10-27 21:45:28] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:14 [I 2022-10-27 21:45:28] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00245 [I 2022-10-27 21:45:28] ...trap/engines/engine.py.230: loss: 0.12383 [I 2022-10-27 21:45:28] ...optimizers/trijoint.py.151: total_norm: 9.702476 [I 2022-10-27 21:45:33] ...trap/engines/engine.py.224: train: epoch 89 | batch 2180/2304 [I 2022-10-27 21:45:33] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:09 [I 2022-10-27 21:45:33] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00245 [I 2022-10-27 21:45:33] ...trap/engines/engine.py.230: loss: 0.18193 [I 2022-10-27 21:45:33] ...optimizers/trijoint.py.151: total_norm: 28.159931 [I 2022-10-27 21:45:39] ...trap/engines/engine.py.224: train: epoch 89 | batch 2190/2304 [I 2022-10-27 21:45:39] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:03 [I 2022-10-27 21:45:39] ...trap/engines/engine.py.229: process: 0.55343 | load: 0.00250 [I 2022-10-27 21:45:39] ...trap/engines/engine.py.230: loss: 0.15862 [I 2022-10-27 21:45:39] ...optimizers/trijoint.py.151: total_norm: 47.640450 [I 2022-10-27 21:45:44] ...trap/engines/engine.py.224: train: epoch 89 | batch 2200/2304 [I 2022-10-27 21:45:44] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:57 [I 2022-10-27 21:45:44] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00247 [I 2022-10-27 21:45:44] ...trap/engines/engine.py.230: loss: 0.12973 [I 2022-10-27 21:45:44] ...optimizers/trijoint.py.151: total_norm: 22.823721 [I 2022-10-27 21:45:50] ...trap/engines/engine.py.224: train: epoch 89 | batch 2210/2304 [I 2022-10-27 21:45:50] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:00:52 [I 2022-10-27 21:45:50] ...trap/engines/engine.py.229: process: 0.54746 | load: 0.00243 [I 2022-10-27 21:45:50] ...trap/engines/engine.py.230: loss: 0.11125 [I 2022-10-27 21:45:50] ...optimizers/trijoint.py.151: total_norm: 10.694851 [I 2022-10-27 21:45:55] ...trap/engines/engine.py.224: train: epoch 89 | batch 2220/2304 [I 2022-10-27 21:45:55] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:00:46 [I 2022-10-27 21:45:55] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00241 [I 2022-10-27 21:45:55] ...trap/engines/engine.py.230: loss: 0.11090 [I 2022-10-27 21:45:55] ...optimizers/trijoint.py.151: total_norm: 16.376026 [I 2022-10-27 21:46:01] ...trap/engines/engine.py.224: train: epoch 89 | batch 2230/2304 [I 2022-10-27 21:46:01] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:41 [I 2022-10-27 21:46:01] ...trap/engines/engine.py.229: process: 0.56824 | load: 0.00246 [I 2022-10-27 21:46:01] ...trap/engines/engine.py.230: loss: 0.17638 [I 2022-10-27 21:46:01] ...optimizers/trijoint.py.151: total_norm: 15.665126 [I 2022-10-27 21:46:07] ...trap/engines/engine.py.224: train: epoch 89 | batch 2240/2304 [I 2022-10-27 21:46:07] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:35 [I 2022-10-27 21:46:07] ...trap/engines/engine.py.229: process: 0.55360 | load: 0.00246 [I 2022-10-27 21:46:07] ...trap/engines/engine.py.230: loss: 0.15915 [I 2022-10-27 21:46:07] ...optimizers/trijoint.py.151: total_norm: 27.012941 [I 2022-10-27 21:46:12] ...trap/engines/engine.py.224: train: epoch 89 | batch 2250/2304 [I 2022-10-27 21:46:12] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:30 [I 2022-10-27 21:46:12] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00245 [I 2022-10-27 21:46:12] ...trap/engines/engine.py.230: loss: 0.13802 [I 2022-10-27 21:46:12] ...optimizers/trijoint.py.151: total_norm: 23.479048 [I 2022-10-27 21:46:18] ...trap/engines/engine.py.224: train: epoch 89 | batch 2260/2304 [I 2022-10-27 21:46:18] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:24 [I 2022-10-27 21:46:18] ...trap/engines/engine.py.229: process: 0.56042 | load: 0.00247 [I 2022-10-27 21:46:18] ...trap/engines/engine.py.230: loss: 0.09483 [I 2022-10-27 21:46:18] ...optimizers/trijoint.py.151: total_norm: 13.562951 [I 2022-10-27 21:46:24] ...trap/engines/engine.py.224: train: epoch 89 | batch 2270/2304 [I 2022-10-27 21:46:24] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:19 [I 2022-10-27 21:46:24] ...trap/engines/engine.py.229: process: 0.58184 | load: 0.00245 [I 2022-10-27 21:46:24] ...trap/engines/engine.py.230: loss: 0.18873 [I 2022-10-27 21:46:24] ...optimizers/trijoint.py.151: total_norm: 15.494880 [I 2022-10-27 21:46:29] ...trap/engines/engine.py.224: train: epoch 89 | batch 2280/2304 [I 2022-10-27 21:46:29] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:13 [I 2022-10-27 21:46:29] ...trap/engines/engine.py.229: process: 0.57264 | load: 0.00242 [I 2022-10-27 21:46:29] ...trap/engines/engine.py.230: loss: 0.10494 [I 2022-10-27 21:46:29] ...optimizers/trijoint.py.151: total_norm: 9.983413 [I 2022-10-27 21:46:35] ...trap/engines/engine.py.224: train: epoch 89 | batch 2290/2304 [I 2022-10-27 21:46:35] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:07 [I 2022-10-27 21:46:35] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00242 [I 2022-10-27 21:46:35] ...trap/engines/engine.py.230: loss: 0.16120 [I 2022-10-27 21:46:35] ...optimizers/trijoint.py.151: total_norm: 16.795773 [I 2022-10-27 21:46:41] ...trap/engines/engine.py.224: train: epoch 89 | batch 2300/2304 [I 2022-10-27 21:46:41] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:02 [I 2022-10-27 21:46:41] ...trap/engines/engine.py.229: process: 0.54519 | load: 0.00015 [I 2022-10-27 21:46:41] ...trap/engines/engine.py.230: loss: 0.12135 [I 2022-10-27 21:46:41] ...optimizers/trijoint.py.151: total_norm: 12.931936 [I 2022-10-27 21:46:43] ...trap/engines/engine.py.224: train: epoch 89 | batch 2304/2304 [I 2022-10-27 21:46:43] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:00 [I 2022-10-27 21:46:43] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00015 [I 2022-10-27 21:46:43] ...trap/engines/engine.py.230: loss: 0.18162 [I 2022-10-27 21:46:43] ...optimizers/trijoint.py.151: total_norm: 14.329278 [S 2022-10-27 21:46:44] ...trap/engines/engine.py.129: train_epoch.epoch: 89 [S 2022-10-27 21:46:44] ...trap/engines/engine.py.129: train_epoch.loss: 0.1323743978640521 [I 2022-10-27 21:46:44] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 21:46:45] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 89 [I 2022-10-27 21:46:45] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 21:46:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 21:46:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 21:46:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 21:46:50] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 21:46:50] ...trap/engines/engine.py.321: eval: epoch 89 | batch 0/510 [I 2022-10-27 21:46:50] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:44:30 [I 2022-10-27 21:46:50] ...trap/engines/engine.py.326: process: 5.23681 | load: 3.78312 [I 2022-10-27 21:46:54] ...trap/engines/engine.py.321: eval: epoch 89 | batch 10/510 [I 2022-10-27 21:46:54] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:09 [I 2022-10-27 21:46:54] ...trap/engines/engine.py.326: process: 0.32065 | load: 0.00018 [I 2022-10-27 21:46:57] ...trap/engines/engine.py.321: eval: epoch 89 | batch 20/510 [I 2022-10-27 21:46:57] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:13 [I 2022-10-27 21:46:57] ...trap/engines/engine.py.326: process: 0.33769 | load: 0.00018 [I 2022-10-27 21:47:00] ...trap/engines/engine.py.321: eval: epoch 89 | batch 30/510 [I 2022-10-27 21:47:00] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:43 [I 2022-10-27 21:47:00] ...trap/engines/engine.py.326: process: 0.34094 | load: 0.00017 [I 2022-10-27 21:47:04] ...trap/engines/engine.py.321: eval: epoch 89 | batch 40/510 [I 2022-10-27 21:47:04] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:36 [I 2022-10-27 21:47:04] ...trap/engines/engine.py.326: process: 0.32817 | load: 0.00017 [I 2022-10-27 21:47:07] ...trap/engines/engine.py.321: eval: epoch 89 | batch 50/510 [I 2022-10-27 21:47:07] ...trap/engines/engine.py.325: elapsed: 0:00:21 | left: 0:02:32 [I 2022-10-27 21:47:07] ...trap/engines/engine.py.326: process: 0.32375 | load: 0.00036 [I 2022-10-27 21:47:10] ...trap/engines/engine.py.321: eval: epoch 89 | batch 60/510 [I 2022-10-27 21:47:10] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:27 [I 2022-10-27 21:47:10] ...trap/engines/engine.py.326: process: 0.33370 | load: 0.00016 [I 2022-10-27 21:47:14] ...trap/engines/engine.py.321: eval: epoch 89 | batch 70/510 [I 2022-10-27 21:47:14] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:28 [I 2022-10-27 21:47:14] ...trap/engines/engine.py.326: process: 0.33164 | load: 0.00018 [I 2022-10-27 21:47:17] ...trap/engines/engine.py.321: eval: epoch 89 | batch 80/510 [I 2022-10-27 21:47:17] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:24 [I 2022-10-27 21:47:17] ...trap/engines/engine.py.326: process: 0.34930 | load: 0.00016 [I 2022-10-27 21:47:20] ...trap/engines/engine.py.321: eval: epoch 89 | batch 90/510 [I 2022-10-27 21:47:20] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:18 [I 2022-10-27 21:47:20] ...trap/engines/engine.py.326: process: 0.33071 | load: 0.00016 [I 2022-10-27 21:47:24] ...trap/engines/engine.py.321: eval: epoch 89 | batch 100/510 [I 2022-10-27 21:47:24] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:14 [I 2022-10-27 21:47:24] ...trap/engines/engine.py.326: process: 0.29461 | load: 0.00037 [I 2022-10-27 21:47:27] ...trap/engines/engine.py.321: eval: epoch 89 | batch 110/510 [I 2022-10-27 21:47:27] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-27 21:47:27] ...trap/engines/engine.py.326: process: 0.28974 | load: 0.00014 [I 2022-10-27 21:47:30] ...trap/engines/engine.py.321: eval: epoch 89 | batch 120/510 [I 2022-10-27 21:47:30] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 21:47:30] ...trap/engines/engine.py.326: process: 0.28961 | load: 0.00017 [I 2022-10-27 21:47:32] ...trap/engines/engine.py.321: eval: epoch 89 | batch 130/510 [I 2022-10-27 21:47:32] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:50 [I 2022-10-27 21:47:32] ...trap/engines/engine.py.326: process: 0.30315 | load: 0.00015 [I 2022-10-27 21:47:35] ...trap/engines/engine.py.321: eval: epoch 89 | batch 140/510 [I 2022-10-27 21:47:35] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-27 21:47:35] ...trap/engines/engine.py.326: process: 0.29337 | load: 0.00016 [I 2022-10-27 21:47:38] ...trap/engines/engine.py.321: eval: epoch 89 | batch 150/510 [I 2022-10-27 21:47:38] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-27 21:47:38] ...trap/engines/engine.py.326: process: 0.28663 | load: 0.00016 [I 2022-10-27 21:47:41] ...trap/engines/engine.py.321: eval: epoch 89 | batch 160/510 [I 2022-10-27 21:47:41] ...trap/engines/engine.py.325: elapsed: 0:00:55 | left: 0:01:40 [I 2022-10-27 21:47:41] ...trap/engines/engine.py.326: process: 0.29002 | load: 0.00018 [I 2022-10-27 21:47:44] ...trap/engines/engine.py.321: eval: epoch 89 | batch 170/510 [I 2022-10-27 21:47:44] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:38 [I 2022-10-27 21:47:44] ...trap/engines/engine.py.326: process: 0.28559 | load: 0.00033 [I 2022-10-27 21:47:47] ...trap/engines/engine.py.321: eval: epoch 89 | batch 180/510 [I 2022-10-27 21:47:47] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-27 21:47:47] ...trap/engines/engine.py.326: process: 0.29248 | load: 0.00016 [I 2022-10-27 21:47:50] ...trap/engines/engine.py.321: eval: epoch 89 | batch 190/510 [I 2022-10-27 21:47:50] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:33 [I 2022-10-27 21:47:50] ...trap/engines/engine.py.326: process: 0.29409 | load: 0.00023 [I 2022-10-27 21:47:53] ...trap/engines/engine.py.321: eval: epoch 89 | batch 200/510 [I 2022-10-27 21:47:53] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:34 [I 2022-10-27 21:47:53] ...trap/engines/engine.py.326: process: 0.28933 | load: 0.00017 [I 2022-10-27 21:47:56] ...trap/engines/engine.py.321: eval: epoch 89 | batch 210/510 [I 2022-10-27 21:47:56] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:28 [I 2022-10-27 21:47:56] ...trap/engines/engine.py.326: process: 0.28604 | load: 0.00022 [I 2022-10-27 21:47:59] ...trap/engines/engine.py.321: eval: epoch 89 | batch 220/510 [I 2022-10-27 21:47:59] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 21:47:59] ...trap/engines/engine.py.326: process: 0.28255 | load: 0.00018 [I 2022-10-27 21:48:02] ...trap/engines/engine.py.321: eval: epoch 89 | batch 230/510 [I 2022-10-27 21:48:02] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:21 [I 2022-10-27 21:48:02] ...trap/engines/engine.py.326: process: 0.29209 | load: 0.00022 [I 2022-10-27 21:48:04] ...trap/engines/engine.py.321: eval: epoch 89 | batch 240/510 [I 2022-10-27 21:48:04] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-27 21:48:04] ...trap/engines/engine.py.326: process: 0.27956 | load: 0.00051 [I 2022-10-27 21:48:07] ...trap/engines/engine.py.321: eval: epoch 89 | batch 250/510 [I 2022-10-27 21:48:07] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-27 21:48:07] ...trap/engines/engine.py.326: process: 0.28756 | load: 0.00015 [I 2022-10-27 21:48:10] ...trap/engines/engine.py.321: eval: epoch 89 | batch 260/510 [I 2022-10-27 21:48:10] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 21:48:10] ...trap/engines/engine.py.326: process: 0.28752 | load: 0.00018 [I 2022-10-27 21:48:13] ...trap/engines/engine.py.321: eval: epoch 89 | batch 270/510 [I 2022-10-27 21:48:13] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:10 [I 2022-10-27 21:48:13] ...trap/engines/engine.py.326: process: 0.28409 | load: 0.00020 [I 2022-10-27 21:48:16] ...trap/engines/engine.py.321: eval: epoch 89 | batch 280/510 [I 2022-10-27 21:48:16] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 21:48:16] ...trap/engines/engine.py.326: process: 0.28364 | load: 0.00018 [I 2022-10-27 21:48:19] ...trap/engines/engine.py.321: eval: epoch 89 | batch 290/510 [I 2022-10-27 21:48:19] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 21:48:19] ...trap/engines/engine.py.326: process: 0.28423 | load: 0.00021 [I 2022-10-27 21:48:22] ...trap/engines/engine.py.321: eval: epoch 89 | batch 300/510 [I 2022-10-27 21:48:22] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:01 [I 2022-10-27 21:48:22] ...trap/engines/engine.py.326: process: 0.29182 | load: 0.00018 [I 2022-10-27 21:48:25] ...trap/engines/engine.py.321: eval: epoch 89 | batch 310/510 [I 2022-10-27 21:48:25] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:58 [I 2022-10-27 21:48:25] ...trap/engines/engine.py.326: process: 0.30737 | load: 0.00019 [I 2022-10-27 21:48:28] ...trap/engines/engine.py.321: eval: epoch 89 | batch 320/510 [I 2022-10-27 21:48:28] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 21:48:28] ...trap/engines/engine.py.326: process: 0.28968 | load: 0.00016 [I 2022-10-27 21:48:31] ...trap/engines/engine.py.321: eval: epoch 89 | batch 330/510 [I 2022-10-27 21:48:31] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-27 21:48:31] ...trap/engines/engine.py.326: process: 0.29121 | load: 0.00015 [I 2022-10-27 21:48:33] ...trap/engines/engine.py.321: eval: epoch 89 | batch 340/510 [I 2022-10-27 21:48:33] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-27 21:48:33] ...trap/engines/engine.py.326: process: 0.28964 | load: 0.00020 [I 2022-10-27 21:48:36] ...trap/engines/engine.py.321: eval: epoch 89 | batch 350/510 [I 2022-10-27 21:48:36] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 21:48:36] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00020 [I 2022-10-27 21:48:39] ...trap/engines/engine.py.321: eval: epoch 89 | batch 360/510 [I 2022-10-27 21:48:39] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-27 21:48:39] ...trap/engines/engine.py.326: process: 0.28102 | load: 0.00020 [I 2022-10-27 21:48:42] ...trap/engines/engine.py.321: eval: epoch 89 | batch 370/510 [I 2022-10-27 21:48:42] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 21:48:42] ...trap/engines/engine.py.326: process: 0.28831 | load: 0.00020 [I 2022-10-27 21:48:45] ...trap/engines/engine.py.321: eval: epoch 89 | batch 380/510 [I 2022-10-27 21:48:45] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 21:48:45] ...trap/engines/engine.py.326: process: 0.28772 | load: 0.00018 [I 2022-10-27 21:48:48] ...trap/engines/engine.py.321: eval: epoch 89 | batch 390/510 [I 2022-10-27 21:48:48] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:35 [I 2022-10-27 21:48:48] ...trap/engines/engine.py.326: process: 0.28938 | load: 0.00020 [I 2022-10-27 21:48:51] ...trap/engines/engine.py.321: eval: epoch 89 | batch 400/510 [I 2022-10-27 21:48:51] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 21:48:51] ...trap/engines/engine.py.326: process: 0.28849 | load: 0.00019 [I 2022-10-27 21:48:54] ...trap/engines/engine.py.321: eval: epoch 89 | batch 410/510 [I 2022-10-27 21:48:54] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 21:48:54] ...trap/engines/engine.py.326: process: 0.28264 | load: 0.00024 [I 2022-10-27 21:48:57] ...trap/engines/engine.py.321: eval: epoch 89 | batch 420/510 [I 2022-10-27 21:48:57] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 21:48:57] ...trap/engines/engine.py.326: process: 0.28799 | load: 0.00021 [I 2022-10-27 21:48:59] ...trap/engines/engine.py.321: eval: epoch 89 | batch 430/510 [I 2022-10-27 21:48:59] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 21:48:59] ...trap/engines/engine.py.326: process: 0.29110 | load: 0.00021 [I 2022-10-27 21:49:02] ...trap/engines/engine.py.321: eval: epoch 89 | batch 440/510 [I 2022-10-27 21:49:02] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 21:49:02] ...trap/engines/engine.py.326: process: 0.28677 | load: 0.00019 [I 2022-10-27 21:49:05] ...trap/engines/engine.py.321: eval: epoch 89 | batch 450/510 [I 2022-10-27 21:49:05] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 21:49:05] ...trap/engines/engine.py.326: process: 0.28824 | load: 0.00020 [I 2022-10-27 21:49:08] ...trap/engines/engine.py.321: eval: epoch 89 | batch 460/510 [I 2022-10-27 21:49:08] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 21:49:08] ...trap/engines/engine.py.326: process: 0.29095 | load: 0.00020 [I 2022-10-27 21:49:11] ...trap/engines/engine.py.321: eval: epoch 89 | batch 470/510 [I 2022-10-27 21:49:11] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 21:49:11] ...trap/engines/engine.py.326: process: 0.28146 | load: 0.00016 [I 2022-10-27 21:49:14] ...trap/engines/engine.py.321: eval: epoch 89 | batch 480/510 [I 2022-10-27 21:49:14] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 21:49:14] ...trap/engines/engine.py.326: process: 0.27912 | load: 0.00022 [I 2022-10-27 21:49:17] ...trap/engines/engine.py.321: eval: epoch 89 | batch 490/510 [I 2022-10-27 21:49:17] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 21:49:17] ...trap/engines/engine.py.326: process: 0.28603 | load: 0.00022 [I 2022-10-27 21:49:20] ...trap/engines/engine.py.321: eval: epoch 89 | batch 500/510 [I 2022-10-27 21:49:20] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 21:49:20] ...trap/engines/engine.py.326: process: 0.29025 | load: 0.00019 [I 2022-10-27 21:49:22] ...trap/engines/engine.py.321: eval: epoch 89 | batch 510/510 [I 2022-10-27 21:49:23] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 21:49:23] ...trap/engines/engine.py.326: process: 0.28891 | load: 0.00017 [S 2022-10-27 21:49:23] ...trap/engines/engine.py.132: eval_epoch.epoch: 89 [I 2022-10-27 21:49:23] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.725900 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908400 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936100 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.729400 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905800 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935900 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009332 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009254 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008814 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012076 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007250 [S 2022-10-27 21:49:27] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009071 [I 2022-10-27 21:49:29] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 21:49:29] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 21:49:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 21:49:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 21:49:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 21:49:34] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 21:49:35] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 21:49:40] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 21:49:40] ...trap/engines/engine.py.163: Training model on trainset for epoch 90 [I 2022-10-27 21:49:40] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 21:49:43] ...trap/engines/engine.py.224: train: epoch 90 | batch 0/2304 [I 2022-10-27 21:49:43] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:41:34 [I 2022-10-27 21:49:43] ...trap/engines/engine.py.229: process: 2.64519 | load: 2.02902 [I 2022-10-27 21:49:43] ...trap/engines/engine.py.230: loss: 0.20735 [I 2022-10-27 21:49:43] ...optimizers/trijoint.py.151: total_norm: 18.906202 [I 2022-10-27 21:49:48] ...trap/engines/engine.py.224: train: epoch 90 | batch 10/2304 [I 2022-10-27 21:49:48] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:15 [I 2022-10-27 21:49:48] ...trap/engines/engine.py.229: process: 0.56731 | load: 0.00466 [I 2022-10-27 21:49:48] ...trap/engines/engine.py.230: loss: 0.09528 [I 2022-10-27 21:49:48] ...optimizers/trijoint.py.151: total_norm: 10.878682 [I 2022-10-27 21:49:54] ...trap/engines/engine.py.224: train: epoch 90 | batch 20/2304 [I 2022-10-27 21:49:54] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:17 [I 2022-10-27 21:49:54] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00346 [I 2022-10-27 21:49:54] ...trap/engines/engine.py.230: loss: 0.11737 [I 2022-10-27 21:49:54] ...optimizers/trijoint.py.151: total_norm: 16.206636 [I 2022-10-27 21:50:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 30/2304 [I 2022-10-27 21:50:00] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:23 [I 2022-10-27 21:50:00] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00270 [I 2022-10-27 21:50:00] ...trap/engines/engine.py.230: loss: 0.10032 [I 2022-10-27 21:50:00] ...optimizers/trijoint.py.151: total_norm: 10.147207 [I 2022-10-27 21:50:05] ...trap/engines/engine.py.224: train: epoch 90 | batch 40/2304 [I 2022-10-27 21:50:05] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:59 [I 2022-10-27 21:50:05] ...trap/engines/engine.py.229: process: 0.55829 | load: 0.00295 [I 2022-10-27 21:50:05] ...trap/engines/engine.py.230: loss: 0.09441 [I 2022-10-27 21:50:05] ...optimizers/trijoint.py.151: total_norm: 11.977530 [I 2022-10-27 21:50:11] ...trap/engines/engine.py.224: train: epoch 90 | batch 50/2304 [I 2022-10-27 21:50:11] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:02 [I 2022-10-27 21:50:11] ...trap/engines/engine.py.229: process: 0.55031 | load: 0.00312 [I 2022-10-27 21:50:11] ...trap/engines/engine.py.230: loss: 0.12745 [I 2022-10-27 21:50:11] ...optimizers/trijoint.py.151: total_norm: 19.407410 [I 2022-10-27 21:50:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 60/2304 [I 2022-10-27 21:50:17] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:56 [I 2022-10-27 21:50:17] ...trap/engines/engine.py.229: process: 0.56578 | load: 0.00290 [I 2022-10-27 21:50:17] ...trap/engines/engine.py.230: loss: 0.12340 [I 2022-10-27 21:50:17] ...optimizers/trijoint.py.151: total_norm: 12.497859 [I 2022-10-27 21:50:22] ...trap/engines/engine.py.224: train: epoch 90 | batch 70/2304 [I 2022-10-27 21:50:22] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:52 [I 2022-10-27 21:50:22] ...trap/engines/engine.py.229: process: 0.56442 | load: 0.00279 [I 2022-10-27 21:50:22] ...trap/engines/engine.py.230: loss: 0.10152 [I 2022-10-27 21:50:22] ...optimizers/trijoint.py.151: total_norm: 8.927226 [I 2022-10-27 21:50:28] ...trap/engines/engine.py.224: train: epoch 90 | batch 80/2304 [I 2022-10-27 21:50:28] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:54 [I 2022-10-27 21:50:28] ...trap/engines/engine.py.229: process: 0.57795 | load: 0.00271 [I 2022-10-27 21:50:28] ...trap/engines/engine.py.230: loss: 0.15344 [I 2022-10-27 21:50:28] ...optimizers/trijoint.py.151: total_norm: 26.054550 [I 2022-10-27 21:50:33] ...trap/engines/engine.py.224: train: epoch 90 | batch 90/2304 [I 2022-10-27 21:50:33] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:38 [I 2022-10-27 21:50:33] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00264 [I 2022-10-27 21:50:33] ...trap/engines/engine.py.230: loss: 0.17281 [I 2022-10-27 21:50:33] ...optimizers/trijoint.py.151: total_norm: 23.898146 [I 2022-10-27 21:50:39] ...trap/engines/engine.py.224: train: epoch 90 | batch 100/2304 [I 2022-10-27 21:50:39] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:28 [I 2022-10-27 21:50:39] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00249 [I 2022-10-27 21:50:39] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-27 21:50:39] ...optimizers/trijoint.py.151: total_norm: 20.030155 [I 2022-10-27 21:50:45] ...trap/engines/engine.py.224: train: epoch 90 | batch 110/2304 [I 2022-10-27 21:50:45] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:22 [I 2022-10-27 21:50:45] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00303 [I 2022-10-27 21:50:45] ...trap/engines/engine.py.230: loss: 0.28809 [I 2022-10-27 21:50:45] ...optimizers/trijoint.py.151: total_norm: 20.725832 [I 2022-10-27 21:50:50] ...trap/engines/engine.py.224: train: epoch 90 | batch 120/2304 [I 2022-10-27 21:50:50] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:16 [I 2022-10-27 21:50:50] ...trap/engines/engine.py.229: process: 0.56749 | load: 0.00246 [I 2022-10-27 21:50:50] ...trap/engines/engine.py.230: loss: 0.09341 [I 2022-10-27 21:50:50] ...optimizers/trijoint.py.151: total_norm: 11.272110 [I 2022-10-27 21:50:56] ...trap/engines/engine.py.224: train: epoch 90 | batch 130/2304 [I 2022-10-27 21:50:56] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:24 [I 2022-10-27 21:50:56] ...trap/engines/engine.py.229: process: 0.54843 | load: 0.00255 [I 2022-10-27 21:50:56] ...trap/engines/engine.py.230: loss: 0.18809 [I 2022-10-27 21:50:56] ...optimizers/trijoint.py.151: total_norm: 20.010813 [I 2022-10-27 21:51:01] ...trap/engines/engine.py.224: train: epoch 90 | batch 140/2304 [I 2022-10-27 21:51:01] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:07 [I 2022-10-27 21:51:01] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00252 [I 2022-10-27 21:51:01] ...trap/engines/engine.py.230: loss: 0.13314 [I 2022-10-27 21:51:01] ...optimizers/trijoint.py.151: total_norm: 11.579050 [I 2022-10-27 21:51:07] ...trap/engines/engine.py.224: train: epoch 90 | batch 150/2304 [I 2022-10-27 21:51:07] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:01 [I 2022-10-27 21:51:07] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00253 [I 2022-10-27 21:51:07] ...trap/engines/engine.py.230: loss: 0.15235 [I 2022-10-27 21:51:07] ...optimizers/trijoint.py.151: total_norm: 21.504887 [I 2022-10-27 21:51:13] ...trap/engines/engine.py.224: train: epoch 90 | batch 160/2304 [I 2022-10-27 21:51:13] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:56 [I 2022-10-27 21:51:13] ...trap/engines/engine.py.229: process: 0.56517 | load: 0.00264 [I 2022-10-27 21:51:13] ...trap/engines/engine.py.230: loss: 0.10738 [I 2022-10-27 21:51:13] ...optimizers/trijoint.py.151: total_norm: 14.100544 [I 2022-10-27 21:51:18] ...trap/engines/engine.py.224: train: epoch 90 | batch 170/2304 [I 2022-10-27 21:51:18] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:42 [I 2022-10-27 21:51:18] ...trap/engines/engine.py.229: process: 0.54518 | load: 0.00253 [I 2022-10-27 21:51:18] ...trap/engines/engine.py.230: loss: 0.16199 [I 2022-10-27 21:51:18] ...optimizers/trijoint.py.151: total_norm: 100.885620 [I 2022-10-27 21:51:24] ...trap/engines/engine.py.224: train: epoch 90 | batch 180/2304 [I 2022-10-27 21:51:24] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:42 [I 2022-10-27 21:51:24] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00266 [I 2022-10-27 21:51:24] ...trap/engines/engine.py.230: loss: 0.08616 [I 2022-10-27 21:51:24] ...optimizers/trijoint.py.151: total_norm: 13.983234 [I 2022-10-27 21:51:29] ...trap/engines/engine.py.224: train: epoch 90 | batch 190/2304 [I 2022-10-27 21:51:29] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:20:14 [I 2022-10-27 21:51:29] ...trap/engines/engine.py.229: process: 0.56306 | load: 0.00249 [I 2022-10-27 21:51:29] ...trap/engines/engine.py.230: loss: 0.12933 [I 2022-10-27 21:51:29] ...optimizers/trijoint.py.151: total_norm: 17.793272 [I 2022-10-27 21:51:35] ...trap/engines/engine.py.224: train: epoch 90 | batch 200/2304 [I 2022-10-27 21:51:35] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:35 [I 2022-10-27 21:51:35] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00247 [I 2022-10-27 21:51:35] ...trap/engines/engine.py.230: loss: 0.22926 [I 2022-10-27 21:51:35] ...optimizers/trijoint.py.151: total_norm: 24.101505 [I 2022-10-27 21:51:41] ...trap/engines/engine.py.224: train: epoch 90 | batch 210/2304 [I 2022-10-27 21:51:41] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:30 [I 2022-10-27 21:51:41] ...trap/engines/engine.py.229: process: 0.55113 | load: 0.00248 [I 2022-10-27 21:51:41] ...trap/engines/engine.py.230: loss: 0.12157 [I 2022-10-27 21:51:41] ...optimizers/trijoint.py.151: total_norm: 11.923162 [I 2022-10-27 21:51:46] ...trap/engines/engine.py.224: train: epoch 90 | batch 220/2304 [I 2022-10-27 21:51:46] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:18 [I 2022-10-27 21:51:46] ...trap/engines/engine.py.229: process: 0.55161 | load: 0.00243 [I 2022-10-27 21:51:46] ...trap/engines/engine.py.230: loss: 0.09565 [I 2022-10-27 21:51:46] ...optimizers/trijoint.py.151: total_norm: 9.933958 [I 2022-10-27 21:51:52] ...trap/engines/engine.py.224: train: epoch 90 | batch 230/2304 [I 2022-10-27 21:51:52] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:34 [I 2022-10-27 21:51:52] ...trap/engines/engine.py.229: process: 0.56555 | load: 0.00252 [I 2022-10-27 21:51:52] ...trap/engines/engine.py.230: loss: 0.09261 [I 2022-10-27 21:51:52] ...optimizers/trijoint.py.151: total_norm: 17.363098 [I 2022-10-27 21:51:58] ...trap/engines/engine.py.224: train: epoch 90 | batch 240/2304 [I 2022-10-27 21:51:58] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:19 [I 2022-10-27 21:51:58] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00256 [I 2022-10-27 21:51:58] ...trap/engines/engine.py.230: loss: 0.14798 [I 2022-10-27 21:51:58] ...optimizers/trijoint.py.151: total_norm: 9.042803 [I 2022-10-27 21:52:03] ...trap/engines/engine.py.224: train: epoch 90 | batch 250/2304 [I 2022-10-27 21:52:03] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:00 [I 2022-10-27 21:52:03] ...trap/engines/engine.py.229: process: 0.55003 | load: 0.00251 [I 2022-10-27 21:52:03] ...trap/engines/engine.py.230: loss: 0.14448 [I 2022-10-27 21:52:03] ...optimizers/trijoint.py.151: total_norm: 13.411066 [I 2022-10-27 21:52:09] ...trap/engines/engine.py.224: train: epoch 90 | batch 260/2304 [I 2022-10-27 21:52:09] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:02 [I 2022-10-27 21:52:09] ...trap/engines/engine.py.229: process: 0.55162 | load: 0.00262 [I 2022-10-27 21:52:09] ...trap/engines/engine.py.230: loss: 0.14528 [I 2022-10-27 21:52:09] ...optimizers/trijoint.py.151: total_norm: 10.201285 [I 2022-10-27 21:52:14] ...trap/engines/engine.py.224: train: epoch 90 | batch 270/2304 [I 2022-10-27 21:52:14] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:49 [I 2022-10-27 21:52:14] ...trap/engines/engine.py.229: process: 0.54756 | load: 0.00241 [I 2022-10-27 21:52:14] ...trap/engines/engine.py.230: loss: 0.12037 [I 2022-10-27 21:52:14] ...optimizers/trijoint.py.151: total_norm: 12.091389 [I 2022-10-27 21:52:20] ...trap/engines/engine.py.224: train: epoch 90 | batch 280/2304 [I 2022-10-27 21:52:20] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:47 [I 2022-10-27 21:52:20] ...trap/engines/engine.py.229: process: 0.56376 | load: 0.00245 [I 2022-10-27 21:52:20] ...trap/engines/engine.py.230: loss: 0.12364 [I 2022-10-27 21:52:20] ...optimizers/trijoint.py.151: total_norm: 32.493118 [I 2022-10-27 21:52:25] ...trap/engines/engine.py.224: train: epoch 90 | batch 290/2304 [I 2022-10-27 21:52:25] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:41 [I 2022-10-27 21:52:25] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00242 [I 2022-10-27 21:52:25] ...trap/engines/engine.py.230: loss: 0.16850 [I 2022-10-27 21:52:25] ...optimizers/trijoint.py.151: total_norm: 26.909880 [I 2022-10-27 21:52:31] ...trap/engines/engine.py.224: train: epoch 90 | batch 300/2304 [I 2022-10-27 21:52:31] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:31 [I 2022-10-27 21:52:31] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00251 [I 2022-10-27 21:52:31] ...trap/engines/engine.py.230: loss: 0.19194 [I 2022-10-27 21:52:31] ...optimizers/trijoint.py.151: total_norm: 18.628349 [I 2022-10-27 21:52:37] ...trap/engines/engine.py.224: train: epoch 90 | batch 310/2304 [I 2022-10-27 21:52:37] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:34 [I 2022-10-27 21:52:37] ...trap/engines/engine.py.229: process: 0.54998 | load: 0.00249 [I 2022-10-27 21:52:37] ...trap/engines/engine.py.230: loss: 0.14259 [I 2022-10-27 21:52:37] ...optimizers/trijoint.py.151: total_norm: 20.765730 [I 2022-10-27 21:52:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 320/2304 [I 2022-10-27 21:52:42] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:25 [I 2022-10-27 21:52:42] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00245 [I 2022-10-27 21:52:42] ...trap/engines/engine.py.230: loss: 0.14735 [I 2022-10-27 21:52:42] ...optimizers/trijoint.py.151: total_norm: 19.306232 [I 2022-10-27 21:52:48] ...trap/engines/engine.py.224: train: epoch 90 | batch 330/2304 [I 2022-10-27 21:52:48] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:20 [I 2022-10-27 21:52:48] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00242 [I 2022-10-27 21:52:48] ...trap/engines/engine.py.230: loss: 0.12909 [I 2022-10-27 21:52:48] ...optimizers/trijoint.py.151: total_norm: 10.454146 [I 2022-10-27 21:52:53] ...trap/engines/engine.py.224: train: epoch 90 | batch 340/2304 [I 2022-10-27 21:52:53] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:08 [I 2022-10-27 21:52:53] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00244 [I 2022-10-27 21:52:53] ...trap/engines/engine.py.230: loss: 0.10400 [I 2022-10-27 21:52:53] ...optimizers/trijoint.py.151: total_norm: 10.123628 [I 2022-10-27 21:52:59] ...trap/engines/engine.py.224: train: epoch 90 | batch 350/2304 [I 2022-10-27 21:52:59] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:11 [I 2022-10-27 21:52:59] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00243 [I 2022-10-27 21:52:59] ...trap/engines/engine.py.230: loss: 0.11013 [I 2022-10-27 21:52:59] ...optimizers/trijoint.py.151: total_norm: 15.264156 [I 2022-10-27 21:53:04] ...trap/engines/engine.py.224: train: epoch 90 | batch 360/2304 [I 2022-10-27 21:53:04] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:02 [I 2022-10-27 21:53:04] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00247 [I 2022-10-27 21:53:04] ...trap/engines/engine.py.230: loss: 0.09794 [I 2022-10-27 21:53:04] ...optimizers/trijoint.py.151: total_norm: 13.022709 [I 2022-10-27 21:53:10] ...trap/engines/engine.py.224: train: epoch 90 | batch 370/2304 [I 2022-10-27 21:53:10] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:11 [I 2022-10-27 21:53:10] ...trap/engines/engine.py.229: process: 0.57330 | load: 0.00250 [I 2022-10-27 21:53:10] ...trap/engines/engine.py.230: loss: 0.11476 [I 2022-10-27 21:53:10] ...optimizers/trijoint.py.151: total_norm: 12.732169 [I 2022-10-27 21:53:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 380/2304 [I 2022-10-27 21:53:16] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:47 [I 2022-10-27 21:53:16] ...trap/engines/engine.py.229: process: 0.54416 | load: 0.00246 [I 2022-10-27 21:53:16] ...trap/engines/engine.py.230: loss: 0.13272 [I 2022-10-27 21:53:16] ...optimizers/trijoint.py.151: total_norm: 17.967482 [I 2022-10-27 21:53:21] ...trap/engines/engine.py.224: train: epoch 90 | batch 390/2304 [I 2022-10-27 21:53:21] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:49 [I 2022-10-27 21:53:21] ...trap/engines/engine.py.229: process: 0.55975 | load: 0.00239 [I 2022-10-27 21:53:21] ...trap/engines/engine.py.230: loss: 0.10620 [I 2022-10-27 21:53:21] ...optimizers/trijoint.py.151: total_norm: 11.624557 [I 2022-10-27 21:53:27] ...trap/engines/engine.py.224: train: epoch 90 | batch 400/2304 [I 2022-10-27 21:53:27] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:41 [I 2022-10-27 21:53:27] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00250 [I 2022-10-27 21:53:27] ...trap/engines/engine.py.230: loss: 0.10531 [I 2022-10-27 21:53:27] ...optimizers/trijoint.py.151: total_norm: 8.954989 [I 2022-10-27 21:53:32] ...trap/engines/engine.py.224: train: epoch 90 | batch 410/2304 [I 2022-10-27 21:53:32] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:34 [I 2022-10-27 21:53:32] ...trap/engines/engine.py.229: process: 0.54887 | load: 0.00239 [I 2022-10-27 21:53:32] ...trap/engines/engine.py.230: loss: 0.12912 [I 2022-10-27 21:53:32] ...optimizers/trijoint.py.151: total_norm: 25.122358 [I 2022-10-27 21:53:38] ...trap/engines/engine.py.224: train: epoch 90 | batch 420/2304 [I 2022-10-27 21:53:38] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:24 [I 2022-10-27 21:53:38] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00244 [I 2022-10-27 21:53:38] ...trap/engines/engine.py.230: loss: 0.06712 [I 2022-10-27 21:53:38] ...optimizers/trijoint.py.151: total_norm: 14.177812 [I 2022-10-27 21:53:43] ...trap/engines/engine.py.224: train: epoch 90 | batch 430/2304 [I 2022-10-27 21:53:43] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:19 [I 2022-10-27 21:53:43] ...trap/engines/engine.py.229: process: 0.56738 | load: 0.00249 [I 2022-10-27 21:53:43] ...trap/engines/engine.py.230: loss: 0.12568 [I 2022-10-27 21:53:43] ...optimizers/trijoint.py.151: total_norm: 15.790536 [I 2022-10-27 21:53:49] ...trap/engines/engine.py.224: train: epoch 90 | batch 440/2304 [I 2022-10-27 21:53:49] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:17 [I 2022-10-27 21:53:49] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00251 [I 2022-10-27 21:53:49] ...trap/engines/engine.py.230: loss: 0.16570 [I 2022-10-27 21:53:49] ...optimizers/trijoint.py.151: total_norm: 17.753992 [I 2022-10-27 21:53:55] ...trap/engines/engine.py.224: train: epoch 90 | batch 450/2304 [I 2022-10-27 21:53:55] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:13 [I 2022-10-27 21:53:55] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00246 [I 2022-10-27 21:53:55] ...trap/engines/engine.py.230: loss: 0.16320 [I 2022-10-27 21:53:55] ...optimizers/trijoint.py.151: total_norm: 16.340633 [I 2022-10-27 21:54:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 460/2304 [I 2022-10-27 21:54:00] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:11 [I 2022-10-27 21:54:00] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00243 [I 2022-10-27 21:54:00] ...trap/engines/engine.py.230: loss: 0.19678 [I 2022-10-27 21:54:00] ...optimizers/trijoint.py.151: total_norm: 25.418600 [I 2022-10-27 21:54:06] ...trap/engines/engine.py.224: train: epoch 90 | batch 470/2304 [I 2022-10-27 21:54:06] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:04 [I 2022-10-27 21:54:06] ...trap/engines/engine.py.229: process: 0.55180 | load: 0.00243 [I 2022-10-27 21:54:06] ...trap/engines/engine.py.230: loss: 0.11981 [I 2022-10-27 21:54:06] ...optimizers/trijoint.py.151: total_norm: 26.610489 [I 2022-10-27 21:54:11] ...trap/engines/engine.py.224: train: epoch 90 | batch 480/2304 [I 2022-10-27 21:54:11] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:56 [I 2022-10-27 21:54:11] ...trap/engines/engine.py.229: process: 0.55793 | load: 0.00243 [I 2022-10-27 21:54:11] ...trap/engines/engine.py.230: loss: 0.11880 [I 2022-10-27 21:54:11] ...optimizers/trijoint.py.151: total_norm: 46.092018 [I 2022-10-27 21:54:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 490/2304 [I 2022-10-27 21:54:17] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:46 [I 2022-10-27 21:54:17] ...trap/engines/engine.py.229: process: 0.55268 | load: 0.00238 [I 2022-10-27 21:54:17] ...trap/engines/engine.py.230: loss: 0.09944 [I 2022-10-27 21:54:17] ...optimizers/trijoint.py.151: total_norm: 19.331324 [I 2022-10-27 21:54:23] ...trap/engines/engine.py.224: train: epoch 90 | batch 500/2304 [I 2022-10-27 21:54:23] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:45 [I 2022-10-27 21:54:23] ...trap/engines/engine.py.229: process: 0.55501 | load: 0.00250 [I 2022-10-27 21:54:23] ...trap/engines/engine.py.230: loss: 0.20225 [I 2022-10-27 21:54:23] ...optimizers/trijoint.py.151: total_norm: 28.779749 [I 2022-10-27 21:54:28] ...trap/engines/engine.py.224: train: epoch 90 | batch 510/2304 [I 2022-10-27 21:54:28] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:33 [I 2022-10-27 21:54:28] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00249 [I 2022-10-27 21:54:28] ...trap/engines/engine.py.230: loss: 0.16041 [I 2022-10-27 21:54:28] ...optimizers/trijoint.py.151: total_norm: 9.254265 [I 2022-10-27 21:54:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 520/2304 [I 2022-10-27 21:54:34] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:35 [I 2022-10-27 21:54:34] ...trap/engines/engine.py.229: process: 0.57133 | load: 0.00241 [I 2022-10-27 21:54:34] ...trap/engines/engine.py.230: loss: 0.14284 [I 2022-10-27 21:54:34] ...optimizers/trijoint.py.151: total_norm: 25.639105 [I 2022-10-27 21:54:39] ...trap/engines/engine.py.224: train: epoch 90 | batch 530/2304 [I 2022-10-27 21:54:39] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:32 [I 2022-10-27 21:54:39] ...trap/engines/engine.py.229: process: 0.56404 | load: 0.00244 [I 2022-10-27 21:54:39] ...trap/engines/engine.py.230: loss: 0.12716 [I 2022-10-27 21:54:39] ...optimizers/trijoint.py.151: total_norm: 33.964249 [I 2022-10-27 21:54:45] ...trap/engines/engine.py.224: train: epoch 90 | batch 540/2304 [I 2022-10-27 21:54:45] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:27 [I 2022-10-27 21:54:45] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00241 [I 2022-10-27 21:54:45] ...trap/engines/engine.py.230: loss: 0.09032 [I 2022-10-27 21:54:45] ...optimizers/trijoint.py.151: total_norm: 12.785870 [I 2022-10-27 21:54:50] ...trap/engines/engine.py.224: train: epoch 90 | batch 550/2304 [I 2022-10-27 21:54:50] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:24 [I 2022-10-27 21:54:50] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00243 [I 2022-10-27 21:54:50] ...trap/engines/engine.py.230: loss: 0.14396 [I 2022-10-27 21:54:50] ...optimizers/trijoint.py.151: total_norm: 13.326504 [I 2022-10-27 21:54:56] ...trap/engines/engine.py.224: train: epoch 90 | batch 560/2304 [I 2022-10-27 21:54:56] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:09 [I 2022-10-27 21:54:56] ...trap/engines/engine.py.229: process: 0.55036 | load: 0.00242 [I 2022-10-27 21:54:56] ...trap/engines/engine.py.230: loss: 0.09729 [I 2022-10-27 21:54:56] ...optimizers/trijoint.py.151: total_norm: 7.573904 [I 2022-10-27 21:55:02] ...trap/engines/engine.py.224: train: epoch 90 | batch 570/2304 [I 2022-10-27 21:55:02] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:03 [I 2022-10-27 21:55:02] ...trap/engines/engine.py.229: process: 0.54779 | load: 0.00263 [I 2022-10-27 21:55:02] ...trap/engines/engine.py.230: loss: 0.20641 [I 2022-10-27 21:55:02] ...optimizers/trijoint.py.151: total_norm: 14.576934 [I 2022-10-27 21:55:07] ...trap/engines/engine.py.224: train: epoch 90 | batch 580/2304 [I 2022-10-27 21:55:07] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:02 [I 2022-10-27 21:55:07] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00244 [I 2022-10-27 21:55:07] ...trap/engines/engine.py.230: loss: 0.12310 [I 2022-10-27 21:55:07] ...optimizers/trijoint.py.151: total_norm: 13.441048 [I 2022-10-27 21:55:33] ...trap/engines/engine.py.224: train: epoch 90 | batch 590/2304 [I 2022-10-27 21:55:33] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 1:02:26 [I 2022-10-27 21:55:33] ...trap/engines/engine.py.229: process: 0.55015 | load: 0.00346 [I 2022-10-27 21:55:33] ...trap/engines/engine.py.230: loss: 0.13837 [I 2022-10-27 21:55:33] ...optimizers/trijoint.py.151: total_norm: 32.065662 [I 2022-10-27 21:55:38] ...trap/engines/engine.py.224: train: epoch 90 | batch 600/2304 [I 2022-10-27 21:55:38] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:20:48 [I 2022-10-27 21:55:38] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00244 [I 2022-10-27 21:55:38] ...trap/engines/engine.py.230: loss: 0.09551 [I 2022-10-27 21:55:38] ...optimizers/trijoint.py.151: total_norm: 9.215110 [I 2022-10-27 21:55:44] ...trap/engines/engine.py.224: train: epoch 90 | batch 610/2304 [I 2022-10-27 21:55:44] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:16:18 [I 2022-10-27 21:55:44] ...trap/engines/engine.py.229: process: 0.55246 | load: 0.00243 [I 2022-10-27 21:55:44] ...trap/engines/engine.py.230: loss: 0.10548 [I 2022-10-27 21:55:44] ...optimizers/trijoint.py.151: total_norm: 21.168676 [I 2022-10-27 21:55:50] ...trap/engines/engine.py.224: train: epoch 90 | batch 620/2304 [I 2022-10-27 21:55:50] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:44 [I 2022-10-27 21:55:50] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00245 [I 2022-10-27 21:55:50] ...trap/engines/engine.py.230: loss: 0.11402 [I 2022-10-27 21:55:50] ...optimizers/trijoint.py.151: total_norm: 46.997787 [I 2022-10-27 21:55:55] ...trap/engines/engine.py.224: train: epoch 90 | batch 630/2304 [I 2022-10-27 21:55:55] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:33 [I 2022-10-27 21:55:55] ...trap/engines/engine.py.229: process: 0.54973 | load: 0.00242 [I 2022-10-27 21:55:55] ...trap/engines/engine.py.230: loss: 0.10443 [I 2022-10-27 21:55:55] ...optimizers/trijoint.py.151: total_norm: 14.265523 [I 2022-10-27 21:56:01] ...trap/engines/engine.py.224: train: epoch 90 | batch 640/2304 [I 2022-10-27 21:56:01] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:24 [I 2022-10-27 21:56:01] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00242 [I 2022-10-27 21:56:01] ...trap/engines/engine.py.230: loss: 0.11914 [I 2022-10-27 21:56:01] ...optimizers/trijoint.py.151: total_norm: 21.662455 [I 2022-10-27 21:56:06] ...trap/engines/engine.py.224: train: epoch 90 | batch 650/2304 [I 2022-10-27 21:56:06] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:30 [I 2022-10-27 21:56:06] ...trap/engines/engine.py.229: process: 0.57137 | load: 0.00252 [I 2022-10-27 21:56:06] ...trap/engines/engine.py.230: loss: 0.19232 [I 2022-10-27 21:56:06] ...optimizers/trijoint.py.151: total_norm: 12.831141 [I 2022-10-27 21:56:12] ...trap/engines/engine.py.224: train: epoch 90 | batch 660/2304 [I 2022-10-27 21:56:12] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:15:18 [I 2022-10-27 21:56:12] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00249 [I 2022-10-27 21:56:12] ...trap/engines/engine.py.230: loss: 0.13814 [I 2022-10-27 21:56:12] ...optimizers/trijoint.py.151: total_norm: 27.619043 [I 2022-10-27 21:56:18] ...trap/engines/engine.py.224: train: epoch 90 | batch 670/2304 [I 2022-10-27 21:56:18] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:07 [I 2022-10-27 21:56:18] ...trap/engines/engine.py.229: process: 0.54699 | load: 0.00246 [I 2022-10-27 21:56:18] ...trap/engines/engine.py.230: loss: 0.20415 [I 2022-10-27 21:56:18] ...optimizers/trijoint.py.151: total_norm: 41.860733 [I 2022-10-27 21:56:23] ...trap/engines/engine.py.224: train: epoch 90 | batch 680/2304 [I 2022-10-27 21:56:23] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:14 [I 2022-10-27 21:56:23] ...trap/engines/engine.py.229: process: 0.57708 | load: 0.00245 [I 2022-10-27 21:56:23] ...trap/engines/engine.py.230: loss: 0.20174 [I 2022-10-27 21:56:23] ...optimizers/trijoint.py.151: total_norm: 23.338669 [I 2022-10-27 21:56:29] ...trap/engines/engine.py.224: train: epoch 90 | batch 690/2304 [I 2022-10-27 21:56:29] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:15:08 [I 2022-10-27 21:56:29] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00248 [I 2022-10-27 21:56:29] ...trap/engines/engine.py.230: loss: 0.12051 [I 2022-10-27 21:56:29] ...optimizers/trijoint.py.151: total_norm: 18.337082 [I 2022-10-27 21:56:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 700/2304 [I 2022-10-27 21:56:34] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:15:03 [I 2022-10-27 21:56:34] ...trap/engines/engine.py.229: process: 0.55101 | load: 0.00248 [I 2022-10-27 21:56:34] ...trap/engines/engine.py.230: loss: 0.09351 [I 2022-10-27 21:56:34] ...optimizers/trijoint.py.151: total_norm: 12.589832 [I 2022-10-27 21:56:40] ...trap/engines/engine.py.224: train: epoch 90 | batch 710/2304 [I 2022-10-27 21:56:40] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:14:45 [I 2022-10-27 21:56:40] ...trap/engines/engine.py.229: process: 0.54629 | load: 0.00241 [I 2022-10-27 21:56:40] ...trap/engines/engine.py.230: loss: 0.09004 [I 2022-10-27 21:56:40] ...optimizers/trijoint.py.151: total_norm: 22.605616 [I 2022-10-27 21:56:46] ...trap/engines/engine.py.224: train: epoch 90 | batch 720/2304 [I 2022-10-27 21:56:46] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:45 [I 2022-10-27 21:56:46] ...trap/engines/engine.py.229: process: 0.54491 | load: 0.00243 [I 2022-10-27 21:56:46] ...trap/engines/engine.py.230: loss: 0.12240 [I 2022-10-27 21:56:46] ...optimizers/trijoint.py.151: total_norm: 29.104916 [I 2022-10-27 21:56:51] ...trap/engines/engine.py.224: train: epoch 90 | batch 730/2304 [I 2022-10-27 21:56:51] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:36 [I 2022-10-27 21:56:51] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00242 [I 2022-10-27 21:56:51] ...trap/engines/engine.py.230: loss: 0.14289 [I 2022-10-27 21:56:51] ...optimizers/trijoint.py.151: total_norm: 18.640566 [I 2022-10-27 21:56:57] ...trap/engines/engine.py.224: train: epoch 90 | batch 740/2304 [I 2022-10-27 21:56:57] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:33 [I 2022-10-27 21:56:57] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00247 [I 2022-10-27 21:56:57] ...trap/engines/engine.py.230: loss: 0.14500 [I 2022-10-27 21:56:57] ...optimizers/trijoint.py.151: total_norm: 10.996253 [I 2022-10-27 21:57:03] ...trap/engines/engine.py.224: train: epoch 90 | batch 750/2304 [I 2022-10-27 21:57:03] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:38 [I 2022-10-27 21:57:03] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00244 [I 2022-10-27 21:57:03] ...trap/engines/engine.py.230: loss: 0.12646 [I 2022-10-27 21:57:03] ...optimizers/trijoint.py.151: total_norm: 13.314567 [I 2022-10-27 21:57:08] ...trap/engines/engine.py.224: train: epoch 90 | batch 760/2304 [I 2022-10-27 21:57:08] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:36 [I 2022-10-27 21:57:08] ...trap/engines/engine.py.229: process: 0.56515 | load: 0.00246 [I 2022-10-27 21:57:08] ...trap/engines/engine.py.230: loss: 0.09617 [I 2022-10-27 21:57:08] ...optimizers/trijoint.py.151: total_norm: 19.641111 [I 2022-10-27 21:57:14] ...trap/engines/engine.py.224: train: epoch 90 | batch 770/2304 [I 2022-10-27 21:57:14] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:14:29 [I 2022-10-27 21:57:14] ...trap/engines/engine.py.229: process: 0.56596 | load: 0.00251 [I 2022-10-27 21:57:14] ...trap/engines/engine.py.230: loss: 0.12755 [I 2022-10-27 21:57:14] ...optimizers/trijoint.py.151: total_norm: 12.733152 [I 2022-10-27 21:57:19] ...trap/engines/engine.py.224: train: epoch 90 | batch 780/2304 [I 2022-10-27 21:57:19] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:20 [I 2022-10-27 21:57:19] ...trap/engines/engine.py.229: process: 0.57224 | load: 0.00248 [I 2022-10-27 21:57:19] ...trap/engines/engine.py.230: loss: 0.11346 [I 2022-10-27 21:57:19] ...optimizers/trijoint.py.151: total_norm: 50.012730 [I 2022-10-27 21:57:25] ...trap/engines/engine.py.224: train: epoch 90 | batch 790/2304 [I 2022-10-27 21:57:25] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:14:22 [I 2022-10-27 21:57:25] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00244 [I 2022-10-27 21:57:25] ...trap/engines/engine.py.230: loss: 0.18346 [I 2022-10-27 21:57:25] ...optimizers/trijoint.py.151: total_norm: 13.971319 [I 2022-10-27 21:57:31] ...trap/engines/engine.py.224: train: epoch 90 | batch 800/2304 [I 2022-10-27 21:57:31] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:14:04 [I 2022-10-27 21:57:31] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00244 [I 2022-10-27 21:57:31] ...trap/engines/engine.py.230: loss: 0.23448 [I 2022-10-27 21:57:31] ...optimizers/trijoint.py.151: total_norm: 17.432056 [I 2022-10-27 21:57:37] ...trap/engines/engine.py.224: train: epoch 90 | batch 810/2304 [I 2022-10-27 21:57:37] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:14:00 [I 2022-10-27 21:57:37] ...trap/engines/engine.py.229: process: 0.56637 | load: 0.00260 [I 2022-10-27 21:57:37] ...trap/engines/engine.py.230: loss: 0.15373 [I 2022-10-27 21:57:37] ...optimizers/trijoint.py.151: total_norm: 22.171911 [I 2022-10-27 21:57:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 820/2304 [I 2022-10-27 21:57:42] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:51 [I 2022-10-27 21:57:42] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00244 [I 2022-10-27 21:57:42] ...trap/engines/engine.py.230: loss: 0.10126 [I 2022-10-27 21:57:42] ...optimizers/trijoint.py.151: total_norm: 15.154930 [I 2022-10-27 21:57:48] ...trap/engines/engine.py.224: train: epoch 90 | batch 830/2304 [I 2022-10-27 21:57:48] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:39 [I 2022-10-27 21:57:48] ...trap/engines/engine.py.229: process: 0.54276 | load: 0.00244 [I 2022-10-27 21:57:48] ...trap/engines/engine.py.230: loss: 0.13327 [I 2022-10-27 21:57:48] ...optimizers/trijoint.py.151: total_norm: 14.803611 [I 2022-10-27 21:57:53] ...trap/engines/engine.py.224: train: epoch 90 | batch 840/2304 [I 2022-10-27 21:57:53] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:34 [I 2022-10-27 21:57:53] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00243 [I 2022-10-27 21:57:53] ...trap/engines/engine.py.230: loss: 0.13679 [I 2022-10-27 21:57:53] ...optimizers/trijoint.py.151: total_norm: 48.327374 [I 2022-10-27 21:57:59] ...trap/engines/engine.py.224: train: epoch 90 | batch 850/2304 [I 2022-10-27 21:57:59] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:37 [I 2022-10-27 21:57:59] ...trap/engines/engine.py.229: process: 0.55067 | load: 0.00245 [I 2022-10-27 21:57:59] ...trap/engines/engine.py.230: loss: 0.15549 [I 2022-10-27 21:57:59] ...optimizers/trijoint.py.151: total_norm: 23.237635 [I 2022-10-27 21:58:05] ...trap/engines/engine.py.224: train: epoch 90 | batch 860/2304 [I 2022-10-27 21:58:05] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:22 [I 2022-10-27 21:58:05] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00241 [I 2022-10-27 21:58:05] ...trap/engines/engine.py.230: loss: 0.13965 [I 2022-10-27 21:58:05] ...optimizers/trijoint.py.151: total_norm: 27.934824 [I 2022-10-27 21:58:10] ...trap/engines/engine.py.224: train: epoch 90 | batch 870/2304 [I 2022-10-27 21:58:10] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:13:20 [I 2022-10-27 21:58:10] ...trap/engines/engine.py.229: process: 0.55694 | load: 0.00243 [I 2022-10-27 21:58:10] ...trap/engines/engine.py.230: loss: 0.11162 [I 2022-10-27 21:58:10] ...optimizers/trijoint.py.151: total_norm: 37.114567 [I 2022-10-27 21:58:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 880/2304 [I 2022-10-27 21:58:16] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:18 [I 2022-10-27 21:58:16] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00242 [I 2022-10-27 21:58:16] ...trap/engines/engine.py.230: loss: 0.12226 [I 2022-10-27 21:58:16] ...optimizers/trijoint.py.151: total_norm: 10.690099 [I 2022-10-27 21:58:21] ...trap/engines/engine.py.224: train: epoch 90 | batch 890/2304 [I 2022-10-27 21:58:21] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:13:07 [I 2022-10-27 21:58:21] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00242 [I 2022-10-27 21:58:21] ...trap/engines/engine.py.230: loss: 0.21205 [I 2022-10-27 21:58:21] ...optimizers/trijoint.py.151: total_norm: 14.237179 [I 2022-10-27 21:58:27] ...trap/engines/engine.py.224: train: epoch 90 | batch 900/2304 [I 2022-10-27 21:58:27] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:13:04 [I 2022-10-27 21:58:27] ...trap/engines/engine.py.229: process: 0.54734 | load: 0.00247 [I 2022-10-27 21:58:27] ...trap/engines/engine.py.230: loss: 0.17251 [I 2022-10-27 21:58:27] ...optimizers/trijoint.py.151: total_norm: 20.250000 [I 2022-10-27 21:58:32] ...trap/engines/engine.py.224: train: epoch 90 | batch 910/2304 [I 2022-10-27 21:58:32] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:56 [I 2022-10-27 21:58:32] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00243 [I 2022-10-27 21:58:32] ...trap/engines/engine.py.230: loss: 0.11295 [I 2022-10-27 21:58:32] ...optimizers/trijoint.py.151: total_norm: 16.764587 [I 2022-10-27 21:58:38] ...trap/engines/engine.py.224: train: epoch 90 | batch 920/2304 [I 2022-10-27 21:58:38] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:52 [I 2022-10-27 21:58:38] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00240 [I 2022-10-27 21:58:38] ...trap/engines/engine.py.230: loss: 0.24968 [I 2022-10-27 21:58:38] ...optimizers/trijoint.py.151: total_norm: 24.253386 [I 2022-10-27 21:58:44] ...trap/engines/engine.py.224: train: epoch 90 | batch 930/2304 [I 2022-10-27 21:58:44] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:47 [I 2022-10-27 21:58:44] ...trap/engines/engine.py.229: process: 0.56556 | load: 0.00244 [I 2022-10-27 21:58:44] ...trap/engines/engine.py.230: loss: 0.17544 [I 2022-10-27 21:58:44] ...optimizers/trijoint.py.151: total_norm: 32.085297 [I 2022-10-27 21:58:49] ...trap/engines/engine.py.224: train: epoch 90 | batch 940/2304 [I 2022-10-27 21:58:49] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:42 [I 2022-10-27 21:58:49] ...trap/engines/engine.py.229: process: 0.56470 | load: 0.00251 [I 2022-10-27 21:58:49] ...trap/engines/engine.py.230: loss: 0.15234 [I 2022-10-27 21:58:49] ...optimizers/trijoint.py.151: total_norm: 16.353870 [I 2022-10-27 21:58:55] ...trap/engines/engine.py.224: train: epoch 90 | batch 950/2304 [I 2022-10-27 21:58:55] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:37 [I 2022-10-27 21:58:55] ...trap/engines/engine.py.229: process: 0.56696 | load: 0.00246 [I 2022-10-27 21:58:55] ...trap/engines/engine.py.230: loss: 0.11498 [I 2022-10-27 21:58:55] ...optimizers/trijoint.py.151: total_norm: 18.639164 [I 2022-10-27 21:59:01] ...trap/engines/engine.py.224: train: epoch 90 | batch 960/2304 [I 2022-10-27 21:59:01] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:33 [I 2022-10-27 21:59:01] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00243 [I 2022-10-27 21:59:01] ...trap/engines/engine.py.230: loss: 0.09493 [I 2022-10-27 21:59:01] ...optimizers/trijoint.py.151: total_norm: 10.521927 [I 2022-10-27 21:59:06] ...trap/engines/engine.py.224: train: epoch 90 | batch 970/2304 [I 2022-10-27 21:59:06] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:27 [I 2022-10-27 21:59:06] ...trap/engines/engine.py.229: process: 0.55656 | load: 0.00249 [I 2022-10-27 21:59:06] ...trap/engines/engine.py.230: loss: 0.14906 [I 2022-10-27 21:59:06] ...optimizers/trijoint.py.151: total_norm: 11.322510 [I 2022-10-27 21:59:12] ...trap/engines/engine.py.224: train: epoch 90 | batch 980/2304 [I 2022-10-27 21:59:12] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:27 [I 2022-10-27 21:59:12] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00245 [I 2022-10-27 21:59:12] ...trap/engines/engine.py.230: loss: 0.09646 [I 2022-10-27 21:59:12] ...optimizers/trijoint.py.151: total_norm: 10.495074 [I 2022-10-27 21:59:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 990/2304 [I 2022-10-27 21:59:17] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:12:14 [I 2022-10-27 21:59:17] ...trap/engines/engine.py.229: process: 0.55096 | load: 0.00243 [I 2022-10-27 21:59:17] ...trap/engines/engine.py.230: loss: 0.14941 [I 2022-10-27 21:59:17] ...optimizers/trijoint.py.151: total_norm: 25.102699 [I 2022-10-27 21:59:23] ...trap/engines/engine.py.224: train: epoch 90 | batch 1000/2304 [I 2022-10-27 21:59:23] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:06 [I 2022-10-27 21:59:23] ...trap/engines/engine.py.229: process: 0.56070 | load: 0.00251 [I 2022-10-27 21:59:23] ...trap/engines/engine.py.230: loss: 0.09582 [I 2022-10-27 21:59:23] ...optimizers/trijoint.py.151: total_norm: 12.152987 [I 2022-10-27 21:59:29] ...trap/engines/engine.py.224: train: epoch 90 | batch 1010/2304 [I 2022-10-27 21:59:29] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:12:04 [I 2022-10-27 21:59:29] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00256 [I 2022-10-27 21:59:29] ...trap/engines/engine.py.230: loss: 0.11181 [I 2022-10-27 21:59:29] ...optimizers/trijoint.py.151: total_norm: 9.551275 [I 2022-10-27 21:59:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 1020/2304 [I 2022-10-27 21:59:34] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:59 [I 2022-10-27 21:59:34] ...trap/engines/engine.py.229: process: 0.55337 | load: 0.00256 [I 2022-10-27 21:59:34] ...trap/engines/engine.py.230: loss: 0.16404 [I 2022-10-27 21:59:34] ...optimizers/trijoint.py.151: total_norm: 25.838406 [I 2022-10-27 21:59:40] ...trap/engines/engine.py.224: train: epoch 90 | batch 1030/2304 [I 2022-10-27 21:59:40] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:48 [I 2022-10-27 21:59:40] ...trap/engines/engine.py.229: process: 0.54581 | load: 0.00251 [I 2022-10-27 21:59:40] ...trap/engines/engine.py.230: loss: 0.11234 [I 2022-10-27 21:59:40] ...optimizers/trijoint.py.151: total_norm: 16.633684 [I 2022-10-27 21:59:46] ...trap/engines/engine.py.224: train: epoch 90 | batch 1040/2304 [I 2022-10-27 21:59:46] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:49 [I 2022-10-27 21:59:46] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00249 [I 2022-10-27 21:59:46] ...trap/engines/engine.py.230: loss: 0.12936 [I 2022-10-27 21:59:46] ...optimizers/trijoint.py.151: total_norm: 15.326594 [I 2022-10-27 21:59:51] ...trap/engines/engine.py.224: train: epoch 90 | batch 1050/2304 [I 2022-10-27 21:59:51] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:42 [I 2022-10-27 21:59:51] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00276 [I 2022-10-27 21:59:51] ...trap/engines/engine.py.230: loss: 0.13718 [I 2022-10-27 21:59:51] ...optimizers/trijoint.py.151: total_norm: 16.364235 [I 2022-10-27 21:59:57] ...trap/engines/engine.py.224: train: epoch 90 | batch 1060/2304 [I 2022-10-27 21:59:57] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:40 [I 2022-10-27 21:59:57] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00263 [I 2022-10-27 21:59:57] ...trap/engines/engine.py.230: loss: 0.11798 [I 2022-10-27 21:59:57] ...optimizers/trijoint.py.151: total_norm: 13.500156 [I 2022-10-27 22:00:02] ...trap/engines/engine.py.224: train: epoch 90 | batch 1070/2304 [I 2022-10-27 22:00:02] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:34 [I 2022-10-27 22:00:02] ...trap/engines/engine.py.229: process: 0.56094 | load: 0.00250 [I 2022-10-27 22:00:02] ...trap/engines/engine.py.230: loss: 0.10057 [I 2022-10-27 22:00:02] ...optimizers/trijoint.py.151: total_norm: 17.798370 [I 2022-10-27 22:00:08] ...trap/engines/engine.py.224: train: epoch 90 | batch 1080/2304 [I 2022-10-27 22:00:08] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:31 [I 2022-10-27 22:00:08] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00246 [I 2022-10-27 22:00:08] ...trap/engines/engine.py.230: loss: 0.19018 [I 2022-10-27 22:00:08] ...optimizers/trijoint.py.151: total_norm: 13.051688 [I 2022-10-27 22:00:14] ...trap/engines/engine.py.224: train: epoch 90 | batch 1090/2304 [I 2022-10-27 22:00:14] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:11:21 [I 2022-10-27 22:00:14] ...trap/engines/engine.py.229: process: 0.55848 | load: 0.00254 [I 2022-10-27 22:00:14] ...trap/engines/engine.py.230: loss: 0.10943 [I 2022-10-27 22:00:14] ...optimizers/trijoint.py.151: total_norm: 9.882559 [I 2022-10-27 22:00:19] ...trap/engines/engine.py.224: train: epoch 90 | batch 1100/2304 [I 2022-10-27 22:00:19] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:11:14 [I 2022-10-27 22:00:19] ...trap/engines/engine.py.229: process: 0.56694 | load: 0.00251 [I 2022-10-27 22:00:19] ...trap/engines/engine.py.230: loss: 0.13741 [I 2022-10-27 22:00:19] ...optimizers/trijoint.py.151: total_norm: 9.312563 [I 2022-10-27 22:00:25] ...trap/engines/engine.py.224: train: epoch 90 | batch 1110/2304 [I 2022-10-27 22:00:25] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:11:13 [I 2022-10-27 22:00:25] ...trap/engines/engine.py.229: process: 0.54526 | load: 0.00250 [I 2022-10-27 22:00:25] ...trap/engines/engine.py.230: loss: 0.09288 [I 2022-10-27 22:00:25] ...optimizers/trijoint.py.151: total_norm: 29.444677 [I 2022-10-27 22:00:31] ...trap/engines/engine.py.224: train: epoch 90 | batch 1120/2304 [I 2022-10-27 22:00:31] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:11:02 [I 2022-10-27 22:00:31] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00249 [I 2022-10-27 22:00:31] ...trap/engines/engine.py.230: loss: 0.19922 [I 2022-10-27 22:00:31] ...optimizers/trijoint.py.151: total_norm: 15.557822 [I 2022-10-27 22:00:36] ...trap/engines/engine.py.224: train: epoch 90 | batch 1130/2304 [I 2022-10-27 22:00:36] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:56 [I 2022-10-27 22:00:36] ...trap/engines/engine.py.229: process: 0.56512 | load: 0.00298 [I 2022-10-27 22:00:36] ...trap/engines/engine.py.230: loss: 0.12808 [I 2022-10-27 22:00:36] ...optimizers/trijoint.py.151: total_norm: 12.844897 [I 2022-10-27 22:00:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 1140/2304 [I 2022-10-27 22:00:42] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:50 [I 2022-10-27 22:00:42] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00248 [I 2022-10-27 22:00:42] ...trap/engines/engine.py.230: loss: 0.12877 [I 2022-10-27 22:00:42] ...optimizers/trijoint.py.151: total_norm: 10.241336 [I 2022-10-27 22:00:47] ...trap/engines/engine.py.224: train: epoch 90 | batch 1150/2304 [I 2022-10-27 22:00:47] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:44 [I 2022-10-27 22:00:47] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00246 [I 2022-10-27 22:00:47] ...trap/engines/engine.py.230: loss: 0.11737 [I 2022-10-27 22:00:47] ...optimizers/trijoint.py.151: total_norm: 13.784154 [I 2022-10-27 22:00:53] ...trap/engines/engine.py.224: train: epoch 90 | batch 1160/2304 [I 2022-10-27 22:00:53] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:36 [I 2022-10-27 22:00:53] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00247 [I 2022-10-27 22:00:53] ...trap/engines/engine.py.230: loss: 0.11680 [I 2022-10-27 22:00:53] ...optimizers/trijoint.py.151: total_norm: 13.702147 [I 2022-10-27 22:00:59] ...trap/engines/engine.py.224: train: epoch 90 | batch 1170/2304 [I 2022-10-27 22:00:59] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:37 [I 2022-10-27 22:00:59] ...trap/engines/engine.py.229: process: 0.56451 | load: 0.00241 [I 2022-10-27 22:00:59] ...trap/engines/engine.py.230: loss: 0.10408 [I 2022-10-27 22:00:59] ...optimizers/trijoint.py.151: total_norm: 18.840586 [I 2022-10-27 22:01:04] ...trap/engines/engine.py.224: train: epoch 90 | batch 1180/2304 [I 2022-10-27 22:01:04] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:23 [I 2022-10-27 22:01:04] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00249 [I 2022-10-27 22:01:04] ...trap/engines/engine.py.230: loss: 0.13437 [I 2022-10-27 22:01:04] ...optimizers/trijoint.py.151: total_norm: 37.229111 [I 2022-10-27 22:01:10] ...trap/engines/engine.py.224: train: epoch 90 | batch 1190/2304 [I 2022-10-27 22:01:10] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:10:20 [I 2022-10-27 22:01:10] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00249 [I 2022-10-27 22:01:10] ...trap/engines/engine.py.230: loss: 0.11278 [I 2022-10-27 22:01:10] ...optimizers/trijoint.py.151: total_norm: 11.925615 [I 2022-10-27 22:01:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 1200/2304 [I 2022-10-27 22:01:16] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:10:22 [I 2022-10-27 22:01:16] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00240 [I 2022-10-27 22:01:16] ...trap/engines/engine.py.230: loss: 0.19055 [I 2022-10-27 22:01:16] ...optimizers/trijoint.py.151: total_norm: 23.383057 [I 2022-10-27 22:01:21] ...trap/engines/engine.py.224: train: epoch 90 | batch 1210/2304 [I 2022-10-27 22:01:21] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:10:07 [I 2022-10-27 22:01:21] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00242 [I 2022-10-27 22:01:21] ...trap/engines/engine.py.230: loss: 0.09908 [I 2022-10-27 22:01:21] ...optimizers/trijoint.py.151: total_norm: 11.905851 [I 2022-10-27 22:01:27] ...trap/engines/engine.py.224: train: epoch 90 | batch 1220/2304 [I 2022-10-27 22:01:27] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:10:06 [I 2022-10-27 22:01:27] ...trap/engines/engine.py.229: process: 0.55425 | load: 0.00245 [I 2022-10-27 22:01:27] ...trap/engines/engine.py.230: loss: 0.09565 [I 2022-10-27 22:01:27] ...optimizers/trijoint.py.151: total_norm: 8.230883 [I 2022-10-27 22:01:32] ...trap/engines/engine.py.224: train: epoch 90 | batch 1230/2304 [I 2022-10-27 22:01:32] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:10:11 [I 2022-10-27 22:01:32] ...trap/engines/engine.py.229: process: 0.54867 | load: 0.00242 [I 2022-10-27 22:01:32] ...trap/engines/engine.py.230: loss: 0.16193 [I 2022-10-27 22:01:32] ...optimizers/trijoint.py.151: total_norm: 60.571018 [I 2022-10-27 22:01:38] ...trap/engines/engine.py.224: train: epoch 90 | batch 1240/2304 [I 2022-10-27 22:01:38] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:55 [I 2022-10-27 22:01:38] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00251 [I 2022-10-27 22:01:38] ...trap/engines/engine.py.230: loss: 0.18015 [I 2022-10-27 22:01:38] ...optimizers/trijoint.py.151: total_norm: 20.910208 [I 2022-10-27 22:01:44] ...trap/engines/engine.py.224: train: epoch 90 | batch 1250/2304 [I 2022-10-27 22:01:44] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:48 [I 2022-10-27 22:01:44] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00241 [I 2022-10-27 22:01:44] ...trap/engines/engine.py.230: loss: 0.09054 [I 2022-10-27 22:01:44] ...optimizers/trijoint.py.151: total_norm: 9.879168 [I 2022-10-27 22:01:49] ...trap/engines/engine.py.224: train: epoch 90 | batch 1260/2304 [I 2022-10-27 22:01:49] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:40 [I 2022-10-27 22:01:49] ...trap/engines/engine.py.229: process: 0.54839 | load: 0.00240 [I 2022-10-27 22:01:49] ...trap/engines/engine.py.230: loss: 0.15198 [I 2022-10-27 22:01:49] ...optimizers/trijoint.py.151: total_norm: 25.750208 [I 2022-10-27 22:01:55] ...trap/engines/engine.py.224: train: epoch 90 | batch 1270/2304 [I 2022-10-27 22:01:55] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:36 [I 2022-10-27 22:01:55] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00240 [I 2022-10-27 22:01:55] ...trap/engines/engine.py.230: loss: 0.12641 [I 2022-10-27 22:01:55] ...optimizers/trijoint.py.151: total_norm: 23.314272 [I 2022-10-27 22:02:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 1280/2304 [I 2022-10-27 22:02:00] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:32 [I 2022-10-27 22:02:00] ...trap/engines/engine.py.229: process: 0.55035 | load: 0.00240 [I 2022-10-27 22:02:00] ...trap/engines/engine.py.230: loss: 0.12232 [I 2022-10-27 22:02:00] ...optimizers/trijoint.py.151: total_norm: 19.827587 [I 2022-10-27 22:02:06] ...trap/engines/engine.py.224: train: epoch 90 | batch 1290/2304 [I 2022-10-27 22:02:06] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:24 [I 2022-10-27 22:02:06] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00241 [I 2022-10-27 22:02:06] ...trap/engines/engine.py.230: loss: 0.10109 [I 2022-10-27 22:02:06] ...optimizers/trijoint.py.151: total_norm: 10.382919 [I 2022-10-27 22:02:12] ...trap/engines/engine.py.224: train: epoch 90 | batch 1300/2304 [I 2022-10-27 22:02:12] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:09:21 [I 2022-10-27 22:02:12] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00248 [I 2022-10-27 22:02:12] ...trap/engines/engine.py.230: loss: 0.14790 [I 2022-10-27 22:02:12] ...optimizers/trijoint.py.151: total_norm: 21.749685 [I 2022-10-27 22:02:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 1310/2304 [I 2022-10-27 22:02:17] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:09:15 [I 2022-10-27 22:02:17] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00239 [I 2022-10-27 22:02:17] ...trap/engines/engine.py.230: loss: 0.13288 [I 2022-10-27 22:02:17] ...optimizers/trijoint.py.151: total_norm: 18.274212 [I 2022-10-27 22:02:23] ...trap/engines/engine.py.224: train: epoch 90 | batch 1320/2304 [I 2022-10-27 22:02:23] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:09:06 [I 2022-10-27 22:02:23] ...trap/engines/engine.py.229: process: 0.55075 | load: 0.00239 [I 2022-10-27 22:02:23] ...trap/engines/engine.py.230: loss: 0.10199 [I 2022-10-27 22:02:23] ...optimizers/trijoint.py.151: total_norm: 22.140057 [I 2022-10-27 22:02:28] ...trap/engines/engine.py.224: train: epoch 90 | batch 1330/2304 [I 2022-10-27 22:02:28] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:09:02 [I 2022-10-27 22:02:28] ...trap/engines/engine.py.229: process: 0.54991 | load: 0.00252 [I 2022-10-27 22:02:28] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-27 22:02:28] ...optimizers/trijoint.py.151: total_norm: 21.847704 [I 2022-10-27 22:02:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 1340/2304 [I 2022-10-27 22:02:34] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:53 [I 2022-10-27 22:02:34] ...trap/engines/engine.py.229: process: 0.54229 | load: 0.00246 [I 2022-10-27 22:02:34] ...trap/engines/engine.py.230: loss: 0.17184 [I 2022-10-27 22:02:34] ...optimizers/trijoint.py.151: total_norm: 25.152052 [I 2022-10-27 22:02:39] ...trap/engines/engine.py.224: train: epoch 90 | batch 1350/2304 [I 2022-10-27 22:02:39] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:51 [I 2022-10-27 22:02:39] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00250 [I 2022-10-27 22:02:39] ...trap/engines/engine.py.230: loss: 0.17070 [I 2022-10-27 22:02:39] ...optimizers/trijoint.py.151: total_norm: 14.039707 [I 2022-10-27 22:02:45] ...trap/engines/engine.py.224: train: epoch 90 | batch 1360/2304 [I 2022-10-27 22:02:45] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:45 [I 2022-10-27 22:02:45] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00247 [I 2022-10-27 22:02:45] ...trap/engines/engine.py.230: loss: 0.10037 [I 2022-10-27 22:02:45] ...optimizers/trijoint.py.151: total_norm: 11.130475 [I 2022-10-27 22:02:51] ...trap/engines/engine.py.224: train: epoch 90 | batch 1370/2304 [I 2022-10-27 22:02:51] ...trap/engines/engine.py.228: elapsed: 0:13:10 | left: 0:08:43 [I 2022-10-27 22:02:51] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00240 [I 2022-10-27 22:02:51] ...trap/engines/engine.py.230: loss: 0.11996 [I 2022-10-27 22:02:51] ...optimizers/trijoint.py.151: total_norm: 8.143194 [I 2022-10-27 22:02:56] ...trap/engines/engine.py.224: train: epoch 90 | batch 1380/2304 [I 2022-10-27 22:02:56] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:36 [I 2022-10-27 22:02:56] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00246 [I 2022-10-27 22:02:56] ...trap/engines/engine.py.230: loss: 0.12835 [I 2022-10-27 22:02:56] ...optimizers/trijoint.py.151: total_norm: 31.327791 [I 2022-10-27 22:03:02] ...trap/engines/engine.py.224: train: epoch 90 | batch 1390/2304 [I 2022-10-27 22:03:02] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:28 [I 2022-10-27 22:03:02] ...trap/engines/engine.py.229: process: 0.56290 | load: 0.00244 [I 2022-10-27 22:03:02] ...trap/engines/engine.py.230: loss: 0.16250 [I 2022-10-27 22:03:02] ...optimizers/trijoint.py.151: total_norm: 21.844233 [I 2022-10-27 22:03:07] ...trap/engines/engine.py.224: train: epoch 90 | batch 1400/2304 [I 2022-10-27 22:03:07] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:26 [I 2022-10-27 22:03:07] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00240 [I 2022-10-27 22:03:07] ...trap/engines/engine.py.230: loss: 0.20539 [I 2022-10-27 22:03:07] ...optimizers/trijoint.py.151: total_norm: 17.279694 [I 2022-10-27 22:03:13] ...trap/engines/engine.py.224: train: epoch 90 | batch 1410/2304 [I 2022-10-27 22:03:13] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:08:22 [I 2022-10-27 22:03:13] ...trap/engines/engine.py.229: process: 0.56599 | load: 0.00244 [I 2022-10-27 22:03:13] ...trap/engines/engine.py.230: loss: 0.14219 [I 2022-10-27 22:03:13] ...optimizers/trijoint.py.151: total_norm: 17.685469 [I 2022-10-27 22:03:19] ...trap/engines/engine.py.224: train: epoch 90 | batch 1420/2304 [I 2022-10-27 22:03:19] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:08:13 [I 2022-10-27 22:03:19] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00246 [I 2022-10-27 22:03:19] ...trap/engines/engine.py.230: loss: 0.09763 [I 2022-10-27 22:03:19] ...optimizers/trijoint.py.151: total_norm: 8.452513 [I 2022-10-27 22:03:24] ...trap/engines/engine.py.224: train: epoch 90 | batch 1430/2304 [I 2022-10-27 22:03:24] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:08:08 [I 2022-10-27 22:03:24] ...trap/engines/engine.py.229: process: 0.56164 | load: 0.00254 [I 2022-10-27 22:03:24] ...trap/engines/engine.py.230: loss: 0.11158 [I 2022-10-27 22:03:24] ...optimizers/trijoint.py.151: total_norm: 12.456881 [I 2022-10-27 22:03:30] ...trap/engines/engine.py.224: train: epoch 90 | batch 1440/2304 [I 2022-10-27 22:03:30] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:08:02 [I 2022-10-27 22:03:30] ...trap/engines/engine.py.229: process: 0.56115 | load: 0.00243 [I 2022-10-27 22:03:30] ...trap/engines/engine.py.230: loss: 0.08431 [I 2022-10-27 22:03:30] ...optimizers/trijoint.py.151: total_norm: 12.325642 [I 2022-10-27 22:03:35] ...trap/engines/engine.py.224: train: epoch 90 | batch 1450/2304 [I 2022-10-27 22:03:35] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:59 [I 2022-10-27 22:03:35] ...trap/engines/engine.py.229: process: 0.55750 | load: 0.00246 [I 2022-10-27 22:03:35] ...trap/engines/engine.py.230: loss: 0.19161 [I 2022-10-27 22:03:35] ...optimizers/trijoint.py.151: total_norm: 23.209068 [I 2022-10-27 22:03:41] ...trap/engines/engine.py.224: train: epoch 90 | batch 1460/2304 [I 2022-10-27 22:03:41] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:53 [I 2022-10-27 22:03:41] ...trap/engines/engine.py.229: process: 0.56741 | load: 0.00241 [I 2022-10-27 22:03:41] ...trap/engines/engine.py.230: loss: 0.09216 [I 2022-10-27 22:03:41] ...optimizers/trijoint.py.151: total_norm: 12.737338 [I 2022-10-27 22:03:47] ...trap/engines/engine.py.224: train: epoch 90 | batch 1470/2304 [I 2022-10-27 22:03:47] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:49 [I 2022-10-27 22:03:47] ...trap/engines/engine.py.229: process: 0.56425 | load: 0.00245 [I 2022-10-27 22:03:47] ...trap/engines/engine.py.230: loss: 0.16369 [I 2022-10-27 22:03:47] ...optimizers/trijoint.py.151: total_norm: 12.214418 [I 2022-10-27 22:03:52] ...trap/engines/engine.py.224: train: epoch 90 | batch 1480/2304 [I 2022-10-27 22:03:52] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:43 [I 2022-10-27 22:03:52] ...trap/engines/engine.py.229: process: 0.56941 | load: 0.00247 [I 2022-10-27 22:03:52] ...trap/engines/engine.py.230: loss: 0.14482 [I 2022-10-27 22:03:52] ...optimizers/trijoint.py.151: total_norm: 11.100645 [I 2022-10-27 22:03:58] ...trap/engines/engine.py.224: train: epoch 90 | batch 1490/2304 [I 2022-10-27 22:03:58] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:33 [I 2022-10-27 22:03:58] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00242 [I 2022-10-27 22:03:58] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-27 22:03:58] ...optimizers/trijoint.py.151: total_norm: 8.471893 [I 2022-10-27 22:04:04] ...trap/engines/engine.py.224: train: epoch 90 | batch 1500/2304 [I 2022-10-27 22:04:04] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:28 [I 2022-10-27 22:04:04] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00246 [I 2022-10-27 22:04:04] ...trap/engines/engine.py.230: loss: 0.10784 [I 2022-10-27 22:04:04] ...optimizers/trijoint.py.151: total_norm: 15.146713 [I 2022-10-27 22:04:09] ...trap/engines/engine.py.224: train: epoch 90 | batch 1510/2304 [I 2022-10-27 22:04:09] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:24 [I 2022-10-27 22:04:09] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00247 [I 2022-10-27 22:04:09] ...trap/engines/engine.py.230: loss: 0.09976 [I 2022-10-27 22:04:09] ...optimizers/trijoint.py.151: total_norm: 11.936094 [I 2022-10-27 22:04:15] ...trap/engines/engine.py.224: train: epoch 90 | batch 1520/2304 [I 2022-10-27 22:04:15] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:07:18 [I 2022-10-27 22:04:15] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00250 [I 2022-10-27 22:04:15] ...trap/engines/engine.py.230: loss: 0.11060 [I 2022-10-27 22:04:15] ...optimizers/trijoint.py.151: total_norm: 19.669477 [I 2022-10-27 22:04:20] ...trap/engines/engine.py.224: train: epoch 90 | batch 1530/2304 [I 2022-10-27 22:04:20] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:07:12 [I 2022-10-27 22:04:20] ...trap/engines/engine.py.229: process: 0.55544 | load: 0.00243 [I 2022-10-27 22:04:20] ...trap/engines/engine.py.230: loss: 0.14780 [I 2022-10-27 22:04:20] ...optimizers/trijoint.py.151: total_norm: 20.710163 [I 2022-10-27 22:04:26] ...trap/engines/engine.py.224: train: epoch 90 | batch 1540/2304 [I 2022-10-27 22:04:26] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:07:09 [I 2022-10-27 22:04:26] ...trap/engines/engine.py.229: process: 0.56227 | load: 0.00243 [I 2022-10-27 22:04:26] ...trap/engines/engine.py.230: loss: 0.15844 [I 2022-10-27 22:04:26] ...optimizers/trijoint.py.151: total_norm: 17.503363 [I 2022-10-27 22:04:32] ...trap/engines/engine.py.224: train: epoch 90 | batch 1550/2304 [I 2022-10-27 22:04:32] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:07:06 [I 2022-10-27 22:04:32] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00240 [I 2022-10-27 22:04:32] ...trap/engines/engine.py.230: loss: 0.09369 [I 2022-10-27 22:04:32] ...optimizers/trijoint.py.151: total_norm: 23.831995 [I 2022-10-27 22:04:37] ...trap/engines/engine.py.224: train: epoch 90 | batch 1560/2304 [I 2022-10-27 22:04:37] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:58 [I 2022-10-27 22:04:37] ...trap/engines/engine.py.229: process: 0.57163 | load: 0.00243 [I 2022-10-27 22:04:37] ...trap/engines/engine.py.230: loss: 0.09585 [I 2022-10-27 22:04:37] ...optimizers/trijoint.py.151: total_norm: 13.097907 [I 2022-10-27 22:04:43] ...trap/engines/engine.py.224: train: epoch 90 | batch 1570/2304 [I 2022-10-27 22:04:43] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:56 [I 2022-10-27 22:04:43] ...trap/engines/engine.py.229: process: 0.57194 | load: 0.00295 [I 2022-10-27 22:04:43] ...trap/engines/engine.py.230: loss: 0.11655 [I 2022-10-27 22:04:43] ...optimizers/trijoint.py.151: total_norm: 10.265529 [I 2022-10-27 22:04:49] ...trap/engines/engine.py.224: train: epoch 90 | batch 1580/2304 [I 2022-10-27 22:04:49] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:45 [I 2022-10-27 22:04:49] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00290 [I 2022-10-27 22:04:49] ...trap/engines/engine.py.230: loss: 0.09073 [I 2022-10-27 22:04:49] ...optimizers/trijoint.py.151: total_norm: 20.836693 [I 2022-10-27 22:04:54] ...trap/engines/engine.py.224: train: epoch 90 | batch 1590/2304 [I 2022-10-27 22:04:54] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:38 [I 2022-10-27 22:04:54] ...trap/engines/engine.py.229: process: 0.54939 | load: 0.00258 [I 2022-10-27 22:04:54] ...trap/engines/engine.py.230: loss: 0.09635 [I 2022-10-27 22:04:54] ...optimizers/trijoint.py.151: total_norm: 9.210429 [I 2022-10-27 22:05:00] ...trap/engines/engine.py.224: train: epoch 90 | batch 1600/2304 [I 2022-10-27 22:05:00] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:30 [I 2022-10-27 22:05:00] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00257 [I 2022-10-27 22:05:00] ...trap/engines/engine.py.230: loss: 0.10518 [I 2022-10-27 22:05:00] ...optimizers/trijoint.py.151: total_norm: 21.189800 [I 2022-10-27 22:05:05] ...trap/engines/engine.py.224: train: epoch 90 | batch 1610/2304 [I 2022-10-27 22:05:05] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:27 [I 2022-10-27 22:05:05] ...trap/engines/engine.py.229: process: 0.55130 | load: 0.00253 [I 2022-10-27 22:05:05] ...trap/engines/engine.py.230: loss: 0.17714 [I 2022-10-27 22:05:05] ...optimizers/trijoint.py.151: total_norm: 17.921001 [I 2022-10-27 22:05:11] ...trap/engines/engine.py.224: train: epoch 90 | batch 1620/2304 [I 2022-10-27 22:05:11] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:23 [I 2022-10-27 22:05:11] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00244 [I 2022-10-27 22:05:11] ...trap/engines/engine.py.230: loss: 0.09145 [I 2022-10-27 22:05:11] ...optimizers/trijoint.py.151: total_norm: 12.101853 [I 2022-10-27 22:05:17] ...trap/engines/engine.py.224: train: epoch 90 | batch 1630/2304 [I 2022-10-27 22:05:17] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:16 [I 2022-10-27 22:05:17] ...trap/engines/engine.py.229: process: 0.54826 | load: 0.00248 [I 2022-10-27 22:05:17] ...trap/engines/engine.py.230: loss: 0.17886 [I 2022-10-27 22:05:17] ...optimizers/trijoint.py.151: total_norm: 31.372423 [I 2022-10-27 22:05:22] ...trap/engines/engine.py.224: train: epoch 90 | batch 1640/2304 [I 2022-10-27 22:05:22] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:06:10 [I 2022-10-27 22:05:22] ...trap/engines/engine.py.229: process: 0.55484 | load: 0.00253 [I 2022-10-27 22:05:22] ...trap/engines/engine.py.230: loss: 0.20348 [I 2022-10-27 22:05:22] ...optimizers/trijoint.py.151: total_norm: 22.236773 [I 2022-10-27 22:05:28] ...trap/engines/engine.py.224: train: epoch 90 | batch 1650/2304 [I 2022-10-27 22:05:28] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:06:07 [I 2022-10-27 22:05:28] ...trap/engines/engine.py.229: process: 0.56250 | load: 0.00246 [I 2022-10-27 22:05:28] ...trap/engines/engine.py.230: loss: 0.15526 [I 2022-10-27 22:05:28] ...optimizers/trijoint.py.151: total_norm: 14.499121 [I 2022-10-27 22:05:33] ...trap/engines/engine.py.224: train: epoch 90 | batch 1660/2304 [I 2022-10-27 22:05:33] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:59 [I 2022-10-27 22:05:33] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00249 [I 2022-10-27 22:05:33] ...trap/engines/engine.py.230: loss: 0.14165 [I 2022-10-27 22:05:33] ...optimizers/trijoint.py.151: total_norm: 34.269314 [I 2022-10-27 22:05:39] ...trap/engines/engine.py.224: train: epoch 90 | batch 1670/2304 [I 2022-10-27 22:05:39] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:55 [I 2022-10-27 22:05:39] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00251 [I 2022-10-27 22:05:39] ...trap/engines/engine.py.230: loss: 0.12038 [I 2022-10-27 22:05:39] ...optimizers/trijoint.py.151: total_norm: 20.138432 [I 2022-10-27 22:05:45] ...trap/engines/engine.py.224: train: epoch 90 | batch 1680/2304 [I 2022-10-27 22:05:45] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:49 [I 2022-10-27 22:05:45] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00248 [I 2022-10-27 22:05:45] ...trap/engines/engine.py.230: loss: 0.10647 [I 2022-10-27 22:05:45] ...optimizers/trijoint.py.151: total_norm: 15.397139 [I 2022-10-27 22:05:50] ...trap/engines/engine.py.224: train: epoch 90 | batch 1690/2304 [I 2022-10-27 22:05:50] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:41 [I 2022-10-27 22:05:50] ...trap/engines/engine.py.229: process: 0.55532 | load: 0.00250 [I 2022-10-27 22:05:50] ...trap/engines/engine.py.230: loss: 0.10032 [I 2022-10-27 22:05:50] ...optimizers/trijoint.py.151: total_norm: 8.898131 [I 2022-10-27 22:05:56] ...trap/engines/engine.py.224: train: epoch 90 | batch 1700/2304 [I 2022-10-27 22:05:56] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:34 [I 2022-10-27 22:05:56] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00247 [I 2022-10-27 22:05:56] ...trap/engines/engine.py.230: loss: 0.07880 [I 2022-10-27 22:05:56] ...optimizers/trijoint.py.151: total_norm: 10.323174 [I 2022-10-27 22:06:01] ...trap/engines/engine.py.224: train: epoch 90 | batch 1710/2304 [I 2022-10-27 22:06:01] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:32 [I 2022-10-27 22:06:01] ...trap/engines/engine.py.229: process: 0.56648 | load: 0.00245 [I 2022-10-27 22:06:01] ...trap/engines/engine.py.230: loss: 0.10874 [I 2022-10-27 22:06:01] ...optimizers/trijoint.py.151: total_norm: 23.911303 [I 2022-10-27 22:06:07] ...trap/engines/engine.py.224: train: epoch 90 | batch 1720/2304 [I 2022-10-27 22:06:07] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:27 [I 2022-10-27 22:06:07] ...trap/engines/engine.py.229: process: 0.56254 | load: 0.00245 [I 2022-10-27 22:06:07] ...trap/engines/engine.py.230: loss: 0.10750 [I 2022-10-27 22:06:07] ...optimizers/trijoint.py.151: total_norm: 17.607113 [I 2022-10-27 22:06:13] ...trap/engines/engine.py.224: train: epoch 90 | batch 1730/2304 [I 2022-10-27 22:06:13] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:19 [I 2022-10-27 22:06:13] ...trap/engines/engine.py.229: process: 0.54993 | load: 0.00245 [I 2022-10-27 22:06:13] ...trap/engines/engine.py.230: loss: 0.13588 [I 2022-10-27 22:06:13] ...optimizers/trijoint.py.151: total_norm: 42.397705 [I 2022-10-27 22:06:18] ...trap/engines/engine.py.224: train: epoch 90 | batch 1740/2304 [I 2022-10-27 22:06:18] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:05:12 [I 2022-10-27 22:06:18] ...trap/engines/engine.py.229: process: 0.54737 | load: 0.00243 [I 2022-10-27 22:06:18] ...trap/engines/engine.py.230: loss: 0.12899 [I 2022-10-27 22:06:18] ...optimizers/trijoint.py.151: total_norm: 8.582135 [I 2022-10-27 22:06:24] ...trap/engines/engine.py.224: train: epoch 90 | batch 1750/2304 [I 2022-10-27 22:06:24] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:11 [I 2022-10-27 22:06:24] ...trap/engines/engine.py.229: process: 0.56776 | load: 0.00289 [I 2022-10-27 22:06:24] ...trap/engines/engine.py.230: loss: 0.12350 [I 2022-10-27 22:06:24] ...optimizers/trijoint.py.151: total_norm: 32.171432 [I 2022-10-27 22:06:29] ...trap/engines/engine.py.224: train: epoch 90 | batch 1760/2304 [I 2022-10-27 22:06:29] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:05:03 [I 2022-10-27 22:06:29] ...trap/engines/engine.py.229: process: 0.56119 | load: 0.00241 [I 2022-10-27 22:06:29] ...trap/engines/engine.py.230: loss: 0.17171 [I 2022-10-27 22:06:29] ...optimizers/trijoint.py.151: total_norm: 12.014259 [I 2022-10-27 22:06:35] ...trap/engines/engine.py.224: train: epoch 90 | batch 1770/2304 [I 2022-10-27 22:06:35] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:05:08 [I 2022-10-27 22:06:35] ...trap/engines/engine.py.229: process: 0.58234 | load: 0.00244 [I 2022-10-27 22:06:35] ...trap/engines/engine.py.230: loss: 0.17233 [I 2022-10-27 22:06:35] ...optimizers/trijoint.py.151: total_norm: 11.351506 [I 2022-10-27 22:06:41] ...trap/engines/engine.py.224: train: epoch 90 | batch 1780/2304 [I 2022-10-27 22:06:41] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:54 [I 2022-10-27 22:06:41] ...trap/engines/engine.py.229: process: 0.55943 | load: 0.00303 [I 2022-10-27 22:06:41] ...trap/engines/engine.py.230: loss: 0.11148 [I 2022-10-27 22:06:41] ...optimizers/trijoint.py.151: total_norm: 26.277262 [I 2022-10-27 22:06:46] ...trap/engines/engine.py.224: train: epoch 90 | batch 1790/2304 [I 2022-10-27 22:06:46] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:54 [I 2022-10-27 22:06:46] ...trap/engines/engine.py.229: process: 0.57101 | load: 0.00246 [I 2022-10-27 22:06:46] ...trap/engines/engine.py.230: loss: 0.10487 [I 2022-10-27 22:06:46] ...optimizers/trijoint.py.151: total_norm: 8.300251 [I 2022-10-27 22:06:52] ...trap/engines/engine.py.224: train: epoch 90 | batch 1800/2304 [I 2022-10-27 22:06:52] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:43 [I 2022-10-27 22:06:52] ...trap/engines/engine.py.229: process: 0.56811 | load: 0.00248 [I 2022-10-27 22:06:52] ...trap/engines/engine.py.230: loss: 0.11088 [I 2022-10-27 22:06:52] ...optimizers/trijoint.py.151: total_norm: 13.579519 [I 2022-10-27 22:06:58] ...trap/engines/engine.py.224: train: epoch 90 | batch 1810/2304 [I 2022-10-27 22:06:58] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:36 [I 2022-10-27 22:06:58] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00243 [I 2022-10-27 22:06:58] ...trap/engines/engine.py.230: loss: 0.13874 [I 2022-10-27 22:06:58] ...optimizers/trijoint.py.151: total_norm: 24.477928 [I 2022-10-27 22:07:03] ...trap/engines/engine.py.224: train: epoch 90 | batch 1820/2304 [I 2022-10-27 22:07:03] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:30 [I 2022-10-27 22:07:03] ...trap/engines/engine.py.229: process: 0.55267 | load: 0.00300 [I 2022-10-27 22:07:03] ...trap/engines/engine.py.230: loss: 0.20280 [I 2022-10-27 22:07:03] ...optimizers/trijoint.py.151: total_norm: 13.975546 [I 2022-10-27 22:07:09] ...trap/engines/engine.py.224: train: epoch 90 | batch 1830/2304 [I 2022-10-27 22:07:09] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:24 [I 2022-10-27 22:07:09] ...trap/engines/engine.py.229: process: 0.55724 | load: 0.00250 [I 2022-10-27 22:07:09] ...trap/engines/engine.py.230: loss: 0.12165 [I 2022-10-27 22:07:09] ...optimizers/trijoint.py.151: total_norm: 31.298321 [I 2022-10-27 22:07:14] ...trap/engines/engine.py.224: train: epoch 90 | batch 1840/2304 [I 2022-10-27 22:07:14] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:04:18 [I 2022-10-27 22:07:14] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00248 [I 2022-10-27 22:07:14] ...trap/engines/engine.py.230: loss: 0.09882 [I 2022-10-27 22:07:14] ...optimizers/trijoint.py.151: total_norm: 11.262825 [I 2022-10-27 22:07:20] ...trap/engines/engine.py.224: train: epoch 90 | batch 1850/2304 [I 2022-10-27 22:07:20] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:04:13 [I 2022-10-27 22:07:20] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00250 [I 2022-10-27 22:07:20] ...trap/engines/engine.py.230: loss: 0.14767 [I 2022-10-27 22:07:20] ...optimizers/trijoint.py.151: total_norm: 9.409514 [I 2022-10-27 22:07:26] ...trap/engines/engine.py.224: train: epoch 90 | batch 1860/2304 [I 2022-10-27 22:07:26] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:04:10 [I 2022-10-27 22:07:26] ...trap/engines/engine.py.229: process: 0.54898 | load: 0.00349 [I 2022-10-27 22:07:26] ...trap/engines/engine.py.230: loss: 0.12287 [I 2022-10-27 22:07:26] ...optimizers/trijoint.py.151: total_norm: 14.969262 [I 2022-10-27 22:07:31] ...trap/engines/engine.py.224: train: epoch 90 | batch 1870/2304 [I 2022-10-27 22:07:31] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:04:02 [I 2022-10-27 22:07:31] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00250 [I 2022-10-27 22:07:31] ...trap/engines/engine.py.230: loss: 0.10087 [I 2022-10-27 22:07:31] ...optimizers/trijoint.py.151: total_norm: 13.805182 [I 2022-10-27 22:07:37] ...trap/engines/engine.py.224: train: epoch 90 | batch 1880/2304 [I 2022-10-27 22:07:37] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:56 [I 2022-10-27 22:07:37] ...trap/engines/engine.py.229: process: 0.54893 | load: 0.00248 [I 2022-10-27 22:07:37] ...trap/engines/engine.py.230: loss: 0.12938 [I 2022-10-27 22:07:37] ...optimizers/trijoint.py.151: total_norm: 11.868878 [I 2022-10-27 22:07:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 1890/2304 [I 2022-10-27 22:07:42] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:50 [I 2022-10-27 22:07:42] ...trap/engines/engine.py.229: process: 0.55017 | load: 0.00241 [I 2022-10-27 22:07:42] ...trap/engines/engine.py.230: loss: 0.11547 [I 2022-10-27 22:07:42] ...optimizers/trijoint.py.151: total_norm: 13.831743 [I 2022-10-27 22:07:48] ...trap/engines/engine.py.224: train: epoch 90 | batch 1900/2304 [I 2022-10-27 22:07:48] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:44 [I 2022-10-27 22:07:48] ...trap/engines/engine.py.229: process: 0.54601 | load: 0.00255 [I 2022-10-27 22:07:48] ...trap/engines/engine.py.230: loss: 0.10469 [I 2022-10-27 22:07:48] ...optimizers/trijoint.py.151: total_norm: 7.374956 [I 2022-10-27 22:07:54] ...trap/engines/engine.py.224: train: epoch 90 | batch 1910/2304 [I 2022-10-27 22:07:54] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:39 [I 2022-10-27 22:07:54] ...trap/engines/engine.py.229: process: 0.54192 | load: 0.00257 [I 2022-10-27 22:07:54] ...trap/engines/engine.py.230: loss: 0.09658 [I 2022-10-27 22:07:54] ...optimizers/trijoint.py.151: total_norm: 13.135313 [I 2022-10-27 22:07:59] ...trap/engines/engine.py.224: train: epoch 90 | batch 1920/2304 [I 2022-10-27 22:07:59] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:34 [I 2022-10-27 22:07:59] ...trap/engines/engine.py.229: process: 0.56806 | load: 0.00260 [I 2022-10-27 22:07:59] ...trap/engines/engine.py.230: loss: 0.09237 [I 2022-10-27 22:07:59] ...optimizers/trijoint.py.151: total_norm: 16.382547 [I 2022-10-27 22:08:05] ...trap/engines/engine.py.224: train: epoch 90 | batch 1930/2304 [I 2022-10-27 22:08:05] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:30 [I 2022-10-27 22:08:05] ...trap/engines/engine.py.229: process: 0.56566 | load: 0.00249 [I 2022-10-27 22:08:05] ...trap/engines/engine.py.230: loss: 0.11457 [I 2022-10-27 22:08:05] ...optimizers/trijoint.py.151: total_norm: 14.844028 [I 2022-10-27 22:08:10] ...trap/engines/engine.py.224: train: epoch 90 | batch 1940/2304 [I 2022-10-27 22:08:10] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:23 [I 2022-10-27 22:08:10] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00253 [I 2022-10-27 22:08:10] ...trap/engines/engine.py.230: loss: 0.11789 [I 2022-10-27 22:08:10] ...optimizers/trijoint.py.151: total_norm: 12.721847 [I 2022-10-27 22:08:16] ...trap/engines/engine.py.224: train: epoch 90 | batch 1950/2304 [I 2022-10-27 22:08:16] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:17 [I 2022-10-27 22:08:16] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00245 [I 2022-10-27 22:08:16] ...trap/engines/engine.py.230: loss: 0.18628 [I 2022-10-27 22:08:16] ...optimizers/trijoint.py.151: total_norm: 18.007776 [I 2022-10-27 22:08:22] ...trap/engines/engine.py.224: train: epoch 90 | batch 1960/2304 [I 2022-10-27 22:08:22] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:03:13 [I 2022-10-27 22:08:22] ...trap/engines/engine.py.229: process: 0.56939 | load: 0.00245 [I 2022-10-27 22:08:22] ...trap/engines/engine.py.230: loss: 0.19265 [I 2022-10-27 22:08:22] ...optimizers/trijoint.py.151: total_norm: 20.711048 [I 2022-10-27 22:08:27] ...trap/engines/engine.py.224: train: epoch 90 | batch 1970/2304 [I 2022-10-27 22:08:27] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:03:04 [I 2022-10-27 22:08:27] ...trap/engines/engine.py.229: process: 0.54604 | load: 0.00252 [I 2022-10-27 22:08:27] ...trap/engines/engine.py.230: loss: 0.20366 [I 2022-10-27 22:08:27] ...optimizers/trijoint.py.151: total_norm: 30.322865 [I 2022-10-27 22:08:33] ...trap/engines/engine.py.224: train: epoch 90 | batch 1980/2304 [I 2022-10-27 22:08:33] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:02:59 [I 2022-10-27 22:08:33] ...trap/engines/engine.py.229: process: 0.54358 | load: 0.00243 [I 2022-10-27 22:08:33] ...trap/engines/engine.py.230: loss: 0.10379 [I 2022-10-27 22:08:33] ...optimizers/trijoint.py.151: total_norm: 20.404034 [I 2022-10-27 22:08:38] ...trap/engines/engine.py.224: train: epoch 90 | batch 1990/2304 [I 2022-10-27 22:08:38] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:54 [I 2022-10-27 22:08:38] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00278 [I 2022-10-27 22:08:38] ...trap/engines/engine.py.230: loss: 0.19803 [I 2022-10-27 22:08:38] ...optimizers/trijoint.py.151: total_norm: 29.173176 [I 2022-10-27 22:08:44] ...trap/engines/engine.py.224: train: epoch 90 | batch 2000/2304 [I 2022-10-27 22:08:44] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:49 [I 2022-10-27 22:08:44] ...trap/engines/engine.py.229: process: 0.56553 | load: 0.00249 [I 2022-10-27 22:08:44] ...trap/engines/engine.py.230: loss: 0.12882 [I 2022-10-27 22:08:44] ...optimizers/trijoint.py.151: total_norm: 22.202665 [I 2022-10-27 22:08:49] ...trap/engines/engine.py.224: train: epoch 90 | batch 2010/2304 [I 2022-10-27 22:08:49] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:43 [I 2022-10-27 22:08:49] ...trap/engines/engine.py.229: process: 0.54957 | load: 0.00246 [I 2022-10-27 22:08:49] ...trap/engines/engine.py.230: loss: 0.11237 [I 2022-10-27 22:08:49] ...optimizers/trijoint.py.151: total_norm: 20.066761 [I 2022-10-27 22:08:55] ...trap/engines/engine.py.224: train: epoch 90 | batch 2020/2304 [I 2022-10-27 22:08:55] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:38 [I 2022-10-27 22:08:55] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00246 [I 2022-10-27 22:08:55] ...trap/engines/engine.py.230: loss: 0.11298 [I 2022-10-27 22:08:55] ...optimizers/trijoint.py.151: total_norm: 19.154627 [I 2022-10-27 22:09:01] ...trap/engines/engine.py.224: train: epoch 90 | batch 2030/2304 [I 2022-10-27 22:09:01] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:33 [I 2022-10-27 22:09:01] ...trap/engines/engine.py.229: process: 0.56616 | load: 0.00269 [I 2022-10-27 22:09:01] ...trap/engines/engine.py.230: loss: 0.10916 [I 2022-10-27 22:09:01] ...optimizers/trijoint.py.151: total_norm: 12.626317 [I 2022-10-27 22:09:06] ...trap/engines/engine.py.224: train: epoch 90 | batch 2040/2304 [I 2022-10-27 22:09:06] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:31 [I 2022-10-27 22:09:06] ...trap/engines/engine.py.229: process: 0.55293 | load: 0.00248 [I 2022-10-27 22:09:06] ...trap/engines/engine.py.230: loss: 0.16002 [I 2022-10-27 22:09:06] ...optimizers/trijoint.py.151: total_norm: 17.791195 [I 2022-10-27 22:09:12] ...trap/engines/engine.py.224: train: epoch 90 | batch 2050/2304 [I 2022-10-27 22:09:12] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:22 [I 2022-10-27 22:09:12] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00243 [I 2022-10-27 22:09:12] ...trap/engines/engine.py.230: loss: 0.16254 [I 2022-10-27 22:09:12] ...optimizers/trijoint.py.151: total_norm: 23.325970 [I 2022-10-27 22:09:18] ...trap/engines/engine.py.224: train: epoch 90 | batch 2060/2304 [I 2022-10-27 22:09:18] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:14 [I 2022-10-27 22:09:18] ...trap/engines/engine.py.229: process: 0.54641 | load: 0.00245 [I 2022-10-27 22:09:18] ...trap/engines/engine.py.230: loss: 0.13962 [I 2022-10-27 22:09:18] ...optimizers/trijoint.py.151: total_norm: 13.784994 [I 2022-10-27 22:09:23] ...trap/engines/engine.py.224: train: epoch 90 | batch 2070/2304 [I 2022-10-27 22:09:23] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:10 [I 2022-10-27 22:09:23] ...trap/engines/engine.py.229: process: 0.55399 | load: 0.00251 [I 2022-10-27 22:09:23] ...trap/engines/engine.py.230: loss: 0.18398 [I 2022-10-27 22:09:23] ...optimizers/trijoint.py.151: total_norm: 37.276451 [I 2022-10-27 22:09:29] ...trap/engines/engine.py.224: train: epoch 90 | batch 2080/2304 [I 2022-10-27 22:09:29] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:02:04 [I 2022-10-27 22:09:29] ...trap/engines/engine.py.229: process: 0.54451 | load: 0.00252 [I 2022-10-27 22:09:29] ...trap/engines/engine.py.230: loss: 0.11790 [I 2022-10-27 22:09:29] ...optimizers/trijoint.py.151: total_norm: 15.302931 [I 2022-10-27 22:09:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 2090/2304 [I 2022-10-27 22:09:34] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:59 [I 2022-10-27 22:09:34] ...trap/engines/engine.py.229: process: 0.55562 | load: 0.00252 [I 2022-10-27 22:09:34] ...trap/engines/engine.py.230: loss: 0.15491 [I 2022-10-27 22:09:34] ...optimizers/trijoint.py.151: total_norm: 18.607401 [I 2022-10-27 22:09:40] ...trap/engines/engine.py.224: train: epoch 90 | batch 2100/2304 [I 2022-10-27 22:09:40] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:54 [I 2022-10-27 22:09:40] ...trap/engines/engine.py.229: process: 0.56250 | load: 0.00247 [I 2022-10-27 22:09:40] ...trap/engines/engine.py.230: loss: 0.09522 [I 2022-10-27 22:09:40] ...optimizers/trijoint.py.151: total_norm: 12.011398 [I 2022-10-27 22:09:45] ...trap/engines/engine.py.224: train: epoch 90 | batch 2110/2304 [I 2022-10-27 22:09:45] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:48 [I 2022-10-27 22:09:45] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00255 [I 2022-10-27 22:09:45] ...trap/engines/engine.py.230: loss: 0.11821 [I 2022-10-27 22:09:45] ...optimizers/trijoint.py.151: total_norm: 10.436344 [I 2022-10-27 22:09:51] ...trap/engines/engine.py.224: train: epoch 90 | batch 2120/2304 [I 2022-10-27 22:09:51] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:42 [I 2022-10-27 22:09:51] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00283 [I 2022-10-27 22:09:51] ...trap/engines/engine.py.230: loss: 0.10223 [I 2022-10-27 22:09:51] ...optimizers/trijoint.py.151: total_norm: 15.657743 [I 2022-10-27 22:09:57] ...trap/engines/engine.py.224: train: epoch 90 | batch 2130/2304 [I 2022-10-27 22:09:57] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:37 [I 2022-10-27 22:09:57] ...trap/engines/engine.py.229: process: 0.56789 | load: 0.00256 [I 2022-10-27 22:09:57] ...trap/engines/engine.py.230: loss: 0.12764 [I 2022-10-27 22:09:57] ...optimizers/trijoint.py.151: total_norm: 13.234726 [I 2022-10-27 22:10:02] ...trap/engines/engine.py.224: train: epoch 90 | batch 2140/2304 [I 2022-10-27 22:10:02] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:31 [I 2022-10-27 22:10:02] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00254 [I 2022-10-27 22:10:02] ...trap/engines/engine.py.230: loss: 0.11611 [I 2022-10-27 22:10:02] ...optimizers/trijoint.py.151: total_norm: 12.191332 [I 2022-10-27 22:10:08] ...trap/engines/engine.py.224: train: epoch 90 | batch 2150/2304 [I 2022-10-27 22:10:08] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:27 [I 2022-10-27 22:10:08] ...trap/engines/engine.py.229: process: 0.55444 | load: 0.00247 [I 2022-10-27 22:10:08] ...trap/engines/engine.py.230: loss: 0.13437 [I 2022-10-27 22:10:08] ...optimizers/trijoint.py.151: total_norm: 39.779358 [I 2022-10-27 22:10:14] ...trap/engines/engine.py.224: train: epoch 90 | batch 2160/2304 [I 2022-10-27 22:10:14] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:20 [I 2022-10-27 22:10:14] ...trap/engines/engine.py.229: process: 0.56168 | load: 0.00247 [I 2022-10-27 22:10:14] ...trap/engines/engine.py.230: loss: 0.09734 [I 2022-10-27 22:10:14] ...optimizers/trijoint.py.151: total_norm: 13.630976 [I 2022-10-27 22:10:19] ...trap/engines/engine.py.224: train: epoch 90 | batch 2170/2304 [I 2022-10-27 22:10:19] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:01:15 [I 2022-10-27 22:10:19] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00246 [I 2022-10-27 22:10:19] ...trap/engines/engine.py.230: loss: 0.13799 [I 2022-10-27 22:10:19] ...optimizers/trijoint.py.151: total_norm: 24.930424 [I 2022-10-27 22:10:25] ...trap/engines/engine.py.224: train: epoch 90 | batch 2180/2304 [I 2022-10-27 22:10:25] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:09 [I 2022-10-27 22:10:25] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00245 [I 2022-10-27 22:10:25] ...trap/engines/engine.py.230: loss: 0.13851 [I 2022-10-27 22:10:25] ...optimizers/trijoint.py.151: total_norm: 22.343637 [I 2022-10-27 22:10:30] ...trap/engines/engine.py.224: train: epoch 90 | batch 2190/2304 [I 2022-10-27 22:10:30] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:01:03 [I 2022-10-27 22:10:30] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00246 [I 2022-10-27 22:10:30] ...trap/engines/engine.py.230: loss: 0.12032 [I 2022-10-27 22:10:30] ...optimizers/trijoint.py.151: total_norm: 12.180615 [I 2022-10-27 22:10:36] ...trap/engines/engine.py.224: train: epoch 90 | batch 2200/2304 [I 2022-10-27 22:10:36] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:58 [I 2022-10-27 22:10:36] ...trap/engines/engine.py.229: process: 0.56545 | load: 0.00247 [I 2022-10-27 22:10:36] ...trap/engines/engine.py.230: loss: 0.11791 [I 2022-10-27 22:10:36] ...optimizers/trijoint.py.151: total_norm: 9.647017 [I 2022-10-27 22:10:42] ...trap/engines/engine.py.224: train: epoch 90 | batch 2210/2304 [I 2022-10-27 22:10:42] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:52 [I 2022-10-27 22:10:42] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00246 [I 2022-10-27 22:10:42] ...trap/engines/engine.py.230: loss: 0.11932 [I 2022-10-27 22:10:42] ...optimizers/trijoint.py.151: total_norm: 10.495440 [I 2022-10-27 22:10:47] ...trap/engines/engine.py.224: train: epoch 90 | batch 2220/2304 [I 2022-10-27 22:10:47] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:46 [I 2022-10-27 22:10:47] ...trap/engines/engine.py.229: process: 0.55080 | load: 0.00247 [I 2022-10-27 22:10:47] ...trap/engines/engine.py.230: loss: 0.09477 [I 2022-10-27 22:10:47] ...optimizers/trijoint.py.151: total_norm: 10.004557 [I 2022-10-27 22:10:53] ...trap/engines/engine.py.224: train: epoch 90 | batch 2230/2304 [I 2022-10-27 22:10:53] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:40 [I 2022-10-27 22:10:53] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00247 [I 2022-10-27 22:10:53] ...trap/engines/engine.py.230: loss: 0.08931 [I 2022-10-27 22:10:53] ...optimizers/trijoint.py.151: total_norm: 10.504766 [I 2022-10-27 22:10:58] ...trap/engines/engine.py.224: train: epoch 90 | batch 2240/2304 [I 2022-10-27 22:10:58] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:35 [I 2022-10-27 22:10:58] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00245 [I 2022-10-27 22:10:58] ...trap/engines/engine.py.230: loss: 0.11373 [I 2022-10-27 22:10:58] ...optimizers/trijoint.py.151: total_norm: 15.522109 [I 2022-10-27 22:11:04] ...trap/engines/engine.py.224: train: epoch 90 | batch 2250/2304 [I 2022-10-27 22:11:04] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:30 [I 2022-10-27 22:11:04] ...trap/engines/engine.py.229: process: 0.55202 | load: 0.00279 [I 2022-10-27 22:11:04] ...trap/engines/engine.py.230: loss: 0.18830 [I 2022-10-27 22:11:04] ...optimizers/trijoint.py.151: total_norm: 20.931753 [I 2022-10-27 22:11:10] ...trap/engines/engine.py.224: train: epoch 90 | batch 2260/2304 [I 2022-10-27 22:11:10] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:25 [I 2022-10-27 22:11:10] ...trap/engines/engine.py.229: process: 0.57187 | load: 0.00265 [I 2022-10-27 22:11:10] ...trap/engines/engine.py.230: loss: 0.17753 [I 2022-10-27 22:11:10] ...optimizers/trijoint.py.151: total_norm: 31.296032 [I 2022-10-27 22:11:15] ...trap/engines/engine.py.224: train: epoch 90 | batch 2270/2304 [I 2022-10-27 22:11:15] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:19 [I 2022-10-27 22:11:15] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00253 [I 2022-10-27 22:11:15] ...trap/engines/engine.py.230: loss: 0.12303 [I 2022-10-27 22:11:15] ...optimizers/trijoint.py.151: total_norm: 11.378045 [I 2022-10-27 22:11:21] ...trap/engines/engine.py.224: train: epoch 90 | batch 2280/2304 [I 2022-10-27 22:11:21] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:13 [I 2022-10-27 22:11:21] ...trap/engines/engine.py.229: process: 0.54433 | load: 0.00250 [I 2022-10-27 22:11:21] ...trap/engines/engine.py.230: loss: 0.19050 [I 2022-10-27 22:11:21] ...optimizers/trijoint.py.151: total_norm: 26.562098 [I 2022-10-27 22:11:27] ...trap/engines/engine.py.224: train: epoch 90 | batch 2290/2304 [I 2022-10-27 22:11:27] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:07 [I 2022-10-27 22:11:27] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00248 [I 2022-10-27 22:11:27] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-27 22:11:27] ...optimizers/trijoint.py.151: total_norm: 18.353865 [I 2022-10-27 22:11:32] ...trap/engines/engine.py.224: train: epoch 90 | batch 2300/2304 [I 2022-10-27 22:11:32] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:00:02 [I 2022-10-27 22:11:32] ...trap/engines/engine.py.229: process: 0.55635 | load: 0.00015 [I 2022-10-27 22:11:32] ...trap/engines/engine.py.230: loss: 0.16645 [I 2022-10-27 22:11:32] ...optimizers/trijoint.py.151: total_norm: 13.896174 [I 2022-10-27 22:11:34] ...trap/engines/engine.py.224: train: epoch 90 | batch 2304/2304 [I 2022-10-27 22:11:34] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:00:00 [I 2022-10-27 22:11:34] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00016 [I 2022-10-27 22:11:34] ...trap/engines/engine.py.230: loss: 0.09329 [I 2022-10-27 22:11:34] ...optimizers/trijoint.py.151: total_norm: 12.054402 [S 2022-10-27 22:11:36] ...trap/engines/engine.py.129: train_epoch.epoch: 90 [S 2022-10-27 22:11:36] ...trap/engines/engine.py.129: train_epoch.loss: 0.1305746202742976 [I 2022-10-27 22:11:36] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 22:11:37] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 90 [I 2022-10-27 22:11:37] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 22:11:39] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 22:11:39] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 22:11:39] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 22:11:42] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 22:11:42] ...trap/engines/engine.py.321: eval: epoch 90 | batch 0/510 [I 2022-10-27 22:11:42] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:45:53 [I 2022-10-27 22:11:42] ...trap/engines/engine.py.326: process: 5.39989 | load: 3.35085 [I 2022-10-27 22:11:46] ...trap/engines/engine.py.321: eval: epoch 90 | batch 10/510 [I 2022-10-27 22:11:46] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:18 [I 2022-10-27 22:11:46] ...trap/engines/engine.py.326: process: 0.33172 | load: 0.00018 [I 2022-10-27 22:11:49] ...trap/engines/engine.py.321: eval: epoch 90 | batch 20/510 [I 2022-10-27 22:11:49] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:15 [I 2022-10-27 22:11:49] ...trap/engines/engine.py.326: process: 0.32207 | load: 0.00021 [I 2022-10-27 22:11:52] ...trap/engines/engine.py.321: eval: epoch 90 | batch 30/510 [I 2022-10-27 22:11:52] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:46 [I 2022-10-27 22:11:52] ...trap/engines/engine.py.326: process: 0.35689 | load: 0.00018 [I 2022-10-27 22:11:56] ...trap/engines/engine.py.321: eval: epoch 90 | batch 40/510 [I 2022-10-27 22:11:56] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:37 [I 2022-10-27 22:11:56] ...trap/engines/engine.py.326: process: 0.32851 | load: 0.00019 [I 2022-10-27 22:11:59] ...trap/engines/engine.py.321: eval: epoch 90 | batch 50/510 [I 2022-10-27 22:11:59] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:35 [I 2022-10-27 22:11:59] ...trap/engines/engine.py.326: process: 0.33557 | load: 0.00019 [I 2022-10-27 22:12:02] ...trap/engines/engine.py.321: eval: epoch 90 | batch 60/510 [I 2022-10-27 22:12:02] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:29 [I 2022-10-27 22:12:02] ...trap/engines/engine.py.326: process: 0.32568 | load: 0.00018 [I 2022-10-27 22:12:06] ...trap/engines/engine.py.321: eval: epoch 90 | batch 70/510 [I 2022-10-27 22:12:06] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:27 [I 2022-10-27 22:12:06] ...trap/engines/engine.py.326: process: 0.34310 | load: 0.00019 [I 2022-10-27 22:12:09] ...trap/engines/engine.py.321: eval: epoch 90 | batch 80/510 [I 2022-10-27 22:12:09] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:25 [I 2022-10-27 22:12:09] ...trap/engines/engine.py.326: process: 0.33191 | load: 0.00020 [I 2022-10-27 22:12:13] ...trap/engines/engine.py.321: eval: epoch 90 | batch 90/510 [I 2022-10-27 22:12:13] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:22 [I 2022-10-27 22:12:13] ...trap/engines/engine.py.326: process: 0.33637 | load: 0.00018 [I 2022-10-27 22:12:16] ...trap/engines/engine.py.321: eval: epoch 90 | batch 100/510 [I 2022-10-27 22:12:16] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-27 22:12:16] ...trap/engines/engine.py.326: process: 0.29202 | load: 0.00019 [I 2022-10-27 22:12:19] ...trap/engines/engine.py.321: eval: epoch 90 | batch 110/510 [I 2022-10-27 22:12:19] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-27 22:12:19] ...trap/engines/engine.py.326: process: 0.28493 | load: 0.00019 [I 2022-10-27 22:12:22] ...trap/engines/engine.py.321: eval: epoch 90 | batch 120/510 [I 2022-10-27 22:12:22] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 22:12:22] ...trap/engines/engine.py.326: process: 0.29069 | load: 0.00015 [I 2022-10-27 22:12:25] ...trap/engines/engine.py.321: eval: epoch 90 | batch 130/510 [I 2022-10-27 22:12:25] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-27 22:12:25] ...trap/engines/engine.py.326: process: 0.28845 | load: 0.00017 [I 2022-10-27 22:12:27] ...trap/engines/engine.py.321: eval: epoch 90 | batch 140/510 [I 2022-10-27 22:12:27] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-27 22:12:28] ...trap/engines/engine.py.326: process: 0.28971 | load: 0.00015 [I 2022-10-27 22:12:30] ...trap/engines/engine.py.321: eval: epoch 90 | batch 150/510 [I 2022-10-27 22:12:30] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:44 [I 2022-10-27 22:12:30] ...trap/engines/engine.py.326: process: 0.29219 | load: 0.00021 [I 2022-10-27 22:12:33] ...trap/engines/engine.py.321: eval: epoch 90 | batch 160/510 [I 2022-10-27 22:12:33] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:42 [I 2022-10-27 22:12:33] ...trap/engines/engine.py.326: process: 0.29082 | load: 0.00015 [I 2022-10-27 22:12:36] ...trap/engines/engine.py.321: eval: epoch 90 | batch 170/510 [I 2022-10-27 22:12:36] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-27 22:12:36] ...trap/engines/engine.py.326: process: 0.28700 | load: 0.00017 [I 2022-10-27 22:12:39] ...trap/engines/engine.py.321: eval: epoch 90 | batch 180/510 [I 2022-10-27 22:12:39] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:36 [I 2022-10-27 22:12:39] ...trap/engines/engine.py.326: process: 0.29447 | load: 0.00014 [I 2022-10-27 22:12:42] ...trap/engines/engine.py.321: eval: epoch 90 | batch 190/510 [I 2022-10-27 22:12:42] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:34 [I 2022-10-27 22:12:42] ...trap/engines/engine.py.326: process: 0.29242 | load: 0.00026 [I 2022-10-27 22:12:45] ...trap/engines/engine.py.321: eval: epoch 90 | batch 200/510 [I 2022-10-27 22:12:45] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:30 [I 2022-10-27 22:12:45] ...trap/engines/engine.py.326: process: 0.29245 | load: 0.00015 [I 2022-10-27 22:12:48] ...trap/engines/engine.py.321: eval: epoch 90 | batch 210/510 [I 2022-10-27 22:12:48] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:27 [I 2022-10-27 22:12:48] ...trap/engines/engine.py.326: process: 0.29205 | load: 0.00027 [I 2022-10-27 22:12:51] ...trap/engines/engine.py.321: eval: epoch 90 | batch 220/510 [I 2022-10-27 22:12:51] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:24 [I 2022-10-27 22:12:51] ...trap/engines/engine.py.326: process: 0.29196 | load: 0.00017 [I 2022-10-27 22:12:54] ...trap/engines/engine.py.321: eval: epoch 90 | batch 230/510 [I 2022-10-27 22:12:54] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:21 [I 2022-10-27 22:12:54] ...trap/engines/engine.py.326: process: 0.29062 | load: 0.00016 [I 2022-10-27 22:12:57] ...trap/engines/engine.py.321: eval: epoch 90 | batch 240/510 [I 2022-10-27 22:12:57] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-27 22:12:57] ...trap/engines/engine.py.326: process: 0.29158 | load: 0.00016 [I 2022-10-27 22:13:00] ...trap/engines/engine.py.321: eval: epoch 90 | batch 250/510 [I 2022-10-27 22:13:00] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-27 22:13:00] ...trap/engines/engine.py.326: process: 0.29204 | load: 0.00015 [I 2022-10-27 22:13:02] ...trap/engines/engine.py.321: eval: epoch 90 | batch 260/510 [I 2022-10-27 22:13:02] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-27 22:13:02] ...trap/engines/engine.py.326: process: 0.28619 | load: 0.00015 [I 2022-10-27 22:13:05] ...trap/engines/engine.py.321: eval: epoch 90 | batch 270/510 [I 2022-10-27 22:13:05] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-27 22:13:05] ...trap/engines/engine.py.326: process: 0.28597 | load: 0.00015 [I 2022-10-27 22:13:08] ...trap/engines/engine.py.321: eval: epoch 90 | batch 280/510 [I 2022-10-27 22:13:08] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-27 22:13:08] ...trap/engines/engine.py.326: process: 0.28742 | load: 0.00015 [I 2022-10-27 22:13:11] ...trap/engines/engine.py.321: eval: epoch 90 | batch 290/510 [I 2022-10-27 22:13:11] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-27 22:13:11] ...trap/engines/engine.py.326: process: 0.29142 | load: 0.00020 [I 2022-10-27 22:13:14] ...trap/engines/engine.py.321: eval: epoch 90 | batch 300/510 [I 2022-10-27 22:13:14] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-27 22:13:14] ...trap/engines/engine.py.326: process: 0.28917 | load: 0.00015 [I 2022-10-27 22:13:17] ...trap/engines/engine.py.321: eval: epoch 90 | batch 310/510 [I 2022-10-27 22:13:17] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 22:13:17] ...trap/engines/engine.py.326: process: 0.28853 | load: 0.00015 [I 2022-10-27 22:13:20] ...trap/engines/engine.py.321: eval: epoch 90 | batch 320/510 [I 2022-10-27 22:13:20] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:55 [I 2022-10-27 22:13:20] ...trap/engines/engine.py.326: process: 0.29140 | load: 0.00015 [I 2022-10-27 22:13:23] ...trap/engines/engine.py.321: eval: epoch 90 | batch 330/510 [I 2022-10-27 22:13:23] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-27 22:13:23] ...trap/engines/engine.py.326: process: 0.28363 | load: 0.00014 [I 2022-10-27 22:13:26] ...trap/engines/engine.py.321: eval: epoch 90 | batch 340/510 [I 2022-10-27 22:13:26] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-27 22:13:26] ...trap/engines/engine.py.326: process: 0.28862 | load: 0.00020 [I 2022-10-27 22:13:28] ...trap/engines/engine.py.321: eval: epoch 90 | batch 350/510 [I 2022-10-27 22:13:28] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 22:13:28] ...trap/engines/engine.py.326: process: 0.28837 | load: 0.00026 [I 2022-10-27 22:13:31] ...trap/engines/engine.py.321: eval: epoch 90 | batch 360/510 [I 2022-10-27 22:13:31] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-27 22:13:31] ...trap/engines/engine.py.326: process: 0.28435 | load: 0.00015 [I 2022-10-27 22:13:34] ...trap/engines/engine.py.321: eval: epoch 90 | batch 370/510 [I 2022-10-27 22:13:34] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-27 22:13:34] ...trap/engines/engine.py.326: process: 0.29120 | load: 0.00015 [I 2022-10-27 22:13:37] ...trap/engines/engine.py.321: eval: epoch 90 | batch 380/510 [I 2022-10-27 22:13:37] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:38 [I 2022-10-27 22:13:37] ...trap/engines/engine.py.326: process: 0.30630 | load: 0.00015 [I 2022-10-27 22:13:40] ...trap/engines/engine.py.321: eval: epoch 90 | batch 390/510 [I 2022-10-27 22:13:40] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-27 22:13:40] ...trap/engines/engine.py.326: process: 0.28792 | load: 0.00019 [I 2022-10-27 22:13:43] ...trap/engines/engine.py.321: eval: epoch 90 | batch 400/510 [I 2022-10-27 22:13:43] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:32 [I 2022-10-27 22:13:43] ...trap/engines/engine.py.326: process: 0.28505 | load: 0.00023 [I 2022-10-27 22:13:46] ...trap/engines/engine.py.321: eval: epoch 90 | batch 410/510 [I 2022-10-27 22:13:46] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 22:13:46] ...trap/engines/engine.py.326: process: 0.28785 | load: 0.00016 [I 2022-10-27 22:13:49] ...trap/engines/engine.py.321: eval: epoch 90 | batch 420/510 [I 2022-10-27 22:13:49] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-27 22:13:49] ...trap/engines/engine.py.326: process: 0.29118 | load: 0.00015 [I 2022-10-27 22:13:52] ...trap/engines/engine.py.321: eval: epoch 90 | batch 430/510 [I 2022-10-27 22:13:52] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:22 [I 2022-10-27 22:13:52] ...trap/engines/engine.py.326: process: 0.28201 | load: 0.00016 [I 2022-10-27 22:13:55] ...trap/engines/engine.py.321: eval: epoch 90 | batch 440/510 [I 2022-10-27 22:13:55] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 22:13:55] ...trap/engines/engine.py.326: process: 0.29115 | load: 0.00015 [I 2022-10-27 22:13:57] ...trap/engines/engine.py.321: eval: epoch 90 | batch 450/510 [I 2022-10-27 22:13:57] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 22:13:57] ...trap/engines/engine.py.326: process: 0.28455 | load: 0.00015 [I 2022-10-27 22:14:00] ...trap/engines/engine.py.321: eval: epoch 90 | batch 460/510 [I 2022-10-27 22:14:00] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-27 22:14:00] ...trap/engines/engine.py.326: process: 0.28991 | load: 0.00014 [I 2022-10-27 22:14:03] ...trap/engines/engine.py.321: eval: epoch 90 | batch 470/510 [I 2022-10-27 22:14:03] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-27 22:14:03] ...trap/engines/engine.py.326: process: 0.28846 | load: 0.00016 [I 2022-10-27 22:14:06] ...trap/engines/engine.py.321: eval: epoch 90 | batch 480/510 [I 2022-10-27 22:14:06] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-27 22:14:06] ...trap/engines/engine.py.326: process: 0.28841 | load: 0.00015 [I 2022-10-27 22:14:09] ...trap/engines/engine.py.321: eval: epoch 90 | batch 490/510 [I 2022-10-27 22:14:09] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-27 22:14:09] ...trap/engines/engine.py.326: process: 0.28992 | load: 0.00017 [I 2022-10-27 22:14:12] ...trap/engines/engine.py.321: eval: epoch 90 | batch 500/510 [I 2022-10-27 22:14:12] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 22:14:12] ...trap/engines/engine.py.326: process: 0.27940 | load: 0.00027 [I 2022-10-27 22:14:15] ...trap/engines/engine.py.321: eval: epoch 90 | batch 510/510 [I 2022-10-27 22:14:15] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 22:14:15] ...trap/engines/engine.py.326: process: 0.28360 | load: 0.00011 [S 2022-10-27 22:14:15] ...trap/engines/engine.py.132: eval_epoch.epoch: 90 [I 2022-10-27 22:14:15] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.733600 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906400 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934100 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.729000 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904400 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932500 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015435 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010938 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005629 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013557 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009243 [S 2022-10-27 22:14:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006888 [I 2022-10-27 22:14:21] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 22:14:21] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 22:14:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 22:14:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 22:14:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 22:14:26] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 22:14:27] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 22:14:33] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 22:14:33] ...trap/engines/engine.py.163: Training model on trainset for epoch 91 [I 2022-10-27 22:14:33] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 22:14:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 0/2304 [I 2022-10-27 22:14:35] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:04 [I 2022-10-27 22:14:35] ...trap/engines/engine.py.229: process: 2.76253 | load: 2.11186 [I 2022-10-27 22:14:35] ...trap/engines/engine.py.230: loss: 0.11693 [I 2022-10-27 22:14:35] ...optimizers/trijoint.py.151: total_norm: 13.601254 [I 2022-10-27 22:14:41] ...trap/engines/engine.py.224: train: epoch 91 | batch 10/2304 [I 2022-10-27 22:14:41] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:27 [I 2022-10-27 22:14:41] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00380 [I 2022-10-27 22:14:41] ...trap/engines/engine.py.230: loss: 0.10422 [I 2022-10-27 22:14:41] ...optimizers/trijoint.py.151: total_norm: 14.328856 [I 2022-10-27 22:14:47] ...trap/engines/engine.py.224: train: epoch 91 | batch 20/2304 [I 2022-10-27 22:14:47] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:13 [I 2022-10-27 22:14:47] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00375 [I 2022-10-27 22:14:47] ...trap/engines/engine.py.230: loss: 0.09353 [I 2022-10-27 22:14:47] ...optimizers/trijoint.py.151: total_norm: 23.528839 [I 2022-10-27 22:14:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 30/2304 [I 2022-10-27 22:14:52] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:12 [I 2022-10-27 22:14:52] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00328 [I 2022-10-27 22:14:52] ...trap/engines/engine.py.230: loss: 0.12748 [I 2022-10-27 22:14:52] ...optimizers/trijoint.py.151: total_norm: 19.665796 [I 2022-10-27 22:14:58] ...trap/engines/engine.py.224: train: epoch 91 | batch 40/2304 [I 2022-10-27 22:14:58] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:05 [I 2022-10-27 22:14:58] ...trap/engines/engine.py.229: process: 0.57797 | load: 0.00282 [I 2022-10-27 22:14:58] ...trap/engines/engine.py.230: loss: 0.14054 [I 2022-10-27 22:14:58] ...optimizers/trijoint.py.151: total_norm: 20.545740 [I 2022-10-27 22:15:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 50/2304 [I 2022-10-27 22:15:03] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:07 [I 2022-10-27 22:15:03] ...trap/engines/engine.py.229: process: 0.54670 | load: 0.00291 [I 2022-10-27 22:15:03] ...trap/engines/engine.py.230: loss: 0.15043 [I 2022-10-27 22:15:03] ...optimizers/trijoint.py.151: total_norm: 27.203432 [I 2022-10-27 22:15:09] ...trap/engines/engine.py.224: train: epoch 91 | batch 60/2304 [I 2022-10-27 22:15:09] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:58 [I 2022-10-27 22:15:09] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00280 [I 2022-10-27 22:15:09] ...trap/engines/engine.py.230: loss: 0.16337 [I 2022-10-27 22:15:09] ...optimizers/trijoint.py.151: total_norm: 13.672613 [I 2022-10-27 22:15:15] ...trap/engines/engine.py.224: train: epoch 91 | batch 70/2304 [I 2022-10-27 22:15:15] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:51 [I 2022-10-27 22:15:15] ...trap/engines/engine.py.229: process: 0.56348 | load: 0.00279 [I 2022-10-27 22:15:15] ...trap/engines/engine.py.230: loss: 0.10708 [I 2022-10-27 22:15:15] ...optimizers/trijoint.py.151: total_norm: 22.252512 [I 2022-10-27 22:15:20] ...trap/engines/engine.py.224: train: epoch 91 | batch 80/2304 [I 2022-10-27 22:15:20] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:45 [I 2022-10-27 22:15:20] ...trap/engines/engine.py.229: process: 0.56537 | load: 0.00298 [I 2022-10-27 22:15:20] ...trap/engines/engine.py.230: loss: 0.09581 [I 2022-10-27 22:15:20] ...optimizers/trijoint.py.151: total_norm: 17.469135 [I 2022-10-27 22:15:26] ...trap/engines/engine.py.224: train: epoch 91 | batch 90/2304 [I 2022-10-27 22:15:26] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:37 [I 2022-10-27 22:15:26] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00263 [I 2022-10-27 22:15:26] ...trap/engines/engine.py.230: loss: 0.11125 [I 2022-10-27 22:15:26] ...optimizers/trijoint.py.151: total_norm: 21.728886 [I 2022-10-27 22:15:31] ...trap/engines/engine.py.224: train: epoch 91 | batch 100/2304 [I 2022-10-27 22:15:31] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:32 [I 2022-10-27 22:15:31] ...trap/engines/engine.py.229: process: 0.56358 | load: 0.00256 [I 2022-10-27 22:15:31] ...trap/engines/engine.py.230: loss: 0.10168 [I 2022-10-27 22:15:31] ...optimizers/trijoint.py.151: total_norm: 10.814444 [I 2022-10-27 22:15:37] ...trap/engines/engine.py.224: train: epoch 91 | batch 110/2304 [I 2022-10-27 22:15:37] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:27 [I 2022-10-27 22:15:37] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00252 [I 2022-10-27 22:15:37] ...trap/engines/engine.py.230: loss: 0.10176 [I 2022-10-27 22:15:37] ...optimizers/trijoint.py.151: total_norm: 13.452977 [I 2022-10-27 22:15:43] ...trap/engines/engine.py.224: train: epoch 91 | batch 120/2304 [I 2022-10-27 22:15:43] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:32 [I 2022-10-27 22:15:43] ...trap/engines/engine.py.229: process: 0.57284 | load: 0.00262 [I 2022-10-27 22:15:43] ...trap/engines/engine.py.230: loss: 0.11838 [I 2022-10-27 22:15:43] ...optimizers/trijoint.py.151: total_norm: 11.716978 [I 2022-10-27 22:15:48] ...trap/engines/engine.py.224: train: epoch 91 | batch 130/2304 [I 2022-10-27 22:15:48] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:19 [I 2022-10-27 22:15:48] ...trap/engines/engine.py.229: process: 0.56432 | load: 0.00262 [I 2022-10-27 22:15:48] ...trap/engines/engine.py.230: loss: 0.14476 [I 2022-10-27 22:15:48] ...optimizers/trijoint.py.151: total_norm: 8.607262 [I 2022-10-27 22:15:54] ...trap/engines/engine.py.224: train: epoch 91 | batch 140/2304 [I 2022-10-27 22:15:54] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:07 [I 2022-10-27 22:15:54] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00249 [I 2022-10-27 22:15:54] ...trap/engines/engine.py.230: loss: 0.14557 [I 2022-10-27 22:15:54] ...optimizers/trijoint.py.151: total_norm: 19.314890 [I 2022-10-27 22:15:59] ...trap/engines/engine.py.224: train: epoch 91 | batch 150/2304 [I 2022-10-27 22:15:59] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:55 [I 2022-10-27 22:15:59] ...trap/engines/engine.py.229: process: 0.55635 | load: 0.00258 [I 2022-10-27 22:15:59] ...trap/engines/engine.py.230: loss: 0.11226 [I 2022-10-27 22:15:59] ...optimizers/trijoint.py.151: total_norm: 8.056046 [I 2022-10-27 22:16:05] ...trap/engines/engine.py.224: train: epoch 91 | batch 160/2304 [I 2022-10-27 22:16:05] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:02 [I 2022-10-27 22:16:05] ...trap/engines/engine.py.229: process: 0.56551 | load: 0.00247 [I 2022-10-27 22:16:05] ...trap/engines/engine.py.230: loss: 0.12041 [I 2022-10-27 22:16:05] ...optimizers/trijoint.py.151: total_norm: 8.626280 [I 2022-10-27 22:16:11] ...trap/engines/engine.py.224: train: epoch 91 | batch 170/2304 [I 2022-10-27 22:16:11] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:20:08 [I 2022-10-27 22:16:11] ...trap/engines/engine.py.229: process: 0.58136 | load: 0.00252 [I 2022-10-27 22:16:11] ...trap/engines/engine.py.230: loss: 0.13193 [I 2022-10-27 22:16:11] ...optimizers/trijoint.py.151: total_norm: 22.326355 [I 2022-10-27 22:16:16] ...trap/engines/engine.py.224: train: epoch 91 | batch 180/2304 [I 2022-10-27 22:16:16] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:50 [I 2022-10-27 22:16:16] ...trap/engines/engine.py.229: process: 0.56896 | load: 0.00266 [I 2022-10-27 22:16:16] ...trap/engines/engine.py.230: loss: 0.14419 [I 2022-10-27 22:16:16] ...optimizers/trijoint.py.151: total_norm: 14.122265 [I 2022-10-27 22:16:22] ...trap/engines/engine.py.224: train: epoch 91 | batch 190/2304 [I 2022-10-27 22:16:22] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:30 [I 2022-10-27 22:16:22] ...trap/engines/engine.py.229: process: 0.54961 | load: 0.00246 [I 2022-10-27 22:16:22] ...trap/engines/engine.py.230: loss: 0.13103 [I 2022-10-27 22:16:22] ...optimizers/trijoint.py.151: total_norm: 16.499052 [I 2022-10-27 22:16:27] ...trap/engines/engine.py.224: train: epoch 91 | batch 200/2304 [I 2022-10-27 22:16:27] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:36 [I 2022-10-27 22:16:27] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00245 [I 2022-10-27 22:16:27] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-27 22:16:27] ...optimizers/trijoint.py.151: total_norm: 12.664322 [I 2022-10-27 22:16:33] ...trap/engines/engine.py.224: train: epoch 91 | batch 210/2304 [I 2022-10-27 22:16:33] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:46 [I 2022-10-27 22:16:33] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00252 [I 2022-10-27 22:16:33] ...trap/engines/engine.py.230: loss: 0.07654 [I 2022-10-27 22:16:33] ...optimizers/trijoint.py.151: total_norm: 9.753610 [I 2022-10-27 22:16:39] ...trap/engines/engine.py.224: train: epoch 91 | batch 220/2304 [I 2022-10-27 22:16:39] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:33 [I 2022-10-27 22:16:39] ...trap/engines/engine.py.229: process: 0.56418 | load: 0.00251 [I 2022-10-27 22:16:39] ...trap/engines/engine.py.230: loss: 0.21168 [I 2022-10-27 22:16:39] ...optimizers/trijoint.py.151: total_norm: 22.919725 [I 2022-10-27 22:16:44] ...trap/engines/engine.py.224: train: epoch 91 | batch 230/2304 [I 2022-10-27 22:16:44] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:49 [I 2022-10-27 22:16:44] ...trap/engines/engine.py.229: process: 0.56403 | load: 0.00250 [I 2022-10-27 22:16:44] ...trap/engines/engine.py.230: loss: 0.14614 [I 2022-10-27 22:16:44] ...optimizers/trijoint.py.151: total_norm: 28.872089 [I 2022-10-27 22:16:50] ...trap/engines/engine.py.224: train: epoch 91 | batch 240/2304 [I 2022-10-27 22:16:50] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:27 [I 2022-10-27 22:16:50] ...trap/engines/engine.py.229: process: 0.56159 | load: 0.00243 [I 2022-10-27 22:16:50] ...trap/engines/engine.py.230: loss: 0.14097 [I 2022-10-27 22:16:50] ...optimizers/trijoint.py.151: total_norm: 27.923183 [I 2022-10-27 22:16:56] ...trap/engines/engine.py.224: train: epoch 91 | batch 250/2304 [I 2022-10-27 22:16:56] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:07 [I 2022-10-27 22:16:56] ...trap/engines/engine.py.229: process: 0.55865 | load: 0.00266 [I 2022-10-27 22:16:56] ...trap/engines/engine.py.230: loss: 0.12499 [I 2022-10-27 22:16:56] ...optimizers/trijoint.py.151: total_norm: 21.078283 [I 2022-10-27 22:17:01] ...trap/engines/engine.py.224: train: epoch 91 | batch 260/2304 [I 2022-10-27 22:17:01] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:13 [I 2022-10-27 22:17:01] ...trap/engines/engine.py.229: process: 0.56788 | load: 0.00247 [I 2022-10-27 22:17:01] ...trap/engines/engine.py.230: loss: 0.17266 [I 2022-10-27 22:17:01] ...optimizers/trijoint.py.151: total_norm: 13.272022 [I 2022-10-27 22:17:07] ...trap/engines/engine.py.224: train: epoch 91 | batch 270/2304 [I 2022-10-27 22:17:07] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:19:02 [I 2022-10-27 22:17:07] ...trap/engines/engine.py.229: process: 0.54617 | load: 0.00246 [I 2022-10-27 22:17:07] ...trap/engines/engine.py.230: loss: 0.16174 [I 2022-10-27 22:17:07] ...optimizers/trijoint.py.151: total_norm: 17.396854 [I 2022-10-27 22:17:13] ...trap/engines/engine.py.224: train: epoch 91 | batch 280/2304 [I 2022-10-27 22:17:13] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:18:51 [I 2022-10-27 22:17:13] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00259 [I 2022-10-27 22:17:13] ...trap/engines/engine.py.230: loss: 0.10492 [I 2022-10-27 22:17:13] ...optimizers/trijoint.py.151: total_norm: 14.342844 [I 2022-10-27 22:17:18] ...trap/engines/engine.py.224: train: epoch 91 | batch 290/2304 [I 2022-10-27 22:17:18] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:19:21 [I 2022-10-27 22:17:18] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00247 [I 2022-10-27 22:17:18] ...trap/engines/engine.py.230: loss: 0.12643 [I 2022-10-27 22:17:18] ...optimizers/trijoint.py.151: total_norm: 12.101970 [I 2022-10-27 22:17:24] ...trap/engines/engine.py.224: train: epoch 91 | batch 300/2304 [I 2022-10-27 22:17:24] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:35 [I 2022-10-27 22:17:24] ...trap/engines/engine.py.229: process: 0.54645 | load: 0.00271 [I 2022-10-27 22:17:24] ...trap/engines/engine.py.230: loss: 0.16488 [I 2022-10-27 22:17:24] ...optimizers/trijoint.py.151: total_norm: 11.512982 [I 2022-10-27 22:17:30] ...trap/engines/engine.py.224: train: epoch 91 | batch 310/2304 [I 2022-10-27 22:17:30] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:43 [I 2022-10-27 22:17:30] ...trap/engines/engine.py.229: process: 0.57908 | load: 0.00251 [I 2022-10-27 22:17:30] ...trap/engines/engine.py.230: loss: 0.10376 [I 2022-10-27 22:17:30] ...optimizers/trijoint.py.151: total_norm: 17.402096 [I 2022-10-27 22:17:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 320/2304 [I 2022-10-27 22:17:35] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:36 [I 2022-10-27 22:17:35] ...trap/engines/engine.py.229: process: 0.56588 | load: 0.00243 [I 2022-10-27 22:17:35] ...trap/engines/engine.py.230: loss: 0.14858 [I 2022-10-27 22:17:35] ...optimizers/trijoint.py.151: total_norm: 14.402934 [I 2022-10-27 22:17:41] ...trap/engines/engine.py.224: train: epoch 91 | batch 330/2304 [I 2022-10-27 22:17:41] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:18:30 [I 2022-10-27 22:17:41] ...trap/engines/engine.py.229: process: 0.56868 | load: 0.00252 [I 2022-10-27 22:17:41] ...trap/engines/engine.py.230: loss: 0.13014 [I 2022-10-27 22:17:41] ...optimizers/trijoint.py.151: total_norm: 17.335400 [I 2022-10-27 22:17:46] ...trap/engines/engine.py.224: train: epoch 91 | batch 340/2304 [I 2022-10-27 22:17:46] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:26 [I 2022-10-27 22:17:46] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00243 [I 2022-10-27 22:17:46] ...trap/engines/engine.py.230: loss: 0.13427 [I 2022-10-27 22:17:46] ...optimizers/trijoint.py.151: total_norm: 14.535347 [I 2022-10-27 22:17:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 350/2304 [I 2022-10-27 22:17:52] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:21 [I 2022-10-27 22:17:52] ...trap/engines/engine.py.229: process: 0.56452 | load: 0.00252 [I 2022-10-27 22:17:52] ...trap/engines/engine.py.230: loss: 0.23252 [I 2022-10-27 22:17:52] ...optimizers/trijoint.py.151: total_norm: 9.736721 [I 2022-10-27 22:17:58] ...trap/engines/engine.py.224: train: epoch 91 | batch 360/2304 [I 2022-10-27 22:17:58] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:18:32 [I 2022-10-27 22:17:58] ...trap/engines/engine.py.229: process: 0.57671 | load: 0.00254 [I 2022-10-27 22:17:58] ...trap/engines/engine.py.230: loss: 0.11817 [I 2022-10-27 22:17:58] ...optimizers/trijoint.py.151: total_norm: 14.722138 [I 2022-10-27 22:18:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 370/2304 [I 2022-10-27 22:18:03] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:17:59 [I 2022-10-27 22:18:03] ...trap/engines/engine.py.229: process: 0.54824 | load: 0.00254 [I 2022-10-27 22:18:03] ...trap/engines/engine.py.230: loss: 0.12329 [I 2022-10-27 22:18:03] ...optimizers/trijoint.py.151: total_norm: 12.241325 [I 2022-10-27 22:18:09] ...trap/engines/engine.py.224: train: epoch 91 | batch 380/2304 [I 2022-10-27 22:18:09] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:17:45 [I 2022-10-27 22:18:09] ...trap/engines/engine.py.229: process: 0.55888 | load: 0.00253 [I 2022-10-27 22:18:09] ...trap/engines/engine.py.230: loss: 0.12886 [I 2022-10-27 22:18:09] ...optimizers/trijoint.py.151: total_norm: 16.649752 [I 2022-10-27 22:18:15] ...trap/engines/engine.py.224: train: epoch 91 | batch 390/2304 [I 2022-10-27 22:18:15] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:43 [I 2022-10-27 22:18:15] ...trap/engines/engine.py.229: process: 0.55453 | load: 0.00298 [I 2022-10-27 22:18:15] ...trap/engines/engine.py.230: loss: 0.13291 [I 2022-10-27 22:18:15] ...optimizers/trijoint.py.151: total_norm: 14.591295 [I 2022-10-27 22:18:20] ...trap/engines/engine.py.224: train: epoch 91 | batch 400/2304 [I 2022-10-27 22:18:20] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:47 [I 2022-10-27 22:18:20] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00249 [I 2022-10-27 22:18:20] ...trap/engines/engine.py.230: loss: 0.13360 [I 2022-10-27 22:18:20] ...optimizers/trijoint.py.151: total_norm: 10.346758 [I 2022-10-27 22:18:26] ...trap/engines/engine.py.224: train: epoch 91 | batch 410/2304 [I 2022-10-27 22:18:26] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:17:34 [I 2022-10-27 22:18:26] ...trap/engines/engine.py.229: process: 0.54678 | load: 0.00246 [I 2022-10-27 22:18:26] ...trap/engines/engine.py.230: loss: 0.18223 [I 2022-10-27 22:18:26] ...optimizers/trijoint.py.151: total_norm: 14.126375 [I 2022-10-27 22:18:31] ...trap/engines/engine.py.224: train: epoch 91 | batch 420/2304 [I 2022-10-27 22:18:31] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:52 [I 2022-10-27 22:18:31] ...trap/engines/engine.py.229: process: 0.56396 | load: 0.00260 [I 2022-10-27 22:18:31] ...trap/engines/engine.py.230: loss: 0.23635 [I 2022-10-27 22:18:31] ...optimizers/trijoint.py.151: total_norm: 50.799984 [I 2022-10-27 22:18:37] ...trap/engines/engine.py.224: train: epoch 91 | batch 430/2304 [I 2022-10-27 22:18:37] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:27 [I 2022-10-27 22:18:37] ...trap/engines/engine.py.229: process: 0.57099 | load: 0.00245 [I 2022-10-27 22:18:37] ...trap/engines/engine.py.230: loss: 0.13186 [I 2022-10-27 22:18:37] ...optimizers/trijoint.py.151: total_norm: 13.780550 [I 2022-10-27 22:18:43] ...trap/engines/engine.py.224: train: epoch 91 | batch 440/2304 [I 2022-10-27 22:18:43] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:17:23 [I 2022-10-27 22:18:43] ...trap/engines/engine.py.229: process: 0.56126 | load: 0.00246 [I 2022-10-27 22:18:43] ...trap/engines/engine.py.230: loss: 0.15769 [I 2022-10-27 22:18:43] ...optimizers/trijoint.py.151: total_norm: 26.264931 [I 2022-10-27 22:18:48] ...trap/engines/engine.py.224: train: epoch 91 | batch 450/2304 [I 2022-10-27 22:18:48] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:16 [I 2022-10-27 22:18:48] ...trap/engines/engine.py.229: process: 0.56778 | load: 0.00243 [I 2022-10-27 22:18:48] ...trap/engines/engine.py.230: loss: 0.11778 [I 2022-10-27 22:18:48] ...optimizers/trijoint.py.151: total_norm: 12.331326 [I 2022-10-27 22:18:54] ...trap/engines/engine.py.224: train: epoch 91 | batch 460/2304 [I 2022-10-27 22:18:54] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:17:04 [I 2022-10-27 22:18:54] ...trap/engines/engine.py.229: process: 0.55034 | load: 0.00243 [I 2022-10-27 22:18:54] ...trap/engines/engine.py.230: loss: 0.15722 [I 2022-10-27 22:18:54] ...optimizers/trijoint.py.151: total_norm: 18.015623 [I 2022-10-27 22:18:59] ...trap/engines/engine.py.224: train: epoch 91 | batch 470/2304 [I 2022-10-27 22:18:59] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:04 [I 2022-10-27 22:18:59] ...trap/engines/engine.py.229: process: 0.56330 | load: 0.00250 [I 2022-10-27 22:18:59] ...trap/engines/engine.py.230: loss: 0.12848 [I 2022-10-27 22:18:59] ...optimizers/trijoint.py.151: total_norm: 18.653095 [I 2022-10-27 22:19:05] ...trap/engines/engine.py.224: train: epoch 91 | batch 480/2304 [I 2022-10-27 22:19:05] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:17:46 [I 2022-10-27 22:19:05] ...trap/engines/engine.py.229: process: 0.69642 | load: 0.00255 [I 2022-10-27 22:19:05] ...trap/engines/engine.py.230: loss: 0.09963 [I 2022-10-27 22:19:05] ...optimizers/trijoint.py.151: total_norm: 16.019159 [I 2022-10-27 22:19:11] ...trap/engines/engine.py.224: train: epoch 91 | batch 490/2304 [I 2022-10-27 22:19:11] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:17:01 [I 2022-10-27 22:19:11] ...trap/engines/engine.py.229: process: 0.55556 | load: 0.00246 [I 2022-10-27 22:19:11] ...trap/engines/engine.py.230: loss: 0.09798 [I 2022-10-27 22:19:11] ...optimizers/trijoint.py.151: total_norm: 8.943026 [I 2022-10-27 22:19:16] ...trap/engines/engine.py.224: train: epoch 91 | batch 500/2304 [I 2022-10-27 22:19:16] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:49 [I 2022-10-27 22:19:16] ...trap/engines/engine.py.229: process: 0.54690 | load: 0.00246 [I 2022-10-27 22:19:16] ...trap/engines/engine.py.230: loss: 0.09767 [I 2022-10-27 22:19:16] ...optimizers/trijoint.py.151: total_norm: 9.973239 [I 2022-10-27 22:19:22] ...trap/engines/engine.py.224: train: epoch 91 | batch 510/2304 [I 2022-10-27 22:19:22] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:38 [I 2022-10-27 22:19:22] ...trap/engines/engine.py.229: process: 0.55177 | load: 0.00249 [I 2022-10-27 22:19:22] ...trap/engines/engine.py.230: loss: 0.17210 [I 2022-10-27 22:19:22] ...optimizers/trijoint.py.151: total_norm: 17.873343 [I 2022-10-27 22:19:28] ...trap/engines/engine.py.224: train: epoch 91 | batch 520/2304 [I 2022-10-27 22:19:28] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:42 [I 2022-10-27 22:19:28] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00244 [I 2022-10-27 22:19:28] ...trap/engines/engine.py.230: loss: 0.11798 [I 2022-10-27 22:19:28] ...optimizers/trijoint.py.151: total_norm: 33.944889 [I 2022-10-27 22:19:33] ...trap/engines/engine.py.224: train: epoch 91 | batch 530/2304 [I 2022-10-27 22:19:33] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:32 [I 2022-10-27 22:19:33] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00245 [I 2022-10-27 22:19:33] ...trap/engines/engine.py.230: loss: 0.10138 [I 2022-10-27 22:19:33] ...optimizers/trijoint.py.151: total_norm: 11.083773 [I 2022-10-27 22:19:39] ...trap/engines/engine.py.224: train: epoch 91 | batch 540/2304 [I 2022-10-27 22:19:39] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:16:26 [I 2022-10-27 22:19:39] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00254 [I 2022-10-27 22:19:39] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-27 22:19:39] ...optimizers/trijoint.py.151: total_norm: 18.172920 [I 2022-10-27 22:19:45] ...trap/engines/engine.py.224: train: epoch 91 | batch 550/2304 [I 2022-10-27 22:19:45] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:46 [I 2022-10-27 22:19:45] ...trap/engines/engine.py.229: process: 0.57777 | load: 0.00247 [I 2022-10-27 22:19:45] ...trap/engines/engine.py.230: loss: 0.13848 [I 2022-10-27 22:19:45] ...optimizers/trijoint.py.151: total_norm: 34.584484 [I 2022-10-27 22:19:50] ...trap/engines/engine.py.224: train: epoch 91 | batch 560/2304 [I 2022-10-27 22:19:50] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:14 [I 2022-10-27 22:19:50] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00245 [I 2022-10-27 22:19:50] ...trap/engines/engine.py.230: loss: 0.10708 [I 2022-10-27 22:19:50] ...optimizers/trijoint.py.151: total_norm: 12.268085 [I 2022-10-27 22:19:56] ...trap/engines/engine.py.224: train: epoch 91 | batch 570/2304 [I 2022-10-27 22:19:56] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:16:04 [I 2022-10-27 22:19:56] ...trap/engines/engine.py.229: process: 0.55426 | load: 0.00248 [I 2022-10-27 22:19:56] ...trap/engines/engine.py.230: loss: 0.10781 [I 2022-10-27 22:19:56] ...optimizers/trijoint.py.151: total_norm: 13.997659 [I 2022-10-27 22:20:01] ...trap/engines/engine.py.224: train: epoch 91 | batch 580/2304 [I 2022-10-27 22:20:01] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:15:57 [I 2022-10-27 22:20:01] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00247 [I 2022-10-27 22:20:01] ...trap/engines/engine.py.230: loss: 0.10679 [I 2022-10-27 22:20:01] ...optimizers/trijoint.py.151: total_norm: 9.058400 [I 2022-10-27 22:20:07] ...trap/engines/engine.py.224: train: epoch 91 | batch 590/2304 [I 2022-10-27 22:20:07] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:15:53 [I 2022-10-27 22:20:07] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00243 [I 2022-10-27 22:20:07] ...trap/engines/engine.py.230: loss: 0.09274 [I 2022-10-27 22:20:07] ...optimizers/trijoint.py.151: total_norm: 10.554614 [I 2022-10-27 22:20:12] ...trap/engines/engine.py.224: train: epoch 91 | batch 600/2304 [I 2022-10-27 22:20:12] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:44 [I 2022-10-27 22:20:12] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00244 [I 2022-10-27 22:20:12] ...trap/engines/engine.py.230: loss: 0.10637 [I 2022-10-27 22:20:12] ...optimizers/trijoint.py.151: total_norm: 17.740063 [I 2022-10-27 22:20:18] ...trap/engines/engine.py.224: train: epoch 91 | batch 610/2304 [I 2022-10-27 22:20:18] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:16:24 [I 2022-10-27 22:20:18] ...trap/engines/engine.py.229: process: 0.69355 | load: 0.00992 [I 2022-10-27 22:20:18] ...trap/engines/engine.py.230: loss: 0.08737 [I 2022-10-27 22:20:18] ...optimizers/trijoint.py.151: total_norm: 14.900377 [I 2022-10-27 22:20:24] ...trap/engines/engine.py.224: train: epoch 91 | batch 620/2304 [I 2022-10-27 22:20:24] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:15:39 [I 2022-10-27 22:20:24] ...trap/engines/engine.py.229: process: 0.55576 | load: 0.00243 [I 2022-10-27 22:20:24] ...trap/engines/engine.py.230: loss: 0.13469 [I 2022-10-27 22:20:24] ...optimizers/trijoint.py.151: total_norm: 14.222740 [I 2022-10-27 22:20:29] ...trap/engines/engine.py.224: train: epoch 91 | batch 630/2304 [I 2022-10-27 22:20:29] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:26 [I 2022-10-27 22:20:29] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00242 [I 2022-10-27 22:20:29] ...trap/engines/engine.py.230: loss: 0.11147 [I 2022-10-27 22:20:29] ...optimizers/trijoint.py.151: total_norm: 25.107103 [I 2022-10-27 22:20:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 640/2304 [I 2022-10-27 22:20:35] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:15:24 [I 2022-10-27 22:20:35] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00241 [I 2022-10-27 22:20:35] ...trap/engines/engine.py.230: loss: 0.20271 [I 2022-10-27 22:20:35] ...optimizers/trijoint.py.151: total_norm: 14.107007 [I 2022-10-27 22:20:40] ...trap/engines/engine.py.224: train: epoch 91 | batch 650/2304 [I 2022-10-27 22:20:40] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:19 [I 2022-10-27 22:20:40] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00277 [I 2022-10-27 22:20:40] ...trap/engines/engine.py.230: loss: 0.08985 [I 2022-10-27 22:20:40] ...optimizers/trijoint.py.151: total_norm: 15.758238 [I 2022-10-27 22:20:46] ...trap/engines/engine.py.224: train: epoch 91 | batch 660/2304 [I 2022-10-27 22:20:46] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:15:10 [I 2022-10-27 22:20:46] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00257 [I 2022-10-27 22:20:46] ...trap/engines/engine.py.230: loss: 0.17002 [I 2022-10-27 22:20:46] ...optimizers/trijoint.py.151: total_norm: 19.271694 [I 2022-10-27 22:20:51] ...trap/engines/engine.py.224: train: epoch 91 | batch 670/2304 [I 2022-10-27 22:20:51] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:06 [I 2022-10-27 22:20:51] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00245 [I 2022-10-27 22:20:51] ...trap/engines/engine.py.230: loss: 0.10440 [I 2022-10-27 22:20:51] ...optimizers/trijoint.py.151: total_norm: 16.564947 [I 2022-10-27 22:20:57] ...trap/engines/engine.py.224: train: epoch 91 | batch 680/2304 [I 2022-10-27 22:20:57] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:19 [I 2022-10-27 22:20:57] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00243 [I 2022-10-27 22:20:57] ...trap/engines/engine.py.230: loss: 0.15255 [I 2022-10-27 22:20:57] ...optimizers/trijoint.py.151: total_norm: 16.156519 [I 2022-10-27 22:21:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 690/2304 [I 2022-10-27 22:21:03] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:14:58 [I 2022-10-27 22:21:03] ...trap/engines/engine.py.229: process: 0.56382 | load: 0.00248 [I 2022-10-27 22:21:03] ...trap/engines/engine.py.230: loss: 0.09250 [I 2022-10-27 22:21:03] ...optimizers/trijoint.py.151: total_norm: 13.533224 [I 2022-10-27 22:21:08] ...trap/engines/engine.py.224: train: epoch 91 | batch 700/2304 [I 2022-10-27 22:21:08] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:14:59 [I 2022-10-27 22:21:08] ...trap/engines/engine.py.229: process: 0.56574 | load: 0.00246 [I 2022-10-27 22:21:08] ...trap/engines/engine.py.230: loss: 0.09018 [I 2022-10-27 22:21:08] ...optimizers/trijoint.py.151: total_norm: 9.725824 [I 2022-10-27 22:21:14] ...trap/engines/engine.py.224: train: epoch 91 | batch 710/2304 [I 2022-10-27 22:21:14] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:14:51 [I 2022-10-27 22:21:14] ...trap/engines/engine.py.229: process: 0.56270 | load: 0.00244 [I 2022-10-27 22:21:14] ...trap/engines/engine.py.230: loss: 0.17158 [I 2022-10-27 22:21:14] ...optimizers/trijoint.py.151: total_norm: 24.070009 [I 2022-10-27 22:21:20] ...trap/engines/engine.py.224: train: epoch 91 | batch 720/2304 [I 2022-10-27 22:21:20] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:47 [I 2022-10-27 22:21:20] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00243 [I 2022-10-27 22:21:20] ...trap/engines/engine.py.230: loss: 0.16093 [I 2022-10-27 22:21:20] ...optimizers/trijoint.py.151: total_norm: 24.822058 [I 2022-10-27 22:21:25] ...trap/engines/engine.py.224: train: epoch 91 | batch 730/2304 [I 2022-10-27 22:21:25] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:14:37 [I 2022-10-27 22:21:25] ...trap/engines/engine.py.229: process: 0.54800 | load: 0.00248 [I 2022-10-27 22:21:25] ...trap/engines/engine.py.230: loss: 0.12613 [I 2022-10-27 22:21:25] ...optimizers/trijoint.py.151: total_norm: 28.260721 [I 2022-10-27 22:21:31] ...trap/engines/engine.py.224: train: epoch 91 | batch 740/2304 [I 2022-10-27 22:21:31] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:14:49 [I 2022-10-27 22:21:31] ...trap/engines/engine.py.229: process: 0.56407 | load: 0.00246 [I 2022-10-27 22:21:31] ...trap/engines/engine.py.230: loss: 0.16026 [I 2022-10-27 22:21:31] ...optimizers/trijoint.py.151: total_norm: 20.881905 [I 2022-10-27 22:21:37] ...trap/engines/engine.py.224: train: epoch 91 | batch 750/2304 [I 2022-10-27 22:21:37] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:34 [I 2022-10-27 22:21:37] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00248 [I 2022-10-27 22:21:37] ...trap/engines/engine.py.230: loss: 0.24113 [I 2022-10-27 22:21:37] ...optimizers/trijoint.py.151: total_norm: 15.334438 [I 2022-10-27 22:21:42] ...trap/engines/engine.py.224: train: epoch 91 | batch 760/2304 [I 2022-10-27 22:21:42] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:26 [I 2022-10-27 22:21:42] ...trap/engines/engine.py.229: process: 0.57774 | load: 0.00247 [I 2022-10-27 22:21:42] ...trap/engines/engine.py.230: loss: 0.12464 [I 2022-10-27 22:21:42] ...optimizers/trijoint.py.151: total_norm: 28.787132 [I 2022-10-27 22:21:48] ...trap/engines/engine.py.224: train: epoch 91 | batch 770/2304 [I 2022-10-27 22:21:48] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:14:08 [I 2022-10-27 22:21:48] ...trap/engines/engine.py.229: process: 0.53518 | load: 0.00245 [I 2022-10-27 22:21:48] ...trap/engines/engine.py.230: loss: 0.09418 [I 2022-10-27 22:21:48] ...optimizers/trijoint.py.151: total_norm: 7.467180 [I 2022-10-27 22:21:53] ...trap/engines/engine.py.224: train: epoch 91 | batch 780/2304 [I 2022-10-27 22:21:53] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:12 [I 2022-10-27 22:21:53] ...trap/engines/engine.py.229: process: 0.56639 | load: 0.00244 [I 2022-10-27 22:21:53] ...trap/engines/engine.py.230: loss: 0.10623 [I 2022-10-27 22:21:53] ...optimizers/trijoint.py.151: total_norm: 14.553739 [I 2022-10-27 22:21:59] ...trap/engines/engine.py.224: train: epoch 91 | batch 790/2304 [I 2022-10-27 22:21:59] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:14:09 [I 2022-10-27 22:21:59] ...trap/engines/engine.py.229: process: 0.56612 | load: 0.00264 [I 2022-10-27 22:21:59] ...trap/engines/engine.py.230: loss: 0.10135 [I 2022-10-27 22:21:59] ...optimizers/trijoint.py.151: total_norm: 9.549914 [I 2022-10-27 22:22:05] ...trap/engines/engine.py.224: train: epoch 91 | batch 800/2304 [I 2022-10-27 22:22:05] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:14:00 [I 2022-10-27 22:22:05] ...trap/engines/engine.py.229: process: 0.56772 | load: 0.00242 [I 2022-10-27 22:22:05] ...trap/engines/engine.py.230: loss: 0.12006 [I 2022-10-27 22:22:05] ...optimizers/trijoint.py.151: total_norm: 24.424761 [I 2022-10-27 22:22:10] ...trap/engines/engine.py.224: train: epoch 91 | batch 810/2304 [I 2022-10-27 22:22:10] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:14:00 [I 2022-10-27 22:22:10] ...trap/engines/engine.py.229: process: 0.57172 | load: 0.00242 [I 2022-10-27 22:22:10] ...trap/engines/engine.py.230: loss: 0.16498 [I 2022-10-27 22:22:10] ...optimizers/trijoint.py.151: total_norm: 9.856133 [I 2022-10-27 22:22:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 820/2304 [I 2022-10-27 22:22:35] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 1:48:50 [I 2022-10-27 22:22:35] ...trap/engines/engine.py.229: process: 19.76706 | load: 0.00243 [I 2022-10-27 22:22:35] ...trap/engines/engine.py.230: loss: 0.11201 [I 2022-10-27 22:22:35] ...optimizers/trijoint.py.151: total_norm: 12.185187 [I 2022-10-27 22:22:40] ...trap/engines/engine.py.224: train: epoch 91 | batch 830/2304 [I 2022-10-27 22:22:40] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:23:42 [I 2022-10-27 22:22:40] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00251 [I 2022-10-27 22:22:40] ...trap/engines/engine.py.230: loss: 0.11717 [I 2022-10-27 22:22:40] ...optimizers/trijoint.py.151: total_norm: 12.252532 [I 2022-10-27 22:22:46] ...trap/engines/engine.py.224: train: epoch 91 | batch 840/2304 [I 2022-10-27 22:22:46] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:14:35 [I 2022-10-27 22:22:46] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00250 [I 2022-10-27 22:22:46] ...trap/engines/engine.py.230: loss: 0.19646 [I 2022-10-27 22:22:46] ...optimizers/trijoint.py.151: total_norm: 21.668488 [I 2022-10-27 22:22:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 850/2304 [I 2022-10-27 22:22:52] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:40 [I 2022-10-27 22:22:52] ...trap/engines/engine.py.229: process: 0.56691 | load: 0.00249 [I 2022-10-27 22:22:52] ...trap/engines/engine.py.230: loss: 0.14947 [I 2022-10-27 22:22:52] ...optimizers/trijoint.py.151: total_norm: 15.339478 [I 2022-10-27 22:22:57] ...trap/engines/engine.py.224: train: epoch 91 | batch 860/2304 [I 2022-10-27 22:22:57] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:18 [I 2022-10-27 22:22:57] ...trap/engines/engine.py.229: process: 0.54006 | load: 0.00306 [I 2022-10-27 22:22:57] ...trap/engines/engine.py.230: loss: 0.13952 [I 2022-10-27 22:22:57] ...optimizers/trijoint.py.151: total_norm: 14.054871 [I 2022-10-27 22:23:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 870/2304 [I 2022-10-27 22:23:03] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:13:21 [I 2022-10-27 22:23:03] ...trap/engines/engine.py.229: process: 0.56545 | load: 0.00244 [I 2022-10-27 22:23:03] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-27 22:23:03] ...optimizers/trijoint.py.151: total_norm: 9.617239 [I 2022-10-27 22:23:08] ...trap/engines/engine.py.224: train: epoch 91 | batch 880/2304 [I 2022-10-27 22:23:08] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:10 [I 2022-10-27 22:23:08] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00244 [I 2022-10-27 22:23:08] ...trap/engines/engine.py.230: loss: 0.15766 [I 2022-10-27 22:23:08] ...optimizers/trijoint.py.151: total_norm: 29.320187 [I 2022-10-27 22:23:14] ...trap/engines/engine.py.224: train: epoch 91 | batch 890/2304 [I 2022-10-27 22:23:14] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:13:06 [I 2022-10-27 22:23:14] ...trap/engines/engine.py.229: process: 0.55387 | load: 0.00256 [I 2022-10-27 22:23:14] ...trap/engines/engine.py.230: loss: 0.10664 [I 2022-10-27 22:23:14] ...optimizers/trijoint.py.151: total_norm: 14.292602 [I 2022-10-27 22:23:20] ...trap/engines/engine.py.224: train: epoch 91 | batch 900/2304 [I 2022-10-27 22:23:20] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:13:02 [I 2022-10-27 22:23:20] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00248 [I 2022-10-27 22:23:20] ...trap/engines/engine.py.230: loss: 0.14814 [I 2022-10-27 22:23:20] ...optimizers/trijoint.py.151: total_norm: 15.457367 [I 2022-10-27 22:23:25] ...trap/engines/engine.py.224: train: epoch 91 | batch 910/2304 [I 2022-10-27 22:23:25] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:57 [I 2022-10-27 22:23:25] ...trap/engines/engine.py.229: process: 0.56515 | load: 0.00246 [I 2022-10-27 22:23:25] ...trap/engines/engine.py.230: loss: 0.08648 [I 2022-10-27 22:23:25] ...optimizers/trijoint.py.151: total_norm: 12.505261 [I 2022-10-27 22:23:31] ...trap/engines/engine.py.224: train: epoch 91 | batch 920/2304 [I 2022-10-27 22:23:31] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:52 [I 2022-10-27 22:23:31] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00250 [I 2022-10-27 22:23:31] ...trap/engines/engine.py.230: loss: 0.15298 [I 2022-10-27 22:23:31] ...optimizers/trijoint.py.151: total_norm: 14.860163 [I 2022-10-27 22:23:36] ...trap/engines/engine.py.224: train: epoch 91 | batch 930/2304 [I 2022-10-27 22:23:36] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:43 [I 2022-10-27 22:23:36] ...trap/engines/engine.py.229: process: 0.54423 | load: 0.00246 [I 2022-10-27 22:23:36] ...trap/engines/engine.py.230: loss: 0.16692 [I 2022-10-27 22:23:36] ...optimizers/trijoint.py.151: total_norm: 15.524995 [I 2022-10-27 22:23:42] ...trap/engines/engine.py.224: train: epoch 91 | batch 940/2304 [I 2022-10-27 22:23:42] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:33 [I 2022-10-27 22:23:42] ...trap/engines/engine.py.229: process: 0.54616 | load: 0.00244 [I 2022-10-27 22:23:42] ...trap/engines/engine.py.230: loss: 0.08684 [I 2022-10-27 22:23:42] ...optimizers/trijoint.py.151: total_norm: 13.475851 [I 2022-10-27 22:23:47] ...trap/engines/engine.py.224: train: epoch 91 | batch 950/2304 [I 2022-10-27 22:23:47] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:33 [I 2022-10-27 22:23:47] ...trap/engines/engine.py.229: process: 0.56924 | load: 0.00241 [I 2022-10-27 22:23:47] ...trap/engines/engine.py.230: loss: 0.16552 [I 2022-10-27 22:23:47] ...optimizers/trijoint.py.151: total_norm: 9.378174 [I 2022-10-27 22:23:53] ...trap/engines/engine.py.224: train: epoch 91 | batch 960/2304 [I 2022-10-27 22:23:53] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:30 [I 2022-10-27 22:23:53] ...trap/engines/engine.py.229: process: 0.55520 | load: 0.00243 [I 2022-10-27 22:23:53] ...trap/engines/engine.py.230: loss: 0.09393 [I 2022-10-27 22:23:53] ...optimizers/trijoint.py.151: total_norm: 9.032495 [I 2022-10-27 22:23:59] ...trap/engines/engine.py.224: train: epoch 91 | batch 970/2304 [I 2022-10-27 22:23:59] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:12:22 [I 2022-10-27 22:23:59] ...trap/engines/engine.py.229: process: 0.55576 | load: 0.00243 [I 2022-10-27 22:23:59] ...trap/engines/engine.py.230: loss: 0.19984 [I 2022-10-27 22:23:59] ...optimizers/trijoint.py.151: total_norm: 16.425919 [I 2022-10-27 22:24:04] ...trap/engines/engine.py.224: train: epoch 91 | batch 980/2304 [I 2022-10-27 22:24:04] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:19 [I 2022-10-27 22:24:04] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00243 [I 2022-10-27 22:24:04] ...trap/engines/engine.py.230: loss: 0.13901 [I 2022-10-27 22:24:04] ...optimizers/trijoint.py.151: total_norm: 24.357754 [I 2022-10-27 22:24:10] ...trap/engines/engine.py.224: train: epoch 91 | batch 990/2304 [I 2022-10-27 22:24:10] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:12:16 [I 2022-10-27 22:24:10] ...trap/engines/engine.py.229: process: 0.57409 | load: 0.00244 [I 2022-10-27 22:24:10] ...trap/engines/engine.py.230: loss: 0.14532 [I 2022-10-27 22:24:10] ...optimizers/trijoint.py.151: total_norm: 40.869263 [I 2022-10-27 22:24:15] ...trap/engines/engine.py.224: train: epoch 91 | batch 1000/2304 [I 2022-10-27 22:24:15] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:04 [I 2022-10-27 22:24:15] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00242 [I 2022-10-27 22:24:15] ...trap/engines/engine.py.230: loss: 0.10429 [I 2022-10-27 22:24:15] ...optimizers/trijoint.py.151: total_norm: 12.732759 [I 2022-10-27 22:24:21] ...trap/engines/engine.py.224: train: epoch 91 | batch 1010/2304 [I 2022-10-27 22:24:21] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:12:02 [I 2022-10-27 22:24:21] ...trap/engines/engine.py.229: process: 0.55388 | load: 0.00241 [I 2022-10-27 22:24:21] ...trap/engines/engine.py.230: loss: 0.11728 [I 2022-10-27 22:24:21] ...optimizers/trijoint.py.151: total_norm: 14.988400 [I 2022-10-27 22:24:27] ...trap/engines/engine.py.224: train: epoch 91 | batch 1020/2304 [I 2022-10-27 22:24:27] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:56 [I 2022-10-27 22:24:27] ...trap/engines/engine.py.229: process: 0.55770 | load: 0.00268 [I 2022-10-27 22:24:27] ...trap/engines/engine.py.230: loss: 0.16713 [I 2022-10-27 22:24:27] ...optimizers/trijoint.py.151: total_norm: 16.686712 [I 2022-10-27 22:24:32] ...trap/engines/engine.py.224: train: epoch 91 | batch 1030/2304 [I 2022-10-27 22:24:32] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:48 [I 2022-10-27 22:24:32] ...trap/engines/engine.py.229: process: 0.55219 | load: 0.00253 [I 2022-10-27 22:24:32] ...trap/engines/engine.py.230: loss: 0.13443 [I 2022-10-27 22:24:32] ...optimizers/trijoint.py.151: total_norm: 14.968053 [I 2022-10-27 22:24:38] ...trap/engines/engine.py.224: train: epoch 91 | batch 1040/2304 [I 2022-10-27 22:24:38] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:41 [I 2022-10-27 22:24:38] ...trap/engines/engine.py.229: process: 0.55547 | load: 0.00252 [I 2022-10-27 22:24:38] ...trap/engines/engine.py.230: loss: 0.14570 [I 2022-10-27 22:24:38] ...optimizers/trijoint.py.151: total_norm: 13.103389 [I 2022-10-27 22:24:43] ...trap/engines/engine.py.224: train: epoch 91 | batch 1050/2304 [I 2022-10-27 22:24:43] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:44 [I 2022-10-27 22:24:43] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00263 [I 2022-10-27 22:24:43] ...trap/engines/engine.py.230: loss: 0.10373 [I 2022-10-27 22:24:43] ...optimizers/trijoint.py.151: total_norm: 17.435003 [I 2022-10-27 22:24:49] ...trap/engines/engine.py.224: train: epoch 91 | batch 1060/2304 [I 2022-10-27 22:24:49] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:34 [I 2022-10-27 22:24:49] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00253 [I 2022-10-27 22:24:49] ...trap/engines/engine.py.230: loss: 0.21541 [I 2022-10-27 22:24:49] ...optimizers/trijoint.py.151: total_norm: 21.722054 [I 2022-10-27 22:24:54] ...trap/engines/engine.py.224: train: epoch 91 | batch 1070/2304 [I 2022-10-27 22:24:54] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:31 [I 2022-10-27 22:24:54] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00293 [I 2022-10-27 22:24:54] ...trap/engines/engine.py.230: loss: 0.08354 [I 2022-10-27 22:24:54] ...optimizers/trijoint.py.151: total_norm: 8.563403 [I 2022-10-27 22:25:00] ...trap/engines/engine.py.224: train: epoch 91 | batch 1080/2304 [I 2022-10-27 22:25:00] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:26 [I 2022-10-27 22:25:00] ...trap/engines/engine.py.229: process: 0.56394 | load: 0.00271 [I 2022-10-27 22:25:00] ...trap/engines/engine.py.230: loss: 0.10142 [I 2022-10-27 22:25:00] ...optimizers/trijoint.py.151: total_norm: 22.147808 [I 2022-10-27 22:25:06] ...trap/engines/engine.py.224: train: epoch 91 | batch 1090/2304 [I 2022-10-27 22:25:06] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:11:18 [I 2022-10-27 22:25:06] ...trap/engines/engine.py.229: process: 0.56767 | load: 0.00268 [I 2022-10-27 22:25:06] ...trap/engines/engine.py.230: loss: 0.13788 [I 2022-10-27 22:25:06] ...optimizers/trijoint.py.151: total_norm: 8.781319 [I 2022-10-27 22:25:11] ...trap/engines/engine.py.224: train: epoch 91 | batch 1100/2304 [I 2022-10-27 22:25:11] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:11:12 [I 2022-10-27 22:25:11] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00258 [I 2022-10-27 22:25:11] ...trap/engines/engine.py.230: loss: 0.16441 [I 2022-10-27 22:25:11] ...optimizers/trijoint.py.151: total_norm: 14.033521 [I 2022-10-27 22:25:17] ...trap/engines/engine.py.224: train: epoch 91 | batch 1110/2304 [I 2022-10-27 22:25:17] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:11:07 [I 2022-10-27 22:25:17] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00250 [I 2022-10-27 22:25:17] ...trap/engines/engine.py.230: loss: 0.13636 [I 2022-10-27 22:25:17] ...optimizers/trijoint.py.151: total_norm: 27.919800 [I 2022-10-27 22:25:22] ...trap/engines/engine.py.224: train: epoch 91 | batch 1120/2304 [I 2022-10-27 22:25:22] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:11:01 [I 2022-10-27 22:25:22] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00252 [I 2022-10-27 22:25:22] ...trap/engines/engine.py.230: loss: 0.12144 [I 2022-10-27 22:25:22] ...optimizers/trijoint.py.151: total_norm: 26.184664 [I 2022-10-27 22:25:28] ...trap/engines/engine.py.224: train: epoch 91 | batch 1130/2304 [I 2022-10-27 22:25:28] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:54 [I 2022-10-27 22:25:28] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00252 [I 2022-10-27 22:25:28] ...trap/engines/engine.py.230: loss: 0.19466 [I 2022-10-27 22:25:28] ...optimizers/trijoint.py.151: total_norm: 59.707550 [I 2022-10-27 22:25:34] ...trap/engines/engine.py.224: train: epoch 91 | batch 1140/2304 [I 2022-10-27 22:25:34] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:49 [I 2022-10-27 22:25:34] ...trap/engines/engine.py.229: process: 0.55594 | load: 0.00289 [I 2022-10-27 22:25:34] ...trap/engines/engine.py.230: loss: 0.11345 [I 2022-10-27 22:25:34] ...optimizers/trijoint.py.151: total_norm: 18.897032 [I 2022-10-27 22:25:39] ...trap/engines/engine.py.224: train: epoch 91 | batch 1150/2304 [I 2022-10-27 22:25:39] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:42 [I 2022-10-27 22:25:39] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00274 [I 2022-10-27 22:25:39] ...trap/engines/engine.py.230: loss: 0.12873 [I 2022-10-27 22:25:39] ...optimizers/trijoint.py.151: total_norm: 28.147984 [I 2022-10-27 22:25:45] ...trap/engines/engine.py.224: train: epoch 91 | batch 1160/2304 [I 2022-10-27 22:25:45] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:37 [I 2022-10-27 22:25:45] ...trap/engines/engine.py.229: process: 0.54837 | load: 0.00269 [I 2022-10-27 22:25:45] ...trap/engines/engine.py.230: loss: 0.14898 [I 2022-10-27 22:25:45] ...optimizers/trijoint.py.151: total_norm: 31.183052 [I 2022-10-27 22:25:50] ...trap/engines/engine.py.224: train: epoch 91 | batch 1170/2304 [I 2022-10-27 22:25:50] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:30 [I 2022-10-27 22:25:50] ...trap/engines/engine.py.229: process: 0.54546 | load: 0.00261 [I 2022-10-27 22:25:50] ...trap/engines/engine.py.230: loss: 0.11500 [I 2022-10-27 22:25:50] ...optimizers/trijoint.py.151: total_norm: 8.764521 [I 2022-10-27 22:25:56] ...trap/engines/engine.py.224: train: epoch 91 | batch 1180/2304 [I 2022-10-27 22:25:56] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:26 [I 2022-10-27 22:25:56] ...trap/engines/engine.py.229: process: 0.56720 | load: 0.00265 [I 2022-10-27 22:25:56] ...trap/engines/engine.py.230: loss: 0.10030 [I 2022-10-27 22:25:56] ...optimizers/trijoint.py.151: total_norm: 14.521674 [I 2022-10-27 22:26:02] ...trap/engines/engine.py.224: train: epoch 91 | batch 1190/2304 [I 2022-10-27 22:26:02] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:21 [I 2022-10-27 22:26:02] ...trap/engines/engine.py.229: process: 0.56220 | load: 0.00276 [I 2022-10-27 22:26:02] ...trap/engines/engine.py.230: loss: 0.13826 [I 2022-10-27 22:26:02] ...optimizers/trijoint.py.151: total_norm: 19.556423 [I 2022-10-27 22:26:07] ...trap/engines/engine.py.224: train: epoch 91 | batch 1200/2304 [I 2022-10-27 22:26:07] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:10:19 [I 2022-10-27 22:26:07] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00257 [I 2022-10-27 22:26:07] ...trap/engines/engine.py.230: loss: 0.09520 [I 2022-10-27 22:26:07] ...optimizers/trijoint.py.151: total_norm: 9.496418 [I 2022-10-27 22:26:13] ...trap/engines/engine.py.224: train: epoch 91 | batch 1210/2304 [I 2022-10-27 22:26:13] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:10:12 [I 2022-10-27 22:26:13] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00257 [I 2022-10-27 22:26:13] ...trap/engines/engine.py.230: loss: 0.10341 [I 2022-10-27 22:26:13] ...optimizers/trijoint.py.151: total_norm: 12.495514 [I 2022-10-27 22:26:18] ...trap/engines/engine.py.224: train: epoch 91 | batch 1220/2304 [I 2022-10-27 22:26:18] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:10:03 [I 2022-10-27 22:26:18] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00249 [I 2022-10-27 22:26:18] ...trap/engines/engine.py.230: loss: 0.14354 [I 2022-10-27 22:26:18] ...optimizers/trijoint.py.151: total_norm: 18.566742 [I 2022-10-27 22:26:24] ...trap/engines/engine.py.224: train: epoch 91 | batch 1230/2304 [I 2022-10-27 22:26:24] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:10:05 [I 2022-10-27 22:26:24] ...trap/engines/engine.py.229: process: 0.54640 | load: 0.00251 [I 2022-10-27 22:26:24] ...trap/engines/engine.py.230: loss: 0.21223 [I 2022-10-27 22:26:24] ...optimizers/trijoint.py.151: total_norm: 37.969307 [I 2022-10-27 22:26:30] ...trap/engines/engine.py.224: train: epoch 91 | batch 1240/2304 [I 2022-10-27 22:26:30] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:10:07 [I 2022-10-27 22:26:30] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00248 [I 2022-10-27 22:26:30] ...trap/engines/engine.py.230: loss: 0.10875 [I 2022-10-27 22:26:30] ...optimizers/trijoint.py.151: total_norm: 14.962411 [I 2022-10-27 22:26:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 1250/2304 [I 2022-10-27 22:26:35] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:49 [I 2022-10-27 22:26:35] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00252 [I 2022-10-27 22:26:35] ...trap/engines/engine.py.230: loss: 0.15189 [I 2022-10-27 22:26:35] ...optimizers/trijoint.py.151: total_norm: 33.025280 [I 2022-10-27 22:26:41] ...trap/engines/engine.py.224: train: epoch 91 | batch 1260/2304 [I 2022-10-27 22:26:41] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:45 [I 2022-10-27 22:26:41] ...trap/engines/engine.py.229: process: 0.56376 | load: 0.00244 [I 2022-10-27 22:26:41] ...trap/engines/engine.py.230: loss: 0.13432 [I 2022-10-27 22:26:41] ...optimizers/trijoint.py.151: total_norm: 14.203815 [I 2022-10-27 22:26:46] ...trap/engines/engine.py.224: train: epoch 91 | batch 1270/2304 [I 2022-10-27 22:26:46] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:35 [I 2022-10-27 22:26:46] ...trap/engines/engine.py.229: process: 0.54539 | load: 0.00246 [I 2022-10-27 22:26:46] ...trap/engines/engine.py.230: loss: 0.14348 [I 2022-10-27 22:26:46] ...optimizers/trijoint.py.151: total_norm: 26.305536 [I 2022-10-27 22:26:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 1280/2304 [I 2022-10-27 22:26:52] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:26 [I 2022-10-27 22:26:52] ...trap/engines/engine.py.229: process: 0.55114 | load: 0.00247 [I 2022-10-27 22:26:52] ...trap/engines/engine.py.230: loss: 0.10229 [I 2022-10-27 22:26:52] ...optimizers/trijoint.py.151: total_norm: 11.456984 [I 2022-10-27 22:26:58] ...trap/engines/engine.py.224: train: epoch 91 | batch 1290/2304 [I 2022-10-27 22:26:58] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:24 [I 2022-10-27 22:26:58] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00255 [I 2022-10-27 22:26:58] ...trap/engines/engine.py.230: loss: 0.10298 [I 2022-10-27 22:26:58] ...optimizers/trijoint.py.151: total_norm: 79.567909 [I 2022-10-27 22:27:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 1300/2304 [I 2022-10-27 22:27:03] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:09:21 [I 2022-10-27 22:27:03] ...trap/engines/engine.py.229: process: 0.55039 | load: 0.00251 [I 2022-10-27 22:27:03] ...trap/engines/engine.py.230: loss: 0.08580 [I 2022-10-27 22:27:03] ...optimizers/trijoint.py.151: total_norm: 15.173481 [I 2022-10-27 22:27:09] ...trap/engines/engine.py.224: train: epoch 91 | batch 1310/2304 [I 2022-10-27 22:27:09] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:09:14 [I 2022-10-27 22:27:09] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00246 [I 2022-10-27 22:27:09] ...trap/engines/engine.py.230: loss: 0.14220 [I 2022-10-27 22:27:09] ...optimizers/trijoint.py.151: total_norm: 34.206158 [I 2022-10-27 22:27:15] ...trap/engines/engine.py.224: train: epoch 91 | batch 1320/2304 [I 2022-10-27 22:27:15] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:09:12 [I 2022-10-27 22:27:15] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00255 [I 2022-10-27 22:27:15] ...trap/engines/engine.py.230: loss: 0.17412 [I 2022-10-27 22:27:15] ...optimizers/trijoint.py.151: total_norm: 9.716200 [I 2022-10-27 22:27:20] ...trap/engines/engine.py.224: train: epoch 91 | batch 1330/2304 [I 2022-10-27 22:27:20] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:09:02 [I 2022-10-27 22:27:20] ...trap/engines/engine.py.229: process: 0.54628 | load: 0.00261 [I 2022-10-27 22:27:20] ...trap/engines/engine.py.230: loss: 0.20700 [I 2022-10-27 22:27:20] ...optimizers/trijoint.py.151: total_norm: 13.500173 [I 2022-10-27 22:27:26] ...trap/engines/engine.py.224: train: epoch 91 | batch 1340/2304 [I 2022-10-27 22:27:26] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:58 [I 2022-10-27 22:27:26] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00252 [I 2022-10-27 22:27:26] ...trap/engines/engine.py.230: loss: 0.13350 [I 2022-10-27 22:27:26] ...optimizers/trijoint.py.151: total_norm: 380.240234 [I 2022-10-27 22:27:31] ...trap/engines/engine.py.224: train: epoch 91 | batch 1350/2304 [I 2022-10-27 22:27:31] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:52 [I 2022-10-27 22:27:31] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00250 [I 2022-10-27 22:27:31] ...trap/engines/engine.py.230: loss: 0.15348 [I 2022-10-27 22:27:31] ...optimizers/trijoint.py.151: total_norm: 15.067801 [I 2022-10-27 22:27:37] ...trap/engines/engine.py.224: train: epoch 91 | batch 1360/2304 [I 2022-10-27 22:27:37] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:48 [I 2022-10-27 22:27:37] ...trap/engines/engine.py.229: process: 0.56439 | load: 0.00255 [I 2022-10-27 22:27:37] ...trap/engines/engine.py.230: loss: 0.12312 [I 2022-10-27 22:27:37] ...optimizers/trijoint.py.151: total_norm: 30.463459 [I 2022-10-27 22:27:42] ...trap/engines/engine.py.224: train: epoch 91 | batch 1370/2304 [I 2022-10-27 22:27:42] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:38 [I 2022-10-27 22:27:42] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00257 [I 2022-10-27 22:27:42] ...trap/engines/engine.py.230: loss: 0.11358 [I 2022-10-27 22:27:42] ...optimizers/trijoint.py.151: total_norm: 16.189777 [I 2022-10-27 22:27:48] ...trap/engines/engine.py.224: train: epoch 91 | batch 1380/2304 [I 2022-10-27 22:27:48] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:57 [I 2022-10-27 22:27:48] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00256 [I 2022-10-27 22:27:48] ...trap/engines/engine.py.230: loss: 0.12452 [I 2022-10-27 22:27:48] ...optimizers/trijoint.py.151: total_norm: 9.360491 [I 2022-10-27 22:27:54] ...trap/engines/engine.py.224: train: epoch 91 | batch 1390/2304 [I 2022-10-27 22:27:54] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:31 [I 2022-10-27 22:27:54] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00248 [I 2022-10-27 22:27:54] ...trap/engines/engine.py.230: loss: 0.10017 [I 2022-10-27 22:27:54] ...optimizers/trijoint.py.151: total_norm: 11.295179 [I 2022-10-27 22:27:59] ...trap/engines/engine.py.224: train: epoch 91 | batch 1400/2304 [I 2022-10-27 22:27:59] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:25 [I 2022-10-27 22:27:59] ...trap/engines/engine.py.229: process: 0.56617 | load: 0.00245 [I 2022-10-27 22:27:59] ...trap/engines/engine.py.230: loss: 0.10295 [I 2022-10-27 22:27:59] ...optimizers/trijoint.py.151: total_norm: 19.814514 [I 2022-10-27 22:28:05] ...trap/engines/engine.py.224: train: epoch 91 | batch 1410/2304 [I 2022-10-27 22:28:05] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:08:21 [I 2022-10-27 22:28:05] ...trap/engines/engine.py.229: process: 0.56318 | load: 0.00247 [I 2022-10-27 22:28:05] ...trap/engines/engine.py.230: loss: 0.10511 [I 2022-10-27 22:28:05] ...optimizers/trijoint.py.151: total_norm: 18.561735 [I 2022-10-27 22:28:11] ...trap/engines/engine.py.224: train: epoch 91 | batch 1420/2304 [I 2022-10-27 22:28:11] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:08:09 [I 2022-10-27 22:28:11] ...trap/engines/engine.py.229: process: 0.55484 | load: 0.00242 [I 2022-10-27 22:28:11] ...trap/engines/engine.py.230: loss: 0.14654 [I 2022-10-27 22:28:11] ...optimizers/trijoint.py.151: total_norm: 13.217195 [I 2022-10-27 22:28:16] ...trap/engines/engine.py.224: train: epoch 91 | batch 1430/2304 [I 2022-10-27 22:28:16] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:08:03 [I 2022-10-27 22:28:16] ...trap/engines/engine.py.229: process: 0.55776 | load: 0.00245 [I 2022-10-27 22:28:16] ...trap/engines/engine.py.230: loss: 0.11177 [I 2022-10-27 22:28:16] ...optimizers/trijoint.py.151: total_norm: 12.763753 [I 2022-10-27 22:28:22] ...trap/engines/engine.py.224: train: epoch 91 | batch 1440/2304 [I 2022-10-27 22:28:22] ...trap/engines/engine.py.228: elapsed: 0:13:49 | left: 0:07:56 [I 2022-10-27 22:28:22] ...trap/engines/engine.py.229: process: 0.54617 | load: 0.00253 [I 2022-10-27 22:28:22] ...trap/engines/engine.py.230: loss: 0.09268 [I 2022-10-27 22:28:22] ...optimizers/trijoint.py.151: total_norm: 24.964184 [I 2022-10-27 22:28:27] ...trap/engines/engine.py.224: train: epoch 91 | batch 1450/2304 [I 2022-10-27 22:28:27] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:53 [I 2022-10-27 22:28:27] ...trap/engines/engine.py.229: process: 0.55128 | load: 0.00250 [I 2022-10-27 22:28:27] ...trap/engines/engine.py.230: loss: 0.11821 [I 2022-10-27 22:28:27] ...optimizers/trijoint.py.151: total_norm: 12.550584 [I 2022-10-27 22:28:33] ...trap/engines/engine.py.224: train: epoch 91 | batch 1460/2304 [I 2022-10-27 22:28:33] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:46 [I 2022-10-27 22:28:33] ...trap/engines/engine.py.229: process: 0.54318 | load: 0.00250 [I 2022-10-27 22:28:33] ...trap/engines/engine.py.230: loss: 0.14090 [I 2022-10-27 22:28:33] ...optimizers/trijoint.py.151: total_norm: 12.272525 [I 2022-10-27 22:28:38] ...trap/engines/engine.py.224: train: epoch 91 | batch 1470/2304 [I 2022-10-27 22:28:38] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:41 [I 2022-10-27 22:28:38] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00254 [I 2022-10-27 22:28:38] ...trap/engines/engine.py.230: loss: 0.12015 [I 2022-10-27 22:28:38] ...optimizers/trijoint.py.151: total_norm: 13.961000 [I 2022-10-27 22:28:44] ...trap/engines/engine.py.224: train: epoch 91 | batch 1480/2304 [I 2022-10-27 22:28:44] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:37 [I 2022-10-27 22:28:44] ...trap/engines/engine.py.229: process: 0.54654 | load: 0.00253 [I 2022-10-27 22:28:44] ...trap/engines/engine.py.230: loss: 0.15170 [I 2022-10-27 22:28:44] ...optimizers/trijoint.py.151: total_norm: 21.788975 [I 2022-10-27 22:28:49] ...trap/engines/engine.py.224: train: epoch 91 | batch 1490/2304 [I 2022-10-27 22:28:49] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:28 [I 2022-10-27 22:28:49] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00257 [I 2022-10-27 22:28:49] ...trap/engines/engine.py.230: loss: 0.09718 [I 2022-10-27 22:28:49] ...optimizers/trijoint.py.151: total_norm: 31.443760 [I 2022-10-27 22:28:55] ...trap/engines/engine.py.224: train: epoch 91 | batch 1500/2304 [I 2022-10-27 22:28:55] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:39 [I 2022-10-27 22:28:55] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00261 [I 2022-10-27 22:28:55] ...trap/engines/engine.py.230: loss: 0.15016 [I 2022-10-27 22:28:55] ...optimizers/trijoint.py.151: total_norm: 40.609966 [I 2022-10-27 22:29:01] ...trap/engines/engine.py.224: train: epoch 91 | batch 1510/2304 [I 2022-10-27 22:29:01] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:20 [I 2022-10-27 22:29:01] ...trap/engines/engine.py.229: process: 0.55383 | load: 0.00254 [I 2022-10-27 22:29:01] ...trap/engines/engine.py.230: loss: 0.09293 [I 2022-10-27 22:29:01] ...optimizers/trijoint.py.151: total_norm: 14.088216 [I 2022-10-27 22:29:06] ...trap/engines/engine.py.224: train: epoch 91 | batch 1520/2304 [I 2022-10-27 22:29:06] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:15 [I 2022-10-27 22:29:06] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00252 [I 2022-10-27 22:29:06] ...trap/engines/engine.py.230: loss: 0.10439 [I 2022-10-27 22:29:06] ...optimizers/trijoint.py.151: total_norm: 16.455351 [I 2022-10-27 22:29:12] ...trap/engines/engine.py.224: train: epoch 91 | batch 1530/2304 [I 2022-10-27 22:29:12] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:07:10 [I 2022-10-27 22:29:12] ...trap/engines/engine.py.229: process: 0.55066 | load: 0.00267 [I 2022-10-27 22:29:12] ...trap/engines/engine.py.230: loss: 0.11874 [I 2022-10-27 22:29:12] ...optimizers/trijoint.py.151: total_norm: 14.444951 [I 2022-10-27 22:29:17] ...trap/engines/engine.py.224: train: epoch 91 | batch 1540/2304 [I 2022-10-27 22:29:17] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:07:04 [I 2022-10-27 22:29:17] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00247 [I 2022-10-27 22:29:17] ...trap/engines/engine.py.230: loss: 0.10951 [I 2022-10-27 22:29:17] ...optimizers/trijoint.py.151: total_norm: 11.385290 [I 2022-10-27 22:29:23] ...trap/engines/engine.py.224: train: epoch 91 | batch 1550/2304 [I 2022-10-27 22:29:23] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:59 [I 2022-10-27 22:29:23] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00248 [I 2022-10-27 22:29:23] ...trap/engines/engine.py.230: loss: 0.11309 [I 2022-10-27 22:29:23] ...optimizers/trijoint.py.151: total_norm: 17.187384 [I 2022-10-27 22:29:28] ...trap/engines/engine.py.224: train: epoch 91 | batch 1560/2304 [I 2022-10-27 22:29:28] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:49 [I 2022-10-27 22:29:28] ...trap/engines/engine.py.229: process: 0.53810 | load: 0.00248 [I 2022-10-27 22:29:28] ...trap/engines/engine.py.230: loss: 0.13018 [I 2022-10-27 22:29:28] ...optimizers/trijoint.py.151: total_norm: 20.855587 [I 2022-10-27 22:29:34] ...trap/engines/engine.py.224: train: epoch 91 | batch 1570/2304 [I 2022-10-27 22:29:34] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:48 [I 2022-10-27 22:29:34] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00244 [I 2022-10-27 22:29:34] ...trap/engines/engine.py.230: loss: 0.10592 [I 2022-10-27 22:29:34] ...optimizers/trijoint.py.151: total_norm: 12.050963 [I 2022-10-27 22:29:40] ...trap/engines/engine.py.224: train: epoch 91 | batch 1580/2304 [I 2022-10-27 22:29:40] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:43 [I 2022-10-27 22:29:40] ...trap/engines/engine.py.229: process: 0.54821 | load: 0.00252 [I 2022-10-27 22:29:40] ...trap/engines/engine.py.230: loss: 0.10167 [I 2022-10-27 22:29:40] ...optimizers/trijoint.py.151: total_norm: 12.898688 [I 2022-10-27 22:29:45] ...trap/engines/engine.py.224: train: epoch 91 | batch 1590/2304 [I 2022-10-27 22:29:45] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:38 [I 2022-10-27 22:29:45] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00254 [I 2022-10-27 22:29:45] ...trap/engines/engine.py.230: loss: 0.12832 [I 2022-10-27 22:29:45] ...optimizers/trijoint.py.151: total_norm: 14.358470 [I 2022-10-27 22:29:51] ...trap/engines/engine.py.224: train: epoch 91 | batch 1600/2304 [I 2022-10-27 22:29:51] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:31 [I 2022-10-27 22:29:51] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00251 [I 2022-10-27 22:29:51] ...trap/engines/engine.py.230: loss: 0.18281 [I 2022-10-27 22:29:51] ...optimizers/trijoint.py.151: total_norm: 18.157928 [I 2022-10-27 22:29:56] ...trap/engines/engine.py.224: train: epoch 91 | batch 1610/2304 [I 2022-10-27 22:29:56] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:25 [I 2022-10-27 22:29:56] ...trap/engines/engine.py.229: process: 0.56338 | load: 0.00256 [I 2022-10-27 22:29:56] ...trap/engines/engine.py.230: loss: 0.14616 [I 2022-10-27 22:29:56] ...optimizers/trijoint.py.151: total_norm: 14.477312 [I 2022-10-27 22:30:02] ...trap/engines/engine.py.224: train: epoch 91 | batch 1620/2304 [I 2022-10-27 22:30:02] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:19 [I 2022-10-27 22:30:02] ...trap/engines/engine.py.229: process: 0.55307 | load: 0.00259 [I 2022-10-27 22:30:02] ...trap/engines/engine.py.230: loss: 0.09515 [I 2022-10-27 22:30:02] ...optimizers/trijoint.py.151: total_norm: 13.540885 [I 2022-10-27 22:30:08] ...trap/engines/engine.py.224: train: epoch 91 | batch 1630/2304 [I 2022-10-27 22:30:08] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:06:18 [I 2022-10-27 22:30:08] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00249 [I 2022-10-27 22:30:08] ...trap/engines/engine.py.230: loss: 0.09198 [I 2022-10-27 22:30:08] ...optimizers/trijoint.py.151: total_norm: 14.236034 [I 2022-10-27 22:30:13] ...trap/engines/engine.py.224: train: epoch 91 | batch 1640/2304 [I 2022-10-27 22:30:13] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:06:09 [I 2022-10-27 22:30:13] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00268 [I 2022-10-27 22:30:13] ...trap/engines/engine.py.230: loss: 0.09596 [I 2022-10-27 22:30:13] ...optimizers/trijoint.py.151: total_norm: 10.377847 [I 2022-10-27 22:30:19] ...trap/engines/engine.py.224: train: epoch 91 | batch 1650/2304 [I 2022-10-27 22:30:19] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:06:05 [I 2022-10-27 22:30:19] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00266 [I 2022-10-27 22:30:19] ...trap/engines/engine.py.230: loss: 0.12947 [I 2022-10-27 22:30:19] ...optimizers/trijoint.py.151: total_norm: 23.382730 [I 2022-10-27 22:30:24] ...trap/engines/engine.py.224: train: epoch 91 | batch 1660/2304 [I 2022-10-27 22:30:24] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:57 [I 2022-10-27 22:30:24] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00251 [I 2022-10-27 22:30:24] ...trap/engines/engine.py.230: loss: 0.13927 [I 2022-10-27 22:30:24] ...optimizers/trijoint.py.151: total_norm: 19.026876 [I 2022-10-27 22:30:30] ...trap/engines/engine.py.224: train: epoch 91 | batch 1670/2304 [I 2022-10-27 22:30:30] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:52 [I 2022-10-27 22:30:30] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00253 [I 2022-10-27 22:30:30] ...trap/engines/engine.py.230: loss: 0.09817 [I 2022-10-27 22:30:30] ...optimizers/trijoint.py.151: total_norm: 15.229589 [I 2022-10-27 22:30:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 1680/2304 [I 2022-10-27 22:30:35] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:47 [I 2022-10-27 22:30:35] ...trap/engines/engine.py.229: process: 0.55266 | load: 0.00247 [I 2022-10-27 22:30:35] ...trap/engines/engine.py.230: loss: 0.21293 [I 2022-10-27 22:30:35] ...optimizers/trijoint.py.151: total_norm: 18.008261 [I 2022-10-27 22:30:41] ...trap/engines/engine.py.224: train: epoch 91 | batch 1690/2304 [I 2022-10-27 22:30:41] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:42 [I 2022-10-27 22:30:41] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00252 [I 2022-10-27 22:30:41] ...trap/engines/engine.py.230: loss: 0.14465 [I 2022-10-27 22:30:41] ...optimizers/trijoint.py.151: total_norm: 19.554358 [I 2022-10-27 22:30:47] ...trap/engines/engine.py.224: train: epoch 91 | batch 1700/2304 [I 2022-10-27 22:30:47] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:35 [I 2022-10-27 22:30:47] ...trap/engines/engine.py.229: process: 0.55557 | load: 0.00251 [I 2022-10-27 22:30:47] ...trap/engines/engine.py.230: loss: 0.11398 [I 2022-10-27 22:30:47] ...optimizers/trijoint.py.151: total_norm: 22.775127 [I 2022-10-27 22:30:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 1710/2304 [I 2022-10-27 22:30:52] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:31 [I 2022-10-27 22:30:52] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00241 [I 2022-10-27 22:30:52] ...trap/engines/engine.py.230: loss: 0.16657 [I 2022-10-27 22:30:52] ...optimizers/trijoint.py.151: total_norm: 26.844923 [I 2022-10-27 22:30:58] ...trap/engines/engine.py.224: train: epoch 91 | batch 1720/2304 [I 2022-10-27 22:30:58] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:24 [I 2022-10-27 22:30:58] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00249 [I 2022-10-27 22:30:58] ...trap/engines/engine.py.230: loss: 0.08936 [I 2022-10-27 22:30:58] ...optimizers/trijoint.py.151: total_norm: 11.986875 [I 2022-10-27 22:31:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 1730/2304 [I 2022-10-27 22:31:03] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:32 [I 2022-10-27 22:31:03] ...trap/engines/engine.py.229: process: 0.66065 | load: 0.00243 [I 2022-10-27 22:31:03] ...trap/engines/engine.py.230: loss: 0.16444 [I 2022-10-27 22:31:03] ...optimizers/trijoint.py.151: total_norm: 11.215890 [I 2022-10-27 22:31:09] ...trap/engines/engine.py.224: train: epoch 91 | batch 1740/2304 [I 2022-10-27 22:31:09] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:05:14 [I 2022-10-27 22:31:09] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00240 [I 2022-10-27 22:31:09] ...trap/engines/engine.py.230: loss: 0.10351 [I 2022-10-27 22:31:09] ...optimizers/trijoint.py.151: total_norm: 11.451031 [I 2022-10-27 22:31:15] ...trap/engines/engine.py.224: train: epoch 91 | batch 1750/2304 [I 2022-10-27 22:31:15] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:05:05 [I 2022-10-27 22:31:15] ...trap/engines/engine.py.229: process: 0.53368 | load: 0.00245 [I 2022-10-27 22:31:15] ...trap/engines/engine.py.230: loss: 0.08454 [I 2022-10-27 22:31:15] ...optimizers/trijoint.py.151: total_norm: 8.417480 [I 2022-10-27 22:31:20] ...trap/engines/engine.py.224: train: epoch 91 | batch 1760/2304 [I 2022-10-27 22:31:20] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:05:02 [I 2022-10-27 22:31:20] ...trap/engines/engine.py.229: process: 0.55531 | load: 0.00241 [I 2022-10-27 22:31:20] ...trap/engines/engine.py.230: loss: 0.17402 [I 2022-10-27 22:31:20] ...optimizers/trijoint.py.151: total_norm: 65.605545 [I 2022-10-27 22:31:26] ...trap/engines/engine.py.224: train: epoch 91 | batch 1770/2304 [I 2022-10-27 22:31:26] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:58 [I 2022-10-27 22:31:26] ...trap/engines/engine.py.229: process: 0.56449 | load: 0.00243 [I 2022-10-27 22:31:26] ...trap/engines/engine.py.230: loss: 0.15253 [I 2022-10-27 22:31:26] ...optimizers/trijoint.py.151: total_norm: 13.681771 [I 2022-10-27 22:31:31] ...trap/engines/engine.py.224: train: epoch 91 | batch 1780/2304 [I 2022-10-27 22:31:31] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:50 [I 2022-10-27 22:31:31] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00250 [I 2022-10-27 22:31:31] ...trap/engines/engine.py.230: loss: 0.11802 [I 2022-10-27 22:31:31] ...optimizers/trijoint.py.151: total_norm: 19.772753 [I 2022-10-27 22:31:37] ...trap/engines/engine.py.224: train: epoch 91 | batch 1790/2304 [I 2022-10-27 22:31:37] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:44 [I 2022-10-27 22:31:37] ...trap/engines/engine.py.229: process: 0.54929 | load: 0.00246 [I 2022-10-27 22:31:37] ...trap/engines/engine.py.230: loss: 0.18822 [I 2022-10-27 22:31:37] ...optimizers/trijoint.py.151: total_norm: 11.649301 [I 2022-10-27 22:31:42] ...trap/engines/engine.py.224: train: epoch 91 | batch 1800/2304 [I 2022-10-27 22:31:42] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:41 [I 2022-10-27 22:31:42] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00246 [I 2022-10-27 22:31:42] ...trap/engines/engine.py.230: loss: 0.12210 [I 2022-10-27 22:31:42] ...optimizers/trijoint.py.151: total_norm: 14.062475 [I 2022-10-27 22:31:48] ...trap/engines/engine.py.224: train: epoch 91 | batch 1810/2304 [I 2022-10-27 22:31:48] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:34 [I 2022-10-27 22:31:48] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00247 [I 2022-10-27 22:31:48] ...trap/engines/engine.py.230: loss: 0.11623 [I 2022-10-27 22:31:48] ...optimizers/trijoint.py.151: total_norm: 20.122293 [I 2022-10-27 22:31:54] ...trap/engines/engine.py.224: train: epoch 91 | batch 1820/2304 [I 2022-10-27 22:31:54] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:29 [I 2022-10-27 22:31:54] ...trap/engines/engine.py.229: process: 0.54812 | load: 0.00255 [I 2022-10-27 22:31:54] ...trap/engines/engine.py.230: loss: 0.08636 [I 2022-10-27 22:31:54] ...optimizers/trijoint.py.151: total_norm: 13.115367 [I 2022-10-27 22:31:59] ...trap/engines/engine.py.224: train: epoch 91 | batch 1830/2304 [I 2022-10-27 22:31:59] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:25 [I 2022-10-27 22:31:59] ...trap/engines/engine.py.229: process: 0.56795 | load: 0.00254 [I 2022-10-27 22:31:59] ...trap/engines/engine.py.230: loss: 0.08620 [I 2022-10-27 22:31:59] ...optimizers/trijoint.py.151: total_norm: 13.499063 [I 2022-10-27 22:32:05] ...trap/engines/engine.py.224: train: epoch 91 | batch 1840/2304 [I 2022-10-27 22:32:05] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:16 [I 2022-10-27 22:32:05] ...trap/engines/engine.py.229: process: 0.54431 | load: 0.00247 [I 2022-10-27 22:32:05] ...trap/engines/engine.py.230: loss: 0.15589 [I 2022-10-27 22:32:05] ...optimizers/trijoint.py.151: total_norm: 20.174160 [I 2022-10-27 22:32:10] ...trap/engines/engine.py.224: train: epoch 91 | batch 1850/2304 [I 2022-10-27 22:32:10] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:04:13 [I 2022-10-27 22:32:10] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00250 [I 2022-10-27 22:32:10] ...trap/engines/engine.py.230: loss: 0.10361 [I 2022-10-27 22:32:10] ...optimizers/trijoint.py.151: total_norm: 32.785954 [I 2022-10-27 22:32:16] ...trap/engines/engine.py.224: train: epoch 91 | batch 1860/2304 [I 2022-10-27 22:32:16] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:04:06 [I 2022-10-27 22:32:16] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00248 [I 2022-10-27 22:32:16] ...trap/engines/engine.py.230: loss: 0.10514 [I 2022-10-27 22:32:16] ...optimizers/trijoint.py.151: total_norm: 21.681992 [I 2022-10-27 22:32:21] ...trap/engines/engine.py.224: train: epoch 91 | batch 1870/2304 [I 2022-10-27 22:32:21] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:04:01 [I 2022-10-27 22:32:21] ...trap/engines/engine.py.229: process: 0.54983 | load: 0.00247 [I 2022-10-27 22:32:21] ...trap/engines/engine.py.230: loss: 0.13463 [I 2022-10-27 22:32:21] ...optimizers/trijoint.py.151: total_norm: 27.797277 [I 2022-10-27 22:32:27] ...trap/engines/engine.py.224: train: epoch 91 | batch 1880/2304 [I 2022-10-27 22:32:27] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:54 [I 2022-10-27 22:32:27] ...trap/engines/engine.py.229: process: 0.55054 | load: 0.00258 [I 2022-10-27 22:32:27] ...trap/engines/engine.py.230: loss: 0.15877 [I 2022-10-27 22:32:27] ...optimizers/trijoint.py.151: total_norm: 10.383767 [I 2022-10-27 22:32:33] ...trap/engines/engine.py.224: train: epoch 91 | batch 1890/2304 [I 2022-10-27 22:32:33] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:51 [I 2022-10-27 22:32:33] ...trap/engines/engine.py.229: process: 0.54820 | load: 0.00258 [I 2022-10-27 22:32:33] ...trap/engines/engine.py.230: loss: 0.09702 [I 2022-10-27 22:32:33] ...optimizers/trijoint.py.151: total_norm: 10.774019 [I 2022-10-27 22:32:38] ...trap/engines/engine.py.224: train: epoch 91 | batch 1900/2304 [I 2022-10-27 22:32:38] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:43 [I 2022-10-27 22:32:38] ...trap/engines/engine.py.229: process: 0.54362 | load: 0.00256 [I 2022-10-27 22:32:38] ...trap/engines/engine.py.230: loss: 0.18488 [I 2022-10-27 22:32:38] ...optimizers/trijoint.py.151: total_norm: 28.255692 [I 2022-10-27 22:32:44] ...trap/engines/engine.py.224: train: epoch 91 | batch 1910/2304 [I 2022-10-27 22:32:44] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:36 [I 2022-10-27 22:32:44] ...trap/engines/engine.py.229: process: 0.54700 | load: 0.00258 [I 2022-10-27 22:32:44] ...trap/engines/engine.py.230: loss: 0.15219 [I 2022-10-27 22:32:44] ...optimizers/trijoint.py.151: total_norm: 11.302262 [I 2022-10-27 22:32:49] ...trap/engines/engine.py.224: train: epoch 91 | batch 1920/2304 [I 2022-10-27 22:32:49] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:33 [I 2022-10-27 22:32:49] ...trap/engines/engine.py.229: process: 0.55568 | load: 0.00246 [I 2022-10-27 22:32:49] ...trap/engines/engine.py.230: loss: 0.09459 [I 2022-10-27 22:32:49] ...optimizers/trijoint.py.151: total_norm: 8.978213 [I 2022-10-27 22:32:55] ...trap/engines/engine.py.224: train: epoch 91 | batch 1930/2304 [I 2022-10-27 22:32:55] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:26 [I 2022-10-27 22:32:55] ...trap/engines/engine.py.229: process: 0.55526 | load: 0.00247 [I 2022-10-27 22:32:55] ...trap/engines/engine.py.230: loss: 0.09146 [I 2022-10-27 22:32:55] ...optimizers/trijoint.py.151: total_norm: 10.785988 [I 2022-10-27 22:33:00] ...trap/engines/engine.py.224: train: epoch 91 | batch 1940/2304 [I 2022-10-27 22:33:00] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:20 [I 2022-10-27 22:33:00] ...trap/engines/engine.py.229: process: 0.54422 | load: 0.00248 [I 2022-10-27 22:33:00] ...trap/engines/engine.py.230: loss: 0.15180 [I 2022-10-27 22:33:00] ...optimizers/trijoint.py.151: total_norm: 12.115544 [I 2022-10-27 22:33:06] ...trap/engines/engine.py.224: train: epoch 91 | batch 1950/2304 [I 2022-10-27 22:33:06] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:17 [I 2022-10-27 22:33:06] ...trap/engines/engine.py.229: process: 0.55662 | load: 0.00245 [I 2022-10-27 22:33:06] ...trap/engines/engine.py.230: loss: 0.13371 [I 2022-10-27 22:33:06] ...optimizers/trijoint.py.151: total_norm: 11.836795 [I 2022-10-27 22:33:12] ...trap/engines/engine.py.224: train: epoch 91 | batch 1960/2304 [I 2022-10-27 22:33:12] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:03:11 [I 2022-10-27 22:33:12] ...trap/engines/engine.py.229: process: 0.55099 | load: 0.00267 [I 2022-10-27 22:33:12] ...trap/engines/engine.py.230: loss: 0.09953 [I 2022-10-27 22:33:12] ...optimizers/trijoint.py.151: total_norm: 12.523155 [I 2022-10-27 22:33:17] ...trap/engines/engine.py.224: train: epoch 91 | batch 1970/2304 [I 2022-10-27 22:33:17] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:03:05 [I 2022-10-27 22:33:17] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00247 [I 2022-10-27 22:33:17] ...trap/engines/engine.py.230: loss: 0.18816 [I 2022-10-27 22:33:17] ...optimizers/trijoint.py.151: total_norm: 30.935171 [I 2022-10-27 22:33:23] ...trap/engines/engine.py.224: train: epoch 91 | batch 1980/2304 [I 2022-10-27 22:33:23] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:59 [I 2022-10-27 22:33:23] ...trap/engines/engine.py.229: process: 0.54946 | load: 0.00244 [I 2022-10-27 22:33:23] ...trap/engines/engine.py.230: loss: 0.14658 [I 2022-10-27 22:33:23] ...optimizers/trijoint.py.151: total_norm: 19.887335 [I 2022-10-27 22:33:28] ...trap/engines/engine.py.224: train: epoch 91 | batch 1990/2304 [I 2022-10-27 22:33:28] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:54 [I 2022-10-27 22:33:28] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00256 [I 2022-10-27 22:33:28] ...trap/engines/engine.py.230: loss: 0.13127 [I 2022-10-27 22:33:28] ...optimizers/trijoint.py.151: total_norm: 19.734514 [I 2022-10-27 22:33:34] ...trap/engines/engine.py.224: train: epoch 91 | batch 2000/2304 [I 2022-10-27 22:33:34] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:49 [I 2022-10-27 22:33:34] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00249 [I 2022-10-27 22:33:34] ...trap/engines/engine.py.230: loss: 0.10864 [I 2022-10-27 22:33:34] ...optimizers/trijoint.py.151: total_norm: 24.062233 [I 2022-10-27 22:33:39] ...trap/engines/engine.py.224: train: epoch 91 | batch 2010/2304 [I 2022-10-27 22:33:39] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:02:42 [I 2022-10-27 22:33:39] ...trap/engines/engine.py.229: process: 0.55319 | load: 0.00246 [I 2022-10-27 22:33:39] ...trap/engines/engine.py.230: loss: 0.11737 [I 2022-10-27 22:33:39] ...optimizers/trijoint.py.151: total_norm: 22.260605 [I 2022-10-27 22:33:45] ...trap/engines/engine.py.224: train: epoch 91 | batch 2020/2304 [I 2022-10-27 22:33:45] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:36 [I 2022-10-27 22:33:45] ...trap/engines/engine.py.229: process: 0.54714 | load: 0.00252 [I 2022-10-27 22:33:45] ...trap/engines/engine.py.230: loss: 0.16489 [I 2022-10-27 22:33:45] ...optimizers/trijoint.py.151: total_norm: 10.917778 [I 2022-10-27 22:33:50] ...trap/engines/engine.py.224: train: epoch 91 | batch 2030/2304 [I 2022-10-27 22:33:50] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:32 [I 2022-10-27 22:33:50] ...trap/engines/engine.py.229: process: 0.55209 | load: 0.00259 [I 2022-10-27 22:33:50] ...trap/engines/engine.py.230: loss: 0.14463 [I 2022-10-27 22:33:50] ...optimizers/trijoint.py.151: total_norm: 13.230556 [I 2022-10-27 22:33:56] ...trap/engines/engine.py.224: train: epoch 91 | batch 2040/2304 [I 2022-10-27 22:33:56] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:26 [I 2022-10-27 22:33:56] ...trap/engines/engine.py.229: process: 0.55131 | load: 0.00270 [I 2022-10-27 22:33:56] ...trap/engines/engine.py.230: loss: 0.10661 [I 2022-10-27 22:33:56] ...optimizers/trijoint.py.151: total_norm: 15.765711 [I 2022-10-27 22:34:02] ...trap/engines/engine.py.224: train: epoch 91 | batch 2050/2304 [I 2022-10-27 22:34:02] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:20 [I 2022-10-27 22:34:02] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00275 [I 2022-10-27 22:34:02] ...trap/engines/engine.py.230: loss: 0.10004 [I 2022-10-27 22:34:02] ...optimizers/trijoint.py.151: total_norm: 12.083591 [I 2022-10-27 22:34:07] ...trap/engines/engine.py.224: train: epoch 91 | batch 2060/2304 [I 2022-10-27 22:34:07] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:02:16 [I 2022-10-27 22:34:07] ...trap/engines/engine.py.229: process: 0.56215 | load: 0.00253 [I 2022-10-27 22:34:07] ...trap/engines/engine.py.230: loss: 0.11165 [I 2022-10-27 22:34:07] ...optimizers/trijoint.py.151: total_norm: 17.207960 [I 2022-10-27 22:34:13] ...trap/engines/engine.py.224: train: epoch 91 | batch 2070/2304 [I 2022-10-27 22:34:13] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:02:10 [I 2022-10-27 22:34:13] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00258 [I 2022-10-27 22:34:13] ...trap/engines/engine.py.230: loss: 0.10849 [I 2022-10-27 22:34:13] ...optimizers/trijoint.py.151: total_norm: 28.507597 [I 2022-10-27 22:34:18] ...trap/engines/engine.py.224: train: epoch 91 | batch 2080/2304 [I 2022-10-27 22:34:18] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:02:04 [I 2022-10-27 22:34:18] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00251 [I 2022-10-27 22:34:18] ...trap/engines/engine.py.230: loss: 0.12760 [I 2022-10-27 22:34:18] ...optimizers/trijoint.py.151: total_norm: 8.908031 [I 2022-10-27 22:34:24] ...trap/engines/engine.py.224: train: epoch 91 | batch 2090/2304 [I 2022-10-27 22:34:24] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:58 [I 2022-10-27 22:34:24] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00252 [I 2022-10-27 22:34:24] ...trap/engines/engine.py.230: loss: 0.19312 [I 2022-10-27 22:34:24] ...optimizers/trijoint.py.151: total_norm: 19.348553 [I 2022-10-27 22:34:29] ...trap/engines/engine.py.224: train: epoch 91 | batch 2100/2304 [I 2022-10-27 22:34:29] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:53 [I 2022-10-27 22:34:29] ...trap/engines/engine.py.229: process: 0.57038 | load: 0.00248 [I 2022-10-27 22:34:29] ...trap/engines/engine.py.230: loss: 0.15925 [I 2022-10-27 22:34:29] ...optimizers/trijoint.py.151: total_norm: 19.761250 [I 2022-10-27 22:34:35] ...trap/engines/engine.py.224: train: epoch 91 | batch 2110/2304 [I 2022-10-27 22:34:35] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:47 [I 2022-10-27 22:34:35] ...trap/engines/engine.py.229: process: 0.54621 | load: 0.00245 [I 2022-10-27 22:34:35] ...trap/engines/engine.py.230: loss: 0.21108 [I 2022-10-27 22:34:35] ...optimizers/trijoint.py.151: total_norm: 27.940163 [I 2022-10-27 22:34:41] ...trap/engines/engine.py.224: train: epoch 91 | batch 2120/2304 [I 2022-10-27 22:34:41] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:42 [I 2022-10-27 22:34:41] ...trap/engines/engine.py.229: process: 0.55354 | load: 0.00246 [I 2022-10-27 22:34:41] ...trap/engines/engine.py.230: loss: 0.12922 [I 2022-10-27 22:34:41] ...optimizers/trijoint.py.151: total_norm: 16.733585 [I 2022-10-27 22:34:46] ...trap/engines/engine.py.224: train: epoch 91 | batch 2130/2304 [I 2022-10-27 22:34:46] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:36 [I 2022-10-27 22:34:46] ...trap/engines/engine.py.229: process: 0.54727 | load: 0.00257 [I 2022-10-27 22:34:46] ...trap/engines/engine.py.230: loss: 0.12248 [I 2022-10-27 22:34:46] ...optimizers/trijoint.py.151: total_norm: 9.843278 [I 2022-10-27 22:34:52] ...trap/engines/engine.py.224: train: epoch 91 | batch 2140/2304 [I 2022-10-27 22:34:52] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:31 [I 2022-10-27 22:34:52] ...trap/engines/engine.py.229: process: 0.56398 | load: 0.00248 [I 2022-10-27 22:34:52] ...trap/engines/engine.py.230: loss: 0.18954 [I 2022-10-27 22:34:52] ...optimizers/trijoint.py.151: total_norm: 9.747245 [I 2022-10-27 22:34:57] ...trap/engines/engine.py.224: train: epoch 91 | batch 2150/2304 [I 2022-10-27 22:34:57] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:26 [I 2022-10-27 22:34:57] ...trap/engines/engine.py.229: process: 0.57376 | load: 0.00244 [I 2022-10-27 22:34:57] ...trap/engines/engine.py.230: loss: 0.11332 [I 2022-10-27 22:34:57] ...optimizers/trijoint.py.151: total_norm: 26.997345 [I 2022-10-27 22:35:03] ...trap/engines/engine.py.224: train: epoch 91 | batch 2160/2304 [I 2022-10-27 22:35:03] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:01:20 [I 2022-10-27 22:35:03] ...trap/engines/engine.py.229: process: 0.55345 | load: 0.00245 [I 2022-10-27 22:35:03] ...trap/engines/engine.py.230: loss: 0.19791 [I 2022-10-27 22:35:03] ...optimizers/trijoint.py.151: total_norm: 28.922724 [I 2022-10-27 22:35:09] ...trap/engines/engine.py.224: train: epoch 91 | batch 2170/2304 [I 2022-10-27 22:35:09] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:14 [I 2022-10-27 22:35:09] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00241 [I 2022-10-27 22:35:09] ...trap/engines/engine.py.230: loss: 0.10268 [I 2022-10-27 22:35:09] ...optimizers/trijoint.py.151: total_norm: 15.962688 [I 2022-10-27 22:35:14] ...trap/engines/engine.py.224: train: epoch 91 | batch 2180/2304 [I 2022-10-27 22:35:14] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:01:08 [I 2022-10-27 22:35:14] ...trap/engines/engine.py.229: process: 0.55205 | load: 0.00252 [I 2022-10-27 22:35:14] ...trap/engines/engine.py.230: loss: 0.10598 [I 2022-10-27 22:35:14] ...optimizers/trijoint.py.151: total_norm: 17.257479 [I 2022-10-27 22:35:20] ...trap/engines/engine.py.224: train: epoch 91 | batch 2190/2304 [I 2022-10-27 22:35:20] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:01:02 [I 2022-10-27 22:35:20] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00252 [I 2022-10-27 22:35:20] ...trap/engines/engine.py.230: loss: 0.11944 [I 2022-10-27 22:35:20] ...optimizers/trijoint.py.151: total_norm: 58.199074 [I 2022-10-27 22:35:25] ...trap/engines/engine.py.224: train: epoch 91 | batch 2200/2304 [I 2022-10-27 22:35:25] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:57 [I 2022-10-27 22:35:25] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00248 [I 2022-10-27 22:35:25] ...trap/engines/engine.py.230: loss: 0.12310 [I 2022-10-27 22:35:25] ...optimizers/trijoint.py.151: total_norm: 12.675361 [I 2022-10-27 22:35:31] ...trap/engines/engine.py.224: train: epoch 91 | batch 2210/2304 [I 2022-10-27 22:35:31] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:52 [I 2022-10-27 22:35:31] ...trap/engines/engine.py.229: process: 0.55567 | load: 0.00249 [I 2022-10-27 22:35:31] ...trap/engines/engine.py.230: loss: 0.15293 [I 2022-10-27 22:35:31] ...optimizers/trijoint.py.151: total_norm: 23.449492 [I 2022-10-27 22:35:36] ...trap/engines/engine.py.224: train: epoch 91 | batch 2220/2304 [I 2022-10-27 22:35:36] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:46 [I 2022-10-27 22:35:36] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00242 [I 2022-10-27 22:35:36] ...trap/engines/engine.py.230: loss: 0.10288 [I 2022-10-27 22:35:36] ...optimizers/trijoint.py.151: total_norm: 19.275879 [I 2022-10-27 22:35:42] ...trap/engines/engine.py.224: train: epoch 91 | batch 2230/2304 [I 2022-10-27 22:35:42] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:40 [I 2022-10-27 22:35:42] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00255 [I 2022-10-27 22:35:42] ...trap/engines/engine.py.230: loss: 0.12121 [I 2022-10-27 22:35:42] ...optimizers/trijoint.py.151: total_norm: 16.120813 [I 2022-10-27 22:35:48] ...trap/engines/engine.py.224: train: epoch 91 | batch 2240/2304 [I 2022-10-27 22:35:48] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:35 [I 2022-10-27 22:35:48] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00255 [I 2022-10-27 22:35:48] ...trap/engines/engine.py.230: loss: 0.16661 [I 2022-10-27 22:35:48] ...optimizers/trijoint.py.151: total_norm: 20.149948 [I 2022-10-27 22:35:53] ...trap/engines/engine.py.224: train: epoch 91 | batch 2250/2304 [I 2022-10-27 22:35:53] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:29 [I 2022-10-27 22:35:53] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00254 [I 2022-10-27 22:35:53] ...trap/engines/engine.py.230: loss: 0.09191 [I 2022-10-27 22:35:53] ...optimizers/trijoint.py.151: total_norm: 11.968895 [I 2022-10-27 22:35:59] ...trap/engines/engine.py.224: train: epoch 91 | batch 2260/2304 [I 2022-10-27 22:35:59] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:24 [I 2022-10-27 22:35:59] ...trap/engines/engine.py.229: process: 0.56578 | load: 0.00247 [I 2022-10-27 22:35:59] ...trap/engines/engine.py.230: loss: 0.10551 [I 2022-10-27 22:35:59] ...optimizers/trijoint.py.151: total_norm: 12.983461 [I 2022-10-27 22:36:04] ...trap/engines/engine.py.224: train: epoch 91 | batch 2270/2304 [I 2022-10-27 22:36:04] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:18 [I 2022-10-27 22:36:04] ...trap/engines/engine.py.229: process: 0.55272 | load: 0.00279 [I 2022-10-27 22:36:04] ...trap/engines/engine.py.230: loss: 0.11128 [I 2022-10-27 22:36:04] ...optimizers/trijoint.py.151: total_norm: 9.283220 [I 2022-10-27 22:36:10] ...trap/engines/engine.py.224: train: epoch 91 | batch 2280/2304 [I 2022-10-27 22:36:10] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:00:13 [I 2022-10-27 22:36:10] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00257 [I 2022-10-27 22:36:10] ...trap/engines/engine.py.230: loss: 0.15644 [I 2022-10-27 22:36:10] ...optimizers/trijoint.py.151: total_norm: 12.987806 [I 2022-10-27 22:36:15] ...trap/engines/engine.py.224: train: epoch 91 | batch 2290/2304 [I 2022-10-27 22:36:15] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:00:07 [I 2022-10-27 22:36:15] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00250 [I 2022-10-27 22:36:15] ...trap/engines/engine.py.230: loss: 0.13086 [I 2022-10-27 22:36:15] ...optimizers/trijoint.py.151: total_norm: 23.935757 [I 2022-10-27 22:36:21] ...trap/engines/engine.py.224: train: epoch 91 | batch 2300/2304 [I 2022-10-27 22:36:21] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:02 [I 2022-10-27 22:36:21] ...trap/engines/engine.py.229: process: 0.54924 | load: 0.00016 [I 2022-10-27 22:36:21] ...trap/engines/engine.py.230: loss: 0.08753 [I 2022-10-27 22:36:21] ...optimizers/trijoint.py.151: total_norm: 15.611055 [I 2022-10-27 22:36:23] ...trap/engines/engine.py.224: train: epoch 91 | batch 2304/2304 [I 2022-10-27 22:36:23] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:00 [I 2022-10-27 22:36:23] ...trap/engines/engine.py.229: process: 0.54671 | load: 0.00019 [I 2022-10-27 22:36:23] ...trap/engines/engine.py.230: loss: 0.13706 [I 2022-10-27 22:36:23] ...optimizers/trijoint.py.151: total_norm: 20.522690 [S 2022-10-27 22:36:25] ...trap/engines/engine.py.129: train_epoch.epoch: 91 [S 2022-10-27 22:36:25] ...trap/engines/engine.py.129: train_epoch.loss: 0.12981990512921082 [I 2022-10-27 22:36:25] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 22:36:26] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 91 [I 2022-10-27 22:36:26] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 22:36:28] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 22:36:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 22:36:28] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 22:36:31] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 22:36:31] ...trap/engines/engine.py.321: eval: epoch 91 | batch 0/510 [I 2022-10-27 22:36:31] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:47:03 [I 2022-10-27 22:36:31] ...trap/engines/engine.py.326: process: 5.53721 | load: 3.78001 [I 2022-10-27 22:36:35] ...trap/engines/engine.py.321: eval: epoch 91 | batch 10/510 [I 2022-10-27 22:36:35] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:25 [I 2022-10-27 22:36:35] ...trap/engines/engine.py.326: process: 0.32567 | load: 0.00018 [I 2022-10-27 22:36:38] ...trap/engines/engine.py.321: eval: epoch 91 | batch 20/510 [I 2022-10-27 22:36:38] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:11 [I 2022-10-27 22:36:38] ...trap/engines/engine.py.326: process: 0.32752 | load: 0.00019 [I 2022-10-27 22:36:41] ...trap/engines/engine.py.321: eval: epoch 91 | batch 30/510 [I 2022-10-27 22:36:41] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:43 [I 2022-10-27 22:36:41] ...trap/engines/engine.py.326: process: 0.34515 | load: 0.00020 [I 2022-10-27 22:36:45] ...trap/engines/engine.py.321: eval: epoch 91 | batch 40/510 [I 2022-10-27 22:36:45] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:37 [I 2022-10-27 22:36:45] ...trap/engines/engine.py.326: process: 0.34276 | load: 0.00024 [I 2022-10-27 22:36:48] ...trap/engines/engine.py.321: eval: epoch 91 | batch 50/510 [I 2022-10-27 22:36:48] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:35 [I 2022-10-27 22:36:48] ...trap/engines/engine.py.326: process: 0.34961 | load: 0.00024 [I 2022-10-27 22:36:52] ...trap/engines/engine.py.321: eval: epoch 91 | batch 60/510 [I 2022-10-27 22:36:52] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:32 [I 2022-10-27 22:36:52] ...trap/engines/engine.py.326: process: 0.35780 | load: 0.00022 [I 2022-10-27 22:36:55] ...trap/engines/engine.py.321: eval: epoch 91 | batch 70/510 [I 2022-10-27 22:36:55] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:27 [I 2022-10-27 22:36:55] ...trap/engines/engine.py.326: process: 0.33290 | load: 0.00020 [I 2022-10-27 22:36:58] ...trap/engines/engine.py.321: eval: epoch 91 | batch 80/510 [I 2022-10-27 22:36:58] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:26 [I 2022-10-27 22:36:58] ...trap/engines/engine.py.326: process: 0.34353 | load: 0.00020 [I 2022-10-27 22:37:02] ...trap/engines/engine.py.321: eval: epoch 91 | batch 90/510 [I 2022-10-27 22:37:02] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:22 [I 2022-10-27 22:37:02] ...trap/engines/engine.py.326: process: 0.35761 | load: 0.00017 [I 2022-10-27 22:37:05] ...trap/engines/engine.py.321: eval: epoch 91 | batch 100/510 [I 2022-10-27 22:37:05] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-27 22:37:05] ...trap/engines/engine.py.326: process: 0.28902 | load: 0.00019 [I 2022-10-27 22:37:08] ...trap/engines/engine.py.321: eval: epoch 91 | batch 110/510 [I 2022-10-27 22:37:08] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-27 22:37:08] ...trap/engines/engine.py.326: process: 0.28620 | load: 0.00020 [I 2022-10-27 22:37:11] ...trap/engines/engine.py.321: eval: epoch 91 | batch 120/510 [I 2022-10-27 22:37:11] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:58 [I 2022-10-27 22:37:11] ...trap/engines/engine.py.326: process: 0.36566 | load: 0.00016 [I 2022-10-27 22:37:14] ...trap/engines/engine.py.321: eval: epoch 91 | batch 130/510 [I 2022-10-27 22:37:14] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:51 [I 2022-10-27 22:37:14] ...trap/engines/engine.py.326: process: 0.28833 | load: 0.00028 [I 2022-10-27 22:37:17] ...trap/engines/engine.py.321: eval: epoch 91 | batch 140/510 [I 2022-10-27 22:37:17] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-27 22:37:17] ...trap/engines/engine.py.326: process: 0.28170 | load: 0.00024 [I 2022-10-27 22:37:20] ...trap/engines/engine.py.321: eval: epoch 91 | batch 150/510 [I 2022-10-27 22:37:20] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:44 [I 2022-10-27 22:37:20] ...trap/engines/engine.py.326: process: 0.29389 | load: 0.00017 [I 2022-10-27 22:37:23] ...trap/engines/engine.py.321: eval: epoch 91 | batch 160/510 [I 2022-10-27 22:37:23] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:42 [I 2022-10-27 22:37:23] ...trap/engines/engine.py.326: process: 0.28276 | load: 0.00024 [I 2022-10-27 22:37:25] ...trap/engines/engine.py.321: eval: epoch 91 | batch 170/510 [I 2022-10-27 22:37:25] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-27 22:37:25] ...trap/engines/engine.py.326: process: 0.28170 | load: 0.00014 [I 2022-10-27 22:37:28] ...trap/engines/engine.py.321: eval: epoch 91 | batch 180/510 [I 2022-10-27 22:37:28] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-27 22:37:28] ...trap/engines/engine.py.326: process: 0.29066 | load: 0.00015 [I 2022-10-27 22:37:31] ...trap/engines/engine.py.321: eval: epoch 91 | batch 190/510 [I 2022-10-27 22:37:31] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:32 [I 2022-10-27 22:37:31] ...trap/engines/engine.py.326: process: 0.28838 | load: 0.00016 [I 2022-10-27 22:37:34] ...trap/engines/engine.py.321: eval: epoch 91 | batch 200/510 [I 2022-10-27 22:37:34] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:29 [I 2022-10-27 22:37:34] ...trap/engines/engine.py.326: process: 0.28663 | load: 0.00015 [I 2022-10-27 22:37:37] ...trap/engines/engine.py.321: eval: epoch 91 | batch 210/510 [I 2022-10-27 22:37:37] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-27 22:37:37] ...trap/engines/engine.py.326: process: 0.28827 | load: 0.00015 [I 2022-10-27 22:37:40] ...trap/engines/engine.py.321: eval: epoch 91 | batch 220/510 [I 2022-10-27 22:37:40] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 22:37:40] ...trap/engines/engine.py.326: process: 0.28510 | load: 0.00017 [I 2022-10-27 22:37:43] ...trap/engines/engine.py.321: eval: epoch 91 | batch 230/510 [I 2022-10-27 22:37:43] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 22:37:43] ...trap/engines/engine.py.326: process: 0.28632 | load: 0.00022 [I 2022-10-27 22:37:46] ...trap/engines/engine.py.321: eval: epoch 91 | batch 240/510 [I 2022-10-27 22:37:46] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-27 22:37:46] ...trap/engines/engine.py.326: process: 0.29080 | load: 0.00018 [I 2022-10-27 22:37:49] ...trap/engines/engine.py.321: eval: epoch 91 | batch 250/510 [I 2022-10-27 22:37:49] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:16 [I 2022-10-27 22:37:49] ...trap/engines/engine.py.326: process: 0.28297 | load: 0.00022 [I 2022-10-27 22:37:52] ...trap/engines/engine.py.321: eval: epoch 91 | batch 260/510 [I 2022-10-27 22:37:52] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-27 22:37:52] ...trap/engines/engine.py.326: process: 0.29124 | load: 0.00016 [I 2022-10-27 22:37:54] ...trap/engines/engine.py.321: eval: epoch 91 | batch 270/510 [I 2022-10-27 22:37:54] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-27 22:37:54] ...trap/engines/engine.py.326: process: 0.29845 | load: 0.00016 [I 2022-10-27 22:37:57] ...trap/engines/engine.py.321: eval: epoch 91 | batch 280/510 [I 2022-10-27 22:37:57] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-27 22:37:57] ...trap/engines/engine.py.326: process: 0.29092 | load: 0.00016 [I 2022-10-27 22:38:00] ...trap/engines/engine.py.321: eval: epoch 91 | batch 290/510 [I 2022-10-27 22:38:00] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-27 22:38:00] ...trap/engines/engine.py.326: process: 0.28808 | load: 0.00017 [I 2022-10-27 22:38:03] ...trap/engines/engine.py.321: eval: epoch 91 | batch 300/510 [I 2022-10-27 22:38:03] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-27 22:38:03] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00015 [I 2022-10-27 22:38:06] ...trap/engines/engine.py.321: eval: epoch 91 | batch 310/510 [I 2022-10-27 22:38:06] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:58 [I 2022-10-27 22:38:06] ...trap/engines/engine.py.326: process: 0.30827 | load: 0.00015 [I 2022-10-27 22:38:09] ...trap/engines/engine.py.321: eval: epoch 91 | batch 320/510 [I 2022-10-27 22:38:09] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 22:38:09] ...trap/engines/engine.py.326: process: 0.28837 | load: 0.00015 [I 2022-10-27 22:38:12] ...trap/engines/engine.py.321: eval: epoch 91 | batch 330/510 [I 2022-10-27 22:38:12] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-27 22:38:12] ...trap/engines/engine.py.326: process: 0.28728 | load: 0.00016 [I 2022-10-27 22:38:15] ...trap/engines/engine.py.321: eval: epoch 91 | batch 340/510 [I 2022-10-27 22:38:15] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-27 22:38:15] ...trap/engines/engine.py.326: process: 0.28711 | load: 0.00015 [I 2022-10-27 22:38:18] ...trap/engines/engine.py.321: eval: epoch 91 | batch 350/510 [I 2022-10-27 22:38:18] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:45 [I 2022-10-27 22:38:18] ...trap/engines/engine.py.326: process: 0.29222 | load: 0.00015 [I 2022-10-27 22:38:20] ...trap/engines/engine.py.321: eval: epoch 91 | batch 360/510 [I 2022-10-27 22:38:20] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-27 22:38:20] ...trap/engines/engine.py.326: process: 0.29193 | load: 0.00018 [I 2022-10-27 22:38:23] ...trap/engines/engine.py.321: eval: epoch 91 | batch 370/510 [I 2022-10-27 22:38:23] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-27 22:38:23] ...trap/engines/engine.py.326: process: 0.28551 | load: 0.00014 [I 2022-10-27 22:38:26] ...trap/engines/engine.py.321: eval: epoch 91 | batch 380/510 [I 2022-10-27 22:38:26] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-27 22:38:26] ...trap/engines/engine.py.326: process: 0.28450 | load: 0.00015 [I 2022-10-27 22:38:29] ...trap/engines/engine.py.321: eval: epoch 91 | batch 390/510 [I 2022-10-27 22:38:29] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-27 22:38:29] ...trap/engines/engine.py.326: process: 0.29211 | load: 0.00017 [I 2022-10-27 22:38:32] ...trap/engines/engine.py.321: eval: epoch 91 | batch 400/510 [I 2022-10-27 22:38:32] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:31 [I 2022-10-27 22:38:32] ...trap/engines/engine.py.326: process: 0.28666 | load: 0.00016 [I 2022-10-27 22:38:35] ...trap/engines/engine.py.321: eval: epoch 91 | batch 410/510 [I 2022-10-27 22:38:35] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 22:38:35] ...trap/engines/engine.py.326: process: 0.28845 | load: 0.00016 [I 2022-10-27 22:38:38] ...trap/engines/engine.py.321: eval: epoch 91 | batch 420/510 [I 2022-10-27 22:38:38] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 22:38:38] ...trap/engines/engine.py.326: process: 0.28785 | load: 0.00022 [I 2022-10-27 22:38:41] ...trap/engines/engine.py.321: eval: epoch 91 | batch 430/510 [I 2022-10-27 22:38:41] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:22 [I 2022-10-27 22:38:41] ...trap/engines/engine.py.326: process: 0.29227 | load: 0.00016 [I 2022-10-27 22:38:44] ...trap/engines/engine.py.321: eval: epoch 91 | batch 440/510 [I 2022-10-27 22:38:44] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 22:38:44] ...trap/engines/engine.py.326: process: 0.29103 | load: 0.00015 [I 2022-10-27 22:38:46] ...trap/engines/engine.py.321: eval: epoch 91 | batch 450/510 [I 2022-10-27 22:38:46] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 22:38:46] ...trap/engines/engine.py.326: process: 0.28667 | load: 0.00017 [I 2022-10-27 22:38:49] ...trap/engines/engine.py.321: eval: epoch 91 | batch 460/510 [I 2022-10-27 22:38:49] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-27 22:38:49] ...trap/engines/engine.py.326: process: 0.28388 | load: 0.00016 [I 2022-10-27 22:38:52] ...trap/engines/engine.py.321: eval: epoch 91 | batch 470/510 [I 2022-10-27 22:38:52] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:12 [I 2022-10-27 22:38:52] ...trap/engines/engine.py.326: process: 0.37489 | load: 0.00021 [I 2022-10-27 22:38:55] ...trap/engines/engine.py.321: eval: epoch 91 | batch 480/510 [I 2022-10-27 22:38:55] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-27 22:38:55] ...trap/engines/engine.py.326: process: 0.28957 | load: 0.00021 [I 2022-10-27 22:38:58] ...trap/engines/engine.py.321: eval: epoch 91 | batch 490/510 [I 2022-10-27 22:38:58] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-27 22:38:58] ...trap/engines/engine.py.326: process: 0.28597 | load: 0.00016 [I 2022-10-27 22:39:01] ...trap/engines/engine.py.321: eval: epoch 91 | batch 500/510 [I 2022-10-27 22:39:01] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 22:39:01] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00015 [I 2022-10-27 22:39:04] ...trap/engines/engine.py.321: eval: epoch 91 | batch 510/510 [I 2022-10-27 22:39:04] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 22:39:04] ...trap/engines/engine.py.326: process: 0.28914 | load: 0.00011 [S 2022-10-27 22:39:05] ...trap/engines/engine.py.132: eval_epoch.epoch: 91 [I 2022-10-27 22:39:05] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.731800 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.914600 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.940500 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.729700 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.910700 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.942500 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015664 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013162 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010385 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012783 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011992 [S 2022-10-27 22:39:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010819 [I 2022-10-27 22:39:10] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 22:39:10] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 22:39:11] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 22:39:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 22:39:11] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 22:39:15] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 22:39:16] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 22:39:22] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 22:39:22] ...trap/engines/engine.py.163: Training model on trainset for epoch 92 [I 2022-10-27 22:39:22] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 22:39:25] ...trap/engines/engine.py.224: train: epoch 92 | batch 0/2304 [I 2022-10-27 22:39:25] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:49 [I 2022-10-27 22:39:25] ...trap/engines/engine.py.229: process: 2.72977 | load: 2.10525 [I 2022-10-27 22:39:25] ...trap/engines/engine.py.230: loss: 0.14249 [I 2022-10-27 22:39:25] ...optimizers/trijoint.py.151: total_norm: 62.556602 [I 2022-10-27 22:39:30] ...trap/engines/engine.py.224: train: epoch 92 | batch 10/2304 [I 2022-10-27 22:39:30] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:28 [I 2022-10-27 22:39:30] ...trap/engines/engine.py.229: process: 0.57586 | load: 0.00391 [I 2022-10-27 22:39:30] ...trap/engines/engine.py.230: loss: 0.09550 [I 2022-10-27 22:39:30] ...optimizers/trijoint.py.151: total_norm: 18.593868 [I 2022-10-27 22:39:36] ...trap/engines/engine.py.224: train: epoch 92 | batch 20/2304 [I 2022-10-27 22:39:36] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:08 [I 2022-10-27 22:39:36] ...trap/engines/engine.py.229: process: 0.54497 | load: 0.00325 [I 2022-10-27 22:39:36] ...trap/engines/engine.py.230: loss: 0.12742 [I 2022-10-27 22:39:36] ...optimizers/trijoint.py.151: total_norm: 30.306023 [I 2022-10-27 22:39:41] ...trap/engines/engine.py.224: train: epoch 92 | batch 30/2304 [I 2022-10-27 22:39:41] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:30 [I 2022-10-27 22:39:41] ...trap/engines/engine.py.229: process: 0.56126 | load: 0.00345 [I 2022-10-27 22:39:41] ...trap/engines/engine.py.230: loss: 0.18436 [I 2022-10-27 22:39:41] ...optimizers/trijoint.py.151: total_norm: 15.339770 [I 2022-10-27 22:39:47] ...trap/engines/engine.py.224: train: epoch 92 | batch 40/2304 [I 2022-10-27 22:39:47] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:08 [I 2022-10-27 22:39:47] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00283 [I 2022-10-27 22:39:47] ...trap/engines/engine.py.230: loss: 0.11784 [I 2022-10-27 22:39:47] ...optimizers/trijoint.py.151: total_norm: 35.140461 [I 2022-10-27 22:39:53] ...trap/engines/engine.py.224: train: epoch 92 | batch 50/2304 [I 2022-10-27 22:39:53] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:53 [I 2022-10-27 22:39:53] ...trap/engines/engine.py.229: process: 0.55082 | load: 0.00345 [I 2022-10-27 22:39:53] ...trap/engines/engine.py.230: loss: 0.12583 [I 2022-10-27 22:39:53] ...optimizers/trijoint.py.151: total_norm: 10.836848 [I 2022-10-27 22:39:58] ...trap/engines/engine.py.224: train: epoch 92 | batch 60/2304 [I 2022-10-27 22:39:58] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:18 [I 2022-10-27 22:39:58] ...trap/engines/engine.py.229: process: 0.55317 | load: 0.00306 [I 2022-10-27 22:39:58] ...trap/engines/engine.py.230: loss: 0.20529 [I 2022-10-27 22:39:58] ...optimizers/trijoint.py.151: total_norm: 15.735594 [I 2022-10-27 22:40:04] ...trap/engines/engine.py.224: train: epoch 92 | batch 70/2304 [I 2022-10-27 22:40:04] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:36 [I 2022-10-27 22:40:04] ...trap/engines/engine.py.229: process: 0.54301 | load: 0.00297 [I 2022-10-27 22:40:04] ...trap/engines/engine.py.230: loss: 0.14727 [I 2022-10-27 22:40:04] ...optimizers/trijoint.py.151: total_norm: 18.658495 [I 2022-10-27 22:40:09] ...trap/engines/engine.py.224: train: epoch 92 | batch 80/2304 [I 2022-10-27 22:40:09] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:44 [I 2022-10-27 22:40:09] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00301 [I 2022-10-27 22:40:09] ...trap/engines/engine.py.230: loss: 0.11840 [I 2022-10-27 22:40:09] ...optimizers/trijoint.py.151: total_norm: 15.180011 [I 2022-10-27 22:40:15] ...trap/engines/engine.py.224: train: epoch 92 | batch 90/2304 [I 2022-10-27 22:40:15] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:37 [I 2022-10-27 22:40:15] ...trap/engines/engine.py.229: process: 0.54658 | load: 0.00286 [I 2022-10-27 22:40:15] ...trap/engines/engine.py.230: loss: 0.11683 [I 2022-10-27 22:40:15] ...optimizers/trijoint.py.151: total_norm: 9.975229 [I 2022-10-27 22:40:21] ...trap/engines/engine.py.224: train: epoch 92 | batch 100/2304 [I 2022-10-27 22:40:21] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:32 [I 2022-10-27 22:40:21] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00273 [I 2022-10-27 22:40:21] ...trap/engines/engine.py.230: loss: 0.10502 [I 2022-10-27 22:40:21] ...optimizers/trijoint.py.151: total_norm: 15.024541 [I 2022-10-27 22:40:26] ...trap/engines/engine.py.224: train: epoch 92 | batch 110/2304 [I 2022-10-27 22:40:26] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:21 [I 2022-10-27 22:40:26] ...trap/engines/engine.py.229: process: 0.55822 | load: 0.00258 [I 2022-10-27 22:40:26] ...trap/engines/engine.py.230: loss: 0.12163 [I 2022-10-27 22:40:26] ...optimizers/trijoint.py.151: total_norm: 15.527281 [I 2022-10-27 22:40:32] ...trap/engines/engine.py.224: train: epoch 92 | batch 120/2304 [I 2022-10-27 22:40:32] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:20 [I 2022-10-27 22:40:32] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00264 [I 2022-10-27 22:40:32] ...trap/engines/engine.py.230: loss: 0.09927 [I 2022-10-27 22:40:32] ...optimizers/trijoint.py.151: total_norm: 14.205590 [I 2022-10-27 22:40:37] ...trap/engines/engine.py.224: train: epoch 92 | batch 130/2304 [I 2022-10-27 22:40:37] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:12 [I 2022-10-27 22:40:37] ...trap/engines/engine.py.229: process: 0.54542 | load: 0.00252 [I 2022-10-27 22:40:37] ...trap/engines/engine.py.230: loss: 0.09215 [I 2022-10-27 22:40:37] ...optimizers/trijoint.py.151: total_norm: 15.072628 [I 2022-10-27 22:40:43] ...trap/engines/engine.py.224: train: epoch 92 | batch 140/2304 [I 2022-10-27 22:40:43] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:07 [I 2022-10-27 22:40:43] ...trap/engines/engine.py.229: process: 0.56097 | load: 0.00252 [I 2022-10-27 22:40:43] ...trap/engines/engine.py.230: loss: 0.11690 [I 2022-10-27 22:40:43] ...optimizers/trijoint.py.151: total_norm: 28.160189 [I 2022-10-27 22:40:49] ...trap/engines/engine.py.224: train: epoch 92 | batch 150/2304 [I 2022-10-27 22:40:49] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:05 [I 2022-10-27 22:40:49] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00255 [I 2022-10-27 22:40:49] ...trap/engines/engine.py.230: loss: 0.12895 [I 2022-10-27 22:40:49] ...optimizers/trijoint.py.151: total_norm: 15.054233 [I 2022-10-27 22:40:54] ...trap/engines/engine.py.224: train: epoch 92 | batch 160/2304 [I 2022-10-27 22:40:54] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:53 [I 2022-10-27 22:40:54] ...trap/engines/engine.py.229: process: 0.54931 | load: 0.00242 [I 2022-10-27 22:40:54] ...trap/engines/engine.py.230: loss: 0.11929 [I 2022-10-27 22:40:54] ...optimizers/trijoint.py.151: total_norm: 8.816130 [I 2022-10-27 22:41:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 170/2304 [I 2022-10-27 22:41:00] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:20:20 [I 2022-10-27 22:41:00] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00284 [I 2022-10-27 22:41:00] ...trap/engines/engine.py.230: loss: 0.11065 [I 2022-10-27 22:41:00] ...optimizers/trijoint.py.151: total_norm: 23.202282 [I 2022-10-27 22:41:06] ...trap/engines/engine.py.224: train: epoch 92 | batch 180/2304 [I 2022-10-27 22:41:06] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:39 [I 2022-10-27 22:41:06] ...trap/engines/engine.py.229: process: 0.54634 | load: 0.00267 [I 2022-10-27 22:41:06] ...trap/engines/engine.py.230: loss: 0.16653 [I 2022-10-27 22:41:06] ...optimizers/trijoint.py.151: total_norm: 42.325424 [I 2022-10-27 22:41:11] ...trap/engines/engine.py.224: train: epoch 92 | batch 190/2304 [I 2022-10-27 22:41:11] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:47 [I 2022-10-27 22:41:11] ...trap/engines/engine.py.229: process: 0.56497 | load: 0.00252 [I 2022-10-27 22:41:11] ...trap/engines/engine.py.230: loss: 0.17948 [I 2022-10-27 22:41:11] ...optimizers/trijoint.py.151: total_norm: 32.583488 [I 2022-10-27 22:41:17] ...trap/engines/engine.py.224: train: epoch 92 | batch 200/2304 [I 2022-10-27 22:41:17] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:32 [I 2022-10-27 22:41:17] ...trap/engines/engine.py.229: process: 0.54907 | load: 0.00246 [I 2022-10-27 22:41:17] ...trap/engines/engine.py.230: loss: 0.10872 [I 2022-10-27 22:41:17] ...optimizers/trijoint.py.151: total_norm: 12.264909 [I 2022-10-27 22:41:22] ...trap/engines/engine.py.224: train: epoch 92 | batch 210/2304 [I 2022-10-27 22:41:22] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:28 [I 2022-10-27 22:41:22] ...trap/engines/engine.py.229: process: 0.56869 | load: 0.00246 [I 2022-10-27 22:41:22] ...trap/engines/engine.py.230: loss: 0.12555 [I 2022-10-27 22:41:22] ...optimizers/trijoint.py.151: total_norm: 24.735817 [I 2022-10-27 22:41:28] ...trap/engines/engine.py.224: train: epoch 92 | batch 220/2304 [I 2022-10-27 22:41:28] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:27 [I 2022-10-27 22:41:28] ...trap/engines/engine.py.229: process: 0.55937 | load: 0.00244 [I 2022-10-27 22:41:28] ...trap/engines/engine.py.230: loss: 0.12709 [I 2022-10-27 22:41:28] ...optimizers/trijoint.py.151: total_norm: 14.472766 [I 2022-10-27 22:41:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 230/2304 [I 2022-10-27 22:41:34] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:51 [I 2022-10-27 22:41:34] ...trap/engines/engine.py.229: process: 0.55484 | load: 0.00251 [I 2022-10-27 22:41:34] ...trap/engines/engine.py.230: loss: 0.10127 [I 2022-10-27 22:41:34] ...optimizers/trijoint.py.151: total_norm: 9.623752 [I 2022-10-27 22:41:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 240/2304 [I 2022-10-27 22:41:39] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:18 [I 2022-10-27 22:41:39] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00272 [I 2022-10-27 22:41:39] ...trap/engines/engine.py.230: loss: 0.11839 [I 2022-10-27 22:41:39] ...optimizers/trijoint.py.151: total_norm: 20.956991 [I 2022-10-27 22:41:45] ...trap/engines/engine.py.224: train: epoch 92 | batch 250/2304 [I 2022-10-27 22:41:45] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:04 [I 2022-10-27 22:41:45] ...trap/engines/engine.py.229: process: 0.54506 | load: 0.00262 [I 2022-10-27 22:41:45] ...trap/engines/engine.py.230: loss: 0.11872 [I 2022-10-27 22:41:45] ...optimizers/trijoint.py.151: total_norm: 14.650870 [I 2022-10-27 22:41:50] ...trap/engines/engine.py.224: train: epoch 92 | batch 260/2304 [I 2022-10-27 22:41:50] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:01 [I 2022-10-27 22:41:50] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00246 [I 2022-10-27 22:41:50] ...trap/engines/engine.py.230: loss: 0.10401 [I 2022-10-27 22:41:50] ...optimizers/trijoint.py.151: total_norm: 19.589396 [I 2022-10-27 22:41:56] ...trap/engines/engine.py.224: train: epoch 92 | batch 270/2304 [I 2022-10-27 22:41:56] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:53 [I 2022-10-27 22:41:56] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00252 [I 2022-10-27 22:41:56] ...trap/engines/engine.py.230: loss: 0.09304 [I 2022-10-27 22:41:56] ...optimizers/trijoint.py.151: total_norm: 14.613534 [I 2022-10-27 22:42:02] ...trap/engines/engine.py.224: train: epoch 92 | batch 280/2304 [I 2022-10-27 22:42:02] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:47 [I 2022-10-27 22:42:02] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00252 [I 2022-10-27 22:42:02] ...trap/engines/engine.py.230: loss: 0.20629 [I 2022-10-27 22:42:02] ...optimizers/trijoint.py.151: total_norm: 13.969606 [I 2022-10-27 22:42:07] ...trap/engines/engine.py.224: train: epoch 92 | batch 290/2304 [I 2022-10-27 22:42:07] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:39 [I 2022-10-27 22:42:07] ...trap/engines/engine.py.229: process: 0.54967 | load: 0.00255 [I 2022-10-27 22:42:07] ...trap/engines/engine.py.230: loss: 0.10982 [I 2022-10-27 22:42:07] ...optimizers/trijoint.py.151: total_norm: 17.094124 [I 2022-10-27 22:42:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 300/2304 [I 2022-10-27 22:42:13] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:36 [I 2022-10-27 22:42:13] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00252 [I 2022-10-27 22:42:13] ...trap/engines/engine.py.230: loss: 0.13618 [I 2022-10-27 22:42:13] ...optimizers/trijoint.py.151: total_norm: 18.745626 [I 2022-10-27 22:42:18] ...trap/engines/engine.py.224: train: epoch 92 | batch 310/2304 [I 2022-10-27 22:42:18] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:41 [I 2022-10-27 22:42:18] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00265 [I 2022-10-27 22:42:18] ...trap/engines/engine.py.230: loss: 0.08958 [I 2022-10-27 22:42:18] ...optimizers/trijoint.py.151: total_norm: 14.213415 [I 2022-10-27 22:42:24] ...trap/engines/engine.py.224: train: epoch 92 | batch 320/2304 [I 2022-10-27 22:42:24] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:28 [I 2022-10-27 22:42:24] ...trap/engines/engine.py.229: process: 0.55338 | load: 0.00261 [I 2022-10-27 22:42:24] ...trap/engines/engine.py.230: loss: 0.16271 [I 2022-10-27 22:42:24] ...optimizers/trijoint.py.151: total_norm: 13.731327 [I 2022-10-27 22:42:30] ...trap/engines/engine.py.224: train: epoch 92 | batch 330/2304 [I 2022-10-27 22:42:30] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:20 [I 2022-10-27 22:42:30] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00243 [I 2022-10-27 22:42:30] ...trap/engines/engine.py.230: loss: 0.09934 [I 2022-10-27 22:42:30] ...optimizers/trijoint.py.151: total_norm: 16.591667 [I 2022-10-27 22:42:35] ...trap/engines/engine.py.224: train: epoch 92 | batch 340/2304 [I 2022-10-27 22:42:35] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:15 [I 2022-10-27 22:42:35] ...trap/engines/engine.py.229: process: 0.55343 | load: 0.00245 [I 2022-10-27 22:42:35] ...trap/engines/engine.py.230: loss: 0.15050 [I 2022-10-27 22:42:35] ...optimizers/trijoint.py.151: total_norm: 104.677200 [I 2022-10-27 22:42:41] ...trap/engines/engine.py.224: train: epoch 92 | batch 350/2304 [I 2022-10-27 22:42:41] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:10 [I 2022-10-27 22:42:41] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00245 [I 2022-10-27 22:42:41] ...trap/engines/engine.py.230: loss: 0.09072 [I 2022-10-27 22:42:41] ...optimizers/trijoint.py.151: total_norm: 13.847804 [I 2022-10-27 22:42:46] ...trap/engines/engine.py.224: train: epoch 92 | batch 360/2304 [I 2022-10-27 22:42:46] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:10 [I 2022-10-27 22:42:46] ...trap/engines/engine.py.229: process: 0.56602 | load: 0.00243 [I 2022-10-27 22:42:46] ...trap/engines/engine.py.230: loss: 0.10150 [I 2022-10-27 22:42:46] ...optimizers/trijoint.py.151: total_norm: 11.060247 [I 2022-10-27 22:42:52] ...trap/engines/engine.py.224: train: epoch 92 | batch 370/2304 [I 2022-10-27 22:42:52] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:01 [I 2022-10-27 22:42:52] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00253 [I 2022-10-27 22:42:52] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-27 22:42:52] ...optimizers/trijoint.py.151: total_norm: 9.127229 [I 2022-10-27 22:42:58] ...trap/engines/engine.py.224: train: epoch 92 | batch 380/2304 [I 2022-10-27 22:42:58] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:18:00 [I 2022-10-27 22:42:58] ...trap/engines/engine.py.229: process: 0.56432 | load: 0.00249 [I 2022-10-27 22:42:58] ...trap/engines/engine.py.230: loss: 0.10367 [I 2022-10-27 22:42:58] ...optimizers/trijoint.py.151: total_norm: 12.452922 [I 2022-10-27 22:43:03] ...trap/engines/engine.py.224: train: epoch 92 | batch 390/2304 [I 2022-10-27 22:43:03] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:38 [I 2022-10-27 22:43:03] ...trap/engines/engine.py.229: process: 0.54563 | load: 0.00247 [I 2022-10-27 22:43:03] ...trap/engines/engine.py.230: loss: 0.09150 [I 2022-10-27 22:43:03] ...optimizers/trijoint.py.151: total_norm: 13.405381 [I 2022-10-27 22:43:09] ...trap/engines/engine.py.224: train: epoch 92 | batch 400/2304 [I 2022-10-27 22:43:09] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:40 [I 2022-10-27 22:43:09] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00243 [I 2022-10-27 22:43:09] ...trap/engines/engine.py.230: loss: 0.12513 [I 2022-10-27 22:43:09] ...optimizers/trijoint.py.151: total_norm: 11.169928 [I 2022-10-27 22:43:14] ...trap/engines/engine.py.224: train: epoch 92 | batch 410/2304 [I 2022-10-27 22:43:14] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:36 [I 2022-10-27 22:43:14] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00267 [I 2022-10-27 22:43:14] ...trap/engines/engine.py.230: loss: 0.12334 [I 2022-10-27 22:43:14] ...optimizers/trijoint.py.151: total_norm: 24.827522 [I 2022-10-27 22:43:20] ...trap/engines/engine.py.224: train: epoch 92 | batch 420/2304 [I 2022-10-27 22:43:20] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:37 [I 2022-10-27 22:43:20] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00249 [I 2022-10-27 22:43:20] ...trap/engines/engine.py.230: loss: 0.14540 [I 2022-10-27 22:43:20] ...optimizers/trijoint.py.151: total_norm: 15.901801 [I 2022-10-27 22:43:26] ...trap/engines/engine.py.224: train: epoch 92 | batch 430/2304 [I 2022-10-27 22:43:26] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:36 [I 2022-10-27 22:43:26] ...trap/engines/engine.py.229: process: 0.55695 | load: 0.00247 [I 2022-10-27 22:43:26] ...trap/engines/engine.py.230: loss: 0.13426 [I 2022-10-27 22:43:26] ...optimizers/trijoint.py.151: total_norm: 18.011917 [I 2022-10-27 22:43:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 440/2304 [I 2022-10-27 22:43:31] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:13 [I 2022-10-27 22:43:31] ...trap/engines/engine.py.229: process: 0.54458 | load: 0.00255 [I 2022-10-27 22:43:31] ...trap/engines/engine.py.230: loss: 0.08240 [I 2022-10-27 22:43:31] ...optimizers/trijoint.py.151: total_norm: 10.925791 [I 2022-10-27 22:43:37] ...trap/engines/engine.py.224: train: epoch 92 | batch 450/2304 [I 2022-10-27 22:43:37] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:20 [I 2022-10-27 22:43:37] ...trap/engines/engine.py.229: process: 0.56312 | load: 0.00253 [I 2022-10-27 22:43:37] ...trap/engines/engine.py.230: loss: 0.14964 [I 2022-10-27 22:43:37] ...optimizers/trijoint.py.151: total_norm: 19.115849 [I 2022-10-27 22:43:43] ...trap/engines/engine.py.224: train: epoch 92 | batch 460/2304 [I 2022-10-27 22:43:43] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:22 [I 2022-10-27 22:43:43] ...trap/engines/engine.py.229: process: 0.57239 | load: 0.00257 [I 2022-10-27 22:43:43] ...trap/engines/engine.py.230: loss: 0.10921 [I 2022-10-27 22:43:43] ...optimizers/trijoint.py.151: total_norm: 19.216139 [I 2022-10-27 22:43:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 470/2304 [I 2022-10-27 22:43:48] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:04 [I 2022-10-27 22:43:48] ...trap/engines/engine.py.229: process: 0.54378 | load: 0.00263 [I 2022-10-27 22:43:48] ...trap/engines/engine.py.230: loss: 0.20988 [I 2022-10-27 22:43:48] ...optimizers/trijoint.py.151: total_norm: 15.562507 [I 2022-10-27 22:43:54] ...trap/engines/engine.py.224: train: epoch 92 | batch 480/2304 [I 2022-10-27 22:43:54] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:01 [I 2022-10-27 22:43:54] ...trap/engines/engine.py.229: process: 0.57111 | load: 0.00243 [I 2022-10-27 22:43:54] ...trap/engines/engine.py.230: loss: 0.15070 [I 2022-10-27 22:43:54] ...optimizers/trijoint.py.151: total_norm: 21.399641 [I 2022-10-27 22:43:59] ...trap/engines/engine.py.224: train: epoch 92 | batch 490/2304 [I 2022-10-27 22:43:59] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:50 [I 2022-10-27 22:43:59] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00256 [I 2022-10-27 22:43:59] ...trap/engines/engine.py.230: loss: 0.12039 [I 2022-10-27 22:43:59] ...optimizers/trijoint.py.151: total_norm: 27.884750 [I 2022-10-27 22:44:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 500/2304 [I 2022-10-27 22:44:05] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:55 [I 2022-10-27 22:44:05] ...trap/engines/engine.py.229: process: 0.57459 | load: 0.00256 [I 2022-10-27 22:44:05] ...trap/engines/engine.py.230: loss: 0.14228 [I 2022-10-27 22:44:05] ...optimizers/trijoint.py.151: total_norm: 13.373912 [I 2022-10-27 22:44:10] ...trap/engines/engine.py.224: train: epoch 92 | batch 510/2304 [I 2022-10-27 22:44:10] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:33 [I 2022-10-27 22:44:10] ...trap/engines/engine.py.229: process: 0.55069 | load: 0.00254 [I 2022-10-27 22:44:10] ...trap/engines/engine.py.230: loss: 0.07574 [I 2022-10-27 22:44:10] ...optimizers/trijoint.py.151: total_norm: 9.812350 [I 2022-10-27 22:44:16] ...trap/engines/engine.py.224: train: epoch 92 | batch 520/2304 [I 2022-10-27 22:44:16] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:36 [I 2022-10-27 22:44:16] ...trap/engines/engine.py.229: process: 0.56240 | load: 0.00247 [I 2022-10-27 22:44:16] ...trap/engines/engine.py.230: loss: 0.09750 [I 2022-10-27 22:44:16] ...optimizers/trijoint.py.151: total_norm: 25.498713 [I 2022-10-27 22:44:22] ...trap/engines/engine.py.224: train: epoch 92 | batch 530/2304 [I 2022-10-27 22:44:22] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:34 [I 2022-10-27 22:44:22] ...trap/engines/engine.py.229: process: 0.56225 | load: 0.00245 [I 2022-10-27 22:44:22] ...trap/engines/engine.py.230: loss: 0.09695 [I 2022-10-27 22:44:22] ...optimizers/trijoint.py.151: total_norm: 8.605498 [I 2022-10-27 22:44:27] ...trap/engines/engine.py.224: train: epoch 92 | batch 540/2304 [I 2022-10-27 22:44:27] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:26 [I 2022-10-27 22:44:27] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00241 [I 2022-10-27 22:44:27] ...trap/engines/engine.py.230: loss: 0.12507 [I 2022-10-27 22:44:27] ...optimizers/trijoint.py.151: total_norm: 13.046182 [I 2022-10-27 22:44:33] ...trap/engines/engine.py.224: train: epoch 92 | batch 550/2304 [I 2022-10-27 22:44:33] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:26 [I 2022-10-27 22:44:33] ...trap/engines/engine.py.229: process: 0.56890 | load: 0.00246 [I 2022-10-27 22:44:33] ...trap/engines/engine.py.230: loss: 0.10742 [I 2022-10-27 22:44:33] ...optimizers/trijoint.py.151: total_norm: 22.427048 [I 2022-10-27 22:44:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 560/2304 [I 2022-10-27 22:44:39] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:18 [I 2022-10-27 22:44:39] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00243 [I 2022-10-27 22:44:39] ...trap/engines/engine.py.230: loss: 0.21176 [I 2022-10-27 22:44:39] ...optimizers/trijoint.py.151: total_norm: 11.049100 [I 2022-10-27 22:44:44] ...trap/engines/engine.py.224: train: epoch 92 | batch 570/2304 [I 2022-10-27 22:44:44] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:15 [I 2022-10-27 22:44:44] ...trap/engines/engine.py.229: process: 0.56476 | load: 0.00266 [I 2022-10-27 22:44:44] ...trap/engines/engine.py.230: loss: 0.12632 [I 2022-10-27 22:44:44] ...optimizers/trijoint.py.151: total_norm: 20.067451 [I 2022-10-27 22:44:50] ...trap/engines/engine.py.224: train: epoch 92 | batch 580/2304 [I 2022-10-27 22:44:50] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:08 [I 2022-10-27 22:44:50] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00252 [I 2022-10-27 22:44:50] ...trap/engines/engine.py.230: loss: 0.14202 [I 2022-10-27 22:44:50] ...optimizers/trijoint.py.151: total_norm: 22.271166 [I 2022-10-27 22:44:55] ...trap/engines/engine.py.224: train: epoch 92 | batch 590/2304 [I 2022-10-27 22:44:55] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:57 [I 2022-10-27 22:44:55] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00253 [I 2022-10-27 22:44:55] ...trap/engines/engine.py.230: loss: 0.11580 [I 2022-10-27 22:44:55] ...optimizers/trijoint.py.151: total_norm: 31.925158 [I 2022-10-27 22:45:01] ...trap/engines/engine.py.224: train: epoch 92 | batch 600/2304 [I 2022-10-27 22:45:01] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:47 [I 2022-10-27 22:45:01] ...trap/engines/engine.py.229: process: 0.55902 | load: 0.00250 [I 2022-10-27 22:45:01] ...trap/engines/engine.py.230: loss: 0.09792 [I 2022-10-27 22:45:01] ...optimizers/trijoint.py.151: total_norm: 10.821754 [I 2022-10-27 22:45:07] ...trap/engines/engine.py.224: train: epoch 92 | batch 610/2304 [I 2022-10-27 22:45:07] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:36 [I 2022-10-27 22:45:07] ...trap/engines/engine.py.229: process: 0.55442 | load: 0.00244 [I 2022-10-27 22:45:07] ...trap/engines/engine.py.230: loss: 0.18524 [I 2022-10-27 22:45:07] ...optimizers/trijoint.py.151: total_norm: 27.592300 [I 2022-10-27 22:45:12] ...trap/engines/engine.py.224: train: epoch 92 | batch 620/2304 [I 2022-10-27 22:45:12] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:43 [I 2022-10-27 22:45:12] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00249 [I 2022-10-27 22:45:12] ...trap/engines/engine.py.230: loss: 0.11326 [I 2022-10-27 22:45:12] ...optimizers/trijoint.py.151: total_norm: 8.892258 [I 2022-10-27 22:45:18] ...trap/engines/engine.py.224: train: epoch 92 | batch 630/2304 [I 2022-10-27 22:45:18] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:34 [I 2022-10-27 22:45:18] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00245 [I 2022-10-27 22:45:18] ...trap/engines/engine.py.230: loss: 0.12317 [I 2022-10-27 22:45:18] ...optimizers/trijoint.py.151: total_norm: 10.320441 [I 2022-10-27 22:45:23] ...trap/engines/engine.py.224: train: epoch 92 | batch 640/2304 [I 2022-10-27 22:45:23] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:31 [I 2022-10-27 22:45:23] ...trap/engines/engine.py.229: process: 0.56704 | load: 0.00241 [I 2022-10-27 22:45:23] ...trap/engines/engine.py.230: loss: 0.09909 [I 2022-10-27 22:45:23] ...optimizers/trijoint.py.151: total_norm: 9.108424 [I 2022-10-27 22:45:29] ...trap/engines/engine.py.224: train: epoch 92 | batch 650/2304 [I 2022-10-27 22:45:29] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:23 [I 2022-10-27 22:45:29] ...trap/engines/engine.py.229: process: 0.54982 | load: 0.00247 [I 2022-10-27 22:45:29] ...trap/engines/engine.py.230: loss: 0.09248 [I 2022-10-27 22:45:29] ...optimizers/trijoint.py.151: total_norm: 8.085283 [I 2022-10-27 22:45:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 660/2304 [I 2022-10-27 22:45:34] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:18 [I 2022-10-27 22:45:34] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00248 [I 2022-10-27 22:45:34] ...trap/engines/engine.py.230: loss: 0.10101 [I 2022-10-27 22:45:34] ...optimizers/trijoint.py.151: total_norm: 10.358898 [I 2022-10-27 22:45:40] ...trap/engines/engine.py.224: train: epoch 92 | batch 670/2304 [I 2022-10-27 22:45:40] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:13 [I 2022-10-27 22:45:40] ...trap/engines/engine.py.229: process: 0.55351 | load: 0.00245 [I 2022-10-27 22:45:40] ...trap/engines/engine.py.230: loss: 0.15442 [I 2022-10-27 22:45:40] ...optimizers/trijoint.py.151: total_norm: 11.311195 [I 2022-10-27 22:45:46] ...trap/engines/engine.py.224: train: epoch 92 | batch 680/2304 [I 2022-10-27 22:45:46] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:06 [I 2022-10-27 22:45:46] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00245 [I 2022-10-27 22:45:46] ...trap/engines/engine.py.230: loss: 0.21597 [I 2022-10-27 22:45:46] ...optimizers/trijoint.py.151: total_norm: 16.028084 [I 2022-10-27 22:45:51] ...trap/engines/engine.py.224: train: epoch 92 | batch 690/2304 [I 2022-10-27 22:45:51] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:02 [I 2022-10-27 22:45:51] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00242 [I 2022-10-27 22:45:51] ...trap/engines/engine.py.230: loss: 0.10599 [I 2022-10-27 22:45:51] ...optimizers/trijoint.py.151: total_norm: 11.249138 [I 2022-10-27 22:45:57] ...trap/engines/engine.py.224: train: epoch 92 | batch 700/2304 [I 2022-10-27 22:45:57] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:15:02 [I 2022-10-27 22:45:57] ...trap/engines/engine.py.229: process: 0.56024 | load: 0.00243 [I 2022-10-27 22:45:57] ...trap/engines/engine.py.230: loss: 0.16173 [I 2022-10-27 22:45:57] ...optimizers/trijoint.py.151: total_norm: 25.565788 [I 2022-10-27 22:46:02] ...trap/engines/engine.py.224: train: epoch 92 | batch 710/2304 [I 2022-10-27 22:46:02] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:50 [I 2022-10-27 22:46:02] ...trap/engines/engine.py.229: process: 0.56665 | load: 0.00245 [I 2022-10-27 22:46:02] ...trap/engines/engine.py.230: loss: 0.11318 [I 2022-10-27 22:46:02] ...optimizers/trijoint.py.151: total_norm: 19.097944 [I 2022-10-27 22:46:08] ...trap/engines/engine.py.224: train: epoch 92 | batch 720/2304 [I 2022-10-27 22:46:08] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:48 [I 2022-10-27 22:46:08] ...trap/engines/engine.py.229: process: 0.56468 | load: 0.00241 [I 2022-10-27 22:46:08] ...trap/engines/engine.py.230: loss: 0.09896 [I 2022-10-27 22:46:08] ...optimizers/trijoint.py.151: total_norm: 9.153981 [I 2022-10-27 22:46:14] ...trap/engines/engine.py.224: train: epoch 92 | batch 730/2304 [I 2022-10-27 22:46:14] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:37 [I 2022-10-27 22:46:14] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00242 [I 2022-10-27 22:46:14] ...trap/engines/engine.py.230: loss: 0.12139 [I 2022-10-27 22:46:14] ...optimizers/trijoint.py.151: total_norm: 17.016502 [I 2022-10-27 22:46:19] ...trap/engines/engine.py.224: train: epoch 92 | batch 740/2304 [I 2022-10-27 22:46:19] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:28 [I 2022-10-27 22:46:19] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00243 [I 2022-10-27 22:46:19] ...trap/engines/engine.py.230: loss: 0.14376 [I 2022-10-27 22:46:19] ...optimizers/trijoint.py.151: total_norm: 14.291116 [I 2022-10-27 22:46:25] ...trap/engines/engine.py.224: train: epoch 92 | batch 750/2304 [I 2022-10-27 22:46:25] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:27 [I 2022-10-27 22:46:25] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00242 [I 2022-10-27 22:46:25] ...trap/engines/engine.py.230: loss: 0.10756 [I 2022-10-27 22:46:25] ...optimizers/trijoint.py.151: total_norm: 14.573570 [I 2022-10-27 22:46:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 760/2304 [I 2022-10-27 22:46:31] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:40 [I 2022-10-27 22:46:31] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00249 [I 2022-10-27 22:46:31] ...trap/engines/engine.py.230: loss: 0.14077 [I 2022-10-27 22:46:31] ...optimizers/trijoint.py.151: total_norm: 16.044783 [I 2022-10-27 22:46:36] ...trap/engines/engine.py.224: train: epoch 92 | batch 770/2304 [I 2022-10-27 22:46:36] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:16 [I 2022-10-27 22:46:36] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00254 [I 2022-10-27 22:46:36] ...trap/engines/engine.py.230: loss: 0.19624 [I 2022-10-27 22:46:36] ...optimizers/trijoint.py.151: total_norm: 18.910978 [I 2022-10-27 22:46:42] ...trap/engines/engine.py.224: train: epoch 92 | batch 780/2304 [I 2022-10-27 22:46:42] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:31 [I 2022-10-27 22:46:42] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00245 [I 2022-10-27 22:46:42] ...trap/engines/engine.py.230: loss: 0.10291 [I 2022-10-27 22:46:42] ...optimizers/trijoint.py.151: total_norm: 13.277966 [I 2022-10-27 22:46:47] ...trap/engines/engine.py.224: train: epoch 92 | batch 790/2304 [I 2022-10-27 22:46:47] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:04 [I 2022-10-27 22:46:47] ...trap/engines/engine.py.229: process: 0.54074 | load: 0.00244 [I 2022-10-27 22:46:47] ...trap/engines/engine.py.230: loss: 0.15300 [I 2022-10-27 22:46:47] ...optimizers/trijoint.py.151: total_norm: 19.792765 [I 2022-10-27 22:46:53] ...trap/engines/engine.py.224: train: epoch 92 | batch 800/2304 [I 2022-10-27 22:46:53] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:13:49 [I 2022-10-27 22:46:53] ...trap/engines/engine.py.229: process: 0.54199 | load: 0.00242 [I 2022-10-27 22:46:53] ...trap/engines/engine.py.230: loss: 0.11990 [I 2022-10-27 22:46:53] ...optimizers/trijoint.py.151: total_norm: 22.488148 [I 2022-10-27 22:46:58] ...trap/engines/engine.py.224: train: epoch 92 | batch 810/2304 [I 2022-10-27 22:46:58] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:43 [I 2022-10-27 22:46:58] ...trap/engines/engine.py.229: process: 0.54830 | load: 0.00243 [I 2022-10-27 22:46:58] ...trap/engines/engine.py.230: loss: 0.10271 [I 2022-10-27 22:46:58] ...optimizers/trijoint.py.151: total_norm: 10.048986 [I 2022-10-27 22:47:04] ...trap/engines/engine.py.224: train: epoch 92 | batch 820/2304 [I 2022-10-27 22:47:04] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:43 [I 2022-10-27 22:47:04] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00243 [I 2022-10-27 22:47:04] ...trap/engines/engine.py.230: loss: 0.13403 [I 2022-10-27 22:47:04] ...optimizers/trijoint.py.151: total_norm: 41.893303 [I 2022-10-27 22:47:10] ...trap/engines/engine.py.224: train: epoch 92 | batch 830/2304 [I 2022-10-27 22:47:10] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:32 [I 2022-10-27 22:47:10] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00262 [I 2022-10-27 22:47:10] ...trap/engines/engine.py.230: loss: 0.09391 [I 2022-10-27 22:47:10] ...optimizers/trijoint.py.151: total_norm: 14.342505 [I 2022-10-27 22:47:15] ...trap/engines/engine.py.224: train: epoch 92 | batch 840/2304 [I 2022-10-27 22:47:15] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:32 [I 2022-10-27 22:47:15] ...trap/engines/engine.py.229: process: 0.54928 | load: 0.00241 [I 2022-10-27 22:47:15] ...trap/engines/engine.py.230: loss: 0.10312 [I 2022-10-27 22:47:15] ...optimizers/trijoint.py.151: total_norm: 10.142986 [I 2022-10-27 22:47:21] ...trap/engines/engine.py.224: train: epoch 92 | batch 850/2304 [I 2022-10-27 22:47:21] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:29 [I 2022-10-27 22:47:21] ...trap/engines/engine.py.229: process: 0.56543 | load: 0.00245 [I 2022-10-27 22:47:21] ...trap/engines/engine.py.230: loss: 0.12967 [I 2022-10-27 22:47:21] ...optimizers/trijoint.py.151: total_norm: 44.744690 [I 2022-10-27 22:47:26] ...trap/engines/engine.py.224: train: epoch 92 | batch 860/2304 [I 2022-10-27 22:47:26] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:33 [I 2022-10-27 22:47:26] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00247 [I 2022-10-27 22:47:26] ...trap/engines/engine.py.230: loss: 0.10864 [I 2022-10-27 22:47:26] ...optimizers/trijoint.py.151: total_norm: 8.304524 [I 2022-10-27 22:47:32] ...trap/engines/engine.py.224: train: epoch 92 | batch 870/2304 [I 2022-10-27 22:47:32] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:21 [I 2022-10-27 22:47:32] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00251 [I 2022-10-27 22:47:32] ...trap/engines/engine.py.230: loss: 0.11757 [I 2022-10-27 22:47:32] ...optimizers/trijoint.py.151: total_norm: 14.640189 [I 2022-10-27 22:47:38] ...trap/engines/engine.py.224: train: epoch 92 | batch 880/2304 [I 2022-10-27 22:47:38] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:19 [I 2022-10-27 22:47:38] ...trap/engines/engine.py.229: process: 0.56094 | load: 0.00245 [I 2022-10-27 22:47:38] ...trap/engines/engine.py.230: loss: 0.14643 [I 2022-10-27 22:47:38] ...optimizers/trijoint.py.151: total_norm: 38.567928 [I 2022-10-27 22:47:43] ...trap/engines/engine.py.224: train: epoch 92 | batch 890/2304 [I 2022-10-27 22:47:43] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:07 [I 2022-10-27 22:47:43] ...trap/engines/engine.py.229: process: 0.56090 | load: 0.00283 [I 2022-10-27 22:47:43] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-27 22:47:43] ...optimizers/trijoint.py.151: total_norm: 25.468864 [I 2022-10-27 22:47:49] ...trap/engines/engine.py.224: train: epoch 92 | batch 900/2304 [I 2022-10-27 22:47:49] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:02 [I 2022-10-27 22:47:49] ...trap/engines/engine.py.229: process: 0.55400 | load: 0.00250 [I 2022-10-27 22:47:49] ...trap/engines/engine.py.230: loss: 0.09143 [I 2022-10-27 22:47:49] ...optimizers/trijoint.py.151: total_norm: 13.395273 [I 2022-10-27 22:47:54] ...trap/engines/engine.py.224: train: epoch 92 | batch 910/2304 [I 2022-10-27 22:47:54] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:56 [I 2022-10-27 22:47:54] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00241 [I 2022-10-27 22:47:54] ...trap/engines/engine.py.230: loss: 0.11285 [I 2022-10-27 22:47:54] ...optimizers/trijoint.py.151: total_norm: 12.059844 [I 2022-10-27 22:48:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 920/2304 [I 2022-10-27 22:48:00] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:13:02 [I 2022-10-27 22:48:00] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00240 [I 2022-10-27 22:48:00] ...trap/engines/engine.py.230: loss: 0.11214 [I 2022-10-27 22:48:00] ...optimizers/trijoint.py.151: total_norm: 12.322497 [I 2022-10-27 22:48:06] ...trap/engines/engine.py.224: train: epoch 92 | batch 930/2304 [I 2022-10-27 22:48:06] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:45 [I 2022-10-27 22:48:06] ...trap/engines/engine.py.229: process: 0.56672 | load: 0.00243 [I 2022-10-27 22:48:06] ...trap/engines/engine.py.230: loss: 0.12715 [I 2022-10-27 22:48:06] ...optimizers/trijoint.py.151: total_norm: 12.246446 [I 2022-10-27 22:48:11] ...trap/engines/engine.py.224: train: epoch 92 | batch 940/2304 [I 2022-10-27 22:48:11] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:42 [I 2022-10-27 22:48:11] ...trap/engines/engine.py.229: process: 0.56368 | load: 0.00243 [I 2022-10-27 22:48:11] ...trap/engines/engine.py.230: loss: 0.11833 [I 2022-10-27 22:48:11] ...optimizers/trijoint.py.151: total_norm: 22.843143 [I 2022-10-27 22:48:17] ...trap/engines/engine.py.224: train: epoch 92 | batch 950/2304 [I 2022-10-27 22:48:17] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:33 [I 2022-10-27 22:48:17] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00242 [I 2022-10-27 22:48:17] ...trap/engines/engine.py.230: loss: 0.13964 [I 2022-10-27 22:48:17] ...optimizers/trijoint.py.151: total_norm: 30.674410 [I 2022-10-27 22:48:23] ...trap/engines/engine.py.224: train: epoch 92 | batch 960/2304 [I 2022-10-27 22:48:23] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:34 [I 2022-10-27 22:48:23] ...trap/engines/engine.py.229: process: 0.56786 | load: 0.00240 [I 2022-10-27 22:48:23] ...trap/engines/engine.py.230: loss: 0.12352 [I 2022-10-27 22:48:23] ...optimizers/trijoint.py.151: total_norm: 18.073957 [I 2022-10-27 22:48:28] ...trap/engines/engine.py.224: train: epoch 92 | batch 970/2304 [I 2022-10-27 22:48:28] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:22 [I 2022-10-27 22:48:28] ...trap/engines/engine.py.229: process: 0.54958 | load: 0.00242 [I 2022-10-27 22:48:28] ...trap/engines/engine.py.230: loss: 0.11516 [I 2022-10-27 22:48:28] ...optimizers/trijoint.py.151: total_norm: 12.824188 [I 2022-10-27 22:48:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 980/2304 [I 2022-10-27 22:48:34] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:17 [I 2022-10-27 22:48:34] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00244 [I 2022-10-27 22:48:34] ...trap/engines/engine.py.230: loss: 0.11145 [I 2022-10-27 22:48:34] ...optimizers/trijoint.py.151: total_norm: 31.851227 [I 2022-10-27 22:48:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 990/2304 [I 2022-10-27 22:48:39] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:08 [I 2022-10-27 22:48:39] ...trap/engines/engine.py.229: process: 0.54749 | load: 0.00240 [I 2022-10-27 22:48:39] ...trap/engines/engine.py.230: loss: 0.14341 [I 2022-10-27 22:48:39] ...optimizers/trijoint.py.151: total_norm: 25.574541 [I 2022-10-27 22:48:45] ...trap/engines/engine.py.224: train: epoch 92 | batch 1000/2304 [I 2022-10-27 22:48:45] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:05 [I 2022-10-27 22:48:45] ...trap/engines/engine.py.229: process: 0.55284 | load: 0.00253 [I 2022-10-27 22:48:45] ...trap/engines/engine.py.230: loss: 0.16245 [I 2022-10-27 22:48:45] ...optimizers/trijoint.py.151: total_norm: 19.699583 [I 2022-10-27 22:48:50] ...trap/engines/engine.py.224: train: epoch 92 | batch 1010/2304 [I 2022-10-27 22:48:50] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:00 [I 2022-10-27 22:48:50] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00247 [I 2022-10-27 22:48:50] ...trap/engines/engine.py.230: loss: 0.09810 [I 2022-10-27 22:48:50] ...optimizers/trijoint.py.151: total_norm: 11.346399 [I 2022-10-27 22:48:56] ...trap/engines/engine.py.224: train: epoch 92 | batch 1020/2304 [I 2022-10-27 22:48:56] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:57 [I 2022-10-27 22:48:56] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00244 [I 2022-10-27 22:48:56] ...trap/engines/engine.py.230: loss: 0.16464 [I 2022-10-27 22:48:56] ...optimizers/trijoint.py.151: total_norm: 13.008755 [I 2022-10-27 22:49:02] ...trap/engines/engine.py.224: train: epoch 92 | batch 1030/2304 [I 2022-10-27 22:49:02] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:54 [I 2022-10-27 22:49:02] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00243 [I 2022-10-27 22:49:02] ...trap/engines/engine.py.230: loss: 0.12433 [I 2022-10-27 22:49:02] ...optimizers/trijoint.py.151: total_norm: 21.890680 [I 2022-10-27 22:49:07] ...trap/engines/engine.py.224: train: epoch 92 | batch 1040/2304 [I 2022-10-27 22:49:07] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:12:09 [I 2022-10-27 22:49:07] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00243 [I 2022-10-27 22:49:07] ...trap/engines/engine.py.230: loss: 0.10369 [I 2022-10-27 22:49:07] ...optimizers/trijoint.py.151: total_norm: 9.830173 [I 2022-10-27 22:49:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 1050/2304 [I 2022-10-27 22:49:13] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:43 [I 2022-10-27 22:49:13] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00245 [I 2022-10-27 22:49:13] ...trap/engines/engine.py.230: loss: 0.10468 [I 2022-10-27 22:49:13] ...optimizers/trijoint.py.151: total_norm: 9.248260 [I 2022-10-27 22:49:19] ...trap/engines/engine.py.224: train: epoch 92 | batch 1060/2304 [I 2022-10-27 22:49:19] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:34 [I 2022-10-27 22:49:19] ...trap/engines/engine.py.229: process: 0.56490 | load: 0.00244 [I 2022-10-27 22:49:19] ...trap/engines/engine.py.230: loss: 0.11553 [I 2022-10-27 22:49:19] ...optimizers/trijoint.py.151: total_norm: 16.087614 [I 2022-10-27 22:49:24] ...trap/engines/engine.py.224: train: epoch 92 | batch 1070/2304 [I 2022-10-27 22:49:24] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:29 [I 2022-10-27 22:49:24] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00246 [I 2022-10-27 22:49:24] ...trap/engines/engine.py.230: loss: 0.16117 [I 2022-10-27 22:49:24] ...optimizers/trijoint.py.151: total_norm: 16.196756 [I 2022-10-27 22:49:30] ...trap/engines/engine.py.224: train: epoch 92 | batch 1080/2304 [I 2022-10-27 22:49:30] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:26 [I 2022-10-27 22:49:30] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00251 [I 2022-10-27 22:49:30] ...trap/engines/engine.py.230: loss: 0.17129 [I 2022-10-27 22:49:30] ...optimizers/trijoint.py.151: total_norm: 99.538757 [I 2022-10-27 22:49:35] ...trap/engines/engine.py.224: train: epoch 92 | batch 1090/2304 [I 2022-10-27 22:49:35] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:25 [I 2022-10-27 22:49:35] ...trap/engines/engine.py.229: process: 0.57431 | load: 0.00245 [I 2022-10-27 22:49:35] ...trap/engines/engine.py.230: loss: 0.18383 [I 2022-10-27 22:49:35] ...optimizers/trijoint.py.151: total_norm: 30.266001 [I 2022-10-27 22:49:41] ...trap/engines/engine.py.224: train: epoch 92 | batch 1100/2304 [I 2022-10-27 22:49:41] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:16 [I 2022-10-27 22:49:41] ...trap/engines/engine.py.229: process: 0.57042 | load: 0.00258 [I 2022-10-27 22:49:41] ...trap/engines/engine.py.230: loss: 0.10274 [I 2022-10-27 22:49:41] ...optimizers/trijoint.py.151: total_norm: 8.813292 [I 2022-10-27 22:49:47] ...trap/engines/engine.py.224: train: epoch 92 | batch 1110/2304 [I 2022-10-27 22:49:47] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:08 [I 2022-10-27 22:49:47] ...trap/engines/engine.py.229: process: 0.55734 | load: 0.00245 [I 2022-10-27 22:49:47] ...trap/engines/engine.py.230: loss: 0.12011 [I 2022-10-27 22:49:47] ...optimizers/trijoint.py.151: total_norm: 9.161455 [I 2022-10-27 22:49:52] ...trap/engines/engine.py.224: train: epoch 92 | batch 1120/2304 [I 2022-10-27 22:49:52] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:01 [I 2022-10-27 22:49:52] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00246 [I 2022-10-27 22:49:52] ...trap/engines/engine.py.230: loss: 0.11648 [I 2022-10-27 22:49:52] ...optimizers/trijoint.py.151: total_norm: 16.647137 [I 2022-10-27 22:49:58] ...trap/engines/engine.py.224: train: epoch 92 | batch 1130/2304 [I 2022-10-27 22:49:58] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:10:54 [I 2022-10-27 22:49:58] ...trap/engines/engine.py.229: process: 0.55482 | load: 0.00244 [I 2022-10-27 22:49:58] ...trap/engines/engine.py.230: loss: 0.13690 [I 2022-10-27 22:49:58] ...optimizers/trijoint.py.151: total_norm: 13.366940 [I 2022-10-27 22:50:04] ...trap/engines/engine.py.224: train: epoch 92 | batch 1140/2304 [I 2022-10-27 22:50:04] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:11:16 [I 2022-10-27 22:50:04] ...trap/engines/engine.py.229: process: 0.56380 | load: 0.00248 [I 2022-10-27 22:50:04] ...trap/engines/engine.py.230: loss: 0.11805 [I 2022-10-27 22:50:04] ...optimizers/trijoint.py.151: total_norm: 14.348295 [I 2022-10-27 22:50:09] ...trap/engines/engine.py.224: train: epoch 92 | batch 1150/2304 [I 2022-10-27 22:50:09] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:50 [I 2022-10-27 22:50:09] ...trap/engines/engine.py.229: process: 0.56201 | load: 0.00246 [I 2022-10-27 22:50:09] ...trap/engines/engine.py.230: loss: 0.10507 [I 2022-10-27 22:50:09] ...optimizers/trijoint.py.151: total_norm: 15.789932 [I 2022-10-27 22:50:15] ...trap/engines/engine.py.224: train: epoch 92 | batch 1160/2304 [I 2022-10-27 22:50:15] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:38 [I 2022-10-27 22:50:15] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00243 [I 2022-10-27 22:50:15] ...trap/engines/engine.py.230: loss: 0.14994 [I 2022-10-27 22:50:15] ...optimizers/trijoint.py.151: total_norm: 16.047234 [I 2022-10-27 22:50:20] ...trap/engines/engine.py.224: train: epoch 92 | batch 1170/2304 [I 2022-10-27 22:50:20] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:36 [I 2022-10-27 22:50:20] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00243 [I 2022-10-27 22:50:20] ...trap/engines/engine.py.230: loss: 0.10012 [I 2022-10-27 22:50:20] ...optimizers/trijoint.py.151: total_norm: 9.823836 [I 2022-10-27 22:50:26] ...trap/engines/engine.py.224: train: epoch 92 | batch 1180/2304 [I 2022-10-27 22:50:26] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:28 [I 2022-10-27 22:50:26] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00244 [I 2022-10-27 22:50:26] ...trap/engines/engine.py.230: loss: 0.15454 [I 2022-10-27 22:50:26] ...optimizers/trijoint.py.151: total_norm: 8.994159 [I 2022-10-27 22:50:32] ...trap/engines/engine.py.224: train: epoch 92 | batch 1190/2304 [I 2022-10-27 22:50:32] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:29 [I 2022-10-27 22:50:32] ...trap/engines/engine.py.229: process: 0.57486 | load: 0.00243 [I 2022-10-27 22:50:32] ...trap/engines/engine.py.230: loss: 0.17208 [I 2022-10-27 22:50:32] ...optimizers/trijoint.py.151: total_norm: 9.133608 [I 2022-10-27 22:50:37] ...trap/engines/engine.py.224: train: epoch 92 | batch 1200/2304 [I 2022-10-27 22:50:37] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:16 [I 2022-10-27 22:50:37] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00242 [I 2022-10-27 22:50:37] ...trap/engines/engine.py.230: loss: 0.15375 [I 2022-10-27 22:50:37] ...optimizers/trijoint.py.151: total_norm: 19.603798 [I 2022-10-27 22:50:43] ...trap/engines/engine.py.224: train: epoch 92 | batch 1210/2304 [I 2022-10-27 22:50:43] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:10 [I 2022-10-27 22:50:43] ...trap/engines/engine.py.229: process: 0.56192 | load: 0.00239 [I 2022-10-27 22:50:43] ...trap/engines/engine.py.230: loss: 0.12479 [I 2022-10-27 22:50:43] ...optimizers/trijoint.py.151: total_norm: 27.357990 [I 2022-10-27 22:50:49] ...trap/engines/engine.py.224: train: epoch 92 | batch 1220/2304 [I 2022-10-27 22:50:49] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:12 [I 2022-10-27 22:50:49] ...trap/engines/engine.py.229: process: 0.56712 | load: 0.00262 [I 2022-10-27 22:50:49] ...trap/engines/engine.py.230: loss: 0.10435 [I 2022-10-27 22:50:49] ...optimizers/trijoint.py.151: total_norm: 11.630853 [I 2022-10-27 22:50:54] ...trap/engines/engine.py.224: train: epoch 92 | batch 1230/2304 [I 2022-10-27 22:50:54] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:09:57 [I 2022-10-27 22:50:54] ...trap/engines/engine.py.229: process: 0.54990 | load: 0.00249 [I 2022-10-27 22:50:54] ...trap/engines/engine.py.230: loss: 0.15614 [I 2022-10-27 22:50:54] ...optimizers/trijoint.py.151: total_norm: 21.465195 [I 2022-10-27 22:51:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 1240/2304 [I 2022-10-27 22:51:00] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:54 [I 2022-10-27 22:51:00] ...trap/engines/engine.py.229: process: 0.56579 | load: 0.00245 [I 2022-10-27 22:51:00] ...trap/engines/engine.py.230: loss: 0.16945 [I 2022-10-27 22:51:00] ...optimizers/trijoint.py.151: total_norm: 23.744623 [I 2022-10-27 22:51:05] ...trap/engines/engine.py.224: train: epoch 92 | batch 1250/2304 [I 2022-10-27 22:51:05] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:52 [I 2022-10-27 22:51:05] ...trap/engines/engine.py.229: process: 0.56631 | load: 0.00247 [I 2022-10-27 22:51:05] ...trap/engines/engine.py.230: loss: 0.10275 [I 2022-10-27 22:51:05] ...optimizers/trijoint.py.151: total_norm: 9.842131 [I 2022-10-27 22:51:11] ...trap/engines/engine.py.224: train: epoch 92 | batch 1260/2304 [I 2022-10-27 22:51:11] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:10:00 [I 2022-10-27 22:51:11] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00266 [I 2022-10-27 22:51:11] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-27 22:51:11] ...optimizers/trijoint.py.151: total_norm: 6.730708 [I 2022-10-27 22:51:17] ...trap/engines/engine.py.224: train: epoch 92 | batch 1270/2304 [I 2022-10-27 22:51:17] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:41 [I 2022-10-27 22:51:17] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00244 [I 2022-10-27 22:51:17] ...trap/engines/engine.py.230: loss: 0.12500 [I 2022-10-27 22:51:17] ...optimizers/trijoint.py.151: total_norm: 14.643414 [I 2022-10-27 22:51:22] ...trap/engines/engine.py.224: train: epoch 92 | batch 1280/2304 [I 2022-10-27 22:51:22] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:09:32 [I 2022-10-27 22:51:22] ...trap/engines/engine.py.229: process: 0.56812 | load: 0.00248 [I 2022-10-27 22:51:22] ...trap/engines/engine.py.230: loss: 0.12431 [I 2022-10-27 22:51:22] ...optimizers/trijoint.py.151: total_norm: 12.327141 [I 2022-10-27 22:51:28] ...trap/engines/engine.py.224: train: epoch 92 | batch 1290/2304 [I 2022-10-27 22:51:28] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:28 [I 2022-10-27 22:51:28] ...trap/engines/engine.py.229: process: 0.57118 | load: 0.00247 [I 2022-10-27 22:51:28] ...trap/engines/engine.py.230: loss: 0.10301 [I 2022-10-27 22:51:28] ...optimizers/trijoint.py.151: total_norm: 12.743519 [I 2022-10-27 22:51:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 1300/2304 [I 2022-10-27 22:51:34] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:24 [I 2022-10-27 22:51:34] ...trap/engines/engine.py.229: process: 0.56220 | load: 0.00244 [I 2022-10-27 22:51:34] ...trap/engines/engine.py.230: loss: 0.11309 [I 2022-10-27 22:51:34] ...optimizers/trijoint.py.151: total_norm: 14.985437 [I 2022-10-27 22:51:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 1310/2304 [I 2022-10-27 22:51:39] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:15 [I 2022-10-27 22:51:39] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00244 [I 2022-10-27 22:51:39] ...trap/engines/engine.py.230: loss: 0.13085 [I 2022-10-27 22:51:39] ...optimizers/trijoint.py.151: total_norm: 15.537778 [I 2022-10-27 22:51:45] ...trap/engines/engine.py.224: train: epoch 92 | batch 1320/2304 [I 2022-10-27 22:51:45] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:10 [I 2022-10-27 22:51:45] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00265 [I 2022-10-27 22:51:45] ...trap/engines/engine.py.230: loss: 0.18294 [I 2022-10-27 22:51:45] ...optimizers/trijoint.py.151: total_norm: 14.735309 [I 2022-10-27 22:51:50] ...trap/engines/engine.py.224: train: epoch 92 | batch 1330/2304 [I 2022-10-27 22:51:50] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:09:05 [I 2022-10-27 22:51:50] ...trap/engines/engine.py.229: process: 0.56386 | load: 0.00250 [I 2022-10-27 22:51:50] ...trap/engines/engine.py.230: loss: 0.15849 [I 2022-10-27 22:51:50] ...optimizers/trijoint.py.151: total_norm: 17.459129 [I 2022-10-27 22:51:56] ...trap/engines/engine.py.224: train: epoch 92 | batch 1340/2304 [I 2022-10-27 22:51:56] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:08:57 [I 2022-10-27 22:51:56] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00249 [I 2022-10-27 22:51:56] ...trap/engines/engine.py.230: loss: 0.09407 [I 2022-10-27 22:51:56] ...optimizers/trijoint.py.151: total_norm: 11.902902 [I 2022-10-27 22:52:01] ...trap/engines/engine.py.224: train: epoch 92 | batch 1350/2304 [I 2022-10-27 22:52:01] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:08:52 [I 2022-10-27 22:52:01] ...trap/engines/engine.py.229: process: 0.56518 | load: 0.00247 [I 2022-10-27 22:52:01] ...trap/engines/engine.py.230: loss: 0.15868 [I 2022-10-27 22:52:01] ...optimizers/trijoint.py.151: total_norm: 16.822660 [I 2022-10-27 22:52:07] ...trap/engines/engine.py.224: train: epoch 92 | batch 1360/2304 [I 2022-10-27 22:52:07] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:08:49 [I 2022-10-27 22:52:07] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00243 [I 2022-10-27 22:52:07] ...trap/engines/engine.py.230: loss: 0.09760 [I 2022-10-27 22:52:07] ...optimizers/trijoint.py.151: total_norm: 12.135851 [I 2022-10-27 22:52:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 1370/2304 [I 2022-10-27 22:52:13] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:40 [I 2022-10-27 22:52:13] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00249 [I 2022-10-27 22:52:13] ...trap/engines/engine.py.230: loss: 0.10094 [I 2022-10-27 22:52:13] ...optimizers/trijoint.py.151: total_norm: 8.018948 [I 2022-10-27 22:52:18] ...trap/engines/engine.py.224: train: epoch 92 | batch 1380/2304 [I 2022-10-27 22:52:18] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:36 [I 2022-10-27 22:52:18] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00245 [I 2022-10-27 22:52:18] ...trap/engines/engine.py.230: loss: 0.09650 [I 2022-10-27 22:52:18] ...optimizers/trijoint.py.151: total_norm: 8.466916 [I 2022-10-27 22:52:24] ...trap/engines/engine.py.224: train: epoch 92 | batch 1390/2304 [I 2022-10-27 22:52:24] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:28 [I 2022-10-27 22:52:24] ...trap/engines/engine.py.229: process: 0.54644 | load: 0.00242 [I 2022-10-27 22:52:24] ...trap/engines/engine.py.230: loss: 0.15353 [I 2022-10-27 22:52:24] ...optimizers/trijoint.py.151: total_norm: 14.496599 [I 2022-10-27 22:52:29] ...trap/engines/engine.py.224: train: epoch 92 | batch 1400/2304 [I 2022-10-27 22:52:29] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:08:26 [I 2022-10-27 22:52:29] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00243 [I 2022-10-27 22:52:29] ...trap/engines/engine.py.230: loss: 0.09522 [I 2022-10-27 22:52:29] ...optimizers/trijoint.py.151: total_norm: 21.126850 [I 2022-10-27 22:52:35] ...trap/engines/engine.py.224: train: epoch 92 | batch 1410/2304 [I 2022-10-27 22:52:35] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:17 [I 2022-10-27 22:52:35] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00242 [I 2022-10-27 22:52:35] ...trap/engines/engine.py.230: loss: 0.16629 [I 2022-10-27 22:52:35] ...optimizers/trijoint.py.151: total_norm: 17.211954 [I 2022-10-27 22:52:41] ...trap/engines/engine.py.224: train: epoch 92 | batch 1420/2304 [I 2022-10-27 22:52:41] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:13 [I 2022-10-27 22:52:41] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00238 [I 2022-10-27 22:52:41] ...trap/engines/engine.py.230: loss: 0.13762 [I 2022-10-27 22:52:41] ...optimizers/trijoint.py.151: total_norm: 32.601994 [I 2022-10-27 22:52:46] ...trap/engines/engine.py.224: train: epoch 92 | batch 1430/2304 [I 2022-10-27 22:52:46] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:04 [I 2022-10-27 22:52:46] ...trap/engines/engine.py.229: process: 0.54755 | load: 0.00243 [I 2022-10-27 22:52:46] ...trap/engines/engine.py.230: loss: 0.13644 [I 2022-10-27 22:52:46] ...optimizers/trijoint.py.151: total_norm: 10.869813 [I 2022-10-27 22:52:52] ...trap/engines/engine.py.224: train: epoch 92 | batch 1440/2304 [I 2022-10-27 22:52:52] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:08:01 [I 2022-10-27 22:52:52] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00244 [I 2022-10-27 22:52:52] ...trap/engines/engine.py.230: loss: 0.13994 [I 2022-10-27 22:52:52] ...optimizers/trijoint.py.151: total_norm: 24.768217 [I 2022-10-27 22:52:57] ...trap/engines/engine.py.224: train: epoch 92 | batch 1450/2304 [I 2022-10-27 22:52:57] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:07:58 [I 2022-10-27 22:52:57] ...trap/engines/engine.py.229: process: 0.55637 | load: 0.00241 [I 2022-10-27 22:52:57] ...trap/engines/engine.py.230: loss: 0.19173 [I 2022-10-27 22:52:57] ...optimizers/trijoint.py.151: total_norm: 12.738490 [I 2022-10-27 22:53:03] ...trap/engines/engine.py.224: train: epoch 92 | batch 1460/2304 [I 2022-10-27 22:53:03] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:07:49 [I 2022-10-27 22:53:03] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00241 [I 2022-10-27 22:53:03] ...trap/engines/engine.py.230: loss: 0.14764 [I 2022-10-27 22:53:03] ...optimizers/trijoint.py.151: total_norm: 14.349876 [I 2022-10-27 22:53:09] ...trap/engines/engine.py.224: train: epoch 92 | batch 1470/2304 [I 2022-10-27 22:53:09] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:46 [I 2022-10-27 22:53:09] ...trap/engines/engine.py.229: process: 0.56163 | load: 0.00247 [I 2022-10-27 22:53:09] ...trap/engines/engine.py.230: loss: 0.11274 [I 2022-10-27 22:53:09] ...optimizers/trijoint.py.151: total_norm: 9.111755 [I 2022-10-27 22:53:14] ...trap/engines/engine.py.224: train: epoch 92 | batch 1480/2304 [I 2022-10-27 22:53:14] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:38 [I 2022-10-27 22:53:14] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00253 [I 2022-10-27 22:53:14] ...trap/engines/engine.py.230: loss: 0.09731 [I 2022-10-27 22:53:14] ...optimizers/trijoint.py.151: total_norm: 10.885332 [I 2022-10-27 22:53:20] ...trap/engines/engine.py.224: train: epoch 92 | batch 1490/2304 [I 2022-10-27 22:53:20] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:31 [I 2022-10-27 22:53:20] ...trap/engines/engine.py.229: process: 0.55218 | load: 0.00251 [I 2022-10-27 22:53:20] ...trap/engines/engine.py.230: loss: 0.16999 [I 2022-10-27 22:53:20] ...optimizers/trijoint.py.151: total_norm: 17.216520 [I 2022-10-27 22:53:25] ...trap/engines/engine.py.224: train: epoch 92 | batch 1500/2304 [I 2022-10-27 22:53:25] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:07:29 [I 2022-10-27 22:53:25] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00245 [I 2022-10-27 22:53:25] ...trap/engines/engine.py.230: loss: 0.09809 [I 2022-10-27 22:53:25] ...optimizers/trijoint.py.151: total_norm: 9.736211 [I 2022-10-27 22:53:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 1510/2304 [I 2022-10-27 22:53:31] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:23 [I 2022-10-27 22:53:31] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00245 [I 2022-10-27 22:53:31] ...trap/engines/engine.py.230: loss: 0.15590 [I 2022-10-27 22:53:31] ...optimizers/trijoint.py.151: total_norm: 32.374519 [I 2022-10-27 22:53:37] ...trap/engines/engine.py.224: train: epoch 92 | batch 1520/2304 [I 2022-10-27 22:53:37] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:07:14 [I 2022-10-27 22:53:37] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00244 [I 2022-10-27 22:53:37] ...trap/engines/engine.py.230: loss: 0.10898 [I 2022-10-27 22:53:37] ...optimizers/trijoint.py.151: total_norm: 12.178308 [I 2022-10-27 22:53:42] ...trap/engines/engine.py.224: train: epoch 92 | batch 1530/2304 [I 2022-10-27 22:53:42] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:10 [I 2022-10-27 22:53:42] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00243 [I 2022-10-27 22:53:42] ...trap/engines/engine.py.230: loss: 0.13414 [I 2022-10-27 22:53:42] ...optimizers/trijoint.py.151: total_norm: 22.198076 [I 2022-10-27 22:53:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 1540/2304 [I 2022-10-27 22:53:48] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:04 [I 2022-10-27 22:53:48] ...trap/engines/engine.py.229: process: 0.54862 | load: 0.00245 [I 2022-10-27 22:53:48] ...trap/engines/engine.py.230: loss: 0.20284 [I 2022-10-27 22:53:48] ...optimizers/trijoint.py.151: total_norm: 18.761124 [I 2022-10-27 22:53:53] ...trap/engines/engine.py.224: train: epoch 92 | batch 1550/2304 [I 2022-10-27 22:53:53] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:07:00 [I 2022-10-27 22:53:53] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00255 [I 2022-10-27 22:53:53] ...trap/engines/engine.py.230: loss: 0.16484 [I 2022-10-27 22:53:53] ...optimizers/trijoint.py.151: total_norm: 28.112558 [I 2022-10-27 22:53:59] ...trap/engines/engine.py.224: train: epoch 92 | batch 1560/2304 [I 2022-10-27 22:53:59] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:06:56 [I 2022-10-27 22:53:59] ...trap/engines/engine.py.229: process: 0.56233 | load: 0.00245 [I 2022-10-27 22:53:59] ...trap/engines/engine.py.230: loss: 0.10569 [I 2022-10-27 22:53:59] ...optimizers/trijoint.py.151: total_norm: 24.040461 [I 2022-10-27 22:54:04] ...trap/engines/engine.py.224: train: epoch 92 | batch 1570/2304 [I 2022-10-27 22:54:04] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:06:50 [I 2022-10-27 22:54:04] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00246 [I 2022-10-27 22:54:04] ...trap/engines/engine.py.230: loss: 0.12047 [I 2022-10-27 22:54:04] ...optimizers/trijoint.py.151: total_norm: 16.793377 [I 2022-10-27 22:54:10] ...trap/engines/engine.py.224: train: epoch 92 | batch 1580/2304 [I 2022-10-27 22:54:10] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:06:43 [I 2022-10-27 22:54:10] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00246 [I 2022-10-27 22:54:10] ...trap/engines/engine.py.230: loss: 0.10924 [I 2022-10-27 22:54:10] ...optimizers/trijoint.py.151: total_norm: 21.609234 [I 2022-10-27 22:54:16] ...trap/engines/engine.py.224: train: epoch 92 | batch 1590/2304 [I 2022-10-27 22:54:16] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:36 [I 2022-10-27 22:54:16] ...trap/engines/engine.py.229: process: 0.55529 | load: 0.00244 [I 2022-10-27 22:54:16] ...trap/engines/engine.py.230: loss: 0.18890 [I 2022-10-27 22:54:16] ...optimizers/trijoint.py.151: total_norm: 51.639965 [I 2022-10-27 22:54:21] ...trap/engines/engine.py.224: train: epoch 92 | batch 1600/2304 [I 2022-10-27 22:54:21] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:32 [I 2022-10-27 22:54:21] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00248 [I 2022-10-27 22:54:21] ...trap/engines/engine.py.230: loss: 0.10202 [I 2022-10-27 22:54:21] ...optimizers/trijoint.py.151: total_norm: 8.620893 [I 2022-10-27 22:54:27] ...trap/engines/engine.py.224: train: epoch 92 | batch 1610/2304 [I 2022-10-27 22:54:27] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:27 [I 2022-10-27 22:54:27] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00246 [I 2022-10-27 22:54:27] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-27 22:54:27] ...optimizers/trijoint.py.151: total_norm: 29.080938 [I 2022-10-27 22:54:32] ...trap/engines/engine.py.224: train: epoch 92 | batch 1620/2304 [I 2022-10-27 22:54:32] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:22 [I 2022-10-27 22:54:32] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00245 [I 2022-10-27 22:54:32] ...trap/engines/engine.py.230: loss: 0.10677 [I 2022-10-27 22:54:32] ...optimizers/trijoint.py.151: total_norm: 16.371107 [I 2022-10-27 22:54:38] ...trap/engines/engine.py.224: train: epoch 92 | batch 1630/2304 [I 2022-10-27 22:54:38] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:12 [I 2022-10-27 22:54:38] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00243 [I 2022-10-27 22:54:38] ...trap/engines/engine.py.230: loss: 0.09578 [I 2022-10-27 22:54:38] ...optimizers/trijoint.py.151: total_norm: 16.918869 [I 2022-10-27 22:54:43] ...trap/engines/engine.py.224: train: epoch 92 | batch 1640/2304 [I 2022-10-27 22:54:43] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:09 [I 2022-10-27 22:54:43] ...trap/engines/engine.py.229: process: 0.55902 | load: 0.00241 [I 2022-10-27 22:54:43] ...trap/engines/engine.py.230: loss: 0.11493 [I 2022-10-27 22:54:43] ...optimizers/trijoint.py.151: total_norm: 8.835921 [I 2022-10-27 22:54:49] ...trap/engines/engine.py.224: train: epoch 92 | batch 1650/2304 [I 2022-10-27 22:54:49] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:03 [I 2022-10-27 22:54:49] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00240 [I 2022-10-27 22:54:49] ...trap/engines/engine.py.230: loss: 0.13469 [I 2022-10-27 22:54:49] ...optimizers/trijoint.py.151: total_norm: 11.891636 [I 2022-10-27 22:54:55] ...trap/engines/engine.py.224: train: epoch 92 | batch 1660/2304 [I 2022-10-27 22:54:55] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:05:57 [I 2022-10-27 22:54:55] ...trap/engines/engine.py.229: process: 0.54625 | load: 0.00242 [I 2022-10-27 22:54:55] ...trap/engines/engine.py.230: loss: 0.23923 [I 2022-10-27 22:54:55] ...optimizers/trijoint.py.151: total_norm: 21.870808 [I 2022-10-27 22:55:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 1670/2304 [I 2022-10-27 22:55:00] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:53 [I 2022-10-27 22:55:00] ...trap/engines/engine.py.229: process: 0.55902 | load: 0.00256 [I 2022-10-27 22:55:00] ...trap/engines/engine.py.230: loss: 0.12728 [I 2022-10-27 22:55:00] ...optimizers/trijoint.py.151: total_norm: 18.524027 [I 2022-10-27 22:55:06] ...trap/engines/engine.py.224: train: epoch 92 | batch 1680/2304 [I 2022-10-27 22:55:06] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:46 [I 2022-10-27 22:55:06] ...trap/engines/engine.py.229: process: 0.54851 | load: 0.00240 [I 2022-10-27 22:55:06] ...trap/engines/engine.py.230: loss: 0.22085 [I 2022-10-27 22:55:06] ...optimizers/trijoint.py.151: total_norm: 32.718929 [I 2022-10-27 22:55:11] ...trap/engines/engine.py.224: train: epoch 92 | batch 1690/2304 [I 2022-10-27 22:55:11] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:40 [I 2022-10-27 22:55:11] ...trap/engines/engine.py.229: process: 0.55478 | load: 0.00242 [I 2022-10-27 22:55:11] ...trap/engines/engine.py.230: loss: 0.12178 [I 2022-10-27 22:55:11] ...optimizers/trijoint.py.151: total_norm: 32.970913 [I 2022-10-27 22:55:17] ...trap/engines/engine.py.224: train: epoch 92 | batch 1700/2304 [I 2022-10-27 22:55:17] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:34 [I 2022-10-27 22:55:17] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00250 [I 2022-10-27 22:55:17] ...trap/engines/engine.py.230: loss: 0.11538 [I 2022-10-27 22:55:17] ...optimizers/trijoint.py.151: total_norm: 56.542480 [I 2022-10-27 22:55:22] ...trap/engines/engine.py.224: train: epoch 92 | batch 1710/2304 [I 2022-10-27 22:55:22] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:29 [I 2022-10-27 22:55:22] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00245 [I 2022-10-27 22:55:22] ...trap/engines/engine.py.230: loss: 0.12256 [I 2022-10-27 22:55:22] ...optimizers/trijoint.py.151: total_norm: 11.575066 [I 2022-10-27 22:55:28] ...trap/engines/engine.py.224: train: epoch 92 | batch 1720/2304 [I 2022-10-27 22:55:28] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:25 [I 2022-10-27 22:55:28] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00242 [I 2022-10-27 22:55:28] ...trap/engines/engine.py.230: loss: 0.22842 [I 2022-10-27 22:55:28] ...optimizers/trijoint.py.151: total_norm: 33.129925 [I 2022-10-27 22:55:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 1730/2304 [I 2022-10-27 22:55:34] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:19 [I 2022-10-27 22:55:34] ...trap/engines/engine.py.229: process: 0.55518 | load: 0.00241 [I 2022-10-27 22:55:34] ...trap/engines/engine.py.230: loss: 0.10486 [I 2022-10-27 22:55:34] ...optimizers/trijoint.py.151: total_norm: 16.366932 [I 2022-10-27 22:55:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 1740/2304 [I 2022-10-27 22:55:39] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:14 [I 2022-10-27 22:55:39] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00263 [I 2022-10-27 22:55:39] ...trap/engines/engine.py.230: loss: 0.16356 [I 2022-10-27 22:55:39] ...optimizers/trijoint.py.151: total_norm: 12.308538 [I 2022-10-27 22:55:45] ...trap/engines/engine.py.224: train: epoch 92 | batch 1750/2304 [I 2022-10-27 22:55:45] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:06 [I 2022-10-27 22:55:45] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00243 [I 2022-10-27 22:55:45] ...trap/engines/engine.py.230: loss: 0.11810 [I 2022-10-27 22:55:45] ...optimizers/trijoint.py.151: total_norm: 13.956607 [I 2022-10-27 22:55:50] ...trap/engines/engine.py.224: train: epoch 92 | batch 1760/2304 [I 2022-10-27 22:55:50] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:04:58 [I 2022-10-27 22:55:50] ...trap/engines/engine.py.229: process: 0.54571 | load: 0.00244 [I 2022-10-27 22:55:50] ...trap/engines/engine.py.230: loss: 0.13410 [I 2022-10-27 22:55:50] ...optimizers/trijoint.py.151: total_norm: 15.489286 [I 2022-10-27 22:55:56] ...trap/engines/engine.py.224: train: epoch 92 | batch 1770/2304 [I 2022-10-27 22:55:56] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:04:56 [I 2022-10-27 22:55:56] ...trap/engines/engine.py.229: process: 0.55295 | load: 0.00244 [I 2022-10-27 22:55:56] ...trap/engines/engine.py.230: loss: 0.10438 [I 2022-10-27 22:55:56] ...optimizers/trijoint.py.151: total_norm: 27.851191 [I 2022-10-27 22:56:01] ...trap/engines/engine.py.224: train: epoch 92 | batch 1780/2304 [I 2022-10-27 22:56:01] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:49 [I 2022-10-27 22:56:01] ...trap/engines/engine.py.229: process: 0.54630 | load: 0.00242 [I 2022-10-27 22:56:01] ...trap/engines/engine.py.230: loss: 0.12062 [I 2022-10-27 22:56:01] ...optimizers/trijoint.py.151: total_norm: 10.589075 [I 2022-10-27 22:56:07] ...trap/engines/engine.py.224: train: epoch 92 | batch 1790/2304 [I 2022-10-27 22:56:07] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:45 [I 2022-10-27 22:56:07] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00249 [I 2022-10-27 22:56:07] ...trap/engines/engine.py.230: loss: 0.15084 [I 2022-10-27 22:56:07] ...optimizers/trijoint.py.151: total_norm: 35.281624 [I 2022-10-27 22:56:12] ...trap/engines/engine.py.224: train: epoch 92 | batch 1800/2304 [I 2022-10-27 22:56:12] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:40 [I 2022-10-27 22:56:12] ...trap/engines/engine.py.229: process: 0.55375 | load: 0.00247 [I 2022-10-27 22:56:12] ...trap/engines/engine.py.230: loss: 0.07794 [I 2022-10-27 22:56:12] ...optimizers/trijoint.py.151: total_norm: 11.477451 [I 2022-10-27 22:56:18] ...trap/engines/engine.py.224: train: epoch 92 | batch 1810/2304 [I 2022-10-27 22:56:18] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:34 [I 2022-10-27 22:56:18] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00245 [I 2022-10-27 22:56:18] ...trap/engines/engine.py.230: loss: 0.10323 [I 2022-10-27 22:56:18] ...optimizers/trijoint.py.151: total_norm: 9.780848 [I 2022-10-27 22:56:24] ...trap/engines/engine.py.224: train: epoch 92 | batch 1820/2304 [I 2022-10-27 22:56:24] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:30 [I 2022-10-27 22:56:24] ...trap/engines/engine.py.229: process: 0.55050 | load: 0.00246 [I 2022-10-27 22:56:24] ...trap/engines/engine.py.230: loss: 0.20043 [I 2022-10-27 22:56:24] ...optimizers/trijoint.py.151: total_norm: 21.688028 [I 2022-10-27 22:56:29] ...trap/engines/engine.py.224: train: epoch 92 | batch 1830/2304 [I 2022-10-27 22:56:29] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:23 [I 2022-10-27 22:56:29] ...trap/engines/engine.py.229: process: 0.55352 | load: 0.00245 [I 2022-10-27 22:56:29] ...trap/engines/engine.py.230: loss: 0.13061 [I 2022-10-27 22:56:29] ...optimizers/trijoint.py.151: total_norm: 29.453587 [I 2022-10-27 22:56:35] ...trap/engines/engine.py.224: train: epoch 92 | batch 1840/2304 [I 2022-10-27 22:56:35] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:24 [I 2022-10-27 22:56:35] ...trap/engines/engine.py.229: process: 0.55941 | load: 0.00243 [I 2022-10-27 22:56:35] ...trap/engines/engine.py.230: loss: 0.13746 [I 2022-10-27 22:56:35] ...optimizers/trijoint.py.151: total_norm: 12.413128 [I 2022-10-27 22:56:41] ...trap/engines/engine.py.224: train: epoch 92 | batch 1850/2304 [I 2022-10-27 22:56:41] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:19 [I 2022-10-27 22:56:41] ...trap/engines/engine.py.229: process: 0.62384 | load: 0.00240 [I 2022-10-27 22:56:41] ...trap/engines/engine.py.230: loss: 0.16268 [I 2022-10-27 22:56:41] ...optimizers/trijoint.py.151: total_norm: 19.489161 [I 2022-10-27 22:56:46] ...trap/engines/engine.py.224: train: epoch 92 | batch 1860/2304 [I 2022-10-27 22:56:46] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:07 [I 2022-10-27 22:56:46] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00244 [I 2022-10-27 22:56:46] ...trap/engines/engine.py.230: loss: 0.08666 [I 2022-10-27 22:56:46] ...optimizers/trijoint.py.151: total_norm: 10.406296 [I 2022-10-27 22:56:52] ...trap/engines/engine.py.224: train: epoch 92 | batch 1870/2304 [I 2022-10-27 22:56:52] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:01 [I 2022-10-27 22:56:52] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00244 [I 2022-10-27 22:56:52] ...trap/engines/engine.py.230: loss: 0.09283 [I 2022-10-27 22:56:52] ...optimizers/trijoint.py.151: total_norm: 8.241610 [I 2022-10-27 22:56:57] ...trap/engines/engine.py.224: train: epoch 92 | batch 1880/2304 [I 2022-10-27 22:56:57] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:03:56 [I 2022-10-27 22:56:57] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00241 [I 2022-10-27 22:56:57] ...trap/engines/engine.py.230: loss: 0.12540 [I 2022-10-27 22:56:57] ...optimizers/trijoint.py.151: total_norm: 9.414367 [I 2022-10-27 22:57:03] ...trap/engines/engine.py.224: train: epoch 92 | batch 1890/2304 [I 2022-10-27 22:57:03] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:51 [I 2022-10-27 22:57:03] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00241 [I 2022-10-27 22:57:03] ...trap/engines/engine.py.230: loss: 0.14541 [I 2022-10-27 22:57:03] ...optimizers/trijoint.py.151: total_norm: 11.512843 [I 2022-10-27 22:57:08] ...trap/engines/engine.py.224: train: epoch 92 | batch 1900/2304 [I 2022-10-27 22:57:08] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:44 [I 2022-10-27 22:57:08] ...trap/engines/engine.py.229: process: 0.55237 | load: 0.00241 [I 2022-10-27 22:57:08] ...trap/engines/engine.py.230: loss: 0.13801 [I 2022-10-27 22:57:08] ...optimizers/trijoint.py.151: total_norm: 20.391542 [I 2022-10-27 22:57:14] ...trap/engines/engine.py.224: train: epoch 92 | batch 1910/2304 [I 2022-10-27 22:57:14] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:39 [I 2022-10-27 22:57:14] ...trap/engines/engine.py.229: process: 0.54421 | load: 0.00243 [I 2022-10-27 22:57:14] ...trap/engines/engine.py.230: loss: 0.09794 [I 2022-10-27 22:57:14] ...optimizers/trijoint.py.151: total_norm: 8.500566 [I 2022-10-27 22:57:20] ...trap/engines/engine.py.224: train: epoch 92 | batch 1920/2304 [I 2022-10-27 22:57:20] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:34 [I 2022-10-27 22:57:20] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00238 [I 2022-10-27 22:57:20] ...trap/engines/engine.py.230: loss: 0.17429 [I 2022-10-27 22:57:20] ...optimizers/trijoint.py.151: total_norm: 15.658575 [I 2022-10-27 22:57:25] ...trap/engines/engine.py.224: train: epoch 92 | batch 1930/2304 [I 2022-10-27 22:57:25] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:29 [I 2022-10-27 22:57:25] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00243 [I 2022-10-27 22:57:25] ...trap/engines/engine.py.230: loss: 0.11162 [I 2022-10-27 22:57:25] ...optimizers/trijoint.py.151: total_norm: 15.068787 [I 2022-10-27 22:57:31] ...trap/engines/engine.py.224: train: epoch 92 | batch 1940/2304 [I 2022-10-27 22:57:31] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:23 [I 2022-10-27 22:57:31] ...trap/engines/engine.py.229: process: 0.55844 | load: 0.00249 [I 2022-10-27 22:57:31] ...trap/engines/engine.py.230: loss: 0.08474 [I 2022-10-27 22:57:31] ...optimizers/trijoint.py.151: total_norm: 9.745660 [I 2022-10-27 22:57:37] ...trap/engines/engine.py.224: train: epoch 92 | batch 1950/2304 [I 2022-10-27 22:57:37] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:16 [I 2022-10-27 22:57:37] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00247 [I 2022-10-27 22:57:37] ...trap/engines/engine.py.230: loss: 0.09569 [I 2022-10-27 22:57:37] ...optimizers/trijoint.py.151: total_norm: 9.190365 [I 2022-10-27 22:57:42] ...trap/engines/engine.py.224: train: epoch 92 | batch 1960/2304 [I 2022-10-27 22:57:42] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:13 [I 2022-10-27 22:57:42] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00246 [I 2022-10-27 22:57:42] ...trap/engines/engine.py.230: loss: 0.19872 [I 2022-10-27 22:57:42] ...optimizers/trijoint.py.151: total_norm: 18.218754 [I 2022-10-27 22:57:48] ...trap/engines/engine.py.224: train: epoch 92 | batch 1970/2304 [I 2022-10-27 22:57:48] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:06 [I 2022-10-27 22:57:48] ...trap/engines/engine.py.229: process: 0.56697 | load: 0.00259 [I 2022-10-27 22:57:48] ...trap/engines/engine.py.230: loss: 0.10682 [I 2022-10-27 22:57:48] ...optimizers/trijoint.py.151: total_norm: 21.261818 [I 2022-10-27 22:57:53] ...trap/engines/engine.py.224: train: epoch 92 | batch 1980/2304 [I 2022-10-27 22:57:53] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:02:59 [I 2022-10-27 22:57:53] ...trap/engines/engine.py.229: process: 0.56427 | load: 0.00242 [I 2022-10-27 22:57:53] ...trap/engines/engine.py.230: loss: 0.11912 [I 2022-10-27 22:57:53] ...optimizers/trijoint.py.151: total_norm: 18.206013 [I 2022-10-27 22:57:59] ...trap/engines/engine.py.224: train: epoch 92 | batch 1990/2304 [I 2022-10-27 22:57:59] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:02:53 [I 2022-10-27 22:57:59] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00244 [I 2022-10-27 22:57:59] ...trap/engines/engine.py.230: loss: 0.14682 [I 2022-10-27 22:57:59] ...optimizers/trijoint.py.151: total_norm: 16.588104 [I 2022-10-27 22:58:04] ...trap/engines/engine.py.224: train: epoch 92 | batch 2000/2304 [I 2022-10-27 22:58:05] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:49 [I 2022-10-27 22:58:05] ...trap/engines/engine.py.229: process: 0.54929 | load: 0.00241 [I 2022-10-27 22:58:05] ...trap/engines/engine.py.230: loss: 0.19291 [I 2022-10-27 22:58:05] ...optimizers/trijoint.py.151: total_norm: 14.070759 [I 2022-10-27 22:58:10] ...trap/engines/engine.py.224: train: epoch 92 | batch 2010/2304 [I 2022-10-27 22:58:10] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:44 [I 2022-10-27 22:58:10] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00240 [I 2022-10-27 22:58:10] ...trap/engines/engine.py.230: loss: 0.11554 [I 2022-10-27 22:58:10] ...optimizers/trijoint.py.151: total_norm: 13.366637 [I 2022-10-27 22:58:16] ...trap/engines/engine.py.224: train: epoch 92 | batch 2020/2304 [I 2022-10-27 22:58:16] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:38 [I 2022-10-27 22:58:16] ...trap/engines/engine.py.229: process: 0.56272 | load: 0.00241 [I 2022-10-27 22:58:16] ...trap/engines/engine.py.230: loss: 0.15501 [I 2022-10-27 22:58:16] ...optimizers/trijoint.py.151: total_norm: 20.014013 [I 2022-10-27 22:58:21] ...trap/engines/engine.py.224: train: epoch 92 | batch 2030/2304 [I 2022-10-27 22:58:21] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:33 [I 2022-10-27 22:58:21] ...trap/engines/engine.py.229: process: 0.56689 | load: 0.00250 [I 2022-10-27 22:58:21] ...trap/engines/engine.py.230: loss: 0.10037 [I 2022-10-27 22:58:21] ...optimizers/trijoint.py.151: total_norm: 10.849594 [I 2022-10-27 22:58:27] ...trap/engines/engine.py.224: train: epoch 92 | batch 2040/2304 [I 2022-10-27 22:58:27] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:26 [I 2022-10-27 22:58:27] ...trap/engines/engine.py.229: process: 0.54326 | load: 0.00246 [I 2022-10-27 22:58:27] ...trap/engines/engine.py.230: loss: 0.12569 [I 2022-10-27 22:58:27] ...optimizers/trijoint.py.151: total_norm: 15.166923 [I 2022-10-27 22:58:32] ...trap/engines/engine.py.224: train: epoch 92 | batch 2050/2304 [I 2022-10-27 22:58:32] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:21 [I 2022-10-27 22:58:32] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00246 [I 2022-10-27 22:58:32] ...trap/engines/engine.py.230: loss: 0.10431 [I 2022-10-27 22:58:32] ...optimizers/trijoint.py.151: total_norm: 128.571625 [I 2022-10-27 22:58:38] ...trap/engines/engine.py.224: train: epoch 92 | batch 2060/2304 [I 2022-10-27 22:58:38] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:15 [I 2022-10-27 22:58:38] ...trap/engines/engine.py.229: process: 0.55042 | load: 0.00244 [I 2022-10-27 22:58:38] ...trap/engines/engine.py.230: loss: 0.23166 [I 2022-10-27 22:58:38] ...optimizers/trijoint.py.151: total_norm: 37.649151 [I 2022-10-27 22:58:44] ...trap/engines/engine.py.224: train: epoch 92 | batch 2070/2304 [I 2022-10-27 22:58:44] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:10 [I 2022-10-27 22:58:44] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00245 [I 2022-10-27 22:58:44] ...trap/engines/engine.py.230: loss: 0.12596 [I 2022-10-27 22:58:44] ...optimizers/trijoint.py.151: total_norm: 15.372269 [I 2022-10-27 22:58:49] ...trap/engines/engine.py.224: train: epoch 92 | batch 2080/2304 [I 2022-10-27 22:58:49] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:04 [I 2022-10-27 22:58:49] ...trap/engines/engine.py.229: process: 0.55162 | load: 0.00239 [I 2022-10-27 22:58:49] ...trap/engines/engine.py.230: loss: 0.09116 [I 2022-10-27 22:58:49] ...optimizers/trijoint.py.151: total_norm: 10.140779 [I 2022-10-27 22:58:55] ...trap/engines/engine.py.224: train: epoch 92 | batch 2090/2304 [I 2022-10-27 22:58:55] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:01:59 [I 2022-10-27 22:58:55] ...trap/engines/engine.py.229: process: 0.56554 | load: 0.00240 [I 2022-10-27 22:58:55] ...trap/engines/engine.py.230: loss: 0.16812 [I 2022-10-27 22:58:55] ...optimizers/trijoint.py.151: total_norm: 12.764621 [I 2022-10-27 22:59:00] ...trap/engines/engine.py.224: train: epoch 92 | batch 2100/2304 [I 2022-10-27 22:59:00] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:01:53 [I 2022-10-27 22:59:00] ...trap/engines/engine.py.229: process: 0.56274 | load: 0.00243 [I 2022-10-27 22:59:00] ...trap/engines/engine.py.230: loss: 0.10121 [I 2022-10-27 22:59:00] ...optimizers/trijoint.py.151: total_norm: 10.213159 [I 2022-10-27 22:59:06] ...trap/engines/engine.py.224: train: epoch 92 | batch 2110/2304 [I 2022-10-27 22:59:06] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:49 [I 2022-10-27 22:59:06] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00245 [I 2022-10-27 22:59:06] ...trap/engines/engine.py.230: loss: 0.11287 [I 2022-10-27 22:59:06] ...optimizers/trijoint.py.151: total_norm: 12.468442 [I 2022-10-27 22:59:12] ...trap/engines/engine.py.224: train: epoch 92 | batch 2120/2304 [I 2022-10-27 22:59:12] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:41 [I 2022-10-27 22:59:12] ...trap/engines/engine.py.229: process: 0.55399 | load: 0.00243 [I 2022-10-27 22:59:12] ...trap/engines/engine.py.230: loss: 0.10521 [I 2022-10-27 22:59:12] ...optimizers/trijoint.py.151: total_norm: 10.669701 [I 2022-10-27 22:59:17] ...trap/engines/engine.py.224: train: epoch 92 | batch 2130/2304 [I 2022-10-27 22:59:17] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:36 [I 2022-10-27 22:59:17] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00242 [I 2022-10-27 22:59:17] ...trap/engines/engine.py.230: loss: 0.15774 [I 2022-10-27 22:59:17] ...optimizers/trijoint.py.151: total_norm: 37.820656 [I 2022-10-27 22:59:23] ...trap/engines/engine.py.224: train: epoch 92 | batch 2140/2304 [I 2022-10-27 22:59:23] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:31 [I 2022-10-27 22:59:23] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00240 [I 2022-10-27 22:59:23] ...trap/engines/engine.py.230: loss: 0.11266 [I 2022-10-27 22:59:23] ...optimizers/trijoint.py.151: total_norm: 20.987225 [I 2022-10-27 22:59:28] ...trap/engines/engine.py.224: train: epoch 92 | batch 2150/2304 [I 2022-10-27 22:59:28] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:25 [I 2022-10-27 22:59:28] ...trap/engines/engine.py.229: process: 0.56408 | load: 0.00269 [I 2022-10-27 22:59:28] ...trap/engines/engine.py.230: loss: 0.09696 [I 2022-10-27 22:59:28] ...optimizers/trijoint.py.151: total_norm: 16.026838 [I 2022-10-27 22:59:34] ...trap/engines/engine.py.224: train: epoch 92 | batch 2160/2304 [I 2022-10-27 22:59:34] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:20 [I 2022-10-27 22:59:34] ...trap/engines/engine.py.229: process: 0.55146 | load: 0.00241 [I 2022-10-27 22:59:34] ...trap/engines/engine.py.230: loss: 0.09423 [I 2022-10-27 22:59:34] ...optimizers/trijoint.py.151: total_norm: 24.850817 [I 2022-10-27 22:59:39] ...trap/engines/engine.py.224: train: epoch 92 | batch 2170/2304 [I 2022-10-27 22:59:39] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:14 [I 2022-10-27 22:59:39] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00243 [I 2022-10-27 22:59:39] ...trap/engines/engine.py.230: loss: 0.14722 [I 2022-10-27 22:59:39] ...optimizers/trijoint.py.151: total_norm: 27.862736 [I 2022-10-27 22:59:45] ...trap/engines/engine.py.224: train: epoch 92 | batch 2180/2304 [I 2022-10-27 22:59:45] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:08 [I 2022-10-27 22:59:45] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00242 [I 2022-10-27 22:59:45] ...trap/engines/engine.py.230: loss: 0.14832 [I 2022-10-27 22:59:45] ...optimizers/trijoint.py.151: total_norm: 14.777001 [I 2022-10-27 22:59:50] ...trap/engines/engine.py.224: train: epoch 92 | batch 2190/2304 [I 2022-10-27 22:59:50] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:02 [I 2022-10-27 22:59:50] ...trap/engines/engine.py.229: process: 0.54489 | load: 0.00243 [I 2022-10-27 22:59:50] ...trap/engines/engine.py.230: loss: 0.11686 [I 2022-10-27 22:59:50] ...optimizers/trijoint.py.151: total_norm: 12.896297 [I 2022-10-27 22:59:56] ...trap/engines/engine.py.224: train: epoch 92 | batch 2200/2304 [I 2022-10-27 22:59:56] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:00:57 [I 2022-10-27 22:59:56] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00250 [I 2022-10-27 22:59:56] ...trap/engines/engine.py.230: loss: 0.10556 [I 2022-10-27 22:59:56] ...optimizers/trijoint.py.151: total_norm: 8.717458 [I 2022-10-27 23:00:02] ...trap/engines/engine.py.224: train: epoch 92 | batch 2210/2304 [I 2022-10-27 23:00:02] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:00:54 [I 2022-10-27 23:00:02] ...trap/engines/engine.py.229: process: 0.61250 | load: 0.00270 [I 2022-10-27 23:00:02] ...trap/engines/engine.py.230: loss: 0.13485 [I 2022-10-27 23:00:02] ...optimizers/trijoint.py.151: total_norm: 20.209822 [I 2022-10-27 23:00:07] ...trap/engines/engine.py.224: train: epoch 92 | batch 2220/2304 [I 2022-10-27 23:00:07] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:00:46 [I 2022-10-27 23:00:07] ...trap/engines/engine.py.229: process: 0.53955 | load: 0.00241 [I 2022-10-27 23:00:07] ...trap/engines/engine.py.230: loss: 0.13682 [I 2022-10-27 23:00:07] ...optimizers/trijoint.py.151: total_norm: 7.454519 [I 2022-10-27 23:00:13] ...trap/engines/engine.py.224: train: epoch 92 | batch 2230/2304 [I 2022-10-27 23:00:13] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:40 [I 2022-10-27 23:00:13] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00244 [I 2022-10-27 23:00:13] ...trap/engines/engine.py.230: loss: 0.10860 [I 2022-10-27 23:00:13] ...optimizers/trijoint.py.151: total_norm: 27.892694 [I 2022-10-27 23:00:18] ...trap/engines/engine.py.224: train: epoch 92 | batch 2240/2304 [I 2022-10-27 23:00:18] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:35 [I 2022-10-27 23:00:18] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00241 [I 2022-10-27 23:00:18] ...trap/engines/engine.py.230: loss: 0.17613 [I 2022-10-27 23:00:18] ...optimizers/trijoint.py.151: total_norm: 23.473789 [I 2022-10-27 23:00:24] ...trap/engines/engine.py.224: train: epoch 92 | batch 2250/2304 [I 2022-10-27 23:00:24] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:29 [I 2022-10-27 23:00:24] ...trap/engines/engine.py.229: process: 0.55058 | load: 0.00243 [I 2022-10-27 23:00:24] ...trap/engines/engine.py.230: loss: 0.21798 [I 2022-10-27 23:00:24] ...optimizers/trijoint.py.151: total_norm: 21.719473 [I 2022-10-27 23:00:29] ...trap/engines/engine.py.224: train: epoch 92 | batch 2260/2304 [I 2022-10-27 23:00:29] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:24 [I 2022-10-27 23:00:29] ...trap/engines/engine.py.229: process: 0.57303 | load: 0.00240 [I 2022-10-27 23:00:29] ...trap/engines/engine.py.230: loss: 0.16399 [I 2022-10-27 23:00:29] ...optimizers/trijoint.py.151: total_norm: 13.682551 [I 2022-10-27 23:00:35] ...trap/engines/engine.py.224: train: epoch 92 | batch 2270/2304 [I 2022-10-27 23:00:35] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:18 [I 2022-10-27 23:00:35] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00241 [I 2022-10-27 23:00:35] ...trap/engines/engine.py.230: loss: 0.11361 [I 2022-10-27 23:00:35] ...optimizers/trijoint.py.151: total_norm: 11.485406 [I 2022-10-27 23:00:41] ...trap/engines/engine.py.224: train: epoch 92 | batch 2280/2304 [I 2022-10-27 23:00:41] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:13 [I 2022-10-27 23:00:41] ...trap/engines/engine.py.229: process: 0.54958 | load: 0.00241 [I 2022-10-27 23:00:41] ...trap/engines/engine.py.230: loss: 0.07898 [I 2022-10-27 23:00:41] ...optimizers/trijoint.py.151: total_norm: 15.608859 [I 2022-10-27 23:00:46] ...trap/engines/engine.py.224: train: epoch 92 | batch 2290/2304 [I 2022-10-27 23:00:46] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:07 [I 2022-10-27 23:00:46] ...trap/engines/engine.py.229: process: 0.55169 | load: 0.00239 [I 2022-10-27 23:00:46] ...trap/engines/engine.py.230: loss: 0.09297 [I 2022-10-27 23:00:46] ...optimizers/trijoint.py.151: total_norm: 10.821750 [I 2022-10-27 23:00:52] ...trap/engines/engine.py.224: train: epoch 92 | batch 2300/2304 [I 2022-10-27 23:00:52] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:02 [I 2022-10-27 23:00:52] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00015 [I 2022-10-27 23:00:52] ...trap/engines/engine.py.230: loss: 0.07810 [I 2022-10-27 23:00:52] ...optimizers/trijoint.py.151: total_norm: 14.016445 [I 2022-10-27 23:00:54] ...trap/engines/engine.py.224: train: epoch 92 | batch 2304/2304 [I 2022-10-27 23:00:54] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:00 [I 2022-10-27 23:00:54] ...trap/engines/engine.py.229: process: 0.54513 | load: 0.00016 [I 2022-10-27 23:00:54] ...trap/engines/engine.py.230: loss: 0.14573 [I 2022-10-27 23:00:54] ...optimizers/trijoint.py.151: total_norm: 18.099993 [S 2022-10-27 23:00:55] ...trap/engines/engine.py.129: train_epoch.epoch: 92 [S 2022-10-27 23:00:55] ...trap/engines/engine.py.129: train_epoch.loss: 0.1297341329156706 [I 2022-10-27 23:00:55] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 23:00:57] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 92 [I 2022-10-27 23:00:57] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 23:00:59] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 23:00:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 23:00:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 23:01:02] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 23:01:02] ...trap/engines/engine.py.321: eval: epoch 92 | batch 0/510 [I 2022-10-27 23:01:02] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:45:09 [I 2022-10-27 23:01:02] ...trap/engines/engine.py.326: process: 5.31297 | load: 2.53173 [I 2022-10-27 23:01:05] ...trap/engines/engine.py.321: eval: epoch 92 | batch 10/510 [I 2022-10-27 23:01:05] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:13 [I 2022-10-27 23:01:05] ...trap/engines/engine.py.326: process: 0.32350 | load: 0.00017 [I 2022-10-27 23:01:09] ...trap/engines/engine.py.321: eval: epoch 92 | batch 20/510 [I 2022-10-27 23:01:09] ...trap/engines/engine.py.325: elapsed: 0:00:11 | left: 0:03:10 [I 2022-10-27 23:01:09] ...trap/engines/engine.py.326: process: 0.33548 | load: 0.00020 [I 2022-10-27 23:01:12] ...trap/engines/engine.py.321: eval: epoch 92 | batch 30/510 [I 2022-10-27 23:01:12] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:43 [I 2022-10-27 23:01:12] ...trap/engines/engine.py.326: process: 0.32760 | load: 0.00016 [I 2022-10-27 23:01:15] ...trap/engines/engine.py.321: eval: epoch 92 | batch 40/510 [I 2022-10-27 23:01:15] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:36 [I 2022-10-27 23:01:15] ...trap/engines/engine.py.326: process: 0.32875 | load: 0.00016 [I 2022-10-27 23:01:19] ...trap/engines/engine.py.321: eval: epoch 92 | batch 50/510 [I 2022-10-27 23:01:19] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:33 [I 2022-10-27 23:01:19] ...trap/engines/engine.py.326: process: 0.33126 | load: 0.00016 [I 2022-10-27 23:01:22] ...trap/engines/engine.py.321: eval: epoch 92 | batch 60/510 [I 2022-10-27 23:01:22] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:29 [I 2022-10-27 23:01:22] ...trap/engines/engine.py.326: process: 0.31911 | load: 0.00016 [I 2022-10-27 23:01:25] ...trap/engines/engine.py.321: eval: epoch 92 | batch 70/510 [I 2022-10-27 23:01:25] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:27 [I 2022-10-27 23:01:25] ...trap/engines/engine.py.326: process: 0.32202 | load: 0.00018 [I 2022-10-27 23:01:29] ...trap/engines/engine.py.321: eval: epoch 92 | batch 80/510 [I 2022-10-27 23:01:29] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:23 [I 2022-10-27 23:01:29] ...trap/engines/engine.py.326: process: 0.32579 | load: 0.00016 [I 2022-10-27 23:01:32] ...trap/engines/engine.py.321: eval: epoch 92 | batch 90/510 [I 2022-10-27 23:01:32] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:21 [I 2022-10-27 23:01:32] ...trap/engines/engine.py.326: process: 0.32689 | load: 0.00017 [I 2022-10-27 23:01:35] ...trap/engines/engine.py.321: eval: epoch 92 | batch 100/510 [I 2022-10-27 23:01:35] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:14 [I 2022-10-27 23:01:35] ...trap/engines/engine.py.326: process: 0.28609 | load: 0.00018 [I 2022-10-27 23:01:38] ...trap/engines/engine.py.321: eval: epoch 92 | batch 110/510 [I 2022-10-27 23:01:38] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 23:01:38] ...trap/engines/engine.py.326: process: 0.29873 | load: 0.00014 [I 2022-10-27 23:01:41] ...trap/engines/engine.py.321: eval: epoch 92 | batch 120/510 [I 2022-10-27 23:01:41] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 23:01:41] ...trap/engines/engine.py.326: process: 0.29410 | load: 0.00014 [I 2022-10-27 23:01:44] ...trap/engines/engine.py.321: eval: epoch 92 | batch 130/510 [I 2022-10-27 23:01:44] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-27 23:01:44] ...trap/engines/engine.py.326: process: 0.28445 | load: 0.00014 [I 2022-10-27 23:01:47] ...trap/engines/engine.py.321: eval: epoch 92 | batch 140/510 [I 2022-10-27 23:01:47] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-27 23:01:47] ...trap/engines/engine.py.326: process: 0.28378 | load: 0.00014 [I 2022-10-27 23:01:50] ...trap/engines/engine.py.321: eval: epoch 92 | batch 150/510 [I 2022-10-27 23:01:50] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-27 23:01:50] ...trap/engines/engine.py.326: process: 0.28314 | load: 0.00014 [I 2022-10-27 23:01:53] ...trap/engines/engine.py.321: eval: epoch 92 | batch 160/510 [I 2022-10-27 23:01:53] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-27 23:01:53] ...trap/engines/engine.py.326: process: 0.29220 | load: 0.00014 [I 2022-10-27 23:01:55] ...trap/engines/engine.py.321: eval: epoch 92 | batch 170/510 [I 2022-10-27 23:01:55] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:37 [I 2022-10-27 23:01:55] ...trap/engines/engine.py.326: process: 0.28424 | load: 0.00014 [I 2022-10-27 23:01:58] ...trap/engines/engine.py.321: eval: epoch 92 | batch 180/510 [I 2022-10-27 23:01:58] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:34 [I 2022-10-27 23:01:58] ...trap/engines/engine.py.326: process: 0.28259 | load: 0.00018 [I 2022-10-27 23:02:01] ...trap/engines/engine.py.321: eval: epoch 92 | batch 190/510 [I 2022-10-27 23:02:01] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-27 23:02:01] ...trap/engines/engine.py.326: process: 0.29009 | load: 0.00015 [I 2022-10-27 23:02:04] ...trap/engines/engine.py.321: eval: epoch 92 | batch 200/510 [I 2022-10-27 23:02:04] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 23:02:04] ...trap/engines/engine.py.326: process: 0.29607 | load: 0.00015 [I 2022-10-27 23:02:07] ...trap/engines/engine.py.321: eval: epoch 92 | batch 210/510 [I 2022-10-27 23:02:07] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:25 [I 2022-10-27 23:02:07] ...trap/engines/engine.py.326: process: 0.28578 | load: 0.00014 [I 2022-10-27 23:02:10] ...trap/engines/engine.py.321: eval: epoch 92 | batch 220/510 [I 2022-10-27 23:02:10] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 23:02:10] ...trap/engines/engine.py.326: process: 0.28700 | load: 0.00014 [I 2022-10-27 23:02:13] ...trap/engines/engine.py.321: eval: epoch 92 | batch 230/510 [I 2022-10-27 23:02:13] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 23:02:13] ...trap/engines/engine.py.326: process: 0.27839 | load: 0.00018 [I 2022-10-27 23:02:16] ...trap/engines/engine.py.321: eval: epoch 92 | batch 240/510 [I 2022-10-27 23:02:16] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-27 23:02:16] ...trap/engines/engine.py.326: process: 0.29419 | load: 0.00023 [I 2022-10-27 23:02:19] ...trap/engines/engine.py.321: eval: epoch 92 | batch 250/510 [I 2022-10-27 23:02:19] ...trap/engines/engine.py.325: elapsed: 0:01:21 | left: 0:01:14 [I 2022-10-27 23:02:19] ...trap/engines/engine.py.326: process: 0.28792 | load: 0.00024 [I 2022-10-27 23:02:21] ...trap/engines/engine.py.321: eval: epoch 92 | batch 260/510 [I 2022-10-27 23:02:21] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:12 [I 2022-10-27 23:02:21] ...trap/engines/engine.py.326: process: 0.28730 | load: 0.00023 [I 2022-10-27 23:02:24] ...trap/engines/engine.py.321: eval: epoch 92 | batch 270/510 [I 2022-10-27 23:02:24] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:09 [I 2022-10-27 23:02:24] ...trap/engines/engine.py.326: process: 0.28973 | load: 0.00026 [I 2022-10-27 23:02:27] ...trap/engines/engine.py.321: eval: epoch 92 | batch 280/510 [I 2022-10-27 23:02:27] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-27 23:02:27] ...trap/engines/engine.py.326: process: 0.29205 | load: 0.00019 [I 2022-10-27 23:02:30] ...trap/engines/engine.py.321: eval: epoch 92 | batch 290/510 [I 2022-10-27 23:02:30] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-27 23:02:30] ...trap/engines/engine.py.326: process: 0.29324 | load: 0.00020 [I 2022-10-27 23:02:33] ...trap/engines/engine.py.321: eval: epoch 92 | batch 300/510 [I 2022-10-27 23:02:33] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-27 23:02:33] ...trap/engines/engine.py.326: process: 0.28473 | load: 0.00018 [I 2022-10-27 23:02:36] ...trap/engines/engine.py.321: eval: epoch 92 | batch 310/510 [I 2022-10-27 23:02:36] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 23:02:36] ...trap/engines/engine.py.326: process: 0.28838 | load: 0.00020 [I 2022-10-27 23:02:39] ...trap/engines/engine.py.321: eval: epoch 92 | batch 320/510 [I 2022-10-27 23:02:39] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 23:02:39] ...trap/engines/engine.py.326: process: 0.29135 | load: 0.00016 [I 2022-10-27 23:02:42] ...trap/engines/engine.py.321: eval: epoch 92 | batch 330/510 [I 2022-10-27 23:02:42] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-27 23:02:42] ...trap/engines/engine.py.326: process: 0.28368 | load: 0.00016 [I 2022-10-27 23:02:45] ...trap/engines/engine.py.321: eval: epoch 92 | batch 340/510 [I 2022-10-27 23:02:45] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:50 [I 2022-10-27 23:02:45] ...trap/engines/engine.py.326: process: 0.29069 | load: 0.00015 [I 2022-10-27 23:02:48] ...trap/engines/engine.py.321: eval: epoch 92 | batch 350/510 [I 2022-10-27 23:02:48] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 23:02:48] ...trap/engines/engine.py.326: process: 0.28380 | load: 0.00017 [I 2022-10-27 23:02:50] ...trap/engines/engine.py.321: eval: epoch 92 | batch 360/510 [I 2022-10-27 23:02:50] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-27 23:02:50] ...trap/engines/engine.py.326: process: 0.29254 | load: 0.00016 [I 2022-10-27 23:02:53] ...trap/engines/engine.py.321: eval: epoch 92 | batch 370/510 [I 2022-10-27 23:02:53] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-27 23:02:53] ...trap/engines/engine.py.326: process: 0.28493 | load: 0.00018 [I 2022-10-27 23:02:56] ...trap/engines/engine.py.321: eval: epoch 92 | batch 380/510 [I 2022-10-27 23:02:56] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-27 23:02:56] ...trap/engines/engine.py.326: process: 0.28414 | load: 0.00016 [I 2022-10-27 23:02:59] ...trap/engines/engine.py.321: eval: epoch 92 | batch 390/510 [I 2022-10-27 23:02:59] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 23:02:59] ...trap/engines/engine.py.326: process: 0.29105 | load: 0.00029 [I 2022-10-27 23:03:02] ...trap/engines/engine.py.321: eval: epoch 92 | batch 400/510 [I 2022-10-27 23:03:02] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 23:03:02] ...trap/engines/engine.py.326: process: 0.29562 | load: 0.00019 [I 2022-10-27 23:03:05] ...trap/engines/engine.py.321: eval: epoch 92 | batch 410/510 [I 2022-10-27 23:03:05] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:29 [I 2022-10-27 23:03:05] ...trap/engines/engine.py.326: process: 0.29120 | load: 0.00016 [I 2022-10-27 23:03:08] ...trap/engines/engine.py.321: eval: epoch 92 | batch 420/510 [I 2022-10-27 23:03:08] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 23:03:08] ...trap/engines/engine.py.326: process: 0.28328 | load: 0.00019 [I 2022-10-27 23:03:11] ...trap/engines/engine.py.321: eval: epoch 92 | batch 430/510 [I 2022-10-27 23:03:11] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 23:03:11] ...trap/engines/engine.py.326: process: 0.28643 | load: 0.00026 [I 2022-10-27 23:03:14] ...trap/engines/engine.py.321: eval: epoch 92 | batch 440/510 [I 2022-10-27 23:03:14] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 23:03:14] ...trap/engines/engine.py.326: process: 0.29074 | load: 0.00021 [I 2022-10-27 23:03:17] ...trap/engines/engine.py.321: eval: epoch 92 | batch 450/510 [I 2022-10-27 23:03:17] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-27 23:03:17] ...trap/engines/engine.py.326: process: 0.28672 | load: 0.00020 [I 2022-10-27 23:03:19] ...trap/engines/engine.py.321: eval: epoch 92 | batch 460/510 [I 2022-10-27 23:03:19] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-27 23:03:19] ...trap/engines/engine.py.326: process: 0.28314 | load: 0.00017 [I 2022-10-27 23:03:22] ...trap/engines/engine.py.321: eval: epoch 92 | batch 470/510 [I 2022-10-27 23:03:22] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-27 23:03:22] ...trap/engines/engine.py.326: process: 0.28424 | load: 0.00018 [I 2022-10-27 23:03:25] ...trap/engines/engine.py.321: eval: epoch 92 | batch 480/510 [I 2022-10-27 23:03:25] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 23:03:25] ...trap/engines/engine.py.326: process: 0.28971 | load: 0.00018 [I 2022-10-27 23:03:28] ...trap/engines/engine.py.321: eval: epoch 92 | batch 490/510 [I 2022-10-27 23:03:28] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 23:03:28] ...trap/engines/engine.py.326: process: 0.29230 | load: 0.00022 [I 2022-10-27 23:03:31] ...trap/engines/engine.py.321: eval: epoch 92 | batch 500/510 [I 2022-10-27 23:03:31] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 23:03:31] ...trap/engines/engine.py.326: process: 0.28442 | load: 0.00021 [I 2022-10-27 23:03:34] ...trap/engines/engine.py.321: eval: epoch 92 | batch 510/510 [I 2022-10-27 23:03:34] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 23:03:34] ...trap/engines/engine.py.326: process: 0.28456 | load: 0.00011 [S 2022-10-27 23:03:35] ...trap/engines/engine.py.132: eval_epoch.epoch: 92 [I 2022-10-27 23:03:35] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.739200 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.910100 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937500 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.733900 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.908800 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936200 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012836 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008068 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007928 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010968 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008964 [S 2022-10-27 23:03:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006242 [I 2022-10-27 23:03:41] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 23:03:41] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 23:03:41] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 23:03:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 23:03:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 23:03:45] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 23:03:46] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 23:03:52] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 23:03:52] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 23:03:52] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 23:03:55] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 23:04:02] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 23:04:02] ...trap/engines/engine.py.163: Training model on trainset for epoch 93 [I 2022-10-27 23:04:02] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 23:04:04] ...trap/engines/engine.py.224: train: epoch 93 | batch 0/2304 [I 2022-10-27 23:04:04] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:54 [I 2022-10-27 23:04:04] ...trap/engines/engine.py.229: process: 2.73208 | load: 2.08333 [I 2022-10-27 23:04:04] ...trap/engines/engine.py.230: loss: 0.16036 [I 2022-10-27 23:04:04] ...optimizers/trijoint.py.151: total_norm: 17.842949 [I 2022-10-27 23:04:10] ...trap/engines/engine.py.224: train: epoch 93 | batch 10/2304 [I 2022-10-27 23:04:10] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:18 [I 2022-10-27 23:04:10] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00378 [I 2022-10-27 23:04:10] ...trap/engines/engine.py.230: loss: 0.17939 [I 2022-10-27 23:04:10] ...optimizers/trijoint.py.151: total_norm: 18.760534 [I 2022-10-27 23:04:16] ...trap/engines/engine.py.224: train: epoch 93 | batch 20/2304 [I 2022-10-27 23:04:16] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:10 [I 2022-10-27 23:04:16] ...trap/engines/engine.py.229: process: 0.54901 | load: 0.00357 [I 2022-10-27 23:04:16] ...trap/engines/engine.py.230: loss: 0.13366 [I 2022-10-27 23:04:16] ...optimizers/trijoint.py.151: total_norm: 21.063995 [I 2022-10-27 23:04:21] ...trap/engines/engine.py.224: train: epoch 93 | batch 30/2304 [I 2022-10-27 23:04:21] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:17 [I 2022-10-27 23:04:21] ...trap/engines/engine.py.229: process: 0.56600 | load: 0.00270 [I 2022-10-27 23:04:21] ...trap/engines/engine.py.230: loss: 0.11650 [I 2022-10-27 23:04:21] ...optimizers/trijoint.py.151: total_norm: 14.091140 [I 2022-10-27 23:04:27] ...trap/engines/engine.py.224: train: epoch 93 | batch 40/2304 [I 2022-10-27 23:04:27] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:09 [I 2022-10-27 23:04:27] ...trap/engines/engine.py.229: process: 0.56723 | load: 0.00303 [I 2022-10-27 23:04:27] ...trap/engines/engine.py.230: loss: 0.08061 [I 2022-10-27 23:04:27] ...optimizers/trijoint.py.151: total_norm: 18.217728 [I 2022-10-27 23:04:32] ...trap/engines/engine.py.224: train: epoch 93 | batch 50/2304 [I 2022-10-27 23:04:32] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:02 [I 2022-10-27 23:04:32] ...trap/engines/engine.py.229: process: 0.56447 | load: 0.00269 [I 2022-10-27 23:04:32] ...trap/engines/engine.py.230: loss: 0.16434 [I 2022-10-27 23:04:32] ...optimizers/trijoint.py.151: total_norm: 18.894068 [I 2022-10-27 23:04:38] ...trap/engines/engine.py.224: train: epoch 93 | batch 60/2304 [I 2022-10-27 23:04:38] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:53 [I 2022-10-27 23:04:38] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00271 [I 2022-10-27 23:04:38] ...trap/engines/engine.py.230: loss: 0.11370 [I 2022-10-27 23:04:38] ...optimizers/trijoint.py.151: total_norm: 16.947224 [I 2022-10-27 23:04:44] ...trap/engines/engine.py.224: train: epoch 93 | batch 70/2304 [I 2022-10-27 23:04:44] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:44 [I 2022-10-27 23:04:44] ...trap/engines/engine.py.229: process: 0.54869 | load: 0.00264 [I 2022-10-27 23:04:44] ...trap/engines/engine.py.230: loss: 0.12277 [I 2022-10-27 23:04:44] ...optimizers/trijoint.py.151: total_norm: 17.345549 [I 2022-10-27 23:04:49] ...trap/engines/engine.py.224: train: epoch 93 | batch 80/2304 [I 2022-10-27 23:04:49] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:42 [I 2022-10-27 23:04:49] ...trap/engines/engine.py.229: process: 0.57170 | load: 0.00275 [I 2022-10-27 23:04:49] ...trap/engines/engine.py.230: loss: 0.10132 [I 2022-10-27 23:04:49] ...optimizers/trijoint.py.151: total_norm: 17.114109 [I 2022-10-27 23:04:55] ...trap/engines/engine.py.224: train: epoch 93 | batch 90/2304 [I 2022-10-27 23:04:55] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:26 [I 2022-10-27 23:04:55] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00263 [I 2022-10-27 23:04:55] ...trap/engines/engine.py.230: loss: 0.11715 [I 2022-10-27 23:04:55] ...optimizers/trijoint.py.151: total_norm: 14.264195 [I 2022-10-27 23:05:20] ...trap/engines/engine.py.224: train: epoch 93 | batch 100/2304 [I 2022-10-27 23:05:20] ...trap/engines/engine.py.228: elapsed: 0:01:18 | left: 1:54:26 [I 2022-10-27 23:05:20] ...trap/engines/engine.py.229: process: 0.55075 | load: 0.00248 [I 2022-10-27 23:05:20] ...trap/engines/engine.py.230: loss: 0.15715 [I 2022-10-27 23:05:20] ...optimizers/trijoint.py.151: total_norm: 23.645981 [I 2022-10-27 23:05:26] ...trap/engines/engine.py.224: train: epoch 93 | batch 110/2304 [I 2022-10-27 23:05:26] ...trap/engines/engine.py.228: elapsed: 0:01:24 | left: 0:30:39 [I 2022-10-27 23:05:26] ...trap/engines/engine.py.229: process: 0.56536 | load: 0.00272 [I 2022-10-27 23:05:26] ...trap/engines/engine.py.230: loss: 0.08494 [I 2022-10-27 23:05:26] ...optimizers/trijoint.py.151: total_norm: 14.194738 [I 2022-10-27 23:05:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 120/2304 [I 2022-10-27 23:05:31] ...trap/engines/engine.py.228: elapsed: 0:01:29 | left: 0:21:17 [I 2022-10-27 23:05:31] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00269 [I 2022-10-27 23:05:31] ...trap/engines/engine.py.230: loss: 0.13357 [I 2022-10-27 23:05:31] ...optimizers/trijoint.py.151: total_norm: 22.751192 [I 2022-10-27 23:05:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 130/2304 [I 2022-10-27 23:05:37] ...trap/engines/engine.py.228: elapsed: 0:01:35 | left: 0:20:23 [I 2022-10-27 23:05:37] ...trap/engines/engine.py.229: process: 0.56157 | load: 0.00260 [I 2022-10-27 23:05:37] ...trap/engines/engine.py.230: loss: 0.22285 [I 2022-10-27 23:05:37] ...optimizers/trijoint.py.151: total_norm: 37.420254 [I 2022-10-27 23:05:43] ...trap/engines/engine.py.224: train: epoch 93 | batch 140/2304 [I 2022-10-27 23:05:43] ...trap/engines/engine.py.228: elapsed: 0:01:41 | left: 0:20:13 [I 2022-10-27 23:05:43] ...trap/engines/engine.py.229: process: 0.56498 | load: 0.00274 [I 2022-10-27 23:05:43] ...trap/engines/engine.py.230: loss: 0.09080 [I 2022-10-27 23:05:43] ...optimizers/trijoint.py.151: total_norm: 23.103226 [I 2022-10-27 23:05:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 150/2304 [I 2022-10-27 23:05:48] ...trap/engines/engine.py.228: elapsed: 0:01:46 | left: 0:19:58 [I 2022-10-27 23:05:48] ...trap/engines/engine.py.229: process: 0.55073 | load: 0.00289 [I 2022-10-27 23:05:48] ...trap/engines/engine.py.230: loss: 0.21672 [I 2022-10-27 23:05:48] ...optimizers/trijoint.py.151: total_norm: 22.559891 [I 2022-10-27 23:05:54] ...trap/engines/engine.py.224: train: epoch 93 | batch 160/2304 [I 2022-10-27 23:05:54] ...trap/engines/engine.py.228: elapsed: 0:01:52 | left: 0:19:52 [I 2022-10-27 23:05:54] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00293 [I 2022-10-27 23:05:54] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-27 23:05:54] ...optimizers/trijoint.py.151: total_norm: 18.250237 [I 2022-10-27 23:05:59] ...trap/engines/engine.py.224: train: epoch 93 | batch 170/2304 [I 2022-10-27 23:05:59] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:19:48 [I 2022-10-27 23:05:59] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00263 [I 2022-10-27 23:05:59] ...trap/engines/engine.py.230: loss: 0.16368 [I 2022-10-27 23:05:59] ...optimizers/trijoint.py.151: total_norm: 18.709705 [I 2022-10-27 23:06:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 180/2304 [I 2022-10-27 23:06:05] ...trap/engines/engine.py.228: elapsed: 0:02:03 | left: 0:19:42 [I 2022-10-27 23:06:05] ...trap/engines/engine.py.229: process: 0.54735 | load: 0.00258 [I 2022-10-27 23:06:05] ...trap/engines/engine.py.230: loss: 0.12991 [I 2022-10-27 23:06:05] ...optimizers/trijoint.py.151: total_norm: 16.453161 [I 2022-10-27 23:06:11] ...trap/engines/engine.py.224: train: epoch 93 | batch 190/2304 [I 2022-10-27 23:06:11] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:19:29 [I 2022-10-27 23:06:11] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00248 [I 2022-10-27 23:06:11] ...trap/engines/engine.py.230: loss: 0.22674 [I 2022-10-27 23:06:11] ...optimizers/trijoint.py.151: total_norm: 20.333475 [I 2022-10-27 23:06:16] ...trap/engines/engine.py.224: train: epoch 93 | batch 200/2304 [I 2022-10-27 23:06:16] ...trap/engines/engine.py.228: elapsed: 0:02:14 | left: 0:19:24 [I 2022-10-27 23:06:16] ...trap/engines/engine.py.229: process: 0.55406 | load: 0.00256 [I 2022-10-27 23:06:16] ...trap/engines/engine.py.230: loss: 0.17068 [I 2022-10-27 23:06:16] ...optimizers/trijoint.py.151: total_norm: 19.961361 [I 2022-10-27 23:06:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 210/2304 [I 2022-10-27 23:06:22] ...trap/engines/engine.py.228: elapsed: 0:02:20 | left: 0:19:17 [I 2022-10-27 23:06:22] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00256 [I 2022-10-27 23:06:22] ...trap/engines/engine.py.230: loss: 0.31201 [I 2022-10-27 23:06:22] ...optimizers/trijoint.py.151: total_norm: 24.971260 [I 2022-10-27 23:06:27] ...trap/engines/engine.py.224: train: epoch 93 | batch 220/2304 [I 2022-10-27 23:06:27] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:19:15 [I 2022-10-27 23:06:27] ...trap/engines/engine.py.229: process: 0.55363 | load: 0.00259 [I 2022-10-27 23:06:27] ...trap/engines/engine.py.230: loss: 0.15148 [I 2022-10-27 23:06:27] ...optimizers/trijoint.py.151: total_norm: 15.519154 [I 2022-10-27 23:06:33] ...trap/engines/engine.py.224: train: epoch 93 | batch 230/2304 [I 2022-10-27 23:06:33] ...trap/engines/engine.py.228: elapsed: 0:02:31 | left: 0:19:07 [I 2022-10-27 23:06:33] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00256 [I 2022-10-27 23:06:33] ...trap/engines/engine.py.230: loss: 0.13925 [I 2022-10-27 23:06:33] ...optimizers/trijoint.py.151: total_norm: 18.977240 [I 2022-10-27 23:06:38] ...trap/engines/engine.py.224: train: epoch 93 | batch 240/2304 [I 2022-10-27 23:06:38] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:19:46 [I 2022-10-27 23:06:38] ...trap/engines/engine.py.229: process: 0.62827 | load: 0.00257 [I 2022-10-27 23:06:38] ...trap/engines/engine.py.230: loss: 0.19620 [I 2022-10-27 23:06:38] ...optimizers/trijoint.py.151: total_norm: 15.534814 [I 2022-10-27 23:06:44] ...trap/engines/engine.py.224: train: epoch 93 | batch 250/2304 [I 2022-10-27 23:06:44] ...trap/engines/engine.py.228: elapsed: 0:02:42 | left: 0:19:06 [I 2022-10-27 23:06:44] ...trap/engines/engine.py.229: process: 0.55576 | load: 0.00245 [I 2022-10-27 23:06:44] ...trap/engines/engine.py.230: loss: 0.14407 [I 2022-10-27 23:06:44] ...optimizers/trijoint.py.151: total_norm: 16.911472 [I 2022-10-27 23:06:50] ...trap/engines/engine.py.224: train: epoch 93 | batch 260/2304 [I 2022-10-27 23:06:50] ...trap/engines/engine.py.228: elapsed: 0:02:48 | left: 0:19:39 [I 2022-10-27 23:06:50] ...trap/engines/engine.py.229: process: 0.58325 | load: 0.00286 [I 2022-10-27 23:06:50] ...trap/engines/engine.py.230: loss: 0.12970 [I 2022-10-27 23:06:50] ...optimizers/trijoint.py.151: total_norm: 12.399147 [I 2022-10-27 23:06:55] ...trap/engines/engine.py.224: train: epoch 93 | batch 270/2304 [I 2022-10-27 23:06:55] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:18:55 [I 2022-10-27 23:06:55] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00284 [I 2022-10-27 23:06:55] ...trap/engines/engine.py.230: loss: 0.11038 [I 2022-10-27 23:06:55] ...optimizers/trijoint.py.151: total_norm: 14.397395 [I 2022-10-27 23:07:01] ...trap/engines/engine.py.224: train: epoch 93 | batch 280/2304 [I 2022-10-27 23:07:01] ...trap/engines/engine.py.228: elapsed: 0:02:59 | left: 0:18:43 [I 2022-10-27 23:07:01] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00263 [I 2022-10-27 23:07:01] ...trap/engines/engine.py.230: loss: 0.20527 [I 2022-10-27 23:07:01] ...optimizers/trijoint.py.151: total_norm: 20.906874 [I 2022-10-27 23:07:06] ...trap/engines/engine.py.224: train: epoch 93 | batch 290/2304 [I 2022-10-27 23:07:06] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:18:37 [I 2022-10-27 23:07:06] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00263 [I 2022-10-27 23:07:06] ...trap/engines/engine.py.230: loss: 0.16336 [I 2022-10-27 23:07:06] ...optimizers/trijoint.py.151: total_norm: 18.082150 [I 2022-10-27 23:07:12] ...trap/engines/engine.py.224: train: epoch 93 | batch 300/2304 [I 2022-10-27 23:07:12] ...trap/engines/engine.py.228: elapsed: 0:03:10 | left: 0:18:30 [I 2022-10-27 23:07:12] ...trap/engines/engine.py.229: process: 0.54301 | load: 0.00260 [I 2022-10-27 23:07:12] ...trap/engines/engine.py.230: loss: 0.15309 [I 2022-10-27 23:07:12] ...optimizers/trijoint.py.151: total_norm: 22.566969 [I 2022-10-27 23:07:18] ...trap/engines/engine.py.224: train: epoch 93 | batch 310/2304 [I 2022-10-27 23:07:18] ...trap/engines/engine.py.228: elapsed: 0:03:15 | left: 0:18:28 [I 2022-10-27 23:07:18] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00273 [I 2022-10-27 23:07:18] ...trap/engines/engine.py.230: loss: 0.10025 [I 2022-10-27 23:07:18] ...optimizers/trijoint.py.151: total_norm: 11.119721 [I 2022-10-27 23:07:23] ...trap/engines/engine.py.224: train: epoch 93 | batch 320/2304 [I 2022-10-27 23:07:23] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:18:18 [I 2022-10-27 23:07:23] ...trap/engines/engine.py.229: process: 0.56337 | load: 0.00267 [I 2022-10-27 23:07:23] ...trap/engines/engine.py.230: loss: 0.18048 [I 2022-10-27 23:07:23] ...optimizers/trijoint.py.151: total_norm: 17.166658 [I 2022-10-27 23:07:29] ...trap/engines/engine.py.224: train: epoch 93 | batch 330/2304 [I 2022-10-27 23:07:29] ...trap/engines/engine.py.228: elapsed: 0:03:27 | left: 0:18:44 [I 2022-10-27 23:07:29] ...trap/engines/engine.py.229: process: 0.54252 | load: 0.00268 [I 2022-10-27 23:07:29] ...trap/engines/engine.py.230: loss: 0.15238 [I 2022-10-27 23:07:29] ...optimizers/trijoint.py.151: total_norm: 33.852055 [I 2022-10-27 23:07:34] ...trap/engines/engine.py.224: train: epoch 93 | batch 340/2304 [I 2022-10-27 23:07:34] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:18:19 [I 2022-10-27 23:07:34] ...trap/engines/engine.py.229: process: 0.56675 | load: 0.00266 [I 2022-10-27 23:07:34] ...trap/engines/engine.py.230: loss: 0.08935 [I 2022-10-27 23:07:34] ...optimizers/trijoint.py.151: total_norm: 14.748456 [I 2022-10-27 23:07:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 350/2304 [I 2022-10-27 23:07:40] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:18:01 [I 2022-10-27 23:07:40] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00250 [I 2022-10-27 23:07:40] ...trap/engines/engine.py.230: loss: 0.09869 [I 2022-10-27 23:07:40] ...optimizers/trijoint.py.151: total_norm: 17.461332 [I 2022-10-27 23:07:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 360/2304 [I 2022-10-27 23:07:45] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:17:52 [I 2022-10-27 23:07:45] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00248 [I 2022-10-27 23:07:45] ...trap/engines/engine.py.230: loss: 0.13730 [I 2022-10-27 23:07:45] ...optimizers/trijoint.py.151: total_norm: 14.970800 [I 2022-10-27 23:07:51] ...trap/engines/engine.py.224: train: epoch 93 | batch 370/2304 [I 2022-10-27 23:07:51] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:17:46 [I 2022-10-27 23:07:51] ...trap/engines/engine.py.229: process: 0.54841 | load: 0.00263 [I 2022-10-27 23:07:51] ...trap/engines/engine.py.230: loss: 0.07549 [I 2022-10-27 23:07:51] ...optimizers/trijoint.py.151: total_norm: 10.292493 [I 2022-10-27 23:07:56] ...trap/engines/engine.py.224: train: epoch 93 | batch 380/2304 [I 2022-10-27 23:07:56] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:17:36 [I 2022-10-27 23:07:56] ...trap/engines/engine.py.229: process: 0.54526 | load: 0.00246 [I 2022-10-27 23:07:56] ...trap/engines/engine.py.230: loss: 0.12640 [I 2022-10-27 23:07:56] ...optimizers/trijoint.py.151: total_norm: 27.343843 [I 2022-10-27 23:08:02] ...trap/engines/engine.py.224: train: epoch 93 | batch 390/2304 [I 2022-10-27 23:08:02] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:17:38 [I 2022-10-27 23:08:02] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00249 [I 2022-10-27 23:08:02] ...trap/engines/engine.py.230: loss: 0.12673 [I 2022-10-27 23:08:02] ...optimizers/trijoint.py.151: total_norm: 13.602316 [I 2022-10-27 23:08:08] ...trap/engines/engine.py.224: train: epoch 93 | batch 400/2304 [I 2022-10-27 23:08:08] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:17:46 [I 2022-10-27 23:08:08] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00252 [I 2022-10-27 23:08:08] ...trap/engines/engine.py.230: loss: 0.08283 [I 2022-10-27 23:08:08] ...optimizers/trijoint.py.151: total_norm: 9.166478 [I 2022-10-27 23:08:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 410/2304 [I 2022-10-27 23:08:13] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:17:43 [I 2022-10-27 23:08:13] ...trap/engines/engine.py.229: process: 0.55544 | load: 0.00261 [I 2022-10-27 23:08:13] ...trap/engines/engine.py.230: loss: 0.12895 [I 2022-10-27 23:08:13] ...optimizers/trijoint.py.151: total_norm: 12.794647 [I 2022-10-27 23:08:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 420/2304 [I 2022-10-27 23:08:19] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:17:28 [I 2022-10-27 23:08:19] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00250 [I 2022-10-27 23:08:19] ...trap/engines/engine.py.230: loss: 0.08798 [I 2022-10-27 23:08:19] ...optimizers/trijoint.py.151: total_norm: 14.143032 [I 2022-10-27 23:08:24] ...trap/engines/engine.py.224: train: epoch 93 | batch 430/2304 [I 2022-10-27 23:08:24] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:17:19 [I 2022-10-27 23:08:24] ...trap/engines/engine.py.229: process: 0.55932 | load: 0.00273 [I 2022-10-27 23:08:24] ...trap/engines/engine.py.230: loss: 0.12654 [I 2022-10-27 23:08:24] ...optimizers/trijoint.py.151: total_norm: 31.610130 [I 2022-10-27 23:08:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 440/2304 [I 2022-10-27 23:08:30] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:17:08 [I 2022-10-27 23:08:30] ...trap/engines/engine.py.229: process: 0.55302 | load: 0.00242 [I 2022-10-27 23:08:30] ...trap/engines/engine.py.230: loss: 0.09768 [I 2022-10-27 23:08:30] ...optimizers/trijoint.py.151: total_norm: 21.946054 [I 2022-10-27 23:08:36] ...trap/engines/engine.py.224: train: epoch 93 | batch 450/2304 [I 2022-10-27 23:08:36] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:17:11 [I 2022-10-27 23:08:36] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00249 [I 2022-10-27 23:08:36] ...trap/engines/engine.py.230: loss: 0.12286 [I 2022-10-27 23:08:36] ...optimizers/trijoint.py.151: total_norm: 8.438100 [I 2022-10-27 23:08:41] ...trap/engines/engine.py.224: train: epoch 93 | batch 460/2304 [I 2022-10-27 23:08:41] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:17:02 [I 2022-10-27 23:08:41] ...trap/engines/engine.py.229: process: 0.55295 | load: 0.00247 [I 2022-10-27 23:08:41] ...trap/engines/engine.py.230: loss: 0.10971 [I 2022-10-27 23:08:41] ...optimizers/trijoint.py.151: total_norm: 13.116567 [I 2022-10-27 23:08:47] ...trap/engines/engine.py.224: train: epoch 93 | batch 470/2304 [I 2022-10-27 23:08:47] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:17:01 [I 2022-10-27 23:08:47] ...trap/engines/engine.py.229: process: 0.56462 | load: 0.00249 [I 2022-10-27 23:08:47] ...trap/engines/engine.py.230: loss: 0.12186 [I 2022-10-27 23:08:47] ...optimizers/trijoint.py.151: total_norm: 19.478735 [I 2022-10-27 23:08:52] ...trap/engines/engine.py.224: train: epoch 93 | batch 480/2304 [I 2022-10-27 23:08:52] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:16:53 [I 2022-10-27 23:08:52] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00250 [I 2022-10-27 23:08:52] ...trap/engines/engine.py.230: loss: 0.09739 [I 2022-10-27 23:08:52] ...optimizers/trijoint.py.151: total_norm: 11.348041 [I 2022-10-27 23:08:58] ...trap/engines/engine.py.224: train: epoch 93 | batch 490/2304 [I 2022-10-27 23:08:58] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:16:48 [I 2022-10-27 23:08:58] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00250 [I 2022-10-27 23:08:58] ...trap/engines/engine.py.230: loss: 0.15007 [I 2022-10-27 23:08:58] ...optimizers/trijoint.py.151: total_norm: 62.747002 [I 2022-10-27 23:09:03] ...trap/engines/engine.py.224: train: epoch 93 | batch 500/2304 [I 2022-10-27 23:09:03] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:16:48 [I 2022-10-27 23:09:03] ...trap/engines/engine.py.229: process: 0.56318 | load: 0.00249 [I 2022-10-27 23:09:03] ...trap/engines/engine.py.230: loss: 0.15196 [I 2022-10-27 23:09:03] ...optimizers/trijoint.py.151: total_norm: 27.345360 [I 2022-10-27 23:09:09] ...trap/engines/engine.py.224: train: epoch 93 | batch 510/2304 [I 2022-10-27 23:09:09] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:16:40 [I 2022-10-27 23:09:09] ...trap/engines/engine.py.229: process: 0.56658 | load: 0.00258 [I 2022-10-27 23:09:09] ...trap/engines/engine.py.230: loss: 0.10130 [I 2022-10-27 23:09:09] ...optimizers/trijoint.py.151: total_norm: 17.153206 [I 2022-10-27 23:09:15] ...trap/engines/engine.py.224: train: epoch 93 | batch 520/2304 [I 2022-10-27 23:09:15] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:16:47 [I 2022-10-27 23:09:15] ...trap/engines/engine.py.229: process: 0.56240 | load: 0.00277 [I 2022-10-27 23:09:15] ...trap/engines/engine.py.230: loss: 0.18726 [I 2022-10-27 23:09:15] ...optimizers/trijoint.py.151: total_norm: 23.229998 [I 2022-10-27 23:09:20] ...trap/engines/engine.py.224: train: epoch 93 | batch 530/2304 [I 2022-10-27 23:09:20] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:16:30 [I 2022-10-27 23:09:20] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00270 [I 2022-10-27 23:09:20] ...trap/engines/engine.py.230: loss: 0.12363 [I 2022-10-27 23:09:20] ...optimizers/trijoint.py.151: total_norm: 28.647955 [I 2022-10-27 23:09:26] ...trap/engines/engine.py.224: train: epoch 93 | batch 540/2304 [I 2022-10-27 23:09:26] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:16:20 [I 2022-10-27 23:09:26] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00246 [I 2022-10-27 23:09:26] ...trap/engines/engine.py.230: loss: 0.10365 [I 2022-10-27 23:09:26] ...optimizers/trijoint.py.151: total_norm: 14.257332 [I 2022-10-27 23:09:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 550/2304 [I 2022-10-27 23:09:31] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:16:05 [I 2022-10-27 23:09:31] ...trap/engines/engine.py.229: process: 0.54794 | load: 0.00269 [I 2022-10-27 23:09:31] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-27 23:09:31] ...optimizers/trijoint.py.151: total_norm: 20.300262 [I 2022-10-27 23:09:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 560/2304 [I 2022-10-27 23:09:37] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:16:08 [I 2022-10-27 23:09:37] ...trap/engines/engine.py.229: process: 0.55684 | load: 0.00247 [I 2022-10-27 23:09:37] ...trap/engines/engine.py.230: loss: 0.08285 [I 2022-10-27 23:09:37] ...optimizers/trijoint.py.151: total_norm: 11.294115 [I 2022-10-27 23:09:43] ...trap/engines/engine.py.224: train: epoch 93 | batch 570/2304 [I 2022-10-27 23:09:43] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:16:06 [I 2022-10-27 23:09:43] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00244 [I 2022-10-27 23:09:43] ...trap/engines/engine.py.230: loss: 0.11995 [I 2022-10-27 23:09:43] ...optimizers/trijoint.py.151: total_norm: 12.856794 [I 2022-10-27 23:09:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 580/2304 [I 2022-10-27 23:09:48] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:16:05 [I 2022-10-27 23:09:48] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00247 [I 2022-10-27 23:09:48] ...trap/engines/engine.py.230: loss: 0.08754 [I 2022-10-27 23:09:48] ...optimizers/trijoint.py.151: total_norm: 12.910540 [I 2022-10-27 23:09:54] ...trap/engines/engine.py.224: train: epoch 93 | batch 590/2304 [I 2022-10-27 23:09:54] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:15:54 [I 2022-10-27 23:09:54] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00242 [I 2022-10-27 23:09:54] ...trap/engines/engine.py.230: loss: 0.08158 [I 2022-10-27 23:09:54] ...optimizers/trijoint.py.151: total_norm: 10.128402 [I 2022-10-27 23:09:59] ...trap/engines/engine.py.224: train: epoch 93 | batch 600/2304 [I 2022-10-27 23:09:59] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:15:44 [I 2022-10-27 23:09:59] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00246 [I 2022-10-27 23:09:59] ...trap/engines/engine.py.230: loss: 0.15577 [I 2022-10-27 23:09:59] ...optimizers/trijoint.py.151: total_norm: 33.694630 [I 2022-10-27 23:10:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 610/2304 [I 2022-10-27 23:10:05] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:15:34 [I 2022-10-27 23:10:05] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00247 [I 2022-10-27 23:10:05] ...trap/engines/engine.py.230: loss: 0.15314 [I 2022-10-27 23:10:05] ...optimizers/trijoint.py.151: total_norm: 16.562281 [I 2022-10-27 23:10:10] ...trap/engines/engine.py.224: train: epoch 93 | batch 620/2304 [I 2022-10-27 23:10:10] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:15:31 [I 2022-10-27 23:10:10] ...trap/engines/engine.py.229: process: 0.54690 | load: 0.00246 [I 2022-10-27 23:10:10] ...trap/engines/engine.py.230: loss: 0.10170 [I 2022-10-27 23:10:10] ...optimizers/trijoint.py.151: total_norm: 12.845500 [I 2022-10-27 23:10:16] ...trap/engines/engine.py.224: train: epoch 93 | batch 630/2304 [I 2022-10-27 23:10:16] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:15:44 [I 2022-10-27 23:10:16] ...trap/engines/engine.py.229: process: 0.58696 | load: 0.00248 [I 2022-10-27 23:10:16] ...trap/engines/engine.py.230: loss: 0.09670 [I 2022-10-27 23:10:16] ...optimizers/trijoint.py.151: total_norm: 8.690539 [I 2022-10-27 23:10:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 640/2304 [I 2022-10-27 23:10:22] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:43 [I 2022-10-27 23:10:22] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00246 [I 2022-10-27 23:10:22] ...trap/engines/engine.py.230: loss: 0.11805 [I 2022-10-27 23:10:22] ...optimizers/trijoint.py.151: total_norm: 45.113461 [I 2022-10-27 23:10:27] ...trap/engines/engine.py.224: train: epoch 93 | batch 650/2304 [I 2022-10-27 23:10:27] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:15:17 [I 2022-10-27 23:10:27] ...trap/engines/engine.py.229: process: 0.54811 | load: 0.00244 [I 2022-10-27 23:10:27] ...trap/engines/engine.py.230: loss: 0.13539 [I 2022-10-27 23:10:27] ...optimizers/trijoint.py.151: total_norm: 25.203922 [I 2022-10-27 23:10:33] ...trap/engines/engine.py.224: train: epoch 93 | batch 660/2304 [I 2022-10-27 23:10:33] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:15:11 [I 2022-10-27 23:10:33] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00261 [I 2022-10-27 23:10:33] ...trap/engines/engine.py.230: loss: 0.19263 [I 2022-10-27 23:10:33] ...optimizers/trijoint.py.151: total_norm: 9.859673 [I 2022-10-27 23:10:38] ...trap/engines/engine.py.224: train: epoch 93 | batch 670/2304 [I 2022-10-27 23:10:38] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:15:06 [I 2022-10-27 23:10:38] ...trap/engines/engine.py.229: process: 0.55417 | load: 0.00242 [I 2022-10-27 23:10:38] ...trap/engines/engine.py.230: loss: 0.15345 [I 2022-10-27 23:10:38] ...optimizers/trijoint.py.151: total_norm: 14.205827 [I 2022-10-27 23:10:44] ...trap/engines/engine.py.224: train: epoch 93 | batch 680/2304 [I 2022-10-27 23:10:44] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:15:00 [I 2022-10-27 23:10:44] ...trap/engines/engine.py.229: process: 0.54987 | load: 0.00248 [I 2022-10-27 23:10:44] ...trap/engines/engine.py.230: loss: 0.16035 [I 2022-10-27 23:10:44] ...optimizers/trijoint.py.151: total_norm: 10.804374 [I 2022-10-27 23:10:50] ...trap/engines/engine.py.224: train: epoch 93 | batch 690/2304 [I 2022-10-27 23:10:50] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:15:01 [I 2022-10-27 23:10:50] ...trap/engines/engine.py.229: process: 0.55831 | load: 0.00244 [I 2022-10-27 23:10:50] ...trap/engines/engine.py.230: loss: 0.12538 [I 2022-10-27 23:10:50] ...optimizers/trijoint.py.151: total_norm: 24.045868 [I 2022-10-27 23:10:55] ...trap/engines/engine.py.224: train: epoch 93 | batch 700/2304 [I 2022-10-27 23:10:55] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:14:57 [I 2022-10-27 23:10:55] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00241 [I 2022-10-27 23:10:55] ...trap/engines/engine.py.230: loss: 0.12434 [I 2022-10-27 23:10:55] ...optimizers/trijoint.py.151: total_norm: 33.450905 [I 2022-10-27 23:11:01] ...trap/engines/engine.py.224: train: epoch 93 | batch 710/2304 [I 2022-10-27 23:11:01] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:14:46 [I 2022-10-27 23:11:01] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00244 [I 2022-10-27 23:11:01] ...trap/engines/engine.py.230: loss: 0.11597 [I 2022-10-27 23:11:01] ...optimizers/trijoint.py.151: total_norm: 9.731830 [I 2022-10-27 23:11:06] ...trap/engines/engine.py.224: train: epoch 93 | batch 720/2304 [I 2022-10-27 23:11:06] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:14:42 [I 2022-10-27 23:11:06] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00242 [I 2022-10-27 23:11:06] ...trap/engines/engine.py.230: loss: 0.19728 [I 2022-10-27 23:11:06] ...optimizers/trijoint.py.151: total_norm: 14.504899 [I 2022-10-27 23:11:12] ...trap/engines/engine.py.224: train: epoch 93 | batch 730/2304 [I 2022-10-27 23:11:12] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:14:30 [I 2022-10-27 23:11:12] ...trap/engines/engine.py.229: process: 0.54986 | load: 0.00241 [I 2022-10-27 23:11:12] ...trap/engines/engine.py.230: loss: 0.08572 [I 2022-10-27 23:11:12] ...optimizers/trijoint.py.151: total_norm: 20.260305 [I 2022-10-27 23:11:17] ...trap/engines/engine.py.224: train: epoch 93 | batch 740/2304 [I 2022-10-27 23:11:17] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:14:29 [I 2022-10-27 23:11:17] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00243 [I 2022-10-27 23:11:17] ...trap/engines/engine.py.230: loss: 0.16186 [I 2022-10-27 23:11:17] ...optimizers/trijoint.py.151: total_norm: 24.856258 [I 2022-10-27 23:11:23] ...trap/engines/engine.py.224: train: epoch 93 | batch 750/2304 [I 2022-10-27 23:11:23] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:14:29 [I 2022-10-27 23:11:23] ...trap/engines/engine.py.229: process: 0.53496 | load: 0.00251 [I 2022-10-27 23:11:23] ...trap/engines/engine.py.230: loss: 0.13725 [I 2022-10-27 23:11:23] ...optimizers/trijoint.py.151: total_norm: 12.480887 [I 2022-10-27 23:11:29] ...trap/engines/engine.py.224: train: epoch 93 | batch 760/2304 [I 2022-10-27 23:11:29] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:14:16 [I 2022-10-27 23:11:29] ...trap/engines/engine.py.229: process: 0.55436 | load: 0.00252 [I 2022-10-27 23:11:29] ...trap/engines/engine.py.230: loss: 0.09631 [I 2022-10-27 23:11:29] ...optimizers/trijoint.py.151: total_norm: 9.533916 [I 2022-10-27 23:11:34] ...trap/engines/engine.py.224: train: epoch 93 | batch 770/2304 [I 2022-10-27 23:11:34] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:14:16 [I 2022-10-27 23:11:34] ...trap/engines/engine.py.229: process: 0.56529 | load: 0.00250 [I 2022-10-27 23:11:34] ...trap/engines/engine.py.230: loss: 0.14578 [I 2022-10-27 23:11:34] ...optimizers/trijoint.py.151: total_norm: 10.027904 [I 2022-10-27 23:11:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 780/2304 [I 2022-10-27 23:11:40] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:14:03 [I 2022-10-27 23:11:40] ...trap/engines/engine.py.229: process: 0.55687 | load: 0.00241 [I 2022-10-27 23:11:40] ...trap/engines/engine.py.230: loss: 0.11386 [I 2022-10-27 23:11:40] ...optimizers/trijoint.py.151: total_norm: 11.086463 [I 2022-10-27 23:11:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 790/2304 [I 2022-10-27 23:11:45] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:14:03 [I 2022-10-27 23:11:45] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00249 [I 2022-10-27 23:11:45] ...trap/engines/engine.py.230: loss: 0.15031 [I 2022-10-27 23:11:45] ...optimizers/trijoint.py.151: total_norm: 10.365455 [I 2022-10-27 23:11:51] ...trap/engines/engine.py.224: train: epoch 93 | batch 800/2304 [I 2022-10-27 23:11:51] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:13:58 [I 2022-10-27 23:11:51] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00242 [I 2022-10-27 23:11:51] ...trap/engines/engine.py.230: loss: 0.10118 [I 2022-10-27 23:11:51] ...optimizers/trijoint.py.151: total_norm: 10.601995 [I 2022-10-27 23:11:57] ...trap/engines/engine.py.224: train: epoch 93 | batch 810/2304 [I 2022-10-27 23:11:57] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:13:53 [I 2022-10-27 23:11:57] ...trap/engines/engine.py.229: process: 0.56086 | load: 0.00244 [I 2022-10-27 23:11:57] ...trap/engines/engine.py.230: loss: 0.14772 [I 2022-10-27 23:11:57] ...optimizers/trijoint.py.151: total_norm: 21.299782 [I 2022-10-27 23:12:02] ...trap/engines/engine.py.224: train: epoch 93 | batch 820/2304 [I 2022-10-27 23:12:02] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:13:41 [I 2022-10-27 23:12:02] ...trap/engines/engine.py.229: process: 0.54783 | load: 0.00244 [I 2022-10-27 23:12:02] ...trap/engines/engine.py.230: loss: 0.15008 [I 2022-10-27 23:12:02] ...optimizers/trijoint.py.151: total_norm: 18.682882 [I 2022-10-27 23:12:08] ...trap/engines/engine.py.224: train: epoch 93 | batch 830/2304 [I 2022-10-27 23:12:08] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:13:44 [I 2022-10-27 23:12:08] ...trap/engines/engine.py.229: process: 0.56261 | load: 0.00249 [I 2022-10-27 23:12:08] ...trap/engines/engine.py.230: loss: 0.15417 [I 2022-10-27 23:12:08] ...optimizers/trijoint.py.151: total_norm: 14.802446 [I 2022-10-27 23:12:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 840/2304 [I 2022-10-27 23:12:13] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:36 [I 2022-10-27 23:12:13] ...trap/engines/engine.py.229: process: 0.55150 | load: 0.00250 [I 2022-10-27 23:12:13] ...trap/engines/engine.py.230: loss: 0.14480 [I 2022-10-27 23:12:13] ...optimizers/trijoint.py.151: total_norm: 8.903964 [I 2022-10-27 23:12:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 850/2304 [I 2022-10-27 23:12:19] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:13:24 [I 2022-10-27 23:12:19] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00252 [I 2022-10-27 23:12:19] ...trap/engines/engine.py.230: loss: 0.09794 [I 2022-10-27 23:12:19] ...optimizers/trijoint.py.151: total_norm: 13.071762 [I 2022-10-27 23:12:25] ...trap/engines/engine.py.224: train: epoch 93 | batch 860/2304 [I 2022-10-27 23:12:25] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:13:22 [I 2022-10-27 23:12:25] ...trap/engines/engine.py.229: process: 0.55834 | load: 0.00250 [I 2022-10-27 23:12:25] ...trap/engines/engine.py.230: loss: 0.14609 [I 2022-10-27 23:12:25] ...optimizers/trijoint.py.151: total_norm: 16.823242 [I 2022-10-27 23:12:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 870/2304 [I 2022-10-27 23:12:30] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:19 [I 2022-10-27 23:12:30] ...trap/engines/engine.py.229: process: 0.55477 | load: 0.00250 [I 2022-10-27 23:12:30] ...trap/engines/engine.py.230: loss: 0.13567 [I 2022-10-27 23:12:30] ...optimizers/trijoint.py.151: total_norm: 19.195316 [I 2022-10-27 23:12:36] ...trap/engines/engine.py.224: train: epoch 93 | batch 880/2304 [I 2022-10-27 23:12:36] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:13:10 [I 2022-10-27 23:12:36] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00258 [I 2022-10-27 23:12:36] ...trap/engines/engine.py.230: loss: 0.15856 [I 2022-10-27 23:12:36] ...optimizers/trijoint.py.151: total_norm: 24.957312 [I 2022-10-27 23:12:41] ...trap/engines/engine.py.224: train: epoch 93 | batch 890/2304 [I 2022-10-27 23:12:41] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:13:06 [I 2022-10-27 23:12:41] ...trap/engines/engine.py.229: process: 0.55361 | load: 0.00250 [I 2022-10-27 23:12:41] ...trap/engines/engine.py.230: loss: 0.13866 [I 2022-10-27 23:12:41] ...optimizers/trijoint.py.151: total_norm: 10.588310 [I 2022-10-27 23:12:47] ...trap/engines/engine.py.224: train: epoch 93 | batch 900/2304 [I 2022-10-27 23:12:47] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:12:58 [I 2022-10-27 23:12:47] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00266 [I 2022-10-27 23:12:47] ...trap/engines/engine.py.230: loss: 0.11813 [I 2022-10-27 23:12:47] ...optimizers/trijoint.py.151: total_norm: 18.968637 [I 2022-10-27 23:12:52] ...trap/engines/engine.py.224: train: epoch 93 | batch 910/2304 [I 2022-10-27 23:12:52] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:12:53 [I 2022-10-27 23:12:52] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00290 [I 2022-10-27 23:12:52] ...trap/engines/engine.py.230: loss: 0.13386 [I 2022-10-27 23:12:52] ...optimizers/trijoint.py.151: total_norm: 15.645054 [I 2022-10-27 23:12:58] ...trap/engines/engine.py.224: train: epoch 93 | batch 920/2304 [I 2022-10-27 23:12:58] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:12:52 [I 2022-10-27 23:12:58] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00251 [I 2022-10-27 23:12:58] ...trap/engines/engine.py.230: loss: 0.19601 [I 2022-10-27 23:12:58] ...optimizers/trijoint.py.151: total_norm: 15.276439 [I 2022-10-27 23:13:04] ...trap/engines/engine.py.224: train: epoch 93 | batch 930/2304 [I 2022-10-27 23:13:04] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:41 [I 2022-10-27 23:13:04] ...trap/engines/engine.py.229: process: 0.55166 | load: 0.00248 [I 2022-10-27 23:13:04] ...trap/engines/engine.py.230: loss: 0.15428 [I 2022-10-27 23:13:04] ...optimizers/trijoint.py.151: total_norm: 11.274903 [I 2022-10-27 23:13:09] ...trap/engines/engine.py.224: train: epoch 93 | batch 940/2304 [I 2022-10-27 23:13:09] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:42 [I 2022-10-27 23:13:09] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00243 [I 2022-10-27 23:13:09] ...trap/engines/engine.py.230: loss: 0.11566 [I 2022-10-27 23:13:09] ...optimizers/trijoint.py.151: total_norm: 10.428195 [I 2022-10-27 23:13:15] ...trap/engines/engine.py.224: train: epoch 93 | batch 950/2304 [I 2022-10-27 23:13:15] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:33 [I 2022-10-27 23:13:15] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00250 [I 2022-10-27 23:13:15] ...trap/engines/engine.py.230: loss: 0.09460 [I 2022-10-27 23:13:15] ...optimizers/trijoint.py.151: total_norm: 13.252850 [I 2022-10-27 23:13:20] ...trap/engines/engine.py.224: train: epoch 93 | batch 960/2304 [I 2022-10-27 23:13:20] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:27 [I 2022-10-27 23:13:20] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00253 [I 2022-10-27 23:13:20] ...trap/engines/engine.py.230: loss: 0.11082 [I 2022-10-27 23:13:20] ...optimizers/trijoint.py.151: total_norm: 11.818038 [I 2022-10-27 23:13:26] ...trap/engines/engine.py.224: train: epoch 93 | batch 970/2304 [I 2022-10-27 23:13:26] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:26 [I 2022-10-27 23:13:26] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00312 [I 2022-10-27 23:13:26] ...trap/engines/engine.py.230: loss: 0.16676 [I 2022-10-27 23:13:26] ...optimizers/trijoint.py.151: total_norm: 12.780914 [I 2022-10-27 23:13:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 980/2304 [I 2022-10-27 23:13:31] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:12:13 [I 2022-10-27 23:13:31] ...trap/engines/engine.py.229: process: 0.55036 | load: 0.00242 [I 2022-10-27 23:13:31] ...trap/engines/engine.py.230: loss: 0.13207 [I 2022-10-27 23:13:31] ...optimizers/trijoint.py.151: total_norm: 16.738668 [I 2022-10-27 23:13:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 990/2304 [I 2022-10-27 23:13:37] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:12:11 [I 2022-10-27 23:13:37] ...trap/engines/engine.py.229: process: 0.56916 | load: 0.00240 [I 2022-10-27 23:13:37] ...trap/engines/engine.py.230: loss: 0.17119 [I 2022-10-27 23:13:37] ...optimizers/trijoint.py.151: total_norm: 27.239439 [I 2022-10-27 23:13:43] ...trap/engines/engine.py.224: train: epoch 93 | batch 1000/2304 [I 2022-10-27 23:13:43] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:12:03 [I 2022-10-27 23:13:43] ...trap/engines/engine.py.229: process: 0.55319 | load: 0.00241 [I 2022-10-27 23:13:43] ...trap/engines/engine.py.230: loss: 0.14858 [I 2022-10-27 23:13:43] ...optimizers/trijoint.py.151: total_norm: 23.458813 [I 2022-10-27 23:13:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 1010/2304 [I 2022-10-27 23:13:48] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:12:01 [I 2022-10-27 23:13:48] ...trap/engines/engine.py.229: process: 0.56192 | load: 0.00243 [I 2022-10-27 23:13:48] ...trap/engines/engine.py.230: loss: 0.08738 [I 2022-10-27 23:13:48] ...optimizers/trijoint.py.151: total_norm: 9.852073 [I 2022-10-27 23:13:54] ...trap/engines/engine.py.224: train: epoch 93 | batch 1020/2304 [I 2022-10-27 23:13:54] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:53 [I 2022-10-27 23:13:54] ...trap/engines/engine.py.229: process: 0.54938 | load: 0.00242 [I 2022-10-27 23:13:54] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-27 23:13:54] ...optimizers/trijoint.py.151: total_norm: 9.261868 [I 2022-10-27 23:13:59] ...trap/engines/engine.py.224: train: epoch 93 | batch 1030/2304 [I 2022-10-27 23:13:59] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:45 [I 2022-10-27 23:13:59] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00241 [I 2022-10-27 23:13:59] ...trap/engines/engine.py.230: loss: 0.13281 [I 2022-10-27 23:13:59] ...optimizers/trijoint.py.151: total_norm: 9.379080 [I 2022-10-27 23:14:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 1040/2304 [I 2022-10-27 23:14:05] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:35 [I 2022-10-27 23:14:05] ...trap/engines/engine.py.229: process: 0.54604 | load: 0.00241 [I 2022-10-27 23:14:05] ...trap/engines/engine.py.230: loss: 0.10684 [I 2022-10-27 23:14:05] ...optimizers/trijoint.py.151: total_norm: 17.041569 [I 2022-10-27 23:14:10] ...trap/engines/engine.py.224: train: epoch 93 | batch 1050/2304 [I 2022-10-27 23:14:10] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:42 [I 2022-10-27 23:14:10] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00243 [I 2022-10-27 23:14:10] ...trap/engines/engine.py.230: loss: 0.15829 [I 2022-10-27 23:14:10] ...optimizers/trijoint.py.151: total_norm: 11.784844 [I 2022-10-27 23:14:16] ...trap/engines/engine.py.224: train: epoch 93 | batch 1060/2304 [I 2022-10-27 23:14:16] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:31 [I 2022-10-27 23:14:16] ...trap/engines/engine.py.229: process: 0.57123 | load: 0.00261 [I 2022-10-27 23:14:16] ...trap/engines/engine.py.230: loss: 0.18484 [I 2022-10-27 23:14:16] ...optimizers/trijoint.py.151: total_norm: 24.314135 [I 2022-10-27 23:14:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 1070/2304 [I 2022-10-27 23:14:22] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:25 [I 2022-10-27 23:14:22] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00260 [I 2022-10-27 23:14:22] ...trap/engines/engine.py.230: loss: 0.11548 [I 2022-10-27 23:14:22] ...optimizers/trijoint.py.151: total_norm: 13.309046 [I 2022-10-27 23:14:27] ...trap/engines/engine.py.224: train: epoch 93 | batch 1080/2304 [I 2022-10-27 23:14:27] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:19 [I 2022-10-27 23:14:27] ...trap/engines/engine.py.229: process: 0.56645 | load: 0.00243 [I 2022-10-27 23:14:27] ...trap/engines/engine.py.230: loss: 0.11673 [I 2022-10-27 23:14:27] ...optimizers/trijoint.py.151: total_norm: 14.701885 [I 2022-10-27 23:14:33] ...trap/engines/engine.py.224: train: epoch 93 | batch 1090/2304 [I 2022-10-27 23:14:33] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:08 [I 2022-10-27 23:14:33] ...trap/engines/engine.py.229: process: 0.55112 | load: 0.00250 [I 2022-10-27 23:14:33] ...trap/engines/engine.py.230: loss: 0.10740 [I 2022-10-27 23:14:33] ...optimizers/trijoint.py.151: total_norm: 14.745454 [I 2022-10-27 23:14:38] ...trap/engines/engine.py.224: train: epoch 93 | batch 1100/2304 [I 2022-10-27 23:14:38] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:11:09 [I 2022-10-27 23:14:38] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00245 [I 2022-10-27 23:14:38] ...trap/engines/engine.py.230: loss: 0.13101 [I 2022-10-27 23:14:38] ...optimizers/trijoint.py.151: total_norm: 39.088440 [I 2022-10-27 23:14:44] ...trap/engines/engine.py.224: train: epoch 93 | batch 1110/2304 [I 2022-10-27 23:14:44] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:58 [I 2022-10-27 23:14:44] ...trap/engines/engine.py.229: process: 0.54859 | load: 0.00250 [I 2022-10-27 23:14:44] ...trap/engines/engine.py.230: loss: 0.09716 [I 2022-10-27 23:14:44] ...optimizers/trijoint.py.151: total_norm: 15.859950 [I 2022-10-27 23:14:49] ...trap/engines/engine.py.224: train: epoch 93 | batch 1120/2304 [I 2022-10-27 23:14:49] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:59 [I 2022-10-27 23:14:49] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00254 [I 2022-10-27 23:14:49] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-27 23:14:49] ...optimizers/trijoint.py.151: total_norm: 25.370743 [I 2022-10-27 23:14:55] ...trap/engines/engine.py.224: train: epoch 93 | batch 1130/2304 [I 2022-10-27 23:14:55] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:54 [I 2022-10-27 23:14:55] ...trap/engines/engine.py.229: process: 0.55485 | load: 0.00251 [I 2022-10-27 23:14:55] ...trap/engines/engine.py.230: loss: 0.08457 [I 2022-10-27 23:14:55] ...optimizers/trijoint.py.151: total_norm: 13.872030 [I 2022-10-27 23:15:00] ...trap/engines/engine.py.224: train: epoch 93 | batch 1140/2304 [I 2022-10-27 23:15:00] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:46 [I 2022-10-27 23:15:00] ...trap/engines/engine.py.229: process: 0.54556 | load: 0.00275 [I 2022-10-27 23:15:00] ...trap/engines/engine.py.230: loss: 0.13528 [I 2022-10-27 23:15:00] ...optimizers/trijoint.py.151: total_norm: 23.262272 [I 2022-10-27 23:15:06] ...trap/engines/engine.py.224: train: epoch 93 | batch 1150/2304 [I 2022-10-27 23:15:06] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:48 [I 2022-10-27 23:15:06] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00255 [I 2022-10-27 23:15:06] ...trap/engines/engine.py.230: loss: 0.12581 [I 2022-10-27 23:15:06] ...optimizers/trijoint.py.151: total_norm: 24.025059 [I 2022-10-27 23:15:12] ...trap/engines/engine.py.224: train: epoch 93 | batch 1160/2304 [I 2022-10-27 23:15:12] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:37 [I 2022-10-27 23:15:12] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00253 [I 2022-10-27 23:15:12] ...trap/engines/engine.py.230: loss: 0.12720 [I 2022-10-27 23:15:12] ...optimizers/trijoint.py.151: total_norm: 14.625526 [I 2022-10-27 23:15:17] ...trap/engines/engine.py.224: train: epoch 93 | batch 1170/2304 [I 2022-10-27 23:15:17] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:29 [I 2022-10-27 23:15:17] ...trap/engines/engine.py.229: process: 0.55137 | load: 0.00249 [I 2022-10-27 23:15:17] ...trap/engines/engine.py.230: loss: 0.11489 [I 2022-10-27 23:15:17] ...optimizers/trijoint.py.151: total_norm: 16.203041 [I 2022-10-27 23:15:23] ...trap/engines/engine.py.224: train: epoch 93 | batch 1180/2304 [I 2022-10-27 23:15:23] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:23 [I 2022-10-27 23:15:23] ...trap/engines/engine.py.229: process: 0.55752 | load: 0.00251 [I 2022-10-27 23:15:23] ...trap/engines/engine.py.230: loss: 0.08502 [I 2022-10-27 23:15:23] ...optimizers/trijoint.py.151: total_norm: 11.735449 [I 2022-10-27 23:15:28] ...trap/engines/engine.py.224: train: epoch 93 | batch 1190/2304 [I 2022-10-27 23:15:28] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:21 [I 2022-10-27 23:15:28] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00247 [I 2022-10-27 23:15:28] ...trap/engines/engine.py.230: loss: 0.09820 [I 2022-10-27 23:15:28] ...optimizers/trijoint.py.151: total_norm: 25.557989 [I 2022-10-27 23:15:34] ...trap/engines/engine.py.224: train: epoch 93 | batch 1200/2304 [I 2022-10-27 23:15:34] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:10:14 [I 2022-10-27 23:15:34] ...trap/engines/engine.py.229: process: 0.55098 | load: 0.00247 [I 2022-10-27 23:15:34] ...trap/engines/engine.py.230: loss: 0.10425 [I 2022-10-27 23:15:34] ...optimizers/trijoint.py.151: total_norm: 16.058842 [I 2022-10-27 23:15:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 1210/2304 [I 2022-10-27 23:15:40] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:10:07 [I 2022-10-27 23:15:40] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00247 [I 2022-10-27 23:15:40] ...trap/engines/engine.py.230: loss: 0.10345 [I 2022-10-27 23:15:40] ...optimizers/trijoint.py.151: total_norm: 12.411896 [I 2022-10-27 23:15:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 1220/2304 [I 2022-10-27 23:15:45] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:10:03 [I 2022-10-27 23:15:45] ...trap/engines/engine.py.229: process: 0.56454 | load: 0.00249 [I 2022-10-27 23:15:45] ...trap/engines/engine.py.230: loss: 0.13362 [I 2022-10-27 23:15:45] ...optimizers/trijoint.py.151: total_norm: 21.073006 [I 2022-10-27 23:15:51] ...trap/engines/engine.py.224: train: epoch 93 | batch 1230/2304 [I 2022-10-27 23:15:51] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:09:59 [I 2022-10-27 23:15:51] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00246 [I 2022-10-27 23:15:51] ...trap/engines/engine.py.230: loss: 0.14641 [I 2022-10-27 23:15:51] ...optimizers/trijoint.py.151: total_norm: 16.006714 [I 2022-10-27 23:15:56] ...trap/engines/engine.py.224: train: epoch 93 | batch 1240/2304 [I 2022-10-27 23:15:56] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:49 [I 2022-10-27 23:15:56] ...trap/engines/engine.py.229: process: 0.54589 | load: 0.00255 [I 2022-10-27 23:15:56] ...trap/engines/engine.py.230: loss: 0.10206 [I 2022-10-27 23:15:56] ...optimizers/trijoint.py.151: total_norm: 14.060072 [I 2022-10-27 23:16:02] ...trap/engines/engine.py.224: train: epoch 93 | batch 1250/2304 [I 2022-10-27 23:16:02] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:09:47 [I 2022-10-27 23:16:02] ...trap/engines/engine.py.229: process: 0.56430 | load: 0.00251 [I 2022-10-27 23:16:02] ...trap/engines/engine.py.230: loss: 0.11428 [I 2022-10-27 23:16:02] ...optimizers/trijoint.py.151: total_norm: 24.863310 [I 2022-10-27 23:16:07] ...trap/engines/engine.py.224: train: epoch 93 | batch 1260/2304 [I 2022-10-27 23:16:07] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:40 [I 2022-10-27 23:16:07] ...trap/engines/engine.py.229: process: 0.55111 | load: 0.00247 [I 2022-10-27 23:16:07] ...trap/engines/engine.py.230: loss: 0.10425 [I 2022-10-27 23:16:07] ...optimizers/trijoint.py.151: total_norm: 14.760634 [I 2022-10-27 23:16:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 1270/2304 [I 2022-10-27 23:16:13] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:35 [I 2022-10-27 23:16:13] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00256 [I 2022-10-27 23:16:13] ...trap/engines/engine.py.230: loss: 0.13410 [I 2022-10-27 23:16:13] ...optimizers/trijoint.py.151: total_norm: 19.702190 [I 2022-10-27 23:16:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 1280/2304 [I 2022-10-27 23:16:19] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:29 [I 2022-10-27 23:16:19] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00244 [I 2022-10-27 23:16:19] ...trap/engines/engine.py.230: loss: 0.16311 [I 2022-10-27 23:16:19] ...optimizers/trijoint.py.151: total_norm: 29.082499 [I 2022-10-27 23:16:24] ...trap/engines/engine.py.224: train: epoch 93 | batch 1290/2304 [I 2022-10-27 23:16:24] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:25 [I 2022-10-27 23:16:24] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00258 [I 2022-10-27 23:16:24] ...trap/engines/engine.py.230: loss: 0.10250 [I 2022-10-27 23:16:24] ...optimizers/trijoint.py.151: total_norm: 32.411533 [I 2022-10-27 23:16:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 1300/2304 [I 2022-10-27 23:16:30] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:09:15 [I 2022-10-27 23:16:30] ...trap/engines/engine.py.229: process: 0.54607 | load: 0.00247 [I 2022-10-27 23:16:30] ...trap/engines/engine.py.230: loss: 0.18080 [I 2022-10-27 23:16:30] ...optimizers/trijoint.py.151: total_norm: 21.960209 [I 2022-10-27 23:16:35] ...trap/engines/engine.py.224: train: epoch 93 | batch 1310/2304 [I 2022-10-27 23:16:35] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:09:11 [I 2022-10-27 23:16:35] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00253 [I 2022-10-27 23:16:35] ...trap/engines/engine.py.230: loss: 0.11064 [I 2022-10-27 23:16:35] ...optimizers/trijoint.py.151: total_norm: 15.338304 [I 2022-10-27 23:16:41] ...trap/engines/engine.py.224: train: epoch 93 | batch 1320/2304 [I 2022-10-27 23:16:41] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:09:11 [I 2022-10-27 23:16:41] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00258 [I 2022-10-27 23:16:41] ...trap/engines/engine.py.230: loss: 0.13505 [I 2022-10-27 23:16:41] ...optimizers/trijoint.py.151: total_norm: 19.925838 [I 2022-10-27 23:16:46] ...trap/engines/engine.py.224: train: epoch 93 | batch 1330/2304 [I 2022-10-27 23:16:46] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:59 [I 2022-10-27 23:16:46] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00253 [I 2022-10-27 23:16:46] ...trap/engines/engine.py.230: loss: 0.11245 [I 2022-10-27 23:16:46] ...optimizers/trijoint.py.151: total_norm: 14.250531 [I 2022-10-27 23:16:52] ...trap/engines/engine.py.224: train: epoch 93 | batch 1340/2304 [I 2022-10-27 23:16:52] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:09:10 [I 2022-10-27 23:16:52] ...trap/engines/engine.py.229: process: 0.56380 | load: 0.00247 [I 2022-10-27 23:16:52] ...trap/engines/engine.py.230: loss: 0.13457 [I 2022-10-27 23:16:52] ...optimizers/trijoint.py.151: total_norm: 14.594549 [I 2022-10-27 23:16:58] ...trap/engines/engine.py.224: train: epoch 93 | batch 1350/2304 [I 2022-10-27 23:16:58] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:50 [I 2022-10-27 23:16:58] ...trap/engines/engine.py.229: process: 0.56972 | load: 0.00249 [I 2022-10-27 23:16:58] ...trap/engines/engine.py.230: loss: 0.11515 [I 2022-10-27 23:16:58] ...optimizers/trijoint.py.151: total_norm: 11.664763 [I 2022-10-27 23:17:03] ...trap/engines/engine.py.224: train: epoch 93 | batch 1360/2304 [I 2022-10-27 23:17:03] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:46 [I 2022-10-27 23:17:03] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00242 [I 2022-10-27 23:17:03] ...trap/engines/engine.py.230: loss: 0.10831 [I 2022-10-27 23:17:03] ...optimizers/trijoint.py.151: total_norm: 17.027147 [I 2022-10-27 23:17:09] ...trap/engines/engine.py.224: train: epoch 93 | batch 1370/2304 [I 2022-10-27 23:17:09] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:08:38 [I 2022-10-27 23:17:09] ...trap/engines/engine.py.229: process: 0.56767 | load: 0.00243 [I 2022-10-27 23:17:09] ...trap/engines/engine.py.230: loss: 0.16356 [I 2022-10-27 23:17:09] ...optimizers/trijoint.py.151: total_norm: 18.106979 [I 2022-10-27 23:17:14] ...trap/engines/engine.py.224: train: epoch 93 | batch 1380/2304 [I 2022-10-27 23:17:14] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:33 [I 2022-10-27 23:17:14] ...trap/engines/engine.py.229: process: 0.54842 | load: 0.00243 [I 2022-10-27 23:17:14] ...trap/engines/engine.py.230: loss: 0.12923 [I 2022-10-27 23:17:14] ...optimizers/trijoint.py.151: total_norm: 12.488498 [I 2022-10-27 23:17:20] ...trap/engines/engine.py.224: train: epoch 93 | batch 1390/2304 [I 2022-10-27 23:17:20] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:26 [I 2022-10-27 23:17:20] ...trap/engines/engine.py.229: process: 0.55851 | load: 0.00242 [I 2022-10-27 23:17:20] ...trap/engines/engine.py.230: loss: 0.21078 [I 2022-10-27 23:17:20] ...optimizers/trijoint.py.151: total_norm: 21.780554 [I 2022-10-27 23:17:26] ...trap/engines/engine.py.224: train: epoch 93 | batch 1400/2304 [I 2022-10-27 23:17:26] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:21 [I 2022-10-27 23:17:26] ...trap/engines/engine.py.229: process: 0.55102 | load: 0.00245 [I 2022-10-27 23:17:26] ...trap/engines/engine.py.230: loss: 0.11394 [I 2022-10-27 23:17:26] ...optimizers/trijoint.py.151: total_norm: 11.717187 [I 2022-10-27 23:17:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 1410/2304 [I 2022-10-27 23:17:31] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:08:15 [I 2022-10-27 23:17:31] ...trap/engines/engine.py.229: process: 0.54967 | load: 0.00249 [I 2022-10-27 23:17:31] ...trap/engines/engine.py.230: loss: 0.10367 [I 2022-10-27 23:17:31] ...optimizers/trijoint.py.151: total_norm: 13.470509 [I 2022-10-27 23:17:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 1420/2304 [I 2022-10-27 23:17:37] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:08:12 [I 2022-10-27 23:17:37] ...trap/engines/engine.py.229: process: 0.56412 | load: 0.00244 [I 2022-10-27 23:17:37] ...trap/engines/engine.py.230: loss: 0.09586 [I 2022-10-27 23:17:37] ...optimizers/trijoint.py.151: total_norm: 24.124060 [I 2022-10-27 23:17:42] ...trap/engines/engine.py.224: train: epoch 93 | batch 1430/2304 [I 2022-10-27 23:17:42] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:08:08 [I 2022-10-27 23:17:42] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00241 [I 2022-10-27 23:17:42] ...trap/engines/engine.py.230: loss: 0.11068 [I 2022-10-27 23:17:42] ...optimizers/trijoint.py.151: total_norm: 9.377945 [I 2022-10-27 23:17:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 1440/2304 [I 2022-10-27 23:17:48] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:57 [I 2022-10-27 23:17:48] ...trap/engines/engine.py.229: process: 0.55284 | load: 0.00254 [I 2022-10-27 23:17:48] ...trap/engines/engine.py.230: loss: 0.10369 [I 2022-10-27 23:17:48] ...optimizers/trijoint.py.151: total_norm: 11.924972 [I 2022-10-27 23:17:53] ...trap/engines/engine.py.224: train: epoch 93 | batch 1450/2304 [I 2022-10-27 23:17:53] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:53 [I 2022-10-27 23:17:53] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00245 [I 2022-10-27 23:17:53] ...trap/engines/engine.py.230: loss: 0.11056 [I 2022-10-27 23:17:53] ...optimizers/trijoint.py.151: total_norm: 13.722006 [I 2022-10-27 23:17:59] ...trap/engines/engine.py.224: train: epoch 93 | batch 1460/2304 [I 2022-10-27 23:17:59] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:46 [I 2022-10-27 23:17:59] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00244 [I 2022-10-27 23:17:59] ...trap/engines/engine.py.230: loss: 0.10551 [I 2022-10-27 23:17:59] ...optimizers/trijoint.py.151: total_norm: 21.813250 [I 2022-10-27 23:18:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 1470/2304 [I 2022-10-27 23:18:05] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:07:55 [I 2022-10-27 23:18:05] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00257 [I 2022-10-27 23:18:05] ...trap/engines/engine.py.230: loss: 0.13043 [I 2022-10-27 23:18:05] ...optimizers/trijoint.py.151: total_norm: 21.162590 [I 2022-10-27 23:18:10] ...trap/engines/engine.py.224: train: epoch 93 | batch 1480/2304 [I 2022-10-27 23:18:10] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:40 [I 2022-10-27 23:18:10] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00253 [I 2022-10-27 23:18:10] ...trap/engines/engine.py.230: loss: 0.21151 [I 2022-10-27 23:18:10] ...optimizers/trijoint.py.151: total_norm: 17.786407 [I 2022-10-27 23:18:16] ...trap/engines/engine.py.224: train: epoch 93 | batch 1490/2304 [I 2022-10-27 23:18:16] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:07:30 [I 2022-10-27 23:18:16] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00253 [I 2022-10-27 23:18:16] ...trap/engines/engine.py.230: loss: 0.26381 [I 2022-10-27 23:18:16] ...optimizers/trijoint.py.151: total_norm: 60.925560 [I 2022-10-27 23:18:21] ...trap/engines/engine.py.224: train: epoch 93 | batch 1500/2304 [I 2022-10-27 23:18:21] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:26 [I 2022-10-27 23:18:21] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00254 [I 2022-10-27 23:18:21] ...trap/engines/engine.py.230: loss: 0.13868 [I 2022-10-27 23:18:21] ...optimizers/trijoint.py.151: total_norm: 13.763633 [I 2022-10-27 23:18:27] ...trap/engines/engine.py.224: train: epoch 93 | batch 1510/2304 [I 2022-10-27 23:18:27] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:21 [I 2022-10-27 23:18:27] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00255 [I 2022-10-27 23:18:27] ...trap/engines/engine.py.230: loss: 0.13637 [I 2022-10-27 23:18:27] ...optimizers/trijoint.py.151: total_norm: 27.579336 [I 2022-10-27 23:18:33] ...trap/engines/engine.py.224: train: epoch 93 | batch 1520/2304 [I 2022-10-27 23:18:33] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:07:15 [I 2022-10-27 23:18:33] ...trap/engines/engine.py.229: process: 0.56672 | load: 0.00252 [I 2022-10-27 23:18:33] ...trap/engines/engine.py.230: loss: 0.17539 [I 2022-10-27 23:18:33] ...optimizers/trijoint.py.151: total_norm: 10.993283 [I 2022-10-27 23:18:38] ...trap/engines/engine.py.224: train: epoch 93 | batch 1530/2304 [I 2022-10-27 23:18:38] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:07:11 [I 2022-10-27 23:18:38] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00251 [I 2022-10-27 23:18:38] ...trap/engines/engine.py.230: loss: 0.15589 [I 2022-10-27 23:18:38] ...optimizers/trijoint.py.151: total_norm: 18.754162 [I 2022-10-27 23:18:44] ...trap/engines/engine.py.224: train: epoch 93 | batch 1540/2304 [I 2022-10-27 23:18:44] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:07:03 [I 2022-10-27 23:18:44] ...trap/engines/engine.py.229: process: 0.55309 | load: 0.00252 [I 2022-10-27 23:18:44] ...trap/engines/engine.py.230: loss: 0.14590 [I 2022-10-27 23:18:44] ...optimizers/trijoint.py.151: total_norm: 25.083111 [I 2022-10-27 23:18:49] ...trap/engines/engine.py.224: train: epoch 93 | batch 1550/2304 [I 2022-10-27 23:18:49] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:07:00 [I 2022-10-27 23:18:49] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00245 [I 2022-10-27 23:18:49] ...trap/engines/engine.py.230: loss: 0.12541 [I 2022-10-27 23:18:49] ...optimizers/trijoint.py.151: total_norm: 18.077660 [I 2022-10-27 23:18:55] ...trap/engines/engine.py.224: train: epoch 93 | batch 1560/2304 [I 2022-10-27 23:18:55] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:54 [I 2022-10-27 23:18:55] ...trap/engines/engine.py.229: process: 0.55787 | load: 0.00252 [I 2022-10-27 23:18:55] ...trap/engines/engine.py.230: loss: 0.14582 [I 2022-10-27 23:18:55] ...optimizers/trijoint.py.151: total_norm: 12.436561 [I 2022-10-27 23:19:01] ...trap/engines/engine.py.224: train: epoch 93 | batch 1570/2304 [I 2022-10-27 23:19:01] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:55 [I 2022-10-27 23:19:01] ...trap/engines/engine.py.229: process: 0.55155 | load: 0.00247 [I 2022-10-27 23:19:01] ...trap/engines/engine.py.230: loss: 0.09220 [I 2022-10-27 23:19:01] ...optimizers/trijoint.py.151: total_norm: 16.894730 [I 2022-10-27 23:19:06] ...trap/engines/engine.py.224: train: epoch 93 | batch 1580/2304 [I 2022-10-27 23:19:06] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:44 [I 2022-10-27 23:19:06] ...trap/engines/engine.py.229: process: 0.54914 | load: 0.00243 [I 2022-10-27 23:19:06] ...trap/engines/engine.py.230: loss: 0.10748 [I 2022-10-27 23:19:06] ...optimizers/trijoint.py.151: total_norm: 27.109266 [I 2022-10-27 23:19:12] ...trap/engines/engine.py.224: train: epoch 93 | batch 1590/2304 [I 2022-10-27 23:19:12] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:36 [I 2022-10-27 23:19:12] ...trap/engines/engine.py.229: process: 0.54839 | load: 0.00257 [I 2022-10-27 23:19:12] ...trap/engines/engine.py.230: loss: 0.11147 [I 2022-10-27 23:19:12] ...optimizers/trijoint.py.151: total_norm: 9.553619 [I 2022-10-27 23:19:17] ...trap/engines/engine.py.224: train: epoch 93 | batch 1600/2304 [I 2022-10-27 23:19:17] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:31 [I 2022-10-27 23:19:17] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00268 [I 2022-10-27 23:19:17] ...trap/engines/engine.py.230: loss: 0.09081 [I 2022-10-27 23:19:17] ...optimizers/trijoint.py.151: total_norm: 9.575617 [I 2022-10-27 23:19:23] ...trap/engines/engine.py.224: train: epoch 93 | batch 1610/2304 [I 2022-10-27 23:19:23] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:22 [I 2022-10-27 23:19:23] ...trap/engines/engine.py.229: process: 0.54568 | load: 0.00258 [I 2022-10-27 23:19:23] ...trap/engines/engine.py.230: loss: 0.11350 [I 2022-10-27 23:19:23] ...optimizers/trijoint.py.151: total_norm: 7.301857 [I 2022-10-27 23:19:28] ...trap/engines/engine.py.224: train: epoch 93 | batch 1620/2304 [I 2022-10-27 23:19:28] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:20 [I 2022-10-27 23:19:28] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00250 [I 2022-10-27 23:19:28] ...trap/engines/engine.py.230: loss: 0.11585 [I 2022-10-27 23:19:28] ...optimizers/trijoint.py.151: total_norm: 24.489027 [I 2022-10-27 23:19:34] ...trap/engines/engine.py.224: train: epoch 93 | batch 1630/2304 [I 2022-10-27 23:19:34] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:06:15 [I 2022-10-27 23:19:34] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00249 [I 2022-10-27 23:19:34] ...trap/engines/engine.py.230: loss: 0.10100 [I 2022-10-27 23:19:34] ...optimizers/trijoint.py.151: total_norm: 14.351624 [I 2022-10-27 23:19:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 1640/2304 [I 2022-10-27 23:19:40] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:06:11 [I 2022-10-27 23:19:40] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00256 [I 2022-10-27 23:19:40] ...trap/engines/engine.py.230: loss: 0.14000 [I 2022-10-27 23:19:40] ...optimizers/trijoint.py.151: total_norm: 13.004335 [I 2022-10-27 23:19:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 1650/2304 [I 2022-10-27 23:19:45] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:06:02 [I 2022-10-27 23:19:45] ...trap/engines/engine.py.229: process: 0.55610 | load: 0.00248 [I 2022-10-27 23:19:45] ...trap/engines/engine.py.230: loss: 0.11450 [I 2022-10-27 23:19:45] ...optimizers/trijoint.py.151: total_norm: 24.713179 [I 2022-10-27 23:19:51] ...trap/engines/engine.py.224: train: epoch 93 | batch 1660/2304 [I 2022-10-27 23:19:51] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:58 [I 2022-10-27 23:19:51] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00254 [I 2022-10-27 23:19:51] ...trap/engines/engine.py.230: loss: 0.13542 [I 2022-10-27 23:19:51] ...optimizers/trijoint.py.151: total_norm: 16.035320 [I 2022-10-27 23:19:56] ...trap/engines/engine.py.224: train: epoch 93 | batch 1670/2304 [I 2022-10-27 23:19:56] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:54 [I 2022-10-27 23:19:56] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00248 [I 2022-10-27 23:19:56] ...trap/engines/engine.py.230: loss: 0.12902 [I 2022-10-27 23:19:56] ...optimizers/trijoint.py.151: total_norm: 25.864622 [I 2022-10-27 23:20:02] ...trap/engines/engine.py.224: train: epoch 93 | batch 1680/2304 [I 2022-10-27 23:20:02] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:46 [I 2022-10-27 23:20:02] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00258 [I 2022-10-27 23:20:02] ...trap/engines/engine.py.230: loss: 0.14394 [I 2022-10-27 23:20:02] ...optimizers/trijoint.py.151: total_norm: 41.658215 [I 2022-10-27 23:20:08] ...trap/engines/engine.py.224: train: epoch 93 | batch 1690/2304 [I 2022-10-27 23:20:08] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:48 [I 2022-10-27 23:20:08] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00254 [I 2022-10-27 23:20:08] ...trap/engines/engine.py.230: loss: 0.12175 [I 2022-10-27 23:20:08] ...optimizers/trijoint.py.151: total_norm: 20.522987 [I 2022-10-27 23:20:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 1700/2304 [I 2022-10-27 23:20:13] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:35 [I 2022-10-27 23:20:13] ...trap/engines/engine.py.229: process: 0.54956 | load: 0.00248 [I 2022-10-27 23:20:13] ...trap/engines/engine.py.230: loss: 0.13529 [I 2022-10-27 23:20:13] ...optimizers/trijoint.py.151: total_norm: 32.285763 [I 2022-10-27 23:20:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 1710/2304 [I 2022-10-27 23:20:19] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:29 [I 2022-10-27 23:20:19] ...trap/engines/engine.py.229: process: 0.55506 | load: 0.00247 [I 2022-10-27 23:20:19] ...trap/engines/engine.py.230: loss: 0.17944 [I 2022-10-27 23:20:19] ...optimizers/trijoint.py.151: total_norm: 16.452469 [I 2022-10-27 23:20:24] ...trap/engines/engine.py.224: train: epoch 93 | batch 1720/2304 [I 2022-10-27 23:20:24] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:27 [I 2022-10-27 23:20:24] ...trap/engines/engine.py.229: process: 0.57531 | load: 0.00245 [I 2022-10-27 23:20:24] ...trap/engines/engine.py.230: loss: 0.14756 [I 2022-10-27 23:20:24] ...optimizers/trijoint.py.151: total_norm: 29.560553 [I 2022-10-27 23:20:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 1730/2304 [I 2022-10-27 23:20:30] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:18 [I 2022-10-27 23:20:30] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00250 [I 2022-10-27 23:20:30] ...trap/engines/engine.py.230: loss: 0.11273 [I 2022-10-27 23:20:30] ...optimizers/trijoint.py.151: total_norm: 9.851164 [I 2022-10-27 23:20:35] ...trap/engines/engine.py.224: train: epoch 93 | batch 1740/2304 [I 2022-10-27 23:20:35] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:05:11 [I 2022-10-27 23:20:35] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00249 [I 2022-10-27 23:20:35] ...trap/engines/engine.py.230: loss: 0.09570 [I 2022-10-27 23:20:35] ...optimizers/trijoint.py.151: total_norm: 8.202579 [I 2022-10-27 23:20:41] ...trap/engines/engine.py.224: train: epoch 93 | batch 1750/2304 [I 2022-10-27 23:20:41] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:05:08 [I 2022-10-27 23:20:41] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00247 [I 2022-10-27 23:20:41] ...trap/engines/engine.py.230: loss: 0.13469 [I 2022-10-27 23:20:41] ...optimizers/trijoint.py.151: total_norm: 12.941966 [I 2022-10-27 23:20:47] ...trap/engines/engine.py.224: train: epoch 93 | batch 1760/2304 [I 2022-10-27 23:20:47] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:05:00 [I 2022-10-27 23:20:47] ...trap/engines/engine.py.229: process: 0.54215 | load: 0.00253 [I 2022-10-27 23:20:47] ...trap/engines/engine.py.230: loss: 0.11721 [I 2022-10-27 23:20:47] ...optimizers/trijoint.py.151: total_norm: 26.230568 [I 2022-10-27 23:20:52] ...trap/engines/engine.py.224: train: epoch 93 | batch 1770/2304 [I 2022-10-27 23:20:52] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:57 [I 2022-10-27 23:20:52] ...trap/engines/engine.py.229: process: 0.54753 | load: 0.00248 [I 2022-10-27 23:20:52] ...trap/engines/engine.py.230: loss: 0.10783 [I 2022-10-27 23:20:52] ...optimizers/trijoint.py.151: total_norm: 15.398625 [I 2022-10-27 23:20:58] ...trap/engines/engine.py.224: train: epoch 93 | batch 1780/2304 [I 2022-10-27 23:20:58] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:52 [I 2022-10-27 23:20:58] ...trap/engines/engine.py.229: process: 0.55257 | load: 0.00248 [I 2022-10-27 23:20:58] ...trap/engines/engine.py.230: loss: 0.10990 [I 2022-10-27 23:20:58] ...optimizers/trijoint.py.151: total_norm: 13.448438 [I 2022-10-27 23:21:03] ...trap/engines/engine.py.224: train: epoch 93 | batch 1790/2304 [I 2022-10-27 23:21:03] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:46 [I 2022-10-27 23:21:03] ...trap/engines/engine.py.229: process: 0.55558 | load: 0.00258 [I 2022-10-27 23:21:03] ...trap/engines/engine.py.230: loss: 0.12118 [I 2022-10-27 23:21:03] ...optimizers/trijoint.py.151: total_norm: 19.566282 [I 2022-10-27 23:21:09] ...trap/engines/engine.py.224: train: epoch 93 | batch 1800/2304 [I 2022-10-27 23:21:09] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:41 [I 2022-10-27 23:21:09] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00254 [I 2022-10-27 23:21:09] ...trap/engines/engine.py.230: loss: 0.11031 [I 2022-10-27 23:21:09] ...optimizers/trijoint.py.151: total_norm: 7.786618 [I 2022-10-27 23:21:15] ...trap/engines/engine.py.224: train: epoch 93 | batch 1810/2304 [I 2022-10-27 23:21:15] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:36 [I 2022-10-27 23:21:15] ...trap/engines/engine.py.229: process: 0.54871 | load: 0.00243 [I 2022-10-27 23:21:15] ...trap/engines/engine.py.230: loss: 0.12432 [I 2022-10-27 23:21:15] ...optimizers/trijoint.py.151: total_norm: 23.266031 [I 2022-10-27 23:21:20] ...trap/engines/engine.py.224: train: epoch 93 | batch 1820/2304 [I 2022-10-27 23:21:20] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:29 [I 2022-10-27 23:21:20] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00250 [I 2022-10-27 23:21:20] ...trap/engines/engine.py.230: loss: 0.11912 [I 2022-10-27 23:21:20] ...optimizers/trijoint.py.151: total_norm: 14.721526 [I 2022-10-27 23:21:26] ...trap/engines/engine.py.224: train: epoch 93 | batch 1830/2304 [I 2022-10-27 23:21:26] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:24 [I 2022-10-27 23:21:26] ...trap/engines/engine.py.229: process: 0.54849 | load: 0.00248 [I 2022-10-27 23:21:26] ...trap/engines/engine.py.230: loss: 0.11868 [I 2022-10-27 23:21:26] ...optimizers/trijoint.py.151: total_norm: 35.955040 [I 2022-10-27 23:21:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 1840/2304 [I 2022-10-27 23:21:31] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:20 [I 2022-10-27 23:21:31] ...trap/engines/engine.py.229: process: 0.56526 | load: 0.00259 [I 2022-10-27 23:21:31] ...trap/engines/engine.py.230: loss: 0.09783 [I 2022-10-27 23:21:31] ...optimizers/trijoint.py.151: total_norm: 19.867704 [I 2022-10-27 23:21:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 1850/2304 [I 2022-10-27 23:21:37] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:14 [I 2022-10-27 23:21:37] ...trap/engines/engine.py.229: process: 0.56198 | load: 0.00252 [I 2022-10-27 23:21:37] ...trap/engines/engine.py.230: loss: 0.14905 [I 2022-10-27 23:21:37] ...optimizers/trijoint.py.151: total_norm: 14.440901 [I 2022-10-27 23:21:43] ...trap/engines/engine.py.224: train: epoch 93 | batch 1860/2304 [I 2022-10-27 23:21:43] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:04:10 [I 2022-10-27 23:21:43] ...trap/engines/engine.py.229: process: 0.57850 | load: 0.00252 [I 2022-10-27 23:21:43] ...trap/engines/engine.py.230: loss: 0.12728 [I 2022-10-27 23:21:43] ...optimizers/trijoint.py.151: total_norm: 19.294228 [I 2022-10-27 23:21:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 1870/2304 [I 2022-10-27 23:21:48] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:04:01 [I 2022-10-27 23:21:48] ...trap/engines/engine.py.229: process: 0.54285 | load: 0.00250 [I 2022-10-27 23:21:48] ...trap/engines/engine.py.230: loss: 0.09954 [I 2022-10-27 23:21:48] ...optimizers/trijoint.py.151: total_norm: 14.620277 [I 2022-10-27 23:21:54] ...trap/engines/engine.py.224: train: epoch 93 | batch 1880/2304 [I 2022-10-27 23:21:54] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:58 [I 2022-10-27 23:21:54] ...trap/engines/engine.py.229: process: 0.56890 | load: 0.00248 [I 2022-10-27 23:21:54] ...trap/engines/engine.py.230: loss: 0.11432 [I 2022-10-27 23:21:54] ...optimizers/trijoint.py.151: total_norm: 10.932932 [I 2022-10-27 23:21:59] ...trap/engines/engine.py.224: train: epoch 93 | batch 1890/2304 [I 2022-10-27 23:21:59] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:50 [I 2022-10-27 23:21:59] ...trap/engines/engine.py.229: process: 0.55770 | load: 0.00254 [I 2022-10-27 23:21:59] ...trap/engines/engine.py.230: loss: 0.12100 [I 2022-10-27 23:21:59] ...optimizers/trijoint.py.151: total_norm: 27.468756 [I 2022-10-27 23:22:05] ...trap/engines/engine.py.224: train: epoch 93 | batch 1900/2304 [I 2022-10-27 23:22:05] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:45 [I 2022-10-27 23:22:05] ...trap/engines/engine.py.229: process: 0.55863 | load: 0.00244 [I 2022-10-27 23:22:05] ...trap/engines/engine.py.230: loss: 0.10490 [I 2022-10-27 23:22:05] ...optimizers/trijoint.py.151: total_norm: 15.038768 [I 2022-10-27 23:22:10] ...trap/engines/engine.py.224: train: epoch 93 | batch 1910/2304 [I 2022-10-27 23:22:10] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:39 [I 2022-10-27 23:22:10] ...trap/engines/engine.py.229: process: 0.55954 | load: 0.00243 [I 2022-10-27 23:22:10] ...trap/engines/engine.py.230: loss: 0.11629 [I 2022-10-27 23:22:10] ...optimizers/trijoint.py.151: total_norm: 9.197531 [I 2022-10-27 23:22:16] ...trap/engines/engine.py.224: train: epoch 93 | batch 1920/2304 [I 2022-10-27 23:22:16] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:34 [I 2022-10-27 23:22:16] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00247 [I 2022-10-27 23:22:16] ...trap/engines/engine.py.230: loss: 0.13128 [I 2022-10-27 23:22:16] ...optimizers/trijoint.py.151: total_norm: 18.411476 [I 2022-10-27 23:22:22] ...trap/engines/engine.py.224: train: epoch 93 | batch 1930/2304 [I 2022-10-27 23:22:22] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:27 [I 2022-10-27 23:22:22] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00245 [I 2022-10-27 23:22:22] ...trap/engines/engine.py.230: loss: 0.13994 [I 2022-10-27 23:22:22] ...optimizers/trijoint.py.151: total_norm: 28.630163 [I 2022-10-27 23:22:27] ...trap/engines/engine.py.224: train: epoch 93 | batch 1940/2304 [I 2022-10-27 23:22:27] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:23 [I 2022-10-27 23:22:27] ...trap/engines/engine.py.229: process: 0.56687 | load: 0.00252 [I 2022-10-27 23:22:27] ...trap/engines/engine.py.230: loss: 0.10692 [I 2022-10-27 23:22:27] ...optimizers/trijoint.py.151: total_norm: 20.291639 [I 2022-10-27 23:22:33] ...trap/engines/engine.py.224: train: epoch 93 | batch 1950/2304 [I 2022-10-27 23:22:33] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:17 [I 2022-10-27 23:22:33] ...trap/engines/engine.py.229: process: 0.55865 | load: 0.00267 [I 2022-10-27 23:22:33] ...trap/engines/engine.py.230: loss: 0.15661 [I 2022-10-27 23:22:33] ...optimizers/trijoint.py.151: total_norm: 16.224258 [I 2022-10-27 23:22:38] ...trap/engines/engine.py.224: train: epoch 93 | batch 1960/2304 [I 2022-10-27 23:22:38] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:03:10 [I 2022-10-27 23:22:38] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00250 [I 2022-10-27 23:22:38] ...trap/engines/engine.py.230: loss: 0.13677 [I 2022-10-27 23:22:38] ...optimizers/trijoint.py.151: total_norm: 17.929695 [I 2022-10-27 23:22:44] ...trap/engines/engine.py.224: train: epoch 93 | batch 1970/2304 [I 2022-10-27 23:22:44] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:03:05 [I 2022-10-27 23:22:44] ...trap/engines/engine.py.229: process: 0.54532 | load: 0.00251 [I 2022-10-27 23:22:44] ...trap/engines/engine.py.230: loss: 0.10066 [I 2022-10-27 23:22:44] ...optimizers/trijoint.py.151: total_norm: 10.937626 [I 2022-10-27 23:22:50] ...trap/engines/engine.py.224: train: epoch 93 | batch 1980/2304 [I 2022-10-27 23:22:50] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:03:01 [I 2022-10-27 23:22:50] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00250 [I 2022-10-27 23:22:50] ...trap/engines/engine.py.230: loss: 0.08801 [I 2022-10-27 23:22:50] ...optimizers/trijoint.py.151: total_norm: 9.567735 [I 2022-10-27 23:22:55] ...trap/engines/engine.py.224: train: epoch 93 | batch 1990/2304 [I 2022-10-27 23:22:55] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:54 [I 2022-10-27 23:22:55] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00257 [I 2022-10-27 23:22:55] ...trap/engines/engine.py.230: loss: 0.15558 [I 2022-10-27 23:22:55] ...optimizers/trijoint.py.151: total_norm: 11.897529 [I 2022-10-27 23:23:01] ...trap/engines/engine.py.224: train: epoch 93 | batch 2000/2304 [I 2022-10-27 23:23:01] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:50 [I 2022-10-27 23:23:01] ...trap/engines/engine.py.229: process: 0.54645 | load: 0.00252 [I 2022-10-27 23:23:01] ...trap/engines/engine.py.230: loss: 0.15075 [I 2022-10-27 23:23:01] ...optimizers/trijoint.py.151: total_norm: 9.203515 [I 2022-10-27 23:23:06] ...trap/engines/engine.py.224: train: epoch 93 | batch 2010/2304 [I 2022-10-27 23:23:06] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:44 [I 2022-10-27 23:23:06] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00257 [I 2022-10-27 23:23:06] ...trap/engines/engine.py.230: loss: 0.09850 [I 2022-10-27 23:23:06] ...optimizers/trijoint.py.151: total_norm: 23.647556 [I 2022-10-27 23:23:12] ...trap/engines/engine.py.224: train: epoch 93 | batch 2020/2304 [I 2022-10-27 23:23:12] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:37 [I 2022-10-27 23:23:12] ...trap/engines/engine.py.229: process: 0.56112 | load: 0.00261 [I 2022-10-27 23:23:12] ...trap/engines/engine.py.230: loss: 0.13781 [I 2022-10-27 23:23:12] ...optimizers/trijoint.py.151: total_norm: 18.781771 [I 2022-10-27 23:23:17] ...trap/engines/engine.py.224: train: epoch 93 | batch 2030/2304 [I 2022-10-27 23:23:17] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:33 [I 2022-10-27 23:23:17] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00255 [I 2022-10-27 23:23:17] ...trap/engines/engine.py.230: loss: 0.11859 [I 2022-10-27 23:23:17] ...optimizers/trijoint.py.151: total_norm: 10.154013 [I 2022-10-27 23:23:23] ...trap/engines/engine.py.224: train: epoch 93 | batch 2040/2304 [I 2022-10-27 23:23:23] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:27 [I 2022-10-27 23:23:23] ...trap/engines/engine.py.229: process: 0.55400 | load: 0.00274 [I 2022-10-27 23:23:23] ...trap/engines/engine.py.230: loss: 0.15435 [I 2022-10-27 23:23:23] ...optimizers/trijoint.py.151: total_norm: 9.358395 [I 2022-10-27 23:23:29] ...trap/engines/engine.py.224: train: epoch 93 | batch 2050/2304 [I 2022-10-27 23:23:29] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:21 [I 2022-10-27 23:23:29] ...trap/engines/engine.py.229: process: 0.54699 | load: 0.00257 [I 2022-10-27 23:23:29] ...trap/engines/engine.py.230: loss: 0.13510 [I 2022-10-27 23:23:29] ...optimizers/trijoint.py.151: total_norm: 33.504086 [I 2022-10-27 23:23:34] ...trap/engines/engine.py.224: train: epoch 93 | batch 2060/2304 [I 2022-10-27 23:23:34] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:02:15 [I 2022-10-27 23:23:34] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00253 [I 2022-10-27 23:23:34] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-27 23:23:34] ...optimizers/trijoint.py.151: total_norm: 9.139695 [I 2022-10-27 23:23:40] ...trap/engines/engine.py.224: train: epoch 93 | batch 2070/2304 [I 2022-10-27 23:23:40] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:10 [I 2022-10-27 23:23:40] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00271 [I 2022-10-27 23:23:40] ...trap/engines/engine.py.230: loss: 0.14965 [I 2022-10-27 23:23:40] ...optimizers/trijoint.py.151: total_norm: 25.711178 [I 2022-10-27 23:23:45] ...trap/engines/engine.py.224: train: epoch 93 | batch 2080/2304 [I 2022-10-27 23:23:45] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:02:04 [I 2022-10-27 23:23:45] ...trap/engines/engine.py.229: process: 0.54293 | load: 0.00257 [I 2022-10-27 23:23:45] ...trap/engines/engine.py.230: loss: 0.17244 [I 2022-10-27 23:23:45] ...optimizers/trijoint.py.151: total_norm: 23.110683 [I 2022-10-27 23:23:51] ...trap/engines/engine.py.224: train: epoch 93 | batch 2090/2304 [I 2022-10-27 23:23:51] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:59 [I 2022-10-27 23:23:51] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00250 [I 2022-10-27 23:23:51] ...trap/engines/engine.py.230: loss: 0.18149 [I 2022-10-27 23:23:51] ...optimizers/trijoint.py.151: total_norm: 18.300646 [I 2022-10-27 23:23:57] ...trap/engines/engine.py.224: train: epoch 93 | batch 2100/2304 [I 2022-10-27 23:23:57] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:53 [I 2022-10-27 23:23:57] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00247 [I 2022-10-27 23:23:57] ...trap/engines/engine.py.230: loss: 0.09224 [I 2022-10-27 23:23:57] ...optimizers/trijoint.py.151: total_norm: 10.633286 [I 2022-10-27 23:24:02] ...trap/engines/engine.py.224: train: epoch 93 | batch 2110/2304 [I 2022-10-27 23:24:02] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:48 [I 2022-10-27 23:24:02] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00250 [I 2022-10-27 23:24:02] ...trap/engines/engine.py.230: loss: 0.08275 [I 2022-10-27 23:24:02] ...optimizers/trijoint.py.151: total_norm: 9.456661 [I 2022-10-27 23:24:08] ...trap/engines/engine.py.224: train: epoch 93 | batch 2120/2304 [I 2022-10-27 23:24:08] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:42 [I 2022-10-27 23:24:08] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00250 [I 2022-10-27 23:24:08] ...trap/engines/engine.py.230: loss: 0.17919 [I 2022-10-27 23:24:08] ...optimizers/trijoint.py.151: total_norm: 15.460467 [I 2022-10-27 23:24:13] ...trap/engines/engine.py.224: train: epoch 93 | batch 2130/2304 [I 2022-10-27 23:24:13] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:36 [I 2022-10-27 23:24:13] ...trap/engines/engine.py.229: process: 0.54414 | load: 0.00256 [I 2022-10-27 23:24:13] ...trap/engines/engine.py.230: loss: 0.11933 [I 2022-10-27 23:24:13] ...optimizers/trijoint.py.151: total_norm: 12.450918 [I 2022-10-27 23:24:19] ...trap/engines/engine.py.224: train: epoch 93 | batch 2140/2304 [I 2022-10-27 23:24:19] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:31 [I 2022-10-27 23:24:19] ...trap/engines/engine.py.229: process: 0.56593 | load: 0.00247 [I 2022-10-27 23:24:19] ...trap/engines/engine.py.230: loss: 0.13373 [I 2022-10-27 23:24:19] ...optimizers/trijoint.py.151: total_norm: 18.627893 [I 2022-10-27 23:24:24] ...trap/engines/engine.py.224: train: epoch 93 | batch 2150/2304 [I 2022-10-27 23:24:24] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:25 [I 2022-10-27 23:24:24] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00245 [I 2022-10-27 23:24:24] ...trap/engines/engine.py.230: loss: 0.13143 [I 2022-10-27 23:24:24] ...optimizers/trijoint.py.151: total_norm: 27.039450 [I 2022-10-27 23:24:30] ...trap/engines/engine.py.224: train: epoch 93 | batch 2160/2304 [I 2022-10-27 23:24:30] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:20 [I 2022-10-27 23:24:30] ...trap/engines/engine.py.229: process: 0.55943 | load: 0.00259 [I 2022-10-27 23:24:30] ...trap/engines/engine.py.230: loss: 0.09941 [I 2022-10-27 23:24:30] ...optimizers/trijoint.py.151: total_norm: 8.722005 [I 2022-10-27 23:24:36] ...trap/engines/engine.py.224: train: epoch 93 | batch 2170/2304 [I 2022-10-27 23:24:36] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:01:14 [I 2022-10-27 23:24:36] ...trap/engines/engine.py.229: process: 0.55024 | load: 0.00248 [I 2022-10-27 23:24:36] ...trap/engines/engine.py.230: loss: 0.07926 [I 2022-10-27 23:24:36] ...optimizers/trijoint.py.151: total_norm: 10.757518 [I 2022-10-27 23:24:41] ...trap/engines/engine.py.224: train: epoch 93 | batch 2180/2304 [I 2022-10-27 23:24:41] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:01:09 [I 2022-10-27 23:24:41] ...trap/engines/engine.py.229: process: 0.55469 | load: 0.00248 [I 2022-10-27 23:24:41] ...trap/engines/engine.py.230: loss: 0.15137 [I 2022-10-27 23:24:41] ...optimizers/trijoint.py.151: total_norm: 21.943327 [I 2022-10-27 23:24:47] ...trap/engines/engine.py.224: train: epoch 93 | batch 2190/2304 [I 2022-10-27 23:24:47] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:01:03 [I 2022-10-27 23:24:47] ...trap/engines/engine.py.229: process: 0.54645 | load: 0.00248 [I 2022-10-27 23:24:47] ...trap/engines/engine.py.230: loss: 0.09514 [I 2022-10-27 23:24:47] ...optimizers/trijoint.py.151: total_norm: 14.976129 [I 2022-10-27 23:24:52] ...trap/engines/engine.py.224: train: epoch 93 | batch 2200/2304 [I 2022-10-27 23:24:52] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:58 [I 2022-10-27 23:24:52] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00252 [I 2022-10-27 23:24:52] ...trap/engines/engine.py.230: loss: 0.13800 [I 2022-10-27 23:24:52] ...optimizers/trijoint.py.151: total_norm: 26.508648 [I 2022-10-27 23:24:58] ...trap/engines/engine.py.224: train: epoch 93 | batch 2210/2304 [I 2022-10-27 23:24:58] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:52 [I 2022-10-27 23:24:58] ...trap/engines/engine.py.229: process: 0.56287 | load: 0.00261 [I 2022-10-27 23:24:58] ...trap/engines/engine.py.230: loss: 0.11500 [I 2022-10-27 23:24:58] ...optimizers/trijoint.py.151: total_norm: 12.629005 [I 2022-10-27 23:25:04] ...trap/engines/engine.py.224: train: epoch 93 | batch 2220/2304 [I 2022-10-27 23:25:04] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:46 [I 2022-10-27 23:25:04] ...trap/engines/engine.py.229: process: 0.54815 | load: 0.00252 [I 2022-10-27 23:25:04] ...trap/engines/engine.py.230: loss: 0.10133 [I 2022-10-27 23:25:04] ...optimizers/trijoint.py.151: total_norm: 13.102397 [I 2022-10-27 23:25:09] ...trap/engines/engine.py.224: train: epoch 93 | batch 2230/2304 [I 2022-10-27 23:25:09] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:40 [I 2022-10-27 23:25:09] ...trap/engines/engine.py.229: process: 0.53973 | load: 0.00251 [I 2022-10-27 23:25:09] ...trap/engines/engine.py.230: loss: 0.09068 [I 2022-10-27 23:25:09] ...optimizers/trijoint.py.151: total_norm: 23.280107 [I 2022-10-27 23:25:15] ...trap/engines/engine.py.224: train: epoch 93 | batch 2240/2304 [I 2022-10-27 23:25:15] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:35 [I 2022-10-27 23:25:15] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00250 [I 2022-10-27 23:25:15] ...trap/engines/engine.py.230: loss: 0.08643 [I 2022-10-27 23:25:15] ...optimizers/trijoint.py.151: total_norm: 14.080445 [I 2022-10-27 23:25:20] ...trap/engines/engine.py.224: train: epoch 93 | batch 2250/2304 [I 2022-10-27 23:25:20] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:30 [I 2022-10-27 23:25:20] ...trap/engines/engine.py.229: process: 0.56220 | load: 0.00256 [I 2022-10-27 23:25:20] ...trap/engines/engine.py.230: loss: 0.11564 [I 2022-10-27 23:25:20] ...optimizers/trijoint.py.151: total_norm: 12.252927 [I 2022-10-27 23:25:26] ...trap/engines/engine.py.224: train: epoch 93 | batch 2260/2304 [I 2022-10-27 23:25:26] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:24 [I 2022-10-27 23:25:26] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00252 [I 2022-10-27 23:25:26] ...trap/engines/engine.py.230: loss: 0.09117 [I 2022-10-27 23:25:26] ...optimizers/trijoint.py.151: total_norm: 16.518257 [I 2022-10-27 23:25:31] ...trap/engines/engine.py.224: train: epoch 93 | batch 2270/2304 [I 2022-10-27 23:25:31] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:18 [I 2022-10-27 23:25:31] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00250 [I 2022-10-27 23:25:31] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-27 23:25:31] ...optimizers/trijoint.py.151: total_norm: 11.247121 [I 2022-10-27 23:25:37] ...trap/engines/engine.py.224: train: epoch 93 | batch 2280/2304 [I 2022-10-27 23:25:37] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:13 [I 2022-10-27 23:25:37] ...trap/engines/engine.py.229: process: 0.55115 | load: 0.00249 [I 2022-10-27 23:25:37] ...trap/engines/engine.py.230: loss: 0.13089 [I 2022-10-27 23:25:37] ...optimizers/trijoint.py.151: total_norm: 23.569164 [I 2022-10-27 23:25:43] ...trap/engines/engine.py.224: train: epoch 93 | batch 2290/2304 [I 2022-10-27 23:25:43] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:07 [I 2022-10-27 23:25:43] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00246 [I 2022-10-27 23:25:43] ...trap/engines/engine.py.230: loss: 0.14877 [I 2022-10-27 23:25:43] ...optimizers/trijoint.py.151: total_norm: 26.731445 [I 2022-10-27 23:25:48] ...trap/engines/engine.py.224: train: epoch 93 | batch 2300/2304 [I 2022-10-27 23:25:48] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:02 [I 2022-10-27 23:25:48] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00015 [I 2022-10-27 23:25:48] ...trap/engines/engine.py.230: loss: 0.12271 [I 2022-10-27 23:25:48] ...optimizers/trijoint.py.151: total_norm: 13.841147 [I 2022-10-27 23:25:50] ...trap/engines/engine.py.224: train: epoch 93 | batch 2304/2304 [I 2022-10-27 23:25:50] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:00 [I 2022-10-27 23:25:50] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00015 [I 2022-10-27 23:25:50] ...trap/engines/engine.py.230: loss: 0.08923 [I 2022-10-27 23:25:50] ...optimizers/trijoint.py.151: total_norm: 25.060299 [S 2022-10-27 23:25:52] ...trap/engines/engine.py.129: train_epoch.epoch: 93 [S 2022-10-27 23:25:52] ...trap/engines/engine.py.129: train_epoch.loss: 0.12965561188353375 [I 2022-10-27 23:25:52] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 23:25:53] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 93 [I 2022-10-27 23:25:53] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 23:25:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 23:25:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 23:25:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 23:25:58] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 23:25:58] ...trap/engines/engine.py.321: eval: epoch 93 | batch 0/510 [I 2022-10-27 23:25:58] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:46:17 [I 2022-10-27 23:25:58] ...trap/engines/engine.py.326: process: 5.44692 | load: 3.14115 [I 2022-10-27 23:26:02] ...trap/engines/engine.py.321: eval: epoch 93 | batch 10/510 [I 2022-10-27 23:26:02] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:21 [I 2022-10-27 23:26:02] ...trap/engines/engine.py.326: process: 0.33046 | load: 0.00022 [I 2022-10-27 23:26:05] ...trap/engines/engine.py.321: eval: epoch 93 | batch 20/510 [I 2022-10-27 23:26:05] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:10 [I 2022-10-27 23:26:05] ...trap/engines/engine.py.326: process: 0.32616 | load: 0.00017 [I 2022-10-27 23:26:09] ...trap/engines/engine.py.321: eval: epoch 93 | batch 30/510 [I 2022-10-27 23:26:09] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:42 [I 2022-10-27 23:26:09] ...trap/engines/engine.py.326: process: 0.32657 | load: 0.00017 [I 2022-10-27 23:26:12] ...trap/engines/engine.py.321: eval: epoch 93 | batch 40/510 [I 2022-10-27 23:26:12] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:38 [I 2022-10-27 23:26:12] ...trap/engines/engine.py.326: process: 0.34210 | load: 0.00022 [I 2022-10-27 23:26:15] ...trap/engines/engine.py.321: eval: epoch 93 | batch 50/510 [I 2022-10-27 23:26:15] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:30 [I 2022-10-27 23:26:15] ...trap/engines/engine.py.326: process: 0.31787 | load: 0.00020 [I 2022-10-27 23:26:18] ...trap/engines/engine.py.321: eval: epoch 93 | batch 60/510 [I 2022-10-27 23:26:18] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:30 [I 2022-10-27 23:26:18] ...trap/engines/engine.py.326: process: 0.33994 | load: 0.00016 [I 2022-10-27 23:26:22] ...trap/engines/engine.py.321: eval: epoch 93 | batch 70/510 [I 2022-10-27 23:26:22] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:30 [I 2022-10-27 23:26:22] ...trap/engines/engine.py.326: process: 0.34966 | load: 0.00018 [I 2022-10-27 23:26:25] ...trap/engines/engine.py.321: eval: epoch 93 | batch 80/510 [I 2022-10-27 23:26:25] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:24 [I 2022-10-27 23:26:25] ...trap/engines/engine.py.326: process: 0.31903 | load: 0.00021 [I 2022-10-27 23:26:29] ...trap/engines/engine.py.321: eval: epoch 93 | batch 90/510 [I 2022-10-27 23:26:29] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:24 [I 2022-10-27 23:26:29] ...trap/engines/engine.py.326: process: 0.33014 | load: 0.00021 [I 2022-10-27 23:26:32] ...trap/engines/engine.py.321: eval: epoch 93 | batch 100/510 [I 2022-10-27 23:26:32] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:18 [I 2022-10-27 23:26:32] ...trap/engines/engine.py.326: process: 0.28952 | load: 0.00019 [I 2022-10-27 23:26:35] ...trap/engines/engine.py.321: eval: epoch 93 | batch 110/510 [I 2022-10-27 23:26:35] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-27 23:26:35] ...trap/engines/engine.py.326: process: 0.28308 | load: 0.00014 [I 2022-10-27 23:26:38] ...trap/engines/engine.py.321: eval: epoch 93 | batch 120/510 [I 2022-10-27 23:26:38] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:51 [I 2022-10-27 23:26:38] ...trap/engines/engine.py.326: process: 0.27535 | load: 0.00015 [I 2022-10-27 23:26:41] ...trap/engines/engine.py.321: eval: epoch 93 | batch 130/510 [I 2022-10-27 23:26:41] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-27 23:26:41] ...trap/engines/engine.py.326: process: 0.28881 | load: 0.00014 [I 2022-10-27 23:26:44] ...trap/engines/engine.py.321: eval: epoch 93 | batch 140/510 [I 2022-10-27 23:26:44] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-27 23:26:44] ...trap/engines/engine.py.326: process: 0.28625 | load: 0.00018 [I 2022-10-27 23:26:47] ...trap/engines/engine.py.321: eval: epoch 93 | batch 150/510 [I 2022-10-27 23:26:47] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-27 23:26:47] ...trap/engines/engine.py.326: process: 0.28822 | load: 0.00016 [I 2022-10-27 23:26:49] ...trap/engines/engine.py.321: eval: epoch 93 | batch 160/510 [I 2022-10-27 23:26:49] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-27 23:26:49] ...trap/engines/engine.py.326: process: 0.27962 | load: 0.00015 [I 2022-10-27 23:26:52] ...trap/engines/engine.py.321: eval: epoch 93 | batch 170/510 [I 2022-10-27 23:26:52] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-27 23:26:52] ...trap/engines/engine.py.326: process: 0.29167 | load: 0.00015 [I 2022-10-27 23:26:55] ...trap/engines/engine.py.321: eval: epoch 93 | batch 180/510 [I 2022-10-27 23:26:55] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:36 [I 2022-10-27 23:26:55] ...trap/engines/engine.py.326: process: 0.29605 | load: 0.00019 [I 2022-10-27 23:26:58] ...trap/engines/engine.py.321: eval: epoch 93 | batch 190/510 [I 2022-10-27 23:26:58] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:33 [I 2022-10-27 23:26:58] ...trap/engines/engine.py.326: process: 0.29511 | load: 0.00024 [I 2022-10-27 23:27:01] ...trap/engines/engine.py.321: eval: epoch 93 | batch 200/510 [I 2022-10-27 23:27:01] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-27 23:27:01] ...trap/engines/engine.py.326: process: 0.29220 | load: 0.00014 [I 2022-10-27 23:27:04] ...trap/engines/engine.py.321: eval: epoch 93 | batch 210/510 [I 2022-10-27 23:27:04] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-27 23:27:04] ...trap/engines/engine.py.326: process: 0.28372 | load: 0.00015 [I 2022-10-27 23:27:07] ...trap/engines/engine.py.321: eval: epoch 93 | batch 220/510 [I 2022-10-27 23:27:07] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:24 [I 2022-10-27 23:27:07] ...trap/engines/engine.py.326: process: 0.28624 | load: 0.00021 [I 2022-10-27 23:27:10] ...trap/engines/engine.py.321: eval: epoch 93 | batch 230/510 [I 2022-10-27 23:27:10] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:22 [I 2022-10-27 23:27:10] ...trap/engines/engine.py.326: process: 0.29215 | load: 0.00019 [I 2022-10-27 23:27:13] ...trap/engines/engine.py.321: eval: epoch 93 | batch 240/510 [I 2022-10-27 23:27:13] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:22 [I 2022-10-27 23:27:13] ...trap/engines/engine.py.326: process: 0.33511 | load: 0.00016 [I 2022-10-27 23:27:16] ...trap/engines/engine.py.321: eval: epoch 93 | batch 250/510 [I 2022-10-27 23:27:16] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-27 23:27:16] ...trap/engines/engine.py.326: process: 0.27835 | load: 0.00014 [I 2022-10-27 23:27:19] ...trap/engines/engine.py.321: eval: epoch 93 | batch 260/510 [I 2022-10-27 23:27:19] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-27 23:27:19] ...trap/engines/engine.py.326: process: 0.28805 | load: 0.00015 [I 2022-10-27 23:27:21] ...trap/engines/engine.py.321: eval: epoch 93 | batch 270/510 [I 2022-10-27 23:27:21] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-27 23:27:21] ...trap/engines/engine.py.326: process: 0.29296 | load: 0.00015 [I 2022-10-27 23:27:24] ...trap/engines/engine.py.321: eval: epoch 93 | batch 280/510 [I 2022-10-27 23:27:24] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-27 23:27:24] ...trap/engines/engine.py.326: process: 0.28932 | load: 0.00015 [I 2022-10-27 23:27:27] ...trap/engines/engine.py.321: eval: epoch 93 | batch 290/510 [I 2022-10-27 23:27:27] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-27 23:27:27] ...trap/engines/engine.py.326: process: 0.28323 | load: 0.00015 [I 2022-10-27 23:27:30] ...trap/engines/engine.py.321: eval: epoch 93 | batch 300/510 [I 2022-10-27 23:27:30] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-27 23:27:30] ...trap/engines/engine.py.326: process: 0.29009 | load: 0.00021 [I 2022-10-27 23:27:33] ...trap/engines/engine.py.321: eval: epoch 93 | batch 310/510 [I 2022-10-27 23:27:33] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:58 [I 2022-10-27 23:27:33] ...trap/engines/engine.py.326: process: 0.29575 | load: 0.00027 [I 2022-10-27 23:27:36] ...trap/engines/engine.py.321: eval: epoch 93 | batch 320/510 [I 2022-10-27 23:27:36] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:55 [I 2022-10-27 23:27:36] ...trap/engines/engine.py.326: process: 0.28839 | load: 0.00020 [I 2022-10-27 23:27:39] ...trap/engines/engine.py.321: eval: epoch 93 | batch 330/510 [I 2022-10-27 23:27:39] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-27 23:27:39] ...trap/engines/engine.py.326: process: 0.28974 | load: 0.00014 [I 2022-10-27 23:27:42] ...trap/engines/engine.py.321: eval: epoch 93 | batch 340/510 [I 2022-10-27 23:27:42] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-27 23:27:42] ...trap/engines/engine.py.326: process: 0.29315 | load: 0.00015 [I 2022-10-27 23:27:45] ...trap/engines/engine.py.321: eval: epoch 93 | batch 350/510 [I 2022-10-27 23:27:45] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 23:27:45] ...trap/engines/engine.py.326: process: 0.28979 | load: 0.00015 [I 2022-10-27 23:27:47] ...trap/engines/engine.py.321: eval: epoch 93 | batch 360/510 [I 2022-10-27 23:27:47] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-27 23:27:47] ...trap/engines/engine.py.326: process: 0.28511 | load: 0.00015 [I 2022-10-27 23:27:50] ...trap/engines/engine.py.321: eval: epoch 93 | batch 370/510 [I 2022-10-27 23:27:50] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-27 23:27:50] ...trap/engines/engine.py.326: process: 0.30729 | load: 0.00015 [I 2022-10-27 23:27:53] ...trap/engines/engine.py.321: eval: epoch 93 | batch 380/510 [I 2022-10-27 23:27:53] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-27 23:27:53] ...trap/engines/engine.py.326: process: 0.28798 | load: 0.00016 [I 2022-10-27 23:27:56] ...trap/engines/engine.py.321: eval: epoch 93 | batch 390/510 [I 2022-10-27 23:27:56] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-27 23:27:56] ...trap/engines/engine.py.326: process: 0.28534 | load: 0.00015 [I 2022-10-27 23:27:59] ...trap/engines/engine.py.321: eval: epoch 93 | batch 400/510 [I 2022-10-27 23:27:59] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:31 [I 2022-10-27 23:27:59] ...trap/engines/engine.py.326: process: 0.28230 | load: 0.00016 [I 2022-10-27 23:28:02] ...trap/engines/engine.py.321: eval: epoch 93 | batch 410/510 [I 2022-10-27 23:28:02] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-27 23:28:02] ...trap/engines/engine.py.326: process: 0.28913 | load: 0.00015 [I 2022-10-27 23:28:05] ...trap/engines/engine.py.321: eval: epoch 93 | batch 420/510 [I 2022-10-27 23:28:05] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 23:28:05] ...trap/engines/engine.py.326: process: 0.28074 | load: 0.00016 [I 2022-10-27 23:28:08] ...trap/engines/engine.py.321: eval: epoch 93 | batch 430/510 [I 2022-10-27 23:28:08] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 23:28:08] ...trap/engines/engine.py.326: process: 0.28857 | load: 0.00015 [I 2022-10-27 23:28:11] ...trap/engines/engine.py.321: eval: epoch 93 | batch 440/510 [I 2022-10-27 23:28:11] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 23:28:11] ...trap/engines/engine.py.326: process: 0.28754 | load: 0.00015 [I 2022-10-27 23:28:13] ...trap/engines/engine.py.321: eval: epoch 93 | batch 450/510 [I 2022-10-27 23:28:13] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 23:28:13] ...trap/engines/engine.py.326: process: 0.27857 | load: 0.00021 [I 2022-10-27 23:28:16] ...trap/engines/engine.py.321: eval: epoch 93 | batch 460/510 [I 2022-10-27 23:28:16] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-27 23:28:16] ...trap/engines/engine.py.326: process: 0.28549 | load: 0.00024 [I 2022-10-27 23:28:19] ...trap/engines/engine.py.321: eval: epoch 93 | batch 470/510 [I 2022-10-27 23:28:19] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-27 23:28:19] ...trap/engines/engine.py.326: process: 0.29692 | load: 0.00022 [I 2022-10-27 23:28:22] ...trap/engines/engine.py.321: eval: epoch 93 | batch 480/510 [I 2022-10-27 23:28:22] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-27 23:28:22] ...trap/engines/engine.py.326: process: 0.28401 | load: 0.00022 [I 2022-10-27 23:28:25] ...trap/engines/engine.py.321: eval: epoch 93 | batch 490/510 [I 2022-10-27 23:28:25] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 23:28:25] ...trap/engines/engine.py.326: process: 0.27730 | load: 0.00021 [I 2022-10-27 23:28:28] ...trap/engines/engine.py.321: eval: epoch 93 | batch 500/510 [I 2022-10-27 23:28:28] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 23:28:28] ...trap/engines/engine.py.326: process: 0.28579 | load: 0.00016 [I 2022-10-27 23:28:31] ...trap/engines/engine.py.321: eval: epoch 93 | batch 510/510 [I 2022-10-27 23:28:31] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 23:28:31] ...trap/engines/engine.py.326: process: 0.28324 | load: 0.00011 [S 2022-10-27 23:28:31] ...trap/engines/engine.py.132: eval_epoch.epoch: 93 [I 2022-10-27 23:28:31] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.740000 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907000 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933100 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.726000 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.901600 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931400 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014960 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009455 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008264 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012353 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008114 [S 2022-10-27 23:28:36] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006296 [I 2022-10-27 23:28:37] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-27 23:28:37] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 23:28:38] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 23:28:38] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 23:28:38] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 23:28:42] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 23:28:43] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 23:28:49] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 23:28:49] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 23:28:49] ...trap/engines/engine.py.470: Saving model... [I 2022-10-27 23:28:52] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 23:28:58] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 23:28:58] ...trap/engines/engine.py.163: Training model on trainset for epoch 94 [I 2022-10-27 23:28:58] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 23:29:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 0/2304 [I 2022-10-27 23:29:01] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:15 [I 2022-10-27 23:29:01] ...trap/engines/engine.py.229: process: 2.68882 | load: 2.04994 [I 2022-10-27 23:29:01] ...trap/engines/engine.py.230: loss: 0.11816 [I 2022-10-27 23:29:01] ...optimizers/trijoint.py.151: total_norm: 82.120621 [I 2022-10-27 23:29:06] ...trap/engines/engine.py.224: train: epoch 94 | batch 10/2304 [I 2022-10-27 23:29:06] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:59 [I 2022-10-27 23:29:06] ...trap/engines/engine.py.229: process: 0.55064 | load: 0.00403 [I 2022-10-27 23:29:06] ...trap/engines/engine.py.230: loss: 0.10150 [I 2022-10-27 23:29:06] ...optimizers/trijoint.py.151: total_norm: 15.044527 [I 2022-10-27 23:29:12] ...trap/engines/engine.py.224: train: epoch 94 | batch 20/2304 [I 2022-10-27 23:29:12] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:04 [I 2022-10-27 23:29:12] ...trap/engines/engine.py.229: process: 0.55203 | load: 0.00425 [I 2022-10-27 23:29:12] ...trap/engines/engine.py.230: loss: 0.17760 [I 2022-10-27 23:29:12] ...optimizers/trijoint.py.151: total_norm: 9.512670 [I 2022-10-27 23:29:17] ...trap/engines/engine.py.224: train: epoch 94 | batch 30/2304 [I 2022-10-27 23:29:17] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:17 [I 2022-10-27 23:29:17] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00317 [I 2022-10-27 23:29:17] ...trap/engines/engine.py.230: loss: 0.21031 [I 2022-10-27 23:29:17] ...optimizers/trijoint.py.151: total_norm: 16.722519 [I 2022-10-27 23:29:23] ...trap/engines/engine.py.224: train: epoch 94 | batch 40/2304 [I 2022-10-27 23:29:23] ...trap/engines/engine.py.228: elapsed: 0:00:24 | left: 0:20:54 [I 2022-10-27 23:29:23] ...trap/engines/engine.py.229: process: 0.55293 | load: 0.00268 [I 2022-10-27 23:29:23] ...trap/engines/engine.py.230: loss: 0.14822 [I 2022-10-27 23:29:23] ...optimizers/trijoint.py.151: total_norm: 21.050997 [I 2022-10-27 23:29:29] ...trap/engines/engine.py.224: train: epoch 94 | batch 50/2304 [I 2022-10-27 23:29:29] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:09 [I 2022-10-27 23:29:29] ...trap/engines/engine.py.229: process: 0.56607 | load: 0.00296 [I 2022-10-27 23:29:29] ...trap/engines/engine.py.230: loss: 0.10261 [I 2022-10-27 23:29:29] ...optimizers/trijoint.py.151: total_norm: 11.879748 [I 2022-10-27 23:29:34] ...trap/engines/engine.py.224: train: epoch 94 | batch 60/2304 [I 2022-10-27 23:29:34] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:54 [I 2022-10-27 23:29:34] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00259 [I 2022-10-27 23:29:34] ...trap/engines/engine.py.230: loss: 0.16776 [I 2022-10-27 23:29:34] ...optimizers/trijoint.py.151: total_norm: 20.308064 [I 2022-10-27 23:29:40] ...trap/engines/engine.py.224: train: epoch 94 | batch 70/2304 [I 2022-10-27 23:29:40] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:53 [I 2022-10-27 23:29:40] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00290 [I 2022-10-27 23:29:40] ...trap/engines/engine.py.230: loss: 0.18241 [I 2022-10-27 23:29:40] ...optimizers/trijoint.py.151: total_norm: 38.898975 [I 2022-10-27 23:29:45] ...trap/engines/engine.py.224: train: epoch 94 | batch 80/2304 [I 2022-10-27 23:29:45] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:53 [I 2022-10-27 23:29:45] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00303 [I 2022-10-27 23:29:45] ...trap/engines/engine.py.230: loss: 0.12001 [I 2022-10-27 23:29:45] ...optimizers/trijoint.py.151: total_norm: 22.086136 [I 2022-10-27 23:29:51] ...trap/engines/engine.py.224: train: epoch 94 | batch 90/2304 [I 2022-10-27 23:29:51] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:39 [I 2022-10-27 23:29:51] ...trap/engines/engine.py.229: process: 0.56141 | load: 0.00251 [I 2022-10-27 23:29:51] ...trap/engines/engine.py.230: loss: 0.11521 [I 2022-10-27 23:29:51] ...optimizers/trijoint.py.151: total_norm: 10.634639 [I 2022-10-27 23:29:57] ...trap/engines/engine.py.224: train: epoch 94 | batch 100/2304 [I 2022-10-27 23:29:57] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:24 [I 2022-10-27 23:29:57] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00339 [I 2022-10-27 23:29:57] ...trap/engines/engine.py.230: loss: 0.11962 [I 2022-10-27 23:29:57] ...optimizers/trijoint.py.151: total_norm: 8.818466 [I 2022-10-27 23:30:02] ...trap/engines/engine.py.224: train: epoch 94 | batch 110/2304 [I 2022-10-27 23:30:02] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:34 [I 2022-10-27 23:30:02] ...trap/engines/engine.py.229: process: 0.55114 | load: 0.00260 [I 2022-10-27 23:30:02] ...trap/engines/engine.py.230: loss: 0.24378 [I 2022-10-27 23:30:02] ...optimizers/trijoint.py.151: total_norm: 20.756100 [I 2022-10-27 23:30:08] ...trap/engines/engine.py.224: train: epoch 94 | batch 120/2304 [I 2022-10-27 23:30:08] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:17 [I 2022-10-27 23:30:08] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00290 [I 2022-10-27 23:30:08] ...trap/engines/engine.py.230: loss: 0.15988 [I 2022-10-27 23:30:08] ...optimizers/trijoint.py.151: total_norm: 13.143414 [I 2022-10-27 23:30:13] ...trap/engines/engine.py.224: train: epoch 94 | batch 130/2304 [I 2022-10-27 23:30:13] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:38 [I 2022-10-27 23:30:13] ...trap/engines/engine.py.229: process: 0.58541 | load: 0.00266 [I 2022-10-27 23:30:13] ...trap/engines/engine.py.230: loss: 0.12005 [I 2022-10-27 23:30:13] ...optimizers/trijoint.py.151: total_norm: 26.610910 [I 2022-10-27 23:30:19] ...trap/engines/engine.py.224: train: epoch 94 | batch 140/2304 [I 2022-10-27 23:30:19] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:14 [I 2022-10-27 23:30:19] ...trap/engines/engine.py.229: process: 0.57840 | load: 0.00266 [I 2022-10-27 23:30:19] ...trap/engines/engine.py.230: loss: 0.11643 [I 2022-10-27 23:30:19] ...optimizers/trijoint.py.151: total_norm: 25.230185 [I 2022-10-27 23:30:25] ...trap/engines/engine.py.224: train: epoch 94 | batch 150/2304 [I 2022-10-27 23:30:25] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:08 [I 2022-10-27 23:30:25] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00252 [I 2022-10-27 23:30:25] ...trap/engines/engine.py.230: loss: 0.11644 [I 2022-10-27 23:30:25] ...optimizers/trijoint.py.151: total_norm: 13.949024 [I 2022-10-27 23:30:30] ...trap/engines/engine.py.224: train: epoch 94 | batch 160/2304 [I 2022-10-27 23:30:30] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:54 [I 2022-10-27 23:30:30] ...trap/engines/engine.py.229: process: 0.55845 | load: 0.00265 [I 2022-10-27 23:30:30] ...trap/engines/engine.py.230: loss: 0.12655 [I 2022-10-27 23:30:30] ...optimizers/trijoint.py.151: total_norm: 20.534931 [I 2022-10-27 23:30:36] ...trap/engines/engine.py.224: train: epoch 94 | batch 170/2304 [I 2022-10-27 23:30:36] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:49 [I 2022-10-27 23:30:36] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00264 [I 2022-10-27 23:30:36] ...trap/engines/engine.py.230: loss: 0.09877 [I 2022-10-27 23:30:36] ...optimizers/trijoint.py.151: total_norm: 18.945269 [I 2022-10-27 23:30:41] ...trap/engines/engine.py.224: train: epoch 94 | batch 180/2304 [I 2022-10-27 23:30:41] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:41 [I 2022-10-27 23:30:41] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00256 [I 2022-10-27 23:30:41] ...trap/engines/engine.py.230: loss: 0.18070 [I 2022-10-27 23:30:41] ...optimizers/trijoint.py.151: total_norm: 12.230880 [I 2022-10-27 23:30:47] ...trap/engines/engine.py.224: train: epoch 94 | batch 190/2304 [I 2022-10-27 23:30:47] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:26 [I 2022-10-27 23:30:47] ...trap/engines/engine.py.229: process: 0.54118 | load: 0.00250 [I 2022-10-27 23:30:47] ...trap/engines/engine.py.230: loss: 0.12141 [I 2022-10-27 23:30:47] ...optimizers/trijoint.py.151: total_norm: 34.583096 [I 2022-10-27 23:30:53] ...trap/engines/engine.py.224: train: epoch 94 | batch 200/2304 [I 2022-10-27 23:30:53] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:32 [I 2022-10-27 23:30:53] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00274 [I 2022-10-27 23:30:53] ...trap/engines/engine.py.230: loss: 0.13598 [I 2022-10-27 23:30:53] ...optimizers/trijoint.py.151: total_norm: 15.787415 [I 2022-10-27 23:30:58] ...trap/engines/engine.py.224: train: epoch 94 | batch 210/2304 [I 2022-10-27 23:30:58] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:48 [I 2022-10-27 23:30:58] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00283 [I 2022-10-27 23:30:58] ...trap/engines/engine.py.230: loss: 0.11163 [I 2022-10-27 23:30:58] ...optimizers/trijoint.py.151: total_norm: 14.714778 [I 2022-10-27 23:31:04] ...trap/engines/engine.py.224: train: epoch 94 | batch 220/2304 [I 2022-10-27 23:31:04] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:22 [I 2022-10-27 23:31:04] ...trap/engines/engine.py.229: process: 0.55194 | load: 0.00263 [I 2022-10-27 23:31:04] ...trap/engines/engine.py.230: loss: 0.09556 [I 2022-10-27 23:31:04] ...optimizers/trijoint.py.151: total_norm: 9.189063 [I 2022-10-27 23:31:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 230/2304 [I 2022-10-27 23:31:09] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:19 [I 2022-10-27 23:31:09] ...trap/engines/engine.py.229: process: 0.55189 | load: 0.00245 [I 2022-10-27 23:31:09] ...trap/engines/engine.py.230: loss: 0.19471 [I 2022-10-27 23:31:09] ...optimizers/trijoint.py.151: total_norm: 20.605179 [I 2022-10-27 23:31:15] ...trap/engines/engine.py.224: train: epoch 94 | batch 240/2304 [I 2022-10-27 23:31:15] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:13 [I 2022-10-27 23:31:15] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00257 [I 2022-10-27 23:31:15] ...trap/engines/engine.py.230: loss: 0.11063 [I 2022-10-27 23:31:15] ...optimizers/trijoint.py.151: total_norm: 19.347790 [I 2022-10-27 23:31:21] ...trap/engines/engine.py.224: train: epoch 94 | batch 250/2304 [I 2022-10-27 23:31:21] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:06 [I 2022-10-27 23:31:21] ...trap/engines/engine.py.229: process: 0.55526 | load: 0.00249 [I 2022-10-27 23:31:21] ...trap/engines/engine.py.230: loss: 0.11713 [I 2022-10-27 23:31:21] ...optimizers/trijoint.py.151: total_norm: 17.196768 [I 2022-10-27 23:31:26] ...trap/engines/engine.py.224: train: epoch 94 | batch 260/2304 [I 2022-10-27 23:31:26] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:10 [I 2022-10-27 23:31:26] ...trap/engines/engine.py.229: process: 0.57088 | load: 0.00256 [I 2022-10-27 23:31:26] ...trap/engines/engine.py.230: loss: 0.10617 [I 2022-10-27 23:31:26] ...optimizers/trijoint.py.151: total_norm: 13.410602 [I 2022-10-27 23:31:32] ...trap/engines/engine.py.224: train: epoch 94 | batch 270/2304 [I 2022-10-27 23:31:32] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:53 [I 2022-10-27 23:31:32] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00248 [I 2022-10-27 23:31:32] ...trap/engines/engine.py.230: loss: 0.18644 [I 2022-10-27 23:31:32] ...optimizers/trijoint.py.151: total_norm: 17.101910 [I 2022-10-27 23:31:37] ...trap/engines/engine.py.224: train: epoch 94 | batch 280/2304 [I 2022-10-27 23:31:37] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:59 [I 2022-10-27 23:31:37] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00258 [I 2022-10-27 23:31:37] ...trap/engines/engine.py.230: loss: 0.14585 [I 2022-10-27 23:31:37] ...optimizers/trijoint.py.151: total_norm: 20.620094 [I 2022-10-27 23:31:43] ...trap/engines/engine.py.224: train: epoch 94 | batch 290/2304 [I 2022-10-27 23:31:43] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:43 [I 2022-10-27 23:31:43] ...trap/engines/engine.py.229: process: 0.55461 | load: 0.00261 [I 2022-10-27 23:31:43] ...trap/engines/engine.py.230: loss: 0.13318 [I 2022-10-27 23:31:43] ...optimizers/trijoint.py.151: total_norm: 18.791475 [I 2022-10-27 23:31:49] ...trap/engines/engine.py.224: train: epoch 94 | batch 300/2304 [I 2022-10-27 23:31:49] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:31 [I 2022-10-27 23:31:49] ...trap/engines/engine.py.229: process: 0.54964 | load: 0.00308 [I 2022-10-27 23:31:49] ...trap/engines/engine.py.230: loss: 0.09108 [I 2022-10-27 23:31:49] ...optimizers/trijoint.py.151: total_norm: 64.287949 [I 2022-10-27 23:31:54] ...trap/engines/engine.py.224: train: epoch 94 | batch 310/2304 [I 2022-10-27 23:31:54] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:31 [I 2022-10-27 23:31:54] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00250 [I 2022-10-27 23:31:54] ...trap/engines/engine.py.230: loss: 0.13388 [I 2022-10-27 23:31:54] ...optimizers/trijoint.py.151: total_norm: 21.361147 [I 2022-10-27 23:32:00] ...trap/engines/engine.py.224: train: epoch 94 | batch 320/2304 [I 2022-10-27 23:32:00] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:27 [I 2022-10-27 23:32:00] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00264 [I 2022-10-27 23:32:00] ...trap/engines/engine.py.230: loss: 0.10415 [I 2022-10-27 23:32:00] ...optimizers/trijoint.py.151: total_norm: 9.818366 [I 2022-10-27 23:32:05] ...trap/engines/engine.py.224: train: epoch 94 | batch 330/2304 [I 2022-10-27 23:32:05] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:23 [I 2022-10-27 23:32:05] ...trap/engines/engine.py.229: process: 0.57080 | load: 0.00252 [I 2022-10-27 23:32:05] ...trap/engines/engine.py.230: loss: 0.12130 [I 2022-10-27 23:32:05] ...optimizers/trijoint.py.151: total_norm: 14.887194 [I 2022-10-27 23:32:11] ...trap/engines/engine.py.224: train: epoch 94 | batch 340/2304 [I 2022-10-27 23:32:11] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:23 [I 2022-10-27 23:32:11] ...trap/engines/engine.py.229: process: 0.56557 | load: 0.00260 [I 2022-10-27 23:32:11] ...trap/engines/engine.py.230: loss: 0.14694 [I 2022-10-27 23:32:11] ...optimizers/trijoint.py.151: total_norm: 21.837339 [I 2022-10-27 23:32:17] ...trap/engines/engine.py.224: train: epoch 94 | batch 350/2304 [I 2022-10-27 23:32:17] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:04 [I 2022-10-27 23:32:17] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00250 [I 2022-10-27 23:32:17] ...trap/engines/engine.py.230: loss: 0.10265 [I 2022-10-27 23:32:17] ...optimizers/trijoint.py.151: total_norm: 13.866849 [I 2022-10-27 23:32:22] ...trap/engines/engine.py.224: train: epoch 94 | batch 360/2304 [I 2022-10-27 23:32:22] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:05 [I 2022-10-27 23:32:22] ...trap/engines/engine.py.229: process: 0.56404 | load: 0.00246 [I 2022-10-27 23:32:22] ...trap/engines/engine.py.230: loss: 0.11026 [I 2022-10-27 23:32:22] ...optimizers/trijoint.py.151: total_norm: 17.548101 [I 2022-10-27 23:32:28] ...trap/engines/engine.py.224: train: epoch 94 | batch 370/2304 [I 2022-10-27 23:32:28] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:08 [I 2022-10-27 23:32:28] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00252 [I 2022-10-27 23:32:28] ...trap/engines/engine.py.230: loss: 0.13014 [I 2022-10-27 23:32:28] ...optimizers/trijoint.py.151: total_norm: 23.444187 [I 2022-10-27 23:32:33] ...trap/engines/engine.py.224: train: epoch 94 | batch 380/2304 [I 2022-10-27 23:32:33] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:18:06 [I 2022-10-27 23:32:33] ...trap/engines/engine.py.229: process: 0.56130 | load: 0.00253 [I 2022-10-27 23:32:33] ...trap/engines/engine.py.230: loss: 0.13067 [I 2022-10-27 23:32:33] ...optimizers/trijoint.py.151: total_norm: 14.707513 [I 2022-10-27 23:32:39] ...trap/engines/engine.py.224: train: epoch 94 | batch 390/2304 [I 2022-10-27 23:32:39] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:53 [I 2022-10-27 23:32:39] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00246 [I 2022-10-27 23:32:39] ...trap/engines/engine.py.230: loss: 0.15207 [I 2022-10-27 23:32:39] ...optimizers/trijoint.py.151: total_norm: 24.914673 [I 2022-10-27 23:32:45] ...trap/engines/engine.py.224: train: epoch 94 | batch 400/2304 [I 2022-10-27 23:32:45] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:52 [I 2022-10-27 23:32:45] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00244 [I 2022-10-27 23:32:45] ...trap/engines/engine.py.230: loss: 0.17986 [I 2022-10-27 23:32:45] ...optimizers/trijoint.py.151: total_norm: 23.172535 [I 2022-10-27 23:32:50] ...trap/engines/engine.py.224: train: epoch 94 | batch 410/2304 [I 2022-10-27 23:32:50] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:34 [I 2022-10-27 23:32:50] ...trap/engines/engine.py.229: process: 0.54482 | load: 0.00283 [I 2022-10-27 23:32:50] ...trap/engines/engine.py.230: loss: 0.09186 [I 2022-10-27 23:32:50] ...optimizers/trijoint.py.151: total_norm: 10.253424 [I 2022-10-27 23:32:56] ...trap/engines/engine.py.224: train: epoch 94 | batch 420/2304 [I 2022-10-27 23:32:56] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:28 [I 2022-10-27 23:32:56] ...trap/engines/engine.py.229: process: 0.56156 | load: 0.00258 [I 2022-10-27 23:32:56] ...trap/engines/engine.py.230: loss: 0.08584 [I 2022-10-27 23:32:56] ...optimizers/trijoint.py.151: total_norm: 13.629333 [I 2022-10-27 23:33:02] ...trap/engines/engine.py.224: train: epoch 94 | batch 430/2304 [I 2022-10-27 23:33:02] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:26 [I 2022-10-27 23:33:02] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00264 [I 2022-10-27 23:33:02] ...trap/engines/engine.py.230: loss: 0.12084 [I 2022-10-27 23:33:02] ...optimizers/trijoint.py.151: total_norm: 19.956320 [I 2022-10-27 23:33:07] ...trap/engines/engine.py.224: train: epoch 94 | batch 440/2304 [I 2022-10-27 23:33:07] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:19 [I 2022-10-27 23:33:07] ...trap/engines/engine.py.229: process: 0.56194 | load: 0.00249 [I 2022-10-27 23:33:07] ...trap/engines/engine.py.230: loss: 0.13070 [I 2022-10-27 23:33:07] ...optimizers/trijoint.py.151: total_norm: 12.397568 [I 2022-10-27 23:33:13] ...trap/engines/engine.py.224: train: epoch 94 | batch 450/2304 [I 2022-10-27 23:33:13] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:15 [I 2022-10-27 23:33:13] ...trap/engines/engine.py.229: process: 0.56260 | load: 0.00254 [I 2022-10-27 23:33:13] ...trap/engines/engine.py.230: loss: 0.12335 [I 2022-10-27 23:33:13] ...optimizers/trijoint.py.151: total_norm: 14.034727 [I 2022-10-27 23:33:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 460/2304 [I 2022-10-27 23:33:18] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:00 [I 2022-10-27 23:33:18] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00540 [I 2022-10-27 23:33:18] ...trap/engines/engine.py.230: loss: 0.11274 [I 2022-10-27 23:33:18] ...optimizers/trijoint.py.151: total_norm: 24.937761 [I 2022-10-27 23:33:24] ...trap/engines/engine.py.224: train: epoch 94 | batch 470/2304 [I 2022-10-27 23:33:24] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:00 [I 2022-10-27 23:33:24] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00251 [I 2022-10-27 23:33:24] ...trap/engines/engine.py.230: loss: 0.11105 [I 2022-10-27 23:33:24] ...optimizers/trijoint.py.151: total_norm: 21.116285 [I 2022-10-27 23:33:29] ...trap/engines/engine.py.224: train: epoch 94 | batch 480/2304 [I 2022-10-27 23:33:29] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:52 [I 2022-10-27 23:33:29] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00246 [I 2022-10-27 23:33:29] ...trap/engines/engine.py.230: loss: 0.10474 [I 2022-10-27 23:33:29] ...optimizers/trijoint.py.151: total_norm: 9.022658 [I 2022-10-27 23:33:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 490/2304 [I 2022-10-27 23:33:35] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:47 [I 2022-10-27 23:33:35] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00262 [I 2022-10-27 23:33:35] ...trap/engines/engine.py.230: loss: 0.09866 [I 2022-10-27 23:33:35] ...optimizers/trijoint.py.151: total_norm: 16.761034 [I 2022-10-27 23:33:41] ...trap/engines/engine.py.224: train: epoch 94 | batch 500/2304 [I 2022-10-27 23:33:41] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:43 [I 2022-10-27 23:33:41] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00243 [I 2022-10-27 23:33:41] ...trap/engines/engine.py.230: loss: 0.15298 [I 2022-10-27 23:33:41] ...optimizers/trijoint.py.151: total_norm: 8.211124 [I 2022-10-27 23:33:46] ...trap/engines/engine.py.224: train: epoch 94 | batch 510/2304 [I 2022-10-27 23:33:46] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:48 [I 2022-10-27 23:33:46] ...trap/engines/engine.py.229: process: 0.56324 | load: 0.00298 [I 2022-10-27 23:33:46] ...trap/engines/engine.py.230: loss: 0.07457 [I 2022-10-27 23:33:46] ...optimizers/trijoint.py.151: total_norm: 16.201639 [I 2022-10-27 23:33:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 520/2304 [I 2022-10-27 23:33:52] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:41 [I 2022-10-27 23:33:52] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00279 [I 2022-10-27 23:33:52] ...trap/engines/engine.py.230: loss: 0.16553 [I 2022-10-27 23:33:52] ...optimizers/trijoint.py.151: total_norm: 33.718452 [I 2022-10-27 23:33:57] ...trap/engines/engine.py.224: train: epoch 94 | batch 530/2304 [I 2022-10-27 23:33:57] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:29 [I 2022-10-27 23:33:57] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00258 [I 2022-10-27 23:33:57] ...trap/engines/engine.py.230: loss: 0.13643 [I 2022-10-27 23:33:57] ...optimizers/trijoint.py.151: total_norm: 22.599476 [I 2022-10-27 23:34:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 540/2304 [I 2022-10-27 23:34:03] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:27 [I 2022-10-27 23:34:03] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00244 [I 2022-10-27 23:34:03] ...trap/engines/engine.py.230: loss: 0.09069 [I 2022-10-27 23:34:03] ...optimizers/trijoint.py.151: total_norm: 9.988609 [I 2022-10-27 23:34:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 550/2304 [I 2022-10-27 23:34:09] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:23 [I 2022-10-27 23:34:09] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00246 [I 2022-10-27 23:34:09] ...trap/engines/engine.py.230: loss: 0.09636 [I 2022-10-27 23:34:09] ...optimizers/trijoint.py.151: total_norm: 23.351332 [I 2022-10-27 23:34:14] ...trap/engines/engine.py.224: train: epoch 94 | batch 560/2304 [I 2022-10-27 23:34:14] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:17 [I 2022-10-27 23:34:14] ...trap/engines/engine.py.229: process: 0.56477 | load: 0.00245 [I 2022-10-27 23:34:14] ...trap/engines/engine.py.230: loss: 0.18727 [I 2022-10-27 23:34:14] ...optimizers/trijoint.py.151: total_norm: 22.975708 [I 2022-10-27 23:34:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 570/2304 [I 2022-10-27 23:34:20] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:14 [I 2022-10-27 23:34:20] ...trap/engines/engine.py.229: process: 0.56454 | load: 0.00246 [I 2022-10-27 23:34:20] ...trap/engines/engine.py.230: loss: 0.10821 [I 2022-10-27 23:34:20] ...optimizers/trijoint.py.151: total_norm: 9.069366 [I 2022-10-27 23:34:25] ...trap/engines/engine.py.224: train: epoch 94 | batch 580/2304 [I 2022-10-27 23:34:25] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:09 [I 2022-10-27 23:34:25] ...trap/engines/engine.py.229: process: 0.56461 | load: 0.00243 [I 2022-10-27 23:34:25] ...trap/engines/engine.py.230: loss: 0.07908 [I 2022-10-27 23:34:25] ...optimizers/trijoint.py.151: total_norm: 8.660289 [I 2022-10-27 23:34:31] ...trap/engines/engine.py.224: train: epoch 94 | batch 590/2304 [I 2022-10-27 23:34:31] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:54 [I 2022-10-27 23:34:31] ...trap/engines/engine.py.229: process: 0.54865 | load: 0.00252 [I 2022-10-27 23:34:31] ...trap/engines/engine.py.230: loss: 0.13659 [I 2022-10-27 23:34:31] ...optimizers/trijoint.py.151: total_norm: 12.223896 [I 2022-10-27 23:34:37] ...trap/engines/engine.py.224: train: epoch 94 | batch 600/2304 [I 2022-10-27 23:34:37] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:56 [I 2022-10-27 23:34:37] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00247 [I 2022-10-27 23:34:37] ...trap/engines/engine.py.230: loss: 0.16415 [I 2022-10-27 23:34:37] ...optimizers/trijoint.py.151: total_norm: 12.099895 [I 2022-10-27 23:34:42] ...trap/engines/engine.py.224: train: epoch 94 | batch 610/2304 [I 2022-10-27 23:34:42] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:45 [I 2022-10-27 23:34:42] ...trap/engines/engine.py.229: process: 0.55556 | load: 0.00246 [I 2022-10-27 23:34:42] ...trap/engines/engine.py.230: loss: 0.15889 [I 2022-10-27 23:34:42] ...optimizers/trijoint.py.151: total_norm: 12.171189 [I 2022-10-27 23:34:48] ...trap/engines/engine.py.224: train: epoch 94 | batch 620/2304 [I 2022-10-27 23:34:48] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:42 [I 2022-10-27 23:34:48] ...trap/engines/engine.py.229: process: 0.56340 | load: 0.00246 [I 2022-10-27 23:34:48] ...trap/engines/engine.py.230: loss: 0.11558 [I 2022-10-27 23:34:48] ...optimizers/trijoint.py.151: total_norm: 12.158628 [I 2022-10-27 23:34:53] ...trap/engines/engine.py.224: train: epoch 94 | batch 630/2304 [I 2022-10-27 23:34:53] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:41 [I 2022-10-27 23:34:53] ...trap/engines/engine.py.229: process: 0.56370 | load: 0.00245 [I 2022-10-27 23:34:53] ...trap/engines/engine.py.230: loss: 0.15635 [I 2022-10-27 23:34:53] ...optimizers/trijoint.py.151: total_norm: 16.741449 [I 2022-10-27 23:34:59] ...trap/engines/engine.py.224: train: epoch 94 | batch 640/2304 [I 2022-10-27 23:34:59] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:36 [I 2022-10-27 23:34:59] ...trap/engines/engine.py.229: process: 0.56759 | load: 0.00245 [I 2022-10-27 23:34:59] ...trap/engines/engine.py.230: loss: 0.14043 [I 2022-10-27 23:34:59] ...optimizers/trijoint.py.151: total_norm: 13.367872 [I 2022-10-27 23:35:05] ...trap/engines/engine.py.224: train: epoch 94 | batch 650/2304 [I 2022-10-27 23:35:05] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:24 [I 2022-10-27 23:35:05] ...trap/engines/engine.py.229: process: 0.55307 | load: 0.00242 [I 2022-10-27 23:35:05] ...trap/engines/engine.py.230: loss: 0.13338 [I 2022-10-27 23:35:05] ...optimizers/trijoint.py.151: total_norm: 16.268387 [I 2022-10-27 23:35:10] ...trap/engines/engine.py.224: train: epoch 94 | batch 660/2304 [I 2022-10-27 23:35:10] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:23 [I 2022-10-27 23:35:10] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00243 [I 2022-10-27 23:35:10] ...trap/engines/engine.py.230: loss: 0.10615 [I 2022-10-27 23:35:10] ...optimizers/trijoint.py.151: total_norm: 15.890857 [I 2022-10-27 23:35:16] ...trap/engines/engine.py.224: train: epoch 94 | batch 670/2304 [I 2022-10-27 23:35:16] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:19 [I 2022-10-27 23:35:16] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00244 [I 2022-10-27 23:35:16] ...trap/engines/engine.py.230: loss: 0.12041 [I 2022-10-27 23:35:16] ...optimizers/trijoint.py.151: total_norm: 22.382906 [I 2022-10-27 23:35:22] ...trap/engines/engine.py.224: train: epoch 94 | batch 680/2304 [I 2022-10-27 23:35:22] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:10 [I 2022-10-27 23:35:22] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00243 [I 2022-10-27 23:35:22] ...trap/engines/engine.py.230: loss: 0.10366 [I 2022-10-27 23:35:22] ...optimizers/trijoint.py.151: total_norm: 9.405115 [I 2022-10-27 23:35:27] ...trap/engines/engine.py.224: train: epoch 94 | batch 690/2304 [I 2022-10-27 23:35:27] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:00 [I 2022-10-27 23:35:27] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00246 [I 2022-10-27 23:35:27] ...trap/engines/engine.py.230: loss: 0.12988 [I 2022-10-27 23:35:27] ...optimizers/trijoint.py.151: total_norm: 16.390163 [I 2022-10-27 23:35:33] ...trap/engines/engine.py.224: train: epoch 94 | batch 700/2304 [I 2022-10-27 23:35:33] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:59 [I 2022-10-27 23:35:33] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00244 [I 2022-10-27 23:35:33] ...trap/engines/engine.py.230: loss: 0.12405 [I 2022-10-27 23:35:33] ...optimizers/trijoint.py.151: total_norm: 11.963464 [I 2022-10-27 23:35:38] ...trap/engines/engine.py.224: train: epoch 94 | batch 710/2304 [I 2022-10-27 23:35:38] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:54 [I 2022-10-27 23:35:38] ...trap/engines/engine.py.229: process: 0.56430 | load: 0.00253 [I 2022-10-27 23:35:38] ...trap/engines/engine.py.230: loss: 0.09671 [I 2022-10-27 23:35:38] ...optimizers/trijoint.py.151: total_norm: 20.232042 [I 2022-10-27 23:35:44] ...trap/engines/engine.py.224: train: epoch 94 | batch 720/2304 [I 2022-10-27 23:35:44] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:49 [I 2022-10-27 23:35:44] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00249 [I 2022-10-27 23:35:44] ...trap/engines/engine.py.230: loss: 0.19012 [I 2022-10-27 23:35:44] ...optimizers/trijoint.py.151: total_norm: 16.323021 [I 2022-10-27 23:35:50] ...trap/engines/engine.py.224: train: epoch 94 | batch 730/2304 [I 2022-10-27 23:35:50] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:37 [I 2022-10-27 23:35:50] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00248 [I 2022-10-27 23:35:50] ...trap/engines/engine.py.230: loss: 0.14251 [I 2022-10-27 23:35:50] ...optimizers/trijoint.py.151: total_norm: 35.298939 [I 2022-10-27 23:35:55] ...trap/engines/engine.py.224: train: epoch 94 | batch 740/2304 [I 2022-10-27 23:35:55] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:31 [I 2022-10-27 23:35:55] ...trap/engines/engine.py.229: process: 0.54943 | load: 0.00247 [I 2022-10-27 23:35:55] ...trap/engines/engine.py.230: loss: 0.09866 [I 2022-10-27 23:35:55] ...optimizers/trijoint.py.151: total_norm: 12.386756 [I 2022-10-27 23:36:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 750/2304 [I 2022-10-27 23:36:01] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:31 [I 2022-10-27 23:36:01] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00245 [I 2022-10-27 23:36:01] ...trap/engines/engine.py.230: loss: 0.10598 [I 2022-10-27 23:36:01] ...optimizers/trijoint.py.151: total_norm: 12.861112 [I 2022-10-27 23:36:07] ...trap/engines/engine.py.224: train: epoch 94 | batch 760/2304 [I 2022-10-27 23:36:07] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:24 [I 2022-10-27 23:36:07] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00244 [I 2022-10-27 23:36:07] ...trap/engines/engine.py.230: loss: 0.13259 [I 2022-10-27 23:36:07] ...optimizers/trijoint.py.151: total_norm: 23.024752 [I 2022-10-27 23:36:12] ...trap/engines/engine.py.224: train: epoch 94 | batch 770/2304 [I 2022-10-27 23:36:12] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:19 [I 2022-10-27 23:36:12] ...trap/engines/engine.py.229: process: 0.56181 | load: 0.00246 [I 2022-10-27 23:36:12] ...trap/engines/engine.py.230: loss: 0.10698 [I 2022-10-27 23:36:12] ...optimizers/trijoint.py.151: total_norm: 11.781864 [I 2022-10-27 23:36:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 780/2304 [I 2022-10-27 23:36:18] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:06 [I 2022-10-27 23:36:18] ...trap/engines/engine.py.229: process: 0.55067 | load: 0.00252 [I 2022-10-27 23:36:18] ...trap/engines/engine.py.230: loss: 0.14972 [I 2022-10-27 23:36:18] ...optimizers/trijoint.py.151: total_norm: 14.940770 [I 2022-10-27 23:36:23] ...trap/engines/engine.py.224: train: epoch 94 | batch 790/2304 [I 2022-10-27 23:36:23] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:13 [I 2022-10-27 23:36:23] ...trap/engines/engine.py.229: process: 0.57653 | load: 0.00250 [I 2022-10-27 23:36:23] ...trap/engines/engine.py.230: loss: 0.11446 [I 2022-10-27 23:36:23] ...optimizers/trijoint.py.151: total_norm: 7.680644 [I 2022-10-27 23:36:29] ...trap/engines/engine.py.224: train: epoch 94 | batch 800/2304 [I 2022-10-27 23:36:29] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:14:12 [I 2022-10-27 23:36:29] ...trap/engines/engine.py.229: process: 0.57034 | load: 0.00252 [I 2022-10-27 23:36:29] ...trap/engines/engine.py.230: loss: 0.13329 [I 2022-10-27 23:36:29] ...optimizers/trijoint.py.151: total_norm: 18.176022 [I 2022-10-27 23:36:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 810/2304 [I 2022-10-27 23:36:35] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:55 [I 2022-10-27 23:36:35] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00250 [I 2022-10-27 23:36:35] ...trap/engines/engine.py.230: loss: 0.16520 [I 2022-10-27 23:36:35] ...optimizers/trijoint.py.151: total_norm: 40.585293 [I 2022-10-27 23:36:40] ...trap/engines/engine.py.224: train: epoch 94 | batch 820/2304 [I 2022-10-27 23:36:40] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:54 [I 2022-10-27 23:36:40] ...trap/engines/engine.py.229: process: 0.56182 | load: 0.00245 [I 2022-10-27 23:36:40] ...trap/engines/engine.py.230: loss: 0.15457 [I 2022-10-27 23:36:40] ...optimizers/trijoint.py.151: total_norm: 15.553180 [I 2022-10-27 23:36:46] ...trap/engines/engine.py.224: train: epoch 94 | batch 830/2304 [I 2022-10-27 23:36:46] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:40 [I 2022-10-27 23:36:46] ...trap/engines/engine.py.229: process: 0.54803 | load: 0.00243 [I 2022-10-27 23:36:46] ...trap/engines/engine.py.230: loss: 0.15864 [I 2022-10-27 23:36:46] ...optimizers/trijoint.py.151: total_norm: 10.983819 [I 2022-10-27 23:36:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 840/2304 [I 2022-10-27 23:36:52] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:36 [I 2022-10-27 23:36:52] ...trap/engines/engine.py.229: process: 0.56149 | load: 0.00251 [I 2022-10-27 23:36:52] ...trap/engines/engine.py.230: loss: 0.12497 [I 2022-10-27 23:36:52] ...optimizers/trijoint.py.151: total_norm: 20.322233 [I 2022-10-27 23:36:57] ...trap/engines/engine.py.224: train: epoch 94 | batch 850/2304 [I 2022-10-27 23:36:57] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:32 [I 2022-10-27 23:36:57] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00246 [I 2022-10-27 23:36:57] ...trap/engines/engine.py.230: loss: 0.18472 [I 2022-10-27 23:36:57] ...optimizers/trijoint.py.151: total_norm: 41.689278 [I 2022-10-27 23:37:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 860/2304 [I 2022-10-27 23:37:03] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:24 [I 2022-10-27 23:37:03] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00251 [I 2022-10-27 23:37:03] ...trap/engines/engine.py.230: loss: 0.10702 [I 2022-10-27 23:37:03] ...optimizers/trijoint.py.151: total_norm: 13.242112 [I 2022-10-27 23:37:08] ...trap/engines/engine.py.224: train: epoch 94 | batch 870/2304 [I 2022-10-27 23:37:08] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:22 [I 2022-10-27 23:37:08] ...trap/engines/engine.py.229: process: 0.56070 | load: 0.00242 [I 2022-10-27 23:37:08] ...trap/engines/engine.py.230: loss: 0.10809 [I 2022-10-27 23:37:08] ...optimizers/trijoint.py.151: total_norm: 21.578924 [I 2022-10-27 23:37:14] ...trap/engines/engine.py.224: train: epoch 94 | batch 880/2304 [I 2022-10-27 23:37:14] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:19 [I 2022-10-27 23:37:14] ...trap/engines/engine.py.229: process: 0.56263 | load: 0.00246 [I 2022-10-27 23:37:14] ...trap/engines/engine.py.230: loss: 0.14759 [I 2022-10-27 23:37:14] ...optimizers/trijoint.py.151: total_norm: 34.407341 [I 2022-10-27 23:37:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 890/2304 [I 2022-10-27 23:37:20] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:17 [I 2022-10-27 23:37:20] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00250 [I 2022-10-27 23:37:20] ...trap/engines/engine.py.230: loss: 0.17460 [I 2022-10-27 23:37:20] ...optimizers/trijoint.py.151: total_norm: 17.606499 [I 2022-10-27 23:37:25] ...trap/engines/engine.py.224: train: epoch 94 | batch 900/2304 [I 2022-10-27 23:37:25] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:09 [I 2022-10-27 23:37:25] ...trap/engines/engine.py.229: process: 0.56574 | load: 0.00246 [I 2022-10-27 23:37:25] ...trap/engines/engine.py.230: loss: 0.16404 [I 2022-10-27 23:37:25] ...optimizers/trijoint.py.151: total_norm: 31.724901 [I 2022-10-27 23:37:31] ...trap/engines/engine.py.224: train: epoch 94 | batch 910/2304 [I 2022-10-27 23:37:31] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:59 [I 2022-10-27 23:37:31] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00273 [I 2022-10-27 23:37:31] ...trap/engines/engine.py.230: loss: 0.14541 [I 2022-10-27 23:37:31] ...optimizers/trijoint.py.151: total_norm: 14.316755 [I 2022-10-27 23:37:36] ...trap/engines/engine.py.224: train: epoch 94 | batch 920/2304 [I 2022-10-27 23:37:36] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:58 [I 2022-10-27 23:37:36] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00256 [I 2022-10-27 23:37:36] ...trap/engines/engine.py.230: loss: 0.21789 [I 2022-10-27 23:37:36] ...optimizers/trijoint.py.151: total_norm: 15.271249 [I 2022-10-27 23:37:42] ...trap/engines/engine.py.224: train: epoch 94 | batch 930/2304 [I 2022-10-27 23:37:42] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:12:48 [I 2022-10-27 23:37:42] ...trap/engines/engine.py.229: process: 0.56179 | load: 0.00259 [I 2022-10-27 23:37:42] ...trap/engines/engine.py.230: loss: 0.11439 [I 2022-10-27 23:37:42] ...optimizers/trijoint.py.151: total_norm: 15.659707 [I 2022-10-27 23:37:48] ...trap/engines/engine.py.224: train: epoch 94 | batch 940/2304 [I 2022-10-27 23:37:48] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:42 [I 2022-10-27 23:37:48] ...trap/engines/engine.py.229: process: 0.54820 | load: 0.00247 [I 2022-10-27 23:37:48] ...trap/engines/engine.py.230: loss: 0.11483 [I 2022-10-27 23:37:48] ...optimizers/trijoint.py.151: total_norm: 25.137705 [I 2022-10-27 23:37:53] ...trap/engines/engine.py.224: train: epoch 94 | batch 950/2304 [I 2022-10-27 23:37:53] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:37 [I 2022-10-27 23:37:53] ...trap/engines/engine.py.229: process: 0.55381 | load: 0.00247 [I 2022-10-27 23:37:53] ...trap/engines/engine.py.230: loss: 0.17354 [I 2022-10-27 23:37:53] ...optimizers/trijoint.py.151: total_norm: 9.055914 [I 2022-10-27 23:37:59] ...trap/engines/engine.py.224: train: epoch 94 | batch 960/2304 [I 2022-10-27 23:37:59] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:40 [I 2022-10-27 23:37:59] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00264 [I 2022-10-27 23:37:59] ...trap/engines/engine.py.230: loss: 0.12602 [I 2022-10-27 23:37:59] ...optimizers/trijoint.py.151: total_norm: 12.044528 [I 2022-10-27 23:38:05] ...trap/engines/engine.py.224: train: epoch 94 | batch 970/2304 [I 2022-10-27 23:38:05] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:28 [I 2022-10-27 23:38:05] ...trap/engines/engine.py.229: process: 0.55543 | load: 0.00261 [I 2022-10-27 23:38:05] ...trap/engines/engine.py.230: loss: 0.16523 [I 2022-10-27 23:38:05] ...optimizers/trijoint.py.151: total_norm: 20.363686 [I 2022-10-27 23:38:10] ...trap/engines/engine.py.224: train: epoch 94 | batch 980/2304 [I 2022-10-27 23:38:10] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:12:20 [I 2022-10-27 23:38:10] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00248 [I 2022-10-27 23:38:10] ...trap/engines/engine.py.230: loss: 0.08730 [I 2022-10-27 23:38:10] ...optimizers/trijoint.py.151: total_norm: 19.244282 [I 2022-10-27 23:38:16] ...trap/engines/engine.py.224: train: epoch 94 | batch 990/2304 [I 2022-10-27 23:38:16] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:10 [I 2022-10-27 23:38:16] ...trap/engines/engine.py.229: process: 0.54987 | load: 0.00247 [I 2022-10-27 23:38:16] ...trap/engines/engine.py.230: loss: 0.16945 [I 2022-10-27 23:38:16] ...optimizers/trijoint.py.151: total_norm: 15.583220 [I 2022-10-27 23:38:21] ...trap/engines/engine.py.224: train: epoch 94 | batch 1000/2304 [I 2022-10-27 23:38:21] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:03 [I 2022-10-27 23:38:21] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00243 [I 2022-10-27 23:38:21] ...trap/engines/engine.py.230: loss: 0.16290 [I 2022-10-27 23:38:21] ...optimizers/trijoint.py.151: total_norm: 18.005522 [I 2022-10-27 23:38:27] ...trap/engines/engine.py.224: train: epoch 94 | batch 1010/2304 [I 2022-10-27 23:38:27] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:12:03 [I 2022-10-27 23:38:27] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00243 [I 2022-10-27 23:38:27] ...trap/engines/engine.py.230: loss: 0.08922 [I 2022-10-27 23:38:27] ...optimizers/trijoint.py.151: total_norm: 9.136884 [I 2022-10-27 23:38:33] ...trap/engines/engine.py.224: train: epoch 94 | batch 1020/2304 [I 2022-10-27 23:38:33] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:59 [I 2022-10-27 23:38:33] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00245 [I 2022-10-27 23:38:33] ...trap/engines/engine.py.230: loss: 0.10773 [I 2022-10-27 23:38:33] ...optimizers/trijoint.py.151: total_norm: 15.418435 [I 2022-10-27 23:38:38] ...trap/engines/engine.py.224: train: epoch 94 | batch 1030/2304 [I 2022-10-27 23:38:38] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:11:58 [I 2022-10-27 23:38:38] ...trap/engines/engine.py.229: process: 0.56395 | load: 0.00245 [I 2022-10-27 23:38:38] ...trap/engines/engine.py.230: loss: 0.10356 [I 2022-10-27 23:38:38] ...optimizers/trijoint.py.151: total_norm: 7.990987 [I 2022-10-27 23:38:44] ...trap/engines/engine.py.224: train: epoch 94 | batch 1040/2304 [I 2022-10-27 23:38:44] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:43 [I 2022-10-27 23:38:44] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00250 [I 2022-10-27 23:38:44] ...trap/engines/engine.py.230: loss: 0.09755 [I 2022-10-27 23:38:44] ...optimizers/trijoint.py.151: total_norm: 33.393875 [I 2022-10-27 23:38:49] ...trap/engines/engine.py.224: train: epoch 94 | batch 1050/2304 [I 2022-10-27 23:38:49] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:44 [I 2022-10-27 23:38:49] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00246 [I 2022-10-27 23:38:49] ...trap/engines/engine.py.230: loss: 0.09671 [I 2022-10-27 23:38:49] ...optimizers/trijoint.py.151: total_norm: 20.558466 [I 2022-10-27 23:38:55] ...trap/engines/engine.py.224: train: epoch 94 | batch 1060/2304 [I 2022-10-27 23:38:55] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:54 [I 2022-10-27 23:38:55] ...trap/engines/engine.py.229: process: 0.56787 | load: 0.00245 [I 2022-10-27 23:38:55] ...trap/engines/engine.py.230: loss: 0.11157 [I 2022-10-27 23:38:55] ...optimizers/trijoint.py.151: total_norm: 8.712570 [I 2022-10-27 23:39:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 1070/2304 [I 2022-10-27 23:39:01] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:40 [I 2022-10-27 23:39:01] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00259 [I 2022-10-27 23:39:01] ...trap/engines/engine.py.230: loss: 0.07555 [I 2022-10-27 23:39:01] ...optimizers/trijoint.py.151: total_norm: 13.175731 [I 2022-10-27 23:39:07] ...trap/engines/engine.py.224: train: epoch 94 | batch 1080/2304 [I 2022-10-27 23:39:07] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:34 [I 2022-10-27 23:39:07] ...trap/engines/engine.py.229: process: 0.55036 | load: 0.00247 [I 2022-10-27 23:39:07] ...trap/engines/engine.py.230: loss: 0.14680 [I 2022-10-27 23:39:07] ...optimizers/trijoint.py.151: total_norm: 32.585014 [I 2022-10-27 23:39:12] ...trap/engines/engine.py.224: train: epoch 94 | batch 1090/2304 [I 2022-10-27 23:39:12] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:28 [I 2022-10-27 23:39:12] ...trap/engines/engine.py.229: process: 0.56859 | load: 0.00243 [I 2022-10-27 23:39:12] ...trap/engines/engine.py.230: loss: 0.12193 [I 2022-10-27 23:39:12] ...optimizers/trijoint.py.151: total_norm: 13.956555 [I 2022-10-27 23:39:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 1100/2304 [I 2022-10-27 23:39:18] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:20 [I 2022-10-27 23:39:18] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00247 [I 2022-10-27 23:39:18] ...trap/engines/engine.py.230: loss: 0.09695 [I 2022-10-27 23:39:18] ...optimizers/trijoint.py.151: total_norm: 11.848453 [I 2022-10-27 23:39:24] ...trap/engines/engine.py.224: train: epoch 94 | batch 1110/2304 [I 2022-10-27 23:39:24] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:24 [I 2022-10-27 23:39:24] ...trap/engines/engine.py.229: process: 0.57187 | load: 0.00245 [I 2022-10-27 23:39:24] ...trap/engines/engine.py.230: loss: 0.11088 [I 2022-10-27 23:39:24] ...optimizers/trijoint.py.151: total_norm: 15.106946 [I 2022-10-27 23:39:29] ...trap/engines/engine.py.224: train: epoch 94 | batch 1120/2304 [I 2022-10-27 23:39:29] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:07 [I 2022-10-27 23:39:29] ...trap/engines/engine.py.229: process: 0.55345 | load: 0.00245 [I 2022-10-27 23:39:29] ...trap/engines/engine.py.230: loss: 0.15385 [I 2022-10-27 23:39:29] ...optimizers/trijoint.py.151: total_norm: 18.190479 [I 2022-10-27 23:39:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 1130/2304 [I 2022-10-27 23:39:35] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:10:57 [I 2022-10-27 23:39:35] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00246 [I 2022-10-27 23:39:35] ...trap/engines/engine.py.230: loss: 0.17182 [I 2022-10-27 23:39:35] ...optimizers/trijoint.py.151: total_norm: 15.163462 [I 2022-10-27 23:39:41] ...trap/engines/engine.py.224: train: epoch 94 | batch 1140/2304 [I 2022-10-27 23:39:41] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:58 [I 2022-10-27 23:39:41] ...trap/engines/engine.py.229: process: 0.56512 | load: 0.00267 [I 2022-10-27 23:39:41] ...trap/engines/engine.py.230: loss: 0.07804 [I 2022-10-27 23:39:41] ...optimizers/trijoint.py.151: total_norm: 12.038085 [I 2022-10-27 23:39:46] ...trap/engines/engine.py.224: train: epoch 94 | batch 1150/2304 [I 2022-10-27 23:39:46] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:51 [I 2022-10-27 23:39:46] ...trap/engines/engine.py.229: process: 0.55356 | load: 0.00255 [I 2022-10-27 23:39:46] ...trap/engines/engine.py.230: loss: 0.08786 [I 2022-10-27 23:39:46] ...optimizers/trijoint.py.151: total_norm: 15.008209 [I 2022-10-27 23:39:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 1160/2304 [I 2022-10-27 23:39:52] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:42 [I 2022-10-27 23:39:52] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00255 [I 2022-10-27 23:39:52] ...trap/engines/engine.py.230: loss: 0.16529 [I 2022-10-27 23:39:52] ...optimizers/trijoint.py.151: total_norm: 29.608170 [I 2022-10-27 23:39:58] ...trap/engines/engine.py.224: train: epoch 94 | batch 1170/2304 [I 2022-10-27 23:39:58] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:41 [I 2022-10-27 23:39:58] ...trap/engines/engine.py.229: process: 0.54946 | load: 0.00250 [I 2022-10-27 23:39:58] ...trap/engines/engine.py.230: loss: 0.09771 [I 2022-10-27 23:39:58] ...optimizers/trijoint.py.151: total_norm: 12.339322 [I 2022-10-27 23:40:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 1180/2304 [I 2022-10-27 23:40:03] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:29 [I 2022-10-27 23:40:03] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00253 [I 2022-10-27 23:40:03] ...trap/engines/engine.py.230: loss: 0.12554 [I 2022-10-27 23:40:03] ...optimizers/trijoint.py.151: total_norm: 13.423988 [I 2022-10-27 23:40:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 1190/2304 [I 2022-10-27 23:40:09] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:22 [I 2022-10-27 23:40:09] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00244 [I 2022-10-27 23:40:09] ...trap/engines/engine.py.230: loss: 0.12786 [I 2022-10-27 23:40:09] ...optimizers/trijoint.py.151: total_norm: 11.305983 [I 2022-10-27 23:40:14] ...trap/engines/engine.py.224: train: epoch 94 | batch 1200/2304 [I 2022-10-27 23:40:14] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:11 [I 2022-10-27 23:40:14] ...trap/engines/engine.py.229: process: 0.54507 | load: 0.00247 [I 2022-10-27 23:40:14] ...trap/engines/engine.py.230: loss: 0.12127 [I 2022-10-27 23:40:14] ...optimizers/trijoint.py.151: total_norm: 7.620891 [I 2022-10-27 23:40:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 1210/2304 [I 2022-10-27 23:40:20] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:10 [I 2022-10-27 23:40:20] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00247 [I 2022-10-27 23:40:20] ...trap/engines/engine.py.230: loss: 0.13554 [I 2022-10-27 23:40:20] ...optimizers/trijoint.py.151: total_norm: 13.965493 [I 2022-10-27 23:40:26] ...trap/engines/engine.py.224: train: epoch 94 | batch 1220/2304 [I 2022-10-27 23:40:26] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:07 [I 2022-10-27 23:40:26] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00243 [I 2022-10-27 23:40:26] ...trap/engines/engine.py.230: loss: 0.25555 [I 2022-10-27 23:40:26] ...optimizers/trijoint.py.151: total_norm: 23.010332 [I 2022-10-27 23:40:31] ...trap/engines/engine.py.224: train: epoch 94 | batch 1230/2304 [I 2022-10-27 23:40:31] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:01 [I 2022-10-27 23:40:31] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00249 [I 2022-10-27 23:40:31] ...trap/engines/engine.py.230: loss: 0.12595 [I 2022-10-27 23:40:31] ...optimizers/trijoint.py.151: total_norm: 23.915600 [I 2022-10-27 23:40:37] ...trap/engines/engine.py.224: train: epoch 94 | batch 1240/2304 [I 2022-10-27 23:40:37] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:09:59 [I 2022-10-27 23:40:37] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00244 [I 2022-10-27 23:40:37] ...trap/engines/engine.py.230: loss: 0.15469 [I 2022-10-27 23:40:37] ...optimizers/trijoint.py.151: total_norm: 12.300290 [I 2022-10-27 23:40:42] ...trap/engines/engine.py.224: train: epoch 94 | batch 1250/2304 [I 2022-10-27 23:40:42] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:09:49 [I 2022-10-27 23:40:42] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00245 [I 2022-10-27 23:40:42] ...trap/engines/engine.py.230: loss: 0.11776 [I 2022-10-27 23:40:42] ...optimizers/trijoint.py.151: total_norm: 8.334159 [I 2022-10-27 23:40:48] ...trap/engines/engine.py.224: train: epoch 94 | batch 1260/2304 [I 2022-10-27 23:40:48] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:43 [I 2022-10-27 23:40:48] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00239 [I 2022-10-27 23:40:48] ...trap/engines/engine.py.230: loss: 0.15339 [I 2022-10-27 23:40:48] ...optimizers/trijoint.py.151: total_norm: 11.664624 [I 2022-10-27 23:40:54] ...trap/engines/engine.py.224: train: epoch 94 | batch 1270/2304 [I 2022-10-27 23:40:54] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:42 [I 2022-10-27 23:40:54] ...trap/engines/engine.py.229: process: 0.56286 | load: 0.00270 [I 2022-10-27 23:40:54] ...trap/engines/engine.py.230: loss: 0.13989 [I 2022-10-27 23:40:54] ...optimizers/trijoint.py.151: total_norm: 19.965731 [I 2022-10-27 23:40:59] ...trap/engines/engine.py.224: train: epoch 94 | batch 1280/2304 [I 2022-10-27 23:40:59] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:39 [I 2022-10-27 23:40:59] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00255 [I 2022-10-27 23:40:59] ...trap/engines/engine.py.230: loss: 0.20988 [I 2022-10-27 23:40:59] ...optimizers/trijoint.py.151: total_norm: 11.595320 [I 2022-10-27 23:41:05] ...trap/engines/engine.py.224: train: epoch 94 | batch 1290/2304 [I 2022-10-27 23:41:05] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:35 [I 2022-10-27 23:41:05] ...trap/engines/engine.py.229: process: 0.55857 | load: 0.00254 [I 2022-10-27 23:41:05] ...trap/engines/engine.py.230: loss: 0.10060 [I 2022-10-27 23:41:05] ...optimizers/trijoint.py.151: total_norm: 15.547567 [I 2022-10-27 23:41:11] ...trap/engines/engine.py.224: train: epoch 94 | batch 1300/2304 [I 2022-10-27 23:41:11] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:21 [I 2022-10-27 23:41:11] ...trap/engines/engine.py.229: process: 0.54798 | load: 0.00254 [I 2022-10-27 23:41:11] ...trap/engines/engine.py.230: loss: 0.14256 [I 2022-10-27 23:41:11] ...optimizers/trijoint.py.151: total_norm: 10.278274 [I 2022-10-27 23:41:16] ...trap/engines/engine.py.224: train: epoch 94 | batch 1310/2304 [I 2022-10-27 23:41:16] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:14 [I 2022-10-27 23:41:16] ...trap/engines/engine.py.229: process: 0.55010 | load: 0.00254 [I 2022-10-27 23:41:16] ...trap/engines/engine.py.230: loss: 0.13427 [I 2022-10-27 23:41:16] ...optimizers/trijoint.py.151: total_norm: 19.372698 [I 2022-10-27 23:41:22] ...trap/engines/engine.py.224: train: epoch 94 | batch 1320/2304 [I 2022-10-27 23:41:22] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:08 [I 2022-10-27 23:41:22] ...trap/engines/engine.py.229: process: 0.56605 | load: 0.00255 [I 2022-10-27 23:41:22] ...trap/engines/engine.py.230: loss: 0.11388 [I 2022-10-27 23:41:22] ...optimizers/trijoint.py.151: total_norm: 19.018785 [I 2022-10-27 23:41:28] ...trap/engines/engine.py.224: train: epoch 94 | batch 1330/2304 [I 2022-10-27 23:41:28] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:05 [I 2022-10-27 23:41:28] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00277 [I 2022-10-27 23:41:28] ...trap/engines/engine.py.230: loss: 0.13602 [I 2022-10-27 23:41:28] ...optimizers/trijoint.py.151: total_norm: 18.907736 [I 2022-10-27 23:41:33] ...trap/engines/engine.py.224: train: epoch 94 | batch 1340/2304 [I 2022-10-27 23:41:33] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:06 [I 2022-10-27 23:41:33] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00267 [I 2022-10-27 23:41:33] ...trap/engines/engine.py.230: loss: 0.11635 [I 2022-10-27 23:41:33] ...optimizers/trijoint.py.151: total_norm: 18.453157 [I 2022-10-27 23:41:39] ...trap/engines/engine.py.224: train: epoch 94 | batch 1350/2304 [I 2022-10-27 23:41:39] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:08:58 [I 2022-10-27 23:41:39] ...trap/engines/engine.py.229: process: 0.57168 | load: 0.00253 [I 2022-10-27 23:41:39] ...trap/engines/engine.py.230: loss: 0.22349 [I 2022-10-27 23:41:39] ...optimizers/trijoint.py.151: total_norm: 28.484446 [I 2022-10-27 23:41:45] ...trap/engines/engine.py.224: train: epoch 94 | batch 1360/2304 [I 2022-10-27 23:41:45] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:08:48 [I 2022-10-27 23:41:45] ...trap/engines/engine.py.229: process: 0.54660 | load: 0.00250 [I 2022-10-27 23:41:45] ...trap/engines/engine.py.230: loss: 0.09140 [I 2022-10-27 23:41:45] ...optimizers/trijoint.py.151: total_norm: 11.628189 [I 2022-10-27 23:41:50] ...trap/engines/engine.py.224: train: epoch 94 | batch 1370/2304 [I 2022-10-27 23:41:50] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:40 [I 2022-10-27 23:41:50] ...trap/engines/engine.py.229: process: 0.54508 | load: 0.00251 [I 2022-10-27 23:41:50] ...trap/engines/engine.py.230: loss: 0.12440 [I 2022-10-27 23:41:50] ...optimizers/trijoint.py.151: total_norm: 15.670074 [I 2022-10-27 23:41:56] ...trap/engines/engine.py.224: train: epoch 94 | batch 1380/2304 [I 2022-10-27 23:41:56] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:34 [I 2022-10-27 23:41:56] ...trap/engines/engine.py.229: process: 0.55283 | load: 0.00250 [I 2022-10-27 23:41:56] ...trap/engines/engine.py.230: loss: 0.15157 [I 2022-10-27 23:41:56] ...optimizers/trijoint.py.151: total_norm: 13.830445 [I 2022-10-27 23:42:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 1390/2304 [I 2022-10-27 23:42:01] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:28 [I 2022-10-27 23:42:01] ...trap/engines/engine.py.229: process: 0.55012 | load: 0.00249 [I 2022-10-27 23:42:01] ...trap/engines/engine.py.230: loss: 0.12239 [I 2022-10-27 23:42:01] ...optimizers/trijoint.py.151: total_norm: 17.126366 [I 2022-10-27 23:42:07] ...trap/engines/engine.py.224: train: epoch 94 | batch 1400/2304 [I 2022-10-27 23:42:07] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:27 [I 2022-10-27 23:42:07] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00245 [I 2022-10-27 23:42:07] ...trap/engines/engine.py.230: loss: 0.10170 [I 2022-10-27 23:42:07] ...optimizers/trijoint.py.151: total_norm: 13.851080 [I 2022-10-27 23:42:13] ...trap/engines/engine.py.224: train: epoch 94 | batch 1410/2304 [I 2022-10-27 23:42:13] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:26 [I 2022-10-27 23:42:13] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00244 [I 2022-10-27 23:42:13] ...trap/engines/engine.py.230: loss: 0.14637 [I 2022-10-27 23:42:13] ...optimizers/trijoint.py.151: total_norm: 10.053713 [I 2022-10-27 23:42:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 1420/2304 [I 2022-10-27 23:42:18] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:17 [I 2022-10-27 23:42:18] ...trap/engines/engine.py.229: process: 0.56710 | load: 0.00249 [I 2022-10-27 23:42:18] ...trap/engines/engine.py.230: loss: 0.13341 [I 2022-10-27 23:42:18] ...optimizers/trijoint.py.151: total_norm: 15.638445 [I 2022-10-27 23:42:24] ...trap/engines/engine.py.224: train: epoch 94 | batch 1430/2304 [I 2022-10-27 23:42:24] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:12 [I 2022-10-27 23:42:24] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00254 [I 2022-10-27 23:42:24] ...trap/engines/engine.py.230: loss: 0.10041 [I 2022-10-27 23:42:24] ...optimizers/trijoint.py.151: total_norm: 13.796489 [I 2022-10-27 23:42:30] ...trap/engines/engine.py.224: train: epoch 94 | batch 1440/2304 [I 2022-10-27 23:42:30] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:06 [I 2022-10-27 23:42:30] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00256 [I 2022-10-27 23:42:30] ...trap/engines/engine.py.230: loss: 0.09681 [I 2022-10-27 23:42:30] ...optimizers/trijoint.py.151: total_norm: 15.454574 [I 2022-10-27 23:42:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 1450/2304 [I 2022-10-27 23:42:35] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:07:57 [I 2022-10-27 23:42:35] ...trap/engines/engine.py.229: process: 0.54891 | load: 0.00246 [I 2022-10-27 23:42:35] ...trap/engines/engine.py.230: loss: 0.12526 [I 2022-10-27 23:42:35] ...optimizers/trijoint.py.151: total_norm: 18.259880 [I 2022-10-27 23:42:41] ...trap/engines/engine.py.224: train: epoch 94 | batch 1460/2304 [I 2022-10-27 23:42:41] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:07:52 [I 2022-10-27 23:42:41] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00252 [I 2022-10-27 23:42:41] ...trap/engines/engine.py.230: loss: 0.14094 [I 2022-10-27 23:42:41] ...optimizers/trijoint.py.151: total_norm: 23.571451 [I 2022-10-27 23:42:46] ...trap/engines/engine.py.224: train: epoch 94 | batch 1470/2304 [I 2022-10-27 23:42:46] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:43 [I 2022-10-27 23:42:46] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00244 [I 2022-10-27 23:42:46] ...trap/engines/engine.py.230: loss: 0.09483 [I 2022-10-27 23:42:46] ...optimizers/trijoint.py.151: total_norm: 13.784228 [I 2022-10-27 23:42:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 1480/2304 [I 2022-10-27 23:42:52] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:40 [I 2022-10-27 23:42:52] ...trap/engines/engine.py.229: process: 0.56430 | load: 0.00247 [I 2022-10-27 23:42:52] ...trap/engines/engine.py.230: loss: 0.15820 [I 2022-10-27 23:42:52] ...optimizers/trijoint.py.151: total_norm: 24.162895 [I 2022-10-27 23:42:58] ...trap/engines/engine.py.224: train: epoch 94 | batch 1490/2304 [I 2022-10-27 23:42:58] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:38 [I 2022-10-27 23:42:58] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00263 [I 2022-10-27 23:42:58] ...trap/engines/engine.py.230: loss: 0.10526 [I 2022-10-27 23:42:58] ...optimizers/trijoint.py.151: total_norm: 8.427464 [I 2022-10-27 23:43:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 1500/2304 [I 2022-10-27 23:43:03] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:31 [I 2022-10-27 23:43:03] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00253 [I 2022-10-27 23:43:03] ...trap/engines/engine.py.230: loss: 0.15271 [I 2022-10-27 23:43:03] ...optimizers/trijoint.py.151: total_norm: 11.417950 [I 2022-10-27 23:43:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 1510/2304 [I 2022-10-27 23:43:09] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:25 [I 2022-10-27 23:43:09] ...trap/engines/engine.py.229: process: 0.56138 | load: 0.00245 [I 2022-10-27 23:43:09] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-27 23:43:09] ...optimizers/trijoint.py.151: total_norm: 18.982651 [I 2022-10-27 23:43:15] ...trap/engines/engine.py.224: train: epoch 94 | batch 1520/2304 [I 2022-10-27 23:43:15] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:14 [I 2022-10-27 23:43:15] ...trap/engines/engine.py.229: process: 0.54476 | load: 0.00259 [I 2022-10-27 23:43:15] ...trap/engines/engine.py.230: loss: 0.08304 [I 2022-10-27 23:43:15] ...optimizers/trijoint.py.151: total_norm: 17.211031 [I 2022-10-27 23:43:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 1530/2304 [I 2022-10-27 23:43:20] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:06 [I 2022-10-27 23:43:20] ...trap/engines/engine.py.229: process: 0.54109 | load: 0.00245 [I 2022-10-27 23:43:20] ...trap/engines/engine.py.230: loss: 0.10785 [I 2022-10-27 23:43:20] ...optimizers/trijoint.py.151: total_norm: 11.337724 [I 2022-10-27 23:43:26] ...trap/engines/engine.py.224: train: epoch 94 | batch 1540/2304 [I 2022-10-27 23:43:26] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:07 [I 2022-10-27 23:43:26] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00245 [I 2022-10-27 23:43:26] ...trap/engines/engine.py.230: loss: 0.14105 [I 2022-10-27 23:43:26] ...optimizers/trijoint.py.151: total_norm: 20.329676 [I 2022-10-27 23:43:31] ...trap/engines/engine.py.224: train: epoch 94 | batch 1550/2304 [I 2022-10-27 23:43:31] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:01 [I 2022-10-27 23:43:31] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00245 [I 2022-10-27 23:43:31] ...trap/engines/engine.py.230: loss: 0.13772 [I 2022-10-27 23:43:31] ...optimizers/trijoint.py.151: total_norm: 15.698411 [I 2022-10-27 23:43:37] ...trap/engines/engine.py.224: train: epoch 94 | batch 1560/2304 [I 2022-10-27 23:43:37] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:06:55 [I 2022-10-27 23:43:37] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00249 [I 2022-10-27 23:43:37] ...trap/engines/engine.py.230: loss: 0.14426 [I 2022-10-27 23:43:37] ...optimizers/trijoint.py.151: total_norm: 13.069739 [I 2022-10-27 23:43:42] ...trap/engines/engine.py.224: train: epoch 94 | batch 1570/2304 [I 2022-10-27 23:43:42] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:06:45 [I 2022-10-27 23:43:42] ...trap/engines/engine.py.229: process: 0.54364 | load: 0.00241 [I 2022-10-27 23:43:42] ...trap/engines/engine.py.230: loss: 0.10307 [I 2022-10-27 23:43:42] ...optimizers/trijoint.py.151: total_norm: 9.557090 [I 2022-10-27 23:43:48] ...trap/engines/engine.py.224: train: epoch 94 | batch 1580/2304 [I 2022-10-27 23:43:48] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:42 [I 2022-10-27 23:43:48] ...trap/engines/engine.py.229: process: 0.56173 | load: 0.00243 [I 2022-10-27 23:43:48] ...trap/engines/engine.py.230: loss: 0.10997 [I 2022-10-27 23:43:48] ...optimizers/trijoint.py.151: total_norm: 15.999140 [I 2022-10-27 23:43:54] ...trap/engines/engine.py.224: train: epoch 94 | batch 1590/2304 [I 2022-10-27 23:43:54] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:38 [I 2022-10-27 23:43:54] ...trap/engines/engine.py.229: process: 0.56648 | load: 0.00241 [I 2022-10-27 23:43:54] ...trap/engines/engine.py.230: loss: 0.12343 [I 2022-10-27 23:43:54] ...optimizers/trijoint.py.151: total_norm: 9.650276 [I 2022-10-27 23:43:59] ...trap/engines/engine.py.224: train: epoch 94 | batch 1600/2304 [I 2022-10-27 23:43:59] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:29 [I 2022-10-27 23:43:59] ...trap/engines/engine.py.229: process: 0.54444 | load: 0.00256 [I 2022-10-27 23:43:59] ...trap/engines/engine.py.230: loss: 0.09053 [I 2022-10-27 23:43:59] ...optimizers/trijoint.py.151: total_norm: 15.735393 [I 2022-10-27 23:44:05] ...trap/engines/engine.py.224: train: epoch 94 | batch 1610/2304 [I 2022-10-27 23:44:05] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:25 [I 2022-10-27 23:44:05] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00248 [I 2022-10-27 23:44:05] ...trap/engines/engine.py.230: loss: 0.11170 [I 2022-10-27 23:44:05] ...optimizers/trijoint.py.151: total_norm: 25.528929 [I 2022-10-27 23:44:10] ...trap/engines/engine.py.224: train: epoch 94 | batch 1620/2304 [I 2022-10-27 23:44:10] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:25 [I 2022-10-27 23:44:10] ...trap/engines/engine.py.229: process: 0.55532 | load: 0.00248 [I 2022-10-27 23:44:10] ...trap/engines/engine.py.230: loss: 0.16612 [I 2022-10-27 23:44:10] ...optimizers/trijoint.py.151: total_norm: 27.272165 [I 2022-10-27 23:44:16] ...trap/engines/engine.py.224: train: epoch 94 | batch 1630/2304 [I 2022-10-27 23:44:16] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:21 [I 2022-10-27 23:44:16] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00246 [I 2022-10-27 23:44:16] ...trap/engines/engine.py.230: loss: 0.14451 [I 2022-10-27 23:44:16] ...optimizers/trijoint.py.151: total_norm: 11.770212 [I 2022-10-27 23:44:22] ...trap/engines/engine.py.224: train: epoch 94 | batch 1640/2304 [I 2022-10-27 23:44:22] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:14 [I 2022-10-27 23:44:22] ...trap/engines/engine.py.229: process: 0.56472 | load: 0.00248 [I 2022-10-27 23:44:22] ...trap/engines/engine.py.230: loss: 0.14949 [I 2022-10-27 23:44:22] ...optimizers/trijoint.py.151: total_norm: 9.195898 [I 2022-10-27 23:44:27] ...trap/engines/engine.py.224: train: epoch 94 | batch 1650/2304 [I 2022-10-27 23:44:27] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:08 [I 2022-10-27 23:44:27] ...trap/engines/engine.py.229: process: 0.55351 | load: 0.00261 [I 2022-10-27 23:44:27] ...trap/engines/engine.py.230: loss: 0.15527 [I 2022-10-27 23:44:27] ...optimizers/trijoint.py.151: total_norm: 14.655079 [I 2022-10-27 23:44:33] ...trap/engines/engine.py.224: train: epoch 94 | batch 1660/2304 [I 2022-10-27 23:44:33] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:06:05 [I 2022-10-27 23:44:33] ...trap/engines/engine.py.229: process: 0.58384 | load: 0.00247 [I 2022-10-27 23:44:33] ...trap/engines/engine.py.230: loss: 0.10277 [I 2022-10-27 23:44:33] ...optimizers/trijoint.py.151: total_norm: 18.653393 [I 2022-10-27 23:44:39] ...trap/engines/engine.py.224: train: epoch 94 | batch 1670/2304 [I 2022-10-27 23:44:39] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:05:57 [I 2022-10-27 23:44:39] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00246 [I 2022-10-27 23:44:39] ...trap/engines/engine.py.230: loss: 0.09512 [I 2022-10-27 23:44:39] ...optimizers/trijoint.py.151: total_norm: 25.944571 [I 2022-10-27 23:44:44] ...trap/engines/engine.py.224: train: epoch 94 | batch 1680/2304 [I 2022-10-27 23:44:44] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:05:51 [I 2022-10-27 23:44:44] ...trap/engines/engine.py.229: process: 0.56847 | load: 0.00243 [I 2022-10-27 23:44:44] ...trap/engines/engine.py.230: loss: 0.11667 [I 2022-10-27 23:44:44] ...optimizers/trijoint.py.151: total_norm: 19.041838 [I 2022-10-27 23:44:50] ...trap/engines/engine.py.224: train: epoch 94 | batch 1690/2304 [I 2022-10-27 23:44:50] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:43 [I 2022-10-27 23:44:50] ...trap/engines/engine.py.229: process: 0.56753 | load: 0.00258 [I 2022-10-27 23:44:50] ...trap/engines/engine.py.230: loss: 0.12309 [I 2022-10-27 23:44:50] ...optimizers/trijoint.py.151: total_norm: 19.074642 [I 2022-10-27 23:44:55] ...trap/engines/engine.py.224: train: epoch 94 | batch 1700/2304 [I 2022-10-27 23:44:55] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:36 [I 2022-10-27 23:44:55] ...trap/engines/engine.py.229: process: 0.54822 | load: 0.00250 [I 2022-10-27 23:44:55] ...trap/engines/engine.py.230: loss: 0.11239 [I 2022-10-27 23:44:55] ...optimizers/trijoint.py.151: total_norm: 12.624846 [I 2022-10-27 23:45:01] ...trap/engines/engine.py.224: train: epoch 94 | batch 1710/2304 [I 2022-10-27 23:45:01] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:31 [I 2022-10-27 23:45:01] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00256 [I 2022-10-27 23:45:01] ...trap/engines/engine.py.230: loss: 0.07371 [I 2022-10-27 23:45:01] ...optimizers/trijoint.py.151: total_norm: 8.249096 [I 2022-10-27 23:45:07] ...trap/engines/engine.py.224: train: epoch 94 | batch 1720/2304 [I 2022-10-27 23:45:07] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:28 [I 2022-10-27 23:45:07] ...trap/engines/engine.py.229: process: 0.55364 | load: 0.00257 [I 2022-10-27 23:45:07] ...trap/engines/engine.py.230: loss: 0.11572 [I 2022-10-27 23:45:07] ...optimizers/trijoint.py.151: total_norm: 16.031101 [I 2022-10-27 23:45:12] ...trap/engines/engine.py.224: train: epoch 94 | batch 1730/2304 [I 2022-10-27 23:45:12] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:22 [I 2022-10-27 23:45:12] ...trap/engines/engine.py.229: process: 0.56671 | load: 0.00250 [I 2022-10-27 23:45:12] ...trap/engines/engine.py.230: loss: 0.08803 [I 2022-10-27 23:45:12] ...optimizers/trijoint.py.151: total_norm: 27.271736 [I 2022-10-27 23:45:18] ...trap/engines/engine.py.224: train: epoch 94 | batch 1740/2304 [I 2022-10-27 23:45:18] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:13 [I 2022-10-27 23:45:18] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00249 [I 2022-10-27 23:45:18] ...trap/engines/engine.py.230: loss: 0.09199 [I 2022-10-27 23:45:18] ...optimizers/trijoint.py.151: total_norm: 16.262922 [I 2022-10-27 23:45:24] ...trap/engines/engine.py.224: train: epoch 94 | batch 1750/2304 [I 2022-10-27 23:45:24] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:13 [I 2022-10-27 23:45:24] ...trap/engines/engine.py.229: process: 0.56412 | load: 0.00340 [I 2022-10-27 23:45:24] ...trap/engines/engine.py.230: loss: 0.13627 [I 2022-10-27 23:45:24] ...optimizers/trijoint.py.151: total_norm: 19.409407 [I 2022-10-27 23:45:29] ...trap/engines/engine.py.224: train: epoch 94 | batch 1760/2304 [I 2022-10-27 23:45:29] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:05:04 [I 2022-10-27 23:45:29] ...trap/engines/engine.py.229: process: 0.56702 | load: 0.00275 [I 2022-10-27 23:45:29] ...trap/engines/engine.py.230: loss: 0.07484 [I 2022-10-27 23:45:29] ...optimizers/trijoint.py.151: total_norm: 9.197406 [I 2022-10-27 23:45:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 1770/2304 [I 2022-10-27 23:45:35] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:05:00 [I 2022-10-27 23:45:35] ...trap/engines/engine.py.229: process: 0.56460 | load: 0.00252 [I 2022-10-27 23:45:35] ...trap/engines/engine.py.230: loss: 0.10512 [I 2022-10-27 23:45:35] ...optimizers/trijoint.py.151: total_norm: 26.990715 [I 2022-10-27 23:45:40] ...trap/engines/engine.py.224: train: epoch 94 | batch 1780/2304 [I 2022-10-27 23:45:40] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:04:53 [I 2022-10-27 23:45:40] ...trap/engines/engine.py.229: process: 0.55521 | load: 0.00247 [I 2022-10-27 23:45:40] ...trap/engines/engine.py.230: loss: 0.08364 [I 2022-10-27 23:45:40] ...optimizers/trijoint.py.151: total_norm: 6.713684 [I 2022-10-27 23:45:46] ...trap/engines/engine.py.224: train: epoch 94 | batch 1790/2304 [I 2022-10-27 23:45:46] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:04:49 [I 2022-10-27 23:45:46] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00245 [I 2022-10-27 23:45:46] ...trap/engines/engine.py.230: loss: 0.11451 [I 2022-10-27 23:45:46] ...optimizers/trijoint.py.151: total_norm: 9.977526 [I 2022-10-27 23:45:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 1800/2304 [I 2022-10-27 23:45:52] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:40 [I 2022-10-27 23:45:52] ...trap/engines/engine.py.229: process: 0.55059 | load: 0.00245 [I 2022-10-27 23:45:52] ...trap/engines/engine.py.230: loss: 0.14794 [I 2022-10-27 23:45:52] ...optimizers/trijoint.py.151: total_norm: 17.392893 [I 2022-10-27 23:45:57] ...trap/engines/engine.py.224: train: epoch 94 | batch 1810/2304 [I 2022-10-27 23:45:57] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:36 [I 2022-10-27 23:45:57] ...trap/engines/engine.py.229: process: 0.55246 | load: 0.00243 [I 2022-10-27 23:45:57] ...trap/engines/engine.py.230: loss: 0.12229 [I 2022-10-27 23:45:57] ...optimizers/trijoint.py.151: total_norm: 14.820724 [I 2022-10-27 23:46:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 1820/2304 [I 2022-10-27 23:46:03] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:30 [I 2022-10-27 23:46:03] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00255 [I 2022-10-27 23:46:03] ...trap/engines/engine.py.230: loss: 0.08110 [I 2022-10-27 23:46:03] ...optimizers/trijoint.py.151: total_norm: 10.684146 [I 2022-10-27 23:46:08] ...trap/engines/engine.py.224: train: epoch 94 | batch 1830/2304 [I 2022-10-27 23:46:08] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:27 [I 2022-10-27 23:46:08] ...trap/engines/engine.py.229: process: 0.56629 | load: 0.00250 [I 2022-10-27 23:46:08] ...trap/engines/engine.py.230: loss: 0.11197 [I 2022-10-27 23:46:08] ...optimizers/trijoint.py.151: total_norm: 26.179771 [I 2022-10-27 23:46:14] ...trap/engines/engine.py.224: train: epoch 94 | batch 1840/2304 [I 2022-10-27 23:46:14] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:19 [I 2022-10-27 23:46:14] ...trap/engines/engine.py.229: process: 0.55404 | load: 0.00246 [I 2022-10-27 23:46:14] ...trap/engines/engine.py.230: loss: 0.11772 [I 2022-10-27 23:46:14] ...optimizers/trijoint.py.151: total_norm: 12.255378 [I 2022-10-27 23:46:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 1850/2304 [I 2022-10-27 23:46:20] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:13 [I 2022-10-27 23:46:20] ...trap/engines/engine.py.229: process: 0.56797 | load: 0.00243 [I 2022-10-27 23:46:20] ...trap/engines/engine.py.230: loss: 0.11858 [I 2022-10-27 23:46:20] ...optimizers/trijoint.py.151: total_norm: 38.922401 [I 2022-10-27 23:46:25] ...trap/engines/engine.py.224: train: epoch 94 | batch 1860/2304 [I 2022-10-27 23:46:25] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:06 [I 2022-10-27 23:46:25] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00246 [I 2022-10-27 23:46:25] ...trap/engines/engine.py.230: loss: 0.10715 [I 2022-10-27 23:46:25] ...optimizers/trijoint.py.151: total_norm: 12.931800 [I 2022-10-27 23:46:31] ...trap/engines/engine.py.224: train: epoch 94 | batch 1870/2304 [I 2022-10-27 23:46:31] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:00 [I 2022-10-27 23:46:31] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00248 [I 2022-10-27 23:46:31] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-27 23:46:31] ...optimizers/trijoint.py.151: total_norm: 17.954357 [I 2022-10-27 23:46:36] ...trap/engines/engine.py.224: train: epoch 94 | batch 1880/2304 [I 2022-10-27 23:46:36] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:04:04 [I 2022-10-27 23:46:36] ...trap/engines/engine.py.229: process: 0.60219 | load: 0.00243 [I 2022-10-27 23:46:36] ...trap/engines/engine.py.230: loss: 0.08739 [I 2022-10-27 23:46:36] ...optimizers/trijoint.py.151: total_norm: 11.971002 [I 2022-10-27 23:46:42] ...trap/engines/engine.py.224: train: epoch 94 | batch 1890/2304 [I 2022-10-27 23:46:42] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:03:52 [I 2022-10-27 23:46:42] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00252 [I 2022-10-27 23:46:42] ...trap/engines/engine.py.230: loss: 0.13590 [I 2022-10-27 23:46:42] ...optimizers/trijoint.py.151: total_norm: 23.971149 [I 2022-10-27 23:46:48] ...trap/engines/engine.py.224: train: epoch 94 | batch 1900/2304 [I 2022-10-27 23:46:48] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:03:45 [I 2022-10-27 23:46:48] ...trap/engines/engine.py.229: process: 0.55800 | load: 0.00241 [I 2022-10-27 23:46:48] ...trap/engines/engine.py.230: loss: 0.16821 [I 2022-10-27 23:46:48] ...optimizers/trijoint.py.151: total_norm: 12.357482 [I 2022-10-27 23:46:53] ...trap/engines/engine.py.224: train: epoch 94 | batch 1910/2304 [I 2022-10-27 23:46:53] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:42 [I 2022-10-27 23:46:53] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00239 [I 2022-10-27 23:46:53] ...trap/engines/engine.py.230: loss: 0.14234 [I 2022-10-27 23:46:53] ...optimizers/trijoint.py.151: total_norm: 13.852624 [I 2022-10-27 23:46:59] ...trap/engines/engine.py.224: train: epoch 94 | batch 1920/2304 [I 2022-10-27 23:46:59] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:33 [I 2022-10-27 23:46:59] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00246 [I 2022-10-27 23:46:59] ...trap/engines/engine.py.230: loss: 0.11959 [I 2022-10-27 23:46:59] ...optimizers/trijoint.py.151: total_norm: 18.018642 [I 2022-10-27 23:47:04] ...trap/engines/engine.py.224: train: epoch 94 | batch 1930/2304 [I 2022-10-27 23:47:04] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:29 [I 2022-10-27 23:47:04] ...trap/engines/engine.py.229: process: 0.56384 | load: 0.00248 [I 2022-10-27 23:47:04] ...trap/engines/engine.py.230: loss: 0.13515 [I 2022-10-27 23:47:04] ...optimizers/trijoint.py.151: total_norm: 27.877106 [I 2022-10-27 23:47:10] ...trap/engines/engine.py.224: train: epoch 94 | batch 1940/2304 [I 2022-10-27 23:47:10] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:21 [I 2022-10-27 23:47:10] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00249 [I 2022-10-27 23:47:10] ...trap/engines/engine.py.230: loss: 0.09910 [I 2022-10-27 23:47:10] ...optimizers/trijoint.py.151: total_norm: 20.915298 [I 2022-10-27 23:47:16] ...trap/engines/engine.py.224: train: epoch 94 | batch 1950/2304 [I 2022-10-27 23:47:16] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:17 [I 2022-10-27 23:47:16] ...trap/engines/engine.py.229: process: 0.56194 | load: 0.00243 [I 2022-10-27 23:47:16] ...trap/engines/engine.py.230: loss: 0.17841 [I 2022-10-27 23:47:16] ...optimizers/trijoint.py.151: total_norm: 26.562593 [I 2022-10-27 23:47:21] ...trap/engines/engine.py.224: train: epoch 94 | batch 1960/2304 [I 2022-10-27 23:47:21] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:11 [I 2022-10-27 23:47:21] ...trap/engines/engine.py.229: process: 0.55672 | load: 0.00246 [I 2022-10-27 23:47:21] ...trap/engines/engine.py.230: loss: 0.14698 [I 2022-10-27 23:47:21] ...optimizers/trijoint.py.151: total_norm: 26.347422 [I 2022-10-27 23:47:27] ...trap/engines/engine.py.224: train: epoch 94 | batch 1970/2304 [I 2022-10-27 23:47:27] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:08 [I 2022-10-27 23:47:27] ...trap/engines/engine.py.229: process: 0.56705 | load: 0.00241 [I 2022-10-27 23:47:27] ...trap/engines/engine.py.230: loss: 0.12858 [I 2022-10-27 23:47:27] ...optimizers/trijoint.py.151: total_norm: 19.121542 [I 2022-10-27 23:47:32] ...trap/engines/engine.py.224: train: epoch 94 | batch 1980/2304 [I 2022-10-27 23:47:32] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:03:01 [I 2022-10-27 23:47:32] ...trap/engines/engine.py.229: process: 0.56288 | load: 0.00240 [I 2022-10-27 23:47:32] ...trap/engines/engine.py.230: loss: 0.10326 [I 2022-10-27 23:47:32] ...optimizers/trijoint.py.151: total_norm: 24.534960 [I 2022-10-27 23:47:38] ...trap/engines/engine.py.224: train: epoch 94 | batch 1990/2304 [I 2022-10-27 23:47:38] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:02:55 [I 2022-10-27 23:47:38] ...trap/engines/engine.py.229: process: 0.56025 | load: 0.00238 [I 2022-10-27 23:47:38] ...trap/engines/engine.py.230: loss: 0.10180 [I 2022-10-27 23:47:38] ...optimizers/trijoint.py.151: total_norm: 18.388313 [I 2022-10-27 23:47:44] ...trap/engines/engine.py.224: train: epoch 94 | batch 2000/2304 [I 2022-10-27 23:47:44] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:02:49 [I 2022-10-27 23:47:44] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00242 [I 2022-10-27 23:47:44] ...trap/engines/engine.py.230: loss: 0.12821 [I 2022-10-27 23:47:44] ...optimizers/trijoint.py.151: total_norm: 17.004185 [I 2022-10-27 23:47:49] ...trap/engines/engine.py.224: train: epoch 94 | batch 2010/2304 [I 2022-10-27 23:47:49] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:43 [I 2022-10-27 23:47:49] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00242 [I 2022-10-27 23:47:49] ...trap/engines/engine.py.230: loss: 0.14953 [I 2022-10-27 23:47:49] ...optimizers/trijoint.py.151: total_norm: 19.286589 [I 2022-10-27 23:47:55] ...trap/engines/engine.py.224: train: epoch 94 | batch 2020/2304 [I 2022-10-27 23:47:55] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:38 [I 2022-10-27 23:47:55] ...trap/engines/engine.py.229: process: 0.57010 | load: 0.00242 [I 2022-10-27 23:47:55] ...trap/engines/engine.py.230: loss: 0.09289 [I 2022-10-27 23:47:55] ...optimizers/trijoint.py.151: total_norm: 10.710006 [I 2022-10-27 23:48:00] ...trap/engines/engine.py.224: train: epoch 94 | batch 2030/2304 [I 2022-10-27 23:48:00] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:34 [I 2022-10-27 23:48:00] ...trap/engines/engine.py.229: process: 0.54960 | load: 0.00252 [I 2022-10-27 23:48:00] ...trap/engines/engine.py.230: loss: 0.15451 [I 2022-10-27 23:48:00] ...optimizers/trijoint.py.151: total_norm: 16.830847 [I 2022-10-27 23:48:06] ...trap/engines/engine.py.224: train: epoch 94 | batch 2040/2304 [I 2022-10-27 23:48:06] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:27 [I 2022-10-27 23:48:06] ...trap/engines/engine.py.229: process: 0.55578 | load: 0.00245 [I 2022-10-27 23:48:06] ...trap/engines/engine.py.230: loss: 0.09823 [I 2022-10-27 23:48:06] ...optimizers/trijoint.py.151: total_norm: 11.933105 [I 2022-10-27 23:48:12] ...trap/engines/engine.py.224: train: epoch 94 | batch 2050/2304 [I 2022-10-27 23:48:12] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:21 [I 2022-10-27 23:48:12] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00246 [I 2022-10-27 23:48:12] ...trap/engines/engine.py.230: loss: 0.15363 [I 2022-10-27 23:48:12] ...optimizers/trijoint.py.151: total_norm: 21.189402 [I 2022-10-27 23:48:17] ...trap/engines/engine.py.224: train: epoch 94 | batch 2060/2304 [I 2022-10-27 23:48:17] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:16 [I 2022-10-27 23:48:17] ...trap/engines/engine.py.229: process: 0.56612 | load: 0.00243 [I 2022-10-27 23:48:17] ...trap/engines/engine.py.230: loss: 0.21832 [I 2022-10-27 23:48:17] ...optimizers/trijoint.py.151: total_norm: 17.534348 [I 2022-10-27 23:48:23] ...trap/engines/engine.py.224: train: epoch 94 | batch 2070/2304 [I 2022-10-27 23:48:23] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:10 [I 2022-10-27 23:48:23] ...trap/engines/engine.py.229: process: 0.56330 | load: 0.00246 [I 2022-10-27 23:48:23] ...trap/engines/engine.py.230: loss: 0.10825 [I 2022-10-27 23:48:23] ...optimizers/trijoint.py.151: total_norm: 10.003999 [I 2022-10-27 23:48:28] ...trap/engines/engine.py.224: train: epoch 94 | batch 2080/2304 [I 2022-10-27 23:48:28] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:02:04 [I 2022-10-27 23:48:28] ...trap/engines/engine.py.229: process: 0.55275 | load: 0.00242 [I 2022-10-27 23:48:28] ...trap/engines/engine.py.230: loss: 0.12647 [I 2022-10-27 23:48:28] ...optimizers/trijoint.py.151: total_norm: 13.615133 [I 2022-10-27 23:48:34] ...trap/engines/engine.py.224: train: epoch 94 | batch 2090/2304 [I 2022-10-27 23:48:34] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:01:58 [I 2022-10-27 23:48:34] ...trap/engines/engine.py.229: process: 0.56114 | load: 0.00245 [I 2022-10-27 23:48:34] ...trap/engines/engine.py.230: loss: 0.11709 [I 2022-10-27 23:48:34] ...optimizers/trijoint.py.151: total_norm: 20.819088 [I 2022-10-27 23:48:40] ...trap/engines/engine.py.224: train: epoch 94 | batch 2100/2304 [I 2022-10-27 23:48:40] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:01:53 [I 2022-10-27 23:48:40] ...trap/engines/engine.py.229: process: 0.54764 | load: 0.00242 [I 2022-10-27 23:48:40] ...trap/engines/engine.py.230: loss: 0.12553 [I 2022-10-27 23:48:40] ...optimizers/trijoint.py.151: total_norm: 15.047513 [I 2022-10-27 23:48:45] ...trap/engines/engine.py.224: train: epoch 94 | batch 2110/2304 [I 2022-10-27 23:48:45] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:01:47 [I 2022-10-27 23:48:45] ...trap/engines/engine.py.229: process: 0.54995 | load: 0.00241 [I 2022-10-27 23:48:45] ...trap/engines/engine.py.230: loss: 0.08491 [I 2022-10-27 23:48:45] ...optimizers/trijoint.py.151: total_norm: 11.030180 [I 2022-10-27 23:48:51] ...trap/engines/engine.py.224: train: epoch 94 | batch 2120/2304 [I 2022-10-27 23:48:51] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:41 [I 2022-10-27 23:48:51] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00243 [I 2022-10-27 23:48:51] ...trap/engines/engine.py.230: loss: 0.11469 [I 2022-10-27 23:48:51] ...optimizers/trijoint.py.151: total_norm: 9.493378 [I 2022-10-27 23:48:56] ...trap/engines/engine.py.224: train: epoch 94 | batch 2130/2304 [I 2022-10-27 23:48:56] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:37 [I 2022-10-27 23:48:56] ...trap/engines/engine.py.229: process: 0.56916 | load: 0.00240 [I 2022-10-27 23:48:56] ...trap/engines/engine.py.230: loss: 0.10555 [I 2022-10-27 23:48:56] ...optimizers/trijoint.py.151: total_norm: 20.566401 [I 2022-10-27 23:49:02] ...trap/engines/engine.py.224: train: epoch 94 | batch 2140/2304 [I 2022-10-27 23:49:02] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:31 [I 2022-10-27 23:49:02] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00243 [I 2022-10-27 23:49:02] ...trap/engines/engine.py.230: loss: 0.10819 [I 2022-10-27 23:49:02] ...optimizers/trijoint.py.151: total_norm: 12.255048 [I 2022-10-27 23:49:07] ...trap/engines/engine.py.224: train: epoch 94 | batch 2150/2304 [I 2022-10-27 23:49:07] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:25 [I 2022-10-27 23:49:07] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00239 [I 2022-10-27 23:49:07] ...trap/engines/engine.py.230: loss: 0.07920 [I 2022-10-27 23:49:07] ...optimizers/trijoint.py.151: total_norm: 18.353451 [I 2022-10-27 23:49:13] ...trap/engines/engine.py.224: train: epoch 94 | batch 2160/2304 [I 2022-10-27 23:49:13] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:23 [I 2022-10-27 23:49:13] ...trap/engines/engine.py.229: process: 0.67348 | load: 0.00239 [I 2022-10-27 23:49:13] ...trap/engines/engine.py.230: loss: 0.17838 [I 2022-10-27 23:49:13] ...optimizers/trijoint.py.151: total_norm: 14.281128 [I 2022-10-27 23:49:19] ...trap/engines/engine.py.224: train: epoch 94 | batch 2170/2304 [I 2022-10-27 23:49:19] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:14 [I 2022-10-27 23:49:19] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00242 [I 2022-10-27 23:49:19] ...trap/engines/engine.py.230: loss: 0.10069 [I 2022-10-27 23:49:19] ...optimizers/trijoint.py.151: total_norm: 11.620308 [I 2022-10-27 23:49:24] ...trap/engines/engine.py.224: train: epoch 94 | batch 2180/2304 [I 2022-10-27 23:49:24] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:08 [I 2022-10-27 23:49:24] ...trap/engines/engine.py.229: process: 0.54826 | load: 0.00240 [I 2022-10-27 23:49:24] ...trap/engines/engine.py.230: loss: 0.13112 [I 2022-10-27 23:49:24] ...optimizers/trijoint.py.151: total_norm: 9.988789 [I 2022-10-27 23:49:30] ...trap/engines/engine.py.224: train: epoch 94 | batch 2190/2304 [I 2022-10-27 23:49:30] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:03 [I 2022-10-27 23:49:30] ...trap/engines/engine.py.229: process: 0.55181 | load: 0.00262 [I 2022-10-27 23:49:30] ...trap/engines/engine.py.230: loss: 0.11195 [I 2022-10-27 23:49:30] ...optimizers/trijoint.py.151: total_norm: 15.293733 [I 2022-10-27 23:49:35] ...trap/engines/engine.py.224: train: epoch 94 | batch 2200/2304 [I 2022-10-27 23:49:35] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:00:57 [I 2022-10-27 23:49:35] ...trap/engines/engine.py.229: process: 0.53976 | load: 0.00241 [I 2022-10-27 23:49:35] ...trap/engines/engine.py.230: loss: 0.18742 [I 2022-10-27 23:49:35] ...optimizers/trijoint.py.151: total_norm: 15.711824 [I 2022-10-27 23:49:41] ...trap/engines/engine.py.224: train: epoch 94 | batch 2210/2304 [I 2022-10-27 23:49:41] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:00:51 [I 2022-10-27 23:49:41] ...trap/engines/engine.py.229: process: 0.55157 | load: 0.00242 [I 2022-10-27 23:49:41] ...trap/engines/engine.py.230: loss: 0.13935 [I 2022-10-27 23:49:41] ...optimizers/trijoint.py.151: total_norm: 13.461044 [I 2022-10-27 23:49:46] ...trap/engines/engine.py.224: train: epoch 94 | batch 2220/2304 [I 2022-10-27 23:49:46] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:46 [I 2022-10-27 23:49:46] ...trap/engines/engine.py.229: process: 0.54930 | load: 0.00241 [I 2022-10-27 23:49:46] ...trap/engines/engine.py.230: loss: 0.11884 [I 2022-10-27 23:49:46] ...optimizers/trijoint.py.151: total_norm: 12.259879 [I 2022-10-27 23:49:52] ...trap/engines/engine.py.224: train: epoch 94 | batch 2230/2304 [I 2022-10-27 23:49:52] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:41 [I 2022-10-27 23:49:52] ...trap/engines/engine.py.229: process: 0.56740 | load: 0.00242 [I 2022-10-27 23:49:52] ...trap/engines/engine.py.230: loss: 0.16107 [I 2022-10-27 23:49:52] ...optimizers/trijoint.py.151: total_norm: 15.592746 [I 2022-10-27 23:49:58] ...trap/engines/engine.py.224: train: epoch 94 | batch 2240/2304 [I 2022-10-27 23:49:58] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:35 [I 2022-10-27 23:49:58] ...trap/engines/engine.py.229: process: 0.57189 | load: 0.00243 [I 2022-10-27 23:49:58] ...trap/engines/engine.py.230: loss: 0.13250 [I 2022-10-27 23:49:58] ...optimizers/trijoint.py.151: total_norm: 15.155383 [I 2022-10-27 23:50:03] ...trap/engines/engine.py.224: train: epoch 94 | batch 2250/2304 [I 2022-10-27 23:50:03] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:30 [I 2022-10-27 23:50:03] ...trap/engines/engine.py.229: process: 0.56673 | load: 0.00252 [I 2022-10-27 23:50:03] ...trap/engines/engine.py.230: loss: 0.21806 [I 2022-10-27 23:50:03] ...optimizers/trijoint.py.151: total_norm: 21.944118 [I 2022-10-27 23:50:09] ...trap/engines/engine.py.224: train: epoch 94 | batch 2260/2304 [I 2022-10-27 23:50:09] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:24 [I 2022-10-27 23:50:09] ...trap/engines/engine.py.229: process: 0.54436 | load: 0.00249 [I 2022-10-27 23:50:09] ...trap/engines/engine.py.230: loss: 0.09940 [I 2022-10-27 23:50:09] ...optimizers/trijoint.py.151: total_norm: 10.282584 [I 2022-10-27 23:50:14] ...trap/engines/engine.py.224: train: epoch 94 | batch 2270/2304 [I 2022-10-27 23:50:14] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:18 [I 2022-10-27 23:50:14] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00247 [I 2022-10-27 23:50:14] ...trap/engines/engine.py.230: loss: 0.11422 [I 2022-10-27 23:50:14] ...optimizers/trijoint.py.151: total_norm: 30.796703 [I 2022-10-27 23:50:20] ...trap/engines/engine.py.224: train: epoch 94 | batch 2280/2304 [I 2022-10-27 23:50:20] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:13 [I 2022-10-27 23:50:20] ...trap/engines/engine.py.229: process: 0.56137 | load: 0.00242 [I 2022-10-27 23:50:20] ...trap/engines/engine.py.230: loss: 0.14960 [I 2022-10-27 23:50:20] ...optimizers/trijoint.py.151: total_norm: 19.622423 [I 2022-10-27 23:50:26] ...trap/engines/engine.py.224: train: epoch 94 | batch 2290/2304 [I 2022-10-27 23:50:26] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:07 [I 2022-10-27 23:50:26] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00244 [I 2022-10-27 23:50:26] ...trap/engines/engine.py.230: loss: 0.09459 [I 2022-10-27 23:50:26] ...optimizers/trijoint.py.151: total_norm: 11.749970 [I 2022-10-27 23:50:31] ...trap/engines/engine.py.224: train: epoch 94 | batch 2300/2304 [I 2022-10-27 23:50:31] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:02 [I 2022-10-27 23:50:31] ...trap/engines/engine.py.229: process: 0.54427 | load: 0.00014 [I 2022-10-27 23:50:31] ...trap/engines/engine.py.230: loss: 0.12351 [I 2022-10-27 23:50:31] ...optimizers/trijoint.py.151: total_norm: 19.848320 [I 2022-10-27 23:50:33] ...trap/engines/engine.py.224: train: epoch 94 | batch 2304/2304 [I 2022-10-27 23:50:33] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:00 [I 2022-10-27 23:50:33] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00014 [I 2022-10-27 23:50:33] ...trap/engines/engine.py.230: loss: 0.12786 [I 2022-10-27 23:50:33] ...optimizers/trijoint.py.151: total_norm: 15.425316 [S 2022-10-27 23:50:35] ...trap/engines/engine.py.129: train_epoch.epoch: 94 [S 2022-10-27 23:50:35] ...trap/engines/engine.py.129: train_epoch.loss: 0.12965353261881432 [I 2022-10-27 23:50:35] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-27 23:50:36] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 94 [I 2022-10-27 23:50:36] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-27 23:50:38] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 23:50:38] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 23:50:38] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 23:50:41] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 23:50:42] ...trap/engines/engine.py.321: eval: epoch 94 | batch 0/510 [I 2022-10-27 23:50:42] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:46:59 [I 2022-10-27 23:50:42] ...trap/engines/engine.py.326: process: 5.52787 | load: 2.67997 [I 2022-10-27 23:50:45] ...trap/engines/engine.py.321: eval: epoch 94 | batch 10/510 [I 2022-10-27 23:50:45] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:25 [I 2022-10-27 23:50:45] ...trap/engines/engine.py.326: process: 0.32826 | load: 0.00020 [I 2022-10-27 23:50:48] ...trap/engines/engine.py.321: eval: epoch 94 | batch 20/510 [I 2022-10-27 23:50:48] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:14 [I 2022-10-27 23:50:48] ...trap/engines/engine.py.326: process: 0.35280 | load: 0.00019 [I 2022-10-27 23:50:52] ...trap/engines/engine.py.321: eval: epoch 94 | batch 30/510 [I 2022-10-27 23:50:52] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:43 [I 2022-10-27 23:50:52] ...trap/engines/engine.py.326: process: 0.33278 | load: 0.00018 [I 2022-10-27 23:50:55] ...trap/engines/engine.py.321: eval: epoch 94 | batch 40/510 [I 2022-10-27 23:50:55] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:38 [I 2022-10-27 23:50:55] ...trap/engines/engine.py.326: process: 0.33935 | load: 0.00017 [I 2022-10-27 23:50:58] ...trap/engines/engine.py.321: eval: epoch 94 | batch 50/510 [I 2022-10-27 23:50:58] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:33 [I 2022-10-27 23:50:58] ...trap/engines/engine.py.326: process: 0.33658 | load: 0.00017 [I 2022-10-27 23:51:02] ...trap/engines/engine.py.321: eval: epoch 94 | batch 60/510 [I 2022-10-27 23:51:02] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:31 [I 2022-10-27 23:51:02] ...trap/engines/engine.py.326: process: 0.32802 | load: 0.00016 [I 2022-10-27 23:51:05] ...trap/engines/engine.py.321: eval: epoch 94 | batch 70/510 [I 2022-10-27 23:51:05] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:25 [I 2022-10-27 23:51:05] ...trap/engines/engine.py.326: process: 0.34811 | load: 0.00018 [I 2022-10-27 23:51:08] ...trap/engines/engine.py.321: eval: epoch 94 | batch 80/510 [I 2022-10-27 23:51:08] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:23 [I 2022-10-27 23:51:08] ...trap/engines/engine.py.326: process: 0.32824 | load: 0.00020 [I 2022-10-27 23:51:12] ...trap/engines/engine.py.321: eval: epoch 94 | batch 90/510 [I 2022-10-27 23:51:12] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:22 [I 2022-10-27 23:51:12] ...trap/engines/engine.py.326: process: 0.32587 | load: 0.00019 [I 2022-10-27 23:51:15] ...trap/engines/engine.py.321: eval: epoch 94 | batch 100/510 [I 2022-10-27 23:51:15] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-27 23:51:15] ...trap/engines/engine.py.326: process: 0.28425 | load: 0.00024 [I 2022-10-27 23:51:18] ...trap/engines/engine.py.321: eval: epoch 94 | batch 110/510 [I 2022-10-27 23:51:18] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:58 [I 2022-10-27 23:51:18] ...trap/engines/engine.py.326: process: 0.28761 | load: 0.00015 [I 2022-10-27 23:51:21] ...trap/engines/engine.py.321: eval: epoch 94 | batch 120/510 [I 2022-10-27 23:51:21] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-27 23:51:21] ...trap/engines/engine.py.326: process: 0.29200 | load: 0.00021 [I 2022-10-27 23:51:24] ...trap/engines/engine.py.321: eval: epoch 94 | batch 130/510 [I 2022-10-27 23:51:24] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-27 23:51:24] ...trap/engines/engine.py.326: process: 0.28637 | load: 0.00017 [I 2022-10-27 23:51:27] ...trap/engines/engine.py.321: eval: epoch 94 | batch 140/510 [I 2022-10-27 23:51:27] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-27 23:51:27] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00014 [I 2022-10-27 23:51:29] ...trap/engines/engine.py.321: eval: epoch 94 | batch 150/510 [I 2022-10-27 23:51:29] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-27 23:51:29] ...trap/engines/engine.py.326: process: 0.29244 | load: 0.00014 [I 2022-10-27 23:51:32] ...trap/engines/engine.py.321: eval: epoch 94 | batch 160/510 [I 2022-10-27 23:51:32] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-27 23:51:32] ...trap/engines/engine.py.326: process: 0.28437 | load: 0.00026 [I 2022-10-27 23:51:35] ...trap/engines/engine.py.321: eval: epoch 94 | batch 170/510 [I 2022-10-27 23:51:35] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:45 [I 2022-10-27 23:51:35] ...trap/engines/engine.py.326: process: 0.29347 | load: 0.00022 [I 2022-10-27 23:51:38] ...trap/engines/engine.py.321: eval: epoch 94 | batch 180/510 [I 2022-10-27 23:51:38] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-27 23:51:38] ...trap/engines/engine.py.326: process: 0.28602 | load: 0.00018 [I 2022-10-27 23:51:41] ...trap/engines/engine.py.321: eval: epoch 94 | batch 190/510 [I 2022-10-27 23:51:41] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:33 [I 2022-10-27 23:51:41] ...trap/engines/engine.py.326: process: 0.30496 | load: 0.00015 [I 2022-10-27 23:51:44] ...trap/engines/engine.py.321: eval: epoch 94 | batch 200/510 [I 2022-10-27 23:51:44] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:30 [I 2022-10-27 23:51:44] ...trap/engines/engine.py.326: process: 0.28665 | load: 0.00028 [I 2022-10-27 23:51:47] ...trap/engines/engine.py.321: eval: epoch 94 | batch 210/510 [I 2022-10-27 23:51:47] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:27 [I 2022-10-27 23:51:47] ...trap/engines/engine.py.326: process: 0.29586 | load: 0.00016 [I 2022-10-27 23:51:50] ...trap/engines/engine.py.321: eval: epoch 94 | batch 220/510 [I 2022-10-27 23:51:50] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-27 23:51:50] ...trap/engines/engine.py.326: process: 0.29399 | load: 0.00026 [I 2022-10-27 23:51:53] ...trap/engines/engine.py.321: eval: epoch 94 | batch 230/510 [I 2022-10-27 23:51:53] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-27 23:51:53] ...trap/engines/engine.py.326: process: 0.28601 | load: 0.00016 [I 2022-10-27 23:51:56] ...trap/engines/engine.py.321: eval: epoch 94 | batch 240/510 [I 2022-10-27 23:51:56] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-27 23:51:56] ...trap/engines/engine.py.326: process: 0.28740 | load: 0.00022 [I 2022-10-27 23:51:59] ...trap/engines/engine.py.321: eval: epoch 94 | batch 250/510 [I 2022-10-27 23:51:59] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-27 23:51:59] ...trap/engines/engine.py.326: process: 0.28640 | load: 0.00017 [I 2022-10-27 23:52:01] ...trap/engines/engine.py.321: eval: epoch 94 | batch 260/510 [I 2022-10-27 23:52:01] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-27 23:52:01] ...trap/engines/engine.py.326: process: 0.30010 | load: 0.00018 [I 2022-10-27 23:52:04] ...trap/engines/engine.py.321: eval: epoch 94 | batch 270/510 [I 2022-10-27 23:52:04] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-27 23:52:04] ...trap/engines/engine.py.326: process: 0.29035 | load: 0.00028 [I 2022-10-27 23:52:07] ...trap/engines/engine.py.321: eval: epoch 94 | batch 280/510 [I 2022-10-27 23:52:07] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-27 23:52:07] ...trap/engines/engine.py.326: process: 0.28490 | load: 0.00023 [I 2022-10-27 23:52:10] ...trap/engines/engine.py.321: eval: epoch 94 | batch 290/510 [I 2022-10-27 23:52:10] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-27 23:52:10] ...trap/engines/engine.py.326: process: 0.28870 | load: 0.00016 [I 2022-10-27 23:52:13] ...trap/engines/engine.py.321: eval: epoch 94 | batch 300/510 [I 2022-10-27 23:52:13] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-27 23:52:13] ...trap/engines/engine.py.326: process: 0.28539 | load: 0.00017 [I 2022-10-27 23:52:16] ...trap/engines/engine.py.321: eval: epoch 94 | batch 310/510 [I 2022-10-27 23:52:16] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-27 23:52:16] ...trap/engines/engine.py.326: process: 0.29298 | load: 0.00017 [I 2022-10-27 23:52:19] ...trap/engines/engine.py.321: eval: epoch 94 | batch 320/510 [I 2022-10-27 23:52:19] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-27 23:52:19] ...trap/engines/engine.py.326: process: 0.28474 | load: 0.00017 [I 2022-10-27 23:52:22] ...trap/engines/engine.py.321: eval: epoch 94 | batch 330/510 [I 2022-10-27 23:52:22] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-27 23:52:22] ...trap/engines/engine.py.326: process: 0.28206 | load: 0.00015 [I 2022-10-27 23:52:25] ...trap/engines/engine.py.321: eval: epoch 94 | batch 340/510 [I 2022-10-27 23:52:25] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-27 23:52:25] ...trap/engines/engine.py.326: process: 0.29428 | load: 0.00014 [I 2022-10-27 23:52:27] ...trap/engines/engine.py.321: eval: epoch 94 | batch 350/510 [I 2022-10-27 23:52:27] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-27 23:52:27] ...trap/engines/engine.py.326: process: 0.29127 | load: 0.00019 [I 2022-10-27 23:52:30] ...trap/engines/engine.py.321: eval: epoch 94 | batch 360/510 [I 2022-10-27 23:52:30] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-27 23:52:30] ...trap/engines/engine.py.326: process: 0.28721 | load: 0.00021 [I 2022-10-27 23:52:33] ...trap/engines/engine.py.321: eval: epoch 94 | batch 370/510 [I 2022-10-27 23:52:33] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-27 23:52:33] ...trap/engines/engine.py.326: process: 0.28037 | load: 0.00019 [I 2022-10-27 23:52:36] ...trap/engines/engine.py.321: eval: epoch 94 | batch 380/510 [I 2022-10-27 23:52:36] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-27 23:52:36] ...trap/engines/engine.py.326: process: 0.29337 | load: 0.00021 [I 2022-10-27 23:52:39] ...trap/engines/engine.py.321: eval: epoch 94 | batch 390/510 [I 2022-10-27 23:52:39] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-27 23:52:39] ...trap/engines/engine.py.326: process: 0.28259 | load: 0.00023 [I 2022-10-27 23:52:42] ...trap/engines/engine.py.321: eval: epoch 94 | batch 400/510 [I 2022-10-27 23:52:42] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-27 23:52:42] ...trap/engines/engine.py.326: process: 0.28489 | load: 0.00017 [I 2022-10-27 23:52:45] ...trap/engines/engine.py.321: eval: epoch 94 | batch 410/510 [I 2022-10-27 23:52:45] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:29 [I 2022-10-27 23:52:45] ...trap/engines/engine.py.326: process: 0.29322 | load: 0.00028 [I 2022-10-27 23:52:48] ...trap/engines/engine.py.321: eval: epoch 94 | batch 420/510 [I 2022-10-27 23:52:48] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-27 23:52:48] ...trap/engines/engine.py.326: process: 0.28634 | load: 0.00019 [I 2022-10-27 23:52:51] ...trap/engines/engine.py.321: eval: epoch 94 | batch 430/510 [I 2022-10-27 23:52:51] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-27 23:52:51] ...trap/engines/engine.py.326: process: 0.28742 | load: 0.00028 [I 2022-10-27 23:52:53] ...trap/engines/engine.py.321: eval: epoch 94 | batch 440/510 [I 2022-10-27 23:52:53] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-27 23:52:53] ...trap/engines/engine.py.326: process: 0.29145 | load: 0.00016 [I 2022-10-27 23:52:56] ...trap/engines/engine.py.321: eval: epoch 94 | batch 450/510 [I 2022-10-27 23:52:56] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-27 23:52:56] ...trap/engines/engine.py.326: process: 0.28924 | load: 0.00020 [I 2022-10-27 23:52:59] ...trap/engines/engine.py.321: eval: epoch 94 | batch 460/510 [I 2022-10-27 23:52:59] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-27 23:52:59] ...trap/engines/engine.py.326: process: 0.29159 | load: 0.00027 [I 2022-10-27 23:53:02] ...trap/engines/engine.py.321: eval: epoch 94 | batch 470/510 [I 2022-10-27 23:53:02] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-27 23:53:02] ...trap/engines/engine.py.326: process: 0.28544 | load: 0.00017 [I 2022-10-27 23:53:05] ...trap/engines/engine.py.321: eval: epoch 94 | batch 480/510 [I 2022-10-27 23:53:05] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-27 23:53:05] ...trap/engines/engine.py.326: process: 0.28877 | load: 0.00018 [I 2022-10-27 23:53:08] ...trap/engines/engine.py.321: eval: epoch 94 | batch 490/510 [I 2022-10-27 23:53:08] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-27 23:53:08] ...trap/engines/engine.py.326: process: 0.28888 | load: 0.00017 [I 2022-10-27 23:53:11] ...trap/engines/engine.py.321: eval: epoch 94 | batch 500/510 [I 2022-10-27 23:53:11] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-27 23:53:11] ...trap/engines/engine.py.326: process: 0.28180 | load: 0.00017 [I 2022-10-27 23:53:14] ...trap/engines/engine.py.321: eval: epoch 94 | batch 510/510 [I 2022-10-27 23:53:14] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-27 23:53:14] ...trap/engines/engine.py.326: process: 0.29021 | load: 0.00013 [S 2022-10-27 23:53:14] ...trap/engines/engine.py.132: eval_epoch.epoch: 94 [I 2022-10-27 23:53:14] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.725800 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.902600 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.930200 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.724200 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.899700 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.929700 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.016111 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008581 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.003816 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016851 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005311 [S 2022-10-27 23:53:18] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005100 [I 2022-10-27 23:53:20] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-27 23:53:20] ...trap/engines/engine.py.470: Saving model... [W 2022-10-27 23:53:21] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-27 23:53:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-27 23:53:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-27 23:53:25] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-27 23:53:26] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-27 23:53:32] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-27 23:53:32] ...trap/engines/engine.py.163: Training model on trainset for epoch 95 [I 2022-10-27 23:53:32] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-27 23:53:35] ...trap/engines/engine.py.224: train: epoch 95 | batch 0/2304 [I 2022-10-27 23:53:35] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:07 [I 2022-10-27 23:53:35] ...trap/engines/engine.py.229: process: 2.76379 | load: 2.11798 [I 2022-10-27 23:53:35] ...trap/engines/engine.py.230: loss: 0.11149 [I 2022-10-27 23:53:35] ...optimizers/trijoint.py.151: total_norm: 19.701754 [I 2022-10-27 23:53:40] ...trap/engines/engine.py.224: train: epoch 95 | batch 10/2304 [I 2022-10-27 23:53:40] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:26 [I 2022-10-27 23:53:40] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00453 [I 2022-10-27 23:53:40] ...trap/engines/engine.py.230: loss: 0.09986 [I 2022-10-27 23:53:40] ...optimizers/trijoint.py.151: total_norm: 8.792616 [I 2022-10-27 23:53:46] ...trap/engines/engine.py.224: train: epoch 95 | batch 20/2304 [I 2022-10-27 23:53:46] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:19 [I 2022-10-27 23:53:46] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00388 [I 2022-10-27 23:53:46] ...trap/engines/engine.py.230: loss: 0.10198 [I 2022-10-27 23:53:46] ...optimizers/trijoint.py.151: total_norm: 11.361121 [I 2022-10-27 23:53:52] ...trap/engines/engine.py.224: train: epoch 95 | batch 30/2304 [I 2022-10-27 23:53:52] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:18 [I 2022-10-27 23:53:52] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00371 [I 2022-10-27 23:53:52] ...trap/engines/engine.py.230: loss: 0.08361 [I 2022-10-27 23:53:52] ...optimizers/trijoint.py.151: total_norm: 14.952423 [I 2022-10-27 23:53:57] ...trap/engines/engine.py.224: train: epoch 95 | batch 40/2304 [I 2022-10-27 23:53:57] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:03 [I 2022-10-27 23:53:57] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00314 [I 2022-10-27 23:53:57] ...trap/engines/engine.py.230: loss: 0.13003 [I 2022-10-27 23:53:57] ...optimizers/trijoint.py.151: total_norm: 9.501714 [I 2022-10-27 23:54:03] ...trap/engines/engine.py.224: train: epoch 95 | batch 50/2304 [I 2022-10-27 23:54:03] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:48 [I 2022-10-27 23:54:03] ...trap/engines/engine.py.229: process: 0.55154 | load: 0.00288 [I 2022-10-27 23:54:03] ...trap/engines/engine.py.230: loss: 0.13538 [I 2022-10-27 23:54:03] ...optimizers/trijoint.py.151: total_norm: 9.246305 [I 2022-10-27 23:54:08] ...trap/engines/engine.py.224: train: epoch 95 | batch 60/2304 [I 2022-10-27 23:54:08] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:48 [I 2022-10-27 23:54:08] ...trap/engines/engine.py.229: process: 0.55101 | load: 0.00292 [I 2022-10-27 23:54:08] ...trap/engines/engine.py.230: loss: 0.11834 [I 2022-10-27 23:54:08] ...optimizers/trijoint.py.151: total_norm: 14.805386 [I 2022-10-27 23:54:14] ...trap/engines/engine.py.224: train: epoch 95 | batch 70/2304 [I 2022-10-27 23:54:14] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:44 [I 2022-10-27 23:54:14] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00296 [I 2022-10-27 23:54:14] ...trap/engines/engine.py.230: loss: 0.14820 [I 2022-10-27 23:54:14] ...optimizers/trijoint.py.151: total_norm: 26.820307 [I 2022-10-27 23:54:20] ...trap/engines/engine.py.224: train: epoch 95 | batch 80/2304 [I 2022-10-27 23:54:20] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:41 [I 2022-10-27 23:54:20] ...trap/engines/engine.py.229: process: 0.56229 | load: 0.00296 [I 2022-10-27 23:54:20] ...trap/engines/engine.py.230: loss: 0.12157 [I 2022-10-27 23:54:20] ...optimizers/trijoint.py.151: total_norm: 14.456285 [I 2022-10-27 23:54:25] ...trap/engines/engine.py.224: train: epoch 95 | batch 90/2304 [I 2022-10-27 23:54:25] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:32 [I 2022-10-27 23:54:25] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00261 [I 2022-10-27 23:54:25] ...trap/engines/engine.py.230: loss: 0.13540 [I 2022-10-27 23:54:25] ...optimizers/trijoint.py.151: total_norm: 12.139252 [I 2022-10-27 23:54:31] ...trap/engines/engine.py.224: train: epoch 95 | batch 100/2304 [I 2022-10-27 23:54:31] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:36 [I 2022-10-27 23:54:31] ...trap/engines/engine.py.229: process: 0.57011 | load: 0.00258 [I 2022-10-27 23:54:31] ...trap/engines/engine.py.230: loss: 0.09695 [I 2022-10-27 23:54:31] ...optimizers/trijoint.py.151: total_norm: 10.497059 [I 2022-10-27 23:54:36] ...trap/engines/engine.py.224: train: epoch 95 | batch 110/2304 [I 2022-10-27 23:54:36] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:28 [I 2022-10-27 23:54:36] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00264 [I 2022-10-27 23:54:36] ...trap/engines/engine.py.230: loss: 0.09627 [I 2022-10-27 23:54:36] ...optimizers/trijoint.py.151: total_norm: 9.484614 [I 2022-10-27 23:54:42] ...trap/engines/engine.py.224: train: epoch 95 | batch 120/2304 [I 2022-10-27 23:54:42] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:14 [I 2022-10-27 23:54:42] ...trap/engines/engine.py.229: process: 0.54938 | load: 0.00276 [I 2022-10-27 23:54:42] ...trap/engines/engine.py.230: loss: 0.25596 [I 2022-10-27 23:54:42] ...optimizers/trijoint.py.151: total_norm: 15.637084 [I 2022-10-27 23:54:47] ...trap/engines/engine.py.224: train: epoch 95 | batch 130/2304 [I 2022-10-27 23:54:47] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:08 [I 2022-10-27 23:54:47] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00261 [I 2022-10-27 23:54:47] ...trap/engines/engine.py.230: loss: 0.13947 [I 2022-10-27 23:54:47] ...optimizers/trijoint.py.151: total_norm: 10.722799 [I 2022-10-27 23:54:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 140/2304 [I 2022-10-27 23:54:53] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:01 [I 2022-10-27 23:54:53] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00258 [I 2022-10-27 23:54:53] ...trap/engines/engine.py.230: loss: 0.11384 [I 2022-10-27 23:54:53] ...optimizers/trijoint.py.151: total_norm: 8.435098 [I 2022-10-27 23:54:59] ...trap/engines/engine.py.224: train: epoch 95 | batch 150/2304 [I 2022-10-27 23:54:59] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:04 [I 2022-10-27 23:54:59] ...trap/engines/engine.py.229: process: 0.56419 | load: 0.00272 [I 2022-10-27 23:54:59] ...trap/engines/engine.py.230: loss: 0.10451 [I 2022-10-27 23:54:59] ...optimizers/trijoint.py.151: total_norm: 14.558169 [I 2022-10-27 23:55:04] ...trap/engines/engine.py.224: train: epoch 95 | batch 160/2304 [I 2022-10-27 23:55:04] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:01 [I 2022-10-27 23:55:04] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00275 [I 2022-10-27 23:55:04] ...trap/engines/engine.py.230: loss: 0.13141 [I 2022-10-27 23:55:04] ...optimizers/trijoint.py.151: total_norm: 13.375217 [I 2022-10-27 23:55:10] ...trap/engines/engine.py.224: train: epoch 95 | batch 170/2304 [I 2022-10-27 23:55:10] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:41 [I 2022-10-27 23:55:10] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00252 [I 2022-10-27 23:55:10] ...trap/engines/engine.py.230: loss: 0.15258 [I 2022-10-27 23:55:10] ...optimizers/trijoint.py.151: total_norm: 23.827349 [I 2022-10-27 23:55:15] ...trap/engines/engine.py.224: train: epoch 95 | batch 180/2304 [I 2022-10-27 23:55:15] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:33 [I 2022-10-27 23:55:15] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00247 [I 2022-10-27 23:55:15] ...trap/engines/engine.py.230: loss: 0.09797 [I 2022-10-27 23:55:15] ...optimizers/trijoint.py.151: total_norm: 9.689691 [I 2022-10-27 23:55:21] ...trap/engines/engine.py.224: train: epoch 95 | batch 190/2304 [I 2022-10-27 23:55:21] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:32 [I 2022-10-27 23:55:21] ...trap/engines/engine.py.229: process: 0.56619 | load: 0.00251 [I 2022-10-27 23:55:21] ...trap/engines/engine.py.230: loss: 0.15549 [I 2022-10-27 23:55:21] ...optimizers/trijoint.py.151: total_norm: 51.299072 [I 2022-10-27 23:55:26] ...trap/engines/engine.py.224: train: epoch 95 | batch 200/2304 [I 2022-10-27 23:55:26] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:29 [I 2022-10-27 23:55:26] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00251 [I 2022-10-27 23:55:26] ...trap/engines/engine.py.230: loss: 0.12291 [I 2022-10-27 23:55:26] ...optimizers/trijoint.py.151: total_norm: 36.063545 [I 2022-10-27 23:55:32] ...trap/engines/engine.py.224: train: epoch 95 | batch 210/2304 [I 2022-10-27 23:55:32] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:28 [I 2022-10-27 23:55:32] ...trap/engines/engine.py.229: process: 0.55369 | load: 0.00247 [I 2022-10-27 23:55:32] ...trap/engines/engine.py.230: loss: 0.13931 [I 2022-10-27 23:55:32] ...optimizers/trijoint.py.151: total_norm: 12.722425 [I 2022-10-27 23:55:38] ...trap/engines/engine.py.224: train: epoch 95 | batch 220/2304 [I 2022-10-27 23:55:38] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:15 [I 2022-10-27 23:55:38] ...trap/engines/engine.py.229: process: 0.54437 | load: 0.00259 [I 2022-10-27 23:55:38] ...trap/engines/engine.py.230: loss: 0.10560 [I 2022-10-27 23:55:38] ...optimizers/trijoint.py.151: total_norm: 29.364031 [I 2022-10-27 23:55:43] ...trap/engines/engine.py.224: train: epoch 95 | batch 230/2304 [I 2022-10-27 23:55:43] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:19 [I 2022-10-27 23:55:43] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00257 [I 2022-10-27 23:55:43] ...trap/engines/engine.py.230: loss: 0.09720 [I 2022-10-27 23:55:43] ...optimizers/trijoint.py.151: total_norm: 7.112950 [I 2022-10-27 23:55:49] ...trap/engines/engine.py.224: train: epoch 95 | batch 240/2304 [I 2022-10-27 23:55:49] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:05 [I 2022-10-27 23:55:49] ...trap/engines/engine.py.229: process: 0.54904 | load: 0.00245 [I 2022-10-27 23:55:49] ...trap/engines/engine.py.230: loss: 0.13324 [I 2022-10-27 23:55:49] ...optimizers/trijoint.py.151: total_norm: 12.218020 [I 2022-10-27 23:55:54] ...trap/engines/engine.py.224: train: epoch 95 | batch 250/2304 [I 2022-10-27 23:55:54] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:05 [I 2022-10-27 23:55:54] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00245 [I 2022-10-27 23:55:54] ...trap/engines/engine.py.230: loss: 0.11418 [I 2022-10-27 23:55:54] ...optimizers/trijoint.py.151: total_norm: 15.713127 [I 2022-10-27 23:56:00] ...trap/engines/engine.py.224: train: epoch 95 | batch 260/2304 [I 2022-10-27 23:56:00] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:48 [I 2022-10-27 23:56:00] ...trap/engines/engine.py.229: process: 0.54096 | load: 0.00258 [I 2022-10-27 23:56:00] ...trap/engines/engine.py.230: loss: 0.09203 [I 2022-10-27 23:56:00] ...optimizers/trijoint.py.151: total_norm: 9.426709 [I 2022-10-27 23:56:06] ...trap/engines/engine.py.224: train: epoch 95 | batch 270/2304 [I 2022-10-27 23:56:06] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:53 [I 2022-10-27 23:56:06] ...trap/engines/engine.py.229: process: 0.54806 | load: 0.00244 [I 2022-10-27 23:56:06] ...trap/engines/engine.py.230: loss: 0.09805 [I 2022-10-27 23:56:06] ...optimizers/trijoint.py.151: total_norm: 8.372991 [I 2022-10-27 23:56:11] ...trap/engines/engine.py.224: train: epoch 95 | batch 280/2304 [I 2022-10-27 23:56:11] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:53 [I 2022-10-27 23:56:11] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00261 [I 2022-10-27 23:56:11] ...trap/engines/engine.py.230: loss: 0.10490 [I 2022-10-27 23:56:11] ...optimizers/trijoint.py.151: total_norm: 10.179895 [I 2022-10-27 23:56:17] ...trap/engines/engine.py.224: train: epoch 95 | batch 290/2304 [I 2022-10-27 23:56:17] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:50 [I 2022-10-27 23:56:17] ...trap/engines/engine.py.229: process: 0.55845 | load: 0.00256 [I 2022-10-27 23:56:17] ...trap/engines/engine.py.230: loss: 0.09432 [I 2022-10-27 23:56:17] ...optimizers/trijoint.py.151: total_norm: 10.541143 [I 2022-10-27 23:56:22] ...trap/engines/engine.py.224: train: epoch 95 | batch 300/2304 [I 2022-10-27 23:56:22] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:27 [I 2022-10-27 23:56:22] ...trap/engines/engine.py.229: process: 0.54294 | load: 0.00259 [I 2022-10-27 23:56:22] ...trap/engines/engine.py.230: loss: 0.17416 [I 2022-10-27 23:56:22] ...optimizers/trijoint.py.151: total_norm: 17.375887 [I 2022-10-27 23:56:28] ...trap/engines/engine.py.224: train: epoch 95 | batch 310/2304 [I 2022-10-27 23:56:28] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:49 [I 2022-10-27 23:56:28] ...trap/engines/engine.py.229: process: 0.56625 | load: 0.00247 [I 2022-10-27 23:56:28] ...trap/engines/engine.py.230: loss: 0.09418 [I 2022-10-27 23:56:28] ...optimizers/trijoint.py.151: total_norm: 20.941204 [I 2022-10-27 23:56:34] ...trap/engines/engine.py.224: train: epoch 95 | batch 320/2304 [I 2022-10-27 23:56:34] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:22 [I 2022-10-27 23:56:34] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00248 [I 2022-10-27 23:56:34] ...trap/engines/engine.py.230: loss: 0.09608 [I 2022-10-27 23:56:34] ...optimizers/trijoint.py.151: total_norm: 13.952843 [I 2022-10-27 23:56:39] ...trap/engines/engine.py.224: train: epoch 95 | batch 330/2304 [I 2022-10-27 23:56:39] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:35 [I 2022-10-27 23:56:39] ...trap/engines/engine.py.229: process: 0.56675 | load: 0.00246 [I 2022-10-27 23:56:39] ...trap/engines/engine.py.230: loss: 0.08599 [I 2022-10-27 23:56:39] ...optimizers/trijoint.py.151: total_norm: 15.655809 [I 2022-10-27 23:56:45] ...trap/engines/engine.py.224: train: epoch 95 | batch 340/2304 [I 2022-10-27 23:56:45] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:14 [I 2022-10-27 23:56:45] ...trap/engines/engine.py.229: process: 0.55164 | load: 0.00248 [I 2022-10-27 23:56:45] ...trap/engines/engine.py.230: loss: 0.15447 [I 2022-10-27 23:56:45] ...optimizers/trijoint.py.151: total_norm: 19.760576 [I 2022-10-27 23:56:50] ...trap/engines/engine.py.224: train: epoch 95 | batch 350/2304 [I 2022-10-27 23:56:50] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:10 [I 2022-10-27 23:56:50] ...trap/engines/engine.py.229: process: 0.56859 | load: 0.00261 [I 2022-10-27 23:56:50] ...trap/engines/engine.py.230: loss: 0.16049 [I 2022-10-27 23:56:50] ...optimizers/trijoint.py.151: total_norm: 20.616556 [I 2022-10-27 23:56:56] ...trap/engines/engine.py.224: train: epoch 95 | batch 360/2304 [I 2022-10-27 23:56:56] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:01 [I 2022-10-27 23:56:56] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00263 [I 2022-10-27 23:56:56] ...trap/engines/engine.py.230: loss: 0.14631 [I 2022-10-27 23:56:56] ...optimizers/trijoint.py.151: total_norm: 94.094215 [I 2022-10-27 23:57:02] ...trap/engines/engine.py.224: train: epoch 95 | batch 370/2304 [I 2022-10-27 23:57:02] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:04 [I 2022-10-27 23:57:02] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00248 [I 2022-10-27 23:57:02] ...trap/engines/engine.py.230: loss: 0.07249 [I 2022-10-27 23:57:02] ...optimizers/trijoint.py.151: total_norm: 28.106806 [I 2022-10-27 23:57:07] ...trap/engines/engine.py.224: train: epoch 95 | batch 380/2304 [I 2022-10-27 23:57:07] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:49 [I 2022-10-27 23:57:07] ...trap/engines/engine.py.229: process: 0.55492 | load: 0.00253 [I 2022-10-27 23:57:07] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-27 23:57:07] ...optimizers/trijoint.py.151: total_norm: 17.498331 [I 2022-10-27 23:57:13] ...trap/engines/engine.py.224: train: epoch 95 | batch 390/2304 [I 2022-10-27 23:57:13] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:44 [I 2022-10-27 23:57:13] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00250 [I 2022-10-27 23:57:13] ...trap/engines/engine.py.230: loss: 0.13560 [I 2022-10-27 23:57:13] ...optimizers/trijoint.py.151: total_norm: 10.580732 [I 2022-10-27 23:57:18] ...trap/engines/engine.py.224: train: epoch 95 | batch 400/2304 [I 2022-10-27 23:57:18] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:54 [I 2022-10-27 23:57:18] ...trap/engines/engine.py.229: process: 0.56737 | load: 0.00243 [I 2022-10-27 23:57:18] ...trap/engines/engine.py.230: loss: 0.10446 [I 2022-10-27 23:57:18] ...optimizers/trijoint.py.151: total_norm: 15.901386 [I 2022-10-27 23:57:24] ...trap/engines/engine.py.224: train: epoch 95 | batch 410/2304 [I 2022-10-27 23:57:24] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:37 [I 2022-10-27 23:57:24] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00256 [I 2022-10-27 23:57:24] ...trap/engines/engine.py.230: loss: 0.11207 [I 2022-10-27 23:57:24] ...optimizers/trijoint.py.151: total_norm: 11.304919 [I 2022-10-27 23:57:30] ...trap/engines/engine.py.224: train: epoch 95 | batch 420/2304 [I 2022-10-27 23:57:30] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:26 [I 2022-10-27 23:57:30] ...trap/engines/engine.py.229: process: 0.54607 | load: 0.00241 [I 2022-10-27 23:57:30] ...trap/engines/engine.py.230: loss: 0.08428 [I 2022-10-27 23:57:30] ...optimizers/trijoint.py.151: total_norm: 11.997158 [I 2022-10-27 23:57:35] ...trap/engines/engine.py.224: train: epoch 95 | batch 430/2304 [I 2022-10-27 23:57:35] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:27 [I 2022-10-27 23:57:35] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00251 [I 2022-10-27 23:57:35] ...trap/engines/engine.py.230: loss: 0.16287 [I 2022-10-27 23:57:35] ...optimizers/trijoint.py.151: total_norm: 12.663672 [I 2022-10-27 23:57:41] ...trap/engines/engine.py.224: train: epoch 95 | batch 440/2304 [I 2022-10-27 23:57:41] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:18 [I 2022-10-27 23:57:41] ...trap/engines/engine.py.229: process: 0.54445 | load: 0.00276 [I 2022-10-27 23:57:41] ...trap/engines/engine.py.230: loss: 0.10561 [I 2022-10-27 23:57:41] ...optimizers/trijoint.py.151: total_norm: 8.426059 [I 2022-10-27 23:57:46] ...trap/engines/engine.py.224: train: epoch 95 | batch 450/2304 [I 2022-10-27 23:57:46] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:17 [I 2022-10-27 23:57:46] ...trap/engines/engine.py.229: process: 0.56221 | load: 0.00256 [I 2022-10-27 23:57:46] ...trap/engines/engine.py.230: loss: 0.19102 [I 2022-10-27 23:57:46] ...optimizers/trijoint.py.151: total_norm: 14.435740 [I 2022-10-27 23:57:52] ...trap/engines/engine.py.224: train: epoch 95 | batch 460/2304 [I 2022-10-27 23:57:52] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:00 [I 2022-10-27 23:57:52] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00255 [I 2022-10-27 23:57:52] ...trap/engines/engine.py.230: loss: 0.15046 [I 2022-10-27 23:57:52] ...optimizers/trijoint.py.151: total_norm: 21.064470 [I 2022-10-27 23:57:58] ...trap/engines/engine.py.224: train: epoch 95 | batch 470/2304 [I 2022-10-27 23:57:58] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:11 [I 2022-10-27 23:57:58] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00251 [I 2022-10-27 23:57:58] ...trap/engines/engine.py.230: loss: 0.15683 [I 2022-10-27 23:57:58] ...optimizers/trijoint.py.151: total_norm: 22.995153 [I 2022-10-27 23:58:03] ...trap/engines/engine.py.224: train: epoch 95 | batch 480/2304 [I 2022-10-27 23:58:03] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:49 [I 2022-10-27 23:58:03] ...trap/engines/engine.py.229: process: 0.55518 | load: 0.00247 [I 2022-10-27 23:58:03] ...trap/engines/engine.py.230: loss: 0.12700 [I 2022-10-27 23:58:03] ...optimizers/trijoint.py.151: total_norm: 18.799107 [I 2022-10-27 23:58:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 490/2304 [I 2022-10-27 23:58:09] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:51 [I 2022-10-27 23:58:09] ...trap/engines/engine.py.229: process: 0.56280 | load: 0.00254 [I 2022-10-27 23:58:09] ...trap/engines/engine.py.230: loss: 0.15512 [I 2022-10-27 23:58:09] ...optimizers/trijoint.py.151: total_norm: 21.420799 [I 2022-10-27 23:58:14] ...trap/engines/engine.py.224: train: epoch 95 | batch 500/2304 [I 2022-10-27 23:58:14] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:46 [I 2022-10-27 23:58:14] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00243 [I 2022-10-27 23:58:14] ...trap/engines/engine.py.230: loss: 0.13157 [I 2022-10-27 23:58:14] ...optimizers/trijoint.py.151: total_norm: 14.346009 [I 2022-10-27 23:58:20] ...trap/engines/engine.py.224: train: epoch 95 | batch 510/2304 [I 2022-10-27 23:58:20] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:35 [I 2022-10-27 23:58:20] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00249 [I 2022-10-27 23:58:20] ...trap/engines/engine.py.230: loss: 0.09711 [I 2022-10-27 23:58:20] ...optimizers/trijoint.py.151: total_norm: 17.432684 [I 2022-10-27 23:58:26] ...trap/engines/engine.py.224: train: epoch 95 | batch 520/2304 [I 2022-10-27 23:58:26] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:34 [I 2022-10-27 23:58:26] ...trap/engines/engine.py.229: process: 0.56245 | load: 0.00250 [I 2022-10-27 23:58:26] ...trap/engines/engine.py.230: loss: 0.13352 [I 2022-10-27 23:58:26] ...optimizers/trijoint.py.151: total_norm: 12.711024 [I 2022-10-27 23:58:31] ...trap/engines/engine.py.224: train: epoch 95 | batch 530/2304 [I 2022-10-27 23:58:31] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:22 [I 2022-10-27 23:58:31] ...trap/engines/engine.py.229: process: 0.55058 | load: 0.00259 [I 2022-10-27 23:58:31] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-27 23:58:31] ...optimizers/trijoint.py.151: total_norm: 10.621565 [I 2022-10-27 23:58:37] ...trap/engines/engine.py.224: train: epoch 95 | batch 540/2304 [I 2022-10-27 23:58:37] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:25 [I 2022-10-27 23:58:37] ...trap/engines/engine.py.229: process: 0.56450 | load: 0.00246 [I 2022-10-27 23:58:37] ...trap/engines/engine.py.230: loss: 0.12226 [I 2022-10-27 23:58:37] ...optimizers/trijoint.py.151: total_norm: 11.431303 [I 2022-10-27 23:58:42] ...trap/engines/engine.py.224: train: epoch 95 | batch 550/2304 [I 2022-10-27 23:58:42] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:13 [I 2022-10-27 23:58:42] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00246 [I 2022-10-27 23:58:42] ...trap/engines/engine.py.230: loss: 0.22795 [I 2022-10-27 23:58:42] ...optimizers/trijoint.py.151: total_norm: 21.784697 [I 2022-10-27 23:58:48] ...trap/engines/engine.py.224: train: epoch 95 | batch 560/2304 [I 2022-10-27 23:58:48] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:04 [I 2022-10-27 23:58:48] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00244 [I 2022-10-27 23:58:48] ...trap/engines/engine.py.230: loss: 0.11648 [I 2022-10-27 23:58:48] ...optimizers/trijoint.py.151: total_norm: 17.576084 [I 2022-10-27 23:59:14] ...trap/engines/engine.py.224: train: epoch 95 | batch 570/2304 [I 2022-10-27 23:59:14] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 2:15:11 [I 2022-10-27 23:59:14] ...trap/engines/engine.py.229: process: 21.15806 | load: 0.00244 [I 2022-10-27 23:59:14] ...trap/engines/engine.py.230: loss: 0.08776 [I 2022-10-27 23:59:14] ...optimizers/trijoint.py.151: total_norm: 9.951569 [I 2022-10-27 23:59:20] ...trap/engines/engine.py.224: train: epoch 95 | batch 580/2304 [I 2022-10-27 23:59:20] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:28:36 [I 2022-10-27 23:59:20] ...trap/engines/engine.py.229: process: 0.55191 | load: 0.00237 [I 2022-10-27 23:59:20] ...trap/engines/engine.py.230: loss: 0.10240 [I 2022-10-27 23:59:20] ...optimizers/trijoint.py.151: total_norm: 24.721643 [I 2022-10-27 23:59:25] ...trap/engines/engine.py.224: train: epoch 95 | batch 590/2304 [I 2022-10-27 23:59:25] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:17:09 [I 2022-10-27 23:59:25] ...trap/engines/engine.py.229: process: 0.55496 | load: 0.00245 [I 2022-10-27 23:59:25] ...trap/engines/engine.py.230: loss: 0.10823 [I 2022-10-27 23:59:25] ...optimizers/trijoint.py.151: total_norm: 20.628349 [I 2022-10-27 23:59:31] ...trap/engines/engine.py.224: train: epoch 95 | batch 600/2304 [I 2022-10-27 23:59:31] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:16:01 [I 2022-10-27 23:59:31] ...trap/engines/engine.py.229: process: 0.56224 | load: 0.00235 [I 2022-10-27 23:59:31] ...trap/engines/engine.py.230: loss: 0.13488 [I 2022-10-27 23:59:31] ...optimizers/trijoint.py.151: total_norm: 14.103296 [I 2022-10-27 23:59:36] ...trap/engines/engine.py.224: train: epoch 95 | batch 610/2304 [I 2022-10-27 23:59:36] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:44 [I 2022-10-27 23:59:36] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00234 [I 2022-10-27 23:59:36] ...trap/engines/engine.py.230: loss: 0.17612 [I 2022-10-27 23:59:36] ...optimizers/trijoint.py.151: total_norm: 12.046335 [I 2022-10-27 23:59:42] ...trap/engines/engine.py.224: train: epoch 95 | batch 620/2304 [I 2022-10-27 23:59:42] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:37 [I 2022-10-27 23:59:42] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00244 [I 2022-10-27 23:59:42] ...trap/engines/engine.py.230: loss: 0.16233 [I 2022-10-27 23:59:42] ...optimizers/trijoint.py.151: total_norm: 17.723335 [I 2022-10-27 23:59:47] ...trap/engines/engine.py.224: train: epoch 95 | batch 630/2304 [I 2022-10-27 23:59:47] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:22 [I 2022-10-27 23:59:47] ...trap/engines/engine.py.229: process: 0.54169 | load: 0.00242 [I 2022-10-27 23:59:47] ...trap/engines/engine.py.230: loss: 0.19334 [I 2022-10-27 23:59:47] ...optimizers/trijoint.py.151: total_norm: 13.816638 [I 2022-10-27 23:59:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 640/2304 [I 2022-10-27 23:59:53] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:23 [I 2022-10-27 23:59:53] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00241 [I 2022-10-27 23:59:53] ...trap/engines/engine.py.230: loss: 0.10910 [I 2022-10-27 23:59:53] ...optimizers/trijoint.py.151: total_norm: 17.134417 [I 2022-10-27 23:59:59] ...trap/engines/engine.py.224: train: epoch 95 | batch 650/2304 [I 2022-10-27 23:59:59] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:23 [I 2022-10-27 23:59:59] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00234 [I 2022-10-27 23:59:59] ...trap/engines/engine.py.230: loss: 0.13971 [I 2022-10-27 23:59:59] ...optimizers/trijoint.py.151: total_norm: 25.952614 [I 2022-10-28 00:00:04] ...trap/engines/engine.py.224: train: epoch 95 | batch 660/2304 [I 2022-10-28 00:00:04] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:15:22 [I 2022-10-28 00:00:04] ...trap/engines/engine.py.229: process: 0.55819 | load: 0.00235 [I 2022-10-28 00:00:04] ...trap/engines/engine.py.230: loss: 0.12656 [I 2022-10-28 00:00:04] ...optimizers/trijoint.py.151: total_norm: 9.417960 [I 2022-10-28 00:00:10] ...trap/engines/engine.py.224: train: epoch 95 | batch 670/2304 [I 2022-10-28 00:00:10] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:30 [I 2022-10-28 00:00:10] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00236 [I 2022-10-28 00:00:10] ...trap/engines/engine.py.230: loss: 0.18051 [I 2022-10-28 00:00:10] ...optimizers/trijoint.py.151: total_norm: 10.188901 [I 2022-10-28 00:00:15] ...trap/engines/engine.py.224: train: epoch 95 | batch 680/2304 [I 2022-10-28 00:00:15] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:03 [I 2022-10-28 00:00:15] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00236 [I 2022-10-28 00:00:15] ...trap/engines/engine.py.230: loss: 0.10673 [I 2022-10-28 00:00:15] ...optimizers/trijoint.py.151: total_norm: 12.402968 [I 2022-10-28 00:00:21] ...trap/engines/engine.py.224: train: epoch 95 | batch 690/2304 [I 2022-10-28 00:00:21] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:15:00 [I 2022-10-28 00:00:21] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00235 [I 2022-10-28 00:00:21] ...trap/engines/engine.py.230: loss: 0.10942 [I 2022-10-28 00:00:21] ...optimizers/trijoint.py.151: total_norm: 14.949754 [I 2022-10-28 00:00:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 700/2304 [I 2022-10-28 00:00:27] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:14:54 [I 2022-10-28 00:00:27] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00247 [I 2022-10-28 00:00:27] ...trap/engines/engine.py.230: loss: 0.15704 [I 2022-10-28 00:00:27] ...optimizers/trijoint.py.151: total_norm: 18.891298 [I 2022-10-28 00:00:32] ...trap/engines/engine.py.224: train: epoch 95 | batch 710/2304 [I 2022-10-28 00:00:32] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:45 [I 2022-10-28 00:00:32] ...trap/engines/engine.py.229: process: 0.55618 | load: 0.00243 [I 2022-10-28 00:00:32] ...trap/engines/engine.py.230: loss: 0.13357 [I 2022-10-28 00:00:32] ...optimizers/trijoint.py.151: total_norm: 34.879032 [I 2022-10-28 00:00:38] ...trap/engines/engine.py.224: train: epoch 95 | batch 720/2304 [I 2022-10-28 00:00:38] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:42 [I 2022-10-28 00:00:38] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00242 [I 2022-10-28 00:00:38] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-28 00:00:38] ...optimizers/trijoint.py.151: total_norm: 16.619255 [I 2022-10-28 00:00:43] ...trap/engines/engine.py.224: train: epoch 95 | batch 730/2304 [I 2022-10-28 00:00:43] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:33 [I 2022-10-28 00:00:43] ...trap/engines/engine.py.229: process: 0.54937 | load: 0.00241 [I 2022-10-28 00:00:43] ...trap/engines/engine.py.230: loss: 0.12396 [I 2022-10-28 00:00:43] ...optimizers/trijoint.py.151: total_norm: 34.118084 [I 2022-10-28 00:00:49] ...trap/engines/engine.py.224: train: epoch 95 | batch 740/2304 [I 2022-10-28 00:00:49] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:27 [I 2022-10-28 00:00:49] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00239 [I 2022-10-28 00:00:49] ...trap/engines/engine.py.230: loss: 0.12867 [I 2022-10-28 00:00:49] ...optimizers/trijoint.py.151: total_norm: 11.714264 [I 2022-10-28 00:00:55] ...trap/engines/engine.py.224: train: epoch 95 | batch 750/2304 [I 2022-10-28 00:00:55] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:29 [I 2022-10-28 00:00:55] ...trap/engines/engine.py.229: process: 0.56681 | load: 0.00243 [I 2022-10-28 00:00:55] ...trap/engines/engine.py.230: loss: 0.14992 [I 2022-10-28 00:00:55] ...optimizers/trijoint.py.151: total_norm: 13.744020 [I 2022-10-28 00:01:00] ...trap/engines/engine.py.224: train: epoch 95 | batch 760/2304 [I 2022-10-28 00:01:00] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:27 [I 2022-10-28 00:01:00] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00249 [I 2022-10-28 00:01:00] ...trap/engines/engine.py.230: loss: 0.18755 [I 2022-10-28 00:01:00] ...optimizers/trijoint.py.151: total_norm: 13.776013 [I 2022-10-28 00:01:06] ...trap/engines/engine.py.224: train: epoch 95 | batch 770/2304 [I 2022-10-28 00:01:06] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:14:17 [I 2022-10-28 00:01:06] ...trap/engines/engine.py.229: process: 0.56563 | load: 0.00243 [I 2022-10-28 00:01:06] ...trap/engines/engine.py.230: loss: 0.10907 [I 2022-10-28 00:01:06] ...optimizers/trijoint.py.151: total_norm: 39.323303 [I 2022-10-28 00:01:11] ...trap/engines/engine.py.224: train: epoch 95 | batch 780/2304 [I 2022-10-28 00:01:11] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:28 [I 2022-10-28 00:01:11] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00244 [I 2022-10-28 00:01:11] ...trap/engines/engine.py.230: loss: 0.14885 [I 2022-10-28 00:01:11] ...optimizers/trijoint.py.151: total_norm: 15.109868 [I 2022-10-28 00:01:17] ...trap/engines/engine.py.224: train: epoch 95 | batch 790/2304 [I 2022-10-28 00:01:17] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:14:08 [I 2022-10-28 00:01:17] ...trap/engines/engine.py.229: process: 0.56165 | load: 0.00259 [I 2022-10-28 00:01:17] ...trap/engines/engine.py.230: loss: 0.08705 [I 2022-10-28 00:01:17] ...optimizers/trijoint.py.151: total_norm: 11.732412 [I 2022-10-28 00:01:23] ...trap/engines/engine.py.224: train: epoch 95 | batch 800/2304 [I 2022-10-28 00:01:23] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:13:59 [I 2022-10-28 00:01:23] ...trap/engines/engine.py.229: process: 0.56403 | load: 0.00249 [I 2022-10-28 00:01:23] ...trap/engines/engine.py.230: loss: 0.10292 [I 2022-10-28 00:01:23] ...optimizers/trijoint.py.151: total_norm: 8.316822 [I 2022-10-28 00:01:28] ...trap/engines/engine.py.224: train: epoch 95 | batch 810/2304 [I 2022-10-28 00:01:28] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:53 [I 2022-10-28 00:01:28] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00245 [I 2022-10-28 00:01:28] ...trap/engines/engine.py.230: loss: 0.10361 [I 2022-10-28 00:01:28] ...optimizers/trijoint.py.151: total_norm: 28.693747 [I 2022-10-28 00:01:34] ...trap/engines/engine.py.224: train: epoch 95 | batch 820/2304 [I 2022-10-28 00:01:34] ...trap/engines/engine.py.228: elapsed: 0:08:01 | left: 0:13:52 [I 2022-10-28 00:01:34] ...trap/engines/engine.py.229: process: 0.56704 | load: 0.00246 [I 2022-10-28 00:01:34] ...trap/engines/engine.py.230: loss: 0.13083 [I 2022-10-28 00:01:34] ...optimizers/trijoint.py.151: total_norm: 12.863012 [I 2022-10-28 00:01:39] ...trap/engines/engine.py.224: train: epoch 95 | batch 830/2304 [I 2022-10-28 00:01:39] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:39 [I 2022-10-28 00:01:39] ...trap/engines/engine.py.229: process: 0.55350 | load: 0.00249 [I 2022-10-28 00:01:39] ...trap/engines/engine.py.230: loss: 0.10475 [I 2022-10-28 00:01:39] ...optimizers/trijoint.py.151: total_norm: 20.135366 [I 2022-10-28 00:01:45] ...trap/engines/engine.py.224: train: epoch 95 | batch 840/2304 [I 2022-10-28 00:01:45] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:36 [I 2022-10-28 00:01:45] ...trap/engines/engine.py.229: process: 0.56343 | load: 0.00247 [I 2022-10-28 00:01:45] ...trap/engines/engine.py.230: loss: 0.10532 [I 2022-10-28 00:01:45] ...optimizers/trijoint.py.151: total_norm: 9.442433 [I 2022-10-28 00:01:51] ...trap/engines/engine.py.224: train: epoch 95 | batch 850/2304 [I 2022-10-28 00:01:51] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:30 [I 2022-10-28 00:01:51] ...trap/engines/engine.py.229: process: 0.56477 | load: 0.00246 [I 2022-10-28 00:01:51] ...trap/engines/engine.py.230: loss: 0.10479 [I 2022-10-28 00:01:51] ...optimizers/trijoint.py.151: total_norm: 9.607231 [I 2022-10-28 00:01:56] ...trap/engines/engine.py.224: train: epoch 95 | batch 860/2304 [I 2022-10-28 00:01:56] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:24 [I 2022-10-28 00:01:56] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00247 [I 2022-10-28 00:01:56] ...trap/engines/engine.py.230: loss: 0.09869 [I 2022-10-28 00:01:56] ...optimizers/trijoint.py.151: total_norm: 14.134016 [I 2022-10-28 00:02:02] ...trap/engines/engine.py.224: train: epoch 95 | batch 870/2304 [I 2022-10-28 00:02:02] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:13:11 [I 2022-10-28 00:02:02] ...trap/engines/engine.py.229: process: 0.54232 | load: 0.00249 [I 2022-10-28 00:02:02] ...trap/engines/engine.py.230: loss: 0.10288 [I 2022-10-28 00:02:02] ...optimizers/trijoint.py.151: total_norm: 18.356998 [I 2022-10-28 00:02:07] ...trap/engines/engine.py.224: train: epoch 95 | batch 880/2304 [I 2022-10-28 00:02:07] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:09 [I 2022-10-28 00:02:07] ...trap/engines/engine.py.229: process: 0.55138 | load: 0.00244 [I 2022-10-28 00:02:07] ...trap/engines/engine.py.230: loss: 0.10447 [I 2022-10-28 00:02:07] ...optimizers/trijoint.py.151: total_norm: 16.761824 [I 2022-10-28 00:02:13] ...trap/engines/engine.py.224: train: epoch 95 | batch 890/2304 [I 2022-10-28 00:02:13] ...trap/engines/engine.py.228: elapsed: 0:08:40 | left: 0:13:13 [I 2022-10-28 00:02:13] ...trap/engines/engine.py.229: process: 0.56658 | load: 0.00246 [I 2022-10-28 00:02:13] ...trap/engines/engine.py.230: loss: 0.12043 [I 2022-10-28 00:02:13] ...optimizers/trijoint.py.151: total_norm: 11.590193 [I 2022-10-28 00:02:19] ...trap/engines/engine.py.224: train: epoch 95 | batch 900/2304 [I 2022-10-28 00:02:19] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:13:03 [I 2022-10-28 00:02:19] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00245 [I 2022-10-28 00:02:19] ...trap/engines/engine.py.230: loss: 0.11264 [I 2022-10-28 00:02:19] ...optimizers/trijoint.py.151: total_norm: 9.436169 [I 2022-10-28 00:02:24] ...trap/engines/engine.py.224: train: epoch 95 | batch 910/2304 [I 2022-10-28 00:02:24] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:56 [I 2022-10-28 00:02:24] ...trap/engines/engine.py.229: process: 0.55171 | load: 0.00241 [I 2022-10-28 00:02:24] ...trap/engines/engine.py.230: loss: 0.17449 [I 2022-10-28 00:02:24] ...optimizers/trijoint.py.151: total_norm: 34.819958 [I 2022-10-28 00:02:30] ...trap/engines/engine.py.224: train: epoch 95 | batch 920/2304 [I 2022-10-28 00:02:30] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:46 [I 2022-10-28 00:02:30] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00243 [I 2022-10-28 00:02:30] ...trap/engines/engine.py.230: loss: 0.09872 [I 2022-10-28 00:02:30] ...optimizers/trijoint.py.151: total_norm: 31.137596 [I 2022-10-28 00:02:35] ...trap/engines/engine.py.224: train: epoch 95 | batch 930/2304 [I 2022-10-28 00:02:35] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:45 [I 2022-10-28 00:02:35] ...trap/engines/engine.py.229: process: 0.54932 | load: 0.00245 [I 2022-10-28 00:02:35] ...trap/engines/engine.py.230: loss: 0.10192 [I 2022-10-28 00:02:35] ...optimizers/trijoint.py.151: total_norm: 12.947200 [I 2022-10-28 00:02:41] ...trap/engines/engine.py.224: train: epoch 95 | batch 940/2304 [I 2022-10-28 00:02:41] ...trap/engines/engine.py.228: elapsed: 0:09:08 | left: 0:12:41 [I 2022-10-28 00:02:41] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00244 [I 2022-10-28 00:02:41] ...trap/engines/engine.py.230: loss: 0.10670 [I 2022-10-28 00:02:41] ...optimizers/trijoint.py.151: total_norm: 11.428029 [I 2022-10-28 00:02:46] ...trap/engines/engine.py.224: train: epoch 95 | batch 950/2304 [I 2022-10-28 00:02:46] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:40 [I 2022-10-28 00:02:46] ...trap/engines/engine.py.229: process: 0.56455 | load: 0.00246 [I 2022-10-28 00:02:46] ...trap/engines/engine.py.230: loss: 0.14702 [I 2022-10-28 00:02:46] ...optimizers/trijoint.py.151: total_norm: 10.163022 [I 2022-10-28 00:02:52] ...trap/engines/engine.py.224: train: epoch 95 | batch 960/2304 [I 2022-10-28 00:02:52] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:29 [I 2022-10-28 00:02:52] ...trap/engines/engine.py.229: process: 0.55425 | load: 0.00259 [I 2022-10-28 00:02:52] ...trap/engines/engine.py.230: loss: 0.09315 [I 2022-10-28 00:02:52] ...optimizers/trijoint.py.151: total_norm: 8.262593 [I 2022-10-28 00:02:58] ...trap/engines/engine.py.224: train: epoch 95 | batch 970/2304 [I 2022-10-28 00:02:58] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:12:25 [I 2022-10-28 00:02:58] ...trap/engines/engine.py.229: process: 0.55220 | load: 0.00244 [I 2022-10-28 00:02:58] ...trap/engines/engine.py.230: loss: 0.17563 [I 2022-10-28 00:02:58] ...optimizers/trijoint.py.151: total_norm: 35.277039 [I 2022-10-28 00:03:03] ...trap/engines/engine.py.224: train: epoch 95 | batch 980/2304 [I 2022-10-28 00:03:03] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:17 [I 2022-10-28 00:03:03] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00240 [I 2022-10-28 00:03:03] ...trap/engines/engine.py.230: loss: 0.09514 [I 2022-10-28 00:03:03] ...optimizers/trijoint.py.151: total_norm: 14.523028 [I 2022-10-28 00:03:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 990/2304 [I 2022-10-28 00:03:09] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:12:19 [I 2022-10-28 00:03:09] ...trap/engines/engine.py.229: process: 0.56468 | load: 0.00247 [I 2022-10-28 00:03:09] ...trap/engines/engine.py.230: loss: 0.12523 [I 2022-10-28 00:03:09] ...optimizers/trijoint.py.151: total_norm: 29.148764 [I 2022-10-28 00:03:14] ...trap/engines/engine.py.224: train: epoch 95 | batch 1000/2304 [I 2022-10-28 00:03:14] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:12:06 [I 2022-10-28 00:03:14] ...trap/engines/engine.py.229: process: 0.54968 | load: 0.00246 [I 2022-10-28 00:03:14] ...trap/engines/engine.py.230: loss: 0.15222 [I 2022-10-28 00:03:14] ...optimizers/trijoint.py.151: total_norm: 16.239660 [I 2022-10-28 00:03:20] ...trap/engines/engine.py.224: train: epoch 95 | batch 1010/2304 [I 2022-10-28 00:03:20] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:12:04 [I 2022-10-28 00:03:20] ...trap/engines/engine.py.229: process: 0.56442 | load: 0.00239 [I 2022-10-28 00:03:20] ...trap/engines/engine.py.230: loss: 0.11239 [I 2022-10-28 00:03:20] ...optimizers/trijoint.py.151: total_norm: 21.169319 [I 2022-10-28 00:03:26] ...trap/engines/engine.py.224: train: epoch 95 | batch 1020/2304 [I 2022-10-28 00:03:26] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:12:03 [I 2022-10-28 00:03:26] ...trap/engines/engine.py.229: process: 0.56659 | load: 0.00248 [I 2022-10-28 00:03:26] ...trap/engines/engine.py.230: loss: 0.11055 [I 2022-10-28 00:03:26] ...optimizers/trijoint.py.151: total_norm: 9.506881 [I 2022-10-28 00:03:31] ...trap/engines/engine.py.224: train: epoch 95 | batch 1030/2304 [I 2022-10-28 00:03:31] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:52 [I 2022-10-28 00:03:31] ...trap/engines/engine.py.229: process: 0.56649 | load: 0.00243 [I 2022-10-28 00:03:31] ...trap/engines/engine.py.230: loss: 0.11170 [I 2022-10-28 00:03:31] ...optimizers/trijoint.py.151: total_norm: 9.997524 [I 2022-10-28 00:03:37] ...trap/engines/engine.py.224: train: epoch 95 | batch 1040/2304 [I 2022-10-28 00:03:37] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:46 [I 2022-10-28 00:03:37] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00243 [I 2022-10-28 00:03:37] ...trap/engines/engine.py.230: loss: 0.10954 [I 2022-10-28 00:03:37] ...optimizers/trijoint.py.151: total_norm: 10.809179 [I 2022-10-28 00:03:42] ...trap/engines/engine.py.224: train: epoch 95 | batch 1050/2304 [I 2022-10-28 00:03:42] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:35 [I 2022-10-28 00:03:42] ...trap/engines/engine.py.229: process: 0.54826 | load: 0.00242 [I 2022-10-28 00:03:42] ...trap/engines/engine.py.230: loss: 0.11950 [I 2022-10-28 00:03:42] ...optimizers/trijoint.py.151: total_norm: 13.919630 [I 2022-10-28 00:03:48] ...trap/engines/engine.py.224: train: epoch 95 | batch 1060/2304 [I 2022-10-28 00:03:48] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:34 [I 2022-10-28 00:03:48] ...trap/engines/engine.py.229: process: 0.54352 | load: 0.00244 [I 2022-10-28 00:03:48] ...trap/engines/engine.py.230: loss: 0.15824 [I 2022-10-28 00:03:48] ...optimizers/trijoint.py.151: total_norm: 24.051367 [I 2022-10-28 00:03:54] ...trap/engines/engine.py.224: train: epoch 95 | batch 1070/2304 [I 2022-10-28 00:03:54] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:29 [I 2022-10-28 00:03:54] ...trap/engines/engine.py.229: process: 0.57290 | load: 0.00246 [I 2022-10-28 00:03:54] ...trap/engines/engine.py.230: loss: 0.10214 [I 2022-10-28 00:03:54] ...optimizers/trijoint.py.151: total_norm: 8.941908 [I 2022-10-28 00:03:59] ...trap/engines/engine.py.224: train: epoch 95 | batch 1080/2304 [I 2022-10-28 00:03:59] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:25 [I 2022-10-28 00:03:59] ...trap/engines/engine.py.229: process: 0.56587 | load: 0.00242 [I 2022-10-28 00:03:59] ...trap/engines/engine.py.230: loss: 0.10512 [I 2022-10-28 00:03:59] ...optimizers/trijoint.py.151: total_norm: 10.357636 [I 2022-10-28 00:04:05] ...trap/engines/engine.py.224: train: epoch 95 | batch 1090/2304 [I 2022-10-28 00:04:05] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:21 [I 2022-10-28 00:04:05] ...trap/engines/engine.py.229: process: 0.56302 | load: 0.00253 [I 2022-10-28 00:04:05] ...trap/engines/engine.py.230: loss: 0.10464 [I 2022-10-28 00:04:05] ...optimizers/trijoint.py.151: total_norm: 17.446426 [I 2022-10-28 00:04:10] ...trap/engines/engine.py.224: train: epoch 95 | batch 1100/2304 [I 2022-10-28 00:04:10] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:11:15 [I 2022-10-28 00:04:10] ...trap/engines/engine.py.229: process: 0.55881 | load: 0.00247 [I 2022-10-28 00:04:10] ...trap/engines/engine.py.230: loss: 0.11658 [I 2022-10-28 00:04:10] ...optimizers/trijoint.py.151: total_norm: 14.942381 [I 2022-10-28 00:04:16] ...trap/engines/engine.py.224: train: epoch 95 | batch 1110/2304 [I 2022-10-28 00:04:16] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:11:05 [I 2022-10-28 00:04:16] ...trap/engines/engine.py.229: process: 0.55227 | load: 0.00247 [I 2022-10-28 00:04:16] ...trap/engines/engine.py.230: loss: 0.16389 [I 2022-10-28 00:04:16] ...optimizers/trijoint.py.151: total_norm: 12.426828 [I 2022-10-28 00:04:22] ...trap/engines/engine.py.224: train: epoch 95 | batch 1120/2304 [I 2022-10-28 00:04:22] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:11:03 [I 2022-10-28 00:04:22] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00243 [I 2022-10-28 00:04:22] ...trap/engines/engine.py.230: loss: 0.08719 [I 2022-10-28 00:04:22] ...optimizers/trijoint.py.151: total_norm: 9.948921 [I 2022-10-28 00:04:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 1130/2304 [I 2022-10-28 00:04:27] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:55 [I 2022-10-28 00:04:27] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00245 [I 2022-10-28 00:04:27] ...trap/engines/engine.py.230: loss: 0.10509 [I 2022-10-28 00:04:27] ...optimizers/trijoint.py.151: total_norm: 16.129620 [I 2022-10-28 00:04:33] ...trap/engines/engine.py.224: train: epoch 95 | batch 1140/2304 [I 2022-10-28 00:04:33] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:53 [I 2022-10-28 00:04:33] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00244 [I 2022-10-28 00:04:33] ...trap/engines/engine.py.230: loss: 0.13256 [I 2022-10-28 00:04:33] ...optimizers/trijoint.py.151: total_norm: 13.708074 [I 2022-10-28 00:04:38] ...trap/engines/engine.py.224: train: epoch 95 | batch 1150/2304 [I 2022-10-28 00:04:38] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:44 [I 2022-10-28 00:04:39] ...trap/engines/engine.py.229: process: 0.55299 | load: 0.00248 [I 2022-10-28 00:04:39] ...trap/engines/engine.py.230: loss: 0.12371 [I 2022-10-28 00:04:39] ...optimizers/trijoint.py.151: total_norm: 10.519794 [I 2022-10-28 00:04:44] ...trap/engines/engine.py.224: train: epoch 95 | batch 1160/2304 [I 2022-10-28 00:04:44] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:38 [I 2022-10-28 00:04:44] ...trap/engines/engine.py.229: process: 0.54587 | load: 0.00243 [I 2022-10-28 00:04:44] ...trap/engines/engine.py.230: loss: 0.11059 [I 2022-10-28 00:04:44] ...optimizers/trijoint.py.151: total_norm: 12.347395 [I 2022-10-28 00:04:50] ...trap/engines/engine.py.224: train: epoch 95 | batch 1170/2304 [I 2022-10-28 00:04:50] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:37 [I 2022-10-28 00:04:50] ...trap/engines/engine.py.229: process: 0.57121 | load: 0.00247 [I 2022-10-28 00:04:50] ...trap/engines/engine.py.230: loss: 0.15493 [I 2022-10-28 00:04:50] ...optimizers/trijoint.py.151: total_norm: 21.331051 [I 2022-10-28 00:04:55] ...trap/engines/engine.py.224: train: epoch 95 | batch 1180/2304 [I 2022-10-28 00:04:55] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:28 [I 2022-10-28 00:04:55] ...trap/engines/engine.py.229: process: 0.55491 | load: 0.00242 [I 2022-10-28 00:04:55] ...trap/engines/engine.py.230: loss: 0.14715 [I 2022-10-28 00:04:55] ...optimizers/trijoint.py.151: total_norm: 42.764088 [I 2022-10-28 00:05:01] ...trap/engines/engine.py.224: train: epoch 95 | batch 1190/2304 [I 2022-10-28 00:05:01] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:24 [I 2022-10-28 00:05:01] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00243 [I 2022-10-28 00:05:01] ...trap/engines/engine.py.230: loss: 0.09490 [I 2022-10-28 00:05:01] ...optimizers/trijoint.py.151: total_norm: 27.242382 [I 2022-10-28 00:05:06] ...trap/engines/engine.py.224: train: epoch 95 | batch 1200/2304 [I 2022-10-28 00:05:06] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:10:14 [I 2022-10-28 00:05:06] ...trap/engines/engine.py.229: process: 0.55442 | load: 0.00244 [I 2022-10-28 00:05:06] ...trap/engines/engine.py.230: loss: 0.18105 [I 2022-10-28 00:05:06] ...optimizers/trijoint.py.151: total_norm: 24.725311 [I 2022-10-28 00:05:12] ...trap/engines/engine.py.224: train: epoch 95 | batch 1210/2304 [I 2022-10-28 00:05:12] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:10:11 [I 2022-10-28 00:05:12] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00241 [I 2022-10-28 00:05:12] ...trap/engines/engine.py.230: loss: 0.10572 [I 2022-10-28 00:05:12] ...optimizers/trijoint.py.151: total_norm: 14.375463 [I 2022-10-28 00:05:18] ...trap/engines/engine.py.224: train: epoch 95 | batch 1220/2304 [I 2022-10-28 00:05:18] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:10:06 [I 2022-10-28 00:05:18] ...trap/engines/engine.py.229: process: 0.56419 | load: 0.00243 [I 2022-10-28 00:05:18] ...trap/engines/engine.py.230: loss: 0.13412 [I 2022-10-28 00:05:18] ...optimizers/trijoint.py.151: total_norm: 19.338543 [I 2022-10-28 00:05:23] ...trap/engines/engine.py.224: train: epoch 95 | batch 1230/2304 [I 2022-10-28 00:05:23] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:10:01 [I 2022-10-28 00:05:23] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00242 [I 2022-10-28 00:05:23] ...trap/engines/engine.py.230: loss: 0.08260 [I 2022-10-28 00:05:23] ...optimizers/trijoint.py.151: total_norm: 10.673011 [I 2022-10-28 00:05:29] ...trap/engines/engine.py.224: train: epoch 95 | batch 1240/2304 [I 2022-10-28 00:05:29] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:54 [I 2022-10-28 00:05:29] ...trap/engines/engine.py.229: process: 0.55184 | load: 0.00242 [I 2022-10-28 00:05:29] ...trap/engines/engine.py.230: loss: 0.08770 [I 2022-10-28 00:05:29] ...optimizers/trijoint.py.151: total_norm: 10.857811 [I 2022-10-28 00:05:34] ...trap/engines/engine.py.224: train: epoch 95 | batch 1250/2304 [I 2022-10-28 00:05:34] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:51 [I 2022-10-28 00:05:34] ...trap/engines/engine.py.229: process: 0.56391 | load: 0.00246 [I 2022-10-28 00:05:34] ...trap/engines/engine.py.230: loss: 0.15437 [I 2022-10-28 00:05:34] ...optimizers/trijoint.py.151: total_norm: 22.393824 [I 2022-10-28 00:05:40] ...trap/engines/engine.py.224: train: epoch 95 | batch 1260/2304 [I 2022-10-28 00:05:40] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:44 [I 2022-10-28 00:05:40] ...trap/engines/engine.py.229: process: 0.54320 | load: 0.00244 [I 2022-10-28 00:05:40] ...trap/engines/engine.py.230: loss: 0.12332 [I 2022-10-28 00:05:40] ...optimizers/trijoint.py.151: total_norm: 18.781103 [I 2022-10-28 00:05:46] ...trap/engines/engine.py.224: train: epoch 95 | batch 1270/2304 [I 2022-10-28 00:05:46] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:35 [I 2022-10-28 00:05:46] ...trap/engines/engine.py.229: process: 0.56501 | load: 0.00241 [I 2022-10-28 00:05:46] ...trap/engines/engine.py.230: loss: 0.19827 [I 2022-10-28 00:05:46] ...optimizers/trijoint.py.151: total_norm: 33.721027 [I 2022-10-28 00:05:51] ...trap/engines/engine.py.224: train: epoch 95 | batch 1280/2304 [I 2022-10-28 00:05:51] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:33 [I 2022-10-28 00:05:51] ...trap/engines/engine.py.229: process: 0.55213 | load: 0.00239 [I 2022-10-28 00:05:51] ...trap/engines/engine.py.230: loss: 0.09524 [I 2022-10-28 00:05:51] ...optimizers/trijoint.py.151: total_norm: 40.514854 [I 2022-10-28 00:05:57] ...trap/engines/engine.py.224: train: epoch 95 | batch 1290/2304 [I 2022-10-28 00:05:57] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:23 [I 2022-10-28 00:05:57] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00250 [I 2022-10-28 00:05:57] ...trap/engines/engine.py.230: loss: 0.12482 [I 2022-10-28 00:05:57] ...optimizers/trijoint.py.151: total_norm: 11.264139 [I 2022-10-28 00:06:02] ...trap/engines/engine.py.224: train: epoch 95 | batch 1300/2304 [I 2022-10-28 00:06:02] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:09:18 [I 2022-10-28 00:06:02] ...trap/engines/engine.py.229: process: 0.54988 | load: 0.00246 [I 2022-10-28 00:06:02] ...trap/engines/engine.py.230: loss: 0.10611 [I 2022-10-28 00:06:02] ...optimizers/trijoint.py.151: total_norm: 10.917163 [I 2022-10-28 00:06:08] ...trap/engines/engine.py.224: train: epoch 95 | batch 1310/2304 [I 2022-10-28 00:06:08] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:09:12 [I 2022-10-28 00:06:08] ...trap/engines/engine.py.229: process: 0.56360 | load: 0.00245 [I 2022-10-28 00:06:08] ...trap/engines/engine.py.230: loss: 0.11030 [I 2022-10-28 00:06:08] ...optimizers/trijoint.py.151: total_norm: 14.580427 [I 2022-10-28 00:06:14] ...trap/engines/engine.py.224: train: epoch 95 | batch 1320/2304 [I 2022-10-28 00:06:14] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:09:10 [I 2022-10-28 00:06:14] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00248 [I 2022-10-28 00:06:14] ...trap/engines/engine.py.230: loss: 0.11847 [I 2022-10-28 00:06:14] ...optimizers/trijoint.py.151: total_norm: 11.565841 [I 2022-10-28 00:06:19] ...trap/engines/engine.py.224: train: epoch 95 | batch 1330/2304 [I 2022-10-28 00:06:19] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:09:03 [I 2022-10-28 00:06:19] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00245 [I 2022-10-28 00:06:19] ...trap/engines/engine.py.230: loss: 0.19638 [I 2022-10-28 00:06:19] ...optimizers/trijoint.py.151: total_norm: 22.813801 [I 2022-10-28 00:06:25] ...trap/engines/engine.py.224: train: epoch 95 | batch 1340/2304 [I 2022-10-28 00:06:25] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:09:00 [I 2022-10-28 00:06:25] ...trap/engines/engine.py.229: process: 0.56207 | load: 0.00248 [I 2022-10-28 00:06:25] ...trap/engines/engine.py.230: loss: 0.12300 [I 2022-10-28 00:06:25] ...optimizers/trijoint.py.151: total_norm: 10.400783 [I 2022-10-28 00:06:30] ...trap/engines/engine.py.224: train: epoch 95 | batch 1350/2304 [I 2022-10-28 00:06:30] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:54 [I 2022-10-28 00:06:30] ...trap/engines/engine.py.229: process: 0.56542 | load: 0.00244 [I 2022-10-28 00:06:30] ...trap/engines/engine.py.230: loss: 0.11356 [I 2022-10-28 00:06:30] ...optimizers/trijoint.py.151: total_norm: 10.931087 [I 2022-10-28 00:06:36] ...trap/engines/engine.py.224: train: epoch 95 | batch 1360/2304 [I 2022-10-28 00:06:36] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:49 [I 2022-10-28 00:06:36] ...trap/engines/engine.py.229: process: 0.54573 | load: 0.00244 [I 2022-10-28 00:06:36] ...trap/engines/engine.py.230: loss: 0.10325 [I 2022-10-28 00:06:36] ...optimizers/trijoint.py.151: total_norm: 15.306091 [I 2022-10-28 00:06:42] ...trap/engines/engine.py.224: train: epoch 95 | batch 1370/2304 [I 2022-10-28 00:06:42] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:59 [I 2022-10-28 00:06:42] ...trap/engines/engine.py.229: process: 0.56816 | load: 0.00261 [I 2022-10-28 00:06:42] ...trap/engines/engine.py.230: loss: 0.20041 [I 2022-10-28 00:06:42] ...optimizers/trijoint.py.151: total_norm: 15.027061 [I 2022-10-28 00:06:47] ...trap/engines/engine.py.224: train: epoch 95 | batch 1380/2304 [I 2022-10-28 00:06:47] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:34 [I 2022-10-28 00:06:47] ...trap/engines/engine.py.229: process: 0.55298 | load: 0.00253 [I 2022-10-28 00:06:47] ...trap/engines/engine.py.230: loss: 0.11559 [I 2022-10-28 00:06:47] ...optimizers/trijoint.py.151: total_norm: 17.242926 [I 2022-10-28 00:06:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 1390/2304 [I 2022-10-28 00:06:53] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:32 [I 2022-10-28 00:06:53] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00244 [I 2022-10-28 00:06:53] ...trap/engines/engine.py.230: loss: 0.12915 [I 2022-10-28 00:06:53] ...optimizers/trijoint.py.151: total_norm: 15.586303 [I 2022-10-28 00:06:59] ...trap/engines/engine.py.224: train: epoch 95 | batch 1400/2304 [I 2022-10-28 00:06:59] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:24 [I 2022-10-28 00:06:59] ...trap/engines/engine.py.229: process: 0.56352 | load: 0.00246 [I 2022-10-28 00:06:59] ...trap/engines/engine.py.230: loss: 0.10375 [I 2022-10-28 00:06:59] ...optimizers/trijoint.py.151: total_norm: 13.036038 [I 2022-10-28 00:07:04] ...trap/engines/engine.py.224: train: epoch 95 | batch 1410/2304 [I 2022-10-28 00:07:04] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:08:19 [I 2022-10-28 00:07:04] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00241 [I 2022-10-28 00:07:04] ...trap/engines/engine.py.230: loss: 0.12536 [I 2022-10-28 00:07:04] ...optimizers/trijoint.py.151: total_norm: 9.474276 [I 2022-10-28 00:07:10] ...trap/engines/engine.py.224: train: epoch 95 | batch 1420/2304 [I 2022-10-28 00:07:10] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:08:09 [I 2022-10-28 00:07:10] ...trap/engines/engine.py.229: process: 0.54428 | load: 0.00245 [I 2022-10-28 00:07:10] ...trap/engines/engine.py.230: loss: 0.15055 [I 2022-10-28 00:07:10] ...optimizers/trijoint.py.151: total_norm: 14.348145 [I 2022-10-28 00:07:15] ...trap/engines/engine.py.224: train: epoch 95 | batch 1430/2304 [I 2022-10-28 00:07:15] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:08:08 [I 2022-10-28 00:07:15] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00242 [I 2022-10-28 00:07:15] ...trap/engines/engine.py.230: loss: 0.13912 [I 2022-10-28 00:07:15] ...optimizers/trijoint.py.151: total_norm: 25.670885 [I 2022-10-28 00:07:21] ...trap/engines/engine.py.224: train: epoch 95 | batch 1440/2304 [I 2022-10-28 00:07:21] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:08:19 [I 2022-10-28 00:07:21] ...trap/engines/engine.py.229: process: 0.64520 | load: 0.00244 [I 2022-10-28 00:07:21] ...trap/engines/engine.py.230: loss: 0.10076 [I 2022-10-28 00:07:21] ...optimizers/trijoint.py.151: total_norm: 14.242794 [I 2022-10-28 00:07:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 1450/2304 [I 2022-10-28 00:07:27] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:08:00 [I 2022-10-28 00:07:27] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00248 [I 2022-10-28 00:07:27] ...trap/engines/engine.py.230: loss: 0.25614 [I 2022-10-28 00:07:27] ...optimizers/trijoint.py.151: total_norm: 24.046597 [I 2022-10-28 00:07:32] ...trap/engines/engine.py.224: train: epoch 95 | batch 1460/2304 [I 2022-10-28 00:07:32] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:56 [I 2022-10-28 00:07:32] ...trap/engines/engine.py.229: process: 0.57518 | load: 0.00244 [I 2022-10-28 00:07:32] ...trap/engines/engine.py.230: loss: 0.13911 [I 2022-10-28 00:07:32] ...optimizers/trijoint.py.151: total_norm: 17.565111 [I 2022-10-28 00:07:38] ...trap/engines/engine.py.224: train: epoch 95 | batch 1470/2304 [I 2022-10-28 00:07:38] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:46 [I 2022-10-28 00:07:38] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00244 [I 2022-10-28 00:07:38] ...trap/engines/engine.py.230: loss: 0.19400 [I 2022-10-28 00:07:38] ...optimizers/trijoint.py.151: total_norm: 20.321232 [I 2022-10-28 00:07:44] ...trap/engines/engine.py.224: train: epoch 95 | batch 1480/2304 [I 2022-10-28 00:07:44] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:39 [I 2022-10-28 00:07:44] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00244 [I 2022-10-28 00:07:44] ...trap/engines/engine.py.230: loss: 0.13089 [I 2022-10-28 00:07:44] ...optimizers/trijoint.py.151: total_norm: 21.090202 [I 2022-10-28 00:07:49] ...trap/engines/engine.py.224: train: epoch 95 | batch 1490/2304 [I 2022-10-28 00:07:49] ...trap/engines/engine.py.228: elapsed: 0:14:17 | left: 0:07:33 [I 2022-10-28 00:07:49] ...trap/engines/engine.py.229: process: 0.55472 | load: 0.00242 [I 2022-10-28 00:07:49] ...trap/engines/engine.py.230: loss: 0.09816 [I 2022-10-28 00:07:49] ...optimizers/trijoint.py.151: total_norm: 13.465117 [I 2022-10-28 00:07:55] ...trap/engines/engine.py.224: train: epoch 95 | batch 1500/2304 [I 2022-10-28 00:07:55] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:30 [I 2022-10-28 00:07:55] ...trap/engines/engine.py.229: process: 0.56423 | load: 0.00241 [I 2022-10-28 00:07:55] ...trap/engines/engine.py.230: loss: 0.16587 [I 2022-10-28 00:07:55] ...optimizers/trijoint.py.151: total_norm: 26.734108 [I 2022-10-28 00:08:00] ...trap/engines/engine.py.224: train: epoch 95 | batch 1510/2304 [I 2022-10-28 00:08:00] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:32 [I 2022-10-28 00:08:00] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00243 [I 2022-10-28 00:08:00] ...trap/engines/engine.py.230: loss: 0.16571 [I 2022-10-28 00:08:00] ...optimizers/trijoint.py.151: total_norm: 10.148720 [I 2022-10-28 00:08:06] ...trap/engines/engine.py.224: train: epoch 95 | batch 1520/2304 [I 2022-10-28 00:08:06] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:07:17 [I 2022-10-28 00:08:06] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00241 [I 2022-10-28 00:08:06] ...trap/engines/engine.py.230: loss: 0.11810 [I 2022-10-28 00:08:06] ...optimizers/trijoint.py.151: total_norm: 19.106932 [I 2022-10-28 00:08:12] ...trap/engines/engine.py.224: train: epoch 95 | batch 1530/2304 [I 2022-10-28 00:08:12] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:07:11 [I 2022-10-28 00:08:12] ...trap/engines/engine.py.229: process: 0.56308 | load: 0.00262 [I 2022-10-28 00:08:12] ...trap/engines/engine.py.230: loss: 0.13683 [I 2022-10-28 00:08:12] ...optimizers/trijoint.py.151: total_norm: 21.747368 [I 2022-10-28 00:08:17] ...trap/engines/engine.py.224: train: epoch 95 | batch 1540/2304 [I 2022-10-28 00:08:17] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:07:03 [I 2022-10-28 00:08:17] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00242 [I 2022-10-28 00:08:17] ...trap/engines/engine.py.230: loss: 0.12924 [I 2022-10-28 00:08:17] ...optimizers/trijoint.py.151: total_norm: 8.525852 [I 2022-10-28 00:08:23] ...trap/engines/engine.py.224: train: epoch 95 | batch 1550/2304 [I 2022-10-28 00:08:23] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:07:01 [I 2022-10-28 00:08:23] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00248 [I 2022-10-28 00:08:23] ...trap/engines/engine.py.230: loss: 0.13095 [I 2022-10-28 00:08:23] ...optimizers/trijoint.py.151: total_norm: 14.701039 [I 2022-10-28 00:08:28] ...trap/engines/engine.py.224: train: epoch 95 | batch 1560/2304 [I 2022-10-28 00:08:28] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:56 [I 2022-10-28 00:08:28] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00247 [I 2022-10-28 00:08:28] ...trap/engines/engine.py.230: loss: 0.15869 [I 2022-10-28 00:08:28] ...optimizers/trijoint.py.151: total_norm: 17.930332 [I 2022-10-28 00:08:34] ...trap/engines/engine.py.224: train: epoch 95 | batch 1570/2304 [I 2022-10-28 00:08:34] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:07:02 [I 2022-10-28 00:08:34] ...trap/engines/engine.py.229: process: 0.64160 | load: 0.00246 [I 2022-10-28 00:08:34] ...trap/engines/engine.py.230: loss: 0.12440 [I 2022-10-28 00:08:34] ...optimizers/trijoint.py.151: total_norm: 16.877077 [I 2022-10-28 00:08:40] ...trap/engines/engine.py.224: train: epoch 95 | batch 1580/2304 [I 2022-10-28 00:08:40] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:46 [I 2022-10-28 00:08:40] ...trap/engines/engine.py.229: process: 0.57237 | load: 0.00244 [I 2022-10-28 00:08:40] ...trap/engines/engine.py.230: loss: 0.08149 [I 2022-10-28 00:08:40] ...optimizers/trijoint.py.151: total_norm: 13.949803 [I 2022-10-28 00:08:45] ...trap/engines/engine.py.224: train: epoch 95 | batch 1590/2304 [I 2022-10-28 00:08:45] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:36 [I 2022-10-28 00:08:45] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00244 [I 2022-10-28 00:08:45] ...trap/engines/engine.py.230: loss: 0.12908 [I 2022-10-28 00:08:45] ...optimizers/trijoint.py.151: total_norm: 15.723402 [I 2022-10-28 00:08:51] ...trap/engines/engine.py.224: train: epoch 95 | batch 1600/2304 [I 2022-10-28 00:08:51] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:31 [I 2022-10-28 00:08:51] ...trap/engines/engine.py.229: process: 0.54798 | load: 0.00247 [I 2022-10-28 00:08:51] ...trap/engines/engine.py.230: loss: 0.10369 [I 2022-10-28 00:08:51] ...optimizers/trijoint.py.151: total_norm: 18.571943 [I 2022-10-28 00:08:56] ...trap/engines/engine.py.224: train: epoch 95 | batch 1610/2304 [I 2022-10-28 00:08:56] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:27 [I 2022-10-28 00:08:56] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00246 [I 2022-10-28 00:08:56] ...trap/engines/engine.py.230: loss: 0.10786 [I 2022-10-28 00:08:56] ...optimizers/trijoint.py.151: total_norm: 9.683396 [I 2022-10-28 00:09:02] ...trap/engines/engine.py.224: train: epoch 95 | batch 1620/2304 [I 2022-10-28 00:09:02] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:20 [I 2022-10-28 00:09:02] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00242 [I 2022-10-28 00:09:02] ...trap/engines/engine.py.230: loss: 0.10247 [I 2022-10-28 00:09:02] ...optimizers/trijoint.py.151: total_norm: 13.436619 [I 2022-10-28 00:09:08] ...trap/engines/engine.py.224: train: epoch 95 | batch 1630/2304 [I 2022-10-28 00:09:08] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:06:15 [I 2022-10-28 00:09:08] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00259 [I 2022-10-28 00:09:08] ...trap/engines/engine.py.230: loss: 0.16547 [I 2022-10-28 00:09:08] ...optimizers/trijoint.py.151: total_norm: 10.446665 [I 2022-10-28 00:09:13] ...trap/engines/engine.py.224: train: epoch 95 | batch 1640/2304 [I 2022-10-28 00:09:13] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:06:12 [I 2022-10-28 00:09:13] ...trap/engines/engine.py.229: process: 0.55499 | load: 0.00247 [I 2022-10-28 00:09:13] ...trap/engines/engine.py.230: loss: 0.21416 [I 2022-10-28 00:09:13] ...optimizers/trijoint.py.151: total_norm: 17.302679 [I 2022-10-28 00:09:19] ...trap/engines/engine.py.224: train: epoch 95 | batch 1650/2304 [I 2022-10-28 00:09:19] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:06:08 [I 2022-10-28 00:09:19] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00250 [I 2022-10-28 00:09:19] ...trap/engines/engine.py.230: loss: 0.14759 [I 2022-10-28 00:09:19] ...optimizers/trijoint.py.151: total_norm: 13.884847 [I 2022-10-28 00:09:24] ...trap/engines/engine.py.224: train: epoch 95 | batch 1660/2304 [I 2022-10-28 00:09:24] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:06:00 [I 2022-10-28 00:09:24] ...trap/engines/engine.py.229: process: 0.56288 | load: 0.00247 [I 2022-10-28 00:09:24] ...trap/engines/engine.py.230: loss: 0.09781 [I 2022-10-28 00:09:24] ...optimizers/trijoint.py.151: total_norm: 10.448565 [I 2022-10-28 00:09:30] ...trap/engines/engine.py.224: train: epoch 95 | batch 1670/2304 [I 2022-10-28 00:09:30] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:54 [I 2022-10-28 00:09:30] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00245 [I 2022-10-28 00:09:30] ...trap/engines/engine.py.230: loss: 0.17467 [I 2022-10-28 00:09:30] ...optimizers/trijoint.py.151: total_norm: 20.604309 [I 2022-10-28 00:09:36] ...trap/engines/engine.py.224: train: epoch 95 | batch 1680/2304 [I 2022-10-28 00:09:36] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:48 [I 2022-10-28 00:09:36] ...trap/engines/engine.py.229: process: 0.55999 | load: 0.00252 [I 2022-10-28 00:09:36] ...trap/engines/engine.py.230: loss: 0.11048 [I 2022-10-28 00:09:36] ...optimizers/trijoint.py.151: total_norm: 17.124218 [I 2022-10-28 00:09:41] ...trap/engines/engine.py.224: train: epoch 95 | batch 1690/2304 [I 2022-10-28 00:09:41] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:42 [I 2022-10-28 00:09:41] ...trap/engines/engine.py.229: process: 0.57054 | load: 0.00247 [I 2022-10-28 00:09:41] ...trap/engines/engine.py.230: loss: 0.10279 [I 2022-10-28 00:09:41] ...optimizers/trijoint.py.151: total_norm: 8.967887 [I 2022-10-28 00:09:47] ...trap/engines/engine.py.224: train: epoch 95 | batch 1700/2304 [I 2022-10-28 00:09:47] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:42 [I 2022-10-28 00:09:47] ...trap/engines/engine.py.229: process: 0.56915 | load: 0.00248 [I 2022-10-28 00:09:47] ...trap/engines/engine.py.230: loss: 0.09401 [I 2022-10-28 00:09:47] ...optimizers/trijoint.py.151: total_norm: 15.815761 [I 2022-10-28 00:09:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 1710/2304 [I 2022-10-28 00:09:53] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:31 [I 2022-10-28 00:09:53] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00246 [I 2022-10-28 00:09:53] ...trap/engines/engine.py.230: loss: 0.16963 [I 2022-10-28 00:09:53] ...optimizers/trijoint.py.151: total_norm: 68.746246 [I 2022-10-28 00:09:58] ...trap/engines/engine.py.224: train: epoch 95 | batch 1720/2304 [I 2022-10-28 00:09:58] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:24 [I 2022-10-28 00:09:58] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00241 [I 2022-10-28 00:09:58] ...trap/engines/engine.py.230: loss: 0.12790 [I 2022-10-28 00:09:58] ...optimizers/trijoint.py.151: total_norm: 27.934231 [I 2022-10-28 00:10:04] ...trap/engines/engine.py.224: train: epoch 95 | batch 1730/2304 [I 2022-10-28 00:10:04] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:05:21 [I 2022-10-28 00:10:04] ...trap/engines/engine.py.229: process: 0.56407 | load: 0.00241 [I 2022-10-28 00:10:04] ...trap/engines/engine.py.230: loss: 0.10863 [I 2022-10-28 00:10:04] ...optimizers/trijoint.py.151: total_norm: 18.184397 [I 2022-10-28 00:10:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 1740/2304 [I 2022-10-28 00:10:09] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:05:16 [I 2022-10-28 00:10:09] ...trap/engines/engine.py.229: process: 0.55198 | load: 0.00243 [I 2022-10-28 00:10:09] ...trap/engines/engine.py.230: loss: 0.15308 [I 2022-10-28 00:10:09] ...optimizers/trijoint.py.151: total_norm: 18.632454 [I 2022-10-28 00:10:15] ...trap/engines/engine.py.224: train: epoch 95 | batch 1750/2304 [I 2022-10-28 00:10:15] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:05:09 [I 2022-10-28 00:10:15] ...trap/engines/engine.py.229: process: 0.55531 | load: 0.00251 [I 2022-10-28 00:10:15] ...trap/engines/engine.py.230: loss: 0.10356 [I 2022-10-28 00:10:15] ...optimizers/trijoint.py.151: total_norm: 35.849083 [I 2022-10-28 00:10:21] ...trap/engines/engine.py.224: train: epoch 95 | batch 1760/2304 [I 2022-10-28 00:10:21] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:05:11 [I 2022-10-28 00:10:21] ...trap/engines/engine.py.229: process: 0.54385 | load: 0.00248 [I 2022-10-28 00:10:21] ...trap/engines/engine.py.230: loss: 0.09962 [I 2022-10-28 00:10:21] ...optimizers/trijoint.py.151: total_norm: 23.430077 [I 2022-10-28 00:10:26] ...trap/engines/engine.py.224: train: epoch 95 | batch 1770/2304 [I 2022-10-28 00:10:26] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:53 [I 2022-10-28 00:10:26] ...trap/engines/engine.py.229: process: 0.53931 | load: 0.00247 [I 2022-10-28 00:10:26] ...trap/engines/engine.py.230: loss: 0.11916 [I 2022-10-28 00:10:26] ...optimizers/trijoint.py.151: total_norm: 23.625975 [I 2022-10-28 00:10:32] ...trap/engines/engine.py.224: train: epoch 95 | batch 1780/2304 [I 2022-10-28 00:10:32] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:49 [I 2022-10-28 00:10:32] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00244 [I 2022-10-28 00:10:32] ...trap/engines/engine.py.230: loss: 0.09677 [I 2022-10-28 00:10:32] ...optimizers/trijoint.py.151: total_norm: 18.114544 [I 2022-10-28 00:10:37] ...trap/engines/engine.py.224: train: epoch 95 | batch 1790/2304 [I 2022-10-28 00:10:37] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:45 [I 2022-10-28 00:10:37] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00247 [I 2022-10-28 00:10:37] ...trap/engines/engine.py.230: loss: 0.12376 [I 2022-10-28 00:10:37] ...optimizers/trijoint.py.151: total_norm: 14.911951 [I 2022-10-28 00:10:43] ...trap/engines/engine.py.224: train: epoch 95 | batch 1800/2304 [I 2022-10-28 00:10:43] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:40 [I 2022-10-28 00:10:43] ...trap/engines/engine.py.229: process: 0.55014 | load: 0.00243 [I 2022-10-28 00:10:43] ...trap/engines/engine.py.230: loss: 0.15328 [I 2022-10-28 00:10:43] ...optimizers/trijoint.py.151: total_norm: 34.418228 [I 2022-10-28 00:10:48] ...trap/engines/engine.py.224: train: epoch 95 | batch 1810/2304 [I 2022-10-28 00:10:48] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:35 [I 2022-10-28 00:10:48] ...trap/engines/engine.py.229: process: 0.55113 | load: 0.00246 [I 2022-10-28 00:10:48] ...trap/engines/engine.py.230: loss: 0.17275 [I 2022-10-28 00:10:48] ...optimizers/trijoint.py.151: total_norm: 18.715157 [I 2022-10-28 00:10:54] ...trap/engines/engine.py.224: train: epoch 95 | batch 1820/2304 [I 2022-10-28 00:10:54] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:30 [I 2022-10-28 00:10:54] ...trap/engines/engine.py.229: process: 0.57097 | load: 0.00252 [I 2022-10-28 00:10:54] ...trap/engines/engine.py.230: loss: 0.18547 [I 2022-10-28 00:10:54] ...optimizers/trijoint.py.151: total_norm: 14.269382 [I 2022-10-28 00:11:00] ...trap/engines/engine.py.224: train: epoch 95 | batch 1830/2304 [I 2022-10-28 00:11:00] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:25 [I 2022-10-28 00:11:00] ...trap/engines/engine.py.229: process: 0.57837 | load: 0.00246 [I 2022-10-28 00:11:00] ...trap/engines/engine.py.230: loss: 0.13567 [I 2022-10-28 00:11:00] ...optimizers/trijoint.py.151: total_norm: 24.087984 [I 2022-10-28 00:11:05] ...trap/engines/engine.py.224: train: epoch 95 | batch 1840/2304 [I 2022-10-28 00:11:05] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:04:19 [I 2022-10-28 00:11:05] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00248 [I 2022-10-28 00:11:05] ...trap/engines/engine.py.230: loss: 0.13987 [I 2022-10-28 00:11:05] ...optimizers/trijoint.py.151: total_norm: 15.287419 [I 2022-10-28 00:11:11] ...trap/engines/engine.py.224: train: epoch 95 | batch 1850/2304 [I 2022-10-28 00:11:11] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:04:13 [I 2022-10-28 00:11:11] ...trap/engines/engine.py.229: process: 0.56352 | load: 0.00245 [I 2022-10-28 00:11:11] ...trap/engines/engine.py.230: loss: 0.12765 [I 2022-10-28 00:11:11] ...optimizers/trijoint.py.151: total_norm: 19.069414 [I 2022-10-28 00:11:16] ...trap/engines/engine.py.224: train: epoch 95 | batch 1860/2304 [I 2022-10-28 00:11:16] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:04:06 [I 2022-10-28 00:11:16] ...trap/engines/engine.py.229: process: 0.55269 | load: 0.00240 [I 2022-10-28 00:11:16] ...trap/engines/engine.py.230: loss: 0.10330 [I 2022-10-28 00:11:16] ...optimizers/trijoint.py.151: total_norm: 8.822242 [I 2022-10-28 00:11:22] ...trap/engines/engine.py.224: train: epoch 95 | batch 1870/2304 [I 2022-10-28 00:11:22] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:04:02 [I 2022-10-28 00:11:22] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00244 [I 2022-10-28 00:11:22] ...trap/engines/engine.py.230: loss: 0.13534 [I 2022-10-28 00:11:22] ...optimizers/trijoint.py.151: total_norm: 11.361759 [I 2022-10-28 00:11:28] ...trap/engines/engine.py.224: train: epoch 95 | batch 1880/2304 [I 2022-10-28 00:11:28] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:56 [I 2022-10-28 00:11:28] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00245 [I 2022-10-28 00:11:28] ...trap/engines/engine.py.230: loss: 0.11530 [I 2022-10-28 00:11:28] ...optimizers/trijoint.py.151: total_norm: 19.548336 [I 2022-10-28 00:11:33] ...trap/engines/engine.py.224: train: epoch 95 | batch 1890/2304 [I 2022-10-28 00:11:33] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:52 [I 2022-10-28 00:11:33] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00248 [I 2022-10-28 00:11:33] ...trap/engines/engine.py.230: loss: 0.10739 [I 2022-10-28 00:11:33] ...optimizers/trijoint.py.151: total_norm: 10.055990 [I 2022-10-28 00:11:39] ...trap/engines/engine.py.224: train: epoch 95 | batch 1900/2304 [I 2022-10-28 00:11:39] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:46 [I 2022-10-28 00:11:39] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00244 [I 2022-10-28 00:11:39] ...trap/engines/engine.py.230: loss: 0.12588 [I 2022-10-28 00:11:39] ...optimizers/trijoint.py.151: total_norm: 13.247105 [I 2022-10-28 00:11:45] ...trap/engines/engine.py.224: train: epoch 95 | batch 1910/2304 [I 2022-10-28 00:11:45] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:41 [I 2022-10-28 00:11:45] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00244 [I 2022-10-28 00:11:45] ...trap/engines/engine.py.230: loss: 0.13708 [I 2022-10-28 00:11:45] ...optimizers/trijoint.py.151: total_norm: 27.177734 [I 2022-10-28 00:11:50] ...trap/engines/engine.py.224: train: epoch 95 | batch 1920/2304 [I 2022-10-28 00:11:50] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:36 [I 2022-10-28 00:11:50] ...trap/engines/engine.py.229: process: 0.56542 | load: 0.00243 [I 2022-10-28 00:11:50] ...trap/engines/engine.py.230: loss: 0.11442 [I 2022-10-28 00:11:50] ...optimizers/trijoint.py.151: total_norm: 14.572599 [I 2022-10-28 00:11:56] ...trap/engines/engine.py.224: train: epoch 95 | batch 1930/2304 [I 2022-10-28 00:11:56] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:29 [I 2022-10-28 00:11:56] ...trap/engines/engine.py.229: process: 0.57288 | load: 0.00258 [I 2022-10-28 00:11:56] ...trap/engines/engine.py.230: loss: 0.14239 [I 2022-10-28 00:11:56] ...optimizers/trijoint.py.151: total_norm: 10.066374 [I 2022-10-28 00:12:01] ...trap/engines/engine.py.224: train: epoch 95 | batch 1940/2304 [I 2022-10-28 00:12:01] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:21 [I 2022-10-28 00:12:01] ...trap/engines/engine.py.229: process: 0.54115 | load: 0.00254 [I 2022-10-28 00:12:01] ...trap/engines/engine.py.230: loss: 0.08761 [I 2022-10-28 00:12:01] ...optimizers/trijoint.py.151: total_norm: 9.356904 [I 2022-10-28 00:12:07] ...trap/engines/engine.py.224: train: epoch 95 | batch 1950/2304 [I 2022-10-28 00:12:07] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:18 [I 2022-10-28 00:12:07] ...trap/engines/engine.py.229: process: 0.57244 | load: 0.00249 [I 2022-10-28 00:12:07] ...trap/engines/engine.py.230: loss: 0.09051 [I 2022-10-28 00:12:07] ...optimizers/trijoint.py.151: total_norm: 8.643551 [I 2022-10-28 00:12:13] ...trap/engines/engine.py.224: train: epoch 95 | batch 1960/2304 [I 2022-10-28 00:12:13] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:03:11 [I 2022-10-28 00:12:13] ...trap/engines/engine.py.229: process: 0.55584 | load: 0.00250 [I 2022-10-28 00:12:13] ...trap/engines/engine.py.230: loss: 0.11483 [I 2022-10-28 00:12:13] ...optimizers/trijoint.py.151: total_norm: 12.891628 [I 2022-10-28 00:12:18] ...trap/engines/engine.py.224: train: epoch 95 | batch 1970/2304 [I 2022-10-28 00:12:18] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:03:06 [I 2022-10-28 00:12:18] ...trap/engines/engine.py.229: process: 0.55258 | load: 0.00241 [I 2022-10-28 00:12:18] ...trap/engines/engine.py.230: loss: 0.17040 [I 2022-10-28 00:12:18] ...optimizers/trijoint.py.151: total_norm: 13.397655 [I 2022-10-28 00:12:24] ...trap/engines/engine.py.224: train: epoch 95 | batch 1980/2304 [I 2022-10-28 00:12:24] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:03:00 [I 2022-10-28 00:12:24] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00245 [I 2022-10-28 00:12:24] ...trap/engines/engine.py.230: loss: 0.11256 [I 2022-10-28 00:12:24] ...optimizers/trijoint.py.151: total_norm: 9.619498 [I 2022-10-28 00:12:29] ...trap/engines/engine.py.224: train: epoch 95 | batch 1990/2304 [I 2022-10-28 00:12:29] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:54 [I 2022-10-28 00:12:29] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00242 [I 2022-10-28 00:12:29] ...trap/engines/engine.py.230: loss: 0.11568 [I 2022-10-28 00:12:29] ...optimizers/trijoint.py.151: total_norm: 13.361616 [I 2022-10-28 00:12:35] ...trap/engines/engine.py.224: train: epoch 95 | batch 2000/2304 [I 2022-10-28 00:12:35] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:50 [I 2022-10-28 00:12:35] ...trap/engines/engine.py.229: process: 0.56927 | load: 0.00243 [I 2022-10-28 00:12:35] ...trap/engines/engine.py.230: loss: 0.14562 [I 2022-10-28 00:12:35] ...optimizers/trijoint.py.151: total_norm: 13.974116 [I 2022-10-28 00:12:41] ...trap/engines/engine.py.224: train: epoch 95 | batch 2010/2304 [I 2022-10-28 00:12:41] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:44 [I 2022-10-28 00:12:41] ...trap/engines/engine.py.229: process: 0.56273 | load: 0.00243 [I 2022-10-28 00:12:41] ...trap/engines/engine.py.230: loss: 0.10866 [I 2022-10-28 00:12:41] ...optimizers/trijoint.py.151: total_norm: 14.036716 [I 2022-10-28 00:12:46] ...trap/engines/engine.py.224: train: epoch 95 | batch 2020/2304 [I 2022-10-28 00:12:46] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:38 [I 2022-10-28 00:12:46] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00252 [I 2022-10-28 00:12:46] ...trap/engines/engine.py.230: loss: 0.13892 [I 2022-10-28 00:12:46] ...optimizers/trijoint.py.151: total_norm: 21.830347 [I 2022-10-28 00:12:52] ...trap/engines/engine.py.224: train: epoch 95 | batch 2030/2304 [I 2022-10-28 00:12:52] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:33 [I 2022-10-28 00:12:52] ...trap/engines/engine.py.229: process: 0.56264 | load: 0.00247 [I 2022-10-28 00:12:52] ...trap/engines/engine.py.230: loss: 0.20020 [I 2022-10-28 00:12:52] ...optimizers/trijoint.py.151: total_norm: 12.205657 [I 2022-10-28 00:12:57] ...trap/engines/engine.py.224: train: epoch 95 | batch 2040/2304 [I 2022-10-28 00:12:57] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:27 [I 2022-10-28 00:12:57] ...trap/engines/engine.py.229: process: 0.54790 | load: 0.00246 [I 2022-10-28 00:12:57] ...trap/engines/engine.py.230: loss: 0.10455 [I 2022-10-28 00:12:57] ...optimizers/trijoint.py.151: total_norm: 10.769489 [I 2022-10-28 00:13:03] ...trap/engines/engine.py.224: train: epoch 95 | batch 2050/2304 [I 2022-10-28 00:13:03] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:02:21 [I 2022-10-28 00:13:03] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00247 [I 2022-10-28 00:13:03] ...trap/engines/engine.py.230: loss: 0.12324 [I 2022-10-28 00:13:03] ...optimizers/trijoint.py.151: total_norm: 12.431073 [I 2022-10-28 00:13:09] ...trap/engines/engine.py.224: train: epoch 95 | batch 2060/2304 [I 2022-10-28 00:13:09] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:02:17 [I 2022-10-28 00:13:09] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00242 [I 2022-10-28 00:13:09] ...trap/engines/engine.py.230: loss: 0.11815 [I 2022-10-28 00:13:09] ...optimizers/trijoint.py.151: total_norm: 10.636572 [I 2022-10-28 00:13:14] ...trap/engines/engine.py.224: train: epoch 95 | batch 2070/2304 [I 2022-10-28 00:13:14] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:10 [I 2022-10-28 00:13:14] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00243 [I 2022-10-28 00:13:14] ...trap/engines/engine.py.230: loss: 0.23378 [I 2022-10-28 00:13:14] ...optimizers/trijoint.py.151: total_norm: 32.701683 [I 2022-10-28 00:13:20] ...trap/engines/engine.py.224: train: epoch 95 | batch 2080/2304 [I 2022-10-28 00:13:20] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:02:05 [I 2022-10-28 00:13:20] ...trap/engines/engine.py.229: process: 0.56547 | load: 0.00244 [I 2022-10-28 00:13:20] ...trap/engines/engine.py.230: loss: 0.16730 [I 2022-10-28 00:13:20] ...optimizers/trijoint.py.151: total_norm: 25.384573 [I 2022-10-28 00:13:25] ...trap/engines/engine.py.224: train: epoch 95 | batch 2090/2304 [I 2022-10-28 00:13:25] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:59 [I 2022-10-28 00:13:25] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00243 [I 2022-10-28 00:13:25] ...trap/engines/engine.py.230: loss: 0.09782 [I 2022-10-28 00:13:25] ...optimizers/trijoint.py.151: total_norm: 9.733120 [I 2022-10-28 00:13:31] ...trap/engines/engine.py.224: train: epoch 95 | batch 2100/2304 [I 2022-10-28 00:13:31] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:53 [I 2022-10-28 00:13:31] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00244 [I 2022-10-28 00:13:31] ...trap/engines/engine.py.230: loss: 0.16161 [I 2022-10-28 00:13:31] ...optimizers/trijoint.py.151: total_norm: 16.136127 [I 2022-10-28 00:13:37] ...trap/engines/engine.py.224: train: epoch 95 | batch 2110/2304 [I 2022-10-28 00:13:37] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:47 [I 2022-10-28 00:13:37] ...trap/engines/engine.py.229: process: 0.55756 | load: 0.00241 [I 2022-10-28 00:13:37] ...trap/engines/engine.py.230: loss: 0.11292 [I 2022-10-28 00:13:37] ...optimizers/trijoint.py.151: total_norm: 21.677052 [I 2022-10-28 00:13:42] ...trap/engines/engine.py.224: train: epoch 95 | batch 2120/2304 [I 2022-10-28 00:13:42] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:42 [I 2022-10-28 00:13:42] ...trap/engines/engine.py.229: process: 0.54839 | load: 0.00248 [I 2022-10-28 00:13:42] ...trap/engines/engine.py.230: loss: 0.09388 [I 2022-10-28 00:13:42] ...optimizers/trijoint.py.151: total_norm: 10.051620 [I 2022-10-28 00:13:48] ...trap/engines/engine.py.224: train: epoch 95 | batch 2130/2304 [I 2022-10-28 00:13:48] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:36 [I 2022-10-28 00:13:48] ...trap/engines/engine.py.229: process: 0.56101 | load: 0.00242 [I 2022-10-28 00:13:48] ...trap/engines/engine.py.230: loss: 0.08531 [I 2022-10-28 00:13:48] ...optimizers/trijoint.py.151: total_norm: 15.664618 [I 2022-10-28 00:13:53] ...trap/engines/engine.py.224: train: epoch 95 | batch 2140/2304 [I 2022-10-28 00:13:53] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:31 [I 2022-10-28 00:13:53] ...trap/engines/engine.py.229: process: 0.55831 | load: 0.00245 [I 2022-10-28 00:13:53] ...trap/engines/engine.py.230: loss: 0.11526 [I 2022-10-28 00:13:53] ...optimizers/trijoint.py.151: total_norm: 14.373307 [I 2022-10-28 00:13:59] ...trap/engines/engine.py.224: train: epoch 95 | batch 2150/2304 [I 2022-10-28 00:13:59] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:26 [I 2022-10-28 00:13:59] ...trap/engines/engine.py.229: process: 0.56909 | load: 0.00249 [I 2022-10-28 00:13:59] ...trap/engines/engine.py.230: loss: 0.18392 [I 2022-10-28 00:13:59] ...optimizers/trijoint.py.151: total_norm: 35.989635 [I 2022-10-28 00:14:04] ...trap/engines/engine.py.224: train: epoch 95 | batch 2160/2304 [I 2022-10-28 00:14:04] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:01:20 [I 2022-10-28 00:14:04] ...trap/engines/engine.py.229: process: 0.55979 | load: 0.00245 [I 2022-10-28 00:14:04] ...trap/engines/engine.py.230: loss: 0.14847 [I 2022-10-28 00:14:04] ...optimizers/trijoint.py.151: total_norm: 10.013052 [I 2022-10-28 00:14:10] ...trap/engines/engine.py.224: train: epoch 95 | batch 2170/2304 [I 2022-10-28 00:14:10] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:14 [I 2022-10-28 00:14:10] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00245 [I 2022-10-28 00:14:10] ...trap/engines/engine.py.230: loss: 0.20261 [I 2022-10-28 00:14:10] ...optimizers/trijoint.py.151: total_norm: 93.791649 [I 2022-10-28 00:14:16] ...trap/engines/engine.py.224: train: epoch 95 | batch 2180/2304 [I 2022-10-28 00:14:16] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:01:08 [I 2022-10-28 00:14:16] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00245 [I 2022-10-28 00:14:16] ...trap/engines/engine.py.230: loss: 0.08661 [I 2022-10-28 00:14:16] ...optimizers/trijoint.py.151: total_norm: 13.707544 [I 2022-10-28 00:14:21] ...trap/engines/engine.py.224: train: epoch 95 | batch 2190/2304 [I 2022-10-28 00:14:21] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:01:03 [I 2022-10-28 00:14:21] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00245 [I 2022-10-28 00:14:21] ...trap/engines/engine.py.230: loss: 0.15904 [I 2022-10-28 00:14:21] ...optimizers/trijoint.py.151: total_norm: 11.698031 [I 2022-10-28 00:14:27] ...trap/engines/engine.py.224: train: epoch 95 | batch 2200/2304 [I 2022-10-28 00:14:27] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:58 [I 2022-10-28 00:14:27] ...trap/engines/engine.py.229: process: 0.55848 | load: 0.00245 [I 2022-10-28 00:14:27] ...trap/engines/engine.py.230: loss: 0.11001 [I 2022-10-28 00:14:27] ...optimizers/trijoint.py.151: total_norm: 43.432461 [I 2022-10-28 00:14:32] ...trap/engines/engine.py.224: train: epoch 95 | batch 2210/2304 [I 2022-10-28 00:14:32] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:52 [I 2022-10-28 00:14:32] ...trap/engines/engine.py.229: process: 0.56727 | load: 0.00244 [I 2022-10-28 00:14:32] ...trap/engines/engine.py.230: loss: 0.12044 [I 2022-10-28 00:14:32] ...optimizers/trijoint.py.151: total_norm: 26.342787 [I 2022-10-28 00:14:38] ...trap/engines/engine.py.224: train: epoch 95 | batch 2220/2304 [I 2022-10-28 00:14:38] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:46 [I 2022-10-28 00:14:38] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00243 [I 2022-10-28 00:14:38] ...trap/engines/engine.py.230: loss: 0.15490 [I 2022-10-28 00:14:38] ...optimizers/trijoint.py.151: total_norm: 14.723762 [I 2022-10-28 00:14:44] ...trap/engines/engine.py.224: train: epoch 95 | batch 2230/2304 [I 2022-10-28 00:14:44] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:41 [I 2022-10-28 00:14:44] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00254 [I 2022-10-28 00:14:44] ...trap/engines/engine.py.230: loss: 0.12089 [I 2022-10-28 00:14:44] ...optimizers/trijoint.py.151: total_norm: 39.246342 [I 2022-10-28 00:14:49] ...trap/engines/engine.py.224: train: epoch 95 | batch 2240/2304 [I 2022-10-28 00:14:49] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:35 [I 2022-10-28 00:14:49] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00247 [I 2022-10-28 00:14:49] ...trap/engines/engine.py.230: loss: 0.10208 [I 2022-10-28 00:14:49] ...optimizers/trijoint.py.151: total_norm: 15.845819 [I 2022-10-28 00:14:55] ...trap/engines/engine.py.224: train: epoch 95 | batch 2250/2304 [I 2022-10-28 00:14:55] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:30 [I 2022-10-28 00:14:55] ...trap/engines/engine.py.229: process: 0.54666 | load: 0.00247 [I 2022-10-28 00:14:55] ...trap/engines/engine.py.230: loss: 0.10265 [I 2022-10-28 00:14:55] ...optimizers/trijoint.py.151: total_norm: 7.146055 [I 2022-10-28 00:15:00] ...trap/engines/engine.py.224: train: epoch 95 | batch 2260/2304 [I 2022-10-28 00:15:00] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:24 [I 2022-10-28 00:15:00] ...trap/engines/engine.py.229: process: 0.56371 | load: 0.00243 [I 2022-10-28 00:15:00] ...trap/engines/engine.py.230: loss: 0.09399 [I 2022-10-28 00:15:00] ...optimizers/trijoint.py.151: total_norm: 8.215055 [I 2022-10-28 00:15:06] ...trap/engines/engine.py.224: train: epoch 95 | batch 2270/2304 [I 2022-10-28 00:15:06] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:18 [I 2022-10-28 00:15:06] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00241 [I 2022-10-28 00:15:06] ...trap/engines/engine.py.230: loss: 0.10367 [I 2022-10-28 00:15:06] ...optimizers/trijoint.py.151: total_norm: 21.570255 [I 2022-10-28 00:15:12] ...trap/engines/engine.py.224: train: epoch 95 | batch 2280/2304 [I 2022-10-28 00:15:12] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:13 [I 2022-10-28 00:15:12] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00243 [I 2022-10-28 00:15:12] ...trap/engines/engine.py.230: loss: 0.09235 [I 2022-10-28 00:15:12] ...optimizers/trijoint.py.151: total_norm: 14.603091 [I 2022-10-28 00:15:17] ...trap/engines/engine.py.224: train: epoch 95 | batch 2290/2304 [I 2022-10-28 00:15:17] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:07 [I 2022-10-28 00:15:17] ...trap/engines/engine.py.229: process: 0.56211 | load: 0.00240 [I 2022-10-28 00:15:17] ...trap/engines/engine.py.230: loss: 0.15805 [I 2022-10-28 00:15:17] ...optimizers/trijoint.py.151: total_norm: 13.598996 [I 2022-10-28 00:15:23] ...trap/engines/engine.py.224: train: epoch 95 | batch 2300/2304 [I 2022-10-28 00:15:23] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:02 [I 2022-10-28 00:15:23] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00015 [I 2022-10-28 00:15:23] ...trap/engines/engine.py.230: loss: 0.16334 [I 2022-10-28 00:15:23] ...optimizers/trijoint.py.151: total_norm: 21.735764 [I 2022-10-28 00:15:25] ...trap/engines/engine.py.224: train: epoch 95 | batch 2304/2304 [I 2022-10-28 00:15:25] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:00:00 [I 2022-10-28 00:15:25] ...trap/engines/engine.py.229: process: 0.53887 | load: 0.00017 [I 2022-10-28 00:15:25] ...trap/engines/engine.py.230: loss: 0.11755 [I 2022-10-28 00:15:25] ...optimizers/trijoint.py.151: total_norm: 17.487753 [S 2022-10-28 00:15:26] ...trap/engines/engine.py.129: train_epoch.epoch: 95 [S 2022-10-28 00:15:26] ...trap/engines/engine.py.129: train_epoch.loss: 0.1282546886168178 [I 2022-10-28 00:15:26] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 00:15:28] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 95 [I 2022-10-28 00:15:28] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 00:15:30] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 00:15:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 00:15:30] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 00:15:33] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 00:15:33] ...trap/engines/engine.py.321: eval: epoch 95 | batch 0/510 [I 2022-10-28 00:15:33] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:46:18 [I 2022-10-28 00:15:33] ...trap/engines/engine.py.326: process: 5.44717 | load: 3.03215 [I 2022-10-28 00:15:36] ...trap/engines/engine.py.321: eval: epoch 95 | batch 10/510 [I 2022-10-28 00:15:36] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:20 [I 2022-10-28 00:15:36] ...trap/engines/engine.py.326: process: 0.33055 | load: 0.00020 [I 2022-10-28 00:15:40] ...trap/engines/engine.py.321: eval: epoch 95 | batch 20/510 [I 2022-10-28 00:15:40] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:11 [I 2022-10-28 00:15:40] ...trap/engines/engine.py.326: process: 0.33194 | load: 0.00019 [I 2022-10-28 00:15:43] ...trap/engines/engine.py.321: eval: epoch 95 | batch 30/510 [I 2022-10-28 00:15:43] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:40 [I 2022-10-28 00:15:43] ...trap/engines/engine.py.326: process: 0.32396 | load: 0.00018 [I 2022-10-28 00:15:46] ...trap/engines/engine.py.321: eval: epoch 95 | batch 40/510 [I 2022-10-28 00:15:46] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:37 [I 2022-10-28 00:15:46] ...trap/engines/engine.py.326: process: 0.32737 | load: 0.00018 [I 2022-10-28 00:15:50] ...trap/engines/engine.py.321: eval: epoch 95 | batch 50/510 [I 2022-10-28 00:15:50] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:34 [I 2022-10-28 00:15:50] ...trap/engines/engine.py.326: process: 0.35195 | load: 0.00023 [I 2022-10-28 00:15:53] ...trap/engines/engine.py.321: eval: epoch 95 | batch 60/510 [I 2022-10-28 00:15:53] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:33 [I 2022-10-28 00:15:53] ...trap/engines/engine.py.326: process: 0.35049 | load: 0.00018 [I 2022-10-28 00:15:57] ...trap/engines/engine.py.321: eval: epoch 95 | batch 70/510 [I 2022-10-28 00:15:57] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:28 [I 2022-10-28 00:15:57] ...trap/engines/engine.py.326: process: 0.32388 | load: 0.00020 [I 2022-10-28 00:16:00] ...trap/engines/engine.py.321: eval: epoch 95 | batch 80/510 [I 2022-10-28 00:16:00] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:23 [I 2022-10-28 00:16:00] ...trap/engines/engine.py.326: process: 0.34033 | load: 0.00021 [I 2022-10-28 00:16:03] ...trap/engines/engine.py.321: eval: epoch 95 | batch 90/510 [I 2022-10-28 00:16:03] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-28 00:16:03] ...trap/engines/engine.py.326: process: 0.33186 | load: 0.00020 [I 2022-10-28 00:16:07] ...trap/engines/engine.py.321: eval: epoch 95 | batch 100/510 [I 2022-10-28 00:16:07] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:12 [I 2022-10-28 00:16:07] ...trap/engines/engine.py.326: process: 0.28895 | load: 0.00019 [I 2022-10-28 00:16:09] ...trap/engines/engine.py.321: eval: epoch 95 | batch 110/510 [I 2022-10-28 00:16:09] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:55 [I 2022-10-28 00:16:09] ...trap/engines/engine.py.326: process: 0.28395 | load: 0.00016 [I 2022-10-28 00:16:12] ...trap/engines/engine.py.321: eval: epoch 95 | batch 120/510 [I 2022-10-28 00:16:12] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-28 00:16:12] ...trap/engines/engine.py.326: process: 0.29010 | load: 0.00016 [I 2022-10-28 00:16:15] ...trap/engines/engine.py.321: eval: epoch 95 | batch 130/510 [I 2022-10-28 00:16:15] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-28 00:16:15] ...trap/engines/engine.py.326: process: 0.29104 | load: 0.00022 [I 2022-10-28 00:16:18] ...trap/engines/engine.py.321: eval: epoch 95 | batch 140/510 [I 2022-10-28 00:16:18] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-28 00:16:18] ...trap/engines/engine.py.326: process: 0.29923 | load: 0.00015 [I 2022-10-28 00:16:21] ...trap/engines/engine.py.321: eval: epoch 95 | batch 150/510 [I 2022-10-28 00:16:21] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-28 00:16:21] ...trap/engines/engine.py.326: process: 0.28852 | load: 0.00016 [I 2022-10-28 00:16:24] ...trap/engines/engine.py.321: eval: epoch 95 | batch 160/510 [I 2022-10-28 00:16:24] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:43 [I 2022-10-28 00:16:24] ...trap/engines/engine.py.326: process: 0.30500 | load: 0.00015 [I 2022-10-28 00:16:27] ...trap/engines/engine.py.321: eval: epoch 95 | batch 170/510 [I 2022-10-28 00:16:27] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:39 [I 2022-10-28 00:16:27] ...trap/engines/engine.py.326: process: 0.28253 | load: 0.00021 [I 2022-10-28 00:16:30] ...trap/engines/engine.py.321: eval: epoch 95 | batch 180/510 [I 2022-10-28 00:16:30] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-28 00:16:30] ...trap/engines/engine.py.326: process: 0.27896 | load: 0.00022 [I 2022-10-28 00:16:33] ...trap/engines/engine.py.321: eval: epoch 95 | batch 190/510 [I 2022-10-28 00:16:33] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:33 [I 2022-10-28 00:16:33] ...trap/engines/engine.py.326: process: 0.29073 | load: 0.00020 [I 2022-10-28 00:16:36] ...trap/engines/engine.py.321: eval: epoch 95 | batch 200/510 [I 2022-10-28 00:16:36] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:28 [I 2022-10-28 00:16:36] ...trap/engines/engine.py.326: process: 0.28056 | load: 0.00015 [I 2022-10-28 00:16:38] ...trap/engines/engine.py.321: eval: epoch 95 | batch 210/510 [I 2022-10-28 00:16:38] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:30 [I 2022-10-28 00:16:38] ...trap/engines/engine.py.326: process: 0.28725 | load: 0.00022 [I 2022-10-28 00:16:41] ...trap/engines/engine.py.321: eval: epoch 95 | batch 220/510 [I 2022-10-28 00:16:41] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-28 00:16:41] ...trap/engines/engine.py.326: process: 0.28528 | load: 0.00015 [I 2022-10-28 00:16:44] ...trap/engines/engine.py.321: eval: epoch 95 | batch 230/510 [I 2022-10-28 00:16:44] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-28 00:16:44] ...trap/engines/engine.py.326: process: 0.28542 | load: 0.00021 [I 2022-10-28 00:16:47] ...trap/engines/engine.py.321: eval: epoch 95 | batch 240/510 [I 2022-10-28 00:16:47] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 00:16:47] ...trap/engines/engine.py.326: process: 0.28735 | load: 0.00015 [I 2022-10-28 00:16:50] ...trap/engines/engine.py.321: eval: epoch 95 | batch 250/510 [I 2022-10-28 00:16:50] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-28 00:16:50] ...trap/engines/engine.py.326: process: 0.28696 | load: 0.00015 [I 2022-10-28 00:16:53] ...trap/engines/engine.py.321: eval: epoch 95 | batch 260/510 [I 2022-10-28 00:16:53] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:11 [I 2022-10-28 00:16:53] ...trap/engines/engine.py.326: process: 0.28511 | load: 0.00016 [I 2022-10-28 00:16:56] ...trap/engines/engine.py.321: eval: epoch 95 | batch 270/510 [I 2022-10-28 00:16:56] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-28 00:16:56] ...trap/engines/engine.py.326: process: 0.28266 | load: 0.00023 [I 2022-10-28 00:16:59] ...trap/engines/engine.py.321: eval: epoch 95 | batch 280/510 [I 2022-10-28 00:16:59] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-28 00:16:59] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00017 [I 2022-10-28 00:17:01] ...trap/engines/engine.py.321: eval: epoch 95 | batch 290/510 [I 2022-10-28 00:17:01] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:02 [I 2022-10-28 00:17:01] ...trap/engines/engine.py.326: process: 0.29128 | load: 0.00019 [I 2022-10-28 00:17:04] ...trap/engines/engine.py.321: eval: epoch 95 | batch 300/510 [I 2022-10-28 00:17:04] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:00:59 [I 2022-10-28 00:17:04] ...trap/engines/engine.py.326: process: 0.28560 | load: 0.00015 [I 2022-10-28 00:17:07] ...trap/engines/engine.py.321: eval: epoch 95 | batch 310/510 [I 2022-10-28 00:17:07] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 00:17:07] ...trap/engines/engine.py.326: process: 0.28802 | load: 0.00015 [I 2022-10-28 00:17:10] ...trap/engines/engine.py.321: eval: epoch 95 | batch 320/510 [I 2022-10-28 00:17:10] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 00:17:10] ...trap/engines/engine.py.326: process: 0.28862 | load: 0.00023 [I 2022-10-28 00:17:13] ...trap/engines/engine.py.321: eval: epoch 95 | batch 330/510 [I 2022-10-28 00:17:13] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-28 00:17:13] ...trap/engines/engine.py.326: process: 0.29074 | load: 0.00021 [I 2022-10-28 00:17:16] ...trap/engines/engine.py.321: eval: epoch 95 | batch 340/510 [I 2022-10-28 00:17:16] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:50 [I 2022-10-28 00:17:16] ...trap/engines/engine.py.326: process: 0.28747 | load: 0.00015 [I 2022-10-28 00:17:19] ...trap/engines/engine.py.321: eval: epoch 95 | batch 350/510 [I 2022-10-28 00:17:19] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 00:17:19] ...trap/engines/engine.py.326: process: 0.29213 | load: 0.00020 [I 2022-10-28 00:17:22] ...trap/engines/engine.py.321: eval: epoch 95 | batch 360/510 [I 2022-10-28 00:17:22] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 00:17:22] ...trap/engines/engine.py.326: process: 0.28521 | load: 0.00015 [I 2022-10-28 00:17:25] ...trap/engines/engine.py.321: eval: epoch 95 | batch 370/510 [I 2022-10-28 00:17:25] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-28 00:17:25] ...trap/engines/engine.py.326: process: 0.29196 | load: 0.00016 [I 2022-10-28 00:17:27] ...trap/engines/engine.py.321: eval: epoch 95 | batch 380/510 [I 2022-10-28 00:17:27] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-28 00:17:27] ...trap/engines/engine.py.326: process: 0.28617 | load: 0.00016 [I 2022-10-28 00:17:30] ...trap/engines/engine.py.321: eval: epoch 95 | batch 390/510 [I 2022-10-28 00:17:30] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 00:17:30] ...trap/engines/engine.py.326: process: 0.28624 | load: 0.00024 [I 2022-10-28 00:17:33] ...trap/engines/engine.py.321: eval: epoch 95 | batch 400/510 [I 2022-10-28 00:17:33] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 00:17:33] ...trap/engines/engine.py.326: process: 0.28274 | load: 0.00016 [I 2022-10-28 00:17:36] ...trap/engines/engine.py.321: eval: epoch 95 | batch 410/510 [I 2022-10-28 00:17:36] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 00:17:36] ...trap/engines/engine.py.326: process: 0.28852 | load: 0.00020 [I 2022-10-28 00:17:39] ...trap/engines/engine.py.321: eval: epoch 95 | batch 420/510 [I 2022-10-28 00:17:39] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-28 00:17:39] ...trap/engines/engine.py.326: process: 0.30976 | load: 0.00016 [I 2022-10-28 00:17:42] ...trap/engines/engine.py.321: eval: epoch 95 | batch 430/510 [I 2022-10-28 00:17:42] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 00:17:42] ...trap/engines/engine.py.326: process: 0.29143 | load: 0.00020 [I 2022-10-28 00:17:45] ...trap/engines/engine.py.321: eval: epoch 95 | batch 440/510 [I 2022-10-28 00:17:45] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 00:17:45] ...trap/engines/engine.py.326: process: 0.29247 | load: 0.00015 [I 2022-10-28 00:17:48] ...trap/engines/engine.py.321: eval: epoch 95 | batch 450/510 [I 2022-10-28 00:17:48] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 00:17:48] ...trap/engines/engine.py.326: process: 0.28734 | load: 0.00022 [I 2022-10-28 00:17:51] ...trap/engines/engine.py.321: eval: epoch 95 | batch 460/510 [I 2022-10-28 00:17:51] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 00:17:51] ...trap/engines/engine.py.326: process: 0.28739 | load: 0.00015 [I 2022-10-28 00:17:54] ...trap/engines/engine.py.321: eval: epoch 95 | batch 470/510 [I 2022-10-28 00:17:54] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-28 00:17:54] ...trap/engines/engine.py.326: process: 0.29335 | load: 0.00020 [I 2022-10-28 00:17:57] ...trap/engines/engine.py.321: eval: epoch 95 | batch 480/510 [I 2022-10-28 00:17:57] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 00:17:57] ...trap/engines/engine.py.326: process: 0.28908 | load: 0.00015 [I 2022-10-28 00:17:59] ...trap/engines/engine.py.321: eval: epoch 95 | batch 490/510 [I 2022-10-28 00:17:59] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 00:17:59] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00022 [I 2022-10-28 00:18:02] ...trap/engines/engine.py.321: eval: epoch 95 | batch 500/510 [I 2022-10-28 00:18:02] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 00:18:02] ...trap/engines/engine.py.326: process: 0.29176 | load: 0.00015 [I 2022-10-28 00:18:05] ...trap/engines/engine.py.321: eval: epoch 95 | batch 510/510 [I 2022-10-28 00:18:05] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 00:18:05] ...trap/engines/engine.py.326: process: 0.28951 | load: 0.00019 [S 2022-10-28 00:18:06] ...trap/engines/engine.py.132: eval_epoch.epoch: 95 [I 2022-10-28 00:18:06] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.735600 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906100 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936700 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.722500 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907100 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935700 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009124 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010894 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.011296 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008913 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011131 [S 2022-10-28 00:18:10] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010238 [I 2022-10-28 00:18:12] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 00:18:12] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 00:18:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 00:18:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 00:18:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 00:18:17] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 00:18:18] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 00:18:24] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 00:18:24] ...trap/engines/engine.py.163: Training model on trainset for epoch 96 [I 2022-10-28 00:18:24] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 00:18:26] ...trap/engines/engine.py.224: train: epoch 96 | batch 0/2304 [I 2022-10-28 00:18:26] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:03 [I 2022-10-28 00:18:26] ...trap/engines/engine.py.229: process: 2.68381 | load: 2.04530 [I 2022-10-28 00:18:26] ...trap/engines/engine.py.230: loss: 0.09863 [I 2022-10-28 00:18:26] ...optimizers/trijoint.py.151: total_norm: 15.038560 [I 2022-10-28 00:18:32] ...trap/engines/engine.py.224: train: epoch 96 | batch 10/2304 [I 2022-10-28 00:18:32] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:51 [I 2022-10-28 00:18:32] ...trap/engines/engine.py.229: process: 0.55095 | load: 0.00381 [I 2022-10-28 00:18:32] ...trap/engines/engine.py.230: loss: 0.14510 [I 2022-10-28 00:18:32] ...optimizers/trijoint.py.151: total_norm: 15.826735 [I 2022-10-28 00:18:37] ...trap/engines/engine.py.224: train: epoch 96 | batch 20/2304 [I 2022-10-28 00:18:37] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:10 [I 2022-10-28 00:18:37] ...trap/engines/engine.py.229: process: 0.54816 | load: 0.00344 [I 2022-10-28 00:18:37] ...trap/engines/engine.py.230: loss: 0.16153 [I 2022-10-28 00:18:37] ...optimizers/trijoint.py.151: total_norm: 13.049264 [I 2022-10-28 00:18:43] ...trap/engines/engine.py.224: train: epoch 96 | batch 30/2304 [I 2022-10-28 00:18:43] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:13 [I 2022-10-28 00:18:43] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00339 [I 2022-10-28 00:18:43] ...trap/engines/engine.py.230: loss: 0.09887 [I 2022-10-28 00:18:43] ...optimizers/trijoint.py.151: total_norm: 13.134521 [I 2022-10-28 00:18:49] ...trap/engines/engine.py.224: train: epoch 96 | batch 40/2304 [I 2022-10-28 00:18:49] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:56 [I 2022-10-28 00:18:49] ...trap/engines/engine.py.229: process: 0.54005 | load: 0.00295 [I 2022-10-28 00:18:49] ...trap/engines/engine.py.230: loss: 0.23321 [I 2022-10-28 00:18:49] ...optimizers/trijoint.py.151: total_norm: 28.332840 [I 2022-10-28 00:18:54] ...trap/engines/engine.py.224: train: epoch 96 | batch 50/2304 [I 2022-10-28 00:18:54] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:01 [I 2022-10-28 00:18:54] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00348 [I 2022-10-28 00:18:54] ...trap/engines/engine.py.230: loss: 0.16377 [I 2022-10-28 00:18:54] ...optimizers/trijoint.py.151: total_norm: 32.309338 [I 2022-10-28 00:19:00] ...trap/engines/engine.py.224: train: epoch 96 | batch 60/2304 [I 2022-10-28 00:19:00] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:42 [I 2022-10-28 00:19:00] ...trap/engines/engine.py.229: process: 0.56888 | load: 0.00294 [I 2022-10-28 00:19:00] ...trap/engines/engine.py.230: loss: 0.11876 [I 2022-10-28 00:19:00] ...optimizers/trijoint.py.151: total_norm: 8.178321 [I 2022-10-28 00:19:06] ...trap/engines/engine.py.224: train: epoch 96 | batch 70/2304 [I 2022-10-28 00:19:06] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:52 [I 2022-10-28 00:19:06] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00264 [I 2022-10-28 00:19:06] ...trap/engines/engine.py.230: loss: 0.10861 [I 2022-10-28 00:19:06] ...optimizers/trijoint.py.151: total_norm: 12.630997 [I 2022-10-28 00:19:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 80/2304 [I 2022-10-28 00:19:11] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:32 [I 2022-10-28 00:19:11] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00269 [I 2022-10-28 00:19:11] ...trap/engines/engine.py.230: loss: 0.09396 [I 2022-10-28 00:19:11] ...optimizers/trijoint.py.151: total_norm: 9.345098 [I 2022-10-28 00:19:17] ...trap/engines/engine.py.224: train: epoch 96 | batch 90/2304 [I 2022-10-28 00:19:17] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:21 [I 2022-10-28 00:19:17] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00261 [I 2022-10-28 00:19:17] ...trap/engines/engine.py.230: loss: 0.08160 [I 2022-10-28 00:19:17] ...optimizers/trijoint.py.151: total_norm: 16.831438 [I 2022-10-28 00:19:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 100/2304 [I 2022-10-28 00:19:22] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:26 [I 2022-10-28 00:19:22] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00264 [I 2022-10-28 00:19:22] ...trap/engines/engine.py.230: loss: 0.16663 [I 2022-10-28 00:19:22] ...optimizers/trijoint.py.151: total_norm: 24.456640 [I 2022-10-28 00:19:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 110/2304 [I 2022-10-28 00:19:28] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:17 [I 2022-10-28 00:19:28] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00250 [I 2022-10-28 00:19:28] ...trap/engines/engine.py.230: loss: 0.09849 [I 2022-10-28 00:19:28] ...optimizers/trijoint.py.151: total_norm: 9.632179 [I 2022-10-28 00:19:33] ...trap/engines/engine.py.224: train: epoch 96 | batch 120/2304 [I 2022-10-28 00:19:33] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:16 [I 2022-10-28 00:19:33] ...trap/engines/engine.py.229: process: 0.56162 | load: 0.00281 [I 2022-10-28 00:19:33] ...trap/engines/engine.py.230: loss: 0.10623 [I 2022-10-28 00:19:33] ...optimizers/trijoint.py.151: total_norm: 45.682831 [I 2022-10-28 00:19:39] ...trap/engines/engine.py.224: train: epoch 96 | batch 130/2304 [I 2022-10-28 00:19:39] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:14 [I 2022-10-28 00:19:39] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00248 [I 2022-10-28 00:19:39] ...trap/engines/engine.py.230: loss: 0.16358 [I 2022-10-28 00:19:39] ...optimizers/trijoint.py.151: total_norm: 10.738871 [I 2022-10-28 00:19:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 140/2304 [I 2022-10-28 00:19:45] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:00 [I 2022-10-28 00:19:45] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00246 [I 2022-10-28 00:19:45] ...trap/engines/engine.py.230: loss: 0.11368 [I 2022-10-28 00:19:45] ...optimizers/trijoint.py.151: total_norm: 27.553572 [I 2022-10-28 00:19:50] ...trap/engines/engine.py.224: train: epoch 96 | batch 150/2304 [I 2022-10-28 00:19:50] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:51 [I 2022-10-28 00:19:50] ...trap/engines/engine.py.229: process: 0.55176 | load: 0.00253 [I 2022-10-28 00:19:50] ...trap/engines/engine.py.230: loss: 0.13907 [I 2022-10-28 00:19:50] ...optimizers/trijoint.py.151: total_norm: 25.784044 [I 2022-10-28 00:19:56] ...trap/engines/engine.py.224: train: epoch 96 | batch 160/2304 [I 2022-10-28 00:19:56] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:02 [I 2022-10-28 00:19:56] ...trap/engines/engine.py.229: process: 0.56439 | load: 0.00264 [I 2022-10-28 00:19:56] ...trap/engines/engine.py.230: loss: 0.14184 [I 2022-10-28 00:19:56] ...optimizers/trijoint.py.151: total_norm: 18.516678 [I 2022-10-28 00:20:01] ...trap/engines/engine.py.224: train: epoch 96 | batch 170/2304 [I 2022-10-28 00:20:01] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:50 [I 2022-10-28 00:20:01] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00253 [I 2022-10-28 00:20:01] ...trap/engines/engine.py.230: loss: 0.08176 [I 2022-10-28 00:20:01] ...optimizers/trijoint.py.151: total_norm: 8.704394 [I 2022-10-28 00:20:07] ...trap/engines/engine.py.224: train: epoch 96 | batch 180/2304 [I 2022-10-28 00:20:07] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:58 [I 2022-10-28 00:20:07] ...trap/engines/engine.py.229: process: 0.56831 | load: 0.00253 [I 2022-10-28 00:20:07] ...trap/engines/engine.py.230: loss: 0.17096 [I 2022-10-28 00:20:07] ...optimizers/trijoint.py.151: total_norm: 19.211664 [I 2022-10-28 00:20:13] ...trap/engines/engine.py.224: train: epoch 96 | batch 190/2304 [I 2022-10-28 00:20:13] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:51 [I 2022-10-28 00:20:13] ...trap/engines/engine.py.229: process: 0.56045 | load: 0.00262 [I 2022-10-28 00:20:13] ...trap/engines/engine.py.230: loss: 0.10483 [I 2022-10-28 00:20:13] ...optimizers/trijoint.py.151: total_norm: 10.315978 [I 2022-10-28 00:20:18] ...trap/engines/engine.py.224: train: epoch 96 | batch 200/2304 [I 2022-10-28 00:20:18] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:38 [I 2022-10-28 00:20:18] ...trap/engines/engine.py.229: process: 0.57163 | load: 0.00246 [I 2022-10-28 00:20:18] ...trap/engines/engine.py.230: loss: 0.12185 [I 2022-10-28 00:20:18] ...optimizers/trijoint.py.151: total_norm: 10.277333 [I 2022-10-28 00:20:24] ...trap/engines/engine.py.224: train: epoch 96 | batch 210/2304 [I 2022-10-28 00:20:24] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:25 [I 2022-10-28 00:20:24] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00269 [I 2022-10-28 00:20:24] ...trap/engines/engine.py.230: loss: 0.09040 [I 2022-10-28 00:20:24] ...optimizers/trijoint.py.151: total_norm: 18.759933 [I 2022-10-28 00:20:29] ...trap/engines/engine.py.224: train: epoch 96 | batch 220/2304 [I 2022-10-28 00:20:29] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:24 [I 2022-10-28 00:20:29] ...trap/engines/engine.py.229: process: 0.57061 | load: 0.00250 [I 2022-10-28 00:20:29] ...trap/engines/engine.py.230: loss: 0.11607 [I 2022-10-28 00:20:29] ...optimizers/trijoint.py.151: total_norm: 9.100004 [I 2022-10-28 00:20:35] ...trap/engines/engine.py.224: train: epoch 96 | batch 230/2304 [I 2022-10-28 00:20:35] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:21 [I 2022-10-28 00:20:35] ...trap/engines/engine.py.229: process: 0.56453 | load: 0.00247 [I 2022-10-28 00:20:35] ...trap/engines/engine.py.230: loss: 0.11674 [I 2022-10-28 00:20:35] ...optimizers/trijoint.py.151: total_norm: 35.716484 [I 2022-10-28 00:21:00] ...trap/engines/engine.py.224: train: epoch 96 | batch 240/2304 [I 2022-10-28 00:21:00] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:46:37 [I 2022-10-28 00:21:00] ...trap/engines/engine.py.229: process: 0.56293 | load: 0.00240 [I 2022-10-28 00:21:00] ...trap/engines/engine.py.230: loss: 0.11039 [I 2022-10-28 00:21:00] ...optimizers/trijoint.py.151: total_norm: 13.230429 [I 2022-10-28 00:21:05] ...trap/engines/engine.py.224: train: epoch 96 | batch 250/2304 [I 2022-10-28 00:21:05] ...trap/engines/engine.py.228: elapsed: 0:02:41 | left: 0:22:08 [I 2022-10-28 00:21:05] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00236 [I 2022-10-28 00:21:05] ...trap/engines/engine.py.230: loss: 0.12491 [I 2022-10-28 00:21:05] ...optimizers/trijoint.py.151: total_norm: 24.861105 [I 2022-10-28 00:21:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 260/2304 [I 2022-10-28 00:21:11] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:19:27 [I 2022-10-28 00:21:11] ...trap/engines/engine.py.229: process: 0.54806 | load: 0.00238 [I 2022-10-28 00:21:11] ...trap/engines/engine.py.230: loss: 0.12149 [I 2022-10-28 00:21:11] ...optimizers/trijoint.py.151: total_norm: 7.334636 [I 2022-10-28 00:21:17] ...trap/engines/engine.py.224: train: epoch 96 | batch 270/2304 [I 2022-10-28 00:21:17] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:19:00 [I 2022-10-28 00:21:17] ...trap/engines/engine.py.229: process: 0.56633 | load: 0.00247 [I 2022-10-28 00:21:17] ...trap/engines/engine.py.230: loss: 0.10526 [I 2022-10-28 00:21:17] ...optimizers/trijoint.py.151: total_norm: 19.563168 [I 2022-10-28 00:21:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 280/2304 [I 2022-10-28 00:21:22] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:18:53 [I 2022-10-28 00:21:22] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00278 [I 2022-10-28 00:21:22] ...trap/engines/engine.py.230: loss: 0.11332 [I 2022-10-28 00:21:22] ...optimizers/trijoint.py.151: total_norm: 24.911661 [I 2022-10-28 00:21:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 290/2304 [I 2022-10-28 00:21:28] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:18:44 [I 2022-10-28 00:21:28] ...trap/engines/engine.py.229: process: 0.55056 | load: 0.00235 [I 2022-10-28 00:21:28] ...trap/engines/engine.py.230: loss: 0.09989 [I 2022-10-28 00:21:28] ...optimizers/trijoint.py.151: total_norm: 9.882600 [I 2022-10-28 00:21:33] ...trap/engines/engine.py.224: train: epoch 96 | batch 300/2304 [I 2022-10-28 00:21:33] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:18:33 [I 2022-10-28 00:21:33] ...trap/engines/engine.py.229: process: 0.55383 | load: 0.00238 [I 2022-10-28 00:21:33] ...trap/engines/engine.py.230: loss: 0.09889 [I 2022-10-28 00:21:33] ...optimizers/trijoint.py.151: total_norm: 17.220617 [I 2022-10-28 00:21:39] ...trap/engines/engine.py.224: train: epoch 96 | batch 310/2304 [I 2022-10-28 00:21:39] ...trap/engines/engine.py.228: elapsed: 0:03:15 | left: 0:18:28 [I 2022-10-28 00:21:39] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00243 [I 2022-10-28 00:21:39] ...trap/engines/engine.py.230: loss: 0.13300 [I 2022-10-28 00:21:39] ...optimizers/trijoint.py.151: total_norm: 13.475193 [I 2022-10-28 00:21:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 320/2304 [I 2022-10-28 00:21:45] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:18:29 [I 2022-10-28 00:21:45] ...trap/engines/engine.py.229: process: 0.56499 | load: 0.00252 [I 2022-10-28 00:21:45] ...trap/engines/engine.py.230: loss: 0.10202 [I 2022-10-28 00:21:45] ...optimizers/trijoint.py.151: total_norm: 41.257381 [I 2022-10-28 00:21:50] ...trap/engines/engine.py.224: train: epoch 96 | batch 330/2304 [I 2022-10-28 00:21:50] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:18:22 [I 2022-10-28 00:21:50] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00236 [I 2022-10-28 00:21:50] ...trap/engines/engine.py.230: loss: 0.11514 [I 2022-10-28 00:21:50] ...optimizers/trijoint.py.151: total_norm: 11.706691 [I 2022-10-28 00:21:56] ...trap/engines/engine.py.224: train: epoch 96 | batch 340/2304 [I 2022-10-28 00:21:56] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:18:19 [I 2022-10-28 00:21:56] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00239 [I 2022-10-28 00:21:56] ...trap/engines/engine.py.230: loss: 0.11644 [I 2022-10-28 00:21:56] ...optimizers/trijoint.py.151: total_norm: 22.485046 [I 2022-10-28 00:22:01] ...trap/engines/engine.py.224: train: epoch 96 | batch 350/2304 [I 2022-10-28 00:22:01] ...trap/engines/engine.py.228: elapsed: 0:03:37 | left: 0:18:11 [I 2022-10-28 00:22:01] ...trap/engines/engine.py.229: process: 0.55679 | load: 0.00243 [I 2022-10-28 00:22:01] ...trap/engines/engine.py.230: loss: 0.10613 [I 2022-10-28 00:22:01] ...optimizers/trijoint.py.151: total_norm: 17.729731 [I 2022-10-28 00:22:07] ...trap/engines/engine.py.224: train: epoch 96 | batch 360/2304 [I 2022-10-28 00:22:07] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:18:03 [I 2022-10-28 00:22:07] ...trap/engines/engine.py.229: process: 0.56246 | load: 0.00261 [I 2022-10-28 00:22:07] ...trap/engines/engine.py.230: loss: 0.10197 [I 2022-10-28 00:22:07] ...optimizers/trijoint.py.151: total_norm: 10.412546 [I 2022-10-28 00:22:12] ...trap/engines/engine.py.224: train: epoch 96 | batch 370/2304 [I 2022-10-28 00:22:12] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:17:51 [I 2022-10-28 00:22:12] ...trap/engines/engine.py.229: process: 0.55269 | load: 0.00236 [I 2022-10-28 00:22:12] ...trap/engines/engine.py.230: loss: 0.08616 [I 2022-10-28 00:22:12] ...optimizers/trijoint.py.151: total_norm: 10.027289 [I 2022-10-28 00:22:18] ...trap/engines/engine.py.224: train: epoch 96 | batch 380/2304 [I 2022-10-28 00:22:18] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:17:48 [I 2022-10-28 00:22:18] ...trap/engines/engine.py.229: process: 0.55113 | load: 0.00242 [I 2022-10-28 00:22:18] ...trap/engines/engine.py.230: loss: 0.08617 [I 2022-10-28 00:22:18] ...optimizers/trijoint.py.151: total_norm: 10.392907 [I 2022-10-28 00:22:24] ...trap/engines/engine.py.224: train: epoch 96 | batch 390/2304 [I 2022-10-28 00:22:24] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:17:43 [I 2022-10-28 00:22:24] ...trap/engines/engine.py.229: process: 0.54663 | load: 0.00254 [I 2022-10-28 00:22:24] ...trap/engines/engine.py.230: loss: 0.11118 [I 2022-10-28 00:22:24] ...optimizers/trijoint.py.151: total_norm: 7.513227 [I 2022-10-28 00:22:29] ...trap/engines/engine.py.224: train: epoch 96 | batch 400/2304 [I 2022-10-28 00:22:29] ...trap/engines/engine.py.228: elapsed: 0:04:05 | left: 0:17:43 [I 2022-10-28 00:22:29] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00241 [I 2022-10-28 00:22:29] ...trap/engines/engine.py.230: loss: 0.15584 [I 2022-10-28 00:22:29] ...optimizers/trijoint.py.151: total_norm: 13.864450 [I 2022-10-28 00:22:35] ...trap/engines/engine.py.224: train: epoch 96 | batch 410/2304 [I 2022-10-28 00:22:35] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:17:34 [I 2022-10-28 00:22:35] ...trap/engines/engine.py.229: process: 0.56598 | load: 0.00248 [I 2022-10-28 00:22:35] ...trap/engines/engine.py.230: loss: 0.10756 [I 2022-10-28 00:22:35] ...optimizers/trijoint.py.151: total_norm: 15.687777 [I 2022-10-28 00:22:40] ...trap/engines/engine.py.224: train: epoch 96 | batch 420/2304 [I 2022-10-28 00:22:40] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:17:33 [I 2022-10-28 00:22:40] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00244 [I 2022-10-28 00:22:40] ...trap/engines/engine.py.230: loss: 0.10565 [I 2022-10-28 00:22:40] ...optimizers/trijoint.py.151: total_norm: 57.781162 [I 2022-10-28 00:22:46] ...trap/engines/engine.py.224: train: epoch 96 | batch 430/2304 [I 2022-10-28 00:22:46] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:17:26 [I 2022-10-28 00:22:46] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00240 [I 2022-10-28 00:22:46] ...trap/engines/engine.py.230: loss: 0.14148 [I 2022-10-28 00:22:46] ...optimizers/trijoint.py.151: total_norm: 23.662374 [I 2022-10-28 00:22:52] ...trap/engines/engine.py.224: train: epoch 96 | batch 440/2304 [I 2022-10-28 00:22:52] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:17:25 [I 2022-10-28 00:22:52] ...trap/engines/engine.py.229: process: 0.56524 | load: 0.00250 [I 2022-10-28 00:22:52] ...trap/engines/engine.py.230: loss: 0.08802 [I 2022-10-28 00:22:52] ...optimizers/trijoint.py.151: total_norm: 12.953860 [I 2022-10-28 00:22:57] ...trap/engines/engine.py.224: train: epoch 96 | batch 450/2304 [I 2022-10-28 00:22:57] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:17:15 [I 2022-10-28 00:22:57] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00242 [I 2022-10-28 00:22:57] ...trap/engines/engine.py.230: loss: 0.16740 [I 2022-10-28 00:22:57] ...optimizers/trijoint.py.151: total_norm: 21.087740 [I 2022-10-28 00:23:03] ...trap/engines/engine.py.224: train: epoch 96 | batch 460/2304 [I 2022-10-28 00:23:03] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:17:04 [I 2022-10-28 00:23:03] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00263 [I 2022-10-28 00:23:03] ...trap/engines/engine.py.230: loss: 0.10299 [I 2022-10-28 00:23:03] ...optimizers/trijoint.py.151: total_norm: 11.414148 [I 2022-10-28 00:23:08] ...trap/engines/engine.py.224: train: epoch 96 | batch 470/2304 [I 2022-10-28 00:23:08] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:17:02 [I 2022-10-28 00:23:08] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00243 [I 2022-10-28 00:23:08] ...trap/engines/engine.py.230: loss: 0.10139 [I 2022-10-28 00:23:08] ...optimizers/trijoint.py.151: total_norm: 10.147282 [I 2022-10-28 00:23:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 480/2304 [I 2022-10-28 00:23:14] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:17:00 [I 2022-10-28 00:23:14] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00242 [I 2022-10-28 00:23:14] ...trap/engines/engine.py.230: loss: 0.07541 [I 2022-10-28 00:23:14] ...optimizers/trijoint.py.151: total_norm: 14.335232 [I 2022-10-28 00:23:20] ...trap/engines/engine.py.224: train: epoch 96 | batch 490/2304 [I 2022-10-28 00:23:20] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:16:52 [I 2022-10-28 00:23:20] ...trap/engines/engine.py.229: process: 0.55519 | load: 0.00275 [I 2022-10-28 00:23:20] ...trap/engines/engine.py.230: loss: 0.15341 [I 2022-10-28 00:23:20] ...optimizers/trijoint.py.151: total_norm: 21.340809 [I 2022-10-28 00:23:25] ...trap/engines/engine.py.224: train: epoch 96 | batch 500/2304 [I 2022-10-28 00:23:25] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:16:56 [I 2022-10-28 00:23:25] ...trap/engines/engine.py.229: process: 0.57445 | load: 0.00246 [I 2022-10-28 00:23:25] ...trap/engines/engine.py.230: loss: 0.22743 [I 2022-10-28 00:23:25] ...optimizers/trijoint.py.151: total_norm: 24.745995 [I 2022-10-28 00:23:31] ...trap/engines/engine.py.224: train: epoch 96 | batch 510/2304 [I 2022-10-28 00:23:31] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:16:42 [I 2022-10-28 00:23:31] ...trap/engines/engine.py.229: process: 0.56378 | load: 0.00246 [I 2022-10-28 00:23:31] ...trap/engines/engine.py.230: loss: 0.11599 [I 2022-10-28 00:23:31] ...optimizers/trijoint.py.151: total_norm: 24.006977 [I 2022-10-28 00:23:36] ...trap/engines/engine.py.224: train: epoch 96 | batch 520/2304 [I 2022-10-28 00:23:36] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:16:28 [I 2022-10-28 00:23:36] ...trap/engines/engine.py.229: process: 0.54848 | load: 0.00243 [I 2022-10-28 00:23:36] ...trap/engines/engine.py.230: loss: 0.13574 [I 2022-10-28 00:23:36] ...optimizers/trijoint.py.151: total_norm: 11.255866 [I 2022-10-28 00:23:42] ...trap/engines/engine.py.224: train: epoch 96 | batch 530/2304 [I 2022-10-28 00:23:42] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:16:25 [I 2022-10-28 00:23:42] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00242 [I 2022-10-28 00:23:42] ...trap/engines/engine.py.230: loss: 0.10482 [I 2022-10-28 00:23:42] ...optimizers/trijoint.py.151: total_norm: 7.383609 [I 2022-10-28 00:23:48] ...trap/engines/engine.py.224: train: epoch 96 | batch 540/2304 [I 2022-10-28 00:23:48] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:16:20 [I 2022-10-28 00:23:48] ...trap/engines/engine.py.229: process: 0.55208 | load: 0.00242 [I 2022-10-28 00:23:48] ...trap/engines/engine.py.230: loss: 0.11366 [I 2022-10-28 00:23:48] ...optimizers/trijoint.py.151: total_norm: 21.665598 [I 2022-10-28 00:23:53] ...trap/engines/engine.py.224: train: epoch 96 | batch 550/2304 [I 2022-10-28 00:23:53] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:16:24 [I 2022-10-28 00:23:53] ...trap/engines/engine.py.229: process: 0.56996 | load: 0.00243 [I 2022-10-28 00:23:53] ...trap/engines/engine.py.230: loss: 0.10068 [I 2022-10-28 00:23:53] ...optimizers/trijoint.py.151: total_norm: 24.991833 [I 2022-10-28 00:23:59] ...trap/engines/engine.py.224: train: epoch 96 | batch 560/2304 [I 2022-10-28 00:23:59] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:16:13 [I 2022-10-28 00:23:59] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00240 [I 2022-10-28 00:23:59] ...trap/engines/engine.py.230: loss: 0.09709 [I 2022-10-28 00:23:59] ...optimizers/trijoint.py.151: total_norm: 15.198519 [I 2022-10-28 00:24:04] ...trap/engines/engine.py.224: train: epoch 96 | batch 570/2304 [I 2022-10-28 00:24:04] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:16:04 [I 2022-10-28 00:24:04] ...trap/engines/engine.py.229: process: 0.54673 | load: 0.00244 [I 2022-10-28 00:24:04] ...trap/engines/engine.py.230: loss: 0.09432 [I 2022-10-28 00:24:04] ...optimizers/trijoint.py.151: total_norm: 10.668637 [I 2022-10-28 00:24:10] ...trap/engines/engine.py.224: train: epoch 96 | batch 580/2304 [I 2022-10-28 00:24:10] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:15:57 [I 2022-10-28 00:24:10] ...trap/engines/engine.py.229: process: 0.55426 | load: 0.00249 [I 2022-10-28 00:24:10] ...trap/engines/engine.py.230: loss: 0.11529 [I 2022-10-28 00:24:10] ...optimizers/trijoint.py.151: total_norm: 14.200419 [I 2022-10-28 00:24:15] ...trap/engines/engine.py.224: train: epoch 96 | batch 590/2304 [I 2022-10-28 00:24:15] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:15:47 [I 2022-10-28 00:24:15] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00242 [I 2022-10-28 00:24:15] ...trap/engines/engine.py.230: loss: 0.07799 [I 2022-10-28 00:24:15] ...optimizers/trijoint.py.151: total_norm: 10.917750 [I 2022-10-28 00:24:21] ...trap/engines/engine.py.224: train: epoch 96 | batch 600/2304 [I 2022-10-28 00:24:21] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:15:50 [I 2022-10-28 00:24:21] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00241 [I 2022-10-28 00:24:21] ...trap/engines/engine.py.230: loss: 0.13106 [I 2022-10-28 00:24:21] ...optimizers/trijoint.py.151: total_norm: 23.669046 [I 2022-10-28 00:24:27] ...trap/engines/engine.py.224: train: epoch 96 | batch 610/2304 [I 2022-10-28 00:24:27] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:15:39 [I 2022-10-28 00:24:27] ...trap/engines/engine.py.229: process: 0.55460 | load: 0.00240 [I 2022-10-28 00:24:27] ...trap/engines/engine.py.230: loss: 0.15404 [I 2022-10-28 00:24:27] ...optimizers/trijoint.py.151: total_norm: 31.411268 [I 2022-10-28 00:24:32] ...trap/engines/engine.py.224: train: epoch 96 | batch 620/2304 [I 2022-10-28 00:24:32] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:15:41 [I 2022-10-28 00:24:32] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00257 [I 2022-10-28 00:24:32] ...trap/engines/engine.py.230: loss: 0.09729 [I 2022-10-28 00:24:32] ...optimizers/trijoint.py.151: total_norm: 10.986149 [I 2022-10-28 00:24:38] ...trap/engines/engine.py.224: train: epoch 96 | batch 630/2304 [I 2022-10-28 00:24:38] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:15:33 [I 2022-10-28 00:24:38] ...trap/engines/engine.py.229: process: 0.56631 | load: 0.00242 [I 2022-10-28 00:24:38] ...trap/engines/engine.py.230: loss: 0.10974 [I 2022-10-28 00:24:38] ...optimizers/trijoint.py.151: total_norm: 8.119113 [I 2022-10-28 00:24:43] ...trap/engines/engine.py.224: train: epoch 96 | batch 640/2304 [I 2022-10-28 00:24:43] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:15:27 [I 2022-10-28 00:24:43] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00241 [I 2022-10-28 00:24:43] ...trap/engines/engine.py.230: loss: 0.09133 [I 2022-10-28 00:24:43] ...optimizers/trijoint.py.151: total_norm: 27.956978 [I 2022-10-28 00:24:49] ...trap/engines/engine.py.224: train: epoch 96 | batch 650/2304 [I 2022-10-28 00:24:49] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:15:22 [I 2022-10-28 00:24:49] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00240 [I 2022-10-28 00:24:49] ...trap/engines/engine.py.230: loss: 0.14343 [I 2022-10-28 00:24:49] ...optimizers/trijoint.py.151: total_norm: 14.616994 [I 2022-10-28 00:24:55] ...trap/engines/engine.py.224: train: epoch 96 | batch 660/2304 [I 2022-10-28 00:24:55] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:15:12 [I 2022-10-28 00:24:55] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00241 [I 2022-10-28 00:24:55] ...trap/engines/engine.py.230: loss: 0.10819 [I 2022-10-28 00:24:55] ...optimizers/trijoint.py.151: total_norm: 9.441503 [I 2022-10-28 00:25:00] ...trap/engines/engine.py.224: train: epoch 96 | batch 670/2304 [I 2022-10-28 00:25:00] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:15:08 [I 2022-10-28 00:25:00] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00242 [I 2022-10-28 00:25:00] ...trap/engines/engine.py.230: loss: 0.10028 [I 2022-10-28 00:25:00] ...optimizers/trijoint.py.151: total_norm: 9.581657 [I 2022-10-28 00:25:06] ...trap/engines/engine.py.224: train: epoch 96 | batch 680/2304 [I 2022-10-28 00:25:06] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:15:02 [I 2022-10-28 00:25:06] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00242 [I 2022-10-28 00:25:06] ...trap/engines/engine.py.230: loss: 0.11869 [I 2022-10-28 00:25:06] ...optimizers/trijoint.py.151: total_norm: 11.875422 [I 2022-10-28 00:25:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 690/2304 [I 2022-10-28 00:25:11] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:14:55 [I 2022-10-28 00:25:11] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00242 [I 2022-10-28 00:25:11] ...trap/engines/engine.py.230: loss: 0.11317 [I 2022-10-28 00:25:11] ...optimizers/trijoint.py.151: total_norm: 16.971998 [I 2022-10-28 00:25:17] ...trap/engines/engine.py.224: train: epoch 96 | batch 700/2304 [I 2022-10-28 00:25:17] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:14:53 [I 2022-10-28 00:25:17] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00238 [I 2022-10-28 00:25:17] ...trap/engines/engine.py.230: loss: 0.08921 [I 2022-10-28 00:25:17] ...optimizers/trijoint.py.151: total_norm: 11.359770 [I 2022-10-28 00:25:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 710/2304 [I 2022-10-28 00:25:22] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:14:46 [I 2022-10-28 00:25:22] ...trap/engines/engine.py.229: process: 0.55911 | load: 0.00239 [I 2022-10-28 00:25:22] ...trap/engines/engine.py.230: loss: 0.16865 [I 2022-10-28 00:25:22] ...optimizers/trijoint.py.151: total_norm: 22.651266 [I 2022-10-28 00:25:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 720/2304 [I 2022-10-28 00:25:28] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:14:42 [I 2022-10-28 00:25:28] ...trap/engines/engine.py.229: process: 0.55200 | load: 0.00242 [I 2022-10-28 00:25:28] ...trap/engines/engine.py.230: loss: 0.15219 [I 2022-10-28 00:25:28] ...optimizers/trijoint.py.151: total_norm: 18.147938 [I 2022-10-28 00:25:34] ...trap/engines/engine.py.224: train: epoch 96 | batch 730/2304 [I 2022-10-28 00:25:34] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:29 [I 2022-10-28 00:25:34] ...trap/engines/engine.py.229: process: 0.55135 | load: 0.00291 [I 2022-10-28 00:25:34] ...trap/engines/engine.py.230: loss: 0.13264 [I 2022-10-28 00:25:34] ...optimizers/trijoint.py.151: total_norm: 21.451052 [I 2022-10-28 00:25:39] ...trap/engines/engine.py.224: train: epoch 96 | batch 740/2304 [I 2022-10-28 00:25:39] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:14:25 [I 2022-10-28 00:25:39] ...trap/engines/engine.py.229: process: 0.55138 | load: 0.00245 [I 2022-10-28 00:25:39] ...trap/engines/engine.py.230: loss: 0.10137 [I 2022-10-28 00:25:39] ...optimizers/trijoint.py.151: total_norm: 14.162393 [I 2022-10-28 00:25:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 750/2304 [I 2022-10-28 00:25:45] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:14:25 [I 2022-10-28 00:25:45] ...trap/engines/engine.py.229: process: 0.55410 | load: 0.00244 [I 2022-10-28 00:25:45] ...trap/engines/engine.py.230: loss: 0.16952 [I 2022-10-28 00:25:45] ...optimizers/trijoint.py.151: total_norm: 16.896971 [I 2022-10-28 00:25:50] ...trap/engines/engine.py.224: train: epoch 96 | batch 760/2304 [I 2022-10-28 00:25:50] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:14:21 [I 2022-10-28 00:25:50] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00242 [I 2022-10-28 00:25:50] ...trap/engines/engine.py.230: loss: 0.10969 [I 2022-10-28 00:25:50] ...optimizers/trijoint.py.151: total_norm: 15.248817 [I 2022-10-28 00:25:56] ...trap/engines/engine.py.224: train: epoch 96 | batch 770/2304 [I 2022-10-28 00:25:56] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:14:16 [I 2022-10-28 00:25:56] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00240 [I 2022-10-28 00:25:56] ...trap/engines/engine.py.230: loss: 0.12447 [I 2022-10-28 00:25:56] ...optimizers/trijoint.py.151: total_norm: 29.727188 [I 2022-10-28 00:26:01] ...trap/engines/engine.py.224: train: epoch 96 | batch 780/2304 [I 2022-10-28 00:26:01] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:14:04 [I 2022-10-28 00:26:01] ...trap/engines/engine.py.229: process: 0.55129 | load: 0.00241 [I 2022-10-28 00:26:01] ...trap/engines/engine.py.230: loss: 0.10669 [I 2022-10-28 00:26:01] ...optimizers/trijoint.py.151: total_norm: 9.249381 [I 2022-10-28 00:26:07] ...trap/engines/engine.py.224: train: epoch 96 | batch 790/2304 [I 2022-10-28 00:26:07] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:14:03 [I 2022-10-28 00:26:07] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00244 [I 2022-10-28 00:26:07] ...trap/engines/engine.py.230: loss: 0.12496 [I 2022-10-28 00:26:07] ...optimizers/trijoint.py.151: total_norm: 17.719864 [I 2022-10-28 00:26:13] ...trap/engines/engine.py.224: train: epoch 96 | batch 800/2304 [I 2022-10-28 00:26:13] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:56 [I 2022-10-28 00:26:13] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00241 [I 2022-10-28 00:26:13] ...trap/engines/engine.py.230: loss: 0.13569 [I 2022-10-28 00:26:13] ...optimizers/trijoint.py.151: total_norm: 17.746750 [I 2022-10-28 00:26:18] ...trap/engines/engine.py.224: train: epoch 96 | batch 810/2304 [I 2022-10-28 00:26:18] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:13:49 [I 2022-10-28 00:26:18] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00246 [I 2022-10-28 00:26:18] ...trap/engines/engine.py.230: loss: 0.14825 [I 2022-10-28 00:26:18] ...optimizers/trijoint.py.151: total_norm: 12.641324 [I 2022-10-28 00:26:24] ...trap/engines/engine.py.224: train: epoch 96 | batch 820/2304 [I 2022-10-28 00:26:24] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:13:47 [I 2022-10-28 00:26:24] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00242 [I 2022-10-28 00:26:24] ...trap/engines/engine.py.230: loss: 0.09833 [I 2022-10-28 00:26:24] ...optimizers/trijoint.py.151: total_norm: 9.483359 [I 2022-10-28 00:26:29] ...trap/engines/engine.py.224: train: epoch 96 | batch 830/2304 [I 2022-10-28 00:26:29] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:13:53 [I 2022-10-28 00:26:29] ...trap/engines/engine.py.229: process: 0.55164 | load: 0.00243 [I 2022-10-28 00:26:29] ...trap/engines/engine.py.230: loss: 0.14530 [I 2022-10-28 00:26:29] ...optimizers/trijoint.py.151: total_norm: 12.033081 [I 2022-10-28 00:26:35] ...trap/engines/engine.py.224: train: epoch 96 | batch 840/2304 [I 2022-10-28 00:26:35] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:14:06 [I 2022-10-28 00:26:35] ...trap/engines/engine.py.229: process: 0.54606 | load: 0.00239 [I 2022-10-28 00:26:35] ...trap/engines/engine.py.230: loss: 0.11191 [I 2022-10-28 00:26:35] ...optimizers/trijoint.py.151: total_norm: 12.164080 [I 2022-10-28 00:26:41] ...trap/engines/engine.py.224: train: epoch 96 | batch 850/2304 [I 2022-10-28 00:26:41] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:13:30 [I 2022-10-28 00:26:41] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00246 [I 2022-10-28 00:26:41] ...trap/engines/engine.py.230: loss: 0.10691 [I 2022-10-28 00:26:41] ...optimizers/trijoint.py.151: total_norm: 16.910898 [I 2022-10-28 00:26:46] ...trap/engines/engine.py.224: train: epoch 96 | batch 860/2304 [I 2022-10-28 00:26:46] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:13:24 [I 2022-10-28 00:26:46] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00245 [I 2022-10-28 00:26:46] ...trap/engines/engine.py.230: loss: 0.10114 [I 2022-10-28 00:26:46] ...optimizers/trijoint.py.151: total_norm: 10.083313 [I 2022-10-28 00:26:52] ...trap/engines/engine.py.224: train: epoch 96 | batch 870/2304 [I 2022-10-28 00:26:52] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:25 [I 2022-10-28 00:26:52] ...trap/engines/engine.py.229: process: 0.56373 | load: 0.00250 [I 2022-10-28 00:26:52] ...trap/engines/engine.py.230: loss: 0.13275 [I 2022-10-28 00:26:52] ...optimizers/trijoint.py.151: total_norm: 20.991760 [I 2022-10-28 00:26:57] ...trap/engines/engine.py.224: train: epoch 96 | batch 880/2304 [I 2022-10-28 00:26:57] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:13:08 [I 2022-10-28 00:26:57] ...trap/engines/engine.py.229: process: 0.54629 | load: 0.00246 [I 2022-10-28 00:26:57] ...trap/engines/engine.py.230: loss: 0.12379 [I 2022-10-28 00:26:57] ...optimizers/trijoint.py.151: total_norm: 16.553766 [I 2022-10-28 00:27:03] ...trap/engines/engine.py.224: train: epoch 96 | batch 890/2304 [I 2022-10-28 00:27:03] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:13:10 [I 2022-10-28 00:27:03] ...trap/engines/engine.py.229: process: 0.56696 | load: 0.00243 [I 2022-10-28 00:27:03] ...trap/engines/engine.py.230: loss: 0.13011 [I 2022-10-28 00:27:03] ...optimizers/trijoint.py.151: total_norm: 17.032732 [I 2022-10-28 00:27:09] ...trap/engines/engine.py.224: train: epoch 96 | batch 900/2304 [I 2022-10-28 00:27:09] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:13:03 [I 2022-10-28 00:27:09] ...trap/engines/engine.py.229: process: 0.55056 | load: 0.00241 [I 2022-10-28 00:27:09] ...trap/engines/engine.py.230: loss: 0.07689 [I 2022-10-28 00:27:09] ...optimizers/trijoint.py.151: total_norm: 12.379331 [I 2022-10-28 00:27:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 910/2304 [I 2022-10-28 00:27:14] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:12:59 [I 2022-10-28 00:27:14] ...trap/engines/engine.py.229: process: 0.55024 | load: 0.00245 [I 2022-10-28 00:27:14] ...trap/engines/engine.py.230: loss: 0.12990 [I 2022-10-28 00:27:14] ...optimizers/trijoint.py.151: total_norm: 17.996569 [I 2022-10-28 00:27:20] ...trap/engines/engine.py.224: train: epoch 96 | batch 920/2304 [I 2022-10-28 00:27:20] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:13:32 [I 2022-10-28 00:27:20] ...trap/engines/engine.py.229: process: 0.65294 | load: 0.00266 [I 2022-10-28 00:27:20] ...trap/engines/engine.py.230: loss: 0.14877 [I 2022-10-28 00:27:20] ...optimizers/trijoint.py.151: total_norm: 30.891146 [I 2022-10-28 00:27:26] ...trap/engines/engine.py.224: train: epoch 96 | batch 930/2304 [I 2022-10-28 00:27:26] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:12:49 [I 2022-10-28 00:27:26] ...trap/engines/engine.py.229: process: 0.55977 | load: 0.00242 [I 2022-10-28 00:27:26] ...trap/engines/engine.py.230: loss: 0.15536 [I 2022-10-28 00:27:26] ...optimizers/trijoint.py.151: total_norm: 18.424452 [I 2022-10-28 00:27:31] ...trap/engines/engine.py.224: train: epoch 96 | batch 940/2304 [I 2022-10-28 00:27:31] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:37 [I 2022-10-28 00:27:31] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00244 [I 2022-10-28 00:27:31] ...trap/engines/engine.py.230: loss: 0.11745 [I 2022-10-28 00:27:31] ...optimizers/trijoint.py.151: total_norm: 20.342190 [I 2022-10-28 00:27:37] ...trap/engines/engine.py.224: train: epoch 96 | batch 950/2304 [I 2022-10-28 00:27:37] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:43 [I 2022-10-28 00:27:37] ...trap/engines/engine.py.229: process: 0.56766 | load: 0.00249 [I 2022-10-28 00:27:37] ...trap/engines/engine.py.230: loss: 0.09551 [I 2022-10-28 00:27:37] ...optimizers/trijoint.py.151: total_norm: 21.582113 [I 2022-10-28 00:27:42] ...trap/engines/engine.py.224: train: epoch 96 | batch 960/2304 [I 2022-10-28 00:27:42] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:30 [I 2022-10-28 00:27:42] ...trap/engines/engine.py.229: process: 0.57651 | load: 0.00243 [I 2022-10-28 00:27:42] ...trap/engines/engine.py.230: loss: 0.09414 [I 2022-10-28 00:27:42] ...optimizers/trijoint.py.151: total_norm: 14.945949 [I 2022-10-28 00:27:48] ...trap/engines/engine.py.224: train: epoch 96 | batch 970/2304 [I 2022-10-28 00:27:48] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:24 [I 2022-10-28 00:27:48] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00239 [I 2022-10-28 00:27:48] ...trap/engines/engine.py.230: loss: 0.17790 [I 2022-10-28 00:27:48] ...optimizers/trijoint.py.151: total_norm: 15.979362 [I 2022-10-28 00:27:54] ...trap/engines/engine.py.224: train: epoch 96 | batch 980/2304 [I 2022-10-28 00:27:54] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:21 [I 2022-10-28 00:27:54] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00241 [I 2022-10-28 00:27:54] ...trap/engines/engine.py.230: loss: 0.14599 [I 2022-10-28 00:27:54] ...optimizers/trijoint.py.151: total_norm: 10.295998 [I 2022-10-28 00:27:59] ...trap/engines/engine.py.224: train: epoch 96 | batch 990/2304 [I 2022-10-28 00:27:59] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:12:21 [I 2022-10-28 00:27:59] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00243 [I 2022-10-28 00:27:59] ...trap/engines/engine.py.230: loss: 0.10879 [I 2022-10-28 00:27:59] ...optimizers/trijoint.py.151: total_norm: 11.119720 [I 2022-10-28 00:28:05] ...trap/engines/engine.py.224: train: epoch 96 | batch 1000/2304 [I 2022-10-28 00:28:05] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:12:08 [I 2022-10-28 00:28:05] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00242 [I 2022-10-28 00:28:05] ...trap/engines/engine.py.230: loss: 0.17252 [I 2022-10-28 00:28:05] ...optimizers/trijoint.py.151: total_norm: 39.421558 [I 2022-10-28 00:28:10] ...trap/engines/engine.py.224: train: epoch 96 | batch 1010/2304 [I 2022-10-28 00:28:10] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:12:01 [I 2022-10-28 00:28:10] ...trap/engines/engine.py.229: process: 0.56401 | load: 0.00241 [I 2022-10-28 00:28:10] ...trap/engines/engine.py.230: loss: 0.10720 [I 2022-10-28 00:28:10] ...optimizers/trijoint.py.151: total_norm: 15.727971 [I 2022-10-28 00:28:16] ...trap/engines/engine.py.224: train: epoch 96 | batch 1020/2304 [I 2022-10-28 00:28:16] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:56 [I 2022-10-28 00:28:16] ...trap/engines/engine.py.229: process: 0.57021 | load: 0.00242 [I 2022-10-28 00:28:16] ...trap/engines/engine.py.230: loss: 0.18505 [I 2022-10-28 00:28:16] ...optimizers/trijoint.py.151: total_norm: 25.725285 [I 2022-10-28 00:28:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 1030/2304 [I 2022-10-28 00:28:22] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:51 [I 2022-10-28 00:28:22] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00239 [I 2022-10-28 00:28:22] ...trap/engines/engine.py.230: loss: 0.16559 [I 2022-10-28 00:28:22] ...optimizers/trijoint.py.151: total_norm: 13.197944 [I 2022-10-28 00:28:27] ...trap/engines/engine.py.224: train: epoch 96 | batch 1040/2304 [I 2022-10-28 00:28:27] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:45 [I 2022-10-28 00:28:27] ...trap/engines/engine.py.229: process: 0.56005 | load: 0.00242 [I 2022-10-28 00:28:27] ...trap/engines/engine.py.230: loss: 0.11486 [I 2022-10-28 00:28:27] ...optimizers/trijoint.py.151: total_norm: 28.189598 [I 2022-10-28 00:28:33] ...trap/engines/engine.py.224: train: epoch 96 | batch 1050/2304 [I 2022-10-28 00:28:33] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:41 [I 2022-10-28 00:28:33] ...trap/engines/engine.py.229: process: 0.56557 | load: 0.00285 [I 2022-10-28 00:28:33] ...trap/engines/engine.py.230: loss: 0.17242 [I 2022-10-28 00:28:33] ...optimizers/trijoint.py.151: total_norm: 20.745157 [I 2022-10-28 00:28:38] ...trap/engines/engine.py.224: train: epoch 96 | batch 1060/2304 [I 2022-10-28 00:28:38] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:31 [I 2022-10-28 00:28:38] ...trap/engines/engine.py.229: process: 0.54887 | load: 0.00241 [I 2022-10-28 00:28:38] ...trap/engines/engine.py.230: loss: 0.08845 [I 2022-10-28 00:28:38] ...optimizers/trijoint.py.151: total_norm: 9.512072 [I 2022-10-28 00:28:44] ...trap/engines/engine.py.224: train: epoch 96 | batch 1070/2304 [I 2022-10-28 00:28:44] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:27 [I 2022-10-28 00:28:44] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00248 [I 2022-10-28 00:28:44] ...trap/engines/engine.py.230: loss: 0.16131 [I 2022-10-28 00:28:44] ...optimizers/trijoint.py.151: total_norm: 25.905762 [I 2022-10-28 00:28:50] ...trap/engines/engine.py.224: train: epoch 96 | batch 1080/2304 [I 2022-10-28 00:28:50] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:22 [I 2022-10-28 00:28:50] ...trap/engines/engine.py.229: process: 0.56584 | load: 0.00247 [I 2022-10-28 00:28:50] ...trap/engines/engine.py.230: loss: 0.13099 [I 2022-10-28 00:28:50] ...optimizers/trijoint.py.151: total_norm: 26.514397 [I 2022-10-28 00:28:55] ...trap/engines/engine.py.224: train: epoch 96 | batch 1090/2304 [I 2022-10-28 00:28:55] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:19 [I 2022-10-28 00:28:55] ...trap/engines/engine.py.229: process: 0.56783 | load: 0.00248 [I 2022-10-28 00:28:55] ...trap/engines/engine.py.230: loss: 0.11161 [I 2022-10-28 00:28:55] ...optimizers/trijoint.py.151: total_norm: 13.935357 [I 2022-10-28 00:29:01] ...trap/engines/engine.py.224: train: epoch 96 | batch 1100/2304 [I 2022-10-28 00:29:01] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:32 [I 2022-10-28 00:29:01] ...trap/engines/engine.py.229: process: 0.64617 | load: 0.00242 [I 2022-10-28 00:29:01] ...trap/engines/engine.py.230: loss: 0.12538 [I 2022-10-28 00:29:01] ...optimizers/trijoint.py.151: total_norm: 18.135805 [I 2022-10-28 00:29:06] ...trap/engines/engine.py.224: train: epoch 96 | batch 1110/2304 [I 2022-10-28 00:29:06] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:11:09 [I 2022-10-28 00:29:06] ...trap/engines/engine.py.229: process: 0.55804 | load: 0.00240 [I 2022-10-28 00:29:06] ...trap/engines/engine.py.230: loss: 0.11319 [I 2022-10-28 00:29:06] ...optimizers/trijoint.py.151: total_norm: 19.653063 [I 2022-10-28 00:29:12] ...trap/engines/engine.py.224: train: epoch 96 | batch 1120/2304 [I 2022-10-28 00:29:12] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:57 [I 2022-10-28 00:29:12] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00243 [I 2022-10-28 00:29:12] ...trap/engines/engine.py.230: loss: 0.10147 [I 2022-10-28 00:29:12] ...optimizers/trijoint.py.151: total_norm: 8.008997 [I 2022-10-28 00:29:18] ...trap/engines/engine.py.224: train: epoch 96 | batch 1130/2304 [I 2022-10-28 00:29:18] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:54 [I 2022-10-28 00:29:18] ...trap/engines/engine.py.229: process: 0.56298 | load: 0.00242 [I 2022-10-28 00:29:18] ...trap/engines/engine.py.230: loss: 0.12426 [I 2022-10-28 00:29:18] ...optimizers/trijoint.py.151: total_norm: 10.494255 [I 2022-10-28 00:29:23] ...trap/engines/engine.py.224: train: epoch 96 | batch 1140/2304 [I 2022-10-28 00:29:23] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:47 [I 2022-10-28 00:29:23] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00243 [I 2022-10-28 00:29:23] ...trap/engines/engine.py.230: loss: 0.12694 [I 2022-10-28 00:29:23] ...optimizers/trijoint.py.151: total_norm: 44.359013 [I 2022-10-28 00:29:29] ...trap/engines/engine.py.224: train: epoch 96 | batch 1150/2304 [I 2022-10-28 00:29:29] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:39 [I 2022-10-28 00:29:29] ...trap/engines/engine.py.229: process: 0.55756 | load: 0.00244 [I 2022-10-28 00:29:29] ...trap/engines/engine.py.230: loss: 0.15556 [I 2022-10-28 00:29:29] ...optimizers/trijoint.py.151: total_norm: 10.104488 [I 2022-10-28 00:29:34] ...trap/engines/engine.py.224: train: epoch 96 | batch 1160/2304 [I 2022-10-28 00:29:34] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:41 [I 2022-10-28 00:29:34] ...trap/engines/engine.py.229: process: 0.56347 | load: 0.00243 [I 2022-10-28 00:29:34] ...trap/engines/engine.py.230: loss: 0.18008 [I 2022-10-28 00:29:34] ...optimizers/trijoint.py.151: total_norm: 18.776690 [I 2022-10-28 00:29:40] ...trap/engines/engine.py.224: train: epoch 96 | batch 1170/2304 [I 2022-10-28 00:29:40] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:35 [I 2022-10-28 00:29:40] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00242 [I 2022-10-28 00:29:40] ...trap/engines/engine.py.230: loss: 0.12270 [I 2022-10-28 00:29:40] ...optimizers/trijoint.py.151: total_norm: 19.181261 [I 2022-10-28 00:29:46] ...trap/engines/engine.py.224: train: epoch 96 | batch 1180/2304 [I 2022-10-28 00:29:46] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:26 [I 2022-10-28 00:29:46] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00244 [I 2022-10-28 00:29:46] ...trap/engines/engine.py.230: loss: 0.13269 [I 2022-10-28 00:29:46] ...optimizers/trijoint.py.151: total_norm: 18.031961 [I 2022-10-28 00:29:51] ...trap/engines/engine.py.224: train: epoch 96 | batch 1190/2304 [I 2022-10-28 00:29:51] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:18 [I 2022-10-28 00:29:51] ...trap/engines/engine.py.229: process: 0.54670 | load: 0.00248 [I 2022-10-28 00:29:51] ...trap/engines/engine.py.230: loss: 0.09908 [I 2022-10-28 00:29:51] ...optimizers/trijoint.py.151: total_norm: 10.601342 [I 2022-10-28 00:29:57] ...trap/engines/engine.py.224: train: epoch 96 | batch 1200/2304 [I 2022-10-28 00:29:57] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:30 [I 2022-10-28 00:29:57] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00272 [I 2022-10-28 00:29:57] ...trap/engines/engine.py.230: loss: 0.21879 [I 2022-10-28 00:29:57] ...optimizers/trijoint.py.151: total_norm: 15.539938 [I 2022-10-28 00:30:02] ...trap/engines/engine.py.224: train: epoch 96 | batch 1210/2304 [I 2022-10-28 00:30:02] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:10:08 [I 2022-10-28 00:30:02] ...trap/engines/engine.py.229: process: 0.54696 | load: 0.00245 [I 2022-10-28 00:30:02] ...trap/engines/engine.py.230: loss: 0.11706 [I 2022-10-28 00:30:02] ...optimizers/trijoint.py.151: total_norm: 20.022503 [I 2022-10-28 00:30:08] ...trap/engines/engine.py.224: train: epoch 96 | batch 1220/2304 [I 2022-10-28 00:30:08] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:10:04 [I 2022-10-28 00:30:08] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00248 [I 2022-10-28 00:30:08] ...trap/engines/engine.py.230: loss: 0.09578 [I 2022-10-28 00:30:08] ...optimizers/trijoint.py.151: total_norm: 10.914456 [I 2022-10-28 00:30:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 1230/2304 [I 2022-10-28 00:30:14] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:10:01 [I 2022-10-28 00:30:14] ...trap/engines/engine.py.229: process: 0.56408 | load: 0.00246 [I 2022-10-28 00:30:14] ...trap/engines/engine.py.230: loss: 0.13156 [I 2022-10-28 00:30:14] ...optimizers/trijoint.py.151: total_norm: 20.063356 [I 2022-10-28 00:30:19] ...trap/engines/engine.py.224: train: epoch 96 | batch 1240/2304 [I 2022-10-28 00:30:19] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:53 [I 2022-10-28 00:30:19] ...trap/engines/engine.py.229: process: 0.55292 | load: 0.00242 [I 2022-10-28 00:30:19] ...trap/engines/engine.py.230: loss: 0.15314 [I 2022-10-28 00:30:19] ...optimizers/trijoint.py.151: total_norm: 8.739614 [I 2022-10-28 00:30:25] ...trap/engines/engine.py.224: train: epoch 96 | batch 1250/2304 [I 2022-10-28 00:30:25] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:50 [I 2022-10-28 00:30:25] ...trap/engines/engine.py.229: process: 0.55535 | load: 0.00242 [I 2022-10-28 00:30:25] ...trap/engines/engine.py.230: loss: 0.10321 [I 2022-10-28 00:30:25] ...optimizers/trijoint.py.151: total_norm: 12.882973 [I 2022-10-28 00:30:30] ...trap/engines/engine.py.224: train: epoch 96 | batch 1260/2304 [I 2022-10-28 00:30:30] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:43 [I 2022-10-28 00:30:30] ...trap/engines/engine.py.229: process: 0.56809 | load: 0.00245 [I 2022-10-28 00:30:30] ...trap/engines/engine.py.230: loss: 0.16819 [I 2022-10-28 00:30:30] ...optimizers/trijoint.py.151: total_norm: 21.494940 [I 2022-10-28 00:30:36] ...trap/engines/engine.py.224: train: epoch 96 | batch 1270/2304 [I 2022-10-28 00:30:36] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:40 [I 2022-10-28 00:30:36] ...trap/engines/engine.py.229: process: 0.56222 | load: 0.00245 [I 2022-10-28 00:30:36] ...trap/engines/engine.py.230: loss: 0.10762 [I 2022-10-28 00:30:36] ...optimizers/trijoint.py.151: total_norm: 17.184986 [I 2022-10-28 00:30:42] ...trap/engines/engine.py.224: train: epoch 96 | batch 1280/2304 [I 2022-10-28 00:30:42] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:26 [I 2022-10-28 00:30:42] ...trap/engines/engine.py.229: process: 0.54245 | load: 0.00243 [I 2022-10-28 00:30:42] ...trap/engines/engine.py.230: loss: 0.12425 [I 2022-10-28 00:30:42] ...optimizers/trijoint.py.151: total_norm: 7.936884 [I 2022-10-28 00:30:47] ...trap/engines/engine.py.224: train: epoch 96 | batch 1290/2304 [I 2022-10-28 00:30:47] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:28 [I 2022-10-28 00:30:47] ...trap/engines/engine.py.229: process: 0.56162 | load: 0.00241 [I 2022-10-28 00:30:47] ...trap/engines/engine.py.230: loss: 0.10221 [I 2022-10-28 00:30:47] ...optimizers/trijoint.py.151: total_norm: 8.892192 [I 2022-10-28 00:30:53] ...trap/engines/engine.py.224: train: epoch 96 | batch 1300/2304 [I 2022-10-28 00:30:53] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:21 [I 2022-10-28 00:30:53] ...trap/engines/engine.py.229: process: 0.56102 | load: 0.00240 [I 2022-10-28 00:30:53] ...trap/engines/engine.py.230: loss: 0.15340 [I 2022-10-28 00:30:53] ...optimizers/trijoint.py.151: total_norm: 17.002357 [I 2022-10-28 00:30:58] ...trap/engines/engine.py.224: train: epoch 96 | batch 1310/2304 [I 2022-10-28 00:30:58] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:09:12 [I 2022-10-28 00:30:58] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00243 [I 2022-10-28 00:30:58] ...trap/engines/engine.py.230: loss: 0.16953 [I 2022-10-28 00:30:58] ...optimizers/trijoint.py.151: total_norm: 17.196312 [I 2022-10-28 00:31:04] ...trap/engines/engine.py.224: train: epoch 96 | batch 1320/2304 [I 2022-10-28 00:31:04] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:16 [I 2022-10-28 00:31:04] ...trap/engines/engine.py.229: process: 0.55360 | load: 0.00242 [I 2022-10-28 00:31:04] ...trap/engines/engine.py.230: loss: 0.12760 [I 2022-10-28 00:31:04] ...optimizers/trijoint.py.151: total_norm: 15.704937 [I 2022-10-28 00:31:10] ...trap/engines/engine.py.224: train: epoch 96 | batch 1330/2304 [I 2022-10-28 00:31:10] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:09:05 [I 2022-10-28 00:31:10] ...trap/engines/engine.py.229: process: 0.56498 | load: 0.00243 [I 2022-10-28 00:31:10] ...trap/engines/engine.py.230: loss: 0.12007 [I 2022-10-28 00:31:10] ...optimizers/trijoint.py.151: total_norm: 10.984865 [I 2022-10-28 00:31:15] ...trap/engines/engine.py.224: train: epoch 96 | batch 1340/2304 [I 2022-10-28 00:31:15] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:58 [I 2022-10-28 00:31:15] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00242 [I 2022-10-28 00:31:15] ...trap/engines/engine.py.230: loss: 0.13855 [I 2022-10-28 00:31:15] ...optimizers/trijoint.py.151: total_norm: 29.900963 [I 2022-10-28 00:31:21] ...trap/engines/engine.py.224: train: epoch 96 | batch 1350/2304 [I 2022-10-28 00:31:21] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:50 [I 2022-10-28 00:31:21] ...trap/engines/engine.py.229: process: 0.56056 | load: 0.00241 [I 2022-10-28 00:31:21] ...trap/engines/engine.py.230: loss: 0.10115 [I 2022-10-28 00:31:21] ...optimizers/trijoint.py.151: total_norm: 10.112197 [I 2022-10-28 00:31:26] ...trap/engines/engine.py.224: train: epoch 96 | batch 1360/2304 [I 2022-10-28 00:31:26] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:47 [I 2022-10-28 00:31:26] ...trap/engines/engine.py.229: process: 0.56423 | load: 0.00243 [I 2022-10-28 00:31:26] ...trap/engines/engine.py.230: loss: 0.13294 [I 2022-10-28 00:31:26] ...optimizers/trijoint.py.151: total_norm: 26.165104 [I 2022-10-28 00:31:32] ...trap/engines/engine.py.224: train: epoch 96 | batch 1370/2304 [I 2022-10-28 00:31:32] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:41 [I 2022-10-28 00:31:32] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00240 [I 2022-10-28 00:31:32] ...trap/engines/engine.py.230: loss: 0.12860 [I 2022-10-28 00:31:32] ...optimizers/trijoint.py.151: total_norm: 27.931801 [I 2022-10-28 00:31:38] ...trap/engines/engine.py.224: train: epoch 96 | batch 1380/2304 [I 2022-10-28 00:31:38] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:50 [I 2022-10-28 00:31:38] ...trap/engines/engine.py.229: process: 0.63249 | load: 0.00243 [I 2022-10-28 00:31:38] ...trap/engines/engine.py.230: loss: 0.12690 [I 2022-10-28 00:31:38] ...optimizers/trijoint.py.151: total_norm: 31.600883 [I 2022-10-28 00:31:43] ...trap/engines/engine.py.224: train: epoch 96 | batch 1390/2304 [I 2022-10-28 00:31:43] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:26 [I 2022-10-28 00:31:43] ...trap/engines/engine.py.229: process: 0.55369 | load: 0.00242 [I 2022-10-28 00:31:43] ...trap/engines/engine.py.230: loss: 0.11462 [I 2022-10-28 00:31:43] ...optimizers/trijoint.py.151: total_norm: 24.831852 [I 2022-10-28 00:31:49] ...trap/engines/engine.py.224: train: epoch 96 | batch 1400/2304 [I 2022-10-28 00:31:49] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:21 [I 2022-10-28 00:31:49] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00243 [I 2022-10-28 00:31:49] ...trap/engines/engine.py.230: loss: 0.13312 [I 2022-10-28 00:31:49] ...optimizers/trijoint.py.151: total_norm: 19.378906 [I 2022-10-28 00:31:54] ...trap/engines/engine.py.224: train: epoch 96 | batch 1410/2304 [I 2022-10-28 00:31:54] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:19 [I 2022-10-28 00:31:54] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00241 [I 2022-10-28 00:31:54] ...trap/engines/engine.py.230: loss: 0.12880 [I 2022-10-28 00:31:54] ...optimizers/trijoint.py.151: total_norm: 11.184570 [I 2022-10-28 00:32:00] ...trap/engines/engine.py.224: train: epoch 96 | batch 1420/2304 [I 2022-10-28 00:32:00] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:17 [I 2022-10-28 00:32:00] ...trap/engines/engine.py.229: process: 0.56781 | load: 0.00242 [I 2022-10-28 00:32:00] ...trap/engines/engine.py.230: loss: 0.12975 [I 2022-10-28 00:32:00] ...optimizers/trijoint.py.151: total_norm: 13.064326 [I 2022-10-28 00:32:06] ...trap/engines/engine.py.224: train: epoch 96 | batch 1430/2304 [I 2022-10-28 00:32:06] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:08:07 [I 2022-10-28 00:32:06] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00249 [I 2022-10-28 00:32:06] ...trap/engines/engine.py.230: loss: 0.10772 [I 2022-10-28 00:32:06] ...optimizers/trijoint.py.151: total_norm: 21.256159 [I 2022-10-28 00:32:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 1440/2304 [I 2022-10-28 00:32:11] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:08:04 [I 2022-10-28 00:32:11] ...trap/engines/engine.py.229: process: 0.56039 | load: 0.00247 [I 2022-10-28 00:32:11] ...trap/engines/engine.py.230: loss: 0.20605 [I 2022-10-28 00:32:11] ...optimizers/trijoint.py.151: total_norm: 47.356182 [I 2022-10-28 00:32:17] ...trap/engines/engine.py.224: train: epoch 96 | batch 1450/2304 [I 2022-10-28 00:32:17] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:08:00 [I 2022-10-28 00:32:17] ...trap/engines/engine.py.229: process: 0.57191 | load: 0.00246 [I 2022-10-28 00:32:17] ...trap/engines/engine.py.230: loss: 0.18103 [I 2022-10-28 00:32:17] ...optimizers/trijoint.py.151: total_norm: 13.587104 [I 2022-10-28 00:32:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 1460/2304 [I 2022-10-28 00:32:22] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:52 [I 2022-10-28 00:32:22] ...trap/engines/engine.py.229: process: 0.56544 | load: 0.00246 [I 2022-10-28 00:32:22] ...trap/engines/engine.py.230: loss: 0.11756 [I 2022-10-28 00:32:22] ...optimizers/trijoint.py.151: total_norm: 12.418227 [I 2022-10-28 00:32:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 1470/2304 [I 2022-10-28 00:32:28] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:43 [I 2022-10-28 00:32:28] ...trap/engines/engine.py.229: process: 0.55292 | load: 0.00243 [I 2022-10-28 00:32:28] ...trap/engines/engine.py.230: loss: 0.10585 [I 2022-10-28 00:32:28] ...optimizers/trijoint.py.151: total_norm: 14.852986 [I 2022-10-28 00:32:34] ...trap/engines/engine.py.224: train: epoch 96 | batch 1480/2304 [I 2022-10-28 00:32:34] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:37 [I 2022-10-28 00:32:34] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00246 [I 2022-10-28 00:32:34] ...trap/engines/engine.py.230: loss: 0.16994 [I 2022-10-28 00:32:34] ...optimizers/trijoint.py.151: total_norm: 9.633251 [I 2022-10-28 00:32:39] ...trap/engines/engine.py.224: train: epoch 96 | batch 1490/2304 [I 2022-10-28 00:32:39] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:36 [I 2022-10-28 00:32:39] ...trap/engines/engine.py.229: process: 0.56686 | load: 0.00244 [I 2022-10-28 00:32:39] ...trap/engines/engine.py.230: loss: 0.12506 [I 2022-10-28 00:32:39] ...optimizers/trijoint.py.151: total_norm: 17.875933 [I 2022-10-28 00:32:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 1500/2304 [I 2022-10-28 00:32:45] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:28 [I 2022-10-28 00:32:45] ...trap/engines/engine.py.229: process: 0.55477 | load: 0.00244 [I 2022-10-28 00:32:45] ...trap/engines/engine.py.230: loss: 0.11275 [I 2022-10-28 00:32:45] ...optimizers/trijoint.py.151: total_norm: 10.618908 [I 2022-10-28 00:32:50] ...trap/engines/engine.py.224: train: epoch 96 | batch 1510/2304 [I 2022-10-28 00:32:50] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:25 [I 2022-10-28 00:32:50] ...trap/engines/engine.py.229: process: 0.56675 | load: 0.00244 [I 2022-10-28 00:32:50] ...trap/engines/engine.py.230: loss: 0.12540 [I 2022-10-28 00:32:50] ...optimizers/trijoint.py.151: total_norm: 23.448557 [I 2022-10-28 00:32:56] ...trap/engines/engine.py.224: train: epoch 96 | batch 1520/2304 [I 2022-10-28 00:32:56] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:16 [I 2022-10-28 00:32:56] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00245 [I 2022-10-28 00:32:56] ...trap/engines/engine.py.230: loss: 0.09885 [I 2022-10-28 00:32:56] ...optimizers/trijoint.py.151: total_norm: 14.720706 [I 2022-10-28 00:33:02] ...trap/engines/engine.py.224: train: epoch 96 | batch 1530/2304 [I 2022-10-28 00:33:02] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:07:12 [I 2022-10-28 00:33:02] ...trap/engines/engine.py.229: process: 0.55023 | load: 0.00246 [I 2022-10-28 00:33:02] ...trap/engines/engine.py.230: loss: 0.08862 [I 2022-10-28 00:33:02] ...optimizers/trijoint.py.151: total_norm: 14.744682 [I 2022-10-28 00:33:07] ...trap/engines/engine.py.224: train: epoch 96 | batch 1540/2304 [I 2022-10-28 00:33:07] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:07:07 [I 2022-10-28 00:33:07] ...trap/engines/engine.py.229: process: 0.57296 | load: 0.00245 [I 2022-10-28 00:33:07] ...trap/engines/engine.py.230: loss: 0.09924 [I 2022-10-28 00:33:07] ...optimizers/trijoint.py.151: total_norm: 8.463656 [I 2022-10-28 00:33:13] ...trap/engines/engine.py.224: train: epoch 96 | batch 1550/2304 [I 2022-10-28 00:33:13] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:07:00 [I 2022-10-28 00:33:13] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00241 [I 2022-10-28 00:33:13] ...trap/engines/engine.py.230: loss: 0.09585 [I 2022-10-28 00:33:13] ...optimizers/trijoint.py.151: total_norm: 14.508399 [I 2022-10-28 00:33:18] ...trap/engines/engine.py.224: train: epoch 96 | batch 1560/2304 [I 2022-10-28 00:33:18] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:55 [I 2022-10-28 00:33:18] ...trap/engines/engine.py.229: process: 0.55097 | load: 0.00261 [I 2022-10-28 00:33:18] ...trap/engines/engine.py.230: loss: 0.09375 [I 2022-10-28 00:33:18] ...optimizers/trijoint.py.151: total_norm: 10.461202 [I 2022-10-28 00:33:24] ...trap/engines/engine.py.224: train: epoch 96 | batch 1570/2304 [I 2022-10-28 00:33:24] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:48 [I 2022-10-28 00:33:24] ...trap/engines/engine.py.229: process: 0.55510 | load: 0.00245 [I 2022-10-28 00:33:24] ...trap/engines/engine.py.230: loss: 0.14452 [I 2022-10-28 00:33:24] ...optimizers/trijoint.py.151: total_norm: 11.770408 [I 2022-10-28 00:33:30] ...trap/engines/engine.py.224: train: epoch 96 | batch 1580/2304 [I 2022-10-28 00:33:30] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:42 [I 2022-10-28 00:33:30] ...trap/engines/engine.py.229: process: 0.54828 | load: 0.00242 [I 2022-10-28 00:33:30] ...trap/engines/engine.py.230: loss: 0.15450 [I 2022-10-28 00:33:30] ...optimizers/trijoint.py.151: total_norm: 14.959574 [I 2022-10-28 00:33:35] ...trap/engines/engine.py.224: train: epoch 96 | batch 1590/2304 [I 2022-10-28 00:33:35] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:39 [I 2022-10-28 00:33:35] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00243 [I 2022-10-28 00:33:35] ...trap/engines/engine.py.230: loss: 0.12409 [I 2022-10-28 00:33:35] ...optimizers/trijoint.py.151: total_norm: 20.445429 [I 2022-10-28 00:33:41] ...trap/engines/engine.py.224: train: epoch 96 | batch 1600/2304 [I 2022-10-28 00:33:41] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:31 [I 2022-10-28 00:33:41] ...trap/engines/engine.py.229: process: 0.55562 | load: 0.00253 [I 2022-10-28 00:33:41] ...trap/engines/engine.py.230: loss: 0.14057 [I 2022-10-28 00:33:41] ...optimizers/trijoint.py.151: total_norm: 12.520620 [I 2022-10-28 00:33:46] ...trap/engines/engine.py.224: train: epoch 96 | batch 1610/2304 [I 2022-10-28 00:33:46] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:28 [I 2022-10-28 00:33:46] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00248 [I 2022-10-28 00:33:46] ...trap/engines/engine.py.230: loss: 0.09088 [I 2022-10-28 00:33:46] ...optimizers/trijoint.py.151: total_norm: 12.084101 [I 2022-10-28 00:33:52] ...trap/engines/engine.py.224: train: epoch 96 | batch 1620/2304 [I 2022-10-28 00:33:52] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:21 [I 2022-10-28 00:33:52] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00245 [I 2022-10-28 00:33:52] ...trap/engines/engine.py.230: loss: 0.17665 [I 2022-10-28 00:33:52] ...optimizers/trijoint.py.151: total_norm: 13.303885 [I 2022-10-28 00:33:57] ...trap/engines/engine.py.224: train: epoch 96 | batch 1630/2304 [I 2022-10-28 00:33:57] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:14 [I 2022-10-28 00:33:57] ...trap/engines/engine.py.229: process: 0.54861 | load: 0.00243 [I 2022-10-28 00:33:57] ...trap/engines/engine.py.230: loss: 0.09562 [I 2022-10-28 00:33:57] ...optimizers/trijoint.py.151: total_norm: 9.751575 [I 2022-10-28 00:34:03] ...trap/engines/engine.py.224: train: epoch 96 | batch 1640/2304 [I 2022-10-28 00:34:03] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:06:10 [I 2022-10-28 00:34:03] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00243 [I 2022-10-28 00:34:03] ...trap/engines/engine.py.230: loss: 0.17053 [I 2022-10-28 00:34:03] ...optimizers/trijoint.py.151: total_norm: 16.352800 [I 2022-10-28 00:34:09] ...trap/engines/engine.py.224: train: epoch 96 | batch 1650/2304 [I 2022-10-28 00:34:09] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:06:04 [I 2022-10-28 00:34:09] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00245 [I 2022-10-28 00:34:09] ...trap/engines/engine.py.230: loss: 0.11902 [I 2022-10-28 00:34:09] ...optimizers/trijoint.py.151: total_norm: 17.523769 [I 2022-10-28 00:34:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 1660/2304 [I 2022-10-28 00:34:14] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:05:59 [I 2022-10-28 00:34:14] ...trap/engines/engine.py.229: process: 0.55231 | load: 0.00244 [I 2022-10-28 00:34:14] ...trap/engines/engine.py.230: loss: 0.12062 [I 2022-10-28 00:34:14] ...optimizers/trijoint.py.151: total_norm: 16.211205 [I 2022-10-28 00:34:20] ...trap/engines/engine.py.224: train: epoch 96 | batch 1670/2304 [I 2022-10-28 00:34:20] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:53 [I 2022-10-28 00:34:20] ...trap/engines/engine.py.229: process: 0.56683 | load: 0.00243 [I 2022-10-28 00:34:20] ...trap/engines/engine.py.230: loss: 0.09340 [I 2022-10-28 00:34:20] ...optimizers/trijoint.py.151: total_norm: 14.409401 [I 2022-10-28 00:34:25] ...trap/engines/engine.py.224: train: epoch 96 | batch 1680/2304 [I 2022-10-28 00:34:25] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:48 [I 2022-10-28 00:34:25] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00241 [I 2022-10-28 00:34:25] ...trap/engines/engine.py.230: loss: 0.13147 [I 2022-10-28 00:34:25] ...optimizers/trijoint.py.151: total_norm: 14.916530 [I 2022-10-28 00:34:31] ...trap/engines/engine.py.224: train: epoch 96 | batch 1690/2304 [I 2022-10-28 00:34:31] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:41 [I 2022-10-28 00:34:31] ...trap/engines/engine.py.229: process: 0.54498 | load: 0.00244 [I 2022-10-28 00:34:31] ...trap/engines/engine.py.230: loss: 0.12309 [I 2022-10-28 00:34:31] ...optimizers/trijoint.py.151: total_norm: 36.312279 [I 2022-10-28 00:34:37] ...trap/engines/engine.py.224: train: epoch 96 | batch 1700/2304 [I 2022-10-28 00:34:37] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:37 [I 2022-10-28 00:34:37] ...trap/engines/engine.py.229: process: 0.54619 | load: 0.00239 [I 2022-10-28 00:34:37] ...trap/engines/engine.py.230: loss: 0.14981 [I 2022-10-28 00:34:37] ...optimizers/trijoint.py.151: total_norm: 32.183895 [I 2022-10-28 00:34:42] ...trap/engines/engine.py.224: train: epoch 96 | batch 1710/2304 [I 2022-10-28 00:34:42] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:29 [I 2022-10-28 00:34:42] ...trap/engines/engine.py.229: process: 0.54970 | load: 0.00246 [I 2022-10-28 00:34:42] ...trap/engines/engine.py.230: loss: 0.11005 [I 2022-10-28 00:34:42] ...optimizers/trijoint.py.151: total_norm: 13.408346 [I 2022-10-28 00:34:48] ...trap/engines/engine.py.224: train: epoch 96 | batch 1720/2304 [I 2022-10-28 00:34:48] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:25 [I 2022-10-28 00:34:48] ...trap/engines/engine.py.229: process: 0.54545 | load: 0.00242 [I 2022-10-28 00:34:48] ...trap/engines/engine.py.230: loss: 0.11271 [I 2022-10-28 00:34:48] ...optimizers/trijoint.py.151: total_norm: 11.027804 [I 2022-10-28 00:34:53] ...trap/engines/engine.py.224: train: epoch 96 | batch 1730/2304 [I 2022-10-28 00:34:53] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:19 [I 2022-10-28 00:34:53] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00242 [I 2022-10-28 00:34:53] ...trap/engines/engine.py.230: loss: 0.09252 [I 2022-10-28 00:34:53] ...optimizers/trijoint.py.151: total_norm: 9.764642 [I 2022-10-28 00:34:59] ...trap/engines/engine.py.224: train: epoch 96 | batch 1740/2304 [I 2022-10-28 00:34:59] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:05:15 [I 2022-10-28 00:34:59] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00241 [I 2022-10-28 00:34:59] ...trap/engines/engine.py.230: loss: 0.12238 [I 2022-10-28 00:34:59] ...optimizers/trijoint.py.151: total_norm: 16.275494 [I 2022-10-28 00:35:05] ...trap/engines/engine.py.224: train: epoch 96 | batch 1750/2304 [I 2022-10-28 00:35:05] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:10 [I 2022-10-28 00:35:05] ...trap/engines/engine.py.229: process: 0.57699 | load: 0.00278 [I 2022-10-28 00:35:05] ...trap/engines/engine.py.230: loss: 0.13281 [I 2022-10-28 00:35:05] ...optimizers/trijoint.py.151: total_norm: 17.084597 [I 2022-10-28 00:35:10] ...trap/engines/engine.py.224: train: epoch 96 | batch 1760/2304 [I 2022-10-28 00:35:10] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:05:04 [I 2022-10-28 00:35:10] ...trap/engines/engine.py.229: process: 0.55831 | load: 0.00239 [I 2022-10-28 00:35:10] ...trap/engines/engine.py.230: loss: 0.13939 [I 2022-10-28 00:35:10] ...optimizers/trijoint.py.151: total_norm: 17.251877 [I 2022-10-28 00:35:16] ...trap/engines/engine.py.224: train: epoch 96 | batch 1770/2304 [I 2022-10-28 00:35:16] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:57 [I 2022-10-28 00:35:16] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00241 [I 2022-10-28 00:35:16] ...trap/engines/engine.py.230: loss: 0.13844 [I 2022-10-28 00:35:16] ...optimizers/trijoint.py.151: total_norm: 54.692406 [I 2022-10-28 00:35:21] ...trap/engines/engine.py.224: train: epoch 96 | batch 1780/2304 [I 2022-10-28 00:35:21] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:53 [I 2022-10-28 00:35:21] ...trap/engines/engine.py.229: process: 0.56265 | load: 0.00243 [I 2022-10-28 00:35:21] ...trap/engines/engine.py.230: loss: 0.10947 [I 2022-10-28 00:35:21] ...optimizers/trijoint.py.151: total_norm: 8.773981 [I 2022-10-28 00:35:27] ...trap/engines/engine.py.224: train: epoch 96 | batch 1790/2304 [I 2022-10-28 00:35:27] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:04:47 [I 2022-10-28 00:35:27] ...trap/engines/engine.py.229: process: 0.56073 | load: 0.00242 [I 2022-10-28 00:35:27] ...trap/engines/engine.py.230: loss: 0.09696 [I 2022-10-28 00:35:27] ...optimizers/trijoint.py.151: total_norm: 9.990943 [I 2022-10-28 00:35:33] ...trap/engines/engine.py.224: train: epoch 96 | batch 1800/2304 [I 2022-10-28 00:35:33] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:41 [I 2022-10-28 00:35:33] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00244 [I 2022-10-28 00:35:33] ...trap/engines/engine.py.230: loss: 0.18084 [I 2022-10-28 00:35:33] ...optimizers/trijoint.py.151: total_norm: 16.056110 [I 2022-10-28 00:35:38] ...trap/engines/engine.py.224: train: epoch 96 | batch 1810/2304 [I 2022-10-28 00:35:38] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:34 [I 2022-10-28 00:35:38] ...trap/engines/engine.py.229: process: 0.56284 | load: 0.00244 [I 2022-10-28 00:35:38] ...trap/engines/engine.py.230: loss: 0.12612 [I 2022-10-28 00:35:38] ...optimizers/trijoint.py.151: total_norm: 22.145462 [I 2022-10-28 00:35:44] ...trap/engines/engine.py.224: train: epoch 96 | batch 1820/2304 [I 2022-10-28 00:35:44] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:30 [I 2022-10-28 00:35:44] ...trap/engines/engine.py.229: process: 0.56397 | load: 0.00241 [I 2022-10-28 00:35:44] ...trap/engines/engine.py.230: loss: 0.11303 [I 2022-10-28 00:35:44] ...optimizers/trijoint.py.151: total_norm: 16.670731 [I 2022-10-28 00:35:49] ...trap/engines/engine.py.224: train: epoch 96 | batch 1830/2304 [I 2022-10-28 00:35:49] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:26 [I 2022-10-28 00:35:49] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00240 [I 2022-10-28 00:35:49] ...trap/engines/engine.py.230: loss: 0.08575 [I 2022-10-28 00:35:49] ...optimizers/trijoint.py.151: total_norm: 12.663703 [I 2022-10-28 00:35:55] ...trap/engines/engine.py.224: train: epoch 96 | batch 1840/2304 [I 2022-10-28 00:35:55] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:04:19 [I 2022-10-28 00:35:55] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00241 [I 2022-10-28 00:35:55] ...trap/engines/engine.py.230: loss: 0.08080 [I 2022-10-28 00:35:55] ...optimizers/trijoint.py.151: total_norm: 12.917801 [I 2022-10-28 00:36:01] ...trap/engines/engine.py.224: train: epoch 96 | batch 1850/2304 [I 2022-10-28 00:36:01] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:04:14 [I 2022-10-28 00:36:01] ...trap/engines/engine.py.229: process: 0.56581 | load: 0.00242 [I 2022-10-28 00:36:01] ...trap/engines/engine.py.230: loss: 0.09451 [I 2022-10-28 00:36:01] ...optimizers/trijoint.py.151: total_norm: 12.427709 [I 2022-10-28 00:36:06] ...trap/engines/engine.py.224: train: epoch 96 | batch 1860/2304 [I 2022-10-28 00:36:06] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:04:08 [I 2022-10-28 00:36:06] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00243 [I 2022-10-28 00:36:06] ...trap/engines/engine.py.230: loss: 0.10888 [I 2022-10-28 00:36:06] ...optimizers/trijoint.py.151: total_norm: 15.406989 [I 2022-10-28 00:36:12] ...trap/engines/engine.py.224: train: epoch 96 | batch 1870/2304 [I 2022-10-28 00:36:12] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:04:02 [I 2022-10-28 00:36:12] ...trap/engines/engine.py.229: process: 0.56871 | load: 0.00243 [I 2022-10-28 00:36:12] ...trap/engines/engine.py.230: loss: 0.11239 [I 2022-10-28 00:36:12] ...optimizers/trijoint.py.151: total_norm: 8.955323 [I 2022-10-28 00:36:17] ...trap/engines/engine.py.224: train: epoch 96 | batch 1880/2304 [I 2022-10-28 00:36:17] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:56 [I 2022-10-28 00:36:17] ...trap/engines/engine.py.229: process: 0.55219 | load: 0.00240 [I 2022-10-28 00:36:17] ...trap/engines/engine.py.230: loss: 0.11135 [I 2022-10-28 00:36:17] ...optimizers/trijoint.py.151: total_norm: 10.766103 [I 2022-10-28 00:36:23] ...trap/engines/engine.py.224: train: epoch 96 | batch 1890/2304 [I 2022-10-28 00:36:23] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:51 [I 2022-10-28 00:36:23] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00244 [I 2022-10-28 00:36:23] ...trap/engines/engine.py.230: loss: 0.13020 [I 2022-10-28 00:36:23] ...optimizers/trijoint.py.151: total_norm: 22.624960 [I 2022-10-28 00:36:29] ...trap/engines/engine.py.224: train: epoch 96 | batch 1900/2304 [I 2022-10-28 00:36:29] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:49 [I 2022-10-28 00:36:29] ...trap/engines/engine.py.229: process: 0.54521 | load: 0.00242 [I 2022-10-28 00:36:29] ...trap/engines/engine.py.230: loss: 0.15547 [I 2022-10-28 00:36:29] ...optimizers/trijoint.py.151: total_norm: 22.820454 [I 2022-10-28 00:36:34] ...trap/engines/engine.py.224: train: epoch 96 | batch 1910/2304 [I 2022-10-28 00:36:34] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:03:41 [I 2022-10-28 00:36:34] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00240 [I 2022-10-28 00:36:34] ...trap/engines/engine.py.230: loss: 0.07934 [I 2022-10-28 00:36:34] ...optimizers/trijoint.py.151: total_norm: 14.606050 [I 2022-10-28 00:36:40] ...trap/engines/engine.py.224: train: epoch 96 | batch 1920/2304 [I 2022-10-28 00:36:40] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:35 [I 2022-10-28 00:36:40] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00243 [I 2022-10-28 00:36:40] ...trap/engines/engine.py.230: loss: 0.10054 [I 2022-10-28 00:36:40] ...optimizers/trijoint.py.151: total_norm: 18.892899 [I 2022-10-28 00:36:46] ...trap/engines/engine.py.224: train: epoch 96 | batch 1930/2304 [I 2022-10-28 00:36:46] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:30 [I 2022-10-28 00:36:46] ...trap/engines/engine.py.229: process: 0.56899 | load: 0.00246 [I 2022-10-28 00:36:46] ...trap/engines/engine.py.230: loss: 0.12462 [I 2022-10-28 00:36:46] ...optimizers/trijoint.py.151: total_norm: 15.726661 [I 2022-10-28 00:36:51] ...trap/engines/engine.py.224: train: epoch 96 | batch 1940/2304 [I 2022-10-28 00:36:51] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:23 [I 2022-10-28 00:36:51] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00241 [I 2022-10-28 00:36:51] ...trap/engines/engine.py.230: loss: 0.09046 [I 2022-10-28 00:36:51] ...optimizers/trijoint.py.151: total_norm: 12.117023 [I 2022-10-28 00:36:57] ...trap/engines/engine.py.224: train: epoch 96 | batch 1950/2304 [I 2022-10-28 00:36:57] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:18 [I 2022-10-28 00:36:57] ...trap/engines/engine.py.229: process: 0.56361 | load: 0.00259 [I 2022-10-28 00:36:57] ...trap/engines/engine.py.230: loss: 0.12226 [I 2022-10-28 00:36:57] ...optimizers/trijoint.py.151: total_norm: 26.079725 [I 2022-10-28 00:37:02] ...trap/engines/engine.py.224: train: epoch 96 | batch 1960/2304 [I 2022-10-28 00:37:02] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:03:12 [I 2022-10-28 00:37:02] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00242 [I 2022-10-28 00:37:02] ...trap/engines/engine.py.230: loss: 0.13432 [I 2022-10-28 00:37:02] ...optimizers/trijoint.py.151: total_norm: 17.400629 [I 2022-10-28 00:37:08] ...trap/engines/engine.py.224: train: epoch 96 | batch 1970/2304 [I 2022-10-28 00:37:08] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:03:06 [I 2022-10-28 00:37:08] ...trap/engines/engine.py.229: process: 0.56331 | load: 0.00243 [I 2022-10-28 00:37:08] ...trap/engines/engine.py.230: loss: 0.07081 [I 2022-10-28 00:37:08] ...optimizers/trijoint.py.151: total_norm: 12.892526 [I 2022-10-28 00:37:14] ...trap/engines/engine.py.224: train: epoch 96 | batch 1980/2304 [I 2022-10-28 00:37:14] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:03:01 [I 2022-10-28 00:37:14] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00243 [I 2022-10-28 00:37:14] ...trap/engines/engine.py.230: loss: 0.10799 [I 2022-10-28 00:37:14] ...optimizers/trijoint.py.151: total_norm: 17.343090 [I 2022-10-28 00:37:19] ...trap/engines/engine.py.224: train: epoch 96 | batch 1990/2304 [I 2022-10-28 00:37:19] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:56 [I 2022-10-28 00:37:19] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00243 [I 2022-10-28 00:37:19] ...trap/engines/engine.py.230: loss: 0.12278 [I 2022-10-28 00:37:19] ...optimizers/trijoint.py.151: total_norm: 16.023260 [I 2022-10-28 00:37:42] ...trap/engines/engine.py.224: train: epoch 96 | batch 2000/2304 [I 2022-10-28 00:37:42] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:08:24 [I 2022-10-28 00:37:42] ...trap/engines/engine.py.229: process: 0.57290 | load: 0.00236 [I 2022-10-28 00:37:42] ...trap/engines/engine.py.230: loss: 0.10304 [I 2022-10-28 00:37:42] ...optimizers/trijoint.py.151: total_norm: 15.169199 [I 2022-10-28 00:37:47] ...trap/engines/engine.py.224: train: epoch 96 | batch 2010/2304 [I 2022-10-28 00:37:47] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:03:19 [I 2022-10-28 00:37:47] ...trap/engines/engine.py.229: process: 0.56598 | load: 0.00236 [I 2022-10-28 00:37:47] ...trap/engines/engine.py.230: loss: 0.22586 [I 2022-10-28 00:37:47] ...optimizers/trijoint.py.151: total_norm: 22.486387 [I 2022-10-28 00:37:53] ...trap/engines/engine.py.224: train: epoch 96 | batch 2020/2304 [I 2022-10-28 00:37:53] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:42 [I 2022-10-28 00:37:53] ...trap/engines/engine.py.229: process: 0.56201 | load: 0.00238 [I 2022-10-28 00:37:53] ...trap/engines/engine.py.230: loss: 0.09847 [I 2022-10-28 00:37:53] ...optimizers/trijoint.py.151: total_norm: 8.901917 [I 2022-10-28 00:37:59] ...trap/engines/engine.py.224: train: epoch 96 | batch 2030/2304 [I 2022-10-28 00:37:59] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:02:33 [I 2022-10-28 00:37:59] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00240 [I 2022-10-28 00:37:59] ...trap/engines/engine.py.230: loss: 0.12296 [I 2022-10-28 00:37:59] ...optimizers/trijoint.py.151: total_norm: 15.443110 [I 2022-10-28 00:38:04] ...trap/engines/engine.py.224: train: epoch 96 | batch 2040/2304 [I 2022-10-28 00:38:04] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:02:27 [I 2022-10-28 00:38:04] ...trap/engines/engine.py.229: process: 0.55863 | load: 0.00237 [I 2022-10-28 00:38:04] ...trap/engines/engine.py.230: loss: 0.13873 [I 2022-10-28 00:38:04] ...optimizers/trijoint.py.151: total_norm: 14.106966 [I 2022-10-28 00:38:10] ...trap/engines/engine.py.224: train: epoch 96 | batch 2050/2304 [I 2022-10-28 00:38:10] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:02:21 [I 2022-10-28 00:38:10] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00237 [I 2022-10-28 00:38:10] ...trap/engines/engine.py.230: loss: 0.10523 [I 2022-10-28 00:38:10] ...optimizers/trijoint.py.151: total_norm: 13.264278 [I 2022-10-28 00:38:15] ...trap/engines/engine.py.224: train: epoch 96 | batch 2060/2304 [I 2022-10-28 00:38:15] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:02:15 [I 2022-10-28 00:38:15] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00247 [I 2022-10-28 00:38:15] ...trap/engines/engine.py.230: loss: 0.13790 [I 2022-10-28 00:38:15] ...optimizers/trijoint.py.151: total_norm: 20.554869 [I 2022-10-28 00:38:21] ...trap/engines/engine.py.224: train: epoch 96 | batch 2070/2304 [I 2022-10-28 00:38:21] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:02:09 [I 2022-10-28 00:38:21] ...trap/engines/engine.py.229: process: 0.54038 | load: 0.00243 [I 2022-10-28 00:38:21] ...trap/engines/engine.py.230: loss: 0.10669 [I 2022-10-28 00:38:21] ...optimizers/trijoint.py.151: total_norm: 13.828399 [I 2022-10-28 00:38:26] ...trap/engines/engine.py.224: train: epoch 96 | batch 2080/2304 [I 2022-10-28 00:38:26] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:02:04 [I 2022-10-28 00:38:26] ...trap/engines/engine.py.229: process: 0.55216 | load: 0.00242 [I 2022-10-28 00:38:26] ...trap/engines/engine.py.230: loss: 0.13076 [I 2022-10-28 00:38:26] ...optimizers/trijoint.py.151: total_norm: 11.983979 [I 2022-10-28 00:38:32] ...trap/engines/engine.py.224: train: epoch 96 | batch 2090/2304 [I 2022-10-28 00:38:32] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:57 [I 2022-10-28 00:38:32] ...trap/engines/engine.py.229: process: 0.53768 | load: 0.00240 [I 2022-10-28 00:38:32] ...trap/engines/engine.py.230: loss: 0.12522 [I 2022-10-28 00:38:32] ...optimizers/trijoint.py.151: total_norm: 16.663116 [I 2022-10-28 00:38:38] ...trap/engines/engine.py.224: train: epoch 96 | batch 2100/2304 [I 2022-10-28 00:38:38] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:52 [I 2022-10-28 00:38:38] ...trap/engines/engine.py.229: process: 0.54817 | load: 0.00237 [I 2022-10-28 00:38:38] ...trap/engines/engine.py.230: loss: 0.12504 [I 2022-10-28 00:38:38] ...optimizers/trijoint.py.151: total_norm: 15.313611 [I 2022-10-28 00:38:43] ...trap/engines/engine.py.224: train: epoch 96 | batch 2110/2304 [I 2022-10-28 00:38:43] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:48 [I 2022-10-28 00:38:43] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00238 [I 2022-10-28 00:38:43] ...trap/engines/engine.py.230: loss: 0.12963 [I 2022-10-28 00:38:43] ...optimizers/trijoint.py.151: total_norm: 8.608148 [I 2022-10-28 00:38:49] ...trap/engines/engine.py.224: train: epoch 96 | batch 2120/2304 [I 2022-10-28 00:38:49] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:41 [I 2022-10-28 00:38:49] ...trap/engines/engine.py.229: process: 0.54551 | load: 0.00260 [I 2022-10-28 00:38:49] ...trap/engines/engine.py.230: loss: 0.12616 [I 2022-10-28 00:38:49] ...optimizers/trijoint.py.151: total_norm: 10.205477 [I 2022-10-28 00:38:54] ...trap/engines/engine.py.224: train: epoch 96 | batch 2130/2304 [I 2022-10-28 00:38:54] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:01:37 [I 2022-10-28 00:38:54] ...trap/engines/engine.py.229: process: 0.56879 | load: 0.00238 [I 2022-10-28 00:38:54] ...trap/engines/engine.py.230: loss: 0.11645 [I 2022-10-28 00:38:54] ...optimizers/trijoint.py.151: total_norm: 14.515086 [I 2022-10-28 00:39:00] ...trap/engines/engine.py.224: train: epoch 96 | batch 2140/2304 [I 2022-10-28 00:39:00] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:35 [I 2022-10-28 00:39:00] ...trap/engines/engine.py.229: process: 0.67906 | load: 0.00247 [I 2022-10-28 00:39:00] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-28 00:39:00] ...optimizers/trijoint.py.151: total_norm: 22.624207 [I 2022-10-28 00:39:06] ...trap/engines/engine.py.224: train: epoch 96 | batch 2150/2304 [I 2022-10-28 00:39:06] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:27 [I 2022-10-28 00:39:06] ...trap/engines/engine.py.229: process: 0.56382 | load: 0.00251 [I 2022-10-28 00:39:06] ...trap/engines/engine.py.230: loss: 0.10611 [I 2022-10-28 00:39:06] ...optimizers/trijoint.py.151: total_norm: 28.388174 [I 2022-10-28 00:39:11] ...trap/engines/engine.py.224: train: epoch 96 | batch 2160/2304 [I 2022-10-28 00:39:11] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:01:20 [I 2022-10-28 00:39:11] ...trap/engines/engine.py.229: process: 0.56799 | load: 0.00246 [I 2022-10-28 00:39:11] ...trap/engines/engine.py.230: loss: 0.09711 [I 2022-10-28 00:39:11] ...optimizers/trijoint.py.151: total_norm: 10.161742 [I 2022-10-28 00:39:17] ...trap/engines/engine.py.224: train: epoch 96 | batch 2170/2304 [I 2022-10-28 00:39:17] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:01:14 [I 2022-10-28 00:39:17] ...trap/engines/engine.py.229: process: 0.56036 | load: 0.00248 [I 2022-10-28 00:39:17] ...trap/engines/engine.py.230: loss: 0.12166 [I 2022-10-28 00:39:17] ...optimizers/trijoint.py.151: total_norm: 19.720346 [I 2022-10-28 00:39:22] ...trap/engines/engine.py.224: train: epoch 96 | batch 2180/2304 [I 2022-10-28 00:39:22] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:01:08 [I 2022-10-28 00:39:22] ...trap/engines/engine.py.229: process: 0.55482 | load: 0.00253 [I 2022-10-28 00:39:22] ...trap/engines/engine.py.230: loss: 0.10858 [I 2022-10-28 00:39:22] ...optimizers/trijoint.py.151: total_norm: 13.565182 [I 2022-10-28 00:39:28] ...trap/engines/engine.py.224: train: epoch 96 | batch 2190/2304 [I 2022-10-28 00:39:28] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:01:03 [I 2022-10-28 00:39:28] ...trap/engines/engine.py.229: process: 0.56185 | load: 0.00246 [I 2022-10-28 00:39:28] ...trap/engines/engine.py.230: loss: 0.09620 [I 2022-10-28 00:39:28] ...optimizers/trijoint.py.151: total_norm: 10.827289 [I 2022-10-28 00:39:34] ...trap/engines/engine.py.224: train: epoch 96 | batch 2200/2304 [I 2022-10-28 00:39:34] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:58 [I 2022-10-28 00:39:34] ...trap/engines/engine.py.229: process: 0.56874 | load: 0.00253 [I 2022-10-28 00:39:34] ...trap/engines/engine.py.230: loss: 0.14312 [I 2022-10-28 00:39:34] ...optimizers/trijoint.py.151: total_norm: 16.859102 [I 2022-10-28 00:39:39] ...trap/engines/engine.py.224: train: epoch 96 | batch 2210/2304 [I 2022-10-28 00:39:39] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:52 [I 2022-10-28 00:39:39] ...trap/engines/engine.py.229: process: 0.56861 | load: 0.00244 [I 2022-10-28 00:39:39] ...trap/engines/engine.py.230: loss: 0.10033 [I 2022-10-28 00:39:39] ...optimizers/trijoint.py.151: total_norm: 11.007633 [I 2022-10-28 00:39:45] ...trap/engines/engine.py.224: train: epoch 96 | batch 2220/2304 [I 2022-10-28 00:39:45] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:46 [I 2022-10-28 00:39:45] ...trap/engines/engine.py.229: process: 0.55055 | load: 0.00246 [I 2022-10-28 00:39:45] ...trap/engines/engine.py.230: loss: 0.11721 [I 2022-10-28 00:39:45] ...optimizers/trijoint.py.151: total_norm: 19.417263 [I 2022-10-28 00:39:50] ...trap/engines/engine.py.224: train: epoch 96 | batch 2230/2304 [I 2022-10-28 00:39:50] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:40 [I 2022-10-28 00:39:50] ...trap/engines/engine.py.229: process: 0.54918 | load: 0.00263 [I 2022-10-28 00:39:50] ...trap/engines/engine.py.230: loss: 0.11154 [I 2022-10-28 00:39:50] ...optimizers/trijoint.py.151: total_norm: 14.150303 [I 2022-10-28 00:39:56] ...trap/engines/engine.py.224: train: epoch 96 | batch 2240/2304 [I 2022-10-28 00:39:56] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:35 [I 2022-10-28 00:39:56] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00273 [I 2022-10-28 00:39:56] ...trap/engines/engine.py.230: loss: 0.19138 [I 2022-10-28 00:39:56] ...optimizers/trijoint.py.151: total_norm: 31.400684 [I 2022-10-28 00:40:02] ...trap/engines/engine.py.224: train: epoch 96 | batch 2250/2304 [I 2022-10-28 00:40:02] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:30 [I 2022-10-28 00:40:02] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00252 [I 2022-10-28 00:40:02] ...trap/engines/engine.py.230: loss: 0.14922 [I 2022-10-28 00:40:02] ...optimizers/trijoint.py.151: total_norm: 29.426128 [I 2022-10-28 00:40:07] ...trap/engines/engine.py.224: train: epoch 96 | batch 2260/2304 [I 2022-10-28 00:40:07] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:00:25 [I 2022-10-28 00:40:07] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00276 [I 2022-10-28 00:40:07] ...trap/engines/engine.py.230: loss: 0.11358 [I 2022-10-28 00:40:07] ...optimizers/trijoint.py.151: total_norm: 15.383248 [I 2022-10-28 00:40:13] ...trap/engines/engine.py.224: train: epoch 96 | batch 2270/2304 [I 2022-10-28 00:40:13] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:18 [I 2022-10-28 00:40:13] ...trap/engines/engine.py.229: process: 0.54545 | load: 0.00263 [I 2022-10-28 00:40:13] ...trap/engines/engine.py.230: loss: 0.09468 [I 2022-10-28 00:40:13] ...optimizers/trijoint.py.151: total_norm: 17.659590 [I 2022-10-28 00:40:19] ...trap/engines/engine.py.224: train: epoch 96 | batch 2280/2304 [I 2022-10-28 00:40:19] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:00:13 [I 2022-10-28 00:40:19] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00306 [I 2022-10-28 00:40:19] ...trap/engines/engine.py.230: loss: 0.12818 [I 2022-10-28 00:40:19] ...optimizers/trijoint.py.151: total_norm: 15.237286 [I 2022-10-28 00:40:24] ...trap/engines/engine.py.224: train: epoch 96 | batch 2290/2304 [I 2022-10-28 00:40:24] ...trap/engines/engine.py.228: elapsed: 0:22:00 | left: 0:00:07 [I 2022-10-28 00:40:24] ...trap/engines/engine.py.229: process: 0.55360 | load: 0.00281 [I 2022-10-28 00:40:24] ...trap/engines/engine.py.230: loss: 0.13144 [I 2022-10-28 00:40:24] ...optimizers/trijoint.py.151: total_norm: 18.398941 [I 2022-10-28 00:40:30] ...trap/engines/engine.py.224: train: epoch 96 | batch 2300/2304 [I 2022-10-28 00:40:30] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:00:02 [I 2022-10-28 00:40:30] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00017 [I 2022-10-28 00:40:30] ...trap/engines/engine.py.230: loss: 0.11002 [I 2022-10-28 00:40:30] ...optimizers/trijoint.py.151: total_norm: 10.947596 [I 2022-10-28 00:40:32] ...trap/engines/engine.py.224: train: epoch 96 | batch 2304/2304 [I 2022-10-28 00:40:32] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:00:00 [I 2022-10-28 00:40:32] ...trap/engines/engine.py.229: process: 0.54192 | load: 0.00020 [I 2022-10-28 00:40:32] ...trap/engines/engine.py.230: loss: 0.12194 [I 2022-10-28 00:40:32] ...optimizers/trijoint.py.151: total_norm: 17.907108 [S 2022-10-28 00:40:33] ...trap/engines/engine.py.129: train_epoch.epoch: 96 [S 2022-10-28 00:40:33] ...trap/engines/engine.py.129: train_epoch.loss: 0.1276917682665677 [I 2022-10-28 00:40:33] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 00:40:35] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 96 [I 2022-10-28 00:40:35] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 00:40:37] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 00:40:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 00:40:37] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 00:40:40] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 00:40:40] ...trap/engines/engine.py.321: eval: epoch 96 | batch 0/510 [I 2022-10-28 00:40:40] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:47:22 [I 2022-10-28 00:40:40] ...trap/engines/engine.py.326: process: 5.57266 | load: 3.85151 [I 2022-10-28 00:40:44] ...trap/engines/engine.py.321: eval: epoch 96 | batch 10/510 [I 2022-10-28 00:40:44] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:26 [I 2022-10-28 00:40:44] ...trap/engines/engine.py.326: process: 0.32604 | load: 0.00017 [I 2022-10-28 00:40:47] ...trap/engines/engine.py.321: eval: epoch 96 | batch 20/510 [I 2022-10-28 00:40:47] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:12 [I 2022-10-28 00:40:47] ...trap/engines/engine.py.326: process: 0.33953 | load: 0.00016 [I 2022-10-28 00:40:50] ...trap/engines/engine.py.321: eval: epoch 96 | batch 30/510 [I 2022-10-28 00:40:50] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:42 [I 2022-10-28 00:40:50] ...trap/engines/engine.py.326: process: 0.33024 | load: 0.00016 [I 2022-10-28 00:40:53] ...trap/engines/engine.py.321: eval: epoch 96 | batch 40/510 [I 2022-10-28 00:40:53] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:36 [I 2022-10-28 00:40:53] ...trap/engines/engine.py.326: process: 0.33399 | load: 0.00018 [I 2022-10-28 00:40:57] ...trap/engines/engine.py.321: eval: epoch 96 | batch 50/510 [I 2022-10-28 00:40:57] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:31 [I 2022-10-28 00:40:57] ...trap/engines/engine.py.326: process: 0.32168 | load: 0.00017 [I 2022-10-28 00:41:00] ...trap/engines/engine.py.321: eval: epoch 96 | batch 60/510 [I 2022-10-28 00:41:00] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:29 [I 2022-10-28 00:41:00] ...trap/engines/engine.py.326: process: 0.33389 | load: 0.00021 [I 2022-10-28 00:41:04] ...trap/engines/engine.py.321: eval: epoch 96 | batch 70/510 [I 2022-10-28 00:41:04] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:29 [I 2022-10-28 00:41:04] ...trap/engines/engine.py.326: process: 0.32016 | load: 0.00016 [I 2022-10-28 00:41:07] ...trap/engines/engine.py.321: eval: epoch 96 | batch 80/510 [I 2022-10-28 00:41:07] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:27 [I 2022-10-28 00:41:07] ...trap/engines/engine.py.326: process: 0.37398 | load: 0.00017 [I 2022-10-28 00:41:10] ...trap/engines/engine.py.321: eval: epoch 96 | batch 90/510 [I 2022-10-28 00:41:10] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:18 [I 2022-10-28 00:41:10] ...trap/engines/engine.py.326: process: 0.32528 | load: 0.00017 [I 2022-10-28 00:41:14] ...trap/engines/engine.py.321: eval: epoch 96 | batch 100/510 [I 2022-10-28 00:41:14] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:15 [I 2022-10-28 00:41:14] ...trap/engines/engine.py.326: process: 0.30764 | load: 0.00015 [I 2022-10-28 00:41:17] ...trap/engines/engine.py.321: eval: epoch 96 | batch 110/510 [I 2022-10-28 00:41:17] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-28 00:41:17] ...trap/engines/engine.py.326: process: 0.28726 | load: 0.00014 [I 2022-10-28 00:41:19] ...trap/engines/engine.py.321: eval: epoch 96 | batch 120/510 [I 2022-10-28 00:41:19] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-28 00:41:19] ...trap/engines/engine.py.326: process: 0.28782 | load: 0.00014 [I 2022-10-28 00:41:22] ...trap/engines/engine.py.321: eval: epoch 96 | batch 130/510 [I 2022-10-28 00:41:22] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:50 [I 2022-10-28 00:41:22] ...trap/engines/engine.py.326: process: 0.29066 | load: 0.00014 [I 2022-10-28 00:41:25] ...trap/engines/engine.py.321: eval: epoch 96 | batch 140/510 [I 2022-10-28 00:41:25] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-28 00:41:25] ...trap/engines/engine.py.326: process: 0.28684 | load: 0.00017 [I 2022-10-28 00:41:28] ...trap/engines/engine.py.321: eval: epoch 96 | batch 150/510 [I 2022-10-28 00:41:28] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:44 [I 2022-10-28 00:41:28] ...trap/engines/engine.py.326: process: 0.29056 | load: 0.00014 [I 2022-10-28 00:41:31] ...trap/engines/engine.py.321: eval: epoch 96 | batch 160/510 [I 2022-10-28 00:41:31] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-28 00:41:31] ...trap/engines/engine.py.326: process: 0.29161 | load: 0.00014 [I 2022-10-28 00:41:34] ...trap/engines/engine.py.321: eval: epoch 96 | batch 170/510 [I 2022-10-28 00:41:34] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 00:41:34] ...trap/engines/engine.py.326: process: 0.28922 | load: 0.00018 [I 2022-10-28 00:41:37] ...trap/engines/engine.py.321: eval: epoch 96 | batch 180/510 [I 2022-10-28 00:41:37] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:33 [I 2022-10-28 00:41:37] ...trap/engines/engine.py.326: process: 0.28492 | load: 0.00016 [I 2022-10-28 00:41:40] ...trap/engines/engine.py.321: eval: epoch 96 | batch 190/510 [I 2022-10-28 00:41:40] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-28 00:41:40] ...trap/engines/engine.py.326: process: 0.28732 | load: 0.00014 [I 2022-10-28 00:41:43] ...trap/engines/engine.py.321: eval: epoch 96 | batch 200/510 [I 2022-10-28 00:41:43] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:30 [I 2022-10-28 00:41:43] ...trap/engines/engine.py.326: process: 0.28614 | load: 0.00015 [I 2022-10-28 00:41:45] ...trap/engines/engine.py.321: eval: epoch 96 | batch 210/510 [I 2022-10-28 00:41:45] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-28 00:41:45] ...trap/engines/engine.py.326: process: 0.28537 | load: 0.00017 [I 2022-10-28 00:41:48] ...trap/engines/engine.py.321: eval: epoch 96 | batch 220/510 [I 2022-10-28 00:41:48] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-28 00:41:48] ...trap/engines/engine.py.326: process: 0.28638 | load: 0.00015 [I 2022-10-28 00:41:51] ...trap/engines/engine.py.321: eval: epoch 96 | batch 230/510 [I 2022-10-28 00:41:51] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:21 [I 2022-10-28 00:41:51] ...trap/engines/engine.py.326: process: 0.28684 | load: 0.00016 [I 2022-10-28 00:41:54] ...trap/engines/engine.py.321: eval: epoch 96 | batch 240/510 [I 2022-10-28 00:41:54] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 00:41:54] ...trap/engines/engine.py.326: process: 0.28957 | load: 0.00016 [I 2022-10-28 00:41:57] ...trap/engines/engine.py.321: eval: epoch 96 | batch 250/510 [I 2022-10-28 00:41:57] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-28 00:41:57] ...trap/engines/engine.py.326: process: 0.28953 | load: 0.00016 [I 2022-10-28 00:42:00] ...trap/engines/engine.py.321: eval: epoch 96 | batch 260/510 [I 2022-10-28 00:42:00] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 00:42:00] ...trap/engines/engine.py.326: process: 0.29367 | load: 0.00016 [I 2022-10-28 00:42:03] ...trap/engines/engine.py.321: eval: epoch 96 | batch 270/510 [I 2022-10-28 00:42:03] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-28 00:42:03] ...trap/engines/engine.py.326: process: 0.28429 | load: 0.00017 [I 2022-10-28 00:42:06] ...trap/engines/engine.py.321: eval: epoch 96 | batch 280/510 [I 2022-10-28 00:42:06] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 00:42:06] ...trap/engines/engine.py.326: process: 0.29024 | load: 0.00015 [I 2022-10-28 00:42:09] ...trap/engines/engine.py.321: eval: epoch 96 | batch 290/510 [I 2022-10-28 00:42:09] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-28 00:42:09] ...trap/engines/engine.py.326: process: 0.28360 | load: 0.00015 [I 2022-10-28 00:42:11] ...trap/engines/engine.py.321: eval: epoch 96 | batch 300/510 [I 2022-10-28 00:42:11] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-28 00:42:11] ...trap/engines/engine.py.326: process: 0.29105 | load: 0.00015 [I 2022-10-28 00:42:14] ...trap/engines/engine.py.321: eval: epoch 96 | batch 310/510 [I 2022-10-28 00:42:14] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 00:42:14] ...trap/engines/engine.py.326: process: 0.28280 | load: 0.00015 [I 2022-10-28 00:42:17] ...trap/engines/engine.py.321: eval: epoch 96 | batch 320/510 [I 2022-10-28 00:42:17] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 00:42:17] ...trap/engines/engine.py.326: process: 0.28658 | load: 0.00015 [I 2022-10-28 00:42:20] ...trap/engines/engine.py.321: eval: epoch 96 | batch 330/510 [I 2022-10-28 00:42:20] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-28 00:42:20] ...trap/engines/engine.py.326: process: 0.28906 | load: 0.00015 [I 2022-10-28 00:42:23] ...trap/engines/engine.py.321: eval: epoch 96 | batch 340/510 [I 2022-10-28 00:42:23] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-28 00:42:23] ...trap/engines/engine.py.326: process: 0.28648 | load: 0.00020 [I 2022-10-28 00:42:26] ...trap/engines/engine.py.321: eval: epoch 96 | batch 350/510 [I 2022-10-28 00:42:26] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 00:42:26] ...trap/engines/engine.py.326: process: 0.28959 | load: 0.00015 [I 2022-10-28 00:42:29] ...trap/engines/engine.py.321: eval: epoch 96 | batch 360/510 [I 2022-10-28 00:42:29] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:42 [I 2022-10-28 00:42:29] ...trap/engines/engine.py.326: process: 0.28471 | load: 0.00015 [I 2022-10-28 00:42:32] ...trap/engines/engine.py.321: eval: epoch 96 | batch 370/510 [I 2022-10-28 00:42:32] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 00:42:32] ...trap/engines/engine.py.326: process: 0.28983 | load: 0.00022 [I 2022-10-28 00:42:35] ...trap/engines/engine.py.321: eval: epoch 96 | batch 380/510 [I 2022-10-28 00:42:35] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-28 00:42:35] ...trap/engines/engine.py.326: process: 0.28737 | load: 0.00016 [I 2022-10-28 00:42:37] ...trap/engines/engine.py.321: eval: epoch 96 | batch 390/510 [I 2022-10-28 00:42:37] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 00:42:37] ...trap/engines/engine.py.326: process: 0.28619 | load: 0.00020 [I 2022-10-28 00:42:40] ...trap/engines/engine.py.321: eval: epoch 96 | batch 400/510 [I 2022-10-28 00:42:40] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 00:42:40] ...trap/engines/engine.py.326: process: 0.28331 | load: 0.00032 [I 2022-10-28 00:42:43] ...trap/engines/engine.py.321: eval: epoch 96 | batch 410/510 [I 2022-10-28 00:42:43] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 00:42:43] ...trap/engines/engine.py.326: process: 0.28380 | load: 0.00021 [I 2022-10-28 00:42:46] ...trap/engines/engine.py.321: eval: epoch 96 | batch 420/510 [I 2022-10-28 00:42:46] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 00:42:46] ...trap/engines/engine.py.326: process: 0.28391 | load: 0.00021 [I 2022-10-28 00:42:49] ...trap/engines/engine.py.321: eval: epoch 96 | batch 430/510 [I 2022-10-28 00:42:49] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:22 [I 2022-10-28 00:42:49] ...trap/engines/engine.py.326: process: 0.28962 | load: 0.00021 [I 2022-10-28 00:42:52] ...trap/engines/engine.py.321: eval: epoch 96 | batch 440/510 [I 2022-10-28 00:42:52] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 00:42:52] ...trap/engines/engine.py.326: process: 0.29764 | load: 0.00016 [I 2022-10-28 00:42:55] ...trap/engines/engine.py.321: eval: epoch 96 | batch 450/510 [I 2022-10-28 00:42:55] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 00:42:55] ...trap/engines/engine.py.326: process: 0.28211 | load: 0.00024 [I 2022-10-28 00:42:58] ...trap/engines/engine.py.321: eval: epoch 96 | batch 460/510 [I 2022-10-28 00:42:58] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 00:42:58] ...trap/engines/engine.py.326: process: 0.28460 | load: 0.00020 [I 2022-10-28 00:43:01] ...trap/engines/engine.py.321: eval: epoch 96 | batch 470/510 [I 2022-10-28 00:43:01] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 00:43:01] ...trap/engines/engine.py.326: process: 0.28712 | load: 0.00039 [I 2022-10-28 00:43:04] ...trap/engines/engine.py.321: eval: epoch 96 | batch 480/510 [I 2022-10-28 00:43:04] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 00:43:04] ...trap/engines/engine.py.326: process: 0.28620 | load: 0.00019 [I 2022-10-28 00:43:06] ...trap/engines/engine.py.321: eval: epoch 96 | batch 490/510 [I 2022-10-28 00:43:06] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 00:43:06] ...trap/engines/engine.py.326: process: 0.29120 | load: 0.00037 [I 2022-10-28 00:43:09] ...trap/engines/engine.py.321: eval: epoch 96 | batch 500/510 [I 2022-10-28 00:43:09] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 00:43:09] ...trap/engines/engine.py.326: process: 0.28799 | load: 0.00021 [I 2022-10-28 00:43:12] ...trap/engines/engine.py.321: eval: epoch 96 | batch 510/510 [I 2022-10-28 00:43:12] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 00:43:12] ...trap/engines/engine.py.326: process: 0.28863 | load: 0.00017 [S 2022-10-28 00:43:13] ...trap/engines/engine.py.132: eval_epoch.epoch: 96 [I 2022-10-28 00:43:13] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.739600 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909900 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936500 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.729900 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909500 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936100 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017884 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008938 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005536 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.018918 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007710 [S 2022-10-28 00:43:17] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004784 [I 2022-10-28 00:43:19] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 00:43:19] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 00:43:19] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 00:43:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 00:43:19] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 00:43:24] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 00:43:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 00:43:30] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 00:43:30] ...trap/engines/engine.py.163: Training model on trainset for epoch 97 [I 2022-10-28 00:43:30] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 00:43:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 0/2304 [I 2022-10-28 00:43:33] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:54 [I 2022-10-28 00:43:33] ...trap/engines/engine.py.229: process: 2.78397 | load: 2.14700 [I 2022-10-28 00:43:33] ...trap/engines/engine.py.230: loss: 0.13833 [I 2022-10-28 00:43:33] ...optimizers/trijoint.py.151: total_norm: 17.236801 [I 2022-10-28 00:43:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 10/2304 [I 2022-10-28 00:43:39] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:34 [I 2022-10-28 00:43:39] ...trap/engines/engine.py.229: process: 0.56541 | load: 0.00462 [I 2022-10-28 00:43:39] ...trap/engines/engine.py.230: loss: 0.11550 [I 2022-10-28 00:43:39] ...optimizers/trijoint.py.151: total_norm: 25.704714 [I 2022-10-28 00:43:44] ...trap/engines/engine.py.224: train: epoch 97 | batch 20/2304 [I 2022-10-28 00:43:44] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:07 [I 2022-10-28 00:43:44] ...trap/engines/engine.py.229: process: 0.55676 | load: 0.00392 [I 2022-10-28 00:43:44] ...trap/engines/engine.py.230: loss: 0.15662 [I 2022-10-28 00:43:44] ...optimizers/trijoint.py.151: total_norm: 64.592476 [I 2022-10-28 00:43:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 30/2304 [I 2022-10-28 00:43:50] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-28 00:43:50] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00302 [I 2022-10-28 00:43:50] ...trap/engines/engine.py.230: loss: 0.13554 [I 2022-10-28 00:43:50] ...optimizers/trijoint.py.151: total_norm: 10.690494 [I 2022-10-28 00:43:56] ...trap/engines/engine.py.224: train: epoch 97 | batch 40/2304 [I 2022-10-28 00:43:56] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:12 [I 2022-10-28 00:43:56] ...trap/engines/engine.py.229: process: 0.56704 | load: 0.00296 [I 2022-10-28 00:43:56] ...trap/engines/engine.py.230: loss: 0.12332 [I 2022-10-28 00:43:56] ...optimizers/trijoint.py.151: total_norm: 10.119248 [I 2022-10-28 00:44:01] ...trap/engines/engine.py.224: train: epoch 97 | batch 50/2304 [I 2022-10-28 00:44:01] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:07 [I 2022-10-28 00:44:01] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00290 [I 2022-10-28 00:44:01] ...trap/engines/engine.py.230: loss: 0.11106 [I 2022-10-28 00:44:01] ...optimizers/trijoint.py.151: total_norm: 9.737221 [I 2022-10-28 00:44:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 60/2304 [I 2022-10-28 00:44:07] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:55 [I 2022-10-28 00:44:07] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00298 [I 2022-10-28 00:44:07] ...trap/engines/engine.py.230: loss: 0.14682 [I 2022-10-28 00:44:07] ...optimizers/trijoint.py.151: total_norm: 13.072126 [I 2022-10-28 00:44:12] ...trap/engines/engine.py.224: train: epoch 97 | batch 70/2304 [I 2022-10-28 00:44:12] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:21:00 [I 2022-10-28 00:44:12] ...trap/engines/engine.py.229: process: 0.56186 | load: 0.00267 [I 2022-10-28 00:44:12] ...trap/engines/engine.py.230: loss: 0.10695 [I 2022-10-28 00:44:12] ...optimizers/trijoint.py.151: total_norm: 14.862250 [I 2022-10-28 00:44:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 80/2304 [I 2022-10-28 00:44:18] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:46 [I 2022-10-28 00:44:18] ...trap/engines/engine.py.229: process: 0.55561 | load: 0.00270 [I 2022-10-28 00:44:18] ...trap/engines/engine.py.230: loss: 0.12524 [I 2022-10-28 00:44:18] ...optimizers/trijoint.py.151: total_norm: 18.717752 [I 2022-10-28 00:44:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 90/2304 [I 2022-10-28 00:44:24] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:46 [I 2022-10-28 00:44:24] ...trap/engines/engine.py.229: process: 0.55842 | load: 0.00291 [I 2022-10-28 00:44:24] ...trap/engines/engine.py.230: loss: 0.10202 [I 2022-10-28 00:44:24] ...optimizers/trijoint.py.151: total_norm: 11.265061 [I 2022-10-28 00:44:29] ...trap/engines/engine.py.224: train: epoch 97 | batch 100/2304 [I 2022-10-28 00:44:29] ...trap/engines/engine.py.228: elapsed: 0:00:59 | left: 0:20:40 [I 2022-10-28 00:44:29] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00275 [I 2022-10-28 00:44:29] ...trap/engines/engine.py.230: loss: 0.09689 [I 2022-10-28 00:44:29] ...optimizers/trijoint.py.151: total_norm: 22.294308 [I 2022-10-28 00:44:35] ...trap/engines/engine.py.224: train: epoch 97 | batch 110/2304 [I 2022-10-28 00:44:35] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:33 [I 2022-10-28 00:44:35] ...trap/engines/engine.py.229: process: 0.56376 | load: 0.00279 [I 2022-10-28 00:44:35] ...trap/engines/engine.py.230: loss: 0.14230 [I 2022-10-28 00:44:35] ...optimizers/trijoint.py.151: total_norm: 12.113642 [I 2022-10-28 00:44:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 120/2304 [I 2022-10-28 00:44:41] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:24 [I 2022-10-28 00:44:41] ...trap/engines/engine.py.229: process: 0.56822 | load: 0.00274 [I 2022-10-28 00:44:41] ...trap/engines/engine.py.230: loss: 0.14010 [I 2022-10-28 00:44:41] ...optimizers/trijoint.py.151: total_norm: 29.644632 [I 2022-10-28 00:44:46] ...trap/engines/engine.py.224: train: epoch 97 | batch 130/2304 [I 2022-10-28 00:44:46] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:26 [I 2022-10-28 00:44:46] ...trap/engines/engine.py.229: process: 0.57228 | load: 0.00285 [I 2022-10-28 00:44:46] ...trap/engines/engine.py.230: loss: 0.12982 [I 2022-10-28 00:44:46] ...optimizers/trijoint.py.151: total_norm: 11.528709 [I 2022-10-28 00:44:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 140/2304 [I 2022-10-28 00:44:52] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:11 [I 2022-10-28 00:44:52] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00259 [I 2022-10-28 00:44:52] ...trap/engines/engine.py.230: loss: 0.10529 [I 2022-10-28 00:44:52] ...optimizers/trijoint.py.151: total_norm: 14.866603 [I 2022-10-28 00:44:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 150/2304 [I 2022-10-28 00:44:57] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:20:06 [I 2022-10-28 00:44:57] ...trap/engines/engine.py.229: process: 0.57801 | load: 0.00260 [I 2022-10-28 00:44:57] ...trap/engines/engine.py.230: loss: 0.11235 [I 2022-10-28 00:44:57] ...optimizers/trijoint.py.151: total_norm: 16.188217 [I 2022-10-28 00:45:03] ...trap/engines/engine.py.224: train: epoch 97 | batch 160/2304 [I 2022-10-28 00:45:03] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:05 [I 2022-10-28 00:45:03] ...trap/engines/engine.py.229: process: 0.56513 | load: 0.00266 [I 2022-10-28 00:45:03] ...trap/engines/engine.py.230: loss: 0.11860 [I 2022-10-28 00:45:03] ...optimizers/trijoint.py.151: total_norm: 14.371046 [I 2022-10-28 00:45:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 170/2304 [I 2022-10-28 00:45:09] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:49 [I 2022-10-28 00:45:09] ...trap/engines/engine.py.229: process: 0.56427 | load: 0.00267 [I 2022-10-28 00:45:09] ...trap/engines/engine.py.230: loss: 0.10416 [I 2022-10-28 00:45:09] ...optimizers/trijoint.py.151: total_norm: 9.818136 [I 2022-10-28 00:45:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 180/2304 [I 2022-10-28 00:45:14] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:43 [I 2022-10-28 00:45:14] ...trap/engines/engine.py.229: process: 0.55017 | load: 0.00251 [I 2022-10-28 00:45:14] ...trap/engines/engine.py.230: loss: 0.09118 [I 2022-10-28 00:45:14] ...optimizers/trijoint.py.151: total_norm: 9.988170 [I 2022-10-28 00:45:20] ...trap/engines/engine.py.224: train: epoch 97 | batch 190/2304 [I 2022-10-28 00:45:20] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:36 [I 2022-10-28 00:45:20] ...trap/engines/engine.py.229: process: 0.57194 | load: 0.00253 [I 2022-10-28 00:45:20] ...trap/engines/engine.py.230: loss: 0.17249 [I 2022-10-28 00:45:20] ...optimizers/trijoint.py.151: total_norm: 11.062203 [I 2022-10-28 00:45:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 200/2304 [I 2022-10-28 00:45:25] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:31 [I 2022-10-28 00:45:25] ...trap/engines/engine.py.229: process: 0.56270 | load: 0.00256 [I 2022-10-28 00:45:25] ...trap/engines/engine.py.230: loss: 0.08972 [I 2022-10-28 00:45:25] ...optimizers/trijoint.py.151: total_norm: 12.318235 [I 2022-10-28 00:45:31] ...trap/engines/engine.py.224: train: epoch 97 | batch 210/2304 [I 2022-10-28 00:45:31] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:14 [I 2022-10-28 00:45:31] ...trap/engines/engine.py.229: process: 0.53881 | load: 0.00254 [I 2022-10-28 00:45:31] ...trap/engines/engine.py.230: loss: 0.07907 [I 2022-10-28 00:45:31] ...optimizers/trijoint.py.151: total_norm: 13.082805 [I 2022-10-28 00:45:36] ...trap/engines/engine.py.224: train: epoch 97 | batch 220/2304 [I 2022-10-28 00:45:36] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:21 [I 2022-10-28 00:45:36] ...trap/engines/engine.py.229: process: 0.56508 | load: 0.00243 [I 2022-10-28 00:45:36] ...trap/engines/engine.py.230: loss: 0.13548 [I 2022-10-28 00:45:36] ...optimizers/trijoint.py.151: total_norm: 14.958509 [I 2022-10-28 00:45:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 230/2304 [I 2022-10-28 00:45:42] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:18 [I 2022-10-28 00:45:42] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00264 [I 2022-10-28 00:45:42] ...trap/engines/engine.py.230: loss: 0.15690 [I 2022-10-28 00:45:42] ...optimizers/trijoint.py.151: total_norm: 29.195145 [I 2022-10-28 00:45:48] ...trap/engines/engine.py.224: train: epoch 97 | batch 240/2304 [I 2022-10-28 00:45:48] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:19 [I 2022-10-28 00:45:48] ...trap/engines/engine.py.229: process: 0.54405 | load: 0.00247 [I 2022-10-28 00:45:48] ...trap/engines/engine.py.230: loss: 0.10918 [I 2022-10-28 00:45:48] ...optimizers/trijoint.py.151: total_norm: 7.888228 [I 2022-10-28 00:45:53] ...trap/engines/engine.py.224: train: epoch 97 | batch 250/2304 [I 2022-10-28 00:45:53] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:05 [I 2022-10-28 00:45:53] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00247 [I 2022-10-28 00:45:53] ...trap/engines/engine.py.230: loss: 0.21190 [I 2022-10-28 00:45:53] ...optimizers/trijoint.py.151: total_norm: 28.577703 [I 2022-10-28 00:45:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 260/2304 [I 2022-10-28 00:45:59] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:15 [I 2022-10-28 00:45:59] ...trap/engines/engine.py.229: process: 0.57046 | load: 0.00243 [I 2022-10-28 00:45:59] ...trap/engines/engine.py.230: loss: 0.10192 [I 2022-10-28 00:45:59] ...optimizers/trijoint.py.151: total_norm: 26.311430 [I 2022-10-28 00:46:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 270/2304 [I 2022-10-28 00:46:04] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:47 [I 2022-10-28 00:46:04] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00253 [I 2022-10-28 00:46:04] ...trap/engines/engine.py.230: loss: 0.14910 [I 2022-10-28 00:46:04] ...optimizers/trijoint.py.151: total_norm: 24.464361 [I 2022-10-28 00:46:10] ...trap/engines/engine.py.224: train: epoch 97 | batch 280/2304 [I 2022-10-28 00:46:10] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:50 [I 2022-10-28 00:46:10] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00263 [I 2022-10-28 00:46:10] ...trap/engines/engine.py.230: loss: 0.12347 [I 2022-10-28 00:46:10] ...optimizers/trijoint.py.151: total_norm: 37.132938 [I 2022-10-28 00:46:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 290/2304 [I 2022-10-28 00:46:16] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:42 [I 2022-10-28 00:46:16] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00249 [I 2022-10-28 00:46:16] ...trap/engines/engine.py.230: loss: 0.14540 [I 2022-10-28 00:46:16] ...optimizers/trijoint.py.151: total_norm: 22.528107 [I 2022-10-28 00:46:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 300/2304 [I 2022-10-28 00:46:21] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:28 [I 2022-10-28 00:46:21] ...trap/engines/engine.py.229: process: 0.54224 | load: 0.00246 [I 2022-10-28 00:46:21] ...trap/engines/engine.py.230: loss: 0.11120 [I 2022-10-28 00:46:21] ...optimizers/trijoint.py.151: total_norm: 22.277800 [I 2022-10-28 00:46:27] ...trap/engines/engine.py.224: train: epoch 97 | batch 310/2304 [I 2022-10-28 00:46:27] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:23 [I 2022-10-28 00:46:27] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00245 [I 2022-10-28 00:46:27] ...trap/engines/engine.py.230: loss: 0.16929 [I 2022-10-28 00:46:27] ...optimizers/trijoint.py.151: total_norm: 17.315184 [I 2022-10-28 00:46:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 320/2304 [I 2022-10-28 00:46:32] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:32 [I 2022-10-28 00:46:32] ...trap/engines/engine.py.229: process: 0.58276 | load: 0.00254 [I 2022-10-28 00:46:32] ...trap/engines/engine.py.230: loss: 0.12042 [I 2022-10-28 00:46:32] ...optimizers/trijoint.py.151: total_norm: 19.857597 [I 2022-10-28 00:46:38] ...trap/engines/engine.py.224: train: epoch 97 | batch 330/2304 [I 2022-10-28 00:46:38] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:59 [I 2022-10-28 00:46:38] ...trap/engines/engine.py.229: process: 0.59165 | load: 0.00267 [I 2022-10-28 00:46:38] ...trap/engines/engine.py.230: loss: 0.08317 [I 2022-10-28 00:46:38] ...optimizers/trijoint.py.151: total_norm: 9.337990 [I 2022-10-28 00:46:44] ...trap/engines/engine.py.224: train: epoch 97 | batch 340/2304 [I 2022-10-28 00:46:44] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:29 [I 2022-10-28 00:46:44] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00248 [I 2022-10-28 00:46:44] ...trap/engines/engine.py.230: loss: 0.09375 [I 2022-10-28 00:46:44] ...optimizers/trijoint.py.151: total_norm: 15.422585 [I 2022-10-28 00:46:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 350/2304 [I 2022-10-28 00:46:49] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:19:09 [I 2022-10-28 00:46:49] ...trap/engines/engine.py.229: process: 0.70086 | load: 0.00256 [I 2022-10-28 00:46:49] ...trap/engines/engine.py.230: loss: 0.10999 [I 2022-10-28 00:46:49] ...optimizers/trijoint.py.151: total_norm: 8.618714 [I 2022-10-28 00:46:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 360/2304 [I 2022-10-28 00:46:55] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:19 [I 2022-10-28 00:46:55] ...trap/engines/engine.py.229: process: 0.57177 | load: 0.00245 [I 2022-10-28 00:46:55] ...trap/engines/engine.py.230: loss: 0.12840 [I 2022-10-28 00:46:55] ...optimizers/trijoint.py.151: total_norm: 17.988874 [I 2022-10-28 00:47:01] ...trap/engines/engine.py.224: train: epoch 97 | batch 370/2304 [I 2022-10-28 00:47:01] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:08 [I 2022-10-28 00:47:01] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00251 [I 2022-10-28 00:47:01] ...trap/engines/engine.py.230: loss: 0.16377 [I 2022-10-28 00:47:01] ...optimizers/trijoint.py.151: total_norm: 17.226030 [I 2022-10-28 00:47:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 380/2304 [I 2022-10-28 00:47:06] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:52 [I 2022-10-28 00:47:06] ...trap/engines/engine.py.229: process: 0.54731 | load: 0.00249 [I 2022-10-28 00:47:06] ...trap/engines/engine.py.230: loss: 0.08566 [I 2022-10-28 00:47:06] ...optimizers/trijoint.py.151: total_norm: 6.074330 [I 2022-10-28 00:47:12] ...trap/engines/engine.py.224: train: epoch 97 | batch 390/2304 [I 2022-10-28 00:47:12] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:49 [I 2022-10-28 00:47:12] ...trap/engines/engine.py.229: process: 0.56567 | load: 0.00258 [I 2022-10-28 00:47:12] ...trap/engines/engine.py.230: loss: 0.14918 [I 2022-10-28 00:47:12] ...optimizers/trijoint.py.151: total_norm: 23.749144 [I 2022-10-28 00:47:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 400/2304 [I 2022-10-28 00:47:17] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:38 [I 2022-10-28 00:47:17] ...trap/engines/engine.py.229: process: 0.55154 | load: 0.00251 [I 2022-10-28 00:47:17] ...trap/engines/engine.py.230: loss: 0.15339 [I 2022-10-28 00:47:17] ...optimizers/trijoint.py.151: total_norm: 9.887515 [I 2022-10-28 00:47:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 410/2304 [I 2022-10-28 00:47:23] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:42 [I 2022-10-28 00:47:23] ...trap/engines/engine.py.229: process: 0.57124 | load: 0.00251 [I 2022-10-28 00:47:23] ...trap/engines/engine.py.230: loss: 0.15167 [I 2022-10-28 00:47:23] ...optimizers/trijoint.py.151: total_norm: 14.072409 [I 2022-10-28 00:47:29] ...trap/engines/engine.py.224: train: epoch 97 | batch 420/2304 [I 2022-10-28 00:47:29] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:38 [I 2022-10-28 00:47:29] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00252 [I 2022-10-28 00:47:29] ...trap/engines/engine.py.230: loss: 0.13134 [I 2022-10-28 00:47:29] ...optimizers/trijoint.py.151: total_norm: 17.658880 [I 2022-10-28 00:47:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 430/2304 [I 2022-10-28 00:47:34] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:30 [I 2022-10-28 00:47:34] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00254 [I 2022-10-28 00:47:34] ...trap/engines/engine.py.230: loss: 0.12478 [I 2022-10-28 00:47:34] ...optimizers/trijoint.py.151: total_norm: 15.769396 [I 2022-10-28 00:47:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 440/2304 [I 2022-10-28 00:47:40] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:20 [I 2022-10-28 00:47:40] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00254 [I 2022-10-28 00:47:40] ...trap/engines/engine.py.230: loss: 0.09264 [I 2022-10-28 00:47:40] ...optimizers/trijoint.py.151: total_norm: 12.895245 [I 2022-10-28 00:47:46] ...trap/engines/engine.py.224: train: epoch 97 | batch 450/2304 [I 2022-10-28 00:47:46] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:23 [I 2022-10-28 00:47:46] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00253 [I 2022-10-28 00:47:46] ...trap/engines/engine.py.230: loss: 0.10844 [I 2022-10-28 00:47:46] ...optimizers/trijoint.py.151: total_norm: 16.263851 [I 2022-10-28 00:47:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 460/2304 [I 2022-10-28 00:47:51] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:06 [I 2022-10-28 00:47:51] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00249 [I 2022-10-28 00:47:51] ...trap/engines/engine.py.230: loss: 0.22026 [I 2022-10-28 00:47:51] ...optimizers/trijoint.py.151: total_norm: 16.277338 [I 2022-10-28 00:47:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 470/2304 [I 2022-10-28 00:47:57] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:11 [I 2022-10-28 00:47:57] ...trap/engines/engine.py.229: process: 0.56931 | load: 0.00243 [I 2022-10-28 00:47:57] ...trap/engines/engine.py.230: loss: 0.15805 [I 2022-10-28 00:47:57] ...optimizers/trijoint.py.151: total_norm: 8.896810 [I 2022-10-28 00:48:03] ...trap/engines/engine.py.224: train: epoch 97 | batch 480/2304 [I 2022-10-28 00:48:03] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:17:29 [I 2022-10-28 00:48:03] ...trap/engines/engine.py.229: process: 0.57430 | load: 0.00254 [I 2022-10-28 00:48:03] ...trap/engines/engine.py.230: loss: 0.11584 [I 2022-10-28 00:48:03] ...optimizers/trijoint.py.151: total_norm: 15.945048 [I 2022-10-28 00:48:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 490/2304 [I 2022-10-28 00:48:08] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:56 [I 2022-10-28 00:48:08] ...trap/engines/engine.py.229: process: 0.56465 | load: 0.00240 [I 2022-10-28 00:48:08] ...trap/engines/engine.py.230: loss: 0.15475 [I 2022-10-28 00:48:08] ...optimizers/trijoint.py.151: total_norm: 23.298817 [I 2022-10-28 00:48:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 500/2304 [I 2022-10-28 00:48:14] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:50 [I 2022-10-28 00:48:14] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00243 [I 2022-10-28 00:48:14] ...trap/engines/engine.py.230: loss: 0.14051 [I 2022-10-28 00:48:14] ...optimizers/trijoint.py.151: total_norm: 18.887135 [I 2022-10-28 00:48:19] ...trap/engines/engine.py.224: train: epoch 97 | batch 510/2304 [I 2022-10-28 00:48:19] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:43 [I 2022-10-28 00:48:19] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00243 [I 2022-10-28 00:48:19] ...trap/engines/engine.py.230: loss: 0.12946 [I 2022-10-28 00:48:19] ...optimizers/trijoint.py.151: total_norm: 15.044494 [I 2022-10-28 00:48:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 520/2304 [I 2022-10-28 00:48:25] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:31 [I 2022-10-28 00:48:25] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00243 [I 2022-10-28 00:48:25] ...trap/engines/engine.py.230: loss: 0.11171 [I 2022-10-28 00:48:25] ...optimizers/trijoint.py.151: total_norm: 10.369333 [I 2022-10-28 00:48:31] ...trap/engines/engine.py.224: train: epoch 97 | batch 530/2304 [I 2022-10-28 00:48:31] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:39 [I 2022-10-28 00:48:31] ...trap/engines/engine.py.229: process: 0.56632 | load: 0.00265 [I 2022-10-28 00:48:31] ...trap/engines/engine.py.230: loss: 0.07993 [I 2022-10-28 00:48:31] ...optimizers/trijoint.py.151: total_norm: 27.120991 [I 2022-10-28 00:48:36] ...trap/engines/engine.py.224: train: epoch 97 | batch 540/2304 [I 2022-10-28 00:48:36] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:37 [I 2022-10-28 00:48:36] ...trap/engines/engine.py.229: process: 0.57158 | load: 0.00255 [I 2022-10-28 00:48:36] ...trap/engines/engine.py.230: loss: 0.16430 [I 2022-10-28 00:48:36] ...optimizers/trijoint.py.151: total_norm: 19.258097 [I 2022-10-28 00:48:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 550/2304 [I 2022-10-28 00:48:42] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:27 [I 2022-10-28 00:48:42] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00244 [I 2022-10-28 00:48:42] ...trap/engines/engine.py.230: loss: 0.14009 [I 2022-10-28 00:48:42] ...optimizers/trijoint.py.151: total_norm: 44.942303 [I 2022-10-28 00:48:47] ...trap/engines/engine.py.224: train: epoch 97 | batch 560/2304 [I 2022-10-28 00:48:47] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:12 [I 2022-10-28 00:48:47] ...trap/engines/engine.py.229: process: 0.55727 | load: 0.00247 [I 2022-10-28 00:48:47] ...trap/engines/engine.py.230: loss: 0.07240 [I 2022-10-28 00:48:47] ...optimizers/trijoint.py.151: total_norm: 10.102568 [I 2022-10-28 00:48:53] ...trap/engines/engine.py.224: train: epoch 97 | batch 570/2304 [I 2022-10-28 00:48:53] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:03 [I 2022-10-28 00:48:53] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00247 [I 2022-10-28 00:48:53] ...trap/engines/engine.py.230: loss: 0.07722 [I 2022-10-28 00:48:53] ...optimizers/trijoint.py.151: total_norm: 18.021832 [I 2022-10-28 00:48:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 580/2304 [I 2022-10-28 00:48:59] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:15:58 [I 2022-10-28 00:48:59] ...trap/engines/engine.py.229: process: 0.54219 | load: 0.00247 [I 2022-10-28 00:48:59] ...trap/engines/engine.py.230: loss: 0.11851 [I 2022-10-28 00:48:59] ...optimizers/trijoint.py.151: total_norm: 11.578192 [I 2022-10-28 00:49:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 590/2304 [I 2022-10-28 00:49:04] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:18 [I 2022-10-28 00:49:04] ...trap/engines/engine.py.229: process: 0.56576 | load: 0.00250 [I 2022-10-28 00:49:04] ...trap/engines/engine.py.230: loss: 0.11889 [I 2022-10-28 00:49:04] ...optimizers/trijoint.py.151: total_norm: 16.201838 [I 2022-10-28 00:49:10] ...trap/engines/engine.py.224: train: epoch 97 | batch 600/2304 [I 2022-10-28 00:49:10] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:16:00 [I 2022-10-28 00:49:10] ...trap/engines/engine.py.229: process: 0.56729 | load: 0.00250 [I 2022-10-28 00:49:10] ...trap/engines/engine.py.230: loss: 0.12163 [I 2022-10-28 00:49:10] ...optimizers/trijoint.py.151: total_norm: 17.900629 [I 2022-10-28 00:49:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 610/2304 [I 2022-10-28 00:49:16] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:15:49 [I 2022-10-28 00:49:16] ...trap/engines/engine.py.229: process: 0.55353 | load: 0.00248 [I 2022-10-28 00:49:16] ...trap/engines/engine.py.230: loss: 0.10864 [I 2022-10-28 00:49:16] ...optimizers/trijoint.py.151: total_norm: 7.905133 [I 2022-10-28 00:49:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 620/2304 [I 2022-10-28 00:49:21] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:41 [I 2022-10-28 00:49:21] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00240 [I 2022-10-28 00:49:21] ...trap/engines/engine.py.230: loss: 0.09537 [I 2022-10-28 00:49:21] ...optimizers/trijoint.py.151: total_norm: 11.467328 [I 2022-10-28 00:49:27] ...trap/engines/engine.py.224: train: epoch 97 | batch 630/2304 [I 2022-10-28 00:49:27] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:38 [I 2022-10-28 00:49:27] ...trap/engines/engine.py.229: process: 0.56705 | load: 0.00244 [I 2022-10-28 00:49:27] ...trap/engines/engine.py.230: loss: 0.12305 [I 2022-10-28 00:49:27] ...optimizers/trijoint.py.151: total_norm: 14.846721 [I 2022-10-28 00:49:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 640/2304 [I 2022-10-28 00:49:32] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:24 [I 2022-10-28 00:49:32] ...trap/engines/engine.py.229: process: 0.55940 | load: 0.00261 [I 2022-10-28 00:49:32] ...trap/engines/engine.py.230: loss: 0.11798 [I 2022-10-28 00:49:32] ...optimizers/trijoint.py.151: total_norm: 14.800685 [I 2022-10-28 00:49:38] ...trap/engines/engine.py.224: train: epoch 97 | batch 650/2304 [I 2022-10-28 00:49:38] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:28 [I 2022-10-28 00:49:38] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00248 [I 2022-10-28 00:49:38] ...trap/engines/engine.py.230: loss: 0.11157 [I 2022-10-28 00:49:38] ...optimizers/trijoint.py.151: total_norm: 13.795806 [I 2022-10-28 00:49:44] ...trap/engines/engine.py.224: train: epoch 97 | batch 660/2304 [I 2022-10-28 00:49:44] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:15:21 [I 2022-10-28 00:49:44] ...trap/engines/engine.py.229: process: 0.56706 | load: 0.00247 [I 2022-10-28 00:49:44] ...trap/engines/engine.py.230: loss: 0.15132 [I 2022-10-28 00:49:44] ...optimizers/trijoint.py.151: total_norm: 26.247717 [I 2022-10-28 00:49:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 670/2304 [I 2022-10-28 00:49:49] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:20 [I 2022-10-28 00:49:49] ...trap/engines/engine.py.229: process: 0.58139 | load: 0.00242 [I 2022-10-28 00:49:49] ...trap/engines/engine.py.230: loss: 0.12672 [I 2022-10-28 00:49:49] ...optimizers/trijoint.py.151: total_norm: 13.371416 [I 2022-10-28 00:49:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 680/2304 [I 2022-10-28 00:49:55] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:09 [I 2022-10-28 00:49:55] ...trap/engines/engine.py.229: process: 0.56406 | load: 0.00257 [I 2022-10-28 00:49:55] ...trap/engines/engine.py.230: loss: 0.15389 [I 2022-10-28 00:49:55] ...optimizers/trijoint.py.151: total_norm: 11.381999 [I 2022-10-28 00:50:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 690/2304 [I 2022-10-28 00:50:00] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:14:59 [I 2022-10-28 00:50:00] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00282 [I 2022-10-28 00:50:00] ...trap/engines/engine.py.230: loss: 0.15421 [I 2022-10-28 00:50:00] ...optimizers/trijoint.py.151: total_norm: 25.059788 [I 2022-10-28 00:50:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 700/2304 [I 2022-10-28 00:50:06] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:15:16 [I 2022-10-28 00:50:06] ...trap/engines/engine.py.229: process: 0.55863 | load: 0.00268 [I 2022-10-28 00:50:06] ...trap/engines/engine.py.230: loss: 0.12035 [I 2022-10-28 00:50:06] ...optimizers/trijoint.py.151: total_norm: 14.368577 [I 2022-10-28 00:50:12] ...trap/engines/engine.py.224: train: epoch 97 | batch 710/2304 [I 2022-10-28 00:50:12] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:14:53 [I 2022-10-28 00:50:12] ...trap/engines/engine.py.229: process: 0.54937 | load: 0.00244 [I 2022-10-28 00:50:12] ...trap/engines/engine.py.230: loss: 0.10239 [I 2022-10-28 00:50:12] ...optimizers/trijoint.py.151: total_norm: 20.593184 [I 2022-10-28 00:50:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 720/2304 [I 2022-10-28 00:50:17] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:53 [I 2022-10-28 00:50:17] ...trap/engines/engine.py.229: process: 0.57642 | load: 0.00247 [I 2022-10-28 00:50:17] ...trap/engines/engine.py.230: loss: 0.10551 [I 2022-10-28 00:50:17] ...optimizers/trijoint.py.151: total_norm: 16.037380 [I 2022-10-28 00:50:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 730/2304 [I 2022-10-28 00:50:23] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:14:35 [I 2022-10-28 00:50:23] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00246 [I 2022-10-28 00:50:23] ...trap/engines/engine.py.230: loss: 0.11430 [I 2022-10-28 00:50:23] ...optimizers/trijoint.py.151: total_norm: 25.674711 [I 2022-10-28 00:50:28] ...trap/engines/engine.py.224: train: epoch 97 | batch 740/2304 [I 2022-10-28 00:50:28] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:14:25 [I 2022-10-28 00:50:28] ...trap/engines/engine.py.229: process: 0.55275 | load: 0.00245 [I 2022-10-28 00:50:28] ...trap/engines/engine.py.230: loss: 0.17197 [I 2022-10-28 00:50:28] ...optimizers/trijoint.py.151: total_norm: 32.189510 [I 2022-10-28 00:50:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 750/2304 [I 2022-10-28 00:50:34] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:24 [I 2022-10-28 00:50:34] ...trap/engines/engine.py.229: process: 0.54544 | load: 0.00252 [I 2022-10-28 00:50:34] ...trap/engines/engine.py.230: loss: 0.09341 [I 2022-10-28 00:50:34] ...optimizers/trijoint.py.151: total_norm: 8.455606 [I 2022-10-28 00:50:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 760/2304 [I 2022-10-28 00:50:40] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:33 [I 2022-10-28 00:50:40] ...trap/engines/engine.py.229: process: 0.57894 | load: 0.00253 [I 2022-10-28 00:50:40] ...trap/engines/engine.py.230: loss: 0.12052 [I 2022-10-28 00:50:40] ...optimizers/trijoint.py.151: total_norm: 24.509790 [I 2022-10-28 00:50:45] ...trap/engines/engine.py.224: train: epoch 97 | batch 770/2304 [I 2022-10-28 00:50:45] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:19 [I 2022-10-28 00:50:45] ...trap/engines/engine.py.229: process: 0.54347 | load: 0.00246 [I 2022-10-28 00:50:45] ...trap/engines/engine.py.230: loss: 0.12343 [I 2022-10-28 00:50:45] ...optimizers/trijoint.py.151: total_norm: 27.776873 [I 2022-10-28 00:50:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 780/2304 [I 2022-10-28 00:50:51] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:09 [I 2022-10-28 00:50:51] ...trap/engines/engine.py.229: process: 0.55945 | load: 0.00250 [I 2022-10-28 00:50:51] ...trap/engines/engine.py.230: loss: 0.14039 [I 2022-10-28 00:50:51] ...optimizers/trijoint.py.151: total_norm: 16.498781 [I 2022-10-28 00:50:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 790/2304 [I 2022-10-28 00:50:57] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:14:09 [I 2022-10-28 00:50:57] ...trap/engines/engine.py.229: process: 0.54647 | load: 0.00240 [I 2022-10-28 00:50:57] ...trap/engines/engine.py.230: loss: 0.11134 [I 2022-10-28 00:50:57] ...optimizers/trijoint.py.151: total_norm: 18.298538 [I 2022-10-28 00:51:02] ...trap/engines/engine.py.224: train: epoch 97 | batch 800/2304 [I 2022-10-28 00:51:02] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:14:02 [I 2022-10-28 00:51:02] ...trap/engines/engine.py.229: process: 0.57145 | load: 0.00242 [I 2022-10-28 00:51:02] ...trap/engines/engine.py.230: loss: 0.08836 [I 2022-10-28 00:51:02] ...optimizers/trijoint.py.151: total_norm: 10.261167 [I 2022-10-28 00:51:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 810/2304 [I 2022-10-28 00:51:08] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:14:01 [I 2022-10-28 00:51:08] ...trap/engines/engine.py.229: process: 0.55192 | load: 0.00246 [I 2022-10-28 00:51:08] ...trap/engines/engine.py.230: loss: 0.23108 [I 2022-10-28 00:51:08] ...optimizers/trijoint.py.151: total_norm: 23.827084 [I 2022-10-28 00:51:14] ...trap/engines/engine.py.224: train: epoch 97 | batch 820/2304 [I 2022-10-28 00:51:14] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:14:00 [I 2022-10-28 00:51:14] ...trap/engines/engine.py.229: process: 0.56642 | load: 0.00246 [I 2022-10-28 00:51:14] ...trap/engines/engine.py.230: loss: 0.18375 [I 2022-10-28 00:51:14] ...optimizers/trijoint.py.151: total_norm: 35.226570 [I 2022-10-28 00:51:19] ...trap/engines/engine.py.224: train: epoch 97 | batch 830/2304 [I 2022-10-28 00:51:19] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:51 [I 2022-10-28 00:51:19] ...trap/engines/engine.py.229: process: 0.56326 | load: 0.00243 [I 2022-10-28 00:51:19] ...trap/engines/engine.py.230: loss: 0.09308 [I 2022-10-28 00:51:19] ...optimizers/trijoint.py.151: total_norm: 8.524974 [I 2022-10-28 00:51:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 840/2304 [I 2022-10-28 00:51:25] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:13:44 [I 2022-10-28 00:51:25] ...trap/engines/engine.py.229: process: 0.56983 | load: 0.00246 [I 2022-10-28 00:51:25] ...trap/engines/engine.py.230: loss: 0.14387 [I 2022-10-28 00:51:25] ...optimizers/trijoint.py.151: total_norm: 12.349195 [I 2022-10-28 00:51:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 850/2304 [I 2022-10-28 00:51:30] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:13:36 [I 2022-10-28 00:51:30] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00260 [I 2022-10-28 00:51:30] ...trap/engines/engine.py.230: loss: 0.11150 [I 2022-10-28 00:51:30] ...optimizers/trijoint.py.151: total_norm: 24.760178 [I 2022-10-28 00:51:36] ...trap/engines/engine.py.224: train: epoch 97 | batch 860/2304 [I 2022-10-28 00:51:36] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:13:24 [I 2022-10-28 00:51:36] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00256 [I 2022-10-28 00:51:36] ...trap/engines/engine.py.230: loss: 0.10750 [I 2022-10-28 00:51:36] ...optimizers/trijoint.py.151: total_norm: 10.897849 [I 2022-10-28 00:51:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 870/2304 [I 2022-10-28 00:51:42] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:26 [I 2022-10-28 00:51:42] ...trap/engines/engine.py.229: process: 0.57074 | load: 0.00255 [I 2022-10-28 00:51:42] ...trap/engines/engine.py.230: loss: 0.09690 [I 2022-10-28 00:51:42] ...optimizers/trijoint.py.151: total_norm: 11.765223 [I 2022-10-28 00:51:47] ...trap/engines/engine.py.224: train: epoch 97 | batch 880/2304 [I 2022-10-28 00:51:47] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:12 [I 2022-10-28 00:51:47] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00245 [I 2022-10-28 00:51:47] ...trap/engines/engine.py.230: loss: 0.09616 [I 2022-10-28 00:51:47] ...optimizers/trijoint.py.151: total_norm: 8.441657 [I 2022-10-28 00:51:53] ...trap/engines/engine.py.224: train: epoch 97 | batch 890/2304 [I 2022-10-28 00:51:53] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:13:12 [I 2022-10-28 00:51:53] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00248 [I 2022-10-28 00:51:53] ...trap/engines/engine.py.230: loss: 0.10588 [I 2022-10-28 00:51:53] ...optimizers/trijoint.py.151: total_norm: 13.350646 [I 2022-10-28 00:51:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 900/2304 [I 2022-10-28 00:51:58] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:06 [I 2022-10-28 00:51:58] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00244 [I 2022-10-28 00:51:58] ...trap/engines/engine.py.230: loss: 0.17467 [I 2022-10-28 00:51:58] ...optimizers/trijoint.py.151: total_norm: 35.347614 [I 2022-10-28 00:52:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 910/2304 [I 2022-10-28 00:52:04] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:13:00 [I 2022-10-28 00:52:04] ...trap/engines/engine.py.229: process: 0.56806 | load: 0.00256 [I 2022-10-28 00:52:04] ...trap/engines/engine.py.230: loss: 0.15365 [I 2022-10-28 00:52:04] ...optimizers/trijoint.py.151: total_norm: 17.533264 [I 2022-10-28 00:52:10] ...trap/engines/engine.py.224: train: epoch 97 | batch 920/2304 [I 2022-10-28 00:52:10] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:12:47 [I 2022-10-28 00:52:10] ...trap/engines/engine.py.229: process: 0.53899 | load: 0.00259 [I 2022-10-28 00:52:10] ...trap/engines/engine.py.230: loss: 0.12459 [I 2022-10-28 00:52:10] ...optimizers/trijoint.py.151: total_norm: 11.588631 [I 2022-10-28 00:52:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 930/2304 [I 2022-10-28 00:52:15] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:12:42 [I 2022-10-28 00:52:15] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00249 [I 2022-10-28 00:52:15] ...trap/engines/engine.py.230: loss: 0.11295 [I 2022-10-28 00:52:15] ...optimizers/trijoint.py.151: total_norm: 16.507460 [I 2022-10-28 00:52:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 940/2304 [I 2022-10-28 00:52:21] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:12:42 [I 2022-10-28 00:52:21] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00249 [I 2022-10-28 00:52:21] ...trap/engines/engine.py.230: loss: 0.10646 [I 2022-10-28 00:52:21] ...optimizers/trijoint.py.151: total_norm: 7.573944 [I 2022-10-28 00:52:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 950/2304 [I 2022-10-28 00:52:26] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:35 [I 2022-10-28 00:52:26] ...trap/engines/engine.py.229: process: 0.55501 | load: 0.00247 [I 2022-10-28 00:52:26] ...trap/engines/engine.py.230: loss: 0.14549 [I 2022-10-28 00:52:26] ...optimizers/trijoint.py.151: total_norm: 13.999655 [I 2022-10-28 00:52:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 960/2304 [I 2022-10-28 00:52:32] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:30 [I 2022-10-28 00:52:32] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00243 [I 2022-10-28 00:52:32] ...trap/engines/engine.py.230: loss: 0.17235 [I 2022-10-28 00:52:32] ...optimizers/trijoint.py.151: total_norm: 9.639034 [I 2022-10-28 00:52:38] ...trap/engines/engine.py.224: train: epoch 97 | batch 970/2304 [I 2022-10-28 00:52:38] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:21 [I 2022-10-28 00:52:38] ...trap/engines/engine.py.229: process: 0.55348 | load: 0.00265 [I 2022-10-28 00:52:38] ...trap/engines/engine.py.230: loss: 0.09914 [I 2022-10-28 00:52:38] ...optimizers/trijoint.py.151: total_norm: 15.417597 [I 2022-10-28 00:52:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 980/2304 [I 2022-10-28 00:52:43] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:12:21 [I 2022-10-28 00:52:43] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00246 [I 2022-10-28 00:52:43] ...trap/engines/engine.py.230: loss: 0.12183 [I 2022-10-28 00:52:43] ...optimizers/trijoint.py.151: total_norm: 32.440010 [I 2022-10-28 00:52:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 990/2304 [I 2022-10-28 00:52:49] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:24 [I 2022-10-28 00:52:49] ...trap/engines/engine.py.229: process: 0.56712 | load: 0.00242 [I 2022-10-28 00:52:49] ...trap/engines/engine.py.230: loss: 0.10780 [I 2022-10-28 00:52:49] ...optimizers/trijoint.py.151: total_norm: 16.759035 [I 2022-10-28 00:52:54] ...trap/engines/engine.py.224: train: epoch 97 | batch 1000/2304 [I 2022-10-28 00:52:54] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:11 [I 2022-10-28 00:52:55] ...trap/engines/engine.py.229: process: 0.54527 | load: 0.00239 [I 2022-10-28 00:52:55] ...trap/engines/engine.py.230: loss: 0.10677 [I 2022-10-28 00:52:55] ...optimizers/trijoint.py.151: total_norm: 25.430285 [I 2022-10-28 00:53:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 1010/2304 [I 2022-10-28 00:53:00] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:12:09 [I 2022-10-28 00:53:00] ...trap/engines/engine.py.229: process: 0.57176 | load: 0.00263 [I 2022-10-28 00:53:00] ...trap/engines/engine.py.230: loss: 0.09780 [I 2022-10-28 00:53:00] ...optimizers/trijoint.py.151: total_norm: 22.328127 [I 2022-10-28 00:53:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 1020/2304 [I 2022-10-28 00:53:06] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:11:57 [I 2022-10-28 00:53:06] ...trap/engines/engine.py.229: process: 0.56311 | load: 0.00259 [I 2022-10-28 00:53:06] ...trap/engines/engine.py.230: loss: 0.13412 [I 2022-10-28 00:53:06] ...optimizers/trijoint.py.151: total_norm: 18.673477 [I 2022-10-28 00:53:11] ...trap/engines/engine.py.224: train: epoch 97 | batch 1030/2304 [I 2022-10-28 00:53:11] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:11:55 [I 2022-10-28 00:53:11] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00262 [I 2022-10-28 00:53:11] ...trap/engines/engine.py.230: loss: 0.12936 [I 2022-10-28 00:53:11] ...optimizers/trijoint.py.151: total_norm: 12.955419 [I 2022-10-28 00:53:17] ...trap/engines/engine.py.224: train: epoch 97 | batch 1040/2304 [I 2022-10-28 00:53:17] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:11:43 [I 2022-10-28 00:53:17] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00248 [I 2022-10-28 00:53:17] ...trap/engines/engine.py.230: loss: 0.15455 [I 2022-10-28 00:53:17] ...optimizers/trijoint.py.151: total_norm: 33.270370 [I 2022-10-28 00:53:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 1050/2304 [I 2022-10-28 00:53:23] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:41 [I 2022-10-28 00:53:23] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00274 [I 2022-10-28 00:53:23] ...trap/engines/engine.py.230: loss: 0.10463 [I 2022-10-28 00:53:23] ...optimizers/trijoint.py.151: total_norm: 10.562957 [I 2022-10-28 00:53:28] ...trap/engines/engine.py.224: train: epoch 97 | batch 1060/2304 [I 2022-10-28 00:53:28] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:37 [I 2022-10-28 00:53:28] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00243 [I 2022-10-28 00:53:28] ...trap/engines/engine.py.230: loss: 0.11189 [I 2022-10-28 00:53:28] ...optimizers/trijoint.py.151: total_norm: 26.804241 [I 2022-10-28 00:53:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 1070/2304 [I 2022-10-28 00:53:34] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:29 [I 2022-10-28 00:53:34] ...trap/engines/engine.py.229: process: 0.56628 | load: 0.00242 [I 2022-10-28 00:53:34] ...trap/engines/engine.py.230: loss: 0.14058 [I 2022-10-28 00:53:34] ...optimizers/trijoint.py.151: total_norm: 13.334931 [I 2022-10-28 00:53:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 1080/2304 [I 2022-10-28 00:53:39] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:25 [I 2022-10-28 00:53:39] ...trap/engines/engine.py.229: process: 0.56391 | load: 0.00269 [I 2022-10-28 00:53:39] ...trap/engines/engine.py.230: loss: 0.10036 [I 2022-10-28 00:53:39] ...optimizers/trijoint.py.151: total_norm: 27.718012 [I 2022-10-28 00:53:45] ...trap/engines/engine.py.224: train: epoch 97 | batch 1090/2304 [I 2022-10-28 00:53:45] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:18 [I 2022-10-28 00:53:45] ...trap/engines/engine.py.229: process: 0.55480 | load: 0.00247 [I 2022-10-28 00:53:45] ...trap/engines/engine.py.230: loss: 0.13051 [I 2022-10-28 00:53:45] ...optimizers/trijoint.py.151: total_norm: 18.488518 [I 2022-10-28 00:53:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 1100/2304 [I 2022-10-28 00:53:51] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:10 [I 2022-10-28 00:53:51] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00247 [I 2022-10-28 00:53:51] ...trap/engines/engine.py.230: loss: 0.14254 [I 2022-10-28 00:53:51] ...optimizers/trijoint.py.151: total_norm: 13.680228 [I 2022-10-28 00:53:56] ...trap/engines/engine.py.224: train: epoch 97 | batch 1110/2304 [I 2022-10-28 00:53:56] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:00 [I 2022-10-28 00:53:56] ...trap/engines/engine.py.229: process: 0.55128 | load: 0.00245 [I 2022-10-28 00:53:56] ...trap/engines/engine.py.230: loss: 0.08453 [I 2022-10-28 00:53:56] ...optimizers/trijoint.py.151: total_norm: 7.876912 [I 2022-10-28 00:54:02] ...trap/engines/engine.py.224: train: epoch 97 | batch 1120/2304 [I 2022-10-28 00:54:02] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:10:58 [I 2022-10-28 00:54:02] ...trap/engines/engine.py.229: process: 0.56925 | load: 0.00246 [I 2022-10-28 00:54:02] ...trap/engines/engine.py.230: loss: 0.08454 [I 2022-10-28 00:54:02] ...optimizers/trijoint.py.151: total_norm: 13.614736 [I 2022-10-28 00:54:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 1130/2304 [I 2022-10-28 00:54:07] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:10:51 [I 2022-10-28 00:54:07] ...trap/engines/engine.py.229: process: 0.55351 | load: 0.00246 [I 2022-10-28 00:54:07] ...trap/engines/engine.py.230: loss: 0.11212 [I 2022-10-28 00:54:07] ...optimizers/trijoint.py.151: total_norm: 15.945368 [I 2022-10-28 00:54:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 1140/2304 [I 2022-10-28 00:54:13] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:52 [I 2022-10-28 00:54:13] ...trap/engines/engine.py.229: process: 0.57258 | load: 0.00241 [I 2022-10-28 00:54:13] ...trap/engines/engine.py.230: loss: 0.24506 [I 2022-10-28 00:54:13] ...optimizers/trijoint.py.151: total_norm: 29.829247 [I 2022-10-28 00:54:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 1150/2304 [I 2022-10-28 00:54:18] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:41 [I 2022-10-28 00:54:18] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00259 [I 2022-10-28 00:54:18] ...trap/engines/engine.py.230: loss: 0.10268 [I 2022-10-28 00:54:18] ...optimizers/trijoint.py.151: total_norm: 10.263928 [I 2022-10-28 00:54:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 1160/2304 [I 2022-10-28 00:54:24] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:36 [I 2022-10-28 00:54:24] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00244 [I 2022-10-28 00:54:24] ...trap/engines/engine.py.230: loss: 0.08715 [I 2022-10-28 00:54:24] ...optimizers/trijoint.py.151: total_norm: 9.930197 [I 2022-10-28 00:54:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 1170/2304 [I 2022-10-28 00:54:30] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:37 [I 2022-10-28 00:54:30] ...trap/engines/engine.py.229: process: 0.56691 | load: 0.00241 [I 2022-10-28 00:54:30] ...trap/engines/engine.py.230: loss: 0.08681 [I 2022-10-28 00:54:30] ...optimizers/trijoint.py.151: total_norm: 8.104421 [I 2022-10-28 00:54:35] ...trap/engines/engine.py.224: train: epoch 97 | batch 1180/2304 [I 2022-10-28 00:54:35] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:30 [I 2022-10-28 00:54:35] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00243 [I 2022-10-28 00:54:35] ...trap/engines/engine.py.230: loss: 0.11211 [I 2022-10-28 00:54:35] ...optimizers/trijoint.py.151: total_norm: 15.619670 [I 2022-10-28 00:54:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 1190/2304 [I 2022-10-28 00:54:41] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:21 [I 2022-10-28 00:54:41] ...trap/engines/engine.py.229: process: 0.55073 | load: 0.00242 [I 2022-10-28 00:54:41] ...trap/engines/engine.py.230: loss: 0.16678 [I 2022-10-28 00:54:41] ...optimizers/trijoint.py.151: total_norm: 16.824394 [I 2022-10-28 00:54:46] ...trap/engines/engine.py.224: train: epoch 97 | batch 1200/2304 [I 2022-10-28 00:54:46] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:15 [I 2022-10-28 00:54:46] ...trap/engines/engine.py.229: process: 0.53974 | load: 0.00241 [I 2022-10-28 00:54:46] ...trap/engines/engine.py.230: loss: 0.18825 [I 2022-10-28 00:54:46] ...optimizers/trijoint.py.151: total_norm: 19.137466 [I 2022-10-28 00:54:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 1210/2304 [I 2022-10-28 00:54:52] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:10 [I 2022-10-28 00:54:52] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00245 [I 2022-10-28 00:54:52] ...trap/engines/engine.py.230: loss: 0.10294 [I 2022-10-28 00:54:52] ...optimizers/trijoint.py.151: total_norm: 7.353308 [I 2022-10-28 00:54:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 1220/2304 [I 2022-10-28 00:54:58] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:05 [I 2022-10-28 00:54:58] ...trap/engines/engine.py.229: process: 0.56118 | load: 0.00243 [I 2022-10-28 00:54:58] ...trap/engines/engine.py.230: loss: 0.15938 [I 2022-10-28 00:54:58] ...optimizers/trijoint.py.151: total_norm: 10.333196 [I 2022-10-28 00:55:03] ...trap/engines/engine.py.224: train: epoch 97 | batch 1230/2304 [I 2022-10-28 00:55:03] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:10:05 [I 2022-10-28 00:55:03] ...trap/engines/engine.py.229: process: 0.55656 | load: 0.00241 [I 2022-10-28 00:55:03] ...trap/engines/engine.py.230: loss: 0.11639 [I 2022-10-28 00:55:03] ...optimizers/trijoint.py.151: total_norm: 15.456065 [I 2022-10-28 00:55:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 1240/2304 [I 2022-10-28 00:55:09] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:09:52 [I 2022-10-28 00:55:09] ...trap/engines/engine.py.229: process: 0.56604 | load: 0.00253 [I 2022-10-28 00:55:09] ...trap/engines/engine.py.230: loss: 0.11580 [I 2022-10-28 00:55:09] ...optimizers/trijoint.py.151: total_norm: 19.385502 [I 2022-10-28 00:55:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 1250/2304 [I 2022-10-28 00:55:15] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:09:52 [I 2022-10-28 00:55:15] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00252 [I 2022-10-28 00:55:15] ...trap/engines/engine.py.230: loss: 0.15671 [I 2022-10-28 00:55:15] ...optimizers/trijoint.py.151: total_norm: 38.820339 [I 2022-10-28 00:55:20] ...trap/engines/engine.py.224: train: epoch 97 | batch 1260/2304 [I 2022-10-28 00:55:20] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:09:46 [I 2022-10-28 00:55:20] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00251 [I 2022-10-28 00:55:20] ...trap/engines/engine.py.230: loss: 0.16220 [I 2022-10-28 00:55:20] ...optimizers/trijoint.py.151: total_norm: 31.135147 [I 2022-10-28 00:55:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 1270/2304 [I 2022-10-28 00:55:26] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:37 [I 2022-10-28 00:55:26] ...trap/engines/engine.py.229: process: 0.55048 | load: 0.00253 [I 2022-10-28 00:55:26] ...trap/engines/engine.py.230: loss: 0.20723 [I 2022-10-28 00:55:26] ...optimizers/trijoint.py.151: total_norm: 26.175690 [I 2022-10-28 00:55:31] ...trap/engines/engine.py.224: train: epoch 97 | batch 1280/2304 [I 2022-10-28 00:55:31] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:31 [I 2022-10-28 00:55:31] ...trap/engines/engine.py.229: process: 0.55202 | load: 0.00252 [I 2022-10-28 00:55:31] ...trap/engines/engine.py.230: loss: 0.15742 [I 2022-10-28 00:55:31] ...optimizers/trijoint.py.151: total_norm: 22.709986 [I 2022-10-28 00:55:37] ...trap/engines/engine.py.224: train: epoch 97 | batch 1290/2304 [I 2022-10-28 00:55:37] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:29 [I 2022-10-28 00:55:37] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00246 [I 2022-10-28 00:55:37] ...trap/engines/engine.py.230: loss: 0.09627 [I 2022-10-28 00:55:37] ...optimizers/trijoint.py.151: total_norm: 16.213091 [I 2022-10-28 00:55:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 1300/2304 [I 2022-10-28 00:55:43] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:21 [I 2022-10-28 00:55:43] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00248 [I 2022-10-28 00:55:43] ...trap/engines/engine.py.230: loss: 0.20320 [I 2022-10-28 00:55:43] ...optimizers/trijoint.py.151: total_norm: 14.382612 [I 2022-10-28 00:55:48] ...trap/engines/engine.py.224: train: epoch 97 | batch 1310/2304 [I 2022-10-28 00:55:48] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:17 [I 2022-10-28 00:55:48] ...trap/engines/engine.py.229: process: 0.56500 | load: 0.00247 [I 2022-10-28 00:55:48] ...trap/engines/engine.py.230: loss: 0.16306 [I 2022-10-28 00:55:48] ...optimizers/trijoint.py.151: total_norm: 16.054661 [I 2022-10-28 00:55:54] ...trap/engines/engine.py.224: train: epoch 97 | batch 1320/2304 [I 2022-10-28 00:55:54] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:13 [I 2022-10-28 00:55:54] ...trap/engines/engine.py.229: process: 0.56366 | load: 0.00243 [I 2022-10-28 00:55:54] ...trap/engines/engine.py.230: loss: 0.15407 [I 2022-10-28 00:55:54] ...optimizers/trijoint.py.151: total_norm: 25.151917 [I 2022-10-28 00:55:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 1330/2304 [I 2022-10-28 00:55:59] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:05 [I 2022-10-28 00:55:59] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00253 [I 2022-10-28 00:55:59] ...trap/engines/engine.py.230: loss: 0.10267 [I 2022-10-28 00:55:59] ...optimizers/trijoint.py.151: total_norm: 28.366217 [I 2022-10-28 00:56:05] ...trap/engines/engine.py.224: train: epoch 97 | batch 1340/2304 [I 2022-10-28 00:56:05] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:08:57 [I 2022-10-28 00:56:05] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00251 [I 2022-10-28 00:56:05] ...trap/engines/engine.py.230: loss: 0.10453 [I 2022-10-28 00:56:05] ...optimizers/trijoint.py.151: total_norm: 20.633482 [I 2022-10-28 00:56:11] ...trap/engines/engine.py.224: train: epoch 97 | batch 1350/2304 [I 2022-10-28 00:56:11] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:08:57 [I 2022-10-28 00:56:11] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00250 [I 2022-10-28 00:56:11] ...trap/engines/engine.py.230: loss: 0.10197 [I 2022-10-28 00:56:11] ...optimizers/trijoint.py.151: total_norm: 8.300514 [I 2022-10-28 00:56:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 1360/2304 [I 2022-10-28 00:56:16] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:08:50 [I 2022-10-28 00:56:16] ...trap/engines/engine.py.229: process: 0.56445 | load: 0.00252 [I 2022-10-28 00:56:16] ...trap/engines/engine.py.230: loss: 0.09365 [I 2022-10-28 00:56:16] ...optimizers/trijoint.py.151: total_norm: 18.752356 [I 2022-10-28 00:56:22] ...trap/engines/engine.py.224: train: epoch 97 | batch 1370/2304 [I 2022-10-28 00:56:22] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:43 [I 2022-10-28 00:56:22] ...trap/engines/engine.py.229: process: 0.55672 | load: 0.00245 [I 2022-10-28 00:56:22] ...trap/engines/engine.py.230: loss: 0.10686 [I 2022-10-28 00:56:22] ...optimizers/trijoint.py.151: total_norm: 11.382956 [I 2022-10-28 00:56:28] ...trap/engines/engine.py.224: train: epoch 97 | batch 1380/2304 [I 2022-10-28 00:56:28] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:37 [I 2022-10-28 00:56:28] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00244 [I 2022-10-28 00:56:28] ...trap/engines/engine.py.230: loss: 0.09897 [I 2022-10-28 00:56:28] ...optimizers/trijoint.py.151: total_norm: 27.747835 [I 2022-10-28 00:56:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 1390/2304 [I 2022-10-28 00:56:33] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:38 [I 2022-10-28 00:56:33] ...trap/engines/engine.py.229: process: 0.58925 | load: 0.00243 [I 2022-10-28 00:56:33] ...trap/engines/engine.py.230: loss: 0.07990 [I 2022-10-28 00:56:33] ...optimizers/trijoint.py.151: total_norm: 11.418412 [I 2022-10-28 00:56:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 1400/2304 [I 2022-10-28 00:56:39] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:26 [I 2022-10-28 00:56:39] ...trap/engines/engine.py.229: process: 0.56222 | load: 0.00243 [I 2022-10-28 00:56:39] ...trap/engines/engine.py.230: loss: 0.09727 [I 2022-10-28 00:56:39] ...optimizers/trijoint.py.151: total_norm: 14.547984 [I 2022-10-28 00:56:45] ...trap/engines/engine.py.224: train: epoch 97 | batch 1410/2304 [I 2022-10-28 00:56:45] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:33 [I 2022-10-28 00:56:45] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00243 [I 2022-10-28 00:56:45] ...trap/engines/engine.py.230: loss: 0.09280 [I 2022-10-28 00:56:45] ...optimizers/trijoint.py.151: total_norm: 10.025206 [I 2022-10-28 00:56:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 1420/2304 [I 2022-10-28 00:56:50] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:20 [I 2022-10-28 00:56:50] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00242 [I 2022-10-28 00:56:50] ...trap/engines/engine.py.230: loss: 0.08612 [I 2022-10-28 00:56:50] ...optimizers/trijoint.py.151: total_norm: 9.605210 [I 2022-10-28 00:56:56] ...trap/engines/engine.py.224: train: epoch 97 | batch 1430/2304 [I 2022-10-28 00:56:56] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:08 [I 2022-10-28 00:56:56] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00244 [I 2022-10-28 00:56:56] ...trap/engines/engine.py.230: loss: 0.12162 [I 2022-10-28 00:56:56] ...optimizers/trijoint.py.151: total_norm: 14.107021 [I 2022-10-28 00:57:01] ...trap/engines/engine.py.224: train: epoch 97 | batch 1440/2304 [I 2022-10-28 00:57:01] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:00 [I 2022-10-28 00:57:01] ...trap/engines/engine.py.229: process: 0.55460 | load: 0.00242 [I 2022-10-28 00:57:01] ...trap/engines/engine.py.230: loss: 0.08539 [I 2022-10-28 00:57:01] ...optimizers/trijoint.py.151: total_norm: 18.088224 [I 2022-10-28 00:57:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 1450/2304 [I 2022-10-28 00:57:07] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:07:59 [I 2022-10-28 00:57:07] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00274 [I 2022-10-28 00:57:07] ...trap/engines/engine.py.230: loss: 0.14807 [I 2022-10-28 00:57:07] ...optimizers/trijoint.py.151: total_norm: 17.053997 [I 2022-10-28 00:57:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 1460/2304 [I 2022-10-28 00:57:13] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:07:53 [I 2022-10-28 00:57:13] ...trap/engines/engine.py.229: process: 0.55008 | load: 0.00257 [I 2022-10-28 00:57:13] ...trap/engines/engine.py.230: loss: 0.20755 [I 2022-10-28 00:57:13] ...optimizers/trijoint.py.151: total_norm: 49.347012 [I 2022-10-28 00:57:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 1470/2304 [I 2022-10-28 00:57:18] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:07:45 [I 2022-10-28 00:57:18] ...trap/engines/engine.py.229: process: 0.55290 | load: 0.00251 [I 2022-10-28 00:57:18] ...trap/engines/engine.py.230: loss: 0.13433 [I 2022-10-28 00:57:18] ...optimizers/trijoint.py.151: total_norm: 10.841496 [I 2022-10-28 00:57:24] ...trap/engines/engine.py.224: train: epoch 97 | batch 1480/2304 [I 2022-10-28 00:57:24] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:44 [I 2022-10-28 00:57:24] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00258 [I 2022-10-28 00:57:24] ...trap/engines/engine.py.230: loss: 0.09070 [I 2022-10-28 00:57:24] ...optimizers/trijoint.py.151: total_norm: 17.935400 [I 2022-10-28 00:57:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 1490/2304 [I 2022-10-28 00:57:30] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:39 [I 2022-10-28 00:57:30] ...trap/engines/engine.py.229: process: 0.56278 | load: 0.00258 [I 2022-10-28 00:57:30] ...trap/engines/engine.py.230: loss: 0.10019 [I 2022-10-28 00:57:30] ...optimizers/trijoint.py.151: total_norm: 8.272942 [I 2022-10-28 00:57:35] ...trap/engines/engine.py.224: train: epoch 97 | batch 1500/2304 [I 2022-10-28 00:57:35] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:32 [I 2022-10-28 00:57:35] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00253 [I 2022-10-28 00:57:35] ...trap/engines/engine.py.230: loss: 0.13874 [I 2022-10-28 00:57:35] ...optimizers/trijoint.py.151: total_norm: 17.596991 [I 2022-10-28 00:57:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 1510/2304 [I 2022-10-28 00:57:41] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:25 [I 2022-10-28 00:57:41] ...trap/engines/engine.py.229: process: 0.56094 | load: 0.00257 [I 2022-10-28 00:57:41] ...trap/engines/engine.py.230: loss: 0.11458 [I 2022-10-28 00:57:41] ...optimizers/trijoint.py.151: total_norm: 25.781660 [I 2022-10-28 00:57:46] ...trap/engines/engine.py.224: train: epoch 97 | batch 1520/2304 [I 2022-10-28 00:57:46] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:17 [I 2022-10-28 00:57:46] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00247 [I 2022-10-28 00:57:46] ...trap/engines/engine.py.230: loss: 0.10661 [I 2022-10-28 00:57:46] ...optimizers/trijoint.py.151: total_norm: 10.831984 [I 2022-10-28 00:57:52] ...trap/engines/engine.py.224: train: epoch 97 | batch 1530/2304 [I 2022-10-28 00:57:52] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:08 [I 2022-10-28 00:57:52] ...trap/engines/engine.py.229: process: 0.54870 | load: 0.00243 [I 2022-10-28 00:57:52] ...trap/engines/engine.py.230: loss: 0.12636 [I 2022-10-28 00:57:52] ...optimizers/trijoint.py.151: total_norm: 24.991970 [I 2022-10-28 00:57:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 1540/2304 [I 2022-10-28 00:57:58] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:09 [I 2022-10-28 00:57:58] ...trap/engines/engine.py.229: process: 0.58411 | load: 0.00248 [I 2022-10-28 00:57:58] ...trap/engines/engine.py.230: loss: 0.11300 [I 2022-10-28 00:57:58] ...optimizers/trijoint.py.151: total_norm: 8.234155 [I 2022-10-28 00:58:03] ...trap/engines/engine.py.224: train: epoch 97 | batch 1550/2304 [I 2022-10-28 00:58:03] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:09 [I 2022-10-28 00:58:03] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00251 [I 2022-10-28 00:58:03] ...trap/engines/engine.py.230: loss: 0.11698 [I 2022-10-28 00:58:03] ...optimizers/trijoint.py.151: total_norm: 11.431779 [I 2022-10-28 00:58:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 1560/2304 [I 2022-10-28 00:58:09] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:06:58 [I 2022-10-28 00:58:09] ...trap/engines/engine.py.229: process: 0.56944 | load: 0.00243 [I 2022-10-28 00:58:09] ...trap/engines/engine.py.230: loss: 0.17065 [I 2022-10-28 00:58:09] ...optimizers/trijoint.py.151: total_norm: 39.392300 [I 2022-10-28 00:58:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 1570/2304 [I 2022-10-28 00:58:15] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:06:49 [I 2022-10-28 00:58:15] ...trap/engines/engine.py.229: process: 0.56238 | load: 0.00272 [I 2022-10-28 00:58:15] ...trap/engines/engine.py.230: loss: 0.13413 [I 2022-10-28 00:58:15] ...optimizers/trijoint.py.151: total_norm: 24.651522 [I 2022-10-28 00:58:20] ...trap/engines/engine.py.224: train: epoch 97 | batch 1580/2304 [I 2022-10-28 00:58:20] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:06:42 [I 2022-10-28 00:58:20] ...trap/engines/engine.py.229: process: 0.54873 | load: 0.00248 [I 2022-10-28 00:58:20] ...trap/engines/engine.py.230: loss: 0.09606 [I 2022-10-28 00:58:20] ...optimizers/trijoint.py.151: total_norm: 11.977412 [I 2022-10-28 00:58:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 1590/2304 [I 2022-10-28 00:58:26] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:36 [I 2022-10-28 00:58:26] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00242 [I 2022-10-28 00:58:26] ...trap/engines/engine.py.230: loss: 0.10661 [I 2022-10-28 00:58:26] ...optimizers/trijoint.py.151: total_norm: 9.958234 [I 2022-10-28 00:58:31] ...trap/engines/engine.py.224: train: epoch 97 | batch 1600/2304 [I 2022-10-28 00:58:31] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:31 [I 2022-10-28 00:58:31] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00244 [I 2022-10-28 00:58:31] ...trap/engines/engine.py.230: loss: 0.15712 [I 2022-10-28 00:58:31] ...optimizers/trijoint.py.151: total_norm: 16.586535 [I 2022-10-28 00:58:37] ...trap/engines/engine.py.224: train: epoch 97 | batch 1610/2304 [I 2022-10-28 00:58:37] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:22 [I 2022-10-28 00:58:37] ...trap/engines/engine.py.229: process: 0.53675 | load: 0.00282 [I 2022-10-28 00:58:37] ...trap/engines/engine.py.230: loss: 0.15879 [I 2022-10-28 00:58:37] ...optimizers/trijoint.py.151: total_norm: 16.094353 [I 2022-10-28 00:58:42] ...trap/engines/engine.py.224: train: epoch 97 | batch 1620/2304 [I 2022-10-28 00:58:42] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:19 [I 2022-10-28 00:58:42] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00243 [I 2022-10-28 00:58:42] ...trap/engines/engine.py.230: loss: 0.12213 [I 2022-10-28 00:58:42] ...optimizers/trijoint.py.151: total_norm: 9.700501 [I 2022-10-28 00:58:48] ...trap/engines/engine.py.224: train: epoch 97 | batch 1630/2304 [I 2022-10-28 00:58:48] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:13 [I 2022-10-28 00:58:48] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00244 [I 2022-10-28 00:58:48] ...trap/engines/engine.py.230: loss: 0.08724 [I 2022-10-28 00:58:48] ...optimizers/trijoint.py.151: total_norm: 10.374344 [I 2022-10-28 00:58:53] ...trap/engines/engine.py.224: train: epoch 97 | batch 1640/2304 [I 2022-10-28 00:58:53] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:07 [I 2022-10-28 00:58:53] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00243 [I 2022-10-28 00:58:53] ...trap/engines/engine.py.230: loss: 0.15138 [I 2022-10-28 00:58:53] ...optimizers/trijoint.py.151: total_norm: 21.997997 [I 2022-10-28 00:58:59] ...trap/engines/engine.py.224: train: epoch 97 | batch 1650/2304 [I 2022-10-28 00:58:59] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:06 [I 2022-10-28 00:58:59] ...trap/engines/engine.py.229: process: 0.56690 | load: 0.00245 [I 2022-10-28 00:58:59] ...trap/engines/engine.py.230: loss: 0.09468 [I 2022-10-28 00:58:59] ...optimizers/trijoint.py.151: total_norm: 9.705657 [I 2022-10-28 00:59:05] ...trap/engines/engine.py.224: train: epoch 97 | batch 1660/2304 [I 2022-10-28 00:59:05] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:05:59 [I 2022-10-28 00:59:05] ...trap/engines/engine.py.229: process: 0.55075 | load: 0.00242 [I 2022-10-28 00:59:05] ...trap/engines/engine.py.230: loss: 0.13905 [I 2022-10-28 00:59:05] ...optimizers/trijoint.py.151: total_norm: 27.291573 [I 2022-10-28 00:59:10] ...trap/engines/engine.py.224: train: epoch 97 | batch 1670/2304 [I 2022-10-28 00:59:10] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:05:55 [I 2022-10-28 00:59:10] ...trap/engines/engine.py.229: process: 0.55274 | load: 0.00243 [I 2022-10-28 00:59:10] ...trap/engines/engine.py.230: loss: 0.11893 [I 2022-10-28 00:59:10] ...optimizers/trijoint.py.151: total_norm: 11.115823 [I 2022-10-28 00:59:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 1680/2304 [I 2022-10-28 00:59:16] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:05:47 [I 2022-10-28 00:59:16] ...trap/engines/engine.py.229: process: 0.55635 | load: 0.00249 [I 2022-10-28 00:59:16] ...trap/engines/engine.py.230: loss: 0.12861 [I 2022-10-28 00:59:16] ...optimizers/trijoint.py.151: total_norm: 23.846451 [I 2022-10-28 00:59:21] ...trap/engines/engine.py.224: train: epoch 97 | batch 1690/2304 [I 2022-10-28 00:59:21] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:43 [I 2022-10-28 00:59:21] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00246 [I 2022-10-28 00:59:21] ...trap/engines/engine.py.230: loss: 0.17129 [I 2022-10-28 00:59:21] ...optimizers/trijoint.py.151: total_norm: 17.927717 [I 2022-10-28 00:59:27] ...trap/engines/engine.py.224: train: epoch 97 | batch 1700/2304 [I 2022-10-28 00:59:27] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:40 [I 2022-10-28 00:59:27] ...trap/engines/engine.py.229: process: 0.57815 | load: 0.00247 [I 2022-10-28 00:59:27] ...trap/engines/engine.py.230: loss: 0.12811 [I 2022-10-28 00:59:27] ...optimizers/trijoint.py.151: total_norm: 21.070595 [I 2022-10-28 00:59:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 1710/2304 [I 2022-10-28 00:59:33] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:32 [I 2022-10-28 00:59:33] ...trap/engines/engine.py.229: process: 0.56127 | load: 0.00252 [I 2022-10-28 00:59:33] ...trap/engines/engine.py.230: loss: 0.17010 [I 2022-10-28 00:59:33] ...optimizers/trijoint.py.151: total_norm: 19.050810 [I 2022-10-28 00:59:38] ...trap/engines/engine.py.224: train: epoch 97 | batch 1720/2304 [I 2022-10-28 00:59:38] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:24 [I 2022-10-28 00:59:38] ...trap/engines/engine.py.229: process: 0.55093 | load: 0.00246 [I 2022-10-28 00:59:38] ...trap/engines/engine.py.230: loss: 0.11722 [I 2022-10-28 00:59:38] ...optimizers/trijoint.py.151: total_norm: 12.325130 [I 2022-10-28 00:59:44] ...trap/engines/engine.py.224: train: epoch 97 | batch 1730/2304 [I 2022-10-28 00:59:44] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:21 [I 2022-10-28 00:59:44] ...trap/engines/engine.py.229: process: 0.56188 | load: 0.00244 [I 2022-10-28 00:59:44] ...trap/engines/engine.py.230: loss: 0.16588 [I 2022-10-28 00:59:44] ...optimizers/trijoint.py.151: total_norm: 17.991774 [I 2022-10-28 00:59:49] ...trap/engines/engine.py.224: train: epoch 97 | batch 1740/2304 [I 2022-10-28 00:59:49] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:14 [I 2022-10-28 00:59:49] ...trap/engines/engine.py.229: process: 0.56126 | load: 0.00244 [I 2022-10-28 00:59:49] ...trap/engines/engine.py.230: loss: 0.09602 [I 2022-10-28 00:59:49] ...optimizers/trijoint.py.151: total_norm: 10.249004 [I 2022-10-28 00:59:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 1750/2304 [I 2022-10-28 00:59:55] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:09 [I 2022-10-28 00:59:55] ...trap/engines/engine.py.229: process: 0.55266 | load: 0.00244 [I 2022-10-28 00:59:55] ...trap/engines/engine.py.230: loss: 0.14717 [I 2022-10-28 00:59:55] ...optimizers/trijoint.py.151: total_norm: 23.668715 [I 2022-10-28 01:00:01] ...trap/engines/engine.py.224: train: epoch 97 | batch 1760/2304 [I 2022-10-28 01:00:01] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:03 [I 2022-10-28 01:00:01] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00250 [I 2022-10-28 01:00:01] ...trap/engines/engine.py.230: loss: 0.18320 [I 2022-10-28 01:00:01] ...optimizers/trijoint.py.151: total_norm: 21.932968 [I 2022-10-28 01:00:06] ...trap/engines/engine.py.224: train: epoch 97 | batch 1770/2304 [I 2022-10-28 01:00:06] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:04:55 [I 2022-10-28 01:00:06] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00242 [I 2022-10-28 01:00:06] ...trap/engines/engine.py.230: loss: 0.10148 [I 2022-10-28 01:00:06] ...optimizers/trijoint.py.151: total_norm: 12.320297 [I 2022-10-28 01:00:12] ...trap/engines/engine.py.224: train: epoch 97 | batch 1780/2304 [I 2022-10-28 01:00:12] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:04:55 [I 2022-10-28 01:00:12] ...trap/engines/engine.py.229: process: 0.56991 | load: 0.00244 [I 2022-10-28 01:00:12] ...trap/engines/engine.py.230: loss: 0.09686 [I 2022-10-28 01:00:12] ...optimizers/trijoint.py.151: total_norm: 8.691586 [I 2022-10-28 01:00:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 1790/2304 [I 2022-10-28 01:00:18] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:45 [I 2022-10-28 01:00:18] ...trap/engines/engine.py.229: process: 0.54801 | load: 0.00254 [I 2022-10-28 01:00:18] ...trap/engines/engine.py.230: loss: 0.10642 [I 2022-10-28 01:00:18] ...optimizers/trijoint.py.151: total_norm: 9.605741 [I 2022-10-28 01:00:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 1800/2304 [I 2022-10-28 01:00:23] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:42 [I 2022-10-28 01:00:23] ...trap/engines/engine.py.229: process: 0.58207 | load: 0.00251 [I 2022-10-28 01:00:23] ...trap/engines/engine.py.230: loss: 0.08664 [I 2022-10-28 01:00:23] ...optimizers/trijoint.py.151: total_norm: 8.844942 [I 2022-10-28 01:00:29] ...trap/engines/engine.py.224: train: epoch 97 | batch 1810/2304 [I 2022-10-28 01:00:29] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:32 [I 2022-10-28 01:00:29] ...trap/engines/engine.py.229: process: 0.54549 | load: 0.00247 [I 2022-10-28 01:00:29] ...trap/engines/engine.py.230: loss: 0.10540 [I 2022-10-28 01:00:29] ...optimizers/trijoint.py.151: total_norm: 11.135616 [I 2022-10-28 01:00:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 1820/2304 [I 2022-10-28 01:00:34] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:04:27 [I 2022-10-28 01:00:34] ...trap/engines/engine.py.229: process: 0.54679 | load: 0.00247 [I 2022-10-28 01:00:34] ...trap/engines/engine.py.230: loss: 0.14859 [I 2022-10-28 01:00:34] ...optimizers/trijoint.py.151: total_norm: 33.676865 [I 2022-10-28 01:00:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 1830/2304 [I 2022-10-28 01:00:40] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:24 [I 2022-10-28 01:00:40] ...trap/engines/engine.py.229: process: 0.56149 | load: 0.00261 [I 2022-10-28 01:00:40] ...trap/engines/engine.py.230: loss: 0.11394 [I 2022-10-28 01:00:40] ...optimizers/trijoint.py.151: total_norm: 10.994845 [I 2022-10-28 01:00:45] ...trap/engines/engine.py.224: train: epoch 97 | batch 1840/2304 [I 2022-10-28 01:00:45] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:17 [I 2022-10-28 01:00:45] ...trap/engines/engine.py.229: process: 0.56787 | load: 0.00244 [I 2022-10-28 01:00:45] ...trap/engines/engine.py.230: loss: 0.10046 [I 2022-10-28 01:00:45] ...optimizers/trijoint.py.151: total_norm: 8.143633 [I 2022-10-28 01:00:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 1850/2304 [I 2022-10-28 01:00:51] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:11 [I 2022-10-28 01:00:51] ...trap/engines/engine.py.229: process: 0.55174 | load: 0.00244 [I 2022-10-28 01:00:51] ...trap/engines/engine.py.230: loss: 0.16823 [I 2022-10-28 01:00:51] ...optimizers/trijoint.py.151: total_norm: 24.131435 [I 2022-10-28 01:00:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 1860/2304 [I 2022-10-28 01:00:57] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:08 [I 2022-10-28 01:00:57] ...trap/engines/engine.py.229: process: 0.55319 | load: 0.00251 [I 2022-10-28 01:00:57] ...trap/engines/engine.py.230: loss: 0.13026 [I 2022-10-28 01:00:57] ...optimizers/trijoint.py.151: total_norm: 32.100780 [I 2022-10-28 01:01:02] ...trap/engines/engine.py.224: train: epoch 97 | batch 1870/2304 [I 2022-10-28 01:01:02] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:04:02 [I 2022-10-28 01:01:02] ...trap/engines/engine.py.229: process: 0.55025 | load: 0.00243 [I 2022-10-28 01:01:02] ...trap/engines/engine.py.230: loss: 0.09138 [I 2022-10-28 01:01:02] ...optimizers/trijoint.py.151: total_norm: 11.413393 [I 2022-10-28 01:01:08] ...trap/engines/engine.py.224: train: epoch 97 | batch 1880/2304 [I 2022-10-28 01:01:08] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:03:55 [I 2022-10-28 01:01:08] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00246 [I 2022-10-28 01:01:08] ...trap/engines/engine.py.230: loss: 0.12384 [I 2022-10-28 01:01:08] ...optimizers/trijoint.py.151: total_norm: 21.765295 [I 2022-10-28 01:01:13] ...trap/engines/engine.py.224: train: epoch 97 | batch 1890/2304 [I 2022-10-28 01:01:13] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:03:53 [I 2022-10-28 01:01:13] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00242 [I 2022-10-28 01:01:13] ...trap/engines/engine.py.230: loss: 0.10406 [I 2022-10-28 01:01:13] ...optimizers/trijoint.py.151: total_norm: 34.183937 [I 2022-10-28 01:01:19] ...trap/engines/engine.py.224: train: epoch 97 | batch 1900/2304 [I 2022-10-28 01:01:19] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:03:45 [I 2022-10-28 01:01:19] ...trap/engines/engine.py.229: process: 0.56234 | load: 0.00252 [I 2022-10-28 01:01:19] ...trap/engines/engine.py.230: loss: 0.10728 [I 2022-10-28 01:01:19] ...optimizers/trijoint.py.151: total_norm: 10.842721 [I 2022-10-28 01:01:25] ...trap/engines/engine.py.224: train: epoch 97 | batch 1910/2304 [I 2022-10-28 01:01:25] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:39 [I 2022-10-28 01:01:25] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00244 [I 2022-10-28 01:01:25] ...trap/engines/engine.py.230: loss: 0.13597 [I 2022-10-28 01:01:25] ...optimizers/trijoint.py.151: total_norm: 13.904160 [I 2022-10-28 01:01:30] ...trap/engines/engine.py.224: train: epoch 97 | batch 1920/2304 [I 2022-10-28 01:01:30] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:34 [I 2022-10-28 01:01:30] ...trap/engines/engine.py.229: process: 0.54367 | load: 0.00242 [I 2022-10-28 01:01:30] ...trap/engines/engine.py.230: loss: 0.10249 [I 2022-10-28 01:01:30] ...optimizers/trijoint.py.151: total_norm: 18.180422 [I 2022-10-28 01:01:36] ...trap/engines/engine.py.224: train: epoch 97 | batch 1930/2304 [I 2022-10-28 01:01:36] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:29 [I 2022-10-28 01:01:36] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00243 [I 2022-10-28 01:01:36] ...trap/engines/engine.py.230: loss: 0.09607 [I 2022-10-28 01:01:36] ...optimizers/trijoint.py.151: total_norm: 19.800520 [I 2022-10-28 01:01:41] ...trap/engines/engine.py.224: train: epoch 97 | batch 1940/2304 [I 2022-10-28 01:01:41] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:23 [I 2022-10-28 01:01:41] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00242 [I 2022-10-28 01:01:41] ...trap/engines/engine.py.230: loss: 0.10956 [I 2022-10-28 01:01:41] ...optimizers/trijoint.py.151: total_norm: 15.048064 [I 2022-10-28 01:01:47] ...trap/engines/engine.py.224: train: epoch 97 | batch 1950/2304 [I 2022-10-28 01:01:47] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:16 [I 2022-10-28 01:01:47] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00250 [I 2022-10-28 01:01:47] ...trap/engines/engine.py.230: loss: 0.11764 [I 2022-10-28 01:01:47] ...optimizers/trijoint.py.151: total_norm: 12.943097 [I 2022-10-28 01:01:53] ...trap/engines/engine.py.224: train: epoch 97 | batch 1960/2304 [I 2022-10-28 01:01:53] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:12 [I 2022-10-28 01:01:53] ...trap/engines/engine.py.229: process: 0.55830 | load: 0.00247 [I 2022-10-28 01:01:53] ...trap/engines/engine.py.230: loss: 0.13990 [I 2022-10-28 01:01:53] ...optimizers/trijoint.py.151: total_norm: 15.323511 [I 2022-10-28 01:01:58] ...trap/engines/engine.py.224: train: epoch 97 | batch 1970/2304 [I 2022-10-28 01:01:58] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:05 [I 2022-10-28 01:01:58] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00247 [I 2022-10-28 01:01:58] ...trap/engines/engine.py.230: loss: 0.14870 [I 2022-10-28 01:01:58] ...optimizers/trijoint.py.151: total_norm: 9.956223 [I 2022-10-28 01:02:04] ...trap/engines/engine.py.224: train: epoch 97 | batch 1980/2304 [I 2022-10-28 01:02:04] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:00 [I 2022-10-28 01:02:04] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00245 [I 2022-10-28 01:02:04] ...trap/engines/engine.py.230: loss: 0.13826 [I 2022-10-28 01:02:04] ...optimizers/trijoint.py.151: total_norm: 13.738470 [I 2022-10-28 01:02:09] ...trap/engines/engine.py.224: train: epoch 97 | batch 1990/2304 [I 2022-10-28 01:02:09] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:02:55 [I 2022-10-28 01:02:09] ...trap/engines/engine.py.229: process: 0.56044 | load: 0.00248 [I 2022-10-28 01:02:09] ...trap/engines/engine.py.230: loss: 0.11280 [I 2022-10-28 01:02:09] ...optimizers/trijoint.py.151: total_norm: 37.271675 [I 2022-10-28 01:02:15] ...trap/engines/engine.py.224: train: epoch 97 | batch 2000/2304 [I 2022-10-28 01:02:15] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:02:49 [I 2022-10-28 01:02:15] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00245 [I 2022-10-28 01:02:15] ...trap/engines/engine.py.230: loss: 0.14695 [I 2022-10-28 01:02:15] ...optimizers/trijoint.py.151: total_norm: 26.163336 [I 2022-10-28 01:02:20] ...trap/engines/engine.py.224: train: epoch 97 | batch 2010/2304 [I 2022-10-28 01:02:20] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:02:44 [I 2022-10-28 01:02:20] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00243 [I 2022-10-28 01:02:20] ...trap/engines/engine.py.230: loss: 0.10270 [I 2022-10-28 01:02:20] ...optimizers/trijoint.py.151: total_norm: 12.829541 [I 2022-10-28 01:02:26] ...trap/engines/engine.py.224: train: epoch 97 | batch 2020/2304 [I 2022-10-28 01:02:26] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:38 [I 2022-10-28 01:02:26] ...trap/engines/engine.py.229: process: 0.55466 | load: 0.00246 [I 2022-10-28 01:02:26] ...trap/engines/engine.py.230: loss: 0.11894 [I 2022-10-28 01:02:26] ...optimizers/trijoint.py.151: total_norm: 27.596027 [I 2022-10-28 01:02:32] ...trap/engines/engine.py.224: train: epoch 97 | batch 2030/2304 [I 2022-10-28 01:02:32] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:32 [I 2022-10-28 01:02:32] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00244 [I 2022-10-28 01:02:32] ...trap/engines/engine.py.230: loss: 0.16294 [I 2022-10-28 01:02:32] ...optimizers/trijoint.py.151: total_norm: 41.398624 [I 2022-10-28 01:02:37] ...trap/engines/engine.py.224: train: epoch 97 | batch 2040/2304 [I 2022-10-28 01:02:37] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:02:26 [I 2022-10-28 01:02:37] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00242 [I 2022-10-28 01:02:37] ...trap/engines/engine.py.230: loss: 0.11409 [I 2022-10-28 01:02:37] ...optimizers/trijoint.py.151: total_norm: 7.445372 [I 2022-10-28 01:02:43] ...trap/engines/engine.py.224: train: epoch 97 | batch 2050/2304 [I 2022-10-28 01:02:43] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:21 [I 2022-10-28 01:02:43] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00249 [I 2022-10-28 01:02:43] ...trap/engines/engine.py.230: loss: 0.13019 [I 2022-10-28 01:02:43] ...optimizers/trijoint.py.151: total_norm: 23.922590 [I 2022-10-28 01:02:48] ...trap/engines/engine.py.224: train: epoch 97 | batch 2060/2304 [I 2022-10-28 01:02:48] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:16 [I 2022-10-28 01:02:48] ...trap/engines/engine.py.229: process: 0.55038 | load: 0.00244 [I 2022-10-28 01:02:48] ...trap/engines/engine.py.230: loss: 0.15565 [I 2022-10-28 01:02:48] ...optimizers/trijoint.py.151: total_norm: 47.876457 [I 2022-10-28 01:02:54] ...trap/engines/engine.py.224: train: epoch 97 | batch 2070/2304 [I 2022-10-28 01:02:54] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:10 [I 2022-10-28 01:02:54] ...trap/engines/engine.py.229: process: 0.55499 | load: 0.00245 [I 2022-10-28 01:02:54] ...trap/engines/engine.py.230: loss: 0.08926 [I 2022-10-28 01:02:54] ...optimizers/trijoint.py.151: total_norm: 22.326263 [I 2022-10-28 01:03:00] ...trap/engines/engine.py.224: train: epoch 97 | batch 2080/2304 [I 2022-10-28 01:03:00] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:04 [I 2022-10-28 01:03:00] ...trap/engines/engine.py.229: process: 0.55488 | load: 0.00250 [I 2022-10-28 01:03:00] ...trap/engines/engine.py.230: loss: 0.11285 [I 2022-10-28 01:03:00] ...optimizers/trijoint.py.151: total_norm: 18.427010 [I 2022-10-28 01:03:05] ...trap/engines/engine.py.224: train: epoch 97 | batch 2090/2304 [I 2022-10-28 01:03:05] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:01:59 [I 2022-10-28 01:03:05] ...trap/engines/engine.py.229: process: 0.55034 | load: 0.00244 [I 2022-10-28 01:03:05] ...trap/engines/engine.py.230: loss: 0.13392 [I 2022-10-28 01:03:05] ...optimizers/trijoint.py.151: total_norm: 26.651886 [I 2022-10-28 01:03:11] ...trap/engines/engine.py.224: train: epoch 97 | batch 2100/2304 [I 2022-10-28 01:03:11] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:01:53 [I 2022-10-28 01:03:11] ...trap/engines/engine.py.229: process: 0.55474 | load: 0.00243 [I 2022-10-28 01:03:11] ...trap/engines/engine.py.230: loss: 0.10801 [I 2022-10-28 01:03:11] ...optimizers/trijoint.py.151: total_norm: 15.442267 [I 2022-10-28 01:03:16] ...trap/engines/engine.py.224: train: epoch 97 | batch 2110/2304 [I 2022-10-28 01:03:16] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:01:48 [I 2022-10-28 01:03:16] ...trap/engines/engine.py.229: process: 0.54990 | load: 0.00243 [I 2022-10-28 01:03:16] ...trap/engines/engine.py.230: loss: 0.10580 [I 2022-10-28 01:03:16] ...optimizers/trijoint.py.151: total_norm: 14.380017 [I 2022-10-28 01:03:22] ...trap/engines/engine.py.224: train: epoch 97 | batch 2120/2304 [I 2022-10-28 01:03:22] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:42 [I 2022-10-28 01:03:22] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00242 [I 2022-10-28 01:03:22] ...trap/engines/engine.py.230: loss: 0.10481 [I 2022-10-28 01:03:22] ...optimizers/trijoint.py.151: total_norm: 10.413606 [I 2022-10-28 01:03:28] ...trap/engines/engine.py.224: train: epoch 97 | batch 2130/2304 [I 2022-10-28 01:03:28] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:37 [I 2022-10-28 01:03:28] ...trap/engines/engine.py.229: process: 0.56626 | load: 0.00244 [I 2022-10-28 01:03:28] ...trap/engines/engine.py.230: loss: 0.10207 [I 2022-10-28 01:03:28] ...optimizers/trijoint.py.151: total_norm: 14.244396 [I 2022-10-28 01:03:33] ...trap/engines/engine.py.224: train: epoch 97 | batch 2140/2304 [I 2022-10-28 01:03:33] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:30 [I 2022-10-28 01:03:33] ...trap/engines/engine.py.229: process: 0.54346 | load: 0.00240 [I 2022-10-28 01:03:33] ...trap/engines/engine.py.230: loss: 0.15291 [I 2022-10-28 01:03:33] ...optimizers/trijoint.py.151: total_norm: 19.008387 [I 2022-10-28 01:03:39] ...trap/engines/engine.py.224: train: epoch 97 | batch 2150/2304 [I 2022-10-28 01:03:39] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:25 [I 2022-10-28 01:03:39] ...trap/engines/engine.py.229: process: 0.55072 | load: 0.00241 [I 2022-10-28 01:03:39] ...trap/engines/engine.py.230: loss: 0.11240 [I 2022-10-28 01:03:39] ...optimizers/trijoint.py.151: total_norm: 38.680794 [I 2022-10-28 01:03:44] ...trap/engines/engine.py.224: train: epoch 97 | batch 2160/2304 [I 2022-10-28 01:03:44] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:19 [I 2022-10-28 01:03:44] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00241 [I 2022-10-28 01:03:44] ...trap/engines/engine.py.230: loss: 0.10768 [I 2022-10-28 01:03:44] ...optimizers/trijoint.py.151: total_norm: 40.463242 [I 2022-10-28 01:03:50] ...trap/engines/engine.py.224: train: epoch 97 | batch 2170/2304 [I 2022-10-28 01:03:50] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:13 [I 2022-10-28 01:03:50] ...trap/engines/engine.py.229: process: 0.55180 | load: 0.00239 [I 2022-10-28 01:03:50] ...trap/engines/engine.py.230: loss: 0.11394 [I 2022-10-28 01:03:50] ...optimizers/trijoint.py.151: total_norm: 15.415693 [I 2022-10-28 01:03:55] ...trap/engines/engine.py.224: train: epoch 97 | batch 2180/2304 [I 2022-10-28 01:03:55] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:08 [I 2022-10-28 01:03:55] ...trap/engines/engine.py.229: process: 0.55542 | load: 0.00280 [I 2022-10-28 01:03:55] ...trap/engines/engine.py.230: loss: 0.13538 [I 2022-10-28 01:03:55] ...optimizers/trijoint.py.151: total_norm: 15.219886 [I 2022-10-28 01:04:01] ...trap/engines/engine.py.224: train: epoch 97 | batch 2190/2304 [I 2022-10-28 01:04:01] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:01:03 [I 2022-10-28 01:04:01] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00241 [I 2022-10-28 01:04:01] ...trap/engines/engine.py.230: loss: 0.15847 [I 2022-10-28 01:04:01] ...optimizers/trijoint.py.151: total_norm: 18.408108 [I 2022-10-28 01:04:07] ...trap/engines/engine.py.224: train: epoch 97 | batch 2200/2304 [I 2022-10-28 01:04:07] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:00:57 [I 2022-10-28 01:04:07] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00241 [I 2022-10-28 01:04:07] ...trap/engines/engine.py.230: loss: 0.10707 [I 2022-10-28 01:04:07] ...optimizers/trijoint.py.151: total_norm: 15.917646 [I 2022-10-28 01:04:12] ...trap/engines/engine.py.224: train: epoch 97 | batch 2210/2304 [I 2022-10-28 01:04:12] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:00:52 [I 2022-10-28 01:04:12] ...trap/engines/engine.py.229: process: 0.55008 | load: 0.00269 [I 2022-10-28 01:04:12] ...trap/engines/engine.py.230: loss: 0.13534 [I 2022-10-28 01:04:12] ...optimizers/trijoint.py.151: total_norm: 7.085263 [I 2022-10-28 01:04:18] ...trap/engines/engine.py.224: train: epoch 97 | batch 2220/2304 [I 2022-10-28 01:04:18] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:00:46 [I 2022-10-28 01:04:18] ...trap/engines/engine.py.229: process: 0.53676 | load: 0.00243 [I 2022-10-28 01:04:18] ...trap/engines/engine.py.230: loss: 0.12799 [I 2022-10-28 01:04:18] ...optimizers/trijoint.py.151: total_norm: 9.557811 [I 2022-10-28 01:04:23] ...trap/engines/engine.py.224: train: epoch 97 | batch 2230/2304 [I 2022-10-28 01:04:23] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:41 [I 2022-10-28 01:04:23] ...trap/engines/engine.py.229: process: 0.56223 | load: 0.00244 [I 2022-10-28 01:04:23] ...trap/engines/engine.py.230: loss: 0.09818 [I 2022-10-28 01:04:23] ...optimizers/trijoint.py.151: total_norm: 23.325590 [I 2022-10-28 01:04:29] ...trap/engines/engine.py.224: train: epoch 97 | batch 2240/2304 [I 2022-10-28 01:04:29] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:35 [I 2022-10-28 01:04:29] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00245 [I 2022-10-28 01:04:29] ...trap/engines/engine.py.230: loss: 0.13618 [I 2022-10-28 01:04:29] ...optimizers/trijoint.py.151: total_norm: 11.989151 [I 2022-10-28 01:04:34] ...trap/engines/engine.py.224: train: epoch 97 | batch 2250/2304 [I 2022-10-28 01:04:34] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:30 [I 2022-10-28 01:04:34] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00245 [I 2022-10-28 01:04:34] ...trap/engines/engine.py.230: loss: 0.09684 [I 2022-10-28 01:04:34] ...optimizers/trijoint.py.151: total_norm: 12.185630 [I 2022-10-28 01:04:40] ...trap/engines/engine.py.224: train: epoch 97 | batch 2260/2304 [I 2022-10-28 01:04:40] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:24 [I 2022-10-28 01:04:40] ...trap/engines/engine.py.229: process: 0.56510 | load: 0.00244 [I 2022-10-28 01:04:40] ...trap/engines/engine.py.230: loss: 0.13782 [I 2022-10-28 01:04:40] ...optimizers/trijoint.py.151: total_norm: 23.568417 [I 2022-10-28 01:04:46] ...trap/engines/engine.py.224: train: epoch 97 | batch 2270/2304 [I 2022-10-28 01:04:46] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:19 [I 2022-10-28 01:04:46] ...trap/engines/engine.py.229: process: 0.56159 | load: 0.00255 [I 2022-10-28 01:04:46] ...trap/engines/engine.py.230: loss: 0.13613 [I 2022-10-28 01:04:46] ...optimizers/trijoint.py.151: total_norm: 10.466132 [I 2022-10-28 01:04:51] ...trap/engines/engine.py.224: train: epoch 97 | batch 2280/2304 [I 2022-10-28 01:04:51] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:13 [I 2022-10-28 01:04:51] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00257 [I 2022-10-28 01:04:51] ...trap/engines/engine.py.230: loss: 0.10688 [I 2022-10-28 01:04:51] ...optimizers/trijoint.py.151: total_norm: 13.762269 [I 2022-10-28 01:04:57] ...trap/engines/engine.py.224: train: epoch 97 | batch 2290/2304 [I 2022-10-28 01:04:57] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:07 [I 2022-10-28 01:04:57] ...trap/engines/engine.py.229: process: 0.54429 | load: 0.00255 [I 2022-10-28 01:04:57] ...trap/engines/engine.py.230: loss: 0.14980 [I 2022-10-28 01:04:57] ...optimizers/trijoint.py.151: total_norm: 22.567574 [I 2022-10-28 01:05:02] ...trap/engines/engine.py.224: train: epoch 97 | batch 2300/2304 [I 2022-10-28 01:05:02] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:02 [I 2022-10-28 01:05:02] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00015 [I 2022-10-28 01:05:02] ...trap/engines/engine.py.230: loss: 0.21179 [I 2022-10-28 01:05:02] ...optimizers/trijoint.py.151: total_norm: 11.441532 [I 2022-10-28 01:05:05] ...trap/engines/engine.py.224: train: epoch 97 | batch 2304/2304 [I 2022-10-28 01:05:05] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:00 [I 2022-10-28 01:05:05] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00017 [I 2022-10-28 01:05:05] ...trap/engines/engine.py.230: loss: 0.10036 [I 2022-10-28 01:05:05] ...optimizers/trijoint.py.151: total_norm: 21.474354 [S 2022-10-28 01:05:06] ...trap/engines/engine.py.129: train_epoch.epoch: 97 [S 2022-10-28 01:05:06] ...trap/engines/engine.py.129: train_epoch.loss: 0.12639051889293623 [I 2022-10-28 01:05:06] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 01:05:07] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 97 [I 2022-10-28 01:05:07] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 01:05:09] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 01:05:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 01:05:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 01:05:13] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 01:05:13] ...trap/engines/engine.py.321: eval: epoch 97 | batch 0/510 [I 2022-10-28 01:05:13] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:47:24 [I 2022-10-28 01:05:13] ...trap/engines/engine.py.326: process: 5.57739 | load: 4.37200 [I 2022-10-28 01:05:16] ...trap/engines/engine.py.321: eval: epoch 97 | batch 10/510 [I 2022-10-28 01:05:16] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:28 [I 2022-10-28 01:05:16] ...trap/engines/engine.py.326: process: 0.32794 | load: 0.00020 [I 2022-10-28 01:05:20] ...trap/engines/engine.py.321: eval: epoch 97 | batch 20/510 [I 2022-10-28 01:05:20] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:17 [I 2022-10-28 01:05:20] ...trap/engines/engine.py.326: process: 0.36167 | load: 0.00017 [I 2022-10-28 01:05:23] ...trap/engines/engine.py.321: eval: epoch 97 | batch 30/510 [I 2022-10-28 01:05:23] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:45 [I 2022-10-28 01:05:23] ...trap/engines/engine.py.326: process: 0.35891 | load: 0.00016 [I 2022-10-28 01:05:26] ...trap/engines/engine.py.321: eval: epoch 97 | batch 40/510 [I 2022-10-28 01:05:26] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:37 [I 2022-10-28 01:05:26] ...trap/engines/engine.py.326: process: 0.33101 | load: 0.00018 [I 2022-10-28 01:05:30] ...trap/engines/engine.py.321: eval: epoch 97 | batch 50/510 [I 2022-10-28 01:05:30] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:34 [I 2022-10-28 01:05:30] ...trap/engines/engine.py.326: process: 0.33041 | load: 0.00017 [I 2022-10-28 01:05:33] ...trap/engines/engine.py.321: eval: epoch 97 | batch 60/510 [I 2022-10-28 01:05:33] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:28 [I 2022-10-28 01:05:33] ...trap/engines/engine.py.326: process: 0.32848 | load: 0.00016 [I 2022-10-28 01:05:36] ...trap/engines/engine.py.321: eval: epoch 97 | batch 70/510 [I 2022-10-28 01:05:36] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:25 [I 2022-10-28 01:05:36] ...trap/engines/engine.py.326: process: 0.33024 | load: 0.00017 [I 2022-10-28 01:05:40] ...trap/engines/engine.py.321: eval: epoch 97 | batch 80/510 [I 2022-10-28 01:05:40] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:21 [I 2022-10-28 01:05:40] ...trap/engines/engine.py.326: process: 0.32502 | load: 0.00016 [I 2022-10-28 01:05:43] ...trap/engines/engine.py.321: eval: epoch 97 | batch 90/510 [I 2022-10-28 01:05:43] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:18 [I 2022-10-28 01:05:43] ...trap/engines/engine.py.326: process: 0.32600 | load: 0.00018 [I 2022-10-28 01:05:46] ...trap/engines/engine.py.321: eval: epoch 97 | batch 100/510 [I 2022-10-28 01:05:46] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-28 01:05:46] ...trap/engines/engine.py.326: process: 0.30321 | load: 0.00019 [I 2022-10-28 01:05:49] ...trap/engines/engine.py.321: eval: epoch 97 | batch 110/510 [I 2022-10-28 01:05:49] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-28 01:05:49] ...trap/engines/engine.py.326: process: 0.29416 | load: 0.00015 [I 2022-10-28 01:05:52] ...trap/engines/engine.py.321: eval: epoch 97 | batch 120/510 [I 2022-10-28 01:05:52] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:54 [I 2022-10-28 01:05:52] ...trap/engines/engine.py.326: process: 0.30535 | load: 0.00019 [I 2022-10-28 01:05:55] ...trap/engines/engine.py.321: eval: epoch 97 | batch 130/510 [I 2022-10-28 01:05:55] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:50 [I 2022-10-28 01:05:55] ...trap/engines/engine.py.326: process: 0.28785 | load: 0.00018 [I 2022-10-28 01:05:58] ...trap/engines/engine.py.321: eval: epoch 97 | batch 140/510 [I 2022-10-28 01:05:58] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-28 01:05:58] ...trap/engines/engine.py.326: process: 0.28861 | load: 0.00014 [I 2022-10-28 01:06:01] ...trap/engines/engine.py.321: eval: epoch 97 | batch 150/510 [I 2022-10-28 01:06:01] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:44 [I 2022-10-28 01:06:01] ...trap/engines/engine.py.326: process: 0.29006 | load: 0.00014 [I 2022-10-28 01:06:04] ...trap/engines/engine.py.321: eval: epoch 97 | batch 160/510 [I 2022-10-28 01:06:04] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-28 01:06:04] ...trap/engines/engine.py.326: process: 0.28555 | load: 0.00020 [I 2022-10-28 01:06:07] ...trap/engines/engine.py.321: eval: epoch 97 | batch 170/510 [I 2022-10-28 01:06:07] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 01:06:07] ...trap/engines/engine.py.326: process: 0.30043 | load: 0.00014 [I 2022-10-28 01:06:09] ...trap/engines/engine.py.321: eval: epoch 97 | batch 180/510 [I 2022-10-28 01:06:09] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-28 01:06:09] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00015 [I 2022-10-28 01:06:12] ...trap/engines/engine.py.321: eval: epoch 97 | batch 190/510 [I 2022-10-28 01:06:12] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-28 01:06:12] ...trap/engines/engine.py.326: process: 0.28727 | load: 0.00016 [I 2022-10-28 01:06:15] ...trap/engines/engine.py.321: eval: epoch 97 | batch 200/510 [I 2022-10-28 01:06:15] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:28 [I 2022-10-28 01:06:15] ...trap/engines/engine.py.326: process: 0.28651 | load: 0.00016 [I 2022-10-28 01:06:18] ...trap/engines/engine.py.321: eval: epoch 97 | batch 210/510 [I 2022-10-28 01:06:18] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:25 [I 2022-10-28 01:06:18] ...trap/engines/engine.py.326: process: 0.28393 | load: 0.00014 [I 2022-10-28 01:06:21] ...trap/engines/engine.py.321: eval: epoch 97 | batch 220/510 [I 2022-10-28 01:06:21] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-28 01:06:21] ...trap/engines/engine.py.326: process: 0.28583 | load: 0.00015 [I 2022-10-28 01:06:24] ...trap/engines/engine.py.321: eval: epoch 97 | batch 230/510 [I 2022-10-28 01:06:24] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-28 01:06:24] ...trap/engines/engine.py.326: process: 0.28660 | load: 0.00014 [I 2022-10-28 01:06:27] ...trap/engines/engine.py.321: eval: epoch 97 | batch 240/510 [I 2022-10-28 01:06:27] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 01:06:27] ...trap/engines/engine.py.326: process: 0.29063 | load: 0.00015 [I 2022-10-28 01:06:30] ...trap/engines/engine.py.321: eval: epoch 97 | batch 250/510 [I 2022-10-28 01:06:30] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-28 01:06:30] ...trap/engines/engine.py.326: process: 0.29042 | load: 0.00023 [I 2022-10-28 01:06:33] ...trap/engines/engine.py.321: eval: epoch 97 | batch 260/510 [I 2022-10-28 01:06:33] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:13 [I 2022-10-28 01:06:33] ...trap/engines/engine.py.326: process: 0.29702 | load: 0.00017 [I 2022-10-28 01:06:35] ...trap/engines/engine.py.321: eval: epoch 97 | batch 270/510 [I 2022-10-28 01:06:35] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-28 01:06:35] ...trap/engines/engine.py.326: process: 0.28970 | load: 0.00015 [I 2022-10-28 01:06:38] ...trap/engines/engine.py.321: eval: epoch 97 | batch 280/510 [I 2022-10-28 01:06:38] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 01:06:38] ...trap/engines/engine.py.326: process: 0.29162 | load: 0.00018 [I 2022-10-28 01:06:41] ...trap/engines/engine.py.321: eval: epoch 97 | batch 290/510 [I 2022-10-28 01:06:41] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-28 01:06:41] ...trap/engines/engine.py.326: process: 0.28713 | load: 0.00016 [I 2022-10-28 01:06:44] ...trap/engines/engine.py.321: eval: epoch 97 | batch 300/510 [I 2022-10-28 01:06:44] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:01 [I 2022-10-28 01:06:44] ...trap/engines/engine.py.326: process: 0.28139 | load: 0.00015 [I 2022-10-28 01:06:47] ...trap/engines/engine.py.321: eval: epoch 97 | batch 310/510 [I 2022-10-28 01:06:47] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 01:06:47] ...trap/engines/engine.py.326: process: 0.28940 | load: 0.00015 [I 2022-10-28 01:06:50] ...trap/engines/engine.py.321: eval: epoch 97 | batch 320/510 [I 2022-10-28 01:06:50] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 01:06:50] ...trap/engines/engine.py.326: process: 0.28675 | load: 0.00021 [I 2022-10-28 01:06:53] ...trap/engines/engine.py.321: eval: epoch 97 | batch 330/510 [I 2022-10-28 01:06:53] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-28 01:06:53] ...trap/engines/engine.py.326: process: 0.29034 | load: 0.00015 [I 2022-10-28 01:06:56] ...trap/engines/engine.py.321: eval: epoch 97 | batch 340/510 [I 2022-10-28 01:06:56] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-28 01:06:56] ...trap/engines/engine.py.326: process: 0.28804 | load: 0.00019 [I 2022-10-28 01:06:59] ...trap/engines/engine.py.321: eval: epoch 97 | batch 350/510 [I 2022-10-28 01:06:59] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:45 [I 2022-10-28 01:06:59] ...trap/engines/engine.py.326: process: 0.28664 | load: 0.00015 [I 2022-10-28 01:07:01] ...trap/engines/engine.py.321: eval: epoch 97 | batch 360/510 [I 2022-10-28 01:07:01] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 01:07:01] ...trap/engines/engine.py.326: process: 0.28878 | load: 0.00016 [I 2022-10-28 01:07:04] ...trap/engines/engine.py.321: eval: epoch 97 | batch 370/510 [I 2022-10-28 01:07:04] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-28 01:07:04] ...trap/engines/engine.py.326: process: 0.29045 | load: 0.00015 [I 2022-10-28 01:07:07] ...trap/engines/engine.py.321: eval: epoch 97 | batch 380/510 [I 2022-10-28 01:07:07] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-28 01:07:07] ...trap/engines/engine.py.326: process: 0.28846 | load: 0.00015 [I 2022-10-28 01:07:10] ...trap/engines/engine.py.321: eval: epoch 97 | batch 390/510 [I 2022-10-28 01:07:10] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 01:07:10] ...trap/engines/engine.py.326: process: 0.28680 | load: 0.00015 [I 2022-10-28 01:07:13] ...trap/engines/engine.py.321: eval: epoch 97 | batch 400/510 [I 2022-10-28 01:07:13] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 01:07:13] ...trap/engines/engine.py.326: process: 0.29136 | load: 0.00016 [I 2022-10-28 01:07:16] ...trap/engines/engine.py.321: eval: epoch 97 | batch 410/510 [I 2022-10-28 01:07:16] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 01:07:16] ...trap/engines/engine.py.326: process: 0.28359 | load: 0.00015 [I 2022-10-28 01:07:19] ...trap/engines/engine.py.321: eval: epoch 97 | batch 420/510 [I 2022-10-28 01:07:19] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 01:07:19] ...trap/engines/engine.py.326: process: 0.28820 | load: 0.00016 [I 2022-10-28 01:07:22] ...trap/engines/engine.py.321: eval: epoch 97 | batch 430/510 [I 2022-10-28 01:07:22] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 01:07:22] ...trap/engines/engine.py.326: process: 0.28598 | load: 0.00037 [I 2022-10-28 01:07:24] ...trap/engines/engine.py.321: eval: epoch 97 | batch 440/510 [I 2022-10-28 01:07:24] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 01:07:24] ...trap/engines/engine.py.326: process: 0.29289 | load: 0.00015 [I 2022-10-28 01:07:27] ...trap/engines/engine.py.321: eval: epoch 97 | batch 450/510 [I 2022-10-28 01:07:27] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-28 01:07:27] ...trap/engines/engine.py.326: process: 0.28843 | load: 0.00016 [I 2022-10-28 01:07:30] ...trap/engines/engine.py.321: eval: epoch 97 | batch 460/510 [I 2022-10-28 01:07:30] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-28 01:07:30] ...trap/engines/engine.py.326: process: 0.28296 | load: 0.00018 [I 2022-10-28 01:07:33] ...trap/engines/engine.py.321: eval: epoch 97 | batch 470/510 [I 2022-10-28 01:07:33] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-28 01:07:33] ...trap/engines/engine.py.326: process: 0.28735 | load: 0.00015 [I 2022-10-28 01:07:36] ...trap/engines/engine.py.321: eval: epoch 97 | batch 480/510 [I 2022-10-28 01:07:36] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 01:07:36] ...trap/engines/engine.py.326: process: 0.29113 | load: 0.00017 [I 2022-10-28 01:07:39] ...trap/engines/engine.py.321: eval: epoch 97 | batch 490/510 [I 2022-10-28 01:07:39] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 01:07:39] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00015 [I 2022-10-28 01:07:42] ...trap/engines/engine.py.321: eval: epoch 97 | batch 500/510 [I 2022-10-28 01:07:42] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 01:07:42] ...trap/engines/engine.py.326: process: 0.28143 | load: 0.00015 [I 2022-10-28 01:07:45] ...trap/engines/engine.py.321: eval: epoch 97 | batch 510/510 [I 2022-10-28 01:07:45] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 01:07:45] ...trap/engines/engine.py.326: process: 0.28763 | load: 0.00011 [S 2022-10-28 01:07:45] ...trap/engines/engine.py.132: eval_epoch.epoch: 97 [I 2022-10-28 01:07:45] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.734500 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907200 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937800 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.731700 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906700 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937100 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013844 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008681 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007040 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012025 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008050 [S 2022-10-28 01:07:50] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007543 [I 2022-10-28 01:07:51] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 01:07:51] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 01:07:52] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 01:07:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 01:07:52] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 01:07:57] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 01:07:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 01:08:03] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 01:08:03] ...trap/engines/engine.py.163: Training model on trainset for epoch 98 [I 2022-10-28 01:08:03] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 01:08:06] ...trap/engines/engine.py.224: train: epoch 98 | batch 0/2304 [I 2022-10-28 01:08:06] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:43 [I 2022-10-28 01:08:06] ...trap/engines/engine.py.229: process: 2.70133 | load: 2.05656 [I 2022-10-28 01:08:06] ...trap/engines/engine.py.230: loss: 0.10491 [I 2022-10-28 01:08:06] ...optimizers/trijoint.py.151: total_norm: 11.389314 [I 2022-10-28 01:08:12] ...trap/engines/engine.py.224: train: epoch 98 | batch 10/2304 [I 2022-10-28 01:08:12] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:11 [I 2022-10-28 01:08:12] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00398 [I 2022-10-28 01:08:12] ...trap/engines/engine.py.230: loss: 0.16453 [I 2022-10-28 01:08:12] ...optimizers/trijoint.py.151: total_norm: 16.735443 [I 2022-10-28 01:08:17] ...trap/engines/engine.py.224: train: epoch 98 | batch 20/2304 [I 2022-10-28 01:08:17] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:10 [I 2022-10-28 01:08:17] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00281 [I 2022-10-28 01:08:17] ...trap/engines/engine.py.230: loss: 0.14689 [I 2022-10-28 01:08:17] ...optimizers/trijoint.py.151: total_norm: 12.052770 [I 2022-10-28 01:08:23] ...trap/engines/engine.py.224: train: epoch 98 | batch 30/2304 [I 2022-10-28 01:08:23] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:14 [I 2022-10-28 01:08:23] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00317 [I 2022-10-28 01:08:23] ...trap/engines/engine.py.230: loss: 0.15937 [I 2022-10-28 01:08:23] ...optimizers/trijoint.py.151: total_norm: 16.364426 [I 2022-10-28 01:08:28] ...trap/engines/engine.py.224: train: epoch 98 | batch 40/2304 [I 2022-10-28 01:08:28] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:03 [I 2022-10-28 01:08:28] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00368 [I 2022-10-28 01:08:28] ...trap/engines/engine.py.230: loss: 0.11529 [I 2022-10-28 01:08:28] ...optimizers/trijoint.py.151: total_norm: 13.294292 [I 2022-10-28 01:08:34] ...trap/engines/engine.py.224: train: epoch 98 | batch 50/2304 [I 2022-10-28 01:08:34] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:09 [I 2022-10-28 01:08:34] ...trap/engines/engine.py.229: process: 0.55359 | load: 0.00293 [I 2022-10-28 01:08:34] ...trap/engines/engine.py.230: loss: 0.08764 [I 2022-10-28 01:08:34] ...optimizers/trijoint.py.151: total_norm: 12.552757 [I 2022-10-28 01:08:40] ...trap/engines/engine.py.224: train: epoch 98 | batch 60/2304 [I 2022-10-28 01:08:40] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:53 [I 2022-10-28 01:08:40] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00304 [I 2022-10-28 01:08:40] ...trap/engines/engine.py.230: loss: 0.12265 [I 2022-10-28 01:08:40] ...optimizers/trijoint.py.151: total_norm: 37.405018 [I 2022-10-28 01:08:45] ...trap/engines/engine.py.224: train: epoch 98 | batch 70/2304 [I 2022-10-28 01:08:45] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:49 [I 2022-10-28 01:08:45] ...trap/engines/engine.py.229: process: 0.56398 | load: 0.00328 [I 2022-10-28 01:08:45] ...trap/engines/engine.py.230: loss: 0.10183 [I 2022-10-28 01:08:45] ...optimizers/trijoint.py.151: total_norm: 16.684898 [I 2022-10-28 01:08:51] ...trap/engines/engine.py.224: train: epoch 98 | batch 80/2304 [I 2022-10-28 01:08:51] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:30 [I 2022-10-28 01:08:51] ...trap/engines/engine.py.229: process: 0.54669 | load: 0.00311 [I 2022-10-28 01:08:51] ...trap/engines/engine.py.230: loss: 0.11108 [I 2022-10-28 01:08:51] ...optimizers/trijoint.py.151: total_norm: 11.238139 [I 2022-10-28 01:08:57] ...trap/engines/engine.py.224: train: epoch 98 | batch 90/2304 [I 2022-10-28 01:08:57] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:46 [I 2022-10-28 01:08:57] ...trap/engines/engine.py.229: process: 0.58191 | load: 0.00277 [I 2022-10-28 01:08:57] ...trap/engines/engine.py.230: loss: 0.17763 [I 2022-10-28 01:08:57] ...optimizers/trijoint.py.151: total_norm: 40.285774 [I 2022-10-28 01:09:02] ...trap/engines/engine.py.224: train: epoch 98 | batch 100/2304 [I 2022-10-28 01:09:02] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:24 [I 2022-10-28 01:09:02] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00264 [I 2022-10-28 01:09:02] ...trap/engines/engine.py.230: loss: 0.10280 [I 2022-10-28 01:09:02] ...optimizers/trijoint.py.151: total_norm: 17.918644 [I 2022-10-28 01:09:08] ...trap/engines/engine.py.224: train: epoch 98 | batch 110/2304 [I 2022-10-28 01:09:08] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:21:27 [I 2022-10-28 01:09:08] ...trap/engines/engine.py.229: process: 0.67943 | load: 0.01083 [I 2022-10-28 01:09:08] ...trap/engines/engine.py.230: loss: 0.11368 [I 2022-10-28 01:09:08] ...optimizers/trijoint.py.151: total_norm: 20.362232 [I 2022-10-28 01:09:13] ...trap/engines/engine.py.224: train: epoch 98 | batch 120/2304 [I 2022-10-28 01:09:13] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:14 [I 2022-10-28 01:09:13] ...trap/engines/engine.py.229: process: 0.54740 | load: 0.00260 [I 2022-10-28 01:09:13] ...trap/engines/engine.py.230: loss: 0.21301 [I 2022-10-28 01:09:13] ...optimizers/trijoint.py.151: total_norm: 21.190220 [I 2022-10-28 01:09:19] ...trap/engines/engine.py.224: train: epoch 98 | batch 130/2304 [I 2022-10-28 01:09:19] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:10 [I 2022-10-28 01:09:19] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00261 [I 2022-10-28 01:09:19] ...trap/engines/engine.py.230: loss: 0.10532 [I 2022-10-28 01:09:19] ...optimizers/trijoint.py.151: total_norm: 11.539527 [I 2022-10-28 01:09:25] ...trap/engines/engine.py.224: train: epoch 98 | batch 140/2304 [I 2022-10-28 01:09:25] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:14 [I 2022-10-28 01:09:25] ...trap/engines/engine.py.229: process: 0.56978 | load: 0.00268 [I 2022-10-28 01:09:25] ...trap/engines/engine.py.230: loss: 0.11783 [I 2022-10-28 01:09:25] ...optimizers/trijoint.py.151: total_norm: 13.227075 [I 2022-10-28 01:09:30] ...trap/engines/engine.py.224: train: epoch 98 | batch 150/2304 [I 2022-10-28 01:09:30] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:06 [I 2022-10-28 01:09:30] ...trap/engines/engine.py.229: process: 0.55892 | load: 0.00246 [I 2022-10-28 01:09:30] ...trap/engines/engine.py.230: loss: 0.14722 [I 2022-10-28 01:09:30] ...optimizers/trijoint.py.151: total_norm: 21.491920 [I 2022-10-28 01:09:36] ...trap/engines/engine.py.224: train: epoch 98 | batch 160/2304 [I 2022-10-28 01:09:36] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:53 [I 2022-10-28 01:09:36] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00249 [I 2022-10-28 01:09:36] ...trap/engines/engine.py.230: loss: 0.14084 [I 2022-10-28 01:09:36] ...optimizers/trijoint.py.151: total_norm: 15.938545 [I 2022-10-28 01:09:41] ...trap/engines/engine.py.224: train: epoch 98 | batch 170/2304 [I 2022-10-28 01:09:41] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:47 [I 2022-10-28 01:09:41] ...trap/engines/engine.py.229: process: 0.54970 | load: 0.00260 [I 2022-10-28 01:09:41] ...trap/engines/engine.py.230: loss: 0.14434 [I 2022-10-28 01:09:41] ...optimizers/trijoint.py.151: total_norm: 45.274223 [I 2022-10-28 01:09:47] ...trap/engines/engine.py.224: train: epoch 98 | batch 180/2304 [I 2022-10-28 01:09:47] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:56 [I 2022-10-28 01:09:47] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00251 [I 2022-10-28 01:09:47] ...trap/engines/engine.py.230: loss: 0.13885 [I 2022-10-28 01:09:47] ...optimizers/trijoint.py.151: total_norm: 19.135914 [I 2022-10-28 01:09:53] ...trap/engines/engine.py.224: train: epoch 98 | batch 190/2304 [I 2022-10-28 01:09:53] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:38 [I 2022-10-28 01:09:53] ...trap/engines/engine.py.229: process: 0.54406 | load: 0.00247 [I 2022-10-28 01:09:53] ...trap/engines/engine.py.230: loss: 0.09945 [I 2022-10-28 01:09:53] ...optimizers/trijoint.py.151: total_norm: 18.408371 [I 2022-10-28 01:09:58] ...trap/engines/engine.py.224: train: epoch 98 | batch 200/2304 [I 2022-10-28 01:09:58] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:26 [I 2022-10-28 01:09:58] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00272 [I 2022-10-28 01:09:58] ...trap/engines/engine.py.230: loss: 0.09228 [I 2022-10-28 01:09:58] ...optimizers/trijoint.py.151: total_norm: 14.437275 [I 2022-10-28 01:10:04] ...trap/engines/engine.py.224: train: epoch 98 | batch 210/2304 [I 2022-10-28 01:10:04] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:22 [I 2022-10-28 01:10:04] ...trap/engines/engine.py.229: process: 0.55786 | load: 0.00244 [I 2022-10-28 01:10:04] ...trap/engines/engine.py.230: loss: 0.16879 [I 2022-10-28 01:10:04] ...optimizers/trijoint.py.151: total_norm: 12.432055 [I 2022-10-28 01:10:09] ...trap/engines/engine.py.224: train: epoch 98 | batch 220/2304 [I 2022-10-28 01:10:09] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:23 [I 2022-10-28 01:10:09] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00260 [I 2022-10-28 01:10:09] ...trap/engines/engine.py.230: loss: 0.11717 [I 2022-10-28 01:10:09] ...optimizers/trijoint.py.151: total_norm: 13.647031 [I 2022-10-28 01:10:15] ...trap/engines/engine.py.224: train: epoch 98 | batch 230/2304 [I 2022-10-28 01:10:15] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:18 [I 2022-10-28 01:10:15] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00243 [I 2022-10-28 01:10:15] ...trap/engines/engine.py.230: loss: 0.10538 [I 2022-10-28 01:10:15] ...optimizers/trijoint.py.151: total_norm: 12.305960 [I 2022-10-28 01:10:21] ...trap/engines/engine.py.224: train: epoch 98 | batch 240/2304 [I 2022-10-28 01:10:21] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:39 [I 2022-10-28 01:10:21] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00263 [I 2022-10-28 01:10:21] ...trap/engines/engine.py.230: loss: 0.16654 [I 2022-10-28 01:10:21] ...optimizers/trijoint.py.151: total_norm: 18.746679 [I 2022-10-28 01:10:26] ...trap/engines/engine.py.224: train: epoch 98 | batch 250/2304 [I 2022-10-28 01:10:26] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:20 [I 2022-10-28 01:10:26] ...trap/engines/engine.py.229: process: 0.56536 | load: 0.00250 [I 2022-10-28 01:10:26] ...trap/engines/engine.py.230: loss: 0.16263 [I 2022-10-28 01:10:26] ...optimizers/trijoint.py.151: total_norm: 10.325178 [I 2022-10-28 01:10:32] ...trap/engines/engine.py.224: train: epoch 98 | batch 260/2304 [I 2022-10-28 01:10:32] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:58 [I 2022-10-28 01:10:32] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00241 [I 2022-10-28 01:10:32] ...trap/engines/engine.py.230: loss: 0.15014 [I 2022-10-28 01:10:32] ...optimizers/trijoint.py.151: total_norm: 9.931839 [I 2022-10-28 01:10:38] ...trap/engines/engine.py.224: train: epoch 98 | batch 270/2304 [I 2022-10-28 01:10:38] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:43 [I 2022-10-28 01:10:38] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00275 [I 2022-10-28 01:10:38] ...trap/engines/engine.py.230: loss: 0.11519 [I 2022-10-28 01:10:38] ...optimizers/trijoint.py.151: total_norm: 20.666712 [I 2022-10-28 01:10:43] ...trap/engines/engine.py.224: train: epoch 98 | batch 280/2304 [I 2022-10-28 01:10:43] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:43 [I 2022-10-28 01:10:43] ...trap/engines/engine.py.229: process: 0.55725 | load: 0.00252 [I 2022-10-28 01:10:43] ...trap/engines/engine.py.230: loss: 0.11129 [I 2022-10-28 01:10:43] ...optimizers/trijoint.py.151: total_norm: 13.216395 [I 2022-10-28 01:10:49] ...trap/engines/engine.py.224: train: epoch 98 | batch 290/2304 [I 2022-10-28 01:10:49] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:39 [I 2022-10-28 01:10:49] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00253 [I 2022-10-28 01:10:49] ...trap/engines/engine.py.230: loss: 0.17735 [I 2022-10-28 01:10:49] ...optimizers/trijoint.py.151: total_norm: 30.959522 [I 2022-10-28 01:10:54] ...trap/engines/engine.py.224: train: epoch 98 | batch 300/2304 [I 2022-10-28 01:10:54] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:19:11 [I 2022-10-28 01:10:54] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00247 [I 2022-10-28 01:10:54] ...trap/engines/engine.py.230: loss: 0.09076 [I 2022-10-28 01:10:54] ...optimizers/trijoint.py.151: total_norm: 12.163541 [I 2022-10-28 01:11:00] ...trap/engines/engine.py.224: train: epoch 98 | batch 310/2304 [I 2022-10-28 01:11:00] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:36 [I 2022-10-28 01:11:00] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00239 [I 2022-10-28 01:11:00] ...trap/engines/engine.py.230: loss: 0.10547 [I 2022-10-28 01:11:00] ...optimizers/trijoint.py.151: total_norm: 11.518095 [I 2022-10-28 01:11:05] ...trap/engines/engine.py.224: train: epoch 98 | batch 320/2304 [I 2022-10-28 01:11:05] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:19 [I 2022-10-28 01:11:05] ...trap/engines/engine.py.229: process: 0.56243 | load: 0.00250 [I 2022-10-28 01:11:05] ...trap/engines/engine.py.230: loss: 0.16104 [I 2022-10-28 01:11:05] ...optimizers/trijoint.py.151: total_norm: 24.215099 [I 2022-10-28 01:11:11] ...trap/engines/engine.py.224: train: epoch 98 | batch 330/2304 [I 2022-10-28 01:11:11] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:21 [I 2022-10-28 01:11:11] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00287 [I 2022-10-28 01:11:11] ...trap/engines/engine.py.230: loss: 0.14140 [I 2022-10-28 01:11:11] ...optimizers/trijoint.py.151: total_norm: 8.478919 [I 2022-10-28 01:11:17] ...trap/engines/engine.py.224: train: epoch 98 | batch 340/2304 [I 2022-10-28 01:11:17] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:24 [I 2022-10-28 01:11:17] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00271 [I 2022-10-28 01:11:17] ...trap/engines/engine.py.230: loss: 0.13005 [I 2022-10-28 01:11:17] ...optimizers/trijoint.py.151: total_norm: 11.577358 [I 2022-10-28 01:11:22] ...trap/engines/engine.py.224: train: epoch 98 | batch 350/2304 [I 2022-10-28 01:11:22] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:06 [I 2022-10-28 01:11:22] ...trap/engines/engine.py.229: process: 0.55343 | load: 0.00244 [I 2022-10-28 01:11:22] ...trap/engines/engine.py.230: loss: 0.16848 [I 2022-10-28 01:11:22] ...optimizers/trijoint.py.151: total_norm: 22.837297 [I 2022-10-28 01:11:28] ...trap/engines/engine.py.224: train: epoch 98 | batch 360/2304 [I 2022-10-28 01:11:28] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:41 [I 2022-10-28 01:11:28] ...trap/engines/engine.py.229: process: 0.64129 | load: 0.00289 [I 2022-10-28 01:11:28] ...trap/engines/engine.py.230: loss: 0.11654 [I 2022-10-28 01:11:28] ...optimizers/trijoint.py.151: total_norm: 19.700726 [I 2022-10-28 01:11:34] ...trap/engines/engine.py.224: train: epoch 98 | batch 370/2304 [I 2022-10-28 01:11:34] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:01 [I 2022-10-28 01:11:34] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00280 [I 2022-10-28 01:11:34] ...trap/engines/engine.py.230: loss: 0.11584 [I 2022-10-28 01:11:34] ...optimizers/trijoint.py.151: total_norm: 13.233513 [I 2022-10-28 01:11:39] ...trap/engines/engine.py.224: train: epoch 98 | batch 380/2304 [I 2022-10-28 01:11:39] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:55 [I 2022-10-28 01:11:39] ...trap/engines/engine.py.229: process: 0.54758 | load: 0.00258 [I 2022-10-28 01:11:39] ...trap/engines/engine.py.230: loss: 0.20321 [I 2022-10-28 01:11:39] ...optimizers/trijoint.py.151: total_norm: 24.385365 [I 2022-10-28 01:11:45] ...trap/engines/engine.py.224: train: epoch 98 | batch 390/2304 [I 2022-10-28 01:11:45] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:48 [I 2022-10-28 01:11:45] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00257 [I 2022-10-28 01:11:45] ...trap/engines/engine.py.230: loss: 0.16943 [I 2022-10-28 01:11:45] ...optimizers/trijoint.py.151: total_norm: 13.716532 [I 2022-10-28 01:11:50] ...trap/engines/engine.py.224: train: epoch 98 | batch 400/2304 [I 2022-10-28 01:11:50] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:38 [I 2022-10-28 01:11:50] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00271 [I 2022-10-28 01:11:50] ...trap/engines/engine.py.230: loss: 0.11763 [I 2022-10-28 01:11:50] ...optimizers/trijoint.py.151: total_norm: 9.377071 [I 2022-10-28 01:11:56] ...trap/engines/engine.py.224: train: epoch 98 | batch 410/2304 [I 2022-10-28 01:11:56] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:43 [I 2022-10-28 01:11:56] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00277 [I 2022-10-28 01:11:56] ...trap/engines/engine.py.230: loss: 0.10738 [I 2022-10-28 01:11:56] ...optimizers/trijoint.py.151: total_norm: 43.305721 [I 2022-10-28 01:12:02] ...trap/engines/engine.py.224: train: epoch 98 | batch 420/2304 [I 2022-10-28 01:12:02] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:28 [I 2022-10-28 01:12:02] ...trap/engines/engine.py.229: process: 0.54325 | load: 0.00260 [I 2022-10-28 01:12:02] ...trap/engines/engine.py.230: loss: 0.09796 [I 2022-10-28 01:12:02] ...optimizers/trijoint.py.151: total_norm: 7.929506 [I 2022-10-28 01:12:07] ...trap/engines/engine.py.224: train: epoch 98 | batch 430/2304 [I 2022-10-28 01:12:07] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:27 [I 2022-10-28 01:12:07] ...trap/engines/engine.py.229: process: 0.55378 | load: 0.00251 [I 2022-10-28 01:12:07] ...trap/engines/engine.py.230: loss: 0.24670 [I 2022-10-28 01:12:07] ...optimizers/trijoint.py.151: total_norm: 23.108940 [I 2022-10-28 01:12:13] ...trap/engines/engine.py.224: train: epoch 98 | batch 440/2304 [I 2022-10-28 01:12:13] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:20 [I 2022-10-28 01:12:13] ...trap/engines/engine.py.229: process: 0.55529 | load: 0.00250 [I 2022-10-28 01:12:13] ...trap/engines/engine.py.230: loss: 0.10962 [I 2022-10-28 01:12:13] ...optimizers/trijoint.py.151: total_norm: 8.687437 [I 2022-10-28 01:12:18] ...trap/engines/engine.py.224: train: epoch 98 | batch 450/2304 [I 2022-10-28 01:12:18] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:12 [I 2022-10-28 01:12:18] ...trap/engines/engine.py.229: process: 0.56154 | load: 0.00250 [I 2022-10-28 01:12:18] ...trap/engines/engine.py.230: loss: 0.09119 [I 2022-10-28 01:12:18] ...optimizers/trijoint.py.151: total_norm: 13.106874 [I 2022-10-28 01:12:24] ...trap/engines/engine.py.224: train: epoch 98 | batch 460/2304 [I 2022-10-28 01:12:24] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:09 [I 2022-10-28 01:12:24] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00259 [I 2022-10-28 01:12:24] ...trap/engines/engine.py.230: loss: 0.10880 [I 2022-10-28 01:12:24] ...optimizers/trijoint.py.151: total_norm: 12.015429 [I 2022-10-28 01:12:30] ...trap/engines/engine.py.224: train: epoch 98 | batch 470/2304 [I 2022-10-28 01:12:30] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:07 [I 2022-10-28 01:12:30] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00252 [I 2022-10-28 01:12:30] ...trap/engines/engine.py.230: loss: 0.13879 [I 2022-10-28 01:12:30] ...optimizers/trijoint.py.151: total_norm: 15.328925 [I 2022-10-28 01:12:35] ...trap/engines/engine.py.224: train: epoch 98 | batch 480/2304 [I 2022-10-28 01:12:35] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:53 [I 2022-10-28 01:12:35] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00250 [I 2022-10-28 01:12:35] ...trap/engines/engine.py.230: loss: 0.11061 [I 2022-10-28 01:12:35] ...optimizers/trijoint.py.151: total_norm: 23.500322 [I 2022-10-28 01:12:41] ...trap/engines/engine.py.224: train: epoch 98 | batch 490/2304 [I 2022-10-28 01:12:41] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:56 [I 2022-10-28 01:12:41] ...trap/engines/engine.py.229: process: 0.56618 | load: 0.00249 [I 2022-10-28 01:12:41] ...trap/engines/engine.py.230: loss: 0.10260 [I 2022-10-28 01:12:41] ...optimizers/trijoint.py.151: total_norm: 16.017578 [I 2022-10-28 01:12:46] ...trap/engines/engine.py.224: train: epoch 98 | batch 500/2304 [I 2022-10-28 01:12:46] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:44 [I 2022-10-28 01:12:46] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00243 [I 2022-10-28 01:12:46] ...trap/engines/engine.py.230: loss: 0.11215 [I 2022-10-28 01:12:46] ...optimizers/trijoint.py.151: total_norm: 10.575991 [I 2022-10-28 01:12:52] ...trap/engines/engine.py.224: train: epoch 98 | batch 510/2304 [I 2022-10-28 01:12:52] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:36 [I 2022-10-28 01:12:52] ...trap/engines/engine.py.229: process: 0.55200 | load: 0.00245 [I 2022-10-28 01:12:52] ...trap/engines/engine.py.230: loss: 0.07760 [I 2022-10-28 01:12:52] ...optimizers/trijoint.py.151: total_norm: 14.767022 [I 2022-10-28 01:12:57] ...trap/engines/engine.py.224: train: epoch 98 | batch 520/2304 [I 2022-10-28 01:12:57] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:25 [I 2022-10-28 01:12:57] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00241 [I 2022-10-28 01:12:57] ...trap/engines/engine.py.230: loss: 0.10458 [I 2022-10-28 01:12:57] ...optimizers/trijoint.py.151: total_norm: 8.528258 [I 2022-10-28 01:13:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 530/2304 [I 2022-10-28 01:13:03] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:28 [I 2022-10-28 01:13:03] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00242 [I 2022-10-28 01:13:03] ...trap/engines/engine.py.230: loss: 0.19242 [I 2022-10-28 01:13:03] ...optimizers/trijoint.py.151: total_norm: 16.296530 [I 2022-10-28 01:13:09] ...trap/engines/engine.py.224: train: epoch 98 | batch 540/2304 [I 2022-10-28 01:13:09] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:21 [I 2022-10-28 01:13:09] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00239 [I 2022-10-28 01:13:09] ...trap/engines/engine.py.230: loss: 0.10753 [I 2022-10-28 01:13:09] ...optimizers/trijoint.py.151: total_norm: 32.044178 [I 2022-10-28 01:13:34] ...trap/engines/engine.py.224: train: epoch 98 | batch 550/2304 [I 2022-10-28 01:13:34] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 1:04:56 [I 2022-10-28 01:13:34] ...trap/engines/engine.py.229: process: 0.57141 | load: 0.00238 [I 2022-10-28 01:13:34] ...trap/engines/engine.py.230: loss: 0.11328 [I 2022-10-28 01:13:34] ...optimizers/trijoint.py.151: total_norm: 23.550146 [I 2022-10-28 01:13:40] ...trap/engines/engine.py.224: train: epoch 98 | batch 560/2304 [I 2022-10-28 01:13:40] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:21:36 [I 2022-10-28 01:13:40] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00241 [I 2022-10-28 01:13:40] ...trap/engines/engine.py.230: loss: 0.10510 [I 2022-10-28 01:13:40] ...optimizers/trijoint.py.151: total_norm: 15.265977 [I 2022-10-28 01:13:46] ...trap/engines/engine.py.224: train: epoch 98 | batch 570/2304 [I 2022-10-28 01:13:46] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:16:39 [I 2022-10-28 01:13:46] ...trap/engines/engine.py.229: process: 0.56621 | load: 0.00241 [I 2022-10-28 01:13:46] ...trap/engines/engine.py.230: loss: 0.15166 [I 2022-10-28 01:13:46] ...optimizers/trijoint.py.151: total_norm: 16.197351 [I 2022-10-28 01:13:51] ...trap/engines/engine.py.224: train: epoch 98 | batch 580/2304 [I 2022-10-28 01:13:51] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:16:02 [I 2022-10-28 01:13:51] ...trap/engines/engine.py.229: process: 0.55821 | load: 0.00239 [I 2022-10-28 01:13:51] ...trap/engines/engine.py.230: loss: 0.10388 [I 2022-10-28 01:13:51] ...optimizers/trijoint.py.151: total_norm: 23.731976 [I 2022-10-28 01:13:57] ...trap/engines/engine.py.224: train: epoch 98 | batch 590/2304 [I 2022-10-28 01:13:57] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:15:55 [I 2022-10-28 01:13:57] ...trap/engines/engine.py.229: process: 0.55205 | load: 0.00242 [I 2022-10-28 01:13:57] ...trap/engines/engine.py.230: loss: 0.13945 [I 2022-10-28 01:13:57] ...optimizers/trijoint.py.151: total_norm: 13.061001 [I 2022-10-28 01:14:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 600/2304 [I 2022-10-28 01:14:03] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:56 [I 2022-10-28 01:14:03] ...trap/engines/engine.py.229: process: 0.55790 | load: 0.00239 [I 2022-10-28 01:14:03] ...trap/engines/engine.py.230: loss: 0.09374 [I 2022-10-28 01:14:03] ...optimizers/trijoint.py.151: total_norm: 15.166774 [I 2022-10-28 01:14:08] ...trap/engines/engine.py.224: train: epoch 98 | batch 610/2304 [I 2022-10-28 01:14:08] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:48 [I 2022-10-28 01:14:08] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00259 [I 2022-10-28 01:14:08] ...trap/engines/engine.py.230: loss: 0.08913 [I 2022-10-28 01:14:08] ...optimizers/trijoint.py.151: total_norm: 15.103796 [I 2022-10-28 01:14:14] ...trap/engines/engine.py.224: train: epoch 98 | batch 620/2304 [I 2022-10-28 01:14:14] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:15:45 [I 2022-10-28 01:14:14] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00250 [I 2022-10-28 01:14:14] ...trap/engines/engine.py.230: loss: 0.15992 [I 2022-10-28 01:14:14] ...optimizers/trijoint.py.151: total_norm: 16.111916 [I 2022-10-28 01:14:19] ...trap/engines/engine.py.224: train: epoch 98 | batch 630/2304 [I 2022-10-28 01:14:19] ...trap/engines/engine.py.228: elapsed: 0:06:16 | left: 0:15:38 [I 2022-10-28 01:14:19] ...trap/engines/engine.py.229: process: 0.56890 | load: 0.00250 [I 2022-10-28 01:14:19] ...trap/engines/engine.py.230: loss: 0.15101 [I 2022-10-28 01:14:19] ...optimizers/trijoint.py.151: total_norm: 16.017794 [I 2022-10-28 01:14:25] ...trap/engines/engine.py.224: train: epoch 98 | batch 640/2304 [I 2022-10-28 01:14:25] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:15:50 [I 2022-10-28 01:14:25] ...trap/engines/engine.py.229: process: 0.56944 | load: 0.00286 [I 2022-10-28 01:14:25] ...trap/engines/engine.py.230: loss: 0.10801 [I 2022-10-28 01:14:25] ...optimizers/trijoint.py.151: total_norm: 17.259886 [I 2022-10-28 01:14:31] ...trap/engines/engine.py.224: train: epoch 98 | batch 650/2304 [I 2022-10-28 01:14:31] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:15:20 [I 2022-10-28 01:14:31] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00246 [I 2022-10-28 01:14:31] ...trap/engines/engine.py.230: loss: 0.13082 [I 2022-10-28 01:14:31] ...optimizers/trijoint.py.151: total_norm: 26.684862 [I 2022-10-28 01:14:36] ...trap/engines/engine.py.224: train: epoch 98 | batch 660/2304 [I 2022-10-28 01:14:36] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:15:24 [I 2022-10-28 01:14:36] ...trap/engines/engine.py.229: process: 0.56897 | load: 0.00245 [I 2022-10-28 01:14:36] ...trap/engines/engine.py.230: loss: 0.11684 [I 2022-10-28 01:14:36] ...optimizers/trijoint.py.151: total_norm: 8.884651 [I 2022-10-28 01:14:42] ...trap/engines/engine.py.224: train: epoch 98 | batch 670/2304 [I 2022-10-28 01:14:42] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:15:10 [I 2022-10-28 01:14:42] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00245 [I 2022-10-28 01:14:42] ...trap/engines/engine.py.230: loss: 0.10777 [I 2022-10-28 01:14:42] ...optimizers/trijoint.py.151: total_norm: 11.811015 [I 2022-10-28 01:14:48] ...trap/engines/engine.py.224: train: epoch 98 | batch 680/2304 [I 2022-10-28 01:14:48] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:15:08 [I 2022-10-28 01:14:48] ...trap/engines/engine.py.229: process: 0.56615 | load: 0.00244 [I 2022-10-28 01:14:48] ...trap/engines/engine.py.230: loss: 0.13945 [I 2022-10-28 01:14:48] ...optimizers/trijoint.py.151: total_norm: 13.650386 [I 2022-10-28 01:14:53] ...trap/engines/engine.py.224: train: epoch 98 | batch 690/2304 [I 2022-10-28 01:14:53] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:15:03 [I 2022-10-28 01:14:53] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00241 [I 2022-10-28 01:14:53] ...trap/engines/engine.py.230: loss: 0.20190 [I 2022-10-28 01:14:53] ...optimizers/trijoint.py.151: total_norm: 20.278654 [I 2022-10-28 01:14:59] ...trap/engines/engine.py.224: train: epoch 98 | batch 700/2304 [I 2022-10-28 01:14:59] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:46 [I 2022-10-28 01:14:59] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00243 [I 2022-10-28 01:14:59] ...trap/engines/engine.py.230: loss: 0.14982 [I 2022-10-28 01:14:59] ...optimizers/trijoint.py.151: total_norm: 16.034830 [I 2022-10-28 01:15:04] ...trap/engines/engine.py.224: train: epoch 98 | batch 710/2304 [I 2022-10-28 01:15:04] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:47 [I 2022-10-28 01:15:04] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00241 [I 2022-10-28 01:15:04] ...trap/engines/engine.py.230: loss: 0.11389 [I 2022-10-28 01:15:04] ...optimizers/trijoint.py.151: total_norm: 20.254444 [I 2022-10-28 01:15:10] ...trap/engines/engine.py.224: train: epoch 98 | batch 720/2304 [I 2022-10-28 01:15:10] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:34 [I 2022-10-28 01:15:10] ...trap/engines/engine.py.229: process: 0.54340 | load: 0.00240 [I 2022-10-28 01:15:10] ...trap/engines/engine.py.230: loss: 0.11471 [I 2022-10-28 01:15:10] ...optimizers/trijoint.py.151: total_norm: 7.658937 [I 2022-10-28 01:15:15] ...trap/engines/engine.py.224: train: epoch 98 | batch 730/2304 [I 2022-10-28 01:15:15] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:38 [I 2022-10-28 01:15:15] ...trap/engines/engine.py.229: process: 0.56240 | load: 0.00240 [I 2022-10-28 01:15:15] ...trap/engines/engine.py.230: loss: 0.14843 [I 2022-10-28 01:15:15] ...optimizers/trijoint.py.151: total_norm: 14.194423 [I 2022-10-28 01:15:21] ...trap/engines/engine.py.224: train: epoch 98 | batch 740/2304 [I 2022-10-28 01:15:21] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:14:34 [I 2022-10-28 01:15:21] ...trap/engines/engine.py.229: process: 0.55899 | load: 0.00242 [I 2022-10-28 01:15:21] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-28 01:15:21] ...optimizers/trijoint.py.151: total_norm: 19.523100 [I 2022-10-28 01:15:27] ...trap/engines/engine.py.224: train: epoch 98 | batch 750/2304 [I 2022-10-28 01:15:27] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:27 [I 2022-10-28 01:15:27] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00241 [I 2022-10-28 01:15:27] ...trap/engines/engine.py.230: loss: 0.09524 [I 2022-10-28 01:15:27] ...optimizers/trijoint.py.151: total_norm: 22.904707 [I 2022-10-28 01:15:32] ...trap/engines/engine.py.224: train: epoch 98 | batch 760/2304 [I 2022-10-28 01:15:32] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:25 [I 2022-10-28 01:15:32] ...trap/engines/engine.py.229: process: 0.55219 | load: 0.00239 [I 2022-10-28 01:15:32] ...trap/engines/engine.py.230: loss: 0.15999 [I 2022-10-28 01:15:32] ...optimizers/trijoint.py.151: total_norm: 15.803445 [I 2022-10-28 01:15:38] ...trap/engines/engine.py.224: train: epoch 98 | batch 770/2304 [I 2022-10-28 01:15:38] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:14:16 [I 2022-10-28 01:15:38] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00240 [I 2022-10-28 01:15:38] ...trap/engines/engine.py.230: loss: 0.10982 [I 2022-10-28 01:15:38] ...optimizers/trijoint.py.151: total_norm: 18.390985 [I 2022-10-28 01:15:43] ...trap/engines/engine.py.224: train: epoch 98 | batch 780/2304 [I 2022-10-28 01:15:43] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:14:15 [I 2022-10-28 01:15:43] ...trap/engines/engine.py.229: process: 0.56720 | load: 0.00243 [I 2022-10-28 01:15:43] ...trap/engines/engine.py.230: loss: 0.10576 [I 2022-10-28 01:15:43] ...optimizers/trijoint.py.151: total_norm: 19.009949 [I 2022-10-28 01:15:49] ...trap/engines/engine.py.224: train: epoch 98 | batch 790/2304 [I 2022-10-28 01:15:49] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:14:03 [I 2022-10-28 01:15:49] ...trap/engines/engine.py.229: process: 0.55350 | load: 0.00246 [I 2022-10-28 01:15:49] ...trap/engines/engine.py.230: loss: 0.10036 [I 2022-10-28 01:15:49] ...optimizers/trijoint.py.151: total_norm: 8.986582 [I 2022-10-28 01:15:54] ...trap/engines/engine.py.224: train: epoch 98 | batch 800/2304 [I 2022-10-28 01:15:54] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:51 [I 2022-10-28 01:15:54] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00243 [I 2022-10-28 01:15:54] ...trap/engines/engine.py.230: loss: 0.18810 [I 2022-10-28 01:15:54] ...optimizers/trijoint.py.151: total_norm: 48.740971 [I 2022-10-28 01:16:00] ...trap/engines/engine.py.224: train: epoch 98 | batch 810/2304 [I 2022-10-28 01:16:00] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:48 [I 2022-10-28 01:16:00] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00239 [I 2022-10-28 01:16:00] ...trap/engines/engine.py.230: loss: 0.12195 [I 2022-10-28 01:16:00] ...optimizers/trijoint.py.151: total_norm: 11.878814 [I 2022-10-28 01:16:06] ...trap/engines/engine.py.224: train: epoch 98 | batch 820/2304 [I 2022-10-28 01:16:06] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:45 [I 2022-10-28 01:16:06] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00241 [I 2022-10-28 01:16:06] ...trap/engines/engine.py.230: loss: 0.15678 [I 2022-10-28 01:16:06] ...optimizers/trijoint.py.151: total_norm: 26.762245 [I 2022-10-28 01:16:11] ...trap/engines/engine.py.224: train: epoch 98 | batch 830/2304 [I 2022-10-28 01:16:11] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:40 [I 2022-10-28 01:16:11] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00241 [I 2022-10-28 01:16:11] ...trap/engines/engine.py.230: loss: 0.11263 [I 2022-10-28 01:16:11] ...optimizers/trijoint.py.151: total_norm: 11.210824 [I 2022-10-28 01:16:17] ...trap/engines/engine.py.224: train: epoch 98 | batch 840/2304 [I 2022-10-28 01:16:17] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:34 [I 2022-10-28 01:16:17] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00240 [I 2022-10-28 01:16:17] ...trap/engines/engine.py.230: loss: 0.16431 [I 2022-10-28 01:16:17] ...optimizers/trijoint.py.151: total_norm: 42.366550 [I 2022-10-28 01:16:22] ...trap/engines/engine.py.224: train: epoch 98 | batch 850/2304 [I 2022-10-28 01:16:22] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:27 [I 2022-10-28 01:16:22] ...trap/engines/engine.py.229: process: 0.55743 | load: 0.00241 [I 2022-10-28 01:16:22] ...trap/engines/engine.py.230: loss: 0.11439 [I 2022-10-28 01:16:22] ...optimizers/trijoint.py.151: total_norm: 30.277504 [I 2022-10-28 01:16:28] ...trap/engines/engine.py.224: train: epoch 98 | batch 860/2304 [I 2022-10-28 01:16:28] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:19 [I 2022-10-28 01:16:28] ...trap/engines/engine.py.229: process: 0.54817 | load: 0.00238 [I 2022-10-28 01:16:28] ...trap/engines/engine.py.230: loss: 0.10549 [I 2022-10-28 01:16:28] ...optimizers/trijoint.py.151: total_norm: 17.847500 [I 2022-10-28 01:16:33] ...trap/engines/engine.py.224: train: epoch 98 | batch 870/2304 [I 2022-10-28 01:16:33] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:13:17 [I 2022-10-28 01:16:33] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00242 [I 2022-10-28 01:16:33] ...trap/engines/engine.py.230: loss: 0.09342 [I 2022-10-28 01:16:33] ...optimizers/trijoint.py.151: total_norm: 15.222683 [I 2022-10-28 01:16:39] ...trap/engines/engine.py.224: train: epoch 98 | batch 880/2304 [I 2022-10-28 01:16:39] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:09 [I 2022-10-28 01:16:39] ...trap/engines/engine.py.229: process: 0.54720 | load: 0.00242 [I 2022-10-28 01:16:39] ...trap/engines/engine.py.230: loss: 0.16831 [I 2022-10-28 01:16:39] ...optimizers/trijoint.py.151: total_norm: 23.290939 [I 2022-10-28 01:16:45] ...trap/engines/engine.py.224: train: epoch 98 | batch 890/2304 [I 2022-10-28 01:16:45] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:13:13 [I 2022-10-28 01:16:45] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00240 [I 2022-10-28 01:16:45] ...trap/engines/engine.py.230: loss: 0.09174 [I 2022-10-28 01:16:45] ...optimizers/trijoint.py.151: total_norm: 11.629288 [I 2022-10-28 01:16:50] ...trap/engines/engine.py.224: train: epoch 98 | batch 900/2304 [I 2022-10-28 01:16:50] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:13:33 [I 2022-10-28 01:16:50] ...trap/engines/engine.py.229: process: 0.60562 | load: 0.01087 [I 2022-10-28 01:16:50] ...trap/engines/engine.py.230: loss: 0.10052 [I 2022-10-28 01:16:50] ...optimizers/trijoint.py.151: total_norm: 18.711948 [I 2022-10-28 01:16:56] ...trap/engines/engine.py.224: train: epoch 98 | batch 910/2304 [I 2022-10-28 01:16:56] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:58 [I 2022-10-28 01:16:56] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00242 [I 2022-10-28 01:16:56] ...trap/engines/engine.py.230: loss: 0.10098 [I 2022-10-28 01:16:56] ...optimizers/trijoint.py.151: total_norm: 9.277586 [I 2022-10-28 01:17:02] ...trap/engines/engine.py.224: train: epoch 98 | batch 920/2304 [I 2022-10-28 01:17:02] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:55 [I 2022-10-28 01:17:02] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00241 [I 2022-10-28 01:17:02] ...trap/engines/engine.py.230: loss: 0.10807 [I 2022-10-28 01:17:02] ...optimizers/trijoint.py.151: total_norm: 7.947207 [I 2022-10-28 01:17:07] ...trap/engines/engine.py.224: train: epoch 98 | batch 930/2304 [I 2022-10-28 01:17:07] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:41 [I 2022-10-28 01:17:07] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00240 [I 2022-10-28 01:17:07] ...trap/engines/engine.py.230: loss: 0.19542 [I 2022-10-28 01:17:07] ...optimizers/trijoint.py.151: total_norm: 15.528631 [I 2022-10-28 01:17:13] ...trap/engines/engine.py.224: train: epoch 98 | batch 940/2304 [I 2022-10-28 01:17:13] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:42 [I 2022-10-28 01:17:13] ...trap/engines/engine.py.229: process: 0.56264 | load: 0.00244 [I 2022-10-28 01:17:13] ...trap/engines/engine.py.230: loss: 0.08700 [I 2022-10-28 01:17:13] ...optimizers/trijoint.py.151: total_norm: 11.854239 [I 2022-10-28 01:17:18] ...trap/engines/engine.py.224: train: epoch 98 | batch 950/2304 [I 2022-10-28 01:17:18] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:35 [I 2022-10-28 01:17:18] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00241 [I 2022-10-28 01:17:18] ...trap/engines/engine.py.230: loss: 0.12140 [I 2022-10-28 01:17:18] ...optimizers/trijoint.py.151: total_norm: 14.191806 [I 2022-10-28 01:17:24] ...trap/engines/engine.py.224: train: epoch 98 | batch 960/2304 [I 2022-10-28 01:17:24] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:25 [I 2022-10-28 01:17:24] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00241 [I 2022-10-28 01:17:24] ...trap/engines/engine.py.230: loss: 0.08977 [I 2022-10-28 01:17:24] ...optimizers/trijoint.py.151: total_norm: 18.666315 [I 2022-10-28 01:17:30] ...trap/engines/engine.py.224: train: epoch 98 | batch 970/2304 [I 2022-10-28 01:17:30] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:46 [I 2022-10-28 01:17:30] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00239 [I 2022-10-28 01:17:30] ...trap/engines/engine.py.230: loss: 0.09743 [I 2022-10-28 01:17:30] ...optimizers/trijoint.py.151: total_norm: 10.218749 [I 2022-10-28 01:17:35] ...trap/engines/engine.py.224: train: epoch 98 | batch 980/2304 [I 2022-10-28 01:17:35] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:21 [I 2022-10-28 01:17:35] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00241 [I 2022-10-28 01:17:35] ...trap/engines/engine.py.230: loss: 0.09908 [I 2022-10-28 01:17:35] ...optimizers/trijoint.py.151: total_norm: 22.351728 [I 2022-10-28 01:17:41] ...trap/engines/engine.py.224: train: epoch 98 | batch 990/2304 [I 2022-10-28 01:17:41] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:12:08 [I 2022-10-28 01:17:41] ...trap/engines/engine.py.229: process: 0.55280 | load: 0.00243 [I 2022-10-28 01:17:41] ...trap/engines/engine.py.230: loss: 0.10268 [I 2022-10-28 01:17:41] ...optimizers/trijoint.py.151: total_norm: 19.452404 [I 2022-10-28 01:17:46] ...trap/engines/engine.py.224: train: epoch 98 | batch 1000/2304 [I 2022-10-28 01:17:46] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:12:09 [I 2022-10-28 01:17:46] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00240 [I 2022-10-28 01:17:46] ...trap/engines/engine.py.230: loss: 0.12383 [I 2022-10-28 01:17:46] ...optimizers/trijoint.py.151: total_norm: 11.522415 [I 2022-10-28 01:17:52] ...trap/engines/engine.py.224: train: epoch 98 | batch 1010/2304 [I 2022-10-28 01:17:52] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:56 [I 2022-10-28 01:17:52] ...trap/engines/engine.py.229: process: 0.54466 | load: 0.00243 [I 2022-10-28 01:17:52] ...trap/engines/engine.py.230: loss: 0.13900 [I 2022-10-28 01:17:52] ...optimizers/trijoint.py.151: total_norm: 13.760421 [I 2022-10-28 01:17:57] ...trap/engines/engine.py.224: train: epoch 98 | batch 1020/2304 [I 2022-10-28 01:17:57] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:56 [I 2022-10-28 01:17:57] ...trap/engines/engine.py.229: process: 0.56273 | load: 0.00242 [I 2022-10-28 01:17:57] ...trap/engines/engine.py.230: loss: 0.14504 [I 2022-10-28 01:17:57] ...optimizers/trijoint.py.151: total_norm: 33.466797 [I 2022-10-28 01:18:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 1030/2304 [I 2022-10-28 01:18:03] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:53 [I 2022-10-28 01:18:03] ...trap/engines/engine.py.229: process: 0.56082 | load: 0.00240 [I 2022-10-28 01:18:03] ...trap/engines/engine.py.230: loss: 0.16155 [I 2022-10-28 01:18:03] ...optimizers/trijoint.py.151: total_norm: 23.206566 [I 2022-10-28 01:18:09] ...trap/engines/engine.py.224: train: epoch 98 | batch 1040/2304 [I 2022-10-28 01:18:09] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:53 [I 2022-10-28 01:18:09] ...trap/engines/engine.py.229: process: 0.54093 | load: 0.00244 [I 2022-10-28 01:18:09] ...trap/engines/engine.py.230: loss: 0.12551 [I 2022-10-28 01:18:09] ...optimizers/trijoint.py.151: total_norm: 21.038483 [I 2022-10-28 01:18:14] ...trap/engines/engine.py.224: train: epoch 98 | batch 1050/2304 [I 2022-10-28 01:18:14] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:38 [I 2022-10-28 01:18:14] ...trap/engines/engine.py.229: process: 0.54891 | load: 0.00239 [I 2022-10-28 01:18:14] ...trap/engines/engine.py.230: loss: 0.14266 [I 2022-10-28 01:18:14] ...optimizers/trijoint.py.151: total_norm: 17.491299 [I 2022-10-28 01:18:20] ...trap/engines/engine.py.224: train: epoch 98 | batch 1060/2304 [I 2022-10-28 01:18:20] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:33 [I 2022-10-28 01:18:20] ...trap/engines/engine.py.229: process: 0.56427 | load: 0.00240 [I 2022-10-28 01:18:20] ...trap/engines/engine.py.230: loss: 0.11554 [I 2022-10-28 01:18:20] ...optimizers/trijoint.py.151: total_norm: 13.313229 [I 2022-10-28 01:18:25] ...trap/engines/engine.py.224: train: epoch 98 | batch 1070/2304 [I 2022-10-28 01:18:25] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:22 [I 2022-10-28 01:18:25] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00250 [I 2022-10-28 01:18:25] ...trap/engines/engine.py.230: loss: 0.16132 [I 2022-10-28 01:18:25] ...optimizers/trijoint.py.151: total_norm: 26.045233 [I 2022-10-28 01:18:31] ...trap/engines/engine.py.224: train: epoch 98 | batch 1080/2304 [I 2022-10-28 01:18:31] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:18 [I 2022-10-28 01:18:31] ...trap/engines/engine.py.229: process: 0.56214 | load: 0.00247 [I 2022-10-28 01:18:31] ...trap/engines/engine.py.230: loss: 0.10475 [I 2022-10-28 01:18:31] ...optimizers/trijoint.py.151: total_norm: 23.318047 [I 2022-10-28 01:18:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 1090/2304 [I 2022-10-28 01:18:37] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:11:17 [I 2022-10-28 01:18:37] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00244 [I 2022-10-28 01:18:37] ...trap/engines/engine.py.230: loss: 0.08134 [I 2022-10-28 01:18:37] ...optimizers/trijoint.py.151: total_norm: 17.124361 [I 2022-10-28 01:18:42] ...trap/engines/engine.py.224: train: epoch 98 | batch 1100/2304 [I 2022-10-28 01:18:42] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:11:11 [I 2022-10-28 01:18:42] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00245 [I 2022-10-28 01:18:42] ...trap/engines/engine.py.230: loss: 0.19676 [I 2022-10-28 01:18:42] ...optimizers/trijoint.py.151: total_norm: 23.881443 [I 2022-10-28 01:18:48] ...trap/engines/engine.py.224: train: epoch 98 | batch 1110/2304 [I 2022-10-28 01:18:48] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:11:03 [I 2022-10-28 01:18:48] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00284 [I 2022-10-28 01:18:48] ...trap/engines/engine.py.230: loss: 0.13088 [I 2022-10-28 01:18:48] ...optimizers/trijoint.py.151: total_norm: 14.103917 [I 2022-10-28 01:18:53] ...trap/engines/engine.py.224: train: epoch 98 | batch 1120/2304 [I 2022-10-28 01:18:53] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:10:58 [I 2022-10-28 01:18:53] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00241 [I 2022-10-28 01:18:53] ...trap/engines/engine.py.230: loss: 0.15637 [I 2022-10-28 01:18:53] ...optimizers/trijoint.py.151: total_norm: 17.877195 [I 2022-10-28 01:18:59] ...trap/engines/engine.py.224: train: epoch 98 | batch 1130/2304 [I 2022-10-28 01:18:59] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:59 [I 2022-10-28 01:18:59] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00241 [I 2022-10-28 01:18:59] ...trap/engines/engine.py.230: loss: 0.09977 [I 2022-10-28 01:18:59] ...optimizers/trijoint.py.151: total_norm: 13.048259 [I 2022-10-28 01:19:05] ...trap/engines/engine.py.224: train: epoch 98 | batch 1140/2304 [I 2022-10-28 01:19:05] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:55 [I 2022-10-28 01:19:05] ...trap/engines/engine.py.229: process: 0.56475 | load: 0.00241 [I 2022-10-28 01:19:05] ...trap/engines/engine.py.230: loss: 0.10903 [I 2022-10-28 01:19:05] ...optimizers/trijoint.py.151: total_norm: 9.986947 [I 2022-10-28 01:19:10] ...trap/engines/engine.py.224: train: epoch 98 | batch 1150/2304 [I 2022-10-28 01:19:10] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:11:05 [I 2022-10-28 01:19:10] ...trap/engines/engine.py.229: process: 0.56939 | load: 0.00246 [I 2022-10-28 01:19:10] ...trap/engines/engine.py.230: loss: 0.11570 [I 2022-10-28 01:19:10] ...optimizers/trijoint.py.151: total_norm: 12.224528 [I 2022-10-28 01:19:16] ...trap/engines/engine.py.224: train: epoch 98 | batch 1160/2304 [I 2022-10-28 01:19:16] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:39 [I 2022-10-28 01:19:16] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00244 [I 2022-10-28 01:19:16] ...trap/engines/engine.py.230: loss: 0.11506 [I 2022-10-28 01:19:16] ...optimizers/trijoint.py.151: total_norm: 19.621872 [I 2022-10-28 01:19:22] ...trap/engines/engine.py.224: train: epoch 98 | batch 1170/2304 [I 2022-10-28 01:19:22] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:29 [I 2022-10-28 01:19:22] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00244 [I 2022-10-28 01:19:22] ...trap/engines/engine.py.230: loss: 0.16021 [I 2022-10-28 01:19:22] ...optimizers/trijoint.py.151: total_norm: 17.948097 [I 2022-10-28 01:19:27] ...trap/engines/engine.py.224: train: epoch 98 | batch 1180/2304 [I 2022-10-28 01:19:27] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:27 [I 2022-10-28 01:19:27] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00262 [I 2022-10-28 01:19:27] ...trap/engines/engine.py.230: loss: 0.09242 [I 2022-10-28 01:19:27] ...optimizers/trijoint.py.151: total_norm: 9.564096 [I 2022-10-28 01:19:33] ...trap/engines/engine.py.224: train: epoch 98 | batch 1190/2304 [I 2022-10-28 01:19:33] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:10:19 [I 2022-10-28 01:19:33] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00246 [I 2022-10-28 01:19:33] ...trap/engines/engine.py.230: loss: 0.17880 [I 2022-10-28 01:19:33] ...optimizers/trijoint.py.151: total_norm: 18.747339 [I 2022-10-28 01:19:38] ...trap/engines/engine.py.224: train: epoch 98 | batch 1200/2304 [I 2022-10-28 01:19:38] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:10:13 [I 2022-10-28 01:19:38] ...trap/engines/engine.py.229: process: 0.55225 | load: 0.00242 [I 2022-10-28 01:19:38] ...trap/engines/engine.py.230: loss: 0.12375 [I 2022-10-28 01:19:38] ...optimizers/trijoint.py.151: total_norm: 18.488594 [I 2022-10-28 01:19:44] ...trap/engines/engine.py.224: train: epoch 98 | batch 1210/2304 [I 2022-10-28 01:19:44] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:10:01 [I 2022-10-28 01:19:44] ...trap/engines/engine.py.229: process: 0.54271 | load: 0.00240 [I 2022-10-28 01:19:44] ...trap/engines/engine.py.230: loss: 0.13285 [I 2022-10-28 01:19:44] ...optimizers/trijoint.py.151: total_norm: 17.750338 [I 2022-10-28 01:19:49] ...trap/engines/engine.py.224: train: epoch 98 | batch 1220/2304 [I 2022-10-28 01:19:49] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:10:01 [I 2022-10-28 01:19:49] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00242 [I 2022-10-28 01:19:49] ...trap/engines/engine.py.230: loss: 0.11340 [I 2022-10-28 01:19:49] ...optimizers/trijoint.py.151: total_norm: 19.271400 [I 2022-10-28 01:19:55] ...trap/engines/engine.py.224: train: epoch 98 | batch 1230/2304 [I 2022-10-28 01:19:55] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:55 [I 2022-10-28 01:19:55] ...trap/engines/engine.py.229: process: 0.54615 | load: 0.00239 [I 2022-10-28 01:19:55] ...trap/engines/engine.py.230: loss: 0.10630 [I 2022-10-28 01:19:55] ...optimizers/trijoint.py.151: total_norm: 7.721768 [I 2022-10-28 01:20:01] ...trap/engines/engine.py.224: train: epoch 98 | batch 1240/2304 [I 2022-10-28 01:20:01] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:53 [I 2022-10-28 01:20:01] ...trap/engines/engine.py.229: process: 0.55624 | load: 0.00243 [I 2022-10-28 01:20:01] ...trap/engines/engine.py.230: loss: 0.10225 [I 2022-10-28 01:20:01] ...optimizers/trijoint.py.151: total_norm: 36.264595 [I 2022-10-28 01:20:06] ...trap/engines/engine.py.224: train: epoch 98 | batch 1250/2304 [I 2022-10-28 01:20:06] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:50 [I 2022-10-28 01:20:06] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00239 [I 2022-10-28 01:20:06] ...trap/engines/engine.py.230: loss: 0.09746 [I 2022-10-28 01:20:06] ...optimizers/trijoint.py.151: total_norm: 19.355778 [I 2022-10-28 01:20:12] ...trap/engines/engine.py.224: train: epoch 98 | batch 1260/2304 [I 2022-10-28 01:20:12] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:51 [I 2022-10-28 01:20:12] ...trap/engines/engine.py.229: process: 0.56293 | load: 0.00249 [I 2022-10-28 01:20:12] ...trap/engines/engine.py.230: loss: 0.10473 [I 2022-10-28 01:20:12] ...optimizers/trijoint.py.151: total_norm: 16.753916 [I 2022-10-28 01:20:18] ...trap/engines/engine.py.224: train: epoch 98 | batch 1270/2304 [I 2022-10-28 01:20:18] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:35 [I 2022-10-28 01:20:18] ...trap/engines/engine.py.229: process: 0.54823 | load: 0.00246 [I 2022-10-28 01:20:18] ...trap/engines/engine.py.230: loss: 0.10703 [I 2022-10-28 01:20:18] ...optimizers/trijoint.py.151: total_norm: 9.981808 [I 2022-10-28 01:20:23] ...trap/engines/engine.py.224: train: epoch 98 | batch 1280/2304 [I 2022-10-28 01:20:23] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:26 [I 2022-10-28 01:20:23] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00244 [I 2022-10-28 01:20:23] ...trap/engines/engine.py.230: loss: 0.10239 [I 2022-10-28 01:20:23] ...optimizers/trijoint.py.151: total_norm: 13.288323 [I 2022-10-28 01:20:29] ...trap/engines/engine.py.224: train: epoch 98 | batch 1290/2304 [I 2022-10-28 01:20:29] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:21 [I 2022-10-28 01:20:29] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00245 [I 2022-10-28 01:20:29] ...trap/engines/engine.py.230: loss: 0.11289 [I 2022-10-28 01:20:29] ...optimizers/trijoint.py.151: total_norm: 13.638953 [I 2022-10-28 01:20:34] ...trap/engines/engine.py.224: train: epoch 98 | batch 1300/2304 [I 2022-10-28 01:20:34] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:09:22 [I 2022-10-28 01:20:34] ...trap/engines/engine.py.229: process: 0.56224 | load: 0.00246 [I 2022-10-28 01:20:34] ...trap/engines/engine.py.230: loss: 0.07905 [I 2022-10-28 01:20:34] ...optimizers/trijoint.py.151: total_norm: 11.213996 [I 2022-10-28 01:20:40] ...trap/engines/engine.py.224: train: epoch 98 | batch 1310/2304 [I 2022-10-28 01:20:40] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:09:16 [I 2022-10-28 01:20:40] ...trap/engines/engine.py.229: process: 0.56786 | load: 0.00242 [I 2022-10-28 01:20:40] ...trap/engines/engine.py.230: loss: 0.13694 [I 2022-10-28 01:20:40] ...optimizers/trijoint.py.151: total_norm: 15.369304 [I 2022-10-28 01:20:45] ...trap/engines/engine.py.224: train: epoch 98 | batch 1320/2304 [I 2022-10-28 01:20:45] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:09:03 [I 2022-10-28 01:20:45] ...trap/engines/engine.py.229: process: 0.55120 | load: 0.00241 [I 2022-10-28 01:20:45] ...trap/engines/engine.py.230: loss: 0.09611 [I 2022-10-28 01:20:45] ...optimizers/trijoint.py.151: total_norm: 11.015631 [I 2022-10-28 01:20:51] ...trap/engines/engine.py.224: train: epoch 98 | batch 1330/2304 [I 2022-10-28 01:20:51] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:09:00 [I 2022-10-28 01:20:51] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00244 [I 2022-10-28 01:20:51] ...trap/engines/engine.py.230: loss: 0.11084 [I 2022-10-28 01:20:51] ...optimizers/trijoint.py.151: total_norm: 18.579319 [I 2022-10-28 01:20:57] ...trap/engines/engine.py.224: train: epoch 98 | batch 1340/2304 [I 2022-10-28 01:20:57] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:52 [I 2022-10-28 01:20:57] ...trap/engines/engine.py.229: process: 0.53834 | load: 0.00239 [I 2022-10-28 01:20:57] ...trap/engines/engine.py.230: loss: 0.11132 [I 2022-10-28 01:20:57] ...optimizers/trijoint.py.151: total_norm: 11.564282 [I 2022-10-28 01:21:02] ...trap/engines/engine.py.224: train: epoch 98 | batch 1350/2304 [I 2022-10-28 01:21:02] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:49 [I 2022-10-28 01:21:02] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00242 [I 2022-10-28 01:21:02] ...trap/engines/engine.py.230: loss: 0.10113 [I 2022-10-28 01:21:02] ...optimizers/trijoint.py.151: total_norm: 11.444195 [I 2022-10-28 01:21:08] ...trap/engines/engine.py.224: train: epoch 98 | batch 1360/2304 [I 2022-10-28 01:21:08] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:45 [I 2022-10-28 01:21:08] ...trap/engines/engine.py.229: process: 0.56141 | load: 0.00242 [I 2022-10-28 01:21:08] ...trap/engines/engine.py.230: loss: 0.15460 [I 2022-10-28 01:21:08] ...optimizers/trijoint.py.151: total_norm: 7.949623 [I 2022-10-28 01:21:13] ...trap/engines/engine.py.224: train: epoch 98 | batch 1370/2304 [I 2022-10-28 01:21:13] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:35 [I 2022-10-28 01:21:13] ...trap/engines/engine.py.229: process: 0.55143 | load: 0.00241 [I 2022-10-28 01:21:13] ...trap/engines/engine.py.230: loss: 0.13031 [I 2022-10-28 01:21:13] ...optimizers/trijoint.py.151: total_norm: 8.477085 [I 2022-10-28 01:21:19] ...trap/engines/engine.py.224: train: epoch 98 | batch 1380/2304 [I 2022-10-28 01:21:19] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:32 [I 2022-10-28 01:21:19] ...trap/engines/engine.py.229: process: 0.55341 | load: 0.00242 [I 2022-10-28 01:21:19] ...trap/engines/engine.py.230: loss: 0.14834 [I 2022-10-28 01:21:19] ...optimizers/trijoint.py.151: total_norm: 18.702534 [I 2022-10-28 01:21:24] ...trap/engines/engine.py.224: train: epoch 98 | batch 1390/2304 [I 2022-10-28 01:21:24] ...trap/engines/engine.py.228: elapsed: 0:13:21 | left: 0:08:32 [I 2022-10-28 01:21:24] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00240 [I 2022-10-28 01:21:24] ...trap/engines/engine.py.230: loss: 0.11911 [I 2022-10-28 01:21:24] ...optimizers/trijoint.py.151: total_norm: 16.263117 [I 2022-10-28 01:21:30] ...trap/engines/engine.py.224: train: epoch 98 | batch 1400/2304 [I 2022-10-28 01:21:30] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:24 [I 2022-10-28 01:21:30] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00242 [I 2022-10-28 01:21:30] ...trap/engines/engine.py.230: loss: 0.12199 [I 2022-10-28 01:21:30] ...optimizers/trijoint.py.151: total_norm: 18.093153 [I 2022-10-28 01:21:36] ...trap/engines/engine.py.224: train: epoch 98 | batch 1410/2304 [I 2022-10-28 01:21:36] ...trap/engines/engine.py.228: elapsed: 0:13:32 | left: 0:08:17 [I 2022-10-28 01:21:36] ...trap/engines/engine.py.229: process: 0.55532 | load: 0.00242 [I 2022-10-28 01:21:36] ...trap/engines/engine.py.230: loss: 0.10261 [I 2022-10-28 01:21:36] ...optimizers/trijoint.py.151: total_norm: 10.312298 [I 2022-10-28 01:21:41] ...trap/engines/engine.py.224: train: epoch 98 | batch 1420/2304 [I 2022-10-28 01:21:41] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:08:08 [I 2022-10-28 01:21:41] ...trap/engines/engine.py.229: process: 0.54501 | load: 0.00243 [I 2022-10-28 01:21:41] ...trap/engines/engine.py.230: loss: 0.08458 [I 2022-10-28 01:21:41] ...optimizers/trijoint.py.151: total_norm: 12.385785 [I 2022-10-28 01:21:47] ...trap/engines/engine.py.224: train: epoch 98 | batch 1430/2304 [I 2022-10-28 01:21:47] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:08:06 [I 2022-10-28 01:21:47] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00241 [I 2022-10-28 01:21:47] ...trap/engines/engine.py.230: loss: 0.12322 [I 2022-10-28 01:21:47] ...optimizers/trijoint.py.151: total_norm: 9.947922 [I 2022-10-28 01:21:52] ...trap/engines/engine.py.224: train: epoch 98 | batch 1440/2304 [I 2022-10-28 01:21:52] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:58 [I 2022-10-28 01:21:52] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00240 [I 2022-10-28 01:21:52] ...trap/engines/engine.py.230: loss: 0.15463 [I 2022-10-28 01:21:52] ...optimizers/trijoint.py.151: total_norm: 16.604181 [I 2022-10-28 01:21:58] ...trap/engines/engine.py.224: train: epoch 98 | batch 1450/2304 [I 2022-10-28 01:21:58] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:53 [I 2022-10-28 01:21:58] ...trap/engines/engine.py.229: process: 0.54922 | load: 0.00245 [I 2022-10-28 01:21:58] ...trap/engines/engine.py.230: loss: 0.14236 [I 2022-10-28 01:21:58] ...optimizers/trijoint.py.151: total_norm: 23.112137 [I 2022-10-28 01:22:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 1460/2304 [I 2022-10-28 01:22:03] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:49 [I 2022-10-28 01:22:03] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00241 [I 2022-10-28 01:22:03] ...trap/engines/engine.py.230: loss: 0.12285 [I 2022-10-28 01:22:03] ...optimizers/trijoint.py.151: total_norm: 15.378388 [I 2022-10-28 01:22:09] ...trap/engines/engine.py.224: train: epoch 98 | batch 1470/2304 [I 2022-10-28 01:22:09] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:44 [I 2022-10-28 01:22:09] ...trap/engines/engine.py.229: process: 0.55343 | load: 0.00242 [I 2022-10-28 01:22:09] ...trap/engines/engine.py.230: loss: 0.09735 [I 2022-10-28 01:22:09] ...optimizers/trijoint.py.151: total_norm: 10.367403 [I 2022-10-28 01:22:15] ...trap/engines/engine.py.224: train: epoch 98 | batch 1480/2304 [I 2022-10-28 01:22:15] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:37 [I 2022-10-28 01:22:15] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00247 [I 2022-10-28 01:22:15] ...trap/engines/engine.py.230: loss: 0.07708 [I 2022-10-28 01:22:15] ...optimizers/trijoint.py.151: total_norm: 8.710989 [I 2022-10-28 01:22:20] ...trap/engines/engine.py.224: train: epoch 98 | batch 1490/2304 [I 2022-10-28 01:22:20] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:33 [I 2022-10-28 01:22:20] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00243 [I 2022-10-28 01:22:20] ...trap/engines/engine.py.230: loss: 0.14609 [I 2022-10-28 01:22:20] ...optimizers/trijoint.py.151: total_norm: 14.512166 [I 2022-10-28 01:22:26] ...trap/engines/engine.py.224: train: epoch 98 | batch 1500/2304 [I 2022-10-28 01:22:26] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:27 [I 2022-10-28 01:22:26] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00243 [I 2022-10-28 01:22:26] ...trap/engines/engine.py.230: loss: 0.10575 [I 2022-10-28 01:22:26] ...optimizers/trijoint.py.151: total_norm: 29.609409 [I 2022-10-28 01:22:31] ...trap/engines/engine.py.224: train: epoch 98 | batch 1510/2304 [I 2022-10-28 01:22:31] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:20 [I 2022-10-28 01:22:31] ...trap/engines/engine.py.229: process: 0.54897 | load: 0.00243 [I 2022-10-28 01:22:31] ...trap/engines/engine.py.230: loss: 0.10558 [I 2022-10-28 01:22:31] ...optimizers/trijoint.py.151: total_norm: 17.151842 [I 2022-10-28 01:22:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 1520/2304 [I 2022-10-28 01:22:37] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:17 [I 2022-10-28 01:22:37] ...trap/engines/engine.py.229: process: 0.57186 | load: 0.00251 [I 2022-10-28 01:22:37] ...trap/engines/engine.py.230: loss: 0.11670 [I 2022-10-28 01:22:37] ...optimizers/trijoint.py.151: total_norm: 21.421240 [I 2022-10-28 01:22:42] ...trap/engines/engine.py.224: train: epoch 98 | batch 1530/2304 [I 2022-10-28 01:22:42] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:07:09 [I 2022-10-28 01:22:42] ...trap/engines/engine.py.229: process: 0.54674 | load: 0.00246 [I 2022-10-28 01:22:42] ...trap/engines/engine.py.230: loss: 0.09266 [I 2022-10-28 01:22:42] ...optimizers/trijoint.py.151: total_norm: 8.536326 [I 2022-10-28 01:22:48] ...trap/engines/engine.py.224: train: epoch 98 | batch 1540/2304 [I 2022-10-28 01:22:48] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:07:03 [I 2022-10-28 01:22:48] ...trap/engines/engine.py.229: process: 0.54030 | load: 0.00248 [I 2022-10-28 01:22:48] ...trap/engines/engine.py.230: loss: 0.11635 [I 2022-10-28 01:22:48] ...optimizers/trijoint.py.151: total_norm: 20.226072 [I 2022-10-28 01:22:54] ...trap/engines/engine.py.224: train: epoch 98 | batch 1550/2304 [I 2022-10-28 01:22:54] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:07:02 [I 2022-10-28 01:22:54] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00243 [I 2022-10-28 01:22:54] ...trap/engines/engine.py.230: loss: 0.09854 [I 2022-10-28 01:22:54] ...optimizers/trijoint.py.151: total_norm: 9.983159 [I 2022-10-28 01:22:59] ...trap/engines/engine.py.224: train: epoch 98 | batch 1560/2304 [I 2022-10-28 01:22:59] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:59 [I 2022-10-28 01:22:59] ...trap/engines/engine.py.229: process: 0.56884 | load: 0.00242 [I 2022-10-28 01:22:59] ...trap/engines/engine.py.230: loss: 0.10120 [I 2022-10-28 01:22:59] ...optimizers/trijoint.py.151: total_norm: 9.929267 [I 2022-10-28 01:23:05] ...trap/engines/engine.py.224: train: epoch 98 | batch 1570/2304 [I 2022-10-28 01:23:05] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:48 [I 2022-10-28 01:23:05] ...trap/engines/engine.py.229: process: 0.54806 | load: 0.00241 [I 2022-10-28 01:23:05] ...trap/engines/engine.py.230: loss: 0.08901 [I 2022-10-28 01:23:05] ...optimizers/trijoint.py.151: total_norm: 13.718376 [I 2022-10-28 01:23:10] ...trap/engines/engine.py.224: train: epoch 98 | batch 1580/2304 [I 2022-10-28 01:23:10] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:38 [I 2022-10-28 01:23:10] ...trap/engines/engine.py.229: process: 0.54798 | load: 0.00241 [I 2022-10-28 01:23:10] ...trap/engines/engine.py.230: loss: 0.17163 [I 2022-10-28 01:23:10] ...optimizers/trijoint.py.151: total_norm: 12.101623 [I 2022-10-28 01:23:16] ...trap/engines/engine.py.224: train: epoch 98 | batch 1590/2304 [I 2022-10-28 01:23:16] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:42 [I 2022-10-28 01:23:16] ...trap/engines/engine.py.229: process: 0.57692 | load: 0.00243 [I 2022-10-28 01:23:16] ...trap/engines/engine.py.230: loss: 0.13021 [I 2022-10-28 01:23:16] ...optimizers/trijoint.py.151: total_norm: 27.328108 [I 2022-10-28 01:23:22] ...trap/engines/engine.py.224: train: epoch 98 | batch 1600/2304 [I 2022-10-28 01:23:22] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:35 [I 2022-10-28 01:23:22] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00242 [I 2022-10-28 01:23:22] ...trap/engines/engine.py.230: loss: 0.12182 [I 2022-10-28 01:23:22] ...optimizers/trijoint.py.151: total_norm: 28.324068 [I 2022-10-28 01:23:27] ...trap/engines/engine.py.224: train: epoch 98 | batch 1610/2304 [I 2022-10-28 01:23:27] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:27 [I 2022-10-28 01:23:27] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00242 [I 2022-10-28 01:23:27] ...trap/engines/engine.py.230: loss: 0.12883 [I 2022-10-28 01:23:27] ...optimizers/trijoint.py.151: total_norm: 20.817436 [I 2022-10-28 01:23:33] ...trap/engines/engine.py.224: train: epoch 98 | batch 1620/2304 [I 2022-10-28 01:23:33] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:17 [I 2022-10-28 01:23:33] ...trap/engines/engine.py.229: process: 0.54540 | load: 0.00260 [I 2022-10-28 01:23:33] ...trap/engines/engine.py.230: loss: 0.08806 [I 2022-10-28 01:23:33] ...optimizers/trijoint.py.151: total_norm: 17.280315 [I 2022-10-28 01:23:38] ...trap/engines/engine.py.224: train: epoch 98 | batch 1630/2304 [I 2022-10-28 01:23:38] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:06:17 [I 2022-10-28 01:23:38] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00241 [I 2022-10-28 01:23:38] ...trap/engines/engine.py.230: loss: 0.09887 [I 2022-10-28 01:23:38] ...optimizers/trijoint.py.151: total_norm: 9.225956 [I 2022-10-28 01:23:44] ...trap/engines/engine.py.224: train: epoch 98 | batch 1640/2304 [I 2022-10-28 01:23:44] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:06:10 [I 2022-10-28 01:23:44] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00243 [I 2022-10-28 01:23:44] ...trap/engines/engine.py.230: loss: 0.10034 [I 2022-10-28 01:23:44] ...optimizers/trijoint.py.151: total_norm: 11.507215 [I 2022-10-28 01:23:49] ...trap/engines/engine.py.224: train: epoch 98 | batch 1650/2304 [I 2022-10-28 01:23:49] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:06:04 [I 2022-10-28 01:23:49] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00240 [I 2022-10-28 01:23:49] ...trap/engines/engine.py.230: loss: 0.12858 [I 2022-10-28 01:23:49] ...optimizers/trijoint.py.151: total_norm: 12.243578 [I 2022-10-28 01:23:55] ...trap/engines/engine.py.224: train: epoch 98 | batch 1660/2304 [I 2022-10-28 01:23:55] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:58 [I 2022-10-28 01:23:55] ...trap/engines/engine.py.229: process: 0.56667 | load: 0.00254 [I 2022-10-28 01:23:55] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-28 01:23:55] ...optimizers/trijoint.py.151: total_norm: 11.396637 [I 2022-10-28 01:24:01] ...trap/engines/engine.py.224: train: epoch 98 | batch 1670/2304 [I 2022-10-28 01:24:01] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:52 [I 2022-10-28 01:24:01] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00247 [I 2022-10-28 01:24:01] ...trap/engines/engine.py.230: loss: 0.10286 [I 2022-10-28 01:24:01] ...optimizers/trijoint.py.151: total_norm: 9.004050 [I 2022-10-28 01:24:06] ...trap/engines/engine.py.224: train: epoch 98 | batch 1680/2304 [I 2022-10-28 01:24:06] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:45 [I 2022-10-28 01:24:06] ...trap/engines/engine.py.229: process: 0.53452 | load: 0.00246 [I 2022-10-28 01:24:06] ...trap/engines/engine.py.230: loss: 0.11316 [I 2022-10-28 01:24:06] ...optimizers/trijoint.py.151: total_norm: 6.597588 [I 2022-10-28 01:24:12] ...trap/engines/engine.py.224: train: epoch 98 | batch 1690/2304 [I 2022-10-28 01:24:12] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:42 [I 2022-10-28 01:24:12] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00245 [I 2022-10-28 01:24:12] ...trap/engines/engine.py.230: loss: 0.12059 [I 2022-10-28 01:24:12] ...optimizers/trijoint.py.151: total_norm: 11.994867 [I 2022-10-28 01:24:17] ...trap/engines/engine.py.224: train: epoch 98 | batch 1700/2304 [I 2022-10-28 01:24:17] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:38 [I 2022-10-28 01:24:17] ...trap/engines/engine.py.229: process: 0.55828 | load: 0.00246 [I 2022-10-28 01:24:17] ...trap/engines/engine.py.230: loss: 0.13536 [I 2022-10-28 01:24:17] ...optimizers/trijoint.py.151: total_norm: 13.547832 [I 2022-10-28 01:24:23] ...trap/engines/engine.py.224: train: epoch 98 | batch 1710/2304 [I 2022-10-28 01:24:23] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:29 [I 2022-10-28 01:24:23] ...trap/engines/engine.py.229: process: 0.54627 | load: 0.00240 [I 2022-10-28 01:24:23] ...trap/engines/engine.py.230: loss: 0.10441 [I 2022-10-28 01:24:23] ...optimizers/trijoint.py.151: total_norm: 8.063811 [I 2022-10-28 01:24:28] ...trap/engines/engine.py.224: train: epoch 98 | batch 1720/2304 [I 2022-10-28 01:24:28] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:25 [I 2022-10-28 01:24:28] ...trap/engines/engine.py.229: process: 0.55858 | load: 0.00243 [I 2022-10-28 01:24:28] ...trap/engines/engine.py.230: loss: 0.09246 [I 2022-10-28 01:24:28] ...optimizers/trijoint.py.151: total_norm: 14.290638 [I 2022-10-28 01:24:34] ...trap/engines/engine.py.224: train: epoch 98 | batch 1730/2304 [I 2022-10-28 01:24:34] ...trap/engines/engine.py.228: elapsed: 0:16:30 | left: 0:05:19 [I 2022-10-28 01:24:34] ...trap/engines/engine.py.229: process: 0.55335 | load: 0.00244 [I 2022-10-28 01:24:34] ...trap/engines/engine.py.230: loss: 0.09273 [I 2022-10-28 01:24:34] ...optimizers/trijoint.py.151: total_norm: 10.132034 [I 2022-10-28 01:24:40] ...trap/engines/engine.py.224: train: epoch 98 | batch 1740/2304 [I 2022-10-28 01:24:40] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:05:13 [I 2022-10-28 01:24:40] ...trap/engines/engine.py.229: process: 0.55920 | load: 0.00243 [I 2022-10-28 01:24:40] ...trap/engines/engine.py.230: loss: 0.18192 [I 2022-10-28 01:24:40] ...optimizers/trijoint.py.151: total_norm: 20.997581 [I 2022-10-28 01:24:45] ...trap/engines/engine.py.224: train: epoch 98 | batch 1750/2304 [I 2022-10-28 01:24:45] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:07 [I 2022-10-28 01:24:45] ...trap/engines/engine.py.229: process: 0.55096 | load: 0.00241 [I 2022-10-28 01:24:45] ...trap/engines/engine.py.230: loss: 0.17634 [I 2022-10-28 01:24:45] ...optimizers/trijoint.py.151: total_norm: 18.888735 [I 2022-10-28 01:24:51] ...trap/engines/engine.py.224: train: epoch 98 | batch 1760/2304 [I 2022-10-28 01:24:51] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:05:03 [I 2022-10-28 01:24:51] ...trap/engines/engine.py.229: process: 0.54329 | load: 0.00242 [I 2022-10-28 01:24:51] ...trap/engines/engine.py.230: loss: 0.10619 [I 2022-10-28 01:24:51] ...optimizers/trijoint.py.151: total_norm: 10.344645 [I 2022-10-28 01:24:56] ...trap/engines/engine.py.224: train: epoch 98 | batch 1770/2304 [I 2022-10-28 01:24:56] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:58 [I 2022-10-28 01:24:56] ...trap/engines/engine.py.229: process: 0.56005 | load: 0.00241 [I 2022-10-28 01:24:56] ...trap/engines/engine.py.230: loss: 0.10999 [I 2022-10-28 01:24:56] ...optimizers/trijoint.py.151: total_norm: 22.968809 [I 2022-10-28 01:25:02] ...trap/engines/engine.py.224: train: epoch 98 | batch 1780/2304 [I 2022-10-28 01:25:02] ...trap/engines/engine.py.228: elapsed: 0:16:58 | left: 0:04:53 [I 2022-10-28 01:25:02] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00241 [I 2022-10-28 01:25:02] ...trap/engines/engine.py.230: loss: 0.08344 [I 2022-10-28 01:25:02] ...optimizers/trijoint.py.151: total_norm: 10.040192 [I 2022-10-28 01:25:08] ...trap/engines/engine.py.224: train: epoch 98 | batch 1790/2304 [I 2022-10-28 01:25:08] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:46 [I 2022-10-28 01:25:08] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00242 [I 2022-10-28 01:25:08] ...trap/engines/engine.py.230: loss: 0.09879 [I 2022-10-28 01:25:08] ...optimizers/trijoint.py.151: total_norm: 13.704249 [I 2022-10-28 01:25:13] ...trap/engines/engine.py.224: train: epoch 98 | batch 1800/2304 [I 2022-10-28 01:25:13] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:39 [I 2022-10-28 01:25:13] ...trap/engines/engine.py.229: process: 0.55156 | load: 0.00240 [I 2022-10-28 01:25:13] ...trap/engines/engine.py.230: loss: 0.09852 [I 2022-10-28 01:25:13] ...optimizers/trijoint.py.151: total_norm: 20.357418 [I 2022-10-28 01:25:19] ...trap/engines/engine.py.224: train: epoch 98 | batch 1810/2304 [I 2022-10-28 01:25:19] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:35 [I 2022-10-28 01:25:19] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00239 [I 2022-10-28 01:25:19] ...trap/engines/engine.py.230: loss: 0.10393 [I 2022-10-28 01:25:19] ...optimizers/trijoint.py.151: total_norm: 18.910519 [I 2022-10-28 01:25:24] ...trap/engines/engine.py.224: train: epoch 98 | batch 1820/2304 [I 2022-10-28 01:25:24] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:29 [I 2022-10-28 01:25:24] ...trap/engines/engine.py.229: process: 0.55725 | load: 0.00239 [I 2022-10-28 01:25:24] ...trap/engines/engine.py.230: loss: 0.12881 [I 2022-10-28 01:25:24] ...optimizers/trijoint.py.151: total_norm: 12.775500 [I 2022-10-28 01:25:30] ...trap/engines/engine.py.224: train: epoch 98 | batch 1830/2304 [I 2022-10-28 01:25:30] ...trap/engines/engine.py.228: elapsed: 0:17:26 | left: 0:04:23 [I 2022-10-28 01:25:30] ...trap/engines/engine.py.229: process: 0.56901 | load: 0.00239 [I 2022-10-28 01:25:30] ...trap/engines/engine.py.230: loss: 0.12705 [I 2022-10-28 01:25:30] ...optimizers/trijoint.py.151: total_norm: 25.475498 [I 2022-10-28 01:25:35] ...trap/engines/engine.py.224: train: epoch 98 | batch 1840/2304 [I 2022-10-28 01:25:35] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:18 [I 2022-10-28 01:25:35] ...trap/engines/engine.py.229: process: 0.56243 | load: 0.00242 [I 2022-10-28 01:25:35] ...trap/engines/engine.py.230: loss: 0.09435 [I 2022-10-28 01:25:35] ...optimizers/trijoint.py.151: total_norm: 10.726404 [I 2022-10-28 01:25:41] ...trap/engines/engine.py.224: train: epoch 98 | batch 1850/2304 [I 2022-10-28 01:25:41] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:04:11 [I 2022-10-28 01:25:41] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00244 [I 2022-10-28 01:25:41] ...trap/engines/engine.py.230: loss: 0.12878 [I 2022-10-28 01:25:41] ...optimizers/trijoint.py.151: total_norm: 20.140453 [I 2022-10-28 01:25:47] ...trap/engines/engine.py.224: train: epoch 98 | batch 1860/2304 [I 2022-10-28 01:25:47] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:04:06 [I 2022-10-28 01:25:47] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00243 [I 2022-10-28 01:25:47] ...trap/engines/engine.py.230: loss: 0.11279 [I 2022-10-28 01:25:47] ...optimizers/trijoint.py.151: total_norm: 10.028114 [I 2022-10-28 01:25:52] ...trap/engines/engine.py.224: train: epoch 98 | batch 1870/2304 [I 2022-10-28 01:25:52] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:04:00 [I 2022-10-28 01:25:52] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00246 [I 2022-10-28 01:25:52] ...trap/engines/engine.py.230: loss: 0.09752 [I 2022-10-28 01:25:52] ...optimizers/trijoint.py.151: total_norm: 17.243454 [I 2022-10-28 01:25:58] ...trap/engines/engine.py.224: train: epoch 98 | batch 1880/2304 [I 2022-10-28 01:25:58] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:56 [I 2022-10-28 01:25:58] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00242 [I 2022-10-28 01:25:58] ...trap/engines/engine.py.230: loss: 0.11397 [I 2022-10-28 01:25:58] ...optimizers/trijoint.py.151: total_norm: 35.488007 [I 2022-10-28 01:26:03] ...trap/engines/engine.py.224: train: epoch 98 | batch 1890/2304 [I 2022-10-28 01:26:03] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:51 [I 2022-10-28 01:26:03] ...trap/engines/engine.py.229: process: 0.56434 | load: 0.00240 [I 2022-10-28 01:26:03] ...trap/engines/engine.py.230: loss: 0.13062 [I 2022-10-28 01:26:03] ...optimizers/trijoint.py.151: total_norm: 13.338247 [I 2022-10-28 01:26:09] ...trap/engines/engine.py.224: train: epoch 98 | batch 1900/2304 [I 2022-10-28 01:26:09] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:44 [I 2022-10-28 01:26:09] ...trap/engines/engine.py.229: process: 0.56110 | load: 0.00244 [I 2022-10-28 01:26:09] ...trap/engines/engine.py.230: loss: 0.15784 [I 2022-10-28 01:26:09] ...optimizers/trijoint.py.151: total_norm: 31.317272 [I 2022-10-28 01:26:14] ...trap/engines/engine.py.224: train: epoch 98 | batch 1910/2304 [I 2022-10-28 01:26:14] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:39 [I 2022-10-28 01:26:14] ...trap/engines/engine.py.229: process: 0.56165 | load: 0.00242 [I 2022-10-28 01:26:14] ...trap/engines/engine.py.230: loss: 0.10834 [I 2022-10-28 01:26:14] ...optimizers/trijoint.py.151: total_norm: 15.095722 [I 2022-10-28 01:26:20] ...trap/engines/engine.py.224: train: epoch 98 | batch 1920/2304 [I 2022-10-28 01:26:20] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:34 [I 2022-10-28 01:26:20] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00242 [I 2022-10-28 01:26:20] ...trap/engines/engine.py.230: loss: 0.15610 [I 2022-10-28 01:26:20] ...optimizers/trijoint.py.151: total_norm: 28.286308 [I 2022-10-28 01:26:26] ...trap/engines/engine.py.224: train: epoch 98 | batch 1930/2304 [I 2022-10-28 01:26:26] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:28 [I 2022-10-28 01:26:26] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00248 [I 2022-10-28 01:26:26] ...trap/engines/engine.py.230: loss: 0.15014 [I 2022-10-28 01:26:26] ...optimizers/trijoint.py.151: total_norm: 19.802931 [I 2022-10-28 01:26:31] ...trap/engines/engine.py.224: train: epoch 98 | batch 1940/2304 [I 2022-10-28 01:26:31] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:23 [I 2022-10-28 01:26:31] ...trap/engines/engine.py.229: process: 0.56796 | load: 0.00245 [I 2022-10-28 01:26:31] ...trap/engines/engine.py.230: loss: 0.10613 [I 2022-10-28 01:26:31] ...optimizers/trijoint.py.151: total_norm: 17.795082 [I 2022-10-28 01:26:37] ...trap/engines/engine.py.224: train: epoch 98 | batch 1950/2304 [I 2022-10-28 01:26:37] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:18 [I 2022-10-28 01:26:37] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00242 [I 2022-10-28 01:26:37] ...trap/engines/engine.py.230: loss: 0.08745 [I 2022-10-28 01:26:37] ...optimizers/trijoint.py.151: total_norm: 14.672728 [I 2022-10-28 01:26:43] ...trap/engines/engine.py.224: train: epoch 98 | batch 1960/2304 [I 2022-10-28 01:26:43] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:03:12 [I 2022-10-28 01:26:43] ...trap/engines/engine.py.229: process: 0.55104 | load: 0.00243 [I 2022-10-28 01:26:43] ...trap/engines/engine.py.230: loss: 0.14764 [I 2022-10-28 01:26:43] ...optimizers/trijoint.py.151: total_norm: 23.514460 [I 2022-10-28 01:26:48] ...trap/engines/engine.py.224: train: epoch 98 | batch 1970/2304 [I 2022-10-28 01:26:48] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:03:06 [I 2022-10-28 01:26:48] ...trap/engines/engine.py.229: process: 0.56644 | load: 0.00241 [I 2022-10-28 01:26:48] ...trap/engines/engine.py.230: loss: 0.08796 [I 2022-10-28 01:26:48] ...optimizers/trijoint.py.151: total_norm: 15.884580 [I 2022-10-28 01:26:54] ...trap/engines/engine.py.224: train: epoch 98 | batch 1980/2304 [I 2022-10-28 01:26:54] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:03:02 [I 2022-10-28 01:26:54] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00243 [I 2022-10-28 01:26:54] ...trap/engines/engine.py.230: loss: 0.11851 [I 2022-10-28 01:26:54] ...optimizers/trijoint.py.151: total_norm: 11.877274 [I 2022-10-28 01:26:59] ...trap/engines/engine.py.224: train: epoch 98 | batch 1990/2304 [I 2022-10-28 01:26:59] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:55 [I 2022-10-28 01:26:59] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00241 [I 2022-10-28 01:26:59] ...trap/engines/engine.py.230: loss: 0.13254 [I 2022-10-28 01:26:59] ...optimizers/trijoint.py.151: total_norm: 12.458092 [I 2022-10-28 01:27:05] ...trap/engines/engine.py.224: train: epoch 98 | batch 2000/2304 [I 2022-10-28 01:27:05] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:48 [I 2022-10-28 01:27:05] ...trap/engines/engine.py.229: process: 0.54538 | load: 0.00241 [I 2022-10-28 01:27:05] ...trap/engines/engine.py.230: loss: 0.15248 [I 2022-10-28 01:27:05] ...optimizers/trijoint.py.151: total_norm: 19.659266 [I 2022-10-28 01:27:11] ...trap/engines/engine.py.224: train: epoch 98 | batch 2010/2304 [I 2022-10-28 01:27:11] ...trap/engines/engine.py.228: elapsed: 0:19:07 | left: 0:02:43 [I 2022-10-28 01:27:11] ...trap/engines/engine.py.229: process: 0.55979 | load: 0.00241 [I 2022-10-28 01:27:11] ...trap/engines/engine.py.230: loss: 0.13783 [I 2022-10-28 01:27:11] ...optimizers/trijoint.py.151: total_norm: 13.386765 [I 2022-10-28 01:27:16] ...trap/engines/engine.py.224: train: epoch 98 | batch 2020/2304 [I 2022-10-28 01:27:16] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:38 [I 2022-10-28 01:27:16] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00260 [I 2022-10-28 01:27:16] ...trap/engines/engine.py.230: loss: 0.08984 [I 2022-10-28 01:27:16] ...optimizers/trijoint.py.151: total_norm: 16.034033 [I 2022-10-28 01:27:22] ...trap/engines/engine.py.224: train: epoch 98 | batch 2030/2304 [I 2022-10-28 01:27:22] ...trap/engines/engine.py.228: elapsed: 0:19:18 | left: 0:02:31 [I 2022-10-28 01:27:22] ...trap/engines/engine.py.229: process: 0.54466 | load: 0.00242 [I 2022-10-28 01:27:22] ...trap/engines/engine.py.230: loss: 0.09510 [I 2022-10-28 01:27:22] ...optimizers/trijoint.py.151: total_norm: 12.783650 [I 2022-10-28 01:27:27] ...trap/engines/engine.py.224: train: epoch 98 | batch 2040/2304 [I 2022-10-28 01:27:27] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:30 [I 2022-10-28 01:27:27] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00245 [I 2022-10-28 01:27:27] ...trap/engines/engine.py.230: loss: 0.09645 [I 2022-10-28 01:27:27] ...optimizers/trijoint.py.151: total_norm: 9.967238 [I 2022-10-28 01:27:33] ...trap/engines/engine.py.224: train: epoch 98 | batch 2050/2304 [I 2022-10-28 01:27:33] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:22 [I 2022-10-28 01:27:33] ...trap/engines/engine.py.229: process: 0.56773 | load: 0.00243 [I 2022-10-28 01:27:33] ...trap/engines/engine.py.230: loss: 0.13366 [I 2022-10-28 01:27:33] ...optimizers/trijoint.py.151: total_norm: 8.898314 [I 2022-10-28 01:27:39] ...trap/engines/engine.py.224: train: epoch 98 | batch 2060/2304 [I 2022-10-28 01:27:39] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:02:16 [I 2022-10-28 01:27:39] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00243 [I 2022-10-28 01:27:39] ...trap/engines/engine.py.230: loss: 0.08140 [I 2022-10-28 01:27:39] ...optimizers/trijoint.py.151: total_norm: 10.473319 [I 2022-10-28 01:27:44] ...trap/engines/engine.py.224: train: epoch 98 | batch 2070/2304 [I 2022-10-28 01:27:44] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:02:10 [I 2022-10-28 01:27:44] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00241 [I 2022-10-28 01:27:44] ...trap/engines/engine.py.230: loss: 0.10356 [I 2022-10-28 01:27:44] ...optimizers/trijoint.py.151: total_norm: 12.064817 [I 2022-10-28 01:27:50] ...trap/engines/engine.py.224: train: epoch 98 | batch 2080/2304 [I 2022-10-28 01:27:50] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:02:04 [I 2022-10-28 01:27:50] ...trap/engines/engine.py.229: process: 0.56458 | load: 0.00242 [I 2022-10-28 01:27:50] ...trap/engines/engine.py.230: loss: 0.16483 [I 2022-10-28 01:27:50] ...optimizers/trijoint.py.151: total_norm: 19.530703 [I 2022-10-28 01:27:55] ...trap/engines/engine.py.224: train: epoch 98 | batch 2090/2304 [I 2022-10-28 01:27:55] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:59 [I 2022-10-28 01:27:55] ...trap/engines/engine.py.229: process: 0.56675 | load: 0.00243 [I 2022-10-28 01:27:55] ...trap/engines/engine.py.230: loss: 0.11965 [I 2022-10-28 01:27:55] ...optimizers/trijoint.py.151: total_norm: 15.315891 [I 2022-10-28 01:28:01] ...trap/engines/engine.py.224: train: epoch 98 | batch 2100/2304 [I 2022-10-28 01:28:01] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:53 [I 2022-10-28 01:28:01] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00246 [I 2022-10-28 01:28:01] ...trap/engines/engine.py.230: loss: 0.14919 [I 2022-10-28 01:28:01] ...optimizers/trijoint.py.151: total_norm: 10.885280 [I 2022-10-28 01:28:06] ...trap/engines/engine.py.224: train: epoch 98 | batch 2110/2304 [I 2022-10-28 01:28:06] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:48 [I 2022-10-28 01:28:06] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00247 [I 2022-10-28 01:28:06] ...trap/engines/engine.py.230: loss: 0.08116 [I 2022-10-28 01:28:06] ...optimizers/trijoint.py.151: total_norm: 12.915862 [I 2022-10-28 01:28:12] ...trap/engines/engine.py.224: train: epoch 98 | batch 2120/2304 [I 2022-10-28 01:28:12] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:42 [I 2022-10-28 01:28:12] ...trap/engines/engine.py.229: process: 0.55526 | load: 0.00246 [I 2022-10-28 01:28:12] ...trap/engines/engine.py.230: loss: 0.17091 [I 2022-10-28 01:28:12] ...optimizers/trijoint.py.151: total_norm: 39.902584 [I 2022-10-28 01:28:18] ...trap/engines/engine.py.224: train: epoch 98 | batch 2130/2304 [I 2022-10-28 01:28:18] ...trap/engines/engine.py.228: elapsed: 0:20:14 | left: 0:01:36 [I 2022-10-28 01:28:18] ...trap/engines/engine.py.229: process: 0.55242 | load: 0.00244 [I 2022-10-28 01:28:18] ...trap/engines/engine.py.230: loss: 0.10272 [I 2022-10-28 01:28:18] ...optimizers/trijoint.py.151: total_norm: 15.897955 [I 2022-10-28 01:28:23] ...trap/engines/engine.py.224: train: epoch 98 | batch 2140/2304 [I 2022-10-28 01:28:23] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:31 [I 2022-10-28 01:28:23] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00244 [I 2022-10-28 01:28:23] ...trap/engines/engine.py.230: loss: 0.16411 [I 2022-10-28 01:28:23] ...optimizers/trijoint.py.151: total_norm: 19.035732 [I 2022-10-28 01:28:29] ...trap/engines/engine.py.224: train: epoch 98 | batch 2150/2304 [I 2022-10-28 01:28:29] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:26 [I 2022-10-28 01:28:29] ...trap/engines/engine.py.229: process: 0.56595 | load: 0.00244 [I 2022-10-28 01:28:29] ...trap/engines/engine.py.230: loss: 0.16272 [I 2022-10-28 01:28:29] ...optimizers/trijoint.py.151: total_norm: 12.810127 [I 2022-10-28 01:28:34] ...trap/engines/engine.py.224: train: epoch 98 | batch 2160/2304 [I 2022-10-28 01:28:34] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:20 [I 2022-10-28 01:28:34] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00244 [I 2022-10-28 01:28:34] ...trap/engines/engine.py.230: loss: 0.07257 [I 2022-10-28 01:28:34] ...optimizers/trijoint.py.151: total_norm: 11.018172 [I 2022-10-28 01:28:40] ...trap/engines/engine.py.224: train: epoch 98 | batch 2170/2304 [I 2022-10-28 01:28:40] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:14 [I 2022-10-28 01:28:40] ...trap/engines/engine.py.229: process: 0.55164 | load: 0.00244 [I 2022-10-28 01:28:40] ...trap/engines/engine.py.230: loss: 0.17153 [I 2022-10-28 01:28:40] ...optimizers/trijoint.py.151: total_norm: 30.383209 [I 2022-10-28 01:28:45] ...trap/engines/engine.py.224: train: epoch 98 | batch 2180/2304 [I 2022-10-28 01:28:45] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:09 [I 2022-10-28 01:28:45] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00243 [I 2022-10-28 01:28:45] ...trap/engines/engine.py.230: loss: 0.15783 [I 2022-10-28 01:28:45] ...optimizers/trijoint.py.151: total_norm: 13.593569 [I 2022-10-28 01:28:51] ...trap/engines/engine.py.224: train: epoch 98 | batch 2190/2304 [I 2022-10-28 01:28:51] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:01:03 [I 2022-10-28 01:28:51] ...trap/engines/engine.py.229: process: 0.55632 | load: 0.00243 [I 2022-10-28 01:28:51] ...trap/engines/engine.py.230: loss: 0.16394 [I 2022-10-28 01:28:51] ...optimizers/trijoint.py.151: total_norm: 13.074660 [I 2022-10-28 01:28:57] ...trap/engines/engine.py.224: train: epoch 98 | batch 2200/2304 [I 2022-10-28 01:28:57] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:58 [I 2022-10-28 01:28:57] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00243 [I 2022-10-28 01:28:57] ...trap/engines/engine.py.230: loss: 0.11509 [I 2022-10-28 01:28:57] ...optimizers/trijoint.py.151: total_norm: 9.486811 [I 2022-10-28 01:29:02] ...trap/engines/engine.py.224: train: epoch 98 | batch 2210/2304 [I 2022-10-28 01:29:02] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:52 [I 2022-10-28 01:29:02] ...trap/engines/engine.py.229: process: 0.55315 | load: 0.00242 [I 2022-10-28 01:29:02] ...trap/engines/engine.py.230: loss: 0.08561 [I 2022-10-28 01:29:02] ...optimizers/trijoint.py.151: total_norm: 10.021989 [I 2022-10-28 01:29:08] ...trap/engines/engine.py.224: train: epoch 98 | batch 2220/2304 [I 2022-10-28 01:29:08] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:47 [I 2022-10-28 01:29:08] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00243 [I 2022-10-28 01:29:08] ...trap/engines/engine.py.230: loss: 0.15185 [I 2022-10-28 01:29:08] ...optimizers/trijoint.py.151: total_norm: 18.170465 [I 2022-10-28 01:29:13] ...trap/engines/engine.py.224: train: epoch 98 | batch 2230/2304 [I 2022-10-28 01:29:13] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:41 [I 2022-10-28 01:29:13] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00242 [I 2022-10-28 01:29:13] ...trap/engines/engine.py.230: loss: 0.11485 [I 2022-10-28 01:29:13] ...optimizers/trijoint.py.151: total_norm: 15.833568 [I 2022-10-28 01:29:19] ...trap/engines/engine.py.224: train: epoch 98 | batch 2240/2304 [I 2022-10-28 01:29:19] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:35 [I 2022-10-28 01:29:19] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00241 [I 2022-10-28 01:29:19] ...trap/engines/engine.py.230: loss: 0.08903 [I 2022-10-28 01:29:19] ...optimizers/trijoint.py.151: total_norm: 9.130023 [I 2022-10-28 01:29:25] ...trap/engines/engine.py.224: train: epoch 98 | batch 2250/2304 [I 2022-10-28 01:29:25] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:29 [I 2022-10-28 01:29:25] ...trap/engines/engine.py.229: process: 0.54260 | load: 0.00242 [I 2022-10-28 01:29:25] ...trap/engines/engine.py.230: loss: 0.11304 [I 2022-10-28 01:29:25] ...optimizers/trijoint.py.151: total_norm: 13.344646 [I 2022-10-28 01:29:30] ...trap/engines/engine.py.224: train: epoch 98 | batch 2260/2304 [I 2022-10-28 01:29:30] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:24 [I 2022-10-28 01:29:30] ...trap/engines/engine.py.229: process: 0.56109 | load: 0.00240 [I 2022-10-28 01:29:30] ...trap/engines/engine.py.230: loss: 0.10763 [I 2022-10-28 01:29:30] ...optimizers/trijoint.py.151: total_norm: 20.800821 [I 2022-10-28 01:29:36] ...trap/engines/engine.py.224: train: epoch 98 | batch 2270/2304 [I 2022-10-28 01:29:36] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:18 [I 2022-10-28 01:29:36] ...trap/engines/engine.py.229: process: 0.56105 | load: 0.00242 [I 2022-10-28 01:29:36] ...trap/engines/engine.py.230: loss: 0.12722 [I 2022-10-28 01:29:36] ...optimizers/trijoint.py.151: total_norm: 12.851996 [I 2022-10-28 01:29:41] ...trap/engines/engine.py.224: train: epoch 98 | batch 2280/2304 [I 2022-10-28 01:29:41] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:13 [I 2022-10-28 01:29:41] ...trap/engines/engine.py.229: process: 0.55790 | load: 0.00240 [I 2022-10-28 01:29:41] ...trap/engines/engine.py.230: loss: 0.12343 [I 2022-10-28 01:29:41] ...optimizers/trijoint.py.151: total_norm: 10.334103 [I 2022-10-28 01:29:47] ...trap/engines/engine.py.224: train: epoch 98 | batch 2290/2304 [I 2022-10-28 01:29:47] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:00:07 [I 2022-10-28 01:29:47] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00243 [I 2022-10-28 01:29:47] ...trap/engines/engine.py.230: loss: 0.14496 [I 2022-10-28 01:29:47] ...optimizers/trijoint.py.151: total_norm: 15.388614 [I 2022-10-28 01:29:53] ...trap/engines/engine.py.224: train: epoch 98 | batch 2300/2304 [I 2022-10-28 01:29:53] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:02 [I 2022-10-28 01:29:53] ...trap/engines/engine.py.229: process: 0.55761 | load: 0.00017 [I 2022-10-28 01:29:53] ...trap/engines/engine.py.230: loss: 0.08620 [I 2022-10-28 01:29:53] ...optimizers/trijoint.py.151: total_norm: 13.721682 [I 2022-10-28 01:29:55] ...trap/engines/engine.py.224: train: epoch 98 | batch 2304/2304 [I 2022-10-28 01:29:55] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:00 [I 2022-10-28 01:29:55] ...trap/engines/engine.py.229: process: 0.55298 | load: 0.00017 [I 2022-10-28 01:29:55] ...trap/engines/engine.py.230: loss: 0.16895 [I 2022-10-28 01:29:55] ...optimizers/trijoint.py.151: total_norm: 22.861023 [S 2022-10-28 01:29:56] ...trap/engines/engine.py.129: train_epoch.epoch: 98 [S 2022-10-28 01:29:56] ...trap/engines/engine.py.129: train_epoch.loss: 0.12771059964296616 [I 2022-10-28 01:29:56] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 01:29:58] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 98 [I 2022-10-28 01:29:58] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 01:30:00] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 01:30:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 01:30:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 01:30:03] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 01:30:03] ...trap/engines/engine.py.321: eval: epoch 98 | batch 0/510 [I 2022-10-28 01:30:03] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:47:48 [I 2022-10-28 01:30:03] ...trap/engines/engine.py.326: process: 5.62529 | load: 2.52967 [I 2022-10-28 01:30:07] ...trap/engines/engine.py.321: eval: epoch 98 | batch 10/510 [I 2022-10-28 01:30:07] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:29 [I 2022-10-28 01:30:07] ...trap/engines/engine.py.326: process: 0.31992 | load: 0.00016 [I 2022-10-28 01:30:10] ...trap/engines/engine.py.321: eval: epoch 98 | batch 20/510 [I 2022-10-28 01:30:10] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:11 [I 2022-10-28 01:30:10] ...trap/engines/engine.py.326: process: 0.33098 | load: 0.00018 [I 2022-10-28 01:30:13] ...trap/engines/engine.py.321: eval: epoch 98 | batch 30/510 [I 2022-10-28 01:30:13] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:41 [I 2022-10-28 01:30:13] ...trap/engines/engine.py.326: process: 0.32626 | load: 0.00016 [I 2022-10-28 01:30:16] ...trap/engines/engine.py.321: eval: epoch 98 | batch 40/510 [I 2022-10-28 01:30:16] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:36 [I 2022-10-28 01:30:16] ...trap/engines/engine.py.326: process: 0.33311 | load: 0.00018 [I 2022-10-28 01:30:20] ...trap/engines/engine.py.321: eval: epoch 98 | batch 50/510 [I 2022-10-28 01:30:20] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:34 [I 2022-10-28 01:30:20] ...trap/engines/engine.py.326: process: 0.32719 | load: 0.00019 [I 2022-10-28 01:30:23] ...trap/engines/engine.py.321: eval: epoch 98 | batch 60/510 [I 2022-10-28 01:30:23] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:29 [I 2022-10-28 01:30:23] ...trap/engines/engine.py.326: process: 0.33196 | load: 0.00016 [I 2022-10-28 01:30:26] ...trap/engines/engine.py.321: eval: epoch 98 | batch 70/510 [I 2022-10-28 01:30:26] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:25 [I 2022-10-28 01:30:26] ...trap/engines/engine.py.326: process: 0.32818 | load: 0.00016 [I 2022-10-28 01:30:30] ...trap/engines/engine.py.321: eval: epoch 98 | batch 80/510 [I 2022-10-28 01:30:30] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:23 [I 2022-10-28 01:30:30] ...trap/engines/engine.py.326: process: 0.32545 | load: 0.00016 [I 2022-10-28 01:30:33] ...trap/engines/engine.py.321: eval: epoch 98 | batch 90/510 [I 2022-10-28 01:30:33] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:20 [I 2022-10-28 01:30:33] ...trap/engines/engine.py.326: process: 0.33435 | load: 0.00016 [I 2022-10-28 01:30:36] ...trap/engines/engine.py.321: eval: epoch 98 | batch 100/510 [I 2022-10-28 01:30:36] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-28 01:30:36] ...trap/engines/engine.py.326: process: 0.28663 | load: 0.00016 [I 2022-10-28 01:30:39] ...trap/engines/engine.py.321: eval: epoch 98 | batch 110/510 [I 2022-10-28 01:30:39] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-28 01:30:39] ...trap/engines/engine.py.326: process: 0.29217 | load: 0.00018 [I 2022-10-28 01:30:42] ...trap/engines/engine.py.321: eval: epoch 98 | batch 120/510 [I 2022-10-28 01:30:42] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-28 01:30:42] ...trap/engines/engine.py.326: process: 0.29180 | load: 0.00016 [I 2022-10-28 01:30:45] ...trap/engines/engine.py.321: eval: epoch 98 | batch 130/510 [I 2022-10-28 01:30:45] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-28 01:30:45] ...trap/engines/engine.py.326: process: 0.29072 | load: 0.00014 [I 2022-10-28 01:30:48] ...trap/engines/engine.py.321: eval: epoch 98 | batch 140/510 [I 2022-10-28 01:30:48] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-28 01:30:48] ...trap/engines/engine.py.326: process: 0.28251 | load: 0.00018 [I 2022-10-28 01:30:51] ...trap/engines/engine.py.321: eval: epoch 98 | batch 150/510 [I 2022-10-28 01:30:51] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-28 01:30:51] ...trap/engines/engine.py.326: process: 0.28680 | load: 0.00015 [I 2022-10-28 01:30:54] ...trap/engines/engine.py.321: eval: epoch 98 | batch 160/510 [I 2022-10-28 01:30:54] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-28 01:30:54] ...trap/engines/engine.py.326: process: 0.28943 | load: 0.00019 [I 2022-10-28 01:30:57] ...trap/engines/engine.py.321: eval: epoch 98 | batch 170/510 [I 2022-10-28 01:30:57] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 01:30:57] ...trap/engines/engine.py.326: process: 0.29089 | load: 0.00018 [I 2022-10-28 01:31:00] ...trap/engines/engine.py.321: eval: epoch 98 | batch 180/510 [I 2022-10-28 01:31:00] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-28 01:31:00] ...trap/engines/engine.py.326: process: 0.28376 | load: 0.00019 [I 2022-10-28 01:31:02] ...trap/engines/engine.py.321: eval: epoch 98 | batch 190/510 [I 2022-10-28 01:31:02] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:33 [I 2022-10-28 01:31:02] ...trap/engines/engine.py.326: process: 0.29315 | load: 0.00023 [I 2022-10-28 01:31:05] ...trap/engines/engine.py.321: eval: epoch 98 | batch 200/510 [I 2022-10-28 01:31:05] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:30 [I 2022-10-28 01:31:05] ...trap/engines/engine.py.326: process: 0.30883 | load: 0.00014 [I 2022-10-28 01:31:08] ...trap/engines/engine.py.321: eval: epoch 98 | batch 210/510 [I 2022-10-28 01:31:08] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:27 [I 2022-10-28 01:31:08] ...trap/engines/engine.py.326: process: 0.28466 | load: 0.00015 [I 2022-10-28 01:31:11] ...trap/engines/engine.py.321: eval: epoch 98 | batch 220/510 [I 2022-10-28 01:31:11] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-28 01:31:11] ...trap/engines/engine.py.326: process: 0.28120 | load: 0.00014 [I 2022-10-28 01:31:14] ...trap/engines/engine.py.321: eval: epoch 98 | batch 230/510 [I 2022-10-28 01:31:14] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:19 [I 2022-10-28 01:31:14] ...trap/engines/engine.py.326: process: 0.28571 | load: 0.00015 [I 2022-10-28 01:31:17] ...trap/engines/engine.py.321: eval: epoch 98 | batch 240/510 [I 2022-10-28 01:31:17] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 01:31:17] ...trap/engines/engine.py.326: process: 0.28702 | load: 0.00015 [I 2022-10-28 01:31:20] ...trap/engines/engine.py.321: eval: epoch 98 | batch 250/510 [I 2022-10-28 01:31:20] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-28 01:31:20] ...trap/engines/engine.py.326: process: 0.28629 | load: 0.00015 [I 2022-10-28 01:31:23] ...trap/engines/engine.py.321: eval: epoch 98 | batch 260/510 [I 2022-10-28 01:31:23] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:11 [I 2022-10-28 01:31:23] ...trap/engines/engine.py.326: process: 0.28509 | load: 0.00015 [I 2022-10-28 01:31:26] ...trap/engines/engine.py.321: eval: epoch 98 | batch 270/510 [I 2022-10-28 01:31:26] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-28 01:31:26] ...trap/engines/engine.py.326: process: 0.28723 | load: 0.00015 [I 2022-10-28 01:31:28] ...trap/engines/engine.py.321: eval: epoch 98 | batch 280/510 [I 2022-10-28 01:31:28] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-28 01:31:28] ...trap/engines/engine.py.326: process: 0.28309 | load: 0.00019 [I 2022-10-28 01:31:31] ...trap/engines/engine.py.321: eval: epoch 98 | batch 290/510 [I 2022-10-28 01:31:31] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-28 01:31:31] ...trap/engines/engine.py.326: process: 0.29125 | load: 0.00037 [I 2022-10-28 01:31:34] ...trap/engines/engine.py.321: eval: epoch 98 | batch 300/510 [I 2022-10-28 01:31:34] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-28 01:31:34] ...trap/engines/engine.py.326: process: 0.28893 | load: 0.00017 [I 2022-10-28 01:31:37] ...trap/engines/engine.py.321: eval: epoch 98 | batch 310/510 [I 2022-10-28 01:31:37] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 01:31:37] ...trap/engines/engine.py.326: process: 0.28801 | load: 0.00015 [I 2022-10-28 01:31:40] ...trap/engines/engine.py.321: eval: epoch 98 | batch 320/510 [I 2022-10-28 01:31:40] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 01:31:40] ...trap/engines/engine.py.326: process: 0.28712 | load: 0.00016 [I 2022-10-28 01:31:43] ...trap/engines/engine.py.321: eval: epoch 98 | batch 330/510 [I 2022-10-28 01:31:43] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:54 [I 2022-10-28 01:31:43] ...trap/engines/engine.py.326: process: 0.28842 | load: 0.00018 [I 2022-10-28 01:31:46] ...trap/engines/engine.py.321: eval: epoch 98 | batch 340/510 [I 2022-10-28 01:31:46] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-28 01:31:46] ...trap/engines/engine.py.326: process: 0.28639 | load: 0.00015 [I 2022-10-28 01:31:49] ...trap/engines/engine.py.321: eval: epoch 98 | batch 350/510 [I 2022-10-28 01:31:49] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 01:31:49] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00015 [I 2022-10-28 01:31:52] ...trap/engines/engine.py.321: eval: epoch 98 | batch 360/510 [I 2022-10-28 01:31:52] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-28 01:31:52] ...trap/engines/engine.py.326: process: 0.28553 | load: 0.00016 [I 2022-10-28 01:31:54] ...trap/engines/engine.py.321: eval: epoch 98 | batch 370/510 [I 2022-10-28 01:31:54] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-28 01:31:54] ...trap/engines/engine.py.326: process: 0.28781 | load: 0.00017 [I 2022-10-28 01:31:57] ...trap/engines/engine.py.321: eval: epoch 98 | batch 380/510 [I 2022-10-28 01:31:57] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-28 01:31:57] ...trap/engines/engine.py.326: process: 0.28718 | load: 0.00016 [I 2022-10-28 01:32:00] ...trap/engines/engine.py.321: eval: epoch 98 | batch 390/510 [I 2022-10-28 01:32:00] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 01:32:00] ...trap/engines/engine.py.326: process: 0.28931 | load: 0.00015 [I 2022-10-28 01:32:03] ...trap/engines/engine.py.321: eval: epoch 98 | batch 400/510 [I 2022-10-28 01:32:03] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 01:32:03] ...trap/engines/engine.py.326: process: 0.28721 | load: 0.00015 [I 2022-10-28 01:32:06] ...trap/engines/engine.py.321: eval: epoch 98 | batch 410/510 [I 2022-10-28 01:32:06] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 01:32:06] ...trap/engines/engine.py.326: process: 0.28729 | load: 0.00014 [I 2022-10-28 01:32:09] ...trap/engines/engine.py.321: eval: epoch 98 | batch 420/510 [I 2022-10-28 01:32:09] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 01:32:09] ...trap/engines/engine.py.326: process: 0.28859 | load: 0.00015 [I 2022-10-28 01:32:12] ...trap/engines/engine.py.321: eval: epoch 98 | batch 430/510 [I 2022-10-28 01:32:12] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:22 [I 2022-10-28 01:32:12] ...trap/engines/engine.py.326: process: 0.28114 | load: 0.00016 [I 2022-10-28 01:32:14] ...trap/engines/engine.py.321: eval: epoch 98 | batch 440/510 [I 2022-10-28 01:32:14] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-28 01:32:14] ...trap/engines/engine.py.326: process: 0.28525 | load: 0.00014 [I 2022-10-28 01:32:17] ...trap/engines/engine.py.321: eval: epoch 98 | batch 450/510 [I 2022-10-28 01:32:17] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-28 01:32:17] ...trap/engines/engine.py.326: process: 0.29102 | load: 0.00016 [I 2022-10-28 01:32:20] ...trap/engines/engine.py.321: eval: epoch 98 | batch 460/510 [I 2022-10-28 01:32:20] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-28 01:32:20] ...trap/engines/engine.py.326: process: 0.28257 | load: 0.00015 [I 2022-10-28 01:32:23] ...trap/engines/engine.py.321: eval: epoch 98 | batch 470/510 [I 2022-10-28 01:32:23] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-28 01:32:23] ...trap/engines/engine.py.326: process: 0.28318 | load: 0.00015 [I 2022-10-28 01:32:26] ...trap/engines/engine.py.321: eval: epoch 98 | batch 480/510 [I 2022-10-28 01:32:26] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 01:32:26] ...trap/engines/engine.py.326: process: 0.28379 | load: 0.00015 [I 2022-10-28 01:32:29] ...trap/engines/engine.py.321: eval: epoch 98 | batch 490/510 [I 2022-10-28 01:32:29] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 01:32:29] ...trap/engines/engine.py.326: process: 0.29066 | load: 0.00015 [I 2022-10-28 01:32:32] ...trap/engines/engine.py.321: eval: epoch 98 | batch 500/510 [I 2022-10-28 01:32:32] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 01:32:32] ...trap/engines/engine.py.326: process: 0.28449 | load: 0.00014 [I 2022-10-28 01:32:34] ...trap/engines/engine.py.321: eval: epoch 98 | batch 510/510 [I 2022-10-28 01:32:34] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-28 01:32:34] ...trap/engines/engine.py.326: process: 0.28825 | load: 0.00011 [S 2022-10-28 01:32:35] ...trap/engines/engine.py.132: eval_epoch.epoch: 98 [I 2022-10-28 01:32:35] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.729800 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.903000 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.932500 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.727300 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.903700 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931200 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010048 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005692 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005766 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013327 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005951 [S 2022-10-28 01:32:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005288 [I 2022-10-28 01:32:41] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 01:32:41] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 01:32:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 01:32:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 01:32:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 01:32:46] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 01:32:47] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 01:32:53] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 01:32:53] ...trap/engines/engine.py.163: Training model on trainset for epoch 99 [I 2022-10-28 01:32:53] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 01:33:14] ...trap/engines/engine.py.224: train: epoch 99 | batch 0/2304 [I 2022-10-28 01:33:14] ...trap/engines/engine.py.228: elapsed: 0:00:20 | left: 13:25:37 [I 2022-10-28 01:33:14] ...trap/engines/engine.py.229: process: 20.97966 | load: 20.36405 [I 2022-10-28 01:33:14] ...trap/engines/engine.py.230: loss: 0.10612 [I 2022-10-28 01:33:14] ...optimizers/trijoint.py.151: total_norm: 14.265245 [I 2022-10-28 01:33:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 10/2304 [I 2022-10-28 01:33:20] ...trap/engines/engine.py.228: elapsed: 0:00:26 | left: 1:45:08 [I 2022-10-28 01:33:20] ...trap/engines/engine.py.229: process: 0.55478 | load: 0.00394 [I 2022-10-28 01:33:20] ...trap/engines/engine.py.230: loss: 0.10345 [I 2022-10-28 01:33:20] ...optimizers/trijoint.py.151: total_norm: 12.815528 [I 2022-10-28 01:33:25] ...trap/engines/engine.py.224: train: epoch 99 | batch 20/2304 [I 2022-10-28 01:33:25] ...trap/engines/engine.py.228: elapsed: 0:00:32 | left: 0:30:06 [I 2022-10-28 01:33:25] ...trap/engines/engine.py.229: process: 0.55268 | load: 0.00319 [I 2022-10-28 01:33:25] ...trap/engines/engine.py.230: loss: 0.10264 [I 2022-10-28 01:33:25] ...optimizers/trijoint.py.151: total_norm: 13.674120 [I 2022-10-28 01:33:31] ...trap/engines/engine.py.224: train: epoch 99 | batch 30/2304 [I 2022-10-28 01:33:31] ...trap/engines/engine.py.228: elapsed: 0:00:37 | left: 0:21:57 [I 2022-10-28 01:33:31] ...trap/engines/engine.py.229: process: 0.54952 | load: 0.00328 [I 2022-10-28 01:33:31] ...trap/engines/engine.py.230: loss: 0.10238 [I 2022-10-28 01:33:31] ...optimizers/trijoint.py.151: total_norm: 10.926888 [I 2022-10-28 01:33:36] ...trap/engines/engine.py.224: train: epoch 99 | batch 40/2304 [I 2022-10-28 01:33:36] ...trap/engines/engine.py.228: elapsed: 0:00:43 | left: 0:20:59 [I 2022-10-28 01:33:36] ...trap/engines/engine.py.229: process: 0.55567 | load: 0.00260 [I 2022-10-28 01:33:36] ...trap/engines/engine.py.230: loss: 0.09745 [I 2022-10-28 01:33:36] ...optimizers/trijoint.py.151: total_norm: 12.604135 [I 2022-10-28 01:33:42] ...trap/engines/engine.py.224: train: epoch 99 | batch 50/2304 [I 2022-10-28 01:33:42] ...trap/engines/engine.py.228: elapsed: 0:00:48 | left: 0:20:56 [I 2022-10-28 01:33:42] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00258 [I 2022-10-28 01:33:42] ...trap/engines/engine.py.230: loss: 0.10500 [I 2022-10-28 01:33:42] ...optimizers/trijoint.py.151: total_norm: 10.956026 [I 2022-10-28 01:33:47] ...trap/engines/engine.py.224: train: epoch 99 | batch 60/2304 [I 2022-10-28 01:33:47] ...trap/engines/engine.py.228: elapsed: 0:00:54 | left: 0:20:45 [I 2022-10-28 01:33:47] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00295 [I 2022-10-28 01:33:47] ...trap/engines/engine.py.230: loss: 0.17733 [I 2022-10-28 01:33:47] ...optimizers/trijoint.py.151: total_norm: 12.048650 [I 2022-10-28 01:33:53] ...trap/engines/engine.py.224: train: epoch 99 | batch 70/2304 [I 2022-10-28 01:33:53] ...trap/engines/engine.py.228: elapsed: 0:00:59 | left: 0:20:37 [I 2022-10-28 01:33:53] ...trap/engines/engine.py.229: process: 0.54402 | load: 0.00254 [I 2022-10-28 01:33:53] ...trap/engines/engine.py.230: loss: 0.12882 [I 2022-10-28 01:33:53] ...optimizers/trijoint.py.151: total_norm: 12.843171 [I 2022-10-28 01:33:59] ...trap/engines/engine.py.224: train: epoch 99 | batch 80/2304 [I 2022-10-28 01:33:59] ...trap/engines/engine.py.228: elapsed: 0:01:05 | left: 0:20:35 [I 2022-10-28 01:33:59] ...trap/engines/engine.py.229: process: 0.54407 | load: 0.00247 [I 2022-10-28 01:33:59] ...trap/engines/engine.py.230: loss: 0.09789 [I 2022-10-28 01:33:59] ...optimizers/trijoint.py.151: total_norm: 14.823754 [I 2022-10-28 01:34:04] ...trap/engines/engine.py.224: train: epoch 99 | batch 90/2304 [I 2022-10-28 01:34:04] ...trap/engines/engine.py.228: elapsed: 0:01:11 | left: 0:20:32 [I 2022-10-28 01:34:04] ...trap/engines/engine.py.229: process: 0.55077 | load: 0.00265 [I 2022-10-28 01:34:04] ...trap/engines/engine.py.230: loss: 0.13315 [I 2022-10-28 01:34:04] ...optimizers/trijoint.py.151: total_norm: 13.950451 [I 2022-10-28 01:34:10] ...trap/engines/engine.py.224: train: epoch 99 | batch 100/2304 [I 2022-10-28 01:34:10] ...trap/engines/engine.py.228: elapsed: 0:01:16 | left: 0:20:36 [I 2022-10-28 01:34:10] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00262 [I 2022-10-28 01:34:10] ...trap/engines/engine.py.230: loss: 0.15230 [I 2022-10-28 01:34:10] ...optimizers/trijoint.py.151: total_norm: 11.882413 [I 2022-10-28 01:34:15] ...trap/engines/engine.py.224: train: epoch 99 | batch 110/2304 [I 2022-10-28 01:34:15] ...trap/engines/engine.py.228: elapsed: 0:01:22 | left: 0:20:15 [I 2022-10-28 01:34:15] ...trap/engines/engine.py.229: process: 0.54924 | load: 0.00252 [I 2022-10-28 01:34:15] ...trap/engines/engine.py.230: loss: 0.09505 [I 2022-10-28 01:34:15] ...optimizers/trijoint.py.151: total_norm: 18.341270 [I 2022-10-28 01:34:21] ...trap/engines/engine.py.224: train: epoch 99 | batch 120/2304 [I 2022-10-28 01:34:21] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:20:22 [I 2022-10-28 01:34:21] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00244 [I 2022-10-28 01:34:21] ...trap/engines/engine.py.230: loss: 0.10391 [I 2022-10-28 01:34:21] ...optimizers/trijoint.py.151: total_norm: 14.454488 [I 2022-10-28 01:34:27] ...trap/engines/engine.py.224: train: epoch 99 | batch 130/2304 [I 2022-10-28 01:34:27] ...trap/engines/engine.py.228: elapsed: 0:01:33 | left: 0:20:03 [I 2022-10-28 01:34:27] ...trap/engines/engine.py.229: process: 0.54876 | load: 0.00259 [I 2022-10-28 01:34:27] ...trap/engines/engine.py.230: loss: 0.12849 [I 2022-10-28 01:34:27] ...optimizers/trijoint.py.151: total_norm: 20.878479 [I 2022-10-28 01:34:32] ...trap/engines/engine.py.224: train: epoch 99 | batch 140/2304 [I 2022-10-28 01:34:32] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:53 [I 2022-10-28 01:34:32] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00256 [I 2022-10-28 01:34:32] ...trap/engines/engine.py.230: loss: 0.10665 [I 2022-10-28 01:34:32] ...optimizers/trijoint.py.151: total_norm: 61.580414 [I 2022-10-28 01:34:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 150/2304 [I 2022-10-28 01:34:38] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:19:57 [I 2022-10-28 01:34:38] ...trap/engines/engine.py.229: process: 0.55790 | load: 0.00236 [I 2022-10-28 01:34:38] ...trap/engines/engine.py.230: loss: 0.14104 [I 2022-10-28 01:34:38] ...optimizers/trijoint.py.151: total_norm: 12.997817 [I 2022-10-28 01:34:43] ...trap/engines/engine.py.224: train: epoch 99 | batch 160/2304 [I 2022-10-28 01:34:43] ...trap/engines/engine.py.228: elapsed: 0:01:50 | left: 0:19:53 [I 2022-10-28 01:34:43] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00236 [I 2022-10-28 01:34:43] ...trap/engines/engine.py.230: loss: 0.20691 [I 2022-10-28 01:34:43] ...optimizers/trijoint.py.151: total_norm: 80.911636 [I 2022-10-28 01:34:49] ...trap/engines/engine.py.224: train: epoch 99 | batch 170/2304 [I 2022-10-28 01:34:49] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:46 [I 2022-10-28 01:34:49] ...trap/engines/engine.py.229: process: 0.56082 | load: 0.00246 [I 2022-10-28 01:34:49] ...trap/engines/engine.py.230: loss: 0.14795 [I 2022-10-28 01:34:49] ...optimizers/trijoint.py.151: total_norm: 31.316551 [I 2022-10-28 01:34:54] ...trap/engines/engine.py.224: train: epoch 99 | batch 180/2304 [I 2022-10-28 01:34:54] ...trap/engines/engine.py.228: elapsed: 0:02:01 | left: 0:19:41 [I 2022-10-28 01:34:54] ...trap/engines/engine.py.229: process: 0.55161 | load: 0.00246 [I 2022-10-28 01:34:54] ...trap/engines/engine.py.230: loss: 0.13934 [I 2022-10-28 01:34:54] ...optimizers/trijoint.py.151: total_norm: 11.902281 [I 2022-10-28 01:35:21] ...trap/engines/engine.py.224: train: epoch 99 | batch 190/2304 [I 2022-10-28 01:35:21] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 2:02:37 [I 2022-10-28 01:35:21] ...trap/engines/engine.py.229: process: 3.79471 | load: 3.20805 [I 2022-10-28 01:35:21] ...trap/engines/engine.py.230: loss: 0.16037 [I 2022-10-28 01:35:21] ...optimizers/trijoint.py.151: total_norm: 20.444025 [I 2022-10-28 01:35:28] ...trap/engines/engine.py.224: train: epoch 99 | batch 200/2304 [I 2022-10-28 01:35:28] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:31:55 [I 2022-10-28 01:35:28] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00248 [I 2022-10-28 01:35:28] ...trap/engines/engine.py.230: loss: 0.11833 [I 2022-10-28 01:35:28] ...optimizers/trijoint.py.151: total_norm: 34.229027 [I 2022-10-28 01:35:33] ...trap/engines/engine.py.224: train: epoch 99 | batch 210/2304 [I 2022-10-28 01:35:33] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:20:48 [I 2022-10-28 01:35:33] ...trap/engines/engine.py.229: process: 0.58187 | load: 0.00256 [I 2022-10-28 01:35:33] ...trap/engines/engine.py.230: loss: 0.10593 [I 2022-10-28 01:35:33] ...optimizers/trijoint.py.151: total_norm: 10.498893 [I 2022-10-28 01:35:39] ...trap/engines/engine.py.224: train: epoch 99 | batch 220/2304 [I 2022-10-28 01:35:39] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:19:29 [I 2022-10-28 01:35:39] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00253 [I 2022-10-28 01:35:39] ...trap/engines/engine.py.230: loss: 0.18871 [I 2022-10-28 01:35:39] ...optimizers/trijoint.py.151: total_norm: 14.873776 [I 2022-10-28 01:35:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 230/2304 [I 2022-10-28 01:35:45] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:19:19 [I 2022-10-28 01:35:45] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00250 [I 2022-10-28 01:35:45] ...trap/engines/engine.py.230: loss: 0.07822 [I 2022-10-28 01:35:45] ...optimizers/trijoint.py.151: total_norm: 8.475385 [I 2022-10-28 01:35:50] ...trap/engines/engine.py.224: train: epoch 99 | batch 240/2304 [I 2022-10-28 01:35:50] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:19:08 [I 2022-10-28 01:35:50] ...trap/engines/engine.py.229: process: 0.54957 | load: 0.00238 [I 2022-10-28 01:35:50] ...trap/engines/engine.py.230: loss: 0.11906 [I 2022-10-28 01:35:50] ...optimizers/trijoint.py.151: total_norm: 14.360973 [I 2022-10-28 01:35:56] ...trap/engines/engine.py.224: train: epoch 99 | batch 250/2304 [I 2022-10-28 01:35:56] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:19:00 [I 2022-10-28 01:35:56] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00237 [I 2022-10-28 01:35:56] ...trap/engines/engine.py.230: loss: 0.15682 [I 2022-10-28 01:35:56] ...optimizers/trijoint.py.151: total_norm: 17.020916 [I 2022-10-28 01:36:01] ...trap/engines/engine.py.224: train: epoch 99 | batch 260/2304 [I 2022-10-28 01:36:01] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:18:54 [I 2022-10-28 01:36:01] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00239 [I 2022-10-28 01:36:01] ...trap/engines/engine.py.230: loss: 0.11698 [I 2022-10-28 01:36:01] ...optimizers/trijoint.py.151: total_norm: 9.133309 [I 2022-10-28 01:36:07] ...trap/engines/engine.py.224: train: epoch 99 | batch 270/2304 [I 2022-10-28 01:36:07] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:55 [I 2022-10-28 01:36:07] ...trap/engines/engine.py.229: process: 0.55192 | load: 0.00241 [I 2022-10-28 01:36:07] ...trap/engines/engine.py.230: loss: 0.08580 [I 2022-10-28 01:36:07] ...optimizers/trijoint.py.151: total_norm: 9.334094 [I 2022-10-28 01:36:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 280/2304 [I 2022-10-28 01:36:13] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:49 [I 2022-10-28 01:36:13] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00247 [I 2022-10-28 01:36:13] ...trap/engines/engine.py.230: loss: 0.08972 [I 2022-10-28 01:36:13] ...optimizers/trijoint.py.151: total_norm: 8.858045 [I 2022-10-28 01:36:18] ...trap/engines/engine.py.224: train: epoch 99 | batch 290/2304 [I 2022-10-28 01:36:18] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:40 [I 2022-10-28 01:36:18] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00238 [I 2022-10-28 01:36:18] ...trap/engines/engine.py.230: loss: 0.13468 [I 2022-10-28 01:36:18] ...optimizers/trijoint.py.151: total_norm: 21.121939 [I 2022-10-28 01:36:24] ...trap/engines/engine.py.224: train: epoch 99 | batch 300/2304 [I 2022-10-28 01:36:24] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:31 [I 2022-10-28 01:36:24] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00239 [I 2022-10-28 01:36:24] ...trap/engines/engine.py.230: loss: 0.08290 [I 2022-10-28 01:36:24] ...optimizers/trijoint.py.151: total_norm: 12.680207 [I 2022-10-28 01:36:29] ...trap/engines/engine.py.224: train: epoch 99 | batch 310/2304 [I 2022-10-28 01:36:29] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:18:32 [I 2022-10-28 01:36:29] ...trap/engines/engine.py.229: process: 0.56341 | load: 0.00244 [I 2022-10-28 01:36:29] ...trap/engines/engine.py.230: loss: 0.09921 [I 2022-10-28 01:36:29] ...optimizers/trijoint.py.151: total_norm: 10.025281 [I 2022-10-28 01:36:35] ...trap/engines/engine.py.224: train: epoch 99 | batch 320/2304 [I 2022-10-28 01:36:35] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:18:24 [I 2022-10-28 01:36:35] ...trap/engines/engine.py.229: process: 0.54821 | load: 0.00242 [I 2022-10-28 01:36:35] ...trap/engines/engine.py.230: loss: 0.18989 [I 2022-10-28 01:36:35] ...optimizers/trijoint.py.151: total_norm: 21.979786 [I 2022-10-28 01:36:41] ...trap/engines/engine.py.224: train: epoch 99 | batch 330/2304 [I 2022-10-28 01:36:41] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:18:30 [I 2022-10-28 01:36:41] ...trap/engines/engine.py.229: process: 0.55568 | load: 0.00240 [I 2022-10-28 01:36:41] ...trap/engines/engine.py.230: loss: 0.12462 [I 2022-10-28 01:36:41] ...optimizers/trijoint.py.151: total_norm: 12.587753 [I 2022-10-28 01:36:46] ...trap/engines/engine.py.224: train: epoch 99 | batch 340/2304 [I 2022-10-28 01:36:46] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:18:11 [I 2022-10-28 01:36:46] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00243 [I 2022-10-28 01:36:46] ...trap/engines/engine.py.230: loss: 0.14013 [I 2022-10-28 01:36:46] ...optimizers/trijoint.py.151: total_norm: 45.672852 [I 2022-10-28 01:36:52] ...trap/engines/engine.py.224: train: epoch 99 | batch 350/2304 [I 2022-10-28 01:36:52] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:18:05 [I 2022-10-28 01:36:52] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00259 [I 2022-10-28 01:36:52] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-28 01:36:52] ...optimizers/trijoint.py.151: total_norm: 20.241028 [I 2022-10-28 01:36:57] ...trap/engines/engine.py.224: train: epoch 99 | batch 360/2304 [I 2022-10-28 01:36:57] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:18:01 [I 2022-10-28 01:36:57] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00246 [I 2022-10-28 01:36:57] ...trap/engines/engine.py.230: loss: 0.15416 [I 2022-10-28 01:36:57] ...optimizers/trijoint.py.151: total_norm: 15.749169 [I 2022-10-28 01:37:03] ...trap/engines/engine.py.224: train: epoch 99 | batch 370/2304 [I 2022-10-28 01:37:03] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:18:01 [I 2022-10-28 01:37:03] ...trap/engines/engine.py.229: process: 0.56287 | load: 0.00251 [I 2022-10-28 01:37:03] ...trap/engines/engine.py.230: loss: 0.14064 [I 2022-10-28 01:37:03] ...optimizers/trijoint.py.151: total_norm: 18.578800 [I 2022-10-28 01:37:08] ...trap/engines/engine.py.224: train: epoch 99 | batch 380/2304 [I 2022-10-28 01:37:08] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:49 [I 2022-10-28 01:37:08] ...trap/engines/engine.py.229: process: 0.56037 | load: 0.00248 [I 2022-10-28 01:37:08] ...trap/engines/engine.py.230: loss: 0.14276 [I 2022-10-28 01:37:08] ...optimizers/trijoint.py.151: total_norm: 20.835529 [I 2022-10-28 01:37:14] ...trap/engines/engine.py.224: train: epoch 99 | batch 390/2304 [I 2022-10-28 01:37:14] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:43 [I 2022-10-28 01:37:14] ...trap/engines/engine.py.229: process: 0.54720 | load: 0.00244 [I 2022-10-28 01:37:14] ...trap/engines/engine.py.230: loss: 0.11847 [I 2022-10-28 01:37:14] ...optimizers/trijoint.py.151: total_norm: 24.701843 [I 2022-10-28 01:37:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 400/2304 [I 2022-10-28 01:37:20] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:40 [I 2022-10-28 01:37:20] ...trap/engines/engine.py.229: process: 0.55162 | load: 0.00240 [I 2022-10-28 01:37:20] ...trap/engines/engine.py.230: loss: 0.12028 [I 2022-10-28 01:37:20] ...optimizers/trijoint.py.151: total_norm: 27.607460 [I 2022-10-28 01:37:25] ...trap/engines/engine.py.224: train: epoch 99 | batch 410/2304 [I 2022-10-28 01:37:25] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:17:59 [I 2022-10-28 01:37:25] ...trap/engines/engine.py.229: process: 0.56082 | load: 0.00263 [I 2022-10-28 01:37:25] ...trap/engines/engine.py.230: loss: 0.12381 [I 2022-10-28 01:37:25] ...optimizers/trijoint.py.151: total_norm: 12.898834 [I 2022-10-28 01:37:31] ...trap/engines/engine.py.224: train: epoch 99 | batch 420/2304 [I 2022-10-28 01:37:31] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:17:29 [I 2022-10-28 01:37:31] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00247 [I 2022-10-28 01:37:31] ...trap/engines/engine.py.230: loss: 0.09531 [I 2022-10-28 01:37:31] ...optimizers/trijoint.py.151: total_norm: 10.731057 [I 2022-10-28 01:37:36] ...trap/engines/engine.py.224: train: epoch 99 | batch 430/2304 [I 2022-10-28 01:37:36] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:17:19 [I 2022-10-28 01:37:36] ...trap/engines/engine.py.229: process: 0.54957 | load: 0.00243 [I 2022-10-28 01:37:36] ...trap/engines/engine.py.230: loss: 0.13075 [I 2022-10-28 01:37:36] ...optimizers/trijoint.py.151: total_norm: 17.535011 [I 2022-10-28 01:37:42] ...trap/engines/engine.py.224: train: epoch 99 | batch 440/2304 [I 2022-10-28 01:37:42] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:17:27 [I 2022-10-28 01:37:42] ...trap/engines/engine.py.229: process: 0.56314 | load: 0.00244 [I 2022-10-28 01:37:42] ...trap/engines/engine.py.230: loss: 0.07674 [I 2022-10-28 01:37:42] ...optimizers/trijoint.py.151: total_norm: 18.680830 [I 2022-10-28 01:37:48] ...trap/engines/engine.py.224: train: epoch 99 | batch 450/2304 [I 2022-10-28 01:37:48] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:17:11 [I 2022-10-28 01:37:48] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00241 [I 2022-10-28 01:37:48] ...trap/engines/engine.py.230: loss: 0.07144 [I 2022-10-28 01:37:48] ...optimizers/trijoint.py.151: total_norm: 22.978464 [I 2022-10-28 01:37:53] ...trap/engines/engine.py.224: train: epoch 99 | batch 460/2304 [I 2022-10-28 01:37:53] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:17:08 [I 2022-10-28 01:37:53] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00242 [I 2022-10-28 01:37:53] ...trap/engines/engine.py.230: loss: 0.17079 [I 2022-10-28 01:37:53] ...optimizers/trijoint.py.151: total_norm: 22.342865 [I 2022-10-28 01:37:59] ...trap/engines/engine.py.224: train: epoch 99 | batch 470/2304 [I 2022-10-28 01:37:59] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:17:09 [I 2022-10-28 01:37:59] ...trap/engines/engine.py.229: process: 0.54171 | load: 0.00246 [I 2022-10-28 01:37:59] ...trap/engines/engine.py.230: loss: 0.12650 [I 2022-10-28 01:37:59] ...optimizers/trijoint.py.151: total_norm: 13.551288 [I 2022-10-28 01:38:05] ...trap/engines/engine.py.224: train: epoch 99 | batch 480/2304 [I 2022-10-28 01:38:05] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:17:04 [I 2022-10-28 01:38:05] ...trap/engines/engine.py.229: process: 0.56141 | load: 0.00244 [I 2022-10-28 01:38:05] ...trap/engines/engine.py.230: loss: 0.09698 [I 2022-10-28 01:38:05] ...optimizers/trijoint.py.151: total_norm: 13.028256 [I 2022-10-28 01:38:10] ...trap/engines/engine.py.224: train: epoch 99 | batch 490/2304 [I 2022-10-28 01:38:10] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:53 [I 2022-10-28 01:38:10] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00246 [I 2022-10-28 01:38:10] ...trap/engines/engine.py.230: loss: 0.10274 [I 2022-10-28 01:38:10] ...optimizers/trijoint.py.151: total_norm: 18.341772 [I 2022-10-28 01:38:16] ...trap/engines/engine.py.224: train: epoch 99 | batch 500/2304 [I 2022-10-28 01:38:16] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:47 [I 2022-10-28 01:38:16] ...trap/engines/engine.py.229: process: 0.56475 | load: 0.00243 [I 2022-10-28 01:38:16] ...trap/engines/engine.py.230: loss: 0.09492 [I 2022-10-28 01:38:16] ...optimizers/trijoint.py.151: total_norm: 11.854967 [I 2022-10-28 01:38:21] ...trap/engines/engine.py.224: train: epoch 99 | batch 510/2304 [I 2022-10-28 01:38:21] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:16:43 [I 2022-10-28 01:38:21] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00243 [I 2022-10-28 01:38:21] ...trap/engines/engine.py.230: loss: 0.13185 [I 2022-10-28 01:38:21] ...optimizers/trijoint.py.151: total_norm: 10.355364 [I 2022-10-28 01:38:27] ...trap/engines/engine.py.224: train: epoch 99 | batch 520/2304 [I 2022-10-28 01:38:27] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:35 [I 2022-10-28 01:38:27] ...trap/engines/engine.py.229: process: 0.55362 | load: 0.00249 [I 2022-10-28 01:38:27] ...trap/engines/engine.py.230: loss: 0.19366 [I 2022-10-28 01:38:27] ...optimizers/trijoint.py.151: total_norm: 11.866553 [I 2022-10-28 01:38:33] ...trap/engines/engine.py.224: train: epoch 99 | batch 530/2304 [I 2022-10-28 01:38:33] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:16:32 [I 2022-10-28 01:38:33] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00242 [I 2022-10-28 01:38:33] ...trap/engines/engine.py.230: loss: 0.09659 [I 2022-10-28 01:38:33] ...optimizers/trijoint.py.151: total_norm: 19.429956 [I 2022-10-28 01:38:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 540/2304 [I 2022-10-28 01:38:38] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:16:28 [I 2022-10-28 01:38:38] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00240 [I 2022-10-28 01:38:38] ...trap/engines/engine.py.230: loss: 0.10973 [I 2022-10-28 01:38:38] ...optimizers/trijoint.py.151: total_norm: 11.713579 [I 2022-10-28 01:38:44] ...trap/engines/engine.py.224: train: epoch 99 | batch 550/2304 [I 2022-10-28 01:38:44] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:16:09 [I 2022-10-28 01:38:44] ...trap/engines/engine.py.229: process: 0.54213 | load: 0.00241 [I 2022-10-28 01:38:44] ...trap/engines/engine.py.230: loss: 0.11318 [I 2022-10-28 01:38:44] ...optimizers/trijoint.py.151: total_norm: 16.251476 [I 2022-10-28 01:38:49] ...trap/engines/engine.py.224: train: epoch 99 | batch 560/2304 [I 2022-10-28 01:38:49] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:16:04 [I 2022-10-28 01:38:49] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00252 [I 2022-10-28 01:38:49] ...trap/engines/engine.py.230: loss: 0.10118 [I 2022-10-28 01:38:49] ...optimizers/trijoint.py.151: total_norm: 11.141575 [I 2022-10-28 01:38:55] ...trap/engines/engine.py.224: train: epoch 99 | batch 570/2304 [I 2022-10-28 01:38:55] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:16:02 [I 2022-10-28 01:38:55] ...trap/engines/engine.py.229: process: 0.55726 | load: 0.00241 [I 2022-10-28 01:38:55] ...trap/engines/engine.py.230: loss: 0.12154 [I 2022-10-28 01:38:55] ...optimizers/trijoint.py.151: total_norm: 11.329444 [I 2022-10-28 01:39:00] ...trap/engines/engine.py.224: train: epoch 99 | batch 580/2304 [I 2022-10-28 01:39:00] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:56 [I 2022-10-28 01:39:00] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00244 [I 2022-10-28 01:39:00] ...trap/engines/engine.py.230: loss: 0.11431 [I 2022-10-28 01:39:00] ...optimizers/trijoint.py.151: total_norm: 12.233913 [I 2022-10-28 01:39:06] ...trap/engines/engine.py.224: train: epoch 99 | batch 590/2304 [I 2022-10-28 01:39:06] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:16:11 [I 2022-10-28 01:39:06] ...trap/engines/engine.py.229: process: 0.56042 | load: 0.00245 [I 2022-10-28 01:39:06] ...trap/engines/engine.py.230: loss: 0.13419 [I 2022-10-28 01:39:06] ...optimizers/trijoint.py.151: total_norm: 9.931157 [I 2022-10-28 01:39:12] ...trap/engines/engine.py.224: train: epoch 99 | batch 600/2304 [I 2022-10-28 01:39:12] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:49 [I 2022-10-28 01:39:12] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00244 [I 2022-10-28 01:39:12] ...trap/engines/engine.py.230: loss: 0.07619 [I 2022-10-28 01:39:12] ...optimizers/trijoint.py.151: total_norm: 12.406936 [I 2022-10-28 01:39:17] ...trap/engines/engine.py.224: train: epoch 99 | batch 610/2304 [I 2022-10-28 01:39:17] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:47 [I 2022-10-28 01:39:17] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00240 [I 2022-10-28 01:39:17] ...trap/engines/engine.py.230: loss: 0.12653 [I 2022-10-28 01:39:17] ...optimizers/trijoint.py.151: total_norm: 15.022734 [I 2022-10-28 01:39:23] ...trap/engines/engine.py.224: train: epoch 99 | batch 620/2304 [I 2022-10-28 01:39:23] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:48 [I 2022-10-28 01:39:23] ...trap/engines/engine.py.229: process: 0.57167 | load: 0.00246 [I 2022-10-28 01:39:23] ...trap/engines/engine.py.230: loss: 0.09869 [I 2022-10-28 01:39:23] ...optimizers/trijoint.py.151: total_norm: 15.438925 [I 2022-10-28 01:39:28] ...trap/engines/engine.py.224: train: epoch 99 | batch 630/2304 [I 2022-10-28 01:39:28] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:15:36 [I 2022-10-28 01:39:28] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00257 [I 2022-10-28 01:39:28] ...trap/engines/engine.py.230: loss: 0.13004 [I 2022-10-28 01:39:28] ...optimizers/trijoint.py.151: total_norm: 25.600254 [I 2022-10-28 01:39:34] ...trap/engines/engine.py.224: train: epoch 99 | batch 640/2304 [I 2022-10-28 01:39:34] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:15:22 [I 2022-10-28 01:39:34] ...trap/engines/engine.py.229: process: 0.54492 | load: 0.00253 [I 2022-10-28 01:39:34] ...trap/engines/engine.py.230: loss: 0.11576 [I 2022-10-28 01:39:34] ...optimizers/trijoint.py.151: total_norm: 21.175894 [I 2022-10-28 01:39:40] ...trap/engines/engine.py.224: train: epoch 99 | batch 650/2304 [I 2022-10-28 01:39:40] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:15:24 [I 2022-10-28 01:39:40] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00251 [I 2022-10-28 01:39:40] ...trap/engines/engine.py.230: loss: 0.09886 [I 2022-10-28 01:39:40] ...optimizers/trijoint.py.151: total_norm: 14.826248 [I 2022-10-28 01:39:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 660/2304 [I 2022-10-28 01:39:45] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:15:14 [I 2022-10-28 01:39:45] ...trap/engines/engine.py.229: process: 0.55016 | load: 0.00244 [I 2022-10-28 01:39:45] ...trap/engines/engine.py.230: loss: 0.14143 [I 2022-10-28 01:39:45] ...optimizers/trijoint.py.151: total_norm: 23.149654 [I 2022-10-28 01:39:51] ...trap/engines/engine.py.224: train: epoch 99 | batch 670/2304 [I 2022-10-28 01:39:51] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:15:11 [I 2022-10-28 01:39:51] ...trap/engines/engine.py.229: process: 0.55348 | load: 0.00247 [I 2022-10-28 01:39:51] ...trap/engines/engine.py.230: loss: 0.11052 [I 2022-10-28 01:39:51] ...optimizers/trijoint.py.151: total_norm: 13.161819 [I 2022-10-28 01:39:56] ...trap/engines/engine.py.224: train: epoch 99 | batch 680/2304 [I 2022-10-28 01:39:56] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:15:08 [I 2022-10-28 01:39:56] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00259 [I 2022-10-28 01:39:56] ...trap/engines/engine.py.230: loss: 0.09020 [I 2022-10-28 01:39:56] ...optimizers/trijoint.py.151: total_norm: 10.910877 [I 2022-10-28 01:40:02] ...trap/engines/engine.py.224: train: epoch 99 | batch 690/2304 [I 2022-10-28 01:40:02] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:59 [I 2022-10-28 01:40:02] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00249 [I 2022-10-28 01:40:02] ...trap/engines/engine.py.230: loss: 0.26716 [I 2022-10-28 01:40:02] ...optimizers/trijoint.py.151: total_norm: 14.522909 [I 2022-10-28 01:40:08] ...trap/engines/engine.py.224: train: epoch 99 | batch 700/2304 [I 2022-10-28 01:40:08] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:15:33 [I 2022-10-28 01:40:08] ...trap/engines/engine.py.229: process: 0.65525 | load: 0.00274 [I 2022-10-28 01:40:08] ...trap/engines/engine.py.230: loss: 0.09622 [I 2022-10-28 01:40:08] ...optimizers/trijoint.py.151: total_norm: 10.109593 [I 2022-10-28 01:40:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 710/2304 [I 2022-10-28 01:40:13] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:58 [I 2022-10-28 01:40:13] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00248 [I 2022-10-28 01:40:13] ...trap/engines/engine.py.230: loss: 0.18122 [I 2022-10-28 01:40:13] ...optimizers/trijoint.py.151: total_norm: 15.872615 [I 2022-10-28 01:40:19] ...trap/engines/engine.py.224: train: epoch 99 | batch 720/2304 [I 2022-10-28 01:40:19] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:44 [I 2022-10-28 01:40:19] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00247 [I 2022-10-28 01:40:19] ...trap/engines/engine.py.230: loss: 0.07835 [I 2022-10-28 01:40:19] ...optimizers/trijoint.py.151: total_norm: 11.664848 [I 2022-10-28 01:40:25] ...trap/engines/engine.py.224: train: epoch 99 | batch 730/2304 [I 2022-10-28 01:40:25] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:14:39 [I 2022-10-28 01:40:25] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00245 [I 2022-10-28 01:40:25] ...trap/engines/engine.py.230: loss: 0.22243 [I 2022-10-28 01:40:25] ...optimizers/trijoint.py.151: total_norm: 15.309668 [I 2022-10-28 01:40:30] ...trap/engines/engine.py.224: train: epoch 99 | batch 740/2304 [I 2022-10-28 01:40:30] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:14:32 [I 2022-10-28 01:40:30] ...trap/engines/engine.py.229: process: 0.56707 | load: 0.00266 [I 2022-10-28 01:40:30] ...trap/engines/engine.py.230: loss: 0.14042 [I 2022-10-28 01:40:30] ...optimizers/trijoint.py.151: total_norm: 14.902183 [I 2022-10-28 01:40:36] ...trap/engines/engine.py.224: train: epoch 99 | batch 750/2304 [I 2022-10-28 01:40:36] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:14:33 [I 2022-10-28 01:40:36] ...trap/engines/engine.py.229: process: 0.56226 | load: 0.00262 [I 2022-10-28 01:40:36] ...trap/engines/engine.py.230: loss: 0.09157 [I 2022-10-28 01:40:36] ...optimizers/trijoint.py.151: total_norm: 11.929009 [I 2022-10-28 01:40:41] ...trap/engines/engine.py.224: train: epoch 99 | batch 760/2304 [I 2022-10-28 01:40:41] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:14:22 [I 2022-10-28 01:40:41] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00255 [I 2022-10-28 01:40:41] ...trap/engines/engine.py.230: loss: 0.09143 [I 2022-10-28 01:40:41] ...optimizers/trijoint.py.151: total_norm: 14.646539 [I 2022-10-28 01:40:47] ...trap/engines/engine.py.224: train: epoch 99 | batch 770/2304 [I 2022-10-28 01:40:47] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:14:26 [I 2022-10-28 01:40:47] ...trap/engines/engine.py.229: process: 0.57171 | load: 0.00253 [I 2022-10-28 01:40:47] ...trap/engines/engine.py.230: loss: 0.13863 [I 2022-10-28 01:40:47] ...optimizers/trijoint.py.151: total_norm: 16.646515 [I 2022-10-28 01:40:53] ...trap/engines/engine.py.224: train: epoch 99 | batch 780/2304 [I 2022-10-28 01:40:53] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:14:06 [I 2022-10-28 01:40:53] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00251 [I 2022-10-28 01:40:53] ...trap/engines/engine.py.230: loss: 0.12184 [I 2022-10-28 01:40:53] ...optimizers/trijoint.py.151: total_norm: 9.594778 [I 2022-10-28 01:40:58] ...trap/engines/engine.py.224: train: epoch 99 | batch 790/2304 [I 2022-10-28 01:40:58] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:14:03 [I 2022-10-28 01:40:58] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00246 [I 2022-10-28 01:40:58] ...trap/engines/engine.py.230: loss: 0.11804 [I 2022-10-28 01:40:58] ...optimizers/trijoint.py.151: total_norm: 25.448874 [I 2022-10-28 01:41:04] ...trap/engines/engine.py.224: train: epoch 99 | batch 800/2304 [I 2022-10-28 01:41:04] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:58 [I 2022-10-28 01:41:04] ...trap/engines/engine.py.229: process: 0.55831 | load: 0.00248 [I 2022-10-28 01:41:04] ...trap/engines/engine.py.230: loss: 0.07727 [I 2022-10-28 01:41:04] ...optimizers/trijoint.py.151: total_norm: 10.658821 [I 2022-10-28 01:41:09] ...trap/engines/engine.py.224: train: epoch 99 | batch 810/2304 [I 2022-10-28 01:41:09] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:49 [I 2022-10-28 01:41:09] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00251 [I 2022-10-28 01:41:09] ...trap/engines/engine.py.230: loss: 0.13809 [I 2022-10-28 01:41:09] ...optimizers/trijoint.py.151: total_norm: 27.668390 [I 2022-10-28 01:41:15] ...trap/engines/engine.py.224: train: epoch 99 | batch 820/2304 [I 2022-10-28 01:41:15] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:46 [I 2022-10-28 01:41:15] ...trap/engines/engine.py.229: process: 0.55723 | load: 0.00247 [I 2022-10-28 01:41:15] ...trap/engines/engine.py.230: loss: 0.11287 [I 2022-10-28 01:41:15] ...optimizers/trijoint.py.151: total_norm: 9.004486 [I 2022-10-28 01:41:21] ...trap/engines/engine.py.224: train: epoch 99 | batch 830/2304 [I 2022-10-28 01:41:21] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:53 [I 2022-10-28 01:41:21] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00250 [I 2022-10-28 01:41:21] ...trap/engines/engine.py.230: loss: 0.14880 [I 2022-10-28 01:41:21] ...optimizers/trijoint.py.151: total_norm: 14.025650 [I 2022-10-28 01:41:26] ...trap/engines/engine.py.224: train: epoch 99 | batch 840/2304 [I 2022-10-28 01:41:26] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:13:32 [I 2022-10-28 01:41:26] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00241 [I 2022-10-28 01:41:26] ...trap/engines/engine.py.230: loss: 0.12203 [I 2022-10-28 01:41:26] ...optimizers/trijoint.py.151: total_norm: 54.850845 [I 2022-10-28 01:41:32] ...trap/engines/engine.py.224: train: epoch 99 | batch 850/2304 [I 2022-10-28 01:41:32] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:13:28 [I 2022-10-28 01:41:32] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00248 [I 2022-10-28 01:41:32] ...trap/engines/engine.py.230: loss: 0.12023 [I 2022-10-28 01:41:32] ...optimizers/trijoint.py.151: total_norm: 8.562351 [I 2022-10-28 01:41:37] ...trap/engines/engine.py.224: train: epoch 99 | batch 860/2304 [I 2022-10-28 01:41:37] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:13:18 [I 2022-10-28 01:41:37] ...trap/engines/engine.py.229: process: 0.55263 | load: 0.00244 [I 2022-10-28 01:41:37] ...trap/engines/engine.py.230: loss: 0.12705 [I 2022-10-28 01:41:37] ...optimizers/trijoint.py.151: total_norm: 14.111362 [I 2022-10-28 01:41:43] ...trap/engines/engine.py.224: train: epoch 99 | batch 870/2304 [I 2022-10-28 01:41:43] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:13:13 [I 2022-10-28 01:41:43] ...trap/engines/engine.py.229: process: 0.54826 | load: 0.00240 [I 2022-10-28 01:41:43] ...trap/engines/engine.py.230: loss: 0.16012 [I 2022-10-28 01:41:43] ...optimizers/trijoint.py.151: total_norm: 28.523251 [I 2022-10-28 01:41:48] ...trap/engines/engine.py.224: train: epoch 99 | batch 880/2304 [I 2022-10-28 01:41:48] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:13:07 [I 2022-10-28 01:41:48] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00245 [I 2022-10-28 01:41:48] ...trap/engines/engine.py.230: loss: 0.11324 [I 2022-10-28 01:41:48] ...optimizers/trijoint.py.151: total_norm: 13.608845 [I 2022-10-28 01:41:54] ...trap/engines/engine.py.224: train: epoch 99 | batch 890/2304 [I 2022-10-28 01:41:54] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:13:07 [I 2022-10-28 01:41:54] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00262 [I 2022-10-28 01:41:54] ...trap/engines/engine.py.230: loss: 0.10904 [I 2022-10-28 01:41:54] ...optimizers/trijoint.py.151: total_norm: 9.790966 [I 2022-10-28 01:41:59] ...trap/engines/engine.py.224: train: epoch 99 | batch 900/2304 [I 2022-10-28 01:41:59] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:59 [I 2022-10-28 01:41:59] ...trap/engines/engine.py.229: process: 0.55804 | load: 0.00245 [I 2022-10-28 01:41:59] ...trap/engines/engine.py.230: loss: 0.08935 [I 2022-10-28 01:41:59] ...optimizers/trijoint.py.151: total_norm: 11.314750 [I 2022-10-28 01:42:05] ...trap/engines/engine.py.224: train: epoch 99 | batch 910/2304 [I 2022-10-28 01:42:05] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:52 [I 2022-10-28 01:42:05] ...trap/engines/engine.py.229: process: 0.54966 | load: 0.00259 [I 2022-10-28 01:42:05] ...trap/engines/engine.py.230: loss: 0.12981 [I 2022-10-28 01:42:05] ...optimizers/trijoint.py.151: total_norm: 25.148546 [I 2022-10-28 01:42:11] ...trap/engines/engine.py.224: train: epoch 99 | batch 920/2304 [I 2022-10-28 01:42:11] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:47 [I 2022-10-28 01:42:11] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00247 [I 2022-10-28 01:42:11] ...trap/engines/engine.py.230: loss: 0.14025 [I 2022-10-28 01:42:11] ...optimizers/trijoint.py.151: total_norm: 17.994837 [I 2022-10-28 01:42:16] ...trap/engines/engine.py.224: train: epoch 99 | batch 930/2304 [I 2022-10-28 01:42:16] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:44 [I 2022-10-28 01:42:16] ...trap/engines/engine.py.229: process: 0.56513 | load: 0.00249 [I 2022-10-28 01:42:16] ...trap/engines/engine.py.230: loss: 0.11409 [I 2022-10-28 01:42:16] ...optimizers/trijoint.py.151: total_norm: 11.462905 [I 2022-10-28 01:42:22] ...trap/engines/engine.py.224: train: epoch 99 | batch 940/2304 [I 2022-10-28 01:42:22] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:33 [I 2022-10-28 01:42:22] ...trap/engines/engine.py.229: process: 0.55134 | load: 0.00246 [I 2022-10-28 01:42:22] ...trap/engines/engine.py.230: loss: 0.17117 [I 2022-10-28 01:42:22] ...optimizers/trijoint.py.151: total_norm: 26.314850 [I 2022-10-28 01:42:27] ...trap/engines/engine.py.224: train: epoch 99 | batch 950/2304 [I 2022-10-28 01:42:27] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:12:33 [I 2022-10-28 01:42:27] ...trap/engines/engine.py.229: process: 0.55549 | load: 0.00246 [I 2022-10-28 01:42:27] ...trap/engines/engine.py.230: loss: 0.10728 [I 2022-10-28 01:42:27] ...optimizers/trijoint.py.151: total_norm: 14.093259 [I 2022-10-28 01:42:33] ...trap/engines/engine.py.224: train: epoch 99 | batch 960/2304 [I 2022-10-28 01:42:33] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:12:20 [I 2022-10-28 01:42:33] ...trap/engines/engine.py.229: process: 0.54393 | load: 0.00245 [I 2022-10-28 01:42:33] ...trap/engines/engine.py.230: loss: 0.15260 [I 2022-10-28 01:42:33] ...optimizers/trijoint.py.151: total_norm: 18.350538 [I 2022-10-28 01:42:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 970/2304 [I 2022-10-28 01:42:38] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:12:24 [I 2022-10-28 01:42:38] ...trap/engines/engine.py.229: process: 0.55224 | load: 0.00243 [I 2022-10-28 01:42:38] ...trap/engines/engine.py.230: loss: 0.11696 [I 2022-10-28 01:42:38] ...optimizers/trijoint.py.151: total_norm: 17.927559 [I 2022-10-28 01:42:44] ...trap/engines/engine.py.224: train: epoch 99 | batch 980/2304 [I 2022-10-28 01:42:44] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:12:19 [I 2022-10-28 01:42:44] ...trap/engines/engine.py.229: process: 0.54897 | load: 0.00253 [I 2022-10-28 01:42:44] ...trap/engines/engine.py.230: loss: 0.18766 [I 2022-10-28 01:42:44] ...optimizers/trijoint.py.151: total_norm: 13.123398 [I 2022-10-28 01:42:50] ...trap/engines/engine.py.224: train: epoch 99 | batch 990/2304 [I 2022-10-28 01:42:50] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:12:14 [I 2022-10-28 01:42:50] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00242 [I 2022-10-28 01:42:50] ...trap/engines/engine.py.230: loss: 0.09881 [I 2022-10-28 01:42:50] ...optimizers/trijoint.py.151: total_norm: 10.752052 [I 2022-10-28 01:42:55] ...trap/engines/engine.py.224: train: epoch 99 | batch 1000/2304 [I 2022-10-28 01:42:55] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:12:08 [I 2022-10-28 01:42:55] ...trap/engines/engine.py.229: process: 0.57224 | load: 0.00241 [I 2022-10-28 01:42:55] ...trap/engines/engine.py.230: loss: 0.16460 [I 2022-10-28 01:42:55] ...optimizers/trijoint.py.151: total_norm: 30.606491 [I 2022-10-28 01:43:01] ...trap/engines/engine.py.224: train: epoch 99 | batch 1010/2304 [I 2022-10-28 01:43:01] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:12:00 [I 2022-10-28 01:43:01] ...trap/engines/engine.py.229: process: 0.56498 | load: 0.00242 [I 2022-10-28 01:43:01] ...trap/engines/engine.py.230: loss: 0.08042 [I 2022-10-28 01:43:01] ...optimizers/trijoint.py.151: total_norm: 12.175184 [I 2022-10-28 01:43:06] ...trap/engines/engine.py.224: train: epoch 99 | batch 1020/2304 [I 2022-10-28 01:43:06] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:51 [I 2022-10-28 01:43:06] ...trap/engines/engine.py.229: process: 0.55018 | load: 0.00243 [I 2022-10-28 01:43:06] ...trap/engines/engine.py.230: loss: 0.10903 [I 2022-10-28 01:43:06] ...optimizers/trijoint.py.151: total_norm: 20.217422 [I 2022-10-28 01:43:12] ...trap/engines/engine.py.224: train: epoch 99 | batch 1030/2304 [I 2022-10-28 01:43:12] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:55 [I 2022-10-28 01:43:12] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00296 [I 2022-10-28 01:43:12] ...trap/engines/engine.py.230: loss: 0.15817 [I 2022-10-28 01:43:12] ...optimizers/trijoint.py.151: total_norm: 11.105327 [I 2022-10-28 01:43:18] ...trap/engines/engine.py.224: train: epoch 99 | batch 1040/2304 [I 2022-10-28 01:43:18] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:40 [I 2022-10-28 01:43:18] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00241 [I 2022-10-28 01:43:18] ...trap/engines/engine.py.230: loss: 0.18222 [I 2022-10-28 01:43:18] ...optimizers/trijoint.py.151: total_norm: 42.215572 [I 2022-10-28 01:43:23] ...trap/engines/engine.py.224: train: epoch 99 | batch 1050/2304 [I 2022-10-28 01:43:23] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:37 [I 2022-10-28 01:43:23] ...trap/engines/engine.py.229: process: 0.54832 | load: 0.00241 [I 2022-10-28 01:43:23] ...trap/engines/engine.py.230: loss: 0.17355 [I 2022-10-28 01:43:23] ...optimizers/trijoint.py.151: total_norm: 28.519407 [I 2022-10-28 01:43:29] ...trap/engines/engine.py.224: train: epoch 99 | batch 1060/2304 [I 2022-10-28 01:43:29] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:11:40 [I 2022-10-28 01:43:29] ...trap/engines/engine.py.229: process: 0.57207 | load: 0.00242 [I 2022-10-28 01:43:29] ...trap/engines/engine.py.230: loss: 0.15764 [I 2022-10-28 01:43:29] ...optimizers/trijoint.py.151: total_norm: 35.801170 [I 2022-10-28 01:43:34] ...trap/engines/engine.py.224: train: epoch 99 | batch 1070/2304 [I 2022-10-28 01:43:34] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:11:31 [I 2022-10-28 01:43:34] ...trap/engines/engine.py.229: process: 0.55558 | load: 0.00242 [I 2022-10-28 01:43:34] ...trap/engines/engine.py.230: loss: 0.11116 [I 2022-10-28 01:43:34] ...optimizers/trijoint.py.151: total_norm: 17.855837 [I 2022-10-28 01:43:40] ...trap/engines/engine.py.224: train: epoch 99 | batch 1080/2304 [I 2022-10-28 01:43:40] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:11:21 [I 2022-10-28 01:43:40] ...trap/engines/engine.py.229: process: 0.56186 | load: 0.00240 [I 2022-10-28 01:43:40] ...trap/engines/engine.py.230: loss: 0.11121 [I 2022-10-28 01:43:40] ...optimizers/trijoint.py.151: total_norm: 15.465144 [I 2022-10-28 01:43:46] ...trap/engines/engine.py.224: train: epoch 99 | batch 1090/2304 [I 2022-10-28 01:43:46] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:11:20 [I 2022-10-28 01:43:46] ...trap/engines/engine.py.229: process: 0.56091 | load: 0.00244 [I 2022-10-28 01:43:46] ...trap/engines/engine.py.230: loss: 0.12756 [I 2022-10-28 01:43:46] ...optimizers/trijoint.py.151: total_norm: 11.953813 [I 2022-10-28 01:43:51] ...trap/engines/engine.py.224: train: epoch 99 | batch 1100/2304 [I 2022-10-28 01:43:51] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:11:06 [I 2022-10-28 01:43:51] ...trap/engines/engine.py.229: process: 0.55532 | load: 0.00241 [I 2022-10-28 01:43:51] ...trap/engines/engine.py.230: loss: 0.13635 [I 2022-10-28 01:43:51] ...optimizers/trijoint.py.151: total_norm: 19.405420 [I 2022-10-28 01:43:57] ...trap/engines/engine.py.224: train: epoch 99 | batch 1110/2304 [I 2022-10-28 01:43:57] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:11:07 [I 2022-10-28 01:43:57] ...trap/engines/engine.py.229: process: 0.56374 | load: 0.00244 [I 2022-10-28 01:43:57] ...trap/engines/engine.py.230: loss: 0.16114 [I 2022-10-28 01:43:57] ...optimizers/trijoint.py.151: total_norm: 16.478279 [I 2022-10-28 01:44:02] ...trap/engines/engine.py.224: train: epoch 99 | batch 1120/2304 [I 2022-10-28 01:44:02] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:59 [I 2022-10-28 01:44:02] ...trap/engines/engine.py.229: process: 0.55128 | load: 0.00242 [I 2022-10-28 01:44:02] ...trap/engines/engine.py.230: loss: 0.13323 [I 2022-10-28 01:44:02] ...optimizers/trijoint.py.151: total_norm: 13.287206 [I 2022-10-28 01:44:08] ...trap/engines/engine.py.224: train: epoch 99 | batch 1130/2304 [I 2022-10-28 01:44:08] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:57 [I 2022-10-28 01:44:08] ...trap/engines/engine.py.229: process: 0.56559 | load: 0.00254 [I 2022-10-28 01:44:08] ...trap/engines/engine.py.230: loss: 0.16928 [I 2022-10-28 01:44:08] ...optimizers/trijoint.py.151: total_norm: 13.633259 [I 2022-10-28 01:44:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 1140/2304 [I 2022-10-28 01:44:13] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:47 [I 2022-10-28 01:44:13] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00252 [I 2022-10-28 01:44:13] ...trap/engines/engine.py.230: loss: 0.12279 [I 2022-10-28 01:44:13] ...optimizers/trijoint.py.151: total_norm: 12.808212 [I 2022-10-28 01:44:19] ...trap/engines/engine.py.224: train: epoch 99 | batch 1150/2304 [I 2022-10-28 01:44:19] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:39 [I 2022-10-28 01:44:19] ...trap/engines/engine.py.229: process: 0.55642 | load: 0.00246 [I 2022-10-28 01:44:19] ...trap/engines/engine.py.230: loss: 0.16808 [I 2022-10-28 01:44:19] ...optimizers/trijoint.py.151: total_norm: 15.372606 [I 2022-10-28 01:44:25] ...trap/engines/engine.py.224: train: epoch 99 | batch 1160/2304 [I 2022-10-28 01:44:25] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:10:35 [I 2022-10-28 01:44:25] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00239 [I 2022-10-28 01:44:25] ...trap/engines/engine.py.230: loss: 0.13535 [I 2022-10-28 01:44:25] ...optimizers/trijoint.py.151: total_norm: 15.330897 [I 2022-10-28 01:44:30] ...trap/engines/engine.py.224: train: epoch 99 | batch 1170/2304 [I 2022-10-28 01:44:30] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:10:26 [I 2022-10-28 01:44:30] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00244 [I 2022-10-28 01:44:30] ...trap/engines/engine.py.230: loss: 0.11050 [I 2022-10-28 01:44:30] ...optimizers/trijoint.py.151: total_norm: 19.837328 [I 2022-10-28 01:44:36] ...trap/engines/engine.py.224: train: epoch 99 | batch 1180/2304 [I 2022-10-28 01:44:36] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:10:24 [I 2022-10-28 01:44:36] ...trap/engines/engine.py.229: process: 0.54824 | load: 0.00245 [I 2022-10-28 01:44:36] ...trap/engines/engine.py.230: loss: 0.10869 [I 2022-10-28 01:44:36] ...optimizers/trijoint.py.151: total_norm: 23.380159 [I 2022-10-28 01:44:41] ...trap/engines/engine.py.224: train: epoch 99 | batch 1190/2304 [I 2022-10-28 01:44:41] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:10:20 [I 2022-10-28 01:44:41] ...trap/engines/engine.py.229: process: 0.55271 | load: 0.00244 [I 2022-10-28 01:44:41] ...trap/engines/engine.py.230: loss: 0.11456 [I 2022-10-28 01:44:41] ...optimizers/trijoint.py.151: total_norm: 22.905355 [I 2022-10-28 01:44:47] ...trap/engines/engine.py.224: train: epoch 99 | batch 1200/2304 [I 2022-10-28 01:44:47] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:10:21 [I 2022-10-28 01:44:47] ...trap/engines/engine.py.229: process: 0.56425 | load: 0.00242 [I 2022-10-28 01:44:47] ...trap/engines/engine.py.230: loss: 0.09245 [I 2022-10-28 01:44:47] ...optimizers/trijoint.py.151: total_norm: 12.299818 [I 2022-10-28 01:44:52] ...trap/engines/engine.py.224: train: epoch 99 | batch 1210/2304 [I 2022-10-28 01:44:52] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:10:09 [I 2022-10-28 01:44:52] ...trap/engines/engine.py.229: process: 0.55641 | load: 0.00241 [I 2022-10-28 01:44:52] ...trap/engines/engine.py.230: loss: 0.13695 [I 2022-10-28 01:44:52] ...optimizers/trijoint.py.151: total_norm: 20.200401 [I 2022-10-28 01:44:58] ...trap/engines/engine.py.224: train: epoch 99 | batch 1220/2304 [I 2022-10-28 01:44:58] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:10:01 [I 2022-10-28 01:44:58] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00240 [I 2022-10-28 01:44:58] ...trap/engines/engine.py.230: loss: 0.10885 [I 2022-10-28 01:44:58] ...optimizers/trijoint.py.151: total_norm: 11.308272 [I 2022-10-28 01:45:04] ...trap/engines/engine.py.224: train: epoch 99 | batch 1230/2304 [I 2022-10-28 01:45:04] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:55 [I 2022-10-28 01:45:04] ...trap/engines/engine.py.229: process: 0.54778 | load: 0.00244 [I 2022-10-28 01:45:04] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-28 01:45:04] ...optimizers/trijoint.py.151: total_norm: 9.645056 [I 2022-10-28 01:45:09] ...trap/engines/engine.py.224: train: epoch 99 | batch 1240/2304 [I 2022-10-28 01:45:09] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:50 [I 2022-10-28 01:45:09] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00244 [I 2022-10-28 01:45:09] ...trap/engines/engine.py.230: loss: 0.23528 [I 2022-10-28 01:45:09] ...optimizers/trijoint.py.151: total_norm: 23.262175 [I 2022-10-28 01:45:15] ...trap/engines/engine.py.224: train: epoch 99 | batch 1250/2304 [I 2022-10-28 01:45:15] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:44 [I 2022-10-28 01:45:15] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00243 [I 2022-10-28 01:45:15] ...trap/engines/engine.py.230: loss: 0.18864 [I 2022-10-28 01:45:15] ...optimizers/trijoint.py.151: total_norm: 36.112812 [I 2022-10-28 01:45:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 1260/2304 [I 2022-10-28 01:45:20] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:35 [I 2022-10-28 01:45:20] ...trap/engines/engine.py.229: process: 0.55292 | load: 0.00248 [I 2022-10-28 01:45:20] ...trap/engines/engine.py.230: loss: 0.16146 [I 2022-10-28 01:45:20] ...optimizers/trijoint.py.151: total_norm: 17.352888 [I 2022-10-28 01:45:26] ...trap/engines/engine.py.224: train: epoch 99 | batch 1270/2304 [I 2022-10-28 01:45:26] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:09:38 [I 2022-10-28 01:45:26] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00244 [I 2022-10-28 01:45:26] ...trap/engines/engine.py.230: loss: 0.10331 [I 2022-10-28 01:45:26] ...optimizers/trijoint.py.151: total_norm: 8.701056 [I 2022-10-28 01:45:31] ...trap/engines/engine.py.224: train: epoch 99 | batch 1280/2304 [I 2022-10-28 01:45:31] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:09:31 [I 2022-10-28 01:45:31] ...trap/engines/engine.py.229: process: 0.54785 | load: 0.00256 [I 2022-10-28 01:45:31] ...trap/engines/engine.py.230: loss: 0.10850 [I 2022-10-28 01:45:31] ...optimizers/trijoint.py.151: total_norm: 12.464086 [I 2022-10-28 01:45:37] ...trap/engines/engine.py.224: train: epoch 99 | batch 1290/2304 [I 2022-10-28 01:45:37] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:09:33 [I 2022-10-28 01:45:37] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00245 [I 2022-10-28 01:45:37] ...trap/engines/engine.py.230: loss: 0.11637 [I 2022-10-28 01:45:37] ...optimizers/trijoint.py.151: total_norm: 24.874786 [I 2022-10-28 01:45:43] ...trap/engines/engine.py.224: train: epoch 99 | batch 1300/2304 [I 2022-10-28 01:45:43] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:09:30 [I 2022-10-28 01:45:43] ...trap/engines/engine.py.229: process: 0.57843 | load: 0.00243 [I 2022-10-28 01:45:43] ...trap/engines/engine.py.230: loss: 0.08507 [I 2022-10-28 01:45:43] ...optimizers/trijoint.py.151: total_norm: 9.687347 [I 2022-10-28 01:45:48] ...trap/engines/engine.py.224: train: epoch 99 | batch 1310/2304 [I 2022-10-28 01:45:48] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:09:11 [I 2022-10-28 01:45:48] ...trap/engines/engine.py.229: process: 0.54939 | load: 0.00245 [I 2022-10-28 01:45:48] ...trap/engines/engine.py.230: loss: 0.09612 [I 2022-10-28 01:45:48] ...optimizers/trijoint.py.151: total_norm: 18.834135 [I 2022-10-28 01:45:54] ...trap/engines/engine.py.224: train: epoch 99 | batch 1320/2304 [I 2022-10-28 01:45:54] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:09:06 [I 2022-10-28 01:45:54] ...trap/engines/engine.py.229: process: 0.56718 | load: 0.00238 [I 2022-10-28 01:45:54] ...trap/engines/engine.py.230: loss: 0.14260 [I 2022-10-28 01:45:54] ...optimizers/trijoint.py.151: total_norm: 20.622828 [I 2022-10-28 01:45:59] ...trap/engines/engine.py.224: train: epoch 99 | batch 1330/2304 [I 2022-10-28 01:45:59] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:58 [I 2022-10-28 01:45:59] ...trap/engines/engine.py.229: process: 0.55506 | load: 0.00242 [I 2022-10-28 01:45:59] ...trap/engines/engine.py.230: loss: 0.09965 [I 2022-10-28 01:45:59] ...optimizers/trijoint.py.151: total_norm: 16.616974 [I 2022-10-28 01:46:05] ...trap/engines/engine.py.224: train: epoch 99 | batch 1340/2304 [I 2022-10-28 01:46:05] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:54 [I 2022-10-28 01:46:05] ...trap/engines/engine.py.229: process: 0.55157 | load: 0.00241 [I 2022-10-28 01:46:05] ...trap/engines/engine.py.230: loss: 0.17953 [I 2022-10-28 01:46:05] ...optimizers/trijoint.py.151: total_norm: 33.923298 [I 2022-10-28 01:46:11] ...trap/engines/engine.py.224: train: epoch 99 | batch 1350/2304 [I 2022-10-28 01:46:11] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:53 [I 2022-10-28 01:46:11] ...trap/engines/engine.py.229: process: 0.56185 | load: 0.00244 [I 2022-10-28 01:46:11] ...trap/engines/engine.py.230: loss: 0.14625 [I 2022-10-28 01:46:11] ...optimizers/trijoint.py.151: total_norm: 23.042528 [I 2022-10-28 01:46:16] ...trap/engines/engine.py.224: train: epoch 99 | batch 1360/2304 [I 2022-10-28 01:46:16] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:42 [I 2022-10-28 01:46:16] ...trap/engines/engine.py.229: process: 0.53969 | load: 0.00243 [I 2022-10-28 01:46:16] ...trap/engines/engine.py.230: loss: 0.09907 [I 2022-10-28 01:46:16] ...optimizers/trijoint.py.151: total_norm: 7.773402 [I 2022-10-28 01:46:22] ...trap/engines/engine.py.224: train: epoch 99 | batch 1370/2304 [I 2022-10-28 01:46:22] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:36 [I 2022-10-28 01:46:22] ...trap/engines/engine.py.229: process: 0.55150 | load: 0.00245 [I 2022-10-28 01:46:22] ...trap/engines/engine.py.230: loss: 0.10893 [I 2022-10-28 01:46:22] ...optimizers/trijoint.py.151: total_norm: 9.662118 [I 2022-10-28 01:46:27] ...trap/engines/engine.py.224: train: epoch 99 | batch 1380/2304 [I 2022-10-28 01:46:27] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:08:31 [I 2022-10-28 01:46:27] ...trap/engines/engine.py.229: process: 0.55880 | load: 0.00243 [I 2022-10-28 01:46:27] ...trap/engines/engine.py.230: loss: 0.12007 [I 2022-10-28 01:46:27] ...optimizers/trijoint.py.151: total_norm: 14.659945 [I 2022-10-28 01:46:33] ...trap/engines/engine.py.224: train: epoch 99 | batch 1390/2304 [I 2022-10-28 01:46:33] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:08:24 [I 2022-10-28 01:46:33] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00245 [I 2022-10-28 01:46:33] ...trap/engines/engine.py.230: loss: 0.16528 [I 2022-10-28 01:46:33] ...optimizers/trijoint.py.151: total_norm: 21.380589 [I 2022-10-28 01:46:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 1400/2304 [I 2022-10-28 01:46:38] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:08:28 [I 2022-10-28 01:46:38] ...trap/engines/engine.py.229: process: 0.54489 | load: 0.00243 [I 2022-10-28 01:46:38] ...trap/engines/engine.py.230: loss: 0.12008 [I 2022-10-28 01:46:38] ...optimizers/trijoint.py.151: total_norm: 7.208378 [I 2022-10-28 01:46:44] ...trap/engines/engine.py.224: train: epoch 99 | batch 1410/2304 [I 2022-10-28 01:46:44] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:08:23 [I 2022-10-28 01:46:44] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00239 [I 2022-10-28 01:46:44] ...trap/engines/engine.py.230: loss: 0.11397 [I 2022-10-28 01:46:44] ...optimizers/trijoint.py.151: total_norm: 12.957702 [I 2022-10-28 01:46:50] ...trap/engines/engine.py.224: train: epoch 99 | batch 1420/2304 [I 2022-10-28 01:46:50] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:08:20 [I 2022-10-28 01:46:50] ...trap/engines/engine.py.229: process: 0.56405 | load: 0.00242 [I 2022-10-28 01:46:50] ...trap/engines/engine.py.230: loss: 0.14987 [I 2022-10-28 01:46:50] ...optimizers/trijoint.py.151: total_norm: 20.992151 [I 2022-10-28 01:46:55] ...trap/engines/engine.py.224: train: epoch 99 | batch 1430/2304 [I 2022-10-28 01:46:55] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:08:04 [I 2022-10-28 01:46:55] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00243 [I 2022-10-28 01:46:55] ...trap/engines/engine.py.230: loss: 0.08477 [I 2022-10-28 01:46:55] ...optimizers/trijoint.py.151: total_norm: 11.226415 [I 2022-10-28 01:47:01] ...trap/engines/engine.py.224: train: epoch 99 | batch 1440/2304 [I 2022-10-28 01:47:01] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:56 [I 2022-10-28 01:47:01] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00244 [I 2022-10-28 01:47:01] ...trap/engines/engine.py.230: loss: 0.19853 [I 2022-10-28 01:47:01] ...optimizers/trijoint.py.151: total_norm: 10.987858 [I 2022-10-28 01:47:06] ...trap/engines/engine.py.224: train: epoch 99 | batch 1450/2304 [I 2022-10-28 01:47:06] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:53 [I 2022-10-28 01:47:06] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00251 [I 2022-10-28 01:47:06] ...trap/engines/engine.py.230: loss: 0.10241 [I 2022-10-28 01:47:06] ...optimizers/trijoint.py.151: total_norm: 8.467071 [I 2022-10-28 01:47:12] ...trap/engines/engine.py.224: train: epoch 99 | batch 1460/2304 [I 2022-10-28 01:47:12] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:48 [I 2022-10-28 01:47:12] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00246 [I 2022-10-28 01:47:12] ...trap/engines/engine.py.230: loss: 0.13032 [I 2022-10-28 01:47:12] ...optimizers/trijoint.py.151: total_norm: 26.741543 [I 2022-10-28 01:47:17] ...trap/engines/engine.py.224: train: epoch 99 | batch 1470/2304 [I 2022-10-28 01:47:17] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:43 [I 2022-10-28 01:47:17] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00266 [I 2022-10-28 01:47:17] ...trap/engines/engine.py.230: loss: 0.10329 [I 2022-10-28 01:47:17] ...optimizers/trijoint.py.151: total_norm: 15.422332 [I 2022-10-28 01:47:23] ...trap/engines/engine.py.224: train: epoch 99 | batch 1480/2304 [I 2022-10-28 01:47:23] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:38 [I 2022-10-28 01:47:23] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00243 [I 2022-10-28 01:47:23] ...trap/engines/engine.py.230: loss: 0.09349 [I 2022-10-28 01:47:23] ...optimizers/trijoint.py.151: total_norm: 16.535376 [I 2022-10-28 01:47:29] ...trap/engines/engine.py.224: train: epoch 99 | batch 1490/2304 [I 2022-10-28 01:47:29] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:07:33 [I 2022-10-28 01:47:29] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00249 [I 2022-10-28 01:47:29] ...trap/engines/engine.py.230: loss: 0.09792 [I 2022-10-28 01:47:29] ...optimizers/trijoint.py.151: total_norm: 26.050261 [I 2022-10-28 01:47:34] ...trap/engines/engine.py.224: train: epoch 99 | batch 1500/2304 [I 2022-10-28 01:47:34] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:07:27 [I 2022-10-28 01:47:34] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00242 [I 2022-10-28 01:47:34] ...trap/engines/engine.py.230: loss: 0.14893 [I 2022-10-28 01:47:34] ...optimizers/trijoint.py.151: total_norm: 15.869678 [I 2022-10-28 01:47:40] ...trap/engines/engine.py.224: train: epoch 99 | batch 1510/2304 [I 2022-10-28 01:47:40] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:07:24 [I 2022-10-28 01:47:40] ...trap/engines/engine.py.229: process: 0.55832 | load: 0.00244 [I 2022-10-28 01:47:40] ...trap/engines/engine.py.230: loss: 0.14194 [I 2022-10-28 01:47:40] ...optimizers/trijoint.py.151: total_norm: 17.470850 [I 2022-10-28 01:47:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 1520/2304 [I 2022-10-28 01:47:45] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:07:15 [I 2022-10-28 01:47:45] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00250 [I 2022-10-28 01:47:45] ...trap/engines/engine.py.230: loss: 0.18964 [I 2022-10-28 01:47:45] ...optimizers/trijoint.py.151: total_norm: 47.180218 [I 2022-10-28 01:47:51] ...trap/engines/engine.py.224: train: epoch 99 | batch 1530/2304 [I 2022-10-28 01:47:51] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:07:08 [I 2022-10-28 01:47:51] ...trap/engines/engine.py.229: process: 0.55039 | load: 0.00242 [I 2022-10-28 01:47:51] ...trap/engines/engine.py.230: loss: 0.16333 [I 2022-10-28 01:47:51] ...optimizers/trijoint.py.151: total_norm: 12.148453 [I 2022-10-28 01:47:56] ...trap/engines/engine.py.224: train: epoch 99 | batch 1540/2304 [I 2022-10-28 01:47:56] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:07:06 [I 2022-10-28 01:47:56] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00240 [I 2022-10-28 01:47:56] ...trap/engines/engine.py.230: loss: 0.15942 [I 2022-10-28 01:47:56] ...optimizers/trijoint.py.151: total_norm: 19.121277 [I 2022-10-28 01:48:02] ...trap/engines/engine.py.224: train: epoch 99 | batch 1550/2304 [I 2022-10-28 01:48:02] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:07:09 [I 2022-10-28 01:48:02] ...trap/engines/engine.py.229: process: 0.54655 | load: 0.00245 [I 2022-10-28 01:48:02] ...trap/engines/engine.py.230: loss: 0.09459 [I 2022-10-28 01:48:02] ...optimizers/trijoint.py.151: total_norm: 7.663483 [I 2022-10-28 01:48:08] ...trap/engines/engine.py.224: train: epoch 99 | batch 1560/2304 [I 2022-10-28 01:48:08] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:53 [I 2022-10-28 01:48:08] ...trap/engines/engine.py.229: process: 0.54825 | load: 0.00242 [I 2022-10-28 01:48:08] ...trap/engines/engine.py.230: loss: 0.11927 [I 2022-10-28 01:48:08] ...optimizers/trijoint.py.151: total_norm: 14.036118 [I 2022-10-28 01:48:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 1570/2304 [I 2022-10-28 01:48:13] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:48 [I 2022-10-28 01:48:13] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00250 [I 2022-10-28 01:48:13] ...trap/engines/engine.py.230: loss: 0.12964 [I 2022-10-28 01:48:13] ...optimizers/trijoint.py.151: total_norm: 18.841940 [I 2022-10-28 01:48:19] ...trap/engines/engine.py.224: train: epoch 99 | batch 1580/2304 [I 2022-10-28 01:48:19] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:45 [I 2022-10-28 01:48:19] ...trap/engines/engine.py.229: process: 0.54940 | load: 0.00248 [I 2022-10-28 01:48:19] ...trap/engines/engine.py.230: loss: 0.17843 [I 2022-10-28 01:48:19] ...optimizers/trijoint.py.151: total_norm: 15.883604 [I 2022-10-28 01:48:25] ...trap/engines/engine.py.224: train: epoch 99 | batch 1590/2304 [I 2022-10-28 01:48:25] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:37 [I 2022-10-28 01:48:25] ...trap/engines/engine.py.229: process: 0.56685 | load: 0.00250 [I 2022-10-28 01:48:25] ...trap/engines/engine.py.230: loss: 0.16583 [I 2022-10-28 01:48:25] ...optimizers/trijoint.py.151: total_norm: 17.322227 [I 2022-10-28 01:48:30] ...trap/engines/engine.py.224: train: epoch 99 | batch 1600/2304 [I 2022-10-28 01:48:30] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:32 [I 2022-10-28 01:48:30] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00241 [I 2022-10-28 01:48:30] ...trap/engines/engine.py.230: loss: 0.09724 [I 2022-10-28 01:48:30] ...optimizers/trijoint.py.151: total_norm: 9.329895 [I 2022-10-28 01:48:36] ...trap/engines/engine.py.224: train: epoch 99 | batch 1610/2304 [I 2022-10-28 01:48:36] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:06:26 [I 2022-10-28 01:48:36] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00241 [I 2022-10-28 01:48:36] ...trap/engines/engine.py.230: loss: 0.17608 [I 2022-10-28 01:48:36] ...optimizers/trijoint.py.151: total_norm: 21.418070 [I 2022-10-28 01:48:41] ...trap/engines/engine.py.224: train: epoch 99 | batch 1620/2304 [I 2022-10-28 01:48:41] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:06:21 [I 2022-10-28 01:48:41] ...trap/engines/engine.py.229: process: 0.56308 | load: 0.00244 [I 2022-10-28 01:48:41] ...trap/engines/engine.py.230: loss: 0.16404 [I 2022-10-28 01:48:41] ...optimizers/trijoint.py.151: total_norm: 35.720959 [I 2022-10-28 01:48:47] ...trap/engines/engine.py.224: train: epoch 99 | batch 1630/2304 [I 2022-10-28 01:48:47] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:06:16 [I 2022-10-28 01:48:47] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00240 [I 2022-10-28 01:48:47] ...trap/engines/engine.py.230: loss: 0.11495 [I 2022-10-28 01:48:47] ...optimizers/trijoint.py.151: total_norm: 37.121765 [I 2022-10-28 01:48:52] ...trap/engines/engine.py.224: train: epoch 99 | batch 1640/2304 [I 2022-10-28 01:48:52] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:06:11 [I 2022-10-28 01:48:52] ...trap/engines/engine.py.229: process: 0.56851 | load: 0.00243 [I 2022-10-28 01:48:52] ...trap/engines/engine.py.230: loss: 0.11567 [I 2022-10-28 01:48:52] ...optimizers/trijoint.py.151: total_norm: 9.490715 [I 2022-10-28 01:48:58] ...trap/engines/engine.py.224: train: epoch 99 | batch 1650/2304 [I 2022-10-28 01:48:58] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:06:02 [I 2022-10-28 01:48:58] ...trap/engines/engine.py.229: process: 0.54722 | load: 0.00244 [I 2022-10-28 01:48:58] ...trap/engines/engine.py.230: loss: 0.17720 [I 2022-10-28 01:48:58] ...optimizers/trijoint.py.151: total_norm: 10.803950 [I 2022-10-28 01:49:04] ...trap/engines/engine.py.224: train: epoch 99 | batch 1660/2304 [I 2022-10-28 01:49:04] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:57 [I 2022-10-28 01:49:04] ...trap/engines/engine.py.229: process: 0.55264 | load: 0.00243 [I 2022-10-28 01:49:04] ...trap/engines/engine.py.230: loss: 0.12863 [I 2022-10-28 01:49:04] ...optimizers/trijoint.py.151: total_norm: 16.264400 [I 2022-10-28 01:49:09] ...trap/engines/engine.py.224: train: epoch 99 | batch 1670/2304 [I 2022-10-28 01:49:09] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:53 [I 2022-10-28 01:49:09] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00243 [I 2022-10-28 01:49:09] ...trap/engines/engine.py.230: loss: 0.12111 [I 2022-10-28 01:49:09] ...optimizers/trijoint.py.151: total_norm: 10.506518 [I 2022-10-28 01:49:15] ...trap/engines/engine.py.224: train: epoch 99 | batch 1680/2304 [I 2022-10-28 01:49:15] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:46 [I 2022-10-28 01:49:15] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00245 [I 2022-10-28 01:49:15] ...trap/engines/engine.py.230: loss: 0.11159 [I 2022-10-28 01:49:15] ...optimizers/trijoint.py.151: total_norm: 20.726744 [I 2022-10-28 01:49:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 1690/2304 [I 2022-10-28 01:49:20] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:41 [I 2022-10-28 01:49:20] ...trap/engines/engine.py.229: process: 0.54431 | load: 0.00243 [I 2022-10-28 01:49:20] ...trap/engines/engine.py.230: loss: 0.14202 [I 2022-10-28 01:49:20] ...optimizers/trijoint.py.151: total_norm: 24.006582 [I 2022-10-28 01:49:26] ...trap/engines/engine.py.224: train: epoch 99 | batch 1700/2304 [I 2022-10-28 01:49:26] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:34 [I 2022-10-28 01:49:26] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00240 [I 2022-10-28 01:49:26] ...trap/engines/engine.py.230: loss: 0.13174 [I 2022-10-28 01:49:26] ...optimizers/trijoint.py.151: total_norm: 41.127281 [I 2022-10-28 01:49:31] ...trap/engines/engine.py.224: train: epoch 99 | batch 1710/2304 [I 2022-10-28 01:49:31] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:05:30 [I 2022-10-28 01:49:31] ...trap/engines/engine.py.229: process: 0.55466 | load: 0.00241 [I 2022-10-28 01:49:31] ...trap/engines/engine.py.230: loss: 0.09135 [I 2022-10-28 01:49:31] ...optimizers/trijoint.py.151: total_norm: 12.519467 [I 2022-10-28 01:49:37] ...trap/engines/engine.py.224: train: epoch 99 | batch 1720/2304 [I 2022-10-28 01:49:37] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:26 [I 2022-10-28 01:49:37] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00247 [I 2022-10-28 01:49:37] ...trap/engines/engine.py.230: loss: 0.16994 [I 2022-10-28 01:49:37] ...optimizers/trijoint.py.151: total_norm: 37.702168 [I 2022-10-28 01:49:43] ...trap/engines/engine.py.224: train: epoch 99 | batch 1730/2304 [I 2022-10-28 01:49:43] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:05:21 [I 2022-10-28 01:49:43] ...trap/engines/engine.py.229: process: 0.56365 | load: 0.00242 [I 2022-10-28 01:49:43] ...trap/engines/engine.py.230: loss: 0.12196 [I 2022-10-28 01:49:43] ...optimizers/trijoint.py.151: total_norm: 16.238422 [I 2022-10-28 01:49:48] ...trap/engines/engine.py.224: train: epoch 99 | batch 1740/2304 [I 2022-10-28 01:49:48] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:05:17 [I 2022-10-28 01:49:48] ...trap/engines/engine.py.229: process: 0.56455 | load: 0.00244 [I 2022-10-28 01:49:48] ...trap/engines/engine.py.230: loss: 0.15454 [I 2022-10-28 01:49:48] ...optimizers/trijoint.py.151: total_norm: 19.795389 [I 2022-10-28 01:49:54] ...trap/engines/engine.py.224: train: epoch 99 | batch 1750/2304 [I 2022-10-28 01:49:54] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:05:08 [I 2022-10-28 01:49:54] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00245 [I 2022-10-28 01:49:54] ...trap/engines/engine.py.230: loss: 0.13493 [I 2022-10-28 01:49:54] ...optimizers/trijoint.py.151: total_norm: 16.647614 [I 2022-10-28 01:49:59] ...trap/engines/engine.py.224: train: epoch 99 | batch 1760/2304 [I 2022-10-28 01:49:59] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:05:04 [I 2022-10-28 01:49:59] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00243 [I 2022-10-28 01:49:59] ...trap/engines/engine.py.230: loss: 0.12629 [I 2022-10-28 01:49:59] ...optimizers/trijoint.py.151: total_norm: 11.536389 [I 2022-10-28 01:50:05] ...trap/engines/engine.py.224: train: epoch 99 | batch 1770/2304 [I 2022-10-28 01:50:05] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:57 [I 2022-10-28 01:50:05] ...trap/engines/engine.py.229: process: 0.55029 | load: 0.00256 [I 2022-10-28 01:50:05] ...trap/engines/engine.py.230: loss: 0.13353 [I 2022-10-28 01:50:05] ...optimizers/trijoint.py.151: total_norm: 17.125401 [I 2022-10-28 01:50:11] ...trap/engines/engine.py.224: train: epoch 99 | batch 1780/2304 [I 2022-10-28 01:50:11] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:49 [I 2022-10-28 01:50:11] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00251 [I 2022-10-28 01:50:11] ...trap/engines/engine.py.230: loss: 0.11362 [I 2022-10-28 01:50:11] ...optimizers/trijoint.py.151: total_norm: 15.793077 [I 2022-10-28 01:50:16] ...trap/engines/engine.py.224: train: epoch 99 | batch 1790/2304 [I 2022-10-28 01:50:16] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:50 [I 2022-10-28 01:50:16] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00246 [I 2022-10-28 01:50:16] ...trap/engines/engine.py.230: loss: 0.10221 [I 2022-10-28 01:50:16] ...optimizers/trijoint.py.151: total_norm: 8.753286 [I 2022-10-28 01:50:22] ...trap/engines/engine.py.224: train: epoch 99 | batch 1800/2304 [I 2022-10-28 01:50:22] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:40 [I 2022-10-28 01:50:22] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00244 [I 2022-10-28 01:50:22] ...trap/engines/engine.py.230: loss: 0.15179 [I 2022-10-28 01:50:22] ...optimizers/trijoint.py.151: total_norm: 41.624504 [I 2022-10-28 01:50:27] ...trap/engines/engine.py.224: train: epoch 99 | batch 1810/2304 [I 2022-10-28 01:50:27] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:04:31 [I 2022-10-28 01:50:27] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00245 [I 2022-10-28 01:50:27] ...trap/engines/engine.py.230: loss: 0.08824 [I 2022-10-28 01:50:27] ...optimizers/trijoint.py.151: total_norm: 17.437584 [I 2022-10-28 01:50:33] ...trap/engines/engine.py.224: train: epoch 99 | batch 1820/2304 [I 2022-10-28 01:50:33] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:04:28 [I 2022-10-28 01:50:33] ...trap/engines/engine.py.229: process: 0.55113 | load: 0.00241 [I 2022-10-28 01:50:33] ...trap/engines/engine.py.230: loss: 0.13311 [I 2022-10-28 01:50:33] ...optimizers/trijoint.py.151: total_norm: 124.037392 [I 2022-10-28 01:50:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 1830/2304 [I 2022-10-28 01:50:38] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:04:24 [I 2022-10-28 01:50:38] ...trap/engines/engine.py.229: process: 0.55925 | load: 0.00242 [I 2022-10-28 01:50:38] ...trap/engines/engine.py.230: loss: 0.07867 [I 2022-10-28 01:50:38] ...optimizers/trijoint.py.151: total_norm: 16.985065 [I 2022-10-28 01:50:44] ...trap/engines/engine.py.224: train: epoch 99 | batch 1840/2304 [I 2022-10-28 01:50:44] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:04:17 [I 2022-10-28 01:50:44] ...trap/engines/engine.py.229: process: 0.55563 | load: 0.00243 [I 2022-10-28 01:50:44] ...trap/engines/engine.py.230: loss: 0.10638 [I 2022-10-28 01:50:44] ...optimizers/trijoint.py.151: total_norm: 9.086589 [I 2022-10-28 01:50:50] ...trap/engines/engine.py.224: train: epoch 99 | batch 1850/2304 [I 2022-10-28 01:50:50] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:04:12 [I 2022-10-28 01:50:50] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00240 [I 2022-10-28 01:50:50] ...trap/engines/engine.py.230: loss: 0.12086 [I 2022-10-28 01:50:50] ...optimizers/trijoint.py.151: total_norm: 14.966432 [I 2022-10-28 01:50:55] ...trap/engines/engine.py.224: train: epoch 99 | batch 1860/2304 [I 2022-10-28 01:50:55] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:04:06 [I 2022-10-28 01:50:55] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00240 [I 2022-10-28 01:50:55] ...trap/engines/engine.py.230: loss: 0.16058 [I 2022-10-28 01:50:55] ...optimizers/trijoint.py.151: total_norm: 10.398222 [I 2022-10-28 01:51:01] ...trap/engines/engine.py.224: train: epoch 99 | batch 1870/2304 [I 2022-10-28 01:51:01] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:04:00 [I 2022-10-28 01:51:01] ...trap/engines/engine.py.229: process: 0.53882 | load: 0.00244 [I 2022-10-28 01:51:01] ...trap/engines/engine.py.230: loss: 0.12587 [I 2022-10-28 01:51:01] ...optimizers/trijoint.py.151: total_norm: 20.446314 [I 2022-10-28 01:51:06] ...trap/engines/engine.py.224: train: epoch 99 | batch 1880/2304 [I 2022-10-28 01:51:06] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:55 [I 2022-10-28 01:51:06] ...trap/engines/engine.py.229: process: 0.55552 | load: 0.00243 [I 2022-10-28 01:51:06] ...trap/engines/engine.py.230: loss: 0.10977 [I 2022-10-28 01:51:06] ...optimizers/trijoint.py.151: total_norm: 17.783394 [I 2022-10-28 01:51:12] ...trap/engines/engine.py.224: train: epoch 99 | batch 1890/2304 [I 2022-10-28 01:51:12] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:50 [I 2022-10-28 01:51:12] ...trap/engines/engine.py.229: process: 0.55526 | load: 0.00243 [I 2022-10-28 01:51:12] ...trap/engines/engine.py.230: loss: 0.15998 [I 2022-10-28 01:51:12] ...optimizers/trijoint.py.151: total_norm: 23.747381 [I 2022-10-28 01:51:18] ...trap/engines/engine.py.224: train: epoch 99 | batch 1900/2304 [I 2022-10-28 01:51:18] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:46 [I 2022-10-28 01:51:18] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00242 [I 2022-10-28 01:51:18] ...trap/engines/engine.py.230: loss: 0.10202 [I 2022-10-28 01:51:18] ...optimizers/trijoint.py.151: total_norm: 39.583801 [I 2022-10-28 01:51:23] ...trap/engines/engine.py.224: train: epoch 99 | batch 1910/2304 [I 2022-10-28 01:51:23] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:37 [I 2022-10-28 01:51:23] ...trap/engines/engine.py.229: process: 0.54755 | load: 0.00241 [I 2022-10-28 01:51:23] ...trap/engines/engine.py.230: loss: 0.15539 [I 2022-10-28 01:51:23] ...optimizers/trijoint.py.151: total_norm: 12.752074 [I 2022-10-28 01:51:29] ...trap/engines/engine.py.224: train: epoch 99 | batch 1920/2304 [I 2022-10-28 01:51:29] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:31 [I 2022-10-28 01:51:29] ...trap/engines/engine.py.229: process: 0.55263 | load: 0.00241 [I 2022-10-28 01:51:29] ...trap/engines/engine.py.230: loss: 0.09466 [I 2022-10-28 01:51:29] ...optimizers/trijoint.py.151: total_norm: 20.565664 [I 2022-10-28 01:51:34] ...trap/engines/engine.py.224: train: epoch 99 | batch 1930/2304 [I 2022-10-28 01:51:34] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:03:27 [I 2022-10-28 01:51:34] ...trap/engines/engine.py.229: process: 0.55523 | load: 0.00240 [I 2022-10-28 01:51:34] ...trap/engines/engine.py.230: loss: 0.08760 [I 2022-10-28 01:51:34] ...optimizers/trijoint.py.151: total_norm: 11.538342 [I 2022-10-28 01:51:40] ...trap/engines/engine.py.224: train: epoch 99 | batch 1940/2304 [I 2022-10-28 01:51:40] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:03:21 [I 2022-10-28 01:51:40] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00243 [I 2022-10-28 01:51:40] ...trap/engines/engine.py.230: loss: 0.14002 [I 2022-10-28 01:51:40] ...optimizers/trijoint.py.151: total_norm: 10.933108 [I 2022-10-28 01:51:45] ...trap/engines/engine.py.224: train: epoch 99 | batch 1950/2304 [I 2022-10-28 01:51:45] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:03:16 [I 2022-10-28 01:51:45] ...trap/engines/engine.py.229: process: 0.54560 | load: 0.00247 [I 2022-10-28 01:51:45] ...trap/engines/engine.py.230: loss: 0.15510 [I 2022-10-28 01:51:45] ...optimizers/trijoint.py.151: total_norm: 7.588359 [I 2022-10-28 01:51:51] ...trap/engines/engine.py.224: train: epoch 99 | batch 1960/2304 [I 2022-10-28 01:51:51] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:03:10 [I 2022-10-28 01:51:51] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00244 [I 2022-10-28 01:51:51] ...trap/engines/engine.py.230: loss: 0.16615 [I 2022-10-28 01:51:51] ...optimizers/trijoint.py.151: total_norm: 9.520003 [I 2022-10-28 01:51:57] ...trap/engines/engine.py.224: train: epoch 99 | batch 1970/2304 [I 2022-10-28 01:51:57] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:03:05 [I 2022-10-28 01:51:57] ...trap/engines/engine.py.229: process: 0.55307 | load: 0.00242 [I 2022-10-28 01:51:57] ...trap/engines/engine.py.230: loss: 0.08292 [I 2022-10-28 01:51:57] ...optimizers/trijoint.py.151: total_norm: 10.466522 [I 2022-10-28 01:52:02] ...trap/engines/engine.py.224: train: epoch 99 | batch 1980/2304 [I 2022-10-28 01:52:02] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:03:00 [I 2022-10-28 01:52:02] ...trap/engines/engine.py.229: process: 0.56319 | load: 0.00245 [I 2022-10-28 01:52:02] ...trap/engines/engine.py.230: loss: 0.11464 [I 2022-10-28 01:52:02] ...optimizers/trijoint.py.151: total_norm: 9.239491 [I 2022-10-28 01:52:08] ...trap/engines/engine.py.224: train: epoch 99 | batch 1990/2304 [I 2022-10-28 01:52:08] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:54 [I 2022-10-28 01:52:08] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00241 [I 2022-10-28 01:52:08] ...trap/engines/engine.py.230: loss: 0.11100 [I 2022-10-28 01:52:08] ...optimizers/trijoint.py.151: total_norm: 9.174647 [I 2022-10-28 01:52:13] ...trap/engines/engine.py.224: train: epoch 99 | batch 2000/2304 [I 2022-10-28 01:52:13] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:49 [I 2022-10-28 01:52:13] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00246 [I 2022-10-28 01:52:13] ...trap/engines/engine.py.230: loss: 0.09774 [I 2022-10-28 01:52:13] ...optimizers/trijoint.py.151: total_norm: 8.651096 [I 2022-10-28 01:52:19] ...trap/engines/engine.py.224: train: epoch 99 | batch 2010/2304 [I 2022-10-28 01:52:19] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:43 [I 2022-10-28 01:52:19] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00246 [I 2022-10-28 01:52:19] ...trap/engines/engine.py.230: loss: 0.14365 [I 2022-10-28 01:52:19] ...optimizers/trijoint.py.151: total_norm: 17.899624 [I 2022-10-28 01:52:24] ...trap/engines/engine.py.224: train: epoch 99 | batch 2020/2304 [I 2022-10-28 01:52:24] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:37 [I 2022-10-28 01:52:24] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00242 [I 2022-10-28 01:52:24] ...trap/engines/engine.py.230: loss: 0.10454 [I 2022-10-28 01:52:24] ...optimizers/trijoint.py.151: total_norm: 15.259649 [I 2022-10-28 01:52:30] ...trap/engines/engine.py.224: train: epoch 99 | batch 2030/2304 [I 2022-10-28 01:52:30] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:02:32 [I 2022-10-28 01:52:30] ...trap/engines/engine.py.229: process: 0.55257 | load: 0.00244 [I 2022-10-28 01:52:30] ...trap/engines/engine.py.230: loss: 0.09816 [I 2022-10-28 01:52:30] ...optimizers/trijoint.py.151: total_norm: 16.530998 [I 2022-10-28 01:52:36] ...trap/engines/engine.py.224: train: epoch 99 | batch 2040/2304 [I 2022-10-28 01:52:36] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:26 [I 2022-10-28 01:52:36] ...trap/engines/engine.py.229: process: 0.55942 | load: 0.00247 [I 2022-10-28 01:52:36] ...trap/engines/engine.py.230: loss: 0.10912 [I 2022-10-28 01:52:36] ...optimizers/trijoint.py.151: total_norm: 17.988924 [I 2022-10-28 01:52:41] ...trap/engines/engine.py.224: train: epoch 99 | batch 2050/2304 [I 2022-10-28 01:52:41] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:02:20 [I 2022-10-28 01:52:41] ...trap/engines/engine.py.229: process: 0.54278 | load: 0.00243 [I 2022-10-28 01:52:41] ...trap/engines/engine.py.230: loss: 0.09010 [I 2022-10-28 01:52:41] ...optimizers/trijoint.py.151: total_norm: 13.072927 [I 2022-10-28 01:52:47] ...trap/engines/engine.py.224: train: epoch 99 | batch 2060/2304 [I 2022-10-28 01:52:47] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:02:15 [I 2022-10-28 01:52:47] ...trap/engines/engine.py.229: process: 0.56344 | load: 0.00244 [I 2022-10-28 01:52:47] ...trap/engines/engine.py.230: loss: 0.10956 [I 2022-10-28 01:52:47] ...optimizers/trijoint.py.151: total_norm: 12.863584 [I 2022-10-28 01:52:52] ...trap/engines/engine.py.224: train: epoch 99 | batch 2070/2304 [I 2022-10-28 01:52:52] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:02:09 [I 2022-10-28 01:52:52] ...trap/engines/engine.py.229: process: 0.54199 | load: 0.00242 [I 2022-10-28 01:52:52] ...trap/engines/engine.py.230: loss: 0.10605 [I 2022-10-28 01:52:52] ...optimizers/trijoint.py.151: total_norm: 12.691080 [I 2022-10-28 01:52:58] ...trap/engines/engine.py.224: train: epoch 99 | batch 2080/2304 [I 2022-10-28 01:52:58] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:02:05 [I 2022-10-28 01:52:58] ...trap/engines/engine.py.229: process: 0.56328 | load: 0.00243 [I 2022-10-28 01:52:58] ...trap/engines/engine.py.230: loss: 0.12131 [I 2022-10-28 01:52:58] ...optimizers/trijoint.py.151: total_norm: 18.574455 [I 2022-10-28 01:53:03] ...trap/engines/engine.py.224: train: epoch 99 | batch 2090/2304 [I 2022-10-28 01:53:03] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:59 [I 2022-10-28 01:53:03] ...trap/engines/engine.py.229: process: 0.54526 | load: 0.00244 [I 2022-10-28 01:53:03] ...trap/engines/engine.py.230: loss: 0.13858 [I 2022-10-28 01:53:03] ...optimizers/trijoint.py.151: total_norm: 17.318045 [I 2022-10-28 01:53:09] ...trap/engines/engine.py.224: train: epoch 99 | batch 2100/2304 [I 2022-10-28 01:53:09] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:53 [I 2022-10-28 01:53:09] ...trap/engines/engine.py.229: process: 0.54791 | load: 0.00243 [I 2022-10-28 01:53:09] ...trap/engines/engine.py.230: loss: 0.16788 [I 2022-10-28 01:53:09] ...optimizers/trijoint.py.151: total_norm: 23.340376 [I 2022-10-28 01:53:15] ...trap/engines/engine.py.224: train: epoch 99 | batch 2110/2304 [I 2022-10-28 01:53:15] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:48 [I 2022-10-28 01:53:15] ...trap/engines/engine.py.229: process: 0.56087 | load: 0.00244 [I 2022-10-28 01:53:15] ...trap/engines/engine.py.230: loss: 0.10785 [I 2022-10-28 01:53:15] ...optimizers/trijoint.py.151: total_norm: 10.504718 [I 2022-10-28 01:53:20] ...trap/engines/engine.py.224: train: epoch 99 | batch 2120/2304 [I 2022-10-28 01:53:20] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:42 [I 2022-10-28 01:53:20] ...trap/engines/engine.py.229: process: 0.56434 | load: 0.00241 [I 2022-10-28 01:53:20] ...trap/engines/engine.py.230: loss: 0.09590 [I 2022-10-28 01:53:20] ...optimizers/trijoint.py.151: total_norm: 14.118723 [I 2022-10-28 01:53:26] ...trap/engines/engine.py.224: train: epoch 99 | batch 2130/2304 [I 2022-10-28 01:53:26] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:01:36 [I 2022-10-28 01:53:26] ...trap/engines/engine.py.229: process: 0.54385 | load: 0.00246 [I 2022-10-28 01:53:26] ...trap/engines/engine.py.230: loss: 0.11873 [I 2022-10-28 01:53:26] ...optimizers/trijoint.py.151: total_norm: 12.200767 [I 2022-10-28 01:53:31] ...trap/engines/engine.py.224: train: epoch 99 | batch 2140/2304 [I 2022-10-28 01:53:31] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:30 [I 2022-10-28 01:53:31] ...trap/engines/engine.py.229: process: 0.55185 | load: 0.00241 [I 2022-10-28 01:53:31] ...trap/engines/engine.py.230: loss: 0.09736 [I 2022-10-28 01:53:31] ...optimizers/trijoint.py.151: total_norm: 28.087511 [I 2022-10-28 01:53:37] ...trap/engines/engine.py.224: train: epoch 99 | batch 2150/2304 [I 2022-10-28 01:53:37] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:01:25 [I 2022-10-28 01:53:37] ...trap/engines/engine.py.229: process: 0.55421 | load: 0.00253 [I 2022-10-28 01:53:37] ...trap/engines/engine.py.230: loss: 0.19687 [I 2022-10-28 01:53:37] ...optimizers/trijoint.py.151: total_norm: 85.226509 [I 2022-10-28 01:53:42] ...trap/engines/engine.py.224: train: epoch 99 | batch 2160/2304 [I 2022-10-28 01:53:42] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:01:20 [I 2022-10-28 01:53:42] ...trap/engines/engine.py.229: process: 0.56821 | load: 0.00248 [I 2022-10-28 01:53:42] ...trap/engines/engine.py.230: loss: 0.08859 [I 2022-10-28 01:53:42] ...optimizers/trijoint.py.151: total_norm: 22.314249 [I 2022-10-28 01:53:48] ...trap/engines/engine.py.224: train: epoch 99 | batch 2170/2304 [I 2022-10-28 01:53:48] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:01:14 [I 2022-10-28 01:53:48] ...trap/engines/engine.py.229: process: 0.55111 | load: 0.00266 [I 2022-10-28 01:53:48] ...trap/engines/engine.py.230: loss: 0.11970 [I 2022-10-28 01:53:48] ...optimizers/trijoint.py.151: total_norm: 19.025793 [I 2022-10-28 01:53:54] ...trap/engines/engine.py.224: train: epoch 99 | batch 2180/2304 [I 2022-10-28 01:53:54] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:01:09 [I 2022-10-28 01:53:54] ...trap/engines/engine.py.229: process: 0.55159 | load: 0.00243 [I 2022-10-28 01:53:54] ...trap/engines/engine.py.230: loss: 0.09044 [I 2022-10-28 01:53:54] ...optimizers/trijoint.py.151: total_norm: 13.028587 [I 2022-10-28 01:53:59] ...trap/engines/engine.py.224: train: epoch 99 | batch 2190/2304 [I 2022-10-28 01:53:59] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:01:03 [I 2022-10-28 01:53:59] ...trap/engines/engine.py.229: process: 0.55344 | load: 0.00242 [I 2022-10-28 01:53:59] ...trap/engines/engine.py.230: loss: 0.13104 [I 2022-10-28 01:53:59] ...optimizers/trijoint.py.151: total_norm: 20.315060 [I 2022-10-28 01:54:05] ...trap/engines/engine.py.224: train: epoch 99 | batch 2200/2304 [I 2022-10-28 01:54:05] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:57 [I 2022-10-28 01:54:05] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00242 [I 2022-10-28 01:54:05] ...trap/engines/engine.py.230: loss: 0.11756 [I 2022-10-28 01:54:05] ...optimizers/trijoint.py.151: total_norm: 12.526732 [I 2022-10-28 01:54:10] ...trap/engines/engine.py.224: train: epoch 99 | batch 2210/2304 [I 2022-10-28 01:54:10] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:52 [I 2022-10-28 01:54:10] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00244 [I 2022-10-28 01:54:10] ...trap/engines/engine.py.230: loss: 0.11900 [I 2022-10-28 01:54:10] ...optimizers/trijoint.py.151: total_norm: 18.702337 [I 2022-10-28 01:54:16] ...trap/engines/engine.py.224: train: epoch 99 | batch 2220/2304 [I 2022-10-28 01:54:16] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:46 [I 2022-10-28 01:54:16] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00244 [I 2022-10-28 01:54:16] ...trap/engines/engine.py.230: loss: 0.13684 [I 2022-10-28 01:54:16] ...optimizers/trijoint.py.151: total_norm: 13.818214 [I 2022-10-28 01:54:22] ...trap/engines/engine.py.224: train: epoch 99 | batch 2230/2304 [I 2022-10-28 01:54:22] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:41 [I 2022-10-28 01:54:22] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00244 [I 2022-10-28 01:54:22] ...trap/engines/engine.py.230: loss: 0.07415 [I 2022-10-28 01:54:22] ...optimizers/trijoint.py.151: total_norm: 12.630775 [I 2022-10-28 01:54:27] ...trap/engines/engine.py.224: train: epoch 99 | batch 2240/2304 [I 2022-10-28 01:54:27] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:35 [I 2022-10-28 01:54:27] ...trap/engines/engine.py.229: process: 0.56418 | load: 0.00248 [I 2022-10-28 01:54:27] ...trap/engines/engine.py.230: loss: 0.14210 [I 2022-10-28 01:54:27] ...optimizers/trijoint.py.151: total_norm: 29.074835 [I 2022-10-28 01:54:33] ...trap/engines/engine.py.224: train: epoch 99 | batch 2250/2304 [I 2022-10-28 01:54:33] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:30 [I 2022-10-28 01:54:33] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00242 [I 2022-10-28 01:54:33] ...trap/engines/engine.py.230: loss: 0.17374 [I 2022-10-28 01:54:33] ...optimizers/trijoint.py.151: total_norm: 17.266872 [I 2022-10-28 01:54:38] ...trap/engines/engine.py.224: train: epoch 99 | batch 2260/2304 [I 2022-10-28 01:54:38] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:24 [I 2022-10-28 01:54:38] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00248 [I 2022-10-28 01:54:38] ...trap/engines/engine.py.230: loss: 0.11412 [I 2022-10-28 01:54:38] ...optimizers/trijoint.py.151: total_norm: 9.793819 [I 2022-10-28 01:54:44] ...trap/engines/engine.py.224: train: epoch 99 | batch 2270/2304 [I 2022-10-28 01:54:44] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:18 [I 2022-10-28 01:54:44] ...trap/engines/engine.py.229: process: 0.54537 | load: 0.00245 [I 2022-10-28 01:54:44] ...trap/engines/engine.py.230: loss: 0.10281 [I 2022-10-28 01:54:44] ...optimizers/trijoint.py.151: total_norm: 13.231112 [I 2022-10-28 01:54:49] ...trap/engines/engine.py.224: train: epoch 99 | batch 2280/2304 [I 2022-10-28 01:54:49] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:00:13 [I 2022-10-28 01:54:49] ...trap/engines/engine.py.229: process: 0.56481 | load: 0.00245 [I 2022-10-28 01:54:49] ...trap/engines/engine.py.230: loss: 0.11846 [I 2022-10-28 01:54:49] ...optimizers/trijoint.py.151: total_norm: 22.485565 [I 2022-10-28 01:54:55] ...trap/engines/engine.py.224: train: epoch 99 | batch 2290/2304 [I 2022-10-28 01:54:55] ...trap/engines/engine.py.228: elapsed: 0:22:01 | left: 0:00:07 [I 2022-10-28 01:54:55] ...trap/engines/engine.py.229: process: 0.54540 | load: 0.00244 [I 2022-10-28 01:54:55] ...trap/engines/engine.py.230: loss: 0.11217 [I 2022-10-28 01:54:55] ...optimizers/trijoint.py.151: total_norm: 17.495665 [I 2022-10-28 01:55:01] ...trap/engines/engine.py.224: train: epoch 99 | batch 2300/2304 [I 2022-10-28 01:55:01] ...trap/engines/engine.py.228: elapsed: 0:22:07 | left: 0:00:02 [I 2022-10-28 01:55:01] ...trap/engines/engine.py.229: process: 0.54910 | load: 0.00016 [I 2022-10-28 01:55:01] ...trap/engines/engine.py.230: loss: 0.11178 [I 2022-10-28 01:55:01] ...optimizers/trijoint.py.151: total_norm: 15.061167 [I 2022-10-28 01:55:03] ...trap/engines/engine.py.224: train: epoch 99 | batch 2304/2304 [I 2022-10-28 01:55:03] ...trap/engines/engine.py.228: elapsed: 0:22:09 | left: 0:00:00 [I 2022-10-28 01:55:03] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00015 [I 2022-10-28 01:55:03] ...trap/engines/engine.py.230: loss: 0.15027 [I 2022-10-28 01:55:03] ...optimizers/trijoint.py.151: total_norm: 53.905903 [S 2022-10-28 01:55:04] ...trap/engines/engine.py.129: train_epoch.epoch: 99 [S 2022-10-28 01:55:04] ...trap/engines/engine.py.129: train_epoch.loss: 0.12626298815820067 [I 2022-10-28 01:55:04] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 01:55:06] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 99 [I 2022-10-28 01:55:06] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 01:55:08] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 01:55:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 01:55:08] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 01:55:11] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 01:55:11] ...trap/engines/engine.py.321: eval: epoch 99 | batch 0/510 [I 2022-10-28 01:55:11] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:47:46 [I 2022-10-28 01:55:11] ...trap/engines/engine.py.326: process: 5.62133 | load: 3.62013 [I 2022-10-28 01:55:15] ...trap/engines/engine.py.321: eval: epoch 99 | batch 10/510 [I 2022-10-28 01:55:15] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:30 [I 2022-10-28 01:55:15] ...trap/engines/engine.py.326: process: 0.32356 | load: 0.00018 [I 2022-10-28 01:55:18] ...trap/engines/engine.py.321: eval: epoch 99 | batch 20/510 [I 2022-10-28 01:55:18] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:15 [I 2022-10-28 01:55:18] ...trap/engines/engine.py.326: process: 0.36557 | load: 0.00019 [I 2022-10-28 01:55:21] ...trap/engines/engine.py.321: eval: epoch 99 | batch 30/510 [I 2022-10-28 01:55:21] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:41 [I 2022-10-28 01:55:21] ...trap/engines/engine.py.326: process: 0.32744 | load: 0.00020 [I 2022-10-28 01:55:25] ...trap/engines/engine.py.321: eval: epoch 99 | batch 40/510 [I 2022-10-28 01:55:25] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:35 [I 2022-10-28 01:55:25] ...trap/engines/engine.py.326: process: 0.33018 | load: 0.00017 [I 2022-10-28 01:55:28] ...trap/engines/engine.py.321: eval: epoch 99 | batch 50/510 [I 2022-10-28 01:55:28] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:32 [I 2022-10-28 01:55:28] ...trap/engines/engine.py.326: process: 0.32890 | load: 0.00015 [I 2022-10-28 01:55:31] ...trap/engines/engine.py.321: eval: epoch 99 | batch 60/510 [I 2022-10-28 01:55:31] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:28 [I 2022-10-28 01:55:31] ...trap/engines/engine.py.326: process: 0.33512 | load: 0.00017 [I 2022-10-28 01:55:35] ...trap/engines/engine.py.321: eval: epoch 99 | batch 70/510 [I 2022-10-28 01:55:35] ...trap/engines/engine.py.325: elapsed: 0:00:28 | left: 0:02:26 [I 2022-10-28 01:55:35] ...trap/engines/engine.py.326: process: 0.33615 | load: 0.00017 [I 2022-10-28 01:55:38] ...trap/engines/engine.py.321: eval: epoch 99 | batch 80/510 [I 2022-10-28 01:55:38] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:22 [I 2022-10-28 01:55:38] ...trap/engines/engine.py.326: process: 0.33344 | load: 0.00016 [I 2022-10-28 01:55:41] ...trap/engines/engine.py.321: eval: epoch 99 | batch 90/510 [I 2022-10-28 01:55:41] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-28 01:55:41] ...trap/engines/engine.py.326: process: 0.33371 | load: 0.00016 [I 2022-10-28 01:55:44] ...trap/engines/engine.py.321: eval: epoch 99 | batch 100/510 [I 2022-10-28 01:55:45] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:12 [I 2022-10-28 01:55:45] ...trap/engines/engine.py.326: process: 0.28392 | load: 0.00020 [I 2022-10-28 01:55:47] ...trap/engines/engine.py.321: eval: epoch 99 | batch 110/510 [I 2022-10-28 01:55:47] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-28 01:55:47] ...trap/engines/engine.py.326: process: 0.28517 | load: 0.00014 [I 2022-10-28 01:55:50] ...trap/engines/engine.py.321: eval: epoch 99 | batch 120/510 [I 2022-10-28 01:55:50] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:53 [I 2022-10-28 01:55:50] ...trap/engines/engine.py.326: process: 0.28309 | load: 0.00014 [I 2022-10-28 01:55:53] ...trap/engines/engine.py.321: eval: epoch 99 | batch 130/510 [I 2022-10-28 01:55:53] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-28 01:55:53] ...trap/engines/engine.py.326: process: 0.28456 | load: 0.00014 [I 2022-10-28 01:55:56] ...trap/engines/engine.py.321: eval: epoch 99 | batch 140/510 [I 2022-10-28 01:55:56] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-28 01:55:56] ...trap/engines/engine.py.326: process: 0.30386 | load: 0.00014 [I 2022-10-28 01:55:59] ...trap/engines/engine.py.321: eval: epoch 99 | batch 150/510 [I 2022-10-28 01:55:59] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:42 [I 2022-10-28 01:55:59] ...trap/engines/engine.py.326: process: 0.28732 | load: 0.00015 [I 2022-10-28 01:56:02] ...trap/engines/engine.py.321: eval: epoch 99 | batch 160/510 [I 2022-10-28 01:56:02] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-28 01:56:02] ...trap/engines/engine.py.326: process: 0.28402 | load: 0.00014 [I 2022-10-28 01:56:05] ...trap/engines/engine.py.321: eval: epoch 99 | batch 170/510 [I 2022-10-28 01:56:05] ...trap/engines/engine.py.325: elapsed: 0:00:58 | left: 0:01:37 [I 2022-10-28 01:56:05] ...trap/engines/engine.py.326: process: 0.28615 | load: 0.00014 [I 2022-10-28 01:56:08] ...trap/engines/engine.py.321: eval: epoch 99 | batch 180/510 [I 2022-10-28 01:56:08] ...trap/engines/engine.py.325: elapsed: 0:01:01 | left: 0:01:35 [I 2022-10-28 01:56:08] ...trap/engines/engine.py.326: process: 0.30196 | load: 0.00015 [I 2022-10-28 01:56:10] ...trap/engines/engine.py.321: eval: epoch 99 | batch 190/510 [I 2022-10-28 01:56:10] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:33 [I 2022-10-28 01:56:10] ...trap/engines/engine.py.326: process: 0.29200 | load: 0.00015 [I 2022-10-28 01:56:13] ...trap/engines/engine.py.321: eval: epoch 99 | batch 200/510 [I 2022-10-28 01:56:13] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-28 01:56:13] ...trap/engines/engine.py.326: process: 0.28022 | load: 0.00014 [I 2022-10-28 01:56:16] ...trap/engines/engine.py.321: eval: epoch 99 | batch 210/510 [I 2022-10-28 01:56:16] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-28 01:56:16] ...trap/engines/engine.py.326: process: 0.28821 | load: 0.00014 [I 2022-10-28 01:56:19] ...trap/engines/engine.py.321: eval: epoch 99 | batch 220/510 [I 2022-10-28 01:56:19] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:24 [I 2022-10-28 01:56:19] ...trap/engines/engine.py.326: process: 0.29849 | load: 0.00015 [I 2022-10-28 01:56:22] ...trap/engines/engine.py.321: eval: epoch 99 | batch 230/510 [I 2022-10-28 01:56:22] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-28 01:56:22] ...trap/engines/engine.py.326: process: 0.28748 | load: 0.00015 [I 2022-10-28 01:56:25] ...trap/engines/engine.py.321: eval: epoch 99 | batch 240/510 [I 2022-10-28 01:56:25] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 01:56:25] ...trap/engines/engine.py.326: process: 0.28292 | load: 0.00014 [I 2022-10-28 01:56:28] ...trap/engines/engine.py.321: eval: epoch 99 | batch 250/510 [I 2022-10-28 01:56:28] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:13 [I 2022-10-28 01:56:28] ...trap/engines/engine.py.326: process: 0.28169 | load: 0.00015 [I 2022-10-28 01:56:31] ...trap/engines/engine.py.321: eval: epoch 99 | batch 260/510 [I 2022-10-28 01:56:31] ...trap/engines/engine.py.325: elapsed: 0:01:24 | left: 0:01:11 [I 2022-10-28 01:56:31] ...trap/engines/engine.py.326: process: 0.28473 | load: 0.00015 [I 2022-10-28 01:56:33] ...trap/engines/engine.py.321: eval: epoch 99 | batch 270/510 [I 2022-10-28 01:56:33] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:08 [I 2022-10-28 01:56:33] ...trap/engines/engine.py.326: process: 0.28626 | load: 0.00016 [I 2022-10-28 01:56:36] ...trap/engines/engine.py.321: eval: epoch 99 | batch 280/510 [I 2022-10-28 01:56:36] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-28 01:56:36] ...trap/engines/engine.py.326: process: 0.29157 | load: 0.00014 [I 2022-10-28 01:56:39] ...trap/engines/engine.py.321: eval: epoch 99 | batch 290/510 [I 2022-10-28 01:56:39] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-28 01:56:39] ...trap/engines/engine.py.326: process: 0.28829 | load: 0.00015 [I 2022-10-28 01:56:42] ...trap/engines/engine.py.321: eval: epoch 99 | batch 300/510 [I 2022-10-28 01:56:42] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:00:59 [I 2022-10-28 01:56:42] ...trap/engines/engine.py.326: process: 0.28468 | load: 0.00015 [I 2022-10-28 01:56:45] ...trap/engines/engine.py.321: eval: epoch 99 | batch 310/510 [I 2022-10-28 01:56:45] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:58 [I 2022-10-28 01:56:45] ...trap/engines/engine.py.326: process: 0.28364 | load: 0.00015 [I 2022-10-28 01:56:48] ...trap/engines/engine.py.321: eval: epoch 99 | batch 320/510 [I 2022-10-28 01:56:48] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 01:56:48] ...trap/engines/engine.py.326: process: 0.28294 | load: 0.00016 [I 2022-10-28 01:56:51] ...trap/engines/engine.py.321: eval: epoch 99 | batch 330/510 [I 2022-10-28 01:56:51] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-28 01:56:51] ...trap/engines/engine.py.326: process: 0.28999 | load: 0.00016 [I 2022-10-28 01:56:54] ...trap/engines/engine.py.321: eval: epoch 99 | batch 340/510 [I 2022-10-28 01:56:54] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-28 01:56:54] ...trap/engines/engine.py.326: process: 0.28789 | load: 0.00014 [I 2022-10-28 01:56:57] ...trap/engines/engine.py.321: eval: epoch 99 | batch 350/510 [I 2022-10-28 01:56:57] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:45 [I 2022-10-28 01:56:57] ...trap/engines/engine.py.326: process: 0.28543 | load: 0.00015 [I 2022-10-28 01:56:59] ...trap/engines/engine.py.321: eval: epoch 99 | batch 360/510 [I 2022-10-28 01:56:59] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:42 [I 2022-10-28 01:56:59] ...trap/engines/engine.py.326: process: 0.28953 | load: 0.00015 [I 2022-10-28 01:57:02] ...trap/engines/engine.py.321: eval: epoch 99 | batch 370/510 [I 2022-10-28 01:57:02] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:39 [I 2022-10-28 01:57:02] ...trap/engines/engine.py.326: process: 0.28251 | load: 0.00015 [I 2022-10-28 01:57:05] ...trap/engines/engine.py.321: eval: epoch 99 | batch 380/510 [I 2022-10-28 01:57:05] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-28 01:57:05] ...trap/engines/engine.py.326: process: 0.29002 | load: 0.00014 [I 2022-10-28 01:57:08] ...trap/engines/engine.py.321: eval: epoch 99 | batch 390/510 [I 2022-10-28 01:57:08] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 01:57:08] ...trap/engines/engine.py.326: process: 0.28645 | load: 0.00016 [I 2022-10-28 01:57:11] ...trap/engines/engine.py.321: eval: epoch 99 | batch 400/510 [I 2022-10-28 01:57:11] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 01:57:11] ...trap/engines/engine.py.326: process: 0.29176 | load: 0.00022 [I 2022-10-28 01:57:14] ...trap/engines/engine.py.321: eval: epoch 99 | batch 410/510 [I 2022-10-28 01:57:14] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 01:57:14] ...trap/engines/engine.py.326: process: 0.29056 | load: 0.00015 [I 2022-10-28 01:57:17] ...trap/engines/engine.py.321: eval: epoch 99 | batch 420/510 [I 2022-10-28 01:57:17] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 01:57:17] ...trap/engines/engine.py.326: process: 0.29073 | load: 0.00016 [I 2022-10-28 01:57:20] ...trap/engines/engine.py.321: eval: epoch 99 | batch 430/510 [I 2022-10-28 01:57:20] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:22 [I 2022-10-28 01:57:20] ...trap/engines/engine.py.326: process: 0.28653 | load: 0.00015 [I 2022-10-28 01:57:22] ...trap/engines/engine.py.321: eval: epoch 99 | batch 440/510 [I 2022-10-28 01:57:22] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-28 01:57:22] ...trap/engines/engine.py.326: process: 0.28752 | load: 0.00015 [I 2022-10-28 01:57:25] ...trap/engines/engine.py.321: eval: epoch 99 | batch 450/510 [I 2022-10-28 01:57:25] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-28 01:57:25] ...trap/engines/engine.py.326: process: 0.28385 | load: 0.00015 [I 2022-10-28 01:57:28] ...trap/engines/engine.py.321: eval: epoch 99 | batch 460/510 [I 2022-10-28 01:57:28] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-28 01:57:28] ...trap/engines/engine.py.326: process: 0.28827 | load: 0.00015 [I 2022-10-28 01:57:31] ...trap/engines/engine.py.321: eval: epoch 99 | batch 470/510 [I 2022-10-28 01:57:31] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-28 01:57:31] ...trap/engines/engine.py.326: process: 0.29034 | load: 0.00016 [I 2022-10-28 01:57:34] ...trap/engines/engine.py.321: eval: epoch 99 | batch 480/510 [I 2022-10-28 01:57:34] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 01:57:34] ...trap/engines/engine.py.326: process: 0.27964 | load: 0.00016 [I 2022-10-28 01:57:37] ...trap/engines/engine.py.321: eval: epoch 99 | batch 490/510 [I 2022-10-28 01:57:37] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 01:57:37] ...trap/engines/engine.py.326: process: 0.28676 | load: 0.00014 [I 2022-10-28 01:57:40] ...trap/engines/engine.py.321: eval: epoch 99 | batch 500/510 [I 2022-10-28 01:57:40] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 01:57:40] ...trap/engines/engine.py.326: process: 0.28979 | load: 0.00014 [I 2022-10-28 01:57:43] ...trap/engines/engine.py.321: eval: epoch 99 | batch 510/510 [I 2022-10-28 01:57:43] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-28 01:57:43] ...trap/engines/engine.py.326: process: 0.28771 | load: 0.00010 [S 2022-10-28 01:57:43] ...trap/engines/engine.py.132: eval_epoch.epoch: 99 [I 2022-10-28 01:57:43] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.732000 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908300 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937800 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.724100 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907300 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936900 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.018852 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006783 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005879 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012589 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006116 [S 2022-10-28 01:57:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007713 [I 2022-10-28 01:57:49] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 01:57:49] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 01:57:50] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 01:57:50] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 01:57:50] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 01:57:54] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 01:57:55] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 01:58:01] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 01:58:01] ...trap/engines/engine.py.163: Training model on trainset for epoch 100 [I 2022-10-28 01:58:01] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 01:58:03] ...trap/engines/engine.py.224: train: epoch 100 | batch 0/2304 [I 2022-10-28 01:58:03] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:45:47 [I 2022-10-28 01:58:03] ...trap/engines/engine.py.229: process: 2.75509 | load: 2.13270 [I 2022-10-28 01:58:03] ...trap/engines/engine.py.230: loss: 0.14116 [I 2022-10-28 01:58:03] ...optimizers/trijoint.py.151: total_norm: 15.565174 [I 2022-10-28 01:58:09] ...trap/engines/engine.py.224: train: epoch 100 | batch 10/2304 [I 2022-10-28 01:58:09] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:18 [I 2022-10-28 01:58:09] ...trap/engines/engine.py.229: process: 0.55063 | load: 0.00397 [I 2022-10-28 01:58:09] ...trap/engines/engine.py.230: loss: 0.13528 [I 2022-10-28 01:58:09] ...optimizers/trijoint.py.151: total_norm: 14.931343 [I 2022-10-28 01:58:15] ...trap/engines/engine.py.224: train: epoch 100 | batch 20/2304 [I 2022-10-28 01:58:15] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:55 [I 2022-10-28 01:58:15] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00366 [I 2022-10-28 01:58:15] ...trap/engines/engine.py.230: loss: 0.13851 [I 2022-10-28 01:58:15] ...optimizers/trijoint.py.151: total_norm: 41.585258 [I 2022-10-28 01:58:20] ...trap/engines/engine.py.224: train: epoch 100 | batch 30/2304 [I 2022-10-28 01:58:20] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:22 [I 2022-10-28 01:58:20] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00326 [I 2022-10-28 01:58:20] ...trap/engines/engine.py.230: loss: 0.09231 [I 2022-10-28 01:58:20] ...optimizers/trijoint.py.151: total_norm: 15.576463 [I 2022-10-28 01:58:26] ...trap/engines/engine.py.224: train: epoch 100 | batch 40/2304 [I 2022-10-28 01:58:26] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:06 [I 2022-10-28 01:58:26] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00337 [I 2022-10-28 01:58:26] ...trap/engines/engine.py.230: loss: 0.09017 [I 2022-10-28 01:58:26] ...optimizers/trijoint.py.151: total_norm: 13.499043 [I 2022-10-28 01:58:32] ...trap/engines/engine.py.224: train: epoch 100 | batch 50/2304 [I 2022-10-28 01:58:32] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:47 [I 2022-10-28 01:58:32] ...trap/engines/engine.py.229: process: 0.55571 | load: 0.00298 [I 2022-10-28 01:58:32] ...trap/engines/engine.py.230: loss: 0.16492 [I 2022-10-28 01:58:32] ...optimizers/trijoint.py.151: total_norm: 21.040617 [I 2022-10-28 01:58:37] ...trap/engines/engine.py.224: train: epoch 100 | batch 60/2304 [I 2022-10-28 01:58:37] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:54 [I 2022-10-28 01:58:37] ...trap/engines/engine.py.229: process: 0.56387 | load: 0.00311 [I 2022-10-28 01:58:37] ...trap/engines/engine.py.230: loss: 0.14309 [I 2022-10-28 01:58:37] ...optimizers/trijoint.py.151: total_norm: 12.059421 [I 2022-10-28 01:58:43] ...trap/engines/engine.py.224: train: epoch 100 | batch 70/2304 [I 2022-10-28 01:58:43] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:53 [I 2022-10-28 01:58:43] ...trap/engines/engine.py.229: process: 0.55834 | load: 0.00268 [I 2022-10-28 01:58:43] ...trap/engines/engine.py.230: loss: 0.09186 [I 2022-10-28 01:58:43] ...optimizers/trijoint.py.151: total_norm: 8.845670 [I 2022-10-28 01:58:48] ...trap/engines/engine.py.224: train: epoch 100 | batch 80/2304 [I 2022-10-28 01:58:48] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:33 [I 2022-10-28 01:58:48] ...trap/engines/engine.py.229: process: 0.55184 | load: 0.00260 [I 2022-10-28 01:58:48] ...trap/engines/engine.py.230: loss: 0.12644 [I 2022-10-28 01:58:48] ...optimizers/trijoint.py.151: total_norm: 14.029521 [I 2022-10-28 01:58:54] ...trap/engines/engine.py.224: train: epoch 100 | batch 90/2304 [I 2022-10-28 01:58:54] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:35 [I 2022-10-28 01:58:54] ...trap/engines/engine.py.229: process: 0.54804 | load: 0.00270 [I 2022-10-28 01:58:54] ...trap/engines/engine.py.230: loss: 0.12738 [I 2022-10-28 01:58:54] ...optimizers/trijoint.py.151: total_norm: 12.677237 [I 2022-10-28 01:59:00] ...trap/engines/engine.py.224: train: epoch 100 | batch 100/2304 [I 2022-10-28 01:59:00] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:40 [I 2022-10-28 01:59:00] ...trap/engines/engine.py.229: process: 0.57723 | load: 0.00268 [I 2022-10-28 01:59:00] ...trap/engines/engine.py.230: loss: 0.08724 [I 2022-10-28 01:59:00] ...optimizers/trijoint.py.151: total_norm: 9.799156 [I 2022-10-28 01:59:05] ...trap/engines/engine.py.224: train: epoch 100 | batch 110/2304 [I 2022-10-28 01:59:05] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:29 [I 2022-10-28 01:59:05] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00269 [I 2022-10-28 01:59:05] ...trap/engines/engine.py.230: loss: 0.12347 [I 2022-10-28 01:59:05] ...optimizers/trijoint.py.151: total_norm: 16.305286 [I 2022-10-28 01:59:11] ...trap/engines/engine.py.224: train: epoch 100 | batch 120/2304 [I 2022-10-28 01:59:11] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:30 [I 2022-10-28 01:59:11] ...trap/engines/engine.py.229: process: 0.55197 | load: 0.00305 [I 2022-10-28 01:59:11] ...trap/engines/engine.py.230: loss: 0.13855 [I 2022-10-28 01:59:11] ...optimizers/trijoint.py.151: total_norm: 15.548690 [I 2022-10-28 01:59:17] ...trap/engines/engine.py.224: train: epoch 100 | batch 130/2304 [I 2022-10-28 01:59:17] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:10 [I 2022-10-28 01:59:17] ...trap/engines/engine.py.229: process: 0.55145 | load: 0.00259 [I 2022-10-28 01:59:17] ...trap/engines/engine.py.230: loss: 0.15641 [I 2022-10-28 01:59:17] ...optimizers/trijoint.py.151: total_norm: 75.521660 [I 2022-10-28 01:59:22] ...trap/engines/engine.py.224: train: epoch 100 | batch 140/2304 [I 2022-10-28 01:59:22] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:12 [I 2022-10-28 01:59:22] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00247 [I 2022-10-28 01:59:22] ...trap/engines/engine.py.230: loss: 0.08795 [I 2022-10-28 01:59:22] ...optimizers/trijoint.py.151: total_norm: 16.451891 [I 2022-10-28 01:59:28] ...trap/engines/engine.py.224: train: epoch 100 | batch 150/2304 [I 2022-10-28 01:59:28] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:19:59 [I 2022-10-28 01:59:28] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00251 [I 2022-10-28 01:59:28] ...trap/engines/engine.py.230: loss: 0.09976 [I 2022-10-28 01:59:28] ...optimizers/trijoint.py.151: total_norm: 11.292812 [I 2022-10-28 01:59:33] ...trap/engines/engine.py.224: train: epoch 100 | batch 160/2304 [I 2022-10-28 01:59:33] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:50 [I 2022-10-28 01:59:33] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00243 [I 2022-10-28 01:59:33] ...trap/engines/engine.py.230: loss: 0.11595 [I 2022-10-28 01:59:33] ...optimizers/trijoint.py.151: total_norm: 9.452231 [I 2022-10-28 01:59:39] ...trap/engines/engine.py.224: train: epoch 100 | batch 170/2304 [I 2022-10-28 01:59:39] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:20:01 [I 2022-10-28 01:59:39] ...trap/engines/engine.py.229: process: 0.56260 | load: 0.00273 [I 2022-10-28 01:59:39] ...trap/engines/engine.py.230: loss: 0.12840 [I 2022-10-28 01:59:39] ...optimizers/trijoint.py.151: total_norm: 21.605850 [I 2022-10-28 01:59:45] ...trap/engines/engine.py.224: train: epoch 100 | batch 180/2304 [I 2022-10-28 01:59:45] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:56 [I 2022-10-28 01:59:45] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00267 [I 2022-10-28 01:59:45] ...trap/engines/engine.py.230: loss: 0.09429 [I 2022-10-28 01:59:45] ...optimizers/trijoint.py.151: total_norm: 17.576395 [I 2022-10-28 01:59:50] ...trap/engines/engine.py.224: train: epoch 100 | batch 190/2304 [I 2022-10-28 01:59:50] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:50 [I 2022-10-28 01:59:50] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00259 [I 2022-10-28 01:59:50] ...trap/engines/engine.py.230: loss: 0.17855 [I 2022-10-28 01:59:50] ...optimizers/trijoint.py.151: total_norm: 23.994946 [I 2022-10-28 01:59:56] ...trap/engines/engine.py.224: train: epoch 100 | batch 200/2304 [I 2022-10-28 01:59:56] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:36 [I 2022-10-28 01:59:56] ...trap/engines/engine.py.229: process: 0.54679 | load: 0.00259 [I 2022-10-28 01:59:56] ...trap/engines/engine.py.230: loss: 0.09906 [I 2022-10-28 01:59:56] ...optimizers/trijoint.py.151: total_norm: 13.220338 [I 2022-10-28 02:00:01] ...trap/engines/engine.py.224: train: epoch 100 | batch 210/2304 [I 2022-10-28 02:00:01] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:28 [I 2022-10-28 02:00:01] ...trap/engines/engine.py.229: process: 0.56780 | load: 0.00279 [I 2022-10-28 02:00:01] ...trap/engines/engine.py.230: loss: 0.11478 [I 2022-10-28 02:00:01] ...optimizers/trijoint.py.151: total_norm: 14.985647 [I 2022-10-28 02:00:07] ...trap/engines/engine.py.224: train: epoch 100 | batch 220/2304 [I 2022-10-28 02:00:07] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:32 [I 2022-10-28 02:00:07] ...trap/engines/engine.py.229: process: 0.56732 | load: 0.00248 [I 2022-10-28 02:00:07] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-28 02:00:07] ...optimizers/trijoint.py.151: total_norm: 24.698677 [I 2022-10-28 02:00:13] ...trap/engines/engine.py.224: train: epoch 100 | batch 230/2304 [I 2022-10-28 02:00:13] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:21 [I 2022-10-28 02:00:13] ...trap/engines/engine.py.229: process: 0.56636 | load: 0.00246 [I 2022-10-28 02:00:13] ...trap/engines/engine.py.230: loss: 0.11017 [I 2022-10-28 02:00:13] ...optimizers/trijoint.py.151: total_norm: 15.319818 [I 2022-10-28 02:00:18] ...trap/engines/engine.py.224: train: epoch 100 | batch 240/2304 [I 2022-10-28 02:00:18] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:27 [I 2022-10-28 02:00:18] ...trap/engines/engine.py.229: process: 0.56691 | load: 0.00254 [I 2022-10-28 02:00:18] ...trap/engines/engine.py.230: loss: 0.10840 [I 2022-10-28 02:00:18] ...optimizers/trijoint.py.151: total_norm: 10.147420 [I 2022-10-28 02:00:24] ...trap/engines/engine.py.224: train: epoch 100 | batch 250/2304 [I 2022-10-28 02:00:24] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:12 [I 2022-10-28 02:00:24] ...trap/engines/engine.py.229: process: 0.56293 | load: 0.00257 [I 2022-10-28 02:00:24] ...trap/engines/engine.py.230: loss: 0.08481 [I 2022-10-28 02:00:24] ...optimizers/trijoint.py.151: total_norm: 15.830317 [I 2022-10-28 02:00:30] ...trap/engines/engine.py.224: train: epoch 100 | batch 260/2304 [I 2022-10-28 02:00:30] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:01 [I 2022-10-28 02:00:30] ...trap/engines/engine.py.229: process: 0.54968 | load: 0.00261 [I 2022-10-28 02:00:30] ...trap/engines/engine.py.230: loss: 0.15182 [I 2022-10-28 02:00:30] ...optimizers/trijoint.py.151: total_norm: 17.575832 [I 2022-10-28 02:00:35] ...trap/engines/engine.py.224: train: epoch 100 | batch 270/2304 [I 2022-10-28 02:00:35] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:50 [I 2022-10-28 02:00:35] ...trap/engines/engine.py.229: process: 0.55213 | load: 0.00256 [I 2022-10-28 02:00:35] ...trap/engines/engine.py.230: loss: 0.08617 [I 2022-10-28 02:00:35] ...optimizers/trijoint.py.151: total_norm: 9.606621 [I 2022-10-28 02:00:41] ...trap/engines/engine.py.224: train: epoch 100 | batch 280/2304 [I 2022-10-28 02:00:41] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:18:51 [I 2022-10-28 02:00:41] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00250 [I 2022-10-28 02:00:41] ...trap/engines/engine.py.230: loss: 0.12871 [I 2022-10-28 02:00:41] ...optimizers/trijoint.py.151: total_norm: 15.013915 [I 2022-10-28 02:00:46] ...trap/engines/engine.py.224: train: epoch 100 | batch 290/2304 [I 2022-10-28 02:00:46] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:47 [I 2022-10-28 02:00:46] ...trap/engines/engine.py.229: process: 0.56845 | load: 0.00245 [I 2022-10-28 02:00:46] ...trap/engines/engine.py.230: loss: 0.15272 [I 2022-10-28 02:00:46] ...optimizers/trijoint.py.151: total_norm: 30.818064 [I 2022-10-28 02:00:52] ...trap/engines/engine.py.224: train: epoch 100 | batch 300/2304 [I 2022-10-28 02:00:52] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:47 [I 2022-10-28 02:00:52] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00300 [I 2022-10-28 02:00:52] ...trap/engines/engine.py.230: loss: 0.14686 [I 2022-10-28 02:00:52] ...optimizers/trijoint.py.151: total_norm: 20.341648 [I 2022-10-28 02:00:58] ...trap/engines/engine.py.224: train: epoch 100 | batch 310/2304 [I 2022-10-28 02:00:58] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:34 [I 2022-10-28 02:00:58] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00263 [I 2022-10-28 02:00:58] ...trap/engines/engine.py.230: loss: 0.10677 [I 2022-10-28 02:00:58] ...optimizers/trijoint.py.151: total_norm: 21.507763 [I 2022-10-28 02:01:03] ...trap/engines/engine.py.224: train: epoch 100 | batch 320/2304 [I 2022-10-28 02:01:03] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:23 [I 2022-10-28 02:01:03] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00296 [I 2022-10-28 02:01:03] ...trap/engines/engine.py.230: loss: 0.12177 [I 2022-10-28 02:01:03] ...optimizers/trijoint.py.151: total_norm: 13.518111 [I 2022-10-28 02:01:09] ...trap/engines/engine.py.224: train: epoch 100 | batch 330/2304 [I 2022-10-28 02:01:09] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:18:27 [I 2022-10-28 02:01:09] ...trap/engines/engine.py.229: process: 0.56851 | load: 0.00253 [I 2022-10-28 02:01:09] ...trap/engines/engine.py.230: loss: 0.13265 [I 2022-10-28 02:01:09] ...optimizers/trijoint.py.151: total_norm: 10.750853 [I 2022-10-28 02:01:14] ...trap/engines/engine.py.224: train: epoch 100 | batch 340/2304 [I 2022-10-28 02:01:14] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:16 [I 2022-10-28 02:01:14] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00254 [I 2022-10-28 02:01:14] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-28 02:01:14] ...optimizers/trijoint.py.151: total_norm: 13.987946 [I 2022-10-28 02:01:20] ...trap/engines/engine.py.224: train: epoch 100 | batch 350/2304 [I 2022-10-28 02:01:20] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:13 [I 2022-10-28 02:01:20] ...trap/engines/engine.py.229: process: 0.56898 | load: 0.00267 [I 2022-10-28 02:01:20] ...trap/engines/engine.py.230: loss: 0.12548 [I 2022-10-28 02:01:20] ...optimizers/trijoint.py.151: total_norm: 11.959469 [I 2022-10-28 02:01:26] ...trap/engines/engine.py.224: train: epoch 100 | batch 360/2304 [I 2022-10-28 02:01:26] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:18:12 [I 2022-10-28 02:01:26] ...trap/engines/engine.py.229: process: 0.53790 | load: 0.00270 [I 2022-10-28 02:01:26] ...trap/engines/engine.py.230: loss: 0.13052 [I 2022-10-28 02:01:26] ...optimizers/trijoint.py.151: total_norm: 36.574184 [I 2022-10-28 02:01:31] ...trap/engines/engine.py.224: train: epoch 100 | batch 370/2304 [I 2022-10-28 02:01:31] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:17:59 [I 2022-10-28 02:01:31] ...trap/engines/engine.py.229: process: 0.55471 | load: 0.00247 [I 2022-10-28 02:01:31] ...trap/engines/engine.py.230: loss: 0.11708 [I 2022-10-28 02:01:31] ...optimizers/trijoint.py.151: total_norm: 25.912611 [I 2022-10-28 02:01:37] ...trap/engines/engine.py.224: train: epoch 100 | batch 380/2304 [I 2022-10-28 02:01:37] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:18:03 [I 2022-10-28 02:01:37] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00246 [I 2022-10-28 02:01:37] ...trap/engines/engine.py.230: loss: 0.12757 [I 2022-10-28 02:01:37] ...optimizers/trijoint.py.151: total_norm: 29.737795 [I 2022-10-28 02:01:43] ...trap/engines/engine.py.224: train: epoch 100 | batch 390/2304 [I 2022-10-28 02:01:43] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:55 [I 2022-10-28 02:01:43] ...trap/engines/engine.py.229: process: 0.56042 | load: 0.00247 [I 2022-10-28 02:01:43] ...trap/engines/engine.py.230: loss: 0.11774 [I 2022-10-28 02:01:43] ...optimizers/trijoint.py.151: total_norm: 19.791992 [I 2022-10-28 02:01:48] ...trap/engines/engine.py.224: train: epoch 100 | batch 400/2304 [I 2022-10-28 02:01:48] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:33 [I 2022-10-28 02:01:48] ...trap/engines/engine.py.229: process: 0.56691 | load: 0.00241 [I 2022-10-28 02:01:48] ...trap/engines/engine.py.230: loss: 0.08494 [I 2022-10-28 02:01:48] ...optimizers/trijoint.py.151: total_norm: 14.236358 [I 2022-10-28 02:01:54] ...trap/engines/engine.py.224: train: epoch 100 | batch 410/2304 [I 2022-10-28 02:01:54] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:17:39 [I 2022-10-28 02:01:54] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00248 [I 2022-10-28 02:01:54] ...trap/engines/engine.py.230: loss: 0.17418 [I 2022-10-28 02:01:54] ...optimizers/trijoint.py.151: total_norm: 8.690726 [I 2022-10-28 02:01:59] ...trap/engines/engine.py.224: train: epoch 100 | batch 420/2304 [I 2022-10-28 02:01:59] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:24 [I 2022-10-28 02:01:59] ...trap/engines/engine.py.229: process: 0.54705 | load: 0.00250 [I 2022-10-28 02:01:59] ...trap/engines/engine.py.230: loss: 0.11249 [I 2022-10-28 02:01:59] ...optimizers/trijoint.py.151: total_norm: 8.579460 [I 2022-10-28 02:02:05] ...trap/engines/engine.py.224: train: epoch 100 | batch 430/2304 [I 2022-10-28 02:02:05] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:30 [I 2022-10-28 02:02:05] ...trap/engines/engine.py.229: process: 0.56998 | load: 0.00274 [I 2022-10-28 02:02:05] ...trap/engines/engine.py.230: loss: 0.15669 [I 2022-10-28 02:02:05] ...optimizers/trijoint.py.151: total_norm: 16.739597 [I 2022-10-28 02:02:11] ...trap/engines/engine.py.224: train: epoch 100 | batch 440/2304 [I 2022-10-28 02:02:11] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:13 [I 2022-10-28 02:02:11] ...trap/engines/engine.py.229: process: 0.54375 | load: 0.00246 [I 2022-10-28 02:02:11] ...trap/engines/engine.py.230: loss: 0.10106 [I 2022-10-28 02:02:11] ...optimizers/trijoint.py.151: total_norm: 10.573080 [I 2022-10-28 02:02:16] ...trap/engines/engine.py.224: train: epoch 100 | batch 450/2304 [I 2022-10-28 02:02:16] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:09 [I 2022-10-28 02:02:16] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00251 [I 2022-10-28 02:02:16] ...trap/engines/engine.py.230: loss: 0.08994 [I 2022-10-28 02:02:16] ...optimizers/trijoint.py.151: total_norm: 12.495028 [I 2022-10-28 02:02:22] ...trap/engines/engine.py.224: train: epoch 100 | batch 460/2304 [I 2022-10-28 02:02:22] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:17:14 [I 2022-10-28 02:02:22] ...trap/engines/engine.py.229: process: 0.56629 | load: 0.00245 [I 2022-10-28 02:02:22] ...trap/engines/engine.py.230: loss: 0.08605 [I 2022-10-28 02:02:22] ...optimizers/trijoint.py.151: total_norm: 10.425422 [I 2022-10-28 02:02:27] ...trap/engines/engine.py.224: train: epoch 100 | batch 470/2304 [I 2022-10-28 02:02:27] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:03 [I 2022-10-28 02:02:27] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00244 [I 2022-10-28 02:02:27] ...trap/engines/engine.py.230: loss: 0.09447 [I 2022-10-28 02:02:27] ...optimizers/trijoint.py.151: total_norm: 10.045842 [I 2022-10-28 02:02:33] ...trap/engines/engine.py.224: train: epoch 100 | batch 480/2304 [I 2022-10-28 02:02:33] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:16:53 [I 2022-10-28 02:02:33] ...trap/engines/engine.py.229: process: 0.54462 | load: 0.00243 [I 2022-10-28 02:02:33] ...trap/engines/engine.py.230: loss: 0.17524 [I 2022-10-28 02:02:33] ...optimizers/trijoint.py.151: total_norm: 15.036271 [I 2022-10-28 02:02:39] ...trap/engines/engine.py.224: train: epoch 100 | batch 490/2304 [I 2022-10-28 02:02:39] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:57 [I 2022-10-28 02:02:39] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00248 [I 2022-10-28 02:02:39] ...trap/engines/engine.py.230: loss: 0.23977 [I 2022-10-28 02:02:39] ...optimizers/trijoint.py.151: total_norm: 25.102472 [I 2022-10-28 02:02:44] ...trap/engines/engine.py.224: train: epoch 100 | batch 500/2304 [I 2022-10-28 02:02:44] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:56 [I 2022-10-28 02:02:44] ...trap/engines/engine.py.229: process: 0.56731 | load: 0.00251 [I 2022-10-28 02:02:44] ...trap/engines/engine.py.230: loss: 0.11988 [I 2022-10-28 02:02:44] ...optimizers/trijoint.py.151: total_norm: 11.896765 [I 2022-10-28 02:02:50] ...trap/engines/engine.py.224: train: epoch 100 | batch 510/2304 [I 2022-10-28 02:02:50] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:49 [I 2022-10-28 02:02:50] ...trap/engines/engine.py.229: process: 0.56746 | load: 0.00314 [I 2022-10-28 02:02:50] ...trap/engines/engine.py.230: loss: 0.12693 [I 2022-10-28 02:02:50] ...optimizers/trijoint.py.151: total_norm: 16.603514 [I 2022-10-28 02:02:55] ...trap/engines/engine.py.224: train: epoch 100 | batch 520/2304 [I 2022-10-28 02:02:55] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:37 [I 2022-10-28 02:02:55] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00245 [I 2022-10-28 02:02:55] ...trap/engines/engine.py.230: loss: 0.10788 [I 2022-10-28 02:02:55] ...optimizers/trijoint.py.151: total_norm: 17.122961 [I 2022-10-28 02:03:01] ...trap/engines/engine.py.224: train: epoch 100 | batch 530/2304 [I 2022-10-28 02:03:01] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:29 [I 2022-10-28 02:03:01] ...trap/engines/engine.py.229: process: 0.55271 | load: 0.00243 [I 2022-10-28 02:03:01] ...trap/engines/engine.py.230: loss: 0.09701 [I 2022-10-28 02:03:01] ...optimizers/trijoint.py.151: total_norm: 9.744984 [I 2022-10-28 02:03:07] ...trap/engines/engine.py.224: train: epoch 100 | batch 540/2304 [I 2022-10-28 02:03:07] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:26 [I 2022-10-28 02:03:07] ...trap/engines/engine.py.229: process: 0.55378 | load: 0.00239 [I 2022-10-28 02:03:07] ...trap/engines/engine.py.230: loss: 0.16568 [I 2022-10-28 02:03:07] ...optimizers/trijoint.py.151: total_norm: 26.747982 [I 2022-10-28 02:03:12] ...trap/engines/engine.py.224: train: epoch 100 | batch 550/2304 [I 2022-10-28 02:03:12] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:11 [I 2022-10-28 02:03:12] ...trap/engines/engine.py.229: process: 0.54068 | load: 0.00243 [I 2022-10-28 02:03:12] ...trap/engines/engine.py.230: loss: 0.11519 [I 2022-10-28 02:03:12] ...optimizers/trijoint.py.151: total_norm: 38.756329 [I 2022-10-28 02:03:18] ...trap/engines/engine.py.224: train: epoch 100 | batch 560/2304 [I 2022-10-28 02:03:18] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:14 [I 2022-10-28 02:03:18] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00246 [I 2022-10-28 02:03:18] ...trap/engines/engine.py.230: loss: 0.15376 [I 2022-10-28 02:03:18] ...optimizers/trijoint.py.151: total_norm: 15.898921 [I 2022-10-28 02:03:23] ...trap/engines/engine.py.224: train: epoch 100 | batch 570/2304 [I 2022-10-28 02:03:23] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:07 [I 2022-10-28 02:03:23] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00245 [I 2022-10-28 02:03:23] ...trap/engines/engine.py.230: loss: 0.08748 [I 2022-10-28 02:03:23] ...optimizers/trijoint.py.151: total_norm: 13.104047 [I 2022-10-28 02:03:29] ...trap/engines/engine.py.224: train: epoch 100 | batch 580/2304 [I 2022-10-28 02:03:29] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:15:59 [I 2022-10-28 02:03:29] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00241 [I 2022-10-28 02:03:29] ...trap/engines/engine.py.230: loss: 0.10221 [I 2022-10-28 02:03:29] ...optimizers/trijoint.py.151: total_norm: 15.426033 [I 2022-10-28 02:03:35] ...trap/engines/engine.py.224: train: epoch 100 | batch 590/2304 [I 2022-10-28 02:03:35] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:01 [I 2022-10-28 02:03:35] ...trap/engines/engine.py.229: process: 0.56776 | load: 0.00253 [I 2022-10-28 02:03:35] ...trap/engines/engine.py.230: loss: 0.15492 [I 2022-10-28 02:03:35] ...optimizers/trijoint.py.151: total_norm: 12.369876 [I 2022-10-28 02:03:40] ...trap/engines/engine.py.224: train: epoch 100 | batch 600/2304 [I 2022-10-28 02:03:40] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:55 [I 2022-10-28 02:03:40] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00245 [I 2022-10-28 02:03:40] ...trap/engines/engine.py.230: loss: 0.13656 [I 2022-10-28 02:03:40] ...optimizers/trijoint.py.151: total_norm: 45.591545 [I 2022-10-28 02:03:46] ...trap/engines/engine.py.224: train: epoch 100 | batch 610/2304 [I 2022-10-28 02:03:46] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:15:50 [I 2022-10-28 02:03:46] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00244 [I 2022-10-28 02:03:46] ...trap/engines/engine.py.230: loss: 0.12216 [I 2022-10-28 02:03:46] ...optimizers/trijoint.py.151: total_norm: 11.360596 [I 2022-10-28 02:03:51] ...trap/engines/engine.py.224: train: epoch 100 | batch 620/2304 [I 2022-10-28 02:03:51] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:47 [I 2022-10-28 02:03:51] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00250 [I 2022-10-28 02:03:51] ...trap/engines/engine.py.230: loss: 0.15661 [I 2022-10-28 02:03:51] ...optimizers/trijoint.py.151: total_norm: 29.726845 [I 2022-10-28 02:03:57] ...trap/engines/engine.py.224: train: epoch 100 | batch 630/2304 [I 2022-10-28 02:03:57] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:32 [I 2022-10-28 02:03:57] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00245 [I 2022-10-28 02:03:57] ...trap/engines/engine.py.230: loss: 0.11184 [I 2022-10-28 02:03:57] ...optimizers/trijoint.py.151: total_norm: 17.917915 [I 2022-10-28 02:04:03] ...trap/engines/engine.py.224: train: epoch 100 | batch 640/2304 [I 2022-10-28 02:04:03] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:25 [I 2022-10-28 02:04:03] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00243 [I 2022-10-28 02:04:03] ...trap/engines/engine.py.230: loss: 0.18857 [I 2022-10-28 02:04:03] ...optimizers/trijoint.py.151: total_norm: 10.360619 [I 2022-10-28 02:04:08] ...trap/engines/engine.py.224: train: epoch 100 | batch 650/2304 [I 2022-10-28 02:04:08] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:16 [I 2022-10-28 02:04:08] ...trap/engines/engine.py.229: process: 0.54328 | load: 0.00251 [I 2022-10-28 02:04:08] ...trap/engines/engine.py.230: loss: 0.11513 [I 2022-10-28 02:04:08] ...optimizers/trijoint.py.151: total_norm: 10.081530 [I 2022-10-28 02:04:14] ...trap/engines/engine.py.224: train: epoch 100 | batch 660/2304 [I 2022-10-28 02:04:14] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:15:25 [I 2022-10-28 02:04:14] ...trap/engines/engine.py.229: process: 0.57780 | load: 0.00246 [I 2022-10-28 02:04:14] ...trap/engines/engine.py.230: loss: 0.13867 [I 2022-10-28 02:04:14] ...optimizers/trijoint.py.151: total_norm: 12.547283 [I 2022-10-28 02:04:19] ...trap/engines/engine.py.224: train: epoch 100 | batch 670/2304 [I 2022-10-28 02:04:19] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:08 [I 2022-10-28 02:04:19] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00248 [I 2022-10-28 02:04:19] ...trap/engines/engine.py.230: loss: 0.16804 [I 2022-10-28 02:04:19] ...optimizers/trijoint.py.151: total_norm: 16.307665 [I 2022-10-28 02:04:25] ...trap/engines/engine.py.224: train: epoch 100 | batch 680/2304 [I 2022-10-28 02:04:25] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:08 [I 2022-10-28 02:04:25] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00259 [I 2022-10-28 02:04:25] ...trap/engines/engine.py.230: loss: 0.10290 [I 2022-10-28 02:04:25] ...optimizers/trijoint.py.151: total_norm: 11.287889 [I 2022-10-28 02:04:31] ...trap/engines/engine.py.224: train: epoch 100 | batch 690/2304 [I 2022-10-28 02:04:31] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:06 [I 2022-10-28 02:04:31] ...trap/engines/engine.py.229: process: 0.56109 | load: 0.00244 [I 2022-10-28 02:04:31] ...trap/engines/engine.py.230: loss: 0.10663 [I 2022-10-28 02:04:31] ...optimizers/trijoint.py.151: total_norm: 7.998266 [I 2022-10-28 02:04:36] ...trap/engines/engine.py.224: train: epoch 100 | batch 700/2304 [I 2022-10-28 02:04:36] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:15:01 [I 2022-10-28 02:04:36] ...trap/engines/engine.py.229: process: 0.55587 | load: 0.00243 [I 2022-10-28 02:04:36] ...trap/engines/engine.py.230: loss: 0.14558 [I 2022-10-28 02:04:36] ...optimizers/trijoint.py.151: total_norm: 12.468062 [I 2022-10-28 02:04:42] ...trap/engines/engine.py.224: train: epoch 100 | batch 710/2304 [I 2022-10-28 02:04:42] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:14:56 [I 2022-10-28 02:04:42] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00243 [I 2022-10-28 02:04:42] ...trap/engines/engine.py.230: loss: 0.17540 [I 2022-10-28 02:04:42] ...optimizers/trijoint.py.151: total_norm: 24.474442 [I 2022-10-28 02:04:47] ...trap/engines/engine.py.224: train: epoch 100 | batch 720/2304 [I 2022-10-28 02:04:47] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:51 [I 2022-10-28 02:04:47] ...trap/engines/engine.py.229: process: 0.56672 | load: 0.00241 [I 2022-10-28 02:04:47] ...trap/engines/engine.py.230: loss: 0.12088 [I 2022-10-28 02:04:47] ...optimizers/trijoint.py.151: total_norm: 19.091812 [I 2022-10-28 02:04:53] ...trap/engines/engine.py.224: train: epoch 100 | batch 730/2304 [I 2022-10-28 02:04:53] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:14:43 [I 2022-10-28 02:04:53] ...trap/engines/engine.py.229: process: 0.56519 | load: 0.00245 [I 2022-10-28 02:04:53] ...trap/engines/engine.py.230: loss: 0.09097 [I 2022-10-28 02:04:53] ...optimizers/trijoint.py.151: total_norm: 16.174326 [I 2022-10-28 02:04:59] ...trap/engines/engine.py.224: train: epoch 100 | batch 740/2304 [I 2022-10-28 02:04:59] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:14:42 [I 2022-10-28 02:04:59] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00243 [I 2022-10-28 02:04:59] ...trap/engines/engine.py.230: loss: 0.09980 [I 2022-10-28 02:04:59] ...optimizers/trijoint.py.151: total_norm: 15.089617 [I 2022-10-28 02:05:04] ...trap/engines/engine.py.224: train: epoch 100 | batch 750/2304 [I 2022-10-28 02:05:04] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:25 [I 2022-10-28 02:05:04] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00257 [I 2022-10-28 02:05:04] ...trap/engines/engine.py.230: loss: 0.11063 [I 2022-10-28 02:05:04] ...optimizers/trijoint.py.151: total_norm: 23.019863 [I 2022-10-28 02:05:10] ...trap/engines/engine.py.224: train: epoch 100 | batch 760/2304 [I 2022-10-28 02:05:10] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:23 [I 2022-10-28 02:05:10] ...trap/engines/engine.py.229: process: 0.57182 | load: 0.00251 [I 2022-10-28 02:05:10] ...trap/engines/engine.py.230: loss: 0.13657 [I 2022-10-28 02:05:10] ...optimizers/trijoint.py.151: total_norm: 16.528942 [I 2022-10-28 02:05:16] ...trap/engines/engine.py.224: train: epoch 100 | batch 770/2304 [I 2022-10-28 02:05:16] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:20 [I 2022-10-28 02:05:16] ...trap/engines/engine.py.229: process: 0.56350 | load: 0.00250 [I 2022-10-28 02:05:16] ...trap/engines/engine.py.230: loss: 0.17390 [I 2022-10-28 02:05:16] ...optimizers/trijoint.py.151: total_norm: 21.869041 [I 2022-10-28 02:05:21] ...trap/engines/engine.py.224: train: epoch 100 | batch 780/2304 [I 2022-10-28 02:05:21] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:08 [I 2022-10-28 02:05:21] ...trap/engines/engine.py.229: process: 0.54724 | load: 0.00247 [I 2022-10-28 02:05:21] ...trap/engines/engine.py.230: loss: 0.07964 [I 2022-10-28 02:05:21] ...optimizers/trijoint.py.151: total_norm: 15.120574 [I 2022-10-28 02:05:27] ...trap/engines/engine.py.224: train: epoch 100 | batch 790/2304 [I 2022-10-28 02:05:27] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:14:11 [I 2022-10-28 02:05:27] ...trap/engines/engine.py.229: process: 0.57804 | load: 0.00241 [I 2022-10-28 02:05:27] ...trap/engines/engine.py.230: loss: 0.13791 [I 2022-10-28 02:05:27] ...optimizers/trijoint.py.151: total_norm: 21.560032 [I 2022-10-28 02:05:32] ...trap/engines/engine.py.224: train: epoch 100 | batch 800/2304 [I 2022-10-28 02:05:32] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:14:04 [I 2022-10-28 02:05:32] ...trap/engines/engine.py.229: process: 0.54872 | load: 0.00281 [I 2022-10-28 02:05:32] ...trap/engines/engine.py.230: loss: 0.11884 [I 2022-10-28 02:05:32] ...optimizers/trijoint.py.151: total_norm: 13.663424 [I 2022-10-28 02:05:38] ...trap/engines/engine.py.224: train: epoch 100 | batch 810/2304 [I 2022-10-28 02:05:38] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:14:02 [I 2022-10-28 02:05:38] ...trap/engines/engine.py.229: process: 0.56773 | load: 0.00255 [I 2022-10-28 02:05:38] ...trap/engines/engine.py.230: loss: 0.16316 [I 2022-10-28 02:05:38] ...optimizers/trijoint.py.151: total_norm: 14.676086 [I 2022-10-28 02:05:44] ...trap/engines/engine.py.224: train: epoch 100 | batch 820/2304 [I 2022-10-28 02:05:44] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:45 [I 2022-10-28 02:05:44] ...trap/engines/engine.py.229: process: 0.54515 | load: 0.00250 [I 2022-10-28 02:05:44] ...trap/engines/engine.py.230: loss: 0.12277 [I 2022-10-28 02:05:44] ...optimizers/trijoint.py.151: total_norm: 13.866757 [I 2022-10-28 02:05:49] ...trap/engines/engine.py.224: train: epoch 100 | batch 830/2304 [I 2022-10-28 02:05:49] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:48 [I 2022-10-28 02:05:49] ...trap/engines/engine.py.229: process: 0.56442 | load: 0.00252 [I 2022-10-28 02:05:49] ...trap/engines/engine.py.230: loss: 0.10143 [I 2022-10-28 02:05:49] ...optimizers/trijoint.py.151: total_norm: 21.555193 [I 2022-10-28 02:05:55] ...trap/engines/engine.py.224: train: epoch 100 | batch 840/2304 [I 2022-10-28 02:05:55] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:13:44 [I 2022-10-28 02:05:55] ...trap/engines/engine.py.229: process: 0.57442 | load: 0.00266 [I 2022-10-28 02:05:55] ...trap/engines/engine.py.230: loss: 0.16264 [I 2022-10-28 02:05:55] ...optimizers/trijoint.py.151: total_norm: 9.855000 [I 2022-10-28 02:06:01] ...trap/engines/engine.py.224: train: epoch 100 | batch 850/2304 [I 2022-10-28 02:06:01] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:35 [I 2022-10-28 02:06:01] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00245 [I 2022-10-28 02:06:01] ...trap/engines/engine.py.230: loss: 0.10317 [I 2022-10-28 02:06:01] ...optimizers/trijoint.py.151: total_norm: 8.843619 [I 2022-10-28 02:06:06] ...trap/engines/engine.py.224: train: epoch 100 | batch 860/2304 [I 2022-10-28 02:06:06] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:13:33 [I 2022-10-28 02:06:06] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00246 [I 2022-10-28 02:06:06] ...trap/engines/engine.py.230: loss: 0.11734 [I 2022-10-28 02:06:06] ...optimizers/trijoint.py.151: total_norm: 18.696684 [I 2022-10-28 02:06:12] ...trap/engines/engine.py.224: train: epoch 100 | batch 870/2304 [I 2022-10-28 02:06:12] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:15 [I 2022-10-28 02:06:12] ...trap/engines/engine.py.229: process: 0.54633 | load: 0.00244 [I 2022-10-28 02:06:12] ...trap/engines/engine.py.230: loss: 0.14106 [I 2022-10-28 02:06:12] ...optimizers/trijoint.py.151: total_norm: 31.533056 [I 2022-10-28 02:06:17] ...trap/engines/engine.py.224: train: epoch 100 | batch 880/2304 [I 2022-10-28 02:06:17] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:16 [I 2022-10-28 02:06:17] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00262 [I 2022-10-28 02:06:17] ...trap/engines/engine.py.230: loss: 0.16841 [I 2022-10-28 02:06:17] ...optimizers/trijoint.py.151: total_norm: 21.053396 [I 2022-10-28 02:06:23] ...trap/engines/engine.py.224: train: epoch 100 | batch 890/2304 [I 2022-10-28 02:06:23] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:13:03 [I 2022-10-28 02:06:23] ...trap/engines/engine.py.229: process: 0.54978 | load: 0.00242 [I 2022-10-28 02:06:23] ...trap/engines/engine.py.230: loss: 0.10442 [I 2022-10-28 02:06:23] ...optimizers/trijoint.py.151: total_norm: 16.730799 [I 2022-10-28 02:06:29] ...trap/engines/engine.py.224: train: epoch 100 | batch 900/2304 [I 2022-10-28 02:06:29] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:02 [I 2022-10-28 02:06:29] ...trap/engines/engine.py.229: process: 0.54905 | load: 0.00243 [I 2022-10-28 02:06:29] ...trap/engines/engine.py.230: loss: 0.10287 [I 2022-10-28 02:06:29] ...optimizers/trijoint.py.151: total_norm: 11.677493 [I 2022-10-28 02:06:34] ...trap/engines/engine.py.224: train: epoch 100 | batch 910/2304 [I 2022-10-28 02:06:34] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:12:56 [I 2022-10-28 02:06:34] ...trap/engines/engine.py.229: process: 0.56599 | load: 0.00248 [I 2022-10-28 02:06:34] ...trap/engines/engine.py.230: loss: 0.14030 [I 2022-10-28 02:06:34] ...optimizers/trijoint.py.151: total_norm: 14.145791 [I 2022-10-28 02:06:40] ...trap/engines/engine.py.224: train: epoch 100 | batch 920/2304 [I 2022-10-28 02:06:40] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:12:56 [I 2022-10-28 02:06:40] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00255 [I 2022-10-28 02:06:40] ...trap/engines/engine.py.230: loss: 0.10167 [I 2022-10-28 02:06:40] ...optimizers/trijoint.py.151: total_norm: 16.236460 [I 2022-10-28 02:06:45] ...trap/engines/engine.py.224: train: epoch 100 | batch 930/2304 [I 2022-10-28 02:06:45] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:12:51 [I 2022-10-28 02:06:45] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00251 [I 2022-10-28 02:06:45] ...trap/engines/engine.py.230: loss: 0.13600 [I 2022-10-28 02:06:45] ...optimizers/trijoint.py.151: total_norm: 10.238788 [I 2022-10-28 02:06:51] ...trap/engines/engine.py.224: train: epoch 100 | batch 940/2304 [I 2022-10-28 02:06:51] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:12:53 [I 2022-10-28 02:06:51] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00247 [I 2022-10-28 02:06:51] ...trap/engines/engine.py.230: loss: 0.10979 [I 2022-10-28 02:06:51] ...optimizers/trijoint.py.151: total_norm: 29.583378 [I 2022-10-28 02:06:57] ...trap/engines/engine.py.224: train: epoch 100 | batch 950/2304 [I 2022-10-28 02:06:57] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:38 [I 2022-10-28 02:06:57] ...trap/engines/engine.py.229: process: 0.55154 | load: 0.00242 [I 2022-10-28 02:06:57] ...trap/engines/engine.py.230: loss: 0.15894 [I 2022-10-28 02:06:57] ...optimizers/trijoint.py.151: total_norm: 16.951691 [I 2022-10-28 02:07:02] ...trap/engines/engine.py.224: train: epoch 100 | batch 960/2304 [I 2022-10-28 02:07:02] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:34 [I 2022-10-28 02:07:02] ...trap/engines/engine.py.229: process: 0.56947 | load: 0.00245 [I 2022-10-28 02:07:02] ...trap/engines/engine.py.230: loss: 0.10543 [I 2022-10-28 02:07:02] ...optimizers/trijoint.py.151: total_norm: 16.950375 [I 2022-10-28 02:07:08] ...trap/engines/engine.py.224: train: epoch 100 | batch 970/2304 [I 2022-10-28 02:07:08] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:25 [I 2022-10-28 02:07:08] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00242 [I 2022-10-28 02:07:08] ...trap/engines/engine.py.230: loss: 0.13204 [I 2022-10-28 02:07:08] ...optimizers/trijoint.py.151: total_norm: 20.079365 [I 2022-10-28 02:07:13] ...trap/engines/engine.py.224: train: epoch 100 | batch 980/2304 [I 2022-10-28 02:07:13] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:12:11 [I 2022-10-28 02:07:13] ...trap/engines/engine.py.229: process: 0.55459 | load: 0.00250 [I 2022-10-28 02:07:13] ...trap/engines/engine.py.230: loss: 0.10239 [I 2022-10-28 02:07:13] ...optimizers/trijoint.py.151: total_norm: 10.729098 [I 2022-10-28 02:07:19] ...trap/engines/engine.py.224: train: epoch 100 | batch 990/2304 [I 2022-10-28 02:07:19] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:08 [I 2022-10-28 02:07:19] ...trap/engines/engine.py.229: process: 0.54378 | load: 0.00253 [I 2022-10-28 02:07:19] ...trap/engines/engine.py.230: loss: 0.10667 [I 2022-10-28 02:07:19] ...optimizers/trijoint.py.151: total_norm: 11.068994 [I 2022-10-28 02:07:25] ...trap/engines/engine.py.224: train: epoch 100 | batch 1000/2304 [I 2022-10-28 02:07:25] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:08 [I 2022-10-28 02:07:25] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00249 [I 2022-10-28 02:07:25] ...trap/engines/engine.py.230: loss: 0.14330 [I 2022-10-28 02:07:25] ...optimizers/trijoint.py.151: total_norm: 13.169071 [I 2022-10-28 02:07:30] ...trap/engines/engine.py.224: train: epoch 100 | batch 1010/2304 [I 2022-10-28 02:07:30] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:12:17 [I 2022-10-28 02:07:30] ...trap/engines/engine.py.229: process: 0.57298 | load: 0.00249 [I 2022-10-28 02:07:30] ...trap/engines/engine.py.230: loss: 0.12818 [I 2022-10-28 02:07:30] ...optimizers/trijoint.py.151: total_norm: 9.519930 [I 2022-10-28 02:07:36] ...trap/engines/engine.py.224: train: epoch 100 | batch 1020/2304 [I 2022-10-28 02:07:36] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:12:01 [I 2022-10-28 02:07:36] ...trap/engines/engine.py.229: process: 0.56050 | load: 0.00244 [I 2022-10-28 02:07:36] ...trap/engines/engine.py.230: loss: 0.12923 [I 2022-10-28 02:07:36] ...optimizers/trijoint.py.151: total_norm: 37.177086 [I 2022-10-28 02:07:42] ...trap/engines/engine.py.224: train: epoch 100 | batch 1030/2304 [I 2022-10-28 02:07:42] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:11:54 [I 2022-10-28 02:07:42] ...trap/engines/engine.py.229: process: 0.57939 | load: 0.00246 [I 2022-10-28 02:07:42] ...trap/engines/engine.py.230: loss: 0.09008 [I 2022-10-28 02:07:42] ...optimizers/trijoint.py.151: total_norm: 13.057827 [I 2022-10-28 02:07:47] ...trap/engines/engine.py.224: train: epoch 100 | batch 1040/2304 [I 2022-10-28 02:07:47] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:11:47 [I 2022-10-28 02:07:47] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00265 [I 2022-10-28 02:07:47] ...trap/engines/engine.py.230: loss: 0.12610 [I 2022-10-28 02:07:47] ...optimizers/trijoint.py.151: total_norm: 24.621080 [I 2022-10-28 02:08:12] ...trap/engines/engine.py.224: train: epoch 100 | batch 1050/2304 [I 2022-10-28 02:08:12] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 1:14:30 [I 2022-10-28 02:08:12] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00238 [I 2022-10-28 02:08:12] ...trap/engines/engine.py.230: loss: 0.14758 [I 2022-10-28 02:08:12] ...optimizers/trijoint.py.151: total_norm: 14.980487 [I 2022-10-28 02:08:17] ...trap/engines/engine.py.224: train: epoch 100 | batch 1060/2304 [I 2022-10-28 02:08:17] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:18:13 [I 2022-10-28 02:08:17] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00238 [I 2022-10-28 02:08:17] ...trap/engines/engine.py.230: loss: 0.13094 [I 2022-10-28 02:08:17] ...optimizers/trijoint.py.151: total_norm: 18.478773 [I 2022-10-28 02:08:23] ...trap/engines/engine.py.224: train: epoch 100 | batch 1070/2304 [I 2022-10-28 02:08:23] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:12:15 [I 2022-10-28 02:08:23] ...trap/engines/engine.py.229: process: 0.56709 | load: 0.00237 [I 2022-10-28 02:08:23] ...trap/engines/engine.py.230: loss: 0.13258 [I 2022-10-28 02:08:23] ...optimizers/trijoint.py.151: total_norm: 20.247217 [I 2022-10-28 02:08:28] ...trap/engines/engine.py.224: train: epoch 100 | batch 1080/2304 [I 2022-10-28 02:08:28] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:31 [I 2022-10-28 02:08:28] ...trap/engines/engine.py.229: process: 0.56327 | load: 0.00250 [I 2022-10-28 02:08:28] ...trap/engines/engine.py.230: loss: 0.15592 [I 2022-10-28 02:08:28] ...optimizers/trijoint.py.151: total_norm: 10.466149 [I 2022-10-28 02:08:34] ...trap/engines/engine.py.224: train: epoch 100 | batch 1090/2304 [I 2022-10-28 02:08:34] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:11:17 [I 2022-10-28 02:08:34] ...trap/engines/engine.py.229: process: 0.55864 | load: 0.00246 [I 2022-10-28 02:08:34] ...trap/engines/engine.py.230: loss: 0.12959 [I 2022-10-28 02:08:34] ...optimizers/trijoint.py.151: total_norm: 13.919955 [I 2022-10-28 02:08:39] ...trap/engines/engine.py.224: train: epoch 100 | batch 1100/2304 [I 2022-10-28 02:08:39] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:11:11 [I 2022-10-28 02:08:39] ...trap/engines/engine.py.229: process: 0.55319 | load: 0.00240 [I 2022-10-28 02:08:39] ...trap/engines/engine.py.230: loss: 0.13383 [I 2022-10-28 02:08:39] ...optimizers/trijoint.py.151: total_norm: 17.943186 [I 2022-10-28 02:08:45] ...trap/engines/engine.py.224: train: epoch 100 | batch 1110/2304 [I 2022-10-28 02:08:45] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:11:07 [I 2022-10-28 02:08:45] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00238 [I 2022-10-28 02:08:45] ...trap/engines/engine.py.230: loss: 0.13653 [I 2022-10-28 02:08:45] ...optimizers/trijoint.py.151: total_norm: 20.004181 [I 2022-10-28 02:08:51] ...trap/engines/engine.py.224: train: epoch 100 | batch 1120/2304 [I 2022-10-28 02:08:51] ...trap/engines/engine.py.228: elapsed: 0:10:50 | left: 0:11:03 [I 2022-10-28 02:08:51] ...trap/engines/engine.py.229: process: 0.57401 | load: 0.00243 [I 2022-10-28 02:08:51] ...trap/engines/engine.py.230: loss: 0.18232 [I 2022-10-28 02:08:51] ...optimizers/trijoint.py.151: total_norm: 30.321207 [I 2022-10-28 02:08:56] ...trap/engines/engine.py.224: train: epoch 100 | batch 1130/2304 [I 2022-10-28 02:08:56] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:53 [I 2022-10-28 02:08:56] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00246 [I 2022-10-28 02:08:56] ...trap/engines/engine.py.230: loss: 0.16688 [I 2022-10-28 02:08:56] ...optimizers/trijoint.py.151: total_norm: 23.539900 [I 2022-10-28 02:09:02] ...trap/engines/engine.py.224: train: epoch 100 | batch 1140/2304 [I 2022-10-28 02:09:02] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:46 [I 2022-10-28 02:09:02] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00241 [I 2022-10-28 02:09:02] ...trap/engines/engine.py.230: loss: 0.08795 [I 2022-10-28 02:09:02] ...optimizers/trijoint.py.151: total_norm: 11.369802 [I 2022-10-28 02:09:07] ...trap/engines/engine.py.224: train: epoch 100 | batch 1150/2304 [I 2022-10-28 02:09:07] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:46 [I 2022-10-28 02:09:07] ...trap/engines/engine.py.229: process: 0.57059 | load: 0.00243 [I 2022-10-28 02:09:07] ...trap/engines/engine.py.230: loss: 0.11508 [I 2022-10-28 02:09:07] ...optimizers/trijoint.py.151: total_norm: 12.020836 [I 2022-10-28 02:09:13] ...trap/engines/engine.py.224: train: epoch 100 | batch 1160/2304 [I 2022-10-28 02:09:13] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:35 [I 2022-10-28 02:09:13] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00241 [I 2022-10-28 02:09:13] ...trap/engines/engine.py.230: loss: 0.16144 [I 2022-10-28 02:09:13] ...optimizers/trijoint.py.151: total_norm: 22.165251 [I 2022-10-28 02:09:19] ...trap/engines/engine.py.224: train: epoch 100 | batch 1170/2304 [I 2022-10-28 02:09:19] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:40 [I 2022-10-28 02:09:19] ...trap/engines/engine.py.229: process: 0.54377 | load: 0.00246 [I 2022-10-28 02:09:19] ...trap/engines/engine.py.230: loss: 0.12645 [I 2022-10-28 02:09:19] ...optimizers/trijoint.py.151: total_norm: 23.829908 [I 2022-10-28 02:09:24] ...trap/engines/engine.py.224: train: epoch 100 | batch 1180/2304 [I 2022-10-28 02:09:24] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:25 [I 2022-10-28 02:09:24] ...trap/engines/engine.py.229: process: 0.55881 | load: 0.00246 [I 2022-10-28 02:09:24] ...trap/engines/engine.py.230: loss: 0.09722 [I 2022-10-28 02:09:24] ...optimizers/trijoint.py.151: total_norm: 23.446827 [I 2022-10-28 02:09:30] ...trap/engines/engine.py.224: train: epoch 100 | batch 1190/2304 [I 2022-10-28 02:09:30] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:10:20 [I 2022-10-28 02:09:30] ...trap/engines/engine.py.229: process: 0.54821 | load: 0.00245 [I 2022-10-28 02:09:30] ...trap/engines/engine.py.230: loss: 0.12918 [I 2022-10-28 02:09:30] ...optimizers/trijoint.py.151: total_norm: 11.084845 [I 2022-10-28 02:09:35] ...trap/engines/engine.py.224: train: epoch 100 | batch 1200/2304 [I 2022-10-28 02:09:35] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:10:13 [I 2022-10-28 02:09:35] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00244 [I 2022-10-28 02:09:35] ...trap/engines/engine.py.230: loss: 0.07854 [I 2022-10-28 02:09:35] ...optimizers/trijoint.py.151: total_norm: 14.342490 [I 2022-10-28 02:09:41] ...trap/engines/engine.py.224: train: epoch 100 | batch 1210/2304 [I 2022-10-28 02:09:41] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:10:12 [I 2022-10-28 02:09:41] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00244 [I 2022-10-28 02:09:41] ...trap/engines/engine.py.230: loss: 0.12913 [I 2022-10-28 02:09:41] ...optimizers/trijoint.py.151: total_norm: 13.491651 [I 2022-10-28 02:09:47] ...trap/engines/engine.py.224: train: epoch 100 | batch 1220/2304 [I 2022-10-28 02:09:47] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:10:04 [I 2022-10-28 02:09:47] ...trap/engines/engine.py.229: process: 0.54908 | load: 0.00246 [I 2022-10-28 02:09:47] ...trap/engines/engine.py.230: loss: 0.11762 [I 2022-10-28 02:09:47] ...optimizers/trijoint.py.151: total_norm: 16.271639 [I 2022-10-28 02:09:52] ...trap/engines/engine.py.224: train: epoch 100 | batch 1230/2304 [I 2022-10-28 02:09:52] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:10:28 [I 2022-10-28 02:09:52] ...trap/engines/engine.py.229: process: 0.64097 | load: 0.00316 [I 2022-10-28 02:09:52] ...trap/engines/engine.py.230: loss: 0.10521 [I 2022-10-28 02:09:52] ...optimizers/trijoint.py.151: total_norm: 11.605248 [I 2022-10-28 02:09:58] ...trap/engines/engine.py.224: train: epoch 100 | batch 1240/2304 [I 2022-10-28 02:09:58] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:59 [I 2022-10-28 02:09:58] ...trap/engines/engine.py.229: process: 0.55977 | load: 0.00243 [I 2022-10-28 02:09:58] ...trap/engines/engine.py.230: loss: 0.09835 [I 2022-10-28 02:09:58] ...optimizers/trijoint.py.151: total_norm: 34.975445 [I 2022-10-28 02:10:04] ...trap/engines/engine.py.224: train: epoch 100 | batch 1250/2304 [I 2022-10-28 02:10:04] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:47 [I 2022-10-28 02:10:04] ...trap/engines/engine.py.229: process: 0.56505 | load: 0.00247 [I 2022-10-28 02:10:04] ...trap/engines/engine.py.230: loss: 0.15743 [I 2022-10-28 02:10:04] ...optimizers/trijoint.py.151: total_norm: 8.620501 [I 2022-10-28 02:10:09] ...trap/engines/engine.py.224: train: epoch 100 | batch 1260/2304 [I 2022-10-28 02:10:09] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:42 [I 2022-10-28 02:10:09] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00243 [I 2022-10-28 02:10:09] ...trap/engines/engine.py.230: loss: 0.09805 [I 2022-10-28 02:10:09] ...optimizers/trijoint.py.151: total_norm: 17.510849 [I 2022-10-28 02:10:15] ...trap/engines/engine.py.224: train: epoch 100 | batch 1270/2304 [I 2022-10-28 02:10:15] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:35 [I 2022-10-28 02:10:15] ...trap/engines/engine.py.229: process: 0.56406 | load: 0.00249 [I 2022-10-28 02:10:15] ...trap/engines/engine.py.230: loss: 0.10363 [I 2022-10-28 02:10:15] ...optimizers/trijoint.py.151: total_norm: 13.773139 [I 2022-10-28 02:10:20] ...trap/engines/engine.py.224: train: epoch 100 | batch 1280/2304 [I 2022-10-28 02:10:20] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:35 [I 2022-10-28 02:10:20] ...trap/engines/engine.py.229: process: 0.55756 | load: 0.00245 [I 2022-10-28 02:10:20] ...trap/engines/engine.py.230: loss: 0.16836 [I 2022-10-28 02:10:20] ...optimizers/trijoint.py.151: total_norm: 15.507271 [I 2022-10-28 02:10:27] ...trap/engines/engine.py.224: train: epoch 100 | batch 1290/2304 [I 2022-10-28 02:10:27] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:11:53 [I 2022-10-28 02:10:27] ...trap/engines/engine.py.229: process: 0.76713 | load: 0.00242 [I 2022-10-28 02:10:27] ...trap/engines/engine.py.230: loss: 0.19108 [I 2022-10-28 02:10:27] ...optimizers/trijoint.py.151: total_norm: 10.515699 [I 2022-10-28 02:10:33] ...trap/engines/engine.py.224: train: epoch 100 | batch 1300/2304 [I 2022-10-28 02:10:33] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:09:44 [I 2022-10-28 02:10:33] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00241 [I 2022-10-28 02:10:33] ...trap/engines/engine.py.230: loss: 0.11506 [I 2022-10-28 02:10:33] ...optimizers/trijoint.py.151: total_norm: 17.197622 [I 2022-10-28 02:10:39] ...trap/engines/engine.py.224: train: epoch 100 | batch 1310/2304 [I 2022-10-28 02:10:39] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:09:13 [I 2022-10-28 02:10:39] ...trap/engines/engine.py.229: process: 0.55594 | load: 0.00241 [I 2022-10-28 02:10:39] ...trap/engines/engine.py.230: loss: 0.15365 [I 2022-10-28 02:10:39] ...optimizers/trijoint.py.151: total_norm: 20.099892 [I 2022-10-28 02:10:44] ...trap/engines/engine.py.224: train: epoch 100 | batch 1320/2304 [I 2022-10-28 02:10:44] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:09:05 [I 2022-10-28 02:10:44] ...trap/engines/engine.py.229: process: 0.53941 | load: 0.00240 [I 2022-10-28 02:10:44] ...trap/engines/engine.py.230: loss: 0.09342 [I 2022-10-28 02:10:44] ...optimizers/trijoint.py.151: total_norm: 10.856006 [I 2022-10-28 02:10:50] ...trap/engines/engine.py.224: train: epoch 100 | batch 1330/2304 [I 2022-10-28 02:10:50] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:09:01 [I 2022-10-28 02:10:50] ...trap/engines/engine.py.229: process: 0.54697 | load: 0.00245 [I 2022-10-28 02:10:50] ...trap/engines/engine.py.230: loss: 0.09824 [I 2022-10-28 02:10:50] ...optimizers/trijoint.py.151: total_norm: 12.171376 [I 2022-10-28 02:10:55] ...trap/engines/engine.py.224: train: epoch 100 | batch 1340/2304 [I 2022-10-28 02:10:55] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:58 [I 2022-10-28 02:10:55] ...trap/engines/engine.py.229: process: 0.56152 | load: 0.00241 [I 2022-10-28 02:10:55] ...trap/engines/engine.py.230: loss: 0.13962 [I 2022-10-28 02:10:55] ...optimizers/trijoint.py.151: total_norm: 15.987902 [I 2022-10-28 02:11:01] ...trap/engines/engine.py.224: train: epoch 100 | batch 1350/2304 [I 2022-10-28 02:11:01] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:53 [I 2022-10-28 02:11:01] ...trap/engines/engine.py.229: process: 0.56301 | load: 0.00243 [I 2022-10-28 02:11:01] ...trap/engines/engine.py.230: loss: 0.13107 [I 2022-10-28 02:11:01] ...optimizers/trijoint.py.151: total_norm: 22.149099 [I 2022-10-28 02:11:07] ...trap/engines/engine.py.224: train: epoch 100 | batch 1360/2304 [I 2022-10-28 02:11:07] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:57 [I 2022-10-28 02:11:07] ...trap/engines/engine.py.229: process: 0.56884 | load: 0.00241 [I 2022-10-28 02:11:07] ...trap/engines/engine.py.230: loss: 0.09229 [I 2022-10-28 02:11:07] ...optimizers/trijoint.py.151: total_norm: 12.594938 [I 2022-10-28 02:11:12] ...trap/engines/engine.py.224: train: epoch 100 | batch 1370/2304 [I 2022-10-28 02:11:12] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:41 [I 2022-10-28 02:11:12] ...trap/engines/engine.py.229: process: 0.54584 | load: 0.00243 [I 2022-10-28 02:11:12] ...trap/engines/engine.py.230: loss: 0.14465 [I 2022-10-28 02:11:12] ...optimizers/trijoint.py.151: total_norm: 11.523807 [I 2022-10-28 02:11:18] ...trap/engines/engine.py.224: train: epoch 100 | batch 1380/2304 [I 2022-10-28 02:11:18] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:32 [I 2022-10-28 02:11:18] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00242 [I 2022-10-28 02:11:18] ...trap/engines/engine.py.230: loss: 0.10003 [I 2022-10-28 02:11:18] ...optimizers/trijoint.py.151: total_norm: 9.833765 [I 2022-10-28 02:11:23] ...trap/engines/engine.py.224: train: epoch 100 | batch 1390/2304 [I 2022-10-28 02:11:23] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:30 [I 2022-10-28 02:11:23] ...trap/engines/engine.py.229: process: 0.57185 | load: 0.00244 [I 2022-10-28 02:11:23] ...trap/engines/engine.py.230: loss: 0.16460 [I 2022-10-28 02:11:23] ...optimizers/trijoint.py.151: total_norm: 12.142221 [I 2022-10-28 02:11:29] ...trap/engines/engine.py.224: train: epoch 100 | batch 1400/2304 [I 2022-10-28 02:11:29] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:25 [I 2022-10-28 02:11:29] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00241 [I 2022-10-28 02:11:29] ...trap/engines/engine.py.230: loss: 0.12195 [I 2022-10-28 02:11:29] ...optimizers/trijoint.py.151: total_norm: 21.012135 [I 2022-10-28 02:11:35] ...trap/engines/engine.py.224: train: epoch 100 | batch 1410/2304 [I 2022-10-28 02:11:35] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:08:17 [I 2022-10-28 02:11:35] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00240 [I 2022-10-28 02:11:35] ...trap/engines/engine.py.230: loss: 0.15507 [I 2022-10-28 02:11:35] ...optimizers/trijoint.py.151: total_norm: 9.642153 [I 2022-10-28 02:11:40] ...trap/engines/engine.py.224: train: epoch 100 | batch 1420/2304 [I 2022-10-28 02:11:40] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:08:16 [I 2022-10-28 02:11:40] ...trap/engines/engine.py.229: process: 0.56444 | load: 0.00254 [I 2022-10-28 02:11:40] ...trap/engines/engine.py.230: loss: 0.08712 [I 2022-10-28 02:11:40] ...optimizers/trijoint.py.151: total_norm: 17.945507 [I 2022-10-28 02:11:46] ...trap/engines/engine.py.224: train: epoch 100 | batch 1430/2304 [I 2022-10-28 02:11:46] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:08:09 [I 2022-10-28 02:11:46] ...trap/engines/engine.py.229: process: 0.54459 | load: 0.00249 [I 2022-10-28 02:11:46] ...trap/engines/engine.py.230: loss: 0.09817 [I 2022-10-28 02:11:46] ...optimizers/trijoint.py.151: total_norm: 12.761645 [I 2022-10-28 02:11:52] ...trap/engines/engine.py.224: train: epoch 100 | batch 1440/2304 [I 2022-10-28 02:11:52] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:08:04 [I 2022-10-28 02:11:52] ...trap/engines/engine.py.229: process: 0.54968 | load: 0.00244 [I 2022-10-28 02:11:52] ...trap/engines/engine.py.230: loss: 0.11282 [I 2022-10-28 02:11:52] ...optimizers/trijoint.py.151: total_norm: 15.213753 [I 2022-10-28 02:11:57] ...trap/engines/engine.py.224: train: epoch 100 | batch 1450/2304 [I 2022-10-28 02:11:57] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:57 [I 2022-10-28 02:11:57] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00244 [I 2022-10-28 02:11:57] ...trap/engines/engine.py.230: loss: 0.10727 [I 2022-10-28 02:11:57] ...optimizers/trijoint.py.151: total_norm: 17.927723 [I 2022-10-28 02:12:03] ...trap/engines/engine.py.224: train: epoch 100 | batch 1460/2304 [I 2022-10-28 02:12:03] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:51 [I 2022-10-28 02:12:03] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00239 [I 2022-10-28 02:12:03] ...trap/engines/engine.py.230: loss: 0.09435 [I 2022-10-28 02:12:03] ...optimizers/trijoint.py.151: total_norm: 10.978483 [I 2022-10-28 02:12:08] ...trap/engines/engine.py.224: train: epoch 100 | batch 1470/2304 [I 2022-10-28 02:12:08] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:45 [I 2022-10-28 02:12:08] ...trap/engines/engine.py.229: process: 0.55923 | load: 0.00241 [I 2022-10-28 02:12:08] ...trap/engines/engine.py.230: loss: 0.12757 [I 2022-10-28 02:12:08] ...optimizers/trijoint.py.151: total_norm: 22.688156 [I 2022-10-28 02:12:14] ...trap/engines/engine.py.224: train: epoch 100 | batch 1480/2304 [I 2022-10-28 02:12:14] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:40 [I 2022-10-28 02:12:14] ...trap/engines/engine.py.229: process: 0.54819 | load: 0.00246 [I 2022-10-28 02:12:14] ...trap/engines/engine.py.230: loss: 0.21581 [I 2022-10-28 02:12:14] ...optimizers/trijoint.py.151: total_norm: 31.882591 [I 2022-10-28 02:12:19] ...trap/engines/engine.py.224: train: epoch 100 | batch 1490/2304 [I 2022-10-28 02:12:19] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:30 [I 2022-10-28 02:12:19] ...trap/engines/engine.py.229: process: 0.54328 | load: 0.00242 [I 2022-10-28 02:12:19] ...trap/engines/engine.py.230: loss: 0.11056 [I 2022-10-28 02:12:19] ...optimizers/trijoint.py.151: total_norm: 25.542545 [I 2022-10-28 02:12:25] ...trap/engines/engine.py.224: train: epoch 100 | batch 1500/2304 [I 2022-10-28 02:12:25] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:26 [I 2022-10-28 02:12:25] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00242 [I 2022-10-28 02:12:25] ...trap/engines/engine.py.230: loss: 0.11311 [I 2022-10-28 02:12:25] ...optimizers/trijoint.py.151: total_norm: 7.196040 [I 2022-10-28 02:12:31] ...trap/engines/engine.py.224: train: epoch 100 | batch 1510/2304 [I 2022-10-28 02:12:31] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:22 [I 2022-10-28 02:12:31] ...trap/engines/engine.py.229: process: 0.55058 | load: 0.00244 [I 2022-10-28 02:12:31] ...trap/engines/engine.py.230: loss: 0.11052 [I 2022-10-28 02:12:31] ...optimizers/trijoint.py.151: total_norm: 24.674561 [I 2022-10-28 02:12:36] ...trap/engines/engine.py.224: train: epoch 100 | batch 1520/2304 [I 2022-10-28 02:12:36] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:07:17 [I 2022-10-28 02:12:36] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00250 [I 2022-10-28 02:12:36] ...trap/engines/engine.py.230: loss: 0.14003 [I 2022-10-28 02:12:36] ...optimizers/trijoint.py.151: total_norm: 13.264210 [I 2022-10-28 02:12:42] ...trap/engines/engine.py.224: train: epoch 100 | batch 1530/2304 [I 2022-10-28 02:12:42] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:07:12 [I 2022-10-28 02:12:42] ...trap/engines/engine.py.229: process: 0.56726 | load: 0.00247 [I 2022-10-28 02:12:42] ...trap/engines/engine.py.230: loss: 0.15118 [I 2022-10-28 02:12:42] ...optimizers/trijoint.py.151: total_norm: 48.312702 [I 2022-10-28 02:12:47] ...trap/engines/engine.py.224: train: epoch 100 | batch 1540/2304 [I 2022-10-28 02:12:47] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:07:07 [I 2022-10-28 02:12:47] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00244 [I 2022-10-28 02:12:47] ...trap/engines/engine.py.230: loss: 0.14374 [I 2022-10-28 02:12:47] ...optimizers/trijoint.py.151: total_norm: 20.337208 [I 2022-10-28 02:12:53] ...trap/engines/engine.py.224: train: epoch 100 | batch 1550/2304 [I 2022-10-28 02:12:53] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:07:01 [I 2022-10-28 02:12:53] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00247 [I 2022-10-28 02:12:53] ...trap/engines/engine.py.230: loss: 0.17629 [I 2022-10-28 02:12:53] ...optimizers/trijoint.py.151: total_norm: 9.239042 [I 2022-10-28 02:12:59] ...trap/engines/engine.py.224: train: epoch 100 | batch 1560/2304 [I 2022-10-28 02:12:59] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:54 [I 2022-10-28 02:12:59] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00244 [I 2022-10-28 02:12:59] ...trap/engines/engine.py.230: loss: 0.17136 [I 2022-10-28 02:12:59] ...optimizers/trijoint.py.151: total_norm: 12.953992 [I 2022-10-28 02:13:04] ...trap/engines/engine.py.224: train: epoch 100 | batch 1570/2304 [I 2022-10-28 02:13:04] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:46 [I 2022-10-28 02:13:04] ...trap/engines/engine.py.229: process: 0.54738 | load: 0.00241 [I 2022-10-28 02:13:04] ...trap/engines/engine.py.230: loss: 0.13916 [I 2022-10-28 02:13:04] ...optimizers/trijoint.py.151: total_norm: 22.183062 [I 2022-10-28 02:13:10] ...trap/engines/engine.py.224: train: epoch 100 | batch 1580/2304 [I 2022-10-28 02:13:10] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:40 [I 2022-10-28 02:13:10] ...trap/engines/engine.py.229: process: 0.54300 | load: 0.00244 [I 2022-10-28 02:13:10] ...trap/engines/engine.py.230: loss: 0.14949 [I 2022-10-28 02:13:10] ...optimizers/trijoint.py.151: total_norm: 23.225136 [I 2022-10-28 02:13:15] ...trap/engines/engine.py.224: train: epoch 100 | batch 1590/2304 [I 2022-10-28 02:13:15] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:37 [I 2022-10-28 02:13:15] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00244 [I 2022-10-28 02:13:15] ...trap/engines/engine.py.230: loss: 0.15577 [I 2022-10-28 02:13:15] ...optimizers/trijoint.py.151: total_norm: 22.797861 [I 2022-10-28 02:13:21] ...trap/engines/engine.py.224: train: epoch 100 | batch 1600/2304 [I 2022-10-28 02:13:21] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:33 [I 2022-10-28 02:13:21] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00244 [I 2022-10-28 02:13:21] ...trap/engines/engine.py.230: loss: 0.09910 [I 2022-10-28 02:13:21] ...optimizers/trijoint.py.151: total_norm: 13.002945 [I 2022-10-28 02:13:26] ...trap/engines/engine.py.224: train: epoch 100 | batch 1610/2304 [I 2022-10-28 02:13:26] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:23 [I 2022-10-28 02:13:26] ...trap/engines/engine.py.229: process: 0.55062 | load: 0.00240 [I 2022-10-28 02:13:26] ...trap/engines/engine.py.230: loss: 0.11450 [I 2022-10-28 02:13:26] ...optimizers/trijoint.py.151: total_norm: 19.697845 [I 2022-10-28 02:13:32] ...trap/engines/engine.py.224: train: epoch 100 | batch 1620/2304 [I 2022-10-28 02:13:32] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:19 [I 2022-10-28 02:13:32] ...trap/engines/engine.py.229: process: 0.54896 | load: 0.00243 [I 2022-10-28 02:13:32] ...trap/engines/engine.py.230: loss: 0.17947 [I 2022-10-28 02:13:32] ...optimizers/trijoint.py.151: total_norm: 25.560783 [I 2022-10-28 02:13:38] ...trap/engines/engine.py.224: train: epoch 100 | batch 1630/2304 [I 2022-10-28 02:13:38] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:15 [I 2022-10-28 02:13:38] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00241 [I 2022-10-28 02:13:38] ...trap/engines/engine.py.230: loss: 0.14186 [I 2022-10-28 02:13:38] ...optimizers/trijoint.py.151: total_norm: 12.081475 [I 2022-10-28 02:13:43] ...trap/engines/engine.py.224: train: epoch 100 | batch 1640/2304 [I 2022-10-28 02:13:43] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:06:10 [I 2022-10-28 02:13:43] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00241 [I 2022-10-28 02:13:43] ...trap/engines/engine.py.230: loss: 0.09472 [I 2022-10-28 02:13:43] ...optimizers/trijoint.py.151: total_norm: 9.509368 [I 2022-10-28 02:13:49] ...trap/engines/engine.py.224: train: epoch 100 | batch 1650/2304 [I 2022-10-28 02:13:49] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:06:04 [I 2022-10-28 02:13:49] ...trap/engines/engine.py.229: process: 0.55807 | load: 0.00241 [I 2022-10-28 02:13:49] ...trap/engines/engine.py.230: loss: 0.08644 [I 2022-10-28 02:13:49] ...optimizers/trijoint.py.151: total_norm: 10.904554 [I 2022-10-28 02:13:54] ...trap/engines/engine.py.224: train: epoch 100 | batch 1660/2304 [I 2022-10-28 02:13:54] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:58 [I 2022-10-28 02:13:54] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00240 [I 2022-10-28 02:13:54] ...trap/engines/engine.py.230: loss: 0.12637 [I 2022-10-28 02:13:54] ...optimizers/trijoint.py.151: total_norm: 16.087072 [I 2022-10-28 02:14:00] ...trap/engines/engine.py.224: train: epoch 100 | batch 1670/2304 [I 2022-10-28 02:14:00] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:52 [I 2022-10-28 02:14:00] ...trap/engines/engine.py.229: process: 0.54954 | load: 0.00241 [I 2022-10-28 02:14:00] ...trap/engines/engine.py.230: loss: 0.14797 [I 2022-10-28 02:14:00] ...optimizers/trijoint.py.151: total_norm: 12.191381 [I 2022-10-28 02:14:06] ...trap/engines/engine.py.224: train: epoch 100 | batch 1680/2304 [I 2022-10-28 02:14:06] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:48 [I 2022-10-28 02:14:06] ...trap/engines/engine.py.229: process: 0.55433 | load: 0.00250 [I 2022-10-28 02:14:06] ...trap/engines/engine.py.230: loss: 0.09179 [I 2022-10-28 02:14:06] ...optimizers/trijoint.py.151: total_norm: 30.220335 [I 2022-10-28 02:14:11] ...trap/engines/engine.py.224: train: epoch 100 | batch 1690/2304 [I 2022-10-28 02:14:11] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:42 [I 2022-10-28 02:14:11] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00248 [I 2022-10-28 02:14:11] ...trap/engines/engine.py.230: loss: 0.12590 [I 2022-10-28 02:14:11] ...optimizers/trijoint.py.151: total_norm: 9.472075 [I 2022-10-28 02:14:17] ...trap/engines/engine.py.224: train: epoch 100 | batch 1700/2304 [I 2022-10-28 02:14:17] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:37 [I 2022-10-28 02:14:17] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00244 [I 2022-10-28 02:14:17] ...trap/engines/engine.py.230: loss: 0.15306 [I 2022-10-28 02:14:17] ...optimizers/trijoint.py.151: total_norm: 24.182741 [I 2022-10-28 02:14:22] ...trap/engines/engine.py.224: train: epoch 100 | batch 1710/2304 [I 2022-10-28 02:14:22] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:30 [I 2022-10-28 02:14:22] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00243 [I 2022-10-28 02:14:22] ...trap/engines/engine.py.230: loss: 0.10724 [I 2022-10-28 02:14:22] ...optimizers/trijoint.py.151: total_norm: 21.906307 [I 2022-10-28 02:14:28] ...trap/engines/engine.py.224: train: epoch 100 | batch 1720/2304 [I 2022-10-28 02:14:28] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:25 [I 2022-10-28 02:14:28] ...trap/engines/engine.py.229: process: 0.54865 | load: 0.00241 [I 2022-10-28 02:14:28] ...trap/engines/engine.py.230: loss: 0.14232 [I 2022-10-28 02:14:28] ...optimizers/trijoint.py.151: total_norm: 22.004553 [I 2022-10-28 02:14:34] ...trap/engines/engine.py.224: train: epoch 100 | batch 1730/2304 [I 2022-10-28 02:14:34] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:19 [I 2022-10-28 02:14:34] ...trap/engines/engine.py.229: process: 0.54253 | load: 0.00239 [I 2022-10-28 02:14:34] ...trap/engines/engine.py.230: loss: 0.17306 [I 2022-10-28 02:14:34] ...optimizers/trijoint.py.151: total_norm: 19.440504 [I 2022-10-28 02:14:39] ...trap/engines/engine.py.224: train: epoch 100 | batch 1740/2304 [I 2022-10-28 02:14:39] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:05:14 [I 2022-10-28 02:14:39] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00241 [I 2022-10-28 02:14:39] ...trap/engines/engine.py.230: loss: 0.13000 [I 2022-10-28 02:14:39] ...optimizers/trijoint.py.151: total_norm: 14.300083 [I 2022-10-28 02:14:45] ...trap/engines/engine.py.224: train: epoch 100 | batch 1750/2304 [I 2022-10-28 02:14:45] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:07 [I 2022-10-28 02:14:45] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00241 [I 2022-10-28 02:14:45] ...trap/engines/engine.py.230: loss: 0.09160 [I 2022-10-28 02:14:45] ...optimizers/trijoint.py.151: total_norm: 14.579114 [I 2022-10-28 02:14:50] ...trap/engines/engine.py.224: train: epoch 100 | batch 1760/2304 [I 2022-10-28 02:14:50] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:05:02 [I 2022-10-28 02:14:50] ...trap/engines/engine.py.229: process: 0.55842 | load: 0.00242 [I 2022-10-28 02:14:50] ...trap/engines/engine.py.230: loss: 0.13957 [I 2022-10-28 02:14:50] ...optimizers/trijoint.py.151: total_norm: 20.611776 [I 2022-10-28 02:14:56] ...trap/engines/engine.py.224: train: epoch 100 | batch 1770/2304 [I 2022-10-28 02:14:56] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:56 [I 2022-10-28 02:14:56] ...trap/engines/engine.py.229: process: 0.55030 | load: 0.00240 [I 2022-10-28 02:14:56] ...trap/engines/engine.py.230: loss: 0.15046 [I 2022-10-28 02:14:56] ...optimizers/trijoint.py.151: total_norm: 25.151871 [I 2022-10-28 02:15:01] ...trap/engines/engine.py.224: train: epoch 100 | batch 1780/2304 [I 2022-10-28 02:15:01] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:53 [I 2022-10-28 02:15:01] ...trap/engines/engine.py.229: process: 0.55821 | load: 0.00239 [I 2022-10-28 02:15:01] ...trap/engines/engine.py.230: loss: 0.09734 [I 2022-10-28 02:15:01] ...optimizers/trijoint.py.151: total_norm: 19.049307 [I 2022-10-28 02:15:07] ...trap/engines/engine.py.224: train: epoch 100 | batch 1790/2304 [I 2022-10-28 02:15:07] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:44 [I 2022-10-28 02:15:07] ...trap/engines/engine.py.229: process: 0.55271 | load: 0.00244 [I 2022-10-28 02:15:07] ...trap/engines/engine.py.230: loss: 0.11655 [I 2022-10-28 02:15:07] ...optimizers/trijoint.py.151: total_norm: 10.761761 [I 2022-10-28 02:15:13] ...trap/engines/engine.py.224: train: epoch 100 | batch 1800/2304 [I 2022-10-28 02:15:13] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:39 [I 2022-10-28 02:15:13] ...trap/engines/engine.py.229: process: 0.55701 | load: 0.00247 [I 2022-10-28 02:15:13] ...trap/engines/engine.py.230: loss: 0.08189 [I 2022-10-28 02:15:13] ...optimizers/trijoint.py.151: total_norm: 17.639902 [I 2022-10-28 02:15:18] ...trap/engines/engine.py.224: train: epoch 100 | batch 1810/2304 [I 2022-10-28 02:15:18] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:37 [I 2022-10-28 02:15:18] ...trap/engines/engine.py.229: process: 0.56678 | load: 0.00248 [I 2022-10-28 02:15:18] ...trap/engines/engine.py.230: loss: 0.11009 [I 2022-10-28 02:15:18] ...optimizers/trijoint.py.151: total_norm: 14.176218 [I 2022-10-28 02:15:24] ...trap/engines/engine.py.224: train: epoch 100 | batch 1820/2304 [I 2022-10-28 02:15:24] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:30 [I 2022-10-28 02:15:24] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00244 [I 2022-10-28 02:15:24] ...trap/engines/engine.py.230: loss: 0.20172 [I 2022-10-28 02:15:24] ...optimizers/trijoint.py.151: total_norm: 41.469940 [I 2022-10-28 02:15:29] ...trap/engines/engine.py.224: train: epoch 100 | batch 1830/2304 [I 2022-10-28 02:15:29] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:25 [I 2022-10-28 02:15:29] ...trap/engines/engine.py.229: process: 0.56036 | load: 0.00247 [I 2022-10-28 02:15:29] ...trap/engines/engine.py.230: loss: 0.08666 [I 2022-10-28 02:15:29] ...optimizers/trijoint.py.151: total_norm: 19.460186 [I 2022-10-28 02:15:35] ...trap/engines/engine.py.224: train: epoch 100 | batch 1840/2304 [I 2022-10-28 02:15:35] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:04:17 [I 2022-10-28 02:15:35] ...trap/engines/engine.py.229: process: 0.55696 | load: 0.00243 [I 2022-10-28 02:15:35] ...trap/engines/engine.py.230: loss: 0.09987 [I 2022-10-28 02:15:35] ...optimizers/trijoint.py.151: total_norm: 13.321022 [I 2022-10-28 02:15:41] ...trap/engines/engine.py.224: train: epoch 100 | batch 1850/2304 [I 2022-10-28 02:15:41] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:04:12 [I 2022-10-28 02:15:41] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00243 [I 2022-10-28 02:15:41] ...trap/engines/engine.py.230: loss: 0.14004 [I 2022-10-28 02:15:41] ...optimizers/trijoint.py.151: total_norm: 14.682752 [I 2022-10-28 02:15:46] ...trap/engines/engine.py.224: train: epoch 100 | batch 1860/2304 [I 2022-10-28 02:15:46] ...trap/engines/engine.py.228: elapsed: 0:17:45 | left: 0:04:07 [I 2022-10-28 02:15:46] ...trap/engines/engine.py.229: process: 0.56182 | load: 0.00246 [I 2022-10-28 02:15:46] ...trap/engines/engine.py.230: loss: 0.12056 [I 2022-10-28 02:15:46] ...optimizers/trijoint.py.151: total_norm: 10.740306 [I 2022-10-28 02:15:52] ...trap/engines/engine.py.224: train: epoch 100 | batch 1870/2304 [I 2022-10-28 02:15:52] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:04:02 [I 2022-10-28 02:15:52] ...trap/engines/engine.py.229: process: 0.54571 | load: 0.00244 [I 2022-10-28 02:15:52] ...trap/engines/engine.py.230: loss: 0.10326 [I 2022-10-28 02:15:52] ...optimizers/trijoint.py.151: total_norm: 8.754180 [I 2022-10-28 02:15:57] ...trap/engines/engine.py.224: train: epoch 100 | batch 1880/2304 [I 2022-10-28 02:15:57] ...trap/engines/engine.py.228: elapsed: 0:17:56 | left: 0:03:56 [I 2022-10-28 02:15:57] ...trap/engines/engine.py.229: process: 0.54232 | load: 0.00246 [I 2022-10-28 02:15:57] ...trap/engines/engine.py.230: loss: 0.09513 [I 2022-10-28 02:15:57] ...optimizers/trijoint.py.151: total_norm: 7.645470 [I 2022-10-28 02:16:03] ...trap/engines/engine.py.224: train: epoch 100 | batch 1890/2304 [I 2022-10-28 02:16:03] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:50 [I 2022-10-28 02:16:03] ...trap/engines/engine.py.229: process: 0.55574 | load: 0.00243 [I 2022-10-28 02:16:03] ...trap/engines/engine.py.230: loss: 0.10845 [I 2022-10-28 02:16:03] ...optimizers/trijoint.py.151: total_norm: 14.243183 [I 2022-10-28 02:16:09] ...trap/engines/engine.py.224: train: epoch 100 | batch 1900/2304 [I 2022-10-28 02:16:09] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:46 [I 2022-10-28 02:16:09] ...trap/engines/engine.py.229: process: 0.55857 | load: 0.00281 [I 2022-10-28 02:16:09] ...trap/engines/engine.py.230: loss: 0.13812 [I 2022-10-28 02:16:09] ...optimizers/trijoint.py.151: total_norm: 10.534186 [I 2022-10-28 02:16:14] ...trap/engines/engine.py.224: train: epoch 100 | batch 1910/2304 [I 2022-10-28 02:16:14] ...trap/engines/engine.py.228: elapsed: 0:18:13 | left: 0:03:39 [I 2022-10-28 02:16:14] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00244 [I 2022-10-28 02:16:14] ...trap/engines/engine.py.230: loss: 0.11225 [I 2022-10-28 02:16:14] ...optimizers/trijoint.py.151: total_norm: 15.861029 [I 2022-10-28 02:16:20] ...trap/engines/engine.py.224: train: epoch 100 | batch 1920/2304 [I 2022-10-28 02:16:20] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:32 [I 2022-10-28 02:16:20] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00243 [I 2022-10-28 02:16:20] ...trap/engines/engine.py.230: loss: 0.13218 [I 2022-10-28 02:16:20] ...optimizers/trijoint.py.151: total_norm: 36.229034 [I 2022-10-28 02:16:25] ...trap/engines/engine.py.224: train: epoch 100 | batch 1930/2304 [I 2022-10-28 02:16:25] ...trap/engines/engine.py.228: elapsed: 0:18:24 | left: 0:03:27 [I 2022-10-28 02:16:25] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00242 [I 2022-10-28 02:16:25] ...trap/engines/engine.py.230: loss: 0.11349 [I 2022-10-28 02:16:25] ...optimizers/trijoint.py.151: total_norm: 18.636648 [I 2022-10-28 02:16:31] ...trap/engines/engine.py.224: train: epoch 100 | batch 1940/2304 [I 2022-10-28 02:16:31] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:27 [I 2022-10-28 02:16:31] ...trap/engines/engine.py.229: process: 0.57452 | load: 0.00252 [I 2022-10-28 02:16:31] ...trap/engines/engine.py.230: loss: 0.15785 [I 2022-10-28 02:16:31] ...optimizers/trijoint.py.151: total_norm: 15.859950 [I 2022-10-28 02:16:37] ...trap/engines/engine.py.224: train: epoch 100 | batch 1950/2304 [I 2022-10-28 02:16:37] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:18 [I 2022-10-28 02:16:37] ...trap/engines/engine.py.229: process: 0.55295 | load: 0.00248 [I 2022-10-28 02:16:37] ...trap/engines/engine.py.230: loss: 0.15202 [I 2022-10-28 02:16:37] ...optimizers/trijoint.py.151: total_norm: 23.473074 [I 2022-10-28 02:16:42] ...trap/engines/engine.py.224: train: epoch 100 | batch 1960/2304 [I 2022-10-28 02:16:42] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:03:13 [I 2022-10-28 02:16:42] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00263 [I 2022-10-28 02:16:42] ...trap/engines/engine.py.230: loss: 0.11255 [I 2022-10-28 02:16:42] ...optimizers/trijoint.py.151: total_norm: 18.840349 [I 2022-10-28 02:16:48] ...trap/engines/engine.py.224: train: epoch 100 | batch 1970/2304 [I 2022-10-28 02:16:48] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:03:07 [I 2022-10-28 02:16:48] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00247 [I 2022-10-28 02:16:48] ...trap/engines/engine.py.230: loss: 0.12057 [I 2022-10-28 02:16:48] ...optimizers/trijoint.py.151: total_norm: 13.441314 [I 2022-10-28 02:16:54] ...trap/engines/engine.py.224: train: epoch 100 | batch 1980/2304 [I 2022-10-28 02:16:54] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:03:00 [I 2022-10-28 02:16:54] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00241 [I 2022-10-28 02:16:54] ...trap/engines/engine.py.230: loss: 0.11421 [I 2022-10-28 02:16:54] ...optimizers/trijoint.py.151: total_norm: 12.858459 [I 2022-10-28 02:16:59] ...trap/engines/engine.py.224: train: epoch 100 | batch 1990/2304 [I 2022-10-28 02:16:59] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:54 [I 2022-10-28 02:16:59] ...trap/engines/engine.py.229: process: 0.55147 | load: 0.00244 [I 2022-10-28 02:16:59] ...trap/engines/engine.py.230: loss: 0.08508 [I 2022-10-28 02:16:59] ...optimizers/trijoint.py.151: total_norm: 11.007160 [I 2022-10-28 02:17:05] ...trap/engines/engine.py.224: train: epoch 100 | batch 2000/2304 [I 2022-10-28 02:17:05] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:49 [I 2022-10-28 02:17:05] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00244 [I 2022-10-28 02:17:05] ...trap/engines/engine.py.230: loss: 0.12348 [I 2022-10-28 02:17:05] ...optimizers/trijoint.py.151: total_norm: 22.902845 [I 2022-10-28 02:17:10] ...trap/engines/engine.py.224: train: epoch 100 | batch 2010/2304 [I 2022-10-28 02:17:10] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:43 [I 2022-10-28 02:17:10] ...trap/engines/engine.py.229: process: 0.55356 | load: 0.00244 [I 2022-10-28 02:17:10] ...trap/engines/engine.py.230: loss: 0.13800 [I 2022-10-28 02:17:10] ...optimizers/trijoint.py.151: total_norm: 33.569267 [I 2022-10-28 02:17:16] ...trap/engines/engine.py.224: train: epoch 100 | batch 2020/2304 [I 2022-10-28 02:17:16] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:38 [I 2022-10-28 02:17:16] ...trap/engines/engine.py.229: process: 0.57147 | load: 0.00243 [I 2022-10-28 02:17:16] ...trap/engines/engine.py.230: loss: 0.10042 [I 2022-10-28 02:17:16] ...optimizers/trijoint.py.151: total_norm: 12.916945 [I 2022-10-28 02:17:21] ...trap/engines/engine.py.224: train: epoch 100 | batch 2030/2304 [I 2022-10-28 02:17:21] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:32 [I 2022-10-28 02:17:21] ...trap/engines/engine.py.229: process: 0.56861 | load: 0.00247 [I 2022-10-28 02:17:21] ...trap/engines/engine.py.230: loss: 0.16445 [I 2022-10-28 02:17:21] ...optimizers/trijoint.py.151: total_norm: 11.406356 [I 2022-10-28 02:17:27] ...trap/engines/engine.py.224: train: epoch 100 | batch 2040/2304 [I 2022-10-28 02:17:27] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:29 [I 2022-10-28 02:17:27] ...trap/engines/engine.py.229: process: 0.56102 | load: 0.00257 [I 2022-10-28 02:17:27] ...trap/engines/engine.py.230: loss: 0.07880 [I 2022-10-28 02:17:27] ...optimizers/trijoint.py.151: total_norm: 9.720283 [I 2022-10-28 02:17:33] ...trap/engines/engine.py.224: train: epoch 100 | batch 2050/2304 [I 2022-10-28 02:17:33] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:21 [I 2022-10-28 02:17:33] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00248 [I 2022-10-28 02:17:33] ...trap/engines/engine.py.230: loss: 0.13819 [I 2022-10-28 02:17:33] ...optimizers/trijoint.py.151: total_norm: 12.493335 [I 2022-10-28 02:17:38] ...trap/engines/engine.py.224: train: epoch 100 | batch 2060/2304 [I 2022-10-28 02:17:38] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:17 [I 2022-10-28 02:17:38] ...trap/engines/engine.py.229: process: 0.56359 | load: 0.00245 [I 2022-10-28 02:17:38] ...trap/engines/engine.py.230: loss: 0.11350 [I 2022-10-28 02:17:38] ...optimizers/trijoint.py.151: total_norm: 16.055975 [I 2022-10-28 02:17:44] ...trap/engines/engine.py.224: train: epoch 100 | batch 2070/2304 [I 2022-10-28 02:17:44] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:02:11 [I 2022-10-28 02:17:44] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00243 [I 2022-10-28 02:17:44] ...trap/engines/engine.py.230: loss: 0.10059 [I 2022-10-28 02:17:44] ...optimizers/trijoint.py.151: total_norm: 11.090951 [I 2022-10-28 02:17:50] ...trap/engines/engine.py.224: train: epoch 100 | batch 2080/2304 [I 2022-10-28 02:17:50] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:02:03 [I 2022-10-28 02:17:50] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00243 [I 2022-10-28 02:17:50] ...trap/engines/engine.py.230: loss: 0.14462 [I 2022-10-28 02:17:50] ...optimizers/trijoint.py.151: total_norm: 22.155586 [I 2022-10-28 02:17:55] ...trap/engines/engine.py.224: train: epoch 100 | batch 2090/2304 [I 2022-10-28 02:17:55] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:59 [I 2022-10-28 02:17:55] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00244 [I 2022-10-28 02:17:55] ...trap/engines/engine.py.230: loss: 0.13214 [I 2022-10-28 02:17:55] ...optimizers/trijoint.py.151: total_norm: 20.164619 [I 2022-10-28 02:18:01] ...trap/engines/engine.py.224: train: epoch 100 | batch 2100/2304 [I 2022-10-28 02:18:01] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:53 [I 2022-10-28 02:18:01] ...trap/engines/engine.py.229: process: 0.54791 | load: 0.00247 [I 2022-10-28 02:18:01] ...trap/engines/engine.py.230: loss: 0.12294 [I 2022-10-28 02:18:01] ...optimizers/trijoint.py.151: total_norm: 29.135799 [I 2022-10-28 02:18:06] ...trap/engines/engine.py.224: train: epoch 100 | batch 2110/2304 [I 2022-10-28 02:18:06] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:48 [I 2022-10-28 02:18:06] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00243 [I 2022-10-28 02:18:06] ...trap/engines/engine.py.230: loss: 0.13940 [I 2022-10-28 02:18:06] ...optimizers/trijoint.py.151: total_norm: 29.516684 [I 2022-10-28 02:18:12] ...trap/engines/engine.py.224: train: epoch 100 | batch 2120/2304 [I 2022-10-28 02:18:12] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:42 [I 2022-10-28 02:18:12] ...trap/engines/engine.py.229: process: 0.55277 | load: 0.00245 [I 2022-10-28 02:18:12] ...trap/engines/engine.py.230: loss: 0.13151 [I 2022-10-28 02:18:12] ...optimizers/trijoint.py.151: total_norm: 25.586573 [I 2022-10-28 02:18:18] ...trap/engines/engine.py.224: train: epoch 100 | batch 2130/2304 [I 2022-10-28 02:18:18] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:37 [I 2022-10-28 02:18:18] ...trap/engines/engine.py.229: process: 0.54843 | load: 0.00243 [I 2022-10-28 02:18:18] ...trap/engines/engine.py.230: loss: 0.10778 [I 2022-10-28 02:18:18] ...optimizers/trijoint.py.151: total_norm: 20.204884 [I 2022-10-28 02:18:23] ...trap/engines/engine.py.224: train: epoch 100 | batch 2140/2304 [I 2022-10-28 02:18:23] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:31 [I 2022-10-28 02:18:23] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00244 [I 2022-10-28 02:18:23] ...trap/engines/engine.py.230: loss: 0.11045 [I 2022-10-28 02:18:23] ...optimizers/trijoint.py.151: total_norm: 12.423041 [I 2022-10-28 02:18:29] ...trap/engines/engine.py.224: train: epoch 100 | batch 2150/2304 [I 2022-10-28 02:18:29] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:26 [I 2022-10-28 02:18:29] ...trap/engines/engine.py.229: process: 0.56976 | load: 0.00263 [I 2022-10-28 02:18:29] ...trap/engines/engine.py.230: loss: 0.10808 [I 2022-10-28 02:18:29] ...optimizers/trijoint.py.151: total_norm: 11.314252 [I 2022-10-28 02:18:34] ...trap/engines/engine.py.224: train: epoch 100 | batch 2160/2304 [I 2022-10-28 02:18:34] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:20 [I 2022-10-28 02:18:34] ...trap/engines/engine.py.229: process: 0.56450 | load: 0.00244 [I 2022-10-28 02:18:34] ...trap/engines/engine.py.230: loss: 0.09859 [I 2022-10-28 02:18:34] ...optimizers/trijoint.py.151: total_norm: 15.716873 [I 2022-10-28 02:18:40] ...trap/engines/engine.py.224: train: epoch 100 | batch 2170/2304 [I 2022-10-28 02:18:40] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:01:14 [I 2022-10-28 02:18:40] ...trap/engines/engine.py.229: process: 0.55618 | load: 0.00241 [I 2022-10-28 02:18:40] ...trap/engines/engine.py.230: loss: 0.10271 [I 2022-10-28 02:18:40] ...optimizers/trijoint.py.151: total_norm: 14.842196 [I 2022-10-28 02:18:45] ...trap/engines/engine.py.224: train: epoch 100 | batch 2180/2304 [I 2022-10-28 02:18:45] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:08 [I 2022-10-28 02:18:45] ...trap/engines/engine.py.229: process: 0.54667 | load: 0.00243 [I 2022-10-28 02:18:45] ...trap/engines/engine.py.230: loss: 0.11750 [I 2022-10-28 02:18:45] ...optimizers/trijoint.py.151: total_norm: 19.509417 [I 2022-10-28 02:18:51] ...trap/engines/engine.py.224: train: epoch 100 | batch 2190/2304 [I 2022-10-28 02:18:51] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:01:03 [I 2022-10-28 02:18:51] ...trap/engines/engine.py.229: process: 0.55353 | load: 0.00242 [I 2022-10-28 02:18:51] ...trap/engines/engine.py.230: loss: 0.08009 [I 2022-10-28 02:18:51] ...optimizers/trijoint.py.151: total_norm: 11.280578 [I 2022-10-28 02:18:57] ...trap/engines/engine.py.224: train: epoch 100 | batch 2200/2304 [I 2022-10-28 02:18:57] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:58 [I 2022-10-28 02:18:57] ...trap/engines/engine.py.229: process: 0.56996 | load: 0.00252 [I 2022-10-28 02:18:57] ...trap/engines/engine.py.230: loss: 0.12184 [I 2022-10-28 02:18:57] ...optimizers/trijoint.py.151: total_norm: 15.546333 [I 2022-10-28 02:19:02] ...trap/engines/engine.py.224: train: epoch 100 | batch 2210/2304 [I 2022-10-28 02:19:02] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:52 [I 2022-10-28 02:19:02] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00247 [I 2022-10-28 02:19:02] ...trap/engines/engine.py.230: loss: 0.10726 [I 2022-10-28 02:19:02] ...optimizers/trijoint.py.151: total_norm: 35.784878 [I 2022-10-28 02:19:08] ...trap/engines/engine.py.224: train: epoch 100 | batch 2220/2304 [I 2022-10-28 02:19:08] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:46 [I 2022-10-28 02:19:08] ...trap/engines/engine.py.229: process: 0.54967 | load: 0.00247 [I 2022-10-28 02:19:08] ...trap/engines/engine.py.230: loss: 0.08318 [I 2022-10-28 02:19:08] ...optimizers/trijoint.py.151: total_norm: 12.177843 [I 2022-10-28 02:19:14] ...trap/engines/engine.py.224: train: epoch 100 | batch 2230/2304 [I 2022-10-28 02:19:14] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:41 [I 2022-10-28 02:19:14] ...trap/engines/engine.py.229: process: 0.56320 | load: 0.00242 [I 2022-10-28 02:19:14] ...trap/engines/engine.py.230: loss: 0.11965 [I 2022-10-28 02:19:14] ...optimizers/trijoint.py.151: total_norm: 12.196633 [I 2022-10-28 02:19:19] ...trap/engines/engine.py.224: train: epoch 100 | batch 2240/2304 [I 2022-10-28 02:19:19] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:35 [I 2022-10-28 02:19:19] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00245 [I 2022-10-28 02:19:19] ...trap/engines/engine.py.230: loss: 0.12336 [I 2022-10-28 02:19:19] ...optimizers/trijoint.py.151: total_norm: 11.384977 [I 2022-10-28 02:19:25] ...trap/engines/engine.py.224: train: epoch 100 | batch 2250/2304 [I 2022-10-28 02:19:25] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:30 [I 2022-10-28 02:19:25] ...trap/engines/engine.py.229: process: 0.54786 | load: 0.00243 [I 2022-10-28 02:19:25] ...trap/engines/engine.py.230: loss: 0.10498 [I 2022-10-28 02:19:25] ...optimizers/trijoint.py.151: total_norm: 13.431749 [I 2022-10-28 02:19:30] ...trap/engines/engine.py.224: train: epoch 100 | batch 2260/2304 [I 2022-10-28 02:19:30] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:24 [I 2022-10-28 02:19:30] ...trap/engines/engine.py.229: process: 0.55824 | load: 0.00245 [I 2022-10-28 02:19:30] ...trap/engines/engine.py.230: loss: 0.12694 [I 2022-10-28 02:19:30] ...optimizers/trijoint.py.151: total_norm: 13.095400 [I 2022-10-28 02:19:36] ...trap/engines/engine.py.224: train: epoch 100 | batch 2270/2304 [I 2022-10-28 02:19:36] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:18 [I 2022-10-28 02:19:36] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00246 [I 2022-10-28 02:19:36] ...trap/engines/engine.py.230: loss: 0.20240 [I 2022-10-28 02:19:36] ...optimizers/trijoint.py.151: total_norm: 16.469883 [I 2022-10-28 02:19:42] ...trap/engines/engine.py.224: train: epoch 100 | batch 2280/2304 [I 2022-10-28 02:19:42] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:13 [I 2022-10-28 02:19:42] ...trap/engines/engine.py.229: process: 0.56376 | load: 0.00245 [I 2022-10-28 02:19:42] ...trap/engines/engine.py.230: loss: 0.15039 [I 2022-10-28 02:19:42] ...optimizers/trijoint.py.151: total_norm: 37.469536 [I 2022-10-28 02:19:47] ...trap/engines/engine.py.224: train: epoch 100 | batch 2290/2304 [I 2022-10-28 02:19:47] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:07 [I 2022-10-28 02:19:47] ...trap/engines/engine.py.229: process: 0.54480 | load: 0.00245 [I 2022-10-28 02:19:47] ...trap/engines/engine.py.230: loss: 0.11188 [I 2022-10-28 02:19:47] ...optimizers/trijoint.py.151: total_norm: 18.446486 [I 2022-10-28 02:19:53] ...trap/engines/engine.py.224: train: epoch 100 | batch 2300/2304 [I 2022-10-28 02:19:53] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:00:02 [I 2022-10-28 02:19:53] ...trap/engines/engine.py.229: process: 0.54664 | load: 0.00015 [I 2022-10-28 02:19:53] ...trap/engines/engine.py.230: loss: 0.13587 [I 2022-10-28 02:19:53] ...optimizers/trijoint.py.151: total_norm: 18.193333 [I 2022-10-28 02:19:55] ...trap/engines/engine.py.224: train: epoch 100 | batch 2304/2304 [I 2022-10-28 02:19:55] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:00:00 [I 2022-10-28 02:19:55] ...trap/engines/engine.py.229: process: 0.55175 | load: 0.00017 [I 2022-10-28 02:19:55] ...trap/engines/engine.py.230: loss: 0.09538 [I 2022-10-28 02:19:55] ...optimizers/trijoint.py.151: total_norm: 9.935387 [S 2022-10-28 02:19:56] ...trap/engines/engine.py.129: train_epoch.epoch: 100 [S 2022-10-28 02:19:56] ...trap/engines/engine.py.129: train_epoch.loss: 0.125539632458361 [I 2022-10-28 02:19:56] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 02:19:58] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 100 [I 2022-10-28 02:19:58] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 02:20:00] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 02:20:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 02:20:00] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 02:20:03] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 02:20:04] ...trap/engines/engine.py.321: eval: epoch 100 | batch 0/510 [I 2022-10-28 02:20:04] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:48:07 [I 2022-10-28 02:20:04] ...trap/engines/engine.py.326: process: 5.66151 | load: 4.99775 [I 2022-10-28 02:20:07] ...trap/engines/engine.py.321: eval: epoch 100 | batch 10/510 [I 2022-10-28 02:20:07] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:32 [I 2022-10-28 02:20:07] ...trap/engines/engine.py.326: process: 0.32774 | load: 0.00017 [I 2022-10-28 02:20:10] ...trap/engines/engine.py.321: eval: epoch 100 | batch 20/510 [I 2022-10-28 02:20:10] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:14 [I 2022-10-28 02:20:10] ...trap/engines/engine.py.326: process: 0.32971 | load: 0.00019 [I 2022-10-28 02:20:14] ...trap/engines/engine.py.321: eval: epoch 100 | batch 30/510 [I 2022-10-28 02:20:14] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:46 [I 2022-10-28 02:20:14] ...trap/engines/engine.py.326: process: 0.36060 | load: 0.00019 [I 2022-10-28 02:20:17] ...trap/engines/engine.py.321: eval: epoch 100 | batch 40/510 [I 2022-10-28 02:20:17] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:38 [I 2022-10-28 02:20:17] ...trap/engines/engine.py.326: process: 0.32087 | load: 0.00017 [I 2022-10-28 02:20:20] ...trap/engines/engine.py.321: eval: epoch 100 | batch 50/510 [I 2022-10-28 02:20:20] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:33 [I 2022-10-28 02:20:20] ...trap/engines/engine.py.326: process: 0.33370 | load: 0.00017 [I 2022-10-28 02:20:24] ...trap/engines/engine.py.321: eval: epoch 100 | batch 60/510 [I 2022-10-28 02:20:24] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:29 [I 2022-10-28 02:20:24] ...trap/engines/engine.py.326: process: 0.32936 | load: 0.00015 [I 2022-10-28 02:20:27] ...trap/engines/engine.py.321: eval: epoch 100 | batch 70/510 [I 2022-10-28 02:20:27] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:24 [I 2022-10-28 02:20:27] ...trap/engines/engine.py.326: process: 0.31930 | load: 0.00016 [I 2022-10-28 02:20:30] ...trap/engines/engine.py.321: eval: epoch 100 | batch 80/510 [I 2022-10-28 02:20:30] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:24 [I 2022-10-28 02:20:30] ...trap/engines/engine.py.326: process: 0.33187 | load: 0.00017 [I 2022-10-28 02:20:34] ...trap/engines/engine.py.321: eval: epoch 100 | batch 90/510 [I 2022-10-28 02:20:34] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-28 02:20:34] ...trap/engines/engine.py.326: process: 0.33021 | load: 0.00027 [I 2022-10-28 02:20:37] ...trap/engines/engine.py.321: eval: epoch 100 | batch 100/510 [I 2022-10-28 02:20:37] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-28 02:20:37] ...trap/engines/engine.py.326: process: 0.28721 | load: 0.00024 [I 2022-10-28 02:20:40] ...trap/engines/engine.py.321: eval: epoch 100 | batch 110/510 [I 2022-10-28 02:20:40] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:58 [I 2022-10-28 02:20:40] ...trap/engines/engine.py.326: process: 0.28826 | load: 0.00016 [I 2022-10-28 02:20:43] ...trap/engines/engine.py.321: eval: epoch 100 | batch 120/510 [I 2022-10-28 02:20:43] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:53 [I 2022-10-28 02:20:43] ...trap/engines/engine.py.326: process: 0.28931 | load: 0.00020 [I 2022-10-28 02:20:46] ...trap/engines/engine.py.321: eval: epoch 100 | batch 130/510 [I 2022-10-28 02:20:46] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-28 02:20:46] ...trap/engines/engine.py.326: process: 0.28295 | load: 0.00015 [I 2022-10-28 02:20:49] ...trap/engines/engine.py.321: eval: epoch 100 | batch 140/510 [I 2022-10-28 02:20:49] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:48 [I 2022-10-28 02:20:49] ...trap/engines/engine.py.326: process: 0.30338 | load: 0.00015 [I 2022-10-28 02:20:52] ...trap/engines/engine.py.321: eval: epoch 100 | batch 150/510 [I 2022-10-28 02:20:52] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-28 02:20:52] ...trap/engines/engine.py.326: process: 0.28262 | load: 0.00015 [I 2022-10-28 02:20:54] ...trap/engines/engine.py.321: eval: epoch 100 | batch 160/510 [I 2022-10-28 02:20:54] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-28 02:20:54] ...trap/engines/engine.py.326: process: 0.28751 | load: 0.00021 [I 2022-10-28 02:20:57] ...trap/engines/engine.py.321: eval: epoch 100 | batch 170/510 [I 2022-10-28 02:20:57] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:37 [I 2022-10-28 02:20:57] ...trap/engines/engine.py.326: process: 0.28091 | load: 0.00014 [I 2022-10-28 02:21:00] ...trap/engines/engine.py.321: eval: epoch 100 | batch 180/510 [I 2022-10-28 02:21:00] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-28 02:21:00] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00016 [I 2022-10-28 02:21:03] ...trap/engines/engine.py.321: eval: epoch 100 | batch 190/510 [I 2022-10-28 02:21:03] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-28 02:21:03] ...trap/engines/engine.py.326: process: 0.28879 | load: 0.00019 [I 2022-10-28 02:21:06] ...trap/engines/engine.py.321: eval: epoch 100 | batch 200/510 [I 2022-10-28 02:21:06] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:28 [I 2022-10-28 02:21:06] ...trap/engines/engine.py.326: process: 0.28620 | load: 0.00015 [I 2022-10-28 02:21:09] ...trap/engines/engine.py.321: eval: epoch 100 | batch 210/510 [I 2022-10-28 02:21:09] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:26 [I 2022-10-28 02:21:09] ...trap/engines/engine.py.326: process: 0.29208 | load: 0.00015 [I 2022-10-28 02:21:12] ...trap/engines/engine.py.321: eval: epoch 100 | batch 220/510 [I 2022-10-28 02:21:12] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:22 [I 2022-10-28 02:21:12] ...trap/engines/engine.py.326: process: 0.27997 | load: 0.00020 [I 2022-10-28 02:21:14] ...trap/engines/engine.py.321: eval: epoch 100 | batch 230/510 [I 2022-10-28 02:21:14] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:19 [I 2022-10-28 02:21:14] ...trap/engines/engine.py.326: process: 0.27942 | load: 0.00017 [I 2022-10-28 02:21:17] ...trap/engines/engine.py.321: eval: epoch 100 | batch 240/510 [I 2022-10-28 02:21:17] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 02:21:17] ...trap/engines/engine.py.326: process: 0.28853 | load: 0.00016 [I 2022-10-28 02:21:20] ...trap/engines/engine.py.321: eval: epoch 100 | batch 250/510 [I 2022-10-28 02:21:20] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-28 02:21:20] ...trap/engines/engine.py.326: process: 0.29070 | load: 0.00015 [I 2022-10-28 02:21:23] ...trap/engines/engine.py.321: eval: epoch 100 | batch 260/510 [I 2022-10-28 02:21:23] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 02:21:23] ...trap/engines/engine.py.326: process: 0.28534 | load: 0.00016 [I 2022-10-28 02:21:26] ...trap/engines/engine.py.321: eval: epoch 100 | batch 270/510 [I 2022-10-28 02:21:26] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-28 02:21:26] ...trap/engines/engine.py.326: process: 0.28245 | load: 0.00017 [I 2022-10-28 02:21:29] ...trap/engines/engine.py.321: eval: epoch 100 | batch 280/510 [I 2022-10-28 02:21:29] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 02:21:29] ...trap/engines/engine.py.326: process: 0.28745 | load: 0.00016 [I 2022-10-28 02:21:32] ...trap/engines/engine.py.321: eval: epoch 100 | batch 290/510 [I 2022-10-28 02:21:32] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-28 02:21:32] ...trap/engines/engine.py.326: process: 0.28764 | load: 0.00016 [I 2022-10-28 02:21:35] ...trap/engines/engine.py.321: eval: epoch 100 | batch 300/510 [I 2022-10-28 02:21:35] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-28 02:21:35] ...trap/engines/engine.py.326: process: 0.28918 | load: 0.00016 [I 2022-10-28 02:21:38] ...trap/engines/engine.py.321: eval: epoch 100 | batch 310/510 [I 2022-10-28 02:21:38] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 02:21:38] ...trap/engines/engine.py.326: process: 0.29040 | load: 0.00022 [I 2022-10-28 02:21:40] ...trap/engines/engine.py.321: eval: epoch 100 | batch 320/510 [I 2022-10-28 02:21:40] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 02:21:40] ...trap/engines/engine.py.326: process: 0.28692 | load: 0.00015 [I 2022-10-28 02:21:43] ...trap/engines/engine.py.321: eval: epoch 100 | batch 330/510 [I 2022-10-28 02:21:43] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-28 02:21:43] ...trap/engines/engine.py.326: process: 0.28075 | load: 0.00015 [I 2022-10-28 02:21:46] ...trap/engines/engine.py.321: eval: epoch 100 | batch 340/510 [I 2022-10-28 02:21:46] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-28 02:21:46] ...trap/engines/engine.py.326: process: 0.30599 | load: 0.00015 [I 2022-10-28 02:21:49] ...trap/engines/engine.py.321: eval: epoch 100 | batch 350/510 [I 2022-10-28 02:21:49] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 02:21:49] ...trap/engines/engine.py.326: process: 0.28825 | load: 0.00015 [I 2022-10-28 02:21:52] ...trap/engines/engine.py.321: eval: epoch 100 | batch 360/510 [I 2022-10-28 02:21:52] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 02:21:52] ...trap/engines/engine.py.326: process: 0.28879 | load: 0.00015 [I 2022-10-28 02:21:55] ...trap/engines/engine.py.321: eval: epoch 100 | batch 370/510 [I 2022-10-28 02:21:55] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 02:21:55] ...trap/engines/engine.py.326: process: 0.29407 | load: 0.00015 [I 2022-10-28 02:21:58] ...trap/engines/engine.py.321: eval: epoch 100 | batch 380/510 [I 2022-10-28 02:21:58] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-28 02:21:58] ...trap/engines/engine.py.326: process: 0.29323 | load: 0.00015 [I 2022-10-28 02:22:01] ...trap/engines/engine.py.321: eval: epoch 100 | batch 390/510 [I 2022-10-28 02:22:01] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 02:22:01] ...trap/engines/engine.py.326: process: 0.28876 | load: 0.00016 [I 2022-10-28 02:22:04] ...trap/engines/engine.py.321: eval: epoch 100 | batch 400/510 [I 2022-10-28 02:22:04] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 02:22:04] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00015 [I 2022-10-28 02:22:06] ...trap/engines/engine.py.321: eval: epoch 100 | batch 410/510 [I 2022-10-28 02:22:06] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 02:22:06] ...trap/engines/engine.py.326: process: 0.28620 | load: 0.00016 [I 2022-10-28 02:22:09] ...trap/engines/engine.py.321: eval: epoch 100 | batch 420/510 [I 2022-10-28 02:22:09] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 02:22:09] ...trap/engines/engine.py.326: process: 0.28887 | load: 0.00015 [I 2022-10-28 02:22:12] ...trap/engines/engine.py.321: eval: epoch 100 | batch 430/510 [I 2022-10-28 02:22:12] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 02:22:12] ...trap/engines/engine.py.326: process: 0.28981 | load: 0.00015 [I 2022-10-28 02:22:15] ...trap/engines/engine.py.321: eval: epoch 100 | batch 440/510 [I 2022-10-28 02:22:15] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 02:22:15] ...trap/engines/engine.py.326: process: 0.28844 | load: 0.00015 [I 2022-10-28 02:22:18] ...trap/engines/engine.py.321: eval: epoch 100 | batch 450/510 [I 2022-10-28 02:22:18] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 02:22:18] ...trap/engines/engine.py.326: process: 0.28960 | load: 0.00016 [I 2022-10-28 02:22:21] ...trap/engines/engine.py.321: eval: epoch 100 | batch 460/510 [I 2022-10-28 02:22:21] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 02:22:21] ...trap/engines/engine.py.326: process: 0.29329 | load: 0.00019 [I 2022-10-28 02:22:24] ...trap/engines/engine.py.321: eval: epoch 100 | batch 470/510 [I 2022-10-28 02:22:24] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 02:22:24] ...trap/engines/engine.py.326: process: 0.28915 | load: 0.00018 [I 2022-10-28 02:22:27] ...trap/engines/engine.py.321: eval: epoch 100 | batch 480/510 [I 2022-10-28 02:22:27] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 02:22:27] ...trap/engines/engine.py.326: process: 0.28593 | load: 0.00015 [I 2022-10-28 02:22:30] ...trap/engines/engine.py.321: eval: epoch 100 | batch 490/510 [I 2022-10-28 02:22:30] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 02:22:30] ...trap/engines/engine.py.326: process: 0.28799 | load: 0.00015 [I 2022-10-28 02:22:33] ...trap/engines/engine.py.321: eval: epoch 100 | batch 500/510 [I 2022-10-28 02:22:33] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 02:22:33] ...trap/engines/engine.py.326: process: 0.28669 | load: 0.00016 [I 2022-10-28 02:22:35] ...trap/engines/engine.py.321: eval: epoch 100 | batch 510/510 [I 2022-10-28 02:22:35] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 02:22:35] ...trap/engines/engine.py.326: process: 0.28585 | load: 0.00017 [S 2022-10-28 02:22:36] ...trap/engines/engine.py.132: eval_epoch.epoch: 100 [I 2022-10-28 02:22:36] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.739800 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.911600 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.938400 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.726400 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.909700 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937200 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017657 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011020 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007671 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015364 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010364 [S 2022-10-28 02:22:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007626 [I 2022-10-28 02:22:42] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 02:22:42] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 02:22:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 02:22:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 02:22:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 02:22:47] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 02:22:48] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 02:22:54] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 02:22:54] ...trap/engines/engine.py.163: Training model on trainset for epoch 101 [I 2022-10-28 02:22:54] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 02:22:57] ...trap/engines/engine.py.224: train: epoch 101 | batch 0/2304 [I 2022-10-28 02:22:57] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:01 [I 2022-10-28 02:22:57] ...trap/engines/engine.py.229: process: 2.65698 | load: 2.01221 [I 2022-10-28 02:22:57] ...trap/engines/engine.py.230: loss: 0.17220 [I 2022-10-28 02:22:57] ...optimizers/trijoint.py.151: total_norm: 20.893654 [I 2022-10-28 02:23:02] ...trap/engines/engine.py.224: train: epoch 101 | batch 10/2304 [I 2022-10-28 02:23:02] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:53 [I 2022-10-28 02:23:02] ...trap/engines/engine.py.229: process: 0.54645 | load: 0.00398 [I 2022-10-28 02:23:02] ...trap/engines/engine.py.230: loss: 0.12427 [I 2022-10-28 02:23:02] ...optimizers/trijoint.py.151: total_norm: 19.686760 [I 2022-10-28 02:23:08] ...trap/engines/engine.py.224: train: epoch 101 | batch 20/2304 [I 2022-10-28 02:23:08] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:11 [I 2022-10-28 02:23:08] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00377 [I 2022-10-28 02:23:08] ...trap/engines/engine.py.230: loss: 0.18217 [I 2022-10-28 02:23:08] ...optimizers/trijoint.py.151: total_norm: 19.072224 [I 2022-10-28 02:23:14] ...trap/engines/engine.py.224: train: epoch 101 | batch 30/2304 [I 2022-10-28 02:23:14] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:22 [I 2022-10-28 02:23:14] ...trap/engines/engine.py.229: process: 0.56701 | load: 0.00295 [I 2022-10-28 02:23:14] ...trap/engines/engine.py.230: loss: 0.09226 [I 2022-10-28 02:23:14] ...optimizers/trijoint.py.151: total_norm: 9.472460 [I 2022-10-28 02:23:19] ...trap/engines/engine.py.224: train: epoch 101 | batch 40/2304 [I 2022-10-28 02:23:19] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:05 [I 2022-10-28 02:23:19] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00273 [I 2022-10-28 02:23:19] ...trap/engines/engine.py.230: loss: 0.15307 [I 2022-10-28 02:23:19] ...optimizers/trijoint.py.151: total_norm: 35.127972 [I 2022-10-28 02:23:25] ...trap/engines/engine.py.224: train: epoch 101 | batch 50/2304 [I 2022-10-28 02:23:25] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:05 [I 2022-10-28 02:23:25] ...trap/engines/engine.py.229: process: 0.56322 | load: 0.00309 [I 2022-10-28 02:23:25] ...trap/engines/engine.py.230: loss: 0.14645 [I 2022-10-28 02:23:25] ...optimizers/trijoint.py.151: total_norm: 16.995087 [I 2022-10-28 02:23:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 60/2304 [I 2022-10-28 02:23:30] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:02 [I 2022-10-28 02:23:30] ...trap/engines/engine.py.229: process: 0.56626 | load: 0.00295 [I 2022-10-28 02:23:30] ...trap/engines/engine.py.230: loss: 0.11062 [I 2022-10-28 02:23:30] ...optimizers/trijoint.py.151: total_norm: 26.323471 [I 2022-10-28 02:23:36] ...trap/engines/engine.py.224: train: epoch 101 | batch 70/2304 [I 2022-10-28 02:23:36] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:37 [I 2022-10-28 02:23:36] ...trap/engines/engine.py.229: process: 0.54749 | load: 0.00271 [I 2022-10-28 02:23:36] ...trap/engines/engine.py.230: loss: 0.13173 [I 2022-10-28 02:23:36] ...optimizers/trijoint.py.151: total_norm: 30.833111 [I 2022-10-28 02:23:42] ...trap/engines/engine.py.224: train: epoch 101 | batch 80/2304 [I 2022-10-28 02:23:42] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:37 [I 2022-10-28 02:23:42] ...trap/engines/engine.py.229: process: 0.54932 | load: 0.00270 [I 2022-10-28 02:23:42] ...trap/engines/engine.py.230: loss: 0.16177 [I 2022-10-28 02:23:42] ...optimizers/trijoint.py.151: total_norm: 36.252796 [I 2022-10-28 02:23:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 90/2304 [I 2022-10-28 02:23:47] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:32 [I 2022-10-28 02:23:47] ...trap/engines/engine.py.229: process: 0.54861 | load: 0.00302 [I 2022-10-28 02:23:47] ...trap/engines/engine.py.230: loss: 0.20498 [I 2022-10-28 02:23:47] ...optimizers/trijoint.py.151: total_norm: 22.425619 [I 2022-10-28 02:23:53] ...trap/engines/engine.py.224: train: epoch 101 | batch 100/2304 [I 2022-10-28 02:23:53] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:35 [I 2022-10-28 02:23:53] ...trap/engines/engine.py.229: process: 0.56308 | load: 0.00281 [I 2022-10-28 02:23:53] ...trap/engines/engine.py.230: loss: 0.11470 [I 2022-10-28 02:23:53] ...optimizers/trijoint.py.151: total_norm: 14.211050 [I 2022-10-28 02:23:58] ...trap/engines/engine.py.224: train: epoch 101 | batch 110/2304 [I 2022-10-28 02:23:58] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:39 [I 2022-10-28 02:23:58] ...trap/engines/engine.py.229: process: 0.55534 | load: 0.00254 [I 2022-10-28 02:23:58] ...trap/engines/engine.py.230: loss: 0.16528 [I 2022-10-28 02:23:58] ...optimizers/trijoint.py.151: total_norm: 15.667559 [I 2022-10-28 02:24:04] ...trap/engines/engine.py.224: train: epoch 101 | batch 120/2304 [I 2022-10-28 02:24:04] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:16 [I 2022-10-28 02:24:04] ...trap/engines/engine.py.229: process: 0.55292 | load: 0.00282 [I 2022-10-28 02:24:04] ...trap/engines/engine.py.230: loss: 0.13204 [I 2022-10-28 02:24:04] ...optimizers/trijoint.py.151: total_norm: 14.750209 [I 2022-10-28 02:24:10] ...trap/engines/engine.py.224: train: epoch 101 | batch 130/2304 [I 2022-10-28 02:24:10] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:21 [I 2022-10-28 02:24:10] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00321 [I 2022-10-28 02:24:10] ...trap/engines/engine.py.230: loss: 0.12428 [I 2022-10-28 02:24:10] ...optimizers/trijoint.py.151: total_norm: 9.479887 [I 2022-10-28 02:24:15] ...trap/engines/engine.py.224: train: epoch 101 | batch 140/2304 [I 2022-10-28 02:24:15] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:05 [I 2022-10-28 02:24:15] ...trap/engines/engine.py.229: process: 0.55734 | load: 0.00276 [I 2022-10-28 02:24:15] ...trap/engines/engine.py.230: loss: 0.10474 [I 2022-10-28 02:24:15] ...optimizers/trijoint.py.151: total_norm: 8.865784 [I 2022-10-28 02:24:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 150/2304 [I 2022-10-28 02:24:21] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:02 [I 2022-10-28 02:24:21] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00271 [I 2022-10-28 02:24:21] ...trap/engines/engine.py.230: loss: 0.13321 [I 2022-10-28 02:24:21] ...optimizers/trijoint.py.151: total_norm: 16.785782 [I 2022-10-28 02:24:26] ...trap/engines/engine.py.224: train: epoch 101 | batch 160/2304 [I 2022-10-28 02:24:26] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:48 [I 2022-10-28 02:24:26] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00272 [I 2022-10-28 02:24:26] ...trap/engines/engine.py.230: loss: 0.14876 [I 2022-10-28 02:24:26] ...optimizers/trijoint.py.151: total_norm: 14.749898 [I 2022-10-28 02:24:32] ...trap/engines/engine.py.224: train: epoch 101 | batch 170/2304 [I 2022-10-28 02:24:32] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:48 [I 2022-10-28 02:24:32] ...trap/engines/engine.py.229: process: 0.54886 | load: 0.00292 [I 2022-10-28 02:24:32] ...trap/engines/engine.py.230: loss: 0.13736 [I 2022-10-28 02:24:32] ...optimizers/trijoint.py.151: total_norm: 16.526590 [I 2022-10-28 02:24:37] ...trap/engines/engine.py.224: train: epoch 101 | batch 180/2304 [I 2022-10-28 02:24:37] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:31 [I 2022-10-28 02:24:37] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00271 [I 2022-10-28 02:24:37] ...trap/engines/engine.py.230: loss: 0.11896 [I 2022-10-28 02:24:37] ...optimizers/trijoint.py.151: total_norm: 11.901695 [I 2022-10-28 02:24:43] ...trap/engines/engine.py.224: train: epoch 101 | batch 190/2304 [I 2022-10-28 02:24:43] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:35 [I 2022-10-28 02:24:43] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00281 [I 2022-10-28 02:24:43] ...trap/engines/engine.py.230: loss: 0.09747 [I 2022-10-28 02:24:43] ...optimizers/trijoint.py.151: total_norm: 11.825807 [I 2022-10-28 02:24:49] ...trap/engines/engine.py.224: train: epoch 101 | batch 200/2304 [I 2022-10-28 02:24:49] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:31 [I 2022-10-28 02:24:49] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00284 [I 2022-10-28 02:24:49] ...trap/engines/engine.py.230: loss: 0.09838 [I 2022-10-28 02:24:49] ...optimizers/trijoint.py.151: total_norm: 12.097014 [I 2022-10-28 02:24:54] ...trap/engines/engine.py.224: train: epoch 101 | batch 210/2304 [I 2022-10-28 02:24:54] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:27 [I 2022-10-28 02:24:54] ...trap/engines/engine.py.229: process: 0.55106 | load: 0.00269 [I 2022-10-28 02:24:54] ...trap/engines/engine.py.230: loss: 0.11412 [I 2022-10-28 02:24:54] ...optimizers/trijoint.py.151: total_norm: 130.359283 [I 2022-10-28 02:25:00] ...trap/engines/engine.py.224: train: epoch 101 | batch 220/2304 [I 2022-10-28 02:25:00] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:26 [I 2022-10-28 02:25:00] ...trap/engines/engine.py.229: process: 0.55482 | load: 0.00262 [I 2022-10-28 02:25:00] ...trap/engines/engine.py.230: loss: 0.08121 [I 2022-10-28 02:25:00] ...optimizers/trijoint.py.151: total_norm: 17.419895 [I 2022-10-28 02:25:05] ...trap/engines/engine.py.224: train: epoch 101 | batch 230/2304 [I 2022-10-28 02:25:05] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:06 [I 2022-10-28 02:25:05] ...trap/engines/engine.py.229: process: 0.54082 | load: 0.00257 [I 2022-10-28 02:25:05] ...trap/engines/engine.py.230: loss: 0.13183 [I 2022-10-28 02:25:05] ...optimizers/trijoint.py.151: total_norm: 16.046801 [I 2022-10-28 02:25:11] ...trap/engines/engine.py.224: train: epoch 101 | batch 240/2304 [I 2022-10-28 02:25:11] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:00 [I 2022-10-28 02:25:11] ...trap/engines/engine.py.229: process: 0.54116 | load: 0.00262 [I 2022-10-28 02:25:11] ...trap/engines/engine.py.230: loss: 0.10128 [I 2022-10-28 02:25:11] ...optimizers/trijoint.py.151: total_norm: 8.732344 [I 2022-10-28 02:25:16] ...trap/engines/engine.py.224: train: epoch 101 | batch 250/2304 [I 2022-10-28 02:25:16] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:46 [I 2022-10-28 02:25:16] ...trap/engines/engine.py.229: process: 0.54847 | load: 0.00263 [I 2022-10-28 02:25:16] ...trap/engines/engine.py.230: loss: 0.13216 [I 2022-10-28 02:25:16] ...optimizers/trijoint.py.151: total_norm: 14.485538 [I 2022-10-28 02:25:22] ...trap/engines/engine.py.224: train: epoch 101 | batch 260/2304 [I 2022-10-28 02:25:22] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:19:02 [I 2022-10-28 02:25:22] ...trap/engines/engine.py.229: process: 0.55391 | load: 0.00273 [I 2022-10-28 02:25:22] ...trap/engines/engine.py.230: loss: 0.10320 [I 2022-10-28 02:25:22] ...optimizers/trijoint.py.151: total_norm: 9.463001 [I 2022-10-28 02:25:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 270/2304 [I 2022-10-28 02:25:28] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:54 [I 2022-10-28 02:25:28] ...trap/engines/engine.py.229: process: 0.55676 | load: 0.00273 [I 2022-10-28 02:25:28] ...trap/engines/engine.py.230: loss: 0.09595 [I 2022-10-28 02:25:28] ...optimizers/trijoint.py.151: total_norm: 9.099511 [I 2022-10-28 02:25:33] ...trap/engines/engine.py.224: train: epoch 101 | batch 280/2304 [I 2022-10-28 02:25:33] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:37 [I 2022-10-28 02:25:33] ...trap/engines/engine.py.229: process: 0.55021 | load: 0.00245 [I 2022-10-28 02:25:33] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-28 02:25:33] ...optimizers/trijoint.py.151: total_norm: 13.165383 [I 2022-10-28 02:25:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 290/2304 [I 2022-10-28 02:25:39] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:42 [I 2022-10-28 02:25:39] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00251 [I 2022-10-28 02:25:39] ...trap/engines/engine.py.230: loss: 0.08615 [I 2022-10-28 02:25:39] ...optimizers/trijoint.py.151: total_norm: 11.465851 [I 2022-10-28 02:25:44] ...trap/engines/engine.py.224: train: epoch 101 | batch 300/2304 [I 2022-10-28 02:25:44] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:35 [I 2022-10-28 02:25:44] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00258 [I 2022-10-28 02:25:44] ...trap/engines/engine.py.230: loss: 0.07978 [I 2022-10-28 02:25:44] ...optimizers/trijoint.py.151: total_norm: 13.391103 [I 2022-10-28 02:25:50] ...trap/engines/engine.py.224: train: epoch 101 | batch 310/2304 [I 2022-10-28 02:25:50] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:25 [I 2022-10-28 02:25:50] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00257 [I 2022-10-28 02:25:50] ...trap/engines/engine.py.230: loss: 0.13048 [I 2022-10-28 02:25:50] ...optimizers/trijoint.py.151: total_norm: 16.785419 [I 2022-10-28 02:25:55] ...trap/engines/engine.py.224: train: epoch 101 | batch 320/2304 [I 2022-10-28 02:25:55] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:27 [I 2022-10-28 02:25:55] ...trap/engines/engine.py.229: process: 0.55787 | load: 0.00254 [I 2022-10-28 02:25:55] ...trap/engines/engine.py.230: loss: 0.09039 [I 2022-10-28 02:25:55] ...optimizers/trijoint.py.151: total_norm: 9.962182 [I 2022-10-28 02:26:01] ...trap/engines/engine.py.224: train: epoch 101 | batch 330/2304 [I 2022-10-28 02:26:01] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:08 [I 2022-10-28 02:26:01] ...trap/engines/engine.py.229: process: 0.54646 | load: 0.00242 [I 2022-10-28 02:26:01] ...trap/engines/engine.py.230: loss: 0.10668 [I 2022-10-28 02:26:01] ...optimizers/trijoint.py.151: total_norm: 9.200563 [I 2022-10-28 02:26:07] ...trap/engines/engine.py.224: train: epoch 101 | batch 340/2304 [I 2022-10-28 02:26:07] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:10 [I 2022-10-28 02:26:07] ...trap/engines/engine.py.229: process: 0.55216 | load: 0.00245 [I 2022-10-28 02:26:07] ...trap/engines/engine.py.230: loss: 0.08158 [I 2022-10-28 02:26:07] ...optimizers/trijoint.py.151: total_norm: 10.514002 [I 2022-10-28 02:26:12] ...trap/engines/engine.py.224: train: epoch 101 | batch 350/2304 [I 2022-10-28 02:26:12] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:01 [I 2022-10-28 02:26:12] ...trap/engines/engine.py.229: process: 0.54634 | load: 0.00241 [I 2022-10-28 02:26:12] ...trap/engines/engine.py.230: loss: 0.17144 [I 2022-10-28 02:26:12] ...optimizers/trijoint.py.151: total_norm: 12.236432 [I 2022-10-28 02:26:18] ...trap/engines/engine.py.224: train: epoch 101 | batch 360/2304 [I 2022-10-28 02:26:18] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:54 [I 2022-10-28 02:26:18] ...trap/engines/engine.py.229: process: 0.54938 | load: 0.00252 [I 2022-10-28 02:26:18] ...trap/engines/engine.py.230: loss: 0.10094 [I 2022-10-28 02:26:18] ...optimizers/trijoint.py.151: total_norm: 28.843767 [I 2022-10-28 02:26:23] ...trap/engines/engine.py.224: train: epoch 101 | batch 370/2304 [I 2022-10-28 02:26:23] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:49 [I 2022-10-28 02:26:23] ...trap/engines/engine.py.229: process: 0.54724 | load: 0.00253 [I 2022-10-28 02:26:23] ...trap/engines/engine.py.230: loss: 0.09565 [I 2022-10-28 02:26:23] ...optimizers/trijoint.py.151: total_norm: 16.201136 [I 2022-10-28 02:26:29] ...trap/engines/engine.py.224: train: epoch 101 | batch 380/2304 [I 2022-10-28 02:26:29] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:49 [I 2022-10-28 02:26:29] ...trap/engines/engine.py.229: process: 0.56225 | load: 0.00247 [I 2022-10-28 02:26:29] ...trap/engines/engine.py.230: loss: 0.11813 [I 2022-10-28 02:26:29] ...optimizers/trijoint.py.151: total_norm: 8.493728 [I 2022-10-28 02:26:34] ...trap/engines/engine.py.224: train: epoch 101 | batch 390/2304 [I 2022-10-28 02:26:34] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:38 [I 2022-10-28 02:26:34] ...trap/engines/engine.py.229: process: 0.54898 | load: 0.00250 [I 2022-10-28 02:26:34] ...trap/engines/engine.py.230: loss: 0.12480 [I 2022-10-28 02:26:34] ...optimizers/trijoint.py.151: total_norm: 23.059008 [I 2022-10-28 02:26:40] ...trap/engines/engine.py.224: train: epoch 101 | batch 400/2304 [I 2022-10-28 02:26:40] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:40 [I 2022-10-28 02:26:40] ...trap/engines/engine.py.229: process: 0.54991 | load: 0.00248 [I 2022-10-28 02:26:40] ...trap/engines/engine.py.230: loss: 0.11347 [I 2022-10-28 02:26:40] ...optimizers/trijoint.py.151: total_norm: 8.656709 [I 2022-10-28 02:26:46] ...trap/engines/engine.py.224: train: epoch 101 | batch 410/2304 [I 2022-10-28 02:26:46] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:33 [I 2022-10-28 02:26:46] ...trap/engines/engine.py.229: process: 0.55835 | load: 0.00247 [I 2022-10-28 02:26:46] ...trap/engines/engine.py.230: loss: 0.11483 [I 2022-10-28 02:26:46] ...optimizers/trijoint.py.151: total_norm: 12.707137 [I 2022-10-28 02:26:51] ...trap/engines/engine.py.224: train: epoch 101 | batch 420/2304 [I 2022-10-28 02:26:51] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:45 [I 2022-10-28 02:26:51] ...trap/engines/engine.py.229: process: 0.55594 | load: 0.00240 [I 2022-10-28 02:26:51] ...trap/engines/engine.py.230: loss: 0.11606 [I 2022-10-28 02:26:51] ...optimizers/trijoint.py.151: total_norm: 28.661123 [I 2022-10-28 02:26:57] ...trap/engines/engine.py.224: train: epoch 101 | batch 430/2304 [I 2022-10-28 02:26:57] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:23 [I 2022-10-28 02:26:57] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00241 [I 2022-10-28 02:26:57] ...trap/engines/engine.py.230: loss: 0.13262 [I 2022-10-28 02:26:57] ...optimizers/trijoint.py.151: total_norm: 16.835321 [I 2022-10-28 02:27:02] ...trap/engines/engine.py.224: train: epoch 101 | batch 440/2304 [I 2022-10-28 02:27:02] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:08 [I 2022-10-28 02:27:02] ...trap/engines/engine.py.229: process: 0.55316 | load: 0.00244 [I 2022-10-28 02:27:02] ...trap/engines/engine.py.230: loss: 0.09519 [I 2022-10-28 02:27:02] ...optimizers/trijoint.py.151: total_norm: 13.331763 [I 2022-10-28 02:27:08] ...trap/engines/engine.py.224: train: epoch 101 | batch 450/2304 [I 2022-10-28 02:27:08] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:09 [I 2022-10-28 02:27:08] ...trap/engines/engine.py.229: process: 0.55299 | load: 0.00243 [I 2022-10-28 02:27:08] ...trap/engines/engine.py.230: loss: 0.17304 [I 2022-10-28 02:27:08] ...optimizers/trijoint.py.151: total_norm: 14.896598 [I 2022-10-28 02:27:14] ...trap/engines/engine.py.224: train: epoch 101 | batch 460/2304 [I 2022-10-28 02:27:14] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:00 [I 2022-10-28 02:27:14] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00240 [I 2022-10-28 02:27:14] ...trap/engines/engine.py.230: loss: 0.12124 [I 2022-10-28 02:27:14] ...optimizers/trijoint.py.151: total_norm: 25.035383 [I 2022-10-28 02:27:19] ...trap/engines/engine.py.224: train: epoch 101 | batch 470/2304 [I 2022-10-28 02:27:19] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:57 [I 2022-10-28 02:27:19] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00249 [I 2022-10-28 02:27:19] ...trap/engines/engine.py.230: loss: 0.09060 [I 2022-10-28 02:27:19] ...optimizers/trijoint.py.151: total_norm: 14.079227 [I 2022-10-28 02:27:25] ...trap/engines/engine.py.224: train: epoch 101 | batch 480/2304 [I 2022-10-28 02:27:25] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:17:29 [I 2022-10-28 02:27:25] ...trap/engines/engine.py.229: process: 0.54135 | load: 0.00247 [I 2022-10-28 02:27:25] ...trap/engines/engine.py.230: loss: 0.14837 [I 2022-10-28 02:27:25] ...optimizers/trijoint.py.151: total_norm: 11.140471 [I 2022-10-28 02:27:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 490/2304 [I 2022-10-28 02:27:30] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:57 [I 2022-10-28 02:27:30] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00241 [I 2022-10-28 02:27:30] ...trap/engines/engine.py.230: loss: 0.11429 [I 2022-10-28 02:27:30] ...optimizers/trijoint.py.151: total_norm: 16.932564 [I 2022-10-28 02:27:36] ...trap/engines/engine.py.224: train: epoch 101 | batch 500/2304 [I 2022-10-28 02:27:36] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:43 [I 2022-10-28 02:27:36] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00242 [I 2022-10-28 02:27:36] ...trap/engines/engine.py.230: loss: 0.11982 [I 2022-10-28 02:27:36] ...optimizers/trijoint.py.151: total_norm: 27.349417 [I 2022-10-28 02:27:42] ...trap/engines/engine.py.224: train: epoch 101 | batch 510/2304 [I 2022-10-28 02:27:42] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:37 [I 2022-10-28 02:27:42] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00239 [I 2022-10-28 02:27:42] ...trap/engines/engine.py.230: loss: 0.11359 [I 2022-10-28 02:27:42] ...optimizers/trijoint.py.151: total_norm: 12.594383 [I 2022-10-28 02:27:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 520/2304 [I 2022-10-28 02:27:47] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:37 [I 2022-10-28 02:27:47] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00249 [I 2022-10-28 02:27:47] ...trap/engines/engine.py.230: loss: 0.09140 [I 2022-10-28 02:27:47] ...optimizers/trijoint.py.151: total_norm: 25.122698 [I 2022-10-28 02:27:53] ...trap/engines/engine.py.224: train: epoch 101 | batch 530/2304 [I 2022-10-28 02:27:53] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:24 [I 2022-10-28 02:27:53] ...trap/engines/engine.py.229: process: 0.54872 | load: 0.00249 [I 2022-10-28 02:27:53] ...trap/engines/engine.py.230: loss: 0.09434 [I 2022-10-28 02:27:53] ...optimizers/trijoint.py.151: total_norm: 15.915188 [I 2022-10-28 02:27:58] ...trap/engines/engine.py.224: train: epoch 101 | batch 540/2304 [I 2022-10-28 02:27:58] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:13 [I 2022-10-28 02:27:58] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00264 [I 2022-10-28 02:27:58] ...trap/engines/engine.py.230: loss: 0.12110 [I 2022-10-28 02:27:58] ...optimizers/trijoint.py.151: total_norm: 26.204874 [I 2022-10-28 02:28:04] ...trap/engines/engine.py.224: train: epoch 101 | batch 550/2304 [I 2022-10-28 02:28:04] ...trap/engines/engine.py.228: elapsed: 0:05:09 | left: 0:16:22 [I 2022-10-28 02:28:04] ...trap/engines/engine.py.229: process: 0.54664 | load: 0.00254 [I 2022-10-28 02:28:04] ...trap/engines/engine.py.230: loss: 0.11918 [I 2022-10-28 02:28:04] ...optimizers/trijoint.py.151: total_norm: 21.834883 [I 2022-10-28 02:28:10] ...trap/engines/engine.py.224: train: epoch 101 | batch 560/2304 [I 2022-10-28 02:28:10] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:12 [I 2022-10-28 02:28:10] ...trap/engines/engine.py.229: process: 0.56055 | load: 0.00246 [I 2022-10-28 02:28:10] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-28 02:28:10] ...optimizers/trijoint.py.151: total_norm: 13.730922 [I 2022-10-28 02:28:15] ...trap/engines/engine.py.224: train: epoch 101 | batch 570/2304 [I 2022-10-28 02:28:15] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:04 [I 2022-10-28 02:28:15] ...trap/engines/engine.py.229: process: 0.54684 | load: 0.00259 [I 2022-10-28 02:28:15] ...trap/engines/engine.py.230: loss: 0.10622 [I 2022-10-28 02:28:15] ...optimizers/trijoint.py.151: total_norm: 9.347246 [I 2022-10-28 02:28:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 580/2304 [I 2022-10-28 02:28:21] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:15:51 [I 2022-10-28 02:28:21] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00249 [I 2022-10-28 02:28:21] ...trap/engines/engine.py.230: loss: 0.17602 [I 2022-10-28 02:28:21] ...optimizers/trijoint.py.151: total_norm: 55.316536 [I 2022-10-28 02:28:26] ...trap/engines/engine.py.224: train: epoch 101 | batch 590/2304 [I 2022-10-28 02:28:26] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:57 [I 2022-10-28 02:28:26] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00264 [I 2022-10-28 02:28:26] ...trap/engines/engine.py.230: loss: 0.15291 [I 2022-10-28 02:28:26] ...optimizers/trijoint.py.151: total_norm: 17.180475 [I 2022-10-28 02:28:32] ...trap/engines/engine.py.224: train: epoch 101 | batch 600/2304 [I 2022-10-28 02:28:32] ...trap/engines/engine.py.228: elapsed: 0:05:37 | left: 0:15:59 [I 2022-10-28 02:28:32] ...trap/engines/engine.py.229: process: 0.56674 | load: 0.00257 [I 2022-10-28 02:28:32] ...trap/engines/engine.py.230: loss: 0.08730 [I 2022-10-28 02:28:32] ...optimizers/trijoint.py.151: total_norm: 17.535927 [I 2022-10-28 02:28:38] ...trap/engines/engine.py.224: train: epoch 101 | batch 610/2304 [I 2022-10-28 02:28:38] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:16:02 [I 2022-10-28 02:28:38] ...trap/engines/engine.py.229: process: 0.54741 | load: 0.00249 [I 2022-10-28 02:28:38] ...trap/engines/engine.py.230: loss: 0.11591 [I 2022-10-28 02:28:38] ...optimizers/trijoint.py.151: total_norm: 15.016171 [I 2022-10-28 02:28:43] ...trap/engines/engine.py.224: train: epoch 101 | batch 620/2304 [I 2022-10-28 02:28:43] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:46 [I 2022-10-28 02:28:43] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00255 [I 2022-10-28 02:28:43] ...trap/engines/engine.py.230: loss: 0.12346 [I 2022-10-28 02:28:43] ...optimizers/trijoint.py.151: total_norm: 20.927320 [I 2022-10-28 02:28:49] ...trap/engines/engine.py.224: train: epoch 101 | batch 630/2304 [I 2022-10-28 02:28:49] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:35 [I 2022-10-28 02:28:49] ...trap/engines/engine.py.229: process: 0.55842 | load: 0.00253 [I 2022-10-28 02:28:49] ...trap/engines/engine.py.230: loss: 0.11919 [I 2022-10-28 02:28:49] ...optimizers/trijoint.py.151: total_norm: 45.215309 [I 2022-10-28 02:28:54] ...trap/engines/engine.py.224: train: epoch 101 | batch 640/2304 [I 2022-10-28 02:28:54] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:29 [I 2022-10-28 02:28:54] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00247 [I 2022-10-28 02:28:54] ...trap/engines/engine.py.230: loss: 0.19109 [I 2022-10-28 02:28:54] ...optimizers/trijoint.py.151: total_norm: 8.454782 [I 2022-10-28 02:29:00] ...trap/engines/engine.py.224: train: epoch 101 | batch 650/2304 [I 2022-10-28 02:29:00] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:19 [I 2022-10-28 02:29:00] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00248 [I 2022-10-28 02:29:00] ...trap/engines/engine.py.230: loss: 0.12336 [I 2022-10-28 02:29:00] ...optimizers/trijoint.py.151: total_norm: 29.562256 [I 2022-10-28 02:29:06] ...trap/engines/engine.py.224: train: epoch 101 | batch 660/2304 [I 2022-10-28 02:29:06] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:06 [I 2022-10-28 02:29:06] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00248 [I 2022-10-28 02:29:06] ...trap/engines/engine.py.230: loss: 0.09385 [I 2022-10-28 02:29:06] ...optimizers/trijoint.py.151: total_norm: 10.061740 [I 2022-10-28 02:29:11] ...trap/engines/engine.py.224: train: epoch 101 | batch 670/2304 [I 2022-10-28 02:29:11] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:09 [I 2022-10-28 02:29:11] ...trap/engines/engine.py.229: process: 0.56123 | load: 0.00243 [I 2022-10-28 02:29:11] ...trap/engines/engine.py.230: loss: 0.14196 [I 2022-10-28 02:29:11] ...optimizers/trijoint.py.151: total_norm: 13.049611 [I 2022-10-28 02:29:17] ...trap/engines/engine.py.224: train: epoch 101 | batch 680/2304 [I 2022-10-28 02:29:17] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:07 [I 2022-10-28 02:29:17] ...trap/engines/engine.py.229: process: 0.53897 | load: 0.00245 [I 2022-10-28 02:29:17] ...trap/engines/engine.py.230: loss: 0.16614 [I 2022-10-28 02:29:17] ...optimizers/trijoint.py.151: total_norm: 20.714703 [I 2022-10-28 02:29:22] ...trap/engines/engine.py.224: train: epoch 101 | batch 690/2304 [I 2022-10-28 02:29:22] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:15:02 [I 2022-10-28 02:29:22] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00251 [I 2022-10-28 02:29:22] ...trap/engines/engine.py.230: loss: 0.08774 [I 2022-10-28 02:29:22] ...optimizers/trijoint.py.151: total_norm: 13.248878 [I 2022-10-28 02:29:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 700/2304 [I 2022-10-28 02:29:28] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:45 [I 2022-10-28 02:29:28] ...trap/engines/engine.py.229: process: 0.54117 | load: 0.00246 [I 2022-10-28 02:29:28] ...trap/engines/engine.py.230: loss: 0.07572 [I 2022-10-28 02:29:28] ...optimizers/trijoint.py.151: total_norm: 11.041763 [I 2022-10-28 02:29:34] ...trap/engines/engine.py.224: train: epoch 101 | batch 710/2304 [I 2022-10-28 02:29:34] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:48 [I 2022-10-28 02:29:34] ...trap/engines/engine.py.229: process: 0.56238 | load: 0.00246 [I 2022-10-28 02:29:34] ...trap/engines/engine.py.230: loss: 0.15686 [I 2022-10-28 02:29:34] ...optimizers/trijoint.py.151: total_norm: 9.716036 [I 2022-10-28 02:29:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 720/2304 [I 2022-10-28 02:29:39] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:43 [I 2022-10-28 02:29:39] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00252 [I 2022-10-28 02:29:39] ...trap/engines/engine.py.230: loss: 0.08948 [I 2022-10-28 02:29:39] ...optimizers/trijoint.py.151: total_norm: 20.294319 [I 2022-10-28 02:29:45] ...trap/engines/engine.py.224: train: epoch 101 | batch 730/2304 [I 2022-10-28 02:29:45] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:38 [I 2022-10-28 02:29:45] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00249 [I 2022-10-28 02:29:45] ...trap/engines/engine.py.230: loss: 0.09189 [I 2022-10-28 02:29:45] ...optimizers/trijoint.py.151: total_norm: 11.013747 [I 2022-10-28 02:29:50] ...trap/engines/engine.py.224: train: epoch 101 | batch 740/2304 [I 2022-10-28 02:29:50] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:58 [I 2022-10-28 02:29:50] ...trap/engines/engine.py.229: process: 0.57719 | load: 0.00255 [I 2022-10-28 02:29:50] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-28 02:29:50] ...optimizers/trijoint.py.151: total_norm: 16.017378 [I 2022-10-28 02:29:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 750/2304 [I 2022-10-28 02:29:56] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:24 [I 2022-10-28 02:29:56] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00252 [I 2022-10-28 02:29:56] ...trap/engines/engine.py.230: loss: 0.15969 [I 2022-10-28 02:29:56] ...optimizers/trijoint.py.151: total_norm: 34.725117 [I 2022-10-28 02:30:02] ...trap/engines/engine.py.224: train: epoch 101 | batch 760/2304 [I 2022-10-28 02:30:02] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:23 [I 2022-10-28 02:30:02] ...trap/engines/engine.py.229: process: 0.56377 | load: 0.00260 [I 2022-10-28 02:30:02] ...trap/engines/engine.py.230: loss: 0.15779 [I 2022-10-28 02:30:02] ...optimizers/trijoint.py.151: total_norm: 38.280830 [I 2022-10-28 02:30:07] ...trap/engines/engine.py.224: train: epoch 101 | batch 770/2304 [I 2022-10-28 02:30:07] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:13 [I 2022-10-28 02:30:07] ...trap/engines/engine.py.229: process: 0.55512 | load: 0.00250 [I 2022-10-28 02:30:07] ...trap/engines/engine.py.230: loss: 0.13664 [I 2022-10-28 02:30:07] ...optimizers/trijoint.py.151: total_norm: 26.152277 [I 2022-10-28 02:30:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 780/2304 [I 2022-10-28 02:30:13] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:06 [I 2022-10-28 02:30:13] ...trap/engines/engine.py.229: process: 0.55477 | load: 0.00245 [I 2022-10-28 02:30:13] ...trap/engines/engine.py.230: loss: 0.16199 [I 2022-10-28 02:30:13] ...optimizers/trijoint.py.151: total_norm: 29.962687 [I 2022-10-28 02:30:18] ...trap/engines/engine.py.224: train: epoch 101 | batch 790/2304 [I 2022-10-28 02:30:18] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:00 [I 2022-10-28 02:30:18] ...trap/engines/engine.py.229: process: 0.56636 | load: 0.00250 [I 2022-10-28 02:30:18] ...trap/engines/engine.py.230: loss: 0.12351 [I 2022-10-28 02:30:18] ...optimizers/trijoint.py.151: total_norm: 14.477766 [I 2022-10-28 02:30:24] ...trap/engines/engine.py.224: train: epoch 101 | batch 800/2304 [I 2022-10-28 02:30:24] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:14:37 [I 2022-10-28 02:30:24] ...trap/engines/engine.py.229: process: 0.69235 | load: 0.00246 [I 2022-10-28 02:30:24] ...trap/engines/engine.py.230: loss: 0.14461 [I 2022-10-28 02:30:24] ...optimizers/trijoint.py.151: total_norm: 18.560028 [I 2022-10-28 02:30:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 810/2304 [I 2022-10-28 02:30:30] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:14:00 [I 2022-10-28 02:30:30] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00244 [I 2022-10-28 02:30:30] ...trap/engines/engine.py.230: loss: 0.14989 [I 2022-10-28 02:30:30] ...optimizers/trijoint.py.151: total_norm: 16.584791 [I 2022-10-28 02:30:35] ...trap/engines/engine.py.224: train: epoch 101 | batch 820/2304 [I 2022-10-28 02:30:35] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:51 [I 2022-10-28 02:30:35] ...trap/engines/engine.py.229: process: 0.56112 | load: 0.00242 [I 2022-10-28 02:30:35] ...trap/engines/engine.py.230: loss: 0.11306 [I 2022-10-28 02:30:35] ...optimizers/trijoint.py.151: total_norm: 23.033642 [I 2022-10-28 02:30:41] ...trap/engines/engine.py.224: train: epoch 101 | batch 830/2304 [I 2022-10-28 02:30:41] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:44 [I 2022-10-28 02:30:41] ...trap/engines/engine.py.229: process: 0.56531 | load: 0.00246 [I 2022-10-28 02:30:41] ...trap/engines/engine.py.230: loss: 0.12737 [I 2022-10-28 02:30:41] ...optimizers/trijoint.py.151: total_norm: 13.737223 [I 2022-10-28 02:30:46] ...trap/engines/engine.py.224: train: epoch 101 | batch 840/2304 [I 2022-10-28 02:30:46] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:33 [I 2022-10-28 02:30:46] ...trap/engines/engine.py.229: process: 0.55228 | load: 0.00244 [I 2022-10-28 02:30:46] ...trap/engines/engine.py.230: loss: 0.14628 [I 2022-10-28 02:30:46] ...optimizers/trijoint.py.151: total_norm: 14.679262 [I 2022-10-28 02:30:52] ...trap/engines/engine.py.224: train: epoch 101 | batch 850/2304 [I 2022-10-28 02:30:52] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:25 [I 2022-10-28 02:30:52] ...trap/engines/engine.py.229: process: 0.55968 | load: 0.00249 [I 2022-10-28 02:30:52] ...trap/engines/engine.py.230: loss: 0.11099 [I 2022-10-28 02:30:52] ...optimizers/trijoint.py.151: total_norm: 9.767469 [I 2022-10-28 02:30:57] ...trap/engines/engine.py.224: train: epoch 101 | batch 860/2304 [I 2022-10-28 02:30:57] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:27 [I 2022-10-28 02:30:57] ...trap/engines/engine.py.229: process: 0.55639 | load: 0.00252 [I 2022-10-28 02:30:57] ...trap/engines/engine.py.230: loss: 0.13624 [I 2022-10-28 02:30:57] ...optimizers/trijoint.py.151: total_norm: 15.137287 [I 2022-10-28 02:31:03] ...trap/engines/engine.py.224: train: epoch 101 | batch 870/2304 [I 2022-10-28 02:31:03] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:22 [I 2022-10-28 02:31:03] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00250 [I 2022-10-28 02:31:03] ...trap/engines/engine.py.230: loss: 0.22411 [I 2022-10-28 02:31:03] ...optimizers/trijoint.py.151: total_norm: 26.323221 [I 2022-10-28 02:31:09] ...trap/engines/engine.py.224: train: epoch 101 | batch 880/2304 [I 2022-10-28 02:31:09] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:11 [I 2022-10-28 02:31:09] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00246 [I 2022-10-28 02:31:09] ...trap/engines/engine.py.230: loss: 0.08347 [I 2022-10-28 02:31:09] ...optimizers/trijoint.py.151: total_norm: 21.493002 [I 2022-10-28 02:31:14] ...trap/engines/engine.py.224: train: epoch 101 | batch 890/2304 [I 2022-10-28 02:31:14] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:12:59 [I 2022-10-28 02:31:14] ...trap/engines/engine.py.229: process: 0.55008 | load: 0.00246 [I 2022-10-28 02:31:14] ...trap/engines/engine.py.230: loss: 0.15152 [I 2022-10-28 02:31:14] ...optimizers/trijoint.py.151: total_norm: 18.315401 [I 2022-10-28 02:31:20] ...trap/engines/engine.py.224: train: epoch 101 | batch 900/2304 [I 2022-10-28 02:31:20] ...trap/engines/engine.py.228: elapsed: 0:08:25 | left: 0:13:00 [I 2022-10-28 02:31:20] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00250 [I 2022-10-28 02:31:20] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-28 02:31:20] ...optimizers/trijoint.py.151: total_norm: 13.830520 [I 2022-10-28 02:31:25] ...trap/engines/engine.py.224: train: epoch 101 | batch 910/2304 [I 2022-10-28 02:31:25] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:13:01 [I 2022-10-28 02:31:25] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00253 [I 2022-10-28 02:31:25] ...trap/engines/engine.py.230: loss: 0.13963 [I 2022-10-28 02:31:25] ...optimizers/trijoint.py.151: total_norm: 33.265091 [I 2022-10-28 02:31:31] ...trap/engines/engine.py.224: train: epoch 101 | batch 920/2304 [I 2022-10-28 02:31:31] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:45 [I 2022-10-28 02:31:31] ...trap/engines/engine.py.229: process: 0.55942 | load: 0.00284 [I 2022-10-28 02:31:31] ...trap/engines/engine.py.230: loss: 0.11779 [I 2022-10-28 02:31:31] ...optimizers/trijoint.py.151: total_norm: 13.441852 [I 2022-10-28 02:31:37] ...trap/engines/engine.py.224: train: epoch 101 | batch 930/2304 [I 2022-10-28 02:31:37] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:57 [I 2022-10-28 02:31:37] ...trap/engines/engine.py.229: process: 0.56297 | load: 0.00252 [I 2022-10-28 02:31:37] ...trap/engines/engine.py.230: loss: 0.08134 [I 2022-10-28 02:31:37] ...optimizers/trijoint.py.151: total_norm: 12.810059 [I 2022-10-28 02:31:42] ...trap/engines/engine.py.224: train: epoch 101 | batch 940/2304 [I 2022-10-28 02:31:42] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:44 [I 2022-10-28 02:31:42] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00257 [I 2022-10-28 02:31:42] ...trap/engines/engine.py.230: loss: 0.16658 [I 2022-10-28 02:31:42] ...optimizers/trijoint.py.151: total_norm: 22.421555 [I 2022-10-28 02:31:48] ...trap/engines/engine.py.224: train: epoch 101 | batch 950/2304 [I 2022-10-28 02:31:48] ...trap/engines/engine.py.228: elapsed: 0:08:53 | left: 0:12:27 [I 2022-10-28 02:31:48] ...trap/engines/engine.py.229: process: 0.54375 | load: 0.00251 [I 2022-10-28 02:31:48] ...trap/engines/engine.py.230: loss: 0.11950 [I 2022-10-28 02:31:48] ...optimizers/trijoint.py.151: total_norm: 14.560636 [I 2022-10-28 02:31:53] ...trap/engines/engine.py.224: train: epoch 101 | batch 960/2304 [I 2022-10-28 02:31:53] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:28 [I 2022-10-28 02:31:53] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00258 [I 2022-10-28 02:31:53] ...trap/engines/engine.py.230: loss: 0.13780 [I 2022-10-28 02:31:53] ...optimizers/trijoint.py.151: total_norm: 36.499348 [I 2022-10-28 02:31:59] ...trap/engines/engine.py.224: train: epoch 101 | batch 970/2304 [I 2022-10-28 02:31:59] ...trap/engines/engine.py.228: elapsed: 0:09:04 | left: 0:12:22 [I 2022-10-28 02:31:59] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00251 [I 2022-10-28 02:31:59] ...trap/engines/engine.py.230: loss: 0.07547 [I 2022-10-28 02:31:59] ...optimizers/trijoint.py.151: total_norm: 10.515296 [I 2022-10-28 02:32:05] ...trap/engines/engine.py.224: train: epoch 101 | batch 980/2304 [I 2022-10-28 02:32:05] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:21 [I 2022-10-28 02:32:05] ...trap/engines/engine.py.229: process: 0.56446 | load: 0.00245 [I 2022-10-28 02:32:05] ...trap/engines/engine.py.230: loss: 0.10847 [I 2022-10-28 02:32:05] ...optimizers/trijoint.py.151: total_norm: 16.412771 [I 2022-10-28 02:32:10] ...trap/engines/engine.py.224: train: epoch 101 | batch 990/2304 [I 2022-10-28 02:32:10] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:12 [I 2022-10-28 02:32:10] ...trap/engines/engine.py.229: process: 0.56411 | load: 0.00246 [I 2022-10-28 02:32:10] ...trap/engines/engine.py.230: loss: 0.19950 [I 2022-10-28 02:32:10] ...optimizers/trijoint.py.151: total_norm: 20.768152 [I 2022-10-28 02:32:16] ...trap/engines/engine.py.224: train: epoch 101 | batch 1000/2304 [I 2022-10-28 02:32:16] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:07 [I 2022-10-28 02:32:16] ...trap/engines/engine.py.229: process: 0.56221 | load: 0.00248 [I 2022-10-28 02:32:16] ...trap/engines/engine.py.230: loss: 0.11518 [I 2022-10-28 02:32:16] ...optimizers/trijoint.py.151: total_norm: 21.365023 [I 2022-10-28 02:32:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 1010/2304 [I 2022-10-28 02:32:21] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:02 [I 2022-10-28 02:32:21] ...trap/engines/engine.py.229: process: 0.56773 | load: 0.00240 [I 2022-10-28 02:32:21] ...trap/engines/engine.py.230: loss: 0.13326 [I 2022-10-28 02:32:21] ...optimizers/trijoint.py.151: total_norm: 11.719625 [I 2022-10-28 02:32:27] ...trap/engines/engine.py.224: train: epoch 101 | batch 1020/2304 [I 2022-10-28 02:32:27] ...trap/engines/engine.py.228: elapsed: 0:09:32 | left: 0:11:48 [I 2022-10-28 02:32:27] ...trap/engines/engine.py.229: process: 0.54976 | load: 0.00244 [I 2022-10-28 02:32:27] ...trap/engines/engine.py.230: loss: 0.14451 [I 2022-10-28 02:32:27] ...optimizers/trijoint.py.151: total_norm: 24.061882 [I 2022-10-28 02:32:32] ...trap/engines/engine.py.224: train: epoch 101 | batch 1030/2304 [I 2022-10-28 02:32:32] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:48 [I 2022-10-28 02:32:32] ...trap/engines/engine.py.229: process: 0.55858 | load: 0.00246 [I 2022-10-28 02:32:32] ...trap/engines/engine.py.230: loss: 0.14744 [I 2022-10-28 02:32:32] ...optimizers/trijoint.py.151: total_norm: 16.207254 [I 2022-10-28 02:32:38] ...trap/engines/engine.py.224: train: epoch 101 | batch 1040/2304 [I 2022-10-28 02:32:38] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:11:44 [I 2022-10-28 02:32:38] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00243 [I 2022-10-28 02:32:38] ...trap/engines/engine.py.230: loss: 0.16127 [I 2022-10-28 02:32:38] ...optimizers/trijoint.py.151: total_norm: 9.844703 [I 2022-10-28 02:32:44] ...trap/engines/engine.py.224: train: epoch 101 | batch 1050/2304 [I 2022-10-28 02:32:44] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:38 [I 2022-10-28 02:32:44] ...trap/engines/engine.py.229: process: 0.55043 | load: 0.00250 [I 2022-10-28 02:32:44] ...trap/engines/engine.py.230: loss: 0.11893 [I 2022-10-28 02:32:44] ...optimizers/trijoint.py.151: total_norm: 22.797306 [I 2022-10-28 02:32:49] ...trap/engines/engine.py.224: train: epoch 101 | batch 1060/2304 [I 2022-10-28 02:32:49] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:32 [I 2022-10-28 02:32:49] ...trap/engines/engine.py.229: process: 0.55779 | load: 0.00242 [I 2022-10-28 02:32:49] ...trap/engines/engine.py.230: loss: 0.11422 [I 2022-10-28 02:32:49] ...optimizers/trijoint.py.151: total_norm: 15.744115 [I 2022-10-28 02:32:55] ...trap/engines/engine.py.224: train: epoch 101 | batch 1070/2304 [I 2022-10-28 02:32:55] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:11:33 [I 2022-10-28 02:32:55] ...trap/engines/engine.py.229: process: 0.56996 | load: 0.00245 [I 2022-10-28 02:32:55] ...trap/engines/engine.py.230: loss: 0.10388 [I 2022-10-28 02:32:55] ...optimizers/trijoint.py.151: total_norm: 11.295194 [I 2022-10-28 02:33:00] ...trap/engines/engine.py.224: train: epoch 101 | batch 1080/2304 [I 2022-10-28 02:33:00] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:25 [I 2022-10-28 02:33:00] ...trap/engines/engine.py.229: process: 0.55819 | load: 0.00252 [I 2022-10-28 02:33:00] ...trap/engines/engine.py.230: loss: 0.10614 [I 2022-10-28 02:33:00] ...optimizers/trijoint.py.151: total_norm: 16.468048 [I 2022-10-28 02:33:06] ...trap/engines/engine.py.224: train: epoch 101 | batch 1090/2304 [I 2022-10-28 02:33:06] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:16 [I 2022-10-28 02:33:06] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00261 [I 2022-10-28 02:33:06] ...trap/engines/engine.py.230: loss: 0.10504 [I 2022-10-28 02:33:06] ...optimizers/trijoint.py.151: total_norm: 8.286802 [I 2022-10-28 02:33:12] ...trap/engines/engine.py.224: train: epoch 101 | batch 1100/2304 [I 2022-10-28 02:33:12] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:13 [I 2022-10-28 02:33:12] ...trap/engines/engine.py.229: process: 0.54859 | load: 0.00252 [I 2022-10-28 02:33:12] ...trap/engines/engine.py.230: loss: 0.10189 [I 2022-10-28 02:33:12] ...optimizers/trijoint.py.151: total_norm: 12.276361 [I 2022-10-28 02:33:17] ...trap/engines/engine.py.224: train: epoch 101 | batch 1110/2304 [I 2022-10-28 02:33:17] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:05 [I 2022-10-28 02:33:17] ...trap/engines/engine.py.229: process: 0.55335 | load: 0.00254 [I 2022-10-28 02:33:17] ...trap/engines/engine.py.230: loss: 0.08053 [I 2022-10-28 02:33:17] ...optimizers/trijoint.py.151: total_norm: 9.502020 [I 2022-10-28 02:33:23] ...trap/engines/engine.py.224: train: epoch 101 | batch 1120/2304 [I 2022-10-28 02:33:23] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:05 [I 2022-10-28 02:33:23] ...trap/engines/engine.py.229: process: 0.56263 | load: 0.00242 [I 2022-10-28 02:33:23] ...trap/engines/engine.py.230: loss: 0.15887 [I 2022-10-28 02:33:23] ...optimizers/trijoint.py.151: total_norm: 96.414139 [I 2022-10-28 02:33:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 1130/2304 [I 2022-10-28 02:33:28] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:48 [I 2022-10-28 02:33:28] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00249 [I 2022-10-28 02:33:28] ...trap/engines/engine.py.230: loss: 0.12707 [I 2022-10-28 02:33:28] ...optimizers/trijoint.py.151: total_norm: 17.136360 [I 2022-10-28 02:33:34] ...trap/engines/engine.py.224: train: epoch 101 | batch 1140/2304 [I 2022-10-28 02:33:34] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:10:49 [I 2022-10-28 02:33:34] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00247 [I 2022-10-28 02:33:34] ...trap/engines/engine.py.230: loss: 0.12449 [I 2022-10-28 02:33:34] ...optimizers/trijoint.py.151: total_norm: 22.165525 [I 2022-10-28 02:33:40] ...trap/engines/engine.py.224: train: epoch 101 | batch 1150/2304 [I 2022-10-28 02:33:40] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:44 [I 2022-10-28 02:33:40] ...trap/engines/engine.py.229: process: 0.55576 | load: 0.00249 [I 2022-10-28 02:33:40] ...trap/engines/engine.py.230: loss: 0.13917 [I 2022-10-28 02:33:40] ...optimizers/trijoint.py.151: total_norm: 18.157486 [I 2022-10-28 02:33:45] ...trap/engines/engine.py.224: train: epoch 101 | batch 1160/2304 [I 2022-10-28 02:33:45] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:39 [I 2022-10-28 02:33:45] ...trap/engines/engine.py.229: process: 0.55667 | load: 0.00247 [I 2022-10-28 02:33:45] ...trap/engines/engine.py.230: loss: 0.13103 [I 2022-10-28 02:33:45] ...optimizers/trijoint.py.151: total_norm: 13.832314 [I 2022-10-28 02:33:51] ...trap/engines/engine.py.224: train: epoch 101 | batch 1170/2304 [I 2022-10-28 02:33:51] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:30 [I 2022-10-28 02:33:51] ...trap/engines/engine.py.229: process: 0.54243 | load: 0.00247 [I 2022-10-28 02:33:51] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-28 02:33:51] ...optimizers/trijoint.py.151: total_norm: 9.767743 [I 2022-10-28 02:33:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 1180/2304 [I 2022-10-28 02:33:56] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:26 [I 2022-10-28 02:33:56] ...trap/engines/engine.py.229: process: 0.56552 | load: 0.00258 [I 2022-10-28 02:33:56] ...trap/engines/engine.py.230: loss: 0.13121 [I 2022-10-28 02:33:56] ...optimizers/trijoint.py.151: total_norm: 22.121799 [I 2022-10-28 02:34:02] ...trap/engines/engine.py.224: train: epoch 101 | batch 1190/2304 [I 2022-10-28 02:34:02] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:22 [I 2022-10-28 02:34:02] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00254 [I 2022-10-28 02:34:02] ...trap/engines/engine.py.230: loss: 0.09142 [I 2022-10-28 02:34:02] ...optimizers/trijoint.py.151: total_norm: 19.785303 [I 2022-10-28 02:34:07] ...trap/engines/engine.py.224: train: epoch 101 | batch 1200/2304 [I 2022-10-28 02:34:07] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:12 [I 2022-10-28 02:34:07] ...trap/engines/engine.py.229: process: 0.55478 | load: 0.00250 [I 2022-10-28 02:34:07] ...trap/engines/engine.py.230: loss: 0.10317 [I 2022-10-28 02:34:07] ...optimizers/trijoint.py.151: total_norm: 13.257047 [I 2022-10-28 02:34:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 1210/2304 [I 2022-10-28 02:34:13] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:05 [I 2022-10-28 02:34:13] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00258 [I 2022-10-28 02:34:13] ...trap/engines/engine.py.230: loss: 0.12120 [I 2022-10-28 02:34:13] ...optimizers/trijoint.py.151: total_norm: 15.763514 [I 2022-10-28 02:34:19] ...trap/engines/engine.py.224: train: epoch 101 | batch 1220/2304 [I 2022-10-28 02:34:19] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:01 [I 2022-10-28 02:34:19] ...trap/engines/engine.py.229: process: 0.55300 | load: 0.00251 [I 2022-10-28 02:34:19] ...trap/engines/engine.py.230: loss: 0.12736 [I 2022-10-28 02:34:19] ...optimizers/trijoint.py.151: total_norm: 26.924892 [I 2022-10-28 02:34:24] ...trap/engines/engine.py.224: train: epoch 101 | batch 1230/2304 [I 2022-10-28 02:34:24] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:57 [I 2022-10-28 02:34:24] ...trap/engines/engine.py.229: process: 0.54571 | load: 0.00249 [I 2022-10-28 02:34:24] ...trap/engines/engine.py.230: loss: 0.11183 [I 2022-10-28 02:34:24] ...optimizers/trijoint.py.151: total_norm: 11.944623 [I 2022-10-28 02:34:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 1240/2304 [I 2022-10-28 02:34:30] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:09:54 [I 2022-10-28 02:34:30] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00260 [I 2022-10-28 02:34:30] ...trap/engines/engine.py.230: loss: 0.10254 [I 2022-10-28 02:34:30] ...optimizers/trijoint.py.151: total_norm: 9.102411 [I 2022-10-28 02:34:35] ...trap/engines/engine.py.224: train: epoch 101 | batch 1250/2304 [I 2022-10-28 02:34:35] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:47 [I 2022-10-28 02:34:35] ...trap/engines/engine.py.229: process: 0.54935 | load: 0.00254 [I 2022-10-28 02:34:35] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-28 02:34:35] ...optimizers/trijoint.py.151: total_norm: 50.749073 [I 2022-10-28 02:34:41] ...trap/engines/engine.py.224: train: epoch 101 | batch 1260/2304 [I 2022-10-28 02:34:41] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:09:43 [I 2022-10-28 02:34:41] ...trap/engines/engine.py.229: process: 0.57129 | load: 0.00254 [I 2022-10-28 02:34:41] ...trap/engines/engine.py.230: loss: 0.10167 [I 2022-10-28 02:34:41] ...optimizers/trijoint.py.151: total_norm: 14.404205 [I 2022-10-28 02:34:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 1270/2304 [I 2022-10-28 02:34:47] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:37 [I 2022-10-28 02:34:47] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00253 [I 2022-10-28 02:34:47] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-28 02:34:47] ...optimizers/trijoint.py.151: total_norm: 21.647856 [I 2022-10-28 02:34:52] ...trap/engines/engine.py.224: train: epoch 101 | batch 1280/2304 [I 2022-10-28 02:34:52] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:30 [I 2022-10-28 02:34:52] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00249 [I 2022-10-28 02:34:52] ...trap/engines/engine.py.230: loss: 0.11488 [I 2022-10-28 02:34:52] ...optimizers/trijoint.py.151: total_norm: 11.710986 [I 2022-10-28 02:34:58] ...trap/engines/engine.py.224: train: epoch 101 | batch 1290/2304 [I 2022-10-28 02:34:58] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:23 [I 2022-10-28 02:34:58] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00251 [I 2022-10-28 02:34:58] ...trap/engines/engine.py.230: loss: 0.08792 [I 2022-10-28 02:34:58] ...optimizers/trijoint.py.151: total_norm: 15.536136 [I 2022-10-28 02:35:03] ...trap/engines/engine.py.224: train: epoch 101 | batch 1300/2304 [I 2022-10-28 02:35:03] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:18 [I 2022-10-28 02:35:03] ...trap/engines/engine.py.229: process: 0.56438 | load: 0.00247 [I 2022-10-28 02:35:03] ...trap/engines/engine.py.230: loss: 0.09552 [I 2022-10-28 02:35:03] ...optimizers/trijoint.py.151: total_norm: 15.524988 [I 2022-10-28 02:35:09] ...trap/engines/engine.py.224: train: epoch 101 | batch 1310/2304 [I 2022-10-28 02:35:09] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:11 [I 2022-10-28 02:35:09] ...trap/engines/engine.py.229: process: 0.56187 | load: 0.00243 [I 2022-10-28 02:35:09] ...trap/engines/engine.py.230: loss: 0.15315 [I 2022-10-28 02:35:09] ...optimizers/trijoint.py.151: total_norm: 15.094902 [I 2022-10-28 02:35:14] ...trap/engines/engine.py.224: train: epoch 101 | batch 1320/2304 [I 2022-10-28 02:35:14] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:05 [I 2022-10-28 02:35:14] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00244 [I 2022-10-28 02:35:14] ...trap/engines/engine.py.230: loss: 0.09768 [I 2022-10-28 02:35:14] ...optimizers/trijoint.py.151: total_norm: 13.100052 [I 2022-10-28 02:35:20] ...trap/engines/engine.py.224: train: epoch 101 | batch 1330/2304 [I 2022-10-28 02:35:20] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:02 [I 2022-10-28 02:35:20] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00243 [I 2022-10-28 02:35:20] ...trap/engines/engine.py.230: loss: 0.10966 [I 2022-10-28 02:35:20] ...optimizers/trijoint.py.151: total_norm: 17.932755 [I 2022-10-28 02:35:25] ...trap/engines/engine.py.224: train: epoch 101 | batch 1340/2304 [I 2022-10-28 02:35:25] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:08:55 [I 2022-10-28 02:35:25] ...trap/engines/engine.py.229: process: 0.55024 | load: 0.00242 [I 2022-10-28 02:35:25] ...trap/engines/engine.py.230: loss: 0.08899 [I 2022-10-28 02:35:25] ...optimizers/trijoint.py.151: total_norm: 9.079803 [I 2022-10-28 02:35:31] ...trap/engines/engine.py.224: train: epoch 101 | batch 1350/2304 [I 2022-10-28 02:35:31] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:50 [I 2022-10-28 02:35:31] ...trap/engines/engine.py.229: process: 0.56719 | load: 0.00247 [I 2022-10-28 02:35:31] ...trap/engines/engine.py.230: loss: 0.10875 [I 2022-10-28 02:35:31] ...optimizers/trijoint.py.151: total_norm: 10.163321 [I 2022-10-28 02:35:37] ...trap/engines/engine.py.224: train: epoch 101 | batch 1360/2304 [I 2022-10-28 02:35:37] ...trap/engines/engine.py.228: elapsed: 0:12:42 | left: 0:08:44 [I 2022-10-28 02:35:37] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00244 [I 2022-10-28 02:35:37] ...trap/engines/engine.py.230: loss: 0.12055 [I 2022-10-28 02:35:37] ...optimizers/trijoint.py.151: total_norm: 11.014061 [I 2022-10-28 02:35:42] ...trap/engines/engine.py.224: train: epoch 101 | batch 1370/2304 [I 2022-10-28 02:35:42] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:08:38 [I 2022-10-28 02:35:42] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00246 [I 2022-10-28 02:35:42] ...trap/engines/engine.py.230: loss: 0.09003 [I 2022-10-28 02:35:42] ...optimizers/trijoint.py.151: total_norm: 10.059185 [I 2022-10-28 02:35:48] ...trap/engines/engine.py.224: train: epoch 101 | batch 1380/2304 [I 2022-10-28 02:35:48] ...trap/engines/engine.py.228: elapsed: 0:12:53 | left: 0:08:32 [I 2022-10-28 02:35:48] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00240 [I 2022-10-28 02:35:48] ...trap/engines/engine.py.230: loss: 0.10653 [I 2022-10-28 02:35:48] ...optimizers/trijoint.py.151: total_norm: 16.533764 [I 2022-10-28 02:35:53] ...trap/engines/engine.py.224: train: epoch 101 | batch 1390/2304 [I 2022-10-28 02:35:53] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:30 [I 2022-10-28 02:35:53] ...trap/engines/engine.py.229: process: 0.55507 | load: 0.00260 [I 2022-10-28 02:35:53] ...trap/engines/engine.py.230: loss: 0.09754 [I 2022-10-28 02:35:53] ...optimizers/trijoint.py.151: total_norm: 9.305789 [I 2022-10-28 02:36:17] ...trap/engines/engine.py.224: train: epoch 101 | batch 1400/2304 [I 2022-10-28 02:36:17] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:52:36 [I 2022-10-28 02:36:17] ...trap/engines/engine.py.229: process: 0.55224 | load: 0.00250 [I 2022-10-28 02:36:17] ...trap/engines/engine.py.230: loss: 0.10624 [I 2022-10-28 02:36:17] ...optimizers/trijoint.py.151: total_norm: 9.252806 [I 2022-10-28 02:36:23] ...trap/engines/engine.py.224: train: epoch 101 | batch 1410/2304 [I 2022-10-28 02:36:23] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:12:57 [I 2022-10-28 02:36:23] ...trap/engines/engine.py.229: process: 0.55108 | load: 0.00243 [I 2022-10-28 02:36:23] ...trap/engines/engine.py.230: loss: 0.09206 [I 2022-10-28 02:36:23] ...optimizers/trijoint.py.151: total_norm: 14.230126 [I 2022-10-28 02:36:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 1420/2304 [I 2022-10-28 02:36:28] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:08:43 [I 2022-10-28 02:36:28] ...trap/engines/engine.py.229: process: 0.56491 | load: 0.00242 [I 2022-10-28 02:36:28] ...trap/engines/engine.py.230: loss: 0.09905 [I 2022-10-28 02:36:28] ...optimizers/trijoint.py.151: total_norm: 9.782892 [I 2022-10-28 02:36:34] ...trap/engines/engine.py.224: train: epoch 101 | batch 1430/2304 [I 2022-10-28 02:36:34] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:08:13 [I 2022-10-28 02:36:34] ...trap/engines/engine.py.229: process: 0.56919 | load: 0.00259 [I 2022-10-28 02:36:34] ...trap/engines/engine.py.230: loss: 0.20730 [I 2022-10-28 02:36:34] ...optimizers/trijoint.py.151: total_norm: 25.416571 [I 2022-10-28 02:36:40] ...trap/engines/engine.py.224: train: epoch 101 | batch 1440/2304 [I 2022-10-28 02:36:40] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:08:03 [I 2022-10-28 02:36:40] ...trap/engines/engine.py.229: process: 0.56350 | load: 0.00254 [I 2022-10-28 02:36:40] ...trap/engines/engine.py.230: loss: 0.10730 [I 2022-10-28 02:36:40] ...optimizers/trijoint.py.151: total_norm: 12.330939 [I 2022-10-28 02:36:45] ...trap/engines/engine.py.224: train: epoch 101 | batch 1450/2304 [I 2022-10-28 02:36:45] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:08:03 [I 2022-10-28 02:36:45] ...trap/engines/engine.py.229: process: 0.56279 | load: 0.00248 [I 2022-10-28 02:36:45] ...trap/engines/engine.py.230: loss: 0.11684 [I 2022-10-28 02:36:45] ...optimizers/trijoint.py.151: total_norm: 31.854528 [I 2022-10-28 02:36:51] ...trap/engines/engine.py.224: train: epoch 101 | batch 1460/2304 [I 2022-10-28 02:36:51] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:50 [I 2022-10-28 02:36:51] ...trap/engines/engine.py.229: process: 0.56299 | load: 0.00249 [I 2022-10-28 02:36:51] ...trap/engines/engine.py.230: loss: 0.09348 [I 2022-10-28 02:36:51] ...optimizers/trijoint.py.151: total_norm: 25.102167 [I 2022-10-28 02:36:57] ...trap/engines/engine.py.224: train: epoch 101 | batch 1470/2304 [I 2022-10-28 02:36:57] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:47 [I 2022-10-28 02:36:57] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00248 [I 2022-10-28 02:36:57] ...trap/engines/engine.py.230: loss: 0.10274 [I 2022-10-28 02:36:57] ...optimizers/trijoint.py.151: total_norm: 11.322820 [I 2022-10-28 02:37:02] ...trap/engines/engine.py.224: train: epoch 101 | batch 1480/2304 [I 2022-10-28 02:37:02] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:38 [I 2022-10-28 02:37:02] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00248 [I 2022-10-28 02:37:02] ...trap/engines/engine.py.230: loss: 0.09720 [I 2022-10-28 02:37:02] ...optimizers/trijoint.py.151: total_norm: 12.648187 [I 2022-10-28 02:37:08] ...trap/engines/engine.py.224: train: epoch 101 | batch 1490/2304 [I 2022-10-28 02:37:08] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:33 [I 2022-10-28 02:37:08] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00247 [I 2022-10-28 02:37:08] ...trap/engines/engine.py.230: loss: 0.14277 [I 2022-10-28 02:37:08] ...optimizers/trijoint.py.151: total_norm: 14.801753 [I 2022-10-28 02:37:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 1500/2304 [I 2022-10-28 02:37:13] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:24 [I 2022-10-28 02:37:13] ...trap/engines/engine.py.229: process: 0.54789 | load: 0.00245 [I 2022-10-28 02:37:13] ...trap/engines/engine.py.230: loss: 0.08111 [I 2022-10-28 02:37:13] ...optimizers/trijoint.py.151: total_norm: 9.941187 [I 2022-10-28 02:37:19] ...trap/engines/engine.py.224: train: epoch 101 | batch 1510/2304 [I 2022-10-28 02:37:19] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:21 [I 2022-10-28 02:37:19] ...trap/engines/engine.py.229: process: 0.56318 | load: 0.00250 [I 2022-10-28 02:37:19] ...trap/engines/engine.py.230: loss: 0.12660 [I 2022-10-28 02:37:19] ...optimizers/trijoint.py.151: total_norm: 15.851231 [I 2022-10-28 02:37:24] ...trap/engines/engine.py.224: train: epoch 101 | batch 1520/2304 [I 2022-10-28 02:37:24] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:16 [I 2022-10-28 02:37:24] ...trap/engines/engine.py.229: process: 0.56453 | load: 0.00246 [I 2022-10-28 02:37:24] ...trap/engines/engine.py.230: loss: 0.17083 [I 2022-10-28 02:37:24] ...optimizers/trijoint.py.151: total_norm: 10.867641 [I 2022-10-28 02:37:30] ...trap/engines/engine.py.224: train: epoch 101 | batch 1530/2304 [I 2022-10-28 02:37:30] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:07:09 [I 2022-10-28 02:37:30] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00248 [I 2022-10-28 02:37:30] ...trap/engines/engine.py.230: loss: 0.09864 [I 2022-10-28 02:37:30] ...optimizers/trijoint.py.151: total_norm: 12.866614 [I 2022-10-28 02:37:36] ...trap/engines/engine.py.224: train: epoch 101 | batch 1540/2304 [I 2022-10-28 02:37:36] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:07:16 [I 2022-10-28 02:37:36] ...trap/engines/engine.py.229: process: 0.55231 | load: 0.00244 [I 2022-10-28 02:37:36] ...trap/engines/engine.py.230: loss: 0.10278 [I 2022-10-28 02:37:36] ...optimizers/trijoint.py.151: total_norm: 19.581812 [I 2022-10-28 02:37:41] ...trap/engines/engine.py.224: train: epoch 101 | batch 1550/2304 [I 2022-10-28 02:37:41] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:07:00 [I 2022-10-28 02:37:41] ...trap/engines/engine.py.229: process: 0.55783 | load: 0.00245 [I 2022-10-28 02:37:41] ...trap/engines/engine.py.230: loss: 0.12277 [I 2022-10-28 02:37:41] ...optimizers/trijoint.py.151: total_norm: 17.308903 [I 2022-10-28 02:37:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 1560/2304 [I 2022-10-28 02:37:47] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:54 [I 2022-10-28 02:37:47] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00249 [I 2022-10-28 02:37:47] ...trap/engines/engine.py.230: loss: 0.13748 [I 2022-10-28 02:37:47] ...optimizers/trijoint.py.151: total_norm: 11.597742 [I 2022-10-28 02:37:52] ...trap/engines/engine.py.224: train: epoch 101 | batch 1570/2304 [I 2022-10-28 02:37:52] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:45 [I 2022-10-28 02:37:52] ...trap/engines/engine.py.229: process: 0.54072 | load: 0.00242 [I 2022-10-28 02:37:52] ...trap/engines/engine.py.230: loss: 0.11237 [I 2022-10-28 02:37:52] ...optimizers/trijoint.py.151: total_norm: 11.940601 [I 2022-10-28 02:37:58] ...trap/engines/engine.py.224: train: epoch 101 | batch 1580/2304 [I 2022-10-28 02:37:58] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:45 [I 2022-10-28 02:37:58] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00261 [I 2022-10-28 02:37:58] ...trap/engines/engine.py.230: loss: 0.08744 [I 2022-10-28 02:37:58] ...optimizers/trijoint.py.151: total_norm: 10.686252 [I 2022-10-28 02:38:03] ...trap/engines/engine.py.224: train: epoch 101 | batch 1590/2304 [I 2022-10-28 02:38:03] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:37 [I 2022-10-28 02:38:03] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00248 [I 2022-10-28 02:38:03] ...trap/engines/engine.py.230: loss: 0.10281 [I 2022-10-28 02:38:03] ...optimizers/trijoint.py.151: total_norm: 13.273659 [I 2022-10-28 02:38:09] ...trap/engines/engine.py.224: train: epoch 101 | batch 1600/2304 [I 2022-10-28 02:38:09] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:32 [I 2022-10-28 02:38:09] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00254 [I 2022-10-28 02:38:09] ...trap/engines/engine.py.230: loss: 0.11159 [I 2022-10-28 02:38:09] ...optimizers/trijoint.py.151: total_norm: 33.532246 [I 2022-10-28 02:38:15] ...trap/engines/engine.py.224: train: epoch 101 | batch 1610/2304 [I 2022-10-28 02:38:15] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:31 [I 2022-10-28 02:38:15] ...trap/engines/engine.py.229: process: 0.54990 | load: 0.00251 [I 2022-10-28 02:38:15] ...trap/engines/engine.py.230: loss: 0.11839 [I 2022-10-28 02:38:15] ...optimizers/trijoint.py.151: total_norm: 15.719135 [I 2022-10-28 02:38:20] ...trap/engines/engine.py.224: train: epoch 101 | batch 1620/2304 [I 2022-10-28 02:38:20] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:19 [I 2022-10-28 02:38:20] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00253 [I 2022-10-28 02:38:20] ...trap/engines/engine.py.230: loss: 0.12119 [I 2022-10-28 02:38:20] ...optimizers/trijoint.py.151: total_norm: 16.132404 [I 2022-10-28 02:38:26] ...trap/engines/engine.py.224: train: epoch 101 | batch 1630/2304 [I 2022-10-28 02:38:26] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:14 [I 2022-10-28 02:38:26] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00269 [I 2022-10-28 02:38:26] ...trap/engines/engine.py.230: loss: 0.11316 [I 2022-10-28 02:38:26] ...optimizers/trijoint.py.151: total_norm: 9.605564 [I 2022-10-28 02:38:32] ...trap/engines/engine.py.224: train: epoch 101 | batch 1640/2304 [I 2022-10-28 02:38:32] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:06:08 [I 2022-10-28 02:38:32] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00252 [I 2022-10-28 02:38:32] ...trap/engines/engine.py.230: loss: 0.09265 [I 2022-10-28 02:38:32] ...optimizers/trijoint.py.151: total_norm: 14.453945 [I 2022-10-28 02:38:37] ...trap/engines/engine.py.224: train: epoch 101 | batch 1650/2304 [I 2022-10-28 02:38:37] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:06:07 [I 2022-10-28 02:38:37] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00245 [I 2022-10-28 02:38:37] ...trap/engines/engine.py.230: loss: 0.10606 [I 2022-10-28 02:38:37] ...optimizers/trijoint.py.151: total_norm: 13.837152 [I 2022-10-28 02:38:43] ...trap/engines/engine.py.224: train: epoch 101 | batch 1660/2304 [I 2022-10-28 02:38:43] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:59 [I 2022-10-28 02:38:43] ...trap/engines/engine.py.229: process: 0.55413 | load: 0.00261 [I 2022-10-28 02:38:43] ...trap/engines/engine.py.230: loss: 0.12237 [I 2022-10-28 02:38:43] ...optimizers/trijoint.py.151: total_norm: 15.724566 [I 2022-10-28 02:38:48] ...trap/engines/engine.py.224: train: epoch 101 | batch 1670/2304 [I 2022-10-28 02:38:48] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:55 [I 2022-10-28 02:38:48] ...trap/engines/engine.py.229: process: 0.55301 | load: 0.00257 [I 2022-10-28 02:38:48] ...trap/engines/engine.py.230: loss: 0.17049 [I 2022-10-28 02:38:48] ...optimizers/trijoint.py.151: total_norm: 14.511092 [I 2022-10-28 02:38:54] ...trap/engines/engine.py.224: train: epoch 101 | batch 1680/2304 [I 2022-10-28 02:38:54] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:50 [I 2022-10-28 02:38:54] ...trap/engines/engine.py.229: process: 0.55280 | load: 0.00259 [I 2022-10-28 02:38:54] ...trap/engines/engine.py.230: loss: 0.12640 [I 2022-10-28 02:38:54] ...optimizers/trijoint.py.151: total_norm: 10.901316 [I 2022-10-28 02:39:00] ...trap/engines/engine.py.224: train: epoch 101 | batch 1690/2304 [I 2022-10-28 02:39:00] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:44 [I 2022-10-28 02:39:00] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00249 [I 2022-10-28 02:39:00] ...trap/engines/engine.py.230: loss: 0.11586 [I 2022-10-28 02:39:00] ...optimizers/trijoint.py.151: total_norm: 29.603325 [I 2022-10-28 02:39:05] ...trap/engines/engine.py.224: train: epoch 101 | batch 1700/2304 [I 2022-10-28 02:39:05] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:39 [I 2022-10-28 02:39:05] ...trap/engines/engine.py.229: process: 0.56584 | load: 0.00246 [I 2022-10-28 02:39:05] ...trap/engines/engine.py.230: loss: 0.06736 [I 2022-10-28 02:39:05] ...optimizers/trijoint.py.151: total_norm: 11.229504 [I 2022-10-28 02:39:11] ...trap/engines/engine.py.224: train: epoch 101 | batch 1710/2304 [I 2022-10-28 02:39:11] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:30 [I 2022-10-28 02:39:11] ...trap/engines/engine.py.229: process: 0.54845 | load: 0.00253 [I 2022-10-28 02:39:11] ...trap/engines/engine.py.230: loss: 0.20341 [I 2022-10-28 02:39:11] ...optimizers/trijoint.py.151: total_norm: 39.665466 [I 2022-10-28 02:39:17] ...trap/engines/engine.py.224: train: epoch 101 | batch 1720/2304 [I 2022-10-28 02:39:17] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:29 [I 2022-10-28 02:39:17] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00250 [I 2022-10-28 02:39:17] ...trap/engines/engine.py.230: loss: 0.08432 [I 2022-10-28 02:39:17] ...optimizers/trijoint.py.151: total_norm: 10.226637 [I 2022-10-28 02:39:22] ...trap/engines/engine.py.224: train: epoch 101 | batch 1730/2304 [I 2022-10-28 02:39:22] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:19 [I 2022-10-28 02:39:22] ...trap/engines/engine.py.229: process: 0.55459 | load: 0.00249 [I 2022-10-28 02:39:22] ...trap/engines/engine.py.230: loss: 0.10023 [I 2022-10-28 02:39:22] ...optimizers/trijoint.py.151: total_norm: 17.073799 [I 2022-10-28 02:39:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 1740/2304 [I 2022-10-28 02:39:28] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:05:14 [I 2022-10-28 02:39:28] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00245 [I 2022-10-28 02:39:28] ...trap/engines/engine.py.230: loss: 0.15421 [I 2022-10-28 02:39:28] ...optimizers/trijoint.py.151: total_norm: 16.865463 [I 2022-10-28 02:39:33] ...trap/engines/engine.py.224: train: epoch 101 | batch 1750/2304 [I 2022-10-28 02:39:33] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:05:08 [I 2022-10-28 02:39:33] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00256 [I 2022-10-28 02:39:33] ...trap/engines/engine.py.230: loss: 0.13262 [I 2022-10-28 02:39:33] ...optimizers/trijoint.py.151: total_norm: 18.025932 [I 2022-10-28 02:39:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 1760/2304 [I 2022-10-28 02:39:39] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:05:03 [I 2022-10-28 02:39:39] ...trap/engines/engine.py.229: process: 0.56754 | load: 0.00253 [I 2022-10-28 02:39:39] ...trap/engines/engine.py.230: loss: 0.10662 [I 2022-10-28 02:39:39] ...optimizers/trijoint.py.151: total_norm: 14.724427 [I 2022-10-28 02:39:45] ...trap/engines/engine.py.224: train: epoch 101 | batch 1770/2304 [I 2022-10-28 02:39:45] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:57 [I 2022-10-28 02:39:45] ...trap/engines/engine.py.229: process: 0.55881 | load: 0.00251 [I 2022-10-28 02:39:45] ...trap/engines/engine.py.230: loss: 0.10972 [I 2022-10-28 02:39:45] ...optimizers/trijoint.py.151: total_norm: 26.798578 [I 2022-10-28 02:39:50] ...trap/engines/engine.py.224: train: epoch 101 | batch 1780/2304 [I 2022-10-28 02:39:50] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:52 [I 2022-10-28 02:39:50] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00252 [I 2022-10-28 02:39:50] ...trap/engines/engine.py.230: loss: 0.09786 [I 2022-10-28 02:39:50] ...optimizers/trijoint.py.151: total_norm: 13.496838 [I 2022-10-28 02:39:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 1790/2304 [I 2022-10-28 02:39:56] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:47 [I 2022-10-28 02:39:56] ...trap/engines/engine.py.229: process: 0.57113 | load: 0.00249 [I 2022-10-28 02:39:56] ...trap/engines/engine.py.230: loss: 0.10459 [I 2022-10-28 02:39:56] ...optimizers/trijoint.py.151: total_norm: 19.294121 [I 2022-10-28 02:40:01] ...trap/engines/engine.py.224: train: epoch 101 | batch 1800/2304 [I 2022-10-28 02:40:01] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:43 [I 2022-10-28 02:40:01] ...trap/engines/engine.py.229: process: 0.56214 | load: 0.00270 [I 2022-10-28 02:40:01] ...trap/engines/engine.py.230: loss: 0.16731 [I 2022-10-28 02:40:01] ...optimizers/trijoint.py.151: total_norm: 16.235273 [I 2022-10-28 02:40:07] ...trap/engines/engine.py.224: train: epoch 101 | batch 1810/2304 [I 2022-10-28 02:40:07] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:35 [I 2022-10-28 02:40:07] ...trap/engines/engine.py.229: process: 0.55596 | load: 0.00254 [I 2022-10-28 02:40:07] ...trap/engines/engine.py.230: loss: 0.15032 [I 2022-10-28 02:40:07] ...optimizers/trijoint.py.151: total_norm: 16.721949 [I 2022-10-28 02:40:13] ...trap/engines/engine.py.224: train: epoch 101 | batch 1820/2304 [I 2022-10-28 02:40:13] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:43 [I 2022-10-28 02:40:13] ...trap/engines/engine.py.229: process: 0.63980 | load: 0.01098 [I 2022-10-28 02:40:13] ...trap/engines/engine.py.230: loss: 0.10695 [I 2022-10-28 02:40:13] ...optimizers/trijoint.py.151: total_norm: 16.454283 [I 2022-10-28 02:40:18] ...trap/engines/engine.py.224: train: epoch 101 | batch 1830/2304 [I 2022-10-28 02:40:18] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:27 [I 2022-10-28 02:40:18] ...trap/engines/engine.py.229: process: 0.56130 | load: 0.00251 [I 2022-10-28 02:40:18] ...trap/engines/engine.py.230: loss: 0.11377 [I 2022-10-28 02:40:18] ...optimizers/trijoint.py.151: total_norm: 20.544634 [I 2022-10-28 02:40:24] ...trap/engines/engine.py.224: train: epoch 101 | batch 1840/2304 [I 2022-10-28 02:40:24] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:18 [I 2022-10-28 02:40:24] ...trap/engines/engine.py.229: process: 0.55492 | load: 0.00251 [I 2022-10-28 02:40:24] ...trap/engines/engine.py.230: loss: 0.13714 [I 2022-10-28 02:40:24] ...optimizers/trijoint.py.151: total_norm: 49.687233 [I 2022-10-28 02:40:29] ...trap/engines/engine.py.224: train: epoch 101 | batch 1850/2304 [I 2022-10-28 02:40:29] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:14 [I 2022-10-28 02:40:29] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00248 [I 2022-10-28 02:40:29] ...trap/engines/engine.py.230: loss: 0.08432 [I 2022-10-28 02:40:29] ...optimizers/trijoint.py.151: total_norm: 11.104774 [I 2022-10-28 02:40:35] ...trap/engines/engine.py.224: train: epoch 101 | batch 1860/2304 [I 2022-10-28 02:40:35] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:04:08 [I 2022-10-28 02:40:35] ...trap/engines/engine.py.229: process: 0.57061 | load: 0.00242 [I 2022-10-28 02:40:35] ...trap/engines/engine.py.230: loss: 0.13163 [I 2022-10-28 02:40:35] ...optimizers/trijoint.py.151: total_norm: 20.294584 [I 2022-10-28 02:40:41] ...trap/engines/engine.py.224: train: epoch 101 | batch 1870/2304 [I 2022-10-28 02:40:41] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:04:01 [I 2022-10-28 02:40:41] ...trap/engines/engine.py.229: process: 0.55488 | load: 0.00249 [I 2022-10-28 02:40:41] ...trap/engines/engine.py.230: loss: 0.10427 [I 2022-10-28 02:40:41] ...optimizers/trijoint.py.151: total_norm: 18.649607 [I 2022-10-28 02:40:46] ...trap/engines/engine.py.224: train: epoch 101 | batch 1880/2304 [I 2022-10-28 02:40:46] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:55 [I 2022-10-28 02:40:46] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00247 [I 2022-10-28 02:40:46] ...trap/engines/engine.py.230: loss: 0.09980 [I 2022-10-28 02:40:46] ...optimizers/trijoint.py.151: total_norm: 15.135815 [I 2022-10-28 02:40:52] ...trap/engines/engine.py.224: train: epoch 101 | batch 1890/2304 [I 2022-10-28 02:40:52] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:51 [I 2022-10-28 02:40:52] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00248 [I 2022-10-28 02:40:52] ...trap/engines/engine.py.230: loss: 0.14784 [I 2022-10-28 02:40:52] ...optimizers/trijoint.py.151: total_norm: 16.814451 [I 2022-10-28 02:40:57] ...trap/engines/engine.py.224: train: epoch 101 | batch 1900/2304 [I 2022-10-28 02:40:57] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:43 [I 2022-10-28 02:40:57] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00248 [I 2022-10-28 02:40:57] ...trap/engines/engine.py.230: loss: 0.14886 [I 2022-10-28 02:40:57] ...optimizers/trijoint.py.151: total_norm: 23.914804 [I 2022-10-28 02:41:03] ...trap/engines/engine.py.224: train: epoch 101 | batch 1910/2304 [I 2022-10-28 02:41:03] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:39 [I 2022-10-28 02:41:03] ...trap/engines/engine.py.229: process: 0.55349 | load: 0.00246 [I 2022-10-28 02:41:03] ...trap/engines/engine.py.230: loss: 0.08342 [I 2022-10-28 02:41:03] ...optimizers/trijoint.py.151: total_norm: 13.671194 [I 2022-10-28 02:41:08] ...trap/engines/engine.py.224: train: epoch 101 | batch 1920/2304 [I 2022-10-28 02:41:08] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:33 [I 2022-10-28 02:41:08] ...trap/engines/engine.py.229: process: 0.56398 | load: 0.00255 [I 2022-10-28 02:41:08] ...trap/engines/engine.py.230: loss: 0.11505 [I 2022-10-28 02:41:08] ...optimizers/trijoint.py.151: total_norm: 11.640603 [I 2022-10-28 02:41:14] ...trap/engines/engine.py.224: train: epoch 101 | batch 1930/2304 [I 2022-10-28 02:41:14] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:26 [I 2022-10-28 02:41:14] ...trap/engines/engine.py.229: process: 0.55215 | load: 0.00249 [I 2022-10-28 02:41:14] ...trap/engines/engine.py.230: loss: 0.12332 [I 2022-10-28 02:41:14] ...optimizers/trijoint.py.151: total_norm: 21.717501 [I 2022-10-28 02:41:20] ...trap/engines/engine.py.224: train: epoch 101 | batch 1940/2304 [I 2022-10-28 02:41:20] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:21 [I 2022-10-28 02:41:20] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00249 [I 2022-10-28 02:41:20] ...trap/engines/engine.py.230: loss: 0.12610 [I 2022-10-28 02:41:20] ...optimizers/trijoint.py.151: total_norm: 16.152618 [I 2022-10-28 02:41:25] ...trap/engines/engine.py.224: train: epoch 101 | batch 1950/2304 [I 2022-10-28 02:41:25] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:17 [I 2022-10-28 02:41:25] ...trap/engines/engine.py.229: process: 0.55284 | load: 0.00249 [I 2022-10-28 02:41:25] ...trap/engines/engine.py.230: loss: 0.16360 [I 2022-10-28 02:41:25] ...optimizers/trijoint.py.151: total_norm: 18.187901 [I 2022-10-28 02:41:31] ...trap/engines/engine.py.224: train: epoch 101 | batch 1960/2304 [I 2022-10-28 02:41:31] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:03:11 [I 2022-10-28 02:41:31] ...trap/engines/engine.py.229: process: 0.55335 | load: 0.00269 [I 2022-10-28 02:41:31] ...trap/engines/engine.py.230: loss: 0.11013 [I 2022-10-28 02:41:31] ...optimizers/trijoint.py.151: total_norm: 11.122284 [I 2022-10-28 02:41:36] ...trap/engines/engine.py.224: train: epoch 101 | batch 1970/2304 [I 2022-10-28 02:41:36] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:03:05 [I 2022-10-28 02:41:36] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00252 [I 2022-10-28 02:41:36] ...trap/engines/engine.py.230: loss: 0.13046 [I 2022-10-28 02:41:36] ...optimizers/trijoint.py.151: total_norm: 31.305456 [I 2022-10-28 02:41:42] ...trap/engines/engine.py.224: train: epoch 101 | batch 1980/2304 [I 2022-10-28 02:41:42] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:03:00 [I 2022-10-28 02:41:42] ...trap/engines/engine.py.229: process: 0.55774 | load: 0.00253 [I 2022-10-28 02:41:42] ...trap/engines/engine.py.230: loss: 0.10923 [I 2022-10-28 02:41:42] ...optimizers/trijoint.py.151: total_norm: 22.590788 [I 2022-10-28 02:41:47] ...trap/engines/engine.py.224: train: epoch 101 | batch 1990/2304 [I 2022-10-28 02:41:47] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:54 [I 2022-10-28 02:41:47] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00250 [I 2022-10-28 02:41:47] ...trap/engines/engine.py.230: loss: 0.21387 [I 2022-10-28 02:41:47] ...optimizers/trijoint.py.151: total_norm: 52.077080 [I 2022-10-28 02:41:53] ...trap/engines/engine.py.224: train: epoch 101 | batch 2000/2304 [I 2022-10-28 02:41:53] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:48 [I 2022-10-28 02:41:53] ...trap/engines/engine.py.229: process: 0.57082 | load: 0.00249 [I 2022-10-28 02:41:53] ...trap/engines/engine.py.230: loss: 0.09591 [I 2022-10-28 02:41:53] ...optimizers/trijoint.py.151: total_norm: 34.142818 [I 2022-10-28 02:41:59] ...trap/engines/engine.py.224: train: epoch 101 | batch 2010/2304 [I 2022-10-28 02:41:59] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:43 [I 2022-10-28 02:41:59] ...trap/engines/engine.py.229: process: 0.56427 | load: 0.00250 [I 2022-10-28 02:41:59] ...trap/engines/engine.py.230: loss: 0.17327 [I 2022-10-28 02:41:59] ...optimizers/trijoint.py.151: total_norm: 38.586498 [I 2022-10-28 02:42:04] ...trap/engines/engine.py.224: train: epoch 101 | batch 2020/2304 [I 2022-10-28 02:42:04] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:38 [I 2022-10-28 02:42:04] ...trap/engines/engine.py.229: process: 0.55672 | load: 0.00251 [I 2022-10-28 02:42:04] ...trap/engines/engine.py.230: loss: 0.17921 [I 2022-10-28 02:42:04] ...optimizers/trijoint.py.151: total_norm: 18.486780 [I 2022-10-28 02:42:10] ...trap/engines/engine.py.224: train: epoch 101 | batch 2030/2304 [I 2022-10-28 02:42:10] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:31 [I 2022-10-28 02:42:10] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00258 [I 2022-10-28 02:42:10] ...trap/engines/engine.py.230: loss: 0.15393 [I 2022-10-28 02:42:10] ...optimizers/trijoint.py.151: total_norm: 10.376505 [I 2022-10-28 02:42:15] ...trap/engines/engine.py.224: train: epoch 101 | batch 2040/2304 [I 2022-10-28 02:42:15] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:27 [I 2022-10-28 02:42:15] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00253 [I 2022-10-28 02:42:15] ...trap/engines/engine.py.230: loss: 0.09824 [I 2022-10-28 02:42:15] ...optimizers/trijoint.py.151: total_norm: 13.386461 [I 2022-10-28 02:42:21] ...trap/engines/engine.py.224: train: epoch 101 | batch 2050/2304 [I 2022-10-28 02:42:21] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:20 [I 2022-10-28 02:42:21] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00254 [I 2022-10-28 02:42:21] ...trap/engines/engine.py.230: loss: 0.10756 [I 2022-10-28 02:42:21] ...optimizers/trijoint.py.151: total_norm: 35.966713 [I 2022-10-28 02:42:26] ...trap/engines/engine.py.224: train: epoch 101 | batch 2060/2304 [I 2022-10-28 02:42:26] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:02:15 [I 2022-10-28 02:42:26] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00253 [I 2022-10-28 02:42:26] ...trap/engines/engine.py.230: loss: 0.11373 [I 2022-10-28 02:42:26] ...optimizers/trijoint.py.151: total_norm: 14.983738 [I 2022-10-28 02:42:32] ...trap/engines/engine.py.224: train: epoch 101 | batch 2070/2304 [I 2022-10-28 02:42:32] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:10 [I 2022-10-28 02:42:32] ...trap/engines/engine.py.229: process: 0.55161 | load: 0.00251 [I 2022-10-28 02:42:32] ...trap/engines/engine.py.230: loss: 0.18028 [I 2022-10-28 02:42:32] ...optimizers/trijoint.py.151: total_norm: 13.705074 [I 2022-10-28 02:42:38] ...trap/engines/engine.py.224: train: epoch 101 | batch 2080/2304 [I 2022-10-28 02:42:38] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:02:05 [I 2022-10-28 02:42:38] ...trap/engines/engine.py.229: process: 0.55575 | load: 0.00250 [I 2022-10-28 02:42:38] ...trap/engines/engine.py.230: loss: 0.21062 [I 2022-10-28 02:42:38] ...optimizers/trijoint.py.151: total_norm: 26.043018 [I 2022-10-28 02:42:43] ...trap/engines/engine.py.224: train: epoch 101 | batch 2090/2304 [I 2022-10-28 02:42:43] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:59 [I 2022-10-28 02:42:43] ...trap/engines/engine.py.229: process: 0.56392 | load: 0.00245 [I 2022-10-28 02:42:43] ...trap/engines/engine.py.230: loss: 0.11081 [I 2022-10-28 02:42:43] ...optimizers/trijoint.py.151: total_norm: 13.919138 [I 2022-10-28 02:42:49] ...trap/engines/engine.py.224: train: epoch 101 | batch 2100/2304 [I 2022-10-28 02:42:49] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:52 [I 2022-10-28 02:42:49] ...trap/engines/engine.py.229: process: 0.54185 | load: 0.00266 [I 2022-10-28 02:42:49] ...trap/engines/engine.py.230: loss: 0.11758 [I 2022-10-28 02:42:49] ...optimizers/trijoint.py.151: total_norm: 12.782194 [I 2022-10-28 02:42:54] ...trap/engines/engine.py.224: train: epoch 101 | batch 2110/2304 [I 2022-10-28 02:42:54] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:48 [I 2022-10-28 02:42:54] ...trap/engines/engine.py.229: process: 0.57191 | load: 0.00247 [I 2022-10-28 02:42:54] ...trap/engines/engine.py.230: loss: 0.14960 [I 2022-10-28 02:42:54] ...optimizers/trijoint.py.151: total_norm: 20.775063 [I 2022-10-28 02:43:00] ...trap/engines/engine.py.224: train: epoch 101 | batch 2120/2304 [I 2022-10-28 02:43:00] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:42 [I 2022-10-28 02:43:00] ...trap/engines/engine.py.229: process: 0.56222 | load: 0.00251 [I 2022-10-28 02:43:00] ...trap/engines/engine.py.230: loss: 0.08980 [I 2022-10-28 02:43:00] ...optimizers/trijoint.py.151: total_norm: 15.968358 [I 2022-10-28 02:43:05] ...trap/engines/engine.py.224: train: epoch 101 | batch 2130/2304 [I 2022-10-28 02:43:05] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:36 [I 2022-10-28 02:43:05] ...trap/engines/engine.py.229: process: 0.54504 | load: 0.00246 [I 2022-10-28 02:43:05] ...trap/engines/engine.py.230: loss: 0.13454 [I 2022-10-28 02:43:05] ...optimizers/trijoint.py.151: total_norm: 12.328802 [I 2022-10-28 02:43:11] ...trap/engines/engine.py.224: train: epoch 101 | batch 2140/2304 [I 2022-10-28 02:43:11] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:31 [I 2022-10-28 02:43:11] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00253 [I 2022-10-28 02:43:11] ...trap/engines/engine.py.230: loss: 0.14629 [I 2022-10-28 02:43:11] ...optimizers/trijoint.py.151: total_norm: 30.156128 [I 2022-10-28 02:43:17] ...trap/engines/engine.py.224: train: epoch 101 | batch 2150/2304 [I 2022-10-28 02:43:17] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:25 [I 2022-10-28 02:43:17] ...trap/engines/engine.py.229: process: 0.54812 | load: 0.00252 [I 2022-10-28 02:43:17] ...trap/engines/engine.py.230: loss: 0.13310 [I 2022-10-28 02:43:17] ...optimizers/trijoint.py.151: total_norm: 11.957873 [I 2022-10-28 02:43:22] ...trap/engines/engine.py.224: train: epoch 101 | batch 2160/2304 [I 2022-10-28 02:43:22] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:20 [I 2022-10-28 02:43:22] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00257 [I 2022-10-28 02:43:22] ...trap/engines/engine.py.230: loss: 0.08976 [I 2022-10-28 02:43:22] ...optimizers/trijoint.py.151: total_norm: 9.765323 [I 2022-10-28 02:43:28] ...trap/engines/engine.py.224: train: epoch 101 | batch 2170/2304 [I 2022-10-28 02:43:28] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:14 [I 2022-10-28 02:43:28] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00249 [I 2022-10-28 02:43:28] ...trap/engines/engine.py.230: loss: 0.13133 [I 2022-10-28 02:43:28] ...optimizers/trijoint.py.151: total_norm: 14.786842 [I 2022-10-28 02:43:33] ...trap/engines/engine.py.224: train: epoch 101 | batch 2180/2304 [I 2022-10-28 02:43:33] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:01:08 [I 2022-10-28 02:43:33] ...trap/engines/engine.py.229: process: 0.54173 | load: 0.00252 [I 2022-10-28 02:43:33] ...trap/engines/engine.py.230: loss: 0.09023 [I 2022-10-28 02:43:33] ...optimizers/trijoint.py.151: total_norm: 18.543371 [I 2022-10-28 02:43:39] ...trap/engines/engine.py.224: train: epoch 101 | batch 2190/2304 [I 2022-10-28 02:43:39] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:03 [I 2022-10-28 02:43:39] ...trap/engines/engine.py.229: process: 0.55385 | load: 0.00251 [I 2022-10-28 02:43:39] ...trap/engines/engine.py.230: loss: 0.12086 [I 2022-10-28 02:43:39] ...optimizers/trijoint.py.151: total_norm: 15.165441 [I 2022-10-28 02:43:44] ...trap/engines/engine.py.224: train: epoch 101 | batch 2200/2304 [I 2022-10-28 02:43:44] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:58 [I 2022-10-28 02:43:44] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00247 [I 2022-10-28 02:43:44] ...trap/engines/engine.py.230: loss: 0.10149 [I 2022-10-28 02:43:44] ...optimizers/trijoint.py.151: total_norm: 22.925575 [I 2022-10-28 02:43:50] ...trap/engines/engine.py.224: train: epoch 101 | batch 2210/2304 [I 2022-10-28 02:43:50] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:52 [I 2022-10-28 02:43:50] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00250 [I 2022-10-28 02:43:50] ...trap/engines/engine.py.230: loss: 0.10901 [I 2022-10-28 02:43:50] ...optimizers/trijoint.py.151: total_norm: 18.192268 [I 2022-10-28 02:43:56] ...trap/engines/engine.py.224: train: epoch 101 | batch 2220/2304 [I 2022-10-28 02:43:56] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:46 [I 2022-10-28 02:43:56] ...trap/engines/engine.py.229: process: 0.55137 | load: 0.00250 [I 2022-10-28 02:43:56] ...trap/engines/engine.py.230: loss: 0.14400 [I 2022-10-28 02:43:56] ...optimizers/trijoint.py.151: total_norm: 29.740410 [I 2022-10-28 02:44:01] ...trap/engines/engine.py.224: train: epoch 101 | batch 2230/2304 [I 2022-10-28 02:44:01] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:41 [I 2022-10-28 02:44:01] ...trap/engines/engine.py.229: process: 0.56277 | load: 0.00255 [I 2022-10-28 02:44:01] ...trap/engines/engine.py.230: loss: 0.15912 [I 2022-10-28 02:44:01] ...optimizers/trijoint.py.151: total_norm: 15.014801 [I 2022-10-28 02:44:07] ...trap/engines/engine.py.224: train: epoch 101 | batch 2240/2304 [I 2022-10-28 02:44:07] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:35 [I 2022-10-28 02:44:07] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00248 [I 2022-10-28 02:44:07] ...trap/engines/engine.py.230: loss: 0.13775 [I 2022-10-28 02:44:07] ...optimizers/trijoint.py.151: total_norm: 10.095493 [I 2022-10-28 02:44:12] ...trap/engines/engine.py.224: train: epoch 101 | batch 2250/2304 [I 2022-10-28 02:44:12] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:30 [I 2022-10-28 02:44:12] ...trap/engines/engine.py.229: process: 0.55413 | load: 0.00253 [I 2022-10-28 02:44:12] ...trap/engines/engine.py.230: loss: 0.10416 [I 2022-10-28 02:44:12] ...optimizers/trijoint.py.151: total_norm: 10.688683 [I 2022-10-28 02:44:18] ...trap/engines/engine.py.224: train: epoch 101 | batch 2260/2304 [I 2022-10-28 02:44:18] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:24 [I 2022-10-28 02:44:18] ...trap/engines/engine.py.229: process: 0.56547 | load: 0.00251 [I 2022-10-28 02:44:18] ...trap/engines/engine.py.230: loss: 0.10476 [I 2022-10-28 02:44:18] ...optimizers/trijoint.py.151: total_norm: 10.258490 [I 2022-10-28 02:44:23] ...trap/engines/engine.py.224: train: epoch 101 | batch 2270/2304 [I 2022-10-28 02:44:23] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:18 [I 2022-10-28 02:44:23] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00248 [I 2022-10-28 02:44:23] ...trap/engines/engine.py.230: loss: 0.08838 [I 2022-10-28 02:44:23] ...optimizers/trijoint.py.151: total_norm: 10.399689 [I 2022-10-28 02:44:29] ...trap/engines/engine.py.224: train: epoch 101 | batch 2280/2304 [I 2022-10-28 02:44:29] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:13 [I 2022-10-28 02:44:29] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00250 [I 2022-10-28 02:44:29] ...trap/engines/engine.py.230: loss: 0.22197 [I 2022-10-28 02:44:29] ...optimizers/trijoint.py.151: total_norm: 15.207354 [I 2022-10-28 02:44:35] ...trap/engines/engine.py.224: train: epoch 101 | batch 2290/2304 [I 2022-10-28 02:44:35] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:07 [I 2022-10-28 02:44:35] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00244 [I 2022-10-28 02:44:35] ...trap/engines/engine.py.230: loss: 0.09507 [I 2022-10-28 02:44:35] ...optimizers/trijoint.py.151: total_norm: 10.607049 [I 2022-10-28 02:44:40] ...trap/engines/engine.py.224: train: epoch 101 | batch 2300/2304 [I 2022-10-28 02:44:40] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:02 [I 2022-10-28 02:44:40] ...trap/engines/engine.py.229: process: 0.56546 | load: 0.00015 [I 2022-10-28 02:44:40] ...trap/engines/engine.py.230: loss: 0.10732 [I 2022-10-28 02:44:40] ...optimizers/trijoint.py.151: total_norm: 9.847266 [I 2022-10-28 02:44:42] ...trap/engines/engine.py.224: train: epoch 101 | batch 2304/2304 [I 2022-10-28 02:44:42] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:00 [I 2022-10-28 02:44:42] ...trap/engines/engine.py.229: process: 0.54267 | load: 0.00019 [I 2022-10-28 02:44:42] ...trap/engines/engine.py.230: loss: 0.14084 [I 2022-10-28 02:44:42] ...optimizers/trijoint.py.151: total_norm: 26.310459 [S 2022-10-28 02:44:44] ...trap/engines/engine.py.129: train_epoch.epoch: 101 [S 2022-10-28 02:44:44] ...trap/engines/engine.py.129: train_epoch.loss: 0.12503079566740938 [I 2022-10-28 02:44:44] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 02:44:45] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 101 [I 2022-10-28 02:44:45] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 02:44:47] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 02:44:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 02:44:47] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 02:44:50] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 02:44:51] ...trap/engines/engine.py.321: eval: epoch 101 | batch 0/510 [I 2022-10-28 02:44:51] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:48:23 [I 2022-10-28 02:44:51] ...trap/engines/engine.py.326: process: 5.69260 | load: 3.14883 [I 2022-10-28 02:44:54] ...trap/engines/engine.py.321: eval: epoch 101 | batch 10/510 [I 2022-10-28 02:44:54] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:31 [I 2022-10-28 02:44:54] ...trap/engines/engine.py.326: process: 0.32325 | load: 0.00021 [I 2022-10-28 02:44:57] ...trap/engines/engine.py.321: eval: epoch 101 | batch 20/510 [I 2022-10-28 02:44:57] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:14 [I 2022-10-28 02:44:57] ...trap/engines/engine.py.326: process: 0.34149 | load: 0.00017 [I 2022-10-28 02:45:01] ...trap/engines/engine.py.321: eval: epoch 101 | batch 30/510 [I 2022-10-28 02:45:01] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:45 [I 2022-10-28 02:45:01] ...trap/engines/engine.py.326: process: 0.36054 | load: 0.00018 [I 2022-10-28 02:45:04] ...trap/engines/engine.py.321: eval: epoch 101 | batch 40/510 [I 2022-10-28 02:45:04] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-28 02:45:04] ...trap/engines/engine.py.326: process: 0.32938 | load: 0.00027 [I 2022-10-28 02:45:07] ...trap/engines/engine.py.321: eval: epoch 101 | batch 50/510 [I 2022-10-28 02:45:07] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:35 [I 2022-10-28 02:45:07] ...trap/engines/engine.py.326: process: 0.35164 | load: 0.00023 [I 2022-10-28 02:45:11] ...trap/engines/engine.py.321: eval: epoch 101 | batch 60/510 [I 2022-10-28 02:45:11] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:34 [I 2022-10-28 02:45:11] ...trap/engines/engine.py.326: process: 0.38110 | load: 0.00019 [I 2022-10-28 02:45:14] ...trap/engines/engine.py.321: eval: epoch 101 | batch 70/510 [I 2022-10-28 02:45:14] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:28 [I 2022-10-28 02:45:14] ...trap/engines/engine.py.326: process: 0.33405 | load: 0.00017 [I 2022-10-28 02:45:18] ...trap/engines/engine.py.321: eval: epoch 101 | batch 80/510 [I 2022-10-28 02:45:18] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:25 [I 2022-10-28 02:45:18] ...trap/engines/engine.py.326: process: 0.33076 | load: 0.00019 [I 2022-10-28 02:45:21] ...trap/engines/engine.py.321: eval: epoch 101 | batch 90/510 [I 2022-10-28 02:45:21] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-28 02:45:21] ...trap/engines/engine.py.326: process: 0.33125 | load: 0.00016 [I 2022-10-28 02:45:24] ...trap/engines/engine.py.321: eval: epoch 101 | batch 100/510 [I 2022-10-28 02:45:24] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:12 [I 2022-10-28 02:45:24] ...trap/engines/engine.py.326: process: 0.28795 | load: 0.00021 [I 2022-10-28 02:45:27] ...trap/engines/engine.py.321: eval: epoch 101 | batch 110/510 [I 2022-10-28 02:45:27] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:56 [I 2022-10-28 02:45:27] ...trap/engines/engine.py.326: process: 0.28634 | load: 0.00015 [I 2022-10-28 02:45:30] ...trap/engines/engine.py.321: eval: epoch 101 | batch 120/510 [I 2022-10-28 02:45:30] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:51 [I 2022-10-28 02:45:30] ...trap/engines/engine.py.326: process: 0.28790 | load: 0.00015 [I 2022-10-28 02:45:33] ...trap/engines/engine.py.321: eval: epoch 101 | batch 130/510 [I 2022-10-28 02:45:33] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-28 02:45:33] ...trap/engines/engine.py.326: process: 0.28805 | load: 0.00027 [I 2022-10-28 02:45:36] ...trap/engines/engine.py.321: eval: epoch 101 | batch 140/510 [I 2022-10-28 02:45:36] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-28 02:45:36] ...trap/engines/engine.py.326: process: 0.28777 | load: 0.00023 [I 2022-10-28 02:45:39] ...trap/engines/engine.py.321: eval: epoch 101 | batch 150/510 [I 2022-10-28 02:45:39] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:44 [I 2022-10-28 02:45:39] ...trap/engines/engine.py.326: process: 0.28690 | load: 0.00021 [I 2022-10-28 02:45:42] ...trap/engines/engine.py.321: eval: epoch 101 | batch 160/510 [I 2022-10-28 02:45:42] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-28 02:45:42] ...trap/engines/engine.py.326: process: 0.29154 | load: 0.00015 [I 2022-10-28 02:45:44] ...trap/engines/engine.py.321: eval: epoch 101 | batch 170/510 [I 2022-10-28 02:45:44] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 02:45:44] ...trap/engines/engine.py.326: process: 0.28034 | load: 0.00014 [I 2022-10-28 02:45:47] ...trap/engines/engine.py.321: eval: epoch 101 | batch 180/510 [I 2022-10-28 02:45:47] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:36 [I 2022-10-28 02:45:47] ...trap/engines/engine.py.326: process: 0.29115 | load: 0.00015 [I 2022-10-28 02:45:50] ...trap/engines/engine.py.321: eval: epoch 101 | batch 190/510 [I 2022-10-28 02:45:50] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-28 02:45:50] ...trap/engines/engine.py.326: process: 0.28004 | load: 0.00018 [I 2022-10-28 02:45:53] ...trap/engines/engine.py.321: eval: epoch 101 | batch 200/510 [I 2022-10-28 02:45:53] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:29 [I 2022-10-28 02:45:53] ...trap/engines/engine.py.326: process: 0.29157 | load: 0.00020 [I 2022-10-28 02:45:56] ...trap/engines/engine.py.321: eval: epoch 101 | batch 210/510 [I 2022-10-28 02:45:56] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:25 [I 2022-10-28 02:45:56] ...trap/engines/engine.py.326: process: 0.28618 | load: 0.00015 [I 2022-10-28 02:45:59] ...trap/engines/engine.py.321: eval: epoch 101 | batch 220/510 [I 2022-10-28 02:45:59] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-28 02:45:59] ...trap/engines/engine.py.326: process: 0.28922 | load: 0.00020 [I 2022-10-28 02:46:02] ...trap/engines/engine.py.321: eval: epoch 101 | batch 230/510 [I 2022-10-28 02:46:02] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-28 02:46:02] ...trap/engines/engine.py.326: process: 0.28830 | load: 0.00018 [I 2022-10-28 02:46:05] ...trap/engines/engine.py.321: eval: epoch 101 | batch 240/510 [I 2022-10-28 02:46:05] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 02:46:05] ...trap/engines/engine.py.326: process: 0.28486 | load: 0.00018 [I 2022-10-28 02:46:08] ...trap/engines/engine.py.321: eval: epoch 101 | batch 250/510 [I 2022-10-28 02:46:08] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-28 02:46:08] ...trap/engines/engine.py.326: process: 0.28985 | load: 0.00016 [I 2022-10-28 02:46:10] ...trap/engines/engine.py.321: eval: epoch 101 | batch 260/510 [I 2022-10-28 02:46:10] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 02:46:10] ...trap/engines/engine.py.326: process: 0.30276 | load: 0.00017 [I 2022-10-28 02:46:13] ...trap/engines/engine.py.321: eval: epoch 101 | batch 270/510 [I 2022-10-28 02:46:13] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-28 02:46:13] ...trap/engines/engine.py.326: process: 0.28583 | load: 0.00021 [I 2022-10-28 02:46:16] ...trap/engines/engine.py.321: eval: epoch 101 | batch 280/510 [I 2022-10-28 02:46:16] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 02:46:16] ...trap/engines/engine.py.326: process: 0.28640 | load: 0.00024 [I 2022-10-28 02:46:19] ...trap/engines/engine.py.321: eval: epoch 101 | batch 290/510 [I 2022-10-28 02:46:19] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 02:46:19] ...trap/engines/engine.py.326: process: 0.28777 | load: 0.00020 [I 2022-10-28 02:46:22] ...trap/engines/engine.py.321: eval: epoch 101 | batch 300/510 [I 2022-10-28 02:46:22] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-28 02:46:22] ...trap/engines/engine.py.326: process: 0.29136 | load: 0.00016 [I 2022-10-28 02:46:25] ...trap/engines/engine.py.321: eval: epoch 101 | batch 310/510 [I 2022-10-28 02:46:25] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 02:46:25] ...trap/engines/engine.py.326: process: 0.28376 | load: 0.00015 [I 2022-10-28 02:46:28] ...trap/engines/engine.py.321: eval: epoch 101 | batch 320/510 [I 2022-10-28 02:46:28] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:55 [I 2022-10-28 02:46:28] ...trap/engines/engine.py.326: process: 0.29222 | load: 0.00025 [I 2022-10-28 02:46:31] ...trap/engines/engine.py.321: eval: epoch 101 | batch 330/510 [I 2022-10-28 02:46:31] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:53 [I 2022-10-28 02:46:31] ...trap/engines/engine.py.326: process: 0.29019 | load: 0.00020 [I 2022-10-28 02:46:34] ...trap/engines/engine.py.321: eval: epoch 101 | batch 340/510 [I 2022-10-28 02:46:34] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-28 02:46:34] ...trap/engines/engine.py.326: process: 0.29094 | load: 0.00014 [I 2022-10-28 02:46:37] ...trap/engines/engine.py.321: eval: epoch 101 | batch 350/510 [I 2022-10-28 02:46:37] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 02:46:37] ...trap/engines/engine.py.326: process: 0.29127 | load: 0.00016 [I 2022-10-28 02:46:39] ...trap/engines/engine.py.321: eval: epoch 101 | batch 360/510 [I 2022-10-28 02:46:39] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:44 [I 2022-10-28 02:46:39] ...trap/engines/engine.py.326: process: 0.30655 | load: 0.00015 [I 2022-10-28 02:46:42] ...trap/engines/engine.py.321: eval: epoch 101 | batch 370/510 [I 2022-10-28 02:46:42] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:41 [I 2022-10-28 02:46:42] ...trap/engines/engine.py.326: process: 0.29051 | load: 0.00015 [I 2022-10-28 02:46:45] ...trap/engines/engine.py.321: eval: epoch 101 | batch 380/510 [I 2022-10-28 02:46:45] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 02:46:45] ...trap/engines/engine.py.326: process: 0.28306 | load: 0.00027 [I 2022-10-28 02:46:48] ...trap/engines/engine.py.321: eval: epoch 101 | batch 390/510 [I 2022-10-28 02:46:48] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 02:46:48] ...trap/engines/engine.py.326: process: 0.28873 | load: 0.00015 [I 2022-10-28 02:46:51] ...trap/engines/engine.py.321: eval: epoch 101 | batch 400/510 [I 2022-10-28 02:46:51] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:32 [I 2022-10-28 02:46:51] ...trap/engines/engine.py.326: process: 0.28364 | load: 0.00025 [I 2022-10-28 02:46:54] ...trap/engines/engine.py.321: eval: epoch 101 | batch 410/510 [I 2022-10-28 02:46:54] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:28 [I 2022-10-28 02:46:54] ...trap/engines/engine.py.326: process: 0.28560 | load: 0.00015 [I 2022-10-28 02:46:57] ...trap/engines/engine.py.321: eval: epoch 101 | batch 420/510 [I 2022-10-28 02:46:57] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:26 [I 2022-10-28 02:46:57] ...trap/engines/engine.py.326: process: 0.28735 | load: 0.00020 [I 2022-10-28 02:47:00] ...trap/engines/engine.py.321: eval: epoch 101 | batch 430/510 [I 2022-10-28 02:47:00] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 02:47:00] ...trap/engines/engine.py.326: process: 0.29180 | load: 0.00015 [I 2022-10-28 02:47:03] ...trap/engines/engine.py.321: eval: epoch 101 | batch 440/510 [I 2022-10-28 02:47:03] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 02:47:03] ...trap/engines/engine.py.326: process: 0.28670 | load: 0.00025 [I 2022-10-28 02:47:06] ...trap/engines/engine.py.321: eval: epoch 101 | batch 450/510 [I 2022-10-28 02:47:06] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 02:47:06] ...trap/engines/engine.py.326: process: 0.28639 | load: 0.00020 [I 2022-10-28 02:47:09] ...trap/engines/engine.py.321: eval: epoch 101 | batch 460/510 [I 2022-10-28 02:47:09] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 02:47:09] ...trap/engines/engine.py.326: process: 0.29212 | load: 0.00016 [I 2022-10-28 02:47:11] ...trap/engines/engine.py.321: eval: epoch 101 | batch 470/510 [I 2022-10-28 02:47:11] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 02:47:11] ...trap/engines/engine.py.326: process: 0.28552 | load: 0.00020 [I 2022-10-28 02:47:14] ...trap/engines/engine.py.321: eval: epoch 101 | batch 480/510 [I 2022-10-28 02:47:14] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 02:47:14] ...trap/engines/engine.py.326: process: 0.29033 | load: 0.00026 [I 2022-10-28 02:47:17] ...trap/engines/engine.py.321: eval: epoch 101 | batch 490/510 [I 2022-10-28 02:47:17] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 02:47:17] ...trap/engines/engine.py.326: process: 0.29077 | load: 0.00015 [I 2022-10-28 02:47:20] ...trap/engines/engine.py.321: eval: epoch 101 | batch 500/510 [I 2022-10-28 02:47:20] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:02 [I 2022-10-28 02:47:20] ...trap/engines/engine.py.326: process: 0.28786 | load: 0.00040 [I 2022-10-28 02:47:23] ...trap/engines/engine.py.321: eval: epoch 101 | batch 510/510 [I 2022-10-28 02:47:23] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:00 [I 2022-10-28 02:47:23] ...trap/engines/engine.py.326: process: 0.28751 | load: 0.00012 [S 2022-10-28 02:47:24] ...trap/engines/engine.py.132: eval_epoch.epoch: 101 [I 2022-10-28 02:47:24] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.735200 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.904300 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.932100 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.727300 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904900 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933800 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011729 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.004051 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004206 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010555 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.005009 [S 2022-10-28 02:47:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005363 [I 2022-10-28 02:47:30] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 02:47:30] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 02:47:31] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 02:47:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 02:47:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 02:47:35] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 02:47:36] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 02:47:42] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 02:47:42] ...trap/engines/engine.py.163: Training model on trainset for epoch 102 [I 2022-10-28 02:47:42] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 02:47:44] ...trap/engines/engine.py.224: train: epoch 102 | batch 0/2304 [I 2022-10-28 02:47:44] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:42:48 [I 2022-10-28 02:47:44] ...trap/engines/engine.py.229: process: 2.67750 | load: 2.03973 [I 2022-10-28 02:47:44] ...trap/engines/engine.py.230: loss: 0.09345 [I 2022-10-28 02:47:44] ...optimizers/trijoint.py.151: total_norm: 21.703012 [I 2022-10-28 02:47:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 10/2304 [I 2022-10-28 02:47:50] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:12 [I 2022-10-28 02:47:50] ...trap/engines/engine.py.229: process: 0.55118 | load: 0.00346 [I 2022-10-28 02:47:50] ...trap/engines/engine.py.230: loss: 0.13106 [I 2022-10-28 02:47:50] ...optimizers/trijoint.py.151: total_norm: 11.613696 [I 2022-10-28 02:47:56] ...trap/engines/engine.py.224: train: epoch 102 | batch 20/2304 [I 2022-10-28 02:47:56] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:06 [I 2022-10-28 02:47:56] ...trap/engines/engine.py.229: process: 0.55369 | load: 0.00344 [I 2022-10-28 02:47:56] ...trap/engines/engine.py.230: loss: 0.08772 [I 2022-10-28 02:47:56] ...optimizers/trijoint.py.151: total_norm: 15.495679 [I 2022-10-28 02:48:01] ...trap/engines/engine.py.224: train: epoch 102 | batch 30/2304 [I 2022-10-28 02:48:01] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:14 [I 2022-10-28 02:48:01] ...trap/engines/engine.py.229: process: 0.54438 | load: 0.00304 [I 2022-10-28 02:48:01] ...trap/engines/engine.py.230: loss: 0.26274 [I 2022-10-28 02:48:01] ...optimizers/trijoint.py.151: total_norm: 25.935186 [I 2022-10-28 02:48:07] ...trap/engines/engine.py.224: train: epoch 102 | batch 40/2304 [I 2022-10-28 02:48:07] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:10 [I 2022-10-28 02:48:07] ...trap/engines/engine.py.229: process: 0.54977 | load: 0.00375 [I 2022-10-28 02:48:07] ...trap/engines/engine.py.230: loss: 0.09143 [I 2022-10-28 02:48:07] ...optimizers/trijoint.py.151: total_norm: 11.873812 [I 2022-10-28 02:48:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 50/2304 [I 2022-10-28 02:48:12] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:02 [I 2022-10-28 02:48:12] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00290 [I 2022-10-28 02:48:12] ...trap/engines/engine.py.230: loss: 0.08580 [I 2022-10-28 02:48:12] ...optimizers/trijoint.py.151: total_norm: 8.376856 [I 2022-10-28 02:48:18] ...trap/engines/engine.py.224: train: epoch 102 | batch 60/2304 [I 2022-10-28 02:48:18] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:50 [I 2022-10-28 02:48:18] ...trap/engines/engine.py.229: process: 0.55031 | load: 0.00275 [I 2022-10-28 02:48:18] ...trap/engines/engine.py.230: loss: 0.11273 [I 2022-10-28 02:48:18] ...optimizers/trijoint.py.151: total_norm: 8.394624 [I 2022-10-28 02:48:24] ...trap/engines/engine.py.224: train: epoch 102 | batch 70/2304 [I 2022-10-28 02:48:24] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:41 [I 2022-10-28 02:48:24] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00274 [I 2022-10-28 02:48:24] ...trap/engines/engine.py.230: loss: 0.14933 [I 2022-10-28 02:48:24] ...optimizers/trijoint.py.151: total_norm: 9.914101 [I 2022-10-28 02:48:29] ...trap/engines/engine.py.224: train: epoch 102 | batch 80/2304 [I 2022-10-28 02:48:29] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:34 [I 2022-10-28 02:48:29] ...trap/engines/engine.py.229: process: 0.55442 | load: 0.00267 [I 2022-10-28 02:48:29] ...trap/engines/engine.py.230: loss: 0.10788 [I 2022-10-28 02:48:29] ...optimizers/trijoint.py.151: total_norm: 14.710454 [I 2022-10-28 02:48:35] ...trap/engines/engine.py.224: train: epoch 102 | batch 90/2304 [I 2022-10-28 02:48:35] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:28 [I 2022-10-28 02:48:35] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00266 [I 2022-10-28 02:48:35] ...trap/engines/engine.py.230: loss: 0.14540 [I 2022-10-28 02:48:35] ...optimizers/trijoint.py.151: total_norm: 22.116711 [I 2022-10-28 02:48:41] ...trap/engines/engine.py.224: train: epoch 102 | batch 100/2304 [I 2022-10-28 02:48:41] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:21:26 [I 2022-10-28 02:48:41] ...trap/engines/engine.py.229: process: 0.56953 | load: 0.00304 [I 2022-10-28 02:48:41] ...trap/engines/engine.py.230: loss: 0.22011 [I 2022-10-28 02:48:41] ...optimizers/trijoint.py.151: total_norm: 41.582912 [I 2022-10-28 02:48:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 110/2304 [I 2022-10-28 02:48:46] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:20 [I 2022-10-28 02:48:46] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00256 [I 2022-10-28 02:48:46] ...trap/engines/engine.py.230: loss: 0.12092 [I 2022-10-28 02:48:46] ...optimizers/trijoint.py.151: total_norm: 14.539980 [I 2022-10-28 02:48:52] ...trap/engines/engine.py.224: train: epoch 102 | batch 120/2304 [I 2022-10-28 02:48:52] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:30 [I 2022-10-28 02:48:52] ...trap/engines/engine.py.229: process: 0.58489 | load: 0.00263 [I 2022-10-28 02:48:52] ...trap/engines/engine.py.230: loss: 0.11251 [I 2022-10-28 02:48:52] ...optimizers/trijoint.py.151: total_norm: 12.583713 [I 2022-10-28 02:48:57] ...trap/engines/engine.py.224: train: epoch 102 | batch 130/2304 [I 2022-10-28 02:48:57] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:16 [I 2022-10-28 02:48:57] ...trap/engines/engine.py.229: process: 0.54916 | load: 0.00347 [I 2022-10-28 02:48:57] ...trap/engines/engine.py.230: loss: 0.10244 [I 2022-10-28 02:48:57] ...optimizers/trijoint.py.151: total_norm: 9.414728 [I 2022-10-28 02:49:03] ...trap/engines/engine.py.224: train: epoch 102 | batch 140/2304 [I 2022-10-28 02:49:03] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:06 [I 2022-10-28 02:49:03] ...trap/engines/engine.py.229: process: 0.56157 | load: 0.00277 [I 2022-10-28 02:49:03] ...trap/engines/engine.py.230: loss: 0.09266 [I 2022-10-28 02:49:03] ...optimizers/trijoint.py.151: total_norm: 16.124323 [I 2022-10-28 02:49:09] ...trap/engines/engine.py.224: train: epoch 102 | batch 150/2304 [I 2022-10-28 02:49:09] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:04 [I 2022-10-28 02:49:09] ...trap/engines/engine.py.229: process: 0.55520 | load: 0.00254 [I 2022-10-28 02:49:09] ...trap/engines/engine.py.230: loss: 0.09237 [I 2022-10-28 02:49:09] ...optimizers/trijoint.py.151: total_norm: 39.099762 [I 2022-10-28 02:49:14] ...trap/engines/engine.py.224: train: epoch 102 | batch 160/2304 [I 2022-10-28 02:49:14] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:59 [I 2022-10-28 02:49:14] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00264 [I 2022-10-28 02:49:14] ...trap/engines/engine.py.230: loss: 0.15977 [I 2022-10-28 02:49:14] ...optimizers/trijoint.py.151: total_norm: 13.680893 [I 2022-10-28 02:49:20] ...trap/engines/engine.py.224: train: epoch 102 | batch 170/2304 [I 2022-10-28 02:49:20] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:20:05 [I 2022-10-28 02:49:20] ...trap/engines/engine.py.229: process: 0.56320 | load: 0.00258 [I 2022-10-28 02:49:20] ...trap/engines/engine.py.230: loss: 0.14939 [I 2022-10-28 02:49:20] ...optimizers/trijoint.py.151: total_norm: 40.051144 [I 2022-10-28 02:49:26] ...trap/engines/engine.py.224: train: epoch 102 | batch 180/2304 [I 2022-10-28 02:49:26] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:55 [I 2022-10-28 02:49:26] ...trap/engines/engine.py.229: process: 0.57172 | load: 0.00250 [I 2022-10-28 02:49:26] ...trap/engines/engine.py.230: loss: 0.09185 [I 2022-10-28 02:49:26] ...optimizers/trijoint.py.151: total_norm: 11.272217 [I 2022-10-28 02:49:31] ...trap/engines/engine.py.224: train: epoch 102 | batch 190/2304 [I 2022-10-28 02:49:31] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:55 [I 2022-10-28 02:49:31] ...trap/engines/engine.py.229: process: 0.56685 | load: 0.00257 [I 2022-10-28 02:49:31] ...trap/engines/engine.py.230: loss: 0.10552 [I 2022-10-28 02:49:31] ...optimizers/trijoint.py.151: total_norm: 10.066451 [I 2022-10-28 02:49:37] ...trap/engines/engine.py.224: train: epoch 102 | batch 200/2304 [I 2022-10-28 02:49:37] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:31 [I 2022-10-28 02:49:37] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00268 [I 2022-10-28 02:49:37] ...trap/engines/engine.py.230: loss: 0.12537 [I 2022-10-28 02:49:37] ...optimizers/trijoint.py.151: total_norm: 20.553238 [I 2022-10-28 02:49:43] ...trap/engines/engine.py.224: train: epoch 102 | batch 210/2304 [I 2022-10-28 02:49:43] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:47 [I 2022-10-28 02:49:43] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00262 [I 2022-10-28 02:49:43] ...trap/engines/engine.py.230: loss: 0.13502 [I 2022-10-28 02:49:43] ...optimizers/trijoint.py.151: total_norm: 14.453825 [I 2022-10-28 02:49:48] ...trap/engines/engine.py.224: train: epoch 102 | batch 220/2304 [I 2022-10-28 02:49:48] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:26 [I 2022-10-28 02:49:48] ...trap/engines/engine.py.229: process: 0.54914 | load: 0.00253 [I 2022-10-28 02:49:48] ...trap/engines/engine.py.230: loss: 0.10166 [I 2022-10-28 02:49:48] ...optimizers/trijoint.py.151: total_norm: 11.284539 [I 2022-10-28 02:49:54] ...trap/engines/engine.py.224: train: epoch 102 | batch 230/2304 [I 2022-10-28 02:49:54] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:19:29 [I 2022-10-28 02:49:54] ...trap/engines/engine.py.229: process: 0.56049 | load: 0.00299 [I 2022-10-28 02:49:54] ...trap/engines/engine.py.230: loss: 0.15784 [I 2022-10-28 02:49:54] ...optimizers/trijoint.py.151: total_norm: 16.469315 [I 2022-10-28 02:49:59] ...trap/engines/engine.py.224: train: epoch 102 | batch 240/2304 [I 2022-10-28 02:49:59] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:26 [I 2022-10-28 02:49:59] ...trap/engines/engine.py.229: process: 0.57440 | load: 0.00283 [I 2022-10-28 02:49:59] ...trap/engines/engine.py.230: loss: 0.12478 [I 2022-10-28 02:49:59] ...optimizers/trijoint.py.151: total_norm: 11.496765 [I 2022-10-28 02:50:05] ...trap/engines/engine.py.224: train: epoch 102 | batch 250/2304 [I 2022-10-28 02:50:05] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:18 [I 2022-10-28 02:50:05] ...trap/engines/engine.py.229: process: 0.56460 | load: 0.00250 [I 2022-10-28 02:50:05] ...trap/engines/engine.py.230: loss: 0.11102 [I 2022-10-28 02:50:05] ...optimizers/trijoint.py.151: total_norm: 9.848061 [I 2022-10-28 02:50:11] ...trap/engines/engine.py.224: train: epoch 102 | batch 260/2304 [I 2022-10-28 02:50:11] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:19:14 [I 2022-10-28 02:50:11] ...trap/engines/engine.py.229: process: 0.56800 | load: 0.00264 [I 2022-10-28 02:50:11] ...trap/engines/engine.py.230: loss: 0.11941 [I 2022-10-28 02:50:11] ...optimizers/trijoint.py.151: total_norm: 24.490454 [I 2022-10-28 02:50:16] ...trap/engines/engine.py.224: train: epoch 102 | batch 270/2304 [I 2022-10-28 02:50:16] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:55 [I 2022-10-28 02:50:16] ...trap/engines/engine.py.229: process: 0.55344 | load: 0.00244 [I 2022-10-28 02:50:16] ...trap/engines/engine.py.230: loss: 0.08170 [I 2022-10-28 02:50:16] ...optimizers/trijoint.py.151: total_norm: 7.840438 [I 2022-10-28 02:50:22] ...trap/engines/engine.py.224: train: epoch 102 | batch 280/2304 [I 2022-10-28 02:50:22] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:18:54 [I 2022-10-28 02:50:22] ...trap/engines/engine.py.229: process: 0.56225 | load: 0.00254 [I 2022-10-28 02:50:22] ...trap/engines/engine.py.230: loss: 0.10763 [I 2022-10-28 02:50:22] ...optimizers/trijoint.py.151: total_norm: 26.112432 [I 2022-10-28 02:50:28] ...trap/engines/engine.py.224: train: epoch 102 | batch 290/2304 [I 2022-10-28 02:50:28] ...trap/engines/engine.py.228: elapsed: 0:02:46 | left: 0:19:23 [I 2022-10-28 02:50:28] ...trap/engines/engine.py.229: process: 0.55376 | load: 0.00269 [I 2022-10-28 02:50:28] ...trap/engines/engine.py.230: loss: 0.09995 [I 2022-10-28 02:50:28] ...optimizers/trijoint.py.151: total_norm: 30.136003 [I 2022-10-28 02:50:33] ...trap/engines/engine.py.224: train: epoch 102 | batch 300/2304 [I 2022-10-28 02:50:33] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:39 [I 2022-10-28 02:50:33] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00243 [I 2022-10-28 02:50:33] ...trap/engines/engine.py.230: loss: 0.12262 [I 2022-10-28 02:50:33] ...optimizers/trijoint.py.151: total_norm: 11.751513 [I 2022-10-28 02:50:39] ...trap/engines/engine.py.224: train: epoch 102 | batch 310/2304 [I 2022-10-28 02:50:39] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:18:34 [I 2022-10-28 02:50:39] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00249 [I 2022-10-28 02:50:39] ...trap/engines/engine.py.230: loss: 0.13219 [I 2022-10-28 02:50:39] ...optimizers/trijoint.py.151: total_norm: 21.798895 [I 2022-10-28 02:50:45] ...trap/engines/engine.py.224: train: epoch 102 | batch 320/2304 [I 2022-10-28 02:50:45] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:33 [I 2022-10-28 02:50:45] ...trap/engines/engine.py.229: process: 0.57306 | load: 0.00255 [I 2022-10-28 02:50:45] ...trap/engines/engine.py.230: loss: 0.09309 [I 2022-10-28 02:50:45] ...optimizers/trijoint.py.151: total_norm: 11.371497 [I 2022-10-28 02:50:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 330/2304 [I 2022-10-28 02:50:50] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:18:32 [I 2022-10-28 02:50:50] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00246 [I 2022-10-28 02:50:50] ...trap/engines/engine.py.230: loss: 0.10085 [I 2022-10-28 02:50:50] ...optimizers/trijoint.py.151: total_norm: 11.426916 [I 2022-10-28 02:50:56] ...trap/engines/engine.py.224: train: epoch 102 | batch 340/2304 [I 2022-10-28 02:50:56] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:18:38 [I 2022-10-28 02:50:56] ...trap/engines/engine.py.229: process: 0.59818 | load: 0.00291 [I 2022-10-28 02:50:56] ...trap/engines/engine.py.230: loss: 0.13967 [I 2022-10-28 02:50:56] ...optimizers/trijoint.py.151: total_norm: 23.698450 [I 2022-10-28 02:51:02] ...trap/engines/engine.py.224: train: epoch 102 | batch 350/2304 [I 2022-10-28 02:51:02] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:13 [I 2022-10-28 02:51:02] ...trap/engines/engine.py.229: process: 0.54459 | load: 0.00253 [I 2022-10-28 02:51:02] ...trap/engines/engine.py.230: loss: 0.08910 [I 2022-10-28 02:51:02] ...optimizers/trijoint.py.151: total_norm: 9.383662 [I 2022-10-28 02:51:07] ...trap/engines/engine.py.224: train: epoch 102 | batch 360/2304 [I 2022-10-28 02:51:07] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:18:09 [I 2022-10-28 02:51:07] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00242 [I 2022-10-28 02:51:07] ...trap/engines/engine.py.230: loss: 0.12442 [I 2022-10-28 02:51:07] ...optimizers/trijoint.py.151: total_norm: 12.107348 [I 2022-10-28 02:51:13] ...trap/engines/engine.py.224: train: epoch 102 | batch 370/2304 [I 2022-10-28 02:51:13] ...trap/engines/engine.py.228: elapsed: 0:03:31 | left: 0:17:58 [I 2022-10-28 02:51:13] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00245 [I 2022-10-28 02:51:13] ...trap/engines/engine.py.230: loss: 0.12044 [I 2022-10-28 02:51:13] ...optimizers/trijoint.py.151: total_norm: 30.957939 [I 2022-10-28 02:51:18] ...trap/engines/engine.py.224: train: epoch 102 | batch 380/2304 [I 2022-10-28 02:51:18] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:17:56 [I 2022-10-28 02:51:18] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00247 [I 2022-10-28 02:51:18] ...trap/engines/engine.py.230: loss: 0.11226 [I 2022-10-28 02:51:18] ...optimizers/trijoint.py.151: total_norm: 11.665085 [I 2022-10-28 02:51:24] ...trap/engines/engine.py.224: train: epoch 102 | batch 390/2304 [I 2022-10-28 02:51:24] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:17:47 [I 2022-10-28 02:51:24] ...trap/engines/engine.py.229: process: 0.55248 | load: 0.00242 [I 2022-10-28 02:51:24] ...trap/engines/engine.py.230: loss: 0.10035 [I 2022-10-28 02:51:24] ...optimizers/trijoint.py.151: total_norm: 10.153493 [I 2022-10-28 02:51:30] ...trap/engines/engine.py.224: train: epoch 102 | batch 400/2304 [I 2022-10-28 02:51:30] ...trap/engines/engine.py.228: elapsed: 0:03:48 | left: 0:17:52 [I 2022-10-28 02:51:30] ...trap/engines/engine.py.229: process: 0.56535 | load: 0.00252 [I 2022-10-28 02:51:30] ...trap/engines/engine.py.230: loss: 0.12689 [I 2022-10-28 02:51:30] ...optimizers/trijoint.py.151: total_norm: 9.204152 [I 2022-10-28 02:51:35] ...trap/engines/engine.py.224: train: epoch 102 | batch 410/2304 [I 2022-10-28 02:51:35] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:17:27 [I 2022-10-28 02:51:35] ...trap/engines/engine.py.229: process: 0.54469 | load: 0.00245 [I 2022-10-28 02:51:35] ...trap/engines/engine.py.230: loss: 0.14473 [I 2022-10-28 02:51:35] ...optimizers/trijoint.py.151: total_norm: 19.919230 [I 2022-10-28 02:51:41] ...trap/engines/engine.py.224: train: epoch 102 | batch 420/2304 [I 2022-10-28 02:51:41] ...trap/engines/engine.py.228: elapsed: 0:03:59 | left: 0:17:53 [I 2022-10-28 02:51:41] ...trap/engines/engine.py.229: process: 0.56612 | load: 0.00241 [I 2022-10-28 02:51:41] ...trap/engines/engine.py.230: loss: 0.10905 [I 2022-10-28 02:51:41] ...optimizers/trijoint.py.151: total_norm: 9.676769 [I 2022-10-28 02:51:47] ...trap/engines/engine.py.224: train: epoch 102 | batch 430/2304 [I 2022-10-28 02:51:47] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:34 [I 2022-10-28 02:51:47] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00243 [I 2022-10-28 02:51:47] ...trap/engines/engine.py.230: loss: 0.15597 [I 2022-10-28 02:51:47] ...optimizers/trijoint.py.151: total_norm: 14.180109 [I 2022-10-28 02:51:52] ...trap/engines/engine.py.224: train: epoch 102 | batch 440/2304 [I 2022-10-28 02:51:52] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:17:27 [I 2022-10-28 02:51:52] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00247 [I 2022-10-28 02:51:52] ...trap/engines/engine.py.230: loss: 0.11383 [I 2022-10-28 02:51:52] ...optimizers/trijoint.py.151: total_norm: 13.933687 [I 2022-10-28 02:51:58] ...trap/engines/engine.py.224: train: epoch 102 | batch 450/2304 [I 2022-10-28 02:51:58] ...trap/engines/engine.py.228: elapsed: 0:04:16 | left: 0:17:15 [I 2022-10-28 02:51:58] ...trap/engines/engine.py.229: process: 0.54722 | load: 0.00251 [I 2022-10-28 02:51:58] ...trap/engines/engine.py.230: loss: 0.13719 [I 2022-10-28 02:51:58] ...optimizers/trijoint.py.151: total_norm: 15.285028 [I 2022-10-28 02:52:04] ...trap/engines/engine.py.224: train: epoch 102 | batch 460/2304 [I 2022-10-28 02:52:04] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:17:16 [I 2022-10-28 02:52:04] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00247 [I 2022-10-28 02:52:04] ...trap/engines/engine.py.230: loss: 0.10423 [I 2022-10-28 02:52:04] ...optimizers/trijoint.py.151: total_norm: 29.305489 [I 2022-10-28 02:52:09] ...trap/engines/engine.py.224: train: epoch 102 | batch 470/2304 [I 2022-10-28 02:52:09] ...trap/engines/engine.py.228: elapsed: 0:04:27 | left: 0:17:01 [I 2022-10-28 02:52:09] ...trap/engines/engine.py.229: process: 0.54874 | load: 0.00249 [I 2022-10-28 02:52:09] ...trap/engines/engine.py.230: loss: 0.12367 [I 2022-10-28 02:52:09] ...optimizers/trijoint.py.151: total_norm: 15.696184 [I 2022-10-28 02:52:15] ...trap/engines/engine.py.224: train: epoch 102 | batch 480/2304 [I 2022-10-28 02:52:15] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:17:03 [I 2022-10-28 02:52:15] ...trap/engines/engine.py.229: process: 0.56911 | load: 0.00241 [I 2022-10-28 02:52:15] ...trap/engines/engine.py.230: loss: 0.09071 [I 2022-10-28 02:52:15] ...optimizers/trijoint.py.151: total_norm: 21.115170 [I 2022-10-28 02:52:20] ...trap/engines/engine.py.224: train: epoch 102 | batch 490/2304 [I 2022-10-28 02:52:20] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:16:59 [I 2022-10-28 02:52:20] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00245 [I 2022-10-28 02:52:20] ...trap/engines/engine.py.230: loss: 0.08717 [I 2022-10-28 02:52:20] ...optimizers/trijoint.py.151: total_norm: 9.779716 [I 2022-10-28 02:52:26] ...trap/engines/engine.py.224: train: epoch 102 | batch 500/2304 [I 2022-10-28 02:52:26] ...trap/engines/engine.py.228: elapsed: 0:04:44 | left: 0:16:44 [I 2022-10-28 02:52:26] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00248 [I 2022-10-28 02:52:26] ...trap/engines/engine.py.230: loss: 0.08270 [I 2022-10-28 02:52:26] ...optimizers/trijoint.py.151: total_norm: 12.054621 [I 2022-10-28 02:52:32] ...trap/engines/engine.py.224: train: epoch 102 | batch 510/2304 [I 2022-10-28 02:52:32] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:43 [I 2022-10-28 02:52:32] ...trap/engines/engine.py.229: process: 0.56757 | load: 0.00244 [I 2022-10-28 02:52:32] ...trap/engines/engine.py.230: loss: 0.09449 [I 2022-10-28 02:52:32] ...optimizers/trijoint.py.151: total_norm: 10.964289 [I 2022-10-28 02:52:37] ...trap/engines/engine.py.224: train: epoch 102 | batch 520/2304 [I 2022-10-28 02:52:37] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:16:38 [I 2022-10-28 02:52:37] ...trap/engines/engine.py.229: process: 0.56283 | load: 0.00247 [I 2022-10-28 02:52:37] ...trap/engines/engine.py.230: loss: 0.12590 [I 2022-10-28 02:52:37] ...optimizers/trijoint.py.151: total_norm: 30.626011 [I 2022-10-28 02:52:43] ...trap/engines/engine.py.224: train: epoch 102 | batch 530/2304 [I 2022-10-28 02:52:43] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 0:16:40 [I 2022-10-28 02:52:43] ...trap/engines/engine.py.229: process: 0.57767 | load: 0.00247 [I 2022-10-28 02:52:43] ...trap/engines/engine.py.230: loss: 0.15412 [I 2022-10-28 02:52:43] ...optimizers/trijoint.py.151: total_norm: 22.902584 [I 2022-10-28 02:52:48] ...trap/engines/engine.py.224: train: epoch 102 | batch 540/2304 [I 2022-10-28 02:52:48] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:16:22 [I 2022-10-28 02:52:48] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00251 [I 2022-10-28 02:52:48] ...trap/engines/engine.py.230: loss: 0.08133 [I 2022-10-28 02:52:48] ...optimizers/trijoint.py.151: total_norm: 16.084511 [I 2022-10-28 02:52:54] ...trap/engines/engine.py.224: train: epoch 102 | batch 550/2304 [I 2022-10-28 02:52:54] ...trap/engines/engine.py.228: elapsed: 0:05:12 | left: 0:16:15 [I 2022-10-28 02:52:54] ...trap/engines/engine.py.229: process: 0.55329 | load: 0.00243 [I 2022-10-28 02:52:54] ...trap/engines/engine.py.230: loss: 0.08955 [I 2022-10-28 02:52:54] ...optimizers/trijoint.py.151: total_norm: 10.442692 [I 2022-10-28 02:53:00] ...trap/engines/engine.py.224: train: epoch 102 | batch 560/2304 [I 2022-10-28 02:53:00] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:19 [I 2022-10-28 02:53:00] ...trap/engines/engine.py.229: process: 0.56788 | load: 0.00244 [I 2022-10-28 02:53:00] ...trap/engines/engine.py.230: loss: 0.10697 [I 2022-10-28 02:53:00] ...optimizers/trijoint.py.151: total_norm: 18.101130 [I 2022-10-28 02:53:05] ...trap/engines/engine.py.224: train: epoch 102 | batch 570/2304 [I 2022-10-28 02:53:05] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:16:11 [I 2022-10-28 02:53:05] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00245 [I 2022-10-28 02:53:05] ...trap/engines/engine.py.230: loss: 0.12606 [I 2022-10-28 02:53:05] ...optimizers/trijoint.py.151: total_norm: 24.072824 [I 2022-10-28 02:53:11] ...trap/engines/engine.py.224: train: epoch 102 | batch 580/2304 [I 2022-10-28 02:53:11] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:16:08 [I 2022-10-28 02:53:11] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00245 [I 2022-10-28 02:53:11] ...trap/engines/engine.py.230: loss: 0.08759 [I 2022-10-28 02:53:11] ...optimizers/trijoint.py.151: total_norm: 10.923592 [I 2022-10-28 02:53:16] ...trap/engines/engine.py.224: train: epoch 102 | batch 590/2304 [I 2022-10-28 02:53:16] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:16:06 [I 2022-10-28 02:53:16] ...trap/engines/engine.py.229: process: 0.56910 | load: 0.00242 [I 2022-10-28 02:53:16] ...trap/engines/engine.py.230: loss: 0.15956 [I 2022-10-28 02:53:16] ...optimizers/trijoint.py.151: total_norm: 18.456354 [I 2022-10-28 02:53:22] ...trap/engines/engine.py.224: train: epoch 102 | batch 600/2304 [I 2022-10-28 02:53:22] ...trap/engines/engine.py.228: elapsed: 0:05:40 | left: 0:15:55 [I 2022-10-28 02:53:22] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00243 [I 2022-10-28 02:53:22] ...trap/engines/engine.py.230: loss: 0.17279 [I 2022-10-28 02:53:22] ...optimizers/trijoint.py.151: total_norm: 20.156631 [I 2022-10-28 02:53:28] ...trap/engines/engine.py.224: train: epoch 102 | batch 610/2304 [I 2022-10-28 02:53:28] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:15:41 [I 2022-10-28 02:53:28] ...trap/engines/engine.py.229: process: 0.55247 | load: 0.00241 [I 2022-10-28 02:53:28] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-28 02:53:28] ...optimizers/trijoint.py.151: total_norm: 13.025816 [I 2022-10-28 02:53:33] ...trap/engines/engine.py.224: train: epoch 102 | batch 620/2304 [I 2022-10-28 02:53:33] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:15:38 [I 2022-10-28 02:53:33] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00244 [I 2022-10-28 02:53:33] ...trap/engines/engine.py.230: loss: 0.14519 [I 2022-10-28 02:53:33] ...optimizers/trijoint.py.151: total_norm: 16.441626 [I 2022-10-28 02:53:39] ...trap/engines/engine.py.224: train: epoch 102 | batch 630/2304 [I 2022-10-28 02:53:39] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:15:33 [I 2022-10-28 02:53:39] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00244 [I 2022-10-28 02:53:39] ...trap/engines/engine.py.230: loss: 0.11626 [I 2022-10-28 02:53:39] ...optimizers/trijoint.py.151: total_norm: 19.538498 [I 2022-10-28 02:53:44] ...trap/engines/engine.py.224: train: epoch 102 | batch 640/2304 [I 2022-10-28 02:53:44] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:15:25 [I 2022-10-28 02:53:44] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00244 [I 2022-10-28 02:53:44] ...trap/engines/engine.py.230: loss: 0.11747 [I 2022-10-28 02:53:44] ...optimizers/trijoint.py.151: total_norm: 10.341437 [I 2022-10-28 02:53:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 650/2304 [I 2022-10-28 02:53:50] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:15:15 [I 2022-10-28 02:53:50] ...trap/engines/engine.py.229: process: 0.54806 | load: 0.00248 [I 2022-10-28 02:53:50] ...trap/engines/engine.py.230: loss: 0.18567 [I 2022-10-28 02:53:50] ...optimizers/trijoint.py.151: total_norm: 28.781443 [I 2022-10-28 02:53:56] ...trap/engines/engine.py.224: train: epoch 102 | batch 660/2304 [I 2022-10-28 02:53:56] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:15:19 [I 2022-10-28 02:53:56] ...trap/engines/engine.py.229: process: 0.56482 | load: 0.00246 [I 2022-10-28 02:53:56] ...trap/engines/engine.py.230: loss: 0.14225 [I 2022-10-28 02:53:56] ...optimizers/trijoint.py.151: total_norm: 33.349251 [I 2022-10-28 02:54:01] ...trap/engines/engine.py.224: train: epoch 102 | batch 670/2304 [I 2022-10-28 02:54:01] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:15:20 [I 2022-10-28 02:54:01] ...trap/engines/engine.py.229: process: 0.57213 | load: 0.00261 [I 2022-10-28 02:54:01] ...trap/engines/engine.py.230: loss: 0.12932 [I 2022-10-28 02:54:01] ...optimizers/trijoint.py.151: total_norm: 12.567921 [I 2022-10-28 02:54:07] ...trap/engines/engine.py.224: train: epoch 102 | batch 680/2304 [I 2022-10-28 02:54:07] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:15:14 [I 2022-10-28 02:54:07] ...trap/engines/engine.py.229: process: 0.56919 | load: 0.00243 [I 2022-10-28 02:54:07] ...trap/engines/engine.py.230: loss: 0.09355 [I 2022-10-28 02:54:07] ...optimizers/trijoint.py.151: total_norm: 15.289489 [I 2022-10-28 02:54:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 690/2304 [I 2022-10-28 02:54:12] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:14:59 [I 2022-10-28 02:54:12] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00246 [I 2022-10-28 02:54:12] ...trap/engines/engine.py.230: loss: 0.12412 [I 2022-10-28 02:54:12] ...optimizers/trijoint.py.151: total_norm: 19.528126 [I 2022-10-28 02:54:18] ...trap/engines/engine.py.224: train: epoch 102 | batch 700/2304 [I 2022-10-28 02:54:18] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:15:05 [I 2022-10-28 02:54:18] ...trap/engines/engine.py.229: process: 0.56602 | load: 0.00242 [I 2022-10-28 02:54:18] ...trap/engines/engine.py.230: loss: 0.09610 [I 2022-10-28 02:54:18] ...optimizers/trijoint.py.151: total_norm: 9.404053 [I 2022-10-28 02:54:24] ...trap/engines/engine.py.224: train: epoch 102 | batch 710/2304 [I 2022-10-28 02:54:24] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:14:55 [I 2022-10-28 02:54:24] ...trap/engines/engine.py.229: process: 0.56320 | load: 0.00241 [I 2022-10-28 02:54:24] ...trap/engines/engine.py.230: loss: 0.14998 [I 2022-10-28 02:54:24] ...optimizers/trijoint.py.151: total_norm: 14.574972 [I 2022-10-28 02:54:29] ...trap/engines/engine.py.224: train: epoch 102 | batch 720/2304 [I 2022-10-28 02:54:29] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:14:44 [I 2022-10-28 02:54:29] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00241 [I 2022-10-28 02:54:29] ...trap/engines/engine.py.230: loss: 0.14584 [I 2022-10-28 02:54:29] ...optimizers/trijoint.py.151: total_norm: 21.667215 [I 2022-10-28 02:54:35] ...trap/engines/engine.py.224: train: epoch 102 | batch 730/2304 [I 2022-10-28 02:54:35] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:14:35 [I 2022-10-28 02:54:35] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00245 [I 2022-10-28 02:54:35] ...trap/engines/engine.py.230: loss: 0.08076 [I 2022-10-28 02:54:35] ...optimizers/trijoint.py.151: total_norm: 16.081249 [I 2022-10-28 02:54:40] ...trap/engines/engine.py.224: train: epoch 102 | batch 740/2304 [I 2022-10-28 02:54:40] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:14:34 [I 2022-10-28 02:54:40] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00253 [I 2022-10-28 02:54:40] ...trap/engines/engine.py.230: loss: 0.12691 [I 2022-10-28 02:54:40] ...optimizers/trijoint.py.151: total_norm: 95.161758 [I 2022-10-28 02:54:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 750/2304 [I 2022-10-28 02:54:46] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:14:26 [I 2022-10-28 02:54:46] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00243 [I 2022-10-28 02:54:46] ...trap/engines/engine.py.230: loss: 0.15235 [I 2022-10-28 02:54:46] ...optimizers/trijoint.py.151: total_norm: 24.101788 [I 2022-10-28 02:54:52] ...trap/engines/engine.py.224: train: epoch 102 | batch 760/2304 [I 2022-10-28 02:54:52] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:14:28 [I 2022-10-28 02:54:52] ...trap/engines/engine.py.229: process: 0.56982 | load: 0.00245 [I 2022-10-28 02:54:52] ...trap/engines/engine.py.230: loss: 0.16440 [I 2022-10-28 02:54:52] ...optimizers/trijoint.py.151: total_norm: 10.275598 [I 2022-10-28 02:54:57] ...trap/engines/engine.py.224: train: epoch 102 | batch 770/2304 [I 2022-10-28 02:54:57] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:14:15 [I 2022-10-28 02:54:57] ...trap/engines/engine.py.229: process: 0.56319 | load: 0.00244 [I 2022-10-28 02:54:57] ...trap/engines/engine.py.230: loss: 0.11414 [I 2022-10-28 02:54:57] ...optimizers/trijoint.py.151: total_norm: 11.509646 [I 2022-10-28 02:55:03] ...trap/engines/engine.py.224: train: epoch 102 | batch 780/2304 [I 2022-10-28 02:55:03] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:14:14 [I 2022-10-28 02:55:03] ...trap/engines/engine.py.229: process: 0.56916 | load: 0.00243 [I 2022-10-28 02:55:03] ...trap/engines/engine.py.230: loss: 0.08644 [I 2022-10-28 02:55:03] ...optimizers/trijoint.py.151: total_norm: 11.684390 [I 2022-10-28 02:55:08] ...trap/engines/engine.py.224: train: epoch 102 | batch 790/2304 [I 2022-10-28 02:55:08] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:14:03 [I 2022-10-28 02:55:08] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00243 [I 2022-10-28 02:55:08] ...trap/engines/engine.py.230: loss: 0.16372 [I 2022-10-28 02:55:08] ...optimizers/trijoint.py.151: total_norm: 25.900661 [I 2022-10-28 02:55:14] ...trap/engines/engine.py.224: train: epoch 102 | batch 800/2304 [I 2022-10-28 02:55:14] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:13:57 [I 2022-10-28 02:55:14] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00239 [I 2022-10-28 02:55:14] ...trap/engines/engine.py.230: loss: 0.10850 [I 2022-10-28 02:55:14] ...optimizers/trijoint.py.151: total_norm: 20.753487 [I 2022-10-28 02:55:20] ...trap/engines/engine.py.224: train: epoch 102 | batch 810/2304 [I 2022-10-28 02:55:20] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:13:58 [I 2022-10-28 02:55:20] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00242 [I 2022-10-28 02:55:20] ...trap/engines/engine.py.230: loss: 0.09057 [I 2022-10-28 02:55:20] ...optimizers/trijoint.py.151: total_norm: 16.232840 [I 2022-10-28 02:55:25] ...trap/engines/engine.py.224: train: epoch 102 | batch 820/2304 [I 2022-10-28 02:55:25] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:13:44 [I 2022-10-28 02:55:25] ...trap/engines/engine.py.229: process: 0.54500 | load: 0.00244 [I 2022-10-28 02:55:25] ...trap/engines/engine.py.230: loss: 0.12051 [I 2022-10-28 02:55:25] ...optimizers/trijoint.py.151: total_norm: 12.054123 [I 2022-10-28 02:55:31] ...trap/engines/engine.py.224: train: epoch 102 | batch 830/2304 [I 2022-10-28 02:55:31] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:13:45 [I 2022-10-28 02:55:31] ...trap/engines/engine.py.229: process: 0.56443 | load: 0.00279 [I 2022-10-28 02:55:31] ...trap/engines/engine.py.230: loss: 0.12420 [I 2022-10-28 02:55:31] ...optimizers/trijoint.py.151: total_norm: 21.590456 [I 2022-10-28 02:55:36] ...trap/engines/engine.py.224: train: epoch 102 | batch 840/2304 [I 2022-10-28 02:55:36] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:13:36 [I 2022-10-28 02:55:36] ...trap/engines/engine.py.229: process: 0.57034 | load: 0.00240 [I 2022-10-28 02:55:36] ...trap/engines/engine.py.230: loss: 0.14689 [I 2022-10-28 02:55:36] ...optimizers/trijoint.py.151: total_norm: 10.031335 [I 2022-10-28 02:55:42] ...trap/engines/engine.py.224: train: epoch 102 | batch 850/2304 [I 2022-10-28 02:55:42] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:13:28 [I 2022-10-28 02:55:42] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00243 [I 2022-10-28 02:55:42] ...trap/engines/engine.py.230: loss: 0.11367 [I 2022-10-28 02:55:42] ...optimizers/trijoint.py.151: total_norm: 21.887083 [I 2022-10-28 02:55:48] ...trap/engines/engine.py.224: train: epoch 102 | batch 860/2304 [I 2022-10-28 02:55:48] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:13:24 [I 2022-10-28 02:55:48] ...trap/engines/engine.py.229: process: 0.54465 | load: 0.00241 [I 2022-10-28 02:55:48] ...trap/engines/engine.py.230: loss: 0.10193 [I 2022-10-28 02:55:48] ...optimizers/trijoint.py.151: total_norm: 17.934721 [I 2022-10-28 02:55:53] ...trap/engines/engine.py.224: train: epoch 102 | batch 870/2304 [I 2022-10-28 02:55:53] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:20 [I 2022-10-28 02:55:53] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00261 [I 2022-10-28 02:55:53] ...trap/engines/engine.py.230: loss: 0.11666 [I 2022-10-28 02:55:53] ...optimizers/trijoint.py.151: total_norm: 18.763893 [I 2022-10-28 02:55:59] ...trap/engines/engine.py.224: train: epoch 102 | batch 880/2304 [I 2022-10-28 02:55:59] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:13:17 [I 2022-10-28 02:55:59] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00250 [I 2022-10-28 02:55:59] ...trap/engines/engine.py.230: loss: 0.12212 [I 2022-10-28 02:55:59] ...optimizers/trijoint.py.151: total_norm: 18.371403 [I 2022-10-28 02:56:04] ...trap/engines/engine.py.224: train: epoch 102 | batch 890/2304 [I 2022-10-28 02:56:04] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:13:11 [I 2022-10-28 02:56:04] ...trap/engines/engine.py.229: process: 0.55676 | load: 0.00244 [I 2022-10-28 02:56:04] ...trap/engines/engine.py.230: loss: 0.08914 [I 2022-10-28 02:56:04] ...optimizers/trijoint.py.151: total_norm: 10.127920 [I 2022-10-28 02:56:10] ...trap/engines/engine.py.224: train: epoch 102 | batch 900/2304 [I 2022-10-28 02:56:10] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:03 [I 2022-10-28 02:56:10] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00247 [I 2022-10-28 02:56:10] ...trap/engines/engine.py.230: loss: 0.15457 [I 2022-10-28 02:56:10] ...optimizers/trijoint.py.151: total_norm: 16.861010 [I 2022-10-28 02:56:16] ...trap/engines/engine.py.224: train: epoch 102 | batch 910/2304 [I 2022-10-28 02:56:16] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:13:00 [I 2022-10-28 02:56:16] ...trap/engines/engine.py.229: process: 0.54771 | load: 0.00242 [I 2022-10-28 02:56:16] ...trap/engines/engine.py.230: loss: 0.10035 [I 2022-10-28 02:56:16] ...optimizers/trijoint.py.151: total_norm: 14.628337 [I 2022-10-28 02:56:21] ...trap/engines/engine.py.224: train: epoch 102 | batch 920/2304 [I 2022-10-28 02:56:21] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:12:49 [I 2022-10-28 02:56:21] ...trap/engines/engine.py.229: process: 0.55225 | load: 0.00244 [I 2022-10-28 02:56:21] ...trap/engines/engine.py.230: loss: 0.15027 [I 2022-10-28 02:56:21] ...optimizers/trijoint.py.151: total_norm: 10.978672 [I 2022-10-28 02:56:27] ...trap/engines/engine.py.224: train: epoch 102 | batch 930/2304 [I 2022-10-28 02:56:27] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:12:42 [I 2022-10-28 02:56:27] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00243 [I 2022-10-28 02:56:27] ...trap/engines/engine.py.230: loss: 0.12005 [I 2022-10-28 02:56:27] ...optimizers/trijoint.py.151: total_norm: 7.812536 [I 2022-10-28 02:56:32] ...trap/engines/engine.py.224: train: epoch 102 | batch 940/2304 [I 2022-10-28 02:56:32] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:12:46 [I 2022-10-28 02:56:32] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00245 [I 2022-10-28 02:56:32] ...trap/engines/engine.py.230: loss: 0.11785 [I 2022-10-28 02:56:32] ...optimizers/trijoint.py.151: total_norm: 13.124271 [I 2022-10-28 02:56:38] ...trap/engines/engine.py.224: train: epoch 102 | batch 950/2304 [I 2022-10-28 02:56:38] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:12:36 [I 2022-10-28 02:56:38] ...trap/engines/engine.py.229: process: 0.56091 | load: 0.00244 [I 2022-10-28 02:56:38] ...trap/engines/engine.py.230: loss: 0.12040 [I 2022-10-28 02:56:38] ...optimizers/trijoint.py.151: total_norm: 15.362045 [I 2022-10-28 02:56:44] ...trap/engines/engine.py.224: train: epoch 102 | batch 960/2304 [I 2022-10-28 02:56:44] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:38 [I 2022-10-28 02:56:44] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00241 [I 2022-10-28 02:56:44] ...trap/engines/engine.py.230: loss: 0.09176 [I 2022-10-28 02:56:44] ...optimizers/trijoint.py.151: total_norm: 10.898133 [I 2022-10-28 02:56:49] ...trap/engines/engine.py.224: train: epoch 102 | batch 970/2304 [I 2022-10-28 02:56:49] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:57 [I 2022-10-28 02:56:49] ...trap/engines/engine.py.229: process: 0.67527 | load: 0.00255 [I 2022-10-28 02:56:49] ...trap/engines/engine.py.230: loss: 0.16401 [I 2022-10-28 02:56:49] ...optimizers/trijoint.py.151: total_norm: 15.925434 [I 2022-10-28 02:56:55] ...trap/engines/engine.py.224: train: epoch 102 | batch 980/2304 [I 2022-10-28 02:56:55] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:19 [I 2022-10-28 02:56:55] ...trap/engines/engine.py.229: process: 0.55879 | load: 0.00246 [I 2022-10-28 02:56:55] ...trap/engines/engine.py.230: loss: 0.12684 [I 2022-10-28 02:56:55] ...optimizers/trijoint.py.151: total_norm: 14.328503 [I 2022-10-28 02:57:01] ...trap/engines/engine.py.224: train: epoch 102 | batch 990/2304 [I 2022-10-28 02:57:01] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:09 [I 2022-10-28 02:57:01] ...trap/engines/engine.py.229: process: 0.55066 | load: 0.00246 [I 2022-10-28 02:57:01] ...trap/engines/engine.py.230: loss: 0.09928 [I 2022-10-28 02:57:01] ...optimizers/trijoint.py.151: total_norm: 28.303078 [I 2022-10-28 02:57:06] ...trap/engines/engine.py.224: train: epoch 102 | batch 1000/2304 [I 2022-10-28 02:57:06] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:05 [I 2022-10-28 02:57:06] ...trap/engines/engine.py.229: process: 0.55425 | load: 0.00246 [I 2022-10-28 02:57:06] ...trap/engines/engine.py.230: loss: 0.14018 [I 2022-10-28 02:57:06] ...optimizers/trijoint.py.151: total_norm: 26.642895 [I 2022-10-28 02:57:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 1010/2304 [I 2022-10-28 02:57:12] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:01 [I 2022-10-28 02:57:12] ...trap/engines/engine.py.229: process: 0.55790 | load: 0.00248 [I 2022-10-28 02:57:12] ...trap/engines/engine.py.230: loss: 0.11863 [I 2022-10-28 02:57:12] ...optimizers/trijoint.py.151: total_norm: 14.897079 [I 2022-10-28 02:57:17] ...trap/engines/engine.py.224: train: epoch 102 | batch 1020/2304 [I 2022-10-28 02:57:17] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:11:55 [I 2022-10-28 02:57:17] ...trap/engines/engine.py.229: process: 0.56597 | load: 0.00241 [I 2022-10-28 02:57:17] ...trap/engines/engine.py.230: loss: 0.16403 [I 2022-10-28 02:57:17] ...optimizers/trijoint.py.151: total_norm: 15.543818 [I 2022-10-28 02:57:23] ...trap/engines/engine.py.224: train: epoch 102 | batch 1030/2304 [I 2022-10-28 02:57:23] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:11:48 [I 2022-10-28 02:57:23] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00244 [I 2022-10-28 02:57:23] ...trap/engines/engine.py.230: loss: 0.13766 [I 2022-10-28 02:57:23] ...optimizers/trijoint.py.151: total_norm: 12.057775 [I 2022-10-28 02:57:28] ...trap/engines/engine.py.224: train: epoch 102 | batch 1040/2304 [I 2022-10-28 02:57:28] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:11:47 [I 2022-10-28 02:57:28] ...trap/engines/engine.py.229: process: 0.56417 | load: 0.00242 [I 2022-10-28 02:57:28] ...trap/engines/engine.py.230: loss: 0.09868 [I 2022-10-28 02:57:28] ...optimizers/trijoint.py.151: total_norm: 13.599130 [I 2022-10-28 02:57:34] ...trap/engines/engine.py.224: train: epoch 102 | batch 1050/2304 [I 2022-10-28 02:57:34] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:46 [I 2022-10-28 02:57:34] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00242 [I 2022-10-28 02:57:34] ...trap/engines/engine.py.230: loss: 0.08930 [I 2022-10-28 02:57:34] ...optimizers/trijoint.py.151: total_norm: 19.504065 [I 2022-10-28 02:57:40] ...trap/engines/engine.py.224: train: epoch 102 | batch 1060/2304 [I 2022-10-28 02:57:40] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:32 [I 2022-10-28 02:57:40] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00241 [I 2022-10-28 02:57:40] ...trap/engines/engine.py.230: loss: 0.09506 [I 2022-10-28 02:57:40] ...optimizers/trijoint.py.151: total_norm: 20.442823 [I 2022-10-28 02:57:45] ...trap/engines/engine.py.224: train: epoch 102 | batch 1070/2304 [I 2022-10-28 02:57:45] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:27 [I 2022-10-28 02:57:45] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00242 [I 2022-10-28 02:57:45] ...trap/engines/engine.py.230: loss: 0.09990 [I 2022-10-28 02:57:45] ...optimizers/trijoint.py.151: total_norm: 20.503330 [I 2022-10-28 02:57:51] ...trap/engines/engine.py.224: train: epoch 102 | batch 1080/2304 [I 2022-10-28 02:57:51] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:20 [I 2022-10-28 02:57:51] ...trap/engines/engine.py.229: process: 0.56413 | load: 0.00242 [I 2022-10-28 02:57:51] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-28 02:57:51] ...optimizers/trijoint.py.151: total_norm: 11.950709 [I 2022-10-28 02:57:57] ...trap/engines/engine.py.224: train: epoch 102 | batch 1090/2304 [I 2022-10-28 02:57:57] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:16 [I 2022-10-28 02:57:57] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00242 [I 2022-10-28 02:57:57] ...trap/engines/engine.py.230: loss: 0.15913 [I 2022-10-28 02:57:57] ...optimizers/trijoint.py.151: total_norm: 16.640831 [I 2022-10-28 02:58:02] ...trap/engines/engine.py.224: train: epoch 102 | batch 1100/2304 [I 2022-10-28 02:58:02] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:08 [I 2022-10-28 02:58:02] ...trap/engines/engine.py.229: process: 0.54995 | load: 0.00254 [I 2022-10-28 02:58:02] ...trap/engines/engine.py.230: loss: 0.14830 [I 2022-10-28 02:58:02] ...optimizers/trijoint.py.151: total_norm: 17.283993 [I 2022-10-28 02:58:08] ...trap/engines/engine.py.224: train: epoch 102 | batch 1110/2304 [I 2022-10-28 02:58:08] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:07 [I 2022-10-28 02:58:08] ...trap/engines/engine.py.229: process: 0.55740 | load: 0.00247 [I 2022-10-28 02:58:08] ...trap/engines/engine.py.230: loss: 0.11226 [I 2022-10-28 02:58:08] ...optimizers/trijoint.py.151: total_norm: 31.768427 [I 2022-10-28 02:58:13] ...trap/engines/engine.py.224: train: epoch 102 | batch 1120/2304 [I 2022-10-28 02:58:13] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:00 [I 2022-10-28 02:58:13] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00246 [I 2022-10-28 02:58:13] ...trap/engines/engine.py.230: loss: 0.08588 [I 2022-10-28 02:58:13] ...optimizers/trijoint.py.151: total_norm: 9.379659 [I 2022-10-28 02:58:19] ...trap/engines/engine.py.224: train: epoch 102 | batch 1130/2304 [I 2022-10-28 02:58:19] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:10:55 [I 2022-10-28 02:58:19] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00239 [I 2022-10-28 02:58:19] ...trap/engines/engine.py.230: loss: 0.19570 [I 2022-10-28 02:58:19] ...optimizers/trijoint.py.151: total_norm: 17.470535 [I 2022-10-28 02:58:25] ...trap/engines/engine.py.224: train: epoch 102 | batch 1140/2304 [I 2022-10-28 02:58:25] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:46 [I 2022-10-28 02:58:25] ...trap/engines/engine.py.229: process: 0.55152 | load: 0.00243 [I 2022-10-28 02:58:25] ...trap/engines/engine.py.230: loss: 0.14421 [I 2022-10-28 02:58:25] ...optimizers/trijoint.py.151: total_norm: 23.838875 [I 2022-10-28 02:58:30] ...trap/engines/engine.py.224: train: epoch 102 | batch 1150/2304 [I 2022-10-28 02:58:30] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:39 [I 2022-10-28 02:58:30] ...trap/engines/engine.py.229: process: 0.55069 | load: 0.00243 [I 2022-10-28 02:58:30] ...trap/engines/engine.py.230: loss: 0.07323 [I 2022-10-28 02:58:30] ...optimizers/trijoint.py.151: total_norm: 17.569876 [I 2022-10-28 02:58:36] ...trap/engines/engine.py.224: train: epoch 102 | batch 1160/2304 [I 2022-10-28 02:58:36] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:37 [I 2022-10-28 02:58:36] ...trap/engines/engine.py.229: process: 0.55212 | load: 0.00242 [I 2022-10-28 02:58:36] ...trap/engines/engine.py.230: loss: 0.13252 [I 2022-10-28 02:58:36] ...optimizers/trijoint.py.151: total_norm: 19.990574 [I 2022-10-28 02:58:41] ...trap/engines/engine.py.224: train: epoch 102 | batch 1170/2304 [I 2022-10-28 02:58:41] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:30 [I 2022-10-28 02:58:41] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00248 [I 2022-10-28 02:58:41] ...trap/engines/engine.py.230: loss: 0.09053 [I 2022-10-28 02:58:41] ...optimizers/trijoint.py.151: total_norm: 12.982200 [I 2022-10-28 02:58:47] ...trap/engines/engine.py.224: train: epoch 102 | batch 1180/2304 [I 2022-10-28 02:58:47] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:28 [I 2022-10-28 02:58:47] ...trap/engines/engine.py.229: process: 0.56138 | load: 0.00247 [I 2022-10-28 02:58:47] ...trap/engines/engine.py.230: loss: 0.16872 [I 2022-10-28 02:58:47] ...optimizers/trijoint.py.151: total_norm: 13.836371 [I 2022-10-28 02:58:52] ...trap/engines/engine.py.224: train: epoch 102 | batch 1190/2304 [I 2022-10-28 02:58:52] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:18 [I 2022-10-28 02:58:52] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00247 [I 2022-10-28 02:58:52] ...trap/engines/engine.py.230: loss: 0.09521 [I 2022-10-28 02:58:52] ...optimizers/trijoint.py.151: total_norm: 10.233756 [I 2022-10-28 02:58:58] ...trap/engines/engine.py.224: train: epoch 102 | batch 1200/2304 [I 2022-10-28 02:58:58] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:15 [I 2022-10-28 02:58:58] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00242 [I 2022-10-28 02:58:58] ...trap/engines/engine.py.230: loss: 0.10185 [I 2022-10-28 02:58:58] ...optimizers/trijoint.py.151: total_norm: 13.492783 [I 2022-10-28 02:59:04] ...trap/engines/engine.py.224: train: epoch 102 | batch 1210/2304 [I 2022-10-28 02:59:04] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:06 [I 2022-10-28 02:59:04] ...trap/engines/engine.py.229: process: 0.55145 | load: 0.00244 [I 2022-10-28 02:59:04] ...trap/engines/engine.py.230: loss: 0.13827 [I 2022-10-28 02:59:04] ...optimizers/trijoint.py.151: total_norm: 24.288897 [I 2022-10-28 02:59:09] ...trap/engines/engine.py.224: train: epoch 102 | batch 1220/2304 [I 2022-10-28 02:59:09] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:06 [I 2022-10-28 02:59:09] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00242 [I 2022-10-28 02:59:09] ...trap/engines/engine.py.230: loss: 0.11990 [I 2022-10-28 02:59:09] ...optimizers/trijoint.py.151: total_norm: 12.577854 [I 2022-10-28 02:59:15] ...trap/engines/engine.py.224: train: epoch 102 | batch 1230/2304 [I 2022-10-28 02:59:15] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:07 [I 2022-10-28 02:59:15] ...trap/engines/engine.py.229: process: 0.57297 | load: 0.00244 [I 2022-10-28 02:59:15] ...trap/engines/engine.py.230: loss: 0.11406 [I 2022-10-28 02:59:15] ...optimizers/trijoint.py.151: total_norm: 20.220900 [I 2022-10-28 02:59:20] ...trap/engines/engine.py.224: train: epoch 102 | batch 1240/2304 [I 2022-10-28 02:59:20] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:09:54 [I 2022-10-28 02:59:20] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00245 [I 2022-10-28 02:59:20] ...trap/engines/engine.py.230: loss: 0.21205 [I 2022-10-28 02:59:20] ...optimizers/trijoint.py.151: total_norm: 19.507877 [I 2022-10-28 02:59:26] ...trap/engines/engine.py.224: train: epoch 102 | batch 1250/2304 [I 2022-10-28 02:59:26] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:09:46 [I 2022-10-28 02:59:26] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00240 [I 2022-10-28 02:59:26] ...trap/engines/engine.py.230: loss: 0.16397 [I 2022-10-28 02:59:26] ...optimizers/trijoint.py.151: total_norm: 16.450176 [I 2022-10-28 02:59:32] ...trap/engines/engine.py.224: train: epoch 102 | batch 1260/2304 [I 2022-10-28 02:59:32] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:09:40 [I 2022-10-28 02:59:32] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00242 [I 2022-10-28 02:59:32] ...trap/engines/engine.py.230: loss: 0.14610 [I 2022-10-28 02:59:32] ...optimizers/trijoint.py.151: total_norm: 10.623254 [I 2022-10-28 02:59:37] ...trap/engines/engine.py.224: train: epoch 102 | batch 1270/2304 [I 2022-10-28 02:59:37] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:32 [I 2022-10-28 02:59:37] ...trap/engines/engine.py.229: process: 0.54124 | load: 0.00242 [I 2022-10-28 02:59:37] ...trap/engines/engine.py.230: loss: 0.11348 [I 2022-10-28 02:59:37] ...optimizers/trijoint.py.151: total_norm: 9.877803 [I 2022-10-28 02:59:43] ...trap/engines/engine.py.224: train: epoch 102 | batch 1280/2304 [I 2022-10-28 02:59:43] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:29 [I 2022-10-28 02:59:43] ...trap/engines/engine.py.229: process: 0.55080 | load: 0.00242 [I 2022-10-28 02:59:43] ...trap/engines/engine.py.230: loss: 0.13824 [I 2022-10-28 02:59:43] ...optimizers/trijoint.py.151: total_norm: 30.185930 [I 2022-10-28 02:59:48] ...trap/engines/engine.py.224: train: epoch 102 | batch 1290/2304 [I 2022-10-28 02:59:48] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:26 [I 2022-10-28 02:59:48] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00252 [I 2022-10-28 02:59:48] ...trap/engines/engine.py.230: loss: 0.12283 [I 2022-10-28 02:59:48] ...optimizers/trijoint.py.151: total_norm: 13.148773 [I 2022-10-28 02:59:54] ...trap/engines/engine.py.224: train: epoch 102 | batch 1300/2304 [I 2022-10-28 02:59:54] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:17 [I 2022-10-28 02:59:54] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00247 [I 2022-10-28 02:59:54] ...trap/engines/engine.py.230: loss: 0.10420 [I 2022-10-28 02:59:54] ...optimizers/trijoint.py.151: total_norm: 12.851215 [I 2022-10-28 02:59:59] ...trap/engines/engine.py.224: train: epoch 102 | batch 1310/2304 [I 2022-10-28 02:59:59] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:12 [I 2022-10-28 02:59:59] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00245 [I 2022-10-28 02:59:59] ...trap/engines/engine.py.230: loss: 0.09989 [I 2022-10-28 02:59:59] ...optimizers/trijoint.py.151: total_norm: 11.304465 [I 2022-10-28 03:00:05] ...trap/engines/engine.py.224: train: epoch 102 | batch 1320/2304 [I 2022-10-28 03:00:05] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:08 [I 2022-10-28 03:00:05] ...trap/engines/engine.py.229: process: 0.56607 | load: 0.00244 [I 2022-10-28 03:00:05] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-28 03:00:05] ...optimizers/trijoint.py.151: total_norm: 17.022314 [I 2022-10-28 03:00:11] ...trap/engines/engine.py.224: train: epoch 102 | batch 1330/2304 [I 2022-10-28 03:00:11] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:06 [I 2022-10-28 03:00:11] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00261 [I 2022-10-28 03:00:11] ...trap/engines/engine.py.230: loss: 0.08769 [I 2022-10-28 03:00:11] ...optimizers/trijoint.py.151: total_norm: 10.028392 [I 2022-10-28 03:00:16] ...trap/engines/engine.py.224: train: epoch 102 | batch 1340/2304 [I 2022-10-28 03:00:16] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:08:59 [I 2022-10-28 03:00:16] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00250 [I 2022-10-28 03:00:16] ...trap/engines/engine.py.230: loss: 0.09412 [I 2022-10-28 03:00:16] ...optimizers/trijoint.py.151: total_norm: 9.261734 [I 2022-10-28 03:00:22] ...trap/engines/engine.py.224: train: epoch 102 | batch 1350/2304 [I 2022-10-28 03:00:22] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:08:51 [I 2022-10-28 03:00:22] ...trap/engines/engine.py.229: process: 0.55384 | load: 0.00246 [I 2022-10-28 03:00:22] ...trap/engines/engine.py.230: loss: 0.09757 [I 2022-10-28 03:00:22] ...optimizers/trijoint.py.151: total_norm: 7.577430 [I 2022-10-28 03:00:27] ...trap/engines/engine.py.224: train: epoch 102 | batch 1360/2304 [I 2022-10-28 03:00:27] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:08:45 [I 2022-10-28 03:00:27] ...trap/engines/engine.py.229: process: 0.55926 | load: 0.00247 [I 2022-10-28 03:00:27] ...trap/engines/engine.py.230: loss: 0.13962 [I 2022-10-28 03:00:27] ...optimizers/trijoint.py.151: total_norm: 18.182022 [I 2022-10-28 03:00:33] ...trap/engines/engine.py.224: train: epoch 102 | batch 1370/2304 [I 2022-10-28 03:00:33] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:40 [I 2022-10-28 03:00:33] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00251 [I 2022-10-28 03:00:33] ...trap/engines/engine.py.230: loss: 0.07985 [I 2022-10-28 03:00:33] ...optimizers/trijoint.py.151: total_norm: 12.059888 [I 2022-10-28 03:00:39] ...trap/engines/engine.py.224: train: epoch 102 | batch 1380/2304 [I 2022-10-28 03:00:39] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:33 [I 2022-10-28 03:00:39] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00241 [I 2022-10-28 03:00:39] ...trap/engines/engine.py.230: loss: 0.11545 [I 2022-10-28 03:00:39] ...optimizers/trijoint.py.151: total_norm: 8.717471 [I 2022-10-28 03:00:44] ...trap/engines/engine.py.224: train: epoch 102 | batch 1390/2304 [I 2022-10-28 03:00:44] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:26 [I 2022-10-28 03:00:44] ...trap/engines/engine.py.229: process: 0.55482 | load: 0.00249 [I 2022-10-28 03:00:44] ...trap/engines/engine.py.230: loss: 0.14783 [I 2022-10-28 03:00:44] ...optimizers/trijoint.py.151: total_norm: 17.009058 [I 2022-10-28 03:00:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 1400/2304 [I 2022-10-28 03:00:50] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:24 [I 2022-10-28 03:00:50] ...trap/engines/engine.py.229: process: 0.56372 | load: 0.00246 [I 2022-10-28 03:00:50] ...trap/engines/engine.py.230: loss: 0.10323 [I 2022-10-28 03:00:50] ...optimizers/trijoint.py.151: total_norm: 11.591594 [I 2022-10-28 03:00:55] ...trap/engines/engine.py.224: train: epoch 102 | batch 1410/2304 [I 2022-10-28 03:00:55] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:20 [I 2022-10-28 03:00:55] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00245 [I 2022-10-28 03:00:55] ...trap/engines/engine.py.230: loss: 0.09511 [I 2022-10-28 03:00:55] ...optimizers/trijoint.py.151: total_norm: 7.582415 [I 2022-10-28 03:01:01] ...trap/engines/engine.py.224: train: epoch 102 | batch 1420/2304 [I 2022-10-28 03:01:01] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:17 [I 2022-10-28 03:01:01] ...trap/engines/engine.py.229: process: 0.56837 | load: 0.00242 [I 2022-10-28 03:01:01] ...trap/engines/engine.py.230: loss: 0.19531 [I 2022-10-28 03:01:01] ...optimizers/trijoint.py.151: total_norm: 36.248615 [I 2022-10-28 03:01:07] ...trap/engines/engine.py.224: train: epoch 102 | batch 1430/2304 [I 2022-10-28 03:01:07] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:08 [I 2022-10-28 03:01:07] ...trap/engines/engine.py.229: process: 0.55777 | load: 0.00243 [I 2022-10-28 03:01:07] ...trap/engines/engine.py.230: loss: 0.11775 [I 2022-10-28 03:01:07] ...optimizers/trijoint.py.151: total_norm: 16.221806 [I 2022-10-28 03:01:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 1440/2304 [I 2022-10-28 03:01:12] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:01 [I 2022-10-28 03:01:12] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00244 [I 2022-10-28 03:01:12] ...trap/engines/engine.py.230: loss: 0.09128 [I 2022-10-28 03:01:12] ...optimizers/trijoint.py.151: total_norm: 11.204783 [I 2022-10-28 03:01:18] ...trap/engines/engine.py.224: train: epoch 102 | batch 1450/2304 [I 2022-10-28 03:01:18] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:07:58 [I 2022-10-28 03:01:18] ...trap/engines/engine.py.229: process: 0.54627 | load: 0.00243 [I 2022-10-28 03:01:18] ...trap/engines/engine.py.230: loss: 0.10214 [I 2022-10-28 03:01:18] ...optimizers/trijoint.py.151: total_norm: 12.348240 [I 2022-10-28 03:01:24] ...trap/engines/engine.py.224: train: epoch 102 | batch 1460/2304 [I 2022-10-28 03:01:24] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:07:51 [I 2022-10-28 03:01:24] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00240 [I 2022-10-28 03:01:24] ...trap/engines/engine.py.230: loss: 0.15825 [I 2022-10-28 03:01:24] ...optimizers/trijoint.py.151: total_norm: 27.761028 [I 2022-10-28 03:01:29] ...trap/engines/engine.py.224: train: epoch 102 | batch 1470/2304 [I 2022-10-28 03:01:29] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:07:47 [I 2022-10-28 03:01:29] ...trap/engines/engine.py.229: process: 0.57558 | load: 0.00264 [I 2022-10-28 03:01:29] ...trap/engines/engine.py.230: loss: 0.10241 [I 2022-10-28 03:01:29] ...optimizers/trijoint.py.151: total_norm: 9.251627 [I 2022-10-28 03:01:35] ...trap/engines/engine.py.224: train: epoch 102 | batch 1480/2304 [I 2022-10-28 03:01:35] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:39 [I 2022-10-28 03:01:35] ...trap/engines/engine.py.229: process: 0.54850 | load: 0.00240 [I 2022-10-28 03:01:35] ...trap/engines/engine.py.230: loss: 0.09338 [I 2022-10-28 03:01:35] ...optimizers/trijoint.py.151: total_norm: 9.047746 [I 2022-10-28 03:01:40] ...trap/engines/engine.py.224: train: epoch 102 | batch 1490/2304 [I 2022-10-28 03:01:40] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:35 [I 2022-10-28 03:01:40] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00243 [I 2022-10-28 03:01:40] ...trap/engines/engine.py.230: loss: 0.08720 [I 2022-10-28 03:01:40] ...optimizers/trijoint.py.151: total_norm: 11.819934 [I 2022-10-28 03:01:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 1500/2304 [I 2022-10-28 03:01:46] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:33 [I 2022-10-28 03:01:46] ...trap/engines/engine.py.229: process: 0.56512 | load: 0.00241 [I 2022-10-28 03:01:46] ...trap/engines/engine.py.230: loss: 0.15906 [I 2022-10-28 03:01:46] ...optimizers/trijoint.py.151: total_norm: 35.571621 [I 2022-10-28 03:01:52] ...trap/engines/engine.py.224: train: epoch 102 | batch 1510/2304 [I 2022-10-28 03:01:52] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:22 [I 2022-10-28 03:01:52] ...trap/engines/engine.py.229: process: 0.55641 | load: 0.00255 [I 2022-10-28 03:01:52] ...trap/engines/engine.py.230: loss: 0.09858 [I 2022-10-28 03:01:52] ...optimizers/trijoint.py.151: total_norm: 30.631672 [I 2022-10-28 03:01:57] ...trap/engines/engine.py.224: train: epoch 102 | batch 1520/2304 [I 2022-10-28 03:01:57] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:17 [I 2022-10-28 03:01:57] ...trap/engines/engine.py.229: process: 0.56181 | load: 0.00247 [I 2022-10-28 03:01:57] ...trap/engines/engine.py.230: loss: 0.18383 [I 2022-10-28 03:01:57] ...optimizers/trijoint.py.151: total_norm: 30.811798 [I 2022-10-28 03:02:03] ...trap/engines/engine.py.224: train: epoch 102 | batch 1530/2304 [I 2022-10-28 03:02:03] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:14 [I 2022-10-28 03:02:03] ...trap/engines/engine.py.229: process: 0.56301 | load: 0.00248 [I 2022-10-28 03:02:03] ...trap/engines/engine.py.230: loss: 0.11517 [I 2022-10-28 03:02:03] ...optimizers/trijoint.py.151: total_norm: 13.688767 [I 2022-10-28 03:02:08] ...trap/engines/engine.py.224: train: epoch 102 | batch 1540/2304 [I 2022-10-28 03:02:08] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:04 [I 2022-10-28 03:02:08] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00244 [I 2022-10-28 03:02:08] ...trap/engines/engine.py.230: loss: 0.13288 [I 2022-10-28 03:02:08] ...optimizers/trijoint.py.151: total_norm: 16.993299 [I 2022-10-28 03:02:14] ...trap/engines/engine.py.224: train: epoch 102 | batch 1550/2304 [I 2022-10-28 03:02:14] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:00 [I 2022-10-28 03:02:14] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00243 [I 2022-10-28 03:02:14] ...trap/engines/engine.py.230: loss: 0.13539 [I 2022-10-28 03:02:14] ...optimizers/trijoint.py.151: total_norm: 23.741062 [I 2022-10-28 03:02:19] ...trap/engines/engine.py.224: train: epoch 102 | batch 1560/2304 [I 2022-10-28 03:02:19] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:06:53 [I 2022-10-28 03:02:19] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00245 [I 2022-10-28 03:02:19] ...trap/engines/engine.py.230: loss: 0.11189 [I 2022-10-28 03:02:19] ...optimizers/trijoint.py.151: total_norm: 18.805969 [I 2022-10-28 03:02:25] ...trap/engines/engine.py.224: train: epoch 102 | batch 1570/2304 [I 2022-10-28 03:02:25] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:06:47 [I 2022-10-28 03:02:25] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00257 [I 2022-10-28 03:02:25] ...trap/engines/engine.py.230: loss: 0.17921 [I 2022-10-28 03:02:25] ...optimizers/trijoint.py.151: total_norm: 25.360600 [I 2022-10-28 03:02:31] ...trap/engines/engine.py.224: train: epoch 102 | batch 1580/2304 [I 2022-10-28 03:02:31] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:06:44 [I 2022-10-28 03:02:31] ...trap/engines/engine.py.229: process: 0.56010 | load: 0.00255 [I 2022-10-28 03:02:31] ...trap/engines/engine.py.230: loss: 0.15113 [I 2022-10-28 03:02:31] ...optimizers/trijoint.py.151: total_norm: 16.569527 [I 2022-10-28 03:02:36] ...trap/engines/engine.py.224: train: epoch 102 | batch 1590/2304 [I 2022-10-28 03:02:36] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:37 [I 2022-10-28 03:02:36] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00251 [I 2022-10-28 03:02:36] ...trap/engines/engine.py.230: loss: 0.11766 [I 2022-10-28 03:02:36] ...optimizers/trijoint.py.151: total_norm: 11.050186 [I 2022-10-28 03:02:42] ...trap/engines/engine.py.224: train: epoch 102 | batch 1600/2304 [I 2022-10-28 03:02:42] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:32 [I 2022-10-28 03:02:42] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00243 [I 2022-10-28 03:02:42] ...trap/engines/engine.py.230: loss: 0.13541 [I 2022-10-28 03:02:42] ...optimizers/trijoint.py.151: total_norm: 8.542924 [I 2022-10-28 03:02:47] ...trap/engines/engine.py.224: train: epoch 102 | batch 1610/2304 [I 2022-10-28 03:02:47] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:25 [I 2022-10-28 03:02:47] ...trap/engines/engine.py.229: process: 0.55408 | load: 0.00253 [I 2022-10-28 03:02:47] ...trap/engines/engine.py.230: loss: 0.23709 [I 2022-10-28 03:02:47] ...optimizers/trijoint.py.151: total_norm: 22.091116 [I 2022-10-28 03:02:53] ...trap/engines/engine.py.224: train: epoch 102 | batch 1620/2304 [I 2022-10-28 03:02:53] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:20 [I 2022-10-28 03:02:53] ...trap/engines/engine.py.229: process: 0.54848 | load: 0.00265 [I 2022-10-28 03:02:53] ...trap/engines/engine.py.230: loss: 0.09772 [I 2022-10-28 03:02:53] ...optimizers/trijoint.py.151: total_norm: 16.069307 [I 2022-10-28 03:02:58] ...trap/engines/engine.py.224: train: epoch 102 | batch 1630/2304 [I 2022-10-28 03:02:58] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:15 [I 2022-10-28 03:02:58] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00247 [I 2022-10-28 03:02:58] ...trap/engines/engine.py.230: loss: 0.15264 [I 2022-10-28 03:02:58] ...optimizers/trijoint.py.151: total_norm: 34.538246 [I 2022-10-28 03:03:04] ...trap/engines/engine.py.224: train: epoch 102 | batch 1640/2304 [I 2022-10-28 03:03:04] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:10 [I 2022-10-28 03:03:04] ...trap/engines/engine.py.229: process: 0.55344 | load: 0.00242 [I 2022-10-28 03:03:04] ...trap/engines/engine.py.230: loss: 0.12685 [I 2022-10-28 03:03:04] ...optimizers/trijoint.py.151: total_norm: 16.115604 [I 2022-10-28 03:03:10] ...trap/engines/engine.py.224: train: epoch 102 | batch 1650/2304 [I 2022-10-28 03:03:10] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:01 [I 2022-10-28 03:03:10] ...trap/engines/engine.py.229: process: 0.55045 | load: 0.00244 [I 2022-10-28 03:03:10] ...trap/engines/engine.py.230: loss: 0.10824 [I 2022-10-28 03:03:10] ...optimizers/trijoint.py.151: total_norm: 9.537541 [I 2022-10-28 03:03:15] ...trap/engines/engine.py.224: train: epoch 102 | batch 1660/2304 [I 2022-10-28 03:03:15] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:05:57 [I 2022-10-28 03:03:15] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00244 [I 2022-10-28 03:03:15] ...trap/engines/engine.py.230: loss: 0.14979 [I 2022-10-28 03:03:15] ...optimizers/trijoint.py.151: total_norm: 37.370529 [I 2022-10-28 03:03:21] ...trap/engines/engine.py.224: train: epoch 102 | batch 1670/2304 [I 2022-10-28 03:03:21] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:05:52 [I 2022-10-28 03:03:21] ...trap/engines/engine.py.229: process: 0.55184 | load: 0.00251 [I 2022-10-28 03:03:21] ...trap/engines/engine.py.230: loss: 0.08660 [I 2022-10-28 03:03:21] ...optimizers/trijoint.py.151: total_norm: 12.260518 [I 2022-10-28 03:03:26] ...trap/engines/engine.py.224: train: epoch 102 | batch 1680/2304 [I 2022-10-28 03:03:26] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:05:47 [I 2022-10-28 03:03:26] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00256 [I 2022-10-28 03:03:26] ...trap/engines/engine.py.230: loss: 0.12021 [I 2022-10-28 03:03:26] ...optimizers/trijoint.py.151: total_norm: 13.932474 [I 2022-10-28 03:03:32] ...trap/engines/engine.py.224: train: epoch 102 | batch 1690/2304 [I 2022-10-28 03:03:32] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:05:40 [I 2022-10-28 03:03:32] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00255 [I 2022-10-28 03:03:32] ...trap/engines/engine.py.230: loss: 0.11759 [I 2022-10-28 03:03:32] ...optimizers/trijoint.py.151: total_norm: 12.347264 [I 2022-10-28 03:03:37] ...trap/engines/engine.py.224: train: epoch 102 | batch 1700/2304 [I 2022-10-28 03:03:37] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:35 [I 2022-10-28 03:03:37] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00245 [I 2022-10-28 03:03:37] ...trap/engines/engine.py.230: loss: 0.11633 [I 2022-10-28 03:03:37] ...optimizers/trijoint.py.151: total_norm: 12.375390 [I 2022-10-28 03:03:43] ...trap/engines/engine.py.224: train: epoch 102 | batch 1710/2304 [I 2022-10-28 03:03:43] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:29 [I 2022-10-28 03:03:43] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00243 [I 2022-10-28 03:03:43] ...trap/engines/engine.py.230: loss: 0.10422 [I 2022-10-28 03:03:43] ...optimizers/trijoint.py.151: total_norm: 11.364553 [I 2022-10-28 03:03:49] ...trap/engines/engine.py.224: train: epoch 102 | batch 1720/2304 [I 2022-10-28 03:03:49] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:23 [I 2022-10-28 03:03:49] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00245 [I 2022-10-28 03:03:49] ...trap/engines/engine.py.230: loss: 0.14061 [I 2022-10-28 03:03:49] ...optimizers/trijoint.py.151: total_norm: 16.649229 [I 2022-10-28 03:03:54] ...trap/engines/engine.py.224: train: epoch 102 | batch 1730/2304 [I 2022-10-28 03:03:54] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:18 [I 2022-10-28 03:03:54] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00241 [I 2022-10-28 03:03:54] ...trap/engines/engine.py.230: loss: 0.10822 [I 2022-10-28 03:03:54] ...optimizers/trijoint.py.151: total_norm: 8.857880 [I 2022-10-28 03:04:00] ...trap/engines/engine.py.224: train: epoch 102 | batch 1740/2304 [I 2022-10-28 03:04:00] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:15 [I 2022-10-28 03:04:00] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00249 [I 2022-10-28 03:04:00] ...trap/engines/engine.py.230: loss: 0.10673 [I 2022-10-28 03:04:00] ...optimizers/trijoint.py.151: total_norm: 14.438064 [I 2022-10-28 03:04:05] ...trap/engines/engine.py.224: train: epoch 102 | batch 1750/2304 [I 2022-10-28 03:04:05] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:09 [I 2022-10-28 03:04:05] ...trap/engines/engine.py.229: process: 0.56416 | load: 0.00246 [I 2022-10-28 03:04:05] ...trap/engines/engine.py.230: loss: 0.11958 [I 2022-10-28 03:04:05] ...optimizers/trijoint.py.151: total_norm: 9.136444 [I 2022-10-28 03:04:11] ...trap/engines/engine.py.224: train: epoch 102 | batch 1760/2304 [I 2022-10-28 03:04:11] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:04 [I 2022-10-28 03:04:11] ...trap/engines/engine.py.229: process: 0.56561 | load: 0.00245 [I 2022-10-28 03:04:11] ...trap/engines/engine.py.230: loss: 0.11358 [I 2022-10-28 03:04:11] ...optimizers/trijoint.py.151: total_norm: 9.621331 [I 2022-10-28 03:04:17] ...trap/engines/engine.py.224: train: epoch 102 | batch 1770/2304 [I 2022-10-28 03:04:17] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:04:57 [I 2022-10-28 03:04:17] ...trap/engines/engine.py.229: process: 0.54904 | load: 0.00245 [I 2022-10-28 03:04:17] ...trap/engines/engine.py.230: loss: 0.16051 [I 2022-10-28 03:04:17] ...optimizers/trijoint.py.151: total_norm: 7.593534 [I 2022-10-28 03:04:22] ...trap/engines/engine.py.224: train: epoch 102 | batch 1780/2304 [I 2022-10-28 03:04:22] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:04:53 [I 2022-10-28 03:04:22] ...trap/engines/engine.py.229: process: 0.56509 | load: 0.00247 [I 2022-10-28 03:04:22] ...trap/engines/engine.py.230: loss: 0.10366 [I 2022-10-28 03:04:22] ...optimizers/trijoint.py.151: total_norm: 15.718086 [I 2022-10-28 03:04:28] ...trap/engines/engine.py.224: train: epoch 102 | batch 1790/2304 [I 2022-10-28 03:04:28] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:04:47 [I 2022-10-28 03:04:28] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00244 [I 2022-10-28 03:04:28] ...trap/engines/engine.py.230: loss: 0.09962 [I 2022-10-28 03:04:28] ...optimizers/trijoint.py.151: total_norm: 9.476540 [I 2022-10-28 03:04:33] ...trap/engines/engine.py.224: train: epoch 102 | batch 1800/2304 [I 2022-10-28 03:04:33] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:40 [I 2022-10-28 03:04:33] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00245 [I 2022-10-28 03:04:33] ...trap/engines/engine.py.230: loss: 0.12880 [I 2022-10-28 03:04:33] ...optimizers/trijoint.py.151: total_norm: 13.942389 [I 2022-10-28 03:04:39] ...trap/engines/engine.py.224: train: epoch 102 | batch 1810/2304 [I 2022-10-28 03:04:39] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:36 [I 2022-10-28 03:04:39] ...trap/engines/engine.py.229: process: 0.55567 | load: 0.00242 [I 2022-10-28 03:04:39] ...trap/engines/engine.py.230: loss: 0.09691 [I 2022-10-28 03:04:39] ...optimizers/trijoint.py.151: total_norm: 14.966393 [I 2022-10-28 03:04:44] ...trap/engines/engine.py.224: train: epoch 102 | batch 1820/2304 [I 2022-10-28 03:04:44] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:28 [I 2022-10-28 03:04:44] ...trap/engines/engine.py.229: process: 0.55954 | load: 0.00253 [I 2022-10-28 03:04:44] ...trap/engines/engine.py.230: loss: 0.08145 [I 2022-10-28 03:04:44] ...optimizers/trijoint.py.151: total_norm: 8.723152 [I 2022-10-28 03:04:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 1830/2304 [I 2022-10-28 03:04:50] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:24 [I 2022-10-28 03:04:50] ...trap/engines/engine.py.229: process: 0.55228 | load: 0.00248 [I 2022-10-28 03:04:50] ...trap/engines/engine.py.230: loss: 0.11579 [I 2022-10-28 03:04:50] ...optimizers/trijoint.py.151: total_norm: 16.515970 [I 2022-10-28 03:04:56] ...trap/engines/engine.py.224: train: epoch 102 | batch 1840/2304 [I 2022-10-28 03:04:56] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:19 [I 2022-10-28 03:04:56] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00249 [I 2022-10-28 03:04:56] ...trap/engines/engine.py.230: loss: 0.14970 [I 2022-10-28 03:04:56] ...optimizers/trijoint.py.151: total_norm: 12.208659 [I 2022-10-28 03:05:01] ...trap/engines/engine.py.224: train: epoch 102 | batch 1850/2304 [I 2022-10-28 03:05:01] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:13 [I 2022-10-28 03:05:01] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00244 [I 2022-10-28 03:05:01] ...trap/engines/engine.py.230: loss: 0.20236 [I 2022-10-28 03:05:01] ...optimizers/trijoint.py.151: total_norm: 28.281858 [I 2022-10-28 03:05:07] ...trap/engines/engine.py.224: train: epoch 102 | batch 1860/2304 [I 2022-10-28 03:05:07] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:06 [I 2022-10-28 03:05:07] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00246 [I 2022-10-28 03:05:07] ...trap/engines/engine.py.230: loss: 0.11415 [I 2022-10-28 03:05:07] ...optimizers/trijoint.py.151: total_norm: 10.117362 [I 2022-10-28 03:05:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 1870/2304 [I 2022-10-28 03:05:12] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:02 [I 2022-10-28 03:05:12] ...trap/engines/engine.py.229: process: 0.55149 | load: 0.00248 [I 2022-10-28 03:05:12] ...trap/engines/engine.py.230: loss: 0.14225 [I 2022-10-28 03:05:12] ...optimizers/trijoint.py.151: total_norm: 28.614716 [I 2022-10-28 03:05:18] ...trap/engines/engine.py.224: train: epoch 102 | batch 1880/2304 [I 2022-10-28 03:05:18] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:03:57 [I 2022-10-28 03:05:18] ...trap/engines/engine.py.229: process: 0.56640 | load: 0.00250 [I 2022-10-28 03:05:18] ...trap/engines/engine.py.230: loss: 0.11158 [I 2022-10-28 03:05:18] ...optimizers/trijoint.py.151: total_norm: 9.641078 [I 2022-10-28 03:05:24] ...trap/engines/engine.py.224: train: epoch 102 | batch 1890/2304 [I 2022-10-28 03:05:24] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:52 [I 2022-10-28 03:05:24] ...trap/engines/engine.py.229: process: 0.56227 | load: 0.00248 [I 2022-10-28 03:05:24] ...trap/engines/engine.py.230: loss: 0.12840 [I 2022-10-28 03:05:24] ...optimizers/trijoint.py.151: total_norm: 10.191244 [I 2022-10-28 03:05:29] ...trap/engines/engine.py.224: train: epoch 102 | batch 1900/2304 [I 2022-10-28 03:05:29] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:03:46 [I 2022-10-28 03:05:29] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00245 [I 2022-10-28 03:05:29] ...trap/engines/engine.py.230: loss: 0.16307 [I 2022-10-28 03:05:29] ...optimizers/trijoint.py.151: total_norm: 32.657501 [I 2022-10-28 03:05:35] ...trap/engines/engine.py.224: train: epoch 102 | batch 1910/2304 [I 2022-10-28 03:05:35] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:38 [I 2022-10-28 03:05:35] ...trap/engines/engine.py.229: process: 0.54778 | load: 0.00246 [I 2022-10-28 03:05:35] ...trap/engines/engine.py.230: loss: 0.15702 [I 2022-10-28 03:05:35] ...optimizers/trijoint.py.151: total_norm: 21.561213 [I 2022-10-28 03:05:41] ...trap/engines/engine.py.224: train: epoch 102 | batch 1920/2304 [I 2022-10-28 03:05:41] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:34 [I 2022-10-28 03:05:41] ...trap/engines/engine.py.229: process: 0.54963 | load: 0.00247 [I 2022-10-28 03:05:41] ...trap/engines/engine.py.230: loss: 0.07980 [I 2022-10-28 03:05:41] ...optimizers/trijoint.py.151: total_norm: 13.051295 [I 2022-10-28 03:05:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 1930/2304 [I 2022-10-28 03:05:46] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:29 [I 2022-10-28 03:05:46] ...trap/engines/engine.py.229: process: 0.55205 | load: 0.00242 [I 2022-10-28 03:05:46] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-28 03:05:46] ...optimizers/trijoint.py.151: total_norm: 12.242315 [I 2022-10-28 03:05:52] ...trap/engines/engine.py.224: train: epoch 102 | batch 1940/2304 [I 2022-10-28 03:05:52] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:21 [I 2022-10-28 03:05:52] ...trap/engines/engine.py.229: process: 0.54512 | load: 0.00247 [I 2022-10-28 03:05:52] ...trap/engines/engine.py.230: loss: 0.08109 [I 2022-10-28 03:05:52] ...optimizers/trijoint.py.151: total_norm: 11.507069 [I 2022-10-28 03:05:57] ...trap/engines/engine.py.224: train: epoch 102 | batch 1950/2304 [I 2022-10-28 03:05:57] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:16 [I 2022-10-28 03:05:57] ...trap/engines/engine.py.229: process: 0.54466 | load: 0.00244 [I 2022-10-28 03:05:57] ...trap/engines/engine.py.230: loss: 0.11569 [I 2022-10-28 03:05:57] ...optimizers/trijoint.py.151: total_norm: 12.499228 [I 2022-10-28 03:06:03] ...trap/engines/engine.py.224: train: epoch 102 | batch 1960/2304 [I 2022-10-28 03:06:03] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:13 [I 2022-10-28 03:06:03] ...trap/engines/engine.py.229: process: 0.57493 | load: 0.00244 [I 2022-10-28 03:06:03] ...trap/engines/engine.py.230: loss: 0.13233 [I 2022-10-28 03:06:03] ...optimizers/trijoint.py.151: total_norm: 206.114090 [I 2022-10-28 03:06:08] ...trap/engines/engine.py.224: train: epoch 102 | batch 1970/2304 [I 2022-10-28 03:06:08] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:07 [I 2022-10-28 03:06:08] ...trap/engines/engine.py.229: process: 0.57486 | load: 0.00244 [I 2022-10-28 03:06:08] ...trap/engines/engine.py.230: loss: 0.07028 [I 2022-10-28 03:06:08] ...optimizers/trijoint.py.151: total_norm: 12.103372 [I 2022-10-28 03:06:14] ...trap/engines/engine.py.224: train: epoch 102 | batch 1980/2304 [I 2022-10-28 03:06:14] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:00 [I 2022-10-28 03:06:14] ...trap/engines/engine.py.229: process: 0.54982 | load: 0.00254 [I 2022-10-28 03:06:14] ...trap/engines/engine.py.230: loss: 0.08683 [I 2022-10-28 03:06:14] ...optimizers/trijoint.py.151: total_norm: 11.928236 [I 2022-10-28 03:06:20] ...trap/engines/engine.py.224: train: epoch 102 | batch 1990/2304 [I 2022-10-28 03:06:20] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:02:54 [I 2022-10-28 03:06:20] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00250 [I 2022-10-28 03:06:20] ...trap/engines/engine.py.230: loss: 0.10788 [I 2022-10-28 03:06:20] ...optimizers/trijoint.py.151: total_norm: 11.134391 [I 2022-10-28 03:06:25] ...trap/engines/engine.py.224: train: epoch 102 | batch 2000/2304 [I 2022-10-28 03:06:25] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:02:50 [I 2022-10-28 03:06:25] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00245 [I 2022-10-28 03:06:25] ...trap/engines/engine.py.230: loss: 0.14976 [I 2022-10-28 03:06:25] ...optimizers/trijoint.py.151: total_norm: 20.929464 [I 2022-10-28 03:06:31] ...trap/engines/engine.py.224: train: epoch 102 | batch 2010/2304 [I 2022-10-28 03:06:31] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:02:44 [I 2022-10-28 03:06:31] ...trap/engines/engine.py.229: process: 0.56518 | load: 0.00247 [I 2022-10-28 03:06:31] ...trap/engines/engine.py.230: loss: 0.10106 [I 2022-10-28 03:06:31] ...optimizers/trijoint.py.151: total_norm: 8.846972 [I 2022-10-28 03:06:36] ...trap/engines/engine.py.224: train: epoch 102 | batch 2020/2304 [I 2022-10-28 03:06:36] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:38 [I 2022-10-28 03:06:36] ...trap/engines/engine.py.229: process: 0.55943 | load: 0.00246 [I 2022-10-28 03:06:36] ...trap/engines/engine.py.230: loss: 0.13400 [I 2022-10-28 03:06:36] ...optimizers/trijoint.py.151: total_norm: 15.616325 [I 2022-10-28 03:06:42] ...trap/engines/engine.py.224: train: epoch 102 | batch 2030/2304 [I 2022-10-28 03:06:42] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:32 [I 2022-10-28 03:06:42] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00244 [I 2022-10-28 03:06:42] ...trap/engines/engine.py.230: loss: 0.08269 [I 2022-10-28 03:06:42] ...optimizers/trijoint.py.151: total_norm: 15.559007 [I 2022-10-28 03:06:48] ...trap/engines/engine.py.224: train: epoch 102 | batch 2040/2304 [I 2022-10-28 03:06:48] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:02:29 [I 2022-10-28 03:06:48] ...trap/engines/engine.py.229: process: 0.55141 | load: 0.00249 [I 2022-10-28 03:06:48] ...trap/engines/engine.py.230: loss: 0.13572 [I 2022-10-28 03:06:48] ...optimizers/trijoint.py.151: total_norm: 15.966142 [I 2022-10-28 03:06:53] ...trap/engines/engine.py.224: train: epoch 102 | batch 2050/2304 [I 2022-10-28 03:06:53] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:24 [I 2022-10-28 03:06:53] ...trap/engines/engine.py.229: process: 0.57872 | load: 0.00250 [I 2022-10-28 03:06:53] ...trap/engines/engine.py.230: loss: 0.13039 [I 2022-10-28 03:06:53] ...optimizers/trijoint.py.151: total_norm: 13.156583 [I 2022-10-28 03:06:59] ...trap/engines/engine.py.224: train: epoch 102 | batch 2060/2304 [I 2022-10-28 03:06:59] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:18 [I 2022-10-28 03:06:59] ...trap/engines/engine.py.229: process: 0.56002 | load: 0.00247 [I 2022-10-28 03:06:59] ...trap/engines/engine.py.230: loss: 0.14661 [I 2022-10-28 03:06:59] ...optimizers/trijoint.py.151: total_norm: 39.040760 [I 2022-10-28 03:07:05] ...trap/engines/engine.py.224: train: epoch 102 | batch 2070/2304 [I 2022-10-28 03:07:05] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:11 [I 2022-10-28 03:07:05] ...trap/engines/engine.py.229: process: 0.54795 | load: 0.00244 [I 2022-10-28 03:07:05] ...trap/engines/engine.py.230: loss: 0.11194 [I 2022-10-28 03:07:05] ...optimizers/trijoint.py.151: total_norm: 12.090778 [I 2022-10-28 03:07:10] ...trap/engines/engine.py.224: train: epoch 102 | batch 2080/2304 [I 2022-10-28 03:07:10] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:05 [I 2022-10-28 03:07:10] ...trap/engines/engine.py.229: process: 0.56327 | load: 0.00246 [I 2022-10-28 03:07:10] ...trap/engines/engine.py.230: loss: 0.09595 [I 2022-10-28 03:07:10] ...optimizers/trijoint.py.151: total_norm: 16.118790 [I 2022-10-28 03:07:16] ...trap/engines/engine.py.224: train: epoch 102 | batch 2090/2304 [I 2022-10-28 03:07:16] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:01:59 [I 2022-10-28 03:07:16] ...trap/engines/engine.py.229: process: 0.56557 | load: 0.00242 [I 2022-10-28 03:07:16] ...trap/engines/engine.py.230: loss: 0.08809 [I 2022-10-28 03:07:16] ...optimizers/trijoint.py.151: total_norm: 11.574059 [I 2022-10-28 03:07:22] ...trap/engines/engine.py.224: train: epoch 102 | batch 2100/2304 [I 2022-10-28 03:07:22] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:01:54 [I 2022-10-28 03:07:22] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00243 [I 2022-10-28 03:07:22] ...trap/engines/engine.py.230: loss: 0.16716 [I 2022-10-28 03:07:22] ...optimizers/trijoint.py.151: total_norm: 18.346458 [I 2022-10-28 03:07:27] ...trap/engines/engine.py.224: train: epoch 102 | batch 2110/2304 [I 2022-10-28 03:07:27] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:01:47 [I 2022-10-28 03:07:27] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00244 [I 2022-10-28 03:07:27] ...trap/engines/engine.py.230: loss: 0.13336 [I 2022-10-28 03:07:27] ...optimizers/trijoint.py.151: total_norm: 22.515263 [I 2022-10-28 03:07:33] ...trap/engines/engine.py.224: train: epoch 102 | batch 2120/2304 [I 2022-10-28 03:07:33] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:41 [I 2022-10-28 03:07:33] ...trap/engines/engine.py.229: process: 0.54883 | load: 0.00243 [I 2022-10-28 03:07:33] ...trap/engines/engine.py.230: loss: 0.13873 [I 2022-10-28 03:07:33] ...optimizers/trijoint.py.151: total_norm: 12.917326 [I 2022-10-28 03:07:38] ...trap/engines/engine.py.224: train: epoch 102 | batch 2130/2304 [I 2022-10-28 03:07:38] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:38 [I 2022-10-28 03:07:38] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00247 [I 2022-10-28 03:07:38] ...trap/engines/engine.py.230: loss: 0.09829 [I 2022-10-28 03:07:38] ...optimizers/trijoint.py.151: total_norm: 10.622597 [I 2022-10-28 03:07:44] ...trap/engines/engine.py.224: train: epoch 102 | batch 2140/2304 [I 2022-10-28 03:07:44] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:32 [I 2022-10-28 03:07:44] ...trap/engines/engine.py.229: process: 0.56431 | load: 0.00251 [I 2022-10-28 03:07:44] ...trap/engines/engine.py.230: loss: 0.13385 [I 2022-10-28 03:07:44] ...optimizers/trijoint.py.151: total_norm: 17.387604 [I 2022-10-28 03:07:50] ...trap/engines/engine.py.224: train: epoch 102 | batch 2150/2304 [I 2022-10-28 03:07:50] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:25 [I 2022-10-28 03:07:50] ...trap/engines/engine.py.229: process: 0.56521 | load: 0.00247 [I 2022-10-28 03:07:50] ...trap/engines/engine.py.230: loss: 0.16026 [I 2022-10-28 03:07:50] ...optimizers/trijoint.py.151: total_norm: 17.964016 [I 2022-10-28 03:07:55] ...trap/engines/engine.py.224: train: epoch 102 | batch 2160/2304 [I 2022-10-28 03:07:55] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:20 [I 2022-10-28 03:07:55] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00246 [I 2022-10-28 03:07:55] ...trap/engines/engine.py.230: loss: 0.13884 [I 2022-10-28 03:07:55] ...optimizers/trijoint.py.151: total_norm: 17.989061 [I 2022-10-28 03:08:01] ...trap/engines/engine.py.224: train: epoch 102 | batch 2170/2304 [I 2022-10-28 03:08:01] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:15 [I 2022-10-28 03:08:01] ...trap/engines/engine.py.229: process: 0.56909 | load: 0.00247 [I 2022-10-28 03:08:01] ...trap/engines/engine.py.230: loss: 0.07575 [I 2022-10-28 03:08:01] ...optimizers/trijoint.py.151: total_norm: 13.019053 [I 2022-10-28 03:08:06] ...trap/engines/engine.py.224: train: epoch 102 | batch 2180/2304 [I 2022-10-28 03:08:06] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:09 [I 2022-10-28 03:08:06] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00243 [I 2022-10-28 03:08:06] ...trap/engines/engine.py.230: loss: 0.12857 [I 2022-10-28 03:08:06] ...optimizers/trijoint.py.151: total_norm: 14.620409 [I 2022-10-28 03:08:12] ...trap/engines/engine.py.224: train: epoch 102 | batch 2190/2304 [I 2022-10-28 03:08:12] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:01:03 [I 2022-10-28 03:08:12] ...trap/engines/engine.py.229: process: 0.54372 | load: 0.00246 [I 2022-10-28 03:08:12] ...trap/engines/engine.py.230: loss: 0.11362 [I 2022-10-28 03:08:12] ...optimizers/trijoint.py.151: total_norm: 28.298096 [I 2022-10-28 03:08:18] ...trap/engines/engine.py.224: train: epoch 102 | batch 2200/2304 [I 2022-10-28 03:08:18] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:00:58 [I 2022-10-28 03:08:18] ...trap/engines/engine.py.229: process: 0.55688 | load: 0.00250 [I 2022-10-28 03:08:18] ...trap/engines/engine.py.230: loss: 0.12370 [I 2022-10-28 03:08:18] ...optimizers/trijoint.py.151: total_norm: 15.850182 [I 2022-10-28 03:08:24] ...trap/engines/engine.py.224: train: epoch 102 | batch 2210/2304 [I 2022-10-28 03:08:24] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:00:53 [I 2022-10-28 03:08:24] ...trap/engines/engine.py.229: process: 0.56678 | load: 0.00250 [I 2022-10-28 03:08:24] ...trap/engines/engine.py.230: loss: 0.16454 [I 2022-10-28 03:08:24] ...optimizers/trijoint.py.151: total_norm: 14.334358 [I 2022-10-28 03:08:29] ...trap/engines/engine.py.224: train: epoch 102 | batch 2220/2304 [I 2022-10-28 03:08:29] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:00:47 [I 2022-10-28 03:08:29] ...trap/engines/engine.py.229: process: 0.56397 | load: 0.00244 [I 2022-10-28 03:08:29] ...trap/engines/engine.py.230: loss: 0.12399 [I 2022-10-28 03:08:29] ...optimizers/trijoint.py.151: total_norm: 21.984819 [I 2022-10-28 03:08:35] ...trap/engines/engine.py.224: train: epoch 102 | batch 2230/2304 [I 2022-10-28 03:08:35] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:41 [I 2022-10-28 03:08:35] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00252 [I 2022-10-28 03:08:35] ...trap/engines/engine.py.230: loss: 0.11134 [I 2022-10-28 03:08:35] ...optimizers/trijoint.py.151: total_norm: 18.957361 [I 2022-10-28 03:08:40] ...trap/engines/engine.py.224: train: epoch 102 | batch 2240/2304 [I 2022-10-28 03:08:40] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:35 [I 2022-10-28 03:08:40] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00245 [I 2022-10-28 03:08:40] ...trap/engines/engine.py.230: loss: 0.10723 [I 2022-10-28 03:08:40] ...optimizers/trijoint.py.151: total_norm: 17.224474 [I 2022-10-28 03:08:46] ...trap/engines/engine.py.224: train: epoch 102 | batch 2250/2304 [I 2022-10-28 03:08:46] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:30 [I 2022-10-28 03:08:46] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00243 [I 2022-10-28 03:08:46] ...trap/engines/engine.py.230: loss: 0.13662 [I 2022-10-28 03:08:46] ...optimizers/trijoint.py.151: total_norm: 7.733673 [I 2022-10-28 03:08:52] ...trap/engines/engine.py.224: train: epoch 102 | batch 2260/2304 [I 2022-10-28 03:08:52] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:24 [I 2022-10-28 03:08:52] ...trap/engines/engine.py.229: process: 0.55565 | load: 0.00243 [I 2022-10-28 03:08:52] ...trap/engines/engine.py.230: loss: 0.16605 [I 2022-10-28 03:08:52] ...optimizers/trijoint.py.151: total_norm: 18.545025 [I 2022-10-28 03:08:57] ...trap/engines/engine.py.224: train: epoch 102 | batch 2270/2304 [I 2022-10-28 03:08:57] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:19 [I 2022-10-28 03:08:57] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00247 [I 2022-10-28 03:08:57] ...trap/engines/engine.py.230: loss: 0.09604 [I 2022-10-28 03:08:57] ...optimizers/trijoint.py.151: total_norm: 9.913825 [I 2022-10-28 03:09:03] ...trap/engines/engine.py.224: train: epoch 102 | batch 2280/2304 [I 2022-10-28 03:09:03] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:13 [I 2022-10-28 03:09:03] ...trap/engines/engine.py.229: process: 0.56280 | load: 0.00242 [I 2022-10-28 03:09:03] ...trap/engines/engine.py.230: loss: 0.10871 [I 2022-10-28 03:09:03] ...optimizers/trijoint.py.151: total_norm: 13.420871 [I 2022-10-28 03:09:08] ...trap/engines/engine.py.224: train: epoch 102 | batch 2290/2304 [I 2022-10-28 03:09:08] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:07 [I 2022-10-28 03:09:08] ...trap/engines/engine.py.229: process: 0.56600 | load: 0.00255 [I 2022-10-28 03:09:08] ...trap/engines/engine.py.230: loss: 0.10993 [I 2022-10-28 03:09:08] ...optimizers/trijoint.py.151: total_norm: 18.925507 [I 2022-10-28 03:09:14] ...trap/engines/engine.py.224: train: epoch 102 | batch 2300/2304 [I 2022-10-28 03:09:14] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:02 [I 2022-10-28 03:09:14] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00015 [I 2022-10-28 03:09:14] ...trap/engines/engine.py.230: loss: 0.15325 [I 2022-10-28 03:09:14] ...optimizers/trijoint.py.151: total_norm: 24.138210 [I 2022-10-28 03:09:16] ...trap/engines/engine.py.224: train: epoch 102 | batch 2304/2304 [I 2022-10-28 03:09:16] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:00 [I 2022-10-28 03:09:16] ...trap/engines/engine.py.229: process: 0.54726 | load: 0.00015 [I 2022-10-28 03:09:16] ...trap/engines/engine.py.230: loss: 0.08523 [I 2022-10-28 03:09:16] ...optimizers/trijoint.py.151: total_norm: 8.240215 [S 2022-10-28 03:09:17] ...trap/engines/engine.py.129: train_epoch.epoch: 102 [S 2022-10-28 03:09:17] ...trap/engines/engine.py.129: train_epoch.loss: 0.12582437188209009 [I 2022-10-28 03:09:17] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 03:09:19] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 102 [I 2022-10-28 03:09:19] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 03:09:21] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 03:09:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 03:09:21] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 03:09:24] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 03:09:25] ...trap/engines/engine.py.321: eval: epoch 102 | batch 0/510 [I 2022-10-28 03:09:25] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:47:32 [I 2022-10-28 03:09:25] ...trap/engines/engine.py.326: process: 5.59378 | load: 3.13525 [I 2022-10-28 03:09:28] ...trap/engines/engine.py.321: eval: epoch 102 | batch 10/510 [I 2022-10-28 03:09:28] ...trap/engines/engine.py.325: elapsed: 0:00:08 | left: 0:07:29 [I 2022-10-28 03:09:28] ...trap/engines/engine.py.326: process: 0.33923 | load: 0.00018 [I 2022-10-28 03:09:31] ...trap/engines/engine.py.321: eval: epoch 102 | batch 20/510 [I 2022-10-28 03:09:31] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:13 [I 2022-10-28 03:09:31] ...trap/engines/engine.py.326: process: 0.32866 | load: 0.00017 [I 2022-10-28 03:09:35] ...trap/engines/engine.py.321: eval: epoch 102 | batch 30/510 [I 2022-10-28 03:09:35] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:43 [I 2022-10-28 03:09:35] ...trap/engines/engine.py.326: process: 0.33867 | load: 0.00018 [I 2022-10-28 03:09:38] ...trap/engines/engine.py.321: eval: epoch 102 | batch 40/510 [I 2022-10-28 03:09:38] ...trap/engines/engine.py.325: elapsed: 0:00:18 | left: 0:02:37 [I 2022-10-28 03:09:38] ...trap/engines/engine.py.326: process: 0.33483 | load: 0.00021 [I 2022-10-28 03:09:41] ...trap/engines/engine.py.321: eval: epoch 102 | batch 50/510 [I 2022-10-28 03:09:41] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:37 [I 2022-10-28 03:09:41] ...trap/engines/engine.py.326: process: 0.32803 | load: 0.00020 [I 2022-10-28 03:09:45] ...trap/engines/engine.py.321: eval: epoch 102 | batch 60/510 [I 2022-10-28 03:09:45] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:31 [I 2022-10-28 03:09:45] ...trap/engines/engine.py.326: process: 0.32828 | load: 0.00019 [I 2022-10-28 03:09:48] ...trap/engines/engine.py.321: eval: epoch 102 | batch 70/510 [I 2022-10-28 03:09:48] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:27 [I 2022-10-28 03:09:48] ...trap/engines/engine.py.326: process: 0.32318 | load: 0.00016 [I 2022-10-28 03:09:52] ...trap/engines/engine.py.321: eval: epoch 102 | batch 80/510 [I 2022-10-28 03:09:52] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:26 [I 2022-10-28 03:09:52] ...trap/engines/engine.py.326: process: 0.33528 | load: 0.00018 [I 2022-10-28 03:09:55] ...trap/engines/engine.py.321: eval: epoch 102 | batch 90/510 [I 2022-10-28 03:09:55] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:20 [I 2022-10-28 03:09:55] ...trap/engines/engine.py.326: process: 0.33365 | load: 0.00018 [I 2022-10-28 03:09:58] ...trap/engines/engine.py.321: eval: epoch 102 | batch 100/510 [I 2022-10-28 03:09:58] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-28 03:09:58] ...trap/engines/engine.py.326: process: 0.28847 | load: 0.00024 [I 2022-10-28 03:10:01] ...trap/engines/engine.py.321: eval: epoch 102 | batch 110/510 [I 2022-10-28 03:10:01] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:57 [I 2022-10-28 03:10:01] ...trap/engines/engine.py.326: process: 0.29684 | load: 0.00014 [I 2022-10-28 03:10:04] ...trap/engines/engine.py.321: eval: epoch 102 | batch 120/510 [I 2022-10-28 03:10:04] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:53 [I 2022-10-28 03:10:04] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00015 [I 2022-10-28 03:10:07] ...trap/engines/engine.py.321: eval: epoch 102 | batch 130/510 [I 2022-10-28 03:10:07] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:50 [I 2022-10-28 03:10:07] ...trap/engines/engine.py.326: process: 0.28601 | load: 0.00020 [I 2022-10-28 03:10:10] ...trap/engines/engine.py.321: eval: epoch 102 | batch 140/510 [I 2022-10-28 03:10:10] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-28 03:10:10] ...trap/engines/engine.py.326: process: 0.28833 | load: 0.00023 [I 2022-10-28 03:10:13] ...trap/engines/engine.py.321: eval: epoch 102 | batch 150/510 [I 2022-10-28 03:10:13] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:45 [I 2022-10-28 03:10:13] ...trap/engines/engine.py.326: process: 0.30278 | load: 0.00016 [I 2022-10-28 03:10:16] ...trap/engines/engine.py.321: eval: epoch 102 | batch 160/510 [I 2022-10-28 03:10:16] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-28 03:10:16] ...trap/engines/engine.py.326: process: 0.28505 | load: 0.00014 [I 2022-10-28 03:10:19] ...trap/engines/engine.py.321: eval: epoch 102 | batch 170/510 [I 2022-10-28 03:10:19] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 03:10:19] ...trap/engines/engine.py.326: process: 0.29072 | load: 0.00014 [I 2022-10-28 03:10:21] ...trap/engines/engine.py.321: eval: epoch 102 | batch 180/510 [I 2022-10-28 03:10:21] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-28 03:10:21] ...trap/engines/engine.py.326: process: 0.29057 | load: 0.00015 [I 2022-10-28 03:10:24] ...trap/engines/engine.py.321: eval: epoch 102 | batch 190/510 [I 2022-10-28 03:10:24] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:33 [I 2022-10-28 03:10:24] ...trap/engines/engine.py.326: process: 0.28723 | load: 0.00014 [I 2022-10-28 03:10:27] ...trap/engines/engine.py.321: eval: epoch 102 | batch 200/510 [I 2022-10-28 03:10:27] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:29 [I 2022-10-28 03:10:27] ...trap/engines/engine.py.326: process: 0.28889 | load: 0.00015 [I 2022-10-28 03:10:30] ...trap/engines/engine.py.321: eval: epoch 102 | batch 210/510 [I 2022-10-28 03:10:30] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 03:10:30] ...trap/engines/engine.py.326: process: 0.28725 | load: 0.00014 [I 2022-10-28 03:10:33] ...trap/engines/engine.py.321: eval: epoch 102 | batch 220/510 [I 2022-10-28 03:10:33] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:24 [I 2022-10-28 03:10:33] ...trap/engines/engine.py.326: process: 0.29198 | load: 0.00046 [I 2022-10-28 03:10:36] ...trap/engines/engine.py.321: eval: epoch 102 | batch 230/510 [I 2022-10-28 03:10:36] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-28 03:10:36] ...trap/engines/engine.py.326: process: 0.28403 | load: 0.00027 [I 2022-10-28 03:10:39] ...trap/engines/engine.py.321: eval: epoch 102 | batch 240/510 [I 2022-10-28 03:10:39] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-28 03:10:39] ...trap/engines/engine.py.326: process: 0.28407 | load: 0.00015 [I 2022-10-28 03:10:42] ...trap/engines/engine.py.321: eval: epoch 102 | batch 250/510 [I 2022-10-28 03:10:42] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-28 03:10:42] ...trap/engines/engine.py.326: process: 0.29561 | load: 0.00016 [I 2022-10-28 03:10:45] ...trap/engines/engine.py.321: eval: epoch 102 | batch 260/510 [I 2022-10-28 03:10:45] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 03:10:45] ...trap/engines/engine.py.326: process: 0.28936 | load: 0.00016 [I 2022-10-28 03:10:48] ...trap/engines/engine.py.321: eval: epoch 102 | batch 270/510 [I 2022-10-28 03:10:48] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-28 03:10:48] ...trap/engines/engine.py.326: process: 0.28790 | load: 0.00015 [I 2022-10-28 03:10:50] ...trap/engines/engine.py.321: eval: epoch 102 | batch 280/510 [I 2022-10-28 03:10:50] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 03:10:50] ...trap/engines/engine.py.326: process: 0.28722 | load: 0.00015 [I 2022-10-28 03:10:53] ...trap/engines/engine.py.321: eval: epoch 102 | batch 290/510 [I 2022-10-28 03:10:53] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 03:10:53] ...trap/engines/engine.py.326: process: 0.28450 | load: 0.00015 [I 2022-10-28 03:10:56] ...trap/engines/engine.py.321: eval: epoch 102 | batch 300/510 [I 2022-10-28 03:10:56] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-28 03:10:56] ...trap/engines/engine.py.326: process: 0.28225 | load: 0.00020 [I 2022-10-28 03:10:59] ...trap/engines/engine.py.321: eval: epoch 102 | batch 310/510 [I 2022-10-28 03:10:59] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:58 [I 2022-10-28 03:10:59] ...trap/engines/engine.py.326: process: 0.28941 | load: 0.00019 [I 2022-10-28 03:11:02] ...trap/engines/engine.py.321: eval: epoch 102 | batch 320/510 [I 2022-10-28 03:11:02] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 03:11:02] ...trap/engines/engine.py.326: process: 0.29201 | load: 0.00014 [I 2022-10-28 03:11:05] ...trap/engines/engine.py.321: eval: epoch 102 | batch 330/510 [I 2022-10-28 03:11:05] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-28 03:11:05] ...trap/engines/engine.py.326: process: 0.30508 | load: 0.00014 [I 2022-10-28 03:11:08] ...trap/engines/engine.py.321: eval: epoch 102 | batch 340/510 [I 2022-10-28 03:11:08] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-28 03:11:08] ...trap/engines/engine.py.326: process: 0.28793 | load: 0.00027 [I 2022-10-28 03:11:11] ...trap/engines/engine.py.321: eval: epoch 102 | batch 350/510 [I 2022-10-28 03:11:11] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 03:11:11] ...trap/engines/engine.py.326: process: 0.29208 | load: 0.00014 [I 2022-10-28 03:11:14] ...trap/engines/engine.py.321: eval: epoch 102 | batch 360/510 [I 2022-10-28 03:11:14] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 03:11:14] ...trap/engines/engine.py.326: process: 0.29546 | load: 0.00014 [I 2022-10-28 03:11:16] ...trap/engines/engine.py.321: eval: epoch 102 | batch 370/510 [I 2022-10-28 03:11:16] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 03:11:16] ...trap/engines/engine.py.326: process: 0.28385 | load: 0.00014 [I 2022-10-28 03:11:19] ...trap/engines/engine.py.321: eval: epoch 102 | batch 380/510 [I 2022-10-28 03:11:19] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 03:11:19] ...trap/engines/engine.py.326: process: 0.29089 | load: 0.00014 [I 2022-10-28 03:11:22] ...trap/engines/engine.py.321: eval: epoch 102 | batch 390/510 [I 2022-10-28 03:11:22] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 03:11:22] ...trap/engines/engine.py.326: process: 0.28730 | load: 0.00015 [I 2022-10-28 03:11:25] ...trap/engines/engine.py.321: eval: epoch 102 | batch 400/510 [I 2022-10-28 03:11:25] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:32 [I 2022-10-28 03:11:25] ...trap/engines/engine.py.326: process: 0.28786 | load: 0.00015 [I 2022-10-28 03:11:28] ...trap/engines/engine.py.321: eval: epoch 102 | batch 410/510 [I 2022-10-28 03:11:28] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:28 [I 2022-10-28 03:11:28] ...trap/engines/engine.py.326: process: 0.28806 | load: 0.00015 [I 2022-10-28 03:11:31] ...trap/engines/engine.py.321: eval: epoch 102 | batch 420/510 [I 2022-10-28 03:11:31] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 03:11:31] ...trap/engines/engine.py.326: process: 0.29071 | load: 0.00015 [I 2022-10-28 03:11:34] ...trap/engines/engine.py.321: eval: epoch 102 | batch 430/510 [I 2022-10-28 03:11:34] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 03:11:34] ...trap/engines/engine.py.326: process: 0.28812 | load: 0.00021 [I 2022-10-28 03:11:37] ...trap/engines/engine.py.321: eval: epoch 102 | batch 440/510 [I 2022-10-28 03:11:37] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 03:11:37] ...trap/engines/engine.py.326: process: 0.29205 | load: 0.00016 [I 2022-10-28 03:11:40] ...trap/engines/engine.py.321: eval: epoch 102 | batch 450/510 [I 2022-10-28 03:11:40] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 03:11:40] ...trap/engines/engine.py.326: process: 0.29000 | load: 0.00018 [I 2022-10-28 03:11:43] ...trap/engines/engine.py.321: eval: epoch 102 | batch 460/510 [I 2022-10-28 03:11:43] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 03:11:43] ...trap/engines/engine.py.326: process: 0.28718 | load: 0.00016 [I 2022-10-28 03:11:45] ...trap/engines/engine.py.321: eval: epoch 102 | batch 470/510 [I 2022-10-28 03:11:45] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 03:11:45] ...trap/engines/engine.py.326: process: 0.29265 | load: 0.00015 [I 2022-10-28 03:11:48] ...trap/engines/engine.py.321: eval: epoch 102 | batch 480/510 [I 2022-10-28 03:11:48] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 03:11:48] ...trap/engines/engine.py.326: process: 0.29699 | load: 0.00016 [I 2022-10-28 03:11:51] ...trap/engines/engine.py.321: eval: epoch 102 | batch 490/510 [I 2022-10-28 03:11:51] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 03:11:51] ...trap/engines/engine.py.326: process: 0.28563 | load: 0.00014 [I 2022-10-28 03:11:54] ...trap/engines/engine.py.321: eval: epoch 102 | batch 500/510 [I 2022-10-28 03:11:54] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:02 [I 2022-10-28 03:11:54] ...trap/engines/engine.py.326: process: 0.29252 | load: 0.00015 [I 2022-10-28 03:11:57] ...trap/engines/engine.py.321: eval: epoch 102 | batch 510/510 [I 2022-10-28 03:11:57] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 03:11:57] ...trap/engines/engine.py.326: process: 0.28876 | load: 0.00010 [S 2022-10-28 03:11:58] ...trap/engines/engine.py.132: eval_epoch.epoch: 102 [I 2022-10-28 03:11:58] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.736600 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906100 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933500 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.732900 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905200 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934800 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011834 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007409 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007393 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010163 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009293 [S 2022-10-28 03:12:02] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006462 [I 2022-10-28 03:12:04] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 03:12:04] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 03:12:04] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 03:12:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 03:12:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 03:12:09] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 03:12:10] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 03:12:16] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 03:12:16] ...trap/engines/engine.py.163: Training model on trainset for epoch 103 [I 2022-10-28 03:12:16] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 03:12:19] ...trap/engines/engine.py.224: train: epoch 103 | batch 0/2304 [I 2022-10-28 03:12:19] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:45:37 [I 2022-10-28 03:12:19] ...trap/engines/engine.py.229: process: 2.75076 | load: 2.07570 [I 2022-10-28 03:12:19] ...trap/engines/engine.py.230: loss: 0.15187 [I 2022-10-28 03:12:19] ...optimizers/trijoint.py.151: total_norm: 12.910853 [I 2022-10-28 03:12:24] ...trap/engines/engine.py.224: train: epoch 103 | batch 10/2304 [I 2022-10-28 03:12:24] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:24 [I 2022-10-28 03:12:24] ...trap/engines/engine.py.229: process: 0.56823 | load: 0.00449 [I 2022-10-28 03:12:24] ...trap/engines/engine.py.230: loss: 0.14284 [I 2022-10-28 03:12:24] ...optimizers/trijoint.py.151: total_norm: 16.873442 [I 2022-10-28 03:12:30] ...trap/engines/engine.py.224: train: epoch 103 | batch 20/2304 [I 2022-10-28 03:12:30] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:09 [I 2022-10-28 03:12:30] ...trap/engines/engine.py.229: process: 0.54828 | load: 0.00423 [I 2022-10-28 03:12:30] ...trap/engines/engine.py.230: loss: 0.13958 [I 2022-10-28 03:12:30] ...optimizers/trijoint.py.151: total_norm: 17.830315 [I 2022-10-28 03:12:35] ...trap/engines/engine.py.224: train: epoch 103 | batch 30/2304 [I 2022-10-28 03:12:35] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:31 [I 2022-10-28 03:12:35] ...trap/engines/engine.py.229: process: 0.58595 | load: 0.00332 [I 2022-10-28 03:12:35] ...trap/engines/engine.py.230: loss: 0.07288 [I 2022-10-28 03:12:35] ...optimizers/trijoint.py.151: total_norm: 14.050589 [I 2022-10-28 03:12:41] ...trap/engines/engine.py.224: train: epoch 103 | batch 40/2304 [I 2022-10-28 03:12:41] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:15 [I 2022-10-28 03:12:41] ...trap/engines/engine.py.229: process: 0.56227 | load: 0.00291 [I 2022-10-28 03:12:41] ...trap/engines/engine.py.230: loss: 0.08323 [I 2022-10-28 03:12:41] ...optimizers/trijoint.py.151: total_norm: 14.899662 [I 2022-10-28 03:12:47] ...trap/engines/engine.py.224: train: epoch 103 | batch 50/2304 [I 2022-10-28 03:12:47] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:46 [I 2022-10-28 03:12:47] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00294 [I 2022-10-28 03:12:47] ...trap/engines/engine.py.230: loss: 0.11397 [I 2022-10-28 03:12:47] ...optimizers/trijoint.py.151: total_norm: 16.703243 [I 2022-10-28 03:12:52] ...trap/engines/engine.py.224: train: epoch 103 | batch 60/2304 [I 2022-10-28 03:12:52] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:49 [I 2022-10-28 03:12:52] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00283 [I 2022-10-28 03:12:52] ...trap/engines/engine.py.230: loss: 0.13266 [I 2022-10-28 03:12:52] ...optimizers/trijoint.py.151: total_norm: 32.115059 [I 2022-10-28 03:12:58] ...trap/engines/engine.py.224: train: epoch 103 | batch 70/2304 [I 2022-10-28 03:12:58] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:36 [I 2022-10-28 03:12:58] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00278 [I 2022-10-28 03:12:58] ...trap/engines/engine.py.230: loss: 0.12273 [I 2022-10-28 03:12:58] ...optimizers/trijoint.py.151: total_norm: 15.683228 [I 2022-10-28 03:13:03] ...trap/engines/engine.py.224: train: epoch 103 | batch 80/2304 [I 2022-10-28 03:13:03] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:32 [I 2022-10-28 03:13:03] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00276 [I 2022-10-28 03:13:03] ...trap/engines/engine.py.230: loss: 0.08372 [I 2022-10-28 03:13:03] ...optimizers/trijoint.py.151: total_norm: 8.732782 [I 2022-10-28 03:13:09] ...trap/engines/engine.py.224: train: epoch 103 | batch 90/2304 [I 2022-10-28 03:13:09] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:36 [I 2022-10-28 03:13:09] ...trap/engines/engine.py.229: process: 0.54568 | load: 0.00277 [I 2022-10-28 03:13:09] ...trap/engines/engine.py.230: loss: 0.15830 [I 2022-10-28 03:13:09] ...optimizers/trijoint.py.151: total_norm: 18.463999 [I 2022-10-28 03:13:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 100/2304 [I 2022-10-28 03:13:14] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:33 [I 2022-10-28 03:13:14] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00282 [I 2022-10-28 03:13:14] ...trap/engines/engine.py.230: loss: 0.14467 [I 2022-10-28 03:13:14] ...optimizers/trijoint.py.151: total_norm: 23.223967 [I 2022-10-28 03:13:20] ...trap/engines/engine.py.224: train: epoch 103 | batch 110/2304 [I 2022-10-28 03:13:20] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:23 [I 2022-10-28 03:13:20] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00242 [I 2022-10-28 03:13:20] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-28 03:13:20] ...optimizers/trijoint.py.151: total_norm: 17.779482 [I 2022-10-28 03:13:26] ...trap/engines/engine.py.224: train: epoch 103 | batch 120/2304 [I 2022-10-28 03:13:26] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:20 [I 2022-10-28 03:13:26] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00279 [I 2022-10-28 03:13:26] ...trap/engines/engine.py.230: loss: 0.09361 [I 2022-10-28 03:13:26] ...optimizers/trijoint.py.151: total_norm: 15.623375 [I 2022-10-28 03:13:31] ...trap/engines/engine.py.224: train: epoch 103 | batch 130/2304 [I 2022-10-28 03:13:31] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:07 [I 2022-10-28 03:13:31] ...trap/engines/engine.py.229: process: 0.55583 | load: 0.00249 [I 2022-10-28 03:13:31] ...trap/engines/engine.py.230: loss: 0.24081 [I 2022-10-28 03:13:31] ...optimizers/trijoint.py.151: total_norm: 19.716852 [I 2022-10-28 03:13:37] ...trap/engines/engine.py.224: train: epoch 103 | batch 140/2304 [I 2022-10-28 03:13:37] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:00 [I 2022-10-28 03:13:37] ...trap/engines/engine.py.229: process: 0.54250 | load: 0.00249 [I 2022-10-28 03:13:37] ...trap/engines/engine.py.230: loss: 0.11213 [I 2022-10-28 03:13:37] ...optimizers/trijoint.py.151: total_norm: 11.843369 [I 2022-10-28 03:13:42] ...trap/engines/engine.py.224: train: epoch 103 | batch 150/2304 [I 2022-10-28 03:13:42] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:03 [I 2022-10-28 03:13:42] ...trap/engines/engine.py.229: process: 0.56572 | load: 0.00259 [I 2022-10-28 03:13:42] ...trap/engines/engine.py.230: loss: 0.19957 [I 2022-10-28 03:13:42] ...optimizers/trijoint.py.151: total_norm: 9.440181 [I 2022-10-28 03:13:48] ...trap/engines/engine.py.224: train: epoch 103 | batch 160/2304 [I 2022-10-28 03:13:48] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:00 [I 2022-10-28 03:13:48] ...trap/engines/engine.py.229: process: 0.57976 | load: 0.00244 [I 2022-10-28 03:13:48] ...trap/engines/engine.py.230: loss: 0.11082 [I 2022-10-28 03:13:48] ...optimizers/trijoint.py.151: total_norm: 14.464967 [I 2022-10-28 03:13:54] ...trap/engines/engine.py.224: train: epoch 103 | batch 170/2304 [I 2022-10-28 03:13:54] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:52 [I 2022-10-28 03:13:54] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00254 [I 2022-10-28 03:13:54] ...trap/engines/engine.py.230: loss: 0.12177 [I 2022-10-28 03:13:54] ...optimizers/trijoint.py.151: total_norm: 29.637518 [I 2022-10-28 03:13:59] ...trap/engines/engine.py.224: train: epoch 103 | batch 180/2304 [I 2022-10-28 03:13:59] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:35 [I 2022-10-28 03:13:59] ...trap/engines/engine.py.229: process: 0.56080 | load: 0.00272 [I 2022-10-28 03:13:59] ...trap/engines/engine.py.230: loss: 0.18909 [I 2022-10-28 03:13:59] ...optimizers/trijoint.py.151: total_norm: 22.318443 [I 2022-10-28 03:14:05] ...trap/engines/engine.py.224: train: epoch 103 | batch 190/2304 [I 2022-10-28 03:14:05] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:34 [I 2022-10-28 03:14:05] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00245 [I 2022-10-28 03:14:05] ...trap/engines/engine.py.230: loss: 0.14299 [I 2022-10-28 03:14:05] ...optimizers/trijoint.py.151: total_norm: 14.621080 [I 2022-10-28 03:14:10] ...trap/engines/engine.py.224: train: epoch 103 | batch 200/2304 [I 2022-10-28 03:14:10] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:24 [I 2022-10-28 03:14:10] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00247 [I 2022-10-28 03:14:10] ...trap/engines/engine.py.230: loss: 0.11636 [I 2022-10-28 03:14:10] ...optimizers/trijoint.py.151: total_norm: 13.659766 [I 2022-10-28 03:14:16] ...trap/engines/engine.py.224: train: epoch 103 | batch 210/2304 [I 2022-10-28 03:14:16] ...trap/engines/engine.py.228: elapsed: 0:01:59 | left: 0:19:16 [I 2022-10-28 03:14:16] ...trap/engines/engine.py.229: process: 0.53828 | load: 0.00285 [I 2022-10-28 03:14:16] ...trap/engines/engine.py.230: loss: 0.16127 [I 2022-10-28 03:14:16] ...optimizers/trijoint.py.151: total_norm: 14.053295 [I 2022-10-28 03:14:21] ...trap/engines/engine.py.224: train: epoch 103 | batch 220/2304 [I 2022-10-28 03:14:21] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:12 [I 2022-10-28 03:14:21] ...trap/engines/engine.py.229: process: 0.54742 | load: 0.00271 [I 2022-10-28 03:14:21] ...trap/engines/engine.py.230: loss: 0.13769 [I 2022-10-28 03:14:21] ...optimizers/trijoint.py.151: total_norm: 11.678440 [I 2022-10-28 03:14:27] ...trap/engines/engine.py.224: train: epoch 103 | batch 230/2304 [I 2022-10-28 03:14:27] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:10 [I 2022-10-28 03:14:27] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00247 [I 2022-10-28 03:14:27] ...trap/engines/engine.py.230: loss: 0.12287 [I 2022-10-28 03:14:27] ...optimizers/trijoint.py.151: total_norm: 14.443661 [I 2022-10-28 03:14:32] ...trap/engines/engine.py.224: train: epoch 103 | batch 240/2304 [I 2022-10-28 03:14:32] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:02 [I 2022-10-28 03:14:32] ...trap/engines/engine.py.229: process: 0.55082 | load: 0.00248 [I 2022-10-28 03:14:32] ...trap/engines/engine.py.230: loss: 0.10236 [I 2022-10-28 03:14:32] ...optimizers/trijoint.py.151: total_norm: 9.802075 [I 2022-10-28 03:14:38] ...trap/engines/engine.py.224: train: epoch 103 | batch 250/2304 [I 2022-10-28 03:14:38] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:52 [I 2022-10-28 03:14:38] ...trap/engines/engine.py.229: process: 0.55270 | load: 0.00252 [I 2022-10-28 03:14:38] ...trap/engines/engine.py.230: loss: 0.08988 [I 2022-10-28 03:14:38] ...optimizers/trijoint.py.151: total_norm: 18.682350 [I 2022-10-28 03:14:43] ...trap/engines/engine.py.224: train: epoch 103 | batch 260/2304 [I 2022-10-28 03:14:43] ...trap/engines/engine.py.228: elapsed: 0:02:27 | left: 0:18:56 [I 2022-10-28 03:14:43] ...trap/engines/engine.py.229: process: 0.55506 | load: 0.00248 [I 2022-10-28 03:14:43] ...trap/engines/engine.py.230: loss: 0.12617 [I 2022-10-28 03:14:43] ...optimizers/trijoint.py.151: total_norm: 14.393014 [I 2022-10-28 03:14:49] ...trap/engines/engine.py.224: train: epoch 103 | batch 270/2304 [I 2022-10-28 03:14:49] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:55 [I 2022-10-28 03:14:49] ...trap/engines/engine.py.229: process: 0.56859 | load: 0.00245 [I 2022-10-28 03:14:49] ...trap/engines/engine.py.230: loss: 0.09796 [I 2022-10-28 03:14:49] ...optimizers/trijoint.py.151: total_norm: 13.080038 [I 2022-10-28 03:14:55] ...trap/engines/engine.py.224: train: epoch 103 | batch 280/2304 [I 2022-10-28 03:14:55] ...trap/engines/engine.py.228: elapsed: 0:02:38 | left: 0:18:45 [I 2022-10-28 03:14:55] ...trap/engines/engine.py.229: process: 0.54897 | load: 0.00248 [I 2022-10-28 03:14:55] ...trap/engines/engine.py.230: loss: 0.24395 [I 2022-10-28 03:14:55] ...optimizers/trijoint.py.151: total_norm: 49.566483 [I 2022-10-28 03:15:00] ...trap/engines/engine.py.224: train: epoch 103 | batch 290/2304 [I 2022-10-28 03:15:00] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:38 [I 2022-10-28 03:15:00] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00266 [I 2022-10-28 03:15:00] ...trap/engines/engine.py.230: loss: 0.08287 [I 2022-10-28 03:15:00] ...optimizers/trijoint.py.151: total_norm: 22.905569 [I 2022-10-28 03:15:06] ...trap/engines/engine.py.224: train: epoch 103 | batch 300/2304 [I 2022-10-28 03:15:06] ...trap/engines/engine.py.228: elapsed: 0:02:49 | left: 0:18:33 [I 2022-10-28 03:15:06] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00249 [I 2022-10-28 03:15:06] ...trap/engines/engine.py.230: loss: 0.08854 [I 2022-10-28 03:15:06] ...optimizers/trijoint.py.151: total_norm: 19.772074 [I 2022-10-28 03:15:11] ...trap/engines/engine.py.224: train: epoch 103 | batch 310/2304 [I 2022-10-28 03:15:11] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:26 [I 2022-10-28 03:15:11] ...trap/engines/engine.py.229: process: 0.55509 | load: 0.00251 [I 2022-10-28 03:15:11] ...trap/engines/engine.py.230: loss: 0.12004 [I 2022-10-28 03:15:11] ...optimizers/trijoint.py.151: total_norm: 52.132137 [I 2022-10-28 03:15:17] ...trap/engines/engine.py.224: train: epoch 103 | batch 320/2304 [I 2022-10-28 03:15:17] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:20 [I 2022-10-28 03:15:17] ...trap/engines/engine.py.229: process: 0.54813 | load: 0.00252 [I 2022-10-28 03:15:17] ...trap/engines/engine.py.230: loss: 0.12917 [I 2022-10-28 03:15:17] ...optimizers/trijoint.py.151: total_norm: 15.841765 [I 2022-10-28 03:15:22] ...trap/engines/engine.py.224: train: epoch 103 | batch 330/2304 [I 2022-10-28 03:15:22] ...trap/engines/engine.py.228: elapsed: 0:03:06 | left: 0:18:14 [I 2022-10-28 03:15:22] ...trap/engines/engine.py.229: process: 0.54573 | load: 0.00254 [I 2022-10-28 03:15:22] ...trap/engines/engine.py.230: loss: 0.15166 [I 2022-10-28 03:15:22] ...optimizers/trijoint.py.151: total_norm: 68.873184 [I 2022-10-28 03:15:28] ...trap/engines/engine.py.224: train: epoch 103 | batch 340/2304 [I 2022-10-28 03:15:28] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:07 [I 2022-10-28 03:15:28] ...trap/engines/engine.py.229: process: 0.54878 | load: 0.00243 [I 2022-10-28 03:15:28] ...trap/engines/engine.py.230: loss: 0.11198 [I 2022-10-28 03:15:28] ...optimizers/trijoint.py.151: total_norm: 8.571668 [I 2022-10-28 03:15:34] ...trap/engines/engine.py.224: train: epoch 103 | batch 350/2304 [I 2022-10-28 03:15:34] ...trap/engines/engine.py.228: elapsed: 0:03:17 | left: 0:18:02 [I 2022-10-28 03:15:34] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00243 [I 2022-10-28 03:15:34] ...trap/engines/engine.py.230: loss: 0.13744 [I 2022-10-28 03:15:34] ...optimizers/trijoint.py.151: total_norm: 15.206191 [I 2022-10-28 03:15:39] ...trap/engines/engine.py.224: train: epoch 103 | batch 360/2304 [I 2022-10-28 03:15:39] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:17:56 [I 2022-10-28 03:15:39] ...trap/engines/engine.py.229: process: 0.54802 | load: 0.00244 [I 2022-10-28 03:15:39] ...trap/engines/engine.py.230: loss: 0.08611 [I 2022-10-28 03:15:39] ...optimizers/trijoint.py.151: total_norm: 12.150619 [I 2022-10-28 03:15:45] ...trap/engines/engine.py.224: train: epoch 103 | batch 370/2304 [I 2022-10-28 03:15:45] ...trap/engines/engine.py.228: elapsed: 0:03:28 | left: 0:17:43 [I 2022-10-28 03:15:45] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00257 [I 2022-10-28 03:15:45] ...trap/engines/engine.py.230: loss: 0.21175 [I 2022-10-28 03:15:45] ...optimizers/trijoint.py.151: total_norm: 30.569031 [I 2022-10-28 03:15:50] ...trap/engines/engine.py.224: train: epoch 103 | batch 380/2304 [I 2022-10-28 03:15:50] ...trap/engines/engine.py.228: elapsed: 0:03:34 | left: 0:17:49 [I 2022-10-28 03:15:50] ...trap/engines/engine.py.229: process: 0.55414 | load: 0.00248 [I 2022-10-28 03:15:50] ...trap/engines/engine.py.230: loss: 0.08916 [I 2022-10-28 03:15:50] ...optimizers/trijoint.py.151: total_norm: 11.237175 [I 2022-10-28 03:15:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 390/2304 [I 2022-10-28 03:15:56] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:43 [I 2022-10-28 03:15:56] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00250 [I 2022-10-28 03:15:56] ...trap/engines/engine.py.230: loss: 0.16338 [I 2022-10-28 03:15:56] ...optimizers/trijoint.py.151: total_norm: 16.478680 [I 2022-10-28 03:16:01] ...trap/engines/engine.py.224: train: epoch 103 | batch 400/2304 [I 2022-10-28 03:16:01] ...trap/engines/engine.py.228: elapsed: 0:03:45 | left: 0:17:34 [I 2022-10-28 03:16:01] ...trap/engines/engine.py.229: process: 0.54682 | load: 0.00247 [I 2022-10-28 03:16:01] ...trap/engines/engine.py.230: loss: 0.07797 [I 2022-10-28 03:16:01] ...optimizers/trijoint.py.151: total_norm: 12.753947 [I 2022-10-28 03:16:07] ...trap/engines/engine.py.224: train: epoch 103 | batch 410/2304 [I 2022-10-28 03:16:07] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:28 [I 2022-10-28 03:16:07] ...trap/engines/engine.py.229: process: 0.55405 | load: 0.00243 [I 2022-10-28 03:16:07] ...trap/engines/engine.py.230: loss: 0.16622 [I 2022-10-28 03:16:07] ...optimizers/trijoint.py.151: total_norm: 67.237579 [I 2022-10-28 03:16:12] ...trap/engines/engine.py.224: train: epoch 103 | batch 420/2304 [I 2022-10-28 03:16:12] ...trap/engines/engine.py.228: elapsed: 0:03:56 | left: 0:17:24 [I 2022-10-28 03:16:12] ...trap/engines/engine.py.229: process: 0.55381 | load: 0.00243 [I 2022-10-28 03:16:12] ...trap/engines/engine.py.230: loss: 0.22963 [I 2022-10-28 03:16:12] ...optimizers/trijoint.py.151: total_norm: 22.616934 [I 2022-10-28 03:16:18] ...trap/engines/engine.py.224: train: epoch 103 | batch 430/2304 [I 2022-10-28 03:16:18] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:17 [I 2022-10-28 03:16:18] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00259 [I 2022-10-28 03:16:18] ...trap/engines/engine.py.230: loss: 0.09965 [I 2022-10-28 03:16:18] ...optimizers/trijoint.py.151: total_norm: 10.520782 [I 2022-10-28 03:16:24] ...trap/engines/engine.py.224: train: epoch 103 | batch 440/2304 [I 2022-10-28 03:16:24] ...trap/engines/engine.py.228: elapsed: 0:04:07 | left: 0:17:08 [I 2022-10-28 03:16:24] ...trap/engines/engine.py.229: process: 0.53728 | load: 0.00244 [I 2022-10-28 03:16:24] ...trap/engines/engine.py.230: loss: 0.12757 [I 2022-10-28 03:16:24] ...optimizers/trijoint.py.151: total_norm: 45.702072 [I 2022-10-28 03:16:29] ...trap/engines/engine.py.224: train: epoch 103 | batch 450/2304 [I 2022-10-28 03:16:29] ...trap/engines/engine.py.228: elapsed: 0:04:13 | left: 0:17:02 [I 2022-10-28 03:16:29] ...trap/engines/engine.py.229: process: 0.55614 | load: 0.00241 [I 2022-10-28 03:16:29] ...trap/engines/engine.py.230: loss: 0.10720 [I 2022-10-28 03:16:29] ...optimizers/trijoint.py.151: total_norm: 12.820767 [I 2022-10-28 03:16:35] ...trap/engines/engine.py.224: train: epoch 103 | batch 460/2304 [I 2022-10-28 03:16:35] ...trap/engines/engine.py.228: elapsed: 0:04:18 | left: 0:16:55 [I 2022-10-28 03:16:35] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00246 [I 2022-10-28 03:16:35] ...trap/engines/engine.py.230: loss: 0.16675 [I 2022-10-28 03:16:35] ...optimizers/trijoint.py.151: total_norm: 15.372824 [I 2022-10-28 03:16:40] ...trap/engines/engine.py.224: train: epoch 103 | batch 470/2304 [I 2022-10-28 03:16:40] ...trap/engines/engine.py.228: elapsed: 0:04:24 | left: 0:16:55 [I 2022-10-28 03:16:40] ...trap/engines/engine.py.229: process: 0.55436 | load: 0.00243 [I 2022-10-28 03:16:40] ...trap/engines/engine.py.230: loss: 0.12888 [I 2022-10-28 03:16:40] ...optimizers/trijoint.py.151: total_norm: 81.091591 [I 2022-10-28 03:16:46] ...trap/engines/engine.py.224: train: epoch 103 | batch 480/2304 [I 2022-10-28 03:16:46] ...trap/engines/engine.py.228: elapsed: 0:04:30 | left: 0:17:14 [I 2022-10-28 03:16:46] ...trap/engines/engine.py.229: process: 0.56918 | load: 0.00242 [I 2022-10-28 03:16:46] ...trap/engines/engine.py.230: loss: 0.12717 [I 2022-10-28 03:16:46] ...optimizers/trijoint.py.151: total_norm: 12.457611 [I 2022-10-28 03:16:51] ...trap/engines/engine.py.224: train: epoch 103 | batch 490/2304 [I 2022-10-28 03:16:51] ...trap/engines/engine.py.228: elapsed: 0:04:35 | left: 0:17:15 [I 2022-10-28 03:16:51] ...trap/engines/engine.py.229: process: 0.61865 | load: 0.00241 [I 2022-10-28 03:16:51] ...trap/engines/engine.py.230: loss: 0.13259 [I 2022-10-28 03:16:51] ...optimizers/trijoint.py.151: total_norm: 13.880507 [I 2022-10-28 03:16:57] ...trap/engines/engine.py.224: train: epoch 103 | batch 500/2304 [I 2022-10-28 03:16:57] ...trap/engines/engine.py.228: elapsed: 0:04:41 | left: 0:16:39 [I 2022-10-28 03:16:57] ...trap/engines/engine.py.229: process: 0.53861 | load: 0.00242 [I 2022-10-28 03:16:57] ...trap/engines/engine.py.230: loss: 0.07510 [I 2022-10-28 03:16:57] ...optimizers/trijoint.py.151: total_norm: 19.925547 [I 2022-10-28 03:17:03] ...trap/engines/engine.py.224: train: epoch 103 | batch 510/2304 [I 2022-10-28 03:17:03] ...trap/engines/engine.py.228: elapsed: 0:04:46 | left: 0:16:32 [I 2022-10-28 03:17:03] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00288 [I 2022-10-28 03:17:03] ...trap/engines/engine.py.230: loss: 0.12514 [I 2022-10-28 03:17:03] ...optimizers/trijoint.py.151: total_norm: 15.734602 [I 2022-10-28 03:17:08] ...trap/engines/engine.py.224: train: epoch 103 | batch 520/2304 [I 2022-10-28 03:17:08] ...trap/engines/engine.py.228: elapsed: 0:04:52 | left: 0:16:34 [I 2022-10-28 03:17:08] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00249 [I 2022-10-28 03:17:08] ...trap/engines/engine.py.230: loss: 0.09429 [I 2022-10-28 03:17:08] ...optimizers/trijoint.py.151: total_norm: 23.909653 [I 2022-10-28 03:17:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 530/2304 [I 2022-10-28 03:17:14] ...trap/engines/engine.py.228: elapsed: 0:04:57 | left: 0:16:20 [I 2022-10-28 03:17:14] ...trap/engines/engine.py.229: process: 0.55065 | load: 0.00246 [I 2022-10-28 03:17:14] ...trap/engines/engine.py.230: loss: 0.13549 [I 2022-10-28 03:17:14] ...optimizers/trijoint.py.151: total_norm: 14.792027 [I 2022-10-28 03:17:19] ...trap/engines/engine.py.224: train: epoch 103 | batch 540/2304 [I 2022-10-28 03:17:19] ...trap/engines/engine.py.228: elapsed: 0:05:03 | left: 0:16:11 [I 2022-10-28 03:17:19] ...trap/engines/engine.py.229: process: 0.55460 | load: 0.00246 [I 2022-10-28 03:17:19] ...trap/engines/engine.py.230: loss: 0.08997 [I 2022-10-28 03:17:19] ...optimizers/trijoint.py.151: total_norm: 10.208174 [I 2022-10-28 03:17:25] ...trap/engines/engine.py.224: train: epoch 103 | batch 550/2304 [I 2022-10-28 03:17:25] ...trap/engines/engine.py.228: elapsed: 0:05:08 | left: 0:16:16 [I 2022-10-28 03:17:25] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00241 [I 2022-10-28 03:17:25] ...trap/engines/engine.py.230: loss: 0.14335 [I 2022-10-28 03:17:25] ...optimizers/trijoint.py.151: total_norm: 38.049335 [I 2022-10-28 03:17:30] ...trap/engines/engine.py.224: train: epoch 103 | batch 560/2304 [I 2022-10-28 03:17:30] ...trap/engines/engine.py.228: elapsed: 0:05:14 | left: 0:16:09 [I 2022-10-28 03:17:30] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00250 [I 2022-10-28 03:17:30] ...trap/engines/engine.py.230: loss: 0.13589 [I 2022-10-28 03:17:30] ...optimizers/trijoint.py.151: total_norm: 10.469569 [I 2022-10-28 03:17:36] ...trap/engines/engine.py.224: train: epoch 103 | batch 570/2304 [I 2022-10-28 03:17:36] ...trap/engines/engine.py.228: elapsed: 0:05:20 | left: 0:16:30 [I 2022-10-28 03:17:36] ...trap/engines/engine.py.229: process: 0.54699 | load: 0.00268 [I 2022-10-28 03:17:36] ...trap/engines/engine.py.230: loss: 0.09530 [I 2022-10-28 03:17:36] ...optimizers/trijoint.py.151: total_norm: 12.340581 [I 2022-10-28 03:17:42] ...trap/engines/engine.py.224: train: epoch 103 | batch 580/2304 [I 2022-10-28 03:17:42] ...trap/engines/engine.py.228: elapsed: 0:05:25 | left: 0:16:01 [I 2022-10-28 03:17:42] ...trap/engines/engine.py.229: process: 0.55845 | load: 0.00243 [I 2022-10-28 03:17:42] ...trap/engines/engine.py.230: loss: 0.19743 [I 2022-10-28 03:17:42] ...optimizers/trijoint.py.151: total_norm: 12.374987 [I 2022-10-28 03:17:47] ...trap/engines/engine.py.224: train: epoch 103 | batch 590/2304 [I 2022-10-28 03:17:47] ...trap/engines/engine.py.228: elapsed: 0:05:31 | left: 0:15:49 [I 2022-10-28 03:17:47] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00242 [I 2022-10-28 03:17:47] ...trap/engines/engine.py.230: loss: 0.10651 [I 2022-10-28 03:17:47] ...optimizers/trijoint.py.151: total_norm: 19.421026 [I 2022-10-28 03:17:53] ...trap/engines/engine.py.224: train: epoch 103 | batch 600/2304 [I 2022-10-28 03:17:53] ...trap/engines/engine.py.228: elapsed: 0:05:36 | left: 0:15:47 [I 2022-10-28 03:17:53] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00245 [I 2022-10-28 03:17:53] ...trap/engines/engine.py.230: loss: 0.10947 [I 2022-10-28 03:17:53] ...optimizers/trijoint.py.151: total_norm: 19.611259 [I 2022-10-28 03:17:58] ...trap/engines/engine.py.224: train: epoch 103 | batch 610/2304 [I 2022-10-28 03:17:58] ...trap/engines/engine.py.228: elapsed: 0:05:42 | left: 0:15:37 [I 2022-10-28 03:17:58] ...trap/engines/engine.py.229: process: 0.54284 | load: 0.00247 [I 2022-10-28 03:17:58] ...trap/engines/engine.py.230: loss: 0.10103 [I 2022-10-28 03:17:58] ...optimizers/trijoint.py.151: total_norm: 8.823661 [I 2022-10-28 03:18:04] ...trap/engines/engine.py.224: train: epoch 103 | batch 620/2304 [I 2022-10-28 03:18:04] ...trap/engines/engine.py.228: elapsed: 0:05:48 | left: 0:15:37 [I 2022-10-28 03:18:04] ...trap/engines/engine.py.229: process: 0.55442 | load: 0.00246 [I 2022-10-28 03:18:04] ...trap/engines/engine.py.230: loss: 0.15743 [I 2022-10-28 03:18:04] ...optimizers/trijoint.py.151: total_norm: 14.566830 [I 2022-10-28 03:18:09] ...trap/engines/engine.py.224: train: epoch 103 | batch 630/2304 [I 2022-10-28 03:18:09] ...trap/engines/engine.py.228: elapsed: 0:05:53 | left: 0:15:29 [I 2022-10-28 03:18:09] ...trap/engines/engine.py.229: process: 0.55937 | load: 0.00252 [I 2022-10-28 03:18:09] ...trap/engines/engine.py.230: loss: 0.11853 [I 2022-10-28 03:18:09] ...optimizers/trijoint.py.151: total_norm: 24.359919 [I 2022-10-28 03:18:15] ...trap/engines/engine.py.224: train: epoch 103 | batch 640/2304 [I 2022-10-28 03:18:15] ...trap/engines/engine.py.228: elapsed: 0:05:59 | left: 0:15:32 [I 2022-10-28 03:18:15] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00246 [I 2022-10-28 03:18:15] ...trap/engines/engine.py.230: loss: 0.09050 [I 2022-10-28 03:18:15] ...optimizers/trijoint.py.151: total_norm: 16.590521 [I 2022-10-28 03:18:21] ...trap/engines/engine.py.224: train: epoch 103 | batch 650/2304 [I 2022-10-28 03:18:21] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:18 [I 2022-10-28 03:18:21] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00243 [I 2022-10-28 03:18:21] ...trap/engines/engine.py.230: loss: 0.18404 [I 2022-10-28 03:18:21] ...optimizers/trijoint.py.151: total_norm: 20.082693 [I 2022-10-28 03:18:26] ...trap/engines/engine.py.224: train: epoch 103 | batch 660/2304 [I 2022-10-28 03:18:26] ...trap/engines/engine.py.228: elapsed: 0:06:10 | left: 0:15:14 [I 2022-10-28 03:18:26] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00267 [I 2022-10-28 03:18:26] ...trap/engines/engine.py.230: loss: 0.13349 [I 2022-10-28 03:18:26] ...optimizers/trijoint.py.151: total_norm: 13.890167 [I 2022-10-28 03:18:32] ...trap/engines/engine.py.224: train: epoch 103 | batch 670/2304 [I 2022-10-28 03:18:32] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:10 [I 2022-10-28 03:18:32] ...trap/engines/engine.py.229: process: 0.56327 | load: 0.00251 [I 2022-10-28 03:18:32] ...trap/engines/engine.py.230: loss: 0.10824 [I 2022-10-28 03:18:32] ...optimizers/trijoint.py.151: total_norm: 9.853372 [I 2022-10-28 03:18:37] ...trap/engines/engine.py.224: train: epoch 103 | batch 680/2304 [I 2022-10-28 03:18:37] ...trap/engines/engine.py.228: elapsed: 0:06:21 | left: 0:15:06 [I 2022-10-28 03:18:37] ...trap/engines/engine.py.229: process: 0.56431 | load: 0.00254 [I 2022-10-28 03:18:37] ...trap/engines/engine.py.230: loss: 0.10452 [I 2022-10-28 03:18:37] ...optimizers/trijoint.py.151: total_norm: 14.254619 [I 2022-10-28 03:18:43] ...trap/engines/engine.py.224: train: epoch 103 | batch 690/2304 [I 2022-10-28 03:18:43] ...trap/engines/engine.py.228: elapsed: 0:06:27 | left: 0:15:00 [I 2022-10-28 03:18:43] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00267 [I 2022-10-28 03:18:43] ...trap/engines/engine.py.230: loss: 0.10281 [I 2022-10-28 03:18:43] ...optimizers/trijoint.py.151: total_norm: 14.754002 [I 2022-10-28 03:18:48] ...trap/engines/engine.py.224: train: epoch 103 | batch 700/2304 [I 2022-10-28 03:18:48] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:14:52 [I 2022-10-28 03:18:48] ...trap/engines/engine.py.229: process: 0.55377 | load: 0.00246 [I 2022-10-28 03:18:48] ...trap/engines/engine.py.230: loss: 0.14850 [I 2022-10-28 03:18:48] ...optimizers/trijoint.py.151: total_norm: 25.182156 [I 2022-10-28 03:18:54] ...trap/engines/engine.py.224: train: epoch 103 | batch 710/2304 [I 2022-10-28 03:18:54] ...trap/engines/engine.py.228: elapsed: 0:06:38 | left: 0:14:43 [I 2022-10-28 03:18:54] ...trap/engines/engine.py.229: process: 0.53939 | load: 0.00243 [I 2022-10-28 03:18:54] ...trap/engines/engine.py.230: loss: 0.11592 [I 2022-10-28 03:18:54] ...optimizers/trijoint.py.151: total_norm: 12.998696 [I 2022-10-28 03:19:00] ...trap/engines/engine.py.224: train: epoch 103 | batch 720/2304 [I 2022-10-28 03:19:00] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:14:38 [I 2022-10-28 03:19:00] ...trap/engines/engine.py.229: process: 0.54334 | load: 0.00250 [I 2022-10-28 03:19:00] ...trap/engines/engine.py.230: loss: 0.09910 [I 2022-10-28 03:19:00] ...optimizers/trijoint.py.151: total_norm: 18.832014 [I 2022-10-28 03:19:05] ...trap/engines/engine.py.224: train: epoch 103 | batch 730/2304 [I 2022-10-28 03:19:05] ...trap/engines/engine.py.228: elapsed: 0:06:49 | left: 0:14:33 [I 2022-10-28 03:19:05] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00250 [I 2022-10-28 03:19:05] ...trap/engines/engine.py.230: loss: 0.09058 [I 2022-10-28 03:19:05] ...optimizers/trijoint.py.151: total_norm: 10.834425 [I 2022-10-28 03:19:11] ...trap/engines/engine.py.224: train: epoch 103 | batch 740/2304 [I 2022-10-28 03:19:11] ...trap/engines/engine.py.228: elapsed: 0:06:55 | left: 0:14:35 [I 2022-10-28 03:19:11] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00241 [I 2022-10-28 03:19:11] ...trap/engines/engine.py.230: loss: 0.10029 [I 2022-10-28 03:19:11] ...optimizers/trijoint.py.151: total_norm: 10.091911 [I 2022-10-28 03:19:16] ...trap/engines/engine.py.224: train: epoch 103 | batch 750/2304 [I 2022-10-28 03:19:16] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:33 [I 2022-10-28 03:19:16] ...trap/engines/engine.py.229: process: 0.56109 | load: 0.00243 [I 2022-10-28 03:19:16] ...trap/engines/engine.py.230: loss: 0.12406 [I 2022-10-28 03:19:16] ...optimizers/trijoint.py.151: total_norm: 15.128811 [I 2022-10-28 03:19:22] ...trap/engines/engine.py.224: train: epoch 103 | batch 760/2304 [I 2022-10-28 03:19:22] ...trap/engines/engine.py.228: elapsed: 0:07:06 | left: 0:14:19 [I 2022-10-28 03:19:22] ...trap/engines/engine.py.229: process: 0.55422 | load: 0.00242 [I 2022-10-28 03:19:22] ...trap/engines/engine.py.230: loss: 0.10099 [I 2022-10-28 03:19:22] ...optimizers/trijoint.py.151: total_norm: 11.618708 [I 2022-10-28 03:19:28] ...trap/engines/engine.py.224: train: epoch 103 | batch 770/2304 [I 2022-10-28 03:19:28] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:12 [I 2022-10-28 03:19:28] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00241 [I 2022-10-28 03:19:28] ...trap/engines/engine.py.230: loss: 0.11965 [I 2022-10-28 03:19:28] ...optimizers/trijoint.py.151: total_norm: 10.280548 [I 2022-10-28 03:19:33] ...trap/engines/engine.py.224: train: epoch 103 | batch 780/2304 [I 2022-10-28 03:19:33] ...trap/engines/engine.py.228: elapsed: 0:07:17 | left: 0:14:08 [I 2022-10-28 03:19:33] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00242 [I 2022-10-28 03:19:33] ...trap/engines/engine.py.230: loss: 0.11629 [I 2022-10-28 03:19:33] ...optimizers/trijoint.py.151: total_norm: 17.642052 [I 2022-10-28 03:19:39] ...trap/engines/engine.py.224: train: epoch 103 | batch 790/2304 [I 2022-10-28 03:19:39] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:13:59 [I 2022-10-28 03:19:39] ...trap/engines/engine.py.229: process: 0.55218 | load: 0.00243 [I 2022-10-28 03:19:39] ...trap/engines/engine.py.230: loss: 0.16587 [I 2022-10-28 03:19:39] ...optimizers/trijoint.py.151: total_norm: 22.500450 [I 2022-10-28 03:19:44] ...trap/engines/engine.py.224: train: epoch 103 | batch 800/2304 [I 2022-10-28 03:19:44] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:01 [I 2022-10-28 03:19:44] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00260 [I 2022-10-28 03:19:44] ...trap/engines/engine.py.230: loss: 0.16491 [I 2022-10-28 03:19:44] ...optimizers/trijoint.py.151: total_norm: 17.109015 [I 2022-10-28 03:19:50] ...trap/engines/engine.py.224: train: epoch 103 | batch 810/2304 [I 2022-10-28 03:19:50] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:13:48 [I 2022-10-28 03:19:50] ...trap/engines/engine.py.229: process: 0.55799 | load: 0.00249 [I 2022-10-28 03:19:50] ...trap/engines/engine.py.230: loss: 0.14500 [I 2022-10-28 03:19:50] ...optimizers/trijoint.py.151: total_norm: 22.086775 [I 2022-10-28 03:19:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 820/2304 [I 2022-10-28 03:19:56] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:13:49 [I 2022-10-28 03:19:56] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00251 [I 2022-10-28 03:19:56] ...trap/engines/engine.py.230: loss: 0.11961 [I 2022-10-28 03:19:56] ...optimizers/trijoint.py.151: total_norm: 25.453447 [I 2022-10-28 03:20:01] ...trap/engines/engine.py.224: train: epoch 103 | batch 830/2304 [I 2022-10-28 03:20:01] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:13:41 [I 2022-10-28 03:20:01] ...trap/engines/engine.py.229: process: 0.55539 | load: 0.00245 [I 2022-10-28 03:20:01] ...trap/engines/engine.py.230: loss: 0.07497 [I 2022-10-28 03:20:01] ...optimizers/trijoint.py.151: total_norm: 17.498249 [I 2022-10-28 03:20:07] ...trap/engines/engine.py.224: train: epoch 103 | batch 840/2304 [I 2022-10-28 03:20:07] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:13:32 [I 2022-10-28 03:20:07] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00246 [I 2022-10-28 03:20:07] ...trap/engines/engine.py.230: loss: 0.11865 [I 2022-10-28 03:20:07] ...optimizers/trijoint.py.151: total_norm: 12.890532 [I 2022-10-28 03:20:12] ...trap/engines/engine.py.224: train: epoch 103 | batch 850/2304 [I 2022-10-28 03:20:12] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:13:35 [I 2022-10-28 03:20:12] ...trap/engines/engine.py.229: process: 0.56378 | load: 0.00247 [I 2022-10-28 03:20:12] ...trap/engines/engine.py.230: loss: 0.09480 [I 2022-10-28 03:20:12] ...optimizers/trijoint.py.151: total_norm: 9.701784 [I 2022-10-28 03:20:18] ...trap/engines/engine.py.224: train: epoch 103 | batch 860/2304 [I 2022-10-28 03:20:18] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:24 [I 2022-10-28 03:20:18] ...trap/engines/engine.py.229: process: 0.55818 | load: 0.00246 [I 2022-10-28 03:20:18] ...trap/engines/engine.py.230: loss: 0.09595 [I 2022-10-28 03:20:18] ...optimizers/trijoint.py.151: total_norm: 8.259772 [I 2022-10-28 03:20:24] ...trap/engines/engine.py.224: train: epoch 103 | batch 870/2304 [I 2022-10-28 03:20:24] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:27 [I 2022-10-28 03:20:24] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00240 [I 2022-10-28 03:20:24] ...trap/engines/engine.py.230: loss: 0.09636 [I 2022-10-28 03:20:24] ...optimizers/trijoint.py.151: total_norm: 26.640163 [I 2022-10-28 03:20:29] ...trap/engines/engine.py.224: train: epoch 103 | batch 880/2304 [I 2022-10-28 03:20:29] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:11 [I 2022-10-28 03:20:29] ...trap/engines/engine.py.229: process: 0.55703 | load: 0.00244 [I 2022-10-28 03:20:29] ...trap/engines/engine.py.230: loss: 0.09281 [I 2022-10-28 03:20:29] ...optimizers/trijoint.py.151: total_norm: 11.883034 [I 2022-10-28 03:20:35] ...trap/engines/engine.py.224: train: epoch 103 | batch 890/2304 [I 2022-10-28 03:20:35] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:04 [I 2022-10-28 03:20:35] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00245 [I 2022-10-28 03:20:35] ...trap/engines/engine.py.230: loss: 0.10723 [I 2022-10-28 03:20:35] ...optimizers/trijoint.py.151: total_norm: 20.516994 [I 2022-10-28 03:20:40] ...trap/engines/engine.py.224: train: epoch 103 | batch 900/2304 [I 2022-10-28 03:20:40] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:12:55 [I 2022-10-28 03:20:40] ...trap/engines/engine.py.229: process: 0.54607 | load: 0.00248 [I 2022-10-28 03:20:40] ...trap/engines/engine.py.230: loss: 0.08588 [I 2022-10-28 03:20:40] ...optimizers/trijoint.py.151: total_norm: 13.531219 [I 2022-10-28 03:20:46] ...trap/engines/engine.py.224: train: epoch 103 | batch 910/2304 [I 2022-10-28 03:20:46] ...trap/engines/engine.py.228: elapsed: 0:08:29 | left: 0:12:53 [I 2022-10-28 03:20:46] ...trap/engines/engine.py.229: process: 0.55291 | load: 0.00248 [I 2022-10-28 03:20:46] ...trap/engines/engine.py.230: loss: 0.10809 [I 2022-10-28 03:20:46] ...optimizers/trijoint.py.151: total_norm: 18.549133 [I 2022-10-28 03:20:51] ...trap/engines/engine.py.224: train: epoch 103 | batch 920/2304 [I 2022-10-28 03:20:51] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:12:49 [I 2022-10-28 03:20:51] ...trap/engines/engine.py.229: process: 0.56242 | load: 0.00245 [I 2022-10-28 03:20:51] ...trap/engines/engine.py.230: loss: 0.11225 [I 2022-10-28 03:20:51] ...optimizers/trijoint.py.151: total_norm: 20.361919 [I 2022-10-28 03:20:57] ...trap/engines/engine.py.224: train: epoch 103 | batch 930/2304 [I 2022-10-28 03:20:57] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:12:49 [I 2022-10-28 03:20:57] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00242 [I 2022-10-28 03:20:57] ...trap/engines/engine.py.230: loss: 0.13304 [I 2022-10-28 03:20:57] ...optimizers/trijoint.py.151: total_norm: 13.459144 [I 2022-10-28 03:21:02] ...trap/engines/engine.py.224: train: epoch 103 | batch 940/2304 [I 2022-10-28 03:21:02] ...trap/engines/engine.py.228: elapsed: 0:08:46 | left: 0:12:29 [I 2022-10-28 03:21:02] ...trap/engines/engine.py.229: process: 0.54098 | load: 0.00244 [I 2022-10-28 03:21:02] ...trap/engines/engine.py.230: loss: 0.09750 [I 2022-10-28 03:21:02] ...optimizers/trijoint.py.151: total_norm: 11.903630 [I 2022-10-28 03:21:08] ...trap/engines/engine.py.224: train: epoch 103 | batch 950/2304 [I 2022-10-28 03:21:08] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:26 [I 2022-10-28 03:21:08] ...trap/engines/engine.py.229: process: 0.55936 | load: 0.00246 [I 2022-10-28 03:21:08] ...trap/engines/engine.py.230: loss: 0.11000 [I 2022-10-28 03:21:08] ...optimizers/trijoint.py.151: total_norm: 15.601937 [I 2022-10-28 03:21:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 960/2304 [I 2022-10-28 03:21:14] ...trap/engines/engine.py.228: elapsed: 0:08:57 | left: 0:12:28 [I 2022-10-28 03:21:14] ...trap/engines/engine.py.229: process: 0.56554 | load: 0.00245 [I 2022-10-28 03:21:14] ...trap/engines/engine.py.230: loss: 0.11373 [I 2022-10-28 03:21:14] ...optimizers/trijoint.py.151: total_norm: 12.830462 [I 2022-10-28 03:21:19] ...trap/engines/engine.py.224: train: epoch 103 | batch 970/2304 [I 2022-10-28 03:21:19] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:20 [I 2022-10-28 03:21:19] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00244 [I 2022-10-28 03:21:19] ...trap/engines/engine.py.230: loss: 0.10966 [I 2022-10-28 03:21:19] ...optimizers/trijoint.py.151: total_norm: 10.590812 [I 2022-10-28 03:21:25] ...trap/engines/engine.py.224: train: epoch 103 | batch 980/2304 [I 2022-10-28 03:21:25] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:25 [I 2022-10-28 03:21:25] ...trap/engines/engine.py.229: process: 0.54724 | load: 0.00241 [I 2022-10-28 03:21:25] ...trap/engines/engine.py.230: loss: 0.10240 [I 2022-10-28 03:21:25] ...optimizers/trijoint.py.151: total_norm: 7.702235 [I 2022-10-28 03:21:30] ...trap/engines/engine.py.224: train: epoch 103 | batch 990/2304 [I 2022-10-28 03:21:30] ...trap/engines/engine.py.228: elapsed: 0:09:14 | left: 0:12:09 [I 2022-10-28 03:21:30] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00242 [I 2022-10-28 03:21:30] ...trap/engines/engine.py.230: loss: 0.11086 [I 2022-10-28 03:21:30] ...optimizers/trijoint.py.151: total_norm: 26.117325 [I 2022-10-28 03:21:36] ...trap/engines/engine.py.224: train: epoch 103 | batch 1000/2304 [I 2022-10-28 03:21:36] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:03 [I 2022-10-28 03:21:36] ...trap/engines/engine.py.229: process: 0.55273 | load: 0.00244 [I 2022-10-28 03:21:36] ...trap/engines/engine.py.230: loss: 0.08257 [I 2022-10-28 03:21:36] ...optimizers/trijoint.py.151: total_norm: 15.858422 [I 2022-10-28 03:21:42] ...trap/engines/engine.py.224: train: epoch 103 | batch 1010/2304 [I 2022-10-28 03:21:42] ...trap/engines/engine.py.228: elapsed: 0:09:25 | left: 0:11:56 [I 2022-10-28 03:21:42] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00242 [I 2022-10-28 03:21:42] ...trap/engines/engine.py.230: loss: 0.12112 [I 2022-10-28 03:21:42] ...optimizers/trijoint.py.151: total_norm: 9.208908 [I 2022-10-28 03:21:47] ...trap/engines/engine.py.224: train: epoch 103 | batch 1020/2304 [I 2022-10-28 03:21:47] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:11:49 [I 2022-10-28 03:21:47] ...trap/engines/engine.py.229: process: 0.54885 | load: 0.00244 [I 2022-10-28 03:21:47] ...trap/engines/engine.py.230: loss: 0.11352 [I 2022-10-28 03:21:47] ...optimizers/trijoint.py.151: total_norm: 15.503416 [I 2022-10-28 03:21:53] ...trap/engines/engine.py.224: train: epoch 103 | batch 1030/2304 [I 2022-10-28 03:21:53] ...trap/engines/engine.py.228: elapsed: 0:09:36 | left: 0:11:46 [I 2022-10-28 03:21:53] ...trap/engines/engine.py.229: process: 0.56527 | load: 0.00241 [I 2022-10-28 03:21:53] ...trap/engines/engine.py.230: loss: 0.08448 [I 2022-10-28 03:21:53] ...optimizers/trijoint.py.151: total_norm: 9.374845 [I 2022-10-28 03:21:58] ...trap/engines/engine.py.224: train: epoch 103 | batch 1040/2304 [I 2022-10-28 03:21:58] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:11:42 [I 2022-10-28 03:21:58] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00242 [I 2022-10-28 03:21:58] ...trap/engines/engine.py.230: loss: 0.13821 [I 2022-10-28 03:21:58] ...optimizers/trijoint.py.151: total_norm: 15.390590 [I 2022-10-28 03:22:04] ...trap/engines/engine.py.224: train: epoch 103 | batch 1050/2304 [I 2022-10-28 03:22:04] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:33 [I 2022-10-28 03:22:04] ...trap/engines/engine.py.229: process: 0.55628 | load: 0.00249 [I 2022-10-28 03:22:04] ...trap/engines/engine.py.230: loss: 0.14396 [I 2022-10-28 03:22:04] ...optimizers/trijoint.py.151: total_norm: 16.240610 [I 2022-10-28 03:22:09] ...trap/engines/engine.py.224: train: epoch 103 | batch 1060/2304 [I 2022-10-28 03:22:09] ...trap/engines/engine.py.228: elapsed: 0:09:53 | left: 0:11:29 [I 2022-10-28 03:22:09] ...trap/engines/engine.py.229: process: 0.54925 | load: 0.00247 [I 2022-10-28 03:22:09] ...trap/engines/engine.py.230: loss: 0.10683 [I 2022-10-28 03:22:09] ...optimizers/trijoint.py.151: total_norm: 13.655892 [I 2022-10-28 03:22:15] ...trap/engines/engine.py.224: train: epoch 103 | batch 1070/2304 [I 2022-10-28 03:22:15] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:24 [I 2022-10-28 03:22:15] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00244 [I 2022-10-28 03:22:15] ...trap/engines/engine.py.230: loss: 0.14188 [I 2022-10-28 03:22:15] ...optimizers/trijoint.py.151: total_norm: 48.145763 [I 2022-10-28 03:22:20] ...trap/engines/engine.py.224: train: epoch 103 | batch 1080/2304 [I 2022-10-28 03:22:20] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:15 [I 2022-10-28 03:22:20] ...trap/engines/engine.py.229: process: 0.55207 | load: 0.00246 [I 2022-10-28 03:22:20] ...trap/engines/engine.py.230: loss: 0.17528 [I 2022-10-28 03:22:20] ...optimizers/trijoint.py.151: total_norm: 21.894400 [I 2022-10-28 03:22:26] ...trap/engines/engine.py.224: train: epoch 103 | batch 1090/2304 [I 2022-10-28 03:22:26] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:16 [I 2022-10-28 03:22:26] ...trap/engines/engine.py.229: process: 0.55462 | load: 0.00244 [I 2022-10-28 03:22:26] ...trap/engines/engine.py.230: loss: 0.11935 [I 2022-10-28 03:22:26] ...optimizers/trijoint.py.151: total_norm: 10.474212 [I 2022-10-28 03:22:32] ...trap/engines/engine.py.224: train: epoch 103 | batch 1100/2304 [I 2022-10-28 03:22:32] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:07 [I 2022-10-28 03:22:32] ...trap/engines/engine.py.229: process: 0.55624 | load: 0.00243 [I 2022-10-28 03:22:32] ...trap/engines/engine.py.230: loss: 0.13448 [I 2022-10-28 03:22:32] ...optimizers/trijoint.py.151: total_norm: 10.670551 [I 2022-10-28 03:22:37] ...trap/engines/engine.py.224: train: epoch 103 | batch 1110/2304 [I 2022-10-28 03:22:37] ...trap/engines/engine.py.228: elapsed: 0:10:21 | left: 0:11:08 [I 2022-10-28 03:22:37] ...trap/engines/engine.py.229: process: 0.56184 | load: 0.00243 [I 2022-10-28 03:22:37] ...trap/engines/engine.py.230: loss: 0.14088 [I 2022-10-28 03:22:37] ...optimizers/trijoint.py.151: total_norm: 27.274662 [I 2022-10-28 03:22:43] ...trap/engines/engine.py.224: train: epoch 103 | batch 1120/2304 [I 2022-10-28 03:22:43] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:10:58 [I 2022-10-28 03:22:43] ...trap/engines/engine.py.229: process: 0.56638 | load: 0.00244 [I 2022-10-28 03:22:43] ...trap/engines/engine.py.230: loss: 0.12650 [I 2022-10-28 03:22:43] ...optimizers/trijoint.py.151: total_norm: 10.878227 [I 2022-10-28 03:22:48] ...trap/engines/engine.py.224: train: epoch 103 | batch 1130/2304 [I 2022-10-28 03:22:48] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:10:55 [I 2022-10-28 03:22:48] ...trap/engines/engine.py.229: process: 0.55768 | load: 0.00250 [I 2022-10-28 03:22:48] ...trap/engines/engine.py.230: loss: 0.14263 [I 2022-10-28 03:22:48] ...optimizers/trijoint.py.151: total_norm: 14.080980 [I 2022-10-28 03:22:54] ...trap/engines/engine.py.224: train: epoch 103 | batch 1140/2304 [I 2022-10-28 03:22:54] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:10:50 [I 2022-10-28 03:22:54] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00241 [I 2022-10-28 03:22:54] ...trap/engines/engine.py.230: loss: 0.18176 [I 2022-10-28 03:22:54] ...optimizers/trijoint.py.151: total_norm: 20.487841 [I 2022-10-28 03:22:59] ...trap/engines/engine.py.224: train: epoch 103 | batch 1150/2304 [I 2022-10-28 03:22:59] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:10:41 [I 2022-10-28 03:22:59] ...trap/engines/engine.py.229: process: 0.55551 | load: 0.00249 [I 2022-10-28 03:22:59] ...trap/engines/engine.py.230: loss: 0.11824 [I 2022-10-28 03:22:59] ...optimizers/trijoint.py.151: total_norm: 15.584365 [I 2022-10-28 03:23:05] ...trap/engines/engine.py.224: train: epoch 103 | batch 1160/2304 [I 2022-10-28 03:23:05] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:10:38 [I 2022-10-28 03:23:05] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00250 [I 2022-10-28 03:23:05] ...trap/engines/engine.py.230: loss: 0.19627 [I 2022-10-28 03:23:05] ...optimizers/trijoint.py.151: total_norm: 14.776978 [I 2022-10-28 03:23:11] ...trap/engines/engine.py.224: train: epoch 103 | batch 1170/2304 [I 2022-10-28 03:23:11] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:33 [I 2022-10-28 03:23:11] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00245 [I 2022-10-28 03:23:11] ...trap/engines/engine.py.230: loss: 0.12560 [I 2022-10-28 03:23:11] ...optimizers/trijoint.py.151: total_norm: 16.173687 [I 2022-10-28 03:23:16] ...trap/engines/engine.py.224: train: epoch 103 | batch 1180/2304 [I 2022-10-28 03:23:16] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:28 [I 2022-10-28 03:23:16] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00247 [I 2022-10-28 03:23:16] ...trap/engines/engine.py.230: loss: 0.08161 [I 2022-10-28 03:23:16] ...optimizers/trijoint.py.151: total_norm: 18.314014 [I 2022-10-28 03:23:22] ...trap/engines/engine.py.224: train: epoch 103 | batch 1190/2304 [I 2022-10-28 03:23:22] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:22 [I 2022-10-28 03:23:22] ...trap/engines/engine.py.229: process: 0.55246 | load: 0.00246 [I 2022-10-28 03:23:22] ...trap/engines/engine.py.230: loss: 0.11692 [I 2022-10-28 03:23:22] ...optimizers/trijoint.py.151: total_norm: 15.683289 [I 2022-10-28 03:23:27] ...trap/engines/engine.py.224: train: epoch 103 | batch 1200/2304 [I 2022-10-28 03:23:27] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:11 [I 2022-10-28 03:23:27] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00244 [I 2022-10-28 03:23:27] ...trap/engines/engine.py.230: loss: 0.13907 [I 2022-10-28 03:23:27] ...optimizers/trijoint.py.151: total_norm: 22.210314 [I 2022-10-28 03:23:33] ...trap/engines/engine.py.224: train: epoch 103 | batch 1210/2304 [I 2022-10-28 03:23:33] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:03 [I 2022-10-28 03:23:33] ...trap/engines/engine.py.229: process: 0.55767 | load: 0.00248 [I 2022-10-28 03:23:33] ...trap/engines/engine.py.230: loss: 0.10193 [I 2022-10-28 03:23:33] ...optimizers/trijoint.py.151: total_norm: 8.891217 [I 2022-10-28 03:23:38] ...trap/engines/engine.py.224: train: epoch 103 | batch 1220/2304 [I 2022-10-28 03:23:38] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:03 [I 2022-10-28 03:23:38] ...trap/engines/engine.py.229: process: 0.56319 | load: 0.00243 [I 2022-10-28 03:23:38] ...trap/engines/engine.py.230: loss: 0.16112 [I 2022-10-28 03:23:38] ...optimizers/trijoint.py.151: total_norm: 14.197042 [I 2022-10-28 03:23:44] ...trap/engines/engine.py.224: train: epoch 103 | batch 1230/2304 [I 2022-10-28 03:23:44] ...trap/engines/engine.py.228: elapsed: 0:11:28 | left: 0:10:00 [I 2022-10-28 03:23:44] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00243 [I 2022-10-28 03:23:44] ...trap/engines/engine.py.230: loss: 0.11382 [I 2022-10-28 03:23:44] ...optimizers/trijoint.py.151: total_norm: 12.774128 [I 2022-10-28 03:23:50] ...trap/engines/engine.py.224: train: epoch 103 | batch 1240/2304 [I 2022-10-28 03:23:50] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:09:49 [I 2022-10-28 03:23:50] ...trap/engines/engine.py.229: process: 0.55993 | load: 0.00245 [I 2022-10-28 03:23:50] ...trap/engines/engine.py.230: loss: 0.15732 [I 2022-10-28 03:23:50] ...optimizers/trijoint.py.151: total_norm: 15.662665 [I 2022-10-28 03:23:55] ...trap/engines/engine.py.224: train: epoch 103 | batch 1250/2304 [I 2022-10-28 03:23:55] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:09:42 [I 2022-10-28 03:23:55] ...trap/engines/engine.py.229: process: 0.55159 | load: 0.00249 [I 2022-10-28 03:23:55] ...trap/engines/engine.py.230: loss: 0.10429 [I 2022-10-28 03:23:55] ...optimizers/trijoint.py.151: total_norm: 13.346601 [I 2022-10-28 03:24:01] ...trap/engines/engine.py.224: train: epoch 103 | batch 1260/2304 [I 2022-10-28 03:24:01] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:09:40 [I 2022-10-28 03:24:01] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00244 [I 2022-10-28 03:24:01] ...trap/engines/engine.py.230: loss: 0.09447 [I 2022-10-28 03:24:01] ...optimizers/trijoint.py.151: total_norm: 13.383517 [I 2022-10-28 03:24:06] ...trap/engines/engine.py.224: train: epoch 103 | batch 1270/2304 [I 2022-10-28 03:24:06] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:30 [I 2022-10-28 03:24:06] ...trap/engines/engine.py.229: process: 0.53955 | load: 0.00243 [I 2022-10-28 03:24:06] ...trap/engines/engine.py.230: loss: 0.17175 [I 2022-10-28 03:24:06] ...optimizers/trijoint.py.151: total_norm: 28.527786 [I 2022-10-28 03:24:12] ...trap/engines/engine.py.224: train: epoch 103 | batch 1280/2304 [I 2022-10-28 03:24:12] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:23 [I 2022-10-28 03:24:12] ...trap/engines/engine.py.229: process: 0.55317 | load: 0.00243 [I 2022-10-28 03:24:12] ...trap/engines/engine.py.230: loss: 0.08452 [I 2022-10-28 03:24:12] ...optimizers/trijoint.py.151: total_norm: 12.583309 [I 2022-10-28 03:24:17] ...trap/engines/engine.py.224: train: epoch 103 | batch 1290/2304 [I 2022-10-28 03:24:17] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:22 [I 2022-10-28 03:24:17] ...trap/engines/engine.py.229: process: 0.56187 | load: 0.00244 [I 2022-10-28 03:24:17] ...trap/engines/engine.py.230: loss: 0.09647 [I 2022-10-28 03:24:17] ...optimizers/trijoint.py.151: total_norm: 11.429876 [I 2022-10-28 03:24:23] ...trap/engines/engine.py.224: train: epoch 103 | batch 1300/2304 [I 2022-10-28 03:24:23] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:20 [I 2022-10-28 03:24:23] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00247 [I 2022-10-28 03:24:23] ...trap/engines/engine.py.230: loss: 0.12374 [I 2022-10-28 03:24:23] ...optimizers/trijoint.py.151: total_norm: 14.588809 [I 2022-10-28 03:24:28] ...trap/engines/engine.py.224: train: epoch 103 | batch 1310/2304 [I 2022-10-28 03:24:28] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:14 [I 2022-10-28 03:24:28] ...trap/engines/engine.py.229: process: 0.55160 | load: 0.00240 [I 2022-10-28 03:24:28] ...trap/engines/engine.py.230: loss: 0.16950 [I 2022-10-28 03:24:28] ...optimizers/trijoint.py.151: total_norm: 18.954660 [I 2022-10-28 03:24:34] ...trap/engines/engine.py.224: train: epoch 103 | batch 1320/2304 [I 2022-10-28 03:24:34] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:07 [I 2022-10-28 03:24:34] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00239 [I 2022-10-28 03:24:34] ...trap/engines/engine.py.230: loss: 0.07784 [I 2022-10-28 03:24:34] ...optimizers/trijoint.py.151: total_norm: 12.432953 [I 2022-10-28 03:24:40] ...trap/engines/engine.py.224: train: epoch 103 | batch 1330/2304 [I 2022-10-28 03:24:40] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:08:58 [I 2022-10-28 03:24:40] ...trap/engines/engine.py.229: process: 0.54649 | load: 0.00240 [I 2022-10-28 03:24:40] ...trap/engines/engine.py.230: loss: 0.14267 [I 2022-10-28 03:24:40] ...optimizers/trijoint.py.151: total_norm: 29.781563 [I 2022-10-28 03:24:45] ...trap/engines/engine.py.224: train: epoch 103 | batch 1340/2304 [I 2022-10-28 03:24:45] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:08:54 [I 2022-10-28 03:24:45] ...trap/engines/engine.py.229: process: 0.54675 | load: 0.00243 [I 2022-10-28 03:24:45] ...trap/engines/engine.py.230: loss: 0.12160 [I 2022-10-28 03:24:45] ...optimizers/trijoint.py.151: total_norm: 10.677785 [I 2022-10-28 03:24:51] ...trap/engines/engine.py.224: train: epoch 103 | batch 1350/2304 [I 2022-10-28 03:24:51] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:08:49 [I 2022-10-28 03:24:51] ...trap/engines/engine.py.229: process: 0.54065 | load: 0.00251 [I 2022-10-28 03:24:51] ...trap/engines/engine.py.230: loss: 0.13315 [I 2022-10-28 03:24:51] ...optimizers/trijoint.py.151: total_norm: 24.974096 [I 2022-10-28 03:24:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 1360/2304 [I 2022-10-28 03:24:56] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:08:47 [I 2022-10-28 03:24:56] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00240 [I 2022-10-28 03:24:56] ...trap/engines/engine.py.230: loss: 0.07632 [I 2022-10-28 03:24:56] ...optimizers/trijoint.py.151: total_norm: 8.696966 [I 2022-10-28 03:25:02] ...trap/engines/engine.py.224: train: epoch 103 | batch 1370/2304 [I 2022-10-28 03:25:02] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:08:41 [I 2022-10-28 03:25:02] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00241 [I 2022-10-28 03:25:02] ...trap/engines/engine.py.230: loss: 0.10658 [I 2022-10-28 03:25:02] ...optimizers/trijoint.py.151: total_norm: 19.099752 [I 2022-10-28 03:25:07] ...trap/engines/engine.py.224: train: epoch 103 | batch 1380/2304 [I 2022-10-28 03:25:07] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:35 [I 2022-10-28 03:25:07] ...trap/engines/engine.py.229: process: 0.56530 | load: 0.00240 [I 2022-10-28 03:25:07] ...trap/engines/engine.py.230: loss: 0.15773 [I 2022-10-28 03:25:07] ...optimizers/trijoint.py.151: total_norm: 15.132405 [I 2022-10-28 03:25:13] ...trap/engines/engine.py.224: train: epoch 103 | batch 1390/2304 [I 2022-10-28 03:25:13] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:26 [I 2022-10-28 03:25:13] ...trap/engines/engine.py.229: process: 0.54328 | load: 0.00239 [I 2022-10-28 03:25:13] ...trap/engines/engine.py.230: loss: 0.07975 [I 2022-10-28 03:25:13] ...optimizers/trijoint.py.151: total_norm: 9.479335 [I 2022-10-28 03:25:19] ...trap/engines/engine.py.224: train: epoch 103 | batch 1400/2304 [I 2022-10-28 03:25:19] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:23 [I 2022-10-28 03:25:19] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00241 [I 2022-10-28 03:25:19] ...trap/engines/engine.py.230: loss: 0.09770 [I 2022-10-28 03:25:19] ...optimizers/trijoint.py.151: total_norm: 8.477348 [I 2022-10-28 03:25:24] ...trap/engines/engine.py.224: train: epoch 103 | batch 1410/2304 [I 2022-10-28 03:25:24] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:16 [I 2022-10-28 03:25:24] ...trap/engines/engine.py.229: process: 0.55240 | load: 0.00256 [I 2022-10-28 03:25:24] ...trap/engines/engine.py.230: loss: 0.13666 [I 2022-10-28 03:25:24] ...optimizers/trijoint.py.151: total_norm: 11.934511 [I 2022-10-28 03:25:30] ...trap/engines/engine.py.224: train: epoch 103 | batch 1420/2304 [I 2022-10-28 03:25:30] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:11 [I 2022-10-28 03:25:30] ...trap/engines/engine.py.229: process: 0.56082 | load: 0.00249 [I 2022-10-28 03:25:30] ...trap/engines/engine.py.230: loss: 0.13143 [I 2022-10-28 03:25:30] ...optimizers/trijoint.py.151: total_norm: 16.970457 [I 2022-10-28 03:25:35] ...trap/engines/engine.py.224: train: epoch 103 | batch 1430/2304 [I 2022-10-28 03:25:35] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:05 [I 2022-10-28 03:25:35] ...trap/engines/engine.py.229: process: 0.54835 | load: 0.00246 [I 2022-10-28 03:25:35] ...trap/engines/engine.py.230: loss: 0.13719 [I 2022-10-28 03:25:35] ...optimizers/trijoint.py.151: total_norm: 12.523384 [I 2022-10-28 03:25:41] ...trap/engines/engine.py.224: train: epoch 103 | batch 1440/2304 [I 2022-10-28 03:25:41] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:00 [I 2022-10-28 03:25:41] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00246 [I 2022-10-28 03:25:41] ...trap/engines/engine.py.230: loss: 0.09365 [I 2022-10-28 03:25:41] ...optimizers/trijoint.py.151: total_norm: 8.773602 [I 2022-10-28 03:25:46] ...trap/engines/engine.py.224: train: epoch 103 | batch 1450/2304 [I 2022-10-28 03:25:46] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:07:51 [I 2022-10-28 03:25:46] ...trap/engines/engine.py.229: process: 0.55484 | load: 0.00244 [I 2022-10-28 03:25:46] ...trap/engines/engine.py.230: loss: 0.15775 [I 2022-10-28 03:25:46] ...optimizers/trijoint.py.151: total_norm: 20.751526 [I 2022-10-28 03:25:52] ...trap/engines/engine.py.224: train: epoch 103 | batch 1460/2304 [I 2022-10-28 03:25:52] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:07:50 [I 2022-10-28 03:25:52] ...trap/engines/engine.py.229: process: 0.56226 | load: 0.00242 [I 2022-10-28 03:25:52] ...trap/engines/engine.py.230: loss: 0.14996 [I 2022-10-28 03:25:52] ...optimizers/trijoint.py.151: total_norm: 10.132003 [I 2022-10-28 03:25:58] ...trap/engines/engine.py.224: train: epoch 103 | batch 1470/2304 [I 2022-10-28 03:25:58] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:07:44 [I 2022-10-28 03:25:58] ...trap/engines/engine.py.229: process: 0.55432 | load: 0.00248 [I 2022-10-28 03:25:58] ...trap/engines/engine.py.230: loss: 0.11840 [I 2022-10-28 03:25:58] ...optimizers/trijoint.py.151: total_norm: 14.232758 [I 2022-10-28 03:26:03] ...trap/engines/engine.py.224: train: epoch 103 | batch 1480/2304 [I 2022-10-28 03:26:03] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:07:36 [I 2022-10-28 03:26:03] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00241 [I 2022-10-28 03:26:03] ...trap/engines/engine.py.230: loss: 0.09352 [I 2022-10-28 03:26:03] ...optimizers/trijoint.py.151: total_norm: 12.657878 [I 2022-10-28 03:26:09] ...trap/engines/engine.py.224: train: epoch 103 | batch 1490/2304 [I 2022-10-28 03:26:09] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:36 [I 2022-10-28 03:26:09] ...trap/engines/engine.py.229: process: 0.56098 | load: 0.00242 [I 2022-10-28 03:26:09] ...trap/engines/engine.py.230: loss: 0.14899 [I 2022-10-28 03:26:09] ...optimizers/trijoint.py.151: total_norm: 24.745592 [I 2022-10-28 03:26:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 1500/2304 [I 2022-10-28 03:26:14] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:23 [I 2022-10-28 03:26:14] ...trap/engines/engine.py.229: process: 0.53718 | load: 0.00241 [I 2022-10-28 03:26:14] ...trap/engines/engine.py.230: loss: 0.12031 [I 2022-10-28 03:26:14] ...optimizers/trijoint.py.151: total_norm: 14.011113 [I 2022-10-28 03:26:20] ...trap/engines/engine.py.224: train: epoch 103 | batch 1510/2304 [I 2022-10-28 03:26:20] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:20 [I 2022-10-28 03:26:20] ...trap/engines/engine.py.229: process: 0.54915 | load: 0.00239 [I 2022-10-28 03:26:20] ...trap/engines/engine.py.230: loss: 0.10260 [I 2022-10-28 03:26:20] ...optimizers/trijoint.py.151: total_norm: 20.331463 [I 2022-10-28 03:26:25] ...trap/engines/engine.py.224: train: epoch 103 | batch 1520/2304 [I 2022-10-28 03:26:25] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:16 [I 2022-10-28 03:26:25] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00240 [I 2022-10-28 03:26:25] ...trap/engines/engine.py.230: loss: 0.15623 [I 2022-10-28 03:26:25] ...optimizers/trijoint.py.151: total_norm: 16.425606 [I 2022-10-28 03:26:31] ...trap/engines/engine.py.224: train: epoch 103 | batch 1530/2304 [I 2022-10-28 03:26:31] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:12 [I 2022-10-28 03:26:31] ...trap/engines/engine.py.229: process: 0.56474 | load: 0.00242 [I 2022-10-28 03:26:31] ...trap/engines/engine.py.230: loss: 0.09089 [I 2022-10-28 03:26:31] ...optimizers/trijoint.py.151: total_norm: 8.356242 [I 2022-10-28 03:26:37] ...trap/engines/engine.py.224: train: epoch 103 | batch 1540/2304 [I 2022-10-28 03:26:37] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:02 [I 2022-10-28 03:26:37] ...trap/engines/engine.py.229: process: 0.55053 | load: 0.00245 [I 2022-10-28 03:26:37] ...trap/engines/engine.py.230: loss: 0.15261 [I 2022-10-28 03:26:37] ...optimizers/trijoint.py.151: total_norm: 18.946371 [I 2022-10-28 03:26:42] ...trap/engines/engine.py.224: train: epoch 103 | batch 1550/2304 [I 2022-10-28 03:26:42] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:06:59 [I 2022-10-28 03:26:42] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00240 [I 2022-10-28 03:26:42] ...trap/engines/engine.py.230: loss: 0.09571 [I 2022-10-28 03:26:42] ...optimizers/trijoint.py.151: total_norm: 14.306337 [I 2022-10-28 03:26:48] ...trap/engines/engine.py.224: train: epoch 103 | batch 1560/2304 [I 2022-10-28 03:26:48] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:06:59 [I 2022-10-28 03:26:48] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00253 [I 2022-10-28 03:26:48] ...trap/engines/engine.py.230: loss: 0.11623 [I 2022-10-28 03:26:48] ...optimizers/trijoint.py.151: total_norm: 27.645529 [I 2022-10-28 03:26:53] ...trap/engines/engine.py.224: train: epoch 103 | batch 1570/2304 [I 2022-10-28 03:26:53] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:06:52 [I 2022-10-28 03:26:53] ...trap/engines/engine.py.229: process: 0.56249 | load: 0.00248 [I 2022-10-28 03:26:53] ...trap/engines/engine.py.230: loss: 0.09991 [I 2022-10-28 03:26:53] ...optimizers/trijoint.py.151: total_norm: 16.503994 [I 2022-10-28 03:26:59] ...trap/engines/engine.py.224: train: epoch 103 | batch 1580/2304 [I 2022-10-28 03:26:59] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:06:55 [I 2022-10-28 03:26:59] ...trap/engines/engine.py.229: process: 0.55848 | load: 0.00246 [I 2022-10-28 03:26:59] ...trap/engines/engine.py.230: loss: 0.14607 [I 2022-10-28 03:26:59] ...optimizers/trijoint.py.151: total_norm: 17.550491 [I 2022-10-28 03:27:05] ...trap/engines/engine.py.224: train: epoch 103 | batch 1590/2304 [I 2022-10-28 03:27:05] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:06:37 [I 2022-10-28 03:27:05] ...trap/engines/engine.py.229: process: 0.54460 | load: 0.00244 [I 2022-10-28 03:27:05] ...trap/engines/engine.py.230: loss: 0.12202 [I 2022-10-28 03:27:05] ...optimizers/trijoint.py.151: total_norm: 27.314325 [I 2022-10-28 03:27:10] ...trap/engines/engine.py.224: train: epoch 103 | batch 1600/2304 [I 2022-10-28 03:27:10] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:31 [I 2022-10-28 03:27:10] ...trap/engines/engine.py.229: process: 0.55639 | load: 0.00245 [I 2022-10-28 03:27:10] ...trap/engines/engine.py.230: loss: 0.09926 [I 2022-10-28 03:27:10] ...optimizers/trijoint.py.151: total_norm: 12.738246 [I 2022-10-28 03:27:16] ...trap/engines/engine.py.224: train: epoch 103 | batch 1610/2304 [I 2022-10-28 03:27:16] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:27 [I 2022-10-28 03:27:16] ...trap/engines/engine.py.229: process: 0.56482 | load: 0.00243 [I 2022-10-28 03:27:16] ...trap/engines/engine.py.230: loss: 0.17601 [I 2022-10-28 03:27:16] ...optimizers/trijoint.py.151: total_norm: 16.550159 [I 2022-10-28 03:27:21] ...trap/engines/engine.py.224: train: epoch 103 | batch 1620/2304 [I 2022-10-28 03:27:21] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:21 [I 2022-10-28 03:27:21] ...trap/engines/engine.py.229: process: 0.57369 | load: 0.00245 [I 2022-10-28 03:27:21] ...trap/engines/engine.py.230: loss: 0.16621 [I 2022-10-28 03:27:21] ...optimizers/trijoint.py.151: total_norm: 17.267715 [I 2022-10-28 03:27:27] ...trap/engines/engine.py.224: train: epoch 103 | batch 1630/2304 [I 2022-10-28 03:27:27] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:16 [I 2022-10-28 03:27:27] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00242 [I 2022-10-28 03:27:27] ...trap/engines/engine.py.230: loss: 0.08624 [I 2022-10-28 03:27:27] ...optimizers/trijoint.py.151: total_norm: 14.177935 [I 2022-10-28 03:27:33] ...trap/engines/engine.py.224: train: epoch 103 | batch 1640/2304 [I 2022-10-28 03:27:33] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:08 [I 2022-10-28 03:27:33] ...trap/engines/engine.py.229: process: 0.55835 | load: 0.00245 [I 2022-10-28 03:27:33] ...trap/engines/engine.py.230: loss: 0.13999 [I 2022-10-28 03:27:33] ...optimizers/trijoint.py.151: total_norm: 10.172272 [I 2022-10-28 03:27:38] ...trap/engines/engine.py.224: train: epoch 103 | batch 1650/2304 [I 2022-10-28 03:27:38] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:04 [I 2022-10-28 03:27:38] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00240 [I 2022-10-28 03:27:38] ...trap/engines/engine.py.230: loss: 0.16890 [I 2022-10-28 03:27:38] ...optimizers/trijoint.py.151: total_norm: 26.816843 [I 2022-10-28 03:27:44] ...trap/engines/engine.py.224: train: epoch 103 | batch 1660/2304 [I 2022-10-28 03:27:44] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:01 [I 2022-10-28 03:27:44] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00242 [I 2022-10-28 03:27:44] ...trap/engines/engine.py.230: loss: 0.09436 [I 2022-10-28 03:27:44] ...optimizers/trijoint.py.151: total_norm: 31.415558 [I 2022-10-28 03:27:49] ...trap/engines/engine.py.224: train: epoch 103 | batch 1670/2304 [I 2022-10-28 03:27:49] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:05:50 [I 2022-10-28 03:27:49] ...trap/engines/engine.py.229: process: 0.54555 | load: 0.00239 [I 2022-10-28 03:27:49] ...trap/engines/engine.py.230: loss: 0.16774 [I 2022-10-28 03:27:49] ...optimizers/trijoint.py.151: total_norm: 12.659447 [I 2022-10-28 03:27:55] ...trap/engines/engine.py.224: train: epoch 103 | batch 1680/2304 [I 2022-10-28 03:27:55] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:05:46 [I 2022-10-28 03:27:55] ...trap/engines/engine.py.229: process: 0.56249 | load: 0.00245 [I 2022-10-28 03:27:55] ...trap/engines/engine.py.230: loss: 0.14554 [I 2022-10-28 03:27:55] ...optimizers/trijoint.py.151: total_norm: 16.244772 [I 2022-10-28 03:28:01] ...trap/engines/engine.py.224: train: epoch 103 | batch 1690/2304 [I 2022-10-28 03:28:01] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:05:40 [I 2022-10-28 03:28:01] ...trap/engines/engine.py.229: process: 0.55423 | load: 0.00243 [I 2022-10-28 03:28:01] ...trap/engines/engine.py.230: loss: 0.13693 [I 2022-10-28 03:28:01] ...optimizers/trijoint.py.151: total_norm: 89.933311 [I 2022-10-28 03:28:06] ...trap/engines/engine.py.224: train: epoch 103 | batch 1700/2304 [I 2022-10-28 03:28:06] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:05:34 [I 2022-10-28 03:28:06] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00245 [I 2022-10-28 03:28:06] ...trap/engines/engine.py.230: loss: 0.14857 [I 2022-10-28 03:28:06] ...optimizers/trijoint.py.151: total_norm: 14.619356 [I 2022-10-28 03:28:12] ...trap/engines/engine.py.224: train: epoch 103 | batch 1710/2304 [I 2022-10-28 03:28:12] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:31 [I 2022-10-28 03:28:12] ...trap/engines/engine.py.229: process: 0.56886 | load: 0.00246 [I 2022-10-28 03:28:12] ...trap/engines/engine.py.230: loss: 0.09426 [I 2022-10-28 03:28:12] ...optimizers/trijoint.py.151: total_norm: 10.501065 [I 2022-10-28 03:28:17] ...trap/engines/engine.py.224: train: epoch 103 | batch 1720/2304 [I 2022-10-28 03:28:17] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:28 [I 2022-10-28 03:28:17] ...trap/engines/engine.py.229: process: 0.56143 | load: 0.00242 [I 2022-10-28 03:28:17] ...trap/engines/engine.py.230: loss: 0.11330 [I 2022-10-28 03:28:17] ...optimizers/trijoint.py.151: total_norm: 12.202765 [I 2022-10-28 03:28:23] ...trap/engines/engine.py.224: train: epoch 103 | batch 1730/2304 [I 2022-10-28 03:28:23] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:18 [I 2022-10-28 03:28:23] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00239 [I 2022-10-28 03:28:23] ...trap/engines/engine.py.230: loss: 0.09655 [I 2022-10-28 03:28:23] ...optimizers/trijoint.py.151: total_norm: 17.507706 [I 2022-10-28 03:28:28] ...trap/engines/engine.py.224: train: epoch 103 | batch 1740/2304 [I 2022-10-28 03:28:28] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:12 [I 2022-10-28 03:28:28] ...trap/engines/engine.py.229: process: 0.55353 | load: 0.00241 [I 2022-10-28 03:28:28] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-28 03:28:28] ...optimizers/trijoint.py.151: total_norm: 9.550235 [I 2022-10-28 03:28:34] ...trap/engines/engine.py.224: train: epoch 103 | batch 1750/2304 [I 2022-10-28 03:28:34] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:06 [I 2022-10-28 03:28:34] ...trap/engines/engine.py.229: process: 0.55562 | load: 0.00239 [I 2022-10-28 03:28:34] ...trap/engines/engine.py.230: loss: 0.16688 [I 2022-10-28 03:28:34] ...optimizers/trijoint.py.151: total_norm: 30.469765 [I 2022-10-28 03:28:40] ...trap/engines/engine.py.224: train: epoch 103 | batch 1760/2304 [I 2022-10-28 03:28:40] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:00 [I 2022-10-28 03:28:40] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00242 [I 2022-10-28 03:28:40] ...trap/engines/engine.py.230: loss: 0.12843 [I 2022-10-28 03:28:40] ...optimizers/trijoint.py.151: total_norm: 10.199560 [I 2022-10-28 03:28:45] ...trap/engines/engine.py.224: train: epoch 103 | batch 1770/2304 [I 2022-10-28 03:28:45] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:04:57 [I 2022-10-28 03:28:45] ...trap/engines/engine.py.229: process: 0.55851 | load: 0.00242 [I 2022-10-28 03:28:45] ...trap/engines/engine.py.230: loss: 0.12704 [I 2022-10-28 03:28:45] ...optimizers/trijoint.py.151: total_norm: 20.469419 [I 2022-10-28 03:28:51] ...trap/engines/engine.py.224: train: epoch 103 | batch 1780/2304 [I 2022-10-28 03:28:51] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:04:51 [I 2022-10-28 03:28:51] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00242 [I 2022-10-28 03:28:51] ...trap/engines/engine.py.230: loss: 0.15858 [I 2022-10-28 03:28:51] ...optimizers/trijoint.py.151: total_norm: 47.597111 [I 2022-10-28 03:28:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 1790/2304 [I 2022-10-28 03:28:56] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:04:46 [I 2022-10-28 03:28:56] ...trap/engines/engine.py.229: process: 0.54824 | load: 0.00242 [I 2022-10-28 03:28:56] ...trap/engines/engine.py.230: loss: 0.09827 [I 2022-10-28 03:28:56] ...optimizers/trijoint.py.151: total_norm: 18.916580 [I 2022-10-28 03:29:02] ...trap/engines/engine.py.224: train: epoch 103 | batch 1800/2304 [I 2022-10-28 03:29:02] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:04:39 [I 2022-10-28 03:29:02] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00241 [I 2022-10-28 03:29:02] ...trap/engines/engine.py.230: loss: 0.14759 [I 2022-10-28 03:29:02] ...optimizers/trijoint.py.151: total_norm: 10.926679 [I 2022-10-28 03:29:07] ...trap/engines/engine.py.224: train: epoch 103 | batch 1810/2304 [I 2022-10-28 03:29:07] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:32 [I 2022-10-28 03:29:07] ...trap/engines/engine.py.229: process: 0.54278 | load: 0.00247 [I 2022-10-28 03:29:07] ...trap/engines/engine.py.230: loss: 0.13519 [I 2022-10-28 03:29:07] ...optimizers/trijoint.py.151: total_norm: 16.670397 [I 2022-10-28 03:29:13] ...trap/engines/engine.py.224: train: epoch 103 | batch 1820/2304 [I 2022-10-28 03:29:13] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:29 [I 2022-10-28 03:29:13] ...trap/engines/engine.py.229: process: 0.56446 | load: 0.00240 [I 2022-10-28 03:29:13] ...trap/engines/engine.py.230: loss: 0.12205 [I 2022-10-28 03:29:13] ...optimizers/trijoint.py.151: total_norm: 16.721632 [I 2022-10-28 03:29:19] ...trap/engines/engine.py.224: train: epoch 103 | batch 1830/2304 [I 2022-10-28 03:29:19] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:24 [I 2022-10-28 03:29:19] ...trap/engines/engine.py.229: process: 0.54417 | load: 0.00260 [I 2022-10-28 03:29:19] ...trap/engines/engine.py.230: loss: 0.18427 [I 2022-10-28 03:29:19] ...optimizers/trijoint.py.151: total_norm: 25.381027 [I 2022-10-28 03:29:24] ...trap/engines/engine.py.224: train: epoch 103 | batch 1840/2304 [I 2022-10-28 03:29:24] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:17 [I 2022-10-28 03:29:24] ...trap/engines/engine.py.229: process: 0.56787 | load: 0.00243 [I 2022-10-28 03:29:24] ...trap/engines/engine.py.230: loss: 0.10245 [I 2022-10-28 03:29:24] ...optimizers/trijoint.py.151: total_norm: 8.422441 [I 2022-10-28 03:29:30] ...trap/engines/engine.py.224: train: epoch 103 | batch 1850/2304 [I 2022-10-28 03:29:30] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:11 [I 2022-10-28 03:29:30] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00243 [I 2022-10-28 03:29:30] ...trap/engines/engine.py.230: loss: 0.11520 [I 2022-10-28 03:29:30] ...optimizers/trijoint.py.151: total_norm: 25.626644 [I 2022-10-28 03:29:35] ...trap/engines/engine.py.224: train: epoch 103 | batch 1860/2304 [I 2022-10-28 03:29:35] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:07 [I 2022-10-28 03:29:35] ...trap/engines/engine.py.229: process: 0.56496 | load: 0.00241 [I 2022-10-28 03:29:35] ...trap/engines/engine.py.230: loss: 0.22301 [I 2022-10-28 03:29:35] ...optimizers/trijoint.py.151: total_norm: 30.297644 [I 2022-10-28 03:29:41] ...trap/engines/engine.py.224: train: epoch 103 | batch 1870/2304 [I 2022-10-28 03:29:41] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:00 [I 2022-10-28 03:29:41] ...trap/engines/engine.py.229: process: 0.54971 | load: 0.00240 [I 2022-10-28 03:29:41] ...trap/engines/engine.py.230: loss: 0.13414 [I 2022-10-28 03:29:41] ...optimizers/trijoint.py.151: total_norm: 22.453203 [I 2022-10-28 03:29:46] ...trap/engines/engine.py.224: train: epoch 103 | batch 1880/2304 [I 2022-10-28 03:29:46] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:03:56 [I 2022-10-28 03:29:46] ...trap/engines/engine.py.229: process: 0.56303 | load: 0.00243 [I 2022-10-28 03:29:46] ...trap/engines/engine.py.230: loss: 0.11047 [I 2022-10-28 03:29:46] ...optimizers/trijoint.py.151: total_norm: 9.300833 [I 2022-10-28 03:29:52] ...trap/engines/engine.py.224: train: epoch 103 | batch 1890/2304 [I 2022-10-28 03:29:52] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:03:50 [I 2022-10-28 03:29:52] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00242 [I 2022-10-28 03:29:52] ...trap/engines/engine.py.230: loss: 0.13483 [I 2022-10-28 03:29:52] ...optimizers/trijoint.py.151: total_norm: 26.898500 [I 2022-10-28 03:29:58] ...trap/engines/engine.py.224: train: epoch 103 | batch 1900/2304 [I 2022-10-28 03:29:58] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:44 [I 2022-10-28 03:29:58] ...trap/engines/engine.py.229: process: 0.56545 | load: 0.00242 [I 2022-10-28 03:29:58] ...trap/engines/engine.py.230: loss: 0.12217 [I 2022-10-28 03:29:58] ...optimizers/trijoint.py.151: total_norm: 20.010817 [I 2022-10-28 03:30:03] ...trap/engines/engine.py.224: train: epoch 103 | batch 1910/2304 [I 2022-10-28 03:30:03] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:03:38 [I 2022-10-28 03:30:03] ...trap/engines/engine.py.229: process: 0.55202 | load: 0.00241 [I 2022-10-28 03:30:03] ...trap/engines/engine.py.230: loss: 0.12596 [I 2022-10-28 03:30:03] ...optimizers/trijoint.py.151: total_norm: 12.659240 [I 2022-10-28 03:30:09] ...trap/engines/engine.py.224: train: epoch 103 | batch 1920/2304 [I 2022-10-28 03:30:09] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:31 [I 2022-10-28 03:30:09] ...trap/engines/engine.py.229: process: 0.55289 | load: 0.00250 [I 2022-10-28 03:30:09] ...trap/engines/engine.py.230: loss: 0.11015 [I 2022-10-28 03:30:09] ...optimizers/trijoint.py.151: total_norm: 7.203832 [I 2022-10-28 03:30:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 1930/2304 [I 2022-10-28 03:30:14] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:26 [I 2022-10-28 03:30:14] ...trap/engines/engine.py.229: process: 0.55154 | load: 0.00246 [I 2022-10-28 03:30:14] ...trap/engines/engine.py.230: loss: 0.13595 [I 2022-10-28 03:30:14] ...optimizers/trijoint.py.151: total_norm: 21.037228 [I 2022-10-28 03:30:20] ...trap/engines/engine.py.224: train: epoch 103 | batch 1940/2304 [I 2022-10-28 03:30:20] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:20 [I 2022-10-28 03:30:20] ...trap/engines/engine.py.229: process: 0.55722 | load: 0.00244 [I 2022-10-28 03:30:20] ...trap/engines/engine.py.230: loss: 0.11868 [I 2022-10-28 03:30:20] ...optimizers/trijoint.py.151: total_norm: 22.034288 [I 2022-10-28 03:30:26] ...trap/engines/engine.py.224: train: epoch 103 | batch 1950/2304 [I 2022-10-28 03:30:26] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:18 [I 2022-10-28 03:30:26] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00247 [I 2022-10-28 03:30:26] ...trap/engines/engine.py.230: loss: 0.12243 [I 2022-10-28 03:30:26] ...optimizers/trijoint.py.151: total_norm: 11.114462 [I 2022-10-28 03:30:31] ...trap/engines/engine.py.224: train: epoch 103 | batch 1960/2304 [I 2022-10-28 03:30:31] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:09 [I 2022-10-28 03:30:31] ...trap/engines/engine.py.229: process: 0.54450 | load: 0.00245 [I 2022-10-28 03:30:31] ...trap/engines/engine.py.230: loss: 0.13006 [I 2022-10-28 03:30:31] ...optimizers/trijoint.py.151: total_norm: 20.565565 [I 2022-10-28 03:30:57] ...trap/engines/engine.py.224: train: epoch 103 | batch 1970/2304 [I 2022-10-28 03:30:57] ...trap/engines/engine.py.228: elapsed: 0:18:41 | left: 0:26:08 [I 2022-10-28 03:30:57] ...trap/engines/engine.py.229: process: 21.25464 | load: 0.00241 [I 2022-10-28 03:30:57] ...trap/engines/engine.py.230: loss: 0.08638 [I 2022-10-28 03:30:57] ...optimizers/trijoint.py.151: total_norm: 12.393454 [I 2022-10-28 03:31:03] ...trap/engines/engine.py.224: train: epoch 103 | batch 1980/2304 [I 2022-10-28 03:31:03] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:05:24 [I 2022-10-28 03:31:03] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00259 [I 2022-10-28 03:31:03] ...trap/engines/engine.py.230: loss: 0.11605 [I 2022-10-28 03:31:03] ...optimizers/trijoint.py.151: total_norm: 23.102135 [I 2022-10-28 03:31:08] ...trap/engines/engine.py.224: train: epoch 103 | batch 1990/2304 [I 2022-10-28 03:31:08] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:03:09 [I 2022-10-28 03:31:08] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00241 [I 2022-10-28 03:31:08] ...trap/engines/engine.py.230: loss: 0.10839 [I 2022-10-28 03:31:08] ...optimizers/trijoint.py.151: total_norm: 16.128618 [I 2022-10-28 03:31:14] ...trap/engines/engine.py.224: train: epoch 103 | batch 2000/2304 [I 2022-10-28 03:31:14] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:49 [I 2022-10-28 03:31:14] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00241 [I 2022-10-28 03:31:14] ...trap/engines/engine.py.230: loss: 0.08315 [I 2022-10-28 03:31:14] ...optimizers/trijoint.py.151: total_norm: 22.724642 [I 2022-10-28 03:31:20] ...trap/engines/engine.py.224: train: epoch 103 | batch 2010/2304 [I 2022-10-28 03:31:20] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:44 [I 2022-10-28 03:31:20] ...trap/engines/engine.py.229: process: 0.55932 | load: 0.00241 [I 2022-10-28 03:31:20] ...trap/engines/engine.py.230: loss: 0.17171 [I 2022-10-28 03:31:20] ...optimizers/trijoint.py.151: total_norm: 19.461824 [I 2022-10-28 03:31:25] ...trap/engines/engine.py.224: train: epoch 103 | batch 2020/2304 [I 2022-10-28 03:31:25] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:40 [I 2022-10-28 03:31:25] ...trap/engines/engine.py.229: process: 0.62307 | load: 0.00241 [I 2022-10-28 03:31:25] ...trap/engines/engine.py.230: loss: 0.10878 [I 2022-10-28 03:31:25] ...optimizers/trijoint.py.151: total_norm: 16.912724 [I 2022-10-28 03:31:31] ...trap/engines/engine.py.224: train: epoch 103 | batch 2030/2304 [I 2022-10-28 03:31:31] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:31 [I 2022-10-28 03:31:31] ...trap/engines/engine.py.229: process: 0.54606 | load: 0.00241 [I 2022-10-28 03:31:31] ...trap/engines/engine.py.230: loss: 0.10768 [I 2022-10-28 03:31:31] ...optimizers/trijoint.py.151: total_norm: 21.800755 [I 2022-10-28 03:31:36] ...trap/engines/engine.py.224: train: epoch 103 | batch 2040/2304 [I 2022-10-28 03:31:36] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:27 [I 2022-10-28 03:31:36] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00245 [I 2022-10-28 03:31:36] ...trap/engines/engine.py.230: loss: 0.10743 [I 2022-10-28 03:31:36] ...optimizers/trijoint.py.151: total_norm: 14.892710 [I 2022-10-28 03:31:42] ...trap/engines/engine.py.224: train: epoch 103 | batch 2050/2304 [I 2022-10-28 03:31:42] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:21 [I 2022-10-28 03:31:42] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00244 [I 2022-10-28 03:31:42] ...trap/engines/engine.py.230: loss: 0.17023 [I 2022-10-28 03:31:42] ...optimizers/trijoint.py.151: total_norm: 33.410339 [I 2022-10-28 03:31:47] ...trap/engines/engine.py.224: train: epoch 103 | batch 2060/2304 [I 2022-10-28 03:31:47] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:14 [I 2022-10-28 03:31:47] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00249 [I 2022-10-28 03:31:47] ...trap/engines/engine.py.230: loss: 0.11984 [I 2022-10-28 03:31:47] ...optimizers/trijoint.py.151: total_norm: 11.457318 [I 2022-10-28 03:31:53] ...trap/engines/engine.py.224: train: epoch 103 | batch 2070/2304 [I 2022-10-28 03:31:53] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:10 [I 2022-10-28 03:31:53] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00244 [I 2022-10-28 03:31:53] ...trap/engines/engine.py.230: loss: 0.15420 [I 2022-10-28 03:31:53] ...optimizers/trijoint.py.151: total_norm: 9.628255 [I 2022-10-28 03:31:59] ...trap/engines/engine.py.224: train: epoch 103 | batch 2080/2304 [I 2022-10-28 03:31:59] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:05 [I 2022-10-28 03:31:59] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00242 [I 2022-10-28 03:31:59] ...trap/engines/engine.py.230: loss: 0.17357 [I 2022-10-28 03:31:59] ...optimizers/trijoint.py.151: total_norm: 19.942291 [I 2022-10-28 03:32:04] ...trap/engines/engine.py.224: train: epoch 103 | batch 2090/2304 [I 2022-10-28 03:32:04] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:02:00 [I 2022-10-28 03:32:04] ...trap/engines/engine.py.229: process: 0.56378 | load: 0.00248 [I 2022-10-28 03:32:04] ...trap/engines/engine.py.230: loss: 0.14418 [I 2022-10-28 03:32:04] ...optimizers/trijoint.py.151: total_norm: 21.433590 [I 2022-10-28 03:32:10] ...trap/engines/engine.py.224: train: epoch 103 | batch 2100/2304 [I 2022-10-28 03:32:10] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:53 [I 2022-10-28 03:32:10] ...trap/engines/engine.py.229: process: 0.55558 | load: 0.00243 [I 2022-10-28 03:32:10] ...trap/engines/engine.py.230: loss: 0.16058 [I 2022-10-28 03:32:10] ...optimizers/trijoint.py.151: total_norm: 24.309837 [I 2022-10-28 03:32:15] ...trap/engines/engine.py.224: train: epoch 103 | batch 2110/2304 [I 2022-10-28 03:32:15] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:47 [I 2022-10-28 03:32:15] ...trap/engines/engine.py.229: process: 0.56740 | load: 0.00244 [I 2022-10-28 03:32:15] ...trap/engines/engine.py.230: loss: 0.10635 [I 2022-10-28 03:32:15] ...optimizers/trijoint.py.151: total_norm: 10.935388 [I 2022-10-28 03:32:21] ...trap/engines/engine.py.224: train: epoch 103 | batch 2120/2304 [I 2022-10-28 03:32:21] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:42 [I 2022-10-28 03:32:21] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00246 [I 2022-10-28 03:32:21] ...trap/engines/engine.py.230: loss: 0.11071 [I 2022-10-28 03:32:21] ...optimizers/trijoint.py.151: total_norm: 11.798714 [I 2022-10-28 03:32:27] ...trap/engines/engine.py.224: train: epoch 103 | batch 2130/2304 [I 2022-10-28 03:32:27] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:36 [I 2022-10-28 03:32:27] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00244 [I 2022-10-28 03:32:27] ...trap/engines/engine.py.230: loss: 0.11390 [I 2022-10-28 03:32:27] ...optimizers/trijoint.py.151: total_norm: 15.868960 [I 2022-10-28 03:32:32] ...trap/engines/engine.py.224: train: epoch 103 | batch 2140/2304 [I 2022-10-28 03:32:32] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:31 [I 2022-10-28 03:32:32] ...trap/engines/engine.py.229: process: 0.55083 | load: 0.00241 [I 2022-10-28 03:32:32] ...trap/engines/engine.py.230: loss: 0.15107 [I 2022-10-28 03:32:32] ...optimizers/trijoint.py.151: total_norm: 30.509256 [I 2022-10-28 03:32:38] ...trap/engines/engine.py.224: train: epoch 103 | batch 2150/2304 [I 2022-10-28 03:32:38] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:25 [I 2022-10-28 03:32:38] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00243 [I 2022-10-28 03:32:38] ...trap/engines/engine.py.230: loss: 0.12833 [I 2022-10-28 03:32:38] ...optimizers/trijoint.py.151: total_norm: 53.305927 [I 2022-10-28 03:32:43] ...trap/engines/engine.py.224: train: epoch 103 | batch 2160/2304 [I 2022-10-28 03:32:43] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:19 [I 2022-10-28 03:32:43] ...trap/engines/engine.py.229: process: 0.55067 | load: 0.00243 [I 2022-10-28 03:32:43] ...trap/engines/engine.py.230: loss: 0.12641 [I 2022-10-28 03:32:43] ...optimizers/trijoint.py.151: total_norm: 34.014404 [I 2022-10-28 03:32:49] ...trap/engines/engine.py.224: train: epoch 103 | batch 2170/2304 [I 2022-10-28 03:32:49] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:14 [I 2022-10-28 03:32:49] ...trap/engines/engine.py.229: process: 0.56763 | load: 0.00243 [I 2022-10-28 03:32:49] ...trap/engines/engine.py.230: loss: 0.08720 [I 2022-10-28 03:32:49] ...optimizers/trijoint.py.151: total_norm: 10.262646 [I 2022-10-28 03:32:55] ...trap/engines/engine.py.224: train: epoch 103 | batch 2180/2304 [I 2022-10-28 03:32:55] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:09 [I 2022-10-28 03:32:55] ...trap/engines/engine.py.229: process: 0.56042 | load: 0.00243 [I 2022-10-28 03:32:55] ...trap/engines/engine.py.230: loss: 0.09086 [I 2022-10-28 03:32:55] ...optimizers/trijoint.py.151: total_norm: 15.851252 [I 2022-10-28 03:33:00] ...trap/engines/engine.py.224: train: epoch 103 | batch 2190/2304 [I 2022-10-28 03:33:00] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:03 [I 2022-10-28 03:33:00] ...trap/engines/engine.py.229: process: 0.56374 | load: 0.00241 [I 2022-10-28 03:33:00] ...trap/engines/engine.py.230: loss: 0.08290 [I 2022-10-28 03:33:00] ...optimizers/trijoint.py.151: total_norm: 21.711952 [I 2022-10-28 03:33:06] ...trap/engines/engine.py.224: train: epoch 103 | batch 2200/2304 [I 2022-10-28 03:33:06] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:58 [I 2022-10-28 03:33:06] ...trap/engines/engine.py.229: process: 0.55315 | load: 0.00242 [I 2022-10-28 03:33:06] ...trap/engines/engine.py.230: loss: 0.12971 [I 2022-10-28 03:33:06] ...optimizers/trijoint.py.151: total_norm: 17.914368 [I 2022-10-28 03:33:11] ...trap/engines/engine.py.224: train: epoch 103 | batch 2210/2304 [I 2022-10-28 03:33:11] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:52 [I 2022-10-28 03:33:11] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00244 [I 2022-10-28 03:33:11] ...trap/engines/engine.py.230: loss: 0.10954 [I 2022-10-28 03:33:11] ...optimizers/trijoint.py.151: total_norm: 10.913468 [I 2022-10-28 03:33:17] ...trap/engines/engine.py.224: train: epoch 103 | batch 2220/2304 [I 2022-10-28 03:33:17] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:47 [I 2022-10-28 03:33:17] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00245 [I 2022-10-28 03:33:17] ...trap/engines/engine.py.230: loss: 0.07299 [I 2022-10-28 03:33:17] ...optimizers/trijoint.py.151: total_norm: 8.929450 [I 2022-10-28 03:33:23] ...trap/engines/engine.py.224: train: epoch 103 | batch 2230/2304 [I 2022-10-28 03:33:23] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:41 [I 2022-10-28 03:33:23] ...trap/engines/engine.py.229: process: 0.56878 | load: 0.00242 [I 2022-10-28 03:33:23] ...trap/engines/engine.py.230: loss: 0.11929 [I 2022-10-28 03:33:23] ...optimizers/trijoint.py.151: total_norm: 14.192070 [I 2022-10-28 03:33:28] ...trap/engines/engine.py.224: train: epoch 103 | batch 2240/2304 [I 2022-10-28 03:33:28] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:35 [I 2022-10-28 03:33:28] ...trap/engines/engine.py.229: process: 0.55618 | load: 0.00241 [I 2022-10-28 03:33:28] ...trap/engines/engine.py.230: loss: 0.09356 [I 2022-10-28 03:33:28] ...optimizers/trijoint.py.151: total_norm: 11.503952 [I 2022-10-28 03:33:34] ...trap/engines/engine.py.224: train: epoch 103 | batch 2250/2304 [I 2022-10-28 03:33:34] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:29 [I 2022-10-28 03:33:34] ...trap/engines/engine.py.229: process: 0.55278 | load: 0.00242 [I 2022-10-28 03:33:34] ...trap/engines/engine.py.230: loss: 0.11397 [I 2022-10-28 03:33:34] ...optimizers/trijoint.py.151: total_norm: 34.335655 [I 2022-10-28 03:33:39] ...trap/engines/engine.py.224: train: epoch 103 | batch 2260/2304 [I 2022-10-28 03:33:39] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:24 [I 2022-10-28 03:33:39] ...trap/engines/engine.py.229: process: 0.55639 | load: 0.00245 [I 2022-10-28 03:33:39] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-28 03:33:39] ...optimizers/trijoint.py.151: total_norm: 20.809145 [I 2022-10-28 03:33:45] ...trap/engines/engine.py.224: train: epoch 103 | batch 2270/2304 [I 2022-10-28 03:33:45] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:19 [I 2022-10-28 03:33:45] ...trap/engines/engine.py.229: process: 0.56305 | load: 0.00244 [I 2022-10-28 03:33:45] ...trap/engines/engine.py.230: loss: 0.11155 [I 2022-10-28 03:33:45] ...optimizers/trijoint.py.151: total_norm: 22.283447 [I 2022-10-28 03:33:50] ...trap/engines/engine.py.224: train: epoch 103 | batch 2280/2304 [I 2022-10-28 03:33:50] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:13 [I 2022-10-28 03:33:50] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00288 [I 2022-10-28 03:33:50] ...trap/engines/engine.py.230: loss: 0.13966 [I 2022-10-28 03:33:50] ...optimizers/trijoint.py.151: total_norm: 18.222612 [I 2022-10-28 03:33:56] ...trap/engines/engine.py.224: train: epoch 103 | batch 2290/2304 [I 2022-10-28 03:33:56] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:07 [I 2022-10-28 03:33:56] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00242 [I 2022-10-28 03:33:56] ...trap/engines/engine.py.230: loss: 0.13424 [I 2022-10-28 03:33:56] ...optimizers/trijoint.py.151: total_norm: 11.297720 [I 2022-10-28 03:34:02] ...trap/engines/engine.py.224: train: epoch 103 | batch 2300/2304 [I 2022-10-28 03:34:02] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:02 [I 2022-10-28 03:34:02] ...trap/engines/engine.py.229: process: 0.53946 | load: 0.00015 [I 2022-10-28 03:34:02] ...trap/engines/engine.py.230: loss: 0.08474 [I 2022-10-28 03:34:02] ...optimizers/trijoint.py.151: total_norm: 7.341309 [I 2022-10-28 03:34:04] ...trap/engines/engine.py.224: train: epoch 103 | batch 2304/2304 [I 2022-10-28 03:34:04] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:00 [I 2022-10-28 03:34:04] ...trap/engines/engine.py.229: process: 0.56078 | load: 0.00015 [I 2022-10-28 03:34:04] ...trap/engines/engine.py.230: loss: 0.14337 [I 2022-10-28 03:34:04] ...optimizers/trijoint.py.151: total_norm: 17.301943 [S 2022-10-28 03:34:05] ...trap/engines/engine.py.129: train_epoch.epoch: 103 [S 2022-10-28 03:34:05] ...trap/engines/engine.py.129: train_epoch.loss: 0.12472395220379509 [I 2022-10-28 03:34:05] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 03:34:07] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 103 [I 2022-10-28 03:34:07] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 03:34:09] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 03:34:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 03:34:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 03:34:12] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 03:34:12] ...trap/engines/engine.py.321: eval: epoch 103 | batch 0/510 [I 2022-10-28 03:34:12] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:49:10 [I 2022-10-28 03:34:12] ...trap/engines/engine.py.326: process: 5.78612 | load: 3.12241 [I 2022-10-28 03:34:16] ...trap/engines/engine.py.321: eval: epoch 103 | batch 10/510 [I 2022-10-28 03:34:16] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:37 [I 2022-10-28 03:34:16] ...trap/engines/engine.py.326: process: 0.32287 | load: 0.00018 [I 2022-10-28 03:34:19] ...trap/engines/engine.py.321: eval: epoch 103 | batch 20/510 [I 2022-10-28 03:34:19] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:12 [I 2022-10-28 03:34:19] ...trap/engines/engine.py.326: process: 0.32891 | load: 0.00018 [I 2022-10-28 03:34:22] ...trap/engines/engine.py.321: eval: epoch 103 | batch 30/510 [I 2022-10-28 03:34:22] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:41 [I 2022-10-28 03:34:22] ...trap/engines/engine.py.326: process: 0.32956 | load: 0.00018 [I 2022-10-28 03:34:26] ...trap/engines/engine.py.321: eval: epoch 103 | batch 40/510 [I 2022-10-28 03:34:26] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-28 03:34:26] ...trap/engines/engine.py.326: process: 0.33714 | load: 0.00016 [I 2022-10-28 03:34:29] ...trap/engines/engine.py.321: eval: epoch 103 | batch 50/510 [I 2022-10-28 03:34:29] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:32 [I 2022-10-28 03:34:29] ...trap/engines/engine.py.326: process: 0.32947 | load: 0.00018 [I 2022-10-28 03:34:32] ...trap/engines/engine.py.321: eval: epoch 103 | batch 60/510 [I 2022-10-28 03:34:32] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:29 [I 2022-10-28 03:34:32] ...trap/engines/engine.py.326: process: 0.32081 | load: 0.00017 [I 2022-10-28 03:34:36] ...trap/engines/engine.py.321: eval: epoch 103 | batch 70/510 [I 2022-10-28 03:34:36] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:25 [I 2022-10-28 03:34:36] ...trap/engines/engine.py.326: process: 0.32462 | load: 0.00019 [I 2022-10-28 03:34:39] ...trap/engines/engine.py.321: eval: epoch 103 | batch 80/510 [I 2022-10-28 03:34:39] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:22 [I 2022-10-28 03:34:39] ...trap/engines/engine.py.326: process: 0.33125 | load: 0.00019 [I 2022-10-28 03:34:42] ...trap/engines/engine.py.321: eval: epoch 103 | batch 90/510 [I 2022-10-28 03:34:42] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:22 [I 2022-10-28 03:34:42] ...trap/engines/engine.py.326: process: 0.35128 | load: 0.00018 [I 2022-10-28 03:34:46] ...trap/engines/engine.py.321: eval: epoch 103 | batch 100/510 [I 2022-10-28 03:34:46] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:14 [I 2022-10-28 03:34:46] ...trap/engines/engine.py.326: process: 0.28521 | load: 0.00020 [I 2022-10-28 03:34:49] ...trap/engines/engine.py.321: eval: epoch 103 | batch 110/510 [I 2022-10-28 03:34:49] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:57 [I 2022-10-28 03:34:49] ...trap/engines/engine.py.326: process: 0.28821 | load: 0.00020 [I 2022-10-28 03:34:52] ...trap/engines/engine.py.321: eval: epoch 103 | batch 120/510 [I 2022-10-28 03:34:52] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:51 [I 2022-10-28 03:34:52] ...trap/engines/engine.py.326: process: 0.28353 | load: 0.00015 [I 2022-10-28 03:34:54] ...trap/engines/engine.py.321: eval: epoch 103 | batch 130/510 [I 2022-10-28 03:34:54] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-28 03:34:54] ...trap/engines/engine.py.326: process: 0.28824 | load: 0.00016 [I 2022-10-28 03:34:57] ...trap/engines/engine.py.321: eval: epoch 103 | batch 140/510 [I 2022-10-28 03:34:57] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-28 03:34:57] ...trap/engines/engine.py.326: process: 0.29070 | load: 0.00014 [I 2022-10-28 03:35:00] ...trap/engines/engine.py.321: eval: epoch 103 | batch 150/510 [I 2022-10-28 03:35:00] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:45 [I 2022-10-28 03:35:00] ...trap/engines/engine.py.326: process: 0.30376 | load: 0.00013 [I 2022-10-28 03:35:03] ...trap/engines/engine.py.321: eval: epoch 103 | batch 160/510 [I 2022-10-28 03:35:03] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-28 03:35:03] ...trap/engines/engine.py.326: process: 0.28827 | load: 0.00015 [I 2022-10-28 03:35:06] ...trap/engines/engine.py.321: eval: epoch 103 | batch 170/510 [I 2022-10-28 03:35:06] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:37 [I 2022-10-28 03:35:06] ...trap/engines/engine.py.326: process: 0.29049 | load: 0.00015 [I 2022-10-28 03:35:09] ...trap/engines/engine.py.321: eval: epoch 103 | batch 180/510 [I 2022-10-28 03:35:09] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-28 03:35:09] ...trap/engines/engine.py.326: process: 0.28706 | load: 0.00015 [I 2022-10-28 03:35:12] ...trap/engines/engine.py.321: eval: epoch 103 | batch 190/510 [I 2022-10-28 03:35:12] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-28 03:35:12] ...trap/engines/engine.py.326: process: 0.28381 | load: 0.00015 [I 2022-10-28 03:35:15] ...trap/engines/engine.py.321: eval: epoch 103 | batch 200/510 [I 2022-10-28 03:35:15] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-28 03:35:15] ...trap/engines/engine.py.326: process: 0.28998 | load: 0.00014 [I 2022-10-28 03:35:17] ...trap/engines/engine.py.321: eval: epoch 103 | batch 210/510 [I 2022-10-28 03:35:17] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:25 [I 2022-10-28 03:35:17] ...trap/engines/engine.py.326: process: 0.29034 | load: 0.00015 [I 2022-10-28 03:35:20] ...trap/engines/engine.py.321: eval: epoch 103 | batch 220/510 [I 2022-10-28 03:35:20] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-28 03:35:20] ...trap/engines/engine.py.326: process: 0.29173 | load: 0.00014 [I 2022-10-28 03:35:23] ...trap/engines/engine.py.321: eval: epoch 103 | batch 230/510 [I 2022-10-28 03:35:23] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-28 03:35:23] ...trap/engines/engine.py.326: process: 0.28591 | load: 0.00015 [I 2022-10-28 03:35:26] ...trap/engines/engine.py.321: eval: epoch 103 | batch 240/510 [I 2022-10-28 03:35:26] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 03:35:26] ...trap/engines/engine.py.326: process: 0.28170 | load: 0.00015 [I 2022-10-28 03:35:29] ...trap/engines/engine.py.321: eval: epoch 103 | batch 250/510 [I 2022-10-28 03:35:29] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-28 03:35:29] ...trap/engines/engine.py.326: process: 0.29322 | load: 0.00016 [I 2022-10-28 03:35:32] ...trap/engines/engine.py.321: eval: epoch 103 | batch 260/510 [I 2022-10-28 03:35:32] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 03:35:32] ...trap/engines/engine.py.326: process: 0.29050 | load: 0.00017 [I 2022-10-28 03:35:35] ...trap/engines/engine.py.321: eval: epoch 103 | batch 270/510 [I 2022-10-28 03:35:35] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-28 03:35:35] ...trap/engines/engine.py.326: process: 0.28347 | load: 0.00015 [I 2022-10-28 03:35:38] ...trap/engines/engine.py.321: eval: epoch 103 | batch 280/510 [I 2022-10-28 03:35:38] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-28 03:35:38] ...trap/engines/engine.py.326: process: 0.29087 | load: 0.00016 [I 2022-10-28 03:35:40] ...trap/engines/engine.py.321: eval: epoch 103 | batch 290/510 [I 2022-10-28 03:35:40] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:02 [I 2022-10-28 03:35:40] ...trap/engines/engine.py.326: process: 0.28468 | load: 0.00017 [I 2022-10-28 03:35:43] ...trap/engines/engine.py.321: eval: epoch 103 | batch 300/510 [I 2022-10-28 03:35:43] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-28 03:35:43] ...trap/engines/engine.py.326: process: 0.28796 | load: 0.00020 [I 2022-10-28 03:35:46] ...trap/engines/engine.py.321: eval: epoch 103 | batch 310/510 [I 2022-10-28 03:35:46] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 03:35:46] ...trap/engines/engine.py.326: process: 0.28891 | load: 0.00021 [I 2022-10-28 03:35:49] ...trap/engines/engine.py.321: eval: epoch 103 | batch 320/510 [I 2022-10-28 03:35:49] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 03:35:49] ...trap/engines/engine.py.326: process: 0.29117 | load: 0.00020 [I 2022-10-28 03:35:52] ...trap/engines/engine.py.321: eval: epoch 103 | batch 330/510 [I 2022-10-28 03:35:52] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-28 03:35:52] ...trap/engines/engine.py.326: process: 0.28636 | load: 0.00016 [I 2022-10-28 03:35:55] ...trap/engines/engine.py.321: eval: epoch 103 | batch 340/510 [I 2022-10-28 03:35:55] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-28 03:35:55] ...trap/engines/engine.py.326: process: 0.28346 | load: 0.00015 [I 2022-10-28 03:35:58] ...trap/engines/engine.py.321: eval: epoch 103 | batch 350/510 [I 2022-10-28 03:35:58] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:45 [I 2022-10-28 03:35:58] ...trap/engines/engine.py.326: process: 0.28676 | load: 0.00021 [I 2022-10-28 03:36:00] ...trap/engines/engine.py.321: eval: epoch 103 | batch 360/510 [I 2022-10-28 03:36:00] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:42 [I 2022-10-28 03:36:00] ...trap/engines/engine.py.326: process: 0.28627 | load: 0.00015 [I 2022-10-28 03:36:03] ...trap/engines/engine.py.321: eval: epoch 103 | batch 370/510 [I 2022-10-28 03:36:03] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:39 [I 2022-10-28 03:36:03] ...trap/engines/engine.py.326: process: 0.28715 | load: 0.00015 [I 2022-10-28 03:36:06] ...trap/engines/engine.py.321: eval: epoch 103 | batch 380/510 [I 2022-10-28 03:36:06] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-28 03:36:06] ...trap/engines/engine.py.326: process: 0.28240 | load: 0.00021 [I 2022-10-28 03:36:09] ...trap/engines/engine.py.321: eval: epoch 103 | batch 390/510 [I 2022-10-28 03:36:09] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 03:36:09] ...trap/engines/engine.py.326: process: 0.27954 | load: 0.00015 [I 2022-10-28 03:36:12] ...trap/engines/engine.py.321: eval: epoch 103 | batch 400/510 [I 2022-10-28 03:36:12] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 03:36:12] ...trap/engines/engine.py.326: process: 0.28712 | load: 0.00015 [I 2022-10-28 03:36:15] ...trap/engines/engine.py.321: eval: epoch 103 | batch 410/510 [I 2022-10-28 03:36:15] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 03:36:15] ...trap/engines/engine.py.326: process: 0.29146 | load: 0.00015 [I 2022-10-28 03:36:18] ...trap/engines/engine.py.321: eval: epoch 103 | batch 420/510 [I 2022-10-28 03:36:18] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:25 [I 2022-10-28 03:36:18] ...trap/engines/engine.py.326: process: 0.28511 | load: 0.00015 [I 2022-10-28 03:36:20] ...trap/engines/engine.py.321: eval: epoch 103 | batch 430/510 [I 2022-10-28 03:36:20] ...trap/engines/engine.py.325: elapsed: 0:02:13 | left: 0:00:22 [I 2022-10-28 03:36:20] ...trap/engines/engine.py.326: process: 0.27944 | load: 0.00015 [I 2022-10-28 03:36:23] ...trap/engines/engine.py.321: eval: epoch 103 | batch 440/510 [I 2022-10-28 03:36:23] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-28 03:36:23] ...trap/engines/engine.py.326: process: 0.28976 | load: 0.00018 [I 2022-10-28 03:36:26] ...trap/engines/engine.py.321: eval: epoch 103 | batch 450/510 [I 2022-10-28 03:36:26] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-28 03:36:26] ...trap/engines/engine.py.326: process: 0.28488 | load: 0.00018 [I 2022-10-28 03:36:29] ...trap/engines/engine.py.321: eval: epoch 103 | batch 460/510 [I 2022-10-28 03:36:29] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-28 03:36:29] ...trap/engines/engine.py.326: process: 0.27819 | load: 0.00017 [I 2022-10-28 03:36:32] ...trap/engines/engine.py.321: eval: epoch 103 | batch 470/510 [I 2022-10-28 03:36:32] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-28 03:36:32] ...trap/engines/engine.py.326: process: 0.28912 | load: 0.00016 [I 2022-10-28 03:36:35] ...trap/engines/engine.py.321: eval: epoch 103 | batch 480/510 [I 2022-10-28 03:36:35] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 03:36:35] ...trap/engines/engine.py.326: process: 0.29080 | load: 0.00017 [I 2022-10-28 03:36:38] ...trap/engines/engine.py.321: eval: epoch 103 | batch 490/510 [I 2022-10-28 03:36:38] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 03:36:38] ...trap/engines/engine.py.326: process: 0.28847 | load: 0.00015 [I 2022-10-28 03:36:41] ...trap/engines/engine.py.321: eval: epoch 103 | batch 500/510 [I 2022-10-28 03:36:41] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:02 [I 2022-10-28 03:36:41] ...trap/engines/engine.py.326: process: 0.28798 | load: 0.00015 [I 2022-10-28 03:36:43] ...trap/engines/engine.py.321: eval: epoch 103 | batch 510/510 [I 2022-10-28 03:36:43] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:00 [I 2022-10-28 03:36:43] ...trap/engines/engine.py.326: process: 0.28438 | load: 0.00011 [S 2022-10-28 03:36:44] ...trap/engines/engine.py.132: eval_epoch.epoch: 103 [I 2022-10-28 03:36:44] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.736600 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909600 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.938100 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.735700 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.908500 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935500 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.019268 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009708 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008573 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.020509 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010385 [S 2022-10-28 03:36:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010548 [I 2022-10-28 03:36:50] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 03:36:50] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 03:36:51] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 03:36:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 03:36:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 03:36:56] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 03:36:56] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 03:37:02] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 03:37:02] ...trap/engines/engine.py.163: Training model on trainset for epoch 104 [I 2022-10-28 03:37:02] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 03:37:05] ...trap/engines/engine.py.224: train: epoch 104 | batch 0/2304 [I 2022-10-28 03:37:05] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:56 [I 2022-10-28 03:37:05] ...trap/engines/engine.py.229: process: 2.70691 | load: 2.05005 [I 2022-10-28 03:37:05] ...trap/engines/engine.py.230: loss: 0.15422 [I 2022-10-28 03:37:05] ...optimizers/trijoint.py.151: total_norm: 22.392929 [I 2022-10-28 03:37:11] ...trap/engines/engine.py.224: train: epoch 104 | batch 10/2304 [I 2022-10-28 03:37:11] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:07 [I 2022-10-28 03:37:11] ...trap/engines/engine.py.229: process: 0.53412 | load: 0.00333 [I 2022-10-28 03:37:11] ...trap/engines/engine.py.230: loss: 0.14949 [I 2022-10-28 03:37:11] ...optimizers/trijoint.py.151: total_norm: 14.934962 [I 2022-10-28 03:37:16] ...trap/engines/engine.py.224: train: epoch 104 | batch 20/2304 [I 2022-10-28 03:37:16] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:21:59 [I 2022-10-28 03:37:16] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00415 [I 2022-10-28 03:37:16] ...trap/engines/engine.py.230: loss: 0.10931 [I 2022-10-28 03:37:16] ...optimizers/trijoint.py.151: total_norm: 17.845076 [I 2022-10-28 03:37:22] ...trap/engines/engine.py.224: train: epoch 104 | batch 30/2304 [I 2022-10-28 03:37:22] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:19 [I 2022-10-28 03:37:22] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00352 [I 2022-10-28 03:37:22] ...trap/engines/engine.py.230: loss: 0.12213 [I 2022-10-28 03:37:22] ...optimizers/trijoint.py.151: total_norm: 18.339895 [I 2022-10-28 03:37:27] ...trap/engines/engine.py.224: train: epoch 104 | batch 40/2304 [I 2022-10-28 03:37:27] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:06 [I 2022-10-28 03:37:27] ...trap/engines/engine.py.229: process: 0.55534 | load: 0.00334 [I 2022-10-28 03:37:27] ...trap/engines/engine.py.230: loss: 0.12806 [I 2022-10-28 03:37:27] ...optimizers/trijoint.py.151: total_norm: 15.925245 [I 2022-10-28 03:37:33] ...trap/engines/engine.py.224: train: epoch 104 | batch 50/2304 [I 2022-10-28 03:37:33] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:08 [I 2022-10-28 03:37:33] ...trap/engines/engine.py.229: process: 0.55828 | load: 0.00280 [I 2022-10-28 03:37:33] ...trap/engines/engine.py.230: loss: 0.11574 [I 2022-10-28 03:37:33] ...optimizers/trijoint.py.151: total_norm: 18.761574 [I 2022-10-28 03:37:39] ...trap/engines/engine.py.224: train: epoch 104 | batch 60/2304 [I 2022-10-28 03:37:39] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:53 [I 2022-10-28 03:37:39] ...trap/engines/engine.py.229: process: 0.56348 | load: 0.00264 [I 2022-10-28 03:37:39] ...trap/engines/engine.py.230: loss: 0.10530 [I 2022-10-28 03:37:39] ...optimizers/trijoint.py.151: total_norm: 26.883936 [I 2022-10-28 03:37:44] ...trap/engines/engine.py.224: train: epoch 104 | batch 70/2304 [I 2022-10-28 03:37:44] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:45 [I 2022-10-28 03:37:44] ...trap/engines/engine.py.229: process: 0.55339 | load: 0.00277 [I 2022-10-28 03:37:44] ...trap/engines/engine.py.230: loss: 0.10258 [I 2022-10-28 03:37:44] ...optimizers/trijoint.py.151: total_norm: 19.699791 [I 2022-10-28 03:37:50] ...trap/engines/engine.py.224: train: epoch 104 | batch 80/2304 [I 2022-10-28 03:37:50] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:39 [I 2022-10-28 03:37:50] ...trap/engines/engine.py.229: process: 0.54912 | load: 0.00251 [I 2022-10-28 03:37:50] ...trap/engines/engine.py.230: loss: 0.12028 [I 2022-10-28 03:37:50] ...optimizers/trijoint.py.151: total_norm: 22.295790 [I 2022-10-28 03:37:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 90/2304 [I 2022-10-28 03:37:55] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:30 [I 2022-10-28 03:37:55] ...trap/engines/engine.py.229: process: 0.55419 | load: 0.00256 [I 2022-10-28 03:37:55] ...trap/engines/engine.py.230: loss: 0.10197 [I 2022-10-28 03:37:55] ...optimizers/trijoint.py.151: total_norm: 11.784252 [I 2022-10-28 03:38:01] ...trap/engines/engine.py.224: train: epoch 104 | batch 100/2304 [I 2022-10-28 03:38:01] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:27 [I 2022-10-28 03:38:01] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00272 [I 2022-10-28 03:38:01] ...trap/engines/engine.py.230: loss: 0.09730 [I 2022-10-28 03:38:01] ...optimizers/trijoint.py.151: total_norm: 16.177048 [I 2022-10-28 03:38:07] ...trap/engines/engine.py.224: train: epoch 104 | batch 110/2304 [I 2022-10-28 03:38:07] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:43 [I 2022-10-28 03:38:07] ...trap/engines/engine.py.229: process: 0.54665 | load: 0.00257 [I 2022-10-28 03:38:07] ...trap/engines/engine.py.230: loss: 0.11475 [I 2022-10-28 03:38:07] ...optimizers/trijoint.py.151: total_norm: 15.619575 [I 2022-10-28 03:38:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 120/2304 [I 2022-10-28 03:38:12] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:19 [I 2022-10-28 03:38:12] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00261 [I 2022-10-28 03:38:12] ...trap/engines/engine.py.230: loss: 0.09859 [I 2022-10-28 03:38:12] ...optimizers/trijoint.py.151: total_norm: 17.176624 [I 2022-10-28 03:38:18] ...trap/engines/engine.py.224: train: epoch 104 | batch 130/2304 [I 2022-10-28 03:38:18] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:10 [I 2022-10-28 03:38:18] ...trap/engines/engine.py.229: process: 0.54924 | load: 0.00242 [I 2022-10-28 03:38:18] ...trap/engines/engine.py.230: loss: 0.11390 [I 2022-10-28 03:38:18] ...optimizers/trijoint.py.151: total_norm: 20.866705 [I 2022-10-28 03:38:24] ...trap/engines/engine.py.224: train: epoch 104 | batch 140/2304 [I 2022-10-28 03:38:24] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:19:59 [I 2022-10-28 03:38:24] ...trap/engines/engine.py.229: process: 0.55068 | load: 0.00284 [I 2022-10-28 03:38:24] ...trap/engines/engine.py.230: loss: 0.22614 [I 2022-10-28 03:38:24] ...optimizers/trijoint.py.151: total_norm: 13.914850 [I 2022-10-28 03:38:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 150/2304 [I 2022-10-28 03:38:29] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:05 [I 2022-10-28 03:38:29] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00266 [I 2022-10-28 03:38:29] ...trap/engines/engine.py.230: loss: 0.09578 [I 2022-10-28 03:38:29] ...optimizers/trijoint.py.151: total_norm: 17.203772 [I 2022-10-28 03:38:35] ...trap/engines/engine.py.224: train: epoch 104 | batch 160/2304 [I 2022-10-28 03:38:35] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:04 [I 2022-10-28 03:38:35] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00258 [I 2022-10-28 03:38:35] ...trap/engines/engine.py.230: loss: 0.28278 [I 2022-10-28 03:38:35] ...optimizers/trijoint.py.151: total_norm: 19.746805 [I 2022-10-28 03:38:40] ...trap/engines/engine.py.224: train: epoch 104 | batch 170/2304 [I 2022-10-28 03:38:40] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:51 [I 2022-10-28 03:38:40] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00284 [I 2022-10-28 03:38:40] ...trap/engines/engine.py.230: loss: 0.16505 [I 2022-10-28 03:38:40] ...optimizers/trijoint.py.151: total_norm: 12.396004 [I 2022-10-28 03:38:46] ...trap/engines/engine.py.224: train: epoch 104 | batch 180/2304 [I 2022-10-28 03:38:46] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:44 [I 2022-10-28 03:38:46] ...trap/engines/engine.py.229: process: 0.56355 | load: 0.00263 [I 2022-10-28 03:38:46] ...trap/engines/engine.py.230: loss: 0.08530 [I 2022-10-28 03:38:46] ...optimizers/trijoint.py.151: total_norm: 9.732971 [I 2022-10-28 03:38:52] ...trap/engines/engine.py.224: train: epoch 104 | batch 190/2304 [I 2022-10-28 03:38:52] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:36 [I 2022-10-28 03:38:52] ...trap/engines/engine.py.229: process: 0.55603 | load: 0.00260 [I 2022-10-28 03:38:52] ...trap/engines/engine.py.230: loss: 0.08824 [I 2022-10-28 03:38:52] ...optimizers/trijoint.py.151: total_norm: 10.007826 [I 2022-10-28 03:38:57] ...trap/engines/engine.py.224: train: epoch 104 | batch 200/2304 [I 2022-10-28 03:38:57] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:22 [I 2022-10-28 03:38:57] ...trap/engines/engine.py.229: process: 0.53951 | load: 0.00263 [I 2022-10-28 03:38:57] ...trap/engines/engine.py.230: loss: 0.09521 [I 2022-10-28 03:38:57] ...optimizers/trijoint.py.151: total_norm: 14.204405 [I 2022-10-28 03:39:03] ...trap/engines/engine.py.224: train: epoch 104 | batch 210/2304 [I 2022-10-28 03:39:03] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:27 [I 2022-10-28 03:39:03] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00251 [I 2022-10-28 03:39:03] ...trap/engines/engine.py.230: loss: 0.09951 [I 2022-10-28 03:39:03] ...optimizers/trijoint.py.151: total_norm: 11.026103 [I 2022-10-28 03:39:08] ...trap/engines/engine.py.224: train: epoch 104 | batch 220/2304 [I 2022-10-28 03:39:08] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:34 [I 2022-10-28 03:39:08] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00259 [I 2022-10-28 03:39:08] ...trap/engines/engine.py.230: loss: 0.11861 [I 2022-10-28 03:39:08] ...optimizers/trijoint.py.151: total_norm: 24.816978 [I 2022-10-28 03:39:14] ...trap/engines/engine.py.224: train: epoch 104 | batch 230/2304 [I 2022-10-28 03:39:14] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:19 [I 2022-10-28 03:39:14] ...trap/engines/engine.py.229: process: 0.56468 | load: 0.00253 [I 2022-10-28 03:39:14] ...trap/engines/engine.py.230: loss: 0.10083 [I 2022-10-28 03:39:14] ...optimizers/trijoint.py.151: total_norm: 9.554458 [I 2022-10-28 03:39:20] ...trap/engines/engine.py.224: train: epoch 104 | batch 240/2304 [I 2022-10-28 03:39:20] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:14 [I 2022-10-28 03:39:20] ...trap/engines/engine.py.229: process: 0.56234 | load: 0.00266 [I 2022-10-28 03:39:20] ...trap/engines/engine.py.230: loss: 0.05523 [I 2022-10-28 03:39:20] ...optimizers/trijoint.py.151: total_norm: 10.362428 [I 2022-10-28 03:39:25] ...trap/engines/engine.py.224: train: epoch 104 | batch 250/2304 [I 2022-10-28 03:39:25] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:01 [I 2022-10-28 03:39:25] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00264 [I 2022-10-28 03:39:25] ...trap/engines/engine.py.230: loss: 0.08429 [I 2022-10-28 03:39:25] ...optimizers/trijoint.py.151: total_norm: 13.754419 [I 2022-10-28 03:39:31] ...trap/engines/engine.py.224: train: epoch 104 | batch 260/2304 [I 2022-10-28 03:39:31] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:55 [I 2022-10-28 03:39:31] ...trap/engines/engine.py.229: process: 0.55148 | load: 0.00254 [I 2022-10-28 03:39:31] ...trap/engines/engine.py.230: loss: 0.10885 [I 2022-10-28 03:39:31] ...optimizers/trijoint.py.151: total_norm: 15.975697 [I 2022-10-28 03:39:36] ...trap/engines/engine.py.224: train: epoch 104 | batch 270/2304 [I 2022-10-28 03:39:36] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:19:02 [I 2022-10-28 03:39:36] ...trap/engines/engine.py.229: process: 0.54598 | load: 0.00260 [I 2022-10-28 03:39:36] ...trap/engines/engine.py.230: loss: 0.09380 [I 2022-10-28 03:39:36] ...optimizers/trijoint.py.151: total_norm: 11.064796 [I 2022-10-28 03:39:42] ...trap/engines/engine.py.224: train: epoch 104 | batch 280/2304 [I 2022-10-28 03:39:42] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:41 [I 2022-10-28 03:39:42] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00252 [I 2022-10-28 03:39:42] ...trap/engines/engine.py.230: loss: 0.10707 [I 2022-10-28 03:39:42] ...optimizers/trijoint.py.151: total_norm: 20.061823 [I 2022-10-28 03:39:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 290/2304 [I 2022-10-28 03:39:47] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:43 [I 2022-10-28 03:39:47] ...trap/engines/engine.py.229: process: 0.56338 | load: 0.00245 [I 2022-10-28 03:39:47] ...trap/engines/engine.py.230: loss: 0.08518 [I 2022-10-28 03:39:47] ...optimizers/trijoint.py.151: total_norm: 24.911007 [I 2022-10-28 03:39:53] ...trap/engines/engine.py.224: train: epoch 104 | batch 300/2304 [I 2022-10-28 03:39:53] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:39 [I 2022-10-28 03:39:53] ...trap/engines/engine.py.229: process: 0.55043 | load: 0.00257 [I 2022-10-28 03:39:53] ...trap/engines/engine.py.230: loss: 0.16072 [I 2022-10-28 03:39:53] ...optimizers/trijoint.py.151: total_norm: 21.867785 [I 2022-10-28 03:39:59] ...trap/engines/engine.py.224: train: epoch 104 | batch 310/2304 [I 2022-10-28 03:39:59] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:29 [I 2022-10-28 03:39:59] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00246 [I 2022-10-28 03:39:59] ...trap/engines/engine.py.230: loss: 0.10105 [I 2022-10-28 03:39:59] ...optimizers/trijoint.py.151: total_norm: 12.811164 [I 2022-10-28 03:40:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 320/2304 [I 2022-10-28 03:40:04] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:29 [I 2022-10-28 03:40:04] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00240 [I 2022-10-28 03:40:04] ...trap/engines/engine.py.230: loss: 0.12191 [I 2022-10-28 03:40:04] ...optimizers/trijoint.py.151: total_norm: 23.016785 [I 2022-10-28 03:40:10] ...trap/engines/engine.py.224: train: epoch 104 | batch 330/2304 [I 2022-10-28 03:40:10] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:45 [I 2022-10-28 03:40:10] ...trap/engines/engine.py.229: process: 0.55371 | load: 0.00250 [I 2022-10-28 03:40:10] ...trap/engines/engine.py.230: loss: 0.19215 [I 2022-10-28 03:40:10] ...optimizers/trijoint.py.151: total_norm: 32.486217 [I 2022-10-28 03:40:16] ...trap/engines/engine.py.224: train: epoch 104 | batch 340/2304 [I 2022-10-28 03:40:16] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:26 [I 2022-10-28 03:40:16] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00250 [I 2022-10-28 03:40:16] ...trap/engines/engine.py.230: loss: 0.10026 [I 2022-10-28 03:40:16] ...optimizers/trijoint.py.151: total_norm: 12.154816 [I 2022-10-28 03:40:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 350/2304 [I 2022-10-28 03:40:21] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:13 [I 2022-10-28 03:40:21] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00242 [I 2022-10-28 03:40:21] ...trap/engines/engine.py.230: loss: 0.10563 [I 2022-10-28 03:40:21] ...optimizers/trijoint.py.151: total_norm: 11.783751 [I 2022-10-28 03:40:27] ...trap/engines/engine.py.224: train: epoch 104 | batch 360/2304 [I 2022-10-28 03:40:27] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:59 [I 2022-10-28 03:40:27] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00242 [I 2022-10-28 03:40:27] ...trap/engines/engine.py.230: loss: 0.12455 [I 2022-10-28 03:40:27] ...optimizers/trijoint.py.151: total_norm: 10.829955 [I 2022-10-28 03:40:32] ...trap/engines/engine.py.224: train: epoch 104 | batch 370/2304 [I 2022-10-28 03:40:32] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:59 [I 2022-10-28 03:40:32] ...trap/engines/engine.py.229: process: 0.56615 | load: 0.00247 [I 2022-10-28 03:40:32] ...trap/engines/engine.py.230: loss: 0.13618 [I 2022-10-28 03:40:32] ...optimizers/trijoint.py.151: total_norm: 18.619921 [I 2022-10-28 03:40:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 380/2304 [I 2022-10-28 03:40:38] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:52 [I 2022-10-28 03:40:38] ...trap/engines/engine.py.229: process: 0.55459 | load: 0.00244 [I 2022-10-28 03:40:38] ...trap/engines/engine.py.230: loss: 0.08650 [I 2022-10-28 03:40:38] ...optimizers/trijoint.py.151: total_norm: 10.264783 [I 2022-10-28 03:40:43] ...trap/engines/engine.py.224: train: epoch 104 | batch 390/2304 [I 2022-10-28 03:40:43] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:45 [I 2022-10-28 03:40:43] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00252 [I 2022-10-28 03:40:43] ...trap/engines/engine.py.230: loss: 0.13578 [I 2022-10-28 03:40:43] ...optimizers/trijoint.py.151: total_norm: 11.569525 [I 2022-10-28 03:40:49] ...trap/engines/engine.py.224: train: epoch 104 | batch 400/2304 [I 2022-10-28 03:40:49] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:44 [I 2022-10-28 03:40:49] ...trap/engines/engine.py.229: process: 0.54663 | load: 0.00241 [I 2022-10-28 03:40:49] ...trap/engines/engine.py.230: loss: 0.09774 [I 2022-10-28 03:40:49] ...optimizers/trijoint.py.151: total_norm: 10.793063 [I 2022-10-28 03:40:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 410/2304 [I 2022-10-28 03:40:55] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:31 [I 2022-10-28 03:40:55] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00244 [I 2022-10-28 03:40:55] ...trap/engines/engine.py.230: loss: 0.15564 [I 2022-10-28 03:40:55] ...optimizers/trijoint.py.151: total_norm: 13.508887 [I 2022-10-28 03:41:00] ...trap/engines/engine.py.224: train: epoch 104 | batch 420/2304 [I 2022-10-28 03:41:00] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:35 [I 2022-10-28 03:41:00] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00243 [I 2022-10-28 03:41:00] ...trap/engines/engine.py.230: loss: 0.10502 [I 2022-10-28 03:41:00] ...optimizers/trijoint.py.151: total_norm: 12.848204 [I 2022-10-28 03:41:06] ...trap/engines/engine.py.224: train: epoch 104 | batch 430/2304 [I 2022-10-28 03:41:06] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:35 [I 2022-10-28 03:41:06] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00255 [I 2022-10-28 03:41:06] ...trap/engines/engine.py.230: loss: 0.08384 [I 2022-10-28 03:41:06] ...optimizers/trijoint.py.151: total_norm: 12.521382 [I 2022-10-28 03:41:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 440/2304 [I 2022-10-28 03:41:12] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:29 [I 2022-10-28 03:41:12] ...trap/engines/engine.py.229: process: 0.56214 | load: 0.00247 [I 2022-10-28 03:41:12] ...trap/engines/engine.py.230: loss: 0.08026 [I 2022-10-28 03:41:12] ...optimizers/trijoint.py.151: total_norm: 9.789825 [I 2022-10-28 03:41:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 450/2304 [I 2022-10-28 03:41:17] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:14 [I 2022-10-28 03:41:17] ...trap/engines/engine.py.229: process: 0.56944 | load: 0.00246 [I 2022-10-28 03:41:17] ...trap/engines/engine.py.230: loss: 0.10229 [I 2022-10-28 03:41:17] ...optimizers/trijoint.py.151: total_norm: 12.738160 [I 2022-10-28 03:41:23] ...trap/engines/engine.py.224: train: epoch 104 | batch 460/2304 [I 2022-10-28 03:41:23] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:00 [I 2022-10-28 03:41:23] ...trap/engines/engine.py.229: process: 0.54056 | load: 0.00242 [I 2022-10-28 03:41:23] ...trap/engines/engine.py.230: loss: 0.10316 [I 2022-10-28 03:41:23] ...optimizers/trijoint.py.151: total_norm: 27.921555 [I 2022-10-28 03:41:28] ...trap/engines/engine.py.224: train: epoch 104 | batch 470/2304 [I 2022-10-28 03:41:28] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:08 [I 2022-10-28 03:41:28] ...trap/engines/engine.py.229: process: 0.56773 | load: 0.00250 [I 2022-10-28 03:41:28] ...trap/engines/engine.py.230: loss: 0.12752 [I 2022-10-28 03:41:28] ...optimizers/trijoint.py.151: total_norm: 17.975061 [I 2022-10-28 03:41:34] ...trap/engines/engine.py.224: train: epoch 104 | batch 480/2304 [I 2022-10-28 03:41:34] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:05 [I 2022-10-28 03:41:34] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00248 [I 2022-10-28 03:41:34] ...trap/engines/engine.py.230: loss: 0.11716 [I 2022-10-28 03:41:34] ...optimizers/trijoint.py.151: total_norm: 15.250304 [I 2022-10-28 03:41:39] ...trap/engines/engine.py.224: train: epoch 104 | batch 490/2304 [I 2022-10-28 03:41:39] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:49 [I 2022-10-28 03:41:39] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00250 [I 2022-10-28 03:41:39] ...trap/engines/engine.py.230: loss: 0.09201 [I 2022-10-28 03:41:39] ...optimizers/trijoint.py.151: total_norm: 16.497440 [I 2022-10-28 03:41:45] ...trap/engines/engine.py.224: train: epoch 104 | batch 500/2304 [I 2022-10-28 03:41:45] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:46 [I 2022-10-28 03:41:45] ...trap/engines/engine.py.229: process: 0.55522 | load: 0.00248 [I 2022-10-28 03:41:45] ...trap/engines/engine.py.230: loss: 0.10463 [I 2022-10-28 03:41:45] ...optimizers/trijoint.py.151: total_norm: 15.588296 [I 2022-10-28 03:41:51] ...trap/engines/engine.py.224: train: epoch 104 | batch 510/2304 [I 2022-10-28 03:41:51] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:17:09 [I 2022-10-28 03:41:51] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00270 [I 2022-10-28 03:41:51] ...trap/engines/engine.py.230: loss: 0.07596 [I 2022-10-28 03:41:51] ...optimizers/trijoint.py.151: total_norm: 12.083698 [I 2022-10-28 03:41:56] ...trap/engines/engine.py.224: train: epoch 104 | batch 520/2304 [I 2022-10-28 03:41:56] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:41 [I 2022-10-28 03:41:56] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00248 [I 2022-10-28 03:41:56] ...trap/engines/engine.py.230: loss: 0.16999 [I 2022-10-28 03:41:56] ...optimizers/trijoint.py.151: total_norm: 17.446268 [I 2022-10-28 03:42:02] ...trap/engines/engine.py.224: train: epoch 104 | batch 530/2304 [I 2022-10-28 03:42:02] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:27 [I 2022-10-28 03:42:02] ...trap/engines/engine.py.229: process: 0.55687 | load: 0.00248 [I 2022-10-28 03:42:02] ...trap/engines/engine.py.230: loss: 0.21774 [I 2022-10-28 03:42:02] ...optimizers/trijoint.py.151: total_norm: 12.248400 [I 2022-10-28 03:42:08] ...trap/engines/engine.py.224: train: epoch 104 | batch 540/2304 [I 2022-10-28 03:42:08] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:24 [I 2022-10-28 03:42:08] ...trap/engines/engine.py.229: process: 0.56116 | load: 0.00253 [I 2022-10-28 03:42:08] ...trap/engines/engine.py.230: loss: 0.11973 [I 2022-10-28 03:42:08] ...optimizers/trijoint.py.151: total_norm: 17.535162 [I 2022-10-28 03:42:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 550/2304 [I 2022-10-28 03:42:13] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:19 [I 2022-10-28 03:42:13] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00252 [I 2022-10-28 03:42:13] ...trap/engines/engine.py.230: loss: 0.12124 [I 2022-10-28 03:42:13] ...optimizers/trijoint.py.151: total_norm: 15.041226 [I 2022-10-28 03:42:19] ...trap/engines/engine.py.224: train: epoch 104 | batch 560/2304 [I 2022-10-28 03:42:19] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:12 [I 2022-10-28 03:42:19] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00255 [I 2022-10-28 03:42:19] ...trap/engines/engine.py.230: loss: 0.13421 [I 2022-10-28 03:42:19] ...optimizers/trijoint.py.151: total_norm: 13.016600 [I 2022-10-28 03:42:24] ...trap/engines/engine.py.224: train: epoch 104 | batch 570/2304 [I 2022-10-28 03:42:24] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:07 [I 2022-10-28 03:42:24] ...trap/engines/engine.py.229: process: 0.55217 | load: 0.00249 [I 2022-10-28 03:42:24] ...trap/engines/engine.py.230: loss: 0.20262 [I 2022-10-28 03:42:24] ...optimizers/trijoint.py.151: total_norm: 11.427797 [I 2022-10-28 03:42:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 580/2304 [I 2022-10-28 03:42:30] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:04 [I 2022-10-28 03:42:30] ...trap/engines/engine.py.229: process: 0.56134 | load: 0.00251 [I 2022-10-28 03:42:30] ...trap/engines/engine.py.230: loss: 0.08844 [I 2022-10-28 03:42:30] ...optimizers/trijoint.py.151: total_norm: 17.932985 [I 2022-10-28 03:42:36] ...trap/engines/engine.py.224: train: epoch 104 | batch 590/2304 [I 2022-10-28 03:42:36] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:56 [I 2022-10-28 03:42:36] ...trap/engines/engine.py.229: process: 0.56196 | load: 0.00250 [I 2022-10-28 03:42:36] ...trap/engines/engine.py.230: loss: 0.10317 [I 2022-10-28 03:42:36] ...optimizers/trijoint.py.151: total_norm: 17.054562 [I 2022-10-28 03:42:41] ...trap/engines/engine.py.224: train: epoch 104 | batch 600/2304 [I 2022-10-28 03:42:41] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:52 [I 2022-10-28 03:42:41] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00244 [I 2022-10-28 03:42:41] ...trap/engines/engine.py.230: loss: 0.12814 [I 2022-10-28 03:42:41] ...optimizers/trijoint.py.151: total_norm: 10.108994 [I 2022-10-28 03:42:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 610/2304 [I 2022-10-28 03:42:47] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:44 [I 2022-10-28 03:42:47] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00244 [I 2022-10-28 03:42:47] ...trap/engines/engine.py.230: loss: 0.10247 [I 2022-10-28 03:42:47] ...optimizers/trijoint.py.151: total_norm: 10.878886 [I 2022-10-28 03:42:52] ...trap/engines/engine.py.224: train: epoch 104 | batch 620/2304 [I 2022-10-28 03:42:52] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:44 [I 2022-10-28 03:42:52] ...trap/engines/engine.py.229: process: 0.56791 | load: 0.00243 [I 2022-10-28 03:42:52] ...trap/engines/engine.py.230: loss: 0.15324 [I 2022-10-28 03:42:52] ...optimizers/trijoint.py.151: total_norm: 17.790180 [I 2022-10-28 03:42:58] ...trap/engines/engine.py.224: train: epoch 104 | batch 630/2304 [I 2022-10-28 03:42:58] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:35 [I 2022-10-28 03:42:58] ...trap/engines/engine.py.229: process: 0.56374 | load: 0.00248 [I 2022-10-28 03:42:58] ...trap/engines/engine.py.230: loss: 0.14360 [I 2022-10-28 03:42:58] ...optimizers/trijoint.py.151: total_norm: 13.063423 [I 2022-10-28 03:43:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 640/2304 [I 2022-10-28 03:43:04] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:32 [I 2022-10-28 03:43:04] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00247 [I 2022-10-28 03:43:04] ...trap/engines/engine.py.230: loss: 0.15710 [I 2022-10-28 03:43:04] ...optimizers/trijoint.py.151: total_norm: 38.515430 [I 2022-10-28 03:43:09] ...trap/engines/engine.py.224: train: epoch 104 | batch 650/2304 [I 2022-10-28 03:43:09] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:30 [I 2022-10-28 03:43:09] ...trap/engines/engine.py.229: process: 0.57511 | load: 0.00244 [I 2022-10-28 03:43:09] ...trap/engines/engine.py.230: loss: 0.14668 [I 2022-10-28 03:43:09] ...optimizers/trijoint.py.151: total_norm: 13.763866 [I 2022-10-28 03:43:15] ...trap/engines/engine.py.224: train: epoch 104 | batch 660/2304 [I 2022-10-28 03:43:15] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:16 [I 2022-10-28 03:43:15] ...trap/engines/engine.py.229: process: 0.54517 | load: 0.00246 [I 2022-10-28 03:43:15] ...trap/engines/engine.py.230: loss: 0.11030 [I 2022-10-28 03:43:15] ...optimizers/trijoint.py.151: total_norm: 63.727425 [I 2022-10-28 03:43:20] ...trap/engines/engine.py.224: train: epoch 104 | batch 670/2304 [I 2022-10-28 03:43:20] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:09 [I 2022-10-28 03:43:20] ...trap/engines/engine.py.229: process: 0.55192 | load: 0.00243 [I 2022-10-28 03:43:20] ...trap/engines/engine.py.230: loss: 0.11048 [I 2022-10-28 03:43:20] ...optimizers/trijoint.py.151: total_norm: 8.903589 [I 2022-10-28 03:43:26] ...trap/engines/engine.py.224: train: epoch 104 | batch 680/2304 [I 2022-10-28 03:43:26] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:11 [I 2022-10-28 03:43:26] ...trap/engines/engine.py.229: process: 0.56112 | load: 0.00250 [I 2022-10-28 03:43:26] ...trap/engines/engine.py.230: loss: 0.09857 [I 2022-10-28 03:43:26] ...optimizers/trijoint.py.151: total_norm: 24.619408 [I 2022-10-28 03:43:32] ...trap/engines/engine.py.224: train: epoch 104 | batch 690/2304 [I 2022-10-28 03:43:32] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:05 [I 2022-10-28 03:43:32] ...trap/engines/engine.py.229: process: 0.56418 | load: 0.00278 [I 2022-10-28 03:43:32] ...trap/engines/engine.py.230: loss: 0.11527 [I 2022-10-28 03:43:32] ...optimizers/trijoint.py.151: total_norm: 12.133606 [I 2022-10-28 03:43:37] ...trap/engines/engine.py.224: train: epoch 104 | batch 700/2304 [I 2022-10-28 03:43:37] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:54 [I 2022-10-28 03:43:37] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00273 [I 2022-10-28 03:43:37] ...trap/engines/engine.py.230: loss: 0.12216 [I 2022-10-28 03:43:37] ...optimizers/trijoint.py.151: total_norm: 13.987387 [I 2022-10-28 03:43:43] ...trap/engines/engine.py.224: train: epoch 104 | batch 710/2304 [I 2022-10-28 03:43:43] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:57 [I 2022-10-28 03:43:43] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00257 [I 2022-10-28 03:43:43] ...trap/engines/engine.py.230: loss: 0.15569 [I 2022-10-28 03:43:43] ...optimizers/trijoint.py.151: total_norm: 26.811598 [I 2022-10-28 03:43:48] ...trap/engines/engine.py.224: train: epoch 104 | batch 720/2304 [I 2022-10-28 03:43:48] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:47 [I 2022-10-28 03:43:48] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00255 [I 2022-10-28 03:43:48] ...trap/engines/engine.py.230: loss: 0.12144 [I 2022-10-28 03:43:48] ...optimizers/trijoint.py.151: total_norm: 17.077375 [I 2022-10-28 03:43:54] ...trap/engines/engine.py.224: train: epoch 104 | batch 730/2304 [I 2022-10-28 03:43:54] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:36 [I 2022-10-28 03:43:54] ...trap/engines/engine.py.229: process: 0.55472 | load: 0.00244 [I 2022-10-28 03:43:54] ...trap/engines/engine.py.230: loss: 0.10445 [I 2022-10-28 03:43:54] ...optimizers/trijoint.py.151: total_norm: 27.806629 [I 2022-10-28 03:44:00] ...trap/engines/engine.py.224: train: epoch 104 | batch 740/2304 [I 2022-10-28 03:44:00] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:29 [I 2022-10-28 03:44:00] ...trap/engines/engine.py.229: process: 0.55072 | load: 0.00245 [I 2022-10-28 03:44:00] ...trap/engines/engine.py.230: loss: 0.15393 [I 2022-10-28 03:44:00] ...optimizers/trijoint.py.151: total_norm: 30.590500 [I 2022-10-28 03:44:05] ...trap/engines/engine.py.224: train: epoch 104 | batch 750/2304 [I 2022-10-28 03:44:05] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:27 [I 2022-10-28 03:44:05] ...trap/engines/engine.py.229: process: 0.55611 | load: 0.00241 [I 2022-10-28 03:44:05] ...trap/engines/engine.py.230: loss: 0.09414 [I 2022-10-28 03:44:05] ...optimizers/trijoint.py.151: total_norm: 11.279395 [I 2022-10-28 03:44:11] ...trap/engines/engine.py.224: train: epoch 104 | batch 760/2304 [I 2022-10-28 03:44:11] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:26 [I 2022-10-28 03:44:11] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00250 [I 2022-10-28 03:44:11] ...trap/engines/engine.py.230: loss: 0.14354 [I 2022-10-28 03:44:11] ...optimizers/trijoint.py.151: total_norm: 21.773911 [I 2022-10-28 03:44:16] ...trap/engines/engine.py.224: train: epoch 104 | batch 770/2304 [I 2022-10-28 03:44:16] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:19 [I 2022-10-28 03:44:16] ...trap/engines/engine.py.229: process: 0.57250 | load: 0.00247 [I 2022-10-28 03:44:16] ...trap/engines/engine.py.230: loss: 0.12583 [I 2022-10-28 03:44:16] ...optimizers/trijoint.py.151: total_norm: 33.713413 [I 2022-10-28 03:44:22] ...trap/engines/engine.py.224: train: epoch 104 | batch 780/2304 [I 2022-10-28 03:44:22] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:11 [I 2022-10-28 03:44:22] ...trap/engines/engine.py.229: process: 0.55259 | load: 0.00243 [I 2022-10-28 03:44:22] ...trap/engines/engine.py.230: loss: 0.08597 [I 2022-10-28 03:44:22] ...optimizers/trijoint.py.151: total_norm: 13.376074 [I 2022-10-28 03:44:28] ...trap/engines/engine.py.224: train: epoch 104 | batch 790/2304 [I 2022-10-28 03:44:28] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:04 [I 2022-10-28 03:44:28] ...trap/engines/engine.py.229: process: 0.54552 | load: 0.00243 [I 2022-10-28 03:44:28] ...trap/engines/engine.py.230: loss: 0.10568 [I 2022-10-28 03:44:28] ...optimizers/trijoint.py.151: total_norm: 7.602317 [I 2022-10-28 03:44:33] ...trap/engines/engine.py.224: train: epoch 104 | batch 800/2304 [I 2022-10-28 03:44:33] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:56 [I 2022-10-28 03:44:33] ...trap/engines/engine.py.229: process: 0.56433 | load: 0.00246 [I 2022-10-28 03:44:33] ...trap/engines/engine.py.230: loss: 0.07259 [I 2022-10-28 03:44:33] ...optimizers/trijoint.py.151: total_norm: 13.998055 [I 2022-10-28 03:44:39] ...trap/engines/engine.py.224: train: epoch 104 | batch 810/2304 [I 2022-10-28 03:44:39] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:59 [I 2022-10-28 03:44:39] ...trap/engines/engine.py.229: process: 0.55720 | load: 0.00257 [I 2022-10-28 03:44:39] ...trap/engines/engine.py.230: loss: 0.15029 [I 2022-10-28 03:44:39] ...optimizers/trijoint.py.151: total_norm: 21.945124 [I 2022-10-28 03:44:44] ...trap/engines/engine.py.224: train: epoch 104 | batch 820/2304 [I 2022-10-28 03:44:44] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:41 [I 2022-10-28 03:44:44] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00253 [I 2022-10-28 03:44:44] ...trap/engines/engine.py.230: loss: 0.15833 [I 2022-10-28 03:44:44] ...optimizers/trijoint.py.151: total_norm: 29.633072 [I 2022-10-28 03:44:50] ...trap/engines/engine.py.224: train: epoch 104 | batch 830/2304 [I 2022-10-28 03:44:50] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:52 [I 2022-10-28 03:44:50] ...trap/engines/engine.py.229: process: 0.58075 | load: 0.00250 [I 2022-10-28 03:44:50] ...trap/engines/engine.py.230: loss: 0.09617 [I 2022-10-28 03:44:50] ...optimizers/trijoint.py.151: total_norm: 13.714547 [I 2022-10-28 03:44:56] ...trap/engines/engine.py.224: train: epoch 104 | batch 840/2304 [I 2022-10-28 03:44:56] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:43 [I 2022-10-28 03:44:56] ...trap/engines/engine.py.229: process: 0.55943 | load: 0.00248 [I 2022-10-28 03:44:56] ...trap/engines/engine.py.230: loss: 0.12392 [I 2022-10-28 03:44:56] ...optimizers/trijoint.py.151: total_norm: 16.634945 [I 2022-10-28 03:45:01] ...trap/engines/engine.py.224: train: epoch 104 | batch 850/2304 [I 2022-10-28 03:45:01] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:39 [I 2022-10-28 03:45:01] ...trap/engines/engine.py.229: process: 0.57098 | load: 0.00252 [I 2022-10-28 03:45:01] ...trap/engines/engine.py.230: loss: 0.13991 [I 2022-10-28 03:45:01] ...optimizers/trijoint.py.151: total_norm: 16.607796 [I 2022-10-28 03:45:07] ...trap/engines/engine.py.224: train: epoch 104 | batch 860/2304 [I 2022-10-28 03:45:07] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:30 [I 2022-10-28 03:45:07] ...trap/engines/engine.py.229: process: 0.56695 | load: 0.00248 [I 2022-10-28 03:45:07] ...trap/engines/engine.py.230: loss: 0.11145 [I 2022-10-28 03:45:07] ...optimizers/trijoint.py.151: total_norm: 34.844921 [I 2022-10-28 03:45:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 870/2304 [I 2022-10-28 03:45:12] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:21 [I 2022-10-28 03:45:12] ...trap/engines/engine.py.229: process: 0.56331 | load: 0.00246 [I 2022-10-28 03:45:12] ...trap/engines/engine.py.230: loss: 0.10023 [I 2022-10-28 03:45:12] ...optimizers/trijoint.py.151: total_norm: 20.520544 [I 2022-10-28 03:45:18] ...trap/engines/engine.py.224: train: epoch 104 | batch 880/2304 [I 2022-10-28 03:45:18] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:13 [I 2022-10-28 03:45:18] ...trap/engines/engine.py.229: process: 0.56698 | load: 0.00245 [I 2022-10-28 03:45:18] ...trap/engines/engine.py.230: loss: 0.11616 [I 2022-10-28 03:45:18] ...optimizers/trijoint.py.151: total_norm: 15.647420 [I 2022-10-28 03:45:24] ...trap/engines/engine.py.224: train: epoch 104 | batch 890/2304 [I 2022-10-28 03:45:24] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:12 [I 2022-10-28 03:45:24] ...trap/engines/engine.py.229: process: 0.57082 | load: 0.00247 [I 2022-10-28 03:45:24] ...trap/engines/engine.py.230: loss: 0.12516 [I 2022-10-28 03:45:24] ...optimizers/trijoint.py.151: total_norm: 33.107121 [I 2022-10-28 03:45:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 900/2304 [I 2022-10-28 03:45:29] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:03 [I 2022-10-28 03:45:29] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00247 [I 2022-10-28 03:45:29] ...trap/engines/engine.py.230: loss: 0.12253 [I 2022-10-28 03:45:29] ...optimizers/trijoint.py.151: total_norm: 19.704132 [I 2022-10-28 03:45:35] ...trap/engines/engine.py.224: train: epoch 104 | batch 910/2304 [I 2022-10-28 03:45:35] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:13:02 [I 2022-10-28 03:45:35] ...trap/engines/engine.py.229: process: 0.56891 | load: 0.00247 [I 2022-10-28 03:45:35] ...trap/engines/engine.py.230: loss: 0.15594 [I 2022-10-28 03:45:35] ...optimizers/trijoint.py.151: total_norm: 16.548298 [I 2022-10-28 03:45:40] ...trap/engines/engine.py.224: train: epoch 104 | batch 920/2304 [I 2022-10-28 03:45:40] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:55 [I 2022-10-28 03:45:40] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00245 [I 2022-10-28 03:45:40] ...trap/engines/engine.py.230: loss: 0.14710 [I 2022-10-28 03:45:40] ...optimizers/trijoint.py.151: total_norm: 26.863783 [I 2022-10-28 03:45:46] ...trap/engines/engine.py.224: train: epoch 104 | batch 930/2304 [I 2022-10-28 03:45:46] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:43 [I 2022-10-28 03:45:46] ...trap/engines/engine.py.229: process: 0.55046 | load: 0.00245 [I 2022-10-28 03:45:46] ...trap/engines/engine.py.230: loss: 0.10816 [I 2022-10-28 03:45:46] ...optimizers/trijoint.py.151: total_norm: 31.598816 [I 2022-10-28 03:45:52] ...trap/engines/engine.py.224: train: epoch 104 | batch 940/2304 [I 2022-10-28 03:45:52] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:46 [I 2022-10-28 03:45:52] ...trap/engines/engine.py.229: process: 0.56627 | load: 0.00245 [I 2022-10-28 03:45:52] ...trap/engines/engine.py.230: loss: 0.11958 [I 2022-10-28 03:45:52] ...optimizers/trijoint.py.151: total_norm: 27.212944 [I 2022-10-28 03:45:57] ...trap/engines/engine.py.224: train: epoch 104 | batch 950/2304 [I 2022-10-28 03:45:57] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:34 [I 2022-10-28 03:45:57] ...trap/engines/engine.py.229: process: 0.55201 | load: 0.00243 [I 2022-10-28 03:45:57] ...trap/engines/engine.py.230: loss: 0.10845 [I 2022-10-28 03:45:57] ...optimizers/trijoint.py.151: total_norm: 11.503113 [I 2022-10-28 03:46:03] ...trap/engines/engine.py.224: train: epoch 104 | batch 960/2304 [I 2022-10-28 03:46:03] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:30 [I 2022-10-28 03:46:03] ...trap/engines/engine.py.229: process: 0.55302 | load: 0.00263 [I 2022-10-28 03:46:03] ...trap/engines/engine.py.230: loss: 0.15189 [I 2022-10-28 03:46:03] ...optimizers/trijoint.py.151: total_norm: 13.046241 [I 2022-10-28 03:46:08] ...trap/engines/engine.py.224: train: epoch 104 | batch 970/2304 [I 2022-10-28 03:46:08] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:18 [I 2022-10-28 03:46:08] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00242 [I 2022-10-28 03:46:08] ...trap/engines/engine.py.230: loss: 0.17590 [I 2022-10-28 03:46:08] ...optimizers/trijoint.py.151: total_norm: 33.905624 [I 2022-10-28 03:46:14] ...trap/engines/engine.py.224: train: epoch 104 | batch 980/2304 [I 2022-10-28 03:46:14] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:15 [I 2022-10-28 03:46:14] ...trap/engines/engine.py.229: process: 0.55326 | load: 0.00253 [I 2022-10-28 03:46:14] ...trap/engines/engine.py.230: loss: 0.14168 [I 2022-10-28 03:46:14] ...optimizers/trijoint.py.151: total_norm: 9.696187 [I 2022-10-28 03:46:20] ...trap/engines/engine.py.224: train: epoch 104 | batch 990/2304 [I 2022-10-28 03:46:20] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:11 [I 2022-10-28 03:46:20] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00253 [I 2022-10-28 03:46:20] ...trap/engines/engine.py.230: loss: 0.09151 [I 2022-10-28 03:46:20] ...optimizers/trijoint.py.151: total_norm: 10.029515 [I 2022-10-28 03:46:25] ...trap/engines/engine.py.224: train: epoch 104 | batch 1000/2304 [I 2022-10-28 03:46:25] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:11:59 [I 2022-10-28 03:46:25] ...trap/engines/engine.py.229: process: 0.54329 | load: 0.00249 [I 2022-10-28 03:46:25] ...trap/engines/engine.py.230: loss: 0.11805 [I 2022-10-28 03:46:25] ...optimizers/trijoint.py.151: total_norm: 12.044304 [I 2022-10-28 03:46:31] ...trap/engines/engine.py.224: train: epoch 104 | batch 1010/2304 [I 2022-10-28 03:46:31] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:04 [I 2022-10-28 03:46:31] ...trap/engines/engine.py.229: process: 0.57365 | load: 0.00247 [I 2022-10-28 03:46:31] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-28 03:46:31] ...optimizers/trijoint.py.151: total_norm: 8.328059 [I 2022-10-28 03:46:36] ...trap/engines/engine.py.224: train: epoch 104 | batch 1020/2304 [I 2022-10-28 03:46:36] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:58 [I 2022-10-28 03:46:36] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00249 [I 2022-10-28 03:46:36] ...trap/engines/engine.py.230: loss: 0.20802 [I 2022-10-28 03:46:36] ...optimizers/trijoint.py.151: total_norm: 23.420265 [I 2022-10-28 03:46:42] ...trap/engines/engine.py.224: train: epoch 104 | batch 1030/2304 [I 2022-10-28 03:46:42] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:46 [I 2022-10-28 03:46:42] ...trap/engines/engine.py.229: process: 0.54253 | load: 0.00245 [I 2022-10-28 03:46:42] ...trap/engines/engine.py.230: loss: 0.16127 [I 2022-10-28 03:46:42] ...optimizers/trijoint.py.151: total_norm: 15.010278 [I 2022-10-28 03:46:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 1040/2304 [I 2022-10-28 03:46:47] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:48 [I 2022-10-28 03:46:47] ...trap/engines/engine.py.229: process: 0.55480 | load: 0.00243 [I 2022-10-28 03:46:47] ...trap/engines/engine.py.230: loss: 0.14666 [I 2022-10-28 03:46:47] ...optimizers/trijoint.py.151: total_norm: 9.475594 [I 2022-10-28 03:46:53] ...trap/engines/engine.py.224: train: epoch 104 | batch 1050/2304 [I 2022-10-28 03:46:53] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:35 [I 2022-10-28 03:46:53] ...trap/engines/engine.py.229: process: 0.55346 | load: 0.00245 [I 2022-10-28 03:46:53] ...trap/engines/engine.py.230: loss: 0.11701 [I 2022-10-28 03:46:53] ...optimizers/trijoint.py.151: total_norm: 20.364382 [I 2022-10-28 03:46:59] ...trap/engines/engine.py.224: train: epoch 104 | batch 1060/2304 [I 2022-10-28 03:46:59] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:49 [I 2022-10-28 03:46:59] ...trap/engines/engine.py.229: process: 0.55789 | load: 0.00250 [I 2022-10-28 03:46:59] ...trap/engines/engine.py.230: loss: 0.12328 [I 2022-10-28 03:46:59] ...optimizers/trijoint.py.151: total_norm: 17.734844 [I 2022-10-28 03:47:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1070/2304 [I 2022-10-28 03:47:04] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:32 [I 2022-10-28 03:47:04] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00245 [I 2022-10-28 03:47:04] ...trap/engines/engine.py.230: loss: 0.19435 [I 2022-10-28 03:47:04] ...optimizers/trijoint.py.151: total_norm: 23.293726 [I 2022-10-28 03:47:10] ...trap/engines/engine.py.224: train: epoch 104 | batch 1080/2304 [I 2022-10-28 03:47:10] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:21 [I 2022-10-28 03:47:10] ...trap/engines/engine.py.229: process: 0.55306 | load: 0.00249 [I 2022-10-28 03:47:10] ...trap/engines/engine.py.230: loss: 0.15853 [I 2022-10-28 03:47:10] ...optimizers/trijoint.py.151: total_norm: 17.185101 [I 2022-10-28 03:47:16] ...trap/engines/engine.py.224: train: epoch 104 | batch 1090/2304 [I 2022-10-28 03:47:16] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:16 [I 2022-10-28 03:47:16] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00248 [I 2022-10-28 03:47:16] ...trap/engines/engine.py.230: loss: 0.10647 [I 2022-10-28 03:47:16] ...optimizers/trijoint.py.151: total_norm: 11.296720 [I 2022-10-28 03:47:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1100/2304 [I 2022-10-28 03:47:21] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:09 [I 2022-10-28 03:47:21] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00247 [I 2022-10-28 03:47:21] ...trap/engines/engine.py.230: loss: 0.16055 [I 2022-10-28 03:47:21] ...optimizers/trijoint.py.151: total_norm: 12.391234 [I 2022-10-28 03:47:27] ...trap/engines/engine.py.224: train: epoch 104 | batch 1110/2304 [I 2022-10-28 03:47:27] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:08 [I 2022-10-28 03:47:27] ...trap/engines/engine.py.229: process: 0.56002 | load: 0.00248 [I 2022-10-28 03:47:27] ...trap/engines/engine.py.230: loss: 0.11433 [I 2022-10-28 03:47:27] ...optimizers/trijoint.py.151: total_norm: 15.020516 [I 2022-10-28 03:47:33] ...trap/engines/engine.py.224: train: epoch 104 | batch 1120/2304 [I 2022-10-28 03:47:33] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:42 [I 2022-10-28 03:47:33] ...trap/engines/engine.py.229: process: 0.63583 | load: 0.00270 [I 2022-10-28 03:47:33] ...trap/engines/engine.py.230: loss: 0.09012 [I 2022-10-28 03:47:33] ...optimizers/trijoint.py.151: total_norm: 14.370169 [I 2022-10-28 03:47:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1130/2304 [I 2022-10-28 03:47:38] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:11:02 [I 2022-10-28 03:47:38] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00243 [I 2022-10-28 03:47:38] ...trap/engines/engine.py.230: loss: 0.10361 [I 2022-10-28 03:47:38] ...optimizers/trijoint.py.151: total_norm: 12.480592 [I 2022-10-28 03:47:44] ...trap/engines/engine.py.224: train: epoch 104 | batch 1140/2304 [I 2022-10-28 03:47:44] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:10:49 [I 2022-10-28 03:47:44] ...trap/engines/engine.py.229: process: 0.56543 | load: 0.00243 [I 2022-10-28 03:47:44] ...trap/engines/engine.py.230: loss: 0.12427 [I 2022-10-28 03:47:44] ...optimizers/trijoint.py.151: total_norm: 20.612856 [I 2022-10-28 03:47:49] ...trap/engines/engine.py.224: train: epoch 104 | batch 1150/2304 [I 2022-10-28 03:47:49] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:43 [I 2022-10-28 03:47:49] ...trap/engines/engine.py.229: process: 0.55695 | load: 0.00245 [I 2022-10-28 03:47:49] ...trap/engines/engine.py.230: loss: 0.09869 [I 2022-10-28 03:47:49] ...optimizers/trijoint.py.151: total_norm: 13.742641 [I 2022-10-28 03:47:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1160/2304 [I 2022-10-28 03:47:55] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:41 [I 2022-10-28 03:47:55] ...trap/engines/engine.py.229: process: 0.56239 | load: 0.00248 [I 2022-10-28 03:47:55] ...trap/engines/engine.py.230: loss: 0.10236 [I 2022-10-28 03:47:55] ...optimizers/trijoint.py.151: total_norm: 12.255022 [I 2022-10-28 03:48:00] ...trap/engines/engine.py.224: train: epoch 104 | batch 1170/2304 [I 2022-10-28 03:48:00] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:29 [I 2022-10-28 03:48:00] ...trap/engines/engine.py.229: process: 0.54855 | load: 0.00248 [I 2022-10-28 03:48:00] ...trap/engines/engine.py.230: loss: 0.14776 [I 2022-10-28 03:48:00] ...optimizers/trijoint.py.151: total_norm: 14.597682 [I 2022-10-28 03:48:06] ...trap/engines/engine.py.224: train: epoch 104 | batch 1180/2304 [I 2022-10-28 03:48:06] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:33 [I 2022-10-28 03:48:06] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00242 [I 2022-10-28 03:48:06] ...trap/engines/engine.py.230: loss: 0.10304 [I 2022-10-28 03:48:06] ...optimizers/trijoint.py.151: total_norm: 11.902937 [I 2022-10-28 03:48:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 1190/2304 [I 2022-10-28 03:48:12] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:28 [I 2022-10-28 03:48:12] ...trap/engines/engine.py.229: process: 0.56850 | load: 0.00247 [I 2022-10-28 03:48:12] ...trap/engines/engine.py.230: loss: 0.13166 [I 2022-10-28 03:48:12] ...optimizers/trijoint.py.151: total_norm: 13.061104 [I 2022-10-28 03:48:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 1200/2304 [I 2022-10-28 03:48:17] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:19 [I 2022-10-28 03:48:17] ...trap/engines/engine.py.229: process: 0.54145 | load: 0.00242 [I 2022-10-28 03:48:17] ...trap/engines/engine.py.230: loss: 0.14379 [I 2022-10-28 03:48:17] ...optimizers/trijoint.py.151: total_norm: 16.694317 [I 2022-10-28 03:48:23] ...trap/engines/engine.py.224: train: epoch 104 | batch 1210/2304 [I 2022-10-28 03:48:23] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:12 [I 2022-10-28 03:48:23] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00243 [I 2022-10-28 03:48:23] ...trap/engines/engine.py.230: loss: 0.17002 [I 2022-10-28 03:48:23] ...optimizers/trijoint.py.151: total_norm: 33.771351 [I 2022-10-28 03:48:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 1220/2304 [I 2022-10-28 03:48:29] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:11 [I 2022-10-28 03:48:29] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00244 [I 2022-10-28 03:48:29] ...trap/engines/engine.py.230: loss: 0.13038 [I 2022-10-28 03:48:29] ...optimizers/trijoint.py.151: total_norm: 10.059029 [I 2022-10-28 03:48:34] ...trap/engines/engine.py.224: train: epoch 104 | batch 1230/2304 [I 2022-10-28 03:48:34] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:10:05 [I 2022-10-28 03:48:34] ...trap/engines/engine.py.229: process: 0.57487 | load: 0.00243 [I 2022-10-28 03:48:34] ...trap/engines/engine.py.230: loss: 0.11482 [I 2022-10-28 03:48:34] ...optimizers/trijoint.py.151: total_norm: 8.634435 [I 2022-10-28 03:48:40] ...trap/engines/engine.py.224: train: epoch 104 | batch 1240/2304 [I 2022-10-28 03:48:40] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:53 [I 2022-10-28 03:48:40] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00244 [I 2022-10-28 03:48:40] ...trap/engines/engine.py.230: loss: 0.09062 [I 2022-10-28 03:48:40] ...optimizers/trijoint.py.151: total_norm: 9.888592 [I 2022-10-28 03:48:46] ...trap/engines/engine.py.224: train: epoch 104 | batch 1250/2304 [I 2022-10-28 03:48:46] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:51 [I 2022-10-28 03:48:46] ...trap/engines/engine.py.229: process: 0.56936 | load: 0.00247 [I 2022-10-28 03:48:46] ...trap/engines/engine.py.230: loss: 0.11825 [I 2022-10-28 03:48:46] ...optimizers/trijoint.py.151: total_norm: 14.919502 [I 2022-10-28 03:48:51] ...trap/engines/engine.py.224: train: epoch 104 | batch 1260/2304 [I 2022-10-28 03:48:51] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:44 [I 2022-10-28 03:48:51] ...trap/engines/engine.py.229: process: 0.56441 | load: 0.00246 [I 2022-10-28 03:48:51] ...trap/engines/engine.py.230: loss: 0.13901 [I 2022-10-28 03:48:51] ...optimizers/trijoint.py.151: total_norm: 10.242111 [I 2022-10-28 03:48:57] ...trap/engines/engine.py.224: train: epoch 104 | batch 1270/2304 [I 2022-10-28 03:48:57] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:42 [I 2022-10-28 03:48:57] ...trap/engines/engine.py.229: process: 0.57112 | load: 0.00245 [I 2022-10-28 03:48:57] ...trap/engines/engine.py.230: loss: 0.14391 [I 2022-10-28 03:48:57] ...optimizers/trijoint.py.151: total_norm: 13.912543 [I 2022-10-28 03:49:02] ...trap/engines/engine.py.224: train: epoch 104 | batch 1280/2304 [I 2022-10-28 03:49:02] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:34 [I 2022-10-28 03:49:02] ...trap/engines/engine.py.229: process: 0.55669 | load: 0.00251 [I 2022-10-28 03:49:02] ...trap/engines/engine.py.230: loss: 0.11218 [I 2022-10-28 03:49:02] ...optimizers/trijoint.py.151: total_norm: 15.084229 [I 2022-10-28 03:49:08] ...trap/engines/engine.py.224: train: epoch 104 | batch 1290/2304 [I 2022-10-28 03:49:08] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:28 [I 2022-10-28 03:49:08] ...trap/engines/engine.py.229: process: 0.56056 | load: 0.00243 [I 2022-10-28 03:49:08] ...trap/engines/engine.py.230: loss: 0.13888 [I 2022-10-28 03:49:08] ...optimizers/trijoint.py.151: total_norm: 9.640492 [I 2022-10-28 03:49:14] ...trap/engines/engine.py.224: train: epoch 104 | batch 1300/2304 [I 2022-10-28 03:49:14] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:19 [I 2022-10-28 03:49:14] ...trap/engines/engine.py.229: process: 0.55557 | load: 0.00242 [I 2022-10-28 03:49:14] ...trap/engines/engine.py.230: loss: 0.11503 [I 2022-10-28 03:49:14] ...optimizers/trijoint.py.151: total_norm: 12.860983 [I 2022-10-28 03:49:19] ...trap/engines/engine.py.224: train: epoch 104 | batch 1310/2304 [I 2022-10-28 03:49:19] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:35 [I 2022-10-28 03:49:19] ...trap/engines/engine.py.229: process: 0.56723 | load: 0.00245 [I 2022-10-28 03:49:19] ...trap/engines/engine.py.230: loss: 0.09510 [I 2022-10-28 03:49:19] ...optimizers/trijoint.py.151: total_norm: 23.281193 [I 2022-10-28 03:49:25] ...trap/engines/engine.py.224: train: epoch 104 | batch 1320/2304 [I 2022-10-28 03:49:25] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:07 [I 2022-10-28 03:49:25] ...trap/engines/engine.py.229: process: 0.54270 | load: 0.00243 [I 2022-10-28 03:49:25] ...trap/engines/engine.py.230: loss: 0.11478 [I 2022-10-28 03:49:25] ...optimizers/trijoint.py.151: total_norm: 15.822477 [I 2022-10-28 03:49:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 1330/2304 [I 2022-10-28 03:49:30] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:09:02 [I 2022-10-28 03:49:30] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00304 [I 2022-10-28 03:49:30] ...trap/engines/engine.py.230: loss: 0.12178 [I 2022-10-28 03:49:30] ...optimizers/trijoint.py.151: total_norm: 16.724012 [I 2022-10-28 03:49:36] ...trap/engines/engine.py.224: train: epoch 104 | batch 1340/2304 [I 2022-10-28 03:49:36] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:09:01 [I 2022-10-28 03:49:36] ...trap/engines/engine.py.229: process: 0.56296 | load: 0.00278 [I 2022-10-28 03:49:36] ...trap/engines/engine.py.230: loss: 0.11097 [I 2022-10-28 03:49:36] ...optimizers/trijoint.py.151: total_norm: 11.946648 [I 2022-10-28 03:49:42] ...trap/engines/engine.py.224: train: epoch 104 | batch 1350/2304 [I 2022-10-28 03:49:42] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:08:56 [I 2022-10-28 03:49:42] ...trap/engines/engine.py.229: process: 0.57458 | load: 0.00262 [I 2022-10-28 03:49:42] ...trap/engines/engine.py.230: loss: 0.09268 [I 2022-10-28 03:49:42] ...optimizers/trijoint.py.151: total_norm: 10.164956 [I 2022-10-28 03:49:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 1360/2304 [I 2022-10-28 03:49:47] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:47 [I 2022-10-28 03:49:47] ...trap/engines/engine.py.229: process: 0.55471 | load: 0.00256 [I 2022-10-28 03:49:47] ...trap/engines/engine.py.230: loss: 0.10089 [I 2022-10-28 03:49:47] ...optimizers/trijoint.py.151: total_norm: 15.634879 [I 2022-10-28 03:49:53] ...trap/engines/engine.py.224: train: epoch 104 | batch 1370/2304 [I 2022-10-28 03:49:53] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:41 [I 2022-10-28 03:49:53] ...trap/engines/engine.py.229: process: 0.55438 | load: 0.00249 [I 2022-10-28 03:49:53] ...trap/engines/engine.py.230: loss: 0.14413 [I 2022-10-28 03:49:53] ...optimizers/trijoint.py.151: total_norm: 21.622364 [I 2022-10-28 03:49:59] ...trap/engines/engine.py.224: train: epoch 104 | batch 1380/2304 [I 2022-10-28 03:49:59] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:38 [I 2022-10-28 03:49:59] ...trap/engines/engine.py.229: process: 0.55383 | load: 0.00241 [I 2022-10-28 03:49:59] ...trap/engines/engine.py.230: loss: 0.09501 [I 2022-10-28 03:49:59] ...optimizers/trijoint.py.151: total_norm: 12.850934 [I 2022-10-28 03:50:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1390/2304 [I 2022-10-28 03:50:04] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:32 [I 2022-10-28 03:50:04] ...trap/engines/engine.py.229: process: 0.57401 | load: 0.00242 [I 2022-10-28 03:50:04] ...trap/engines/engine.py.230: loss: 0.12134 [I 2022-10-28 03:50:04] ...optimizers/trijoint.py.151: total_norm: 17.726225 [I 2022-10-28 03:50:10] ...trap/engines/engine.py.224: train: epoch 104 | batch 1400/2304 [I 2022-10-28 03:50:10] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:08:26 [I 2022-10-28 03:50:10] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00243 [I 2022-10-28 03:50:10] ...trap/engines/engine.py.230: loss: 0.14598 [I 2022-10-28 03:50:10] ...optimizers/trijoint.py.151: total_norm: 17.401894 [I 2022-10-28 03:50:15] ...trap/engines/engine.py.224: train: epoch 104 | batch 1410/2304 [I 2022-10-28 03:50:15] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:19 [I 2022-10-28 03:50:15] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00246 [I 2022-10-28 03:50:15] ...trap/engines/engine.py.230: loss: 0.15777 [I 2022-10-28 03:50:15] ...optimizers/trijoint.py.151: total_norm: 16.722149 [I 2022-10-28 03:50:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1420/2304 [I 2022-10-28 03:50:21] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:11 [I 2022-10-28 03:50:21] ...trap/engines/engine.py.229: process: 0.55067 | load: 0.00246 [I 2022-10-28 03:50:21] ...trap/engines/engine.py.230: loss: 0.13494 [I 2022-10-28 03:50:21] ...optimizers/trijoint.py.151: total_norm: 12.889814 [I 2022-10-28 03:50:27] ...trap/engines/engine.py.224: train: epoch 104 | batch 1430/2304 [I 2022-10-28 03:50:27] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:13 [I 2022-10-28 03:50:27] ...trap/engines/engine.py.229: process: 0.54468 | load: 0.00240 [I 2022-10-28 03:50:27] ...trap/engines/engine.py.230: loss: 0.16535 [I 2022-10-28 03:50:27] ...optimizers/trijoint.py.151: total_norm: 27.001173 [I 2022-10-28 03:50:32] ...trap/engines/engine.py.224: train: epoch 104 | batch 1440/2304 [I 2022-10-28 03:50:32] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:08:00 [I 2022-10-28 03:50:32] ...trap/engines/engine.py.229: process: 0.55669 | load: 0.00248 [I 2022-10-28 03:50:32] ...trap/engines/engine.py.230: loss: 0.08896 [I 2022-10-28 03:50:32] ...optimizers/trijoint.py.151: total_norm: 16.678978 [I 2022-10-28 03:50:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1450/2304 [I 2022-10-28 03:50:38] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:07:55 [I 2022-10-28 03:50:38] ...trap/engines/engine.py.229: process: 0.55153 | load: 0.00242 [I 2022-10-28 03:50:38] ...trap/engines/engine.py.230: loss: 0.14489 [I 2022-10-28 03:50:38] ...optimizers/trijoint.py.151: total_norm: 8.231031 [I 2022-10-28 03:50:43] ...trap/engines/engine.py.224: train: epoch 104 | batch 1460/2304 [I 2022-10-28 03:50:43] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:49 [I 2022-10-28 03:50:43] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00243 [I 2022-10-28 03:50:43] ...trap/engines/engine.py.230: loss: 0.18463 [I 2022-10-28 03:50:43] ...optimizers/trijoint.py.151: total_norm: 17.413445 [I 2022-10-28 03:50:49] ...trap/engines/engine.py.224: train: epoch 104 | batch 1470/2304 [I 2022-10-28 03:50:49] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:47 [I 2022-10-28 03:50:49] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00246 [I 2022-10-28 03:50:49] ...trap/engines/engine.py.230: loss: 0.12230 [I 2022-10-28 03:50:49] ...optimizers/trijoint.py.151: total_norm: 11.807595 [I 2022-10-28 03:50:54] ...trap/engines/engine.py.224: train: epoch 104 | batch 1480/2304 [I 2022-10-28 03:50:54] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:36 [I 2022-10-28 03:50:54] ...trap/engines/engine.py.229: process: 0.54089 | load: 0.00244 [I 2022-10-28 03:50:54] ...trap/engines/engine.py.230: loss: 0.12963 [I 2022-10-28 03:50:54] ...optimizers/trijoint.py.151: total_norm: 8.041771 [I 2022-10-28 03:51:00] ...trap/engines/engine.py.224: train: epoch 104 | batch 1490/2304 [I 2022-10-28 03:51:00] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:31 [I 2022-10-28 03:51:00] ...trap/engines/engine.py.229: process: 0.54709 | load: 0.00241 [I 2022-10-28 03:51:00] ...trap/engines/engine.py.230: loss: 0.09211 [I 2022-10-28 03:51:00] ...optimizers/trijoint.py.151: total_norm: 9.945603 [I 2022-10-28 03:51:05] ...trap/engines/engine.py.224: train: epoch 104 | batch 1500/2304 [I 2022-10-28 03:51:05] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:07:25 [I 2022-10-28 03:51:05] ...trap/engines/engine.py.229: process: 0.55216 | load: 0.00244 [I 2022-10-28 03:51:05] ...trap/engines/engine.py.230: loss: 0.06889 [I 2022-10-28 03:51:05] ...optimizers/trijoint.py.151: total_norm: 9.922798 [I 2022-10-28 03:51:11] ...trap/engines/engine.py.224: train: epoch 104 | batch 1510/2304 [I 2022-10-28 03:51:11] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:19 [I 2022-10-28 03:51:11] ...trap/engines/engine.py.229: process: 0.55440 | load: 0.00241 [I 2022-10-28 03:51:11] ...trap/engines/engine.py.230: loss: 0.11341 [I 2022-10-28 03:51:11] ...optimizers/trijoint.py.151: total_norm: 10.376787 [I 2022-10-28 03:51:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 1520/2304 [I 2022-10-28 03:51:17] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:07:17 [I 2022-10-28 03:51:17] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00243 [I 2022-10-28 03:51:17] ...trap/engines/engine.py.230: loss: 0.13965 [I 2022-10-28 03:51:17] ...optimizers/trijoint.py.151: total_norm: 13.992606 [I 2022-10-28 03:51:22] ...trap/engines/engine.py.224: train: epoch 104 | batch 1530/2304 [I 2022-10-28 03:51:22] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:19 [I 2022-10-28 03:51:22] ...trap/engines/engine.py.229: process: 0.55458 | load: 0.00249 [I 2022-10-28 03:51:22] ...trap/engines/engine.py.230: loss: 0.09033 [I 2022-10-28 03:51:22] ...optimizers/trijoint.py.151: total_norm: 15.516439 [I 2022-10-28 03:51:28] ...trap/engines/engine.py.224: train: epoch 104 | batch 1540/2304 [I 2022-10-28 03:51:28] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:03 [I 2022-10-28 03:51:28] ...trap/engines/engine.py.229: process: 0.54667 | load: 0.00244 [I 2022-10-28 03:51:28] ...trap/engines/engine.py.230: loss: 0.12215 [I 2022-10-28 03:51:28] ...optimizers/trijoint.py.151: total_norm: 34.721210 [I 2022-10-28 03:51:33] ...trap/engines/engine.py.224: train: epoch 104 | batch 1550/2304 [I 2022-10-28 03:51:33] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:07:02 [I 2022-10-28 03:51:33] ...trap/engines/engine.py.229: process: 0.56539 | load: 0.00243 [I 2022-10-28 03:51:33] ...trap/engines/engine.py.230: loss: 0.17744 [I 2022-10-28 03:51:33] ...optimizers/trijoint.py.151: total_norm: 18.943802 [I 2022-10-28 03:51:39] ...trap/engines/engine.py.224: train: epoch 104 | batch 1560/2304 [I 2022-10-28 03:51:39] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:55 [I 2022-10-28 03:51:39] ...trap/engines/engine.py.229: process: 0.54323 | load: 0.00247 [I 2022-10-28 03:51:39] ...trap/engines/engine.py.230: loss: 0.20253 [I 2022-10-28 03:51:39] ...optimizers/trijoint.py.151: total_norm: 10.990519 [I 2022-10-28 03:51:45] ...trap/engines/engine.py.224: train: epoch 104 | batch 1570/2304 [I 2022-10-28 03:51:45] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:06:52 [I 2022-10-28 03:51:45] ...trap/engines/engine.py.229: process: 0.56224 | load: 0.00246 [I 2022-10-28 03:51:45] ...trap/engines/engine.py.230: loss: 0.10233 [I 2022-10-28 03:51:45] ...optimizers/trijoint.py.151: total_norm: 19.482540 [I 2022-10-28 03:51:50] ...trap/engines/engine.py.224: train: epoch 104 | batch 1580/2304 [I 2022-10-28 03:51:50] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:41 [I 2022-10-28 03:51:50] ...trap/engines/engine.py.229: process: 0.55005 | load: 0.00245 [I 2022-10-28 03:51:50] ...trap/engines/engine.py.230: loss: 0.19437 [I 2022-10-28 03:51:50] ...optimizers/trijoint.py.151: total_norm: 26.736750 [I 2022-10-28 03:51:56] ...trap/engines/engine.py.224: train: epoch 104 | batch 1590/2304 [I 2022-10-28 03:51:56] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:37 [I 2022-10-28 03:51:56] ...trap/engines/engine.py.229: process: 0.55131 | load: 0.00244 [I 2022-10-28 03:51:56] ...trap/engines/engine.py.230: loss: 0.11095 [I 2022-10-28 03:51:56] ...optimizers/trijoint.py.151: total_norm: 25.256014 [I 2022-10-28 03:52:01] ...trap/engines/engine.py.224: train: epoch 104 | batch 1600/2304 [I 2022-10-28 03:52:01] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:32 [I 2022-10-28 03:52:01] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00243 [I 2022-10-28 03:52:01] ...trap/engines/engine.py.230: loss: 0.09865 [I 2022-10-28 03:52:01] ...optimizers/trijoint.py.151: total_norm: 15.741424 [I 2022-10-28 03:52:07] ...trap/engines/engine.py.224: train: epoch 104 | batch 1610/2304 [I 2022-10-28 03:52:07] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:27 [I 2022-10-28 03:52:07] ...trap/engines/engine.py.229: process: 0.55539 | load: 0.00244 [I 2022-10-28 03:52:07] ...trap/engines/engine.py.230: loss: 0.16810 [I 2022-10-28 03:52:07] ...optimizers/trijoint.py.151: total_norm: 20.235826 [I 2022-10-28 03:52:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 1620/2304 [I 2022-10-28 03:52:13] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:22 [I 2022-10-28 03:52:13] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00241 [I 2022-10-28 03:52:13] ...trap/engines/engine.py.230: loss: 0.08225 [I 2022-10-28 03:52:13] ...optimizers/trijoint.py.151: total_norm: 8.027291 [I 2022-10-28 03:52:18] ...trap/engines/engine.py.224: train: epoch 104 | batch 1630/2304 [I 2022-10-28 03:52:18] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:18 [I 2022-10-28 03:52:18] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00242 [I 2022-10-28 03:52:18] ...trap/engines/engine.py.230: loss: 0.14138 [I 2022-10-28 03:52:18] ...optimizers/trijoint.py.151: total_norm: 14.941782 [I 2022-10-28 03:52:24] ...trap/engines/engine.py.224: train: epoch 104 | batch 1640/2304 [I 2022-10-28 03:52:24] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:11 [I 2022-10-28 03:52:24] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00303 [I 2022-10-28 03:52:24] ...trap/engines/engine.py.230: loss: 0.11311 [I 2022-10-28 03:52:24] ...optimizers/trijoint.py.151: total_norm: 23.295683 [I 2022-10-28 03:52:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 1650/2304 [I 2022-10-28 03:52:29] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:04 [I 2022-10-28 03:52:29] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00242 [I 2022-10-28 03:52:29] ...trap/engines/engine.py.230: loss: 0.10307 [I 2022-10-28 03:52:29] ...optimizers/trijoint.py.151: total_norm: 13.733643 [I 2022-10-28 03:52:35] ...trap/engines/engine.py.224: train: epoch 104 | batch 1660/2304 [I 2022-10-28 03:52:35] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:05:56 [I 2022-10-28 03:52:35] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00239 [I 2022-10-28 03:52:35] ...trap/engines/engine.py.230: loss: 0.17079 [I 2022-10-28 03:52:35] ...optimizers/trijoint.py.151: total_norm: 15.130303 [I 2022-10-28 03:52:41] ...trap/engines/engine.py.224: train: epoch 104 | batch 1670/2304 [I 2022-10-28 03:52:41] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:51 [I 2022-10-28 03:52:41] ...trap/engines/engine.py.229: process: 0.54182 | load: 0.00243 [I 2022-10-28 03:52:41] ...trap/engines/engine.py.230: loss: 0.10379 [I 2022-10-28 03:52:41] ...optimizers/trijoint.py.151: total_norm: 11.095469 [I 2022-10-28 03:52:46] ...trap/engines/engine.py.224: train: epoch 104 | batch 1680/2304 [I 2022-10-28 03:52:46] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:45 [I 2022-10-28 03:52:46] ...trap/engines/engine.py.229: process: 0.54854 | load: 0.00242 [I 2022-10-28 03:52:46] ...trap/engines/engine.py.230: loss: 0.16104 [I 2022-10-28 03:52:46] ...optimizers/trijoint.py.151: total_norm: 17.269331 [I 2022-10-28 03:52:52] ...trap/engines/engine.py.224: train: epoch 104 | batch 1690/2304 [I 2022-10-28 03:52:52] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:43 [I 2022-10-28 03:52:52] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00244 [I 2022-10-28 03:52:52] ...trap/engines/engine.py.230: loss: 0.12873 [I 2022-10-28 03:52:52] ...optimizers/trijoint.py.151: total_norm: 16.180044 [I 2022-10-28 03:52:57] ...trap/engines/engine.py.224: train: epoch 104 | batch 1700/2304 [I 2022-10-28 03:52:57] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:39 [I 2022-10-28 03:52:57] ...trap/engines/engine.py.229: process: 0.56002 | load: 0.00245 [I 2022-10-28 03:52:57] ...trap/engines/engine.py.230: loss: 0.08194 [I 2022-10-28 03:52:57] ...optimizers/trijoint.py.151: total_norm: 10.172106 [I 2022-10-28 03:53:03] ...trap/engines/engine.py.224: train: epoch 104 | batch 1710/2304 [I 2022-10-28 03:53:03] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:32 [I 2022-10-28 03:53:03] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00240 [I 2022-10-28 03:53:03] ...trap/engines/engine.py.230: loss: 0.07363 [I 2022-10-28 03:53:03] ...optimizers/trijoint.py.151: total_norm: 10.218859 [I 2022-10-28 03:53:09] ...trap/engines/engine.py.224: train: epoch 104 | batch 1720/2304 [I 2022-10-28 03:53:09] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:25 [I 2022-10-28 03:53:09] ...trap/engines/engine.py.229: process: 0.54729 | load: 0.00244 [I 2022-10-28 03:53:09] ...trap/engines/engine.py.230: loss: 0.12595 [I 2022-10-28 03:53:09] ...optimizers/trijoint.py.151: total_norm: 23.525320 [I 2022-10-28 03:53:14] ...trap/engines/engine.py.224: train: epoch 104 | batch 1730/2304 [I 2022-10-28 03:53:14] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:20 [I 2022-10-28 03:53:14] ...trap/engines/engine.py.229: process: 0.55348 | load: 0.00241 [I 2022-10-28 03:53:14] ...trap/engines/engine.py.230: loss: 0.14632 [I 2022-10-28 03:53:14] ...optimizers/trijoint.py.151: total_norm: 13.021167 [I 2022-10-28 03:53:20] ...trap/engines/engine.py.224: train: epoch 104 | batch 1740/2304 [I 2022-10-28 03:53:20] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:15 [I 2022-10-28 03:53:20] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00243 [I 2022-10-28 03:53:20] ...trap/engines/engine.py.230: loss: 0.16267 [I 2022-10-28 03:53:20] ...optimizers/trijoint.py.151: total_norm: 13.259838 [I 2022-10-28 03:53:25] ...trap/engines/engine.py.224: train: epoch 104 | batch 1750/2304 [I 2022-10-28 03:53:25] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:08 [I 2022-10-28 03:53:25] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00241 [I 2022-10-28 03:53:25] ...trap/engines/engine.py.230: loss: 0.18634 [I 2022-10-28 03:53:25] ...optimizers/trijoint.py.151: total_norm: 82.606918 [I 2022-10-28 03:53:31] ...trap/engines/engine.py.224: train: epoch 104 | batch 1760/2304 [I 2022-10-28 03:53:31] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:04 [I 2022-10-28 03:53:31] ...trap/engines/engine.py.229: process: 0.55192 | load: 0.00242 [I 2022-10-28 03:53:31] ...trap/engines/engine.py.230: loss: 0.10064 [I 2022-10-28 03:53:31] ...optimizers/trijoint.py.151: total_norm: 11.890993 [I 2022-10-28 03:53:37] ...trap/engines/engine.py.224: train: epoch 104 | batch 1770/2304 [I 2022-10-28 03:53:37] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:04:56 [I 2022-10-28 03:53:37] ...trap/engines/engine.py.229: process: 0.54789 | load: 0.00241 [I 2022-10-28 03:53:37] ...trap/engines/engine.py.230: loss: 0.08029 [I 2022-10-28 03:53:37] ...optimizers/trijoint.py.151: total_norm: 11.082100 [I 2022-10-28 03:53:42] ...trap/engines/engine.py.224: train: epoch 104 | batch 1780/2304 [I 2022-10-28 03:53:42] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:52 [I 2022-10-28 03:53:42] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00242 [I 2022-10-28 03:53:42] ...trap/engines/engine.py.230: loss: 0.16973 [I 2022-10-28 03:53:42] ...optimizers/trijoint.py.151: total_norm: 13.598457 [I 2022-10-28 03:53:48] ...trap/engines/engine.py.224: train: epoch 104 | batch 1790/2304 [I 2022-10-28 03:53:48] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:47 [I 2022-10-28 03:53:48] ...trap/engines/engine.py.229: process: 0.56049 | load: 0.00241 [I 2022-10-28 03:53:48] ...trap/engines/engine.py.230: loss: 0.07608 [I 2022-10-28 03:53:48] ...optimizers/trijoint.py.151: total_norm: 15.779574 [I 2022-10-28 03:53:53] ...trap/engines/engine.py.224: train: epoch 104 | batch 1800/2304 [I 2022-10-28 03:53:53] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:41 [I 2022-10-28 03:53:53] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00247 [I 2022-10-28 03:53:53] ...trap/engines/engine.py.230: loss: 0.13685 [I 2022-10-28 03:53:53] ...optimizers/trijoint.py.151: total_norm: 11.444010 [I 2022-10-28 03:53:59] ...trap/engines/engine.py.224: train: epoch 104 | batch 1810/2304 [I 2022-10-28 03:53:59] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:35 [I 2022-10-28 03:53:59] ...trap/engines/engine.py.229: process: 0.56445 | load: 0.00249 [I 2022-10-28 03:53:59] ...trap/engines/engine.py.230: loss: 0.09431 [I 2022-10-28 03:53:59] ...optimizers/trijoint.py.151: total_norm: 19.096048 [I 2022-10-28 03:54:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 1820/2304 [I 2022-10-28 03:54:04] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:29 [I 2022-10-28 03:54:04] ...trap/engines/engine.py.229: process: 0.54620 | load: 0.00242 [I 2022-10-28 03:54:04] ...trap/engines/engine.py.230: loss: 0.08120 [I 2022-10-28 03:54:04] ...optimizers/trijoint.py.151: total_norm: 10.009822 [I 2022-10-28 03:54:10] ...trap/engines/engine.py.224: train: epoch 104 | batch 1830/2304 [I 2022-10-28 03:54:10] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:23 [I 2022-10-28 03:54:10] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00243 [I 2022-10-28 03:54:10] ...trap/engines/engine.py.230: loss: 0.11681 [I 2022-10-28 03:54:10] ...optimizers/trijoint.py.151: total_norm: 24.404692 [I 2022-10-28 03:54:16] ...trap/engines/engine.py.224: train: epoch 104 | batch 1840/2304 [I 2022-10-28 03:54:16] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:19 [I 2022-10-28 03:54:16] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00242 [I 2022-10-28 03:54:16] ...trap/engines/engine.py.230: loss: 0.08883 [I 2022-10-28 03:54:16] ...optimizers/trijoint.py.151: total_norm: 10.321461 [I 2022-10-28 03:54:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 1850/2304 [I 2022-10-28 03:54:21] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:13 [I 2022-10-28 03:54:21] ...trap/engines/engine.py.229: process: 0.56225 | load: 0.00243 [I 2022-10-28 03:54:21] ...trap/engines/engine.py.230: loss: 0.14153 [I 2022-10-28 03:54:21] ...optimizers/trijoint.py.151: total_norm: 26.377352 [I 2022-10-28 03:54:27] ...trap/engines/engine.py.224: train: epoch 104 | batch 1860/2304 [I 2022-10-28 03:54:27] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:09 [I 2022-10-28 03:54:27] ...trap/engines/engine.py.229: process: 0.56179 | load: 0.00242 [I 2022-10-28 03:54:27] ...trap/engines/engine.py.230: loss: 0.13014 [I 2022-10-28 03:54:27] ...optimizers/trijoint.py.151: total_norm: 19.817839 [I 2022-10-28 03:54:33] ...trap/engines/engine.py.224: train: epoch 104 | batch 1870/2304 [I 2022-10-28 03:54:33] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:04 [I 2022-10-28 03:54:33] ...trap/engines/engine.py.229: process: 0.57435 | load: 0.00243 [I 2022-10-28 03:54:33] ...trap/engines/engine.py.230: loss: 0.11036 [I 2022-10-28 03:54:33] ...optimizers/trijoint.py.151: total_norm: 13.264790 [I 2022-10-28 03:54:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 1880/2304 [I 2022-10-28 03:54:38] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:03:56 [I 2022-10-28 03:54:38] ...trap/engines/engine.py.229: process: 0.55357 | load: 0.00246 [I 2022-10-28 03:54:38] ...trap/engines/engine.py.230: loss: 0.10610 [I 2022-10-28 03:54:38] ...optimizers/trijoint.py.151: total_norm: 17.623632 [I 2022-10-28 03:54:44] ...trap/engines/engine.py.224: train: epoch 104 | batch 1890/2304 [I 2022-10-28 03:54:44] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:52 [I 2022-10-28 03:54:44] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00244 [I 2022-10-28 03:54:44] ...trap/engines/engine.py.230: loss: 0.13016 [I 2022-10-28 03:54:44] ...optimizers/trijoint.py.151: total_norm: 28.108025 [I 2022-10-28 03:54:49] ...trap/engines/engine.py.224: train: epoch 104 | batch 1900/2304 [I 2022-10-28 03:54:49] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:46 [I 2022-10-28 03:54:49] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00261 [I 2022-10-28 03:54:49] ...trap/engines/engine.py.230: loss: 0.09789 [I 2022-10-28 03:54:49] ...optimizers/trijoint.py.151: total_norm: 7.858206 [I 2022-10-28 03:54:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 1910/2304 [I 2022-10-28 03:54:55] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:38 [I 2022-10-28 03:54:55] ...trap/engines/engine.py.229: process: 0.54343 | load: 0.00249 [I 2022-10-28 03:54:55] ...trap/engines/engine.py.230: loss: 0.08950 [I 2022-10-28 03:54:55] ...optimizers/trijoint.py.151: total_norm: 13.936212 [I 2022-10-28 03:55:01] ...trap/engines/engine.py.224: train: epoch 104 | batch 1920/2304 [I 2022-10-28 03:55:01] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:33 [I 2022-10-28 03:55:01] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00254 [I 2022-10-28 03:55:01] ...trap/engines/engine.py.230: loss: 0.09835 [I 2022-10-28 03:55:01] ...optimizers/trijoint.py.151: total_norm: 11.829000 [I 2022-10-28 03:55:06] ...trap/engines/engine.py.224: train: epoch 104 | batch 1930/2304 [I 2022-10-28 03:55:06] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:28 [I 2022-10-28 03:55:06] ...trap/engines/engine.py.229: process: 0.54738 | load: 0.00247 [I 2022-10-28 03:55:06] ...trap/engines/engine.py.230: loss: 0.10943 [I 2022-10-28 03:55:06] ...optimizers/trijoint.py.151: total_norm: 10.477550 [I 2022-10-28 03:55:12] ...trap/engines/engine.py.224: train: epoch 104 | batch 1940/2304 [I 2022-10-28 03:55:12] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:22 [I 2022-10-28 03:55:12] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00244 [I 2022-10-28 03:55:12] ...trap/engines/engine.py.230: loss: 0.13646 [I 2022-10-28 03:55:12] ...optimizers/trijoint.py.151: total_norm: 12.277504 [I 2022-10-28 03:55:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 1950/2304 [I 2022-10-28 03:55:17] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:18 [I 2022-10-28 03:55:17] ...trap/engines/engine.py.229: process: 0.56369 | load: 0.00245 [I 2022-10-28 03:55:17] ...trap/engines/engine.py.230: loss: 0.15612 [I 2022-10-28 03:55:17] ...optimizers/trijoint.py.151: total_norm: 21.321205 [I 2022-10-28 03:55:23] ...trap/engines/engine.py.224: train: epoch 104 | batch 1960/2304 [I 2022-10-28 03:55:23] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:10 [I 2022-10-28 03:55:23] ...trap/engines/engine.py.229: process: 0.54279 | load: 0.00244 [I 2022-10-28 03:55:23] ...trap/engines/engine.py.230: loss: 0.12486 [I 2022-10-28 03:55:23] ...optimizers/trijoint.py.151: total_norm: 15.643303 [I 2022-10-28 03:55:29] ...trap/engines/engine.py.224: train: epoch 104 | batch 1970/2304 [I 2022-10-28 03:55:29] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:07 [I 2022-10-28 03:55:29] ...trap/engines/engine.py.229: process: 0.56574 | load: 0.00245 [I 2022-10-28 03:55:29] ...trap/engines/engine.py.230: loss: 0.08777 [I 2022-10-28 03:55:29] ...optimizers/trijoint.py.151: total_norm: 10.933359 [I 2022-10-28 03:55:34] ...trap/engines/engine.py.224: train: epoch 104 | batch 1980/2304 [I 2022-10-28 03:55:34] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:01 [I 2022-10-28 03:55:34] ...trap/engines/engine.py.229: process: 0.55017 | load: 0.00244 [I 2022-10-28 03:55:34] ...trap/engines/engine.py.230: loss: 0.13581 [I 2022-10-28 03:55:34] ...optimizers/trijoint.py.151: total_norm: 11.057290 [I 2022-10-28 03:55:40] ...trap/engines/engine.py.224: train: epoch 104 | batch 1990/2304 [I 2022-10-28 03:55:40] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:02:55 [I 2022-10-28 03:55:40] ...trap/engines/engine.py.229: process: 0.56702 | load: 0.00244 [I 2022-10-28 03:55:40] ...trap/engines/engine.py.230: loss: 0.07953 [I 2022-10-28 03:55:40] ...optimizers/trijoint.py.151: total_norm: 13.733750 [I 2022-10-28 03:55:45] ...trap/engines/engine.py.224: train: epoch 104 | batch 2000/2304 [I 2022-10-28 03:55:45] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:49 [I 2022-10-28 03:55:45] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00243 [I 2022-10-28 03:55:45] ...trap/engines/engine.py.230: loss: 0.14620 [I 2022-10-28 03:55:45] ...optimizers/trijoint.py.151: total_norm: 10.275111 [I 2022-10-28 03:55:51] ...trap/engines/engine.py.224: train: epoch 104 | batch 2010/2304 [I 2022-10-28 03:55:51] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:44 [I 2022-10-28 03:55:51] ...trap/engines/engine.py.229: process: 0.56246 | load: 0.00254 [I 2022-10-28 03:55:51] ...trap/engines/engine.py.230: loss: 0.11173 [I 2022-10-28 03:55:51] ...optimizers/trijoint.py.151: total_norm: 11.517378 [I 2022-10-28 03:55:56] ...trap/engines/engine.py.224: train: epoch 104 | batch 2020/2304 [I 2022-10-28 03:55:56] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:38 [I 2022-10-28 03:55:56] ...trap/engines/engine.py.229: process: 0.56355 | load: 0.00248 [I 2022-10-28 03:55:56] ...trap/engines/engine.py.230: loss: 0.14024 [I 2022-10-28 03:55:56] ...optimizers/trijoint.py.151: total_norm: 14.226672 [I 2022-10-28 03:56:02] ...trap/engines/engine.py.224: train: epoch 104 | batch 2030/2304 [I 2022-10-28 03:56:02] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:33 [I 2022-10-28 03:56:02] ...trap/engines/engine.py.229: process: 0.56594 | load: 0.00246 [I 2022-10-28 03:56:02] ...trap/engines/engine.py.230: loss: 0.11645 [I 2022-10-28 03:56:02] ...optimizers/trijoint.py.151: total_norm: 14.319848 [I 2022-10-28 03:56:08] ...trap/engines/engine.py.224: train: epoch 104 | batch 2040/2304 [I 2022-10-28 03:56:08] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:28 [I 2022-10-28 03:56:08] ...trap/engines/engine.py.229: process: 0.56880 | load: 0.00246 [I 2022-10-28 03:56:08] ...trap/engines/engine.py.230: loss: 0.23813 [I 2022-10-28 03:56:08] ...optimizers/trijoint.py.151: total_norm: 29.437695 [I 2022-10-28 03:56:13] ...trap/engines/engine.py.224: train: epoch 104 | batch 2050/2304 [I 2022-10-28 03:56:13] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:22 [I 2022-10-28 03:56:13] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00249 [I 2022-10-28 03:56:13] ...trap/engines/engine.py.230: loss: 0.10029 [I 2022-10-28 03:56:13] ...optimizers/trijoint.py.151: total_norm: 12.571071 [I 2022-10-28 03:56:19] ...trap/engines/engine.py.224: train: epoch 104 | batch 2060/2304 [I 2022-10-28 03:56:19] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:15 [I 2022-10-28 03:56:19] ...trap/engines/engine.py.229: process: 0.54561 | load: 0.00253 [I 2022-10-28 03:56:19] ...trap/engines/engine.py.230: loss: 0.09925 [I 2022-10-28 03:56:19] ...optimizers/trijoint.py.151: total_norm: 11.768315 [I 2022-10-28 03:56:25] ...trap/engines/engine.py.224: train: epoch 104 | batch 2070/2304 [I 2022-10-28 03:56:25] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:10 [I 2022-10-28 03:56:25] ...trap/engines/engine.py.229: process: 0.54735 | load: 0.00254 [I 2022-10-28 03:56:25] ...trap/engines/engine.py.230: loss: 0.09559 [I 2022-10-28 03:56:25] ...optimizers/trijoint.py.151: total_norm: 22.313551 [I 2022-10-28 03:56:30] ...trap/engines/engine.py.224: train: epoch 104 | batch 2080/2304 [I 2022-10-28 03:56:30] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:05 [I 2022-10-28 03:56:30] ...trap/engines/engine.py.229: process: 0.55787 | load: 0.00253 [I 2022-10-28 03:56:30] ...trap/engines/engine.py.230: loss: 0.15028 [I 2022-10-28 03:56:30] ...optimizers/trijoint.py.151: total_norm: 11.795117 [I 2022-10-28 03:56:36] ...trap/engines/engine.py.224: train: epoch 104 | batch 2090/2304 [I 2022-10-28 03:56:36] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:00 [I 2022-10-28 03:56:36] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00250 [I 2022-10-28 03:56:36] ...trap/engines/engine.py.230: loss: 0.15448 [I 2022-10-28 03:56:36] ...optimizers/trijoint.py.151: total_norm: 40.351978 [I 2022-10-28 03:56:41] ...trap/engines/engine.py.224: train: epoch 104 | batch 2100/2304 [I 2022-10-28 03:56:41] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:01:54 [I 2022-10-28 03:56:41] ...trap/engines/engine.py.229: process: 0.54794 | load: 0.00246 [I 2022-10-28 03:56:41] ...trap/engines/engine.py.230: loss: 0.11188 [I 2022-10-28 03:56:41] ...optimizers/trijoint.py.151: total_norm: 13.516891 [I 2022-10-28 03:56:47] ...trap/engines/engine.py.224: train: epoch 104 | batch 2110/2304 [I 2022-10-28 03:56:47] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:48 [I 2022-10-28 03:56:47] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00245 [I 2022-10-28 03:56:47] ...trap/engines/engine.py.230: loss: 0.12221 [I 2022-10-28 03:56:47] ...optimizers/trijoint.py.151: total_norm: 32.154572 [I 2022-10-28 03:56:53] ...trap/engines/engine.py.224: train: epoch 104 | batch 2120/2304 [I 2022-10-28 03:56:53] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:44 [I 2022-10-28 03:56:53] ...trap/engines/engine.py.229: process: 0.55223 | load: 0.00245 [I 2022-10-28 03:56:53] ...trap/engines/engine.py.230: loss: 0.10144 [I 2022-10-28 03:56:53] ...optimizers/trijoint.py.151: total_norm: 26.284521 [I 2022-10-28 03:56:58] ...trap/engines/engine.py.224: train: epoch 104 | batch 2130/2304 [I 2022-10-28 03:56:58] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:41 [I 2022-10-28 03:56:58] ...trap/engines/engine.py.229: process: 0.62186 | load: 0.00272 [I 2022-10-28 03:56:58] ...trap/engines/engine.py.230: loss: 0.08061 [I 2022-10-28 03:56:58] ...optimizers/trijoint.py.151: total_norm: 9.514335 [I 2022-10-28 03:57:04] ...trap/engines/engine.py.224: train: epoch 104 | batch 2140/2304 [I 2022-10-28 03:57:04] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:32 [I 2022-10-28 03:57:04] ...trap/engines/engine.py.229: process: 0.56795 | load: 0.00245 [I 2022-10-28 03:57:04] ...trap/engines/engine.py.230: loss: 0.07956 [I 2022-10-28 03:57:04] ...optimizers/trijoint.py.151: total_norm: 10.854096 [I 2022-10-28 03:57:10] ...trap/engines/engine.py.224: train: epoch 104 | batch 2150/2304 [I 2022-10-28 03:57:10] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:26 [I 2022-10-28 03:57:10] ...trap/engines/engine.py.229: process: 0.57332 | load: 0.00245 [I 2022-10-28 03:57:10] ...trap/engines/engine.py.230: loss: 0.07147 [I 2022-10-28 03:57:10] ...optimizers/trijoint.py.151: total_norm: 9.430259 [I 2022-10-28 03:57:15] ...trap/engines/engine.py.224: train: epoch 104 | batch 2160/2304 [I 2022-10-28 03:57:15] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:19 [I 2022-10-28 03:57:15] ...trap/engines/engine.py.229: process: 0.55514 | load: 0.00243 [I 2022-10-28 03:57:15] ...trap/engines/engine.py.230: loss: 0.12646 [I 2022-10-28 03:57:15] ...optimizers/trijoint.py.151: total_norm: 32.243324 [I 2022-10-28 03:57:21] ...trap/engines/engine.py.224: train: epoch 104 | batch 2170/2304 [I 2022-10-28 03:57:21] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:14 [I 2022-10-28 03:57:21] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00242 [I 2022-10-28 03:57:21] ...trap/engines/engine.py.230: loss: 0.13022 [I 2022-10-28 03:57:21] ...optimizers/trijoint.py.151: total_norm: 11.941525 [I 2022-10-28 03:57:26] ...trap/engines/engine.py.224: train: epoch 104 | batch 2180/2304 [I 2022-10-28 03:57:26] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:09 [I 2022-10-28 03:57:26] ...trap/engines/engine.py.229: process: 0.56325 | load: 0.00290 [I 2022-10-28 03:57:26] ...trap/engines/engine.py.230: loss: 0.12526 [I 2022-10-28 03:57:26] ...optimizers/trijoint.py.151: total_norm: 16.937269 [I 2022-10-28 03:57:32] ...trap/engines/engine.py.224: train: epoch 104 | batch 2190/2304 [I 2022-10-28 03:57:32] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:04 [I 2022-10-28 03:57:32] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00256 [I 2022-10-28 03:57:32] ...trap/engines/engine.py.230: loss: 0.12615 [I 2022-10-28 03:57:32] ...optimizers/trijoint.py.151: total_norm: 32.238026 [I 2022-10-28 03:57:38] ...trap/engines/engine.py.224: train: epoch 104 | batch 2200/2304 [I 2022-10-28 03:57:38] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:01:00 [I 2022-10-28 03:57:38] ...trap/engines/engine.py.229: process: 0.65819 | load: 0.00253 [I 2022-10-28 03:57:38] ...trap/engines/engine.py.230: loss: 0.12629 [I 2022-10-28 03:57:38] ...optimizers/trijoint.py.151: total_norm: 10.701073 [I 2022-10-28 03:57:43] ...trap/engines/engine.py.224: train: epoch 104 | batch 2210/2304 [I 2022-10-28 03:57:43] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:00:53 [I 2022-10-28 03:57:43] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00247 [I 2022-10-28 03:57:43] ...trap/engines/engine.py.230: loss: 0.10321 [I 2022-10-28 03:57:43] ...optimizers/trijoint.py.151: total_norm: 9.396503 [I 2022-10-28 03:57:49] ...trap/engines/engine.py.224: train: epoch 104 | batch 2220/2304 [I 2022-10-28 03:57:49] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:00:46 [I 2022-10-28 03:57:49] ...trap/engines/engine.py.229: process: 0.54830 | load: 0.00243 [I 2022-10-28 03:57:49] ...trap/engines/engine.py.230: loss: 0.19128 [I 2022-10-28 03:57:49] ...optimizers/trijoint.py.151: total_norm: 21.681841 [I 2022-10-28 03:57:55] ...trap/engines/engine.py.224: train: epoch 104 | batch 2230/2304 [I 2022-10-28 03:57:55] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:00:41 [I 2022-10-28 03:57:55] ...trap/engines/engine.py.229: process: 0.57756 | load: 0.00247 [I 2022-10-28 03:57:55] ...trap/engines/engine.py.230: loss: 0.12232 [I 2022-10-28 03:57:55] ...optimizers/trijoint.py.151: total_norm: 8.243535 [I 2022-10-28 03:58:00] ...trap/engines/engine.py.224: train: epoch 104 | batch 2240/2304 [I 2022-10-28 03:58:00] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:35 [I 2022-10-28 03:58:00] ...trap/engines/engine.py.229: process: 0.55075 | load: 0.00245 [I 2022-10-28 03:58:00] ...trap/engines/engine.py.230: loss: 0.13690 [I 2022-10-28 03:58:00] ...optimizers/trijoint.py.151: total_norm: 17.987429 [I 2022-10-28 03:58:06] ...trap/engines/engine.py.224: train: epoch 104 | batch 2250/2304 [I 2022-10-28 03:58:06] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:30 [I 2022-10-28 03:58:06] ...trap/engines/engine.py.229: process: 0.55361 | load: 0.00247 [I 2022-10-28 03:58:06] ...trap/engines/engine.py.230: loss: 0.18453 [I 2022-10-28 03:58:06] ...optimizers/trijoint.py.151: total_norm: 22.294096 [I 2022-10-28 03:58:11] ...trap/engines/engine.py.224: train: epoch 104 | batch 2260/2304 [I 2022-10-28 03:58:11] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:24 [I 2022-10-28 03:58:11] ...trap/engines/engine.py.229: process: 0.56283 | load: 0.00243 [I 2022-10-28 03:58:11] ...trap/engines/engine.py.230: loss: 0.11146 [I 2022-10-28 03:58:11] ...optimizers/trijoint.py.151: total_norm: 19.907742 [I 2022-10-28 03:58:17] ...trap/engines/engine.py.224: train: epoch 104 | batch 2270/2304 [I 2022-10-28 03:58:17] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:19 [I 2022-10-28 03:58:17] ...trap/engines/engine.py.229: process: 0.61576 | load: 0.00271 [I 2022-10-28 03:58:17] ...trap/engines/engine.py.230: loss: 0.10563 [I 2022-10-28 03:58:17] ...optimizers/trijoint.py.151: total_norm: 11.724931 [I 2022-10-28 03:58:23] ...trap/engines/engine.py.224: train: epoch 104 | batch 2280/2304 [I 2022-10-28 03:58:23] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:13 [I 2022-10-28 03:58:23] ...trap/engines/engine.py.229: process: 0.57406 | load: 0.00248 [I 2022-10-28 03:58:23] ...trap/engines/engine.py.230: loss: 0.11191 [I 2022-10-28 03:58:23] ...optimizers/trijoint.py.151: total_norm: 18.906872 [I 2022-10-28 03:58:28] ...trap/engines/engine.py.224: train: epoch 104 | batch 2290/2304 [I 2022-10-28 03:58:28] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:07 [I 2022-10-28 03:58:28] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00246 [I 2022-10-28 03:58:28] ...trap/engines/engine.py.230: loss: 0.10671 [I 2022-10-28 03:58:28] ...optimizers/trijoint.py.151: total_norm: 12.076996 [I 2022-10-28 03:58:34] ...trap/engines/engine.py.224: train: epoch 104 | batch 2300/2304 [I 2022-10-28 03:58:34] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:02 [I 2022-10-28 03:58:34] ...trap/engines/engine.py.229: process: 0.55064 | load: 0.00017 [I 2022-10-28 03:58:34] ...trap/engines/engine.py.230: loss: 0.10180 [I 2022-10-28 03:58:34] ...optimizers/trijoint.py.151: total_norm: 11.009464 [I 2022-10-28 03:58:36] ...trap/engines/engine.py.224: train: epoch 104 | batch 2304/2304 [I 2022-10-28 03:58:36] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:00 [I 2022-10-28 03:58:36] ...trap/engines/engine.py.229: process: 0.54280 | load: 0.00016 [I 2022-10-28 03:58:36] ...trap/engines/engine.py.230: loss: 0.20261 [I 2022-10-28 03:58:36] ...optimizers/trijoint.py.151: total_norm: 21.150091 [S 2022-10-28 03:58:37] ...trap/engines/engine.py.129: train_epoch.epoch: 104 [S 2022-10-28 03:58:37] ...trap/engines/engine.py.129: train_epoch.loss: 0.12451584001683878 [I 2022-10-28 03:58:37] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 03:58:39] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 104 [I 2022-10-28 03:58:39] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 03:58:41] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 03:58:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 03:58:41] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 03:58:44] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 03:58:45] ...trap/engines/engine.py.321: eval: epoch 104 | batch 0/510 [I 2022-10-28 03:58:45] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:49:17 [I 2022-10-28 03:58:45] ...trap/engines/engine.py.326: process: 5.79985 | load: 3.10063 [I 2022-10-28 03:58:48] ...trap/engines/engine.py.321: eval: epoch 104 | batch 10/510 [I 2022-10-28 03:58:48] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:39 [I 2022-10-28 03:58:48] ...trap/engines/engine.py.326: process: 0.34088 | load: 0.00026 [I 2022-10-28 03:58:51] ...trap/engines/engine.py.321: eval: epoch 104 | batch 20/510 [I 2022-10-28 03:58:51] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:10 [I 2022-10-28 03:58:51] ...trap/engines/engine.py.326: process: 0.32530 | load: 0.00016 [I 2022-10-28 03:58:55] ...trap/engines/engine.py.321: eval: epoch 104 | batch 30/510 [I 2022-10-28 03:58:55] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:43 [I 2022-10-28 03:58:55] ...trap/engines/engine.py.326: process: 0.33000 | load: 0.00016 [I 2022-10-28 03:58:58] ...trap/engines/engine.py.321: eval: epoch 104 | batch 40/510 [I 2022-10-28 03:58:58] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:41 [I 2022-10-28 03:58:58] ...trap/engines/engine.py.326: process: 0.37167 | load: 0.00020 [I 2022-10-28 03:59:01] ...trap/engines/engine.py.321: eval: epoch 104 | batch 50/510 [I 2022-10-28 03:59:01] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:34 [I 2022-10-28 03:59:01] ...trap/engines/engine.py.326: process: 0.32615 | load: 0.00018 [I 2022-10-28 03:59:05] ...trap/engines/engine.py.321: eval: epoch 104 | batch 60/510 [I 2022-10-28 03:59:05] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:34 [I 2022-10-28 03:59:05] ...trap/engines/engine.py.326: process: 0.34909 | load: 0.00016 [I 2022-10-28 03:59:08] ...trap/engines/engine.py.321: eval: epoch 104 | batch 70/510 [I 2022-10-28 03:59:08] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:29 [I 2022-10-28 03:59:08] ...trap/engines/engine.py.326: process: 0.33278 | load: 0.00021 [I 2022-10-28 03:59:12] ...trap/engines/engine.py.321: eval: epoch 104 | batch 80/510 [I 2022-10-28 03:59:12] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:26 [I 2022-10-28 03:59:12] ...trap/engines/engine.py.326: process: 0.37583 | load: 0.00017 [I 2022-10-28 03:59:15] ...trap/engines/engine.py.321: eval: epoch 104 | batch 90/510 [I 2022-10-28 03:59:15] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:20 [I 2022-10-28 03:59:15] ...trap/engines/engine.py.326: process: 0.32487 | load: 0.00017 [I 2022-10-28 03:59:18] ...trap/engines/engine.py.321: eval: epoch 104 | batch 100/510 [I 2022-10-28 03:59:18] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-28 03:59:18] ...trap/engines/engine.py.326: process: 0.28859 | load: 0.00019 [I 2022-10-28 03:59:21] ...trap/engines/engine.py.321: eval: epoch 104 | batch 110/510 [I 2022-10-28 03:59:21] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:57 [I 2022-10-28 03:59:21] ...trap/engines/engine.py.326: process: 0.29507 | load: 0.00017 [I 2022-10-28 03:59:24] ...trap/engines/engine.py.321: eval: epoch 104 | batch 120/510 [I 2022-10-28 03:59:24] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:54 [I 2022-10-28 03:59:24] ...trap/engines/engine.py.326: process: 0.30186 | load: 0.00017 [I 2022-10-28 03:59:27] ...trap/engines/engine.py.321: eval: epoch 104 | batch 130/510 [I 2022-10-28 03:59:27] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:49 [I 2022-10-28 03:59:27] ...trap/engines/engine.py.326: process: 0.28891 | load: 0.00016 [I 2022-10-28 03:59:30] ...trap/engines/engine.py.321: eval: epoch 104 | batch 140/510 [I 2022-10-28 03:59:30] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:47 [I 2022-10-28 03:59:30] ...trap/engines/engine.py.326: process: 0.28173 | load: 0.00016 [I 2022-10-28 03:59:33] ...trap/engines/engine.py.321: eval: epoch 104 | batch 150/510 [I 2022-10-28 03:59:33] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:44 [I 2022-10-28 03:59:33] ...trap/engines/engine.py.326: process: 0.28333 | load: 0.00016 [I 2022-10-28 03:59:36] ...trap/engines/engine.py.321: eval: epoch 104 | batch 160/510 [I 2022-10-28 03:59:36] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-28 03:59:36] ...trap/engines/engine.py.326: process: 0.29109 | load: 0.00018 [I 2022-10-28 03:59:39] ...trap/engines/engine.py.321: eval: epoch 104 | batch 170/510 [I 2022-10-28 03:59:39] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 03:59:39] ...trap/engines/engine.py.326: process: 0.29070 | load: 0.00016 [I 2022-10-28 03:59:42] ...trap/engines/engine.py.321: eval: epoch 104 | batch 180/510 [I 2022-10-28 03:59:42] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-28 03:59:42] ...trap/engines/engine.py.326: process: 0.29112 | load: 0.00017 [I 2022-10-28 03:59:44] ...trap/engines/engine.py.321: eval: epoch 104 | batch 190/510 [I 2022-10-28 03:59:44] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-28 03:59:44] ...trap/engines/engine.py.326: process: 0.28468 | load: 0.00016 [I 2022-10-28 03:59:47] ...trap/engines/engine.py.321: eval: epoch 104 | batch 200/510 [I 2022-10-28 03:59:47] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:29 [I 2022-10-28 03:59:47] ...trap/engines/engine.py.326: process: 0.29076 | load: 0.00014 [I 2022-10-28 03:59:50] ...trap/engines/engine.py.321: eval: epoch 104 | batch 210/510 [I 2022-10-28 03:59:50] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 03:59:50] ...trap/engines/engine.py.326: process: 0.28888 | load: 0.00026 [I 2022-10-28 03:59:53] ...trap/engines/engine.py.321: eval: epoch 104 | batch 220/510 [I 2022-10-28 03:59:53] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:23 [I 2022-10-28 03:59:53] ...trap/engines/engine.py.326: process: 0.29494 | load: 0.00016 [I 2022-10-28 03:59:56] ...trap/engines/engine.py.321: eval: epoch 104 | batch 230/510 [I 2022-10-28 03:59:56] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:20 [I 2022-10-28 03:59:56] ...trap/engines/engine.py.326: process: 0.28673 | load: 0.00019 [I 2022-10-28 03:59:59] ...trap/engines/engine.py.321: eval: epoch 104 | batch 240/510 [I 2022-10-28 03:59:59] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 03:59:59] ...trap/engines/engine.py.326: process: 0.28938 | load: 0.00018 [I 2022-10-28 04:00:02] ...trap/engines/engine.py.321: eval: epoch 104 | batch 250/510 [I 2022-10-28 04:00:02] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-28 04:00:02] ...trap/engines/engine.py.326: process: 0.28781 | load: 0.00031 [I 2022-10-28 04:00:05] ...trap/engines/engine.py.321: eval: epoch 104 | batch 260/510 [I 2022-10-28 04:00:05] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 04:00:05] ...trap/engines/engine.py.326: process: 0.28999 | load: 0.00018 [I 2022-10-28 04:00:08] ...trap/engines/engine.py.321: eval: epoch 104 | batch 270/510 [I 2022-10-28 04:00:08] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-28 04:00:08] ...trap/engines/engine.py.326: process: 0.28810 | load: 0.00017 [I 2022-10-28 04:00:10] ...trap/engines/engine.py.321: eval: epoch 104 | batch 280/510 [I 2022-10-28 04:00:10] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 04:00:10] ...trap/engines/engine.py.326: process: 0.28727 | load: 0.00016 [I 2022-10-28 04:00:13] ...trap/engines/engine.py.321: eval: epoch 104 | batch 290/510 [I 2022-10-28 04:00:13] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:07 [I 2022-10-28 04:00:13] ...trap/engines/engine.py.326: process: 0.38814 | load: 0.00019 [I 2022-10-28 04:00:16] ...trap/engines/engine.py.321: eval: epoch 104 | batch 300/510 [I 2022-10-28 04:00:16] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:01 [I 2022-10-28 04:00:16] ...trap/engines/engine.py.326: process: 0.30174 | load: 0.00014 [I 2022-10-28 04:00:19] ...trap/engines/engine.py.321: eval: epoch 104 | batch 310/510 [I 2022-10-28 04:00:19] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:57 [I 2022-10-28 04:00:19] ...trap/engines/engine.py.326: process: 0.28983 | load: 0.00026 [I 2022-10-28 04:00:22] ...trap/engines/engine.py.321: eval: epoch 104 | batch 320/510 [I 2022-10-28 04:00:22] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:54 [I 2022-10-28 04:00:22] ...trap/engines/engine.py.326: process: 0.29385 | load: 0.00017 [I 2022-10-28 04:00:25] ...trap/engines/engine.py.321: eval: epoch 104 | batch 330/510 [I 2022-10-28 04:00:25] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:52 [I 2022-10-28 04:00:25] ...trap/engines/engine.py.326: process: 0.29089 | load: 0.00017 [I 2022-10-28 04:00:28] ...trap/engines/engine.py.321: eval: epoch 104 | batch 340/510 [I 2022-10-28 04:00:28] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-28 04:00:28] ...trap/engines/engine.py.326: process: 0.28277 | load: 0.00015 [I 2022-10-28 04:00:31] ...trap/engines/engine.py.321: eval: epoch 104 | batch 350/510 [I 2022-10-28 04:00:31] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:45 [I 2022-10-28 04:00:31] ...trap/engines/engine.py.326: process: 0.28798 | load: 0.00017 [I 2022-10-28 04:00:34] ...trap/engines/engine.py.321: eval: epoch 104 | batch 360/510 [I 2022-10-28 04:00:34] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 04:00:34] ...trap/engines/engine.py.326: process: 0.28970 | load: 0.00015 [I 2022-10-28 04:00:36] ...trap/engines/engine.py.321: eval: epoch 104 | batch 370/510 [I 2022-10-28 04:00:36] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 04:00:36] ...trap/engines/engine.py.326: process: 0.28745 | load: 0.00018 [I 2022-10-28 04:00:39] ...trap/engines/engine.py.321: eval: epoch 104 | batch 380/510 [I 2022-10-28 04:00:39] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 04:00:39] ...trap/engines/engine.py.326: process: 0.28810 | load: 0.00015 [I 2022-10-28 04:00:42] ...trap/engines/engine.py.321: eval: epoch 104 | batch 390/510 [I 2022-10-28 04:00:42] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 04:00:42] ...trap/engines/engine.py.326: process: 0.28540 | load: 0.00015 [I 2022-10-28 04:00:45] ...trap/engines/engine.py.321: eval: epoch 104 | batch 400/510 [I 2022-10-28 04:00:45] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:31 [I 2022-10-28 04:00:45] ...trap/engines/engine.py.326: process: 0.28587 | load: 0.00014 [I 2022-10-28 04:00:48] ...trap/engines/engine.py.321: eval: epoch 104 | batch 410/510 [I 2022-10-28 04:00:48] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 04:00:48] ...trap/engines/engine.py.326: process: 0.29003 | load: 0.00014 [I 2022-10-28 04:00:51] ...trap/engines/engine.py.321: eval: epoch 104 | batch 420/510 [I 2022-10-28 04:00:51] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 04:00:51] ...trap/engines/engine.py.326: process: 0.28442 | load: 0.00015 [I 2022-10-28 04:00:54] ...trap/engines/engine.py.321: eval: epoch 104 | batch 430/510 [I 2022-10-28 04:00:54] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 04:00:54] ...trap/engines/engine.py.326: process: 0.29017 | load: 0.00016 [I 2022-10-28 04:00:57] ...trap/engines/engine.py.321: eval: epoch 104 | batch 440/510 [I 2022-10-28 04:00:57] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 04:00:57] ...trap/engines/engine.py.326: process: 0.28942 | load: 0.00021 [I 2022-10-28 04:00:59] ...trap/engines/engine.py.321: eval: epoch 104 | batch 450/510 [I 2022-10-28 04:00:59] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 04:00:59] ...trap/engines/engine.py.326: process: 0.28332 | load: 0.00015 [I 2022-10-28 04:01:02] ...trap/engines/engine.py.321: eval: epoch 104 | batch 460/510 [I 2022-10-28 04:01:02] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 04:01:02] ...trap/engines/engine.py.326: process: 0.28782 | load: 0.00016 [I 2022-10-28 04:01:05] ...trap/engines/engine.py.321: eval: epoch 104 | batch 470/510 [I 2022-10-28 04:01:05] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 04:01:05] ...trap/engines/engine.py.326: process: 0.28479 | load: 0.00016 [I 2022-10-28 04:01:08] ...trap/engines/engine.py.321: eval: epoch 104 | batch 480/510 [I 2022-10-28 04:01:08] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 04:01:08] ...trap/engines/engine.py.326: process: 0.28506 | load: 0.00022 [I 2022-10-28 04:01:11] ...trap/engines/engine.py.321: eval: epoch 104 | batch 490/510 [I 2022-10-28 04:01:11] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 04:01:11] ...trap/engines/engine.py.326: process: 0.28705 | load: 0.00021 [I 2022-10-28 04:01:14] ...trap/engines/engine.py.321: eval: epoch 104 | batch 500/510 [I 2022-10-28 04:01:14] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 04:01:14] ...trap/engines/engine.py.326: process: 0.28890 | load: 0.00015 [I 2022-10-28 04:01:17] ...trap/engines/engine.py.321: eval: epoch 104 | batch 510/510 [I 2022-10-28 04:01:17] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 04:01:17] ...trap/engines/engine.py.326: process: 0.29019 | load: 0.00014 [S 2022-10-28 04:01:17] ...trap/engines/engine.py.132: eval_epoch.epoch: 104 [I 2022-10-28 04:01:17] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.741000 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909200 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936900 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.730900 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907000 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935300 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011636 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008424 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005108 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012996 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008050 [S 2022-10-28 04:01:22] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004649 [I 2022-10-28 04:01:24] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-28 04:01:24] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 04:01:24] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 04:01:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 04:01:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 04:01:29] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 04:01:30] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 04:01:36] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 04:01:36] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 04:01:36] ...trap/engines/engine.py.470: Saving model... [I 2022-10-28 04:01:39] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 04:01:44] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 04:01:44] ...trap/engines/engine.py.163: Training model on trainset for epoch 105 [I 2022-10-28 04:01:44] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 04:01:47] ...trap/engines/engine.py.224: train: epoch 105 | batch 0/2304 [I 2022-10-28 04:01:47] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:59 [I 2022-10-28 04:01:47] ...trap/engines/engine.py.229: process: 2.70801 | load: 2.05616 [I 2022-10-28 04:01:47] ...trap/engines/engine.py.230: loss: 0.10605 [I 2022-10-28 04:01:47] ...optimizers/trijoint.py.151: total_norm: 8.593562 [I 2022-10-28 04:01:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 10/2304 [I 2022-10-28 04:01:53] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:04 [I 2022-10-28 04:01:53] ...trap/engines/engine.py.229: process: 0.55027 | load: 0.00449 [I 2022-10-28 04:01:53] ...trap/engines/engine.py.230: loss: 0.10109 [I 2022-10-28 04:01:53] ...optimizers/trijoint.py.151: total_norm: 16.911318 [I 2022-10-28 04:01:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 20/2304 [I 2022-10-28 04:01:58] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:21 [I 2022-10-28 04:01:58] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00293 [I 2022-10-28 04:01:58] ...trap/engines/engine.py.230: loss: 0.10718 [I 2022-10-28 04:01:58] ...optimizers/trijoint.py.151: total_norm: 13.251578 [I 2022-10-28 04:02:04] ...trap/engines/engine.py.224: train: epoch 105 | batch 30/2304 [I 2022-10-28 04:02:04] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:14 [I 2022-10-28 04:02:04] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00266 [I 2022-10-28 04:02:04] ...trap/engines/engine.py.230: loss: 0.07167 [I 2022-10-28 04:02:04] ...optimizers/trijoint.py.151: total_norm: 16.662542 [I 2022-10-28 04:02:09] ...trap/engines/engine.py.224: train: epoch 105 | batch 40/2304 [I 2022-10-28 04:02:09] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:59 [I 2022-10-28 04:02:09] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00289 [I 2022-10-28 04:02:09] ...trap/engines/engine.py.230: loss: 0.15460 [I 2022-10-28 04:02:09] ...optimizers/trijoint.py.151: total_norm: 25.235004 [I 2022-10-28 04:02:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 50/2304 [I 2022-10-28 04:02:15] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:10 [I 2022-10-28 04:02:15] ...trap/engines/engine.py.229: process: 0.56815 | load: 0.00283 [I 2022-10-28 04:02:15] ...trap/engines/engine.py.230: loss: 0.14015 [I 2022-10-28 04:02:15] ...optimizers/trijoint.py.151: total_norm: 13.734529 [I 2022-10-28 04:02:21] ...trap/engines/engine.py.224: train: epoch 105 | batch 60/2304 [I 2022-10-28 04:02:21] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:53 [I 2022-10-28 04:02:21] ...trap/engines/engine.py.229: process: 0.55164 | load: 0.00268 [I 2022-10-28 04:02:21] ...trap/engines/engine.py.230: loss: 0.13566 [I 2022-10-28 04:02:21] ...optimizers/trijoint.py.151: total_norm: 18.276667 [I 2022-10-28 04:02:26] ...trap/engines/engine.py.224: train: epoch 105 | batch 70/2304 [I 2022-10-28 04:02:26] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:53 [I 2022-10-28 04:02:26] ...trap/engines/engine.py.229: process: 0.56097 | load: 0.00294 [I 2022-10-28 04:02:26] ...trap/engines/engine.py.230: loss: 0.12214 [I 2022-10-28 04:02:26] ...optimizers/trijoint.py.151: total_norm: 30.332190 [I 2022-10-28 04:02:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 80/2304 [I 2022-10-28 04:02:32] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:45 [I 2022-10-28 04:02:32] ...trap/engines/engine.py.229: process: 0.56268 | load: 0.00271 [I 2022-10-28 04:02:32] ...trap/engines/engine.py.230: loss: 0.10613 [I 2022-10-28 04:02:32] ...optimizers/trijoint.py.151: total_norm: 11.881553 [I 2022-10-28 04:02:37] ...trap/engines/engine.py.224: train: epoch 105 | batch 90/2304 [I 2022-10-28 04:02:37] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:33 [I 2022-10-28 04:02:37] ...trap/engines/engine.py.229: process: 0.54449 | load: 0.00269 [I 2022-10-28 04:02:37] ...trap/engines/engine.py.230: loss: 0.10650 [I 2022-10-28 04:02:37] ...optimizers/trijoint.py.151: total_norm: 11.461350 [I 2022-10-28 04:02:43] ...trap/engines/engine.py.224: train: epoch 105 | batch 100/2304 [I 2022-10-28 04:02:43] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:28 [I 2022-10-28 04:02:43] ...trap/engines/engine.py.229: process: 0.56753 | load: 0.00304 [I 2022-10-28 04:02:43] ...trap/engines/engine.py.230: loss: 0.17713 [I 2022-10-28 04:02:43] ...optimizers/trijoint.py.151: total_norm: 12.778979 [I 2022-10-28 04:02:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 110/2304 [I 2022-10-28 04:02:49] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:18 [I 2022-10-28 04:02:49] ...trap/engines/engine.py.229: process: 0.55881 | load: 0.00274 [I 2022-10-28 04:02:49] ...trap/engines/engine.py.230: loss: 0.09449 [I 2022-10-28 04:02:49] ...optimizers/trijoint.py.151: total_norm: 19.972250 [I 2022-10-28 04:02:54] ...trap/engines/engine.py.224: train: epoch 105 | batch 120/2304 [I 2022-10-28 04:02:54] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:30 [I 2022-10-28 04:02:54] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00257 [I 2022-10-28 04:02:54] ...trap/engines/engine.py.230: loss: 0.09726 [I 2022-10-28 04:02:54] ...optimizers/trijoint.py.151: total_norm: 9.834057 [I 2022-10-28 04:03:00] ...trap/engines/engine.py.224: train: epoch 105 | batch 130/2304 [I 2022-10-28 04:03:00] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:18 [I 2022-10-28 04:03:00] ...trap/engines/engine.py.229: process: 0.56872 | load: 0.00265 [I 2022-10-28 04:03:00] ...trap/engines/engine.py.230: loss: 0.12376 [I 2022-10-28 04:03:00] ...optimizers/trijoint.py.151: total_norm: 15.922943 [I 2022-10-28 04:03:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 140/2304 [I 2022-10-28 04:03:05] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:10 [I 2022-10-28 04:03:05] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00265 [I 2022-10-28 04:03:05] ...trap/engines/engine.py.230: loss: 0.10767 [I 2022-10-28 04:03:05] ...optimizers/trijoint.py.151: total_norm: 17.086557 [I 2022-10-28 04:03:11] ...trap/engines/engine.py.224: train: epoch 105 | batch 150/2304 [I 2022-10-28 04:03:11] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:05 [I 2022-10-28 04:03:11] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00267 [I 2022-10-28 04:03:11] ...trap/engines/engine.py.230: loss: 0.11896 [I 2022-10-28 04:03:11] ...optimizers/trijoint.py.151: total_norm: 18.270655 [I 2022-10-28 04:03:17] ...trap/engines/engine.py.224: train: epoch 105 | batch 160/2304 [I 2022-10-28 04:03:17] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:57 [I 2022-10-28 04:03:17] ...trap/engines/engine.py.229: process: 0.57085 | load: 0.00268 [I 2022-10-28 04:03:17] ...trap/engines/engine.py.230: loss: 0.10888 [I 2022-10-28 04:03:17] ...optimizers/trijoint.py.151: total_norm: 13.139354 [I 2022-10-28 04:03:22] ...trap/engines/engine.py.224: train: epoch 105 | batch 170/2304 [I 2022-10-28 04:03:22] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:41 [I 2022-10-28 04:03:22] ...trap/engines/engine.py.229: process: 0.54789 | load: 0.00243 [I 2022-10-28 04:03:22] ...trap/engines/engine.py.230: loss: 0.22743 [I 2022-10-28 04:03:22] ...optimizers/trijoint.py.151: total_norm: 16.630774 [I 2022-10-28 04:03:28] ...trap/engines/engine.py.224: train: epoch 105 | batch 180/2304 [I 2022-10-28 04:03:28] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:41 [I 2022-10-28 04:03:28] ...trap/engines/engine.py.229: process: 0.54931 | load: 0.00241 [I 2022-10-28 04:03:28] ...trap/engines/engine.py.230: loss: 0.11082 [I 2022-10-28 04:03:28] ...optimizers/trijoint.py.151: total_norm: 10.088912 [I 2022-10-28 04:03:33] ...trap/engines/engine.py.224: train: epoch 105 | batch 190/2304 [I 2022-10-28 04:03:33] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:50 [I 2022-10-28 04:03:33] ...trap/engines/engine.py.229: process: 0.54744 | load: 0.00272 [I 2022-10-28 04:03:33] ...trap/engines/engine.py.230: loss: 0.15749 [I 2022-10-28 04:03:33] ...optimizers/trijoint.py.151: total_norm: 23.635429 [I 2022-10-28 04:03:39] ...trap/engines/engine.py.224: train: epoch 105 | batch 200/2304 [I 2022-10-28 04:03:39] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:27 [I 2022-10-28 04:03:39] ...trap/engines/engine.py.229: process: 0.54433 | load: 0.00245 [I 2022-10-28 04:03:39] ...trap/engines/engine.py.230: loss: 0.15370 [I 2022-10-28 04:03:39] ...optimizers/trijoint.py.151: total_norm: 18.128666 [I 2022-10-28 04:03:44] ...trap/engines/engine.py.224: train: epoch 105 | batch 210/2304 [I 2022-10-28 04:03:44] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:28 [I 2022-10-28 04:03:44] ...trap/engines/engine.py.229: process: 0.55993 | load: 0.00247 [I 2022-10-28 04:03:44] ...trap/engines/engine.py.230: loss: 0.07878 [I 2022-10-28 04:03:44] ...optimizers/trijoint.py.151: total_norm: 11.417471 [I 2022-10-28 04:03:50] ...trap/engines/engine.py.224: train: epoch 105 | batch 220/2304 [I 2022-10-28 04:03:50] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:25 [I 2022-10-28 04:03:50] ...trap/engines/engine.py.229: process: 0.57316 | load: 0.00253 [I 2022-10-28 04:03:50] ...trap/engines/engine.py.230: loss: 0.10930 [I 2022-10-28 04:03:50] ...optimizers/trijoint.py.151: total_norm: 20.259190 [I 2022-10-28 04:03:56] ...trap/engines/engine.py.224: train: epoch 105 | batch 230/2304 [I 2022-10-28 04:03:56] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:24 [I 2022-10-28 04:03:56] ...trap/engines/engine.py.229: process: 0.56467 | load: 0.00254 [I 2022-10-28 04:03:56] ...trap/engines/engine.py.230: loss: 0.11120 [I 2022-10-28 04:03:56] ...optimizers/trijoint.py.151: total_norm: 9.681998 [I 2022-10-28 04:04:01] ...trap/engines/engine.py.224: train: epoch 105 | batch 240/2304 [I 2022-10-28 04:04:01] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:11 [I 2022-10-28 04:04:01] ...trap/engines/engine.py.229: process: 0.55217 | load: 0.00246 [I 2022-10-28 04:04:01] ...trap/engines/engine.py.230: loss: 0.10332 [I 2022-10-28 04:04:01] ...optimizers/trijoint.py.151: total_norm: 14.844755 [I 2022-10-28 04:04:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 250/2304 [I 2022-10-28 04:04:07] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:59 [I 2022-10-28 04:04:07] ...trap/engines/engine.py.229: process: 0.53944 | load: 0.00256 [I 2022-10-28 04:04:07] ...trap/engines/engine.py.230: loss: 0.10435 [I 2022-10-28 04:04:07] ...optimizers/trijoint.py.151: total_norm: 36.021080 [I 2022-10-28 04:04:12] ...trap/engines/engine.py.224: train: epoch 105 | batch 260/2304 [I 2022-10-28 04:04:12] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:55 [I 2022-10-28 04:04:12] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00244 [I 2022-10-28 04:04:12] ...trap/engines/engine.py.230: loss: 0.09017 [I 2022-10-28 04:04:12] ...optimizers/trijoint.py.151: total_norm: 17.324604 [I 2022-10-28 04:04:18] ...trap/engines/engine.py.224: train: epoch 105 | batch 270/2304 [I 2022-10-28 04:04:18] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:55 [I 2022-10-28 04:04:18] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00247 [I 2022-10-28 04:04:18] ...trap/engines/engine.py.230: loss: 0.14420 [I 2022-10-28 04:04:18] ...optimizers/trijoint.py.151: total_norm: 18.354895 [I 2022-10-28 04:04:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 280/2304 [I 2022-10-28 04:04:24] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:49 [I 2022-10-28 04:04:24] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00266 [I 2022-10-28 04:04:24] ...trap/engines/engine.py.230: loss: 0.10762 [I 2022-10-28 04:04:24] ...optimizers/trijoint.py.151: total_norm: 52.952755 [I 2022-10-28 04:04:29] ...trap/engines/engine.py.224: train: epoch 105 | batch 290/2304 [I 2022-10-28 04:04:29] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:52 [I 2022-10-28 04:04:29] ...trap/engines/engine.py.229: process: 0.56989 | load: 0.00256 [I 2022-10-28 04:04:29] ...trap/engines/engine.py.230: loss: 0.14198 [I 2022-10-28 04:04:29] ...optimizers/trijoint.py.151: total_norm: 20.695400 [I 2022-10-28 04:04:35] ...trap/engines/engine.py.224: train: epoch 105 | batch 300/2304 [I 2022-10-28 04:04:35] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:39 [I 2022-10-28 04:04:35] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00252 [I 2022-10-28 04:04:35] ...trap/engines/engine.py.230: loss: 0.12604 [I 2022-10-28 04:04:35] ...optimizers/trijoint.py.151: total_norm: 18.796925 [I 2022-10-28 04:04:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 310/2304 [I 2022-10-28 04:04:40] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:27 [I 2022-10-28 04:04:40] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00243 [I 2022-10-28 04:04:40] ...trap/engines/engine.py.230: loss: 0.12803 [I 2022-10-28 04:04:40] ...optimizers/trijoint.py.151: total_norm: 11.443996 [I 2022-10-28 04:04:46] ...trap/engines/engine.py.224: train: epoch 105 | batch 320/2304 [I 2022-10-28 04:04:46] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:19 [I 2022-10-28 04:04:46] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00248 [I 2022-10-28 04:04:46] ...trap/engines/engine.py.230: loss: 0.10033 [I 2022-10-28 04:04:46] ...optimizers/trijoint.py.151: total_norm: 25.552382 [I 2022-10-28 04:04:52] ...trap/engines/engine.py.224: train: epoch 105 | batch 330/2304 [I 2022-10-28 04:04:52] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:32 [I 2022-10-28 04:04:52] ...trap/engines/engine.py.229: process: 0.56577 | load: 0.00256 [I 2022-10-28 04:04:52] ...trap/engines/engine.py.230: loss: 0.13311 [I 2022-10-28 04:04:52] ...optimizers/trijoint.py.151: total_norm: 10.906266 [I 2022-10-28 04:04:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 340/2304 [I 2022-10-28 04:04:57] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:15 [I 2022-10-28 04:04:57] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00260 [I 2022-10-28 04:04:57] ...trap/engines/engine.py.230: loss: 0.11007 [I 2022-10-28 04:04:57] ...optimizers/trijoint.py.151: total_norm: 11.103953 [I 2022-10-28 04:05:03] ...trap/engines/engine.py.224: train: epoch 105 | batch 350/2304 [I 2022-10-28 04:05:03] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:14 [I 2022-10-28 04:05:03] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00245 [I 2022-10-28 04:05:03] ...trap/engines/engine.py.230: loss: 0.18945 [I 2022-10-28 04:05:03] ...optimizers/trijoint.py.151: total_norm: 19.089952 [I 2022-10-28 04:05:08] ...trap/engines/engine.py.224: train: epoch 105 | batch 360/2304 [I 2022-10-28 04:05:08] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:06 [I 2022-10-28 04:05:08] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00254 [I 2022-10-28 04:05:08] ...trap/engines/engine.py.230: loss: 0.15279 [I 2022-10-28 04:05:08] ...optimizers/trijoint.py.151: total_norm: 22.431026 [I 2022-10-28 04:05:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 370/2304 [I 2022-10-28 04:05:14] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:57 [I 2022-10-28 04:05:14] ...trap/engines/engine.py.229: process: 0.54985 | load: 0.00243 [I 2022-10-28 04:05:14] ...trap/engines/engine.py.230: loss: 0.13209 [I 2022-10-28 04:05:14] ...optimizers/trijoint.py.151: total_norm: 10.468235 [I 2022-10-28 04:05:20] ...trap/engines/engine.py.224: train: epoch 105 | batch 380/2304 [I 2022-10-28 04:05:20] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:51 [I 2022-10-28 04:05:20] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00247 [I 2022-10-28 04:05:20] ...trap/engines/engine.py.230: loss: 0.19135 [I 2022-10-28 04:05:20] ...optimizers/trijoint.py.151: total_norm: 14.967912 [I 2022-10-28 04:05:25] ...trap/engines/engine.py.224: train: epoch 105 | batch 390/2304 [I 2022-10-28 04:05:25] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:43 [I 2022-10-28 04:05:25] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00252 [I 2022-10-28 04:05:25] ...trap/engines/engine.py.230: loss: 0.10802 [I 2022-10-28 04:05:25] ...optimizers/trijoint.py.151: total_norm: 15.228195 [I 2022-10-28 04:05:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 400/2304 [I 2022-10-28 04:05:31] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:45 [I 2022-10-28 04:05:31] ...trap/engines/engine.py.229: process: 0.56249 | load: 0.00244 [I 2022-10-28 04:05:31] ...trap/engines/engine.py.230: loss: 0.10472 [I 2022-10-28 04:05:31] ...optimizers/trijoint.py.151: total_norm: 13.907215 [I 2022-10-28 04:05:36] ...trap/engines/engine.py.224: train: epoch 105 | batch 410/2304 [I 2022-10-28 04:05:36] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:41 [I 2022-10-28 04:05:36] ...trap/engines/engine.py.229: process: 0.56274 | load: 0.00261 [I 2022-10-28 04:05:36] ...trap/engines/engine.py.230: loss: 0.13438 [I 2022-10-28 04:05:36] ...optimizers/trijoint.py.151: total_norm: 19.740679 [I 2022-10-28 04:05:42] ...trap/engines/engine.py.224: train: epoch 105 | batch 420/2304 [I 2022-10-28 04:05:42] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:33 [I 2022-10-28 04:05:42] ...trap/engines/engine.py.229: process: 0.54592 | load: 0.00254 [I 2022-10-28 04:05:42] ...trap/engines/engine.py.230: loss: 0.10193 [I 2022-10-28 04:05:42] ...optimizers/trijoint.py.151: total_norm: 11.895775 [I 2022-10-28 04:05:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 430/2304 [I 2022-10-28 04:05:48] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:25 [I 2022-10-28 04:05:48] ...trap/engines/engine.py.229: process: 0.56614 | load: 0.00251 [I 2022-10-28 04:05:48] ...trap/engines/engine.py.230: loss: 0.16463 [I 2022-10-28 04:05:48] ...optimizers/trijoint.py.151: total_norm: 16.370373 [I 2022-10-28 04:05:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 440/2304 [I 2022-10-28 04:05:53] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:19 [I 2022-10-28 04:05:53] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00241 [I 2022-10-28 04:05:53] ...trap/engines/engine.py.230: loss: 0.13869 [I 2022-10-28 04:05:53] ...optimizers/trijoint.py.151: total_norm: 26.470280 [I 2022-10-28 04:05:59] ...trap/engines/engine.py.224: train: epoch 105 | batch 450/2304 [I 2022-10-28 04:05:59] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:22 [I 2022-10-28 04:05:59] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00241 [I 2022-10-28 04:05:59] ...trap/engines/engine.py.230: loss: 0.12080 [I 2022-10-28 04:05:59] ...optimizers/trijoint.py.151: total_norm: 28.816566 [I 2022-10-28 04:06:04] ...trap/engines/engine.py.224: train: epoch 105 | batch 460/2304 [I 2022-10-28 04:06:04] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:11 [I 2022-10-28 04:06:04] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00242 [I 2022-10-28 04:06:04] ...trap/engines/engine.py.230: loss: 0.09419 [I 2022-10-28 04:06:04] ...optimizers/trijoint.py.151: total_norm: 12.017920 [I 2022-10-28 04:06:10] ...trap/engines/engine.py.224: train: epoch 105 | batch 470/2304 [I 2022-10-28 04:06:10] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:00 [I 2022-10-28 04:06:10] ...trap/engines/engine.py.229: process: 0.54804 | load: 0.00253 [I 2022-10-28 04:06:10] ...trap/engines/engine.py.230: loss: 0.14981 [I 2022-10-28 04:06:10] ...optimizers/trijoint.py.151: total_norm: 153.021912 [I 2022-10-28 04:06:16] ...trap/engines/engine.py.224: train: epoch 105 | batch 480/2304 [I 2022-10-28 04:06:16] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:54 [I 2022-10-28 04:06:16] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00250 [I 2022-10-28 04:06:16] ...trap/engines/engine.py.230: loss: 0.10008 [I 2022-10-28 04:06:16] ...optimizers/trijoint.py.151: total_norm: 33.140427 [I 2022-10-28 04:06:21] ...trap/engines/engine.py.224: train: epoch 105 | batch 490/2304 [I 2022-10-28 04:06:21] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:53 [I 2022-10-28 04:06:21] ...trap/engines/engine.py.229: process: 0.56216 | load: 0.00299 [I 2022-10-28 04:06:21] ...trap/engines/engine.py.230: loss: 0.10197 [I 2022-10-28 04:06:21] ...optimizers/trijoint.py.151: total_norm: 9.413812 [I 2022-10-28 04:06:46] ...trap/engines/engine.py.224: train: epoch 105 | batch 500/2304 [I 2022-10-28 04:06:46] ...trap/engines/engine.py.228: elapsed: 0:05:01 | left: 1:16:54 [I 2022-10-28 04:06:46] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00258 [I 2022-10-28 04:06:46] ...trap/engines/engine.py.230: loss: 0.11485 [I 2022-10-28 04:06:46] ...optimizers/trijoint.py.151: total_norm: 34.251053 [I 2022-10-28 04:06:52] ...trap/engines/engine.py.224: train: epoch 105 | batch 510/2304 [I 2022-10-28 04:06:52] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:23:03 [I 2022-10-28 04:06:52] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00238 [I 2022-10-28 04:06:52] ...trap/engines/engine.py.230: loss: 0.15773 [I 2022-10-28 04:06:52] ...optimizers/trijoint.py.151: total_norm: 34.878162 [I 2022-10-28 04:06:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 520/2304 [I 2022-10-28 04:06:57] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:17:11 [I 2022-10-28 04:06:57] ...trap/engines/engine.py.229: process: 0.54759 | load: 0.00238 [I 2022-10-28 04:06:57] ...trap/engines/engine.py.230: loss: 0.10587 [I 2022-10-28 04:06:57] ...optimizers/trijoint.py.151: total_norm: 17.586170 [I 2022-10-28 04:07:03] ...trap/engines/engine.py.224: train: epoch 105 | batch 530/2304 [I 2022-10-28 04:07:03] ...trap/engines/engine.py.228: elapsed: 0:05:18 | left: 0:16:33 [I 2022-10-28 04:07:03] ...trap/engines/engine.py.229: process: 0.56187 | load: 0.00251 [I 2022-10-28 04:07:03] ...trap/engines/engine.py.230: loss: 0.14806 [I 2022-10-28 04:07:03] ...optimizers/trijoint.py.151: total_norm: 14.391430 [I 2022-10-28 04:07:09] ...trap/engines/engine.py.224: train: epoch 105 | batch 540/2304 [I 2022-10-28 04:07:09] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:16:20 [I 2022-10-28 04:07:09] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00249 [I 2022-10-28 04:07:09] ...trap/engines/engine.py.230: loss: 0.16493 [I 2022-10-28 04:07:09] ...optimizers/trijoint.py.151: total_norm: 14.137576 [I 2022-10-28 04:07:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 550/2304 [I 2022-10-28 04:07:14] ...trap/engines/engine.py.228: elapsed: 0:05:29 | left: 0:16:19 [I 2022-10-28 04:07:14] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00244 [I 2022-10-28 04:07:14] ...trap/engines/engine.py.230: loss: 0.09593 [I 2022-10-28 04:07:14] ...optimizers/trijoint.py.151: total_norm: 18.528143 [I 2022-10-28 04:07:20] ...trap/engines/engine.py.224: train: epoch 105 | batch 560/2304 [I 2022-10-28 04:07:20] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:16:13 [I 2022-10-28 04:07:20] ...trap/engines/engine.py.229: process: 0.55289 | load: 0.00241 [I 2022-10-28 04:07:20] ...trap/engines/engine.py.230: loss: 0.18138 [I 2022-10-28 04:07:20] ...optimizers/trijoint.py.151: total_norm: 18.133549 [I 2022-10-28 04:07:25] ...trap/engines/engine.py.224: train: epoch 105 | batch 570/2304 [I 2022-10-28 04:07:25] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:16:20 [I 2022-10-28 04:07:25] ...trap/engines/engine.py.229: process: 0.57096 | load: 0.00243 [I 2022-10-28 04:07:25] ...trap/engines/engine.py.230: loss: 0.10387 [I 2022-10-28 04:07:25] ...optimizers/trijoint.py.151: total_norm: 14.885649 [I 2022-10-28 04:07:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 580/2304 [I 2022-10-28 04:07:31] ...trap/engines/engine.py.228: elapsed: 0:05:46 | left: 0:16:01 [I 2022-10-28 04:07:31] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00244 [I 2022-10-28 04:07:31] ...trap/engines/engine.py.230: loss: 0.14654 [I 2022-10-28 04:07:31] ...optimizers/trijoint.py.151: total_norm: 12.602668 [I 2022-10-28 04:07:37] ...trap/engines/engine.py.224: train: epoch 105 | batch 590/2304 [I 2022-10-28 04:07:37] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:15:51 [I 2022-10-28 04:07:37] ...trap/engines/engine.py.229: process: 0.56137 | load: 0.00240 [I 2022-10-28 04:07:37] ...trap/engines/engine.py.230: loss: 0.10968 [I 2022-10-28 04:07:37] ...optimizers/trijoint.py.151: total_norm: 13.453046 [I 2022-10-28 04:07:42] ...trap/engines/engine.py.224: train: epoch 105 | batch 600/2304 [I 2022-10-28 04:07:42] ...trap/engines/engine.py.228: elapsed: 0:05:57 | left: 0:15:49 [I 2022-10-28 04:07:42] ...trap/engines/engine.py.229: process: 0.56365 | load: 0.00240 [I 2022-10-28 04:07:42] ...trap/engines/engine.py.230: loss: 0.07861 [I 2022-10-28 04:07:42] ...optimizers/trijoint.py.151: total_norm: 15.388025 [I 2022-10-28 04:07:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 610/2304 [I 2022-10-28 04:07:48] ...trap/engines/engine.py.228: elapsed: 0:06:03 | left: 0:15:39 [I 2022-10-28 04:07:48] ...trap/engines/engine.py.229: process: 0.55580 | load: 0.00239 [I 2022-10-28 04:07:48] ...trap/engines/engine.py.230: loss: 0.10957 [I 2022-10-28 04:07:48] ...optimizers/trijoint.py.151: total_norm: 17.241520 [I 2022-10-28 04:07:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 620/2304 [I 2022-10-28 04:07:53] ...trap/engines/engine.py.228: elapsed: 0:06:08 | left: 0:15:35 [I 2022-10-28 04:07:53] ...trap/engines/engine.py.229: process: 0.56419 | load: 0.00241 [I 2022-10-28 04:07:53] ...trap/engines/engine.py.230: loss: 0.13212 [I 2022-10-28 04:07:53] ...optimizers/trijoint.py.151: total_norm: 15.470228 [I 2022-10-28 04:07:59] ...trap/engines/engine.py.224: train: epoch 105 | batch 630/2304 [I 2022-10-28 04:07:59] ...trap/engines/engine.py.228: elapsed: 0:06:14 | left: 0:15:46 [I 2022-10-28 04:07:59] ...trap/engines/engine.py.229: process: 0.55490 | load: 0.00238 [I 2022-10-28 04:07:59] ...trap/engines/engine.py.230: loss: 0.13006 [I 2022-10-28 04:07:59] ...optimizers/trijoint.py.151: total_norm: 46.224174 [I 2022-10-28 04:08:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 640/2304 [I 2022-10-28 04:08:05] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:31 [I 2022-10-28 04:08:05] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00242 [I 2022-10-28 04:08:05] ...trap/engines/engine.py.230: loss: 0.14759 [I 2022-10-28 04:08:05] ...optimizers/trijoint.py.151: total_norm: 11.180082 [I 2022-10-28 04:08:10] ...trap/engines/engine.py.224: train: epoch 105 | batch 650/2304 [I 2022-10-28 04:08:10] ...trap/engines/engine.py.228: elapsed: 0:06:25 | left: 0:15:27 [I 2022-10-28 04:08:10] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00240 [I 2022-10-28 04:08:10] ...trap/engines/engine.py.230: loss: 0.10570 [I 2022-10-28 04:08:10] ...optimizers/trijoint.py.151: total_norm: 11.969872 [I 2022-10-28 04:08:16] ...trap/engines/engine.py.224: train: epoch 105 | batch 660/2304 [I 2022-10-28 04:08:16] ...trap/engines/engine.py.228: elapsed: 0:06:31 | left: 0:15:10 [I 2022-10-28 04:08:16] ...trap/engines/engine.py.229: process: 0.54204 | load: 0.00239 [I 2022-10-28 04:08:16] ...trap/engines/engine.py.230: loss: 0.08422 [I 2022-10-28 04:08:16] ...optimizers/trijoint.py.151: total_norm: 11.052263 [I 2022-10-28 04:08:21] ...trap/engines/engine.py.224: train: epoch 105 | batch 670/2304 [I 2022-10-28 04:08:21] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:15:05 [I 2022-10-28 04:08:21] ...trap/engines/engine.py.229: process: 0.55844 | load: 0.00244 [I 2022-10-28 04:08:21] ...trap/engines/engine.py.230: loss: 0.10937 [I 2022-10-28 04:08:21] ...optimizers/trijoint.py.151: total_norm: 35.605354 [I 2022-10-28 04:08:27] ...trap/engines/engine.py.224: train: epoch 105 | batch 680/2304 [I 2022-10-28 04:08:27] ...trap/engines/engine.py.228: elapsed: 0:06:42 | left: 0:15:05 [I 2022-10-28 04:08:27] ...trap/engines/engine.py.229: process: 0.56829 | load: 0.00236 [I 2022-10-28 04:08:27] ...trap/engines/engine.py.230: loss: 0.13442 [I 2022-10-28 04:08:27] ...optimizers/trijoint.py.151: total_norm: 11.746744 [I 2022-10-28 04:08:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 690/2304 [I 2022-10-28 04:08:32] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:14:53 [I 2022-10-28 04:08:32] ...trap/engines/engine.py.229: process: 0.55033 | load: 0.00240 [I 2022-10-28 04:08:32] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-28 04:08:32] ...optimizers/trijoint.py.151: total_norm: 12.175422 [I 2022-10-28 04:08:38] ...trap/engines/engine.py.224: train: epoch 105 | batch 700/2304 [I 2022-10-28 04:08:38] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:14:54 [I 2022-10-28 04:08:38] ...trap/engines/engine.py.229: process: 0.55662 | load: 0.00240 [I 2022-10-28 04:08:38] ...trap/engines/engine.py.230: loss: 0.08184 [I 2022-10-28 04:08:38] ...optimizers/trijoint.py.151: total_norm: 12.248630 [I 2022-10-28 04:08:44] ...trap/engines/engine.py.224: train: epoch 105 | batch 710/2304 [I 2022-10-28 04:08:44] ...trap/engines/engine.py.228: elapsed: 0:06:59 | left: 0:14:47 [I 2022-10-28 04:08:44] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00246 [I 2022-10-28 04:08:44] ...trap/engines/engine.py.230: loss: 0.11040 [I 2022-10-28 04:08:44] ...optimizers/trijoint.py.151: total_norm: 11.703232 [I 2022-10-28 04:08:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 720/2304 [I 2022-10-28 04:08:49] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:14:41 [I 2022-10-28 04:08:49] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00239 [I 2022-10-28 04:08:49] ...trap/engines/engine.py.230: loss: 0.12373 [I 2022-10-28 04:08:49] ...optimizers/trijoint.py.151: total_norm: 21.279640 [I 2022-10-28 04:08:55] ...trap/engines/engine.py.224: train: epoch 105 | batch 730/2304 [I 2022-10-28 04:08:55] ...trap/engines/engine.py.228: elapsed: 0:07:10 | left: 0:14:40 [I 2022-10-28 04:08:55] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00243 [I 2022-10-28 04:08:55] ...trap/engines/engine.py.230: loss: 0.07863 [I 2022-10-28 04:08:55] ...optimizers/trijoint.py.151: total_norm: 10.928997 [I 2022-10-28 04:09:00] ...trap/engines/engine.py.224: train: epoch 105 | batch 740/2304 [I 2022-10-28 04:09:00] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:31 [I 2022-10-28 04:09:00] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00241 [I 2022-10-28 04:09:00] ...trap/engines/engine.py.230: loss: 0.10090 [I 2022-10-28 04:09:00] ...optimizers/trijoint.py.151: total_norm: 10.400566 [I 2022-10-28 04:09:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 750/2304 [I 2022-10-28 04:09:06] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:14:32 [I 2022-10-28 04:09:06] ...trap/engines/engine.py.229: process: 0.55831 | load: 0.00240 [I 2022-10-28 04:09:06] ...trap/engines/engine.py.230: loss: 0.11732 [I 2022-10-28 04:09:06] ...optimizers/trijoint.py.151: total_norm: 13.983635 [I 2022-10-28 04:09:12] ...trap/engines/engine.py.224: train: epoch 105 | batch 760/2304 [I 2022-10-28 04:09:12] ...trap/engines/engine.py.228: elapsed: 0:07:27 | left: 0:14:25 [I 2022-10-28 04:09:12] ...trap/engines/engine.py.229: process: 0.56887 | load: 0.00244 [I 2022-10-28 04:09:12] ...trap/engines/engine.py.230: loss: 0.12097 [I 2022-10-28 04:09:12] ...optimizers/trijoint.py.151: total_norm: 11.250836 [I 2022-10-28 04:09:17] ...trap/engines/engine.py.224: train: epoch 105 | batch 770/2304 [I 2022-10-28 04:09:17] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:14:08 [I 2022-10-28 04:09:17] ...trap/engines/engine.py.229: process: 0.54410 | load: 0.00243 [I 2022-10-28 04:09:17] ...trap/engines/engine.py.230: loss: 0.08144 [I 2022-10-28 04:09:17] ...optimizers/trijoint.py.151: total_norm: 22.148926 [I 2022-10-28 04:09:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 780/2304 [I 2022-10-28 04:09:23] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:14:05 [I 2022-10-28 04:09:23] ...trap/engines/engine.py.229: process: 0.55524 | load: 0.00241 [I 2022-10-28 04:09:23] ...trap/engines/engine.py.230: loss: 0.16521 [I 2022-10-28 04:09:23] ...optimizers/trijoint.py.151: total_norm: 21.999121 [I 2022-10-28 04:09:28] ...trap/engines/engine.py.224: train: epoch 105 | batch 790/2304 [I 2022-10-28 04:09:28] ...trap/engines/engine.py.228: elapsed: 0:07:44 | left: 0:14:05 [I 2022-10-28 04:09:28] ...trap/engines/engine.py.229: process: 0.55307 | load: 0.00240 [I 2022-10-28 04:09:28] ...trap/engines/engine.py.230: loss: 0.12766 [I 2022-10-28 04:09:28] ...optimizers/trijoint.py.151: total_norm: 20.647150 [I 2022-10-28 04:09:34] ...trap/engines/engine.py.224: train: epoch 105 | batch 800/2304 [I 2022-10-28 04:09:34] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:13:59 [I 2022-10-28 04:09:34] ...trap/engines/engine.py.229: process: 0.55975 | load: 0.00241 [I 2022-10-28 04:09:34] ...trap/engines/engine.py.230: loss: 0.17016 [I 2022-10-28 04:09:34] ...optimizers/trijoint.py.151: total_norm: 23.104908 [I 2022-10-28 04:09:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 810/2304 [I 2022-10-28 04:09:40] ...trap/engines/engine.py.228: elapsed: 0:07:55 | left: 0:13:56 [I 2022-10-28 04:09:40] ...trap/engines/engine.py.229: process: 0.56153 | load: 0.00238 [I 2022-10-28 04:09:40] ...trap/engines/engine.py.230: loss: 0.11614 [I 2022-10-28 04:09:40] ...optimizers/trijoint.py.151: total_norm: 36.203377 [I 2022-10-28 04:09:45] ...trap/engines/engine.py.224: train: epoch 105 | batch 820/2304 [I 2022-10-28 04:09:45] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:13:45 [I 2022-10-28 04:09:45] ...trap/engines/engine.py.229: process: 0.54849 | load: 0.00244 [I 2022-10-28 04:09:45] ...trap/engines/engine.py.230: loss: 0.10342 [I 2022-10-28 04:09:45] ...optimizers/trijoint.py.151: total_norm: 20.934870 [I 2022-10-28 04:09:51] ...trap/engines/engine.py.224: train: epoch 105 | batch 830/2304 [I 2022-10-28 04:09:51] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:13:37 [I 2022-10-28 04:09:51] ...trap/engines/engine.py.229: process: 0.54761 | load: 0.00240 [I 2022-10-28 04:09:51] ...trap/engines/engine.py.230: loss: 0.10062 [I 2022-10-28 04:09:51] ...optimizers/trijoint.py.151: total_norm: 13.694792 [I 2022-10-28 04:09:56] ...trap/engines/engine.py.224: train: epoch 105 | batch 840/2304 [I 2022-10-28 04:09:56] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:28 [I 2022-10-28 04:09:56] ...trap/engines/engine.py.229: process: 0.56556 | load: 0.00241 [I 2022-10-28 04:09:56] ...trap/engines/engine.py.230: loss: 0.19608 [I 2022-10-28 04:09:56] ...optimizers/trijoint.py.151: total_norm: 19.520107 [I 2022-10-28 04:10:02] ...trap/engines/engine.py.224: train: epoch 105 | batch 850/2304 [I 2022-10-28 04:10:02] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:13:34 [I 2022-10-28 04:10:02] ...trap/engines/engine.py.229: process: 0.56464 | load: 0.00240 [I 2022-10-28 04:10:02] ...trap/engines/engine.py.230: loss: 0.08633 [I 2022-10-28 04:10:02] ...optimizers/trijoint.py.151: total_norm: 9.854526 [I 2022-10-28 04:10:08] ...trap/engines/engine.py.224: train: epoch 105 | batch 860/2304 [I 2022-10-28 04:10:08] ...trap/engines/engine.py.228: elapsed: 0:08:23 | left: 0:13:41 [I 2022-10-28 04:10:08] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00243 [I 2022-10-28 04:10:08] ...trap/engines/engine.py.230: loss: 0.17166 [I 2022-10-28 04:10:08] ...optimizers/trijoint.py.151: total_norm: 31.634319 [I 2022-10-28 04:10:13] ...trap/engines/engine.py.224: train: epoch 105 | batch 870/2304 [I 2022-10-28 04:10:13] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:19 [I 2022-10-28 04:10:13] ...trap/engines/engine.py.229: process: 0.54828 | load: 0.00242 [I 2022-10-28 04:10:13] ...trap/engines/engine.py.230: loss: 0.09629 [I 2022-10-28 04:10:13] ...optimizers/trijoint.py.151: total_norm: 10.406599 [I 2022-10-28 04:10:19] ...trap/engines/engine.py.224: train: epoch 105 | batch 880/2304 [I 2022-10-28 04:10:19] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:13:10 [I 2022-10-28 04:10:19] ...trap/engines/engine.py.229: process: 0.55638 | load: 0.00239 [I 2022-10-28 04:10:19] ...trap/engines/engine.py.230: loss: 0.11732 [I 2022-10-28 04:10:19] ...optimizers/trijoint.py.151: total_norm: 9.408307 [I 2022-10-28 04:10:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 890/2304 [I 2022-10-28 04:10:24] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:13:14 [I 2022-10-28 04:10:24] ...trap/engines/engine.py.229: process: 0.56334 | load: 0.00242 [I 2022-10-28 04:10:24] ...trap/engines/engine.py.230: loss: 0.08755 [I 2022-10-28 04:10:24] ...optimizers/trijoint.py.151: total_norm: 10.214610 [I 2022-10-28 04:10:30] ...trap/engines/engine.py.224: train: epoch 105 | batch 900/2304 [I 2022-10-28 04:10:30] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:12:57 [I 2022-10-28 04:10:30] ...trap/engines/engine.py.229: process: 0.55042 | load: 0.00236 [I 2022-10-28 04:10:30] ...trap/engines/engine.py.230: loss: 0.08308 [I 2022-10-28 04:10:30] ...optimizers/trijoint.py.151: total_norm: 15.444133 [I 2022-10-28 04:10:35] ...trap/engines/engine.py.224: train: epoch 105 | batch 910/2304 [I 2022-10-28 04:10:35] ...trap/engines/engine.py.228: elapsed: 0:08:51 | left: 0:12:54 [I 2022-10-28 04:10:35] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00241 [I 2022-10-28 04:10:35] ...trap/engines/engine.py.230: loss: 0.09782 [I 2022-10-28 04:10:35] ...optimizers/trijoint.py.151: total_norm: 17.903231 [I 2022-10-28 04:10:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 920/2304 [I 2022-10-28 04:10:41] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:12:45 [I 2022-10-28 04:10:41] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00283 [I 2022-10-28 04:10:41] ...trap/engines/engine.py.230: loss: 0.12908 [I 2022-10-28 04:10:41] ...optimizers/trijoint.py.151: total_norm: 22.462152 [I 2022-10-28 04:10:47] ...trap/engines/engine.py.224: train: epoch 105 | batch 930/2304 [I 2022-10-28 04:10:47] ...trap/engines/engine.py.228: elapsed: 0:09:02 | left: 0:12:48 [I 2022-10-28 04:10:47] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00242 [I 2022-10-28 04:10:47] ...trap/engines/engine.py.230: loss: 0.12115 [I 2022-10-28 04:10:47] ...optimizers/trijoint.py.151: total_norm: 11.770761 [I 2022-10-28 04:10:52] ...trap/engines/engine.py.224: train: epoch 105 | batch 940/2304 [I 2022-10-28 04:10:52] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:37 [I 2022-10-28 04:10:52] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00241 [I 2022-10-28 04:10:52] ...trap/engines/engine.py.230: loss: 0.13147 [I 2022-10-28 04:10:52] ...optimizers/trijoint.py.151: total_norm: 19.810814 [I 2022-10-28 04:10:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 950/2304 [I 2022-10-28 04:10:58] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:32 [I 2022-10-28 04:10:58] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00242 [I 2022-10-28 04:10:58] ...trap/engines/engine.py.230: loss: 0.17750 [I 2022-10-28 04:10:58] ...optimizers/trijoint.py.151: total_norm: 12.393496 [I 2022-10-28 04:11:03] ...trap/engines/engine.py.224: train: epoch 105 | batch 960/2304 [I 2022-10-28 04:11:03] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:26 [I 2022-10-28 04:11:03] ...trap/engines/engine.py.229: process: 0.54637 | load: 0.00240 [I 2022-10-28 04:11:03] ...trap/engines/engine.py.230: loss: 0.12387 [I 2022-10-28 04:11:03] ...optimizers/trijoint.py.151: total_norm: 16.380077 [I 2022-10-28 04:11:09] ...trap/engines/engine.py.224: train: epoch 105 | batch 970/2304 [I 2022-10-28 04:11:09] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:44 [I 2022-10-28 04:11:09] ...trap/engines/engine.py.229: process: 0.56372 | load: 0.00241 [I 2022-10-28 04:11:09] ...trap/engines/engine.py.230: loss: 0.11273 [I 2022-10-28 04:11:09] ...optimizers/trijoint.py.151: total_norm: 16.856247 [I 2022-10-28 04:11:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 980/2304 [I 2022-10-28 04:11:15] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:26 [I 2022-10-28 04:11:15] ...trap/engines/engine.py.229: process: 0.55553 | load: 0.00315 [I 2022-10-28 04:11:15] ...trap/engines/engine.py.230: loss: 0.16472 [I 2022-10-28 04:11:15] ...optimizers/trijoint.py.151: total_norm: 25.302914 [I 2022-10-28 04:11:20] ...trap/engines/engine.py.224: train: epoch 105 | batch 990/2304 [I 2022-10-28 04:11:20] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:12:19 [I 2022-10-28 04:11:20] ...trap/engines/engine.py.229: process: 0.55676 | load: 0.00240 [I 2022-10-28 04:11:20] ...trap/engines/engine.py.230: loss: 0.08248 [I 2022-10-28 04:11:20] ...optimizers/trijoint.py.151: total_norm: 17.137058 [I 2022-10-28 04:11:26] ...trap/engines/engine.py.224: train: epoch 105 | batch 1000/2304 [I 2022-10-28 04:11:26] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:12:07 [I 2022-10-28 04:11:26] ...trap/engines/engine.py.229: process: 0.55068 | load: 0.00242 [I 2022-10-28 04:11:26] ...trap/engines/engine.py.230: loss: 0.08281 [I 2022-10-28 04:11:26] ...optimizers/trijoint.py.151: total_norm: 14.936861 [I 2022-10-28 04:11:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 1010/2304 [I 2022-10-28 04:11:32] ...trap/engines/engine.py.228: elapsed: 0:09:47 | left: 0:11:58 [I 2022-10-28 04:11:32] ...trap/engines/engine.py.229: process: 0.56342 | load: 0.00243 [I 2022-10-28 04:11:32] ...trap/engines/engine.py.230: loss: 0.12231 [I 2022-10-28 04:11:32] ...optimizers/trijoint.py.151: total_norm: 15.147480 [I 2022-10-28 04:11:37] ...trap/engines/engine.py.224: train: epoch 105 | batch 1020/2304 [I 2022-10-28 04:11:37] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:58 [I 2022-10-28 04:11:37] ...trap/engines/engine.py.229: process: 0.56636 | load: 0.00244 [I 2022-10-28 04:11:37] ...trap/engines/engine.py.230: loss: 0.09130 [I 2022-10-28 04:11:37] ...optimizers/trijoint.py.151: total_norm: 14.854517 [I 2022-10-28 04:11:43] ...trap/engines/engine.py.224: train: epoch 105 | batch 1030/2304 [I 2022-10-28 04:11:43] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:12:15 [I 2022-10-28 04:11:43] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00243 [I 2022-10-28 04:11:43] ...trap/engines/engine.py.230: loss: 0.09631 [I 2022-10-28 04:11:43] ...optimizers/trijoint.py.151: total_norm: 18.116848 [I 2022-10-28 04:11:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 1040/2304 [I 2022-10-28 04:11:48] ...trap/engines/engine.py.228: elapsed: 0:10:04 | left: 0:11:45 [I 2022-10-28 04:11:48] ...trap/engines/engine.py.229: process: 0.54481 | load: 0.00241 [I 2022-10-28 04:11:48] ...trap/engines/engine.py.230: loss: 0.07950 [I 2022-10-28 04:11:48] ...optimizers/trijoint.py.151: total_norm: 11.346155 [I 2022-10-28 04:11:54] ...trap/engines/engine.py.224: train: epoch 105 | batch 1050/2304 [I 2022-10-28 04:11:54] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:39 [I 2022-10-28 04:11:54] ...trap/engines/engine.py.229: process: 0.55025 | load: 0.00240 [I 2022-10-28 04:11:54] ...trap/engines/engine.py.230: loss: 0.15632 [I 2022-10-28 04:11:54] ...optimizers/trijoint.py.151: total_norm: 21.575758 [I 2022-10-28 04:12:00] ...trap/engines/engine.py.224: train: epoch 105 | batch 1060/2304 [I 2022-10-28 04:12:00] ...trap/engines/engine.py.228: elapsed: 0:10:15 | left: 0:11:33 [I 2022-10-28 04:12:00] ...trap/engines/engine.py.229: process: 0.56204 | load: 0.00240 [I 2022-10-28 04:12:00] ...trap/engines/engine.py.230: loss: 0.14358 [I 2022-10-28 04:12:00] ...optimizers/trijoint.py.151: total_norm: 29.345688 [I 2022-10-28 04:12:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 1070/2304 [I 2022-10-28 04:12:05] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:29 [I 2022-10-28 04:12:05] ...trap/engines/engine.py.229: process: 0.56314 | load: 0.00249 [I 2022-10-28 04:12:05] ...trap/engines/engine.py.230: loss: 0.11428 [I 2022-10-28 04:12:05] ...optimizers/trijoint.py.151: total_norm: 11.214426 [I 2022-10-28 04:12:11] ...trap/engines/engine.py.224: train: epoch 105 | batch 1080/2304 [I 2022-10-28 04:12:11] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:25 [I 2022-10-28 04:12:11] ...trap/engines/engine.py.229: process: 0.56653 | load: 0.00251 [I 2022-10-28 04:12:11] ...trap/engines/engine.py.230: loss: 0.11029 [I 2022-10-28 04:12:11] ...optimizers/trijoint.py.151: total_norm: 24.526669 [I 2022-10-28 04:12:16] ...trap/engines/engine.py.224: train: epoch 105 | batch 1090/2304 [I 2022-10-28 04:12:16] ...trap/engines/engine.py.228: elapsed: 0:10:32 | left: 0:11:21 [I 2022-10-28 04:12:16] ...trap/engines/engine.py.229: process: 0.55832 | load: 0.00246 [I 2022-10-28 04:12:16] ...trap/engines/engine.py.230: loss: 0.13972 [I 2022-10-28 04:12:16] ...optimizers/trijoint.py.151: total_norm: 104.605804 [I 2022-10-28 04:12:22] ...trap/engines/engine.py.224: train: epoch 105 | batch 1100/2304 [I 2022-10-28 04:12:22] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:11 [I 2022-10-28 04:12:22] ...trap/engines/engine.py.229: process: 0.54886 | load: 0.00244 [I 2022-10-28 04:12:22] ...trap/engines/engine.py.230: loss: 0.19197 [I 2022-10-28 04:12:22] ...optimizers/trijoint.py.151: total_norm: 34.485985 [I 2022-10-28 04:12:28] ...trap/engines/engine.py.224: train: epoch 105 | batch 1110/2304 [I 2022-10-28 04:12:28] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:11:06 [I 2022-10-28 04:12:28] ...trap/engines/engine.py.229: process: 0.55017 | load: 0.00245 [I 2022-10-28 04:12:28] ...trap/engines/engine.py.230: loss: 0.10725 [I 2022-10-28 04:12:28] ...optimizers/trijoint.py.151: total_norm: 17.862457 [I 2022-10-28 04:12:33] ...trap/engines/engine.py.224: train: epoch 105 | batch 1120/2304 [I 2022-10-28 04:12:33] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:59 [I 2022-10-28 04:12:33] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00242 [I 2022-10-28 04:12:33] ...trap/engines/engine.py.230: loss: 0.12378 [I 2022-10-28 04:12:33] ...optimizers/trijoint.py.151: total_norm: 12.260237 [I 2022-10-28 04:12:39] ...trap/engines/engine.py.224: train: epoch 105 | batch 1130/2304 [I 2022-10-28 04:12:39] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:50 [I 2022-10-28 04:12:39] ...trap/engines/engine.py.229: process: 0.54920 | load: 0.00245 [I 2022-10-28 04:12:39] ...trap/engines/engine.py.230: loss: 0.21018 [I 2022-10-28 04:12:39] ...optimizers/trijoint.py.151: total_norm: 19.108461 [I 2022-10-28 04:12:44] ...trap/engines/engine.py.224: train: epoch 105 | batch 1140/2304 [I 2022-10-28 04:12:44] ...trap/engines/engine.py.228: elapsed: 0:11:00 | left: 0:10:46 [I 2022-10-28 04:12:44] ...trap/engines/engine.py.229: process: 0.55101 | load: 0.00242 [I 2022-10-28 04:12:44] ...trap/engines/engine.py.230: loss: 0.15274 [I 2022-10-28 04:12:44] ...optimizers/trijoint.py.151: total_norm: 32.154915 [I 2022-10-28 04:12:50] ...trap/engines/engine.py.224: train: epoch 105 | batch 1150/2304 [I 2022-10-28 04:12:50] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:43 [I 2022-10-28 04:12:50] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00243 [I 2022-10-28 04:12:50] ...trap/engines/engine.py.230: loss: 0.14450 [I 2022-10-28 04:12:50] ...optimizers/trijoint.py.151: total_norm: 24.494125 [I 2022-10-28 04:12:56] ...trap/engines/engine.py.224: train: epoch 105 | batch 1160/2304 [I 2022-10-28 04:12:56] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:38 [I 2022-10-28 04:12:56] ...trap/engines/engine.py.229: process: 0.56533 | load: 0.00244 [I 2022-10-28 04:12:56] ...trap/engines/engine.py.230: loss: 0.08730 [I 2022-10-28 04:12:56] ...optimizers/trijoint.py.151: total_norm: 8.870325 [I 2022-10-28 04:13:01] ...trap/engines/engine.py.224: train: epoch 105 | batch 1170/2304 [I 2022-10-28 04:13:01] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:31 [I 2022-10-28 04:13:01] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00242 [I 2022-10-28 04:13:01] ...trap/engines/engine.py.230: loss: 0.12722 [I 2022-10-28 04:13:01] ...optimizers/trijoint.py.151: total_norm: 12.549497 [I 2022-10-28 04:13:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 1180/2304 [I 2022-10-28 04:13:07] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:27 [I 2022-10-28 04:13:07] ...trap/engines/engine.py.229: process: 0.56576 | load: 0.00242 [I 2022-10-28 04:13:07] ...trap/engines/engine.py.230: loss: 0.09050 [I 2022-10-28 04:13:07] ...optimizers/trijoint.py.151: total_norm: 9.590496 [I 2022-10-28 04:13:12] ...trap/engines/engine.py.224: train: epoch 105 | batch 1190/2304 [I 2022-10-28 04:13:12] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:18 [I 2022-10-28 04:13:12] ...trap/engines/engine.py.229: process: 0.55397 | load: 0.00245 [I 2022-10-28 04:13:12] ...trap/engines/engine.py.230: loss: 0.13435 [I 2022-10-28 04:13:12] ...optimizers/trijoint.py.151: total_norm: 21.634241 [I 2022-10-28 04:13:18] ...trap/engines/engine.py.224: train: epoch 105 | batch 1200/2304 [I 2022-10-28 04:13:18] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:14 [I 2022-10-28 04:13:18] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00240 [I 2022-10-28 04:13:18] ...trap/engines/engine.py.230: loss: 0.11446 [I 2022-10-28 04:13:18] ...optimizers/trijoint.py.151: total_norm: 11.721381 [I 2022-10-28 04:13:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 1210/2304 [I 2022-10-28 04:13:24] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:10:10 [I 2022-10-28 04:13:24] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00242 [I 2022-10-28 04:13:24] ...trap/engines/engine.py.230: loss: 0.11251 [I 2022-10-28 04:13:24] ...optimizers/trijoint.py.151: total_norm: 13.247602 [I 2022-10-28 04:13:29] ...trap/engines/engine.py.224: train: epoch 105 | batch 1220/2304 [I 2022-10-28 04:13:29] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:10:05 [I 2022-10-28 04:13:29] ...trap/engines/engine.py.229: process: 0.55824 | load: 0.00241 [I 2022-10-28 04:13:29] ...trap/engines/engine.py.230: loss: 0.11990 [I 2022-10-28 04:13:29] ...optimizers/trijoint.py.151: total_norm: 13.817570 [I 2022-10-28 04:13:35] ...trap/engines/engine.py.224: train: epoch 105 | batch 1230/2304 [I 2022-10-28 04:13:35] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:57 [I 2022-10-28 04:13:35] ...trap/engines/engine.py.229: process: 0.54748 | load: 0.00279 [I 2022-10-28 04:13:35] ...trap/engines/engine.py.230: loss: 0.10854 [I 2022-10-28 04:13:35] ...optimizers/trijoint.py.151: total_norm: 13.545116 [I 2022-10-28 04:13:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 1240/2304 [I 2022-10-28 04:13:40] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:50 [I 2022-10-28 04:13:40] ...trap/engines/engine.py.229: process: 0.55431 | load: 0.00245 [I 2022-10-28 04:13:40] ...trap/engines/engine.py.230: loss: 0.12184 [I 2022-10-28 04:13:40] ...optimizers/trijoint.py.151: total_norm: 22.713808 [I 2022-10-28 04:13:46] ...trap/engines/engine.py.224: train: epoch 105 | batch 1250/2304 [I 2022-10-28 04:13:46] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:47 [I 2022-10-28 04:13:46] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00239 [I 2022-10-28 04:13:46] ...trap/engines/engine.py.230: loss: 0.12660 [I 2022-10-28 04:13:46] ...optimizers/trijoint.py.151: total_norm: 12.005721 [I 2022-10-28 04:13:51] ...trap/engines/engine.py.224: train: epoch 105 | batch 1260/2304 [I 2022-10-28 04:13:51] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:41 [I 2022-10-28 04:13:51] ...trap/engines/engine.py.229: process: 0.55877 | load: 0.00242 [I 2022-10-28 04:13:51] ...trap/engines/engine.py.230: loss: 0.08824 [I 2022-10-28 04:13:51] ...optimizers/trijoint.py.151: total_norm: 14.784883 [I 2022-10-28 04:13:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 1270/2304 [I 2022-10-28 04:13:57] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:36 [I 2022-10-28 04:13:57] ...trap/engines/engine.py.229: process: 0.55477 | load: 0.00240 [I 2022-10-28 04:13:57] ...trap/engines/engine.py.230: loss: 0.19559 [I 2022-10-28 04:13:57] ...optimizers/trijoint.py.151: total_norm: 30.301031 [I 2022-10-28 04:14:03] ...trap/engines/engine.py.224: train: epoch 105 | batch 1280/2304 [I 2022-10-28 04:14:03] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:33 [I 2022-10-28 04:14:03] ...trap/engines/engine.py.229: process: 0.56366 | load: 0.00243 [I 2022-10-28 04:14:03] ...trap/engines/engine.py.230: loss: 0.09785 [I 2022-10-28 04:14:03] ...optimizers/trijoint.py.151: total_norm: 22.775629 [I 2022-10-28 04:14:08] ...trap/engines/engine.py.224: train: epoch 105 | batch 1290/2304 [I 2022-10-28 04:14:08] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:26 [I 2022-10-28 04:14:08] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00244 [I 2022-10-28 04:14:08] ...trap/engines/engine.py.230: loss: 0.10565 [I 2022-10-28 04:14:08] ...optimizers/trijoint.py.151: total_norm: 14.151236 [I 2022-10-28 04:14:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 1300/2304 [I 2022-10-28 04:14:14] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:21 [I 2022-10-28 04:14:14] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00242 [I 2022-10-28 04:14:14] ...trap/engines/engine.py.230: loss: 0.09115 [I 2022-10-28 04:14:14] ...optimizers/trijoint.py.151: total_norm: 16.387012 [I 2022-10-28 04:14:19] ...trap/engines/engine.py.224: train: epoch 105 | batch 1310/2304 [I 2022-10-28 04:14:19] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:12 [I 2022-10-28 04:14:19] ...trap/engines/engine.py.229: process: 0.54470 | load: 0.00243 [I 2022-10-28 04:14:19] ...trap/engines/engine.py.230: loss: 0.15520 [I 2022-10-28 04:14:19] ...optimizers/trijoint.py.151: total_norm: 23.450100 [I 2022-10-28 04:14:25] ...trap/engines/engine.py.224: train: epoch 105 | batch 1320/2304 [I 2022-10-28 04:14:25] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:09:07 [I 2022-10-28 04:14:25] ...trap/engines/engine.py.229: process: 0.54558 | load: 0.00241 [I 2022-10-28 04:14:25] ...trap/engines/engine.py.230: loss: 0.09798 [I 2022-10-28 04:14:25] ...optimizers/trijoint.py.151: total_norm: 14.276565 [I 2022-10-28 04:14:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 1330/2304 [I 2022-10-28 04:14:31] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:09:01 [I 2022-10-28 04:14:31] ...trap/engines/engine.py.229: process: 0.55770 | load: 0.00242 [I 2022-10-28 04:14:31] ...trap/engines/engine.py.230: loss: 0.13227 [I 2022-10-28 04:14:31] ...optimizers/trijoint.py.151: total_norm: 18.606277 [I 2022-10-28 04:14:36] ...trap/engines/engine.py.224: train: epoch 105 | batch 1340/2304 [I 2022-10-28 04:14:36] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:58 [I 2022-10-28 04:14:36] ...trap/engines/engine.py.229: process: 0.56843 | load: 0.00241 [I 2022-10-28 04:14:36] ...trap/engines/engine.py.230: loss: 0.12681 [I 2022-10-28 04:14:36] ...optimizers/trijoint.py.151: total_norm: 10.527588 [I 2022-10-28 04:14:42] ...trap/engines/engine.py.224: train: epoch 105 | batch 1350/2304 [I 2022-10-28 04:14:42] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:50 [I 2022-10-28 04:14:42] ...trap/engines/engine.py.229: process: 0.54470 | load: 0.00251 [I 2022-10-28 04:14:42] ...trap/engines/engine.py.230: loss: 0.12451 [I 2022-10-28 04:14:42] ...optimizers/trijoint.py.151: total_norm: 25.860615 [I 2022-10-28 04:14:47] ...trap/engines/engine.py.224: train: epoch 105 | batch 1360/2304 [I 2022-10-28 04:14:47] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:45 [I 2022-10-28 04:14:47] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00245 [I 2022-10-28 04:14:47] ...trap/engines/engine.py.230: loss: 0.14940 [I 2022-10-28 04:14:47] ...optimizers/trijoint.py.151: total_norm: 21.680731 [I 2022-10-28 04:14:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 1370/2304 [I 2022-10-28 04:14:53] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:41 [I 2022-10-28 04:14:53] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00246 [I 2022-10-28 04:14:53] ...trap/engines/engine.py.230: loss: 0.21135 [I 2022-10-28 04:14:53] ...optimizers/trijoint.py.151: total_norm: 22.597752 [I 2022-10-28 04:14:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 1380/2304 [I 2022-10-28 04:14:58] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:34 [I 2022-10-28 04:14:58] ...trap/engines/engine.py.229: process: 0.55493 | load: 0.00245 [I 2022-10-28 04:14:58] ...trap/engines/engine.py.230: loss: 0.12230 [I 2022-10-28 04:14:58] ...optimizers/trijoint.py.151: total_norm: 33.173141 [I 2022-10-28 04:15:04] ...trap/engines/engine.py.224: train: epoch 105 | batch 1390/2304 [I 2022-10-28 04:15:04] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:29 [I 2022-10-28 04:15:04] ...trap/engines/engine.py.229: process: 0.55755 | load: 0.00243 [I 2022-10-28 04:15:04] ...trap/engines/engine.py.230: loss: 0.16237 [I 2022-10-28 04:15:04] ...optimizers/trijoint.py.151: total_norm: 20.247345 [I 2022-10-28 04:15:10] ...trap/engines/engine.py.224: train: epoch 105 | batch 1400/2304 [I 2022-10-28 04:15:10] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:25 [I 2022-10-28 04:15:10] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00245 [I 2022-10-28 04:15:10] ...trap/engines/engine.py.230: loss: 0.13047 [I 2022-10-28 04:15:10] ...optimizers/trijoint.py.151: total_norm: 22.042257 [I 2022-10-28 04:15:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 1410/2304 [I 2022-10-28 04:15:15] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:20 [I 2022-10-28 04:15:15] ...trap/engines/engine.py.229: process: 0.55369 | load: 0.00244 [I 2022-10-28 04:15:15] ...trap/engines/engine.py.230: loss: 0.09625 [I 2022-10-28 04:15:15] ...optimizers/trijoint.py.151: total_norm: 17.458897 [I 2022-10-28 04:15:21] ...trap/engines/engine.py.224: train: epoch 105 | batch 1420/2304 [I 2022-10-28 04:15:21] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:08:17 [I 2022-10-28 04:15:21] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00248 [I 2022-10-28 04:15:21] ...trap/engines/engine.py.230: loss: 0.11815 [I 2022-10-28 04:15:21] ...optimizers/trijoint.py.151: total_norm: 11.528730 [I 2022-10-28 04:15:27] ...trap/engines/engine.py.224: train: epoch 105 | batch 1430/2304 [I 2022-10-28 04:15:27] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:08:07 [I 2022-10-28 04:15:27] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00248 [I 2022-10-28 04:15:27] ...trap/engines/engine.py.230: loss: 0.10584 [I 2022-10-28 04:15:27] ...optimizers/trijoint.py.151: total_norm: 13.569324 [I 2022-10-28 04:15:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 1440/2304 [I 2022-10-28 04:15:32] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:08:03 [I 2022-10-28 04:15:32] ...trap/engines/engine.py.229: process: 0.55615 | load: 0.00243 [I 2022-10-28 04:15:32] ...trap/engines/engine.py.230: loss: 0.12973 [I 2022-10-28 04:15:32] ...optimizers/trijoint.py.151: total_norm: 12.099364 [I 2022-10-28 04:15:38] ...trap/engines/engine.py.224: train: epoch 105 | batch 1450/2304 [I 2022-10-28 04:15:38] ...trap/engines/engine.py.228: elapsed: 0:13:53 | left: 0:07:54 [I 2022-10-28 04:15:38] ...trap/engines/engine.py.229: process: 0.54804 | load: 0.00243 [I 2022-10-28 04:15:38] ...trap/engines/engine.py.230: loss: 0.09304 [I 2022-10-28 04:15:38] ...optimizers/trijoint.py.151: total_norm: 11.171287 [I 2022-10-28 04:15:43] ...trap/engines/engine.py.224: train: epoch 105 | batch 1460/2304 [I 2022-10-28 04:15:43] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:51 [I 2022-10-28 04:15:43] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00241 [I 2022-10-28 04:15:43] ...trap/engines/engine.py.230: loss: 0.15047 [I 2022-10-28 04:15:43] ...optimizers/trijoint.py.151: total_norm: 10.483784 [I 2022-10-28 04:15:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 1470/2304 [I 2022-10-28 04:15:49] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:44 [I 2022-10-28 04:15:49] ...trap/engines/engine.py.229: process: 0.54751 | load: 0.00243 [I 2022-10-28 04:15:49] ...trap/engines/engine.py.230: loss: 0.07988 [I 2022-10-28 04:15:49] ...optimizers/trijoint.py.151: total_norm: 11.349787 [I 2022-10-28 04:15:54] ...trap/engines/engine.py.224: train: epoch 105 | batch 1480/2304 [I 2022-10-28 04:15:54] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:35 [I 2022-10-28 04:15:54] ...trap/engines/engine.py.229: process: 0.54270 | load: 0.00241 [I 2022-10-28 04:15:54] ...trap/engines/engine.py.230: loss: 0.11835 [I 2022-10-28 04:15:54] ...optimizers/trijoint.py.151: total_norm: 20.592237 [I 2022-10-28 04:16:00] ...trap/engines/engine.py.224: train: epoch 105 | batch 1490/2304 [I 2022-10-28 04:16:00] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:27 [I 2022-10-28 04:16:00] ...trap/engines/engine.py.229: process: 0.54484 | load: 0.00243 [I 2022-10-28 04:16:00] ...trap/engines/engine.py.230: loss: 0.11144 [I 2022-10-28 04:16:00] ...optimizers/trijoint.py.151: total_norm: 8.568850 [I 2022-10-28 04:16:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 1500/2304 [I 2022-10-28 04:16:06] ...trap/engines/engine.py.228: elapsed: 0:14:21 | left: 0:07:23 [I 2022-10-28 04:16:06] ...trap/engines/engine.py.229: process: 0.55688 | load: 0.00251 [I 2022-10-28 04:16:06] ...trap/engines/engine.py.230: loss: 0.14474 [I 2022-10-28 04:16:06] ...optimizers/trijoint.py.151: total_norm: 20.205561 [I 2022-10-28 04:16:11] ...trap/engines/engine.py.224: train: epoch 105 | batch 1510/2304 [I 2022-10-28 04:16:11] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:22 [I 2022-10-28 04:16:11] ...trap/engines/engine.py.229: process: 0.55271 | load: 0.00245 [I 2022-10-28 04:16:11] ...trap/engines/engine.py.230: loss: 0.20705 [I 2022-10-28 04:16:11] ...optimizers/trijoint.py.151: total_norm: 32.368649 [I 2022-10-28 04:16:17] ...trap/engines/engine.py.224: train: epoch 105 | batch 1520/2304 [I 2022-10-28 04:16:17] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:07:17 [I 2022-10-28 04:16:17] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00262 [I 2022-10-28 04:16:17] ...trap/engines/engine.py.230: loss: 0.14504 [I 2022-10-28 04:16:17] ...optimizers/trijoint.py.151: total_norm: 17.519850 [I 2022-10-28 04:16:22] ...trap/engines/engine.py.224: train: epoch 105 | batch 1530/2304 [I 2022-10-28 04:16:22] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:07:06 [I 2022-10-28 04:16:22] ...trap/engines/engine.py.229: process: 0.53570 | load: 0.00243 [I 2022-10-28 04:16:22] ...trap/engines/engine.py.230: loss: 0.10174 [I 2022-10-28 04:16:22] ...optimizers/trijoint.py.151: total_norm: 16.458977 [I 2022-10-28 04:16:28] ...trap/engines/engine.py.224: train: epoch 105 | batch 1540/2304 [I 2022-10-28 04:16:28] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:07:03 [I 2022-10-28 04:16:28] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00240 [I 2022-10-28 04:16:28] ...trap/engines/engine.py.230: loss: 0.12907 [I 2022-10-28 04:16:28] ...optimizers/trijoint.py.151: total_norm: 20.694260 [I 2022-10-28 04:16:33] ...trap/engines/engine.py.224: train: epoch 105 | batch 1550/2304 [I 2022-10-28 04:16:33] ...trap/engines/engine.py.228: elapsed: 0:14:49 | left: 0:06:56 [I 2022-10-28 04:16:33] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00244 [I 2022-10-28 04:16:33] ...trap/engines/engine.py.230: loss: 0.13682 [I 2022-10-28 04:16:33] ...optimizers/trijoint.py.151: total_norm: 20.315552 [I 2022-10-28 04:16:39] ...trap/engines/engine.py.224: train: epoch 105 | batch 1560/2304 [I 2022-10-28 04:16:39] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:52 [I 2022-10-28 04:16:39] ...trap/engines/engine.py.229: process: 0.54663 | load: 0.00242 [I 2022-10-28 04:16:39] ...trap/engines/engine.py.230: loss: 0.09666 [I 2022-10-28 04:16:39] ...optimizers/trijoint.py.151: total_norm: 20.277538 [I 2022-10-28 04:16:45] ...trap/engines/engine.py.224: train: epoch 105 | batch 1570/2304 [I 2022-10-28 04:16:45] ...trap/engines/engine.py.228: elapsed: 0:15:00 | left: 0:06:45 [I 2022-10-28 04:16:45] ...trap/engines/engine.py.229: process: 0.55189 | load: 0.00242 [I 2022-10-28 04:16:45] ...trap/engines/engine.py.230: loss: 0.09093 [I 2022-10-28 04:16:45] ...optimizers/trijoint.py.151: total_norm: 17.434330 [I 2022-10-28 04:16:50] ...trap/engines/engine.py.224: train: epoch 105 | batch 1580/2304 [I 2022-10-28 04:16:50] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:53 [I 2022-10-28 04:16:50] ...trap/engines/engine.py.229: process: 0.55833 | load: 0.00242 [I 2022-10-28 04:16:50] ...trap/engines/engine.py.230: loss: 0.09819 [I 2022-10-28 04:16:50] ...optimizers/trijoint.py.151: total_norm: 10.800277 [I 2022-10-28 04:16:56] ...trap/engines/engine.py.224: train: epoch 105 | batch 1590/2304 [I 2022-10-28 04:16:56] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:35 [I 2022-10-28 04:16:56] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00243 [I 2022-10-28 04:16:56] ...trap/engines/engine.py.230: loss: 0.10501 [I 2022-10-28 04:16:56] ...optimizers/trijoint.py.151: total_norm: 12.890894 [I 2022-10-28 04:17:02] ...trap/engines/engine.py.224: train: epoch 105 | batch 1600/2304 [I 2022-10-28 04:17:02] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:39 [I 2022-10-28 04:17:02] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00240 [I 2022-10-28 04:17:02] ...trap/engines/engine.py.230: loss: 0.20232 [I 2022-10-28 04:17:02] ...optimizers/trijoint.py.151: total_norm: 19.495182 [I 2022-10-28 04:17:07] ...trap/engines/engine.py.224: train: epoch 105 | batch 1610/2304 [I 2022-10-28 04:17:07] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:25 [I 2022-10-28 04:17:07] ...trap/engines/engine.py.229: process: 0.55547 | load: 0.00240 [I 2022-10-28 04:17:07] ...trap/engines/engine.py.230: loss: 0.13207 [I 2022-10-28 04:17:07] ...optimizers/trijoint.py.151: total_norm: 21.974583 [I 2022-10-28 04:17:13] ...trap/engines/engine.py.224: train: epoch 105 | batch 1620/2304 [I 2022-10-28 04:17:13] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:06:22 [I 2022-10-28 04:17:13] ...trap/engines/engine.py.229: process: 0.56522 | load: 0.00239 [I 2022-10-28 04:17:13] ...trap/engines/engine.py.230: loss: 0.16564 [I 2022-10-28 04:17:13] ...optimizers/trijoint.py.151: total_norm: 27.143011 [I 2022-10-28 04:17:18] ...trap/engines/engine.py.224: train: epoch 105 | batch 1630/2304 [I 2022-10-28 04:17:18] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:16 [I 2022-10-28 04:17:18] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00243 [I 2022-10-28 04:17:18] ...trap/engines/engine.py.230: loss: 0.13061 [I 2022-10-28 04:17:18] ...optimizers/trijoint.py.151: total_norm: 12.495902 [I 2022-10-28 04:17:24] ...trap/engines/engine.py.224: train: epoch 105 | batch 1640/2304 [I 2022-10-28 04:17:24] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:06:08 [I 2022-10-28 04:17:24] ...trap/engines/engine.py.229: process: 0.54679 | load: 0.00242 [I 2022-10-28 04:17:24] ...trap/engines/engine.py.230: loss: 0.15548 [I 2022-10-28 04:17:24] ...optimizers/trijoint.py.151: total_norm: 27.532600 [I 2022-10-28 04:17:29] ...trap/engines/engine.py.224: train: epoch 105 | batch 1650/2304 [I 2022-10-28 04:17:29] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:06:04 [I 2022-10-28 04:17:29] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00242 [I 2022-10-28 04:17:29] ...trap/engines/engine.py.230: loss: 0.15847 [I 2022-10-28 04:17:29] ...optimizers/trijoint.py.151: total_norm: 19.393560 [I 2022-10-28 04:17:35] ...trap/engines/engine.py.224: train: epoch 105 | batch 1660/2304 [I 2022-10-28 04:17:35] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:05:59 [I 2022-10-28 04:17:35] ...trap/engines/engine.py.229: process: 0.56428 | load: 0.00240 [I 2022-10-28 04:17:35] ...trap/engines/engine.py.230: loss: 0.10619 [I 2022-10-28 04:17:35] ...optimizers/trijoint.py.151: total_norm: 9.160148 [I 2022-10-28 04:17:41] ...trap/engines/engine.py.224: train: epoch 105 | batch 1670/2304 [I 2022-10-28 04:17:41] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:55 [I 2022-10-28 04:17:41] ...trap/engines/engine.py.229: process: 0.54876 | load: 0.00243 [I 2022-10-28 04:17:41] ...trap/engines/engine.py.230: loss: 0.10667 [I 2022-10-28 04:17:41] ...optimizers/trijoint.py.151: total_norm: 11.483791 [I 2022-10-28 04:17:46] ...trap/engines/engine.py.224: train: epoch 105 | batch 1680/2304 [I 2022-10-28 04:17:46] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:47 [I 2022-10-28 04:17:46] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00242 [I 2022-10-28 04:17:46] ...trap/engines/engine.py.230: loss: 0.12519 [I 2022-10-28 04:17:46] ...optimizers/trijoint.py.151: total_norm: 17.769981 [I 2022-10-28 04:17:52] ...trap/engines/engine.py.224: train: epoch 105 | batch 1690/2304 [I 2022-10-28 04:17:52] ...trap/engines/engine.py.228: elapsed: 0:16:07 | left: 0:05:40 [I 2022-10-28 04:17:52] ...trap/engines/engine.py.229: process: 0.55209 | load: 0.00260 [I 2022-10-28 04:17:52] ...trap/engines/engine.py.230: loss: 0.10476 [I 2022-10-28 04:17:52] ...optimizers/trijoint.py.151: total_norm: 12.598368 [I 2022-10-28 04:17:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 1700/2304 [I 2022-10-28 04:17:57] ...trap/engines/engine.py.228: elapsed: 0:16:13 | left: 0:05:35 [I 2022-10-28 04:17:57] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00242 [I 2022-10-28 04:17:57] ...trap/engines/engine.py.230: loss: 0.16738 [I 2022-10-28 04:17:57] ...optimizers/trijoint.py.151: total_norm: 35.402512 [I 2022-10-28 04:18:03] ...trap/engines/engine.py.224: train: epoch 105 | batch 1710/2304 [I 2022-10-28 04:18:03] ...trap/engines/engine.py.228: elapsed: 0:16:18 | left: 0:05:32 [I 2022-10-28 04:18:03] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00241 [I 2022-10-28 04:18:03] ...trap/engines/engine.py.230: loss: 0.15932 [I 2022-10-28 04:18:03] ...optimizers/trijoint.py.151: total_norm: 18.438120 [I 2022-10-28 04:18:09] ...trap/engines/engine.py.224: train: epoch 105 | batch 1720/2304 [I 2022-10-28 04:18:09] ...trap/engines/engine.py.228: elapsed: 0:16:24 | left: 0:05:25 [I 2022-10-28 04:18:09] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00241 [I 2022-10-28 04:18:09] ...trap/engines/engine.py.230: loss: 0.09984 [I 2022-10-28 04:18:09] ...optimizers/trijoint.py.151: total_norm: 29.778467 [I 2022-10-28 04:18:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 1730/2304 [I 2022-10-28 04:18:14] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:05:20 [I 2022-10-28 04:18:14] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00250 [I 2022-10-28 04:18:14] ...trap/engines/engine.py.230: loss: 0.13793 [I 2022-10-28 04:18:14] ...optimizers/trijoint.py.151: total_norm: 31.807829 [I 2022-10-28 04:18:20] ...trap/engines/engine.py.224: train: epoch 105 | batch 1740/2304 [I 2022-10-28 04:18:20] ...trap/engines/engine.py.228: elapsed: 0:16:35 | left: 0:05:15 [I 2022-10-28 04:18:20] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00245 [I 2022-10-28 04:18:20] ...trap/engines/engine.py.230: loss: 0.10401 [I 2022-10-28 04:18:20] ...optimizers/trijoint.py.151: total_norm: 11.860744 [I 2022-10-28 04:18:25] ...trap/engines/engine.py.224: train: epoch 105 | batch 1750/2304 [I 2022-10-28 04:18:25] ...trap/engines/engine.py.228: elapsed: 0:16:41 | left: 0:05:08 [I 2022-10-28 04:18:25] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00245 [I 2022-10-28 04:18:25] ...trap/engines/engine.py.230: loss: 0.10900 [I 2022-10-28 04:18:25] ...optimizers/trijoint.py.151: total_norm: 16.005493 [I 2022-10-28 04:18:31] ...trap/engines/engine.py.224: train: epoch 105 | batch 1760/2304 [I 2022-10-28 04:18:31] ...trap/engines/engine.py.228: elapsed: 0:16:46 | left: 0:05:05 [I 2022-10-28 04:18:31] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00243 [I 2022-10-28 04:18:31] ...trap/engines/engine.py.230: loss: 0.13172 [I 2022-10-28 04:18:31] ...optimizers/trijoint.py.151: total_norm: 11.554326 [I 2022-10-28 04:18:37] ...trap/engines/engine.py.224: train: epoch 105 | batch 1770/2304 [I 2022-10-28 04:18:37] ...trap/engines/engine.py.228: elapsed: 0:16:52 | left: 0:04:58 [I 2022-10-28 04:18:37] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00244 [I 2022-10-28 04:18:37] ...trap/engines/engine.py.230: loss: 0.10362 [I 2022-10-28 04:18:37] ...optimizers/trijoint.py.151: total_norm: 23.080076 [I 2022-10-28 04:18:42] ...trap/engines/engine.py.224: train: epoch 105 | batch 1780/2304 [I 2022-10-28 04:18:42] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:52 [I 2022-10-28 04:18:42] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00239 [I 2022-10-28 04:18:42] ...trap/engines/engine.py.230: loss: 0.09435 [I 2022-10-28 04:18:42] ...optimizers/trijoint.py.151: total_norm: 12.809682 [I 2022-10-28 04:18:48] ...trap/engines/engine.py.224: train: epoch 105 | batch 1790/2304 [I 2022-10-28 04:18:48] ...trap/engines/engine.py.228: elapsed: 0:17:03 | left: 0:04:45 [I 2022-10-28 04:18:48] ...trap/engines/engine.py.229: process: 0.54713 | load: 0.00248 [I 2022-10-28 04:18:48] ...trap/engines/engine.py.230: loss: 0.11146 [I 2022-10-28 04:18:48] ...optimizers/trijoint.py.151: total_norm: 24.157175 [I 2022-10-28 04:18:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 1800/2304 [I 2022-10-28 04:18:53] ...trap/engines/engine.py.228: elapsed: 0:17:09 | left: 0:04:43 [I 2022-10-28 04:18:53] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00248 [I 2022-10-28 04:18:53] ...trap/engines/engine.py.230: loss: 0.18781 [I 2022-10-28 04:18:53] ...optimizers/trijoint.py.151: total_norm: 24.523548 [I 2022-10-28 04:18:59] ...trap/engines/engine.py.224: train: epoch 105 | batch 1810/2304 [I 2022-10-28 04:18:59] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:39 [I 2022-10-28 04:18:59] ...trap/engines/engine.py.229: process: 0.56717 | load: 0.00246 [I 2022-10-28 04:18:59] ...trap/engines/engine.py.230: loss: 0.08088 [I 2022-10-28 04:18:59] ...optimizers/trijoint.py.151: total_norm: 17.099779 [I 2022-10-28 04:19:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 1820/2304 [I 2022-10-28 04:19:05] ...trap/engines/engine.py.228: elapsed: 0:17:20 | left: 0:04:33 [I 2022-10-28 04:19:05] ...trap/engines/engine.py.229: process: 0.57366 | load: 0.00249 [I 2022-10-28 04:19:05] ...trap/engines/engine.py.230: loss: 0.14402 [I 2022-10-28 04:19:05] ...optimizers/trijoint.py.151: total_norm: 11.844810 [I 2022-10-28 04:19:10] ...trap/engines/engine.py.224: train: epoch 105 | batch 1830/2304 [I 2022-10-28 04:19:10] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:23 [I 2022-10-28 04:19:10] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00248 [I 2022-10-28 04:19:10] ...trap/engines/engine.py.230: loss: 0.10042 [I 2022-10-28 04:19:10] ...optimizers/trijoint.py.151: total_norm: 11.979718 [I 2022-10-28 04:19:16] ...trap/engines/engine.py.224: train: epoch 105 | batch 1840/2304 [I 2022-10-28 04:19:16] ...trap/engines/engine.py.228: elapsed: 0:17:31 | left: 0:04:20 [I 2022-10-28 04:19:16] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00242 [I 2022-10-28 04:19:16] ...trap/engines/engine.py.230: loss: 0.13386 [I 2022-10-28 04:19:16] ...optimizers/trijoint.py.151: total_norm: 10.796957 [I 2022-10-28 04:19:22] ...trap/engines/engine.py.224: train: epoch 105 | batch 1850/2304 [I 2022-10-28 04:19:22] ...trap/engines/engine.py.228: elapsed: 0:17:37 | left: 0:04:14 [I 2022-10-28 04:19:22] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00245 [I 2022-10-28 04:19:22] ...trap/engines/engine.py.230: loss: 0.15695 [I 2022-10-28 04:19:22] ...optimizers/trijoint.py.151: total_norm: 12.380527 [I 2022-10-28 04:19:27] ...trap/engines/engine.py.224: train: epoch 105 | batch 1860/2304 [I 2022-10-28 04:19:27] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:04:06 [I 2022-10-28 04:19:27] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00239 [I 2022-10-28 04:19:27] ...trap/engines/engine.py.230: loss: 0.21730 [I 2022-10-28 04:19:27] ...optimizers/trijoint.py.151: total_norm: 21.187599 [I 2022-10-28 04:19:33] ...trap/engines/engine.py.224: train: epoch 105 | batch 1870/2304 [I 2022-10-28 04:19:33] ...trap/engines/engine.py.228: elapsed: 0:17:48 | left: 0:04:02 [I 2022-10-28 04:19:33] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00248 [I 2022-10-28 04:19:33] ...trap/engines/engine.py.230: loss: 0.12034 [I 2022-10-28 04:19:33] ...optimizers/trijoint.py.151: total_norm: 17.275469 [I 2022-10-28 04:19:38] ...trap/engines/engine.py.224: train: epoch 105 | batch 1880/2304 [I 2022-10-28 04:19:38] ...trap/engines/engine.py.228: elapsed: 0:17:54 | left: 0:03:56 [I 2022-10-28 04:19:38] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00250 [I 2022-10-28 04:19:38] ...trap/engines/engine.py.230: loss: 0.11356 [I 2022-10-28 04:19:38] ...optimizers/trijoint.py.151: total_norm: 12.628925 [I 2022-10-28 04:19:44] ...trap/engines/engine.py.224: train: epoch 105 | batch 1890/2304 [I 2022-10-28 04:19:44] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:51 [I 2022-10-28 04:19:44] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00246 [I 2022-10-28 04:19:44] ...trap/engines/engine.py.230: loss: 0.12880 [I 2022-10-28 04:19:44] ...optimizers/trijoint.py.151: total_norm: 13.567964 [I 2022-10-28 04:19:50] ...trap/engines/engine.py.224: train: epoch 105 | batch 1900/2304 [I 2022-10-28 04:19:50] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:45 [I 2022-10-28 04:19:50] ...trap/engines/engine.py.229: process: 0.56469 | load: 0.00247 [I 2022-10-28 04:19:50] ...trap/engines/engine.py.230: loss: 0.11884 [I 2022-10-28 04:19:50] ...optimizers/trijoint.py.151: total_norm: 38.888168 [I 2022-10-28 04:19:55] ...trap/engines/engine.py.224: train: epoch 105 | batch 1910/2304 [I 2022-10-28 04:19:55] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:03:41 [I 2022-10-28 04:19:55] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00245 [I 2022-10-28 04:19:55] ...trap/engines/engine.py.230: loss: 0.08683 [I 2022-10-28 04:19:55] ...optimizers/trijoint.py.151: total_norm: 10.569561 [I 2022-10-28 04:20:01] ...trap/engines/engine.py.224: train: epoch 105 | batch 1920/2304 [I 2022-10-28 04:20:01] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:34 [I 2022-10-28 04:20:01] ...trap/engines/engine.py.229: process: 0.55923 | load: 0.00246 [I 2022-10-28 04:20:01] ...trap/engines/engine.py.230: loss: 0.15815 [I 2022-10-28 04:20:01] ...optimizers/trijoint.py.151: total_norm: 17.877262 [I 2022-10-28 04:20:06] ...trap/engines/engine.py.224: train: epoch 105 | batch 1930/2304 [I 2022-10-28 04:20:06] ...trap/engines/engine.py.228: elapsed: 0:18:22 | left: 0:03:30 [I 2022-10-28 04:20:06] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00245 [I 2022-10-28 04:20:06] ...trap/engines/engine.py.230: loss: 0.12482 [I 2022-10-28 04:20:06] ...optimizers/trijoint.py.151: total_norm: 12.274384 [I 2022-10-28 04:20:12] ...trap/engines/engine.py.224: train: epoch 105 | batch 1940/2304 [I 2022-10-28 04:20:12] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:22 [I 2022-10-28 04:20:12] ...trap/engines/engine.py.229: process: 0.56222 | load: 0.00243 [I 2022-10-28 04:20:12] ...trap/engines/engine.py.230: loss: 0.12141 [I 2022-10-28 04:20:12] ...optimizers/trijoint.py.151: total_norm: 9.758654 [I 2022-10-28 04:20:18] ...trap/engines/engine.py.224: train: epoch 105 | batch 1950/2304 [I 2022-10-28 04:20:18] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:18 [I 2022-10-28 04:20:18] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00247 [I 2022-10-28 04:20:18] ...trap/engines/engine.py.230: loss: 0.09711 [I 2022-10-28 04:20:18] ...optimizers/trijoint.py.151: total_norm: 14.388749 [I 2022-10-28 04:20:23] ...trap/engines/engine.py.224: train: epoch 105 | batch 1960/2304 [I 2022-10-28 04:20:23] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:03:13 [I 2022-10-28 04:20:23] ...trap/engines/engine.py.229: process: 0.58831 | load: 0.00243 [I 2022-10-28 04:20:23] ...trap/engines/engine.py.230: loss: 0.11940 [I 2022-10-28 04:20:23] ...optimizers/trijoint.py.151: total_norm: 13.128889 [I 2022-10-28 04:20:29] ...trap/engines/engine.py.224: train: epoch 105 | batch 1970/2304 [I 2022-10-28 04:20:29] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:03:05 [I 2022-10-28 04:20:29] ...trap/engines/engine.py.229: process: 0.54182 | load: 0.00245 [I 2022-10-28 04:20:29] ...trap/engines/engine.py.230: loss: 0.12916 [I 2022-10-28 04:20:29] ...optimizers/trijoint.py.151: total_norm: 14.845112 [I 2022-10-28 04:20:34] ...trap/engines/engine.py.224: train: epoch 105 | batch 1980/2304 [I 2022-10-28 04:20:34] ...trap/engines/engine.py.228: elapsed: 0:18:50 | left: 0:03:00 [I 2022-10-28 04:20:34] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00244 [I 2022-10-28 04:20:34] ...trap/engines/engine.py.230: loss: 0.12702 [I 2022-10-28 04:20:34] ...optimizers/trijoint.py.151: total_norm: 30.918310 [I 2022-10-28 04:20:40] ...trap/engines/engine.py.224: train: epoch 105 | batch 1990/2304 [I 2022-10-28 04:20:40] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:55 [I 2022-10-28 04:20:40] ...trap/engines/engine.py.229: process: 0.56710 | load: 0.00242 [I 2022-10-28 04:20:40] ...trap/engines/engine.py.230: loss: 0.10071 [I 2022-10-28 04:20:40] ...optimizers/trijoint.py.151: total_norm: 8.896025 [I 2022-10-28 04:20:46] ...trap/engines/engine.py.224: train: epoch 105 | batch 2000/2304 [I 2022-10-28 04:20:46] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:50 [I 2022-10-28 04:20:46] ...trap/engines/engine.py.229: process: 0.57015 | load: 0.00241 [I 2022-10-28 04:20:46] ...trap/engines/engine.py.230: loss: 0.14572 [I 2022-10-28 04:20:46] ...optimizers/trijoint.py.151: total_norm: 9.162776 [I 2022-10-28 04:20:51] ...trap/engines/engine.py.224: train: epoch 105 | batch 2010/2304 [I 2022-10-28 04:20:51] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:02:43 [I 2022-10-28 04:20:51] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00246 [I 2022-10-28 04:20:51] ...trap/engines/engine.py.230: loss: 0.17796 [I 2022-10-28 04:20:51] ...optimizers/trijoint.py.151: total_norm: 18.232048 [I 2022-10-28 04:20:57] ...trap/engines/engine.py.224: train: epoch 105 | batch 2020/2304 [I 2022-10-28 04:20:57] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:37 [I 2022-10-28 04:20:57] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00243 [I 2022-10-28 04:20:57] ...trap/engines/engine.py.230: loss: 0.09562 [I 2022-10-28 04:20:57] ...optimizers/trijoint.py.151: total_norm: 10.061017 [I 2022-10-28 04:21:02] ...trap/engines/engine.py.224: train: epoch 105 | batch 2030/2304 [I 2022-10-28 04:21:02] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:32 [I 2022-10-28 04:21:02] ...trap/engines/engine.py.229: process: 0.55266 | load: 0.00242 [I 2022-10-28 04:21:02] ...trap/engines/engine.py.230: loss: 0.17477 [I 2022-10-28 04:21:02] ...optimizers/trijoint.py.151: total_norm: 23.327015 [I 2022-10-28 04:21:08] ...trap/engines/engine.py.224: train: epoch 105 | batch 2040/2304 [I 2022-10-28 04:21:08] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:28 [I 2022-10-28 04:21:08] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00243 [I 2022-10-28 04:21:08] ...trap/engines/engine.py.230: loss: 0.15481 [I 2022-10-28 04:21:08] ...optimizers/trijoint.py.151: total_norm: 45.784416 [I 2022-10-28 04:21:14] ...trap/engines/engine.py.224: train: epoch 105 | batch 2050/2304 [I 2022-10-28 04:21:14] ...trap/engines/engine.py.228: elapsed: 0:19:29 | left: 0:02:21 [I 2022-10-28 04:21:14] ...trap/engines/engine.py.229: process: 0.55959 | load: 0.00241 [I 2022-10-28 04:21:14] ...trap/engines/engine.py.230: loss: 0.11645 [I 2022-10-28 04:21:14] ...optimizers/trijoint.py.151: total_norm: 19.282316 [I 2022-10-28 04:21:19] ...trap/engines/engine.py.224: train: epoch 105 | batch 2060/2304 [I 2022-10-28 04:21:19] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:02:16 [I 2022-10-28 04:21:19] ...trap/engines/engine.py.229: process: 0.56949 | load: 0.00243 [I 2022-10-28 04:21:19] ...trap/engines/engine.py.230: loss: 0.12702 [I 2022-10-28 04:21:19] ...optimizers/trijoint.py.151: total_norm: 13.249608 [I 2022-10-28 04:21:25] ...trap/engines/engine.py.224: train: epoch 105 | batch 2070/2304 [I 2022-10-28 04:21:25] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:02:10 [I 2022-10-28 04:21:25] ...trap/engines/engine.py.229: process: 0.55838 | load: 0.00242 [I 2022-10-28 04:21:25] ...trap/engines/engine.py.230: loss: 0.08288 [I 2022-10-28 04:21:25] ...optimizers/trijoint.py.151: total_norm: 12.506424 [I 2022-10-28 04:21:30] ...trap/engines/engine.py.224: train: epoch 105 | batch 2080/2304 [I 2022-10-28 04:21:30] ...trap/engines/engine.py.228: elapsed: 0:19:46 | left: 0:02:05 [I 2022-10-28 04:21:30] ...trap/engines/engine.py.229: process: 0.54971 | load: 0.00240 [I 2022-10-28 04:21:30] ...trap/engines/engine.py.230: loss: 0.10492 [I 2022-10-28 04:21:30] ...optimizers/trijoint.py.151: total_norm: 15.647703 [I 2022-10-28 04:21:36] ...trap/engines/engine.py.224: train: epoch 105 | batch 2090/2304 [I 2022-10-28 04:21:36] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:58 [I 2022-10-28 04:21:36] ...trap/engines/engine.py.229: process: 0.56009 | load: 0.00244 [I 2022-10-28 04:21:36] ...trap/engines/engine.py.230: loss: 0.13129 [I 2022-10-28 04:21:36] ...optimizers/trijoint.py.151: total_norm: 6.586421 [I 2022-10-28 04:21:42] ...trap/engines/engine.py.224: train: epoch 105 | batch 2100/2304 [I 2022-10-28 04:21:42] ...trap/engines/engine.py.228: elapsed: 0:19:57 | left: 0:01:53 [I 2022-10-28 04:21:42] ...trap/engines/engine.py.229: process: 0.56374 | load: 0.00242 [I 2022-10-28 04:21:42] ...trap/engines/engine.py.230: loss: 0.12690 [I 2022-10-28 04:21:42] ...optimizers/trijoint.py.151: total_norm: 10.929830 [I 2022-10-28 04:21:47] ...trap/engines/engine.py.224: train: epoch 105 | batch 2110/2304 [I 2022-10-28 04:21:47] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:48 [I 2022-10-28 04:21:47] ...trap/engines/engine.py.229: process: 0.55128 | load: 0.00250 [I 2022-10-28 04:21:47] ...trap/engines/engine.py.230: loss: 0.09964 [I 2022-10-28 04:21:47] ...optimizers/trijoint.py.151: total_norm: 20.823280 [I 2022-10-28 04:21:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 2120/2304 [I 2022-10-28 04:21:53] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:42 [I 2022-10-28 04:21:53] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00249 [I 2022-10-28 04:21:53] ...trap/engines/engine.py.230: loss: 0.09737 [I 2022-10-28 04:21:53] ...optimizers/trijoint.py.151: total_norm: 13.531794 [I 2022-10-28 04:21:58] ...trap/engines/engine.py.224: train: epoch 105 | batch 2130/2304 [I 2022-10-28 04:21:58] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:36 [I 2022-10-28 04:21:58] ...trap/engines/engine.py.229: process: 0.55538 | load: 0.00245 [I 2022-10-28 04:21:58] ...trap/engines/engine.py.230: loss: 0.11466 [I 2022-10-28 04:21:58] ...optimizers/trijoint.py.151: total_norm: 18.849897 [I 2022-10-28 04:22:04] ...trap/engines/engine.py.224: train: epoch 105 | batch 2140/2304 [I 2022-10-28 04:22:04] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:30 [I 2022-10-28 04:22:04] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00248 [I 2022-10-28 04:22:04] ...trap/engines/engine.py.230: loss: 0.10237 [I 2022-10-28 04:22:04] ...optimizers/trijoint.py.151: total_norm: 8.850431 [I 2022-10-28 04:22:09] ...trap/engines/engine.py.224: train: epoch 105 | batch 2150/2304 [I 2022-10-28 04:22:09] ...trap/engines/engine.py.228: elapsed: 0:20:25 | left: 0:01:25 [I 2022-10-28 04:22:09] ...trap/engines/engine.py.229: process: 0.55342 | load: 0.00245 [I 2022-10-28 04:22:09] ...trap/engines/engine.py.230: loss: 0.12434 [I 2022-10-28 04:22:09] ...optimizers/trijoint.py.151: total_norm: 27.592327 [I 2022-10-28 04:22:15] ...trap/engines/engine.py.224: train: epoch 105 | batch 2160/2304 [I 2022-10-28 04:22:15] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:01:20 [I 2022-10-28 04:22:15] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00243 [I 2022-10-28 04:22:15] ...trap/engines/engine.py.230: loss: 0.12725 [I 2022-10-28 04:22:15] ...optimizers/trijoint.py.151: total_norm: 14.655536 [I 2022-10-28 04:22:21] ...trap/engines/engine.py.224: train: epoch 105 | batch 2170/2304 [I 2022-10-28 04:22:21] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:15 [I 2022-10-28 04:22:21] ...trap/engines/engine.py.229: process: 0.57215 | load: 0.00243 [I 2022-10-28 04:22:21] ...trap/engines/engine.py.230: loss: 0.13760 [I 2022-10-28 04:22:21] ...optimizers/trijoint.py.151: total_norm: 19.286337 [I 2022-10-28 04:22:26] ...trap/engines/engine.py.224: train: epoch 105 | batch 2180/2304 [I 2022-10-28 04:22:26] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:01:09 [I 2022-10-28 04:22:26] ...trap/engines/engine.py.229: process: 0.54768 | load: 0.00246 [I 2022-10-28 04:22:26] ...trap/engines/engine.py.230: loss: 0.14836 [I 2022-10-28 04:22:26] ...optimizers/trijoint.py.151: total_norm: 11.207807 [I 2022-10-28 04:22:32] ...trap/engines/engine.py.224: train: epoch 105 | batch 2190/2304 [I 2022-10-28 04:22:32] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:01:03 [I 2022-10-28 04:22:32] ...trap/engines/engine.py.229: process: 0.54901 | load: 0.00250 [I 2022-10-28 04:22:32] ...trap/engines/engine.py.230: loss: 0.11562 [I 2022-10-28 04:22:32] ...optimizers/trijoint.py.151: total_norm: 21.451410 [I 2022-10-28 04:22:37] ...trap/engines/engine.py.224: train: epoch 105 | batch 2200/2304 [I 2022-10-28 04:22:37] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:58 [I 2022-10-28 04:22:37] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00246 [I 2022-10-28 04:22:37] ...trap/engines/engine.py.230: loss: 0.11441 [I 2022-10-28 04:22:37] ...optimizers/trijoint.py.151: total_norm: 77.442703 [I 2022-10-28 04:22:43] ...trap/engines/engine.py.224: train: epoch 105 | batch 2210/2304 [I 2022-10-28 04:22:43] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:51 [I 2022-10-28 04:22:43] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00247 [I 2022-10-28 04:22:43] ...trap/engines/engine.py.230: loss: 0.15658 [I 2022-10-28 04:22:43] ...optimizers/trijoint.py.151: total_norm: 10.506935 [I 2022-10-28 04:22:49] ...trap/engines/engine.py.224: train: epoch 105 | batch 2220/2304 [I 2022-10-28 04:22:49] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:46 [I 2022-10-28 04:22:49] ...trap/engines/engine.py.229: process: 0.56314 | load: 0.00245 [I 2022-10-28 04:22:49] ...trap/engines/engine.py.230: loss: 0.10229 [I 2022-10-28 04:22:49] ...optimizers/trijoint.py.151: total_norm: 12.943185 [I 2022-10-28 04:22:54] ...trap/engines/engine.py.224: train: epoch 105 | batch 2230/2304 [I 2022-10-28 04:22:54] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:41 [I 2022-10-28 04:22:54] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00253 [I 2022-10-28 04:22:54] ...trap/engines/engine.py.230: loss: 0.09013 [I 2022-10-28 04:22:54] ...optimizers/trijoint.py.151: total_norm: 12.730503 [I 2022-10-28 04:23:00] ...trap/engines/engine.py.224: train: epoch 105 | batch 2240/2304 [I 2022-10-28 04:23:00] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:35 [I 2022-10-28 04:23:00] ...trap/engines/engine.py.229: process: 0.56797 | load: 0.00251 [I 2022-10-28 04:23:00] ...trap/engines/engine.py.230: loss: 0.11468 [I 2022-10-28 04:23:00] ...optimizers/trijoint.py.151: total_norm: 68.435265 [I 2022-10-28 04:23:05] ...trap/engines/engine.py.224: train: epoch 105 | batch 2250/2304 [I 2022-10-28 04:23:05] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:30 [I 2022-10-28 04:23:05] ...trap/engines/engine.py.229: process: 0.55317 | load: 0.00244 [I 2022-10-28 04:23:05] ...trap/engines/engine.py.230: loss: 0.15472 [I 2022-10-28 04:23:05] ...optimizers/trijoint.py.151: total_norm: 12.687740 [I 2022-10-28 04:23:28] ...trap/engines/engine.py.224: train: epoch 105 | batch 2260/2304 [I 2022-10-28 04:23:28] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:02:54 [I 2022-10-28 04:23:28] ...trap/engines/engine.py.229: process: 17.58515 | load: 0.00247 [I 2022-10-28 04:23:28] ...trap/engines/engine.py.230: loss: 0.11575 [I 2022-10-28 04:23:28] ...optimizers/trijoint.py.151: total_norm: 14.343655 [I 2022-10-28 04:23:34] ...trap/engines/engine.py.224: train: epoch 105 | batch 2270/2304 [I 2022-10-28 04:23:34] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:31 [I 2022-10-28 04:23:34] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00238 [I 2022-10-28 04:23:34] ...trap/engines/engine.py.230: loss: 0.14304 [I 2022-10-28 04:23:34] ...optimizers/trijoint.py.151: total_norm: 20.834049 [I 2022-10-28 04:23:39] ...trap/engines/engine.py.224: train: epoch 105 | batch 2280/2304 [I 2022-10-28 04:23:39] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:00:14 [I 2022-10-28 04:23:39] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00235 [I 2022-10-28 04:23:39] ...trap/engines/engine.py.230: loss: 0.10645 [I 2022-10-28 04:23:39] ...optimizers/trijoint.py.151: total_norm: 9.264259 [I 2022-10-28 04:23:45] ...trap/engines/engine.py.224: train: epoch 105 | batch 2290/2304 [I 2022-10-28 04:23:45] ...trap/engines/engine.py.228: elapsed: 0:22:00 | left: 0:00:07 [I 2022-10-28 04:23:45] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00237 [I 2022-10-28 04:23:45] ...trap/engines/engine.py.230: loss: 0.11454 [I 2022-10-28 04:23:45] ...optimizers/trijoint.py.151: total_norm: 28.271006 [I 2022-10-28 04:23:50] ...trap/engines/engine.py.224: train: epoch 105 | batch 2300/2304 [I 2022-10-28 04:23:50] ...trap/engines/engine.py.228: elapsed: 0:22:06 | left: 0:00:02 [I 2022-10-28 04:23:50] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00015 [I 2022-10-28 04:23:50] ...trap/engines/engine.py.230: loss: 0.13895 [I 2022-10-28 04:23:50] ...optimizers/trijoint.py.151: total_norm: 20.539478 [I 2022-10-28 04:23:53] ...trap/engines/engine.py.224: train: epoch 105 | batch 2304/2304 [I 2022-10-28 04:23:53] ...trap/engines/engine.py.228: elapsed: 0:22:08 | left: 0:00:00 [I 2022-10-28 04:23:53] ...trap/engines/engine.py.229: process: 0.53380 | load: 0.00015 [I 2022-10-28 04:23:53] ...trap/engines/engine.py.230: loss: 0.17291 [I 2022-10-28 04:23:53] ...optimizers/trijoint.py.151: total_norm: 23.590818 [S 2022-10-28 04:23:54] ...trap/engines/engine.py.129: train_epoch.epoch: 105 [S 2022-10-28 04:23:54] ...trap/engines/engine.py.129: train_epoch.loss: 0.1237061970743983 [I 2022-10-28 04:23:54] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 04:23:55] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 105 [I 2022-10-28 04:23:55] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 04:23:57] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 04:23:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 04:23:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 04:24:01] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 04:24:01] ...trap/engines/engine.py.321: eval: epoch 105 | batch 0/510 [I 2022-10-28 04:24:01] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:49:24 [I 2022-10-28 04:24:01] ...trap/engines/engine.py.326: process: 5.81287 | load: 3.38834 [I 2022-10-28 04:24:04] ...trap/engines/engine.py.321: eval: epoch 105 | batch 10/510 [I 2022-10-28 04:24:04] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:41 [I 2022-10-28 04:24:04] ...trap/engines/engine.py.326: process: 0.31930 | load: 0.00017 [I 2022-10-28 04:24:08] ...trap/engines/engine.py.321: eval: epoch 105 | batch 20/510 [I 2022-10-28 04:24:08] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:14 [I 2022-10-28 04:24:08] ...trap/engines/engine.py.326: process: 0.34917 | load: 0.00016 [I 2022-10-28 04:24:11] ...trap/engines/engine.py.321: eval: epoch 105 | batch 30/510 [I 2022-10-28 04:24:11] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:41 [I 2022-10-28 04:24:11] ...trap/engines/engine.py.326: process: 0.32669 | load: 0.00016 [I 2022-10-28 04:24:14] ...trap/engines/engine.py.321: eval: epoch 105 | batch 40/510 [I 2022-10-28 04:24:14] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-28 04:24:14] ...trap/engines/engine.py.326: process: 0.32990 | load: 0.00015 [I 2022-10-28 04:24:18] ...trap/engines/engine.py.321: eval: epoch 105 | batch 50/510 [I 2022-10-28 04:24:18] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:30 [I 2022-10-28 04:24:18] ...trap/engines/engine.py.326: process: 0.33207 | load: 0.00016 [I 2022-10-28 04:24:21] ...trap/engines/engine.py.321: eval: epoch 105 | batch 60/510 [I 2022-10-28 04:24:21] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:28 [I 2022-10-28 04:24:21] ...trap/engines/engine.py.326: process: 0.32243 | load: 0.00018 [I 2022-10-28 04:24:24] ...trap/engines/engine.py.321: eval: epoch 105 | batch 70/510 [I 2022-10-28 04:24:24] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:26 [I 2022-10-28 04:24:24] ...trap/engines/engine.py.326: process: 0.33031 | load: 0.00020 [I 2022-10-28 04:24:28] ...trap/engines/engine.py.321: eval: epoch 105 | batch 80/510 [I 2022-10-28 04:24:28] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:21 [I 2022-10-28 04:24:28] ...trap/engines/engine.py.326: process: 0.33300 | load: 0.00015 [I 2022-10-28 04:24:31] ...trap/engines/engine.py.321: eval: epoch 105 | batch 90/510 [I 2022-10-28 04:24:31] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-28 04:24:31] ...trap/engines/engine.py.326: process: 0.33296 | load: 0.00017 [I 2022-10-28 04:24:34] ...trap/engines/engine.py.321: eval: epoch 105 | batch 100/510 [I 2022-10-28 04:24:34] ...trap/engines/engine.py.325: elapsed: 0:00:38 | left: 0:02:13 [I 2022-10-28 04:24:34] ...trap/engines/engine.py.326: process: 0.28829 | load: 0.00019 [I 2022-10-28 04:24:37] ...trap/engines/engine.py.321: eval: epoch 105 | batch 110/510 [I 2022-10-28 04:24:37] ...trap/engines/engine.py.325: elapsed: 0:00:41 | left: 0:01:56 [I 2022-10-28 04:24:37] ...trap/engines/engine.py.326: process: 0.28179 | load: 0.00022 [I 2022-10-28 04:24:40] ...trap/engines/engine.py.321: eval: epoch 105 | batch 120/510 [I 2022-10-28 04:24:40] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:52 [I 2022-10-28 04:24:40] ...trap/engines/engine.py.326: process: 0.28468 | load: 0.00016 [I 2022-10-28 04:24:43] ...trap/engines/engine.py.321: eval: epoch 105 | batch 130/510 [I 2022-10-28 04:24:43] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-28 04:24:43] ...trap/engines/engine.py.326: process: 0.28602 | load: 0.00017 [I 2022-10-28 04:24:46] ...trap/engines/engine.py.321: eval: epoch 105 | batch 140/510 [I 2022-10-28 04:24:46] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-28 04:24:46] ...trap/engines/engine.py.326: process: 0.28813 | load: 0.00014 [I 2022-10-28 04:24:49] ...trap/engines/engine.py.321: eval: epoch 105 | batch 150/510 [I 2022-10-28 04:24:49] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-28 04:24:49] ...trap/engines/engine.py.326: process: 0.29168 | load: 0.00016 [I 2022-10-28 04:24:52] ...trap/engines/engine.py.321: eval: epoch 105 | batch 160/510 [I 2022-10-28 04:24:52] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-28 04:24:52] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00014 [I 2022-10-28 04:24:54] ...trap/engines/engine.py.321: eval: epoch 105 | batch 170/510 [I 2022-10-28 04:24:54] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 04:24:54] ...trap/engines/engine.py.326: process: 0.28566 | load: 0.00017 [I 2022-10-28 04:24:57] ...trap/engines/engine.py.321: eval: epoch 105 | batch 180/510 [I 2022-10-28 04:24:57] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:36 [I 2022-10-28 04:24:57] ...trap/engines/engine.py.326: process: 0.29081 | load: 0.00016 [I 2022-10-28 04:25:00] ...trap/engines/engine.py.321: eval: epoch 105 | batch 190/510 [I 2022-10-28 04:25:00] ...trap/engines/engine.py.325: elapsed: 0:01:04 | left: 0:01:32 [I 2022-10-28 04:25:00] ...trap/engines/engine.py.326: process: 0.28953 | load: 0.00015 [I 2022-10-28 04:25:03] ...trap/engines/engine.py.321: eval: epoch 105 | batch 200/510 [I 2022-10-28 04:25:03] ...trap/engines/engine.py.325: elapsed: 0:01:07 | left: 0:01:29 [I 2022-10-28 04:25:03] ...trap/engines/engine.py.326: process: 0.28526 | load: 0.00015 [I 2022-10-28 04:25:06] ...trap/engines/engine.py.321: eval: epoch 105 | batch 210/510 [I 2022-10-28 04:25:06] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:25 [I 2022-10-28 04:25:06] ...trap/engines/engine.py.326: process: 0.28463 | load: 0.00016 [I 2022-10-28 04:25:09] ...trap/engines/engine.py.321: eval: epoch 105 | batch 220/510 [I 2022-10-28 04:25:09] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:24 [I 2022-10-28 04:25:09] ...trap/engines/engine.py.326: process: 0.29529 | load: 0.00014 [I 2022-10-28 04:25:12] ...trap/engines/engine.py.321: eval: epoch 105 | batch 230/510 [I 2022-10-28 04:25:12] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-28 04:25:12] ...trap/engines/engine.py.326: process: 0.28965 | load: 0.00016 [I 2022-10-28 04:25:15] ...trap/engines/engine.py.321: eval: epoch 105 | batch 240/510 [I 2022-10-28 04:25:15] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 04:25:15] ...trap/engines/engine.py.326: process: 0.28405 | load: 0.00015 [I 2022-10-28 04:25:18] ...trap/engines/engine.py.321: eval: epoch 105 | batch 250/510 [I 2022-10-28 04:25:18] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-28 04:25:18] ...trap/engines/engine.py.326: process: 0.28309 | load: 0.00015 [I 2022-10-28 04:25:20] ...trap/engines/engine.py.321: eval: epoch 105 | batch 260/510 [I 2022-10-28 04:25:20] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:11 [I 2022-10-28 04:25:20] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00015 [I 2022-10-28 04:25:23] ...trap/engines/engine.py.321: eval: epoch 105 | batch 270/510 [I 2022-10-28 04:25:23] ...trap/engines/engine.py.325: elapsed: 0:01:27 | left: 0:01:08 [I 2022-10-28 04:25:23] ...trap/engines/engine.py.326: process: 0.28550 | load: 0.00015 [I 2022-10-28 04:25:26] ...trap/engines/engine.py.321: eval: epoch 105 | batch 280/510 [I 2022-10-28 04:25:26] ...trap/engines/engine.py.325: elapsed: 0:01:30 | left: 0:01:06 [I 2022-10-28 04:25:26] ...trap/engines/engine.py.326: process: 0.28655 | load: 0.00016 [I 2022-10-28 04:25:29] ...trap/engines/engine.py.321: eval: epoch 105 | batch 290/510 [I 2022-10-28 04:25:29] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-28 04:25:29] ...trap/engines/engine.py.326: process: 0.28590 | load: 0.00020 [I 2022-10-28 04:25:32] ...trap/engines/engine.py.321: eval: epoch 105 | batch 300/510 [I 2022-10-28 04:25:32] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-28 04:25:32] ...trap/engines/engine.py.326: process: 0.28012 | load: 0.00015 [I 2022-10-28 04:25:35] ...trap/engines/engine.py.321: eval: epoch 105 | batch 310/510 [I 2022-10-28 04:25:35] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 04:25:35] ...trap/engines/engine.py.326: process: 0.28437 | load: 0.00015 [I 2022-10-28 04:25:38] ...trap/engines/engine.py.321: eval: epoch 105 | batch 320/510 [I 2022-10-28 04:25:38] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 04:25:38] ...trap/engines/engine.py.326: process: 0.28854 | load: 0.00016 [I 2022-10-28 04:25:41] ...trap/engines/engine.py.321: eval: epoch 105 | batch 330/510 [I 2022-10-28 04:25:41] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-28 04:25:41] ...trap/engines/engine.py.326: process: 0.28979 | load: 0.00032 [I 2022-10-28 04:25:43] ...trap/engines/engine.py.321: eval: epoch 105 | batch 340/510 [I 2022-10-28 04:25:43] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-28 04:25:43] ...trap/engines/engine.py.326: process: 0.29034 | load: 0.00020 [I 2022-10-28 04:25:46] ...trap/engines/engine.py.321: eval: epoch 105 | batch 350/510 [I 2022-10-28 04:25:46] ...trap/engines/engine.py.325: elapsed: 0:01:50 | left: 0:00:46 [I 2022-10-28 04:25:46] ...trap/engines/engine.py.326: process: 0.29730 | load: 0.00014 [I 2022-10-28 04:25:49] ...trap/engines/engine.py.321: eval: epoch 105 | batch 360/510 [I 2022-10-28 04:25:49] ...trap/engines/engine.py.325: elapsed: 0:01:53 | left: 0:00:43 [I 2022-10-28 04:25:49] ...trap/engines/engine.py.326: process: 0.29066 | load: 0.00015 [I 2022-10-28 04:25:52] ...trap/engines/engine.py.321: eval: epoch 105 | batch 370/510 [I 2022-10-28 04:25:52] ...trap/engines/engine.py.325: elapsed: 0:01:56 | left: 0:00:40 [I 2022-10-28 04:25:52] ...trap/engines/engine.py.326: process: 0.29505 | load: 0.00018 [I 2022-10-28 04:25:55] ...trap/engines/engine.py.321: eval: epoch 105 | batch 380/510 [I 2022-10-28 04:25:55] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-28 04:25:55] ...trap/engines/engine.py.326: process: 0.28986 | load: 0.00016 [I 2022-10-28 04:25:58] ...trap/engines/engine.py.321: eval: epoch 105 | batch 390/510 [I 2022-10-28 04:25:58] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 04:25:58] ...trap/engines/engine.py.326: process: 0.29552 | load: 0.00021 [I 2022-10-28 04:26:01] ...trap/engines/engine.py.321: eval: epoch 105 | batch 400/510 [I 2022-10-28 04:26:01] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 04:26:01] ...trap/engines/engine.py.326: process: 0.28488 | load: 0.00019 [I 2022-10-28 04:26:04] ...trap/engines/engine.py.321: eval: epoch 105 | batch 410/510 [I 2022-10-28 04:26:04] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 04:26:04] ...trap/engines/engine.py.326: process: 0.29214 | load: 0.00020 [I 2022-10-28 04:26:06] ...trap/engines/engine.py.321: eval: epoch 105 | batch 420/510 [I 2022-10-28 04:26:06] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 04:26:06] ...trap/engines/engine.py.326: process: 0.28424 | load: 0.00016 [I 2022-10-28 04:26:09] ...trap/engines/engine.py.321: eval: epoch 105 | batch 430/510 [I 2022-10-28 04:26:09] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 04:26:09] ...trap/engines/engine.py.326: process: 0.29009 | load: 0.00015 [I 2022-10-28 04:26:12] ...trap/engines/engine.py.321: eval: epoch 105 | batch 440/510 [I 2022-10-28 04:26:12] ...trap/engines/engine.py.325: elapsed: 0:02:16 | left: 0:00:20 [I 2022-10-28 04:26:12] ...trap/engines/engine.py.326: process: 0.28926 | load: 0.00015 [I 2022-10-28 04:26:15] ...trap/engines/engine.py.321: eval: epoch 105 | batch 450/510 [I 2022-10-28 04:26:15] ...trap/engines/engine.py.325: elapsed: 0:02:19 | left: 0:00:17 [I 2022-10-28 04:26:15] ...trap/engines/engine.py.326: process: 0.28936 | load: 0.00015 [I 2022-10-28 04:26:18] ...trap/engines/engine.py.321: eval: epoch 105 | batch 460/510 [I 2022-10-28 04:26:18] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-28 04:26:18] ...trap/engines/engine.py.326: process: 0.29058 | load: 0.00016 [I 2022-10-28 04:26:21] ...trap/engines/engine.py.321: eval: epoch 105 | batch 470/510 [I 2022-10-28 04:26:21] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-28 04:26:21] ...trap/engines/engine.py.326: process: 0.28733 | load: 0.00017 [I 2022-10-28 04:26:24] ...trap/engines/engine.py.321: eval: epoch 105 | batch 480/510 [I 2022-10-28 04:26:24] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 04:26:24] ...trap/engines/engine.py.326: process: 0.29112 | load: 0.00016 [I 2022-10-28 04:26:27] ...trap/engines/engine.py.321: eval: epoch 105 | batch 490/510 [I 2022-10-28 04:26:27] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 04:26:27] ...trap/engines/engine.py.326: process: 0.29145 | load: 0.00021 [I 2022-10-28 04:26:30] ...trap/engines/engine.py.321: eval: epoch 105 | batch 500/510 [I 2022-10-28 04:26:30] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 04:26:30] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00015 [I 2022-10-28 04:26:32] ...trap/engines/engine.py.321: eval: epoch 105 | batch 510/510 [I 2022-10-28 04:26:32] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 04:26:32] ...trap/engines/engine.py.326: process: 0.29332 | load: 0.00010 [S 2022-10-28 04:26:33] ...trap/engines/engine.py.132: eval_epoch.epoch: 105 [I 2022-10-28 04:26:33] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.738500 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.909400 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.937000 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.732300 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.910100 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.937500 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.015200 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008040 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006132 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012166 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007778 [S 2022-10-28 04:26:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006201 [I 2022-10-28 04:26:39] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 04:26:39] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 04:26:40] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 04:26:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 04:26:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 04:26:45] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 04:26:45] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 04:26:51] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 04:26:51] ...trap/engines/engine.py.163: Training model on trainset for epoch 106 [I 2022-10-28 04:26:51] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 04:26:54] ...trap/engines/engine.py.224: train: epoch 106 | batch 0/2304 [I 2022-10-28 04:26:54] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:46 [I 2022-10-28 04:26:54] ...trap/engines/engine.py.229: process: 2.72863 | load: 2.10498 [I 2022-10-28 04:26:54] ...trap/engines/engine.py.230: loss: 0.09532 [I 2022-10-28 04:26:54] ...optimizers/trijoint.py.151: total_norm: 12.227517 [I 2022-10-28 04:26:59] ...trap/engines/engine.py.224: train: epoch 106 | batch 10/2304 [I 2022-10-28 04:26:59] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:16 [I 2022-10-28 04:26:59] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00436 [I 2022-10-28 04:26:59] ...trap/engines/engine.py.230: loss: 0.11622 [I 2022-10-28 04:26:59] ...optimizers/trijoint.py.151: total_norm: 11.207277 [I 2022-10-28 04:27:05] ...trap/engines/engine.py.224: train: epoch 106 | batch 20/2304 [I 2022-10-28 04:27:05] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:21 [I 2022-10-28 04:27:05] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00304 [I 2022-10-28 04:27:05] ...trap/engines/engine.py.230: loss: 0.12181 [I 2022-10-28 04:27:05] ...optimizers/trijoint.py.151: total_norm: 23.232761 [I 2022-10-28 04:27:11] ...trap/engines/engine.py.224: train: epoch 106 | batch 30/2304 [I 2022-10-28 04:27:11] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-28 04:27:11] ...trap/engines/engine.py.229: process: 0.56067 | load: 0.00360 [I 2022-10-28 04:27:11] ...trap/engines/engine.py.230: loss: 0.10589 [I 2022-10-28 04:27:11] ...optimizers/trijoint.py.151: total_norm: 13.727036 [I 2022-10-28 04:27:16] ...trap/engines/engine.py.224: train: epoch 106 | batch 40/2304 [I 2022-10-28 04:27:16] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:54 [I 2022-10-28 04:27:16] ...trap/engines/engine.py.229: process: 0.55030 | load: 0.00298 [I 2022-10-28 04:27:16] ...trap/engines/engine.py.230: loss: 0.13116 [I 2022-10-28 04:27:16] ...optimizers/trijoint.py.151: total_norm: 10.805314 [I 2022-10-28 04:27:22] ...trap/engines/engine.py.224: train: epoch 106 | batch 50/2304 [I 2022-10-28 04:27:22] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:53 [I 2022-10-28 04:27:22] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00297 [I 2022-10-28 04:27:22] ...trap/engines/engine.py.230: loss: 0.10262 [I 2022-10-28 04:27:22] ...optimizers/trijoint.py.151: total_norm: 19.711273 [I 2022-10-28 04:27:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 60/2304 [I 2022-10-28 04:27:27] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:48 [I 2022-10-28 04:27:27] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00246 [I 2022-10-28 04:27:27] ...trap/engines/engine.py.230: loss: 0.15429 [I 2022-10-28 04:27:27] ...optimizers/trijoint.py.151: total_norm: 27.423506 [I 2022-10-28 04:27:33] ...trap/engines/engine.py.224: train: epoch 106 | batch 70/2304 [I 2022-10-28 04:27:33] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:43 [I 2022-10-28 04:27:33] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00307 [I 2022-10-28 04:27:33] ...trap/engines/engine.py.230: loss: 0.13183 [I 2022-10-28 04:27:33] ...optimizers/trijoint.py.151: total_norm: 18.734011 [I 2022-10-28 04:27:39] ...trap/engines/engine.py.224: train: epoch 106 | batch 80/2304 [I 2022-10-28 04:27:39] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:50 [I 2022-10-28 04:27:39] ...trap/engines/engine.py.229: process: 0.56568 | load: 0.00272 [I 2022-10-28 04:27:39] ...trap/engines/engine.py.230: loss: 0.12008 [I 2022-10-28 04:27:39] ...optimizers/trijoint.py.151: total_norm: 15.213872 [I 2022-10-28 04:27:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 90/2304 [I 2022-10-28 04:27:44] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:43 [I 2022-10-28 04:27:44] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00266 [I 2022-10-28 04:27:44] ...trap/engines/engine.py.230: loss: 0.10829 [I 2022-10-28 04:27:44] ...optimizers/trijoint.py.151: total_norm: 11.308522 [I 2022-10-28 04:27:50] ...trap/engines/engine.py.224: train: epoch 106 | batch 100/2304 [I 2022-10-28 04:27:50] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:29 [I 2022-10-28 04:27:50] ...trap/engines/engine.py.229: process: 0.54832 | load: 0.00265 [I 2022-10-28 04:27:50] ...trap/engines/engine.py.230: loss: 0.17133 [I 2022-10-28 04:27:50] ...optimizers/trijoint.py.151: total_norm: 10.122962 [I 2022-10-28 04:27:55] ...trap/engines/engine.py.224: train: epoch 106 | batch 110/2304 [I 2022-10-28 04:27:55] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:29 [I 2022-10-28 04:27:55] ...trap/engines/engine.py.229: process: 0.56508 | load: 0.00289 [I 2022-10-28 04:27:55] ...trap/engines/engine.py.230: loss: 0.09013 [I 2022-10-28 04:27:55] ...optimizers/trijoint.py.151: total_norm: 14.822556 [I 2022-10-28 04:28:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 120/2304 [I 2022-10-28 04:28:01] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:29 [I 2022-10-28 04:28:01] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00259 [I 2022-10-28 04:28:01] ...trap/engines/engine.py.230: loss: 0.14044 [I 2022-10-28 04:28:01] ...optimizers/trijoint.py.151: total_norm: 27.186111 [I 2022-10-28 04:28:07] ...trap/engines/engine.py.224: train: epoch 106 | batch 130/2304 [I 2022-10-28 04:28:07] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:19:58 [I 2022-10-28 04:28:07] ...trap/engines/engine.py.229: process: 0.53878 | load: 0.00261 [I 2022-10-28 04:28:07] ...trap/engines/engine.py.230: loss: 0.12635 [I 2022-10-28 04:28:07] ...optimizers/trijoint.py.151: total_norm: 11.574977 [I 2022-10-28 04:28:12] ...trap/engines/engine.py.224: train: epoch 106 | batch 140/2304 [I 2022-10-28 04:28:12] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:03 [I 2022-10-28 04:28:12] ...trap/engines/engine.py.229: process: 0.55384 | load: 0.00275 [I 2022-10-28 04:28:12] ...trap/engines/engine.py.230: loss: 0.10803 [I 2022-10-28 04:28:12] ...optimizers/trijoint.py.151: total_norm: 9.564990 [I 2022-10-28 04:28:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 150/2304 [I 2022-10-28 04:28:18] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:09 [I 2022-10-28 04:28:18] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00253 [I 2022-10-28 04:28:18] ...trap/engines/engine.py.230: loss: 0.10559 [I 2022-10-28 04:28:18] ...optimizers/trijoint.py.151: total_norm: 11.816316 [I 2022-10-28 04:28:23] ...trap/engines/engine.py.224: train: epoch 106 | batch 160/2304 [I 2022-10-28 04:28:23] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:00 [I 2022-10-28 04:28:23] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00250 [I 2022-10-28 04:28:23] ...trap/engines/engine.py.230: loss: 0.10272 [I 2022-10-28 04:28:23] ...optimizers/trijoint.py.151: total_norm: 28.837976 [I 2022-10-28 04:28:29] ...trap/engines/engine.py.224: train: epoch 106 | batch 170/2304 [I 2022-10-28 04:28:29] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:53 [I 2022-10-28 04:28:29] ...trap/engines/engine.py.229: process: 0.55985 | load: 0.00259 [I 2022-10-28 04:28:29] ...trap/engines/engine.py.230: loss: 0.10369 [I 2022-10-28 04:28:29] ...optimizers/trijoint.py.151: total_norm: 17.647051 [I 2022-10-28 04:28:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 180/2304 [I 2022-10-28 04:28:35] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:48 [I 2022-10-28 04:28:35] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00250 [I 2022-10-28 04:28:35] ...trap/engines/engine.py.230: loss: 0.12943 [I 2022-10-28 04:28:35] ...optimizers/trijoint.py.151: total_norm: 32.744076 [I 2022-10-28 04:28:40] ...trap/engines/engine.py.224: train: epoch 106 | batch 190/2304 [I 2022-10-28 04:28:40] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:42 [I 2022-10-28 04:28:40] ...trap/engines/engine.py.229: process: 0.56570 | load: 0.00254 [I 2022-10-28 04:28:40] ...trap/engines/engine.py.230: loss: 0.10863 [I 2022-10-28 04:28:40] ...optimizers/trijoint.py.151: total_norm: 13.348987 [I 2022-10-28 04:28:46] ...trap/engines/engine.py.224: train: epoch 106 | batch 200/2304 [I 2022-10-28 04:28:46] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:33 [I 2022-10-28 04:28:46] ...trap/engines/engine.py.229: process: 0.54481 | load: 0.00257 [I 2022-10-28 04:28:46] ...trap/engines/engine.py.230: loss: 0.07227 [I 2022-10-28 04:28:46] ...optimizers/trijoint.py.151: total_norm: 14.940489 [I 2022-10-28 04:28:51] ...trap/engines/engine.py.224: train: epoch 106 | batch 210/2304 [I 2022-10-28 04:28:51] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:31 [I 2022-10-28 04:28:51] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00246 [I 2022-10-28 04:28:51] ...trap/engines/engine.py.230: loss: 0.16013 [I 2022-10-28 04:28:51] ...optimizers/trijoint.py.151: total_norm: 9.448484 [I 2022-10-28 04:28:57] ...trap/engines/engine.py.224: train: epoch 106 | batch 220/2304 [I 2022-10-28 04:28:57] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:24 [I 2022-10-28 04:28:57] ...trap/engines/engine.py.229: process: 0.54771 | load: 0.00244 [I 2022-10-28 04:28:57] ...trap/engines/engine.py.230: loss: 0.16007 [I 2022-10-28 04:28:57] ...optimizers/trijoint.py.151: total_norm: 10.072860 [I 2022-10-28 04:29:03] ...trap/engines/engine.py.224: train: epoch 106 | batch 230/2304 [I 2022-10-28 04:29:03] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:11 [I 2022-10-28 04:29:03] ...trap/engines/engine.py.229: process: 0.54843 | load: 0.00240 [I 2022-10-28 04:29:03] ...trap/engines/engine.py.230: loss: 0.08507 [I 2022-10-28 04:29:03] ...optimizers/trijoint.py.151: total_norm: 9.237415 [I 2022-10-28 04:29:08] ...trap/engines/engine.py.224: train: epoch 106 | batch 240/2304 [I 2022-10-28 04:29:08] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:09 [I 2022-10-28 04:29:08] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00252 [I 2022-10-28 04:29:08] ...trap/engines/engine.py.230: loss: 0.11201 [I 2022-10-28 04:29:08] ...optimizers/trijoint.py.151: total_norm: 32.024902 [I 2022-10-28 04:29:14] ...trap/engines/engine.py.224: train: epoch 106 | batch 250/2304 [I 2022-10-28 04:29:14] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:32 [I 2022-10-28 04:29:14] ...trap/engines/engine.py.229: process: 0.56165 | load: 0.00262 [I 2022-10-28 04:29:14] ...trap/engines/engine.py.230: loss: 0.10279 [I 2022-10-28 04:29:14] ...optimizers/trijoint.py.151: total_norm: 48.603256 [I 2022-10-28 04:29:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 260/2304 [I 2022-10-28 04:29:19] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:05 [I 2022-10-28 04:29:19] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00246 [I 2022-10-28 04:29:19] ...trap/engines/engine.py.230: loss: 0.10398 [I 2022-10-28 04:29:19] ...optimizers/trijoint.py.151: total_norm: 21.211185 [I 2022-10-28 04:29:25] ...trap/engines/engine.py.224: train: epoch 106 | batch 270/2304 [I 2022-10-28 04:29:25] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:49 [I 2022-10-28 04:29:25] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00243 [I 2022-10-28 04:29:25] ...trap/engines/engine.py.230: loss: 0.10351 [I 2022-10-28 04:29:25] ...optimizers/trijoint.py.151: total_norm: 12.586590 [I 2022-10-28 04:29:31] ...trap/engines/engine.py.224: train: epoch 106 | batch 280/2304 [I 2022-10-28 04:29:31] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:42 [I 2022-10-28 04:29:31] ...trap/engines/engine.py.229: process: 0.53870 | load: 0.00250 [I 2022-10-28 04:29:31] ...trap/engines/engine.py.230: loss: 0.11695 [I 2022-10-28 04:29:31] ...optimizers/trijoint.py.151: total_norm: 6.749296 [I 2022-10-28 04:29:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 290/2304 [I 2022-10-28 04:29:36] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:40 [I 2022-10-28 04:29:36] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00244 [I 2022-10-28 04:29:36] ...trap/engines/engine.py.230: loss: 0.11889 [I 2022-10-28 04:29:36] ...optimizers/trijoint.py.151: total_norm: 19.138840 [I 2022-10-28 04:29:42] ...trap/engines/engine.py.224: train: epoch 106 | batch 300/2304 [I 2022-10-28 04:29:42] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:37 [I 2022-10-28 04:29:42] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00242 [I 2022-10-28 04:29:42] ...trap/engines/engine.py.230: loss: 0.09382 [I 2022-10-28 04:29:42] ...optimizers/trijoint.py.151: total_norm: 9.647315 [I 2022-10-28 04:29:47] ...trap/engines/engine.py.224: train: epoch 106 | batch 310/2304 [I 2022-10-28 04:29:47] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:29 [I 2022-10-28 04:29:47] ...trap/engines/engine.py.229: process: 0.56234 | load: 0.00258 [I 2022-10-28 04:29:47] ...trap/engines/engine.py.230: loss: 0.09815 [I 2022-10-28 04:29:47] ...optimizers/trijoint.py.151: total_norm: 8.512983 [I 2022-10-28 04:29:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 320/2304 [I 2022-10-28 04:29:53] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:36 [I 2022-10-28 04:29:53] ...trap/engines/engine.py.229: process: 0.55943 | load: 0.00253 [I 2022-10-28 04:29:53] ...trap/engines/engine.py.230: loss: 0.13352 [I 2022-10-28 04:29:53] ...optimizers/trijoint.py.151: total_norm: 24.688566 [I 2022-10-28 04:29:59] ...trap/engines/engine.py.224: train: epoch 106 | batch 330/2304 [I 2022-10-28 04:29:59] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:20 [I 2022-10-28 04:29:59] ...trap/engines/engine.py.229: process: 0.56828 | load: 0.00246 [I 2022-10-28 04:29:59] ...trap/engines/engine.py.230: loss: 0.11431 [I 2022-10-28 04:29:59] ...optimizers/trijoint.py.151: total_norm: 22.696764 [I 2022-10-28 04:30:04] ...trap/engines/engine.py.224: train: epoch 106 | batch 340/2304 [I 2022-10-28 04:30:04] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:15 [I 2022-10-28 04:30:04] ...trap/engines/engine.py.229: process: 0.56958 | load: 0.00247 [I 2022-10-28 04:30:04] ...trap/engines/engine.py.230: loss: 0.10452 [I 2022-10-28 04:30:04] ...optimizers/trijoint.py.151: total_norm: 11.308289 [I 2022-10-28 04:30:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 350/2304 [I 2022-10-28 04:30:10] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:12 [I 2022-10-28 04:30:10] ...trap/engines/engine.py.229: process: 0.56416 | load: 0.00254 [I 2022-10-28 04:30:10] ...trap/engines/engine.py.230: loss: 0.15771 [I 2022-10-28 04:30:10] ...optimizers/trijoint.py.151: total_norm: 14.966486 [I 2022-10-28 04:30:15] ...trap/engines/engine.py.224: train: epoch 106 | batch 360/2304 [I 2022-10-28 04:30:15] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:10 [I 2022-10-28 04:30:15] ...trap/engines/engine.py.229: process: 0.56486 | load: 0.00243 [I 2022-10-28 04:30:15] ...trap/engines/engine.py.230: loss: 0.08918 [I 2022-10-28 04:30:15] ...optimizers/trijoint.py.151: total_norm: 30.972998 [I 2022-10-28 04:30:21] ...trap/engines/engine.py.224: train: epoch 106 | batch 370/2304 [I 2022-10-28 04:30:21] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:09 [I 2022-10-28 04:30:21] ...trap/engines/engine.py.229: process: 0.56254 | load: 0.00256 [I 2022-10-28 04:30:21] ...trap/engines/engine.py.230: loss: 0.09883 [I 2022-10-28 04:30:21] ...optimizers/trijoint.py.151: total_norm: 20.148998 [I 2022-10-28 04:30:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 380/2304 [I 2022-10-28 04:30:27] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:58 [I 2022-10-28 04:30:27] ...trap/engines/engine.py.229: process: 0.56237 | load: 0.00246 [I 2022-10-28 04:30:27] ...trap/engines/engine.py.230: loss: 0.14100 [I 2022-10-28 04:30:27] ...optimizers/trijoint.py.151: total_norm: 30.547386 [I 2022-10-28 04:30:32] ...trap/engines/engine.py.224: train: epoch 106 | batch 390/2304 [I 2022-10-28 04:30:32] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:49 [I 2022-10-28 04:30:32] ...trap/engines/engine.py.229: process: 0.56339 | load: 0.00246 [I 2022-10-28 04:30:32] ...trap/engines/engine.py.230: loss: 0.12784 [I 2022-10-28 04:30:32] ...optimizers/trijoint.py.151: total_norm: 10.478789 [I 2022-10-28 04:30:38] ...trap/engines/engine.py.224: train: epoch 106 | batch 400/2304 [I 2022-10-28 04:30:38] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:45 [I 2022-10-28 04:30:38] ...trap/engines/engine.py.229: process: 0.54393 | load: 0.00264 [I 2022-10-28 04:30:38] ...trap/engines/engine.py.230: loss: 0.10111 [I 2022-10-28 04:30:38] ...optimizers/trijoint.py.151: total_norm: 10.879608 [I 2022-10-28 04:30:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 410/2304 [I 2022-10-28 04:30:44] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:34 [I 2022-10-28 04:30:44] ...trap/engines/engine.py.229: process: 0.54866 | load: 0.00245 [I 2022-10-28 04:30:44] ...trap/engines/engine.py.230: loss: 0.11946 [I 2022-10-28 04:30:44] ...optimizers/trijoint.py.151: total_norm: 12.153467 [I 2022-10-28 04:30:49] ...trap/engines/engine.py.224: train: epoch 106 | batch 420/2304 [I 2022-10-28 04:30:49] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:29 [I 2022-10-28 04:30:49] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00251 [I 2022-10-28 04:30:49] ...trap/engines/engine.py.230: loss: 0.19093 [I 2022-10-28 04:30:49] ...optimizers/trijoint.py.151: total_norm: 30.337780 [I 2022-10-28 04:30:55] ...trap/engines/engine.py.224: train: epoch 106 | batch 430/2304 [I 2022-10-28 04:30:55] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:14 [I 2022-10-28 04:30:55] ...trap/engines/engine.py.229: process: 0.54885 | load: 0.00249 [I 2022-10-28 04:30:55] ...trap/engines/engine.py.230: loss: 0.07738 [I 2022-10-28 04:30:55] ...optimizers/trijoint.py.151: total_norm: 13.360033 [I 2022-10-28 04:31:00] ...trap/engines/engine.py.224: train: epoch 106 | batch 440/2304 [I 2022-10-28 04:31:00] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:22 [I 2022-10-28 04:31:00] ...trap/engines/engine.py.229: process: 0.56867 | load: 0.00242 [I 2022-10-28 04:31:00] ...trap/engines/engine.py.230: loss: 0.11416 [I 2022-10-28 04:31:00] ...optimizers/trijoint.py.151: total_norm: 23.262976 [I 2022-10-28 04:31:06] ...trap/engines/engine.py.224: train: epoch 106 | batch 450/2304 [I 2022-10-28 04:31:06] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:11 [I 2022-10-28 04:31:06] ...trap/engines/engine.py.229: process: 0.55557 | load: 0.00242 [I 2022-10-28 04:31:06] ...trap/engines/engine.py.230: loss: 0.07993 [I 2022-10-28 04:31:06] ...optimizers/trijoint.py.151: total_norm: 9.120498 [I 2022-10-28 04:31:12] ...trap/engines/engine.py.224: train: epoch 106 | batch 460/2304 [I 2022-10-28 04:31:12] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:11 [I 2022-10-28 04:31:12] ...trap/engines/engine.py.229: process: 0.54962 | load: 0.00254 [I 2022-10-28 04:31:12] ...trap/engines/engine.py.230: loss: 0.15735 [I 2022-10-28 04:31:12] ...optimizers/trijoint.py.151: total_norm: 19.555099 [I 2022-10-28 04:31:17] ...trap/engines/engine.py.224: train: epoch 106 | batch 470/2304 [I 2022-10-28 04:31:17] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:03 [I 2022-10-28 04:31:17] ...trap/engines/engine.py.229: process: 0.55471 | load: 0.00245 [I 2022-10-28 04:31:17] ...trap/engines/engine.py.230: loss: 0.10693 [I 2022-10-28 04:31:17] ...optimizers/trijoint.py.151: total_norm: 17.199259 [I 2022-10-28 04:31:23] ...trap/engines/engine.py.224: train: epoch 106 | batch 480/2304 [I 2022-10-28 04:31:23] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:00 [I 2022-10-28 04:31:23] ...trap/engines/engine.py.229: process: 0.55190 | load: 0.00246 [I 2022-10-28 04:31:23] ...trap/engines/engine.py.230: loss: 0.10242 [I 2022-10-28 04:31:23] ...optimizers/trijoint.py.151: total_norm: 10.611515 [I 2022-10-28 04:31:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 490/2304 [I 2022-10-28 04:31:28] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:17:02 [I 2022-10-28 04:31:28] ...trap/engines/engine.py.229: process: 0.56407 | load: 0.00247 [I 2022-10-28 04:31:28] ...trap/engines/engine.py.230: loss: 0.11224 [I 2022-10-28 04:31:28] ...optimizers/trijoint.py.151: total_norm: 32.434559 [I 2022-10-28 04:31:34] ...trap/engines/engine.py.224: train: epoch 106 | batch 500/2304 [I 2022-10-28 04:31:34] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:45 [I 2022-10-28 04:31:34] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00242 [I 2022-10-28 04:31:34] ...trap/engines/engine.py.230: loss: 0.14263 [I 2022-10-28 04:31:34] ...optimizers/trijoint.py.151: total_norm: 20.978577 [I 2022-10-28 04:31:40] ...trap/engines/engine.py.224: train: epoch 106 | batch 510/2304 [I 2022-10-28 04:31:40] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:40 [I 2022-10-28 04:31:40] ...trap/engines/engine.py.229: process: 0.56065 | load: 0.00244 [I 2022-10-28 04:31:40] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-28 04:31:40] ...optimizers/trijoint.py.151: total_norm: 13.788161 [I 2022-10-28 04:31:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 520/2304 [I 2022-10-28 04:31:45] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:41 [I 2022-10-28 04:31:45] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00245 [I 2022-10-28 04:31:45] ...trap/engines/engine.py.230: loss: 0.11981 [I 2022-10-28 04:31:45] ...optimizers/trijoint.py.151: total_norm: 21.839914 [I 2022-10-28 04:31:51] ...trap/engines/engine.py.224: train: epoch 106 | batch 530/2304 [I 2022-10-28 04:31:51] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:41 [I 2022-10-28 04:31:51] ...trap/engines/engine.py.229: process: 0.57434 | load: 0.02274 [I 2022-10-28 04:31:51] ...trap/engines/engine.py.230: loss: 0.12091 [I 2022-10-28 04:31:51] ...optimizers/trijoint.py.151: total_norm: 12.803821 [I 2022-10-28 04:31:56] ...trap/engines/engine.py.224: train: epoch 106 | batch 540/2304 [I 2022-10-28 04:31:56] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:24 [I 2022-10-28 04:31:56] ...trap/engines/engine.py.229: process: 0.54525 | load: 0.00247 [I 2022-10-28 04:31:56] ...trap/engines/engine.py.230: loss: 0.13596 [I 2022-10-28 04:31:56] ...optimizers/trijoint.py.151: total_norm: 28.839096 [I 2022-10-28 04:32:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 550/2304 [I 2022-10-28 04:32:02] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:18 [I 2022-10-28 04:32:02] ...trap/engines/engine.py.229: process: 0.56643 | load: 0.00246 [I 2022-10-28 04:32:02] ...trap/engines/engine.py.230: loss: 0.11318 [I 2022-10-28 04:32:02] ...optimizers/trijoint.py.151: total_norm: 16.434063 [I 2022-10-28 04:32:08] ...trap/engines/engine.py.224: train: epoch 106 | batch 560/2304 [I 2022-10-28 04:32:08] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:14 [I 2022-10-28 04:32:08] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00244 [I 2022-10-28 04:32:08] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-28 04:32:08] ...optimizers/trijoint.py.151: total_norm: 8.807777 [I 2022-10-28 04:32:13] ...trap/engines/engine.py.224: train: epoch 106 | batch 570/2304 [I 2022-10-28 04:32:13] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:02 [I 2022-10-28 04:32:13] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00244 [I 2022-10-28 04:32:13] ...trap/engines/engine.py.230: loss: 0.07998 [I 2022-10-28 04:32:13] ...optimizers/trijoint.py.151: total_norm: 10.964203 [I 2022-10-28 04:32:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 580/2304 [I 2022-10-28 04:32:19] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:01 [I 2022-10-28 04:32:19] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00244 [I 2022-10-28 04:32:19] ...trap/engines/engine.py.230: loss: 0.11712 [I 2022-10-28 04:32:19] ...optimizers/trijoint.py.151: total_norm: 12.320498 [I 2022-10-28 04:32:24] ...trap/engines/engine.py.224: train: epoch 106 | batch 590/2304 [I 2022-10-28 04:32:24] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:02 [I 2022-10-28 04:32:24] ...trap/engines/engine.py.229: process: 0.56451 | load: 0.00245 [I 2022-10-28 04:32:24] ...trap/engines/engine.py.230: loss: 0.09947 [I 2022-10-28 04:32:24] ...optimizers/trijoint.py.151: total_norm: 11.153482 [I 2022-10-28 04:32:30] ...trap/engines/engine.py.224: train: epoch 106 | batch 600/2304 [I 2022-10-28 04:32:30] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:47 [I 2022-10-28 04:32:30] ...trap/engines/engine.py.229: process: 0.55509 | load: 0.00242 [I 2022-10-28 04:32:30] ...trap/engines/engine.py.230: loss: 0.14629 [I 2022-10-28 04:32:30] ...optimizers/trijoint.py.151: total_norm: 20.499743 [I 2022-10-28 04:32:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 610/2304 [I 2022-10-28 04:32:36] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:46 [I 2022-10-28 04:32:36] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00252 [I 2022-10-28 04:32:36] ...trap/engines/engine.py.230: loss: 0.13087 [I 2022-10-28 04:32:36] ...optimizers/trijoint.py.151: total_norm: 31.278263 [I 2022-10-28 04:32:41] ...trap/engines/engine.py.224: train: epoch 106 | batch 620/2304 [I 2022-10-28 04:32:41] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:37 [I 2022-10-28 04:32:41] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00243 [I 2022-10-28 04:32:41] ...trap/engines/engine.py.230: loss: 0.13429 [I 2022-10-28 04:32:41] ...optimizers/trijoint.py.151: total_norm: 18.459791 [I 2022-10-28 04:32:47] ...trap/engines/engine.py.224: train: epoch 106 | batch 630/2304 [I 2022-10-28 04:32:47] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:28 [I 2022-10-28 04:32:47] ...trap/engines/engine.py.229: process: 0.55158 | load: 0.00244 [I 2022-10-28 04:32:47] ...trap/engines/engine.py.230: loss: 0.12439 [I 2022-10-28 04:32:47] ...optimizers/trijoint.py.151: total_norm: 13.600129 [I 2022-10-28 04:32:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 640/2304 [I 2022-10-28 04:32:52] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:33 [I 2022-10-28 04:32:52] ...trap/engines/engine.py.229: process: 0.55822 | load: 0.00248 [I 2022-10-28 04:32:52] ...trap/engines/engine.py.230: loss: 0.12116 [I 2022-10-28 04:32:52] ...optimizers/trijoint.py.151: total_norm: 21.298618 [I 2022-10-28 04:32:58] ...trap/engines/engine.py.224: train: epoch 106 | batch 650/2304 [I 2022-10-28 04:32:58] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:20 [I 2022-10-28 04:32:58] ...trap/engines/engine.py.229: process: 0.55042 | load: 0.00265 [I 2022-10-28 04:32:58] ...trap/engines/engine.py.230: loss: 0.07703 [I 2022-10-28 04:32:58] ...optimizers/trijoint.py.151: total_norm: 20.890305 [I 2022-10-28 04:33:04] ...trap/engines/engine.py.224: train: epoch 106 | batch 660/2304 [I 2022-10-28 04:33:04] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:22 [I 2022-10-28 04:33:04] ...trap/engines/engine.py.229: process: 0.56056 | load: 0.00245 [I 2022-10-28 04:33:04] ...trap/engines/engine.py.230: loss: 0.11702 [I 2022-10-28 04:33:04] ...optimizers/trijoint.py.151: total_norm: 17.721163 [I 2022-10-28 04:33:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 670/2304 [I 2022-10-28 04:33:09] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:09 [I 2022-10-28 04:33:09] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00242 [I 2022-10-28 04:33:09] ...trap/engines/engine.py.230: loss: 0.10813 [I 2022-10-28 04:33:09] ...optimizers/trijoint.py.151: total_norm: 15.140600 [I 2022-10-28 04:33:15] ...trap/engines/engine.py.224: train: epoch 106 | batch 680/2304 [I 2022-10-28 04:33:15] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:09 [I 2022-10-28 04:33:15] ...trap/engines/engine.py.229: process: 0.56583 | load: 0.00241 [I 2022-10-28 04:33:15] ...trap/engines/engine.py.230: loss: 0.14597 [I 2022-10-28 04:33:15] ...optimizers/trijoint.py.151: total_norm: 26.942045 [I 2022-10-28 04:33:20] ...trap/engines/engine.py.224: train: epoch 106 | batch 690/2304 [I 2022-10-28 04:33:20] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:03 [I 2022-10-28 04:33:20] ...trap/engines/engine.py.229: process: 0.56255 | load: 0.00245 [I 2022-10-28 04:33:20] ...trap/engines/engine.py.230: loss: 0.12595 [I 2022-10-28 04:33:20] ...optimizers/trijoint.py.151: total_norm: 19.263939 [I 2022-10-28 04:33:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 700/2304 [I 2022-10-28 04:33:26] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:46 [I 2022-10-28 04:33:26] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00266 [I 2022-10-28 04:33:26] ...trap/engines/engine.py.230: loss: 0.10323 [I 2022-10-28 04:33:26] ...optimizers/trijoint.py.151: total_norm: 22.193954 [I 2022-10-28 04:33:31] ...trap/engines/engine.py.224: train: epoch 106 | batch 710/2304 [I 2022-10-28 04:33:31] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:50 [I 2022-10-28 04:33:31] ...trap/engines/engine.py.229: process: 0.56422 | load: 0.00241 [I 2022-10-28 04:33:31] ...trap/engines/engine.py.230: loss: 0.16811 [I 2022-10-28 04:33:31] ...optimizers/trijoint.py.151: total_norm: 15.701465 [I 2022-10-28 04:33:37] ...trap/engines/engine.py.224: train: epoch 106 | batch 720/2304 [I 2022-10-28 04:33:37] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:39 [I 2022-10-28 04:33:37] ...trap/engines/engine.py.229: process: 0.55127 | load: 0.00240 [I 2022-10-28 04:33:37] ...trap/engines/engine.py.230: loss: 0.13146 [I 2022-10-28 04:33:37] ...optimizers/trijoint.py.151: total_norm: 19.359468 [I 2022-10-28 04:33:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 730/2304 [I 2022-10-28 04:33:43] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:38 [I 2022-10-28 04:33:43] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00247 [I 2022-10-28 04:33:43] ...trap/engines/engine.py.230: loss: 0.16812 [I 2022-10-28 04:33:43] ...optimizers/trijoint.py.151: total_norm: 13.828301 [I 2022-10-28 04:33:48] ...trap/engines/engine.py.224: train: epoch 106 | batch 740/2304 [I 2022-10-28 04:33:48] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:29 [I 2022-10-28 04:33:48] ...trap/engines/engine.py.229: process: 0.55088 | load: 0.00249 [I 2022-10-28 04:33:48] ...trap/engines/engine.py.230: loss: 0.10130 [I 2022-10-28 04:33:48] ...optimizers/trijoint.py.151: total_norm: 18.680849 [I 2022-10-28 04:33:54] ...trap/engines/engine.py.224: train: epoch 106 | batch 750/2304 [I 2022-10-28 04:33:54] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:31 [I 2022-10-28 04:33:54] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00248 [I 2022-10-28 04:33:54] ...trap/engines/engine.py.230: loss: 0.19350 [I 2022-10-28 04:33:54] ...optimizers/trijoint.py.151: total_norm: 21.824526 [I 2022-10-28 04:33:59] ...trap/engines/engine.py.224: train: epoch 106 | batch 760/2304 [I 2022-10-28 04:33:59] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:14 [I 2022-10-28 04:33:59] ...trap/engines/engine.py.229: process: 0.54888 | load: 0.00245 [I 2022-10-28 04:33:59] ...trap/engines/engine.py.230: loss: 0.11416 [I 2022-10-28 04:33:59] ...optimizers/trijoint.py.151: total_norm: 18.154053 [I 2022-10-28 04:34:05] ...trap/engines/engine.py.224: train: epoch 106 | batch 770/2304 [I 2022-10-28 04:34:05] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:16 [I 2022-10-28 04:34:05] ...trap/engines/engine.py.229: process: 0.56795 | load: 0.00244 [I 2022-10-28 04:34:05] ...trap/engines/engine.py.230: loss: 0.13395 [I 2022-10-28 04:34:05] ...optimizers/trijoint.py.151: total_norm: 17.694410 [I 2022-10-28 04:34:11] ...trap/engines/engine.py.224: train: epoch 106 | batch 780/2304 [I 2022-10-28 04:34:11] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:07 [I 2022-10-28 04:34:11] ...trap/engines/engine.py.229: process: 0.55717 | load: 0.00243 [I 2022-10-28 04:34:11] ...trap/engines/engine.py.230: loss: 0.14512 [I 2022-10-28 04:34:11] ...optimizers/trijoint.py.151: total_norm: 22.000135 [I 2022-10-28 04:34:16] ...trap/engines/engine.py.224: train: epoch 106 | batch 790/2304 [I 2022-10-28 04:34:16] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:04 [I 2022-10-28 04:34:16] ...trap/engines/engine.py.229: process: 0.55893 | load: 0.00239 [I 2022-10-28 04:34:16] ...trap/engines/engine.py.230: loss: 0.09770 [I 2022-10-28 04:34:16] ...optimizers/trijoint.py.151: total_norm: 21.656616 [I 2022-10-28 04:34:22] ...trap/engines/engine.py.224: train: epoch 106 | batch 800/2304 [I 2022-10-28 04:34:22] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:00 [I 2022-10-28 04:34:22] ...trap/engines/engine.py.229: process: 0.55243 | load: 0.00239 [I 2022-10-28 04:34:22] ...trap/engines/engine.py.230: loss: 0.14510 [I 2022-10-28 04:34:22] ...optimizers/trijoint.py.151: total_norm: 17.708330 [I 2022-10-28 04:34:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 810/2304 [I 2022-10-28 04:34:27] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:52 [I 2022-10-28 04:34:27] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00251 [I 2022-10-28 04:34:27] ...trap/engines/engine.py.230: loss: 0.09223 [I 2022-10-28 04:34:27] ...optimizers/trijoint.py.151: total_norm: 10.621260 [I 2022-10-28 04:34:33] ...trap/engines/engine.py.224: train: epoch 106 | batch 820/2304 [I 2022-10-28 04:34:33] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:51 [I 2022-10-28 04:34:33] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00246 [I 2022-10-28 04:34:33] ...trap/engines/engine.py.230: loss: 0.10428 [I 2022-10-28 04:34:33] ...optimizers/trijoint.py.151: total_norm: 12.896604 [I 2022-10-28 04:34:39] ...trap/engines/engine.py.224: train: epoch 106 | batch 830/2304 [I 2022-10-28 04:34:39] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:42 [I 2022-10-28 04:34:39] ...trap/engines/engine.py.229: process: 0.54904 | load: 0.00244 [I 2022-10-28 04:34:39] ...trap/engines/engine.py.230: loss: 0.11245 [I 2022-10-28 04:34:39] ...optimizers/trijoint.py.151: total_norm: 14.038962 [I 2022-10-28 04:34:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 840/2304 [I 2022-10-28 04:34:44] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:37 [I 2022-10-28 04:34:44] ...trap/engines/engine.py.229: process: 0.54846 | load: 0.00242 [I 2022-10-28 04:34:44] ...trap/engines/engine.py.230: loss: 0.10939 [I 2022-10-28 04:34:44] ...optimizers/trijoint.py.151: total_norm: 9.983885 [I 2022-10-28 04:34:50] ...trap/engines/engine.py.224: train: epoch 106 | batch 850/2304 [I 2022-10-28 04:34:50] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:29 [I 2022-10-28 04:34:50] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00246 [I 2022-10-28 04:34:50] ...trap/engines/engine.py.230: loss: 0.07320 [I 2022-10-28 04:34:50] ...optimizers/trijoint.py.151: total_norm: 12.292620 [I 2022-10-28 04:34:55] ...trap/engines/engine.py.224: train: epoch 106 | batch 860/2304 [I 2022-10-28 04:34:55] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:29 [I 2022-10-28 04:34:55] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00244 [I 2022-10-28 04:34:55] ...trap/engines/engine.py.230: loss: 0.11148 [I 2022-10-28 04:34:55] ...optimizers/trijoint.py.151: total_norm: 8.444641 [I 2022-10-28 04:35:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 870/2304 [I 2022-10-28 04:35:01] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:24 [I 2022-10-28 04:35:01] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00240 [I 2022-10-28 04:35:01] ...trap/engines/engine.py.230: loss: 0.14181 [I 2022-10-28 04:35:01] ...optimizers/trijoint.py.151: total_norm: 13.546004 [I 2022-10-28 04:35:07] ...trap/engines/engine.py.224: train: epoch 106 | batch 880/2304 [I 2022-10-28 04:35:07] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:13 [I 2022-10-28 04:35:07] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00247 [I 2022-10-28 04:35:07] ...trap/engines/engine.py.230: loss: 0.06834 [I 2022-10-28 04:35:07] ...optimizers/trijoint.py.151: total_norm: 19.871431 [I 2022-10-28 04:35:12] ...trap/engines/engine.py.224: train: epoch 106 | batch 890/2304 [I 2022-10-28 04:35:12] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:12 [I 2022-10-28 04:35:12] ...trap/engines/engine.py.229: process: 0.56516 | load: 0.00245 [I 2022-10-28 04:35:12] ...trap/engines/engine.py.230: loss: 0.17502 [I 2022-10-28 04:35:12] ...optimizers/trijoint.py.151: total_norm: 32.522785 [I 2022-10-28 04:35:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 900/2304 [I 2022-10-28 04:35:18] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:06 [I 2022-10-28 04:35:18] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00246 [I 2022-10-28 04:35:18] ...trap/engines/engine.py.230: loss: 0.25096 [I 2022-10-28 04:35:18] ...optimizers/trijoint.py.151: total_norm: 18.574232 [I 2022-10-28 04:35:23] ...trap/engines/engine.py.224: train: epoch 106 | batch 910/2304 [I 2022-10-28 04:35:23] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:54 [I 2022-10-28 04:35:23] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00242 [I 2022-10-28 04:35:23] ...trap/engines/engine.py.230: loss: 0.20995 [I 2022-10-28 04:35:23] ...optimizers/trijoint.py.151: total_norm: 41.180962 [I 2022-10-28 04:35:29] ...trap/engines/engine.py.224: train: epoch 106 | batch 920/2304 [I 2022-10-28 04:35:29] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:51 [I 2022-10-28 04:35:29] ...trap/engines/engine.py.229: process: 0.55855 | load: 0.00242 [I 2022-10-28 04:35:29] ...trap/engines/engine.py.230: loss: 0.13938 [I 2022-10-28 04:35:29] ...optimizers/trijoint.py.151: total_norm: 23.911633 [I 2022-10-28 04:35:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 930/2304 [I 2022-10-28 04:35:35] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:52 [I 2022-10-28 04:35:35] ...trap/engines/engine.py.229: process: 0.56422 | load: 0.00241 [I 2022-10-28 04:35:35] ...trap/engines/engine.py.230: loss: 0.14108 [I 2022-10-28 04:35:35] ...optimizers/trijoint.py.151: total_norm: 14.851661 [I 2022-10-28 04:35:40] ...trap/engines/engine.py.224: train: epoch 106 | batch 940/2304 [I 2022-10-28 04:35:40] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:37 [I 2022-10-28 04:35:40] ...trap/engines/engine.py.229: process: 0.55450 | load: 0.00242 [I 2022-10-28 04:35:40] ...trap/engines/engine.py.230: loss: 0.12134 [I 2022-10-28 04:35:40] ...optimizers/trijoint.py.151: total_norm: 13.072225 [I 2022-10-28 04:35:46] ...trap/engines/engine.py.224: train: epoch 106 | batch 950/2304 [I 2022-10-28 04:35:46] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:28 [I 2022-10-28 04:35:46] ...trap/engines/engine.py.229: process: 0.53393 | load: 0.00240 [I 2022-10-28 04:35:46] ...trap/engines/engine.py.230: loss: 0.17875 [I 2022-10-28 04:35:46] ...optimizers/trijoint.py.151: total_norm: 23.230270 [I 2022-10-28 04:35:51] ...trap/engines/engine.py.224: train: epoch 106 | batch 960/2304 [I 2022-10-28 04:35:51] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:28 [I 2022-10-28 04:35:51] ...trap/engines/engine.py.229: process: 0.56394 | load: 0.00239 [I 2022-10-28 04:35:51] ...trap/engines/engine.py.230: loss: 0.09272 [I 2022-10-28 04:35:51] ...optimizers/trijoint.py.151: total_norm: 10.709689 [I 2022-10-28 04:35:57] ...trap/engines/engine.py.224: train: epoch 106 | batch 970/2304 [I 2022-10-28 04:35:57] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:25 [I 2022-10-28 04:35:57] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00243 [I 2022-10-28 04:35:57] ...trap/engines/engine.py.230: loss: 0.14153 [I 2022-10-28 04:35:57] ...optimizers/trijoint.py.151: total_norm: 10.621690 [I 2022-10-28 04:36:03] ...trap/engines/engine.py.224: train: epoch 106 | batch 980/2304 [I 2022-10-28 04:36:03] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:28 [I 2022-10-28 04:36:03] ...trap/engines/engine.py.229: process: 0.56645 | load: 0.00249 [I 2022-10-28 04:36:03] ...trap/engines/engine.py.230: loss: 0.06756 [I 2022-10-28 04:36:03] ...optimizers/trijoint.py.151: total_norm: 10.936532 [I 2022-10-28 04:36:08] ...trap/engines/engine.py.224: train: epoch 106 | batch 990/2304 [I 2022-10-28 04:36:08] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:15 [I 2022-10-28 04:36:08] ...trap/engines/engine.py.229: process: 0.55427 | load: 0.00245 [I 2022-10-28 04:36:08] ...trap/engines/engine.py.230: loss: 0.15632 [I 2022-10-28 04:36:08] ...optimizers/trijoint.py.151: total_norm: 16.487425 [I 2022-10-28 04:36:14] ...trap/engines/engine.py.224: train: epoch 106 | batch 1000/2304 [I 2022-10-28 04:36:14] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:11 [I 2022-10-28 04:36:14] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00246 [I 2022-10-28 04:36:14] ...trap/engines/engine.py.230: loss: 0.10529 [I 2022-10-28 04:36:14] ...optimizers/trijoint.py.151: total_norm: 12.802868 [I 2022-10-28 04:36:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 1010/2304 [I 2022-10-28 04:36:19] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:11:57 [I 2022-10-28 04:36:19] ...trap/engines/engine.py.229: process: 0.54351 | load: 0.00246 [I 2022-10-28 04:36:19] ...trap/engines/engine.py.230: loss: 0.12318 [I 2022-10-28 04:36:19] ...optimizers/trijoint.py.151: total_norm: 37.747650 [I 2022-10-28 04:36:25] ...trap/engines/engine.py.224: train: epoch 106 | batch 1020/2304 [I 2022-10-28 04:36:25] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:51 [I 2022-10-28 04:36:25] ...trap/engines/engine.py.229: process: 0.54608 | load: 0.00242 [I 2022-10-28 04:36:25] ...trap/engines/engine.py.230: loss: 0.11219 [I 2022-10-28 04:36:25] ...optimizers/trijoint.py.151: total_norm: 12.379519 [I 2022-10-28 04:36:31] ...trap/engines/engine.py.224: train: epoch 106 | batch 1030/2304 [I 2022-10-28 04:36:31] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:51 [I 2022-10-28 04:36:31] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00245 [I 2022-10-28 04:36:31] ...trap/engines/engine.py.230: loss: 0.13588 [I 2022-10-28 04:36:31] ...optimizers/trijoint.py.151: total_norm: 12.664231 [I 2022-10-28 04:36:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 1040/2304 [I 2022-10-28 04:36:36] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:45 [I 2022-10-28 04:36:36] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00245 [I 2022-10-28 04:36:36] ...trap/engines/engine.py.230: loss: 0.12889 [I 2022-10-28 04:36:36] ...optimizers/trijoint.py.151: total_norm: 17.914530 [I 2022-10-28 04:36:42] ...trap/engines/engine.py.224: train: epoch 106 | batch 1050/2304 [I 2022-10-28 04:36:42] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:39 [I 2022-10-28 04:36:42] ...trap/engines/engine.py.229: process: 0.56000 | load: 0.00241 [I 2022-10-28 04:36:42] ...trap/engines/engine.py.230: loss: 0.13387 [I 2022-10-28 04:36:42] ...optimizers/trijoint.py.151: total_norm: 13.228314 [I 2022-10-28 04:36:47] ...trap/engines/engine.py.224: train: epoch 106 | batch 1060/2304 [I 2022-10-28 04:36:47] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:32 [I 2022-10-28 04:36:47] ...trap/engines/engine.py.229: process: 0.55656 | load: 0.00239 [I 2022-10-28 04:36:47] ...trap/engines/engine.py.230: loss: 0.08757 [I 2022-10-28 04:36:47] ...optimizers/trijoint.py.151: total_norm: 9.928414 [I 2022-10-28 04:36:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 1070/2304 [I 2022-10-28 04:36:53] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:52 [I 2022-10-28 04:36:53] ...trap/engines/engine.py.229: process: 0.56860 | load: 0.00241 [I 2022-10-28 04:36:53] ...trap/engines/engine.py.230: loss: 0.12988 [I 2022-10-28 04:36:53] ...optimizers/trijoint.py.151: total_norm: 21.876522 [I 2022-10-28 04:36:59] ...trap/engines/engine.py.224: train: epoch 106 | batch 1080/2304 [I 2022-10-28 04:36:59] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:25 [I 2022-10-28 04:36:59] ...trap/engines/engine.py.229: process: 0.55262 | load: 0.00244 [I 2022-10-28 04:36:59] ...trap/engines/engine.py.230: loss: 0.13547 [I 2022-10-28 04:36:59] ...optimizers/trijoint.py.151: total_norm: 17.050323 [I 2022-10-28 04:37:04] ...trap/engines/engine.py.224: train: epoch 106 | batch 1090/2304 [I 2022-10-28 04:37:04] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:27 [I 2022-10-28 04:37:04] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00238 [I 2022-10-28 04:37:04] ...trap/engines/engine.py.230: loss: 0.14843 [I 2022-10-28 04:37:04] ...optimizers/trijoint.py.151: total_norm: 14.259276 [I 2022-10-28 04:37:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 1100/2304 [I 2022-10-28 04:37:10] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:13 [I 2022-10-28 04:37:10] ...trap/engines/engine.py.229: process: 0.56071 | load: 0.00239 [I 2022-10-28 04:37:10] ...trap/engines/engine.py.230: loss: 0.12554 [I 2022-10-28 04:37:10] ...optimizers/trijoint.py.151: total_norm: 23.592443 [I 2022-10-28 04:37:15] ...trap/engines/engine.py.224: train: epoch 106 | batch 1110/2304 [I 2022-10-28 04:37:15] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:09 [I 2022-10-28 04:37:15] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00248 [I 2022-10-28 04:37:15] ...trap/engines/engine.py.230: loss: 0.15223 [I 2022-10-28 04:37:15] ...optimizers/trijoint.py.151: total_norm: 18.302128 [I 2022-10-28 04:37:21] ...trap/engines/engine.py.224: train: epoch 106 | batch 1120/2304 [I 2022-10-28 04:37:21] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:04 [I 2022-10-28 04:37:21] ...trap/engines/engine.py.229: process: 0.56513 | load: 0.00245 [I 2022-10-28 04:37:21] ...trap/engines/engine.py.230: loss: 0.15566 [I 2022-10-28 04:37:21] ...optimizers/trijoint.py.151: total_norm: 10.623506 [I 2022-10-28 04:37:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 1130/2304 [I 2022-10-28 04:37:27] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:56 [I 2022-10-28 04:37:27] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00240 [I 2022-10-28 04:37:27] ...trap/engines/engine.py.230: loss: 0.16940 [I 2022-10-28 04:37:27] ...optimizers/trijoint.py.151: total_norm: 14.018187 [I 2022-10-28 04:37:32] ...trap/engines/engine.py.224: train: epoch 106 | batch 1140/2304 [I 2022-10-28 04:37:32] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:10:50 [I 2022-10-28 04:37:32] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00240 [I 2022-10-28 04:37:32] ...trap/engines/engine.py.230: loss: 0.10154 [I 2022-10-28 04:37:32] ...optimizers/trijoint.py.151: total_norm: 31.905882 [I 2022-10-28 04:37:38] ...trap/engines/engine.py.224: train: epoch 106 | batch 1150/2304 [I 2022-10-28 04:37:38] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:40 [I 2022-10-28 04:37:38] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00244 [I 2022-10-28 04:37:38] ...trap/engines/engine.py.230: loss: 0.11483 [I 2022-10-28 04:37:38] ...optimizers/trijoint.py.151: total_norm: 10.585917 [I 2022-10-28 04:37:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 1160/2304 [I 2022-10-28 04:37:44] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:52 [I 2022-10-28 04:37:44] ...trap/engines/engine.py.229: process: 0.56116 | load: 0.00242 [I 2022-10-28 04:37:44] ...trap/engines/engine.py.230: loss: 0.16981 [I 2022-10-28 04:37:44] ...optimizers/trijoint.py.151: total_norm: 16.909208 [I 2022-10-28 04:37:49] ...trap/engines/engine.py.224: train: epoch 106 | batch 1170/2304 [I 2022-10-28 04:37:49] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:37 [I 2022-10-28 04:37:49] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00240 [I 2022-10-28 04:37:49] ...trap/engines/engine.py.230: loss: 0.13072 [I 2022-10-28 04:37:49] ...optimizers/trijoint.py.151: total_norm: 11.629848 [I 2022-10-28 04:37:55] ...trap/engines/engine.py.224: train: epoch 106 | batch 1180/2304 [I 2022-10-28 04:37:55] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:26 [I 2022-10-28 04:37:55] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00242 [I 2022-10-28 04:37:55] ...trap/engines/engine.py.230: loss: 0.10959 [I 2022-10-28 04:37:55] ...optimizers/trijoint.py.151: total_norm: 14.349370 [I 2022-10-28 04:38:00] ...trap/engines/engine.py.224: train: epoch 106 | batch 1190/2304 [I 2022-10-28 04:38:00] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:20 [I 2022-10-28 04:38:00] ...trap/engines/engine.py.229: process: 0.54355 | load: 0.00278 [I 2022-10-28 04:38:00] ...trap/engines/engine.py.230: loss: 0.09317 [I 2022-10-28 04:38:00] ...optimizers/trijoint.py.151: total_norm: 10.763152 [I 2022-10-28 04:38:06] ...trap/engines/engine.py.224: train: epoch 106 | batch 1200/2304 [I 2022-10-28 04:38:06] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:12 [I 2022-10-28 04:38:06] ...trap/engines/engine.py.229: process: 0.55009 | load: 0.00265 [I 2022-10-28 04:38:06] ...trap/engines/engine.py.230: loss: 0.09118 [I 2022-10-28 04:38:06] ...optimizers/trijoint.py.151: total_norm: 9.656071 [I 2022-10-28 04:38:11] ...trap/engines/engine.py.224: train: epoch 106 | batch 1210/2304 [I 2022-10-28 04:38:11] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:10 [I 2022-10-28 04:38:11] ...trap/engines/engine.py.229: process: 0.56830 | load: 0.00262 [I 2022-10-28 04:38:11] ...trap/engines/engine.py.230: loss: 0.09471 [I 2022-10-28 04:38:11] ...optimizers/trijoint.py.151: total_norm: 10.512917 [I 2022-10-28 04:38:17] ...trap/engines/engine.py.224: train: epoch 106 | batch 1220/2304 [I 2022-10-28 04:38:17] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:29 [I 2022-10-28 04:38:17] ...trap/engines/engine.py.229: process: 0.57252 | load: 0.00270 [I 2022-10-28 04:38:17] ...trap/engines/engine.py.230: loss: 0.11788 [I 2022-10-28 04:38:17] ...optimizers/trijoint.py.151: total_norm: 21.449951 [I 2022-10-28 04:38:23] ...trap/engines/engine.py.224: train: epoch 106 | batch 1230/2304 [I 2022-10-28 04:38:23] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:10:03 [I 2022-10-28 04:38:23] ...trap/engines/engine.py.229: process: 0.56563 | load: 0.00240 [I 2022-10-28 04:38:23] ...trap/engines/engine.py.230: loss: 0.08911 [I 2022-10-28 04:38:23] ...optimizers/trijoint.py.151: total_norm: 11.144981 [I 2022-10-28 04:38:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 1240/2304 [I 2022-10-28 04:38:28] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:57 [I 2022-10-28 04:38:28] ...trap/engines/engine.py.229: process: 0.56613 | load: 0.00241 [I 2022-10-28 04:38:28] ...trap/engines/engine.py.230: loss: 0.20428 [I 2022-10-28 04:38:28] ...optimizers/trijoint.py.151: total_norm: 23.491173 [I 2022-10-28 04:38:34] ...trap/engines/engine.py.224: train: epoch 106 | batch 1250/2304 [I 2022-10-28 04:38:34] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:50 [I 2022-10-28 04:38:34] ...trap/engines/engine.py.229: process: 0.56489 | load: 0.00246 [I 2022-10-28 04:38:34] ...trap/engines/engine.py.230: loss: 0.15687 [I 2022-10-28 04:38:34] ...optimizers/trijoint.py.151: total_norm: 15.402910 [I 2022-10-28 04:38:40] ...trap/engines/engine.py.224: train: epoch 106 | batch 1260/2304 [I 2022-10-28 04:38:40] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:44 [I 2022-10-28 04:38:40] ...trap/engines/engine.py.229: process: 0.54072 | load: 0.00244 [I 2022-10-28 04:38:40] ...trap/engines/engine.py.230: loss: 0.12618 [I 2022-10-28 04:38:40] ...optimizers/trijoint.py.151: total_norm: 16.824308 [I 2022-10-28 04:38:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 1270/2304 [I 2022-10-28 04:38:45] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:35 [I 2022-10-28 04:38:45] ...trap/engines/engine.py.229: process: 0.55335 | load: 0.00255 [I 2022-10-28 04:38:45] ...trap/engines/engine.py.230: loss: 0.09298 [I 2022-10-28 04:38:45] ...optimizers/trijoint.py.151: total_norm: 12.650232 [I 2022-10-28 04:38:51] ...trap/engines/engine.py.224: train: epoch 106 | batch 1280/2304 [I 2022-10-28 04:38:51] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:32 [I 2022-10-28 04:38:51] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00249 [I 2022-10-28 04:38:51] ...trap/engines/engine.py.230: loss: 0.09979 [I 2022-10-28 04:38:51] ...optimizers/trijoint.py.151: total_norm: 11.214835 [I 2022-10-28 04:38:56] ...trap/engines/engine.py.224: train: epoch 106 | batch 1290/2304 [I 2022-10-28 04:38:56] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:28 [I 2022-10-28 04:38:56] ...trap/engines/engine.py.229: process: 0.56575 | load: 0.00244 [I 2022-10-28 04:38:56] ...trap/engines/engine.py.230: loss: 0.12535 [I 2022-10-28 04:38:56] ...optimizers/trijoint.py.151: total_norm: 23.364532 [I 2022-10-28 04:39:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 1300/2304 [I 2022-10-28 04:39:02] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:22 [I 2022-10-28 04:39:02] ...trap/engines/engine.py.229: process: 0.55625 | load: 0.00241 [I 2022-10-28 04:39:02] ...trap/engines/engine.py.230: loss: 0.14820 [I 2022-10-28 04:39:02] ...optimizers/trijoint.py.151: total_norm: 42.646904 [I 2022-10-28 04:39:08] ...trap/engines/engine.py.224: train: epoch 106 | batch 1310/2304 [I 2022-10-28 04:39:08] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:12 [I 2022-10-28 04:39:08] ...trap/engines/engine.py.229: process: 0.55028 | load: 0.00240 [I 2022-10-28 04:39:08] ...trap/engines/engine.py.230: loss: 0.12692 [I 2022-10-28 04:39:08] ...optimizers/trijoint.py.151: total_norm: 17.052984 [I 2022-10-28 04:39:13] ...trap/engines/engine.py.224: train: epoch 106 | batch 1320/2304 [I 2022-10-28 04:39:13] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:07 [I 2022-10-28 04:39:13] ...trap/engines/engine.py.229: process: 0.56495 | load: 0.00243 [I 2022-10-28 04:39:13] ...trap/engines/engine.py.230: loss: 0.10897 [I 2022-10-28 04:39:13] ...optimizers/trijoint.py.151: total_norm: 14.309824 [I 2022-10-28 04:39:19] ...trap/engines/engine.py.224: train: epoch 106 | batch 1330/2304 [I 2022-10-28 04:39:19] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:04 [I 2022-10-28 04:39:19] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00241 [I 2022-10-28 04:39:19] ...trap/engines/engine.py.230: loss: 0.09785 [I 2022-10-28 04:39:19] ...optimizers/trijoint.py.151: total_norm: 13.461132 [I 2022-10-28 04:39:25] ...trap/engines/engine.py.224: train: epoch 106 | batch 1340/2304 [I 2022-10-28 04:39:25] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:09:20 [I 2022-10-28 04:39:25] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00243 [I 2022-10-28 04:39:25] ...trap/engines/engine.py.230: loss: 0.12424 [I 2022-10-28 04:39:25] ...optimizers/trijoint.py.151: total_norm: 14.386559 [I 2022-10-28 04:39:30] ...trap/engines/engine.py.224: train: epoch 106 | batch 1350/2304 [I 2022-10-28 04:39:30] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:08:53 [I 2022-10-28 04:39:30] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00241 [I 2022-10-28 04:39:30] ...trap/engines/engine.py.230: loss: 0.09636 [I 2022-10-28 04:39:30] ...optimizers/trijoint.py.151: total_norm: 9.945964 [I 2022-10-28 04:39:36] ...trap/engines/engine.py.224: train: epoch 106 | batch 1360/2304 [I 2022-10-28 04:39:36] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:47 [I 2022-10-28 04:39:36] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00245 [I 2022-10-28 04:39:36] ...trap/engines/engine.py.230: loss: 0.09437 [I 2022-10-28 04:39:36] ...optimizers/trijoint.py.151: total_norm: 21.201662 [I 2022-10-28 04:39:41] ...trap/engines/engine.py.224: train: epoch 106 | batch 1370/2304 [I 2022-10-28 04:39:41] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:42 [I 2022-10-28 04:39:41] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00239 [I 2022-10-28 04:39:41] ...trap/engines/engine.py.230: loss: 0.09748 [I 2022-10-28 04:39:41] ...optimizers/trijoint.py.151: total_norm: 21.785410 [I 2022-10-28 04:39:47] ...trap/engines/engine.py.224: train: epoch 106 | batch 1380/2304 [I 2022-10-28 04:39:47] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:38 [I 2022-10-28 04:39:47] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00242 [I 2022-10-28 04:39:47] ...trap/engines/engine.py.230: loss: 0.18949 [I 2022-10-28 04:39:47] ...optimizers/trijoint.py.151: total_norm: 15.285191 [I 2022-10-28 04:39:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 1390/2304 [I 2022-10-28 04:39:53] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:32 [I 2022-10-28 04:39:53] ...trap/engines/engine.py.229: process: 0.56067 | load: 0.00242 [I 2022-10-28 04:39:53] ...trap/engines/engine.py.230: loss: 0.08723 [I 2022-10-28 04:39:53] ...optimizers/trijoint.py.151: total_norm: 18.081509 [I 2022-10-28 04:39:58] ...trap/engines/engine.py.224: train: epoch 106 | batch 1400/2304 [I 2022-10-28 04:39:58] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:08:24 [I 2022-10-28 04:39:58] ...trap/engines/engine.py.229: process: 0.54957 | load: 0.00252 [I 2022-10-28 04:39:58] ...trap/engines/engine.py.230: loss: 0.12242 [I 2022-10-28 04:39:58] ...optimizers/trijoint.py.151: total_norm: 11.945655 [I 2022-10-28 04:40:04] ...trap/engines/engine.py.224: train: epoch 106 | batch 1410/2304 [I 2022-10-28 04:40:04] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:19 [I 2022-10-28 04:40:04] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00248 [I 2022-10-28 04:40:04] ...trap/engines/engine.py.230: loss: 0.16298 [I 2022-10-28 04:40:04] ...optimizers/trijoint.py.151: total_norm: 19.346542 [I 2022-10-28 04:40:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 1420/2304 [I 2022-10-28 04:40:09] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:13 [I 2022-10-28 04:40:09] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00246 [I 2022-10-28 04:40:09] ...trap/engines/engine.py.230: loss: 0.13585 [I 2022-10-28 04:40:09] ...optimizers/trijoint.py.151: total_norm: 27.791376 [I 2022-10-28 04:40:15] ...trap/engines/engine.py.224: train: epoch 106 | batch 1430/2304 [I 2022-10-28 04:40:15] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:09 [I 2022-10-28 04:40:15] ...trap/engines/engine.py.229: process: 0.56394 | load: 0.00243 [I 2022-10-28 04:40:15] ...trap/engines/engine.py.230: loss: 0.11659 [I 2022-10-28 04:40:15] ...optimizers/trijoint.py.151: total_norm: 15.761477 [I 2022-10-28 04:40:21] ...trap/engines/engine.py.224: train: epoch 106 | batch 1440/2304 [I 2022-10-28 04:40:21] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:08:04 [I 2022-10-28 04:40:21] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00246 [I 2022-10-28 04:40:21] ...trap/engines/engine.py.230: loss: 0.10487 [I 2022-10-28 04:40:21] ...optimizers/trijoint.py.151: total_norm: 10.629757 [I 2022-10-28 04:40:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 1450/2304 [I 2022-10-28 04:40:26] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:07:50 [I 2022-10-28 04:40:26] ...trap/engines/engine.py.229: process: 0.55366 | load: 0.00243 [I 2022-10-28 04:40:26] ...trap/engines/engine.py.230: loss: 0.11787 [I 2022-10-28 04:40:26] ...optimizers/trijoint.py.151: total_norm: 14.422894 [I 2022-10-28 04:40:32] ...trap/engines/engine.py.224: train: epoch 106 | batch 1460/2304 [I 2022-10-28 04:40:32] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:50 [I 2022-10-28 04:40:32] ...trap/engines/engine.py.229: process: 0.56164 | load: 0.00240 [I 2022-10-28 04:40:32] ...trap/engines/engine.py.230: loss: 0.12389 [I 2022-10-28 04:40:32] ...optimizers/trijoint.py.151: total_norm: 12.957072 [I 2022-10-28 04:40:37] ...trap/engines/engine.py.224: train: epoch 106 | batch 1470/2304 [I 2022-10-28 04:40:37] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:55 [I 2022-10-28 04:40:37] ...trap/engines/engine.py.229: process: 0.55871 | load: 0.00239 [I 2022-10-28 04:40:37] ...trap/engines/engine.py.230: loss: 0.10723 [I 2022-10-28 04:40:37] ...optimizers/trijoint.py.151: total_norm: 16.793592 [I 2022-10-28 04:40:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 1480/2304 [I 2022-10-28 04:40:43] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:39 [I 2022-10-28 04:40:43] ...trap/engines/engine.py.229: process: 0.54556 | load: 0.00248 [I 2022-10-28 04:40:43] ...trap/engines/engine.py.230: loss: 0.18472 [I 2022-10-28 04:40:43] ...optimizers/trijoint.py.151: total_norm: 18.336235 [I 2022-10-28 04:40:49] ...trap/engines/engine.py.224: train: epoch 106 | batch 1490/2304 [I 2022-10-28 04:40:49] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:33 [I 2022-10-28 04:40:49] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00244 [I 2022-10-28 04:40:49] ...trap/engines/engine.py.230: loss: 0.11332 [I 2022-10-28 04:40:49] ...optimizers/trijoint.py.151: total_norm: 24.494696 [I 2022-10-28 04:40:54] ...trap/engines/engine.py.224: train: epoch 106 | batch 1500/2304 [I 2022-10-28 04:40:54] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:07:27 [I 2022-10-28 04:40:54] ...trap/engines/engine.py.229: process: 0.55022 | load: 0.00246 [I 2022-10-28 04:40:54] ...trap/engines/engine.py.230: loss: 0.09783 [I 2022-10-28 04:40:54] ...optimizers/trijoint.py.151: total_norm: 12.651428 [I 2022-10-28 04:41:00] ...trap/engines/engine.py.224: train: epoch 106 | batch 1510/2304 [I 2022-10-28 04:41:00] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:22 [I 2022-10-28 04:41:00] ...trap/engines/engine.py.229: process: 0.55758 | load: 0.00244 [I 2022-10-28 04:41:00] ...trap/engines/engine.py.230: loss: 0.13853 [I 2022-10-28 04:41:00] ...optimizers/trijoint.py.151: total_norm: 23.120123 [I 2022-10-28 04:41:05] ...trap/engines/engine.py.224: train: epoch 106 | batch 1520/2304 [I 2022-10-28 04:41:05] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:07:17 [I 2022-10-28 04:41:05] ...trap/engines/engine.py.229: process: 0.56865 | load: 0.00240 [I 2022-10-28 04:41:05] ...trap/engines/engine.py.230: loss: 0.10015 [I 2022-10-28 04:41:05] ...optimizers/trijoint.py.151: total_norm: 11.118722 [I 2022-10-28 04:41:11] ...trap/engines/engine.py.224: train: epoch 106 | batch 1530/2304 [I 2022-10-28 04:41:11] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:11 [I 2022-10-28 04:41:11] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00241 [I 2022-10-28 04:41:11] ...trap/engines/engine.py.230: loss: 0.12363 [I 2022-10-28 04:41:11] ...optimizers/trijoint.py.151: total_norm: 15.174381 [I 2022-10-28 04:41:17] ...trap/engines/engine.py.224: train: epoch 106 | batch 1540/2304 [I 2022-10-28 04:41:17] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:08 [I 2022-10-28 04:41:17] ...trap/engines/engine.py.229: process: 0.56137 | load: 0.00242 [I 2022-10-28 04:41:17] ...trap/engines/engine.py.230: loss: 0.10064 [I 2022-10-28 04:41:17] ...optimizers/trijoint.py.151: total_norm: 49.485996 [I 2022-10-28 04:41:22] ...trap/engines/engine.py.224: train: epoch 106 | batch 1550/2304 [I 2022-10-28 04:41:22] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:06:59 [I 2022-10-28 04:41:22] ...trap/engines/engine.py.229: process: 0.54742 | load: 0.00241 [I 2022-10-28 04:41:22] ...trap/engines/engine.py.230: loss: 0.11406 [I 2022-10-28 04:41:22] ...optimizers/trijoint.py.151: total_norm: 11.402681 [I 2022-10-28 04:41:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 1560/2304 [I 2022-10-28 04:41:28] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:55 [I 2022-10-28 04:41:28] ...trap/engines/engine.py.229: process: 0.55561 | load: 0.00240 [I 2022-10-28 04:41:28] ...trap/engines/engine.py.230: loss: 0.10421 [I 2022-10-28 04:41:28] ...optimizers/trijoint.py.151: total_norm: 15.639514 [I 2022-10-28 04:41:33] ...trap/engines/engine.py.224: train: epoch 106 | batch 1570/2304 [I 2022-10-28 04:41:33] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:06:48 [I 2022-10-28 04:41:33] ...trap/engines/engine.py.229: process: 0.56346 | load: 0.00242 [I 2022-10-28 04:41:33] ...trap/engines/engine.py.230: loss: 0.07889 [I 2022-10-28 04:41:33] ...optimizers/trijoint.py.151: total_norm: 10.251747 [I 2022-10-28 04:41:39] ...trap/engines/engine.py.224: train: epoch 106 | batch 1580/2304 [I 2022-10-28 04:41:39] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:43 [I 2022-10-28 04:41:39] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00237 [I 2022-10-28 04:41:39] ...trap/engines/engine.py.230: loss: 0.14311 [I 2022-10-28 04:41:39] ...optimizers/trijoint.py.151: total_norm: 15.644024 [I 2022-10-28 04:41:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 1590/2304 [I 2022-10-28 04:41:45] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:53 [I 2022-10-28 04:41:45] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00237 [I 2022-10-28 04:41:45] ...trap/engines/engine.py.230: loss: 0.21060 [I 2022-10-28 04:41:45] ...optimizers/trijoint.py.151: total_norm: 26.831421 [I 2022-10-28 04:41:50] ...trap/engines/engine.py.224: train: epoch 106 | batch 1600/2304 [I 2022-10-28 04:41:50] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:34 [I 2022-10-28 04:41:50] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00243 [I 2022-10-28 04:41:50] ...trap/engines/engine.py.230: loss: 0.09372 [I 2022-10-28 04:41:50] ...optimizers/trijoint.py.151: total_norm: 8.196466 [I 2022-10-28 04:41:56] ...trap/engines/engine.py.224: train: epoch 106 | batch 1610/2304 [I 2022-10-28 04:41:56] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:26 [I 2022-10-28 04:41:56] ...trap/engines/engine.py.229: process: 0.54844 | load: 0.00241 [I 2022-10-28 04:41:56] ...trap/engines/engine.py.230: loss: 0.12031 [I 2022-10-28 04:41:56] ...optimizers/trijoint.py.151: total_norm: 10.090824 [I 2022-10-28 04:42:01] ...trap/engines/engine.py.224: train: epoch 106 | batch 1620/2304 [I 2022-10-28 04:42:01] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:21 [I 2022-10-28 04:42:01] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00251 [I 2022-10-28 04:42:01] ...trap/engines/engine.py.230: loss: 0.18516 [I 2022-10-28 04:42:01] ...optimizers/trijoint.py.151: total_norm: 15.934447 [I 2022-10-28 04:42:07] ...trap/engines/engine.py.224: train: epoch 106 | batch 1630/2304 [I 2022-10-28 04:42:07] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:15 [I 2022-10-28 04:42:07] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00247 [I 2022-10-28 04:42:07] ...trap/engines/engine.py.230: loss: 0.13704 [I 2022-10-28 04:42:07] ...optimizers/trijoint.py.151: total_norm: 19.897243 [I 2022-10-28 04:42:13] ...trap/engines/engine.py.224: train: epoch 106 | batch 1640/2304 [I 2022-10-28 04:42:13] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:11 [I 2022-10-28 04:42:13] ...trap/engines/engine.py.229: process: 0.55883 | load: 0.00248 [I 2022-10-28 04:42:13] ...trap/engines/engine.py.230: loss: 0.15767 [I 2022-10-28 04:42:13] ...optimizers/trijoint.py.151: total_norm: 23.024956 [I 2022-10-28 04:42:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 1650/2304 [I 2022-10-28 04:42:18] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:05 [I 2022-10-28 04:42:18] ...trap/engines/engine.py.229: process: 0.56777 | load: 0.00243 [I 2022-10-28 04:42:18] ...trap/engines/engine.py.230: loss: 0.12373 [I 2022-10-28 04:42:18] ...optimizers/trijoint.py.151: total_norm: 16.471472 [I 2022-10-28 04:42:24] ...trap/engines/engine.py.224: train: epoch 106 | batch 1660/2304 [I 2022-10-28 04:42:24] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:05:59 [I 2022-10-28 04:42:24] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00244 [I 2022-10-28 04:42:24] ...trap/engines/engine.py.230: loss: 0.13005 [I 2022-10-28 04:42:24] ...optimizers/trijoint.py.151: total_norm: 35.613491 [I 2022-10-28 04:42:29] ...trap/engines/engine.py.224: train: epoch 106 | batch 1670/2304 [I 2022-10-28 04:42:29] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:54 [I 2022-10-28 04:42:29] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00245 [I 2022-10-28 04:42:29] ...trap/engines/engine.py.230: loss: 0.08088 [I 2022-10-28 04:42:29] ...optimizers/trijoint.py.151: total_norm: 21.701784 [I 2022-10-28 04:42:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 1680/2304 [I 2022-10-28 04:42:35] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:05:49 [I 2022-10-28 04:42:35] ...trap/engines/engine.py.229: process: 0.56182 | load: 0.00243 [I 2022-10-28 04:42:35] ...trap/engines/engine.py.230: loss: 0.09426 [I 2022-10-28 04:42:35] ...optimizers/trijoint.py.151: total_norm: 37.352432 [I 2022-10-28 04:42:41] ...trap/engines/engine.py.224: train: epoch 106 | batch 1690/2304 [I 2022-10-28 04:42:41] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:43 [I 2022-10-28 04:42:41] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00246 [I 2022-10-28 04:42:41] ...trap/engines/engine.py.230: loss: 0.17440 [I 2022-10-28 04:42:41] ...optimizers/trijoint.py.151: total_norm: 25.140518 [I 2022-10-28 04:42:46] ...trap/engines/engine.py.224: train: epoch 106 | batch 1700/2304 [I 2022-10-28 04:42:46] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:39 [I 2022-10-28 04:42:46] ...trap/engines/engine.py.229: process: 0.55135 | load: 0.00253 [I 2022-10-28 04:42:46] ...trap/engines/engine.py.230: loss: 0.18968 [I 2022-10-28 04:42:46] ...optimizers/trijoint.py.151: total_norm: 34.030472 [I 2022-10-28 04:42:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 1710/2304 [I 2022-10-28 04:42:52] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:31 [I 2022-10-28 04:42:52] ...trap/engines/engine.py.229: process: 0.56376 | load: 0.00264 [I 2022-10-28 04:42:52] ...trap/engines/engine.py.230: loss: 0.14432 [I 2022-10-28 04:42:52] ...optimizers/trijoint.py.151: total_norm: 20.899660 [I 2022-10-28 04:42:57] ...trap/engines/engine.py.224: train: epoch 106 | batch 1720/2304 [I 2022-10-28 04:42:57] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:25 [I 2022-10-28 04:42:57] ...trap/engines/engine.py.229: process: 0.56086 | load: 0.00244 [I 2022-10-28 04:42:57] ...trap/engines/engine.py.230: loss: 0.15420 [I 2022-10-28 04:42:57] ...optimizers/trijoint.py.151: total_norm: 17.835100 [I 2022-10-28 04:43:03] ...trap/engines/engine.py.224: train: epoch 106 | batch 1730/2304 [I 2022-10-28 04:43:03] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:22 [I 2022-10-28 04:43:03] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00243 [I 2022-10-28 04:43:03] ...trap/engines/engine.py.230: loss: 0.11118 [I 2022-10-28 04:43:03] ...optimizers/trijoint.py.151: total_norm: 20.237463 [I 2022-10-28 04:43:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 1740/2304 [I 2022-10-28 04:43:09] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:15 [I 2022-10-28 04:43:09] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00241 [I 2022-10-28 04:43:09] ...trap/engines/engine.py.230: loss: 0.13284 [I 2022-10-28 04:43:09] ...optimizers/trijoint.py.151: total_norm: 12.848183 [I 2022-10-28 04:43:14] ...trap/engines/engine.py.224: train: epoch 106 | batch 1750/2304 [I 2022-10-28 04:43:14] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:08 [I 2022-10-28 04:43:14] ...trap/engines/engine.py.229: process: 0.56810 | load: 0.00242 [I 2022-10-28 04:43:14] ...trap/engines/engine.py.230: loss: 0.16971 [I 2022-10-28 04:43:14] ...optimizers/trijoint.py.151: total_norm: 16.168678 [I 2022-10-28 04:43:20] ...trap/engines/engine.py.224: train: epoch 106 | batch 1760/2304 [I 2022-10-28 04:43:20] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:03 [I 2022-10-28 04:43:20] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00241 [I 2022-10-28 04:43:20] ...trap/engines/engine.py.230: loss: 0.10240 [I 2022-10-28 04:43:20] ...optimizers/trijoint.py.151: total_norm: 11.163861 [I 2022-10-28 04:43:25] ...trap/engines/engine.py.224: train: epoch 106 | batch 1770/2304 [I 2022-10-28 04:43:25] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:04:56 [I 2022-10-28 04:43:25] ...trap/engines/engine.py.229: process: 0.54792 | load: 0.00243 [I 2022-10-28 04:43:25] ...trap/engines/engine.py.230: loss: 0.12991 [I 2022-10-28 04:43:25] ...optimizers/trijoint.py.151: total_norm: 12.698902 [I 2022-10-28 04:43:31] ...trap/engines/engine.py.224: train: epoch 106 | batch 1780/2304 [I 2022-10-28 04:43:31] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:04:52 [I 2022-10-28 04:43:31] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00243 [I 2022-10-28 04:43:31] ...trap/engines/engine.py.230: loss: 0.10864 [I 2022-10-28 04:43:31] ...optimizers/trijoint.py.151: total_norm: 12.116102 [I 2022-10-28 04:43:37] ...trap/engines/engine.py.224: train: epoch 106 | batch 1790/2304 [I 2022-10-28 04:43:37] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:47 [I 2022-10-28 04:43:37] ...trap/engines/engine.py.229: process: 0.56068 | load: 0.00244 [I 2022-10-28 04:43:37] ...trap/engines/engine.py.230: loss: 0.11130 [I 2022-10-28 04:43:37] ...optimizers/trijoint.py.151: total_norm: 15.903199 [I 2022-10-28 04:43:42] ...trap/engines/engine.py.224: train: epoch 106 | batch 1800/2304 [I 2022-10-28 04:43:42] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:42 [I 2022-10-28 04:43:42] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00241 [I 2022-10-28 04:43:42] ...trap/engines/engine.py.230: loss: 0.11720 [I 2022-10-28 04:43:42] ...optimizers/trijoint.py.151: total_norm: 21.069359 [I 2022-10-28 04:43:48] ...trap/engines/engine.py.224: train: epoch 106 | batch 1810/2304 [I 2022-10-28 04:43:48] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:36 [I 2022-10-28 04:43:48] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00265 [I 2022-10-28 04:43:48] ...trap/engines/engine.py.230: loss: 0.13999 [I 2022-10-28 04:43:48] ...optimizers/trijoint.py.151: total_norm: 9.820993 [I 2022-10-28 04:43:53] ...trap/engines/engine.py.224: train: epoch 106 | batch 1820/2304 [I 2022-10-28 04:43:53] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:30 [I 2022-10-28 04:43:53] ...trap/engines/engine.py.229: process: 0.54912 | load: 0.00247 [I 2022-10-28 04:43:53] ...trap/engines/engine.py.230: loss: 0.14393 [I 2022-10-28 04:43:53] ...optimizers/trijoint.py.151: total_norm: 18.600500 [I 2022-10-28 04:43:59] ...trap/engines/engine.py.224: train: epoch 106 | batch 1830/2304 [I 2022-10-28 04:43:59] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:24 [I 2022-10-28 04:43:59] ...trap/engines/engine.py.229: process: 0.55503 | load: 0.00248 [I 2022-10-28 04:43:59] ...trap/engines/engine.py.230: loss: 0.14528 [I 2022-10-28 04:43:59] ...optimizers/trijoint.py.151: total_norm: 23.503183 [I 2022-10-28 04:44:05] ...trap/engines/engine.py.224: train: epoch 106 | batch 1840/2304 [I 2022-10-28 04:44:05] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:16 [I 2022-10-28 04:44:05] ...trap/engines/engine.py.229: process: 0.54820 | load: 0.00245 [I 2022-10-28 04:44:05] ...trap/engines/engine.py.230: loss: 0.07822 [I 2022-10-28 04:44:05] ...optimizers/trijoint.py.151: total_norm: 9.865940 [I 2022-10-28 04:44:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 1850/2304 [I 2022-10-28 04:44:10] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:13 [I 2022-10-28 04:44:10] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00241 [I 2022-10-28 04:44:10] ...trap/engines/engine.py.230: loss: 0.11382 [I 2022-10-28 04:44:10] ...optimizers/trijoint.py.151: total_norm: 17.307772 [I 2022-10-28 04:44:16] ...trap/engines/engine.py.224: train: epoch 106 | batch 1860/2304 [I 2022-10-28 04:44:16] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:06 [I 2022-10-28 04:44:16] ...trap/engines/engine.py.229: process: 0.55036 | load: 0.00241 [I 2022-10-28 04:44:16] ...trap/engines/engine.py.230: loss: 0.09605 [I 2022-10-28 04:44:16] ...optimizers/trijoint.py.151: total_norm: 9.655499 [I 2022-10-28 04:44:21] ...trap/engines/engine.py.224: train: epoch 106 | batch 1870/2304 [I 2022-10-28 04:44:21] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:01 [I 2022-10-28 04:44:21] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00241 [I 2022-10-28 04:44:21] ...trap/engines/engine.py.230: loss: 0.10787 [I 2022-10-28 04:44:21] ...optimizers/trijoint.py.151: total_norm: 24.146622 [I 2022-10-28 04:44:27] ...trap/engines/engine.py.224: train: epoch 106 | batch 1880/2304 [I 2022-10-28 04:44:27] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:03:56 [I 2022-10-28 04:44:27] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00249 [I 2022-10-28 04:44:27] ...trap/engines/engine.py.230: loss: 0.14786 [I 2022-10-28 04:44:27] ...optimizers/trijoint.py.151: total_norm: 9.720055 [I 2022-10-28 04:44:32] ...trap/engines/engine.py.224: train: epoch 106 | batch 1890/2304 [I 2022-10-28 04:44:32] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:51 [I 2022-10-28 04:44:32] ...trap/engines/engine.py.229: process: 0.56216 | load: 0.00247 [I 2022-10-28 04:44:32] ...trap/engines/engine.py.230: loss: 0.10424 [I 2022-10-28 04:44:32] ...optimizers/trijoint.py.151: total_norm: 17.780056 [I 2022-10-28 04:44:38] ...trap/engines/engine.py.224: train: epoch 106 | batch 1900/2304 [I 2022-10-28 04:44:38] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:43 [I 2022-10-28 04:44:38] ...trap/engines/engine.py.229: process: 0.54088 | load: 0.00247 [I 2022-10-28 04:44:38] ...trap/engines/engine.py.230: loss: 0.07766 [I 2022-10-28 04:44:38] ...optimizers/trijoint.py.151: total_norm: 32.450413 [I 2022-10-28 04:44:44] ...trap/engines/engine.py.224: train: epoch 106 | batch 1910/2304 [I 2022-10-28 04:44:44] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:37 [I 2022-10-28 04:44:44] ...trap/engines/engine.py.229: process: 0.54896 | load: 0.00246 [I 2022-10-28 04:44:44] ...trap/engines/engine.py.230: loss: 0.10744 [I 2022-10-28 04:44:44] ...optimizers/trijoint.py.151: total_norm: 13.221517 [I 2022-10-28 04:44:49] ...trap/engines/engine.py.224: train: epoch 106 | batch 1920/2304 [I 2022-10-28 04:44:49] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:33 [I 2022-10-28 04:44:49] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00258 [I 2022-10-28 04:44:49] ...trap/engines/engine.py.230: loss: 0.08330 [I 2022-10-28 04:44:49] ...optimizers/trijoint.py.151: total_norm: 20.830404 [I 2022-10-28 04:44:55] ...trap/engines/engine.py.224: train: epoch 106 | batch 1930/2304 [I 2022-10-28 04:44:55] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:28 [I 2022-10-28 04:44:55] ...trap/engines/engine.py.229: process: 0.57608 | load: 0.00240 [I 2022-10-28 04:44:55] ...trap/engines/engine.py.230: loss: 0.10547 [I 2022-10-28 04:44:55] ...optimizers/trijoint.py.151: total_norm: 11.077703 [I 2022-10-28 04:45:00] ...trap/engines/engine.py.224: train: epoch 106 | batch 1940/2304 [I 2022-10-28 04:45:00] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:23 [I 2022-10-28 04:45:00] ...trap/engines/engine.py.229: process: 0.55685 | load: 0.00243 [I 2022-10-28 04:45:00] ...trap/engines/engine.py.230: loss: 0.11319 [I 2022-10-28 04:45:00] ...optimizers/trijoint.py.151: total_norm: 13.232769 [I 2022-10-28 04:45:06] ...trap/engines/engine.py.224: train: epoch 106 | batch 1950/2304 [I 2022-10-28 04:45:06] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:16 [I 2022-10-28 04:45:06] ...trap/engines/engine.py.229: process: 0.54429 | load: 0.00240 [I 2022-10-28 04:45:06] ...trap/engines/engine.py.230: loss: 0.11930 [I 2022-10-28 04:45:06] ...optimizers/trijoint.py.151: total_norm: 32.997173 [I 2022-10-28 04:45:11] ...trap/engines/engine.py.224: train: epoch 106 | batch 1960/2304 [I 2022-10-28 04:45:11] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:10 [I 2022-10-28 04:45:11] ...trap/engines/engine.py.229: process: 0.55113 | load: 0.00242 [I 2022-10-28 04:45:11] ...trap/engines/engine.py.230: loss: 0.23420 [I 2022-10-28 04:45:11] ...optimizers/trijoint.py.151: total_norm: 21.853989 [I 2022-10-28 04:45:17] ...trap/engines/engine.py.224: train: epoch 106 | batch 1970/2304 [I 2022-10-28 04:45:17] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:06 [I 2022-10-28 04:45:17] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00239 [I 2022-10-28 04:45:17] ...trap/engines/engine.py.230: loss: 0.09327 [I 2022-10-28 04:45:17] ...optimizers/trijoint.py.151: total_norm: 16.613628 [I 2022-10-28 04:45:23] ...trap/engines/engine.py.224: train: epoch 106 | batch 1980/2304 [I 2022-10-28 04:45:23] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:00 [I 2022-10-28 04:45:23] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00244 [I 2022-10-28 04:45:23] ...trap/engines/engine.py.230: loss: 0.11016 [I 2022-10-28 04:45:23] ...optimizers/trijoint.py.151: total_norm: 32.572731 [I 2022-10-28 04:45:28] ...trap/engines/engine.py.224: train: epoch 106 | batch 1990/2304 [I 2022-10-28 04:45:28] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:02:54 [I 2022-10-28 04:45:28] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00239 [I 2022-10-28 04:45:28] ...trap/engines/engine.py.230: loss: 0.16732 [I 2022-10-28 04:45:28] ...optimizers/trijoint.py.151: total_norm: 12.906165 [I 2022-10-28 04:45:34] ...trap/engines/engine.py.224: train: epoch 106 | batch 2000/2304 [I 2022-10-28 04:45:34] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:49 [I 2022-10-28 04:45:34] ...trap/engines/engine.py.229: process: 0.55724 | load: 0.00241 [I 2022-10-28 04:45:34] ...trap/engines/engine.py.230: loss: 0.10676 [I 2022-10-28 04:45:34] ...optimizers/trijoint.py.151: total_norm: 16.569424 [I 2022-10-28 04:45:39] ...trap/engines/engine.py.224: train: epoch 106 | batch 2010/2304 [I 2022-10-28 04:45:39] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:43 [I 2022-10-28 04:45:39] ...trap/engines/engine.py.229: process: 0.54917 | load: 0.00238 [I 2022-10-28 04:45:39] ...trap/engines/engine.py.230: loss: 0.08350 [I 2022-10-28 04:45:39] ...optimizers/trijoint.py.151: total_norm: 15.150952 [I 2022-10-28 04:45:45] ...trap/engines/engine.py.224: train: epoch 106 | batch 2020/2304 [I 2022-10-28 04:45:45] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:38 [I 2022-10-28 04:45:45] ...trap/engines/engine.py.229: process: 0.56522 | load: 0.00241 [I 2022-10-28 04:45:45] ...trap/engines/engine.py.230: loss: 0.18554 [I 2022-10-28 04:45:45] ...optimizers/trijoint.py.151: total_norm: 27.426256 [I 2022-10-28 04:45:50] ...trap/engines/engine.py.224: train: epoch 106 | batch 2030/2304 [I 2022-10-28 04:45:50] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:30 [I 2022-10-28 04:45:50] ...trap/engines/engine.py.229: process: 0.54367 | load: 0.00241 [I 2022-10-28 04:45:50] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-28 04:45:50] ...optimizers/trijoint.py.151: total_norm: 16.510519 [I 2022-10-28 04:45:56] ...trap/engines/engine.py.224: train: epoch 106 | batch 2040/2304 [I 2022-10-28 04:45:56] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:26 [I 2022-10-28 04:45:56] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00239 [I 2022-10-28 04:45:56] ...trap/engines/engine.py.230: loss: 0.11011 [I 2022-10-28 04:45:56] ...optimizers/trijoint.py.151: total_norm: 16.134069 [I 2022-10-28 04:46:02] ...trap/engines/engine.py.224: train: epoch 106 | batch 2050/2304 [I 2022-10-28 04:46:02] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:21 [I 2022-10-28 04:46:02] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00241 [I 2022-10-28 04:46:02] ...trap/engines/engine.py.230: loss: 0.08309 [I 2022-10-28 04:46:02] ...optimizers/trijoint.py.151: total_norm: 14.726109 [I 2022-10-28 04:46:07] ...trap/engines/engine.py.224: train: epoch 106 | batch 2060/2304 [I 2022-10-28 04:46:07] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:16 [I 2022-10-28 04:46:07] ...trap/engines/engine.py.229: process: 0.55260 | load: 0.00250 [I 2022-10-28 04:46:07] ...trap/engines/engine.py.230: loss: 0.14321 [I 2022-10-28 04:46:07] ...optimizers/trijoint.py.151: total_norm: 20.256195 [I 2022-10-28 04:46:13] ...trap/engines/engine.py.224: train: epoch 106 | batch 2070/2304 [I 2022-10-28 04:46:13] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:11 [I 2022-10-28 04:46:13] ...trap/engines/engine.py.229: process: 0.56807 | load: 0.00243 [I 2022-10-28 04:46:13] ...trap/engines/engine.py.230: loss: 0.14385 [I 2022-10-28 04:46:13] ...optimizers/trijoint.py.151: total_norm: 36.130020 [I 2022-10-28 04:46:18] ...trap/engines/engine.py.224: train: epoch 106 | batch 2080/2304 [I 2022-10-28 04:46:18] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:04 [I 2022-10-28 04:46:18] ...trap/engines/engine.py.229: process: 0.55827 | load: 0.00249 [I 2022-10-28 04:46:18] ...trap/engines/engine.py.230: loss: 0.12168 [I 2022-10-28 04:46:18] ...optimizers/trijoint.py.151: total_norm: 15.156034 [I 2022-10-28 04:46:24] ...trap/engines/engine.py.224: train: epoch 106 | batch 2090/2304 [I 2022-10-28 04:46:24] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:01:58 [I 2022-10-28 04:46:24] ...trap/engines/engine.py.229: process: 0.55747 | load: 0.00240 [I 2022-10-28 04:46:24] ...trap/engines/engine.py.230: loss: 0.12624 [I 2022-10-28 04:46:24] ...optimizers/trijoint.py.151: total_norm: 17.260338 [I 2022-10-28 04:46:30] ...trap/engines/engine.py.224: train: epoch 106 | batch 2100/2304 [I 2022-10-28 04:46:30] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:01:54 [I 2022-10-28 04:46:30] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00241 [I 2022-10-28 04:46:30] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-28 04:46:30] ...optimizers/trijoint.py.151: total_norm: 13.015119 [I 2022-10-28 04:46:35] ...trap/engines/engine.py.224: train: epoch 106 | batch 2110/2304 [I 2022-10-28 04:46:35] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:47 [I 2022-10-28 04:46:35] ...trap/engines/engine.py.229: process: 0.55267 | load: 0.00242 [I 2022-10-28 04:46:35] ...trap/engines/engine.py.230: loss: 0.09120 [I 2022-10-28 04:46:35] ...optimizers/trijoint.py.151: total_norm: 24.718019 [I 2022-10-28 04:46:41] ...trap/engines/engine.py.224: train: epoch 106 | batch 2120/2304 [I 2022-10-28 04:46:41] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:42 [I 2022-10-28 04:46:41] ...trap/engines/engine.py.229: process: 0.55227 | load: 0.00245 [I 2022-10-28 04:46:41] ...trap/engines/engine.py.230: loss: 0.08647 [I 2022-10-28 04:46:41] ...optimizers/trijoint.py.151: total_norm: 11.355854 [I 2022-10-28 04:46:46] ...trap/engines/engine.py.224: train: epoch 106 | batch 2130/2304 [I 2022-10-28 04:46:46] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:36 [I 2022-10-28 04:46:46] ...trap/engines/engine.py.229: process: 0.55426 | load: 0.00244 [I 2022-10-28 04:46:46] ...trap/engines/engine.py.230: loss: 0.19768 [I 2022-10-28 04:46:46] ...optimizers/trijoint.py.151: total_norm: 31.258139 [I 2022-10-28 04:46:52] ...trap/engines/engine.py.224: train: epoch 106 | batch 2140/2304 [I 2022-10-28 04:46:52] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:33 [I 2022-10-28 04:46:52] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00243 [I 2022-10-28 04:46:52] ...trap/engines/engine.py.230: loss: 0.07969 [I 2022-10-28 04:46:52] ...optimizers/trijoint.py.151: total_norm: 31.313351 [I 2022-10-28 04:46:58] ...trap/engines/engine.py.224: train: epoch 106 | batch 2150/2304 [I 2022-10-28 04:46:58] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:26 [I 2022-10-28 04:46:58] ...trap/engines/engine.py.229: process: 0.56294 | load: 0.00240 [I 2022-10-28 04:46:58] ...trap/engines/engine.py.230: loss: 0.12055 [I 2022-10-28 04:46:58] ...optimizers/trijoint.py.151: total_norm: 13.271378 [I 2022-10-28 04:47:03] ...trap/engines/engine.py.224: train: epoch 106 | batch 2160/2304 [I 2022-10-28 04:47:03] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:21 [I 2022-10-28 04:47:03] ...trap/engines/engine.py.229: process: 0.57873 | load: 0.00288 [I 2022-10-28 04:47:03] ...trap/engines/engine.py.230: loss: 0.09367 [I 2022-10-28 04:47:03] ...optimizers/trijoint.py.151: total_norm: 8.912683 [I 2022-10-28 04:47:09] ...trap/engines/engine.py.224: train: epoch 106 | batch 2170/2304 [I 2022-10-28 04:47:09] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:15 [I 2022-10-28 04:47:09] ...trap/engines/engine.py.229: process: 0.55889 | load: 0.00246 [I 2022-10-28 04:47:09] ...trap/engines/engine.py.230: loss: 0.11019 [I 2022-10-28 04:47:09] ...optimizers/trijoint.py.151: total_norm: 11.621659 [I 2022-10-28 04:47:14] ...trap/engines/engine.py.224: train: epoch 106 | batch 2180/2304 [I 2022-10-28 04:47:14] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:09 [I 2022-10-28 04:47:14] ...trap/engines/engine.py.229: process: 0.55529 | load: 0.00245 [I 2022-10-28 04:47:14] ...trap/engines/engine.py.230: loss: 0.10326 [I 2022-10-28 04:47:14] ...optimizers/trijoint.py.151: total_norm: 10.526378 [I 2022-10-28 04:47:20] ...trap/engines/engine.py.224: train: epoch 106 | batch 2190/2304 [I 2022-10-28 04:47:20] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:03 [I 2022-10-28 04:47:20] ...trap/engines/engine.py.229: process: 0.55404 | load: 0.00246 [I 2022-10-28 04:47:20] ...trap/engines/engine.py.230: loss: 0.11593 [I 2022-10-28 04:47:20] ...optimizers/trijoint.py.151: total_norm: 12.469576 [I 2022-10-28 04:47:26] ...trap/engines/engine.py.224: train: epoch 106 | batch 2200/2304 [I 2022-10-28 04:47:26] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:00:57 [I 2022-10-28 04:47:26] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00245 [I 2022-10-28 04:47:26] ...trap/engines/engine.py.230: loss: 0.11453 [I 2022-10-28 04:47:26] ...optimizers/trijoint.py.151: total_norm: 8.683882 [I 2022-10-28 04:47:31] ...trap/engines/engine.py.224: train: epoch 106 | batch 2210/2304 [I 2022-10-28 04:47:31] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:00:52 [I 2022-10-28 04:47:31] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00240 [I 2022-10-28 04:47:31] ...trap/engines/engine.py.230: loss: 0.15539 [I 2022-10-28 04:47:31] ...optimizers/trijoint.py.151: total_norm: 28.907141 [I 2022-10-28 04:47:37] ...trap/engines/engine.py.224: train: epoch 106 | batch 2220/2304 [I 2022-10-28 04:47:37] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:00:46 [I 2022-10-28 04:47:37] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00242 [I 2022-10-28 04:47:37] ...trap/engines/engine.py.230: loss: 0.09758 [I 2022-10-28 04:47:37] ...optimizers/trijoint.py.151: total_norm: 14.968194 [I 2022-10-28 04:47:43] ...trap/engines/engine.py.224: train: epoch 106 | batch 2230/2304 [I 2022-10-28 04:47:43] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:41 [I 2022-10-28 04:47:43] ...trap/engines/engine.py.229: process: 0.55628 | load: 0.00241 [I 2022-10-28 04:47:43] ...trap/engines/engine.py.230: loss: 0.10916 [I 2022-10-28 04:47:43] ...optimizers/trijoint.py.151: total_norm: 20.911825 [I 2022-10-28 04:47:48] ...trap/engines/engine.py.224: train: epoch 106 | batch 2240/2304 [I 2022-10-28 04:47:48] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:35 [I 2022-10-28 04:47:48] ...trap/engines/engine.py.229: process: 0.56643 | load: 0.00243 [I 2022-10-28 04:47:48] ...trap/engines/engine.py.230: loss: 0.15263 [I 2022-10-28 04:47:48] ...optimizers/trijoint.py.151: total_norm: 22.343708 [I 2022-10-28 04:47:54] ...trap/engines/engine.py.224: train: epoch 106 | batch 2250/2304 [I 2022-10-28 04:47:54] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:30 [I 2022-10-28 04:47:54] ...trap/engines/engine.py.229: process: 0.55017 | load: 0.00238 [I 2022-10-28 04:47:54] ...trap/engines/engine.py.230: loss: 0.08920 [I 2022-10-28 04:47:54] ...optimizers/trijoint.py.151: total_norm: 9.681160 [I 2022-10-28 04:47:59] ...trap/engines/engine.py.224: train: epoch 106 | batch 2260/2304 [I 2022-10-28 04:47:59] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:24 [I 2022-10-28 04:47:59] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00241 [I 2022-10-28 04:47:59] ...trap/engines/engine.py.230: loss: 0.17159 [I 2022-10-28 04:47:59] ...optimizers/trijoint.py.151: total_norm: 12.577976 [I 2022-10-28 04:48:05] ...trap/engines/engine.py.224: train: epoch 106 | batch 2270/2304 [I 2022-10-28 04:48:05] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:18 [I 2022-10-28 04:48:05] ...trap/engines/engine.py.229: process: 0.55770 | load: 0.00241 [I 2022-10-28 04:48:05] ...trap/engines/engine.py.230: loss: 0.15221 [I 2022-10-28 04:48:05] ...optimizers/trijoint.py.151: total_norm: 14.808249 [I 2022-10-28 04:48:10] ...trap/engines/engine.py.224: train: epoch 106 | batch 2280/2304 [I 2022-10-28 04:48:10] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:13 [I 2022-10-28 04:48:10] ...trap/engines/engine.py.229: process: 0.54590 | load: 0.00240 [I 2022-10-28 04:48:10] ...trap/engines/engine.py.230: loss: 0.11689 [I 2022-10-28 04:48:10] ...optimizers/trijoint.py.151: total_norm: 35.134521 [I 2022-10-28 04:48:16] ...trap/engines/engine.py.224: train: epoch 106 | batch 2290/2304 [I 2022-10-28 04:48:16] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:07 [I 2022-10-28 04:48:16] ...trap/engines/engine.py.229: process: 0.63270 | load: 0.00244 [I 2022-10-28 04:48:16] ...trap/engines/engine.py.230: loss: 0.12457 [I 2022-10-28 04:48:16] ...optimizers/trijoint.py.151: total_norm: 31.839924 [I 2022-10-28 04:48:22] ...trap/engines/engine.py.224: train: epoch 106 | batch 2300/2304 [I 2022-10-28 04:48:22] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:02 [I 2022-10-28 04:48:22] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00014 [I 2022-10-28 04:48:22] ...trap/engines/engine.py.230: loss: 0.09828 [I 2022-10-28 04:48:22] ...optimizers/trijoint.py.151: total_norm: 21.594423 [I 2022-10-28 04:48:24] ...trap/engines/engine.py.224: train: epoch 106 | batch 2304/2304 [I 2022-10-28 04:48:24] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:00 [I 2022-10-28 04:48:24] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00014 [I 2022-10-28 04:48:24] ...trap/engines/engine.py.230: loss: 0.10765 [I 2022-10-28 04:48:24] ...optimizers/trijoint.py.151: total_norm: 15.362802 [S 2022-10-28 04:48:25] ...trap/engines/engine.py.129: train_epoch.epoch: 106 [S 2022-10-28 04:48:25] ...trap/engines/engine.py.129: train_epoch.loss: 0.1227079471803539 [I 2022-10-28 04:48:25] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 04:48:27] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 106 [I 2022-10-28 04:48:27] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 04:48:29] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 04:48:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 04:48:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 04:48:32] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 04:48:32] ...trap/engines/engine.py.321: eval: epoch 106 | batch 0/510 [I 2022-10-28 04:48:32] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:48:42 [I 2022-10-28 04:48:32] ...trap/engines/engine.py.326: process: 5.73122 | load: 2.56840 [I 2022-10-28 04:48:36] ...trap/engines/engine.py.321: eval: epoch 106 | batch 10/510 [I 2022-10-28 04:48:36] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:36 [I 2022-10-28 04:48:36] ...trap/engines/engine.py.326: process: 0.33569 | load: 0.00018 [I 2022-10-28 04:48:39] ...trap/engines/engine.py.321: eval: epoch 106 | batch 20/510 [I 2022-10-28 04:48:39] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:15 [I 2022-10-28 04:48:39] ...trap/engines/engine.py.326: process: 0.33535 | load: 0.00017 [I 2022-10-28 04:48:42] ...trap/engines/engine.py.321: eval: epoch 106 | batch 30/510 [I 2022-10-28 04:48:42] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:41 [I 2022-10-28 04:48:42] ...trap/engines/engine.py.326: process: 0.32649 | load: 0.00017 [I 2022-10-28 04:48:46] ...trap/engines/engine.py.321: eval: epoch 106 | batch 40/510 [I 2022-10-28 04:48:46] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:37 [I 2022-10-28 04:48:46] ...trap/engines/engine.py.326: process: 0.32862 | load: 0.00018 [I 2022-10-28 04:48:49] ...trap/engines/engine.py.321: eval: epoch 106 | batch 50/510 [I 2022-10-28 04:48:49] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:34 [I 2022-10-28 04:48:49] ...trap/engines/engine.py.326: process: 0.33290 | load: 0.00022 [I 2022-10-28 04:48:52] ...trap/engines/engine.py.321: eval: epoch 106 | batch 60/510 [I 2022-10-28 04:48:52] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:31 [I 2022-10-28 04:48:52] ...trap/engines/engine.py.326: process: 0.33917 | load: 0.00037 [I 2022-10-28 04:48:56] ...trap/engines/engine.py.321: eval: epoch 106 | batch 70/510 [I 2022-10-28 04:48:56] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:29 [I 2022-10-28 04:48:56] ...trap/engines/engine.py.326: process: 0.34115 | load: 0.00016 [I 2022-10-28 04:48:59] ...trap/engines/engine.py.321: eval: epoch 106 | batch 80/510 [I 2022-10-28 04:48:59] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:27 [I 2022-10-28 04:48:59] ...trap/engines/engine.py.326: process: 0.34836 | load: 0.00017 [I 2022-10-28 04:49:03] ...trap/engines/engine.py.321: eval: epoch 106 | batch 90/510 [I 2022-10-28 04:49:03] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:20 [I 2022-10-28 04:49:03] ...trap/engines/engine.py.326: process: 0.32746 | load: 0.00021 [I 2022-10-28 04:49:06] ...trap/engines/engine.py.321: eval: epoch 106 | batch 100/510 [I 2022-10-28 04:49:06] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-28 04:49:06] ...trap/engines/engine.py.326: process: 0.28545 | load: 0.00017 [I 2022-10-28 04:49:09] ...trap/engines/engine.py.321: eval: epoch 106 | batch 110/510 [I 2022-10-28 04:49:09] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:56 [I 2022-10-28 04:49:09] ...trap/engines/engine.py.326: process: 0.28073 | load: 0.00014 [I 2022-10-28 04:49:12] ...trap/engines/engine.py.321: eval: epoch 106 | batch 120/510 [I 2022-10-28 04:49:12] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:53 [I 2022-10-28 04:49:12] ...trap/engines/engine.py.326: process: 0.29179 | load: 0.00014 [I 2022-10-28 04:49:15] ...trap/engines/engine.py.321: eval: epoch 106 | batch 130/510 [I 2022-10-28 04:49:15] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-28 04:49:15] ...trap/engines/engine.py.326: process: 0.28811 | load: 0.00018 [I 2022-10-28 04:49:18] ...trap/engines/engine.py.321: eval: epoch 106 | batch 140/510 [I 2022-10-28 04:49:18] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:48 [I 2022-10-28 04:49:18] ...trap/engines/engine.py.326: process: 0.29205 | load: 0.00026 [I 2022-10-28 04:49:20] ...trap/engines/engine.py.321: eval: epoch 106 | batch 150/510 [I 2022-10-28 04:49:20] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:45 [I 2022-10-28 04:49:20] ...trap/engines/engine.py.326: process: 0.28741 | load: 0.00022 [I 2022-10-28 04:49:23] ...trap/engines/engine.py.321: eval: epoch 106 | batch 160/510 [I 2022-10-28 04:49:23] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-28 04:49:23] ...trap/engines/engine.py.326: process: 0.29358 | load: 0.00020 [I 2022-10-28 04:49:26] ...trap/engines/engine.py.321: eval: epoch 106 | batch 170/510 [I 2022-10-28 04:49:26] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 04:49:26] ...trap/engines/engine.py.326: process: 0.28108 | load: 0.00022 [I 2022-10-28 04:49:29] ...trap/engines/engine.py.321: eval: epoch 106 | batch 180/510 [I 2022-10-28 04:49:29] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:36 [I 2022-10-28 04:49:29] ...trap/engines/engine.py.326: process: 0.28052 | load: 0.00019 [I 2022-10-28 04:49:32] ...trap/engines/engine.py.321: eval: epoch 106 | batch 190/510 [I 2022-10-28 04:49:32] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:32 [I 2022-10-28 04:49:32] ...trap/engines/engine.py.326: process: 0.29284 | load: 0.00015 [I 2022-10-28 04:49:35] ...trap/engines/engine.py.321: eval: epoch 106 | batch 200/510 [I 2022-10-28 04:49:35] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:29 [I 2022-10-28 04:49:35] ...trap/engines/engine.py.326: process: 0.28463 | load: 0.00018 [I 2022-10-28 04:49:38] ...trap/engines/engine.py.321: eval: epoch 106 | batch 210/510 [I 2022-10-28 04:49:38] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 04:49:38] ...trap/engines/engine.py.326: process: 0.28671 | load: 0.00015 [I 2022-10-28 04:49:41] ...trap/engines/engine.py.321: eval: epoch 106 | batch 220/510 [I 2022-10-28 04:49:41] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:23 [I 2022-10-28 04:49:41] ...trap/engines/engine.py.326: process: 0.29427 | load: 0.00015 [I 2022-10-28 04:49:44] ...trap/engines/engine.py.321: eval: epoch 106 | batch 230/510 [I 2022-10-28 04:49:44] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:20 [I 2022-10-28 04:49:44] ...trap/engines/engine.py.326: process: 0.29101 | load: 0.00015 [I 2022-10-28 04:49:47] ...trap/engines/engine.py.321: eval: epoch 106 | batch 240/510 [I 2022-10-28 04:49:47] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 04:49:47] ...trap/engines/engine.py.326: process: 0.28557 | load: 0.00021 [I 2022-10-28 04:49:49] ...trap/engines/engine.py.321: eval: epoch 106 | batch 250/510 [I 2022-10-28 04:49:49] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-28 04:49:49] ...trap/engines/engine.py.326: process: 0.28538 | load: 0.00017 [I 2022-10-28 04:49:52] ...trap/engines/engine.py.321: eval: epoch 106 | batch 260/510 [I 2022-10-28 04:49:52] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 04:49:52] ...trap/engines/engine.py.326: process: 0.28583 | load: 0.00017 [I 2022-10-28 04:49:55] ...trap/engines/engine.py.321: eval: epoch 106 | batch 270/510 [I 2022-10-28 04:49:55] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-28 04:49:55] ...trap/engines/engine.py.326: process: 0.28619 | load: 0.00027 [I 2022-10-28 04:49:58] ...trap/engines/engine.py.321: eval: epoch 106 | batch 280/510 [I 2022-10-28 04:49:58] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:05 [I 2022-10-28 04:49:58] ...trap/engines/engine.py.326: process: 0.28523 | load: 0.00015 [I 2022-10-28 04:50:01] ...trap/engines/engine.py.321: eval: epoch 106 | batch 290/510 [I 2022-10-28 04:50:01] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 04:50:01] ...trap/engines/engine.py.326: process: 0.29222 | load: 0.00015 [I 2022-10-28 04:50:04] ...trap/engines/engine.py.321: eval: epoch 106 | batch 300/510 [I 2022-10-28 04:50:04] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-28 04:50:04] ...trap/engines/engine.py.326: process: 0.29103 | load: 0.00022 [I 2022-10-28 04:50:07] ...trap/engines/engine.py.321: eval: epoch 106 | batch 310/510 [I 2022-10-28 04:50:07] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:57 [I 2022-10-28 04:50:07] ...trap/engines/engine.py.326: process: 0.28941 | load: 0.00019 [I 2022-10-28 04:50:10] ...trap/engines/engine.py.321: eval: epoch 106 | batch 320/510 [I 2022-10-28 04:50:10] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 04:50:10] ...trap/engines/engine.py.326: process: 0.28822 | load: 0.00018 [I 2022-10-28 04:50:13] ...trap/engines/engine.py.321: eval: epoch 106 | batch 330/510 [I 2022-10-28 04:50:13] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-28 04:50:13] ...trap/engines/engine.py.326: process: 0.29050 | load: 0.00020 [I 2022-10-28 04:50:15] ...trap/engines/engine.py.321: eval: epoch 106 | batch 340/510 [I 2022-10-28 04:50:15] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-28 04:50:15] ...trap/engines/engine.py.326: process: 0.28616 | load: 0.00016 [I 2022-10-28 04:50:18] ...trap/engines/engine.py.321: eval: epoch 106 | batch 350/510 [I 2022-10-28 04:50:18] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 04:50:18] ...trap/engines/engine.py.326: process: 0.28781 | load: 0.00021 [I 2022-10-28 04:50:21] ...trap/engines/engine.py.321: eval: epoch 106 | batch 360/510 [I 2022-10-28 04:50:21] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 04:50:21] ...trap/engines/engine.py.326: process: 0.28204 | load: 0.00031 [I 2022-10-28 04:50:24] ...trap/engines/engine.py.321: eval: epoch 106 | batch 370/510 [I 2022-10-28 04:50:24] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:39 [I 2022-10-28 04:50:24] ...trap/engines/engine.py.326: process: 0.28115 | load: 0.00015 [I 2022-10-28 04:50:27] ...trap/engines/engine.py.321: eval: epoch 106 | batch 380/510 [I 2022-10-28 04:50:27] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 04:50:27] ...trap/engines/engine.py.326: process: 0.30912 | load: 0.00015 [I 2022-10-28 04:50:30] ...trap/engines/engine.py.321: eval: epoch 106 | batch 390/510 [I 2022-10-28 04:50:30] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 04:50:30] ...trap/engines/engine.py.326: process: 0.28813 | load: 0.00022 [I 2022-10-28 04:50:33] ...trap/engines/engine.py.321: eval: epoch 106 | batch 400/510 [I 2022-10-28 04:50:33] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:31 [I 2022-10-28 04:50:33] ...trap/engines/engine.py.326: process: 0.29678 | load: 0.00015 [I 2022-10-28 04:50:36] ...trap/engines/engine.py.321: eval: epoch 106 | batch 410/510 [I 2022-10-28 04:50:36] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:28 [I 2022-10-28 04:50:36] ...trap/engines/engine.py.326: process: 0.28131 | load: 0.00019 [I 2022-10-28 04:50:39] ...trap/engines/engine.py.321: eval: epoch 106 | batch 420/510 [I 2022-10-28 04:50:39] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 04:50:39] ...trap/engines/engine.py.326: process: 0.28753 | load: 0.00019 [I 2022-10-28 04:50:41] ...trap/engines/engine.py.321: eval: epoch 106 | batch 430/510 [I 2022-10-28 04:50:41] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 04:50:41] ...trap/engines/engine.py.326: process: 0.28759 | load: 0.00015 [I 2022-10-28 04:50:44] ...trap/engines/engine.py.321: eval: epoch 106 | batch 440/510 [I 2022-10-28 04:50:44] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 04:50:44] ...trap/engines/engine.py.326: process: 0.28640 | load: 0.00016 [I 2022-10-28 04:50:47] ...trap/engines/engine.py.321: eval: epoch 106 | batch 450/510 [I 2022-10-28 04:50:47] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 04:50:47] ...trap/engines/engine.py.326: process: 0.27903 | load: 0.00018 [I 2022-10-28 04:50:50] ...trap/engines/engine.py.321: eval: epoch 106 | batch 460/510 [I 2022-10-28 04:50:50] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 04:50:50] ...trap/engines/engine.py.326: process: 0.28560 | load: 0.00020 [I 2022-10-28 04:50:53] ...trap/engines/engine.py.321: eval: epoch 106 | batch 470/510 [I 2022-10-28 04:50:53] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 04:50:53] ...trap/engines/engine.py.326: process: 0.28615 | load: 0.00016 [I 2022-10-28 04:50:56] ...trap/engines/engine.py.321: eval: epoch 106 | batch 480/510 [I 2022-10-28 04:50:56] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 04:50:56] ...trap/engines/engine.py.326: process: 0.28327 | load: 0.00020 [I 2022-10-28 04:50:59] ...trap/engines/engine.py.321: eval: epoch 106 | batch 490/510 [I 2022-10-28 04:50:59] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 04:50:59] ...trap/engines/engine.py.326: process: 0.28636 | load: 0.00021 [I 2022-10-28 04:51:02] ...trap/engines/engine.py.321: eval: epoch 106 | batch 500/510 [I 2022-10-28 04:51:02] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 04:51:02] ...trap/engines/engine.py.326: process: 0.29571 | load: 0.00020 [I 2022-10-28 04:51:04] ...trap/engines/engine.py.321: eval: epoch 106 | batch 510/510 [I 2022-10-28 04:51:04] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 04:51:04] ...trap/engines/engine.py.326: process: 0.28316 | load: 0.00011 [S 2022-10-28 04:51:05] ...trap/engines/engine.py.132: eval_epoch.epoch: 106 [I 2022-10-28 04:51:05] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.737800 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.904500 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.932700 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.731100 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905200 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933700 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017273 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009780 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010479 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011140 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009948 [S 2022-10-28 04:51:09] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010179 [I 2022-10-28 04:51:11] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 04:51:11] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 04:51:12] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 04:51:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 04:51:12] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 04:51:17] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 04:51:17] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 04:51:23] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 04:51:23] ...trap/engines/engine.py.163: Training model on trainset for epoch 107 [I 2022-10-28 04:51:23] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 04:51:26] ...trap/engines/engine.py.224: train: epoch 107 | batch 0/2304 [I 2022-10-28 04:51:26] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:46:32 [I 2022-10-28 04:51:26] ...trap/engines/engine.py.229: process: 2.77474 | load: 2.14639 [I 2022-10-28 04:51:26] ...trap/engines/engine.py.230: loss: 0.09306 [I 2022-10-28 04:51:26] ...optimizers/trijoint.py.151: total_norm: 22.646626 [I 2022-10-28 04:51:32] ...trap/engines/engine.py.224: train: epoch 107 | batch 10/2304 [I 2022-10-28 04:51:32] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:39 [I 2022-10-28 04:51:32] ...trap/engines/engine.py.229: process: 0.55008 | load: 0.00407 [I 2022-10-28 04:51:32] ...trap/engines/engine.py.230: loss: 0.10989 [I 2022-10-28 04:51:32] ...optimizers/trijoint.py.151: total_norm: 12.574002 [I 2022-10-28 04:51:37] ...trap/engines/engine.py.224: train: epoch 107 | batch 20/2304 [I 2022-10-28 04:51:37] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:07 [I 2022-10-28 04:51:37] ...trap/engines/engine.py.229: process: 0.55592 | load: 0.00319 [I 2022-10-28 04:51:37] ...trap/engines/engine.py.230: loss: 0.09471 [I 2022-10-28 04:51:37] ...optimizers/trijoint.py.151: total_norm: 25.565580 [I 2022-10-28 04:51:43] ...trap/engines/engine.py.224: train: epoch 107 | batch 30/2304 [I 2022-10-28 04:51:43] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:09 [I 2022-10-28 04:51:43] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00273 [I 2022-10-28 04:51:43] ...trap/engines/engine.py.230: loss: 0.13067 [I 2022-10-28 04:51:43] ...optimizers/trijoint.py.151: total_norm: 15.333144 [I 2022-10-28 04:51:49] ...trap/engines/engine.py.224: train: epoch 107 | batch 40/2304 [I 2022-10-28 04:51:49] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:00 [I 2022-10-28 04:51:49] ...trap/engines/engine.py.229: process: 0.56131 | load: 0.00282 [I 2022-10-28 04:51:49] ...trap/engines/engine.py.230: loss: 0.11135 [I 2022-10-28 04:51:49] ...optimizers/trijoint.py.151: total_norm: 16.634987 [I 2022-10-28 04:51:54] ...trap/engines/engine.py.224: train: epoch 107 | batch 50/2304 [I 2022-10-28 04:51:54] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:45 [I 2022-10-28 04:51:54] ...trap/engines/engine.py.229: process: 0.54377 | load: 0.00315 [I 2022-10-28 04:51:54] ...trap/engines/engine.py.230: loss: 0.13888 [I 2022-10-28 04:51:54] ...optimizers/trijoint.py.151: total_norm: 10.831697 [I 2022-10-28 04:52:00] ...trap/engines/engine.py.224: train: epoch 107 | batch 60/2304 [I 2022-10-28 04:52:00] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:45 [I 2022-10-28 04:52:00] ...trap/engines/engine.py.229: process: 0.54931 | load: 0.00288 [I 2022-10-28 04:52:00] ...trap/engines/engine.py.230: loss: 0.09926 [I 2022-10-28 04:52:00] ...optimizers/trijoint.py.151: total_norm: 20.825317 [I 2022-10-28 04:52:05] ...trap/engines/engine.py.224: train: epoch 107 | batch 70/2304 [I 2022-10-28 04:52:05] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:45 [I 2022-10-28 04:52:05] ...trap/engines/engine.py.229: process: 0.56401 | load: 0.00242 [I 2022-10-28 04:52:05] ...trap/engines/engine.py.230: loss: 0.13054 [I 2022-10-28 04:52:05] ...optimizers/trijoint.py.151: total_norm: 12.051157 [I 2022-10-28 04:52:11] ...trap/engines/engine.py.224: train: epoch 107 | batch 80/2304 [I 2022-10-28 04:52:11] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:48 [I 2022-10-28 04:52:11] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00315 [I 2022-10-28 04:52:11] ...trap/engines/engine.py.230: loss: 0.10642 [I 2022-10-28 04:52:11] ...optimizers/trijoint.py.151: total_norm: 10.197857 [I 2022-10-28 04:52:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 90/2304 [I 2022-10-28 04:52:35] ...trap/engines/engine.py.228: elapsed: 0:01:12 | left: 1:18:07 [I 2022-10-28 04:52:35] ...trap/engines/engine.py.229: process: 0.56659 | load: 0.00286 [I 2022-10-28 04:52:35] ...trap/engines/engine.py.230: loss: 0.08405 [I 2022-10-28 04:52:35] ...optimizers/trijoint.py.151: total_norm: 8.898668 [I 2022-10-28 04:52:41] ...trap/engines/engine.py.224: train: epoch 107 | batch 100/2304 [I 2022-10-28 04:52:41] ...trap/engines/engine.py.228: elapsed: 0:01:17 | left: 0:26:53 [I 2022-10-28 04:52:41] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00252 [I 2022-10-28 04:52:41] ...trap/engines/engine.py.230: loss: 0.13762 [I 2022-10-28 04:52:41] ...optimizers/trijoint.py.151: total_norm: 21.850697 [I 2022-10-28 04:52:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 110/2304 [I 2022-10-28 04:52:47] ...trap/engines/engine.py.228: elapsed: 0:01:23 | left: 0:21:05 [I 2022-10-28 04:52:47] ...trap/engines/engine.py.229: process: 0.55949 | load: 0.00251 [I 2022-10-28 04:52:47] ...trap/engines/engine.py.230: loss: 0.11384 [I 2022-10-28 04:52:47] ...optimizers/trijoint.py.151: total_norm: 25.493437 [I 2022-10-28 04:52:52] ...trap/engines/engine.py.224: train: epoch 107 | batch 120/2304 [I 2022-10-28 04:52:52] ...trap/engines/engine.py.228: elapsed: 0:01:28 | left: 0:20:22 [I 2022-10-28 04:52:52] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00246 [I 2022-10-28 04:52:52] ...trap/engines/engine.py.230: loss: 0.11930 [I 2022-10-28 04:52:52] ...optimizers/trijoint.py.151: total_norm: 28.178568 [I 2022-10-28 04:52:58] ...trap/engines/engine.py.224: train: epoch 107 | batch 130/2304 [I 2022-10-28 04:52:58] ...trap/engines/engine.py.228: elapsed: 0:01:34 | left: 0:20:26 [I 2022-10-28 04:52:58] ...trap/engines/engine.py.229: process: 0.57241 | load: 0.00267 [I 2022-10-28 04:52:58] ...trap/engines/engine.py.230: loss: 0.09803 [I 2022-10-28 04:52:58] ...optimizers/trijoint.py.151: total_norm: 12.120959 [I 2022-10-28 04:53:03] ...trap/engines/engine.py.224: train: epoch 107 | batch 140/2304 [I 2022-10-28 04:53:03] ...trap/engines/engine.py.228: elapsed: 0:01:40 | left: 0:20:07 [I 2022-10-28 04:53:03] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00261 [I 2022-10-28 04:53:03] ...trap/engines/engine.py.230: loss: 0.15919 [I 2022-10-28 04:53:03] ...optimizers/trijoint.py.151: total_norm: 16.725018 [I 2022-10-28 04:53:09] ...trap/engines/engine.py.224: train: epoch 107 | batch 150/2304 [I 2022-10-28 04:53:09] ...trap/engines/engine.py.228: elapsed: 0:01:45 | left: 0:20:06 [I 2022-10-28 04:53:09] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00245 [I 2022-10-28 04:53:09] ...trap/engines/engine.py.230: loss: 0.10153 [I 2022-10-28 04:53:09] ...optimizers/trijoint.py.151: total_norm: 12.878762 [I 2022-10-28 04:53:15] ...trap/engines/engine.py.224: train: epoch 107 | batch 160/2304 [I 2022-10-28 04:53:15] ...trap/engines/engine.py.228: elapsed: 0:01:51 | left: 0:19:59 [I 2022-10-28 04:53:15] ...trap/engines/engine.py.229: process: 0.55918 | load: 0.00242 [I 2022-10-28 04:53:15] ...trap/engines/engine.py.230: loss: 0.14760 [I 2022-10-28 04:53:15] ...optimizers/trijoint.py.151: total_norm: 12.614964 [I 2022-10-28 04:53:20] ...trap/engines/engine.py.224: train: epoch 107 | batch 170/2304 [I 2022-10-28 04:53:20] ...trap/engines/engine.py.228: elapsed: 0:01:57 | left: 0:20:04 [I 2022-10-28 04:53:20] ...trap/engines/engine.py.229: process: 0.56591 | load: 0.00290 [I 2022-10-28 04:53:20] ...trap/engines/engine.py.230: loss: 0.14914 [I 2022-10-28 04:53:20] ...optimizers/trijoint.py.151: total_norm: 27.305725 [I 2022-10-28 04:53:26] ...trap/engines/engine.py.224: train: epoch 107 | batch 180/2304 [I 2022-10-28 04:53:26] ...trap/engines/engine.py.228: elapsed: 0:02:02 | left: 0:19:52 [I 2022-10-28 04:53:26] ...trap/engines/engine.py.229: process: 0.55900 | load: 0.00245 [I 2022-10-28 04:53:26] ...trap/engines/engine.py.230: loss: 0.14513 [I 2022-10-28 04:53:26] ...optimizers/trijoint.py.151: total_norm: 38.850212 [I 2022-10-28 04:53:32] ...trap/engines/engine.py.224: train: epoch 107 | batch 190/2304 [I 2022-10-28 04:53:32] ...trap/engines/engine.py.228: elapsed: 0:02:08 | left: 0:19:43 [I 2022-10-28 04:53:32] ...trap/engines/engine.py.229: process: 0.56134 | load: 0.00248 [I 2022-10-28 04:53:32] ...trap/engines/engine.py.230: loss: 0.11297 [I 2022-10-28 04:53:32] ...optimizers/trijoint.py.151: total_norm: 17.280243 [I 2022-10-28 04:53:37] ...trap/engines/engine.py.224: train: epoch 107 | batch 200/2304 [I 2022-10-28 04:53:37] ...trap/engines/engine.py.228: elapsed: 0:02:13 | left: 0:19:37 [I 2022-10-28 04:53:37] ...trap/engines/engine.py.229: process: 0.55800 | load: 0.00268 [I 2022-10-28 04:53:37] ...trap/engines/engine.py.230: loss: 0.10629 [I 2022-10-28 04:53:37] ...optimizers/trijoint.py.151: total_norm: 10.849928 [I 2022-10-28 04:53:43] ...trap/engines/engine.py.224: train: epoch 107 | batch 210/2304 [I 2022-10-28 04:53:43] ...trap/engines/engine.py.228: elapsed: 0:02:19 | left: 0:19:28 [I 2022-10-28 04:53:43] ...trap/engines/engine.py.229: process: 0.55718 | load: 0.00237 [I 2022-10-28 04:53:43] ...trap/engines/engine.py.230: loss: 0.12246 [I 2022-10-28 04:53:43] ...optimizers/trijoint.py.151: total_norm: 21.883495 [I 2022-10-28 04:53:48] ...trap/engines/engine.py.224: train: epoch 107 | batch 220/2304 [I 2022-10-28 04:53:48] ...trap/engines/engine.py.228: elapsed: 0:02:25 | left: 0:19:28 [I 2022-10-28 04:53:48] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00242 [I 2022-10-28 04:53:48] ...trap/engines/engine.py.230: loss: 0.17372 [I 2022-10-28 04:53:48] ...optimizers/trijoint.py.151: total_norm: 16.792208 [I 2022-10-28 04:53:54] ...trap/engines/engine.py.224: train: epoch 107 | batch 230/2304 [I 2022-10-28 04:53:54] ...trap/engines/engine.py.228: elapsed: 0:02:30 | left: 0:19:15 [I 2022-10-28 04:53:54] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00254 [I 2022-10-28 04:53:54] ...trap/engines/engine.py.230: loss: 0.18561 [I 2022-10-28 04:53:54] ...optimizers/trijoint.py.151: total_norm: 13.233284 [I 2022-10-28 04:54:00] ...trap/engines/engine.py.224: train: epoch 107 | batch 240/2304 [I 2022-10-28 04:54:00] ...trap/engines/engine.py.228: elapsed: 0:02:36 | left: 0:19:20 [I 2022-10-28 04:54:00] ...trap/engines/engine.py.229: process: 0.56843 | load: 0.00256 [I 2022-10-28 04:54:00] ...trap/engines/engine.py.230: loss: 0.06752 [I 2022-10-28 04:54:00] ...optimizers/trijoint.py.151: total_norm: 19.267178 [I 2022-10-28 04:54:05] ...trap/engines/engine.py.224: train: epoch 107 | batch 250/2304 [I 2022-10-28 04:54:05] ...trap/engines/engine.py.228: elapsed: 0:02:41 | left: 0:19:11 [I 2022-10-28 04:54:05] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00249 [I 2022-10-28 04:54:05] ...trap/engines/engine.py.230: loss: 0.12564 [I 2022-10-28 04:54:05] ...optimizers/trijoint.py.151: total_norm: 26.667809 [I 2022-10-28 04:54:11] ...trap/engines/engine.py.224: train: epoch 107 | batch 260/2304 [I 2022-10-28 04:54:11] ...trap/engines/engine.py.228: elapsed: 0:02:47 | left: 0:19:08 [I 2022-10-28 04:54:11] ...trap/engines/engine.py.229: process: 0.56542 | load: 0.00238 [I 2022-10-28 04:54:11] ...trap/engines/engine.py.230: loss: 0.14350 [I 2022-10-28 04:54:11] ...optimizers/trijoint.py.151: total_norm: 18.506180 [I 2022-10-28 04:54:16] ...trap/engines/engine.py.224: train: epoch 107 | batch 270/2304 [I 2022-10-28 04:54:16] ...trap/engines/engine.py.228: elapsed: 0:02:53 | left: 0:18:52 [I 2022-10-28 04:54:16] ...trap/engines/engine.py.229: process: 0.54821 | load: 0.00241 [I 2022-10-28 04:54:16] ...trap/engines/engine.py.230: loss: 0.19856 [I 2022-10-28 04:54:16] ...optimizers/trijoint.py.151: total_norm: 18.555737 [I 2022-10-28 04:54:22] ...trap/engines/engine.py.224: train: epoch 107 | batch 280/2304 [I 2022-10-28 04:54:22] ...trap/engines/engine.py.228: elapsed: 0:02:58 | left: 0:18:51 [I 2022-10-28 04:54:22] ...trap/engines/engine.py.229: process: 0.55575 | load: 0.00268 [I 2022-10-28 04:54:22] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-28 04:54:22] ...optimizers/trijoint.py.151: total_norm: 12.016444 [I 2022-10-28 04:54:28] ...trap/engines/engine.py.224: train: epoch 107 | batch 290/2304 [I 2022-10-28 04:54:28] ...trap/engines/engine.py.228: elapsed: 0:03:04 | left: 0:18:46 [I 2022-10-28 04:54:28] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00258 [I 2022-10-28 04:54:28] ...trap/engines/engine.py.230: loss: 0.10783 [I 2022-10-28 04:54:28] ...optimizers/trijoint.py.151: total_norm: 18.250504 [I 2022-10-28 04:54:33] ...trap/engines/engine.py.224: train: epoch 107 | batch 300/2304 [I 2022-10-28 04:54:33] ...trap/engines/engine.py.228: elapsed: 0:03:09 | left: 0:18:42 [I 2022-10-28 04:54:33] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00248 [I 2022-10-28 04:54:33] ...trap/engines/engine.py.230: loss: 0.12162 [I 2022-10-28 04:54:33] ...optimizers/trijoint.py.151: total_norm: 12.462723 [I 2022-10-28 04:54:39] ...trap/engines/engine.py.224: train: epoch 107 | batch 310/2304 [I 2022-10-28 04:54:39] ...trap/engines/engine.py.228: elapsed: 0:03:15 | left: 0:18:45 [I 2022-10-28 04:54:39] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00258 [I 2022-10-28 04:54:39] ...trap/engines/engine.py.230: loss: 0.12142 [I 2022-10-28 04:54:39] ...optimizers/trijoint.py.151: total_norm: 15.214061 [I 2022-10-28 04:54:45] ...trap/engines/engine.py.224: train: epoch 107 | batch 320/2304 [I 2022-10-28 04:54:45] ...trap/engines/engine.py.228: elapsed: 0:03:21 | left: 0:18:39 [I 2022-10-28 04:54:45] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00249 [I 2022-10-28 04:54:45] ...trap/engines/engine.py.230: loss: 0.11231 [I 2022-10-28 04:54:45] ...optimizers/trijoint.py.151: total_norm: 20.423357 [I 2022-10-28 04:54:50] ...trap/engines/engine.py.224: train: epoch 107 | batch 330/2304 [I 2022-10-28 04:54:50] ...trap/engines/engine.py.228: elapsed: 0:03:26 | left: 0:18:23 [I 2022-10-28 04:54:50] ...trap/engines/engine.py.229: process: 0.56207 | load: 0.00259 [I 2022-10-28 04:54:50] ...trap/engines/engine.py.230: loss: 0.17156 [I 2022-10-28 04:54:50] ...optimizers/trijoint.py.151: total_norm: 9.360662 [I 2022-10-28 04:54:56] ...trap/engines/engine.py.224: train: epoch 107 | batch 340/2304 [I 2022-10-28 04:54:56] ...trap/engines/engine.py.228: elapsed: 0:03:32 | left: 0:18:21 [I 2022-10-28 04:54:56] ...trap/engines/engine.py.229: process: 0.56535 | load: 0.00255 [I 2022-10-28 04:54:56] ...trap/engines/engine.py.230: loss: 0.13456 [I 2022-10-28 04:54:56] ...optimizers/trijoint.py.151: total_norm: 28.076509 [I 2022-10-28 04:55:01] ...trap/engines/engine.py.224: train: epoch 107 | batch 350/2304 [I 2022-10-28 04:55:01] ...trap/engines/engine.py.228: elapsed: 0:03:38 | left: 0:18:20 [I 2022-10-28 04:55:01] ...trap/engines/engine.py.229: process: 0.56313 | load: 0.00245 [I 2022-10-28 04:55:01] ...trap/engines/engine.py.230: loss: 0.10702 [I 2022-10-28 04:55:01] ...optimizers/trijoint.py.151: total_norm: 35.503025 [I 2022-10-28 04:55:07] ...trap/engines/engine.py.224: train: epoch 107 | batch 360/2304 [I 2022-10-28 04:55:07] ...trap/engines/engine.py.228: elapsed: 0:03:43 | left: 0:18:03 [I 2022-10-28 04:55:07] ...trap/engines/engine.py.229: process: 0.55280 | load: 0.00250 [I 2022-10-28 04:55:07] ...trap/engines/engine.py.230: loss: 0.12092 [I 2022-10-28 04:55:07] ...optimizers/trijoint.py.151: total_norm: 23.481453 [I 2022-10-28 04:55:13] ...trap/engines/engine.py.224: train: epoch 107 | batch 370/2304 [I 2022-10-28 04:55:13] ...trap/engines/engine.py.228: elapsed: 0:03:49 | left: 0:18:00 [I 2022-10-28 04:55:13] ...trap/engines/engine.py.229: process: 0.54765 | load: 0.00255 [I 2022-10-28 04:55:13] ...trap/engines/engine.py.230: loss: 0.11199 [I 2022-10-28 04:55:13] ...optimizers/trijoint.py.151: total_norm: 22.903605 [I 2022-10-28 04:55:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 380/2304 [I 2022-10-28 04:55:18] ...trap/engines/engine.py.228: elapsed: 0:03:54 | left: 0:17:47 [I 2022-10-28 04:55:18] ...trap/engines/engine.py.229: process: 0.55066 | load: 0.00251 [I 2022-10-28 04:55:18] ...trap/engines/engine.py.230: loss: 0.09095 [I 2022-10-28 04:55:18] ...optimizers/trijoint.py.151: total_norm: 22.675259 [I 2022-10-28 04:55:24] ...trap/engines/engine.py.224: train: epoch 107 | batch 390/2304 [I 2022-10-28 04:55:24] ...trap/engines/engine.py.228: elapsed: 0:04:00 | left: 0:17:44 [I 2022-10-28 04:55:24] ...trap/engines/engine.py.229: process: 0.55140 | load: 0.00242 [I 2022-10-28 04:55:24] ...trap/engines/engine.py.230: loss: 0.16715 [I 2022-10-28 04:55:24] ...optimizers/trijoint.py.151: total_norm: 32.860146 [I 2022-10-28 04:55:29] ...trap/engines/engine.py.224: train: epoch 107 | batch 400/2304 [I 2022-10-28 04:55:29] ...trap/engines/engine.py.228: elapsed: 0:04:06 | left: 0:17:50 [I 2022-10-28 04:55:29] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00247 [I 2022-10-28 04:55:29] ...trap/engines/engine.py.230: loss: 0.14469 [I 2022-10-28 04:55:29] ...optimizers/trijoint.py.151: total_norm: 18.567488 [I 2022-10-28 04:55:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 410/2304 [I 2022-10-28 04:55:35] ...trap/engines/engine.py.228: elapsed: 0:04:11 | left: 0:17:36 [I 2022-10-28 04:55:35] ...trap/engines/engine.py.229: process: 0.54185 | load: 0.00243 [I 2022-10-28 04:55:35] ...trap/engines/engine.py.230: loss: 0.08832 [I 2022-10-28 04:55:35] ...optimizers/trijoint.py.151: total_norm: 10.287059 [I 2022-10-28 04:55:40] ...trap/engines/engine.py.224: train: epoch 107 | batch 420/2304 [I 2022-10-28 04:55:40] ...trap/engines/engine.py.228: elapsed: 0:04:17 | left: 0:17:18 [I 2022-10-28 04:55:40] ...trap/engines/engine.py.229: process: 0.56032 | load: 0.00242 [I 2022-10-28 04:55:40] ...trap/engines/engine.py.230: loss: 0.09664 [I 2022-10-28 04:55:40] ...optimizers/trijoint.py.151: total_norm: 17.330593 [I 2022-10-28 04:55:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 430/2304 [I 2022-10-28 04:55:46] ...trap/engines/engine.py.228: elapsed: 0:04:22 | left: 0:17:26 [I 2022-10-28 04:55:46] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00244 [I 2022-10-28 04:55:46] ...trap/engines/engine.py.230: loss: 0.08521 [I 2022-10-28 04:55:46] ...optimizers/trijoint.py.151: total_norm: 14.376628 [I 2022-10-28 04:55:52] ...trap/engines/engine.py.224: train: epoch 107 | batch 440/2304 [I 2022-10-28 04:55:52] ...trap/engines/engine.py.228: elapsed: 0:04:28 | left: 0:17:31 [I 2022-10-28 04:55:52] ...trap/engines/engine.py.229: process: 0.56237 | load: 0.00243 [I 2022-10-28 04:55:52] ...trap/engines/engine.py.230: loss: 0.12502 [I 2022-10-28 04:55:52] ...optimizers/trijoint.py.151: total_norm: 16.086208 [I 2022-10-28 04:55:57] ...trap/engines/engine.py.224: train: epoch 107 | batch 450/2304 [I 2022-10-28 04:55:57] ...trap/engines/engine.py.228: elapsed: 0:04:33 | left: 0:17:19 [I 2022-10-28 04:55:57] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00250 [I 2022-10-28 04:55:57] ...trap/engines/engine.py.230: loss: 0.09960 [I 2022-10-28 04:55:57] ...optimizers/trijoint.py.151: total_norm: 13.812083 [I 2022-10-28 04:56:03] ...trap/engines/engine.py.224: train: epoch 107 | batch 460/2304 [I 2022-10-28 04:56:03] ...trap/engines/engine.py.228: elapsed: 0:04:39 | left: 0:17:04 [I 2022-10-28 04:56:03] ...trap/engines/engine.py.229: process: 0.55152 | load: 0.00242 [I 2022-10-28 04:56:03] ...trap/engines/engine.py.230: loss: 0.10231 [I 2022-10-28 04:56:03] ...optimizers/trijoint.py.151: total_norm: 10.425130 [I 2022-10-28 04:56:08] ...trap/engines/engine.py.224: train: epoch 107 | batch 470/2304 [I 2022-10-28 04:56:08] ...trap/engines/engine.py.228: elapsed: 0:04:45 | left: 0:16:57 [I 2022-10-28 04:56:08] ...trap/engines/engine.py.229: process: 0.55499 | load: 0.00263 [I 2022-10-28 04:56:08] ...trap/engines/engine.py.230: loss: 0.11289 [I 2022-10-28 04:56:08] ...optimizers/trijoint.py.151: total_norm: 11.548393 [I 2022-10-28 04:56:14] ...trap/engines/engine.py.224: train: epoch 107 | batch 480/2304 [I 2022-10-28 04:56:14] ...trap/engines/engine.py.228: elapsed: 0:04:50 | left: 0:17:03 [I 2022-10-28 04:56:14] ...trap/engines/engine.py.229: process: 0.57316 | load: 0.00254 [I 2022-10-28 04:56:14] ...trap/engines/engine.py.230: loss: 0.15476 [I 2022-10-28 04:56:14] ...optimizers/trijoint.py.151: total_norm: 27.264484 [I 2022-10-28 04:56:20] ...trap/engines/engine.py.224: train: epoch 107 | batch 490/2304 [I 2022-10-28 04:56:20] ...trap/engines/engine.py.228: elapsed: 0:04:56 | left: 0:17:17 [I 2022-10-28 04:56:20] ...trap/engines/engine.py.229: process: 0.57895 | load: 0.00246 [I 2022-10-28 04:56:20] ...trap/engines/engine.py.230: loss: 0.12124 [I 2022-10-28 04:56:20] ...optimizers/trijoint.py.151: total_norm: 11.999786 [I 2022-10-28 04:56:25] ...trap/engines/engine.py.224: train: epoch 107 | batch 500/2304 [I 2022-10-28 04:56:25] ...trap/engines/engine.py.228: elapsed: 0:05:02 | left: 0:16:52 [I 2022-10-28 04:56:25] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00260 [I 2022-10-28 04:56:25] ...trap/engines/engine.py.230: loss: 0.14383 [I 2022-10-28 04:56:25] ...optimizers/trijoint.py.151: total_norm: 24.054338 [I 2022-10-28 04:56:31] ...trap/engines/engine.py.224: train: epoch 107 | batch 510/2304 [I 2022-10-28 04:56:31] ...trap/engines/engine.py.228: elapsed: 0:05:07 | left: 0:16:49 [I 2022-10-28 04:56:31] ...trap/engines/engine.py.229: process: 0.57146 | load: 0.00252 [I 2022-10-28 04:56:31] ...trap/engines/engine.py.230: loss: 0.09438 [I 2022-10-28 04:56:31] ...optimizers/trijoint.py.151: total_norm: 11.689193 [I 2022-10-28 04:56:37] ...trap/engines/engine.py.224: train: epoch 107 | batch 520/2304 [I 2022-10-28 04:56:37] ...trap/engines/engine.py.228: elapsed: 0:05:13 | left: 0:16:44 [I 2022-10-28 04:56:37] ...trap/engines/engine.py.229: process: 0.55588 | load: 0.00261 [I 2022-10-28 04:56:37] ...trap/engines/engine.py.230: loss: 0.11053 [I 2022-10-28 04:56:37] ...optimizers/trijoint.py.151: total_norm: 10.308587 [I 2022-10-28 04:56:42] ...trap/engines/engine.py.224: train: epoch 107 | batch 530/2304 [I 2022-10-28 04:56:42] ...trap/engines/engine.py.228: elapsed: 0:05:19 | left: 0:16:30 [I 2022-10-28 04:56:42] ...trap/engines/engine.py.229: process: 0.56465 | load: 0.00269 [I 2022-10-28 04:56:42] ...trap/engines/engine.py.230: loss: 0.08807 [I 2022-10-28 04:56:42] ...optimizers/trijoint.py.151: total_norm: 13.477049 [I 2022-10-28 04:56:48] ...trap/engines/engine.py.224: train: epoch 107 | batch 540/2304 [I 2022-10-28 04:56:48] ...trap/engines/engine.py.228: elapsed: 0:05:24 | left: 0:16:39 [I 2022-10-28 04:56:48] ...trap/engines/engine.py.229: process: 0.56875 | load: 0.00269 [I 2022-10-28 04:56:48] ...trap/engines/engine.py.230: loss: 0.10971 [I 2022-10-28 04:56:48] ...optimizers/trijoint.py.151: total_norm: 7.305295 [I 2022-10-28 04:56:54] ...trap/engines/engine.py.224: train: epoch 107 | batch 550/2304 [I 2022-10-28 04:56:54] ...trap/engines/engine.py.228: elapsed: 0:05:30 | left: 0:16:26 [I 2022-10-28 04:56:54] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00260 [I 2022-10-28 04:56:54] ...trap/engines/engine.py.230: loss: 0.16957 [I 2022-10-28 04:56:54] ...optimizers/trijoint.py.151: total_norm: 21.313204 [I 2022-10-28 04:56:59] ...trap/engines/engine.py.224: train: epoch 107 | batch 560/2304 [I 2022-10-28 04:56:59] ...trap/engines/engine.py.228: elapsed: 0:05:35 | left: 0:16:41 [I 2022-10-28 04:56:59] ...trap/engines/engine.py.229: process: 0.64549 | load: 0.00262 [I 2022-10-28 04:56:59] ...trap/engines/engine.py.230: loss: 0.11360 [I 2022-10-28 04:56:59] ...optimizers/trijoint.py.151: total_norm: 25.140390 [I 2022-10-28 04:57:05] ...trap/engines/engine.py.224: train: epoch 107 | batch 570/2304 [I 2022-10-28 04:57:05] ...trap/engines/engine.py.228: elapsed: 0:05:41 | left: 0:16:20 [I 2022-10-28 04:57:05] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00253 [I 2022-10-28 04:57:05] ...trap/engines/engine.py.230: loss: 0.09163 [I 2022-10-28 04:57:05] ...optimizers/trijoint.py.151: total_norm: 8.590029 [I 2022-10-28 04:57:11] ...trap/engines/engine.py.224: train: epoch 107 | batch 580/2304 [I 2022-10-28 04:57:11] ...trap/engines/engine.py.228: elapsed: 0:05:47 | left: 0:15:55 [I 2022-10-28 04:57:11] ...trap/engines/engine.py.229: process: 0.54519 | load: 0.00256 [I 2022-10-28 04:57:11] ...trap/engines/engine.py.230: loss: 0.09274 [I 2022-10-28 04:57:11] ...optimizers/trijoint.py.151: total_norm: 8.542117 [I 2022-10-28 04:57:16] ...trap/engines/engine.py.224: train: epoch 107 | batch 590/2304 [I 2022-10-28 04:57:16] ...trap/engines/engine.py.228: elapsed: 0:05:52 | left: 0:15:56 [I 2022-10-28 04:57:16] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00243 [I 2022-10-28 04:57:16] ...trap/engines/engine.py.230: loss: 0.13758 [I 2022-10-28 04:57:16] ...optimizers/trijoint.py.151: total_norm: 16.853369 [I 2022-10-28 04:57:22] ...trap/engines/engine.py.224: train: epoch 107 | batch 600/2304 [I 2022-10-28 04:57:22] ...trap/engines/engine.py.228: elapsed: 0:05:58 | left: 0:15:53 [I 2022-10-28 04:57:22] ...trap/engines/engine.py.229: process: 0.56503 | load: 0.00248 [I 2022-10-28 04:57:22] ...trap/engines/engine.py.230: loss: 0.11338 [I 2022-10-28 04:57:22] ...optimizers/trijoint.py.151: total_norm: 17.769278 [I 2022-10-28 04:57:27] ...trap/engines/engine.py.224: train: epoch 107 | batch 610/2304 [I 2022-10-28 04:57:27] ...trap/engines/engine.py.228: elapsed: 0:06:04 | left: 0:15:48 [I 2022-10-28 04:57:27] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00253 [I 2022-10-28 04:57:27] ...trap/engines/engine.py.230: loss: 0.13357 [I 2022-10-28 04:57:27] ...optimizers/trijoint.py.151: total_norm: 14.592123 [I 2022-10-28 04:57:33] ...trap/engines/engine.py.224: train: epoch 107 | batch 620/2304 [I 2022-10-28 04:57:33] ...trap/engines/engine.py.228: elapsed: 0:06:09 | left: 0:15:37 [I 2022-10-28 04:57:33] ...trap/engines/engine.py.229: process: 0.54419 | load: 0.00246 [I 2022-10-28 04:57:33] ...trap/engines/engine.py.230: loss: 0.12877 [I 2022-10-28 04:57:33] ...optimizers/trijoint.py.151: total_norm: 13.384212 [I 2022-10-28 04:57:38] ...trap/engines/engine.py.224: train: epoch 107 | batch 630/2304 [I 2022-10-28 04:57:38] ...trap/engines/engine.py.228: elapsed: 0:06:15 | left: 0:15:34 [I 2022-10-28 04:57:38] ...trap/engines/engine.py.229: process: 0.57255 | load: 0.00248 [I 2022-10-28 04:57:38] ...trap/engines/engine.py.230: loss: 0.08984 [I 2022-10-28 04:57:38] ...optimizers/trijoint.py.151: total_norm: 13.935185 [I 2022-10-28 04:57:44] ...trap/engines/engine.py.224: train: epoch 107 | batch 640/2304 [I 2022-10-28 04:57:44] ...trap/engines/engine.py.228: elapsed: 0:06:20 | left: 0:15:43 [I 2022-10-28 04:57:44] ...trap/engines/engine.py.229: process: 0.56772 | load: 0.00248 [I 2022-10-28 04:57:44] ...trap/engines/engine.py.230: loss: 0.21513 [I 2022-10-28 04:57:44] ...optimizers/trijoint.py.151: total_norm: 24.682249 [I 2022-10-28 04:57:50] ...trap/engines/engine.py.224: train: epoch 107 | batch 650/2304 [I 2022-10-28 04:57:50] ...trap/engines/engine.py.228: elapsed: 0:06:26 | left: 0:15:22 [I 2022-10-28 04:57:50] ...trap/engines/engine.py.229: process: 0.55897 | load: 0.00245 [I 2022-10-28 04:57:50] ...trap/engines/engine.py.230: loss: 0.09342 [I 2022-10-28 04:57:50] ...optimizers/trijoint.py.151: total_norm: 12.515296 [I 2022-10-28 04:57:55] ...trap/engines/engine.py.224: train: epoch 107 | batch 660/2304 [I 2022-10-28 04:57:55] ...trap/engines/engine.py.228: elapsed: 0:06:32 | left: 0:15:13 [I 2022-10-28 04:57:55] ...trap/engines/engine.py.229: process: 0.56130 | load: 0.00247 [I 2022-10-28 04:57:55] ...trap/engines/engine.py.230: loss: 0.08792 [I 2022-10-28 04:57:55] ...optimizers/trijoint.py.151: total_norm: 16.501934 [I 2022-10-28 04:58:01] ...trap/engines/engine.py.224: train: epoch 107 | batch 670/2304 [I 2022-10-28 04:58:01] ...trap/engines/engine.py.228: elapsed: 0:06:37 | left: 0:15:06 [I 2022-10-28 04:58:01] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00243 [I 2022-10-28 04:58:01] ...trap/engines/engine.py.230: loss: 0.07961 [I 2022-10-28 04:58:01] ...optimizers/trijoint.py.151: total_norm: 11.213334 [I 2022-10-28 04:58:07] ...trap/engines/engine.py.224: train: epoch 107 | batch 680/2304 [I 2022-10-28 04:58:07] ...trap/engines/engine.py.228: elapsed: 0:06:43 | left: 0:15:04 [I 2022-10-28 04:58:07] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00243 [I 2022-10-28 04:58:07] ...trap/engines/engine.py.230: loss: 0.11583 [I 2022-10-28 04:58:07] ...optimizers/trijoint.py.151: total_norm: 9.338746 [I 2022-10-28 04:58:12] ...trap/engines/engine.py.224: train: epoch 107 | batch 690/2304 [I 2022-10-28 04:58:12] ...trap/engines/engine.py.228: elapsed: 0:06:48 | left: 0:15:06 [I 2022-10-28 04:58:12] ...trap/engines/engine.py.229: process: 0.57514 | load: 0.00241 [I 2022-10-28 04:58:12] ...trap/engines/engine.py.230: loss: 0.13650 [I 2022-10-28 04:58:12] ...optimizers/trijoint.py.151: total_norm: 14.763934 [I 2022-10-28 04:58:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 700/2304 [I 2022-10-28 04:58:18] ...trap/engines/engine.py.228: elapsed: 0:06:54 | left: 0:15:08 [I 2022-10-28 04:58:18] ...trap/engines/engine.py.229: process: 0.55679 | load: 0.00258 [I 2022-10-28 04:58:18] ...trap/engines/engine.py.230: loss: 0.15183 [I 2022-10-28 04:58:18] ...optimizers/trijoint.py.151: total_norm: 31.586555 [I 2022-10-28 04:58:23] ...trap/engines/engine.py.224: train: epoch 107 | batch 710/2304 [I 2022-10-28 04:58:23] ...trap/engines/engine.py.228: elapsed: 0:07:00 | left: 0:14:49 [I 2022-10-28 04:58:23] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00252 [I 2022-10-28 04:58:23] ...trap/engines/engine.py.230: loss: 0.12003 [I 2022-10-28 04:58:23] ...optimizers/trijoint.py.151: total_norm: 15.555399 [I 2022-10-28 04:58:29] ...trap/engines/engine.py.224: train: epoch 107 | batch 720/2304 [I 2022-10-28 04:58:29] ...trap/engines/engine.py.228: elapsed: 0:07:05 | left: 0:14:48 [I 2022-10-28 04:58:29] ...trap/engines/engine.py.229: process: 0.57468 | load: 0.00253 [I 2022-10-28 04:58:29] ...trap/engines/engine.py.230: loss: 0.16609 [I 2022-10-28 04:58:29] ...optimizers/trijoint.py.151: total_norm: 36.991047 [I 2022-10-28 04:58:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 730/2304 [I 2022-10-28 04:58:35] ...trap/engines/engine.py.228: elapsed: 0:07:11 | left: 0:14:43 [I 2022-10-28 04:58:35] ...trap/engines/engine.py.229: process: 0.56349 | load: 0.00248 [I 2022-10-28 04:58:35] ...trap/engines/engine.py.230: loss: 0.11294 [I 2022-10-28 04:58:35] ...optimizers/trijoint.py.151: total_norm: 15.457419 [I 2022-10-28 04:58:40] ...trap/engines/engine.py.224: train: epoch 107 | batch 740/2304 [I 2022-10-28 04:58:40] ...trap/engines/engine.py.228: elapsed: 0:07:16 | left: 0:14:30 [I 2022-10-28 04:58:40] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00243 [I 2022-10-28 04:58:40] ...trap/engines/engine.py.230: loss: 0.11292 [I 2022-10-28 04:58:40] ...optimizers/trijoint.py.151: total_norm: 8.580058 [I 2022-10-28 04:58:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 750/2304 [I 2022-10-28 04:58:46] ...trap/engines/engine.py.228: elapsed: 0:07:22 | left: 0:14:29 [I 2022-10-28 04:58:46] ...trap/engines/engine.py.229: process: 0.56005 | load: 0.00245 [I 2022-10-28 04:58:46] ...trap/engines/engine.py.230: loss: 0.16134 [I 2022-10-28 04:58:46] ...optimizers/trijoint.py.151: total_norm: 12.067765 [I 2022-10-28 04:58:51] ...trap/engines/engine.py.224: train: epoch 107 | batch 760/2304 [I 2022-10-28 04:58:51] ...trap/engines/engine.py.228: elapsed: 0:07:28 | left: 0:14:24 [I 2022-10-28 04:58:51] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00248 [I 2022-10-28 04:58:51] ...trap/engines/engine.py.230: loss: 0.14244 [I 2022-10-28 04:58:51] ...optimizers/trijoint.py.151: total_norm: 27.694740 [I 2022-10-28 04:58:57] ...trap/engines/engine.py.224: train: epoch 107 | batch 770/2304 [I 2022-10-28 04:58:57] ...trap/engines/engine.py.228: elapsed: 0:07:33 | left: 0:14:32 [I 2022-10-28 04:58:57] ...trap/engines/engine.py.229: process: 0.57664 | load: 0.00250 [I 2022-10-28 04:58:57] ...trap/engines/engine.py.230: loss: 0.08822 [I 2022-10-28 04:58:57] ...optimizers/trijoint.py.151: total_norm: 18.838036 [I 2022-10-28 04:59:03] ...trap/engines/engine.py.224: train: epoch 107 | batch 780/2304 [I 2022-10-28 04:59:03] ...trap/engines/engine.py.228: elapsed: 0:07:39 | left: 0:14:21 [I 2022-10-28 04:59:03] ...trap/engines/engine.py.229: process: 0.57266 | load: 0.00243 [I 2022-10-28 04:59:03] ...trap/engines/engine.py.230: loss: 0.10444 [I 2022-10-28 04:59:03] ...optimizers/trijoint.py.151: total_norm: 8.913205 [I 2022-10-28 04:59:08] ...trap/engines/engine.py.224: train: epoch 107 | batch 790/2304 [I 2022-10-28 04:59:08] ...trap/engines/engine.py.228: elapsed: 0:07:45 | left: 0:14:08 [I 2022-10-28 04:59:08] ...trap/engines/engine.py.229: process: 0.56907 | load: 0.00247 [I 2022-10-28 04:59:08] ...trap/engines/engine.py.230: loss: 0.09048 [I 2022-10-28 04:59:08] ...optimizers/trijoint.py.151: total_norm: 8.856424 [I 2022-10-28 04:59:14] ...trap/engines/engine.py.224: train: epoch 107 | batch 800/2304 [I 2022-10-28 04:59:14] ...trap/engines/engine.py.228: elapsed: 0:07:50 | left: 0:14:00 [I 2022-10-28 04:59:14] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00249 [I 2022-10-28 04:59:14] ...trap/engines/engine.py.230: loss: 0.14495 [I 2022-10-28 04:59:14] ...optimizers/trijoint.py.151: total_norm: 18.434340 [I 2022-10-28 04:59:20] ...trap/engines/engine.py.224: train: epoch 107 | batch 810/2304 [I 2022-10-28 04:59:20] ...trap/engines/engine.py.228: elapsed: 0:07:56 | left: 0:14:03 [I 2022-10-28 04:59:20] ...trap/engines/engine.py.229: process: 0.55094 | load: 0.00250 [I 2022-10-28 04:59:20] ...trap/engines/engine.py.230: loss: 0.12466 [I 2022-10-28 04:59:20] ...optimizers/trijoint.py.151: total_norm: 16.377573 [I 2022-10-28 04:59:25] ...trap/engines/engine.py.224: train: epoch 107 | batch 820/2304 [I 2022-10-28 04:59:25] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:48 [I 2022-10-28 04:59:25] ...trap/engines/engine.py.229: process: 0.54060 | load: 0.00244 [I 2022-10-28 04:59:25] ...trap/engines/engine.py.230: loss: 0.09763 [I 2022-10-28 04:59:25] ...optimizers/trijoint.py.151: total_norm: 8.612749 [I 2022-10-28 04:59:31] ...trap/engines/engine.py.224: train: epoch 107 | batch 830/2304 [I 2022-10-28 04:59:31] ...trap/engines/engine.py.228: elapsed: 0:08:07 | left: 0:13:51 [I 2022-10-28 04:59:31] ...trap/engines/engine.py.229: process: 0.56542 | load: 0.00256 [I 2022-10-28 04:59:31] ...trap/engines/engine.py.230: loss: 0.11045 [I 2022-10-28 04:59:31] ...optimizers/trijoint.py.151: total_norm: 14.858367 [I 2022-10-28 04:59:37] ...trap/engines/engine.py.224: train: epoch 107 | batch 840/2304 [I 2022-10-28 04:59:37] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:36 [I 2022-10-28 04:59:37] ...trap/engines/engine.py.229: process: 0.55232 | load: 0.00250 [I 2022-10-28 04:59:37] ...trap/engines/engine.py.230: loss: 0.09907 [I 2022-10-28 04:59:37] ...optimizers/trijoint.py.151: total_norm: 11.351792 [I 2022-10-28 04:59:42] ...trap/engines/engine.py.224: train: epoch 107 | batch 850/2304 [I 2022-10-28 04:59:42] ...trap/engines/engine.py.228: elapsed: 0:08:18 | left: 0:13:34 [I 2022-10-28 04:59:42] ...trap/engines/engine.py.229: process: 0.57057 | load: 0.00249 [I 2022-10-28 04:59:42] ...trap/engines/engine.py.230: loss: 0.15538 [I 2022-10-28 04:59:42] ...optimizers/trijoint.py.151: total_norm: 17.074627 [I 2022-10-28 04:59:48] ...trap/engines/engine.py.224: train: epoch 107 | batch 860/2304 [I 2022-10-28 04:59:48] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:13:38 [I 2022-10-28 04:59:48] ...trap/engines/engine.py.229: process: 0.57382 | load: 0.00249 [I 2022-10-28 04:59:48] ...trap/engines/engine.py.230: loss: 0.15917 [I 2022-10-28 04:59:48] ...optimizers/trijoint.py.151: total_norm: 16.771185 [I 2022-10-28 04:59:53] ...trap/engines/engine.py.224: train: epoch 107 | batch 870/2304 [I 2022-10-28 04:59:53] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:13:21 [I 2022-10-28 04:59:53] ...trap/engines/engine.py.229: process: 0.55693 | load: 0.00241 [I 2022-10-28 04:59:53] ...trap/engines/engine.py.230: loss: 0.12004 [I 2022-10-28 04:59:53] ...optimizers/trijoint.py.151: total_norm: 40.786995 [I 2022-10-28 04:59:59] ...trap/engines/engine.py.224: train: epoch 107 | batch 880/2304 [I 2022-10-28 04:59:59] ...trap/engines/engine.py.228: elapsed: 0:08:35 | left: 0:13:17 [I 2022-10-28 04:59:59] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00248 [I 2022-10-28 04:59:59] ...trap/engines/engine.py.230: loss: 0.08746 [I 2022-10-28 04:59:59] ...optimizers/trijoint.py.151: total_norm: 12.142142 [I 2022-10-28 05:00:05] ...trap/engines/engine.py.224: train: epoch 107 | batch 890/2304 [I 2022-10-28 05:00:05] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:13:04 [I 2022-10-28 05:00:05] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00243 [I 2022-10-28 05:00:05] ...trap/engines/engine.py.230: loss: 0.14981 [I 2022-10-28 05:00:05] ...optimizers/trijoint.py.151: total_norm: 16.490711 [I 2022-10-28 05:00:10] ...trap/engines/engine.py.224: train: epoch 107 | batch 900/2304 [I 2022-10-28 05:00:10] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:13:12 [I 2022-10-28 05:00:10] ...trap/engines/engine.py.229: process: 0.56893 | load: 0.00244 [I 2022-10-28 05:00:10] ...trap/engines/engine.py.230: loss: 0.13958 [I 2022-10-28 05:00:10] ...optimizers/trijoint.py.151: total_norm: 15.061183 [I 2022-10-28 05:00:16] ...trap/engines/engine.py.224: train: epoch 107 | batch 910/2304 [I 2022-10-28 05:00:16] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:13:07 [I 2022-10-28 05:00:16] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00244 [I 2022-10-28 05:00:16] ...trap/engines/engine.py.230: loss: 0.11914 [I 2022-10-28 05:00:16] ...optimizers/trijoint.py.151: total_norm: 8.443799 [I 2022-10-28 05:00:22] ...trap/engines/engine.py.224: train: epoch 107 | batch 920/2304 [I 2022-10-28 05:00:22] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:53 [I 2022-10-28 05:00:22] ...trap/engines/engine.py.229: process: 0.56763 | load: 0.00245 [I 2022-10-28 05:00:22] ...trap/engines/engine.py.230: loss: 0.09812 [I 2022-10-28 05:00:22] ...optimizers/trijoint.py.151: total_norm: 14.000587 [I 2022-10-28 05:00:27] ...trap/engines/engine.py.224: train: epoch 107 | batch 930/2304 [I 2022-10-28 05:00:27] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:45 [I 2022-10-28 05:00:27] ...trap/engines/engine.py.229: process: 0.55302 | load: 0.00245 [I 2022-10-28 05:00:27] ...trap/engines/engine.py.230: loss: 0.15440 [I 2022-10-28 05:00:27] ...optimizers/trijoint.py.151: total_norm: 13.669766 [I 2022-10-28 05:00:33] ...trap/engines/engine.py.224: train: epoch 107 | batch 940/2304 [I 2022-10-28 05:00:33] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:52 [I 2022-10-28 05:00:33] ...trap/engines/engine.py.229: process: 0.58805 | load: 0.00245 [I 2022-10-28 05:00:33] ...trap/engines/engine.py.230: loss: 0.10599 [I 2022-10-28 05:00:33] ...optimizers/trijoint.py.151: total_norm: 14.229881 [I 2022-10-28 05:00:38] ...trap/engines/engine.py.224: train: epoch 107 | batch 950/2304 [I 2022-10-28 05:00:38] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:37 [I 2022-10-28 05:00:38] ...trap/engines/engine.py.229: process: 0.55500 | load: 0.00246 [I 2022-10-28 05:00:38] ...trap/engines/engine.py.230: loss: 0.16209 [I 2022-10-28 05:00:38] ...optimizers/trijoint.py.151: total_norm: 42.392193 [I 2022-10-28 05:00:44] ...trap/engines/engine.py.224: train: epoch 107 | batch 960/2304 [I 2022-10-28 05:00:44] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:38 [I 2022-10-28 05:00:44] ...trap/engines/engine.py.229: process: 0.57387 | load: 0.00241 [I 2022-10-28 05:00:44] ...trap/engines/engine.py.230: loss: 0.12654 [I 2022-10-28 05:00:44] ...optimizers/trijoint.py.151: total_norm: 12.791947 [I 2022-10-28 05:00:50] ...trap/engines/engine.py.224: train: epoch 107 | batch 970/2304 [I 2022-10-28 05:00:50] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:12:21 [I 2022-10-28 05:00:50] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00243 [I 2022-10-28 05:00:50] ...trap/engines/engine.py.230: loss: 0.14486 [I 2022-10-28 05:00:50] ...optimizers/trijoint.py.151: total_norm: 26.086596 [I 2022-10-28 05:00:55] ...trap/engines/engine.py.224: train: epoch 107 | batch 980/2304 [I 2022-10-28 05:00:55] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:12:18 [I 2022-10-28 05:00:55] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00245 [I 2022-10-28 05:00:55] ...trap/engines/engine.py.230: loss: 0.09208 [I 2022-10-28 05:00:55] ...optimizers/trijoint.py.151: total_norm: 21.863050 [I 2022-10-28 05:01:01] ...trap/engines/engine.py.224: train: epoch 107 | batch 990/2304 [I 2022-10-28 05:01:01] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:12:13 [I 2022-10-28 05:01:01] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00255 [I 2022-10-28 05:01:01] ...trap/engines/engine.py.230: loss: 0.11635 [I 2022-10-28 05:01:01] ...optimizers/trijoint.py.151: total_norm: 13.665421 [I 2022-10-28 05:01:06] ...trap/engines/engine.py.224: train: epoch 107 | batch 1000/2304 [I 2022-10-28 05:01:06] ...trap/engines/engine.py.228: elapsed: 0:09:43 | left: 0:12:10 [I 2022-10-28 05:01:06] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00253 [I 2022-10-28 05:01:06] ...trap/engines/engine.py.230: loss: 0.14520 [I 2022-10-28 05:01:06] ...optimizers/trijoint.py.151: total_norm: 24.753059 [I 2022-10-28 05:01:12] ...trap/engines/engine.py.224: train: epoch 107 | batch 1010/2304 [I 2022-10-28 05:01:12] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:12:03 [I 2022-10-28 05:01:12] ...trap/engines/engine.py.229: process: 0.54923 | load: 0.00252 [I 2022-10-28 05:01:12] ...trap/engines/engine.py.230: loss: 0.11168 [I 2022-10-28 05:01:12] ...optimizers/trijoint.py.151: total_norm: 11.522463 [I 2022-10-28 05:01:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 1020/2304 [I 2022-10-28 05:01:18] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:12:04 [I 2022-10-28 05:01:18] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00246 [I 2022-10-28 05:01:18] ...trap/engines/engine.py.230: loss: 0.10507 [I 2022-10-28 05:01:18] ...optimizers/trijoint.py.151: total_norm: 22.252605 [I 2022-10-28 05:01:24] ...trap/engines/engine.py.224: train: epoch 107 | batch 1030/2304 [I 2022-10-28 05:01:24] ...trap/engines/engine.py.228: elapsed: 0:10:00 | left: 0:12:09 [I 2022-10-28 05:01:24] ...trap/engines/engine.py.229: process: 0.56394 | load: 0.00250 [I 2022-10-28 05:01:24] ...trap/engines/engine.py.230: loss: 0.15229 [I 2022-10-28 05:01:24] ...optimizers/trijoint.py.151: total_norm: 13.994510 [I 2022-10-28 05:01:29] ...trap/engines/engine.py.224: train: epoch 107 | batch 1040/2304 [I 2022-10-28 05:01:29] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:55 [I 2022-10-28 05:01:29] ...trap/engines/engine.py.229: process: 0.56704 | load: 0.00254 [I 2022-10-28 05:01:29] ...trap/engines/engine.py.230: loss: 0.12913 [I 2022-10-28 05:01:29] ...optimizers/trijoint.py.151: total_norm: 35.939335 [I 2022-10-28 05:01:35] ...trap/engines/engine.py.224: train: epoch 107 | batch 1050/2304 [I 2022-10-28 05:01:35] ...trap/engines/engine.py.228: elapsed: 0:10:11 | left: 0:11:42 [I 2022-10-28 05:01:35] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00250 [I 2022-10-28 05:01:35] ...trap/engines/engine.py.230: loss: 0.16120 [I 2022-10-28 05:01:35] ...optimizers/trijoint.py.151: total_norm: 11.688434 [I 2022-10-28 05:01:41] ...trap/engines/engine.py.224: train: epoch 107 | batch 1060/2304 [I 2022-10-28 05:01:41] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:39 [I 2022-10-28 05:01:41] ...trap/engines/engine.py.229: process: 0.57238 | load: 0.00251 [I 2022-10-28 05:01:41] ...trap/engines/engine.py.230: loss: 0.10555 [I 2022-10-28 05:01:41] ...optimizers/trijoint.py.151: total_norm: 37.883125 [I 2022-10-28 05:01:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 1070/2304 [I 2022-10-28 05:01:46] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:35 [I 2022-10-28 05:01:46] ...trap/engines/engine.py.229: process: 0.56062 | load: 0.00257 [I 2022-10-28 05:01:46] ...trap/engines/engine.py.230: loss: 0.17385 [I 2022-10-28 05:01:46] ...optimizers/trijoint.py.151: total_norm: 21.574917 [I 2022-10-28 05:01:52] ...trap/engines/engine.py.224: train: epoch 107 | batch 1080/2304 [I 2022-10-28 05:01:52] ...trap/engines/engine.py.228: elapsed: 0:10:28 | left: 0:11:28 [I 2022-10-28 05:01:52] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00258 [I 2022-10-28 05:01:52] ...trap/engines/engine.py.230: loss: 0.10927 [I 2022-10-28 05:01:52] ...optimizers/trijoint.py.151: total_norm: 13.265862 [I 2022-10-28 05:01:58] ...trap/engines/engine.py.224: train: epoch 107 | batch 1090/2304 [I 2022-10-28 05:01:58] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:11:29 [I 2022-10-28 05:01:58] ...trap/engines/engine.py.229: process: 0.57193 | load: 0.00256 [I 2022-10-28 05:01:58] ...trap/engines/engine.py.230: loss: 0.09728 [I 2022-10-28 05:01:58] ...optimizers/trijoint.py.151: total_norm: 10.432363 [I 2022-10-28 05:02:03] ...trap/engines/engine.py.224: train: epoch 107 | batch 1100/2304 [I 2022-10-28 05:02:03] ...trap/engines/engine.py.228: elapsed: 0:10:39 | left: 0:11:15 [I 2022-10-28 05:02:03] ...trap/engines/engine.py.229: process: 0.55493 | load: 0.00248 [I 2022-10-28 05:02:03] ...trap/engines/engine.py.230: loss: 0.15031 [I 2022-10-28 05:02:03] ...optimizers/trijoint.py.151: total_norm: 13.544414 [I 2022-10-28 05:02:09] ...trap/engines/engine.py.224: train: epoch 107 | batch 1110/2304 [I 2022-10-28 05:02:09] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:11:08 [I 2022-10-28 05:02:09] ...trap/engines/engine.py.229: process: 0.54987 | load: 0.00249 [I 2022-10-28 05:02:09] ...trap/engines/engine.py.230: loss: 0.11498 [I 2022-10-28 05:02:09] ...optimizers/trijoint.py.151: total_norm: 12.668222 [I 2022-10-28 05:02:14] ...trap/engines/engine.py.224: train: epoch 107 | batch 1120/2304 [I 2022-10-28 05:02:14] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:11:09 [I 2022-10-28 05:02:14] ...trap/engines/engine.py.229: process: 0.57635 | load: 0.00249 [I 2022-10-28 05:02:14] ...trap/engines/engine.py.230: loss: 0.11994 [I 2022-10-28 05:02:14] ...optimizers/trijoint.py.151: total_norm: 30.814417 [I 2022-10-28 05:02:20] ...trap/engines/engine.py.224: train: epoch 107 | batch 1130/2304 [I 2022-10-28 05:02:20] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:56 [I 2022-10-28 05:02:20] ...trap/engines/engine.py.229: process: 0.56339 | load: 0.00247 [I 2022-10-28 05:02:20] ...trap/engines/engine.py.230: loss: 0.17228 [I 2022-10-28 05:02:20] ...optimizers/trijoint.py.151: total_norm: 19.817930 [I 2022-10-28 05:02:26] ...trap/engines/engine.py.224: train: epoch 107 | batch 1140/2304 [I 2022-10-28 05:02:26] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:47 [I 2022-10-28 05:02:26] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00247 [I 2022-10-28 05:02:26] ...trap/engines/engine.py.230: loss: 0.10730 [I 2022-10-28 05:02:26] ...optimizers/trijoint.py.151: total_norm: 11.272072 [I 2022-10-28 05:02:31] ...trap/engines/engine.py.224: train: epoch 107 | batch 1150/2304 [I 2022-10-28 05:02:31] ...trap/engines/engine.py.228: elapsed: 0:11:07 | left: 0:10:40 [I 2022-10-28 05:02:31] ...trap/engines/engine.py.229: process: 0.55202 | load: 0.00242 [I 2022-10-28 05:02:31] ...trap/engines/engine.py.230: loss: 0.16287 [I 2022-10-28 05:02:31] ...optimizers/trijoint.py.151: total_norm: 7.683171 [I 2022-10-28 05:02:37] ...trap/engines/engine.py.224: train: epoch 107 | batch 1160/2304 [I 2022-10-28 05:02:37] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:37 [I 2022-10-28 05:02:37] ...trap/engines/engine.py.229: process: 0.55383 | load: 0.00246 [I 2022-10-28 05:02:37] ...trap/engines/engine.py.230: loss: 0.13685 [I 2022-10-28 05:02:37] ...optimizers/trijoint.py.151: total_norm: 12.042851 [I 2022-10-28 05:02:42] ...trap/engines/engine.py.224: train: epoch 107 | batch 1170/2304 [I 2022-10-28 05:02:42] ...trap/engines/engine.py.228: elapsed: 0:11:18 | left: 0:10:33 [I 2022-10-28 05:02:42] ...trap/engines/engine.py.229: process: 0.55491 | load: 0.00245 [I 2022-10-28 05:02:42] ...trap/engines/engine.py.230: loss: 0.10594 [I 2022-10-28 05:02:42] ...optimizers/trijoint.py.151: total_norm: 18.685421 [I 2022-10-28 05:02:48] ...trap/engines/engine.py.224: train: epoch 107 | batch 1180/2304 [I 2022-10-28 05:02:48] ...trap/engines/engine.py.228: elapsed: 0:11:24 | left: 0:10:31 [I 2022-10-28 05:02:48] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00246 [I 2022-10-28 05:02:48] ...trap/engines/engine.py.230: loss: 0.14670 [I 2022-10-28 05:02:48] ...optimizers/trijoint.py.151: total_norm: 11.354406 [I 2022-10-28 05:02:53] ...trap/engines/engine.py.224: train: epoch 107 | batch 1190/2304 [I 2022-10-28 05:02:53] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:10:20 [I 2022-10-28 05:02:53] ...trap/engines/engine.py.229: process: 0.55941 | load: 0.00246 [I 2022-10-28 05:02:53] ...trap/engines/engine.py.230: loss: 0.11725 [I 2022-10-28 05:02:53] ...optimizers/trijoint.py.151: total_norm: 13.947424 [I 2022-10-28 05:02:59] ...trap/engines/engine.py.224: train: epoch 107 | batch 1200/2304 [I 2022-10-28 05:02:59] ...trap/engines/engine.py.228: elapsed: 0:11:35 | left: 0:10:16 [I 2022-10-28 05:02:59] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00244 [I 2022-10-28 05:02:59] ...trap/engines/engine.py.230: loss: 0.12845 [I 2022-10-28 05:02:59] ...optimizers/trijoint.py.151: total_norm: 17.290577 [I 2022-10-28 05:03:05] ...trap/engines/engine.py.224: train: epoch 107 | batch 1210/2304 [I 2022-10-28 05:03:05] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:10:14 [I 2022-10-28 05:03:05] ...trap/engines/engine.py.229: process: 0.56682 | load: 0.00248 [I 2022-10-28 05:03:05] ...trap/engines/engine.py.230: loss: 0.14097 [I 2022-10-28 05:03:05] ...optimizers/trijoint.py.151: total_norm: 25.337919 [I 2022-10-28 05:03:10] ...trap/engines/engine.py.224: train: epoch 107 | batch 1220/2304 [I 2022-10-28 05:03:10] ...trap/engines/engine.py.228: elapsed: 0:11:46 | left: 0:10:03 [I 2022-10-28 05:03:10] ...trap/engines/engine.py.229: process: 0.53642 | load: 0.00242 [I 2022-10-28 05:03:10] ...trap/engines/engine.py.230: loss: 0.10986 [I 2022-10-28 05:03:10] ...optimizers/trijoint.py.151: total_norm: 19.966768 [I 2022-10-28 05:03:16] ...trap/engines/engine.py.224: train: epoch 107 | batch 1230/2304 [I 2022-10-28 05:03:16] ...trap/engines/engine.py.228: elapsed: 0:11:52 | left: 0:09:59 [I 2022-10-28 05:03:16] ...trap/engines/engine.py.229: process: 0.56292 | load: 0.00242 [I 2022-10-28 05:03:16] ...trap/engines/engine.py.230: loss: 0.19348 [I 2022-10-28 05:03:16] ...optimizers/trijoint.py.151: total_norm: 41.474941 [I 2022-10-28 05:03:21] ...trap/engines/engine.py.224: train: epoch 107 | batch 1240/2304 [I 2022-10-28 05:03:21] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:53 [I 2022-10-28 05:03:21] ...trap/engines/engine.py.229: process: 0.55821 | load: 0.00246 [I 2022-10-28 05:03:21] ...trap/engines/engine.py.230: loss: 0.10968 [I 2022-10-28 05:03:21] ...optimizers/trijoint.py.151: total_norm: 20.340868 [I 2022-10-28 05:03:27] ...trap/engines/engine.py.224: train: epoch 107 | batch 1250/2304 [I 2022-10-28 05:03:27] ...trap/engines/engine.py.228: elapsed: 0:12:03 | left: 0:09:48 [I 2022-10-28 05:03:27] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00261 [I 2022-10-28 05:03:27] ...trap/engines/engine.py.230: loss: 0.13650 [I 2022-10-28 05:03:27] ...optimizers/trijoint.py.151: total_norm: 19.639618 [I 2022-10-28 05:03:33] ...trap/engines/engine.py.224: train: epoch 107 | batch 1260/2304 [I 2022-10-28 05:03:33] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:47 [I 2022-10-28 05:03:33] ...trap/engines/engine.py.229: process: 0.57086 | load: 0.00250 [I 2022-10-28 05:03:33] ...trap/engines/engine.py.230: loss: 0.11857 [I 2022-10-28 05:03:33] ...optimizers/trijoint.py.151: total_norm: 11.540639 [I 2022-10-28 05:03:38] ...trap/engines/engine.py.224: train: epoch 107 | batch 1270/2304 [I 2022-10-28 05:03:38] ...trap/engines/engine.py.228: elapsed: 0:12:14 | left: 0:09:36 [I 2022-10-28 05:03:38] ...trap/engines/engine.py.229: process: 0.55075 | load: 0.00246 [I 2022-10-28 05:03:38] ...trap/engines/engine.py.230: loss: 0.12276 [I 2022-10-28 05:03:38] ...optimizers/trijoint.py.151: total_norm: 10.767575 [I 2022-10-28 05:03:44] ...trap/engines/engine.py.224: train: epoch 107 | batch 1280/2304 [I 2022-10-28 05:03:44] ...trap/engines/engine.py.228: elapsed: 0:12:20 | left: 0:09:32 [I 2022-10-28 05:03:44] ...trap/engines/engine.py.229: process: 0.55158 | load: 0.00249 [I 2022-10-28 05:03:44] ...trap/engines/engine.py.230: loss: 0.15037 [I 2022-10-28 05:03:44] ...optimizers/trijoint.py.151: total_norm: 28.782005 [I 2022-10-28 05:03:50] ...trap/engines/engine.py.224: train: epoch 107 | batch 1290/2304 [I 2022-10-28 05:03:50] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:28 [I 2022-10-28 05:03:50] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00246 [I 2022-10-28 05:03:50] ...trap/engines/engine.py.230: loss: 0.13245 [I 2022-10-28 05:03:50] ...optimizers/trijoint.py.151: total_norm: 17.640522 [I 2022-10-28 05:03:55] ...trap/engines/engine.py.224: train: epoch 107 | batch 1300/2304 [I 2022-10-28 05:03:55] ...trap/engines/engine.py.228: elapsed: 0:12:31 | left: 0:09:20 [I 2022-10-28 05:03:55] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00250 [I 2022-10-28 05:03:55] ...trap/engines/engine.py.230: loss: 0.17122 [I 2022-10-28 05:03:55] ...optimizers/trijoint.py.151: total_norm: 14.772083 [I 2022-10-28 05:04:01] ...trap/engines/engine.py.224: train: epoch 107 | batch 1310/2304 [I 2022-10-28 05:04:01] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:09:14 [I 2022-10-28 05:04:01] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00253 [I 2022-10-28 05:04:01] ...trap/engines/engine.py.230: loss: 0.11255 [I 2022-10-28 05:04:01] ...optimizers/trijoint.py.151: total_norm: 14.244774 [I 2022-10-28 05:04:06] ...trap/engines/engine.py.224: train: epoch 107 | batch 1320/2304 [I 2022-10-28 05:04:06] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:09:11 [I 2022-10-28 05:04:06] ...trap/engines/engine.py.229: process: 0.55927 | load: 0.00250 [I 2022-10-28 05:04:06] ...trap/engines/engine.py.230: loss: 0.14242 [I 2022-10-28 05:04:06] ...optimizers/trijoint.py.151: total_norm: 14.596272 [I 2022-10-28 05:04:12] ...trap/engines/engine.py.224: train: epoch 107 | batch 1330/2304 [I 2022-10-28 05:04:12] ...trap/engines/engine.py.228: elapsed: 0:12:48 | left: 0:09:07 [I 2022-10-28 05:04:12] ...trap/engines/engine.py.229: process: 0.56906 | load: 0.00276 [I 2022-10-28 05:04:12] ...trap/engines/engine.py.230: loss: 0.13384 [I 2022-10-28 05:04:12] ...optimizers/trijoint.py.151: total_norm: 14.668824 [I 2022-10-28 05:04:18] ...trap/engines/engine.py.224: train: epoch 107 | batch 1340/2304 [I 2022-10-28 05:04:18] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:09:00 [I 2022-10-28 05:04:18] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00262 [I 2022-10-28 05:04:18] ...trap/engines/engine.py.230: loss: 0.19468 [I 2022-10-28 05:04:18] ...optimizers/trijoint.py.151: total_norm: 20.168474 [I 2022-10-28 05:04:23] ...trap/engines/engine.py.224: train: epoch 107 | batch 1350/2304 [I 2022-10-28 05:04:23] ...trap/engines/engine.py.228: elapsed: 0:12:59 | left: 0:08:53 [I 2022-10-28 05:04:23] ...trap/engines/engine.py.229: process: 0.56502 | load: 0.00259 [I 2022-10-28 05:04:23] ...trap/engines/engine.py.230: loss: 0.12522 [I 2022-10-28 05:04:23] ...optimizers/trijoint.py.151: total_norm: 26.636684 [I 2022-10-28 05:04:29] ...trap/engines/engine.py.224: train: epoch 107 | batch 1360/2304 [I 2022-10-28 05:04:29] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:45 [I 2022-10-28 05:04:29] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00262 [I 2022-10-28 05:04:29] ...trap/engines/engine.py.230: loss: 0.14693 [I 2022-10-28 05:04:29] ...optimizers/trijoint.py.151: total_norm: 14.951555 [I 2022-10-28 05:04:34] ...trap/engines/engine.py.224: train: epoch 107 | batch 1370/2304 [I 2022-10-28 05:04:34] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:40 [I 2022-10-28 05:04:34] ...trap/engines/engine.py.229: process: 0.55745 | load: 0.00248 [I 2022-10-28 05:04:34] ...trap/engines/engine.py.230: loss: 0.11876 [I 2022-10-28 05:04:34] ...optimizers/trijoint.py.151: total_norm: 11.463010 [I 2022-10-28 05:04:40] ...trap/engines/engine.py.224: train: epoch 107 | batch 1380/2304 [I 2022-10-28 05:04:40] ...trap/engines/engine.py.228: elapsed: 0:13:16 | left: 0:08:34 [I 2022-10-28 05:04:40] ...trap/engines/engine.py.229: process: 0.54569 | load: 0.00248 [I 2022-10-28 05:04:40] ...trap/engines/engine.py.230: loss: 0.11421 [I 2022-10-28 05:04:40] ...optimizers/trijoint.py.151: total_norm: 15.022373 [I 2022-10-28 05:04:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 1390/2304 [I 2022-10-28 05:04:46] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:26 [I 2022-10-28 05:04:46] ...trap/engines/engine.py.229: process: 0.54965 | load: 0.00251 [I 2022-10-28 05:04:46] ...trap/engines/engine.py.230: loss: 0.14386 [I 2022-10-28 05:04:46] ...optimizers/trijoint.py.151: total_norm: 24.708973 [I 2022-10-28 05:04:51] ...trap/engines/engine.py.224: train: epoch 107 | batch 1400/2304 [I 2022-10-28 05:04:51] ...trap/engines/engine.py.228: elapsed: 0:13:27 | left: 0:08:23 [I 2022-10-28 05:04:51] ...trap/engines/engine.py.229: process: 0.56309 | load: 0.00245 [I 2022-10-28 05:04:51] ...trap/engines/engine.py.230: loss: 0.15145 [I 2022-10-28 05:04:51] ...optimizers/trijoint.py.151: total_norm: 18.926407 [I 2022-10-28 05:04:57] ...trap/engines/engine.py.224: train: epoch 107 | batch 1410/2304 [I 2022-10-28 05:04:57] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:08:16 [I 2022-10-28 05:04:57] ...trap/engines/engine.py.229: process: 0.54747 | load: 0.00247 [I 2022-10-28 05:04:57] ...trap/engines/engine.py.230: loss: 0.10232 [I 2022-10-28 05:04:57] ...optimizers/trijoint.py.151: total_norm: 24.031080 [I 2022-10-28 05:05:02] ...trap/engines/engine.py.224: train: epoch 107 | batch 1420/2304 [I 2022-10-28 05:05:02] ...trap/engines/engine.py.228: elapsed: 0:13:38 | left: 0:08:17 [I 2022-10-28 05:05:02] ...trap/engines/engine.py.229: process: 0.57753 | load: 0.00248 [I 2022-10-28 05:05:02] ...trap/engines/engine.py.230: loss: 0.18332 [I 2022-10-28 05:05:02] ...optimizers/trijoint.py.151: total_norm: 13.109715 [I 2022-10-28 05:05:08] ...trap/engines/engine.py.224: train: epoch 107 | batch 1430/2304 [I 2022-10-28 05:05:08] ...trap/engines/engine.py.228: elapsed: 0:13:44 | left: 0:08:07 [I 2022-10-28 05:05:08] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00246 [I 2022-10-28 05:05:08] ...trap/engines/engine.py.230: loss: 0.10159 [I 2022-10-28 05:05:08] ...optimizers/trijoint.py.151: total_norm: 13.748747 [I 2022-10-28 05:05:13] ...trap/engines/engine.py.224: train: epoch 107 | batch 1440/2304 [I 2022-10-28 05:05:13] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:08:00 [I 2022-10-28 05:05:13] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00246 [I 2022-10-28 05:05:13] ...trap/engines/engine.py.230: loss: 0.10136 [I 2022-10-28 05:05:13] ...optimizers/trijoint.py.151: total_norm: 12.822064 [I 2022-10-28 05:05:19] ...trap/engines/engine.py.224: train: epoch 107 | batch 1450/2304 [I 2022-10-28 05:05:19] ...trap/engines/engine.py.228: elapsed: 0:13:55 | left: 0:07:56 [I 2022-10-28 05:05:19] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00244 [I 2022-10-28 05:05:19] ...trap/engines/engine.py.230: loss: 0.09036 [I 2022-10-28 05:05:19] ...optimizers/trijoint.py.151: total_norm: 7.850003 [I 2022-10-28 05:05:25] ...trap/engines/engine.py.224: train: epoch 107 | batch 1460/2304 [I 2022-10-28 05:05:25] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:51 [I 2022-10-28 05:05:25] ...trap/engines/engine.py.229: process: 0.55252 | load: 0.00242 [I 2022-10-28 05:05:25] ...trap/engines/engine.py.230: loss: 0.10784 [I 2022-10-28 05:05:25] ...optimizers/trijoint.py.151: total_norm: 10.145457 [I 2022-10-28 05:05:30] ...trap/engines/engine.py.224: train: epoch 107 | batch 1470/2304 [I 2022-10-28 05:05:30] ...trap/engines/engine.py.228: elapsed: 0:14:06 | left: 0:07:44 [I 2022-10-28 05:05:30] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00269 [I 2022-10-28 05:05:30] ...trap/engines/engine.py.230: loss: 0.10027 [I 2022-10-28 05:05:30] ...optimizers/trijoint.py.151: total_norm: 11.070090 [I 2022-10-28 05:05:36] ...trap/engines/engine.py.224: train: epoch 107 | batch 1480/2304 [I 2022-10-28 05:05:36] ...trap/engines/engine.py.228: elapsed: 0:14:12 | left: 0:07:40 [I 2022-10-28 05:05:36] ...trap/engines/engine.py.229: process: 0.54899 | load: 0.00260 [I 2022-10-28 05:05:36] ...trap/engines/engine.py.230: loss: 0.18506 [I 2022-10-28 05:05:36] ...optimizers/trijoint.py.151: total_norm: 30.882189 [I 2022-10-28 05:05:41] ...trap/engines/engine.py.224: train: epoch 107 | batch 1490/2304 [I 2022-10-28 05:05:41] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:36 [I 2022-10-28 05:05:41] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00249 [I 2022-10-28 05:05:41] ...trap/engines/engine.py.230: loss: 0.10947 [I 2022-10-28 05:05:41] ...optimizers/trijoint.py.151: total_norm: 9.949538 [I 2022-10-28 05:05:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 1500/2304 [I 2022-10-28 05:05:47] ...trap/engines/engine.py.228: elapsed: 0:14:23 | left: 0:07:28 [I 2022-10-28 05:05:47] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00247 [I 2022-10-28 05:05:47] ...trap/engines/engine.py.230: loss: 0.11523 [I 2022-10-28 05:05:47] ...optimizers/trijoint.py.151: total_norm: 20.698336 [I 2022-10-28 05:05:53] ...trap/engines/engine.py.224: train: epoch 107 | batch 1510/2304 [I 2022-10-28 05:05:53] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:25 [I 2022-10-28 05:05:53] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00245 [I 2022-10-28 05:05:53] ...trap/engines/engine.py.230: loss: 0.13318 [I 2022-10-28 05:05:53] ...optimizers/trijoint.py.151: total_norm: 23.182316 [I 2022-10-28 05:05:58] ...trap/engines/engine.py.224: train: epoch 107 | batch 1520/2304 [I 2022-10-28 05:05:58] ...trap/engines/engine.py.228: elapsed: 0:14:34 | left: 0:07:19 [I 2022-10-28 05:05:58] ...trap/engines/engine.py.229: process: 0.56769 | load: 0.00251 [I 2022-10-28 05:05:58] ...trap/engines/engine.py.230: loss: 0.12688 [I 2022-10-28 05:05:58] ...optimizers/trijoint.py.151: total_norm: 24.390600 [I 2022-10-28 05:06:04] ...trap/engines/engine.py.224: train: epoch 107 | batch 1530/2304 [I 2022-10-28 05:06:04] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:07:09 [I 2022-10-28 05:06:04] ...trap/engines/engine.py.229: process: 0.55751 | load: 0.00254 [I 2022-10-28 05:06:04] ...trap/engines/engine.py.230: loss: 0.14756 [I 2022-10-28 05:06:04] ...optimizers/trijoint.py.151: total_norm: 15.902105 [I 2022-10-28 05:06:09] ...trap/engines/engine.py.224: train: epoch 107 | batch 1540/2304 [I 2022-10-28 05:06:09] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:07:06 [I 2022-10-28 05:06:09] ...trap/engines/engine.py.229: process: 0.55667 | load: 0.00251 [I 2022-10-28 05:06:09] ...trap/engines/engine.py.230: loss: 0.10206 [I 2022-10-28 05:06:09] ...optimizers/trijoint.py.151: total_norm: 11.419909 [I 2022-10-28 05:06:15] ...trap/engines/engine.py.224: train: epoch 107 | batch 1550/2304 [I 2022-10-28 05:06:15] ...trap/engines/engine.py.228: elapsed: 0:14:51 | left: 0:07:03 [I 2022-10-28 05:06:15] ...trap/engines/engine.py.229: process: 0.55696 | load: 0.00280 [I 2022-10-28 05:06:15] ...trap/engines/engine.py.230: loss: 0.12332 [I 2022-10-28 05:06:15] ...optimizers/trijoint.py.151: total_norm: 14.390002 [I 2022-10-28 05:06:21] ...trap/engines/engine.py.224: train: epoch 107 | batch 1560/2304 [I 2022-10-28 05:06:21] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:58 [I 2022-10-28 05:06:21] ...trap/engines/engine.py.229: process: 0.55857 | load: 0.00277 [I 2022-10-28 05:06:21] ...trap/engines/engine.py.230: loss: 0.14014 [I 2022-10-28 05:06:21] ...optimizers/trijoint.py.151: total_norm: 15.457577 [I 2022-10-28 05:06:26] ...trap/engines/engine.py.224: train: epoch 107 | batch 1570/2304 [I 2022-10-28 05:06:26] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:49 [I 2022-10-28 05:06:26] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00256 [I 2022-10-28 05:06:26] ...trap/engines/engine.py.230: loss: 0.10998 [I 2022-10-28 05:06:26] ...optimizers/trijoint.py.151: total_norm: 13.747628 [I 2022-10-28 05:06:32] ...trap/engines/engine.py.224: train: epoch 107 | batch 1580/2304 [I 2022-10-28 05:06:32] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:42 [I 2022-10-28 05:06:32] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00256 [I 2022-10-28 05:06:32] ...trap/engines/engine.py.230: loss: 0.14439 [I 2022-10-28 05:06:32] ...optimizers/trijoint.py.151: total_norm: 27.172916 [I 2022-10-28 05:06:38] ...trap/engines/engine.py.224: train: epoch 107 | batch 1590/2304 [I 2022-10-28 05:06:38] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:38 [I 2022-10-28 05:06:38] ...trap/engines/engine.py.229: process: 0.55573 | load: 0.00250 [I 2022-10-28 05:06:38] ...trap/engines/engine.py.230: loss: 0.09900 [I 2022-10-28 05:06:38] ...optimizers/trijoint.py.151: total_norm: 61.842155 [I 2022-10-28 05:06:43] ...trap/engines/engine.py.224: train: epoch 107 | batch 1600/2304 [I 2022-10-28 05:06:43] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:34 [I 2022-10-28 05:06:43] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00245 [I 2022-10-28 05:06:43] ...trap/engines/engine.py.230: loss: 0.12435 [I 2022-10-28 05:06:43] ...optimizers/trijoint.py.151: total_norm: 10.830856 [I 2022-10-28 05:06:49] ...trap/engines/engine.py.224: train: epoch 107 | batch 1610/2304 [I 2022-10-28 05:06:49] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:42 [I 2022-10-28 05:06:49] ...trap/engines/engine.py.229: process: 0.59764 | load: 0.00260 [I 2022-10-28 05:06:49] ...trap/engines/engine.py.230: loss: 0.11829 [I 2022-10-28 05:06:49] ...optimizers/trijoint.py.151: total_norm: 14.461699 [I 2022-10-28 05:06:54] ...trap/engines/engine.py.224: train: epoch 107 | batch 1620/2304 [I 2022-10-28 05:06:54] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:22 [I 2022-10-28 05:06:54] ...trap/engines/engine.py.229: process: 0.56641 | load: 0.00248 [I 2022-10-28 05:06:54] ...trap/engines/engine.py.230: loss: 0.09520 [I 2022-10-28 05:06:54] ...optimizers/trijoint.py.151: total_norm: 15.669706 [I 2022-10-28 05:07:00] ...trap/engines/engine.py.224: train: epoch 107 | batch 1630/2304 [I 2022-10-28 05:07:00] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:23 [I 2022-10-28 05:07:00] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00254 [I 2022-10-28 05:07:00] ...trap/engines/engine.py.230: loss: 0.15404 [I 2022-10-28 05:07:00] ...optimizers/trijoint.py.151: total_norm: 12.570458 [I 2022-10-28 05:07:06] ...trap/engines/engine.py.224: train: epoch 107 | batch 1640/2304 [I 2022-10-28 05:07:06] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:06:11 [I 2022-10-28 05:07:06] ...trap/engines/engine.py.229: process: 0.55013 | load: 0.00252 [I 2022-10-28 05:07:06] ...trap/engines/engine.py.230: loss: 0.06817 [I 2022-10-28 05:07:06] ...optimizers/trijoint.py.151: total_norm: 13.393915 [I 2022-10-28 05:07:11] ...trap/engines/engine.py.224: train: epoch 107 | batch 1650/2304 [I 2022-10-28 05:07:11] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:06:04 [I 2022-10-28 05:07:11] ...trap/engines/engine.py.229: process: 0.56481 | load: 0.00251 [I 2022-10-28 05:07:11] ...trap/engines/engine.py.230: loss: 0.12859 [I 2022-10-28 05:07:11] ...optimizers/trijoint.py.151: total_norm: 27.860640 [I 2022-10-28 05:07:17] ...trap/engines/engine.py.224: train: epoch 107 | batch 1660/2304 [I 2022-10-28 05:07:17] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:06:03 [I 2022-10-28 05:07:17] ...trap/engines/engine.py.229: process: 0.57015 | load: 0.00246 [I 2022-10-28 05:07:17] ...trap/engines/engine.py.230: loss: 0.18420 [I 2022-10-28 05:07:17] ...optimizers/trijoint.py.151: total_norm: 28.685099 [I 2022-10-28 05:07:23] ...trap/engines/engine.py.224: train: epoch 107 | batch 1670/2304 [I 2022-10-28 05:07:23] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:51 [I 2022-10-28 05:07:23] ...trap/engines/engine.py.229: process: 0.55665 | load: 0.00258 [I 2022-10-28 05:07:23] ...trap/engines/engine.py.230: loss: 0.16814 [I 2022-10-28 05:07:23] ...optimizers/trijoint.py.151: total_norm: 17.549858 [I 2022-10-28 05:07:28] ...trap/engines/engine.py.224: train: epoch 107 | batch 1680/2304 [I 2022-10-28 05:07:28] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:49 [I 2022-10-28 05:07:28] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00254 [I 2022-10-28 05:07:28] ...trap/engines/engine.py.230: loss: 0.12639 [I 2022-10-28 05:07:28] ...optimizers/trijoint.py.151: total_norm: 20.568529 [I 2022-10-28 05:07:34] ...trap/engines/engine.py.224: train: epoch 107 | batch 1690/2304 [I 2022-10-28 05:07:34] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:44 [I 2022-10-28 05:07:34] ...trap/engines/engine.py.229: process: 0.56655 | load: 0.00249 [I 2022-10-28 05:07:34] ...trap/engines/engine.py.230: loss: 0.13889 [I 2022-10-28 05:07:34] ...optimizers/trijoint.py.151: total_norm: 15.686231 [I 2022-10-28 05:07:39] ...trap/engines/engine.py.224: train: epoch 107 | batch 1700/2304 [I 2022-10-28 05:07:39] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:39 [I 2022-10-28 05:07:39] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00256 [I 2022-10-28 05:07:39] ...trap/engines/engine.py.230: loss: 0.08924 [I 2022-10-28 05:07:39] ...optimizers/trijoint.py.151: total_norm: 9.700688 [I 2022-10-28 05:07:45] ...trap/engines/engine.py.224: train: epoch 107 | batch 1710/2304 [I 2022-10-28 05:07:45] ...trap/engines/engine.py.228: elapsed: 0:16:21 | left: 0:05:33 [I 2022-10-28 05:07:45] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00249 [I 2022-10-28 05:07:45] ...trap/engines/engine.py.230: loss: 0.13841 [I 2022-10-28 05:07:45] ...optimizers/trijoint.py.151: total_norm: 10.859908 [I 2022-10-28 05:07:51] ...trap/engines/engine.py.224: train: epoch 107 | batch 1720/2304 [I 2022-10-28 05:07:51] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:29 [I 2022-10-28 05:07:51] ...trap/engines/engine.py.229: process: 0.56986 | load: 0.00247 [I 2022-10-28 05:07:51] ...trap/engines/engine.py.230: loss: 0.11250 [I 2022-10-28 05:07:51] ...optimizers/trijoint.py.151: total_norm: 14.168340 [I 2022-10-28 05:07:56] ...trap/engines/engine.py.224: train: epoch 107 | batch 1730/2304 [I 2022-10-28 05:07:56] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:05:22 [I 2022-10-28 05:07:56] ...trap/engines/engine.py.229: process: 0.56396 | load: 0.00247 [I 2022-10-28 05:07:56] ...trap/engines/engine.py.230: loss: 0.16696 [I 2022-10-28 05:07:56] ...optimizers/trijoint.py.151: total_norm: 26.554953 [I 2022-10-28 05:08:02] ...trap/engines/engine.py.224: train: epoch 107 | batch 1740/2304 [I 2022-10-28 05:08:02] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:05:19 [I 2022-10-28 05:08:02] ...trap/engines/engine.py.229: process: 0.58902 | load: 0.00252 [I 2022-10-28 05:08:02] ...trap/engines/engine.py.230: loss: 0.14054 [I 2022-10-28 05:08:02] ...optimizers/trijoint.py.151: total_norm: 9.973183 [I 2022-10-28 05:08:08] ...trap/engines/engine.py.224: train: epoch 107 | batch 1750/2304 [I 2022-10-28 05:08:08] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:05:11 [I 2022-10-28 05:08:08] ...trap/engines/engine.py.229: process: 0.56486 | load: 0.00252 [I 2022-10-28 05:08:08] ...trap/engines/engine.py.230: loss: 0.12668 [I 2022-10-28 05:08:08] ...optimizers/trijoint.py.151: total_norm: 10.507384 [I 2022-10-28 05:08:13] ...trap/engines/engine.py.224: train: epoch 107 | batch 1760/2304 [I 2022-10-28 05:08:13] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:05:07 [I 2022-10-28 05:08:13] ...trap/engines/engine.py.229: process: 0.56866 | load: 0.00249 [I 2022-10-28 05:08:13] ...trap/engines/engine.py.230: loss: 0.12595 [I 2022-10-28 05:08:13] ...optimizers/trijoint.py.151: total_norm: 24.691986 [I 2022-10-28 05:08:19] ...trap/engines/engine.py.224: train: epoch 107 | batch 1770/2304 [I 2022-10-28 05:08:19] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:05:01 [I 2022-10-28 05:08:19] ...trap/engines/engine.py.229: process: 0.55547 | load: 0.00252 [I 2022-10-28 05:08:19] ...trap/engines/engine.py.230: loss: 0.07874 [I 2022-10-28 05:08:19] ...optimizers/trijoint.py.151: total_norm: 15.820893 [I 2022-10-28 05:08:25] ...trap/engines/engine.py.224: train: epoch 107 | batch 1780/2304 [I 2022-10-28 05:08:25] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:53 [I 2022-10-28 05:08:25] ...trap/engines/engine.py.229: process: 0.54615 | load: 0.00253 [I 2022-10-28 05:08:25] ...trap/engines/engine.py.230: loss: 0.14512 [I 2022-10-28 05:08:25] ...optimizers/trijoint.py.151: total_norm: 11.890667 [I 2022-10-28 05:08:30] ...trap/engines/engine.py.224: train: epoch 107 | batch 1790/2304 [I 2022-10-28 05:08:30] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:51 [I 2022-10-28 05:08:30] ...trap/engines/engine.py.229: process: 0.57791 | load: 0.00249 [I 2022-10-28 05:08:30] ...trap/engines/engine.py.230: loss: 0.08425 [I 2022-10-28 05:08:30] ...optimizers/trijoint.py.151: total_norm: 24.382641 [I 2022-10-28 05:08:36] ...trap/engines/engine.py.224: train: epoch 107 | batch 1800/2304 [I 2022-10-28 05:08:36] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:46 [I 2022-10-28 05:08:36] ...trap/engines/engine.py.229: process: 0.58913 | load: 0.00243 [I 2022-10-28 05:08:36] ...trap/engines/engine.py.230: loss: 0.14122 [I 2022-10-28 05:08:36] ...optimizers/trijoint.py.151: total_norm: 16.677322 [I 2022-10-28 05:08:42] ...trap/engines/engine.py.224: train: epoch 107 | batch 1810/2304 [I 2022-10-28 05:08:42] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:34 [I 2022-10-28 05:08:42] ...trap/engines/engine.py.229: process: 0.55175 | load: 0.00260 [I 2022-10-28 05:08:42] ...trap/engines/engine.py.230: loss: 0.15288 [I 2022-10-28 05:08:42] ...optimizers/trijoint.py.151: total_norm: 17.245049 [I 2022-10-28 05:08:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 1820/2304 [I 2022-10-28 05:08:47] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:29 [I 2022-10-28 05:08:47] ...trap/engines/engine.py.229: process: 0.55124 | load: 0.00243 [I 2022-10-28 05:08:47] ...trap/engines/engine.py.230: loss: 0.13612 [I 2022-10-28 05:08:47] ...optimizers/trijoint.py.151: total_norm: 11.123223 [I 2022-10-28 05:08:53] ...trap/engines/engine.py.224: train: epoch 107 | batch 1830/2304 [I 2022-10-28 05:08:53] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:24 [I 2022-10-28 05:08:53] ...trap/engines/engine.py.229: process: 0.55012 | load: 0.00242 [I 2022-10-28 05:08:53] ...trap/engines/engine.py.230: loss: 0.06490 [I 2022-10-28 05:08:53] ...optimizers/trijoint.py.151: total_norm: 9.220396 [I 2022-10-28 05:08:58] ...trap/engines/engine.py.224: train: epoch 107 | batch 1840/2304 [I 2022-10-28 05:08:58] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:19 [I 2022-10-28 05:08:58] ...trap/engines/engine.py.229: process: 0.57009 | load: 0.00244 [I 2022-10-28 05:08:58] ...trap/engines/engine.py.230: loss: 0.11835 [I 2022-10-28 05:08:58] ...optimizers/trijoint.py.151: total_norm: 16.903177 [I 2022-10-28 05:09:04] ...trap/engines/engine.py.224: train: epoch 107 | batch 1850/2304 [I 2022-10-28 05:09:04] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:04:13 [I 2022-10-28 05:09:04] ...trap/engines/engine.py.229: process: 0.54346 | load: 0.00246 [I 2022-10-28 05:09:04] ...trap/engines/engine.py.230: loss: 0.11691 [I 2022-10-28 05:09:04] ...optimizers/trijoint.py.151: total_norm: 13.260955 [I 2022-10-28 05:09:10] ...trap/engines/engine.py.224: train: epoch 107 | batch 1860/2304 [I 2022-10-28 05:09:10] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:04:10 [I 2022-10-28 05:09:10] ...trap/engines/engine.py.229: process: 0.56641 | load: 0.00244 [I 2022-10-28 05:09:10] ...trap/engines/engine.py.230: loss: 0.12902 [I 2022-10-28 05:09:10] ...optimizers/trijoint.py.151: total_norm: 9.180802 [I 2022-10-28 05:09:15] ...trap/engines/engine.py.224: train: epoch 107 | batch 1870/2304 [I 2022-10-28 05:09:15] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:04:02 [I 2022-10-28 05:09:15] ...trap/engines/engine.py.229: process: 0.55868 | load: 0.00243 [I 2022-10-28 05:09:15] ...trap/engines/engine.py.230: loss: 0.13401 [I 2022-10-28 05:09:15] ...optimizers/trijoint.py.151: total_norm: 10.759007 [I 2022-10-28 05:09:21] ...trap/engines/engine.py.224: train: epoch 107 | batch 1880/2304 [I 2022-10-28 05:09:21] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:59 [I 2022-10-28 05:09:21] ...trap/engines/engine.py.229: process: 0.56289 | load: 0.00267 [I 2022-10-28 05:09:21] ...trap/engines/engine.py.230: loss: 0.12534 [I 2022-10-28 05:09:21] ...optimizers/trijoint.py.151: total_norm: 14.835903 [I 2022-10-28 05:09:27] ...trap/engines/engine.py.224: train: epoch 107 | batch 1890/2304 [I 2022-10-28 05:09:27] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:51 [I 2022-10-28 05:09:27] ...trap/engines/engine.py.229: process: 0.54866 | load: 0.00257 [I 2022-10-28 05:09:27] ...trap/engines/engine.py.230: loss: 0.12891 [I 2022-10-28 05:09:27] ...optimizers/trijoint.py.151: total_norm: 20.033026 [I 2022-10-28 05:09:32] ...trap/engines/engine.py.224: train: epoch 107 | batch 1900/2304 [I 2022-10-28 05:09:32] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:43 [I 2022-10-28 05:09:32] ...trap/engines/engine.py.229: process: 0.54795 | load: 0.00248 [I 2022-10-28 05:09:32] ...trap/engines/engine.py.230: loss: 0.12400 [I 2022-10-28 05:09:32] ...optimizers/trijoint.py.151: total_norm: 16.184879 [I 2022-10-28 05:09:38] ...trap/engines/engine.py.224: train: epoch 107 | batch 1910/2304 [I 2022-10-28 05:09:38] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:40 [I 2022-10-28 05:09:38] ...trap/engines/engine.py.229: process: 0.57136 | load: 0.00245 [I 2022-10-28 05:09:38] ...trap/engines/engine.py.230: loss: 0.10454 [I 2022-10-28 05:09:38] ...optimizers/trijoint.py.151: total_norm: 17.957226 [I 2022-10-28 05:09:43] ...trap/engines/engine.py.224: train: epoch 107 | batch 1920/2304 [I 2022-10-28 05:09:43] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:37 [I 2022-10-28 05:09:43] ...trap/engines/engine.py.229: process: 0.57920 | load: 0.00260 [I 2022-10-28 05:09:43] ...trap/engines/engine.py.230: loss: 0.09505 [I 2022-10-28 05:09:43] ...optimizers/trijoint.py.151: total_norm: 21.926905 [I 2022-10-28 05:09:49] ...trap/engines/engine.py.224: train: epoch 107 | batch 1930/2304 [I 2022-10-28 05:09:49] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:32 [I 2022-10-28 05:09:49] ...trap/engines/engine.py.229: process: 0.55707 | load: 0.00263 [I 2022-10-28 05:09:49] ...trap/engines/engine.py.230: loss: 0.16007 [I 2022-10-28 05:09:49] ...optimizers/trijoint.py.151: total_norm: 16.761917 [I 2022-10-28 05:09:55] ...trap/engines/engine.py.224: train: epoch 107 | batch 1940/2304 [I 2022-10-28 05:09:55] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:23 [I 2022-10-28 05:09:55] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00252 [I 2022-10-28 05:09:55] ...trap/engines/engine.py.230: loss: 0.17669 [I 2022-10-28 05:09:55] ...optimizers/trijoint.py.151: total_norm: 9.735826 [I 2022-10-28 05:10:00] ...trap/engines/engine.py.224: train: epoch 107 | batch 1950/2304 [I 2022-10-28 05:10:00] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:03:16 [I 2022-10-28 05:10:00] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00250 [I 2022-10-28 05:10:00] ...trap/engines/engine.py.230: loss: 0.11153 [I 2022-10-28 05:10:00] ...optimizers/trijoint.py.151: total_norm: 11.338363 [I 2022-10-28 05:10:06] ...trap/engines/engine.py.224: train: epoch 107 | batch 1960/2304 [I 2022-10-28 05:10:06] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:03:13 [I 2022-10-28 05:10:06] ...trap/engines/engine.py.229: process: 0.54985 | load: 0.00269 [I 2022-10-28 05:10:06] ...trap/engines/engine.py.230: loss: 0.09111 [I 2022-10-28 05:10:06] ...optimizers/trijoint.py.151: total_norm: 12.248123 [I 2022-10-28 05:10:12] ...trap/engines/engine.py.224: train: epoch 107 | batch 1970/2304 [I 2022-10-28 05:10:12] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:03:05 [I 2022-10-28 05:10:12] ...trap/engines/engine.py.229: process: 0.54956 | load: 0.00253 [I 2022-10-28 05:10:12] ...trap/engines/engine.py.230: loss: 0.14034 [I 2022-10-28 05:10:12] ...optimizers/trijoint.py.151: total_norm: 23.714249 [I 2022-10-28 05:10:17] ...trap/engines/engine.py.224: train: epoch 107 | batch 1980/2304 [I 2022-10-28 05:10:17] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:03:08 [I 2022-10-28 05:10:17] ...trap/engines/engine.py.229: process: 0.55149 | load: 0.00263 [I 2022-10-28 05:10:17] ...trap/engines/engine.py.230: loss: 0.09039 [I 2022-10-28 05:10:17] ...optimizers/trijoint.py.151: total_norm: 28.458561 [I 2022-10-28 05:10:23] ...trap/engines/engine.py.224: train: epoch 107 | batch 1990/2304 [I 2022-10-28 05:10:23] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:56 [I 2022-10-28 05:10:23] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00257 [I 2022-10-28 05:10:23] ...trap/engines/engine.py.230: loss: 0.11256 [I 2022-10-28 05:10:23] ...optimizers/trijoint.py.151: total_norm: 12.108976 [I 2022-10-28 05:10:29] ...trap/engines/engine.py.224: train: epoch 107 | batch 2000/2304 [I 2022-10-28 05:10:29] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:50 [I 2022-10-28 05:10:29] ...trap/engines/engine.py.229: process: 0.56328 | load: 0.00260 [I 2022-10-28 05:10:29] ...trap/engines/engine.py.230: loss: 0.10321 [I 2022-10-28 05:10:29] ...optimizers/trijoint.py.151: total_norm: 10.373806 [I 2022-10-28 05:10:34] ...trap/engines/engine.py.224: train: epoch 107 | batch 2010/2304 [I 2022-10-28 05:10:34] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:46 [I 2022-10-28 05:10:34] ...trap/engines/engine.py.229: process: 0.56087 | load: 0.00283 [I 2022-10-28 05:10:34] ...trap/engines/engine.py.230: loss: 0.13307 [I 2022-10-28 05:10:34] ...optimizers/trijoint.py.151: total_norm: 15.136017 [I 2022-10-28 05:10:40] ...trap/engines/engine.py.224: train: epoch 107 | batch 2020/2304 [I 2022-10-28 05:10:40] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:39 [I 2022-10-28 05:10:40] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00252 [I 2022-10-28 05:10:40] ...trap/engines/engine.py.230: loss: 0.10495 [I 2022-10-28 05:10:40] ...optimizers/trijoint.py.151: total_norm: 9.335441 [I 2022-10-28 05:10:46] ...trap/engines/engine.py.224: train: epoch 107 | batch 2030/2304 [I 2022-10-28 05:10:46] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:34 [I 2022-10-28 05:10:46] ...trap/engines/engine.py.229: process: 0.56456 | load: 0.00247 [I 2022-10-28 05:10:46] ...trap/engines/engine.py.230: loss: 0.11779 [I 2022-10-28 05:10:46] ...optimizers/trijoint.py.151: total_norm: 21.091406 [I 2022-10-28 05:10:51] ...trap/engines/engine.py.224: train: epoch 107 | batch 2040/2304 [I 2022-10-28 05:10:51] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:28 [I 2022-10-28 05:10:51] ...trap/engines/engine.py.229: process: 0.56399 | load: 0.00248 [I 2022-10-28 05:10:51] ...trap/engines/engine.py.230: loss: 0.18268 [I 2022-10-28 05:10:51] ...optimizers/trijoint.py.151: total_norm: 31.940477 [I 2022-10-28 05:10:57] ...trap/engines/engine.py.224: train: epoch 107 | batch 2050/2304 [I 2022-10-28 05:10:57] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:21 [I 2022-10-28 05:10:57] ...trap/engines/engine.py.229: process: 0.55773 | load: 0.00248 [I 2022-10-28 05:10:57] ...trap/engines/engine.py.230: loss: 0.14129 [I 2022-10-28 05:10:57] ...optimizers/trijoint.py.151: total_norm: 18.734467 [I 2022-10-28 05:11:02] ...trap/engines/engine.py.224: train: epoch 107 | batch 2060/2304 [I 2022-10-28 05:11:02] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:16 [I 2022-10-28 05:11:02] ...trap/engines/engine.py.229: process: 0.56778 | load: 0.00250 [I 2022-10-28 05:11:02] ...trap/engines/engine.py.230: loss: 0.14272 [I 2022-10-28 05:11:02] ...optimizers/trijoint.py.151: total_norm: 17.728170 [I 2022-10-28 05:11:08] ...trap/engines/engine.py.224: train: epoch 107 | batch 2070/2304 [I 2022-10-28 05:11:08] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:02:11 [I 2022-10-28 05:11:08] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00246 [I 2022-10-28 05:11:08] ...trap/engines/engine.py.230: loss: 0.10270 [I 2022-10-28 05:11:08] ...optimizers/trijoint.py.151: total_norm: 9.551316 [I 2022-10-28 05:11:13] ...trap/engines/engine.py.224: train: epoch 107 | batch 2080/2304 [I 2022-10-28 05:11:13] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:02:05 [I 2022-10-28 05:11:13] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00246 [I 2022-10-28 05:11:13] ...trap/engines/engine.py.230: loss: 0.09980 [I 2022-10-28 05:11:13] ...optimizers/trijoint.py.151: total_norm: 8.200153 [I 2022-10-28 05:11:19] ...trap/engines/engine.py.224: train: epoch 107 | batch 2090/2304 [I 2022-10-28 05:11:19] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:58 [I 2022-10-28 05:11:19] ...trap/engines/engine.py.229: process: 0.54864 | load: 0.00286 [I 2022-10-28 05:11:19] ...trap/engines/engine.py.230: loss: 0.12577 [I 2022-10-28 05:11:19] ...optimizers/trijoint.py.151: total_norm: 36.419231 [I 2022-10-28 05:11:25] ...trap/engines/engine.py.224: train: epoch 107 | batch 2100/2304 [I 2022-10-28 05:11:25] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:57 [I 2022-10-28 05:11:25] ...trap/engines/engine.py.229: process: 0.56658 | load: 0.00274 [I 2022-10-28 05:11:25] ...trap/engines/engine.py.230: loss: 0.15389 [I 2022-10-28 05:11:25] ...optimizers/trijoint.py.151: total_norm: 11.068098 [I 2022-10-28 05:11:30] ...trap/engines/engine.py.224: train: epoch 107 | batch 2110/2304 [I 2022-10-28 05:11:30] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:48 [I 2022-10-28 05:11:30] ...trap/engines/engine.py.229: process: 0.56168 | load: 0.00285 [I 2022-10-28 05:11:30] ...trap/engines/engine.py.230: loss: 0.10705 [I 2022-10-28 05:11:30] ...optimizers/trijoint.py.151: total_norm: 25.670387 [I 2022-10-28 05:11:36] ...trap/engines/engine.py.224: train: epoch 107 | batch 2120/2304 [I 2022-10-28 05:11:36] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:42 [I 2022-10-28 05:11:36] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00253 [I 2022-10-28 05:11:36] ...trap/engines/engine.py.230: loss: 0.13135 [I 2022-10-28 05:11:36] ...optimizers/trijoint.py.151: total_norm: 13.997276 [I 2022-10-28 05:11:42] ...trap/engines/engine.py.224: train: epoch 107 | batch 2130/2304 [I 2022-10-28 05:11:42] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:36 [I 2022-10-28 05:11:42] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00250 [I 2022-10-28 05:11:42] ...trap/engines/engine.py.230: loss: 0.07076 [I 2022-10-28 05:11:42] ...optimizers/trijoint.py.151: total_norm: 12.943857 [I 2022-10-28 05:11:47] ...trap/engines/engine.py.224: train: epoch 107 | batch 2140/2304 [I 2022-10-28 05:11:47] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:31 [I 2022-10-28 05:11:47] ...trap/engines/engine.py.229: process: 0.54634 | load: 0.00251 [I 2022-10-28 05:11:47] ...trap/engines/engine.py.230: loss: 0.13761 [I 2022-10-28 05:11:47] ...optimizers/trijoint.py.151: total_norm: 11.798346 [I 2022-10-28 05:11:53] ...trap/engines/engine.py.224: train: epoch 107 | batch 2150/2304 [I 2022-10-28 05:11:53] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:26 [I 2022-10-28 05:11:53] ...trap/engines/engine.py.229: process: 0.56209 | load: 0.00256 [I 2022-10-28 05:11:53] ...trap/engines/engine.py.230: loss: 0.09188 [I 2022-10-28 05:11:53] ...optimizers/trijoint.py.151: total_norm: 15.039049 [I 2022-10-28 05:11:58] ...trap/engines/engine.py.224: train: epoch 107 | batch 2160/2304 [I 2022-10-28 05:11:58] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:01:21 [I 2022-10-28 05:11:58] ...trap/engines/engine.py.229: process: 0.56420 | load: 0.00260 [I 2022-10-28 05:11:58] ...trap/engines/engine.py.230: loss: 0.10440 [I 2022-10-28 05:11:58] ...optimizers/trijoint.py.151: total_norm: 14.850027 [I 2022-10-28 05:12:04] ...trap/engines/engine.py.224: train: epoch 107 | batch 2170/2304 [I 2022-10-28 05:12:04] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:15 [I 2022-10-28 05:12:04] ...trap/engines/engine.py.229: process: 0.56523 | load: 0.00249 [I 2022-10-28 05:12:04] ...trap/engines/engine.py.230: loss: 0.09216 [I 2022-10-28 05:12:04] ...optimizers/trijoint.py.151: total_norm: 28.436764 [I 2022-10-28 05:12:10] ...trap/engines/engine.py.224: train: epoch 107 | batch 2180/2304 [I 2022-10-28 05:12:10] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:01:09 [I 2022-10-28 05:12:10] ...trap/engines/engine.py.229: process: 0.56985 | load: 0.00251 [I 2022-10-28 05:12:10] ...trap/engines/engine.py.230: loss: 0.10258 [I 2022-10-28 05:12:10] ...optimizers/trijoint.py.151: total_norm: 24.883066 [I 2022-10-28 05:12:15] ...trap/engines/engine.py.224: train: epoch 107 | batch 2190/2304 [I 2022-10-28 05:12:15] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:01:04 [I 2022-10-28 05:12:15] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00246 [I 2022-10-28 05:12:15] ...trap/engines/engine.py.230: loss: 0.09794 [I 2022-10-28 05:12:15] ...optimizers/trijoint.py.151: total_norm: 39.001366 [I 2022-10-28 05:12:21] ...trap/engines/engine.py.224: train: epoch 107 | batch 2200/2304 [I 2022-10-28 05:12:21] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:58 [I 2022-10-28 05:12:21] ...trap/engines/engine.py.229: process: 0.56678 | load: 0.00246 [I 2022-10-28 05:12:21] ...trap/engines/engine.py.230: loss: 0.11237 [I 2022-10-28 05:12:21] ...optimizers/trijoint.py.151: total_norm: 14.186906 [I 2022-10-28 05:12:26] ...trap/engines/engine.py.224: train: epoch 107 | batch 2210/2304 [I 2022-10-28 05:12:26] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:53 [I 2022-10-28 05:12:26] ...trap/engines/engine.py.229: process: 0.57205 | load: 0.00246 [I 2022-10-28 05:12:26] ...trap/engines/engine.py.230: loss: 0.09040 [I 2022-10-28 05:12:26] ...optimizers/trijoint.py.151: total_norm: 9.562474 [I 2022-10-28 05:12:32] ...trap/engines/engine.py.224: train: epoch 107 | batch 2220/2304 [I 2022-10-28 05:12:32] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:47 [I 2022-10-28 05:12:32] ...trap/engines/engine.py.229: process: 0.56159 | load: 0.00246 [I 2022-10-28 05:12:32] ...trap/engines/engine.py.230: loss: 0.13188 [I 2022-10-28 05:12:32] ...optimizers/trijoint.py.151: total_norm: 22.001862 [I 2022-10-28 05:12:38] ...trap/engines/engine.py.224: train: epoch 107 | batch 2230/2304 [I 2022-10-28 05:12:38] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:41 [I 2022-10-28 05:12:38] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00245 [I 2022-10-28 05:12:38] ...trap/engines/engine.py.230: loss: 0.14216 [I 2022-10-28 05:12:38] ...optimizers/trijoint.py.151: total_norm: 19.492235 [I 2022-10-28 05:12:43] ...trap/engines/engine.py.224: train: epoch 107 | batch 2240/2304 [I 2022-10-28 05:12:43] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:35 [I 2022-10-28 05:12:43] ...trap/engines/engine.py.229: process: 0.54191 | load: 0.00243 [I 2022-10-28 05:12:43] ...trap/engines/engine.py.230: loss: 0.11147 [I 2022-10-28 05:12:43] ...optimizers/trijoint.py.151: total_norm: 8.749245 [I 2022-10-28 05:12:49] ...trap/engines/engine.py.224: train: epoch 107 | batch 2250/2304 [I 2022-10-28 05:12:49] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:30 [I 2022-10-28 05:12:49] ...trap/engines/engine.py.229: process: 0.56438 | load: 0.00249 [I 2022-10-28 05:12:49] ...trap/engines/engine.py.230: loss: 0.16228 [I 2022-10-28 05:12:49] ...optimizers/trijoint.py.151: total_norm: 14.790298 [I 2022-10-28 05:12:55] ...trap/engines/engine.py.224: train: epoch 107 | batch 2260/2304 [I 2022-10-28 05:12:55] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:24 [I 2022-10-28 05:12:55] ...trap/engines/engine.py.229: process: 0.56130 | load: 0.00245 [I 2022-10-28 05:12:55] ...trap/engines/engine.py.230: loss: 0.15423 [I 2022-10-28 05:12:55] ...optimizers/trijoint.py.151: total_norm: 13.688864 [I 2022-10-28 05:13:00] ...trap/engines/engine.py.224: train: epoch 107 | batch 2270/2304 [I 2022-10-28 05:13:00] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:18 [I 2022-10-28 05:13:00] ...trap/engines/engine.py.229: process: 0.55657 | load: 0.00244 [I 2022-10-28 05:13:00] ...trap/engines/engine.py.230: loss: 0.09430 [I 2022-10-28 05:13:00] ...optimizers/trijoint.py.151: total_norm: 9.417425 [I 2022-10-28 05:13:06] ...trap/engines/engine.py.224: train: epoch 107 | batch 2280/2304 [I 2022-10-28 05:13:06] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:00:13 [I 2022-10-28 05:13:06] ...trap/engines/engine.py.229: process: 0.54455 | load: 0.00258 [I 2022-10-28 05:13:06] ...trap/engines/engine.py.230: loss: 0.14117 [I 2022-10-28 05:13:06] ...optimizers/trijoint.py.151: total_norm: 12.533603 [I 2022-10-28 05:13:11] ...trap/engines/engine.py.224: train: epoch 107 | batch 2290/2304 [I 2022-10-28 05:13:11] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:07 [I 2022-10-28 05:13:11] ...trap/engines/engine.py.229: process: 0.55831 | load: 0.00249 [I 2022-10-28 05:13:11] ...trap/engines/engine.py.230: loss: 0.09026 [I 2022-10-28 05:13:11] ...optimizers/trijoint.py.151: total_norm: 11.832556 [I 2022-10-28 05:13:17] ...trap/engines/engine.py.224: train: epoch 107 | batch 2300/2304 [I 2022-10-28 05:13:17] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:00:02 [I 2022-10-28 05:13:17] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00015 [I 2022-10-28 05:13:17] ...trap/engines/engine.py.230: loss: 0.15818 [I 2022-10-28 05:13:17] ...optimizers/trijoint.py.151: total_norm: 13.629767 [I 2022-10-28 05:13:19] ...trap/engines/engine.py.224: train: epoch 107 | batch 2304/2304 [I 2022-10-28 05:13:19] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:00:00 [I 2022-10-28 05:13:19] ...trap/engines/engine.py.229: process: 0.54389 | load: 0.00015 [I 2022-10-28 05:13:19] ...trap/engines/engine.py.230: loss: 0.10639 [I 2022-10-28 05:13:19] ...optimizers/trijoint.py.151: total_norm: 15.608938 [S 2022-10-28 05:13:20] ...trap/engines/engine.py.129: train_epoch.epoch: 107 [S 2022-10-28 05:13:20] ...trap/engines/engine.py.129: train_epoch.loss: 0.12359345429890067 [I 2022-10-28 05:13:20] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 05:13:22] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 107 [I 2022-10-28 05:13:22] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 05:13:24] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 05:13:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 05:13:24] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 05:13:27] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 05:13:28] ...trap/engines/engine.py.321: eval: epoch 107 | batch 0/510 [I 2022-10-28 05:13:28] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:49:29 [I 2022-10-28 05:13:28] ...trap/engines/engine.py.326: process: 5.82245 | load: 2.74181 [I 2022-10-28 05:13:31] ...trap/engines/engine.py.321: eval: epoch 107 | batch 10/510 [I 2022-10-28 05:13:31] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:38 [I 2022-10-28 05:13:31] ...trap/engines/engine.py.326: process: 0.32768 | load: 0.00017 [I 2022-10-28 05:13:34] ...trap/engines/engine.py.321: eval: epoch 107 | batch 20/510 [I 2022-10-28 05:13:34] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:13 [I 2022-10-28 05:13:34] ...trap/engines/engine.py.326: process: 0.32677 | load: 0.00019 [I 2022-10-28 05:13:38] ...trap/engines/engine.py.321: eval: epoch 107 | batch 30/510 [I 2022-10-28 05:13:38] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:43 [I 2022-10-28 05:13:38] ...trap/engines/engine.py.326: process: 0.32475 | load: 0.00020 [I 2022-10-28 05:13:41] ...trap/engines/engine.py.321: eval: epoch 107 | batch 40/510 [I 2022-10-28 05:13:41] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:38 [I 2022-10-28 05:13:41] ...trap/engines/engine.py.326: process: 0.34294 | load: 0.00029 [I 2022-10-28 05:13:44] ...trap/engines/engine.py.321: eval: epoch 107 | batch 50/510 [I 2022-10-28 05:13:44] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:35 [I 2022-10-28 05:13:44] ...trap/engines/engine.py.326: process: 0.35591 | load: 0.00019 [I 2022-10-28 05:13:48] ...trap/engines/engine.py.321: eval: epoch 107 | batch 60/510 [I 2022-10-28 05:13:48] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:29 [I 2022-10-28 05:13:48] ...trap/engines/engine.py.326: process: 0.33301 | load: 0.00016 [I 2022-10-28 05:13:51] ...trap/engines/engine.py.321: eval: epoch 107 | batch 70/510 [I 2022-10-28 05:13:51] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:25 [I 2022-10-28 05:13:51] ...trap/engines/engine.py.326: process: 0.32051 | load: 0.00018 [I 2022-10-28 05:13:54] ...trap/engines/engine.py.321: eval: epoch 107 | batch 80/510 [I 2022-10-28 05:13:54] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:22 [I 2022-10-28 05:13:54] ...trap/engines/engine.py.326: process: 0.32885 | load: 0.00019 [I 2022-10-28 05:13:58] ...trap/engines/engine.py.321: eval: epoch 107 | batch 90/510 [I 2022-10-28 05:13:58] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:20 [I 2022-10-28 05:13:58] ...trap/engines/engine.py.326: process: 0.32898 | load: 0.00017 [I 2022-10-28 05:14:01] ...trap/engines/engine.py.321: eval: epoch 107 | batch 100/510 [I 2022-10-28 05:14:01] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-28 05:14:01] ...trap/engines/engine.py.326: process: 0.28639 | load: 0.00019 [I 2022-10-28 05:14:04] ...trap/engines/engine.py.321: eval: epoch 107 | batch 110/510 [I 2022-10-28 05:14:04] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:56 [I 2022-10-28 05:14:04] ...trap/engines/engine.py.326: process: 0.28379 | load: 0.00014 [I 2022-10-28 05:14:07] ...trap/engines/engine.py.321: eval: epoch 107 | batch 120/510 [I 2022-10-28 05:14:07] ...trap/engines/engine.py.325: elapsed: 0:00:44 | left: 0:01:53 [I 2022-10-28 05:14:07] ...trap/engines/engine.py.326: process: 0.30149 | load: 0.00015 [I 2022-10-28 05:14:10] ...trap/engines/engine.py.321: eval: epoch 107 | batch 130/510 [I 2022-10-28 05:14:10] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-28 05:14:10] ...trap/engines/engine.py.326: process: 0.28408 | load: 0.00016 [I 2022-10-28 05:14:13] ...trap/engines/engine.py.321: eval: epoch 107 | batch 140/510 [I 2022-10-28 05:14:13] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:47 [I 2022-10-28 05:14:13] ...trap/engines/engine.py.326: process: 0.28951 | load: 0.00018 [I 2022-10-28 05:14:15] ...trap/engines/engine.py.321: eval: epoch 107 | batch 150/510 [I 2022-10-28 05:14:15] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-28 05:14:15] ...trap/engines/engine.py.326: process: 0.29176 | load: 0.00016 [I 2022-10-28 05:14:18] ...trap/engines/engine.py.321: eval: epoch 107 | batch 160/510 [I 2022-10-28 05:14:18] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-28 05:14:18] ...trap/engines/engine.py.326: process: 0.28718 | load: 0.00020 [I 2022-10-28 05:14:21] ...trap/engines/engine.py.321: eval: epoch 107 | batch 170/510 [I 2022-10-28 05:14:21] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 05:14:21] ...trap/engines/engine.py.326: process: 0.29975 | load: 0.00026 [I 2022-10-28 05:14:24] ...trap/engines/engine.py.321: eval: epoch 107 | batch 180/510 [I 2022-10-28 05:14:24] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-28 05:14:24] ...trap/engines/engine.py.326: process: 0.29049 | load: 0.00024 [I 2022-10-28 05:14:27] ...trap/engines/engine.py.321: eval: epoch 107 | batch 190/510 [I 2022-10-28 05:14:27] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:32 [I 2022-10-28 05:14:27] ...trap/engines/engine.py.326: process: 0.29335 | load: 0.00026 [I 2022-10-28 05:14:30] ...trap/engines/engine.py.321: eval: epoch 107 | batch 200/510 [I 2022-10-28 05:14:30] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:30 [I 2022-10-28 05:14:30] ...trap/engines/engine.py.326: process: 0.28727 | load: 0.00016 [I 2022-10-28 05:14:33] ...trap/engines/engine.py.321: eval: epoch 107 | batch 210/510 [I 2022-10-28 05:14:33] ...trap/engines/engine.py.325: elapsed: 0:01:10 | left: 0:01:27 [I 2022-10-28 05:14:33] ...trap/engines/engine.py.326: process: 0.30603 | load: 0.00018 [I 2022-10-28 05:14:36] ...trap/engines/engine.py.321: eval: epoch 107 | batch 220/510 [I 2022-10-28 05:14:36] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:24 [I 2022-10-28 05:14:36] ...trap/engines/engine.py.326: process: 0.28532 | load: 0.00024 [I 2022-10-28 05:14:39] ...trap/engines/engine.py.321: eval: epoch 107 | batch 230/510 [I 2022-10-28 05:14:39] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-28 05:14:39] ...trap/engines/engine.py.326: process: 0.29226 | load: 0.00022 [I 2022-10-28 05:14:42] ...trap/engines/engine.py.321: eval: epoch 107 | batch 240/510 [I 2022-10-28 05:14:42] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-28 05:14:42] ...trap/engines/engine.py.326: process: 0.28733 | load: 0.00020 [I 2022-10-28 05:14:44] ...trap/engines/engine.py.321: eval: epoch 107 | batch 250/510 [I 2022-10-28 05:14:44] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-28 05:14:44] ...trap/engines/engine.py.326: process: 0.28510 | load: 0.00018 [I 2022-10-28 05:14:47] ...trap/engines/engine.py.321: eval: epoch 107 | batch 260/510 [I 2022-10-28 05:14:47] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 05:14:47] ...trap/engines/engine.py.326: process: 0.28152 | load: 0.00018 [I 2022-10-28 05:14:50] ...trap/engines/engine.py.321: eval: epoch 107 | batch 270/510 [I 2022-10-28 05:14:50] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-28 05:14:50] ...trap/engines/engine.py.326: process: 0.29079 | load: 0.00020 [I 2022-10-28 05:14:53] ...trap/engines/engine.py.321: eval: epoch 107 | batch 280/510 [I 2022-10-28 05:14:53] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 05:14:53] ...trap/engines/engine.py.326: process: 0.28961 | load: 0.00016 [I 2022-10-28 05:14:56] ...trap/engines/engine.py.321: eval: epoch 107 | batch 290/510 [I 2022-10-28 05:14:56] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 05:14:56] ...trap/engines/engine.py.326: process: 0.28248 | load: 0.00022 [I 2022-10-28 05:14:59] ...trap/engines/engine.py.321: eval: epoch 107 | batch 300/510 [I 2022-10-28 05:14:59] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-28 05:14:59] ...trap/engines/engine.py.326: process: 0.28959 | load: 0.00027 [I 2022-10-28 05:15:02] ...trap/engines/engine.py.321: eval: epoch 107 | batch 310/510 [I 2022-10-28 05:15:02] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 05:15:02] ...trap/engines/engine.py.326: process: 0.28843 | load: 0.00023 [I 2022-10-28 05:15:05] ...trap/engines/engine.py.321: eval: epoch 107 | batch 320/510 [I 2022-10-28 05:15:05] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 05:15:05] ...trap/engines/engine.py.326: process: 0.28738 | load: 0.00025 [I 2022-10-28 05:15:08] ...trap/engines/engine.py.321: eval: epoch 107 | batch 330/510 [I 2022-10-28 05:15:08] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-28 05:15:08] ...trap/engines/engine.py.326: process: 0.28805 | load: 0.00022 [I 2022-10-28 05:15:10] ...trap/engines/engine.py.321: eval: epoch 107 | batch 340/510 [I 2022-10-28 05:15:10] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-28 05:15:10] ...trap/engines/engine.py.326: process: 0.28561 | load: 0.00020 [I 2022-10-28 05:15:13] ...trap/engines/engine.py.321: eval: epoch 107 | batch 350/510 [I 2022-10-28 05:15:13] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:45 [I 2022-10-28 05:15:13] ...trap/engines/engine.py.326: process: 0.29226 | load: 0.00024 [I 2022-10-28 05:15:16] ...trap/engines/engine.py.321: eval: epoch 107 | batch 360/510 [I 2022-10-28 05:15:16] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 05:15:16] ...trap/engines/engine.py.326: process: 0.29040 | load: 0.00015 [I 2022-10-28 05:15:19] ...trap/engines/engine.py.321: eval: epoch 107 | batch 370/510 [I 2022-10-28 05:15:19] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 05:15:19] ...trap/engines/engine.py.326: process: 0.28825 | load: 0.00024 [I 2022-10-28 05:15:22] ...trap/engines/engine.py.321: eval: epoch 107 | batch 380/510 [I 2022-10-28 05:15:22] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 05:15:22] ...trap/engines/engine.py.326: process: 0.28828 | load: 0.00022 [I 2022-10-28 05:15:25] ...trap/engines/engine.py.321: eval: epoch 107 | batch 390/510 [I 2022-10-28 05:15:25] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 05:15:25] ...trap/engines/engine.py.326: process: 0.29216 | load: 0.00025 [I 2022-10-28 05:15:28] ...trap/engines/engine.py.321: eval: epoch 107 | batch 400/510 [I 2022-10-28 05:15:28] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 05:15:28] ...trap/engines/engine.py.326: process: 0.30368 | load: 0.00016 [I 2022-10-28 05:15:31] ...trap/engines/engine.py.321: eval: epoch 107 | batch 410/510 [I 2022-10-28 05:15:31] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 05:15:31] ...trap/engines/engine.py.326: process: 0.28772 | load: 0.00025 [I 2022-10-28 05:15:34] ...trap/engines/engine.py.321: eval: epoch 107 | batch 420/510 [I 2022-10-28 05:15:34] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 05:15:34] ...trap/engines/engine.py.326: process: 0.28973 | load: 0.00015 [I 2022-10-28 05:15:36] ...trap/engines/engine.py.321: eval: epoch 107 | batch 430/510 [I 2022-10-28 05:15:36] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 05:15:36] ...trap/engines/engine.py.326: process: 0.29152 | load: 0.00029 [I 2022-10-28 05:15:39] ...trap/engines/engine.py.321: eval: epoch 107 | batch 440/510 [I 2022-10-28 05:15:39] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 05:15:39] ...trap/engines/engine.py.326: process: 0.28952 | load: 0.00020 [I 2022-10-28 05:15:42] ...trap/engines/engine.py.321: eval: epoch 107 | batch 450/510 [I 2022-10-28 05:15:42] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 05:15:42] ...trap/engines/engine.py.326: process: 0.28809 | load: 0.00020 [I 2022-10-28 05:15:45] ...trap/engines/engine.py.321: eval: epoch 107 | batch 460/510 [I 2022-10-28 05:15:45] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 05:15:45] ...trap/engines/engine.py.326: process: 0.27956 | load: 0.00016 [I 2022-10-28 05:15:48] ...trap/engines/engine.py.321: eval: epoch 107 | batch 470/510 [I 2022-10-28 05:15:48] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 05:15:48] ...trap/engines/engine.py.326: process: 0.29171 | load: 0.00027 [I 2022-10-28 05:15:51] ...trap/engines/engine.py.321: eval: epoch 107 | batch 480/510 [I 2022-10-28 05:15:51] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 05:15:51] ...trap/engines/engine.py.326: process: 0.29137 | load: 0.00015 [I 2022-10-28 05:15:54] ...trap/engines/engine.py.321: eval: epoch 107 | batch 490/510 [I 2022-10-28 05:15:54] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 05:15:54] ...trap/engines/engine.py.326: process: 0.28656 | load: 0.00027 [I 2022-10-28 05:15:57] ...trap/engines/engine.py.321: eval: epoch 107 | batch 500/510 [I 2022-10-28 05:15:57] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 05:15:57] ...trap/engines/engine.py.326: process: 0.28854 | load: 0.00020 [I 2022-10-28 05:15:59] ...trap/engines/engine.py.321: eval: epoch 107 | batch 510/510 [I 2022-10-28 05:15:59] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 05:15:59] ...trap/engines/engine.py.326: process: 0.28789 | load: 0.00012 [S 2022-10-28 05:16:00] ...trap/engines/engine.py.132: eval_epoch.epoch: 107 [I 2022-10-28 05:16:00] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.734500 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.902900 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.931500 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.724000 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.901000 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931000 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009124 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010995 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.006329 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012798 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008775 [S 2022-10-28 05:16:04] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006450 [I 2022-10-28 05:16:06] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 05:16:06] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 05:16:07] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 05:16:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 05:16:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 05:16:12] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 05:16:12] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 05:16:19] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 05:16:19] ...trap/engines/engine.py.163: Training model on trainset for epoch 108 [I 2022-10-28 05:16:19] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 05:16:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 0/2304 [I 2022-10-28 05:16:22] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:58 [I 2022-10-28 05:16:22] ...trap/engines/engine.py.229: process: 2.73358 | load: 2.07941 [I 2022-10-28 05:16:22] ...trap/engines/engine.py.230: loss: 0.12986 [I 2022-10-28 05:16:22] ...optimizers/trijoint.py.151: total_norm: 22.542988 [I 2022-10-28 05:16:27] ...trap/engines/engine.py.224: train: epoch 108 | batch 10/2304 [I 2022-10-28 05:16:27] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:10 [I 2022-10-28 05:16:27] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00412 [I 2022-10-28 05:16:27] ...trap/engines/engine.py.230: loss: 0.11779 [I 2022-10-28 05:16:27] ...optimizers/trijoint.py.151: total_norm: 19.379812 [I 2022-10-28 05:16:33] ...trap/engines/engine.py.224: train: epoch 108 | batch 20/2304 [I 2022-10-28 05:16:33] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:25 [I 2022-10-28 05:16:33] ...trap/engines/engine.py.229: process: 0.57403 | load: 0.00361 [I 2022-10-28 05:16:33] ...trap/engines/engine.py.230: loss: 0.13902 [I 2022-10-28 05:16:33] ...optimizers/trijoint.py.151: total_norm: 27.304255 [I 2022-10-28 05:16:38] ...trap/engines/engine.py.224: train: epoch 108 | batch 30/2304 [I 2022-10-28 05:16:38] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:16 [I 2022-10-28 05:16:38] ...trap/engines/engine.py.229: process: 0.55933 | load: 0.00298 [I 2022-10-28 05:16:38] ...trap/engines/engine.py.230: loss: 0.14318 [I 2022-10-28 05:16:38] ...optimizers/trijoint.py.151: total_norm: 25.150604 [I 2022-10-28 05:16:44] ...trap/engines/engine.py.224: train: epoch 108 | batch 40/2304 [I 2022-10-28 05:16:44] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:19 [I 2022-10-28 05:16:44] ...trap/engines/engine.py.229: process: 0.56113 | load: 0.00310 [I 2022-10-28 05:16:44] ...trap/engines/engine.py.230: loss: 0.10081 [I 2022-10-28 05:16:44] ...optimizers/trijoint.py.151: total_norm: 29.065016 [I 2022-10-28 05:16:50] ...trap/engines/engine.py.224: train: epoch 108 | batch 50/2304 [I 2022-10-28 05:16:50] ...trap/engines/engine.py.228: elapsed: 0:00:31 | left: 0:21:45 [I 2022-10-28 05:16:50] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00293 [I 2022-10-28 05:16:50] ...trap/engines/engine.py.230: loss: 0.10419 [I 2022-10-28 05:16:50] ...optimizers/trijoint.py.151: total_norm: 16.844427 [I 2022-10-28 05:16:55] ...trap/engines/engine.py.224: train: epoch 108 | batch 60/2304 [I 2022-10-28 05:16:55] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:55 [I 2022-10-28 05:16:55] ...trap/engines/engine.py.229: process: 0.55490 | load: 0.00296 [I 2022-10-28 05:16:55] ...trap/engines/engine.py.230: loss: 0.06925 [I 2022-10-28 05:16:55] ...optimizers/trijoint.py.151: total_norm: 12.802191 [I 2022-10-28 05:17:01] ...trap/engines/engine.py.224: train: epoch 108 | batch 70/2304 [I 2022-10-28 05:17:01] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:52 [I 2022-10-28 05:17:01] ...trap/engines/engine.py.229: process: 0.56383 | load: 0.00291 [I 2022-10-28 05:17:01] ...trap/engines/engine.py.230: loss: 0.14388 [I 2022-10-28 05:17:01] ...optimizers/trijoint.py.151: total_norm: 12.547488 [I 2022-10-28 05:17:07] ...trap/engines/engine.py.224: train: epoch 108 | batch 80/2304 [I 2022-10-28 05:17:07] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:31 [I 2022-10-28 05:17:07] ...trap/engines/engine.py.229: process: 0.54974 | load: 0.00268 [I 2022-10-28 05:17:07] ...trap/engines/engine.py.230: loss: 0.15963 [I 2022-10-28 05:17:07] ...optimizers/trijoint.py.151: total_norm: 22.293804 [I 2022-10-28 05:17:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 90/2304 [I 2022-10-28 05:17:12] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:32 [I 2022-10-28 05:17:12] ...trap/engines/engine.py.229: process: 0.56738 | load: 0.00259 [I 2022-10-28 05:17:12] ...trap/engines/engine.py.230: loss: 0.14900 [I 2022-10-28 05:17:12] ...optimizers/trijoint.py.151: total_norm: 24.680416 [I 2022-10-28 05:17:18] ...trap/engines/engine.py.224: train: epoch 108 | batch 100/2304 [I 2022-10-28 05:17:18] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:18 [I 2022-10-28 05:17:18] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00276 [I 2022-10-28 05:17:18] ...trap/engines/engine.py.230: loss: 0.10918 [I 2022-10-28 05:17:18] ...optimizers/trijoint.py.151: total_norm: 14.313858 [I 2022-10-28 05:17:23] ...trap/engines/engine.py.224: train: epoch 108 | batch 110/2304 [I 2022-10-28 05:17:23] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:23 [I 2022-10-28 05:17:23] ...trap/engines/engine.py.229: process: 0.56426 | load: 0.00281 [I 2022-10-28 05:17:23] ...trap/engines/engine.py.230: loss: 0.15330 [I 2022-10-28 05:17:23] ...optimizers/trijoint.py.151: total_norm: 30.253069 [I 2022-10-28 05:17:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 120/2304 [I 2022-10-28 05:17:29] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:29 [I 2022-10-28 05:17:29] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00252 [I 2022-10-28 05:17:29] ...trap/engines/engine.py.230: loss: 0.17956 [I 2022-10-28 05:17:29] ...optimizers/trijoint.py.151: total_norm: 27.623413 [I 2022-10-28 05:17:35] ...trap/engines/engine.py.224: train: epoch 108 | batch 130/2304 [I 2022-10-28 05:17:35] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:24 [I 2022-10-28 05:17:35] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00252 [I 2022-10-28 05:17:35] ...trap/engines/engine.py.230: loss: 0.14152 [I 2022-10-28 05:17:35] ...optimizers/trijoint.py.151: total_norm: 12.746739 [I 2022-10-28 05:17:40] ...trap/engines/engine.py.224: train: epoch 108 | batch 140/2304 [I 2022-10-28 05:17:40] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:06 [I 2022-10-28 05:17:40] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00273 [I 2022-10-28 05:17:40] ...trap/engines/engine.py.230: loss: 0.15197 [I 2022-10-28 05:17:40] ...optimizers/trijoint.py.151: total_norm: 16.876085 [I 2022-10-28 05:17:46] ...trap/engines/engine.py.224: train: epoch 108 | batch 150/2304 [I 2022-10-28 05:17:46] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:49 [I 2022-10-28 05:17:46] ...trap/engines/engine.py.229: process: 0.55119 | load: 0.00245 [I 2022-10-28 05:17:46] ...trap/engines/engine.py.230: loss: 0.10855 [I 2022-10-28 05:17:46] ...optimizers/trijoint.py.151: total_norm: 17.565643 [I 2022-10-28 05:17:51] ...trap/engines/engine.py.224: train: epoch 108 | batch 160/2304 [I 2022-10-28 05:17:51] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:03 [I 2022-10-28 05:17:51] ...trap/engines/engine.py.229: process: 0.56975 | load: 0.00253 [I 2022-10-28 05:17:51] ...trap/engines/engine.py.230: loss: 0.08328 [I 2022-10-28 05:17:51] ...optimizers/trijoint.py.151: total_norm: 12.277061 [I 2022-10-28 05:17:57] ...trap/engines/engine.py.224: train: epoch 108 | batch 170/2304 [I 2022-10-28 05:17:57] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:54 [I 2022-10-28 05:17:57] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00317 [I 2022-10-28 05:17:57] ...trap/engines/engine.py.230: loss: 0.12348 [I 2022-10-28 05:17:57] ...optimizers/trijoint.py.151: total_norm: 47.236958 [I 2022-10-28 05:18:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 180/2304 [I 2022-10-28 05:18:03] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:48 [I 2022-10-28 05:18:03] ...trap/engines/engine.py.229: process: 0.55861 | load: 0.00272 [I 2022-10-28 05:18:03] ...trap/engines/engine.py.230: loss: 0.09298 [I 2022-10-28 05:18:03] ...optimizers/trijoint.py.151: total_norm: 11.146887 [I 2022-10-28 05:18:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 190/2304 [I 2022-10-28 05:18:08] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:38 [I 2022-10-28 05:18:08] ...trap/engines/engine.py.229: process: 0.54514 | load: 0.00257 [I 2022-10-28 05:18:08] ...trap/engines/engine.py.230: loss: 0.12749 [I 2022-10-28 05:18:08] ...optimizers/trijoint.py.151: total_norm: 14.538842 [I 2022-10-28 05:18:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 200/2304 [I 2022-10-28 05:18:14] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:29 [I 2022-10-28 05:18:14] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00245 [I 2022-10-28 05:18:14] ...trap/engines/engine.py.230: loss: 0.10226 [I 2022-10-28 05:18:14] ...optimizers/trijoint.py.151: total_norm: 18.107004 [I 2022-10-28 05:18:19] ...trap/engines/engine.py.224: train: epoch 108 | batch 210/2304 [I 2022-10-28 05:18:19] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:30 [I 2022-10-28 05:18:19] ...trap/engines/engine.py.229: process: 0.57079 | load: 0.00262 [I 2022-10-28 05:18:19] ...trap/engines/engine.py.230: loss: 0.14555 [I 2022-10-28 05:18:19] ...optimizers/trijoint.py.151: total_norm: 34.469830 [I 2022-10-28 05:18:25] ...trap/engines/engine.py.224: train: epoch 108 | batch 220/2304 [I 2022-10-28 05:18:25] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:21 [I 2022-10-28 05:18:25] ...trap/engines/engine.py.229: process: 0.54468 | load: 0.00260 [I 2022-10-28 05:18:25] ...trap/engines/engine.py.230: loss: 0.16996 [I 2022-10-28 05:18:25] ...optimizers/trijoint.py.151: total_norm: 29.121912 [I 2022-10-28 05:18:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 230/2304 [I 2022-10-28 05:18:31] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:14 [I 2022-10-28 05:18:31] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00259 [I 2022-10-28 05:18:31] ...trap/engines/engine.py.230: loss: 0.14799 [I 2022-10-28 05:18:31] ...optimizers/trijoint.py.151: total_norm: 10.455418 [I 2022-10-28 05:18:36] ...trap/engines/engine.py.224: train: epoch 108 | batch 240/2304 [I 2022-10-28 05:18:36] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:09 [I 2022-10-28 05:18:36] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00248 [I 2022-10-28 05:18:36] ...trap/engines/engine.py.230: loss: 0.13636 [I 2022-10-28 05:18:36] ...optimizers/trijoint.py.151: total_norm: 10.121405 [I 2022-10-28 05:18:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 250/2304 [I 2022-10-28 05:18:42] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:03 [I 2022-10-28 05:18:42] ...trap/engines/engine.py.229: process: 0.56685 | load: 0.00248 [I 2022-10-28 05:18:42] ...trap/engines/engine.py.230: loss: 0.13852 [I 2022-10-28 05:18:42] ...optimizers/trijoint.py.151: total_norm: 10.307115 [I 2022-10-28 05:18:47] ...trap/engines/engine.py.224: train: epoch 108 | batch 260/2304 [I 2022-10-28 05:18:47] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:51 [I 2022-10-28 05:18:47] ...trap/engines/engine.py.229: process: 0.54642 | load: 0.00253 [I 2022-10-28 05:18:47] ...trap/engines/engine.py.230: loss: 0.10339 [I 2022-10-28 05:18:47] ...optimizers/trijoint.py.151: total_norm: 15.423113 [I 2022-10-28 05:18:53] ...trap/engines/engine.py.224: train: epoch 108 | batch 270/2304 [I 2022-10-28 05:18:53] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:56 [I 2022-10-28 05:18:53] ...trap/engines/engine.py.229: process: 0.56487 | load: 0.00251 [I 2022-10-28 05:18:53] ...trap/engines/engine.py.230: loss: 0.09918 [I 2022-10-28 05:18:53] ...optimizers/trijoint.py.151: total_norm: 23.308952 [I 2022-10-28 05:18:58] ...trap/engines/engine.py.224: train: epoch 108 | batch 280/2304 [I 2022-10-28 05:18:58] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:49 [I 2022-10-28 05:18:58] ...trap/engines/engine.py.229: process: 0.56596 | load: 0.00255 [I 2022-10-28 05:18:58] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-28 05:18:58] ...optimizers/trijoint.py.151: total_norm: 8.897434 [I 2022-10-28 05:19:04] ...trap/engines/engine.py.224: train: epoch 108 | batch 290/2304 [I 2022-10-28 05:19:04] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:40 [I 2022-10-28 05:19:04] ...trap/engines/engine.py.229: process: 0.54900 | load: 0.00254 [I 2022-10-28 05:19:04] ...trap/engines/engine.py.230: loss: 0.15207 [I 2022-10-28 05:19:04] ...optimizers/trijoint.py.151: total_norm: 20.057323 [I 2022-10-28 05:19:10] ...trap/engines/engine.py.224: train: epoch 108 | batch 300/2304 [I 2022-10-28 05:19:10] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:38 [I 2022-10-28 05:19:10] ...trap/engines/engine.py.229: process: 0.55768 | load: 0.00248 [I 2022-10-28 05:19:10] ...trap/engines/engine.py.230: loss: 0.12023 [I 2022-10-28 05:19:10] ...optimizers/trijoint.py.151: total_norm: 22.794212 [I 2022-10-28 05:19:15] ...trap/engines/engine.py.224: train: epoch 108 | batch 310/2304 [I 2022-10-28 05:19:15] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:19:12 [I 2022-10-28 05:19:15] ...trap/engines/engine.py.229: process: 0.65889 | load: 0.00246 [I 2022-10-28 05:19:15] ...trap/engines/engine.py.230: loss: 0.10380 [I 2022-10-28 05:19:15] ...optimizers/trijoint.py.151: total_norm: 9.600101 [I 2022-10-28 05:19:21] ...trap/engines/engine.py.224: train: epoch 108 | batch 320/2304 [I 2022-10-28 05:19:21] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:28 [I 2022-10-28 05:19:21] ...trap/engines/engine.py.229: process: 0.55362 | load: 0.00260 [I 2022-10-28 05:19:21] ...trap/engines/engine.py.230: loss: 0.14813 [I 2022-10-28 05:19:21] ...optimizers/trijoint.py.151: total_norm: 13.955936 [I 2022-10-28 05:19:26] ...trap/engines/engine.py.224: train: epoch 108 | batch 330/2304 [I 2022-10-28 05:19:26] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:14 [I 2022-10-28 05:19:26] ...trap/engines/engine.py.229: process: 0.55221 | load: 0.00261 [I 2022-10-28 05:19:26] ...trap/engines/engine.py.230: loss: 0.14015 [I 2022-10-28 05:19:26] ...optimizers/trijoint.py.151: total_norm: 11.096972 [I 2022-10-28 05:19:32] ...trap/engines/engine.py.224: train: epoch 108 | batch 340/2304 [I 2022-10-28 05:19:32] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:06 [I 2022-10-28 05:19:32] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00254 [I 2022-10-28 05:19:32] ...trap/engines/engine.py.230: loss: 0.13256 [I 2022-10-28 05:19:32] ...optimizers/trijoint.py.151: total_norm: 19.172291 [I 2022-10-28 05:19:38] ...trap/engines/engine.py.224: train: epoch 108 | batch 350/2304 [I 2022-10-28 05:19:38] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:06 [I 2022-10-28 05:19:38] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00258 [I 2022-10-28 05:19:38] ...trap/engines/engine.py.230: loss: 0.18268 [I 2022-10-28 05:19:38] ...optimizers/trijoint.py.151: total_norm: 24.110739 [I 2022-10-28 05:19:43] ...trap/engines/engine.py.224: train: epoch 108 | batch 360/2304 [I 2022-10-28 05:19:43] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:05 [I 2022-10-28 05:19:43] ...trap/engines/engine.py.229: process: 0.56048 | load: 0.00249 [I 2022-10-28 05:19:43] ...trap/engines/engine.py.230: loss: 0.12455 [I 2022-10-28 05:19:43] ...optimizers/trijoint.py.151: total_norm: 11.618784 [I 2022-10-28 05:19:49] ...trap/engines/engine.py.224: train: epoch 108 | batch 370/2304 [I 2022-10-28 05:19:49] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:00 [I 2022-10-28 05:19:49] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00243 [I 2022-10-28 05:19:49] ...trap/engines/engine.py.230: loss: 0.17730 [I 2022-10-28 05:19:49] ...optimizers/trijoint.py.151: total_norm: 10.404107 [I 2022-10-28 05:19:54] ...trap/engines/engine.py.224: train: epoch 108 | batch 380/2304 [I 2022-10-28 05:19:54] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:46 [I 2022-10-28 05:19:54] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00251 [I 2022-10-28 05:19:54] ...trap/engines/engine.py.230: loss: 0.15971 [I 2022-10-28 05:19:54] ...optimizers/trijoint.py.151: total_norm: 17.363234 [I 2022-10-28 05:20:00] ...trap/engines/engine.py.224: train: epoch 108 | batch 390/2304 [I 2022-10-28 05:20:00] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:52 [I 2022-10-28 05:20:00] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00249 [I 2022-10-28 05:20:00] ...trap/engines/engine.py.230: loss: 0.11925 [I 2022-10-28 05:20:00] ...optimizers/trijoint.py.151: total_norm: 16.065907 [I 2022-10-28 05:20:06] ...trap/engines/engine.py.224: train: epoch 108 | batch 400/2304 [I 2022-10-28 05:20:06] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:46 [I 2022-10-28 05:20:06] ...trap/engines/engine.py.229: process: 0.56570 | load: 0.00248 [I 2022-10-28 05:20:06] ...trap/engines/engine.py.230: loss: 0.13406 [I 2022-10-28 05:20:06] ...optimizers/trijoint.py.151: total_norm: 21.764164 [I 2022-10-28 05:20:11] ...trap/engines/engine.py.224: train: epoch 108 | batch 410/2304 [I 2022-10-28 05:20:11] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:35 [I 2022-10-28 05:20:11] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00248 [I 2022-10-28 05:20:11] ...trap/engines/engine.py.230: loss: 0.16004 [I 2022-10-28 05:20:11] ...optimizers/trijoint.py.151: total_norm: 28.713938 [I 2022-10-28 05:20:17] ...trap/engines/engine.py.224: train: epoch 108 | batch 420/2304 [I 2022-10-28 05:20:17] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:18:09 [I 2022-10-28 05:20:17] ...trap/engines/engine.py.229: process: 0.56459 | load: 0.00246 [I 2022-10-28 05:20:17] ...trap/engines/engine.py.230: loss: 0.08084 [I 2022-10-28 05:20:17] ...optimizers/trijoint.py.151: total_norm: 12.856859 [I 2022-10-28 05:20:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 430/2304 [I 2022-10-28 05:20:22] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:22 [I 2022-10-28 05:20:22] ...trap/engines/engine.py.229: process: 0.55115 | load: 0.00259 [I 2022-10-28 05:20:22] ...trap/engines/engine.py.230: loss: 0.14219 [I 2022-10-28 05:20:22] ...optimizers/trijoint.py.151: total_norm: 24.244934 [I 2022-10-28 05:20:28] ...trap/engines/engine.py.224: train: epoch 108 | batch 440/2304 [I 2022-10-28 05:20:28] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:26 [I 2022-10-28 05:20:28] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00247 [I 2022-10-28 05:20:28] ...trap/engines/engine.py.230: loss: 0.11874 [I 2022-10-28 05:20:28] ...optimizers/trijoint.py.151: total_norm: 10.155643 [I 2022-10-28 05:20:34] ...trap/engines/engine.py.224: train: epoch 108 | batch 450/2304 [I 2022-10-28 05:20:34] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:19 [I 2022-10-28 05:20:34] ...trap/engines/engine.py.229: process: 0.56641 | load: 0.00292 [I 2022-10-28 05:20:34] ...trap/engines/engine.py.230: loss: 0.15372 [I 2022-10-28 05:20:34] ...optimizers/trijoint.py.151: total_norm: 9.859528 [I 2022-10-28 05:20:39] ...trap/engines/engine.py.224: train: epoch 108 | batch 460/2304 [I 2022-10-28 05:20:39] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:13 [I 2022-10-28 05:20:39] ...trap/engines/engine.py.229: process: 0.56297 | load: 0.00244 [I 2022-10-28 05:20:39] ...trap/engines/engine.py.230: loss: 0.12634 [I 2022-10-28 05:20:39] ...optimizers/trijoint.py.151: total_norm: 13.457845 [I 2022-10-28 05:20:45] ...trap/engines/engine.py.224: train: epoch 108 | batch 470/2304 [I 2022-10-28 05:20:45] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:58 [I 2022-10-28 05:20:45] ...trap/engines/engine.py.229: process: 0.54584 | load: 0.00244 [I 2022-10-28 05:20:45] ...trap/engines/engine.py.230: loss: 0.09378 [I 2022-10-28 05:20:45] ...optimizers/trijoint.py.151: total_norm: 13.659584 [I 2022-10-28 05:20:50] ...trap/engines/engine.py.224: train: epoch 108 | batch 480/2304 [I 2022-10-28 05:20:50] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:00 [I 2022-10-28 05:20:50] ...trap/engines/engine.py.229: process: 0.56353 | load: 0.00244 [I 2022-10-28 05:20:50] ...trap/engines/engine.py.230: loss: 0.09272 [I 2022-10-28 05:20:50] ...optimizers/trijoint.py.151: total_norm: 15.738302 [I 2022-10-28 05:20:56] ...trap/engines/engine.py.224: train: epoch 108 | batch 490/2304 [I 2022-10-28 05:20:56] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:48 [I 2022-10-28 05:20:56] ...trap/engines/engine.py.229: process: 0.54647 | load: 0.00247 [I 2022-10-28 05:20:56] ...trap/engines/engine.py.230: loss: 0.11762 [I 2022-10-28 05:20:56] ...optimizers/trijoint.py.151: total_norm: 21.192600 [I 2022-10-28 05:21:02] ...trap/engines/engine.py.224: train: epoch 108 | batch 500/2304 [I 2022-10-28 05:21:02] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:46 [I 2022-10-28 05:21:02] ...trap/engines/engine.py.229: process: 0.56951 | load: 0.00246 [I 2022-10-28 05:21:02] ...trap/engines/engine.py.230: loss: 0.16602 [I 2022-10-28 05:21:02] ...optimizers/trijoint.py.151: total_norm: 31.578596 [I 2022-10-28 05:21:07] ...trap/engines/engine.py.224: train: epoch 108 | batch 510/2304 [I 2022-10-28 05:21:07] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:43 [I 2022-10-28 05:21:07] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00251 [I 2022-10-28 05:21:07] ...trap/engines/engine.py.230: loss: 0.11298 [I 2022-10-28 05:21:07] ...optimizers/trijoint.py.151: total_norm: 31.801676 [I 2022-10-28 05:21:13] ...trap/engines/engine.py.224: train: epoch 108 | batch 520/2304 [I 2022-10-28 05:21:13] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:41 [I 2022-10-28 05:21:13] ...trap/engines/engine.py.229: process: 0.56549 | load: 0.00251 [I 2022-10-28 05:21:13] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-28 05:21:13] ...optimizers/trijoint.py.151: total_norm: 26.191473 [I 2022-10-28 05:21:18] ...trap/engines/engine.py.224: train: epoch 108 | batch 530/2304 [I 2022-10-28 05:21:18] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:28 [I 2022-10-28 05:21:18] ...trap/engines/engine.py.229: process: 0.56428 | load: 0.00264 [I 2022-10-28 05:21:18] ...trap/engines/engine.py.230: loss: 0.14171 [I 2022-10-28 05:21:18] ...optimizers/trijoint.py.151: total_norm: 10.178267 [I 2022-10-28 05:21:24] ...trap/engines/engine.py.224: train: epoch 108 | batch 540/2304 [I 2022-10-28 05:21:24] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:33 [I 2022-10-28 05:21:24] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00248 [I 2022-10-28 05:21:24] ...trap/engines/engine.py.230: loss: 0.11174 [I 2022-10-28 05:21:24] ...optimizers/trijoint.py.151: total_norm: 11.208720 [I 2022-10-28 05:21:30] ...trap/engines/engine.py.224: train: epoch 108 | batch 550/2304 [I 2022-10-28 05:21:30] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:25 [I 2022-10-28 05:21:30] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00254 [I 2022-10-28 05:21:30] ...trap/engines/engine.py.230: loss: 0.17015 [I 2022-10-28 05:21:30] ...optimizers/trijoint.py.151: total_norm: 15.767616 [I 2022-10-28 05:21:35] ...trap/engines/engine.py.224: train: epoch 108 | batch 560/2304 [I 2022-10-28 05:21:35] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:17 [I 2022-10-28 05:21:35] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00250 [I 2022-10-28 05:21:35] ...trap/engines/engine.py.230: loss: 0.07890 [I 2022-10-28 05:21:35] ...optimizers/trijoint.py.151: total_norm: 12.853765 [I 2022-10-28 05:21:41] ...trap/engines/engine.py.224: train: epoch 108 | batch 570/2304 [I 2022-10-28 05:21:41] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:08 [I 2022-10-28 05:21:41] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00244 [I 2022-10-28 05:21:41] ...trap/engines/engine.py.230: loss: 0.13482 [I 2022-10-28 05:21:41] ...optimizers/trijoint.py.151: total_norm: 29.461338 [I 2022-10-28 05:21:47] ...trap/engines/engine.py.224: train: epoch 108 | batch 580/2304 [I 2022-10-28 05:21:47] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:07 [I 2022-10-28 05:21:47] ...trap/engines/engine.py.229: process: 0.57306 | load: 0.00280 [I 2022-10-28 05:21:47] ...trap/engines/engine.py.230: loss: 0.10678 [I 2022-10-28 05:21:47] ...optimizers/trijoint.py.151: total_norm: 14.089002 [I 2022-10-28 05:21:52] ...trap/engines/engine.py.224: train: epoch 108 | batch 590/2304 [I 2022-10-28 05:21:52] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:57 [I 2022-10-28 05:21:52] ...trap/engines/engine.py.229: process: 0.55772 | load: 0.00293 [I 2022-10-28 05:21:52] ...trap/engines/engine.py.230: loss: 0.10027 [I 2022-10-28 05:21:52] ...optimizers/trijoint.py.151: total_norm: 14.473080 [I 2022-10-28 05:21:58] ...trap/engines/engine.py.224: train: epoch 108 | batch 600/2304 [I 2022-10-28 05:21:58] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:58 [I 2022-10-28 05:21:58] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00306 [I 2022-10-28 05:21:58] ...trap/engines/engine.py.230: loss: 0.10549 [I 2022-10-28 05:21:58] ...optimizers/trijoint.py.151: total_norm: 10.743493 [I 2022-10-28 05:22:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 610/2304 [I 2022-10-28 05:22:03] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:39 [I 2022-10-28 05:22:03] ...trap/engines/engine.py.229: process: 0.54929 | load: 0.00245 [I 2022-10-28 05:22:03] ...trap/engines/engine.py.230: loss: 0.10172 [I 2022-10-28 05:22:03] ...optimizers/trijoint.py.151: total_norm: 18.328543 [I 2022-10-28 05:22:09] ...trap/engines/engine.py.224: train: epoch 108 | batch 620/2304 [I 2022-10-28 05:22:09] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:39 [I 2022-10-28 05:22:09] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00247 [I 2022-10-28 05:22:09] ...trap/engines/engine.py.230: loss: 0.15847 [I 2022-10-28 05:22:09] ...optimizers/trijoint.py.151: total_norm: 12.366661 [I 2022-10-28 05:22:15] ...trap/engines/engine.py.224: train: epoch 108 | batch 630/2304 [I 2022-10-28 05:22:15] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:36 [I 2022-10-28 05:22:15] ...trap/engines/engine.py.229: process: 0.56987 | load: 0.00266 [I 2022-10-28 05:22:15] ...trap/engines/engine.py.230: loss: 0.13358 [I 2022-10-28 05:22:15] ...optimizers/trijoint.py.151: total_norm: 20.034840 [I 2022-10-28 05:22:20] ...trap/engines/engine.py.224: train: epoch 108 | batch 640/2304 [I 2022-10-28 05:22:20] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:33 [I 2022-10-28 05:22:20] ...trap/engines/engine.py.229: process: 0.57289 | load: 0.00250 [I 2022-10-28 05:22:20] ...trap/engines/engine.py.230: loss: 0.10476 [I 2022-10-28 05:22:20] ...optimizers/trijoint.py.151: total_norm: 22.560604 [I 2022-10-28 05:22:26] ...trap/engines/engine.py.224: train: epoch 108 | batch 650/2304 [I 2022-10-28 05:22:26] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:23 [I 2022-10-28 05:22:26] ...trap/engines/engine.py.229: process: 0.55198 | load: 0.00247 [I 2022-10-28 05:22:26] ...trap/engines/engine.py.230: loss: 0.09734 [I 2022-10-28 05:22:26] ...optimizers/trijoint.py.151: total_norm: 11.411446 [I 2022-10-28 05:22:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 660/2304 [I 2022-10-28 05:22:31] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:15 [I 2022-10-28 05:22:31] ...trap/engines/engine.py.229: process: 0.55477 | load: 0.00246 [I 2022-10-28 05:22:31] ...trap/engines/engine.py.230: loss: 0.20793 [I 2022-10-28 05:22:31] ...optimizers/trijoint.py.151: total_norm: 42.773102 [I 2022-10-28 05:22:37] ...trap/engines/engine.py.224: train: epoch 108 | batch 670/2304 [I 2022-10-28 05:22:37] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:13 [I 2022-10-28 05:22:37] ...trap/engines/engine.py.229: process: 0.55543 | load: 0.00254 [I 2022-10-28 05:22:37] ...trap/engines/engine.py.230: loss: 0.10323 [I 2022-10-28 05:22:37] ...optimizers/trijoint.py.151: total_norm: 9.684162 [I 2022-10-28 05:22:43] ...trap/engines/engine.py.224: train: epoch 108 | batch 680/2304 [I 2022-10-28 05:22:43] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:03 [I 2022-10-28 05:22:43] ...trap/engines/engine.py.229: process: 0.56746 | load: 0.00254 [I 2022-10-28 05:22:43] ...trap/engines/engine.py.230: loss: 0.13953 [I 2022-10-28 05:22:43] ...optimizers/trijoint.py.151: total_norm: 23.448605 [I 2022-10-28 05:22:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 690/2304 [I 2022-10-28 05:22:48] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:08 [I 2022-10-28 05:22:48] ...trap/engines/engine.py.229: process: 0.56253 | load: 0.00248 [I 2022-10-28 05:22:48] ...trap/engines/engine.py.230: loss: 0.13538 [I 2022-10-28 05:22:48] ...optimizers/trijoint.py.151: total_norm: 17.738180 [I 2022-10-28 05:22:54] ...trap/engines/engine.py.224: train: epoch 108 | batch 700/2304 [I 2022-10-28 05:22:54] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:58 [I 2022-10-28 05:22:54] ...trap/engines/engine.py.229: process: 0.56656 | load: 0.00248 [I 2022-10-28 05:22:54] ...trap/engines/engine.py.230: loss: 0.14033 [I 2022-10-28 05:22:54] ...optimizers/trijoint.py.151: total_norm: 24.852064 [I 2022-10-28 05:22:59] ...trap/engines/engine.py.224: train: epoch 108 | batch 710/2304 [I 2022-10-28 05:22:59] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:59 [I 2022-10-28 05:22:59] ...trap/engines/engine.py.229: process: 0.56649 | load: 0.00248 [I 2022-10-28 05:22:59] ...trap/engines/engine.py.230: loss: 0.12265 [I 2022-10-28 05:22:59] ...optimizers/trijoint.py.151: total_norm: 18.708035 [I 2022-10-28 05:23:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 720/2304 [I 2022-10-28 05:23:05] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:52 [I 2022-10-28 05:23:05] ...trap/engines/engine.py.229: process: 0.55961 | load: 0.00244 [I 2022-10-28 05:23:05] ...trap/engines/engine.py.230: loss: 0.12380 [I 2022-10-28 05:23:05] ...optimizers/trijoint.py.151: total_norm: 11.488198 [I 2022-10-28 05:23:11] ...trap/engines/engine.py.224: train: epoch 108 | batch 730/2304 [I 2022-10-28 05:23:11] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:36 [I 2022-10-28 05:23:11] ...trap/engines/engine.py.229: process: 0.55283 | load: 0.00243 [I 2022-10-28 05:23:11] ...trap/engines/engine.py.230: loss: 0.09606 [I 2022-10-28 05:23:11] ...optimizers/trijoint.py.151: total_norm: 13.088361 [I 2022-10-28 05:23:16] ...trap/engines/engine.py.224: train: epoch 108 | batch 740/2304 [I 2022-10-28 05:23:16] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:31 [I 2022-10-28 05:23:16] ...trap/engines/engine.py.229: process: 0.56594 | load: 0.00244 [I 2022-10-28 05:23:16] ...trap/engines/engine.py.230: loss: 0.15345 [I 2022-10-28 05:23:16] ...optimizers/trijoint.py.151: total_norm: 14.384319 [I 2022-10-28 05:23:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 750/2304 [I 2022-10-28 05:23:22] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:30 [I 2022-10-28 05:23:22] ...trap/engines/engine.py.229: process: 0.56054 | load: 0.00244 [I 2022-10-28 05:23:22] ...trap/engines/engine.py.230: loss: 0.16964 [I 2022-10-28 05:23:22] ...optimizers/trijoint.py.151: total_norm: 17.554008 [I 2022-10-28 05:23:27] ...trap/engines/engine.py.224: train: epoch 108 | batch 760/2304 [I 2022-10-28 05:23:27] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:25 [I 2022-10-28 05:23:27] ...trap/engines/engine.py.229: process: 0.56497 | load: 0.00246 [I 2022-10-28 05:23:27] ...trap/engines/engine.py.230: loss: 0.12734 [I 2022-10-28 05:23:27] ...optimizers/trijoint.py.151: total_norm: 21.444979 [I 2022-10-28 05:23:33] ...trap/engines/engine.py.224: train: epoch 108 | batch 770/2304 [I 2022-10-28 05:23:33] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:17 [I 2022-10-28 05:23:33] ...trap/engines/engine.py.229: process: 0.56821 | load: 0.00246 [I 2022-10-28 05:23:33] ...trap/engines/engine.py.230: loss: 0.14450 [I 2022-10-28 05:23:33] ...optimizers/trijoint.py.151: total_norm: 11.640141 [I 2022-10-28 05:23:39] ...trap/engines/engine.py.224: train: epoch 108 | batch 780/2304 [I 2022-10-28 05:23:39] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:12 [I 2022-10-28 05:23:39] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00245 [I 2022-10-28 05:23:39] ...trap/engines/engine.py.230: loss: 0.13976 [I 2022-10-28 05:23:39] ...optimizers/trijoint.py.151: total_norm: 15.552394 [I 2022-10-28 05:23:44] ...trap/engines/engine.py.224: train: epoch 108 | batch 790/2304 [I 2022-10-28 05:23:44] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:05 [I 2022-10-28 05:23:44] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00256 [I 2022-10-28 05:23:44] ...trap/engines/engine.py.230: loss: 0.15431 [I 2022-10-28 05:23:44] ...optimizers/trijoint.py.151: total_norm: 18.949682 [I 2022-10-28 05:23:50] ...trap/engines/engine.py.224: train: epoch 108 | batch 800/2304 [I 2022-10-28 05:23:50] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:01 [I 2022-10-28 05:23:50] ...trap/engines/engine.py.229: process: 0.55369 | load: 0.00252 [I 2022-10-28 05:23:50] ...trap/engines/engine.py.230: loss: 0.07434 [I 2022-10-28 05:23:50] ...optimizers/trijoint.py.151: total_norm: 15.091237 [I 2022-10-28 05:23:55] ...trap/engines/engine.py.224: train: epoch 108 | batch 810/2304 [I 2022-10-28 05:23:55] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:48 [I 2022-10-28 05:23:55] ...trap/engines/engine.py.229: process: 0.54407 | load: 0.00244 [I 2022-10-28 05:23:55] ...trap/engines/engine.py.230: loss: 0.11291 [I 2022-10-28 05:23:55] ...optimizers/trijoint.py.151: total_norm: 13.473226 [I 2022-10-28 05:24:01] ...trap/engines/engine.py.224: train: epoch 108 | batch 820/2304 [I 2022-10-28 05:24:01] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:47 [I 2022-10-28 05:24:01] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00244 [I 2022-10-28 05:24:01] ...trap/engines/engine.py.230: loss: 0.13717 [I 2022-10-28 05:24:01] ...optimizers/trijoint.py.151: total_norm: 27.404993 [I 2022-10-28 05:24:07] ...trap/engines/engine.py.224: train: epoch 108 | batch 830/2304 [I 2022-10-28 05:24:07] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:42 [I 2022-10-28 05:24:07] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00257 [I 2022-10-28 05:24:07] ...trap/engines/engine.py.230: loss: 0.16435 [I 2022-10-28 05:24:07] ...optimizers/trijoint.py.151: total_norm: 30.524784 [I 2022-10-28 05:24:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 840/2304 [I 2022-10-28 05:24:12] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:36 [I 2022-10-28 05:24:12] ...trap/engines/engine.py.229: process: 0.55007 | load: 0.00247 [I 2022-10-28 05:24:12] ...trap/engines/engine.py.230: loss: 0.11649 [I 2022-10-28 05:24:12] ...optimizers/trijoint.py.151: total_norm: 17.237980 [I 2022-10-28 05:24:18] ...trap/engines/engine.py.224: train: epoch 108 | batch 850/2304 [I 2022-10-28 05:24:18] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:32 [I 2022-10-28 05:24:18] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00261 [I 2022-10-28 05:24:18] ...trap/engines/engine.py.230: loss: 0.10077 [I 2022-10-28 05:24:18] ...optimizers/trijoint.py.151: total_norm: 11.963359 [I 2022-10-28 05:24:23] ...trap/engines/engine.py.224: train: epoch 108 | batch 860/2304 [I 2022-10-28 05:24:23] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:38 [I 2022-10-28 05:24:23] ...trap/engines/engine.py.229: process: 0.58420 | load: 0.00271 [I 2022-10-28 05:24:23] ...trap/engines/engine.py.230: loss: 0.13436 [I 2022-10-28 05:24:24] ...optimizers/trijoint.py.151: total_norm: 12.672464 [I 2022-10-28 05:24:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 870/2304 [I 2022-10-28 05:24:29] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:28 [I 2022-10-28 05:24:29] ...trap/engines/engine.py.229: process: 0.56371 | load: 0.00244 [I 2022-10-28 05:24:29] ...trap/engines/engine.py.230: loss: 0.12752 [I 2022-10-28 05:24:29] ...optimizers/trijoint.py.151: total_norm: 10.130492 [I 2022-10-28 05:24:35] ...trap/engines/engine.py.224: train: epoch 108 | batch 880/2304 [I 2022-10-28 05:24:35] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:16 [I 2022-10-28 05:24:35] ...trap/engines/engine.py.229: process: 0.56639 | load: 0.00285 [I 2022-10-28 05:24:35] ...trap/engines/engine.py.230: loss: 0.09057 [I 2022-10-28 05:24:35] ...optimizers/trijoint.py.151: total_norm: 11.588321 [I 2022-10-28 05:24:40] ...trap/engines/engine.py.224: train: epoch 108 | batch 890/2304 [I 2022-10-28 05:24:40] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:12 [I 2022-10-28 05:24:40] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00248 [I 2022-10-28 05:24:40] ...trap/engines/engine.py.230: loss: 0.14888 [I 2022-10-28 05:24:40] ...optimizers/trijoint.py.151: total_norm: 102.147018 [I 2022-10-28 05:24:46] ...trap/engines/engine.py.224: train: epoch 108 | batch 900/2304 [I 2022-10-28 05:24:46] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:12:58 [I 2022-10-28 05:24:46] ...trap/engines/engine.py.229: process: 0.54529 | load: 0.00247 [I 2022-10-28 05:24:46] ...trap/engines/engine.py.230: loss: 0.05644 [I 2022-10-28 05:24:46] ...optimizers/trijoint.py.151: total_norm: 19.411776 [I 2022-10-28 05:24:52] ...trap/engines/engine.py.224: train: epoch 108 | batch 910/2304 [I 2022-10-28 05:24:52] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:56 [I 2022-10-28 05:24:52] ...trap/engines/engine.py.229: process: 0.55008 | load: 0.00249 [I 2022-10-28 05:24:52] ...trap/engines/engine.py.230: loss: 0.17077 [I 2022-10-28 05:24:52] ...optimizers/trijoint.py.151: total_norm: 21.267408 [I 2022-10-28 05:24:57] ...trap/engines/engine.py.224: train: epoch 108 | batch 920/2304 [I 2022-10-28 05:24:57] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:53 [I 2022-10-28 05:24:57] ...trap/engines/engine.py.229: process: 0.57048 | load: 0.00243 [I 2022-10-28 05:24:57] ...trap/engines/engine.py.230: loss: 0.09315 [I 2022-10-28 05:24:57] ...optimizers/trijoint.py.151: total_norm: 21.418858 [I 2022-10-28 05:25:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 930/2304 [I 2022-10-28 05:25:03] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:43 [I 2022-10-28 05:25:03] ...trap/engines/engine.py.229: process: 0.55507 | load: 0.00242 [I 2022-10-28 05:25:03] ...trap/engines/engine.py.230: loss: 0.12733 [I 2022-10-28 05:25:03] ...optimizers/trijoint.py.151: total_norm: 13.616665 [I 2022-10-28 05:25:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 940/2304 [I 2022-10-28 05:25:08] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:41 [I 2022-10-28 05:25:08] ...trap/engines/engine.py.229: process: 0.56487 | load: 0.00245 [I 2022-10-28 05:25:08] ...trap/engines/engine.py.230: loss: 0.12221 [I 2022-10-28 05:25:08] ...optimizers/trijoint.py.151: total_norm: 13.214824 [I 2022-10-28 05:25:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 950/2304 [I 2022-10-28 05:25:14] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:38 [I 2022-10-28 05:25:14] ...trap/engines/engine.py.229: process: 0.55713 | load: 0.00300 [I 2022-10-28 05:25:14] ...trap/engines/engine.py.230: loss: 0.14232 [I 2022-10-28 05:25:14] ...optimizers/trijoint.py.151: total_norm: 27.503017 [I 2022-10-28 05:25:20] ...trap/engines/engine.py.224: train: epoch 108 | batch 960/2304 [I 2022-10-28 05:25:20] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:46 [I 2022-10-28 05:25:20] ...trap/engines/engine.py.229: process: 0.57447 | load: 0.00251 [I 2022-10-28 05:25:20] ...trap/engines/engine.py.230: loss: 0.13788 [I 2022-10-28 05:25:20] ...optimizers/trijoint.py.151: total_norm: 20.843285 [I 2022-10-28 05:25:25] ...trap/engines/engine.py.224: train: epoch 108 | batch 970/2304 [I 2022-10-28 05:25:25] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:27 [I 2022-10-28 05:25:25] ...trap/engines/engine.py.229: process: 0.54655 | load: 0.00244 [I 2022-10-28 05:25:25] ...trap/engines/engine.py.230: loss: 0.15188 [I 2022-10-28 05:25:25] ...optimizers/trijoint.py.151: total_norm: 7.904698 [I 2022-10-28 05:25:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 980/2304 [I 2022-10-28 05:25:31] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:21 [I 2022-10-28 05:25:31] ...trap/engines/engine.py.229: process: 0.55704 | load: 0.00246 [I 2022-10-28 05:25:31] ...trap/engines/engine.py.230: loss: 0.12225 [I 2022-10-28 05:25:31] ...optimizers/trijoint.py.151: total_norm: 19.363075 [I 2022-10-28 05:25:37] ...trap/engines/engine.py.224: train: epoch 108 | batch 990/2304 [I 2022-10-28 05:25:37] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:19 [I 2022-10-28 05:25:37] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00249 [I 2022-10-28 05:25:37] ...trap/engines/engine.py.230: loss: 0.22334 [I 2022-10-28 05:25:37] ...optimizers/trijoint.py.151: total_norm: 10.096515 [I 2022-10-28 05:25:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 1000/2304 [I 2022-10-28 05:25:42] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:17 [I 2022-10-28 05:25:42] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00244 [I 2022-10-28 05:25:42] ...trap/engines/engine.py.230: loss: 0.15328 [I 2022-10-28 05:25:42] ...optimizers/trijoint.py.151: total_norm: 21.813740 [I 2022-10-28 05:25:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 1010/2304 [I 2022-10-28 05:25:48] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:04 [I 2022-10-28 05:25:48] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00250 [I 2022-10-28 05:25:48] ...trap/engines/engine.py.230: loss: 0.12124 [I 2022-10-28 05:25:48] ...optimizers/trijoint.py.151: total_norm: 38.394176 [I 2022-10-28 05:25:53] ...trap/engines/engine.py.224: train: epoch 108 | batch 1020/2304 [I 2022-10-28 05:25:53] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:59 [I 2022-10-28 05:25:53] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00250 [I 2022-10-28 05:25:53] ...trap/engines/engine.py.230: loss: 0.10707 [I 2022-10-28 05:25:53] ...optimizers/trijoint.py.151: total_norm: 12.223951 [I 2022-10-28 05:25:59] ...trap/engines/engine.py.224: train: epoch 108 | batch 1030/2304 [I 2022-10-28 05:25:59] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:11:56 [I 2022-10-28 05:25:59] ...trap/engines/engine.py.229: process: 0.56370 | load: 0.00264 [I 2022-10-28 05:25:59] ...trap/engines/engine.py.230: loss: 0.09502 [I 2022-10-28 05:25:59] ...optimizers/trijoint.py.151: total_norm: 7.853753 [I 2022-10-28 05:26:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 1040/2304 [I 2022-10-28 05:26:05] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:47 [I 2022-10-28 05:26:05] ...trap/engines/engine.py.229: process: 0.56082 | load: 0.00245 [I 2022-10-28 05:26:05] ...trap/engines/engine.py.230: loss: 0.10011 [I 2022-10-28 05:26:05] ...optimizers/trijoint.py.151: total_norm: 10.385731 [I 2022-10-28 05:26:10] ...trap/engines/engine.py.224: train: epoch 108 | batch 1050/2304 [I 2022-10-28 05:26:10] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:43 [I 2022-10-28 05:26:10] ...trap/engines/engine.py.229: process: 0.55937 | load: 0.00247 [I 2022-10-28 05:26:10] ...trap/engines/engine.py.230: loss: 0.13364 [I 2022-10-28 05:26:10] ...optimizers/trijoint.py.151: total_norm: 12.693186 [I 2022-10-28 05:26:16] ...trap/engines/engine.py.224: train: epoch 108 | batch 1060/2304 [I 2022-10-28 05:26:16] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:42 [I 2022-10-28 05:26:16] ...trap/engines/engine.py.229: process: 0.57067 | load: 0.00248 [I 2022-10-28 05:26:16] ...trap/engines/engine.py.230: loss: 0.10173 [I 2022-10-28 05:26:16] ...optimizers/trijoint.py.151: total_norm: 23.491428 [I 2022-10-28 05:26:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 1070/2304 [I 2022-10-28 05:26:22] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:35 [I 2022-10-28 05:26:22] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00245 [I 2022-10-28 05:26:22] ...trap/engines/engine.py.230: loss: 0.10118 [I 2022-10-28 05:26:22] ...optimizers/trijoint.py.151: total_norm: 19.195202 [I 2022-10-28 05:26:27] ...trap/engines/engine.py.224: train: epoch 108 | batch 1080/2304 [I 2022-10-28 05:26:27] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:25 [I 2022-10-28 05:26:27] ...trap/engines/engine.py.229: process: 0.56580 | load: 0.00244 [I 2022-10-28 05:26:27] ...trap/engines/engine.py.230: loss: 0.10980 [I 2022-10-28 05:26:27] ...optimizers/trijoint.py.151: total_norm: 9.383526 [I 2022-10-28 05:26:33] ...trap/engines/engine.py.224: train: epoch 108 | batch 1090/2304 [I 2022-10-28 05:26:33] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:25 [I 2022-10-28 05:26:33] ...trap/engines/engine.py.229: process: 0.56642 | load: 0.00248 [I 2022-10-28 05:26:33] ...trap/engines/engine.py.230: loss: 0.15112 [I 2022-10-28 05:26:33] ...optimizers/trijoint.py.151: total_norm: 17.911989 [I 2022-10-28 05:26:38] ...trap/engines/engine.py.224: train: epoch 108 | batch 1100/2304 [I 2022-10-28 05:26:38] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:09 [I 2022-10-28 05:26:38] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00242 [I 2022-10-28 05:26:38] ...trap/engines/engine.py.230: loss: 0.13897 [I 2022-10-28 05:26:38] ...optimizers/trijoint.py.151: total_norm: 10.345340 [I 2022-10-28 05:26:44] ...trap/engines/engine.py.224: train: epoch 108 | batch 1110/2304 [I 2022-10-28 05:26:44] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:21 [I 2022-10-28 05:26:44] ...trap/engines/engine.py.229: process: 0.56902 | load: 0.00274 [I 2022-10-28 05:26:44] ...trap/engines/engine.py.230: loss: 0.15417 [I 2022-10-28 05:26:44] ...optimizers/trijoint.py.151: total_norm: 14.257174 [I 2022-10-28 05:26:50] ...trap/engines/engine.py.224: train: epoch 108 | batch 1120/2304 [I 2022-10-28 05:26:50] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:11 [I 2022-10-28 05:26:50] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00246 [I 2022-10-28 05:26:50] ...trap/engines/engine.py.230: loss: 0.14090 [I 2022-10-28 05:26:50] ...optimizers/trijoint.py.151: total_norm: 28.300961 [I 2022-10-28 05:26:56] ...trap/engines/engine.py.224: train: epoch 108 | batch 1130/2304 [I 2022-10-28 05:26:56] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:11:03 [I 2022-10-28 05:26:56] ...trap/engines/engine.py.229: process: 0.56967 | load: 0.00243 [I 2022-10-28 05:26:56] ...trap/engines/engine.py.230: loss: 0.09152 [I 2022-10-28 05:26:56] ...optimizers/trijoint.py.151: total_norm: 14.562014 [I 2022-10-28 05:27:01] ...trap/engines/engine.py.224: train: epoch 108 | batch 1140/2304 [I 2022-10-28 05:27:01] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:55 [I 2022-10-28 05:27:01] ...trap/engines/engine.py.229: process: 0.56649 | load: 0.00253 [I 2022-10-28 05:27:01] ...trap/engines/engine.py.230: loss: 0.11882 [I 2022-10-28 05:27:01] ...optimizers/trijoint.py.151: total_norm: 22.290026 [I 2022-10-28 05:27:07] ...trap/engines/engine.py.224: train: epoch 108 | batch 1150/2304 [I 2022-10-28 05:27:07] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:52 [I 2022-10-28 05:27:07] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00249 [I 2022-10-28 05:27:07] ...trap/engines/engine.py.230: loss: 0.14544 [I 2022-10-28 05:27:07] ...optimizers/trijoint.py.151: total_norm: 18.222654 [I 2022-10-28 05:27:13] ...trap/engines/engine.py.224: train: epoch 108 | batch 1160/2304 [I 2022-10-28 05:27:13] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:40 [I 2022-10-28 05:27:13] ...trap/engines/engine.py.229: process: 0.55030 | load: 0.00246 [I 2022-10-28 05:27:13] ...trap/engines/engine.py.230: loss: 0.10739 [I 2022-10-28 05:27:13] ...optimizers/trijoint.py.151: total_norm: 13.746758 [I 2022-10-28 05:27:18] ...trap/engines/engine.py.224: train: epoch 108 | batch 1170/2304 [I 2022-10-28 05:27:18] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:34 [I 2022-10-28 05:27:18] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00245 [I 2022-10-28 05:27:18] ...trap/engines/engine.py.230: loss: 0.18565 [I 2022-10-28 05:27:18] ...optimizers/trijoint.py.151: total_norm: 17.736076 [I 2022-10-28 05:27:24] ...trap/engines/engine.py.224: train: epoch 108 | batch 1180/2304 [I 2022-10-28 05:27:24] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:31 [I 2022-10-28 05:27:24] ...trap/engines/engine.py.229: process: 0.55836 | load: 0.00244 [I 2022-10-28 05:27:24] ...trap/engines/engine.py.230: loss: 0.13748 [I 2022-10-28 05:27:24] ...optimizers/trijoint.py.151: total_norm: 17.132132 [I 2022-10-28 05:27:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 1190/2304 [I 2022-10-28 05:27:29] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:20 [I 2022-10-28 05:27:29] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00243 [I 2022-10-28 05:27:29] ...trap/engines/engine.py.230: loss: 0.16666 [I 2022-10-28 05:27:29] ...optimizers/trijoint.py.151: total_norm: 36.501610 [I 2022-10-28 05:27:35] ...trap/engines/engine.py.224: train: epoch 108 | batch 1200/2304 [I 2022-10-28 05:27:35] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:26 [I 2022-10-28 05:27:35] ...trap/engines/engine.py.229: process: 0.56025 | load: 0.00242 [I 2022-10-28 05:27:35] ...trap/engines/engine.py.230: loss: 0.11262 [I 2022-10-28 05:27:35] ...optimizers/trijoint.py.151: total_norm: 16.167236 [I 2022-10-28 05:27:41] ...trap/engines/engine.py.224: train: epoch 108 | batch 1210/2304 [I 2022-10-28 05:27:41] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:15 [I 2022-10-28 05:27:41] ...trap/engines/engine.py.229: process: 0.57113 | load: 0.00244 [I 2022-10-28 05:27:41] ...trap/engines/engine.py.230: loss: 0.08438 [I 2022-10-28 05:27:41] ...optimizers/trijoint.py.151: total_norm: 11.663528 [I 2022-10-28 05:27:46] ...trap/engines/engine.py.224: train: epoch 108 | batch 1220/2304 [I 2022-10-28 05:27:46] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:07 [I 2022-10-28 05:27:46] ...trap/engines/engine.py.229: process: 0.55354 | load: 0.00249 [I 2022-10-28 05:27:46] ...trap/engines/engine.py.230: loss: 0.11033 [I 2022-10-28 05:27:46] ...optimizers/trijoint.py.151: total_norm: 12.840185 [I 2022-10-28 05:27:52] ...trap/engines/engine.py.224: train: epoch 108 | batch 1230/2304 [I 2022-10-28 05:27:52] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:09:58 [I 2022-10-28 05:27:52] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00248 [I 2022-10-28 05:27:52] ...trap/engines/engine.py.230: loss: 0.09888 [I 2022-10-28 05:27:52] ...optimizers/trijoint.py.151: total_norm: 11.952750 [I 2022-10-28 05:27:58] ...trap/engines/engine.py.224: train: epoch 108 | batch 1240/2304 [I 2022-10-28 05:27:58] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:10:02 [I 2022-10-28 05:27:58] ...trap/engines/engine.py.229: process: 0.56648 | load: 0.00251 [I 2022-10-28 05:27:58] ...trap/engines/engine.py.230: loss: 0.12904 [I 2022-10-28 05:27:58] ...optimizers/trijoint.py.151: total_norm: 11.540631 [I 2022-10-28 05:28:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 1250/2304 [I 2022-10-28 05:28:03] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:09:47 [I 2022-10-28 05:28:03] ...trap/engines/engine.py.229: process: 0.53756 | load: 0.00250 [I 2022-10-28 05:28:03] ...trap/engines/engine.py.230: loss: 0.14573 [I 2022-10-28 05:28:03] ...optimizers/trijoint.py.151: total_norm: 70.566910 [I 2022-10-28 05:28:09] ...trap/engines/engine.py.224: train: epoch 108 | batch 1260/2304 [I 2022-10-28 05:28:09] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:47 [I 2022-10-28 05:28:09] ...trap/engines/engine.py.229: process: 0.55720 | load: 0.00251 [I 2022-10-28 05:28:09] ...trap/engines/engine.py.230: loss: 0.11215 [I 2022-10-28 05:28:09] ...optimizers/trijoint.py.151: total_norm: 24.736565 [I 2022-10-28 05:28:15] ...trap/engines/engine.py.224: train: epoch 108 | batch 1270/2304 [I 2022-10-28 05:28:15] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:42 [I 2022-10-28 05:28:15] ...trap/engines/engine.py.229: process: 0.56206 | load: 0.00251 [I 2022-10-28 05:28:15] ...trap/engines/engine.py.230: loss: 0.09985 [I 2022-10-28 05:28:15] ...optimizers/trijoint.py.151: total_norm: 10.677461 [I 2022-10-28 05:28:20] ...trap/engines/engine.py.224: train: epoch 108 | batch 1280/2304 [I 2022-10-28 05:28:20] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:37 [I 2022-10-28 05:28:20] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00246 [I 2022-10-28 05:28:20] ...trap/engines/engine.py.230: loss: 0.14376 [I 2022-10-28 05:28:20] ...optimizers/trijoint.py.151: total_norm: 30.306091 [I 2022-10-28 05:28:26] ...trap/engines/engine.py.224: train: epoch 108 | batch 1290/2304 [I 2022-10-28 05:28:26] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:29 [I 2022-10-28 05:28:26] ...trap/engines/engine.py.229: process: 0.55804 | load: 0.00298 [I 2022-10-28 05:28:26] ...trap/engines/engine.py.230: loss: 0.15874 [I 2022-10-28 05:28:26] ...optimizers/trijoint.py.151: total_norm: 59.753242 [I 2022-10-28 05:28:32] ...trap/engines/engine.py.224: train: epoch 108 | batch 1300/2304 [I 2022-10-28 05:28:32] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:23 [I 2022-10-28 05:28:32] ...trap/engines/engine.py.229: process: 0.54772 | load: 0.00244 [I 2022-10-28 05:28:32] ...trap/engines/engine.py.230: loss: 0.10359 [I 2022-10-28 05:28:32] ...optimizers/trijoint.py.151: total_norm: 10.742240 [I 2022-10-28 05:28:37] ...trap/engines/engine.py.224: train: epoch 108 | batch 1310/2304 [I 2022-10-28 05:28:37] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:17 [I 2022-10-28 05:28:37] ...trap/engines/engine.py.229: process: 0.55628 | load: 0.00246 [I 2022-10-28 05:28:37] ...trap/engines/engine.py.230: loss: 0.12255 [I 2022-10-28 05:28:37] ...optimizers/trijoint.py.151: total_norm: 13.239238 [I 2022-10-28 05:28:43] ...trap/engines/engine.py.224: train: epoch 108 | batch 1320/2304 [I 2022-10-28 05:28:43] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:13 [I 2022-10-28 05:28:43] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00243 [I 2022-10-28 05:28:43] ...trap/engines/engine.py.230: loss: 0.14987 [I 2022-10-28 05:28:43] ...optimizers/trijoint.py.151: total_norm: 11.178631 [I 2022-10-28 05:28:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 1330/2304 [I 2022-10-28 05:28:48] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:05 [I 2022-10-28 05:28:48] ...trap/engines/engine.py.229: process: 0.55596 | load: 0.00255 [I 2022-10-28 05:28:48] ...trap/engines/engine.py.230: loss: 0.13314 [I 2022-10-28 05:28:48] ...optimizers/trijoint.py.151: total_norm: 18.038893 [I 2022-10-28 05:28:54] ...trap/engines/engine.py.224: train: epoch 108 | batch 1340/2304 [I 2022-10-28 05:28:54] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:01 [I 2022-10-28 05:28:54] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00247 [I 2022-10-28 05:28:54] ...trap/engines/engine.py.230: loss: 0.12480 [I 2022-10-28 05:28:54] ...optimizers/trijoint.py.151: total_norm: 22.170628 [I 2022-10-28 05:29:00] ...trap/engines/engine.py.224: train: epoch 108 | batch 1350/2304 [I 2022-10-28 05:29:00] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:08:53 [I 2022-10-28 05:29:00] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00266 [I 2022-10-28 05:29:00] ...trap/engines/engine.py.230: loss: 0.11512 [I 2022-10-28 05:29:00] ...optimizers/trijoint.py.151: total_norm: 14.605727 [I 2022-10-28 05:29:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 1360/2304 [I 2022-10-28 05:29:05] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:08:49 [I 2022-10-28 05:29:05] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00246 [I 2022-10-28 05:29:05] ...trap/engines/engine.py.230: loss: 0.11457 [I 2022-10-28 05:29:05] ...optimizers/trijoint.py.151: total_norm: 15.783707 [I 2022-10-28 05:29:11] ...trap/engines/engine.py.224: train: epoch 108 | batch 1370/2304 [I 2022-10-28 05:29:11] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:40 [I 2022-10-28 05:29:11] ...trap/engines/engine.py.229: process: 0.55531 | load: 0.00247 [I 2022-10-28 05:29:11] ...trap/engines/engine.py.230: loss: 0.16372 [I 2022-10-28 05:29:11] ...optimizers/trijoint.py.151: total_norm: 24.686829 [I 2022-10-28 05:29:17] ...trap/engines/engine.py.224: train: epoch 108 | batch 1380/2304 [I 2022-10-28 05:29:17] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:51 [I 2022-10-28 05:29:17] ...trap/engines/engine.py.229: process: 0.63563 | load: 0.00244 [I 2022-10-28 05:29:17] ...trap/engines/engine.py.230: loss: 0.06433 [I 2022-10-28 05:29:17] ...optimizers/trijoint.py.151: total_norm: 12.461996 [I 2022-10-28 05:29:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 1390/2304 [I 2022-10-28 05:29:22] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:32 [I 2022-10-28 05:29:22] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00244 [I 2022-10-28 05:29:22] ...trap/engines/engine.py.230: loss: 0.13168 [I 2022-10-28 05:29:22] ...optimizers/trijoint.py.151: total_norm: 10.168245 [I 2022-10-28 05:29:28] ...trap/engines/engine.py.224: train: epoch 108 | batch 1400/2304 [I 2022-10-28 05:29:28] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:29 [I 2022-10-28 05:29:28] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00252 [I 2022-10-28 05:29:28] ...trap/engines/engine.py.230: loss: 0.10643 [I 2022-10-28 05:29:28] ...optimizers/trijoint.py.151: total_norm: 8.958410 [I 2022-10-28 05:29:34] ...trap/engines/engine.py.224: train: epoch 108 | batch 1410/2304 [I 2022-10-28 05:29:34] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:20 [I 2022-10-28 05:29:34] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00254 [I 2022-10-28 05:29:34] ...trap/engines/engine.py.230: loss: 0.13365 [I 2022-10-28 05:29:34] ...optimizers/trijoint.py.151: total_norm: 24.048759 [I 2022-10-28 05:29:39] ...trap/engines/engine.py.224: train: epoch 108 | batch 1420/2304 [I 2022-10-28 05:29:39] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:14 [I 2022-10-28 05:29:39] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00246 [I 2022-10-28 05:29:39] ...trap/engines/engine.py.230: loss: 0.11696 [I 2022-10-28 05:29:39] ...optimizers/trijoint.py.151: total_norm: 18.053129 [I 2022-10-28 05:29:45] ...trap/engines/engine.py.224: train: epoch 108 | batch 1430/2304 [I 2022-10-28 05:29:45] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:12 [I 2022-10-28 05:29:45] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00244 [I 2022-10-28 05:29:45] ...trap/engines/engine.py.230: loss: 0.10772 [I 2022-10-28 05:29:45] ...optimizers/trijoint.py.151: total_norm: 11.520511 [I 2022-10-28 05:29:50] ...trap/engines/engine.py.224: train: epoch 108 | batch 1440/2304 [I 2022-10-28 05:29:50] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:03 [I 2022-10-28 05:29:50] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00254 [I 2022-10-28 05:29:50] ...trap/engines/engine.py.230: loss: 0.14948 [I 2022-10-28 05:29:50] ...optimizers/trijoint.py.151: total_norm: 21.070131 [I 2022-10-28 05:29:56] ...trap/engines/engine.py.224: train: epoch 108 | batch 1450/2304 [I 2022-10-28 05:29:56] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:07:56 [I 2022-10-28 05:29:56] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00245 [I 2022-10-28 05:29:56] ...trap/engines/engine.py.230: loss: 0.13745 [I 2022-10-28 05:29:56] ...optimizers/trijoint.py.151: total_norm: 26.942184 [I 2022-10-28 05:30:02] ...trap/engines/engine.py.224: train: epoch 108 | batch 1460/2304 [I 2022-10-28 05:30:02] ...trap/engines/engine.py.228: elapsed: 0:13:42 | left: 0:07:55 [I 2022-10-28 05:30:02] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00244 [I 2022-10-28 05:30:02] ...trap/engines/engine.py.230: loss: 0.12716 [I 2022-10-28 05:30:02] ...optimizers/trijoint.py.151: total_norm: 23.532217 [I 2022-10-28 05:30:07] ...trap/engines/engine.py.224: train: epoch 108 | batch 1470/2304 [I 2022-10-28 05:30:07] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:48 [I 2022-10-28 05:30:07] ...trap/engines/engine.py.229: process: 0.56104 | load: 0.00243 [I 2022-10-28 05:30:07] ...trap/engines/engine.py.230: loss: 0.09381 [I 2022-10-28 05:30:07] ...optimizers/trijoint.py.151: total_norm: 29.139635 [I 2022-10-28 05:30:13] ...trap/engines/engine.py.224: train: epoch 108 | batch 1480/2304 [I 2022-10-28 05:30:13] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:39 [I 2022-10-28 05:30:13] ...trap/engines/engine.py.229: process: 0.56970 | load: 0.00244 [I 2022-10-28 05:30:13] ...trap/engines/engine.py.230: loss: 0.15326 [I 2022-10-28 05:30:13] ...optimizers/trijoint.py.151: total_norm: 48.600597 [I 2022-10-28 05:30:18] ...trap/engines/engine.py.224: train: epoch 108 | batch 1490/2304 [I 2022-10-28 05:30:18] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:38 [I 2022-10-28 05:30:18] ...trap/engines/engine.py.229: process: 0.57989 | load: 0.00244 [I 2022-10-28 05:30:18] ...trap/engines/engine.py.230: loss: 0.08373 [I 2022-10-28 05:30:18] ...optimizers/trijoint.py.151: total_norm: 9.173896 [I 2022-10-28 05:30:24] ...trap/engines/engine.py.224: train: epoch 108 | batch 1500/2304 [I 2022-10-28 05:30:24] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:44 [I 2022-10-28 05:30:24] ...trap/engines/engine.py.229: process: 0.56369 | load: 0.00247 [I 2022-10-28 05:30:24] ...trap/engines/engine.py.230: loss: 0.16820 [I 2022-10-28 05:30:24] ...optimizers/trijoint.py.151: total_norm: 18.206617 [I 2022-10-28 05:30:30] ...trap/engines/engine.py.224: train: epoch 108 | batch 1510/2304 [I 2022-10-28 05:30:30] ...trap/engines/engine.py.228: elapsed: 0:14:10 | left: 0:07:26 [I 2022-10-28 05:30:30] ...trap/engines/engine.py.229: process: 0.55147 | load: 0.00247 [I 2022-10-28 05:30:30] ...trap/engines/engine.py.230: loss: 0.09012 [I 2022-10-28 05:30:30] ...optimizers/trijoint.py.151: total_norm: 8.241134 [I 2022-10-28 05:30:35] ...trap/engines/engine.py.224: train: epoch 108 | batch 1520/2304 [I 2022-10-28 05:30:35] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:17 [I 2022-10-28 05:30:35] ...trap/engines/engine.py.229: process: 0.54506 | load: 0.00249 [I 2022-10-28 05:30:35] ...trap/engines/engine.py.230: loss: 0.08529 [I 2022-10-28 05:30:35] ...optimizers/trijoint.py.151: total_norm: 17.717707 [I 2022-10-28 05:30:41] ...trap/engines/engine.py.224: train: epoch 108 | batch 1530/2304 [I 2022-10-28 05:30:41] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:14 [I 2022-10-28 05:30:41] ...trap/engines/engine.py.229: process: 0.56826 | load: 0.00247 [I 2022-10-28 05:30:41] ...trap/engines/engine.py.230: loss: 0.21622 [I 2022-10-28 05:30:41] ...optimizers/trijoint.py.151: total_norm: 13.017808 [I 2022-10-28 05:30:47] ...trap/engines/engine.py.224: train: epoch 108 | batch 1540/2304 [I 2022-10-28 05:30:47] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:07:04 [I 2022-10-28 05:30:47] ...trap/engines/engine.py.229: process: 0.54815 | load: 0.00247 [I 2022-10-28 05:30:47] ...trap/engines/engine.py.230: loss: 0.13133 [I 2022-10-28 05:30:47] ...optimizers/trijoint.py.151: total_norm: 9.643542 [I 2022-10-28 05:30:52] ...trap/engines/engine.py.224: train: epoch 108 | batch 1550/2304 [I 2022-10-28 05:30:52] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:00 [I 2022-10-28 05:30:52] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00244 [I 2022-10-28 05:30:52] ...trap/engines/engine.py.230: loss: 0.23057 [I 2022-10-28 05:30:52] ...optimizers/trijoint.py.151: total_norm: 39.189713 [I 2022-10-28 05:30:58] ...trap/engines/engine.py.224: train: epoch 108 | batch 1560/2304 [I 2022-10-28 05:30:58] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:06:54 [I 2022-10-28 05:30:58] ...trap/engines/engine.py.229: process: 0.55706 | load: 0.00242 [I 2022-10-28 05:30:58] ...trap/engines/engine.py.230: loss: 0.11018 [I 2022-10-28 05:30:58] ...optimizers/trijoint.py.151: total_norm: 15.992768 [I 2022-10-28 05:31:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 1570/2304 [I 2022-10-28 05:31:03] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:06:47 [I 2022-10-28 05:31:03] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00250 [I 2022-10-28 05:31:03] ...trap/engines/engine.py.230: loss: 0.11218 [I 2022-10-28 05:31:03] ...optimizers/trijoint.py.151: total_norm: 17.960571 [I 2022-10-28 05:31:09] ...trap/engines/engine.py.224: train: epoch 108 | batch 1580/2304 [I 2022-10-28 05:31:09] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:41 [I 2022-10-28 05:31:09] ...trap/engines/engine.py.229: process: 0.54665 | load: 0.00245 [I 2022-10-28 05:31:09] ...trap/engines/engine.py.230: loss: 0.11934 [I 2022-10-28 05:31:09] ...optimizers/trijoint.py.151: total_norm: 18.422812 [I 2022-10-28 05:31:15] ...trap/engines/engine.py.224: train: epoch 108 | batch 1590/2304 [I 2022-10-28 05:31:15] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:40 [I 2022-10-28 05:31:15] ...trap/engines/engine.py.229: process: 0.56370 | load: 0.00245 [I 2022-10-28 05:31:15] ...trap/engines/engine.py.230: loss: 0.13334 [I 2022-10-28 05:31:15] ...optimizers/trijoint.py.151: total_norm: 15.276450 [I 2022-10-28 05:31:20] ...trap/engines/engine.py.224: train: epoch 108 | batch 1600/2304 [I 2022-10-28 05:31:20] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:32 [I 2022-10-28 05:31:20] ...trap/engines/engine.py.229: process: 0.54689 | load: 0.00252 [I 2022-10-28 05:31:20] ...trap/engines/engine.py.230: loss: 0.17140 [I 2022-10-28 05:31:20] ...optimizers/trijoint.py.151: total_norm: 18.584976 [I 2022-10-28 05:31:26] ...trap/engines/engine.py.224: train: epoch 108 | batch 1610/2304 [I 2022-10-28 05:31:26] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:39 [I 2022-10-28 05:31:26] ...trap/engines/engine.py.229: process: 0.57926 | load: 0.00249 [I 2022-10-28 05:31:26] ...trap/engines/engine.py.230: loss: 0.11002 [I 2022-10-28 05:31:26] ...optimizers/trijoint.py.151: total_norm: 28.295244 [I 2022-10-28 05:31:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 1620/2304 [I 2022-10-28 05:31:31] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:21 [I 2022-10-28 05:31:31] ...trap/engines/engine.py.229: process: 0.55935 | load: 0.00247 [I 2022-10-28 05:31:31] ...trap/engines/engine.py.230: loss: 0.11729 [I 2022-10-28 05:31:31] ...optimizers/trijoint.py.151: total_norm: 26.509752 [I 2022-10-28 05:31:37] ...trap/engines/engine.py.224: train: epoch 108 | batch 1630/2304 [I 2022-10-28 05:31:37] ...trap/engines/engine.py.228: elapsed: 0:15:18 | left: 0:06:18 [I 2022-10-28 05:31:37] ...trap/engines/engine.py.229: process: 0.56696 | load: 0.00249 [I 2022-10-28 05:31:37] ...trap/engines/engine.py.230: loss: 0.10599 [I 2022-10-28 05:31:37] ...optimizers/trijoint.py.151: total_norm: 9.671401 [I 2022-10-28 05:31:43] ...trap/engines/engine.py.224: train: epoch 108 | batch 1640/2304 [I 2022-10-28 05:31:43] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:09 [I 2022-10-28 05:31:43] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00244 [I 2022-10-28 05:31:43] ...trap/engines/engine.py.230: loss: 0.13778 [I 2022-10-28 05:31:43] ...optimizers/trijoint.py.151: total_norm: 37.103573 [I 2022-10-28 05:31:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 1650/2304 [I 2022-10-28 05:31:48] ...trap/engines/engine.py.228: elapsed: 0:15:29 | left: 0:06:09 [I 2022-10-28 05:31:48] ...trap/engines/engine.py.229: process: 0.57787 | load: 0.00248 [I 2022-10-28 05:31:48] ...trap/engines/engine.py.230: loss: 0.17080 [I 2022-10-28 05:31:48] ...optimizers/trijoint.py.151: total_norm: 20.034107 [I 2022-10-28 05:31:54] ...trap/engines/engine.py.224: train: epoch 108 | batch 1660/2304 [I 2022-10-28 05:31:54] ...trap/engines/engine.py.228: elapsed: 0:15:35 | left: 0:06:01 [I 2022-10-28 05:31:54] ...trap/engines/engine.py.229: process: 0.54834 | load: 0.00245 [I 2022-10-28 05:31:54] ...trap/engines/engine.py.230: loss: 0.11711 [I 2022-10-28 05:31:54] ...optimizers/trijoint.py.151: total_norm: 20.606653 [I 2022-10-28 05:32:00] ...trap/engines/engine.py.224: train: epoch 108 | batch 1670/2304 [I 2022-10-28 05:32:00] ...trap/engines/engine.py.228: elapsed: 0:15:40 | left: 0:05:55 [I 2022-10-28 05:32:00] ...trap/engines/engine.py.229: process: 0.56681 | load: 0.00250 [I 2022-10-28 05:32:00] ...trap/engines/engine.py.230: loss: 0.18208 [I 2022-10-28 05:32:00] ...optimizers/trijoint.py.151: total_norm: 19.923590 [I 2022-10-28 05:32:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 1680/2304 [I 2022-10-28 05:32:05] ...trap/engines/engine.py.228: elapsed: 0:15:46 | left: 0:05:49 [I 2022-10-28 05:32:05] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00241 [I 2022-10-28 05:32:05] ...trap/engines/engine.py.230: loss: 0.10453 [I 2022-10-28 05:32:05] ...optimizers/trijoint.py.151: total_norm: 16.318945 [I 2022-10-28 05:32:11] ...trap/engines/engine.py.224: train: epoch 108 | batch 1690/2304 [I 2022-10-28 05:32:11] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:42 [I 2022-10-28 05:32:11] ...trap/engines/engine.py.229: process: 0.55931 | load: 0.00245 [I 2022-10-28 05:32:11] ...trap/engines/engine.py.230: loss: 0.17467 [I 2022-10-28 05:32:11] ...optimizers/trijoint.py.151: total_norm: 14.187788 [I 2022-10-28 05:32:16] ...trap/engines/engine.py.224: train: epoch 108 | batch 1700/2304 [I 2022-10-28 05:32:16] ...trap/engines/engine.py.228: elapsed: 0:15:57 | left: 0:05:38 [I 2022-10-28 05:32:16] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00242 [I 2022-10-28 05:32:16] ...trap/engines/engine.py.230: loss: 0.13262 [I 2022-10-28 05:32:16] ...optimizers/trijoint.py.151: total_norm: 26.092941 [I 2022-10-28 05:32:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 1710/2304 [I 2022-10-28 05:32:22] ...trap/engines/engine.py.228: elapsed: 0:16:03 | left: 0:05:33 [I 2022-10-28 05:32:22] ...trap/engines/engine.py.229: process: 0.56981 | load: 0.00251 [I 2022-10-28 05:32:22] ...trap/engines/engine.py.230: loss: 0.13732 [I 2022-10-28 05:32:22] ...optimizers/trijoint.py.151: total_norm: 39.438351 [I 2022-10-28 05:32:28] ...trap/engines/engine.py.224: train: epoch 108 | batch 1720/2304 [I 2022-10-28 05:32:28] ...trap/engines/engine.py.228: elapsed: 0:16:08 | left: 0:05:27 [I 2022-10-28 05:32:28] ...trap/engines/engine.py.229: process: 0.55466 | load: 0.00247 [I 2022-10-28 05:32:28] ...trap/engines/engine.py.230: loss: 0.10203 [I 2022-10-28 05:32:28] ...optimizers/trijoint.py.151: total_norm: 25.108461 [I 2022-10-28 05:32:33] ...trap/engines/engine.py.224: train: epoch 108 | batch 1730/2304 [I 2022-10-28 05:32:33] ...trap/engines/engine.py.228: elapsed: 0:16:14 | left: 0:05:19 [I 2022-10-28 05:32:33] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00245 [I 2022-10-28 05:32:33] ...trap/engines/engine.py.230: loss: 0.13651 [I 2022-10-28 05:32:33] ...optimizers/trijoint.py.151: total_norm: 15.841399 [I 2022-10-28 05:32:39] ...trap/engines/engine.py.224: train: epoch 108 | batch 1740/2304 [I 2022-10-28 05:32:39] ...trap/engines/engine.py.228: elapsed: 0:16:19 | left: 0:05:15 [I 2022-10-28 05:32:39] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00245 [I 2022-10-28 05:32:39] ...trap/engines/engine.py.230: loss: 0.11823 [I 2022-10-28 05:32:39] ...optimizers/trijoint.py.151: total_norm: 13.573102 [I 2022-10-28 05:32:44] ...trap/engines/engine.py.224: train: epoch 108 | batch 1750/2304 [I 2022-10-28 05:32:44] ...trap/engines/engine.py.228: elapsed: 0:16:25 | left: 0:05:11 [I 2022-10-28 05:32:44] ...trap/engines/engine.py.229: process: 0.56650 | load: 0.00256 [I 2022-10-28 05:32:44] ...trap/engines/engine.py.230: loss: 0.19040 [I 2022-10-28 05:32:44] ...optimizers/trijoint.py.151: total_norm: 9.548906 [I 2022-10-28 05:32:50] ...trap/engines/engine.py.224: train: epoch 108 | batch 1760/2304 [I 2022-10-28 05:32:50] ...trap/engines/engine.py.228: elapsed: 0:16:31 | left: 0:05:04 [I 2022-10-28 05:32:50] ...trap/engines/engine.py.229: process: 0.55264 | load: 0.00257 [I 2022-10-28 05:32:50] ...trap/engines/engine.py.230: loss: 0.12976 [I 2022-10-28 05:32:50] ...optimizers/trijoint.py.151: total_norm: 8.361556 [I 2022-10-28 05:32:56] ...trap/engines/engine.py.224: train: epoch 108 | batch 1770/2304 [I 2022-10-28 05:32:56] ...trap/engines/engine.py.228: elapsed: 0:16:36 | left: 0:04:58 [I 2022-10-28 05:32:56] ...trap/engines/engine.py.229: process: 0.56284 | load: 0.00252 [I 2022-10-28 05:32:56] ...trap/engines/engine.py.230: loss: 0.11501 [I 2022-10-28 05:32:56] ...optimizers/trijoint.py.151: total_norm: 28.724333 [I 2022-10-28 05:33:01] ...trap/engines/engine.py.224: train: epoch 108 | batch 1780/2304 [I 2022-10-28 05:33:01] ...trap/engines/engine.py.228: elapsed: 0:16:42 | left: 0:04:51 [I 2022-10-28 05:33:01] ...trap/engines/engine.py.229: process: 0.55061 | load: 0.00250 [I 2022-10-28 05:33:01] ...trap/engines/engine.py.230: loss: 0.08884 [I 2022-10-28 05:33:01] ...optimizers/trijoint.py.151: total_norm: 12.207079 [I 2022-10-28 05:33:07] ...trap/engines/engine.py.224: train: epoch 108 | batch 1790/2304 [I 2022-10-28 05:33:07] ...trap/engines/engine.py.228: elapsed: 0:16:47 | left: 0:04:48 [I 2022-10-28 05:33:07] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00253 [I 2022-10-28 05:33:07] ...trap/engines/engine.py.230: loss: 0.12547 [I 2022-10-28 05:33:07] ...optimizers/trijoint.py.151: total_norm: 24.957371 [I 2022-10-28 05:33:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 1800/2304 [I 2022-10-28 05:33:12] ...trap/engines/engine.py.228: elapsed: 0:16:53 | left: 0:04:42 [I 2022-10-28 05:33:12] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00260 [I 2022-10-28 05:33:12] ...trap/engines/engine.py.230: loss: 0.13510 [I 2022-10-28 05:33:12] ...optimizers/trijoint.py.151: total_norm: 42.368935 [I 2022-10-28 05:33:18] ...trap/engines/engine.py.224: train: epoch 108 | batch 1810/2304 [I 2022-10-28 05:33:18] ...trap/engines/engine.py.228: elapsed: 0:16:59 | left: 0:04:35 [I 2022-10-28 05:33:18] ...trap/engines/engine.py.229: process: 0.54758 | load: 0.00255 [I 2022-10-28 05:33:18] ...trap/engines/engine.py.230: loss: 0.09595 [I 2022-10-28 05:33:18] ...optimizers/trijoint.py.151: total_norm: 20.628790 [I 2022-10-28 05:33:24] ...trap/engines/engine.py.224: train: epoch 108 | batch 1820/2304 [I 2022-10-28 05:33:24] ...trap/engines/engine.py.228: elapsed: 0:17:04 | left: 0:04:31 [I 2022-10-28 05:33:24] ...trap/engines/engine.py.229: process: 0.56868 | load: 0.00256 [I 2022-10-28 05:33:24] ...trap/engines/engine.py.230: loss: 0.08931 [I 2022-10-28 05:33:24] ...optimizers/trijoint.py.151: total_norm: 9.868223 [I 2022-10-28 05:33:29] ...trap/engines/engine.py.224: train: epoch 108 | batch 1830/2304 [I 2022-10-28 05:33:29] ...trap/engines/engine.py.228: elapsed: 0:17:10 | left: 0:04:25 [I 2022-10-28 05:33:29] ...trap/engines/engine.py.229: process: 0.56324 | load: 0.00257 [I 2022-10-28 05:33:29] ...trap/engines/engine.py.230: loss: 0.08671 [I 2022-10-28 05:33:29] ...optimizers/trijoint.py.151: total_norm: 8.862357 [I 2022-10-28 05:33:35] ...trap/engines/engine.py.224: train: epoch 108 | batch 1840/2304 [I 2022-10-28 05:33:35] ...trap/engines/engine.py.228: elapsed: 0:17:15 | left: 0:04:20 [I 2022-10-28 05:33:35] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00255 [I 2022-10-28 05:33:35] ...trap/engines/engine.py.230: loss: 0.14998 [I 2022-10-28 05:33:35] ...optimizers/trijoint.py.151: total_norm: 43.258816 [I 2022-10-28 05:33:40] ...trap/engines/engine.py.224: train: epoch 108 | batch 1850/2304 [I 2022-10-28 05:33:40] ...trap/engines/engine.py.228: elapsed: 0:17:21 | left: 0:04:16 [I 2022-10-28 05:33:40] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00250 [I 2022-10-28 05:33:40] ...trap/engines/engine.py.230: loss: 0.20936 [I 2022-10-28 05:33:40] ...optimizers/trijoint.py.151: total_norm: 57.484615 [I 2022-10-28 05:33:46] ...trap/engines/engine.py.224: train: epoch 108 | batch 1860/2304 [I 2022-10-28 05:33:46] ...trap/engines/engine.py.228: elapsed: 0:17:27 | left: 0:04:08 [I 2022-10-28 05:33:46] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00247 [I 2022-10-28 05:33:46] ...trap/engines/engine.py.230: loss: 0.13658 [I 2022-10-28 05:33:46] ...optimizers/trijoint.py.151: total_norm: 11.565135 [I 2022-10-28 05:33:52] ...trap/engines/engine.py.224: train: epoch 108 | batch 1870/2304 [I 2022-10-28 05:33:52] ...trap/engines/engine.py.228: elapsed: 0:17:32 | left: 0:04:01 [I 2022-10-28 05:33:52] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00253 [I 2022-10-28 05:33:52] ...trap/engines/engine.py.230: loss: 0.16824 [I 2022-10-28 05:33:52] ...optimizers/trijoint.py.151: total_norm: 13.746433 [I 2022-10-28 05:33:57] ...trap/engines/engine.py.224: train: epoch 108 | batch 1880/2304 [I 2022-10-28 05:33:57] ...trap/engines/engine.py.228: elapsed: 0:17:38 | left: 0:03:56 [I 2022-10-28 05:33:57] ...trap/engines/engine.py.229: process: 0.57005 | load: 0.00248 [I 2022-10-28 05:33:57] ...trap/engines/engine.py.230: loss: 0.07495 [I 2022-10-28 05:33:57] ...optimizers/trijoint.py.151: total_norm: 19.862434 [I 2022-10-28 05:34:03] ...trap/engines/engine.py.224: train: epoch 108 | batch 1890/2304 [I 2022-10-28 05:34:03] ...trap/engines/engine.py.228: elapsed: 0:17:43 | left: 0:03:50 [I 2022-10-28 05:34:03] ...trap/engines/engine.py.229: process: 0.54709 | load: 0.00251 [I 2022-10-28 05:34:03] ...trap/engines/engine.py.230: loss: 0.13943 [I 2022-10-28 05:34:03] ...optimizers/trijoint.py.151: total_norm: 12.429475 [I 2022-10-28 05:34:08] ...trap/engines/engine.py.224: train: epoch 108 | batch 1900/2304 [I 2022-10-28 05:34:08] ...trap/engines/engine.py.228: elapsed: 0:17:49 | left: 0:03:44 [I 2022-10-28 05:34:08] ...trap/engines/engine.py.229: process: 0.55097 | load: 0.00251 [I 2022-10-28 05:34:08] ...trap/engines/engine.py.230: loss: 0.16341 [I 2022-10-28 05:34:08] ...optimizers/trijoint.py.151: total_norm: 23.359308 [I 2022-10-28 05:34:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 1910/2304 [I 2022-10-28 05:34:14] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:39 [I 2022-10-28 05:34:14] ...trap/engines/engine.py.229: process: 0.54611 | load: 0.00246 [I 2022-10-28 05:34:14] ...trap/engines/engine.py.230: loss: 0.08849 [I 2022-10-28 05:34:14] ...optimizers/trijoint.py.151: total_norm: 11.669844 [I 2022-10-28 05:34:20] ...trap/engines/engine.py.224: train: epoch 108 | batch 1920/2304 [I 2022-10-28 05:34:20] ...trap/engines/engine.py.228: elapsed: 0:18:00 | left: 0:03:32 [I 2022-10-28 05:34:20] ...trap/engines/engine.py.229: process: 0.54888 | load: 0.00252 [I 2022-10-28 05:34:20] ...trap/engines/engine.py.230: loss: 0.10371 [I 2022-10-28 05:34:20] ...optimizers/trijoint.py.151: total_norm: 9.660698 [I 2022-10-28 05:34:25] ...trap/engines/engine.py.224: train: epoch 108 | batch 1930/2304 [I 2022-10-28 05:34:25] ...trap/engines/engine.py.228: elapsed: 0:18:06 | left: 0:03:29 [I 2022-10-28 05:34:25] ...trap/engines/engine.py.229: process: 0.56153 | load: 0.00248 [I 2022-10-28 05:34:25] ...trap/engines/engine.py.230: loss: 0.16508 [I 2022-10-28 05:34:25] ...optimizers/trijoint.py.151: total_norm: 40.252327 [I 2022-10-28 05:34:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 1940/2304 [I 2022-10-28 05:34:31] ...trap/engines/engine.py.228: elapsed: 0:18:11 | left: 0:03:24 [I 2022-10-28 05:34:31] ...trap/engines/engine.py.229: process: 0.55190 | load: 0.00291 [I 2022-10-28 05:34:31] ...trap/engines/engine.py.230: loss: 0.09354 [I 2022-10-28 05:34:31] ...optimizers/trijoint.py.151: total_norm: 11.009486 [I 2022-10-28 05:34:36] ...trap/engines/engine.py.224: train: epoch 108 | batch 1950/2304 [I 2022-10-28 05:34:36] ...trap/engines/engine.py.228: elapsed: 0:18:17 | left: 0:03:19 [I 2022-10-28 05:34:36] ...trap/engines/engine.py.229: process: 0.55108 | load: 0.00273 [I 2022-10-28 05:34:36] ...trap/engines/engine.py.230: loss: 0.12079 [I 2022-10-28 05:34:36] ...optimizers/trijoint.py.151: total_norm: 15.531110 [I 2022-10-28 05:34:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 1960/2304 [I 2022-10-28 05:34:42] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:12 [I 2022-10-28 05:34:42] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00255 [I 2022-10-28 05:34:42] ...trap/engines/engine.py.230: loss: 0.08613 [I 2022-10-28 05:34:42] ...optimizers/trijoint.py.151: total_norm: 14.123349 [I 2022-10-28 05:34:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 1970/2304 [I 2022-10-28 05:34:48] ...trap/engines/engine.py.228: elapsed: 0:18:28 | left: 0:03:06 [I 2022-10-28 05:34:48] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00242 [I 2022-10-28 05:34:48] ...trap/engines/engine.py.230: loss: 0.12179 [I 2022-10-28 05:34:48] ...optimizers/trijoint.py.151: total_norm: 13.607735 [I 2022-10-28 05:34:53] ...trap/engines/engine.py.224: train: epoch 108 | batch 1980/2304 [I 2022-10-28 05:34:53] ...trap/engines/engine.py.228: elapsed: 0:18:34 | left: 0:03:00 [I 2022-10-28 05:34:53] ...trap/engines/engine.py.229: process: 0.55587 | load: 0.00243 [I 2022-10-28 05:34:53] ...trap/engines/engine.py.230: loss: 0.09036 [I 2022-10-28 05:34:53] ...optimizers/trijoint.py.151: total_norm: 12.708169 [I 2022-10-28 05:34:59] ...trap/engines/engine.py.224: train: epoch 108 | batch 1990/2304 [I 2022-10-28 05:34:59] ...trap/engines/engine.py.228: elapsed: 0:18:39 | left: 0:02:57 [I 2022-10-28 05:34:59] ...trap/engines/engine.py.229: process: 0.56866 | load: 0.00246 [I 2022-10-28 05:34:59] ...trap/engines/engine.py.230: loss: 0.14607 [I 2022-10-28 05:34:59] ...optimizers/trijoint.py.151: total_norm: 42.955719 [I 2022-10-28 05:35:04] ...trap/engines/engine.py.224: train: epoch 108 | batch 2000/2304 [I 2022-10-28 05:35:04] ...trap/engines/engine.py.228: elapsed: 0:18:45 | left: 0:02:50 [I 2022-10-28 05:35:04] ...trap/engines/engine.py.229: process: 0.56743 | load: 0.00261 [I 2022-10-28 05:35:04] ...trap/engines/engine.py.230: loss: 0.09015 [I 2022-10-28 05:35:04] ...optimizers/trijoint.py.151: total_norm: 12.893596 [I 2022-10-28 05:35:10] ...trap/engines/engine.py.224: train: epoch 108 | batch 2010/2304 [I 2022-10-28 05:35:10] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:45 [I 2022-10-28 05:35:10] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00251 [I 2022-10-28 05:35:10] ...trap/engines/engine.py.230: loss: 0.08842 [I 2022-10-28 05:35:10] ...optimizers/trijoint.py.151: total_norm: 10.775628 [I 2022-10-28 05:35:16] ...trap/engines/engine.py.224: train: epoch 108 | batch 2020/2304 [I 2022-10-28 05:35:16] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:02:40 [I 2022-10-28 05:35:16] ...trap/engines/engine.py.229: process: 0.56837 | load: 0.00247 [I 2022-10-28 05:35:16] ...trap/engines/engine.py.230: loss: 0.13881 [I 2022-10-28 05:35:16] ...optimizers/trijoint.py.151: total_norm: 15.042714 [I 2022-10-28 05:35:21] ...trap/engines/engine.py.224: train: epoch 108 | batch 2030/2304 [I 2022-10-28 05:35:21] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:02:33 [I 2022-10-28 05:35:21] ...trap/engines/engine.py.229: process: 0.56525 | load: 0.00248 [I 2022-10-28 05:35:21] ...trap/engines/engine.py.230: loss: 0.13936 [I 2022-10-28 05:35:21] ...optimizers/trijoint.py.151: total_norm: 16.566633 [I 2022-10-28 05:35:27] ...trap/engines/engine.py.224: train: epoch 108 | batch 2040/2304 [I 2022-10-28 05:35:27] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:27 [I 2022-10-28 05:35:27] ...trap/engines/engine.py.229: process: 0.55041 | load: 0.00244 [I 2022-10-28 05:35:27] ...trap/engines/engine.py.230: loss: 0.15231 [I 2022-10-28 05:35:27] ...optimizers/trijoint.py.151: total_norm: 21.944895 [I 2022-10-28 05:35:33] ...trap/engines/engine.py.224: train: epoch 108 | batch 2050/2304 [I 2022-10-28 05:35:33] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:21 [I 2022-10-28 05:35:33] ...trap/engines/engine.py.229: process: 0.56279 | load: 0.00245 [I 2022-10-28 05:35:33] ...trap/engines/engine.py.230: loss: 0.14224 [I 2022-10-28 05:35:33] ...optimizers/trijoint.py.151: total_norm: 11.694387 [I 2022-10-28 05:35:38] ...trap/engines/engine.py.224: train: epoch 108 | batch 2060/2304 [I 2022-10-28 05:35:38] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:16 [I 2022-10-28 05:35:38] ...trap/engines/engine.py.229: process: 0.56793 | load: 0.00242 [I 2022-10-28 05:35:38] ...trap/engines/engine.py.230: loss: 0.10069 [I 2022-10-28 05:35:38] ...optimizers/trijoint.py.151: total_norm: 15.390520 [I 2022-10-28 05:35:44] ...trap/engines/engine.py.224: train: epoch 108 | batch 2070/2304 [I 2022-10-28 05:35:44] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:10 [I 2022-10-28 05:35:44] ...trap/engines/engine.py.229: process: 0.56187 | load: 0.00243 [I 2022-10-28 05:35:44] ...trap/engines/engine.py.230: loss: 0.11651 [I 2022-10-28 05:35:44] ...optimizers/trijoint.py.151: total_norm: 10.696239 [I 2022-10-28 05:36:09] ...trap/engines/engine.py.224: train: epoch 108 | batch 2080/2304 [I 2022-10-28 05:36:09] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:16:28 [I 2022-10-28 05:36:09] ...trap/engines/engine.py.229: process: 19.80133 | load: 0.00249 [I 2022-10-28 05:36:09] ...trap/engines/engine.py.230: loss: 0.11303 [I 2022-10-28 05:36:09] ...optimizers/trijoint.py.151: total_norm: 9.361933 [I 2022-10-28 05:36:14] ...trap/engines/engine.py.224: train: epoch 108 | batch 2090/2304 [I 2022-10-28 05:36:14] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:03:28 [I 2022-10-28 05:36:14] ...trap/engines/engine.py.229: process: 0.55513 | load: 0.00250 [I 2022-10-28 05:36:14] ...trap/engines/engine.py.230: loss: 0.12109 [I 2022-10-28 05:36:14] ...optimizers/trijoint.py.151: total_norm: 77.688385 [I 2022-10-28 05:36:20] ...trap/engines/engine.py.224: train: epoch 108 | batch 2100/2304 [I 2022-10-28 05:36:20] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:02:03 [I 2022-10-28 05:36:20] ...trap/engines/engine.py.229: process: 0.57550 | load: 0.00246 [I 2022-10-28 05:36:20] ...trap/engines/engine.py.230: loss: 0.08272 [I 2022-10-28 05:36:20] ...optimizers/trijoint.py.151: total_norm: 17.459791 [I 2022-10-28 05:36:26] ...trap/engines/engine.py.224: train: epoch 108 | batch 2110/2304 [I 2022-10-28 05:36:26] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:50 [I 2022-10-28 05:36:26] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00242 [I 2022-10-28 05:36:26] ...trap/engines/engine.py.230: loss: 0.17653 [I 2022-10-28 05:36:26] ...optimizers/trijoint.py.151: total_norm: 19.899122 [I 2022-10-28 05:36:31] ...trap/engines/engine.py.224: train: epoch 108 | batch 2120/2304 [I 2022-10-28 05:36:31] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:43 [I 2022-10-28 05:36:31] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00244 [I 2022-10-28 05:36:31] ...trap/engines/engine.py.230: loss: 0.09397 [I 2022-10-28 05:36:31] ...optimizers/trijoint.py.151: total_norm: 13.337922 [I 2022-10-28 05:36:37] ...trap/engines/engine.py.224: train: epoch 108 | batch 2130/2304 [I 2022-10-28 05:36:37] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:37 [I 2022-10-28 05:36:37] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00243 [I 2022-10-28 05:36:37] ...trap/engines/engine.py.230: loss: 0.10176 [I 2022-10-28 05:36:37] ...optimizers/trijoint.py.151: total_norm: 12.585243 [I 2022-10-28 05:36:42] ...trap/engines/engine.py.224: train: epoch 108 | batch 2140/2304 [I 2022-10-28 05:36:42] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:33 [I 2022-10-28 05:36:42] ...trap/engines/engine.py.229: process: 0.57100 | load: 0.00245 [I 2022-10-28 05:36:42] ...trap/engines/engine.py.230: loss: 0.16389 [I 2022-10-28 05:36:42] ...optimizers/trijoint.py.151: total_norm: 20.759132 [I 2022-10-28 05:36:48] ...trap/engines/engine.py.224: train: epoch 108 | batch 2150/2304 [I 2022-10-28 05:36:48] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:29 [I 2022-10-28 05:36:48] ...trap/engines/engine.py.229: process: 0.57651 | load: 0.00255 [I 2022-10-28 05:36:48] ...trap/engines/engine.py.230: loss: 0.15346 [I 2022-10-28 05:36:48] ...optimizers/trijoint.py.151: total_norm: 14.710371 [I 2022-10-28 05:36:54] ...trap/engines/engine.py.224: train: epoch 108 | batch 2160/2304 [I 2022-10-28 05:36:54] ...trap/engines/engine.py.228: elapsed: 0:20:35 | left: 0:01:21 [I 2022-10-28 05:36:54] ...trap/engines/engine.py.229: process: 0.57255 | load: 0.00247 [I 2022-10-28 05:36:54] ...trap/engines/engine.py.230: loss: 0.16632 [I 2022-10-28 05:36:54] ...optimizers/trijoint.py.151: total_norm: 17.084141 [I 2022-10-28 05:37:00] ...trap/engines/engine.py.224: train: epoch 108 | batch 2170/2304 [I 2022-10-28 05:37:00] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:15 [I 2022-10-28 05:37:00] ...trap/engines/engine.py.229: process: 0.54632 | load: 0.00251 [I 2022-10-28 05:37:00] ...trap/engines/engine.py.230: loss: 0.08881 [I 2022-10-28 05:37:00] ...optimizers/trijoint.py.151: total_norm: 10.722612 [I 2022-10-28 05:37:05] ...trap/engines/engine.py.224: train: epoch 108 | batch 2180/2304 [I 2022-10-28 05:37:05] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:01:09 [I 2022-10-28 05:37:05] ...trap/engines/engine.py.229: process: 0.56771 | load: 0.00250 [I 2022-10-28 05:37:05] ...trap/engines/engine.py.230: loss: 0.10440 [I 2022-10-28 05:37:05] ...optimizers/trijoint.py.151: total_norm: 11.843338 [I 2022-10-28 05:37:11] ...trap/engines/engine.py.224: train: epoch 108 | batch 2190/2304 [I 2022-10-28 05:37:11] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:01:03 [I 2022-10-28 05:37:11] ...trap/engines/engine.py.229: process: 0.56711 | load: 0.00249 [I 2022-10-28 05:37:11] ...trap/engines/engine.py.230: loss: 0.08110 [I 2022-10-28 05:37:11] ...optimizers/trijoint.py.151: total_norm: 20.648966 [I 2022-10-28 05:37:16] ...trap/engines/engine.py.224: train: epoch 108 | batch 2200/2304 [I 2022-10-28 05:37:16] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:58 [I 2022-10-28 05:37:16] ...trap/engines/engine.py.229: process: 0.57154 | load: 0.00243 [I 2022-10-28 05:37:16] ...trap/engines/engine.py.230: loss: 0.10614 [I 2022-10-28 05:37:16] ...optimizers/trijoint.py.151: total_norm: 13.283393 [I 2022-10-28 05:37:22] ...trap/engines/engine.py.224: train: epoch 108 | batch 2210/2304 [I 2022-10-28 05:37:22] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:52 [I 2022-10-28 05:37:22] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00244 [I 2022-10-28 05:37:22] ...trap/engines/engine.py.230: loss: 0.10721 [I 2022-10-28 05:37:22] ...optimizers/trijoint.py.151: total_norm: 9.910291 [I 2022-10-28 05:37:27] ...trap/engines/engine.py.224: train: epoch 108 | batch 2220/2304 [I 2022-10-28 05:37:27] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:46 [I 2022-10-28 05:37:27] ...trap/engines/engine.py.229: process: 0.54790 | load: 0.00247 [I 2022-10-28 05:37:27] ...trap/engines/engine.py.230: loss: 0.13355 [I 2022-10-28 05:37:27] ...optimizers/trijoint.py.151: total_norm: 12.034146 [I 2022-10-28 05:37:33] ...trap/engines/engine.py.224: train: epoch 108 | batch 2230/2304 [I 2022-10-28 05:37:33] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:41 [I 2022-10-28 05:37:33] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00248 [I 2022-10-28 05:37:33] ...trap/engines/engine.py.230: loss: 0.16207 [I 2022-10-28 05:37:33] ...optimizers/trijoint.py.151: total_norm: 21.877422 [I 2022-10-28 05:37:39] ...trap/engines/engine.py.224: train: epoch 108 | batch 2240/2304 [I 2022-10-28 05:37:39] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:35 [I 2022-10-28 05:37:39] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00248 [I 2022-10-28 05:37:39] ...trap/engines/engine.py.230: loss: 0.19601 [I 2022-10-28 05:37:39] ...optimizers/trijoint.py.151: total_norm: 11.046761 [I 2022-10-28 05:37:44] ...trap/engines/engine.py.224: train: epoch 108 | batch 2250/2304 [I 2022-10-28 05:37:44] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:30 [I 2022-10-28 05:37:44] ...trap/engines/engine.py.229: process: 0.55070 | load: 0.00272 [I 2022-10-28 05:37:44] ...trap/engines/engine.py.230: loss: 0.17858 [I 2022-10-28 05:37:44] ...optimizers/trijoint.py.151: total_norm: 29.921759 [I 2022-10-28 05:37:50] ...trap/engines/engine.py.224: train: epoch 108 | batch 2260/2304 [I 2022-10-28 05:37:50] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:24 [I 2022-10-28 05:37:50] ...trap/engines/engine.py.229: process: 0.56466 | load: 0.00253 [I 2022-10-28 05:37:50] ...trap/engines/engine.py.230: loss: 0.08298 [I 2022-10-28 05:37:50] ...optimizers/trijoint.py.151: total_norm: 8.540072 [I 2022-10-28 05:37:56] ...trap/engines/engine.py.224: train: epoch 108 | batch 2270/2304 [I 2022-10-28 05:37:56] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:18 [I 2022-10-28 05:37:56] ...trap/engines/engine.py.229: process: 0.55953 | load: 0.00243 [I 2022-10-28 05:37:56] ...trap/engines/engine.py.230: loss: 0.11551 [I 2022-10-28 05:37:56] ...optimizers/trijoint.py.151: total_norm: 10.322746 [I 2022-10-28 05:38:01] ...trap/engines/engine.py.224: train: epoch 108 | batch 2280/2304 [I 2022-10-28 05:38:01] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:00:13 [I 2022-10-28 05:38:01] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00247 [I 2022-10-28 05:38:01] ...trap/engines/engine.py.230: loss: 0.10585 [I 2022-10-28 05:38:01] ...optimizers/trijoint.py.151: total_norm: 14.029119 [I 2022-10-28 05:38:07] ...trap/engines/engine.py.224: train: epoch 108 | batch 2290/2304 [I 2022-10-28 05:38:07] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:07 [I 2022-10-28 05:38:07] ...trap/engines/engine.py.229: process: 0.55843 | load: 0.00243 [I 2022-10-28 05:38:07] ...trap/engines/engine.py.230: loss: 0.14577 [I 2022-10-28 05:38:07] ...optimizers/trijoint.py.151: total_norm: 19.409231 [I 2022-10-28 05:38:12] ...trap/engines/engine.py.224: train: epoch 108 | batch 2300/2304 [I 2022-10-28 05:38:12] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:00:02 [I 2022-10-28 05:38:12] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00014 [I 2022-10-28 05:38:12] ...trap/engines/engine.py.230: loss: 0.17127 [I 2022-10-28 05:38:12] ...optimizers/trijoint.py.151: total_norm: 19.061693 [I 2022-10-28 05:38:15] ...trap/engines/engine.py.224: train: epoch 108 | batch 2304/2304 [I 2022-10-28 05:38:15] ...trap/engines/engine.py.228: elapsed: 0:21:55 | left: 0:00:00 [I 2022-10-28 05:38:15] ...trap/engines/engine.py.229: process: 0.55816 | load: 0.00014 [I 2022-10-28 05:38:15] ...trap/engines/engine.py.230: loss: 0.09810 [I 2022-10-28 05:38:15] ...optimizers/trijoint.py.151: total_norm: 10.909726 [S 2022-10-28 05:38:16] ...trap/engines/engine.py.129: train_epoch.epoch: 108 [S 2022-10-28 05:38:16] ...trap/engines/engine.py.129: train_epoch.loss: 0.12355249528318579 [I 2022-10-28 05:38:16] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 05:38:18] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 108 [I 2022-10-28 05:38:18] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 05:38:20] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 05:38:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 05:38:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 05:38:23] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 05:38:24] ...trap/engines/engine.py.321: eval: epoch 108 | batch 0/510 [I 2022-10-28 05:38:24] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:50:46 [I 2022-10-28 05:38:24] ...trap/engines/engine.py.326: process: 5.97420 | load: 3.10489 [I 2022-10-28 05:38:27] ...trap/engines/engine.py.321: eval: epoch 108 | batch 10/510 [I 2022-10-28 05:38:27] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:49 [I 2022-10-28 05:38:27] ...trap/engines/engine.py.326: process: 0.33777 | load: 0.00016 [I 2022-10-28 05:38:30] ...trap/engines/engine.py.321: eval: epoch 108 | batch 20/510 [I 2022-10-28 05:38:30] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:12 [I 2022-10-28 05:38:30] ...trap/engines/engine.py.326: process: 0.31861 | load: 0.00016 [I 2022-10-28 05:38:34] ...trap/engines/engine.py.321: eval: epoch 108 | batch 30/510 [I 2022-10-28 05:38:34] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:45 [I 2022-10-28 05:38:34] ...trap/engines/engine.py.326: process: 0.35457 | load: 0.00017 [I 2022-10-28 05:38:37] ...trap/engines/engine.py.321: eval: epoch 108 | batch 40/510 [I 2022-10-28 05:38:37] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-28 05:38:37] ...trap/engines/engine.py.326: process: 0.32909 | load: 0.00019 [I 2022-10-28 05:38:40] ...trap/engines/engine.py.321: eval: epoch 108 | batch 50/510 [I 2022-10-28 05:38:40] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:33 [I 2022-10-28 05:38:40] ...trap/engines/engine.py.326: process: 0.33089 | load: 0.00027 [I 2022-10-28 05:38:44] ...trap/engines/engine.py.321: eval: epoch 108 | batch 60/510 [I 2022-10-28 05:38:44] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:29 [I 2022-10-28 05:38:44] ...trap/engines/engine.py.326: process: 0.32910 | load: 0.00020 [I 2022-10-28 05:38:47] ...trap/engines/engine.py.321: eval: epoch 108 | batch 70/510 [I 2022-10-28 05:38:47] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:26 [I 2022-10-28 05:38:47] ...trap/engines/engine.py.326: process: 0.34379 | load: 0.00017 [I 2022-10-28 05:38:50] ...trap/engines/engine.py.321: eval: epoch 108 | batch 80/510 [I 2022-10-28 05:38:50] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:23 [I 2022-10-28 05:38:50] ...trap/engines/engine.py.326: process: 0.32448 | load: 0.00038 [I 2022-10-28 05:38:54] ...trap/engines/engine.py.321: eval: epoch 108 | batch 90/510 [I 2022-10-28 05:38:54] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:21 [I 2022-10-28 05:38:54] ...trap/engines/engine.py.326: process: 0.33118 | load: 0.00022 [I 2022-10-28 05:38:57] ...trap/engines/engine.py.321: eval: epoch 108 | batch 100/510 [I 2022-10-28 05:38:57] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-28 05:38:57] ...trap/engines/engine.py.326: process: 0.28279 | load: 0.00024 [I 2022-10-28 05:39:00] ...trap/engines/engine.py.321: eval: epoch 108 | batch 110/510 [I 2022-10-28 05:39:00] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:57 [I 2022-10-28 05:39:00] ...trap/engines/engine.py.326: process: 0.28649 | load: 0.00014 [I 2022-10-28 05:39:03] ...trap/engines/engine.py.321: eval: epoch 108 | batch 120/510 [I 2022-10-28 05:39:03] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:54 [I 2022-10-28 05:39:03] ...trap/engines/engine.py.326: process: 0.30572 | load: 0.00023 [I 2022-10-28 05:39:06] ...trap/engines/engine.py.321: eval: epoch 108 | batch 130/510 [I 2022-10-28 05:39:06] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:50 [I 2022-10-28 05:39:06] ...trap/engines/engine.py.326: process: 0.28913 | load: 0.00018 [I 2022-10-28 05:39:09] ...trap/engines/engine.py.321: eval: epoch 108 | batch 140/510 [I 2022-10-28 05:39:09] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:48 [I 2022-10-28 05:39:09] ...trap/engines/engine.py.326: process: 0.29038 | load: 0.00018 [I 2022-10-28 05:39:12] ...trap/engines/engine.py.321: eval: epoch 108 | batch 150/510 [I 2022-10-28 05:39:12] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:47 [I 2022-10-28 05:39:12] ...trap/engines/engine.py.326: process: 0.28631 | load: 0.00016 [I 2022-10-28 05:39:15] ...trap/engines/engine.py.321: eval: epoch 108 | batch 160/510 [I 2022-10-28 05:39:15] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:42 [I 2022-10-28 05:39:15] ...trap/engines/engine.py.326: process: 0.28951 | load: 0.00018 [I 2022-10-28 05:39:17] ...trap/engines/engine.py.321: eval: epoch 108 | batch 170/510 [I 2022-10-28 05:39:17] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:39 [I 2022-10-28 05:39:17] ...trap/engines/engine.py.326: process: 0.28833 | load: 0.00025 [I 2022-10-28 05:39:20] ...trap/engines/engine.py.321: eval: epoch 108 | batch 180/510 [I 2022-10-28 05:39:20] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-28 05:39:20] ...trap/engines/engine.py.326: process: 0.29117 | load: 0.00024 [I 2022-10-28 05:39:23] ...trap/engines/engine.py.321: eval: epoch 108 | batch 190/510 [I 2022-10-28 05:39:23] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:32 [I 2022-10-28 05:39:23] ...trap/engines/engine.py.326: process: 0.28534 | load: 0.00017 [I 2022-10-28 05:39:26] ...trap/engines/engine.py.321: eval: epoch 108 | batch 200/510 [I 2022-10-28 05:39:26] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:28 [I 2022-10-28 05:39:26] ...trap/engines/engine.py.326: process: 0.28423 | load: 0.00023 [I 2022-10-28 05:39:29] ...trap/engines/engine.py.321: eval: epoch 108 | batch 210/510 [I 2022-10-28 05:39:29] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 05:39:29] ...trap/engines/engine.py.326: process: 0.28704 | load: 0.00017 [I 2022-10-28 05:39:32] ...trap/engines/engine.py.321: eval: epoch 108 | batch 220/510 [I 2022-10-28 05:39:32] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:24 [I 2022-10-28 05:39:32] ...trap/engines/engine.py.326: process: 0.29064 | load: 0.00016 [I 2022-10-28 05:39:35] ...trap/engines/engine.py.321: eval: epoch 108 | batch 230/510 [I 2022-10-28 05:39:35] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:20 [I 2022-10-28 05:39:35] ...trap/engines/engine.py.326: process: 0.27881 | load: 0.00014 [I 2022-10-28 05:39:38] ...trap/engines/engine.py.321: eval: epoch 108 | batch 240/510 [I 2022-10-28 05:39:38] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:18 [I 2022-10-28 05:39:38] ...trap/engines/engine.py.326: process: 0.29158 | load: 0.00016 [I 2022-10-28 05:39:41] ...trap/engines/engine.py.321: eval: epoch 108 | batch 250/510 [I 2022-10-28 05:39:41] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-28 05:39:41] ...trap/engines/engine.py.326: process: 0.28633 | load: 0.00019 [I 2022-10-28 05:39:43] ...trap/engines/engine.py.321: eval: epoch 108 | batch 260/510 [I 2022-10-28 05:39:43] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 05:39:43] ...trap/engines/engine.py.326: process: 0.28987 | load: 0.00021 [I 2022-10-28 05:39:46] ...trap/engines/engine.py.321: eval: epoch 108 | batch 270/510 [I 2022-10-28 05:39:46] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-28 05:39:46] ...trap/engines/engine.py.326: process: 0.28793 | load: 0.00016 [I 2022-10-28 05:39:49] ...trap/engines/engine.py.321: eval: epoch 108 | batch 280/510 [I 2022-10-28 05:39:49] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 05:39:49] ...trap/engines/engine.py.326: process: 0.28374 | load: 0.00015 [I 2022-10-28 05:39:52] ...trap/engines/engine.py.321: eval: epoch 108 | batch 290/510 [I 2022-10-28 05:39:52] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 05:39:52] ...trap/engines/engine.py.326: process: 0.28543 | load: 0.00015 [I 2022-10-28 05:39:55] ...trap/engines/engine.py.321: eval: epoch 108 | batch 300/510 [I 2022-10-28 05:39:55] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-28 05:39:55] ...trap/engines/engine.py.326: process: 0.29098 | load: 0.00015 [I 2022-10-28 05:39:58] ...trap/engines/engine.py.321: eval: epoch 108 | batch 310/510 [I 2022-10-28 05:39:58] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:58 [I 2022-10-28 05:39:58] ...trap/engines/engine.py.326: process: 0.29212 | load: 0.00016 [I 2022-10-28 05:40:01] ...trap/engines/engine.py.321: eval: epoch 108 | batch 320/510 [I 2022-10-28 05:40:01] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:55 [I 2022-10-28 05:40:01] ...trap/engines/engine.py.326: process: 0.28172 | load: 0.00021 [I 2022-10-28 05:40:04] ...trap/engines/engine.py.321: eval: epoch 108 | batch 330/510 [I 2022-10-28 05:40:04] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:52 [I 2022-10-28 05:40:04] ...trap/engines/engine.py.326: process: 0.29575 | load: 0.00015 [I 2022-10-28 05:40:07] ...trap/engines/engine.py.321: eval: epoch 108 | batch 340/510 [I 2022-10-28 05:40:07] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-28 05:40:07] ...trap/engines/engine.py.326: process: 0.28991 | load: 0.00020 [I 2022-10-28 05:40:10] ...trap/engines/engine.py.321: eval: epoch 108 | batch 350/510 [I 2022-10-28 05:40:10] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 05:40:10] ...trap/engines/engine.py.326: process: 0.28229 | load: 0.00015 [I 2022-10-28 05:40:13] ...trap/engines/engine.py.321: eval: epoch 108 | batch 360/510 [I 2022-10-28 05:40:13] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:44 [I 2022-10-28 05:40:13] ...trap/engines/engine.py.326: process: 0.28742 | load: 0.00018 [I 2022-10-28 05:40:15] ...trap/engines/engine.py.321: eval: epoch 108 | batch 370/510 [I 2022-10-28 05:40:15] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 05:40:15] ...trap/engines/engine.py.326: process: 0.28293 | load: 0.00014 [I 2022-10-28 05:40:18] ...trap/engines/engine.py.321: eval: epoch 108 | batch 380/510 [I 2022-10-28 05:40:18] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:38 [I 2022-10-28 05:40:18] ...trap/engines/engine.py.326: process: 0.31194 | load: 0.00019 [I 2022-10-28 05:40:21] ...trap/engines/engine.py.321: eval: epoch 108 | batch 390/510 [I 2022-10-28 05:40:21] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 05:40:21] ...trap/engines/engine.py.326: process: 0.29154 | load: 0.00038 [I 2022-10-28 05:40:24] ...trap/engines/engine.py.321: eval: epoch 108 | batch 400/510 [I 2022-10-28 05:40:24] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:32 [I 2022-10-28 05:40:24] ...trap/engines/engine.py.326: process: 0.30341 | load: 0.00023 [I 2022-10-28 05:40:27] ...trap/engines/engine.py.321: eval: epoch 108 | batch 410/510 [I 2022-10-28 05:40:27] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:28 [I 2022-10-28 05:40:27] ...trap/engines/engine.py.326: process: 0.28119 | load: 0.00024 [I 2022-10-28 05:40:30] ...trap/engines/engine.py.321: eval: epoch 108 | batch 420/510 [I 2022-10-28 05:40:30] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:26 [I 2022-10-28 05:40:30] ...trap/engines/engine.py.326: process: 0.29117 | load: 0.00015 [I 2022-10-28 05:40:33] ...trap/engines/engine.py.321: eval: epoch 108 | batch 430/510 [I 2022-10-28 05:40:33] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:22 [I 2022-10-28 05:40:33] ...trap/engines/engine.py.326: process: 0.28464 | load: 0.00018 [I 2022-10-28 05:40:36] ...trap/engines/engine.py.321: eval: epoch 108 | batch 440/510 [I 2022-10-28 05:40:36] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:20 [I 2022-10-28 05:40:36] ...trap/engines/engine.py.326: process: 0.28855 | load: 0.00020 [I 2022-10-28 05:40:39] ...trap/engines/engine.py.321: eval: epoch 108 | batch 450/510 [I 2022-10-28 05:40:39] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:17 [I 2022-10-28 05:40:39] ...trap/engines/engine.py.326: process: 0.28210 | load: 0.00017 [I 2022-10-28 05:40:42] ...trap/engines/engine.py.321: eval: epoch 108 | batch 460/510 [I 2022-10-28 05:40:42] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 05:40:42] ...trap/engines/engine.py.326: process: 0.28575 | load: 0.00017 [I 2022-10-28 05:40:44] ...trap/engines/engine.py.321: eval: epoch 108 | batch 470/510 [I 2022-10-28 05:40:44] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 05:40:44] ...trap/engines/engine.py.326: process: 0.28554 | load: 0.00021 [I 2022-10-28 05:40:47] ...trap/engines/engine.py.321: eval: epoch 108 | batch 480/510 [I 2022-10-28 05:40:47] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 05:40:47] ...trap/engines/engine.py.326: process: 0.28875 | load: 0.00024 [I 2022-10-28 05:40:50] ...trap/engines/engine.py.321: eval: epoch 108 | batch 490/510 [I 2022-10-28 05:40:50] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 05:40:50] ...trap/engines/engine.py.326: process: 0.28832 | load: 0.00020 [I 2022-10-28 05:40:53] ...trap/engines/engine.py.321: eval: epoch 108 | batch 500/510 [I 2022-10-28 05:40:53] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:02 [I 2022-10-28 05:40:53] ...trap/engines/engine.py.326: process: 0.29346 | load: 0.00020 [I 2022-10-28 05:40:56] ...trap/engines/engine.py.321: eval: epoch 108 | batch 510/510 [I 2022-10-28 05:40:56] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:00 [I 2022-10-28 05:40:56] ...trap/engines/engine.py.326: process: 0.28185 | load: 0.00011 [S 2022-10-28 05:40:57] ...trap/engines/engine.py.132: eval_epoch.epoch: 108 [I 2022-10-28 05:40:57] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.737500 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906500 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933400 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.732800 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906500 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.931600 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011918 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.003442 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004565 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010438 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007256 [S 2022-10-28 05:41:01] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006756 [I 2022-10-28 05:41:03] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 05:41:03] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 05:41:04] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 05:41:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 05:41:04] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 05:41:08] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 05:41:09] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 05:41:15] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 05:41:15] ...trap/engines/engine.py.163: Training model on trainset for epoch 109 [I 2022-10-28 05:41:15] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 05:41:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 0/2304 [I 2022-10-28 05:41:18] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:30 [I 2022-10-28 05:41:18] ...trap/engines/engine.py.229: process: 2.79978 | load: 2.14214 [I 2022-10-28 05:41:18] ...trap/engines/engine.py.230: loss: 0.15139 [I 2022-10-28 05:41:18] ...optimizers/trijoint.py.151: total_norm: 19.494013 [I 2022-10-28 05:41:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 10/2304 [I 2022-10-28 05:41:24] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:50 [I 2022-10-28 05:41:24] ...trap/engines/engine.py.229: process: 0.58995 | load: 0.00439 [I 2022-10-28 05:41:24] ...trap/engines/engine.py.230: loss: 0.12738 [I 2022-10-28 05:41:24] ...optimizers/trijoint.py.151: total_norm: 36.984600 [I 2022-10-28 05:41:29] ...trap/engines/engine.py.224: train: epoch 109 | batch 20/2304 [I 2022-10-28 05:41:29] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:19 [I 2022-10-28 05:41:29] ...trap/engines/engine.py.229: process: 0.55423 | load: 0.00364 [I 2022-10-28 05:41:29] ...trap/engines/engine.py.230: loss: 0.12576 [I 2022-10-28 05:41:29] ...optimizers/trijoint.py.151: total_norm: 24.110422 [I 2022-10-28 05:41:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 30/2304 [I 2022-10-28 05:41:35] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:14 [I 2022-10-28 05:41:35] ...trap/engines/engine.py.229: process: 0.55601 | load: 0.00282 [I 2022-10-28 05:41:35] ...trap/engines/engine.py.230: loss: 0.09613 [I 2022-10-28 05:41:35] ...optimizers/trijoint.py.151: total_norm: 10.483120 [I 2022-10-28 05:41:40] ...trap/engines/engine.py.224: train: epoch 109 | batch 40/2304 [I 2022-10-28 05:41:40] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:08 [I 2022-10-28 05:41:40] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00315 [I 2022-10-28 05:41:40] ...trap/engines/engine.py.230: loss: 0.08954 [I 2022-10-28 05:41:40] ...optimizers/trijoint.py.151: total_norm: 19.584793 [I 2022-10-28 05:41:46] ...trap/engines/engine.py.224: train: epoch 109 | batch 50/2304 [I 2022-10-28 05:41:46] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:54 [I 2022-10-28 05:41:46] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00329 [I 2022-10-28 05:41:46] ...trap/engines/engine.py.230: loss: 0.18184 [I 2022-10-28 05:41:46] ...optimizers/trijoint.py.151: total_norm: 11.754777 [I 2022-10-28 05:41:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 60/2304 [I 2022-10-28 05:41:52] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:02 [I 2022-10-28 05:41:52] ...trap/engines/engine.py.229: process: 0.57850 | load: 0.00281 [I 2022-10-28 05:41:52] ...trap/engines/engine.py.230: loss: 0.12884 [I 2022-10-28 05:41:52] ...optimizers/trijoint.py.151: total_norm: 17.000698 [I 2022-10-28 05:41:57] ...trap/engines/engine.py.224: train: epoch 109 | batch 70/2304 [I 2022-10-28 05:41:57] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:21:02 [I 2022-10-28 05:41:57] ...trap/engines/engine.py.229: process: 0.56050 | load: 0.00278 [I 2022-10-28 05:41:57] ...trap/engines/engine.py.230: loss: 0.10879 [I 2022-10-28 05:41:57] ...optimizers/trijoint.py.151: total_norm: 25.190886 [I 2022-10-28 05:42:03] ...trap/engines/engine.py.224: train: epoch 109 | batch 80/2304 [I 2022-10-28 05:42:03] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:46 [I 2022-10-28 05:42:03] ...trap/engines/engine.py.229: process: 0.55321 | load: 0.00286 [I 2022-10-28 05:42:03] ...trap/engines/engine.py.230: loss: 0.15009 [I 2022-10-28 05:42:03] ...optimizers/trijoint.py.151: total_norm: 17.253029 [I 2022-10-28 05:42:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 90/2304 [I 2022-10-28 05:42:09] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:40 [I 2022-10-28 05:42:09] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00272 [I 2022-10-28 05:42:09] ...trap/engines/engine.py.230: loss: 0.11715 [I 2022-10-28 05:42:09] ...optimizers/trijoint.py.151: total_norm: 23.527739 [I 2022-10-28 05:42:14] ...trap/engines/engine.py.224: train: epoch 109 | batch 100/2304 [I 2022-10-28 05:42:14] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:28 [I 2022-10-28 05:42:14] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00288 [I 2022-10-28 05:42:14] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-28 05:42:14] ...optimizers/trijoint.py.151: total_norm: 8.214191 [I 2022-10-28 05:42:20] ...trap/engines/engine.py.224: train: epoch 109 | batch 110/2304 [I 2022-10-28 05:42:20] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:21 [I 2022-10-28 05:42:20] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00253 [I 2022-10-28 05:42:20] ...trap/engines/engine.py.230: loss: 0.08782 [I 2022-10-28 05:42:20] ...optimizers/trijoint.py.151: total_norm: 9.449423 [I 2022-10-28 05:42:25] ...trap/engines/engine.py.224: train: epoch 109 | batch 120/2304 [I 2022-10-28 05:42:25] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:11 [I 2022-10-28 05:42:25] ...trap/engines/engine.py.229: process: 0.54113 | load: 0.00267 [I 2022-10-28 05:42:25] ...trap/engines/engine.py.230: loss: 0.08282 [I 2022-10-28 05:42:25] ...optimizers/trijoint.py.151: total_norm: 7.946694 [I 2022-10-28 05:42:31] ...trap/engines/engine.py.224: train: epoch 109 | batch 130/2304 [I 2022-10-28 05:42:31] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:13 [I 2022-10-28 05:42:31] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00273 [I 2022-10-28 05:42:31] ...trap/engines/engine.py.230: loss: 0.08904 [I 2022-10-28 05:42:31] ...optimizers/trijoint.py.151: total_norm: 26.309502 [I 2022-10-28 05:42:36] ...trap/engines/engine.py.224: train: epoch 109 | batch 140/2304 [I 2022-10-28 05:42:36] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:13 [I 2022-10-28 05:42:36] ...trap/engines/engine.py.229: process: 0.54939 | load: 0.00249 [I 2022-10-28 05:42:36] ...trap/engines/engine.py.230: loss: 0.12844 [I 2022-10-28 05:42:36] ...optimizers/trijoint.py.151: total_norm: 12.992958 [I 2022-10-28 05:42:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 150/2304 [I 2022-10-28 05:42:42] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:01 [I 2022-10-28 05:42:42] ...trap/engines/engine.py.229: process: 0.54148 | load: 0.00253 [I 2022-10-28 05:42:42] ...trap/engines/engine.py.230: loss: 0.08497 [I 2022-10-28 05:42:42] ...optimizers/trijoint.py.151: total_norm: 14.602207 [I 2022-10-28 05:42:48] ...trap/engines/engine.py.224: train: epoch 109 | batch 160/2304 [I 2022-10-28 05:42:48] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:52 [I 2022-10-28 05:42:48] ...trap/engines/engine.py.229: process: 0.53909 | load: 0.00268 [I 2022-10-28 05:42:48] ...trap/engines/engine.py.230: loss: 0.13954 [I 2022-10-28 05:42:48] ...optimizers/trijoint.py.151: total_norm: 24.019236 [I 2022-10-28 05:42:53] ...trap/engines/engine.py.224: train: epoch 109 | batch 170/2304 [I 2022-10-28 05:42:53] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:43 [I 2022-10-28 05:42:53] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00267 [I 2022-10-28 05:42:53] ...trap/engines/engine.py.230: loss: 0.09972 [I 2022-10-28 05:42:53] ...optimizers/trijoint.py.151: total_norm: 27.203861 [I 2022-10-28 05:42:59] ...trap/engines/engine.py.224: train: epoch 109 | batch 180/2304 [I 2022-10-28 05:42:59] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:49 [I 2022-10-28 05:42:59] ...trap/engines/engine.py.229: process: 0.56758 | load: 0.00243 [I 2022-10-28 05:42:59] ...trap/engines/engine.py.230: loss: 0.12040 [I 2022-10-28 05:42:59] ...optimizers/trijoint.py.151: total_norm: 14.137040 [I 2022-10-28 05:43:04] ...trap/engines/engine.py.224: train: epoch 109 | batch 190/2304 [I 2022-10-28 05:43:04] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:27 [I 2022-10-28 05:43:04] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00260 [I 2022-10-28 05:43:04] ...trap/engines/engine.py.230: loss: 0.12538 [I 2022-10-28 05:43:04] ...optimizers/trijoint.py.151: total_norm: 15.814080 [I 2022-10-28 05:43:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 200/2304 [I 2022-10-28 05:43:10] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:33 [I 2022-10-28 05:43:10] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00263 [I 2022-10-28 05:43:10] ...trap/engines/engine.py.230: loss: 0.11160 [I 2022-10-28 05:43:10] ...optimizers/trijoint.py.151: total_norm: 14.151032 [I 2022-10-28 05:43:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 210/2304 [I 2022-10-28 05:43:16] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:24 [I 2022-10-28 05:43:16] ...trap/engines/engine.py.229: process: 0.54702 | load: 0.00243 [I 2022-10-28 05:43:16] ...trap/engines/engine.py.230: loss: 0.11538 [I 2022-10-28 05:43:16] ...optimizers/trijoint.py.151: total_norm: 17.675690 [I 2022-10-28 05:43:21] ...trap/engines/engine.py.224: train: epoch 109 | batch 220/2304 [I 2022-10-28 05:43:21] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:22 [I 2022-10-28 05:43:21] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00245 [I 2022-10-28 05:43:21] ...trap/engines/engine.py.230: loss: 0.12350 [I 2022-10-28 05:43:21] ...optimizers/trijoint.py.151: total_norm: 23.206944 [I 2022-10-28 05:43:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 230/2304 [I 2022-10-28 05:43:27] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:08 [I 2022-10-28 05:43:27] ...trap/engines/engine.py.229: process: 0.55112 | load: 0.00267 [I 2022-10-28 05:43:27] ...trap/engines/engine.py.230: loss: 0.12745 [I 2022-10-28 05:43:27] ...optimizers/trijoint.py.151: total_norm: 32.897537 [I 2022-10-28 05:43:32] ...trap/engines/engine.py.224: train: epoch 109 | batch 240/2304 [I 2022-10-28 05:43:32] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:19 [I 2022-10-28 05:43:32] ...trap/engines/engine.py.229: process: 0.55484 | load: 0.00249 [I 2022-10-28 05:43:32] ...trap/engines/engine.py.230: loss: 0.09525 [I 2022-10-28 05:43:32] ...optimizers/trijoint.py.151: total_norm: 13.938856 [I 2022-10-28 05:43:38] ...trap/engines/engine.py.224: train: epoch 109 | batch 250/2304 [I 2022-10-28 05:43:38] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:02 [I 2022-10-28 05:43:38] ...trap/engines/engine.py.229: process: 0.55051 | load: 0.00292 [I 2022-10-28 05:43:38] ...trap/engines/engine.py.230: loss: 0.13259 [I 2022-10-28 05:43:38] ...optimizers/trijoint.py.151: total_norm: 22.601978 [I 2022-10-28 05:43:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 260/2304 [I 2022-10-28 05:43:43] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:59 [I 2022-10-28 05:43:43] ...trap/engines/engine.py.229: process: 0.55090 | load: 0.00246 [I 2022-10-28 05:43:43] ...trap/engines/engine.py.230: loss: 0.16213 [I 2022-10-28 05:43:43] ...optimizers/trijoint.py.151: total_norm: 16.414316 [I 2022-10-28 05:43:49] ...trap/engines/engine.py.224: train: epoch 109 | batch 270/2304 [I 2022-10-28 05:43:49] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:59 [I 2022-10-28 05:43:49] ...trap/engines/engine.py.229: process: 0.55239 | load: 0.00243 [I 2022-10-28 05:43:49] ...trap/engines/engine.py.230: loss: 0.15549 [I 2022-10-28 05:43:49] ...optimizers/trijoint.py.151: total_norm: 11.150949 [I 2022-10-28 05:43:55] ...trap/engines/engine.py.224: train: epoch 109 | batch 280/2304 [I 2022-10-28 05:43:55] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:51 [I 2022-10-28 05:43:55] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00252 [I 2022-10-28 05:43:55] ...trap/engines/engine.py.230: loss: 0.10854 [I 2022-10-28 05:43:55] ...optimizers/trijoint.py.151: total_norm: 34.272091 [I 2022-10-28 05:44:00] ...trap/engines/engine.py.224: train: epoch 109 | batch 290/2304 [I 2022-10-28 05:44:00] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:43 [I 2022-10-28 05:44:00] ...trap/engines/engine.py.229: process: 0.55714 | load: 0.00251 [I 2022-10-28 05:44:00] ...trap/engines/engine.py.230: loss: 0.09093 [I 2022-10-28 05:44:00] ...optimizers/trijoint.py.151: total_norm: 14.438532 [I 2022-10-28 05:44:06] ...trap/engines/engine.py.224: train: epoch 109 | batch 300/2304 [I 2022-10-28 05:44:06] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:39 [I 2022-10-28 05:44:06] ...trap/engines/engine.py.229: process: 0.56392 | load: 0.00260 [I 2022-10-28 05:44:06] ...trap/engines/engine.py.230: loss: 0.09664 [I 2022-10-28 05:44:06] ...optimizers/trijoint.py.151: total_norm: 21.732475 [I 2022-10-28 05:44:12] ...trap/engines/engine.py.224: train: epoch 109 | batch 310/2304 [I 2022-10-28 05:44:12] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:30 [I 2022-10-28 05:44:12] ...trap/engines/engine.py.229: process: 0.55382 | load: 0.00248 [I 2022-10-28 05:44:12] ...trap/engines/engine.py.230: loss: 0.16706 [I 2022-10-28 05:44:12] ...optimizers/trijoint.py.151: total_norm: 16.538439 [I 2022-10-28 05:44:17] ...trap/engines/engine.py.224: train: epoch 109 | batch 320/2304 [I 2022-10-28 05:44:17] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:23 [I 2022-10-28 05:44:17] ...trap/engines/engine.py.229: process: 0.55375 | load: 0.00242 [I 2022-10-28 05:44:17] ...trap/engines/engine.py.230: loss: 0.18510 [I 2022-10-28 05:44:17] ...optimizers/trijoint.py.151: total_norm: 17.595346 [I 2022-10-28 05:44:23] ...trap/engines/engine.py.224: train: epoch 109 | batch 330/2304 [I 2022-10-28 05:44:23] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:08 [I 2022-10-28 05:44:23] ...trap/engines/engine.py.229: process: 0.54602 | load: 0.00245 [I 2022-10-28 05:44:23] ...trap/engines/engine.py.230: loss: 0.10024 [I 2022-10-28 05:44:23] ...optimizers/trijoint.py.151: total_norm: 15.719666 [I 2022-10-28 05:44:28] ...trap/engines/engine.py.224: train: epoch 109 | batch 340/2304 [I 2022-10-28 05:44:28] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:09 [I 2022-10-28 05:44:28] ...trap/engines/engine.py.229: process: 0.56187 | load: 0.00253 [I 2022-10-28 05:44:28] ...trap/engines/engine.py.230: loss: 0.13008 [I 2022-10-28 05:44:28] ...optimizers/trijoint.py.151: total_norm: 17.243336 [I 2022-10-28 05:44:34] ...trap/engines/engine.py.224: train: epoch 109 | batch 350/2304 [I 2022-10-28 05:44:34] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:06 [I 2022-10-28 05:44:34] ...trap/engines/engine.py.229: process: 0.54664 | load: 0.00244 [I 2022-10-28 05:44:34] ...trap/engines/engine.py.230: loss: 0.09943 [I 2022-10-28 05:44:34] ...optimizers/trijoint.py.151: total_norm: 9.264515 [I 2022-10-28 05:44:39] ...trap/engines/engine.py.224: train: epoch 109 | batch 360/2304 [I 2022-10-28 05:44:39] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:17 [I 2022-10-28 05:44:39] ...trap/engines/engine.py.229: process: 0.56501 | load: 0.00258 [I 2022-10-28 05:44:39] ...trap/engines/engine.py.230: loss: 0.13507 [I 2022-10-28 05:44:39] ...optimizers/trijoint.py.151: total_norm: 33.908733 [I 2022-10-28 05:44:45] ...trap/engines/engine.py.224: train: epoch 109 | batch 370/2304 [I 2022-10-28 05:44:45] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:08 [I 2022-10-28 05:44:45] ...trap/engines/engine.py.229: process: 0.56897 | load: 0.00249 [I 2022-10-28 05:44:45] ...trap/engines/engine.py.230: loss: 0.13082 [I 2022-10-28 05:44:45] ...optimizers/trijoint.py.151: total_norm: 16.288456 [I 2022-10-28 05:44:51] ...trap/engines/engine.py.224: train: epoch 109 | batch 380/2304 [I 2022-10-28 05:44:51] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:54 [I 2022-10-28 05:44:51] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00246 [I 2022-10-28 05:44:51] ...trap/engines/engine.py.230: loss: 0.10207 [I 2022-10-28 05:44:51] ...optimizers/trijoint.py.151: total_norm: 10.211215 [I 2022-10-28 05:44:56] ...trap/engines/engine.py.224: train: epoch 109 | batch 390/2304 [I 2022-10-28 05:44:56] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:50 [I 2022-10-28 05:44:56] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00256 [I 2022-10-28 05:44:56] ...trap/engines/engine.py.230: loss: 0.15890 [I 2022-10-28 05:44:56] ...optimizers/trijoint.py.151: total_norm: 23.727072 [I 2022-10-28 05:45:02] ...trap/engines/engine.py.224: train: epoch 109 | batch 400/2304 [I 2022-10-28 05:45:02] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:51 [I 2022-10-28 05:45:02] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00244 [I 2022-10-28 05:45:02] ...trap/engines/engine.py.230: loss: 0.13353 [I 2022-10-28 05:45:02] ...optimizers/trijoint.py.151: total_norm: 26.330332 [I 2022-10-28 05:45:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 410/2304 [I 2022-10-28 05:45:07] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:33 [I 2022-10-28 05:45:07] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00259 [I 2022-10-28 05:45:07] ...trap/engines/engine.py.230: loss: 0.09342 [I 2022-10-28 05:45:07] ...optimizers/trijoint.py.151: total_norm: 10.457936 [I 2022-10-28 05:45:13] ...trap/engines/engine.py.224: train: epoch 109 | batch 420/2304 [I 2022-10-28 05:45:13] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:32 [I 2022-10-28 05:45:13] ...trap/engines/engine.py.229: process: 0.55542 | load: 0.00255 [I 2022-10-28 05:45:13] ...trap/engines/engine.py.230: loss: 0.09864 [I 2022-10-28 05:45:13] ...optimizers/trijoint.py.151: total_norm: 10.971572 [I 2022-10-28 05:45:19] ...trap/engines/engine.py.224: train: epoch 109 | batch 430/2304 [I 2022-10-28 05:45:19] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:22 [I 2022-10-28 05:45:19] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00246 [I 2022-10-28 05:45:19] ...trap/engines/engine.py.230: loss: 0.13560 [I 2022-10-28 05:45:19] ...optimizers/trijoint.py.151: total_norm: 16.367222 [I 2022-10-28 05:45:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 440/2304 [I 2022-10-28 05:45:24] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:16 [I 2022-10-28 05:45:24] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00246 [I 2022-10-28 05:45:24] ...trap/engines/engine.py.230: loss: 0.10716 [I 2022-10-28 05:45:24] ...optimizers/trijoint.py.151: total_norm: 15.334183 [I 2022-10-28 05:45:30] ...trap/engines/engine.py.224: train: epoch 109 | batch 450/2304 [I 2022-10-28 05:45:30] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:11 [I 2022-10-28 05:45:30] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00249 [I 2022-10-28 05:45:30] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-28 05:45:30] ...optimizers/trijoint.py.151: total_norm: 15.413877 [I 2022-10-28 05:45:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 460/2304 [I 2022-10-28 05:45:35] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:01 [I 2022-10-28 05:45:35] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00255 [I 2022-10-28 05:45:35] ...trap/engines/engine.py.230: loss: 0.19811 [I 2022-10-28 05:45:35] ...optimizers/trijoint.py.151: total_norm: 22.560072 [I 2022-10-28 05:45:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 470/2304 [I 2022-10-28 05:45:41] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:55 [I 2022-10-28 05:45:41] ...trap/engines/engine.py.229: process: 0.54647 | load: 0.00246 [I 2022-10-28 05:45:41] ...trap/engines/engine.py.230: loss: 0.10454 [I 2022-10-28 05:45:41] ...optimizers/trijoint.py.151: total_norm: 11.913738 [I 2022-10-28 05:45:47] ...trap/engines/engine.py.224: train: epoch 109 | batch 480/2304 [I 2022-10-28 05:45:47] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:08 [I 2022-10-28 05:45:47] ...trap/engines/engine.py.229: process: 0.55399 | load: 0.00246 [I 2022-10-28 05:45:47] ...trap/engines/engine.py.230: loss: 0.13372 [I 2022-10-28 05:45:47] ...optimizers/trijoint.py.151: total_norm: 22.540752 [I 2022-10-28 05:45:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 490/2304 [I 2022-10-28 05:45:52] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:53 [I 2022-10-28 05:45:52] ...trap/engines/engine.py.229: process: 0.55544 | load: 0.00246 [I 2022-10-28 05:45:52] ...trap/engines/engine.py.230: loss: 0.20200 [I 2022-10-28 05:45:52] ...optimizers/trijoint.py.151: total_norm: 14.455599 [I 2022-10-28 05:45:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 500/2304 [I 2022-10-28 05:45:58] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:42 [I 2022-10-28 05:45:58] ...trap/engines/engine.py.229: process: 0.55079 | load: 0.00243 [I 2022-10-28 05:45:58] ...trap/engines/engine.py.230: loss: 0.08267 [I 2022-10-28 05:45:58] ...optimizers/trijoint.py.151: total_norm: 14.705678 [I 2022-10-28 05:46:03] ...trap/engines/engine.py.224: train: epoch 109 | batch 510/2304 [I 2022-10-28 05:46:03] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:34 [I 2022-10-28 05:46:03] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00243 [I 2022-10-28 05:46:03] ...trap/engines/engine.py.230: loss: 0.08046 [I 2022-10-28 05:46:03] ...optimizers/trijoint.py.151: total_norm: 12.988603 [I 2022-10-28 05:46:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 520/2304 [I 2022-10-28 05:46:09] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:35 [I 2022-10-28 05:46:09] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00243 [I 2022-10-28 05:46:09] ...trap/engines/engine.py.230: loss: 0.13069 [I 2022-10-28 05:46:09] ...optimizers/trijoint.py.151: total_norm: 11.091862 [I 2022-10-28 05:46:14] ...trap/engines/engine.py.224: train: epoch 109 | batch 530/2304 [I 2022-10-28 05:46:14] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:23 [I 2022-10-28 05:46:14] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00242 [I 2022-10-28 05:46:14] ...trap/engines/engine.py.230: loss: 0.11200 [I 2022-10-28 05:46:14] ...optimizers/trijoint.py.151: total_norm: 11.196543 [I 2022-10-28 05:46:20] ...trap/engines/engine.py.224: train: epoch 109 | batch 540/2304 [I 2022-10-28 05:46:20] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:30 [I 2022-10-28 05:46:20] ...trap/engines/engine.py.229: process: 0.56462 | load: 0.00248 [I 2022-10-28 05:46:20] ...trap/engines/engine.py.230: loss: 0.12873 [I 2022-10-28 05:46:20] ...optimizers/trijoint.py.151: total_norm: 22.097052 [I 2022-10-28 05:46:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 550/2304 [I 2022-10-28 05:46:26] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:18 [I 2022-10-28 05:46:26] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00251 [I 2022-10-28 05:46:26] ...trap/engines/engine.py.230: loss: 0.19348 [I 2022-10-28 05:46:26] ...optimizers/trijoint.py.151: total_norm: 37.874943 [I 2022-10-28 05:46:31] ...trap/engines/engine.py.224: train: epoch 109 | batch 560/2304 [I 2022-10-28 05:46:31] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:15 [I 2022-10-28 05:46:31] ...trap/engines/engine.py.229: process: 0.56121 | load: 0.00253 [I 2022-10-28 05:46:31] ...trap/engines/engine.py.230: loss: 0.09796 [I 2022-10-28 05:46:31] ...optimizers/trijoint.py.151: total_norm: 23.726772 [I 2022-10-28 05:46:37] ...trap/engines/engine.py.224: train: epoch 109 | batch 570/2304 [I 2022-10-28 05:46:37] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:06 [I 2022-10-28 05:46:37] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00244 [I 2022-10-28 05:46:37] ...trap/engines/engine.py.230: loss: 0.12740 [I 2022-10-28 05:46:37] ...optimizers/trijoint.py.151: total_norm: 12.550554 [I 2022-10-28 05:46:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 580/2304 [I 2022-10-28 05:46:42] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:01 [I 2022-10-28 05:46:42] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00245 [I 2022-10-28 05:46:42] ...trap/engines/engine.py.230: loss: 0.10779 [I 2022-10-28 05:46:42] ...optimizers/trijoint.py.151: total_norm: 9.468069 [I 2022-10-28 05:46:48] ...trap/engines/engine.py.224: train: epoch 109 | batch 590/2304 [I 2022-10-28 05:46:48] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:41 [I 2022-10-28 05:46:48] ...trap/engines/engine.py.229: process: 0.53429 | load: 0.00246 [I 2022-10-28 05:46:48] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-28 05:46:48] ...optimizers/trijoint.py.151: total_norm: 50.802731 [I 2022-10-28 05:46:54] ...trap/engines/engine.py.224: train: epoch 109 | batch 600/2304 [I 2022-10-28 05:46:54] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:16:07 [I 2022-10-28 05:46:54] ...trap/engines/engine.py.229: process: 0.57269 | load: 0.00243 [I 2022-10-28 05:46:54] ...trap/engines/engine.py.230: loss: 0.19092 [I 2022-10-28 05:46:54] ...optimizers/trijoint.py.151: total_norm: 17.847212 [I 2022-10-28 05:46:59] ...trap/engines/engine.py.224: train: epoch 109 | batch 610/2304 [I 2022-10-28 05:46:59] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:16:34 [I 2022-10-28 05:46:59] ...trap/engines/engine.py.229: process: 0.68976 | load: 0.01016 [I 2022-10-28 05:46:59] ...trap/engines/engine.py.230: loss: 0.09680 [I 2022-10-28 05:46:59] ...optimizers/trijoint.py.151: total_norm: 19.534725 [I 2022-10-28 05:47:05] ...trap/engines/engine.py.224: train: epoch 109 | batch 620/2304 [I 2022-10-28 05:47:05] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:38 [I 2022-10-28 05:47:05] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00255 [I 2022-10-28 05:47:05] ...trap/engines/engine.py.230: loss: 0.13409 [I 2022-10-28 05:47:05] ...optimizers/trijoint.py.151: total_norm: 18.593676 [I 2022-10-28 05:47:11] ...trap/engines/engine.py.224: train: epoch 109 | batch 630/2304 [I 2022-10-28 05:47:11] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:29 [I 2022-10-28 05:47:11] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00254 [I 2022-10-28 05:47:11] ...trap/engines/engine.py.230: loss: 0.13819 [I 2022-10-28 05:47:11] ...optimizers/trijoint.py.151: total_norm: 10.875978 [I 2022-10-28 05:47:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 640/2304 [I 2022-10-28 05:47:16] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:39 [I 2022-10-28 05:47:16] ...trap/engines/engine.py.229: process: 0.56808 | load: 0.00278 [I 2022-10-28 05:47:16] ...trap/engines/engine.py.230: loss: 0.11276 [I 2022-10-28 05:47:16] ...optimizers/trijoint.py.151: total_norm: 12.570468 [I 2022-10-28 05:47:22] ...trap/engines/engine.py.224: train: epoch 109 | batch 650/2304 [I 2022-10-28 05:47:22] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:21 [I 2022-10-28 05:47:22] ...trap/engines/engine.py.229: process: 0.56741 | load: 0.00243 [I 2022-10-28 05:47:22] ...trap/engines/engine.py.230: loss: 0.15698 [I 2022-10-28 05:47:22] ...optimizers/trijoint.py.151: total_norm: 15.578339 [I 2022-10-28 05:47:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 660/2304 [I 2022-10-28 05:47:27] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:15 [I 2022-10-28 05:47:27] ...trap/engines/engine.py.229: process: 0.55422 | load: 0.00241 [I 2022-10-28 05:47:27] ...trap/engines/engine.py.230: loss: 0.16210 [I 2022-10-28 05:47:27] ...optimizers/trijoint.py.151: total_norm: 22.886621 [I 2022-10-28 05:47:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 670/2304 [I 2022-10-28 05:47:33] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:08 [I 2022-10-28 05:47:33] ...trap/engines/engine.py.229: process: 0.55589 | load: 0.00246 [I 2022-10-28 05:47:33] ...trap/engines/engine.py.230: loss: 0.11564 [I 2022-10-28 05:47:33] ...optimizers/trijoint.py.151: total_norm: 10.330311 [I 2022-10-28 05:47:39] ...trap/engines/engine.py.224: train: epoch 109 | batch 680/2304 [I 2022-10-28 05:47:39] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:08 [I 2022-10-28 05:47:39] ...trap/engines/engine.py.229: process: 0.55049 | load: 0.00244 [I 2022-10-28 05:47:39] ...trap/engines/engine.py.230: loss: 0.10249 [I 2022-10-28 05:47:39] ...optimizers/trijoint.py.151: total_norm: 16.545721 [I 2022-10-28 05:47:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 690/2304 [I 2022-10-28 05:47:44] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:06 [I 2022-10-28 05:47:44] ...trap/engines/engine.py.229: process: 0.56423 | load: 0.00291 [I 2022-10-28 05:47:44] ...trap/engines/engine.py.230: loss: 0.13782 [I 2022-10-28 05:47:44] ...optimizers/trijoint.py.151: total_norm: 15.801770 [I 2022-10-28 05:47:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 700/2304 [I 2022-10-28 05:47:50] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:54 [I 2022-10-28 05:47:50] ...trap/engines/engine.py.229: process: 0.55089 | load: 0.00243 [I 2022-10-28 05:47:50] ...trap/engines/engine.py.230: loss: 0.16125 [I 2022-10-28 05:47:50] ...optimizers/trijoint.py.151: total_norm: 15.058154 [I 2022-10-28 05:47:55] ...trap/engines/engine.py.224: train: epoch 109 | batch 710/2304 [I 2022-10-28 05:47:55] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:53 [I 2022-10-28 05:47:55] ...trap/engines/engine.py.229: process: 0.55211 | load: 0.00254 [I 2022-10-28 05:47:55] ...trap/engines/engine.py.230: loss: 0.11637 [I 2022-10-28 05:47:55] ...optimizers/trijoint.py.151: total_norm: 34.412224 [I 2022-10-28 05:48:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 720/2304 [I 2022-10-28 05:48:01] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:44 [I 2022-10-28 05:48:01] ...trap/engines/engine.py.229: process: 0.55873 | load: 0.00251 [I 2022-10-28 05:48:01] ...trap/engines/engine.py.230: loss: 0.11249 [I 2022-10-28 05:48:01] ...optimizers/trijoint.py.151: total_norm: 26.525141 [I 2022-10-28 05:48:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 730/2304 [I 2022-10-28 05:48:07] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:37 [I 2022-10-28 05:48:07] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00253 [I 2022-10-28 05:48:07] ...trap/engines/engine.py.230: loss: 0.09317 [I 2022-10-28 05:48:07] ...optimizers/trijoint.py.151: total_norm: 16.682150 [I 2022-10-28 05:48:12] ...trap/engines/engine.py.224: train: epoch 109 | batch 740/2304 [I 2022-10-28 05:48:12] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:28 [I 2022-10-28 05:48:12] ...trap/engines/engine.py.229: process: 0.55823 | load: 0.00246 [I 2022-10-28 05:48:12] ...trap/engines/engine.py.230: loss: 0.12457 [I 2022-10-28 05:48:12] ...optimizers/trijoint.py.151: total_norm: 10.431401 [I 2022-10-28 05:48:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 750/2304 [I 2022-10-28 05:48:18] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:41 [I 2022-10-28 05:48:18] ...trap/engines/engine.py.229: process: 0.61686 | load: 0.00241 [I 2022-10-28 05:48:18] ...trap/engines/engine.py.230: loss: 0.13355 [I 2022-10-28 05:48:18] ...optimizers/trijoint.py.151: total_norm: 10.651387 [I 2022-10-28 05:48:23] ...trap/engines/engine.py.224: train: epoch 109 | batch 760/2304 [I 2022-10-28 05:48:23] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:29 [I 2022-10-28 05:48:23] ...trap/engines/engine.py.229: process: 0.56786 | load: 0.00246 [I 2022-10-28 05:48:23] ...trap/engines/engine.py.230: loss: 0.11219 [I 2022-10-28 05:48:23] ...optimizers/trijoint.py.151: total_norm: 18.470480 [I 2022-10-28 05:48:29] ...trap/engines/engine.py.224: train: epoch 109 | batch 770/2304 [I 2022-10-28 05:48:29] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:18 [I 2022-10-28 05:48:29] ...trap/engines/engine.py.229: process: 0.55884 | load: 0.00244 [I 2022-10-28 05:48:29] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-28 05:48:29] ...optimizers/trijoint.py.151: total_norm: 18.958916 [I 2022-10-28 05:48:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 780/2304 [I 2022-10-28 05:48:35] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:12 [I 2022-10-28 05:48:35] ...trap/engines/engine.py.229: process: 0.54448 | load: 0.00245 [I 2022-10-28 05:48:35] ...trap/engines/engine.py.230: loss: 0.13291 [I 2022-10-28 05:48:35] ...optimizers/trijoint.py.151: total_norm: 26.239674 [I 2022-10-28 05:48:40] ...trap/engines/engine.py.224: train: epoch 109 | batch 790/2304 [I 2022-10-28 05:48:40] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:13:57 [I 2022-10-28 05:48:40] ...trap/engines/engine.py.229: process: 0.54395 | load: 0.00245 [I 2022-10-28 05:48:40] ...trap/engines/engine.py.230: loss: 0.13491 [I 2022-10-28 05:48:40] ...optimizers/trijoint.py.151: total_norm: 15.479413 [I 2022-10-28 05:48:46] ...trap/engines/engine.py.224: train: epoch 109 | batch 800/2304 [I 2022-10-28 05:48:46] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:54 [I 2022-10-28 05:48:46] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00257 [I 2022-10-28 05:48:46] ...trap/engines/engine.py.230: loss: 0.21824 [I 2022-10-28 05:48:46] ...optimizers/trijoint.py.151: total_norm: 12.547223 [I 2022-10-28 05:48:51] ...trap/engines/engine.py.224: train: epoch 109 | batch 810/2304 [I 2022-10-28 05:48:51] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:59 [I 2022-10-28 05:48:51] ...trap/engines/engine.py.229: process: 0.58417 | load: 0.00286 [I 2022-10-28 05:48:51] ...trap/engines/engine.py.230: loss: 0.11434 [I 2022-10-28 05:48:51] ...optimizers/trijoint.py.151: total_norm: 8.291975 [I 2022-10-28 05:48:57] ...trap/engines/engine.py.224: train: epoch 109 | batch 820/2304 [I 2022-10-28 05:48:57] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:52 [I 2022-10-28 05:48:57] ...trap/engines/engine.py.229: process: 0.54663 | load: 0.00254 [I 2022-10-28 05:48:57] ...trap/engines/engine.py.230: loss: 0.12225 [I 2022-10-28 05:48:57] ...optimizers/trijoint.py.151: total_norm: 13.731956 [I 2022-10-28 05:49:03] ...trap/engines/engine.py.224: train: epoch 109 | batch 830/2304 [I 2022-10-28 05:49:03] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:44 [I 2022-10-28 05:49:03] ...trap/engines/engine.py.229: process: 0.55112 | load: 0.00246 [I 2022-10-28 05:49:03] ...trap/engines/engine.py.230: loss: 0.10226 [I 2022-10-28 05:49:03] ...optimizers/trijoint.py.151: total_norm: 14.878820 [I 2022-10-28 05:49:08] ...trap/engines/engine.py.224: train: epoch 109 | batch 840/2304 [I 2022-10-28 05:49:08] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:43 [I 2022-10-28 05:49:08] ...trap/engines/engine.py.229: process: 0.57547 | load: 0.00247 [I 2022-10-28 05:49:08] ...trap/engines/engine.py.230: loss: 0.10054 [I 2022-10-28 05:49:08] ...optimizers/trijoint.py.151: total_norm: 11.737923 [I 2022-10-28 05:49:14] ...trap/engines/engine.py.224: train: epoch 109 | batch 850/2304 [I 2022-10-28 05:49:14] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:30 [I 2022-10-28 05:49:14] ...trap/engines/engine.py.229: process: 0.54893 | load: 0.00241 [I 2022-10-28 05:49:14] ...trap/engines/engine.py.230: loss: 0.12307 [I 2022-10-28 05:49:14] ...optimizers/trijoint.py.151: total_norm: 13.896352 [I 2022-10-28 05:49:20] ...trap/engines/engine.py.224: train: epoch 109 | batch 860/2304 [I 2022-10-28 05:49:20] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:25 [I 2022-10-28 05:49:20] ...trap/engines/engine.py.229: process: 0.56114 | load: 0.00244 [I 2022-10-28 05:49:20] ...trap/engines/engine.py.230: loss: 0.10733 [I 2022-10-28 05:49:20] ...optimizers/trijoint.py.151: total_norm: 19.738594 [I 2022-10-28 05:49:25] ...trap/engines/engine.py.224: train: epoch 109 | batch 870/2304 [I 2022-10-28 05:49:25] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:19 [I 2022-10-28 05:49:25] ...trap/engines/engine.py.229: process: 0.56368 | load: 0.00243 [I 2022-10-28 05:49:25] ...trap/engines/engine.py.230: loss: 0.17243 [I 2022-10-28 05:49:25] ...optimizers/trijoint.py.151: total_norm: 24.394104 [I 2022-10-28 05:49:31] ...trap/engines/engine.py.224: train: epoch 109 | batch 880/2304 [I 2022-10-28 05:49:31] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:23 [I 2022-10-28 05:49:31] ...trap/engines/engine.py.229: process: 0.55160 | load: 0.00245 [I 2022-10-28 05:49:31] ...trap/engines/engine.py.230: loss: 0.14725 [I 2022-10-28 05:49:31] ...optimizers/trijoint.py.151: total_norm: 16.744684 [I 2022-10-28 05:49:36] ...trap/engines/engine.py.224: train: epoch 109 | batch 890/2304 [I 2022-10-28 05:49:36] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:08 [I 2022-10-28 05:49:36] ...trap/engines/engine.py.229: process: 0.55256 | load: 0.00243 [I 2022-10-28 05:49:36] ...trap/engines/engine.py.230: loss: 0.08224 [I 2022-10-28 05:49:36] ...optimizers/trijoint.py.151: total_norm: 15.142985 [I 2022-10-28 05:49:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 900/2304 [I 2022-10-28 05:49:42] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:03 [I 2022-10-28 05:49:42] ...trap/engines/engine.py.229: process: 0.56508 | load: 0.00244 [I 2022-10-28 05:49:42] ...trap/engines/engine.py.230: loss: 0.10588 [I 2022-10-28 05:49:42] ...optimizers/trijoint.py.151: total_norm: 8.986519 [I 2022-10-28 05:49:48] ...trap/engines/engine.py.224: train: epoch 109 | batch 910/2304 [I 2022-10-28 05:49:48] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:13:01 [I 2022-10-28 05:49:48] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00241 [I 2022-10-28 05:49:48] ...trap/engines/engine.py.230: loss: 0.12797 [I 2022-10-28 05:49:48] ...optimizers/trijoint.py.151: total_norm: 10.734299 [I 2022-10-28 05:49:53] ...trap/engines/engine.py.224: train: epoch 109 | batch 920/2304 [I 2022-10-28 05:49:53] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:55 [I 2022-10-28 05:49:53] ...trap/engines/engine.py.229: process: 0.56639 | load: 0.00241 [I 2022-10-28 05:49:53] ...trap/engines/engine.py.230: loss: 0.16131 [I 2022-10-28 05:49:53] ...optimizers/trijoint.py.151: total_norm: 40.982433 [I 2022-10-28 05:49:59] ...trap/engines/engine.py.224: train: epoch 109 | batch 930/2304 [I 2022-10-28 05:49:59] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:43 [I 2022-10-28 05:49:59] ...trap/engines/engine.py.229: process: 0.54744 | load: 0.00251 [I 2022-10-28 05:49:59] ...trap/engines/engine.py.230: loss: 0.12358 [I 2022-10-28 05:49:59] ...optimizers/trijoint.py.151: total_norm: 18.598976 [I 2022-10-28 05:50:04] ...trap/engines/engine.py.224: train: epoch 109 | batch 940/2304 [I 2022-10-28 05:50:04] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:40 [I 2022-10-28 05:50:04] ...trap/engines/engine.py.229: process: 0.56452 | load: 0.00253 [I 2022-10-28 05:50:04] ...trap/engines/engine.py.230: loss: 0.09896 [I 2022-10-28 05:50:04] ...optimizers/trijoint.py.151: total_norm: 14.968207 [I 2022-10-28 05:50:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 950/2304 [I 2022-10-28 05:50:10] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:32 [I 2022-10-28 05:50:10] ...trap/engines/engine.py.229: process: 0.54618 | load: 0.00249 [I 2022-10-28 05:50:10] ...trap/engines/engine.py.230: loss: 0.14082 [I 2022-10-28 05:50:10] ...optimizers/trijoint.py.151: total_norm: 15.013270 [I 2022-10-28 05:50:15] ...trap/engines/engine.py.224: train: epoch 109 | batch 960/2304 [I 2022-10-28 05:50:15] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:33 [I 2022-10-28 05:50:15] ...trap/engines/engine.py.229: process: 0.56460 | load: 0.00245 [I 2022-10-28 05:50:15] ...trap/engines/engine.py.230: loss: 0.10239 [I 2022-10-28 05:50:15] ...optimizers/trijoint.py.151: total_norm: 13.382928 [I 2022-10-28 05:50:21] ...trap/engines/engine.py.224: train: epoch 109 | batch 970/2304 [I 2022-10-28 05:50:21] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:23 [I 2022-10-28 05:50:21] ...trap/engines/engine.py.229: process: 0.54960 | load: 0.00247 [I 2022-10-28 05:50:21] ...trap/engines/engine.py.230: loss: 0.11351 [I 2022-10-28 05:50:21] ...optimizers/trijoint.py.151: total_norm: 7.687190 [I 2022-10-28 05:50:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 980/2304 [I 2022-10-28 05:50:27] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:19 [I 2022-10-28 05:50:27] ...trap/engines/engine.py.229: process: 0.57028 | load: 0.00247 [I 2022-10-28 05:50:27] ...trap/engines/engine.py.230: loss: 0.10856 [I 2022-10-28 05:50:27] ...optimizers/trijoint.py.151: total_norm: 23.448847 [I 2022-10-28 05:50:32] ...trap/engines/engine.py.224: train: epoch 109 | batch 990/2304 [I 2022-10-28 05:50:32] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:17 [I 2022-10-28 05:50:32] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00244 [I 2022-10-28 05:50:32] ...trap/engines/engine.py.230: loss: 0.08758 [I 2022-10-28 05:50:32] ...optimizers/trijoint.py.151: total_norm: 9.398020 [I 2022-10-28 05:50:38] ...trap/engines/engine.py.224: train: epoch 109 | batch 1000/2304 [I 2022-10-28 05:50:38] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:17 [I 2022-10-28 05:50:38] ...trap/engines/engine.py.229: process: 0.55028 | load: 0.00249 [I 2022-10-28 05:50:38] ...trap/engines/engine.py.230: loss: 0.21255 [I 2022-10-28 05:50:38] ...optimizers/trijoint.py.151: total_norm: 25.556143 [I 2022-10-28 05:50:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 1010/2304 [I 2022-10-28 05:50:44] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:10 [I 2022-10-28 05:50:44] ...trap/engines/engine.py.229: process: 0.56869 | load: 0.00250 [I 2022-10-28 05:50:44] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-28 05:50:44] ...optimizers/trijoint.py.151: total_norm: 22.474569 [I 2022-10-28 05:50:49] ...trap/engines/engine.py.224: train: epoch 109 | batch 1020/2304 [I 2022-10-28 05:50:49] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:55 [I 2022-10-28 05:50:49] ...trap/engines/engine.py.229: process: 0.54812 | load: 0.00248 [I 2022-10-28 05:50:49] ...trap/engines/engine.py.230: loss: 0.11564 [I 2022-10-28 05:50:49] ...optimizers/trijoint.py.151: total_norm: 19.140114 [I 2022-10-28 05:50:55] ...trap/engines/engine.py.224: train: epoch 109 | batch 1030/2304 [I 2022-10-28 05:50:55] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:56 [I 2022-10-28 05:50:55] ...trap/engines/engine.py.229: process: 0.56716 | load: 0.00249 [I 2022-10-28 05:50:55] ...trap/engines/engine.py.230: loss: 0.11524 [I 2022-10-28 05:50:55] ...optimizers/trijoint.py.151: total_norm: 24.331970 [I 2022-10-28 05:51:00] ...trap/engines/engine.py.224: train: epoch 109 | batch 1040/2304 [I 2022-10-28 05:51:00] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:41 [I 2022-10-28 05:51:00] ...trap/engines/engine.py.229: process: 0.56260 | load: 0.00246 [I 2022-10-28 05:51:00] ...trap/engines/engine.py.230: loss: 0.15367 [I 2022-10-28 05:51:00] ...optimizers/trijoint.py.151: total_norm: 11.203614 [I 2022-10-28 05:51:06] ...trap/engines/engine.py.224: train: epoch 109 | batch 1050/2304 [I 2022-10-28 05:51:06] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:39 [I 2022-10-28 05:51:06] ...trap/engines/engine.py.229: process: 0.55624 | load: 0.00247 [I 2022-10-28 05:51:06] ...trap/engines/engine.py.230: loss: 0.14503 [I 2022-10-28 05:51:06] ...optimizers/trijoint.py.151: total_norm: 19.015358 [I 2022-10-28 05:51:12] ...trap/engines/engine.py.224: train: epoch 109 | batch 1060/2304 [I 2022-10-28 05:51:12] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:36 [I 2022-10-28 05:51:12] ...trap/engines/engine.py.229: process: 0.56298 | load: 0.00248 [I 2022-10-28 05:51:12] ...trap/engines/engine.py.230: loss: 0.10875 [I 2022-10-28 05:51:12] ...optimizers/trijoint.py.151: total_norm: 16.791348 [I 2022-10-28 05:51:17] ...trap/engines/engine.py.224: train: epoch 109 | batch 1070/2304 [I 2022-10-28 05:51:17] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:29 [I 2022-10-28 05:51:17] ...trap/engines/engine.py.229: process: 0.57740 | load: 0.00246 [I 2022-10-28 05:51:17] ...trap/engines/engine.py.230: loss: 0.17845 [I 2022-10-28 05:51:17] ...optimizers/trijoint.py.151: total_norm: 9.910170 [I 2022-10-28 05:51:23] ...trap/engines/engine.py.224: train: epoch 109 | batch 1080/2304 [I 2022-10-28 05:51:23] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:16 [I 2022-10-28 05:51:23] ...trap/engines/engine.py.229: process: 0.54656 | load: 0.00242 [I 2022-10-28 05:51:23] ...trap/engines/engine.py.230: loss: 0.16987 [I 2022-10-28 05:51:23] ...optimizers/trijoint.py.151: total_norm: 10.575706 [I 2022-10-28 05:51:28] ...trap/engines/engine.py.224: train: epoch 109 | batch 1090/2304 [I 2022-10-28 05:51:28] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:19 [I 2022-10-28 05:51:28] ...trap/engines/engine.py.229: process: 0.56722 | load: 0.00239 [I 2022-10-28 05:51:28] ...trap/engines/engine.py.230: loss: 0.08371 [I 2022-10-28 05:51:28] ...optimizers/trijoint.py.151: total_norm: 11.059410 [I 2022-10-28 05:51:34] ...trap/engines/engine.py.224: train: epoch 109 | batch 1100/2304 [I 2022-10-28 05:51:34] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:11 [I 2022-10-28 05:51:34] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00244 [I 2022-10-28 05:51:34] ...trap/engines/engine.py.230: loss: 0.10586 [I 2022-10-28 05:51:34] ...optimizers/trijoint.py.151: total_norm: 12.447736 [I 2022-10-28 05:51:39] ...trap/engines/engine.py.224: train: epoch 109 | batch 1110/2304 [I 2022-10-28 05:51:39] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:04 [I 2022-10-28 05:51:39] ...trap/engines/engine.py.229: process: 0.56320 | load: 0.00240 [I 2022-10-28 05:51:39] ...trap/engines/engine.py.230: loss: 0.19434 [I 2022-10-28 05:51:39] ...optimizers/trijoint.py.151: total_norm: 18.163111 [I 2022-10-28 05:51:45] ...trap/engines/engine.py.224: train: epoch 109 | batch 1120/2304 [I 2022-10-28 05:51:45] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:08 [I 2022-10-28 05:51:45] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00271 [I 2022-10-28 05:51:45] ...trap/engines/engine.py.230: loss: 0.13198 [I 2022-10-28 05:51:45] ...optimizers/trijoint.py.151: total_norm: 18.611977 [I 2022-10-28 05:51:51] ...trap/engines/engine.py.224: train: epoch 109 | batch 1130/2304 [I 2022-10-28 05:51:51] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:53 [I 2022-10-28 05:51:51] ...trap/engines/engine.py.229: process: 0.55635 | load: 0.00247 [I 2022-10-28 05:51:51] ...trap/engines/engine.py.230: loss: 0.10576 [I 2022-10-28 05:51:51] ...optimizers/trijoint.py.151: total_norm: 27.475382 [I 2022-10-28 05:51:56] ...trap/engines/engine.py.224: train: epoch 109 | batch 1140/2304 [I 2022-10-28 05:51:56] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:10:49 [I 2022-10-28 05:51:56] ...trap/engines/engine.py.229: process: 0.55814 | load: 0.00246 [I 2022-10-28 05:51:56] ...trap/engines/engine.py.230: loss: 0.09735 [I 2022-10-28 05:51:56] ...optimizers/trijoint.py.151: total_norm: 13.528446 [I 2022-10-28 05:52:02] ...trap/engines/engine.py.224: train: epoch 109 | batch 1150/2304 [I 2022-10-28 05:52:02] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:45 [I 2022-10-28 05:52:02] ...trap/engines/engine.py.229: process: 0.56329 | load: 0.00244 [I 2022-10-28 05:52:02] ...trap/engines/engine.py.230: loss: 0.13851 [I 2022-10-28 05:52:02] ...optimizers/trijoint.py.151: total_norm: 27.535276 [I 2022-10-28 05:52:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 1160/2304 [I 2022-10-28 05:52:07] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:38 [I 2022-10-28 05:52:07] ...trap/engines/engine.py.229: process: 0.56748 | load: 0.00245 [I 2022-10-28 05:52:07] ...trap/engines/engine.py.230: loss: 0.13510 [I 2022-10-28 05:52:07] ...optimizers/trijoint.py.151: total_norm: 29.347168 [I 2022-10-28 05:52:13] ...trap/engines/engine.py.224: train: epoch 109 | batch 1170/2304 [I 2022-10-28 05:52:13] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:30 [I 2022-10-28 05:52:13] ...trap/engines/engine.py.229: process: 0.56299 | load: 0.00248 [I 2022-10-28 05:52:13] ...trap/engines/engine.py.230: loss: 0.14718 [I 2022-10-28 05:52:13] ...optimizers/trijoint.py.151: total_norm: 17.117910 [I 2022-10-28 05:52:19] ...trap/engines/engine.py.224: train: epoch 109 | batch 1180/2304 [I 2022-10-28 05:52:19] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:25 [I 2022-10-28 05:52:19] ...trap/engines/engine.py.229: process: 0.55294 | load: 0.00247 [I 2022-10-28 05:52:19] ...trap/engines/engine.py.230: loss: 0.10259 [I 2022-10-28 05:52:19] ...optimizers/trijoint.py.151: total_norm: 12.858520 [I 2022-10-28 05:52:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 1190/2304 [I 2022-10-28 05:52:24] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:22 [I 2022-10-28 05:52:24] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00244 [I 2022-10-28 05:52:24] ...trap/engines/engine.py.230: loss: 0.11069 [I 2022-10-28 05:52:24] ...optimizers/trijoint.py.151: total_norm: 14.261999 [I 2022-10-28 05:52:30] ...trap/engines/engine.py.224: train: epoch 109 | batch 1200/2304 [I 2022-10-28 05:52:30] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:14 [I 2022-10-28 05:52:30] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00255 [I 2022-10-28 05:52:30] ...trap/engines/engine.py.230: loss: 0.15871 [I 2022-10-28 05:52:30] ...optimizers/trijoint.py.151: total_norm: 26.429251 [I 2022-10-28 05:52:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 1210/2304 [I 2022-10-28 05:52:35] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:13 [I 2022-10-28 05:52:35] ...trap/engines/engine.py.229: process: 0.56625 | load: 0.00250 [I 2022-10-28 05:52:35] ...trap/engines/engine.py.230: loss: 0.12958 [I 2022-10-28 05:52:35] ...optimizers/trijoint.py.151: total_norm: 21.431482 [I 2022-10-28 05:52:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 1220/2304 [I 2022-10-28 05:52:41] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:06 [I 2022-10-28 05:52:41] ...trap/engines/engine.py.229: process: 0.55805 | load: 0.00248 [I 2022-10-28 05:52:41] ...trap/engines/engine.py.230: loss: 0.11271 [I 2022-10-28 05:52:41] ...optimizers/trijoint.py.151: total_norm: 13.217107 [I 2022-10-28 05:52:47] ...trap/engines/engine.py.224: train: epoch 109 | batch 1230/2304 [I 2022-10-28 05:52:47] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:09:58 [I 2022-10-28 05:52:47] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00249 [I 2022-10-28 05:52:47] ...trap/engines/engine.py.230: loss: 0.08831 [I 2022-10-28 05:52:47] ...optimizers/trijoint.py.151: total_norm: 8.571030 [I 2022-10-28 05:52:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 1240/2304 [I 2022-10-28 05:52:52] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:56 [I 2022-10-28 05:52:52] ...trap/engines/engine.py.229: process: 0.56008 | load: 0.00251 [I 2022-10-28 05:52:52] ...trap/engines/engine.py.230: loss: 0.10935 [I 2022-10-28 05:52:52] ...optimizers/trijoint.py.151: total_norm: 12.723668 [I 2022-10-28 05:52:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 1250/2304 [I 2022-10-28 05:52:58] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:51 [I 2022-10-28 05:52:58] ...trap/engines/engine.py.229: process: 0.57386 | load: 0.00259 [I 2022-10-28 05:52:58] ...trap/engines/engine.py.230: loss: 0.13721 [I 2022-10-28 05:52:58] ...optimizers/trijoint.py.151: total_norm: 17.282871 [I 2022-10-28 05:53:03] ...trap/engines/engine.py.224: train: epoch 109 | batch 1260/2304 [I 2022-10-28 05:53:03] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:41 [I 2022-10-28 05:53:03] ...trap/engines/engine.py.229: process: 0.55726 | load: 0.00245 [I 2022-10-28 05:53:03] ...trap/engines/engine.py.230: loss: 0.11139 [I 2022-10-28 05:53:03] ...optimizers/trijoint.py.151: total_norm: 20.669542 [I 2022-10-28 05:53:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 1270/2304 [I 2022-10-28 05:53:09] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:32 [I 2022-10-28 05:53:09] ...trap/engines/engine.py.229: process: 0.54798 | load: 0.00244 [I 2022-10-28 05:53:09] ...trap/engines/engine.py.230: loss: 0.09181 [I 2022-10-28 05:53:09] ...optimizers/trijoint.py.151: total_norm: 8.972300 [I 2022-10-28 05:53:14] ...trap/engines/engine.py.224: train: epoch 109 | batch 1280/2304 [I 2022-10-28 05:53:14] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:33 [I 2022-10-28 05:53:14] ...trap/engines/engine.py.229: process: 0.56406 | load: 0.00246 [I 2022-10-28 05:53:14] ...trap/engines/engine.py.230: loss: 0.08140 [I 2022-10-28 05:53:14] ...optimizers/trijoint.py.151: total_norm: 14.643601 [I 2022-10-28 05:53:20] ...trap/engines/engine.py.224: train: epoch 109 | batch 1290/2304 [I 2022-10-28 05:53:20] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:20 [I 2022-10-28 05:53:20] ...trap/engines/engine.py.229: process: 0.55484 | load: 0.00243 [I 2022-10-28 05:53:20] ...trap/engines/engine.py.230: loss: 0.11479 [I 2022-10-28 05:53:20] ...optimizers/trijoint.py.151: total_norm: 22.080734 [I 2022-10-28 05:53:26] ...trap/engines/engine.py.224: train: epoch 109 | batch 1300/2304 [I 2022-10-28 05:53:26] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:18 [I 2022-10-28 05:53:26] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00246 [I 2022-10-28 05:53:26] ...trap/engines/engine.py.230: loss: 0.15590 [I 2022-10-28 05:53:26] ...optimizers/trijoint.py.151: total_norm: 27.491331 [I 2022-10-28 05:53:31] ...trap/engines/engine.py.224: train: epoch 109 | batch 1310/2304 [I 2022-10-28 05:53:31] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:14 [I 2022-10-28 05:53:31] ...trap/engines/engine.py.229: process: 0.56676 | load: 0.00245 [I 2022-10-28 05:53:31] ...trap/engines/engine.py.230: loss: 0.15313 [I 2022-10-28 05:53:31] ...optimizers/trijoint.py.151: total_norm: 22.954323 [I 2022-10-28 05:53:37] ...trap/engines/engine.py.224: train: epoch 109 | batch 1320/2304 [I 2022-10-28 05:53:37] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:08 [I 2022-10-28 05:53:37] ...trap/engines/engine.py.229: process: 0.56062 | load: 0.00255 [I 2022-10-28 05:53:37] ...trap/engines/engine.py.230: loss: 0.14427 [I 2022-10-28 05:53:37] ...optimizers/trijoint.py.151: total_norm: 13.882592 [I 2022-10-28 05:53:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 1330/2304 [I 2022-10-28 05:53:42] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:02 [I 2022-10-28 05:53:42] ...trap/engines/engine.py.229: process: 0.55460 | load: 0.00247 [I 2022-10-28 05:53:42] ...trap/engines/engine.py.230: loss: 0.13631 [I 2022-10-28 05:53:42] ...optimizers/trijoint.py.151: total_norm: 38.178974 [I 2022-10-28 05:53:48] ...trap/engines/engine.py.224: train: epoch 109 | batch 1340/2304 [I 2022-10-28 05:53:48] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:57 [I 2022-10-28 05:53:48] ...trap/engines/engine.py.229: process: 0.54655 | load: 0.00251 [I 2022-10-28 05:53:48] ...trap/engines/engine.py.230: loss: 0.08239 [I 2022-10-28 05:53:48] ...optimizers/trijoint.py.151: total_norm: 9.396615 [I 2022-10-28 05:53:54] ...trap/engines/engine.py.224: train: epoch 109 | batch 1350/2304 [I 2022-10-28 05:53:54] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:52 [I 2022-10-28 05:53:54] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00268 [I 2022-10-28 05:53:54] ...trap/engines/engine.py.230: loss: 0.12049 [I 2022-10-28 05:53:54] ...optimizers/trijoint.py.151: total_norm: 15.770913 [I 2022-10-28 05:53:59] ...trap/engines/engine.py.224: train: epoch 109 | batch 1360/2304 [I 2022-10-28 05:53:59] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:46 [I 2022-10-28 05:53:59] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00247 [I 2022-10-28 05:53:59] ...trap/engines/engine.py.230: loss: 0.14915 [I 2022-10-28 05:53:59] ...optimizers/trijoint.py.151: total_norm: 23.055481 [I 2022-10-28 05:54:05] ...trap/engines/engine.py.224: train: epoch 109 | batch 1370/2304 [I 2022-10-28 05:54:05] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:40 [I 2022-10-28 05:54:05] ...trap/engines/engine.py.229: process: 0.56262 | load: 0.00247 [I 2022-10-28 05:54:05] ...trap/engines/engine.py.230: loss: 0.09260 [I 2022-10-28 05:54:05] ...optimizers/trijoint.py.151: total_norm: 13.273416 [I 2022-10-28 05:54:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 1380/2304 [I 2022-10-28 05:54:10] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:40 [I 2022-10-28 05:54:10] ...trap/engines/engine.py.229: process: 0.57126 | load: 0.00247 [I 2022-10-28 05:54:10] ...trap/engines/engine.py.230: loss: 0.10472 [I 2022-10-28 05:54:10] ...optimizers/trijoint.py.151: total_norm: 10.579255 [I 2022-10-28 05:54:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 1390/2304 [I 2022-10-28 05:54:16] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:34 [I 2022-10-28 05:54:16] ...trap/engines/engine.py.229: process: 0.57668 | load: 0.00252 [I 2022-10-28 05:54:16] ...trap/engines/engine.py.230: loss: 0.11381 [I 2022-10-28 05:54:16] ...optimizers/trijoint.py.151: total_norm: 12.988860 [I 2022-10-28 05:54:22] ...trap/engines/engine.py.224: train: epoch 109 | batch 1400/2304 [I 2022-10-28 05:54:22] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:27 [I 2022-10-28 05:54:22] ...trap/engines/engine.py.229: process: 0.56223 | load: 0.00247 [I 2022-10-28 05:54:22] ...trap/engines/engine.py.230: loss: 0.12538 [I 2022-10-28 05:54:22] ...optimizers/trijoint.py.151: total_norm: 16.171831 [I 2022-10-28 05:54:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 1410/2304 [I 2022-10-28 05:54:27] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:20 [I 2022-10-28 05:54:27] ...trap/engines/engine.py.229: process: 0.56216 | load: 0.00247 [I 2022-10-28 05:54:27] ...trap/engines/engine.py.230: loss: 0.13978 [I 2022-10-28 05:54:27] ...optimizers/trijoint.py.151: total_norm: 12.781422 [I 2022-10-28 05:54:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 1420/2304 [I 2022-10-28 05:54:33] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:14 [I 2022-10-28 05:54:33] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00245 [I 2022-10-28 05:54:33] ...trap/engines/engine.py.230: loss: 0.14093 [I 2022-10-28 05:54:33] ...optimizers/trijoint.py.151: total_norm: 12.671752 [I 2022-10-28 05:54:38] ...trap/engines/engine.py.224: train: epoch 109 | batch 1430/2304 [I 2022-10-28 05:54:38] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:05 [I 2022-10-28 05:54:38] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00244 [I 2022-10-28 05:54:38] ...trap/engines/engine.py.230: loss: 0.10311 [I 2022-10-28 05:54:38] ...optimizers/trijoint.py.151: total_norm: 20.987204 [I 2022-10-28 05:54:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 1440/2304 [I 2022-10-28 05:54:44] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:07:59 [I 2022-10-28 05:54:44] ...trap/engines/engine.py.229: process: 0.55198 | load: 0.00244 [I 2022-10-28 05:54:44] ...trap/engines/engine.py.230: loss: 0.08344 [I 2022-10-28 05:54:44] ...optimizers/trijoint.py.151: total_norm: 13.837197 [I 2022-10-28 05:54:49] ...trap/engines/engine.py.224: train: epoch 109 | batch 1450/2304 [I 2022-10-28 05:54:49] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:07:57 [I 2022-10-28 05:54:49] ...trap/engines/engine.py.229: process: 0.56408 | load: 0.00250 [I 2022-10-28 05:54:50] ...trap/engines/engine.py.230: loss: 0.12128 [I 2022-10-28 05:54:50] ...optimizers/trijoint.py.151: total_norm: 10.544537 [I 2022-10-28 05:54:55] ...trap/engines/engine.py.224: train: epoch 109 | batch 1460/2304 [I 2022-10-28 05:54:55] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:52 [I 2022-10-28 05:54:55] ...trap/engines/engine.py.229: process: 0.55721 | load: 0.00324 [I 2022-10-28 05:54:55] ...trap/engines/engine.py.230: loss: 0.12831 [I 2022-10-28 05:54:55] ...optimizers/trijoint.py.151: total_norm: 11.872161 [I 2022-10-28 05:55:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 1470/2304 [I 2022-10-28 05:55:01] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:47 [I 2022-10-28 05:55:01] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00252 [I 2022-10-28 05:55:01] ...trap/engines/engine.py.230: loss: 0.10067 [I 2022-10-28 05:55:01] ...optimizers/trijoint.py.151: total_norm: 15.880927 [I 2022-10-28 05:55:06] ...trap/engines/engine.py.224: train: epoch 109 | batch 1480/2304 [I 2022-10-28 05:55:06] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:39 [I 2022-10-28 05:55:06] ...trap/engines/engine.py.229: process: 0.55993 | load: 0.00242 [I 2022-10-28 05:55:06] ...trap/engines/engine.py.230: loss: 0.11537 [I 2022-10-28 05:55:06] ...optimizers/trijoint.py.151: total_norm: 15.795718 [I 2022-10-28 05:55:12] ...trap/engines/engine.py.224: train: epoch 109 | batch 1490/2304 [I 2022-10-28 05:55:12] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:41 [I 2022-10-28 05:55:12] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00256 [I 2022-10-28 05:55:12] ...trap/engines/engine.py.230: loss: 0.16981 [I 2022-10-28 05:55:12] ...optimizers/trijoint.py.151: total_norm: 14.053047 [I 2022-10-28 05:55:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 1500/2304 [I 2022-10-28 05:55:18] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:33 [I 2022-10-28 05:55:18] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00249 [I 2022-10-28 05:55:18] ...trap/engines/engine.py.230: loss: 0.11249 [I 2022-10-28 05:55:18] ...optimizers/trijoint.py.151: total_norm: 10.142869 [I 2022-10-28 05:55:23] ...trap/engines/engine.py.224: train: epoch 109 | batch 1510/2304 [I 2022-10-28 05:55:23] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:26 [I 2022-10-28 05:55:23] ...trap/engines/engine.py.229: process: 0.56267 | load: 0.00248 [I 2022-10-28 05:55:23] ...trap/engines/engine.py.230: loss: 0.08271 [I 2022-10-28 05:55:23] ...optimizers/trijoint.py.151: total_norm: 13.928496 [I 2022-10-28 05:55:29] ...trap/engines/engine.py.224: train: epoch 109 | batch 1520/2304 [I 2022-10-28 05:55:29] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:18 [I 2022-10-28 05:55:29] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00247 [I 2022-10-28 05:55:29] ...trap/engines/engine.py.230: loss: 0.11153 [I 2022-10-28 05:55:29] ...optimizers/trijoint.py.151: total_norm: 21.364948 [I 2022-10-28 05:55:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 1530/2304 [I 2022-10-28 05:55:35] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:14 [I 2022-10-28 05:55:35] ...trap/engines/engine.py.229: process: 0.56334 | load: 0.00247 [I 2022-10-28 05:55:35] ...trap/engines/engine.py.230: loss: 0.11827 [I 2022-10-28 05:55:35] ...optimizers/trijoint.py.151: total_norm: 11.486302 [I 2022-10-28 05:55:40] ...trap/engines/engine.py.224: train: epoch 109 | batch 1540/2304 [I 2022-10-28 05:55:40] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:04 [I 2022-10-28 05:55:40] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00244 [I 2022-10-28 05:55:40] ...trap/engines/engine.py.230: loss: 0.08372 [I 2022-10-28 05:55:40] ...optimizers/trijoint.py.151: total_norm: 12.105124 [I 2022-10-28 05:55:46] ...trap/engines/engine.py.224: train: epoch 109 | batch 1550/2304 [I 2022-10-28 05:55:46] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:03 [I 2022-10-28 05:55:46] ...trap/engines/engine.py.229: process: 0.57735 | load: 0.00261 [I 2022-10-28 05:55:46] ...trap/engines/engine.py.230: loss: 0.16720 [I 2022-10-28 05:55:46] ...optimizers/trijoint.py.151: total_norm: 24.235888 [I 2022-10-28 05:55:51] ...trap/engines/engine.py.224: train: epoch 109 | batch 1560/2304 [I 2022-10-28 05:55:51] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:57 [I 2022-10-28 05:55:51] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00251 [I 2022-10-28 05:55:51] ...trap/engines/engine.py.230: loss: 0.10589 [I 2022-10-28 05:55:51] ...optimizers/trijoint.py.151: total_norm: 10.185925 [I 2022-10-28 05:55:57] ...trap/engines/engine.py.224: train: epoch 109 | batch 1570/2304 [I 2022-10-28 05:55:57] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:50 [I 2022-10-28 05:55:57] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00247 [I 2022-10-28 05:55:57] ...trap/engines/engine.py.230: loss: 0.08172 [I 2022-10-28 05:55:57] ...optimizers/trijoint.py.151: total_norm: 13.630585 [I 2022-10-28 05:56:03] ...trap/engines/engine.py.224: train: epoch 109 | batch 1580/2304 [I 2022-10-28 05:56:03] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:45 [I 2022-10-28 05:56:03] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00243 [I 2022-10-28 05:56:03] ...trap/engines/engine.py.230: loss: 0.12490 [I 2022-10-28 05:56:03] ...optimizers/trijoint.py.151: total_norm: 20.392952 [I 2022-10-28 05:56:08] ...trap/engines/engine.py.224: train: epoch 109 | batch 1590/2304 [I 2022-10-28 05:56:08] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:39 [I 2022-10-28 05:56:08] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00247 [I 2022-10-28 05:56:08] ...trap/engines/engine.py.230: loss: 0.21325 [I 2022-10-28 05:56:08] ...optimizers/trijoint.py.151: total_norm: 15.615867 [I 2022-10-28 05:56:14] ...trap/engines/engine.py.224: train: epoch 109 | batch 1600/2304 [I 2022-10-28 05:56:14] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:31 [I 2022-10-28 05:56:14] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00246 [I 2022-10-28 05:56:14] ...trap/engines/engine.py.230: loss: 0.14925 [I 2022-10-28 05:56:14] ...optimizers/trijoint.py.151: total_norm: 20.808229 [I 2022-10-28 05:56:19] ...trap/engines/engine.py.224: train: epoch 109 | batch 1610/2304 [I 2022-10-28 05:56:19] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:29 [I 2022-10-28 05:56:19] ...trap/engines/engine.py.229: process: 0.56651 | load: 0.00249 [I 2022-10-28 05:56:19] ...trap/engines/engine.py.230: loss: 0.10919 [I 2022-10-28 05:56:19] ...optimizers/trijoint.py.151: total_norm: 18.737650 [I 2022-10-28 05:56:25] ...trap/engines/engine.py.224: train: epoch 109 | batch 1620/2304 [I 2022-10-28 05:56:25] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:23 [I 2022-10-28 05:56:25] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00244 [I 2022-10-28 05:56:25] ...trap/engines/engine.py.230: loss: 0.17957 [I 2022-10-28 05:56:25] ...optimizers/trijoint.py.151: total_norm: 20.312389 [I 2022-10-28 05:56:31] ...trap/engines/engine.py.224: train: epoch 109 | batch 1630/2304 [I 2022-10-28 05:56:31] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:16 [I 2022-10-28 05:56:31] ...trap/engines/engine.py.229: process: 0.55582 | load: 0.00244 [I 2022-10-28 05:56:31] ...trap/engines/engine.py.230: loss: 0.11405 [I 2022-10-28 05:56:31] ...optimizers/trijoint.py.151: total_norm: 12.203333 [I 2022-10-28 05:56:36] ...trap/engines/engine.py.224: train: epoch 109 | batch 1640/2304 [I 2022-10-28 05:56:36] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:07 [I 2022-10-28 05:56:36] ...trap/engines/engine.py.229: process: 0.54796 | load: 0.00253 [I 2022-10-28 05:56:36] ...trap/engines/engine.py.230: loss: 0.14156 [I 2022-10-28 05:56:36] ...optimizers/trijoint.py.151: total_norm: 15.003006 [I 2022-10-28 05:56:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 1650/2304 [I 2022-10-28 05:56:42] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:06 [I 2022-10-28 05:56:42] ...trap/engines/engine.py.229: process: 0.56948 | load: 0.00247 [I 2022-10-28 05:56:42] ...trap/engines/engine.py.230: loss: 0.14548 [I 2022-10-28 05:56:42] ...optimizers/trijoint.py.151: total_norm: 13.206808 [I 2022-10-28 05:56:48] ...trap/engines/engine.py.224: train: epoch 109 | batch 1660/2304 [I 2022-10-28 05:56:48] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:06:11 [I 2022-10-28 05:56:48] ...trap/engines/engine.py.229: process: 0.58494 | load: 0.00245 [I 2022-10-28 05:56:48] ...trap/engines/engine.py.230: loss: 0.08850 [I 2022-10-28 05:56:48] ...optimizers/trijoint.py.151: total_norm: 13.944391 [I 2022-10-28 05:56:53] ...trap/engines/engine.py.224: train: epoch 109 | batch 1670/2304 [I 2022-10-28 05:56:53] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:06:00 [I 2022-10-28 05:56:53] ...trap/engines/engine.py.229: process: 0.56115 | load: 0.00246 [I 2022-10-28 05:56:53] ...trap/engines/engine.py.230: loss: 0.10261 [I 2022-10-28 05:56:53] ...optimizers/trijoint.py.151: total_norm: 11.325768 [I 2022-10-28 05:56:59] ...trap/engines/engine.py.224: train: epoch 109 | batch 1680/2304 [I 2022-10-28 05:56:59] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:53 [I 2022-10-28 05:56:59] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00245 [I 2022-10-28 05:56:59] ...trap/engines/engine.py.230: loss: 0.15963 [I 2022-10-28 05:56:59] ...optimizers/trijoint.py.151: total_norm: 9.593560 [I 2022-10-28 05:57:05] ...trap/engines/engine.py.224: train: epoch 109 | batch 1690/2304 [I 2022-10-28 05:57:05] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:42 [I 2022-10-28 05:57:05] ...trap/engines/engine.py.229: process: 0.54780 | load: 0.00252 [I 2022-10-28 05:57:05] ...trap/engines/engine.py.230: loss: 0.10458 [I 2022-10-28 05:57:05] ...optimizers/trijoint.py.151: total_norm: 10.185487 [I 2022-10-28 05:57:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 1700/2304 [I 2022-10-28 05:57:10] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:42 [I 2022-10-28 05:57:10] ...trap/engines/engine.py.229: process: 0.56584 | load: 0.00251 [I 2022-10-28 05:57:10] ...trap/engines/engine.py.230: loss: 0.11086 [I 2022-10-28 05:57:10] ...optimizers/trijoint.py.151: total_norm: 9.633628 [I 2022-10-28 05:57:16] ...trap/engines/engine.py.224: train: epoch 109 | batch 1710/2304 [I 2022-10-28 05:57:16] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:32 [I 2022-10-28 05:57:16] ...trap/engines/engine.py.229: process: 0.55654 | load: 0.00246 [I 2022-10-28 05:57:16] ...trap/engines/engine.py.230: loss: 0.13238 [I 2022-10-28 05:57:16] ...optimizers/trijoint.py.151: total_norm: 13.048584 [I 2022-10-28 05:57:21] ...trap/engines/engine.py.224: train: epoch 109 | batch 1720/2304 [I 2022-10-28 05:57:21] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:28 [I 2022-10-28 05:57:21] ...trap/engines/engine.py.229: process: 0.56931 | load: 0.00244 [I 2022-10-28 05:57:21] ...trap/engines/engine.py.230: loss: 0.14805 [I 2022-10-28 05:57:21] ...optimizers/trijoint.py.151: total_norm: 17.538271 [I 2022-10-28 05:57:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 1730/2304 [I 2022-10-28 05:57:27] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:20 [I 2022-10-28 05:57:27] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00247 [I 2022-10-28 05:57:27] ...trap/engines/engine.py.230: loss: 0.13437 [I 2022-10-28 05:57:27] ...optimizers/trijoint.py.151: total_norm: 18.014933 [I 2022-10-28 05:57:33] ...trap/engines/engine.py.224: train: epoch 109 | batch 1740/2304 [I 2022-10-28 05:57:33] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:24 [I 2022-10-28 05:57:33] ...trap/engines/engine.py.229: process: 0.56188 | load: 0.00260 [I 2022-10-28 05:57:33] ...trap/engines/engine.py.230: loss: 0.11389 [I 2022-10-28 05:57:33] ...optimizers/trijoint.py.151: total_norm: 15.603465 [I 2022-10-28 05:57:38] ...trap/engines/engine.py.224: train: epoch 109 | batch 1750/2304 [I 2022-10-28 05:57:38] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:13 [I 2022-10-28 05:57:38] ...trap/engines/engine.py.229: process: 0.55850 | load: 0.00243 [I 2022-10-28 05:57:38] ...trap/engines/engine.py.230: loss: 0.13667 [I 2022-10-28 05:57:38] ...optimizers/trijoint.py.151: total_norm: 22.644720 [I 2022-10-28 05:57:44] ...trap/engines/engine.py.224: train: epoch 109 | batch 1760/2304 [I 2022-10-28 05:57:44] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:06 [I 2022-10-28 05:57:44] ...trap/engines/engine.py.229: process: 0.56050 | load: 0.00247 [I 2022-10-28 05:57:44] ...trap/engines/engine.py.230: loss: 0.10477 [I 2022-10-28 05:57:44] ...optimizers/trijoint.py.151: total_norm: 14.884132 [I 2022-10-28 05:57:50] ...trap/engines/engine.py.224: train: epoch 109 | batch 1770/2304 [I 2022-10-28 05:57:50] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:05:02 [I 2022-10-28 05:57:50] ...trap/engines/engine.py.229: process: 0.57102 | load: 0.00287 [I 2022-10-28 05:57:50] ...trap/engines/engine.py.230: loss: 0.09251 [I 2022-10-28 05:57:50] ...optimizers/trijoint.py.151: total_norm: 13.615436 [I 2022-10-28 05:57:55] ...trap/engines/engine.py.224: train: epoch 109 | batch 1780/2304 [I 2022-10-28 05:57:55] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:04:53 [I 2022-10-28 05:57:55] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00249 [I 2022-10-28 05:57:55] ...trap/engines/engine.py.230: loss: 0.17949 [I 2022-10-28 05:57:55] ...optimizers/trijoint.py.151: total_norm: 18.405825 [I 2022-10-28 05:58:01] ...trap/engines/engine.py.224: train: epoch 109 | batch 1790/2304 [I 2022-10-28 05:58:01] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:48 [I 2022-10-28 05:58:01] ...trap/engines/engine.py.229: process: 0.54722 | load: 0.00250 [I 2022-10-28 05:58:01] ...trap/engines/engine.py.230: loss: 0.10926 [I 2022-10-28 05:58:01] ...optimizers/trijoint.py.151: total_norm: 68.271843 [I 2022-10-28 05:58:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 1800/2304 [I 2022-10-28 05:58:07] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:43 [I 2022-10-28 05:58:07] ...trap/engines/engine.py.229: process: 0.56042 | load: 0.00244 [I 2022-10-28 05:58:07] ...trap/engines/engine.py.230: loss: 0.11611 [I 2022-10-28 05:58:07] ...optimizers/trijoint.py.151: total_norm: 9.056293 [I 2022-10-28 05:58:12] ...trap/engines/engine.py.224: train: epoch 109 | batch 1810/2304 [I 2022-10-28 05:58:12] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:38 [I 2022-10-28 05:58:12] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00248 [I 2022-10-28 05:58:12] ...trap/engines/engine.py.230: loss: 0.14746 [I 2022-10-28 05:58:12] ...optimizers/trijoint.py.151: total_norm: 16.436409 [I 2022-10-28 05:58:18] ...trap/engines/engine.py.224: train: epoch 109 | batch 1820/2304 [I 2022-10-28 05:58:18] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:32 [I 2022-10-28 05:58:18] ...trap/engines/engine.py.229: process: 0.55092 | load: 0.00244 [I 2022-10-28 05:58:18] ...trap/engines/engine.py.230: loss: 0.11065 [I 2022-10-28 05:58:18] ...optimizers/trijoint.py.151: total_norm: 21.179548 [I 2022-10-28 05:58:24] ...trap/engines/engine.py.224: train: epoch 109 | batch 1830/2304 [I 2022-10-28 05:58:24] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:28 [I 2022-10-28 05:58:24] ...trap/engines/engine.py.229: process: 0.56711 | load: 0.00247 [I 2022-10-28 05:58:24] ...trap/engines/engine.py.230: loss: 0.10779 [I 2022-10-28 05:58:24] ...optimizers/trijoint.py.151: total_norm: 10.352380 [I 2022-10-28 05:58:29] ...trap/engines/engine.py.224: train: epoch 109 | batch 1840/2304 [I 2022-10-28 05:58:29] ...trap/engines/engine.py.228: elapsed: 0:17:14 | left: 0:04:21 [I 2022-10-28 05:58:29] ...trap/engines/engine.py.229: process: 0.56375 | load: 0.00254 [I 2022-10-28 05:58:29] ...trap/engines/engine.py.230: loss: 0.09168 [I 2022-10-28 05:58:29] ...optimizers/trijoint.py.151: total_norm: 23.050953 [I 2022-10-28 05:58:35] ...trap/engines/engine.py.224: train: epoch 109 | batch 1850/2304 [I 2022-10-28 05:58:35] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:15 [I 2022-10-28 05:58:35] ...trap/engines/engine.py.229: process: 0.56186 | load: 0.00248 [I 2022-10-28 05:58:35] ...trap/engines/engine.py.230: loss: 0.15247 [I 2022-10-28 05:58:35] ...optimizers/trijoint.py.151: total_norm: 14.740297 [I 2022-10-28 05:58:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 1860/2304 [I 2022-10-28 05:58:41] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:08 [I 2022-10-28 05:58:41] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00246 [I 2022-10-28 05:58:41] ...trap/engines/engine.py.230: loss: 0.10591 [I 2022-10-28 05:58:41] ...optimizers/trijoint.py.151: total_norm: 25.931372 [I 2022-10-28 05:58:46] ...trap/engines/engine.py.224: train: epoch 109 | batch 1870/2304 [I 2022-10-28 05:58:46] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:00 [I 2022-10-28 05:58:46] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00246 [I 2022-10-28 05:58:46] ...trap/engines/engine.py.230: loss: 0.17324 [I 2022-10-28 05:58:46] ...optimizers/trijoint.py.151: total_norm: 15.536053 [I 2022-10-28 05:58:52] ...trap/engines/engine.py.224: train: epoch 109 | batch 1880/2304 [I 2022-10-28 05:58:52] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:03:56 [I 2022-10-28 05:58:52] ...trap/engines/engine.py.229: process: 0.53965 | load: 0.00250 [I 2022-10-28 05:58:52] ...trap/engines/engine.py.230: loss: 0.09286 [I 2022-10-28 05:58:52] ...optimizers/trijoint.py.151: total_norm: 7.796139 [I 2022-10-28 05:58:57] ...trap/engines/engine.py.224: train: epoch 109 | batch 1890/2304 [I 2022-10-28 05:58:57] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:03:52 [I 2022-10-28 05:58:57] ...trap/engines/engine.py.229: process: 0.57084 | load: 0.00308 [I 2022-10-28 05:58:57] ...trap/engines/engine.py.230: loss: 0.14174 [I 2022-10-28 05:58:57] ...optimizers/trijoint.py.151: total_norm: 19.528782 [I 2022-10-28 05:59:03] ...trap/engines/engine.py.224: train: epoch 109 | batch 1900/2304 [I 2022-10-28 05:59:03] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:03:45 [I 2022-10-28 05:59:03] ...trap/engines/engine.py.229: process: 0.54961 | load: 0.00245 [I 2022-10-28 05:59:03] ...trap/engines/engine.py.230: loss: 0.10208 [I 2022-10-28 05:59:03] ...optimizers/trijoint.py.151: total_norm: 8.485935 [I 2022-10-28 05:59:09] ...trap/engines/engine.py.224: train: epoch 109 | batch 1910/2304 [I 2022-10-28 05:59:09] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:38 [I 2022-10-28 05:59:09] ...trap/engines/engine.py.229: process: 0.55279 | load: 0.00245 [I 2022-10-28 05:59:09] ...trap/engines/engine.py.230: loss: 0.11317 [I 2022-10-28 05:59:09] ...optimizers/trijoint.py.151: total_norm: 32.812454 [I 2022-10-28 05:59:14] ...trap/engines/engine.py.224: train: epoch 109 | batch 1920/2304 [I 2022-10-28 05:59:14] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:03:38 [I 2022-10-28 05:59:14] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00245 [I 2022-10-28 05:59:14] ...trap/engines/engine.py.230: loss: 0.14378 [I 2022-10-28 05:59:14] ...optimizers/trijoint.py.151: total_norm: 13.897424 [I 2022-10-28 05:59:20] ...trap/engines/engine.py.224: train: epoch 109 | batch 1930/2304 [I 2022-10-28 05:59:20] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:28 [I 2022-10-28 05:59:20] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00255 [I 2022-10-28 05:59:20] ...trap/engines/engine.py.230: loss: 0.12679 [I 2022-10-28 05:59:20] ...optimizers/trijoint.py.151: total_norm: 17.745605 [I 2022-10-28 05:59:25] ...trap/engines/engine.py.224: train: epoch 109 | batch 1940/2304 [I 2022-10-28 05:59:25] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:03:23 [I 2022-10-28 05:59:25] ...trap/engines/engine.py.229: process: 0.56600 | load: 0.00248 [I 2022-10-28 05:59:25] ...trap/engines/engine.py.230: loss: 0.11144 [I 2022-10-28 05:59:25] ...optimizers/trijoint.py.151: total_norm: 32.630413 [I 2022-10-28 05:59:31] ...trap/engines/engine.py.224: train: epoch 109 | batch 1950/2304 [I 2022-10-28 05:59:31] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:16 [I 2022-10-28 05:59:31] ...trap/engines/engine.py.229: process: 0.55637 | load: 0.00249 [I 2022-10-28 05:59:31] ...trap/engines/engine.py.230: loss: 0.12263 [I 2022-10-28 05:59:31] ...optimizers/trijoint.py.151: total_norm: 16.186207 [I 2022-10-28 05:59:37] ...trap/engines/engine.py.224: train: epoch 109 | batch 1960/2304 [I 2022-10-28 05:59:37] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:12 [I 2022-10-28 05:59:37] ...trap/engines/engine.py.229: process: 0.56635 | load: 0.00242 [I 2022-10-28 05:59:37] ...trap/engines/engine.py.230: loss: 0.14137 [I 2022-10-28 05:59:37] ...optimizers/trijoint.py.151: total_norm: 17.705339 [I 2022-10-28 05:59:42] ...trap/engines/engine.py.224: train: epoch 109 | batch 1970/2304 [I 2022-10-28 05:59:42] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:06 [I 2022-10-28 05:59:42] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00249 [I 2022-10-28 05:59:42] ...trap/engines/engine.py.230: loss: 0.10009 [I 2022-10-28 05:59:42] ...optimizers/trijoint.py.151: total_norm: 11.379496 [I 2022-10-28 05:59:48] ...trap/engines/engine.py.224: train: epoch 109 | batch 1980/2304 [I 2022-10-28 05:59:48] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:01 [I 2022-10-28 05:59:48] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00245 [I 2022-10-28 05:59:48] ...trap/engines/engine.py.230: loss: 0.12475 [I 2022-10-28 05:59:48] ...optimizers/trijoint.py.151: total_norm: 11.079420 [I 2022-10-28 06:00:13] ...trap/engines/engine.py.224: train: epoch 109 | batch 1990/2304 [I 2022-10-28 06:00:13] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:06:25 [I 2022-10-28 06:00:13] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00240 [I 2022-10-28 06:00:13] ...trap/engines/engine.py.230: loss: 0.09821 [I 2022-10-28 06:00:13] ...optimizers/trijoint.py.151: total_norm: 10.431971 [I 2022-10-28 06:00:19] ...trap/engines/engine.py.224: train: epoch 109 | batch 2000/2304 [I 2022-10-28 06:00:19] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:03:14 [I 2022-10-28 06:00:19] ...trap/engines/engine.py.229: process: 0.55737 | load: 0.00239 [I 2022-10-28 06:00:19] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-28 06:00:19] ...optimizers/trijoint.py.151: total_norm: 30.753996 [I 2022-10-28 06:00:25] ...trap/engines/engine.py.224: train: epoch 109 | batch 2010/2304 [I 2022-10-28 06:00:25] ...trap/engines/engine.py.228: elapsed: 0:19:09 | left: 0:02:46 [I 2022-10-28 06:00:25] ...trap/engines/engine.py.229: process: 0.55186 | load: 0.00240 [I 2022-10-28 06:00:25] ...trap/engines/engine.py.230: loss: 0.09146 [I 2022-10-28 06:00:25] ...optimizers/trijoint.py.151: total_norm: 10.063263 [I 2022-10-28 06:00:30] ...trap/engines/engine.py.224: train: epoch 109 | batch 2020/2304 [I 2022-10-28 06:00:30] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:39 [I 2022-10-28 06:00:30] ...trap/engines/engine.py.229: process: 0.56480 | load: 0.00240 [I 2022-10-28 06:00:30] ...trap/engines/engine.py.230: loss: 0.11122 [I 2022-10-28 06:00:30] ...optimizers/trijoint.py.151: total_norm: 17.496016 [I 2022-10-28 06:00:36] ...trap/engines/engine.py.224: train: epoch 109 | batch 2030/2304 [I 2022-10-28 06:00:36] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:34 [I 2022-10-28 06:00:36] ...trap/engines/engine.py.229: process: 0.56310 | load: 0.00242 [I 2022-10-28 06:00:36] ...trap/engines/engine.py.230: loss: 0.12720 [I 2022-10-28 06:00:36] ...optimizers/trijoint.py.151: total_norm: 11.709944 [I 2022-10-28 06:00:41] ...trap/engines/engine.py.224: train: epoch 109 | batch 2040/2304 [I 2022-10-28 06:00:41] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:27 [I 2022-10-28 06:00:41] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00241 [I 2022-10-28 06:00:41] ...trap/engines/engine.py.230: loss: 0.10425 [I 2022-10-28 06:00:41] ...optimizers/trijoint.py.151: total_norm: 16.343838 [I 2022-10-28 06:00:47] ...trap/engines/engine.py.224: train: epoch 109 | batch 2050/2304 [I 2022-10-28 06:00:47] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:22 [I 2022-10-28 06:00:47] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00245 [I 2022-10-28 06:00:47] ...trap/engines/engine.py.230: loss: 0.15164 [I 2022-10-28 06:00:47] ...optimizers/trijoint.py.151: total_norm: 19.370522 [I 2022-10-28 06:00:53] ...trap/engines/engine.py.224: train: epoch 109 | batch 2060/2304 [I 2022-10-28 06:00:53] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:18 [I 2022-10-28 06:00:53] ...trap/engines/engine.py.229: process: 0.56367 | load: 0.00248 [I 2022-10-28 06:00:53] ...trap/engines/engine.py.230: loss: 0.10853 [I 2022-10-28 06:00:53] ...optimizers/trijoint.py.151: total_norm: 9.639448 [I 2022-10-28 06:00:58] ...trap/engines/engine.py.224: train: epoch 109 | batch 2070/2304 [I 2022-10-28 06:00:58] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:02:11 [I 2022-10-28 06:00:58] ...trap/engines/engine.py.229: process: 0.54647 | load: 0.00239 [I 2022-10-28 06:00:58] ...trap/engines/engine.py.230: loss: 0.09618 [I 2022-10-28 06:00:58] ...optimizers/trijoint.py.151: total_norm: 9.674354 [I 2022-10-28 06:01:04] ...trap/engines/engine.py.224: train: epoch 109 | batch 2080/2304 [I 2022-10-28 06:01:04] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:02:05 [I 2022-10-28 06:01:04] ...trap/engines/engine.py.229: process: 0.56243 | load: 0.00237 [I 2022-10-28 06:01:04] ...trap/engines/engine.py.230: loss: 0.12262 [I 2022-10-28 06:01:04] ...optimizers/trijoint.py.151: total_norm: 13.432231 [I 2022-10-28 06:01:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 2090/2304 [I 2022-10-28 06:01:10] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:02:00 [I 2022-10-28 06:01:10] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00240 [I 2022-10-28 06:01:10] ...trap/engines/engine.py.230: loss: 0.13638 [I 2022-10-28 06:01:10] ...optimizers/trijoint.py.151: total_norm: 32.067848 [I 2022-10-28 06:01:15] ...trap/engines/engine.py.224: train: epoch 109 | batch 2100/2304 [I 2022-10-28 06:01:15] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:54 [I 2022-10-28 06:01:15] ...trap/engines/engine.py.229: process: 0.56246 | load: 0.00242 [I 2022-10-28 06:01:15] ...trap/engines/engine.py.230: loss: 0.12125 [I 2022-10-28 06:01:15] ...optimizers/trijoint.py.151: total_norm: 19.578983 [I 2022-10-28 06:01:21] ...trap/engines/engine.py.224: train: epoch 109 | batch 2110/2304 [I 2022-10-28 06:01:21] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:49 [I 2022-10-28 06:01:21] ...trap/engines/engine.py.229: process: 0.55987 | load: 0.00240 [I 2022-10-28 06:01:21] ...trap/engines/engine.py.230: loss: 0.12482 [I 2022-10-28 06:01:21] ...optimizers/trijoint.py.151: total_norm: 13.820329 [I 2022-10-28 06:01:27] ...trap/engines/engine.py.224: train: epoch 109 | batch 2120/2304 [I 2022-10-28 06:01:27] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:42 [I 2022-10-28 06:01:27] ...trap/engines/engine.py.229: process: 0.53628 | load: 0.00240 [I 2022-10-28 06:01:27] ...trap/engines/engine.py.230: loss: 0.09944 [I 2022-10-28 06:01:27] ...optimizers/trijoint.py.151: total_norm: 12.404432 [I 2022-10-28 06:01:32] ...trap/engines/engine.py.224: train: epoch 109 | batch 2130/2304 [I 2022-10-28 06:01:32] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:37 [I 2022-10-28 06:01:32] ...trap/engines/engine.py.229: process: 0.55887 | load: 0.00241 [I 2022-10-28 06:01:32] ...trap/engines/engine.py.230: loss: 0.21246 [I 2022-10-28 06:01:32] ...optimizers/trijoint.py.151: total_norm: 24.616932 [I 2022-10-28 06:01:38] ...trap/engines/engine.py.224: train: epoch 109 | batch 2140/2304 [I 2022-10-28 06:01:38] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:31 [I 2022-10-28 06:01:38] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00242 [I 2022-10-28 06:01:38] ...trap/engines/engine.py.230: loss: 0.08674 [I 2022-10-28 06:01:38] ...optimizers/trijoint.py.151: total_norm: 8.728102 [I 2022-10-28 06:01:43] ...trap/engines/engine.py.224: train: epoch 109 | batch 2150/2304 [I 2022-10-28 06:01:43] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:26 [I 2022-10-28 06:01:43] ...trap/engines/engine.py.229: process: 0.56900 | load: 0.00245 [I 2022-10-28 06:01:43] ...trap/engines/engine.py.230: loss: 0.11415 [I 2022-10-28 06:01:43] ...optimizers/trijoint.py.151: total_norm: 14.128920 [I 2022-10-28 06:01:49] ...trap/engines/engine.py.224: train: epoch 109 | batch 2160/2304 [I 2022-10-28 06:01:49] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:20 [I 2022-10-28 06:01:49] ...trap/engines/engine.py.229: process: 0.56319 | load: 0.00243 [I 2022-10-28 06:01:49] ...trap/engines/engine.py.230: loss: 0.16746 [I 2022-10-28 06:01:49] ...optimizers/trijoint.py.151: total_norm: 14.322762 [I 2022-10-28 06:01:55] ...trap/engines/engine.py.224: train: epoch 109 | batch 2170/2304 [I 2022-10-28 06:01:55] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:01:17 [I 2022-10-28 06:01:55] ...trap/engines/engine.py.229: process: 0.56050 | load: 0.00245 [I 2022-10-28 06:01:55] ...trap/engines/engine.py.230: loss: 0.18504 [I 2022-10-28 06:01:55] ...optimizers/trijoint.py.151: total_norm: 20.307341 [I 2022-10-28 06:02:00] ...trap/engines/engine.py.224: train: epoch 109 | batch 2180/2304 [I 2022-10-28 06:02:00] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:01:09 [I 2022-10-28 06:02:00] ...trap/engines/engine.py.229: process: 0.55691 | load: 0.00237 [I 2022-10-28 06:02:00] ...trap/engines/engine.py.230: loss: 0.12818 [I 2022-10-28 06:02:00] ...optimizers/trijoint.py.151: total_norm: 25.278482 [I 2022-10-28 06:02:06] ...trap/engines/engine.py.224: train: epoch 109 | batch 2190/2304 [I 2022-10-28 06:02:06] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:01:03 [I 2022-10-28 06:02:06] ...trap/engines/engine.py.229: process: 0.56157 | load: 0.00243 [I 2022-10-28 06:02:06] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-28 06:02:06] ...optimizers/trijoint.py.151: total_norm: 9.022800 [I 2022-10-28 06:02:12] ...trap/engines/engine.py.224: train: epoch 109 | batch 2200/2304 [I 2022-10-28 06:02:12] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:58 [I 2022-10-28 06:02:12] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00241 [I 2022-10-28 06:02:12] ...trap/engines/engine.py.230: loss: 0.12521 [I 2022-10-28 06:02:12] ...optimizers/trijoint.py.151: total_norm: 16.018362 [I 2022-10-28 06:02:17] ...trap/engines/engine.py.224: train: epoch 109 | batch 2210/2304 [I 2022-10-28 06:02:17] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:52 [I 2022-10-28 06:02:17] ...trap/engines/engine.py.229: process: 0.56819 | load: 0.00247 [I 2022-10-28 06:02:17] ...trap/engines/engine.py.230: loss: 0.09816 [I 2022-10-28 06:02:17] ...optimizers/trijoint.py.151: total_norm: 11.768506 [I 2022-10-28 06:02:23] ...trap/engines/engine.py.224: train: epoch 109 | batch 2220/2304 [I 2022-10-28 06:02:23] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:46 [I 2022-10-28 06:02:23] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00254 [I 2022-10-28 06:02:23] ...trap/engines/engine.py.230: loss: 0.10717 [I 2022-10-28 06:02:23] ...optimizers/trijoint.py.151: total_norm: 19.844849 [I 2022-10-28 06:02:28] ...trap/engines/engine.py.224: train: epoch 109 | batch 2230/2304 [I 2022-10-28 06:02:28] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:41 [I 2022-10-28 06:02:28] ...trap/engines/engine.py.229: process: 0.55847 | load: 0.00247 [I 2022-10-28 06:02:28] ...trap/engines/engine.py.230: loss: 0.10947 [I 2022-10-28 06:02:28] ...optimizers/trijoint.py.151: total_norm: 24.780893 [I 2022-10-28 06:02:34] ...trap/engines/engine.py.224: train: epoch 109 | batch 2240/2304 [I 2022-10-28 06:02:34] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:35 [I 2022-10-28 06:02:34] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00250 [I 2022-10-28 06:02:34] ...trap/engines/engine.py.230: loss: 0.17510 [I 2022-10-28 06:02:34] ...optimizers/trijoint.py.151: total_norm: 23.233908 [I 2022-10-28 06:02:39] ...trap/engines/engine.py.224: train: epoch 109 | batch 2250/2304 [I 2022-10-28 06:02:39] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:29 [I 2022-10-28 06:02:39] ...trap/engines/engine.py.229: process: 0.55656 | load: 0.00246 [I 2022-10-28 06:02:39] ...trap/engines/engine.py.230: loss: 0.10569 [I 2022-10-28 06:02:39] ...optimizers/trijoint.py.151: total_norm: 20.992292 [I 2022-10-28 06:02:45] ...trap/engines/engine.py.224: train: epoch 109 | batch 2260/2304 [I 2022-10-28 06:02:45] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:24 [I 2022-10-28 06:02:45] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00243 [I 2022-10-28 06:02:45] ...trap/engines/engine.py.230: loss: 0.19844 [I 2022-10-28 06:02:45] ...optimizers/trijoint.py.151: total_norm: 39.073978 [I 2022-10-28 06:02:51] ...trap/engines/engine.py.224: train: epoch 109 | batch 2270/2304 [I 2022-10-28 06:02:51] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:19 [I 2022-10-28 06:02:51] ...trap/engines/engine.py.229: process: 0.56992 | load: 0.00243 [I 2022-10-28 06:02:51] ...trap/engines/engine.py.230: loss: 0.13886 [I 2022-10-28 06:02:51] ...optimizers/trijoint.py.151: total_norm: 29.449339 [I 2022-10-28 06:02:56] ...trap/engines/engine.py.224: train: epoch 109 | batch 2280/2304 [I 2022-10-28 06:02:56] ...trap/engines/engine.py.228: elapsed: 0:21:41 | left: 0:00:13 [I 2022-10-28 06:02:56] ...trap/engines/engine.py.229: process: 0.56814 | load: 0.00246 [I 2022-10-28 06:02:56] ...trap/engines/engine.py.230: loss: 0.13715 [I 2022-10-28 06:02:56] ...optimizers/trijoint.py.151: total_norm: 14.431659 [I 2022-10-28 06:03:02] ...trap/engines/engine.py.224: train: epoch 109 | batch 2290/2304 [I 2022-10-28 06:03:02] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:07 [I 2022-10-28 06:03:02] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00243 [I 2022-10-28 06:03:02] ...trap/engines/engine.py.230: loss: 0.13001 [I 2022-10-28 06:03:02] ...optimizers/trijoint.py.151: total_norm: 12.949030 [I 2022-10-28 06:03:07] ...trap/engines/engine.py.224: train: epoch 109 | batch 2300/2304 [I 2022-10-28 06:03:07] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:00:02 [I 2022-10-28 06:03:07] ...trap/engines/engine.py.229: process: 0.55041 | load: 0.00014 [I 2022-10-28 06:03:07] ...trap/engines/engine.py.230: loss: 0.16515 [I 2022-10-28 06:03:07] ...optimizers/trijoint.py.151: total_norm: 9.990782 [I 2022-10-28 06:03:10] ...trap/engines/engine.py.224: train: epoch 109 | batch 2304/2304 [I 2022-10-28 06:03:10] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:00:00 [I 2022-10-28 06:03:10] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00014 [I 2022-10-28 06:03:10] ...trap/engines/engine.py.230: loss: 0.12796 [I 2022-10-28 06:03:10] ...optimizers/trijoint.py.151: total_norm: 20.566055 [S 2022-10-28 06:03:11] ...trap/engines/engine.py.129: train_epoch.epoch: 109 [S 2022-10-28 06:03:11] ...trap/engines/engine.py.129: train_epoch.loss: 0.12203558307795617 [I 2022-10-28 06:03:11] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 06:03:12] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 109 [I 2022-10-28 06:03:12] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 06:03:14] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 06:03:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 06:03:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 06:03:18] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 06:03:18] ...trap/engines/engine.py.321: eval: epoch 109 | batch 0/510 [I 2022-10-28 06:03:18] ...trap/engines/engine.py.325: elapsed: 0:00:05 | left: 0:50:14 [I 2022-10-28 06:03:18] ...trap/engines/engine.py.326: process: 5.91150 | load: 5.01114 [I 2022-10-28 06:03:22] ...trap/engines/engine.py.321: eval: epoch 109 | batch 10/510 [I 2022-10-28 06:03:22] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:49 [I 2022-10-28 06:03:22] ...trap/engines/engine.py.326: process: 0.35421 | load: 0.00017 [I 2022-10-28 06:03:25] ...trap/engines/engine.py.321: eval: epoch 109 | batch 20/510 [I 2022-10-28 06:03:25] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:18 [I 2022-10-28 06:03:25] ...trap/engines/engine.py.326: process: 0.37542 | load: 0.00016 [I 2022-10-28 06:03:28] ...trap/engines/engine.py.321: eval: epoch 109 | batch 30/510 [I 2022-10-28 06:03:28] ...trap/engines/engine.py.325: elapsed: 0:00:15 | left: 0:02:42 [I 2022-10-28 06:03:28] ...trap/engines/engine.py.326: process: 0.33934 | load: 0.00016 [I 2022-10-28 06:03:32] ...trap/engines/engine.py.321: eval: epoch 109 | batch 40/510 [I 2022-10-28 06:03:32] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-28 06:03:32] ...trap/engines/engine.py.326: process: 0.32665 | load: 0.00017 [I 2022-10-28 06:03:35] ...trap/engines/engine.py.321: eval: epoch 109 | batch 50/510 [I 2022-10-28 06:03:35] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:31 [I 2022-10-28 06:03:35] ...trap/engines/engine.py.326: process: 0.33101 | load: 0.00016 [I 2022-10-28 06:03:38] ...trap/engines/engine.py.321: eval: epoch 109 | batch 60/510 [I 2022-10-28 06:03:38] ...trap/engines/engine.py.325: elapsed: 0:00:25 | left: 0:02:28 [I 2022-10-28 06:03:38] ...trap/engines/engine.py.326: process: 0.32931 | load: 0.00017 [I 2022-10-28 06:03:42] ...trap/engines/engine.py.321: eval: epoch 109 | batch 70/510 [I 2022-10-28 06:03:42] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:27 [I 2022-10-28 06:03:42] ...trap/engines/engine.py.326: process: 0.33353 | load: 0.00017 [I 2022-10-28 06:03:45] ...trap/engines/engine.py.321: eval: epoch 109 | batch 80/510 [I 2022-10-28 06:03:45] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:21 [I 2022-10-28 06:03:45] ...trap/engines/engine.py.326: process: 0.32354 | load: 0.00016 [I 2022-10-28 06:03:48] ...trap/engines/engine.py.321: eval: epoch 109 | batch 90/510 [I 2022-10-28 06:03:48] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:18 [I 2022-10-28 06:03:48] ...trap/engines/engine.py.326: process: 0.33063 | load: 0.00019 [I 2022-10-28 06:03:52] ...trap/engines/engine.py.321: eval: epoch 109 | batch 100/510 [I 2022-10-28 06:03:52] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:14 [I 2022-10-28 06:03:52] ...trap/engines/engine.py.326: process: 0.29243 | load: 0.00020 [I 2022-10-28 06:03:55] ...trap/engines/engine.py.321: eval: epoch 109 | batch 110/510 [I 2022-10-28 06:03:55] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:57 [I 2022-10-28 06:03:55] ...trap/engines/engine.py.326: process: 0.29075 | load: 0.00014 [I 2022-10-28 06:03:57] ...trap/engines/engine.py.321: eval: epoch 109 | batch 120/510 [I 2022-10-28 06:03:57] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:53 [I 2022-10-28 06:03:57] ...trap/engines/engine.py.326: process: 0.29375 | load: 0.00014 [I 2022-10-28 06:04:00] ...trap/engines/engine.py.321: eval: epoch 109 | batch 130/510 [I 2022-10-28 06:04:00] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:50 [I 2022-10-28 06:04:00] ...trap/engines/engine.py.326: process: 0.28579 | load: 0.00020 [I 2022-10-28 06:04:03] ...trap/engines/engine.py.321: eval: epoch 109 | batch 140/510 [I 2022-10-28 06:04:03] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:48 [I 2022-10-28 06:04:03] ...trap/engines/engine.py.326: process: 0.30331 | load: 0.00018 [I 2022-10-28 06:04:06] ...trap/engines/engine.py.321: eval: epoch 109 | batch 150/510 [I 2022-10-28 06:04:06] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:44 [I 2022-10-28 06:04:06] ...trap/engines/engine.py.326: process: 0.28920 | load: 0.00013 [I 2022-10-28 06:04:09] ...trap/engines/engine.py.321: eval: epoch 109 | batch 160/510 [I 2022-10-28 06:04:09] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:41 [I 2022-10-28 06:04:09] ...trap/engines/engine.py.326: process: 0.28535 | load: 0.00017 [I 2022-10-28 06:04:12] ...trap/engines/engine.py.321: eval: epoch 109 | batch 170/510 [I 2022-10-28 06:04:12] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 06:04:12] ...trap/engines/engine.py.326: process: 0.29214 | load: 0.00015 [I 2022-10-28 06:04:15] ...trap/engines/engine.py.321: eval: epoch 109 | batch 180/510 [I 2022-10-28 06:04:15] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-28 06:04:15] ...trap/engines/engine.py.326: process: 0.28683 | load: 0.00016 [I 2022-10-28 06:04:18] ...trap/engines/engine.py.321: eval: epoch 109 | batch 190/510 [I 2022-10-28 06:04:18] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-28 06:04:18] ...trap/engines/engine.py.326: process: 0.28970 | load: 0.00015 [I 2022-10-28 06:04:21] ...trap/engines/engine.py.321: eval: epoch 109 | batch 200/510 [I 2022-10-28 06:04:21] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:29 [I 2022-10-28 06:04:21] ...trap/engines/engine.py.326: process: 0.28918 | load: 0.00016 [I 2022-10-28 06:04:23] ...trap/engines/engine.py.321: eval: epoch 109 | batch 210/510 [I 2022-10-28 06:04:23] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 06:04:23] ...trap/engines/engine.py.326: process: 0.28702 | load: 0.00014 [I 2022-10-28 06:04:26] ...trap/engines/engine.py.321: eval: epoch 109 | batch 220/510 [I 2022-10-28 06:04:26] ...trap/engines/engine.py.325: elapsed: 0:01:13 | left: 0:01:23 [I 2022-10-28 06:04:26] ...trap/engines/engine.py.326: process: 0.28648 | load: 0.00017 [I 2022-10-28 06:04:29] ...trap/engines/engine.py.321: eval: epoch 109 | batch 230/510 [I 2022-10-28 06:04:29] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:20 [I 2022-10-28 06:04:29] ...trap/engines/engine.py.326: process: 0.29090 | load: 0.00015 [I 2022-10-28 06:04:32] ...trap/engines/engine.py.321: eval: epoch 109 | batch 240/510 [I 2022-10-28 06:04:32] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 06:04:32] ...trap/engines/engine.py.326: process: 0.28882 | load: 0.00018 [I 2022-10-28 06:04:35] ...trap/engines/engine.py.321: eval: epoch 109 | batch 250/510 [I 2022-10-28 06:04:35] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-28 06:04:35] ...trap/engines/engine.py.326: process: 0.29245 | load: 0.00016 [I 2022-10-28 06:04:38] ...trap/engines/engine.py.321: eval: epoch 109 | batch 260/510 [I 2022-10-28 06:04:38] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:11 [I 2022-10-28 06:04:38] ...trap/engines/engine.py.326: process: 0.27793 | load: 0.00018 [I 2022-10-28 06:04:41] ...trap/engines/engine.py.321: eval: epoch 109 | batch 270/510 [I 2022-10-28 06:04:41] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-28 06:04:41] ...trap/engines/engine.py.326: process: 0.29246 | load: 0.00016 [I 2022-10-28 06:04:44] ...trap/engines/engine.py.321: eval: epoch 109 | batch 280/510 [I 2022-10-28 06:04:44] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 06:04:44] ...trap/engines/engine.py.326: process: 0.29107 | load: 0.00017 [I 2022-10-28 06:04:46] ...trap/engines/engine.py.321: eval: epoch 109 | batch 290/510 [I 2022-10-28 06:04:46] ...trap/engines/engine.py.325: elapsed: 0:01:33 | left: 0:01:03 [I 2022-10-28 06:04:46] ...trap/engines/engine.py.326: process: 0.28662 | load: 0.00019 [I 2022-10-28 06:04:49] ...trap/engines/engine.py.321: eval: epoch 109 | batch 300/510 [I 2022-10-28 06:04:49] ...trap/engines/engine.py.325: elapsed: 0:01:36 | left: 0:01:00 [I 2022-10-28 06:04:49] ...trap/engines/engine.py.326: process: 0.28622 | load: 0.00014 [I 2022-10-28 06:04:52] ...trap/engines/engine.py.321: eval: epoch 109 | batch 310/510 [I 2022-10-28 06:04:52] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 06:04:52] ...trap/engines/engine.py.326: process: 0.28664 | load: 0.00014 [I 2022-10-28 06:04:55] ...trap/engines/engine.py.321: eval: epoch 109 | batch 320/510 [I 2022-10-28 06:04:55] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 06:04:55] ...trap/engines/engine.py.326: process: 0.29218 | load: 0.00014 [I 2022-10-28 06:04:58] ...trap/engines/engine.py.321: eval: epoch 109 | batch 330/510 [I 2022-10-28 06:04:58] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-28 06:04:58] ...trap/engines/engine.py.326: process: 0.28767 | load: 0.00014 [I 2022-10-28 06:05:01] ...trap/engines/engine.py.321: eval: epoch 109 | batch 340/510 [I 2022-10-28 06:05:01] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-28 06:05:01] ...trap/engines/engine.py.326: process: 0.28418 | load: 0.00019 [I 2022-10-28 06:05:04] ...trap/engines/engine.py.321: eval: epoch 109 | batch 350/510 [I 2022-10-28 06:05:04] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:45 [I 2022-10-28 06:05:04] ...trap/engines/engine.py.326: process: 0.29086 | load: 0.00015 [I 2022-10-28 06:05:07] ...trap/engines/engine.py.321: eval: epoch 109 | batch 360/510 [I 2022-10-28 06:05:07] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 06:05:07] ...trap/engines/engine.py.326: process: 0.28938 | load: 0.00016 [I 2022-10-28 06:05:09] ...trap/engines/engine.py.321: eval: epoch 109 | batch 370/510 [I 2022-10-28 06:05:09] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 06:05:09] ...trap/engines/engine.py.326: process: 0.28469 | load: 0.00015 [I 2022-10-28 06:05:12] ...trap/engines/engine.py.321: eval: epoch 109 | batch 380/510 [I 2022-10-28 06:05:12] ...trap/engines/engine.py.325: elapsed: 0:01:59 | left: 0:00:37 [I 2022-10-28 06:05:12] ...trap/engines/engine.py.326: process: 0.28670 | load: 0.00015 [I 2022-10-28 06:05:15] ...trap/engines/engine.py.321: eval: epoch 109 | batch 390/510 [I 2022-10-28 06:05:15] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 06:05:15] ...trap/engines/engine.py.326: process: 0.28871 | load: 0.00014 [I 2022-10-28 06:05:18] ...trap/engines/engine.py.321: eval: epoch 109 | batch 400/510 [I 2022-10-28 06:05:18] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 06:05:18] ...trap/engines/engine.py.326: process: 0.29140 | load: 0.00015 [I 2022-10-28 06:05:21] ...trap/engines/engine.py.321: eval: epoch 109 | batch 410/510 [I 2022-10-28 06:05:21] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 06:05:21] ...trap/engines/engine.py.326: process: 0.28886 | load: 0.00014 [I 2022-10-28 06:05:24] ...trap/engines/engine.py.321: eval: epoch 109 | batch 420/510 [I 2022-10-28 06:05:24] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 06:05:24] ...trap/engines/engine.py.326: process: 0.28642 | load: 0.00014 [I 2022-10-28 06:05:27] ...trap/engines/engine.py.321: eval: epoch 109 | batch 430/510 [I 2022-10-28 06:05:27] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 06:05:27] ...trap/engines/engine.py.326: process: 0.29116 | load: 0.00015 [I 2022-10-28 06:05:30] ...trap/engines/engine.py.321: eval: epoch 109 | batch 440/510 [I 2022-10-28 06:05:30] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 06:05:30] ...trap/engines/engine.py.326: process: 0.29033 | load: 0.00019 [I 2022-10-28 06:05:32] ...trap/engines/engine.py.321: eval: epoch 109 | batch 450/510 [I 2022-10-28 06:05:32] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 06:05:32] ...trap/engines/engine.py.326: process: 0.28281 | load: 0.00015 [I 2022-10-28 06:05:35] ...trap/engines/engine.py.321: eval: epoch 109 | batch 460/510 [I 2022-10-28 06:05:35] ...trap/engines/engine.py.325: elapsed: 0:02:22 | left: 0:00:14 [I 2022-10-28 06:05:35] ...trap/engines/engine.py.326: process: 0.28753 | load: 0.00016 [I 2022-10-28 06:05:38] ...trap/engines/engine.py.321: eval: epoch 109 | batch 470/510 [I 2022-10-28 06:05:38] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-28 06:05:38] ...trap/engines/engine.py.326: process: 0.28910 | load: 0.00014 [I 2022-10-28 06:05:41] ...trap/engines/engine.py.321: eval: epoch 109 | batch 480/510 [I 2022-10-28 06:05:41] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 06:05:41] ...trap/engines/engine.py.326: process: 0.28986 | load: 0.00016 [I 2022-10-28 06:05:44] ...trap/engines/engine.py.321: eval: epoch 109 | batch 490/510 [I 2022-10-28 06:05:44] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 06:05:44] ...trap/engines/engine.py.326: process: 0.28795 | load: 0.00019 [I 2022-10-28 06:05:47] ...trap/engines/engine.py.321: eval: epoch 109 | batch 500/510 [I 2022-10-28 06:05:47] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 06:05:47] ...trap/engines/engine.py.326: process: 0.28310 | load: 0.00015 [I 2022-10-28 06:05:50] ...trap/engines/engine.py.321: eval: epoch 109 | batch 510/510 [I 2022-10-28 06:05:50] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 06:05:50] ...trap/engines/engine.py.326: process: 0.28988 | load: 0.00011 [S 2022-10-28 06:05:50] ...trap/engines/engine.py.132: eval_epoch.epoch: 109 [I 2022-10-28 06:05:50] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.740100 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906200 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933300 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.738200 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905600 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.930700 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012095 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007679 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008149 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.008693 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007826 [S 2022-10-28 06:05:54] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007363 [I 2022-10-28 06:05:57] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 06:05:57] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 06:05:57] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 06:05:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 06:05:57] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 06:06:02] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 06:06:03] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 06:06:09] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 06:06:09] ...trap/engines/engine.py.163: Training model on trainset for epoch 110 [I 2022-10-28 06:06:09] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 06:06:12] ...trap/engines/engine.py.224: train: epoch 110 | batch 0/2304 [I 2022-10-28 06:06:12] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:15 [I 2022-10-28 06:06:12] ...trap/engines/engine.py.229: process: 2.68897 | load: 2.04724 [I 2022-10-28 06:06:12] ...trap/engines/engine.py.230: loss: 0.20518 [I 2022-10-28 06:06:12] ...optimizers/trijoint.py.151: total_norm: 19.450766 [I 2022-10-28 06:06:17] ...trap/engines/engine.py.224: train: epoch 110 | batch 10/2304 [I 2022-10-28 06:06:17] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:03 [I 2022-10-28 06:06:17] ...trap/engines/engine.py.229: process: 0.56819 | load: 0.00442 [I 2022-10-28 06:06:17] ...trap/engines/engine.py.230: loss: 0.09874 [I 2022-10-28 06:06:17] ...optimizers/trijoint.py.151: total_norm: 14.447294 [I 2022-10-28 06:06:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 20/2304 [I 2022-10-28 06:06:23] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:15 [I 2022-10-28 06:06:23] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00333 [I 2022-10-28 06:06:23] ...trap/engines/engine.py.230: loss: 0.09953 [I 2022-10-28 06:06:23] ...optimizers/trijoint.py.151: total_norm: 13.991643 [I 2022-10-28 06:06:28] ...trap/engines/engine.py.224: train: epoch 110 | batch 30/2304 [I 2022-10-28 06:06:28] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:12 [I 2022-10-28 06:06:28] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00371 [I 2022-10-28 06:06:28] ...trap/engines/engine.py.230: loss: 0.11125 [I 2022-10-28 06:06:28] ...optimizers/trijoint.py.151: total_norm: 27.662920 [I 2022-10-28 06:06:34] ...trap/engines/engine.py.224: train: epoch 110 | batch 40/2304 [I 2022-10-28 06:06:34] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:03 [I 2022-10-28 06:06:34] ...trap/engines/engine.py.229: process: 0.56053 | load: 0.00322 [I 2022-10-28 06:06:34] ...trap/engines/engine.py.230: loss: 0.08570 [I 2022-10-28 06:06:34] ...optimizers/trijoint.py.151: total_norm: 13.508234 [I 2022-10-28 06:06:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 50/2304 [I 2022-10-28 06:06:39] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:58 [I 2022-10-28 06:06:39] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00407 [I 2022-10-28 06:06:39] ...trap/engines/engine.py.230: loss: 0.16097 [I 2022-10-28 06:06:39] ...optimizers/trijoint.py.151: total_norm: 13.657869 [I 2022-10-28 06:06:45] ...trap/engines/engine.py.224: train: epoch 110 | batch 60/2304 [I 2022-10-28 06:06:45] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:45 [I 2022-10-28 06:06:45] ...trap/engines/engine.py.229: process: 0.54786 | load: 0.00284 [I 2022-10-28 06:06:45] ...trap/engines/engine.py.230: loss: 0.11715 [I 2022-10-28 06:06:45] ...optimizers/trijoint.py.151: total_norm: 21.162457 [I 2022-10-28 06:06:51] ...trap/engines/engine.py.224: train: epoch 110 | batch 70/2304 [I 2022-10-28 06:06:51] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:44 [I 2022-10-28 06:06:51] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00270 [I 2022-10-28 06:06:51] ...trap/engines/engine.py.230: loss: 0.12552 [I 2022-10-28 06:06:51] ...optimizers/trijoint.py.151: total_norm: 15.445743 [I 2022-10-28 06:06:56] ...trap/engines/engine.py.224: train: epoch 110 | batch 80/2304 [I 2022-10-28 06:06:56] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:21:01 [I 2022-10-28 06:06:56] ...trap/engines/engine.py.229: process: 0.55404 | load: 0.00286 [I 2022-10-28 06:06:56] ...trap/engines/engine.py.230: loss: 0.15190 [I 2022-10-28 06:06:56] ...optimizers/trijoint.py.151: total_norm: 13.144554 [I 2022-10-28 06:07:02] ...trap/engines/engine.py.224: train: epoch 110 | batch 90/2304 [I 2022-10-28 06:07:02] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:26 [I 2022-10-28 06:07:02] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00266 [I 2022-10-28 06:07:02] ...trap/engines/engine.py.230: loss: 0.15853 [I 2022-10-28 06:07:02] ...optimizers/trijoint.py.151: total_norm: 15.760490 [I 2022-10-28 06:07:08] ...trap/engines/engine.py.224: train: epoch 110 | batch 100/2304 [I 2022-10-28 06:07:08] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:35 [I 2022-10-28 06:07:08] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00259 [I 2022-10-28 06:07:08] ...trap/engines/engine.py.230: loss: 0.10248 [I 2022-10-28 06:07:08] ...optimizers/trijoint.py.151: total_norm: 8.811320 [I 2022-10-28 06:07:13] ...trap/engines/engine.py.224: train: epoch 110 | batch 110/2304 [I 2022-10-28 06:07:13] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:28 [I 2022-10-28 06:07:13] ...trap/engines/engine.py.229: process: 0.56066 | load: 0.00245 [I 2022-10-28 06:07:13] ...trap/engines/engine.py.230: loss: 0.09512 [I 2022-10-28 06:07:13] ...optimizers/trijoint.py.151: total_norm: 16.249218 [I 2022-10-28 06:07:19] ...trap/engines/engine.py.224: train: epoch 110 | batch 120/2304 [I 2022-10-28 06:07:19] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:18 [I 2022-10-28 06:07:19] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00256 [I 2022-10-28 06:07:19] ...trap/engines/engine.py.230: loss: 0.19200 [I 2022-10-28 06:07:19] ...optimizers/trijoint.py.151: total_norm: 16.173962 [I 2022-10-28 06:07:24] ...trap/engines/engine.py.224: train: epoch 110 | batch 130/2304 [I 2022-10-28 06:07:24] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:19:59 [I 2022-10-28 06:07:24] ...trap/engines/engine.py.229: process: 0.54856 | load: 0.00269 [I 2022-10-28 06:07:24] ...trap/engines/engine.py.230: loss: 0.13825 [I 2022-10-28 06:07:24] ...optimizers/trijoint.py.151: total_norm: 28.459862 [I 2022-10-28 06:07:30] ...trap/engines/engine.py.224: train: epoch 110 | batch 140/2304 [I 2022-10-28 06:07:30] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:05 [I 2022-10-28 06:07:30] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00261 [I 2022-10-28 06:07:30] ...trap/engines/engine.py.230: loss: 0.17826 [I 2022-10-28 06:07:30] ...optimizers/trijoint.py.151: total_norm: 24.509859 [I 2022-10-28 06:07:35] ...trap/engines/engine.py.224: train: epoch 110 | batch 150/2304 [I 2022-10-28 06:07:35] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:01 [I 2022-10-28 06:07:35] ...trap/engines/engine.py.229: process: 0.56476 | load: 0.00260 [I 2022-10-28 06:07:35] ...trap/engines/engine.py.230: loss: 0.11097 [I 2022-10-28 06:07:35] ...optimizers/trijoint.py.151: total_norm: 10.791726 [I 2022-10-28 06:07:41] ...trap/engines/engine.py.224: train: epoch 110 | batch 160/2304 [I 2022-10-28 06:07:41] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:24 [I 2022-10-28 06:07:41] ...trap/engines/engine.py.229: process: 0.57025 | load: 0.00265 [I 2022-10-28 06:07:41] ...trap/engines/engine.py.230: loss: 0.12456 [I 2022-10-28 06:07:41] ...optimizers/trijoint.py.151: total_norm: 20.792103 [I 2022-10-28 06:07:47] ...trap/engines/engine.py.224: train: epoch 110 | batch 170/2304 [I 2022-10-28 06:07:47] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:48 [I 2022-10-28 06:07:47] ...trap/engines/engine.py.229: process: 0.56108 | load: 0.00275 [I 2022-10-28 06:07:47] ...trap/engines/engine.py.230: loss: 0.10921 [I 2022-10-28 06:07:47] ...optimizers/trijoint.py.151: total_norm: 10.532002 [I 2022-10-28 06:07:52] ...trap/engines/engine.py.224: train: epoch 110 | batch 180/2304 [I 2022-10-28 06:07:52] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:44 [I 2022-10-28 06:07:52] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00252 [I 2022-10-28 06:07:52] ...trap/engines/engine.py.230: loss: 0.14996 [I 2022-10-28 06:07:52] ...optimizers/trijoint.py.151: total_norm: 21.355886 [I 2022-10-28 06:07:58] ...trap/engines/engine.py.224: train: epoch 110 | batch 190/2304 [I 2022-10-28 06:07:58] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:38 [I 2022-10-28 06:07:58] ...trap/engines/engine.py.229: process: 0.55183 | load: 0.00262 [I 2022-10-28 06:07:58] ...trap/engines/engine.py.230: loss: 0.13611 [I 2022-10-28 06:07:58] ...optimizers/trijoint.py.151: total_norm: 11.072343 [I 2022-10-28 06:08:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 200/2304 [I 2022-10-28 06:08:03] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:36 [I 2022-10-28 06:08:03] ...trap/engines/engine.py.229: process: 0.55575 | load: 0.00257 [I 2022-10-28 06:08:03] ...trap/engines/engine.py.230: loss: 0.13687 [I 2022-10-28 06:08:03] ...optimizers/trijoint.py.151: total_norm: 20.207985 [I 2022-10-28 06:08:09] ...trap/engines/engine.py.224: train: epoch 110 | batch 210/2304 [I 2022-10-28 06:08:09] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:35 [I 2022-10-28 06:08:09] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00244 [I 2022-10-28 06:08:09] ...trap/engines/engine.py.230: loss: 0.08233 [I 2022-10-28 06:08:09] ...optimizers/trijoint.py.151: total_norm: 12.143773 [I 2022-10-28 06:08:15] ...trap/engines/engine.py.224: train: epoch 110 | batch 220/2304 [I 2022-10-28 06:08:15] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:11 [I 2022-10-28 06:08:15] ...trap/engines/engine.py.229: process: 0.53533 | load: 0.00243 [I 2022-10-28 06:08:15] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-28 06:08:15] ...optimizers/trijoint.py.151: total_norm: 39.496365 [I 2022-10-28 06:08:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 230/2304 [I 2022-10-28 06:08:20] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:26 [I 2022-10-28 06:08:20] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00243 [I 2022-10-28 06:08:20] ...trap/engines/engine.py.230: loss: 0.08946 [I 2022-10-28 06:08:20] ...optimizers/trijoint.py.151: total_norm: 12.763613 [I 2022-10-28 06:08:26] ...trap/engines/engine.py.224: train: epoch 110 | batch 240/2304 [I 2022-10-28 06:08:26] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:08 [I 2022-10-28 06:08:26] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00244 [I 2022-10-28 06:08:26] ...trap/engines/engine.py.230: loss: 0.18108 [I 2022-10-28 06:08:26] ...optimizers/trijoint.py.151: total_norm: 12.150872 [I 2022-10-28 06:08:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 250/2304 [I 2022-10-28 06:08:31] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:00 [I 2022-10-28 06:08:31] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00259 [I 2022-10-28 06:08:31] ...trap/engines/engine.py.230: loss: 0.18323 [I 2022-10-28 06:08:31] ...optimizers/trijoint.py.151: total_norm: 18.830490 [I 2022-10-28 06:08:37] ...trap/engines/engine.py.224: train: epoch 110 | batch 260/2304 [I 2022-10-28 06:08:37] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:57 [I 2022-10-28 06:08:37] ...trap/engines/engine.py.229: process: 0.55756 | load: 0.00256 [I 2022-10-28 06:08:37] ...trap/engines/engine.py.230: loss: 0.10658 [I 2022-10-28 06:08:37] ...optimizers/trijoint.py.151: total_norm: 9.497504 [I 2022-10-28 06:08:43] ...trap/engines/engine.py.224: train: epoch 110 | batch 270/2304 [I 2022-10-28 06:08:43] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:43 [I 2022-10-28 06:08:43] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00255 [I 2022-10-28 06:08:43] ...trap/engines/engine.py.230: loss: 0.12008 [I 2022-10-28 06:08:43] ...optimizers/trijoint.py.151: total_norm: 17.890495 [I 2022-10-28 06:08:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 280/2304 [I 2022-10-28 06:08:48] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:46 [I 2022-10-28 06:08:48] ...trap/engines/engine.py.229: process: 0.54736 | load: 0.00244 [I 2022-10-28 06:08:48] ...trap/engines/engine.py.230: loss: 0.08832 [I 2022-10-28 06:08:48] ...optimizers/trijoint.py.151: total_norm: 17.855621 [I 2022-10-28 06:08:54] ...trap/engines/engine.py.224: train: epoch 110 | batch 290/2304 [I 2022-10-28 06:08:54] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:44 [I 2022-10-28 06:08:54] ...trap/engines/engine.py.229: process: 0.56886 | load: 0.00244 [I 2022-10-28 06:08:54] ...trap/engines/engine.py.230: loss: 0.11556 [I 2022-10-28 06:08:54] ...optimizers/trijoint.py.151: total_norm: 15.323977 [I 2022-10-28 06:08:59] ...trap/engines/engine.py.224: train: epoch 110 | batch 300/2304 [I 2022-10-28 06:08:59] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:46 [I 2022-10-28 06:08:59] ...trap/engines/engine.py.229: process: 0.56171 | load: 0.00245 [I 2022-10-28 06:08:59] ...trap/engines/engine.py.230: loss: 0.07204 [I 2022-10-28 06:08:59] ...optimizers/trijoint.py.151: total_norm: 8.640460 [I 2022-10-28 06:09:05] ...trap/engines/engine.py.224: train: epoch 110 | batch 310/2304 [I 2022-10-28 06:09:05] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:27 [I 2022-10-28 06:09:05] ...trap/engines/engine.py.229: process: 0.55125 | load: 0.00254 [I 2022-10-28 06:09:05] ...trap/engines/engine.py.230: loss: 0.08248 [I 2022-10-28 06:09:05] ...optimizers/trijoint.py.151: total_norm: 10.443426 [I 2022-10-28 06:09:11] ...trap/engines/engine.py.224: train: epoch 110 | batch 320/2304 [I 2022-10-28 06:09:11] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:16 [I 2022-10-28 06:09:11] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00248 [I 2022-10-28 06:09:11] ...trap/engines/engine.py.230: loss: 0.10221 [I 2022-10-28 06:09:11] ...optimizers/trijoint.py.151: total_norm: 9.758114 [I 2022-10-28 06:09:16] ...trap/engines/engine.py.224: train: epoch 110 | batch 330/2304 [I 2022-10-28 06:09:16] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:15 [I 2022-10-28 06:09:16] ...trap/engines/engine.py.229: process: 0.56731 | load: 0.00248 [I 2022-10-28 06:09:16] ...trap/engines/engine.py.230: loss: 0.19954 [I 2022-10-28 06:09:16] ...optimizers/trijoint.py.151: total_norm: 13.147324 [I 2022-10-28 06:09:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 340/2304 [I 2022-10-28 06:09:22] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:11 [I 2022-10-28 06:09:22] ...trap/engines/engine.py.229: process: 0.55275 | load: 0.00243 [I 2022-10-28 06:09:22] ...trap/engines/engine.py.230: loss: 0.10188 [I 2022-10-28 06:09:22] ...optimizers/trijoint.py.151: total_norm: 10.105008 [I 2022-10-28 06:09:27] ...trap/engines/engine.py.224: train: epoch 110 | batch 350/2304 [I 2022-10-28 06:09:27] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:08 [I 2022-10-28 06:09:27] ...trap/engines/engine.py.229: process: 0.54868 | load: 0.00261 [I 2022-10-28 06:09:27] ...trap/engines/engine.py.230: loss: 0.20819 [I 2022-10-28 06:09:27] ...optimizers/trijoint.py.151: total_norm: 17.450583 [I 2022-10-28 06:09:33] ...trap/engines/engine.py.224: train: epoch 110 | batch 360/2304 [I 2022-10-28 06:09:33] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:11 [I 2022-10-28 06:09:33] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00247 [I 2022-10-28 06:09:33] ...trap/engines/engine.py.230: loss: 0.08177 [I 2022-10-28 06:09:33] ...optimizers/trijoint.py.151: total_norm: 22.859837 [I 2022-10-28 06:09:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 370/2304 [I 2022-10-28 06:09:39] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:59 [I 2022-10-28 06:09:39] ...trap/engines/engine.py.229: process: 0.56467 | load: 0.00243 [I 2022-10-28 06:09:39] ...trap/engines/engine.py.230: loss: 0.10025 [I 2022-10-28 06:09:39] ...optimizers/trijoint.py.151: total_norm: 17.863016 [I 2022-10-28 06:09:44] ...trap/engines/engine.py.224: train: epoch 110 | batch 380/2304 [I 2022-10-28 06:09:44] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:50 [I 2022-10-28 06:09:44] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00245 [I 2022-10-28 06:09:44] ...trap/engines/engine.py.230: loss: 0.13595 [I 2022-10-28 06:09:44] ...optimizers/trijoint.py.151: total_norm: 16.801041 [I 2022-10-28 06:09:50] ...trap/engines/engine.py.224: train: epoch 110 | batch 390/2304 [I 2022-10-28 06:09:50] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:52 [I 2022-10-28 06:09:50] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00257 [I 2022-10-28 06:09:50] ...trap/engines/engine.py.230: loss: 0.09923 [I 2022-10-28 06:09:50] ...optimizers/trijoint.py.151: total_norm: 9.656038 [I 2022-10-28 06:09:55] ...trap/engines/engine.py.224: train: epoch 110 | batch 400/2304 [I 2022-10-28 06:09:55] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:54 [I 2022-10-28 06:09:55] ...trap/engines/engine.py.229: process: 0.58595 | load: 0.00255 [I 2022-10-28 06:09:55] ...trap/engines/engine.py.230: loss: 0.09533 [I 2022-10-28 06:09:55] ...optimizers/trijoint.py.151: total_norm: 14.695811 [I 2022-10-28 06:10:01] ...trap/engines/engine.py.224: train: epoch 110 | batch 410/2304 [I 2022-10-28 06:10:01] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:44 [I 2022-10-28 06:10:01] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00247 [I 2022-10-28 06:10:01] ...trap/engines/engine.py.230: loss: 0.12339 [I 2022-10-28 06:10:01] ...optimizers/trijoint.py.151: total_norm: 12.525946 [I 2022-10-28 06:10:07] ...trap/engines/engine.py.224: train: epoch 110 | batch 420/2304 [I 2022-10-28 06:10:07] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:45 [I 2022-10-28 06:10:07] ...trap/engines/engine.py.229: process: 0.55572 | load: 0.00246 [I 2022-10-28 06:10:07] ...trap/engines/engine.py.230: loss: 0.11394 [I 2022-10-28 06:10:07] ...optimizers/trijoint.py.151: total_norm: 12.193622 [I 2022-10-28 06:10:12] ...trap/engines/engine.py.224: train: epoch 110 | batch 430/2304 [I 2022-10-28 06:10:12] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:26 [I 2022-10-28 06:10:12] ...trap/engines/engine.py.229: process: 0.56207 | load: 0.00247 [I 2022-10-28 06:10:12] ...trap/engines/engine.py.230: loss: 0.16845 [I 2022-10-28 06:10:12] ...optimizers/trijoint.py.151: total_norm: 15.398619 [I 2022-10-28 06:10:18] ...trap/engines/engine.py.224: train: epoch 110 | batch 440/2304 [I 2022-10-28 06:10:18] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:10 [I 2022-10-28 06:10:18] ...trap/engines/engine.py.229: process: 0.54957 | load: 0.00246 [I 2022-10-28 06:10:18] ...trap/engines/engine.py.230: loss: 0.18778 [I 2022-10-28 06:10:18] ...optimizers/trijoint.py.151: total_norm: 16.750076 [I 2022-10-28 06:10:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 450/2304 [I 2022-10-28 06:10:23] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:09 [I 2022-10-28 06:10:23] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00246 [I 2022-10-28 06:10:23] ...trap/engines/engine.py.230: loss: 0.13352 [I 2022-10-28 06:10:23] ...optimizers/trijoint.py.151: total_norm: 15.527059 [I 2022-10-28 06:10:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 460/2304 [I 2022-10-28 06:10:29] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:05 [I 2022-10-28 06:10:29] ...trap/engines/engine.py.229: process: 0.55851 | load: 0.00242 [I 2022-10-28 06:10:29] ...trap/engines/engine.py.230: loss: 0.10975 [I 2022-10-28 06:10:29] ...optimizers/trijoint.py.151: total_norm: 14.688030 [I 2022-10-28 06:10:35] ...trap/engines/engine.py.224: train: epoch 110 | batch 470/2304 [I 2022-10-28 06:10:35] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:59 [I 2022-10-28 06:10:35] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00240 [I 2022-10-28 06:10:35] ...trap/engines/engine.py.230: loss: 0.12550 [I 2022-10-28 06:10:35] ...optimizers/trijoint.py.151: total_norm: 8.817169 [I 2022-10-28 06:10:40] ...trap/engines/engine.py.224: train: epoch 110 | batch 480/2304 [I 2022-10-28 06:10:40] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:54 [I 2022-10-28 06:10:40] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00241 [I 2022-10-28 06:10:40] ...trap/engines/engine.py.230: loss: 0.08547 [I 2022-10-28 06:10:40] ...optimizers/trijoint.py.151: total_norm: 13.525187 [I 2022-10-28 06:10:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 490/2304 [I 2022-10-28 06:10:46] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:17:01 [I 2022-10-28 06:10:46] ...trap/engines/engine.py.229: process: 0.54843 | load: 0.00244 [I 2022-10-28 06:10:46] ...trap/engines/engine.py.230: loss: 0.16124 [I 2022-10-28 06:10:46] ...optimizers/trijoint.py.151: total_norm: 27.057470 [I 2022-10-28 06:10:52] ...trap/engines/engine.py.224: train: epoch 110 | batch 500/2304 [I 2022-10-28 06:10:52] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:45 [I 2022-10-28 06:10:52] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00249 [I 2022-10-28 06:10:52] ...trap/engines/engine.py.230: loss: 0.09138 [I 2022-10-28 06:10:52] ...optimizers/trijoint.py.151: total_norm: 11.289345 [I 2022-10-28 06:10:57] ...trap/engines/engine.py.224: train: epoch 110 | batch 510/2304 [I 2022-10-28 06:10:57] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:37 [I 2022-10-28 06:10:57] ...trap/engines/engine.py.229: process: 0.56909 | load: 0.00248 [I 2022-10-28 06:10:57] ...trap/engines/engine.py.230: loss: 0.09605 [I 2022-10-28 06:10:57] ...optimizers/trijoint.py.151: total_norm: 8.625903 [I 2022-10-28 06:11:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 520/2304 [I 2022-10-28 06:11:03] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:29 [I 2022-10-28 06:11:03] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00247 [I 2022-10-28 06:11:03] ...trap/engines/engine.py.230: loss: 0.11174 [I 2022-10-28 06:11:03] ...optimizers/trijoint.py.151: total_norm: 39.877575 [I 2022-10-28 06:11:08] ...trap/engines/engine.py.224: train: epoch 110 | batch 530/2304 [I 2022-10-28 06:11:08] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:31 [I 2022-10-28 06:11:08] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00246 [I 2022-10-28 06:11:08] ...trap/engines/engine.py.230: loss: 0.12367 [I 2022-10-28 06:11:08] ...optimizers/trijoint.py.151: total_norm: 14.292715 [I 2022-10-28 06:11:14] ...trap/engines/engine.py.224: train: epoch 110 | batch 540/2304 [I 2022-10-28 06:11:14] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:20 [I 2022-10-28 06:11:14] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00244 [I 2022-10-28 06:11:14] ...trap/engines/engine.py.230: loss: 0.10133 [I 2022-10-28 06:11:14] ...optimizers/trijoint.py.151: total_norm: 12.960613 [I 2022-10-28 06:11:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 550/2304 [I 2022-10-28 06:11:20] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:53 [I 2022-10-28 06:11:20] ...trap/engines/engine.py.229: process: 0.56733 | load: 0.00245 [I 2022-10-28 06:11:20] ...trap/engines/engine.py.230: loss: 0.10536 [I 2022-10-28 06:11:20] ...optimizers/trijoint.py.151: total_norm: 33.021164 [I 2022-10-28 06:11:25] ...trap/engines/engine.py.224: train: epoch 110 | batch 560/2304 [I 2022-10-28 06:11:25] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:13 [I 2022-10-28 06:11:25] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00254 [I 2022-10-28 06:11:25] ...trap/engines/engine.py.230: loss: 0.09563 [I 2022-10-28 06:11:25] ...optimizers/trijoint.py.151: total_norm: 18.159073 [I 2022-10-28 06:11:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 570/2304 [I 2022-10-28 06:11:31] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:10 [I 2022-10-28 06:11:31] ...trap/engines/engine.py.229: process: 0.56055 | load: 0.00242 [I 2022-10-28 06:11:31] ...trap/engines/engine.py.230: loss: 0.20330 [I 2022-10-28 06:11:31] ...optimizers/trijoint.py.151: total_norm: 27.261305 [I 2022-10-28 06:11:36] ...trap/engines/engine.py.224: train: epoch 110 | batch 580/2304 [I 2022-10-28 06:11:36] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:04 [I 2022-10-28 06:11:36] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00244 [I 2022-10-28 06:11:36] ...trap/engines/engine.py.230: loss: 0.08664 [I 2022-10-28 06:11:36] ...optimizers/trijoint.py.151: total_norm: 13.089004 [I 2022-10-28 06:11:42] ...trap/engines/engine.py.224: train: epoch 110 | batch 590/2304 [I 2022-10-28 06:11:42] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:07 [I 2022-10-28 06:11:42] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00263 [I 2022-10-28 06:11:42] ...trap/engines/engine.py.230: loss: 0.09073 [I 2022-10-28 06:11:42] ...optimizers/trijoint.py.151: total_norm: 12.227361 [I 2022-10-28 06:11:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 600/2304 [I 2022-10-28 06:11:48] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:48 [I 2022-10-28 06:11:48] ...trap/engines/engine.py.229: process: 0.55591 | load: 0.00251 [I 2022-10-28 06:11:48] ...trap/engines/engine.py.230: loss: 0.16213 [I 2022-10-28 06:11:48] ...optimizers/trijoint.py.151: total_norm: 14.640477 [I 2022-10-28 06:11:53] ...trap/engines/engine.py.224: train: epoch 110 | batch 610/2304 [I 2022-10-28 06:11:53] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:51 [I 2022-10-28 06:11:53] ...trap/engines/engine.py.229: process: 0.58046 | load: 0.00245 [I 2022-10-28 06:11:53] ...trap/engines/engine.py.230: loss: 0.09616 [I 2022-10-28 06:11:53] ...optimizers/trijoint.py.151: total_norm: 14.310019 [I 2022-10-28 06:11:59] ...trap/engines/engine.py.224: train: epoch 110 | batch 620/2304 [I 2022-10-28 06:11:59] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:44 [I 2022-10-28 06:11:59] ...trap/engines/engine.py.229: process: 0.56111 | load: 0.00250 [I 2022-10-28 06:11:59] ...trap/engines/engine.py.230: loss: 0.13078 [I 2022-10-28 06:11:59] ...optimizers/trijoint.py.151: total_norm: 12.756726 [I 2022-10-28 06:12:04] ...trap/engines/engine.py.224: train: epoch 110 | batch 630/2304 [I 2022-10-28 06:12:04] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:29 [I 2022-10-28 06:12:04] ...trap/engines/engine.py.229: process: 0.55045 | load: 0.00244 [I 2022-10-28 06:12:04] ...trap/engines/engine.py.230: loss: 0.10275 [I 2022-10-28 06:12:04] ...optimizers/trijoint.py.151: total_norm: 35.808052 [I 2022-10-28 06:12:10] ...trap/engines/engine.py.224: train: epoch 110 | batch 640/2304 [I 2022-10-28 06:12:10] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:24 [I 2022-10-28 06:12:10] ...trap/engines/engine.py.229: process: 0.55631 | load: 0.00246 [I 2022-10-28 06:12:10] ...trap/engines/engine.py.230: loss: 0.15422 [I 2022-10-28 06:12:10] ...optimizers/trijoint.py.151: total_norm: 11.454524 [I 2022-10-28 06:12:16] ...trap/engines/engine.py.224: train: epoch 110 | batch 650/2304 [I 2022-10-28 06:12:16] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:16 [I 2022-10-28 06:12:16] ...trap/engines/engine.py.229: process: 0.55003 | load: 0.00240 [I 2022-10-28 06:12:16] ...trap/engines/engine.py.230: loss: 0.10651 [I 2022-10-28 06:12:16] ...optimizers/trijoint.py.151: total_norm: 20.131790 [I 2022-10-28 06:12:21] ...trap/engines/engine.py.224: train: epoch 110 | batch 660/2304 [I 2022-10-28 06:12:21] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:20 [I 2022-10-28 06:12:21] ...trap/engines/engine.py.229: process: 0.56395 | load: 0.00245 [I 2022-10-28 06:12:21] ...trap/engines/engine.py.230: loss: 0.17188 [I 2022-10-28 06:12:21] ...optimizers/trijoint.py.151: total_norm: 11.354497 [I 2022-10-28 06:12:27] ...trap/engines/engine.py.224: train: epoch 110 | batch 670/2304 [I 2022-10-28 06:12:27] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:13 [I 2022-10-28 06:12:27] ...trap/engines/engine.py.229: process: 0.56042 | load: 0.00243 [I 2022-10-28 06:12:27] ...trap/engines/engine.py.230: loss: 0.10945 [I 2022-10-28 06:12:27] ...optimizers/trijoint.py.151: total_norm: 14.914813 [I 2022-10-28 06:12:32] ...trap/engines/engine.py.224: train: epoch 110 | batch 680/2304 [I 2022-10-28 06:12:32] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:04 [I 2022-10-28 06:12:32] ...trap/engines/engine.py.229: process: 0.54644 | load: 0.00247 [I 2022-10-28 06:12:32] ...trap/engines/engine.py.230: loss: 0.15042 [I 2022-10-28 06:12:32] ...optimizers/trijoint.py.151: total_norm: 16.544889 [I 2022-10-28 06:12:38] ...trap/engines/engine.py.224: train: epoch 110 | batch 690/2304 [I 2022-10-28 06:12:38] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:14:57 [I 2022-10-28 06:12:38] ...trap/engines/engine.py.229: process: 0.55024 | load: 0.00258 [I 2022-10-28 06:12:38] ...trap/engines/engine.py.230: loss: 0.13767 [I 2022-10-28 06:12:38] ...optimizers/trijoint.py.151: total_norm: 10.097326 [I 2022-10-28 06:12:43] ...trap/engines/engine.py.224: train: epoch 110 | batch 700/2304 [I 2022-10-28 06:12:43] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:51 [I 2022-10-28 06:12:43] ...trap/engines/engine.py.229: process: 0.55619 | load: 0.00244 [I 2022-10-28 06:12:43] ...trap/engines/engine.py.230: loss: 0.10303 [I 2022-10-28 06:12:43] ...optimizers/trijoint.py.151: total_norm: 8.997268 [I 2022-10-28 06:12:49] ...trap/engines/engine.py.224: train: epoch 110 | batch 710/2304 [I 2022-10-28 06:12:49] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:44 [I 2022-10-28 06:12:49] ...trap/engines/engine.py.229: process: 0.55392 | load: 0.00251 [I 2022-10-28 06:12:49] ...trap/engines/engine.py.230: loss: 0.09546 [I 2022-10-28 06:12:49] ...optimizers/trijoint.py.151: total_norm: 9.559136 [I 2022-10-28 06:12:55] ...trap/engines/engine.py.224: train: epoch 110 | batch 720/2304 [I 2022-10-28 06:12:55] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:47 [I 2022-10-28 06:12:55] ...trap/engines/engine.py.229: process: 0.56421 | load: 0.00240 [I 2022-10-28 06:12:55] ...trap/engines/engine.py.230: loss: 0.16782 [I 2022-10-28 06:12:55] ...optimizers/trijoint.py.151: total_norm: 14.281290 [I 2022-10-28 06:13:00] ...trap/engines/engine.py.224: train: epoch 110 | batch 730/2304 [I 2022-10-28 06:13:00] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:39 [I 2022-10-28 06:13:00] ...trap/engines/engine.py.229: process: 0.55699 | load: 0.00248 [I 2022-10-28 06:13:00] ...trap/engines/engine.py.230: loss: 0.09761 [I 2022-10-28 06:13:00] ...optimizers/trijoint.py.151: total_norm: 14.453905 [I 2022-10-28 06:13:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 740/2304 [I 2022-10-28 06:13:06] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:34 [I 2022-10-28 06:13:06] ...trap/engines/engine.py.229: process: 0.56369 | load: 0.00246 [I 2022-10-28 06:13:06] ...trap/engines/engine.py.230: loss: 0.10784 [I 2022-10-28 06:13:06] ...optimizers/trijoint.py.151: total_norm: 10.106297 [I 2022-10-28 06:13:11] ...trap/engines/engine.py.224: train: epoch 110 | batch 750/2304 [I 2022-10-28 06:13:11] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:31 [I 2022-10-28 06:13:11] ...trap/engines/engine.py.229: process: 0.57248 | load: 0.00244 [I 2022-10-28 06:13:11] ...trap/engines/engine.py.230: loss: 0.11781 [I 2022-10-28 06:13:11] ...optimizers/trijoint.py.151: total_norm: 26.438265 [I 2022-10-28 06:13:17] ...trap/engines/engine.py.224: train: epoch 110 | batch 760/2304 [I 2022-10-28 06:13:17] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:22 [I 2022-10-28 06:13:17] ...trap/engines/engine.py.229: process: 0.56629 | load: 0.00241 [I 2022-10-28 06:13:17] ...trap/engines/engine.py.230: loss: 0.15507 [I 2022-10-28 06:13:17] ...optimizers/trijoint.py.151: total_norm: 11.438337 [I 2022-10-28 06:13:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 770/2304 [I 2022-10-28 06:13:23] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:14 [I 2022-10-28 06:13:23] ...trap/engines/engine.py.229: process: 0.55041 | load: 0.00250 [I 2022-10-28 06:13:23] ...trap/engines/engine.py.230: loss: 0.09766 [I 2022-10-28 06:13:23] ...optimizers/trijoint.py.151: total_norm: 20.365660 [I 2022-10-28 06:13:28] ...trap/engines/engine.py.224: train: epoch 110 | batch 780/2304 [I 2022-10-28 06:13:28] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:11 [I 2022-10-28 06:13:28] ...trap/engines/engine.py.229: process: 0.55931 | load: 0.00252 [I 2022-10-28 06:13:28] ...trap/engines/engine.py.230: loss: 0.12670 [I 2022-10-28 06:13:28] ...optimizers/trijoint.py.151: total_norm: 14.676760 [I 2022-10-28 06:13:34] ...trap/engines/engine.py.224: train: epoch 110 | batch 790/2304 [I 2022-10-28 06:13:34] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:14 [I 2022-10-28 06:13:34] ...trap/engines/engine.py.229: process: 0.57949 | load: 0.00245 [I 2022-10-28 06:13:34] ...trap/engines/engine.py.230: loss: 0.13980 [I 2022-10-28 06:13:34] ...optimizers/trijoint.py.151: total_norm: 13.221148 [I 2022-10-28 06:13:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 800/2304 [I 2022-10-28 06:13:39] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:01 [I 2022-10-28 06:13:39] ...trap/engines/engine.py.229: process: 0.56859 | load: 0.00243 [I 2022-10-28 06:13:39] ...trap/engines/engine.py.230: loss: 0.13503 [I 2022-10-28 06:13:39] ...optimizers/trijoint.py.151: total_norm: 11.798526 [I 2022-10-28 06:13:45] ...trap/engines/engine.py.224: train: epoch 110 | batch 810/2304 [I 2022-10-28 06:13:45] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:49 [I 2022-10-28 06:13:45] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00246 [I 2022-10-28 06:13:45] ...trap/engines/engine.py.230: loss: 0.12096 [I 2022-10-28 06:13:45] ...optimizers/trijoint.py.151: total_norm: 15.358404 [I 2022-10-28 06:13:51] ...trap/engines/engine.py.224: train: epoch 110 | batch 820/2304 [I 2022-10-28 06:13:51] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:47 [I 2022-10-28 06:13:51] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00247 [I 2022-10-28 06:13:51] ...trap/engines/engine.py.230: loss: 0.15882 [I 2022-10-28 06:13:51] ...optimizers/trijoint.py.151: total_norm: 11.119057 [I 2022-10-28 06:13:56] ...trap/engines/engine.py.224: train: epoch 110 | batch 830/2304 [I 2022-10-28 06:13:56] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:33 [I 2022-10-28 06:13:56] ...trap/engines/engine.py.229: process: 0.56773 | load: 0.00247 [I 2022-10-28 06:13:56] ...trap/engines/engine.py.230: loss: 0.07612 [I 2022-10-28 06:13:56] ...optimizers/trijoint.py.151: total_norm: 7.304978 [I 2022-10-28 06:14:02] ...trap/engines/engine.py.224: train: epoch 110 | batch 840/2304 [I 2022-10-28 06:14:02] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:34 [I 2022-10-28 06:14:02] ...trap/engines/engine.py.229: process: 0.56306 | load: 0.00245 [I 2022-10-28 06:14:02] ...trap/engines/engine.py.230: loss: 0.14205 [I 2022-10-28 06:14:02] ...optimizers/trijoint.py.151: total_norm: 14.585969 [I 2022-10-28 06:14:07] ...trap/engines/engine.py.224: train: epoch 110 | batch 850/2304 [I 2022-10-28 06:14:07] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:28 [I 2022-10-28 06:14:07] ...trap/engines/engine.py.229: process: 0.56983 | load: 0.00245 [I 2022-10-28 06:14:07] ...trap/engines/engine.py.230: loss: 0.13823 [I 2022-10-28 06:14:07] ...optimizers/trijoint.py.151: total_norm: 16.860455 [I 2022-10-28 06:14:13] ...trap/engines/engine.py.224: train: epoch 110 | batch 860/2304 [I 2022-10-28 06:14:13] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:27 [I 2022-10-28 06:14:13] ...trap/engines/engine.py.229: process: 0.56737 | load: 0.00246 [I 2022-10-28 06:14:13] ...trap/engines/engine.py.230: loss: 0.09407 [I 2022-10-28 06:14:13] ...optimizers/trijoint.py.151: total_norm: 16.184088 [I 2022-10-28 06:14:18] ...trap/engines/engine.py.224: train: epoch 110 | batch 870/2304 [I 2022-10-28 06:14:18] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:12 [I 2022-10-28 06:14:18] ...trap/engines/engine.py.229: process: 0.55109 | load: 0.00246 [I 2022-10-28 06:14:18] ...trap/engines/engine.py.230: loss: 0.10519 [I 2022-10-28 06:14:18] ...optimizers/trijoint.py.151: total_norm: 11.073045 [I 2022-10-28 06:14:24] ...trap/engines/engine.py.224: train: epoch 110 | batch 880/2304 [I 2022-10-28 06:14:24] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:10 [I 2022-10-28 06:14:24] ...trap/engines/engine.py.229: process: 0.55334 | load: 0.00242 [I 2022-10-28 06:14:24] ...trap/engines/engine.py.230: loss: 0.09100 [I 2022-10-28 06:14:24] ...optimizers/trijoint.py.151: total_norm: 9.867934 [I 2022-10-28 06:14:30] ...trap/engines/engine.py.224: train: epoch 110 | batch 890/2304 [I 2022-10-28 06:14:30] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:09 [I 2022-10-28 06:14:30] ...trap/engines/engine.py.229: process: 0.56771 | load: 0.00266 [I 2022-10-28 06:14:30] ...trap/engines/engine.py.230: loss: 0.21201 [I 2022-10-28 06:14:30] ...optimizers/trijoint.py.151: total_norm: 24.152103 [I 2022-10-28 06:14:35] ...trap/engines/engine.py.224: train: epoch 110 | batch 900/2304 [I 2022-10-28 06:14:35] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:12:59 [I 2022-10-28 06:14:35] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00244 [I 2022-10-28 06:14:35] ...trap/engines/engine.py.230: loss: 0.09210 [I 2022-10-28 06:14:35] ...optimizers/trijoint.py.151: total_norm: 15.241606 [I 2022-10-28 06:14:41] ...trap/engines/engine.py.224: train: epoch 110 | batch 910/2304 [I 2022-10-28 06:14:41] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:12:53 [I 2022-10-28 06:14:41] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00248 [I 2022-10-28 06:14:41] ...trap/engines/engine.py.230: loss: 0.12179 [I 2022-10-28 06:14:41] ...optimizers/trijoint.py.151: total_norm: 16.058287 [I 2022-10-28 06:14:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 920/2304 [I 2022-10-28 06:14:46] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:51 [I 2022-10-28 06:14:46] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00247 [I 2022-10-28 06:14:46] ...trap/engines/engine.py.230: loss: 0.09601 [I 2022-10-28 06:14:46] ...optimizers/trijoint.py.151: total_norm: 7.861767 [I 2022-10-28 06:14:52] ...trap/engines/engine.py.224: train: epoch 110 | batch 930/2304 [I 2022-10-28 06:14:52] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:40 [I 2022-10-28 06:14:52] ...trap/engines/engine.py.229: process: 0.55980 | load: 0.00245 [I 2022-10-28 06:14:52] ...trap/engines/engine.py.230: loss: 0.09930 [I 2022-10-28 06:14:52] ...optimizers/trijoint.py.151: total_norm: 9.801162 [I 2022-10-28 06:14:57] ...trap/engines/engine.py.224: train: epoch 110 | batch 940/2304 [I 2022-10-28 06:14:57] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:39 [I 2022-10-28 06:14:57] ...trap/engines/engine.py.229: process: 0.54921 | load: 0.00241 [I 2022-10-28 06:14:57] ...trap/engines/engine.py.230: loss: 0.07530 [I 2022-10-28 06:14:57] ...optimizers/trijoint.py.151: total_norm: 11.038150 [I 2022-10-28 06:15:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 950/2304 [I 2022-10-28 06:15:03] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:38 [I 2022-10-28 06:15:03] ...trap/engines/engine.py.229: process: 0.56985 | load: 0.00244 [I 2022-10-28 06:15:03] ...trap/engines/engine.py.230: loss: 0.13004 [I 2022-10-28 06:15:03] ...optimizers/trijoint.py.151: total_norm: 24.922876 [I 2022-10-28 06:15:09] ...trap/engines/engine.py.224: train: epoch 110 | batch 960/2304 [I 2022-10-28 06:15:09] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:36 [I 2022-10-28 06:15:09] ...trap/engines/engine.py.229: process: 0.58073 | load: 0.00250 [I 2022-10-28 06:15:09] ...trap/engines/engine.py.230: loss: 0.11160 [I 2022-10-28 06:15:09] ...optimizers/trijoint.py.151: total_norm: 12.406108 [I 2022-10-28 06:15:14] ...trap/engines/engine.py.224: train: epoch 110 | batch 970/2304 [I 2022-10-28 06:15:14] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:26 [I 2022-10-28 06:15:14] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00257 [I 2022-10-28 06:15:14] ...trap/engines/engine.py.230: loss: 0.14694 [I 2022-10-28 06:15:14] ...optimizers/trijoint.py.151: total_norm: 28.444284 [I 2022-10-28 06:15:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 980/2304 [I 2022-10-28 06:15:20] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:21 [I 2022-10-28 06:15:20] ...trap/engines/engine.py.229: process: 0.54442 | load: 0.00243 [I 2022-10-28 06:15:20] ...trap/engines/engine.py.230: loss: 0.14122 [I 2022-10-28 06:15:20] ...optimizers/trijoint.py.151: total_norm: 11.488379 [I 2022-10-28 06:15:25] ...trap/engines/engine.py.224: train: epoch 110 | batch 990/2304 [I 2022-10-28 06:15:25] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:13 [I 2022-10-28 06:15:25] ...trap/engines/engine.py.229: process: 0.56450 | load: 0.00247 [I 2022-10-28 06:15:25] ...trap/engines/engine.py.230: loss: 0.09966 [I 2022-10-28 06:15:25] ...optimizers/trijoint.py.151: total_norm: 11.715500 [I 2022-10-28 06:15:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 1000/2304 [I 2022-10-28 06:15:31] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:06 [I 2022-10-28 06:15:31] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00243 [I 2022-10-28 06:15:31] ...trap/engines/engine.py.230: loss: 0.14289 [I 2022-10-28 06:15:31] ...optimizers/trijoint.py.151: total_norm: 32.377117 [I 2022-10-28 06:15:37] ...trap/engines/engine.py.224: train: epoch 110 | batch 1010/2304 [I 2022-10-28 06:15:37] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:11:57 [I 2022-10-28 06:15:37] ...trap/engines/engine.py.229: process: 0.54717 | load: 0.00248 [I 2022-10-28 06:15:37] ...trap/engines/engine.py.230: loss: 0.10899 [I 2022-10-28 06:15:37] ...optimizers/trijoint.py.151: total_norm: 12.648313 [I 2022-10-28 06:15:42] ...trap/engines/engine.py.224: train: epoch 110 | batch 1020/2304 [I 2022-10-28 06:15:42] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:54 [I 2022-10-28 06:15:42] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00262 [I 2022-10-28 06:15:42] ...trap/engines/engine.py.230: loss: 0.12961 [I 2022-10-28 06:15:42] ...optimizers/trijoint.py.151: total_norm: 20.411715 [I 2022-10-28 06:15:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 1030/2304 [I 2022-10-28 06:15:48] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:56 [I 2022-10-28 06:15:48] ...trap/engines/engine.py.229: process: 0.56885 | load: 0.00241 [I 2022-10-28 06:15:48] ...trap/engines/engine.py.230: loss: 0.12005 [I 2022-10-28 06:15:48] ...optimizers/trijoint.py.151: total_norm: 15.643590 [I 2022-10-28 06:15:53] ...trap/engines/engine.py.224: train: epoch 110 | batch 1040/2304 [I 2022-10-28 06:15:53] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:49 [I 2022-10-28 06:15:53] ...trap/engines/engine.py.229: process: 0.56839 | load: 0.00248 [I 2022-10-28 06:15:53] ...trap/engines/engine.py.230: loss: 0.09279 [I 2022-10-28 06:15:53] ...optimizers/trijoint.py.151: total_norm: 13.232956 [I 2022-10-28 06:15:59] ...trap/engines/engine.py.224: train: epoch 110 | batch 1050/2304 [I 2022-10-28 06:15:59] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:37 [I 2022-10-28 06:15:59] ...trap/engines/engine.py.229: process: 0.55490 | load: 0.00246 [I 2022-10-28 06:15:59] ...trap/engines/engine.py.230: loss: 0.11887 [I 2022-10-28 06:15:59] ...optimizers/trijoint.py.151: total_norm: 10.405968 [I 2022-10-28 06:16:05] ...trap/engines/engine.py.224: train: epoch 110 | batch 1060/2304 [I 2022-10-28 06:16:05] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:28 [I 2022-10-28 06:16:05] ...trap/engines/engine.py.229: process: 0.55139 | load: 0.00243 [I 2022-10-28 06:16:05] ...trap/engines/engine.py.230: loss: 0.14621 [I 2022-10-28 06:16:05] ...optimizers/trijoint.py.151: total_norm: 38.947964 [I 2022-10-28 06:16:10] ...trap/engines/engine.py.224: train: epoch 110 | batch 1070/2304 [I 2022-10-28 06:16:10] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:26 [I 2022-10-28 06:16:10] ...trap/engines/engine.py.229: process: 0.54696 | load: 0.00245 [I 2022-10-28 06:16:10] ...trap/engines/engine.py.230: loss: 0.15174 [I 2022-10-28 06:16:10] ...optimizers/trijoint.py.151: total_norm: 15.416498 [I 2022-10-28 06:16:16] ...trap/engines/engine.py.224: train: epoch 110 | batch 1080/2304 [I 2022-10-28 06:16:16] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:18 [I 2022-10-28 06:16:16] ...trap/engines/engine.py.229: process: 0.56052 | load: 0.00243 [I 2022-10-28 06:16:16] ...trap/engines/engine.py.230: loss: 0.07216 [I 2022-10-28 06:16:16] ...optimizers/trijoint.py.151: total_norm: 13.096579 [I 2022-10-28 06:16:21] ...trap/engines/engine.py.224: train: epoch 110 | batch 1090/2304 [I 2022-10-28 06:16:21] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:18 [I 2022-10-28 06:16:21] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00243 [I 2022-10-28 06:16:21] ...trap/engines/engine.py.230: loss: 0.11519 [I 2022-10-28 06:16:21] ...optimizers/trijoint.py.151: total_norm: 19.899851 [I 2022-10-28 06:16:27] ...trap/engines/engine.py.224: train: epoch 110 | batch 1100/2304 [I 2022-10-28 06:16:27] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:10 [I 2022-10-28 06:16:27] ...trap/engines/engine.py.229: process: 0.53836 | load: 0.00245 [I 2022-10-28 06:16:27] ...trap/engines/engine.py.230: loss: 0.11042 [I 2022-10-28 06:16:27] ...optimizers/trijoint.py.151: total_norm: 7.840622 [I 2022-10-28 06:16:33] ...trap/engines/engine.py.224: train: epoch 110 | batch 1110/2304 [I 2022-10-28 06:16:33] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:09 [I 2022-10-28 06:16:33] ...trap/engines/engine.py.229: process: 0.57051 | load: 0.00271 [I 2022-10-28 06:16:33] ...trap/engines/engine.py.230: loss: 0.09400 [I 2022-10-28 06:16:33] ...optimizers/trijoint.py.151: total_norm: 14.947967 [I 2022-10-28 06:16:38] ...trap/engines/engine.py.224: train: epoch 110 | batch 1120/2304 [I 2022-10-28 06:16:38] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:02 [I 2022-10-28 06:16:38] ...trap/engines/engine.py.229: process: 0.55505 | load: 0.00248 [I 2022-10-28 06:16:38] ...trap/engines/engine.py.230: loss: 0.11798 [I 2022-10-28 06:16:38] ...optimizers/trijoint.py.151: total_norm: 14.167325 [I 2022-10-28 06:16:44] ...trap/engines/engine.py.224: train: epoch 110 | batch 1130/2304 [I 2022-10-28 06:16:44] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:11:01 [I 2022-10-28 06:16:44] ...trap/engines/engine.py.229: process: 0.57505 | load: 0.00246 [I 2022-10-28 06:16:44] ...trap/engines/engine.py.230: loss: 0.09098 [I 2022-10-28 06:16:44] ...optimizers/trijoint.py.151: total_norm: 9.875349 [I 2022-10-28 06:16:49] ...trap/engines/engine.py.224: train: epoch 110 | batch 1140/2304 [I 2022-10-28 06:16:49] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:11:04 [I 2022-10-28 06:16:49] ...trap/engines/engine.py.229: process: 0.55673 | load: 0.00245 [I 2022-10-28 06:16:49] ...trap/engines/engine.py.230: loss: 0.10814 [I 2022-10-28 06:16:49] ...optimizers/trijoint.py.151: total_norm: 12.448552 [I 2022-10-28 06:16:55] ...trap/engines/engine.py.224: train: epoch 110 | batch 1150/2304 [I 2022-10-28 06:16:55] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:51 [I 2022-10-28 06:16:55] ...trap/engines/engine.py.229: process: 0.57043 | load: 0.00242 [I 2022-10-28 06:16:55] ...trap/engines/engine.py.230: loss: 0.12804 [I 2022-10-28 06:16:55] ...optimizers/trijoint.py.151: total_norm: 10.406793 [I 2022-10-28 06:17:01] ...trap/engines/engine.py.224: train: epoch 110 | batch 1160/2304 [I 2022-10-28 06:17:01] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:44 [I 2022-10-28 06:17:01] ...trap/engines/engine.py.229: process: 0.56410 | load: 0.00247 [I 2022-10-28 06:17:01] ...trap/engines/engine.py.230: loss: 0.16806 [I 2022-10-28 06:17:01] ...optimizers/trijoint.py.151: total_norm: 12.192780 [I 2022-10-28 06:17:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 1170/2304 [I 2022-10-28 06:17:06] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:36 [I 2022-10-28 06:17:06] ...trap/engines/engine.py.229: process: 0.56286 | load: 0.00241 [I 2022-10-28 06:17:06] ...trap/engines/engine.py.230: loss: 0.12664 [I 2022-10-28 06:17:06] ...optimizers/trijoint.py.151: total_norm: 25.421326 [I 2022-10-28 06:17:12] ...trap/engines/engine.py.224: train: epoch 110 | batch 1180/2304 [I 2022-10-28 06:17:12] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:26 [I 2022-10-28 06:17:12] ...trap/engines/engine.py.229: process: 0.55282 | load: 0.00243 [I 2022-10-28 06:17:12] ...trap/engines/engine.py.230: loss: 0.08655 [I 2022-10-28 06:17:12] ...optimizers/trijoint.py.151: total_norm: 13.132270 [I 2022-10-28 06:17:18] ...trap/engines/engine.py.224: train: epoch 110 | batch 1190/2304 [I 2022-10-28 06:17:18] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:28 [I 2022-10-28 06:17:18] ...trap/engines/engine.py.229: process: 0.57562 | load: 0.00240 [I 2022-10-28 06:17:18] ...trap/engines/engine.py.230: loss: 0.15659 [I 2022-10-28 06:17:18] ...optimizers/trijoint.py.151: total_norm: 16.252268 [I 2022-10-28 06:17:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 1200/2304 [I 2022-10-28 06:17:23] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:19 [I 2022-10-28 06:17:23] ...trap/engines/engine.py.229: process: 0.55356 | load: 0.00245 [I 2022-10-28 06:17:23] ...trap/engines/engine.py.230: loss: 0.08953 [I 2022-10-28 06:17:23] ...optimizers/trijoint.py.151: total_norm: 17.257820 [I 2022-10-28 06:17:29] ...trap/engines/engine.py.224: train: epoch 110 | batch 1210/2304 [I 2022-10-28 06:17:29] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:19 [I 2022-10-28 06:17:29] ...trap/engines/engine.py.229: process: 0.57785 | load: 0.00243 [I 2022-10-28 06:17:29] ...trap/engines/engine.py.230: loss: 0.07702 [I 2022-10-28 06:17:29] ...optimizers/trijoint.py.151: total_norm: 12.502783 [I 2022-10-28 06:17:35] ...trap/engines/engine.py.224: train: epoch 110 | batch 1220/2304 [I 2022-10-28 06:17:35] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:07 [I 2022-10-28 06:17:35] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00243 [I 2022-10-28 06:17:35] ...trap/engines/engine.py.230: loss: 0.16790 [I 2022-10-28 06:17:35] ...optimizers/trijoint.py.151: total_norm: 17.236807 [I 2022-10-28 06:17:40] ...trap/engines/engine.py.224: train: epoch 110 | batch 1230/2304 [I 2022-10-28 06:17:40] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:10:17 [I 2022-10-28 06:17:40] ...trap/engines/engine.py.229: process: 0.58387 | load: 0.00240 [I 2022-10-28 06:17:40] ...trap/engines/engine.py.230: loss: 0.14031 [I 2022-10-28 06:17:40] ...optimizers/trijoint.py.151: total_norm: 27.655531 [I 2022-10-28 06:17:46] ...trap/engines/engine.py.224: train: epoch 110 | batch 1240/2304 [I 2022-10-28 06:17:46] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:55 [I 2022-10-28 06:17:46] ...trap/engines/engine.py.229: process: 0.56489 | load: 0.00241 [I 2022-10-28 06:17:46] ...trap/engines/engine.py.230: loss: 0.11461 [I 2022-10-28 06:17:46] ...optimizers/trijoint.py.151: total_norm: 13.153907 [I 2022-10-28 06:17:51] ...trap/engines/engine.py.224: train: epoch 110 | batch 1250/2304 [I 2022-10-28 06:17:51] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:50 [I 2022-10-28 06:17:51] ...trap/engines/engine.py.229: process: 0.56777 | load: 0.00255 [I 2022-10-28 06:17:51] ...trap/engines/engine.py.230: loss: 0.12370 [I 2022-10-28 06:17:51] ...optimizers/trijoint.py.151: total_norm: 12.289129 [I 2022-10-28 06:17:57] ...trap/engines/engine.py.224: train: epoch 110 | batch 1260/2304 [I 2022-10-28 06:17:57] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:43 [I 2022-10-28 06:17:57] ...trap/engines/engine.py.229: process: 0.56419 | load: 0.00252 [I 2022-10-28 06:17:57] ...trap/engines/engine.py.230: loss: 0.12813 [I 2022-10-28 06:17:57] ...optimizers/trijoint.py.151: total_norm: 10.393942 [I 2022-10-28 06:18:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 1270/2304 [I 2022-10-28 06:18:03] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:32 [I 2022-10-28 06:18:03] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00251 [I 2022-10-28 06:18:03] ...trap/engines/engine.py.230: loss: 0.14702 [I 2022-10-28 06:18:03] ...optimizers/trijoint.py.151: total_norm: 15.532070 [I 2022-10-28 06:18:08] ...trap/engines/engine.py.224: train: epoch 110 | batch 1280/2304 [I 2022-10-28 06:18:08] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:29 [I 2022-10-28 06:18:08] ...trap/engines/engine.py.229: process: 0.55661 | load: 0.00248 [I 2022-10-28 06:18:08] ...trap/engines/engine.py.230: loss: 0.12704 [I 2022-10-28 06:18:08] ...optimizers/trijoint.py.151: total_norm: 24.410168 [I 2022-10-28 06:18:14] ...trap/engines/engine.py.224: train: epoch 110 | batch 1290/2304 [I 2022-10-28 06:18:14] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:24 [I 2022-10-28 06:18:14] ...trap/engines/engine.py.229: process: 0.56637 | load: 0.00243 [I 2022-10-28 06:18:14] ...trap/engines/engine.py.230: loss: 0.08382 [I 2022-10-28 06:18:14] ...optimizers/trijoint.py.151: total_norm: 10.004290 [I 2022-10-28 06:18:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 1300/2304 [I 2022-10-28 06:18:20] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:30 [I 2022-10-28 06:18:20] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00251 [I 2022-10-28 06:18:20] ...trap/engines/engine.py.230: loss: 0.16497 [I 2022-10-28 06:18:20] ...optimizers/trijoint.py.151: total_norm: 33.113148 [I 2022-10-28 06:18:25] ...trap/engines/engine.py.224: train: epoch 110 | batch 1310/2304 [I 2022-10-28 06:18:25] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:14 [I 2022-10-28 06:18:25] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00242 [I 2022-10-28 06:18:25] ...trap/engines/engine.py.230: loss: 0.10881 [I 2022-10-28 06:18:25] ...optimizers/trijoint.py.151: total_norm: 12.900558 [I 2022-10-28 06:18:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 1320/2304 [I 2022-10-28 06:18:31] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:11 [I 2022-10-28 06:18:31] ...trap/engines/engine.py.229: process: 0.56040 | load: 0.00244 [I 2022-10-28 06:18:31] ...trap/engines/engine.py.230: loss: 0.11720 [I 2022-10-28 06:18:31] ...optimizers/trijoint.py.151: total_norm: 16.716885 [I 2022-10-28 06:18:36] ...trap/engines/engine.py.224: train: epoch 110 | batch 1330/2304 [I 2022-10-28 06:18:36] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:02 [I 2022-10-28 06:18:36] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00249 [I 2022-10-28 06:18:36] ...trap/engines/engine.py.230: loss: 0.11862 [I 2022-10-28 06:18:36] ...optimizers/trijoint.py.151: total_norm: 11.016759 [I 2022-10-28 06:18:42] ...trap/engines/engine.py.224: train: epoch 110 | batch 1340/2304 [I 2022-10-28 06:18:42] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:08:55 [I 2022-10-28 06:18:42] ...trap/engines/engine.py.229: process: 0.54649 | load: 0.00254 [I 2022-10-28 06:18:42] ...trap/engines/engine.py.230: loss: 0.17074 [I 2022-10-28 06:18:42] ...optimizers/trijoint.py.151: total_norm: 11.016300 [I 2022-10-28 06:18:47] ...trap/engines/engine.py.224: train: epoch 110 | batch 1350/2304 [I 2022-10-28 06:18:47] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:51 [I 2022-10-28 06:18:47] ...trap/engines/engine.py.229: process: 0.56101 | load: 0.00248 [I 2022-10-28 06:18:47] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-28 06:18:47] ...optimizers/trijoint.py.151: total_norm: 16.764933 [I 2022-10-28 06:18:53] ...trap/engines/engine.py.224: train: epoch 110 | batch 1360/2304 [I 2022-10-28 06:18:53] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:43 [I 2022-10-28 06:18:53] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00250 [I 2022-10-28 06:18:53] ...trap/engines/engine.py.230: loss: 0.08720 [I 2022-10-28 06:18:53] ...optimizers/trijoint.py.151: total_norm: 8.010930 [I 2022-10-28 06:18:59] ...trap/engines/engine.py.224: train: epoch 110 | batch 1370/2304 [I 2022-10-28 06:18:59] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:39 [I 2022-10-28 06:18:59] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00244 [I 2022-10-28 06:18:59] ...trap/engines/engine.py.230: loss: 0.16437 [I 2022-10-28 06:18:59] ...optimizers/trijoint.py.151: total_norm: 24.173023 [I 2022-10-28 06:19:04] ...trap/engines/engine.py.224: train: epoch 110 | batch 1380/2304 [I 2022-10-28 06:19:04] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:36 [I 2022-10-28 06:19:04] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00245 [I 2022-10-28 06:19:04] ...trap/engines/engine.py.230: loss: 0.08563 [I 2022-10-28 06:19:04] ...optimizers/trijoint.py.151: total_norm: 9.844357 [I 2022-10-28 06:19:10] ...trap/engines/engine.py.224: train: epoch 110 | batch 1390/2304 [I 2022-10-28 06:19:10] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:29 [I 2022-10-28 06:19:10] ...trap/engines/engine.py.229: process: 0.55758 | load: 0.00240 [I 2022-10-28 06:19:10] ...trap/engines/engine.py.230: loss: 0.24587 [I 2022-10-28 06:19:10] ...optimizers/trijoint.py.151: total_norm: 66.220078 [I 2022-10-28 06:19:15] ...trap/engines/engine.py.224: train: epoch 110 | batch 1400/2304 [I 2022-10-28 06:19:15] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:25 [I 2022-10-28 06:19:15] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00245 [I 2022-10-28 06:19:15] ...trap/engines/engine.py.230: loss: 0.17204 [I 2022-10-28 06:19:15] ...optimizers/trijoint.py.151: total_norm: 15.186612 [I 2022-10-28 06:19:21] ...trap/engines/engine.py.224: train: epoch 110 | batch 1410/2304 [I 2022-10-28 06:19:21] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:23 [I 2022-10-28 06:19:21] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00289 [I 2022-10-28 06:19:21] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-28 06:19:21] ...optimizers/trijoint.py.151: total_norm: 15.989870 [I 2022-10-28 06:19:27] ...trap/engines/engine.py.224: train: epoch 110 | batch 1420/2304 [I 2022-10-28 06:19:27] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:12 [I 2022-10-28 06:19:27] ...trap/engines/engine.py.229: process: 0.55097 | load: 0.00263 [I 2022-10-28 06:19:27] ...trap/engines/engine.py.230: loss: 0.19759 [I 2022-10-28 06:19:27] ...optimizers/trijoint.py.151: total_norm: 27.656437 [I 2022-10-28 06:19:32] ...trap/engines/engine.py.224: train: epoch 110 | batch 1430/2304 [I 2022-10-28 06:19:32] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:06 [I 2022-10-28 06:19:32] ...trap/engines/engine.py.229: process: 0.55970 | load: 0.00248 [I 2022-10-28 06:19:32] ...trap/engines/engine.py.230: loss: 0.15922 [I 2022-10-28 06:19:32] ...optimizers/trijoint.py.151: total_norm: 21.990168 [I 2022-10-28 06:19:38] ...trap/engines/engine.py.224: train: epoch 110 | batch 1440/2304 [I 2022-10-28 06:19:38] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:08:01 [I 2022-10-28 06:19:38] ...trap/engines/engine.py.229: process: 0.55097 | load: 0.00252 [I 2022-10-28 06:19:38] ...trap/engines/engine.py.230: loss: 0.10897 [I 2022-10-28 06:19:38] ...optimizers/trijoint.py.151: total_norm: 10.456488 [I 2022-10-28 06:19:43] ...trap/engines/engine.py.224: train: epoch 110 | batch 1450/2304 [I 2022-10-28 06:19:43] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:07:55 [I 2022-10-28 06:19:43] ...trap/engines/engine.py.229: process: 0.54667 | load: 0.00248 [I 2022-10-28 06:19:43] ...trap/engines/engine.py.230: loss: 0.10382 [I 2022-10-28 06:19:43] ...optimizers/trijoint.py.151: total_norm: 6.862474 [I 2022-10-28 06:19:49] ...trap/engines/engine.py.224: train: epoch 110 | batch 1460/2304 [I 2022-10-28 06:19:49] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:50 [I 2022-10-28 06:19:49] ...trap/engines/engine.py.229: process: 0.55209 | load: 0.00245 [I 2022-10-28 06:19:49] ...trap/engines/engine.py.230: loss: 0.12539 [I 2022-10-28 06:19:49] ...optimizers/trijoint.py.151: total_norm: 10.275345 [I 2022-10-28 06:19:55] ...trap/engines/engine.py.224: train: epoch 110 | batch 1470/2304 [I 2022-10-28 06:19:55] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:43 [I 2022-10-28 06:19:55] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00245 [I 2022-10-28 06:19:55] ...trap/engines/engine.py.230: loss: 0.10952 [I 2022-10-28 06:19:55] ...optimizers/trijoint.py.151: total_norm: 10.202328 [I 2022-10-28 06:20:00] ...trap/engines/engine.py.224: train: epoch 110 | batch 1480/2304 [I 2022-10-28 06:20:00] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:40 [I 2022-10-28 06:20:00] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00245 [I 2022-10-28 06:20:00] ...trap/engines/engine.py.230: loss: 0.08498 [I 2022-10-28 06:20:00] ...optimizers/trijoint.py.151: total_norm: 25.929253 [I 2022-10-28 06:20:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 1490/2304 [I 2022-10-28 06:20:06] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:33 [I 2022-10-28 06:20:06] ...trap/engines/engine.py.229: process: 0.56280 | load: 0.00244 [I 2022-10-28 06:20:06] ...trap/engines/engine.py.230: loss: 0.10139 [I 2022-10-28 06:20:06] ...optimizers/trijoint.py.151: total_norm: 11.682089 [I 2022-10-28 06:20:11] ...trap/engines/engine.py.224: train: epoch 110 | batch 1500/2304 [I 2022-10-28 06:20:11] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:27 [I 2022-10-28 06:20:11] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00247 [I 2022-10-28 06:20:11] ...trap/engines/engine.py.230: loss: 0.11912 [I 2022-10-28 06:20:11] ...optimizers/trijoint.py.151: total_norm: 28.644909 [I 2022-10-28 06:20:17] ...trap/engines/engine.py.224: train: epoch 110 | batch 1510/2304 [I 2022-10-28 06:20:17] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:19 [I 2022-10-28 06:20:17] ...trap/engines/engine.py.229: process: 0.53873 | load: 0.00248 [I 2022-10-28 06:20:17] ...trap/engines/engine.py.230: loss: 0.18782 [I 2022-10-28 06:20:17] ...optimizers/trijoint.py.151: total_norm: 38.553555 [I 2022-10-28 06:20:23] ...trap/engines/engine.py.224: train: epoch 110 | batch 1520/2304 [I 2022-10-28 06:20:23] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:20 [I 2022-10-28 06:20:23] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00249 [I 2022-10-28 06:20:23] ...trap/engines/engine.py.230: loss: 0.10681 [I 2022-10-28 06:20:23] ...optimizers/trijoint.py.151: total_norm: 25.526278 [I 2022-10-28 06:20:28] ...trap/engines/engine.py.224: train: epoch 110 | batch 1530/2304 [I 2022-10-28 06:20:28] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:19 [I 2022-10-28 06:20:28] ...trap/engines/engine.py.229: process: 0.55817 | load: 0.00251 [I 2022-10-28 06:20:28] ...trap/engines/engine.py.230: loss: 0.09913 [I 2022-10-28 06:20:28] ...optimizers/trijoint.py.151: total_norm: 18.617292 [I 2022-10-28 06:20:34] ...trap/engines/engine.py.224: train: epoch 110 | batch 1540/2304 [I 2022-10-28 06:20:34] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:05 [I 2022-10-28 06:20:34] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00247 [I 2022-10-28 06:20:34] ...trap/engines/engine.py.230: loss: 0.10756 [I 2022-10-28 06:20:34] ...optimizers/trijoint.py.151: total_norm: 17.026642 [I 2022-10-28 06:20:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 1550/2304 [I 2022-10-28 06:20:39] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:01 [I 2022-10-28 06:20:39] ...trap/engines/engine.py.229: process: 0.56763 | load: 0.00248 [I 2022-10-28 06:20:39] ...trap/engines/engine.py.230: loss: 0.08368 [I 2022-10-28 06:20:39] ...optimizers/trijoint.py.151: total_norm: 9.956079 [I 2022-10-28 06:20:45] ...trap/engines/engine.py.224: train: epoch 110 | batch 1560/2304 [I 2022-10-28 06:20:45] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:54 [I 2022-10-28 06:20:45] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00243 [I 2022-10-28 06:20:45] ...trap/engines/engine.py.230: loss: 0.12514 [I 2022-10-28 06:20:45] ...optimizers/trijoint.py.151: total_norm: 14.712019 [I 2022-10-28 06:20:51] ...trap/engines/engine.py.224: train: epoch 110 | batch 1570/2304 [I 2022-10-28 06:20:51] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:48 [I 2022-10-28 06:20:51] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00242 [I 2022-10-28 06:20:51] ...trap/engines/engine.py.230: loss: 0.07319 [I 2022-10-28 06:20:51] ...optimizers/trijoint.py.151: total_norm: 11.471569 [I 2022-10-28 06:20:56] ...trap/engines/engine.py.224: train: epoch 110 | batch 1580/2304 [I 2022-10-28 06:20:56] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:40 [I 2022-10-28 06:20:56] ...trap/engines/engine.py.229: process: 0.54644 | load: 0.00243 [I 2022-10-28 06:20:56] ...trap/engines/engine.py.230: loss: 0.12118 [I 2022-10-28 06:20:56] ...optimizers/trijoint.py.151: total_norm: 61.881535 [I 2022-10-28 06:21:02] ...trap/engines/engine.py.224: train: epoch 110 | batch 1590/2304 [I 2022-10-28 06:21:02] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:37 [I 2022-10-28 06:21:02] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00242 [I 2022-10-28 06:21:02] ...trap/engines/engine.py.230: loss: 0.08347 [I 2022-10-28 06:21:02] ...optimizers/trijoint.py.151: total_norm: 9.057000 [I 2022-10-28 06:21:07] ...trap/engines/engine.py.224: train: epoch 110 | batch 1600/2304 [I 2022-10-28 06:21:07] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:32 [I 2022-10-28 06:21:07] ...trap/engines/engine.py.229: process: 0.54540 | load: 0.00240 [I 2022-10-28 06:21:07] ...trap/engines/engine.py.230: loss: 0.11691 [I 2022-10-28 06:21:07] ...optimizers/trijoint.py.151: total_norm: 11.031464 [I 2022-10-28 06:21:13] ...trap/engines/engine.py.224: train: epoch 110 | batch 1610/2304 [I 2022-10-28 06:21:13] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:24 [I 2022-10-28 06:21:13] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00239 [I 2022-10-28 06:21:13] ...trap/engines/engine.py.230: loss: 0.14055 [I 2022-10-28 06:21:13] ...optimizers/trijoint.py.151: total_norm: 13.993772 [I 2022-10-28 06:21:19] ...trap/engines/engine.py.224: train: epoch 110 | batch 1620/2304 [I 2022-10-28 06:21:19] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:22 [I 2022-10-28 06:21:19] ...trap/engines/engine.py.229: process: 0.56093 | load: 0.00244 [I 2022-10-28 06:21:19] ...trap/engines/engine.py.230: loss: 0.10815 [I 2022-10-28 06:21:19] ...optimizers/trijoint.py.151: total_norm: 36.786953 [I 2022-10-28 06:21:24] ...trap/engines/engine.py.224: train: epoch 110 | batch 1630/2304 [I 2022-10-28 06:21:24] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:16 [I 2022-10-28 06:21:24] ...trap/engines/engine.py.229: process: 0.56816 | load: 0.00243 [I 2022-10-28 06:21:24] ...trap/engines/engine.py.230: loss: 0.09563 [I 2022-10-28 06:21:24] ...optimizers/trijoint.py.151: total_norm: 10.272610 [I 2022-10-28 06:21:30] ...trap/engines/engine.py.224: train: epoch 110 | batch 1640/2304 [I 2022-10-28 06:21:30] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:16 [I 2022-10-28 06:21:30] ...trap/engines/engine.py.229: process: 0.56302 | load: 0.00250 [I 2022-10-28 06:21:30] ...trap/engines/engine.py.230: loss: 0.15857 [I 2022-10-28 06:21:30] ...optimizers/trijoint.py.151: total_norm: 11.788324 [I 2022-10-28 06:21:35] ...trap/engines/engine.py.224: train: epoch 110 | batch 1650/2304 [I 2022-10-28 06:21:35] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:03 [I 2022-10-28 06:21:35] ...trap/engines/engine.py.229: process: 0.54878 | load: 0.00246 [I 2022-10-28 06:21:35] ...trap/engines/engine.py.230: loss: 0.14446 [I 2022-10-28 06:21:35] ...optimizers/trijoint.py.151: total_norm: 10.168249 [I 2022-10-28 06:21:41] ...trap/engines/engine.py.224: train: epoch 110 | batch 1660/2304 [I 2022-10-28 06:21:41] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:05:58 [I 2022-10-28 06:21:41] ...trap/engines/engine.py.229: process: 0.54556 | load: 0.00244 [I 2022-10-28 06:21:41] ...trap/engines/engine.py.230: loss: 0.08222 [I 2022-10-28 06:21:41] ...optimizers/trijoint.py.151: total_norm: 15.594035 [I 2022-10-28 06:21:47] ...trap/engines/engine.py.224: train: epoch 110 | batch 1670/2304 [I 2022-10-28 06:21:47] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:05:54 [I 2022-10-28 06:21:47] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00246 [I 2022-10-28 06:21:47] ...trap/engines/engine.py.230: loss: 0.09034 [I 2022-10-28 06:21:47] ...optimizers/trijoint.py.151: total_norm: 14.508858 [I 2022-10-28 06:21:52] ...trap/engines/engine.py.224: train: epoch 110 | batch 1680/2304 [I 2022-10-28 06:21:52] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:48 [I 2022-10-28 06:21:52] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00246 [I 2022-10-28 06:21:52] ...trap/engines/engine.py.230: loss: 0.10562 [I 2022-10-28 06:21:52] ...optimizers/trijoint.py.151: total_norm: 9.256121 [I 2022-10-28 06:21:58] ...trap/engines/engine.py.224: train: epoch 110 | batch 1690/2304 [I 2022-10-28 06:21:58] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:43 [I 2022-10-28 06:21:58] ...trap/engines/engine.py.229: process: 0.56413 | load: 0.00247 [I 2022-10-28 06:21:58] ...trap/engines/engine.py.230: loss: 0.09569 [I 2022-10-28 06:21:58] ...optimizers/trijoint.py.151: total_norm: 22.049219 [I 2022-10-28 06:22:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 1700/2304 [I 2022-10-28 06:22:03] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:35 [I 2022-10-28 06:22:03] ...trap/engines/engine.py.229: process: 0.55394 | load: 0.00246 [I 2022-10-28 06:22:03] ...trap/engines/engine.py.230: loss: 0.12052 [I 2022-10-28 06:22:03] ...optimizers/trijoint.py.151: total_norm: 17.783207 [I 2022-10-28 06:22:09] ...trap/engines/engine.py.224: train: epoch 110 | batch 1710/2304 [I 2022-10-28 06:22:09] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:29 [I 2022-10-28 06:22:09] ...trap/engines/engine.py.229: process: 0.54635 | load: 0.00245 [I 2022-10-28 06:22:09] ...trap/engines/engine.py.230: loss: 0.18391 [I 2022-10-28 06:22:09] ...optimizers/trijoint.py.151: total_norm: 15.859725 [I 2022-10-28 06:22:15] ...trap/engines/engine.py.224: train: epoch 110 | batch 1720/2304 [I 2022-10-28 06:22:15] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:23 [I 2022-10-28 06:22:15] ...trap/engines/engine.py.229: process: 0.54416 | load: 0.00247 [I 2022-10-28 06:22:15] ...trap/engines/engine.py.230: loss: 0.07734 [I 2022-10-28 06:22:15] ...optimizers/trijoint.py.151: total_norm: 7.346941 [I 2022-10-28 06:22:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 1730/2304 [I 2022-10-28 06:22:20] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:23 [I 2022-10-28 06:22:20] ...trap/engines/engine.py.229: process: 0.57792 | load: 0.00247 [I 2022-10-28 06:22:20] ...trap/engines/engine.py.230: loss: 0.21877 [I 2022-10-28 06:22:20] ...optimizers/trijoint.py.151: total_norm: 28.312057 [I 2022-10-28 06:22:26] ...trap/engines/engine.py.224: train: epoch 110 | batch 1740/2304 [I 2022-10-28 06:22:26] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:15 [I 2022-10-28 06:22:26] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00247 [I 2022-10-28 06:22:26] ...trap/engines/engine.py.230: loss: 0.09152 [I 2022-10-28 06:22:26] ...optimizers/trijoint.py.151: total_norm: 8.503531 [I 2022-10-28 06:22:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 1750/2304 [I 2022-10-28 06:22:31] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:06 [I 2022-10-28 06:22:31] ...trap/engines/engine.py.229: process: 0.55913 | load: 0.00246 [I 2022-10-28 06:22:31] ...trap/engines/engine.py.230: loss: 0.12917 [I 2022-10-28 06:22:31] ...optimizers/trijoint.py.151: total_norm: 43.833248 [I 2022-10-28 06:22:37] ...trap/engines/engine.py.224: train: epoch 110 | batch 1760/2304 [I 2022-10-28 06:22:37] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:02 [I 2022-10-28 06:22:37] ...trap/engines/engine.py.229: process: 0.55684 | load: 0.00245 [I 2022-10-28 06:22:37] ...trap/engines/engine.py.230: loss: 0.16556 [I 2022-10-28 06:22:37] ...optimizers/trijoint.py.151: total_norm: 16.016171 [I 2022-10-28 06:22:42] ...trap/engines/engine.py.224: train: epoch 110 | batch 1770/2304 [I 2022-10-28 06:22:42] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:04:54 [I 2022-10-28 06:22:42] ...trap/engines/engine.py.229: process: 0.55010 | load: 0.00244 [I 2022-10-28 06:22:42] ...trap/engines/engine.py.230: loss: 0.09005 [I 2022-10-28 06:22:42] ...optimizers/trijoint.py.151: total_norm: 12.099874 [I 2022-10-28 06:22:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 1780/2304 [I 2022-10-28 06:22:48] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:52 [I 2022-10-28 06:22:48] ...trap/engines/engine.py.229: process: 0.56270 | load: 0.00245 [I 2022-10-28 06:22:48] ...trap/engines/engine.py.230: loss: 0.10279 [I 2022-10-28 06:22:48] ...optimizers/trijoint.py.151: total_norm: 12.686303 [I 2022-10-28 06:22:54] ...trap/engines/engine.py.224: train: epoch 110 | batch 1790/2304 [I 2022-10-28 06:22:54] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:48 [I 2022-10-28 06:22:54] ...trap/engines/engine.py.229: process: 0.55799 | load: 0.00242 [I 2022-10-28 06:22:54] ...trap/engines/engine.py.230: loss: 0.11148 [I 2022-10-28 06:22:54] ...optimizers/trijoint.py.151: total_norm: 12.125735 [I 2022-10-28 06:22:59] ...trap/engines/engine.py.224: train: epoch 110 | batch 1800/2304 [I 2022-10-28 06:22:59] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:40 [I 2022-10-28 06:22:59] ...trap/engines/engine.py.229: process: 0.54785 | load: 0.00243 [I 2022-10-28 06:22:59] ...trap/engines/engine.py.230: loss: 0.12406 [I 2022-10-28 06:22:59] ...optimizers/trijoint.py.151: total_norm: 32.378284 [I 2022-10-28 06:23:05] ...trap/engines/engine.py.224: train: epoch 110 | batch 1810/2304 [I 2022-10-28 06:23:05] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:35 [I 2022-10-28 06:23:05] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00242 [I 2022-10-28 06:23:05] ...trap/engines/engine.py.230: loss: 0.13050 [I 2022-10-28 06:23:05] ...optimizers/trijoint.py.151: total_norm: 24.179050 [I 2022-10-28 06:23:10] ...trap/engines/engine.py.224: train: epoch 110 | batch 1820/2304 [I 2022-10-28 06:23:10] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:27 [I 2022-10-28 06:23:10] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00247 [I 2022-10-28 06:23:10] ...trap/engines/engine.py.230: loss: 0.16741 [I 2022-10-28 06:23:10] ...optimizers/trijoint.py.151: total_norm: 25.342184 [I 2022-10-28 06:23:16] ...trap/engines/engine.py.224: train: epoch 110 | batch 1830/2304 [I 2022-10-28 06:23:16] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:24 [I 2022-10-28 06:23:16] ...trap/engines/engine.py.229: process: 0.56366 | load: 0.00244 [I 2022-10-28 06:23:16] ...trap/engines/engine.py.230: loss: 0.12258 [I 2022-10-28 06:23:16] ...optimizers/trijoint.py.151: total_norm: 18.946907 [I 2022-10-28 06:23:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 1840/2304 [I 2022-10-28 06:23:22] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:17 [I 2022-10-28 06:23:22] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00253 [I 2022-10-28 06:23:22] ...trap/engines/engine.py.230: loss: 0.10008 [I 2022-10-28 06:23:22] ...optimizers/trijoint.py.151: total_norm: 14.263455 [I 2022-10-28 06:23:27] ...trap/engines/engine.py.224: train: epoch 110 | batch 1850/2304 [I 2022-10-28 06:23:27] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:13 [I 2022-10-28 06:23:27] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00249 [I 2022-10-28 06:23:27] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-28 06:23:27] ...optimizers/trijoint.py.151: total_norm: 8.812881 [I 2022-10-28 06:23:33] ...trap/engines/engine.py.224: train: epoch 110 | batch 1860/2304 [I 2022-10-28 06:23:33] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:07 [I 2022-10-28 06:23:33] ...trap/engines/engine.py.229: process: 0.54576 | load: 0.00241 [I 2022-10-28 06:23:33] ...trap/engines/engine.py.230: loss: 0.11492 [I 2022-10-28 06:23:33] ...optimizers/trijoint.py.151: total_norm: 34.443428 [I 2022-10-28 06:23:38] ...trap/engines/engine.py.224: train: epoch 110 | batch 1870/2304 [I 2022-10-28 06:23:38] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:02 [I 2022-10-28 06:23:38] ...trap/engines/engine.py.229: process: 0.56425 | load: 0.00243 [I 2022-10-28 06:23:38] ...trap/engines/engine.py.230: loss: 0.11142 [I 2022-10-28 06:23:38] ...optimizers/trijoint.py.151: total_norm: 18.825489 [I 2022-10-28 06:24:03] ...trap/engines/engine.py.224: train: epoch 110 | batch 1880/2304 [I 2022-10-28 06:24:03] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:09:29 [I 2022-10-28 06:24:03] ...trap/engines/engine.py.229: process: 0.55017 | load: 0.00237 [I 2022-10-28 06:24:03] ...trap/engines/engine.py.230: loss: 0.16943 [I 2022-10-28 06:24:03] ...optimizers/trijoint.py.151: total_norm: 34.864311 [I 2022-10-28 06:24:08] ...trap/engines/engine.py.224: train: epoch 110 | batch 1890/2304 [I 2022-10-28 06:24:08] ...trap/engines/engine.py.228: elapsed: 0:17:59 | left: 0:04:25 [I 2022-10-28 06:24:08] ...trap/engines/engine.py.229: process: 0.56600 | load: 0.00238 [I 2022-10-28 06:24:08] ...trap/engines/engine.py.230: loss: 0.14824 [I 2022-10-28 06:24:08] ...optimizers/trijoint.py.151: total_norm: 34.815449 [I 2022-10-28 06:24:14] ...trap/engines/engine.py.224: train: epoch 110 | batch 1900/2304 [I 2022-10-28 06:24:14] ...trap/engines/engine.py.228: elapsed: 0:18:05 | left: 0:03:48 [I 2022-10-28 06:24:14] ...trap/engines/engine.py.229: process: 0.54805 | load: 0.00238 [I 2022-10-28 06:24:14] ...trap/engines/engine.py.230: loss: 0.11543 [I 2022-10-28 06:24:14] ...optimizers/trijoint.py.151: total_norm: 15.954026 [I 2022-10-28 06:24:20] ...trap/engines/engine.py.224: train: epoch 110 | batch 1910/2304 [I 2022-10-28 06:24:20] ...trap/engines/engine.py.228: elapsed: 0:18:10 | left: 0:03:41 [I 2022-10-28 06:24:20] ...trap/engines/engine.py.229: process: 0.57122 | load: 0.00244 [I 2022-10-28 06:24:20] ...trap/engines/engine.py.230: loss: 0.16088 [I 2022-10-28 06:24:20] ...optimizers/trijoint.py.151: total_norm: 27.156454 [I 2022-10-28 06:24:25] ...trap/engines/engine.py.224: train: epoch 110 | batch 1920/2304 [I 2022-10-28 06:24:25] ...trap/engines/engine.py.228: elapsed: 0:18:16 | left: 0:03:36 [I 2022-10-28 06:24:25] ...trap/engines/engine.py.229: process: 0.57280 | load: 0.00243 [I 2022-10-28 06:24:25] ...trap/engines/engine.py.230: loss: 0.10563 [I 2022-10-28 06:24:25] ...optimizers/trijoint.py.151: total_norm: 18.531443 [I 2022-10-28 06:24:31] ...trap/engines/engine.py.224: train: epoch 110 | batch 1930/2304 [I 2022-10-28 06:24:31] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:27 [I 2022-10-28 06:24:31] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00238 [I 2022-10-28 06:24:31] ...trap/engines/engine.py.230: loss: 0.19467 [I 2022-10-28 06:24:31] ...optimizers/trijoint.py.151: total_norm: 17.013067 [I 2022-10-28 06:24:36] ...trap/engines/engine.py.224: train: epoch 110 | batch 1940/2304 [I 2022-10-28 06:24:36] ...trap/engines/engine.py.228: elapsed: 0:18:27 | left: 0:03:22 [I 2022-10-28 06:24:36] ...trap/engines/engine.py.229: process: 0.56349 | load: 0.00240 [I 2022-10-28 06:24:36] ...trap/engines/engine.py.230: loss: 0.09361 [I 2022-10-28 06:24:36] ...optimizers/trijoint.py.151: total_norm: 11.143365 [I 2022-10-28 06:24:42] ...trap/engines/engine.py.224: train: epoch 110 | batch 1950/2304 [I 2022-10-28 06:24:42] ...trap/engines/engine.py.228: elapsed: 0:18:33 | left: 0:03:18 [I 2022-10-28 06:24:42] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00239 [I 2022-10-28 06:24:42] ...trap/engines/engine.py.230: loss: 0.11539 [I 2022-10-28 06:24:42] ...optimizers/trijoint.py.151: total_norm: 19.848286 [I 2022-10-28 06:24:48] ...trap/engines/engine.py.224: train: epoch 110 | batch 1960/2304 [I 2022-10-28 06:24:48] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:03:12 [I 2022-10-28 06:24:48] ...trap/engines/engine.py.229: process: 0.55694 | load: 0.00241 [I 2022-10-28 06:24:48] ...trap/engines/engine.py.230: loss: 0.09319 [I 2022-10-28 06:24:48] ...optimizers/trijoint.py.151: total_norm: 8.619608 [I 2022-10-28 06:24:53] ...trap/engines/engine.py.224: train: epoch 110 | batch 1970/2304 [I 2022-10-28 06:24:53] ...trap/engines/engine.py.228: elapsed: 0:18:44 | left: 0:03:05 [I 2022-10-28 06:24:53] ...trap/engines/engine.py.229: process: 0.55046 | load: 0.00237 [I 2022-10-28 06:24:53] ...trap/engines/engine.py.230: loss: 0.09989 [I 2022-10-28 06:24:53] ...optimizers/trijoint.py.151: total_norm: 10.558597 [I 2022-10-28 06:24:59] ...trap/engines/engine.py.224: train: epoch 110 | batch 1980/2304 [I 2022-10-28 06:24:59] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:03:00 [I 2022-10-28 06:24:59] ...trap/engines/engine.py.229: process: 0.54847 | load: 0.00238 [I 2022-10-28 06:24:59] ...trap/engines/engine.py.230: loss: 0.13928 [I 2022-10-28 06:24:59] ...optimizers/trijoint.py.151: total_norm: 20.071842 [I 2022-10-28 06:25:04] ...trap/engines/engine.py.224: train: epoch 110 | batch 1990/2304 [I 2022-10-28 06:25:04] ...trap/engines/engine.py.228: elapsed: 0:18:55 | left: 0:02:54 [I 2022-10-28 06:25:04] ...trap/engines/engine.py.229: process: 0.55298 | load: 0.00237 [I 2022-10-28 06:25:04] ...trap/engines/engine.py.230: loss: 0.16115 [I 2022-10-28 06:25:04] ...optimizers/trijoint.py.151: total_norm: 17.120567 [I 2022-10-28 06:25:10] ...trap/engines/engine.py.224: train: epoch 110 | batch 2000/2304 [I 2022-10-28 06:25:10] ...trap/engines/engine.py.228: elapsed: 0:19:01 | left: 0:02:50 [I 2022-10-28 06:25:10] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00236 [I 2022-10-28 06:25:10] ...trap/engines/engine.py.230: loss: 0.09348 [I 2022-10-28 06:25:10] ...optimizers/trijoint.py.151: total_norm: 15.817940 [I 2022-10-28 06:25:15] ...trap/engines/engine.py.224: train: epoch 110 | batch 2010/2304 [I 2022-10-28 06:25:15] ...trap/engines/engine.py.228: elapsed: 0:19:06 | left: 0:02:43 [I 2022-10-28 06:25:15] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00234 [I 2022-10-28 06:25:15] ...trap/engines/engine.py.230: loss: 0.09403 [I 2022-10-28 06:25:15] ...optimizers/trijoint.py.151: total_norm: 12.312969 [I 2022-10-28 06:25:21] ...trap/engines/engine.py.224: train: epoch 110 | batch 2020/2304 [I 2022-10-28 06:25:21] ...trap/engines/engine.py.228: elapsed: 0:19:12 | left: 0:02:36 [I 2022-10-28 06:25:21] ...trap/engines/engine.py.229: process: 0.54130 | load: 0.00238 [I 2022-10-28 06:25:21] ...trap/engines/engine.py.230: loss: 0.14519 [I 2022-10-28 06:25:21] ...optimizers/trijoint.py.151: total_norm: 17.972380 [I 2022-10-28 06:25:27] ...trap/engines/engine.py.224: train: epoch 110 | batch 2030/2304 [I 2022-10-28 06:25:27] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:31 [I 2022-10-28 06:25:27] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00235 [I 2022-10-28 06:25:27] ...trap/engines/engine.py.230: loss: 0.15453 [I 2022-10-28 06:25:27] ...optimizers/trijoint.py.151: total_norm: 15.673991 [I 2022-10-28 06:25:32] ...trap/engines/engine.py.224: train: epoch 110 | batch 2040/2304 [I 2022-10-28 06:25:32] ...trap/engines/engine.py.228: elapsed: 0:19:23 | left: 0:02:26 [I 2022-10-28 06:25:32] ...trap/engines/engine.py.229: process: 0.55938 | load: 0.00235 [I 2022-10-28 06:25:32] ...trap/engines/engine.py.230: loss: 0.10974 [I 2022-10-28 06:25:32] ...optimizers/trijoint.py.151: total_norm: 13.661472 [I 2022-10-28 06:25:38] ...trap/engines/engine.py.224: train: epoch 110 | batch 2050/2304 [I 2022-10-28 06:25:38] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:22 [I 2022-10-28 06:25:38] ...trap/engines/engine.py.229: process: 0.56992 | load: 0.00236 [I 2022-10-28 06:25:38] ...trap/engines/engine.py.230: loss: 0.11810 [I 2022-10-28 06:25:38] ...optimizers/trijoint.py.151: total_norm: 24.510744 [I 2022-10-28 06:25:43] ...trap/engines/engine.py.224: train: epoch 110 | batch 2060/2304 [I 2022-10-28 06:25:43] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:02:16 [I 2022-10-28 06:25:43] ...trap/engines/engine.py.229: process: 0.55183 | load: 0.00238 [I 2022-10-28 06:25:43] ...trap/engines/engine.py.230: loss: 0.17765 [I 2022-10-28 06:25:43] ...optimizers/trijoint.py.151: total_norm: 17.335041 [I 2022-10-28 06:25:49] ...trap/engines/engine.py.224: train: epoch 110 | batch 2070/2304 [I 2022-10-28 06:25:49] ...trap/engines/engine.py.228: elapsed: 0:19:40 | left: 0:02:10 [I 2022-10-28 06:25:49] ...trap/engines/engine.py.229: process: 0.56367 | load: 0.00234 [I 2022-10-28 06:25:49] ...trap/engines/engine.py.230: loss: 0.11495 [I 2022-10-28 06:25:49] ...optimizers/trijoint.py.151: total_norm: 19.363850 [I 2022-10-28 06:25:54] ...trap/engines/engine.py.224: train: epoch 110 | batch 2080/2304 [I 2022-10-28 06:25:54] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:02:04 [I 2022-10-28 06:25:54] ...trap/engines/engine.py.229: process: 0.57046 | load: 0.00237 [I 2022-10-28 06:25:54] ...trap/engines/engine.py.230: loss: 0.13892 [I 2022-10-28 06:25:54] ...optimizers/trijoint.py.151: total_norm: 26.555084 [I 2022-10-28 06:26:00] ...trap/engines/engine.py.224: train: epoch 110 | batch 2090/2304 [I 2022-10-28 06:26:00] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:59 [I 2022-10-28 06:26:00] ...trap/engines/engine.py.229: process: 0.55795 | load: 0.00262 [I 2022-10-28 06:26:00] ...trap/engines/engine.py.230: loss: 0.16489 [I 2022-10-28 06:26:00] ...optimizers/trijoint.py.151: total_norm: 15.106005 [I 2022-10-28 06:26:06] ...trap/engines/engine.py.224: train: epoch 110 | batch 2100/2304 [I 2022-10-28 06:26:06] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:53 [I 2022-10-28 06:26:06] ...trap/engines/engine.py.229: process: 0.55713 | load: 0.00244 [I 2022-10-28 06:26:06] ...trap/engines/engine.py.230: loss: 0.10031 [I 2022-10-28 06:26:06] ...optimizers/trijoint.py.151: total_norm: 28.690115 [I 2022-10-28 06:26:11] ...trap/engines/engine.py.224: train: epoch 110 | batch 2110/2304 [I 2022-10-28 06:26:11] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:48 [I 2022-10-28 06:26:11] ...trap/engines/engine.py.229: process: 0.56667 | load: 0.00246 [I 2022-10-28 06:26:11] ...trap/engines/engine.py.230: loss: 0.16054 [I 2022-10-28 06:26:11] ...optimizers/trijoint.py.151: total_norm: 18.848316 [I 2022-10-28 06:26:17] ...trap/engines/engine.py.224: train: epoch 110 | batch 2120/2304 [I 2022-10-28 06:26:17] ...trap/engines/engine.py.228: elapsed: 0:20:08 | left: 0:01:43 [I 2022-10-28 06:26:17] ...trap/engines/engine.py.229: process: 0.56526 | load: 0.00240 [I 2022-10-28 06:26:17] ...trap/engines/engine.py.230: loss: 0.12863 [I 2022-10-28 06:26:17] ...optimizers/trijoint.py.151: total_norm: 12.156148 [I 2022-10-28 06:26:22] ...trap/engines/engine.py.224: train: epoch 110 | batch 2130/2304 [I 2022-10-28 06:26:22] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:37 [I 2022-10-28 06:26:22] ...trap/engines/engine.py.229: process: 0.56674 | load: 0.00246 [I 2022-10-28 06:26:22] ...trap/engines/engine.py.230: loss: 0.14062 [I 2022-10-28 06:26:22] ...optimizers/trijoint.py.151: total_norm: 15.833334 [I 2022-10-28 06:26:28] ...trap/engines/engine.py.224: train: epoch 110 | batch 2140/2304 [I 2022-10-28 06:26:28] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:31 [I 2022-10-28 06:26:28] ...trap/engines/engine.py.229: process: 0.55632 | load: 0.00242 [I 2022-10-28 06:26:28] ...trap/engines/engine.py.230: loss: 0.12079 [I 2022-10-28 06:26:28] ...optimizers/trijoint.py.151: total_norm: 26.316502 [I 2022-10-28 06:26:34] ...trap/engines/engine.py.224: train: epoch 110 | batch 2150/2304 [I 2022-10-28 06:26:34] ...trap/engines/engine.py.228: elapsed: 0:20:24 | left: 0:01:25 [I 2022-10-28 06:26:34] ...trap/engines/engine.py.229: process: 0.56838 | load: 0.00242 [I 2022-10-28 06:26:34] ...trap/engines/engine.py.230: loss: 0.11456 [I 2022-10-28 06:26:34] ...optimizers/trijoint.py.151: total_norm: 8.531564 [I 2022-10-28 06:26:39] ...trap/engines/engine.py.224: train: epoch 110 | batch 2160/2304 [I 2022-10-28 06:26:39] ...trap/engines/engine.py.228: elapsed: 0:20:30 | left: 0:01:20 [I 2022-10-28 06:26:39] ...trap/engines/engine.py.229: process: 0.54488 | load: 0.00243 [I 2022-10-28 06:26:39] ...trap/engines/engine.py.230: loss: 0.09269 [I 2022-10-28 06:26:39] ...optimizers/trijoint.py.151: total_norm: 35.460640 [I 2022-10-28 06:26:45] ...trap/engines/engine.py.224: train: epoch 110 | batch 2170/2304 [I 2022-10-28 06:26:45] ...trap/engines/engine.py.228: elapsed: 0:20:36 | left: 0:01:14 [I 2022-10-28 06:26:45] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00247 [I 2022-10-28 06:26:45] ...trap/engines/engine.py.230: loss: 0.10562 [I 2022-10-28 06:26:45] ...optimizers/trijoint.py.151: total_norm: 20.432997 [I 2022-10-28 06:26:50] ...trap/engines/engine.py.224: train: epoch 110 | batch 2180/2304 [I 2022-10-28 06:26:50] ...trap/engines/engine.py.228: elapsed: 0:20:41 | left: 0:01:09 [I 2022-10-28 06:26:50] ...trap/engines/engine.py.229: process: 0.56178 | load: 0.00241 [I 2022-10-28 06:26:50] ...trap/engines/engine.py.230: loss: 0.11196 [I 2022-10-28 06:26:50] ...optimizers/trijoint.py.151: total_norm: 13.669259 [I 2022-10-28 06:26:56] ...trap/engines/engine.py.224: train: epoch 110 | batch 2190/2304 [I 2022-10-28 06:26:56] ...trap/engines/engine.py.228: elapsed: 0:20:47 | left: 0:01:04 [I 2022-10-28 06:26:56] ...trap/engines/engine.py.229: process: 0.56436 | load: 0.00248 [I 2022-10-28 06:26:56] ...trap/engines/engine.py.230: loss: 0.09929 [I 2022-10-28 06:26:56] ...optimizers/trijoint.py.151: total_norm: 12.359104 [I 2022-10-28 06:27:02] ...trap/engines/engine.py.224: train: epoch 110 | batch 2200/2304 [I 2022-10-28 06:27:02] ...trap/engines/engine.py.228: elapsed: 0:20:53 | left: 0:00:59 [I 2022-10-28 06:27:02] ...trap/engines/engine.py.229: process: 0.55077 | load: 0.00250 [I 2022-10-28 06:27:02] ...trap/engines/engine.py.230: loss: 0.16756 [I 2022-10-28 06:27:02] ...optimizers/trijoint.py.151: total_norm: 22.135729 [I 2022-10-28 06:27:07] ...trap/engines/engine.py.224: train: epoch 110 | batch 2210/2304 [I 2022-10-28 06:27:07] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:00:52 [I 2022-10-28 06:27:07] ...trap/engines/engine.py.229: process: 0.56433 | load: 0.00249 [I 2022-10-28 06:27:07] ...trap/engines/engine.py.230: loss: 0.09784 [I 2022-10-28 06:27:07] ...optimizers/trijoint.py.151: total_norm: 17.032251 [I 2022-10-28 06:27:13] ...trap/engines/engine.py.224: train: epoch 110 | batch 2220/2304 [I 2022-10-28 06:27:13] ...trap/engines/engine.py.228: elapsed: 0:21:04 | left: 0:00:46 [I 2022-10-28 06:27:13] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00247 [I 2022-10-28 06:27:13] ...trap/engines/engine.py.230: loss: 0.08563 [I 2022-10-28 06:27:13] ...optimizers/trijoint.py.151: total_norm: 19.108263 [I 2022-10-28 06:27:19] ...trap/engines/engine.py.224: train: epoch 110 | batch 2230/2304 [I 2022-10-28 06:27:19] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:41 [I 2022-10-28 06:27:19] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00244 [I 2022-10-28 06:27:19] ...trap/engines/engine.py.230: loss: 0.08185 [I 2022-10-28 06:27:19] ...optimizers/trijoint.py.151: total_norm: 25.123148 [I 2022-10-28 06:27:24] ...trap/engines/engine.py.224: train: epoch 110 | batch 2240/2304 [I 2022-10-28 06:27:24] ...trap/engines/engine.py.228: elapsed: 0:21:15 | left: 0:00:35 [I 2022-10-28 06:27:24] ...trap/engines/engine.py.229: process: 0.54149 | load: 0.00246 [I 2022-10-28 06:27:24] ...trap/engines/engine.py.230: loss: 0.11560 [I 2022-10-28 06:27:24] ...optimizers/trijoint.py.151: total_norm: 7.992917 [I 2022-10-28 06:27:30] ...trap/engines/engine.py.224: train: epoch 110 | batch 2250/2304 [I 2022-10-28 06:27:30] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:30 [I 2022-10-28 06:27:30] ...trap/engines/engine.py.229: process: 0.57317 | load: 0.00249 [I 2022-10-28 06:27:30] ...trap/engines/engine.py.230: loss: 0.15227 [I 2022-10-28 06:27:30] ...optimizers/trijoint.py.151: total_norm: 19.151751 [I 2022-10-28 06:27:35] ...trap/engines/engine.py.224: train: epoch 110 | batch 2260/2304 [I 2022-10-28 06:27:35] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:24 [I 2022-10-28 06:27:35] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00246 [I 2022-10-28 06:27:35] ...trap/engines/engine.py.230: loss: 0.10428 [I 2022-10-28 06:27:35] ...optimizers/trijoint.py.151: total_norm: 21.573877 [I 2022-10-28 06:27:41] ...trap/engines/engine.py.224: train: epoch 110 | batch 2270/2304 [I 2022-10-28 06:27:41] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:19 [I 2022-10-28 06:27:41] ...trap/engines/engine.py.229: process: 0.56273 | load: 0.00247 [I 2022-10-28 06:27:41] ...trap/engines/engine.py.230: loss: 0.11155 [I 2022-10-28 06:27:41] ...optimizers/trijoint.py.151: total_norm: 37.948318 [I 2022-10-28 06:27:47] ...trap/engines/engine.py.224: train: epoch 110 | batch 2280/2304 [I 2022-10-28 06:27:47] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:00:13 [I 2022-10-28 06:27:47] ...trap/engines/engine.py.229: process: 0.55925 | load: 0.00242 [I 2022-10-28 06:27:47] ...trap/engines/engine.py.230: loss: 0.11947 [I 2022-10-28 06:27:47] ...optimizers/trijoint.py.151: total_norm: 10.624199 [I 2022-10-28 06:27:52] ...trap/engines/engine.py.224: train: epoch 110 | batch 2290/2304 [I 2022-10-28 06:27:52] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:00:07 [I 2022-10-28 06:27:52] ...trap/engines/engine.py.229: process: 0.56282 | load: 0.00256 [I 2022-10-28 06:27:52] ...trap/engines/engine.py.230: loss: 0.06981 [I 2022-10-28 06:27:52] ...optimizers/trijoint.py.151: total_norm: 11.501890 [I 2022-10-28 06:27:58] ...trap/engines/engine.py.224: train: epoch 110 | batch 2300/2304 [I 2022-10-28 06:27:58] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:02 [I 2022-10-28 06:27:58] ...trap/engines/engine.py.229: process: 0.54654 | load: 0.00015 [I 2022-10-28 06:27:58] ...trap/engines/engine.py.230: loss: 0.14201 [I 2022-10-28 06:27:58] ...optimizers/trijoint.py.151: total_norm: 15.937018 [I 2022-10-28 06:28:00] ...trap/engines/engine.py.224: train: epoch 110 | batch 2304/2304 [I 2022-10-28 06:28:00] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:00 [I 2022-10-28 06:28:00] ...trap/engines/engine.py.229: process: 0.54407 | load: 0.00014 [I 2022-10-28 06:28:00] ...trap/engines/engine.py.230: loss: 0.14143 [I 2022-10-28 06:28:00] ...optimizers/trijoint.py.151: total_norm: 16.486494 [S 2022-10-28 06:28:01] ...trap/engines/engine.py.129: train_epoch.epoch: 110 [S 2022-10-28 06:28:01] ...trap/engines/engine.py.129: train_epoch.loss: 0.12144508697819813 [I 2022-10-28 06:28:01] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 06:28:03] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 110 [I 2022-10-28 06:28:03] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 06:28:05] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 06:28:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 06:28:05] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 06:28:09] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 06:28:09] ...trap/engines/engine.py.321: eval: epoch 110 | batch 0/510 [I 2022-10-28 06:28:09] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:51:43 [I 2022-10-28 06:28:09] ...trap/engines/engine.py.326: process: 6.08509 | load: 2.99520 [I 2022-10-28 06:28:12] ...trap/engines/engine.py.321: eval: epoch 110 | batch 10/510 [I 2022-10-28 06:28:12] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:53 [I 2022-10-28 06:28:12] ...trap/engines/engine.py.326: process: 0.32483 | load: 0.00016 [I 2022-10-28 06:28:16] ...trap/engines/engine.py.321: eval: epoch 110 | batch 20/510 [I 2022-10-28 06:28:16] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:13 [I 2022-10-28 06:28:16] ...trap/engines/engine.py.326: process: 0.32490 | load: 0.00016 [I 2022-10-28 06:28:19] ...trap/engines/engine.py.321: eval: epoch 110 | batch 30/510 [I 2022-10-28 06:28:19] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:45 [I 2022-10-28 06:28:19] ...trap/engines/engine.py.326: process: 0.37716 | load: 0.00017 [I 2022-10-28 06:28:22] ...trap/engines/engine.py.321: eval: epoch 110 | batch 40/510 [I 2022-10-28 06:28:22] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-28 06:28:22] ...trap/engines/engine.py.326: process: 0.33101 | load: 0.00018 [I 2022-10-28 06:28:26] ...trap/engines/engine.py.321: eval: epoch 110 | batch 50/510 [I 2022-10-28 06:28:26] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:34 [I 2022-10-28 06:28:26] ...trap/engines/engine.py.326: process: 0.34404 | load: 0.00017 [I 2022-10-28 06:28:29] ...trap/engines/engine.py.321: eval: epoch 110 | batch 60/510 [I 2022-10-28 06:28:29] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:29 [I 2022-10-28 06:28:29] ...trap/engines/engine.py.326: process: 0.33527 | load: 0.00019 [I 2022-10-28 06:28:32] ...trap/engines/engine.py.321: eval: epoch 110 | batch 70/510 [I 2022-10-28 06:28:32] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:27 [I 2022-10-28 06:28:32] ...trap/engines/engine.py.326: process: 0.32434 | load: 0.00017 [I 2022-10-28 06:28:36] ...trap/engines/engine.py.321: eval: epoch 110 | batch 80/510 [I 2022-10-28 06:28:36] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:23 [I 2022-10-28 06:28:36] ...trap/engines/engine.py.326: process: 0.33570 | load: 0.00017 [I 2022-10-28 06:28:39] ...trap/engines/engine.py.321: eval: epoch 110 | batch 90/510 [I 2022-10-28 06:28:39] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:20 [I 2022-10-28 06:28:39] ...trap/engines/engine.py.326: process: 0.34370 | load: 0.00020 [I 2022-10-28 06:28:42] ...trap/engines/engine.py.321: eval: epoch 110 | batch 100/510 [I 2022-10-28 06:28:42] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:14 [I 2022-10-28 06:28:42] ...trap/engines/engine.py.326: process: 0.29491 | load: 0.00041 [I 2022-10-28 06:28:45] ...trap/engines/engine.py.321: eval: epoch 110 | batch 110/510 [I 2022-10-28 06:28:45] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:57 [I 2022-10-28 06:28:45] ...trap/engines/engine.py.326: process: 0.28393 | load: 0.00021 [I 2022-10-28 06:28:48] ...trap/engines/engine.py.321: eval: epoch 110 | batch 120/510 [I 2022-10-28 06:28:48] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:51 [I 2022-10-28 06:28:48] ...trap/engines/engine.py.326: process: 0.28386 | load: 0.00016 [I 2022-10-28 06:28:51] ...trap/engines/engine.py.321: eval: epoch 110 | batch 130/510 [I 2022-10-28 06:28:51] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:50 [I 2022-10-28 06:28:51] ...trap/engines/engine.py.326: process: 0.30716 | load: 0.00019 [I 2022-10-28 06:28:54] ...trap/engines/engine.py.321: eval: epoch 110 | batch 140/510 [I 2022-10-28 06:28:54] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-28 06:28:54] ...trap/engines/engine.py.326: process: 0.28596 | load: 0.00018 [I 2022-10-28 06:28:57] ...trap/engines/engine.py.321: eval: epoch 110 | batch 150/510 [I 2022-10-28 06:28:57] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:43 [I 2022-10-28 06:28:57] ...trap/engines/engine.py.326: process: 0.28915 | load: 0.00015 [I 2022-10-28 06:29:00] ...trap/engines/engine.py.321: eval: epoch 110 | batch 160/510 [I 2022-10-28 06:29:00] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-28 06:29:00] ...trap/engines/engine.py.326: process: 0.28208 | load: 0.00017 [I 2022-10-28 06:29:03] ...trap/engines/engine.py.321: eval: epoch 110 | batch 170/510 [I 2022-10-28 06:29:03] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:37 [I 2022-10-28 06:29:03] ...trap/engines/engine.py.326: process: 0.28332 | load: 0.00022 [I 2022-10-28 06:29:06] ...trap/engines/engine.py.321: eval: epoch 110 | batch 180/510 [I 2022-10-28 06:29:06] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-28 06:29:06] ...trap/engines/engine.py.326: process: 0.28703 | load: 0.00015 [I 2022-10-28 06:29:08] ...trap/engines/engine.py.321: eval: epoch 110 | batch 190/510 [I 2022-10-28 06:29:08] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:32 [I 2022-10-28 06:29:08] ...trap/engines/engine.py.326: process: 0.29141 | load: 0.00017 [I 2022-10-28 06:29:11] ...trap/engines/engine.py.321: eval: epoch 110 | batch 200/510 [I 2022-10-28 06:29:11] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:28 [I 2022-10-28 06:29:11] ...trap/engines/engine.py.326: process: 0.28393 | load: 0.00018 [I 2022-10-28 06:29:14] ...trap/engines/engine.py.321: eval: epoch 110 | batch 210/510 [I 2022-10-28 06:29:14] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:25 [I 2022-10-28 06:29:14] ...trap/engines/engine.py.326: process: 0.28982 | load: 0.00017 [I 2022-10-28 06:29:17] ...trap/engines/engine.py.321: eval: epoch 110 | batch 220/510 [I 2022-10-28 06:29:17] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:23 [I 2022-10-28 06:29:17] ...trap/engines/engine.py.326: process: 0.28694 | load: 0.00018 [I 2022-10-28 06:29:20] ...trap/engines/engine.py.321: eval: epoch 110 | batch 230/510 [I 2022-10-28 06:29:20] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:24 [I 2022-10-28 06:29:20] ...trap/engines/engine.py.326: process: 0.28661 | load: 0.00018 [I 2022-10-28 06:29:23] ...trap/engines/engine.py.321: eval: epoch 110 | batch 240/510 [I 2022-10-28 06:29:23] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 06:29:23] ...trap/engines/engine.py.326: process: 0.28399 | load: 0.00019 [I 2022-10-28 06:29:26] ...trap/engines/engine.py.321: eval: epoch 110 | batch 250/510 [I 2022-10-28 06:29:26] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-28 06:29:26] ...trap/engines/engine.py.326: process: 0.29036 | load: 0.00019 [I 2022-10-28 06:29:29] ...trap/engines/engine.py.321: eval: epoch 110 | batch 260/510 [I 2022-10-28 06:29:29] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:11 [I 2022-10-28 06:29:29] ...trap/engines/engine.py.326: process: 0.28797 | load: 0.00015 [I 2022-10-28 06:29:32] ...trap/engines/engine.py.321: eval: epoch 110 | batch 270/510 [I 2022-10-28 06:29:32] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-28 06:29:32] ...trap/engines/engine.py.326: process: 0.28653 | load: 0.00016 [I 2022-10-28 06:29:34] ...trap/engines/engine.py.321: eval: epoch 110 | batch 280/510 [I 2022-10-28 06:29:34] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:05 [I 2022-10-28 06:29:34] ...trap/engines/engine.py.326: process: 0.28123 | load: 0.00019 [I 2022-10-28 06:29:37] ...trap/engines/engine.py.321: eval: epoch 110 | batch 290/510 [I 2022-10-28 06:29:37] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 06:29:37] ...trap/engines/engine.py.326: process: 0.28905 | load: 0.00016 [I 2022-10-28 06:29:40] ...trap/engines/engine.py.321: eval: epoch 110 | batch 300/510 [I 2022-10-28 06:29:40] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-28 06:29:40] ...trap/engines/engine.py.326: process: 0.29039 | load: 0.00015 [I 2022-10-28 06:29:43] ...trap/engines/engine.py.321: eval: epoch 110 | batch 310/510 [I 2022-10-28 06:29:43] ...trap/engines/engine.py.325: elapsed: 0:01:39 | left: 0:00:57 [I 2022-10-28 06:29:43] ...trap/engines/engine.py.326: process: 0.28026 | load: 0.00019 [I 2022-10-28 06:29:46] ...trap/engines/engine.py.321: eval: epoch 110 | batch 320/510 [I 2022-10-28 06:29:46] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 06:29:46] ...trap/engines/engine.py.326: process: 0.28612 | load: 0.00018 [I 2022-10-28 06:29:49] ...trap/engines/engine.py.321: eval: epoch 110 | batch 330/510 [I 2022-10-28 06:29:49] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-28 06:29:49] ...trap/engines/engine.py.326: process: 0.28795 | load: 0.00015 [I 2022-10-28 06:29:52] ...trap/engines/engine.py.321: eval: epoch 110 | batch 340/510 [I 2022-10-28 06:29:52] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-28 06:29:52] ...trap/engines/engine.py.326: process: 0.28893 | load: 0.00016 [I 2022-10-28 06:29:55] ...trap/engines/engine.py.321: eval: epoch 110 | batch 350/510 [I 2022-10-28 06:29:55] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:45 [I 2022-10-28 06:29:55] ...trap/engines/engine.py.326: process: 0.28305 | load: 0.00018 [I 2022-10-28 06:29:57] ...trap/engines/engine.py.321: eval: epoch 110 | batch 360/510 [I 2022-10-28 06:29:57] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 06:29:57] ...trap/engines/engine.py.326: process: 0.28446 | load: 0.00017 [I 2022-10-28 06:30:00] ...trap/engines/engine.py.321: eval: epoch 110 | batch 370/510 [I 2022-10-28 06:30:00] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 06:30:00] ...trap/engines/engine.py.326: process: 0.28271 | load: 0.00015 [I 2022-10-28 06:30:03] ...trap/engines/engine.py.321: eval: epoch 110 | batch 380/510 [I 2022-10-28 06:30:03] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 06:30:03] ...trap/engines/engine.py.326: process: 0.28307 | load: 0.00015 [I 2022-10-28 06:30:06] ...trap/engines/engine.py.321: eval: epoch 110 | batch 390/510 [I 2022-10-28 06:30:06] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 06:30:06] ...trap/engines/engine.py.326: process: 0.28993 | load: 0.00015 [I 2022-10-28 06:30:09] ...trap/engines/engine.py.321: eval: epoch 110 | batch 400/510 [I 2022-10-28 06:30:09] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 06:30:09] ...trap/engines/engine.py.326: process: 0.29005 | load: 0.00019 [I 2022-10-28 06:30:12] ...trap/engines/engine.py.321: eval: epoch 110 | batch 410/510 [I 2022-10-28 06:30:12] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 06:30:12] ...trap/engines/engine.py.326: process: 0.28725 | load: 0.00015 [I 2022-10-28 06:30:15] ...trap/engines/engine.py.321: eval: epoch 110 | batch 420/510 [I 2022-10-28 06:30:15] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 06:30:15] ...trap/engines/engine.py.326: process: 0.29068 | load: 0.00016 [I 2022-10-28 06:30:18] ...trap/engines/engine.py.321: eval: epoch 110 | batch 430/510 [I 2022-10-28 06:30:18] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:22 [I 2022-10-28 06:30:18] ...trap/engines/engine.py.326: process: 0.28072 | load: 0.00015 [I 2022-10-28 06:30:20] ...trap/engines/engine.py.321: eval: epoch 110 | batch 440/510 [I 2022-10-28 06:30:20] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 06:30:20] ...trap/engines/engine.py.326: process: 0.28456 | load: 0.00015 [I 2022-10-28 06:30:23] ...trap/engines/engine.py.321: eval: epoch 110 | batch 450/510 [I 2022-10-28 06:30:23] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 06:30:23] ...trap/engines/engine.py.326: process: 0.28925 | load: 0.00016 [I 2022-10-28 06:30:26] ...trap/engines/engine.py.321: eval: epoch 110 | batch 460/510 [I 2022-10-28 06:30:26] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 06:30:26] ...trap/engines/engine.py.326: process: 0.28490 | load: 0.00016 [I 2022-10-28 06:30:29] ...trap/engines/engine.py.321: eval: epoch 110 | batch 470/510 [I 2022-10-28 06:30:29] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 06:30:29] ...trap/engines/engine.py.326: process: 0.27650 | load: 0.00021 [I 2022-10-28 06:30:32] ...trap/engines/engine.py.321: eval: epoch 110 | batch 480/510 [I 2022-10-28 06:30:32] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 06:30:32] ...trap/engines/engine.py.326: process: 0.28777 | load: 0.00017 [I 2022-10-28 06:30:35] ...trap/engines/engine.py.321: eval: epoch 110 | batch 490/510 [I 2022-10-28 06:30:35] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 06:30:35] ...trap/engines/engine.py.326: process: 0.28534 | load: 0.00016 [I 2022-10-28 06:30:38] ...trap/engines/engine.py.321: eval: epoch 110 | batch 500/510 [I 2022-10-28 06:30:38] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 06:30:38] ...trap/engines/engine.py.326: process: 0.28757 | load: 0.00016 [I 2022-10-28 06:30:41] ...trap/engines/engine.py.321: eval: epoch 110 | batch 510/510 [I 2022-10-28 06:30:41] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 06:30:41] ...trap/engines/engine.py.326: process: 0.28604 | load: 0.00011 [S 2022-10-28 06:30:41] ...trap/engines/engine.py.132: eval_epoch.epoch: 110 [I 2022-10-28 06:30:41] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.737200 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.906900 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935200 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.731700 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.905600 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934100 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010861 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008455 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008085 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014636 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010547 [S 2022-10-28 06:30:45] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007803 [I 2022-10-28 06:30:47] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 06:30:47] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 06:30:48] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 06:30:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 06:30:48] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 06:30:53] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 06:30:54] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 06:30:59] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 06:30:59] ...trap/engines/engine.py.163: Training model on trainset for epoch 111 [I 2022-10-28 06:30:59] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 06:31:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 0/2304 [I 2022-10-28 06:31:02] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:44:37 [I 2022-10-28 06:31:02] ...trap/engines/engine.py.229: process: 2.72468 | load: 2.09984 [I 2022-10-28 06:31:02] ...trap/engines/engine.py.230: loss: 0.12037 [I 2022-10-28 06:31:02] ...optimizers/trijoint.py.151: total_norm: 18.364305 [I 2022-10-28 06:31:08] ...trap/engines/engine.py.224: train: epoch 111 | batch 10/2304 [I 2022-10-28 06:31:08] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:15 [I 2022-10-28 06:31:08] ...trap/engines/engine.py.229: process: 0.55726 | load: 0.00377 [I 2022-10-28 06:31:08] ...trap/engines/engine.py.230: loss: 0.09833 [I 2022-10-28 06:31:08] ...optimizers/trijoint.py.151: total_norm: 12.937083 [I 2022-10-28 06:31:13] ...trap/engines/engine.py.224: train: epoch 111 | batch 20/2304 [I 2022-10-28 06:31:13] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:03 [I 2022-10-28 06:31:13] ...trap/engines/engine.py.229: process: 0.55135 | load: 0.00356 [I 2022-10-28 06:31:13] ...trap/engines/engine.py.230: loss: 0.14184 [I 2022-10-28 06:31:13] ...optimizers/trijoint.py.151: total_norm: 46.399956 [I 2022-10-28 06:31:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 30/2304 [I 2022-10-28 06:31:19] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:20 [I 2022-10-28 06:31:19] ...trap/engines/engine.py.229: process: 0.56683 | load: 0.00355 [I 2022-10-28 06:31:19] ...trap/engines/engine.py.230: loss: 0.13902 [I 2022-10-28 06:31:19] ...optimizers/trijoint.py.151: total_norm: 16.293278 [I 2022-10-28 06:31:24] ...trap/engines/engine.py.224: train: epoch 111 | batch 40/2304 [I 2022-10-28 06:31:24] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:08 [I 2022-10-28 06:31:24] ...trap/engines/engine.py.229: process: 0.56512 | load: 0.00269 [I 2022-10-28 06:31:24] ...trap/engines/engine.py.230: loss: 0.09754 [I 2022-10-28 06:31:24] ...optimizers/trijoint.py.151: total_norm: 12.901025 [I 2022-10-28 06:31:30] ...trap/engines/engine.py.224: train: epoch 111 | batch 50/2304 [I 2022-10-28 06:31:30] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:54 [I 2022-10-28 06:31:30] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00314 [I 2022-10-28 06:31:30] ...trap/engines/engine.py.230: loss: 0.12854 [I 2022-10-28 06:31:30] ...optimizers/trijoint.py.151: total_norm: 15.986753 [I 2022-10-28 06:31:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 60/2304 [I 2022-10-28 06:31:36] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:30 [I 2022-10-28 06:31:36] ...trap/engines/engine.py.229: process: 0.58151 | load: 0.00290 [I 2022-10-28 06:31:36] ...trap/engines/engine.py.230: loss: 0.10782 [I 2022-10-28 06:31:36] ...optimizers/trijoint.py.151: total_norm: 11.833303 [I 2022-10-28 06:31:41] ...trap/engines/engine.py.224: train: epoch 111 | batch 70/2304 [I 2022-10-28 06:31:41] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:41 [I 2022-10-28 06:31:41] ...trap/engines/engine.py.229: process: 0.55488 | load: 0.00281 [I 2022-10-28 06:31:41] ...trap/engines/engine.py.230: loss: 0.09000 [I 2022-10-28 06:31:41] ...optimizers/trijoint.py.151: total_norm: 8.671506 [I 2022-10-28 06:31:47] ...trap/engines/engine.py.224: train: epoch 111 | batch 80/2304 [I 2022-10-28 06:31:47] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:35 [I 2022-10-28 06:31:47] ...trap/engines/engine.py.229: process: 0.55807 | load: 0.00248 [I 2022-10-28 06:31:47] ...trap/engines/engine.py.230: loss: 0.11734 [I 2022-10-28 06:31:47] ...optimizers/trijoint.py.151: total_norm: 29.067369 [I 2022-10-28 06:31:52] ...trap/engines/engine.py.224: train: epoch 111 | batch 90/2304 [I 2022-10-28 06:31:52] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:39 [I 2022-10-28 06:31:52] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00283 [I 2022-10-28 06:31:52] ...trap/engines/engine.py.230: loss: 0.09759 [I 2022-10-28 06:31:52] ...optimizers/trijoint.py.151: total_norm: 11.645100 [I 2022-10-28 06:31:58] ...trap/engines/engine.py.224: train: epoch 111 | batch 100/2304 [I 2022-10-28 06:31:58] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:35 [I 2022-10-28 06:31:58] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00268 [I 2022-10-28 06:31:58] ...trap/engines/engine.py.230: loss: 0.12478 [I 2022-10-28 06:31:58] ...optimizers/trijoint.py.151: total_norm: 25.066322 [I 2022-10-28 06:32:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 110/2304 [I 2022-10-28 06:32:04] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:22 [I 2022-10-28 06:32:04] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00291 [I 2022-10-28 06:32:04] ...trap/engines/engine.py.230: loss: 0.14177 [I 2022-10-28 06:32:04] ...optimizers/trijoint.py.151: total_norm: 17.931322 [I 2022-10-28 06:32:09] ...trap/engines/engine.py.224: train: epoch 111 | batch 120/2304 [I 2022-10-28 06:32:09] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:12 [I 2022-10-28 06:32:09] ...trap/engines/engine.py.229: process: 0.54641 | load: 0.00258 [I 2022-10-28 06:32:09] ...trap/engines/engine.py.230: loss: 0.12984 [I 2022-10-28 06:32:09] ...optimizers/trijoint.py.151: total_norm: 53.237507 [I 2022-10-28 06:32:15] ...trap/engines/engine.py.224: train: epoch 111 | batch 130/2304 [I 2022-10-28 06:32:15] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:14 [I 2022-10-28 06:32:15] ...trap/engines/engine.py.229: process: 0.55612 | load: 0.00272 [I 2022-10-28 06:32:15] ...trap/engines/engine.py.230: loss: 0.15959 [I 2022-10-28 06:32:15] ...optimizers/trijoint.py.151: total_norm: 20.859678 [I 2022-10-28 06:32:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 140/2304 [I 2022-10-28 06:32:20] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:09 [I 2022-10-28 06:32:20] ...trap/engines/engine.py.229: process: 0.57155 | load: 0.00250 [I 2022-10-28 06:32:20] ...trap/engines/engine.py.230: loss: 0.12367 [I 2022-10-28 06:32:20] ...optimizers/trijoint.py.151: total_norm: 23.932129 [I 2022-10-28 06:32:26] ...trap/engines/engine.py.224: train: epoch 111 | batch 150/2304 [I 2022-10-28 06:32:26] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:50 [I 2022-10-28 06:32:26] ...trap/engines/engine.py.229: process: 0.54761 | load: 0.00249 [I 2022-10-28 06:32:26] ...trap/engines/engine.py.230: loss: 0.12654 [I 2022-10-28 06:32:26] ...optimizers/trijoint.py.151: total_norm: 10.613349 [I 2022-10-28 06:32:31] ...trap/engines/engine.py.224: train: epoch 111 | batch 160/2304 [I 2022-10-28 06:32:31] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:53 [I 2022-10-28 06:32:31] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00255 [I 2022-10-28 06:32:31] ...trap/engines/engine.py.230: loss: 0.13365 [I 2022-10-28 06:32:31] ...optimizers/trijoint.py.151: total_norm: 11.644166 [I 2022-10-28 06:32:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 170/2304 [I 2022-10-28 06:32:37] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:51 [I 2022-10-28 06:32:37] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00239 [I 2022-10-28 06:32:37] ...trap/engines/engine.py.230: loss: 0.10363 [I 2022-10-28 06:32:37] ...optimizers/trijoint.py.151: total_norm: 8.693379 [I 2022-10-28 06:32:43] ...trap/engines/engine.py.224: train: epoch 111 | batch 180/2304 [I 2022-10-28 06:32:43] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:43 [I 2022-10-28 06:32:43] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00250 [I 2022-10-28 06:32:43] ...trap/engines/engine.py.230: loss: 0.12101 [I 2022-10-28 06:32:43] ...optimizers/trijoint.py.151: total_norm: 13.120722 [I 2022-10-28 06:32:48] ...trap/engines/engine.py.224: train: epoch 111 | batch 190/2304 [I 2022-10-28 06:32:48] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:40 [I 2022-10-28 06:32:48] ...trap/engines/engine.py.229: process: 0.56309 | load: 0.00250 [I 2022-10-28 06:32:48] ...trap/engines/engine.py.230: loss: 0.14148 [I 2022-10-28 06:32:48] ...optimizers/trijoint.py.151: total_norm: 20.145027 [I 2022-10-28 06:32:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 200/2304 [I 2022-10-28 06:32:54] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:28 [I 2022-10-28 06:32:54] ...trap/engines/engine.py.229: process: 0.56021 | load: 0.00247 [I 2022-10-28 06:32:54] ...trap/engines/engine.py.230: loss: 0.09264 [I 2022-10-28 06:32:54] ...optimizers/trijoint.py.151: total_norm: 22.178600 [I 2022-10-28 06:32:59] ...trap/engines/engine.py.224: train: epoch 111 | batch 210/2304 [I 2022-10-28 06:32:59] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:26 [I 2022-10-28 06:32:59] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00249 [I 2022-10-28 06:32:59] ...trap/engines/engine.py.230: loss: 0.11780 [I 2022-10-28 06:32:59] ...optimizers/trijoint.py.151: total_norm: 15.506507 [I 2022-10-28 06:33:05] ...trap/engines/engine.py.224: train: epoch 111 | batch 220/2304 [I 2022-10-28 06:33:05] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:18 [I 2022-10-28 06:33:05] ...trap/engines/engine.py.229: process: 0.55000 | load: 0.00255 [I 2022-10-28 06:33:05] ...trap/engines/engine.py.230: loss: 0.10651 [I 2022-10-28 06:33:05] ...optimizers/trijoint.py.151: total_norm: 19.066343 [I 2022-10-28 06:33:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 230/2304 [I 2022-10-28 06:33:10] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:19 [I 2022-10-28 06:33:10] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00241 [I 2022-10-28 06:33:10] ...trap/engines/engine.py.230: loss: 0.06680 [I 2022-10-28 06:33:10] ...optimizers/trijoint.py.151: total_norm: 11.982547 [I 2022-10-28 06:33:16] ...trap/engines/engine.py.224: train: epoch 111 | batch 240/2304 [I 2022-10-28 06:33:16] ...trap/engines/engine.py.228: elapsed: 0:02:16 | left: 0:19:13 [I 2022-10-28 06:33:16] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00239 [I 2022-10-28 06:33:16] ...trap/engines/engine.py.230: loss: 0.10442 [I 2022-10-28 06:33:16] ...optimizers/trijoint.py.151: total_norm: 19.556976 [I 2022-10-28 06:33:22] ...trap/engines/engine.py.224: train: epoch 111 | batch 250/2304 [I 2022-10-28 06:33:22] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:02 [I 2022-10-28 06:33:22] ...trap/engines/engine.py.229: process: 0.56195 | load: 0.00244 [I 2022-10-28 06:33:22] ...trap/engines/engine.py.230: loss: 0.12919 [I 2022-10-28 06:33:22] ...optimizers/trijoint.py.151: total_norm: 10.594402 [I 2022-10-28 06:33:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 260/2304 [I 2022-10-28 06:33:27] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:55 [I 2022-10-28 06:33:27] ...trap/engines/engine.py.229: process: 0.54478 | load: 0.00252 [I 2022-10-28 06:33:27] ...trap/engines/engine.py.230: loss: 0.15445 [I 2022-10-28 06:33:27] ...optimizers/trijoint.py.151: total_norm: 18.907387 [I 2022-10-28 06:33:33] ...trap/engines/engine.py.224: train: epoch 111 | batch 270/2304 [I 2022-10-28 06:33:33] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:47 [I 2022-10-28 06:33:33] ...trap/engines/engine.py.229: process: 0.54332 | load: 0.00238 [I 2022-10-28 06:33:33] ...trap/engines/engine.py.230: loss: 0.10002 [I 2022-10-28 06:33:33] ...optimizers/trijoint.py.151: total_norm: 13.917031 [I 2022-10-28 06:33:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 280/2304 [I 2022-10-28 06:33:38] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:50 [I 2022-10-28 06:33:38] ...trap/engines/engine.py.229: process: 0.55590 | load: 0.00246 [I 2022-10-28 06:33:38] ...trap/engines/engine.py.230: loss: 0.13818 [I 2022-10-28 06:33:38] ...optimizers/trijoint.py.151: total_norm: 13.044428 [I 2022-10-28 06:33:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 290/2304 [I 2022-10-28 06:33:44] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:34 [I 2022-10-28 06:33:44] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00243 [I 2022-10-28 06:33:44] ...trap/engines/engine.py.230: loss: 0.12213 [I 2022-10-28 06:33:44] ...optimizers/trijoint.py.151: total_norm: 24.577410 [I 2022-10-28 06:33:50] ...trap/engines/engine.py.224: train: epoch 111 | batch 300/2304 [I 2022-10-28 06:33:50] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:40 [I 2022-10-28 06:33:50] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00246 [I 2022-10-28 06:33:50] ...trap/engines/engine.py.230: loss: 0.07815 [I 2022-10-28 06:33:50] ...optimizers/trijoint.py.151: total_norm: 16.714308 [I 2022-10-28 06:33:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 310/2304 [I 2022-10-28 06:33:55] ...trap/engines/engine.py.228: elapsed: 0:02:55 | left: 0:18:35 [I 2022-10-28 06:33:55] ...trap/engines/engine.py.229: process: 0.56711 | load: 0.00245 [I 2022-10-28 06:33:55] ...trap/engines/engine.py.230: loss: 0.12345 [I 2022-10-28 06:33:55] ...optimizers/trijoint.py.151: total_norm: 14.730323 [I 2022-10-28 06:34:01] ...trap/engines/engine.py.224: train: epoch 111 | batch 320/2304 [I 2022-10-28 06:34:01] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:28 [I 2022-10-28 06:34:01] ...trap/engines/engine.py.229: process: 0.57766 | load: 0.00260 [I 2022-10-28 06:34:01] ...trap/engines/engine.py.230: loss: 0.13140 [I 2022-10-28 06:34:01] ...optimizers/trijoint.py.151: total_norm: 21.898693 [I 2022-10-28 06:34:06] ...trap/engines/engine.py.224: train: epoch 111 | batch 330/2304 [I 2022-10-28 06:34:06] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:17 [I 2022-10-28 06:34:06] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00247 [I 2022-10-28 06:34:06] ...trap/engines/engine.py.230: loss: 0.10474 [I 2022-10-28 06:34:06] ...optimizers/trijoint.py.151: total_norm: 12.642037 [I 2022-10-28 06:34:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 340/2304 [I 2022-10-28 06:34:12] ...trap/engines/engine.py.228: elapsed: 0:03:12 | left: 0:18:18 [I 2022-10-28 06:34:12] ...trap/engines/engine.py.229: process: 0.56382 | load: 0.00249 [I 2022-10-28 06:34:12] ...trap/engines/engine.py.230: loss: 0.14717 [I 2022-10-28 06:34:12] ...optimizers/trijoint.py.151: total_norm: 51.297394 [I 2022-10-28 06:34:18] ...trap/engines/engine.py.224: train: epoch 111 | batch 350/2304 [I 2022-10-28 06:34:18] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:20 [I 2022-10-28 06:34:18] ...trap/engines/engine.py.229: process: 0.56439 | load: 0.00245 [I 2022-10-28 06:34:18] ...trap/engines/engine.py.230: loss: 0.16978 [I 2022-10-28 06:34:18] ...optimizers/trijoint.py.151: total_norm: 29.366140 [I 2022-10-28 06:34:23] ...trap/engines/engine.py.224: train: epoch 111 | batch 360/2304 [I 2022-10-28 06:34:23] ...trap/engines/engine.py.228: elapsed: 0:03:23 | left: 0:18:02 [I 2022-10-28 06:34:23] ...trap/engines/engine.py.229: process: 0.56022 | load: 0.00242 [I 2022-10-28 06:34:23] ...trap/engines/engine.py.230: loss: 0.13092 [I 2022-10-28 06:34:23] ...optimizers/trijoint.py.151: total_norm: 14.290113 [I 2022-10-28 06:34:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 370/2304 [I 2022-10-28 06:34:29] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:07 [I 2022-10-28 06:34:29] ...trap/engines/engine.py.229: process: 0.56103 | load: 0.00246 [I 2022-10-28 06:34:29] ...trap/engines/engine.py.230: loss: 0.15229 [I 2022-10-28 06:34:29] ...optimizers/trijoint.py.151: total_norm: 16.440323 [I 2022-10-28 06:34:34] ...trap/engines/engine.py.224: train: epoch 111 | batch 380/2304 [I 2022-10-28 06:34:34] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:49 [I 2022-10-28 06:34:34] ...trap/engines/engine.py.229: process: 0.56549 | load: 0.00243 [I 2022-10-28 06:34:34] ...trap/engines/engine.py.230: loss: 0.11394 [I 2022-10-28 06:34:34] ...optimizers/trijoint.py.151: total_norm: 31.296354 [I 2022-10-28 06:34:40] ...trap/engines/engine.py.224: train: epoch 111 | batch 390/2304 [I 2022-10-28 06:34:40] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:40 [I 2022-10-28 06:34:40] ...trap/engines/engine.py.229: process: 0.55116 | load: 0.00242 [I 2022-10-28 06:34:40] ...trap/engines/engine.py.230: loss: 0.08804 [I 2022-10-28 06:34:40] ...optimizers/trijoint.py.151: total_norm: 11.909096 [I 2022-10-28 06:34:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 400/2304 [I 2022-10-28 06:34:45] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:48 [I 2022-10-28 06:34:45] ...trap/engines/engine.py.229: process: 0.57098 | load: 0.00261 [I 2022-10-28 06:34:45] ...trap/engines/engine.py.230: loss: 0.11236 [I 2022-10-28 06:34:45] ...optimizers/trijoint.py.151: total_norm: 11.014358 [I 2022-10-28 06:34:51] ...trap/engines/engine.py.224: train: epoch 111 | batch 410/2304 [I 2022-10-28 06:34:51] ...trap/engines/engine.py.228: elapsed: 0:03:51 | left: 0:17:34 [I 2022-10-28 06:34:51] ...trap/engines/engine.py.229: process: 0.54970 | load: 0.00240 [I 2022-10-28 06:34:51] ...trap/engines/engine.py.230: loss: 0.16414 [I 2022-10-28 06:34:51] ...optimizers/trijoint.py.151: total_norm: 17.010027 [I 2022-10-28 06:34:57] ...trap/engines/engine.py.224: train: epoch 111 | batch 420/2304 [I 2022-10-28 06:34:57] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:34 [I 2022-10-28 06:34:57] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00242 [I 2022-10-28 06:34:57] ...trap/engines/engine.py.230: loss: 0.08119 [I 2022-10-28 06:34:57] ...optimizers/trijoint.py.151: total_norm: 11.164158 [I 2022-10-28 06:35:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 430/2304 [I 2022-10-28 06:35:02] ...trap/engines/engine.py.228: elapsed: 0:04:02 | left: 0:17:22 [I 2022-10-28 06:35:02] ...trap/engines/engine.py.229: process: 0.54972 | load: 0.00241 [I 2022-10-28 06:35:02] ...trap/engines/engine.py.230: loss: 0.12781 [I 2022-10-28 06:35:02] ...optimizers/trijoint.py.151: total_norm: 16.840324 [I 2022-10-28 06:35:08] ...trap/engines/engine.py.224: train: epoch 111 | batch 440/2304 [I 2022-10-28 06:35:08] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:21 [I 2022-10-28 06:35:08] ...trap/engines/engine.py.229: process: 0.56160 | load: 0.00244 [I 2022-10-28 06:35:08] ...trap/engines/engine.py.230: loss: 0.15555 [I 2022-10-28 06:35:08] ...optimizers/trijoint.py.151: total_norm: 37.680794 [I 2022-10-28 06:35:13] ...trap/engines/engine.py.224: train: epoch 111 | batch 450/2304 [I 2022-10-28 06:35:13] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:10 [I 2022-10-28 06:35:13] ...trap/engines/engine.py.229: process: 0.54928 | load: 0.00243 [I 2022-10-28 06:35:13] ...trap/engines/engine.py.230: loss: 0.13913 [I 2022-10-28 06:35:13] ...optimizers/trijoint.py.151: total_norm: 29.588715 [I 2022-10-28 06:35:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 460/2304 [I 2022-10-28 06:35:19] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:08 [I 2022-10-28 06:35:19] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00244 [I 2022-10-28 06:35:19] ...trap/engines/engine.py.230: loss: 0.10197 [I 2022-10-28 06:35:19] ...optimizers/trijoint.py.151: total_norm: 13.326104 [I 2022-10-28 06:35:25] ...trap/engines/engine.py.224: train: epoch 111 | batch 470/2304 [I 2022-10-28 06:35:25] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:05 [I 2022-10-28 06:35:25] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00244 [I 2022-10-28 06:35:25] ...trap/engines/engine.py.230: loss: 0.11266 [I 2022-10-28 06:35:25] ...optimizers/trijoint.py.151: total_norm: 17.388477 [I 2022-10-28 06:35:30] ...trap/engines/engine.py.224: train: epoch 111 | batch 480/2304 [I 2022-10-28 06:35:30] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:06 [I 2022-10-28 06:35:30] ...trap/engines/engine.py.229: process: 0.56745 | load: 0.00242 [I 2022-10-28 06:35:30] ...trap/engines/engine.py.230: loss: 0.14432 [I 2022-10-28 06:35:30] ...optimizers/trijoint.py.151: total_norm: 13.238576 [I 2022-10-28 06:35:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 490/2304 [I 2022-10-28 06:35:36] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:56 [I 2022-10-28 06:35:36] ...trap/engines/engine.py.229: process: 0.54726 | load: 0.00243 [I 2022-10-28 06:35:36] ...trap/engines/engine.py.230: loss: 0.08337 [I 2022-10-28 06:35:36] ...optimizers/trijoint.py.151: total_norm: 12.917225 [I 2022-10-28 06:35:41] ...trap/engines/engine.py.224: train: epoch 111 | batch 500/2304 [I 2022-10-28 06:35:41] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:52 [I 2022-10-28 06:35:41] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00239 [I 2022-10-28 06:35:41] ...trap/engines/engine.py.230: loss: 0.11191 [I 2022-10-28 06:35:41] ...optimizers/trijoint.py.151: total_norm: 9.841002 [I 2022-10-28 06:35:47] ...trap/engines/engine.py.224: train: epoch 111 | batch 510/2304 [I 2022-10-28 06:35:47] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:43 [I 2022-10-28 06:35:47] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00242 [I 2022-10-28 06:35:47] ...trap/engines/engine.py.230: loss: 0.12037 [I 2022-10-28 06:35:47] ...optimizers/trijoint.py.151: total_norm: 19.555490 [I 2022-10-28 06:35:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 520/2304 [I 2022-10-28 06:35:53] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:35 [I 2022-10-28 06:35:53] ...trap/engines/engine.py.229: process: 0.55925 | load: 0.00242 [I 2022-10-28 06:35:53] ...trap/engines/engine.py.230: loss: 0.10243 [I 2022-10-28 06:35:53] ...optimizers/trijoint.py.151: total_norm: 17.538200 [I 2022-10-28 06:35:58] ...trap/engines/engine.py.224: train: epoch 111 | batch 530/2304 [I 2022-10-28 06:35:58] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:30 [I 2022-10-28 06:35:58] ...trap/engines/engine.py.229: process: 0.55399 | load: 0.00241 [I 2022-10-28 06:35:58] ...trap/engines/engine.py.230: loss: 0.11181 [I 2022-10-28 06:35:58] ...optimizers/trijoint.py.151: total_norm: 9.111934 [I 2022-10-28 06:36:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 540/2304 [I 2022-10-28 06:36:04] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:35 [I 2022-10-28 06:36:04] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00247 [I 2022-10-28 06:36:04] ...trap/engines/engine.py.230: loss: 0.12457 [I 2022-10-28 06:36:04] ...optimizers/trijoint.py.151: total_norm: 13.094374 [I 2022-10-28 06:36:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 550/2304 [I 2022-10-28 06:36:10] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:22 [I 2022-10-28 06:36:10] ...trap/engines/engine.py.229: process: 0.56482 | load: 0.00243 [I 2022-10-28 06:36:10] ...trap/engines/engine.py.230: loss: 0.13386 [I 2022-10-28 06:36:10] ...optimizers/trijoint.py.151: total_norm: 13.864529 [I 2022-10-28 06:36:15] ...trap/engines/engine.py.224: train: epoch 111 | batch 560/2304 [I 2022-10-28 06:36:15] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:20 [I 2022-10-28 06:36:15] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00243 [I 2022-10-28 06:36:15] ...trap/engines/engine.py.230: loss: 0.08979 [I 2022-10-28 06:36:15] ...optimizers/trijoint.py.151: total_norm: 15.435462 [I 2022-10-28 06:36:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 570/2304 [I 2022-10-28 06:36:21] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:17 [I 2022-10-28 06:36:21] ...trap/engines/engine.py.229: process: 0.56455 | load: 0.00271 [I 2022-10-28 06:36:21] ...trap/engines/engine.py.230: loss: 0.11706 [I 2022-10-28 06:36:21] ...optimizers/trijoint.py.151: total_norm: 27.482454 [I 2022-10-28 06:36:26] ...trap/engines/engine.py.224: train: epoch 111 | batch 580/2304 [I 2022-10-28 06:36:26] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:09 [I 2022-10-28 06:36:26] ...trap/engines/engine.py.229: process: 0.56292 | load: 0.00260 [I 2022-10-28 06:36:26] ...trap/engines/engine.py.230: loss: 0.09157 [I 2022-10-28 06:36:26] ...optimizers/trijoint.py.151: total_norm: 12.562901 [I 2022-10-28 06:36:32] ...trap/engines/engine.py.224: train: epoch 111 | batch 590/2304 [I 2022-10-28 06:36:32] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:16:00 [I 2022-10-28 06:36:32] ...trap/engines/engine.py.229: process: 0.57052 | load: 0.00255 [I 2022-10-28 06:36:32] ...trap/engines/engine.py.230: loss: 0.09788 [I 2022-10-28 06:36:32] ...optimizers/trijoint.py.151: total_norm: 14.410370 [I 2022-10-28 06:36:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 600/2304 [I 2022-10-28 06:36:38] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:48 [I 2022-10-28 06:36:38] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00249 [I 2022-10-28 06:36:38] ...trap/engines/engine.py.230: loss: 0.10031 [I 2022-10-28 06:36:38] ...optimizers/trijoint.py.151: total_norm: 36.226231 [I 2022-10-28 06:36:43] ...trap/engines/engine.py.224: train: epoch 111 | batch 610/2304 [I 2022-10-28 06:36:43] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:52 [I 2022-10-28 06:36:43] ...trap/engines/engine.py.229: process: 0.56432 | load: 0.00244 [I 2022-10-28 06:36:43] ...trap/engines/engine.py.230: loss: 0.07789 [I 2022-10-28 06:36:43] ...optimizers/trijoint.py.151: total_norm: 14.216997 [I 2022-10-28 06:36:49] ...trap/engines/engine.py.224: train: epoch 111 | batch 620/2304 [I 2022-10-28 06:36:49] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:50 [I 2022-10-28 06:36:49] ...trap/engines/engine.py.229: process: 0.56178 | load: 0.00243 [I 2022-10-28 06:36:49] ...trap/engines/engine.py.230: loss: 0.15861 [I 2022-10-28 06:36:49] ...optimizers/trijoint.py.151: total_norm: 21.174213 [I 2022-10-28 06:36:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 630/2304 [I 2022-10-28 06:36:55] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:37 [I 2022-10-28 06:36:55] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00248 [I 2022-10-28 06:36:55] ...trap/engines/engine.py.230: loss: 0.09288 [I 2022-10-28 06:36:55] ...optimizers/trijoint.py.151: total_norm: 9.272430 [I 2022-10-28 06:37:00] ...trap/engines/engine.py.224: train: epoch 111 | batch 640/2304 [I 2022-10-28 06:37:00] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:42 [I 2022-10-28 06:37:00] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00243 [I 2022-10-28 06:37:00] ...trap/engines/engine.py.230: loss: 0.14334 [I 2022-10-28 06:37:00] ...optimizers/trijoint.py.151: total_norm: 15.901673 [I 2022-10-28 06:37:06] ...trap/engines/engine.py.224: train: epoch 111 | batch 650/2304 [I 2022-10-28 06:37:06] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:29 [I 2022-10-28 06:37:06] ...trap/engines/engine.py.229: process: 0.56694 | load: 0.00269 [I 2022-10-28 06:37:06] ...trap/engines/engine.py.230: loss: 0.14027 [I 2022-10-28 06:37:06] ...optimizers/trijoint.py.151: total_norm: 48.454300 [I 2022-10-28 06:37:12] ...trap/engines/engine.py.224: train: epoch 111 | batch 660/2304 [I 2022-10-28 06:37:12] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:20 [I 2022-10-28 06:37:12] ...trap/engines/engine.py.229: process: 0.56370 | load: 0.00256 [I 2022-10-28 06:37:12] ...trap/engines/engine.py.230: loss: 0.11658 [I 2022-10-28 06:37:12] ...optimizers/trijoint.py.151: total_norm: 13.021369 [I 2022-10-28 06:37:17] ...trap/engines/engine.py.224: train: epoch 111 | batch 670/2304 [I 2022-10-28 06:37:17] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:21 [I 2022-10-28 06:37:17] ...trap/engines/engine.py.229: process: 0.56754 | load: 0.00250 [I 2022-10-28 06:37:17] ...trap/engines/engine.py.230: loss: 0.11058 [I 2022-10-28 06:37:17] ...optimizers/trijoint.py.151: total_norm: 12.158854 [I 2022-10-28 06:37:23] ...trap/engines/engine.py.224: train: epoch 111 | batch 680/2304 [I 2022-10-28 06:37:23] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:18 [I 2022-10-28 06:37:23] ...trap/engines/engine.py.229: process: 0.57095 | load: 0.00247 [I 2022-10-28 06:37:23] ...trap/engines/engine.py.230: loss: 0.15951 [I 2022-10-28 06:37:23] ...optimizers/trijoint.py.151: total_norm: 27.176140 [I 2022-10-28 06:37:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 690/2304 [I 2022-10-28 06:37:29] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:12 [I 2022-10-28 06:37:29] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00265 [I 2022-10-28 06:37:29] ...trap/engines/engine.py.230: loss: 0.10572 [I 2022-10-28 06:37:29] ...optimizers/trijoint.py.151: total_norm: 26.045689 [I 2022-10-28 06:37:34] ...trap/engines/engine.py.224: train: epoch 111 | batch 700/2304 [I 2022-10-28 06:37:34] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:15:12 [I 2022-10-28 06:37:34] ...trap/engines/engine.py.229: process: 0.55806 | load: 0.00252 [I 2022-10-28 06:37:34] ...trap/engines/engine.py.230: loss: 0.12684 [I 2022-10-28 06:37:34] ...optimizers/trijoint.py.151: total_norm: 21.002632 [I 2022-10-28 06:37:40] ...trap/engines/engine.py.224: train: epoch 111 | batch 710/2304 [I 2022-10-28 06:37:40] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:52 [I 2022-10-28 06:37:40] ...trap/engines/engine.py.229: process: 0.55404 | load: 0.00276 [I 2022-10-28 06:37:40] ...trap/engines/engine.py.230: loss: 0.09045 [I 2022-10-28 06:37:40] ...optimizers/trijoint.py.151: total_norm: 11.901294 [I 2022-10-28 06:37:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 720/2304 [I 2022-10-28 06:37:45] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:47 [I 2022-10-28 06:37:45] ...trap/engines/engine.py.229: process: 0.54946 | load: 0.00266 [I 2022-10-28 06:37:45] ...trap/engines/engine.py.230: loss: 0.08339 [I 2022-10-28 06:37:45] ...optimizers/trijoint.py.151: total_norm: 7.884812 [I 2022-10-28 06:37:51] ...trap/engines/engine.py.224: train: epoch 111 | batch 730/2304 [I 2022-10-28 06:37:51] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:41 [I 2022-10-28 06:37:51] ...trap/engines/engine.py.229: process: 0.55782 | load: 0.00258 [I 2022-10-28 06:37:51] ...trap/engines/engine.py.230: loss: 0.12171 [I 2022-10-28 06:37:51] ...optimizers/trijoint.py.151: total_norm: 47.011776 [I 2022-10-28 06:37:57] ...trap/engines/engine.py.224: train: epoch 111 | batch 740/2304 [I 2022-10-28 06:37:57] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:33 [I 2022-10-28 06:37:57] ...trap/engines/engine.py.229: process: 0.55880 | load: 0.00252 [I 2022-10-28 06:37:57] ...trap/engines/engine.py.230: loss: 0.10828 [I 2022-10-28 06:37:57] ...optimizers/trijoint.py.151: total_norm: 14.165465 [I 2022-10-28 06:38:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 750/2304 [I 2022-10-28 06:38:02] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:31 [I 2022-10-28 06:38:02] ...trap/engines/engine.py.229: process: 0.56597 | load: 0.00251 [I 2022-10-28 06:38:02] ...trap/engines/engine.py.230: loss: 0.09602 [I 2022-10-28 06:38:02] ...optimizers/trijoint.py.151: total_norm: 23.649036 [I 2022-10-28 06:38:08] ...trap/engines/engine.py.224: train: epoch 111 | batch 760/2304 [I 2022-10-28 06:38:08] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:20 [I 2022-10-28 06:38:08] ...trap/engines/engine.py.229: process: 0.55215 | load: 0.00250 [I 2022-10-28 06:38:08] ...trap/engines/engine.py.230: loss: 0.12398 [I 2022-10-28 06:38:08] ...optimizers/trijoint.py.151: total_norm: 24.626003 [I 2022-10-28 06:38:14] ...trap/engines/engine.py.224: train: epoch 111 | batch 770/2304 [I 2022-10-28 06:38:14] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:32 [I 2022-10-28 06:38:14] ...trap/engines/engine.py.229: process: 0.55938 | load: 0.00257 [I 2022-10-28 06:38:14] ...trap/engines/engine.py.230: loss: 0.10887 [I 2022-10-28 06:38:14] ...optimizers/trijoint.py.151: total_norm: 15.608050 [I 2022-10-28 06:38:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 780/2304 [I 2022-10-28 06:38:19] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:08 [I 2022-10-28 06:38:19] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00253 [I 2022-10-28 06:38:19] ...trap/engines/engine.py.230: loss: 0.09746 [I 2022-10-28 06:38:19] ...optimizers/trijoint.py.151: total_norm: 14.475517 [I 2022-10-28 06:38:25] ...trap/engines/engine.py.224: train: epoch 111 | batch 790/2304 [I 2022-10-28 06:38:25] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:05 [I 2022-10-28 06:38:25] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00246 [I 2022-10-28 06:38:25] ...trap/engines/engine.py.230: loss: 0.10091 [I 2022-10-28 06:38:25] ...optimizers/trijoint.py.151: total_norm: 17.553957 [I 2022-10-28 06:38:30] ...trap/engines/engine.py.224: train: epoch 111 | batch 800/2304 [I 2022-10-28 06:38:30] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:13:55 [I 2022-10-28 06:38:30] ...trap/engines/engine.py.229: process: 0.54819 | load: 0.00270 [I 2022-10-28 06:38:30] ...trap/engines/engine.py.230: loss: 0.19015 [I 2022-10-28 06:38:30] ...optimizers/trijoint.py.151: total_norm: 34.840649 [I 2022-10-28 06:38:36] ...trap/engines/engine.py.224: train: epoch 111 | batch 810/2304 [I 2022-10-28 06:38:36] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:46 [I 2022-10-28 06:38:36] ...trap/engines/engine.py.229: process: 0.55739 | load: 0.00248 [I 2022-10-28 06:38:36] ...trap/engines/engine.py.230: loss: 0.15999 [I 2022-10-28 06:38:36] ...optimizers/trijoint.py.151: total_norm: 11.305057 [I 2022-10-28 06:38:41] ...trap/engines/engine.py.224: train: epoch 111 | batch 820/2304 [I 2022-10-28 06:38:41] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:41 [I 2022-10-28 06:38:41] ...trap/engines/engine.py.229: process: 0.55269 | load: 0.00247 [I 2022-10-28 06:38:41] ...trap/engines/engine.py.230: loss: 0.15419 [I 2022-10-28 06:38:41] ...optimizers/trijoint.py.151: total_norm: 19.544807 [I 2022-10-28 06:38:47] ...trap/engines/engine.py.224: train: epoch 111 | batch 830/2304 [I 2022-10-28 06:38:47] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:46 [I 2022-10-28 06:38:47] ...trap/engines/engine.py.229: process: 0.56840 | load: 0.00250 [I 2022-10-28 06:38:47] ...trap/engines/engine.py.230: loss: 0.11371 [I 2022-10-28 06:38:47] ...optimizers/trijoint.py.151: total_norm: 22.800966 [I 2022-10-28 06:38:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 840/2304 [I 2022-10-28 06:38:53] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:31 [I 2022-10-28 06:38:53] ...trap/engines/engine.py.229: process: 0.54592 | load: 0.00247 [I 2022-10-28 06:38:53] ...trap/engines/engine.py.230: loss: 0.11895 [I 2022-10-28 06:38:53] ...optimizers/trijoint.py.151: total_norm: 18.715988 [I 2022-10-28 06:38:58] ...trap/engines/engine.py.224: train: epoch 111 | batch 850/2304 [I 2022-10-28 06:38:58] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:26 [I 2022-10-28 06:38:58] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00249 [I 2022-10-28 06:38:58] ...trap/engines/engine.py.230: loss: 0.10263 [I 2022-10-28 06:38:58] ...optimizers/trijoint.py.151: total_norm: 37.003403 [I 2022-10-28 06:39:04] ...trap/engines/engine.py.224: train: epoch 111 | batch 860/2304 [I 2022-10-28 06:39:04] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:32 [I 2022-10-28 06:39:04] ...trap/engines/engine.py.229: process: 0.57001 | load: 0.00242 [I 2022-10-28 06:39:04] ...trap/engines/engine.py.230: loss: 0.16962 [I 2022-10-28 06:39:04] ...optimizers/trijoint.py.151: total_norm: 84.929893 [I 2022-10-28 06:39:09] ...trap/engines/engine.py.224: train: epoch 111 | batch 870/2304 [I 2022-10-28 06:39:09] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:16 [I 2022-10-28 06:39:09] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00298 [I 2022-10-28 06:39:09] ...trap/engines/engine.py.230: loss: 0.10152 [I 2022-10-28 06:39:09] ...optimizers/trijoint.py.151: total_norm: 13.467367 [I 2022-10-28 06:39:15] ...trap/engines/engine.py.224: train: epoch 111 | batch 880/2304 [I 2022-10-28 06:39:15] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:20 [I 2022-10-28 06:39:15] ...trap/engines/engine.py.229: process: 0.56252 | load: 0.00272 [I 2022-10-28 06:39:15] ...trap/engines/engine.py.230: loss: 0.09410 [I 2022-10-28 06:39:15] ...optimizers/trijoint.py.151: total_norm: 21.561825 [I 2022-10-28 06:39:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 890/2304 [I 2022-10-28 06:39:21] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:16 [I 2022-10-28 06:39:21] ...trap/engines/engine.py.229: process: 0.55559 | load: 0.00264 [I 2022-10-28 06:39:21] ...trap/engines/engine.py.230: loss: 0.08602 [I 2022-10-28 06:39:21] ...optimizers/trijoint.py.151: total_norm: 30.993706 [I 2022-10-28 06:39:26] ...trap/engines/engine.py.224: train: epoch 111 | batch 900/2304 [I 2022-10-28 06:39:26] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:06 [I 2022-10-28 06:39:26] ...trap/engines/engine.py.229: process: 0.54888 | load: 0.00256 [I 2022-10-28 06:39:26] ...trap/engines/engine.py.230: loss: 0.09536 [I 2022-10-28 06:39:26] ...optimizers/trijoint.py.151: total_norm: 18.892429 [I 2022-10-28 06:39:32] ...trap/engines/engine.py.224: train: epoch 111 | batch 910/2304 [I 2022-10-28 06:39:32] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:53 [I 2022-10-28 06:39:32] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00247 [I 2022-10-28 06:39:32] ...trap/engines/engine.py.230: loss: 0.13134 [I 2022-10-28 06:39:32] ...optimizers/trijoint.py.151: total_norm: 25.298117 [I 2022-10-28 06:39:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 920/2304 [I 2022-10-28 06:39:37] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:50 [I 2022-10-28 06:39:37] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00252 [I 2022-10-28 06:39:37] ...trap/engines/engine.py.230: loss: 0.07725 [I 2022-10-28 06:39:37] ...optimizers/trijoint.py.151: total_norm: 17.283136 [I 2022-10-28 06:39:43] ...trap/engines/engine.py.224: train: epoch 111 | batch 930/2304 [I 2022-10-28 06:39:43] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:50 [I 2022-10-28 06:39:43] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00256 [I 2022-10-28 06:39:43] ...trap/engines/engine.py.230: loss: 0.10768 [I 2022-10-28 06:39:43] ...optimizers/trijoint.py.151: total_norm: 15.790790 [I 2022-10-28 06:39:49] ...trap/engines/engine.py.224: train: epoch 111 | batch 940/2304 [I 2022-10-28 06:39:49] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:39 [I 2022-10-28 06:39:49] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00261 [I 2022-10-28 06:39:49] ...trap/engines/engine.py.230: loss: 0.08992 [I 2022-10-28 06:39:49] ...optimizers/trijoint.py.151: total_norm: 9.474399 [I 2022-10-28 06:39:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 950/2304 [I 2022-10-28 06:39:54] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:34 [I 2022-10-28 06:39:54] ...trap/engines/engine.py.229: process: 0.55024 | load: 0.00247 [I 2022-10-28 06:39:54] ...trap/engines/engine.py.230: loss: 0.09219 [I 2022-10-28 06:39:54] ...optimizers/trijoint.py.151: total_norm: 22.903772 [I 2022-10-28 06:40:00] ...trap/engines/engine.py.224: train: epoch 111 | batch 960/2304 [I 2022-10-28 06:40:00] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:31 [I 2022-10-28 06:40:00] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00254 [I 2022-10-28 06:40:00] ...trap/engines/engine.py.230: loss: 0.12213 [I 2022-10-28 06:40:00] ...optimizers/trijoint.py.151: total_norm: 16.150347 [I 2022-10-28 06:40:05] ...trap/engines/engine.py.224: train: epoch 111 | batch 970/2304 [I 2022-10-28 06:40:05] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:23 [I 2022-10-28 06:40:05] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00265 [I 2022-10-28 06:40:05] ...trap/engines/engine.py.230: loss: 0.15842 [I 2022-10-28 06:40:05] ...optimizers/trijoint.py.151: total_norm: 32.120213 [I 2022-10-28 06:40:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 980/2304 [I 2022-10-28 06:40:11] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:25 [I 2022-10-28 06:40:11] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00256 [I 2022-10-28 06:40:11] ...trap/engines/engine.py.230: loss: 0.16139 [I 2022-10-28 06:40:11] ...optimizers/trijoint.py.151: total_norm: 24.412691 [I 2022-10-28 06:40:17] ...trap/engines/engine.py.224: train: epoch 111 | batch 990/2304 [I 2022-10-28 06:40:17] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:08 [I 2022-10-28 06:40:17] ...trap/engines/engine.py.229: process: 0.55337 | load: 0.00252 [I 2022-10-28 06:40:17] ...trap/engines/engine.py.230: loss: 0.15803 [I 2022-10-28 06:40:17] ...optimizers/trijoint.py.151: total_norm: 18.836252 [I 2022-10-28 06:40:22] ...trap/engines/engine.py.224: train: epoch 111 | batch 1000/2304 [I 2022-10-28 06:40:22] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:21 [I 2022-10-28 06:40:22] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00253 [I 2022-10-28 06:40:22] ...trap/engines/engine.py.230: loss: 0.15033 [I 2022-10-28 06:40:22] ...optimizers/trijoint.py.151: total_norm: 39.051907 [I 2022-10-28 06:40:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 1010/2304 [I 2022-10-28 06:40:28] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:02 [I 2022-10-28 06:40:28] ...trap/engines/engine.py.229: process: 0.56246 | load: 0.00251 [I 2022-10-28 06:40:28] ...trap/engines/engine.py.230: loss: 0.11853 [I 2022-10-28 06:40:28] ...optimizers/trijoint.py.151: total_norm: 14.725476 [I 2022-10-28 06:40:34] ...trap/engines/engine.py.224: train: epoch 111 | batch 1020/2304 [I 2022-10-28 06:40:34] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:59 [I 2022-10-28 06:40:34] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00248 [I 2022-10-28 06:40:34] ...trap/engines/engine.py.230: loss: 0.05486 [I 2022-10-28 06:40:34] ...optimizers/trijoint.py.151: total_norm: 12.293153 [I 2022-10-28 06:40:39] ...trap/engines/engine.py.224: train: epoch 111 | batch 1030/2304 [I 2022-10-28 06:40:39] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:47 [I 2022-10-28 06:40:39] ...trap/engines/engine.py.229: process: 0.55605 | load: 0.00251 [I 2022-10-28 06:40:39] ...trap/engines/engine.py.230: loss: 0.15398 [I 2022-10-28 06:40:39] ...optimizers/trijoint.py.151: total_norm: 12.264331 [I 2022-10-28 06:40:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 1040/2304 [I 2022-10-28 06:40:45] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:42 [I 2022-10-28 06:40:45] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00252 [I 2022-10-28 06:40:45] ...trap/engines/engine.py.230: loss: 0.11185 [I 2022-10-28 06:40:45] ...optimizers/trijoint.py.151: total_norm: 20.846172 [I 2022-10-28 06:40:50] ...trap/engines/engine.py.224: train: epoch 111 | batch 1050/2304 [I 2022-10-28 06:40:50] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:36 [I 2022-10-28 06:40:50] ...trap/engines/engine.py.229: process: 0.54899 | load: 0.00251 [I 2022-10-28 06:40:50] ...trap/engines/engine.py.230: loss: 0.10505 [I 2022-10-28 06:40:50] ...optimizers/trijoint.py.151: total_norm: 10.223068 [I 2022-10-28 06:40:56] ...trap/engines/engine.py.224: train: epoch 111 | batch 1060/2304 [I 2022-10-28 06:40:56] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:36 [I 2022-10-28 06:40:56] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00263 [I 2022-10-28 06:40:56] ...trap/engines/engine.py.230: loss: 0.11324 [I 2022-10-28 06:40:56] ...optimizers/trijoint.py.151: total_norm: 22.955734 [I 2022-10-28 06:41:01] ...trap/engines/engine.py.224: train: epoch 111 | batch 1070/2304 [I 2022-10-28 06:41:01] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:33 [I 2022-10-28 06:41:01] ...trap/engines/engine.py.229: process: 0.56795 | load: 0.00247 [I 2022-10-28 06:41:01] ...trap/engines/engine.py.230: loss: 0.14046 [I 2022-10-28 06:41:01] ...optimizers/trijoint.py.151: total_norm: 23.708719 [I 2022-10-28 06:41:07] ...trap/engines/engine.py.224: train: epoch 111 | batch 1080/2304 [I 2022-10-28 06:41:07] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:25 [I 2022-10-28 06:41:07] ...trap/engines/engine.py.229: process: 0.57113 | load: 0.00251 [I 2022-10-28 06:41:07] ...trap/engines/engine.py.230: loss: 0.08295 [I 2022-10-28 06:41:07] ...optimizers/trijoint.py.151: total_norm: 11.807152 [I 2022-10-28 06:41:13] ...trap/engines/engine.py.224: train: epoch 111 | batch 1090/2304 [I 2022-10-28 06:41:13] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:18 [I 2022-10-28 06:41:13] ...trap/engines/engine.py.229: process: 0.56535 | load: 0.00244 [I 2022-10-28 06:41:13] ...trap/engines/engine.py.230: loss: 0.14050 [I 2022-10-28 06:41:13] ...optimizers/trijoint.py.151: total_norm: 26.114645 [I 2022-10-28 06:41:18] ...trap/engines/engine.py.224: train: epoch 111 | batch 1100/2304 [I 2022-10-28 06:41:18] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:09 [I 2022-10-28 06:41:18] ...trap/engines/engine.py.229: process: 0.55029 | load: 0.00242 [I 2022-10-28 06:41:18] ...trap/engines/engine.py.230: loss: 0.16113 [I 2022-10-28 06:41:18] ...optimizers/trijoint.py.151: total_norm: 21.388144 [I 2022-10-28 06:41:24] ...trap/engines/engine.py.224: train: epoch 111 | batch 1110/2304 [I 2022-10-28 06:41:24] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:23 [I 2022-10-28 06:41:24] ...trap/engines/engine.py.229: process: 0.54999 | load: 0.00276 [I 2022-10-28 06:41:24] ...trap/engines/engine.py.230: loss: 0.20029 [I 2022-10-28 06:41:24] ...optimizers/trijoint.py.151: total_norm: 40.914478 [I 2022-10-28 06:41:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 1120/2304 [I 2022-10-28 06:41:29] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:02 [I 2022-10-28 06:41:29] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00249 [I 2022-10-28 06:41:29] ...trap/engines/engine.py.230: loss: 0.19605 [I 2022-10-28 06:41:29] ...optimizers/trijoint.py.151: total_norm: 25.263729 [I 2022-10-28 06:41:35] ...trap/engines/engine.py.224: train: epoch 111 | batch 1130/2304 [I 2022-10-28 06:41:35] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:11:01 [I 2022-10-28 06:41:35] ...trap/engines/engine.py.229: process: 0.56876 | load: 0.00246 [I 2022-10-28 06:41:35] ...trap/engines/engine.py.230: loss: 0.12612 [I 2022-10-28 06:41:35] ...optimizers/trijoint.py.151: total_norm: 27.391417 [I 2022-10-28 06:41:41] ...trap/engines/engine.py.224: train: epoch 111 | batch 1140/2304 [I 2022-10-28 06:41:41] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:10:48 [I 2022-10-28 06:41:41] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00244 [I 2022-10-28 06:41:41] ...trap/engines/engine.py.230: loss: 0.12773 [I 2022-10-28 06:41:41] ...optimizers/trijoint.py.151: total_norm: 12.050073 [I 2022-10-28 06:41:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 1150/2304 [I 2022-10-28 06:41:46] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:47 [I 2022-10-28 06:41:46] ...trap/engines/engine.py.229: process: 0.55461 | load: 0.00247 [I 2022-10-28 06:41:46] ...trap/engines/engine.py.230: loss: 0.13191 [I 2022-10-28 06:41:46] ...optimizers/trijoint.py.151: total_norm: 25.723072 [I 2022-10-28 06:41:52] ...trap/engines/engine.py.224: train: epoch 111 | batch 1160/2304 [I 2022-10-28 06:41:52] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:39 [I 2022-10-28 06:41:52] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00248 [I 2022-10-28 06:41:52] ...trap/engines/engine.py.230: loss: 0.10461 [I 2022-10-28 06:41:52] ...optimizers/trijoint.py.151: total_norm: 14.101897 [I 2022-10-28 06:41:58] ...trap/engines/engine.py.224: train: epoch 111 | batch 1170/2304 [I 2022-10-28 06:41:58] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:33 [I 2022-10-28 06:41:58] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00245 [I 2022-10-28 06:41:58] ...trap/engines/engine.py.230: loss: 0.12418 [I 2022-10-28 06:41:58] ...optimizers/trijoint.py.151: total_norm: 18.224115 [I 2022-10-28 06:42:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 1180/2304 [I 2022-10-28 06:42:03] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:25 [I 2022-10-28 06:42:03] ...trap/engines/engine.py.229: process: 0.55126 | load: 0.00249 [I 2022-10-28 06:42:03] ...trap/engines/engine.py.230: loss: 0.13057 [I 2022-10-28 06:42:03] ...optimizers/trijoint.py.151: total_norm: 17.994560 [I 2022-10-28 06:42:09] ...trap/engines/engine.py.224: train: epoch 111 | batch 1190/2304 [I 2022-10-28 06:42:09] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:16 [I 2022-10-28 06:42:09] ...trap/engines/engine.py.229: process: 0.55058 | load: 0.00243 [I 2022-10-28 06:42:09] ...trap/engines/engine.py.230: loss: 0.08793 [I 2022-10-28 06:42:09] ...optimizers/trijoint.py.151: total_norm: 11.682208 [I 2022-10-28 06:42:14] ...trap/engines/engine.py.224: train: epoch 111 | batch 1200/2304 [I 2022-10-28 06:42:14] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:16 [I 2022-10-28 06:42:14] ...trap/engines/engine.py.229: process: 0.56313 | load: 0.00265 [I 2022-10-28 06:42:14] ...trap/engines/engine.py.230: loss: 0.13440 [I 2022-10-28 06:42:14] ...optimizers/trijoint.py.151: total_norm: 12.567432 [I 2022-10-28 06:42:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 1210/2304 [I 2022-10-28 06:42:20] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:13 [I 2022-10-28 06:42:20] ...trap/engines/engine.py.229: process: 0.56363 | load: 0.00255 [I 2022-10-28 06:42:20] ...trap/engines/engine.py.230: loss: 0.15100 [I 2022-10-28 06:42:20] ...optimizers/trijoint.py.151: total_norm: 9.570972 [I 2022-10-28 06:42:25] ...trap/engines/engine.py.224: train: epoch 111 | batch 1220/2304 [I 2022-10-28 06:42:25] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:02 [I 2022-10-28 06:42:25] ...trap/engines/engine.py.229: process: 0.55621 | load: 0.00246 [I 2022-10-28 06:42:25] ...trap/engines/engine.py.230: loss: 0.07707 [I 2022-10-28 06:42:25] ...optimizers/trijoint.py.151: total_norm: 8.302836 [I 2022-10-28 06:42:31] ...trap/engines/engine.py.224: train: epoch 111 | batch 1230/2304 [I 2022-10-28 06:42:31] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:10:01 [I 2022-10-28 06:42:31] ...trap/engines/engine.py.229: process: 0.56484 | load: 0.00251 [I 2022-10-28 06:42:31] ...trap/engines/engine.py.230: loss: 0.09871 [I 2022-10-28 06:42:31] ...optimizers/trijoint.py.151: total_norm: 17.024719 [I 2022-10-28 06:42:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 1240/2304 [I 2022-10-28 06:42:37] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:51 [I 2022-10-28 06:42:37] ...trap/engines/engine.py.229: process: 0.56285 | load: 0.00249 [I 2022-10-28 06:42:37] ...trap/engines/engine.py.230: loss: 0.14168 [I 2022-10-28 06:42:37] ...optimizers/trijoint.py.151: total_norm: 38.512840 [I 2022-10-28 06:42:42] ...trap/engines/engine.py.224: train: epoch 111 | batch 1250/2304 [I 2022-10-28 06:42:42] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:45 [I 2022-10-28 06:42:42] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00251 [I 2022-10-28 06:42:42] ...trap/engines/engine.py.230: loss: 0.12720 [I 2022-10-28 06:42:42] ...optimizers/trijoint.py.151: total_norm: 17.961853 [I 2022-10-28 06:42:48] ...trap/engines/engine.py.224: train: epoch 111 | batch 1260/2304 [I 2022-10-28 06:42:48] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:42 [I 2022-10-28 06:42:48] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00251 [I 2022-10-28 06:42:48] ...trap/engines/engine.py.230: loss: 0.11914 [I 2022-10-28 06:42:48] ...optimizers/trijoint.py.151: total_norm: 18.867710 [I 2022-10-28 06:42:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 1270/2304 [I 2022-10-28 06:42:53] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:34 [I 2022-10-28 06:42:53] ...trap/engines/engine.py.229: process: 0.55172 | load: 0.00244 [I 2022-10-28 06:42:53] ...trap/engines/engine.py.230: loss: 0.09735 [I 2022-10-28 06:42:53] ...optimizers/trijoint.py.151: total_norm: 15.576062 [I 2022-10-28 06:42:59] ...trap/engines/engine.py.224: train: epoch 111 | batch 1280/2304 [I 2022-10-28 06:42:59] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:30 [I 2022-10-28 06:42:59] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00247 [I 2022-10-28 06:42:59] ...trap/engines/engine.py.230: loss: 0.11568 [I 2022-10-28 06:42:59] ...optimizers/trijoint.py.151: total_norm: 14.451164 [I 2022-10-28 06:43:05] ...trap/engines/engine.py.224: train: epoch 111 | batch 1290/2304 [I 2022-10-28 06:43:05] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:28 [I 2022-10-28 06:43:05] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00252 [I 2022-10-28 06:43:05] ...trap/engines/engine.py.230: loss: 0.15707 [I 2022-10-28 06:43:05] ...optimizers/trijoint.py.151: total_norm: 67.610924 [I 2022-10-28 06:43:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 1300/2304 [I 2022-10-28 06:43:10] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:19 [I 2022-10-28 06:43:10] ...trap/engines/engine.py.229: process: 0.55281 | load: 0.00248 [I 2022-10-28 06:43:10] ...trap/engines/engine.py.230: loss: 0.10624 [I 2022-10-28 06:43:10] ...optimizers/trijoint.py.151: total_norm: 11.426457 [I 2022-10-28 06:43:16] ...trap/engines/engine.py.224: train: epoch 111 | batch 1310/2304 [I 2022-10-28 06:43:16] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:18 [I 2022-10-28 06:43:16] ...trap/engines/engine.py.229: process: 0.56631 | load: 0.00248 [I 2022-10-28 06:43:16] ...trap/engines/engine.py.230: loss: 0.12723 [I 2022-10-28 06:43:16] ...optimizers/trijoint.py.151: total_norm: 14.200512 [I 2022-10-28 06:43:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 1320/2304 [I 2022-10-28 06:43:21] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:03 [I 2022-10-28 06:43:21] ...trap/engines/engine.py.229: process: 0.55037 | load: 0.00243 [I 2022-10-28 06:43:21] ...trap/engines/engine.py.230: loss: 0.09815 [I 2022-10-28 06:43:21] ...optimizers/trijoint.py.151: total_norm: 12.717625 [I 2022-10-28 06:43:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 1330/2304 [I 2022-10-28 06:43:27] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:02 [I 2022-10-28 06:43:27] ...trap/engines/engine.py.229: process: 0.54965 | load: 0.00251 [I 2022-10-28 06:43:27] ...trap/engines/engine.py.230: loss: 0.13096 [I 2022-10-28 06:43:27] ...optimizers/trijoint.py.151: total_norm: 14.306171 [I 2022-10-28 06:43:33] ...trap/engines/engine.py.224: train: epoch 111 | batch 1340/2304 [I 2022-10-28 06:43:33] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:08:57 [I 2022-10-28 06:43:33] ...trap/engines/engine.py.229: process: 0.56842 | load: 0.00269 [I 2022-10-28 06:43:33] ...trap/engines/engine.py.230: loss: 0.13263 [I 2022-10-28 06:43:33] ...optimizers/trijoint.py.151: total_norm: 40.086411 [I 2022-10-28 06:43:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 1350/2304 [I 2022-10-28 06:43:38] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:48 [I 2022-10-28 06:43:38] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00244 [I 2022-10-28 06:43:38] ...trap/engines/engine.py.230: loss: 0.08966 [I 2022-10-28 06:43:38] ...optimizers/trijoint.py.151: total_norm: 9.513096 [I 2022-10-28 06:43:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 1360/2304 [I 2022-10-28 06:43:44] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:43 [I 2022-10-28 06:43:44] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00240 [I 2022-10-28 06:43:44] ...trap/engines/engine.py.230: loss: 0.14922 [I 2022-10-28 06:43:44] ...optimizers/trijoint.py.151: total_norm: 19.309597 [I 2022-10-28 06:43:49] ...trap/engines/engine.py.224: train: epoch 111 | batch 1370/2304 [I 2022-10-28 06:43:49] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:39 [I 2022-10-28 06:43:49] ...trap/engines/engine.py.229: process: 0.56653 | load: 0.00256 [I 2022-10-28 06:43:49] ...trap/engines/engine.py.230: loss: 0.15286 [I 2022-10-28 06:43:49] ...optimizers/trijoint.py.151: total_norm: 9.644835 [I 2022-10-28 06:43:55] ...trap/engines/engine.py.224: train: epoch 111 | batch 1380/2304 [I 2022-10-28 06:43:55] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:32 [I 2022-10-28 06:43:55] ...trap/engines/engine.py.229: process: 0.55441 | load: 0.00242 [I 2022-10-28 06:43:55] ...trap/engines/engine.py.230: loss: 0.08578 [I 2022-10-28 06:43:55] ...optimizers/trijoint.py.151: total_norm: 23.832516 [I 2022-10-28 06:44:00] ...trap/engines/engine.py.224: train: epoch 111 | batch 1390/2304 [I 2022-10-28 06:44:00] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:26 [I 2022-10-28 06:44:00] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00252 [I 2022-10-28 06:44:00] ...trap/engines/engine.py.230: loss: 0.15184 [I 2022-10-28 06:44:00] ...optimizers/trijoint.py.151: total_norm: 10.137072 [I 2022-10-28 06:44:06] ...trap/engines/engine.py.224: train: epoch 111 | batch 1400/2304 [I 2022-10-28 06:44:06] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:20 [I 2022-10-28 06:44:06] ...trap/engines/engine.py.229: process: 0.55567 | load: 0.00246 [I 2022-10-28 06:44:06] ...trap/engines/engine.py.230: loss: 0.20024 [I 2022-10-28 06:44:06] ...optimizers/trijoint.py.151: total_norm: 18.401812 [I 2022-10-28 06:44:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 1410/2304 [I 2022-10-28 06:44:11] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:17 [I 2022-10-28 06:44:11] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00247 [I 2022-10-28 06:44:11] ...trap/engines/engine.py.230: loss: 0.13970 [I 2022-10-28 06:44:11] ...optimizers/trijoint.py.151: total_norm: 67.164581 [I 2022-10-28 06:44:17] ...trap/engines/engine.py.224: train: epoch 111 | batch 1420/2304 [I 2022-10-28 06:44:17] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:11 [I 2022-10-28 06:44:17] ...trap/engines/engine.py.229: process: 0.55533 | load: 0.00252 [I 2022-10-28 06:44:17] ...trap/engines/engine.py.230: loss: 0.11100 [I 2022-10-28 06:44:17] ...optimizers/trijoint.py.151: total_norm: 8.581218 [I 2022-10-28 06:44:23] ...trap/engines/engine.py.224: train: epoch 111 | batch 1430/2304 [I 2022-10-28 06:44:23] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:07 [I 2022-10-28 06:44:23] ...trap/engines/engine.py.229: process: 0.56401 | load: 0.00242 [I 2022-10-28 06:44:23] ...trap/engines/engine.py.230: loss: 0.18145 [I 2022-10-28 06:44:23] ...optimizers/trijoint.py.151: total_norm: 36.740585 [I 2022-10-28 06:44:28] ...trap/engines/engine.py.224: train: epoch 111 | batch 1440/2304 [I 2022-10-28 06:44:28] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:05 [I 2022-10-28 06:44:28] ...trap/engines/engine.py.229: process: 0.57664 | load: 0.00258 [I 2022-10-28 06:44:28] ...trap/engines/engine.py.230: loss: 0.14284 [I 2022-10-28 06:44:28] ...optimizers/trijoint.py.151: total_norm: 25.290031 [I 2022-10-28 06:44:34] ...trap/engines/engine.py.224: train: epoch 111 | batch 1450/2304 [I 2022-10-28 06:44:34] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:08:02 [I 2022-10-28 06:44:34] ...trap/engines/engine.py.229: process: 0.56458 | load: 0.00247 [I 2022-10-28 06:44:34] ...trap/engines/engine.py.230: loss: 0.11846 [I 2022-10-28 06:44:34] ...optimizers/trijoint.py.151: total_norm: 16.416704 [I 2022-10-28 06:44:39] ...trap/engines/engine.py.224: train: epoch 111 | batch 1460/2304 [I 2022-10-28 06:44:39] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:53 [I 2022-10-28 06:44:39] ...trap/engines/engine.py.229: process: 0.57030 | load: 0.00256 [I 2022-10-28 06:44:39] ...trap/engines/engine.py.230: loss: 0.08094 [I 2022-10-28 06:44:39] ...optimizers/trijoint.py.151: total_norm: 17.051119 [I 2022-10-28 06:44:45] ...trap/engines/engine.py.224: train: epoch 111 | batch 1470/2304 [I 2022-10-28 06:44:45] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:41 [I 2022-10-28 06:44:45] ...trap/engines/engine.py.229: process: 0.54944 | load: 0.00247 [I 2022-10-28 06:44:45] ...trap/engines/engine.py.230: loss: 0.13015 [I 2022-10-28 06:44:45] ...optimizers/trijoint.py.151: total_norm: 11.145943 [I 2022-10-28 06:44:51] ...trap/engines/engine.py.224: train: epoch 111 | batch 1480/2304 [I 2022-10-28 06:44:51] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:43 [I 2022-10-28 06:44:51] ...trap/engines/engine.py.229: process: 0.57145 | load: 0.00249 [I 2022-10-28 06:44:51] ...trap/engines/engine.py.230: loss: 0.16699 [I 2022-10-28 06:44:51] ...optimizers/trijoint.py.151: total_norm: 13.479891 [I 2022-10-28 06:44:56] ...trap/engines/engine.py.224: train: epoch 111 | batch 1490/2304 [I 2022-10-28 06:44:56] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:32 [I 2022-10-28 06:44:56] ...trap/engines/engine.py.229: process: 0.55178 | load: 0.00251 [I 2022-10-28 06:44:56] ...trap/engines/engine.py.230: loss: 0.14017 [I 2022-10-28 06:44:56] ...optimizers/trijoint.py.151: total_norm: 13.534501 [I 2022-10-28 06:45:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 1500/2304 [I 2022-10-28 06:45:02] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:30 [I 2022-10-28 06:45:02] ...trap/engines/engine.py.229: process: 0.57675 | load: 0.00258 [I 2022-10-28 06:45:02] ...trap/engines/engine.py.230: loss: 0.12654 [I 2022-10-28 06:45:02] ...optimizers/trijoint.py.151: total_norm: 15.056811 [I 2022-10-28 06:45:07] ...trap/engines/engine.py.224: train: epoch 111 | batch 1510/2304 [I 2022-10-28 06:45:07] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:24 [I 2022-10-28 06:45:07] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00256 [I 2022-10-28 06:45:07] ...trap/engines/engine.py.230: loss: 0.16874 [I 2022-10-28 06:45:07] ...optimizers/trijoint.py.151: total_norm: 10.886540 [I 2022-10-28 06:45:13] ...trap/engines/engine.py.224: train: epoch 111 | batch 1520/2304 [I 2022-10-28 06:45:13] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:20 [I 2022-10-28 06:45:13] ...trap/engines/engine.py.229: process: 0.57779 | load: 0.00255 [I 2022-10-28 06:45:13] ...trap/engines/engine.py.230: loss: 0.09314 [I 2022-10-28 06:45:13] ...optimizers/trijoint.py.151: total_norm: 16.766603 [I 2022-10-28 06:45:19] ...trap/engines/engine.py.224: train: epoch 111 | batch 1530/2304 [I 2022-10-28 06:45:19] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:13 [I 2022-10-28 06:45:19] ...trap/engines/engine.py.229: process: 0.55106 | load: 0.00252 [I 2022-10-28 06:45:19] ...trap/engines/engine.py.230: loss: 0.08263 [I 2022-10-28 06:45:19] ...optimizers/trijoint.py.151: total_norm: 9.578650 [I 2022-10-28 06:45:24] ...trap/engines/engine.py.224: train: epoch 111 | batch 1540/2304 [I 2022-10-28 06:45:24] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:05 [I 2022-10-28 06:45:24] ...trap/engines/engine.py.229: process: 0.54293 | load: 0.00253 [I 2022-10-28 06:45:24] ...trap/engines/engine.py.230: loss: 0.09835 [I 2022-10-28 06:45:24] ...optimizers/trijoint.py.151: total_norm: 9.835674 [I 2022-10-28 06:45:30] ...trap/engines/engine.py.224: train: epoch 111 | batch 1550/2304 [I 2022-10-28 06:45:30] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:06:57 [I 2022-10-28 06:45:30] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00271 [I 2022-10-28 06:45:30] ...trap/engines/engine.py.230: loss: 0.11312 [I 2022-10-28 06:45:30] ...optimizers/trijoint.py.151: total_norm: 19.234762 [I 2022-10-28 06:45:35] ...trap/engines/engine.py.224: train: epoch 111 | batch 1560/2304 [I 2022-10-28 06:45:35] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:52 [I 2022-10-28 06:45:35] ...trap/engines/engine.py.229: process: 0.55333 | load: 0.00259 [I 2022-10-28 06:45:35] ...trap/engines/engine.py.230: loss: 0.09213 [I 2022-10-28 06:45:35] ...optimizers/trijoint.py.151: total_norm: 10.157722 [I 2022-10-28 06:45:41] ...trap/engines/engine.py.224: train: epoch 111 | batch 1570/2304 [I 2022-10-28 06:45:41] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:46 [I 2022-10-28 06:45:41] ...trap/engines/engine.py.229: process: 0.55868 | load: 0.00251 [I 2022-10-28 06:45:41] ...trap/engines/engine.py.230: loss: 0.09001 [I 2022-10-28 06:45:41] ...optimizers/trijoint.py.151: total_norm: 12.572870 [I 2022-10-28 06:45:47] ...trap/engines/engine.py.224: train: epoch 111 | batch 1580/2304 [I 2022-10-28 06:45:47] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:41 [I 2022-10-28 06:45:47] ...trap/engines/engine.py.229: process: 0.54232 | load: 0.00243 [I 2022-10-28 06:45:47] ...trap/engines/engine.py.230: loss: 0.11086 [I 2022-10-28 06:45:47] ...optimizers/trijoint.py.151: total_norm: 14.700718 [I 2022-10-28 06:45:52] ...trap/engines/engine.py.224: train: epoch 111 | batch 1590/2304 [I 2022-10-28 06:45:52] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:40 [I 2022-10-28 06:45:52] ...trap/engines/engine.py.229: process: 0.55880 | load: 0.00244 [I 2022-10-28 06:45:52] ...trap/engines/engine.py.230: loss: 0.14987 [I 2022-10-28 06:45:52] ...optimizers/trijoint.py.151: total_norm: 18.656841 [I 2022-10-28 06:45:58] ...trap/engines/engine.py.224: train: epoch 111 | batch 1600/2304 [I 2022-10-28 06:45:58] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:35 [I 2022-10-28 06:45:58] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00242 [I 2022-10-28 06:45:58] ...trap/engines/engine.py.230: loss: 0.16072 [I 2022-10-28 06:45:58] ...optimizers/trijoint.py.151: total_norm: 29.006859 [I 2022-10-28 06:46:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 1610/2304 [I 2022-10-28 06:46:03] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:25 [I 2022-10-28 06:46:03] ...trap/engines/engine.py.229: process: 0.55069 | load: 0.00246 [I 2022-10-28 06:46:03] ...trap/engines/engine.py.230: loss: 0.15218 [I 2022-10-28 06:46:03] ...optimizers/trijoint.py.151: total_norm: 7.905796 [I 2022-10-28 06:46:09] ...trap/engines/engine.py.224: train: epoch 111 | batch 1620/2304 [I 2022-10-28 06:46:09] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:20 [I 2022-10-28 06:46:09] ...trap/engines/engine.py.229: process: 0.55928 | load: 0.00251 [I 2022-10-28 06:46:09] ...trap/engines/engine.py.230: loss: 0.17636 [I 2022-10-28 06:46:09] ...optimizers/trijoint.py.151: total_norm: 16.144606 [I 2022-10-28 06:46:14] ...trap/engines/engine.py.224: train: epoch 111 | batch 1630/2304 [I 2022-10-28 06:46:14] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:13 [I 2022-10-28 06:46:14] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00246 [I 2022-10-28 06:46:14] ...trap/engines/engine.py.230: loss: 0.17841 [I 2022-10-28 06:46:14] ...optimizers/trijoint.py.151: total_norm: 18.384605 [I 2022-10-28 06:46:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 1640/2304 [I 2022-10-28 06:46:20] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:09 [I 2022-10-28 06:46:20] ...trap/engines/engine.py.229: process: 0.55780 | load: 0.00242 [I 2022-10-28 06:46:20] ...trap/engines/engine.py.230: loss: 0.12888 [I 2022-10-28 06:46:20] ...optimizers/trijoint.py.151: total_norm: 19.815941 [I 2022-10-28 06:46:26] ...trap/engines/engine.py.224: train: epoch 111 | batch 1650/2304 [I 2022-10-28 06:46:26] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:01 [I 2022-10-28 06:46:26] ...trap/engines/engine.py.229: process: 0.54609 | load: 0.00252 [I 2022-10-28 06:46:26] ...trap/engines/engine.py.230: loss: 0.14044 [I 2022-10-28 06:46:26] ...optimizers/trijoint.py.151: total_norm: 17.909191 [I 2022-10-28 06:46:31] ...trap/engines/engine.py.224: train: epoch 111 | batch 1660/2304 [I 2022-10-28 06:46:31] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:58 [I 2022-10-28 06:46:31] ...trap/engines/engine.py.229: process: 0.54789 | load: 0.00240 [I 2022-10-28 06:46:31] ...trap/engines/engine.py.230: loss: 0.08645 [I 2022-10-28 06:46:31] ...optimizers/trijoint.py.151: total_norm: 10.068260 [I 2022-10-28 06:46:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 1670/2304 [I 2022-10-28 06:46:37] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:05:55 [I 2022-10-28 06:46:37] ...trap/engines/engine.py.229: process: 0.56669 | load: 0.00244 [I 2022-10-28 06:46:37] ...trap/engines/engine.py.230: loss: 0.11204 [I 2022-10-28 06:46:37] ...optimizers/trijoint.py.151: total_norm: 19.042728 [I 2022-10-28 06:46:42] ...trap/engines/engine.py.224: train: epoch 111 | batch 1680/2304 [I 2022-10-28 06:46:42] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:49 [I 2022-10-28 06:46:42] ...trap/engines/engine.py.229: process: 0.57267 | load: 0.00271 [I 2022-10-28 06:46:42] ...trap/engines/engine.py.230: loss: 0.10402 [I 2022-10-28 06:46:42] ...optimizers/trijoint.py.151: total_norm: 13.943020 [I 2022-10-28 06:46:48] ...trap/engines/engine.py.224: train: epoch 111 | batch 1690/2304 [I 2022-10-28 06:46:48] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:45 [I 2022-10-28 06:46:48] ...trap/engines/engine.py.229: process: 0.56618 | load: 0.00246 [I 2022-10-28 06:46:48] ...trap/engines/engine.py.230: loss: 0.11589 [I 2022-10-28 06:46:48] ...optimizers/trijoint.py.151: total_norm: 33.414646 [I 2022-10-28 06:46:54] ...trap/engines/engine.py.224: train: epoch 111 | batch 1700/2304 [I 2022-10-28 06:46:54] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:39 [I 2022-10-28 06:46:54] ...trap/engines/engine.py.229: process: 0.56923 | load: 0.00244 [I 2022-10-28 06:46:54] ...trap/engines/engine.py.230: loss: 0.12351 [I 2022-10-28 06:46:54] ...optimizers/trijoint.py.151: total_norm: 12.335010 [I 2022-10-28 06:46:59] ...trap/engines/engine.py.224: train: epoch 111 | batch 1710/2304 [I 2022-10-28 06:46:59] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:34 [I 2022-10-28 06:46:59] ...trap/engines/engine.py.229: process: 0.54871 | load: 0.00254 [I 2022-10-28 06:46:59] ...trap/engines/engine.py.230: loss: 0.12146 [I 2022-10-28 06:46:59] ...optimizers/trijoint.py.151: total_norm: 11.701339 [I 2022-10-28 06:47:05] ...trap/engines/engine.py.224: train: epoch 111 | batch 1720/2304 [I 2022-10-28 06:47:05] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:25 [I 2022-10-28 06:47:05] ...trap/engines/engine.py.229: process: 0.56404 | load: 0.00254 [I 2022-10-28 06:47:05] ...trap/engines/engine.py.230: loss: 0.11167 [I 2022-10-28 06:47:05] ...optimizers/trijoint.py.151: total_norm: 13.996548 [I 2022-10-28 06:47:11] ...trap/engines/engine.py.224: train: epoch 111 | batch 1730/2304 [I 2022-10-28 06:47:11] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:20 [I 2022-10-28 06:47:11] ...trap/engines/engine.py.229: process: 0.55814 | load: 0.00255 [I 2022-10-28 06:47:11] ...trap/engines/engine.py.230: loss: 0.14929 [I 2022-10-28 06:47:11] ...optimizers/trijoint.py.151: total_norm: 10.473206 [I 2022-10-28 06:47:16] ...trap/engines/engine.py.224: train: epoch 111 | batch 1740/2304 [I 2022-10-28 06:47:16] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:17 [I 2022-10-28 06:47:16] ...trap/engines/engine.py.229: process: 0.58713 | load: 0.00255 [I 2022-10-28 06:47:16] ...trap/engines/engine.py.230: loss: 0.13983 [I 2022-10-28 06:47:16] ...optimizers/trijoint.py.151: total_norm: 16.743774 [I 2022-10-28 06:47:22] ...trap/engines/engine.py.224: train: epoch 111 | batch 1750/2304 [I 2022-10-28 06:47:22] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:10 [I 2022-10-28 06:47:22] ...trap/engines/engine.py.229: process: 0.54899 | load: 0.00251 [I 2022-10-28 06:47:22] ...trap/engines/engine.py.230: loss: 0.13895 [I 2022-10-28 06:47:22] ...optimizers/trijoint.py.151: total_norm: 48.841232 [I 2022-10-28 06:47:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 1760/2304 [I 2022-10-28 06:47:27] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:11 [I 2022-10-28 06:47:27] ...trap/engines/engine.py.229: process: 0.59419 | load: 0.00252 [I 2022-10-28 06:47:28] ...trap/engines/engine.py.230: loss: 0.15486 [I 2022-10-28 06:47:28] ...optimizers/trijoint.py.151: total_norm: 67.550850 [I 2022-10-28 06:47:33] ...trap/engines/engine.py.224: train: epoch 111 | batch 1770/2304 [I 2022-10-28 06:47:33] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:05:09 [I 2022-10-28 06:47:33] ...trap/engines/engine.py.229: process: 0.54717 | load: 0.00254 [I 2022-10-28 06:47:33] ...trap/engines/engine.py.230: loss: 0.22732 [I 2022-10-28 06:47:33] ...optimizers/trijoint.py.151: total_norm: 16.672810 [I 2022-10-28 06:47:39] ...trap/engines/engine.py.224: train: epoch 111 | batch 1780/2304 [I 2022-10-28 06:47:39] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:56 [I 2022-10-28 06:47:39] ...trap/engines/engine.py.229: process: 0.57337 | load: 0.00251 [I 2022-10-28 06:47:39] ...trap/engines/engine.py.230: loss: 0.11490 [I 2022-10-28 06:47:39] ...optimizers/trijoint.py.151: total_norm: 10.611462 [I 2022-10-28 06:47:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 1790/2304 [I 2022-10-28 06:47:44] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:47 [I 2022-10-28 06:47:44] ...trap/engines/engine.py.229: process: 0.55133 | load: 0.00250 [I 2022-10-28 06:47:44] ...trap/engines/engine.py.230: loss: 0.09467 [I 2022-10-28 06:47:44] ...optimizers/trijoint.py.151: total_norm: 14.046675 [I 2022-10-28 06:47:50] ...trap/engines/engine.py.224: train: epoch 111 | batch 1800/2304 [I 2022-10-28 06:47:50] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:42 [I 2022-10-28 06:47:50] ...trap/engines/engine.py.229: process: 0.56202 | load: 0.00244 [I 2022-10-28 06:47:50] ...trap/engines/engine.py.230: loss: 0.12323 [I 2022-10-28 06:47:50] ...optimizers/trijoint.py.151: total_norm: 10.052052 [I 2022-10-28 06:47:56] ...trap/engines/engine.py.224: train: epoch 111 | batch 1810/2304 [I 2022-10-28 06:47:56] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:36 [I 2022-10-28 06:47:56] ...trap/engines/engine.py.229: process: 0.54923 | load: 0.00251 [I 2022-10-28 06:47:56] ...trap/engines/engine.py.230: loss: 0.10036 [I 2022-10-28 06:47:56] ...optimizers/trijoint.py.151: total_norm: 15.345319 [I 2022-10-28 06:48:01] ...trap/engines/engine.py.224: train: epoch 111 | batch 1820/2304 [I 2022-10-28 06:48:01] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:30 [I 2022-10-28 06:48:01] ...trap/engines/engine.py.229: process: 0.56528 | load: 0.00245 [I 2022-10-28 06:48:01] ...trap/engines/engine.py.230: loss: 0.07498 [I 2022-10-28 06:48:01] ...optimizers/trijoint.py.151: total_norm: 12.225981 [I 2022-10-28 06:48:07] ...trap/engines/engine.py.224: train: epoch 111 | batch 1830/2304 [I 2022-10-28 06:48:07] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:23 [I 2022-10-28 06:48:07] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00250 [I 2022-10-28 06:48:07] ...trap/engines/engine.py.230: loss: 0.14742 [I 2022-10-28 06:48:07] ...optimizers/trijoint.py.151: total_norm: 16.780388 [I 2022-10-28 06:48:13] ...trap/engines/engine.py.224: train: epoch 111 | batch 1840/2304 [I 2022-10-28 06:48:13] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:22 [I 2022-10-28 06:48:13] ...trap/engines/engine.py.229: process: 0.55339 | load: 0.00252 [I 2022-10-28 06:48:13] ...trap/engines/engine.py.230: loss: 0.10019 [I 2022-10-28 06:48:13] ...optimizers/trijoint.py.151: total_norm: 15.382339 [I 2022-10-28 06:48:18] ...trap/engines/engine.py.224: train: epoch 111 | batch 1850/2304 [I 2022-10-28 06:48:18] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:11 [I 2022-10-28 06:48:18] ...trap/engines/engine.py.229: process: 0.54793 | load: 0.00245 [I 2022-10-28 06:48:18] ...trap/engines/engine.py.230: loss: 0.12576 [I 2022-10-28 06:48:18] ...optimizers/trijoint.py.151: total_norm: 14.250118 [I 2022-10-28 06:48:24] ...trap/engines/engine.py.224: train: epoch 111 | batch 1860/2304 [I 2022-10-28 06:48:24] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:06 [I 2022-10-28 06:48:24] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00251 [I 2022-10-28 06:48:24] ...trap/engines/engine.py.230: loss: 0.06688 [I 2022-10-28 06:48:24] ...optimizers/trijoint.py.151: total_norm: 9.114706 [I 2022-10-28 06:48:29] ...trap/engines/engine.py.224: train: epoch 111 | batch 1870/2304 [I 2022-10-28 06:48:29] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:01 [I 2022-10-28 06:48:29] ...trap/engines/engine.py.229: process: 0.55583 | load: 0.00251 [I 2022-10-28 06:48:29] ...trap/engines/engine.py.230: loss: 0.07542 [I 2022-10-28 06:48:29] ...optimizers/trijoint.py.151: total_norm: 14.668263 [I 2022-10-28 06:48:35] ...trap/engines/engine.py.224: train: epoch 111 | batch 1880/2304 [I 2022-10-28 06:48:35] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:03:56 [I 2022-10-28 06:48:35] ...trap/engines/engine.py.229: process: 0.56607 | load: 0.00246 [I 2022-10-28 06:48:35] ...trap/engines/engine.py.230: loss: 0.14138 [I 2022-10-28 06:48:35] ...optimizers/trijoint.py.151: total_norm: 13.251959 [I 2022-10-28 06:48:40] ...trap/engines/engine.py.224: train: epoch 111 | batch 1890/2304 [I 2022-10-28 06:48:40] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:50 [I 2022-10-28 06:48:40] ...trap/engines/engine.py.229: process: 0.54807 | load: 0.00244 [I 2022-10-28 06:48:40] ...trap/engines/engine.py.230: loss: 0.14831 [I 2022-10-28 06:48:40] ...optimizers/trijoint.py.151: total_norm: 17.655163 [I 2022-10-28 06:48:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 1900/2304 [I 2022-10-28 06:48:46] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:42 [I 2022-10-28 06:48:46] ...trap/engines/engine.py.229: process: 0.53960 | load: 0.00248 [I 2022-10-28 06:48:46] ...trap/engines/engine.py.230: loss: 0.10133 [I 2022-10-28 06:48:46] ...optimizers/trijoint.py.151: total_norm: 7.018186 [I 2022-10-28 06:48:52] ...trap/engines/engine.py.224: train: epoch 111 | batch 1910/2304 [I 2022-10-28 06:48:52] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:41 [I 2022-10-28 06:48:52] ...trap/engines/engine.py.229: process: 0.56851 | load: 0.00250 [I 2022-10-28 06:48:52] ...trap/engines/engine.py.230: loss: 0.09276 [I 2022-10-28 06:48:52] ...optimizers/trijoint.py.151: total_norm: 14.198888 [I 2022-10-28 06:48:57] ...trap/engines/engine.py.224: train: epoch 111 | batch 1920/2304 [I 2022-10-28 06:48:57] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:34 [I 2022-10-28 06:48:57] ...trap/engines/engine.py.229: process: 0.56823 | load: 0.00257 [I 2022-10-28 06:48:57] ...trap/engines/engine.py.230: loss: 0.11476 [I 2022-10-28 06:48:57] ...optimizers/trijoint.py.151: total_norm: 19.140120 [I 2022-10-28 06:49:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 1930/2304 [I 2022-10-28 06:49:03] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:29 [I 2022-10-28 06:49:03] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00251 [I 2022-10-28 06:49:03] ...trap/engines/engine.py.230: loss: 0.11235 [I 2022-10-28 06:49:03] ...optimizers/trijoint.py.151: total_norm: 22.397152 [I 2022-10-28 06:49:08] ...trap/engines/engine.py.224: train: epoch 111 | batch 1940/2304 [I 2022-10-28 06:49:08] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:23 [I 2022-10-28 06:49:08] ...trap/engines/engine.py.229: process: 0.56268 | load: 0.00252 [I 2022-10-28 06:49:08] ...trap/engines/engine.py.230: loss: 0.10610 [I 2022-10-28 06:49:08] ...optimizers/trijoint.py.151: total_norm: 12.357385 [I 2022-10-28 06:49:14] ...trap/engines/engine.py.224: train: epoch 111 | batch 1950/2304 [I 2022-10-28 06:49:14] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:17 [I 2022-10-28 06:49:14] ...trap/engines/engine.py.229: process: 0.56671 | load: 0.00252 [I 2022-10-28 06:49:14] ...trap/engines/engine.py.230: loss: 0.11840 [I 2022-10-28 06:49:14] ...optimizers/trijoint.py.151: total_norm: 11.865521 [I 2022-10-28 06:49:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 1960/2304 [I 2022-10-28 06:49:20] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:19 [I 2022-10-28 06:49:20] ...trap/engines/engine.py.229: process: 0.56517 | load: 0.00267 [I 2022-10-28 06:49:20] ...trap/engines/engine.py.230: loss: 0.12103 [I 2022-10-28 06:49:20] ...optimizers/trijoint.py.151: total_norm: 16.130796 [I 2022-10-28 06:49:25] ...trap/engines/engine.py.224: train: epoch 111 | batch 1970/2304 [I 2022-10-28 06:49:25] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:07 [I 2022-10-28 06:49:25] ...trap/engines/engine.py.229: process: 0.56526 | load: 0.00256 [I 2022-10-28 06:49:25] ...trap/engines/engine.py.230: loss: 0.11213 [I 2022-10-28 06:49:25] ...optimizers/trijoint.py.151: total_norm: 16.367918 [I 2022-10-28 06:49:31] ...trap/engines/engine.py.224: train: epoch 111 | batch 1980/2304 [I 2022-10-28 06:49:31] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:02 [I 2022-10-28 06:49:31] ...trap/engines/engine.py.229: process: 0.55575 | load: 0.00253 [I 2022-10-28 06:49:31] ...trap/engines/engine.py.230: loss: 0.13050 [I 2022-10-28 06:49:31] ...optimizers/trijoint.py.151: total_norm: 14.028858 [I 2022-10-28 06:49:56] ...trap/engines/engine.py.224: train: epoch 111 | batch 1990/2304 [I 2022-10-28 06:49:56] ...trap/engines/engine.py.228: elapsed: 0:18:56 | left: 0:19:21 [I 2022-10-28 06:49:56] ...trap/engines/engine.py.229: process: 0.54287 | load: 0.00238 [I 2022-10-28 06:49:56] ...trap/engines/engine.py.230: loss: 0.09167 [I 2022-10-28 06:49:56] ...optimizers/trijoint.py.151: total_norm: 13.832325 [I 2022-10-28 06:50:02] ...trap/engines/engine.py.224: train: epoch 111 | batch 2000/2304 [I 2022-10-28 06:50:02] ...trap/engines/engine.py.228: elapsed: 0:19:02 | left: 0:04:33 [I 2022-10-28 06:50:02] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00240 [I 2022-10-28 06:50:02] ...trap/engines/engine.py.230: loss: 0.10780 [I 2022-10-28 06:50:02] ...optimizers/trijoint.py.151: total_norm: 21.050724 [I 2022-10-28 06:50:07] ...trap/engines/engine.py.224: train: epoch 111 | batch 2010/2304 [I 2022-10-28 06:50:07] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:55 [I 2022-10-28 06:50:07] ...trap/engines/engine.py.229: process: 0.55744 | load: 0.00235 [I 2022-10-28 06:50:07] ...trap/engines/engine.py.230: loss: 0.17060 [I 2022-10-28 06:50:07] ...optimizers/trijoint.py.151: total_norm: 12.796011 [I 2022-10-28 06:50:13] ...trap/engines/engine.py.224: train: epoch 111 | batch 2020/2304 [I 2022-10-28 06:50:13] ...trap/engines/engine.py.228: elapsed: 0:19:13 | left: 0:02:39 [I 2022-10-28 06:50:13] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00238 [I 2022-10-28 06:50:13] ...trap/engines/engine.py.230: loss: 0.11390 [I 2022-10-28 06:50:13] ...optimizers/trijoint.py.151: total_norm: 13.499740 [I 2022-10-28 06:50:18] ...trap/engines/engine.py.224: train: epoch 111 | batch 2030/2304 [I 2022-10-28 06:50:18] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:31 [I 2022-10-28 06:50:18] ...trap/engines/engine.py.229: process: 0.53829 | load: 0.00256 [I 2022-10-28 06:50:18] ...trap/engines/engine.py.230: loss: 0.08619 [I 2022-10-28 06:50:18] ...optimizers/trijoint.py.151: total_norm: 9.697851 [I 2022-10-28 06:50:24] ...trap/engines/engine.py.224: train: epoch 111 | batch 2040/2304 [I 2022-10-28 06:50:24] ...trap/engines/engine.py.228: elapsed: 0:19:24 | left: 0:02:29 [I 2022-10-28 06:50:24] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00249 [I 2022-10-28 06:50:24] ...trap/engines/engine.py.230: loss: 0.11641 [I 2022-10-28 06:50:24] ...optimizers/trijoint.py.151: total_norm: 18.280512 [I 2022-10-28 06:50:30] ...trap/engines/engine.py.224: train: epoch 111 | batch 2050/2304 [I 2022-10-28 06:50:30] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:02:20 [I 2022-10-28 06:50:30] ...trap/engines/engine.py.229: process: 0.54367 | load: 0.00241 [I 2022-10-28 06:50:30] ...trap/engines/engine.py.230: loss: 0.15565 [I 2022-10-28 06:50:30] ...optimizers/trijoint.py.151: total_norm: 20.540977 [I 2022-10-28 06:50:35] ...trap/engines/engine.py.224: train: epoch 111 | batch 2060/2304 [I 2022-10-28 06:50:35] ...trap/engines/engine.py.228: elapsed: 0:19:35 | left: 0:02:14 [I 2022-10-28 06:50:35] ...trap/engines/engine.py.229: process: 0.53782 | load: 0.00245 [I 2022-10-28 06:50:35] ...trap/engines/engine.py.230: loss: 0.09701 [I 2022-10-28 06:50:35] ...optimizers/trijoint.py.151: total_norm: 10.002562 [I 2022-10-28 06:50:41] ...trap/engines/engine.py.224: train: epoch 111 | batch 2070/2304 [I 2022-10-28 06:50:41] ...trap/engines/engine.py.228: elapsed: 0:19:41 | left: 0:02:09 [I 2022-10-28 06:50:41] ...trap/engines/engine.py.229: process: 0.54781 | load: 0.00240 [I 2022-10-28 06:50:41] ...trap/engines/engine.py.230: loss: 0.10114 [I 2022-10-28 06:50:41] ...optimizers/trijoint.py.151: total_norm: 30.725075 [I 2022-10-28 06:50:46] ...trap/engines/engine.py.224: train: epoch 111 | batch 2080/2304 [I 2022-10-28 06:50:46] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:02:04 [I 2022-10-28 06:50:46] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00255 [I 2022-10-28 06:50:46] ...trap/engines/engine.py.230: loss: 0.11311 [I 2022-10-28 06:50:46] ...optimizers/trijoint.py.151: total_norm: 17.934443 [I 2022-10-28 06:50:52] ...trap/engines/engine.py.224: train: epoch 111 | batch 2090/2304 [I 2022-10-28 06:50:52] ...trap/engines/engine.py.228: elapsed: 0:19:52 | left: 0:01:59 [I 2022-10-28 06:50:52] ...trap/engines/engine.py.229: process: 0.56507 | load: 0.00254 [I 2022-10-28 06:50:52] ...trap/engines/engine.py.230: loss: 0.13670 [I 2022-10-28 06:50:52] ...optimizers/trijoint.py.151: total_norm: 11.865108 [I 2022-10-28 06:50:57] ...trap/engines/engine.py.224: train: epoch 111 | batch 2100/2304 [I 2022-10-28 06:50:58] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:53 [I 2022-10-28 06:50:58] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00272 [I 2022-10-28 06:50:58] ...trap/engines/engine.py.230: loss: 0.14575 [I 2022-10-28 06:50:58] ...optimizers/trijoint.py.151: total_norm: 14.914657 [I 2022-10-28 06:51:03] ...trap/engines/engine.py.224: train: epoch 111 | batch 2110/2304 [I 2022-10-28 06:51:03] ...trap/engines/engine.py.228: elapsed: 0:20:03 | left: 0:01:48 [I 2022-10-28 06:51:03] ...trap/engines/engine.py.229: process: 0.56290 | load: 0.00271 [I 2022-10-28 06:51:03] ...trap/engines/engine.py.230: loss: 0.11396 [I 2022-10-28 06:51:03] ...optimizers/trijoint.py.151: total_norm: 15.628133 [I 2022-10-28 06:51:09] ...trap/engines/engine.py.224: train: epoch 111 | batch 2120/2304 [I 2022-10-28 06:51:09] ...trap/engines/engine.py.228: elapsed: 0:20:09 | left: 0:01:42 [I 2022-10-28 06:51:09] ...trap/engines/engine.py.229: process: 0.55511 | load: 0.00258 [I 2022-10-28 06:51:09] ...trap/engines/engine.py.230: loss: 0.10272 [I 2022-10-28 06:51:09] ...optimizers/trijoint.py.151: total_norm: 16.318281 [I 2022-10-28 06:51:14] ...trap/engines/engine.py.224: train: epoch 111 | batch 2130/2304 [I 2022-10-28 06:51:14] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:37 [I 2022-10-28 06:51:14] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00254 [I 2022-10-28 06:51:14] ...trap/engines/engine.py.230: loss: 0.09891 [I 2022-10-28 06:51:14] ...optimizers/trijoint.py.151: total_norm: 9.156584 [I 2022-10-28 06:51:20] ...trap/engines/engine.py.224: train: epoch 111 | batch 2140/2304 [I 2022-10-28 06:51:20] ...trap/engines/engine.py.228: elapsed: 0:20:20 | left: 0:01:33 [I 2022-10-28 06:51:20] ...trap/engines/engine.py.229: process: 0.64912 | load: 0.00260 [I 2022-10-28 06:51:20] ...trap/engines/engine.py.230: loss: 0.10089 [I 2022-10-28 06:51:20] ...optimizers/trijoint.py.151: total_norm: 15.904757 [I 2022-10-28 06:51:25] ...trap/engines/engine.py.224: train: epoch 111 | batch 2150/2304 [I 2022-10-28 06:51:25] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:25 [I 2022-10-28 06:51:25] ...trap/engines/engine.py.229: process: 0.55205 | load: 0.00255 [I 2022-10-28 06:51:25] ...trap/engines/engine.py.230: loss: 0.17829 [I 2022-10-28 06:51:25] ...optimizers/trijoint.py.151: total_norm: 17.834251 [I 2022-10-28 06:51:31] ...trap/engines/engine.py.224: train: epoch 111 | batch 2160/2304 [I 2022-10-28 06:51:31] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:19 [I 2022-10-28 06:51:31] ...trap/engines/engine.py.229: process: 0.54297 | load: 0.00256 [I 2022-10-28 06:51:31] ...trap/engines/engine.py.230: loss: 0.10150 [I 2022-10-28 06:51:31] ...optimizers/trijoint.py.151: total_norm: 13.576271 [I 2022-10-28 06:51:37] ...trap/engines/engine.py.224: train: epoch 111 | batch 2170/2304 [I 2022-10-28 06:51:37] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:01:15 [I 2022-10-28 06:51:37] ...trap/engines/engine.py.229: process: 0.55859 | load: 0.00256 [I 2022-10-28 06:51:37] ...trap/engines/engine.py.230: loss: 0.07982 [I 2022-10-28 06:51:37] ...optimizers/trijoint.py.151: total_norm: 10.608646 [I 2022-10-28 06:51:42] ...trap/engines/engine.py.224: train: epoch 111 | batch 2180/2304 [I 2022-10-28 06:51:42] ...trap/engines/engine.py.228: elapsed: 0:20:42 | left: 0:01:08 [I 2022-10-28 06:51:42] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00253 [I 2022-10-28 06:51:42] ...trap/engines/engine.py.230: loss: 0.11087 [I 2022-10-28 06:51:42] ...optimizers/trijoint.py.151: total_norm: 10.634541 [I 2022-10-28 06:51:48] ...trap/engines/engine.py.224: train: epoch 111 | batch 2190/2304 [I 2022-10-28 06:51:48] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:01:03 [I 2022-10-28 06:51:48] ...trap/engines/engine.py.229: process: 0.55586 | load: 0.00250 [I 2022-10-28 06:51:48] ...trap/engines/engine.py.230: loss: 0.07205 [I 2022-10-28 06:51:48] ...optimizers/trijoint.py.151: total_norm: 9.702465 [I 2022-10-28 06:51:53] ...trap/engines/engine.py.224: train: epoch 111 | batch 2200/2304 [I 2022-10-28 06:51:53] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:58 [I 2022-10-28 06:51:53] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00264 [I 2022-10-28 06:51:53] ...trap/engines/engine.py.230: loss: 0.11036 [I 2022-10-28 06:51:53] ...optimizers/trijoint.py.151: total_norm: 8.625821 [I 2022-10-28 06:51:59] ...trap/engines/engine.py.224: train: epoch 111 | batch 2210/2304 [I 2022-10-28 06:51:59] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:52 [I 2022-10-28 06:51:59] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00256 [I 2022-10-28 06:51:59] ...trap/engines/engine.py.230: loss: 0.11156 [I 2022-10-28 06:51:59] ...optimizers/trijoint.py.151: total_norm: 11.499376 [I 2022-10-28 06:52:05] ...trap/engines/engine.py.224: train: epoch 111 | batch 2220/2304 [I 2022-10-28 06:52:05] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:46 [I 2022-10-28 06:52:05] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00259 [I 2022-10-28 06:52:05] ...trap/engines/engine.py.230: loss: 0.07023 [I 2022-10-28 06:52:05] ...optimizers/trijoint.py.151: total_norm: 53.427219 [I 2022-10-28 06:52:10] ...trap/engines/engine.py.224: train: epoch 111 | batch 2230/2304 [I 2022-10-28 06:52:10] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:41 [I 2022-10-28 06:52:10] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00252 [I 2022-10-28 06:52:10] ...trap/engines/engine.py.230: loss: 0.19527 [I 2022-10-28 06:52:10] ...optimizers/trijoint.py.151: total_norm: 14.915596 [I 2022-10-28 06:52:16] ...trap/engines/engine.py.224: train: epoch 111 | batch 2240/2304 [I 2022-10-28 06:52:16] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:35 [I 2022-10-28 06:52:16] ...trap/engines/engine.py.229: process: 0.55076 | load: 0.00261 [I 2022-10-28 06:52:16] ...trap/engines/engine.py.230: loss: 0.10264 [I 2022-10-28 06:52:16] ...optimizers/trijoint.py.151: total_norm: 14.157354 [I 2022-10-28 06:52:21] ...trap/engines/engine.py.224: train: epoch 111 | batch 2250/2304 [I 2022-10-28 06:52:21] ...trap/engines/engine.py.228: elapsed: 0:21:21 | left: 0:00:30 [I 2022-10-28 06:52:21] ...trap/engines/engine.py.229: process: 0.55804 | load: 0.00254 [I 2022-10-28 06:52:21] ...trap/engines/engine.py.230: loss: 0.11409 [I 2022-10-28 06:52:21] ...optimizers/trijoint.py.151: total_norm: 17.847765 [I 2022-10-28 06:52:27] ...trap/engines/engine.py.224: train: epoch 111 | batch 2260/2304 [I 2022-10-28 06:52:27] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:24 [I 2022-10-28 06:52:27] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00254 [I 2022-10-28 06:52:27] ...trap/engines/engine.py.230: loss: 0.08782 [I 2022-10-28 06:52:27] ...optimizers/trijoint.py.151: total_norm: 11.591461 [I 2022-10-28 06:52:32] ...trap/engines/engine.py.224: train: epoch 111 | batch 2270/2304 [I 2022-10-28 06:52:32] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:18 [I 2022-10-28 06:52:32] ...trap/engines/engine.py.229: process: 0.54494 | load: 0.00259 [I 2022-10-28 06:52:32] ...trap/engines/engine.py.230: loss: 0.08756 [I 2022-10-28 06:52:32] ...optimizers/trijoint.py.151: total_norm: 25.710945 [I 2022-10-28 06:52:38] ...trap/engines/engine.py.224: train: epoch 111 | batch 2280/2304 [I 2022-10-28 06:52:38] ...trap/engines/engine.py.228: elapsed: 0:21:38 | left: 0:00:13 [I 2022-10-28 06:52:38] ...trap/engines/engine.py.229: process: 0.56894 | load: 0.00253 [I 2022-10-28 06:52:38] ...trap/engines/engine.py.230: loss: 0.11788 [I 2022-10-28 06:52:38] ...optimizers/trijoint.py.151: total_norm: 13.898729 [I 2022-10-28 06:52:44] ...trap/engines/engine.py.224: train: epoch 111 | batch 2290/2304 [I 2022-10-28 06:52:44] ...trap/engines/engine.py.228: elapsed: 0:21:44 | left: 0:00:07 [I 2022-10-28 06:52:44] ...trap/engines/engine.py.229: process: 0.55458 | load: 0.00260 [I 2022-10-28 06:52:44] ...trap/engines/engine.py.230: loss: 0.14560 [I 2022-10-28 06:52:44] ...optimizers/trijoint.py.151: total_norm: 16.780428 [I 2022-10-28 06:52:49] ...trap/engines/engine.py.224: train: epoch 111 | batch 2300/2304 [I 2022-10-28 06:52:49] ...trap/engines/engine.py.228: elapsed: 0:21:49 | left: 0:00:02 [I 2022-10-28 06:52:49] ...trap/engines/engine.py.229: process: 0.54581 | load: 0.00015 [I 2022-10-28 06:52:49] ...trap/engines/engine.py.230: loss: 0.10232 [I 2022-10-28 06:52:49] ...optimizers/trijoint.py.151: total_norm: 33.907265 [I 2022-10-28 06:52:51] ...trap/engines/engine.py.224: train: epoch 111 | batch 2304/2304 [I 2022-10-28 06:52:51] ...trap/engines/engine.py.228: elapsed: 0:21:52 | left: 0:00:00 [I 2022-10-28 06:52:51] ...trap/engines/engine.py.229: process: 0.54286 | load: 0.00015 [I 2022-10-28 06:52:51] ...trap/engines/engine.py.230: loss: 0.14098 [I 2022-10-28 06:52:51] ...optimizers/trijoint.py.151: total_norm: 17.034403 [S 2022-10-28 06:52:53] ...trap/engines/engine.py.129: train_epoch.epoch: 111 [S 2022-10-28 06:52:53] ...trap/engines/engine.py.129: train_epoch.loss: 0.12190570338687773 [I 2022-10-28 06:52:53] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 06:52:54] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 111 [I 2022-10-28 06:52:54] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 06:52:56] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 06:52:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 06:52:56] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 06:53:00] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 06:53:00] ...trap/engines/engine.py.321: eval: epoch 111 | batch 0/510 [I 2022-10-28 06:53:00] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:51:22 [I 2022-10-28 06:53:00] ...trap/engines/engine.py.326: process: 6.04406 | load: 3.22689 [I 2022-10-28 06:53:04] ...trap/engines/engine.py.321: eval: epoch 111 | batch 10/510 [I 2022-10-28 06:53:04] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:55 [I 2022-10-28 06:53:04] ...trap/engines/engine.py.326: process: 0.34955 | load: 0.00019 [I 2022-10-28 06:53:07] ...trap/engines/engine.py.321: eval: epoch 111 | batch 20/510 [I 2022-10-28 06:53:07] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:17 [I 2022-10-28 06:53:07] ...trap/engines/engine.py.326: process: 0.37135 | load: 0.00016 [I 2022-10-28 06:53:10] ...trap/engines/engine.py.321: eval: epoch 111 | batch 30/510 [I 2022-10-28 06:53:10] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:42 [I 2022-10-28 06:53:10] ...trap/engines/engine.py.326: process: 0.33435 | load: 0.00018 [I 2022-10-28 06:53:14] ...trap/engines/engine.py.321: eval: epoch 111 | batch 40/510 [I 2022-10-28 06:53:14] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-28 06:53:14] ...trap/engines/engine.py.326: process: 0.33227 | load: 0.00018 [I 2022-10-28 06:53:17] ...trap/engines/engine.py.321: eval: epoch 111 | batch 50/510 [I 2022-10-28 06:53:17] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:34 [I 2022-10-28 06:53:17] ...trap/engines/engine.py.326: process: 0.35538 | load: 0.00023 [I 2022-10-28 06:53:20] ...trap/engines/engine.py.321: eval: epoch 111 | batch 60/510 [I 2022-10-28 06:53:20] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:29 [I 2022-10-28 06:53:20] ...trap/engines/engine.py.326: process: 0.32757 | load: 0.00016 [I 2022-10-28 06:53:24] ...trap/engines/engine.py.321: eval: epoch 111 | batch 70/510 [I 2022-10-28 06:53:24] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:24 [I 2022-10-28 06:53:24] ...trap/engines/engine.py.326: process: 0.32894 | load: 0.00017 [I 2022-10-28 06:53:27] ...trap/engines/engine.py.321: eval: epoch 111 | batch 80/510 [I 2022-10-28 06:53:27] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:22 [I 2022-10-28 06:53:27] ...trap/engines/engine.py.326: process: 0.32960 | load: 0.00016 [I 2022-10-28 06:53:30] ...trap/engines/engine.py.321: eval: epoch 111 | batch 90/510 [I 2022-10-28 06:53:30] ...trap/engines/engine.py.325: elapsed: 0:00:35 | left: 0:02:19 [I 2022-10-28 06:53:30] ...trap/engines/engine.py.326: process: 0.32524 | load: 0.00020 [I 2022-10-28 06:53:34] ...trap/engines/engine.py.321: eval: epoch 111 | batch 100/510 [I 2022-10-28 06:53:34] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:14 [I 2022-10-28 06:53:34] ...trap/engines/engine.py.326: process: 0.29563 | load: 0.00017 [I 2022-10-28 06:53:36] ...trap/engines/engine.py.321: eval: epoch 111 | batch 110/510 [I 2022-10-28 06:53:36] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:57 [I 2022-10-28 06:53:36] ...trap/engines/engine.py.326: process: 0.28881 | load: 0.00016 [I 2022-10-28 06:53:39] ...trap/engines/engine.py.321: eval: epoch 111 | batch 120/510 [I 2022-10-28 06:53:39] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:52 [I 2022-10-28 06:53:39] ...trap/engines/engine.py.326: process: 0.29472 | load: 0.00013 [I 2022-10-28 06:53:42] ...trap/engines/engine.py.321: eval: epoch 111 | batch 130/510 [I 2022-10-28 06:53:42] ...trap/engines/engine.py.325: elapsed: 0:00:47 | left: 0:01:49 [I 2022-10-28 06:53:42] ...trap/engines/engine.py.326: process: 0.28857 | load: 0.00014 [I 2022-10-28 06:53:45] ...trap/engines/engine.py.321: eval: epoch 111 | batch 140/510 [I 2022-10-28 06:53:45] ...trap/engines/engine.py.325: elapsed: 0:00:50 | left: 0:01:46 [I 2022-10-28 06:53:45] ...trap/engines/engine.py.326: process: 0.29337 | load: 0.00016 [I 2022-10-28 06:53:48] ...trap/engines/engine.py.321: eval: epoch 111 | batch 150/510 [I 2022-10-28 06:53:48] ...trap/engines/engine.py.325: elapsed: 0:00:53 | left: 0:01:45 [I 2022-10-28 06:53:48] ...trap/engines/engine.py.326: process: 0.28653 | load: 0.00017 [I 2022-10-28 06:53:51] ...trap/engines/engine.py.321: eval: epoch 111 | batch 160/510 [I 2022-10-28 06:53:51] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:42 [I 2022-10-28 06:53:51] ...trap/engines/engine.py.326: process: 0.30502 | load: 0.00015 [I 2022-10-28 06:53:54] ...trap/engines/engine.py.321: eval: epoch 111 | batch 170/510 [I 2022-10-28 06:53:54] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:37 [I 2022-10-28 06:53:54] ...trap/engines/engine.py.326: process: 0.28005 | load: 0.00026 [I 2022-10-28 06:53:57] ...trap/engines/engine.py.321: eval: epoch 111 | batch 180/510 [I 2022-10-28 06:53:57] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-28 06:53:57] ...trap/engines/engine.py.326: process: 0.29029 | load: 0.00018 [I 2022-10-28 06:54:00] ...trap/engines/engine.py.321: eval: epoch 111 | batch 190/510 [I 2022-10-28 06:54:00] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:32 [I 2022-10-28 06:54:00] ...trap/engines/engine.py.326: process: 0.28601 | load: 0.00018 [I 2022-10-28 06:54:03] ...trap/engines/engine.py.321: eval: epoch 111 | batch 200/510 [I 2022-10-28 06:54:03] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:29 [I 2022-10-28 06:54:03] ...trap/engines/engine.py.326: process: 0.29390 | load: 0.00014 [I 2022-10-28 06:54:05] ...trap/engines/engine.py.321: eval: epoch 111 | batch 210/510 [I 2022-10-28 06:54:05] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 06:54:05] ...trap/engines/engine.py.326: process: 0.28894 | load: 0.00026 [I 2022-10-28 06:54:08] ...trap/engines/engine.py.321: eval: epoch 111 | batch 220/510 [I 2022-10-28 06:54:08] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:23 [I 2022-10-28 06:54:08] ...trap/engines/engine.py.326: process: 0.29441 | load: 0.00016 [I 2022-10-28 06:54:11] ...trap/engines/engine.py.321: eval: epoch 111 | batch 230/510 [I 2022-10-28 06:54:11] ...trap/engines/engine.py.325: elapsed: 0:01:16 | left: 0:01:21 [I 2022-10-28 06:54:11] ...trap/engines/engine.py.326: process: 0.30269 | load: 0.00014 [I 2022-10-28 06:54:14] ...trap/engines/engine.py.321: eval: epoch 111 | batch 240/510 [I 2022-10-28 06:54:14] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:18 [I 2022-10-28 06:54:14] ...trap/engines/engine.py.326: process: 0.29161 | load: 0.00021 [I 2022-10-28 06:54:17] ...trap/engines/engine.py.321: eval: epoch 111 | batch 250/510 [I 2022-10-28 06:54:17] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:16 [I 2022-10-28 06:54:17] ...trap/engines/engine.py.326: process: 0.29136 | load: 0.00015 [I 2022-10-28 06:54:20] ...trap/engines/engine.py.321: eval: epoch 111 | batch 260/510 [I 2022-10-28 06:54:20] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 06:54:20] ...trap/engines/engine.py.326: process: 0.29181 | load: 0.00015 [I 2022-10-28 06:54:23] ...trap/engines/engine.py.321: eval: epoch 111 | batch 270/510 [I 2022-10-28 06:54:23] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-28 06:54:23] ...trap/engines/engine.py.326: process: 0.28846 | load: 0.00015 [I 2022-10-28 06:54:26] ...trap/engines/engine.py.321: eval: epoch 111 | batch 280/510 [I 2022-10-28 06:54:26] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 06:54:26] ...trap/engines/engine.py.326: process: 0.29339 | load: 0.00020 [I 2022-10-28 06:54:29] ...trap/engines/engine.py.321: eval: epoch 111 | batch 290/510 [I 2022-10-28 06:54:29] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 06:54:29] ...trap/engines/engine.py.326: process: 0.28583 | load: 0.00018 [I 2022-10-28 06:54:32] ...trap/engines/engine.py.321: eval: epoch 111 | batch 300/510 [I 2022-10-28 06:54:32] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-28 06:54:32] ...trap/engines/engine.py.326: process: 0.29032 | load: 0.00020 [I 2022-10-28 06:54:34] ...trap/engines/engine.py.321: eval: epoch 111 | batch 310/510 [I 2022-10-28 06:54:34] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:57 [I 2022-10-28 06:54:34] ...trap/engines/engine.py.326: process: 0.28713 | load: 0.00025 [I 2022-10-28 06:54:37] ...trap/engines/engine.py.321: eval: epoch 111 | batch 320/510 [I 2022-10-28 06:54:37] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:54 [I 2022-10-28 06:54:37] ...trap/engines/engine.py.326: process: 0.28335 | load: 0.00019 [I 2022-10-28 06:54:40] ...trap/engines/engine.py.321: eval: epoch 111 | batch 330/510 [I 2022-10-28 06:54:40] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:52 [I 2022-10-28 06:54:40] ...trap/engines/engine.py.326: process: 0.28898 | load: 0.00020 [I 2022-10-28 06:54:43] ...trap/engines/engine.py.321: eval: epoch 111 | batch 340/510 [I 2022-10-28 06:54:43] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-28 06:54:43] ...trap/engines/engine.py.326: process: 0.30235 | load: 0.00016 [I 2022-10-28 06:54:46] ...trap/engines/engine.py.321: eval: epoch 111 | batch 350/510 [I 2022-10-28 06:54:46] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 06:54:46] ...trap/engines/engine.py.326: process: 0.29028 | load: 0.00022 [I 2022-10-28 06:54:49] ...trap/engines/engine.py.321: eval: epoch 111 | batch 360/510 [I 2022-10-28 06:54:49] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:44 [I 2022-10-28 06:54:49] ...trap/engines/engine.py.326: process: 0.29883 | load: 0.00016 [I 2022-10-28 06:54:52] ...trap/engines/engine.py.321: eval: epoch 111 | batch 370/510 [I 2022-10-28 06:54:52] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:41 [I 2022-10-28 06:54:52] ...trap/engines/engine.py.326: process: 0.29708 | load: 0.00015 [I 2022-10-28 06:54:55] ...trap/engines/engine.py.321: eval: epoch 111 | batch 380/510 [I 2022-10-28 06:54:55] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 06:54:55] ...trap/engines/engine.py.326: process: 0.29277 | load: 0.00015 [I 2022-10-28 06:54:58] ...trap/engines/engine.py.321: eval: epoch 111 | batch 390/510 [I 2022-10-28 06:54:58] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 06:54:58] ...trap/engines/engine.py.326: process: 0.28511 | load: 0.00020 [I 2022-10-28 06:55:01] ...trap/engines/engine.py.321: eval: epoch 111 | batch 400/510 [I 2022-10-28 06:55:01] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:31 [I 2022-10-28 06:55:01] ...trap/engines/engine.py.326: process: 0.28859 | load: 0.00015 [I 2022-10-28 06:55:03] ...trap/engines/engine.py.321: eval: epoch 111 | batch 410/510 [I 2022-10-28 06:55:03] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:28 [I 2022-10-28 06:55:03] ...trap/engines/engine.py.326: process: 0.29480 | load: 0.00014 [I 2022-10-28 06:55:06] ...trap/engines/engine.py.321: eval: epoch 111 | batch 420/510 [I 2022-10-28 06:55:06] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:26 [I 2022-10-28 06:55:06] ...trap/engines/engine.py.326: process: 0.29298 | load: 0.00021 [I 2022-10-28 06:55:09] ...trap/engines/engine.py.321: eval: epoch 111 | batch 430/510 [I 2022-10-28 06:55:09] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 06:55:09] ...trap/engines/engine.py.326: process: 0.28784 | load: 0.00016 [I 2022-10-28 06:55:12] ...trap/engines/engine.py.321: eval: epoch 111 | batch 440/510 [I 2022-10-28 06:55:12] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 06:55:12] ...trap/engines/engine.py.326: process: 0.28370 | load: 0.00017 [I 2022-10-28 06:55:15] ...trap/engines/engine.py.321: eval: epoch 111 | batch 450/510 [I 2022-10-28 06:55:15] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 06:55:15] ...trap/engines/engine.py.326: process: 0.28970 | load: 0.00023 [I 2022-10-28 06:55:18] ...trap/engines/engine.py.321: eval: epoch 111 | batch 460/510 [I 2022-10-28 06:55:18] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 06:55:18] ...trap/engines/engine.py.326: process: 0.29208 | load: 0.00017 [I 2022-10-28 06:55:21] ...trap/engines/engine.py.321: eval: epoch 111 | batch 470/510 [I 2022-10-28 06:55:21] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 06:55:21] ...trap/engines/engine.py.326: process: 0.28896 | load: 0.00015 [I 2022-10-28 06:55:24] ...trap/engines/engine.py.321: eval: epoch 111 | batch 480/510 [I 2022-10-28 06:55:24] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 06:55:24] ...trap/engines/engine.py.326: process: 0.29226 | load: 0.00017 [I 2022-10-28 06:55:26] ...trap/engines/engine.py.321: eval: epoch 111 | batch 490/510 [I 2022-10-28 06:55:26] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 06:55:26] ...trap/engines/engine.py.326: process: 0.29082 | load: 0.00015 [I 2022-10-28 06:55:29] ...trap/engines/engine.py.321: eval: epoch 111 | batch 500/510 [I 2022-10-28 06:55:29] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:02 [I 2022-10-28 06:55:29] ...trap/engines/engine.py.326: process: 0.29050 | load: 0.00023 [I 2022-10-28 06:55:32] ...trap/engines/engine.py.321: eval: epoch 111 | batch 510/510 [I 2022-10-28 06:55:32] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 06:55:32] ...trap/engines/engine.py.326: process: 0.28789 | load: 0.00011 [S 2022-10-28 06:55:33] ...trap/engines/engine.py.132: eval_epoch.epoch: 111 [I 2022-10-28 06:55:33] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.730200 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.903600 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.929200 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.727600 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.901600 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.929700 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010265 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005161 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004512 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.014603 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.004737 [S 2022-10-28 06:55:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005460 [I 2022-10-28 06:55:39] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 06:55:39] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 06:55:40] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 06:55:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 06:55:40] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 06:55:45] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 06:55:45] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 06:55:51] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 06:55:51] ...trap/engines/engine.py.163: Training model on trainset for epoch 112 [I 2022-10-28 06:55:51] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 06:55:54] ...trap/engines/engine.py.224: train: epoch 112 | batch 0/2304 [I 2022-10-28 06:55:54] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:45:42 [I 2022-10-28 06:55:54] ...trap/engines/engine.py.229: process: 2.75284 | load: 2.12273 [I 2022-10-28 06:55:54] ...trap/engines/engine.py.230: loss: 0.11201 [I 2022-10-28 06:55:54] ...optimizers/trijoint.py.151: total_norm: 12.841290 [I 2022-10-28 06:55:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 10/2304 [I 2022-10-28 06:55:59] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:17 [I 2022-10-28 06:55:59] ...trap/engines/engine.py.229: process: 0.55759 | load: 0.00413 [I 2022-10-28 06:55:59] ...trap/engines/engine.py.230: loss: 0.13619 [I 2022-10-28 06:55:59] ...optimizers/trijoint.py.151: total_norm: 13.986172 [I 2022-10-28 06:56:05] ...trap/engines/engine.py.224: train: epoch 112 | batch 20/2304 [I 2022-10-28 06:56:05] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:19 [I 2022-10-28 06:56:05] ...trap/engines/engine.py.229: process: 0.57573 | load: 0.00343 [I 2022-10-28 06:56:05] ...trap/engines/engine.py.230: loss: 0.15058 [I 2022-10-28 06:56:05] ...optimizers/trijoint.py.151: total_norm: 27.709236 [I 2022-10-28 06:56:11] ...trap/engines/engine.py.224: train: epoch 112 | batch 30/2304 [I 2022-10-28 06:56:11] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:13 [I 2022-10-28 06:56:11] ...trap/engines/engine.py.229: process: 0.55882 | load: 0.00330 [I 2022-10-28 06:56:11] ...trap/engines/engine.py.230: loss: 0.12217 [I 2022-10-28 06:56:11] ...optimizers/trijoint.py.151: total_norm: 18.024916 [I 2022-10-28 06:56:16] ...trap/engines/engine.py.224: train: epoch 112 | batch 40/2304 [I 2022-10-28 06:56:16] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:05 [I 2022-10-28 06:56:16] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00286 [I 2022-10-28 06:56:16] ...trap/engines/engine.py.230: loss: 0.12793 [I 2022-10-28 06:56:16] ...optimizers/trijoint.py.151: total_norm: 9.520695 [I 2022-10-28 06:56:22] ...trap/engines/engine.py.224: train: epoch 112 | batch 50/2304 [I 2022-10-28 06:56:22] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:51 [I 2022-10-28 06:56:22] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00316 [I 2022-10-28 06:56:22] ...trap/engines/engine.py.230: loss: 0.06561 [I 2022-10-28 06:56:22] ...optimizers/trijoint.py.151: total_norm: 17.641933 [I 2022-10-28 06:56:27] ...trap/engines/engine.py.224: train: epoch 112 | batch 60/2304 [I 2022-10-28 06:56:27] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:48 [I 2022-10-28 06:56:27] ...trap/engines/engine.py.229: process: 0.55929 | load: 0.00263 [I 2022-10-28 06:56:27] ...trap/engines/engine.py.230: loss: 0.07863 [I 2022-10-28 06:56:27] ...optimizers/trijoint.py.151: total_norm: 16.717058 [I 2022-10-28 06:56:33] ...trap/engines/engine.py.224: train: epoch 112 | batch 70/2304 [I 2022-10-28 06:56:33] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:41 [I 2022-10-28 06:56:33] ...trap/engines/engine.py.229: process: 0.54888 | load: 0.00268 [I 2022-10-28 06:56:33] ...trap/engines/engine.py.230: loss: 0.13198 [I 2022-10-28 06:56:33] ...optimizers/trijoint.py.151: total_norm: 13.899107 [I 2022-10-28 06:56:39] ...trap/engines/engine.py.224: train: epoch 112 | batch 80/2304 [I 2022-10-28 06:56:39] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:34 [I 2022-10-28 06:56:39] ...trap/engines/engine.py.229: process: 0.55071 | load: 0.00269 [I 2022-10-28 06:56:39] ...trap/engines/engine.py.230: loss: 0.14110 [I 2022-10-28 06:56:39] ...optimizers/trijoint.py.151: total_norm: 14.616359 [I 2022-10-28 06:56:44] ...trap/engines/engine.py.224: train: epoch 112 | batch 90/2304 [I 2022-10-28 06:56:44] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:27 [I 2022-10-28 06:56:44] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00288 [I 2022-10-28 06:56:44] ...trap/engines/engine.py.230: loss: 0.14103 [I 2022-10-28 06:56:44] ...optimizers/trijoint.py.151: total_norm: 32.232372 [I 2022-10-28 06:56:50] ...trap/engines/engine.py.224: train: epoch 112 | batch 100/2304 [I 2022-10-28 06:56:50] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:24 [I 2022-10-28 06:56:50] ...trap/engines/engine.py.229: process: 0.55124 | load: 0.00242 [I 2022-10-28 06:56:50] ...trap/engines/engine.py.230: loss: 0.10954 [I 2022-10-28 06:56:50] ...optimizers/trijoint.py.151: total_norm: 21.796541 [I 2022-10-28 06:56:55] ...trap/engines/engine.py.224: train: epoch 112 | batch 110/2304 [I 2022-10-28 06:56:55] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:30 [I 2022-10-28 06:56:55] ...trap/engines/engine.py.229: process: 0.53770 | load: 0.00254 [I 2022-10-28 06:56:55] ...trap/engines/engine.py.230: loss: 0.15428 [I 2022-10-28 06:56:55] ...optimizers/trijoint.py.151: total_norm: 24.805218 [I 2022-10-28 06:57:01] ...trap/engines/engine.py.224: train: epoch 112 | batch 120/2304 [I 2022-10-28 06:57:01] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:41 [I 2022-10-28 06:57:01] ...trap/engines/engine.py.229: process: 0.56658 | load: 0.00256 [I 2022-10-28 06:57:01] ...trap/engines/engine.py.230: loss: 0.10184 [I 2022-10-28 06:57:01] ...optimizers/trijoint.py.151: total_norm: 15.143108 [I 2022-10-28 06:57:07] ...trap/engines/engine.py.224: train: epoch 112 | batch 130/2304 [I 2022-10-28 06:57:07] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:15 [I 2022-10-28 06:57:07] ...trap/engines/engine.py.229: process: 0.55184 | load: 0.00254 [I 2022-10-28 06:57:07] ...trap/engines/engine.py.230: loss: 0.13723 [I 2022-10-28 06:57:07] ...optimizers/trijoint.py.151: total_norm: 13.970703 [I 2022-10-28 06:57:12] ...trap/engines/engine.py.224: train: epoch 112 | batch 140/2304 [I 2022-10-28 06:57:12] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:06 [I 2022-10-28 06:57:12] ...trap/engines/engine.py.229: process: 0.55085 | load: 0.00244 [I 2022-10-28 06:57:12] ...trap/engines/engine.py.230: loss: 0.10454 [I 2022-10-28 06:57:12] ...optimizers/trijoint.py.151: total_norm: 12.642061 [I 2022-10-28 06:57:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 150/2304 [I 2022-10-28 06:57:18] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:57 [I 2022-10-28 06:57:18] ...trap/engines/engine.py.229: process: 0.54444 | load: 0.00247 [I 2022-10-28 06:57:18] ...trap/engines/engine.py.230: loss: 0.10937 [I 2022-10-28 06:57:18] ...optimizers/trijoint.py.151: total_norm: 13.841354 [I 2022-10-28 06:57:23] ...trap/engines/engine.py.224: train: epoch 112 | batch 160/2304 [I 2022-10-28 06:57:23] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:00 [I 2022-10-28 06:57:23] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00255 [I 2022-10-28 06:57:23] ...trap/engines/engine.py.230: loss: 0.16204 [I 2022-10-28 06:57:23] ...optimizers/trijoint.py.151: total_norm: 21.751446 [I 2022-10-28 06:57:29] ...trap/engines/engine.py.224: train: epoch 112 | batch 170/2304 [I 2022-10-28 06:57:29] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:50 [I 2022-10-28 06:57:29] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00274 [I 2022-10-28 06:57:29] ...trap/engines/engine.py.230: loss: 0.17397 [I 2022-10-28 06:57:29] ...optimizers/trijoint.py.151: total_norm: 14.998199 [I 2022-10-28 06:57:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 180/2304 [I 2022-10-28 06:57:35] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:36 [I 2022-10-28 06:57:35] ...trap/engines/engine.py.229: process: 0.55023 | load: 0.00280 [I 2022-10-28 06:57:35] ...trap/engines/engine.py.230: loss: 0.09967 [I 2022-10-28 06:57:35] ...optimizers/trijoint.py.151: total_norm: 13.417879 [I 2022-10-28 06:57:40] ...trap/engines/engine.py.224: train: epoch 112 | batch 190/2304 [I 2022-10-28 06:57:40] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:34 [I 2022-10-28 06:57:40] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00256 [I 2022-10-28 06:57:40] ...trap/engines/engine.py.230: loss: 0.19880 [I 2022-10-28 06:57:40] ...optimizers/trijoint.py.151: total_norm: 22.043875 [I 2022-10-28 06:57:46] ...trap/engines/engine.py.224: train: epoch 112 | batch 200/2304 [I 2022-10-28 06:57:46] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:49 [I 2022-10-28 06:57:46] ...trap/engines/engine.py.229: process: 0.55865 | load: 0.00261 [I 2022-10-28 06:57:46] ...trap/engines/engine.py.230: loss: 0.17206 [I 2022-10-28 06:57:46] ...optimizers/trijoint.py.151: total_norm: 18.371752 [I 2022-10-28 06:57:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 210/2304 [I 2022-10-28 06:57:51] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:32 [I 2022-10-28 06:57:51] ...trap/engines/engine.py.229: process: 0.56030 | load: 0.00266 [I 2022-10-28 06:57:51] ...trap/engines/engine.py.230: loss: 0.15649 [I 2022-10-28 06:57:51] ...optimizers/trijoint.py.151: total_norm: 25.313076 [I 2022-10-28 06:57:57] ...trap/engines/engine.py.224: train: epoch 112 | batch 220/2304 [I 2022-10-28 06:57:57] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:20 [I 2022-10-28 06:57:57] ...trap/engines/engine.py.229: process: 0.54690 | load: 0.00254 [I 2022-10-28 06:57:57] ...trap/engines/engine.py.230: loss: 0.10880 [I 2022-10-28 06:57:57] ...optimizers/trijoint.py.151: total_norm: 24.829830 [I 2022-10-28 06:58:03] ...trap/engines/engine.py.224: train: epoch 112 | batch 230/2304 [I 2022-10-28 06:58:03] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:09 [I 2022-10-28 06:58:03] ...trap/engines/engine.py.229: process: 0.54644 | load: 0.00251 [I 2022-10-28 06:58:03] ...trap/engines/engine.py.230: loss: 0.09071 [I 2022-10-28 06:58:03] ...optimizers/trijoint.py.151: total_norm: 16.332344 [I 2022-10-28 06:58:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 240/2304 [I 2022-10-28 06:58:08] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:18 [I 2022-10-28 06:58:08] ...trap/engines/engine.py.229: process: 0.55896 | load: 0.00279 [I 2022-10-28 06:58:08] ...trap/engines/engine.py.230: loss: 0.11009 [I 2022-10-28 06:58:08] ...optimizers/trijoint.py.151: total_norm: 17.827314 [I 2022-10-28 06:58:14] ...trap/engines/engine.py.224: train: epoch 112 | batch 250/2304 [I 2022-10-28 06:58:14] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:03 [I 2022-10-28 06:58:14] ...trap/engines/engine.py.229: process: 0.56012 | load: 0.00265 [I 2022-10-28 06:58:14] ...trap/engines/engine.py.230: loss: 0.09100 [I 2022-10-28 06:58:14] ...optimizers/trijoint.py.151: total_norm: 13.357941 [I 2022-10-28 06:58:19] ...trap/engines/engine.py.224: train: epoch 112 | batch 260/2304 [I 2022-10-28 06:58:19] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:03 [I 2022-10-28 06:58:19] ...trap/engines/engine.py.229: process: 0.55413 | load: 0.00271 [I 2022-10-28 06:58:19] ...trap/engines/engine.py.230: loss: 0.11265 [I 2022-10-28 06:58:19] ...optimizers/trijoint.py.151: total_norm: 11.562456 [I 2022-10-28 06:58:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 270/2304 [I 2022-10-28 06:58:25] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:54 [I 2022-10-28 06:58:25] ...trap/engines/engine.py.229: process: 0.54781 | load: 0.00263 [I 2022-10-28 06:58:25] ...trap/engines/engine.py.230: loss: 0.15518 [I 2022-10-28 06:58:25] ...optimizers/trijoint.py.151: total_norm: 13.067310 [I 2022-10-28 06:58:31] ...trap/engines/engine.py.224: train: epoch 112 | batch 280/2304 [I 2022-10-28 06:58:31] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:48 [I 2022-10-28 06:58:31] ...trap/engines/engine.py.229: process: 0.55539 | load: 0.00249 [I 2022-10-28 06:58:31] ...trap/engines/engine.py.230: loss: 0.09824 [I 2022-10-28 06:58:31] ...optimizers/trijoint.py.151: total_norm: 15.774547 [I 2022-10-28 06:58:36] ...trap/engines/engine.py.224: train: epoch 112 | batch 290/2304 [I 2022-10-28 06:58:36] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:42 [I 2022-10-28 06:58:36] ...trap/engines/engine.py.229: process: 0.54869 | load: 0.00244 [I 2022-10-28 06:58:36] ...trap/engines/engine.py.230: loss: 0.15682 [I 2022-10-28 06:58:36] ...optimizers/trijoint.py.151: total_norm: 18.985115 [I 2022-10-28 06:58:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 300/2304 [I 2022-10-28 06:58:42] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:32 [I 2022-10-28 06:58:42] ...trap/engines/engine.py.229: process: 0.55163 | load: 0.00258 [I 2022-10-28 06:58:42] ...trap/engines/engine.py.230: loss: 0.11537 [I 2022-10-28 06:58:42] ...optimizers/trijoint.py.151: total_norm: 14.553873 [I 2022-10-28 06:58:47] ...trap/engines/engine.py.224: train: epoch 112 | batch 310/2304 [I 2022-10-28 06:58:47] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:32 [I 2022-10-28 06:58:47] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00255 [I 2022-10-28 06:58:47] ...trap/engines/engine.py.230: loss: 0.23499 [I 2022-10-28 06:58:47] ...optimizers/trijoint.py.151: total_norm: 24.858425 [I 2022-10-28 06:58:53] ...trap/engines/engine.py.224: train: epoch 112 | batch 320/2304 [I 2022-10-28 06:58:53] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:30 [I 2022-10-28 06:58:53] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00254 [I 2022-10-28 06:58:53] ...trap/engines/engine.py.230: loss: 0.10631 [I 2022-10-28 06:58:53] ...optimizers/trijoint.py.151: total_norm: 12.087153 [I 2022-10-28 06:58:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 330/2304 [I 2022-10-28 06:58:59] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:20 [I 2022-10-28 06:58:59] ...trap/engines/engine.py.229: process: 0.56033 | load: 0.00252 [I 2022-10-28 06:58:59] ...trap/engines/engine.py.230: loss: 0.09756 [I 2022-10-28 06:58:59] ...optimizers/trijoint.py.151: total_norm: 14.696531 [I 2022-10-28 06:59:04] ...trap/engines/engine.py.224: train: epoch 112 | batch 340/2304 [I 2022-10-28 06:59:04] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:11 [I 2022-10-28 06:59:04] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00250 [I 2022-10-28 06:59:04] ...trap/engines/engine.py.230: loss: 0.17812 [I 2022-10-28 06:59:04] ...optimizers/trijoint.py.151: total_norm: 40.671795 [I 2022-10-28 06:59:10] ...trap/engines/engine.py.224: train: epoch 112 | batch 350/2304 [I 2022-10-28 06:59:10] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:09 [I 2022-10-28 06:59:10] ...trap/engines/engine.py.229: process: 0.55489 | load: 0.00269 [I 2022-10-28 06:59:10] ...trap/engines/engine.py.230: loss: 0.16801 [I 2022-10-28 06:59:10] ...optimizers/trijoint.py.151: total_norm: 10.194797 [I 2022-10-28 06:59:15] ...trap/engines/engine.py.224: train: epoch 112 | batch 360/2304 [I 2022-10-28 06:59:15] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:04 [I 2022-10-28 06:59:15] ...trap/engines/engine.py.229: process: 0.53985 | load: 0.00254 [I 2022-10-28 06:59:15] ...trap/engines/engine.py.230: loss: 0.12724 [I 2022-10-28 06:59:15] ...optimizers/trijoint.py.151: total_norm: 21.753319 [I 2022-10-28 06:59:21] ...trap/engines/engine.py.224: train: epoch 112 | batch 370/2304 [I 2022-10-28 06:59:21] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:18:04 [I 2022-10-28 06:59:21] ...trap/engines/engine.py.229: process: 0.54954 | load: 0.00253 [I 2022-10-28 06:59:21] ...trap/engines/engine.py.230: loss: 0.12353 [I 2022-10-28 06:59:21] ...optimizers/trijoint.py.151: total_norm: 18.887703 [I 2022-10-28 06:59:27] ...trap/engines/engine.py.224: train: epoch 112 | batch 380/2304 [I 2022-10-28 06:59:27] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:56 [I 2022-10-28 06:59:27] ...trap/engines/engine.py.229: process: 0.55659 | load: 0.00246 [I 2022-10-28 06:59:27] ...trap/engines/engine.py.230: loss: 0.06390 [I 2022-10-28 06:59:27] ...optimizers/trijoint.py.151: total_norm: 14.197118 [I 2022-10-28 06:59:32] ...trap/engines/engine.py.224: train: epoch 112 | batch 390/2304 [I 2022-10-28 06:59:32] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:46 [I 2022-10-28 06:59:32] ...trap/engines/engine.py.229: process: 0.54597 | load: 0.00264 [I 2022-10-28 06:59:32] ...trap/engines/engine.py.230: loss: 0.20696 [I 2022-10-28 06:59:32] ...optimizers/trijoint.py.151: total_norm: 14.945611 [I 2022-10-28 06:59:38] ...trap/engines/engine.py.224: train: epoch 112 | batch 400/2304 [I 2022-10-28 06:59:38] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:35 [I 2022-10-28 06:59:38] ...trap/engines/engine.py.229: process: 0.56145 | load: 0.00256 [I 2022-10-28 06:59:38] ...trap/engines/engine.py.230: loss: 0.11909 [I 2022-10-28 06:59:38] ...optimizers/trijoint.py.151: total_norm: 16.055904 [I 2022-10-28 06:59:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 410/2304 [I 2022-10-28 06:59:43] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:39 [I 2022-10-28 06:59:43] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00255 [I 2022-10-28 06:59:43] ...trap/engines/engine.py.230: loss: 0.12160 [I 2022-10-28 06:59:43] ...optimizers/trijoint.py.151: total_norm: 54.858997 [I 2022-10-28 06:59:49] ...trap/engines/engine.py.224: train: epoch 112 | batch 420/2304 [I 2022-10-28 06:59:49] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:30 [I 2022-10-28 06:59:49] ...trap/engines/engine.py.229: process: 0.56185 | load: 0.00250 [I 2022-10-28 06:59:49] ...trap/engines/engine.py.230: loss: 0.08486 [I 2022-10-28 06:59:49] ...optimizers/trijoint.py.151: total_norm: 13.497730 [I 2022-10-28 06:59:54] ...trap/engines/engine.py.224: train: epoch 112 | batch 430/2304 [I 2022-10-28 06:59:54] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:17 [I 2022-10-28 06:59:54] ...trap/engines/engine.py.229: process: 0.54360 | load: 0.00260 [I 2022-10-28 06:59:54] ...trap/engines/engine.py.230: loss: 0.15013 [I 2022-10-28 06:59:54] ...optimizers/trijoint.py.151: total_norm: 9.806561 [I 2022-10-28 07:00:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 440/2304 [I 2022-10-28 07:00:00] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:15 [I 2022-10-28 07:00:00] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00250 [I 2022-10-28 07:00:00] ...trap/engines/engine.py.230: loss: 0.15758 [I 2022-10-28 07:00:00] ...optimizers/trijoint.py.151: total_norm: 19.640181 [I 2022-10-28 07:00:06] ...trap/engines/engine.py.224: train: epoch 112 | batch 450/2304 [I 2022-10-28 07:00:06] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:02 [I 2022-10-28 07:00:06] ...trap/engines/engine.py.229: process: 0.54305 | load: 0.00247 [I 2022-10-28 07:00:06] ...trap/engines/engine.py.230: loss: 0.24266 [I 2022-10-28 07:00:06] ...optimizers/trijoint.py.151: total_norm: 36.177094 [I 2022-10-28 07:00:11] ...trap/engines/engine.py.224: train: epoch 112 | batch 460/2304 [I 2022-10-28 07:00:11] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:02 [I 2022-10-28 07:00:11] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00246 [I 2022-10-28 07:00:11] ...trap/engines/engine.py.230: loss: 0.12040 [I 2022-10-28 07:00:11] ...optimizers/trijoint.py.151: total_norm: 15.439208 [I 2022-10-28 07:00:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 470/2304 [I 2022-10-28 07:00:17] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:16:55 [I 2022-10-28 07:00:17] ...trap/engines/engine.py.229: process: 0.54704 | load: 0.00241 [I 2022-10-28 07:00:17] ...trap/engines/engine.py.230: loss: 0.09210 [I 2022-10-28 07:00:17] ...optimizers/trijoint.py.151: total_norm: 11.825769 [I 2022-10-28 07:00:22] ...trap/engines/engine.py.224: train: epoch 112 | batch 480/2304 [I 2022-10-28 07:00:22] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:21 [I 2022-10-28 07:00:22] ...trap/engines/engine.py.229: process: 0.54944 | load: 0.00239 [I 2022-10-28 07:00:22] ...trap/engines/engine.py.230: loss: 0.22483 [I 2022-10-28 07:00:22] ...optimizers/trijoint.py.151: total_norm: 29.298756 [I 2022-10-28 07:00:28] ...trap/engines/engine.py.224: train: epoch 112 | batch 490/2304 [I 2022-10-28 07:00:28] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:53 [I 2022-10-28 07:00:28] ...trap/engines/engine.py.229: process: 0.55456 | load: 0.00247 [I 2022-10-28 07:00:28] ...trap/engines/engine.py.230: loss: 0.10440 [I 2022-10-28 07:00:28] ...optimizers/trijoint.py.151: total_norm: 13.454195 [I 2022-10-28 07:00:34] ...trap/engines/engine.py.224: train: epoch 112 | batch 500/2304 [I 2022-10-28 07:00:34] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:42 [I 2022-10-28 07:00:34] ...trap/engines/engine.py.229: process: 0.55233 | load: 0.00245 [I 2022-10-28 07:00:34] ...trap/engines/engine.py.230: loss: 0.14392 [I 2022-10-28 07:00:34] ...optimizers/trijoint.py.151: total_norm: 12.513319 [I 2022-10-28 07:00:39] ...trap/engines/engine.py.224: train: epoch 112 | batch 510/2304 [I 2022-10-28 07:00:39] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:37 [I 2022-10-28 07:00:39] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00244 [I 2022-10-28 07:00:39] ...trap/engines/engine.py.230: loss: 0.08094 [I 2022-10-28 07:00:39] ...optimizers/trijoint.py.151: total_norm: 15.374640 [I 2022-10-28 07:00:45] ...trap/engines/engine.py.224: train: epoch 112 | batch 520/2304 [I 2022-10-28 07:00:45] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:31 [I 2022-10-28 07:00:45] ...trap/engines/engine.py.229: process: 0.53704 | load: 0.00241 [I 2022-10-28 07:00:45] ...trap/engines/engine.py.230: loss: 0.11644 [I 2022-10-28 07:00:45] ...optimizers/trijoint.py.151: total_norm: 9.522649 [I 2022-10-28 07:00:50] ...trap/engines/engine.py.224: train: epoch 112 | batch 530/2304 [I 2022-10-28 07:00:50] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:21 [I 2022-10-28 07:00:50] ...trap/engines/engine.py.229: process: 0.54421 | load: 0.00248 [I 2022-10-28 07:00:50] ...trap/engines/engine.py.230: loss: 0.11729 [I 2022-10-28 07:00:50] ...optimizers/trijoint.py.151: total_norm: 10.216344 [I 2022-10-28 07:00:56] ...trap/engines/engine.py.224: train: epoch 112 | batch 540/2304 [I 2022-10-28 07:00:56] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:10 [I 2022-10-28 07:00:56] ...trap/engines/engine.py.229: process: 0.54918 | load: 0.00252 [I 2022-10-28 07:00:56] ...trap/engines/engine.py.230: loss: 0.11075 [I 2022-10-28 07:00:56] ...optimizers/trijoint.py.151: total_norm: 22.882895 [I 2022-10-28 07:01:01] ...trap/engines/engine.py.224: train: epoch 112 | batch 550/2304 [I 2022-10-28 07:01:01] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:12 [I 2022-10-28 07:01:01] ...trap/engines/engine.py.229: process: 0.55308 | load: 0.00251 [I 2022-10-28 07:01:01] ...trap/engines/engine.py.230: loss: 0.16399 [I 2022-10-28 07:01:01] ...optimizers/trijoint.py.151: total_norm: 20.299452 [I 2022-10-28 07:01:07] ...trap/engines/engine.py.224: train: epoch 112 | batch 560/2304 [I 2022-10-28 07:01:07] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:08 [I 2022-10-28 07:01:07] ...trap/engines/engine.py.229: process: 0.53835 | load: 0.00247 [I 2022-10-28 07:01:07] ...trap/engines/engine.py.230: loss: 0.12099 [I 2022-10-28 07:01:07] ...optimizers/trijoint.py.151: total_norm: 9.847301 [I 2022-10-28 07:01:13] ...trap/engines/engine.py.224: train: epoch 112 | batch 570/2304 [I 2022-10-28 07:01:13] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:04 [I 2022-10-28 07:01:13] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00256 [I 2022-10-28 07:01:13] ...trap/engines/engine.py.230: loss: 0.15978 [I 2022-10-28 07:01:13] ...optimizers/trijoint.py.151: total_norm: 31.559505 [I 2022-10-28 07:01:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 580/2304 [I 2022-10-28 07:01:18] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:57 [I 2022-10-28 07:01:18] ...trap/engines/engine.py.229: process: 0.54472 | load: 0.00244 [I 2022-10-28 07:01:18] ...trap/engines/engine.py.230: loss: 0.14195 [I 2022-10-28 07:01:18] ...optimizers/trijoint.py.151: total_norm: 27.482191 [I 2022-10-28 07:01:24] ...trap/engines/engine.py.224: train: epoch 112 | batch 590/2304 [I 2022-10-28 07:01:24] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:16:13 [I 2022-10-28 07:01:24] ...trap/engines/engine.py.229: process: 0.61648 | load: 0.00252 [I 2022-10-28 07:01:24] ...trap/engines/engine.py.230: loss: 0.21430 [I 2022-10-28 07:01:24] ...optimizers/trijoint.py.151: total_norm: 19.533737 [I 2022-10-28 07:01:29] ...trap/engines/engine.py.224: train: epoch 112 | batch 600/2304 [I 2022-10-28 07:01:29] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:43 [I 2022-10-28 07:01:29] ...trap/engines/engine.py.229: process: 0.54543 | load: 0.00251 [I 2022-10-28 07:01:29] ...trap/engines/engine.py.230: loss: 0.20097 [I 2022-10-28 07:01:29] ...optimizers/trijoint.py.151: total_norm: 36.761238 [I 2022-10-28 07:01:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 610/2304 [I 2022-10-28 07:01:35] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:44 [I 2022-10-28 07:01:35] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00245 [I 2022-10-28 07:01:35] ...trap/engines/engine.py.230: loss: 0.08576 [I 2022-10-28 07:01:35] ...optimizers/trijoint.py.151: total_norm: 13.989017 [I 2022-10-28 07:01:40] ...trap/engines/engine.py.224: train: epoch 112 | batch 620/2304 [I 2022-10-28 07:01:40] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:33 [I 2022-10-28 07:01:40] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00269 [I 2022-10-28 07:01:40] ...trap/engines/engine.py.230: loss: 0.13110 [I 2022-10-28 07:01:40] ...optimizers/trijoint.py.151: total_norm: 23.461672 [I 2022-10-28 07:01:46] ...trap/engines/engine.py.224: train: epoch 112 | batch 630/2304 [I 2022-10-28 07:01:46] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:34 [I 2022-10-28 07:01:46] ...trap/engines/engine.py.229: process: 0.55231 | load: 0.00256 [I 2022-10-28 07:01:46] ...trap/engines/engine.py.230: loss: 0.18023 [I 2022-10-28 07:01:46] ...optimizers/trijoint.py.151: total_norm: 21.446194 [I 2022-10-28 07:01:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 640/2304 [I 2022-10-28 07:01:52] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:25 [I 2022-10-28 07:01:52] ...trap/engines/engine.py.229: process: 0.53900 | load: 0.00254 [I 2022-10-28 07:01:52] ...trap/engines/engine.py.230: loss: 0.09962 [I 2022-10-28 07:01:52] ...optimizers/trijoint.py.151: total_norm: 15.636865 [I 2022-10-28 07:01:57] ...trap/engines/engine.py.224: train: epoch 112 | batch 650/2304 [I 2022-10-28 07:01:57] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:29 [I 2022-10-28 07:01:57] ...trap/engines/engine.py.229: process: 0.55297 | load: 0.00255 [I 2022-10-28 07:01:57] ...trap/engines/engine.py.230: loss: 0.10793 [I 2022-10-28 07:01:57] ...optimizers/trijoint.py.151: total_norm: 16.486755 [I 2022-10-28 07:02:03] ...trap/engines/engine.py.224: train: epoch 112 | batch 660/2304 [I 2022-10-28 07:02:03] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:11 [I 2022-10-28 07:02:03] ...trap/engines/engine.py.229: process: 0.55445 | load: 0.00247 [I 2022-10-28 07:02:03] ...trap/engines/engine.py.230: loss: 0.09262 [I 2022-10-28 07:02:03] ...optimizers/trijoint.py.151: total_norm: 10.112458 [I 2022-10-28 07:02:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 670/2304 [I 2022-10-28 07:02:08] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:06 [I 2022-10-28 07:02:08] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00242 [I 2022-10-28 07:02:08] ...trap/engines/engine.py.230: loss: 0.09889 [I 2022-10-28 07:02:08] ...optimizers/trijoint.py.151: total_norm: 11.940139 [I 2022-10-28 07:02:14] ...trap/engines/engine.py.224: train: epoch 112 | batch 680/2304 [I 2022-10-28 07:02:14] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:04 [I 2022-10-28 07:02:14] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00243 [I 2022-10-28 07:02:14] ...trap/engines/engine.py.230: loss: 0.08444 [I 2022-10-28 07:02:14] ...optimizers/trijoint.py.151: total_norm: 14.645073 [I 2022-10-28 07:02:20] ...trap/engines/engine.py.224: train: epoch 112 | batch 690/2304 [I 2022-10-28 07:02:20] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:49 [I 2022-10-28 07:02:20] ...trap/engines/engine.py.229: process: 0.55219 | load: 0.00252 [I 2022-10-28 07:02:20] ...trap/engines/engine.py.230: loss: 0.09426 [I 2022-10-28 07:02:20] ...optimizers/trijoint.py.151: total_norm: 14.474901 [I 2022-10-28 07:02:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 700/2304 [I 2022-10-28 07:02:25] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:41 [I 2022-10-28 07:02:25] ...trap/engines/engine.py.229: process: 0.55210 | load: 0.00240 [I 2022-10-28 07:02:25] ...trap/engines/engine.py.230: loss: 0.13869 [I 2022-10-28 07:02:25] ...optimizers/trijoint.py.151: total_norm: 74.988747 [I 2022-10-28 07:02:31] ...trap/engines/engine.py.224: train: epoch 112 | batch 710/2304 [I 2022-10-28 07:02:31] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:37 [I 2022-10-28 07:02:31] ...trap/engines/engine.py.229: process: 0.54509 | load: 0.00247 [I 2022-10-28 07:02:31] ...trap/engines/engine.py.230: loss: 0.13859 [I 2022-10-28 07:02:31] ...optimizers/trijoint.py.151: total_norm: 21.662004 [I 2022-10-28 07:02:36] ...trap/engines/engine.py.224: train: epoch 112 | batch 720/2304 [I 2022-10-28 07:02:36] ...trap/engines/engine.py.228: elapsed: 0:06:44 | left: 0:14:38 [I 2022-10-28 07:02:36] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00238 [I 2022-10-28 07:02:36] ...trap/engines/engine.py.230: loss: 0.18125 [I 2022-10-28 07:02:36] ...optimizers/trijoint.py.151: total_norm: 16.360760 [I 2022-10-28 07:02:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 730/2304 [I 2022-10-28 07:02:42] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:28 [I 2022-10-28 07:02:42] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00240 [I 2022-10-28 07:02:42] ...trap/engines/engine.py.230: loss: 0.10565 [I 2022-10-28 07:02:42] ...optimizers/trijoint.py.151: total_norm: 13.382032 [I 2022-10-28 07:02:47] ...trap/engines/engine.py.224: train: epoch 112 | batch 740/2304 [I 2022-10-28 07:02:47] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:32 [I 2022-10-28 07:02:47] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00244 [I 2022-10-28 07:02:47] ...trap/engines/engine.py.230: loss: 0.12533 [I 2022-10-28 07:02:47] ...optimizers/trijoint.py.151: total_norm: 11.675634 [I 2022-10-28 07:02:53] ...trap/engines/engine.py.224: train: epoch 112 | batch 750/2304 [I 2022-10-28 07:02:53] ...trap/engines/engine.py.228: elapsed: 0:07:01 | left: 0:14:20 [I 2022-10-28 07:02:53] ...trap/engines/engine.py.229: process: 0.55044 | load: 0.00241 [I 2022-10-28 07:02:53] ...trap/engines/engine.py.230: loss: 0.10968 [I 2022-10-28 07:02:53] ...optimizers/trijoint.py.151: total_norm: 7.826590 [I 2022-10-28 07:02:58] ...trap/engines/engine.py.224: train: epoch 112 | batch 760/2304 [I 2022-10-28 07:02:58] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:16 [I 2022-10-28 07:02:58] ...trap/engines/engine.py.229: process: 0.55671 | load: 0.00240 [I 2022-10-28 07:02:58] ...trap/engines/engine.py.230: loss: 0.10804 [I 2022-10-28 07:02:58] ...optimizers/trijoint.py.151: total_norm: 17.451696 [I 2022-10-28 07:03:04] ...trap/engines/engine.py.224: train: epoch 112 | batch 770/2304 [I 2022-10-28 07:03:04] ...trap/engines/engine.py.228: elapsed: 0:07:12 | left: 0:14:07 [I 2022-10-28 07:03:04] ...trap/engines/engine.py.229: process: 0.54639 | load: 0.00241 [I 2022-10-28 07:03:04] ...trap/engines/engine.py.230: loss: 0.10590 [I 2022-10-28 07:03:04] ...optimizers/trijoint.py.151: total_norm: 16.409349 [I 2022-10-28 07:03:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 780/2304 [I 2022-10-28 07:03:09] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:01 [I 2022-10-28 07:03:09] ...trap/engines/engine.py.229: process: 0.55267 | load: 0.00242 [I 2022-10-28 07:03:09] ...trap/engines/engine.py.230: loss: 0.14616 [I 2022-10-28 07:03:09] ...optimizers/trijoint.py.151: total_norm: 41.354427 [I 2022-10-28 07:03:15] ...trap/engines/engine.py.224: train: epoch 112 | batch 790/2304 [I 2022-10-28 07:03:15] ...trap/engines/engine.py.228: elapsed: 0:07:23 | left: 0:14:01 [I 2022-10-28 07:03:15] ...trap/engines/engine.py.229: process: 0.56935 | load: 0.00242 [I 2022-10-28 07:03:15] ...trap/engines/engine.py.230: loss: 0.09242 [I 2022-10-28 07:03:15] ...optimizers/trijoint.py.151: total_norm: 23.639431 [I 2022-10-28 07:03:21] ...trap/engines/engine.py.224: train: epoch 112 | batch 800/2304 [I 2022-10-28 07:03:21] ...trap/engines/engine.py.228: elapsed: 0:07:29 | left: 0:13:55 [I 2022-10-28 07:03:21] ...trap/engines/engine.py.229: process: 0.54704 | load: 0.00242 [I 2022-10-28 07:03:21] ...trap/engines/engine.py.230: loss: 0.10464 [I 2022-10-28 07:03:21] ...optimizers/trijoint.py.151: total_norm: 24.254047 [I 2022-10-28 07:03:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 810/2304 [I 2022-10-28 07:03:26] ...trap/engines/engine.py.228: elapsed: 0:07:34 | left: 0:13:47 [I 2022-10-28 07:03:26] ...trap/engines/engine.py.229: process: 0.56112 | load: 0.00242 [I 2022-10-28 07:03:26] ...trap/engines/engine.py.230: loss: 0.10000 [I 2022-10-28 07:03:26] ...optimizers/trijoint.py.151: total_norm: 12.746655 [I 2022-10-28 07:03:32] ...trap/engines/engine.py.224: train: epoch 112 | batch 820/2304 [I 2022-10-28 07:03:32] ...trap/engines/engine.py.228: elapsed: 0:07:40 | left: 0:13:42 [I 2022-10-28 07:03:32] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00242 [I 2022-10-28 07:03:32] ...trap/engines/engine.py.230: loss: 0.14691 [I 2022-10-28 07:03:32] ...optimizers/trijoint.py.151: total_norm: 28.775282 [I 2022-10-28 07:03:37] ...trap/engines/engine.py.224: train: epoch 112 | batch 830/2304 [I 2022-10-28 07:03:37] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:33 [I 2022-10-28 07:03:37] ...trap/engines/engine.py.229: process: 0.54795 | load: 0.00241 [I 2022-10-28 07:03:37] ...trap/engines/engine.py.230: loss: 0.13366 [I 2022-10-28 07:03:37] ...optimizers/trijoint.py.151: total_norm: 10.043017 [I 2022-10-28 07:03:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 840/2304 [I 2022-10-28 07:03:43] ...trap/engines/engine.py.228: elapsed: 0:07:51 | left: 0:13:29 [I 2022-10-28 07:03:43] ...trap/engines/engine.py.229: process: 0.54904 | load: 0.00246 [I 2022-10-28 07:03:43] ...trap/engines/engine.py.230: loss: 0.16465 [I 2022-10-28 07:03:43] ...optimizers/trijoint.py.151: total_norm: 9.238128 [I 2022-10-28 07:03:48] ...trap/engines/engine.py.224: train: epoch 112 | batch 850/2304 [I 2022-10-28 07:03:48] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:25 [I 2022-10-28 07:03:48] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00250 [I 2022-10-28 07:03:48] ...trap/engines/engine.py.230: loss: 0.08794 [I 2022-10-28 07:03:48] ...optimizers/trijoint.py.151: total_norm: 19.804409 [I 2022-10-28 07:03:54] ...trap/engines/engine.py.224: train: epoch 112 | batch 860/2304 [I 2022-10-28 07:03:54] ...trap/engines/engine.py.228: elapsed: 0:08:02 | left: 0:13:19 [I 2022-10-28 07:03:54] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00251 [I 2022-10-28 07:03:54] ...trap/engines/engine.py.230: loss: 0.09947 [I 2022-10-28 07:03:54] ...optimizers/trijoint.py.151: total_norm: 21.201281 [I 2022-10-28 07:03:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 870/2304 [I 2022-10-28 07:03:59] ...trap/engines/engine.py.228: elapsed: 0:08:08 | left: 0:13:12 [I 2022-10-28 07:03:59] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00239 [I 2022-10-28 07:03:59] ...trap/engines/engine.py.230: loss: 0.12739 [I 2022-10-28 07:03:59] ...optimizers/trijoint.py.151: total_norm: 15.128517 [I 2022-10-28 07:04:05] ...trap/engines/engine.py.224: train: epoch 112 | batch 880/2304 [I 2022-10-28 07:04:05] ...trap/engines/engine.py.228: elapsed: 0:08:13 | left: 0:13:03 [I 2022-10-28 07:04:05] ...trap/engines/engine.py.229: process: 0.54765 | load: 0.00242 [I 2022-10-28 07:04:05] ...trap/engines/engine.py.230: loss: 0.13828 [I 2022-10-28 07:04:05] ...optimizers/trijoint.py.151: total_norm: 18.400398 [I 2022-10-28 07:04:10] ...trap/engines/engine.py.224: train: epoch 112 | batch 890/2304 [I 2022-10-28 07:04:10] ...trap/engines/engine.py.228: elapsed: 0:08:19 | left: 0:13:07 [I 2022-10-28 07:04:10] ...trap/engines/engine.py.229: process: 0.56677 | load: 0.00247 [I 2022-10-28 07:04:10] ...trap/engines/engine.py.230: loss: 0.12072 [I 2022-10-28 07:04:10] ...optimizers/trijoint.py.151: total_norm: 16.410984 [I 2022-10-28 07:04:16] ...trap/engines/engine.py.224: train: epoch 112 | batch 900/2304 [I 2022-10-28 07:04:16] ...trap/engines/engine.py.228: elapsed: 0:08:24 | left: 0:12:57 [I 2022-10-28 07:04:16] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00256 [I 2022-10-28 07:04:16] ...trap/engines/engine.py.230: loss: 0.07123 [I 2022-10-28 07:04:16] ...optimizers/trijoint.py.151: total_norm: 13.038468 [I 2022-10-28 07:04:22] ...trap/engines/engine.py.224: train: epoch 112 | batch 910/2304 [I 2022-10-28 07:04:22] ...trap/engines/engine.py.228: elapsed: 0:08:30 | left: 0:12:56 [I 2022-10-28 07:04:22] ...trap/engines/engine.py.229: process: 0.54723 | load: 0.00260 [I 2022-10-28 07:04:22] ...trap/engines/engine.py.230: loss: 0.09194 [I 2022-10-28 07:04:22] ...optimizers/trijoint.py.151: total_norm: 12.060574 [I 2022-10-28 07:04:27] ...trap/engines/engine.py.224: train: epoch 112 | batch 920/2304 [I 2022-10-28 07:04:27] ...trap/engines/engine.py.228: elapsed: 0:08:36 | left: 0:12:52 [I 2022-10-28 07:04:27] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00249 [I 2022-10-28 07:04:27] ...trap/engines/engine.py.230: loss: 0.16266 [I 2022-10-28 07:04:27] ...optimizers/trijoint.py.151: total_norm: 24.283064 [I 2022-10-28 07:04:33] ...trap/engines/engine.py.224: train: epoch 112 | batch 930/2304 [I 2022-10-28 07:04:33] ...trap/engines/engine.py.228: elapsed: 0:08:41 | left: 0:12:43 [I 2022-10-28 07:04:33] ...trap/engines/engine.py.229: process: 0.55049 | load: 0.00249 [I 2022-10-28 07:04:33] ...trap/engines/engine.py.230: loss: 0.11917 [I 2022-10-28 07:04:33] ...optimizers/trijoint.py.151: total_norm: 14.594053 [I 2022-10-28 07:04:38] ...trap/engines/engine.py.224: train: epoch 112 | batch 940/2304 [I 2022-10-28 07:04:38] ...trap/engines/engine.py.228: elapsed: 0:08:47 | left: 0:12:35 [I 2022-10-28 07:04:38] ...trap/engines/engine.py.229: process: 0.55238 | load: 0.00260 [I 2022-10-28 07:04:38] ...trap/engines/engine.py.230: loss: 0.13881 [I 2022-10-28 07:04:38] ...optimizers/trijoint.py.151: total_norm: 10.665524 [I 2022-10-28 07:04:44] ...trap/engines/engine.py.224: train: epoch 112 | batch 950/2304 [I 2022-10-28 07:04:44] ...trap/engines/engine.py.228: elapsed: 0:08:52 | left: 0:12:41 [I 2022-10-28 07:04:44] ...trap/engines/engine.py.229: process: 0.55555 | load: 0.00253 [I 2022-10-28 07:04:44] ...trap/engines/engine.py.230: loss: 0.10737 [I 2022-10-28 07:04:44] ...optimizers/trijoint.py.151: total_norm: 10.973116 [I 2022-10-28 07:04:50] ...trap/engines/engine.py.224: train: epoch 112 | batch 960/2304 [I 2022-10-28 07:04:50] ...trap/engines/engine.py.228: elapsed: 0:08:58 | left: 0:12:29 [I 2022-10-28 07:04:50] ...trap/engines/engine.py.229: process: 0.56258 | load: 0.00250 [I 2022-10-28 07:04:50] ...trap/engines/engine.py.230: loss: 0.12218 [I 2022-10-28 07:04:50] ...optimizers/trijoint.py.151: total_norm: 12.590346 [I 2022-10-28 07:04:55] ...trap/engines/engine.py.224: train: epoch 112 | batch 970/2304 [I 2022-10-28 07:04:55] ...trap/engines/engine.py.228: elapsed: 0:09:03 | left: 0:12:16 [I 2022-10-28 07:04:55] ...trap/engines/engine.py.229: process: 0.56706 | load: 0.00245 [I 2022-10-28 07:04:55] ...trap/engines/engine.py.230: loss: 0.22147 [I 2022-10-28 07:04:55] ...optimizers/trijoint.py.151: total_norm: 31.900124 [I 2022-10-28 07:05:01] ...trap/engines/engine.py.224: train: epoch 112 | batch 980/2304 [I 2022-10-28 07:05:01] ...trap/engines/engine.py.228: elapsed: 0:09:09 | left: 0:12:18 [I 2022-10-28 07:05:01] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00250 [I 2022-10-28 07:05:01] ...trap/engines/engine.py.230: loss: 0.09446 [I 2022-10-28 07:05:01] ...optimizers/trijoint.py.151: total_norm: 13.888034 [I 2022-10-28 07:05:06] ...trap/engines/engine.py.224: train: epoch 112 | batch 990/2304 [I 2022-10-28 07:05:06] ...trap/engines/engine.py.228: elapsed: 0:09:15 | left: 0:12:07 [I 2022-10-28 07:05:06] ...trap/engines/engine.py.229: process: 0.54044 | load: 0.00243 [I 2022-10-28 07:05:06] ...trap/engines/engine.py.230: loss: 0.10258 [I 2022-10-28 07:05:06] ...optimizers/trijoint.py.151: total_norm: 12.546474 [I 2022-10-28 07:05:12] ...trap/engines/engine.py.224: train: epoch 112 | batch 1000/2304 [I 2022-10-28 07:05:12] ...trap/engines/engine.py.228: elapsed: 0:09:20 | left: 0:12:08 [I 2022-10-28 07:05:12] ...trap/engines/engine.py.229: process: 0.56696 | load: 0.00247 [I 2022-10-28 07:05:12] ...trap/engines/engine.py.230: loss: 0.10683 [I 2022-10-28 07:05:12] ...optimizers/trijoint.py.151: total_norm: 16.928253 [I 2022-10-28 07:05:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 1010/2304 [I 2022-10-28 07:05:17] ...trap/engines/engine.py.228: elapsed: 0:09:26 | left: 0:11:58 [I 2022-10-28 07:05:17] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00244 [I 2022-10-28 07:05:17] ...trap/engines/engine.py.230: loss: 0.11095 [I 2022-10-28 07:05:17] ...optimizers/trijoint.py.151: total_norm: 11.286195 [I 2022-10-28 07:05:23] ...trap/engines/engine.py.224: train: epoch 112 | batch 1020/2304 [I 2022-10-28 07:05:23] ...trap/engines/engine.py.228: elapsed: 0:09:31 | left: 0:11:57 [I 2022-10-28 07:05:23] ...trap/engines/engine.py.229: process: 0.55366 | load: 0.00250 [I 2022-10-28 07:05:23] ...trap/engines/engine.py.230: loss: 0.08178 [I 2022-10-28 07:05:23] ...optimizers/trijoint.py.151: total_norm: 12.534558 [I 2022-10-28 07:05:29] ...trap/engines/engine.py.224: train: epoch 112 | batch 1030/2304 [I 2022-10-28 07:05:29] ...trap/engines/engine.py.228: elapsed: 0:09:37 | left: 0:11:45 [I 2022-10-28 07:05:29] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00243 [I 2022-10-28 07:05:29] ...trap/engines/engine.py.230: loss: 0.08950 [I 2022-10-28 07:05:29] ...optimizers/trijoint.py.151: total_norm: 12.776681 [I 2022-10-28 07:05:34] ...trap/engines/engine.py.224: train: epoch 112 | batch 1040/2304 [I 2022-10-28 07:05:34] ...trap/engines/engine.py.228: elapsed: 0:09:42 | left: 0:11:43 [I 2022-10-28 07:05:34] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00245 [I 2022-10-28 07:05:34] ...trap/engines/engine.py.230: loss: 0.07771 [I 2022-10-28 07:05:34] ...optimizers/trijoint.py.151: total_norm: 19.485992 [I 2022-10-28 07:05:40] ...trap/engines/engine.py.224: train: epoch 112 | batch 1050/2304 [I 2022-10-28 07:05:40] ...trap/engines/engine.py.228: elapsed: 0:09:48 | left: 0:11:35 [I 2022-10-28 07:05:40] ...trap/engines/engine.py.229: process: 0.55190 | load: 0.00239 [I 2022-10-28 07:05:40] ...trap/engines/engine.py.230: loss: 0.12123 [I 2022-10-28 07:05:40] ...optimizers/trijoint.py.151: total_norm: 18.698965 [I 2022-10-28 07:05:45] ...trap/engines/engine.py.224: train: epoch 112 | batch 1060/2304 [I 2022-10-28 07:05:45] ...trap/engines/engine.py.228: elapsed: 0:09:54 | left: 0:11:30 [I 2022-10-28 07:05:45] ...trap/engines/engine.py.229: process: 0.54403 | load: 0.00244 [I 2022-10-28 07:05:45] ...trap/engines/engine.py.230: loss: 0.11879 [I 2022-10-28 07:05:45] ...optimizers/trijoint.py.151: total_norm: 11.713962 [I 2022-10-28 07:05:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 1070/2304 [I 2022-10-28 07:05:51] ...trap/engines/engine.py.228: elapsed: 0:09:59 | left: 0:11:29 [I 2022-10-28 07:05:51] ...trap/engines/engine.py.229: process: 0.56227 | load: 0.00243 [I 2022-10-28 07:05:51] ...trap/engines/engine.py.230: loss: 0.17171 [I 2022-10-28 07:05:51] ...optimizers/trijoint.py.151: total_norm: 16.765179 [I 2022-10-28 07:05:56] ...trap/engines/engine.py.224: train: epoch 112 | batch 1080/2304 [I 2022-10-28 07:05:56] ...trap/engines/engine.py.228: elapsed: 0:10:05 | left: 0:11:21 [I 2022-10-28 07:05:56] ...trap/engines/engine.py.229: process: 0.55634 | load: 0.00249 [I 2022-10-28 07:05:56] ...trap/engines/engine.py.230: loss: 0.11507 [I 2022-10-28 07:05:56] ...optimizers/trijoint.py.151: total_norm: 21.222784 [I 2022-10-28 07:06:02] ...trap/engines/engine.py.224: train: epoch 112 | batch 1090/2304 [I 2022-10-28 07:06:02] ...trap/engines/engine.py.228: elapsed: 0:10:10 | left: 0:11:10 [I 2022-10-28 07:06:02] ...trap/engines/engine.py.229: process: 0.55184 | load: 0.00248 [I 2022-10-28 07:06:02] ...trap/engines/engine.py.230: loss: 0.16879 [I 2022-10-28 07:06:02] ...optimizers/trijoint.py.151: total_norm: 20.334824 [I 2022-10-28 07:06:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 1100/2304 [I 2022-10-28 07:06:08] ...trap/engines/engine.py.228: elapsed: 0:10:16 | left: 0:11:08 [I 2022-10-28 07:06:08] ...trap/engines/engine.py.229: process: 0.55775 | load: 0.00244 [I 2022-10-28 07:06:08] ...trap/engines/engine.py.230: loss: 0.09696 [I 2022-10-28 07:06:08] ...optimizers/trijoint.py.151: total_norm: 10.891775 [I 2022-10-28 07:06:13] ...trap/engines/engine.py.224: train: epoch 112 | batch 1110/2304 [I 2022-10-28 07:06:13] ...trap/engines/engine.py.228: elapsed: 0:10:22 | left: 0:11:07 [I 2022-10-28 07:06:13] ...trap/engines/engine.py.229: process: 0.57056 | load: 0.00246 [I 2022-10-28 07:06:13] ...trap/engines/engine.py.230: loss: 0.09932 [I 2022-10-28 07:06:13] ...optimizers/trijoint.py.151: total_norm: 24.979389 [I 2022-10-28 07:06:19] ...trap/engines/engine.py.224: train: epoch 112 | batch 1120/2304 [I 2022-10-28 07:06:19] ...trap/engines/engine.py.228: elapsed: 0:10:27 | left: 0:11:01 [I 2022-10-28 07:06:19] ...trap/engines/engine.py.229: process: 0.56183 | load: 0.00248 [I 2022-10-28 07:06:19] ...trap/engines/engine.py.230: loss: 0.09003 [I 2022-10-28 07:06:19] ...optimizers/trijoint.py.151: total_norm: 19.853079 [I 2022-10-28 07:06:24] ...trap/engines/engine.py.224: train: epoch 112 | batch 1130/2304 [I 2022-10-28 07:06:24] ...trap/engines/engine.py.228: elapsed: 0:10:33 | left: 0:10:54 [I 2022-10-28 07:06:24] ...trap/engines/engine.py.229: process: 0.55875 | load: 0.00252 [I 2022-10-28 07:06:24] ...trap/engines/engine.py.230: loss: 0.12264 [I 2022-10-28 07:06:24] ...optimizers/trijoint.py.151: total_norm: 17.500982 [I 2022-10-28 07:06:30] ...trap/engines/engine.py.224: train: epoch 112 | batch 1140/2304 [I 2022-10-28 07:06:30] ...trap/engines/engine.py.228: elapsed: 0:10:38 | left: 0:10:46 [I 2022-10-28 07:06:30] ...trap/engines/engine.py.229: process: 0.55636 | load: 0.00249 [I 2022-10-28 07:06:30] ...trap/engines/engine.py.230: loss: 0.11633 [I 2022-10-28 07:06:30] ...optimizers/trijoint.py.151: total_norm: 12.613868 [I 2022-10-28 07:06:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 1150/2304 [I 2022-10-28 07:06:35] ...trap/engines/engine.py.228: elapsed: 0:10:44 | left: 0:10:40 [I 2022-10-28 07:06:35] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00249 [I 2022-10-28 07:06:35] ...trap/engines/engine.py.230: loss: 0.11513 [I 2022-10-28 07:06:35] ...optimizers/trijoint.py.151: total_norm: 9.779986 [I 2022-10-28 07:06:41] ...trap/engines/engine.py.224: train: epoch 112 | batch 1160/2304 [I 2022-10-28 07:06:41] ...trap/engines/engine.py.228: elapsed: 0:10:49 | left: 0:10:37 [I 2022-10-28 07:06:41] ...trap/engines/engine.py.229: process: 0.54176 | load: 0.00258 [I 2022-10-28 07:06:41] ...trap/engines/engine.py.230: loss: 0.11301 [I 2022-10-28 07:06:41] ...optimizers/trijoint.py.151: total_norm: 10.418638 [I 2022-10-28 07:06:47] ...trap/engines/engine.py.224: train: epoch 112 | batch 1170/2304 [I 2022-10-28 07:06:47] ...trap/engines/engine.py.228: elapsed: 0:10:55 | left: 0:10:26 [I 2022-10-28 07:06:47] ...trap/engines/engine.py.229: process: 0.54251 | load: 0.00252 [I 2022-10-28 07:06:47] ...trap/engines/engine.py.230: loss: 0.10331 [I 2022-10-28 07:06:47] ...optimizers/trijoint.py.151: total_norm: 14.609930 [I 2022-10-28 07:06:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1180/2304 [I 2022-10-28 07:06:52] ...trap/engines/engine.py.228: elapsed: 0:11:01 | left: 0:10:34 [I 2022-10-28 07:06:52] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00252 [I 2022-10-28 07:06:52] ...trap/engines/engine.py.230: loss: 0.15409 [I 2022-10-28 07:06:52] ...optimizers/trijoint.py.151: total_norm: 12.307090 [I 2022-10-28 07:06:58] ...trap/engines/engine.py.224: train: epoch 112 | batch 1190/2304 [I 2022-10-28 07:06:58] ...trap/engines/engine.py.228: elapsed: 0:11:06 | left: 0:10:15 [I 2022-10-28 07:06:58] ...trap/engines/engine.py.229: process: 0.54296 | load: 0.00258 [I 2022-10-28 07:06:58] ...trap/engines/engine.py.230: loss: 0.13707 [I 2022-10-28 07:06:58] ...optimizers/trijoint.py.151: total_norm: 18.702303 [I 2022-10-28 07:07:03] ...trap/engines/engine.py.224: train: epoch 112 | batch 1200/2304 [I 2022-10-28 07:07:03] ...trap/engines/engine.py.228: elapsed: 0:11:12 | left: 0:10:29 [I 2022-10-28 07:07:03] ...trap/engines/engine.py.229: process: 0.55965 | load: 0.00251 [I 2022-10-28 07:07:03] ...trap/engines/engine.py.230: loss: 0.09372 [I 2022-10-28 07:07:03] ...optimizers/trijoint.py.151: total_norm: 19.846867 [I 2022-10-28 07:07:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1210/2304 [I 2022-10-28 07:07:09] ...trap/engines/engine.py.228: elapsed: 0:11:17 | left: 0:10:09 [I 2022-10-28 07:07:09] ...trap/engines/engine.py.229: process: 0.55138 | load: 0.00247 [I 2022-10-28 07:07:09] ...trap/engines/engine.py.230: loss: 0.12013 [I 2022-10-28 07:07:09] ...optimizers/trijoint.py.151: total_norm: 13.286104 [I 2022-10-28 07:07:15] ...trap/engines/engine.py.224: train: epoch 112 | batch 1220/2304 [I 2022-10-28 07:07:15] ...trap/engines/engine.py.228: elapsed: 0:11:23 | left: 0:10:01 [I 2022-10-28 07:07:15] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00242 [I 2022-10-28 07:07:15] ...trap/engines/engine.py.230: loss: 0.09208 [I 2022-10-28 07:07:15] ...optimizers/trijoint.py.151: total_norm: 12.271422 [I 2022-10-28 07:07:20] ...trap/engines/engine.py.224: train: epoch 112 | batch 1230/2304 [I 2022-10-28 07:07:20] ...trap/engines/engine.py.228: elapsed: 0:11:29 | left: 0:09:54 [I 2022-10-28 07:07:20] ...trap/engines/engine.py.229: process: 0.54224 | load: 0.00247 [I 2022-10-28 07:07:20] ...trap/engines/engine.py.230: loss: 0.13641 [I 2022-10-28 07:07:20] ...optimizers/trijoint.py.151: total_norm: 20.259016 [I 2022-10-28 07:07:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 1240/2304 [I 2022-10-28 07:07:26] ...trap/engines/engine.py.228: elapsed: 0:11:34 | left: 0:09:50 [I 2022-10-28 07:07:26] ...trap/engines/engine.py.229: process: 0.56147 | load: 0.00244 [I 2022-10-28 07:07:26] ...trap/engines/engine.py.230: loss: 0.08830 [I 2022-10-28 07:07:26] ...optimizers/trijoint.py.151: total_norm: 11.582222 [I 2022-10-28 07:07:31] ...trap/engines/engine.py.224: train: epoch 112 | batch 1250/2304 [I 2022-10-28 07:07:31] ...trap/engines/engine.py.228: elapsed: 0:11:40 | left: 0:09:46 [I 2022-10-28 07:07:31] ...trap/engines/engine.py.229: process: 0.56185 | load: 0.00246 [I 2022-10-28 07:07:31] ...trap/engines/engine.py.230: loss: 0.09996 [I 2022-10-28 07:07:31] ...optimizers/trijoint.py.151: total_norm: 13.370106 [I 2022-10-28 07:07:37] ...trap/engines/engine.py.224: train: epoch 112 | batch 1260/2304 [I 2022-10-28 07:07:37] ...trap/engines/engine.py.228: elapsed: 0:11:45 | left: 0:09:41 [I 2022-10-28 07:07:37] ...trap/engines/engine.py.229: process: 0.54802 | load: 0.00241 [I 2022-10-28 07:07:37] ...trap/engines/engine.py.230: loss: 0.12261 [I 2022-10-28 07:07:37] ...optimizers/trijoint.py.151: total_norm: 8.173178 [I 2022-10-28 07:07:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 1270/2304 [I 2022-10-28 07:07:43] ...trap/engines/engine.py.228: elapsed: 0:11:51 | left: 0:09:37 [I 2022-10-28 07:07:43] ...trap/engines/engine.py.229: process: 0.55988 | load: 0.00264 [I 2022-10-28 07:07:43] ...trap/engines/engine.py.230: loss: 0.06549 [I 2022-10-28 07:07:43] ...optimizers/trijoint.py.151: total_norm: 21.211466 [I 2022-10-28 07:07:48] ...trap/engines/engine.py.224: train: epoch 112 | batch 1280/2304 [I 2022-10-28 07:07:48] ...trap/engines/engine.py.228: elapsed: 0:11:57 | left: 0:09:32 [I 2022-10-28 07:07:48] ...trap/engines/engine.py.229: process: 0.55078 | load: 0.00246 [I 2022-10-28 07:07:48] ...trap/engines/engine.py.230: loss: 0.17141 [I 2022-10-28 07:07:48] ...optimizers/trijoint.py.151: total_norm: 13.818887 [I 2022-10-28 07:07:54] ...trap/engines/engine.py.224: train: epoch 112 | batch 1290/2304 [I 2022-10-28 07:07:54] ...trap/engines/engine.py.228: elapsed: 0:12:02 | left: 0:09:27 [I 2022-10-28 07:07:54] ...trap/engines/engine.py.229: process: 0.56263 | load: 0.00249 [I 2022-10-28 07:07:54] ...trap/engines/engine.py.230: loss: 0.14911 [I 2022-10-28 07:07:54] ...optimizers/trijoint.py.151: total_norm: 12.943661 [I 2022-10-28 07:07:59] ...trap/engines/engine.py.224: train: epoch 112 | batch 1300/2304 [I 2022-10-28 07:07:59] ...trap/engines/engine.py.228: elapsed: 0:12:08 | left: 0:09:19 [I 2022-10-28 07:07:59] ...trap/engines/engine.py.229: process: 0.56422 | load: 0.00278 [I 2022-10-28 07:07:59] ...trap/engines/engine.py.230: loss: 0.13567 [I 2022-10-28 07:07:59] ...optimizers/trijoint.py.151: total_norm: 15.601268 [I 2022-10-28 07:08:05] ...trap/engines/engine.py.224: train: epoch 112 | batch 1310/2304 [I 2022-10-28 07:08:05] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:14 [I 2022-10-28 07:08:05] ...trap/engines/engine.py.229: process: 0.55947 | load: 0.00246 [I 2022-10-28 07:08:05] ...trap/engines/engine.py.230: loss: 0.16830 [I 2022-10-28 07:08:05] ...optimizers/trijoint.py.151: total_norm: 53.116756 [I 2022-10-28 07:08:11] ...trap/engines/engine.py.224: train: epoch 112 | batch 1320/2304 [I 2022-10-28 07:08:11] ...trap/engines/engine.py.228: elapsed: 0:12:19 | left: 0:09:09 [I 2022-10-28 07:08:11] ...trap/engines/engine.py.229: process: 0.56089 | load: 0.00250 [I 2022-10-28 07:08:11] ...trap/engines/engine.py.230: loss: 0.12082 [I 2022-10-28 07:08:11] ...optimizers/trijoint.py.151: total_norm: 13.962708 [I 2022-10-28 07:08:16] ...trap/engines/engine.py.224: train: epoch 112 | batch 1330/2304 [I 2022-10-28 07:08:16] ...trap/engines/engine.py.228: elapsed: 0:12:25 | left: 0:09:04 [I 2022-10-28 07:08:16] ...trap/engines/engine.py.229: process: 0.56168 | load: 0.00253 [I 2022-10-28 07:08:16] ...trap/engines/engine.py.230: loss: 0.13256 [I 2022-10-28 07:08:16] ...optimizers/trijoint.py.151: total_norm: 10.250967 [I 2022-10-28 07:08:22] ...trap/engines/engine.py.224: train: epoch 112 | batch 1340/2304 [I 2022-10-28 07:08:22] ...trap/engines/engine.py.228: elapsed: 0:12:30 | left: 0:08:58 [I 2022-10-28 07:08:22] ...trap/engines/engine.py.229: process: 0.54434 | load: 0.00250 [I 2022-10-28 07:08:22] ...trap/engines/engine.py.230: loss: 0.12642 [I 2022-10-28 07:08:22] ...optimizers/trijoint.py.151: total_norm: 31.923634 [I 2022-10-28 07:08:27] ...trap/engines/engine.py.224: train: epoch 112 | batch 1350/2304 [I 2022-10-28 07:08:27] ...trap/engines/engine.py.228: elapsed: 0:12:36 | left: 0:08:50 [I 2022-10-28 07:08:27] ...trap/engines/engine.py.229: process: 0.56303 | load: 0.00249 [I 2022-10-28 07:08:27] ...trap/engines/engine.py.230: loss: 0.18426 [I 2022-10-28 07:08:27] ...optimizers/trijoint.py.151: total_norm: 22.328011 [I 2022-10-28 07:08:33] ...trap/engines/engine.py.224: train: epoch 112 | batch 1360/2304 [I 2022-10-28 07:08:33] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:08:43 [I 2022-10-28 07:08:33] ...trap/engines/engine.py.229: process: 0.55939 | load: 0.00240 [I 2022-10-28 07:08:33] ...trap/engines/engine.py.230: loss: 0.21562 [I 2022-10-28 07:08:33] ...optimizers/trijoint.py.151: total_norm: 49.796391 [I 2022-10-28 07:08:38] ...trap/engines/engine.py.224: train: epoch 112 | batch 1370/2304 [I 2022-10-28 07:08:38] ...trap/engines/engine.py.228: elapsed: 0:12:47 | left: 0:08:35 [I 2022-10-28 07:08:38] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00253 [I 2022-10-28 07:08:38] ...trap/engines/engine.py.230: loss: 0.14923 [I 2022-10-28 07:08:38] ...optimizers/trijoint.py.151: total_norm: 15.445063 [I 2022-10-28 07:08:44] ...trap/engines/engine.py.224: train: epoch 112 | batch 1380/2304 [I 2022-10-28 07:08:44] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:31 [I 2022-10-28 07:08:44] ...trap/engines/engine.py.229: process: 0.55552 | load: 0.00251 [I 2022-10-28 07:08:44] ...trap/engines/engine.py.230: loss: 0.10455 [I 2022-10-28 07:08:44] ...optimizers/trijoint.py.151: total_norm: 15.080136 [I 2022-10-28 07:08:50] ...trap/engines/engine.py.224: train: epoch 112 | batch 1390/2304 [I 2022-10-28 07:08:50] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:32 [I 2022-10-28 07:08:50] ...trap/engines/engine.py.229: process: 0.58415 | load: 0.00252 [I 2022-10-28 07:08:50] ...trap/engines/engine.py.230: loss: 0.11614 [I 2022-10-28 07:08:50] ...optimizers/trijoint.py.151: total_norm: 19.635214 [I 2022-10-28 07:08:55] ...trap/engines/engine.py.224: train: epoch 112 | batch 1400/2304 [I 2022-10-28 07:08:55] ...trap/engines/engine.py.228: elapsed: 0:13:04 | left: 0:08:23 [I 2022-10-28 07:08:55] ...trap/engines/engine.py.229: process: 0.55897 | load: 0.00249 [I 2022-10-28 07:08:55] ...trap/engines/engine.py.230: loss: 0.20824 [I 2022-10-28 07:08:55] ...optimizers/trijoint.py.151: total_norm: 20.241192 [I 2022-10-28 07:09:01] ...trap/engines/engine.py.224: train: epoch 112 | batch 1410/2304 [I 2022-10-28 07:09:01] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:12 [I 2022-10-28 07:09:01] ...trap/engines/engine.py.229: process: 0.55039 | load: 0.00249 [I 2022-10-28 07:09:01] ...trap/engines/engine.py.230: loss: 0.15311 [I 2022-10-28 07:09:01] ...optimizers/trijoint.py.151: total_norm: 14.840948 [I 2022-10-28 07:09:06] ...trap/engines/engine.py.224: train: epoch 112 | batch 1420/2304 [I 2022-10-28 07:09:06] ...trap/engines/engine.py.228: elapsed: 0:13:15 | left: 0:08:12 [I 2022-10-28 07:09:06] ...trap/engines/engine.py.229: process: 0.57129 | load: 0.00240 [I 2022-10-28 07:09:06] ...trap/engines/engine.py.230: loss: 0.06640 [I 2022-10-28 07:09:06] ...optimizers/trijoint.py.151: total_norm: 22.678526 [I 2022-10-28 07:09:12] ...trap/engines/engine.py.224: train: epoch 112 | batch 1430/2304 [I 2022-10-28 07:09:12] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:06 [I 2022-10-28 07:09:12] ...trap/engines/engine.py.229: process: 0.55857 | load: 0.00246 [I 2022-10-28 07:09:12] ...trap/engines/engine.py.230: loss: 0.10583 [I 2022-10-28 07:09:12] ...optimizers/trijoint.py.151: total_norm: 16.668489 [I 2022-10-28 07:09:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 1440/2304 [I 2022-10-28 07:09:17] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:01 [I 2022-10-28 07:09:17] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00242 [I 2022-10-28 07:09:17] ...trap/engines/engine.py.230: loss: 0.08518 [I 2022-10-28 07:09:17] ...optimizers/trijoint.py.151: total_norm: 14.617895 [I 2022-10-28 07:09:23] ...trap/engines/engine.py.224: train: epoch 112 | batch 1450/2304 [I 2022-10-28 07:09:23] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:07 [I 2022-10-28 07:09:23] ...trap/engines/engine.py.229: process: 0.55287 | load: 0.00243 [I 2022-10-28 07:09:23] ...trap/engines/engine.py.230: loss: 0.09770 [I 2022-10-28 07:09:23] ...optimizers/trijoint.py.151: total_norm: 7.727954 [I 2022-10-28 07:09:29] ...trap/engines/engine.py.224: train: epoch 112 | batch 1460/2304 [I 2022-10-28 07:09:29] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:07:52 [I 2022-10-28 07:09:29] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00245 [I 2022-10-28 07:09:29] ...trap/engines/engine.py.230: loss: 0.10885 [I 2022-10-28 07:09:29] ...optimizers/trijoint.py.151: total_norm: 28.491688 [I 2022-10-28 07:09:34] ...trap/engines/engine.py.224: train: epoch 112 | batch 1470/2304 [I 2022-10-28 07:09:34] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:07:43 [I 2022-10-28 07:09:34] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00241 [I 2022-10-28 07:09:34] ...trap/engines/engine.py.230: loss: 0.07769 [I 2022-10-28 07:09:34] ...optimizers/trijoint.py.151: total_norm: 18.279057 [I 2022-10-28 07:09:40] ...trap/engines/engine.py.224: train: epoch 112 | batch 1480/2304 [I 2022-10-28 07:09:40] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:40 [I 2022-10-28 07:09:40] ...trap/engines/engine.py.229: process: 0.56554 | load: 0.00248 [I 2022-10-28 07:09:40] ...trap/engines/engine.py.230: loss: 0.16695 [I 2022-10-28 07:09:40] ...optimizers/trijoint.py.151: total_norm: 12.789423 [I 2022-10-28 07:09:45] ...trap/engines/engine.py.224: train: epoch 112 | batch 1490/2304 [I 2022-10-28 07:09:45] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:33 [I 2022-10-28 07:09:45] ...trap/engines/engine.py.229: process: 0.55664 | load: 0.00240 [I 2022-10-28 07:09:45] ...trap/engines/engine.py.230: loss: 0.19069 [I 2022-10-28 07:09:45] ...optimizers/trijoint.py.151: total_norm: 43.025517 [I 2022-10-28 07:09:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 1500/2304 [I 2022-10-28 07:09:51] ...trap/engines/engine.py.228: elapsed: 0:13:59 | left: 0:07:28 [I 2022-10-28 07:09:51] ...trap/engines/engine.py.229: process: 0.56127 | load: 0.00248 [I 2022-10-28 07:09:51] ...trap/engines/engine.py.230: loss: 0.12823 [I 2022-10-28 07:09:51] ...optimizers/trijoint.py.151: total_norm: 27.457354 [I 2022-10-28 07:09:57] ...trap/engines/engine.py.224: train: epoch 112 | batch 1510/2304 [I 2022-10-28 07:09:57] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:23 [I 2022-10-28 07:09:57] ...trap/engines/engine.py.229: process: 0.56491 | load: 0.00247 [I 2022-10-28 07:09:57] ...trap/engines/engine.py.230: loss: 0.18805 [I 2022-10-28 07:09:57] ...optimizers/trijoint.py.151: total_norm: 19.861700 [I 2022-10-28 07:10:02] ...trap/engines/engine.py.224: train: epoch 112 | batch 1520/2304 [I 2022-10-28 07:10:02] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:17 [I 2022-10-28 07:10:02] ...trap/engines/engine.py.229: process: 0.55968 | load: 0.00243 [I 2022-10-28 07:10:02] ...trap/engines/engine.py.230: loss: 0.10372 [I 2022-10-28 07:10:02] ...optimizers/trijoint.py.151: total_norm: 13.083421 [I 2022-10-28 07:10:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 1530/2304 [I 2022-10-28 07:10:08] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:11 [I 2022-10-28 07:10:08] ...trap/engines/engine.py.229: process: 0.55979 | load: 0.00240 [I 2022-10-28 07:10:08] ...trap/engines/engine.py.230: loss: 0.07898 [I 2022-10-28 07:10:08] ...optimizers/trijoint.py.151: total_norm: 11.238600 [I 2022-10-28 07:10:13] ...trap/engines/engine.py.224: train: epoch 112 | batch 1540/2304 [I 2022-10-28 07:10:13] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:02 [I 2022-10-28 07:10:13] ...trap/engines/engine.py.229: process: 0.54640 | load: 0.00246 [I 2022-10-28 07:10:13] ...trap/engines/engine.py.230: loss: 0.12081 [I 2022-10-28 07:10:13] ...optimizers/trijoint.py.151: total_norm: 12.018556 [I 2022-10-28 07:10:19] ...trap/engines/engine.py.224: train: epoch 112 | batch 1550/2304 [I 2022-10-28 07:10:19] ...trap/engines/engine.py.228: elapsed: 0:14:27 | left: 0:06:57 [I 2022-10-28 07:10:19] ...trap/engines/engine.py.229: process: 0.54726 | load: 0.00244 [I 2022-10-28 07:10:19] ...trap/engines/engine.py.230: loss: 0.13313 [I 2022-10-28 07:10:19] ...optimizers/trijoint.py.151: total_norm: 17.315786 [I 2022-10-28 07:10:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 1560/2304 [I 2022-10-28 07:10:25] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:06:53 [I 2022-10-28 07:10:25] ...trap/engines/engine.py.229: process: 0.54666 | load: 0.00240 [I 2022-10-28 07:10:25] ...trap/engines/engine.py.230: loss: 0.10710 [I 2022-10-28 07:10:25] ...optimizers/trijoint.py.151: total_norm: 18.433302 [I 2022-10-28 07:10:30] ...trap/engines/engine.py.224: train: epoch 112 | batch 1570/2304 [I 2022-10-28 07:10:30] ...trap/engines/engine.py.228: elapsed: 0:14:38 | left: 0:06:45 [I 2022-10-28 07:10:30] ...trap/engines/engine.py.229: process: 0.54867 | load: 0.00245 [I 2022-10-28 07:10:30] ...trap/engines/engine.py.230: loss: 0.15848 [I 2022-10-28 07:10:30] ...optimizers/trijoint.py.151: total_norm: 21.965994 [I 2022-10-28 07:10:36] ...trap/engines/engine.py.224: train: epoch 112 | batch 1580/2304 [I 2022-10-28 07:10:36] ...trap/engines/engine.py.228: elapsed: 0:14:44 | left: 0:06:41 [I 2022-10-28 07:10:36] ...trap/engines/engine.py.229: process: 0.55023 | load: 0.00240 [I 2022-10-28 07:10:36] ...trap/engines/engine.py.230: loss: 0.09254 [I 2022-10-28 07:10:36] ...optimizers/trijoint.py.151: total_norm: 10.248096 [I 2022-10-28 07:10:41] ...trap/engines/engine.py.224: train: epoch 112 | batch 1590/2304 [I 2022-10-28 07:10:41] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:33 [I 2022-10-28 07:10:41] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00252 [I 2022-10-28 07:10:41] ...trap/engines/engine.py.230: loss: 0.10013 [I 2022-10-28 07:10:41] ...optimizers/trijoint.py.151: total_norm: 13.226284 [I 2022-10-28 07:10:47] ...trap/engines/engine.py.224: train: epoch 112 | batch 1600/2304 [I 2022-10-28 07:10:47] ...trap/engines/engine.py.228: elapsed: 0:14:55 | left: 0:06:30 [I 2022-10-28 07:10:47] ...trap/engines/engine.py.229: process: 0.54434 | load: 0.00247 [I 2022-10-28 07:10:47] ...trap/engines/engine.py.230: loss: 0.10552 [I 2022-10-28 07:10:47] ...optimizers/trijoint.py.151: total_norm: 10.862299 [I 2022-10-28 07:10:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 1610/2304 [I 2022-10-28 07:10:52] ...trap/engines/engine.py.228: elapsed: 0:15:01 | left: 0:06:23 [I 2022-10-28 07:10:52] ...trap/engines/engine.py.229: process: 0.54065 | load: 0.00245 [I 2022-10-28 07:10:52] ...trap/engines/engine.py.230: loss: 0.10956 [I 2022-10-28 07:10:52] ...optimizers/trijoint.py.151: total_norm: 16.098331 [I 2022-10-28 07:10:58] ...trap/engines/engine.py.224: train: epoch 112 | batch 1620/2304 [I 2022-10-28 07:10:58] ...trap/engines/engine.py.228: elapsed: 0:15:06 | left: 0:06:21 [I 2022-10-28 07:10:58] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00243 [I 2022-10-28 07:10:58] ...trap/engines/engine.py.230: loss: 0.10680 [I 2022-10-28 07:10:58] ...optimizers/trijoint.py.151: total_norm: 30.006096 [I 2022-10-28 07:11:03] ...trap/engines/engine.py.224: train: epoch 112 | batch 1630/2304 [I 2022-10-28 07:11:03] ...trap/engines/engine.py.228: elapsed: 0:15:12 | left: 0:06:12 [I 2022-10-28 07:11:03] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00246 [I 2022-10-28 07:11:03] ...trap/engines/engine.py.230: loss: 0.08153 [I 2022-10-28 07:11:03] ...optimizers/trijoint.py.151: total_norm: 8.875319 [I 2022-10-28 07:11:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1640/2304 [I 2022-10-28 07:11:09] ...trap/engines/engine.py.228: elapsed: 0:15:17 | left: 0:06:03 [I 2022-10-28 07:11:09] ...trap/engines/engine.py.229: process: 0.54584 | load: 0.00241 [I 2022-10-28 07:11:09] ...trap/engines/engine.py.230: loss: 0.09544 [I 2022-10-28 07:11:09] ...optimizers/trijoint.py.151: total_norm: 8.494558 [I 2022-10-28 07:11:14] ...trap/engines/engine.py.224: train: epoch 112 | batch 1650/2304 [I 2022-10-28 07:11:14] ...trap/engines/engine.py.228: elapsed: 0:15:23 | left: 0:06:01 [I 2022-10-28 07:11:14] ...trap/engines/engine.py.229: process: 0.54858 | load: 0.00245 [I 2022-10-28 07:11:14] ...trap/engines/engine.py.230: loss: 0.11271 [I 2022-10-28 07:11:14] ...optimizers/trijoint.py.151: total_norm: 9.269859 [I 2022-10-28 07:11:20] ...trap/engines/engine.py.224: train: epoch 112 | batch 1660/2304 [I 2022-10-28 07:11:20] ...trap/engines/engine.py.228: elapsed: 0:15:28 | left: 0:05:55 [I 2022-10-28 07:11:20] ...trap/engines/engine.py.229: process: 0.55099 | load: 0.00245 [I 2022-10-28 07:11:20] ...trap/engines/engine.py.230: loss: 0.16003 [I 2022-10-28 07:11:20] ...optimizers/trijoint.py.151: total_norm: 24.411007 [I 2022-10-28 07:11:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 1670/2304 [I 2022-10-28 07:11:25] ...trap/engines/engine.py.228: elapsed: 0:15:34 | left: 0:05:50 [I 2022-10-28 07:11:25] ...trap/engines/engine.py.229: process: 0.54470 | load: 0.00256 [I 2022-10-28 07:11:25] ...trap/engines/engine.py.230: loss: 0.10536 [I 2022-10-28 07:11:25] ...optimizers/trijoint.py.151: total_norm: 8.261832 [I 2022-10-28 07:11:31] ...trap/engines/engine.py.224: train: epoch 112 | batch 1680/2304 [I 2022-10-28 07:11:31] ...trap/engines/engine.py.228: elapsed: 0:15:39 | left: 0:05:48 [I 2022-10-28 07:11:31] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00241 [I 2022-10-28 07:11:31] ...trap/engines/engine.py.230: loss: 0.10245 [I 2022-10-28 07:11:31] ...optimizers/trijoint.py.151: total_norm: 9.624208 [I 2022-10-28 07:11:37] ...trap/engines/engine.py.224: train: epoch 112 | batch 1690/2304 [I 2022-10-28 07:11:37] ...trap/engines/engine.py.228: elapsed: 0:15:45 | left: 0:05:40 [I 2022-10-28 07:11:37] ...trap/engines/engine.py.229: process: 0.54521 | load: 0.00251 [I 2022-10-28 07:11:37] ...trap/engines/engine.py.230: loss: 0.10086 [I 2022-10-28 07:11:37] ...optimizers/trijoint.py.151: total_norm: 23.560150 [I 2022-10-28 07:11:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 1700/2304 [I 2022-10-28 07:11:42] ...trap/engines/engine.py.228: elapsed: 0:15:51 | left: 0:05:37 [I 2022-10-28 07:11:42] ...trap/engines/engine.py.229: process: 0.56332 | load: 0.00252 [I 2022-10-28 07:11:42] ...trap/engines/engine.py.230: loss: 0.11303 [I 2022-10-28 07:11:42] ...optimizers/trijoint.py.151: total_norm: 26.139061 [I 2022-10-28 07:11:48] ...trap/engines/engine.py.224: train: epoch 112 | batch 1710/2304 [I 2022-10-28 07:11:48] ...trap/engines/engine.py.228: elapsed: 0:15:56 | left: 0:05:28 [I 2022-10-28 07:11:48] ...trap/engines/engine.py.229: process: 0.54254 | load: 0.00245 [I 2022-10-28 07:11:48] ...trap/engines/engine.py.230: loss: 0.15504 [I 2022-10-28 07:11:48] ...optimizers/trijoint.py.151: total_norm: 14.314765 [I 2022-10-28 07:11:53] ...trap/engines/engine.py.224: train: epoch 112 | batch 1720/2304 [I 2022-10-28 07:11:53] ...trap/engines/engine.py.228: elapsed: 0:16:02 | left: 0:05:23 [I 2022-10-28 07:11:53] ...trap/engines/engine.py.229: process: 0.55376 | load: 0.00247 [I 2022-10-28 07:11:53] ...trap/engines/engine.py.230: loss: 0.09973 [I 2022-10-28 07:11:53] ...optimizers/trijoint.py.151: total_norm: 14.473847 [I 2022-10-28 07:12:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 1730/2304 [I 2022-10-28 07:12:18] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:17:08 [I 2022-10-28 07:12:18] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00241 [I 2022-10-28 07:12:18] ...trap/engines/engine.py.230: loss: 0.10657 [I 2022-10-28 07:12:18] ...optimizers/trijoint.py.151: total_norm: 27.731174 [I 2022-10-28 07:12:23] ...trap/engines/engine.py.224: train: epoch 112 | batch 1740/2304 [I 2022-10-28 07:12:23] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:06:26 [I 2022-10-28 07:12:23] ...trap/engines/engine.py.229: process: 0.54509 | load: 0.00237 [I 2022-10-28 07:12:23] ...trap/engines/engine.py.230: loss: 0.11545 [I 2022-10-28 07:12:23] ...optimizers/trijoint.py.151: total_norm: 25.750969 [I 2022-10-28 07:12:29] ...trap/engines/engine.py.224: train: epoch 112 | batch 1750/2304 [I 2022-10-28 07:12:29] ...trap/engines/engine.py.228: elapsed: 0:16:37 | left: 0:05:14 [I 2022-10-28 07:12:29] ...trap/engines/engine.py.229: process: 0.55234 | load: 0.00247 [I 2022-10-28 07:12:29] ...trap/engines/engine.py.230: loss: 0.18389 [I 2022-10-28 07:12:29] ...optimizers/trijoint.py.151: total_norm: 19.072666 [I 2022-10-28 07:12:34] ...trap/engines/engine.py.224: train: epoch 112 | batch 1760/2304 [I 2022-10-28 07:12:34] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:02 [I 2022-10-28 07:12:34] ...trap/engines/engine.py.229: process: 0.55483 | load: 0.00239 [I 2022-10-28 07:12:34] ...trap/engines/engine.py.230: loss: 0.08968 [I 2022-10-28 07:12:34] ...optimizers/trijoint.py.151: total_norm: 10.048966 [I 2022-10-28 07:12:40] ...trap/engines/engine.py.224: train: epoch 112 | batch 1770/2304 [I 2022-10-28 07:12:40] ...trap/engines/engine.py.228: elapsed: 0:16:48 | left: 0:04:56 [I 2022-10-28 07:12:40] ...trap/engines/engine.py.229: process: 0.55187 | load: 0.00238 [I 2022-10-28 07:12:40] ...trap/engines/engine.py.230: loss: 0.14622 [I 2022-10-28 07:12:40] ...optimizers/trijoint.py.151: total_norm: 8.472318 [I 2022-10-28 07:12:46] ...trap/engines/engine.py.224: train: epoch 112 | batch 1780/2304 [I 2022-10-28 07:12:46] ...trap/engines/engine.py.228: elapsed: 0:16:54 | left: 0:04:52 [I 2022-10-28 07:12:46] ...trap/engines/engine.py.229: process: 0.56734 | load: 0.00239 [I 2022-10-28 07:12:46] ...trap/engines/engine.py.230: loss: 0.09644 [I 2022-10-28 07:12:46] ...optimizers/trijoint.py.151: total_norm: 8.428067 [I 2022-10-28 07:12:51] ...trap/engines/engine.py.224: train: epoch 112 | batch 1790/2304 [I 2022-10-28 07:12:51] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:47 [I 2022-10-28 07:12:51] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00240 [I 2022-10-28 07:12:51] ...trap/engines/engine.py.230: loss: 0.10466 [I 2022-10-28 07:12:51] ...optimizers/trijoint.py.151: total_norm: 21.049049 [I 2022-10-28 07:12:57] ...trap/engines/engine.py.224: train: epoch 112 | batch 1800/2304 [I 2022-10-28 07:12:57] ...trap/engines/engine.py.228: elapsed: 0:17:05 | left: 0:04:41 [I 2022-10-28 07:12:57] ...trap/engines/engine.py.229: process: 0.56597 | load: 0.00242 [I 2022-10-28 07:12:57] ...trap/engines/engine.py.230: loss: 0.09258 [I 2022-10-28 07:12:57] ...optimizers/trijoint.py.151: total_norm: 9.046645 [I 2022-10-28 07:13:02] ...trap/engines/engine.py.224: train: epoch 112 | batch 1810/2304 [I 2022-10-28 07:13:02] ...trap/engines/engine.py.228: elapsed: 0:17:11 | left: 0:04:33 [I 2022-10-28 07:13:02] ...trap/engines/engine.py.229: process: 0.54881 | load: 0.00235 [I 2022-10-28 07:13:02] ...trap/engines/engine.py.230: loss: 0.14601 [I 2022-10-28 07:13:02] ...optimizers/trijoint.py.151: total_norm: 17.469404 [I 2022-10-28 07:13:08] ...trap/engines/engine.py.224: train: epoch 112 | batch 1820/2304 [I 2022-10-28 07:13:08] ...trap/engines/engine.py.228: elapsed: 0:17:16 | left: 0:04:30 [I 2022-10-28 07:13:08] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00233 [I 2022-10-28 07:13:08] ...trap/engines/engine.py.230: loss: 0.11253 [I 2022-10-28 07:13:08] ...optimizers/trijoint.py.151: total_norm: 17.491091 [I 2022-10-28 07:13:14] ...trap/engines/engine.py.224: train: epoch 112 | batch 1830/2304 [I 2022-10-28 07:13:14] ...trap/engines/engine.py.228: elapsed: 0:17:22 | left: 0:04:25 [I 2022-10-28 07:13:14] ...trap/engines/engine.py.229: process: 0.56347 | load: 0.00237 [I 2022-10-28 07:13:14] ...trap/engines/engine.py.230: loss: 0.09841 [I 2022-10-28 07:13:14] ...optimizers/trijoint.py.151: total_norm: 10.370288 [I 2022-10-28 07:13:19] ...trap/engines/engine.py.224: train: epoch 112 | batch 1840/2304 [I 2022-10-28 07:13:19] ...trap/engines/engine.py.228: elapsed: 0:17:28 | left: 0:04:18 [I 2022-10-28 07:13:19] ...trap/engines/engine.py.229: process: 0.56234 | load: 0.00238 [I 2022-10-28 07:13:19] ...trap/engines/engine.py.230: loss: 0.14653 [I 2022-10-28 07:13:19] ...optimizers/trijoint.py.151: total_norm: 30.848368 [I 2022-10-28 07:13:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 1850/2304 [I 2022-10-28 07:13:25] ...trap/engines/engine.py.228: elapsed: 0:17:33 | left: 0:04:12 [I 2022-10-28 07:13:25] ...trap/engines/engine.py.229: process: 0.55263 | load: 0.00244 [I 2022-10-28 07:13:25] ...trap/engines/engine.py.230: loss: 0.07701 [I 2022-10-28 07:13:25] ...optimizers/trijoint.py.151: total_norm: 16.670397 [I 2022-10-28 07:13:30] ...trap/engines/engine.py.224: train: epoch 112 | batch 1860/2304 [I 2022-10-28 07:13:30] ...trap/engines/engine.py.228: elapsed: 0:17:39 | left: 0:04:06 [I 2022-10-28 07:13:30] ...trap/engines/engine.py.229: process: 0.54259 | load: 0.00237 [I 2022-10-28 07:13:30] ...trap/engines/engine.py.230: loss: 0.11217 [I 2022-10-28 07:13:30] ...optimizers/trijoint.py.151: total_norm: 7.222697 [I 2022-10-28 07:13:36] ...trap/engines/engine.py.224: train: epoch 112 | batch 1870/2304 [I 2022-10-28 07:13:36] ...trap/engines/engine.py.228: elapsed: 0:17:44 | left: 0:04:01 [I 2022-10-28 07:13:36] ...trap/engines/engine.py.229: process: 0.55835 | load: 0.00245 [I 2022-10-28 07:13:36] ...trap/engines/engine.py.230: loss: 0.09887 [I 2022-10-28 07:13:36] ...optimizers/trijoint.py.151: total_norm: 10.691363 [I 2022-10-28 07:13:42] ...trap/engines/engine.py.224: train: epoch 112 | batch 1880/2304 [I 2022-10-28 07:13:42] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:03:55 [I 2022-10-28 07:13:42] ...trap/engines/engine.py.229: process: 0.53641 | load: 0.00239 [I 2022-10-28 07:13:42] ...trap/engines/engine.py.230: loss: 0.13340 [I 2022-10-28 07:13:42] ...optimizers/trijoint.py.151: total_norm: 27.820683 [I 2022-10-28 07:13:47] ...trap/engines/engine.py.224: train: epoch 112 | batch 1890/2304 [I 2022-10-28 07:13:47] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:03:50 [I 2022-10-28 07:13:47] ...trap/engines/engine.py.229: process: 0.55019 | load: 0.00237 [I 2022-10-28 07:13:47] ...trap/engines/engine.py.230: loss: 0.10107 [I 2022-10-28 07:13:47] ...optimizers/trijoint.py.151: total_norm: 25.420486 [I 2022-10-28 07:13:53] ...trap/engines/engine.py.224: train: epoch 112 | batch 1900/2304 [I 2022-10-28 07:13:53] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:03:43 [I 2022-10-28 07:13:53] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00244 [I 2022-10-28 07:13:53] ...trap/engines/engine.py.230: loss: 0.14580 [I 2022-10-28 07:13:53] ...optimizers/trijoint.py.151: total_norm: 20.155561 [I 2022-10-28 07:13:58] ...trap/engines/engine.py.224: train: epoch 112 | batch 1910/2304 [I 2022-10-28 07:13:58] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:39 [I 2022-10-28 07:13:58] ...trap/engines/engine.py.229: process: 0.54427 | load: 0.00240 [I 2022-10-28 07:13:58] ...trap/engines/engine.py.230: loss: 0.08003 [I 2022-10-28 07:13:58] ...optimizers/trijoint.py.151: total_norm: 9.351871 [I 2022-10-28 07:14:04] ...trap/engines/engine.py.224: train: epoch 112 | batch 1920/2304 [I 2022-10-28 07:14:04] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:33 [I 2022-10-28 07:14:04] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00239 [I 2022-10-28 07:14:04] ...trap/engines/engine.py.230: loss: 0.08729 [I 2022-10-28 07:14:04] ...optimizers/trijoint.py.151: total_norm: 24.197147 [I 2022-10-28 07:14:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 1930/2304 [I 2022-10-28 07:14:09] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:29 [I 2022-10-28 07:14:09] ...trap/engines/engine.py.229: process: 0.55870 | load: 0.00239 [I 2022-10-28 07:14:09] ...trap/engines/engine.py.230: loss: 0.09476 [I 2022-10-28 07:14:09] ...optimizers/trijoint.py.151: total_norm: 15.196239 [I 2022-10-28 07:14:15] ...trap/engines/engine.py.224: train: epoch 112 | batch 1940/2304 [I 2022-10-28 07:14:15] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:23 [I 2022-10-28 07:14:15] ...trap/engines/engine.py.229: process: 0.55111 | load: 0.00238 [I 2022-10-28 07:14:15] ...trap/engines/engine.py.230: loss: 0.10679 [I 2022-10-28 07:14:15] ...optimizers/trijoint.py.151: total_norm: 31.526691 [I 2022-10-28 07:14:21] ...trap/engines/engine.py.224: train: epoch 112 | batch 1950/2304 [I 2022-10-28 07:14:21] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:17 [I 2022-10-28 07:14:21] ...trap/engines/engine.py.229: process: 0.55206 | load: 0.00243 [I 2022-10-28 07:14:21] ...trap/engines/engine.py.230: loss: 0.18437 [I 2022-10-28 07:14:21] ...optimizers/trijoint.py.151: total_norm: 27.493753 [I 2022-10-28 07:14:26] ...trap/engines/engine.py.224: train: epoch 112 | batch 1960/2304 [I 2022-10-28 07:14:26] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:11 [I 2022-10-28 07:14:26] ...trap/engines/engine.py.229: process: 0.56423 | load: 0.00241 [I 2022-10-28 07:14:26] ...trap/engines/engine.py.230: loss: 0.08757 [I 2022-10-28 07:14:26] ...optimizers/trijoint.py.151: total_norm: 49.132843 [I 2022-10-28 07:14:32] ...trap/engines/engine.py.224: train: epoch 112 | batch 1970/2304 [I 2022-10-28 07:14:32] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:03:06 [I 2022-10-28 07:14:32] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00237 [I 2022-10-28 07:14:32] ...trap/engines/engine.py.230: loss: 0.08980 [I 2022-10-28 07:14:32] ...optimizers/trijoint.py.151: total_norm: 11.331492 [I 2022-10-28 07:14:37] ...trap/engines/engine.py.224: train: epoch 112 | batch 1980/2304 [I 2022-10-28 07:14:37] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:03:01 [I 2022-10-28 07:14:37] ...trap/engines/engine.py.229: process: 0.56441 | load: 0.00238 [I 2022-10-28 07:14:37] ...trap/engines/engine.py.230: loss: 0.10968 [I 2022-10-28 07:14:37] ...optimizers/trijoint.py.151: total_norm: 10.342117 [I 2022-10-28 07:14:43] ...trap/engines/engine.py.224: train: epoch 112 | batch 1990/2304 [I 2022-10-28 07:14:43] ...trap/engines/engine.py.228: elapsed: 0:18:51 | left: 0:02:54 [I 2022-10-28 07:14:43] ...trap/engines/engine.py.229: process: 0.53784 | load: 0.00243 [I 2022-10-28 07:14:43] ...trap/engines/engine.py.230: loss: 0.09225 [I 2022-10-28 07:14:43] ...optimizers/trijoint.py.151: total_norm: 7.382189 [I 2022-10-28 07:14:49] ...trap/engines/engine.py.224: train: epoch 112 | batch 2000/2304 [I 2022-10-28 07:14:49] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:50 [I 2022-10-28 07:14:49] ...trap/engines/engine.py.229: process: 0.56425 | load: 0.00238 [I 2022-10-28 07:14:49] ...trap/engines/engine.py.230: loss: 0.09890 [I 2022-10-28 07:14:49] ...optimizers/trijoint.py.151: total_norm: 15.411942 [I 2022-10-28 07:14:54] ...trap/engines/engine.py.224: train: epoch 112 | batch 2010/2304 [I 2022-10-28 07:14:54] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:44 [I 2022-10-28 07:14:54] ...trap/engines/engine.py.229: process: 0.56370 | load: 0.00242 [I 2022-10-28 07:14:54] ...trap/engines/engine.py.230: loss: 0.15997 [I 2022-10-28 07:14:54] ...optimizers/trijoint.py.151: total_norm: 16.926840 [I 2022-10-28 07:15:00] ...trap/engines/engine.py.224: train: epoch 112 | batch 2020/2304 [I 2022-10-28 07:15:00] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:39 [I 2022-10-28 07:15:00] ...trap/engines/engine.py.229: process: 0.56275 | load: 0.00243 [I 2022-10-28 07:15:00] ...trap/engines/engine.py.230: loss: 0.18130 [I 2022-10-28 07:15:00] ...optimizers/trijoint.py.151: total_norm: 23.009871 [I 2022-10-28 07:15:05] ...trap/engines/engine.py.224: train: epoch 112 | batch 2030/2304 [I 2022-10-28 07:15:05] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:32 [I 2022-10-28 07:15:05] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00241 [I 2022-10-28 07:15:05] ...trap/engines/engine.py.230: loss: 0.11276 [I 2022-10-28 07:15:05] ...optimizers/trijoint.py.151: total_norm: 22.233648 [I 2022-10-28 07:15:11] ...trap/engines/engine.py.224: train: epoch 112 | batch 2040/2304 [I 2022-10-28 07:15:11] ...trap/engines/engine.py.228: elapsed: 0:19:19 | left: 0:02:27 [I 2022-10-28 07:15:11] ...trap/engines/engine.py.229: process: 0.56365 | load: 0.00243 [I 2022-10-28 07:15:11] ...trap/engines/engine.py.230: loss: 0.15148 [I 2022-10-28 07:15:11] ...optimizers/trijoint.py.151: total_norm: 34.074921 [I 2022-10-28 07:15:17] ...trap/engines/engine.py.224: train: epoch 112 | batch 2050/2304 [I 2022-10-28 07:15:17] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:21 [I 2022-10-28 07:15:17] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00250 [I 2022-10-28 07:15:17] ...trap/engines/engine.py.230: loss: 0.08700 [I 2022-10-28 07:15:17] ...optimizers/trijoint.py.151: total_norm: 9.695504 [I 2022-10-28 07:15:22] ...trap/engines/engine.py.224: train: epoch 112 | batch 2060/2304 [I 2022-10-28 07:15:22] ...trap/engines/engine.py.228: elapsed: 0:19:30 | left: 0:02:15 [I 2022-10-28 07:15:22] ...trap/engines/engine.py.229: process: 0.55784 | load: 0.00241 [I 2022-10-28 07:15:22] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-28 07:15:22] ...optimizers/trijoint.py.151: total_norm: 9.327016 [I 2022-10-28 07:15:28] ...trap/engines/engine.py.224: train: epoch 112 | batch 2070/2304 [I 2022-10-28 07:15:28] ...trap/engines/engine.py.228: elapsed: 0:19:36 | left: 0:02:10 [I 2022-10-28 07:15:28] ...trap/engines/engine.py.229: process: 0.54916 | load: 0.00239 [I 2022-10-28 07:15:28] ...trap/engines/engine.py.230: loss: 0.10902 [I 2022-10-28 07:15:28] ...optimizers/trijoint.py.151: total_norm: 14.226745 [I 2022-10-28 07:15:33] ...trap/engines/engine.py.224: train: epoch 112 | batch 2080/2304 [I 2022-10-28 07:15:33] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:05 [I 2022-10-28 07:15:33] ...trap/engines/engine.py.229: process: 0.56230 | load: 0.00239 [I 2022-10-28 07:15:33] ...trap/engines/engine.py.230: loss: 0.10672 [I 2022-10-28 07:15:33] ...optimizers/trijoint.py.151: total_norm: 26.506298 [I 2022-10-28 07:15:39] ...trap/engines/engine.py.224: train: epoch 112 | batch 2090/2304 [I 2022-10-28 07:15:39] ...trap/engines/engine.py.228: elapsed: 0:19:47 | left: 0:01:59 [I 2022-10-28 07:15:39] ...trap/engines/engine.py.229: process: 0.55536 | load: 0.00239 [I 2022-10-28 07:15:39] ...trap/engines/engine.py.230: loss: 0.10347 [I 2022-10-28 07:15:39] ...optimizers/trijoint.py.151: total_norm: 14.376901 [I 2022-10-28 07:15:44] ...trap/engines/engine.py.224: train: epoch 112 | batch 2100/2304 [I 2022-10-28 07:15:44] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:01:54 [I 2022-10-28 07:15:44] ...trap/engines/engine.py.229: process: 0.56757 | load: 0.00256 [I 2022-10-28 07:15:44] ...trap/engines/engine.py.230: loss: 0.15779 [I 2022-10-28 07:15:44] ...optimizers/trijoint.py.151: total_norm: 18.432358 [I 2022-10-28 07:15:50] ...trap/engines/engine.py.224: train: epoch 112 | batch 2110/2304 [I 2022-10-28 07:15:50] ...trap/engines/engine.py.228: elapsed: 0:19:58 | left: 0:01:48 [I 2022-10-28 07:15:50] ...trap/engines/engine.py.229: process: 0.56394 | load: 0.00279 [I 2022-10-28 07:15:50] ...trap/engines/engine.py.230: loss: 0.09025 [I 2022-10-28 07:15:50] ...optimizers/trijoint.py.151: total_norm: 12.772323 [I 2022-10-28 07:15:56] ...trap/engines/engine.py.224: train: epoch 112 | batch 2120/2304 [I 2022-10-28 07:15:56] ...trap/engines/engine.py.228: elapsed: 0:20:04 | left: 0:01:43 [I 2022-10-28 07:15:56] ...trap/engines/engine.py.229: process: 0.57021 | load: 0.00256 [I 2022-10-28 07:15:56] ...trap/engines/engine.py.230: loss: 0.09082 [I 2022-10-28 07:15:56] ...optimizers/trijoint.py.151: total_norm: 19.959707 [I 2022-10-28 07:16:01] ...trap/engines/engine.py.224: train: epoch 112 | batch 2130/2304 [I 2022-10-28 07:16:01] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:37 [I 2022-10-28 07:16:01] ...trap/engines/engine.py.229: process: 0.55985 | load: 0.00245 [I 2022-10-28 07:16:01] ...trap/engines/engine.py.230: loss: 0.12677 [I 2022-10-28 07:16:01] ...optimizers/trijoint.py.151: total_norm: 20.458639 [I 2022-10-28 07:16:07] ...trap/engines/engine.py.224: train: epoch 112 | batch 2140/2304 [I 2022-10-28 07:16:07] ...trap/engines/engine.py.228: elapsed: 0:20:15 | left: 0:01:32 [I 2022-10-28 07:16:07] ...trap/engines/engine.py.229: process: 0.56379 | load: 0.00252 [I 2022-10-28 07:16:07] ...trap/engines/engine.py.230: loss: 0.08805 [I 2022-10-28 07:16:07] ...optimizers/trijoint.py.151: total_norm: 11.114431 [I 2022-10-28 07:16:13] ...trap/engines/engine.py.224: train: epoch 112 | batch 2150/2304 [I 2022-10-28 07:16:13] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:26 [I 2022-10-28 07:16:13] ...trap/engines/engine.py.229: process: 0.56867 | load: 0.00250 [I 2022-10-28 07:16:13] ...trap/engines/engine.py.230: loss: 0.10211 [I 2022-10-28 07:16:13] ...optimizers/trijoint.py.151: total_norm: 7.621114 [I 2022-10-28 07:16:18] ...trap/engines/engine.py.224: train: epoch 112 | batch 2160/2304 [I 2022-10-28 07:16:18] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:20 [I 2022-10-28 07:16:18] ...trap/engines/engine.py.229: process: 0.56024 | load: 0.00248 [I 2022-10-28 07:16:18] ...trap/engines/engine.py.230: loss: 0.17500 [I 2022-10-28 07:16:18] ...optimizers/trijoint.py.151: total_norm: 20.791037 [I 2022-10-28 07:16:24] ...trap/engines/engine.py.224: train: epoch 112 | batch 2170/2304 [I 2022-10-28 07:16:24] ...trap/engines/engine.py.228: elapsed: 0:20:32 | left: 0:01:15 [I 2022-10-28 07:16:24] ...trap/engines/engine.py.229: process: 0.56175 | load: 0.00249 [I 2022-10-28 07:16:24] ...trap/engines/engine.py.230: loss: 0.09837 [I 2022-10-28 07:16:24] ...optimizers/trijoint.py.151: total_norm: 10.391948 [I 2022-10-28 07:16:29] ...trap/engines/engine.py.224: train: epoch 112 | batch 2180/2304 [I 2022-10-28 07:16:29] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:09 [I 2022-10-28 07:16:29] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00241 [I 2022-10-28 07:16:29] ...trap/engines/engine.py.230: loss: 0.15097 [I 2022-10-28 07:16:29] ...optimizers/trijoint.py.151: total_norm: 19.692835 [I 2022-10-28 07:16:35] ...trap/engines/engine.py.224: train: epoch 112 | batch 2190/2304 [I 2022-10-28 07:16:35] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:01:03 [I 2022-10-28 07:16:35] ...trap/engines/engine.py.229: process: 0.55769 | load: 0.00254 [I 2022-10-28 07:16:35] ...trap/engines/engine.py.230: loss: 0.14006 [I 2022-10-28 07:16:35] ...optimizers/trijoint.py.151: total_norm: 12.579051 [I 2022-10-28 07:16:41] ...trap/engines/engine.py.224: train: epoch 112 | batch 2200/2304 [I 2022-10-28 07:16:41] ...trap/engines/engine.py.228: elapsed: 0:20:49 | left: 0:00:58 [I 2022-10-28 07:16:41] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00252 [I 2022-10-28 07:16:41] ...trap/engines/engine.py.230: loss: 0.08840 [I 2022-10-28 07:16:41] ...optimizers/trijoint.py.151: total_norm: 11.011028 [I 2022-10-28 07:16:46] ...trap/engines/engine.py.224: train: epoch 112 | batch 2210/2304 [I 2022-10-28 07:16:46] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:52 [I 2022-10-28 07:16:46] ...trap/engines/engine.py.229: process: 0.55149 | load: 0.00253 [I 2022-10-28 07:16:46] ...trap/engines/engine.py.230: loss: 0.17845 [I 2022-10-28 07:16:46] ...optimizers/trijoint.py.151: total_norm: 34.899109 [I 2022-10-28 07:16:52] ...trap/engines/engine.py.224: train: epoch 112 | batch 2220/2304 [I 2022-10-28 07:16:52] ...trap/engines/engine.py.228: elapsed: 0:21:00 | left: 0:00:47 [I 2022-10-28 07:16:52] ...trap/engines/engine.py.229: process: 0.56558 | load: 0.00253 [I 2022-10-28 07:16:52] ...trap/engines/engine.py.230: loss: 0.08406 [I 2022-10-28 07:16:52] ...optimizers/trijoint.py.151: total_norm: 15.820785 [I 2022-10-28 07:16:57] ...trap/engines/engine.py.224: train: epoch 112 | batch 2230/2304 [I 2022-10-28 07:16:57] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:41 [I 2022-10-28 07:16:57] ...trap/engines/engine.py.229: process: 0.54991 | load: 0.00248 [I 2022-10-28 07:16:57] ...trap/engines/engine.py.230: loss: 0.16525 [I 2022-10-28 07:16:57] ...optimizers/trijoint.py.151: total_norm: 19.956911 [I 2022-10-28 07:17:03] ...trap/engines/engine.py.224: train: epoch 112 | batch 2240/2304 [I 2022-10-28 07:17:03] ...trap/engines/engine.py.228: elapsed: 0:21:11 | left: 0:00:37 [I 2022-10-28 07:17:03] ...trap/engines/engine.py.229: process: 0.70222 | load: 0.00254 [I 2022-10-28 07:17:03] ...trap/engines/engine.py.230: loss: 0.14301 [I 2022-10-28 07:17:03] ...optimizers/trijoint.py.151: total_norm: 23.310104 [I 2022-10-28 07:17:09] ...trap/engines/engine.py.224: train: epoch 112 | batch 2250/2304 [I 2022-10-28 07:17:09] ...trap/engines/engine.py.228: elapsed: 0:21:17 | left: 0:00:30 [I 2022-10-28 07:17:09] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00257 [I 2022-10-28 07:17:09] ...trap/engines/engine.py.230: loss: 0.11115 [I 2022-10-28 07:17:09] ...optimizers/trijoint.py.151: total_norm: 20.861012 [I 2022-10-28 07:17:14] ...trap/engines/engine.py.224: train: epoch 112 | batch 2260/2304 [I 2022-10-28 07:17:14] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:24 [I 2022-10-28 07:17:14] ...trap/engines/engine.py.229: process: 0.56291 | load: 0.00253 [I 2022-10-28 07:17:14] ...trap/engines/engine.py.230: loss: 0.14317 [I 2022-10-28 07:17:14] ...optimizers/trijoint.py.151: total_norm: 18.667503 [I 2022-10-28 07:17:20] ...trap/engines/engine.py.224: train: epoch 112 | batch 2270/2304 [I 2022-10-28 07:17:20] ...trap/engines/engine.py.228: elapsed: 0:21:28 | left: 0:00:18 [I 2022-10-28 07:17:20] ...trap/engines/engine.py.229: process: 0.55330 | load: 0.00249 [I 2022-10-28 07:17:20] ...trap/engines/engine.py.230: loss: 0.09387 [I 2022-10-28 07:17:20] ...optimizers/trijoint.py.151: total_norm: 11.116204 [I 2022-10-28 07:17:25] ...trap/engines/engine.py.224: train: epoch 112 | batch 2280/2304 [I 2022-10-28 07:17:25] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:13 [I 2022-10-28 07:17:25] ...trap/engines/engine.py.229: process: 0.55544 | load: 0.00255 [I 2022-10-28 07:17:25] ...trap/engines/engine.py.230: loss: 0.10085 [I 2022-10-28 07:17:25] ...optimizers/trijoint.py.151: total_norm: 20.053368 [I 2022-10-28 07:17:31] ...trap/engines/engine.py.224: train: epoch 112 | batch 2290/2304 [I 2022-10-28 07:17:31] ...trap/engines/engine.py.228: elapsed: 0:21:39 | left: 0:00:07 [I 2022-10-28 07:17:31] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00252 [I 2022-10-28 07:17:31] ...trap/engines/engine.py.230: loss: 0.10750 [I 2022-10-28 07:17:31] ...optimizers/trijoint.py.151: total_norm: 13.855163 [I 2022-10-28 07:17:37] ...trap/engines/engine.py.224: train: epoch 112 | batch 2300/2304 [I 2022-10-28 07:17:37] ...trap/engines/engine.py.228: elapsed: 0:21:45 | left: 0:00:02 [I 2022-10-28 07:17:37] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00015 [I 2022-10-28 07:17:37] ...trap/engines/engine.py.230: loss: 0.12853 [I 2022-10-28 07:17:37] ...optimizers/trijoint.py.151: total_norm: 22.104980 [I 2022-10-28 07:17:39] ...trap/engines/engine.py.224: train: epoch 112 | batch 2304/2304 [I 2022-10-28 07:17:39] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:00 [I 2022-10-28 07:17:39] ...trap/engines/engine.py.229: process: 0.56351 | load: 0.00019 [I 2022-10-28 07:17:39] ...trap/engines/engine.py.230: loss: 0.09200 [I 2022-10-28 07:17:39] ...optimizers/trijoint.py.151: total_norm: 10.163596 [S 2022-10-28 07:17:40] ...trap/engines/engine.py.129: train_epoch.epoch: 112 [S 2022-10-28 07:17:40] ...trap/engines/engine.py.129: train_epoch.loss: 0.12072558557683372 [I 2022-10-28 07:17:40] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 07:17:42] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 112 [I 2022-10-28 07:17:42] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 07:17:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 07:17:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 07:17:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 07:17:48] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 07:17:48] ...trap/engines/engine.py.321: eval: epoch 112 | batch 0/510 [I 2022-10-28 07:17:48] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:52:41 [I 2022-10-28 07:17:48] ...trap/engines/engine.py.326: process: 6.19807 | load: 3.89814 [I 2022-10-28 07:17:51] ...trap/engines/engine.py.321: eval: epoch 112 | batch 10/510 [I 2022-10-28 07:17:51] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:59 [I 2022-10-28 07:17:51] ...trap/engines/engine.py.326: process: 0.32676 | load: 0.00019 [I 2022-10-28 07:17:55] ...trap/engines/engine.py.321: eval: epoch 112 | batch 20/510 [I 2022-10-28 07:17:55] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:15 [I 2022-10-28 07:17:55] ...trap/engines/engine.py.326: process: 0.33201 | load: 0.00022 [I 2022-10-28 07:17:58] ...trap/engines/engine.py.321: eval: epoch 112 | batch 30/510 [I 2022-10-28 07:17:58] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:41 [I 2022-10-28 07:17:58] ...trap/engines/engine.py.326: process: 0.32361 | load: 0.00035 [I 2022-10-28 07:18:01] ...trap/engines/engine.py.321: eval: epoch 112 | batch 40/510 [I 2022-10-28 07:18:01] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:38 [I 2022-10-28 07:18:01] ...trap/engines/engine.py.326: process: 0.34544 | load: 0.00023 [I 2022-10-28 07:18:05] ...trap/engines/engine.py.321: eval: epoch 112 | batch 50/510 [I 2022-10-28 07:18:05] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:34 [I 2022-10-28 07:18:05] ...trap/engines/engine.py.326: process: 0.34358 | load: 0.00020 [I 2022-10-28 07:18:08] ...trap/engines/engine.py.321: eval: epoch 112 | batch 60/510 [I 2022-10-28 07:18:08] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:30 [I 2022-10-28 07:18:08] ...trap/engines/engine.py.326: process: 0.32234 | load: 0.00051 [I 2022-10-28 07:18:11] ...trap/engines/engine.py.321: eval: epoch 112 | batch 70/510 [I 2022-10-28 07:18:11] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:28 [I 2022-10-28 07:18:11] ...trap/engines/engine.py.326: process: 0.32748 | load: 0.00016 [I 2022-10-28 07:18:15] ...trap/engines/engine.py.321: eval: epoch 112 | batch 80/510 [I 2022-10-28 07:18:15] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:23 [I 2022-10-28 07:18:15] ...trap/engines/engine.py.326: process: 0.32963 | load: 0.00025 [I 2022-10-28 07:18:18] ...trap/engines/engine.py.321: eval: epoch 112 | batch 90/510 [I 2022-10-28 07:18:18] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:22 [I 2022-10-28 07:18:18] ...trap/engines/engine.py.326: process: 0.33119 | load: 0.00021 [I 2022-10-28 07:18:22] ...trap/engines/engine.py.321: eval: epoch 112 | batch 100/510 [I 2022-10-28 07:18:22] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:16 [I 2022-10-28 07:18:22] ...trap/engines/engine.py.326: process: 0.30049 | load: 0.00017 [I 2022-10-28 07:18:25] ...trap/engines/engine.py.321: eval: epoch 112 | batch 110/510 [I 2022-10-28 07:18:25] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:57 [I 2022-10-28 07:18:25] ...trap/engines/engine.py.326: process: 0.27625 | load: 0.00020 [I 2022-10-28 07:18:27] ...trap/engines/engine.py.321: eval: epoch 112 | batch 120/510 [I 2022-10-28 07:18:27] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:52 [I 2022-10-28 07:18:27] ...trap/engines/engine.py.326: process: 0.28729 | load: 0.00014 [I 2022-10-28 07:18:30] ...trap/engines/engine.py.321: eval: epoch 112 | batch 130/510 [I 2022-10-28 07:18:30] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:48 [I 2022-10-28 07:18:30] ...trap/engines/engine.py.326: process: 0.28265 | load: 0.00014 [I 2022-10-28 07:18:33] ...trap/engines/engine.py.321: eval: epoch 112 | batch 140/510 [I 2022-10-28 07:18:33] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:45 [I 2022-10-28 07:18:33] ...trap/engines/engine.py.326: process: 0.28328 | load: 0.00014 [I 2022-10-28 07:18:36] ...trap/engines/engine.py.321: eval: epoch 112 | batch 150/510 [I 2022-10-28 07:18:36] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:43 [I 2022-10-28 07:18:36] ...trap/engines/engine.py.326: process: 0.28266 | load: 0.00019 [I 2022-10-28 07:18:39] ...trap/engines/engine.py.321: eval: epoch 112 | batch 160/510 [I 2022-10-28 07:18:39] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:40 [I 2022-10-28 07:18:39] ...trap/engines/engine.py.326: process: 0.28899 | load: 0.00020 [I 2022-10-28 07:18:42] ...trap/engines/engine.py.321: eval: epoch 112 | batch 170/510 [I 2022-10-28 07:18:42] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:37 [I 2022-10-28 07:18:42] ...trap/engines/engine.py.326: process: 0.28134 | load: 0.00014 [I 2022-10-28 07:18:45] ...trap/engines/engine.py.321: eval: epoch 112 | batch 180/510 [I 2022-10-28 07:18:45] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-28 07:18:45] ...trap/engines/engine.py.326: process: 0.28866 | load: 0.00014 [I 2022-10-28 07:18:48] ...trap/engines/engine.py.321: eval: epoch 112 | batch 190/510 [I 2022-10-28 07:18:48] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:32 [I 2022-10-28 07:18:48] ...trap/engines/engine.py.326: process: 0.28991 | load: 0.00016 [I 2022-10-28 07:18:50] ...trap/engines/engine.py.321: eval: epoch 112 | batch 200/510 [I 2022-10-28 07:18:50] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:30 [I 2022-10-28 07:18:50] ...trap/engines/engine.py.326: process: 0.30861 | load: 0.00014 [I 2022-10-28 07:18:53] ...trap/engines/engine.py.321: eval: epoch 112 | batch 210/510 [I 2022-10-28 07:18:53] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:28 [I 2022-10-28 07:18:53] ...trap/engines/engine.py.326: process: 0.28521 | load: 0.00015 [I 2022-10-28 07:18:56] ...trap/engines/engine.py.321: eval: epoch 112 | batch 220/510 [I 2022-10-28 07:18:56] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:23 [I 2022-10-28 07:18:56] ...trap/engines/engine.py.326: process: 0.28867 | load: 0.00015 [I 2022-10-28 07:18:59] ...trap/engines/engine.py.321: eval: epoch 112 | batch 230/510 [I 2022-10-28 07:18:59] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:20 [I 2022-10-28 07:18:59] ...trap/engines/engine.py.326: process: 0.28839 | load: 0.00015 [I 2022-10-28 07:19:02] ...trap/engines/engine.py.321: eval: epoch 112 | batch 240/510 [I 2022-10-28 07:19:02] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:17 [I 2022-10-28 07:19:02] ...trap/engines/engine.py.326: process: 0.28996 | load: 0.00016 [I 2022-10-28 07:19:05] ...trap/engines/engine.py.321: eval: epoch 112 | batch 250/510 [I 2022-10-28 07:19:05] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:14 [I 2022-10-28 07:19:05] ...trap/engines/engine.py.326: process: 0.27869 | load: 0.00016 [I 2022-10-28 07:19:08] ...trap/engines/engine.py.321: eval: epoch 112 | batch 260/510 [I 2022-10-28 07:19:08] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 07:19:08] ...trap/engines/engine.py.326: process: 0.28917 | load: 0.00016 [I 2022-10-28 07:19:11] ...trap/engines/engine.py.321: eval: epoch 112 | batch 270/510 [I 2022-10-28 07:19:11] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-28 07:19:11] ...trap/engines/engine.py.326: process: 0.28631 | load: 0.00016 [I 2022-10-28 07:19:14] ...trap/engines/engine.py.321: eval: epoch 112 | batch 280/510 [I 2022-10-28 07:19:14] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 07:19:14] ...trap/engines/engine.py.326: process: 0.28442 | load: 0.00014 [I 2022-10-28 07:19:17] ...trap/engines/engine.py.321: eval: epoch 112 | batch 290/510 [I 2022-10-28 07:19:17] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 07:19:17] ...trap/engines/engine.py.326: process: 0.28888 | load: 0.00016 [I 2022-10-28 07:19:19] ...trap/engines/engine.py.321: eval: epoch 112 | batch 300/510 [I 2022-10-28 07:19:19] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-28 07:19:19] ...trap/engines/engine.py.326: process: 0.28822 | load: 0.00016 [I 2022-10-28 07:19:22] ...trap/engines/engine.py.321: eval: epoch 112 | batch 310/510 [I 2022-10-28 07:19:22] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:57 [I 2022-10-28 07:19:22] ...trap/engines/engine.py.326: process: 0.28753 | load: 0.00016 [I 2022-10-28 07:19:25] ...trap/engines/engine.py.321: eval: epoch 112 | batch 320/510 [I 2022-10-28 07:19:25] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:54 [I 2022-10-28 07:19:25] ...trap/engines/engine.py.326: process: 0.29189 | load: 0.00026 [I 2022-10-28 07:19:28] ...trap/engines/engine.py.321: eval: epoch 112 | batch 330/510 [I 2022-10-28 07:19:28] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:51 [I 2022-10-28 07:19:28] ...trap/engines/engine.py.326: process: 0.28075 | load: 0.00022 [I 2022-10-28 07:19:31] ...trap/engines/engine.py.321: eval: epoch 112 | batch 340/510 [I 2022-10-28 07:19:31] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:49 [I 2022-10-28 07:19:31] ...trap/engines/engine.py.326: process: 0.28870 | load: 0.00015 [I 2022-10-28 07:19:34] ...trap/engines/engine.py.321: eval: epoch 112 | batch 350/510 [I 2022-10-28 07:19:34] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:45 [I 2022-10-28 07:19:34] ...trap/engines/engine.py.326: process: 0.28628 | load: 0.00016 [I 2022-10-28 07:19:37] ...trap/engines/engine.py.321: eval: epoch 112 | batch 360/510 [I 2022-10-28 07:19:37] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 07:19:37] ...trap/engines/engine.py.326: process: 0.29175 | load: 0.00015 [I 2022-10-28 07:19:40] ...trap/engines/engine.py.321: eval: epoch 112 | batch 370/510 [I 2022-10-28 07:19:40] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 07:19:40] ...trap/engines/engine.py.326: process: 0.28303 | load: 0.00014 [I 2022-10-28 07:19:42] ...trap/engines/engine.py.321: eval: epoch 112 | batch 380/510 [I 2022-10-28 07:19:42] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 07:19:42] ...trap/engines/engine.py.326: process: 0.28622 | load: 0.00015 [I 2022-10-28 07:19:45] ...trap/engines/engine.py.321: eval: epoch 112 | batch 390/510 [I 2022-10-28 07:19:45] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 07:19:45] ...trap/engines/engine.py.326: process: 0.28764 | load: 0.00016 [I 2022-10-28 07:19:48] ...trap/engines/engine.py.321: eval: epoch 112 | batch 400/510 [I 2022-10-28 07:19:48] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:31 [I 2022-10-28 07:19:48] ...trap/engines/engine.py.326: process: 0.28552 | load: 0.00014 [I 2022-10-28 07:19:51] ...trap/engines/engine.py.321: eval: epoch 112 | batch 410/510 [I 2022-10-28 07:19:51] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:28 [I 2022-10-28 07:19:51] ...trap/engines/engine.py.326: process: 0.28913 | load: 0.00015 [I 2022-10-28 07:19:54] ...trap/engines/engine.py.321: eval: epoch 112 | batch 420/510 [I 2022-10-28 07:19:54] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:25 [I 2022-10-28 07:19:54] ...trap/engines/engine.py.326: process: 0.28267 | load: 0.00015 [I 2022-10-28 07:19:57] ...trap/engines/engine.py.321: eval: epoch 112 | batch 430/510 [I 2022-10-28 07:19:57] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:23 [I 2022-10-28 07:19:57] ...trap/engines/engine.py.326: process: 0.29016 | load: 0.00020 [I 2022-10-28 07:20:00] ...trap/engines/engine.py.321: eval: epoch 112 | batch 440/510 [I 2022-10-28 07:20:00] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 07:20:00] ...trap/engines/engine.py.326: process: 0.28776 | load: 0.00016 [I 2022-10-28 07:20:03] ...trap/engines/engine.py.321: eval: epoch 112 | batch 450/510 [I 2022-10-28 07:20:03] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 07:20:03] ...trap/engines/engine.py.326: process: 0.28704 | load: 0.00016 [I 2022-10-28 07:20:06] ...trap/engines/engine.py.321: eval: epoch 112 | batch 460/510 [I 2022-10-28 07:20:06] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 07:20:06] ...trap/engines/engine.py.326: process: 0.28695 | load: 0.00016 [I 2022-10-28 07:20:08] ...trap/engines/engine.py.321: eval: epoch 112 | batch 470/510 [I 2022-10-28 07:20:08] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 07:20:08] ...trap/engines/engine.py.326: process: 0.28387 | load: 0.00020 [I 2022-10-28 07:20:11] ...trap/engines/engine.py.321: eval: epoch 112 | batch 480/510 [I 2022-10-28 07:20:11] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 07:20:11] ...trap/engines/engine.py.326: process: 0.28935 | load: 0.00017 [I 2022-10-28 07:20:14] ...trap/engines/engine.py.321: eval: epoch 112 | batch 490/510 [I 2022-10-28 07:20:14] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 07:20:14] ...trap/engines/engine.py.326: process: 0.28480 | load: 0.00015 [I 2022-10-28 07:20:17] ...trap/engines/engine.py.321: eval: epoch 112 | batch 500/510 [I 2022-10-28 07:20:17] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:02 [I 2022-10-28 07:20:17] ...trap/engines/engine.py.326: process: 0.28146 | load: 0.00015 [I 2022-10-28 07:20:20] ...trap/engines/engine.py.321: eval: epoch 112 | batch 510/510 [I 2022-10-28 07:20:20] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 07:20:20] ...trap/engines/engine.py.326: process: 0.28056 | load: 0.00014 [S 2022-10-28 07:20:21] ...trap/engines/engine.py.132: eval_epoch.epoch: 112 [I 2022-10-28 07:20:21] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.738200 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905200 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933000 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.728900 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904000 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933200 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010098 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010137 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009088 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011536 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010835 [S 2022-10-28 07:20:25] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.010962 [I 2022-10-28 07:20:27] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 07:20:27] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 07:20:27] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 07:20:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 07:20:27] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 07:20:32] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 07:20:33] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 07:20:39] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 07:20:39] ...trap/engines/engine.py.163: Training model on trainset for epoch 113 [I 2022-10-28 07:20:39] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 07:20:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 0/2304 [I 2022-10-28 07:20:42] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:51:04 [I 2022-10-28 07:20:42] ...trap/engines/engine.py.229: process: 2.89273 | load: 2.26123 [I 2022-10-28 07:20:42] ...trap/engines/engine.py.230: loss: 0.10993 [I 2022-10-28 07:20:42] ...optimizers/trijoint.py.151: total_norm: 10.635939 [I 2022-10-28 07:20:48] ...trap/engines/engine.py.224: train: epoch 113 | batch 10/2304 [I 2022-10-28 07:20:48] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:58 [I 2022-10-28 07:20:48] ...trap/engines/engine.py.229: process: 0.56480 | load: 0.00504 [I 2022-10-28 07:20:48] ...trap/engines/engine.py.230: loss: 0.09670 [I 2022-10-28 07:20:48] ...optimizers/trijoint.py.151: total_norm: 20.013016 [I 2022-10-28 07:20:53] ...trap/engines/engine.py.224: train: epoch 113 | batch 20/2304 [I 2022-10-28 07:20:53] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:17 [I 2022-10-28 07:20:53] ...trap/engines/engine.py.229: process: 0.55712 | load: 0.00301 [I 2022-10-28 07:20:53] ...trap/engines/engine.py.230: loss: 0.17238 [I 2022-10-28 07:20:53] ...optimizers/trijoint.py.151: total_norm: 34.519321 [I 2022-10-28 07:20:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 30/2304 [I 2022-10-28 07:20:59] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:12 [I 2022-10-28 07:20:59] ...trap/engines/engine.py.229: process: 0.55340 | load: 0.00347 [I 2022-10-28 07:20:59] ...trap/engines/engine.py.230: loss: 0.12680 [I 2022-10-28 07:20:59] ...optimizers/trijoint.py.151: total_norm: 14.225297 [I 2022-10-28 07:21:04] ...trap/engines/engine.py.224: train: epoch 113 | batch 40/2304 [I 2022-10-28 07:21:04] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:55 [I 2022-10-28 07:21:04] ...trap/engines/engine.py.229: process: 0.53368 | load: 0.00299 [I 2022-10-28 07:21:04] ...trap/engines/engine.py.230: loss: 0.07820 [I 2022-10-28 07:21:04] ...optimizers/trijoint.py.151: total_norm: 21.691458 [I 2022-10-28 07:21:10] ...trap/engines/engine.py.224: train: epoch 113 | batch 50/2304 [I 2022-10-28 07:21:10] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:08 [I 2022-10-28 07:21:10] ...trap/engines/engine.py.229: process: 0.57181 | load: 0.00312 [I 2022-10-28 07:21:10] ...trap/engines/engine.py.230: loss: 0.09413 [I 2022-10-28 07:21:10] ...optimizers/trijoint.py.151: total_norm: 9.341117 [I 2022-10-28 07:21:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 60/2304 [I 2022-10-28 07:21:16] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:57 [I 2022-10-28 07:21:16] ...trap/engines/engine.py.229: process: 0.55310 | load: 0.00286 [I 2022-10-28 07:21:16] ...trap/engines/engine.py.230: loss: 0.11821 [I 2022-10-28 07:21:16] ...optimizers/trijoint.py.151: total_norm: 32.723179 [I 2022-10-28 07:21:21] ...trap/engines/engine.py.224: train: epoch 113 | batch 70/2304 [I 2022-10-28 07:21:21] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:51 [I 2022-10-28 07:21:21] ...trap/engines/engine.py.229: process: 0.56246 | load: 0.00313 [I 2022-10-28 07:21:21] ...trap/engines/engine.py.230: loss: 0.12252 [I 2022-10-28 07:21:21] ...optimizers/trijoint.py.151: total_norm: 19.329535 [I 2022-10-28 07:21:27] ...trap/engines/engine.py.224: train: epoch 113 | batch 80/2304 [I 2022-10-28 07:21:27] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:32 [I 2022-10-28 07:21:27] ...trap/engines/engine.py.229: process: 0.55296 | load: 0.00278 [I 2022-10-28 07:21:27] ...trap/engines/engine.py.230: loss: 0.14815 [I 2022-10-28 07:21:27] ...optimizers/trijoint.py.151: total_norm: 43.409523 [I 2022-10-28 07:21:32] ...trap/engines/engine.py.224: train: epoch 113 | batch 90/2304 [I 2022-10-28 07:21:32] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:29 [I 2022-10-28 07:21:32] ...trap/engines/engine.py.229: process: 0.54961 | load: 0.00256 [I 2022-10-28 07:21:32] ...trap/engines/engine.py.230: loss: 0.15881 [I 2022-10-28 07:21:32] ...optimizers/trijoint.py.151: total_norm: 25.664755 [I 2022-10-28 07:21:38] ...trap/engines/engine.py.224: train: epoch 113 | batch 100/2304 [I 2022-10-28 07:21:38] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:24 [I 2022-10-28 07:21:38] ...trap/engines/engine.py.229: process: 0.54440 | load: 0.00257 [I 2022-10-28 07:21:38] ...trap/engines/engine.py.230: loss: 0.13947 [I 2022-10-28 07:21:38] ...optimizers/trijoint.py.151: total_norm: 27.492584 [I 2022-10-28 07:21:43] ...trap/engines/engine.py.224: train: epoch 113 | batch 110/2304 [I 2022-10-28 07:21:43] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:18 [I 2022-10-28 07:21:43] ...trap/engines/engine.py.229: process: 0.55554 | load: 0.00262 [I 2022-10-28 07:21:43] ...trap/engines/engine.py.230: loss: 0.11386 [I 2022-10-28 07:21:43] ...optimizers/trijoint.py.151: total_norm: 9.976374 [I 2022-10-28 07:21:49] ...trap/engines/engine.py.224: train: epoch 113 | batch 120/2304 [I 2022-10-28 07:21:49] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:23 [I 2022-10-28 07:21:49] ...trap/engines/engine.py.229: process: 0.55312 | load: 0.00270 [I 2022-10-28 07:21:49] ...trap/engines/engine.py.230: loss: 0.12978 [I 2022-10-28 07:21:49] ...optimizers/trijoint.py.151: total_norm: 9.786662 [I 2022-10-28 07:21:55] ...trap/engines/engine.py.224: train: epoch 113 | batch 130/2304 [I 2022-10-28 07:21:55] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:37 [I 2022-10-28 07:21:55] ...trap/engines/engine.py.229: process: 0.55641 | load: 0.00263 [I 2022-10-28 07:21:55] ...trap/engines/engine.py.230: loss: 0.17712 [I 2022-10-28 07:21:55] ...optimizers/trijoint.py.151: total_norm: 8.974092 [I 2022-10-28 07:22:00] ...trap/engines/engine.py.224: train: epoch 113 | batch 140/2304 [I 2022-10-28 07:22:00] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:11 [I 2022-10-28 07:22:00] ...trap/engines/engine.py.229: process: 0.55250 | load: 0.00263 [I 2022-10-28 07:22:00] ...trap/engines/engine.py.230: loss: 0.10998 [I 2022-10-28 07:22:00] ...optimizers/trijoint.py.151: total_norm: 9.149938 [I 2022-10-28 07:22:06] ...trap/engines/engine.py.224: train: epoch 113 | batch 150/2304 [I 2022-10-28 07:22:06] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:50 [I 2022-10-28 07:22:06] ...trap/engines/engine.py.229: process: 0.55186 | load: 0.00267 [I 2022-10-28 07:22:06] ...trap/engines/engine.py.230: loss: 0.10363 [I 2022-10-28 07:22:06] ...optimizers/trijoint.py.151: total_norm: 9.744910 [I 2022-10-28 07:22:11] ...trap/engines/engine.py.224: train: epoch 113 | batch 160/2304 [I 2022-10-28 07:22:11] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:52 [I 2022-10-28 07:22:11] ...trap/engines/engine.py.229: process: 0.55228 | load: 0.00249 [I 2022-10-28 07:22:11] ...trap/engines/engine.py.230: loss: 0.10244 [I 2022-10-28 07:22:11] ...optimizers/trijoint.py.151: total_norm: 12.225805 [I 2022-10-28 07:22:17] ...trap/engines/engine.py.224: train: epoch 113 | batch 170/2304 [I 2022-10-28 07:22:17] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:57 [I 2022-10-28 07:22:17] ...trap/engines/engine.py.229: process: 0.56161 | load: 0.00260 [I 2022-10-28 07:22:17] ...trap/engines/engine.py.230: loss: 0.13339 [I 2022-10-28 07:22:17] ...optimizers/trijoint.py.151: total_norm: 45.307362 [I 2022-10-28 07:22:23] ...trap/engines/engine.py.224: train: epoch 113 | batch 180/2304 [I 2022-10-28 07:22:23] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:45 [I 2022-10-28 07:22:23] ...trap/engines/engine.py.229: process: 0.55364 | load: 0.00293 [I 2022-10-28 07:22:23] ...trap/engines/engine.py.230: loss: 0.08111 [I 2022-10-28 07:22:23] ...optimizers/trijoint.py.151: total_norm: 34.270794 [I 2022-10-28 07:22:28] ...trap/engines/engine.py.224: train: epoch 113 | batch 190/2304 [I 2022-10-28 07:22:28] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:37 [I 2022-10-28 07:22:28] ...trap/engines/engine.py.229: process: 0.55468 | load: 0.00277 [I 2022-10-28 07:22:28] ...trap/engines/engine.py.230: loss: 0.17760 [I 2022-10-28 07:22:28] ...optimizers/trijoint.py.151: total_norm: 26.677961 [I 2022-10-28 07:22:34] ...trap/engines/engine.py.224: train: epoch 113 | batch 200/2304 [I 2022-10-28 07:22:34] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:19 [I 2022-10-28 07:22:34] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00263 [I 2022-10-28 07:22:34] ...trap/engines/engine.py.230: loss: 0.08318 [I 2022-10-28 07:22:34] ...optimizers/trijoint.py.151: total_norm: 13.476648 [I 2022-10-28 07:22:39] ...trap/engines/engine.py.224: train: epoch 113 | batch 210/2304 [I 2022-10-28 07:22:39] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:32 [I 2022-10-28 07:22:39] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00299 [I 2022-10-28 07:22:39] ...trap/engines/engine.py.230: loss: 0.13519 [I 2022-10-28 07:22:39] ...optimizers/trijoint.py.151: total_norm: 17.914497 [I 2022-10-28 07:22:45] ...trap/engines/engine.py.224: train: epoch 113 | batch 220/2304 [I 2022-10-28 07:22:45] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:21 [I 2022-10-28 07:22:45] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00254 [I 2022-10-28 07:22:45] ...trap/engines/engine.py.230: loss: 0.10414 [I 2022-10-28 07:22:45] ...optimizers/trijoint.py.151: total_norm: 18.840628 [I 2022-10-28 07:22:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 230/2304 [I 2022-10-28 07:22:51] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:11 [I 2022-10-28 07:22:51] ...trap/engines/engine.py.229: process: 0.55629 | load: 0.00254 [I 2022-10-28 07:22:51] ...trap/engines/engine.py.230: loss: 0.14351 [I 2022-10-28 07:22:51] ...optimizers/trijoint.py.151: total_norm: 63.305180 [I 2022-10-28 07:22:56] ...trap/engines/engine.py.224: train: epoch 113 | batch 240/2304 [I 2022-10-28 07:22:56] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:08 [I 2022-10-28 07:22:56] ...trap/engines/engine.py.229: process: 0.54630 | load: 0.00259 [I 2022-10-28 07:22:56] ...trap/engines/engine.py.230: loss: 0.15331 [I 2022-10-28 07:22:56] ...optimizers/trijoint.py.151: total_norm: 33.429489 [I 2022-10-28 07:23:02] ...trap/engines/engine.py.224: train: epoch 113 | batch 250/2304 [I 2022-10-28 07:23:02] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:55 [I 2022-10-28 07:23:02] ...trap/engines/engine.py.229: process: 0.54560 | load: 0.00287 [I 2022-10-28 07:23:02] ...trap/engines/engine.py.230: loss: 0.11685 [I 2022-10-28 07:23:02] ...optimizers/trijoint.py.151: total_norm: 9.808329 [I 2022-10-28 07:23:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 260/2304 [I 2022-10-28 07:23:07] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:59 [I 2022-10-28 07:23:07] ...trap/engines/engine.py.229: process: 0.56549 | load: 0.00271 [I 2022-10-28 07:23:07] ...trap/engines/engine.py.230: loss: 0.10418 [I 2022-10-28 07:23:07] ...optimizers/trijoint.py.151: total_norm: 25.176672 [I 2022-10-28 07:23:13] ...trap/engines/engine.py.224: train: epoch 113 | batch 270/2304 [I 2022-10-28 07:23:13] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:56 [I 2022-10-28 07:23:13] ...trap/engines/engine.py.229: process: 0.56657 | load: 0.00272 [I 2022-10-28 07:23:13] ...trap/engines/engine.py.230: loss: 0.15920 [I 2022-10-28 07:23:13] ...optimizers/trijoint.py.151: total_norm: 10.260153 [I 2022-10-28 07:23:18] ...trap/engines/engine.py.224: train: epoch 113 | batch 280/2304 [I 2022-10-28 07:23:18] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:50 [I 2022-10-28 07:23:18] ...trap/engines/engine.py.229: process: 0.55346 | load: 0.00270 [I 2022-10-28 07:23:18] ...trap/engines/engine.py.230: loss: 0.21092 [I 2022-10-28 07:23:18] ...optimizers/trijoint.py.151: total_norm: 11.787501 [I 2022-10-28 07:23:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 290/2304 [I 2022-10-28 07:23:24] ...trap/engines/engine.py.228: elapsed: 0:02:44 | left: 0:18:39 [I 2022-10-28 07:23:24] ...trap/engines/engine.py.229: process: 0.55762 | load: 0.00253 [I 2022-10-28 07:23:24] ...trap/engines/engine.py.230: loss: 0.12346 [I 2022-10-28 07:23:24] ...optimizers/trijoint.py.151: total_norm: 21.860931 [I 2022-10-28 07:23:30] ...trap/engines/engine.py.224: train: epoch 113 | batch 300/2304 [I 2022-10-28 07:23:30] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:32 [I 2022-10-28 07:23:30] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00263 [I 2022-10-28 07:23:30] ...trap/engines/engine.py.230: loss: 0.17666 [I 2022-10-28 07:23:30] ...optimizers/trijoint.py.151: total_norm: 15.813112 [I 2022-10-28 07:23:35] ...trap/engines/engine.py.224: train: epoch 113 | batch 310/2304 [I 2022-10-28 07:23:35] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:40 [I 2022-10-28 07:23:35] ...trap/engines/engine.py.229: process: 0.55481 | load: 0.00273 [I 2022-10-28 07:23:35] ...trap/engines/engine.py.230: loss: 0.11028 [I 2022-10-28 07:23:35] ...optimizers/trijoint.py.151: total_norm: 19.810303 [I 2022-10-28 07:23:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 320/2304 [I 2022-10-28 07:23:41] ...trap/engines/engine.py.228: elapsed: 0:03:01 | left: 0:18:31 [I 2022-10-28 07:23:41] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00279 [I 2022-10-28 07:23:41] ...trap/engines/engine.py.230: loss: 0.10191 [I 2022-10-28 07:23:41] ...optimizers/trijoint.py.151: total_norm: 11.041313 [I 2022-10-28 07:23:46] ...trap/engines/engine.py.224: train: epoch 113 | batch 330/2304 [I 2022-10-28 07:23:46] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:25 [I 2022-10-28 07:23:46] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00271 [I 2022-10-28 07:23:46] ...trap/engines/engine.py.230: loss: 0.11402 [I 2022-10-28 07:23:46] ...optimizers/trijoint.py.151: total_norm: 11.954742 [I 2022-10-28 07:23:52] ...trap/engines/engine.py.224: train: epoch 113 | batch 340/2304 [I 2022-10-28 07:23:52] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:15 [I 2022-10-28 07:23:52] ...trap/engines/engine.py.229: process: 0.54710 | load: 0.00257 [I 2022-10-28 07:23:52] ...trap/engines/engine.py.230: loss: 0.16297 [I 2022-10-28 07:23:52] ...optimizers/trijoint.py.151: total_norm: 39.741631 [I 2022-10-28 07:23:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 350/2304 [I 2022-10-28 07:23:58] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:07 [I 2022-10-28 07:23:58] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00257 [I 2022-10-28 07:23:58] ...trap/engines/engine.py.230: loss: 0.10359 [I 2022-10-28 07:23:58] ...optimizers/trijoint.py.151: total_norm: 19.111673 [I 2022-10-28 07:24:03] ...trap/engines/engine.py.224: train: epoch 113 | batch 360/2304 [I 2022-10-28 07:24:03] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:56 [I 2022-10-28 07:24:03] ...trap/engines/engine.py.229: process: 0.54609 | load: 0.00289 [I 2022-10-28 07:24:03] ...trap/engines/engine.py.230: loss: 0.11390 [I 2022-10-28 07:24:03] ...optimizers/trijoint.py.151: total_norm: 13.268591 [I 2022-10-28 07:24:09] ...trap/engines/engine.py.224: train: epoch 113 | batch 370/2304 [I 2022-10-28 07:24:09] ...trap/engines/engine.py.228: elapsed: 0:03:29 | left: 0:17:43 [I 2022-10-28 07:24:09] ...trap/engines/engine.py.229: process: 0.53860 | load: 0.00259 [I 2022-10-28 07:24:09] ...trap/engines/engine.py.230: loss: 0.17160 [I 2022-10-28 07:24:09] ...optimizers/trijoint.py.151: total_norm: 17.502457 [I 2022-10-28 07:24:14] ...trap/engines/engine.py.224: train: epoch 113 | batch 380/2304 [I 2022-10-28 07:24:14] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:51 [I 2022-10-28 07:24:14] ...trap/engines/engine.py.229: process: 0.55719 | load: 0.00259 [I 2022-10-28 07:24:14] ...trap/engines/engine.py.230: loss: 0.15417 [I 2022-10-28 07:24:14] ...optimizers/trijoint.py.151: total_norm: 21.558832 [I 2022-10-28 07:24:20] ...trap/engines/engine.py.224: train: epoch 113 | batch 390/2304 [I 2022-10-28 07:24:20] ...trap/engines/engine.py.228: elapsed: 0:03:40 | left: 0:17:44 [I 2022-10-28 07:24:20] ...trap/engines/engine.py.229: process: 0.55322 | load: 0.00250 [I 2022-10-28 07:24:20] ...trap/engines/engine.py.230: loss: 0.09576 [I 2022-10-28 07:24:20] ...optimizers/trijoint.py.151: total_norm: 13.706659 [I 2022-10-28 07:24:26] ...trap/engines/engine.py.224: train: epoch 113 | batch 400/2304 [I 2022-10-28 07:24:26] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:44 [I 2022-10-28 07:24:26] ...trap/engines/engine.py.229: process: 0.55032 | load: 0.00255 [I 2022-10-28 07:24:26] ...trap/engines/engine.py.230: loss: 0.09178 [I 2022-10-28 07:24:26] ...optimizers/trijoint.py.151: total_norm: 24.137106 [I 2022-10-28 07:24:31] ...trap/engines/engine.py.224: train: epoch 113 | batch 410/2304 [I 2022-10-28 07:24:31] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:28 [I 2022-10-28 07:24:31] ...trap/engines/engine.py.229: process: 0.55547 | load: 0.00258 [I 2022-10-28 07:24:31] ...trap/engines/engine.py.230: loss: 0.08517 [I 2022-10-28 07:24:31] ...optimizers/trijoint.py.151: total_norm: 11.238167 [I 2022-10-28 07:24:37] ...trap/engines/engine.py.224: train: epoch 113 | batch 420/2304 [I 2022-10-28 07:24:37] ...trap/engines/engine.py.228: elapsed: 0:03:57 | left: 0:17:32 [I 2022-10-28 07:24:37] ...trap/engines/engine.py.229: process: 0.55873 | load: 0.00254 [I 2022-10-28 07:24:37] ...trap/engines/engine.py.230: loss: 0.16108 [I 2022-10-28 07:24:37] ...optimizers/trijoint.py.151: total_norm: 10.142649 [I 2022-10-28 07:24:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 430/2304 [I 2022-10-28 07:24:42] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:15 [I 2022-10-28 07:24:42] ...trap/engines/engine.py.229: process: 0.55188 | load: 0.00247 [I 2022-10-28 07:24:42] ...trap/engines/engine.py.230: loss: 0.09813 [I 2022-10-28 07:24:42] ...optimizers/trijoint.py.151: total_norm: 13.990220 [I 2022-10-28 07:24:48] ...trap/engines/engine.py.224: train: epoch 113 | batch 440/2304 [I 2022-10-28 07:24:48] ...trap/engines/engine.py.228: elapsed: 0:04:08 | left: 0:17:10 [I 2022-10-28 07:24:48] ...trap/engines/engine.py.229: process: 0.55272 | load: 0.00243 [I 2022-10-28 07:24:48] ...trap/engines/engine.py.230: loss: 0.13454 [I 2022-10-28 07:24:48] ...optimizers/trijoint.py.151: total_norm: 87.060417 [I 2022-10-28 07:24:53] ...trap/engines/engine.py.224: train: epoch 113 | batch 450/2304 [I 2022-10-28 07:24:53] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:13 [I 2022-10-28 07:24:53] ...trap/engines/engine.py.229: process: 0.55088 | load: 0.00247 [I 2022-10-28 07:24:53] ...trap/engines/engine.py.230: loss: 0.16047 [I 2022-10-28 07:24:53] ...optimizers/trijoint.py.151: total_norm: 16.735931 [I 2022-10-28 07:24:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 460/2304 [I 2022-10-28 07:24:59] ...trap/engines/engine.py.228: elapsed: 0:04:19 | left: 0:17:06 [I 2022-10-28 07:24:59] ...trap/engines/engine.py.229: process: 0.55835 | load: 0.00254 [I 2022-10-28 07:24:59] ...trap/engines/engine.py.230: loss: 0.14336 [I 2022-10-28 07:24:59] ...optimizers/trijoint.py.151: total_norm: 15.420159 [I 2022-10-28 07:25:05] ...trap/engines/engine.py.224: train: epoch 113 | batch 470/2304 [I 2022-10-28 07:25:05] ...trap/engines/engine.py.228: elapsed: 0:04:25 | left: 0:17:14 [I 2022-10-28 07:25:05] ...trap/engines/engine.py.229: process: 0.56433 | load: 0.00250 [I 2022-10-28 07:25:05] ...trap/engines/engine.py.230: loss: 0.10649 [I 2022-10-28 07:25:05] ...optimizers/trijoint.py.151: total_norm: 15.294438 [I 2022-10-28 07:25:10] ...trap/engines/engine.py.224: train: epoch 113 | batch 480/2304 [I 2022-10-28 07:25:10] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:58 [I 2022-10-28 07:25:10] ...trap/engines/engine.py.229: process: 0.56502 | load: 0.00255 [I 2022-10-28 07:25:10] ...trap/engines/engine.py.230: loss: 0.13450 [I 2022-10-28 07:25:10] ...optimizers/trijoint.py.151: total_norm: 46.713203 [I 2022-10-28 07:25:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 490/2304 [I 2022-10-28 07:25:16] ...trap/engines/engine.py.228: elapsed: 0:04:36 | left: 0:16:57 [I 2022-10-28 07:25:16] ...trap/engines/engine.py.229: process: 0.56525 | load: 0.00248 [I 2022-10-28 07:25:16] ...trap/engines/engine.py.230: loss: 0.07839 [I 2022-10-28 07:25:16] ...optimizers/trijoint.py.151: total_norm: 13.560511 [I 2022-10-28 07:25:21] ...trap/engines/engine.py.224: train: epoch 113 | batch 500/2304 [I 2022-10-28 07:25:21] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:47 [I 2022-10-28 07:25:21] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00252 [I 2022-10-28 07:25:21] ...trap/engines/engine.py.230: loss: 0.08840 [I 2022-10-28 07:25:21] ...optimizers/trijoint.py.151: total_norm: 14.303439 [I 2022-10-28 07:25:27] ...trap/engines/engine.py.224: train: epoch 113 | batch 510/2304 [I 2022-10-28 07:25:27] ...trap/engines/engine.py.228: elapsed: 0:04:47 | left: 0:16:34 [I 2022-10-28 07:25:27] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00248 [I 2022-10-28 07:25:27] ...trap/engines/engine.py.230: loss: 0.12537 [I 2022-10-28 07:25:27] ...optimizers/trijoint.py.151: total_norm: 12.641307 [I 2022-10-28 07:25:32] ...trap/engines/engine.py.224: train: epoch 113 | batch 520/2304 [I 2022-10-28 07:25:32] ...trap/engines/engine.py.228: elapsed: 0:04:53 | left: 0:16:34 [I 2022-10-28 07:25:32] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00257 [I 2022-10-28 07:25:32] ...trap/engines/engine.py.230: loss: 0.13450 [I 2022-10-28 07:25:32] ...optimizers/trijoint.py.151: total_norm: 10.208214 [I 2022-10-28 07:25:38] ...trap/engines/engine.py.224: train: epoch 113 | batch 530/2304 [I 2022-10-28 07:25:38] ...trap/engines/engine.py.228: elapsed: 0:04:58 | left: 0:16:23 [I 2022-10-28 07:25:38] ...trap/engines/engine.py.229: process: 0.55599 | load: 0.00244 [I 2022-10-28 07:25:38] ...trap/engines/engine.py.230: loss: 0.17466 [I 2022-10-28 07:25:38] ...optimizers/trijoint.py.151: total_norm: 21.522940 [I 2022-10-28 07:25:44] ...trap/engines/engine.py.224: train: epoch 113 | batch 540/2304 [I 2022-10-28 07:25:44] ...trap/engines/engine.py.228: elapsed: 0:05:04 | left: 0:16:14 [I 2022-10-28 07:25:44] ...trap/engines/engine.py.229: process: 0.54970 | load: 0.00248 [I 2022-10-28 07:25:44] ...trap/engines/engine.py.230: loss: 0.08768 [I 2022-10-28 07:25:44] ...optimizers/trijoint.py.151: total_norm: 11.559287 [I 2022-10-28 07:25:49] ...trap/engines/engine.py.224: train: epoch 113 | batch 550/2304 [I 2022-10-28 07:25:49] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:22 [I 2022-10-28 07:25:49] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00264 [I 2022-10-28 07:25:49] ...trap/engines/engine.py.230: loss: 0.08775 [I 2022-10-28 07:25:49] ...optimizers/trijoint.py.151: total_norm: 21.412144 [I 2022-10-28 07:25:55] ...trap/engines/engine.py.224: train: epoch 113 | batch 560/2304 [I 2022-10-28 07:25:55] ...trap/engines/engine.py.228: elapsed: 0:05:15 | left: 0:16:12 [I 2022-10-28 07:25:55] ...trap/engines/engine.py.229: process: 0.55817 | load: 0.00258 [I 2022-10-28 07:25:55] ...trap/engines/engine.py.230: loss: 0.09720 [I 2022-10-28 07:25:55] ...optimizers/trijoint.py.151: total_norm: 20.693233 [I 2022-10-28 07:26:00] ...trap/engines/engine.py.224: train: epoch 113 | batch 570/2304 [I 2022-10-28 07:26:00] ...trap/engines/engine.py.228: elapsed: 0:05:21 | left: 0:16:10 [I 2022-10-28 07:26:00] ...trap/engines/engine.py.229: process: 0.56484 | load: 0.00254 [I 2022-10-28 07:26:00] ...trap/engines/engine.py.230: loss: 0.14439 [I 2022-10-28 07:26:00] ...optimizers/trijoint.py.151: total_norm: 13.847260 [I 2022-10-28 07:26:06] ...trap/engines/engine.py.224: train: epoch 113 | batch 580/2304 [I 2022-10-28 07:26:06] ...trap/engines/engine.py.228: elapsed: 0:05:26 | left: 0:16:03 [I 2022-10-28 07:26:06] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00246 [I 2022-10-28 07:26:06] ...trap/engines/engine.py.230: loss: 0.10380 [I 2022-10-28 07:26:06] ...optimizers/trijoint.py.151: total_norm: 8.991405 [I 2022-10-28 07:26:12] ...trap/engines/engine.py.224: train: epoch 113 | batch 590/2304 [I 2022-10-28 07:26:12] ...trap/engines/engine.py.228: elapsed: 0:05:32 | left: 0:15:50 [I 2022-10-28 07:26:12] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00254 [I 2022-10-28 07:26:12] ...trap/engines/engine.py.230: loss: 0.11236 [I 2022-10-28 07:26:12] ...optimizers/trijoint.py.151: total_norm: 27.316380 [I 2022-10-28 07:26:17] ...trap/engines/engine.py.224: train: epoch 113 | batch 600/2304 [I 2022-10-28 07:26:17] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:36 [I 2022-10-28 07:26:17] ...trap/engines/engine.py.229: process: 0.53371 | load: 0.00252 [I 2022-10-28 07:26:17] ...trap/engines/engine.py.230: loss: 0.10320 [I 2022-10-28 07:26:17] ...optimizers/trijoint.py.151: total_norm: 16.507341 [I 2022-10-28 07:26:23] ...trap/engines/engine.py.224: train: epoch 113 | batch 610/2304 [I 2022-10-28 07:26:23] ...trap/engines/engine.py.228: elapsed: 0:05:43 | left: 0:15:46 [I 2022-10-28 07:26:23] ...trap/engines/engine.py.229: process: 0.56999 | load: 0.00243 [I 2022-10-28 07:26:23] ...trap/engines/engine.py.230: loss: 0.09379 [I 2022-10-28 07:26:23] ...optimizers/trijoint.py.151: total_norm: 16.364536 [I 2022-10-28 07:26:28] ...trap/engines/engine.py.224: train: epoch 113 | batch 620/2304 [I 2022-10-28 07:26:28] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:39 [I 2022-10-28 07:26:28] ...trap/engines/engine.py.229: process: 0.55428 | load: 0.00254 [I 2022-10-28 07:26:28] ...trap/engines/engine.py.230: loss: 0.18187 [I 2022-10-28 07:26:28] ...optimizers/trijoint.py.151: total_norm: 30.197275 [I 2022-10-28 07:26:34] ...trap/engines/engine.py.224: train: epoch 113 | batch 630/2304 [I 2022-10-28 07:26:34] ...trap/engines/engine.py.228: elapsed: 0:05:54 | left: 0:15:42 [I 2022-10-28 07:26:34] ...trap/engines/engine.py.229: process: 0.56709 | load: 0.00264 [I 2022-10-28 07:26:34] ...trap/engines/engine.py.230: loss: 0.11912 [I 2022-10-28 07:26:34] ...optimizers/trijoint.py.151: total_norm: 10.869564 [I 2022-10-28 07:26:39] ...trap/engines/engine.py.224: train: epoch 113 | batch 640/2304 [I 2022-10-28 07:26:39] ...trap/engines/engine.py.228: elapsed: 0:06:00 | left: 0:15:25 [I 2022-10-28 07:26:39] ...trap/engines/engine.py.229: process: 0.55840 | load: 0.00249 [I 2022-10-28 07:26:39] ...trap/engines/engine.py.230: loss: 0.08648 [I 2022-10-28 07:26:39] ...optimizers/trijoint.py.151: total_norm: 10.650938 [I 2022-10-28 07:26:45] ...trap/engines/engine.py.224: train: epoch 113 | batch 650/2304 [I 2022-10-28 07:26:45] ...trap/engines/engine.py.228: elapsed: 0:06:05 | left: 0:15:24 [I 2022-10-28 07:26:45] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00248 [I 2022-10-28 07:26:45] ...trap/engines/engine.py.230: loss: 0.09353 [I 2022-10-28 07:26:45] ...optimizers/trijoint.py.151: total_norm: 9.194439 [I 2022-10-28 07:26:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 660/2304 [I 2022-10-28 07:26:51] ...trap/engines/engine.py.228: elapsed: 0:06:11 | left: 0:15:27 [I 2022-10-28 07:26:51] ...trap/engines/engine.py.229: process: 0.55338 | load: 0.00248 [I 2022-10-28 07:26:51] ...trap/engines/engine.py.230: loss: 0.08789 [I 2022-10-28 07:26:51] ...optimizers/trijoint.py.151: total_norm: 9.393582 [I 2022-10-28 07:26:56] ...trap/engines/engine.py.224: train: epoch 113 | batch 670/2304 [I 2022-10-28 07:26:56] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:06 [I 2022-10-28 07:26:56] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00240 [I 2022-10-28 07:26:56] ...trap/engines/engine.py.230: loss: 0.13128 [I 2022-10-28 07:26:56] ...optimizers/trijoint.py.151: total_norm: 10.199915 [I 2022-10-28 07:27:02] ...trap/engines/engine.py.224: train: epoch 113 | batch 680/2304 [I 2022-10-28 07:27:02] ...trap/engines/engine.py.228: elapsed: 0:06:22 | left: 0:15:13 [I 2022-10-28 07:27:02] ...trap/engines/engine.py.229: process: 0.54266 | load: 0.00251 [I 2022-10-28 07:27:02] ...trap/engines/engine.py.230: loss: 0.14926 [I 2022-10-28 07:27:02] ...optimizers/trijoint.py.151: total_norm: 7.772337 [I 2022-10-28 07:27:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 690/2304 [I 2022-10-28 07:27:07] ...trap/engines/engine.py.228: elapsed: 0:06:28 | left: 0:14:54 [I 2022-10-28 07:27:07] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00252 [I 2022-10-28 07:27:07] ...trap/engines/engine.py.230: loss: 0.12429 [I 2022-10-28 07:27:07] ...optimizers/trijoint.py.151: total_norm: 7.945258 [I 2022-10-28 07:27:13] ...trap/engines/engine.py.224: train: epoch 113 | batch 700/2304 [I 2022-10-28 07:27:13] ...trap/engines/engine.py.228: elapsed: 0:06:33 | left: 0:14:53 [I 2022-10-28 07:27:13] ...trap/engines/engine.py.229: process: 0.55660 | load: 0.00243 [I 2022-10-28 07:27:13] ...trap/engines/engine.py.230: loss: 0.09977 [I 2022-10-28 07:27:13] ...optimizers/trijoint.py.151: total_norm: 12.314986 [I 2022-10-28 07:27:19] ...trap/engines/engine.py.224: train: epoch 113 | batch 710/2304 [I 2022-10-28 07:27:19] ...trap/engines/engine.py.228: elapsed: 0:06:39 | left: 0:14:47 [I 2022-10-28 07:27:19] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00246 [I 2022-10-28 07:27:19] ...trap/engines/engine.py.230: loss: 0.13860 [I 2022-10-28 07:27:19] ...optimizers/trijoint.py.151: total_norm: 27.818348 [I 2022-10-28 07:27:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 720/2304 [I 2022-10-28 07:27:24] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:42 [I 2022-10-28 07:27:24] ...trap/engines/engine.py.229: process: 0.55978 | load: 0.00248 [I 2022-10-28 07:27:24] ...trap/engines/engine.py.230: loss: 0.15481 [I 2022-10-28 07:27:24] ...optimizers/trijoint.py.151: total_norm: 26.802187 [I 2022-10-28 07:27:30] ...trap/engines/engine.py.224: train: epoch 113 | batch 730/2304 [I 2022-10-28 07:27:30] ...trap/engines/engine.py.228: elapsed: 0:06:50 | left: 0:14:41 [I 2022-10-28 07:27:30] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00249 [I 2022-10-28 07:27:30] ...trap/engines/engine.py.230: loss: 0.10236 [I 2022-10-28 07:27:30] ...optimizers/trijoint.py.151: total_norm: 8.460136 [I 2022-10-28 07:27:35] ...trap/engines/engine.py.224: train: epoch 113 | batch 740/2304 [I 2022-10-28 07:27:35] ...trap/engines/engine.py.228: elapsed: 0:06:56 | left: 0:14:35 [I 2022-10-28 07:27:35] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00248 [I 2022-10-28 07:27:35] ...trap/engines/engine.py.230: loss: 0.09907 [I 2022-10-28 07:27:35] ...optimizers/trijoint.py.151: total_norm: 9.685498 [I 2022-10-28 07:27:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 750/2304 [I 2022-10-28 07:27:41] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:35 [I 2022-10-28 07:27:41] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00261 [I 2022-10-28 07:27:41] ...trap/engines/engine.py.230: loss: 0.09974 [I 2022-10-28 07:27:41] ...optimizers/trijoint.py.151: total_norm: 11.209693 [I 2022-10-28 07:27:47] ...trap/engines/engine.py.224: train: epoch 113 | batch 760/2304 [I 2022-10-28 07:27:47] ...trap/engines/engine.py.228: elapsed: 0:07:07 | left: 0:14:29 [I 2022-10-28 07:27:47] ...trap/engines/engine.py.229: process: 0.56633 | load: 0.00260 [I 2022-10-28 07:27:47] ...trap/engines/engine.py.230: loss: 0.16250 [I 2022-10-28 07:27:47] ...optimizers/trijoint.py.151: total_norm: 10.364566 [I 2022-10-28 07:27:52] ...trap/engines/engine.py.224: train: epoch 113 | batch 770/2304 [I 2022-10-28 07:27:52] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:19 [I 2022-10-28 07:27:52] ...trap/engines/engine.py.229: process: 0.57167 | load: 0.00253 [I 2022-10-28 07:27:52] ...trap/engines/engine.py.230: loss: 0.11899 [I 2022-10-28 07:27:52] ...optimizers/trijoint.py.151: total_norm: 10.869645 [I 2022-10-28 07:27:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 780/2304 [I 2022-10-28 07:27:58] ...trap/engines/engine.py.228: elapsed: 0:07:18 | left: 0:14:10 [I 2022-10-28 07:27:58] ...trap/engines/engine.py.229: process: 0.55516 | load: 0.00256 [I 2022-10-28 07:27:58] ...trap/engines/engine.py.230: loss: 0.11897 [I 2022-10-28 07:27:58] ...optimizers/trijoint.py.151: total_norm: 15.279951 [I 2022-10-28 07:28:03] ...trap/engines/engine.py.224: train: epoch 113 | batch 790/2304 [I 2022-10-28 07:28:03] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:05 [I 2022-10-28 07:28:03] ...trap/engines/engine.py.229: process: 0.55694 | load: 0.00252 [I 2022-10-28 07:28:03] ...trap/engines/engine.py.230: loss: 0.10258 [I 2022-10-28 07:28:03] ...optimizers/trijoint.py.151: total_norm: 12.831440 [I 2022-10-28 07:28:09] ...trap/engines/engine.py.224: train: epoch 113 | batch 800/2304 [I 2022-10-28 07:28:09] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:14:05 [I 2022-10-28 07:28:09] ...trap/engines/engine.py.229: process: 0.56205 | load: 0.00252 [I 2022-10-28 07:28:09] ...trap/engines/engine.py.230: loss: 0.12352 [I 2022-10-28 07:28:09] ...optimizers/trijoint.py.151: total_norm: 23.547689 [I 2022-10-28 07:28:15] ...trap/engines/engine.py.224: train: epoch 113 | batch 810/2304 [I 2022-10-28 07:28:15] ...trap/engines/engine.py.228: elapsed: 0:07:35 | left: 0:13:53 [I 2022-10-28 07:28:15] ...trap/engines/engine.py.229: process: 0.56086 | load: 0.00249 [I 2022-10-28 07:28:15] ...trap/engines/engine.py.230: loss: 0.15754 [I 2022-10-28 07:28:15] ...optimizers/trijoint.py.151: total_norm: 9.513719 [I 2022-10-28 07:28:20] ...trap/engines/engine.py.224: train: epoch 113 | batch 820/2304 [I 2022-10-28 07:28:20] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:51 [I 2022-10-28 07:28:20] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00249 [I 2022-10-28 07:28:20] ...trap/engines/engine.py.230: loss: 0.15129 [I 2022-10-28 07:28:20] ...optimizers/trijoint.py.151: total_norm: 16.123991 [I 2022-10-28 07:28:26] ...trap/engines/engine.py.224: train: epoch 113 | batch 830/2304 [I 2022-10-28 07:28:26] ...trap/engines/engine.py.228: elapsed: 0:07:46 | left: 0:13:38 [I 2022-10-28 07:28:26] ...trap/engines/engine.py.229: process: 0.55430 | load: 0.00241 [I 2022-10-28 07:28:26] ...trap/engines/engine.py.230: loss: 0.12152 [I 2022-10-28 07:28:26] ...optimizers/trijoint.py.151: total_norm: 14.909490 [I 2022-10-28 07:28:31] ...trap/engines/engine.py.224: train: epoch 113 | batch 840/2304 [I 2022-10-28 07:28:31] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:36 [I 2022-10-28 07:28:31] ...trap/engines/engine.py.229: process: 0.56437 | load: 0.00242 [I 2022-10-28 07:28:31] ...trap/engines/engine.py.230: loss: 0.10542 [I 2022-10-28 07:28:31] ...optimizers/trijoint.py.151: total_norm: 10.602358 [I 2022-10-28 07:28:37] ...trap/engines/engine.py.224: train: epoch 113 | batch 850/2304 [I 2022-10-28 07:28:37] ...trap/engines/engine.py.228: elapsed: 0:07:57 | left: 0:13:26 [I 2022-10-28 07:28:37] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00241 [I 2022-10-28 07:28:37] ...trap/engines/engine.py.230: loss: 0.11309 [I 2022-10-28 07:28:37] ...optimizers/trijoint.py.151: total_norm: 6.738470 [I 2022-10-28 07:28:43] ...trap/engines/engine.py.224: train: epoch 113 | batch 860/2304 [I 2022-10-28 07:28:43] ...trap/engines/engine.py.228: elapsed: 0:08:03 | left: 0:13:28 [I 2022-10-28 07:28:43] ...trap/engines/engine.py.229: process: 0.55878 | load: 0.00243 [I 2022-10-28 07:28:43] ...trap/engines/engine.py.230: loss: 0.10187 [I 2022-10-28 07:28:43] ...optimizers/trijoint.py.151: total_norm: 44.149254 [I 2022-10-28 07:28:48] ...trap/engines/engine.py.224: train: epoch 113 | batch 870/2304 [I 2022-10-28 07:28:48] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:25 [I 2022-10-28 07:28:48] ...trap/engines/engine.py.229: process: 0.56558 | load: 0.00263 [I 2022-10-28 07:28:48] ...trap/engines/engine.py.230: loss: 0.08240 [I 2022-10-28 07:28:48] ...optimizers/trijoint.py.151: total_norm: 11.259411 [I 2022-10-28 07:28:54] ...trap/engines/engine.py.224: train: epoch 113 | batch 880/2304 [I 2022-10-28 07:28:54] ...trap/engines/engine.py.228: elapsed: 0:08:14 | left: 0:13:16 [I 2022-10-28 07:28:54] ...trap/engines/engine.py.229: process: 0.57036 | load: 0.00251 [I 2022-10-28 07:28:54] ...trap/engines/engine.py.230: loss: 0.10185 [I 2022-10-28 07:28:54] ...optimizers/trijoint.py.151: total_norm: 12.895269 [I 2022-10-28 07:28:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 890/2304 [I 2022-10-28 07:28:59] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:08 [I 2022-10-28 07:28:59] ...trap/engines/engine.py.229: process: 0.55398 | load: 0.00246 [I 2022-10-28 07:28:59] ...trap/engines/engine.py.230: loss: 0.13123 [I 2022-10-28 07:28:59] ...optimizers/trijoint.py.151: total_norm: 12.089839 [I 2022-10-28 07:29:05] ...trap/engines/engine.py.224: train: epoch 113 | batch 900/2304 [I 2022-10-28 07:29:05] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:04 [I 2022-10-28 07:29:05] ...trap/engines/engine.py.229: process: 0.56467 | load: 0.00250 [I 2022-10-28 07:29:05] ...trap/engines/engine.py.230: loss: 0.09033 [I 2022-10-28 07:29:05] ...optimizers/trijoint.py.151: total_norm: 11.154690 [I 2022-10-28 07:29:11] ...trap/engines/engine.py.224: train: epoch 113 | batch 910/2304 [I 2022-10-28 07:29:11] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:13:01 [I 2022-10-28 07:29:11] ...trap/engines/engine.py.229: process: 0.55923 | load: 0.00250 [I 2022-10-28 07:29:11] ...trap/engines/engine.py.230: loss: 0.20600 [I 2022-10-28 07:29:11] ...optimizers/trijoint.py.151: total_norm: 13.886594 [I 2022-10-28 07:29:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 920/2304 [I 2022-10-28 07:29:16] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:55 [I 2022-10-28 07:29:16] ...trap/engines/engine.py.229: process: 0.55497 | load: 0.00250 [I 2022-10-28 07:29:16] ...trap/engines/engine.py.230: loss: 0.10327 [I 2022-10-28 07:29:16] ...optimizers/trijoint.py.151: total_norm: 15.419926 [I 2022-10-28 07:29:22] ...trap/engines/engine.py.224: train: epoch 113 | batch 930/2304 [I 2022-10-28 07:29:22] ...trap/engines/engine.py.228: elapsed: 0:08:42 | left: 0:12:42 [I 2022-10-28 07:29:22] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00254 [I 2022-10-28 07:29:22] ...trap/engines/engine.py.230: loss: 0.10207 [I 2022-10-28 07:29:22] ...optimizers/trijoint.py.151: total_norm: 16.675886 [I 2022-10-28 07:29:27] ...trap/engines/engine.py.224: train: epoch 113 | batch 940/2304 [I 2022-10-28 07:29:27] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:35 [I 2022-10-28 07:29:27] ...trap/engines/engine.py.229: process: 0.54862 | load: 0.00251 [I 2022-10-28 07:29:27] ...trap/engines/engine.py.230: loss: 0.18934 [I 2022-10-28 07:29:27] ...optimizers/trijoint.py.151: total_norm: 20.058832 [I 2022-10-28 07:29:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 950/2304 [I 2022-10-28 07:29:33] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:35 [I 2022-10-28 07:29:33] ...trap/engines/engine.py.229: process: 0.55964 | load: 0.00252 [I 2022-10-28 07:29:33] ...trap/engines/engine.py.230: loss: 0.13499 [I 2022-10-28 07:29:33] ...optimizers/trijoint.py.151: total_norm: 17.584349 [I 2022-10-28 07:29:39] ...trap/engines/engine.py.224: train: epoch 113 | batch 960/2304 [I 2022-10-28 07:29:39] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:30 [I 2022-10-28 07:29:39] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00261 [I 2022-10-28 07:29:39] ...trap/engines/engine.py.230: loss: 0.08428 [I 2022-10-28 07:29:39] ...optimizers/trijoint.py.151: total_norm: 18.690956 [I 2022-10-28 07:29:44] ...trap/engines/engine.py.224: train: epoch 113 | batch 970/2304 [I 2022-10-28 07:29:44] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:30 [I 2022-10-28 07:29:44] ...trap/engines/engine.py.229: process: 0.56127 | load: 0.00255 [I 2022-10-28 07:29:44] ...trap/engines/engine.py.230: loss: 0.13171 [I 2022-10-28 07:29:44] ...optimizers/trijoint.py.151: total_norm: 14.413356 [I 2022-10-28 07:29:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 980/2304 [I 2022-10-28 07:29:50] ...trap/engines/engine.py.228: elapsed: 0:09:10 | left: 0:12:11 [I 2022-10-28 07:29:50] ...trap/engines/engine.py.229: process: 0.54784 | load: 0.00251 [I 2022-10-28 07:29:50] ...trap/engines/engine.py.230: loss: 0.16861 [I 2022-10-28 07:29:50] ...optimizers/trijoint.py.151: total_norm: 26.176624 [I 2022-10-28 07:29:55] ...trap/engines/engine.py.224: train: epoch 113 | batch 990/2304 [I 2022-10-28 07:29:55] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:02 [I 2022-10-28 07:29:55] ...trap/engines/engine.py.229: process: 0.54384 | load: 0.00250 [I 2022-10-28 07:29:55] ...trap/engines/engine.py.230: loss: 0.12957 [I 2022-10-28 07:29:55] ...optimizers/trijoint.py.151: total_norm: 9.592653 [I 2022-10-28 07:30:01] ...trap/engines/engine.py.224: train: epoch 113 | batch 1000/2304 [I 2022-10-28 07:30:01] ...trap/engines/engine.py.228: elapsed: 0:09:21 | left: 0:12:04 [I 2022-10-28 07:30:01] ...trap/engines/engine.py.229: process: 0.54874 | load: 0.00247 [I 2022-10-28 07:30:01] ...trap/engines/engine.py.230: loss: 0.09683 [I 2022-10-28 07:30:01] ...optimizers/trijoint.py.151: total_norm: 17.880133 [I 2022-10-28 07:30:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 1010/2304 [I 2022-10-28 07:30:07] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:11:56 [I 2022-10-28 07:30:07] ...trap/engines/engine.py.229: process: 0.53913 | load: 0.00248 [I 2022-10-28 07:30:07] ...trap/engines/engine.py.230: loss: 0.10263 [I 2022-10-28 07:30:07] ...optimizers/trijoint.py.151: total_norm: 12.922321 [I 2022-10-28 07:30:12] ...trap/engines/engine.py.224: train: epoch 113 | batch 1020/2304 [I 2022-10-28 07:30:12] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:56 [I 2022-10-28 07:30:12] ...trap/engines/engine.py.229: process: 0.56091 | load: 0.00243 [I 2022-10-28 07:30:12] ...trap/engines/engine.py.230: loss: 0.09145 [I 2022-10-28 07:30:12] ...optimizers/trijoint.py.151: total_norm: 12.509147 [I 2022-10-28 07:30:18] ...trap/engines/engine.py.224: train: epoch 113 | batch 1030/2304 [I 2022-10-28 07:30:18] ...trap/engines/engine.py.228: elapsed: 0:09:38 | left: 0:11:51 [I 2022-10-28 07:30:18] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00244 [I 2022-10-28 07:30:18] ...trap/engines/engine.py.230: loss: 0.09041 [I 2022-10-28 07:30:18] ...optimizers/trijoint.py.151: total_norm: 24.061069 [I 2022-10-28 07:30:23] ...trap/engines/engine.py.224: train: epoch 113 | batch 1040/2304 [I 2022-10-28 07:30:23] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:51 [I 2022-10-28 07:30:23] ...trap/engines/engine.py.229: process: 0.56014 | load: 0.00252 [I 2022-10-28 07:30:23] ...trap/engines/engine.py.230: loss: 0.11111 [I 2022-10-28 07:30:23] ...optimizers/trijoint.py.151: total_norm: 9.377142 [I 2022-10-28 07:30:29] ...trap/engines/engine.py.224: train: epoch 113 | batch 1050/2304 [I 2022-10-28 07:30:29] ...trap/engines/engine.py.228: elapsed: 0:09:49 | left: 0:11:40 [I 2022-10-28 07:30:29] ...trap/engines/engine.py.229: process: 0.57114 | load: 0.00245 [I 2022-10-28 07:30:29] ...trap/engines/engine.py.230: loss: 0.14561 [I 2022-10-28 07:30:29] ...optimizers/trijoint.py.151: total_norm: 22.891882 [I 2022-10-28 07:30:35] ...trap/engines/engine.py.224: train: epoch 113 | batch 1060/2304 [I 2022-10-28 07:30:35] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:33 [I 2022-10-28 07:30:35] ...trap/engines/engine.py.229: process: 0.56369 | load: 0.00246 [I 2022-10-28 07:30:35] ...trap/engines/engine.py.230: loss: 0.14532 [I 2022-10-28 07:30:35] ...optimizers/trijoint.py.151: total_norm: 58.557892 [I 2022-10-28 07:30:40] ...trap/engines/engine.py.224: train: epoch 113 | batch 1070/2304 [I 2022-10-28 07:30:40] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:18 [I 2022-10-28 07:30:40] ...trap/engines/engine.py.229: process: 0.53804 | load: 0.00243 [I 2022-10-28 07:30:40] ...trap/engines/engine.py.230: loss: 0.09802 [I 2022-10-28 07:30:40] ...optimizers/trijoint.py.151: total_norm: 9.768692 [I 2022-10-28 07:30:46] ...trap/engines/engine.py.224: train: epoch 113 | batch 1080/2304 [I 2022-10-28 07:30:46] ...trap/engines/engine.py.228: elapsed: 0:10:06 | left: 0:11:27 [I 2022-10-28 07:30:46] ...trap/engines/engine.py.229: process: 0.57759 | load: 0.00245 [I 2022-10-28 07:30:46] ...trap/engines/engine.py.230: loss: 0.11240 [I 2022-10-28 07:30:46] ...optimizers/trijoint.py.151: total_norm: 20.024689 [I 2022-10-28 07:30:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 1090/2304 [I 2022-10-28 07:30:51] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:13 [I 2022-10-28 07:30:51] ...trap/engines/engine.py.229: process: 0.55220 | load: 0.00249 [I 2022-10-28 07:30:51] ...trap/engines/engine.py.230: loss: 0.09441 [I 2022-10-28 07:30:51] ...optimizers/trijoint.py.151: total_norm: 20.171898 [I 2022-10-28 07:30:57] ...trap/engines/engine.py.224: train: epoch 113 | batch 1100/2304 [I 2022-10-28 07:30:57] ...trap/engines/engine.py.228: elapsed: 0:10:17 | left: 0:11:12 [I 2022-10-28 07:30:57] ...trap/engines/engine.py.229: process: 0.56237 | load: 0.00252 [I 2022-10-28 07:30:57] ...trap/engines/engine.py.230: loss: 0.10621 [I 2022-10-28 07:30:57] ...optimizers/trijoint.py.151: total_norm: 10.697543 [I 2022-10-28 07:31:02] ...trap/engines/engine.py.224: train: epoch 113 | batch 1110/2304 [I 2022-10-28 07:31:02] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:04 [I 2022-10-28 07:31:02] ...trap/engines/engine.py.229: process: 0.55341 | load: 0.00269 [I 2022-10-28 07:31:02] ...trap/engines/engine.py.230: loss: 0.17148 [I 2022-10-28 07:31:02] ...optimizers/trijoint.py.151: total_norm: 23.090572 [I 2022-10-28 07:31:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 1120/2304 [I 2022-10-28 07:31:08] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:11:01 [I 2022-10-28 07:31:08] ...trap/engines/engine.py.229: process: 0.55958 | load: 0.00244 [I 2022-10-28 07:31:08] ...trap/engines/engine.py.230: loss: 0.10446 [I 2022-10-28 07:31:08] ...optimizers/trijoint.py.151: total_norm: 13.033572 [I 2022-10-28 07:31:14] ...trap/engines/engine.py.224: train: epoch 113 | batch 1130/2304 [I 2022-10-28 07:31:14] ...trap/engines/engine.py.228: elapsed: 0:10:34 | left: 0:10:58 [I 2022-10-28 07:31:14] ...trap/engines/engine.py.229: process: 0.55839 | load: 0.00247 [I 2022-10-28 07:31:14] ...trap/engines/engine.py.230: loss: 0.09355 [I 2022-10-28 07:31:14] ...optimizers/trijoint.py.151: total_norm: 7.810813 [I 2022-10-28 07:31:19] ...trap/engines/engine.py.224: train: epoch 113 | batch 1140/2304 [I 2022-10-28 07:31:19] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:43 [I 2022-10-28 07:31:19] ...trap/engines/engine.py.229: process: 0.55110 | load: 0.00251 [I 2022-10-28 07:31:19] ...trap/engines/engine.py.230: loss: 0.16135 [I 2022-10-28 07:31:19] ...optimizers/trijoint.py.151: total_norm: 26.517424 [I 2022-10-28 07:31:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 1150/2304 [I 2022-10-28 07:31:25] ...trap/engines/engine.py.228: elapsed: 0:10:45 | left: 0:10:48 [I 2022-10-28 07:31:25] ...trap/engines/engine.py.229: process: 0.54003 | load: 0.00253 [I 2022-10-28 07:31:25] ...trap/engines/engine.py.230: loss: 0.10774 [I 2022-10-28 07:31:25] ...optimizers/trijoint.py.151: total_norm: 18.508318 [I 2022-10-28 07:31:30] ...trap/engines/engine.py.224: train: epoch 113 | batch 1160/2304 [I 2022-10-28 07:31:30] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:39 [I 2022-10-28 07:31:30] ...trap/engines/engine.py.229: process: 0.56722 | load: 0.00248 [I 2022-10-28 07:31:30] ...trap/engines/engine.py.230: loss: 0.16120 [I 2022-10-28 07:31:30] ...optimizers/trijoint.py.151: total_norm: 18.982958 [I 2022-10-28 07:31:36] ...trap/engines/engine.py.224: train: epoch 113 | batch 1170/2304 [I 2022-10-28 07:31:36] ...trap/engines/engine.py.228: elapsed: 0:10:56 | left: 0:10:28 [I 2022-10-28 07:31:36] ...trap/engines/engine.py.229: process: 0.55339 | load: 0.00247 [I 2022-10-28 07:31:36] ...trap/engines/engine.py.230: loss: 0.11362 [I 2022-10-28 07:31:36] ...optimizers/trijoint.py.151: total_norm: 16.255301 [I 2022-10-28 07:31:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1180/2304 [I 2022-10-28 07:31:42] ...trap/engines/engine.py.228: elapsed: 0:11:02 | left: 0:10:27 [I 2022-10-28 07:31:42] ...trap/engines/engine.py.229: process: 0.56939 | load: 0.00270 [I 2022-10-28 07:31:42] ...trap/engines/engine.py.230: loss: 0.15361 [I 2022-10-28 07:31:42] ...optimizers/trijoint.py.151: total_norm: 11.909745 [I 2022-10-28 07:31:47] ...trap/engines/engine.py.224: train: epoch 113 | batch 1190/2304 [I 2022-10-28 07:31:47] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:20 [I 2022-10-28 07:31:47] ...trap/engines/engine.py.229: process: 0.54491 | load: 0.00256 [I 2022-10-28 07:31:47] ...trap/engines/engine.py.230: loss: 0.09111 [I 2022-10-28 07:31:47] ...optimizers/trijoint.py.151: total_norm: 19.248487 [I 2022-10-28 07:31:53] ...trap/engines/engine.py.224: train: epoch 113 | batch 1200/2304 [I 2022-10-28 07:31:53] ...trap/engines/engine.py.228: elapsed: 0:11:13 | left: 0:10:14 [I 2022-10-28 07:31:53] ...trap/engines/engine.py.229: process: 0.55213 | load: 0.00256 [I 2022-10-28 07:31:53] ...trap/engines/engine.py.230: loss: 0.11601 [I 2022-10-28 07:31:53] ...optimizers/trijoint.py.151: total_norm: 13.896535 [I 2022-10-28 07:31:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 1210/2304 [I 2022-10-28 07:31:58] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:25 [I 2022-10-28 07:31:58] ...trap/engines/engine.py.229: process: 0.56530 | load: 0.00252 [I 2022-10-28 07:31:59] ...trap/engines/engine.py.230: loss: 0.08866 [I 2022-10-28 07:31:59] ...optimizers/trijoint.py.151: total_norm: 21.633287 [I 2022-10-28 07:32:04] ...trap/engines/engine.py.224: train: epoch 113 | batch 1220/2304 [I 2022-10-28 07:32:04] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:04 [I 2022-10-28 07:32:04] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00247 [I 2022-10-28 07:32:04] ...trap/engines/engine.py.230: loss: 0.08115 [I 2022-10-28 07:32:04] ...optimizers/trijoint.py.151: total_norm: 12.991947 [I 2022-10-28 07:32:10] ...trap/engines/engine.py.224: train: epoch 113 | batch 1230/2304 [I 2022-10-28 07:32:10] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:55 [I 2022-10-28 07:32:10] ...trap/engines/engine.py.229: process: 0.55080 | load: 0.00251 [I 2022-10-28 07:32:10] ...trap/engines/engine.py.230: loss: 0.09722 [I 2022-10-28 07:32:10] ...optimizers/trijoint.py.151: total_norm: 15.622933 [I 2022-10-28 07:32:15] ...trap/engines/engine.py.224: train: epoch 113 | batch 1240/2304 [I 2022-10-28 07:32:15] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:48 [I 2022-10-28 07:32:15] ...trap/engines/engine.py.229: process: 0.56163 | load: 0.00244 [I 2022-10-28 07:32:15] ...trap/engines/engine.py.230: loss: 0.07682 [I 2022-10-28 07:32:15] ...optimizers/trijoint.py.151: total_norm: 21.139732 [I 2022-10-28 07:32:21] ...trap/engines/engine.py.224: train: epoch 113 | batch 1250/2304 [I 2022-10-28 07:32:21] ...trap/engines/engine.py.228: elapsed: 0:11:41 | left: 0:09:52 [I 2022-10-28 07:32:21] ...trap/engines/engine.py.229: process: 0.56565 | load: 0.00252 [I 2022-10-28 07:32:21] ...trap/engines/engine.py.230: loss: 0.09182 [I 2022-10-28 07:32:21] ...optimizers/trijoint.py.151: total_norm: 11.679865 [I 2022-10-28 07:32:26] ...trap/engines/engine.py.224: train: epoch 113 | batch 1260/2304 [I 2022-10-28 07:32:26] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:43 [I 2022-10-28 07:32:26] ...trap/engines/engine.py.229: process: 0.55652 | load: 0.00247 [I 2022-10-28 07:32:26] ...trap/engines/engine.py.230: loss: 0.12476 [I 2022-10-28 07:32:26] ...optimizers/trijoint.py.151: total_norm: 14.479850 [I 2022-10-28 07:32:32] ...trap/engines/engine.py.224: train: epoch 113 | batch 1270/2304 [I 2022-10-28 07:32:32] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:41 [I 2022-10-28 07:32:32] ...trap/engines/engine.py.229: process: 0.57117 | load: 0.00242 [I 2022-10-28 07:32:32] ...trap/engines/engine.py.230: loss: 0.09131 [I 2022-10-28 07:32:32] ...optimizers/trijoint.py.151: total_norm: 9.615044 [I 2022-10-28 07:32:38] ...trap/engines/engine.py.224: train: epoch 113 | batch 1280/2304 [I 2022-10-28 07:32:38] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:37 [I 2022-10-28 07:32:38] ...trap/engines/engine.py.229: process: 0.57078 | load: 0.00245 [I 2022-10-28 07:32:38] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-28 07:32:38] ...optimizers/trijoint.py.151: total_norm: 20.011789 [I 2022-10-28 07:32:43] ...trap/engines/engine.py.224: train: epoch 113 | batch 1290/2304 [I 2022-10-28 07:32:43] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:27 [I 2022-10-28 07:32:43] ...trap/engines/engine.py.229: process: 0.56918 | load: 0.00247 [I 2022-10-28 07:32:43] ...trap/engines/engine.py.230: loss: 0.17419 [I 2022-10-28 07:32:43] ...optimizers/trijoint.py.151: total_norm: 14.392917 [I 2022-10-28 07:32:49] ...trap/engines/engine.py.224: train: epoch 113 | batch 1300/2304 [I 2022-10-28 07:32:49] ...trap/engines/engine.py.228: elapsed: 0:12:09 | left: 0:09:26 [I 2022-10-28 07:32:49] ...trap/engines/engine.py.229: process: 0.56503 | load: 0.00261 [I 2022-10-28 07:32:49] ...trap/engines/engine.py.230: loss: 0.13299 [I 2022-10-28 07:32:49] ...optimizers/trijoint.py.151: total_norm: 16.523283 [I 2022-10-28 07:32:55] ...trap/engines/engine.py.224: train: epoch 113 | batch 1310/2304 [I 2022-10-28 07:32:55] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:17 [I 2022-10-28 07:32:55] ...trap/engines/engine.py.229: process: 0.56041 | load: 0.00250 [I 2022-10-28 07:32:55] ...trap/engines/engine.py.230: loss: 0.10089 [I 2022-10-28 07:32:55] ...optimizers/trijoint.py.151: total_norm: 15.505023 [I 2022-10-28 07:33:00] ...trap/engines/engine.py.224: train: epoch 113 | batch 1320/2304 [I 2022-10-28 07:33:00] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:11 [I 2022-10-28 07:33:00] ...trap/engines/engine.py.229: process: 0.57181 | load: 0.00244 [I 2022-10-28 07:33:00] ...trap/engines/engine.py.230: loss: 0.08812 [I 2022-10-28 07:33:00] ...optimizers/trijoint.py.151: total_norm: 11.829657 [I 2022-10-28 07:33:06] ...trap/engines/engine.py.224: train: epoch 113 | batch 1330/2304 [I 2022-10-28 07:33:06] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:03 [I 2022-10-28 07:33:06] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00246 [I 2022-10-28 07:33:06] ...trap/engines/engine.py.230: loss: 0.10651 [I 2022-10-28 07:33:06] ...optimizers/trijoint.py.151: total_norm: 10.145050 [I 2022-10-28 07:33:11] ...trap/engines/engine.py.224: train: epoch 113 | batch 1340/2304 [I 2022-10-28 07:33:11] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:09:04 [I 2022-10-28 07:33:11] ...trap/engines/engine.py.229: process: 0.57480 | load: 0.00257 [I 2022-10-28 07:33:11] ...trap/engines/engine.py.230: loss: 0.08133 [I 2022-10-28 07:33:11] ...optimizers/trijoint.py.151: total_norm: 17.557041 [I 2022-10-28 07:33:17] ...trap/engines/engine.py.224: train: epoch 113 | batch 1350/2304 [I 2022-10-28 07:33:17] ...trap/engines/engine.py.228: elapsed: 0:12:37 | left: 0:08:57 [I 2022-10-28 07:33:17] ...trap/engines/engine.py.229: process: 0.56431 | load: 0.00254 [I 2022-10-28 07:33:17] ...trap/engines/engine.py.230: loss: 0.17542 [I 2022-10-28 07:33:17] ...optimizers/trijoint.py.151: total_norm: 52.538090 [I 2022-10-28 07:33:23] ...trap/engines/engine.py.224: train: epoch 113 | batch 1360/2304 [I 2022-10-28 07:33:23] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:50 [I 2022-10-28 07:33:23] ...trap/engines/engine.py.229: process: 0.57657 | load: 0.00245 [I 2022-10-28 07:33:23] ...trap/engines/engine.py.230: loss: 0.11757 [I 2022-10-28 07:33:23] ...optimizers/trijoint.py.151: total_norm: 18.113159 [I 2022-10-28 07:33:28] ...trap/engines/engine.py.224: train: epoch 113 | batch 1370/2304 [I 2022-10-28 07:33:28] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:47 [I 2022-10-28 07:33:28] ...trap/engines/engine.py.229: process: 0.56895 | load: 0.00255 [I 2022-10-28 07:33:28] ...trap/engines/engine.py.230: loss: 0.08621 [I 2022-10-28 07:33:28] ...optimizers/trijoint.py.151: total_norm: 10.109694 [I 2022-10-28 07:33:34] ...trap/engines/engine.py.224: train: epoch 113 | batch 1380/2304 [I 2022-10-28 07:33:34] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:34 [I 2022-10-28 07:33:34] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00259 [I 2022-10-28 07:33:34] ...trap/engines/engine.py.230: loss: 0.13082 [I 2022-10-28 07:33:34] ...optimizers/trijoint.py.151: total_norm: 27.291618 [I 2022-10-28 07:33:39] ...trap/engines/engine.py.224: train: epoch 113 | batch 1390/2304 [I 2022-10-28 07:33:39] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:33 [I 2022-10-28 07:33:39] ...trap/engines/engine.py.229: process: 0.57930 | load: 0.00254 [I 2022-10-28 07:33:39] ...trap/engines/engine.py.230: loss: 0.11710 [I 2022-10-28 07:33:39] ...optimizers/trijoint.py.151: total_norm: 12.514023 [I 2022-10-28 07:33:45] ...trap/engines/engine.py.224: train: epoch 113 | batch 1400/2304 [I 2022-10-28 07:33:45] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:21 [I 2022-10-28 07:33:45] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00250 [I 2022-10-28 07:33:45] ...trap/engines/engine.py.230: loss: 0.11383 [I 2022-10-28 07:33:45] ...optimizers/trijoint.py.151: total_norm: 15.804481 [I 2022-10-28 07:33:51] ...trap/engines/engine.py.224: train: epoch 113 | batch 1410/2304 [I 2022-10-28 07:33:51] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:17 [I 2022-10-28 07:33:51] ...trap/engines/engine.py.229: process: 0.54800 | load: 0.00251 [I 2022-10-28 07:33:51] ...trap/engines/engine.py.230: loss: 0.13983 [I 2022-10-28 07:33:51] ...optimizers/trijoint.py.151: total_norm: 20.542006 [I 2022-10-28 07:33:56] ...trap/engines/engine.py.224: train: epoch 113 | batch 1420/2304 [I 2022-10-28 07:33:56] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:15 [I 2022-10-28 07:33:56] ...trap/engines/engine.py.229: process: 0.56643 | load: 0.00249 [I 2022-10-28 07:33:56] ...trap/engines/engine.py.230: loss: 0.13686 [I 2022-10-28 07:33:56] ...optimizers/trijoint.py.151: total_norm: 12.098008 [I 2022-10-28 07:34:02] ...trap/engines/engine.py.224: train: epoch 113 | batch 1430/2304 [I 2022-10-28 07:34:02] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:04 [I 2022-10-28 07:34:02] ...trap/engines/engine.py.229: process: 0.55094 | load: 0.00244 [I 2022-10-28 07:34:02] ...trap/engines/engine.py.230: loss: 0.11024 [I 2022-10-28 07:34:02] ...optimizers/trijoint.py.151: total_norm: 17.335926 [I 2022-10-28 07:34:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 1440/2304 [I 2022-10-28 07:34:07] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:00 [I 2022-10-28 07:34:07] ...trap/engines/engine.py.229: process: 0.56452 | load: 0.00243 [I 2022-10-28 07:34:07] ...trap/engines/engine.py.230: loss: 0.09458 [I 2022-10-28 07:34:07] ...optimizers/trijoint.py.151: total_norm: 14.860527 [I 2022-10-28 07:34:13] ...trap/engines/engine.py.224: train: epoch 113 | batch 1450/2304 [I 2022-10-28 07:34:13] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:55 [I 2022-10-28 07:34:13] ...trap/engines/engine.py.229: process: 0.55730 | load: 0.00249 [I 2022-10-28 07:34:13] ...trap/engines/engine.py.230: loss: 0.11170 [I 2022-10-28 07:34:13] ...optimizers/trijoint.py.151: total_norm: 44.349758 [I 2022-10-28 07:34:19] ...trap/engines/engine.py.224: train: epoch 113 | batch 1460/2304 [I 2022-10-28 07:34:19] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:49 [I 2022-10-28 07:34:19] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00246 [I 2022-10-28 07:34:19] ...trap/engines/engine.py.230: loss: 0.07085 [I 2022-10-28 07:34:19] ...optimizers/trijoint.py.151: total_norm: 24.590984 [I 2022-10-28 07:34:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 1470/2304 [I 2022-10-28 07:34:24] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:43 [I 2022-10-28 07:34:24] ...trap/engines/engine.py.229: process: 0.54887 | load: 0.00241 [I 2022-10-28 07:34:24] ...trap/engines/engine.py.230: loss: 0.14524 [I 2022-10-28 07:34:24] ...optimizers/trijoint.py.151: total_norm: 18.279001 [I 2022-10-28 07:34:30] ...trap/engines/engine.py.224: train: epoch 113 | batch 1480/2304 [I 2022-10-28 07:34:30] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:38 [I 2022-10-28 07:34:30] ...trap/engines/engine.py.229: process: 0.56159 | load: 0.00245 [I 2022-10-28 07:34:30] ...trap/engines/engine.py.230: loss: 0.12768 [I 2022-10-28 07:34:30] ...optimizers/trijoint.py.151: total_norm: 14.602884 [I 2022-10-28 07:34:35] ...trap/engines/engine.py.224: train: epoch 113 | batch 1490/2304 [I 2022-10-28 07:34:35] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:32 [I 2022-10-28 07:34:35] ...trap/engines/engine.py.229: process: 0.54908 | load: 0.00256 [I 2022-10-28 07:34:35] ...trap/engines/engine.py.230: loss: 0.15886 [I 2022-10-28 07:34:35] ...optimizers/trijoint.py.151: total_norm: 18.522039 [I 2022-10-28 07:34:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 1500/2304 [I 2022-10-28 07:34:41] ...trap/engines/engine.py.228: elapsed: 0:14:01 | left: 0:07:28 [I 2022-10-28 07:34:41] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00253 [I 2022-10-28 07:34:41] ...trap/engines/engine.py.230: loss: 0.09134 [I 2022-10-28 07:34:41] ...optimizers/trijoint.py.151: total_norm: 14.246665 [I 2022-10-28 07:34:47] ...trap/engines/engine.py.224: train: epoch 113 | batch 1510/2304 [I 2022-10-28 07:34:47] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:24 [I 2022-10-28 07:34:47] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00248 [I 2022-10-28 07:34:47] ...trap/engines/engine.py.230: loss: 0.17863 [I 2022-10-28 07:34:47] ...optimizers/trijoint.py.151: total_norm: 16.944698 [I 2022-10-28 07:34:52] ...trap/engines/engine.py.224: train: epoch 113 | batch 1520/2304 [I 2022-10-28 07:34:52] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:17 [I 2022-10-28 07:34:52] ...trap/engines/engine.py.229: process: 0.55369 | load: 0.00246 [I 2022-10-28 07:34:52] ...trap/engines/engine.py.230: loss: 0.09730 [I 2022-10-28 07:34:52] ...optimizers/trijoint.py.151: total_norm: 15.633569 [I 2022-10-28 07:34:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 1530/2304 [I 2022-10-28 07:34:58] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:09 [I 2022-10-28 07:34:58] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00255 [I 2022-10-28 07:34:58] ...trap/engines/engine.py.230: loss: 0.13115 [I 2022-10-28 07:34:58] ...optimizers/trijoint.py.151: total_norm: 25.130407 [I 2022-10-28 07:35:03] ...trap/engines/engine.py.224: train: epoch 113 | batch 1540/2304 [I 2022-10-28 07:35:03] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:07 [I 2022-10-28 07:35:03] ...trap/engines/engine.py.229: process: 0.56646 | load: 0.00249 [I 2022-10-28 07:35:03] ...trap/engines/engine.py.230: loss: 0.09428 [I 2022-10-28 07:35:03] ...optimizers/trijoint.py.151: total_norm: 13.668573 [I 2022-10-28 07:35:09] ...trap/engines/engine.py.224: train: epoch 113 | batch 1550/2304 [I 2022-10-28 07:35:09] ...trap/engines/engine.py.228: elapsed: 0:14:29 | left: 0:07:00 [I 2022-10-28 07:35:09] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00250 [I 2022-10-28 07:35:09] ...trap/engines/engine.py.230: loss: 0.14307 [I 2022-10-28 07:35:09] ...optimizers/trijoint.py.151: total_norm: 10.285045 [I 2022-10-28 07:35:14] ...trap/engines/engine.py.224: train: epoch 113 | batch 1560/2304 [I 2022-10-28 07:35:14] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:55 [I 2022-10-28 07:35:14] ...trap/engines/engine.py.229: process: 0.56472 | load: 0.00254 [I 2022-10-28 07:35:14] ...trap/engines/engine.py.230: loss: 0.12672 [I 2022-10-28 07:35:14] ...optimizers/trijoint.py.151: total_norm: 13.135579 [I 2022-10-28 07:35:20] ...trap/engines/engine.py.224: train: epoch 113 | batch 1570/2304 [I 2022-10-28 07:35:20] ...trap/engines/engine.py.228: elapsed: 0:14:40 | left: 0:06:48 [I 2022-10-28 07:35:20] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00251 [I 2022-10-28 07:35:20] ...trap/engines/engine.py.230: loss: 0.17225 [I 2022-10-28 07:35:20] ...optimizers/trijoint.py.151: total_norm: 29.898426 [I 2022-10-28 07:35:26] ...trap/engines/engine.py.224: train: epoch 113 | batch 1580/2304 [I 2022-10-28 07:35:26] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:41 [I 2022-10-28 07:35:26] ...trap/engines/engine.py.229: process: 0.55244 | load: 0.00262 [I 2022-10-28 07:35:26] ...trap/engines/engine.py.230: loss: 0.10387 [I 2022-10-28 07:35:26] ...optimizers/trijoint.py.151: total_norm: 13.707060 [I 2022-10-28 07:35:31] ...trap/engines/engine.py.224: train: epoch 113 | batch 1590/2304 [I 2022-10-28 07:35:31] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:37 [I 2022-10-28 07:35:31] ...trap/engines/engine.py.229: process: 0.55473 | load: 0.00255 [I 2022-10-28 07:35:31] ...trap/engines/engine.py.230: loss: 0.12161 [I 2022-10-28 07:35:31] ...optimizers/trijoint.py.151: total_norm: 9.852156 [I 2022-10-28 07:35:37] ...trap/engines/engine.py.224: train: epoch 113 | batch 1600/2304 [I 2022-10-28 07:35:37] ...trap/engines/engine.py.228: elapsed: 0:14:57 | left: 0:06:32 [I 2022-10-28 07:35:37] ...trap/engines/engine.py.229: process: 0.56172 | load: 0.00246 [I 2022-10-28 07:35:37] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-28 07:35:37] ...optimizers/trijoint.py.151: total_norm: 9.554454 [I 2022-10-28 07:35:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 1610/2304 [I 2022-10-28 07:35:42] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:28 [I 2022-10-28 07:35:42] ...trap/engines/engine.py.229: process: 0.56308 | load: 0.00255 [I 2022-10-28 07:35:42] ...trap/engines/engine.py.230: loss: 0.19002 [I 2022-10-28 07:35:42] ...optimizers/trijoint.py.151: total_norm: 18.703920 [I 2022-10-28 07:35:48] ...trap/engines/engine.py.224: train: epoch 113 | batch 1620/2304 [I 2022-10-28 07:35:48] ...trap/engines/engine.py.228: elapsed: 0:15:08 | left: 0:06:21 [I 2022-10-28 07:35:48] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00248 [I 2022-10-28 07:35:48] ...trap/engines/engine.py.230: loss: 0.07992 [I 2022-10-28 07:35:48] ...optimizers/trijoint.py.151: total_norm: 13.732437 [I 2022-10-28 07:35:54] ...trap/engines/engine.py.224: train: epoch 113 | batch 1630/2304 [I 2022-10-28 07:35:54] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:14 [I 2022-10-28 07:35:54] ...trap/engines/engine.py.229: process: 0.56117 | load: 0.00250 [I 2022-10-28 07:35:54] ...trap/engines/engine.py.230: loss: 0.10394 [I 2022-10-28 07:35:54] ...optimizers/trijoint.py.151: total_norm: 18.612076 [I 2022-10-28 07:35:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 1640/2304 [I 2022-10-28 07:35:59] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:11 [I 2022-10-28 07:35:59] ...trap/engines/engine.py.229: process: 0.56319 | load: 0.00249 [I 2022-10-28 07:35:59] ...trap/engines/engine.py.230: loss: 0.15206 [I 2022-10-28 07:35:59] ...optimizers/trijoint.py.151: total_norm: 25.328049 [I 2022-10-28 07:36:05] ...trap/engines/engine.py.224: train: epoch 113 | batch 1650/2304 [I 2022-10-28 07:36:05] ...trap/engines/engine.py.228: elapsed: 0:15:25 | left: 0:06:04 [I 2022-10-28 07:36:05] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00250 [I 2022-10-28 07:36:05] ...trap/engines/engine.py.230: loss: 0.14310 [I 2022-10-28 07:36:05] ...optimizers/trijoint.py.151: total_norm: 14.084241 [I 2022-10-28 07:36:10] ...trap/engines/engine.py.224: train: epoch 113 | batch 1660/2304 [I 2022-10-28 07:36:10] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:05:56 [I 2022-10-28 07:36:10] ...trap/engines/engine.py.229: process: 0.54610 | load: 0.00246 [I 2022-10-28 07:36:10] ...trap/engines/engine.py.230: loss: 0.16758 [I 2022-10-28 07:36:10] ...optimizers/trijoint.py.151: total_norm: 47.344223 [I 2022-10-28 07:36:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1670/2304 [I 2022-10-28 07:36:16] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:05:52 [I 2022-10-28 07:36:16] ...trap/engines/engine.py.229: process: 0.55465 | load: 0.00250 [I 2022-10-28 07:36:16] ...trap/engines/engine.py.230: loss: 0.14038 [I 2022-10-28 07:36:16] ...optimizers/trijoint.py.151: total_norm: 14.496678 [I 2022-10-28 07:36:21] ...trap/engines/engine.py.224: train: epoch 113 | batch 1680/2304 [I 2022-10-28 07:36:21] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:47 [I 2022-10-28 07:36:21] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00246 [I 2022-10-28 07:36:21] ...trap/engines/engine.py.230: loss: 0.11151 [I 2022-10-28 07:36:21] ...optimizers/trijoint.py.151: total_norm: 37.387474 [I 2022-10-28 07:36:27] ...trap/engines/engine.py.224: train: epoch 113 | batch 1690/2304 [I 2022-10-28 07:36:27] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:42 [I 2022-10-28 07:36:27] ...trap/engines/engine.py.229: process: 0.55954 | load: 0.00246 [I 2022-10-28 07:36:27] ...trap/engines/engine.py.230: loss: 0.09482 [I 2022-10-28 07:36:27] ...optimizers/trijoint.py.151: total_norm: 13.496540 [I 2022-10-28 07:36:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1700/2304 [I 2022-10-28 07:36:33] ...trap/engines/engine.py.228: elapsed: 0:15:53 | left: 0:05:37 [I 2022-10-28 07:36:33] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00247 [I 2022-10-28 07:36:33] ...trap/engines/engine.py.230: loss: 0.12524 [I 2022-10-28 07:36:33] ...optimizers/trijoint.py.151: total_norm: 29.182131 [I 2022-10-28 07:36:38] ...trap/engines/engine.py.224: train: epoch 113 | batch 1710/2304 [I 2022-10-28 07:36:38] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:30 [I 2022-10-28 07:36:38] ...trap/engines/engine.py.229: process: 0.55459 | load: 0.00243 [I 2022-10-28 07:36:38] ...trap/engines/engine.py.230: loss: 0.10369 [I 2022-10-28 07:36:38] ...optimizers/trijoint.py.151: total_norm: 21.237108 [I 2022-10-28 07:36:44] ...trap/engines/engine.py.224: train: epoch 113 | batch 1720/2304 [I 2022-10-28 07:36:44] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:27 [I 2022-10-28 07:36:44] ...trap/engines/engine.py.229: process: 0.56543 | load: 0.00262 [I 2022-10-28 07:36:44] ...trap/engines/engine.py.230: loss: 0.15025 [I 2022-10-28 07:36:44] ...optimizers/trijoint.py.151: total_norm: 14.512897 [I 2022-10-28 07:36:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 1730/2304 [I 2022-10-28 07:36:50] ...trap/engines/engine.py.228: elapsed: 0:16:10 | left: 0:05:24 [I 2022-10-28 07:36:50] ...trap/engines/engine.py.229: process: 0.55312 | load: 0.00245 [I 2022-10-28 07:36:50] ...trap/engines/engine.py.230: loss: 0.09779 [I 2022-10-28 07:36:50] ...optimizers/trijoint.py.151: total_norm: 16.318657 [I 2022-10-28 07:36:55] ...trap/engines/engine.py.224: train: epoch 113 | batch 1740/2304 [I 2022-10-28 07:36:55] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:16 [I 2022-10-28 07:36:55] ...trap/engines/engine.py.229: process: 0.57102 | load: 0.00243 [I 2022-10-28 07:36:55] ...trap/engines/engine.py.230: loss: 0.07784 [I 2022-10-28 07:36:55] ...optimizers/trijoint.py.151: total_norm: 14.643427 [I 2022-10-28 07:37:19] ...trap/engines/engine.py.224: train: epoch 113 | batch 1750/2304 [I 2022-10-28 07:37:19] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:14:01 [I 2022-10-28 07:37:19] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00237 [I 2022-10-28 07:37:19] ...trap/engines/engine.py.230: loss: 0.14891 [I 2022-10-28 07:37:19] ...optimizers/trijoint.py.151: total_norm: 14.510430 [I 2022-10-28 07:37:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 1760/2304 [I 2022-10-28 07:37:25] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:06:01 [I 2022-10-28 07:37:25] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00248 [I 2022-10-28 07:37:25] ...trap/engines/engine.py.230: loss: 0.17694 [I 2022-10-28 07:37:25] ...optimizers/trijoint.py.151: total_norm: 77.562164 [I 2022-10-28 07:37:30] ...trap/engines/engine.py.224: train: epoch 113 | batch 1770/2304 [I 2022-10-28 07:37:30] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:05:04 [I 2022-10-28 07:37:30] ...trap/engines/engine.py.229: process: 0.54634 | load: 0.00254 [I 2022-10-28 07:37:30] ...trap/engines/engine.py.230: loss: 0.16287 [I 2022-10-28 07:37:30] ...optimizers/trijoint.py.151: total_norm: 26.501423 [I 2022-10-28 07:37:36] ...trap/engines/engine.py.224: train: epoch 113 | batch 1780/2304 [I 2022-10-28 07:37:36] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:50 [I 2022-10-28 07:37:36] ...trap/engines/engine.py.229: process: 0.55142 | load: 0.00256 [I 2022-10-28 07:37:36] ...trap/engines/engine.py.230: loss: 0.08543 [I 2022-10-28 07:37:36] ...optimizers/trijoint.py.151: total_norm: 8.312063 [I 2022-10-28 07:37:41] ...trap/engines/engine.py.224: train: epoch 113 | batch 1790/2304 [I 2022-10-28 07:37:41] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:45 [I 2022-10-28 07:37:41] ...trap/engines/engine.py.229: process: 0.55355 | load: 0.00259 [I 2022-10-28 07:37:41] ...trap/engines/engine.py.230: loss: 0.10187 [I 2022-10-28 07:37:41] ...optimizers/trijoint.py.151: total_norm: 9.851158 [I 2022-10-28 07:37:47] ...trap/engines/engine.py.224: train: epoch 113 | batch 1800/2304 [I 2022-10-28 07:37:47] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:38 [I 2022-10-28 07:37:47] ...trap/engines/engine.py.229: process: 0.54470 | load: 0.00256 [I 2022-10-28 07:37:47] ...trap/engines/engine.py.230: loss: 0.11891 [I 2022-10-28 07:37:47] ...optimizers/trijoint.py.151: total_norm: 23.775318 [I 2022-10-28 07:37:53] ...trap/engines/engine.py.224: train: epoch 113 | batch 1810/2304 [I 2022-10-28 07:37:53] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:35 [I 2022-10-28 07:37:53] ...trap/engines/engine.py.229: process: 0.55798 | load: 0.00257 [I 2022-10-28 07:37:53] ...trap/engines/engine.py.230: loss: 0.15940 [I 2022-10-28 07:37:53] ...optimizers/trijoint.py.151: total_norm: 11.660590 [I 2022-10-28 07:37:58] ...trap/engines/engine.py.224: train: epoch 113 | batch 1820/2304 [I 2022-10-28 07:37:58] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:28 [I 2022-10-28 07:37:58] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00256 [I 2022-10-28 07:37:58] ...trap/engines/engine.py.230: loss: 0.09152 [I 2022-10-28 07:37:58] ...optimizers/trijoint.py.151: total_norm: 11.096648 [I 2022-10-28 07:38:04] ...trap/engines/engine.py.224: train: epoch 113 | batch 1830/2304 [I 2022-10-28 07:38:04] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:23 [I 2022-10-28 07:38:04] ...trap/engines/engine.py.229: process: 0.55937 | load: 0.00255 [I 2022-10-28 07:38:04] ...trap/engines/engine.py.230: loss: 0.12720 [I 2022-10-28 07:38:04] ...optimizers/trijoint.py.151: total_norm: 13.884678 [I 2022-10-28 07:38:09] ...trap/engines/engine.py.224: train: epoch 113 | batch 1840/2304 [I 2022-10-28 07:38:09] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:19 [I 2022-10-28 07:38:09] ...trap/engines/engine.py.229: process: 0.55916 | load: 0.00253 [I 2022-10-28 07:38:09] ...trap/engines/engine.py.230: loss: 0.08753 [I 2022-10-28 07:38:09] ...optimizers/trijoint.py.151: total_norm: 16.026207 [I 2022-10-28 07:38:15] ...trap/engines/engine.py.224: train: epoch 113 | batch 1850/2304 [I 2022-10-28 07:38:15] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:13 [I 2022-10-28 07:38:15] ...trap/engines/engine.py.229: process: 0.55960 | load: 0.00264 [I 2022-10-28 07:38:15] ...trap/engines/engine.py.230: loss: 0.08418 [I 2022-10-28 07:38:15] ...optimizers/trijoint.py.151: total_norm: 10.865138 [I 2022-10-28 07:38:21] ...trap/engines/engine.py.224: train: epoch 113 | batch 1860/2304 [I 2022-10-28 07:38:21] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:04:07 [I 2022-10-28 07:38:21] ...trap/engines/engine.py.229: process: 0.56079 | load: 0.00261 [I 2022-10-28 07:38:21] ...trap/engines/engine.py.230: loss: 0.11305 [I 2022-10-28 07:38:21] ...optimizers/trijoint.py.151: total_norm: 45.469200 [I 2022-10-28 07:38:26] ...trap/engines/engine.py.224: train: epoch 113 | batch 1870/2304 [I 2022-10-28 07:38:26] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:04:01 [I 2022-10-28 07:38:26] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00259 [I 2022-10-28 07:38:26] ...trap/engines/engine.py.230: loss: 0.16279 [I 2022-10-28 07:38:26] ...optimizers/trijoint.py.151: total_norm: 30.657061 [I 2022-10-28 07:38:32] ...trap/engines/engine.py.224: train: epoch 113 | batch 1880/2304 [I 2022-10-28 07:38:32] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:54 [I 2022-10-28 07:38:32] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00262 [I 2022-10-28 07:38:32] ...trap/engines/engine.py.230: loss: 0.15713 [I 2022-10-28 07:38:32] ...optimizers/trijoint.py.151: total_norm: 17.670534 [I 2022-10-28 07:38:37] ...trap/engines/engine.py.224: train: epoch 113 | batch 1890/2304 [I 2022-10-28 07:38:37] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:50 [I 2022-10-28 07:38:37] ...trap/engines/engine.py.229: process: 0.55868 | load: 0.00255 [I 2022-10-28 07:38:37] ...trap/engines/engine.py.230: loss: 0.10326 [I 2022-10-28 07:38:37] ...optimizers/trijoint.py.151: total_norm: 8.887191 [I 2022-10-28 07:38:43] ...trap/engines/engine.py.224: train: epoch 113 | batch 1900/2304 [I 2022-10-28 07:38:43] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:43 [I 2022-10-28 07:38:43] ...trap/engines/engine.py.229: process: 0.54547 | load: 0.00251 [I 2022-10-28 07:38:43] ...trap/engines/engine.py.230: loss: 0.11948 [I 2022-10-28 07:38:43] ...optimizers/trijoint.py.151: total_norm: 12.812639 [I 2022-10-28 07:38:48] ...trap/engines/engine.py.224: train: epoch 113 | batch 1910/2304 [I 2022-10-28 07:38:48] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:38 [I 2022-10-28 07:38:48] ...trap/engines/engine.py.229: process: 0.54816 | load: 0.00254 [I 2022-10-28 07:38:48] ...trap/engines/engine.py.230: loss: 0.11867 [I 2022-10-28 07:38:48] ...optimizers/trijoint.py.151: total_norm: 19.747143 [I 2022-10-28 07:38:54] ...trap/engines/engine.py.224: train: epoch 113 | batch 1920/2304 [I 2022-10-28 07:38:54] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:33 [I 2022-10-28 07:38:54] ...trap/engines/engine.py.229: process: 0.55426 | load: 0.00254 [I 2022-10-28 07:38:54] ...trap/engines/engine.py.230: loss: 0.10937 [I 2022-10-28 07:38:54] ...optimizers/trijoint.py.151: total_norm: 27.528978 [I 2022-10-28 07:39:00] ...trap/engines/engine.py.224: train: epoch 113 | batch 1930/2304 [I 2022-10-28 07:39:00] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:27 [I 2022-10-28 07:39:00] ...trap/engines/engine.py.229: process: 0.54757 | load: 0.00255 [I 2022-10-28 07:39:00] ...trap/engines/engine.py.230: loss: 0.14012 [I 2022-10-28 07:39:00] ...optimizers/trijoint.py.151: total_norm: 12.704995 [I 2022-10-28 07:39:05] ...trap/engines/engine.py.224: train: epoch 113 | batch 1940/2304 [I 2022-10-28 07:39:05] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:30 [I 2022-10-28 07:39:05] ...trap/engines/engine.py.229: process: 0.65518 | load: 0.00249 [I 2022-10-28 07:39:05] ...trap/engines/engine.py.230: loss: 0.09968 [I 2022-10-28 07:39:05] ...optimizers/trijoint.py.151: total_norm: 13.087132 [I 2022-10-28 07:39:11] ...trap/engines/engine.py.224: train: epoch 113 | batch 1950/2304 [I 2022-10-28 07:39:11] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:19 [I 2022-10-28 07:39:11] ...trap/engines/engine.py.229: process: 0.56223 | load: 0.00256 [I 2022-10-28 07:39:11] ...trap/engines/engine.py.230: loss: 0.13075 [I 2022-10-28 07:39:11] ...optimizers/trijoint.py.151: total_norm: 22.891323 [I 2022-10-28 07:39:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 1960/2304 [I 2022-10-28 07:39:16] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:03:11 [I 2022-10-28 07:39:16] ...trap/engines/engine.py.229: process: 0.56168 | load: 0.00258 [I 2022-10-28 07:39:16] ...trap/engines/engine.py.230: loss: 0.08626 [I 2022-10-28 07:39:16] ...optimizers/trijoint.py.151: total_norm: 12.752233 [I 2022-10-28 07:39:22] ...trap/engines/engine.py.224: train: epoch 113 | batch 1970/2304 [I 2022-10-28 07:39:22] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:03:05 [I 2022-10-28 07:39:22] ...trap/engines/engine.py.229: process: 0.54667 | load: 0.00251 [I 2022-10-28 07:39:22] ...trap/engines/engine.py.230: loss: 0.10273 [I 2022-10-28 07:39:22] ...optimizers/trijoint.py.151: total_norm: 10.447464 [I 2022-10-28 07:39:28] ...trap/engines/engine.py.224: train: epoch 113 | batch 1980/2304 [I 2022-10-28 07:39:28] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:03:00 [I 2022-10-28 07:39:28] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00254 [I 2022-10-28 07:39:28] ...trap/engines/engine.py.230: loss: 0.12977 [I 2022-10-28 07:39:28] ...optimizers/trijoint.py.151: total_norm: 14.722861 [I 2022-10-28 07:39:33] ...trap/engines/engine.py.224: train: epoch 113 | batch 1990/2304 [I 2022-10-28 07:39:33] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:54 [I 2022-10-28 07:39:33] ...trap/engines/engine.py.229: process: 0.54653 | load: 0.00247 [I 2022-10-28 07:39:33] ...trap/engines/engine.py.230: loss: 0.09843 [I 2022-10-28 07:39:33] ...optimizers/trijoint.py.151: total_norm: 10.350357 [I 2022-10-28 07:39:39] ...trap/engines/engine.py.224: train: epoch 113 | batch 2000/2304 [I 2022-10-28 07:39:39] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:47 [I 2022-10-28 07:39:39] ...trap/engines/engine.py.229: process: 0.55108 | load: 0.00255 [I 2022-10-28 07:39:39] ...trap/engines/engine.py.230: loss: 0.16533 [I 2022-10-28 07:39:39] ...optimizers/trijoint.py.151: total_norm: 11.991785 [I 2022-10-28 07:39:44] ...trap/engines/engine.py.224: train: epoch 113 | batch 2010/2304 [I 2022-10-28 07:39:44] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:44 [I 2022-10-28 07:39:44] ...trap/engines/engine.py.229: process: 0.56234 | load: 0.00260 [I 2022-10-28 07:39:44] ...trap/engines/engine.py.230: loss: 0.14034 [I 2022-10-28 07:39:44] ...optimizers/trijoint.py.151: total_norm: 18.751225 [I 2022-10-28 07:39:50] ...trap/engines/engine.py.224: train: epoch 113 | batch 2020/2304 [I 2022-10-28 07:39:50] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:38 [I 2022-10-28 07:39:50] ...trap/engines/engine.py.229: process: 0.56501 | load: 0.00259 [I 2022-10-28 07:39:50] ...trap/engines/engine.py.230: loss: 0.10109 [I 2022-10-28 07:39:50] ...optimizers/trijoint.py.151: total_norm: 15.286066 [I 2022-10-28 07:39:56] ...trap/engines/engine.py.224: train: epoch 113 | batch 2030/2304 [I 2022-10-28 07:39:56] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:33 [I 2022-10-28 07:39:56] ...trap/engines/engine.py.229: process: 0.56176 | load: 0.00264 [I 2022-10-28 07:39:56] ...trap/engines/engine.py.230: loss: 0.10073 [I 2022-10-28 07:39:56] ...optimizers/trijoint.py.151: total_norm: 14.068932 [I 2022-10-28 07:40:01] ...trap/engines/engine.py.224: train: epoch 113 | batch 2040/2304 [I 2022-10-28 07:40:01] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:26 [I 2022-10-28 07:40:01] ...trap/engines/engine.py.229: process: 0.57276 | load: 0.00264 [I 2022-10-28 07:40:01] ...trap/engines/engine.py.230: loss: 0.11979 [I 2022-10-28 07:40:01] ...optimizers/trijoint.py.151: total_norm: 13.297604 [I 2022-10-28 07:40:07] ...trap/engines/engine.py.224: train: epoch 113 | batch 2050/2304 [I 2022-10-28 07:40:07] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:21 [I 2022-10-28 07:40:07] ...trap/engines/engine.py.229: process: 0.55495 | load: 0.00265 [I 2022-10-28 07:40:07] ...trap/engines/engine.py.230: loss: 0.14796 [I 2022-10-28 07:40:07] ...optimizers/trijoint.py.151: total_norm: 19.867403 [I 2022-10-28 07:40:12] ...trap/engines/engine.py.224: train: epoch 113 | batch 2060/2304 [I 2022-10-28 07:40:12] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:02:19 [I 2022-10-28 07:40:12] ...trap/engines/engine.py.229: process: 0.55643 | load: 0.00256 [I 2022-10-28 07:40:12] ...trap/engines/engine.py.230: loss: 0.12453 [I 2022-10-28 07:40:12] ...optimizers/trijoint.py.151: total_norm: 14.865386 [I 2022-10-28 07:40:18] ...trap/engines/engine.py.224: train: epoch 113 | batch 2070/2304 [I 2022-10-28 07:40:18] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:10 [I 2022-10-28 07:40:18] ...trap/engines/engine.py.229: process: 0.55600 | load: 0.00251 [I 2022-10-28 07:40:18] ...trap/engines/engine.py.230: loss: 0.09767 [I 2022-10-28 07:40:18] ...optimizers/trijoint.py.151: total_norm: 9.360978 [I 2022-10-28 07:40:24] ...trap/engines/engine.py.224: train: epoch 113 | batch 2080/2304 [I 2022-10-28 07:40:24] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:02:05 [I 2022-10-28 07:40:24] ...trap/engines/engine.py.229: process: 0.57034 | load: 0.00254 [I 2022-10-28 07:40:24] ...trap/engines/engine.py.230: loss: 0.08685 [I 2022-10-28 07:40:24] ...optimizers/trijoint.py.151: total_norm: 8.410739 [I 2022-10-28 07:40:29] ...trap/engines/engine.py.224: train: epoch 113 | batch 2090/2304 [I 2022-10-28 07:40:29] ...trap/engines/engine.py.228: elapsed: 0:19:50 | left: 0:01:57 [I 2022-10-28 07:40:29] ...trap/engines/engine.py.229: process: 0.53898 | load: 0.00259 [I 2022-10-28 07:40:29] ...trap/engines/engine.py.230: loss: 0.10071 [I 2022-10-28 07:40:29] ...optimizers/trijoint.py.151: total_norm: 11.831758 [I 2022-10-28 07:40:35] ...trap/engines/engine.py.224: train: epoch 113 | batch 2100/2304 [I 2022-10-28 07:40:35] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:53 [I 2022-10-28 07:40:35] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00255 [I 2022-10-28 07:40:35] ...trap/engines/engine.py.230: loss: 0.09751 [I 2022-10-28 07:40:35] ...optimizers/trijoint.py.151: total_norm: 15.398627 [I 2022-10-28 07:40:40] ...trap/engines/engine.py.224: train: epoch 113 | batch 2110/2304 [I 2022-10-28 07:40:40] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:47 [I 2022-10-28 07:40:40] ...trap/engines/engine.py.229: process: 0.55729 | load: 0.00259 [I 2022-10-28 07:40:40] ...trap/engines/engine.py.230: loss: 0.15814 [I 2022-10-28 07:40:40] ...optimizers/trijoint.py.151: total_norm: 11.368020 [I 2022-10-28 07:40:46] ...trap/engines/engine.py.224: train: epoch 113 | batch 2120/2304 [I 2022-10-28 07:40:46] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:43 [I 2022-10-28 07:40:46] ...trap/engines/engine.py.229: process: 0.56201 | load: 0.00261 [I 2022-10-28 07:40:46] ...trap/engines/engine.py.230: loss: 0.09365 [I 2022-10-28 07:40:46] ...optimizers/trijoint.py.151: total_norm: 10.350303 [I 2022-10-28 07:40:52] ...trap/engines/engine.py.224: train: epoch 113 | batch 2130/2304 [I 2022-10-28 07:40:52] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:36 [I 2022-10-28 07:40:52] ...trap/engines/engine.py.229: process: 0.55386 | load: 0.00258 [I 2022-10-28 07:40:52] ...trap/engines/engine.py.230: loss: 0.09229 [I 2022-10-28 07:40:52] ...optimizers/trijoint.py.151: total_norm: 10.784379 [I 2022-10-28 07:40:57] ...trap/engines/engine.py.224: train: epoch 113 | batch 2140/2304 [I 2022-10-28 07:40:57] ...trap/engines/engine.py.228: elapsed: 0:20:18 | left: 0:01:31 [I 2022-10-28 07:40:57] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00258 [I 2022-10-28 07:40:57] ...trap/engines/engine.py.230: loss: 0.11803 [I 2022-10-28 07:40:57] ...optimizers/trijoint.py.151: total_norm: 13.618625 [I 2022-10-28 07:41:03] ...trap/engines/engine.py.224: train: epoch 113 | batch 2150/2304 [I 2022-10-28 07:41:03] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:25 [I 2022-10-28 07:41:03] ...trap/engines/engine.py.229: process: 0.54833 | load: 0.00255 [I 2022-10-28 07:41:03] ...trap/engines/engine.py.230: loss: 0.17618 [I 2022-10-28 07:41:03] ...optimizers/trijoint.py.151: total_norm: 17.146006 [I 2022-10-28 07:41:08] ...trap/engines/engine.py.224: train: epoch 113 | batch 2160/2304 [I 2022-10-28 07:41:08] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:20 [I 2022-10-28 07:41:08] ...trap/engines/engine.py.229: process: 0.55675 | load: 0.00255 [I 2022-10-28 07:41:08] ...trap/engines/engine.py.230: loss: 0.09192 [I 2022-10-28 07:41:08] ...optimizers/trijoint.py.151: total_norm: 8.685318 [I 2022-10-28 07:41:14] ...trap/engines/engine.py.224: train: epoch 113 | batch 2170/2304 [I 2022-10-28 07:41:14] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:01:15 [I 2022-10-28 07:41:14] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00253 [I 2022-10-28 07:41:14] ...trap/engines/engine.py.230: loss: 0.07660 [I 2022-10-28 07:41:14] ...optimizers/trijoint.py.151: total_norm: 13.307949 [I 2022-10-28 07:41:19] ...trap/engines/engine.py.224: train: epoch 113 | batch 2180/2304 [I 2022-10-28 07:41:19] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:01:08 [I 2022-10-28 07:41:19] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00273 [I 2022-10-28 07:41:19] ...trap/engines/engine.py.230: loss: 0.17002 [I 2022-10-28 07:41:19] ...optimizers/trijoint.py.151: total_norm: 14.476198 [I 2022-10-28 07:41:25] ...trap/engines/engine.py.224: train: epoch 113 | batch 2190/2304 [I 2022-10-28 07:41:25] ...trap/engines/engine.py.228: elapsed: 0:20:46 | left: 0:01:04 [I 2022-10-28 07:41:25] ...trap/engines/engine.py.229: process: 0.56880 | load: 0.00257 [I 2022-10-28 07:41:25] ...trap/engines/engine.py.230: loss: 0.11138 [I 2022-10-28 07:41:25] ...optimizers/trijoint.py.151: total_norm: 18.118540 [I 2022-10-28 07:41:31] ...trap/engines/engine.py.224: train: epoch 113 | batch 2200/2304 [I 2022-10-28 07:41:31] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:57 [I 2022-10-28 07:41:31] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00251 [I 2022-10-28 07:41:31] ...trap/engines/engine.py.230: loss: 0.08416 [I 2022-10-28 07:41:31] ...optimizers/trijoint.py.151: total_norm: 11.261073 [I 2022-10-28 07:41:36] ...trap/engines/engine.py.224: train: epoch 113 | batch 2210/2304 [I 2022-10-28 07:41:36] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:52 [I 2022-10-28 07:41:36] ...trap/engines/engine.py.229: process: 0.55090 | load: 0.00262 [I 2022-10-28 07:41:36] ...trap/engines/engine.py.230: loss: 0.07861 [I 2022-10-28 07:41:36] ...optimizers/trijoint.py.151: total_norm: 11.804989 [I 2022-10-28 07:41:42] ...trap/engines/engine.py.224: train: epoch 113 | batch 2220/2304 [I 2022-10-28 07:41:42] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:46 [I 2022-10-28 07:41:42] ...trap/engines/engine.py.229: process: 0.56259 | load: 0.00259 [I 2022-10-28 07:41:42] ...trap/engines/engine.py.230: loss: 0.19884 [I 2022-10-28 07:41:42] ...optimizers/trijoint.py.151: total_norm: 19.691628 [I 2022-10-28 07:41:48] ...trap/engines/engine.py.224: train: epoch 113 | batch 2230/2304 [I 2022-10-28 07:41:48] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:43 [I 2022-10-28 07:41:48] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00258 [I 2022-10-28 07:41:48] ...trap/engines/engine.py.230: loss: 0.10432 [I 2022-10-28 07:41:48] ...optimizers/trijoint.py.151: total_norm: 9.506444 [I 2022-10-28 07:41:53] ...trap/engines/engine.py.224: train: epoch 113 | batch 2240/2304 [I 2022-10-28 07:41:53] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:35 [I 2022-10-28 07:41:53] ...trap/engines/engine.py.229: process: 0.55455 | load: 0.00254 [I 2022-10-28 07:41:53] ...trap/engines/engine.py.230: loss: 0.08905 [I 2022-10-28 07:41:53] ...optimizers/trijoint.py.151: total_norm: 9.638044 [I 2022-10-28 07:41:59] ...trap/engines/engine.py.224: train: epoch 113 | batch 2250/2304 [I 2022-10-28 07:41:59] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:30 [I 2022-10-28 07:41:59] ...trap/engines/engine.py.229: process: 0.56238 | load: 0.00257 [I 2022-10-28 07:41:59] ...trap/engines/engine.py.230: loss: 0.12524 [I 2022-10-28 07:41:59] ...optimizers/trijoint.py.151: total_norm: 9.725997 [I 2022-10-28 07:42:04] ...trap/engines/engine.py.224: train: epoch 113 | batch 2260/2304 [I 2022-10-28 07:42:04] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:24 [I 2022-10-28 07:42:04] ...trap/engines/engine.py.229: process: 0.55733 | load: 0.00255 [I 2022-10-28 07:42:04] ...trap/engines/engine.py.230: loss: 0.12002 [I 2022-10-28 07:42:04] ...optimizers/trijoint.py.151: total_norm: 14.757957 [I 2022-10-28 07:42:10] ...trap/engines/engine.py.224: train: epoch 113 | batch 2270/2304 [I 2022-10-28 07:42:10] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:18 [I 2022-10-28 07:42:10] ...trap/engines/engine.py.229: process: 0.56173 | load: 0.00259 [I 2022-10-28 07:42:10] ...trap/engines/engine.py.230: loss: 0.12590 [I 2022-10-28 07:42:10] ...optimizers/trijoint.py.151: total_norm: 29.196350 [I 2022-10-28 07:42:16] ...trap/engines/engine.py.224: train: epoch 113 | batch 2280/2304 [I 2022-10-28 07:42:16] ...trap/engines/engine.py.228: elapsed: 0:21:36 | left: 0:00:13 [I 2022-10-28 07:42:16] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00260 [I 2022-10-28 07:42:16] ...trap/engines/engine.py.230: loss: 0.08326 [I 2022-10-28 07:42:16] ...optimizers/trijoint.py.151: total_norm: 12.555625 [I 2022-10-28 07:42:21] ...trap/engines/engine.py.224: train: epoch 113 | batch 2290/2304 [I 2022-10-28 07:42:21] ...trap/engines/engine.py.228: elapsed: 0:21:42 | left: 0:00:07 [I 2022-10-28 07:42:21] ...trap/engines/engine.py.229: process: 0.56656 | load: 0.00258 [I 2022-10-28 07:42:21] ...trap/engines/engine.py.230: loss: 0.08371 [I 2022-10-28 07:42:21] ...optimizers/trijoint.py.151: total_norm: 10.851126 [I 2022-10-28 07:42:27] ...trap/engines/engine.py.224: train: epoch 113 | batch 2300/2304 [I 2022-10-28 07:42:27] ...trap/engines/engine.py.228: elapsed: 0:21:47 | left: 0:00:02 [I 2022-10-28 07:42:27] ...trap/engines/engine.py.229: process: 0.56042 | load: 0.00015 [I 2022-10-28 07:42:27] ...trap/engines/engine.py.230: loss: 0.12991 [I 2022-10-28 07:42:27] ...optimizers/trijoint.py.151: total_norm: 52.154449 [I 2022-10-28 07:42:29] ...trap/engines/engine.py.224: train: epoch 113 | batch 2304/2304 [I 2022-10-28 07:42:29] ...trap/engines/engine.py.228: elapsed: 0:21:50 | left: 0:00:00 [I 2022-10-28 07:42:29] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00016 [I 2022-10-28 07:42:29] ...trap/engines/engine.py.230: loss: 0.14331 [I 2022-10-28 07:42:29] ...optimizers/trijoint.py.151: total_norm: 19.837730 [S 2022-10-28 07:42:30] ...trap/engines/engine.py.129: train_epoch.epoch: 113 [S 2022-10-28 07:42:30] ...trap/engines/engine.py.129: train_epoch.loss: 0.12099876436940243 [I 2022-10-28 07:42:30] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 07:42:32] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 113 [I 2022-10-28 07:42:32] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 07:42:34] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 07:42:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 07:42:34] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 07:42:38] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 07:42:38] ...trap/engines/engine.py.321: eval: epoch 113 | batch 0/510 [I 2022-10-28 07:42:38] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:52:51 [I 2022-10-28 07:42:38] ...trap/engines/engine.py.326: process: 6.21933 | load: 4.00905 [I 2022-10-28 07:42:41] ...trap/engines/engine.py.321: eval: epoch 113 | batch 10/510 [I 2022-10-28 07:42:41] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:01 [I 2022-10-28 07:42:41] ...trap/engines/engine.py.326: process: 0.33542 | load: 0.00017 [I 2022-10-28 07:42:45] ...trap/engines/engine.py.321: eval: epoch 113 | batch 20/510 [I 2022-10-28 07:42:45] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:16 [I 2022-10-28 07:42:45] ...trap/engines/engine.py.326: process: 0.32938 | load: 0.00017 [I 2022-10-28 07:42:48] ...trap/engines/engine.py.321: eval: epoch 113 | batch 30/510 [I 2022-10-28 07:42:48] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:42 [I 2022-10-28 07:42:48] ...trap/engines/engine.py.326: process: 0.33711 | load: 0.00017 [I 2022-10-28 07:42:51] ...trap/engines/engine.py.321: eval: epoch 113 | batch 40/510 [I 2022-10-28 07:42:51] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:38 [I 2022-10-28 07:42:51] ...trap/engines/engine.py.326: process: 0.34193 | load: 0.00031 [I 2022-10-28 07:42:55] ...trap/engines/engine.py.321: eval: epoch 113 | batch 50/510 [I 2022-10-28 07:42:55] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:38 [I 2022-10-28 07:42:55] ...trap/engines/engine.py.326: process: 0.36999 | load: 0.00026 [I 2022-10-28 07:42:58] ...trap/engines/engine.py.321: eval: epoch 113 | batch 60/510 [I 2022-10-28 07:42:58] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:30 [I 2022-10-28 07:42:58] ...trap/engines/engine.py.326: process: 0.32865 | load: 0.00021 [I 2022-10-28 07:43:02] ...trap/engines/engine.py.321: eval: epoch 113 | batch 70/510 [I 2022-10-28 07:43:02] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:29 [I 2022-10-28 07:43:02] ...trap/engines/engine.py.326: process: 0.32238 | load: 0.00018 [I 2022-10-28 07:43:05] ...trap/engines/engine.py.321: eval: epoch 113 | batch 80/510 [I 2022-10-28 07:43:05] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:27 [I 2022-10-28 07:43:05] ...trap/engines/engine.py.326: process: 0.35447 | load: 0.00024 [I 2022-10-28 07:43:08] ...trap/engines/engine.py.321: eval: epoch 113 | batch 90/510 [I 2022-10-28 07:43:08] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:19 [I 2022-10-28 07:43:08] ...trap/engines/engine.py.326: process: 0.33004 | load: 0.00019 [I 2022-10-28 07:43:12] ...trap/engines/engine.py.321: eval: epoch 113 | batch 100/510 [I 2022-10-28 07:43:12] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-28 07:43:12] ...trap/engines/engine.py.326: process: 0.29133 | load: 0.00021 [I 2022-10-28 07:43:15] ...trap/engines/engine.py.321: eval: epoch 113 | batch 110/510 [I 2022-10-28 07:43:15] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:57 [I 2022-10-28 07:43:15] ...trap/engines/engine.py.326: process: 0.28338 | load: 0.00019 [I 2022-10-28 07:43:17] ...trap/engines/engine.py.321: eval: epoch 113 | batch 120/510 [I 2022-10-28 07:43:17] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:53 [I 2022-10-28 07:43:17] ...trap/engines/engine.py.326: process: 0.28922 | load: 0.00016 [I 2022-10-28 07:43:20] ...trap/engines/engine.py.321: eval: epoch 113 | batch 130/510 [I 2022-10-28 07:43:20] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:50 [I 2022-10-28 07:43:20] ...trap/engines/engine.py.326: process: 0.28964 | load: 0.00016 [I 2022-10-28 07:43:23] ...trap/engines/engine.py.321: eval: epoch 113 | batch 140/510 [I 2022-10-28 07:43:23] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:47 [I 2022-10-28 07:43:23] ...trap/engines/engine.py.326: process: 0.29196 | load: 0.00016 [I 2022-10-28 07:43:26] ...trap/engines/engine.py.321: eval: epoch 113 | batch 150/510 [I 2022-10-28 07:43:26] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:44 [I 2022-10-28 07:43:26] ...trap/engines/engine.py.326: process: 0.29675 | load: 0.00017 [I 2022-10-28 07:43:29] ...trap/engines/engine.py.321: eval: epoch 113 | batch 160/510 [I 2022-10-28 07:43:29] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:42 [I 2022-10-28 07:43:29] ...trap/engines/engine.py.326: process: 0.29378 | load: 0.00019 [I 2022-10-28 07:43:32] ...trap/engines/engine.py.321: eval: epoch 113 | batch 170/510 [I 2022-10-28 07:43:32] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:39 [I 2022-10-28 07:43:32] ...trap/engines/engine.py.326: process: 0.28764 | load: 0.00042 [I 2022-10-28 07:43:35] ...trap/engines/engine.py.321: eval: epoch 113 | batch 180/510 [I 2022-10-28 07:43:35] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:37 [I 2022-10-28 07:43:35] ...trap/engines/engine.py.326: process: 0.30989 | load: 0.00017 [I 2022-10-28 07:43:38] ...trap/engines/engine.py.321: eval: epoch 113 | batch 190/510 [I 2022-10-28 07:43:38] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:33 [I 2022-10-28 07:43:38] ...trap/engines/engine.py.326: process: 0.28966 | load: 0.00016 [I 2022-10-28 07:43:41] ...trap/engines/engine.py.321: eval: epoch 113 | batch 200/510 [I 2022-10-28 07:43:41] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:30 [I 2022-10-28 07:43:41] ...trap/engines/engine.py.326: process: 0.29111 | load: 0.00019 [I 2022-10-28 07:43:44] ...trap/engines/engine.py.321: eval: epoch 113 | batch 210/510 [I 2022-10-28 07:43:44] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 07:43:44] ...trap/engines/engine.py.326: process: 0.28634 | load: 0.00017 [I 2022-10-28 07:43:47] ...trap/engines/engine.py.321: eval: epoch 113 | batch 220/510 [I 2022-10-28 07:43:47] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:24 [I 2022-10-28 07:43:47] ...trap/engines/engine.py.326: process: 0.29113 | load: 0.00014 [I 2022-10-28 07:43:50] ...trap/engines/engine.py.321: eval: epoch 113 | batch 230/510 [I 2022-10-28 07:43:50] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:20 [I 2022-10-28 07:43:50] ...trap/engines/engine.py.326: process: 0.28614 | load: 0.00016 [I 2022-10-28 07:43:52] ...trap/engines/engine.py.321: eval: epoch 113 | batch 240/510 [I 2022-10-28 07:43:52] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:17 [I 2022-10-28 07:43:52] ...trap/engines/engine.py.326: process: 0.28223 | load: 0.00019 [I 2022-10-28 07:43:55] ...trap/engines/engine.py.321: eval: epoch 113 | batch 250/510 [I 2022-10-28 07:43:55] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:14 [I 2022-10-28 07:43:55] ...trap/engines/engine.py.326: process: 0.28122 | load: 0.00019 [I 2022-10-28 07:43:58] ...trap/engines/engine.py.321: eval: epoch 113 | batch 260/510 [I 2022-10-28 07:43:58] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:12 [I 2022-10-28 07:43:58] ...trap/engines/engine.py.326: process: 0.28921 | load: 0.00030 [I 2022-10-28 07:44:01] ...trap/engines/engine.py.321: eval: epoch 113 | batch 270/510 [I 2022-10-28 07:44:01] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:09 [I 2022-10-28 07:44:01] ...trap/engines/engine.py.326: process: 0.28662 | load: 0.00020 [I 2022-10-28 07:44:04] ...trap/engines/engine.py.321: eval: epoch 113 | batch 280/510 [I 2022-10-28 07:44:04] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:06 [I 2022-10-28 07:44:04] ...trap/engines/engine.py.326: process: 0.28856 | load: 0.00016 [I 2022-10-28 07:44:07] ...trap/engines/engine.py.321: eval: epoch 113 | batch 290/510 [I 2022-10-28 07:44:07] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 07:44:07] ...trap/engines/engine.py.326: process: 0.28091 | load: 0.00017 [I 2022-10-28 07:44:10] ...trap/engines/engine.py.321: eval: epoch 113 | batch 300/510 [I 2022-10-28 07:44:10] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-28 07:44:10] ...trap/engines/engine.py.326: process: 0.29117 | load: 0.00017 [I 2022-10-28 07:44:13] ...trap/engines/engine.py.321: eval: epoch 113 | batch 310/510 [I 2022-10-28 07:44:13] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:57 [I 2022-10-28 07:44:13] ...trap/engines/engine.py.326: process: 0.28815 | load: 0.00024 [I 2022-10-28 07:44:16] ...trap/engines/engine.py.321: eval: epoch 113 | batch 320/510 [I 2022-10-28 07:44:16] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:54 [I 2022-10-28 07:44:16] ...trap/engines/engine.py.326: process: 0.29010 | load: 0.00016 [I 2022-10-28 07:44:18] ...trap/engines/engine.py.321: eval: epoch 113 | batch 330/510 [I 2022-10-28 07:44:18] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:52 [I 2022-10-28 07:44:18] ...trap/engines/engine.py.326: process: 0.28720 | load: 0.00017 [I 2022-10-28 07:44:21] ...trap/engines/engine.py.321: eval: epoch 113 | batch 340/510 [I 2022-10-28 07:44:21] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:49 [I 2022-10-28 07:44:21] ...trap/engines/engine.py.326: process: 0.28607 | load: 0.00019 [I 2022-10-28 07:44:24] ...trap/engines/engine.py.321: eval: epoch 113 | batch 350/510 [I 2022-10-28 07:44:24] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:45 [I 2022-10-28 07:44:24] ...trap/engines/engine.py.326: process: 0.28290 | load: 0.00026 [I 2022-10-28 07:44:27] ...trap/engines/engine.py.321: eval: epoch 113 | batch 360/510 [I 2022-10-28 07:44:27] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:43 [I 2022-10-28 07:44:27] ...trap/engines/engine.py.326: process: 0.28899 | load: 0.00021 [I 2022-10-28 07:44:30] ...trap/engines/engine.py.321: eval: epoch 113 | batch 370/510 [I 2022-10-28 07:44:30] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:40 [I 2022-10-28 07:44:30] ...trap/engines/engine.py.326: process: 0.28613 | load: 0.00022 [I 2022-10-28 07:44:33] ...trap/engines/engine.py.321: eval: epoch 113 | batch 380/510 [I 2022-10-28 07:44:33] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 07:44:33] ...trap/engines/engine.py.326: process: 0.28962 | load: 0.00017 [I 2022-10-28 07:44:36] ...trap/engines/engine.py.321: eval: epoch 113 | batch 390/510 [I 2022-10-28 07:44:36] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:35 [I 2022-10-28 07:44:36] ...trap/engines/engine.py.326: process: 0.29999 | load: 0.00017 [I 2022-10-28 07:44:39] ...trap/engines/engine.py.321: eval: epoch 113 | batch 400/510 [I 2022-10-28 07:44:39] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:31 [I 2022-10-28 07:44:39] ...trap/engines/engine.py.326: process: 0.28915 | load: 0.00016 [I 2022-10-28 07:44:42] ...trap/engines/engine.py.321: eval: epoch 113 | batch 410/510 [I 2022-10-28 07:44:42] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:28 [I 2022-10-28 07:44:42] ...trap/engines/engine.py.326: process: 0.28315 | load: 0.00015 [I 2022-10-28 07:44:45] ...trap/engines/engine.py.321: eval: epoch 113 | batch 420/510 [I 2022-10-28 07:44:45] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:26 [I 2022-10-28 07:44:45] ...trap/engines/engine.py.326: process: 0.29221 | load: 0.00015 [I 2022-10-28 07:44:47] ...trap/engines/engine.py.321: eval: epoch 113 | batch 430/510 [I 2022-10-28 07:44:47] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:23 [I 2022-10-28 07:44:47] ...trap/engines/engine.py.326: process: 0.29090 | load: 0.00016 [I 2022-10-28 07:44:50] ...trap/engines/engine.py.321: eval: epoch 113 | batch 440/510 [I 2022-10-28 07:44:50] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:20 [I 2022-10-28 07:44:50] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00018 [I 2022-10-28 07:44:53] ...trap/engines/engine.py.321: eval: epoch 113 | batch 450/510 [I 2022-10-28 07:44:53] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:17 [I 2022-10-28 07:44:53] ...trap/engines/engine.py.326: process: 0.28956 | load: 0.00017 [I 2022-10-28 07:44:56] ...trap/engines/engine.py.321: eval: epoch 113 | batch 460/510 [I 2022-10-28 07:44:56] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:14 [I 2022-10-28 07:44:56] ...trap/engines/engine.py.326: process: 0.29284 | load: 0.00019 [I 2022-10-28 07:44:59] ...trap/engines/engine.py.321: eval: epoch 113 | batch 470/510 [I 2022-10-28 07:44:59] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:11 [I 2022-10-28 07:44:59] ...trap/engines/engine.py.326: process: 0.28612 | load: 0.00018 [I 2022-10-28 07:45:02] ...trap/engines/engine.py.321: eval: epoch 113 | batch 480/510 [I 2022-10-28 07:45:02] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 07:45:02] ...trap/engines/engine.py.326: process: 0.28714 | load: 0.00020 [I 2022-10-28 07:45:05] ...trap/engines/engine.py.321: eval: epoch 113 | batch 490/510 [I 2022-10-28 07:45:05] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 07:45:05] ...trap/engines/engine.py.326: process: 0.28547 | load: 0.00016 [I 2022-10-28 07:45:08] ...trap/engines/engine.py.321: eval: epoch 113 | batch 500/510 [I 2022-10-28 07:45:08] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:02 [I 2022-10-28 07:45:08] ...trap/engines/engine.py.326: process: 0.28763 | load: 0.00017 [I 2022-10-28 07:45:11] ...trap/engines/engine.py.321: eval: epoch 113 | batch 510/510 [I 2022-10-28 07:45:11] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:00 [I 2022-10-28 07:45:11] ...trap/engines/engine.py.326: process: 0.28514 | load: 0.00011 [S 2022-10-28 07:45:11] ...trap/engines/engine.py.132: eval_epoch.epoch: 113 [I 2022-10-28 07:45:11] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.738300 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.908500 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933300 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.734300 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906700 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935200 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010189 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010481 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008307 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009644 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010120 [S 2022-10-28 07:45:15] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008158 [I 2022-10-28 07:45:17] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 07:45:17] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 07:45:18] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 07:45:18] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 07:45:18] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 07:45:23] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 07:45:24] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 07:45:30] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 07:45:30] ...trap/engines/engine.py.163: Training model on trainset for epoch 114 [I 2022-10-28 07:45:30] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 07:45:33] ...trap/engines/engine.py.224: train: epoch 114 | batch 0/2304 [I 2022-10-28 07:45:33] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:44 [I 2022-10-28 07:45:33] ...trap/engines/engine.py.229: process: 2.80563 | load: 2.12657 [I 2022-10-28 07:45:33] ...trap/engines/engine.py.230: loss: 0.09274 [I 2022-10-28 07:45:33] ...optimizers/trijoint.py.151: total_norm: 13.146793 [I 2022-10-28 07:45:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 10/2304 [I 2022-10-28 07:45:38] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:41 [I 2022-10-28 07:45:38] ...trap/engines/engine.py.229: process: 0.57145 | load: 0.00456 [I 2022-10-28 07:45:38] ...trap/engines/engine.py.230: loss: 0.12587 [I 2022-10-28 07:45:38] ...optimizers/trijoint.py.151: total_norm: 46.222336 [I 2022-10-28 07:45:44] ...trap/engines/engine.py.224: train: epoch 114 | batch 20/2304 [I 2022-10-28 07:45:44] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:25 [I 2022-10-28 07:45:44] ...trap/engines/engine.py.229: process: 0.56724 | load: 0.00337 [I 2022-10-28 07:45:44] ...trap/engines/engine.py.230: loss: 0.09137 [I 2022-10-28 07:45:44] ...optimizers/trijoint.py.151: total_norm: 9.859414 [I 2022-10-28 07:45:50] ...trap/engines/engine.py.224: train: epoch 114 | batch 30/2304 [I 2022-10-28 07:45:50] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:19 [I 2022-10-28 07:45:50] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00319 [I 2022-10-28 07:45:50] ...trap/engines/engine.py.230: loss: 0.08798 [I 2022-10-28 07:45:50] ...optimizers/trijoint.py.151: total_norm: 13.380569 [I 2022-10-28 07:45:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 40/2304 [I 2022-10-28 07:45:55] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:14 [I 2022-10-28 07:45:55] ...trap/engines/engine.py.229: process: 0.56640 | load: 0.00387 [I 2022-10-28 07:45:55] ...trap/engines/engine.py.230: loss: 0.08362 [I 2022-10-28 07:45:55] ...optimizers/trijoint.py.151: total_norm: 11.138088 [I 2022-10-28 07:46:01] ...trap/engines/engine.py.224: train: epoch 114 | batch 50/2304 [I 2022-10-28 07:46:01] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:57 [I 2022-10-28 07:46:01] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00292 [I 2022-10-28 07:46:01] ...trap/engines/engine.py.230: loss: 0.14368 [I 2022-10-28 07:46:01] ...optimizers/trijoint.py.151: total_norm: 13.957204 [I 2022-10-28 07:46:06] ...trap/engines/engine.py.224: train: epoch 114 | batch 60/2304 [I 2022-10-28 07:46:06] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:50 [I 2022-10-28 07:46:06] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00278 [I 2022-10-28 07:46:06] ...trap/engines/engine.py.230: loss: 0.12310 [I 2022-10-28 07:46:06] ...optimizers/trijoint.py.151: total_norm: 15.406708 [I 2022-10-28 07:46:12] ...trap/engines/engine.py.224: train: epoch 114 | batch 70/2304 [I 2022-10-28 07:46:12] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:46 [I 2022-10-28 07:46:12] ...trap/engines/engine.py.229: process: 0.56179 | load: 0.00278 [I 2022-10-28 07:46:12] ...trap/engines/engine.py.230: loss: 0.14471 [I 2022-10-28 07:46:12] ...optimizers/trijoint.py.151: total_norm: 12.062358 [I 2022-10-28 07:46:18] ...trap/engines/engine.py.224: train: epoch 114 | batch 80/2304 [I 2022-10-28 07:46:18] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:44 [I 2022-10-28 07:46:18] ...trap/engines/engine.py.229: process: 0.54791 | load: 0.00308 [I 2022-10-28 07:46:18] ...trap/engines/engine.py.230: loss: 0.10930 [I 2022-10-28 07:46:18] ...optimizers/trijoint.py.151: total_norm: 46.900230 [I 2022-10-28 07:46:23] ...trap/engines/engine.py.224: train: epoch 114 | batch 90/2304 [I 2022-10-28 07:46:23] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:33 [I 2022-10-28 07:46:23] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00272 [I 2022-10-28 07:46:23] ...trap/engines/engine.py.230: loss: 0.17044 [I 2022-10-28 07:46:23] ...optimizers/trijoint.py.151: total_norm: 50.194153 [I 2022-10-28 07:46:29] ...trap/engines/engine.py.224: train: epoch 114 | batch 100/2304 [I 2022-10-28 07:46:29] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:29 [I 2022-10-28 07:46:29] ...trap/engines/engine.py.229: process: 0.55004 | load: 0.00270 [I 2022-10-28 07:46:29] ...trap/engines/engine.py.230: loss: 0.17782 [I 2022-10-28 07:46:29] ...optimizers/trijoint.py.151: total_norm: 29.764257 [I 2022-10-28 07:46:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 110/2304 [I 2022-10-28 07:46:34] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:16 [I 2022-10-28 07:46:34] ...trap/engines/engine.py.229: process: 0.54729 | load: 0.00268 [I 2022-10-28 07:46:34] ...trap/engines/engine.py.230: loss: 0.11216 [I 2022-10-28 07:46:34] ...optimizers/trijoint.py.151: total_norm: 32.709850 [I 2022-10-28 07:46:40] ...trap/engines/engine.py.224: train: epoch 114 | batch 120/2304 [I 2022-10-28 07:46:40] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:13 [I 2022-10-28 07:46:40] ...trap/engines/engine.py.229: process: 0.55433 | load: 0.00268 [I 2022-10-28 07:46:40] ...trap/engines/engine.py.230: loss: 0.08901 [I 2022-10-28 07:46:40] ...optimizers/trijoint.py.151: total_norm: 18.151709 [I 2022-10-28 07:46:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 130/2304 [I 2022-10-28 07:46:45] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:09 [I 2022-10-28 07:46:45] ...trap/engines/engine.py.229: process: 0.55135 | load: 0.00251 [I 2022-10-28 07:46:45] ...trap/engines/engine.py.230: loss: 0.11998 [I 2022-10-28 07:46:45] ...optimizers/trijoint.py.151: total_norm: 7.791424 [I 2022-10-28 07:46:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 140/2304 [I 2022-10-28 07:46:51] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:26 [I 2022-10-28 07:46:51] ...trap/engines/engine.py.229: process: 0.55710 | load: 0.00243 [I 2022-10-28 07:46:51] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-28 07:46:51] ...optimizers/trijoint.py.151: total_norm: 19.808411 [I 2022-10-28 07:46:57] ...trap/engines/engine.py.224: train: epoch 114 | batch 150/2304 [I 2022-10-28 07:46:57] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:21 [I 2022-10-28 07:46:57] ...trap/engines/engine.py.229: process: 0.57906 | load: 0.00263 [I 2022-10-28 07:46:57] ...trap/engines/engine.py.230: loss: 0.11208 [I 2022-10-28 07:46:57] ...optimizers/trijoint.py.151: total_norm: 18.755671 [I 2022-10-28 07:47:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 160/2304 [I 2022-10-28 07:47:02] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:01 [I 2022-10-28 07:47:02] ...trap/engines/engine.py.229: process: 0.57616 | load: 0.00265 [I 2022-10-28 07:47:02] ...trap/engines/engine.py.230: loss: 0.09941 [I 2022-10-28 07:47:02] ...optimizers/trijoint.py.151: total_norm: 9.286115 [I 2022-10-28 07:47:08] ...trap/engines/engine.py.224: train: epoch 114 | batch 170/2304 [I 2022-10-28 07:47:08] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:59 [I 2022-10-28 07:47:08] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00260 [I 2022-10-28 07:47:08] ...trap/engines/engine.py.230: loss: 0.12268 [I 2022-10-28 07:47:08] ...optimizers/trijoint.py.151: total_norm: 10.068372 [I 2022-10-28 07:47:14] ...trap/engines/engine.py.224: train: epoch 114 | batch 180/2304 [I 2022-10-28 07:47:14] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:55 [I 2022-10-28 07:47:14] ...trap/engines/engine.py.229: process: 0.56232 | load: 0.00261 [I 2022-10-28 07:47:14] ...trap/engines/engine.py.230: loss: 0.11376 [I 2022-10-28 07:47:14] ...optimizers/trijoint.py.151: total_norm: 22.806921 [I 2022-10-28 07:47:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 190/2304 [I 2022-10-28 07:47:19] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:41 [I 2022-10-28 07:47:19] ...trap/engines/engine.py.229: process: 0.56423 | load: 0.00252 [I 2022-10-28 07:47:19] ...trap/engines/engine.py.230: loss: 0.10472 [I 2022-10-28 07:47:19] ...optimizers/trijoint.py.151: total_norm: 15.634677 [I 2022-10-28 07:47:25] ...trap/engines/engine.py.224: train: epoch 114 | batch 200/2304 [I 2022-10-28 07:47:25] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:29 [I 2022-10-28 07:47:25] ...trap/engines/engine.py.229: process: 0.54445 | load: 0.00252 [I 2022-10-28 07:47:25] ...trap/engines/engine.py.230: loss: 0.11768 [I 2022-10-28 07:47:25] ...optimizers/trijoint.py.151: total_norm: 20.528936 [I 2022-10-28 07:47:31] ...trap/engines/engine.py.224: train: epoch 114 | batch 210/2304 [I 2022-10-28 07:47:31] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:26 [I 2022-10-28 07:47:31] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00264 [I 2022-10-28 07:47:31] ...trap/engines/engine.py.230: loss: 0.08398 [I 2022-10-28 07:47:31] ...optimizers/trijoint.py.151: total_norm: 19.097065 [I 2022-10-28 07:47:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 220/2304 [I 2022-10-28 07:47:36] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:24 [I 2022-10-28 07:47:36] ...trap/engines/engine.py.229: process: 0.55494 | load: 0.00245 [I 2022-10-28 07:47:36] ...trap/engines/engine.py.230: loss: 0.21582 [I 2022-10-28 07:47:36] ...optimizers/trijoint.py.151: total_norm: 82.513680 [I 2022-10-28 07:47:42] ...trap/engines/engine.py.224: train: epoch 114 | batch 230/2304 [I 2022-10-28 07:47:42] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:39 [I 2022-10-28 07:47:42] ...trap/engines/engine.py.229: process: 0.61781 | load: 0.00253 [I 2022-10-28 07:47:42] ...trap/engines/engine.py.230: loss: 0.11404 [I 2022-10-28 07:47:42] ...optimizers/trijoint.py.151: total_norm: 11.758297 [I 2022-10-28 07:47:47] ...trap/engines/engine.py.224: train: epoch 114 | batch 240/2304 [I 2022-10-28 07:47:47] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:20 [I 2022-10-28 07:47:47] ...trap/engines/engine.py.229: process: 0.54995 | load: 0.00248 [I 2022-10-28 07:47:47] ...trap/engines/engine.py.230: loss: 0.12609 [I 2022-10-28 07:47:47] ...optimizers/trijoint.py.151: total_norm: 32.513187 [I 2022-10-28 07:47:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 250/2304 [I 2022-10-28 07:47:53] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:15 [I 2022-10-28 07:47:53] ...trap/engines/engine.py.229: process: 0.56801 | load: 0.00245 [I 2022-10-28 07:47:53] ...trap/engines/engine.py.230: loss: 0.18226 [I 2022-10-28 07:47:53] ...optimizers/trijoint.py.151: total_norm: 13.803961 [I 2022-10-28 07:47:59] ...trap/engines/engine.py.224: train: epoch 114 | batch 260/2304 [I 2022-10-28 07:47:59] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:08 [I 2022-10-28 07:47:59] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00254 [I 2022-10-28 07:47:59] ...trap/engines/engine.py.230: loss: 0.08423 [I 2022-10-28 07:47:59] ...optimizers/trijoint.py.151: total_norm: 8.885486 [I 2022-10-28 07:48:04] ...trap/engines/engine.py.224: train: epoch 114 | batch 270/2304 [I 2022-10-28 07:48:04] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:19:00 [I 2022-10-28 07:48:04] ...trap/engines/engine.py.229: process: 0.57592 | load: 0.00292 [I 2022-10-28 07:48:04] ...trap/engines/engine.py.230: loss: 0.13455 [I 2022-10-28 07:48:04] ...optimizers/trijoint.py.151: total_norm: 15.905236 [I 2022-10-28 07:48:10] ...trap/engines/engine.py.224: train: epoch 114 | batch 280/2304 [I 2022-10-28 07:48:10] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:18:56 [I 2022-10-28 07:48:10] ...trap/engines/engine.py.229: process: 0.57634 | load: 0.00330 [I 2022-10-28 07:48:10] ...trap/engines/engine.py.230: loss: 0.09736 [I 2022-10-28 07:48:10] ...optimizers/trijoint.py.151: total_norm: 19.824982 [I 2022-10-28 07:48:15] ...trap/engines/engine.py.224: train: epoch 114 | batch 290/2304 [I 2022-10-28 07:48:16] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:40 [I 2022-10-28 07:48:16] ...trap/engines/engine.py.229: process: 0.54791 | load: 0.00250 [I 2022-10-28 07:48:16] ...trap/engines/engine.py.230: loss: 0.11768 [I 2022-10-28 07:48:16] ...optimizers/trijoint.py.151: total_norm: 11.204527 [I 2022-10-28 07:48:21] ...trap/engines/engine.py.224: train: epoch 114 | batch 300/2304 [I 2022-10-28 07:48:21] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:57 [I 2022-10-28 07:48:21] ...trap/engines/engine.py.229: process: 0.56582 | load: 0.00248 [I 2022-10-28 07:48:21] ...trap/engines/engine.py.230: loss: 0.10536 [I 2022-10-28 07:48:21] ...optimizers/trijoint.py.151: total_norm: 14.570466 [I 2022-10-28 07:48:27] ...trap/engines/engine.py.224: train: epoch 114 | batch 310/2304 [I 2022-10-28 07:48:27] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:18:38 [I 2022-10-28 07:48:27] ...trap/engines/engine.py.229: process: 0.57436 | load: 0.00247 [I 2022-10-28 07:48:27] ...trap/engines/engine.py.230: loss: 0.09426 [I 2022-10-28 07:48:27] ...optimizers/trijoint.py.151: total_norm: 24.006268 [I 2022-10-28 07:48:32] ...trap/engines/engine.py.224: train: epoch 114 | batch 320/2304 [I 2022-10-28 07:48:32] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:20 [I 2022-10-28 07:48:32] ...trap/engines/engine.py.229: process: 0.54402 | load: 0.00260 [I 2022-10-28 07:48:32] ...trap/engines/engine.py.230: loss: 0.11108 [I 2022-10-28 07:48:32] ...optimizers/trijoint.py.151: total_norm: 23.546148 [I 2022-10-28 07:48:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 330/2304 [I 2022-10-28 07:48:38] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:18:29 [I 2022-10-28 07:48:38] ...trap/engines/engine.py.229: process: 0.59099 | load: 0.00251 [I 2022-10-28 07:48:38] ...trap/engines/engine.py.230: loss: 0.09341 [I 2022-10-28 07:48:38] ...optimizers/trijoint.py.151: total_norm: 22.326031 [I 2022-10-28 07:48:44] ...trap/engines/engine.py.224: train: epoch 114 | batch 340/2304 [I 2022-10-28 07:48:44] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:06 [I 2022-10-28 07:48:44] ...trap/engines/engine.py.229: process: 0.56101 | load: 0.00259 [I 2022-10-28 07:48:44] ...trap/engines/engine.py.230: loss: 0.09977 [I 2022-10-28 07:48:44] ...optimizers/trijoint.py.151: total_norm: 9.329093 [I 2022-10-28 07:48:49] ...trap/engines/engine.py.224: train: epoch 114 | batch 350/2304 [I 2022-10-28 07:48:49] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:03 [I 2022-10-28 07:48:49] ...trap/engines/engine.py.229: process: 0.55973 | load: 0.00245 [I 2022-10-28 07:48:49] ...trap/engines/engine.py.230: loss: 0.11829 [I 2022-10-28 07:48:49] ...optimizers/trijoint.py.151: total_norm: 15.996230 [I 2022-10-28 07:48:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 360/2304 [I 2022-10-28 07:48:55] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:59 [I 2022-10-28 07:48:55] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00250 [I 2022-10-28 07:48:55] ...trap/engines/engine.py.230: loss: 0.20311 [I 2022-10-28 07:48:55] ...optimizers/trijoint.py.151: total_norm: 17.697048 [I 2022-10-28 07:49:00] ...trap/engines/engine.py.224: train: epoch 114 | batch 370/2304 [I 2022-10-28 07:49:00] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:17:57 [I 2022-10-28 07:49:00] ...trap/engines/engine.py.229: process: 0.57041 | load: 0.00257 [I 2022-10-28 07:49:00] ...trap/engines/engine.py.230: loss: 0.10157 [I 2022-10-28 07:49:00] ...optimizers/trijoint.py.151: total_norm: 18.927469 [I 2022-10-28 07:49:06] ...trap/engines/engine.py.224: train: epoch 114 | batch 380/2304 [I 2022-10-28 07:49:06] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:44 [I 2022-10-28 07:49:06] ...trap/engines/engine.py.229: process: 0.54705 | load: 0.00241 [I 2022-10-28 07:49:06] ...trap/engines/engine.py.230: loss: 0.07247 [I 2022-10-28 07:49:06] ...optimizers/trijoint.py.151: total_norm: 15.412544 [I 2022-10-28 07:49:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 390/2304 [I 2022-10-28 07:49:11] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:47 [I 2022-10-28 07:49:11] ...trap/engines/engine.py.229: process: 0.55124 | load: 0.00253 [I 2022-10-28 07:49:11] ...trap/engines/engine.py.230: loss: 0.11333 [I 2022-10-28 07:49:11] ...optimizers/trijoint.py.151: total_norm: 19.537565 [I 2022-10-28 07:49:17] ...trap/engines/engine.py.224: train: epoch 114 | batch 400/2304 [I 2022-10-28 07:49:17] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:36 [I 2022-10-28 07:49:17] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00253 [I 2022-10-28 07:49:17] ...trap/engines/engine.py.230: loss: 0.08286 [I 2022-10-28 07:49:17] ...optimizers/trijoint.py.151: total_norm: 9.815369 [I 2022-10-28 07:49:23] ...trap/engines/engine.py.224: train: epoch 114 | batch 410/2304 [I 2022-10-28 07:49:23] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:36 [I 2022-10-28 07:49:23] ...trap/engines/engine.py.229: process: 0.56026 | load: 0.00251 [I 2022-10-28 07:49:23] ...trap/engines/engine.py.230: loss: 0.12598 [I 2022-10-28 07:49:23] ...optimizers/trijoint.py.151: total_norm: 29.095522 [I 2022-10-28 07:49:28] ...trap/engines/engine.py.224: train: epoch 114 | batch 420/2304 [I 2022-10-28 07:49:28] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:18:10 [I 2022-10-28 07:49:28] ...trap/engines/engine.py.229: process: 0.56659 | load: 0.00244 [I 2022-10-28 07:49:28] ...trap/engines/engine.py.230: loss: 0.10341 [I 2022-10-28 07:49:28] ...optimizers/trijoint.py.151: total_norm: 17.896587 [I 2022-10-28 07:49:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 430/2304 [I 2022-10-28 07:49:34] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:35 [I 2022-10-28 07:49:34] ...trap/engines/engine.py.229: process: 0.56250 | load: 0.00246 [I 2022-10-28 07:49:34] ...trap/engines/engine.py.230: loss: 0.16221 [I 2022-10-28 07:49:34] ...optimizers/trijoint.py.151: total_norm: 21.485996 [I 2022-10-28 07:49:40] ...trap/engines/engine.py.224: train: epoch 114 | batch 440/2304 [I 2022-10-28 07:49:40] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:24 [I 2022-10-28 07:49:40] ...trap/engines/engine.py.229: process: 0.56413 | load: 0.00244 [I 2022-10-28 07:49:40] ...trap/engines/engine.py.230: loss: 0.09513 [I 2022-10-28 07:49:40] ...optimizers/trijoint.py.151: total_norm: 13.069891 [I 2022-10-28 07:49:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 450/2304 [I 2022-10-28 07:49:45] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:08 [I 2022-10-28 07:49:45] ...trap/engines/engine.py.229: process: 0.55036 | load: 0.00244 [I 2022-10-28 07:49:45] ...trap/engines/engine.py.230: loss: 0.10501 [I 2022-10-28 07:49:45] ...optimizers/trijoint.py.151: total_norm: 7.697451 [I 2022-10-28 07:49:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 460/2304 [I 2022-10-28 07:49:51] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:06 [I 2022-10-28 07:49:51] ...trap/engines/engine.py.229: process: 0.56426 | load: 0.00241 [I 2022-10-28 07:49:51] ...trap/engines/engine.py.230: loss: 0.10229 [I 2022-10-28 07:49:51] ...optimizers/trijoint.py.151: total_norm: 16.299763 [I 2022-10-28 07:49:56] ...trap/engines/engine.py.224: train: epoch 114 | batch 470/2304 [I 2022-10-28 07:49:56] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:16:59 [I 2022-10-28 07:49:56] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00249 [I 2022-10-28 07:49:56] ...trap/engines/engine.py.230: loss: 0.08742 [I 2022-10-28 07:49:56] ...optimizers/trijoint.py.151: total_norm: 15.279697 [I 2022-10-28 07:50:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 480/2304 [I 2022-10-28 07:50:02] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:49 [I 2022-10-28 07:50:02] ...trap/engines/engine.py.229: process: 0.55678 | load: 0.00247 [I 2022-10-28 07:50:02] ...trap/engines/engine.py.230: loss: 0.10199 [I 2022-10-28 07:50:02] ...optimizers/trijoint.py.151: total_norm: 24.100035 [I 2022-10-28 07:50:07] ...trap/engines/engine.py.224: train: epoch 114 | batch 490/2304 [I 2022-10-28 07:50:07] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:45 [I 2022-10-28 07:50:07] ...trap/engines/engine.py.229: process: 0.55579 | load: 0.00245 [I 2022-10-28 07:50:07] ...trap/engines/engine.py.230: loss: 0.10622 [I 2022-10-28 07:50:07] ...optimizers/trijoint.py.151: total_norm: 15.218954 [I 2022-10-28 07:50:13] ...trap/engines/engine.py.224: train: epoch 114 | batch 500/2304 [I 2022-10-28 07:50:13] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:44 [I 2022-10-28 07:50:13] ...trap/engines/engine.py.229: process: 0.54705 | load: 0.00247 [I 2022-10-28 07:50:13] ...trap/engines/engine.py.230: loss: 0.12856 [I 2022-10-28 07:50:13] ...optimizers/trijoint.py.151: total_norm: 18.395741 [I 2022-10-28 07:50:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 510/2304 [I 2022-10-28 07:50:19] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:34 [I 2022-10-28 07:50:19] ...trap/engines/engine.py.229: process: 0.55524 | load: 0.00244 [I 2022-10-28 07:50:19] ...trap/engines/engine.py.230: loss: 0.09919 [I 2022-10-28 07:50:19] ...optimizers/trijoint.py.151: total_norm: 13.605257 [I 2022-10-28 07:50:24] ...trap/engines/engine.py.224: train: epoch 114 | batch 520/2304 [I 2022-10-28 07:50:24] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:31 [I 2022-10-28 07:50:24] ...trap/engines/engine.py.229: process: 0.54858 | load: 0.00245 [I 2022-10-28 07:50:24] ...trap/engines/engine.py.230: loss: 0.11517 [I 2022-10-28 07:50:24] ...optimizers/trijoint.py.151: total_norm: 9.430456 [I 2022-10-28 07:50:30] ...trap/engines/engine.py.224: train: epoch 114 | batch 530/2304 [I 2022-10-28 07:50:30] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:36 [I 2022-10-28 07:50:30] ...trap/engines/engine.py.229: process: 0.57334 | load: 0.00244 [I 2022-10-28 07:50:30] ...trap/engines/engine.py.230: loss: 0.07978 [I 2022-10-28 07:50:30] ...optimizers/trijoint.py.151: total_norm: 15.902613 [I 2022-10-28 07:50:35] ...trap/engines/engine.py.224: train: epoch 114 | batch 540/2304 [I 2022-10-28 07:50:35] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:28 [I 2022-10-28 07:50:35] ...trap/engines/engine.py.229: process: 0.56479 | load: 0.00243 [I 2022-10-28 07:50:35] ...trap/engines/engine.py.230: loss: 0.12292 [I 2022-10-28 07:50:35] ...optimizers/trijoint.py.151: total_norm: 18.509481 [I 2022-10-28 07:50:41] ...trap/engines/engine.py.224: train: epoch 114 | batch 550/2304 [I 2022-10-28 07:50:41] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:16 [I 2022-10-28 07:50:41] ...trap/engines/engine.py.229: process: 0.56083 | load: 0.00245 [I 2022-10-28 07:50:41] ...trap/engines/engine.py.230: loss: 0.08953 [I 2022-10-28 07:50:41] ...optimizers/trijoint.py.151: total_norm: 17.731895 [I 2022-10-28 07:50:46] ...trap/engines/engine.py.224: train: epoch 114 | batch 560/2304 [I 2022-10-28 07:50:46] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:13 [I 2022-10-28 07:50:46] ...trap/engines/engine.py.229: process: 0.55954 | load: 0.00244 [I 2022-10-28 07:50:46] ...trap/engines/engine.py.230: loss: 0.18246 [I 2022-10-28 07:50:46] ...optimizers/trijoint.py.151: total_norm: 36.242828 [I 2022-10-28 07:50:52] ...trap/engines/engine.py.224: train: epoch 114 | batch 570/2304 [I 2022-10-28 07:50:52] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:04 [I 2022-10-28 07:50:52] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00239 [I 2022-10-28 07:50:52] ...trap/engines/engine.py.230: loss: 0.09456 [I 2022-10-28 07:50:52] ...optimizers/trijoint.py.151: total_norm: 10.181420 [I 2022-10-28 07:50:58] ...trap/engines/engine.py.224: train: epoch 114 | batch 580/2304 [I 2022-10-28 07:50:58] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:50 [I 2022-10-28 07:50:58] ...trap/engines/engine.py.229: process: 0.55033 | load: 0.00244 [I 2022-10-28 07:50:58] ...trap/engines/engine.py.230: loss: 0.09807 [I 2022-10-28 07:50:58] ...optimizers/trijoint.py.151: total_norm: 20.331465 [I 2022-10-28 07:51:03] ...trap/engines/engine.py.224: train: epoch 114 | batch 590/2304 [I 2022-10-28 07:51:03] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:52 [I 2022-10-28 07:51:03] ...trap/engines/engine.py.229: process: 0.55971 | load: 0.00256 [I 2022-10-28 07:51:03] ...trap/engines/engine.py.230: loss: 0.09815 [I 2022-10-28 07:51:03] ...optimizers/trijoint.py.151: total_norm: 15.637007 [I 2022-10-28 07:51:09] ...trap/engines/engine.py.224: train: epoch 114 | batch 600/2304 [I 2022-10-28 07:51:09] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:15:43 [I 2022-10-28 07:51:09] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00244 [I 2022-10-28 07:51:09] ...trap/engines/engine.py.230: loss: 0.08816 [I 2022-10-28 07:51:09] ...optimizers/trijoint.py.151: total_norm: 25.801523 [I 2022-10-28 07:51:14] ...trap/engines/engine.py.224: train: epoch 114 | batch 610/2304 [I 2022-10-28 07:51:14] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:43 [I 2022-10-28 07:51:14] ...trap/engines/engine.py.229: process: 0.54700 | load: 0.00240 [I 2022-10-28 07:51:14] ...trap/engines/engine.py.230: loss: 0.14013 [I 2022-10-28 07:51:14] ...optimizers/trijoint.py.151: total_norm: 17.482067 [I 2022-10-28 07:51:20] ...trap/engines/engine.py.224: train: epoch 114 | batch 620/2304 [I 2022-10-28 07:51:20] ...trap/engines/engine.py.228: elapsed: 0:05:49 | left: 0:15:37 [I 2022-10-28 07:51:20] ...trap/engines/engine.py.229: process: 0.56608 | load: 0.00242 [I 2022-10-28 07:51:20] ...trap/engines/engine.py.230: loss: 0.17142 [I 2022-10-28 07:51:20] ...optimizers/trijoint.py.151: total_norm: 10.031543 [I 2022-10-28 07:51:25] ...trap/engines/engine.py.224: train: epoch 114 | batch 630/2304 [I 2022-10-28 07:51:25] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:32 [I 2022-10-28 07:51:25] ...trap/engines/engine.py.229: process: 0.53973 | load: 0.00241 [I 2022-10-28 07:51:25] ...trap/engines/engine.py.230: loss: 0.10954 [I 2022-10-28 07:51:25] ...optimizers/trijoint.py.151: total_norm: 14.347680 [I 2022-10-28 07:51:31] ...trap/engines/engine.py.224: train: epoch 114 | batch 640/2304 [I 2022-10-28 07:51:31] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:24 [I 2022-10-28 07:51:31] ...trap/engines/engine.py.229: process: 0.54598 | load: 0.00243 [I 2022-10-28 07:51:31] ...trap/engines/engine.py.230: loss: 0.12148 [I 2022-10-28 07:51:31] ...optimizers/trijoint.py.151: total_norm: 13.254224 [I 2022-10-28 07:51:37] ...trap/engines/engine.py.224: train: epoch 114 | batch 650/2304 [I 2022-10-28 07:51:37] ...trap/engines/engine.py.228: elapsed: 0:06:06 | left: 0:15:17 [I 2022-10-28 07:51:37] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00248 [I 2022-10-28 07:51:37] ...trap/engines/engine.py.230: loss: 0.12478 [I 2022-10-28 07:51:37] ...optimizers/trijoint.py.151: total_norm: 16.788006 [I 2022-10-28 07:51:42] ...trap/engines/engine.py.224: train: epoch 114 | batch 660/2304 [I 2022-10-28 07:51:42] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:15 [I 2022-10-28 07:51:42] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00244 [I 2022-10-28 07:51:42] ...trap/engines/engine.py.230: loss: 0.15785 [I 2022-10-28 07:51:42] ...optimizers/trijoint.py.151: total_norm: 11.615588 [I 2022-10-28 07:51:48] ...trap/engines/engine.py.224: train: epoch 114 | batch 670/2304 [I 2022-10-28 07:51:48] ...trap/engines/engine.py.228: elapsed: 0:06:17 | left: 0:15:09 [I 2022-10-28 07:51:48] ...trap/engines/engine.py.229: process: 0.55463 | load: 0.00247 [I 2022-10-28 07:51:48] ...trap/engines/engine.py.230: loss: 0.10723 [I 2022-10-28 07:51:48] ...optimizers/trijoint.py.151: total_norm: 12.799949 [I 2022-10-28 07:51:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 680/2304 [I 2022-10-28 07:51:53] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:02 [I 2022-10-28 07:51:53] ...trap/engines/engine.py.229: process: 0.55622 | load: 0.00254 [I 2022-10-28 07:51:53] ...trap/engines/engine.py.230: loss: 0.12806 [I 2022-10-28 07:51:53] ...optimizers/trijoint.py.151: total_norm: 10.942301 [I 2022-10-28 07:51:59] ...trap/engines/engine.py.224: train: epoch 114 | batch 690/2304 [I 2022-10-28 07:51:59] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:15:01 [I 2022-10-28 07:51:59] ...trap/engines/engine.py.229: process: 0.56126 | load: 0.00248 [I 2022-10-28 07:51:59] ...trap/engines/engine.py.230: loss: 0.11149 [I 2022-10-28 07:51:59] ...optimizers/trijoint.py.151: total_norm: 15.849833 [I 2022-10-28 07:52:05] ...trap/engines/engine.py.224: train: epoch 114 | batch 700/2304 [I 2022-10-28 07:52:05] ...trap/engines/engine.py.228: elapsed: 0:06:34 | left: 0:14:55 [I 2022-10-28 07:52:05] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00244 [I 2022-10-28 07:52:05] ...trap/engines/engine.py.230: loss: 0.18253 [I 2022-10-28 07:52:05] ...optimizers/trijoint.py.151: total_norm: 11.329715 [I 2022-10-28 07:52:10] ...trap/engines/engine.py.224: train: epoch 114 | batch 710/2304 [I 2022-10-28 07:52:10] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:46 [I 2022-10-28 07:52:10] ...trap/engines/engine.py.229: process: 0.55544 | load: 0.00248 [I 2022-10-28 07:52:10] ...trap/engines/engine.py.230: loss: 0.08339 [I 2022-10-28 07:52:10] ...optimizers/trijoint.py.151: total_norm: 10.515437 [I 2022-10-28 07:52:16] ...trap/engines/engine.py.224: train: epoch 114 | batch 720/2304 [I 2022-10-28 07:52:16] ...trap/engines/engine.py.228: elapsed: 0:06:45 | left: 0:14:42 [I 2022-10-28 07:52:16] ...trap/engines/engine.py.229: process: 0.55844 | load: 0.00251 [I 2022-10-28 07:52:16] ...trap/engines/engine.py.230: loss: 0.08780 [I 2022-10-28 07:52:16] ...optimizers/trijoint.py.151: total_norm: 11.448255 [I 2022-10-28 07:52:21] ...trap/engines/engine.py.224: train: epoch 114 | batch 730/2304 [I 2022-10-28 07:52:21] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:42 [I 2022-10-28 07:52:21] ...trap/engines/engine.py.229: process: 0.58806 | load: 0.00245 [I 2022-10-28 07:52:21] ...trap/engines/engine.py.230: loss: 0.14890 [I 2022-10-28 07:52:21] ...optimizers/trijoint.py.151: total_norm: 14.342196 [I 2022-10-28 07:52:27] ...trap/engines/engine.py.224: train: epoch 114 | batch 740/2304 [I 2022-10-28 07:52:27] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:27 [I 2022-10-28 07:52:27] ...trap/engines/engine.py.229: process: 0.54493 | load: 0.00249 [I 2022-10-28 07:52:27] ...trap/engines/engine.py.230: loss: 0.14053 [I 2022-10-28 07:52:27] ...optimizers/trijoint.py.151: total_norm: 12.591729 [I 2022-10-28 07:52:33] ...trap/engines/engine.py.224: train: epoch 114 | batch 750/2304 [I 2022-10-28 07:52:33] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:31 [I 2022-10-28 07:52:33] ...trap/engines/engine.py.229: process: 0.56658 | load: 0.00243 [I 2022-10-28 07:52:33] ...trap/engines/engine.py.230: loss: 0.06895 [I 2022-10-28 07:52:33] ...optimizers/trijoint.py.151: total_norm: 14.100112 [I 2022-10-28 07:52:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 760/2304 [I 2022-10-28 07:52:38] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:14 [I 2022-10-28 07:52:38] ...trap/engines/engine.py.229: process: 0.54492 | load: 0.00244 [I 2022-10-28 07:52:38] ...trap/engines/engine.py.230: loss: 0.14184 [I 2022-10-28 07:52:38] ...optimizers/trijoint.py.151: total_norm: 22.089594 [I 2022-10-28 07:52:44] ...trap/engines/engine.py.224: train: epoch 114 | batch 770/2304 [I 2022-10-28 07:52:44] ...trap/engines/engine.py.228: elapsed: 0:07:13 | left: 0:14:18 [I 2022-10-28 07:52:44] ...trap/engines/engine.py.229: process: 0.56359 | load: 0.00243 [I 2022-10-28 07:52:44] ...trap/engines/engine.py.230: loss: 0.11447 [I 2022-10-28 07:52:44] ...optimizers/trijoint.py.151: total_norm: 9.568515 [I 2022-10-28 07:52:49] ...trap/engines/engine.py.224: train: epoch 114 | batch 780/2304 [I 2022-10-28 07:52:49] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:04 [I 2022-10-28 07:52:49] ...trap/engines/engine.py.229: process: 0.55170 | load: 0.00243 [I 2022-10-28 07:52:49] ...trap/engines/engine.py.230: loss: 0.08045 [I 2022-10-28 07:52:49] ...optimizers/trijoint.py.151: total_norm: 13.180822 [I 2022-10-28 07:52:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 790/2304 [I 2022-10-28 07:52:55] ...trap/engines/engine.py.228: elapsed: 0:07:24 | left: 0:14:09 [I 2022-10-28 07:52:55] ...trap/engines/engine.py.229: process: 0.57183 | load: 0.00244 [I 2022-10-28 07:52:55] ...trap/engines/engine.py.230: loss: 0.10234 [I 2022-10-28 07:52:55] ...optimizers/trijoint.py.151: total_norm: 24.582138 [I 2022-10-28 07:53:00] ...trap/engines/engine.py.224: train: epoch 114 | batch 800/2304 [I 2022-10-28 07:53:00] ...trap/engines/engine.py.228: elapsed: 0:07:30 | left: 0:13:55 [I 2022-10-28 07:53:00] ...trap/engines/engine.py.229: process: 0.55477 | load: 0.00240 [I 2022-10-28 07:53:00] ...trap/engines/engine.py.230: loss: 0.10007 [I 2022-10-28 07:53:00] ...optimizers/trijoint.py.151: total_norm: 12.989618 [I 2022-10-28 07:53:06] ...trap/engines/engine.py.224: train: epoch 114 | batch 810/2304 [I 2022-10-28 07:53:06] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:54 [I 2022-10-28 07:53:06] ...trap/engines/engine.py.229: process: 0.56574 | load: 0.00241 [I 2022-10-28 07:53:06] ...trap/engines/engine.py.230: loss: 0.16238 [I 2022-10-28 07:53:06] ...optimizers/trijoint.py.151: total_norm: 29.682819 [I 2022-10-28 07:53:12] ...trap/engines/engine.py.224: train: epoch 114 | batch 820/2304 [I 2022-10-28 07:53:12] ...trap/engines/engine.py.228: elapsed: 0:07:41 | left: 0:13:48 [I 2022-10-28 07:53:12] ...trap/engines/engine.py.229: process: 0.53970 | load: 0.00256 [I 2022-10-28 07:53:12] ...trap/engines/engine.py.230: loss: 0.15916 [I 2022-10-28 07:53:12] ...optimizers/trijoint.py.151: total_norm: 19.491726 [I 2022-10-28 07:53:17] ...trap/engines/engine.py.224: train: epoch 114 | batch 830/2304 [I 2022-10-28 07:53:17] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:41 [I 2022-10-28 07:53:17] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00252 [I 2022-10-28 07:53:17] ...trap/engines/engine.py.230: loss: 0.19714 [I 2022-10-28 07:53:17] ...optimizers/trijoint.py.151: total_norm: 28.449457 [I 2022-10-28 07:53:23] ...trap/engines/engine.py.224: train: epoch 114 | batch 840/2304 [I 2022-10-28 07:53:23] ...trap/engines/engine.py.228: elapsed: 0:07:52 | left: 0:13:42 [I 2022-10-28 07:53:23] ...trap/engines/engine.py.229: process: 0.55890 | load: 0.00243 [I 2022-10-28 07:53:23] ...trap/engines/engine.py.230: loss: 0.10603 [I 2022-10-28 07:53:23] ...optimizers/trijoint.py.151: total_norm: 14.941373 [I 2022-10-28 07:53:28] ...trap/engines/engine.py.224: train: epoch 114 | batch 850/2304 [I 2022-10-28 07:53:28] ...trap/engines/engine.py.228: elapsed: 0:07:58 | left: 0:13:27 [I 2022-10-28 07:53:28] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00248 [I 2022-10-28 07:53:28] ...trap/engines/engine.py.230: loss: 0.08107 [I 2022-10-28 07:53:28] ...optimizers/trijoint.py.151: total_norm: 13.816513 [I 2022-10-28 07:53:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 860/2304 [I 2022-10-28 07:53:34] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:19 [I 2022-10-28 07:53:34] ...trap/engines/engine.py.229: process: 0.55176 | load: 0.00243 [I 2022-10-28 07:53:34] ...trap/engines/engine.py.230: loss: 0.17240 [I 2022-10-28 07:53:34] ...optimizers/trijoint.py.151: total_norm: 14.530770 [I 2022-10-28 07:53:39] ...trap/engines/engine.py.224: train: epoch 114 | batch 870/2304 [I 2022-10-28 07:53:39] ...trap/engines/engine.py.228: elapsed: 0:08:09 | left: 0:13:19 [I 2022-10-28 07:53:39] ...trap/engines/engine.py.229: process: 0.56795 | load: 0.00244 [I 2022-10-28 07:53:39] ...trap/engines/engine.py.230: loss: 0.11715 [I 2022-10-28 07:53:39] ...optimizers/trijoint.py.151: total_norm: 12.211028 [I 2022-10-28 07:53:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 880/2304 [I 2022-10-28 07:53:45] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:18 [I 2022-10-28 07:53:45] ...trap/engines/engine.py.229: process: 0.56600 | load: 0.00249 [I 2022-10-28 07:53:45] ...trap/engines/engine.py.230: loss: 0.08884 [I 2022-10-28 07:53:45] ...optimizers/trijoint.py.151: total_norm: 13.131584 [I 2022-10-28 07:53:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 890/2304 [I 2022-10-28 07:53:51] ...trap/engines/engine.py.228: elapsed: 0:08:20 | left: 0:13:08 [I 2022-10-28 07:53:51] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00242 [I 2022-10-28 07:53:51] ...trap/engines/engine.py.230: loss: 0.15567 [I 2022-10-28 07:53:51] ...optimizers/trijoint.py.151: total_norm: 10.713422 [I 2022-10-28 07:53:56] ...trap/engines/engine.py.224: train: epoch 114 | batch 900/2304 [I 2022-10-28 07:53:56] ...trap/engines/engine.py.228: elapsed: 0:08:26 | left: 0:13:02 [I 2022-10-28 07:53:56] ...trap/engines/engine.py.229: process: 0.56073 | load: 0.00247 [I 2022-10-28 07:53:56] ...trap/engines/engine.py.230: loss: 0.08888 [I 2022-10-28 07:53:56] ...optimizers/trijoint.py.151: total_norm: 18.053535 [I 2022-10-28 07:54:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 910/2304 [I 2022-10-28 07:54:02] ...trap/engines/engine.py.228: elapsed: 0:08:31 | left: 0:13:01 [I 2022-10-28 07:54:02] ...trap/engines/engine.py.229: process: 0.57007 | load: 0.00248 [I 2022-10-28 07:54:02] ...trap/engines/engine.py.230: loss: 0.13032 [I 2022-10-28 07:54:02] ...optimizers/trijoint.py.151: total_norm: 14.543327 [I 2022-10-28 07:54:07] ...trap/engines/engine.py.224: train: epoch 114 | batch 920/2304 [I 2022-10-28 07:54:07] ...trap/engines/engine.py.228: elapsed: 0:08:37 | left: 0:12:50 [I 2022-10-28 07:54:07] ...trap/engines/engine.py.229: process: 0.54903 | load: 0.00243 [I 2022-10-28 07:54:07] ...trap/engines/engine.py.230: loss: 0.12698 [I 2022-10-28 07:54:07] ...optimizers/trijoint.py.151: total_norm: 13.229167 [I 2022-10-28 07:54:13] ...trap/engines/engine.py.224: train: epoch 114 | batch 930/2304 [I 2022-10-28 07:54:13] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:43 [I 2022-10-28 07:54:13] ...trap/engines/engine.py.229: process: 0.55834 | load: 0.00242 [I 2022-10-28 07:54:13] ...trap/engines/engine.py.230: loss: 0.08227 [I 2022-10-28 07:54:13] ...optimizers/trijoint.py.151: total_norm: 17.702217 [I 2022-10-28 07:54:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 940/2304 [I 2022-10-28 07:54:19] ...trap/engines/engine.py.228: elapsed: 0:08:48 | left: 0:12:41 [I 2022-10-28 07:54:19] ...trap/engines/engine.py.229: process: 0.55624 | load: 0.00245 [I 2022-10-28 07:54:19] ...trap/engines/engine.py.230: loss: 0.09132 [I 2022-10-28 07:54:19] ...optimizers/trijoint.py.151: total_norm: 18.020323 [I 2022-10-28 07:54:24] ...trap/engines/engine.py.224: train: epoch 114 | batch 950/2304 [I 2022-10-28 07:54:24] ...trap/engines/engine.py.228: elapsed: 0:08:54 | left: 0:12:35 [I 2022-10-28 07:54:24] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00260 [I 2022-10-28 07:54:24] ...trap/engines/engine.py.230: loss: 0.12971 [I 2022-10-28 07:54:24] ...optimizers/trijoint.py.151: total_norm: 13.905233 [I 2022-10-28 07:54:30] ...trap/engines/engine.py.224: train: epoch 114 | batch 960/2304 [I 2022-10-28 07:54:30] ...trap/engines/engine.py.228: elapsed: 0:08:59 | left: 0:12:30 [I 2022-10-28 07:54:30] ...trap/engines/engine.py.229: process: 0.56718 | load: 0.00241 [I 2022-10-28 07:54:30] ...trap/engines/engine.py.230: loss: 0.10701 [I 2022-10-28 07:54:30] ...optimizers/trijoint.py.151: total_norm: 15.299328 [I 2022-10-28 07:54:35] ...trap/engines/engine.py.224: train: epoch 114 | batch 970/2304 [I 2022-10-28 07:54:35] ...trap/engines/engine.py.228: elapsed: 0:09:05 | left: 0:12:22 [I 2022-10-28 07:54:35] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00242 [I 2022-10-28 07:54:35] ...trap/engines/engine.py.230: loss: 0.09570 [I 2022-10-28 07:54:35] ...optimizers/trijoint.py.151: total_norm: 12.049496 [I 2022-10-28 07:54:41] ...trap/engines/engine.py.224: train: epoch 114 | batch 980/2304 [I 2022-10-28 07:54:41] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:15 [I 2022-10-28 07:54:41] ...trap/engines/engine.py.229: process: 0.55764 | load: 0.00243 [I 2022-10-28 07:54:41] ...trap/engines/engine.py.230: loss: 0.12643 [I 2022-10-28 07:54:41] ...optimizers/trijoint.py.151: total_norm: 17.991589 [I 2022-10-28 07:54:46] ...trap/engines/engine.py.224: train: epoch 114 | batch 990/2304 [I 2022-10-28 07:54:46] ...trap/engines/engine.py.228: elapsed: 0:09:16 | left: 0:12:13 [I 2022-10-28 07:54:46] ...trap/engines/engine.py.229: process: 0.57261 | load: 0.00240 [I 2022-10-28 07:54:46] ...trap/engines/engine.py.230: loss: 0.12046 [I 2022-10-28 07:54:46] ...optimizers/trijoint.py.151: total_norm: 8.425006 [I 2022-10-28 07:54:52] ...trap/engines/engine.py.224: train: epoch 114 | batch 1000/2304 [I 2022-10-28 07:54:52] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:06 [I 2022-10-28 07:54:52] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00243 [I 2022-10-28 07:54:52] ...trap/engines/engine.py.230: loss: 0.09825 [I 2022-10-28 07:54:52] ...optimizers/trijoint.py.151: total_norm: 18.579361 [I 2022-10-28 07:54:58] ...trap/engines/engine.py.224: train: epoch 114 | batch 1010/2304 [I 2022-10-28 07:54:58] ...trap/engines/engine.py.228: elapsed: 0:09:27 | left: 0:12:04 [I 2022-10-28 07:54:58] ...trap/engines/engine.py.229: process: 0.56321 | load: 0.00258 [I 2022-10-28 07:54:58] ...trap/engines/engine.py.230: loss: 0.11170 [I 2022-10-28 07:54:58] ...optimizers/trijoint.py.151: total_norm: 13.946392 [I 2022-10-28 07:55:03] ...trap/engines/engine.py.224: train: epoch 114 | batch 1020/2304 [I 2022-10-28 07:55:03] ...trap/engines/engine.py.228: elapsed: 0:09:33 | left: 0:11:56 [I 2022-10-28 07:55:03] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00250 [I 2022-10-28 07:55:03] ...trap/engines/engine.py.230: loss: 0.15678 [I 2022-10-28 07:55:03] ...optimizers/trijoint.py.151: total_norm: 13.878158 [I 2022-10-28 07:55:09] ...trap/engines/engine.py.224: train: epoch 114 | batch 1030/2304 [I 2022-10-28 07:55:09] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:58 [I 2022-10-28 07:55:09] ...trap/engines/engine.py.229: process: 0.56887 | load: 0.00246 [I 2022-10-28 07:55:09] ...trap/engines/engine.py.230: loss: 0.10237 [I 2022-10-28 07:55:09] ...optimizers/trijoint.py.151: total_norm: 10.990697 [I 2022-10-28 07:55:15] ...trap/engines/engine.py.224: train: epoch 114 | batch 1040/2304 [I 2022-10-28 07:55:15] ...trap/engines/engine.py.228: elapsed: 0:09:44 | left: 0:11:50 [I 2022-10-28 07:55:15] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00243 [I 2022-10-28 07:55:15] ...trap/engines/engine.py.230: loss: 0.10052 [I 2022-10-28 07:55:15] ...optimizers/trijoint.py.151: total_norm: 15.496697 [I 2022-10-28 07:55:20] ...trap/engines/engine.py.224: train: epoch 114 | batch 1050/2304 [I 2022-10-28 07:55:20] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:44 [I 2022-10-28 07:55:20] ...trap/engines/engine.py.229: process: 0.57762 | load: 0.00243 [I 2022-10-28 07:55:20] ...trap/engines/engine.py.230: loss: 0.10372 [I 2022-10-28 07:55:20] ...optimizers/trijoint.py.151: total_norm: 8.600467 [I 2022-10-28 07:55:26] ...trap/engines/engine.py.224: train: epoch 114 | batch 1060/2304 [I 2022-10-28 07:55:26] ...trap/engines/engine.py.228: elapsed: 0:09:55 | left: 0:11:40 [I 2022-10-28 07:55:26] ...trap/engines/engine.py.229: process: 0.56707 | load: 0.00241 [I 2022-10-28 07:55:26] ...trap/engines/engine.py.230: loss: 0.17276 [I 2022-10-28 07:55:26] ...optimizers/trijoint.py.151: total_norm: 16.698128 [I 2022-10-28 07:55:31] ...trap/engines/engine.py.224: train: epoch 114 | batch 1070/2304 [I 2022-10-28 07:55:31] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:30 [I 2022-10-28 07:55:31] ...trap/engines/engine.py.229: process: 0.56454 | load: 0.00245 [I 2022-10-28 07:55:31] ...trap/engines/engine.py.230: loss: 0.12607 [I 2022-10-28 07:55:31] ...optimizers/trijoint.py.151: total_norm: 70.496513 [I 2022-10-28 07:55:37] ...trap/engines/engine.py.224: train: epoch 114 | batch 1080/2304 [I 2022-10-28 07:55:37] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:25 [I 2022-10-28 07:55:37] ...trap/engines/engine.py.229: process: 0.55293 | load: 0.00240 [I 2022-10-28 07:55:37] ...trap/engines/engine.py.230: loss: 0.07930 [I 2022-10-28 07:55:37] ...optimizers/trijoint.py.151: total_norm: 10.134225 [I 2022-10-28 07:55:43] ...trap/engines/engine.py.224: train: epoch 114 | batch 1090/2304 [I 2022-10-28 07:55:43] ...trap/engines/engine.py.228: elapsed: 0:10:12 | left: 0:11:22 [I 2022-10-28 07:55:43] ...trap/engines/engine.py.229: process: 0.58182 | load: 0.00245 [I 2022-10-28 07:55:43] ...trap/engines/engine.py.230: loss: 0.12134 [I 2022-10-28 07:55:43] ...optimizers/trijoint.py.151: total_norm: 14.386325 [I 2022-10-28 07:55:48] ...trap/engines/engine.py.224: train: epoch 114 | batch 1100/2304 [I 2022-10-28 07:55:48] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:18 [I 2022-10-28 07:55:48] ...trap/engines/engine.py.229: process: 0.56056 | load: 0.00243 [I 2022-10-28 07:55:48] ...trap/engines/engine.py.230: loss: 0.11758 [I 2022-10-28 07:55:48] ...optimizers/trijoint.py.151: total_norm: 55.981613 [I 2022-10-28 07:55:54] ...trap/engines/engine.py.224: train: epoch 114 | batch 1110/2304 [I 2022-10-28 07:55:54] ...trap/engines/engine.py.228: elapsed: 0:10:23 | left: 0:11:09 [I 2022-10-28 07:55:54] ...trap/engines/engine.py.229: process: 0.55980 | load: 0.00245 [I 2022-10-28 07:55:54] ...trap/engines/engine.py.230: loss: 0.15366 [I 2022-10-28 07:55:54] ...optimizers/trijoint.py.151: total_norm: 25.682365 [I 2022-10-28 07:55:59] ...trap/engines/engine.py.224: train: epoch 114 | batch 1120/2304 [I 2022-10-28 07:55:59] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:10:58 [I 2022-10-28 07:55:59] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00243 [I 2022-10-28 07:55:59] ...trap/engines/engine.py.230: loss: 0.07712 [I 2022-10-28 07:55:59] ...optimizers/trijoint.py.151: total_norm: 10.140757 [I 2022-10-28 07:56:05] ...trap/engines/engine.py.224: train: epoch 114 | batch 1130/2304 [I 2022-10-28 07:56:05] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:55 [I 2022-10-28 07:56:05] ...trap/engines/engine.py.229: process: 0.56283 | load: 0.00246 [I 2022-10-28 07:56:05] ...trap/engines/engine.py.230: loss: 0.12730 [I 2022-10-28 07:56:05] ...optimizers/trijoint.py.151: total_norm: 17.554968 [I 2022-10-28 07:56:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 1140/2304 [I 2022-10-28 07:56:11] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:46 [I 2022-10-28 07:56:11] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00240 [I 2022-10-28 07:56:11] ...trap/engines/engine.py.230: loss: 0.07579 [I 2022-10-28 07:56:11] ...optimizers/trijoint.py.151: total_norm: 13.788373 [I 2022-10-28 07:56:16] ...trap/engines/engine.py.224: train: epoch 114 | batch 1150/2304 [I 2022-10-28 07:56:16] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:40 [I 2022-10-28 07:56:16] ...trap/engines/engine.py.229: process: 0.55646 | load: 0.00244 [I 2022-10-28 07:56:16] ...trap/engines/engine.py.230: loss: 0.10405 [I 2022-10-28 07:56:16] ...optimizers/trijoint.py.151: total_norm: 11.808804 [I 2022-10-28 07:56:22] ...trap/engines/engine.py.224: train: epoch 114 | batch 1160/2304 [I 2022-10-28 07:56:22] ...trap/engines/engine.py.228: elapsed: 0:10:51 | left: 0:10:38 [I 2022-10-28 07:56:22] ...trap/engines/engine.py.229: process: 0.56726 | load: 0.00244 [I 2022-10-28 07:56:22] ...trap/engines/engine.py.230: loss: 0.09360 [I 2022-10-28 07:56:22] ...optimizers/trijoint.py.151: total_norm: 7.839855 [I 2022-10-28 07:56:27] ...trap/engines/engine.py.224: train: epoch 114 | batch 1170/2304 [I 2022-10-28 07:56:27] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:29 [I 2022-10-28 07:56:27] ...trap/engines/engine.py.229: process: 0.54708 | load: 0.00243 [I 2022-10-28 07:56:27] ...trap/engines/engine.py.230: loss: 0.09668 [I 2022-10-28 07:56:27] ...optimizers/trijoint.py.151: total_norm: 16.991480 [I 2022-10-28 07:56:33] ...trap/engines/engine.py.224: train: epoch 114 | batch 1180/2304 [I 2022-10-28 07:56:33] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:25 [I 2022-10-28 07:56:33] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00244 [I 2022-10-28 07:56:33] ...trap/engines/engine.py.230: loss: 0.10811 [I 2022-10-28 07:56:33] ...optimizers/trijoint.py.151: total_norm: 36.198719 [I 2022-10-28 07:56:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 1190/2304 [I 2022-10-28 07:56:38] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:18 [I 2022-10-28 07:56:38] ...trap/engines/engine.py.229: process: 0.55035 | load: 0.00241 [I 2022-10-28 07:56:38] ...trap/engines/engine.py.230: loss: 0.12394 [I 2022-10-28 07:56:38] ...optimizers/trijoint.py.151: total_norm: 7.895545 [I 2022-10-28 07:56:44] ...trap/engines/engine.py.224: train: epoch 114 | batch 1200/2304 [I 2022-10-28 07:56:44] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:10 [I 2022-10-28 07:56:44] ...trap/engines/engine.py.229: process: 0.54759 | load: 0.00239 [I 2022-10-28 07:56:44] ...trap/engines/engine.py.230: loss: 0.11299 [I 2022-10-28 07:56:44] ...optimizers/trijoint.py.151: total_norm: 11.745975 [I 2022-10-28 07:56:50] ...trap/engines/engine.py.224: train: epoch 114 | batch 1210/2304 [I 2022-10-28 07:56:50] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:06 [I 2022-10-28 07:56:50] ...trap/engines/engine.py.229: process: 0.54766 | load: 0.00239 [I 2022-10-28 07:56:50] ...trap/engines/engine.py.230: loss: 0.09648 [I 2022-10-28 07:56:50] ...optimizers/trijoint.py.151: total_norm: 9.340307 [I 2022-10-28 07:56:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 1220/2304 [I 2022-10-28 07:56:55] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:09 [I 2022-10-28 07:56:55] ...trap/engines/engine.py.229: process: 0.55793 | load: 0.00241 [I 2022-10-28 07:56:55] ...trap/engines/engine.py.230: loss: 0.14326 [I 2022-10-28 07:56:55] ...optimizers/trijoint.py.151: total_norm: 14.890188 [I 2022-10-28 07:57:01] ...trap/engines/engine.py.224: train: epoch 114 | batch 1230/2304 [I 2022-10-28 07:57:01] ...trap/engines/engine.py.228: elapsed: 0:11:30 | left: 0:09:56 [I 2022-10-28 07:57:01] ...trap/engines/engine.py.229: process: 0.54574 | load: 0.00240 [I 2022-10-28 07:57:01] ...trap/engines/engine.py.230: loss: 0.10031 [I 2022-10-28 07:57:01] ...optimizers/trijoint.py.151: total_norm: 24.313854 [I 2022-10-28 07:57:06] ...trap/engines/engine.py.224: train: epoch 114 | batch 1240/2304 [I 2022-10-28 07:57:06] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:56 [I 2022-10-28 07:57:06] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00250 [I 2022-10-28 07:57:06] ...trap/engines/engine.py.230: loss: 0.10194 [I 2022-10-28 07:57:06] ...optimizers/trijoint.py.151: total_norm: 21.515030 [I 2022-10-28 07:57:12] ...trap/engines/engine.py.224: train: epoch 114 | batch 1250/2304 [I 2022-10-28 07:57:12] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:46 [I 2022-10-28 07:57:12] ...trap/engines/engine.py.229: process: 0.55213 | load: 0.00248 [I 2022-10-28 07:57:12] ...trap/engines/engine.py.230: loss: 0.13762 [I 2022-10-28 07:57:12] ...optimizers/trijoint.py.151: total_norm: 6.910105 [I 2022-10-28 07:57:18] ...trap/engines/engine.py.224: train: epoch 114 | batch 1260/2304 [I 2022-10-28 07:57:18] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:40 [I 2022-10-28 07:57:18] ...trap/engines/engine.py.229: process: 0.57918 | load: 0.00241 [I 2022-10-28 07:57:18] ...trap/engines/engine.py.230: loss: 0.09771 [I 2022-10-28 07:57:18] ...optimizers/trijoint.py.151: total_norm: 17.468987 [I 2022-10-28 07:57:23] ...trap/engines/engine.py.224: train: epoch 114 | batch 1270/2304 [I 2022-10-28 07:57:23] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:37 [I 2022-10-28 07:57:23] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00245 [I 2022-10-28 07:57:23] ...trap/engines/engine.py.230: loss: 0.18652 [I 2022-10-28 07:57:23] ...optimizers/trijoint.py.151: total_norm: 43.197571 [I 2022-10-28 07:57:29] ...trap/engines/engine.py.224: train: epoch 114 | batch 1280/2304 [I 2022-10-28 07:57:29] ...trap/engines/engine.py.228: elapsed: 0:11:58 | left: 0:09:31 [I 2022-10-28 07:57:29] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00242 [I 2022-10-28 07:57:29] ...trap/engines/engine.py.230: loss: 0.09377 [I 2022-10-28 07:57:29] ...optimizers/trijoint.py.151: total_norm: 10.624000 [I 2022-10-28 07:57:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 1290/2304 [I 2022-10-28 07:57:34] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:25 [I 2022-10-28 07:57:34] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00243 [I 2022-10-28 07:57:34] ...trap/engines/engine.py.230: loss: 0.17214 [I 2022-10-28 07:57:34] ...optimizers/trijoint.py.151: total_norm: 29.935402 [I 2022-10-28 07:57:40] ...trap/engines/engine.py.224: train: epoch 114 | batch 1300/2304 [I 2022-10-28 07:57:40] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:19 [I 2022-10-28 07:57:40] ...trap/engines/engine.py.229: process: 0.55488 | load: 0.00244 [I 2022-10-28 07:57:40] ...trap/engines/engine.py.230: loss: 0.12726 [I 2022-10-28 07:57:40] ...optimizers/trijoint.py.151: total_norm: 14.353203 [I 2022-10-28 07:57:46] ...trap/engines/engine.py.224: train: epoch 114 | batch 1310/2304 [I 2022-10-28 07:57:46] ...trap/engines/engine.py.228: elapsed: 0:12:15 | left: 0:09:13 [I 2022-10-28 07:57:46] ...trap/engines/engine.py.229: process: 0.54587 | load: 0.00242 [I 2022-10-28 07:57:46] ...trap/engines/engine.py.230: loss: 0.11060 [I 2022-10-28 07:57:46] ...optimizers/trijoint.py.151: total_norm: 12.266044 [I 2022-10-28 07:57:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 1320/2304 [I 2022-10-28 07:57:51] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:06 [I 2022-10-28 07:57:51] ...trap/engines/engine.py.229: process: 0.56452 | load: 0.00244 [I 2022-10-28 07:57:51] ...trap/engines/engine.py.230: loss: 0.13060 [I 2022-10-28 07:57:51] ...optimizers/trijoint.py.151: total_norm: 18.827345 [I 2022-10-28 07:57:57] ...trap/engines/engine.py.224: train: epoch 114 | batch 1330/2304 [I 2022-10-28 07:57:57] ...trap/engines/engine.py.228: elapsed: 0:12:26 | left: 0:09:06 [I 2022-10-28 07:57:57] ...trap/engines/engine.py.229: process: 0.56752 | load: 0.00247 [I 2022-10-28 07:57:57] ...trap/engines/engine.py.230: loss: 0.11462 [I 2022-10-28 07:57:57] ...optimizers/trijoint.py.151: total_norm: 15.569921 [I 2022-10-28 07:58:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 1340/2304 [I 2022-10-28 07:58:02] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:54 [I 2022-10-28 07:58:02] ...trap/engines/engine.py.229: process: 0.54909 | load: 0.00244 [I 2022-10-28 07:58:02] ...trap/engines/engine.py.230: loss: 0.10614 [I 2022-10-28 07:58:02] ...optimizers/trijoint.py.151: total_norm: 13.913674 [I 2022-10-28 07:58:08] ...trap/engines/engine.py.224: train: epoch 114 | batch 1350/2304 [I 2022-10-28 07:58:08] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:49 [I 2022-10-28 07:58:08] ...trap/engines/engine.py.229: process: 0.55996 | load: 0.00244 [I 2022-10-28 07:58:08] ...trap/engines/engine.py.230: loss: 0.10046 [I 2022-10-28 07:58:08] ...optimizers/trijoint.py.151: total_norm: 10.062038 [I 2022-10-28 07:58:14] ...trap/engines/engine.py.224: train: epoch 114 | batch 1360/2304 [I 2022-10-28 07:58:14] ...trap/engines/engine.py.228: elapsed: 0:12:43 | left: 0:08:50 [I 2022-10-28 07:58:14] ...trap/engines/engine.py.229: process: 0.57568 | load: 0.00243 [I 2022-10-28 07:58:14] ...trap/engines/engine.py.230: loss: 0.09924 [I 2022-10-28 07:58:14] ...optimizers/trijoint.py.151: total_norm: 10.840754 [I 2022-10-28 07:58:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 1370/2304 [I 2022-10-28 07:58:19] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:46 [I 2022-10-28 07:58:19] ...trap/engines/engine.py.229: process: 0.56313 | load: 0.00242 [I 2022-10-28 07:58:19] ...trap/engines/engine.py.230: loss: 0.17674 [I 2022-10-28 07:58:19] ...optimizers/trijoint.py.151: total_norm: 32.014832 [I 2022-10-28 07:58:25] ...trap/engines/engine.py.224: train: epoch 114 | batch 1380/2304 [I 2022-10-28 07:58:25] ...trap/engines/engine.py.228: elapsed: 0:12:54 | left: 0:08:32 [I 2022-10-28 07:58:25] ...trap/engines/engine.py.229: process: 0.56325 | load: 0.00240 [I 2022-10-28 07:58:25] ...trap/engines/engine.py.230: loss: 0.17309 [I 2022-10-28 07:58:25] ...optimizers/trijoint.py.151: total_norm: 13.918813 [I 2022-10-28 07:58:30] ...trap/engines/engine.py.224: train: epoch 114 | batch 1390/2304 [I 2022-10-28 07:58:30] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:27 [I 2022-10-28 07:58:30] ...trap/engines/engine.py.229: process: 0.55644 | load: 0.00244 [I 2022-10-28 07:58:30] ...trap/engines/engine.py.230: loss: 0.10324 [I 2022-10-28 07:58:30] ...optimizers/trijoint.py.151: total_norm: 31.935373 [I 2022-10-28 07:58:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 1400/2304 [I 2022-10-28 07:58:36] ...trap/engines/engine.py.228: elapsed: 0:13:05 | left: 0:08:25 [I 2022-10-28 07:58:36] ...trap/engines/engine.py.229: process: 0.56244 | load: 0.00240 [I 2022-10-28 07:58:36] ...trap/engines/engine.py.230: loss: 0.08848 [I 2022-10-28 07:58:36] ...optimizers/trijoint.py.151: total_norm: 31.982498 [I 2022-10-28 07:58:41] ...trap/engines/engine.py.224: train: epoch 114 | batch 1410/2304 [I 2022-10-28 07:58:41] ...trap/engines/engine.py.228: elapsed: 0:13:11 | left: 0:08:21 [I 2022-10-28 07:58:41] ...trap/engines/engine.py.229: process: 0.56501 | load: 0.00259 [I 2022-10-28 07:58:41] ...trap/engines/engine.py.230: loss: 0.09962 [I 2022-10-28 07:58:41] ...optimizers/trijoint.py.151: total_norm: 20.248566 [I 2022-10-28 07:58:47] ...trap/engines/engine.py.224: train: epoch 114 | batch 1420/2304 [I 2022-10-28 07:58:47] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:14 [I 2022-10-28 07:58:47] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00240 [I 2022-10-28 07:58:47] ...trap/engines/engine.py.230: loss: 0.09909 [I 2022-10-28 07:58:47] ...optimizers/trijoint.py.151: total_norm: 10.358435 [I 2022-10-28 07:58:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 1430/2304 [I 2022-10-28 07:58:53] ...trap/engines/engine.py.228: elapsed: 0:13:22 | left: 0:08:07 [I 2022-10-28 07:58:53] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00241 [I 2022-10-28 07:58:53] ...trap/engines/engine.py.230: loss: 0.13941 [I 2022-10-28 07:58:53] ...optimizers/trijoint.py.151: total_norm: 21.370392 [I 2022-10-28 07:58:58] ...trap/engines/engine.py.224: train: epoch 114 | batch 1440/2304 [I 2022-10-28 07:58:58] ...trap/engines/engine.py.228: elapsed: 0:13:28 | left: 0:08:00 [I 2022-10-28 07:58:58] ...trap/engines/engine.py.229: process: 0.55000 | load: 0.00241 [I 2022-10-28 07:58:58] ...trap/engines/engine.py.230: loss: 0.16491 [I 2022-10-28 07:58:58] ...optimizers/trijoint.py.151: total_norm: 8.978216 [I 2022-10-28 07:59:04] ...trap/engines/engine.py.224: train: epoch 114 | batch 1450/2304 [I 2022-10-28 07:59:04] ...trap/engines/engine.py.228: elapsed: 0:13:33 | left: 0:07:55 [I 2022-10-28 07:59:04] ...trap/engines/engine.py.229: process: 0.55162 | load: 0.00240 [I 2022-10-28 07:59:04] ...trap/engines/engine.py.230: loss: 0.14256 [I 2022-10-28 07:59:04] ...optimizers/trijoint.py.151: total_norm: 19.769300 [I 2022-10-28 07:59:09] ...trap/engines/engine.py.224: train: epoch 114 | batch 1460/2304 [I 2022-10-28 07:59:09] ...trap/engines/engine.py.228: elapsed: 0:13:39 | left: 0:07:49 [I 2022-10-28 07:59:09] ...trap/engines/engine.py.229: process: 0.55930 | load: 0.00251 [I 2022-10-28 07:59:09] ...trap/engines/engine.py.230: loss: 0.17258 [I 2022-10-28 07:59:09] ...optimizers/trijoint.py.151: total_norm: 14.021932 [I 2022-10-28 07:59:15] ...trap/engines/engine.py.224: train: epoch 114 | batch 1470/2304 [I 2022-10-28 07:59:15] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:48 [I 2022-10-28 07:59:15] ...trap/engines/engine.py.229: process: 0.56875 | load: 0.00261 [I 2022-10-28 07:59:15] ...trap/engines/engine.py.230: loss: 0.18340 [I 2022-10-28 07:59:15] ...optimizers/trijoint.py.151: total_norm: 12.369764 [I 2022-10-28 07:59:21] ...trap/engines/engine.py.224: train: epoch 114 | batch 1480/2304 [I 2022-10-28 07:59:21] ...trap/engines/engine.py.228: elapsed: 0:13:50 | left: 0:07:42 [I 2022-10-28 07:59:21] ...trap/engines/engine.py.229: process: 0.56565 | load: 0.00259 [I 2022-10-28 07:59:21] ...trap/engines/engine.py.230: loss: 0.12705 [I 2022-10-28 07:59:21] ...optimizers/trijoint.py.151: total_norm: 18.284834 [I 2022-10-28 07:59:26] ...trap/engines/engine.py.224: train: epoch 114 | batch 1490/2304 [I 2022-10-28 07:59:26] ...trap/engines/engine.py.228: elapsed: 0:13:56 | left: 0:07:39 [I 2022-10-28 07:59:26] ...trap/engines/engine.py.229: process: 0.54900 | load: 0.00247 [I 2022-10-28 07:59:26] ...trap/engines/engine.py.230: loss: 0.11811 [I 2022-10-28 07:59:26] ...optimizers/trijoint.py.151: total_norm: 24.942835 [I 2022-10-28 07:59:32] ...trap/engines/engine.py.224: train: epoch 114 | batch 1500/2304 [I 2022-10-28 07:59:32] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:29 [I 2022-10-28 07:59:32] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00245 [I 2022-10-28 07:59:32] ...trap/engines/engine.py.230: loss: 0.10247 [I 2022-10-28 07:59:32] ...optimizers/trijoint.py.151: total_norm: 9.183378 [I 2022-10-28 07:59:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 1510/2304 [I 2022-10-28 07:59:38] ...trap/engines/engine.py.228: elapsed: 0:14:07 | left: 0:07:24 [I 2022-10-28 07:59:38] ...trap/engines/engine.py.229: process: 0.56013 | load: 0.00244 [I 2022-10-28 07:59:38] ...trap/engines/engine.py.230: loss: 0.11160 [I 2022-10-28 07:59:38] ...optimizers/trijoint.py.151: total_norm: 16.717426 [I 2022-10-28 07:59:43] ...trap/engines/engine.py.224: train: epoch 114 | batch 1520/2304 [I 2022-10-28 07:59:43] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:17 [I 2022-10-28 07:59:43] ...trap/engines/engine.py.229: process: 0.56188 | load: 0.00248 [I 2022-10-28 07:59:43] ...trap/engines/engine.py.230: loss: 0.08908 [I 2022-10-28 07:59:43] ...optimizers/trijoint.py.151: total_norm: 18.284750 [I 2022-10-28 07:59:49] ...trap/engines/engine.py.224: train: epoch 114 | batch 1530/2304 [I 2022-10-28 07:59:49] ...trap/engines/engine.py.228: elapsed: 0:14:18 | left: 0:07:15 [I 2022-10-28 07:59:49] ...trap/engines/engine.py.229: process: 0.56919 | load: 0.00250 [I 2022-10-28 07:59:49] ...trap/engines/engine.py.230: loss: 0.09096 [I 2022-10-28 07:59:49] ...optimizers/trijoint.py.151: total_norm: 15.008447 [I 2022-10-28 07:59:54] ...trap/engines/engine.py.224: train: epoch 114 | batch 1540/2304 [I 2022-10-28 07:59:54] ...trap/engines/engine.py.228: elapsed: 0:14:24 | left: 0:07:08 [I 2022-10-28 07:59:54] ...trap/engines/engine.py.229: process: 0.57738 | load: 0.00253 [I 2022-10-28 07:59:54] ...trap/engines/engine.py.230: loss: 0.06472 [I 2022-10-28 07:59:54] ...optimizers/trijoint.py.151: total_norm: 13.053161 [I 2022-10-28 08:00:00] ...trap/engines/engine.py.224: train: epoch 114 | batch 1550/2304 [I 2022-10-28 08:00:00] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:06:58 [I 2022-10-28 08:00:00] ...trap/engines/engine.py.229: process: 0.54609 | load: 0.00247 [I 2022-10-28 08:00:00] ...trap/engines/engine.py.230: loss: 0.12124 [I 2022-10-28 08:00:00] ...optimizers/trijoint.py.151: total_norm: 15.707903 [I 2022-10-28 08:00:06] ...trap/engines/engine.py.224: train: epoch 114 | batch 1560/2304 [I 2022-10-28 08:00:06] ...trap/engines/engine.py.228: elapsed: 0:14:35 | left: 0:06:55 [I 2022-10-28 08:00:06] ...trap/engines/engine.py.229: process: 0.55956 | load: 0.00246 [I 2022-10-28 08:00:06] ...trap/engines/engine.py.230: loss: 0.09930 [I 2022-10-28 08:00:06] ...optimizers/trijoint.py.151: total_norm: 27.224476 [I 2022-10-28 08:00:11] ...trap/engines/engine.py.224: train: epoch 114 | batch 1570/2304 [I 2022-10-28 08:00:11] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:49 [I 2022-10-28 08:00:11] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00245 [I 2022-10-28 08:00:11] ...trap/engines/engine.py.230: loss: 0.10042 [I 2022-10-28 08:00:11] ...optimizers/trijoint.py.151: total_norm: 9.883790 [I 2022-10-28 08:00:17] ...trap/engines/engine.py.224: train: epoch 114 | batch 1580/2304 [I 2022-10-28 08:00:17] ...trap/engines/engine.py.228: elapsed: 0:14:46 | left: 0:06:44 [I 2022-10-28 08:00:17] ...trap/engines/engine.py.229: process: 0.56295 | load: 0.00245 [I 2022-10-28 08:00:17] ...trap/engines/engine.py.230: loss: 0.13999 [I 2022-10-28 08:00:17] ...optimizers/trijoint.py.151: total_norm: 18.826805 [I 2022-10-28 08:00:22] ...trap/engines/engine.py.224: train: epoch 114 | batch 1590/2304 [I 2022-10-28 08:00:22] ...trap/engines/engine.py.228: elapsed: 0:14:52 | left: 0:06:39 [I 2022-10-28 08:00:22] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00243 [I 2022-10-28 08:00:22] ...trap/engines/engine.py.230: loss: 0.14918 [I 2022-10-28 08:00:22] ...optimizers/trijoint.py.151: total_norm: 24.458084 [I 2022-10-28 08:00:28] ...trap/engines/engine.py.224: train: epoch 114 | batch 1600/2304 [I 2022-10-28 08:00:28] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:32 [I 2022-10-28 08:00:28] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00244 [I 2022-10-28 08:00:28] ...trap/engines/engine.py.230: loss: 0.16482 [I 2022-10-28 08:00:28] ...optimizers/trijoint.py.151: total_norm: 16.073404 [I 2022-10-28 08:00:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 1610/2304 [I 2022-10-28 08:00:34] ...trap/engines/engine.py.228: elapsed: 0:15:03 | left: 0:06:33 [I 2022-10-28 08:00:34] ...trap/engines/engine.py.229: process: 0.55057 | load: 0.00245 [I 2022-10-28 08:00:34] ...trap/engines/engine.py.230: loss: 0.08897 [I 2022-10-28 08:00:34] ...optimizers/trijoint.py.151: total_norm: 13.682107 [I 2022-10-28 08:00:39] ...trap/engines/engine.py.224: train: epoch 114 | batch 1620/2304 [I 2022-10-28 08:00:39] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:19 [I 2022-10-28 08:00:39] ...trap/engines/engine.py.229: process: 0.54844 | load: 0.00240 [I 2022-10-28 08:00:39] ...trap/engines/engine.py.230: loss: 0.10082 [I 2022-10-28 08:00:39] ...optimizers/trijoint.py.151: total_norm: 10.593632 [I 2022-10-28 08:00:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 1630/2304 [I 2022-10-28 08:00:45] ...trap/engines/engine.py.228: elapsed: 0:15:14 | left: 0:06:16 [I 2022-10-28 08:00:45] ...trap/engines/engine.py.229: process: 0.56571 | load: 0.00241 [I 2022-10-28 08:00:45] ...trap/engines/engine.py.230: loss: 0.08237 [I 2022-10-28 08:00:45] ...optimizers/trijoint.py.151: total_norm: 11.573201 [I 2022-10-28 08:00:50] ...trap/engines/engine.py.224: train: epoch 114 | batch 1640/2304 [I 2022-10-28 08:00:50] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:11 [I 2022-10-28 08:00:50] ...trap/engines/engine.py.229: process: 0.56744 | load: 0.00243 [I 2022-10-28 08:00:50] ...trap/engines/engine.py.230: loss: 0.09558 [I 2022-10-28 08:00:50] ...optimizers/trijoint.py.151: total_norm: 25.286949 [I 2022-10-28 08:00:56] ...trap/engines/engine.py.224: train: epoch 114 | batch 1650/2304 [I 2022-10-28 08:00:56] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:03 [I 2022-10-28 08:00:56] ...trap/engines/engine.py.229: process: 0.54533 | load: 0.00248 [I 2022-10-28 08:00:56] ...trap/engines/engine.py.230: loss: 0.13165 [I 2022-10-28 08:00:56] ...optimizers/trijoint.py.151: total_norm: 19.533039 [I 2022-10-28 08:01:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 1660/2304 [I 2022-10-28 08:01:02] ...trap/engines/engine.py.228: elapsed: 0:15:31 | left: 0:06:03 [I 2022-10-28 08:01:02] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00247 [I 2022-10-28 08:01:02] ...trap/engines/engine.py.230: loss: 0.12979 [I 2022-10-28 08:01:02] ...optimizers/trijoint.py.151: total_norm: 28.485430 [I 2022-10-28 08:01:07] ...trap/engines/engine.py.224: train: epoch 114 | batch 1670/2304 [I 2022-10-28 08:01:07] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:05:53 [I 2022-10-28 08:01:07] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00243 [I 2022-10-28 08:01:07] ...trap/engines/engine.py.230: loss: 0.10012 [I 2022-10-28 08:01:07] ...optimizers/trijoint.py.151: total_norm: 13.005619 [I 2022-10-28 08:01:13] ...trap/engines/engine.py.224: train: epoch 114 | batch 1680/2304 [I 2022-10-28 08:01:13] ...trap/engines/engine.py.228: elapsed: 0:15:42 | left: 0:05:50 [I 2022-10-28 08:01:13] ...trap/engines/engine.py.229: process: 0.56508 | load: 0.00242 [I 2022-10-28 08:01:13] ...trap/engines/engine.py.230: loss: 0.11678 [I 2022-10-28 08:01:13] ...optimizers/trijoint.py.151: total_norm: 13.689189 [I 2022-10-28 08:01:18] ...trap/engines/engine.py.224: train: epoch 114 | batch 1690/2304 [I 2022-10-28 08:01:18] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:41 [I 2022-10-28 08:01:18] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00246 [I 2022-10-28 08:01:18] ...trap/engines/engine.py.230: loss: 0.11469 [I 2022-10-28 08:01:18] ...optimizers/trijoint.py.151: total_norm: 14.123891 [I 2022-10-28 08:01:24] ...trap/engines/engine.py.224: train: epoch 114 | batch 1700/2304 [I 2022-10-28 08:01:24] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:35 [I 2022-10-28 08:01:24] ...trap/engines/engine.py.229: process: 0.55052 | load: 0.00248 [I 2022-10-28 08:01:24] ...trap/engines/engine.py.230: loss: 0.16578 [I 2022-10-28 08:01:24] ...optimizers/trijoint.py.151: total_norm: 21.497398 [I 2022-10-28 08:01:30] ...trap/engines/engine.py.224: train: epoch 114 | batch 1710/2304 [I 2022-10-28 08:01:30] ...trap/engines/engine.py.228: elapsed: 0:15:59 | left: 0:05:30 [I 2022-10-28 08:01:30] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00243 [I 2022-10-28 08:01:30] ...trap/engines/engine.py.230: loss: 0.11774 [I 2022-10-28 08:01:30] ...optimizers/trijoint.py.151: total_norm: 16.026627 [I 2022-10-28 08:01:35] ...trap/engines/engine.py.224: train: epoch 114 | batch 1720/2304 [I 2022-10-28 08:01:35] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:26 [I 2022-10-28 08:01:35] ...trap/engines/engine.py.229: process: 0.55771 | load: 0.00250 [I 2022-10-28 08:01:35] ...trap/engines/engine.py.230: loss: 0.10747 [I 2022-10-28 08:01:35] ...optimizers/trijoint.py.151: total_norm: 17.132017 [I 2022-10-28 08:01:41] ...trap/engines/engine.py.224: train: epoch 114 | batch 1730/2304 [I 2022-10-28 08:01:41] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:27 [I 2022-10-28 08:01:41] ...trap/engines/engine.py.229: process: 0.57517 | load: 0.00249 [I 2022-10-28 08:01:41] ...trap/engines/engine.py.230: loss: 0.11462 [I 2022-10-28 08:01:41] ...optimizers/trijoint.py.151: total_norm: 21.991154 [I 2022-10-28 08:01:47] ...trap/engines/engine.py.224: train: epoch 114 | batch 1740/2304 [I 2022-10-28 08:01:47] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:16 [I 2022-10-28 08:01:47] ...trap/engines/engine.py.229: process: 0.56768 | load: 0.00256 [I 2022-10-28 08:01:47] ...trap/engines/engine.py.230: loss: 0.16503 [I 2022-10-28 08:01:47] ...optimizers/trijoint.py.151: total_norm: 61.151596 [I 2022-10-28 08:01:52] ...trap/engines/engine.py.224: train: epoch 114 | batch 1750/2304 [I 2022-10-28 08:01:52] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:07 [I 2022-10-28 08:01:52] ...trap/engines/engine.py.229: process: 0.55444 | load: 0.00247 [I 2022-10-28 08:01:52] ...trap/engines/engine.py.230: loss: 0.10316 [I 2022-10-28 08:01:52] ...optimizers/trijoint.py.151: total_norm: 15.823586 [I 2022-10-28 08:01:58] ...trap/engines/engine.py.224: train: epoch 114 | batch 1760/2304 [I 2022-10-28 08:01:58] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:04 [I 2022-10-28 08:01:58] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00247 [I 2022-10-28 08:01:58] ...trap/engines/engine.py.230: loss: 0.09697 [I 2022-10-28 08:01:58] ...optimizers/trijoint.py.151: total_norm: 7.521537 [I 2022-10-28 08:02:03] ...trap/engines/engine.py.224: train: epoch 114 | batch 1770/2304 [I 2022-10-28 08:02:03] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:04:57 [I 2022-10-28 08:02:03] ...trap/engines/engine.py.229: process: 0.55332 | load: 0.00245 [I 2022-10-28 08:02:03] ...trap/engines/engine.py.230: loss: 0.07182 [I 2022-10-28 08:02:03] ...optimizers/trijoint.py.151: total_norm: 9.526617 [I 2022-10-28 08:02:09] ...trap/engines/engine.py.224: train: epoch 114 | batch 1780/2304 [I 2022-10-28 08:02:09] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:04:51 [I 2022-10-28 08:02:09] ...trap/engines/engine.py.229: process: 0.54728 | load: 0.00242 [I 2022-10-28 08:02:09] ...trap/engines/engine.py.230: loss: 0.16544 [I 2022-10-28 08:02:09] ...optimizers/trijoint.py.151: total_norm: 24.660334 [I 2022-10-28 08:02:14] ...trap/engines/engine.py.224: train: epoch 114 | batch 1790/2304 [I 2022-10-28 08:02:14] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:46 [I 2022-10-28 08:02:14] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00244 [I 2022-10-28 08:02:14] ...trap/engines/engine.py.230: loss: 0.13385 [I 2022-10-28 08:02:14] ...optimizers/trijoint.py.151: total_norm: 11.930358 [I 2022-10-28 08:02:20] ...trap/engines/engine.py.224: train: epoch 114 | batch 1800/2304 [I 2022-10-28 08:02:20] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:41 [I 2022-10-28 08:02:20] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00241 [I 2022-10-28 08:02:20] ...trap/engines/engine.py.230: loss: 0.14326 [I 2022-10-28 08:02:20] ...optimizers/trijoint.py.151: total_norm: 10.746037 [I 2022-10-28 08:02:26] ...trap/engines/engine.py.224: train: epoch 114 | batch 1810/2304 [I 2022-10-28 08:02:26] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:35 [I 2022-10-28 08:02:26] ...trap/engines/engine.py.229: process: 0.54810 | load: 0.00244 [I 2022-10-28 08:02:26] ...trap/engines/engine.py.230: loss: 0.13931 [I 2022-10-28 08:02:26] ...optimizers/trijoint.py.151: total_norm: 23.204823 [I 2022-10-28 08:02:31] ...trap/engines/engine.py.224: train: epoch 114 | batch 1820/2304 [I 2022-10-28 08:02:31] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:30 [I 2022-10-28 08:02:31] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00242 [I 2022-10-28 08:02:31] ...trap/engines/engine.py.230: loss: 0.11419 [I 2022-10-28 08:02:31] ...optimizers/trijoint.py.151: total_norm: 17.370201 [I 2022-10-28 08:02:37] ...trap/engines/engine.py.224: train: epoch 114 | batch 1830/2304 [I 2022-10-28 08:02:37] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:25 [I 2022-10-28 08:02:37] ...trap/engines/engine.py.229: process: 0.56050 | load: 0.00244 [I 2022-10-28 08:02:37] ...trap/engines/engine.py.230: loss: 0.13554 [I 2022-10-28 08:02:37] ...optimizers/trijoint.py.151: total_norm: 87.589897 [I 2022-10-28 08:02:42] ...trap/engines/engine.py.224: train: epoch 114 | batch 1840/2304 [I 2022-10-28 08:02:42] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:17 [I 2022-10-28 08:02:42] ...trap/engines/engine.py.229: process: 0.53974 | load: 0.00250 [I 2022-10-28 08:02:42] ...trap/engines/engine.py.230: loss: 0.11222 [I 2022-10-28 08:02:42] ...optimizers/trijoint.py.151: total_norm: 17.449757 [I 2022-10-28 08:02:48] ...trap/engines/engine.py.224: train: epoch 114 | batch 1850/2304 [I 2022-10-28 08:02:48] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:13 [I 2022-10-28 08:02:48] ...trap/engines/engine.py.229: process: 0.55858 | load: 0.00248 [I 2022-10-28 08:02:48] ...trap/engines/engine.py.230: loss: 0.12319 [I 2022-10-28 08:02:48] ...optimizers/trijoint.py.151: total_norm: 16.881390 [I 2022-10-28 08:02:54] ...trap/engines/engine.py.224: train: epoch 114 | batch 1860/2304 [I 2022-10-28 08:02:54] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:09 [I 2022-10-28 08:02:54] ...trap/engines/engine.py.229: process: 0.56424 | load: 0.00246 [I 2022-10-28 08:02:54] ...trap/engines/engine.py.230: loss: 0.11460 [I 2022-10-28 08:02:54] ...optimizers/trijoint.py.151: total_norm: 19.057487 [I 2022-10-28 08:02:59] ...trap/engines/engine.py.224: train: epoch 114 | batch 1870/2304 [I 2022-10-28 08:02:59] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:01 [I 2022-10-28 08:02:59] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00244 [I 2022-10-28 08:02:59] ...trap/engines/engine.py.230: loss: 0.09854 [I 2022-10-28 08:02:59] ...optimizers/trijoint.py.151: total_norm: 12.348236 [I 2022-10-28 08:03:05] ...trap/engines/engine.py.224: train: epoch 114 | batch 1880/2304 [I 2022-10-28 08:03:05] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:03:55 [I 2022-10-28 08:03:05] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00243 [I 2022-10-28 08:03:05] ...trap/engines/engine.py.230: loss: 0.15507 [I 2022-10-28 08:03:05] ...optimizers/trijoint.py.151: total_norm: 21.386730 [I 2022-10-28 08:03:10] ...trap/engines/engine.py.224: train: epoch 114 | batch 1890/2304 [I 2022-10-28 08:03:10] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:03:50 [I 2022-10-28 08:03:10] ...trap/engines/engine.py.229: process: 0.54840 | load: 0.00241 [I 2022-10-28 08:03:10] ...trap/engines/engine.py.230: loss: 0.11124 [I 2022-10-28 08:03:10] ...optimizers/trijoint.py.151: total_norm: 13.154551 [I 2022-10-28 08:03:16] ...trap/engines/engine.py.224: train: epoch 114 | batch 1900/2304 [I 2022-10-28 08:03:16] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:45 [I 2022-10-28 08:03:16] ...trap/engines/engine.py.229: process: 0.54819 | load: 0.00251 [I 2022-10-28 08:03:16] ...trap/engines/engine.py.230: loss: 0.14078 [I 2022-10-28 08:03:16] ...optimizers/trijoint.py.151: total_norm: 11.361924 [I 2022-10-28 08:03:22] ...trap/engines/engine.py.224: train: epoch 114 | batch 1910/2304 [I 2022-10-28 08:03:22] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:41 [I 2022-10-28 08:03:22] ...trap/engines/engine.py.229: process: 0.57829 | load: 0.00249 [I 2022-10-28 08:03:22] ...trap/engines/engine.py.230: loss: 0.08949 [I 2022-10-28 08:03:22] ...optimizers/trijoint.py.151: total_norm: 14.024266 [I 2022-10-28 08:03:27] ...trap/engines/engine.py.224: train: epoch 114 | batch 1920/2304 [I 2022-10-28 08:03:27] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:34 [I 2022-10-28 08:03:27] ...trap/engines/engine.py.229: process: 0.56481 | load: 0.00246 [I 2022-10-28 08:03:27] ...trap/engines/engine.py.230: loss: 0.12096 [I 2022-10-28 08:03:27] ...optimizers/trijoint.py.151: total_norm: 16.669977 [I 2022-10-28 08:03:33] ...trap/engines/engine.py.224: train: epoch 114 | batch 1930/2304 [I 2022-10-28 08:03:33] ...trap/engines/engine.py.228: elapsed: 0:18:02 | left: 0:03:28 [I 2022-10-28 08:03:33] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00243 [I 2022-10-28 08:03:33] ...trap/engines/engine.py.230: loss: 0.21225 [I 2022-10-28 08:03:33] ...optimizers/trijoint.py.151: total_norm: 17.088072 [I 2022-10-28 08:03:38] ...trap/engines/engine.py.224: train: epoch 114 | batch 1940/2304 [I 2022-10-28 08:03:38] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:24 [I 2022-10-28 08:03:38] ...trap/engines/engine.py.229: process: 0.56629 | load: 0.00245 [I 2022-10-28 08:03:38] ...trap/engines/engine.py.230: loss: 0.09549 [I 2022-10-28 08:03:38] ...optimizers/trijoint.py.151: total_norm: 31.008865 [I 2022-10-28 08:03:44] ...trap/engines/engine.py.224: train: epoch 114 | batch 1950/2304 [I 2022-10-28 08:03:44] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:16 [I 2022-10-28 08:03:44] ...trap/engines/engine.py.229: process: 0.54960 | load: 0.00243 [I 2022-10-28 08:03:44] ...trap/engines/engine.py.230: loss: 0.11017 [I 2022-10-28 08:03:44] ...optimizers/trijoint.py.151: total_norm: 15.311217 [I 2022-10-28 08:03:50] ...trap/engines/engine.py.224: train: epoch 114 | batch 1960/2304 [I 2022-10-28 08:03:50] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:11 [I 2022-10-28 08:03:50] ...trap/engines/engine.py.229: process: 0.56506 | load: 0.00257 [I 2022-10-28 08:03:50] ...trap/engines/engine.py.230: loss: 0.11556 [I 2022-10-28 08:03:50] ...optimizers/trijoint.py.151: total_norm: 28.056242 [I 2022-10-28 08:03:55] ...trap/engines/engine.py.224: train: epoch 114 | batch 1970/2304 [I 2022-10-28 08:03:55] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:05 [I 2022-10-28 08:03:55] ...trap/engines/engine.py.229: process: 0.55155 | load: 0.00242 [I 2022-10-28 08:03:55] ...trap/engines/engine.py.230: loss: 0.14331 [I 2022-10-28 08:03:55] ...optimizers/trijoint.py.151: total_norm: 23.678778 [I 2022-10-28 08:04:01] ...trap/engines/engine.py.224: train: epoch 114 | batch 1980/2304 [I 2022-10-28 08:04:01] ...trap/engines/engine.py.228: elapsed: 0:18:30 | left: 0:03:01 [I 2022-10-28 08:04:01] ...trap/engines/engine.py.229: process: 0.57630 | load: 0.00243 [I 2022-10-28 08:04:01] ...trap/engines/engine.py.230: loss: 0.11087 [I 2022-10-28 08:04:01] ...optimizers/trijoint.py.151: total_norm: 36.970875 [I 2022-10-28 08:04:06] ...trap/engines/engine.py.224: train: epoch 114 | batch 1990/2304 [I 2022-10-28 08:04:06] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:02:53 [I 2022-10-28 08:04:06] ...trap/engines/engine.py.229: process: 0.54128 | load: 0.00242 [I 2022-10-28 08:04:06] ...trap/engines/engine.py.230: loss: 0.11770 [I 2022-10-28 08:04:06] ...optimizers/trijoint.py.151: total_norm: 22.592587 [I 2022-10-28 08:04:12] ...trap/engines/engine.py.224: train: epoch 114 | batch 2000/2304 [I 2022-10-28 08:04:12] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:50 [I 2022-10-28 08:04:12] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00242 [I 2022-10-28 08:04:12] ...trap/engines/engine.py.230: loss: 0.10517 [I 2022-10-28 08:04:12] ...optimizers/trijoint.py.151: total_norm: 12.904892 [I 2022-10-28 08:04:18] ...trap/engines/engine.py.224: train: epoch 114 | batch 2010/2304 [I 2022-10-28 08:04:18] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:02:44 [I 2022-10-28 08:04:18] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00250 [I 2022-10-28 08:04:18] ...trap/engines/engine.py.230: loss: 0.11544 [I 2022-10-28 08:04:18] ...optimizers/trijoint.py.151: total_norm: 15.701162 [I 2022-10-28 08:04:23] ...trap/engines/engine.py.224: train: epoch 114 | batch 2020/2304 [I 2022-10-28 08:04:23] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:38 [I 2022-10-28 08:04:23] ...trap/engines/engine.py.229: process: 0.54701 | load: 0.00250 [I 2022-10-28 08:04:23] ...trap/engines/engine.py.230: loss: 0.08010 [I 2022-10-28 08:04:23] ...optimizers/trijoint.py.151: total_norm: 12.734509 [I 2022-10-28 08:04:29] ...trap/engines/engine.py.224: train: epoch 114 | batch 2030/2304 [I 2022-10-28 08:04:29] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:33 [I 2022-10-28 08:04:29] ...trap/engines/engine.py.229: process: 0.56606 | load: 0.00246 [I 2022-10-28 08:04:29] ...trap/engines/engine.py.230: loss: 0.14378 [I 2022-10-28 08:04:29] ...optimizers/trijoint.py.151: total_norm: 25.937502 [I 2022-10-28 08:04:34] ...trap/engines/engine.py.224: train: epoch 114 | batch 2040/2304 [I 2022-10-28 08:04:34] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:26 [I 2022-10-28 08:04:34] ...trap/engines/engine.py.229: process: 0.55635 | load: 0.00240 [I 2022-10-28 08:04:34] ...trap/engines/engine.py.230: loss: 0.09944 [I 2022-10-28 08:04:34] ...optimizers/trijoint.py.151: total_norm: 11.240578 [I 2022-10-28 08:04:40] ...trap/engines/engine.py.224: train: epoch 114 | batch 2050/2304 [I 2022-10-28 08:04:40] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:21 [I 2022-10-28 08:04:40] ...trap/engines/engine.py.229: process: 0.54918 | load: 0.00245 [I 2022-10-28 08:04:40] ...trap/engines/engine.py.230: loss: 0.11441 [I 2022-10-28 08:04:40] ...optimizers/trijoint.py.151: total_norm: 11.799314 [I 2022-10-28 08:04:45] ...trap/engines/engine.py.224: train: epoch 114 | batch 2060/2304 [I 2022-10-28 08:04:45] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:14 [I 2022-10-28 08:04:45] ...trap/engines/engine.py.229: process: 0.55165 | load: 0.00243 [I 2022-10-28 08:04:45] ...trap/engines/engine.py.230: loss: 0.09347 [I 2022-10-28 08:04:45] ...optimizers/trijoint.py.151: total_norm: 11.775834 [I 2022-10-28 08:04:51] ...trap/engines/engine.py.224: train: epoch 114 | batch 2070/2304 [I 2022-10-28 08:04:51] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:10 [I 2022-10-28 08:04:51] ...trap/engines/engine.py.229: process: 0.56037 | load: 0.00242 [I 2022-10-28 08:04:51] ...trap/engines/engine.py.230: loss: 0.08554 [I 2022-10-28 08:04:51] ...optimizers/trijoint.py.151: total_norm: 19.380316 [I 2022-10-28 08:04:57] ...trap/engines/engine.py.224: train: epoch 114 | batch 2080/2304 [I 2022-10-28 08:04:57] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:03 [I 2022-10-28 08:04:57] ...trap/engines/engine.py.229: process: 0.54781 | load: 0.00240 [I 2022-10-28 08:04:57] ...trap/engines/engine.py.230: loss: 0.12386 [I 2022-10-28 08:04:57] ...optimizers/trijoint.py.151: total_norm: 11.535179 [I 2022-10-28 08:05:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 2090/2304 [I 2022-10-28 08:05:02] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:01:58 [I 2022-10-28 08:05:02] ...trap/engines/engine.py.229: process: 0.55129 | load: 0.00242 [I 2022-10-28 08:05:02] ...trap/engines/engine.py.230: loss: 0.07062 [I 2022-10-28 08:05:02] ...optimizers/trijoint.py.151: total_norm: 8.853881 [I 2022-10-28 08:05:08] ...trap/engines/engine.py.224: train: epoch 114 | batch 2100/2304 [I 2022-10-28 08:05:08] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:01:54 [I 2022-10-28 08:05:08] ...trap/engines/engine.py.229: process: 0.56179 | load: 0.00244 [I 2022-10-28 08:05:08] ...trap/engines/engine.py.230: loss: 0.11305 [I 2022-10-28 08:05:08] ...optimizers/trijoint.py.151: total_norm: 10.619983 [I 2022-10-28 08:05:13] ...trap/engines/engine.py.224: train: epoch 114 | batch 2110/2304 [I 2022-10-28 08:05:13] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:01:48 [I 2022-10-28 08:05:13] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00253 [I 2022-10-28 08:05:13] ...trap/engines/engine.py.230: loss: 0.09719 [I 2022-10-28 08:05:13] ...optimizers/trijoint.py.151: total_norm: 11.886414 [I 2022-10-28 08:05:19] ...trap/engines/engine.py.224: train: epoch 114 | batch 2120/2304 [I 2022-10-28 08:05:19] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:43 [I 2022-10-28 08:05:19] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00266 [I 2022-10-28 08:05:19] ...trap/engines/engine.py.230: loss: 0.10194 [I 2022-10-28 08:05:19] ...optimizers/trijoint.py.151: total_norm: 18.929260 [I 2022-10-28 08:05:24] ...trap/engines/engine.py.224: train: epoch 114 | batch 2130/2304 [I 2022-10-28 08:05:24] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:37 [I 2022-10-28 08:05:24] ...trap/engines/engine.py.229: process: 0.55919 | load: 0.00247 [I 2022-10-28 08:05:24] ...trap/engines/engine.py.230: loss: 0.16639 [I 2022-10-28 08:05:24] ...optimizers/trijoint.py.151: total_norm: 18.042160 [I 2022-10-28 08:05:30] ...trap/engines/engine.py.224: train: epoch 114 | batch 2140/2304 [I 2022-10-28 08:05:30] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:31 [I 2022-10-28 08:05:30] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00248 [I 2022-10-28 08:05:30] ...trap/engines/engine.py.230: loss: 0.09614 [I 2022-10-28 08:05:30] ...optimizers/trijoint.py.151: total_norm: 13.824501 [I 2022-10-28 08:05:36] ...trap/engines/engine.py.224: train: epoch 114 | batch 2150/2304 [I 2022-10-28 08:05:36] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:26 [I 2022-10-28 08:05:36] ...trap/engines/engine.py.229: process: 0.57254 | load: 0.00246 [I 2022-10-28 08:05:36] ...trap/engines/engine.py.230: loss: 0.16114 [I 2022-10-28 08:05:36] ...optimizers/trijoint.py.151: total_norm: 12.895295 [I 2022-10-28 08:05:41] ...trap/engines/engine.py.224: train: epoch 114 | batch 2160/2304 [I 2022-10-28 08:05:41] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:21 [I 2022-10-28 08:05:41] ...trap/engines/engine.py.229: process: 0.56835 | load: 0.00243 [I 2022-10-28 08:05:41] ...trap/engines/engine.py.230: loss: 0.16507 [I 2022-10-28 08:05:41] ...optimizers/trijoint.py.151: total_norm: 28.132553 [I 2022-10-28 08:05:47] ...trap/engines/engine.py.224: train: epoch 114 | batch 2170/2304 [I 2022-10-28 08:05:47] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:14 [I 2022-10-28 08:05:47] ...trap/engines/engine.py.229: process: 0.54916 | load: 0.00246 [I 2022-10-28 08:05:47] ...trap/engines/engine.py.230: loss: 0.06647 [I 2022-10-28 08:05:47] ...optimizers/trijoint.py.151: total_norm: 11.776508 [I 2022-10-28 08:05:53] ...trap/engines/engine.py.224: train: epoch 114 | batch 2180/2304 [I 2022-10-28 08:05:53] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:09 [I 2022-10-28 08:05:53] ...trap/engines/engine.py.229: process: 0.56067 | load: 0.00242 [I 2022-10-28 08:05:53] ...trap/engines/engine.py.230: loss: 0.09530 [I 2022-10-28 08:05:53] ...optimizers/trijoint.py.151: total_norm: 12.657317 [I 2022-10-28 08:05:58] ...trap/engines/engine.py.224: train: epoch 114 | batch 2190/2304 [I 2022-10-28 08:05:58] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:03 [I 2022-10-28 08:05:58] ...trap/engines/engine.py.229: process: 0.56512 | load: 0.00250 [I 2022-10-28 08:05:58] ...trap/engines/engine.py.230: loss: 0.14130 [I 2022-10-28 08:05:58] ...optimizers/trijoint.py.151: total_norm: 12.828962 [I 2022-10-28 08:06:04] ...trap/engines/engine.py.224: train: epoch 114 | batch 2200/2304 [I 2022-10-28 08:06:04] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:58 [I 2022-10-28 08:06:04] ...trap/engines/engine.py.229: process: 0.55989 | load: 0.00241 [I 2022-10-28 08:06:04] ...trap/engines/engine.py.230: loss: 0.13118 [I 2022-10-28 08:06:04] ...optimizers/trijoint.py.151: total_norm: 22.112190 [I 2022-10-28 08:06:09] ...trap/engines/engine.py.224: train: epoch 114 | batch 2210/2304 [I 2022-10-28 08:06:09] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:00:52 [I 2022-10-28 08:06:09] ...trap/engines/engine.py.229: process: 0.55614 | load: 0.00304 [I 2022-10-28 08:06:09] ...trap/engines/engine.py.230: loss: 0.15281 [I 2022-10-28 08:06:09] ...optimizers/trijoint.py.151: total_norm: 11.512216 [I 2022-10-28 08:06:15] ...trap/engines/engine.py.224: train: epoch 114 | batch 2220/2304 [I 2022-10-28 08:06:15] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:00:46 [I 2022-10-28 08:06:15] ...trap/engines/engine.py.229: process: 0.55197 | load: 0.00242 [I 2022-10-28 08:06:15] ...trap/engines/engine.py.230: loss: 0.08970 [I 2022-10-28 08:06:15] ...optimizers/trijoint.py.151: total_norm: 10.894176 [I 2022-10-28 08:06:20] ...trap/engines/engine.py.224: train: epoch 114 | batch 2230/2304 [I 2022-10-28 08:06:20] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:40 [I 2022-10-28 08:06:20] ...trap/engines/engine.py.229: process: 0.53837 | load: 0.00254 [I 2022-10-28 08:06:20] ...trap/engines/engine.py.230: loss: 0.09054 [I 2022-10-28 08:06:20] ...optimizers/trijoint.py.151: total_norm: 12.538307 [I 2022-10-28 08:06:26] ...trap/engines/engine.py.224: train: epoch 114 | batch 2240/2304 [I 2022-10-28 08:06:26] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:35 [I 2022-10-28 08:06:26] ...trap/engines/engine.py.229: process: 0.56258 | load: 0.00247 [I 2022-10-28 08:06:26] ...trap/engines/engine.py.230: loss: 0.10607 [I 2022-10-28 08:06:26] ...optimizers/trijoint.py.151: total_norm: 13.629347 [I 2022-10-28 08:06:32] ...trap/engines/engine.py.224: train: epoch 114 | batch 2250/2304 [I 2022-10-28 08:06:32] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:30 [I 2022-10-28 08:06:32] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00246 [I 2022-10-28 08:06:32] ...trap/engines/engine.py.230: loss: 0.11175 [I 2022-10-28 08:06:32] ...optimizers/trijoint.py.151: total_norm: 14.635557 [I 2022-10-28 08:06:37] ...trap/engines/engine.py.224: train: epoch 114 | batch 2260/2304 [I 2022-10-28 08:06:37] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:24 [I 2022-10-28 08:06:37] ...trap/engines/engine.py.229: process: 0.54684 | load: 0.00246 [I 2022-10-28 08:06:37] ...trap/engines/engine.py.230: loss: 0.11655 [I 2022-10-28 08:06:37] ...optimizers/trijoint.py.151: total_norm: 24.622780 [I 2022-10-28 08:06:43] ...trap/engines/engine.py.224: train: epoch 114 | batch 2270/2304 [I 2022-10-28 08:06:43] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:18 [I 2022-10-28 08:06:43] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00245 [I 2022-10-28 08:06:43] ...trap/engines/engine.py.230: loss: 0.14173 [I 2022-10-28 08:06:43] ...optimizers/trijoint.py.151: total_norm: 52.645168 [I 2022-10-28 08:06:48] ...trap/engines/engine.py.224: train: epoch 114 | batch 2280/2304 [I 2022-10-28 08:06:48] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:13 [I 2022-10-28 08:06:48] ...trap/engines/engine.py.229: process: 0.56857 | load: 0.00244 [I 2022-10-28 08:06:48] ...trap/engines/engine.py.230: loss: 0.10555 [I 2022-10-28 08:06:48] ...optimizers/trijoint.py.151: total_norm: 14.453868 [I 2022-10-28 08:06:54] ...trap/engines/engine.py.224: train: epoch 114 | batch 2290/2304 [I 2022-10-28 08:06:54] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:07 [I 2022-10-28 08:06:54] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00243 [I 2022-10-28 08:06:54] ...trap/engines/engine.py.230: loss: 0.11739 [I 2022-10-28 08:06:54] ...optimizers/trijoint.py.151: total_norm: 66.970802 [I 2022-10-28 08:07:00] ...trap/engines/engine.py.224: train: epoch 114 | batch 2300/2304 [I 2022-10-28 08:07:00] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:02 [I 2022-10-28 08:07:00] ...trap/engines/engine.py.229: process: 0.55728 | load: 0.00017 [I 2022-10-28 08:07:00] ...trap/engines/engine.py.230: loss: 0.14772 [I 2022-10-28 08:07:00] ...optimizers/trijoint.py.151: total_norm: 14.733717 [I 2022-10-28 08:07:02] ...trap/engines/engine.py.224: train: epoch 114 | batch 2304/2304 [I 2022-10-28 08:07:02] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:00 [I 2022-10-28 08:07:02] ...trap/engines/engine.py.229: process: 0.57146 | load: 0.00016 [I 2022-10-28 08:07:02] ...trap/engines/engine.py.230: loss: 0.13880 [I 2022-10-28 08:07:02] ...optimizers/trijoint.py.151: total_norm: 16.760973 [S 2022-10-28 08:07:03] ...trap/engines/engine.py.129: train_epoch.epoch: 114 [S 2022-10-28 08:07:03] ...trap/engines/engine.py.129: train_epoch.loss: 0.1203283699467746 [I 2022-10-28 08:07:03] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 08:07:05] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 114 [I 2022-10-28 08:07:05] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 08:07:07] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 08:07:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 08:07:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 08:07:11] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 08:07:11] ...trap/engines/engine.py.321: eval: epoch 114 | batch 0/510 [I 2022-10-28 08:07:11] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:52:24 [I 2022-10-28 08:07:11] ...trap/engines/engine.py.326: process: 6.16480 | load: 3.16282 [I 2022-10-28 08:07:14] ...trap/engines/engine.py.321: eval: epoch 114 | batch 10/510 [I 2022-10-28 08:07:14] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:02 [I 2022-10-28 08:07:14] ...trap/engines/engine.py.326: process: 0.32366 | load: 0.00017 [I 2022-10-28 08:07:18] ...trap/engines/engine.py.321: eval: epoch 114 | batch 20/510 [I 2022-10-28 08:07:18] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:16 [I 2022-10-28 08:07:18] ...trap/engines/engine.py.326: process: 0.32407 | load: 0.00018 [I 2022-10-28 08:07:21] ...trap/engines/engine.py.321: eval: epoch 114 | batch 30/510 [I 2022-10-28 08:07:21] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:42 [I 2022-10-28 08:07:21] ...trap/engines/engine.py.326: process: 0.33308 | load: 0.00018 [I 2022-10-28 08:07:24] ...trap/engines/engine.py.321: eval: epoch 114 | batch 40/510 [I 2022-10-28 08:07:24] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-28 08:07:24] ...trap/engines/engine.py.326: process: 0.33526 | load: 0.00019 [I 2022-10-28 08:07:28] ...trap/engines/engine.py.321: eval: epoch 114 | batch 50/510 [I 2022-10-28 08:07:28] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:34 [I 2022-10-28 08:07:28] ...trap/engines/engine.py.326: process: 0.33643 | load: 0.00027 [I 2022-10-28 08:07:31] ...trap/engines/engine.py.321: eval: epoch 114 | batch 60/510 [I 2022-10-28 08:07:31] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:34 [I 2022-10-28 08:07:31] ...trap/engines/engine.py.326: process: 0.36095 | load: 0.00024 [I 2022-10-28 08:07:35] ...trap/engines/engine.py.321: eval: epoch 114 | batch 70/510 [I 2022-10-28 08:07:35] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:29 [I 2022-10-28 08:07:35] ...trap/engines/engine.py.326: process: 0.32935 | load: 0.00020 [I 2022-10-28 08:07:38] ...trap/engines/engine.py.321: eval: epoch 114 | batch 80/510 [I 2022-10-28 08:07:38] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:27 [I 2022-10-28 08:07:38] ...trap/engines/engine.py.326: process: 0.36189 | load: 0.00023 [I 2022-10-28 08:07:41] ...trap/engines/engine.py.321: eval: epoch 114 | batch 90/510 [I 2022-10-28 08:07:41] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:19 [I 2022-10-28 08:07:41] ...trap/engines/engine.py.326: process: 0.32853 | load: 0.00017 [I 2022-10-28 08:07:45] ...trap/engines/engine.py.321: eval: epoch 114 | batch 100/510 [I 2022-10-28 08:07:45] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:12 [I 2022-10-28 08:07:45] ...trap/engines/engine.py.326: process: 0.28970 | load: 0.00017 [I 2022-10-28 08:07:47] ...trap/engines/engine.py.321: eval: epoch 114 | batch 110/510 [I 2022-10-28 08:07:47] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:56 [I 2022-10-28 08:07:47] ...trap/engines/engine.py.326: process: 0.28407 | load: 0.00014 [I 2022-10-28 08:07:50] ...trap/engines/engine.py.321: eval: epoch 114 | batch 120/510 [I 2022-10-28 08:07:50] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:54 [I 2022-10-28 08:07:50] ...trap/engines/engine.py.326: process: 0.31753 | load: 0.00015 [I 2022-10-28 08:07:53] ...trap/engines/engine.py.321: eval: epoch 114 | batch 130/510 [I 2022-10-28 08:07:53] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:53 [I 2022-10-28 08:07:53] ...trap/engines/engine.py.326: process: 0.31340 | load: 0.00017 [I 2022-10-28 08:07:56] ...trap/engines/engine.py.321: eval: epoch 114 | batch 140/510 [I 2022-10-28 08:07:56] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:47 [I 2022-10-28 08:07:56] ...trap/engines/engine.py.326: process: 0.28480 | load: 0.00015 [I 2022-10-28 08:07:59] ...trap/engines/engine.py.321: eval: epoch 114 | batch 150/510 [I 2022-10-28 08:07:59] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:45 [I 2022-10-28 08:07:59] ...trap/engines/engine.py.326: process: 0.28700 | load: 0.00015 [I 2022-10-28 08:08:02] ...trap/engines/engine.py.321: eval: epoch 114 | batch 160/510 [I 2022-10-28 08:08:02] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:42 [I 2022-10-28 08:08:02] ...trap/engines/engine.py.326: process: 0.28658 | load: 0.00017 [I 2022-10-28 08:08:05] ...trap/engines/engine.py.321: eval: epoch 114 | batch 170/510 [I 2022-10-28 08:08:05] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:39 [I 2022-10-28 08:08:05] ...trap/engines/engine.py.326: process: 0.29020 | load: 0.00014 [I 2022-10-28 08:08:08] ...trap/engines/engine.py.321: eval: epoch 114 | batch 180/510 [I 2022-10-28 08:08:08] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:35 [I 2022-10-28 08:08:08] ...trap/engines/engine.py.326: process: 0.28793 | load: 0.00014 [I 2022-10-28 08:08:11] ...trap/engines/engine.py.321: eval: epoch 114 | batch 190/510 [I 2022-10-28 08:08:11] ...trap/engines/engine.py.325: elapsed: 0:01:06 | left: 0:01:32 [I 2022-10-28 08:08:11] ...trap/engines/engine.py.326: process: 0.29013 | load: 0.00015 [I 2022-10-28 08:08:14] ...trap/engines/engine.py.321: eval: epoch 114 | batch 200/510 [I 2022-10-28 08:08:14] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:29 [I 2022-10-28 08:08:14] ...trap/engines/engine.py.326: process: 0.28670 | load: 0.00016 [I 2022-10-28 08:08:17] ...trap/engines/engine.py.321: eval: epoch 114 | batch 210/510 [I 2022-10-28 08:08:17] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 08:08:17] ...trap/engines/engine.py.326: process: 0.28480 | load: 0.00016 [I 2022-10-28 08:08:19] ...trap/engines/engine.py.321: eval: epoch 114 | batch 220/510 [I 2022-10-28 08:08:19] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:23 [I 2022-10-28 08:08:19] ...trap/engines/engine.py.326: process: 0.28478 | load: 0.00015 [I 2022-10-28 08:08:22] ...trap/engines/engine.py.321: eval: epoch 114 | batch 230/510 [I 2022-10-28 08:08:22] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:19 [I 2022-10-28 08:08:22] ...trap/engines/engine.py.326: process: 0.28628 | load: 0.00038 [I 2022-10-28 08:08:25] ...trap/engines/engine.py.321: eval: epoch 114 | batch 240/510 [I 2022-10-28 08:08:25] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:18 [I 2022-10-28 08:08:25] ...trap/engines/engine.py.326: process: 0.28754 | load: 0.00023 [I 2022-10-28 08:08:28] ...trap/engines/engine.py.321: eval: epoch 114 | batch 250/510 [I 2022-10-28 08:08:28] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:17 [I 2022-10-28 08:08:28] ...trap/engines/engine.py.326: process: 0.29066 | load: 0.00024 [I 2022-10-28 08:08:31] ...trap/engines/engine.py.321: eval: epoch 114 | batch 260/510 [I 2022-10-28 08:08:31] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:12 [I 2022-10-28 08:08:31] ...trap/engines/engine.py.326: process: 0.29104 | load: 0.00017 [I 2022-10-28 08:08:34] ...trap/engines/engine.py.321: eval: epoch 114 | batch 270/510 [I 2022-10-28 08:08:34] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:10 [I 2022-10-28 08:08:34] ...trap/engines/engine.py.326: process: 0.30812 | load: 0.00018 [I 2022-10-28 08:08:37] ...trap/engines/engine.py.321: eval: epoch 114 | batch 280/510 [I 2022-10-28 08:08:37] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:06 [I 2022-10-28 08:08:37] ...trap/engines/engine.py.326: process: 0.28764 | load: 0.00020 [I 2022-10-28 08:08:40] ...trap/engines/engine.py.321: eval: epoch 114 | batch 290/510 [I 2022-10-28 08:08:40] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:03 [I 2022-10-28 08:08:40] ...trap/engines/engine.py.326: process: 0.28873 | load: 0.00019 [I 2022-10-28 08:08:43] ...trap/engines/engine.py.321: eval: epoch 114 | batch 300/510 [I 2022-10-28 08:08:43] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:01 [I 2022-10-28 08:08:43] ...trap/engines/engine.py.326: process: 0.30710 | load: 0.00017 [I 2022-10-28 08:08:46] ...trap/engines/engine.py.321: eval: epoch 114 | batch 310/510 [I 2022-10-28 08:08:46] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:57 [I 2022-10-28 08:08:46] ...trap/engines/engine.py.326: process: 0.29138 | load: 0.00017 [I 2022-10-28 08:08:49] ...trap/engines/engine.py.321: eval: epoch 114 | batch 320/510 [I 2022-10-28 08:08:49] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:55 [I 2022-10-28 08:08:49] ...trap/engines/engine.py.326: process: 0.28344 | load: 0.00015 [I 2022-10-28 08:08:51] ...trap/engines/engine.py.321: eval: epoch 114 | batch 330/510 [I 2022-10-28 08:08:51] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:52 [I 2022-10-28 08:08:51] ...trap/engines/engine.py.326: process: 0.28996 | load: 0.00022 [I 2022-10-28 08:08:54] ...trap/engines/engine.py.321: eval: epoch 114 | batch 340/510 [I 2022-10-28 08:08:54] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:49 [I 2022-10-28 08:08:54] ...trap/engines/engine.py.326: process: 0.28696 | load: 0.00021 [I 2022-10-28 08:08:57] ...trap/engines/engine.py.321: eval: epoch 114 | batch 350/510 [I 2022-10-28 08:08:57] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:46 [I 2022-10-28 08:08:57] ...trap/engines/engine.py.326: process: 0.28945 | load: 0.00014 [I 2022-10-28 08:09:00] ...trap/engines/engine.py.321: eval: epoch 114 | batch 360/510 [I 2022-10-28 08:09:00] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:43 [I 2022-10-28 08:09:00] ...trap/engines/engine.py.326: process: 0.28841 | load: 0.00016 [I 2022-10-28 08:09:03] ...trap/engines/engine.py.321: eval: epoch 114 | batch 370/510 [I 2022-10-28 08:09:03] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:41 [I 2022-10-28 08:09:03] ...trap/engines/engine.py.326: process: 0.28942 | load: 0.00014 [I 2022-10-28 08:09:06] ...trap/engines/engine.py.321: eval: epoch 114 | batch 380/510 [I 2022-10-28 08:09:06] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:37 [I 2022-10-28 08:09:06] ...trap/engines/engine.py.326: process: 0.28471 | load: 0.00015 [I 2022-10-28 08:09:09] ...trap/engines/engine.py.321: eval: epoch 114 | batch 390/510 [I 2022-10-28 08:09:09] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:34 [I 2022-10-28 08:09:09] ...trap/engines/engine.py.326: process: 0.28947 | load: 0.00015 [I 2022-10-28 08:09:12] ...trap/engines/engine.py.321: eval: epoch 114 | batch 400/510 [I 2022-10-28 08:09:12] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:31 [I 2022-10-28 08:09:12] ...trap/engines/engine.py.326: process: 0.28930 | load: 0.00015 [I 2022-10-28 08:09:15] ...trap/engines/engine.py.321: eval: epoch 114 | batch 410/510 [I 2022-10-28 08:09:15] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:28 [I 2022-10-28 08:09:15] ...trap/engines/engine.py.326: process: 0.28759 | load: 0.00016 [I 2022-10-28 08:09:17] ...trap/engines/engine.py.321: eval: epoch 114 | batch 420/510 [I 2022-10-28 08:09:17] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:25 [I 2022-10-28 08:09:17] ...trap/engines/engine.py.326: process: 0.29264 | load: 0.00016 [I 2022-10-28 08:09:20] ...trap/engines/engine.py.321: eval: epoch 114 | batch 430/510 [I 2022-10-28 08:09:20] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:23 [I 2022-10-28 08:09:20] ...trap/engines/engine.py.326: process: 0.29132 | load: 0.00019 [I 2022-10-28 08:09:23] ...trap/engines/engine.py.321: eval: epoch 114 | batch 440/510 [I 2022-10-28 08:09:23] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:19 [I 2022-10-28 08:09:23] ...trap/engines/engine.py.326: process: 0.28537 | load: 0.00015 [I 2022-10-28 08:09:26] ...trap/engines/engine.py.321: eval: epoch 114 | batch 450/510 [I 2022-10-28 08:09:26] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:17 [I 2022-10-28 08:09:26] ...trap/engines/engine.py.326: process: 0.29595 | load: 0.00017 [I 2022-10-28 08:09:29] ...trap/engines/engine.py.321: eval: epoch 114 | batch 460/510 [I 2022-10-28 08:09:29] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:14 [I 2022-10-28 08:09:29] ...trap/engines/engine.py.326: process: 0.29028 | load: 0.00020 [I 2022-10-28 08:09:32] ...trap/engines/engine.py.321: eval: epoch 114 | batch 470/510 [I 2022-10-28 08:09:32] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:11 [I 2022-10-28 08:09:32] ...trap/engines/engine.py.326: process: 0.28697 | load: 0.00024 [I 2022-10-28 08:09:35] ...trap/engines/engine.py.321: eval: epoch 114 | batch 480/510 [I 2022-10-28 08:09:35] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 08:09:35] ...trap/engines/engine.py.326: process: 0.28909 | load: 0.00016 [I 2022-10-28 08:09:38] ...trap/engines/engine.py.321: eval: epoch 114 | batch 490/510 [I 2022-10-28 08:09:38] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 08:09:38] ...trap/engines/engine.py.326: process: 0.28854 | load: 0.00018 [I 2022-10-28 08:09:40] ...trap/engines/engine.py.321: eval: epoch 114 | batch 500/510 [I 2022-10-28 08:09:40] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:02 [I 2022-10-28 08:09:40] ...trap/engines/engine.py.326: process: 0.28909 | load: 0.00015 [I 2022-10-28 08:09:43] ...trap/engines/engine.py.321: eval: epoch 114 | batch 510/510 [I 2022-10-28 08:09:43] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:00 [I 2022-10-28 08:09:43] ...trap/engines/engine.py.326: process: 0.28374 | load: 0.00012 [S 2022-10-28 08:09:44] ...trap/engines/engine.py.132: eval_epoch.epoch: 114 [I 2022-10-28 08:09:44] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.728100 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.904800 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.932000 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.733200 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.902700 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933000 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014802 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009958 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009980 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013467 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009665 [S 2022-10-28 08:09:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007443 [I 2022-10-28 08:09:50] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 08:09:50] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 08:09:51] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 08:09:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 08:09:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 08:09:56] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 08:09:56] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 08:10:03] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 08:10:03] ...trap/engines/engine.py.163: Training model on trainset for epoch 115 [I 2022-10-28 08:10:03] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 08:10:05] ...trap/engines/engine.py.224: train: epoch 115 | batch 0/2304 [I 2022-10-28 08:10:05] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:11 [I 2022-10-28 08:10:05] ...trap/engines/engine.py.229: process: 2.79125 | load: 2.13766 [I 2022-10-28 08:10:05] ...trap/engines/engine.py.230: loss: 0.09489 [I 2022-10-28 08:10:05] ...optimizers/trijoint.py.151: total_norm: 12.754784 [I 2022-10-28 08:10:11] ...trap/engines/engine.py.224: train: epoch 115 | batch 10/2304 [I 2022-10-28 08:10:11] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:28 [I 2022-10-28 08:10:11] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00343 [I 2022-10-28 08:10:11] ...trap/engines/engine.py.230: loss: 0.08529 [I 2022-10-28 08:10:11] ...optimizers/trijoint.py.151: total_norm: 11.112350 [I 2022-10-28 08:10:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 20/2304 [I 2022-10-28 08:10:17] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:50 [I 2022-10-28 08:10:17] ...trap/engines/engine.py.229: process: 0.56749 | load: 0.00369 [I 2022-10-28 08:10:17] ...trap/engines/engine.py.230: loss: 0.10997 [I 2022-10-28 08:10:17] ...optimizers/trijoint.py.151: total_norm: 20.733208 [I 2022-10-28 08:10:22] ...trap/engines/engine.py.224: train: epoch 115 | batch 30/2304 [I 2022-10-28 08:10:22] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:15 [I 2022-10-28 08:10:22] ...trap/engines/engine.py.229: process: 0.56257 | load: 0.00285 [I 2022-10-28 08:10:22] ...trap/engines/engine.py.230: loss: 0.09228 [I 2022-10-28 08:10:22] ...optimizers/trijoint.py.151: total_norm: 12.768468 [I 2022-10-28 08:10:28] ...trap/engines/engine.py.224: train: epoch 115 | batch 40/2304 [I 2022-10-28 08:10:28] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:06 [I 2022-10-28 08:10:28] ...trap/engines/engine.py.229: process: 0.55851 | load: 0.00319 [I 2022-10-28 08:10:28] ...trap/engines/engine.py.230: loss: 0.09479 [I 2022-10-28 08:10:28] ...optimizers/trijoint.py.151: total_norm: 9.889312 [I 2022-10-28 08:10:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 50/2304 [I 2022-10-28 08:10:34] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:07 [I 2022-10-28 08:10:34] ...trap/engines/engine.py.229: process: 0.56754 | load: 0.00313 [I 2022-10-28 08:10:34] ...trap/engines/engine.py.230: loss: 0.12860 [I 2022-10-28 08:10:34] ...optimizers/trijoint.py.151: total_norm: 16.318001 [I 2022-10-28 08:10:39] ...trap/engines/engine.py.224: train: epoch 115 | batch 60/2304 [I 2022-10-28 08:10:39] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:00 [I 2022-10-28 08:10:39] ...trap/engines/engine.py.229: process: 0.56181 | load: 0.00293 [I 2022-10-28 08:10:39] ...trap/engines/engine.py.230: loss: 0.13510 [I 2022-10-28 08:10:39] ...optimizers/trijoint.py.151: total_norm: 25.349430 [I 2022-10-28 08:10:45] ...trap/engines/engine.py.224: train: epoch 115 | batch 70/2304 [I 2022-10-28 08:10:45] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:38 [I 2022-10-28 08:10:45] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00287 [I 2022-10-28 08:10:45] ...trap/engines/engine.py.230: loss: 0.12451 [I 2022-10-28 08:10:45] ...optimizers/trijoint.py.151: total_norm: 29.976353 [I 2022-10-28 08:10:50] ...trap/engines/engine.py.224: train: epoch 115 | batch 80/2304 [I 2022-10-28 08:10:50] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:40 [I 2022-10-28 08:10:50] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00284 [I 2022-10-28 08:10:50] ...trap/engines/engine.py.230: loss: 0.15273 [I 2022-10-28 08:10:50] ...optimizers/trijoint.py.151: total_norm: 11.744273 [I 2022-10-28 08:10:56] ...trap/engines/engine.py.224: train: epoch 115 | batch 90/2304 [I 2022-10-28 08:10:56] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:39 [I 2022-10-28 08:10:56] ...trap/engines/engine.py.229: process: 0.56902 | load: 0.00277 [I 2022-10-28 08:10:56] ...trap/engines/engine.py.230: loss: 0.09391 [I 2022-10-28 08:10:56] ...optimizers/trijoint.py.151: total_norm: 11.491224 [I 2022-10-28 08:11:02] ...trap/engines/engine.py.224: train: epoch 115 | batch 100/2304 [I 2022-10-28 08:11:02] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:23 [I 2022-10-28 08:11:02] ...trap/engines/engine.py.229: process: 0.55487 | load: 0.00263 [I 2022-10-28 08:11:02] ...trap/engines/engine.py.230: loss: 0.14828 [I 2022-10-28 08:11:02] ...optimizers/trijoint.py.151: total_norm: 14.193254 [I 2022-10-28 08:11:07] ...trap/engines/engine.py.224: train: epoch 115 | batch 110/2304 [I 2022-10-28 08:11:07] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:12 [I 2022-10-28 08:11:07] ...trap/engines/engine.py.229: process: 0.54528 | load: 0.00276 [I 2022-10-28 08:11:07] ...trap/engines/engine.py.230: loss: 0.09692 [I 2022-10-28 08:11:07] ...optimizers/trijoint.py.151: total_norm: 27.752064 [I 2022-10-28 08:11:13] ...trap/engines/engine.py.224: train: epoch 115 | batch 120/2304 [I 2022-10-28 08:11:13] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:15 [I 2022-10-28 08:11:13] ...trap/engines/engine.py.229: process: 0.55057 | load: 0.00283 [I 2022-10-28 08:11:13] ...trap/engines/engine.py.230: loss: 0.07800 [I 2022-10-28 08:11:13] ...optimizers/trijoint.py.151: total_norm: 15.241311 [I 2022-10-28 08:11:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 130/2304 [I 2022-10-28 08:11:18] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:25 [I 2022-10-28 08:11:18] ...trap/engines/engine.py.229: process: 0.56915 | load: 0.00245 [I 2022-10-28 08:11:18] ...trap/engines/engine.py.230: loss: 0.10514 [I 2022-10-28 08:11:18] ...optimizers/trijoint.py.151: total_norm: 10.847662 [I 2022-10-28 08:11:24] ...trap/engines/engine.py.224: train: epoch 115 | batch 140/2304 [I 2022-10-28 08:11:24] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:20 [I 2022-10-28 08:11:24] ...trap/engines/engine.py.229: process: 0.55423 | load: 0.00251 [I 2022-10-28 08:11:24] ...trap/engines/engine.py.230: loss: 0.09747 [I 2022-10-28 08:11:24] ...optimizers/trijoint.py.151: total_norm: 11.228033 [I 2022-10-28 08:11:30] ...trap/engines/engine.py.224: train: epoch 115 | batch 150/2304 [I 2022-10-28 08:11:30] ...trap/engines/engine.py.228: elapsed: 0:01:27 | left: 0:20:10 [I 2022-10-28 08:11:30] ...trap/engines/engine.py.229: process: 0.54860 | load: 0.00248 [I 2022-10-28 08:11:30] ...trap/engines/engine.py.230: loss: 0.13349 [I 2022-10-28 08:11:30] ...optimizers/trijoint.py.151: total_norm: 16.311417 [I 2022-10-28 08:11:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 160/2304 [I 2022-10-28 08:11:35] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:04 [I 2022-10-28 08:11:35] ...trap/engines/engine.py.229: process: 0.55564 | load: 0.00261 [I 2022-10-28 08:11:35] ...trap/engines/engine.py.230: loss: 0.08061 [I 2022-10-28 08:11:35] ...optimizers/trijoint.py.151: total_norm: 9.367768 [I 2022-10-28 08:11:41] ...trap/engines/engine.py.224: train: epoch 115 | batch 170/2304 [I 2022-10-28 08:11:41] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:20:04 [I 2022-10-28 08:11:41] ...trap/engines/engine.py.229: process: 0.55002 | load: 0.00268 [I 2022-10-28 08:11:41] ...trap/engines/engine.py.230: loss: 0.11602 [I 2022-10-28 08:11:41] ...optimizers/trijoint.py.151: total_norm: 19.178421 [I 2022-10-28 08:11:47] ...trap/engines/engine.py.224: train: epoch 115 | batch 180/2304 [I 2022-10-28 08:11:47] ...trap/engines/engine.py.228: elapsed: 0:01:44 | left: 0:19:43 [I 2022-10-28 08:11:47] ...trap/engines/engine.py.229: process: 0.55668 | load: 0.00243 [I 2022-10-28 08:11:47] ...trap/engines/engine.py.230: loss: 0.08562 [I 2022-10-28 08:11:47] ...optimizers/trijoint.py.151: total_norm: 11.127913 [I 2022-10-28 08:11:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 190/2304 [I 2022-10-28 08:11:52] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:40 [I 2022-10-28 08:11:52] ...trap/engines/engine.py.229: process: 0.54788 | load: 0.00263 [I 2022-10-28 08:11:52] ...trap/engines/engine.py.230: loss: 0.11730 [I 2022-10-28 08:11:52] ...optimizers/trijoint.py.151: total_norm: 58.562603 [I 2022-10-28 08:11:58] ...trap/engines/engine.py.224: train: epoch 115 | batch 200/2304 [I 2022-10-28 08:11:58] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:31 [I 2022-10-28 08:11:58] ...trap/engines/engine.py.229: process: 0.54725 | load: 0.00256 [I 2022-10-28 08:11:58] ...trap/engines/engine.py.230: loss: 0.11909 [I 2022-10-28 08:11:58] ...optimizers/trijoint.py.151: total_norm: 18.992977 [I 2022-10-28 08:12:04] ...trap/engines/engine.py.224: train: epoch 115 | batch 210/2304 [I 2022-10-28 08:12:04] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:42 [I 2022-10-28 08:12:04] ...trap/engines/engine.py.229: process: 0.56391 | load: 0.00255 [I 2022-10-28 08:12:04] ...trap/engines/engine.py.230: loss: 0.09222 [I 2022-10-28 08:12:04] ...optimizers/trijoint.py.151: total_norm: 14.424906 [I 2022-10-28 08:12:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 220/2304 [I 2022-10-28 08:12:09] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:18 [I 2022-10-28 08:12:09] ...trap/engines/engine.py.229: process: 0.54971 | load: 0.00256 [I 2022-10-28 08:12:09] ...trap/engines/engine.py.230: loss: 0.12192 [I 2022-10-28 08:12:09] ...optimizers/trijoint.py.151: total_norm: 12.766860 [I 2022-10-28 08:12:15] ...trap/engines/engine.py.224: train: epoch 115 | batch 230/2304 [I 2022-10-28 08:12:15] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:19:37 [I 2022-10-28 08:12:15] ...trap/engines/engine.py.229: process: 0.57803 | load: 0.00266 [I 2022-10-28 08:12:15] ...trap/engines/engine.py.230: loss: 0.09142 [I 2022-10-28 08:12:15] ...optimizers/trijoint.py.151: total_norm: 13.032783 [I 2022-10-28 08:12:20] ...trap/engines/engine.py.224: train: epoch 115 | batch 240/2304 [I 2022-10-28 08:12:20] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:25 [I 2022-10-28 08:12:20] ...trap/engines/engine.py.229: process: 0.55912 | load: 0.00251 [I 2022-10-28 08:12:20] ...trap/engines/engine.py.230: loss: 0.09981 [I 2022-10-28 08:12:20] ...optimizers/trijoint.py.151: total_norm: 29.734781 [I 2022-10-28 08:12:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 250/2304 [I 2022-10-28 08:12:26] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:16 [I 2022-10-28 08:12:26] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00252 [I 2022-10-28 08:12:26] ...trap/engines/engine.py.230: loss: 0.10849 [I 2022-10-28 08:12:26] ...optimizers/trijoint.py.151: total_norm: 10.716589 [I 2022-10-28 08:12:32] ...trap/engines/engine.py.224: train: epoch 115 | batch 260/2304 [I 2022-10-28 08:12:32] ...trap/engines/engine.py.228: elapsed: 0:02:29 | left: 0:19:05 [I 2022-10-28 08:12:32] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00256 [I 2022-10-28 08:12:32] ...trap/engines/engine.py.230: loss: 0.12434 [I 2022-10-28 08:12:32] ...optimizers/trijoint.py.151: total_norm: 29.402819 [I 2022-10-28 08:12:37] ...trap/engines/engine.py.224: train: epoch 115 | batch 270/2304 [I 2022-10-28 08:12:37] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:56 [I 2022-10-28 08:12:37] ...trap/engines/engine.py.229: process: 0.54906 | load: 0.00258 [I 2022-10-28 08:12:37] ...trap/engines/engine.py.230: loss: 0.12180 [I 2022-10-28 08:12:37] ...optimizers/trijoint.py.151: total_norm: 10.559552 [I 2022-10-28 08:12:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 280/2304 [I 2022-10-28 08:12:43] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:19:10 [I 2022-10-28 08:12:43] ...trap/engines/engine.py.229: process: 0.57690 | load: 0.00254 [I 2022-10-28 08:12:43] ...trap/engines/engine.py.230: loss: 0.14879 [I 2022-10-28 08:12:43] ...optimizers/trijoint.py.151: total_norm: 29.360018 [I 2022-10-28 08:12:49] ...trap/engines/engine.py.224: train: epoch 115 | batch 290/2304 [I 2022-10-28 08:12:49] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:53 [I 2022-10-28 08:12:49] ...trap/engines/engine.py.229: process: 0.57413 | load: 0.00257 [I 2022-10-28 08:12:49] ...trap/engines/engine.py.230: loss: 0.07074 [I 2022-10-28 08:12:49] ...optimizers/trijoint.py.151: total_norm: 23.264095 [I 2022-10-28 08:12:54] ...trap/engines/engine.py.224: train: epoch 115 | batch 300/2304 [I 2022-10-28 08:12:54] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:34 [I 2022-10-28 08:12:54] ...trap/engines/engine.py.229: process: 0.54953 | load: 0.00251 [I 2022-10-28 08:12:54] ...trap/engines/engine.py.230: loss: 0.12500 [I 2022-10-28 08:12:54] ...optimizers/trijoint.py.151: total_norm: 41.141895 [I 2022-10-28 08:13:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 310/2304 [I 2022-10-28 08:13:00] ...trap/engines/engine.py.228: elapsed: 0:02:57 | left: 0:18:31 [I 2022-10-28 08:13:00] ...trap/engines/engine.py.229: process: 0.55177 | load: 0.00259 [I 2022-10-28 08:13:00] ...trap/engines/engine.py.230: loss: 0.10040 [I 2022-10-28 08:13:00] ...optimizers/trijoint.py.151: total_norm: 26.406002 [I 2022-10-28 08:13:05] ...trap/engines/engine.py.224: train: epoch 115 | batch 320/2304 [I 2022-10-28 08:13:05] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:34 [I 2022-10-28 08:13:05] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00253 [I 2022-10-28 08:13:05] ...trap/engines/engine.py.230: loss: 0.09200 [I 2022-10-28 08:13:05] ...optimizers/trijoint.py.151: total_norm: 13.277513 [I 2022-10-28 08:13:11] ...trap/engines/engine.py.224: train: epoch 115 | batch 330/2304 [I 2022-10-28 08:13:11] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:18:27 [I 2022-10-28 08:13:11] ...trap/engines/engine.py.229: process: 0.56245 | load: 0.00253 [I 2022-10-28 08:13:11] ...trap/engines/engine.py.230: loss: 0.13585 [I 2022-10-28 08:13:11] ...optimizers/trijoint.py.151: total_norm: 15.456376 [I 2022-10-28 08:13:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 340/2304 [I 2022-10-28 08:13:17] ...trap/engines/engine.py.228: elapsed: 0:03:14 | left: 0:18:25 [I 2022-10-28 08:13:17] ...trap/engines/engine.py.229: process: 0.56953 | load: 0.00257 [I 2022-10-28 08:13:17] ...trap/engines/engine.py.230: loss: 0.10982 [I 2022-10-28 08:13:17] ...optimizers/trijoint.py.151: total_norm: 12.766523 [I 2022-10-28 08:13:22] ...trap/engines/engine.py.224: train: epoch 115 | batch 350/2304 [I 2022-10-28 08:13:22] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:09 [I 2022-10-28 08:13:22] ...trap/engines/engine.py.229: process: 0.56447 | load: 0.00268 [I 2022-10-28 08:13:22] ...trap/engines/engine.py.230: loss: 0.11870 [I 2022-10-28 08:13:22] ...optimizers/trijoint.py.151: total_norm: 10.731692 [I 2022-10-28 08:13:28] ...trap/engines/engine.py.224: train: epoch 115 | batch 360/2304 [I 2022-10-28 08:13:28] ...trap/engines/engine.py.228: elapsed: 0:03:25 | left: 0:18:14 [I 2022-10-28 08:13:28] ...trap/engines/engine.py.229: process: 0.57806 | load: 0.00264 [I 2022-10-28 08:13:28] ...trap/engines/engine.py.230: loss: 0.08574 [I 2022-10-28 08:13:28] ...optimizers/trijoint.py.151: total_norm: 9.718081 [I 2022-10-28 08:13:33] ...trap/engines/engine.py.224: train: epoch 115 | batch 370/2304 [I 2022-10-28 08:13:33] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:01 [I 2022-10-28 08:13:33] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00265 [I 2022-10-28 08:13:33] ...trap/engines/engine.py.230: loss: 0.12307 [I 2022-10-28 08:13:33] ...optimizers/trijoint.py.151: total_norm: 16.948994 [I 2022-10-28 08:13:39] ...trap/engines/engine.py.224: train: epoch 115 | batch 380/2304 [I 2022-10-28 08:13:39] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:17:51 [I 2022-10-28 08:13:39] ...trap/engines/engine.py.229: process: 0.55375 | load: 0.00251 [I 2022-10-28 08:13:39] ...trap/engines/engine.py.230: loss: 0.08885 [I 2022-10-28 08:13:39] ...optimizers/trijoint.py.151: total_norm: 19.014771 [I 2022-10-28 08:13:45] ...trap/engines/engine.py.224: train: epoch 115 | batch 390/2304 [I 2022-10-28 08:13:45] ...trap/engines/engine.py.228: elapsed: 0:03:42 | left: 0:17:53 [I 2022-10-28 08:13:45] ...trap/engines/engine.py.229: process: 0.55101 | load: 0.00252 [I 2022-10-28 08:13:45] ...trap/engines/engine.py.230: loss: 0.10971 [I 2022-10-28 08:13:45] ...optimizers/trijoint.py.151: total_norm: 10.172429 [I 2022-10-28 08:13:50] ...trap/engines/engine.py.224: train: epoch 115 | batch 400/2304 [I 2022-10-28 08:13:50] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:48 [I 2022-10-28 08:13:50] ...trap/engines/engine.py.229: process: 0.55339 | load: 0.00251 [I 2022-10-28 08:13:50] ...trap/engines/engine.py.230: loss: 0.08594 [I 2022-10-28 08:13:50] ...optimizers/trijoint.py.151: total_norm: 9.942154 [I 2022-10-28 08:13:56] ...trap/engines/engine.py.224: train: epoch 115 | batch 410/2304 [I 2022-10-28 08:13:56] ...trap/engines/engine.py.228: elapsed: 0:03:53 | left: 0:17:32 [I 2022-10-28 08:13:56] ...trap/engines/engine.py.229: process: 0.54073 | load: 0.00247 [I 2022-10-28 08:13:56] ...trap/engines/engine.py.230: loss: 0.17944 [I 2022-10-28 08:13:56] ...optimizers/trijoint.py.151: total_norm: 17.991077 [I 2022-10-28 08:14:01] ...trap/engines/engine.py.224: train: epoch 115 | batch 420/2304 [I 2022-10-28 08:14:01] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:33 [I 2022-10-28 08:14:01] ...trap/engines/engine.py.229: process: 0.55039 | load: 0.00246 [I 2022-10-28 08:14:01] ...trap/engines/engine.py.230: loss: 0.14030 [I 2022-10-28 08:14:01] ...optimizers/trijoint.py.151: total_norm: 13.166651 [I 2022-10-28 08:14:07] ...trap/engines/engine.py.224: train: epoch 115 | batch 430/2304 [I 2022-10-28 08:14:07] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:27 [I 2022-10-28 08:14:07] ...trap/engines/engine.py.229: process: 0.56086 | load: 0.00244 [I 2022-10-28 08:14:07] ...trap/engines/engine.py.230: loss: 0.09563 [I 2022-10-28 08:14:07] ...optimizers/trijoint.py.151: total_norm: 10.831190 [I 2022-10-28 08:14:13] ...trap/engines/engine.py.224: train: epoch 115 | batch 440/2304 [I 2022-10-28 08:14:13] ...trap/engines/engine.py.228: elapsed: 0:04:10 | left: 0:17:33 [I 2022-10-28 08:14:13] ...trap/engines/engine.py.229: process: 0.55534 | load: 0.00258 [I 2022-10-28 08:14:13] ...trap/engines/engine.py.230: loss: 0.13474 [I 2022-10-28 08:14:13] ...optimizers/trijoint.py.151: total_norm: 18.517740 [I 2022-10-28 08:14:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 450/2304 [I 2022-10-28 08:14:18] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:11 [I 2022-10-28 08:14:18] ...trap/engines/engine.py.229: process: 0.54552 | load: 0.00247 [I 2022-10-28 08:14:18] ...trap/engines/engine.py.230: loss: 0.08804 [I 2022-10-28 08:14:18] ...optimizers/trijoint.py.151: total_norm: 11.766076 [I 2022-10-28 08:14:24] ...trap/engines/engine.py.224: train: epoch 115 | batch 460/2304 [I 2022-10-28 08:14:24] ...trap/engines/engine.py.228: elapsed: 0:04:21 | left: 0:17:06 [I 2022-10-28 08:14:24] ...trap/engines/engine.py.229: process: 0.55686 | load: 0.00248 [I 2022-10-28 08:14:24] ...trap/engines/engine.py.230: loss: 0.08756 [I 2022-10-28 08:14:24] ...optimizers/trijoint.py.151: total_norm: 18.274212 [I 2022-10-28 08:14:30] ...trap/engines/engine.py.224: train: epoch 115 | batch 470/2304 [I 2022-10-28 08:14:30] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:09 [I 2022-10-28 08:14:30] ...trap/engines/engine.py.229: process: 0.56550 | load: 0.00249 [I 2022-10-28 08:14:30] ...trap/engines/engine.py.230: loss: 0.09444 [I 2022-10-28 08:14:30] ...optimizers/trijoint.py.151: total_norm: 11.869569 [I 2022-10-28 08:14:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 480/2304 [I 2022-10-28 08:14:35] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:17:05 [I 2022-10-28 08:14:35] ...trap/engines/engine.py.229: process: 0.57006 | load: 0.00246 [I 2022-10-28 08:14:35] ...trap/engines/engine.py.230: loss: 0.09477 [I 2022-10-28 08:14:35] ...optimizers/trijoint.py.151: total_norm: 15.749233 [I 2022-10-28 08:14:41] ...trap/engines/engine.py.224: train: epoch 115 | batch 490/2304 [I 2022-10-28 08:14:41] ...trap/engines/engine.py.228: elapsed: 0:04:38 | left: 0:16:57 [I 2022-10-28 08:14:41] ...trap/engines/engine.py.229: process: 0.56080 | load: 0.00243 [I 2022-10-28 08:14:41] ...trap/engines/engine.py.230: loss: 0.13263 [I 2022-10-28 08:14:41] ...optimizers/trijoint.py.151: total_norm: 11.149574 [I 2022-10-28 08:14:46] ...trap/engines/engine.py.224: train: epoch 115 | batch 500/2304 [I 2022-10-28 08:14:46] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:17:00 [I 2022-10-28 08:14:46] ...trap/engines/engine.py.229: process: 0.56356 | load: 0.00246 [I 2022-10-28 08:14:46] ...trap/engines/engine.py.230: loss: 0.08764 [I 2022-10-28 08:14:46] ...optimizers/trijoint.py.151: total_norm: 18.399309 [I 2022-10-28 08:14:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 510/2304 [I 2022-10-28 08:14:52] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:41 [I 2022-10-28 08:14:52] ...trap/engines/engine.py.229: process: 0.54517 | load: 0.00243 [I 2022-10-28 08:14:52] ...trap/engines/engine.py.230: loss: 0.09322 [I 2022-10-28 08:14:52] ...optimizers/trijoint.py.151: total_norm: 18.890026 [I 2022-10-28 08:14:58] ...trap/engines/engine.py.224: train: epoch 115 | batch 520/2304 [I 2022-10-28 08:14:58] ...trap/engines/engine.py.228: elapsed: 0:04:55 | left: 0:16:36 [I 2022-10-28 08:14:58] ...trap/engines/engine.py.229: process: 0.54224 | load: 0.00257 [I 2022-10-28 08:14:58] ...trap/engines/engine.py.230: loss: 0.12520 [I 2022-10-28 08:14:58] ...optimizers/trijoint.py.151: total_norm: 19.493795 [I 2022-10-28 08:15:03] ...trap/engines/engine.py.224: train: epoch 115 | batch 530/2304 [I 2022-10-28 08:15:03] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:32 [I 2022-10-28 08:15:03] ...trap/engines/engine.py.229: process: 0.55179 | load: 0.00249 [I 2022-10-28 08:15:03] ...trap/engines/engine.py.230: loss: 0.12746 [I 2022-10-28 08:15:03] ...optimizers/trijoint.py.151: total_norm: 19.158348 [I 2022-10-28 08:15:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 540/2304 [I 2022-10-28 08:15:09] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:16:26 [I 2022-10-28 08:15:09] ...trap/engines/engine.py.229: process: 0.55943 | load: 0.00249 [I 2022-10-28 08:15:09] ...trap/engines/engine.py.230: loss: 0.10375 [I 2022-10-28 08:15:09] ...optimizers/trijoint.py.151: total_norm: 19.515116 [I 2022-10-28 08:15:15] ...trap/engines/engine.py.224: train: epoch 115 | batch 550/2304 [I 2022-10-28 08:15:15] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:20 [I 2022-10-28 08:15:15] ...trap/engines/engine.py.229: process: 0.55502 | load: 0.00244 [I 2022-10-28 08:15:15] ...trap/engines/engine.py.230: loss: 0.09505 [I 2022-10-28 08:15:15] ...optimizers/trijoint.py.151: total_norm: 18.757677 [I 2022-10-28 08:15:20] ...trap/engines/engine.py.224: train: epoch 115 | batch 560/2304 [I 2022-10-28 08:15:20] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:10 [I 2022-10-28 08:15:20] ...trap/engines/engine.py.229: process: 0.54910 | load: 0.00246 [I 2022-10-28 08:15:20] ...trap/engines/engine.py.230: loss: 0.09012 [I 2022-10-28 08:15:20] ...optimizers/trijoint.py.151: total_norm: 11.029453 [I 2022-10-28 08:15:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 570/2304 [I 2022-10-28 08:15:26] ...trap/engines/engine.py.228: elapsed: 0:05:23 | left: 0:16:09 [I 2022-10-28 08:15:26] ...trap/engines/engine.py.229: process: 0.55962 | load: 0.00247 [I 2022-10-28 08:15:26] ...trap/engines/engine.py.230: loss: 0.15074 [I 2022-10-28 08:15:26] ...optimizers/trijoint.py.151: total_norm: 20.751312 [I 2022-10-28 08:15:31] ...trap/engines/engine.py.224: train: epoch 115 | batch 580/2304 [I 2022-10-28 08:15:31] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:15:56 [I 2022-10-28 08:15:31] ...trap/engines/engine.py.229: process: 0.55557 | load: 0.00246 [I 2022-10-28 08:15:31] ...trap/engines/engine.py.230: loss: 0.16377 [I 2022-10-28 08:15:31] ...optimizers/trijoint.py.151: total_norm: 98.541550 [I 2022-10-28 08:15:37] ...trap/engines/engine.py.224: train: epoch 115 | batch 590/2304 [I 2022-10-28 08:15:37] ...trap/engines/engine.py.228: elapsed: 0:05:34 | left: 0:15:56 [I 2022-10-28 08:15:37] ...trap/engines/engine.py.229: process: 0.54702 | load: 0.00262 [I 2022-10-28 08:15:37] ...trap/engines/engine.py.230: loss: 0.11384 [I 2022-10-28 08:15:37] ...optimizers/trijoint.py.151: total_norm: 17.788336 [I 2022-10-28 08:15:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 600/2304 [I 2022-10-28 08:15:42] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:48 [I 2022-10-28 08:15:42] ...trap/engines/engine.py.229: process: 0.55886 | load: 0.00250 [I 2022-10-28 08:15:42] ...trap/engines/engine.py.230: loss: 0.10925 [I 2022-10-28 08:15:42] ...optimizers/trijoint.py.151: total_norm: 8.030368 [I 2022-10-28 08:15:48] ...trap/engines/engine.py.224: train: epoch 115 | batch 610/2304 [I 2022-10-28 08:15:48] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:15:45 [I 2022-10-28 08:15:48] ...trap/engines/engine.py.229: process: 0.54541 | load: 0.00251 [I 2022-10-28 08:15:48] ...trap/engines/engine.py.230: loss: 0.09071 [I 2022-10-28 08:15:48] ...optimizers/trijoint.py.151: total_norm: 31.131887 [I 2022-10-28 08:15:54] ...trap/engines/engine.py.224: train: epoch 115 | batch 620/2304 [I 2022-10-28 08:15:54] ...trap/engines/engine.py.228: elapsed: 0:05:51 | left: 0:15:38 [I 2022-10-28 08:15:54] ...trap/engines/engine.py.229: process: 0.55537 | load: 0.00245 [I 2022-10-28 08:15:54] ...trap/engines/engine.py.230: loss: 0.10119 [I 2022-10-28 08:15:54] ...optimizers/trijoint.py.151: total_norm: 10.515616 [I 2022-10-28 08:15:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 630/2304 [I 2022-10-28 08:15:59] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:39 [I 2022-10-28 08:15:59] ...trap/engines/engine.py.229: process: 0.56061 | load: 0.00263 [I 2022-10-28 08:15:59] ...trap/engines/engine.py.230: loss: 0.18611 [I 2022-10-28 08:15:59] ...optimizers/trijoint.py.151: total_norm: 10.230009 [I 2022-10-28 08:16:05] ...trap/engines/engine.py.224: train: epoch 115 | batch 640/2304 [I 2022-10-28 08:16:05] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:15:31 [I 2022-10-28 08:16:05] ...trap/engines/engine.py.229: process: 0.56314 | load: 0.00246 [I 2022-10-28 08:16:05] ...trap/engines/engine.py.230: loss: 0.12501 [I 2022-10-28 08:16:05] ...optimizers/trijoint.py.151: total_norm: 17.485929 [I 2022-10-28 08:16:10] ...trap/engines/engine.py.224: train: epoch 115 | batch 650/2304 [I 2022-10-28 08:16:10] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:25 [I 2022-10-28 08:16:10] ...trap/engines/engine.py.229: process: 0.56315 | load: 0.00244 [I 2022-10-28 08:16:10] ...trap/engines/engine.py.230: loss: 0.10281 [I 2022-10-28 08:16:10] ...optimizers/trijoint.py.151: total_norm: 10.326259 [I 2022-10-28 08:16:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 660/2304 [I 2022-10-28 08:16:16] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:15:30 [I 2022-10-28 08:16:16] ...trap/engines/engine.py.229: process: 0.56372 | load: 0.00244 [I 2022-10-28 08:16:16] ...trap/engines/engine.py.230: loss: 0.11354 [I 2022-10-28 08:16:16] ...optimizers/trijoint.py.151: total_norm: 25.859329 [I 2022-10-28 08:16:22] ...trap/engines/engine.py.224: train: epoch 115 | batch 670/2304 [I 2022-10-28 08:16:22] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:15:22 [I 2022-10-28 08:16:22] ...trap/engines/engine.py.229: process: 0.56218 | load: 0.00244 [I 2022-10-28 08:16:22] ...trap/engines/engine.py.230: loss: 0.13604 [I 2022-10-28 08:16:22] ...optimizers/trijoint.py.151: total_norm: 16.606171 [I 2022-10-28 08:16:27] ...trap/engines/engine.py.224: train: epoch 115 | batch 680/2304 [I 2022-10-28 08:16:27] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:04 [I 2022-10-28 08:16:27] ...trap/engines/engine.py.229: process: 0.55272 | load: 0.00243 [I 2022-10-28 08:16:27] ...trap/engines/engine.py.230: loss: 0.09784 [I 2022-10-28 08:16:27] ...optimizers/trijoint.py.151: total_norm: 20.512932 [I 2022-10-28 08:16:33] ...trap/engines/engine.py.224: train: epoch 115 | batch 690/2304 [I 2022-10-28 08:16:33] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:15:12 [I 2022-10-28 08:16:33] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00245 [I 2022-10-28 08:16:33] ...trap/engines/engine.py.230: loss: 0.12416 [I 2022-10-28 08:16:33] ...optimizers/trijoint.py.151: total_norm: 16.874256 [I 2022-10-28 08:16:39] ...trap/engines/engine.py.224: train: epoch 115 | batch 700/2304 [I 2022-10-28 08:16:39] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:15:01 [I 2022-10-28 08:16:39] ...trap/engines/engine.py.229: process: 0.56236 | load: 0.00252 [I 2022-10-28 08:16:39] ...trap/engines/engine.py.230: loss: 0.10343 [I 2022-10-28 08:16:39] ...optimizers/trijoint.py.151: total_norm: 16.642387 [I 2022-10-28 08:16:44] ...trap/engines/engine.py.224: train: epoch 115 | batch 710/2304 [I 2022-10-28 08:16:44] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:14:53 [I 2022-10-28 08:16:44] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00247 [I 2022-10-28 08:16:44] ...trap/engines/engine.py.230: loss: 0.13784 [I 2022-10-28 08:16:44] ...optimizers/trijoint.py.151: total_norm: 12.155447 [I 2022-10-28 08:16:50] ...trap/engines/engine.py.224: train: epoch 115 | batch 720/2304 [I 2022-10-28 08:16:50] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:15:00 [I 2022-10-28 08:16:50] ...trap/engines/engine.py.229: process: 0.54495 | load: 0.00246 [I 2022-10-28 08:16:50] ...trap/engines/engine.py.230: loss: 0.11748 [I 2022-10-28 08:16:50] ...optimizers/trijoint.py.151: total_norm: 10.554049 [I 2022-10-28 08:16:56] ...trap/engines/engine.py.224: train: epoch 115 | batch 730/2304 [I 2022-10-28 08:16:56] ...trap/engines/engine.py.228: elapsed: 0:06:53 | left: 0:15:04 [I 2022-10-28 08:16:56] ...trap/engines/engine.py.229: process: 0.55215 | load: 0.00252 [I 2022-10-28 08:16:56] ...trap/engines/engine.py.230: loss: 0.13989 [I 2022-10-28 08:16:56] ...optimizers/trijoint.py.151: total_norm: 13.500068 [I 2022-10-28 08:17:01] ...trap/engines/engine.py.224: train: epoch 115 | batch 740/2304 [I 2022-10-28 08:17:01] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:14:34 [I 2022-10-28 08:17:01] ...trap/engines/engine.py.229: process: 0.54592 | load: 0.00245 [I 2022-10-28 08:17:01] ...trap/engines/engine.py.230: loss: 0.12256 [I 2022-10-28 08:17:01] ...optimizers/trijoint.py.151: total_norm: 40.887691 [I 2022-10-28 08:17:07] ...trap/engines/engine.py.224: train: epoch 115 | batch 750/2304 [I 2022-10-28 08:17:07] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:14:28 [I 2022-10-28 08:17:07] ...trap/engines/engine.py.229: process: 0.55719 | load: 0.00247 [I 2022-10-28 08:17:07] ...trap/engines/engine.py.230: loss: 0.12385 [I 2022-10-28 08:17:07] ...optimizers/trijoint.py.151: total_norm: 10.772322 [I 2022-10-28 08:17:13] ...trap/engines/engine.py.224: train: epoch 115 | batch 760/2304 [I 2022-10-28 08:17:13] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:23 [I 2022-10-28 08:17:13] ...trap/engines/engine.py.229: process: 0.54115 | load: 0.00250 [I 2022-10-28 08:17:13] ...trap/engines/engine.py.230: loss: 0.08347 [I 2022-10-28 08:17:13] ...optimizers/trijoint.py.151: total_norm: 15.604923 [I 2022-10-28 08:17:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 770/2304 [I 2022-10-28 08:17:18] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:14:16 [I 2022-10-28 08:17:18] ...trap/engines/engine.py.229: process: 0.55159 | load: 0.00246 [I 2022-10-28 08:17:18] ...trap/engines/engine.py.230: loss: 0.11706 [I 2022-10-28 08:17:18] ...optimizers/trijoint.py.151: total_norm: 16.364506 [I 2022-10-28 08:17:24] ...trap/engines/engine.py.224: train: epoch 115 | batch 780/2304 [I 2022-10-28 08:17:24] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:14:09 [I 2022-10-28 08:17:24] ...trap/engines/engine.py.229: process: 0.54935 | load: 0.00246 [I 2022-10-28 08:17:24] ...trap/engines/engine.py.230: loss: 0.12519 [I 2022-10-28 08:17:24] ...optimizers/trijoint.py.151: total_norm: 18.490639 [I 2022-10-28 08:17:29] ...trap/engines/engine.py.224: train: epoch 115 | batch 790/2304 [I 2022-10-28 08:17:29] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:14:07 [I 2022-10-28 08:17:29] ...trap/engines/engine.py.229: process: 0.55709 | load: 0.00248 [I 2022-10-28 08:17:29] ...trap/engines/engine.py.230: loss: 0.16430 [I 2022-10-28 08:17:29] ...optimizers/trijoint.py.151: total_norm: 15.911047 [I 2022-10-28 08:17:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 800/2304 [I 2022-10-28 08:17:35] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:14:03 [I 2022-10-28 08:17:35] ...trap/engines/engine.py.229: process: 0.57448 | load: 0.00241 [I 2022-10-28 08:17:35] ...trap/engines/engine.py.230: loss: 0.11261 [I 2022-10-28 08:17:35] ...optimizers/trijoint.py.151: total_norm: 9.851305 [I 2022-10-28 08:17:41] ...trap/engines/engine.py.224: train: epoch 115 | batch 810/2304 [I 2022-10-28 08:17:41] ...trap/engines/engine.py.228: elapsed: 0:07:38 | left: 0:13:59 [I 2022-10-28 08:17:41] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00245 [I 2022-10-28 08:17:41] ...trap/engines/engine.py.230: loss: 0.09717 [I 2022-10-28 08:17:41] ...optimizers/trijoint.py.151: total_norm: 12.256872 [I 2022-10-28 08:17:46] ...trap/engines/engine.py.224: train: epoch 115 | batch 820/2304 [I 2022-10-28 08:17:46] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:13:45 [I 2022-10-28 08:17:46] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00245 [I 2022-10-28 08:17:46] ...trap/engines/engine.py.230: loss: 0.10859 [I 2022-10-28 08:17:46] ...optimizers/trijoint.py.151: total_norm: 11.095302 [I 2022-10-28 08:17:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 830/2304 [I 2022-10-28 08:17:52] ...trap/engines/engine.py.228: elapsed: 0:07:49 | left: 0:13:44 [I 2022-10-28 08:17:52] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00258 [I 2022-10-28 08:17:52] ...trap/engines/engine.py.230: loss: 0.14259 [I 2022-10-28 08:17:52] ...optimizers/trijoint.py.151: total_norm: 29.373228 [I 2022-10-28 08:17:57] ...trap/engines/engine.py.224: train: epoch 115 | batch 840/2304 [I 2022-10-28 08:17:57] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:13:35 [I 2022-10-28 08:17:57] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00250 [I 2022-10-28 08:17:57] ...trap/engines/engine.py.230: loss: 0.12588 [I 2022-10-28 08:17:57] ...optimizers/trijoint.py.151: total_norm: 18.219763 [I 2022-10-28 08:18:03] ...trap/engines/engine.py.224: train: epoch 115 | batch 850/2304 [I 2022-10-28 08:18:03] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:13:35 [I 2022-10-28 08:18:03] ...trap/engines/engine.py.229: process: 0.55817 | load: 0.00251 [I 2022-10-28 08:18:03] ...trap/engines/engine.py.230: loss: 0.12850 [I 2022-10-28 08:18:03] ...optimizers/trijoint.py.151: total_norm: 38.069630 [I 2022-10-28 08:18:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 860/2304 [I 2022-10-28 08:18:09] ...trap/engines/engine.py.228: elapsed: 0:08:06 | left: 0:13:27 [I 2022-10-28 08:18:09] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00250 [I 2022-10-28 08:18:09] ...trap/engines/engine.py.230: loss: 0.09766 [I 2022-10-28 08:18:09] ...optimizers/trijoint.py.151: total_norm: 9.804444 [I 2022-10-28 08:18:14] ...trap/engines/engine.py.224: train: epoch 115 | batch 870/2304 [I 2022-10-28 08:18:14] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:45 [I 2022-10-28 08:18:14] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00247 [I 2022-10-28 08:18:14] ...trap/engines/engine.py.230: loss: 0.09956 [I 2022-10-28 08:18:14] ...optimizers/trijoint.py.151: total_norm: 13.377772 [I 2022-10-28 08:18:20] ...trap/engines/engine.py.224: train: epoch 115 | batch 880/2304 [I 2022-10-28 08:18:20] ...trap/engines/engine.py.228: elapsed: 0:08:17 | left: 0:13:14 [I 2022-10-28 08:18:20] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00241 [I 2022-10-28 08:18:20] ...trap/engines/engine.py.230: loss: 0.12422 [I 2022-10-28 08:18:20] ...optimizers/trijoint.py.151: total_norm: 26.246782 [I 2022-10-28 08:18:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 890/2304 [I 2022-10-28 08:18:26] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:13:05 [I 2022-10-28 08:18:26] ...trap/engines/engine.py.229: process: 0.56004 | load: 0.00253 [I 2022-10-28 08:18:26] ...trap/engines/engine.py.230: loss: 0.14895 [I 2022-10-28 08:18:26] ...optimizers/trijoint.py.151: total_norm: 60.338375 [I 2022-10-28 08:18:31] ...trap/engines/engine.py.224: train: epoch 115 | batch 900/2304 [I 2022-10-28 08:18:31] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:04 [I 2022-10-28 08:18:31] ...trap/engines/engine.py.229: process: 0.55931 | load: 0.00245 [I 2022-10-28 08:18:31] ...trap/engines/engine.py.230: loss: 0.11584 [I 2022-10-28 08:18:31] ...optimizers/trijoint.py.151: total_norm: 19.842253 [I 2022-10-28 08:18:37] ...trap/engines/engine.py.224: train: epoch 115 | batch 910/2304 [I 2022-10-28 08:18:37] ...trap/engines/engine.py.228: elapsed: 0:08:34 | left: 0:13:02 [I 2022-10-28 08:18:37] ...trap/engines/engine.py.229: process: 0.57225 | load: 0.00245 [I 2022-10-28 08:18:37] ...trap/engines/engine.py.230: loss: 0.09907 [I 2022-10-28 08:18:37] ...optimizers/trijoint.py.151: total_norm: 11.943270 [I 2022-10-28 08:18:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 920/2304 [I 2022-10-28 08:18:42] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:12:49 [I 2022-10-28 08:18:42] ...trap/engines/engine.py.229: process: 0.55297 | load: 0.00245 [I 2022-10-28 08:18:42] ...trap/engines/engine.py.230: loss: 0.10816 [I 2022-10-28 08:18:42] ...optimizers/trijoint.py.151: total_norm: 23.078152 [I 2022-10-28 08:18:48] ...trap/engines/engine.py.224: train: epoch 115 | batch 930/2304 [I 2022-10-28 08:18:48] ...trap/engines/engine.py.228: elapsed: 0:08:45 | left: 0:12:40 [I 2022-10-28 08:18:48] ...trap/engines/engine.py.229: process: 0.54979 | load: 0.00245 [I 2022-10-28 08:18:48] ...trap/engines/engine.py.230: loss: 0.17620 [I 2022-10-28 08:18:48] ...optimizers/trijoint.py.151: total_norm: 17.561161 [I 2022-10-28 08:18:53] ...trap/engines/engine.py.224: train: epoch 115 | batch 940/2304 [I 2022-10-28 08:18:53] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:12:37 [I 2022-10-28 08:18:53] ...trap/engines/engine.py.229: process: 0.54675 | load: 0.00242 [I 2022-10-28 08:18:53] ...trap/engines/engine.py.230: loss: 0.12447 [I 2022-10-28 08:18:53] ...optimizers/trijoint.py.151: total_norm: 13.357080 [I 2022-10-28 08:18:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 950/2304 [I 2022-10-28 08:18:59] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:12:39 [I 2022-10-28 08:18:59] ...trap/engines/engine.py.229: process: 0.56197 | load: 0.00248 [I 2022-10-28 08:18:59] ...trap/engines/engine.py.230: loss: 0.13118 [I 2022-10-28 08:18:59] ...optimizers/trijoint.py.151: total_norm: 12.685619 [I 2022-10-28 08:19:05] ...trap/engines/engine.py.224: train: epoch 115 | batch 960/2304 [I 2022-10-28 08:19:05] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:29 [I 2022-10-28 08:19:05] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00241 [I 2022-10-28 08:19:05] ...trap/engines/engine.py.230: loss: 0.12461 [I 2022-10-28 08:19:05] ...optimizers/trijoint.py.151: total_norm: 15.127980 [I 2022-10-28 08:19:10] ...trap/engines/engine.py.224: train: epoch 115 | batch 970/2304 [I 2022-10-28 08:19:10] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:24 [I 2022-10-28 08:19:10] ...trap/engines/engine.py.229: process: 0.55908 | load: 0.00244 [I 2022-10-28 08:19:10] ...trap/engines/engine.py.230: loss: 0.11459 [I 2022-10-28 08:19:10] ...optimizers/trijoint.py.151: total_norm: 11.332732 [I 2022-10-28 08:19:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 980/2304 [I 2022-10-28 08:19:16] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:14 [I 2022-10-28 08:19:16] ...trap/engines/engine.py.229: process: 0.54678 | load: 0.00243 [I 2022-10-28 08:19:16] ...trap/engines/engine.py.230: loss: 0.08611 [I 2022-10-28 08:19:16] ...optimizers/trijoint.py.151: total_norm: 7.311346 [I 2022-10-28 08:19:21] ...trap/engines/engine.py.224: train: epoch 115 | batch 990/2304 [I 2022-10-28 08:19:21] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:17 [I 2022-10-28 08:19:21] ...trap/engines/engine.py.229: process: 0.57509 | load: 0.00239 [I 2022-10-28 08:19:21] ...trap/engines/engine.py.230: loss: 0.08420 [I 2022-10-28 08:19:21] ...optimizers/trijoint.py.151: total_norm: 15.659555 [I 2022-10-28 08:19:27] ...trap/engines/engine.py.224: train: epoch 115 | batch 1000/2304 [I 2022-10-28 08:19:27] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:11 [I 2022-10-28 08:19:27] ...trap/engines/engine.py.229: process: 0.55373 | load: 0.00241 [I 2022-10-28 08:19:27] ...trap/engines/engine.py.230: loss: 0.11679 [I 2022-10-28 08:19:27] ...optimizers/trijoint.py.151: total_norm: 13.472809 [I 2022-10-28 08:19:33] ...trap/engines/engine.py.224: train: epoch 115 | batch 1010/2304 [I 2022-10-28 08:19:33] ...trap/engines/engine.py.228: elapsed: 0:09:30 | left: 0:12:05 [I 2022-10-28 08:19:33] ...trap/engines/engine.py.229: process: 0.55095 | load: 0.00241 [I 2022-10-28 08:19:33] ...trap/engines/engine.py.230: loss: 0.14717 [I 2022-10-28 08:19:33] ...optimizers/trijoint.py.151: total_norm: 7.866167 [I 2022-10-28 08:19:38] ...trap/engines/engine.py.224: train: epoch 115 | batch 1020/2304 [I 2022-10-28 08:19:38] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:12:01 [I 2022-10-28 08:19:38] ...trap/engines/engine.py.229: process: 0.56779 | load: 0.00249 [I 2022-10-28 08:19:38] ...trap/engines/engine.py.230: loss: 0.09763 [I 2022-10-28 08:19:38] ...optimizers/trijoint.py.151: total_norm: 12.518278 [I 2022-10-28 08:19:44] ...trap/engines/engine.py.224: train: epoch 115 | batch 1030/2304 [I 2022-10-28 08:19:44] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:11:49 [I 2022-10-28 08:19:44] ...trap/engines/engine.py.229: process: 0.56128 | load: 0.00247 [I 2022-10-28 08:19:44] ...trap/engines/engine.py.230: loss: 0.11253 [I 2022-10-28 08:19:44] ...optimizers/trijoint.py.151: total_norm: 18.150045 [I 2022-10-28 08:19:49] ...trap/engines/engine.py.224: train: epoch 115 | batch 1040/2304 [I 2022-10-28 08:19:49] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:11:40 [I 2022-10-28 08:19:49] ...trap/engines/engine.py.229: process: 0.55246 | load: 0.00244 [I 2022-10-28 08:19:49] ...trap/engines/engine.py.230: loss: 0.09895 [I 2022-10-28 08:19:49] ...optimizers/trijoint.py.151: total_norm: 29.670549 [I 2022-10-28 08:19:55] ...trap/engines/engine.py.224: train: epoch 115 | batch 1050/2304 [I 2022-10-28 08:19:55] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:40 [I 2022-10-28 08:19:55] ...trap/engines/engine.py.229: process: 0.56916 | load: 0.00245 [I 2022-10-28 08:19:55] ...trap/engines/engine.py.230: loss: 0.09303 [I 2022-10-28 08:19:55] ...optimizers/trijoint.py.151: total_norm: 12.047143 [I 2022-10-28 08:20:01] ...trap/engines/engine.py.224: train: epoch 115 | batch 1060/2304 [I 2022-10-28 08:20:01] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:33 [I 2022-10-28 08:20:01] ...trap/engines/engine.py.229: process: 0.55611 | load: 0.00243 [I 2022-10-28 08:20:01] ...trap/engines/engine.py.230: loss: 0.14913 [I 2022-10-28 08:20:01] ...optimizers/trijoint.py.151: total_norm: 21.758038 [I 2022-10-28 08:20:06] ...trap/engines/engine.py.224: train: epoch 115 | batch 1070/2304 [I 2022-10-28 08:20:06] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:26 [I 2022-10-28 08:20:06] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00245 [I 2022-10-28 08:20:06] ...trap/engines/engine.py.230: loss: 0.16007 [I 2022-10-28 08:20:06] ...optimizers/trijoint.py.151: total_norm: 15.793599 [I 2022-10-28 08:20:12] ...trap/engines/engine.py.224: train: epoch 115 | batch 1080/2304 [I 2022-10-28 08:20:12] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:20 [I 2022-10-28 08:20:12] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00240 [I 2022-10-28 08:20:12] ...trap/engines/engine.py.230: loss: 0.12661 [I 2022-10-28 08:20:12] ...optimizers/trijoint.py.151: total_norm: 35.148022 [I 2022-10-28 08:20:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 1090/2304 [I 2022-10-28 08:20:17] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:20 [I 2022-10-28 08:20:17] ...trap/engines/engine.py.229: process: 0.56631 | load: 0.00249 [I 2022-10-28 08:20:17] ...trap/engines/engine.py.230: loss: 0.17288 [I 2022-10-28 08:20:17] ...optimizers/trijoint.py.151: total_norm: 11.236760 [I 2022-10-28 08:20:23] ...trap/engines/engine.py.224: train: epoch 115 | batch 1100/2304 [I 2022-10-28 08:20:23] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:13 [I 2022-10-28 08:20:23] ...trap/engines/engine.py.229: process: 0.55682 | load: 0.00248 [I 2022-10-28 08:20:23] ...trap/engines/engine.py.230: loss: 0.09084 [I 2022-10-28 08:20:23] ...optimizers/trijoint.py.151: total_norm: 13.399968 [I 2022-10-28 08:20:29] ...trap/engines/engine.py.224: train: epoch 115 | batch 1110/2304 [I 2022-10-28 08:20:29] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:23 [I 2022-10-28 08:20:29] ...trap/engines/engine.py.229: process: 0.56421 | load: 0.00259 [I 2022-10-28 08:20:29] ...trap/engines/engine.py.230: loss: 0.09898 [I 2022-10-28 08:20:29] ...optimizers/trijoint.py.151: total_norm: 8.855707 [I 2022-10-28 08:20:34] ...trap/engines/engine.py.224: train: epoch 115 | batch 1120/2304 [I 2022-10-28 08:20:34] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:00 [I 2022-10-28 08:20:34] ...trap/engines/engine.py.229: process: 0.56337 | load: 0.00244 [I 2022-10-28 08:20:34] ...trap/engines/engine.py.230: loss: 0.08906 [I 2022-10-28 08:20:34] ...optimizers/trijoint.py.151: total_norm: 18.190346 [I 2022-10-28 08:20:40] ...trap/engines/engine.py.224: train: epoch 115 | batch 1130/2304 [I 2022-10-28 08:20:40] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:10:52 [I 2022-10-28 08:20:40] ...trap/engines/engine.py.229: process: 0.55891 | load: 0.00244 [I 2022-10-28 08:20:40] ...trap/engines/engine.py.230: loss: 0.08788 [I 2022-10-28 08:20:40] ...optimizers/trijoint.py.151: total_norm: 17.853729 [I 2022-10-28 08:20:45] ...trap/engines/engine.py.224: train: epoch 115 | batch 1140/2304 [I 2022-10-28 08:20:45] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:49 [I 2022-10-28 08:20:45] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00247 [I 2022-10-28 08:20:45] ...trap/engines/engine.py.230: loss: 0.14986 [I 2022-10-28 08:20:45] ...optimizers/trijoint.py.151: total_norm: 14.557460 [I 2022-10-28 08:20:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 1150/2304 [I 2022-10-28 08:20:51] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:38 [I 2022-10-28 08:20:51] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00246 [I 2022-10-28 08:20:51] ...trap/engines/engine.py.230: loss: 0.10656 [I 2022-10-28 08:20:51] ...optimizers/trijoint.py.151: total_norm: 19.083019 [I 2022-10-28 08:20:57] ...trap/engines/engine.py.224: train: epoch 115 | batch 1160/2304 [I 2022-10-28 08:20:57] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:39 [I 2022-10-28 08:20:57] ...trap/engines/engine.py.229: process: 0.55637 | load: 0.00244 [I 2022-10-28 08:20:57] ...trap/engines/engine.py.230: loss: 0.09843 [I 2022-10-28 08:20:57] ...optimizers/trijoint.py.151: total_norm: 21.173218 [I 2022-10-28 08:21:02] ...trap/engines/engine.py.224: train: epoch 115 | batch 1170/2304 [I 2022-10-28 08:21:02] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:33 [I 2022-10-28 08:21:02] ...trap/engines/engine.py.229: process: 0.55945 | load: 0.00249 [I 2022-10-28 08:21:02] ...trap/engines/engine.py.230: loss: 0.14345 [I 2022-10-28 08:21:02] ...optimizers/trijoint.py.151: total_norm: 23.286871 [I 2022-10-28 08:21:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 1180/2304 [I 2022-10-28 08:21:08] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:21 [I 2022-10-28 08:21:08] ...trap/engines/engine.py.229: process: 0.53450 | load: 0.00244 [I 2022-10-28 08:21:08] ...trap/engines/engine.py.230: loss: 0.25303 [I 2022-10-28 08:21:08] ...optimizers/trijoint.py.151: total_norm: 10.466037 [I 2022-10-28 08:21:13] ...trap/engines/engine.py.224: train: epoch 115 | batch 1190/2304 [I 2022-10-28 08:21:13] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:21 [I 2022-10-28 08:21:13] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00243 [I 2022-10-28 08:21:13] ...trap/engines/engine.py.230: loss: 0.10421 [I 2022-10-28 08:21:13] ...optimizers/trijoint.py.151: total_norm: 24.498388 [I 2022-10-28 08:21:19] ...trap/engines/engine.py.224: train: epoch 115 | batch 1200/2304 [I 2022-10-28 08:21:19] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:16 [I 2022-10-28 08:21:19] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00244 [I 2022-10-28 08:21:19] ...trap/engines/engine.py.230: loss: 0.10282 [I 2022-10-28 08:21:19] ...optimizers/trijoint.py.151: total_norm: 10.373063 [I 2022-10-28 08:21:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 1210/2304 [I 2022-10-28 08:21:25] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:10 [I 2022-10-28 08:21:25] ...trap/engines/engine.py.229: process: 0.56884 | load: 0.00240 [I 2022-10-28 08:21:25] ...trap/engines/engine.py.230: loss: 0.09733 [I 2022-10-28 08:21:25] ...optimizers/trijoint.py.151: total_norm: 17.222082 [I 2022-10-28 08:21:30] ...trap/engines/engine.py.224: train: epoch 115 | batch 1220/2304 [I 2022-10-28 08:21:30] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:03 [I 2022-10-28 08:21:30] ...trap/engines/engine.py.229: process: 0.54843 | load: 0.00254 [I 2022-10-28 08:21:30] ...trap/engines/engine.py.230: loss: 0.10898 [I 2022-10-28 08:21:30] ...optimizers/trijoint.py.151: total_norm: 18.429399 [I 2022-10-28 08:21:36] ...trap/engines/engine.py.224: train: epoch 115 | batch 1230/2304 [I 2022-10-28 08:21:36] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:15 [I 2022-10-28 08:21:36] ...trap/engines/engine.py.229: process: 0.55902 | load: 0.00250 [I 2022-10-28 08:21:36] ...trap/engines/engine.py.230: loss: 0.10672 [I 2022-10-28 08:21:36] ...optimizers/trijoint.py.151: total_norm: 29.139517 [I 2022-10-28 08:21:41] ...trap/engines/engine.py.224: train: epoch 115 | batch 1240/2304 [I 2022-10-28 08:21:41] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:09:54 [I 2022-10-28 08:21:41] ...trap/engines/engine.py.229: process: 0.55865 | load: 0.00245 [I 2022-10-28 08:21:41] ...trap/engines/engine.py.230: loss: 0.16581 [I 2022-10-28 08:21:41] ...optimizers/trijoint.py.151: total_norm: 37.880856 [I 2022-10-28 08:21:47] ...trap/engines/engine.py.224: train: epoch 115 | batch 1250/2304 [I 2022-10-28 08:21:47] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:09:49 [I 2022-10-28 08:21:47] ...trap/engines/engine.py.229: process: 0.55828 | load: 0.00256 [I 2022-10-28 08:21:47] ...trap/engines/engine.py.230: loss: 0.10197 [I 2022-10-28 08:21:47] ...optimizers/trijoint.py.151: total_norm: 41.657032 [I 2022-10-28 08:21:53] ...trap/engines/engine.py.224: train: epoch 115 | batch 1260/2304 [I 2022-10-28 08:21:53] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:41 [I 2022-10-28 08:21:53] ...trap/engines/engine.py.229: process: 0.56167 | load: 0.00245 [I 2022-10-28 08:21:53] ...trap/engines/engine.py.230: loss: 0.09315 [I 2022-10-28 08:21:53] ...optimizers/trijoint.py.151: total_norm: 10.703105 [I 2022-10-28 08:21:58] ...trap/engines/engine.py.224: train: epoch 115 | batch 1270/2304 [I 2022-10-28 08:21:58] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:34 [I 2022-10-28 08:21:58] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00250 [I 2022-10-28 08:21:58] ...trap/engines/engine.py.230: loss: 0.11079 [I 2022-10-28 08:21:58] ...optimizers/trijoint.py.151: total_norm: 24.364172 [I 2022-10-28 08:22:04] ...trap/engines/engine.py.224: train: epoch 115 | batch 1280/2304 [I 2022-10-28 08:22:04] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:32 [I 2022-10-28 08:22:04] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00248 [I 2022-10-28 08:22:04] ...trap/engines/engine.py.230: loss: 0.11724 [I 2022-10-28 08:22:04] ...optimizers/trijoint.py.151: total_norm: 21.765177 [I 2022-10-28 08:22:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1290/2304 [I 2022-10-28 08:22:09] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:27 [I 2022-10-28 08:22:09] ...trap/engines/engine.py.229: process: 0.57133 | load: 0.00249 [I 2022-10-28 08:22:09] ...trap/engines/engine.py.230: loss: 0.10545 [I 2022-10-28 08:22:09] ...optimizers/trijoint.py.151: total_norm: 15.649286 [I 2022-10-28 08:22:15] ...trap/engines/engine.py.224: train: epoch 115 | batch 1300/2304 [I 2022-10-28 08:22:15] ...trap/engines/engine.py.228: elapsed: 0:12:12 | left: 0:09:21 [I 2022-10-28 08:22:15] ...trap/engines/engine.py.229: process: 0.55073 | load: 0.00249 [I 2022-10-28 08:22:15] ...trap/engines/engine.py.230: loss: 0.08642 [I 2022-10-28 08:22:15] ...optimizers/trijoint.py.151: total_norm: 17.918989 [I 2022-10-28 08:22:21] ...trap/engines/engine.py.224: train: epoch 115 | batch 1310/2304 [I 2022-10-28 08:22:21] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:17 [I 2022-10-28 08:22:21] ...trap/engines/engine.py.229: process: 0.56726 | load: 0.00248 [I 2022-10-28 08:22:21] ...trap/engines/engine.py.230: loss: 0.11181 [I 2022-10-28 08:22:21] ...optimizers/trijoint.py.151: total_norm: 9.937427 [I 2022-10-28 08:22:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 1320/2304 [I 2022-10-28 08:22:26] ...trap/engines/engine.py.228: elapsed: 0:12:23 | left: 0:09:05 [I 2022-10-28 08:22:26] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00246 [I 2022-10-28 08:22:26] ...trap/engines/engine.py.230: loss: 0.09662 [I 2022-10-28 08:22:26] ...optimizers/trijoint.py.151: total_norm: 10.302258 [I 2022-10-28 08:22:32] ...trap/engines/engine.py.224: train: epoch 115 | batch 1330/2304 [I 2022-10-28 08:22:32] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:06 [I 2022-10-28 08:22:32] ...trap/engines/engine.py.229: process: 0.56213 | load: 0.00250 [I 2022-10-28 08:22:32] ...trap/engines/engine.py.230: loss: 0.17527 [I 2022-10-28 08:22:32] ...optimizers/trijoint.py.151: total_norm: 17.229166 [I 2022-10-28 08:22:37] ...trap/engines/engine.py.224: train: epoch 115 | batch 1340/2304 [I 2022-10-28 08:22:37] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:08:55 [I 2022-10-28 08:22:37] ...trap/engines/engine.py.229: process: 0.55147 | load: 0.00245 [I 2022-10-28 08:22:37] ...trap/engines/engine.py.230: loss: 0.12158 [I 2022-10-28 08:22:37] ...optimizers/trijoint.py.151: total_norm: 22.966923 [I 2022-10-28 08:22:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 1350/2304 [I 2022-10-28 08:22:43] ...trap/engines/engine.py.228: elapsed: 0:12:40 | left: 0:08:48 [I 2022-10-28 08:22:43] ...trap/engines/engine.py.229: process: 0.55363 | load: 0.00245 [I 2022-10-28 08:22:43] ...trap/engines/engine.py.230: loss: 0.10080 [I 2022-10-28 08:22:43] ...optimizers/trijoint.py.151: total_norm: 17.623104 [I 2022-10-28 08:22:49] ...trap/engines/engine.py.224: train: epoch 115 | batch 1360/2304 [I 2022-10-28 08:22:49] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:08:45 [I 2022-10-28 08:22:49] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00245 [I 2022-10-28 08:22:49] ...trap/engines/engine.py.230: loss: 0.11763 [I 2022-10-28 08:22:49] ...optimizers/trijoint.py.151: total_norm: 16.890669 [I 2022-10-28 08:22:54] ...trap/engines/engine.py.224: train: epoch 115 | batch 1370/2304 [I 2022-10-28 08:22:54] ...trap/engines/engine.py.228: elapsed: 0:12:51 | left: 0:08:40 [I 2022-10-28 08:22:54] ...trap/engines/engine.py.229: process: 0.55932 | load: 0.00242 [I 2022-10-28 08:22:54] ...trap/engines/engine.py.230: loss: 0.13200 [I 2022-10-28 08:22:54] ...optimizers/trijoint.py.151: total_norm: 29.303497 [I 2022-10-28 08:23:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 1380/2304 [I 2022-10-28 08:23:00] ...trap/engines/engine.py.228: elapsed: 0:12:57 | left: 0:08:36 [I 2022-10-28 08:23:00] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00243 [I 2022-10-28 08:23:00] ...trap/engines/engine.py.230: loss: 0.11237 [I 2022-10-28 08:23:00] ...optimizers/trijoint.py.151: total_norm: 10.964920 [I 2022-10-28 08:23:05] ...trap/engines/engine.py.224: train: epoch 115 | batch 1390/2304 [I 2022-10-28 08:23:05] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:32 [I 2022-10-28 08:23:05] ...trap/engines/engine.py.229: process: 0.57203 | load: 0.00245 [I 2022-10-28 08:23:05] ...trap/engines/engine.py.230: loss: 0.12308 [I 2022-10-28 08:23:05] ...optimizers/trijoint.py.151: total_norm: 13.539924 [I 2022-10-28 08:23:11] ...trap/engines/engine.py.224: train: epoch 115 | batch 1400/2304 [I 2022-10-28 08:23:11] ...trap/engines/engine.py.228: elapsed: 0:13:08 | left: 0:08:28 [I 2022-10-28 08:23:11] ...trap/engines/engine.py.229: process: 0.55990 | load: 0.00245 [I 2022-10-28 08:23:11] ...trap/engines/engine.py.230: loss: 0.17597 [I 2022-10-28 08:23:11] ...optimizers/trijoint.py.151: total_norm: 11.737031 [I 2022-10-28 08:23:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 1410/2304 [I 2022-10-28 08:23:16] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:19 [I 2022-10-28 08:23:16] ...trap/engines/engine.py.229: process: 0.55624 | load: 0.00259 [I 2022-10-28 08:23:16] ...trap/engines/engine.py.230: loss: 0.20480 [I 2022-10-28 08:23:16] ...optimizers/trijoint.py.151: total_norm: 41.246658 [I 2022-10-28 08:23:22] ...trap/engines/engine.py.224: train: epoch 115 | batch 1420/2304 [I 2022-10-28 08:23:22] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:09 [I 2022-10-28 08:23:22] ...trap/engines/engine.py.229: process: 0.54815 | load: 0.00248 [I 2022-10-28 08:23:22] ...trap/engines/engine.py.230: loss: 0.12627 [I 2022-10-28 08:23:22] ...optimizers/trijoint.py.151: total_norm: 10.152120 [I 2022-10-28 08:23:28] ...trap/engines/engine.py.224: train: epoch 115 | batch 1430/2304 [I 2022-10-28 08:23:28] ...trap/engines/engine.py.228: elapsed: 0:13:25 | left: 0:08:07 [I 2022-10-28 08:23:28] ...trap/engines/engine.py.229: process: 0.55734 | load: 0.00248 [I 2022-10-28 08:23:28] ...trap/engines/engine.py.230: loss: 0.11841 [I 2022-10-28 08:23:28] ...optimizers/trijoint.py.151: total_norm: 12.186711 [I 2022-10-28 08:23:33] ...trap/engines/engine.py.224: train: epoch 115 | batch 1440/2304 [I 2022-10-28 08:23:33] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:08:01 [I 2022-10-28 08:23:33] ...trap/engines/engine.py.229: process: 0.55245 | load: 0.00244 [I 2022-10-28 08:23:33] ...trap/engines/engine.py.230: loss: 0.09676 [I 2022-10-28 08:23:33] ...optimizers/trijoint.py.151: total_norm: 17.261778 [I 2022-10-28 08:23:39] ...trap/engines/engine.py.224: train: epoch 115 | batch 1450/2304 [I 2022-10-28 08:23:39] ...trap/engines/engine.py.228: elapsed: 0:13:36 | left: 0:07:54 [I 2022-10-28 08:23:39] ...trap/engines/engine.py.229: process: 0.54730 | load: 0.00247 [I 2022-10-28 08:23:39] ...trap/engines/engine.py.230: loss: 0.17529 [I 2022-10-28 08:23:39] ...optimizers/trijoint.py.151: total_norm: 17.941654 [I 2022-10-28 08:23:44] ...trap/engines/engine.py.224: train: epoch 115 | batch 1460/2304 [I 2022-10-28 08:23:44] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:07:45 [I 2022-10-28 08:23:44] ...trap/engines/engine.py.229: process: 0.54035 | load: 0.00245 [I 2022-10-28 08:23:44] ...trap/engines/engine.py.230: loss: 0.10218 [I 2022-10-28 08:23:44] ...optimizers/trijoint.py.151: total_norm: 7.870191 [I 2022-10-28 08:23:50] ...trap/engines/engine.py.224: train: epoch 115 | batch 1470/2304 [I 2022-10-28 08:23:50] ...trap/engines/engine.py.228: elapsed: 0:13:47 | left: 0:07:44 [I 2022-10-28 08:23:50] ...trap/engines/engine.py.229: process: 0.55064 | load: 0.00245 [I 2022-10-28 08:23:50] ...trap/engines/engine.py.230: loss: 0.08651 [I 2022-10-28 08:23:50] ...optimizers/trijoint.py.151: total_norm: 14.383782 [I 2022-10-28 08:23:56] ...trap/engines/engine.py.224: train: epoch 115 | batch 1480/2304 [I 2022-10-28 08:23:56] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:39 [I 2022-10-28 08:23:56] ...trap/engines/engine.py.229: process: 0.55111 | load: 0.00242 [I 2022-10-28 08:23:56] ...trap/engines/engine.py.230: loss: 0.07300 [I 2022-10-28 08:23:56] ...optimizers/trijoint.py.151: total_norm: 12.988836 [I 2022-10-28 08:24:01] ...trap/engines/engine.py.224: train: epoch 115 | batch 1490/2304 [I 2022-10-28 08:24:01] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:35 [I 2022-10-28 08:24:01] ...trap/engines/engine.py.229: process: 0.56034 | load: 0.00246 [I 2022-10-28 08:24:01] ...trap/engines/engine.py.230: loss: 0.11363 [I 2022-10-28 08:24:01] ...optimizers/trijoint.py.151: total_norm: 13.435274 [I 2022-10-28 08:24:07] ...trap/engines/engine.py.224: train: epoch 115 | batch 1500/2304 [I 2022-10-28 08:24:07] ...trap/engines/engine.py.228: elapsed: 0:14:04 | left: 0:07:26 [I 2022-10-28 08:24:07] ...trap/engines/engine.py.229: process: 0.55047 | load: 0.00248 [I 2022-10-28 08:24:07] ...trap/engines/engine.py.230: loss: 0.10456 [I 2022-10-28 08:24:07] ...optimizers/trijoint.py.151: total_norm: 9.977811 [I 2022-10-28 08:24:12] ...trap/engines/engine.py.224: train: epoch 115 | batch 1510/2304 [I 2022-10-28 08:24:12] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:22 [I 2022-10-28 08:24:12] ...trap/engines/engine.py.229: process: 0.54962 | load: 0.00251 [I 2022-10-28 08:24:12] ...trap/engines/engine.py.230: loss: 0.13027 [I 2022-10-28 08:24:12] ...optimizers/trijoint.py.151: total_norm: 13.375797 [I 2022-10-28 08:24:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 1520/2304 [I 2022-10-28 08:24:18] ...trap/engines/engine.py.228: elapsed: 0:14:15 | left: 0:07:15 [I 2022-10-28 08:24:18] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00243 [I 2022-10-28 08:24:18] ...trap/engines/engine.py.230: loss: 0.10427 [I 2022-10-28 08:24:18] ...optimizers/trijoint.py.151: total_norm: 10.585826 [I 2022-10-28 08:24:24] ...trap/engines/engine.py.224: train: epoch 115 | batch 1530/2304 [I 2022-10-28 08:24:24] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:13 [I 2022-10-28 08:24:24] ...trap/engines/engine.py.229: process: 0.55837 | load: 0.00246 [I 2022-10-28 08:24:24] ...trap/engines/engine.py.230: loss: 0.16815 [I 2022-10-28 08:24:24] ...optimizers/trijoint.py.151: total_norm: 10.547082 [I 2022-10-28 08:24:29] ...trap/engines/engine.py.224: train: epoch 115 | batch 1540/2304 [I 2022-10-28 08:24:29] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:06 [I 2022-10-28 08:24:29] ...trap/engines/engine.py.229: process: 0.55045 | load: 0.00245 [I 2022-10-28 08:24:29] ...trap/engines/engine.py.230: loss: 0.11440 [I 2022-10-28 08:24:29] ...optimizers/trijoint.py.151: total_norm: 17.343164 [I 2022-10-28 08:24:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 1550/2304 [I 2022-10-28 08:24:35] ...trap/engines/engine.py.228: elapsed: 0:14:32 | left: 0:06:59 [I 2022-10-28 08:24:35] ...trap/engines/engine.py.229: process: 0.56748 | load: 0.00245 [I 2022-10-28 08:24:35] ...trap/engines/engine.py.230: loss: 0.10412 [I 2022-10-28 08:24:35] ...optimizers/trijoint.py.151: total_norm: 44.725426 [I 2022-10-28 08:24:40] ...trap/engines/engine.py.224: train: epoch 115 | batch 1560/2304 [I 2022-10-28 08:24:40] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:06:56 [I 2022-10-28 08:24:40] ...trap/engines/engine.py.229: process: 0.56248 | load: 0.00244 [I 2022-10-28 08:24:40] ...trap/engines/engine.py.230: loss: 0.10336 [I 2022-10-28 08:24:40] ...optimizers/trijoint.py.151: total_norm: 12.072210 [I 2022-10-28 08:24:46] ...trap/engines/engine.py.224: train: epoch 115 | batch 1570/2304 [I 2022-10-28 08:24:46] ...trap/engines/engine.py.228: elapsed: 0:14:43 | left: 0:06:48 [I 2022-10-28 08:24:46] ...trap/engines/engine.py.229: process: 0.55324 | load: 0.00242 [I 2022-10-28 08:24:46] ...trap/engines/engine.py.230: loss: 0.12613 [I 2022-10-28 08:24:46] ...optimizers/trijoint.py.151: total_norm: 12.478937 [I 2022-10-28 08:24:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 1580/2304 [I 2022-10-28 08:24:51] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:06:41 [I 2022-10-28 08:24:51] ...trap/engines/engine.py.229: process: 0.54594 | load: 0.00242 [I 2022-10-28 08:24:51] ...trap/engines/engine.py.230: loss: 0.11727 [I 2022-10-28 08:24:51] ...optimizers/trijoint.py.151: total_norm: 13.205992 [I 2022-10-28 08:24:57] ...trap/engines/engine.py.224: train: epoch 115 | batch 1590/2304 [I 2022-10-28 08:24:57] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:40 [I 2022-10-28 08:24:57] ...trap/engines/engine.py.229: process: 0.56537 | load: 0.00245 [I 2022-10-28 08:24:57] ...trap/engines/engine.py.230: loss: 0.15750 [I 2022-10-28 08:24:57] ...optimizers/trijoint.py.151: total_norm: 16.662428 [I 2022-10-28 08:25:03] ...trap/engines/engine.py.224: train: epoch 115 | batch 1600/2304 [I 2022-10-28 08:25:03] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:33 [I 2022-10-28 08:25:03] ...trap/engines/engine.py.229: process: 0.55997 | load: 0.00246 [I 2022-10-28 08:25:03] ...trap/engines/engine.py.230: loss: 0.11603 [I 2022-10-28 08:25:03] ...optimizers/trijoint.py.151: total_norm: 10.459418 [I 2022-10-28 08:25:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 1610/2304 [I 2022-10-28 08:25:08] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:26 [I 2022-10-28 08:25:08] ...trap/engines/engine.py.229: process: 0.54844 | load: 0.00243 [I 2022-10-28 08:25:08] ...trap/engines/engine.py.230: loss: 0.10307 [I 2022-10-28 08:25:08] ...optimizers/trijoint.py.151: total_norm: 10.806895 [I 2022-10-28 08:25:14] ...trap/engines/engine.py.224: train: epoch 115 | batch 1620/2304 [I 2022-10-28 08:25:14] ...trap/engines/engine.py.228: elapsed: 0:15:11 | left: 0:06:20 [I 2022-10-28 08:25:14] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00490 [I 2022-10-28 08:25:14] ...trap/engines/engine.py.230: loss: 0.10588 [I 2022-10-28 08:25:14] ...optimizers/trijoint.py.151: total_norm: 13.136063 [I 2022-10-28 08:25:19] ...trap/engines/engine.py.224: train: epoch 115 | batch 1630/2304 [I 2022-10-28 08:25:19] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:13 [I 2022-10-28 08:25:19] ...trap/engines/engine.py.229: process: 0.54558 | load: 0.00245 [I 2022-10-28 08:25:19] ...trap/engines/engine.py.230: loss: 0.10611 [I 2022-10-28 08:25:19] ...optimizers/trijoint.py.151: total_norm: 10.205825 [I 2022-10-28 08:25:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 1640/2304 [I 2022-10-28 08:25:25] ...trap/engines/engine.py.228: elapsed: 0:15:22 | left: 0:06:10 [I 2022-10-28 08:25:25] ...trap/engines/engine.py.229: process: 0.55285 | load: 0.00242 [I 2022-10-28 08:25:25] ...trap/engines/engine.py.230: loss: 0.12602 [I 2022-10-28 08:25:25] ...optimizers/trijoint.py.151: total_norm: 13.498402 [I 2022-10-28 08:25:30] ...trap/engines/engine.py.224: train: epoch 115 | batch 1650/2304 [I 2022-10-28 08:25:30] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:01 [I 2022-10-28 08:25:30] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00253 [I 2022-10-28 08:25:30] ...trap/engines/engine.py.230: loss: 0.10905 [I 2022-10-28 08:25:30] ...optimizers/trijoint.py.151: total_norm: 17.948643 [I 2022-10-28 08:25:36] ...trap/engines/engine.py.224: train: epoch 115 | batch 1660/2304 [I 2022-10-28 08:25:36] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:05:57 [I 2022-10-28 08:25:36] ...trap/engines/engine.py.229: process: 0.55370 | load: 0.00247 [I 2022-10-28 08:25:36] ...trap/engines/engine.py.230: loss: 0.13834 [I 2022-10-28 08:25:36] ...optimizers/trijoint.py.151: total_norm: 11.947104 [I 2022-10-28 08:25:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 1670/2304 [I 2022-10-28 08:25:42] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:52 [I 2022-10-28 08:25:42] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00264 [I 2022-10-28 08:25:42] ...trap/engines/engine.py.230: loss: 0.09265 [I 2022-10-28 08:25:42] ...optimizers/trijoint.py.151: total_norm: 20.616087 [I 2022-10-28 08:25:47] ...trap/engines/engine.py.224: train: epoch 115 | batch 1680/2304 [I 2022-10-28 08:25:47] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:05:47 [I 2022-10-28 08:25:47] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00260 [I 2022-10-28 08:25:47] ...trap/engines/engine.py.230: loss: 0.19063 [I 2022-10-28 08:25:47] ...optimizers/trijoint.py.151: total_norm: 39.124752 [I 2022-10-28 08:25:53] ...trap/engines/engine.py.224: train: epoch 115 | batch 1690/2304 [I 2022-10-28 08:25:53] ...trap/engines/engine.py.228: elapsed: 0:15:50 | left: 0:05:41 [I 2022-10-28 08:25:53] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00248 [I 2022-10-28 08:25:53] ...trap/engines/engine.py.230: loss: 0.12736 [I 2022-10-28 08:25:53] ...optimizers/trijoint.py.151: total_norm: 15.387644 [I 2022-10-28 08:25:58] ...trap/engines/engine.py.224: train: epoch 115 | batch 1700/2304 [I 2022-10-28 08:25:58] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:35 [I 2022-10-28 08:25:58] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00248 [I 2022-10-28 08:25:58] ...trap/engines/engine.py.230: loss: 0.11026 [I 2022-10-28 08:25:58] ...optimizers/trijoint.py.151: total_norm: 8.403658 [I 2022-10-28 08:26:04] ...trap/engines/engine.py.224: train: epoch 115 | batch 1710/2304 [I 2022-10-28 08:26:04] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:30 [I 2022-10-28 08:26:04] ...trap/engines/engine.py.229: process: 0.56297 | load: 0.00248 [I 2022-10-28 08:26:04] ...trap/engines/engine.py.230: loss: 0.10912 [I 2022-10-28 08:26:04] ...optimizers/trijoint.py.151: total_norm: 10.020296 [I 2022-10-28 08:26:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 1720/2304 [I 2022-10-28 08:26:09] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:24 [I 2022-10-28 08:26:09] ...trap/engines/engine.py.229: process: 0.56036 | load: 0.00246 [I 2022-10-28 08:26:09] ...trap/engines/engine.py.230: loss: 0.12615 [I 2022-10-28 08:26:09] ...optimizers/trijoint.py.151: total_norm: 46.856972 [I 2022-10-28 08:26:15] ...trap/engines/engine.py.224: train: epoch 115 | batch 1730/2304 [I 2022-10-28 08:26:15] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:17 [I 2022-10-28 08:26:15] ...trap/engines/engine.py.229: process: 0.55261 | load: 0.00250 [I 2022-10-28 08:26:15] ...trap/engines/engine.py.230: loss: 0.08391 [I 2022-10-28 08:26:15] ...optimizers/trijoint.py.151: total_norm: 18.717005 [I 2022-10-28 08:26:21] ...trap/engines/engine.py.224: train: epoch 115 | batch 1740/2304 [I 2022-10-28 08:26:21] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:14 [I 2022-10-28 08:26:21] ...trap/engines/engine.py.229: process: 0.55446 | load: 0.00248 [I 2022-10-28 08:26:21] ...trap/engines/engine.py.230: loss: 0.12700 [I 2022-10-28 08:26:21] ...optimizers/trijoint.py.151: total_norm: 11.688233 [I 2022-10-28 08:26:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 1750/2304 [I 2022-10-28 08:26:26] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:10 [I 2022-10-28 08:26:26] ...trap/engines/engine.py.229: process: 0.56757 | load: 0.00246 [I 2022-10-28 08:26:26] ...trap/engines/engine.py.230: loss: 0.13721 [I 2022-10-28 08:26:26] ...optimizers/trijoint.py.151: total_norm: 28.988167 [I 2022-10-28 08:26:32] ...trap/engines/engine.py.224: train: epoch 115 | batch 1760/2304 [I 2022-10-28 08:26:32] ...trap/engines/engine.py.228: elapsed: 0:16:29 | left: 0:04:59 [I 2022-10-28 08:26:32] ...trap/engines/engine.py.229: process: 0.53810 | load: 0.00244 [I 2022-10-28 08:26:32] ...trap/engines/engine.py.230: loss: 0.12412 [I 2022-10-28 08:26:32] ...optimizers/trijoint.py.151: total_norm: 10.146976 [I 2022-10-28 08:26:37] ...trap/engines/engine.py.224: train: epoch 115 | batch 1770/2304 [I 2022-10-28 08:26:37] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:04:57 [I 2022-10-28 08:26:37] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00244 [I 2022-10-28 08:26:37] ...trap/engines/engine.py.230: loss: 0.09258 [I 2022-10-28 08:26:37] ...optimizers/trijoint.py.151: total_norm: 14.164162 [I 2022-10-28 08:26:43] ...trap/engines/engine.py.224: train: epoch 115 | batch 1780/2304 [I 2022-10-28 08:26:43] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:04:51 [I 2022-10-28 08:26:43] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00247 [I 2022-10-28 08:26:43] ...trap/engines/engine.py.230: loss: 0.15052 [I 2022-10-28 08:26:43] ...optimizers/trijoint.py.151: total_norm: 14.558050 [I 2022-10-28 08:26:48] ...trap/engines/engine.py.224: train: epoch 115 | batch 1790/2304 [I 2022-10-28 08:26:48] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:46 [I 2022-10-28 08:26:48] ...trap/engines/engine.py.229: process: 0.55314 | load: 0.00247 [I 2022-10-28 08:26:48] ...trap/engines/engine.py.230: loss: 0.18951 [I 2022-10-28 08:26:48] ...optimizers/trijoint.py.151: total_norm: 13.401337 [I 2022-10-28 08:26:54] ...trap/engines/engine.py.224: train: epoch 115 | batch 1800/2304 [I 2022-10-28 08:26:54] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:53 [I 2022-10-28 08:26:54] ...trap/engines/engine.py.229: process: 0.60099 | load: 0.00245 [I 2022-10-28 08:26:54] ...trap/engines/engine.py.230: loss: 0.11808 [I 2022-10-28 08:26:54] ...optimizers/trijoint.py.151: total_norm: 11.499002 [I 2022-10-28 08:27:00] ...trap/engines/engine.py.224: train: epoch 115 | batch 1810/2304 [I 2022-10-28 08:27:00] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:37 [I 2022-10-28 08:27:00] ...trap/engines/engine.py.229: process: 0.57329 | load: 0.00244 [I 2022-10-28 08:27:00] ...trap/engines/engine.py.230: loss: 0.16258 [I 2022-10-28 08:27:00] ...optimizers/trijoint.py.151: total_norm: 79.384300 [I 2022-10-28 08:27:05] ...trap/engines/engine.py.224: train: epoch 115 | batch 1820/2304 [I 2022-10-28 08:27:05] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:31 [I 2022-10-28 08:27:05] ...trap/engines/engine.py.229: process: 0.55792 | load: 0.00241 [I 2022-10-28 08:27:05] ...trap/engines/engine.py.230: loss: 0.13494 [I 2022-10-28 08:27:05] ...optimizers/trijoint.py.151: total_norm: 15.183275 [I 2022-10-28 08:27:11] ...trap/engines/engine.py.224: train: epoch 115 | batch 1830/2304 [I 2022-10-28 08:27:11] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:24 [I 2022-10-28 08:27:11] ...trap/engines/engine.py.229: process: 0.55957 | load: 0.00244 [I 2022-10-28 08:27:11] ...trap/engines/engine.py.230: loss: 0.09807 [I 2022-10-28 08:27:11] ...optimizers/trijoint.py.151: total_norm: 9.877645 [I 2022-10-28 08:27:17] ...trap/engines/engine.py.224: train: epoch 115 | batch 1840/2304 [I 2022-10-28 08:27:17] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:17 [I 2022-10-28 08:27:17] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00247 [I 2022-10-28 08:27:17] ...trap/engines/engine.py.230: loss: 0.07292 [I 2022-10-28 08:27:17] ...optimizers/trijoint.py.151: total_norm: 8.092700 [I 2022-10-28 08:27:22] ...trap/engines/engine.py.224: train: epoch 115 | batch 1850/2304 [I 2022-10-28 08:27:22] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:12 [I 2022-10-28 08:27:22] ...trap/engines/engine.py.229: process: 0.56303 | load: 0.00249 [I 2022-10-28 08:27:22] ...trap/engines/engine.py.230: loss: 0.08577 [I 2022-10-28 08:27:22] ...optimizers/trijoint.py.151: total_norm: 8.662431 [I 2022-10-28 08:27:28] ...trap/engines/engine.py.224: train: epoch 115 | batch 1860/2304 [I 2022-10-28 08:27:28] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:07 [I 2022-10-28 08:27:28] ...trap/engines/engine.py.229: process: 0.55057 | load: 0.00245 [I 2022-10-28 08:27:28] ...trap/engines/engine.py.230: loss: 0.10824 [I 2022-10-28 08:27:28] ...optimizers/trijoint.py.151: total_norm: 20.571978 [I 2022-10-28 08:27:33] ...trap/engines/engine.py.224: train: epoch 115 | batch 1870/2304 [I 2022-10-28 08:27:33] ...trap/engines/engine.py.228: elapsed: 0:17:30 | left: 0:04:01 [I 2022-10-28 08:27:33] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00246 [I 2022-10-28 08:27:33] ...trap/engines/engine.py.230: loss: 0.09398 [I 2022-10-28 08:27:33] ...optimizers/trijoint.py.151: total_norm: 16.634251 [I 2022-10-28 08:27:39] ...trap/engines/engine.py.224: train: epoch 115 | batch 1880/2304 [I 2022-10-28 08:27:39] ...trap/engines/engine.py.228: elapsed: 0:17:36 | left: 0:03:55 [I 2022-10-28 08:27:39] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00244 [I 2022-10-28 08:27:39] ...trap/engines/engine.py.230: loss: 0.10669 [I 2022-10-28 08:27:39] ...optimizers/trijoint.py.151: total_norm: 11.573457 [I 2022-10-28 08:27:45] ...trap/engines/engine.py.224: train: epoch 115 | batch 1890/2304 [I 2022-10-28 08:27:45] ...trap/engines/engine.py.228: elapsed: 0:17:42 | left: 0:03:53 [I 2022-10-28 08:27:45] ...trap/engines/engine.py.229: process: 0.55934 | load: 0.00244 [I 2022-10-28 08:27:45] ...trap/engines/engine.py.230: loss: 0.10875 [I 2022-10-28 08:27:45] ...optimizers/trijoint.py.151: total_norm: 22.192062 [I 2022-10-28 08:27:50] ...trap/engines/engine.py.224: train: epoch 115 | batch 1900/2304 [I 2022-10-28 08:27:50] ...trap/engines/engine.py.228: elapsed: 0:17:47 | left: 0:03:44 [I 2022-10-28 08:27:50] ...trap/engines/engine.py.229: process: 0.55642 | load: 0.00248 [I 2022-10-28 08:27:50] ...trap/engines/engine.py.230: loss: 0.19868 [I 2022-10-28 08:27:50] ...optimizers/trijoint.py.151: total_norm: 20.338057 [I 2022-10-28 08:27:56] ...trap/engines/engine.py.224: train: epoch 115 | batch 1910/2304 [I 2022-10-28 08:27:56] ...trap/engines/engine.py.228: elapsed: 0:17:53 | left: 0:03:40 [I 2022-10-28 08:27:56] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00248 [I 2022-10-28 08:27:56] ...trap/engines/engine.py.230: loss: 0.14840 [I 2022-10-28 08:27:56] ...optimizers/trijoint.py.151: total_norm: 37.921318 [I 2022-10-28 08:28:01] ...trap/engines/engine.py.224: train: epoch 115 | batch 1920/2304 [I 2022-10-28 08:28:01] ...trap/engines/engine.py.228: elapsed: 0:17:58 | left: 0:03:34 [I 2022-10-28 08:28:01] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00246 [I 2022-10-28 08:28:01] ...trap/engines/engine.py.230: loss: 0.08757 [I 2022-10-28 08:28:01] ...optimizers/trijoint.py.151: total_norm: 11.795459 [I 2022-10-28 08:28:07] ...trap/engines/engine.py.224: train: epoch 115 | batch 1930/2304 [I 2022-10-28 08:28:07] ...trap/engines/engine.py.228: elapsed: 0:18:04 | left: 0:03:28 [I 2022-10-28 08:28:07] ...trap/engines/engine.py.229: process: 0.56015 | load: 0.00246 [I 2022-10-28 08:28:07] ...trap/engines/engine.py.230: loss: 0.12083 [I 2022-10-28 08:28:07] ...optimizers/trijoint.py.151: total_norm: 34.195271 [I 2022-10-28 08:28:13] ...trap/engines/engine.py.224: train: epoch 115 | batch 1940/2304 [I 2022-10-28 08:28:13] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:24 [I 2022-10-28 08:28:13] ...trap/engines/engine.py.229: process: 0.55984 | load: 0.00248 [I 2022-10-28 08:28:13] ...trap/engines/engine.py.230: loss: 0.10672 [I 2022-10-28 08:28:13] ...optimizers/trijoint.py.151: total_norm: 9.693299 [I 2022-10-28 08:28:18] ...trap/engines/engine.py.224: train: epoch 115 | batch 1950/2304 [I 2022-10-28 08:28:18] ...trap/engines/engine.py.228: elapsed: 0:18:15 | left: 0:03:18 [I 2022-10-28 08:28:18] ...trap/engines/engine.py.229: process: 0.56125 | load: 0.00245 [I 2022-10-28 08:28:18] ...trap/engines/engine.py.230: loss: 0.14101 [I 2022-10-28 08:28:18] ...optimizers/trijoint.py.151: total_norm: 18.410576 [I 2022-10-28 08:28:24] ...trap/engines/engine.py.224: train: epoch 115 | batch 1960/2304 [I 2022-10-28 08:28:24] ...trap/engines/engine.py.228: elapsed: 0:18:21 | left: 0:03:15 [I 2022-10-28 08:28:24] ...trap/engines/engine.py.229: process: 0.54598 | load: 0.00245 [I 2022-10-28 08:28:24] ...trap/engines/engine.py.230: loss: 0.08629 [I 2022-10-28 08:28:24] ...optimizers/trijoint.py.151: total_norm: 12.868662 [I 2022-10-28 08:28:30] ...trap/engines/engine.py.224: train: epoch 115 | batch 1970/2304 [I 2022-10-28 08:28:30] ...trap/engines/engine.py.228: elapsed: 0:18:26 | left: 0:03:06 [I 2022-10-28 08:28:30] ...trap/engines/engine.py.229: process: 0.56385 | load: 0.00243 [I 2022-10-28 08:28:30] ...trap/engines/engine.py.230: loss: 0.10012 [I 2022-10-28 08:28:30] ...optimizers/trijoint.py.151: total_norm: 8.586449 [I 2022-10-28 08:28:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 1980/2304 [I 2022-10-28 08:28:35] ...trap/engines/engine.py.228: elapsed: 0:18:32 | left: 0:03:00 [I 2022-10-28 08:28:35] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00246 [I 2022-10-28 08:28:35] ...trap/engines/engine.py.230: loss: 0.13500 [I 2022-10-28 08:28:35] ...optimizers/trijoint.py.151: total_norm: 13.653154 [I 2022-10-28 08:28:41] ...trap/engines/engine.py.224: train: epoch 115 | batch 1990/2304 [I 2022-10-28 08:28:41] ...trap/engines/engine.py.228: elapsed: 0:18:38 | left: 0:02:53 [I 2022-10-28 08:28:41] ...trap/engines/engine.py.229: process: 0.55099 | load: 0.00244 [I 2022-10-28 08:28:41] ...trap/engines/engine.py.230: loss: 0.07585 [I 2022-10-28 08:28:41] ...optimizers/trijoint.py.151: total_norm: 11.655465 [I 2022-10-28 08:28:46] ...trap/engines/engine.py.224: train: epoch 115 | batch 2000/2304 [I 2022-10-28 08:28:46] ...trap/engines/engine.py.228: elapsed: 0:18:43 | left: 0:02:49 [I 2022-10-28 08:28:46] ...trap/engines/engine.py.229: process: 0.57096 | load: 0.00243 [I 2022-10-28 08:28:46] ...trap/engines/engine.py.230: loss: 0.14883 [I 2022-10-28 08:28:46] ...optimizers/trijoint.py.151: total_norm: 9.754401 [I 2022-10-28 08:28:52] ...trap/engines/engine.py.224: train: epoch 115 | batch 2010/2304 [I 2022-10-28 08:28:52] ...trap/engines/engine.py.228: elapsed: 0:18:49 | left: 0:02:43 [I 2022-10-28 08:28:52] ...trap/engines/engine.py.229: process: 0.54662 | load: 0.00254 [I 2022-10-28 08:28:52] ...trap/engines/engine.py.230: loss: 0.13294 [I 2022-10-28 08:28:52] ...optimizers/trijoint.py.151: total_norm: 19.387991 [I 2022-10-28 08:28:57] ...trap/engines/engine.py.224: train: epoch 115 | batch 2020/2304 [I 2022-10-28 08:28:57] ...trap/engines/engine.py.228: elapsed: 0:18:54 | left: 0:02:38 [I 2022-10-28 08:28:57] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00245 [I 2022-10-28 08:28:57] ...trap/engines/engine.py.230: loss: 0.11534 [I 2022-10-28 08:28:57] ...optimizers/trijoint.py.151: total_norm: 13.864525 [I 2022-10-28 08:29:03] ...trap/engines/engine.py.224: train: epoch 115 | batch 2030/2304 [I 2022-10-28 08:29:03] ...trap/engines/engine.py.228: elapsed: 0:19:00 | left: 0:02:33 [I 2022-10-28 08:29:03] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00244 [I 2022-10-28 08:29:03] ...trap/engines/engine.py.230: loss: 0.11804 [I 2022-10-28 08:29:03] ...optimizers/trijoint.py.151: total_norm: 21.738134 [I 2022-10-28 08:29:09] ...trap/engines/engine.py.224: train: epoch 115 | batch 2040/2304 [I 2022-10-28 08:29:09] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:27 [I 2022-10-28 08:29:09] ...trap/engines/engine.py.229: process: 0.56350 | load: 0.00252 [I 2022-10-28 08:29:09] ...trap/engines/engine.py.230: loss: 0.08516 [I 2022-10-28 08:29:09] ...optimizers/trijoint.py.151: total_norm: 13.736764 [I 2022-10-28 08:29:14] ...trap/engines/engine.py.224: train: epoch 115 | batch 2050/2304 [I 2022-10-28 08:29:14] ...trap/engines/engine.py.228: elapsed: 0:19:11 | left: 0:02:22 [I 2022-10-28 08:29:14] ...trap/engines/engine.py.229: process: 0.55731 | load: 0.00244 [I 2022-10-28 08:29:14] ...trap/engines/engine.py.230: loss: 0.10789 [I 2022-10-28 08:29:14] ...optimizers/trijoint.py.151: total_norm: 16.370810 [I 2022-10-28 08:29:20] ...trap/engines/engine.py.224: train: epoch 115 | batch 2060/2304 [I 2022-10-28 08:29:20] ...trap/engines/engine.py.228: elapsed: 0:19:17 | left: 0:02:18 [I 2022-10-28 08:29:20] ...trap/engines/engine.py.229: process: 0.57074 | load: 0.00257 [I 2022-10-28 08:29:20] ...trap/engines/engine.py.230: loss: 0.10527 [I 2022-10-28 08:29:20] ...optimizers/trijoint.py.151: total_norm: 16.430773 [I 2022-10-28 08:29:26] ...trap/engines/engine.py.224: train: epoch 115 | batch 2070/2304 [I 2022-10-28 08:29:26] ...trap/engines/engine.py.228: elapsed: 0:19:22 | left: 0:02:14 [I 2022-10-28 08:29:26] ...trap/engines/engine.py.229: process: 0.55569 | load: 0.00259 [I 2022-10-28 08:29:26] ...trap/engines/engine.py.230: loss: 0.11849 [I 2022-10-28 08:29:26] ...optimizers/trijoint.py.151: total_norm: 17.706268 [I 2022-10-28 08:29:31] ...trap/engines/engine.py.224: train: epoch 115 | batch 2080/2304 [I 2022-10-28 08:29:31] ...trap/engines/engine.py.228: elapsed: 0:19:28 | left: 0:02:05 [I 2022-10-28 08:29:31] ...trap/engines/engine.py.229: process: 0.55102 | load: 0.00250 [I 2022-10-28 08:29:31] ...trap/engines/engine.py.230: loss: 0.07957 [I 2022-10-28 08:29:31] ...optimizers/trijoint.py.151: total_norm: 10.327609 [I 2022-10-28 08:29:37] ...trap/engines/engine.py.224: train: epoch 115 | batch 2090/2304 [I 2022-10-28 08:29:37] ...trap/engines/engine.py.228: elapsed: 0:19:34 | left: 0:01:59 [I 2022-10-28 08:29:37] ...trap/engines/engine.py.229: process: 0.54175 | load: 0.00250 [I 2022-10-28 08:29:37] ...trap/engines/engine.py.230: loss: 0.13049 [I 2022-10-28 08:29:37] ...optimizers/trijoint.py.151: total_norm: 20.584320 [I 2022-10-28 08:29:42] ...trap/engines/engine.py.224: train: epoch 115 | batch 2100/2304 [I 2022-10-28 08:29:42] ...trap/engines/engine.py.228: elapsed: 0:19:39 | left: 0:01:54 [I 2022-10-28 08:29:42] ...trap/engines/engine.py.229: process: 0.54985 | load: 0.00245 [I 2022-10-28 08:29:42] ...trap/engines/engine.py.230: loss: 0.09902 [I 2022-10-28 08:29:42] ...optimizers/trijoint.py.151: total_norm: 14.409657 [I 2022-10-28 08:29:48] ...trap/engines/engine.py.224: train: epoch 115 | batch 2110/2304 [I 2022-10-28 08:29:48] ...trap/engines/engine.py.228: elapsed: 0:19:45 | left: 0:01:47 [I 2022-10-28 08:29:48] ...trap/engines/engine.py.229: process: 0.55019 | load: 0.00243 [I 2022-10-28 08:29:48] ...trap/engines/engine.py.230: loss: 0.08684 [I 2022-10-28 08:29:48] ...optimizers/trijoint.py.151: total_norm: 11.130760 [I 2022-10-28 08:29:54] ...trap/engines/engine.py.224: train: epoch 115 | batch 2120/2304 [I 2022-10-28 08:29:54] ...trap/engines/engine.py.228: elapsed: 0:19:51 | left: 0:01:42 [I 2022-10-28 08:29:54] ...trap/engines/engine.py.229: process: 0.56170 | load: 0.00244 [I 2022-10-28 08:29:54] ...trap/engines/engine.py.230: loss: 0.10833 [I 2022-10-28 08:29:54] ...optimizers/trijoint.py.151: total_norm: 11.025995 [I 2022-10-28 08:29:59] ...trap/engines/engine.py.224: train: epoch 115 | batch 2130/2304 [I 2022-10-28 08:29:59] ...trap/engines/engine.py.228: elapsed: 0:19:56 | left: 0:01:36 [I 2022-10-28 08:29:59] ...trap/engines/engine.py.229: process: 0.54356 | load: 0.00242 [I 2022-10-28 08:29:59] ...trap/engines/engine.py.230: loss: 0.09860 [I 2022-10-28 08:29:59] ...optimizers/trijoint.py.151: total_norm: 13.595166 [I 2022-10-28 08:30:05] ...trap/engines/engine.py.224: train: epoch 115 | batch 2140/2304 [I 2022-10-28 08:30:05] ...trap/engines/engine.py.228: elapsed: 0:20:02 | left: 0:01:31 [I 2022-10-28 08:30:05] ...trap/engines/engine.py.229: process: 0.54324 | load: 0.00243 [I 2022-10-28 08:30:05] ...trap/engines/engine.py.230: loss: 0.08689 [I 2022-10-28 08:30:05] ...optimizers/trijoint.py.151: total_norm: 9.189241 [I 2022-10-28 08:30:10] ...trap/engines/engine.py.224: train: epoch 115 | batch 2150/2304 [I 2022-10-28 08:30:10] ...trap/engines/engine.py.228: elapsed: 0:20:07 | left: 0:01:26 [I 2022-10-28 08:30:10] ...trap/engines/engine.py.229: process: 0.55902 | load: 0.00243 [I 2022-10-28 08:30:10] ...trap/engines/engine.py.230: loss: 0.10983 [I 2022-10-28 08:30:10] ...optimizers/trijoint.py.151: total_norm: 12.360197 [I 2022-10-28 08:30:16] ...trap/engines/engine.py.224: train: epoch 115 | batch 2160/2304 [I 2022-10-28 08:30:16] ...trap/engines/engine.py.228: elapsed: 0:20:13 | left: 0:01:21 [I 2022-10-28 08:30:16] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00254 [I 2022-10-28 08:30:16] ...trap/engines/engine.py.230: loss: 0.10004 [I 2022-10-28 08:30:16] ...optimizers/trijoint.py.151: total_norm: 11.827749 [I 2022-10-28 08:30:22] ...trap/engines/engine.py.224: train: epoch 115 | batch 2170/2304 [I 2022-10-28 08:30:22] ...trap/engines/engine.py.228: elapsed: 0:20:19 | left: 0:01:15 [I 2022-10-28 08:30:22] ...trap/engines/engine.py.229: process: 0.56405 | load: 0.00253 [I 2022-10-28 08:30:22] ...trap/engines/engine.py.230: loss: 0.07887 [I 2022-10-28 08:30:22] ...optimizers/trijoint.py.151: total_norm: 8.872137 [I 2022-10-28 08:30:29] ...trap/engines/engine.py.224: train: epoch 115 | batch 2180/2304 [I 2022-10-28 08:30:29] ...trap/engines/engine.py.228: elapsed: 0:20:26 | left: 0:01:21 [I 2022-10-28 08:30:29] ...trap/engines/engine.py.229: process: 0.55726 | load: 0.00246 [I 2022-10-28 08:30:29] ...trap/engines/engine.py.230: loss: 0.07961 [I 2022-10-28 08:30:29] ...optimizers/trijoint.py.151: total_norm: 16.939587 [I 2022-10-28 08:30:35] ...trap/engines/engine.py.224: train: epoch 115 | batch 2190/2304 [I 2022-10-28 08:30:35] ...trap/engines/engine.py.228: elapsed: 0:20:31 | left: 0:01:05 [I 2022-10-28 08:30:35] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00246 [I 2022-10-28 08:30:35] ...trap/engines/engine.py.230: loss: 0.09151 [I 2022-10-28 08:30:35] ...optimizers/trijoint.py.151: total_norm: 12.783017 [I 2022-10-28 08:30:40] ...trap/engines/engine.py.224: train: epoch 115 | batch 2200/2304 [I 2022-10-28 08:30:40] ...trap/engines/engine.py.228: elapsed: 0:20:37 | left: 0:00:58 [I 2022-10-28 08:30:40] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00243 [I 2022-10-28 08:30:40] ...trap/engines/engine.py.230: loss: 0.23237 [I 2022-10-28 08:30:40] ...optimizers/trijoint.py.151: total_norm: 16.753469 [I 2022-10-28 08:30:46] ...trap/engines/engine.py.224: train: epoch 115 | batch 2210/2304 [I 2022-10-28 08:30:46] ...trap/engines/engine.py.228: elapsed: 0:20:43 | left: 0:00:52 [I 2022-10-28 08:30:46] ...trap/engines/engine.py.229: process: 0.56837 | load: 0.00248 [I 2022-10-28 08:30:46] ...trap/engines/engine.py.230: loss: 0.11062 [I 2022-10-28 08:30:46] ...optimizers/trijoint.py.151: total_norm: 14.259177 [I 2022-10-28 08:30:51] ...trap/engines/engine.py.224: train: epoch 115 | batch 2220/2304 [I 2022-10-28 08:30:51] ...trap/engines/engine.py.228: elapsed: 0:20:48 | left: 0:00:46 [I 2022-10-28 08:30:51] ...trap/engines/engine.py.229: process: 0.56541 | load: 0.00253 [I 2022-10-28 08:30:51] ...trap/engines/engine.py.230: loss: 0.14050 [I 2022-10-28 08:30:51] ...optimizers/trijoint.py.151: total_norm: 21.144526 [I 2022-10-28 08:30:57] ...trap/engines/engine.py.224: train: epoch 115 | batch 2230/2304 [I 2022-10-28 08:30:57] ...trap/engines/engine.py.228: elapsed: 0:20:54 | left: 0:00:41 [I 2022-10-28 08:30:57] ...trap/engines/engine.py.229: process: 0.55356 | load: 0.00247 [I 2022-10-28 08:30:57] ...trap/engines/engine.py.230: loss: 0.11452 [I 2022-10-28 08:30:57] ...optimizers/trijoint.py.151: total_norm: 15.314303 [I 2022-10-28 08:31:02] ...trap/engines/engine.py.224: train: epoch 115 | batch 2240/2304 [I 2022-10-28 08:31:02] ...trap/engines/engine.py.228: elapsed: 0:20:59 | left: 0:00:35 [I 2022-10-28 08:31:02] ...trap/engines/engine.py.229: process: 0.53489 | load: 0.00245 [I 2022-10-28 08:31:02] ...trap/engines/engine.py.230: loss: 0.07811 [I 2022-10-28 08:31:02] ...optimizers/trijoint.py.151: total_norm: 10.864782 [I 2022-10-28 08:31:08] ...trap/engines/engine.py.224: train: epoch 115 | batch 2250/2304 [I 2022-10-28 08:31:08] ...trap/engines/engine.py.228: elapsed: 0:21:05 | left: 0:00:30 [I 2022-10-28 08:31:08] ...trap/engines/engine.py.229: process: 0.55501 | load: 0.00244 [I 2022-10-28 08:31:08] ...trap/engines/engine.py.230: loss: 0.10013 [I 2022-10-28 08:31:08] ...optimizers/trijoint.py.151: total_norm: 20.809757 [I 2022-10-28 08:31:14] ...trap/engines/engine.py.224: train: epoch 115 | batch 2260/2304 [I 2022-10-28 08:31:14] ...trap/engines/engine.py.228: elapsed: 0:21:10 | left: 0:00:24 [I 2022-10-28 08:31:14] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00244 [I 2022-10-28 08:31:14] ...trap/engines/engine.py.230: loss: 0.13885 [I 2022-10-28 08:31:14] ...optimizers/trijoint.py.151: total_norm: 26.258358 [I 2022-10-28 08:31:19] ...trap/engines/engine.py.224: train: epoch 115 | batch 2270/2304 [I 2022-10-28 08:31:19] ...trap/engines/engine.py.228: elapsed: 0:21:16 | left: 0:00:19 [I 2022-10-28 08:31:19] ...trap/engines/engine.py.229: process: 0.56240 | load: 0.00251 [I 2022-10-28 08:31:19] ...trap/engines/engine.py.230: loss: 0.11181 [I 2022-10-28 08:31:19] ...optimizers/trijoint.py.151: total_norm: 18.696119 [I 2022-10-28 08:31:25] ...trap/engines/engine.py.224: train: epoch 115 | batch 2280/2304 [I 2022-10-28 08:31:25] ...trap/engines/engine.py.228: elapsed: 0:21:22 | left: 0:00:13 [I 2022-10-28 08:31:25] ...trap/engines/engine.py.229: process: 0.55532 | load: 0.00245 [I 2022-10-28 08:31:25] ...trap/engines/engine.py.230: loss: 0.12042 [I 2022-10-28 08:31:25] ...optimizers/trijoint.py.151: total_norm: 26.851212 [I 2022-10-28 08:31:30] ...trap/engines/engine.py.224: train: epoch 115 | batch 2290/2304 [I 2022-10-28 08:31:30] ...trap/engines/engine.py.228: elapsed: 0:21:27 | left: 0:00:07 [I 2022-10-28 08:31:30] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00241 [I 2022-10-28 08:31:30] ...trap/engines/engine.py.230: loss: 0.11661 [I 2022-10-28 08:31:30] ...optimizers/trijoint.py.151: total_norm: 12.393024 [I 2022-10-28 08:31:36] ...trap/engines/engine.py.224: train: epoch 115 | batch 2300/2304 [I 2022-10-28 08:31:36] ...trap/engines/engine.py.228: elapsed: 0:21:33 | left: 0:00:02 [I 2022-10-28 08:31:36] ...trap/engines/engine.py.229: process: 0.55708 | load: 0.00014 [I 2022-10-28 08:31:36] ...trap/engines/engine.py.230: loss: 0.09396 [I 2022-10-28 08:31:36] ...optimizers/trijoint.py.151: total_norm: 11.194330 [I 2022-10-28 08:31:38] ...trap/engines/engine.py.224: train: epoch 115 | batch 2304/2304 [I 2022-10-28 08:31:38] ...trap/engines/engine.py.228: elapsed: 0:21:35 | left: 0:00:00 [I 2022-10-28 08:31:38] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00015 [I 2022-10-28 08:31:38] ...trap/engines/engine.py.230: loss: 0.09512 [I 2022-10-28 08:31:38] ...optimizers/trijoint.py.151: total_norm: 55.353439 [S 2022-10-28 08:31:39] ...trap/engines/engine.py.129: train_epoch.epoch: 115 [S 2022-10-28 08:31:39] ...trap/engines/engine.py.129: train_epoch.loss: 0.12051229968893296 [I 2022-10-28 08:31:39] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 08:31:41] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 115 [I 2022-10-28 08:31:41] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 08:31:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 08:31:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 08:31:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 08:31:47] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 08:31:47] ...trap/engines/engine.py.321: eval: epoch 115 | batch 0/510 [I 2022-10-28 08:31:47] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:52:21 [I 2022-10-28 08:31:47] ...trap/engines/engine.py.326: process: 6.16037 | load: 3.84547 [I 2022-10-28 08:31:51] ...trap/engines/engine.py.321: eval: epoch 115 | batch 10/510 [I 2022-10-28 08:31:51] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:07:58 [I 2022-10-28 08:31:51] ...trap/engines/engine.py.326: process: 0.32527 | load: 0.00016 [I 2022-10-28 08:31:54] ...trap/engines/engine.py.321: eval: epoch 115 | batch 20/510 [I 2022-10-28 08:31:54] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:15 [I 2022-10-28 08:31:54] ...trap/engines/engine.py.326: process: 0.32462 | load: 0.00018 [I 2022-10-28 08:31:57] ...trap/engines/engine.py.321: eval: epoch 115 | batch 30/510 [I 2022-10-28 08:31:57] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:46 [I 2022-10-28 08:31:57] ...trap/engines/engine.py.326: process: 0.32743 | load: 0.00018 [I 2022-10-28 08:32:01] ...trap/engines/engine.py.321: eval: epoch 115 | batch 40/510 [I 2022-10-28 08:32:01] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-28 08:32:01] ...trap/engines/engine.py.326: process: 0.31793 | load: 0.00018 [I 2022-10-28 08:32:04] ...trap/engines/engine.py.321: eval: epoch 115 | batch 50/510 [I 2022-10-28 08:32:04] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:31 [I 2022-10-28 08:32:04] ...trap/engines/engine.py.326: process: 0.32975 | load: 0.00024 [I 2022-10-28 08:32:07] ...trap/engines/engine.py.321: eval: epoch 115 | batch 60/510 [I 2022-10-28 08:32:07] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-28 08:32:07] ...trap/engines/engine.py.326: process: 0.33499 | load: 0.00016 [I 2022-10-28 08:32:11] ...trap/engines/engine.py.321: eval: epoch 115 | batch 70/510 [I 2022-10-28 08:32:11] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:26 [I 2022-10-28 08:32:11] ...trap/engines/engine.py.326: process: 0.33201 | load: 0.00022 [I 2022-10-28 08:32:14] ...trap/engines/engine.py.321: eval: epoch 115 | batch 80/510 [I 2022-10-28 08:32:14] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:24 [I 2022-10-28 08:32:14] ...trap/engines/engine.py.326: process: 0.33305 | load: 0.00016 [I 2022-10-28 08:32:17] ...trap/engines/engine.py.321: eval: epoch 115 | batch 90/510 [I 2022-10-28 08:32:17] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:19 [I 2022-10-28 08:32:17] ...trap/engines/engine.py.326: process: 0.32942 | load: 0.00021 [I 2022-10-28 08:32:21] ...trap/engines/engine.py.321: eval: epoch 115 | batch 100/510 [I 2022-10-28 08:32:21] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:12 [I 2022-10-28 08:32:21] ...trap/engines/engine.py.326: process: 0.28987 | load: 0.00019 [I 2022-10-28 08:32:24] ...trap/engines/engine.py.321: eval: epoch 115 | batch 110/510 [I 2022-10-28 08:32:24] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:59 [I 2022-10-28 08:32:24] ...trap/engines/engine.py.326: process: 0.31324 | load: 0.00015 [I 2022-10-28 08:32:26] ...trap/engines/engine.py.321: eval: epoch 115 | batch 120/510 [I 2022-10-28 08:32:26] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:53 [I 2022-10-28 08:32:26] ...trap/engines/engine.py.326: process: 0.28495 | load: 0.00026 [I 2022-10-28 08:32:29] ...trap/engines/engine.py.321: eval: epoch 115 | batch 130/510 [I 2022-10-28 08:32:29] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:51 [I 2022-10-28 08:32:29] ...trap/engines/engine.py.326: process: 0.30595 | load: 0.00015 [I 2022-10-28 08:32:32] ...trap/engines/engine.py.321: eval: epoch 115 | batch 140/510 [I 2022-10-28 08:32:32] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:49 [I 2022-10-28 08:32:32] ...trap/engines/engine.py.326: process: 0.28742 | load: 0.00019 [I 2022-10-28 08:32:35] ...trap/engines/engine.py.321: eval: epoch 115 | batch 150/510 [I 2022-10-28 08:32:35] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:43 [I 2022-10-28 08:32:35] ...trap/engines/engine.py.326: process: 0.28837 | load: 0.00020 [I 2022-10-28 08:32:38] ...trap/engines/engine.py.321: eval: epoch 115 | batch 160/510 [I 2022-10-28 08:32:38] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-28 08:32:38] ...trap/engines/engine.py.326: process: 0.28124 | load: 0.00016 [I 2022-10-28 08:32:41] ...trap/engines/engine.py.321: eval: epoch 115 | batch 170/510 [I 2022-10-28 08:32:41] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:37 [I 2022-10-28 08:32:41] ...trap/engines/engine.py.326: process: 0.28906 | load: 0.00019 [I 2022-10-28 08:32:44] ...trap/engines/engine.py.321: eval: epoch 115 | batch 180/510 [I 2022-10-28 08:32:44] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-28 08:32:44] ...trap/engines/engine.py.326: process: 0.28327 | load: 0.00021 [I 2022-10-28 08:32:47] ...trap/engines/engine.py.321: eval: epoch 115 | batch 190/510 [I 2022-10-28 08:32:47] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:33 [I 2022-10-28 08:32:47] ...trap/engines/engine.py.326: process: 0.30459 | load: 0.00019 [I 2022-10-28 08:32:50] ...trap/engines/engine.py.321: eval: epoch 115 | batch 200/510 [I 2022-10-28 08:32:50] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:30 [I 2022-10-28 08:32:50] ...trap/engines/engine.py.326: process: 0.28956 | load: 0.00016 [I 2022-10-28 08:32:53] ...trap/engines/engine.py.321: eval: epoch 115 | batch 210/510 [I 2022-10-28 08:32:53] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 08:32:53] ...trap/engines/engine.py.326: process: 0.28323 | load: 0.00016 [I 2022-10-28 08:32:55] ...trap/engines/engine.py.321: eval: epoch 115 | batch 220/510 [I 2022-10-28 08:32:55] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:24 [I 2022-10-28 08:32:55] ...trap/engines/engine.py.326: process: 0.28664 | load: 0.00015 [I 2022-10-28 08:32:58] ...trap/engines/engine.py.321: eval: epoch 115 | batch 230/510 [I 2022-10-28 08:32:58] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:21 [I 2022-10-28 08:32:58] ...trap/engines/engine.py.326: process: 0.30989 | load: 0.00014 [I 2022-10-28 08:33:01] ...trap/engines/engine.py.321: eval: epoch 115 | batch 240/510 [I 2022-10-28 08:33:01] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:18 [I 2022-10-28 08:33:01] ...trap/engines/engine.py.326: process: 0.28666 | load: 0.00023 [I 2022-10-28 08:33:04] ...trap/engines/engine.py.321: eval: epoch 115 | batch 250/510 [I 2022-10-28 08:33:04] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:15 [I 2022-10-28 08:33:04] ...trap/engines/engine.py.326: process: 0.28839 | load: 0.00028 [I 2022-10-28 08:33:07] ...trap/engines/engine.py.321: eval: epoch 115 | batch 260/510 [I 2022-10-28 08:33:07] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 08:33:07] ...trap/engines/engine.py.326: process: 0.28956 | load: 0.00021 [I 2022-10-28 08:33:10] ...trap/engines/engine.py.321: eval: epoch 115 | batch 270/510 [I 2022-10-28 08:33:10] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-28 08:33:10] ...trap/engines/engine.py.326: process: 0.28799 | load: 0.00028 [I 2022-10-28 08:33:13] ...trap/engines/engine.py.321: eval: epoch 115 | batch 280/510 [I 2022-10-28 08:33:13] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 08:33:13] ...trap/engines/engine.py.326: process: 0.28484 | load: 0.00033 [I 2022-10-28 08:33:16] ...trap/engines/engine.py.321: eval: epoch 115 | batch 290/510 [I 2022-10-28 08:33:16] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:04 [I 2022-10-28 08:33:16] ...trap/engines/engine.py.326: process: 0.29734 | load: 0.00028 [I 2022-10-28 08:33:19] ...trap/engines/engine.py.321: eval: epoch 115 | batch 300/510 [I 2022-10-28 08:33:19] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:01 [I 2022-10-28 08:33:19] ...trap/engines/engine.py.326: process: 0.28667 | load: 0.00025 [I 2022-10-28 08:33:22] ...trap/engines/engine.py.321: eval: epoch 115 | batch 310/510 [I 2022-10-28 08:33:22] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:59 [I 2022-10-28 08:33:22] ...trap/engines/engine.py.326: process: 0.29214 | load: 0.00026 [I 2022-10-28 08:33:25] ...trap/engines/engine.py.321: eval: epoch 115 | batch 320/510 [I 2022-10-28 08:33:25] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:55 [I 2022-10-28 08:33:25] ...trap/engines/engine.py.326: process: 0.29275 | load: 0.00015 [I 2022-10-28 08:33:27] ...trap/engines/engine.py.321: eval: epoch 115 | batch 330/510 [I 2022-10-28 08:33:27] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:52 [I 2022-10-28 08:33:27] ...trap/engines/engine.py.326: process: 0.28903 | load: 0.00024 [I 2022-10-28 08:33:30] ...trap/engines/engine.py.321: eval: epoch 115 | batch 340/510 [I 2022-10-28 08:33:30] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:48 [I 2022-10-28 08:33:30] ...trap/engines/engine.py.326: process: 0.29237 | load: 0.00016 [I 2022-10-28 08:33:33] ...trap/engines/engine.py.321: eval: epoch 115 | batch 350/510 [I 2022-10-28 08:33:33] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:46 [I 2022-10-28 08:33:33] ...trap/engines/engine.py.326: process: 0.29200 | load: 0.00025 [I 2022-10-28 08:33:36] ...trap/engines/engine.py.321: eval: epoch 115 | batch 360/510 [I 2022-10-28 08:33:36] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:42 [I 2022-10-28 08:33:36] ...trap/engines/engine.py.326: process: 0.28819 | load: 0.00015 [I 2022-10-28 08:33:39] ...trap/engines/engine.py.321: eval: epoch 115 | batch 370/510 [I 2022-10-28 08:33:39] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 08:33:39] ...trap/engines/engine.py.326: process: 0.28883 | load: 0.00024 [I 2022-10-28 08:33:42] ...trap/engines/engine.py.321: eval: epoch 115 | batch 380/510 [I 2022-10-28 08:33:42] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 08:33:42] ...trap/engines/engine.py.326: process: 0.28879 | load: 0.00016 [I 2022-10-28 08:33:45] ...trap/engines/engine.py.321: eval: epoch 115 | batch 390/510 [I 2022-10-28 08:33:45] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 08:33:45] ...trap/engines/engine.py.326: process: 0.29155 | load: 0.00025 [I 2022-10-28 08:33:48] ...trap/engines/engine.py.321: eval: epoch 115 | batch 400/510 [I 2022-10-28 08:33:48] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:31 [I 2022-10-28 08:33:48] ...trap/engines/engine.py.326: process: 0.30253 | load: 0.00015 [I 2022-10-28 08:33:51] ...trap/engines/engine.py.321: eval: epoch 115 | batch 410/510 [I 2022-10-28 08:33:51] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:29 [I 2022-10-28 08:33:51] ...trap/engines/engine.py.326: process: 0.28995 | load: 0.00025 [I 2022-10-28 08:33:53] ...trap/engines/engine.py.321: eval: epoch 115 | batch 420/510 [I 2022-10-28 08:33:53] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:25 [I 2022-10-28 08:33:53] ...trap/engines/engine.py.326: process: 0.28601 | load: 0.00015 [I 2022-10-28 08:33:56] ...trap/engines/engine.py.321: eval: epoch 115 | batch 430/510 [I 2022-10-28 08:33:56] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:22 [I 2022-10-28 08:33:56] ...trap/engines/engine.py.326: process: 0.28995 | load: 0.00022 [I 2022-10-28 08:33:59] ...trap/engines/engine.py.321: eval: epoch 115 | batch 440/510 [I 2022-10-28 08:33:59] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 08:33:59] ...trap/engines/engine.py.326: process: 0.28573 | load: 0.00016 [I 2022-10-28 08:34:02] ...trap/engines/engine.py.321: eval: epoch 115 | batch 450/510 [I 2022-10-28 08:34:02] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 08:34:02] ...trap/engines/engine.py.326: process: 0.28613 | load: 0.00027 [I 2022-10-28 08:34:05] ...trap/engines/engine.py.321: eval: epoch 115 | batch 460/510 [I 2022-10-28 08:34:05] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 08:34:05] ...trap/engines/engine.py.326: process: 0.28673 | load: 0.00016 [I 2022-10-28 08:34:08] ...trap/engines/engine.py.321: eval: epoch 115 | batch 470/510 [I 2022-10-28 08:34:08] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 08:34:08] ...trap/engines/engine.py.326: process: 0.29035 | load: 0.00021 [I 2022-10-28 08:34:11] ...trap/engines/engine.py.321: eval: epoch 115 | batch 480/510 [I 2022-10-28 08:34:11] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 08:34:11] ...trap/engines/engine.py.326: process: 0.29257 | load: 0.00016 [I 2022-10-28 08:34:14] ...trap/engines/engine.py.321: eval: epoch 115 | batch 490/510 [I 2022-10-28 08:34:14] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 08:34:14] ...trap/engines/engine.py.326: process: 0.28852 | load: 0.00021 [I 2022-10-28 08:34:17] ...trap/engines/engine.py.321: eval: epoch 115 | batch 500/510 [I 2022-10-28 08:34:17] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:02 [I 2022-10-28 08:34:17] ...trap/engines/engine.py.326: process: 0.28981 | load: 0.00016 [I 2022-10-28 08:34:19] ...trap/engines/engine.py.321: eval: epoch 115 | batch 510/510 [I 2022-10-28 08:34:19] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:00 [I 2022-10-28 08:34:19] ...trap/engines/engine.py.326: process: 0.27976 | load: 0.00011 [S 2022-10-28 08:34:20] ...trap/engines/engine.py.132: eval_epoch.epoch: 115 [I 2022-10-28 08:34:20] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.737900 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.904700 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934500 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.730300 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904700 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933900 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010625 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009819 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009646 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.015850 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.009166 [S 2022-10-28 08:34:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008142 [I 2022-10-28 08:34:26] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 08:34:26] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 08:34:45] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 08:34:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 08:34:45] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 08:34:50] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 08:34:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 08:34:56] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 08:34:56] ...trap/engines/engine.py.163: Training model on trainset for epoch 116 [I 2022-10-28 08:34:56] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 08:34:59] ...trap/engines/engine.py.224: train: epoch 116 | batch 0/2304 [I 2022-10-28 08:34:59] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:47:00 [I 2022-10-28 08:34:59] ...trap/engines/engine.py.229: process: 2.78668 | load: 2.12316 [I 2022-10-28 08:34:59] ...trap/engines/engine.py.230: loss: 0.12433 [I 2022-10-28 08:34:59] ...optimizers/trijoint.py.151: total_norm: 11.227962 [I 2022-10-28 08:35:05] ...trap/engines/engine.py.224: train: epoch 116 | batch 10/2304 [I 2022-10-28 08:35:05] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:18 [I 2022-10-28 08:35:05] ...trap/engines/engine.py.229: process: 0.55470 | load: 0.00374 [I 2022-10-28 08:35:05] ...trap/engines/engine.py.230: loss: 0.12432 [I 2022-10-28 08:35:05] ...optimizers/trijoint.py.151: total_norm: 13.393980 [I 2022-10-28 08:35:10] ...trap/engines/engine.py.224: train: epoch 116 | batch 20/2304 [I 2022-10-28 08:35:10] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:13 [I 2022-10-28 08:35:10] ...trap/engines/engine.py.229: process: 0.55464 | load: 0.00343 [I 2022-10-28 08:35:10] ...trap/engines/engine.py.230: loss: 0.12684 [I 2022-10-28 08:35:10] ...optimizers/trijoint.py.151: total_norm: 8.094133 [I 2022-10-28 08:35:16] ...trap/engines/engine.py.224: train: epoch 116 | batch 30/2304 [I 2022-10-28 08:35:16] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:13 [I 2022-10-28 08:35:16] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00361 [I 2022-10-28 08:35:16] ...trap/engines/engine.py.230: loss: 0.14520 [I 2022-10-28 08:35:16] ...optimizers/trijoint.py.151: total_norm: 14.703207 [I 2022-10-28 08:35:22] ...trap/engines/engine.py.224: train: epoch 116 | batch 40/2304 [I 2022-10-28 08:35:22] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:20:59 [I 2022-10-28 08:35:22] ...trap/engines/engine.py.229: process: 0.56527 | load: 0.00283 [I 2022-10-28 08:35:22] ...trap/engines/engine.py.230: loss: 0.13888 [I 2022-10-28 08:35:22] ...optimizers/trijoint.py.151: total_norm: 15.161657 [I 2022-10-28 08:35:27] ...trap/engines/engine.py.224: train: epoch 116 | batch 50/2304 [I 2022-10-28 08:35:27] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:47 [I 2022-10-28 08:35:27] ...trap/engines/engine.py.229: process: 0.54981 | load: 0.00307 [I 2022-10-28 08:35:27] ...trap/engines/engine.py.230: loss: 0.09844 [I 2022-10-28 08:35:27] ...optimizers/trijoint.py.151: total_norm: 32.700119 [I 2022-10-28 08:35:33] ...trap/engines/engine.py.224: train: epoch 116 | batch 60/2304 [I 2022-10-28 08:35:33] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:46 [I 2022-10-28 08:35:33] ...trap/engines/engine.py.229: process: 0.54652 | load: 0.00269 [I 2022-10-28 08:35:33] ...trap/engines/engine.py.230: loss: 0.11629 [I 2022-10-28 08:35:33] ...optimizers/trijoint.py.151: total_norm: 23.391930 [I 2022-10-28 08:35:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 70/2304 [I 2022-10-28 08:35:38] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:44 [I 2022-10-28 08:35:38] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00298 [I 2022-10-28 08:35:38] ...trap/engines/engine.py.230: loss: 0.10828 [I 2022-10-28 08:35:38] ...optimizers/trijoint.py.151: total_norm: 13.636411 [I 2022-10-28 08:35:44] ...trap/engines/engine.py.224: train: epoch 116 | batch 80/2304 [I 2022-10-28 08:35:44] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:36 [I 2022-10-28 08:35:44] ...trap/engines/engine.py.229: process: 0.55407 | load: 0.00264 [I 2022-10-28 08:35:44] ...trap/engines/engine.py.230: loss: 0.11097 [I 2022-10-28 08:35:44] ...optimizers/trijoint.py.151: total_norm: 17.097980 [I 2022-10-28 08:35:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 90/2304 [I 2022-10-28 08:35:49] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:40 [I 2022-10-28 08:35:49] ...trap/engines/engine.py.229: process: 0.58133 | load: 0.00266 [I 2022-10-28 08:35:49] ...trap/engines/engine.py.230: loss: 0.10274 [I 2022-10-28 08:35:49] ...optimizers/trijoint.py.151: total_norm: 17.258286 [I 2022-10-28 08:35:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 100/2304 [I 2022-10-28 08:35:55] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:29 [I 2022-10-28 08:35:55] ...trap/engines/engine.py.229: process: 0.54521 | load: 0.00294 [I 2022-10-28 08:35:55] ...trap/engines/engine.py.230: loss: 0.10218 [I 2022-10-28 08:35:55] ...optimizers/trijoint.py.151: total_norm: 26.764669 [I 2022-10-28 08:36:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 110/2304 [I 2022-10-28 08:36:01] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:18 [I 2022-10-28 08:36:01] ...trap/engines/engine.py.229: process: 0.55757 | load: 0.00259 [I 2022-10-28 08:36:01] ...trap/engines/engine.py.230: loss: 0.08434 [I 2022-10-28 08:36:01] ...optimizers/trijoint.py.151: total_norm: 22.365866 [I 2022-10-28 08:36:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 120/2304 [I 2022-10-28 08:36:06] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:06 [I 2022-10-28 08:36:06] ...trap/engines/engine.py.229: process: 0.54929 | load: 0.00256 [I 2022-10-28 08:36:06] ...trap/engines/engine.py.230: loss: 0.15303 [I 2022-10-28 08:36:06] ...optimizers/trijoint.py.151: total_norm: 43.672848 [I 2022-10-28 08:36:12] ...trap/engines/engine.py.224: train: epoch 116 | batch 130/2304 [I 2022-10-28 08:36:12] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:15 [I 2022-10-28 08:36:12] ...trap/engines/engine.py.229: process: 0.56987 | load: 0.00279 [I 2022-10-28 08:36:12] ...trap/engines/engine.py.230: loss: 0.15099 [I 2022-10-28 08:36:12] ...optimizers/trijoint.py.151: total_norm: 12.696934 [I 2022-10-28 08:36:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 140/2304 [I 2022-10-28 08:36:17] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:36 [I 2022-10-28 08:36:17] ...trap/engines/engine.py.229: process: 0.58976 | load: 0.00257 [I 2022-10-28 08:36:17] ...trap/engines/engine.py.230: loss: 0.13327 [I 2022-10-28 08:36:17] ...optimizers/trijoint.py.151: total_norm: 18.249163 [I 2022-10-28 08:36:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 150/2304 [I 2022-10-28 08:36:23] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:11 [I 2022-10-28 08:36:23] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00291 [I 2022-10-28 08:36:23] ...trap/engines/engine.py.230: loss: 0.13699 [I 2022-10-28 08:36:23] ...optimizers/trijoint.py.151: total_norm: 17.916800 [I 2022-10-28 08:36:29] ...trap/engines/engine.py.224: train: epoch 116 | batch 160/2304 [I 2022-10-28 08:36:29] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:19:56 [I 2022-10-28 08:36:29] ...trap/engines/engine.py.229: process: 0.54588 | load: 0.00271 [I 2022-10-28 08:36:29] ...trap/engines/engine.py.230: loss: 0.10485 [I 2022-10-28 08:36:29] ...optimizers/trijoint.py.151: total_norm: 16.593697 [I 2022-10-28 08:36:34] ...trap/engines/engine.py.224: train: epoch 116 | batch 170/2304 [I 2022-10-28 08:36:34] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:56 [I 2022-10-28 08:36:34] ...trap/engines/engine.py.229: process: 0.56607 | load: 0.00254 [I 2022-10-28 08:36:34] ...trap/engines/engine.py.230: loss: 0.11950 [I 2022-10-28 08:36:34] ...optimizers/trijoint.py.151: total_norm: 18.136423 [I 2022-10-28 08:36:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 180/2304 [I 2022-10-28 08:36:40] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:53 [I 2022-10-28 08:36:40] ...trap/engines/engine.py.229: process: 0.56373 | load: 0.00245 [I 2022-10-28 08:36:40] ...trap/engines/engine.py.230: loss: 0.17293 [I 2022-10-28 08:36:40] ...optimizers/trijoint.py.151: total_norm: 29.728834 [I 2022-10-28 08:36:46] ...trap/engines/engine.py.224: train: epoch 116 | batch 190/2304 [I 2022-10-28 08:36:46] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:51 [I 2022-10-28 08:36:46] ...trap/engines/engine.py.229: process: 0.57918 | load: 0.00258 [I 2022-10-28 08:36:46] ...trap/engines/engine.py.230: loss: 0.12156 [I 2022-10-28 08:36:46] ...optimizers/trijoint.py.151: total_norm: 13.904529 [I 2022-10-28 08:36:51] ...trap/engines/engine.py.224: train: epoch 116 | batch 200/2304 [I 2022-10-28 08:36:51] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:52 [I 2022-10-28 08:36:51] ...trap/engines/engine.py.229: process: 0.55847 | load: 0.00252 [I 2022-10-28 08:36:51] ...trap/engines/engine.py.230: loss: 0.14745 [I 2022-10-28 08:36:51] ...optimizers/trijoint.py.151: total_norm: 20.647705 [I 2022-10-28 08:36:57] ...trap/engines/engine.py.224: train: epoch 116 | batch 210/2304 [I 2022-10-28 08:36:57] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:48 [I 2022-10-28 08:36:57] ...trap/engines/engine.py.229: process: 0.58278 | load: 0.00246 [I 2022-10-28 08:36:57] ...trap/engines/engine.py.230: loss: 0.10125 [I 2022-10-28 08:36:57] ...optimizers/trijoint.py.151: total_norm: 16.240110 [I 2022-10-28 08:37:03] ...trap/engines/engine.py.224: train: epoch 116 | batch 220/2304 [I 2022-10-28 08:37:03] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:29 [I 2022-10-28 08:37:03] ...trap/engines/engine.py.229: process: 0.57944 | load: 0.00244 [I 2022-10-28 08:37:03] ...trap/engines/engine.py.230: loss: 0.14419 [I 2022-10-28 08:37:03] ...optimizers/trijoint.py.151: total_norm: 13.575228 [I 2022-10-28 08:37:08] ...trap/engines/engine.py.224: train: epoch 116 | batch 230/2304 [I 2022-10-28 08:37:08] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:15 [I 2022-10-28 08:37:08] ...trap/engines/engine.py.229: process: 0.55151 | load: 0.00269 [I 2022-10-28 08:37:08] ...trap/engines/engine.py.230: loss: 0.10164 [I 2022-10-28 08:37:08] ...optimizers/trijoint.py.151: total_norm: 12.837639 [I 2022-10-28 08:37:14] ...trap/engines/engine.py.224: train: epoch 116 | batch 240/2304 [I 2022-10-28 08:37:14] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:06 [I 2022-10-28 08:37:14] ...trap/engines/engine.py.229: process: 0.55262 | load: 0.00264 [I 2022-10-28 08:37:14] ...trap/engines/engine.py.230: loss: 0.09249 [I 2022-10-28 08:37:14] ...optimizers/trijoint.py.151: total_norm: 11.684123 [I 2022-10-28 08:37:19] ...trap/engines/engine.py.224: train: epoch 116 | batch 250/2304 [I 2022-10-28 08:37:19] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:18:58 [I 2022-10-28 08:37:19] ...trap/engines/engine.py.229: process: 0.55415 | load: 0.00251 [I 2022-10-28 08:37:19] ...trap/engines/engine.py.230: loss: 0.13531 [I 2022-10-28 08:37:19] ...optimizers/trijoint.py.151: total_norm: 25.390747 [I 2022-10-28 08:37:25] ...trap/engines/engine.py.224: train: epoch 116 | batch 260/2304 [I 2022-10-28 08:37:25] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:44 [I 2022-10-28 08:37:25] ...trap/engines/engine.py.229: process: 0.54814 | load: 0.00253 [I 2022-10-28 08:37:25] ...trap/engines/engine.py.230: loss: 0.11155 [I 2022-10-28 08:37:25] ...optimizers/trijoint.py.151: total_norm: 43.582336 [I 2022-10-28 08:37:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 270/2304 [I 2022-10-28 08:37:30] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:18:57 [I 2022-10-28 08:37:30] ...trap/engines/engine.py.229: process: 0.55796 | load: 0.00245 [I 2022-10-28 08:37:30] ...trap/engines/engine.py.230: loss: 0.10232 [I 2022-10-28 08:37:30] ...optimizers/trijoint.py.151: total_norm: 28.460575 [I 2022-10-28 08:37:36] ...trap/engines/engine.py.224: train: epoch 116 | batch 280/2304 [I 2022-10-28 08:37:36] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:19:07 [I 2022-10-28 08:37:36] ...trap/engines/engine.py.229: process: 0.55869 | load: 0.00251 [I 2022-10-28 08:37:36] ...trap/engines/engine.py.230: loss: 0.12142 [I 2022-10-28 08:37:36] ...optimizers/trijoint.py.151: total_norm: 17.523781 [I 2022-10-28 08:37:42] ...trap/engines/engine.py.224: train: epoch 116 | batch 290/2304 [I 2022-10-28 08:37:42] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:56 [I 2022-10-28 08:37:42] ...trap/engines/engine.py.229: process: 0.56527 | load: 0.00258 [I 2022-10-28 08:37:42] ...trap/engines/engine.py.230: loss: 0.12014 [I 2022-10-28 08:37:42] ...optimizers/trijoint.py.151: total_norm: 20.956247 [I 2022-10-28 08:37:47] ...trap/engines/engine.py.224: train: epoch 116 | batch 300/2304 [I 2022-10-28 08:37:47] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:46 [I 2022-10-28 08:37:47] ...trap/engines/engine.py.229: process: 0.56448 | load: 0.00250 [I 2022-10-28 08:37:47] ...trap/engines/engine.py.230: loss: 0.09533 [I 2022-10-28 08:37:47] ...optimizers/trijoint.py.151: total_norm: 11.223003 [I 2022-10-28 08:37:53] ...trap/engines/engine.py.224: train: epoch 116 | batch 310/2304 [I 2022-10-28 08:37:53] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:35 [I 2022-10-28 08:37:53] ...trap/engines/engine.py.229: process: 0.54903 | load: 0.00250 [I 2022-10-28 08:37:53] ...trap/engines/engine.py.230: loss: 0.15943 [I 2022-10-28 08:37:53] ...optimizers/trijoint.py.151: total_norm: 19.677048 [I 2022-10-28 08:37:59] ...trap/engines/engine.py.224: train: epoch 116 | batch 320/2304 [I 2022-10-28 08:37:59] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:34 [I 2022-10-28 08:37:59] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00250 [I 2022-10-28 08:37:59] ...trap/engines/engine.py.230: loss: 0.15672 [I 2022-10-28 08:37:59] ...optimizers/trijoint.py.151: total_norm: 57.879566 [I 2022-10-28 08:38:04] ...trap/engines/engine.py.224: train: epoch 116 | batch 330/2304 [I 2022-10-28 08:38:04] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:27 [I 2022-10-28 08:38:04] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00247 [I 2022-10-28 08:38:04] ...trap/engines/engine.py.230: loss: 0.11550 [I 2022-10-28 08:38:04] ...optimizers/trijoint.py.151: total_norm: 17.490923 [I 2022-10-28 08:38:10] ...trap/engines/engine.py.224: train: epoch 116 | batch 340/2304 [I 2022-10-28 08:38:10] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:19 [I 2022-10-28 08:38:10] ...trap/engines/engine.py.229: process: 0.56425 | load: 0.00246 [I 2022-10-28 08:38:10] ...trap/engines/engine.py.230: loss: 0.18404 [I 2022-10-28 08:38:10] ...optimizers/trijoint.py.151: total_norm: 17.039614 [I 2022-10-28 08:38:16] ...trap/engines/engine.py.224: train: epoch 116 | batch 350/2304 [I 2022-10-28 08:38:16] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:16 [I 2022-10-28 08:38:16] ...trap/engines/engine.py.229: process: 0.54699 | load: 0.00244 [I 2022-10-28 08:38:16] ...trap/engines/engine.py.230: loss: 0.13897 [I 2022-10-28 08:38:16] ...optimizers/trijoint.py.151: total_norm: 23.589106 [I 2022-10-28 08:38:21] ...trap/engines/engine.py.224: train: epoch 116 | batch 360/2304 [I 2022-10-28 08:38:21] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:01 [I 2022-10-28 08:38:21] ...trap/engines/engine.py.229: process: 0.54908 | load: 0.00257 [I 2022-10-28 08:38:21] ...trap/engines/engine.py.230: loss: 0.11693 [I 2022-10-28 08:38:21] ...optimizers/trijoint.py.151: total_norm: 20.512484 [I 2022-10-28 08:38:27] ...trap/engines/engine.py.224: train: epoch 116 | batch 370/2304 [I 2022-10-28 08:38:27] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:01 [I 2022-10-28 08:38:27] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00250 [I 2022-10-28 08:38:27] ...trap/engines/engine.py.230: loss: 0.12936 [I 2022-10-28 08:38:27] ...optimizers/trijoint.py.151: total_norm: 17.919872 [I 2022-10-28 08:38:32] ...trap/engines/engine.py.224: train: epoch 116 | batch 380/2304 [I 2022-10-28 08:38:32] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:56 [I 2022-10-28 08:38:32] ...trap/engines/engine.py.229: process: 0.54076 | load: 0.00248 [I 2022-10-28 08:38:32] ...trap/engines/engine.py.230: loss: 0.09716 [I 2022-10-28 08:38:32] ...optimizers/trijoint.py.151: total_norm: 15.965353 [I 2022-10-28 08:38:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 390/2304 [I 2022-10-28 08:38:38] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:54 [I 2022-10-28 08:38:38] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00244 [I 2022-10-28 08:38:38] ...trap/engines/engine.py.230: loss: 0.08442 [I 2022-10-28 08:38:38] ...optimizers/trijoint.py.151: total_norm: 11.522684 [I 2022-10-28 08:38:44] ...trap/engines/engine.py.224: train: epoch 116 | batch 400/2304 [I 2022-10-28 08:38:44] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:42 [I 2022-10-28 08:38:44] ...trap/engines/engine.py.229: process: 0.56278 | load: 0.00257 [I 2022-10-28 08:38:44] ...trap/engines/engine.py.230: loss: 0.07562 [I 2022-10-28 08:38:44] ...optimizers/trijoint.py.151: total_norm: 12.973126 [I 2022-10-28 08:38:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 410/2304 [I 2022-10-28 08:38:49] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:56 [I 2022-10-28 08:38:49] ...trap/engines/engine.py.229: process: 0.55849 | load: 0.00256 [I 2022-10-28 08:38:49] ...trap/engines/engine.py.230: loss: 0.08615 [I 2022-10-28 08:38:49] ...optimizers/trijoint.py.151: total_norm: 12.351383 [I 2022-10-28 08:38:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 420/2304 [I 2022-10-28 08:38:55] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:36 [I 2022-10-28 08:38:55] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00252 [I 2022-10-28 08:38:55] ...trap/engines/engine.py.230: loss: 0.09194 [I 2022-10-28 08:38:55] ...optimizers/trijoint.py.151: total_norm: 19.641182 [I 2022-10-28 08:39:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 430/2304 [I 2022-10-28 08:39:01] ...trap/engines/engine.py.228: elapsed: 0:04:04 | left: 0:17:38 [I 2022-10-28 08:39:01] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00246 [I 2022-10-28 08:39:01] ...trap/engines/engine.py.230: loss: 0.11474 [I 2022-10-28 08:39:01] ...optimizers/trijoint.py.151: total_norm: 24.375378 [I 2022-10-28 08:39:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 440/2304 [I 2022-10-28 08:39:06] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:21 [I 2022-10-28 08:39:06] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00246 [I 2022-10-28 08:39:06] ...trap/engines/engine.py.230: loss: 0.08983 [I 2022-10-28 08:39:06] ...optimizers/trijoint.py.151: total_norm: 12.365471 [I 2022-10-28 08:39:12] ...trap/engines/engine.py.224: train: epoch 116 | batch 450/2304 [I 2022-10-28 08:39:12] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:19 [I 2022-10-28 08:39:12] ...trap/engines/engine.py.229: process: 0.55727 | load: 0.00244 [I 2022-10-28 08:39:12] ...trap/engines/engine.py.230: loss: 0.10898 [I 2022-10-28 08:39:12] ...optimizers/trijoint.py.151: total_norm: 18.995691 [I 2022-10-28 08:39:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 460/2304 [I 2022-10-28 08:39:17] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:08 [I 2022-10-28 08:39:17] ...trap/engines/engine.py.229: process: 0.56276 | load: 0.00244 [I 2022-10-28 08:39:17] ...trap/engines/engine.py.230: loss: 0.08955 [I 2022-10-28 08:39:17] ...optimizers/trijoint.py.151: total_norm: 17.246473 [I 2022-10-28 08:39:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 470/2304 [I 2022-10-28 08:39:23] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:12 [I 2022-10-28 08:39:23] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00243 [I 2022-10-28 08:39:23] ...trap/engines/engine.py.230: loss: 0.08924 [I 2022-10-28 08:39:23] ...optimizers/trijoint.py.151: total_norm: 12.899334 [I 2022-10-28 08:39:29] ...trap/engines/engine.py.224: train: epoch 116 | batch 480/2304 [I 2022-10-28 08:39:29] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:17:02 [I 2022-10-28 08:39:29] ...trap/engines/engine.py.229: process: 0.56512 | load: 0.00244 [I 2022-10-28 08:39:29] ...trap/engines/engine.py.230: loss: 0.09784 [I 2022-10-28 08:39:29] ...optimizers/trijoint.py.151: total_norm: 9.487700 [I 2022-10-28 08:39:34] ...trap/engines/engine.py.224: train: epoch 116 | batch 490/2304 [I 2022-10-28 08:39:34] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:58 [I 2022-10-28 08:39:34] ...trap/engines/engine.py.229: process: 0.55923 | load: 0.00248 [I 2022-10-28 08:39:34] ...trap/engines/engine.py.230: loss: 0.18719 [I 2022-10-28 08:39:34] ...optimizers/trijoint.py.151: total_norm: 10.737791 [I 2022-10-28 08:39:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 500/2304 [I 2022-10-28 08:39:40] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:46 [I 2022-10-28 08:39:40] ...trap/engines/engine.py.229: process: 0.56311 | load: 0.00250 [I 2022-10-28 08:39:40] ...trap/engines/engine.py.230: loss: 0.07529 [I 2022-10-28 08:39:40] ...optimizers/trijoint.py.151: total_norm: 8.941873 [I 2022-10-28 08:39:46] ...trap/engines/engine.py.224: train: epoch 116 | batch 510/2304 [I 2022-10-28 08:39:46] ...trap/engines/engine.py.228: elapsed: 0:04:49 | left: 0:16:45 [I 2022-10-28 08:39:46] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00255 [I 2022-10-28 08:39:46] ...trap/engines/engine.py.230: loss: 0.11246 [I 2022-10-28 08:39:46] ...optimizers/trijoint.py.151: total_norm: 15.291921 [I 2022-10-28 08:39:51] ...trap/engines/engine.py.224: train: epoch 116 | batch 520/2304 [I 2022-10-28 08:39:51] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:35 [I 2022-10-28 08:39:51] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00238 [I 2022-10-28 08:39:51] ...trap/engines/engine.py.230: loss: 0.11504 [I 2022-10-28 08:39:51] ...optimizers/trijoint.py.151: total_norm: 21.078518 [I 2022-10-28 08:39:57] ...trap/engines/engine.py.224: train: epoch 116 | batch 530/2304 [I 2022-10-28 08:39:57] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:41 [I 2022-10-28 08:39:57] ...trap/engines/engine.py.229: process: 0.56763 | load: 0.00244 [I 2022-10-28 08:39:57] ...trap/engines/engine.py.230: loss: 0.14534 [I 2022-10-28 08:39:57] ...optimizers/trijoint.py.151: total_norm: 14.232403 [I 2022-10-28 08:40:02] ...trap/engines/engine.py.224: train: epoch 116 | batch 540/2304 [I 2022-10-28 08:40:02] ...trap/engines/engine.py.228: elapsed: 0:05:06 | left: 0:16:37 [I 2022-10-28 08:40:02] ...trap/engines/engine.py.229: process: 0.56422 | load: 0.00243 [I 2022-10-28 08:40:02] ...trap/engines/engine.py.230: loss: 0.11542 [I 2022-10-28 08:40:02] ...optimizers/trijoint.py.151: total_norm: 10.833076 [I 2022-10-28 08:40:08] ...trap/engines/engine.py.224: train: epoch 116 | batch 550/2304 [I 2022-10-28 08:40:08] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:29 [I 2022-10-28 08:40:08] ...trap/engines/engine.py.229: process: 0.58300 | load: 0.00253 [I 2022-10-28 08:40:08] ...trap/engines/engine.py.230: loss: 0.07913 [I 2022-10-28 08:40:08] ...optimizers/trijoint.py.151: total_norm: 12.404335 [I 2022-10-28 08:40:14] ...trap/engines/engine.py.224: train: epoch 116 | batch 560/2304 [I 2022-10-28 08:40:14] ...trap/engines/engine.py.228: elapsed: 0:05:17 | left: 0:16:15 [I 2022-10-28 08:40:14] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00257 [I 2022-10-28 08:40:14] ...trap/engines/engine.py.230: loss: 0.08964 [I 2022-10-28 08:40:14] ...optimizers/trijoint.py.151: total_norm: 16.376730 [I 2022-10-28 08:40:19] ...trap/engines/engine.py.224: train: epoch 116 | batch 570/2304 [I 2022-10-28 08:40:19] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:05 [I 2022-10-28 08:40:19] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00244 [I 2022-10-28 08:40:19] ...trap/engines/engine.py.230: loss: 0.12884 [I 2022-10-28 08:40:19] ...optimizers/trijoint.py.151: total_norm: 9.670498 [I 2022-10-28 08:40:25] ...trap/engines/engine.py.224: train: epoch 116 | batch 580/2304 [I 2022-10-28 08:40:25] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:15:57 [I 2022-10-28 08:40:25] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00252 [I 2022-10-28 08:40:25] ...trap/engines/engine.py.230: loss: 0.15300 [I 2022-10-28 08:40:25] ...optimizers/trijoint.py.151: total_norm: 26.864622 [I 2022-10-28 08:40:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 590/2304 [I 2022-10-28 08:40:30] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:53 [I 2022-10-28 08:40:30] ...trap/engines/engine.py.229: process: 0.55810 | load: 0.00242 [I 2022-10-28 08:40:30] ...trap/engines/engine.py.230: loss: 0.15092 [I 2022-10-28 08:40:30] ...optimizers/trijoint.py.151: total_norm: 26.796526 [I 2022-10-28 08:40:36] ...trap/engines/engine.py.224: train: epoch 116 | batch 600/2304 [I 2022-10-28 08:40:36] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:16:13 [I 2022-10-28 08:40:36] ...trap/engines/engine.py.229: process: 0.55915 | load: 0.00242 [I 2022-10-28 08:40:36] ...trap/engines/engine.py.230: loss: 0.09175 [I 2022-10-28 08:40:36] ...optimizers/trijoint.py.151: total_norm: 9.667233 [I 2022-10-28 08:40:42] ...trap/engines/engine.py.224: train: epoch 116 | batch 610/2304 [I 2022-10-28 08:40:42] ...trap/engines/engine.py.228: elapsed: 0:05:45 | left: 0:15:52 [I 2022-10-28 08:40:42] ...trap/engines/engine.py.229: process: 0.56091 | load: 0.00264 [I 2022-10-28 08:40:42] ...trap/engines/engine.py.230: loss: 0.09328 [I 2022-10-28 08:40:42] ...optimizers/trijoint.py.151: total_norm: 9.467940 [I 2022-10-28 08:40:47] ...trap/engines/engine.py.224: train: epoch 116 | batch 620/2304 [I 2022-10-28 08:40:47] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:40 [I 2022-10-28 08:40:47] ...trap/engines/engine.py.229: process: 0.54809 | load: 0.00243 [I 2022-10-28 08:40:47] ...trap/engines/engine.py.230: loss: 0.08770 [I 2022-10-28 08:40:47] ...optimizers/trijoint.py.151: total_norm: 18.537964 [I 2022-10-28 08:40:53] ...trap/engines/engine.py.224: train: epoch 116 | batch 630/2304 [I 2022-10-28 08:40:53] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:38 [I 2022-10-28 08:40:53] ...trap/engines/engine.py.229: process: 0.56381 | load: 0.00243 [I 2022-10-28 08:40:53] ...trap/engines/engine.py.230: loss: 0.11707 [I 2022-10-28 08:40:53] ...optimizers/trijoint.py.151: total_norm: 8.898745 [I 2022-10-28 08:40:59] ...trap/engines/engine.py.224: train: epoch 116 | batch 640/2304 [I 2022-10-28 08:40:59] ...trap/engines/engine.py.228: elapsed: 0:06:02 | left: 0:15:27 [I 2022-10-28 08:40:59] ...trap/engines/engine.py.229: process: 0.55450 | load: 0.00243 [I 2022-10-28 08:40:59] ...trap/engines/engine.py.230: loss: 0.13457 [I 2022-10-28 08:40:59] ...optimizers/trijoint.py.151: total_norm: 32.571560 [I 2022-10-28 08:41:04] ...trap/engines/engine.py.224: train: epoch 116 | batch 650/2304 [I 2022-10-28 08:41:04] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:32 [I 2022-10-28 08:41:04] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00255 [I 2022-10-28 08:41:04] ...trap/engines/engine.py.230: loss: 0.10514 [I 2022-10-28 08:41:04] ...optimizers/trijoint.py.151: total_norm: 11.572676 [I 2022-10-28 08:41:10] ...trap/engines/engine.py.224: train: epoch 116 | batch 660/2304 [I 2022-10-28 08:41:10] ...trap/engines/engine.py.228: elapsed: 0:06:13 | left: 0:15:49 [I 2022-10-28 08:41:10] ...trap/engines/engine.py.229: process: 0.63312 | load: 0.00248 [I 2022-10-28 08:41:10] ...trap/engines/engine.py.230: loss: 0.07889 [I 2022-10-28 08:41:10] ...optimizers/trijoint.py.151: total_norm: 9.040962 [I 2022-10-28 08:41:16] ...trap/engines/engine.py.224: train: epoch 116 | batch 670/2304 [I 2022-10-28 08:41:16] ...trap/engines/engine.py.228: elapsed: 0:06:19 | left: 0:15:24 [I 2022-10-28 08:41:16] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00242 [I 2022-10-28 08:41:16] ...trap/engines/engine.py.230: loss: 0.14377 [I 2022-10-28 08:41:16] ...optimizers/trijoint.py.151: total_norm: 11.586221 [I 2022-10-28 08:41:21] ...trap/engines/engine.py.224: train: epoch 116 | batch 680/2304 [I 2022-10-28 08:41:21] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:10 [I 2022-10-28 08:41:21] ...trap/engines/engine.py.229: process: 0.55313 | load: 0.00255 [I 2022-10-28 08:41:21] ...trap/engines/engine.py.230: loss: 0.10204 [I 2022-10-28 08:41:21] ...optimizers/trijoint.py.151: total_norm: 19.660070 [I 2022-10-28 08:41:27] ...trap/engines/engine.py.224: train: epoch 116 | batch 690/2304 [I 2022-10-28 08:41:27] ...trap/engines/engine.py.228: elapsed: 0:06:30 | left: 0:14:57 [I 2022-10-28 08:41:27] ...trap/engines/engine.py.229: process: 0.54142 | load: 0.00243 [I 2022-10-28 08:41:27] ...trap/engines/engine.py.230: loss: 0.10082 [I 2022-10-28 08:41:27] ...optimizers/trijoint.py.151: total_norm: 16.673534 [I 2022-10-28 08:41:32] ...trap/engines/engine.py.224: train: epoch 116 | batch 700/2304 [I 2022-10-28 08:41:32] ...trap/engines/engine.py.228: elapsed: 0:06:36 | left: 0:14:55 [I 2022-10-28 08:41:32] ...trap/engines/engine.py.229: process: 0.55304 | load: 0.00244 [I 2022-10-28 08:41:32] ...trap/engines/engine.py.230: loss: 0.14823 [I 2022-10-28 08:41:32] ...optimizers/trijoint.py.151: total_norm: 22.269793 [I 2022-10-28 08:41:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 710/2304 [I 2022-10-28 08:41:38] ...trap/engines/engine.py.228: elapsed: 0:06:41 | left: 0:14:43 [I 2022-10-28 08:41:38] ...trap/engines/engine.py.229: process: 0.55183 | load: 0.00242 [I 2022-10-28 08:41:38] ...trap/engines/engine.py.230: loss: 0.12200 [I 2022-10-28 08:41:38] ...optimizers/trijoint.py.151: total_norm: 13.842790 [I 2022-10-28 08:41:44] ...trap/engines/engine.py.224: train: epoch 116 | batch 720/2304 [I 2022-10-28 08:41:44] ...trap/engines/engine.py.228: elapsed: 0:06:47 | left: 0:14:43 [I 2022-10-28 08:41:44] ...trap/engines/engine.py.229: process: 0.56722 | load: 0.00259 [I 2022-10-28 08:41:44] ...trap/engines/engine.py.230: loss: 0.12911 [I 2022-10-28 08:41:44] ...optimizers/trijoint.py.151: total_norm: 33.102642 [I 2022-10-28 08:41:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 730/2304 [I 2022-10-28 08:41:49] ...trap/engines/engine.py.228: elapsed: 0:06:52 | left: 0:14:39 [I 2022-10-28 08:41:49] ...trap/engines/engine.py.229: process: 0.54596 | load: 0.00247 [I 2022-10-28 08:41:49] ...trap/engines/engine.py.230: loss: 0.10784 [I 2022-10-28 08:41:49] ...optimizers/trijoint.py.151: total_norm: 11.319511 [I 2022-10-28 08:41:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 740/2304 [I 2022-10-28 08:41:55] ...trap/engines/engine.py.228: elapsed: 0:06:58 | left: 0:14:39 [I 2022-10-28 08:41:55] ...trap/engines/engine.py.229: process: 0.56279 | load: 0.00245 [I 2022-10-28 08:41:55] ...trap/engines/engine.py.230: loss: 0.07403 [I 2022-10-28 08:41:55] ...optimizers/trijoint.py.151: total_norm: 11.132318 [I 2022-10-28 08:42:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 750/2304 [I 2022-10-28 08:42:01] ...trap/engines/engine.py.228: elapsed: 0:07:04 | left: 0:14:31 [I 2022-10-28 08:42:01] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00247 [I 2022-10-28 08:42:01] ...trap/engines/engine.py.230: loss: 0.10339 [I 2022-10-28 08:42:01] ...optimizers/trijoint.py.151: total_norm: 10.469996 [I 2022-10-28 08:42:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 760/2304 [I 2022-10-28 08:42:06] ...trap/engines/engine.py.228: elapsed: 0:07:09 | left: 0:14:26 [I 2022-10-28 08:42:06] ...trap/engines/engine.py.229: process: 0.57214 | load: 0.00247 [I 2022-10-28 08:42:06] ...trap/engines/engine.py.230: loss: 0.11472 [I 2022-10-28 08:42:06] ...optimizers/trijoint.py.151: total_norm: 8.754730 [I 2022-10-28 08:42:12] ...trap/engines/engine.py.224: train: epoch 116 | batch 770/2304 [I 2022-10-28 08:42:12] ...trap/engines/engine.py.228: elapsed: 0:07:15 | left: 0:14:21 [I 2022-10-28 08:42:12] ...trap/engines/engine.py.229: process: 0.55420 | load: 0.00243 [I 2022-10-28 08:42:12] ...trap/engines/engine.py.230: loss: 0.14538 [I 2022-10-28 08:42:12] ...optimizers/trijoint.py.151: total_norm: 13.374549 [I 2022-10-28 08:42:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 780/2304 [I 2022-10-28 08:42:17] ...trap/engines/engine.py.228: elapsed: 0:07:21 | left: 0:14:18 [I 2022-10-28 08:42:17] ...trap/engines/engine.py.229: process: 0.56746 | load: 0.00247 [I 2022-10-28 08:42:17] ...trap/engines/engine.py.230: loss: 0.13435 [I 2022-10-28 08:42:17] ...optimizers/trijoint.py.151: total_norm: 14.020359 [I 2022-10-28 08:42:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 790/2304 [I 2022-10-28 08:42:23] ...trap/engines/engine.py.228: elapsed: 0:07:26 | left: 0:14:02 [I 2022-10-28 08:42:23] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00246 [I 2022-10-28 08:42:23] ...trap/engines/engine.py.230: loss: 0.10800 [I 2022-10-28 08:42:23] ...optimizers/trijoint.py.151: total_norm: 9.231450 [I 2022-10-28 08:42:29] ...trap/engines/engine.py.224: train: epoch 116 | batch 800/2304 [I 2022-10-28 08:42:29] ...trap/engines/engine.py.228: elapsed: 0:07:32 | left: 0:13:57 [I 2022-10-28 08:42:29] ...trap/engines/engine.py.229: process: 0.57951 | load: 0.00267 [I 2022-10-28 08:42:29] ...trap/engines/engine.py.230: loss: 0.10544 [I 2022-10-28 08:42:29] ...optimizers/trijoint.py.151: total_norm: 22.168974 [I 2022-10-28 08:42:34] ...trap/engines/engine.py.224: train: epoch 116 | batch 810/2304 [I 2022-10-28 08:42:34] ...trap/engines/engine.py.228: elapsed: 0:07:37 | left: 0:13:42 [I 2022-10-28 08:42:34] ...trap/engines/engine.py.229: process: 0.53949 | load: 0.00242 [I 2022-10-28 08:42:34] ...trap/engines/engine.py.230: loss: 0.13415 [I 2022-10-28 08:42:34] ...optimizers/trijoint.py.151: total_norm: 11.044444 [I 2022-10-28 08:42:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 820/2304 [I 2022-10-28 08:42:40] ...trap/engines/engine.py.228: elapsed: 0:07:43 | left: 0:13:43 [I 2022-10-28 08:42:40] ...trap/engines/engine.py.229: process: 0.56613 | load: 0.00243 [I 2022-10-28 08:42:40] ...trap/engines/engine.py.230: loss: 0.11480 [I 2022-10-28 08:42:40] ...optimizers/trijoint.py.151: total_norm: 41.674992 [I 2022-10-28 08:42:45] ...trap/engines/engine.py.224: train: epoch 116 | batch 830/2304 [I 2022-10-28 08:42:45] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:41 [I 2022-10-28 08:42:45] ...trap/engines/engine.py.229: process: 0.55429 | load: 0.00244 [I 2022-10-28 08:42:45] ...trap/engines/engine.py.230: loss: 0.10628 [I 2022-10-28 08:42:45] ...optimizers/trijoint.py.151: total_norm: 9.685239 [I 2022-10-28 08:42:51] ...trap/engines/engine.py.224: train: epoch 116 | batch 840/2304 [I 2022-10-28 08:42:51] ...trap/engines/engine.py.228: elapsed: 0:07:54 | left: 0:13:34 [I 2022-10-28 08:42:51] ...trap/engines/engine.py.229: process: 0.55530 | load: 0.00253 [I 2022-10-28 08:42:51] ...trap/engines/engine.py.230: loss: 0.11826 [I 2022-10-28 08:42:51] ...optimizers/trijoint.py.151: total_norm: 23.876209 [I 2022-10-28 08:42:56] ...trap/engines/engine.py.224: train: epoch 116 | batch 850/2304 [I 2022-10-28 08:42:56] ...trap/engines/engine.py.228: elapsed: 0:08:00 | left: 0:13:30 [I 2022-10-28 08:42:56] ...trap/engines/engine.py.229: process: 0.56092 | load: 0.00247 [I 2022-10-28 08:42:56] ...trap/engines/engine.py.230: loss: 0.09820 [I 2022-10-28 08:42:56] ...optimizers/trijoint.py.151: total_norm: 8.909010 [I 2022-10-28 08:43:02] ...trap/engines/engine.py.224: train: epoch 116 | batch 860/2304 [I 2022-10-28 08:43:02] ...trap/engines/engine.py.228: elapsed: 0:08:05 | left: 0:13:26 [I 2022-10-28 08:43:02] ...trap/engines/engine.py.229: process: 0.54624 | load: 0.00266 [I 2022-10-28 08:43:02] ...trap/engines/engine.py.230: loss: 0.08942 [I 2022-10-28 08:43:02] ...optimizers/trijoint.py.151: total_norm: 10.929465 [I 2022-10-28 08:43:08] ...trap/engines/engine.py.224: train: epoch 116 | batch 870/2304 [I 2022-10-28 08:43:08] ...trap/engines/engine.py.228: elapsed: 0:08:11 | left: 0:13:23 [I 2022-10-28 08:43:08] ...trap/engines/engine.py.229: process: 0.56968 | load: 0.00245 [I 2022-10-28 08:43:08] ...trap/engines/engine.py.230: loss: 0.16076 [I 2022-10-28 08:43:08] ...optimizers/trijoint.py.151: total_norm: 16.851715 [I 2022-10-28 08:43:13] ...trap/engines/engine.py.224: train: epoch 116 | batch 880/2304 [I 2022-10-28 08:43:13] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:22 [I 2022-10-28 08:43:13] ...trap/engines/engine.py.229: process: 0.56489 | load: 0.00266 [I 2022-10-28 08:43:13] ...trap/engines/engine.py.230: loss: 0.12955 [I 2022-10-28 08:43:13] ...optimizers/trijoint.py.151: total_norm: 11.485026 [I 2022-10-28 08:43:19] ...trap/engines/engine.py.224: train: epoch 116 | batch 890/2304 [I 2022-10-28 08:43:19] ...trap/engines/engine.py.228: elapsed: 0:08:22 | left: 0:13:14 [I 2022-10-28 08:43:19] ...trap/engines/engine.py.229: process: 0.56016 | load: 0.00250 [I 2022-10-28 08:43:19] ...trap/engines/engine.py.230: loss: 0.19285 [I 2022-10-28 08:43:19] ...optimizers/trijoint.py.151: total_norm: 13.970331 [I 2022-10-28 08:43:25] ...trap/engines/engine.py.224: train: epoch 116 | batch 900/2304 [I 2022-10-28 08:43:25] ...trap/engines/engine.py.228: elapsed: 0:08:28 | left: 0:13:07 [I 2022-10-28 08:43:25] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00241 [I 2022-10-28 08:43:25] ...trap/engines/engine.py.230: loss: 0.08967 [I 2022-10-28 08:43:25] ...optimizers/trijoint.py.151: total_norm: 13.085281 [I 2022-10-28 08:43:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 910/2304 [I 2022-10-28 08:43:30] ...trap/engines/engine.py.228: elapsed: 0:08:33 | left: 0:12:58 [I 2022-10-28 08:43:30] ...trap/engines/engine.py.229: process: 0.54754 | load: 0.00247 [I 2022-10-28 08:43:30] ...trap/engines/engine.py.230: loss: 0.12306 [I 2022-10-28 08:43:30] ...optimizers/trijoint.py.151: total_norm: 18.848392 [I 2022-10-28 08:43:36] ...trap/engines/engine.py.224: train: epoch 116 | batch 920/2304 [I 2022-10-28 08:43:36] ...trap/engines/engine.py.228: elapsed: 0:08:39 | left: 0:12:54 [I 2022-10-28 08:43:36] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00256 [I 2022-10-28 08:43:36] ...trap/engines/engine.py.230: loss: 0.14185 [I 2022-10-28 08:43:36] ...optimizers/trijoint.py.151: total_norm: 12.351339 [I 2022-10-28 08:43:41] ...trap/engines/engine.py.224: train: epoch 116 | batch 930/2304 [I 2022-10-28 08:43:41] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:12:50 [I 2022-10-28 08:43:41] ...trap/engines/engine.py.229: process: 0.55219 | load: 0.00248 [I 2022-10-28 08:43:41] ...trap/engines/engine.py.230: loss: 0.10290 [I 2022-10-28 08:43:41] ...optimizers/trijoint.py.151: total_norm: 26.541731 [I 2022-10-28 08:43:47] ...trap/engines/engine.py.224: train: epoch 116 | batch 940/2304 [I 2022-10-28 08:43:47] ...trap/engines/engine.py.228: elapsed: 0:08:50 | left: 0:12:47 [I 2022-10-28 08:43:47] ...trap/engines/engine.py.229: process: 0.55457 | load: 0.00251 [I 2022-10-28 08:43:47] ...trap/engines/engine.py.230: loss: 0.07544 [I 2022-10-28 08:43:47] ...optimizers/trijoint.py.151: total_norm: 19.251020 [I 2022-10-28 08:43:53] ...trap/engines/engine.py.224: train: epoch 116 | batch 950/2304 [I 2022-10-28 08:43:53] ...trap/engines/engine.py.228: elapsed: 0:08:56 | left: 0:12:36 [I 2022-10-28 08:43:53] ...trap/engines/engine.py.229: process: 0.55475 | load: 0.00246 [I 2022-10-28 08:43:53] ...trap/engines/engine.py.230: loss: 0.13306 [I 2022-10-28 08:43:53] ...optimizers/trijoint.py.151: total_norm: 22.663727 [I 2022-10-28 08:43:58] ...trap/engines/engine.py.224: train: epoch 116 | batch 960/2304 [I 2022-10-28 08:43:58] ...trap/engines/engine.py.228: elapsed: 0:09:01 | left: 0:12:31 [I 2022-10-28 08:43:58] ...trap/engines/engine.py.229: process: 0.56529 | load: 0.00247 [I 2022-10-28 08:43:58] ...trap/engines/engine.py.230: loss: 0.14936 [I 2022-10-28 08:43:58] ...optimizers/trijoint.py.151: total_norm: 17.454580 [I 2022-10-28 08:44:04] ...trap/engines/engine.py.224: train: epoch 116 | batch 970/2304 [I 2022-10-28 08:44:04] ...trap/engines/engine.py.228: elapsed: 0:09:07 | left: 0:12:31 [I 2022-10-28 08:44:04] ...trap/engines/engine.py.229: process: 0.56551 | load: 0.00244 [I 2022-10-28 08:44:04] ...trap/engines/engine.py.230: loss: 0.09852 [I 2022-10-28 08:44:04] ...optimizers/trijoint.py.151: total_norm: 13.696066 [I 2022-10-28 08:44:10] ...trap/engines/engine.py.224: train: epoch 116 | batch 980/2304 [I 2022-10-28 08:44:10] ...trap/engines/engine.py.228: elapsed: 0:09:13 | left: 0:12:23 [I 2022-10-28 08:44:10] ...trap/engines/engine.py.229: process: 0.55596 | load: 0.00242 [I 2022-10-28 08:44:10] ...trap/engines/engine.py.230: loss: 0.07215 [I 2022-10-28 08:44:10] ...optimizers/trijoint.py.151: total_norm: 18.225279 [I 2022-10-28 08:44:15] ...trap/engines/engine.py.224: train: epoch 116 | batch 990/2304 [I 2022-10-28 08:44:15] ...trap/engines/engine.py.228: elapsed: 0:09:18 | left: 0:12:18 [I 2022-10-28 08:44:15] ...trap/engines/engine.py.229: process: 0.55872 | load: 0.00246 [I 2022-10-28 08:44:15] ...trap/engines/engine.py.230: loss: 0.08371 [I 2022-10-28 08:44:15] ...optimizers/trijoint.py.151: total_norm: 18.641329 [I 2022-10-28 08:44:21] ...trap/engines/engine.py.224: train: epoch 116 | batch 1000/2304 [I 2022-10-28 08:44:21] ...trap/engines/engine.py.228: elapsed: 0:09:24 | left: 0:12:10 [I 2022-10-28 08:44:21] ...trap/engines/engine.py.229: process: 0.56238 | load: 0.00242 [I 2022-10-28 08:44:21] ...trap/engines/engine.py.230: loss: 0.20294 [I 2022-10-28 08:44:21] ...optimizers/trijoint.py.151: total_norm: 25.196449 [I 2022-10-28 08:44:26] ...trap/engines/engine.py.224: train: epoch 116 | batch 1010/2304 [I 2022-10-28 08:44:26] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:12:08 [I 2022-10-28 08:44:26] ...trap/engines/engine.py.229: process: 0.56841 | load: 0.00242 [I 2022-10-28 08:44:26] ...trap/engines/engine.py.230: loss: 0.09518 [I 2022-10-28 08:44:26] ...optimizers/trijoint.py.151: total_norm: 9.864470 [I 2022-10-28 08:44:32] ...trap/engines/engine.py.224: train: epoch 116 | batch 1020/2304 [I 2022-10-28 08:44:32] ...trap/engines/engine.py.228: elapsed: 0:09:35 | left: 0:11:58 [I 2022-10-28 08:44:32] ...trap/engines/engine.py.229: process: 0.55395 | load: 0.00241 [I 2022-10-28 08:44:32] ...trap/engines/engine.py.230: loss: 0.15403 [I 2022-10-28 08:44:32] ...optimizers/trijoint.py.151: total_norm: 15.146943 [I 2022-10-28 08:44:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 1030/2304 [I 2022-10-28 08:44:38] ...trap/engines/engine.py.228: elapsed: 0:09:41 | left: 0:11:55 [I 2022-10-28 08:44:38] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00252 [I 2022-10-28 08:44:38] ...trap/engines/engine.py.230: loss: 0.17103 [I 2022-10-28 08:44:38] ...optimizers/trijoint.py.151: total_norm: 42.821087 [I 2022-10-28 08:44:43] ...trap/engines/engine.py.224: train: epoch 116 | batch 1040/2304 [I 2022-10-28 08:44:43] ...trap/engines/engine.py.228: elapsed: 0:09:46 | left: 0:11:50 [I 2022-10-28 08:44:43] ...trap/engines/engine.py.229: process: 0.56220 | load: 0.00258 [I 2022-10-28 08:44:43] ...trap/engines/engine.py.230: loss: 0.11288 [I 2022-10-28 08:44:43] ...optimizers/trijoint.py.151: total_norm: 15.345806 [I 2022-10-28 08:44:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 1050/2304 [I 2022-10-28 08:44:49] ...trap/engines/engine.py.228: elapsed: 0:09:52 | left: 0:11:45 [I 2022-10-28 08:44:49] ...trap/engines/engine.py.229: process: 0.57327 | load: 0.00248 [I 2022-10-28 08:44:49] ...trap/engines/engine.py.230: loss: 0.11367 [I 2022-10-28 08:44:49] ...optimizers/trijoint.py.151: total_norm: 29.012295 [I 2022-10-28 08:44:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 1060/2304 [I 2022-10-28 08:44:55] ...trap/engines/engine.py.228: elapsed: 0:09:58 | left: 0:11:40 [I 2022-10-28 08:44:55] ...trap/engines/engine.py.229: process: 0.55926 | load: 0.00243 [I 2022-10-28 08:44:55] ...trap/engines/engine.py.230: loss: 0.08413 [I 2022-10-28 08:44:55] ...optimizers/trijoint.py.151: total_norm: 10.034057 [I 2022-10-28 08:45:00] ...trap/engines/engine.py.224: train: epoch 116 | batch 1070/2304 [I 2022-10-28 08:45:00] ...trap/engines/engine.py.228: elapsed: 0:10:03 | left: 0:11:36 [I 2022-10-28 08:45:00] ...trap/engines/engine.py.229: process: 0.57669 | load: 0.00251 [I 2022-10-28 08:45:00] ...trap/engines/engine.py.230: loss: 0.10022 [I 2022-10-28 08:45:00] ...optimizers/trijoint.py.151: total_norm: 16.524141 [I 2022-10-28 08:45:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 1080/2304 [I 2022-10-28 08:45:06] ...trap/engines/engine.py.228: elapsed: 0:10:09 | left: 0:11:23 [I 2022-10-28 08:45:06] ...trap/engines/engine.py.229: process: 0.55716 | load: 0.00250 [I 2022-10-28 08:45:06] ...trap/engines/engine.py.230: loss: 0.16788 [I 2022-10-28 08:45:06] ...optimizers/trijoint.py.151: total_norm: 22.892784 [I 2022-10-28 08:45:11] ...trap/engines/engine.py.224: train: epoch 116 | batch 1090/2304 [I 2022-10-28 08:45:11] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:16 [I 2022-10-28 08:45:11] ...trap/engines/engine.py.229: process: 0.56064 | load: 0.00246 [I 2022-10-28 08:45:11] ...trap/engines/engine.py.230: loss: 0.10416 [I 2022-10-28 08:45:11] ...optimizers/trijoint.py.151: total_norm: 18.835970 [I 2022-10-28 08:45:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 1100/2304 [I 2022-10-28 08:45:17] ...trap/engines/engine.py.228: elapsed: 0:10:20 | left: 0:11:16 [I 2022-10-28 08:45:17] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00247 [I 2022-10-28 08:45:17] ...trap/engines/engine.py.230: loss: 0.13621 [I 2022-10-28 08:45:17] ...optimizers/trijoint.py.151: total_norm: 9.087940 [I 2022-10-28 08:45:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 1110/2304 [I 2022-10-28 08:45:23] ...trap/engines/engine.py.228: elapsed: 0:10:26 | left: 0:11:14 [I 2022-10-28 08:45:23] ...trap/engines/engine.py.229: process: 0.57109 | load: 0.00246 [I 2022-10-28 08:45:23] ...trap/engines/engine.py.230: loss: 0.22004 [I 2022-10-28 08:45:23] ...optimizers/trijoint.py.151: total_norm: 37.095470 [I 2022-10-28 08:45:28] ...trap/engines/engine.py.224: train: epoch 116 | batch 1120/2304 [I 2022-10-28 08:45:28] ...trap/engines/engine.py.228: elapsed: 0:10:31 | left: 0:11:09 [I 2022-10-28 08:45:28] ...trap/engines/engine.py.229: process: 0.56185 | load: 0.00248 [I 2022-10-28 08:45:28] ...trap/engines/engine.py.230: loss: 0.15577 [I 2022-10-28 08:45:28] ...optimizers/trijoint.py.151: total_norm: 12.930310 [I 2022-10-28 08:45:34] ...trap/engines/engine.py.224: train: epoch 116 | batch 1130/2304 [I 2022-10-28 08:45:34] ...trap/engines/engine.py.228: elapsed: 0:10:37 | left: 0:11:02 [I 2022-10-28 08:45:34] ...trap/engines/engine.py.229: process: 0.57024 | load: 0.00246 [I 2022-10-28 08:45:34] ...trap/engines/engine.py.230: loss: 0.09775 [I 2022-10-28 08:45:34] ...optimizers/trijoint.py.151: total_norm: 10.690585 [I 2022-10-28 08:45:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 1140/2304 [I 2022-10-28 08:45:40] ...trap/engines/engine.py.228: elapsed: 0:10:43 | left: 0:10:45 [I 2022-10-28 08:45:40] ...trap/engines/engine.py.229: process: 0.54206 | load: 0.00245 [I 2022-10-28 08:45:40] ...trap/engines/engine.py.230: loss: 0.10666 [I 2022-10-28 08:45:40] ...optimizers/trijoint.py.151: total_norm: 21.297915 [I 2022-10-28 08:45:45] ...trap/engines/engine.py.224: train: epoch 116 | batch 1150/2304 [I 2022-10-28 08:45:45] ...trap/engines/engine.py.228: elapsed: 0:10:48 | left: 0:10:44 [I 2022-10-28 08:45:45] ...trap/engines/engine.py.229: process: 0.54562 | load: 0.00243 [I 2022-10-28 08:45:45] ...trap/engines/engine.py.230: loss: 0.11167 [I 2022-10-28 08:45:45] ...optimizers/trijoint.py.151: total_norm: 13.505185 [I 2022-10-28 08:45:51] ...trap/engines/engine.py.224: train: epoch 116 | batch 1160/2304 [I 2022-10-28 08:45:51] ...trap/engines/engine.py.228: elapsed: 0:10:54 | left: 0:10:42 [I 2022-10-28 08:45:51] ...trap/engines/engine.py.229: process: 0.55211 | load: 0.00247 [I 2022-10-28 08:45:51] ...trap/engines/engine.py.230: loss: 0.20846 [I 2022-10-28 08:45:51] ...optimizers/trijoint.py.151: total_norm: 26.202728 [I 2022-10-28 08:45:56] ...trap/engines/engine.py.224: train: epoch 116 | batch 1170/2304 [I 2022-10-28 08:45:56] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:39 [I 2022-10-28 08:45:56] ...trap/engines/engine.py.229: process: 0.55829 | load: 0.00244 [I 2022-10-28 08:45:56] ...trap/engines/engine.py.230: loss: 0.10574 [I 2022-10-28 08:45:56] ...optimizers/trijoint.py.151: total_norm: 17.294905 [I 2022-10-28 08:46:02] ...trap/engines/engine.py.224: train: epoch 116 | batch 1180/2304 [I 2022-10-28 08:46:02] ...trap/engines/engine.py.228: elapsed: 0:11:05 | left: 0:10:24 [I 2022-10-28 08:46:02] ...trap/engines/engine.py.229: process: 0.55309 | load: 0.00249 [I 2022-10-28 08:46:02] ...trap/engines/engine.py.230: loss: 0.09115 [I 2022-10-28 08:46:02] ...optimizers/trijoint.py.151: total_norm: 15.323525 [I 2022-10-28 08:46:08] ...trap/engines/engine.py.224: train: epoch 116 | batch 1190/2304 [I 2022-10-28 08:46:08] ...trap/engines/engine.py.228: elapsed: 0:11:11 | left: 0:10:23 [I 2022-10-28 08:46:08] ...trap/engines/engine.py.229: process: 0.56593 | load: 0.00245 [I 2022-10-28 08:46:08] ...trap/engines/engine.py.230: loss: 0.12155 [I 2022-10-28 08:46:08] ...optimizers/trijoint.py.151: total_norm: 27.329098 [I 2022-10-28 08:46:13] ...trap/engines/engine.py.224: train: epoch 116 | batch 1200/2304 [I 2022-10-28 08:46:13] ...trap/engines/engine.py.228: elapsed: 0:11:16 | left: 0:10:19 [I 2022-10-28 08:46:13] ...trap/engines/engine.py.229: process: 0.56198 | load: 0.00252 [I 2022-10-28 08:46:13] ...trap/engines/engine.py.230: loss: 0.19406 [I 2022-10-28 08:46:13] ...optimizers/trijoint.py.151: total_norm: 20.272030 [I 2022-10-28 08:46:19] ...trap/engines/engine.py.224: train: epoch 116 | batch 1210/2304 [I 2022-10-28 08:46:19] ...trap/engines/engine.py.228: elapsed: 0:11:22 | left: 0:10:11 [I 2022-10-28 08:46:19] ...trap/engines/engine.py.229: process: 0.55618 | load: 0.00251 [I 2022-10-28 08:46:19] ...trap/engines/engine.py.230: loss: 0.13014 [I 2022-10-28 08:46:19] ...optimizers/trijoint.py.151: total_norm: 17.040926 [I 2022-10-28 08:46:24] ...trap/engines/engine.py.224: train: epoch 116 | batch 1220/2304 [I 2022-10-28 08:46:24] ...trap/engines/engine.py.228: elapsed: 0:11:27 | left: 0:10:07 [I 2022-10-28 08:46:24] ...trap/engines/engine.py.229: process: 0.56571 | load: 0.00252 [I 2022-10-28 08:46:24] ...trap/engines/engine.py.230: loss: 0.11920 [I 2022-10-28 08:46:24] ...optimizers/trijoint.py.151: total_norm: 26.126640 [I 2022-10-28 08:46:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 1230/2304 [I 2022-10-28 08:46:30] ...trap/engines/engine.py.228: elapsed: 0:11:33 | left: 0:10:02 [I 2022-10-28 08:46:30] ...trap/engines/engine.py.229: process: 0.55175 | load: 0.00246 [I 2022-10-28 08:46:30] ...trap/engines/engine.py.230: loss: 0.13824 [I 2022-10-28 08:46:30] ...optimizers/trijoint.py.151: total_norm: 17.890545 [I 2022-10-28 08:46:36] ...trap/engines/engine.py.224: train: epoch 116 | batch 1240/2304 [I 2022-10-28 08:46:36] ...trap/engines/engine.py.228: elapsed: 0:11:39 | left: 0:09:58 [I 2022-10-28 08:46:36] ...trap/engines/engine.py.229: process: 0.56520 | load: 0.00245 [I 2022-10-28 08:46:36] ...trap/engines/engine.py.230: loss: 0.13571 [I 2022-10-28 08:46:36] ...optimizers/trijoint.py.151: total_norm: 12.625566 [I 2022-10-28 08:46:41] ...trap/engines/engine.py.224: train: epoch 116 | batch 1250/2304 [I 2022-10-28 08:46:41] ...trap/engines/engine.py.228: elapsed: 0:11:44 | left: 0:09:49 [I 2022-10-28 08:46:41] ...trap/engines/engine.py.229: process: 0.55401 | load: 0.00243 [I 2022-10-28 08:46:41] ...trap/engines/engine.py.230: loss: 0.14315 [I 2022-10-28 08:46:41] ...optimizers/trijoint.py.151: total_norm: 9.885199 [I 2022-10-28 08:46:47] ...trap/engines/engine.py.224: train: epoch 116 | batch 1260/2304 [I 2022-10-28 08:46:47] ...trap/engines/engine.py.228: elapsed: 0:11:50 | left: 0:09:43 [I 2022-10-28 08:46:47] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00243 [I 2022-10-28 08:46:47] ...trap/engines/engine.py.230: loss: 0.19179 [I 2022-10-28 08:46:47] ...optimizers/trijoint.py.151: total_norm: 15.614976 [I 2022-10-28 08:46:53] ...trap/engines/engine.py.224: train: epoch 116 | batch 1270/2304 [I 2022-10-28 08:46:53] ...trap/engines/engine.py.228: elapsed: 0:11:56 | left: 0:09:49 [I 2022-10-28 08:46:53] ...trap/engines/engine.py.229: process: 0.56641 | load: 0.00251 [I 2022-10-28 08:46:53] ...trap/engines/engine.py.230: loss: 0.12246 [I 2022-10-28 08:46:53] ...optimizers/trijoint.py.151: total_norm: 33.874187 [I 2022-10-28 08:46:58] ...trap/engines/engine.py.224: train: epoch 116 | batch 1280/2304 [I 2022-10-28 08:46:58] ...trap/engines/engine.py.228: elapsed: 0:12:01 | left: 0:09:33 [I 2022-10-28 08:46:58] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00278 [I 2022-10-28 08:46:58] ...trap/engines/engine.py.230: loss: 0.12350 [I 2022-10-28 08:46:58] ...optimizers/trijoint.py.151: total_norm: 10.129566 [I 2022-10-28 08:47:04] ...trap/engines/engine.py.224: train: epoch 116 | batch 1290/2304 [I 2022-10-28 08:47:04] ...trap/engines/engine.py.228: elapsed: 0:12:07 | left: 0:09:31 [I 2022-10-28 08:47:04] ...trap/engines/engine.py.229: process: 0.56048 | load: 0.00247 [I 2022-10-28 08:47:04] ...trap/engines/engine.py.230: loss: 0.11533 [I 2022-10-28 08:47:04] ...optimizers/trijoint.py.151: total_norm: 26.113689 [I 2022-10-28 08:47:09] ...trap/engines/engine.py.224: train: epoch 116 | batch 1300/2304 [I 2022-10-28 08:47:09] ...trap/engines/engine.py.228: elapsed: 0:12:13 | left: 0:09:18 [I 2022-10-28 08:47:09] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00241 [I 2022-10-28 08:47:09] ...trap/engines/engine.py.230: loss: 0.08686 [I 2022-10-28 08:47:09] ...optimizers/trijoint.py.151: total_norm: 51.532307 [I 2022-10-28 08:47:15] ...trap/engines/engine.py.224: train: epoch 116 | batch 1310/2304 [I 2022-10-28 08:47:15] ...trap/engines/engine.py.228: elapsed: 0:12:18 | left: 0:09:11 [I 2022-10-28 08:47:15] ...trap/engines/engine.py.229: process: 0.55666 | load: 0.00243 [I 2022-10-28 08:47:15] ...trap/engines/engine.py.230: loss: 0.25642 [I 2022-10-28 08:47:15] ...optimizers/trijoint.py.151: total_norm: 64.103210 [I 2022-10-28 08:47:21] ...trap/engines/engine.py.224: train: epoch 116 | batch 1320/2304 [I 2022-10-28 08:47:21] ...trap/engines/engine.py.228: elapsed: 0:12:24 | left: 0:09:10 [I 2022-10-28 08:47:21] ...trap/engines/engine.py.229: process: 0.57125 | load: 0.00243 [I 2022-10-28 08:47:21] ...trap/engines/engine.py.230: loss: 0.13439 [I 2022-10-28 08:47:21] ...optimizers/trijoint.py.151: total_norm: 11.253198 [I 2022-10-28 08:47:26] ...trap/engines/engine.py.224: train: epoch 116 | batch 1330/2304 [I 2022-10-28 08:47:26] ...trap/engines/engine.py.228: elapsed: 0:12:29 | left: 0:09:00 [I 2022-10-28 08:47:26] ...trap/engines/engine.py.229: process: 0.55158 | load: 0.00242 [I 2022-10-28 08:47:26] ...trap/engines/engine.py.230: loss: 0.11566 [I 2022-10-28 08:47:26] ...optimizers/trijoint.py.151: total_norm: 9.359852 [I 2022-10-28 08:47:32] ...trap/engines/engine.py.224: train: epoch 116 | batch 1340/2304 [I 2022-10-28 08:47:32] ...trap/engines/engine.py.228: elapsed: 0:12:35 | left: 0:09:07 [I 2022-10-28 08:47:32] ...trap/engines/engine.py.229: process: 0.56734 | load: 0.00249 [I 2022-10-28 08:47:32] ...trap/engines/engine.py.230: loss: 0.11375 [I 2022-10-28 08:47:32] ...optimizers/trijoint.py.151: total_norm: 14.280664 [I 2022-10-28 08:47:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 1350/2304 [I 2022-10-28 08:47:38] ...trap/engines/engine.py.228: elapsed: 0:12:41 | left: 0:08:57 [I 2022-10-28 08:47:38] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00243 [I 2022-10-28 08:47:38] ...trap/engines/engine.py.230: loss: 0.17327 [I 2022-10-28 08:47:38] ...optimizers/trijoint.py.151: total_norm: 28.442078 [I 2022-10-28 08:47:43] ...trap/engines/engine.py.224: train: epoch 116 | batch 1360/2304 [I 2022-10-28 08:47:43] ...trap/engines/engine.py.228: elapsed: 0:12:46 | left: 0:08:59 [I 2022-10-28 08:47:43] ...trap/engines/engine.py.229: process: 0.54740 | load: 0.00245 [I 2022-10-28 08:47:43] ...trap/engines/engine.py.230: loss: 0.15956 [I 2022-10-28 08:47:43] ...optimizers/trijoint.py.151: total_norm: 11.488474 [I 2022-10-28 08:47:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 1370/2304 [I 2022-10-28 08:47:49] ...trap/engines/engine.py.228: elapsed: 0:12:52 | left: 0:08:44 [I 2022-10-28 08:47:49] ...trap/engines/engine.py.229: process: 0.55276 | load: 0.00244 [I 2022-10-28 08:47:49] ...trap/engines/engine.py.230: loss: 0.10867 [I 2022-10-28 08:47:49] ...optimizers/trijoint.py.151: total_norm: 22.579866 [I 2022-10-28 08:47:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 1380/2304 [I 2022-10-28 08:47:55] ...trap/engines/engine.py.228: elapsed: 0:12:58 | left: 0:08:35 [I 2022-10-28 08:47:55] ...trap/engines/engine.py.229: process: 0.55396 | load: 0.00243 [I 2022-10-28 08:47:55] ...trap/engines/engine.py.230: loss: 0.09977 [I 2022-10-28 08:47:55] ...optimizers/trijoint.py.151: total_norm: 23.249901 [I 2022-10-28 08:48:00] ...trap/engines/engine.py.224: train: epoch 116 | batch 1390/2304 [I 2022-10-28 08:48:00] ...trap/engines/engine.py.228: elapsed: 0:13:03 | left: 0:08:33 [I 2022-10-28 08:48:00] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00255 [I 2022-10-28 08:48:00] ...trap/engines/engine.py.230: loss: 0.12861 [I 2022-10-28 08:48:00] ...optimizers/trijoint.py.151: total_norm: 8.125812 [I 2022-10-28 08:48:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 1400/2304 [I 2022-10-28 08:48:06] ...trap/engines/engine.py.228: elapsed: 0:13:09 | left: 0:08:32 [I 2022-10-28 08:48:06] ...trap/engines/engine.py.229: process: 0.57145 | load: 0.00253 [I 2022-10-28 08:48:06] ...trap/engines/engine.py.230: loss: 0.08706 [I 2022-10-28 08:48:06] ...optimizers/trijoint.py.151: total_norm: 13.801641 [I 2022-10-28 08:48:11] ...trap/engines/engine.py.224: train: epoch 116 | batch 1410/2304 [I 2022-10-28 08:48:11] ...trap/engines/engine.py.228: elapsed: 0:13:14 | left: 0:08:24 [I 2022-10-28 08:48:11] ...trap/engines/engine.py.229: process: 0.56731 | load: 0.00247 [I 2022-10-28 08:48:11] ...trap/engines/engine.py.230: loss: 0.09174 [I 2022-10-28 08:48:11] ...optimizers/trijoint.py.151: total_norm: 19.366964 [I 2022-10-28 08:48:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 1420/2304 [I 2022-10-28 08:48:17] ...trap/engines/engine.py.228: elapsed: 0:13:20 | left: 0:08:34 [I 2022-10-28 08:48:17] ...trap/engines/engine.py.229: process: 0.66920 | load: 0.00244 [I 2022-10-28 08:48:17] ...trap/engines/engine.py.230: loss: 0.14336 [I 2022-10-28 08:48:17] ...optimizers/trijoint.py.151: total_norm: 19.482328 [I 2022-10-28 08:48:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 1430/2304 [I 2022-10-28 08:48:23] ...trap/engines/engine.py.228: elapsed: 0:13:26 | left: 0:08:14 [I 2022-10-28 08:48:23] ...trap/engines/engine.py.229: process: 0.56511 | load: 0.00244 [I 2022-10-28 08:48:23] ...trap/engines/engine.py.230: loss: 0.14364 [I 2022-10-28 08:48:23] ...optimizers/trijoint.py.151: total_norm: 18.242903 [I 2022-10-28 08:48:28] ...trap/engines/engine.py.224: train: epoch 116 | batch 1440/2304 [I 2022-10-28 08:48:28] ...trap/engines/engine.py.228: elapsed: 0:13:31 | left: 0:08:02 [I 2022-10-28 08:48:28] ...trap/engines/engine.py.229: process: 0.55541 | load: 0.00247 [I 2022-10-28 08:48:28] ...trap/engines/engine.py.230: loss: 0.09650 [I 2022-10-28 08:48:28] ...optimizers/trijoint.py.151: total_norm: 10.260043 [I 2022-10-28 08:48:34] ...trap/engines/engine.py.224: train: epoch 116 | batch 1450/2304 [I 2022-10-28 08:48:34] ...trap/engines/engine.py.228: elapsed: 0:13:37 | left: 0:07:58 [I 2022-10-28 08:48:34] ...trap/engines/engine.py.229: process: 0.55802 | load: 0.00244 [I 2022-10-28 08:48:34] ...trap/engines/engine.py.230: loss: 0.14707 [I 2022-10-28 08:48:34] ...optimizers/trijoint.py.151: total_norm: 16.569059 [I 2022-10-28 08:48:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 1460/2304 [I 2022-10-28 08:48:40] ...trap/engines/engine.py.228: elapsed: 0:13:43 | left: 0:07:55 [I 2022-10-28 08:48:40] ...trap/engines/engine.py.229: process: 0.56845 | load: 0.00258 [I 2022-10-28 08:48:40] ...trap/engines/engine.py.230: loss: 0.12096 [I 2022-10-28 08:48:40] ...optimizers/trijoint.py.151: total_norm: 14.306910 [I 2022-10-28 08:48:45] ...trap/engines/engine.py.224: train: epoch 116 | batch 1470/2304 [I 2022-10-28 08:48:45] ...trap/engines/engine.py.228: elapsed: 0:13:48 | left: 0:07:48 [I 2022-10-28 08:48:45] ...trap/engines/engine.py.229: process: 0.55776 | load: 0.00256 [I 2022-10-28 08:48:45] ...trap/engines/engine.py.230: loss: 0.08457 [I 2022-10-28 08:48:45] ...optimizers/trijoint.py.151: total_norm: 13.781788 [I 2022-10-28 08:48:51] ...trap/engines/engine.py.224: train: epoch 116 | batch 1480/2304 [I 2022-10-28 08:48:51] ...trap/engines/engine.py.228: elapsed: 0:13:54 | left: 0:07:41 [I 2022-10-28 08:48:51] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00245 [I 2022-10-28 08:48:51] ...trap/engines/engine.py.230: loss: 0.10876 [I 2022-10-28 08:48:51] ...optimizers/trijoint.py.151: total_norm: 11.827264 [I 2022-10-28 08:48:57] ...trap/engines/engine.py.224: train: epoch 116 | batch 1490/2304 [I 2022-10-28 08:48:57] ...trap/engines/engine.py.228: elapsed: 0:14:00 | left: 0:07:37 [I 2022-10-28 08:48:57] ...trap/engines/engine.py.229: process: 0.56698 | load: 0.00273 [I 2022-10-28 08:48:57] ...trap/engines/engine.py.230: loss: 0.07773 [I 2022-10-28 08:48:57] ...optimizers/trijoint.py.151: total_norm: 14.386042 [I 2022-10-28 08:49:02] ...trap/engines/engine.py.224: train: epoch 116 | batch 1500/2304 [I 2022-10-28 08:49:02] ...trap/engines/engine.py.228: elapsed: 0:14:05 | left: 0:07:28 [I 2022-10-28 08:49:02] ...trap/engines/engine.py.229: process: 0.55275 | load: 0.00246 [I 2022-10-28 08:49:02] ...trap/engines/engine.py.230: loss: 0.12336 [I 2022-10-28 08:49:02] ...optimizers/trijoint.py.151: total_norm: 14.783920 [I 2022-10-28 08:49:08] ...trap/engines/engine.py.224: train: epoch 116 | batch 1510/2304 [I 2022-10-28 08:49:08] ...trap/engines/engine.py.228: elapsed: 0:14:11 | left: 0:07:27 [I 2022-10-28 08:49:08] ...trap/engines/engine.py.229: process: 0.56952 | load: 0.00243 [I 2022-10-28 08:49:08] ...trap/engines/engine.py.230: loss: 0.15234 [I 2022-10-28 08:49:08] ...optimizers/trijoint.py.151: total_norm: 14.858394 [I 2022-10-28 08:49:13] ...trap/engines/engine.py.224: train: epoch 116 | batch 1520/2304 [I 2022-10-28 08:49:13] ...trap/engines/engine.py.228: elapsed: 0:14:16 | left: 0:07:21 [I 2022-10-28 08:49:13] ...trap/engines/engine.py.229: process: 0.57759 | load: 0.00330 [I 2022-10-28 08:49:13] ...trap/engines/engine.py.230: loss: 0.10171 [I 2022-10-28 08:49:13] ...optimizers/trijoint.py.151: total_norm: 18.651777 [I 2022-10-28 08:49:19] ...trap/engines/engine.py.224: train: epoch 116 | batch 1530/2304 [I 2022-10-28 08:49:19] ...trap/engines/engine.py.228: elapsed: 0:14:22 | left: 0:07:13 [I 2022-10-28 08:49:19] ...trap/engines/engine.py.229: process: 0.55746 | load: 0.00247 [I 2022-10-28 08:49:19] ...trap/engines/engine.py.230: loss: 0.11610 [I 2022-10-28 08:49:19] ...optimizers/trijoint.py.151: total_norm: 9.862342 [I 2022-10-28 08:49:25] ...trap/engines/engine.py.224: train: epoch 116 | batch 1540/2304 [I 2022-10-28 08:49:25] ...trap/engines/engine.py.228: elapsed: 0:14:28 | left: 0:07:21 [I 2022-10-28 08:49:25] ...trap/engines/engine.py.229: process: 0.55655 | load: 0.00247 [I 2022-10-28 08:49:25] ...trap/engines/engine.py.230: loss: 0.13471 [I 2022-10-28 08:49:25] ...optimizers/trijoint.py.151: total_norm: 13.752485 [I 2022-10-28 08:49:30] ...trap/engines/engine.py.224: train: epoch 116 | batch 1550/2304 [I 2022-10-28 08:49:30] ...trap/engines/engine.py.228: elapsed: 0:14:33 | left: 0:07:02 [I 2022-10-28 08:49:30] ...trap/engines/engine.py.229: process: 0.56198 | load: 0.00267 [I 2022-10-28 08:49:30] ...trap/engines/engine.py.230: loss: 0.10781 [I 2022-10-28 08:49:30] ...optimizers/trijoint.py.151: total_norm: 29.392176 [I 2022-10-28 08:49:36] ...trap/engines/engine.py.224: train: epoch 116 | batch 1560/2304 [I 2022-10-28 08:49:36] ...trap/engines/engine.py.228: elapsed: 0:14:39 | left: 0:06:56 [I 2022-10-28 08:49:36] ...trap/engines/engine.py.229: process: 0.57326 | load: 0.00255 [I 2022-10-28 08:49:36] ...trap/engines/engine.py.230: loss: 0.13142 [I 2022-10-28 08:49:36] ...optimizers/trijoint.py.151: total_norm: 23.173971 [I 2022-10-28 08:49:42] ...trap/engines/engine.py.224: train: epoch 116 | batch 1570/2304 [I 2022-10-28 08:49:42] ...trap/engines/engine.py.228: elapsed: 0:14:45 | left: 0:06:51 [I 2022-10-28 08:49:42] ...trap/engines/engine.py.229: process: 0.55764 | load: 0.00248 [I 2022-10-28 08:49:42] ...trap/engines/engine.py.230: loss: 0.06886 [I 2022-10-28 08:49:42] ...optimizers/trijoint.py.151: total_norm: 10.055546 [I 2022-10-28 08:49:47] ...trap/engines/engine.py.224: train: epoch 116 | batch 1580/2304 [I 2022-10-28 08:49:47] ...trap/engines/engine.py.228: elapsed: 0:14:50 | left: 0:06:46 [I 2022-10-28 08:49:47] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00249 [I 2022-10-28 08:49:47] ...trap/engines/engine.py.230: loss: 0.14607 [I 2022-10-28 08:49:47] ...optimizers/trijoint.py.151: total_norm: 13.046002 [I 2022-10-28 08:49:53] ...trap/engines/engine.py.224: train: epoch 116 | batch 1590/2304 [I 2022-10-28 08:49:53] ...trap/engines/engine.py.228: elapsed: 0:14:56 | left: 0:06:38 [I 2022-10-28 08:49:53] ...trap/engines/engine.py.229: process: 0.54597 | load: 0.00247 [I 2022-10-28 08:49:53] ...trap/engines/engine.py.230: loss: 0.14945 [I 2022-10-28 08:49:53] ...optimizers/trijoint.py.151: total_norm: 38.329998 [I 2022-10-28 08:49:59] ...trap/engines/engine.py.224: train: epoch 116 | batch 1600/2304 [I 2022-10-28 08:49:59] ...trap/engines/engine.py.228: elapsed: 0:15:02 | left: 0:06:34 [I 2022-10-28 08:49:59] ...trap/engines/engine.py.229: process: 0.55608 | load: 0.00248 [I 2022-10-28 08:49:59] ...trap/engines/engine.py.230: loss: 0.08092 [I 2022-10-28 08:49:59] ...optimizers/trijoint.py.151: total_norm: 15.755650 [I 2022-10-28 08:50:04] ...trap/engines/engine.py.224: train: epoch 116 | batch 1610/2304 [I 2022-10-28 08:50:04] ...trap/engines/engine.py.228: elapsed: 0:15:07 | left: 0:06:35 [I 2022-10-28 08:50:04] ...trap/engines/engine.py.229: process: 0.58911 | load: 0.00247 [I 2022-10-28 08:50:04] ...trap/engines/engine.py.230: loss: 0.13678 [I 2022-10-28 08:50:04] ...optimizers/trijoint.py.151: total_norm: 15.825973 [I 2022-10-28 08:50:10] ...trap/engines/engine.py.224: train: epoch 116 | batch 1620/2304 [I 2022-10-28 08:50:10] ...trap/engines/engine.py.228: elapsed: 0:15:13 | left: 0:06:24 [I 2022-10-28 08:50:10] ...trap/engines/engine.py.229: process: 0.55690 | load: 0.00245 [I 2022-10-28 08:50:10] ...trap/engines/engine.py.230: loss: 0.11661 [I 2022-10-28 08:50:10] ...optimizers/trijoint.py.151: total_norm: 14.458679 [I 2022-10-28 08:50:15] ...trap/engines/engine.py.224: train: epoch 116 | batch 1630/2304 [I 2022-10-28 08:50:15] ...trap/engines/engine.py.228: elapsed: 0:15:19 | left: 0:06:21 [I 2022-10-28 08:50:15] ...trap/engines/engine.py.229: process: 0.57505 | load: 0.00260 [I 2022-10-28 08:50:15] ...trap/engines/engine.py.230: loss: 0.09493 [I 2022-10-28 08:50:15] ...optimizers/trijoint.py.151: total_norm: 12.019541 [I 2022-10-28 08:50:21] ...trap/engines/engine.py.224: train: epoch 116 | batch 1640/2304 [I 2022-10-28 08:50:21] ...trap/engines/engine.py.228: elapsed: 0:15:24 | left: 0:06:18 [I 2022-10-28 08:50:21] ...trap/engines/engine.py.229: process: 0.57354 | load: 0.00306 [I 2022-10-28 08:50:21] ...trap/engines/engine.py.230: loss: 0.10186 [I 2022-10-28 08:50:21] ...optimizers/trijoint.py.151: total_norm: 10.168292 [I 2022-10-28 08:50:27] ...trap/engines/engine.py.224: train: epoch 116 | batch 1650/2304 [I 2022-10-28 08:50:27] ...trap/engines/engine.py.228: elapsed: 0:15:30 | left: 0:06:09 [I 2022-10-28 08:50:27] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00256 [I 2022-10-28 08:50:27] ...trap/engines/engine.py.230: loss: 0.11132 [I 2022-10-28 08:50:27] ...optimizers/trijoint.py.151: total_norm: 16.912916 [I 2022-10-28 08:50:32] ...trap/engines/engine.py.224: train: epoch 116 | batch 1660/2304 [I 2022-10-28 08:50:32] ...trap/engines/engine.py.228: elapsed: 0:15:36 | left: 0:06:02 [I 2022-10-28 08:50:32] ...trap/engines/engine.py.229: process: 0.55880 | load: 0.00262 [I 2022-10-28 08:50:32] ...trap/engines/engine.py.230: loss: 0.12257 [I 2022-10-28 08:50:32] ...optimizers/trijoint.py.151: total_norm: 8.801912 [I 2022-10-28 08:50:38] ...trap/engines/engine.py.224: train: epoch 116 | batch 1670/2304 [I 2022-10-28 08:50:38] ...trap/engines/engine.py.228: elapsed: 0:15:41 | left: 0:05:55 [I 2022-10-28 08:50:38] ...trap/engines/engine.py.229: process: 0.55259 | load: 0.00246 [I 2022-10-28 08:50:38] ...trap/engines/engine.py.230: loss: 0.16239 [I 2022-10-28 08:50:38] ...optimizers/trijoint.py.151: total_norm: 16.702610 [I 2022-10-28 08:50:44] ...trap/engines/engine.py.224: train: epoch 116 | batch 1680/2304 [I 2022-10-28 08:50:44] ...trap/engines/engine.py.228: elapsed: 0:15:47 | left: 0:05:48 [I 2022-10-28 08:50:44] ...trap/engines/engine.py.229: process: 0.55479 | load: 0.00247 [I 2022-10-28 08:50:44] ...trap/engines/engine.py.230: loss: 0.14241 [I 2022-10-28 08:50:44] ...optimizers/trijoint.py.151: total_norm: 27.353867 [I 2022-10-28 08:50:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 1690/2304 [I 2022-10-28 08:50:49] ...trap/engines/engine.py.228: elapsed: 0:15:52 | left: 0:05:42 [I 2022-10-28 08:50:49] ...trap/engines/engine.py.229: process: 0.55145 | load: 0.00274 [I 2022-10-28 08:50:49] ...trap/engines/engine.py.230: loss: 0.10274 [I 2022-10-28 08:50:49] ...optimizers/trijoint.py.151: total_norm: 12.531628 [I 2022-10-28 08:50:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 1700/2304 [I 2022-10-28 08:50:55] ...trap/engines/engine.py.228: elapsed: 0:15:58 | left: 0:05:35 [I 2022-10-28 08:50:55] ...trap/engines/engine.py.229: process: 0.56228 | load: 0.00242 [I 2022-10-28 08:50:55] ...trap/engines/engine.py.230: loss: 0.12572 [I 2022-10-28 08:50:55] ...optimizers/trijoint.py.151: total_norm: 18.828981 [I 2022-10-28 08:51:00] ...trap/engines/engine.py.224: train: epoch 116 | batch 1710/2304 [I 2022-10-28 08:51:00] ...trap/engines/engine.py.228: elapsed: 0:16:04 | left: 0:05:31 [I 2022-10-28 08:51:00] ...trap/engines/engine.py.229: process: 0.56219 | load: 0.00243 [I 2022-10-28 08:51:00] ...trap/engines/engine.py.230: loss: 0.14341 [I 2022-10-28 08:51:00] ...optimizers/trijoint.py.151: total_norm: 22.290934 [I 2022-10-28 08:51:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 1720/2304 [I 2022-10-28 08:51:06] ...trap/engines/engine.py.228: elapsed: 0:16:09 | left: 0:05:24 [I 2022-10-28 08:51:06] ...trap/engines/engine.py.229: process: 0.55650 | load: 0.00268 [I 2022-10-28 08:51:06] ...trap/engines/engine.py.230: loss: 0.11380 [I 2022-10-28 08:51:06] ...optimizers/trijoint.py.151: total_norm: 22.946590 [I 2022-10-28 08:51:12] ...trap/engines/engine.py.224: train: epoch 116 | batch 1730/2304 [I 2022-10-28 08:51:12] ...trap/engines/engine.py.228: elapsed: 0:16:15 | left: 0:05:20 [I 2022-10-28 08:51:12] ...trap/engines/engine.py.229: process: 0.54997 | load: 0.00258 [I 2022-10-28 08:51:12] ...trap/engines/engine.py.230: loss: 0.08028 [I 2022-10-28 08:51:12] ...optimizers/trijoint.py.151: total_norm: 13.770939 [I 2022-10-28 08:51:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 1740/2304 [I 2022-10-28 08:51:17] ...trap/engines/engine.py.228: elapsed: 0:16:20 | left: 0:05:18 [I 2022-10-28 08:51:17] ...trap/engines/engine.py.229: process: 0.55390 | load: 0.00249 [I 2022-10-28 08:51:17] ...trap/engines/engine.py.230: loss: 0.15325 [I 2022-10-28 08:51:17] ...optimizers/trijoint.py.151: total_norm: 39.758053 [I 2022-10-28 08:51:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 1750/2304 [I 2022-10-28 08:51:23] ...trap/engines/engine.py.228: elapsed: 0:16:26 | left: 0:05:13 [I 2022-10-28 08:51:23] ...trap/engines/engine.py.229: process: 0.56430 | load: 0.00264 [I 2022-10-28 08:51:23] ...trap/engines/engine.py.230: loss: 0.09489 [I 2022-10-28 08:51:23] ...optimizers/trijoint.py.151: total_norm: 11.046099 [I 2022-10-28 08:51:29] ...trap/engines/engine.py.224: train: epoch 116 | batch 1760/2304 [I 2022-10-28 08:51:29] ...trap/engines/engine.py.228: elapsed: 0:16:32 | left: 0:05:07 [I 2022-10-28 08:51:29] ...trap/engines/engine.py.229: process: 0.56854 | load: 0.00264 [I 2022-10-28 08:51:29] ...trap/engines/engine.py.230: loss: 0.08882 [I 2022-10-28 08:51:29] ...optimizers/trijoint.py.151: total_norm: 10.059509 [I 2022-10-28 08:51:34] ...trap/engines/engine.py.224: train: epoch 116 | batch 1770/2304 [I 2022-10-28 08:51:34] ...trap/engines/engine.py.228: elapsed: 0:16:38 | left: 0:05:04 [I 2022-10-28 08:51:34] ...trap/engines/engine.py.229: process: 0.56192 | load: 0.00253 [I 2022-10-28 08:51:34] ...trap/engines/engine.py.230: loss: 0.08677 [I 2022-10-28 08:51:34] ...optimizers/trijoint.py.151: total_norm: 23.438431 [I 2022-10-28 08:51:40] ...trap/engines/engine.py.224: train: epoch 116 | batch 1780/2304 [I 2022-10-28 08:51:40] ...trap/engines/engine.py.228: elapsed: 0:16:43 | left: 0:05:05 [I 2022-10-28 08:51:40] ...trap/engines/engine.py.229: process: 0.57000 | load: 0.00252 [I 2022-10-28 08:51:40] ...trap/engines/engine.py.230: loss: 0.20921 [I 2022-10-28 08:51:40] ...optimizers/trijoint.py.151: total_norm: 29.703417 [I 2022-10-28 08:51:46] ...trap/engines/engine.py.224: train: epoch 116 | batch 1790/2304 [I 2022-10-28 08:51:46] ...trap/engines/engine.py.228: elapsed: 0:16:49 | left: 0:04:49 [I 2022-10-28 08:51:46] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00247 [I 2022-10-28 08:51:46] ...trap/engines/engine.py.230: loss: 0.10570 [I 2022-10-28 08:51:46] ...optimizers/trijoint.py.151: total_norm: 12.252556 [I 2022-10-28 08:51:51] ...trap/engines/engine.py.224: train: epoch 116 | batch 1800/2304 [I 2022-10-28 08:51:51] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:39 [I 2022-10-28 08:51:51] ...trap/engines/engine.py.229: process: 0.54902 | load: 0.00261 [I 2022-10-28 08:51:51] ...trap/engines/engine.py.230: loss: 0.10339 [I 2022-10-28 08:51:51] ...optimizers/trijoint.py.151: total_norm: 16.952682 [I 2022-10-28 08:51:57] ...trap/engines/engine.py.224: train: epoch 116 | batch 1810/2304 [I 2022-10-28 08:51:57] ...trap/engines/engine.py.228: elapsed: 0:17:00 | left: 0:04:40 [I 2022-10-28 08:51:57] ...trap/engines/engine.py.229: process: 0.55946 | load: 0.00247 [I 2022-10-28 08:51:57] ...trap/engines/engine.py.230: loss: 0.10510 [I 2022-10-28 08:51:57] ...optimizers/trijoint.py.151: total_norm: 10.290421 [I 2022-10-28 08:52:03] ...trap/engines/engine.py.224: train: epoch 116 | batch 1820/2304 [I 2022-10-28 08:52:03] ...trap/engines/engine.py.228: elapsed: 0:17:06 | left: 0:04:35 [I 2022-10-28 08:52:03] ...trap/engines/engine.py.229: process: 0.57046 | load: 0.00243 [I 2022-10-28 08:52:03] ...trap/engines/engine.py.230: loss: 0.09013 [I 2022-10-28 08:52:03] ...optimizers/trijoint.py.151: total_norm: 18.433575 [I 2022-10-28 08:52:08] ...trap/engines/engine.py.224: train: epoch 116 | batch 1830/2304 [I 2022-10-28 08:52:08] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:26 [I 2022-10-28 08:52:08] ...trap/engines/engine.py.229: process: 0.56903 | load: 0.00243 [I 2022-10-28 08:52:08] ...trap/engines/engine.py.230: loss: 0.09669 [I 2022-10-28 08:52:08] ...optimizers/trijoint.py.151: total_norm: 31.217348 [I 2022-10-28 08:52:14] ...trap/engines/engine.py.224: train: epoch 116 | batch 1840/2304 [I 2022-10-28 08:52:14] ...trap/engines/engine.py.228: elapsed: 0:17:17 | left: 0:04:21 [I 2022-10-28 08:52:14] ...trap/engines/engine.py.229: process: 0.55633 | load: 0.00275 [I 2022-10-28 08:52:14] ...trap/engines/engine.py.230: loss: 0.10911 [I 2022-10-28 08:52:14] ...optimizers/trijoint.py.151: total_norm: 19.652510 [I 2022-10-28 08:52:20] ...trap/engines/engine.py.224: train: epoch 116 | batch 1850/2304 [I 2022-10-28 08:52:20] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:14 [I 2022-10-28 08:52:20] ...trap/engines/engine.py.229: process: 0.55011 | load: 0.00245 [I 2022-10-28 08:52:20] ...trap/engines/engine.py.230: loss: 0.08503 [I 2022-10-28 08:52:20] ...optimizers/trijoint.py.151: total_norm: 10.678024 [I 2022-10-28 08:52:25] ...trap/engines/engine.py.224: train: epoch 116 | batch 1860/2304 [I 2022-10-28 08:52:25] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:12 [I 2022-10-28 08:52:25] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00253 [I 2022-10-28 08:52:25] ...trap/engines/engine.py.230: loss: 0.10638 [I 2022-10-28 08:52:25] ...optimizers/trijoint.py.151: total_norm: 11.620618 [I 2022-10-28 08:52:31] ...trap/engines/engine.py.224: train: epoch 116 | batch 1870/2304 [I 2022-10-28 08:52:31] ...trap/engines/engine.py.228: elapsed: 0:17:34 | left: 0:04:07 [I 2022-10-28 08:52:31] ...trap/engines/engine.py.229: process: 0.57945 | load: 0.00247 [I 2022-10-28 08:52:31] ...trap/engines/engine.py.230: loss: 0.17372 [I 2022-10-28 08:52:31] ...optimizers/trijoint.py.151: total_norm: 14.418226 [I 2022-10-28 08:52:37] ...trap/engines/engine.py.224: train: epoch 116 | batch 1880/2304 [I 2022-10-28 08:52:37] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:03:58 [I 2022-10-28 08:52:37] ...trap/engines/engine.py.229: process: 0.56122 | load: 0.00245 [I 2022-10-28 08:52:37] ...trap/engines/engine.py.230: loss: 0.10014 [I 2022-10-28 08:52:37] ...optimizers/trijoint.py.151: total_norm: 12.845757 [I 2022-10-28 08:52:43] ...trap/engines/engine.py.224: train: epoch 116 | batch 1890/2304 [I 2022-10-28 08:52:43] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:54 [I 2022-10-28 08:52:43] ...trap/engines/engine.py.229: process: 0.57100 | load: 0.00247 [I 2022-10-28 08:52:43] ...trap/engines/engine.py.230: loss: 0.12410 [I 2022-10-28 08:52:43] ...optimizers/trijoint.py.151: total_norm: 11.011589 [I 2022-10-28 08:52:48] ...trap/engines/engine.py.224: train: epoch 116 | batch 1900/2304 [I 2022-10-28 08:52:48] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:49 [I 2022-10-28 08:52:48] ...trap/engines/engine.py.229: process: 0.56536 | load: 0.00249 [I 2022-10-28 08:52:48] ...trap/engines/engine.py.230: loss: 0.14270 [I 2022-10-28 08:52:48] ...optimizers/trijoint.py.151: total_norm: 15.647726 [I 2022-10-28 08:52:54] ...trap/engines/engine.py.224: train: epoch 116 | batch 1910/2304 [I 2022-10-28 08:52:54] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:43 [I 2022-10-28 08:52:54] ...trap/engines/engine.py.229: process: 0.57840 | load: 0.00255 [I 2022-10-28 08:52:54] ...trap/engines/engine.py.230: loss: 0.11197 [I 2022-10-28 08:52:54] ...optimizers/trijoint.py.151: total_norm: 14.847148 [I 2022-10-28 08:53:00] ...trap/engines/engine.py.224: train: epoch 116 | batch 1920/2304 [I 2022-10-28 08:53:00] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:40 [I 2022-10-28 08:53:00] ...trap/engines/engine.py.229: process: 0.55969 | load: 0.00259 [I 2022-10-28 08:53:00] ...trap/engines/engine.py.230: loss: 0.12301 [I 2022-10-28 08:53:00] ...optimizers/trijoint.py.151: total_norm: 10.018022 [I 2022-10-28 08:53:05] ...trap/engines/engine.py.224: train: epoch 116 | batch 1930/2304 [I 2022-10-28 08:53:05] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:32 [I 2022-10-28 08:53:05] ...trap/engines/engine.py.229: process: 0.57385 | load: 0.00254 [I 2022-10-28 08:53:05] ...trap/engines/engine.py.230: loss: 0.10119 [I 2022-10-28 08:53:05] ...optimizers/trijoint.py.151: total_norm: 13.233418 [I 2022-10-28 08:53:11] ...trap/engines/engine.py.224: train: epoch 116 | batch 1940/2304 [I 2022-10-28 08:53:11] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:27 [I 2022-10-28 08:53:11] ...trap/engines/engine.py.229: process: 0.57060 | load: 0.00295 [I 2022-10-28 08:53:11] ...trap/engines/engine.py.230: loss: 0.09407 [I 2022-10-28 08:53:11] ...optimizers/trijoint.py.151: total_norm: 10.431952 [I 2022-10-28 08:53:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 1950/2304 [I 2022-10-28 08:53:17] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:18 [I 2022-10-28 08:53:17] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00245 [I 2022-10-28 08:53:17] ...trap/engines/engine.py.230: loss: 0.17218 [I 2022-10-28 08:53:17] ...optimizers/trijoint.py.151: total_norm: 17.953762 [I 2022-10-28 08:53:22] ...trap/engines/engine.py.224: train: epoch 116 | batch 1960/2304 [I 2022-10-28 08:53:22] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:12 [I 2022-10-28 08:53:22] ...trap/engines/engine.py.229: process: 0.56359 | load: 0.00244 [I 2022-10-28 08:53:22] ...trap/engines/engine.py.230: loss: 0.08153 [I 2022-10-28 08:53:22] ...optimizers/trijoint.py.151: total_norm: 12.498416 [I 2022-10-28 08:53:28] ...trap/engines/engine.py.224: train: epoch 116 | batch 1970/2304 [I 2022-10-28 08:53:28] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:08 [I 2022-10-28 08:53:28] ...trap/engines/engine.py.229: process: 0.55606 | load: 0.00255 [I 2022-10-28 08:53:28] ...trap/engines/engine.py.230: loss: 0.13186 [I 2022-10-28 08:53:28] ...optimizers/trijoint.py.151: total_norm: 13.350575 [I 2022-10-28 08:53:33] ...trap/engines/engine.py.224: train: epoch 116 | batch 1980/2304 [I 2022-10-28 08:53:33] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:03:01 [I 2022-10-28 08:53:33] ...trap/engines/engine.py.229: process: 0.55931 | load: 0.00245 [I 2022-10-28 08:53:33] ...trap/engines/engine.py.230: loss: 0.11865 [I 2022-10-28 08:53:33] ...optimizers/trijoint.py.151: total_norm: 20.089767 [I 2022-10-28 08:53:39] ...trap/engines/engine.py.224: train: epoch 116 | batch 1990/2304 [I 2022-10-28 08:53:39] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:55 [I 2022-10-28 08:53:39] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00259 [I 2022-10-28 08:53:39] ...trap/engines/engine.py.230: loss: 0.07660 [I 2022-10-28 08:53:39] ...optimizers/trijoint.py.151: total_norm: 14.325849 [I 2022-10-28 08:53:45] ...trap/engines/engine.py.224: train: epoch 116 | batch 2000/2304 [I 2022-10-28 08:53:45] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:50 [I 2022-10-28 08:53:45] ...trap/engines/engine.py.229: process: 0.56296 | load: 0.00240 [I 2022-10-28 08:53:45] ...trap/engines/engine.py.230: loss: 0.11317 [I 2022-10-28 08:53:45] ...optimizers/trijoint.py.151: total_norm: 15.703300 [I 2022-10-28 08:53:50] ...trap/engines/engine.py.224: train: epoch 116 | batch 2010/2304 [I 2022-10-28 08:53:50] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:45 [I 2022-10-28 08:53:50] ...trap/engines/engine.py.229: process: 0.56367 | load: 0.00241 [I 2022-10-28 08:53:50] ...trap/engines/engine.py.230: loss: 0.16544 [I 2022-10-28 08:53:50] ...optimizers/trijoint.py.151: total_norm: 22.996489 [I 2022-10-28 08:53:56] ...trap/engines/engine.py.224: train: epoch 116 | batch 2020/2304 [I 2022-10-28 08:53:56] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:38 [I 2022-10-28 08:53:56] ...trap/engines/engine.py.229: process: 0.55670 | load: 0.00243 [I 2022-10-28 08:53:56] ...trap/engines/engine.py.230: loss: 0.12141 [I 2022-10-28 08:53:56] ...optimizers/trijoint.py.151: total_norm: 19.092293 [I 2022-10-28 08:54:01] ...trap/engines/engine.py.224: train: epoch 116 | batch 2030/2304 [I 2022-10-28 08:54:01] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:32 [I 2022-10-28 08:54:01] ...trap/engines/engine.py.229: process: 0.56217 | load: 0.00264 [I 2022-10-28 08:54:01] ...trap/engines/engine.py.230: loss: 0.09614 [I 2022-10-28 08:54:01] ...optimizers/trijoint.py.151: total_norm: 13.096698 [I 2022-10-28 08:54:07] ...trap/engines/engine.py.224: train: epoch 116 | batch 2040/2304 [I 2022-10-28 08:54:07] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:26 [I 2022-10-28 08:54:07] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00251 [I 2022-10-28 08:54:07] ...trap/engines/engine.py.230: loss: 0.15177 [I 2022-10-28 08:54:07] ...optimizers/trijoint.py.151: total_norm: 11.849886 [I 2022-10-28 08:54:13] ...trap/engines/engine.py.224: train: epoch 116 | batch 2050/2304 [I 2022-10-28 08:54:13] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:21 [I 2022-10-28 08:54:13] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00275 [I 2022-10-28 08:54:13] ...trap/engines/engine.py.230: loss: 0.06338 [I 2022-10-28 08:54:13] ...optimizers/trijoint.py.151: total_norm: 17.924721 [I 2022-10-28 08:54:18] ...trap/engines/engine.py.224: train: epoch 116 | batch 2060/2304 [I 2022-10-28 08:54:18] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:15 [I 2022-10-28 08:54:18] ...trap/engines/engine.py.229: process: 0.55345 | load: 0.00247 [I 2022-10-28 08:54:18] ...trap/engines/engine.py.230: loss: 0.10134 [I 2022-10-28 08:54:18] ...optimizers/trijoint.py.151: total_norm: 8.392189 [I 2022-10-28 08:54:24] ...trap/engines/engine.py.224: train: epoch 116 | batch 2070/2304 [I 2022-10-28 08:54:24] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:09 [I 2022-10-28 08:54:24] ...trap/engines/engine.py.229: process: 0.55111 | load: 0.00250 [I 2022-10-28 08:54:24] ...trap/engines/engine.py.230: loss: 0.24933 [I 2022-10-28 08:54:24] ...optimizers/trijoint.py.151: total_norm: 40.279598 [I 2022-10-28 08:54:29] ...trap/engines/engine.py.224: train: epoch 116 | batch 2080/2304 [I 2022-10-28 08:54:29] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:02:04 [I 2022-10-28 08:54:29] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00245 [I 2022-10-28 08:54:29] ...trap/engines/engine.py.230: loss: 0.07093 [I 2022-10-28 08:54:29] ...optimizers/trijoint.py.151: total_norm: 15.602025 [I 2022-10-28 08:54:35] ...trap/engines/engine.py.224: train: epoch 116 | batch 2090/2304 [I 2022-10-28 08:54:35] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:02:00 [I 2022-10-28 08:54:35] ...trap/engines/engine.py.229: process: 0.57799 | load: 0.00244 [I 2022-10-28 08:54:35] ...trap/engines/engine.py.230: loss: 0.13341 [I 2022-10-28 08:54:35] ...optimizers/trijoint.py.151: total_norm: 17.831182 [I 2022-10-28 08:54:41] ...trap/engines/engine.py.224: train: epoch 116 | batch 2100/2304 [I 2022-10-28 08:54:41] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:53 [I 2022-10-28 08:54:41] ...trap/engines/engine.py.229: process: 0.55647 | load: 0.00248 [I 2022-10-28 08:54:41] ...trap/engines/engine.py.230: loss: 0.12515 [I 2022-10-28 08:54:41] ...optimizers/trijoint.py.151: total_norm: 48.129990 [I 2022-10-28 08:54:46] ...trap/engines/engine.py.224: train: epoch 116 | batch 2110/2304 [I 2022-10-28 08:54:46] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:48 [I 2022-10-28 08:54:46] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00244 [I 2022-10-28 08:54:46] ...trap/engines/engine.py.230: loss: 0.12787 [I 2022-10-28 08:54:46] ...optimizers/trijoint.py.151: total_norm: 10.653864 [I 2022-10-28 08:54:52] ...trap/engines/engine.py.224: train: epoch 116 | batch 2120/2304 [I 2022-10-28 08:54:52] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:43 [I 2022-10-28 08:54:52] ...trap/engines/engine.py.229: process: 0.55515 | load: 0.00248 [I 2022-10-28 08:54:52] ...trap/engines/engine.py.230: loss: 0.17275 [I 2022-10-28 08:54:52] ...optimizers/trijoint.py.151: total_norm: 16.947676 [I 2022-10-28 08:54:57] ...trap/engines/engine.py.224: train: epoch 116 | batch 2130/2304 [I 2022-10-28 08:54:57] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:37 [I 2022-10-28 08:54:57] ...trap/engines/engine.py.229: process: 0.55626 | load: 0.00257 [I 2022-10-28 08:54:57] ...trap/engines/engine.py.230: loss: 0.09015 [I 2022-10-28 08:54:57] ...optimizers/trijoint.py.151: total_norm: 12.650221 [I 2022-10-28 08:55:03] ...trap/engines/engine.py.224: train: epoch 116 | batch 2140/2304 [I 2022-10-28 08:55:03] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:31 [I 2022-10-28 08:55:03] ...trap/engines/engine.py.229: process: 0.55860 | load: 0.00246 [I 2022-10-28 08:55:03] ...trap/engines/engine.py.230: loss: 0.15488 [I 2022-10-28 08:55:03] ...optimizers/trijoint.py.151: total_norm: 16.933226 [I 2022-10-28 08:55:09] ...trap/engines/engine.py.224: train: epoch 116 | batch 2150/2304 [I 2022-10-28 08:55:09] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:26 [I 2022-10-28 08:55:09] ...trap/engines/engine.py.229: process: 0.55765 | load: 0.00244 [I 2022-10-28 08:55:09] ...trap/engines/engine.py.230: loss: 0.09457 [I 2022-10-28 08:55:09] ...optimizers/trijoint.py.151: total_norm: 9.313704 [I 2022-10-28 08:55:14] ...trap/engines/engine.py.224: train: epoch 116 | batch 2160/2304 [I 2022-10-28 08:55:14] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:19 [I 2022-10-28 08:55:14] ...trap/engines/engine.py.229: process: 0.54736 | load: 0.00245 [I 2022-10-28 08:55:14] ...trap/engines/engine.py.230: loss: 0.10115 [I 2022-10-28 08:55:14] ...optimizers/trijoint.py.151: total_norm: 14.646613 [I 2022-10-28 08:55:20] ...trap/engines/engine.py.224: train: epoch 116 | batch 2170/2304 [I 2022-10-28 08:55:20] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:14 [I 2022-10-28 08:55:20] ...trap/engines/engine.py.229: process: 0.55127 | load: 0.00245 [I 2022-10-28 08:55:20] ...trap/engines/engine.py.230: loss: 0.15197 [I 2022-10-28 08:55:20] ...optimizers/trijoint.py.151: total_norm: 22.704210 [I 2022-10-28 08:55:25] ...trap/engines/engine.py.224: train: epoch 116 | batch 2180/2304 [I 2022-10-28 08:55:25] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:09 [I 2022-10-28 08:55:25] ...trap/engines/engine.py.229: process: 0.55086 | load: 0.00241 [I 2022-10-28 08:55:25] ...trap/engines/engine.py.230: loss: 0.11626 [I 2022-10-28 08:55:25] ...optimizers/trijoint.py.151: total_norm: 15.423848 [I 2022-10-28 08:55:49] ...trap/engines/engine.py.224: train: epoch 116 | batch 2190/2304 [I 2022-10-28 08:55:49] ...trap/engines/engine.py.228: elapsed: 0:20:52 | left: 0:01:59 [I 2022-10-28 08:55:49] ...trap/engines/engine.py.229: process: 0.57012 | load: 0.00259 [I 2022-10-28 08:55:49] ...trap/engines/engine.py.230: loss: 0.10222 [I 2022-10-28 08:55:49] ...optimizers/trijoint.py.151: total_norm: 33.898708 [I 2022-10-28 08:55:55] ...trap/engines/engine.py.224: train: epoch 116 | batch 2200/2304 [I 2022-10-28 08:55:55] ...trap/engines/engine.py.228: elapsed: 0:20:58 | left: 0:01:03 [I 2022-10-28 08:55:55] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00262 [I 2022-10-28 08:55:55] ...trap/engines/engine.py.230: loss: 0.09018 [I 2022-10-28 08:55:55] ...optimizers/trijoint.py.151: total_norm: 13.613675 [I 2022-10-28 08:56:00] ...trap/engines/engine.py.224: train: epoch 116 | batch 2210/2304 [I 2022-10-28 08:56:00] ...trap/engines/engine.py.228: elapsed: 0:21:03 | left: 0:00:53 [I 2022-10-28 08:56:00] ...trap/engines/engine.py.229: process: 0.57159 | load: 0.00241 [I 2022-10-28 08:56:00] ...trap/engines/engine.py.230: loss: 0.12063 [I 2022-10-28 08:56:00] ...optimizers/trijoint.py.151: total_norm: 21.099833 [I 2022-10-28 08:56:06] ...trap/engines/engine.py.224: train: epoch 116 | batch 2220/2304 [I 2022-10-28 08:56:06] ...trap/engines/engine.py.228: elapsed: 0:21:09 | left: 0:00:47 [I 2022-10-28 08:56:06] ...trap/engines/engine.py.229: process: 0.54897 | load: 0.00253 [I 2022-10-28 08:56:06] ...trap/engines/engine.py.230: loss: 0.10924 [I 2022-10-28 08:56:06] ...optimizers/trijoint.py.151: total_norm: 12.467370 [I 2022-10-28 08:56:11] ...trap/engines/engine.py.224: train: epoch 116 | batch 2230/2304 [I 2022-10-28 08:56:11] ...trap/engines/engine.py.228: elapsed: 0:21:14 | left: 0:00:41 [I 2022-10-28 08:56:11] ...trap/engines/engine.py.229: process: 0.56660 | load: 0.00254 [I 2022-10-28 08:56:11] ...trap/engines/engine.py.230: loss: 0.12664 [I 2022-10-28 08:56:11] ...optimizers/trijoint.py.151: total_norm: 32.480034 [I 2022-10-28 08:56:17] ...trap/engines/engine.py.224: train: epoch 116 | batch 2240/2304 [I 2022-10-28 08:56:17] ...trap/engines/engine.py.228: elapsed: 0:21:20 | left: 0:00:35 [I 2022-10-28 08:56:17] ...trap/engines/engine.py.229: process: 0.57700 | load: 0.00261 [I 2022-10-28 08:56:17] ...trap/engines/engine.py.230: loss: 0.09845 [I 2022-10-28 08:56:17] ...optimizers/trijoint.py.151: total_norm: 15.919118 [I 2022-10-28 08:56:23] ...trap/engines/engine.py.224: train: epoch 116 | batch 2250/2304 [I 2022-10-28 08:56:23] ...trap/engines/engine.py.228: elapsed: 0:21:26 | left: 0:00:30 [I 2022-10-28 08:56:23] ...trap/engines/engine.py.229: process: 0.56174 | load: 0.00260 [I 2022-10-28 08:56:23] ...trap/engines/engine.py.230: loss: 0.11824 [I 2022-10-28 08:56:23] ...optimizers/trijoint.py.151: total_norm: 16.294449 [I 2022-10-28 08:56:28] ...trap/engines/engine.py.224: train: epoch 116 | batch 2260/2304 [I 2022-10-28 08:56:28] ...trap/engines/engine.py.228: elapsed: 0:21:31 | left: 0:00:24 [I 2022-10-28 08:56:28] ...trap/engines/engine.py.229: process: 0.55983 | load: 0.00256 [I 2022-10-28 08:56:28] ...trap/engines/engine.py.230: loss: 0.10805 [I 2022-10-28 08:56:28] ...optimizers/trijoint.py.151: total_norm: 17.933910 [I 2022-10-28 08:56:34] ...trap/engines/engine.py.224: train: epoch 116 | batch 2270/2304 [I 2022-10-28 08:56:34] ...trap/engines/engine.py.228: elapsed: 0:21:37 | left: 0:00:19 [I 2022-10-28 08:56:34] ...trap/engines/engine.py.229: process: 0.55892 | load: 0.00257 [I 2022-10-28 08:56:34] ...trap/engines/engine.py.230: loss: 0.09081 [I 2022-10-28 08:56:34] ...optimizers/trijoint.py.151: total_norm: 14.062970 [I 2022-10-28 08:56:39] ...trap/engines/engine.py.224: train: epoch 116 | batch 2280/2304 [I 2022-10-28 08:56:39] ...trap/engines/engine.py.228: elapsed: 0:21:43 | left: 0:00:13 [I 2022-10-28 08:56:39] ...trap/engines/engine.py.229: process: 0.55942 | load: 0.00259 [I 2022-10-28 08:56:39] ...trap/engines/engine.py.230: loss: 0.10093 [I 2022-10-28 08:56:39] ...optimizers/trijoint.py.151: total_norm: 30.279509 [I 2022-10-28 08:56:45] ...trap/engines/engine.py.224: train: epoch 116 | batch 2290/2304 [I 2022-10-28 08:56:45] ...trap/engines/engine.py.228: elapsed: 0:21:48 | left: 0:00:07 [I 2022-10-28 08:56:45] ...trap/engines/engine.py.229: process: 0.58282 | load: 0.00275 [I 2022-10-28 08:56:45] ...trap/engines/engine.py.230: loss: 0.08038 [I 2022-10-28 08:56:45] ...optimizers/trijoint.py.151: total_norm: 12.166875 [I 2022-10-28 08:56:51] ...trap/engines/engine.py.224: train: epoch 116 | batch 2300/2304 [I 2022-10-28 08:56:51] ...trap/engines/engine.py.228: elapsed: 0:21:54 | left: 0:00:02 [I 2022-10-28 08:56:51] ...trap/engines/engine.py.229: process: 0.55510 | load: 0.00014 [I 2022-10-28 08:56:51] ...trap/engines/engine.py.230: loss: 0.09968 [I 2022-10-28 08:56:51] ...optimizers/trijoint.py.151: total_norm: 21.891005 [I 2022-10-28 08:56:53] ...trap/engines/engine.py.224: train: epoch 116 | batch 2304/2304 [I 2022-10-28 08:56:53] ...trap/engines/engine.py.228: elapsed: 0:21:56 | left: 0:00:00 [I 2022-10-28 08:56:53] ...trap/engines/engine.py.229: process: 0.54435 | load: 0.00018 [I 2022-10-28 08:56:53] ...trap/engines/engine.py.230: loss: 0.11050 [I 2022-10-28 08:56:53] ...optimizers/trijoint.py.151: total_norm: 15.011637 [S 2022-10-28 08:56:54] ...trap/engines/engine.py.129: train_epoch.epoch: 116 [S 2022-10-28 08:56:54] ...trap/engines/engine.py.129: train_epoch.loss: 0.11909152133927686 [I 2022-10-28 08:56:54] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 08:56:56] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 116 [I 2022-10-28 08:56:56] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 08:56:58] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 08:56:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 08:56:58] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 08:57:02] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 08:57:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/510 [I 2022-10-28 08:57:02] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:53:44 [I 2022-10-28 08:57:02] ...trap/engines/engine.py.326: process: 6.32325 | load: 3.95040 [I 2022-10-28 08:57:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/510 [I 2022-10-28 08:57:06] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:07 [I 2022-10-28 08:57:06] ...trap/engines/engine.py.326: process: 0.33734 | load: 0.00016 [I 2022-10-28 08:57:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/510 [I 2022-10-28 08:57:09] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:17 [I 2022-10-28 08:57:09] ...trap/engines/engine.py.326: process: 0.34003 | load: 0.00016 [I 2022-10-28 08:57:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/510 [I 2022-10-28 08:57:12] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:43 [I 2022-10-28 08:57:12] ...trap/engines/engine.py.326: process: 0.33242 | load: 0.00015 [I 2022-10-28 08:57:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/510 [I 2022-10-28 08:57:16] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:36 [I 2022-10-28 08:57:16] ...trap/engines/engine.py.326: process: 0.32869 | load: 0.00015 [I 2022-10-28 08:57:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/510 [I 2022-10-28 08:57:19] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:33 [I 2022-10-28 08:57:19] ...trap/engines/engine.py.326: process: 0.32091 | load: 0.00017 [I 2022-10-28 08:57:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/510 [I 2022-10-28 08:57:22] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-28 08:57:22] ...trap/engines/engine.py.326: process: 0.32753 | load: 0.00017 [I 2022-10-28 08:57:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/510 [I 2022-10-28 08:57:26] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:27 [I 2022-10-28 08:57:26] ...trap/engines/engine.py.326: process: 0.33348 | load: 0.00016 [I 2022-10-28 08:57:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/510 [I 2022-10-28 08:57:29] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:23 [I 2022-10-28 08:57:29] ...trap/engines/engine.py.326: process: 0.33465 | load: 0.00016 [I 2022-10-28 08:57:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/510 [I 2022-10-28 08:57:32] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:19 [I 2022-10-28 08:57:32] ...trap/engines/engine.py.326: process: 0.33762 | load: 0.00015 [I 2022-10-28 08:57:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/510 [I 2022-10-28 08:57:36] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-28 08:57:36] ...trap/engines/engine.py.326: process: 0.29668 | load: 0.00020 [I 2022-10-28 08:57:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/510 [I 2022-10-28 08:57:39] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:02:05 [I 2022-10-28 08:57:39] ...trap/engines/engine.py.326: process: 0.37588 | load: 0.00014 [I 2022-10-28 08:57:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/510 [I 2022-10-28 08:57:42] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:53 [I 2022-10-28 08:57:42] ...trap/engines/engine.py.326: process: 0.28843 | load: 0.00015 [I 2022-10-28 08:57:44] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/510 [I 2022-10-28 08:57:44] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:49 [I 2022-10-28 08:57:44] ...trap/engines/engine.py.326: process: 0.28483 | load: 0.00018 [I 2022-10-28 08:57:47] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/510 [I 2022-10-28 08:57:47] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:46 [I 2022-10-28 08:57:47] ...trap/engines/engine.py.326: process: 0.28555 | load: 0.00016 [I 2022-10-28 08:57:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/510 [I 2022-10-28 08:57:50] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:43 [I 2022-10-28 08:57:50] ...trap/engines/engine.py.326: process: 0.28644 | load: 0.00014 [I 2022-10-28 08:57:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/510 [I 2022-10-28 08:57:53] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:40 [I 2022-10-28 08:57:53] ...trap/engines/engine.py.326: process: 0.28875 | load: 0.00015 [I 2022-10-28 08:57:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/510 [I 2022-10-28 08:57:56] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:38 [I 2022-10-28 08:57:56] ...trap/engines/engine.py.326: process: 0.29290 | load: 0.00014 [I 2022-10-28 08:57:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/510 [I 2022-10-28 08:57:59] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-28 08:57:59] ...trap/engines/engine.py.326: process: 0.28429 | load: 0.00014 [I 2022-10-28 08:58:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/510 [I 2022-10-28 08:58:02] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-28 08:58:02] ...trap/engines/engine.py.326: process: 0.29052 | load: 0.00015 [I 2022-10-28 08:58:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/510 [I 2022-10-28 08:58:05] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:29 [I 2022-10-28 08:58:05] ...trap/engines/engine.py.326: process: 0.28999 | load: 0.00024 [I 2022-10-28 08:58:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/510 [I 2022-10-28 08:58:07] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 08:58:07] ...trap/engines/engine.py.326: process: 0.28768 | load: 0.00015 [I 2022-10-28 08:58:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/510 [I 2022-10-28 08:58:10] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:23 [I 2022-10-28 08:58:10] ...trap/engines/engine.py.326: process: 0.28591 | load: 0.00015 [I 2022-10-28 08:58:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/510 [I 2022-10-28 08:58:13] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:20 [I 2022-10-28 08:58:13] ...trap/engines/engine.py.326: process: 0.28314 | load: 0.00014 [I 2022-10-28 08:58:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/510 [I 2022-10-28 08:58:16] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:18 [I 2022-10-28 08:58:16] ...trap/engines/engine.py.326: process: 0.28589 | load: 0.00016 [I 2022-10-28 08:58:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/510 [I 2022-10-28 08:58:19] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:15 [I 2022-10-28 08:58:19] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00022 [I 2022-10-28 08:58:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/510 [I 2022-10-28 08:58:22] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:12 [I 2022-10-28 08:58:22] ...trap/engines/engine.py.326: process: 0.29056 | load: 0.00020 [I 2022-10-28 08:58:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/510 [I 2022-10-28 08:58:25] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:09 [I 2022-10-28 08:58:25] ...trap/engines/engine.py.326: process: 0.29207 | load: 0.00015 [I 2022-10-28 08:58:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/510 [I 2022-10-28 08:58:28] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:05 [I 2022-10-28 08:58:28] ...trap/engines/engine.py.326: process: 0.28375 | load: 0.00021 [I 2022-10-28 08:58:31] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/510 [I 2022-10-28 08:58:31] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 08:58:31] ...trap/engines/engine.py.326: process: 0.28659 | load: 0.00015 [I 2022-10-28 08:58:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/510 [I 2022-10-28 08:58:33] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-28 08:58:33] ...trap/engines/engine.py.326: process: 0.28883 | load: 0.00021 [I 2022-10-28 08:58:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/510 [I 2022-10-28 08:58:36] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:57 [I 2022-10-28 08:58:36] ...trap/engines/engine.py.326: process: 0.28789 | load: 0.00021 [I 2022-10-28 08:58:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/510 [I 2022-10-28 08:58:39] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:54 [I 2022-10-28 08:58:39] ...trap/engines/engine.py.326: process: 0.28358 | load: 0.00015 [I 2022-10-28 08:58:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/510 [I 2022-10-28 08:58:42] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:51 [I 2022-10-28 08:58:42] ...trap/engines/engine.py.326: process: 0.28736 | load: 0.00016 [I 2022-10-28 08:58:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/510 [I 2022-10-28 08:58:45] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:48 [I 2022-10-28 08:58:45] ...trap/engines/engine.py.326: process: 0.28614 | load: 0.00015 [I 2022-10-28 08:58:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/510 [I 2022-10-28 08:58:48] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 08:58:48] ...trap/engines/engine.py.326: process: 0.28826 | load: 0.00022 [I 2022-10-28 08:58:51] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/510 [I 2022-10-28 08:58:51] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:42 [I 2022-10-28 08:58:51] ...trap/engines/engine.py.326: process: 0.28595 | load: 0.00017 [I 2022-10-28 08:58:54] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/510 [I 2022-10-28 08:58:54] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 08:58:54] ...trap/engines/engine.py.326: process: 0.29054 | load: 0.00021 [I 2022-10-28 08:58:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/510 [I 2022-10-28 08:58:56] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 08:58:56] ...trap/engines/engine.py.326: process: 0.29300 | load: 0.00021 [I 2022-10-28 08:58:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/510 [I 2022-10-28 08:58:59] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 08:58:59] ...trap/engines/engine.py.326: process: 0.28367 | load: 0.00020 [I 2022-10-28 08:59:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/510 [I 2022-10-28 08:59:02] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:32 [I 2022-10-28 08:59:02] ...trap/engines/engine.py.326: process: 0.28599 | load: 0.00022 [I 2022-10-28 08:59:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/510 [I 2022-10-28 08:59:05] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:29 [I 2022-10-28 08:59:05] ...trap/engines/engine.py.326: process: 0.29641 | load: 0.00014 [I 2022-10-28 08:59:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/510 [I 2022-10-28 08:59:08] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:26 [I 2022-10-28 08:59:08] ...trap/engines/engine.py.326: process: 0.29496 | load: 0.00015 [I 2022-10-28 08:59:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/510 [I 2022-10-28 08:59:11] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:23 [I 2022-10-28 08:59:11] ...trap/engines/engine.py.326: process: 0.28851 | load: 0.00021 [I 2022-10-28 08:59:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/510 [I 2022-10-28 08:59:14] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 08:59:14] ...trap/engines/engine.py.326: process: 0.28246 | load: 0.00021 [I 2022-10-28 08:59:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/510 [I 2022-10-28 08:59:17] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 08:59:17] ...trap/engines/engine.py.326: process: 0.28246 | load: 0.00016 [I 2022-10-28 08:59:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/510 [I 2022-10-28 08:59:20] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 08:59:20] ...trap/engines/engine.py.326: process: 0.29391 | load: 0.00016 [I 2022-10-28 08:59:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/510 [I 2022-10-28 08:59:23] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:12 [I 2022-10-28 08:59:23] ...trap/engines/engine.py.326: process: 0.36234 | load: 0.00018 [I 2022-10-28 08:59:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/510 [I 2022-10-28 08:59:26] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 08:59:26] ...trap/engines/engine.py.326: process: 0.28667 | load: 0.00017 [I 2022-10-28 08:59:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/510 [I 2022-10-28 08:59:28] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 08:59:28] ...trap/engines/engine.py.326: process: 0.29243 | load: 0.00016 [I 2022-10-28 08:59:31] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/510 [I 2022-10-28 08:59:31] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:02 [I 2022-10-28 08:59:31] ...trap/engines/engine.py.326: process: 0.29386 | load: 0.00019 [I 2022-10-28 08:59:34] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/510 [I 2022-10-28 08:59:34] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:00 [I 2022-10-28 08:59:34] ...trap/engines/engine.py.326: process: 0.28694 | load: 0.00045 [S 2022-10-28 08:59:35] ...trap/engines/engine.py.132: eval_epoch.epoch: 116 [I 2022-10-28 08:59:35] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.741700 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907600 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933100 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.733500 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907100 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932600 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012594 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006003 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007382 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011342 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007687 [S 2022-10-28 08:59:39] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006681 [I 2022-10-28 08:59:41] ...trap/engines/engine.py.138: Saving best checkpoint for strategy eval_epoch.metric.recall_at_1_im2recipe_mean [I 2022-10-28 08:59:41] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 08:59:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 08:59:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 08:59:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 08:59:47] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 08:59:47] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 08:59:53] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 08:59:53] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 08:59:53] ...trap/engines/engine.py.470: Saving model... [I 2022-10-28 08:59:56] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 09:00:02] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 09:00:02] ...trap/engines/engine.py.163: Training model on trainset for epoch 117 [I 2022-10-28 09:00:02] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 09:00:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 0/2304 [I 2022-10-28 09:00:05] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:10 [I 2022-10-28 09:00:05] ...trap/engines/engine.py.229: process: 2.68680 | load: 2.02613 [I 2022-10-28 09:00:05] ...trap/engines/engine.py.230: loss: 0.10813 [I 2022-10-28 09:00:05] ...optimizers/trijoint.py.151: total_norm: 13.174149 [I 2022-10-28 09:00:10] ...trap/engines/engine.py.224: train: epoch 117 | batch 10/2304 [I 2022-10-28 09:00:10] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:16 [I 2022-10-28 09:00:10] ...trap/engines/engine.py.229: process: 0.56559 | load: 0.00450 [I 2022-10-28 09:00:10] ...trap/engines/engine.py.230: loss: 0.12315 [I 2022-10-28 09:00:10] ...optimizers/trijoint.py.151: total_norm: 15.436382 [I 2022-10-28 09:00:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 20/2304 [I 2022-10-28 09:00:16] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:23 [I 2022-10-28 09:00:16] ...trap/engines/engine.py.229: process: 0.56919 | load: 0.00353 [I 2022-10-28 09:00:16] ...trap/engines/engine.py.230: loss: 0.09411 [I 2022-10-28 09:00:16] ...optimizers/trijoint.py.151: total_norm: 8.878608 [I 2022-10-28 09:00:22] ...trap/engines/engine.py.224: train: epoch 117 | batch 30/2304 [I 2022-10-28 09:00:22] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:32 [I 2022-10-28 09:00:22] ...trap/engines/engine.py.229: process: 0.57292 | load: 0.00301 [I 2022-10-28 09:00:22] ...trap/engines/engine.py.230: loss: 0.09942 [I 2022-10-28 09:00:22] ...optimizers/trijoint.py.151: total_norm: 8.864337 [I 2022-10-28 09:00:27] ...trap/engines/engine.py.224: train: epoch 117 | batch 40/2304 [I 2022-10-28 09:00:27] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:06 [I 2022-10-28 09:00:27] ...trap/engines/engine.py.229: process: 0.58003 | load: 0.00291 [I 2022-10-28 09:00:27] ...trap/engines/engine.py.230: loss: 0.14711 [I 2022-10-28 09:00:27] ...optimizers/trijoint.py.151: total_norm: 13.480612 [I 2022-10-28 09:00:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 50/2304 [I 2022-10-28 09:00:33] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:35 [I 2022-10-28 09:00:33] ...trap/engines/engine.py.229: process: 0.57509 | load: 0.00284 [I 2022-10-28 09:00:33] ...trap/engines/engine.py.230: loss: 0.11466 [I 2022-10-28 09:00:33] ...optimizers/trijoint.py.151: total_norm: 13.377568 [I 2022-10-28 09:00:39] ...trap/engines/engine.py.224: train: epoch 117 | batch 60/2304 [I 2022-10-28 09:00:39] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:21:04 [I 2022-10-28 09:00:39] ...trap/engines/engine.py.229: process: 0.56760 | load: 0.00265 [I 2022-10-28 09:00:39] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-28 09:00:39] ...optimizers/trijoint.py.151: total_norm: 32.364891 [I 2022-10-28 09:00:44] ...trap/engines/engine.py.224: train: epoch 117 | batch 70/2304 [I 2022-10-28 09:00:44] ...trap/engines/engine.py.228: elapsed: 0:00:42 | left: 0:20:39 [I 2022-10-28 09:00:44] ...trap/engines/engine.py.229: process: 0.54038 | load: 0.00249 [I 2022-10-28 09:00:44] ...trap/engines/engine.py.230: loss: 0.11236 [I 2022-10-28 09:00:44] ...optimizers/trijoint.py.151: total_norm: 12.130053 [I 2022-10-28 09:00:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 80/2304 [I 2022-10-28 09:00:50] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:52 [I 2022-10-28 09:00:50] ...trap/engines/engine.py.229: process: 0.56368 | load: 0.00279 [I 2022-10-28 09:00:50] ...trap/engines/engine.py.230: loss: 0.14035 [I 2022-10-28 09:00:50] ...optimizers/trijoint.py.151: total_norm: 18.997311 [I 2022-10-28 09:00:55] ...trap/engines/engine.py.224: train: epoch 117 | batch 90/2304 [I 2022-10-28 09:00:55] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:32 [I 2022-10-28 09:00:55] ...trap/engines/engine.py.229: process: 0.54991 | load: 0.00260 [I 2022-10-28 09:00:55] ...trap/engines/engine.py.230: loss: 0.11781 [I 2022-10-28 09:00:55] ...optimizers/trijoint.py.151: total_norm: 13.506468 [I 2022-10-28 09:01:01] ...trap/engines/engine.py.224: train: epoch 117 | batch 100/2304 [I 2022-10-28 09:01:01] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:35 [I 2022-10-28 09:01:01] ...trap/engines/engine.py.229: process: 0.56811 | load: 0.00263 [I 2022-10-28 09:01:01] ...trap/engines/engine.py.230: loss: 0.14072 [I 2022-10-28 09:01:01] ...optimizers/trijoint.py.151: total_norm: 16.776857 [I 2022-10-28 09:01:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 110/2304 [I 2022-10-28 09:01:06] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:21 [I 2022-10-28 09:01:06] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00271 [I 2022-10-28 09:01:06] ...trap/engines/engine.py.230: loss: 0.11859 [I 2022-10-28 09:01:06] ...optimizers/trijoint.py.151: total_norm: 21.568048 [I 2022-10-28 09:01:12] ...trap/engines/engine.py.224: train: epoch 117 | batch 120/2304 [I 2022-10-28 09:01:12] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:25 [I 2022-10-28 09:01:12] ...trap/engines/engine.py.229: process: 0.55376 | load: 0.00250 [I 2022-10-28 09:01:12] ...trap/engines/engine.py.230: loss: 0.08972 [I 2022-10-28 09:01:12] ...optimizers/trijoint.py.151: total_norm: 14.122783 [I 2022-10-28 09:01:18] ...trap/engines/engine.py.224: train: epoch 117 | batch 130/2304 [I 2022-10-28 09:01:18] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:20 [I 2022-10-28 09:01:18] ...trap/engines/engine.py.229: process: 0.56107 | load: 0.00249 [I 2022-10-28 09:01:18] ...trap/engines/engine.py.230: loss: 0.15676 [I 2022-10-28 09:01:18] ...optimizers/trijoint.py.151: total_norm: 18.569677 [I 2022-10-28 09:01:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 140/2304 [I 2022-10-28 09:01:23] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:20:09 [I 2022-10-28 09:01:23] ...trap/engines/engine.py.229: process: 0.56796 | load: 0.00265 [I 2022-10-28 09:01:23] ...trap/engines/engine.py.230: loss: 0.11132 [I 2022-10-28 09:01:23] ...optimizers/trijoint.py.151: total_norm: 14.490783 [I 2022-10-28 09:01:29] ...trap/engines/engine.py.224: train: epoch 117 | batch 150/2304 [I 2022-10-28 09:01:29] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:10 [I 2022-10-28 09:01:29] ...trap/engines/engine.py.229: process: 0.56708 | load: 0.00257 [I 2022-10-28 09:01:29] ...trap/engines/engine.py.230: loss: 0.14924 [I 2022-10-28 09:01:29] ...optimizers/trijoint.py.151: total_norm: 12.247490 [I 2022-10-28 09:01:35] ...trap/engines/engine.py.224: train: epoch 117 | batch 160/2304 [I 2022-10-28 09:01:35] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:10 [I 2022-10-28 09:01:35] ...trap/engines/engine.py.229: process: 0.54194 | load: 0.00252 [I 2022-10-28 09:01:35] ...trap/engines/engine.py.230: loss: 0.09673 [I 2022-10-28 09:01:35] ...optimizers/trijoint.py.151: total_norm: 11.572896 [I 2022-10-28 09:01:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 170/2304 [I 2022-10-28 09:01:40] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:56 [I 2022-10-28 09:01:40] ...trap/engines/engine.py.229: process: 0.54711 | load: 0.00253 [I 2022-10-28 09:01:40] ...trap/engines/engine.py.230: loss: 0.12684 [I 2022-10-28 09:01:40] ...optimizers/trijoint.py.151: total_norm: 13.081569 [I 2022-10-28 09:01:46] ...trap/engines/engine.py.224: train: epoch 117 | batch 180/2304 [I 2022-10-28 09:01:46] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:53 [I 2022-10-28 09:01:46] ...trap/engines/engine.py.229: process: 0.55754 | load: 0.00254 [I 2022-10-28 09:01:46] ...trap/engines/engine.py.230: loss: 0.15872 [I 2022-10-28 09:01:46] ...optimizers/trijoint.py.151: total_norm: 15.419634 [I 2022-10-28 09:01:52] ...trap/engines/engine.py.224: train: epoch 117 | batch 190/2304 [I 2022-10-28 09:01:52] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:52 [I 2022-10-28 09:01:52] ...trap/engines/engine.py.229: process: 0.56907 | load: 0.00255 [I 2022-10-28 09:01:52] ...trap/engines/engine.py.230: loss: 0.09600 [I 2022-10-28 09:01:52] ...optimizers/trijoint.py.151: total_norm: 10.828574 [I 2022-10-28 09:01:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 200/2304 [I 2022-10-28 09:01:57] ...trap/engines/engine.py.228: elapsed: 0:01:55 | left: 0:19:32 [I 2022-10-28 09:01:57] ...trap/engines/engine.py.229: process: 0.54894 | load: 0.00245 [I 2022-10-28 09:01:57] ...trap/engines/engine.py.230: loss: 0.09633 [I 2022-10-28 09:01:57] ...optimizers/trijoint.py.151: total_norm: 54.408813 [I 2022-10-28 09:02:03] ...trap/engines/engine.py.224: train: epoch 117 | batch 210/2304 [I 2022-10-28 09:02:03] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:32 [I 2022-10-28 09:02:03] ...trap/engines/engine.py.229: process: 0.56100 | load: 0.00248 [I 2022-10-28 09:02:03] ...trap/engines/engine.py.230: loss: 0.07844 [I 2022-10-28 09:02:03] ...optimizers/trijoint.py.151: total_norm: 9.667949 [I 2022-10-28 09:02:08] ...trap/engines/engine.py.224: train: epoch 117 | batch 220/2304 [I 2022-10-28 09:02:08] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:30 [I 2022-10-28 09:02:08] ...trap/engines/engine.py.229: process: 0.55705 | load: 0.00271 [I 2022-10-28 09:02:08] ...trap/engines/engine.py.230: loss: 0.11783 [I 2022-10-28 09:02:08] ...optimizers/trijoint.py.151: total_norm: 24.026125 [I 2022-10-28 09:02:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 230/2304 [I 2022-10-28 09:02:14] ...trap/engines/engine.py.228: elapsed: 0:02:12 | left: 0:19:19 [I 2022-10-28 09:02:14] ...trap/engines/engine.py.229: process: 0.55556 | load: 0.00279 [I 2022-10-28 09:02:14] ...trap/engines/engine.py.230: loss: 0.11957 [I 2022-10-28 09:02:14] ...optimizers/trijoint.py.151: total_norm: 13.191988 [I 2022-10-28 09:02:20] ...trap/engines/engine.py.224: train: epoch 117 | batch 240/2304 [I 2022-10-28 09:02:20] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:16 [I 2022-10-28 09:02:20] ...trap/engines/engine.py.229: process: 0.56069 | load: 0.00265 [I 2022-10-28 09:02:20] ...trap/engines/engine.py.230: loss: 0.16807 [I 2022-10-28 09:02:20] ...optimizers/trijoint.py.151: total_norm: 13.656911 [I 2022-10-28 09:02:25] ...trap/engines/engine.py.224: train: epoch 117 | batch 250/2304 [I 2022-10-28 09:02:25] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:26 [I 2022-10-28 09:02:25] ...trap/engines/engine.py.229: process: 0.57875 | load: 0.00278 [I 2022-10-28 09:02:25] ...trap/engines/engine.py.230: loss: 0.11749 [I 2022-10-28 09:02:25] ...optimizers/trijoint.py.151: total_norm: 10.324391 [I 2022-10-28 09:02:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 260/2304 [I 2022-10-28 09:02:31] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:05 [I 2022-10-28 09:02:31] ...trap/engines/engine.py.229: process: 0.55191 | load: 0.00277 [I 2022-10-28 09:02:31] ...trap/engines/engine.py.230: loss: 0.09073 [I 2022-10-28 09:02:31] ...optimizers/trijoint.py.151: total_norm: 14.221151 [I 2022-10-28 09:02:37] ...trap/engines/engine.py.224: train: epoch 117 | batch 270/2304 [I 2022-10-28 09:02:37] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:19:10 [I 2022-10-28 09:02:37] ...trap/engines/engine.py.229: process: 0.57278 | load: 0.00254 [I 2022-10-28 09:02:37] ...trap/engines/engine.py.230: loss: 0.11526 [I 2022-10-28 09:02:37] ...optimizers/trijoint.py.151: total_norm: 20.769449 [I 2022-10-28 09:02:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 280/2304 [I 2022-10-28 09:02:42] ...trap/engines/engine.py.228: elapsed: 0:02:40 | left: 0:18:44 [I 2022-10-28 09:02:42] ...trap/engines/engine.py.229: process: 0.54777 | load: 0.00271 [I 2022-10-28 09:02:42] ...trap/engines/engine.py.230: loss: 0.09337 [I 2022-10-28 09:02:42] ...optimizers/trijoint.py.151: total_norm: 10.539404 [I 2022-10-28 09:02:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 290/2304 [I 2022-10-28 09:02:48] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:43 [I 2022-10-28 09:02:48] ...trap/engines/engine.py.229: process: 0.56429 | load: 0.00266 [I 2022-10-28 09:02:48] ...trap/engines/engine.py.230: loss: 0.11397 [I 2022-10-28 09:02:48] ...optimizers/trijoint.py.151: total_norm: 11.193878 [I 2022-10-28 09:02:53] ...trap/engines/engine.py.224: train: epoch 117 | batch 300/2304 [I 2022-10-28 09:02:53] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:32 [I 2022-10-28 09:02:53] ...trap/engines/engine.py.229: process: 0.55061 | load: 0.00259 [I 2022-10-28 09:02:53] ...trap/engines/engine.py.230: loss: 0.16765 [I 2022-10-28 09:02:53] ...optimizers/trijoint.py.151: total_norm: 30.486237 [I 2022-10-28 09:02:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 310/2304 [I 2022-10-28 09:02:59] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:30 [I 2022-10-28 09:02:59] ...trap/engines/engine.py.229: process: 0.55040 | load: 0.00256 [I 2022-10-28 09:02:59] ...trap/engines/engine.py.230: loss: 0.12669 [I 2022-10-28 09:02:59] ...optimizers/trijoint.py.151: total_norm: 13.513245 [I 2022-10-28 09:03:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 320/2304 [I 2022-10-28 09:03:05] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:25 [I 2022-10-28 09:03:05] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00269 [I 2022-10-28 09:03:05] ...trap/engines/engine.py.230: loss: 0.15139 [I 2022-10-28 09:03:05] ...optimizers/trijoint.py.151: total_norm: 18.673391 [I 2022-10-28 09:03:10] ...trap/engines/engine.py.224: train: epoch 117 | batch 330/2304 [I 2022-10-28 09:03:10] ...trap/engines/engine.py.228: elapsed: 0:03:08 | left: 0:18:25 [I 2022-10-28 09:03:10] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00316 [I 2022-10-28 09:03:10] ...trap/engines/engine.py.230: loss: 0.08380 [I 2022-10-28 09:03:10] ...optimizers/trijoint.py.151: total_norm: 14.033117 [I 2022-10-28 09:03:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 340/2304 [I 2022-10-28 09:03:16] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:21 [I 2022-10-28 09:03:16] ...trap/engines/engine.py.229: process: 0.55560 | load: 0.00260 [I 2022-10-28 09:03:16] ...trap/engines/engine.py.230: loss: 0.19342 [I 2022-10-28 09:03:16] ...optimizers/trijoint.py.151: total_norm: 11.562018 [I 2022-10-28 09:03:21] ...trap/engines/engine.py.224: train: epoch 117 | batch 350/2304 [I 2022-10-28 09:03:21] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:03 [I 2022-10-28 09:03:21] ...trap/engines/engine.py.229: process: 0.55131 | load: 0.00247 [I 2022-10-28 09:03:21] ...trap/engines/engine.py.230: loss: 0.10288 [I 2022-10-28 09:03:21] ...optimizers/trijoint.py.151: total_norm: 16.236773 [I 2022-10-28 09:03:27] ...trap/engines/engine.py.224: train: epoch 117 | batch 360/2304 [I 2022-10-28 09:03:27] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:17:57 [I 2022-10-28 09:03:27] ...trap/engines/engine.py.229: process: 0.56132 | load: 0.00245 [I 2022-10-28 09:03:27] ...trap/engines/engine.py.230: loss: 0.13687 [I 2022-10-28 09:03:27] ...optimizers/trijoint.py.151: total_norm: 39.126507 [I 2022-10-28 09:03:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 370/2304 [I 2022-10-28 09:03:32] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:03 [I 2022-10-28 09:03:32] ...trap/engines/engine.py.229: process: 0.57140 | load: 0.00244 [I 2022-10-28 09:03:32] ...trap/engines/engine.py.230: loss: 0.14319 [I 2022-10-28 09:03:32] ...optimizers/trijoint.py.151: total_norm: 11.820146 [I 2022-10-28 09:03:38] ...trap/engines/engine.py.224: train: epoch 117 | batch 380/2304 [I 2022-10-28 09:03:38] ...trap/engines/engine.py.228: elapsed: 0:03:36 | left: 0:17:54 [I 2022-10-28 09:03:38] ...trap/engines/engine.py.229: process: 0.56447 | load: 0.00254 [I 2022-10-28 09:03:38] ...trap/engines/engine.py.230: loss: 0.12540 [I 2022-10-28 09:03:38] ...optimizers/trijoint.py.151: total_norm: 14.051565 [I 2022-10-28 09:03:44] ...trap/engines/engine.py.224: train: epoch 117 | batch 390/2304 [I 2022-10-28 09:03:44] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:50 [I 2022-10-28 09:03:44] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00246 [I 2022-10-28 09:03:44] ...trap/engines/engine.py.230: loss: 0.09715 [I 2022-10-28 09:03:44] ...optimizers/trijoint.py.151: total_norm: 9.707982 [I 2022-10-28 09:03:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 400/2304 [I 2022-10-28 09:03:49] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:44 [I 2022-10-28 09:03:49] ...trap/engines/engine.py.229: process: 0.56364 | load: 0.00245 [I 2022-10-28 09:03:49] ...trap/engines/engine.py.230: loss: 0.11098 [I 2022-10-28 09:03:49] ...optimizers/trijoint.py.151: total_norm: 11.446407 [I 2022-10-28 09:03:55] ...trap/engines/engine.py.224: train: epoch 117 | batch 410/2304 [I 2022-10-28 09:03:55] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:42 [I 2022-10-28 09:03:55] ...trap/engines/engine.py.229: process: 0.55768 | load: 0.00243 [I 2022-10-28 09:03:55] ...trap/engines/engine.py.230: loss: 0.12205 [I 2022-10-28 09:03:55] ...optimizers/trijoint.py.151: total_norm: 15.985784 [I 2022-10-28 09:04:00] ...trap/engines/engine.py.224: train: epoch 117 | batch 420/2304 [I 2022-10-28 09:04:00] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:37 [I 2022-10-28 09:04:00] ...trap/engines/engine.py.229: process: 0.56468 | load: 0.00247 [I 2022-10-28 09:04:00] ...trap/engines/engine.py.230: loss: 0.12579 [I 2022-10-28 09:04:00] ...optimizers/trijoint.py.151: total_norm: 13.099995 [I 2022-10-28 09:04:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 430/2304 [I 2022-10-28 09:04:06] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:24 [I 2022-10-28 09:04:06] ...trap/engines/engine.py.229: process: 0.55701 | load: 0.00245 [I 2022-10-28 09:04:06] ...trap/engines/engine.py.230: loss: 0.10473 [I 2022-10-28 09:04:06] ...optimizers/trijoint.py.151: total_norm: 8.890641 [I 2022-10-28 09:04:12] ...trap/engines/engine.py.224: train: epoch 117 | batch 440/2304 [I 2022-10-28 09:04:12] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:25 [I 2022-10-28 09:04:12] ...trap/engines/engine.py.229: process: 0.55090 | load: 0.00244 [I 2022-10-28 09:04:12] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-28 09:04:12] ...optimizers/trijoint.py.151: total_norm: 48.500786 [I 2022-10-28 09:04:17] ...trap/engines/engine.py.224: train: epoch 117 | batch 450/2304 [I 2022-10-28 09:04:17] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:13 [I 2022-10-28 09:04:17] ...trap/engines/engine.py.229: process: 0.55368 | load: 0.00252 [I 2022-10-28 09:04:17] ...trap/engines/engine.py.230: loss: 0.09017 [I 2022-10-28 09:04:17] ...optimizers/trijoint.py.151: total_norm: 8.166677 [I 2022-10-28 09:04:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 460/2304 [I 2022-10-28 09:04:23] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:08 [I 2022-10-28 09:04:23] ...trap/engines/engine.py.229: process: 0.56329 | load: 0.00251 [I 2022-10-28 09:04:23] ...trap/engines/engine.py.230: loss: 0.11521 [I 2022-10-28 09:04:23] ...optimizers/trijoint.py.151: total_norm: 14.333756 [I 2022-10-28 09:04:28] ...trap/engines/engine.py.224: train: epoch 117 | batch 470/2304 [I 2022-10-28 09:04:28] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:07 [I 2022-10-28 09:04:28] ...trap/engines/engine.py.229: process: 0.55766 | load: 0.00264 [I 2022-10-28 09:04:28] ...trap/engines/engine.py.230: loss: 0.10556 [I 2022-10-28 09:04:28] ...optimizers/trijoint.py.151: total_norm: 25.385744 [I 2022-10-28 09:04:34] ...trap/engines/engine.py.224: train: epoch 117 | batch 480/2304 [I 2022-10-28 09:04:34] ...trap/engines/engine.py.228: elapsed: 0:04:32 | left: 0:16:59 [I 2022-10-28 09:04:34] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00243 [I 2022-10-28 09:04:34] ...trap/engines/engine.py.230: loss: 0.11219 [I 2022-10-28 09:04:34] ...optimizers/trijoint.py.151: total_norm: 18.945597 [I 2022-10-28 09:04:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 490/2304 [I 2022-10-28 09:04:40] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:51 [I 2022-10-28 09:04:40] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00252 [I 2022-10-28 09:04:40] ...trap/engines/engine.py.230: loss: 0.10469 [I 2022-10-28 09:04:40] ...optimizers/trijoint.py.151: total_norm: 45.527527 [I 2022-10-28 09:04:45] ...trap/engines/engine.py.224: train: epoch 117 | batch 500/2304 [I 2022-10-28 09:04:45] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:51 [I 2022-10-28 09:04:45] ...trap/engines/engine.py.229: process: 0.56266 | load: 0.00248 [I 2022-10-28 09:04:45] ...trap/engines/engine.py.230: loss: 0.10401 [I 2022-10-28 09:04:45] ...optimizers/trijoint.py.151: total_norm: 18.265478 [I 2022-10-28 09:04:51] ...trap/engines/engine.py.224: train: epoch 117 | batch 510/2304 [I 2022-10-28 09:04:51] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:42 [I 2022-10-28 09:04:51] ...trap/engines/engine.py.229: process: 0.55474 | load: 0.00251 [I 2022-10-28 09:04:51] ...trap/engines/engine.py.230: loss: 0.10615 [I 2022-10-28 09:04:51] ...optimizers/trijoint.py.151: total_norm: 18.229887 [I 2022-10-28 09:04:56] ...trap/engines/engine.py.224: train: epoch 117 | batch 520/2304 [I 2022-10-28 09:04:56] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:30 [I 2022-10-28 09:04:56] ...trap/engines/engine.py.229: process: 0.54924 | load: 0.00244 [I 2022-10-28 09:04:56] ...trap/engines/engine.py.230: loss: 0.14859 [I 2022-10-28 09:04:56] ...optimizers/trijoint.py.151: total_norm: 14.384572 [I 2022-10-28 09:05:02] ...trap/engines/engine.py.224: train: epoch 117 | batch 530/2304 [I 2022-10-28 09:05:02] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:34 [I 2022-10-28 09:05:02] ...trap/engines/engine.py.229: process: 0.55909 | load: 0.00246 [I 2022-10-28 09:05:02] ...trap/engines/engine.py.230: loss: 0.11226 [I 2022-10-28 09:05:02] ...optimizers/trijoint.py.151: total_norm: 33.593971 [I 2022-10-28 09:05:08] ...trap/engines/engine.py.224: train: epoch 117 | batch 540/2304 [I 2022-10-28 09:05:08] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:22 [I 2022-10-28 09:05:08] ...trap/engines/engine.py.229: process: 0.55950 | load: 0.00241 [I 2022-10-28 09:05:08] ...trap/engines/engine.py.230: loss: 0.09313 [I 2022-10-28 09:05:08] ...optimizers/trijoint.py.151: total_norm: 13.040440 [I 2022-10-28 09:05:13] ...trap/engines/engine.py.224: train: epoch 117 | batch 550/2304 [I 2022-10-28 09:05:13] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:16 [I 2022-10-28 09:05:13] ...trap/engines/engine.py.229: process: 0.55431 | load: 0.00242 [I 2022-10-28 09:05:13] ...trap/engines/engine.py.230: loss: 0.13243 [I 2022-10-28 09:05:13] ...optimizers/trijoint.py.151: total_norm: 19.272234 [I 2022-10-28 09:05:19] ...trap/engines/engine.py.224: train: epoch 117 | batch 560/2304 [I 2022-10-28 09:05:19] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:06 [I 2022-10-28 09:05:19] ...trap/engines/engine.py.229: process: 0.54115 | load: 0.00242 [I 2022-10-28 09:05:19] ...trap/engines/engine.py.230: loss: 0.12209 [I 2022-10-28 09:05:19] ...optimizers/trijoint.py.151: total_norm: 17.668449 [I 2022-10-28 09:05:24] ...trap/engines/engine.py.224: train: epoch 117 | batch 570/2304 [I 2022-10-28 09:05:24] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:09 [I 2022-10-28 09:05:24] ...trap/engines/engine.py.229: process: 0.55328 | load: 0.00243 [I 2022-10-28 09:05:24] ...trap/engines/engine.py.230: loss: 0.09817 [I 2022-10-28 09:05:24] ...optimizers/trijoint.py.151: total_norm: 11.467880 [I 2022-10-28 09:05:30] ...trap/engines/engine.py.224: train: epoch 117 | batch 580/2304 [I 2022-10-28 09:05:30] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:15:59 [I 2022-10-28 09:05:30] ...trap/engines/engine.py.229: process: 0.55505 | load: 0.00241 [I 2022-10-28 09:05:30] ...trap/engines/engine.py.230: loss: 0.15347 [I 2022-10-28 09:05:30] ...optimizers/trijoint.py.151: total_norm: 29.346748 [I 2022-10-28 09:05:36] ...trap/engines/engine.py.224: train: epoch 117 | batch 590/2304 [I 2022-10-28 09:05:36] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:50 [I 2022-10-28 09:05:36] ...trap/engines/engine.py.229: process: 0.54745 | load: 0.00241 [I 2022-10-28 09:05:36] ...trap/engines/engine.py.230: loss: 0.07545 [I 2022-10-28 09:05:36] ...optimizers/trijoint.py.151: total_norm: 10.626105 [I 2022-10-28 09:05:41] ...trap/engines/engine.py.224: train: epoch 117 | batch 600/2304 [I 2022-10-28 09:05:41] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:50 [I 2022-10-28 09:05:41] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00242 [I 2022-10-28 09:05:41] ...trap/engines/engine.py.230: loss: 0.11545 [I 2022-10-28 09:05:41] ...optimizers/trijoint.py.151: total_norm: 13.544793 [I 2022-10-28 09:05:47] ...trap/engines/engine.py.224: train: epoch 117 | batch 610/2304 [I 2022-10-28 09:05:47] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:44 [I 2022-10-28 09:05:47] ...trap/engines/engine.py.229: process: 0.56212 | load: 0.00252 [I 2022-10-28 09:05:47] ...trap/engines/engine.py.230: loss: 0.11754 [I 2022-10-28 09:05:47] ...optimizers/trijoint.py.151: total_norm: 19.834784 [I 2022-10-28 09:05:52] ...trap/engines/engine.py.224: train: epoch 117 | batch 620/2304 [I 2022-10-28 09:05:52] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:39 [I 2022-10-28 09:05:52] ...trap/engines/engine.py.229: process: 0.55311 | load: 0.00247 [I 2022-10-28 09:05:52] ...trap/engines/engine.py.230: loss: 0.08722 [I 2022-10-28 09:05:52] ...optimizers/trijoint.py.151: total_norm: 10.904362 [I 2022-10-28 09:05:58] ...trap/engines/engine.py.224: train: epoch 117 | batch 630/2304 [I 2022-10-28 09:05:58] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:35 [I 2022-10-28 09:05:58] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00245 [I 2022-10-28 09:05:58] ...trap/engines/engine.py.230: loss: 0.11470 [I 2022-10-28 09:05:58] ...optimizers/trijoint.py.151: total_norm: 23.371510 [I 2022-10-28 09:06:03] ...trap/engines/engine.py.224: train: epoch 117 | batch 640/2304 [I 2022-10-28 09:06:03] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:30 [I 2022-10-28 09:06:03] ...trap/engines/engine.py.229: process: 0.55937 | load: 0.00242 [I 2022-10-28 09:06:03] ...trap/engines/engine.py.230: loss: 0.10410 [I 2022-10-28 09:06:03] ...optimizers/trijoint.py.151: total_norm: 17.461611 [I 2022-10-28 09:06:09] ...trap/engines/engine.py.224: train: epoch 117 | batch 650/2304 [I 2022-10-28 09:06:09] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:19 [I 2022-10-28 09:06:09] ...trap/engines/engine.py.229: process: 0.54926 | load: 0.00247 [I 2022-10-28 09:06:09] ...trap/engines/engine.py.230: loss: 0.11212 [I 2022-10-28 09:06:09] ...optimizers/trijoint.py.151: total_norm: 9.349884 [I 2022-10-28 09:06:15] ...trap/engines/engine.py.224: train: epoch 117 | batch 660/2304 [I 2022-10-28 09:06:15] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:16 [I 2022-10-28 09:06:15] ...trap/engines/engine.py.229: process: 0.55966 | load: 0.00247 [I 2022-10-28 09:06:15] ...trap/engines/engine.py.230: loss: 0.16815 [I 2022-10-28 09:06:15] ...optimizers/trijoint.py.151: total_norm: 13.219716 [I 2022-10-28 09:06:20] ...trap/engines/engine.py.224: train: epoch 117 | batch 670/2304 [I 2022-10-28 09:06:20] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:07 [I 2022-10-28 09:06:20] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00248 [I 2022-10-28 09:06:20] ...trap/engines/engine.py.230: loss: 0.11175 [I 2022-10-28 09:06:20] ...optimizers/trijoint.py.151: total_norm: 9.162443 [I 2022-10-28 09:06:26] ...trap/engines/engine.py.224: train: epoch 117 | batch 680/2304 [I 2022-10-28 09:06:26] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:09 [I 2022-10-28 09:06:26] ...trap/engines/engine.py.229: process: 0.55412 | load: 0.00245 [I 2022-10-28 09:06:26] ...trap/engines/engine.py.230: loss: 0.14749 [I 2022-10-28 09:06:26] ...optimizers/trijoint.py.151: total_norm: 12.558414 [I 2022-10-28 09:06:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 690/2304 [I 2022-10-28 09:06:31] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:14:56 [I 2022-10-28 09:06:31] ...trap/engines/engine.py.229: process: 0.55702 | load: 0.00245 [I 2022-10-28 09:06:31] ...trap/engines/engine.py.230: loss: 0.07422 [I 2022-10-28 09:06:31] ...optimizers/trijoint.py.151: total_norm: 13.063440 [I 2022-10-28 09:06:37] ...trap/engines/engine.py.224: train: epoch 117 | batch 700/2304 [I 2022-10-28 09:06:37] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:14:48 [I 2022-10-28 09:06:37] ...trap/engines/engine.py.229: process: 0.55196 | load: 0.00245 [I 2022-10-28 09:06:37] ...trap/engines/engine.py.230: loss: 0.12974 [I 2022-10-28 09:06:37] ...optimizers/trijoint.py.151: total_norm: 10.553179 [I 2022-10-28 09:06:43] ...trap/engines/engine.py.224: train: epoch 117 | batch 710/2304 [I 2022-10-28 09:06:43] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:43 [I 2022-10-28 09:06:43] ...trap/engines/engine.py.229: process: 0.56203 | load: 0.00247 [I 2022-10-28 09:06:43] ...trap/engines/engine.py.230: loss: 0.11757 [I 2022-10-28 09:06:43] ...optimizers/trijoint.py.151: total_norm: 12.631319 [I 2022-10-28 09:06:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 720/2304 [I 2022-10-28 09:06:48] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:36 [I 2022-10-28 09:06:48] ...trap/engines/engine.py.229: process: 0.54565 | load: 0.00243 [I 2022-10-28 09:06:48] ...trap/engines/engine.py.230: loss: 0.12921 [I 2022-10-28 09:06:48] ...optimizers/trijoint.py.151: total_norm: 12.853580 [I 2022-10-28 09:06:54] ...trap/engines/engine.py.224: train: epoch 117 | batch 730/2304 [I 2022-10-28 09:06:54] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:37 [I 2022-10-28 09:06:54] ...trap/engines/engine.py.229: process: 0.55876 | load: 0.00241 [I 2022-10-28 09:06:54] ...trap/engines/engine.py.230: loss: 0.19744 [I 2022-10-28 09:06:54] ...optimizers/trijoint.py.151: total_norm: 31.970362 [I 2022-10-28 09:06:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 740/2304 [I 2022-10-28 09:06:59] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:29 [I 2022-10-28 09:06:59] ...trap/engines/engine.py.229: process: 0.55681 | load: 0.00244 [I 2022-10-28 09:06:59] ...trap/engines/engine.py.230: loss: 0.15958 [I 2022-10-28 09:06:59] ...optimizers/trijoint.py.151: total_norm: 10.459812 [I 2022-10-28 09:07:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 750/2304 [I 2022-10-28 09:07:05] ...trap/engines/engine.py.228: elapsed: 0:07:02 | left: 0:14:31 [I 2022-10-28 09:07:05] ...trap/engines/engine.py.229: process: 0.56668 | load: 0.00249 [I 2022-10-28 09:07:05] ...trap/engines/engine.py.230: loss: 0.10497 [I 2022-10-28 09:07:05] ...optimizers/trijoint.py.151: total_norm: 19.715630 [I 2022-10-28 09:07:11] ...trap/engines/engine.py.224: train: epoch 117 | batch 760/2304 [I 2022-10-28 09:07:11] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:57 [I 2022-10-28 09:07:11] ...trap/engines/engine.py.229: process: 0.56515 | load: 0.00247 [I 2022-10-28 09:07:11] ...trap/engines/engine.py.230: loss: 0.07702 [I 2022-10-28 09:07:11] ...optimizers/trijoint.py.151: total_norm: 10.948484 [I 2022-10-28 09:07:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 770/2304 [I 2022-10-28 09:07:16] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:20 [I 2022-10-28 09:07:16] ...trap/engines/engine.py.229: process: 0.56524 | load: 0.00244 [I 2022-10-28 09:07:16] ...trap/engines/engine.py.230: loss: 0.09531 [I 2022-10-28 09:07:16] ...optimizers/trijoint.py.151: total_norm: 10.617085 [I 2022-10-28 09:07:22] ...trap/engines/engine.py.224: train: epoch 117 | batch 780/2304 [I 2022-10-28 09:07:22] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:14 [I 2022-10-28 09:07:22] ...trap/engines/engine.py.229: process: 0.56727 | load: 0.00247 [I 2022-10-28 09:07:22] ...trap/engines/engine.py.230: loss: 0.07843 [I 2022-10-28 09:07:22] ...optimizers/trijoint.py.151: total_norm: 15.899300 [I 2022-10-28 09:07:27] ...trap/engines/engine.py.224: train: epoch 117 | batch 790/2304 [I 2022-10-28 09:07:27] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:05 [I 2022-10-28 09:07:27] ...trap/engines/engine.py.229: process: 0.55344 | load: 0.00243 [I 2022-10-28 09:07:27] ...trap/engines/engine.py.230: loss: 0.12475 [I 2022-10-28 09:07:27] ...optimizers/trijoint.py.151: total_norm: 13.444844 [I 2022-10-28 09:07:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 800/2304 [I 2022-10-28 09:07:33] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:13:58 [I 2022-10-28 09:07:33] ...trap/engines/engine.py.229: process: 0.56158 | load: 0.00264 [I 2022-10-28 09:07:33] ...trap/engines/engine.py.230: loss: 0.15825 [I 2022-10-28 09:07:33] ...optimizers/trijoint.py.151: total_norm: 38.281784 [I 2022-10-28 09:07:39] ...trap/engines/engine.py.224: train: epoch 117 | batch 810/2304 [I 2022-10-28 09:07:39] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:54 [I 2022-10-28 09:07:39] ...trap/engines/engine.py.229: process: 0.55808 | load: 0.00244 [I 2022-10-28 09:07:39] ...trap/engines/engine.py.230: loss: 0.07056 [I 2022-10-28 09:07:39] ...optimizers/trijoint.py.151: total_norm: 12.522374 [I 2022-10-28 09:07:44] ...trap/engines/engine.py.224: train: epoch 117 | batch 820/2304 [I 2022-10-28 09:07:44] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:49 [I 2022-10-28 09:07:44] ...trap/engines/engine.py.229: process: 0.54825 | load: 0.00249 [I 2022-10-28 09:07:44] ...trap/engines/engine.py.230: loss: 0.12716 [I 2022-10-28 09:07:44] ...optimizers/trijoint.py.151: total_norm: 17.301186 [I 2022-10-28 09:07:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 830/2304 [I 2022-10-28 09:07:50] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:50 [I 2022-10-28 09:07:50] ...trap/engines/engine.py.229: process: 0.57137 | load: 0.00248 [I 2022-10-28 09:07:50] ...trap/engines/engine.py.230: loss: 0.10764 [I 2022-10-28 09:07:50] ...optimizers/trijoint.py.151: total_norm: 19.824440 [I 2022-10-28 09:07:56] ...trap/engines/engine.py.224: train: epoch 117 | batch 840/2304 [I 2022-10-28 09:07:56] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:45 [I 2022-10-28 09:07:56] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00247 [I 2022-10-28 09:07:56] ...trap/engines/engine.py.230: loss: 0.12387 [I 2022-10-28 09:07:56] ...optimizers/trijoint.py.151: total_norm: 12.766232 [I 2022-10-28 09:08:01] ...trap/engines/engine.py.224: train: epoch 117 | batch 850/2304 [I 2022-10-28 09:08:01] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:31 [I 2022-10-28 09:08:01] ...trap/engines/engine.py.229: process: 0.56028 | load: 0.00244 [I 2022-10-28 09:08:01] ...trap/engines/engine.py.230: loss: 0.11112 [I 2022-10-28 09:08:01] ...optimizers/trijoint.py.151: total_norm: 15.220695 [I 2022-10-28 09:08:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 860/2304 [I 2022-10-28 09:08:07] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:27 [I 2022-10-28 09:08:07] ...trap/engines/engine.py.229: process: 0.56273 | load: 0.00246 [I 2022-10-28 09:08:07] ...trap/engines/engine.py.230: loss: 0.13565 [I 2022-10-28 09:08:07] ...optimizers/trijoint.py.151: total_norm: 18.616798 [I 2022-10-28 09:08:12] ...trap/engines/engine.py.224: train: epoch 117 | batch 870/2304 [I 2022-10-28 09:08:12] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:23 [I 2022-10-28 09:08:12] ...trap/engines/engine.py.229: process: 0.56370 | load: 0.00272 [I 2022-10-28 09:08:12] ...trap/engines/engine.py.230: loss: 0.14763 [I 2022-10-28 09:08:12] ...optimizers/trijoint.py.151: total_norm: 15.209279 [I 2022-10-28 09:08:18] ...trap/engines/engine.py.224: train: epoch 117 | batch 880/2304 [I 2022-10-28 09:08:18] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:16 [I 2022-10-28 09:08:18] ...trap/engines/engine.py.229: process: 0.55053 | load: 0.00243 [I 2022-10-28 09:08:18] ...trap/engines/engine.py.230: loss: 0.13883 [I 2022-10-28 09:08:18] ...optimizers/trijoint.py.151: total_norm: 20.692867 [I 2022-10-28 09:08:24] ...trap/engines/engine.py.224: train: epoch 117 | batch 890/2304 [I 2022-10-28 09:08:24] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:09 [I 2022-10-28 09:08:24] ...trap/engines/engine.py.229: process: 0.56284 | load: 0.00242 [I 2022-10-28 09:08:24] ...trap/engines/engine.py.230: loss: 0.13618 [I 2022-10-28 09:08:24] ...optimizers/trijoint.py.151: total_norm: 10.488894 [I 2022-10-28 09:08:29] ...trap/engines/engine.py.224: train: epoch 117 | batch 900/2304 [I 2022-10-28 09:08:29] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:19 [I 2022-10-28 09:08:29] ...trap/engines/engine.py.229: process: 0.56120 | load: 0.00243 [I 2022-10-28 09:08:29] ...trap/engines/engine.py.230: loss: 0.15180 [I 2022-10-28 09:08:29] ...optimizers/trijoint.py.151: total_norm: 12.198120 [I 2022-10-28 09:08:35] ...trap/engines/engine.py.224: train: epoch 117 | batch 910/2304 [I 2022-10-28 09:08:35] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:13:01 [I 2022-10-28 09:08:35] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00248 [I 2022-10-28 09:08:35] ...trap/engines/engine.py.230: loss: 0.10494 [I 2022-10-28 09:08:35] ...optimizers/trijoint.py.151: total_norm: 39.581226 [I 2022-10-28 09:08:41] ...trap/engines/engine.py.224: train: epoch 117 | batch 920/2304 [I 2022-10-28 09:08:41] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:58 [I 2022-10-28 09:08:41] ...trap/engines/engine.py.229: process: 0.56543 | load: 0.00245 [I 2022-10-28 09:08:41] ...trap/engines/engine.py.230: loss: 0.19205 [I 2022-10-28 09:08:41] ...optimizers/trijoint.py.151: total_norm: 15.994761 [I 2022-10-28 09:08:46] ...trap/engines/engine.py.224: train: epoch 117 | batch 930/2304 [I 2022-10-28 09:08:46] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:12:48 [I 2022-10-28 09:08:46] ...trap/engines/engine.py.229: process: 0.56059 | load: 0.00245 [I 2022-10-28 09:08:46] ...trap/engines/engine.py.230: loss: 0.11603 [I 2022-10-28 09:08:46] ...optimizers/trijoint.py.151: total_norm: 11.424244 [I 2022-10-28 09:08:52] ...trap/engines/engine.py.224: train: epoch 117 | batch 940/2304 [I 2022-10-28 09:08:52] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:44 [I 2022-10-28 09:08:52] ...trap/engines/engine.py.229: process: 0.56610 | load: 0.00255 [I 2022-10-28 09:08:52] ...trap/engines/engine.py.230: loss: 0.13002 [I 2022-10-28 09:08:52] ...optimizers/trijoint.py.151: total_norm: 13.620652 [I 2022-10-28 09:08:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 950/2304 [I 2022-10-28 09:08:57] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:38 [I 2022-10-28 09:08:57] ...trap/engines/engine.py.229: process: 0.55402 | load: 0.00279 [I 2022-10-28 09:08:57] ...trap/engines/engine.py.230: loss: 0.14092 [I 2022-10-28 09:08:57] ...optimizers/trijoint.py.151: total_norm: 22.921310 [I 2022-10-28 09:09:03] ...trap/engines/engine.py.224: train: epoch 117 | batch 960/2304 [I 2022-10-28 09:09:03] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:32 [I 2022-10-28 09:09:03] ...trap/engines/engine.py.229: process: 0.55906 | load: 0.00243 [I 2022-10-28 09:09:03] ...trap/engines/engine.py.230: loss: 0.12659 [I 2022-10-28 09:09:03] ...optimizers/trijoint.py.151: total_norm: 29.087242 [I 2022-10-28 09:09:09] ...trap/engines/engine.py.224: train: epoch 117 | batch 970/2304 [I 2022-10-28 09:09:09] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:24 [I 2022-10-28 09:09:09] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00246 [I 2022-10-28 09:09:09] ...trap/engines/engine.py.230: loss: 0.11446 [I 2022-10-28 09:09:09] ...optimizers/trijoint.py.151: total_norm: 25.310675 [I 2022-10-28 09:09:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 980/2304 [I 2022-10-28 09:09:14] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:12:24 [I 2022-10-28 09:09:14] ...trap/engines/engine.py.229: process: 0.56760 | load: 0.00242 [I 2022-10-28 09:09:14] ...trap/engines/engine.py.230: loss: 0.10647 [I 2022-10-28 09:09:14] ...optimizers/trijoint.py.151: total_norm: 13.737185 [I 2022-10-28 09:09:20] ...trap/engines/engine.py.224: train: epoch 117 | batch 990/2304 [I 2022-10-28 09:09:20] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:17 [I 2022-10-28 09:09:20] ...trap/engines/engine.py.229: process: 0.56547 | load: 0.00246 [I 2022-10-28 09:09:20] ...trap/engines/engine.py.230: loss: 0.08368 [I 2022-10-28 09:09:20] ...optimizers/trijoint.py.151: total_norm: 9.135665 [I 2022-10-28 09:09:25] ...trap/engines/engine.py.224: train: epoch 117 | batch 1000/2304 [I 2022-10-28 09:09:25] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:10 [I 2022-10-28 09:09:25] ...trap/engines/engine.py.229: process: 0.56200 | load: 0.00246 [I 2022-10-28 09:09:25] ...trap/engines/engine.py.230: loss: 0.10939 [I 2022-10-28 09:09:25] ...optimizers/trijoint.py.151: total_norm: 11.492123 [I 2022-10-28 09:09:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 1010/2304 [I 2022-10-28 09:09:31] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:12:04 [I 2022-10-28 09:09:31] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00245 [I 2022-10-28 09:09:31] ...trap/engines/engine.py.230: loss: 0.15687 [I 2022-10-28 09:09:31] ...optimizers/trijoint.py.151: total_norm: 24.073694 [I 2022-10-28 09:09:37] ...trap/engines/engine.py.224: train: epoch 117 | batch 1020/2304 [I 2022-10-28 09:09:37] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:12:18 [I 2022-10-28 09:09:37] ...trap/engines/engine.py.229: process: 0.56922 | load: 0.00245 [I 2022-10-28 09:09:37] ...trap/engines/engine.py.230: loss: 0.13464 [I 2022-10-28 09:09:37] ...optimizers/trijoint.py.151: total_norm: 23.032936 [I 2022-10-28 09:09:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 1030/2304 [I 2022-10-28 09:09:42] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:11:53 [I 2022-10-28 09:09:42] ...trap/engines/engine.py.229: process: 0.56332 | load: 0.00247 [I 2022-10-28 09:09:42] ...trap/engines/engine.py.230: loss: 0.13435 [I 2022-10-28 09:09:42] ...optimizers/trijoint.py.151: total_norm: 12.486311 [I 2022-10-28 09:09:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 1040/2304 [I 2022-10-28 09:09:48] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:42 [I 2022-10-28 09:09:48] ...trap/engines/engine.py.229: process: 0.55409 | load: 0.00250 [I 2022-10-28 09:09:48] ...trap/engines/engine.py.230: loss: 0.10217 [I 2022-10-28 09:09:48] ...optimizers/trijoint.py.151: total_norm: 25.011206 [I 2022-10-28 09:09:54] ...trap/engines/engine.py.224: train: epoch 117 | batch 1050/2304 [I 2022-10-28 09:09:54] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:44 [I 2022-10-28 09:09:54] ...trap/engines/engine.py.229: process: 0.55785 | load: 0.00248 [I 2022-10-28 09:09:54] ...trap/engines/engine.py.230: loss: 0.10359 [I 2022-10-28 09:09:54] ...optimizers/trijoint.py.151: total_norm: 14.054103 [I 2022-10-28 09:09:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 1060/2304 [I 2022-10-28 09:09:59] ...trap/engines/engine.py.228: elapsed: 0:09:57 | left: 0:11:42 [I 2022-10-28 09:09:59] ...trap/engines/engine.py.229: process: 0.56142 | load: 0.00248 [I 2022-10-28 09:09:59] ...trap/engines/engine.py.230: loss: 0.07424 [I 2022-10-28 09:09:59] ...optimizers/trijoint.py.151: total_norm: 12.709851 [I 2022-10-28 09:10:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 1070/2304 [I 2022-10-28 09:10:05] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:30 [I 2022-10-28 09:10:05] ...trap/engines/engine.py.229: process: 0.56467 | load: 0.00248 [I 2022-10-28 09:10:05] ...trap/engines/engine.py.230: loss: 0.07837 [I 2022-10-28 09:10:05] ...optimizers/trijoint.py.151: total_norm: 13.433994 [I 2022-10-28 09:10:10] ...trap/engines/engine.py.224: train: epoch 117 | batch 1080/2304 [I 2022-10-28 09:10:10] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:26 [I 2022-10-28 09:10:10] ...trap/engines/engine.py.229: process: 0.55952 | load: 0.00244 [I 2022-10-28 09:10:10] ...trap/engines/engine.py.230: loss: 0.14952 [I 2022-10-28 09:10:10] ...optimizers/trijoint.py.151: total_norm: 13.511477 [I 2022-10-28 09:10:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 1090/2304 [I 2022-10-28 09:10:16] ...trap/engines/engine.py.228: elapsed: 0:10:14 | left: 0:11:19 [I 2022-10-28 09:10:16] ...trap/engines/engine.py.229: process: 0.56222 | load: 0.00244 [I 2022-10-28 09:10:16] ...trap/engines/engine.py.230: loss: 0.12520 [I 2022-10-28 09:10:16] ...optimizers/trijoint.py.151: total_norm: 17.389284 [I 2022-10-28 09:10:22] ...trap/engines/engine.py.224: train: epoch 117 | batch 1100/2304 [I 2022-10-28 09:10:22] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:15 [I 2022-10-28 09:10:22] ...trap/engines/engine.py.229: process: 0.56190 | load: 0.00247 [I 2022-10-28 09:10:22] ...trap/engines/engine.py.230: loss: 0.09655 [I 2022-10-28 09:10:22] ...optimizers/trijoint.py.151: total_norm: 12.904846 [I 2022-10-28 09:10:27] ...trap/engines/engine.py.224: train: epoch 117 | batch 1110/2304 [I 2022-10-28 09:10:27] ...trap/engines/engine.py.228: elapsed: 0:10:25 | left: 0:11:08 [I 2022-10-28 09:10:27] ...trap/engines/engine.py.229: process: 0.55649 | load: 0.00243 [I 2022-10-28 09:10:27] ...trap/engines/engine.py.230: loss: 0.10613 [I 2022-10-28 09:10:27] ...optimizers/trijoint.py.151: total_norm: 14.712475 [I 2022-10-28 09:10:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 1120/2304 [I 2022-10-28 09:10:33] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:11:03 [I 2022-10-28 09:10:33] ...trap/engines/engine.py.229: process: 0.56355 | load: 0.00242 [I 2022-10-28 09:10:33] ...trap/engines/engine.py.230: loss: 0.14437 [I 2022-10-28 09:10:33] ...optimizers/trijoint.py.151: total_norm: 57.586979 [I 2022-10-28 09:10:39] ...trap/engines/engine.py.224: train: epoch 117 | batch 1130/2304 [I 2022-10-28 09:10:39] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:11:03 [I 2022-10-28 09:10:39] ...trap/engines/engine.py.229: process: 0.56809 | load: 0.00250 [I 2022-10-28 09:10:39] ...trap/engines/engine.py.230: loss: 0.09392 [I 2022-10-28 09:10:39] ...optimizers/trijoint.py.151: total_norm: 17.768242 [I 2022-10-28 09:10:44] ...trap/engines/engine.py.224: train: epoch 117 | batch 1140/2304 [I 2022-10-28 09:10:44] ...trap/engines/engine.py.228: elapsed: 0:10:42 | left: 0:10:51 [I 2022-10-28 09:10:44] ...trap/engines/engine.py.229: process: 0.55797 | load: 0.00242 [I 2022-10-28 09:10:44] ...trap/engines/engine.py.230: loss: 0.07254 [I 2022-10-28 09:10:44] ...optimizers/trijoint.py.151: total_norm: 20.778381 [I 2022-10-28 09:10:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 1150/2304 [I 2022-10-28 09:10:50] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:43 [I 2022-10-28 09:10:50] ...trap/engines/engine.py.229: process: 0.55493 | load: 0.00243 [I 2022-10-28 09:10:50] ...trap/engines/engine.py.230: loss: 0.11172 [I 2022-10-28 09:10:50] ...optimizers/trijoint.py.151: total_norm: 11.702077 [I 2022-10-28 09:10:55] ...trap/engines/engine.py.224: train: epoch 117 | batch 1160/2304 [I 2022-10-28 09:10:55] ...trap/engines/engine.py.228: elapsed: 0:10:53 | left: 0:10:38 [I 2022-10-28 09:10:55] ...trap/engines/engine.py.229: process: 0.55091 | load: 0.00241 [I 2022-10-28 09:10:55] ...trap/engines/engine.py.230: loss: 0.08039 [I 2022-10-28 09:10:55] ...optimizers/trijoint.py.151: total_norm: 17.886972 [I 2022-10-28 09:11:01] ...trap/engines/engine.py.224: train: epoch 117 | batch 1170/2304 [I 2022-10-28 09:11:01] ...trap/engines/engine.py.228: elapsed: 0:10:59 | left: 0:10:30 [I 2022-10-28 09:11:01] ...trap/engines/engine.py.229: process: 0.55222 | load: 0.00243 [I 2022-10-28 09:11:01] ...trap/engines/engine.py.230: loss: 0.20446 [I 2022-10-28 09:11:01] ...optimizers/trijoint.py.151: total_norm: 11.909379 [I 2022-10-28 09:11:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 1180/2304 [I 2022-10-28 09:11:07] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:29 [I 2022-10-28 09:11:07] ...trap/engines/engine.py.229: process: 0.57190 | load: 0.00245 [I 2022-10-28 09:11:07] ...trap/engines/engine.py.230: loss: 0.09329 [I 2022-10-28 09:11:07] ...optimizers/trijoint.py.151: total_norm: 10.602867 [I 2022-10-28 09:11:12] ...trap/engines/engine.py.224: train: epoch 117 | batch 1190/2304 [I 2022-10-28 09:11:12] ...trap/engines/engine.py.228: elapsed: 0:11:10 | left: 0:10:20 [I 2022-10-28 09:11:12] ...trap/engines/engine.py.229: process: 0.55130 | load: 0.00239 [I 2022-10-28 09:11:12] ...trap/engines/engine.py.230: loss: 0.15894 [I 2022-10-28 09:11:12] ...optimizers/trijoint.py.151: total_norm: 14.537296 [I 2022-10-28 09:11:18] ...trap/engines/engine.py.224: train: epoch 117 | batch 1200/2304 [I 2022-10-28 09:11:18] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:16 [I 2022-10-28 09:11:18] ...trap/engines/engine.py.229: process: 0.56362 | load: 0.00242 [I 2022-10-28 09:11:18] ...trap/engines/engine.py.230: loss: 0.13028 [I 2022-10-28 09:11:18] ...optimizers/trijoint.py.151: total_norm: 30.028980 [I 2022-10-28 09:11:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 1210/2304 [I 2022-10-28 09:11:23] ...trap/engines/engine.py.228: elapsed: 0:11:21 | left: 0:10:09 [I 2022-10-28 09:11:23] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00239 [I 2022-10-28 09:11:23] ...trap/engines/engine.py.230: loss: 0.09752 [I 2022-10-28 09:11:23] ...optimizers/trijoint.py.151: total_norm: 17.002432 [I 2022-10-28 09:11:29] ...trap/engines/engine.py.224: train: epoch 117 | batch 1220/2304 [I 2022-10-28 09:11:29] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:04 [I 2022-10-28 09:11:29] ...trap/engines/engine.py.229: process: 0.56076 | load: 0.00243 [I 2022-10-28 09:11:29] ...trap/engines/engine.py.230: loss: 0.12208 [I 2022-10-28 09:11:29] ...optimizers/trijoint.py.151: total_norm: 21.479006 [I 2022-10-28 09:11:35] ...trap/engines/engine.py.224: train: epoch 117 | batch 1230/2304 [I 2022-10-28 09:11:35] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:09:55 [I 2022-10-28 09:11:35] ...trap/engines/engine.py.229: process: 0.55186 | load: 0.00251 [I 2022-10-28 09:11:35] ...trap/engines/engine.py.230: loss: 0.20747 [I 2022-10-28 09:11:35] ...optimizers/trijoint.py.151: total_norm: 26.003408 [I 2022-10-28 09:11:40] ...trap/engines/engine.py.224: train: epoch 117 | batch 1240/2304 [I 2022-10-28 09:11:40] ...trap/engines/engine.py.228: elapsed: 0:11:38 | left: 0:09:53 [I 2022-10-28 09:11:40] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00246 [I 2022-10-28 09:11:40] ...trap/engines/engine.py.230: loss: 0.13526 [I 2022-10-28 09:11:40] ...optimizers/trijoint.py.151: total_norm: 33.643188 [I 2022-10-28 09:11:46] ...trap/engines/engine.py.224: train: epoch 117 | batch 1250/2304 [I 2022-10-28 09:11:46] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:51 [I 2022-10-28 09:11:46] ...trap/engines/engine.py.229: process: 0.55732 | load: 0.00246 [I 2022-10-28 09:11:46] ...trap/engines/engine.py.230: loss: 0.09625 [I 2022-10-28 09:11:46] ...optimizers/trijoint.py.151: total_norm: 13.814082 [I 2022-10-28 09:11:51] ...trap/engines/engine.py.224: train: epoch 117 | batch 1260/2304 [I 2022-10-28 09:11:51] ...trap/engines/engine.py.228: elapsed: 0:11:49 | left: 0:09:45 [I 2022-10-28 09:11:51] ...trap/engines/engine.py.229: process: 0.56603 | load: 0.00246 [I 2022-10-28 09:11:51] ...trap/engines/engine.py.230: loss: 0.09418 [I 2022-10-28 09:11:51] ...optimizers/trijoint.py.151: total_norm: 10.771084 [I 2022-10-28 09:11:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 1270/2304 [I 2022-10-28 09:11:57] ...trap/engines/engine.py.228: elapsed: 0:11:55 | left: 0:09:36 [I 2022-10-28 09:11:57] ...trap/engines/engine.py.229: process: 0.54317 | load: 0.00243 [I 2022-10-28 09:11:57] ...trap/engines/engine.py.230: loss: 0.12861 [I 2022-10-28 09:11:57] ...optimizers/trijoint.py.151: total_norm: 7.807889 [I 2022-10-28 09:12:03] ...trap/engines/engine.py.224: train: epoch 117 | batch 1280/2304 [I 2022-10-28 09:12:03] ...trap/engines/engine.py.228: elapsed: 0:12:00 | left: 0:09:32 [I 2022-10-28 09:12:03] ...trap/engines/engine.py.229: process: 0.55715 | load: 0.00245 [I 2022-10-28 09:12:03] ...trap/engines/engine.py.230: loss: 0.08188 [I 2022-10-28 09:12:03] ...optimizers/trijoint.py.151: total_norm: 13.835811 [I 2022-10-28 09:12:08] ...trap/engines/engine.py.224: train: epoch 117 | batch 1290/2304 [I 2022-10-28 09:12:08] ...trap/engines/engine.py.228: elapsed: 0:12:06 | left: 0:09:25 [I 2022-10-28 09:12:08] ...trap/engines/engine.py.229: process: 0.56412 | load: 0.00262 [I 2022-10-28 09:12:08] ...trap/engines/engine.py.230: loss: 0.09605 [I 2022-10-28 09:12:08] ...optimizers/trijoint.py.151: total_norm: 10.011683 [I 2022-10-28 09:12:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 1300/2304 [I 2022-10-28 09:12:14] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:25 [I 2022-10-28 09:12:14] ...trap/engines/engine.py.229: process: 0.57657 | load: 0.00251 [I 2022-10-28 09:12:14] ...trap/engines/engine.py.230: loss: 0.10936 [I 2022-10-28 09:12:14] ...optimizers/trijoint.py.151: total_norm: 29.610422 [I 2022-10-28 09:12:19] ...trap/engines/engine.py.224: train: epoch 117 | batch 1310/2304 [I 2022-10-28 09:12:19] ...trap/engines/engine.py.228: elapsed: 0:12:17 | left: 0:09:16 [I 2022-10-28 09:12:19] ...trap/engines/engine.py.229: process: 0.56072 | load: 0.00246 [I 2022-10-28 09:12:19] ...trap/engines/engine.py.230: loss: 0.15447 [I 2022-10-28 09:12:19] ...optimizers/trijoint.py.151: total_norm: 24.433313 [I 2022-10-28 09:12:25] ...trap/engines/engine.py.224: train: epoch 117 | batch 1320/2304 [I 2022-10-28 09:12:25] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:08 [I 2022-10-28 09:12:25] ...trap/engines/engine.py.229: process: 0.55741 | load: 0.00243 [I 2022-10-28 09:12:25] ...trap/engines/engine.py.230: loss: 0.21153 [I 2022-10-28 09:12:25] ...optimizers/trijoint.py.151: total_norm: 12.467809 [I 2022-10-28 09:12:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 1330/2304 [I 2022-10-28 09:12:31] ...trap/engines/engine.py.228: elapsed: 0:12:28 | left: 0:09:06 [I 2022-10-28 09:12:31] ...trap/engines/engine.py.229: process: 0.56136 | load: 0.00246 [I 2022-10-28 09:12:31] ...trap/engines/engine.py.230: loss: 0.09517 [I 2022-10-28 09:12:31] ...optimizers/trijoint.py.151: total_norm: 14.994170 [I 2022-10-28 09:12:36] ...trap/engines/engine.py.224: train: epoch 117 | batch 1340/2304 [I 2022-10-28 09:12:36] ...trap/engines/engine.py.228: elapsed: 0:12:34 | left: 0:09:01 [I 2022-10-28 09:12:36] ...trap/engines/engine.py.229: process: 0.56614 | load: 0.00247 [I 2022-10-28 09:12:36] ...trap/engines/engine.py.230: loss: 0.12764 [I 2022-10-28 09:12:36] ...optimizers/trijoint.py.151: total_norm: 19.072916 [I 2022-10-28 09:12:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 1350/2304 [I 2022-10-28 09:12:42] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:08:50 [I 2022-10-28 09:12:42] ...trap/engines/engine.py.229: process: 0.55760 | load: 0.00249 [I 2022-10-28 09:12:42] ...trap/engines/engine.py.230: loss: 0.09159 [I 2022-10-28 09:12:42] ...optimizers/trijoint.py.151: total_norm: 23.331358 [I 2022-10-28 09:12:47] ...trap/engines/engine.py.224: train: epoch 117 | batch 1360/2304 [I 2022-10-28 09:12:47] ...trap/engines/engine.py.228: elapsed: 0:12:45 | left: 0:08:46 [I 2022-10-28 09:12:47] ...trap/engines/engine.py.229: process: 0.55550 | load: 0.00247 [I 2022-10-28 09:12:47] ...trap/engines/engine.py.230: loss: 0.16159 [I 2022-10-28 09:12:47] ...optimizers/trijoint.py.151: total_norm: 48.544132 [I 2022-10-28 09:12:53] ...trap/engines/engine.py.224: train: epoch 117 | batch 1370/2304 [I 2022-10-28 09:12:53] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:45 [I 2022-10-28 09:12:53] ...trap/engines/engine.py.229: process: 0.56388 | load: 0.00242 [I 2022-10-28 09:12:53] ...trap/engines/engine.py.230: loss: 0.09799 [I 2022-10-28 09:12:53] ...optimizers/trijoint.py.151: total_norm: 8.636581 [I 2022-10-28 09:12:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 1380/2304 [I 2022-10-28 09:12:59] ...trap/engines/engine.py.228: elapsed: 0:12:56 | left: 0:08:40 [I 2022-10-28 09:12:59] ...trap/engines/engine.py.229: process: 0.56435 | load: 0.00257 [I 2022-10-28 09:12:59] ...trap/engines/engine.py.230: loss: 0.11610 [I 2022-10-28 09:12:59] ...optimizers/trijoint.py.151: total_norm: 12.007335 [I 2022-10-28 09:13:04] ...trap/engines/engine.py.224: train: epoch 117 | batch 1390/2304 [I 2022-10-28 09:13:04] ...trap/engines/engine.py.228: elapsed: 0:13:02 | left: 0:08:36 [I 2022-10-28 09:13:04] ...trap/engines/engine.py.229: process: 0.58034 | load: 0.00243 [I 2022-10-28 09:13:04] ...trap/engines/engine.py.230: loss: 0.17261 [I 2022-10-28 09:13:04] ...optimizers/trijoint.py.151: total_norm: 11.958159 [I 2022-10-28 09:13:10] ...trap/engines/engine.py.224: train: epoch 117 | batch 1400/2304 [I 2022-10-28 09:13:10] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:08:25 [I 2022-10-28 09:13:10] ...trap/engines/engine.py.229: process: 0.55697 | load: 0.00250 [I 2022-10-28 09:13:10] ...trap/engines/engine.py.230: loss: 0.10365 [I 2022-10-28 09:13:10] ...optimizers/trijoint.py.151: total_norm: 31.044243 [I 2022-10-28 09:13:15] ...trap/engines/engine.py.224: train: epoch 117 | batch 1410/2304 [I 2022-10-28 09:13:15] ...trap/engines/engine.py.228: elapsed: 0:13:13 | left: 0:08:16 [I 2022-10-28 09:13:15] ...trap/engines/engine.py.229: process: 0.55389 | load: 0.00245 [I 2022-10-28 09:13:15] ...trap/engines/engine.py.230: loss: 0.19104 [I 2022-10-28 09:13:15] ...optimizers/trijoint.py.151: total_norm: 28.218683 [I 2022-10-28 09:13:21] ...trap/engines/engine.py.224: train: epoch 117 | batch 1420/2304 [I 2022-10-28 09:13:21] ...trap/engines/engine.py.228: elapsed: 0:13:19 | left: 0:08:15 [I 2022-10-28 09:13:21] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00249 [I 2022-10-28 09:13:21] ...trap/engines/engine.py.230: loss: 0.08104 [I 2022-10-28 09:13:21] ...optimizers/trijoint.py.151: total_norm: 8.816782 [I 2022-10-28 09:13:27] ...trap/engines/engine.py.224: train: epoch 117 | batch 1430/2304 [I 2022-10-28 09:13:27] ...trap/engines/engine.py.228: elapsed: 0:13:24 | left: 0:08:08 [I 2022-10-28 09:13:27] ...trap/engines/engine.py.229: process: 0.56006 | load: 0.00248 [I 2022-10-28 09:13:27] ...trap/engines/engine.py.230: loss: 0.13159 [I 2022-10-28 09:13:27] ...optimizers/trijoint.py.151: total_norm: 15.182549 [I 2022-10-28 09:13:32] ...trap/engines/engine.py.224: train: epoch 117 | batch 1440/2304 [I 2022-10-28 09:13:32] ...trap/engines/engine.py.228: elapsed: 0:13:30 | left: 0:07:58 [I 2022-10-28 09:13:32] ...trap/engines/engine.py.229: process: 0.54462 | load: 0.00247 [I 2022-10-28 09:13:32] ...trap/engines/engine.py.230: loss: 0.08625 [I 2022-10-28 09:13:32] ...optimizers/trijoint.py.151: total_norm: 14.392899 [I 2022-10-28 09:13:38] ...trap/engines/engine.py.224: train: epoch 117 | batch 1450/2304 [I 2022-10-28 09:13:38] ...trap/engines/engine.py.228: elapsed: 0:13:35 | left: 0:07:55 [I 2022-10-28 09:13:38] ...trap/engines/engine.py.229: process: 0.55527 | load: 0.00246 [I 2022-10-28 09:13:38] ...trap/engines/engine.py.230: loss: 0.07745 [I 2022-10-28 09:13:38] ...optimizers/trijoint.py.151: total_norm: 11.063575 [I 2022-10-28 09:13:43] ...trap/engines/engine.py.224: train: epoch 117 | batch 1460/2304 [I 2022-10-28 09:13:43] ...trap/engines/engine.py.228: elapsed: 0:13:41 | left: 0:07:48 [I 2022-10-28 09:13:43] ...trap/engines/engine.py.229: process: 0.56307 | load: 0.00242 [I 2022-10-28 09:13:43] ...trap/engines/engine.py.230: loss: 0.11602 [I 2022-10-28 09:13:43] ...optimizers/trijoint.py.151: total_norm: 11.767378 [I 2022-10-28 09:13:49] ...trap/engines/engine.py.224: train: epoch 117 | batch 1470/2304 [I 2022-10-28 09:13:49] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:43 [I 2022-10-28 09:13:49] ...trap/engines/engine.py.229: process: 0.54630 | load: 0.00248 [I 2022-10-28 09:13:49] ...trap/engines/engine.py.230: loss: 0.12250 [I 2022-10-28 09:13:49] ...optimizers/trijoint.py.151: total_norm: 11.099924 [I 2022-10-28 09:13:55] ...trap/engines/engine.py.224: train: epoch 117 | batch 1480/2304 [I 2022-10-28 09:13:55] ...trap/engines/engine.py.228: elapsed: 0:13:52 | left: 0:07:42 [I 2022-10-28 09:13:55] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00247 [I 2022-10-28 09:13:55] ...trap/engines/engine.py.230: loss: 0.10622 [I 2022-10-28 09:13:55] ...optimizers/trijoint.py.151: total_norm: 12.743551 [I 2022-10-28 09:14:00] ...trap/engines/engine.py.224: train: epoch 117 | batch 1490/2304 [I 2022-10-28 09:14:00] ...trap/engines/engine.py.228: elapsed: 0:13:58 | left: 0:07:33 [I 2022-10-28 09:14:00] ...trap/engines/engine.py.229: process: 0.55087 | load: 0.00247 [I 2022-10-28 09:14:00] ...trap/engines/engine.py.230: loss: 0.17060 [I 2022-10-28 09:14:00] ...optimizers/trijoint.py.151: total_norm: 38.844044 [I 2022-10-28 09:14:06] ...trap/engines/engine.py.224: train: epoch 117 | batch 1500/2304 [I 2022-10-28 09:14:06] ...trap/engines/engine.py.228: elapsed: 0:14:03 | left: 0:07:27 [I 2022-10-28 09:14:06] ...trap/engines/engine.py.229: process: 0.55122 | load: 0.00247 [I 2022-10-28 09:14:06] ...trap/engines/engine.py.230: loss: 0.12655 [I 2022-10-28 09:14:06] ...optimizers/trijoint.py.151: total_norm: 19.688816 [I 2022-10-28 09:14:11] ...trap/engines/engine.py.224: train: epoch 117 | batch 1510/2304 [I 2022-10-28 09:14:11] ...trap/engines/engine.py.228: elapsed: 0:14:09 | left: 0:07:25 [I 2022-10-28 09:14:11] ...trap/engines/engine.py.229: process: 0.56629 | load: 0.00245 [I 2022-10-28 09:14:11] ...trap/engines/engine.py.230: loss: 0.14213 [I 2022-10-28 09:14:11] ...optimizers/trijoint.py.151: total_norm: 16.232313 [I 2022-10-28 09:14:17] ...trap/engines/engine.py.224: train: epoch 117 | batch 1520/2304 [I 2022-10-28 09:14:17] ...trap/engines/engine.py.228: elapsed: 0:14:14 | left: 0:07:18 [I 2022-10-28 09:14:17] ...trap/engines/engine.py.229: process: 0.55900 | load: 0.00240 [I 2022-10-28 09:14:17] ...trap/engines/engine.py.230: loss: 0.12736 [I 2022-10-28 09:14:17] ...optimizers/trijoint.py.151: total_norm: 30.949772 [I 2022-10-28 09:14:23] ...trap/engines/engine.py.224: train: epoch 117 | batch 1530/2304 [I 2022-10-28 09:14:23] ...trap/engines/engine.py.228: elapsed: 0:14:20 | left: 0:07:11 [I 2022-10-28 09:14:23] ...trap/engines/engine.py.229: process: 0.56218 | load: 0.00243 [I 2022-10-28 09:14:23] ...trap/engines/engine.py.230: loss: 0.09222 [I 2022-10-28 09:14:23] ...optimizers/trijoint.py.151: total_norm: 14.973127 [I 2022-10-28 09:14:28] ...trap/engines/engine.py.224: train: epoch 117 | batch 1540/2304 [I 2022-10-28 09:14:28] ...trap/engines/engine.py.228: elapsed: 0:14:26 | left: 0:07:02 [I 2022-10-28 09:14:28] ...trap/engines/engine.py.229: process: 0.55126 | load: 0.00244 [I 2022-10-28 09:14:28] ...trap/engines/engine.py.230: loss: 0.09236 [I 2022-10-28 09:14:28] ...optimizers/trijoint.py.151: total_norm: 15.906012 [I 2022-10-28 09:14:34] ...trap/engines/engine.py.224: train: epoch 117 | batch 1550/2304 [I 2022-10-28 09:14:34] ...trap/engines/engine.py.228: elapsed: 0:14:31 | left: 0:06:57 [I 2022-10-28 09:14:34] ...trap/engines/engine.py.229: process: 0.55846 | load: 0.00244 [I 2022-10-28 09:14:34] ...trap/engines/engine.py.230: loss: 0.17319 [I 2022-10-28 09:14:34] ...optimizers/trijoint.py.151: total_norm: 40.146244 [I 2022-10-28 09:14:39] ...trap/engines/engine.py.224: train: epoch 117 | batch 1560/2304 [I 2022-10-28 09:14:39] ...trap/engines/engine.py.228: elapsed: 0:14:37 | left: 0:06:57 [I 2022-10-28 09:14:39] ...trap/engines/engine.py.229: process: 0.55815 | load: 0.00243 [I 2022-10-28 09:14:39] ...trap/engines/engine.py.230: loss: 0.18104 [I 2022-10-28 09:14:39] ...optimizers/trijoint.py.151: total_norm: 13.909100 [I 2022-10-28 09:14:45] ...trap/engines/engine.py.224: train: epoch 117 | batch 1570/2304 [I 2022-10-28 09:14:45] ...trap/engines/engine.py.228: elapsed: 0:14:42 | left: 0:06:49 [I 2022-10-28 09:14:45] ...trap/engines/engine.py.229: process: 0.55544 | load: 0.00246 [I 2022-10-28 09:14:45] ...trap/engines/engine.py.230: loss: 0.09376 [I 2022-10-28 09:14:45] ...optimizers/trijoint.py.151: total_norm: 10.366403 [I 2022-10-28 09:14:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 1580/2304 [I 2022-10-28 09:14:50] ...trap/engines/engine.py.228: elapsed: 0:14:48 | left: 0:06:42 [I 2022-10-28 09:14:50] ...trap/engines/engine.py.229: process: 0.55517 | load: 0.00254 [I 2022-10-28 09:14:50] ...trap/engines/engine.py.230: loss: 0.14391 [I 2022-10-28 09:14:50] ...optimizers/trijoint.py.151: total_norm: 12.979674 [I 2022-10-28 09:14:56] ...trap/engines/engine.py.224: train: epoch 117 | batch 1590/2304 [I 2022-10-28 09:14:56] ...trap/engines/engine.py.228: elapsed: 0:14:54 | left: 0:06:37 [I 2022-10-28 09:14:56] ...trap/engines/engine.py.229: process: 0.55226 | load: 0.00247 [I 2022-10-28 09:14:56] ...trap/engines/engine.py.230: loss: 0.16393 [I 2022-10-28 09:14:56] ...optimizers/trijoint.py.151: total_norm: 14.393389 [I 2022-10-28 09:15:02] ...trap/engines/engine.py.224: train: epoch 117 | batch 1600/2304 [I 2022-10-28 09:15:02] ...trap/engines/engine.py.228: elapsed: 0:14:59 | left: 0:06:34 [I 2022-10-28 09:15:02] ...trap/engines/engine.py.229: process: 0.57037 | load: 0.00249 [I 2022-10-28 09:15:02] ...trap/engines/engine.py.230: loss: 0.16302 [I 2022-10-28 09:15:02] ...optimizers/trijoint.py.151: total_norm: 27.558668 [I 2022-10-28 09:15:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 1610/2304 [I 2022-10-28 09:15:07] ...trap/engines/engine.py.228: elapsed: 0:15:05 | left: 0:06:25 [I 2022-10-28 09:15:07] ...trap/engines/engine.py.229: process: 0.54080 | load: 0.00246 [I 2022-10-28 09:15:07] ...trap/engines/engine.py.230: loss: 0.12235 [I 2022-10-28 09:15:07] ...optimizers/trijoint.py.151: total_norm: 10.040740 [I 2022-10-28 09:15:13] ...trap/engines/engine.py.224: train: epoch 117 | batch 1620/2304 [I 2022-10-28 09:15:13] ...trap/engines/engine.py.228: elapsed: 0:15:10 | left: 0:06:22 [I 2022-10-28 09:15:13] ...trap/engines/engine.py.229: process: 0.56274 | load: 0.00245 [I 2022-10-28 09:15:13] ...trap/engines/engine.py.230: loss: 0.14877 [I 2022-10-28 09:15:13] ...optimizers/trijoint.py.151: total_norm: 13.756847 [I 2022-10-28 09:15:18] ...trap/engines/engine.py.224: train: epoch 117 | batch 1630/2304 [I 2022-10-28 09:15:18] ...trap/engines/engine.py.228: elapsed: 0:15:16 | left: 0:06:18 [I 2022-10-28 09:15:18] ...trap/engines/engine.py.229: process: 0.57180 | load: 0.00242 [I 2022-10-28 09:15:18] ...trap/engines/engine.py.230: loss: 0.21168 [I 2022-10-28 09:15:18] ...optimizers/trijoint.py.151: total_norm: 13.097000 [I 2022-10-28 09:15:24] ...trap/engines/engine.py.224: train: epoch 117 | batch 1640/2304 [I 2022-10-28 09:15:24] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:09 [I 2022-10-28 09:15:24] ...trap/engines/engine.py.229: process: 0.55443 | load: 0.00245 [I 2022-10-28 09:15:24] ...trap/engines/engine.py.230: loss: 0.17403 [I 2022-10-28 09:15:24] ...optimizers/trijoint.py.151: total_norm: 17.332512 [I 2022-10-28 09:15:30] ...trap/engines/engine.py.224: train: epoch 117 | batch 1650/2304 [I 2022-10-28 09:15:30] ...trap/engines/engine.py.228: elapsed: 0:15:27 | left: 0:06:06 [I 2022-10-28 09:15:30] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00246 [I 2022-10-28 09:15:30] ...trap/engines/engine.py.230: loss: 0.08599 [I 2022-10-28 09:15:30] ...optimizers/trijoint.py.151: total_norm: 14.009480 [I 2022-10-28 09:15:35] ...trap/engines/engine.py.224: train: epoch 117 | batch 1660/2304 [I 2022-10-28 09:15:35] ...trap/engines/engine.py.228: elapsed: 0:15:33 | left: 0:06:00 [I 2022-10-28 09:15:35] ...trap/engines/engine.py.229: process: 0.56372 | load: 0.00247 [I 2022-10-28 09:15:35] ...trap/engines/engine.py.230: loss: 0.14890 [I 2022-10-28 09:15:35] ...optimizers/trijoint.py.151: total_norm: 26.440563 [I 2022-10-28 09:15:41] ...trap/engines/engine.py.224: train: epoch 117 | batch 1670/2304 [I 2022-10-28 09:15:41] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:49 [I 2022-10-28 09:15:41] ...trap/engines/engine.py.229: process: 0.54572 | load: 0.00243 [I 2022-10-28 09:15:41] ...trap/engines/engine.py.230: loss: 0.11551 [I 2022-10-28 09:15:41] ...optimizers/trijoint.py.151: total_norm: 18.570707 [I 2022-10-28 09:15:46] ...trap/engines/engine.py.224: train: epoch 117 | batch 1680/2304 [I 2022-10-28 09:15:46] ...trap/engines/engine.py.228: elapsed: 0:15:44 | left: 0:05:50 [I 2022-10-28 09:15:46] ...trap/engines/engine.py.229: process: 0.56421 | load: 0.00243 [I 2022-10-28 09:15:46] ...trap/engines/engine.py.230: loss: 0.11981 [I 2022-10-28 09:15:46] ...optimizers/trijoint.py.151: total_norm: 19.151039 [I 2022-10-28 09:15:52] ...trap/engines/engine.py.224: train: epoch 117 | batch 1690/2304 [I 2022-10-28 09:15:52] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:42 [I 2022-10-28 09:15:52] ...trap/engines/engine.py.229: process: 0.56046 | load: 0.00243 [I 2022-10-28 09:15:52] ...trap/engines/engine.py.230: loss: 0.12652 [I 2022-10-28 09:15:52] ...optimizers/trijoint.py.151: total_norm: 51.182735 [I 2022-10-28 09:15:57] ...trap/engines/engine.py.224: train: epoch 117 | batch 1700/2304 [I 2022-10-28 09:15:57] ...trap/engines/engine.py.228: elapsed: 0:15:55 | left: 0:05:35 [I 2022-10-28 09:15:57] ...trap/engines/engine.py.229: process: 0.54745 | load: 0.00246 [I 2022-10-28 09:15:57] ...trap/engines/engine.py.230: loss: 0.16638 [I 2022-10-28 09:15:57] ...optimizers/trijoint.py.151: total_norm: 15.909559 [I 2022-10-28 09:16:03] ...trap/engines/engine.py.224: train: epoch 117 | batch 1710/2304 [I 2022-10-28 09:16:03] ...trap/engines/engine.py.228: elapsed: 0:16:01 | left: 0:05:30 [I 2022-10-28 09:16:03] ...trap/engines/engine.py.229: process: 0.56273 | load: 0.00244 [I 2022-10-28 09:16:03] ...trap/engines/engine.py.230: loss: 0.12721 [I 2022-10-28 09:16:03] ...optimizers/trijoint.py.151: total_norm: 29.700083 [I 2022-10-28 09:16:09] ...trap/engines/engine.py.224: train: epoch 117 | batch 1720/2304 [I 2022-10-28 09:16:09] ...trap/engines/engine.py.228: elapsed: 0:16:06 | left: 0:05:26 [I 2022-10-28 09:16:09] ...trap/engines/engine.py.229: process: 0.56951 | load: 0.00244 [I 2022-10-28 09:16:09] ...trap/engines/engine.py.230: loss: 0.11095 [I 2022-10-28 09:16:09] ...optimizers/trijoint.py.151: total_norm: 20.637695 [I 2022-10-28 09:16:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 1730/2304 [I 2022-10-28 09:16:14] ...trap/engines/engine.py.228: elapsed: 0:16:12 | left: 0:05:18 [I 2022-10-28 09:16:14] ...trap/engines/engine.py.229: process: 0.55032 | load: 0.00246 [I 2022-10-28 09:16:14] ...trap/engines/engine.py.230: loss: 0.13790 [I 2022-10-28 09:16:14] ...optimizers/trijoint.py.151: total_norm: 37.514957 [I 2022-10-28 09:16:20] ...trap/engines/engine.py.224: train: epoch 117 | batch 1740/2304 [I 2022-10-28 09:16:20] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:13 [I 2022-10-28 09:16:20] ...trap/engines/engine.py.229: process: 0.55066 | load: 0.00241 [I 2022-10-28 09:16:20] ...trap/engines/engine.py.230: loss: 0.13347 [I 2022-10-28 09:16:20] ...optimizers/trijoint.py.151: total_norm: 85.706619 [I 2022-10-28 09:16:25] ...trap/engines/engine.py.224: train: epoch 117 | batch 1750/2304 [I 2022-10-28 09:16:25] ...trap/engines/engine.py.228: elapsed: 0:16:23 | left: 0:05:10 [I 2022-10-28 09:16:25] ...trap/engines/engine.py.229: process: 0.56932 | load: 0.00243 [I 2022-10-28 09:16:25] ...trap/engines/engine.py.230: loss: 0.08565 [I 2022-10-28 09:16:25] ...optimizers/trijoint.py.151: total_norm: 12.062860 [I 2022-10-28 09:16:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 1760/2304 [I 2022-10-28 09:16:31] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:03 [I 2022-10-28 09:16:31] ...trap/engines/engine.py.229: process: 0.55998 | load: 0.00243 [I 2022-10-28 09:16:31] ...trap/engines/engine.py.230: loss: 0.10623 [I 2022-10-28 09:16:31] ...optimizers/trijoint.py.151: total_norm: 9.673045 [I 2022-10-28 09:16:37] ...trap/engines/engine.py.224: train: epoch 117 | batch 1770/2304 [I 2022-10-28 09:16:37] ...trap/engines/engine.py.228: elapsed: 0:16:34 | left: 0:04:58 [I 2022-10-28 09:16:37] ...trap/engines/engine.py.229: process: 0.54879 | load: 0.00251 [I 2022-10-28 09:16:37] ...trap/engines/engine.py.230: loss: 0.11053 [I 2022-10-28 09:16:37] ...optimizers/trijoint.py.151: total_norm: 11.071893 [I 2022-10-28 09:16:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 1780/2304 [I 2022-10-28 09:16:42] ...trap/engines/engine.py.228: elapsed: 0:16:40 | left: 0:04:55 [I 2022-10-28 09:16:42] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00248 [I 2022-10-28 09:16:42] ...trap/engines/engine.py.230: loss: 0.12141 [I 2022-10-28 09:16:42] ...optimizers/trijoint.py.151: total_norm: 18.190493 [I 2022-10-28 09:16:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 1790/2304 [I 2022-10-28 09:16:48] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:48 [I 2022-10-28 09:16:48] ...trap/engines/engine.py.229: process: 0.55885 | load: 0.00242 [I 2022-10-28 09:16:48] ...trap/engines/engine.py.230: loss: 0.22502 [I 2022-10-28 09:16:48] ...optimizers/trijoint.py.151: total_norm: 17.085840 [I 2022-10-28 09:16:54] ...trap/engines/engine.py.224: train: epoch 117 | batch 1800/2304 [I 2022-10-28 09:16:54] ...trap/engines/engine.py.228: elapsed: 0:16:51 | left: 0:04:44 [I 2022-10-28 09:16:54] ...trap/engines/engine.py.229: process: 0.56889 | load: 0.00244 [I 2022-10-28 09:16:54] ...trap/engines/engine.py.230: loss: 0.09049 [I 2022-10-28 09:16:54] ...optimizers/trijoint.py.151: total_norm: 11.166536 [I 2022-10-28 09:16:59] ...trap/engines/engine.py.224: train: epoch 117 | batch 1810/2304 [I 2022-10-28 09:16:59] ...trap/engines/engine.py.228: elapsed: 0:16:57 | left: 0:04:36 [I 2022-10-28 09:16:59] ...trap/engines/engine.py.229: process: 0.56347 | load: 0.00241 [I 2022-10-28 09:16:59] ...trap/engines/engine.py.230: loss: 0.08445 [I 2022-10-28 09:16:59] ...optimizers/trijoint.py.151: total_norm: 14.265096 [I 2022-10-28 09:17:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 1820/2304 [I 2022-10-28 09:17:05] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:29 [I 2022-10-28 09:17:05] ...trap/engines/engine.py.229: process: 0.55630 | load: 0.00243 [I 2022-10-28 09:17:05] ...trap/engines/engine.py.230: loss: 0.13625 [I 2022-10-28 09:17:05] ...optimizers/trijoint.py.151: total_norm: 14.738660 [I 2022-10-28 09:17:10] ...trap/engines/engine.py.224: train: epoch 117 | batch 1830/2304 [I 2022-10-28 09:17:10] ...trap/engines/engine.py.228: elapsed: 0:17:08 | left: 0:04:24 [I 2022-10-28 09:17:10] ...trap/engines/engine.py.229: process: 0.54940 | load: 0.00242 [I 2022-10-28 09:17:10] ...trap/engines/engine.py.230: loss: 0.16572 [I 2022-10-28 09:17:10] ...optimizers/trijoint.py.151: total_norm: 14.608144 [I 2022-10-28 09:17:16] ...trap/engines/engine.py.224: train: epoch 117 | batch 1840/2304 [I 2022-10-28 09:17:16] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:18 [I 2022-10-28 09:17:16] ...trap/engines/engine.py.229: process: 0.55683 | load: 0.00244 [I 2022-10-28 09:17:16] ...trap/engines/engine.py.230: loss: 0.10708 [I 2022-10-28 09:17:16] ...optimizers/trijoint.py.151: total_norm: 14.938834 [I 2022-10-28 09:17:22] ...trap/engines/engine.py.224: train: epoch 117 | batch 1850/2304 [I 2022-10-28 09:17:22] ...trap/engines/engine.py.228: elapsed: 0:17:19 | left: 0:04:12 [I 2022-10-28 09:17:22] ...trap/engines/engine.py.229: process: 0.54530 | load: 0.00245 [I 2022-10-28 09:17:22] ...trap/engines/engine.py.230: loss: 0.10564 [I 2022-10-28 09:17:22] ...optimizers/trijoint.py.151: total_norm: 11.134367 [I 2022-10-28 09:17:27] ...trap/engines/engine.py.224: train: epoch 117 | batch 1860/2304 [I 2022-10-28 09:17:27] ...trap/engines/engine.py.228: elapsed: 0:17:25 | left: 0:04:12 [I 2022-10-28 09:17:27] ...trap/engines/engine.py.229: process: 0.56513 | load: 0.00246 [I 2022-10-28 09:17:27] ...trap/engines/engine.py.230: loss: 0.09451 [I 2022-10-28 09:17:27] ...optimizers/trijoint.py.151: total_norm: 8.906391 [I 2022-10-28 09:17:52] ...trap/engines/engine.py.224: train: epoch 117 | batch 1870/2304 [I 2022-10-28 09:17:52] ...trap/engines/engine.py.228: elapsed: 0:17:50 | left: 0:07:46 [I 2022-10-28 09:17:52] ...trap/engines/engine.py.229: process: 0.56180 | load: 0.00242 [I 2022-10-28 09:17:52] ...trap/engines/engine.py.230: loss: 0.09421 [I 2022-10-28 09:17:52] ...optimizers/trijoint.py.151: total_norm: 11.450894 [I 2022-10-28 09:17:58] ...trap/engines/engine.py.224: train: epoch 117 | batch 1880/2304 [I 2022-10-28 09:17:58] ...trap/engines/engine.py.228: elapsed: 0:17:55 | left: 0:04:19 [I 2022-10-28 09:17:58] ...trap/engines/engine.py.229: process: 0.55123 | load: 0.00244 [I 2022-10-28 09:17:58] ...trap/engines/engine.py.230: loss: 0.10305 [I 2022-10-28 09:17:58] ...optimizers/trijoint.py.151: total_norm: 16.597521 [I 2022-10-28 09:18:03] ...trap/engines/engine.py.224: train: epoch 117 | batch 1890/2304 [I 2022-10-28 09:18:03] ...trap/engines/engine.py.228: elapsed: 0:18:01 | left: 0:04:03 [I 2022-10-28 09:18:03] ...trap/engines/engine.py.229: process: 0.56039 | load: 0.00242 [I 2022-10-28 09:18:03] ...trap/engines/engine.py.230: loss: 0.09327 [I 2022-10-28 09:18:03] ...optimizers/trijoint.py.151: total_norm: 19.176216 [I 2022-10-28 09:18:09] ...trap/engines/engine.py.224: train: epoch 117 | batch 1900/2304 [I 2022-10-28 09:18:09] ...trap/engines/engine.py.228: elapsed: 0:18:07 | left: 0:03:47 [I 2022-10-28 09:18:09] ...trap/engines/engine.py.229: process: 0.54466 | load: 0.00241 [I 2022-10-28 09:18:09] ...trap/engines/engine.py.230: loss: 0.15708 [I 2022-10-28 09:18:09] ...optimizers/trijoint.py.151: total_norm: 11.111657 [I 2022-10-28 09:18:15] ...trap/engines/engine.py.224: train: epoch 117 | batch 1910/2304 [I 2022-10-28 09:18:15] ...trap/engines/engine.py.228: elapsed: 0:18:12 | left: 0:03:39 [I 2022-10-28 09:18:15] ...trap/engines/engine.py.229: process: 0.54900 | load: 0.00238 [I 2022-10-28 09:18:15] ...trap/engines/engine.py.230: loss: 0.08274 [I 2022-10-28 09:18:15] ...optimizers/trijoint.py.151: total_norm: 15.399253 [I 2022-10-28 09:18:20] ...trap/engines/engine.py.224: train: epoch 117 | batch 1920/2304 [I 2022-10-28 09:18:20] ...trap/engines/engine.py.228: elapsed: 0:18:18 | left: 0:03:35 [I 2022-10-28 09:18:20] ...trap/engines/engine.py.229: process: 0.56115 | load: 0.00247 [I 2022-10-28 09:18:20] ...trap/engines/engine.py.230: loss: 0.17435 [I 2022-10-28 09:18:20] ...optimizers/trijoint.py.151: total_norm: 14.003910 [I 2022-10-28 09:18:26] ...trap/engines/engine.py.224: train: epoch 117 | batch 1930/2304 [I 2022-10-28 09:18:26] ...trap/engines/engine.py.228: elapsed: 0:18:23 | left: 0:03:30 [I 2022-10-28 09:18:26] ...trap/engines/engine.py.229: process: 0.56399 | load: 0.00239 [I 2022-10-28 09:18:26] ...trap/engines/engine.py.230: loss: 0.15506 [I 2022-10-28 09:18:26] ...optimizers/trijoint.py.151: total_norm: 19.646200 [I 2022-10-28 09:18:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 1940/2304 [I 2022-10-28 09:18:31] ...trap/engines/engine.py.228: elapsed: 0:18:29 | left: 0:03:23 [I 2022-10-28 09:18:31] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00240 [I 2022-10-28 09:18:31] ...trap/engines/engine.py.230: loss: 0.20605 [I 2022-10-28 09:18:31] ...optimizers/trijoint.py.151: total_norm: 13.612546 [I 2022-10-28 09:18:37] ...trap/engines/engine.py.224: train: epoch 117 | batch 1950/2304 [I 2022-10-28 09:18:37] ...trap/engines/engine.py.228: elapsed: 0:18:35 | left: 0:03:18 [I 2022-10-28 09:18:37] ...trap/engines/engine.py.229: process: 0.56445 | load: 0.00241 [I 2022-10-28 09:18:37] ...trap/engines/engine.py.230: loss: 0.11276 [I 2022-10-28 09:18:37] ...optimizers/trijoint.py.151: total_norm: 20.072390 [I 2022-10-28 09:18:43] ...trap/engines/engine.py.224: train: epoch 117 | batch 1960/2304 [I 2022-10-28 09:18:43] ...trap/engines/engine.py.228: elapsed: 0:18:40 | left: 0:03:15 [I 2022-10-28 09:18:43] ...trap/engines/engine.py.229: process: 0.56543 | load: 0.00240 [I 2022-10-28 09:18:43] ...trap/engines/engine.py.230: loss: 0.08314 [I 2022-10-28 09:18:43] ...optimizers/trijoint.py.151: total_norm: 32.936485 [I 2022-10-28 09:18:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 1970/2304 [I 2022-10-28 09:18:48] ...trap/engines/engine.py.228: elapsed: 0:18:46 | left: 0:03:07 [I 2022-10-28 09:18:48] ...trap/engines/engine.py.229: process: 0.54678 | load: 0.00249 [I 2022-10-28 09:18:48] ...trap/engines/engine.py.230: loss: 0.12942 [I 2022-10-28 09:18:48] ...optimizers/trijoint.py.151: total_norm: 9.470510 [I 2022-10-28 09:18:54] ...trap/engines/engine.py.224: train: epoch 117 | batch 1980/2304 [I 2022-10-28 09:18:54] ...trap/engines/engine.py.228: elapsed: 0:18:52 | left: 0:03:01 [I 2022-10-28 09:18:54] ...trap/engines/engine.py.229: process: 0.56043 | load: 0.00237 [I 2022-10-28 09:18:54] ...trap/engines/engine.py.230: loss: 0.13986 [I 2022-10-28 09:18:54] ...optimizers/trijoint.py.151: total_norm: 17.332508 [I 2022-10-28 09:19:00] ...trap/engines/engine.py.224: train: epoch 117 | batch 1990/2304 [I 2022-10-28 09:19:00] ...trap/engines/engine.py.228: elapsed: 0:18:57 | left: 0:02:55 [I 2022-10-28 09:19:00] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00241 [I 2022-10-28 09:19:00] ...trap/engines/engine.py.230: loss: 0.12233 [I 2022-10-28 09:19:00] ...optimizers/trijoint.py.151: total_norm: 33.517410 [I 2022-10-28 09:19:05] ...trap/engines/engine.py.224: train: epoch 117 | batch 2000/2304 [I 2022-10-28 09:19:05] ...trap/engines/engine.py.228: elapsed: 0:19:03 | left: 0:02:51 [I 2022-10-28 09:19:05] ...trap/engines/engine.py.229: process: 0.56208 | load: 0.00238 [I 2022-10-28 09:19:05] ...trap/engines/engine.py.230: loss: 0.09861 [I 2022-10-28 09:19:05] ...optimizers/trijoint.py.151: total_norm: 15.245921 [I 2022-10-28 09:19:11] ...trap/engines/engine.py.224: train: epoch 117 | batch 2010/2304 [I 2022-10-28 09:19:11] ...trap/engines/engine.py.228: elapsed: 0:19:08 | left: 0:02:44 [I 2022-10-28 09:19:11] ...trap/engines/engine.py.229: process: 0.55467 | load: 0.00251 [I 2022-10-28 09:19:11] ...trap/engines/engine.py.230: loss: 0.13213 [I 2022-10-28 09:19:11] ...optimizers/trijoint.py.151: total_norm: 9.239769 [I 2022-10-28 09:19:17] ...trap/engines/engine.py.224: train: epoch 117 | batch 2020/2304 [I 2022-10-28 09:19:17] ...trap/engines/engine.py.228: elapsed: 0:19:14 | left: 0:02:43 [I 2022-10-28 09:19:17] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00245 [I 2022-10-28 09:19:17] ...trap/engines/engine.py.230: loss: 0.10146 [I 2022-10-28 09:19:17] ...optimizers/trijoint.py.151: total_norm: 10.498647 [I 2022-10-28 09:19:22] ...trap/engines/engine.py.224: train: epoch 117 | batch 2030/2304 [I 2022-10-28 09:19:22] ...trap/engines/engine.py.228: elapsed: 0:19:20 | left: 0:02:33 [I 2022-10-28 09:19:22] ...trap/engines/engine.py.229: process: 0.57245 | load: 0.00249 [I 2022-10-28 09:19:22] ...trap/engines/engine.py.230: loss: 0.09567 [I 2022-10-28 09:19:22] ...optimizers/trijoint.py.151: total_norm: 25.718622 [I 2022-10-28 09:19:28] ...trap/engines/engine.py.224: train: epoch 117 | batch 2040/2304 [I 2022-10-28 09:19:28] ...trap/engines/engine.py.228: elapsed: 0:19:25 | left: 0:02:27 [I 2022-10-28 09:19:28] ...trap/engines/engine.py.229: process: 0.55976 | load: 0.00243 [I 2022-10-28 09:19:28] ...trap/engines/engine.py.230: loss: 0.10991 [I 2022-10-28 09:19:28] ...optimizers/trijoint.py.151: total_norm: 23.864233 [I 2022-10-28 09:19:33] ...trap/engines/engine.py.224: train: epoch 117 | batch 2050/2304 [I 2022-10-28 09:19:33] ...trap/engines/engine.py.228: elapsed: 0:19:31 | left: 0:02:22 [I 2022-10-28 09:19:33] ...trap/engines/engine.py.229: process: 0.56001 | load: 0.00241 [I 2022-10-28 09:19:33] ...trap/engines/engine.py.230: loss: 0.12782 [I 2022-10-28 09:19:33] ...optimizers/trijoint.py.151: total_norm: 23.684536 [I 2022-10-28 09:19:39] ...trap/engines/engine.py.224: train: epoch 117 | batch 2060/2304 [I 2022-10-28 09:19:39] ...trap/engines/engine.py.228: elapsed: 0:19:37 | left: 0:02:15 [I 2022-10-28 09:19:39] ...trap/engines/engine.py.229: process: 0.55594 | load: 0.00295 [I 2022-10-28 09:19:39] ...trap/engines/engine.py.230: loss: 0.15464 [I 2022-10-28 09:19:39] ...optimizers/trijoint.py.151: total_norm: 12.594118 [I 2022-10-28 09:19:45] ...trap/engines/engine.py.224: train: epoch 117 | batch 2070/2304 [I 2022-10-28 09:19:45] ...trap/engines/engine.py.228: elapsed: 0:19:42 | left: 0:02:10 [I 2022-10-28 09:19:45] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00245 [I 2022-10-28 09:19:45] ...trap/engines/engine.py.230: loss: 0.11267 [I 2022-10-28 09:19:45] ...optimizers/trijoint.py.151: total_norm: 13.216643 [I 2022-10-28 09:19:50] ...trap/engines/engine.py.224: train: epoch 117 | batch 2080/2304 [I 2022-10-28 09:19:50] ...trap/engines/engine.py.228: elapsed: 0:19:48 | left: 0:02:11 [I 2022-10-28 09:19:50] ...trap/engines/engine.py.229: process: 0.60745 | load: 0.01680 [I 2022-10-28 09:19:50] ...trap/engines/engine.py.230: loss: 0.08189 [I 2022-10-28 09:19:50] ...optimizers/trijoint.py.151: total_norm: 14.964135 [I 2022-10-28 09:19:56] ...trap/engines/engine.py.224: train: epoch 117 | batch 2090/2304 [I 2022-10-28 09:19:56] ...trap/engines/engine.py.228: elapsed: 0:19:53 | left: 0:02:00 [I 2022-10-28 09:19:56] ...trap/engines/engine.py.229: process: 0.55892 | load: 0.00244 [I 2022-10-28 09:19:56] ...trap/engines/engine.py.230: loss: 0.17498 [I 2022-10-28 09:19:56] ...optimizers/trijoint.py.151: total_norm: 76.025085 [I 2022-10-28 09:20:02] ...trap/engines/engine.py.224: train: epoch 117 | batch 2100/2304 [I 2022-10-28 09:20:02] ...trap/engines/engine.py.228: elapsed: 0:19:59 | left: 0:01:54 [I 2022-10-28 09:20:02] ...trap/engines/engine.py.229: process: 0.55992 | load: 0.00245 [I 2022-10-28 09:20:02] ...trap/engines/engine.py.230: loss: 0.14804 [I 2022-10-28 09:20:02] ...optimizers/trijoint.py.151: total_norm: 32.639355 [I 2022-10-28 09:20:07] ...trap/engines/engine.py.224: train: epoch 117 | batch 2110/2304 [I 2022-10-28 09:20:07] ...trap/engines/engine.py.228: elapsed: 0:20:05 | left: 0:01:48 [I 2022-10-28 09:20:07] ...trap/engines/engine.py.229: process: 0.55173 | load: 0.00252 [I 2022-10-28 09:20:07] ...trap/engines/engine.py.230: loss: 0.09036 [I 2022-10-28 09:20:07] ...optimizers/trijoint.py.151: total_norm: 16.271049 [I 2022-10-28 09:20:13] ...trap/engines/engine.py.224: train: epoch 117 | batch 2120/2304 [I 2022-10-28 09:20:13] ...trap/engines/engine.py.228: elapsed: 0:20:10 | left: 0:01:42 [I 2022-10-28 09:20:13] ...trap/engines/engine.py.229: process: 0.55012 | load: 0.00251 [I 2022-10-28 09:20:13] ...trap/engines/engine.py.230: loss: 0.08820 [I 2022-10-28 09:20:13] ...optimizers/trijoint.py.151: total_norm: 9.927122 [I 2022-10-28 09:20:18] ...trap/engines/engine.py.224: train: epoch 117 | batch 2130/2304 [I 2022-10-28 09:20:18] ...trap/engines/engine.py.228: elapsed: 0:20:16 | left: 0:01:37 [I 2022-10-28 09:20:18] ...trap/engines/engine.py.229: process: 0.56235 | load: 0.00245 [I 2022-10-28 09:20:18] ...trap/engines/engine.py.230: loss: 0.11583 [I 2022-10-28 09:20:18] ...optimizers/trijoint.py.151: total_norm: 39.657349 [I 2022-10-28 09:20:24] ...trap/engines/engine.py.224: train: epoch 117 | batch 2140/2304 [I 2022-10-28 09:20:24] ...trap/engines/engine.py.228: elapsed: 0:20:21 | left: 0:01:31 [I 2022-10-28 09:20:24] ...trap/engines/engine.py.229: process: 0.55418 | load: 0.00246 [I 2022-10-28 09:20:24] ...trap/engines/engine.py.230: loss: 0.21619 [I 2022-10-28 09:20:24] ...optimizers/trijoint.py.151: total_norm: 24.617649 [I 2022-10-28 09:20:30] ...trap/engines/engine.py.224: train: epoch 117 | batch 2150/2304 [I 2022-10-28 09:20:30] ...trap/engines/engine.py.228: elapsed: 0:20:27 | left: 0:01:26 [I 2022-10-28 09:20:30] ...trap/engines/engine.py.229: process: 0.56017 | load: 0.00245 [I 2022-10-28 09:20:30] ...trap/engines/engine.py.230: loss: 0.09431 [I 2022-10-28 09:20:30] ...optimizers/trijoint.py.151: total_norm: 15.250929 [I 2022-10-28 09:20:35] ...trap/engines/engine.py.224: train: epoch 117 | batch 2160/2304 [I 2022-10-28 09:20:35] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:01:20 [I 2022-10-28 09:20:35] ...trap/engines/engine.py.229: process: 0.56057 | load: 0.00243 [I 2022-10-28 09:20:35] ...trap/engines/engine.py.230: loss: 0.12734 [I 2022-10-28 09:20:35] ...optimizers/trijoint.py.151: total_norm: 20.650412 [I 2022-10-28 09:20:41] ...trap/engines/engine.py.224: train: epoch 117 | batch 2170/2304 [I 2022-10-28 09:20:41] ...trap/engines/engine.py.228: elapsed: 0:20:38 | left: 0:01:14 [I 2022-10-28 09:20:41] ...trap/engines/engine.py.229: process: 0.56741 | load: 0.00247 [I 2022-10-28 09:20:41] ...trap/engines/engine.py.230: loss: 0.09885 [I 2022-10-28 09:20:41] ...optimizers/trijoint.py.151: total_norm: 10.388683 [I 2022-10-28 09:20:46] ...trap/engines/engine.py.224: train: epoch 117 | batch 2180/2304 [I 2022-10-28 09:20:46] ...trap/engines/engine.py.228: elapsed: 0:20:44 | left: 0:01:09 [I 2022-10-28 09:20:46] ...trap/engines/engine.py.229: process: 0.56063 | load: 0.00245 [I 2022-10-28 09:20:46] ...trap/engines/engine.py.230: loss: 0.14740 [I 2022-10-28 09:20:46] ...optimizers/trijoint.py.151: total_norm: 114.040962 [I 2022-10-28 09:20:52] ...trap/engines/engine.py.224: train: epoch 117 | batch 2190/2304 [I 2022-10-28 09:20:52] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:01:03 [I 2022-10-28 09:20:52] ...trap/engines/engine.py.229: process: 0.57211 | load: 0.00245 [I 2022-10-28 09:20:52] ...trap/engines/engine.py.230: loss: 0.09454 [I 2022-10-28 09:20:52] ...optimizers/trijoint.py.151: total_norm: 9.877214 [I 2022-10-28 09:20:58] ...trap/engines/engine.py.224: train: epoch 117 | batch 2200/2304 [I 2022-10-28 09:20:58] ...trap/engines/engine.py.228: elapsed: 0:20:55 | left: 0:00:57 [I 2022-10-28 09:20:58] ...trap/engines/engine.py.229: process: 0.55476 | load: 0.00250 [I 2022-10-28 09:20:58] ...trap/engines/engine.py.230: loss: 0.11738 [I 2022-10-28 09:20:58] ...optimizers/trijoint.py.151: total_norm: 18.286922 [I 2022-10-28 09:21:03] ...trap/engines/engine.py.224: train: epoch 117 | batch 2210/2304 [I 2022-10-28 09:21:03] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:52 [I 2022-10-28 09:21:03] ...trap/engines/engine.py.229: process: 0.55593 | load: 0.00243 [I 2022-10-28 09:21:03] ...trap/engines/engine.py.230: loss: 0.13724 [I 2022-10-28 09:21:03] ...optimizers/trijoint.py.151: total_norm: 11.253950 [I 2022-10-28 09:21:09] ...trap/engines/engine.py.224: train: epoch 117 | batch 2220/2304 [I 2022-10-28 09:21:09] ...trap/engines/engine.py.228: elapsed: 0:21:06 | left: 0:00:46 [I 2022-10-28 09:21:09] ...trap/engines/engine.py.229: process: 0.55857 | load: 0.00244 [I 2022-10-28 09:21:09] ...trap/engines/engine.py.230: loss: 0.10254 [I 2022-10-28 09:21:09] ...optimizers/trijoint.py.151: total_norm: 18.508682 [I 2022-10-28 09:21:14] ...trap/engines/engine.py.224: train: epoch 117 | batch 2230/2304 [I 2022-10-28 09:21:14] ...trap/engines/engine.py.228: elapsed: 0:21:12 | left: 0:00:41 [I 2022-10-28 09:21:14] ...trap/engines/engine.py.229: process: 0.55735 | load: 0.00245 [I 2022-10-28 09:21:14] ...trap/engines/engine.py.230: loss: 0.21392 [I 2022-10-28 09:21:14] ...optimizers/trijoint.py.151: total_norm: 19.591780 [I 2022-10-28 09:21:20] ...trap/engines/engine.py.224: train: epoch 117 | batch 2240/2304 [I 2022-10-28 09:21:20] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:35 [I 2022-10-28 09:21:20] ...trap/engines/engine.py.229: process: 0.55366 | load: 0.00248 [I 2022-10-28 09:21:20] ...trap/engines/engine.py.230: loss: 0.11537 [I 2022-10-28 09:21:20] ...optimizers/trijoint.py.151: total_norm: 19.528023 [I 2022-10-28 09:21:26] ...trap/engines/engine.py.224: train: epoch 117 | batch 2250/2304 [I 2022-10-28 09:21:26] ...trap/engines/engine.py.228: elapsed: 0:21:23 | left: 0:00:30 [I 2022-10-28 09:21:26] ...trap/engines/engine.py.229: process: 0.56056 | load: 0.00243 [I 2022-10-28 09:21:26] ...trap/engines/engine.py.230: loss: 0.08520 [I 2022-10-28 09:21:26] ...optimizers/trijoint.py.151: total_norm: 15.163778 [I 2022-10-28 09:21:31] ...trap/engines/engine.py.224: train: epoch 117 | batch 2260/2304 [I 2022-10-28 09:21:31] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:24 [I 2022-10-28 09:21:31] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00245 [I 2022-10-28 09:21:31] ...trap/engines/engine.py.230: loss: 0.13878 [I 2022-10-28 09:21:31] ...optimizers/trijoint.py.151: total_norm: 8.756086 [I 2022-10-28 09:21:37] ...trap/engines/engine.py.224: train: epoch 117 | batch 2270/2304 [I 2022-10-28 09:21:37] ...trap/engines/engine.py.228: elapsed: 0:21:34 | left: 0:00:19 [I 2022-10-28 09:21:37] ...trap/engines/engine.py.229: process: 0.62073 | load: 0.00251 [I 2022-10-28 09:21:37] ...trap/engines/engine.py.230: loss: 0.13630 [I 2022-10-28 09:21:37] ...optimizers/trijoint.py.151: total_norm: 66.279236 [I 2022-10-28 09:21:42] ...trap/engines/engine.py.224: train: epoch 117 | batch 2280/2304 [I 2022-10-28 09:21:42] ...trap/engines/engine.py.228: elapsed: 0:21:40 | left: 0:00:13 [I 2022-10-28 09:21:42] ...trap/engines/engine.py.229: process: 0.55845 | load: 0.00247 [I 2022-10-28 09:21:42] ...trap/engines/engine.py.230: loss: 0.14303 [I 2022-10-28 09:21:42] ...optimizers/trijoint.py.151: total_norm: 16.441612 [I 2022-10-28 09:21:48] ...trap/engines/engine.py.224: train: epoch 117 | batch 2290/2304 [I 2022-10-28 09:21:48] ...trap/engines/engine.py.228: elapsed: 0:21:46 | left: 0:00:07 [I 2022-10-28 09:21:48] ...trap/engines/engine.py.229: process: 0.55416 | load: 0.00249 [I 2022-10-28 09:21:48] ...trap/engines/engine.py.230: loss: 0.10772 [I 2022-10-28 09:21:48] ...optimizers/trijoint.py.151: total_norm: 19.087751 [I 2022-10-28 09:21:54] ...trap/engines/engine.py.224: train: epoch 117 | batch 2300/2304 [I 2022-10-28 09:21:54] ...trap/engines/engine.py.228: elapsed: 0:21:51 | left: 0:00:02 [I 2022-10-28 09:21:54] ...trap/engines/engine.py.229: process: 0.54755 | load: 0.00016 [I 2022-10-28 09:21:54] ...trap/engines/engine.py.230: loss: 0.09517 [I 2022-10-28 09:21:54] ...optimizers/trijoint.py.151: total_norm: 9.459176 [I 2022-10-28 09:21:56] ...trap/engines/engine.py.224: train: epoch 117 | batch 2304/2304 [I 2022-10-28 09:21:56] ...trap/engines/engine.py.228: elapsed: 0:21:53 | left: 0:00:00 [I 2022-10-28 09:21:56] ...trap/engines/engine.py.229: process: 0.55570 | load: 0.00015 [I 2022-10-28 09:21:56] ...trap/engines/engine.py.230: loss: 0.10081 [I 2022-10-28 09:21:56] ...optimizers/trijoint.py.151: total_norm: 9.795591 [S 2022-10-28 09:21:57] ...trap/engines/engine.py.129: train_epoch.epoch: 117 [S 2022-10-28 09:21:57] ...trap/engines/engine.py.129: train_epoch.loss: 0.1196811329710768 [I 2022-10-28 09:21:57] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 09:21:59] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 117 [I 2022-10-28 09:21:59] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 09:22:01] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 09:22:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 09:22:01] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 09:22:05] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 09:22:05] ...trap/engines/engine.py.321: eval: epoch 117 | batch 0/510 [I 2022-10-28 09:22:05] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:53:50 [I 2022-10-28 09:22:05] ...trap/engines/engine.py.326: process: 6.33464 | load: 4.54862 [I 2022-10-28 09:22:08] ...trap/engines/engine.py.321: eval: epoch 117 | batch 10/510 [I 2022-10-28 09:22:08] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:07 [I 2022-10-28 09:22:08] ...trap/engines/engine.py.326: process: 0.32849 | load: 0.00020 [I 2022-10-28 09:22:12] ...trap/engines/engine.py.321: eval: epoch 117 | batch 20/510 [I 2022-10-28 09:22:12] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:15 [I 2022-10-28 09:22:12] ...trap/engines/engine.py.326: process: 0.32634 | load: 0.00025 [I 2022-10-28 09:22:15] ...trap/engines/engine.py.321: eval: epoch 117 | batch 30/510 [I 2022-10-28 09:22:15] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:39 [I 2022-10-28 09:22:15] ...trap/engines/engine.py.326: process: 0.32028 | load: 0.00023 [I 2022-10-28 09:22:18] ...trap/engines/engine.py.321: eval: epoch 117 | batch 40/510 [I 2022-10-28 09:22:18] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-28 09:22:18] ...trap/engines/engine.py.326: process: 0.32698 | load: 0.00020 [I 2022-10-28 09:22:22] ...trap/engines/engine.py.321: eval: epoch 117 | batch 50/510 [I 2022-10-28 09:22:22] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:32 [I 2022-10-28 09:22:22] ...trap/engines/engine.py.326: process: 0.33555 | load: 0.00023 [I 2022-10-28 09:22:25] ...trap/engines/engine.py.321: eval: epoch 117 | batch 60/510 [I 2022-10-28 09:22:25] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:29 [I 2022-10-28 09:22:25] ...trap/engines/engine.py.326: process: 0.32669 | load: 0.00020 [I 2022-10-28 09:22:28] ...trap/engines/engine.py.321: eval: epoch 117 | batch 70/510 [I 2022-10-28 09:22:28] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:25 [I 2022-10-28 09:22:28] ...trap/engines/engine.py.326: process: 0.32893 | load: 0.00030 [I 2022-10-28 09:22:32] ...trap/engines/engine.py.321: eval: epoch 117 | batch 80/510 [I 2022-10-28 09:22:32] ...trap/engines/engine.py.325: elapsed: 0:00:32 | left: 0:02:25 [I 2022-10-28 09:22:32] ...trap/engines/engine.py.326: process: 0.32704 | load: 0.00022 [I 2022-10-28 09:22:35] ...trap/engines/engine.py.321: eval: epoch 117 | batch 90/510 [I 2022-10-28 09:22:35] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:20 [I 2022-10-28 09:22:35] ...trap/engines/engine.py.326: process: 0.34278 | load: 0.00022 [I 2022-10-28 09:22:38] ...trap/engines/engine.py.321: eval: epoch 117 | batch 100/510 [I 2022-10-28 09:22:38] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-28 09:22:38] ...trap/engines/engine.py.326: process: 0.29023 | load: 0.00025 [I 2022-10-28 09:22:41] ...trap/engines/engine.py.321: eval: epoch 117 | batch 110/510 [I 2022-10-28 09:22:41] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:56 [I 2022-10-28 09:22:41] ...trap/engines/engine.py.326: process: 0.28925 | load: 0.00023 [I 2022-10-28 09:22:44] ...trap/engines/engine.py.321: eval: epoch 117 | batch 120/510 [I 2022-10-28 09:22:44] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:53 [I 2022-10-28 09:22:44] ...trap/engines/engine.py.326: process: 0.28765 | load: 0.00024 [I 2022-10-28 09:22:47] ...trap/engines/engine.py.321: eval: epoch 117 | batch 130/510 [I 2022-10-28 09:22:47] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:50 [I 2022-10-28 09:22:47] ...trap/engines/engine.py.326: process: 0.28800 | load: 0.00016 [I 2022-10-28 09:22:50] ...trap/engines/engine.py.321: eval: epoch 117 | batch 140/510 [I 2022-10-28 09:22:50] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:48 [I 2022-10-28 09:22:50] ...trap/engines/engine.py.326: process: 0.30623 | load: 0.00014 [I 2022-10-28 09:22:53] ...trap/engines/engine.py.321: eval: epoch 117 | batch 150/510 [I 2022-10-28 09:22:53] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:44 [I 2022-10-28 09:22:53] ...trap/engines/engine.py.326: process: 0.28915 | load: 0.00016 [I 2022-10-28 09:22:56] ...trap/engines/engine.py.321: eval: epoch 117 | batch 160/510 [I 2022-10-28 09:22:56] ...trap/engines/engine.py.325: elapsed: 0:00:56 | left: 0:01:40 [I 2022-10-28 09:22:56] ...trap/engines/engine.py.326: process: 0.28680 | load: 0.00016 [I 2022-10-28 09:22:59] ...trap/engines/engine.py.321: eval: epoch 117 | batch 170/510 [I 2022-10-28 09:22:59] ...trap/engines/engine.py.325: elapsed: 0:00:59 | left: 0:01:36 [I 2022-10-28 09:22:59] ...trap/engines/engine.py.326: process: 0.27819 | load: 0.00016 [I 2022-10-28 09:23:01] ...trap/engines/engine.py.321: eval: epoch 117 | batch 180/510 [I 2022-10-28 09:23:01] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:34 [I 2022-10-28 09:23:01] ...trap/engines/engine.py.326: process: 0.28488 | load: 0.00018 [I 2022-10-28 09:23:04] ...trap/engines/engine.py.321: eval: epoch 117 | batch 190/510 [I 2022-10-28 09:23:04] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:31 [I 2022-10-28 09:23:04] ...trap/engines/engine.py.326: process: 0.27697 | load: 0.00015 [I 2022-10-28 09:23:07] ...trap/engines/engine.py.321: eval: epoch 117 | batch 200/510 [I 2022-10-28 09:23:07] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:28 [I 2022-10-28 09:23:07] ...trap/engines/engine.py.326: process: 0.28929 | load: 0.00015 [I 2022-10-28 09:23:10] ...trap/engines/engine.py.321: eval: epoch 117 | batch 210/510 [I 2022-10-28 09:23:10] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 09:23:10] ...trap/engines/engine.py.326: process: 0.28611 | load: 0.00018 [I 2022-10-28 09:23:13] ...trap/engines/engine.py.321: eval: epoch 117 | batch 220/510 [I 2022-10-28 09:23:13] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:23 [I 2022-10-28 09:23:13] ...trap/engines/engine.py.326: process: 0.28454 | load: 0.00014 [I 2022-10-28 09:23:16] ...trap/engines/engine.py.321: eval: epoch 117 | batch 230/510 [I 2022-10-28 09:23:16] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:20 [I 2022-10-28 09:23:16] ...trap/engines/engine.py.326: process: 0.28380 | load: 0.00014 [I 2022-10-28 09:23:19] ...trap/engines/engine.py.321: eval: epoch 117 | batch 240/510 [I 2022-10-28 09:23:19] ...trap/engines/engine.py.325: elapsed: 0:01:19 | left: 0:01:17 [I 2022-10-28 09:23:19] ...trap/engines/engine.py.326: process: 0.28527 | load: 0.00015 [I 2022-10-28 09:23:22] ...trap/engines/engine.py.321: eval: epoch 117 | batch 250/510 [I 2022-10-28 09:23:22] ...trap/engines/engine.py.325: elapsed: 0:01:22 | left: 0:01:14 [I 2022-10-28 09:23:22] ...trap/engines/engine.py.326: process: 0.28413 | load: 0.00016 [I 2022-10-28 09:23:24] ...trap/engines/engine.py.321: eval: epoch 117 | batch 260/510 [I 2022-10-28 09:23:24] ...trap/engines/engine.py.325: elapsed: 0:01:25 | left: 0:01:11 [I 2022-10-28 09:23:24] ...trap/engines/engine.py.326: process: 0.28792 | load: 0.00016 [I 2022-10-28 09:23:27] ...trap/engines/engine.py.321: eval: epoch 117 | batch 270/510 [I 2022-10-28 09:23:27] ...trap/engines/engine.py.325: elapsed: 0:01:28 | left: 0:01:08 [I 2022-10-28 09:23:27] ...trap/engines/engine.py.326: process: 0.28306 | load: 0.00015 [I 2022-10-28 09:23:30] ...trap/engines/engine.py.321: eval: epoch 117 | batch 280/510 [I 2022-10-28 09:23:30] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:05 [I 2022-10-28 09:23:30] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00015 [I 2022-10-28 09:23:33] ...trap/engines/engine.py.321: eval: epoch 117 | batch 290/510 [I 2022-10-28 09:23:33] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 09:23:33] ...trap/engines/engine.py.326: process: 0.28345 | load: 0.00018 [I 2022-10-28 09:23:36] ...trap/engines/engine.py.321: eval: epoch 117 | batch 300/510 [I 2022-10-28 09:23:36] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-28 09:23:36] ...trap/engines/engine.py.326: process: 0.28236 | load: 0.00015 [I 2022-10-28 09:23:39] ...trap/engines/engine.py.321: eval: epoch 117 | batch 310/510 [I 2022-10-28 09:23:39] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:57 [I 2022-10-28 09:23:39] ...trap/engines/engine.py.326: process: 0.28720 | load: 0.00015 [I 2022-10-28 09:23:42] ...trap/engines/engine.py.321: eval: epoch 117 | batch 320/510 [I 2022-10-28 09:23:42] ...trap/engines/engine.py.325: elapsed: 0:01:42 | left: 0:00:54 [I 2022-10-28 09:23:42] ...trap/engines/engine.py.326: process: 0.28617 | load: 0.00021 [I 2022-10-28 09:23:45] ...trap/engines/engine.py.321: eval: epoch 117 | batch 330/510 [I 2022-10-28 09:23:45] ...trap/engines/engine.py.325: elapsed: 0:01:45 | left: 0:00:51 [I 2022-10-28 09:23:45] ...trap/engines/engine.py.326: process: 0.29115 | load: 0.00015 [I 2022-10-28 09:23:47] ...trap/engines/engine.py.321: eval: epoch 117 | batch 340/510 [I 2022-10-28 09:23:47] ...trap/engines/engine.py.325: elapsed: 0:01:48 | left: 0:00:49 [I 2022-10-28 09:23:47] ...trap/engines/engine.py.326: process: 0.29038 | load: 0.00014 [I 2022-10-28 09:23:50] ...trap/engines/engine.py.321: eval: epoch 117 | batch 350/510 [I 2022-10-28 09:23:50] ...trap/engines/engine.py.325: elapsed: 0:01:51 | left: 0:00:46 [I 2022-10-28 09:23:50] ...trap/engines/engine.py.326: process: 0.28982 | load: 0.00021 [I 2022-10-28 09:23:53] ...trap/engines/engine.py.321: eval: epoch 117 | batch 360/510 [I 2022-10-28 09:23:53] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:42 [I 2022-10-28 09:23:53] ...trap/engines/engine.py.326: process: 0.28875 | load: 0.00016 [I 2022-10-28 09:23:56] ...trap/engines/engine.py.321: eval: epoch 117 | batch 370/510 [I 2022-10-28 09:23:56] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 09:23:56] ...trap/engines/engine.py.326: process: 0.28450 | load: 0.00015 [I 2022-10-28 09:23:59] ...trap/engines/engine.py.321: eval: epoch 117 | batch 380/510 [I 2022-10-28 09:23:59] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 09:23:59] ...trap/engines/engine.py.326: process: 0.29069 | load: 0.00016 [I 2022-10-28 09:24:02] ...trap/engines/engine.py.321: eval: epoch 117 | batch 390/510 [I 2022-10-28 09:24:02] ...trap/engines/engine.py.325: elapsed: 0:02:02 | left: 0:00:34 [I 2022-10-28 09:24:02] ...trap/engines/engine.py.326: process: 0.28887 | load: 0.00016 [I 2022-10-28 09:24:05] ...trap/engines/engine.py.321: eval: epoch 117 | batch 400/510 [I 2022-10-28 09:24:05] ...trap/engines/engine.py.325: elapsed: 0:02:05 | left: 0:00:31 [I 2022-10-28 09:24:05] ...trap/engines/engine.py.326: process: 0.28336 | load: 0.00015 [I 2022-10-28 09:24:08] ...trap/engines/engine.py.321: eval: epoch 117 | batch 410/510 [I 2022-10-28 09:24:08] ...trap/engines/engine.py.325: elapsed: 0:02:08 | left: 0:00:28 [I 2022-10-28 09:24:08] ...trap/engines/engine.py.326: process: 0.28564 | load: 0.00020 [I 2022-10-28 09:24:10] ...trap/engines/engine.py.321: eval: epoch 117 | batch 420/510 [I 2022-10-28 09:24:10] ...trap/engines/engine.py.325: elapsed: 0:02:11 | left: 0:00:25 [I 2022-10-28 09:24:10] ...trap/engines/engine.py.326: process: 0.29122 | load: 0.00015 [I 2022-10-28 09:24:13] ...trap/engines/engine.py.321: eval: epoch 117 | batch 430/510 [I 2022-10-28 09:24:13] ...trap/engines/engine.py.325: elapsed: 0:02:14 | left: 0:00:22 [I 2022-10-28 09:24:13] ...trap/engines/engine.py.326: process: 0.28797 | load: 0.00014 [I 2022-10-28 09:24:16] ...trap/engines/engine.py.321: eval: epoch 117 | batch 440/510 [I 2022-10-28 09:24:16] ...trap/engines/engine.py.325: elapsed: 0:02:17 | left: 0:00:20 [I 2022-10-28 09:24:16] ...trap/engines/engine.py.326: process: 0.28256 | load: 0.00017 [I 2022-10-28 09:24:19] ...trap/engines/engine.py.321: eval: epoch 117 | batch 450/510 [I 2022-10-28 09:24:19] ...trap/engines/engine.py.325: elapsed: 0:02:20 | left: 0:00:17 [I 2022-10-28 09:24:19] ...trap/engines/engine.py.326: process: 0.28832 | load: 0.00016 [I 2022-10-28 09:24:22] ...trap/engines/engine.py.321: eval: epoch 117 | batch 460/510 [I 2022-10-28 09:24:22] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 09:24:22] ...trap/engines/engine.py.326: process: 0.29068 | load: 0.00016 [I 2022-10-28 09:24:25] ...trap/engines/engine.py.321: eval: epoch 117 | batch 470/510 [I 2022-10-28 09:24:25] ...trap/engines/engine.py.325: elapsed: 0:02:25 | left: 0:00:11 [I 2022-10-28 09:24:25] ...trap/engines/engine.py.326: process: 0.28673 | load: 0.00021 [I 2022-10-28 09:24:28] ...trap/engines/engine.py.321: eval: epoch 117 | batch 480/510 [I 2022-10-28 09:24:28] ...trap/engines/engine.py.325: elapsed: 0:02:28 | left: 0:00:08 [I 2022-10-28 09:24:28] ...trap/engines/engine.py.326: process: 0.28922 | load: 0.00017 [I 2022-10-28 09:24:31] ...trap/engines/engine.py.321: eval: epoch 117 | batch 490/510 [I 2022-10-28 09:24:31] ...trap/engines/engine.py.325: elapsed: 0:02:31 | left: 0:00:05 [I 2022-10-28 09:24:31] ...trap/engines/engine.py.326: process: 0.28493 | load: 0.00015 [I 2022-10-28 09:24:33] ...trap/engines/engine.py.321: eval: epoch 117 | batch 500/510 [I 2022-10-28 09:24:33] ...trap/engines/engine.py.325: elapsed: 0:02:34 | left: 0:00:02 [I 2022-10-28 09:24:33] ...trap/engines/engine.py.326: process: 0.29071 | load: 0.00021 [I 2022-10-28 09:24:36] ...trap/engines/engine.py.321: eval: epoch 117 | batch 510/510 [I 2022-10-28 09:24:36] ...trap/engines/engine.py.325: elapsed: 0:02:37 | left: 0:00:00 [I 2022-10-28 09:24:36] ...trap/engines/engine.py.326: process: 0.28756 | load: 0.00013 [S 2022-10-28 09:24:37] ...trap/engines/engine.py.132: eval_epoch.epoch: 117 [I 2022-10-28 09:24:37] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.736400 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.905100 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.934400 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.731000 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.906200 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.935300 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013109 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.010084 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009891 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.016721 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011250 [S 2022-10-28 09:24:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008798 [I 2022-10-28 09:24:43] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 09:24:43] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 09:24:44] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 09:24:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 09:24:44] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 09:24:49] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 09:24:50] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 09:24:55] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 09:24:55] ...trap/engines/engine.py.163: Training model on trainset for epoch 118 [I 2022-10-28 09:24:55] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 09:24:58] ...trap/engines/engine.py.224: train: epoch 118 | batch 0/2304 [I 2022-10-28 09:24:58] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:43:23 [I 2022-10-28 09:24:58] ...trap/engines/engine.py.229: process: 2.69250 | load: 2.08236 [I 2022-10-28 09:24:58] ...trap/engines/engine.py.230: loss: 0.09181 [I 2022-10-28 09:24:58] ...optimizers/trijoint.py.151: total_norm: 12.163300 [I 2022-10-28 09:25:04] ...trap/engines/engine.py.224: train: epoch 118 | batch 10/2304 [I 2022-10-28 09:25:04] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:29:54 [I 2022-10-28 09:25:04] ...trap/engines/engine.py.229: process: 0.54693 | load: 0.00321 [I 2022-10-28 09:25:04] ...trap/engines/engine.py.230: loss: 0.09172 [I 2022-10-28 09:25:04] ...optimizers/trijoint.py.151: total_norm: 10.071888 [I 2022-10-28 09:25:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 20/2304 [I 2022-10-28 09:25:09] ...trap/engines/engine.py.228: elapsed: 0:00:13 | left: 0:22:06 [I 2022-10-28 09:25:09] ...trap/engines/engine.py.229: process: 0.56106 | load: 0.00396 [I 2022-10-28 09:25:09] ...trap/engines/engine.py.230: loss: 0.08312 [I 2022-10-28 09:25:09] ...optimizers/trijoint.py.151: total_norm: 11.761698 [I 2022-10-28 09:25:15] ...trap/engines/engine.py.224: train: epoch 118 | batch 30/2304 [I 2022-10-28 09:25:15] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:14 [I 2022-10-28 09:25:15] ...trap/engines/engine.py.229: process: 0.55119 | load: 0.00294 [I 2022-10-28 09:25:15] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-28 09:25:15] ...optimizers/trijoint.py.151: total_norm: 33.294907 [I 2022-10-28 09:25:20] ...trap/engines/engine.py.224: train: epoch 118 | batch 40/2304 [I 2022-10-28 09:25:20] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:06 [I 2022-10-28 09:25:20] ...trap/engines/engine.py.229: process: 0.55995 | load: 0.00294 [I 2022-10-28 09:25:20] ...trap/engines/engine.py.230: loss: 0.11121 [I 2022-10-28 09:25:20] ...optimizers/trijoint.py.151: total_norm: 14.404733 [I 2022-10-28 09:25:26] ...trap/engines/engine.py.224: train: epoch 118 | batch 50/2304 [I 2022-10-28 09:25:26] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:21:10 [I 2022-10-28 09:25:26] ...trap/engines/engine.py.229: process: 0.58269 | load: 0.00271 [I 2022-10-28 09:25:26] ...trap/engines/engine.py.230: loss: 0.11960 [I 2022-10-28 09:25:26] ...optimizers/trijoint.py.151: total_norm: 13.674074 [I 2022-10-28 09:25:32] ...trap/engines/engine.py.224: train: epoch 118 | batch 60/2304 [I 2022-10-28 09:25:32] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:49 [I 2022-10-28 09:25:32] ...trap/engines/engine.py.229: process: 0.56084 | load: 0.00282 [I 2022-10-28 09:25:32] ...trap/engines/engine.py.230: loss: 0.08806 [I 2022-10-28 09:25:32] ...optimizers/trijoint.py.151: total_norm: 10.576051 [I 2022-10-28 09:25:37] ...trap/engines/engine.py.224: train: epoch 118 | batch 70/2304 [I 2022-10-28 09:25:37] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:36 [I 2022-10-28 09:25:37] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00287 [I 2022-10-28 09:25:37] ...trap/engines/engine.py.230: loss: 0.10524 [I 2022-10-28 09:25:37] ...optimizers/trijoint.py.151: total_norm: 9.807411 [I 2022-10-28 09:25:43] ...trap/engines/engine.py.224: train: epoch 118 | batch 80/2304 [I 2022-10-28 09:25:43] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:35 [I 2022-10-28 09:25:43] ...trap/engines/engine.py.229: process: 0.55548 | load: 0.00331 [I 2022-10-28 09:25:43] ...trap/engines/engine.py.230: loss: 0.10012 [I 2022-10-28 09:25:43] ...optimizers/trijoint.py.151: total_norm: 15.253609 [I 2022-10-28 09:25:48] ...trap/engines/engine.py.224: train: epoch 118 | batch 90/2304 [I 2022-10-28 09:25:48] ...trap/engines/engine.py.228: elapsed: 0:00:52 | left: 0:20:26 [I 2022-10-28 09:25:48] ...trap/engines/engine.py.229: process: 0.55227 | load: 0.00286 [I 2022-10-28 09:25:48] ...trap/engines/engine.py.230: loss: 0.12901 [I 2022-10-28 09:25:48] ...optimizers/trijoint.py.151: total_norm: 21.449425 [I 2022-10-28 09:25:54] ...trap/engines/engine.py.224: train: epoch 118 | batch 100/2304 [I 2022-10-28 09:25:54] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:18 [I 2022-10-28 09:25:54] ...trap/engines/engine.py.229: process: 0.55986 | load: 0.00255 [I 2022-10-28 09:25:54] ...trap/engines/engine.py.230: loss: 0.09110 [I 2022-10-28 09:25:54] ...optimizers/trijoint.py.151: total_norm: 11.258337 [I 2022-10-28 09:25:59] ...trap/engines/engine.py.224: train: epoch 118 | batch 110/2304 [I 2022-10-28 09:25:59] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:14 [I 2022-10-28 09:25:59] ...trap/engines/engine.py.229: process: 0.55198 | load: 0.00254 [I 2022-10-28 09:25:59] ...trap/engines/engine.py.230: loss: 0.18062 [I 2022-10-28 09:25:59] ...optimizers/trijoint.py.151: total_norm: 18.650955 [I 2022-10-28 09:26:05] ...trap/engines/engine.py.224: train: epoch 118 | batch 120/2304 [I 2022-10-28 09:26:05] ...trap/engines/engine.py.228: elapsed: 0:01:09 | left: 0:20:21 [I 2022-10-28 09:26:05] ...trap/engines/engine.py.229: process: 0.56587 | load: 0.00240 [I 2022-10-28 09:26:05] ...trap/engines/engine.py.230: loss: 0.09541 [I 2022-10-28 09:26:05] ...optimizers/trijoint.py.151: total_norm: 14.996379 [I 2022-10-28 09:26:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 130/2304 [I 2022-10-28 09:26:10] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:12 [I 2022-10-28 09:26:10] ...trap/engines/engine.py.229: process: 0.55259 | load: 0.00250 [I 2022-10-28 09:26:10] ...trap/engines/engine.py.230: loss: 0.07028 [I 2022-10-28 09:26:10] ...optimizers/trijoint.py.151: total_norm: 13.685017 [I 2022-10-28 09:26:16] ...trap/engines/engine.py.224: train: epoch 118 | batch 140/2304 [I 2022-10-28 09:26:16] ...trap/engines/engine.py.228: elapsed: 0:01:20 | left: 0:20:02 [I 2022-10-28 09:26:16] ...trap/engines/engine.py.229: process: 0.56945 | load: 0.00273 [I 2022-10-28 09:26:16] ...trap/engines/engine.py.230: loss: 0.11545 [I 2022-10-28 09:26:16] ...optimizers/trijoint.py.151: total_norm: 17.801117 [I 2022-10-28 09:26:22] ...trap/engines/engine.py.224: train: epoch 118 | batch 150/2304 [I 2022-10-28 09:26:22] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:20:00 [I 2022-10-28 09:26:22] ...trap/engines/engine.py.229: process: 0.55161 | load: 0.00247 [I 2022-10-28 09:26:22] ...trap/engines/engine.py.230: loss: 0.10064 [I 2022-10-28 09:26:22] ...optimizers/trijoint.py.151: total_norm: 10.284952 [I 2022-10-28 09:26:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 160/2304 [I 2022-10-28 09:26:27] ...trap/engines/engine.py.228: elapsed: 0:01:31 | left: 0:20:00 [I 2022-10-28 09:26:27] ...trap/engines/engine.py.229: process: 0.56150 | load: 0.00262 [I 2022-10-28 09:26:27] ...trap/engines/engine.py.230: loss: 0.11286 [I 2022-10-28 09:26:27] ...optimizers/trijoint.py.151: total_norm: 19.019512 [I 2022-10-28 09:26:33] ...trap/engines/engine.py.224: train: epoch 118 | batch 170/2304 [I 2022-10-28 09:26:33] ...trap/engines/engine.py.228: elapsed: 0:01:37 | left: 0:19:59 [I 2022-10-28 09:26:33] ...trap/engines/engine.py.229: process: 0.55081 | load: 0.00242 [I 2022-10-28 09:26:33] ...trap/engines/engine.py.230: loss: 0.16003 [I 2022-10-28 09:26:33] ...optimizers/trijoint.py.151: total_norm: 10.230327 [I 2022-10-28 09:26:38] ...trap/engines/engine.py.224: train: epoch 118 | batch 180/2304 [I 2022-10-28 09:26:38] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:40 [I 2022-10-28 09:26:38] ...trap/engines/engine.py.229: process: 0.55249 | load: 0.00250 [I 2022-10-28 09:26:38] ...trap/engines/engine.py.230: loss: 0.09802 [I 2022-10-28 09:26:38] ...optimizers/trijoint.py.151: total_norm: 11.768656 [I 2022-10-28 09:26:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 190/2304 [I 2022-10-28 09:26:44] ...trap/engines/engine.py.228: elapsed: 0:01:48 | left: 0:19:31 [I 2022-10-28 09:26:44] ...trap/engines/engine.py.229: process: 0.54569 | load: 0.00280 [I 2022-10-28 09:26:44] ...trap/engines/engine.py.230: loss: 0.10750 [I 2022-10-28 09:26:44] ...optimizers/trijoint.py.151: total_norm: 10.340217 [I 2022-10-28 09:26:50] ...trap/engines/engine.py.224: train: epoch 118 | batch 200/2304 [I 2022-10-28 09:26:50] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:41 [I 2022-10-28 09:26:50] ...trap/engines/engine.py.229: process: 0.55734 | load: 0.00245 [I 2022-10-28 09:26:50] ...trap/engines/engine.py.230: loss: 0.10549 [I 2022-10-28 09:26:50] ...optimizers/trijoint.py.151: total_norm: 13.360467 [I 2022-10-28 09:26:55] ...trap/engines/engine.py.224: train: epoch 118 | batch 210/2304 [I 2022-10-28 09:26:55] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:20:03 [I 2022-10-28 09:26:55] ...trap/engines/engine.py.229: process: 0.60598 | load: 0.00244 [I 2022-10-28 09:26:55] ...trap/engines/engine.py.230: loss: 0.13418 [I 2022-10-28 09:26:55] ...optimizers/trijoint.py.151: total_norm: 28.376808 [I 2022-10-28 09:27:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 220/2304 [I 2022-10-28 09:27:01] ...trap/engines/engine.py.228: elapsed: 0:02:05 | left: 0:19:39 [I 2022-10-28 09:27:01] ...trap/engines/engine.py.229: process: 0.55598 | load: 0.00285 [I 2022-10-28 09:27:01] ...trap/engines/engine.py.230: loss: 0.16924 [I 2022-10-28 09:27:01] ...optimizers/trijoint.py.151: total_norm: 16.824471 [I 2022-10-28 09:27:07] ...trap/engines/engine.py.224: train: epoch 118 | batch 230/2304 [I 2022-10-28 09:27:07] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:20:00 [I 2022-10-28 09:27:07] ...trap/engines/engine.py.229: process: 0.56728 | load: 0.00268 [I 2022-10-28 09:27:07] ...trap/engines/engine.py.230: loss: 0.11992 [I 2022-10-28 09:27:07] ...optimizers/trijoint.py.151: total_norm: 17.569452 [I 2022-10-28 09:27:12] ...trap/engines/engine.py.224: train: epoch 118 | batch 240/2304 [I 2022-10-28 09:27:12] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:17 [I 2022-10-28 09:27:12] ...trap/engines/engine.py.229: process: 0.55866 | load: 0.00260 [I 2022-10-28 09:27:12] ...trap/engines/engine.py.230: loss: 0.18267 [I 2022-10-28 09:27:12] ...optimizers/trijoint.py.151: total_norm: 12.734487 [I 2022-10-28 09:27:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 250/2304 [I 2022-10-28 09:27:18] ...trap/engines/engine.py.228: elapsed: 0:02:22 | left: 0:19:08 [I 2022-10-28 09:27:18] ...trap/engines/engine.py.229: process: 0.55974 | load: 0.00245 [I 2022-10-28 09:27:18] ...trap/engines/engine.py.230: loss: 0.08643 [I 2022-10-28 09:27:18] ...optimizers/trijoint.py.151: total_norm: 12.238497 [I 2022-10-28 09:27:24] ...trap/engines/engine.py.224: train: epoch 118 | batch 260/2304 [I 2022-10-28 09:27:24] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:19:05 [I 2022-10-28 09:27:24] ...trap/engines/engine.py.229: process: 0.55911 | load: 0.00245 [I 2022-10-28 09:27:24] ...trap/engines/engine.py.230: loss: 0.11052 [I 2022-10-28 09:27:24] ...optimizers/trijoint.py.151: total_norm: 19.175304 [I 2022-10-28 09:27:29] ...trap/engines/engine.py.224: train: epoch 118 | batch 270/2304 [I 2022-10-28 09:27:29] ...trap/engines/engine.py.228: elapsed: 0:02:33 | left: 0:18:49 [I 2022-10-28 09:27:29] ...trap/engines/engine.py.229: process: 0.55794 | load: 0.00262 [I 2022-10-28 09:27:29] ...trap/engines/engine.py.230: loss: 0.10073 [I 2022-10-28 09:27:29] ...optimizers/trijoint.py.151: total_norm: 13.439948 [I 2022-10-28 09:27:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 280/2304 [I 2022-10-28 09:27:35] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:42 [I 2022-10-28 09:27:35] ...trap/engines/engine.py.229: process: 0.55258 | load: 0.00248 [I 2022-10-28 09:27:35] ...trap/engines/engine.py.230: loss: 0.08784 [I 2022-10-28 09:27:35] ...optimizers/trijoint.py.151: total_norm: 9.672312 [I 2022-10-28 09:27:40] ...trap/engines/engine.py.224: train: epoch 118 | batch 290/2304 [I 2022-10-28 09:27:40] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:51 [I 2022-10-28 09:27:40] ...trap/engines/engine.py.229: process: 0.56475 | load: 0.00262 [I 2022-10-28 09:27:40] ...trap/engines/engine.py.230: loss: 0.16233 [I 2022-10-28 09:27:40] ...optimizers/trijoint.py.151: total_norm: 19.159012 [I 2022-10-28 09:27:46] ...trap/engines/engine.py.224: train: epoch 118 | batch 300/2304 [I 2022-10-28 09:27:46] ...trap/engines/engine.py.228: elapsed: 0:02:50 | left: 0:18:38 [I 2022-10-28 09:27:46] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00264 [I 2022-10-28 09:27:46] ...trap/engines/engine.py.230: loss: 0.08695 [I 2022-10-28 09:27:46] ...optimizers/trijoint.py.151: total_norm: 9.620103 [I 2022-10-28 09:27:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 310/2304 [I 2022-10-28 09:27:52] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:27 [I 2022-10-28 09:27:52] ...trap/engines/engine.py.229: process: 0.54393 | load: 0.00264 [I 2022-10-28 09:27:52] ...trap/engines/engine.py.230: loss: 0.10410 [I 2022-10-28 09:27:52] ...optimizers/trijoint.py.151: total_norm: 12.254667 [I 2022-10-28 09:27:57] ...trap/engines/engine.py.224: train: epoch 118 | batch 320/2304 [I 2022-10-28 09:27:57] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:29 [I 2022-10-28 09:27:57] ...trap/engines/engine.py.229: process: 0.56048 | load: 0.00250 [I 2022-10-28 09:27:57] ...trap/engines/engine.py.230: loss: 0.09513 [I 2022-10-28 09:27:57] ...optimizers/trijoint.py.151: total_norm: 12.538463 [I 2022-10-28 09:28:03] ...trap/engines/engine.py.224: train: epoch 118 | batch 330/2304 [I 2022-10-28 09:28:03] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:11 [I 2022-10-28 09:28:03] ...trap/engines/engine.py.229: process: 0.54351 | load: 0.00244 [I 2022-10-28 09:28:03] ...trap/engines/engine.py.230: loss: 0.09680 [I 2022-10-28 09:28:03] ...optimizers/trijoint.py.151: total_norm: 11.615390 [I 2022-10-28 09:28:08] ...trap/engines/engine.py.224: train: epoch 118 | batch 340/2304 [I 2022-10-28 09:28:08] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:09 [I 2022-10-28 09:28:08] ...trap/engines/engine.py.229: process: 0.54971 | load: 0.00244 [I 2022-10-28 09:28:08] ...trap/engines/engine.py.230: loss: 0.12021 [I 2022-10-28 09:28:08] ...optimizers/trijoint.py.151: total_norm: 12.374198 [I 2022-10-28 09:28:14] ...trap/engines/engine.py.224: train: epoch 118 | batch 350/2304 [I 2022-10-28 09:28:14] ...trap/engines/engine.py.228: elapsed: 0:03:18 | left: 0:18:01 [I 2022-10-28 09:28:14] ...trap/engines/engine.py.229: process: 0.55253 | load: 0.00239 [I 2022-10-28 09:28:14] ...trap/engines/engine.py.230: loss: 0.07977 [I 2022-10-28 09:28:14] ...optimizers/trijoint.py.151: total_norm: 11.140732 [I 2022-10-28 09:28:20] ...trap/engines/engine.py.224: train: epoch 118 | batch 360/2304 [I 2022-10-28 09:28:20] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:00 [I 2022-10-28 09:28:20] ...trap/engines/engine.py.229: process: 0.55997 | load: 0.00241 [I 2022-10-28 09:28:20] ...trap/engines/engine.py.230: loss: 0.10958 [I 2022-10-28 09:28:20] ...optimizers/trijoint.py.151: total_norm: 13.586097 [I 2022-10-28 09:28:25] ...trap/engines/engine.py.224: train: epoch 118 | batch 370/2304 [I 2022-10-28 09:28:25] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:18:18 [I 2022-10-28 09:28:25] ...trap/engines/engine.py.229: process: 0.56441 | load: 0.00244 [I 2022-10-28 09:28:25] ...trap/engines/engine.py.230: loss: 0.08053 [I 2022-10-28 09:28:25] ...optimizers/trijoint.py.151: total_norm: 16.787970 [I 2022-10-28 09:28:31] ...trap/engines/engine.py.224: train: epoch 118 | batch 380/2304 [I 2022-10-28 09:28:31] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:59 [I 2022-10-28 09:28:31] ...trap/engines/engine.py.229: process: 0.55354 | load: 0.00254 [I 2022-10-28 09:28:31] ...trap/engines/engine.py.230: loss: 0.09592 [I 2022-10-28 09:28:31] ...optimizers/trijoint.py.151: total_norm: 19.490549 [I 2022-10-28 09:28:37] ...trap/engines/engine.py.224: train: epoch 118 | batch 390/2304 [I 2022-10-28 09:28:37] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:50 [I 2022-10-28 09:28:37] ...trap/engines/engine.py.229: process: 0.55811 | load: 0.00248 [I 2022-10-28 09:28:37] ...trap/engines/engine.py.230: loss: 0.11203 [I 2022-10-28 09:28:37] ...optimizers/trijoint.py.151: total_norm: 8.239923 [I 2022-10-28 09:28:42] ...trap/engines/engine.py.224: train: epoch 118 | batch 400/2304 [I 2022-10-28 09:28:42] ...trap/engines/engine.py.228: elapsed: 0:03:46 | left: 0:17:38 [I 2022-10-28 09:28:42] ...trap/engines/engine.py.229: process: 0.55353 | load: 0.00245 [I 2022-10-28 09:28:42] ...trap/engines/engine.py.230: loss: 0.13048 [I 2022-10-28 09:28:42] ...optimizers/trijoint.py.151: total_norm: 16.088934 [I 2022-10-28 09:28:48] ...trap/engines/engine.py.224: train: epoch 118 | batch 410/2304 [I 2022-10-28 09:28:48] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:41 [I 2022-10-28 09:28:48] ...trap/engines/engine.py.229: process: 0.55613 | load: 0.00244 [I 2022-10-28 09:28:48] ...trap/engines/engine.py.230: loss: 0.06481 [I 2022-10-28 09:28:48] ...optimizers/trijoint.py.151: total_norm: 10.011934 [I 2022-10-28 09:28:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 420/2304 [I 2022-10-28 09:28:53] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:28 [I 2022-10-28 09:28:53] ...trap/engines/engine.py.229: process: 0.54689 | load: 0.00267 [I 2022-10-28 09:28:53] ...trap/engines/engine.py.230: loss: 0.15366 [I 2022-10-28 09:28:53] ...optimizers/trijoint.py.151: total_norm: 12.701613 [I 2022-10-28 09:28:59] ...trap/engines/engine.py.224: train: epoch 118 | batch 430/2304 [I 2022-10-28 09:28:59] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:23 [I 2022-10-28 09:28:59] ...trap/engines/engine.py.229: process: 0.53925 | load: 0.00244 [I 2022-10-28 09:28:59] ...trap/engines/engine.py.230: loss: 0.12072 [I 2022-10-28 09:28:59] ...optimizers/trijoint.py.151: total_norm: 25.895935 [I 2022-10-28 09:29:05] ...trap/engines/engine.py.224: train: epoch 118 | batch 440/2304 [I 2022-10-28 09:29:05] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:25 [I 2022-10-28 09:29:05] ...trap/engines/engine.py.229: process: 0.56412 | load: 0.00264 [I 2022-10-28 09:29:05] ...trap/engines/engine.py.230: loss: 0.16675 [I 2022-10-28 09:29:05] ...optimizers/trijoint.py.151: total_norm: 17.215565 [I 2022-10-28 09:29:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 450/2304 [I 2022-10-28 09:29:10] ...trap/engines/engine.py.228: elapsed: 0:04:14 | left: 0:17:13 [I 2022-10-28 09:29:10] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00262 [I 2022-10-28 09:29:10] ...trap/engines/engine.py.230: loss: 0.09765 [I 2022-10-28 09:29:10] ...optimizers/trijoint.py.151: total_norm: 9.580102 [I 2022-10-28 09:29:16] ...trap/engines/engine.py.224: train: epoch 118 | batch 460/2304 [I 2022-10-28 09:29:16] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:15 [I 2022-10-28 09:29:16] ...trap/engines/engine.py.229: process: 0.54962 | load: 0.00253 [I 2022-10-28 09:29:16] ...trap/engines/engine.py.230: loss: 0.11095 [I 2022-10-28 09:29:16] ...optimizers/trijoint.py.151: total_norm: 16.916731 [I 2022-10-28 09:29:21] ...trap/engines/engine.py.224: train: epoch 118 | batch 470/2304 [I 2022-10-28 09:29:21] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:00 [I 2022-10-28 09:29:21] ...trap/engines/engine.py.229: process: 0.54862 | load: 0.00244 [I 2022-10-28 09:29:21] ...trap/engines/engine.py.230: loss: 0.11321 [I 2022-10-28 09:29:21] ...optimizers/trijoint.py.151: total_norm: 32.126560 [I 2022-10-28 09:29:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 480/2304 [I 2022-10-28 09:29:27] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:17:07 [I 2022-10-28 09:29:27] ...trap/engines/engine.py.229: process: 0.55851 | load: 0.00255 [I 2022-10-28 09:29:27] ...trap/engines/engine.py.230: loss: 0.09381 [I 2022-10-28 09:29:27] ...optimizers/trijoint.py.151: total_norm: 10.451757 [I 2022-10-28 09:29:33] ...trap/engines/engine.py.224: train: epoch 118 | batch 490/2304 [I 2022-10-28 09:29:33] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:57 [I 2022-10-28 09:29:33] ...trap/engines/engine.py.229: process: 0.56805 | load: 0.00254 [I 2022-10-28 09:29:33] ...trap/engines/engine.py.230: loss: 0.12490 [I 2022-10-28 09:29:33] ...optimizers/trijoint.py.151: total_norm: 16.993439 [I 2022-10-28 09:29:38] ...trap/engines/engine.py.224: train: epoch 118 | batch 500/2304 [I 2022-10-28 09:29:38] ...trap/engines/engine.py.228: elapsed: 0:04:42 | left: 0:16:52 [I 2022-10-28 09:29:38] ...trap/engines/engine.py.229: process: 0.56400 | load: 0.00249 [I 2022-10-28 09:29:38] ...trap/engines/engine.py.230: loss: 0.15475 [I 2022-10-28 09:29:38] ...optimizers/trijoint.py.151: total_norm: 15.011438 [I 2022-10-28 09:29:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 510/2304 [I 2022-10-28 09:29:44] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:39 [I 2022-10-28 09:29:44] ...trap/engines/engine.py.229: process: 0.55825 | load: 0.00242 [I 2022-10-28 09:29:44] ...trap/engines/engine.py.230: loss: 0.09707 [I 2022-10-28 09:29:44] ...optimizers/trijoint.py.151: total_norm: 22.569466 [I 2022-10-28 09:29:49] ...trap/engines/engine.py.224: train: epoch 118 | batch 520/2304 [I 2022-10-28 09:29:49] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:32 [I 2022-10-28 09:29:49] ...trap/engines/engine.py.229: process: 0.55888 | load: 0.00243 [I 2022-10-28 09:29:49] ...trap/engines/engine.py.230: loss: 0.14453 [I 2022-10-28 09:29:49] ...optimizers/trijoint.py.151: total_norm: 12.436913 [I 2022-10-28 09:29:55] ...trap/engines/engine.py.224: train: epoch 118 | batch 530/2304 [I 2022-10-28 09:29:55] ...trap/engines/engine.py.228: elapsed: 0:04:59 | left: 0:16:28 [I 2022-10-28 09:29:55] ...trap/engines/engine.py.229: process: 0.55089 | load: 0.00241 [I 2022-10-28 09:29:55] ...trap/engines/engine.py.230: loss: 0.07475 [I 2022-10-28 09:29:55] ...optimizers/trijoint.py.151: total_norm: 13.895535 [I 2022-10-28 09:30:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 540/2304 [I 2022-10-28 09:30:01] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:25 [I 2022-10-28 09:30:01] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00242 [I 2022-10-28 09:30:01] ...trap/engines/engine.py.230: loss: 0.12930 [I 2022-10-28 09:30:01] ...optimizers/trijoint.py.151: total_norm: 26.109707 [I 2022-10-28 09:30:06] ...trap/engines/engine.py.224: train: epoch 118 | batch 550/2304 [I 2022-10-28 09:30:06] ...trap/engines/engine.py.228: elapsed: 0:05:10 | left: 0:16:15 [I 2022-10-28 09:30:06] ...trap/engines/engine.py.229: process: 0.55251 | load: 0.00241 [I 2022-10-28 09:30:06] ...trap/engines/engine.py.230: loss: 0.11235 [I 2022-10-28 09:30:06] ...optimizers/trijoint.py.151: total_norm: 11.552972 [I 2022-10-28 09:30:12] ...trap/engines/engine.py.224: train: epoch 118 | batch 560/2304 [I 2022-10-28 09:30:12] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:17 [I 2022-10-28 09:30:12] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00242 [I 2022-10-28 09:30:12] ...trap/engines/engine.py.230: loss: 0.13557 [I 2022-10-28 09:30:12] ...optimizers/trijoint.py.151: total_norm: 123.040306 [I 2022-10-28 09:30:17] ...trap/engines/engine.py.224: train: epoch 118 | batch 570/2304 [I 2022-10-28 09:30:17] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:07 [I 2022-10-28 09:30:17] ...trap/engines/engine.py.229: process: 0.55901 | load: 0.00241 [I 2022-10-28 09:30:17] ...trap/engines/engine.py.230: loss: 0.11822 [I 2022-10-28 09:30:17] ...optimizers/trijoint.py.151: total_norm: 9.611098 [I 2022-10-28 09:30:23] ...trap/engines/engine.py.224: train: epoch 118 | batch 580/2304 [I 2022-10-28 09:30:23] ...trap/engines/engine.py.228: elapsed: 0:05:27 | left: 0:16:03 [I 2022-10-28 09:30:23] ...trap/engines/engine.py.229: process: 0.56517 | load: 0.00277 [I 2022-10-28 09:30:23] ...trap/engines/engine.py.230: loss: 0.09663 [I 2022-10-28 09:30:23] ...optimizers/trijoint.py.151: total_norm: 35.273308 [I 2022-10-28 09:30:29] ...trap/engines/engine.py.224: train: epoch 118 | batch 590/2304 [I 2022-10-28 09:30:29] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:15:55 [I 2022-10-28 09:30:29] ...trap/engines/engine.py.229: process: 0.55844 | load: 0.00261 [I 2022-10-28 09:30:29] ...trap/engines/engine.py.230: loss: 0.07518 [I 2022-10-28 09:30:29] ...optimizers/trijoint.py.151: total_norm: 10.636587 [I 2022-10-28 09:30:34] ...trap/engines/engine.py.224: train: epoch 118 | batch 600/2304 [I 2022-10-28 09:30:34] ...trap/engines/engine.py.228: elapsed: 0:05:38 | left: 0:16:18 [I 2022-10-28 09:30:34] ...trap/engines/engine.py.229: process: 0.63980 | load: 0.00249 [I 2022-10-28 09:30:34] ...trap/engines/engine.py.230: loss: 0.07861 [I 2022-10-28 09:30:34] ...optimizers/trijoint.py.151: total_norm: 22.170027 [I 2022-10-28 09:30:40] ...trap/engines/engine.py.224: train: epoch 118 | batch 610/2304 [I 2022-10-28 09:30:40] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:52 [I 2022-10-28 09:30:40] ...trap/engines/engine.py.229: process: 0.56191 | load: 0.00250 [I 2022-10-28 09:30:40] ...trap/engines/engine.py.230: loss: 0.18595 [I 2022-10-28 09:30:40] ...optimizers/trijoint.py.151: total_norm: 16.282984 [I 2022-10-28 09:30:46] ...trap/engines/engine.py.224: train: epoch 118 | batch 620/2304 [I 2022-10-28 09:30:46] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:45 [I 2022-10-28 09:30:46] ...trap/engines/engine.py.229: process: 0.55566 | load: 0.00253 [I 2022-10-28 09:30:46] ...trap/engines/engine.py.230: loss: 0.11632 [I 2022-10-28 09:30:46] ...optimizers/trijoint.py.151: total_norm: 17.708975 [I 2022-10-28 09:30:51] ...trap/engines/engine.py.224: train: epoch 118 | batch 630/2304 [I 2022-10-28 09:30:51] ...trap/engines/engine.py.228: elapsed: 0:05:55 | left: 0:15:28 [I 2022-10-28 09:30:51] ...trap/engines/engine.py.229: process: 0.54848 | load: 0.00264 [I 2022-10-28 09:30:51] ...trap/engines/engine.py.230: loss: 0.14092 [I 2022-10-28 09:30:51] ...optimizers/trijoint.py.151: total_norm: 16.108721 [I 2022-10-28 09:30:57] ...trap/engines/engine.py.224: train: epoch 118 | batch 640/2304 [I 2022-10-28 09:30:57] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:23 [I 2022-10-28 09:30:57] ...trap/engines/engine.py.229: process: 0.55972 | load: 0.00241 [I 2022-10-28 09:30:57] ...trap/engines/engine.py.230: loss: 0.07238 [I 2022-10-28 09:30:57] ...optimizers/trijoint.py.151: total_norm: 11.734694 [I 2022-10-28 09:31:02] ...trap/engines/engine.py.224: train: epoch 118 | batch 650/2304 [I 2022-10-28 09:31:02] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:25 [I 2022-10-28 09:31:02] ...trap/engines/engine.py.229: process: 0.55800 | load: 0.00242 [I 2022-10-28 09:31:02] ...trap/engines/engine.py.230: loss: 0.13445 [I 2022-10-28 09:31:02] ...optimizers/trijoint.py.151: total_norm: 15.706628 [I 2022-10-28 09:31:08] ...trap/engines/engine.py.224: train: epoch 118 | batch 660/2304 [I 2022-10-28 09:31:08] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:22 [I 2022-10-28 09:31:08] ...trap/engines/engine.py.229: process: 0.55575 | load: 0.00250 [I 2022-10-28 09:31:08] ...trap/engines/engine.py.230: loss: 0.10267 [I 2022-10-28 09:31:08] ...optimizers/trijoint.py.151: total_norm: 24.621866 [I 2022-10-28 09:31:14] ...trap/engines/engine.py.224: train: epoch 118 | batch 670/2304 [I 2022-10-28 09:31:14] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:14 [I 2022-10-28 09:31:14] ...trap/engines/engine.py.229: process: 0.55781 | load: 0.00240 [I 2022-10-28 09:31:14] ...trap/engines/engine.py.230: loss: 0.15039 [I 2022-10-28 09:31:14] ...optimizers/trijoint.py.151: total_norm: 16.642483 [I 2022-10-28 09:31:19] ...trap/engines/engine.py.224: train: epoch 118 | batch 680/2304 [I 2022-10-28 09:31:19] ...trap/engines/engine.py.228: elapsed: 0:06:23 | left: 0:15:03 [I 2022-10-28 09:31:19] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00250 [I 2022-10-28 09:31:19] ...trap/engines/engine.py.230: loss: 0.09551 [I 2022-10-28 09:31:19] ...optimizers/trijoint.py.151: total_norm: 13.570786 [I 2022-10-28 09:31:25] ...trap/engines/engine.py.224: train: epoch 118 | batch 690/2304 [I 2022-10-28 09:31:25] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:14:54 [I 2022-10-28 09:31:25] ...trap/engines/engine.py.229: process: 0.54774 | load: 0.00250 [I 2022-10-28 09:31:25] ...trap/engines/engine.py.230: loss: 0.11264 [I 2022-10-28 09:31:25] ...optimizers/trijoint.py.151: total_norm: 16.111292 [I 2022-10-28 09:31:30] ...trap/engines/engine.py.224: train: epoch 118 | batch 700/2304 [I 2022-10-28 09:31:30] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:14:54 [I 2022-10-28 09:31:30] ...trap/engines/engine.py.229: process: 0.55028 | load: 0.00248 [I 2022-10-28 09:31:30] ...trap/engines/engine.py.230: loss: 0.10729 [I 2022-10-28 09:31:30] ...optimizers/trijoint.py.151: total_norm: 16.264729 [I 2022-10-28 09:31:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 710/2304 [I 2022-10-28 09:31:36] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:58 [I 2022-10-28 09:31:36] ...trap/engines/engine.py.229: process: 0.56003 | load: 0.00247 [I 2022-10-28 09:31:36] ...trap/engines/engine.py.230: loss: 0.11219 [I 2022-10-28 09:31:36] ...optimizers/trijoint.py.151: total_norm: 13.640623 [I 2022-10-28 09:31:42] ...trap/engines/engine.py.224: train: epoch 118 | batch 720/2304 [I 2022-10-28 09:31:42] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:43 [I 2022-10-28 09:31:42] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00246 [I 2022-10-28 09:31:42] ...trap/engines/engine.py.230: loss: 0.18258 [I 2022-10-28 09:31:42] ...optimizers/trijoint.py.151: total_norm: 38.155201 [I 2022-10-28 09:31:47] ...trap/engines/engine.py.224: train: epoch 118 | batch 730/2304 [I 2022-10-28 09:31:47] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:34 [I 2022-10-28 09:31:47] ...trap/engines/engine.py.229: process: 0.55518 | load: 0.00292 [I 2022-10-28 09:31:47] ...trap/engines/engine.py.230: loss: 0.08454 [I 2022-10-28 09:31:47] ...optimizers/trijoint.py.151: total_norm: 13.456043 [I 2022-10-28 09:31:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 740/2304 [I 2022-10-28 09:31:53] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:42 [I 2022-10-28 09:31:53] ...trap/engines/engine.py.229: process: 0.56955 | load: 0.00261 [I 2022-10-28 09:31:53] ...trap/engines/engine.py.230: loss: 0.11443 [I 2022-10-28 09:31:53] ...optimizers/trijoint.py.151: total_norm: 17.878716 [I 2022-10-28 09:31:58] ...trap/engines/engine.py.224: train: epoch 118 | batch 750/2304 [I 2022-10-28 09:31:58] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:24 [I 2022-10-28 09:31:58] ...trap/engines/engine.py.229: process: 0.54505 | load: 0.00245 [I 2022-10-28 09:31:58] ...trap/engines/engine.py.230: loss: 0.09231 [I 2022-10-28 09:31:58] ...optimizers/trijoint.py.151: total_norm: 9.932071 [I 2022-10-28 09:32:04] ...trap/engines/engine.py.224: train: epoch 118 | batch 760/2304 [I 2022-10-28 09:32:04] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:26 [I 2022-10-28 09:32:04] ...trap/engines/engine.py.229: process: 0.55555 | load: 0.00245 [I 2022-10-28 09:32:04] ...trap/engines/engine.py.230: loss: 0.14112 [I 2022-10-28 09:32:04] ...optimizers/trijoint.py.151: total_norm: 9.996699 [I 2022-10-28 09:32:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 770/2304 [I 2022-10-28 09:32:10] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:25 [I 2022-10-28 09:32:10] ...trap/engines/engine.py.229: process: 0.55982 | load: 0.00245 [I 2022-10-28 09:32:10] ...trap/engines/engine.py.230: loss: 0.09361 [I 2022-10-28 09:32:10] ...optimizers/trijoint.py.151: total_norm: 14.235699 [I 2022-10-28 09:32:15] ...trap/engines/engine.py.224: train: epoch 118 | batch 780/2304 [I 2022-10-28 09:32:15] ...trap/engines/engine.py.228: elapsed: 0:07:20 | left: 0:14:09 [I 2022-10-28 09:32:15] ...trap/engines/engine.py.229: process: 0.56039 | load: 0.00246 [I 2022-10-28 09:32:15] ...trap/engines/engine.py.230: loss: 0.12299 [I 2022-10-28 09:32:15] ...optimizers/trijoint.py.151: total_norm: 14.928342 [I 2022-10-28 09:32:21] ...trap/engines/engine.py.224: train: epoch 118 | batch 790/2304 [I 2022-10-28 09:32:21] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:06 [I 2022-10-28 09:32:21] ...trap/engines/engine.py.229: process: 0.55917 | load: 0.00246 [I 2022-10-28 09:32:21] ...trap/engines/engine.py.230: loss: 0.16300 [I 2022-10-28 09:32:21] ...optimizers/trijoint.py.151: total_norm: 31.534712 [I 2022-10-28 09:32:26] ...trap/engines/engine.py.224: train: epoch 118 | batch 800/2304 [I 2022-10-28 09:32:26] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:13:57 [I 2022-10-28 09:32:26] ...trap/engines/engine.py.229: process: 0.54679 | load: 0.00247 [I 2022-10-28 09:32:26] ...trap/engines/engine.py.230: loss: 0.09496 [I 2022-10-28 09:32:26] ...optimizers/trijoint.py.151: total_norm: 14.733963 [I 2022-10-28 09:32:32] ...trap/engines/engine.py.224: train: epoch 118 | batch 810/2304 [I 2022-10-28 09:32:32] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:53 [I 2022-10-28 09:32:32] ...trap/engines/engine.py.229: process: 0.54860 | load: 0.00251 [I 2022-10-28 09:32:32] ...trap/engines/engine.py.230: loss: 0.11423 [I 2022-10-28 09:32:32] ...optimizers/trijoint.py.151: total_norm: 29.067568 [I 2022-10-28 09:32:38] ...trap/engines/engine.py.224: train: epoch 118 | batch 820/2304 [I 2022-10-28 09:32:38] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:13:51 [I 2022-10-28 09:32:38] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00250 [I 2022-10-28 09:32:38] ...trap/engines/engine.py.230: loss: 0.08995 [I 2022-10-28 09:32:38] ...optimizers/trijoint.py.151: total_norm: 9.389431 [I 2022-10-28 09:32:43] ...trap/engines/engine.py.224: train: epoch 118 | batch 830/2304 [I 2022-10-28 09:32:43] ...trap/engines/engine.py.228: elapsed: 0:07:48 | left: 0:13:44 [I 2022-10-28 09:32:43] ...trap/engines/engine.py.229: process: 0.54988 | load: 0.00245 [I 2022-10-28 09:32:43] ...trap/engines/engine.py.230: loss: 0.11773 [I 2022-10-28 09:32:43] ...optimizers/trijoint.py.151: total_norm: 15.919086 [I 2022-10-28 09:32:49] ...trap/engines/engine.py.224: train: epoch 118 | batch 840/2304 [I 2022-10-28 09:32:49] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:34 [I 2022-10-28 09:32:49] ...trap/engines/engine.py.229: process: 0.54917 | load: 0.00243 [I 2022-10-28 09:32:49] ...trap/engines/engine.py.230: loss: 0.10599 [I 2022-10-28 09:32:49] ...optimizers/trijoint.py.151: total_norm: 13.195436 [I 2022-10-28 09:32:55] ...trap/engines/engine.py.224: train: epoch 118 | batch 850/2304 [I 2022-10-28 09:32:55] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:37 [I 2022-10-28 09:32:55] ...trap/engines/engine.py.229: process: 0.56771 | load: 0.00242 [I 2022-10-28 09:32:55] ...trap/engines/engine.py.230: loss: 0.12159 [I 2022-10-28 09:32:55] ...optimizers/trijoint.py.151: total_norm: 11.318155 [I 2022-10-28 09:33:00] ...trap/engines/engine.py.224: train: epoch 118 | batch 860/2304 [I 2022-10-28 09:33:00] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:23 [I 2022-10-28 09:33:00] ...trap/engines/engine.py.229: process: 0.54930 | load: 0.00244 [I 2022-10-28 09:33:00] ...trap/engines/engine.py.230: loss: 0.10812 [I 2022-10-28 09:33:00] ...optimizers/trijoint.py.151: total_norm: 13.692184 [I 2022-10-28 09:33:06] ...trap/engines/engine.py.224: train: epoch 118 | batch 870/2304 [I 2022-10-28 09:33:06] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:17 [I 2022-10-28 09:33:06] ...trap/engines/engine.py.229: process: 0.55617 | load: 0.00242 [I 2022-10-28 09:33:06] ...trap/engines/engine.py.230: loss: 0.10606 [I 2022-10-28 09:33:06] ...optimizers/trijoint.py.151: total_norm: 11.262517 [I 2022-10-28 09:33:11] ...trap/engines/engine.py.224: train: epoch 118 | batch 880/2304 [I 2022-10-28 09:33:11] ...trap/engines/engine.py.228: elapsed: 0:08:15 | left: 0:13:12 [I 2022-10-28 09:33:11] ...trap/engines/engine.py.229: process: 0.55374 | load: 0.00242 [I 2022-10-28 09:33:11] ...trap/engines/engine.py.230: loss: 0.12566 [I 2022-10-28 09:33:11] ...optimizers/trijoint.py.151: total_norm: 7.362154 [I 2022-10-28 09:33:17] ...trap/engines/engine.py.224: train: epoch 118 | batch 890/2304 [I 2022-10-28 09:33:17] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:07 [I 2022-10-28 09:33:17] ...trap/engines/engine.py.229: process: 0.56229 | load: 0.00244 [I 2022-10-28 09:33:17] ...trap/engines/engine.py.230: loss: 0.10910 [I 2022-10-28 09:33:17] ...optimizers/trijoint.py.151: total_norm: 11.987410 [I 2022-10-28 09:33:22] ...trap/engines/engine.py.224: train: epoch 118 | batch 900/2304 [I 2022-10-28 09:33:22] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:13:08 [I 2022-10-28 09:33:22] ...trap/engines/engine.py.229: process: 0.56545 | load: 0.00249 [I 2022-10-28 09:33:22] ...trap/engines/engine.py.230: loss: 0.12393 [I 2022-10-28 09:33:22] ...optimizers/trijoint.py.151: total_norm: 11.722827 [I 2022-10-28 09:33:28] ...trap/engines/engine.py.224: train: epoch 118 | batch 910/2304 [I 2022-10-28 09:33:28] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:12:56 [I 2022-10-28 09:33:28] ...trap/engines/engine.py.229: process: 0.55042 | load: 0.00246 [I 2022-10-28 09:33:28] ...trap/engines/engine.py.230: loss: 0.10084 [I 2022-10-28 09:33:28] ...optimizers/trijoint.py.151: total_norm: 10.492304 [I 2022-10-28 09:33:34] ...trap/engines/engine.py.224: train: epoch 118 | batch 920/2304 [I 2022-10-28 09:33:34] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:54 [I 2022-10-28 09:33:34] ...trap/engines/engine.py.229: process: 0.56336 | load: 0.00245 [I 2022-10-28 09:33:34] ...trap/engines/engine.py.230: loss: 0.11236 [I 2022-10-28 09:33:34] ...optimizers/trijoint.py.151: total_norm: 12.468744 [I 2022-10-28 09:33:39] ...trap/engines/engine.py.224: train: epoch 118 | batch 930/2304 [I 2022-10-28 09:33:39] ...trap/engines/engine.py.228: elapsed: 0:08:43 | left: 0:12:46 [I 2022-10-28 09:33:39] ...trap/engines/engine.py.229: process: 0.55408 | load: 0.00241 [I 2022-10-28 09:33:39] ...trap/engines/engine.py.230: loss: 0.10346 [I 2022-10-28 09:33:39] ...optimizers/trijoint.py.151: total_norm: 31.058395 [I 2022-10-28 09:33:45] ...trap/engines/engine.py.224: train: epoch 118 | batch 940/2304 [I 2022-10-28 09:33:45] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:30 [I 2022-10-28 09:33:45] ...trap/engines/engine.py.229: process: 0.52848 | load: 0.00243 [I 2022-10-28 09:33:45] ...trap/engines/engine.py.230: loss: 0.12743 [I 2022-10-28 09:33:45] ...optimizers/trijoint.py.151: total_norm: 32.371330 [I 2022-10-28 09:33:50] ...trap/engines/engine.py.224: train: epoch 118 | batch 950/2304 [I 2022-10-28 09:33:50] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:30 [I 2022-10-28 09:33:50] ...trap/engines/engine.py.229: process: 0.55963 | load: 0.00243 [I 2022-10-28 09:33:50] ...trap/engines/engine.py.230: loss: 0.12658 [I 2022-10-28 09:33:50] ...optimizers/trijoint.py.151: total_norm: 26.409801 [I 2022-10-28 09:33:56] ...trap/engines/engine.py.224: train: epoch 118 | batch 960/2304 [I 2022-10-28 09:33:56] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:29 [I 2022-10-28 09:33:56] ...trap/engines/engine.py.229: process: 0.55607 | load: 0.00243 [I 2022-10-28 09:33:56] ...trap/engines/engine.py.230: loss: 0.17788 [I 2022-10-28 09:33:56] ...optimizers/trijoint.py.151: total_norm: 14.812486 [I 2022-10-28 09:34:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 970/2304 [I 2022-10-28 09:34:01] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:23 [I 2022-10-28 09:34:01] ...trap/engines/engine.py.229: process: 0.56464 | load: 0.00293 [I 2022-10-28 09:34:01] ...trap/engines/engine.py.230: loss: 0.13589 [I 2022-10-28 09:34:01] ...optimizers/trijoint.py.151: total_norm: 21.578789 [I 2022-10-28 09:34:07] ...trap/engines/engine.py.224: train: epoch 118 | batch 980/2304 [I 2022-10-28 09:34:07] ...trap/engines/engine.py.228: elapsed: 0:09:11 | left: 0:12:15 [I 2022-10-28 09:34:07] ...trap/engines/engine.py.229: process: 0.54660 | load: 0.00245 [I 2022-10-28 09:34:07] ...trap/engines/engine.py.230: loss: 0.17191 [I 2022-10-28 09:34:07] ...optimizers/trijoint.py.151: total_norm: 23.092632 [I 2022-10-28 09:34:13] ...trap/engines/engine.py.224: train: epoch 118 | batch 990/2304 [I 2022-10-28 09:34:13] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:17 [I 2022-10-28 09:34:13] ...trap/engines/engine.py.229: process: 0.55501 | load: 0.00246 [I 2022-10-28 09:34:13] ...trap/engines/engine.py.230: loss: 0.08219 [I 2022-10-28 09:34:13] ...optimizers/trijoint.py.151: total_norm: 13.043941 [I 2022-10-28 09:34:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 1000/2304 [I 2022-10-28 09:34:18] ...trap/engines/engine.py.228: elapsed: 0:09:22 | left: 0:12:11 [I 2022-10-28 09:34:18] ...trap/engines/engine.py.229: process: 0.56109 | load: 0.00242 [I 2022-10-28 09:34:18] ...trap/engines/engine.py.230: loss: 0.12466 [I 2022-10-28 09:34:18] ...optimizers/trijoint.py.151: total_norm: 12.814346 [I 2022-10-28 09:34:24] ...trap/engines/engine.py.224: train: epoch 118 | batch 1010/2304 [I 2022-10-28 09:34:24] ...trap/engines/engine.py.228: elapsed: 0:09:28 | left: 0:12:00 [I 2022-10-28 09:34:24] ...trap/engines/engine.py.229: process: 0.55941 | load: 0.00243 [I 2022-10-28 09:34:24] ...trap/engines/engine.py.230: loss: 0.17825 [I 2022-10-28 09:34:24] ...optimizers/trijoint.py.151: total_norm: 57.505733 [I 2022-10-28 09:34:29] ...trap/engines/engine.py.224: train: epoch 118 | batch 1020/2304 [I 2022-10-28 09:34:29] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:52 [I 2022-10-28 09:34:29] ...trap/engines/engine.py.229: process: 0.55096 | load: 0.00242 [I 2022-10-28 09:34:29] ...trap/engines/engine.py.230: loss: 0.11961 [I 2022-10-28 09:34:29] ...optimizers/trijoint.py.151: total_norm: 12.057854 [I 2022-10-28 09:34:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 1030/2304 [I 2022-10-28 09:34:35] ...trap/engines/engine.py.228: elapsed: 0:09:39 | left: 0:11:43 [I 2022-10-28 09:34:35] ...trap/engines/engine.py.229: process: 0.55367 | load: 0.00249 [I 2022-10-28 09:34:35] ...trap/engines/engine.py.230: loss: 0.11767 [I 2022-10-28 09:34:35] ...optimizers/trijoint.py.151: total_norm: 21.789162 [I 2022-10-28 09:34:41] ...trap/engines/engine.py.224: train: epoch 118 | batch 1040/2304 [I 2022-10-28 09:34:41] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:39 [I 2022-10-28 09:34:41] ...trap/engines/engine.py.229: process: 0.55084 | load: 0.00246 [I 2022-10-28 09:34:41] ...trap/engines/engine.py.230: loss: 0.11856 [I 2022-10-28 09:34:41] ...optimizers/trijoint.py.151: total_norm: 77.440369 [I 2022-10-28 09:34:46] ...trap/engines/engine.py.224: train: epoch 118 | batch 1050/2304 [I 2022-10-28 09:34:46] ...trap/engines/engine.py.228: elapsed: 0:09:50 | left: 0:11:34 [I 2022-10-28 09:34:46] ...trap/engines/engine.py.229: process: 0.55828 | load: 0.00244 [I 2022-10-28 09:34:46] ...trap/engines/engine.py.230: loss: 0.11771 [I 2022-10-28 09:34:46] ...optimizers/trijoint.py.151: total_norm: 7.871247 [I 2022-10-28 09:34:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 1060/2304 [I 2022-10-28 09:34:52] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:32 [I 2022-10-28 09:34:52] ...trap/engines/engine.py.229: process: 0.55048 | load: 0.00240 [I 2022-10-28 09:34:52] ...trap/engines/engine.py.230: loss: 0.10332 [I 2022-10-28 09:34:52] ...optimizers/trijoint.py.151: total_norm: 11.522975 [I 2022-10-28 09:34:57] ...trap/engines/engine.py.224: train: epoch 118 | batch 1070/2304 [I 2022-10-28 09:34:57] ...trap/engines/engine.py.228: elapsed: 0:10:01 | left: 0:11:29 [I 2022-10-28 09:34:57] ...trap/engines/engine.py.229: process: 0.56151 | load: 0.00242 [I 2022-10-28 09:34:57] ...trap/engines/engine.py.230: loss: 0.10648 [I 2022-10-28 09:34:57] ...optimizers/trijoint.py.151: total_norm: 12.638932 [I 2022-10-28 09:35:03] ...trap/engines/engine.py.224: train: epoch 118 | batch 1080/2304 [I 2022-10-28 09:35:03] ...trap/engines/engine.py.228: elapsed: 0:10:07 | left: 0:11:22 [I 2022-10-28 09:35:03] ...trap/engines/engine.py.229: process: 0.55069 | load: 0.00244 [I 2022-10-28 09:35:03] ...trap/engines/engine.py.230: loss: 0.08105 [I 2022-10-28 09:35:03] ...optimizers/trijoint.py.151: total_norm: 14.439100 [I 2022-10-28 09:35:08] ...trap/engines/engine.py.224: train: epoch 118 | batch 1090/2304 [I 2022-10-28 09:35:08] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:12 [I 2022-10-28 09:35:08] ...trap/engines/engine.py.229: process: 0.54957 | load: 0.00260 [I 2022-10-28 09:35:08] ...trap/engines/engine.py.230: loss: 0.09912 [I 2022-10-28 09:35:08] ...optimizers/trijoint.py.151: total_norm: 19.838812 [I 2022-10-28 09:35:14] ...trap/engines/engine.py.224: train: epoch 118 | batch 1100/2304 [I 2022-10-28 09:35:14] ...trap/engines/engine.py.228: elapsed: 0:10:18 | left: 0:11:08 [I 2022-10-28 09:35:14] ...trap/engines/engine.py.229: process: 0.55809 | load: 0.00249 [I 2022-10-28 09:35:14] ...trap/engines/engine.py.230: loss: 0.13444 [I 2022-10-28 09:35:14] ...optimizers/trijoint.py.151: total_norm: 13.864005 [I 2022-10-28 09:35:20] ...trap/engines/engine.py.224: train: epoch 118 | batch 1110/2304 [I 2022-10-28 09:35:20] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:05 [I 2022-10-28 09:35:20] ...trap/engines/engine.py.229: process: 0.56198 | load: 0.00248 [I 2022-10-28 09:35:20] ...trap/engines/engine.py.230: loss: 0.16530 [I 2022-10-28 09:35:20] ...optimizers/trijoint.py.151: total_norm: 26.254745 [I 2022-10-28 09:35:25] ...trap/engines/engine.py.224: train: epoch 118 | batch 1120/2304 [I 2022-10-28 09:35:25] ...trap/engines/engine.py.228: elapsed: 0:10:29 | left: 0:10:53 [I 2022-10-28 09:35:25] ...trap/engines/engine.py.229: process: 0.55384 | load: 0.00245 [I 2022-10-28 09:35:25] ...trap/engines/engine.py.230: loss: 0.09729 [I 2022-10-28 09:35:25] ...optimizers/trijoint.py.151: total_norm: 24.420822 [I 2022-10-28 09:35:31] ...trap/engines/engine.py.224: train: epoch 118 | batch 1130/2304 [I 2022-10-28 09:35:31] ...trap/engines/engine.py.228: elapsed: 0:10:35 | left: 0:10:54 [I 2022-10-28 09:35:31] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00246 [I 2022-10-28 09:35:31] ...trap/engines/engine.py.230: loss: 0.09365 [I 2022-10-28 09:35:31] ...optimizers/trijoint.py.151: total_norm: 12.756349 [I 2022-10-28 09:35:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 1140/2304 [I 2022-10-28 09:35:36] ...trap/engines/engine.py.228: elapsed: 0:10:40 | left: 0:10:46 [I 2022-10-28 09:35:36] ...trap/engines/engine.py.229: process: 0.54958 | load: 0.00246 [I 2022-10-28 09:35:36] ...trap/engines/engine.py.230: loss: 0.10379 [I 2022-10-28 09:35:36] ...optimizers/trijoint.py.151: total_norm: 13.958506 [I 2022-10-28 09:35:42] ...trap/engines/engine.py.224: train: epoch 118 | batch 1150/2304 [I 2022-10-28 09:35:42] ...trap/engines/engine.py.228: elapsed: 0:10:46 | left: 0:10:39 [I 2022-10-28 09:35:42] ...trap/engines/engine.py.229: process: 0.55944 | load: 0.00243 [I 2022-10-28 09:35:42] ...trap/engines/engine.py.230: loss: 0.12914 [I 2022-10-28 09:35:42] ...optimizers/trijoint.py.151: total_norm: 9.622492 [I 2022-10-28 09:35:47] ...trap/engines/engine.py.224: train: epoch 118 | batch 1160/2304 [I 2022-10-28 09:35:47] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:33 [I 2022-10-28 09:35:47] ...trap/engines/engine.py.229: process: 0.54684 | load: 0.00245 [I 2022-10-28 09:35:47] ...trap/engines/engine.py.230: loss: 0.18108 [I 2022-10-28 09:35:47] ...optimizers/trijoint.py.151: total_norm: 29.262365 [I 2022-10-28 09:35:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 1170/2304 [I 2022-10-28 09:35:53] ...trap/engines/engine.py.228: elapsed: 0:10:57 | left: 0:10:31 [I 2022-10-28 09:35:53] ...trap/engines/engine.py.229: process: 0.55748 | load: 0.00244 [I 2022-10-28 09:35:53] ...trap/engines/engine.py.230: loss: 0.12472 [I 2022-10-28 09:35:53] ...optimizers/trijoint.py.151: total_norm: 56.752426 [I 2022-10-28 09:35:59] ...trap/engines/engine.py.224: train: epoch 118 | batch 1180/2304 [I 2022-10-28 09:35:59] ...trap/engines/engine.py.228: elapsed: 0:11:03 | left: 0:10:25 [I 2022-10-28 09:35:59] ...trap/engines/engine.py.229: process: 0.55648 | load: 0.00243 [I 2022-10-28 09:35:59] ...trap/engines/engine.py.230: loss: 0.08148 [I 2022-10-28 09:35:59] ...optimizers/trijoint.py.151: total_norm: 9.766324 [I 2022-10-28 09:36:04] ...trap/engines/engine.py.224: train: epoch 118 | batch 1190/2304 [I 2022-10-28 09:36:04] ...trap/engines/engine.py.228: elapsed: 0:11:08 | left: 0:10:15 [I 2022-10-28 09:36:04] ...trap/engines/engine.py.229: process: 0.55506 | load: 0.00244 [I 2022-10-28 09:36:04] ...trap/engines/engine.py.230: loss: 0.08124 [I 2022-10-28 09:36:04] ...optimizers/trijoint.py.151: total_norm: 9.468358 [I 2022-10-28 09:36:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 1200/2304 [I 2022-10-28 09:36:10] ...trap/engines/engine.py.228: elapsed: 0:11:14 | left: 0:10:17 [I 2022-10-28 09:36:10] ...trap/engines/engine.py.229: process: 0.56766 | load: 0.00244 [I 2022-10-28 09:36:10] ...trap/engines/engine.py.230: loss: 0.12744 [I 2022-10-28 09:36:10] ...optimizers/trijoint.py.151: total_norm: 24.655762 [I 2022-10-28 09:36:15] ...trap/engines/engine.py.224: train: epoch 118 | batch 1210/2304 [I 2022-10-28 09:36:15] ...trap/engines/engine.py.228: elapsed: 0:11:19 | left: 0:10:07 [I 2022-10-28 09:36:15] ...trap/engines/engine.py.229: process: 0.55331 | load: 0.00254 [I 2022-10-28 09:36:15] ...trap/engines/engine.py.230: loss: 0.09393 [I 2022-10-28 09:36:15] ...optimizers/trijoint.py.151: total_norm: 17.903538 [I 2022-10-28 09:36:21] ...trap/engines/engine.py.224: train: epoch 118 | batch 1220/2304 [I 2022-10-28 09:36:21] ...trap/engines/engine.py.228: elapsed: 0:11:25 | left: 0:10:06 [I 2022-10-28 09:36:21] ...trap/engines/engine.py.229: process: 0.56586 | load: 0.00266 [I 2022-10-28 09:36:21] ...trap/engines/engine.py.230: loss: 0.18193 [I 2022-10-28 09:36:21] ...optimizers/trijoint.py.151: total_norm: 26.592394 [I 2022-10-28 09:36:26] ...trap/engines/engine.py.224: train: epoch 118 | batch 1230/2304 [I 2022-10-28 09:36:26] ...trap/engines/engine.py.228: elapsed: 0:11:31 | left: 0:10:00 [I 2022-10-28 09:36:26] ...trap/engines/engine.py.229: process: 0.55902 | load: 0.00245 [I 2022-10-28 09:36:26] ...trap/engines/engine.py.230: loss: 0.10203 [I 2022-10-28 09:36:26] ...optimizers/trijoint.py.151: total_norm: 16.138279 [I 2022-10-28 09:36:32] ...trap/engines/engine.py.224: train: epoch 118 | batch 1240/2304 [I 2022-10-28 09:36:32] ...trap/engines/engine.py.228: elapsed: 0:11:36 | left: 0:09:57 [I 2022-10-28 09:36:32] ...trap/engines/engine.py.229: process: 0.56399 | load: 0.00248 [I 2022-10-28 09:36:32] ...trap/engines/engine.py.230: loss: 0.13535 [I 2022-10-28 09:36:32] ...optimizers/trijoint.py.151: total_norm: 26.912109 [I 2022-10-28 09:36:38] ...trap/engines/engine.py.224: train: epoch 118 | batch 1250/2304 [I 2022-10-28 09:36:38] ...trap/engines/engine.py.228: elapsed: 0:11:42 | left: 0:09:50 [I 2022-10-28 09:36:38] ...trap/engines/engine.py.229: process: 0.54318 | load: 0.00245 [I 2022-10-28 09:36:38] ...trap/engines/engine.py.230: loss: 0.11749 [I 2022-10-28 09:36:38] ...optimizers/trijoint.py.151: total_norm: 13.044909 [I 2022-10-28 09:36:43] ...trap/engines/engine.py.224: train: epoch 118 | batch 1260/2304 [I 2022-10-28 09:36:43] ...trap/engines/engine.py.228: elapsed: 0:11:47 | left: 0:09:49 [I 2022-10-28 09:36:43] ...trap/engines/engine.py.229: process: 0.56700 | load: 0.00243 [I 2022-10-28 09:36:43] ...trap/engines/engine.py.230: loss: 0.11619 [I 2022-10-28 09:36:43] ...optimizers/trijoint.py.151: total_norm: 14.345636 [I 2022-10-28 09:36:49] ...trap/engines/engine.py.224: train: epoch 118 | batch 1270/2304 [I 2022-10-28 09:36:49] ...trap/engines/engine.py.228: elapsed: 0:11:53 | left: 0:09:44 [I 2022-10-28 09:36:49] ...trap/engines/engine.py.229: process: 0.56074 | load: 0.00252 [I 2022-10-28 09:36:49] ...trap/engines/engine.py.230: loss: 0.09531 [I 2022-10-28 09:36:49] ...optimizers/trijoint.py.151: total_norm: 23.850071 [I 2022-10-28 09:36:55] ...trap/engines/engine.py.224: train: epoch 118 | batch 1280/2304 [I 2022-10-28 09:36:55] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:41 [I 2022-10-28 09:36:55] ...trap/engines/engine.py.229: process: 0.56183 | load: 0.00269 [I 2022-10-28 09:36:55] ...trap/engines/engine.py.230: loss: 0.13610 [I 2022-10-28 09:36:55] ...optimizers/trijoint.py.151: total_norm: 15.854838 [I 2022-10-28 09:37:00] ...trap/engines/engine.py.224: train: epoch 118 | batch 1290/2304 [I 2022-10-28 09:37:00] ...trap/engines/engine.py.228: elapsed: 0:12:04 | left: 0:09:27 [I 2022-10-28 09:37:00] ...trap/engines/engine.py.229: process: 0.56124 | load: 0.00246 [I 2022-10-28 09:37:00] ...trap/engines/engine.py.230: loss: 0.14750 [I 2022-10-28 09:37:00] ...optimizers/trijoint.py.151: total_norm: 17.941679 [I 2022-10-28 09:37:06] ...trap/engines/engine.py.224: train: epoch 118 | batch 1300/2304 [I 2022-10-28 09:37:06] ...trap/engines/engine.py.228: elapsed: 0:12:10 | left: 0:09:25 [I 2022-10-28 09:37:06] ...trap/engines/engine.py.229: process: 0.55922 | load: 0.00245 [I 2022-10-28 09:37:06] ...trap/engines/engine.py.230: loss: 0.11376 [I 2022-10-28 09:37:06] ...optimizers/trijoint.py.151: total_norm: 8.831021 [I 2022-10-28 09:37:11] ...trap/engines/engine.py.224: train: epoch 118 | batch 1310/2304 [I 2022-10-28 09:37:11] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:17 [I 2022-10-28 09:37:11] ...trap/engines/engine.py.229: process: 0.55905 | load: 0.00247 [I 2022-10-28 09:37:11] ...trap/engines/engine.py.230: loss: 0.08341 [I 2022-10-28 09:37:11] ...optimizers/trijoint.py.151: total_norm: 16.759951 [I 2022-10-28 09:37:17] ...trap/engines/engine.py.224: train: epoch 118 | batch 1320/2304 [I 2022-10-28 09:37:17] ...trap/engines/engine.py.228: elapsed: 0:12:21 | left: 0:09:08 [I 2022-10-28 09:37:17] ...trap/engines/engine.py.229: process: 0.56594 | load: 0.00242 [I 2022-10-28 09:37:17] ...trap/engines/engine.py.230: loss: 0.11158 [I 2022-10-28 09:37:17] ...optimizers/trijoint.py.151: total_norm: 12.163420 [I 2022-10-28 09:37:23] ...trap/engines/engine.py.224: train: epoch 118 | batch 1330/2304 [I 2022-10-28 09:37:23] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:03 [I 2022-10-28 09:37:23] ...trap/engines/engine.py.229: process: 0.55176 | load: 0.00244 [I 2022-10-28 09:37:23] ...trap/engines/engine.py.230: loss: 0.12805 [I 2022-10-28 09:37:23] ...optimizers/trijoint.py.151: total_norm: 31.569683 [I 2022-10-28 09:37:28] ...trap/engines/engine.py.224: train: epoch 118 | batch 1340/2304 [I 2022-10-28 09:37:28] ...trap/engines/engine.py.228: elapsed: 0:12:32 | left: 0:08:58 [I 2022-10-28 09:37:28] ...trap/engines/engine.py.229: process: 0.55451 | load: 0.00248 [I 2022-10-28 09:37:28] ...trap/engines/engine.py.230: loss: 0.14062 [I 2022-10-28 09:37:28] ...optimizers/trijoint.py.151: total_norm: 13.155147 [I 2022-10-28 09:37:34] ...trap/engines/engine.py.224: train: epoch 118 | batch 1350/2304 [I 2022-10-28 09:37:34] ...trap/engines/engine.py.228: elapsed: 0:12:38 | left: 0:08:50 [I 2022-10-28 09:37:34] ...trap/engines/engine.py.229: process: 0.55826 | load: 0.00241 [I 2022-10-28 09:37:34] ...trap/engines/engine.py.230: loss: 0.13201 [I 2022-10-28 09:37:34] ...optimizers/trijoint.py.151: total_norm: 21.306990 [I 2022-10-28 09:37:39] ...trap/engines/engine.py.224: train: epoch 118 | batch 1360/2304 [I 2022-10-28 09:37:39] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:48 [I 2022-10-28 09:37:39] ...trap/engines/engine.py.229: process: 0.56589 | load: 0.00244 [I 2022-10-28 09:37:39] ...trap/engines/engine.py.230: loss: 0.09435 [I 2022-10-28 09:37:39] ...optimizers/trijoint.py.151: total_norm: 18.479290 [I 2022-10-28 09:37:45] ...trap/engines/engine.py.224: train: epoch 118 | batch 1370/2304 [I 2022-10-28 09:37:45] ...trap/engines/engine.py.228: elapsed: 0:12:49 | left: 0:08:42 [I 2022-10-28 09:37:45] ...trap/engines/engine.py.229: process: 0.54011 | load: 0.00252 [I 2022-10-28 09:37:45] ...trap/engines/engine.py.230: loss: 0.12860 [I 2022-10-28 09:37:45] ...optimizers/trijoint.py.151: total_norm: 17.531843 [I 2022-10-28 09:37:51] ...trap/engines/engine.py.224: train: epoch 118 | batch 1380/2304 [I 2022-10-28 09:37:51] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:32 [I 2022-10-28 09:37:51] ...trap/engines/engine.py.229: process: 0.54533 | load: 0.00247 [I 2022-10-28 09:37:51] ...trap/engines/engine.py.230: loss: 0.09698 [I 2022-10-28 09:37:51] ...optimizers/trijoint.py.151: total_norm: 17.460539 [I 2022-10-28 09:37:56] ...trap/engines/engine.py.224: train: epoch 118 | batch 1390/2304 [I 2022-10-28 09:37:56] ...trap/engines/engine.py.228: elapsed: 0:13:00 | left: 0:08:28 [I 2022-10-28 09:37:56] ...trap/engines/engine.py.229: process: 0.56201 | load: 0.00245 [I 2022-10-28 09:37:56] ...trap/engines/engine.py.230: loss: 0.15659 [I 2022-10-28 09:37:56] ...optimizers/trijoint.py.151: total_norm: 14.402985 [I 2022-10-28 09:38:02] ...trap/engines/engine.py.224: train: epoch 118 | batch 1400/2304 [I 2022-10-28 09:38:02] ...trap/engines/engine.py.228: elapsed: 0:13:06 | left: 0:08:27 [I 2022-10-28 09:38:02] ...trap/engines/engine.py.229: process: 0.56345 | load: 0.00247 [I 2022-10-28 09:38:02] ...trap/engines/engine.py.230: loss: 0.10757 [I 2022-10-28 09:38:02] ...optimizers/trijoint.py.151: total_norm: 38.922016 [I 2022-10-28 09:38:08] ...trap/engines/engine.py.224: train: epoch 118 | batch 1410/2304 [I 2022-10-28 09:38:08] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:21 [I 2022-10-28 09:38:08] ...trap/engines/engine.py.229: process: 0.56414 | load: 0.00242 [I 2022-10-28 09:38:08] ...trap/engines/engine.py.230: loss: 0.14113 [I 2022-10-28 09:38:08] ...optimizers/trijoint.py.151: total_norm: 13.061457 [I 2022-10-28 09:38:13] ...trap/engines/engine.py.224: train: epoch 118 | batch 1420/2304 [I 2022-10-28 09:38:13] ...trap/engines/engine.py.228: elapsed: 0:13:17 | left: 0:08:11 [I 2022-10-28 09:38:13] ...trap/engines/engine.py.229: process: 0.54764 | load: 0.00259 [I 2022-10-28 09:38:13] ...trap/engines/engine.py.230: loss: 0.12733 [I 2022-10-28 09:38:13] ...optimizers/trijoint.py.151: total_norm: 9.846762 [I 2022-10-28 09:38:19] ...trap/engines/engine.py.224: train: epoch 118 | batch 1430/2304 [I 2022-10-28 09:38:19] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:34 [I 2022-10-28 09:38:19] ...trap/engines/engine.py.229: process: 0.69694 | load: 0.00248 [I 2022-10-28 09:38:19] ...trap/engines/engine.py.230: loss: 0.10596 [I 2022-10-28 09:38:19] ...optimizers/trijoint.py.151: total_norm: 14.111771 [I 2022-10-28 09:38:24] ...trap/engines/engine.py.224: train: epoch 118 | batch 1440/2304 [I 2022-10-28 09:38:24] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:08:03 [I 2022-10-28 09:38:24] ...trap/engines/engine.py.229: process: 0.53978 | load: 0.00252 [I 2022-10-28 09:38:24] ...trap/engines/engine.py.230: loss: 0.11649 [I 2022-10-28 09:38:24] ...optimizers/trijoint.py.151: total_norm: 14.203318 [I 2022-10-28 09:38:30] ...trap/engines/engine.py.224: train: epoch 118 | batch 1450/2304 [I 2022-10-28 09:38:30] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:07:55 [I 2022-10-28 09:38:30] ...trap/engines/engine.py.229: process: 0.55209 | load: 0.00244 [I 2022-10-28 09:38:30] ...trap/engines/engine.py.230: loss: 0.09963 [I 2022-10-28 09:38:30] ...optimizers/trijoint.py.151: total_norm: 11.564890 [I 2022-10-28 09:38:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 1460/2304 [I 2022-10-28 09:38:36] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:52 [I 2022-10-28 09:38:36] ...trap/engines/engine.py.229: process: 0.55449 | load: 0.00243 [I 2022-10-28 09:38:36] ...trap/engines/engine.py.230: loss: 0.09767 [I 2022-10-28 09:38:36] ...optimizers/trijoint.py.151: total_norm: 13.096202 [I 2022-10-28 09:38:41] ...trap/engines/engine.py.224: train: epoch 118 | batch 1470/2304 [I 2022-10-28 09:38:41] ...trap/engines/engine.py.228: elapsed: 0:13:45 | left: 0:07:47 [I 2022-10-28 09:38:41] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00244 [I 2022-10-28 09:38:41] ...trap/engines/engine.py.230: loss: 0.10731 [I 2022-10-28 09:38:41] ...optimizers/trijoint.py.151: total_norm: 22.217430 [I 2022-10-28 09:38:47] ...trap/engines/engine.py.224: train: epoch 118 | batch 1480/2304 [I 2022-10-28 09:38:47] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:40 [I 2022-10-28 09:38:47] ...trap/engines/engine.py.229: process: 0.56281 | load: 0.00254 [I 2022-10-28 09:38:47] ...trap/engines/engine.py.230: loss: 0.13312 [I 2022-10-28 09:38:47] ...optimizers/trijoint.py.151: total_norm: 11.320757 [I 2022-10-28 09:38:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 1490/2304 [I 2022-10-28 09:38:52] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:35 [I 2022-10-28 09:38:52] ...trap/engines/engine.py.229: process: 0.56088 | load: 0.00245 [I 2022-10-28 09:38:52] ...trap/engines/engine.py.230: loss: 0.11667 [I 2022-10-28 09:38:52] ...optimizers/trijoint.py.151: total_norm: 17.531998 [I 2022-10-28 09:38:58] ...trap/engines/engine.py.224: train: epoch 118 | batch 1500/2304 [I 2022-10-28 09:38:58] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:30 [I 2022-10-28 09:38:58] ...trap/engines/engine.py.229: process: 0.56099 | load: 0.00244 [I 2022-10-28 09:38:58] ...trap/engines/engine.py.230: loss: 0.09656 [I 2022-10-28 09:38:58] ...optimizers/trijoint.py.151: total_norm: 13.970190 [I 2022-10-28 09:39:04] ...trap/engines/engine.py.224: train: epoch 118 | batch 1510/2304 [I 2022-10-28 09:39:04] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:20 [I 2022-10-28 09:39:04] ...trap/engines/engine.py.229: process: 0.54897 | load: 0.00241 [I 2022-10-28 09:39:04] ...trap/engines/engine.py.230: loss: 0.12741 [I 2022-10-28 09:39:04] ...optimizers/trijoint.py.151: total_norm: 24.702438 [I 2022-10-28 09:39:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 1520/2304 [I 2022-10-28 09:39:09] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:15 [I 2022-10-28 09:39:09] ...trap/engines/engine.py.229: process: 0.55182 | load: 0.00244 [I 2022-10-28 09:39:09] ...trap/engines/engine.py.230: loss: 0.09270 [I 2022-10-28 09:39:09] ...optimizers/trijoint.py.151: total_norm: 11.337661 [I 2022-10-28 09:39:15] ...trap/engines/engine.py.224: train: epoch 118 | batch 1530/2304 [I 2022-10-28 09:39:15] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:11 [I 2022-10-28 09:39:15] ...trap/engines/engine.py.229: process: 0.55439 | load: 0.00250 [I 2022-10-28 09:39:15] ...trap/engines/engine.py.230: loss: 0.10179 [I 2022-10-28 09:39:15] ...optimizers/trijoint.py.151: total_norm: 10.309661 [I 2022-10-28 09:39:21] ...trap/engines/engine.py.224: train: epoch 118 | batch 1540/2304 [I 2022-10-28 09:39:21] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:22 [I 2022-10-28 09:39:21] ...trap/engines/engine.py.229: process: 0.55277 | load: 0.00250 [I 2022-10-28 09:39:21] ...trap/engines/engine.py.230: loss: 0.13582 [I 2022-10-28 09:39:21] ...optimizers/trijoint.py.151: total_norm: 16.353813 [I 2022-10-28 09:39:26] ...trap/engines/engine.py.224: train: epoch 118 | batch 1550/2304 [I 2022-10-28 09:39:26] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:01 [I 2022-10-28 09:39:26] ...trap/engines/engine.py.229: process: 0.55199 | load: 0.00249 [I 2022-10-28 09:39:26] ...trap/engines/engine.py.230: loss: 0.10994 [I 2022-10-28 09:39:26] ...optimizers/trijoint.py.151: total_norm: 17.932760 [I 2022-10-28 09:39:32] ...trap/engines/engine.py.224: train: epoch 118 | batch 1560/2304 [I 2022-10-28 09:39:32] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:56 [I 2022-10-28 09:39:32] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00242 [I 2022-10-28 09:39:32] ...trap/engines/engine.py.230: loss: 0.20552 [I 2022-10-28 09:39:32] ...optimizers/trijoint.py.151: total_norm: 10.880097 [I 2022-10-28 09:39:37] ...trap/engines/engine.py.224: train: epoch 118 | batch 1570/2304 [I 2022-10-28 09:39:37] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:48 [I 2022-10-28 09:39:37] ...trap/engines/engine.py.229: process: 0.55358 | load: 0.00246 [I 2022-10-28 09:39:37] ...trap/engines/engine.py.230: loss: 0.09953 [I 2022-10-28 09:39:37] ...optimizers/trijoint.py.151: total_norm: 14.268271 [I 2022-10-28 09:39:43] ...trap/engines/engine.py.224: train: epoch 118 | batch 1580/2304 [I 2022-10-28 09:39:43] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:42 [I 2022-10-28 09:39:43] ...trap/engines/engine.py.229: process: 0.55577 | load: 0.00244 [I 2022-10-28 09:39:43] ...trap/engines/engine.py.230: loss: 0.10129 [I 2022-10-28 09:39:43] ...optimizers/trijoint.py.151: total_norm: 11.005175 [I 2022-10-28 09:39:48] ...trap/engines/engine.py.224: train: epoch 118 | batch 1590/2304 [I 2022-10-28 09:39:48] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:39 [I 2022-10-28 09:39:48] ...trap/engines/engine.py.229: process: 0.55620 | load: 0.00249 [I 2022-10-28 09:39:48] ...trap/engines/engine.py.230: loss: 0.08097 [I 2022-10-28 09:39:48] ...optimizers/trijoint.py.151: total_norm: 19.838211 [I 2022-10-28 09:39:54] ...trap/engines/engine.py.224: train: epoch 118 | batch 1600/2304 [I 2022-10-28 09:39:54] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:31 [I 2022-10-28 09:39:54] ...trap/engines/engine.py.229: process: 0.55481 | load: 0.00247 [I 2022-10-28 09:39:54] ...trap/engines/engine.py.230: loss: 0.09931 [I 2022-10-28 09:39:54] ...optimizers/trijoint.py.151: total_norm: 17.696501 [I 2022-10-28 09:40:00] ...trap/engines/engine.py.224: train: epoch 118 | batch 1610/2304 [I 2022-10-28 09:40:00] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:26 [I 2022-10-28 09:40:00] ...trap/engines/engine.py.229: process: 0.55365 | load: 0.00248 [I 2022-10-28 09:40:00] ...trap/engines/engine.py.230: loss: 0.09424 [I 2022-10-28 09:40:00] ...optimizers/trijoint.py.151: total_norm: 12.452733 [I 2022-10-28 09:40:05] ...trap/engines/engine.py.224: train: epoch 118 | batch 1620/2304 [I 2022-10-28 09:40:05] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:25 [I 2022-10-28 09:40:05] ...trap/engines/engine.py.229: process: 0.56487 | load: 0.00243 [I 2022-10-28 09:40:05] ...trap/engines/engine.py.230: loss: 0.09645 [I 2022-10-28 09:40:05] ...optimizers/trijoint.py.151: total_norm: 9.714053 [I 2022-10-28 09:40:11] ...trap/engines/engine.py.224: train: epoch 118 | batch 1630/2304 [I 2022-10-28 09:40:11] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:14 [I 2022-10-28 09:40:11] ...trap/engines/engine.py.229: process: 0.54596 | load: 0.00247 [I 2022-10-28 09:40:11] ...trap/engines/engine.py.230: loss: 0.12956 [I 2022-10-28 09:40:11] ...optimizers/trijoint.py.151: total_norm: 25.189318 [I 2022-10-28 09:40:16] ...trap/engines/engine.py.224: train: epoch 118 | batch 1640/2304 [I 2022-10-28 09:40:16] ...trap/engines/engine.py.228: elapsed: 0:15:21 | left: 0:06:13 [I 2022-10-28 09:40:16] ...trap/engines/engine.py.229: process: 0.56189 | load: 0.00318 [I 2022-10-28 09:40:16] ...trap/engines/engine.py.230: loss: 0.12152 [I 2022-10-28 09:40:16] ...optimizers/trijoint.py.151: total_norm: 22.663849 [I 2022-10-28 09:40:22] ...trap/engines/engine.py.224: train: epoch 118 | batch 1650/2304 [I 2022-10-28 09:40:22] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:02 [I 2022-10-28 09:40:22] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00244 [I 2022-10-28 09:40:22] ...trap/engines/engine.py.230: loss: 0.13412 [I 2022-10-28 09:40:22] ...optimizers/trijoint.py.151: total_norm: 26.190742 [I 2022-10-28 09:40:28] ...trap/engines/engine.py.224: train: epoch 118 | batch 1660/2304 [I 2022-10-28 09:40:28] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:05:58 [I 2022-10-28 09:40:28] ...trap/engines/engine.py.229: process: 0.55820 | load: 0.00243 [I 2022-10-28 09:40:28] ...trap/engines/engine.py.230: loss: 0.09160 [I 2022-10-28 09:40:28] ...optimizers/trijoint.py.151: total_norm: 17.417131 [I 2022-10-28 09:40:33] ...trap/engines/engine.py.224: train: epoch 118 | batch 1670/2304 [I 2022-10-28 09:40:33] ...trap/engines/engine.py.228: elapsed: 0:15:38 | left: 0:05:56 [I 2022-10-28 09:40:33] ...trap/engines/engine.py.229: process: 0.55558 | load: 0.00248 [I 2022-10-28 09:40:33] ...trap/engines/engine.py.230: loss: 0.08974 [I 2022-10-28 09:40:33] ...optimizers/trijoint.py.151: total_norm: 8.671459 [I 2022-10-28 09:40:39] ...trap/engines/engine.py.224: train: epoch 118 | batch 1680/2304 [I 2022-10-28 09:40:39] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:49 [I 2022-10-28 09:40:39] ...trap/engines/engine.py.229: process: 0.56454 | load: 0.00269 [I 2022-10-28 09:40:39] ...trap/engines/engine.py.230: loss: 0.09619 [I 2022-10-28 09:40:39] ...optimizers/trijoint.py.151: total_norm: 12.341694 [I 2022-10-28 09:40:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 1690/2304 [I 2022-10-28 09:40:44] ...trap/engines/engine.py.228: elapsed: 0:15:49 | left: 0:05:41 [I 2022-10-28 09:40:44] ...trap/engines/engine.py.229: process: 0.54941 | load: 0.00245 [I 2022-10-28 09:40:44] ...trap/engines/engine.py.230: loss: 0.09551 [I 2022-10-28 09:40:44] ...optimizers/trijoint.py.151: total_norm: 7.654395 [I 2022-10-28 09:40:50] ...trap/engines/engine.py.224: train: epoch 118 | batch 1700/2304 [I 2022-10-28 09:40:50] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:36 [I 2022-10-28 09:40:50] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00244 [I 2022-10-28 09:40:50] ...trap/engines/engine.py.230: loss: 0.13243 [I 2022-10-28 09:40:50] ...optimizers/trijoint.py.151: total_norm: 20.981066 [I 2022-10-28 09:40:56] ...trap/engines/engine.py.224: train: epoch 118 | batch 1710/2304 [I 2022-10-28 09:40:56] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:32 [I 2022-10-28 09:40:56] ...trap/engines/engine.py.229: process: 0.56327 | load: 0.00251 [I 2022-10-28 09:40:56] ...trap/engines/engine.py.230: loss: 0.18947 [I 2022-10-28 09:40:56] ...optimizers/trijoint.py.151: total_norm: 73.359627 [I 2022-10-28 09:41:01] ...trap/engines/engine.py.224: train: epoch 118 | batch 1720/2304 [I 2022-10-28 09:41:01] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:25 [I 2022-10-28 09:41:01] ...trap/engines/engine.py.229: process: 0.56293 | load: 0.00248 [I 2022-10-28 09:41:01] ...trap/engines/engine.py.230: loss: 0.21141 [I 2022-10-28 09:41:01] ...optimizers/trijoint.py.151: total_norm: 30.189154 [I 2022-10-28 09:41:07] ...trap/engines/engine.py.224: train: epoch 118 | batch 1730/2304 [I 2022-10-28 09:41:07] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:20 [I 2022-10-28 09:41:07] ...trap/engines/engine.py.229: process: 0.56031 | load: 0.00248 [I 2022-10-28 09:41:07] ...trap/engines/engine.py.230: loss: 0.15339 [I 2022-10-28 09:41:07] ...optimizers/trijoint.py.151: total_norm: 16.699085 [I 2022-10-28 09:41:12] ...trap/engines/engine.py.224: train: epoch 118 | batch 1740/2304 [I 2022-10-28 09:41:12] ...trap/engines/engine.py.228: elapsed: 0:16:17 | left: 0:05:15 [I 2022-10-28 09:41:12] ...trap/engines/engine.py.229: process: 0.55900 | load: 0.00243 [I 2022-10-28 09:41:12] ...trap/engines/engine.py.230: loss: 0.11993 [I 2022-10-28 09:41:12] ...optimizers/trijoint.py.151: total_norm: 40.181953 [I 2022-10-28 09:41:18] ...trap/engines/engine.py.224: train: epoch 118 | batch 1750/2304 [I 2022-10-28 09:41:18] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:09 [I 2022-10-28 09:41:18] ...trap/engines/engine.py.229: process: 0.54951 | load: 0.00284 [I 2022-10-28 09:41:18] ...trap/engines/engine.py.230: loss: 0.12273 [I 2022-10-28 09:41:18] ...optimizers/trijoint.py.151: total_norm: 8.349711 [I 2022-10-28 09:41:24] ...trap/engines/engine.py.224: train: epoch 118 | batch 1760/2304 [I 2022-10-28 09:41:24] ...trap/engines/engine.py.228: elapsed: 0:16:28 | left: 0:05:02 [I 2022-10-28 09:41:24] ...trap/engines/engine.py.229: process: 0.54895 | load: 0.00248 [I 2022-10-28 09:41:24] ...trap/engines/engine.py.230: loss: 0.11622 [I 2022-10-28 09:41:24] ...optimizers/trijoint.py.151: total_norm: 15.201779 [I 2022-10-28 09:41:29] ...trap/engines/engine.py.224: train: epoch 118 | batch 1770/2304 [I 2022-10-28 09:41:29] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:04:56 [I 2022-10-28 09:41:29] ...trap/engines/engine.py.229: process: 0.56075 | load: 0.00243 [I 2022-10-28 09:41:29] ...trap/engines/engine.py.230: loss: 0.12490 [I 2022-10-28 09:41:29] ...optimizers/trijoint.py.151: total_norm: 11.836450 [I 2022-10-28 09:41:35] ...trap/engines/engine.py.224: train: epoch 118 | batch 1780/2304 [I 2022-10-28 09:41:35] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:05:01 [I 2022-10-28 09:41:35] ...trap/engines/engine.py.229: process: 0.65133 | load: 0.00244 [I 2022-10-28 09:41:35] ...trap/engines/engine.py.230: loss: 0.19665 [I 2022-10-28 09:41:35] ...optimizers/trijoint.py.151: total_norm: 14.265490 [I 2022-10-28 09:41:41] ...trap/engines/engine.py.224: train: epoch 118 | batch 1790/2304 [I 2022-10-28 09:41:41] ...trap/engines/engine.py.228: elapsed: 0:16:45 | left: 0:04:50 [I 2022-10-28 09:41:41] ...trap/engines/engine.py.229: process: 0.56018 | load: 0.00243 [I 2022-10-28 09:41:41] ...trap/engines/engine.py.230: loss: 0.09147 [I 2022-10-28 09:41:41] ...optimizers/trijoint.py.151: total_norm: 9.123663 [I 2022-10-28 09:41:46] ...trap/engines/engine.py.224: train: epoch 118 | batch 1800/2304 [I 2022-10-28 09:41:46] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:40 [I 2022-10-28 09:41:46] ...trap/engines/engine.py.229: process: 0.54665 | load: 0.00258 [I 2022-10-28 09:41:46] ...trap/engines/engine.py.230: loss: 0.09485 [I 2022-10-28 09:41:46] ...optimizers/trijoint.py.151: total_norm: 14.911180 [I 2022-10-28 09:41:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 1810/2304 [I 2022-10-28 09:41:52] ...trap/engines/engine.py.228: elapsed: 0:16:56 | left: 0:04:35 [I 2022-10-28 09:41:52] ...trap/engines/engine.py.229: process: 0.55141 | load: 0.00249 [I 2022-10-28 09:41:52] ...trap/engines/engine.py.230: loss: 0.09158 [I 2022-10-28 09:41:52] ...optimizers/trijoint.py.151: total_norm: 14.364328 [I 2022-10-28 09:41:57] ...trap/engines/engine.py.224: train: epoch 118 | batch 1820/2304 [I 2022-10-28 09:41:57] ...trap/engines/engine.py.228: elapsed: 0:17:02 | left: 0:04:30 [I 2022-10-28 09:41:57] ...trap/engines/engine.py.229: process: 0.54816 | load: 0.00245 [I 2022-10-28 09:41:57] ...trap/engines/engine.py.230: loss: 0.10867 [I 2022-10-28 09:41:57] ...optimizers/trijoint.py.151: total_norm: 14.865400 [I 2022-10-28 09:42:03] ...trap/engines/engine.py.224: train: epoch 118 | batch 1830/2304 [I 2022-10-28 09:42:03] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:25 [I 2022-10-28 09:42:03] ...trap/engines/engine.py.229: process: 0.57506 | load: 0.00247 [I 2022-10-28 09:42:03] ...trap/engines/engine.py.230: loss: 0.20067 [I 2022-10-28 09:42:03] ...optimizers/trijoint.py.151: total_norm: 13.946037 [I 2022-10-28 09:42:08] ...trap/engines/engine.py.224: train: epoch 118 | batch 1840/2304 [I 2022-10-28 09:42:08] ...trap/engines/engine.py.228: elapsed: 0:17:13 | left: 0:04:17 [I 2022-10-28 09:42:08] ...trap/engines/engine.py.229: process: 0.55454 | load: 0.00248 [I 2022-10-28 09:42:08] ...trap/engines/engine.py.230: loss: 0.14759 [I 2022-10-28 09:42:08] ...optimizers/trijoint.py.151: total_norm: 12.129624 [I 2022-10-28 09:42:14] ...trap/engines/engine.py.224: train: epoch 118 | batch 1850/2304 [I 2022-10-28 09:42:14] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:13 [I 2022-10-28 09:42:14] ...trap/engines/engine.py.229: process: 0.55286 | load: 0.00253 [I 2022-10-28 09:42:14] ...trap/engines/engine.py.230: loss: 0.13950 [I 2022-10-28 09:42:14] ...optimizers/trijoint.py.151: total_norm: 17.998400 [I 2022-10-28 09:42:20] ...trap/engines/engine.py.224: train: epoch 118 | batch 1860/2304 [I 2022-10-28 09:42:20] ...trap/engines/engine.py.228: elapsed: 0:17:24 | left: 0:04:09 [I 2022-10-28 09:42:20] ...trap/engines/engine.py.229: process: 0.55640 | load: 0.00249 [I 2022-10-28 09:42:20] ...trap/engines/engine.py.230: loss: 0.13672 [I 2022-10-28 09:42:20] ...optimizers/trijoint.py.151: total_norm: 24.070061 [I 2022-10-28 09:42:25] ...trap/engines/engine.py.224: train: epoch 118 | batch 1870/2304 [I 2022-10-28 09:42:25] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:03:59 [I 2022-10-28 09:42:25] ...trap/engines/engine.py.229: process: 0.55305 | load: 0.00246 [I 2022-10-28 09:42:25] ...trap/engines/engine.py.230: loss: 0.14811 [I 2022-10-28 09:42:25] ...optimizers/trijoint.py.151: total_norm: 8.899775 [I 2022-10-28 09:42:31] ...trap/engines/engine.py.224: train: epoch 118 | batch 1880/2304 [I 2022-10-28 09:42:31] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:03:56 [I 2022-10-28 09:42:31] ...trap/engines/engine.py.229: process: 0.55841 | load: 0.00245 [I 2022-10-28 09:42:31] ...trap/engines/engine.py.230: loss: 0.14037 [I 2022-10-28 09:42:31] ...optimizers/trijoint.py.151: total_norm: 12.412492 [I 2022-10-28 09:42:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 1890/2304 [I 2022-10-28 09:42:36] ...trap/engines/engine.py.228: elapsed: 0:17:41 | left: 0:03:50 [I 2022-10-28 09:42:36] ...trap/engines/engine.py.229: process: 0.54980 | load: 0.00242 [I 2022-10-28 09:42:36] ...trap/engines/engine.py.230: loss: 0.15637 [I 2022-10-28 09:42:36] ...optimizers/trijoint.py.151: total_norm: 44.846710 [I 2022-10-28 09:42:42] ...trap/engines/engine.py.224: train: epoch 118 | batch 1900/2304 [I 2022-10-28 09:42:42] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:45 [I 2022-10-28 09:42:42] ...trap/engines/engine.py.229: process: 0.55341 | load: 0.00244 [I 2022-10-28 09:42:42] ...trap/engines/engine.py.230: loss: 0.12401 [I 2022-10-28 09:42:42] ...optimizers/trijoint.py.151: total_norm: 8.926456 [I 2022-10-28 09:42:48] ...trap/engines/engine.py.224: train: epoch 118 | batch 1910/2304 [I 2022-10-28 09:42:48] ...trap/engines/engine.py.228: elapsed: 0:17:52 | left: 0:03:40 [I 2022-10-28 09:42:48] ...trap/engines/engine.py.229: process: 0.55812 | load: 0.00245 [I 2022-10-28 09:42:48] ...trap/engines/engine.py.230: loss: 0.10888 [I 2022-10-28 09:42:48] ...optimizers/trijoint.py.151: total_norm: 11.167628 [I 2022-10-28 09:42:53] ...trap/engines/engine.py.224: train: epoch 118 | batch 1920/2304 [I 2022-10-28 09:42:53] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:34 [I 2022-10-28 09:42:53] ...trap/engines/engine.py.229: process: 0.55667 | load: 0.00251 [I 2022-10-28 09:42:53] ...trap/engines/engine.py.230: loss: 0.12284 [I 2022-10-28 09:42:53] ...optimizers/trijoint.py.151: total_norm: 14.592631 [I 2022-10-28 09:42:59] ...trap/engines/engine.py.224: train: epoch 118 | batch 1930/2304 [I 2022-10-28 09:42:59] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:27 [I 2022-10-28 09:42:59] ...trap/engines/engine.py.229: process: 0.54966 | load: 0.00245 [I 2022-10-28 09:42:59] ...trap/engines/engine.py.230: loss: 0.10766 [I 2022-10-28 09:42:59] ...optimizers/trijoint.py.151: total_norm: 13.116930 [I 2022-10-28 09:43:04] ...trap/engines/engine.py.224: train: epoch 118 | batch 1940/2304 [I 2022-10-28 09:43:04] ...trap/engines/engine.py.228: elapsed: 0:18:09 | left: 0:03:22 [I 2022-10-28 09:43:04] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00243 [I 2022-10-28 09:43:04] ...trap/engines/engine.py.230: loss: 0.10034 [I 2022-10-28 09:43:04] ...optimizers/trijoint.py.151: total_norm: 12.599809 [I 2022-10-28 09:43:10] ...trap/engines/engine.py.224: train: epoch 118 | batch 1950/2304 [I 2022-10-28 09:43:10] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:16 [I 2022-10-28 09:43:10] ...trap/engines/engine.py.229: process: 0.54841 | load: 0.00244 [I 2022-10-28 09:43:10] ...trap/engines/engine.py.230: loss: 0.12186 [I 2022-10-28 09:43:10] ...optimizers/trijoint.py.151: total_norm: 14.936057 [I 2022-10-28 09:43:16] ...trap/engines/engine.py.224: train: epoch 118 | batch 1960/2304 [I 2022-10-28 09:43:16] ...trap/engines/engine.py.228: elapsed: 0:18:20 | left: 0:03:11 [I 2022-10-28 09:43:16] ...trap/engines/engine.py.229: process: 0.55255 | load: 0.00243 [I 2022-10-28 09:43:16] ...trap/engines/engine.py.230: loss: 0.10507 [I 2022-10-28 09:43:16] ...optimizers/trijoint.py.151: total_norm: 21.401951 [I 2022-10-28 09:43:21] ...trap/engines/engine.py.224: train: epoch 118 | batch 1970/2304 [I 2022-10-28 09:43:21] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:08 [I 2022-10-28 09:43:21] ...trap/engines/engine.py.229: process: 0.57031 | load: 0.00256 [I 2022-10-28 09:43:21] ...trap/engines/engine.py.230: loss: 0.11939 [I 2022-10-28 09:43:21] ...optimizers/trijoint.py.151: total_norm: 16.303600 [I 2022-10-28 09:43:27] ...trap/engines/engine.py.224: train: epoch 118 | batch 1980/2304 [I 2022-10-28 09:43:27] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:01 [I 2022-10-28 09:43:27] ...trap/engines/engine.py.229: process: 0.56269 | load: 0.00249 [I 2022-10-28 09:43:27] ...trap/engines/engine.py.230: loss: 0.11984 [I 2022-10-28 09:43:27] ...optimizers/trijoint.py.151: total_norm: 19.516838 [I 2022-10-28 09:43:32] ...trap/engines/engine.py.224: train: epoch 118 | batch 1990/2304 [I 2022-10-28 09:43:32] ...trap/engines/engine.py.228: elapsed: 0:18:37 | left: 0:02:56 [I 2022-10-28 09:43:32] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00246 [I 2022-10-28 09:43:32] ...trap/engines/engine.py.230: loss: 0.07840 [I 2022-10-28 09:43:32] ...optimizers/trijoint.py.151: total_norm: 9.051123 [I 2022-10-28 09:43:38] ...trap/engines/engine.py.224: train: epoch 118 | batch 2000/2304 [I 2022-10-28 09:43:38] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:48 [I 2022-10-28 09:43:38] ...trap/engines/engine.py.229: process: 0.55423 | load: 0.00243 [I 2022-10-28 09:43:38] ...trap/engines/engine.py.230: loss: 0.08089 [I 2022-10-28 09:43:38] ...optimizers/trijoint.py.151: total_norm: 8.443588 [I 2022-10-28 09:43:44] ...trap/engines/engine.py.224: train: epoch 118 | batch 2010/2304 [I 2022-10-28 09:43:44] ...trap/engines/engine.py.228: elapsed: 0:18:48 | left: 0:02:44 [I 2022-10-28 09:43:44] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00246 [I 2022-10-28 09:43:44] ...trap/engines/engine.py.230: loss: 0.09098 [I 2022-10-28 09:43:44] ...optimizers/trijoint.py.151: total_norm: 10.282785 [I 2022-10-28 09:43:49] ...trap/engines/engine.py.224: train: epoch 118 | batch 2020/2304 [I 2022-10-28 09:43:49] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:39 [I 2022-10-28 09:43:49] ...trap/engines/engine.py.229: process: 0.56027 | load: 0.00244 [I 2022-10-28 09:43:49] ...trap/engines/engine.py.230: loss: 0.13035 [I 2022-10-28 09:43:49] ...optimizers/trijoint.py.151: total_norm: 15.749562 [I 2022-10-28 09:43:55] ...trap/engines/engine.py.224: train: epoch 118 | batch 2030/2304 [I 2022-10-28 09:43:55] ...trap/engines/engine.py.228: elapsed: 0:18:59 | left: 0:02:32 [I 2022-10-28 09:43:55] ...trap/engines/engine.py.229: process: 0.55895 | load: 0.00243 [I 2022-10-28 09:43:55] ...trap/engines/engine.py.230: loss: 0.12583 [I 2022-10-28 09:43:55] ...optimizers/trijoint.py.151: total_norm: 26.125559 [I 2022-10-28 09:44:00] ...trap/engines/engine.py.224: train: epoch 118 | batch 2040/2304 [I 2022-10-28 09:44:00] ...trap/engines/engine.py.228: elapsed: 0:19:05 | left: 0:02:27 [I 2022-10-28 09:44:00] ...trap/engines/engine.py.229: process: 0.55499 | load: 0.00243 [I 2022-10-28 09:44:00] ...trap/engines/engine.py.230: loss: 0.09971 [I 2022-10-28 09:44:00] ...optimizers/trijoint.py.151: total_norm: 11.105208 [I 2022-10-28 09:44:06] ...trap/engines/engine.py.224: train: epoch 118 | batch 2050/2304 [I 2022-10-28 09:44:06] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:20 [I 2022-10-28 09:44:06] ...trap/engines/engine.py.229: process: 0.56334 | load: 0.00242 [I 2022-10-28 09:44:06] ...trap/engines/engine.py.230: loss: 0.11037 [I 2022-10-28 09:44:06] ...optimizers/trijoint.py.151: total_norm: 10.506643 [I 2022-10-28 09:44:12] ...trap/engines/engine.py.224: train: epoch 118 | batch 2060/2304 [I 2022-10-28 09:44:12] ...trap/engines/engine.py.228: elapsed: 0:19:16 | left: 0:02:17 [I 2022-10-28 09:44:12] ...trap/engines/engine.py.229: process: 0.57329 | load: 0.00241 [I 2022-10-28 09:44:12] ...trap/engines/engine.py.230: loss: 0.08987 [I 2022-10-28 09:44:12] ...optimizers/trijoint.py.151: total_norm: 12.792353 [I 2022-10-28 09:44:17] ...trap/engines/engine.py.224: train: epoch 118 | batch 2070/2304 [I 2022-10-28 09:44:17] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:10 [I 2022-10-28 09:44:17] ...trap/engines/engine.py.229: process: 0.56223 | load: 0.00243 [I 2022-10-28 09:44:17] ...trap/engines/engine.py.230: loss: 0.11487 [I 2022-10-28 09:44:17] ...optimizers/trijoint.py.151: total_norm: 11.972824 [I 2022-10-28 09:44:23] ...trap/engines/engine.py.224: train: epoch 118 | batch 2080/2304 [I 2022-10-28 09:44:23] ...trap/engines/engine.py.228: elapsed: 0:19:27 | left: 0:02:05 [I 2022-10-28 09:44:23] ...trap/engines/engine.py.229: process: 0.55967 | load: 0.00253 [I 2022-10-28 09:44:23] ...trap/engines/engine.py.230: loss: 0.06780 [I 2022-10-28 09:44:23] ...optimizers/trijoint.py.151: total_norm: 12.740478 [I 2022-10-28 09:44:28] ...trap/engines/engine.py.224: train: epoch 118 | batch 2090/2304 [I 2022-10-28 09:44:28] ...trap/engines/engine.py.228: elapsed: 0:19:33 | left: 0:01:59 [I 2022-10-28 09:44:28] ...trap/engines/engine.py.229: process: 0.55496 | load: 0.00247 [I 2022-10-28 09:44:28] ...trap/engines/engine.py.230: loss: 0.08785 [I 2022-10-28 09:44:28] ...optimizers/trijoint.py.151: total_norm: 18.729179 [I 2022-10-28 09:44:34] ...trap/engines/engine.py.224: train: epoch 118 | batch 2100/2304 [I 2022-10-28 09:44:34] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:01:53 [I 2022-10-28 09:44:34] ...trap/engines/engine.py.229: process: 0.55000 | load: 0.00245 [I 2022-10-28 09:44:34] ...trap/engines/engine.py.230: loss: 0.09876 [I 2022-10-28 09:44:34] ...optimizers/trijoint.py.151: total_norm: 25.874710 [I 2022-10-28 09:44:40] ...trap/engines/engine.py.224: train: epoch 118 | batch 2110/2304 [I 2022-10-28 09:44:40] ...trap/engines/engine.py.228: elapsed: 0:19:44 | left: 0:01:48 [I 2022-10-28 09:44:40] ...trap/engines/engine.py.229: process: 0.54676 | load: 0.00242 [I 2022-10-28 09:44:40] ...trap/engines/engine.py.230: loss: 0.11847 [I 2022-10-28 09:44:40] ...optimizers/trijoint.py.151: total_norm: 16.679132 [I 2022-10-28 09:44:45] ...trap/engines/engine.py.224: train: epoch 118 | batch 2120/2304 [I 2022-10-28 09:44:45] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:43 [I 2022-10-28 09:44:45] ...trap/engines/engine.py.229: process: 0.56231 | load: 0.00245 [I 2022-10-28 09:44:45] ...trap/engines/engine.py.230: loss: 0.10796 [I 2022-10-28 09:44:45] ...optimizers/trijoint.py.151: total_norm: 9.621564 [I 2022-10-28 09:44:51] ...trap/engines/engine.py.224: train: epoch 118 | batch 2130/2304 [I 2022-10-28 09:44:51] ...trap/engines/engine.py.228: elapsed: 0:19:55 | left: 0:01:37 [I 2022-10-28 09:44:51] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00242 [I 2022-10-28 09:44:51] ...trap/engines/engine.py.230: loss: 0.13874 [I 2022-10-28 09:44:51] ...optimizers/trijoint.py.151: total_norm: 19.260294 [I 2022-10-28 09:44:56] ...trap/engines/engine.py.224: train: epoch 118 | batch 2140/2304 [I 2022-10-28 09:44:56] ...trap/engines/engine.py.228: elapsed: 0:20:01 | left: 0:01:31 [I 2022-10-28 09:44:56] ...trap/engines/engine.py.229: process: 0.55852 | load: 0.00241 [I 2022-10-28 09:44:56] ...trap/engines/engine.py.230: loss: 0.11916 [I 2022-10-28 09:44:56] ...optimizers/trijoint.py.151: total_norm: 9.076640 [I 2022-10-28 09:45:02] ...trap/engines/engine.py.224: train: epoch 118 | batch 2150/2304 [I 2022-10-28 09:45:02] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:26 [I 2022-10-28 09:45:02] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00254 [I 2022-10-28 09:45:02] ...trap/engines/engine.py.230: loss: 0.12505 [I 2022-10-28 09:45:02] ...optimizers/trijoint.py.151: total_norm: 18.837572 [I 2022-10-28 09:45:08] ...trap/engines/engine.py.224: train: epoch 118 | batch 2160/2304 [I 2022-10-28 09:45:08] ...trap/engines/engine.py.228: elapsed: 0:20:12 | left: 0:01:20 [I 2022-10-28 09:45:08] ...trap/engines/engine.py.229: process: 0.54557 | load: 0.00245 [I 2022-10-28 09:45:08] ...trap/engines/engine.py.230: loss: 0.10951 [I 2022-10-28 09:45:08] ...optimizers/trijoint.py.151: total_norm: 7.163571 [I 2022-10-28 09:45:13] ...trap/engines/engine.py.224: train: epoch 118 | batch 2170/2304 [I 2022-10-28 09:45:13] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:15 [I 2022-10-28 09:45:13] ...trap/engines/engine.py.229: process: 0.54971 | load: 0.00246 [I 2022-10-28 09:45:13] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-28 09:45:13] ...optimizers/trijoint.py.151: total_norm: 13.183520 [I 2022-10-28 09:45:19] ...trap/engines/engine.py.224: train: epoch 118 | batch 2180/2304 [I 2022-10-28 09:45:19] ...trap/engines/engine.py.228: elapsed: 0:20:23 | left: 0:01:09 [I 2022-10-28 09:45:19] ...trap/engines/engine.py.229: process: 0.55268 | load: 0.00247 [I 2022-10-28 09:45:19] ...trap/engines/engine.py.230: loss: 0.07014 [I 2022-10-28 09:45:19] ...optimizers/trijoint.py.151: total_norm: 13.591498 [I 2022-10-28 09:45:24] ...trap/engines/engine.py.224: train: epoch 118 | batch 2190/2304 [I 2022-10-28 09:45:24] ...trap/engines/engine.py.228: elapsed: 0:20:29 | left: 0:01:03 [I 2022-10-28 09:45:24] ...trap/engines/engine.py.229: process: 0.55504 | load: 0.00243 [I 2022-10-28 09:45:24] ...trap/engines/engine.py.230: loss: 0.13317 [I 2022-10-28 09:45:24] ...optimizers/trijoint.py.151: total_norm: 103.722618 [I 2022-10-28 09:45:30] ...trap/engines/engine.py.224: train: epoch 118 | batch 2200/2304 [I 2022-10-28 09:45:30] ...trap/engines/engine.py.228: elapsed: 0:20:34 | left: 0:00:57 [I 2022-10-28 09:45:30] ...trap/engines/engine.py.229: process: 0.55447 | load: 0.00243 [I 2022-10-28 09:45:30] ...trap/engines/engine.py.230: loss: 0.11800 [I 2022-10-28 09:45:30] ...optimizers/trijoint.py.151: total_norm: 11.122403 [I 2022-10-28 09:45:36] ...trap/engines/engine.py.224: train: epoch 118 | batch 2210/2304 [I 2022-10-28 09:45:36] ...trap/engines/engine.py.228: elapsed: 0:20:40 | left: 0:00:52 [I 2022-10-28 09:45:36] ...trap/engines/engine.py.229: process: 0.55545 | load: 0.00249 [I 2022-10-28 09:45:36] ...trap/engines/engine.py.230: loss: 0.08875 [I 2022-10-28 09:45:36] ...optimizers/trijoint.py.151: total_norm: 18.094349 [I 2022-10-28 09:45:41] ...trap/engines/engine.py.224: train: epoch 118 | batch 2220/2304 [I 2022-10-28 09:45:41] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:00:46 [I 2022-10-28 09:45:41] ...trap/engines/engine.py.229: process: 0.54553 | load: 0.00254 [I 2022-10-28 09:45:41] ...trap/engines/engine.py.230: loss: 0.10764 [I 2022-10-28 09:45:41] ...optimizers/trijoint.py.151: total_norm: 25.095026 [I 2022-10-28 09:45:47] ...trap/engines/engine.py.224: train: epoch 118 | batch 2230/2304 [I 2022-10-28 09:45:47] ...trap/engines/engine.py.228: elapsed: 0:20:51 | left: 0:00:41 [I 2022-10-28 09:45:47] ...trap/engines/engine.py.229: process: 0.56029 | load: 0.00241 [I 2022-10-28 09:45:47] ...trap/engines/engine.py.230: loss: 0.11738 [I 2022-10-28 09:45:47] ...optimizers/trijoint.py.151: total_norm: 8.677711 [I 2022-10-28 09:45:52] ...trap/engines/engine.py.224: train: epoch 118 | batch 2240/2304 [I 2022-10-28 09:45:52] ...trap/engines/engine.py.228: elapsed: 0:20:57 | left: 0:00:35 [I 2022-10-28 09:45:52] ...trap/engines/engine.py.229: process: 0.56038 | load: 0.00244 [I 2022-10-28 09:45:52] ...trap/engines/engine.py.230: loss: 0.18505 [I 2022-10-28 09:45:52] ...optimizers/trijoint.py.151: total_norm: 12.048661 [I 2022-10-28 09:45:58] ...trap/engines/engine.py.224: train: epoch 118 | batch 2250/2304 [I 2022-10-28 09:45:58] ...trap/engines/engine.py.228: elapsed: 0:21:02 | left: 0:00:30 [I 2022-10-28 09:45:58] ...trap/engines/engine.py.229: process: 0.55136 | load: 0.00249 [I 2022-10-28 09:45:58] ...trap/engines/engine.py.230: loss: 0.10307 [I 2022-10-28 09:45:58] ...optimizers/trijoint.py.151: total_norm: 8.784930 [I 2022-10-28 09:46:04] ...trap/engines/engine.py.224: train: epoch 118 | batch 2260/2304 [I 2022-10-28 09:46:04] ...trap/engines/engine.py.228: elapsed: 0:21:08 | left: 0:00:24 [I 2022-10-28 09:46:04] ...trap/engines/engine.py.229: process: 0.56384 | load: 0.00248 [I 2022-10-28 09:46:04] ...trap/engines/engine.py.230: loss: 0.09654 [I 2022-10-28 09:46:04] ...optimizers/trijoint.py.151: total_norm: 11.966713 [I 2022-10-28 09:46:09] ...trap/engines/engine.py.224: train: epoch 118 | batch 2270/2304 [I 2022-10-28 09:46:09] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:18 [I 2022-10-28 09:46:09] ...trap/engines/engine.py.229: process: 0.54449 | load: 0.00244 [I 2022-10-28 09:46:09] ...trap/engines/engine.py.230: loss: 0.10133 [I 2022-10-28 09:46:09] ...optimizers/trijoint.py.151: total_norm: 13.565932 [I 2022-10-28 09:46:15] ...trap/engines/engine.py.224: train: epoch 118 | batch 2280/2304 [I 2022-10-28 09:46:15] ...trap/engines/engine.py.228: elapsed: 0:21:19 | left: 0:00:13 [I 2022-10-28 09:46:15] ...trap/engines/engine.py.229: process: 0.55435 | load: 0.00243 [I 2022-10-28 09:46:15] ...trap/engines/engine.py.230: loss: 0.15400 [I 2022-10-28 09:46:15] ...optimizers/trijoint.py.151: total_norm: 15.182566 [I 2022-10-28 09:46:20] ...trap/engines/engine.py.224: train: epoch 118 | batch 2290/2304 [I 2022-10-28 09:46:20] ...trap/engines/engine.py.228: elapsed: 0:21:25 | left: 0:00:07 [I 2022-10-28 09:46:20] ...trap/engines/engine.py.229: process: 0.55753 | load: 0.00246 [I 2022-10-28 09:46:20] ...trap/engines/engine.py.230: loss: 0.09710 [I 2022-10-28 09:46:20] ...optimizers/trijoint.py.151: total_norm: 44.735512 [I 2022-10-28 09:46:26] ...trap/engines/engine.py.224: train: epoch 118 | batch 2300/2304 [I 2022-10-28 09:46:26] ...trap/engines/engine.py.228: elapsed: 0:21:30 | left: 0:00:02 [I 2022-10-28 09:46:26] ...trap/engines/engine.py.229: process: 0.55981 | load: 0.00015 [I 2022-10-28 09:46:26] ...trap/engines/engine.py.230: loss: 0.09657 [I 2022-10-28 09:46:26] ...optimizers/trijoint.py.151: total_norm: 18.932154 [I 2022-10-28 09:46:28] ...trap/engines/engine.py.224: train: epoch 118 | batch 2304/2304 [I 2022-10-28 09:46:28] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:00 [I 2022-10-28 09:46:28] ...trap/engines/engine.py.229: process: 0.56047 | load: 0.00015 [I 2022-10-28 09:46:28] ...trap/engines/engine.py.230: loss: 0.11059 [I 2022-10-28 09:46:28] ...optimizers/trijoint.py.151: total_norm: 15.287291 [S 2022-10-28 09:46:29] ...trap/engines/engine.py.129: train_epoch.epoch: 118 [S 2022-10-28 09:46:29] ...trap/engines/engine.py.129: train_epoch.loss: 0.1188473645912851 [I 2022-10-28 09:46:29] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 09:46:31] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 118 [I 2022-10-28 09:46:31] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 09:46:33] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 09:46:33] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 09:46:33] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 09:46:37] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 09:46:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 0/510 [I 2022-10-28 09:46:37] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:53:06 [I 2022-10-28 09:46:37] ...trap/engines/engine.py.326: process: 6.24813 | load: 4.28695 [I 2022-10-28 09:46:41] ...trap/engines/engine.py.321: eval: epoch 118 | batch 10/510 [I 2022-10-28 09:46:41] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:02 [I 2022-10-28 09:46:41] ...trap/engines/engine.py.326: process: 0.32461 | load: 0.00018 [I 2022-10-28 09:46:44] ...trap/engines/engine.py.321: eval: epoch 118 | batch 20/510 [I 2022-10-28 09:46:44] ...trap/engines/engine.py.325: elapsed: 0:00:12 | left: 0:03:17 [I 2022-10-28 09:46:44] ...trap/engines/engine.py.326: process: 0.33397 | load: 0.00020 [I 2022-10-28 09:46:47] ...trap/engines/engine.py.321: eval: epoch 118 | batch 30/510 [I 2022-10-28 09:46:47] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:43 [I 2022-10-28 09:46:47] ...trap/engines/engine.py.326: process: 0.33012 | load: 0.00017 [I 2022-10-28 09:46:51] ...trap/engines/engine.py.321: eval: epoch 118 | batch 40/510 [I 2022-10-28 09:46:51] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:38 [I 2022-10-28 09:46:51] ...trap/engines/engine.py.326: process: 0.34130 | load: 0.00019 [I 2022-10-28 09:46:54] ...trap/engines/engine.py.321: eval: epoch 118 | batch 50/510 [I 2022-10-28 09:46:54] ...trap/engines/engine.py.325: elapsed: 0:00:22 | left: 0:02:33 [I 2022-10-28 09:46:54] ...trap/engines/engine.py.326: process: 0.33289 | load: 0.00016 [I 2022-10-28 09:46:57] ...trap/engines/engine.py.321: eval: epoch 118 | batch 60/510 [I 2022-10-28 09:46:57] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:30 [I 2022-10-28 09:46:57] ...trap/engines/engine.py.326: process: 0.33808 | load: 0.00018 [I 2022-10-28 09:47:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 70/510 [I 2022-10-28 09:47:01] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:26 [I 2022-10-28 09:47:01] ...trap/engines/engine.py.326: process: 0.32452 | load: 0.00016 [I 2022-10-28 09:47:04] ...trap/engines/engine.py.321: eval: epoch 118 | batch 80/510 [I 2022-10-28 09:47:04] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:23 [I 2022-10-28 09:47:04] ...trap/engines/engine.py.326: process: 0.33520 | load: 0.00017 [I 2022-10-28 09:47:07] ...trap/engines/engine.py.321: eval: epoch 118 | batch 90/510 [I 2022-10-28 09:47:07] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:23 [I 2022-10-28 09:47:07] ...trap/engines/engine.py.326: process: 0.36232 | load: 0.00017 [I 2022-10-28 09:47:11] ...trap/engines/engine.py.321: eval: epoch 118 | batch 100/510 [I 2022-10-28 09:47:11] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:13 [I 2022-10-28 09:47:11] ...trap/engines/engine.py.326: process: 0.28115 | load: 0.00017 [I 2022-10-28 09:47:14] ...trap/engines/engine.py.321: eval: epoch 118 | batch 110/510 [I 2022-10-28 09:47:14] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:57 [I 2022-10-28 09:47:14] ...trap/engines/engine.py.326: process: 0.28766 | load: 0.00018 [I 2022-10-28 09:47:17] ...trap/engines/engine.py.321: eval: epoch 118 | batch 120/510 [I 2022-10-28 09:47:17] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:54 [I 2022-10-28 09:47:17] ...trap/engines/engine.py.326: process: 0.29214 | load: 0.00017 [I 2022-10-28 09:47:20] ...trap/engines/engine.py.321: eval: epoch 118 | batch 130/510 [I 2022-10-28 09:47:20] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:51 [I 2022-10-28 09:47:20] ...trap/engines/engine.py.326: process: 0.28573 | load: 0.00014 [I 2022-10-28 09:47:22] ...trap/engines/engine.py.321: eval: epoch 118 | batch 140/510 [I 2022-10-28 09:47:22] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:47 [I 2022-10-28 09:47:22] ...trap/engines/engine.py.326: process: 0.28959 | load: 0.00018 [I 2022-10-28 09:47:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 150/510 [I 2022-10-28 09:47:25] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:44 [I 2022-10-28 09:47:25] ...trap/engines/engine.py.326: process: 0.28991 | load: 0.00014 [I 2022-10-28 09:47:28] ...trap/engines/engine.py.321: eval: epoch 118 | batch 160/510 [I 2022-10-28 09:47:28] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:41 [I 2022-10-28 09:47:28] ...trap/engines/engine.py.326: process: 0.29128 | load: 0.00017 [I 2022-10-28 09:47:31] ...trap/engines/engine.py.321: eval: epoch 118 | batch 170/510 [I 2022-10-28 09:47:31] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:38 [I 2022-10-28 09:47:31] ...trap/engines/engine.py.326: process: 0.28794 | load: 0.00014 [I 2022-10-28 09:47:34] ...trap/engines/engine.py.321: eval: epoch 118 | batch 180/510 [I 2022-10-28 09:47:34] ...trap/engines/engine.py.325: elapsed: 0:01:02 | left: 0:01:35 [I 2022-10-28 09:47:34] ...trap/engines/engine.py.326: process: 0.28984 | load: 0.00017 [I 2022-10-28 09:47:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 190/510 [I 2022-10-28 09:47:37] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:32 [I 2022-10-28 09:47:37] ...trap/engines/engine.py.326: process: 0.28972 | load: 0.00014 [I 2022-10-28 09:47:40] ...trap/engines/engine.py.321: eval: epoch 118 | batch 200/510 [I 2022-10-28 09:47:40] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:28 [I 2022-10-28 09:47:40] ...trap/engines/engine.py.326: process: 0.28387 | load: 0.00017 [I 2022-10-28 09:47:43] ...trap/engines/engine.py.321: eval: epoch 118 | batch 210/510 [I 2022-10-28 09:47:43] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:26 [I 2022-10-28 09:47:43] ...trap/engines/engine.py.326: process: 0.28897 | load: 0.00014 [I 2022-10-28 09:47:46] ...trap/engines/engine.py.321: eval: epoch 118 | batch 220/510 [I 2022-10-28 09:47:46] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:23 [I 2022-10-28 09:47:46] ...trap/engines/engine.py.326: process: 0.28826 | load: 0.00021 [I 2022-10-28 09:47:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 230/510 [I 2022-10-28 09:47:48] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:20 [I 2022-10-28 09:47:48] ...trap/engines/engine.py.326: process: 0.29177 | load: 0.00020 [I 2022-10-28 09:47:51] ...trap/engines/engine.py.321: eval: epoch 118 | batch 240/510 [I 2022-10-28 09:47:51] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:18 [I 2022-10-28 09:47:51] ...trap/engines/engine.py.326: process: 0.29320 | load: 0.00016 [I 2022-10-28 09:47:54] ...trap/engines/engine.py.321: eval: epoch 118 | batch 250/510 [I 2022-10-28 09:47:54] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:14 [I 2022-10-28 09:47:54] ...trap/engines/engine.py.326: process: 0.28033 | load: 0.00019 [I 2022-10-28 09:47:57] ...trap/engines/engine.py.321: eval: epoch 118 | batch 260/510 [I 2022-10-28 09:47:57] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:12 [I 2022-10-28 09:47:57] ...trap/engines/engine.py.326: process: 0.28554 | load: 0.00016 [I 2022-10-28 09:48:00] ...trap/engines/engine.py.321: eval: epoch 118 | batch 270/510 [I 2022-10-28 09:48:00] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:09 [I 2022-10-28 09:48:00] ...trap/engines/engine.py.326: process: 0.29235 | load: 0.00022 [I 2022-10-28 09:48:03] ...trap/engines/engine.py.321: eval: epoch 118 | batch 280/510 [I 2022-10-28 09:48:03] ...trap/engines/engine.py.325: elapsed: 0:01:31 | left: 0:01:06 [I 2022-10-28 09:48:03] ...trap/engines/engine.py.326: process: 0.29265 | load: 0.00015 [I 2022-10-28 09:48:06] ...trap/engines/engine.py.321: eval: epoch 118 | batch 290/510 [I 2022-10-28 09:48:06] ...trap/engines/engine.py.325: elapsed: 0:01:34 | left: 0:01:03 [I 2022-10-28 09:48:06] ...trap/engines/engine.py.326: process: 0.28841 | load: 0.00017 [I 2022-10-28 09:48:09] ...trap/engines/engine.py.321: eval: epoch 118 | batch 300/510 [I 2022-10-28 09:48:09] ...trap/engines/engine.py.325: elapsed: 0:01:37 | left: 0:01:00 [I 2022-10-28 09:48:09] ...trap/engines/engine.py.326: process: 0.28118 | load: 0.00016 [I 2022-10-28 09:48:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 310/510 [I 2022-10-28 09:48:12] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:57 [I 2022-10-28 09:48:12] ...trap/engines/engine.py.326: process: 0.28986 | load: 0.00019 [I 2022-10-28 09:48:15] ...trap/engines/engine.py.321: eval: epoch 118 | batch 320/510 [I 2022-10-28 09:48:15] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:54 [I 2022-10-28 09:48:15] ...trap/engines/engine.py.326: process: 0.29022 | load: 0.00015 [I 2022-10-28 09:48:17] ...trap/engines/engine.py.321: eval: epoch 118 | batch 330/510 [I 2022-10-28 09:48:17] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:52 [I 2022-10-28 09:48:17] ...trap/engines/engine.py.326: process: 0.28888 | load: 0.00018 [I 2022-10-28 09:48:20] ...trap/engines/engine.py.321: eval: epoch 118 | batch 340/510 [I 2022-10-28 09:48:20] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:48 [I 2022-10-28 09:48:20] ...trap/engines/engine.py.326: process: 0.28531 | load: 0.00016 [I 2022-10-28 09:48:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 350/510 [I 2022-10-28 09:48:23] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:46 [I 2022-10-28 09:48:23] ...trap/engines/engine.py.326: process: 0.28823 | load: 0.00018 [I 2022-10-28 09:48:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 360/510 [I 2022-10-28 09:48:26] ...trap/engines/engine.py.325: elapsed: 0:01:54 | left: 0:00:43 [I 2022-10-28 09:48:26] ...trap/engines/engine.py.326: process: 0.29114 | load: 0.00021 [I 2022-10-28 09:48:29] ...trap/engines/engine.py.321: eval: epoch 118 | batch 370/510 [I 2022-10-28 09:48:29] ...trap/engines/engine.py.325: elapsed: 0:01:57 | left: 0:00:40 [I 2022-10-28 09:48:29] ...trap/engines/engine.py.326: process: 0.28552 | load: 0.00018 [I 2022-10-28 09:48:32] ...trap/engines/engine.py.321: eval: epoch 118 | batch 380/510 [I 2022-10-28 09:48:32] ...trap/engines/engine.py.325: elapsed: 0:02:00 | left: 0:00:37 [I 2022-10-28 09:48:32] ...trap/engines/engine.py.326: process: 0.28172 | load: 0.00019 [I 2022-10-28 09:48:35] ...trap/engines/engine.py.321: eval: epoch 118 | batch 390/510 [I 2022-10-28 09:48:35] ...trap/engines/engine.py.325: elapsed: 0:02:03 | left: 0:00:34 [I 2022-10-28 09:48:35] ...trap/engines/engine.py.326: process: 0.28929 | load: 0.00019 [I 2022-10-28 09:48:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 400/510 [I 2022-10-28 09:48:38] ...trap/engines/engine.py.325: elapsed: 0:02:06 | left: 0:00:31 [I 2022-10-28 09:48:38] ...trap/engines/engine.py.326: process: 0.29382 | load: 0.00024 [I 2022-10-28 09:48:41] ...trap/engines/engine.py.321: eval: epoch 118 | batch 410/510 [I 2022-10-28 09:48:41] ...trap/engines/engine.py.325: elapsed: 0:02:09 | left: 0:00:28 [I 2022-10-28 09:48:41] ...trap/engines/engine.py.326: process: 0.28798 | load: 0.00018 [I 2022-10-28 09:48:43] ...trap/engines/engine.py.321: eval: epoch 118 | batch 420/510 [I 2022-10-28 09:48:43] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:26 [I 2022-10-28 09:48:43] ...trap/engines/engine.py.326: process: 0.28957 | load: 0.00016 [I 2022-10-28 09:48:46] ...trap/engines/engine.py.321: eval: epoch 118 | batch 430/510 [I 2022-10-28 09:48:46] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:22 [I 2022-10-28 09:48:46] ...trap/engines/engine.py.326: process: 0.28263 | load: 0.00019 [I 2022-10-28 09:48:49] ...trap/engines/engine.py.321: eval: epoch 118 | batch 440/510 [I 2022-10-28 09:48:49] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:20 [I 2022-10-28 09:48:49] ...trap/engines/engine.py.326: process: 0.28613 | load: 0.00016 [I 2022-10-28 09:48:52] ...trap/engines/engine.py.321: eval: epoch 118 | batch 450/510 [I 2022-10-28 09:48:52] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:17 [I 2022-10-28 09:48:52] ...trap/engines/engine.py.326: process: 0.28911 | load: 0.00020 [I 2022-10-28 09:48:55] ...trap/engines/engine.py.321: eval: epoch 118 | batch 460/510 [I 2022-10-28 09:48:55] ...trap/engines/engine.py.325: elapsed: 0:02:23 | left: 0:00:14 [I 2022-10-28 09:48:55] ...trap/engines/engine.py.326: process: 0.28713 | load: 0.00018 [I 2022-10-28 09:48:58] ...trap/engines/engine.py.321: eval: epoch 118 | batch 470/510 [I 2022-10-28 09:48:58] ...trap/engines/engine.py.325: elapsed: 0:02:26 | left: 0:00:11 [I 2022-10-28 09:48:58] ...trap/engines/engine.py.326: process: 0.29048 | load: 0.00016 [I 2022-10-28 09:49:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 480/510 [I 2022-10-28 09:49:01] ...trap/engines/engine.py.325: elapsed: 0:02:29 | left: 0:00:08 [I 2022-10-28 09:49:01] ...trap/engines/engine.py.326: process: 0.28342 | load: 0.00021 [I 2022-10-28 09:49:04] ...trap/engines/engine.py.321: eval: epoch 118 | batch 490/510 [I 2022-10-28 09:49:04] ...trap/engines/engine.py.325: elapsed: 0:02:32 | left: 0:00:05 [I 2022-10-28 09:49:04] ...trap/engines/engine.py.326: process: 0.28510 | load: 0.00016 [I 2022-10-28 09:49:07] ...trap/engines/engine.py.321: eval: epoch 118 | batch 500/510 [I 2022-10-28 09:49:07] ...trap/engines/engine.py.325: elapsed: 0:02:35 | left: 0:00:02 [I 2022-10-28 09:49:07] ...trap/engines/engine.py.326: process: 0.28739 | load: 0.00045 [I 2022-10-28 09:49:10] ...trap/engines/engine.py.321: eval: epoch 118 | batch 510/510 [I 2022-10-28 09:49:10] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:00 [I 2022-10-28 09:49:10] ...trap/engines/engine.py.326: process: 0.28607 | load: 0.00015 [S 2022-10-28 09:49:10] ...trap/engines/engine.py.132: eval_epoch.epoch: 118 [I 2022-10-28 09:49:10] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.741500 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907800 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.933800 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.735300 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.907500 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933500 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.013552 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007666 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007125 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011714 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007762 [S 2022-10-28 09:49:14] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007365 [I 2022-10-28 09:49:17] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 09:49:17] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 09:49:17] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 09:49:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 09:49:17] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 09:49:23] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 09:49:23] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 09:49:29] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 09:49:29] ...trap/engines/engine.py.163: Training model on trainset for epoch 119 [I 2022-10-28 09:49:29] ...m/datasets/recipe1m.py.396: Dataset will be sampled with "triplet_classif" batch_sampler. [I 2022-10-28 09:49:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 0/2304 [I 2022-10-28 09:49:32] ...trap/engines/engine.py.228: elapsed: 0:00:02 | left: 1:49:46 [I 2022-10-28 09:49:32] ...trap/engines/engine.py.229: process: 2.85854 | load: 2.19580 [I 2022-10-28 09:49:32] ...trap/engines/engine.py.230: loss: 0.07690 [I 2022-10-28 09:49:32] ...optimizers/trijoint.py.151: total_norm: 9.135317 [I 2022-10-28 09:49:37] ...trap/engines/engine.py.224: train: epoch 119 | batch 10/2304 [I 2022-10-28 09:49:37] ...trap/engines/engine.py.228: elapsed: 0:00:08 | left: 0:30:52 [I 2022-10-28 09:49:37] ...trap/engines/engine.py.229: process: 0.56837 | load: 0.00372 [I 2022-10-28 09:49:37] ...trap/engines/engine.py.230: loss: 0.17057 [I 2022-10-28 09:49:37] ...optimizers/trijoint.py.151: total_norm: 17.799128 [I 2022-10-28 09:49:43] ...trap/engines/engine.py.224: train: epoch 119 | batch 20/2304 [I 2022-10-28 09:49:43] ...trap/engines/engine.py.228: elapsed: 0:00:14 | left: 0:22:07 [I 2022-10-28 09:49:43] ...trap/engines/engine.py.229: process: 0.54577 | load: 0.00413 [I 2022-10-28 09:49:43] ...trap/engines/engine.py.230: loss: 0.12051 [I 2022-10-28 09:49:43] ...optimizers/trijoint.py.151: total_norm: 19.956081 [I 2022-10-28 09:49:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 30/2304 [I 2022-10-28 09:49:49] ...trap/engines/engine.py.228: elapsed: 0:00:19 | left: 0:21:13 [I 2022-10-28 09:49:49] ...trap/engines/engine.py.229: process: 0.55641 | load: 0.00299 [I 2022-10-28 09:49:49] ...trap/engines/engine.py.230: loss: 0.09469 [I 2022-10-28 09:49:49] ...optimizers/trijoint.py.151: total_norm: 13.288980 [I 2022-10-28 09:49:54] ...trap/engines/engine.py.224: train: epoch 119 | batch 40/2304 [I 2022-10-28 09:49:54] ...trap/engines/engine.py.228: elapsed: 0:00:25 | left: 0:21:05 [I 2022-10-28 09:49:54] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00283 [I 2022-10-28 09:49:54] ...trap/engines/engine.py.230: loss: 0.13013 [I 2022-10-28 09:49:54] ...optimizers/trijoint.py.151: total_norm: 17.653856 [I 2022-10-28 09:50:00] ...trap/engines/engine.py.224: train: epoch 119 | batch 50/2304 [I 2022-10-28 09:50:00] ...trap/engines/engine.py.228: elapsed: 0:00:30 | left: 0:20:50 [I 2022-10-28 09:50:00] ...trap/engines/engine.py.229: process: 0.55894 | load: 0.00297 [I 2022-10-28 09:50:00] ...trap/engines/engine.py.230: loss: 0.12224 [I 2022-10-28 09:50:00] ...optimizers/trijoint.py.151: total_norm: 14.299828 [I 2022-10-28 09:50:05] ...trap/engines/engine.py.224: train: epoch 119 | batch 60/2304 [I 2022-10-28 09:50:05] ...trap/engines/engine.py.228: elapsed: 0:00:36 | left: 0:20:37 [I 2022-10-28 09:50:05] ...trap/engines/engine.py.229: process: 0.55529 | load: 0.00302 [I 2022-10-28 09:50:05] ...trap/engines/engine.py.230: loss: 0.11572 [I 2022-10-28 09:50:05] ...optimizers/trijoint.py.151: total_norm: 12.843706 [I 2022-10-28 09:50:11] ...trap/engines/engine.py.224: train: epoch 119 | batch 70/2304 [I 2022-10-28 09:50:11] ...trap/engines/engine.py.228: elapsed: 0:00:41 | left: 0:20:42 [I 2022-10-28 09:50:11] ...trap/engines/engine.py.229: process: 0.55107 | load: 0.00350 [I 2022-10-28 09:50:11] ...trap/engines/engine.py.230: loss: 0.07509 [I 2022-10-28 09:50:11] ...optimizers/trijoint.py.151: total_norm: 18.881489 [I 2022-10-28 09:50:16] ...trap/engines/engine.py.224: train: epoch 119 | batch 80/2304 [I 2022-10-28 09:50:16] ...trap/engines/engine.py.228: elapsed: 0:00:47 | left: 0:20:43 [I 2022-10-28 09:50:16] ...trap/engines/engine.py.229: process: 0.56675 | load: 0.00265 [I 2022-10-28 09:50:16] ...trap/engines/engine.py.230: loss: 0.11310 [I 2022-10-28 09:50:16] ...optimizers/trijoint.py.151: total_norm: 8.450773 [I 2022-10-28 09:50:22] ...trap/engines/engine.py.224: train: epoch 119 | batch 90/2304 [I 2022-10-28 09:50:22] ...trap/engines/engine.py.228: elapsed: 0:00:53 | left: 0:20:38 [I 2022-10-28 09:50:22] ...trap/engines/engine.py.229: process: 0.54222 | load: 0.00260 [I 2022-10-28 09:50:22] ...trap/engines/engine.py.230: loss: 0.09099 [I 2022-10-28 09:50:22] ...optimizers/trijoint.py.151: total_norm: 12.235136 [I 2022-10-28 09:50:28] ...trap/engines/engine.py.224: train: epoch 119 | batch 100/2304 [I 2022-10-28 09:50:28] ...trap/engines/engine.py.228: elapsed: 0:00:58 | left: 0:20:49 [I 2022-10-28 09:50:28] ...trap/engines/engine.py.229: process: 0.57018 | load: 0.00264 [I 2022-10-28 09:50:28] ...trap/engines/engine.py.230: loss: 0.07758 [I 2022-10-28 09:50:28] ...optimizers/trijoint.py.151: total_norm: 15.051394 [I 2022-10-28 09:50:33] ...trap/engines/engine.py.224: train: epoch 119 | batch 110/2304 [I 2022-10-28 09:50:33] ...trap/engines/engine.py.228: elapsed: 0:01:04 | left: 0:20:54 [I 2022-10-28 09:50:33] ...trap/engines/engine.py.229: process: 0.56918 | load: 0.00281 [I 2022-10-28 09:50:33] ...trap/engines/engine.py.230: loss: 0.10133 [I 2022-10-28 09:50:33] ...optimizers/trijoint.py.151: total_norm: 11.373324 [I 2022-10-28 09:50:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 120/2304 [I 2022-10-28 09:50:39] ...trap/engines/engine.py.228: elapsed: 0:01:10 | left: 0:20:23 [I 2022-10-28 09:50:39] ...trap/engines/engine.py.229: process: 0.56433 | load: 0.00259 [I 2022-10-28 09:50:39] ...trap/engines/engine.py.230: loss: 0.14363 [I 2022-10-28 09:50:39] ...optimizers/trijoint.py.151: total_norm: 15.680406 [I 2022-10-28 09:50:45] ...trap/engines/engine.py.224: train: epoch 119 | batch 130/2304 [I 2022-10-28 09:50:45] ...trap/engines/engine.py.228: elapsed: 0:01:15 | left: 0:20:19 [I 2022-10-28 09:50:45] ...trap/engines/engine.py.229: process: 0.56199 | load: 0.00269 [I 2022-10-28 09:50:45] ...trap/engines/engine.py.230: loss: 0.09489 [I 2022-10-28 09:50:45] ...optimizers/trijoint.py.151: total_norm: 13.810243 [I 2022-10-28 09:50:50] ...trap/engines/engine.py.224: train: epoch 119 | batch 140/2304 [I 2022-10-28 09:50:50] ...trap/engines/engine.py.228: elapsed: 0:01:21 | left: 0:19:58 [I 2022-10-28 09:50:50] ...trap/engines/engine.py.229: process: 0.54585 | load: 0.00271 [I 2022-10-28 09:50:50] ...trap/engines/engine.py.230: loss: 0.11929 [I 2022-10-28 09:50:50] ...optimizers/trijoint.py.151: total_norm: 33.427834 [I 2022-10-28 09:50:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 150/2304 [I 2022-10-28 09:50:56] ...trap/engines/engine.py.228: elapsed: 0:01:26 | left: 0:19:57 [I 2022-10-28 09:50:56] ...trap/engines/engine.py.229: process: 0.54522 | load: 0.00248 [I 2022-10-28 09:50:56] ...trap/engines/engine.py.230: loss: 0.13854 [I 2022-10-28 09:50:56] ...optimizers/trijoint.py.151: total_norm: 14.022914 [I 2022-10-28 09:51:01] ...trap/engines/engine.py.224: train: epoch 119 | batch 160/2304 [I 2022-10-28 09:51:01] ...trap/engines/engine.py.228: elapsed: 0:01:32 | left: 0:20:06 [I 2022-10-28 09:51:01] ...trap/engines/engine.py.229: process: 0.58263 | load: 0.00252 [I 2022-10-28 09:51:01] ...trap/engines/engine.py.230: loss: 0.10022 [I 2022-10-28 09:51:01] ...optimizers/trijoint.py.151: total_norm: 16.401524 [I 2022-10-28 09:51:07] ...trap/engines/engine.py.224: train: epoch 119 | batch 170/2304 [I 2022-10-28 09:51:07] ...trap/engines/engine.py.228: elapsed: 0:01:38 | left: 0:19:55 [I 2022-10-28 09:51:07] ...trap/engines/engine.py.229: process: 0.55604 | load: 0.00261 [I 2022-10-28 09:51:07] ...trap/engines/engine.py.230: loss: 0.16800 [I 2022-10-28 09:51:07] ...optimizers/trijoint.py.151: total_norm: 17.253353 [I 2022-10-28 09:51:13] ...trap/engines/engine.py.224: train: epoch 119 | batch 180/2304 [I 2022-10-28 09:51:13] ...trap/engines/engine.py.228: elapsed: 0:01:43 | left: 0:19:50 [I 2022-10-28 09:51:13] ...trap/engines/engine.py.229: process: 0.56575 | load: 0.00268 [I 2022-10-28 09:51:13] ...trap/engines/engine.py.230: loss: 0.09663 [I 2022-10-28 09:51:13] ...optimizers/trijoint.py.151: total_norm: 16.104364 [I 2022-10-28 09:51:18] ...trap/engines/engine.py.224: train: epoch 119 | batch 190/2304 [I 2022-10-28 09:51:18] ...trap/engines/engine.py.228: elapsed: 0:01:49 | left: 0:19:47 [I 2022-10-28 09:51:18] ...trap/engines/engine.py.229: process: 0.56342 | load: 0.00257 [I 2022-10-28 09:51:18] ...trap/engines/engine.py.230: loss: 0.16071 [I 2022-10-28 09:51:18] ...optimizers/trijoint.py.151: total_norm: 30.425602 [I 2022-10-28 09:51:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 200/2304 [I 2022-10-28 09:51:24] ...trap/engines/engine.py.228: elapsed: 0:01:54 | left: 0:19:37 [I 2022-10-28 09:51:24] ...trap/engines/engine.py.229: process: 0.55674 | load: 0.00244 [I 2022-10-28 09:51:24] ...trap/engines/engine.py.230: loss: 0.15943 [I 2022-10-28 09:51:24] ...optimizers/trijoint.py.151: total_norm: 16.638292 [I 2022-10-28 09:51:29] ...trap/engines/engine.py.224: train: epoch 119 | batch 210/2304 [I 2022-10-28 09:51:29] ...trap/engines/engine.py.228: elapsed: 0:02:00 | left: 0:19:39 [I 2022-10-28 09:51:29] ...trap/engines/engine.py.229: process: 0.56402 | load: 0.00263 [I 2022-10-28 09:51:29] ...trap/engines/engine.py.230: loss: 0.14642 [I 2022-10-28 09:51:29] ...optimizers/trijoint.py.151: total_norm: 15.860604 [I 2022-10-28 09:51:35] ...trap/engines/engine.py.224: train: epoch 119 | batch 220/2304 [I 2022-10-28 09:51:35] ...trap/engines/engine.py.228: elapsed: 0:02:06 | left: 0:19:29 [I 2022-10-28 09:51:35] ...trap/engines/engine.py.229: process: 0.56095 | load: 0.00248 [I 2022-10-28 09:51:35] ...trap/engines/engine.py.230: loss: 0.09117 [I 2022-10-28 09:51:35] ...optimizers/trijoint.py.151: total_norm: 20.000093 [I 2022-10-28 09:51:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 230/2304 [I 2022-10-28 09:51:41] ...trap/engines/engine.py.228: elapsed: 0:02:11 | left: 0:19:34 [I 2022-10-28 09:51:41] ...trap/engines/engine.py.229: process: 0.55645 | load: 0.00267 [I 2022-10-28 09:51:41] ...trap/engines/engine.py.230: loss: 0.11013 [I 2022-10-28 09:51:41] ...optimizers/trijoint.py.151: total_norm: 9.933667 [I 2022-10-28 09:51:46] ...trap/engines/engine.py.224: train: epoch 119 | batch 240/2304 [I 2022-10-28 09:51:46] ...trap/engines/engine.py.228: elapsed: 0:02:17 | left: 0:19:25 [I 2022-10-28 09:51:46] ...trap/engines/engine.py.229: process: 0.56939 | load: 0.00263 [I 2022-10-28 09:51:46] ...trap/engines/engine.py.230: loss: 0.10072 [I 2022-10-28 09:51:46] ...optimizers/trijoint.py.151: total_norm: 10.414557 [I 2022-10-28 09:51:52] ...trap/engines/engine.py.224: train: epoch 119 | batch 250/2304 [I 2022-10-28 09:51:52] ...trap/engines/engine.py.228: elapsed: 0:02:23 | left: 0:19:13 [I 2022-10-28 09:51:52] ...trap/engines/engine.py.229: process: 0.54728 | load: 0.00261 [I 2022-10-28 09:51:52] ...trap/engines/engine.py.230: loss: 0.11516 [I 2022-10-28 09:51:52] ...optimizers/trijoint.py.151: total_norm: 14.640739 [I 2022-10-28 09:51:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 260/2304 [I 2022-10-28 09:51:58] ...trap/engines/engine.py.228: elapsed: 0:02:28 | left: 0:18:59 [I 2022-10-28 09:51:58] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00253 [I 2022-10-28 09:51:58] ...trap/engines/engine.py.230: loss: 0.19149 [I 2022-10-28 09:51:58] ...optimizers/trijoint.py.151: total_norm: 27.149282 [I 2022-10-28 09:52:03] ...trap/engines/engine.py.224: train: epoch 119 | batch 270/2304 [I 2022-10-28 09:52:03] ...trap/engines/engine.py.228: elapsed: 0:02:34 | left: 0:19:01 [I 2022-10-28 09:52:03] ...trap/engines/engine.py.229: process: 0.55842 | load: 0.00248 [I 2022-10-28 09:52:03] ...trap/engines/engine.py.230: loss: 0.12789 [I 2022-10-28 09:52:03] ...optimizers/trijoint.py.151: total_norm: 23.952816 [I 2022-10-28 09:52:09] ...trap/engines/engine.py.224: train: epoch 119 | batch 280/2304 [I 2022-10-28 09:52:09] ...trap/engines/engine.py.228: elapsed: 0:02:39 | left: 0:18:48 [I 2022-10-28 09:52:09] ...trap/engines/engine.py.229: process: 0.55698 | load: 0.00247 [I 2022-10-28 09:52:09] ...trap/engines/engine.py.230: loss: 0.13815 [I 2022-10-28 09:52:09] ...optimizers/trijoint.py.151: total_norm: 13.319926 [I 2022-10-28 09:52:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 290/2304 [I 2022-10-28 09:52:14] ...trap/engines/engine.py.228: elapsed: 0:02:45 | left: 0:18:43 [I 2022-10-28 09:52:14] ...trap/engines/engine.py.229: process: 0.55874 | load: 0.00245 [I 2022-10-28 09:52:14] ...trap/engines/engine.py.230: loss: 0.11233 [I 2022-10-28 09:52:14] ...optimizers/trijoint.py.151: total_norm: 24.764078 [I 2022-10-28 09:52:20] ...trap/engines/engine.py.224: train: epoch 119 | batch 300/2304 [I 2022-10-28 09:52:20] ...trap/engines/engine.py.228: elapsed: 0:02:51 | left: 0:18:42 [I 2022-10-28 09:52:20] ...trap/engines/engine.py.229: process: 0.56148 | load: 0.00253 [I 2022-10-28 09:52:20] ...trap/engines/engine.py.230: loss: 0.16474 [I 2022-10-28 09:52:20] ...optimizers/trijoint.py.151: total_norm: 8.657677 [I 2022-10-28 09:52:26] ...trap/engines/engine.py.224: train: epoch 119 | batch 310/2304 [I 2022-10-28 09:52:26] ...trap/engines/engine.py.228: elapsed: 0:02:56 | left: 0:18:39 [I 2022-10-28 09:52:26] ...trap/engines/engine.py.229: process: 0.55616 | load: 0.00256 [I 2022-10-28 09:52:26] ...trap/engines/engine.py.230: loss: 0.09441 [I 2022-10-28 09:52:26] ...optimizers/trijoint.py.151: total_norm: 13.928780 [I 2022-10-28 09:52:31] ...trap/engines/engine.py.224: train: epoch 119 | batch 320/2304 [I 2022-10-28 09:52:31] ...trap/engines/engine.py.228: elapsed: 0:03:02 | left: 0:18:34 [I 2022-10-28 09:52:31] ...trap/engines/engine.py.229: process: 0.55951 | load: 0.00331 [I 2022-10-28 09:52:31] ...trap/engines/engine.py.230: loss: 0.07383 [I 2022-10-28 09:52:31] ...optimizers/trijoint.py.151: total_norm: 13.900767 [I 2022-10-28 09:52:37] ...trap/engines/engine.py.224: train: epoch 119 | batch 330/2304 [I 2022-10-28 09:52:37] ...trap/engines/engine.py.228: elapsed: 0:03:07 | left: 0:18:29 [I 2022-10-28 09:52:37] ...trap/engines/engine.py.229: process: 0.56271 | load: 0.00244 [I 2022-10-28 09:52:37] ...trap/engines/engine.py.230: loss: 0.09527 [I 2022-10-28 09:52:37] ...optimizers/trijoint.py.151: total_norm: 11.144548 [I 2022-10-28 09:52:43] ...trap/engines/engine.py.224: train: epoch 119 | batch 340/2304 [I 2022-10-28 09:52:43] ...trap/engines/engine.py.228: elapsed: 0:03:13 | left: 0:18:14 [I 2022-10-28 09:52:43] ...trap/engines/engine.py.229: process: 0.54467 | load: 0.00306 [I 2022-10-28 09:52:43] ...trap/engines/engine.py.230: loss: 0.08220 [I 2022-10-28 09:52:43] ...optimizers/trijoint.py.151: total_norm: 9.327760 [I 2022-10-28 09:52:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 350/2304 [I 2022-10-28 09:52:48] ...trap/engines/engine.py.228: elapsed: 0:03:19 | left: 0:18:07 [I 2022-10-28 09:52:48] ...trap/engines/engine.py.229: process: 0.56728 | load: 0.00250 [I 2022-10-28 09:52:48] ...trap/engines/engine.py.230: loss: 0.14649 [I 2022-10-28 09:52:48] ...optimizers/trijoint.py.151: total_norm: 36.556656 [I 2022-10-28 09:52:54] ...trap/engines/engine.py.224: train: epoch 119 | batch 360/2304 [I 2022-10-28 09:52:54] ...trap/engines/engine.py.228: elapsed: 0:03:24 | left: 0:18:00 [I 2022-10-28 09:52:54] ...trap/engines/engine.py.229: process: 0.55320 | load: 0.00248 [I 2022-10-28 09:52:54] ...trap/engines/engine.py.230: loss: 0.09710 [I 2022-10-28 09:52:54] ...optimizers/trijoint.py.151: total_norm: 13.384030 [I 2022-10-28 09:52:59] ...trap/engines/engine.py.224: train: epoch 119 | batch 370/2304 [I 2022-10-28 09:52:59] ...trap/engines/engine.py.228: elapsed: 0:03:30 | left: 0:17:55 [I 2022-10-28 09:52:59] ...trap/engines/engine.py.229: process: 0.55801 | load: 0.00246 [I 2022-10-28 09:52:59] ...trap/engines/engine.py.230: loss: 0.09238 [I 2022-10-28 09:52:59] ...optimizers/trijoint.py.151: total_norm: 15.251187 [I 2022-10-28 09:53:05] ...trap/engines/engine.py.224: train: epoch 119 | batch 380/2304 [I 2022-10-28 09:53:05] ...trap/engines/engine.py.228: elapsed: 0:03:35 | left: 0:17:52 [I 2022-10-28 09:53:05] ...trap/engines/engine.py.229: process: 0.56410 | load: 0.00247 [I 2022-10-28 09:53:05] ...trap/engines/engine.py.230: loss: 0.16519 [I 2022-10-28 09:53:05] ...optimizers/trijoint.py.151: total_norm: 21.028507 [I 2022-10-28 09:53:10] ...trap/engines/engine.py.224: train: epoch 119 | batch 390/2304 [I 2022-10-28 09:53:10] ...trap/engines/engine.py.228: elapsed: 0:03:41 | left: 0:17:51 [I 2022-10-28 09:53:10] ...trap/engines/engine.py.229: process: 0.54688 | load: 0.00252 [I 2022-10-28 09:53:10] ...trap/engines/engine.py.230: loss: 0.14116 [I 2022-10-28 09:53:10] ...optimizers/trijoint.py.151: total_norm: 13.906587 [I 2022-10-28 09:53:16] ...trap/engines/engine.py.224: train: epoch 119 | batch 400/2304 [I 2022-10-28 09:53:16] ...trap/engines/engine.py.228: elapsed: 0:03:47 | left: 0:17:49 [I 2022-10-28 09:53:16] ...trap/engines/engine.py.229: process: 0.56786 | load: 0.00256 [I 2022-10-28 09:53:16] ...trap/engines/engine.py.230: loss: 0.10118 [I 2022-10-28 09:53:16] ...optimizers/trijoint.py.151: total_norm: 18.744459 [I 2022-10-28 09:53:22] ...trap/engines/engine.py.224: train: epoch 119 | batch 410/2304 [I 2022-10-28 09:53:22] ...trap/engines/engine.py.228: elapsed: 0:03:52 | left: 0:17:42 [I 2022-10-28 09:53:22] ...trap/engines/engine.py.229: process: 0.56323 | load: 0.00242 [I 2022-10-28 09:53:22] ...trap/engines/engine.py.230: loss: 0.18036 [I 2022-10-28 09:53:22] ...optimizers/trijoint.py.151: total_norm: 53.521217 [I 2022-10-28 09:53:27] ...trap/engines/engine.py.224: train: epoch 119 | batch 420/2304 [I 2022-10-28 09:53:27] ...trap/engines/engine.py.228: elapsed: 0:03:58 | left: 0:17:35 [I 2022-10-28 09:53:27] ...trap/engines/engine.py.229: process: 0.56556 | load: 0.00245 [I 2022-10-28 09:53:27] ...trap/engines/engine.py.230: loss: 0.18789 [I 2022-10-28 09:53:27] ...optimizers/trijoint.py.151: total_norm: 20.917164 [I 2022-10-28 09:53:33] ...trap/engines/engine.py.224: train: epoch 119 | batch 430/2304 [I 2022-10-28 09:53:33] ...trap/engines/engine.py.228: elapsed: 0:04:03 | left: 0:17:31 [I 2022-10-28 09:53:33] ...trap/engines/engine.py.229: process: 0.56077 | load: 0.00245 [I 2022-10-28 09:53:33] ...trap/engines/engine.py.230: loss: 0.12947 [I 2022-10-28 09:53:33] ...optimizers/trijoint.py.151: total_norm: 14.808016 [I 2022-10-28 09:53:38] ...trap/engines/engine.py.224: train: epoch 119 | batch 440/2304 [I 2022-10-28 09:53:38] ...trap/engines/engine.py.228: elapsed: 0:04:09 | left: 0:17:16 [I 2022-10-28 09:53:38] ...trap/engines/engine.py.229: process: 0.55546 | load: 0.00243 [I 2022-10-28 09:53:38] ...trap/engines/engine.py.230: loss: 0.13575 [I 2022-10-28 09:53:38] ...optimizers/trijoint.py.151: total_norm: 16.028118 [I 2022-10-28 09:53:44] ...trap/engines/engine.py.224: train: epoch 119 | batch 450/2304 [I 2022-10-28 09:53:44] ...trap/engines/engine.py.228: elapsed: 0:04:15 | left: 0:17:17 [I 2022-10-28 09:53:44] ...trap/engines/engine.py.229: process: 0.56455 | load: 0.00244 [I 2022-10-28 09:53:44] ...trap/engines/engine.py.230: loss: 0.14781 [I 2022-10-28 09:53:44] ...optimizers/trijoint.py.151: total_norm: 42.875820 [I 2022-10-28 09:53:50] ...trap/engines/engine.py.224: train: epoch 119 | batch 460/2304 [I 2022-10-28 09:53:50] ...trap/engines/engine.py.228: elapsed: 0:04:20 | left: 0:17:11 [I 2022-10-28 09:53:50] ...trap/engines/engine.py.229: process: 0.57474 | load: 0.00262 [I 2022-10-28 09:53:50] ...trap/engines/engine.py.230: loss: 0.11914 [I 2022-10-28 09:53:50] ...optimizers/trijoint.py.151: total_norm: 12.300520 [I 2022-10-28 09:53:55] ...trap/engines/engine.py.224: train: epoch 119 | batch 470/2304 [I 2022-10-28 09:53:55] ...trap/engines/engine.py.228: elapsed: 0:04:26 | left: 0:17:08 [I 2022-10-28 09:53:55] ...trap/engines/engine.py.229: process: 0.56085 | load: 0.00241 [I 2022-10-28 09:53:55] ...trap/engines/engine.py.230: loss: 0.14398 [I 2022-10-28 09:53:55] ...optimizers/trijoint.py.151: total_norm: 19.168648 [I 2022-10-28 09:54:01] ...trap/engines/engine.py.224: train: epoch 119 | batch 480/2304 [I 2022-10-28 09:54:01] ...trap/engines/engine.py.228: elapsed: 0:04:31 | left: 0:16:59 [I 2022-10-28 09:54:01] ...trap/engines/engine.py.229: process: 0.55914 | load: 0.00242 [I 2022-10-28 09:54:01] ...trap/engines/engine.py.230: loss: 0.09466 [I 2022-10-28 09:54:01] ...optimizers/trijoint.py.151: total_norm: 11.268565 [I 2022-10-28 09:54:07] ...trap/engines/engine.py.224: train: epoch 119 | batch 490/2304 [I 2022-10-28 09:54:07] ...trap/engines/engine.py.228: elapsed: 0:04:37 | left: 0:16:58 [I 2022-10-28 09:54:07] ...trap/engines/engine.py.229: process: 0.55627 | load: 0.00249 [I 2022-10-28 09:54:07] ...trap/engines/engine.py.230: loss: 0.10638 [I 2022-10-28 09:54:07] ...optimizers/trijoint.py.151: total_norm: 16.278294 [I 2022-10-28 09:54:12] ...trap/engines/engine.py.224: train: epoch 119 | batch 500/2304 [I 2022-10-28 09:54:12] ...trap/engines/engine.py.228: elapsed: 0:04:43 | left: 0:16:51 [I 2022-10-28 09:54:12] ...trap/engines/engine.py.229: process: 0.56443 | load: 0.00260 [I 2022-10-28 09:54:12] ...trap/engines/engine.py.230: loss: 0.18045 [I 2022-10-28 09:54:12] ...optimizers/trijoint.py.151: total_norm: 26.526691 [I 2022-10-28 09:54:18] ...trap/engines/engine.py.224: train: epoch 119 | batch 510/2304 [I 2022-10-28 09:54:18] ...trap/engines/engine.py.228: elapsed: 0:04:48 | left: 0:16:47 [I 2022-10-28 09:54:18] ...trap/engines/engine.py.229: process: 0.56558 | load: 0.00241 [I 2022-10-28 09:54:18] ...trap/engines/engine.py.230: loss: 0.08030 [I 2022-10-28 09:54:18] ...optimizers/trijoint.py.151: total_norm: 12.670902 [I 2022-10-28 09:54:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 520/2304 [I 2022-10-28 09:54:23] ...trap/engines/engine.py.228: elapsed: 0:04:54 | left: 0:16:40 [I 2022-10-28 09:54:23] ...trap/engines/engine.py.229: process: 0.56241 | load: 0.00243 [I 2022-10-28 09:54:23] ...trap/engines/engine.py.230: loss: 0.10535 [I 2022-10-28 09:54:23] ...optimizers/trijoint.py.151: total_norm: 9.440391 [I 2022-10-28 09:54:29] ...trap/engines/engine.py.224: train: epoch 119 | batch 530/2304 [I 2022-10-28 09:54:29] ...trap/engines/engine.py.228: elapsed: 0:05:00 | left: 0:16:36 [I 2022-10-28 09:54:29] ...trap/engines/engine.py.229: process: 0.54864 | load: 0.00243 [I 2022-10-28 09:54:29] ...trap/engines/engine.py.230: loss: 0.09138 [I 2022-10-28 09:54:29] ...optimizers/trijoint.py.151: total_norm: 15.223319 [I 2022-10-28 09:54:35] ...trap/engines/engine.py.224: train: epoch 119 | batch 540/2304 [I 2022-10-28 09:54:35] ...trap/engines/engine.py.228: elapsed: 0:05:05 | left: 0:16:21 [I 2022-10-28 09:54:35] ...trap/engines/engine.py.229: process: 0.54797 | load: 0.00244 [I 2022-10-28 09:54:35] ...trap/engines/engine.py.230: loss: 0.13395 [I 2022-10-28 09:54:35] ...optimizers/trijoint.py.151: total_norm: 12.910266 [I 2022-10-28 09:54:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 550/2304 [I 2022-10-28 09:54:40] ...trap/engines/engine.py.228: elapsed: 0:05:11 | left: 0:16:23 [I 2022-10-28 09:54:40] ...trap/engines/engine.py.229: process: 0.56215 | load: 0.00250 [I 2022-10-28 09:54:40] ...trap/engines/engine.py.230: loss: 0.09585 [I 2022-10-28 09:54:40] ...optimizers/trijoint.py.151: total_norm: 10.859798 [I 2022-10-28 09:54:46] ...trap/engines/engine.py.224: train: epoch 119 | batch 560/2304 [I 2022-10-28 09:54:46] ...trap/engines/engine.py.228: elapsed: 0:05:16 | left: 0:16:15 [I 2022-10-28 09:54:46] ...trap/engines/engine.py.229: process: 0.55380 | load: 0.00250 [I 2022-10-28 09:54:46] ...trap/engines/engine.py.230: loss: 0.14372 [I 2022-10-28 09:54:46] ...optimizers/trijoint.py.151: total_norm: 19.709284 [I 2022-10-28 09:54:51] ...trap/engines/engine.py.224: train: epoch 119 | batch 570/2304 [I 2022-10-28 09:54:51] ...trap/engines/engine.py.228: elapsed: 0:05:22 | left: 0:16:13 [I 2022-10-28 09:54:51] ...trap/engines/engine.py.229: process: 0.56007 | load: 0.00252 [I 2022-10-28 09:54:51] ...trap/engines/engine.py.230: loss: 0.14072 [I 2022-10-28 09:54:51] ...optimizers/trijoint.py.151: total_norm: 15.255843 [I 2022-10-28 09:54:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 580/2304 [I 2022-10-28 09:54:57] ...trap/engines/engine.py.228: elapsed: 0:05:28 | left: 0:16:08 [I 2022-10-28 09:54:57] ...trap/engines/engine.py.229: process: 0.56980 | load: 0.00248 [I 2022-10-28 09:54:57] ...trap/engines/engine.py.230: loss: 0.13549 [I 2022-10-28 09:54:57] ...optimizers/trijoint.py.151: total_norm: 28.532557 [I 2022-10-28 09:55:03] ...trap/engines/engine.py.224: train: epoch 119 | batch 590/2304 [I 2022-10-28 09:55:03] ...trap/engines/engine.py.228: elapsed: 0:05:33 | left: 0:16:02 [I 2022-10-28 09:55:03] ...trap/engines/engine.py.229: process: 0.56304 | load: 0.00244 [I 2022-10-28 09:55:03] ...trap/engines/engine.py.230: loss: 0.08802 [I 2022-10-28 09:55:03] ...optimizers/trijoint.py.151: total_norm: 8.496905 [I 2022-10-28 09:55:08] ...trap/engines/engine.py.224: train: epoch 119 | batch 600/2304 [I 2022-10-28 09:55:08] ...trap/engines/engine.py.228: elapsed: 0:05:39 | left: 0:15:49 [I 2022-10-28 09:55:08] ...trap/engines/engine.py.229: process: 0.55403 | load: 0.00241 [I 2022-10-28 09:55:08] ...trap/engines/engine.py.230: loss: 0.10611 [I 2022-10-28 09:55:08] ...optimizers/trijoint.py.151: total_norm: 16.799700 [I 2022-10-28 09:55:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 610/2304 [I 2022-10-28 09:55:14] ...trap/engines/engine.py.228: elapsed: 0:05:44 | left: 0:15:46 [I 2022-10-28 09:55:14] ...trap/engines/engine.py.229: process: 0.55711 | load: 0.00242 [I 2022-10-28 09:55:14] ...trap/engines/engine.py.230: loss: 0.08899 [I 2022-10-28 09:55:14] ...optimizers/trijoint.py.151: total_norm: 12.208128 [I 2022-10-28 09:55:20] ...trap/engines/engine.py.224: train: epoch 119 | batch 620/2304 [I 2022-10-28 09:55:20] ...trap/engines/engine.py.228: elapsed: 0:05:50 | left: 0:15:44 [I 2022-10-28 09:55:20] ...trap/engines/engine.py.229: process: 0.56146 | load: 0.00242 [I 2022-10-28 09:55:20] ...trap/engines/engine.py.230: loss: 0.09740 [I 2022-10-28 09:55:20] ...optimizers/trijoint.py.151: total_norm: 31.653439 [I 2022-10-28 09:55:25] ...trap/engines/engine.py.224: train: epoch 119 | batch 630/2304 [I 2022-10-28 09:55:25] ...trap/engines/engine.py.228: elapsed: 0:05:56 | left: 0:15:38 [I 2022-10-28 09:55:25] ...trap/engines/engine.py.229: process: 0.56081 | load: 0.00243 [I 2022-10-28 09:55:25] ...trap/engines/engine.py.230: loss: 0.07950 [I 2022-10-28 09:55:25] ...optimizers/trijoint.py.151: total_norm: 11.077658 [I 2022-10-28 09:55:31] ...trap/engines/engine.py.224: train: epoch 119 | batch 640/2304 [I 2022-10-28 09:55:31] ...trap/engines/engine.py.228: elapsed: 0:06:01 | left: 0:15:28 [I 2022-10-28 09:55:31] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00242 [I 2022-10-28 09:55:31] ...trap/engines/engine.py.230: loss: 0.09239 [I 2022-10-28 09:55:31] ...optimizers/trijoint.py.151: total_norm: 24.196346 [I 2022-10-28 09:55:36] ...trap/engines/engine.py.224: train: epoch 119 | batch 650/2304 [I 2022-10-28 09:55:36] ...trap/engines/engine.py.228: elapsed: 0:06:07 | left: 0:15:18 [I 2022-10-28 09:55:36] ...trap/engines/engine.py.229: process: 0.54681 | load: 0.00240 [I 2022-10-28 09:55:36] ...trap/engines/engine.py.230: loss: 0.11755 [I 2022-10-28 09:55:36] ...optimizers/trijoint.py.151: total_norm: 14.346282 [I 2022-10-28 09:55:42] ...trap/engines/engine.py.224: train: epoch 119 | batch 660/2304 [I 2022-10-28 09:55:42] ...trap/engines/engine.py.228: elapsed: 0:06:12 | left: 0:15:12 [I 2022-10-28 09:55:42] ...trap/engines/engine.py.229: process: 0.54518 | load: 0.00241 [I 2022-10-28 09:55:42] ...trap/engines/engine.py.230: loss: 0.11933 [I 2022-10-28 09:55:42] ...optimizers/trijoint.py.151: total_norm: 26.666023 [I 2022-10-28 09:55:47] ...trap/engines/engine.py.224: train: epoch 119 | batch 670/2304 [I 2022-10-28 09:55:47] ...trap/engines/engine.py.228: elapsed: 0:06:18 | left: 0:15:10 [I 2022-10-28 09:55:47] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00240 [I 2022-10-28 09:55:47] ...trap/engines/engine.py.230: loss: 0.07170 [I 2022-10-28 09:55:47] ...optimizers/trijoint.py.151: total_norm: 11.952861 [I 2022-10-28 09:55:53] ...trap/engines/engine.py.224: train: epoch 119 | batch 680/2304 [I 2022-10-28 09:55:53] ...trap/engines/engine.py.228: elapsed: 0:06:24 | left: 0:15:06 [I 2022-10-28 09:55:53] ...trap/engines/engine.py.229: process: 0.54911 | load: 0.00239 [I 2022-10-28 09:55:53] ...trap/engines/engine.py.230: loss: 0.08244 [I 2022-10-28 09:55:53] ...optimizers/trijoint.py.151: total_norm: 41.393482 [I 2022-10-28 09:55:59] ...trap/engines/engine.py.224: train: epoch 119 | batch 690/2304 [I 2022-10-28 09:55:59] ...trap/engines/engine.py.228: elapsed: 0:06:29 | left: 0:14:58 [I 2022-10-28 09:55:59] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00242 [I 2022-10-28 09:55:59] ...trap/engines/engine.py.230: loss: 0.10807 [I 2022-10-28 09:55:59] ...optimizers/trijoint.py.151: total_norm: 16.465965 [I 2022-10-28 09:56:04] ...trap/engines/engine.py.224: train: epoch 119 | batch 700/2304 [I 2022-10-28 09:56:04] ...trap/engines/engine.py.228: elapsed: 0:06:35 | left: 0:14:48 [I 2022-10-28 09:56:04] ...trap/engines/engine.py.229: process: 0.54523 | load: 0.00242 [I 2022-10-28 09:56:04] ...trap/engines/engine.py.230: loss: 0.13127 [I 2022-10-28 09:56:04] ...optimizers/trijoint.py.151: total_norm: 19.741543 [I 2022-10-28 09:56:10] ...trap/engines/engine.py.224: train: epoch 119 | batch 710/2304 [I 2022-10-28 09:56:10] ...trap/engines/engine.py.228: elapsed: 0:06:40 | left: 0:14:36 [I 2022-10-28 09:56:10] ...trap/engines/engine.py.229: process: 0.54682 | load: 0.00241 [I 2022-10-28 09:56:10] ...trap/engines/engine.py.230: loss: 0.09507 [I 2022-10-28 09:56:10] ...optimizers/trijoint.py.151: total_norm: 14.188452 [I 2022-10-28 09:56:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 720/2304 [I 2022-10-28 09:56:15] ...trap/engines/engine.py.228: elapsed: 0:06:46 | left: 0:14:38 [I 2022-10-28 09:56:15] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00242 [I 2022-10-28 09:56:15] ...trap/engines/engine.py.230: loss: 0.11909 [I 2022-10-28 09:56:15] ...optimizers/trijoint.py.151: total_norm: 17.045246 [I 2022-10-28 09:56:21] ...trap/engines/engine.py.224: train: epoch 119 | batch 730/2304 [I 2022-10-28 09:56:21] ...trap/engines/engine.py.228: elapsed: 0:06:51 | left: 0:14:30 [I 2022-10-28 09:56:21] ...trap/engines/engine.py.229: process: 0.55788 | load: 0.00240 [I 2022-10-28 09:56:21] ...trap/engines/engine.py.230: loss: 0.10052 [I 2022-10-28 09:56:21] ...optimizers/trijoint.py.151: total_norm: 17.619322 [I 2022-10-28 09:56:26] ...trap/engines/engine.py.224: train: epoch 119 | batch 740/2304 [I 2022-10-28 09:56:26] ...trap/engines/engine.py.228: elapsed: 0:06:57 | left: 0:14:34 [I 2022-10-28 09:56:26] ...trap/engines/engine.py.229: process: 0.55904 | load: 0.00240 [I 2022-10-28 09:56:26] ...trap/engines/engine.py.230: loss: 0.13473 [I 2022-10-28 09:56:26] ...optimizers/trijoint.py.151: total_norm: 18.975554 [I 2022-10-28 09:56:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 750/2304 [I 2022-10-28 09:56:32] ...trap/engines/engine.py.228: elapsed: 0:07:03 | left: 0:14:25 [I 2022-10-28 09:56:32] ...trap/engines/engine.py.229: process: 0.55911 | load: 0.00259 [I 2022-10-28 09:56:32] ...trap/engines/engine.py.230: loss: 0.10096 [I 2022-10-28 09:56:32] ...optimizers/trijoint.py.151: total_norm: 47.837261 [I 2022-10-28 09:56:38] ...trap/engines/engine.py.224: train: epoch 119 | batch 760/2304 [I 2022-10-28 09:56:38] ...trap/engines/engine.py.228: elapsed: 0:07:08 | left: 0:14:24 [I 2022-10-28 09:56:38] ...trap/engines/engine.py.229: process: 0.55540 | load: 0.00247 [I 2022-10-28 09:56:38] ...trap/engines/engine.py.230: loss: 0.08905 [I 2022-10-28 09:56:38] ...optimizers/trijoint.py.151: total_norm: 12.981575 [I 2022-10-28 09:56:43] ...trap/engines/engine.py.224: train: epoch 119 | batch 770/2304 [I 2022-10-28 09:56:43] ...trap/engines/engine.py.228: elapsed: 0:07:14 | left: 0:14:13 [I 2022-10-28 09:56:43] ...trap/engines/engine.py.229: process: 0.55448 | load: 0.00245 [I 2022-10-28 09:56:43] ...trap/engines/engine.py.230: loss: 0.09187 [I 2022-10-28 09:56:43] ...optimizers/trijoint.py.151: total_norm: 10.283651 [I 2022-10-28 09:56:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 780/2304 [I 2022-10-28 09:56:49] ...trap/engines/engine.py.228: elapsed: 0:07:19 | left: 0:14:14 [I 2022-10-28 09:56:49] ...trap/engines/engine.py.229: process: 0.57190 | load: 0.00243 [I 2022-10-28 09:56:49] ...trap/engines/engine.py.230: loss: 0.09641 [I 2022-10-28 09:56:49] ...optimizers/trijoint.py.151: total_norm: 11.299486 [I 2022-10-28 09:56:54] ...trap/engines/engine.py.224: train: epoch 119 | batch 790/2304 [I 2022-10-28 09:56:54] ...trap/engines/engine.py.228: elapsed: 0:07:25 | left: 0:14:04 [I 2022-10-28 09:56:54] ...trap/engines/engine.py.229: process: 0.55948 | load: 0.00242 [I 2022-10-28 09:56:54] ...trap/engines/engine.py.230: loss: 0.07275 [I 2022-10-28 09:56:54] ...optimizers/trijoint.py.151: total_norm: 24.279903 [I 2022-10-28 09:57:00] ...trap/engines/engine.py.224: train: epoch 119 | batch 800/2304 [I 2022-10-28 09:57:00] ...trap/engines/engine.py.228: elapsed: 0:07:31 | left: 0:13:51 [I 2022-10-28 09:57:00] ...trap/engines/engine.py.229: process: 0.54800 | load: 0.00248 [I 2022-10-28 09:57:00] ...trap/engines/engine.py.230: loss: 0.18909 [I 2022-10-28 09:57:00] ...optimizers/trijoint.py.151: total_norm: 19.852285 [I 2022-10-28 09:57:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 810/2304 [I 2022-10-28 09:57:06] ...trap/engines/engine.py.228: elapsed: 0:07:36 | left: 0:13:54 [I 2022-10-28 09:57:06] ...trap/engines/engine.py.229: process: 0.55853 | load: 0.00245 [I 2022-10-28 09:57:06] ...trap/engines/engine.py.230: loss: 0.10806 [I 2022-10-28 09:57:06] ...optimizers/trijoint.py.151: total_norm: 34.114689 [I 2022-10-28 09:57:11] ...trap/engines/engine.py.224: train: epoch 119 | batch 820/2304 [I 2022-10-28 09:57:11] ...trap/engines/engine.py.228: elapsed: 0:07:42 | left: 0:14:06 [I 2022-10-28 09:57:11] ...trap/engines/engine.py.229: process: 0.54507 | load: 0.00244 [I 2022-10-28 09:57:11] ...trap/engines/engine.py.230: loss: 0.10932 [I 2022-10-28 09:57:11] ...optimizers/trijoint.py.151: total_norm: 8.921096 [I 2022-10-28 09:57:17] ...trap/engines/engine.py.224: train: epoch 119 | batch 830/2304 [I 2022-10-28 09:57:17] ...trap/engines/engine.py.228: elapsed: 0:07:47 | left: 0:13:46 [I 2022-10-28 09:57:17] ...trap/engines/engine.py.229: process: 0.56454 | load: 0.00256 [I 2022-10-28 09:57:17] ...trap/engines/engine.py.230: loss: 0.11630 [I 2022-10-28 09:57:17] ...optimizers/trijoint.py.151: total_norm: 17.749434 [I 2022-10-28 09:57:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 840/2304 [I 2022-10-28 09:57:23] ...trap/engines/engine.py.228: elapsed: 0:07:53 | left: 0:13:57 [I 2022-10-28 09:57:23] ...trap/engines/engine.py.229: process: 0.54898 | load: 0.00247 [I 2022-10-28 09:57:23] ...trap/engines/engine.py.230: loss: 0.08231 [I 2022-10-28 09:57:23] ...optimizers/trijoint.py.151: total_norm: 11.920316 [I 2022-10-28 09:57:28] ...trap/engines/engine.py.224: train: epoch 119 | batch 850/2304 [I 2022-10-28 09:57:28] ...trap/engines/engine.py.228: elapsed: 0:07:59 | left: 0:13:34 [I 2022-10-28 09:57:28] ...trap/engines/engine.py.229: process: 0.56781 | load: 0.00250 [I 2022-10-28 09:57:28] ...trap/engines/engine.py.230: loss: 0.14957 [I 2022-10-28 09:57:28] ...optimizers/trijoint.py.151: total_norm: 14.898097 [I 2022-10-28 09:57:34] ...trap/engines/engine.py.224: train: epoch 119 | batch 860/2304 [I 2022-10-28 09:57:34] ...trap/engines/engine.py.228: elapsed: 0:08:04 | left: 0:13:23 [I 2022-10-28 09:57:34] ...trap/engines/engine.py.229: process: 0.54320 | load: 0.00248 [I 2022-10-28 09:57:34] ...trap/engines/engine.py.230: loss: 0.11285 [I 2022-10-28 09:57:34] ...optimizers/trijoint.py.151: total_norm: 13.337652 [I 2022-10-28 09:57:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 870/2304 [I 2022-10-28 09:57:40] ...trap/engines/engine.py.228: elapsed: 0:08:10 | left: 0:13:20 [I 2022-10-28 09:57:40] ...trap/engines/engine.py.229: process: 0.56140 | load: 0.00246 [I 2022-10-28 09:57:40] ...trap/engines/engine.py.230: loss: 0.09516 [I 2022-10-28 09:57:40] ...optimizers/trijoint.py.151: total_norm: 14.926156 [I 2022-10-28 09:57:45] ...trap/engines/engine.py.224: train: epoch 119 | batch 880/2304 [I 2022-10-28 09:57:45] ...trap/engines/engine.py.228: elapsed: 0:08:16 | left: 0:13:12 [I 2022-10-28 09:57:45] ...trap/engines/engine.py.229: process: 0.55813 | load: 0.00247 [I 2022-10-28 09:57:45] ...trap/engines/engine.py.230: loss: 0.14195 [I 2022-10-28 09:57:45] ...optimizers/trijoint.py.151: total_norm: 10.904281 [I 2022-10-28 09:57:51] ...trap/engines/engine.py.224: train: epoch 119 | batch 890/2304 [I 2022-10-28 09:57:51] ...trap/engines/engine.py.228: elapsed: 0:08:21 | left: 0:13:07 [I 2022-10-28 09:57:51] ...trap/engines/engine.py.229: process: 0.55768 | load: 0.00243 [I 2022-10-28 09:57:51] ...trap/engines/engine.py.230: loss: 0.16879 [I 2022-10-28 09:57:51] ...optimizers/trijoint.py.151: total_norm: 14.262514 [I 2022-10-28 09:57:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 900/2304 [I 2022-10-28 09:57:56] ...trap/engines/engine.py.228: elapsed: 0:08:27 | left: 0:12:59 [I 2022-10-28 09:57:56] ...trap/engines/engine.py.229: process: 0.55595 | load: 0.00241 [I 2022-10-28 09:57:56] ...trap/engines/engine.py.230: loss: 0.10115 [I 2022-10-28 09:57:56] ...optimizers/trijoint.py.151: total_norm: 13.935639 [I 2022-10-28 09:58:02] ...trap/engines/engine.py.224: train: epoch 119 | batch 910/2304 [I 2022-10-28 09:58:02] ...trap/engines/engine.py.228: elapsed: 0:08:32 | left: 0:13:11 [I 2022-10-28 09:58:02] ...trap/engines/engine.py.229: process: 0.57702 | load: 0.00244 [I 2022-10-28 09:58:02] ...trap/engines/engine.py.230: loss: 0.11033 [I 2022-10-28 09:58:02] ...optimizers/trijoint.py.151: total_norm: 26.243246 [I 2022-10-28 09:58:08] ...trap/engines/engine.py.224: train: epoch 119 | batch 920/2304 [I 2022-10-28 09:58:08] ...trap/engines/engine.py.228: elapsed: 0:08:38 | left: 0:12:47 [I 2022-10-28 09:58:08] ...trap/engines/engine.py.229: process: 0.53830 | load: 0.00241 [I 2022-10-28 09:58:08] ...trap/engines/engine.py.230: loss: 0.11931 [I 2022-10-28 09:58:08] ...optimizers/trijoint.py.151: total_norm: 37.823715 [I 2022-10-28 09:58:13] ...trap/engines/engine.py.224: train: epoch 119 | batch 930/2304 [I 2022-10-28 09:58:13] ...trap/engines/engine.py.228: elapsed: 0:08:44 | left: 0:12:44 [I 2022-10-28 09:58:13] ...trap/engines/engine.py.229: process: 0.56020 | load: 0.00240 [I 2022-10-28 09:58:13] ...trap/engines/engine.py.230: loss: 0.24704 [I 2022-10-28 09:58:13] ...optimizers/trijoint.py.151: total_norm: 30.179672 [I 2022-10-28 09:58:19] ...trap/engines/engine.py.224: train: epoch 119 | batch 940/2304 [I 2022-10-28 09:58:19] ...trap/engines/engine.py.228: elapsed: 0:08:49 | left: 0:12:43 [I 2022-10-28 09:58:19] ...trap/engines/engine.py.229: process: 0.56662 | load: 0.00248 [I 2022-10-28 09:58:19] ...trap/engines/engine.py.230: loss: 0.08292 [I 2022-10-28 09:58:19] ...optimizers/trijoint.py.151: total_norm: 22.129642 [I 2022-10-28 09:58:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 950/2304 [I 2022-10-28 09:58:24] ...trap/engines/engine.py.228: elapsed: 0:08:55 | left: 0:12:40 [I 2022-10-28 09:58:24] ...trap/engines/engine.py.229: process: 0.56610 | load: 0.00245 [I 2022-10-28 09:58:24] ...trap/engines/engine.py.230: loss: 0.17301 [I 2022-10-28 09:58:24] ...optimizers/trijoint.py.151: total_norm: 19.827644 [I 2022-10-28 09:58:30] ...trap/engines/engine.py.224: train: epoch 119 | batch 960/2304 [I 2022-10-28 09:58:30] ...trap/engines/engine.py.228: elapsed: 0:09:00 | left: 0:12:29 [I 2022-10-28 09:58:30] ...trap/engines/engine.py.229: process: 0.56270 | load: 0.00242 [I 2022-10-28 09:58:30] ...trap/engines/engine.py.230: loss: 0.14540 [I 2022-10-28 09:58:30] ...optimizers/trijoint.py.151: total_norm: 36.400051 [I 2022-10-28 09:58:36] ...trap/engines/engine.py.224: train: epoch 119 | batch 970/2304 [I 2022-10-28 09:58:36] ...trap/engines/engine.py.228: elapsed: 0:09:06 | left: 0:12:37 [I 2022-10-28 09:58:36] ...trap/engines/engine.py.229: process: 0.56380 | load: 0.00244 [I 2022-10-28 09:58:36] ...trap/engines/engine.py.230: loss: 0.12937 [I 2022-10-28 09:58:36] ...optimizers/trijoint.py.151: total_norm: 14.846183 [I 2022-10-28 09:58:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 980/2304 [I 2022-10-28 09:58:41] ...trap/engines/engine.py.228: elapsed: 0:09:12 | left: 0:12:22 [I 2022-10-28 09:58:41] ...trap/engines/engine.py.229: process: 0.56261 | load: 0.00242 [I 2022-10-28 09:58:41] ...trap/engines/engine.py.230: loss: 0.10972 [I 2022-10-28 09:58:41] ...optimizers/trijoint.py.151: total_norm: 14.832852 [I 2022-10-28 09:58:47] ...trap/engines/engine.py.224: train: epoch 119 | batch 990/2304 [I 2022-10-28 09:58:47] ...trap/engines/engine.py.228: elapsed: 0:09:17 | left: 0:12:12 [I 2022-10-28 09:58:47] ...trap/engines/engine.py.229: process: 0.55498 | load: 0.00244 [I 2022-10-28 09:58:47] ...trap/engines/engine.py.230: loss: 0.10188 [I 2022-10-28 09:58:47] ...optimizers/trijoint.py.151: total_norm: 12.164215 [I 2022-10-28 09:58:52] ...trap/engines/engine.py.224: train: epoch 119 | batch 1000/2304 [I 2022-10-28 09:58:52] ...trap/engines/engine.py.228: elapsed: 0:09:23 | left: 0:12:11 [I 2022-10-28 09:58:52] ...trap/engines/engine.py.229: process: 0.56543 | load: 0.00244 [I 2022-10-28 09:58:52] ...trap/engines/engine.py.230: loss: 0.09576 [I 2022-10-28 09:58:52] ...optimizers/trijoint.py.151: total_norm: 14.774612 [I 2022-10-28 09:58:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 1010/2304 [I 2022-10-28 09:58:58] ...trap/engines/engine.py.228: elapsed: 0:09:29 | left: 0:11:57 [I 2022-10-28 09:58:58] ...trap/engines/engine.py.229: process: 0.56129 | load: 0.00248 [I 2022-10-28 09:58:58] ...trap/engines/engine.py.230: loss: 0.07704 [I 2022-10-28 09:58:58] ...optimizers/trijoint.py.151: total_norm: 11.005344 [I 2022-10-28 09:59:04] ...trap/engines/engine.py.224: train: epoch 119 | batch 1020/2304 [I 2022-10-28 09:59:04] ...trap/engines/engine.py.228: elapsed: 0:09:34 | left: 0:11:57 [I 2022-10-28 09:59:04] ...trap/engines/engine.py.229: process: 0.55225 | load: 0.00245 [I 2022-10-28 09:59:04] ...trap/engines/engine.py.230: loss: 0.12094 [I 2022-10-28 09:59:04] ...optimizers/trijoint.py.151: total_norm: 14.515552 [I 2022-10-28 09:59:09] ...trap/engines/engine.py.224: train: epoch 119 | batch 1030/2304 [I 2022-10-28 09:59:09] ...trap/engines/engine.py.228: elapsed: 0:09:40 | left: 0:11:53 [I 2022-10-28 09:59:09] ...trap/engines/engine.py.229: process: 0.55692 | load: 0.00248 [I 2022-10-28 09:59:09] ...trap/engines/engine.py.230: loss: 0.07294 [I 2022-10-28 09:59:09] ...optimizers/trijoint.py.151: total_norm: 20.276485 [I 2022-10-28 09:59:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 1040/2304 [I 2022-10-28 09:59:15] ...trap/engines/engine.py.228: elapsed: 0:09:45 | left: 0:11:41 [I 2022-10-28 09:59:15] ...trap/engines/engine.py.229: process: 0.55791 | load: 0.00243 [I 2022-10-28 09:59:15] ...trap/engines/engine.py.230: loss: 0.09983 [I 2022-10-28 09:59:15] ...optimizers/trijoint.py.151: total_norm: 20.449965 [I 2022-10-28 09:59:20] ...trap/engines/engine.py.224: train: epoch 119 | batch 1050/2304 [I 2022-10-28 09:59:20] ...trap/engines/engine.py.228: elapsed: 0:09:51 | left: 0:11:40 [I 2022-10-28 09:59:20] ...trap/engines/engine.py.229: process: 0.57305 | load: 0.00243 [I 2022-10-28 09:59:20] ...trap/engines/engine.py.230: loss: 0.13772 [I 2022-10-28 09:59:20] ...optimizers/trijoint.py.151: total_norm: 15.413933 [I 2022-10-28 09:59:26] ...trap/engines/engine.py.224: train: epoch 119 | batch 1060/2304 [I 2022-10-28 09:59:26] ...trap/engines/engine.py.228: elapsed: 0:09:56 | left: 0:11:35 [I 2022-10-28 09:59:26] ...trap/engines/engine.py.229: process: 0.56334 | load: 0.00240 [I 2022-10-28 09:59:26] ...trap/engines/engine.py.230: loss: 0.08071 [I 2022-10-28 09:59:26] ...optimizers/trijoint.py.151: total_norm: 10.725919 [I 2022-10-28 09:59:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 1070/2304 [I 2022-10-28 09:59:32] ...trap/engines/engine.py.228: elapsed: 0:10:02 | left: 0:11:25 [I 2022-10-28 09:59:32] ...trap/engines/engine.py.229: process: 0.55205 | load: 0.00241 [I 2022-10-28 09:59:32] ...trap/engines/engine.py.230: loss: 0.15551 [I 2022-10-28 09:59:32] ...optimizers/trijoint.py.151: total_norm: 22.387672 [I 2022-10-28 09:59:37] ...trap/engines/engine.py.224: train: epoch 119 | batch 1080/2304 [I 2022-10-28 09:59:37] ...trap/engines/engine.py.228: elapsed: 0:10:08 | left: 0:11:24 [I 2022-10-28 09:59:37] ...trap/engines/engine.py.229: process: 0.56762 | load: 0.00243 [I 2022-10-28 09:59:37] ...trap/engines/engine.py.230: loss: 0.11395 [I 2022-10-28 09:59:37] ...optimizers/trijoint.py.151: total_norm: 17.112673 [I 2022-10-28 09:59:43] ...trap/engines/engine.py.224: train: epoch 119 | batch 1090/2304 [I 2022-10-28 09:59:43] ...trap/engines/engine.py.228: elapsed: 0:10:13 | left: 0:11:18 [I 2022-10-28 09:59:43] ...trap/engines/engine.py.229: process: 0.56420 | load: 0.00242 [I 2022-10-28 09:59:43] ...trap/engines/engine.py.230: loss: 0.10098 [I 2022-10-28 09:59:43] ...optimizers/trijoint.py.151: total_norm: 14.503450 [I 2022-10-28 09:59:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 1100/2304 [I 2022-10-28 09:59:48] ...trap/engines/engine.py.228: elapsed: 0:10:19 | left: 0:11:08 [I 2022-10-28 09:59:48] ...trap/engines/engine.py.229: process: 0.54952 | load: 0.00248 [I 2022-10-28 09:59:48] ...trap/engines/engine.py.230: loss: 0.08714 [I 2022-10-28 09:59:48] ...optimizers/trijoint.py.151: total_norm: 17.395601 [I 2022-10-28 09:59:54] ...trap/engines/engine.py.224: train: epoch 119 | batch 1110/2304 [I 2022-10-28 09:59:54] ...trap/engines/engine.py.228: elapsed: 0:10:24 | left: 0:11:07 [I 2022-10-28 09:59:54] ...trap/engines/engine.py.229: process: 0.56678 | load: 0.00241 [I 2022-10-28 09:59:54] ...trap/engines/engine.py.230: loss: 0.12848 [I 2022-10-28 09:59:54] ...optimizers/trijoint.py.151: total_norm: 17.304708 [I 2022-10-28 10:00:00] ...trap/engines/engine.py.224: train: epoch 119 | batch 1120/2304 [I 2022-10-28 10:00:00] ...trap/engines/engine.py.228: elapsed: 0:10:30 | left: 0:10:59 [I 2022-10-28 10:00:00] ...trap/engines/engine.py.229: process: 0.55424 | load: 0.00247 [I 2022-10-28 10:00:00] ...trap/engines/engine.py.230: loss: 0.09156 [I 2022-10-28 10:00:00] ...optimizers/trijoint.py.151: total_norm: 23.086849 [I 2022-10-28 10:00:05] ...trap/engines/engine.py.224: train: epoch 119 | batch 1130/2304 [I 2022-10-28 10:00:05] ...trap/engines/engine.py.228: elapsed: 0:10:36 | left: 0:10:58 [I 2022-10-28 10:00:05] ...trap/engines/engine.py.229: process: 0.55749 | load: 0.00244 [I 2022-10-28 10:00:05] ...trap/engines/engine.py.230: loss: 0.14932 [I 2022-10-28 10:00:05] ...optimizers/trijoint.py.151: total_norm: 23.962055 [I 2022-10-28 10:00:11] ...trap/engines/engine.py.224: train: epoch 119 | batch 1140/2304 [I 2022-10-28 10:00:11] ...trap/engines/engine.py.228: elapsed: 0:10:41 | left: 0:10:48 [I 2022-10-28 10:00:11] ...trap/engines/engine.py.229: process: 0.56154 | load: 0.00244 [I 2022-10-28 10:00:11] ...trap/engines/engine.py.230: loss: 0.09039 [I 2022-10-28 10:00:11] ...optimizers/trijoint.py.151: total_norm: 14.784872 [I 2022-10-28 10:00:16] ...trap/engines/engine.py.224: train: epoch 119 | batch 1150/2304 [I 2022-10-28 10:00:16] ...trap/engines/engine.py.228: elapsed: 0:10:47 | left: 0:10:40 [I 2022-10-28 10:00:16] ...trap/engines/engine.py.229: process: 0.55651 | load: 0.00244 [I 2022-10-28 10:00:16] ...trap/engines/engine.py.230: loss: 0.14017 [I 2022-10-28 10:00:16] ...optimizers/trijoint.py.151: total_norm: 10.329590 [I 2022-10-28 10:00:22] ...trap/engines/engine.py.224: train: epoch 119 | batch 1160/2304 [I 2022-10-28 10:00:22] ...trap/engines/engine.py.228: elapsed: 0:10:52 | left: 0:10:37 [I 2022-10-28 10:00:22] ...trap/engines/engine.py.229: process: 0.56250 | load: 0.00244 [I 2022-10-28 10:00:22] ...trap/engines/engine.py.230: loss: 0.14732 [I 2022-10-28 10:00:22] ...optimizers/trijoint.py.151: total_norm: 33.136284 [I 2022-10-28 10:00:27] ...trap/engines/engine.py.224: train: epoch 119 | batch 1170/2304 [I 2022-10-28 10:00:27] ...trap/engines/engine.py.228: elapsed: 0:10:58 | left: 0:10:32 [I 2022-10-28 10:00:27] ...trap/engines/engine.py.229: process: 0.55910 | load: 0.00245 [I 2022-10-28 10:00:27] ...trap/engines/engine.py.230: loss: 0.11688 [I 2022-10-28 10:00:27] ...optimizers/trijoint.py.151: total_norm: 16.208666 [I 2022-10-28 10:00:33] ...trap/engines/engine.py.224: train: epoch 119 | batch 1180/2304 [I 2022-10-28 10:00:33] ...trap/engines/engine.py.228: elapsed: 0:11:04 | left: 0:10:26 [I 2022-10-28 10:00:33] ...trap/engines/engine.py.229: process: 0.55907 | load: 0.00248 [I 2022-10-28 10:00:33] ...trap/engines/engine.py.230: loss: 0.12808 [I 2022-10-28 10:00:33] ...optimizers/trijoint.py.151: total_norm: 10.929070 [I 2022-10-28 10:00:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 1190/2304 [I 2022-10-28 10:00:39] ...trap/engines/engine.py.228: elapsed: 0:11:09 | left: 0:10:30 [I 2022-10-28 10:00:39] ...trap/engines/engine.py.229: process: 0.55763 | load: 0.00245 [I 2022-10-28 10:00:39] ...trap/engines/engine.py.230: loss: 0.13441 [I 2022-10-28 10:00:39] ...optimizers/trijoint.py.151: total_norm: 15.344529 [I 2022-10-28 10:00:44] ...trap/engines/engine.py.224: train: epoch 119 | batch 1200/2304 [I 2022-10-28 10:00:44] ...trap/engines/engine.py.228: elapsed: 0:11:15 | left: 0:10:18 [I 2022-10-28 10:00:44] ...trap/engines/engine.py.229: process: 0.56613 | load: 0.00246 [I 2022-10-28 10:00:44] ...trap/engines/engine.py.230: loss: 0.11365 [I 2022-10-28 10:00:44] ...optimizers/trijoint.py.151: total_norm: 26.368547 [I 2022-10-28 10:00:50] ...trap/engines/engine.py.224: train: epoch 119 | batch 1210/2304 [I 2022-10-28 10:00:50] ...trap/engines/engine.py.228: elapsed: 0:11:20 | left: 0:10:07 [I 2022-10-28 10:00:50] ...trap/engines/engine.py.229: process: 0.55677 | load: 0.00244 [I 2022-10-28 10:00:50] ...trap/engines/engine.py.230: loss: 0.09092 [I 2022-10-28 10:00:50] ...optimizers/trijoint.py.151: total_norm: 15.624992 [I 2022-10-28 10:00:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 1220/2304 [I 2022-10-28 10:00:56] ...trap/engines/engine.py.228: elapsed: 0:11:26 | left: 0:10:04 [I 2022-10-28 10:00:56] ...trap/engines/engine.py.229: process: 0.56051 | load: 0.00243 [I 2022-10-28 10:00:56] ...trap/engines/engine.py.230: loss: 0.13672 [I 2022-10-28 10:00:56] ...optimizers/trijoint.py.151: total_norm: 9.106938 [I 2022-10-28 10:01:01] ...trap/engines/engine.py.224: train: epoch 119 | batch 1230/2304 [I 2022-10-28 10:01:01] ...trap/engines/engine.py.228: elapsed: 0:11:32 | left: 0:09:55 [I 2022-10-28 10:01:01] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00243 [I 2022-10-28 10:01:01] ...trap/engines/engine.py.230: loss: 0.11906 [I 2022-10-28 10:01:01] ...optimizers/trijoint.py.151: total_norm: 14.358482 [I 2022-10-28 10:01:07] ...trap/engines/engine.py.224: train: epoch 119 | batch 1240/2304 [I 2022-10-28 10:01:07] ...trap/engines/engine.py.228: elapsed: 0:11:37 | left: 0:09:53 [I 2022-10-28 10:01:07] ...trap/engines/engine.py.229: process: 0.55955 | load: 0.00241 [I 2022-10-28 10:01:07] ...trap/engines/engine.py.230: loss: 0.09995 [I 2022-10-28 10:01:07] ...optimizers/trijoint.py.151: total_norm: 10.734629 [I 2022-10-28 10:01:12] ...trap/engines/engine.py.224: train: epoch 119 | batch 1250/2304 [I 2022-10-28 10:01:12] ...trap/engines/engine.py.228: elapsed: 0:11:43 | left: 0:09:49 [I 2022-10-28 10:01:12] ...trap/engines/engine.py.229: process: 0.56514 | load: 0.00244 [I 2022-10-28 10:01:12] ...trap/engines/engine.py.230: loss: 0.09475 [I 2022-10-28 10:01:12] ...optimizers/trijoint.py.151: total_norm: 16.355669 [I 2022-10-28 10:01:18] ...trap/engines/engine.py.224: train: epoch 119 | batch 1260/2304 [I 2022-10-28 10:01:18] ...trap/engines/engine.py.228: elapsed: 0:11:48 | left: 0:09:40 [I 2022-10-28 10:01:18] ...trap/engines/engine.py.229: process: 0.55348 | load: 0.00242 [I 2022-10-28 10:01:18] ...trap/engines/engine.py.230: loss: 0.12782 [I 2022-10-28 10:01:18] ...optimizers/trijoint.py.151: total_norm: 12.271996 [I 2022-10-28 10:01:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 1270/2304 [I 2022-10-28 10:01:23] ...trap/engines/engine.py.228: elapsed: 0:11:54 | left: 0:09:37 [I 2022-10-28 10:01:23] ...trap/engines/engine.py.229: process: 0.56883 | load: 0.00241 [I 2022-10-28 10:01:23] ...trap/engines/engine.py.230: loss: 0.14897 [I 2022-10-28 10:01:23] ...optimizers/trijoint.py.151: total_norm: 22.868614 [I 2022-10-28 10:01:29] ...trap/engines/engine.py.224: train: epoch 119 | batch 1280/2304 [I 2022-10-28 10:01:29] ...trap/engines/engine.py.228: elapsed: 0:11:59 | left: 0:09:28 [I 2022-10-28 10:01:29] ...trap/engines/engine.py.229: process: 0.54197 | load: 0.00243 [I 2022-10-28 10:01:29] ...trap/engines/engine.py.230: loss: 0.10672 [I 2022-10-28 10:01:29] ...optimizers/trijoint.py.151: total_norm: 7.277290 [I 2022-10-28 10:01:35] ...trap/engines/engine.py.224: train: epoch 119 | batch 1290/2304 [I 2022-10-28 10:01:35] ...trap/engines/engine.py.228: elapsed: 0:12:05 | left: 0:09:24 [I 2022-10-28 10:01:35] ...trap/engines/engine.py.229: process: 0.55492 | load: 0.00244 [I 2022-10-28 10:01:35] ...trap/engines/engine.py.230: loss: 0.13721 [I 2022-10-28 10:01:35] ...optimizers/trijoint.py.151: total_norm: 31.155680 [I 2022-10-28 10:01:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 1300/2304 [I 2022-10-28 10:01:40] ...trap/engines/engine.py.228: elapsed: 0:12:11 | left: 0:09:16 [I 2022-10-28 10:01:40] ...trap/engines/engine.py.229: process: 0.55562 | load: 0.00251 [I 2022-10-28 10:01:40] ...trap/engines/engine.py.230: loss: 0.11863 [I 2022-10-28 10:01:40] ...optimizers/trijoint.py.151: total_norm: 17.592417 [I 2022-10-28 10:01:46] ...trap/engines/engine.py.224: train: epoch 119 | batch 1310/2304 [I 2022-10-28 10:01:46] ...trap/engines/engine.py.228: elapsed: 0:12:16 | left: 0:09:14 [I 2022-10-28 10:01:46] ...trap/engines/engine.py.229: process: 0.56011 | load: 0.00245 [I 2022-10-28 10:01:46] ...trap/engines/engine.py.230: loss: 0.13391 [I 2022-10-28 10:01:46] ...optimizers/trijoint.py.151: total_norm: 28.268040 [I 2022-10-28 10:01:51] ...trap/engines/engine.py.224: train: epoch 119 | batch 1320/2304 [I 2022-10-28 10:01:51] ...trap/engines/engine.py.228: elapsed: 0:12:22 | left: 0:09:17 [I 2022-10-28 10:01:51] ...trap/engines/engine.py.229: process: 0.56383 | load: 0.00246 [I 2022-10-28 10:01:51] ...trap/engines/engine.py.230: loss: 0.10251 [I 2022-10-28 10:01:51] ...optimizers/trijoint.py.151: total_norm: 8.542185 [I 2022-10-28 10:01:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 1330/2304 [I 2022-10-28 10:01:57] ...trap/engines/engine.py.228: elapsed: 0:12:27 | left: 0:09:04 [I 2022-10-28 10:01:57] ...trap/engines/engine.py.229: process: 0.56419 | load: 0.00246 [I 2022-10-28 10:01:57] ...trap/engines/engine.py.230: loss: 0.11944 [I 2022-10-28 10:01:57] ...optimizers/trijoint.py.151: total_norm: 9.102085 [I 2022-10-28 10:02:02] ...trap/engines/engine.py.224: train: epoch 119 | batch 1340/2304 [I 2022-10-28 10:02:02] ...trap/engines/engine.py.228: elapsed: 0:12:33 | left: 0:08:54 [I 2022-10-28 10:02:02] ...trap/engines/engine.py.229: process: 0.56453 | load: 0.00244 [I 2022-10-28 10:02:02] ...trap/engines/engine.py.230: loss: 0.09238 [I 2022-10-28 10:02:02] ...optimizers/trijoint.py.151: total_norm: 10.934809 [I 2022-10-28 10:02:08] ...trap/engines/engine.py.224: train: epoch 119 | batch 1350/2304 [I 2022-10-28 10:02:08] ...trap/engines/engine.py.228: elapsed: 0:12:39 | left: 0:08:53 [I 2022-10-28 10:02:08] ...trap/engines/engine.py.229: process: 0.56286 | load: 0.00245 [I 2022-10-28 10:02:08] ...trap/engines/engine.py.230: loss: 0.10783 [I 2022-10-28 10:02:08] ...optimizers/trijoint.py.151: total_norm: 10.976151 [I 2022-10-28 10:02:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 1360/2304 [I 2022-10-28 10:02:14] ...trap/engines/engine.py.228: elapsed: 0:12:44 | left: 0:08:47 [I 2022-10-28 10:02:14] ...trap/engines/engine.py.229: process: 0.55689 | load: 0.00258 [I 2022-10-28 10:02:14] ...trap/engines/engine.py.230: loss: 0.12999 [I 2022-10-28 10:02:14] ...optimizers/trijoint.py.151: total_norm: 23.963142 [I 2022-10-28 10:02:19] ...trap/engines/engine.py.224: train: epoch 119 | batch 1370/2304 [I 2022-10-28 10:02:19] ...trap/engines/engine.py.228: elapsed: 0:12:50 | left: 0:08:44 [I 2022-10-28 10:02:19] ...trap/engines/engine.py.229: process: 0.57013 | load: 0.00247 [I 2022-10-28 10:02:19] ...trap/engines/engine.py.230: loss: 0.11379 [I 2022-10-28 10:02:19] ...optimizers/trijoint.py.151: total_norm: 8.263819 [I 2022-10-28 10:02:25] ...trap/engines/engine.py.224: train: epoch 119 | batch 1380/2304 [I 2022-10-28 10:02:25] ...trap/engines/engine.py.228: elapsed: 0:12:55 | left: 0:08:32 [I 2022-10-28 10:02:25] ...trap/engines/engine.py.229: process: 0.54541 | load: 0.00249 [I 2022-10-28 10:02:25] ...trap/engines/engine.py.230: loss: 0.13782 [I 2022-10-28 10:02:25] ...optimizers/trijoint.py.151: total_norm: 15.340640 [I 2022-10-28 10:02:30] ...trap/engines/engine.py.224: train: epoch 119 | batch 1390/2304 [I 2022-10-28 10:02:30] ...trap/engines/engine.py.228: elapsed: 0:13:01 | left: 0:08:31 [I 2022-10-28 10:02:30] ...trap/engines/engine.py.229: process: 0.57268 | load: 0.00246 [I 2022-10-28 10:02:30] ...trap/engines/engine.py.230: loss: 0.08145 [I 2022-10-28 10:02:30] ...optimizers/trijoint.py.151: total_norm: 10.147719 [I 2022-10-28 10:02:36] ...trap/engines/engine.py.224: train: epoch 119 | batch 1400/2304 [I 2022-10-28 10:02:36] ...trap/engines/engine.py.228: elapsed: 0:13:07 | left: 0:08:28 [I 2022-10-28 10:02:36] ...trap/engines/engine.py.229: process: 0.56112 | load: 0.00242 [I 2022-10-28 10:02:36] ...trap/engines/engine.py.230: loss: 0.10615 [I 2022-10-28 10:02:36] ...optimizers/trijoint.py.151: total_norm: 18.622635 [I 2022-10-28 10:02:42] ...trap/engines/engine.py.224: train: epoch 119 | batch 1410/2304 [I 2022-10-28 10:02:42] ...trap/engines/engine.py.228: elapsed: 0:13:12 | left: 0:08:16 [I 2022-10-28 10:02:42] ...trap/engines/engine.py.229: process: 0.55581 | load: 0.00244 [I 2022-10-28 10:02:42] ...trap/engines/engine.py.230: loss: 0.15298 [I 2022-10-28 10:02:42] ...optimizers/trijoint.py.151: total_norm: 18.202593 [I 2022-10-28 10:02:47] ...trap/engines/engine.py.224: train: epoch 119 | batch 1420/2304 [I 2022-10-28 10:02:47] ...trap/engines/engine.py.228: elapsed: 0:13:18 | left: 0:08:08 [I 2022-10-28 10:02:47] ...trap/engines/engine.py.229: process: 0.54447 | load: 0.00243 [I 2022-10-28 10:02:47] ...trap/engines/engine.py.230: loss: 0.08370 [I 2022-10-28 10:02:47] ...optimizers/trijoint.py.151: total_norm: 33.737644 [I 2022-10-28 10:02:53] ...trap/engines/engine.py.224: train: epoch 119 | batch 1430/2304 [I 2022-10-28 10:02:53] ...trap/engines/engine.py.228: elapsed: 0:13:23 | left: 0:08:08 [I 2022-10-28 10:02:53] ...trap/engines/engine.py.229: process: 0.56135 | load: 0.00241 [I 2022-10-28 10:02:53] ...trap/engines/engine.py.230: loss: 0.08807 [I 2022-10-28 10:02:53] ...optimizers/trijoint.py.151: total_norm: 9.892989 [I 2022-10-28 10:02:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 1440/2304 [I 2022-10-28 10:02:58] ...trap/engines/engine.py.228: elapsed: 0:13:29 | left: 0:08:01 [I 2022-10-28 10:02:58] ...trap/engines/engine.py.229: process: 0.56247 | load: 0.00241 [I 2022-10-28 10:02:58] ...trap/engines/engine.py.230: loss: 0.16255 [I 2022-10-28 10:02:58] ...optimizers/trijoint.py.151: total_norm: 18.995859 [I 2022-10-28 10:03:04] ...trap/engines/engine.py.224: train: epoch 119 | batch 1450/2304 [I 2022-10-28 10:03:04] ...trap/engines/engine.py.228: elapsed: 0:13:34 | left: 0:07:55 [I 2022-10-28 10:03:04] ...trap/engines/engine.py.229: process: 0.55898 | load: 0.00243 [I 2022-10-28 10:03:04] ...trap/engines/engine.py.230: loss: 0.12529 [I 2022-10-28 10:03:04] ...optimizers/trijoint.py.151: total_norm: 25.911711 [I 2022-10-28 10:03:10] ...trap/engines/engine.py.224: train: epoch 119 | batch 1460/2304 [I 2022-10-28 10:03:10] ...trap/engines/engine.py.228: elapsed: 0:13:40 | left: 0:07:49 [I 2022-10-28 10:03:10] ...trap/engines/engine.py.229: process: 0.55237 | load: 0.00252 [I 2022-10-28 10:03:10] ...trap/engines/engine.py.230: loss: 0.08869 [I 2022-10-28 10:03:10] ...optimizers/trijoint.py.151: total_norm: 15.608727 [I 2022-10-28 10:03:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 1470/2304 [I 2022-10-28 10:03:15] ...trap/engines/engine.py.228: elapsed: 0:13:46 | left: 0:07:45 [I 2022-10-28 10:03:15] ...trap/engines/engine.py.229: process: 0.56607 | load: 0.00252 [I 2022-10-28 10:03:15] ...trap/engines/engine.py.230: loss: 0.11230 [I 2022-10-28 10:03:15] ...optimizers/trijoint.py.151: total_norm: 17.471798 [I 2022-10-28 10:03:21] ...trap/engines/engine.py.224: train: epoch 119 | batch 1480/2304 [I 2022-10-28 10:03:21] ...trap/engines/engine.py.228: elapsed: 0:13:51 | left: 0:07:41 [I 2022-10-28 10:03:21] ...trap/engines/engine.py.229: process: 0.55803 | load: 0.00246 [I 2022-10-28 10:03:21] ...trap/engines/engine.py.230: loss: 0.09548 [I 2022-10-28 10:03:21] ...optimizers/trijoint.py.151: total_norm: 18.121664 [I 2022-10-28 10:03:26] ...trap/engines/engine.py.224: train: epoch 119 | batch 1490/2304 [I 2022-10-28 10:03:26] ...trap/engines/engine.py.228: elapsed: 0:13:57 | left: 0:07:36 [I 2022-10-28 10:03:26] ...trap/engines/engine.py.229: process: 0.56494 | load: 0.00245 [I 2022-10-28 10:03:26] ...trap/engines/engine.py.230: loss: 0.13037 [I 2022-10-28 10:03:26] ...optimizers/trijoint.py.151: total_norm: 13.189363 [I 2022-10-28 10:03:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 1500/2304 [I 2022-10-28 10:03:32] ...trap/engines/engine.py.228: elapsed: 0:14:02 | left: 0:07:26 [I 2022-10-28 10:03:32] ...trap/engines/engine.py.229: process: 0.55379 | load: 0.00244 [I 2022-10-28 10:03:32] ...trap/engines/engine.py.230: loss: 0.11723 [I 2022-10-28 10:03:32] ...optimizers/trijoint.py.151: total_norm: 17.842522 [I 2022-10-28 10:03:37] ...trap/engines/engine.py.224: train: epoch 119 | batch 1510/2304 [I 2022-10-28 10:03:37] ...trap/engines/engine.py.228: elapsed: 0:14:08 | left: 0:07:19 [I 2022-10-28 10:03:37] ...trap/engines/engine.py.229: process: 0.54942 | load: 0.00240 [I 2022-10-28 10:03:37] ...trap/engines/engine.py.230: loss: 0.12891 [I 2022-10-28 10:03:37] ...optimizers/trijoint.py.151: total_norm: 17.670925 [I 2022-10-28 10:03:43] ...trap/engines/engine.py.224: train: epoch 119 | batch 1520/2304 [I 2022-10-28 10:03:43] ...trap/engines/engine.py.228: elapsed: 0:14:13 | left: 0:07:13 [I 2022-10-28 10:03:43] ...trap/engines/engine.py.229: process: 0.54795 | load: 0.00249 [I 2022-10-28 10:03:43] ...trap/engines/engine.py.230: loss: 0.09954 [I 2022-10-28 10:03:43] ...optimizers/trijoint.py.151: total_norm: 11.612552 [I 2022-10-28 10:03:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 1530/2304 [I 2022-10-28 10:03:49] ...trap/engines/engine.py.228: elapsed: 0:14:19 | left: 0:07:09 [I 2022-10-28 10:03:49] ...trap/engines/engine.py.229: process: 0.55043 | load: 0.00247 [I 2022-10-28 10:03:49] ...trap/engines/engine.py.230: loss: 0.11939 [I 2022-10-28 10:03:49] ...optimizers/trijoint.py.151: total_norm: 20.889933 [I 2022-10-28 10:03:54] ...trap/engines/engine.py.224: train: epoch 119 | batch 1540/2304 [I 2022-10-28 10:03:54] ...trap/engines/engine.py.228: elapsed: 0:14:25 | left: 0:07:05 [I 2022-10-28 10:03:54] ...trap/engines/engine.py.229: process: 0.56324 | load: 0.00243 [I 2022-10-28 10:03:54] ...trap/engines/engine.py.230: loss: 0.12362 [I 2022-10-28 10:03:54] ...optimizers/trijoint.py.151: total_norm: 9.432452 [I 2022-10-28 10:04:00] ...trap/engines/engine.py.224: train: epoch 119 | batch 1550/2304 [I 2022-10-28 10:04:00] ...trap/engines/engine.py.228: elapsed: 0:14:30 | left: 0:07:01 [I 2022-10-28 10:04:00] ...trap/engines/engine.py.229: process: 0.56177 | load: 0.00245 [I 2022-10-28 10:04:00] ...trap/engines/engine.py.230: loss: 0.12378 [I 2022-10-28 10:04:00] ...optimizers/trijoint.py.151: total_norm: 13.593053 [I 2022-10-28 10:04:05] ...trap/engines/engine.py.224: train: epoch 119 | batch 1560/2304 [I 2022-10-28 10:04:05] ...trap/engines/engine.py.228: elapsed: 0:14:36 | left: 0:06:53 [I 2022-10-28 10:04:05] ...trap/engines/engine.py.229: process: 0.56496 | load: 0.00241 [I 2022-10-28 10:04:05] ...trap/engines/engine.py.230: loss: 0.08765 [I 2022-10-28 10:04:05] ...optimizers/trijoint.py.151: total_norm: 11.556832 [I 2022-10-28 10:04:11] ...trap/engines/engine.py.224: train: epoch 119 | batch 1570/2304 [I 2022-10-28 10:04:11] ...trap/engines/engine.py.228: elapsed: 0:14:41 | left: 0:06:49 [I 2022-10-28 10:04:11] ...trap/engines/engine.py.229: process: 0.56333 | load: 0.00241 [I 2022-10-28 10:04:11] ...trap/engines/engine.py.230: loss: 0.09915 [I 2022-10-28 10:04:11] ...optimizers/trijoint.py.151: total_norm: 9.828810 [I 2022-10-28 10:04:16] ...trap/engines/engine.py.224: train: epoch 119 | batch 1580/2304 [I 2022-10-28 10:04:16] ...trap/engines/engine.py.228: elapsed: 0:14:47 | left: 0:06:43 [I 2022-10-28 10:04:16] ...trap/engines/engine.py.229: process: 0.55510 | load: 0.00243 [I 2022-10-28 10:04:16] ...trap/engines/engine.py.230: loss: 0.13520 [I 2022-10-28 10:04:16] ...optimizers/trijoint.py.151: total_norm: 19.212120 [I 2022-10-28 10:04:22] ...trap/engines/engine.py.224: train: epoch 119 | batch 1590/2304 [I 2022-10-28 10:04:22] ...trap/engines/engine.py.228: elapsed: 0:14:53 | left: 0:06:36 [I 2022-10-28 10:04:22] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00243 [I 2022-10-28 10:04:22] ...trap/engines/engine.py.230: loss: 0.09026 [I 2022-10-28 10:04:22] ...optimizers/trijoint.py.151: total_norm: 10.192708 [I 2022-10-28 10:04:28] ...trap/engines/engine.py.224: train: epoch 119 | batch 1600/2304 [I 2022-10-28 10:04:28] ...trap/engines/engine.py.228: elapsed: 0:14:58 | left: 0:06:29 [I 2022-10-28 10:04:28] ...trap/engines/engine.py.229: process: 0.55060 | load: 0.00240 [I 2022-10-28 10:04:28] ...trap/engines/engine.py.230: loss: 0.13655 [I 2022-10-28 10:04:28] ...optimizers/trijoint.py.151: total_norm: 8.664652 [I 2022-10-28 10:04:33] ...trap/engines/engine.py.224: train: epoch 119 | batch 1610/2304 [I 2022-10-28 10:04:33] ...trap/engines/engine.py.228: elapsed: 0:15:04 | left: 0:06:28 [I 2022-10-28 10:04:33] ...trap/engines/engine.py.229: process: 0.55700 | load: 0.00243 [I 2022-10-28 10:04:33] ...trap/engines/engine.py.230: loss: 0.10549 [I 2022-10-28 10:04:33] ...optimizers/trijoint.py.151: total_norm: 18.114603 [I 2022-10-28 10:04:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 1620/2304 [I 2022-10-28 10:04:39] ...trap/engines/engine.py.228: elapsed: 0:15:09 | left: 0:06:23 [I 2022-10-28 10:04:39] ...trap/engines/engine.py.229: process: 0.55508 | load: 0.00243 [I 2022-10-28 10:04:39] ...trap/engines/engine.py.230: loss: 0.12740 [I 2022-10-28 10:04:39] ...optimizers/trijoint.py.151: total_norm: 18.361641 [I 2022-10-28 10:04:44] ...trap/engines/engine.py.224: train: epoch 119 | batch 1630/2304 [I 2022-10-28 10:04:44] ...trap/engines/engine.py.228: elapsed: 0:15:15 | left: 0:06:15 [I 2022-10-28 10:04:44] ...trap/engines/engine.py.229: process: 0.55941 | load: 0.00240 [I 2022-10-28 10:04:44] ...trap/engines/engine.py.230: loss: 0.09394 [I 2022-10-28 10:04:44] ...optimizers/trijoint.py.151: total_norm: 12.750092 [I 2022-10-28 10:04:50] ...trap/engines/engine.py.224: train: epoch 119 | batch 1640/2304 [I 2022-10-28 10:04:50] ...trap/engines/engine.py.228: elapsed: 0:15:20 | left: 0:06:08 [I 2022-10-28 10:04:50] ...trap/engines/engine.py.229: process: 0.54832 | load: 0.00249 [I 2022-10-28 10:04:50] ...trap/engines/engine.py.230: loss: 0.10163 [I 2022-10-28 10:04:50] ...optimizers/trijoint.py.151: total_norm: 15.412110 [I 2022-10-28 10:04:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 1650/2304 [I 2022-10-28 10:04:56] ...trap/engines/engine.py.228: elapsed: 0:15:26 | left: 0:06:04 [I 2022-10-28 10:04:56] ...trap/engines/engine.py.229: process: 0.57038 | load: 0.00247 [I 2022-10-28 10:04:56] ...trap/engines/engine.py.230: loss: 0.13601 [I 2022-10-28 10:04:56] ...optimizers/trijoint.py.151: total_norm: 13.151341 [I 2022-10-28 10:05:01] ...trap/engines/engine.py.224: train: epoch 119 | batch 1660/2304 [I 2022-10-28 10:05:01] ...trap/engines/engine.py.228: elapsed: 0:15:32 | left: 0:05:55 [I 2022-10-28 10:05:01] ...trap/engines/engine.py.229: process: 0.53532 | load: 0.00245 [I 2022-10-28 10:05:01] ...trap/engines/engine.py.230: loss: 0.10367 [I 2022-10-28 10:05:01] ...optimizers/trijoint.py.151: total_norm: 17.984015 [I 2022-10-28 10:05:07] ...trap/engines/engine.py.224: train: epoch 119 | batch 1670/2304 [I 2022-10-28 10:05:07] ...trap/engines/engine.py.228: elapsed: 0:15:37 | left: 0:05:52 [I 2022-10-28 10:05:07] ...trap/engines/engine.py.229: process: 0.56019 | load: 0.00242 [I 2022-10-28 10:05:07] ...trap/engines/engine.py.230: loss: 0.13221 [I 2022-10-28 10:05:07] ...optimizers/trijoint.py.151: total_norm: 17.513824 [I 2022-10-28 10:05:12] ...trap/engines/engine.py.224: train: epoch 119 | batch 1680/2304 [I 2022-10-28 10:05:12] ...trap/engines/engine.py.228: elapsed: 0:15:43 | left: 0:05:47 [I 2022-10-28 10:05:12] ...trap/engines/engine.py.229: process: 0.55323 | load: 0.00242 [I 2022-10-28 10:05:12] ...trap/engines/engine.py.230: loss: 0.09974 [I 2022-10-28 10:05:12] ...optimizers/trijoint.py.151: total_norm: 12.420156 [I 2022-10-28 10:05:18] ...trap/engines/engine.py.224: train: epoch 119 | batch 1690/2304 [I 2022-10-28 10:05:18] ...trap/engines/engine.py.228: elapsed: 0:15:48 | left: 0:05:44 [I 2022-10-28 10:05:18] ...trap/engines/engine.py.229: process: 0.57396 | load: 0.00248 [I 2022-10-28 10:05:18] ...trap/engines/engine.py.230: loss: 0.12565 [I 2022-10-28 10:05:18] ...optimizers/trijoint.py.151: total_norm: 10.521281 [I 2022-10-28 10:05:23] ...trap/engines/engine.py.224: train: epoch 119 | batch 1700/2304 [I 2022-10-28 10:05:23] ...trap/engines/engine.py.228: elapsed: 0:15:54 | left: 0:05:38 [I 2022-10-28 10:05:23] ...trap/engines/engine.py.229: process: 0.57505 | load: 0.00248 [I 2022-10-28 10:05:23] ...trap/engines/engine.py.230: loss: 0.12293 [I 2022-10-28 10:05:23] ...optimizers/trijoint.py.151: total_norm: 12.585013 [I 2022-10-28 10:05:29] ...trap/engines/engine.py.224: train: epoch 119 | batch 1710/2304 [I 2022-10-28 10:05:29] ...trap/engines/engine.py.228: elapsed: 0:16:00 | left: 0:05:33 [I 2022-10-28 10:05:29] ...trap/engines/engine.py.229: process: 0.56452 | load: 0.00249 [I 2022-10-28 10:05:29] ...trap/engines/engine.py.230: loss: 0.13955 [I 2022-10-28 10:05:29] ...optimizers/trijoint.py.151: total_norm: 16.613039 [I 2022-10-28 10:05:35] ...trap/engines/engine.py.224: train: epoch 119 | batch 1720/2304 [I 2022-10-28 10:05:35] ...trap/engines/engine.py.228: elapsed: 0:16:05 | left: 0:05:24 [I 2022-10-28 10:05:35] ...trap/engines/engine.py.229: process: 0.55236 | load: 0.00246 [I 2022-10-28 10:05:35] ...trap/engines/engine.py.230: loss: 0.11297 [I 2022-10-28 10:05:35] ...optimizers/trijoint.py.151: total_norm: 15.331463 [I 2022-10-28 10:05:40] ...trap/engines/engine.py.224: train: epoch 119 | batch 1730/2304 [I 2022-10-28 10:05:40] ...trap/engines/engine.py.228: elapsed: 0:16:11 | left: 0:05:17 [I 2022-10-28 10:05:40] ...trap/engines/engine.py.229: process: 0.53924 | load: 0.00246 [I 2022-10-28 10:05:40] ...trap/engines/engine.py.230: loss: 0.08970 [I 2022-10-28 10:05:40] ...optimizers/trijoint.py.151: total_norm: 21.628197 [I 2022-10-28 10:05:46] ...trap/engines/engine.py.224: train: epoch 119 | batch 1740/2304 [I 2022-10-28 10:05:46] ...trap/engines/engine.py.228: elapsed: 0:16:16 | left: 0:05:14 [I 2022-10-28 10:05:46] ...trap/engines/engine.py.229: process: 0.55658 | load: 0.00244 [I 2022-10-28 10:05:46] ...trap/engines/engine.py.230: loss: 0.13708 [I 2022-10-28 10:05:46] ...optimizers/trijoint.py.151: total_norm: 53.352421 [I 2022-10-28 10:05:51] ...trap/engines/engine.py.224: train: epoch 119 | batch 1750/2304 [I 2022-10-28 10:05:51] ...trap/engines/engine.py.228: elapsed: 0:16:22 | left: 0:05:07 [I 2022-10-28 10:05:51] ...trap/engines/engine.py.229: process: 0.55991 | load: 0.00245 [I 2022-10-28 10:05:51] ...trap/engines/engine.py.230: loss: 0.11150 [I 2022-10-28 10:05:51] ...optimizers/trijoint.py.151: total_norm: 18.257437 [I 2022-10-28 10:05:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 1760/2304 [I 2022-10-28 10:05:57] ...trap/engines/engine.py.228: elapsed: 0:16:27 | left: 0:05:02 [I 2022-10-28 10:05:57] ...trap/engines/engine.py.229: process: 0.56599 | load: 0.00243 [I 2022-10-28 10:05:57] ...trap/engines/engine.py.230: loss: 0.13386 [I 2022-10-28 10:05:57] ...optimizers/trijoint.py.151: total_norm: 16.144331 [I 2022-10-28 10:06:02] ...trap/engines/engine.py.224: train: epoch 119 | batch 1770/2304 [I 2022-10-28 10:06:02] ...trap/engines/engine.py.228: elapsed: 0:16:33 | left: 0:04:56 [I 2022-10-28 10:06:02] ...trap/engines/engine.py.229: process: 0.55327 | load: 0.00245 [I 2022-10-28 10:06:02] ...trap/engines/engine.py.230: loss: 0.14196 [I 2022-10-28 10:06:02] ...optimizers/trijoint.py.151: total_norm: 30.960190 [I 2022-10-28 10:06:08] ...trap/engines/engine.py.224: train: epoch 119 | batch 1780/2304 [I 2022-10-28 10:06:08] ...trap/engines/engine.py.228: elapsed: 0:16:39 | left: 0:04:51 [I 2022-10-28 10:06:08] ...trap/engines/engine.py.229: process: 0.56139 | load: 0.00241 [I 2022-10-28 10:06:08] ...trap/engines/engine.py.230: loss: 0.10510 [I 2022-10-28 10:06:08] ...optimizers/trijoint.py.151: total_norm: 9.382011 [I 2022-10-28 10:06:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 1790/2304 [I 2022-10-28 10:06:14] ...trap/engines/engine.py.228: elapsed: 0:16:44 | left: 0:04:47 [I 2022-10-28 10:06:14] ...trap/engines/engine.py.229: process: 0.55767 | load: 0.00252 [I 2022-10-28 10:06:14] ...trap/engines/engine.py.230: loss: 0.12817 [I 2022-10-28 10:06:14] ...optimizers/trijoint.py.151: total_norm: 37.823345 [I 2022-10-28 10:06:19] ...trap/engines/engine.py.224: train: epoch 119 | batch 1800/2304 [I 2022-10-28 10:06:19] ...trap/engines/engine.py.228: elapsed: 0:16:50 | left: 0:04:40 [I 2022-10-28 10:06:19] ...trap/engines/engine.py.229: process: 0.55303 | load: 0.00248 [I 2022-10-28 10:06:19] ...trap/engines/engine.py.230: loss: 0.13094 [I 2022-10-28 10:06:19] ...optimizers/trijoint.py.151: total_norm: 12.158889 [I 2022-10-28 10:06:25] ...trap/engines/engine.py.224: train: epoch 119 | batch 1810/2304 [I 2022-10-28 10:06:25] ...trap/engines/engine.py.228: elapsed: 0:16:55 | left: 0:04:36 [I 2022-10-28 10:06:25] ...trap/engines/engine.py.229: process: 0.56286 | load: 0.00246 [I 2022-10-28 10:06:25] ...trap/engines/engine.py.230: loss: 0.13286 [I 2022-10-28 10:06:25] ...optimizers/trijoint.py.151: total_norm: 24.384956 [I 2022-10-28 10:06:30] ...trap/engines/engine.py.224: train: epoch 119 | batch 1820/2304 [I 2022-10-28 10:06:30] ...trap/engines/engine.py.228: elapsed: 0:17:01 | left: 0:04:29 [I 2022-10-28 10:06:30] ...trap/engines/engine.py.229: process: 0.55856 | load: 0.00245 [I 2022-10-28 10:06:30] ...trap/engines/engine.py.230: loss: 0.10164 [I 2022-10-28 10:06:30] ...optimizers/trijoint.py.151: total_norm: 20.339870 [I 2022-10-28 10:06:36] ...trap/engines/engine.py.224: train: epoch 119 | batch 1830/2304 [I 2022-10-28 10:06:36] ...trap/engines/engine.py.228: elapsed: 0:17:07 | left: 0:04:25 [I 2022-10-28 10:06:36] ...trap/engines/engine.py.229: process: 0.56700 | load: 0.00245 [I 2022-10-28 10:06:36] ...trap/engines/engine.py.230: loss: 0.17123 [I 2022-10-28 10:06:36] ...optimizers/trijoint.py.151: total_norm: 19.358307 [I 2022-10-28 10:06:42] ...trap/engines/engine.py.224: train: epoch 119 | batch 1840/2304 [I 2022-10-28 10:06:42] ...trap/engines/engine.py.228: elapsed: 0:17:12 | left: 0:04:17 [I 2022-10-28 10:06:42] ...trap/engines/engine.py.229: process: 0.55862 | load: 0.00246 [I 2022-10-28 10:06:42] ...trap/engines/engine.py.230: loss: 0.09638 [I 2022-10-28 10:06:42] ...optimizers/trijoint.py.151: total_norm: 15.735676 [I 2022-10-28 10:06:47] ...trap/engines/engine.py.224: train: epoch 119 | batch 1850/2304 [I 2022-10-28 10:06:47] ...trap/engines/engine.py.228: elapsed: 0:17:18 | left: 0:04:12 [I 2022-10-28 10:06:47] ...trap/engines/engine.py.229: process: 0.55994 | load: 0.00250 [I 2022-10-28 10:06:47] ...trap/engines/engine.py.230: loss: 0.09979 [I 2022-10-28 10:06:47] ...optimizers/trijoint.py.151: total_norm: 14.820566 [I 2022-10-28 10:06:53] ...trap/engines/engine.py.224: train: epoch 119 | batch 1860/2304 [I 2022-10-28 10:06:53] ...trap/engines/engine.py.228: elapsed: 0:17:23 | left: 0:04:08 [I 2022-10-28 10:06:53] ...trap/engines/engine.py.229: process: 0.56144 | load: 0.00246 [I 2022-10-28 10:06:53] ...trap/engines/engine.py.230: loss: 0.09717 [I 2022-10-28 10:06:53] ...optimizers/trijoint.py.151: total_norm: 14.131133 [I 2022-10-28 10:06:58] ...trap/engines/engine.py.224: train: epoch 119 | batch 1870/2304 [I 2022-10-28 10:06:58] ...trap/engines/engine.py.228: elapsed: 0:17:29 | left: 0:04:01 [I 2022-10-28 10:06:58] ...trap/engines/engine.py.229: process: 0.55926 | load: 0.00243 [I 2022-10-28 10:06:58] ...trap/engines/engine.py.230: loss: 0.14946 [I 2022-10-28 10:06:58] ...optimizers/trijoint.py.151: total_norm: 33.552765 [I 2022-10-28 10:07:04] ...trap/engines/engine.py.224: train: epoch 119 | batch 1880/2304 [I 2022-10-28 10:07:04] ...trap/engines/engine.py.228: elapsed: 0:17:35 | left: 0:04:01 [I 2022-10-28 10:07:04] ...trap/engines/engine.py.229: process: 0.59753 | load: 0.00256 [I 2022-10-28 10:07:04] ...trap/engines/engine.py.230: loss: 0.16694 [I 2022-10-28 10:07:04] ...optimizers/trijoint.py.151: total_norm: 41.484501 [I 2022-10-28 10:07:10] ...trap/engines/engine.py.224: train: epoch 119 | batch 1890/2304 [I 2022-10-28 10:07:10] ...trap/engines/engine.py.228: elapsed: 0:17:40 | left: 0:03:53 [I 2022-10-28 10:07:10] ...trap/engines/engine.py.229: process: 0.56479 | load: 0.00243 [I 2022-10-28 10:07:10] ...trap/engines/engine.py.230: loss: 0.13612 [I 2022-10-28 10:07:10] ...optimizers/trijoint.py.151: total_norm: 21.284739 [I 2022-10-28 10:07:15] ...trap/engines/engine.py.224: train: epoch 119 | batch 1900/2304 [I 2022-10-28 10:07:15] ...trap/engines/engine.py.228: elapsed: 0:17:46 | left: 0:03:52 [I 2022-10-28 10:07:15] ...trap/engines/engine.py.229: process: 0.64970 | load: 0.00244 [I 2022-10-28 10:07:15] ...trap/engines/engine.py.230: loss: 0.11194 [I 2022-10-28 10:07:15] ...optimizers/trijoint.py.151: total_norm: 17.871151 [I 2022-10-28 10:07:21] ...trap/engines/engine.py.224: train: epoch 119 | batch 1910/2304 [I 2022-10-28 10:07:21] ...trap/engines/engine.py.228: elapsed: 0:17:51 | left: 0:03:39 [I 2022-10-28 10:07:21] ...trap/engines/engine.py.229: process: 0.55074 | load: 0.00246 [I 2022-10-28 10:07:21] ...trap/engines/engine.py.230: loss: 0.10194 [I 2022-10-28 10:07:21] ...optimizers/trijoint.py.151: total_norm: 15.896612 [I 2022-10-28 10:07:26] ...trap/engines/engine.py.224: train: epoch 119 | batch 1920/2304 [I 2022-10-28 10:07:26] ...trap/engines/engine.py.228: elapsed: 0:17:57 | left: 0:03:33 [I 2022-10-28 10:07:26] ...trap/engines/engine.py.229: process: 0.56386 | load: 0.00245 [I 2022-10-28 10:07:26] ...trap/engines/engine.py.230: loss: 0.08734 [I 2022-10-28 10:07:26] ...optimizers/trijoint.py.151: total_norm: 24.600918 [I 2022-10-28 10:07:32] ...trap/engines/engine.py.224: train: epoch 119 | batch 1930/2304 [I 2022-10-28 10:07:32] ...trap/engines/engine.py.228: elapsed: 0:18:03 | left: 0:03:28 [I 2022-10-28 10:07:32] ...trap/engines/engine.py.229: process: 0.55679 | load: 0.00251 [I 2022-10-28 10:07:32] ...trap/engines/engine.py.230: loss: 0.11762 [I 2022-10-28 10:07:32] ...optimizers/trijoint.py.151: total_norm: 38.324593 [I 2022-10-28 10:07:38] ...trap/engines/engine.py.224: train: epoch 119 | batch 1940/2304 [I 2022-10-28 10:07:38] ...trap/engines/engine.py.228: elapsed: 0:18:08 | left: 0:03:21 [I 2022-10-28 10:07:38] ...trap/engines/engine.py.229: process: 0.54731 | load: 0.00248 [I 2022-10-28 10:07:38] ...trap/engines/engine.py.230: loss: 0.08047 [I 2022-10-28 10:07:38] ...optimizers/trijoint.py.151: total_norm: 11.860978 [I 2022-10-28 10:07:43] ...trap/engines/engine.py.224: train: epoch 119 | batch 1950/2304 [I 2022-10-28 10:07:43] ...trap/engines/engine.py.228: elapsed: 0:18:14 | left: 0:03:18 [I 2022-10-28 10:07:43] ...trap/engines/engine.py.229: process: 0.57114 | load: 0.00247 [I 2022-10-28 10:07:43] ...trap/engines/engine.py.230: loss: 0.12830 [I 2022-10-28 10:07:43] ...optimizers/trijoint.py.151: total_norm: 9.287847 [I 2022-10-28 10:07:49] ...trap/engines/engine.py.224: train: epoch 119 | batch 1960/2304 [I 2022-10-28 10:07:49] ...trap/engines/engine.py.228: elapsed: 0:18:19 | left: 0:03:11 [I 2022-10-28 10:07:49] ...trap/engines/engine.py.229: process: 0.56115 | load: 0.00244 [I 2022-10-28 10:07:49] ...trap/engines/engine.py.230: loss: 0.09178 [I 2022-10-28 10:07:49] ...optimizers/trijoint.py.151: total_norm: 9.645499 [I 2022-10-28 10:07:54] ...trap/engines/engine.py.224: train: epoch 119 | batch 1970/2304 [I 2022-10-28 10:07:54] ...trap/engines/engine.py.228: elapsed: 0:18:25 | left: 0:03:08 [I 2022-10-28 10:07:54] ...trap/engines/engine.py.229: process: 0.55844 | load: 0.00254 [I 2022-10-28 10:07:54] ...trap/engines/engine.py.230: loss: 0.13531 [I 2022-10-28 10:07:54] ...optimizers/trijoint.py.151: total_norm: 40.334476 [I 2022-10-28 10:08:00] ...trap/engines/engine.py.224: train: epoch 119 | batch 1980/2304 [I 2022-10-28 10:08:00] ...trap/engines/engine.py.228: elapsed: 0:18:31 | left: 0:03:00 [I 2022-10-28 10:08:00] ...trap/engines/engine.py.229: process: 0.55986 | load: 0.00249 [I 2022-10-28 10:08:00] ...trap/engines/engine.py.230: loss: 0.10727 [I 2022-10-28 10:08:00] ...optimizers/trijoint.py.151: total_norm: 20.630629 [I 2022-10-28 10:08:06] ...trap/engines/engine.py.224: train: epoch 119 | batch 1990/2304 [I 2022-10-28 10:08:06] ...trap/engines/engine.py.228: elapsed: 0:18:36 | left: 0:02:54 [I 2022-10-28 10:08:06] ...trap/engines/engine.py.229: process: 0.55653 | load: 0.00249 [I 2022-10-28 10:08:06] ...trap/engines/engine.py.230: loss: 0.08735 [I 2022-10-28 10:08:06] ...optimizers/trijoint.py.151: total_norm: 10.031692 [I 2022-10-28 10:08:11] ...trap/engines/engine.py.224: train: epoch 119 | batch 2000/2304 [I 2022-10-28 10:08:11] ...trap/engines/engine.py.228: elapsed: 0:18:42 | left: 0:02:49 [I 2022-10-28 10:08:11] ...trap/engines/engine.py.229: process: 0.56447 | load: 0.00239 [I 2022-10-28 10:08:11] ...trap/engines/engine.py.230: loss: 0.12503 [I 2022-10-28 10:08:11] ...optimizers/trijoint.py.151: total_norm: 21.777229 [I 2022-10-28 10:08:17] ...trap/engines/engine.py.224: train: epoch 119 | batch 2010/2304 [I 2022-10-28 10:08:17] ...trap/engines/engine.py.228: elapsed: 0:18:47 | left: 0:02:43 [I 2022-10-28 10:08:17] ...trap/engines/engine.py.229: process: 0.55742 | load: 0.00243 [I 2022-10-28 10:08:17] ...trap/engines/engine.py.230: loss: 0.09968 [I 2022-10-28 10:08:17] ...optimizers/trijoint.py.151: total_norm: 7.665783 [I 2022-10-28 10:08:22] ...trap/engines/engine.py.224: train: epoch 119 | batch 2020/2304 [I 2022-10-28 10:08:22] ...trap/engines/engine.py.228: elapsed: 0:18:53 | left: 0:02:38 [I 2022-10-28 10:08:22] ...trap/engines/engine.py.229: process: 0.54990 | load: 0.00243 [I 2022-10-28 10:08:22] ...trap/engines/engine.py.230: loss: 0.09765 [I 2022-10-28 10:08:22] ...optimizers/trijoint.py.151: total_norm: 15.601234 [I 2022-10-28 10:08:28] ...trap/engines/engine.py.224: train: epoch 119 | batch 2030/2304 [I 2022-10-28 10:08:28] ...trap/engines/engine.py.228: elapsed: 0:18:58 | left: 0:02:32 [I 2022-10-28 10:08:28] ...trap/engines/engine.py.229: process: 0.55663 | load: 0.00243 [I 2022-10-28 10:08:28] ...trap/engines/engine.py.230: loss: 0.08228 [I 2022-10-28 10:08:28] ...optimizers/trijoint.py.151: total_norm: 19.774981 [I 2022-10-28 10:08:34] ...trap/engines/engine.py.224: train: epoch 119 | batch 2040/2304 [I 2022-10-28 10:08:34] ...trap/engines/engine.py.228: elapsed: 0:19:04 | left: 0:02:28 [I 2022-10-28 10:08:34] ...trap/engines/engine.py.229: process: 0.56317 | load: 0.00244 [I 2022-10-28 10:08:34] ...trap/engines/engine.py.230: loss: 0.10864 [I 2022-10-28 10:08:34] ...optimizers/trijoint.py.151: total_norm: 15.191362 [I 2022-10-28 10:08:39] ...trap/engines/engine.py.224: train: epoch 119 | batch 2050/2304 [I 2022-10-28 10:08:39] ...trap/engines/engine.py.228: elapsed: 0:19:10 | left: 0:02:21 [I 2022-10-28 10:08:39] ...trap/engines/engine.py.229: process: 0.54605 | load: 0.00250 [I 2022-10-28 10:08:39] ...trap/engines/engine.py.230: loss: 0.09671 [I 2022-10-28 10:08:39] ...optimizers/trijoint.py.151: total_norm: 10.901351 [I 2022-10-28 10:08:45] ...trap/engines/engine.py.224: train: epoch 119 | batch 2060/2304 [I 2022-10-28 10:08:45] ...trap/engines/engine.py.228: elapsed: 0:19:15 | left: 0:02:16 [I 2022-10-28 10:08:45] ...trap/engines/engine.py.229: process: 0.56288 | load: 0.00246 [I 2022-10-28 10:08:45] ...trap/engines/engine.py.230: loss: 0.16251 [I 2022-10-28 10:08:45] ...optimizers/trijoint.py.151: total_norm: 10.449152 [I 2022-10-28 10:08:50] ...trap/engines/engine.py.224: train: epoch 119 | batch 2070/2304 [I 2022-10-28 10:08:50] ...trap/engines/engine.py.228: elapsed: 0:19:21 | left: 0:02:09 [I 2022-10-28 10:08:50] ...trap/engines/engine.py.229: process: 0.55736 | load: 0.00246 [I 2022-10-28 10:08:50] ...trap/engines/engine.py.230: loss: 0.12320 [I 2022-10-28 10:08:50] ...optimizers/trijoint.py.151: total_norm: 12.966382 [I 2022-10-28 10:08:56] ...trap/engines/engine.py.224: train: epoch 119 | batch 2080/2304 [I 2022-10-28 10:08:56] ...trap/engines/engine.py.228: elapsed: 0:19:26 | left: 0:02:04 [I 2022-10-28 10:08:56] ...trap/engines/engine.py.229: process: 0.55738 | load: 0.00244 [I 2022-10-28 10:08:56] ...trap/engines/engine.py.230: loss: 0.19269 [I 2022-10-28 10:08:56] ...optimizers/trijoint.py.151: total_norm: 19.975256 [I 2022-10-28 10:09:02] ...trap/engines/engine.py.224: train: epoch 119 | batch 2090/2304 [I 2022-10-28 10:09:02] ...trap/engines/engine.py.228: elapsed: 0:19:32 | left: 0:01:59 [I 2022-10-28 10:09:02] ...trap/engines/engine.py.229: process: 0.56133 | load: 0.00246 [I 2022-10-28 10:09:02] ...trap/engines/engine.py.230: loss: 0.12047 [I 2022-10-28 10:09:02] ...optimizers/trijoint.py.151: total_norm: 14.726756 [I 2022-10-28 10:09:07] ...trap/engines/engine.py.224: train: epoch 119 | batch 2100/2304 [I 2022-10-28 10:09:07] ...trap/engines/engine.py.228: elapsed: 0:19:38 | left: 0:01:53 [I 2022-10-28 10:09:07] ...trap/engines/engine.py.229: process: 0.56035 | load: 0.00244 [I 2022-10-28 10:09:07] ...trap/engines/engine.py.230: loss: 0.09433 [I 2022-10-28 10:09:07] ...optimizers/trijoint.py.151: total_norm: 11.278112 [I 2022-10-28 10:09:13] ...trap/engines/engine.py.224: train: epoch 119 | batch 2110/2304 [I 2022-10-28 10:09:13] ...trap/engines/engine.py.228: elapsed: 0:19:43 | left: 0:01:47 [I 2022-10-28 10:09:13] ...trap/engines/engine.py.229: process: 0.56251 | load: 0.00256 [I 2022-10-28 10:09:13] ...trap/engines/engine.py.230: loss: 0.09405 [I 2022-10-28 10:09:13] ...optimizers/trijoint.py.151: total_norm: 12.787424 [I 2022-10-28 10:09:18] ...trap/engines/engine.py.224: train: epoch 119 | batch 2120/2304 [I 2022-10-28 10:09:18] ...trap/engines/engine.py.228: elapsed: 0:19:49 | left: 0:01:42 [I 2022-10-28 10:09:18] ...trap/engines/engine.py.229: process: 0.54989 | load: 0.00249 [I 2022-10-28 10:09:18] ...trap/engines/engine.py.230: loss: 0.11271 [I 2022-10-28 10:09:18] ...optimizers/trijoint.py.151: total_norm: 9.414439 [I 2022-10-28 10:09:24] ...trap/engines/engine.py.224: train: epoch 119 | batch 2130/2304 [I 2022-10-28 10:09:24] ...trap/engines/engine.py.228: elapsed: 0:19:54 | left: 0:01:36 [I 2022-10-28 10:09:24] ...trap/engines/engine.py.229: process: 0.54124 | load: 0.00248 [I 2022-10-28 10:09:24] ...trap/engines/engine.py.230: loss: 0.14490 [I 2022-10-28 10:09:24] ...optimizers/trijoint.py.151: total_norm: 17.107594 [I 2022-10-28 10:09:30] ...trap/engines/engine.py.224: train: epoch 119 | batch 2140/2304 [I 2022-10-28 10:09:30] ...trap/engines/engine.py.228: elapsed: 0:20:00 | left: 0:01:33 [I 2022-10-28 10:09:30] ...trap/engines/engine.py.229: process: 0.55277 | load: 0.00245 [I 2022-10-28 10:09:30] ...trap/engines/engine.py.230: loss: 0.10036 [I 2022-10-28 10:09:30] ...optimizers/trijoint.py.151: total_norm: 9.761550 [I 2022-10-28 10:09:35] ...trap/engines/engine.py.224: train: epoch 119 | batch 2150/2304 [I 2022-10-28 10:09:35] ...trap/engines/engine.py.228: elapsed: 0:20:06 | left: 0:01:25 [I 2022-10-28 10:09:35] ...trap/engines/engine.py.229: process: 0.56060 | load: 0.00245 [I 2022-10-28 10:09:35] ...trap/engines/engine.py.230: loss: 0.15205 [I 2022-10-28 10:09:35] ...optimizers/trijoint.py.151: total_norm: 27.244944 [I 2022-10-28 10:09:41] ...trap/engines/engine.py.224: train: epoch 119 | batch 2160/2304 [I 2022-10-28 10:09:41] ...trap/engines/engine.py.228: elapsed: 0:20:11 | left: 0:01:19 [I 2022-10-28 10:09:41] ...trap/engines/engine.py.229: process: 0.55867 | load: 0.00245 [I 2022-10-28 10:09:41] ...trap/engines/engine.py.230: loss: 0.08677 [I 2022-10-28 10:09:41] ...optimizers/trijoint.py.151: total_norm: 15.465627 [I 2022-10-28 10:09:46] ...trap/engines/engine.py.224: train: epoch 119 | batch 2170/2304 [I 2022-10-28 10:09:46] ...trap/engines/engine.py.228: elapsed: 0:20:17 | left: 0:01:14 [I 2022-10-28 10:09:46] ...trap/engines/engine.py.229: process: 0.55931 | load: 0.00243 [I 2022-10-28 10:09:46] ...trap/engines/engine.py.230: loss: 0.10668 [I 2022-10-28 10:09:46] ...optimizers/trijoint.py.151: total_norm: 19.237616 [I 2022-10-28 10:09:52] ...trap/engines/engine.py.224: train: epoch 119 | batch 2180/2304 [I 2022-10-28 10:09:52] ...trap/engines/engine.py.228: elapsed: 0:20:22 | left: 0:01:08 [I 2022-10-28 10:09:52] ...trap/engines/engine.py.229: process: 0.55945 | load: 0.00242 [I 2022-10-28 10:09:52] ...trap/engines/engine.py.230: loss: 0.09496 [I 2022-10-28 10:09:52] ...optimizers/trijoint.py.151: total_norm: 12.662529 [I 2022-10-28 10:09:57] ...trap/engines/engine.py.224: train: epoch 119 | batch 2190/2304 [I 2022-10-28 10:09:57] ...trap/engines/engine.py.228: elapsed: 0:20:28 | left: 0:01:03 [I 2022-10-28 10:09:57] ...trap/engines/engine.py.229: process: 0.56422 | load: 0.00244 [I 2022-10-28 10:09:57] ...trap/engines/engine.py.230: loss: 0.12955 [I 2022-10-28 10:09:57] ...optimizers/trijoint.py.151: total_norm: 11.303223 [I 2022-10-28 10:10:03] ...trap/engines/engine.py.224: train: epoch 119 | batch 2200/2304 [I 2022-10-28 10:10:03] ...trap/engines/engine.py.228: elapsed: 0:20:33 | left: 0:00:58 [I 2022-10-28 10:10:03] ...trap/engines/engine.py.229: process: 0.55585 | load: 0.00244 [I 2022-10-28 10:10:03] ...trap/engines/engine.py.230: loss: 0.11579 [I 2022-10-28 10:10:03] ...optimizers/trijoint.py.151: total_norm: 11.427682 [I 2022-10-28 10:10:09] ...trap/engines/engine.py.224: train: epoch 119 | batch 2210/2304 [I 2022-10-28 10:10:09] ...trap/engines/engine.py.228: elapsed: 0:20:39 | left: 0:00:52 [I 2022-10-28 10:10:09] ...trap/engines/engine.py.229: process: 0.56169 | load: 0.00242 [I 2022-10-28 10:10:09] ...trap/engines/engine.py.230: loss: 0.18914 [I 2022-10-28 10:10:09] ...optimizers/trijoint.py.151: total_norm: 14.312579 [I 2022-10-28 10:10:14] ...trap/engines/engine.py.224: train: epoch 119 | batch 2220/2304 [I 2022-10-28 10:10:14] ...trap/engines/engine.py.228: elapsed: 0:20:45 | left: 0:00:47 [I 2022-10-28 10:10:14] ...trap/engines/engine.py.229: process: 0.56559 | load: 0.00242 [I 2022-10-28 10:10:14] ...trap/engines/engine.py.230: loss: 0.15816 [I 2022-10-28 10:10:14] ...optimizers/trijoint.py.151: total_norm: 25.521503 [I 2022-10-28 10:10:20] ...trap/engines/engine.py.224: train: epoch 119 | batch 2230/2304 [I 2022-10-28 10:10:20] ...trap/engines/engine.py.228: elapsed: 0:20:50 | left: 0:00:41 [I 2022-10-28 10:10:20] ...trap/engines/engine.py.229: process: 0.55921 | load: 0.00242 [I 2022-10-28 10:10:20] ...trap/engines/engine.py.230: loss: 0.11496 [I 2022-10-28 10:10:20] ...optimizers/trijoint.py.151: total_norm: 27.223167 [I 2022-10-28 10:10:25] ...trap/engines/engine.py.224: train: epoch 119 | batch 2240/2304 [I 2022-10-28 10:10:25] ...trap/engines/engine.py.228: elapsed: 0:20:56 | left: 0:00:35 [I 2022-10-28 10:10:25] ...trap/engines/engine.py.229: process: 0.56058 | load: 0.00241 [I 2022-10-28 10:10:25] ...trap/engines/engine.py.230: loss: 0.11455 [I 2022-10-28 10:10:25] ...optimizers/trijoint.py.151: total_norm: 26.957176 [I 2022-10-28 10:10:31] ...trap/engines/engine.py.224: train: epoch 119 | batch 2250/2304 [I 2022-10-28 10:10:31] ...trap/engines/engine.py.228: elapsed: 0:21:01 | left: 0:00:30 [I 2022-10-28 10:10:31] ...trap/engines/engine.py.229: process: 0.56635 | load: 0.00252 [I 2022-10-28 10:10:31] ...trap/engines/engine.py.230: loss: 0.12729 [I 2022-10-28 10:10:31] ...optimizers/trijoint.py.151: total_norm: 43.289135 [I 2022-10-28 10:10:37] ...trap/engines/engine.py.224: train: epoch 119 | batch 2260/2304 [I 2022-10-28 10:10:37] ...trap/engines/engine.py.228: elapsed: 0:21:07 | left: 0:00:24 [I 2022-10-28 10:10:37] ...trap/engines/engine.py.229: process: 0.55140 | load: 0.00246 [I 2022-10-28 10:10:37] ...trap/engines/engine.py.230: loss: 0.11262 [I 2022-10-28 10:10:37] ...optimizers/trijoint.py.151: total_norm: 24.048832 [I 2022-10-28 10:10:42] ...trap/engines/engine.py.224: train: epoch 119 | batch 2270/2304 [I 2022-10-28 10:10:42] ...trap/engines/engine.py.228: elapsed: 0:21:13 | left: 0:00:18 [I 2022-10-28 10:10:42] ...trap/engines/engine.py.229: process: 0.55321 | load: 0.00244 [I 2022-10-28 10:10:42] ...trap/engines/engine.py.230: loss: 0.09375 [I 2022-10-28 10:10:42] ...optimizers/trijoint.py.151: total_norm: 10.814552 [I 2022-10-28 10:10:48] ...trap/engines/engine.py.224: train: epoch 119 | batch 2280/2304 [I 2022-10-28 10:10:48] ...trap/engines/engine.py.228: elapsed: 0:21:18 | left: 0:00:13 [I 2022-10-28 10:10:48] ...trap/engines/engine.py.229: process: 0.55195 | load: 0.00244 [I 2022-10-28 10:10:48] ...trap/engines/engine.py.230: loss: 0.15810 [I 2022-10-28 10:10:48] ...optimizers/trijoint.py.151: total_norm: 12.834753 [I 2022-10-28 10:10:53] ...trap/engines/engine.py.224: train: epoch 119 | batch 2290/2304 [I 2022-10-28 10:10:53] ...trap/engines/engine.py.228: elapsed: 0:21:24 | left: 0:00:07 [I 2022-10-28 10:10:53] ...trap/engines/engine.py.229: process: 0.55778 | load: 0.00241 [I 2022-10-28 10:10:53] ...trap/engines/engine.py.230: loss: 0.09016 [I 2022-10-28 10:10:53] ...optimizers/trijoint.py.151: total_norm: 14.998877 [I 2022-10-28 10:10:59] ...trap/engines/engine.py.224: train: epoch 119 | batch 2300/2304 [I 2022-10-28 10:10:59] ...trap/engines/engine.py.228: elapsed: 0:21:29 | left: 0:00:02 [I 2022-10-28 10:10:59] ...trap/engines/engine.py.229: process: 0.55318 | load: 0.00015 [I 2022-10-28 10:10:59] ...trap/engines/engine.py.230: loss: 0.08802 [I 2022-10-28 10:10:59] ...optimizers/trijoint.py.151: total_norm: 44.909084 [I 2022-10-28 10:11:01] ...trap/engines/engine.py.224: train: epoch 119 | batch 2304/2304 [I 2022-10-28 10:11:01] ...trap/engines/engine.py.228: elapsed: 0:21:32 | left: 0:00:00 [I 2022-10-28 10:11:01] ...trap/engines/engine.py.229: process: 0.54667 | load: 0.00014 [I 2022-10-28 10:11:01] ...trap/engines/engine.py.230: loss: 0.06979 [I 2022-10-28 10:11:01] ...optimizers/trijoint.py.151: total_norm: 25.095440 [S 2022-10-28 10:11:02] ...trap/engines/engine.py.129: train_epoch.epoch: 119 [S 2022-10-28 10:11:02] ...trap/engines/engine.py.129: train_epoch.loss: 0.11915684844692462 [I 2022-10-28 10:11:02] ...s/criterions/triplet.py.84: margin = 0.3000000000000002 [I 2022-10-28 10:11:04] ...trap/engines/engine.py.268: Evaluating model on valset for epoch 119 [I 2022-10-28 10:11:04] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler. [W 2022-10-28 10:11:06] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 10:11:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 10:11:06] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 10:11:10] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 10:11:10] ...trap/engines/engine.py.321: eval: epoch 119 | batch 0/510 [I 2022-10-28 10:11:10] ...trap/engines/engine.py.325: elapsed: 0:00:06 | left: 0:53:53 [I 2022-10-28 10:11:10] ...trap/engines/engine.py.326: process: 6.34058 | load: 4.26327 [I 2022-10-28 10:11:14] ...trap/engines/engine.py.321: eval: epoch 119 | batch 10/510 [I 2022-10-28 10:11:14] ...trap/engines/engine.py.325: elapsed: 0:00:09 | left: 0:08:09 [I 2022-10-28 10:11:14] ...trap/engines/engine.py.326: process: 0.33994 | load: 0.00020 [I 2022-10-28 10:11:17] ...trap/engines/engine.py.321: eval: epoch 119 | batch 20/510 [I 2022-10-28 10:11:17] ...trap/engines/engine.py.325: elapsed: 0:00:13 | left: 0:03:16 [I 2022-10-28 10:11:17] ...trap/engines/engine.py.326: process: 0.32453 | load: 0.00018 [I 2022-10-28 10:11:20] ...trap/engines/engine.py.321: eval: epoch 119 | batch 30/510 [I 2022-10-28 10:11:20] ...trap/engines/engine.py.325: elapsed: 0:00:16 | left: 0:02:43 [I 2022-10-28 10:11:20] ...trap/engines/engine.py.326: process: 0.33796 | load: 0.00037 [I 2022-10-28 10:11:24] ...trap/engines/engine.py.321: eval: epoch 119 | batch 40/510 [I 2022-10-28 10:11:24] ...trap/engines/engine.py.325: elapsed: 0:00:19 | left: 0:02:35 [I 2022-10-28 10:11:24] ...trap/engines/engine.py.326: process: 0.32458 | load: 0.00018 [I 2022-10-28 10:11:27] ...trap/engines/engine.py.321: eval: epoch 119 | batch 50/510 [I 2022-10-28 10:11:27] ...trap/engines/engine.py.325: elapsed: 0:00:23 | left: 0:02:34 [I 2022-10-28 10:11:27] ...trap/engines/engine.py.326: process: 0.32578 | load: 0.00017 [I 2022-10-28 10:11:30] ...trap/engines/engine.py.321: eval: epoch 119 | batch 60/510 [I 2022-10-28 10:11:30] ...trap/engines/engine.py.325: elapsed: 0:00:26 | left: 0:02:28 [I 2022-10-28 10:11:30] ...trap/engines/engine.py.326: process: 0.32349 | load: 0.00017 [I 2022-10-28 10:11:34] ...trap/engines/engine.py.321: eval: epoch 119 | batch 70/510 [I 2022-10-28 10:11:34] ...trap/engines/engine.py.325: elapsed: 0:00:29 | left: 0:02:27 [I 2022-10-28 10:11:34] ...trap/engines/engine.py.326: process: 0.33407 | load: 0.00018 [I 2022-10-28 10:11:37] ...trap/engines/engine.py.321: eval: epoch 119 | batch 80/510 [I 2022-10-28 10:11:37] ...trap/engines/engine.py.325: elapsed: 0:00:33 | left: 0:02:23 [I 2022-10-28 10:11:37] ...trap/engines/engine.py.326: process: 0.32389 | load: 0.00016 [I 2022-10-28 10:11:40] ...trap/engines/engine.py.321: eval: epoch 119 | batch 90/510 [I 2022-10-28 10:11:40] ...trap/engines/engine.py.325: elapsed: 0:00:36 | left: 0:02:20 [I 2022-10-28 10:11:40] ...trap/engines/engine.py.326: process: 0.35085 | load: 0.00017 [I 2022-10-28 10:11:44] ...trap/engines/engine.py.321: eval: epoch 119 | batch 100/510 [I 2022-10-28 10:11:44] ...trap/engines/engine.py.325: elapsed: 0:00:39 | left: 0:02:17 [I 2022-10-28 10:11:44] ...trap/engines/engine.py.326: process: 0.28669 | load: 0.00020 [I 2022-10-28 10:11:47] ...trap/engines/engine.py.321: eval: epoch 119 | batch 110/510 [I 2022-10-28 10:11:47] ...trap/engines/engine.py.325: elapsed: 0:00:42 | left: 0:01:58 [I 2022-10-28 10:11:47] ...trap/engines/engine.py.326: process: 0.29360 | load: 0.00015 [I 2022-10-28 10:11:50] ...trap/engines/engine.py.321: eval: epoch 119 | batch 120/510 [I 2022-10-28 10:11:50] ...trap/engines/engine.py.325: elapsed: 0:00:45 | left: 0:01:52 [I 2022-10-28 10:11:50] ...trap/engines/engine.py.326: process: 0.28677 | load: 0.00015 [I 2022-10-28 10:11:53] ...trap/engines/engine.py.321: eval: epoch 119 | batch 130/510 [I 2022-10-28 10:11:53] ...trap/engines/engine.py.325: elapsed: 0:00:48 | left: 0:01:50 [I 2022-10-28 10:11:53] ...trap/engines/engine.py.326: process: 0.28808 | load: 0.00014 [I 2022-10-28 10:11:56] ...trap/engines/engine.py.321: eval: epoch 119 | batch 140/510 [I 2022-10-28 10:11:56] ...trap/engines/engine.py.325: elapsed: 0:00:51 | left: 0:01:46 [I 2022-10-28 10:11:56] ...trap/engines/engine.py.326: process: 0.28845 | load: 0.00014 [I 2022-10-28 10:11:58] ...trap/engines/engine.py.321: eval: epoch 119 | batch 150/510 [I 2022-10-28 10:11:58] ...trap/engines/engine.py.325: elapsed: 0:00:54 | left: 0:01:43 [I 2022-10-28 10:11:58] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00015 [I 2022-10-28 10:12:01] ...trap/engines/engine.py.321: eval: epoch 119 | batch 160/510 [I 2022-10-28 10:12:01] ...trap/engines/engine.py.325: elapsed: 0:00:57 | left: 0:01:41 [I 2022-10-28 10:12:01] ...trap/engines/engine.py.326: process: 0.28316 | load: 0.00015 [I 2022-10-28 10:12:04] ...trap/engines/engine.py.321: eval: epoch 119 | batch 170/510 [I 2022-10-28 10:12:04] ...trap/engines/engine.py.325: elapsed: 0:01:00 | left: 0:01:37 [I 2022-10-28 10:12:04] ...trap/engines/engine.py.326: process: 0.28755 | load: 0.00017 [I 2022-10-28 10:12:07] ...trap/engines/engine.py.321: eval: epoch 119 | batch 180/510 [I 2022-10-28 10:12:07] ...trap/engines/engine.py.325: elapsed: 0:01:03 | left: 0:01:35 [I 2022-10-28 10:12:07] ...trap/engines/engine.py.326: process: 0.28689 | load: 0.00014 [I 2022-10-28 10:12:10] ...trap/engines/engine.py.321: eval: epoch 119 | batch 190/510 [I 2022-10-28 10:12:10] ...trap/engines/engine.py.325: elapsed: 0:01:05 | left: 0:01:32 [I 2022-10-28 10:12:10] ...trap/engines/engine.py.326: process: 0.29237 | load: 0.00022 [I 2022-10-28 10:12:13] ...trap/engines/engine.py.321: eval: epoch 119 | batch 200/510 [I 2022-10-28 10:12:13] ...trap/engines/engine.py.325: elapsed: 0:01:08 | left: 0:01:31 [I 2022-10-28 10:12:13] ...trap/engines/engine.py.326: process: 0.30217 | load: 0.00019 [I 2022-10-28 10:12:16] ...trap/engines/engine.py.321: eval: epoch 119 | batch 210/510 [I 2022-10-28 10:12:16] ...trap/engines/engine.py.325: elapsed: 0:01:11 | left: 0:01:28 [I 2022-10-28 10:12:16] ...trap/engines/engine.py.326: process: 0.29060 | load: 0.00020 [I 2022-10-28 10:12:19] ...trap/engines/engine.py.321: eval: epoch 119 | batch 220/510 [I 2022-10-28 10:12:19] ...trap/engines/engine.py.325: elapsed: 0:01:14 | left: 0:01:25 [I 2022-10-28 10:12:19] ...trap/engines/engine.py.326: process: 0.29103 | load: 0.00022 [I 2022-10-28 10:12:22] ...trap/engines/engine.py.321: eval: epoch 119 | batch 230/510 [I 2022-10-28 10:12:22] ...trap/engines/engine.py.325: elapsed: 0:01:17 | left: 0:01:22 [I 2022-10-28 10:12:22] ...trap/engines/engine.py.326: process: 0.28869 | load: 0.00019 [I 2022-10-28 10:12:25] ...trap/engines/engine.py.321: eval: epoch 119 | batch 240/510 [I 2022-10-28 10:12:25] ...trap/engines/engine.py.325: elapsed: 0:01:20 | left: 0:01:18 [I 2022-10-28 10:12:25] ...trap/engines/engine.py.326: process: 0.28348 | load: 0.00015 [I 2022-10-28 10:12:28] ...trap/engines/engine.py.321: eval: epoch 119 | batch 250/510 [I 2022-10-28 10:12:28] ...trap/engines/engine.py.325: elapsed: 0:01:23 | left: 0:01:15 [I 2022-10-28 10:12:28] ...trap/engines/engine.py.326: process: 0.28380 | load: 0.00015 [I 2022-10-28 10:12:30] ...trap/engines/engine.py.321: eval: epoch 119 | batch 260/510 [I 2022-10-28 10:12:30] ...trap/engines/engine.py.325: elapsed: 0:01:26 | left: 0:01:12 [I 2022-10-28 10:12:30] ...trap/engines/engine.py.326: process: 0.28857 | load: 0.00018 [I 2022-10-28 10:12:33] ...trap/engines/engine.py.321: eval: epoch 119 | batch 270/510 [I 2022-10-28 10:12:33] ...trap/engines/engine.py.325: elapsed: 0:01:29 | left: 0:01:09 [I 2022-10-28 10:12:33] ...trap/engines/engine.py.326: process: 0.28833 | load: 0.00022 [I 2022-10-28 10:12:36] ...trap/engines/engine.py.321: eval: epoch 119 | batch 280/510 [I 2022-10-28 10:12:36] ...trap/engines/engine.py.325: elapsed: 0:01:32 | left: 0:01:06 [I 2022-10-28 10:12:36] ...trap/engines/engine.py.326: process: 0.28946 | load: 0.00018 [I 2022-10-28 10:12:39] ...trap/engines/engine.py.321: eval: epoch 119 | batch 290/510 [I 2022-10-28 10:12:39] ...trap/engines/engine.py.325: elapsed: 0:01:35 | left: 0:01:03 [I 2022-10-28 10:12:39] ...trap/engines/engine.py.326: process: 0.29065 | load: 0.00016 [I 2022-10-28 10:12:42] ...trap/engines/engine.py.321: eval: epoch 119 | batch 300/510 [I 2022-10-28 10:12:42] ...trap/engines/engine.py.325: elapsed: 0:01:38 | left: 0:01:00 [I 2022-10-28 10:12:42] ...trap/engines/engine.py.326: process: 0.28568 | load: 0.00020 [I 2022-10-28 10:12:45] ...trap/engines/engine.py.321: eval: epoch 119 | batch 310/510 [I 2022-10-28 10:12:45] ...trap/engines/engine.py.325: elapsed: 0:01:40 | left: 0:00:58 [I 2022-10-28 10:12:45] ...trap/engines/engine.py.326: process: 0.28683 | load: 0.00019 [I 2022-10-28 10:12:48] ...trap/engines/engine.py.321: eval: epoch 119 | batch 320/510 [I 2022-10-28 10:12:48] ...trap/engines/engine.py.325: elapsed: 0:01:43 | left: 0:00:55 [I 2022-10-28 10:12:48] ...trap/engines/engine.py.326: process: 0.28817 | load: 0.00015 [I 2022-10-28 10:12:51] ...trap/engines/engine.py.321: eval: epoch 119 | batch 330/510 [I 2022-10-28 10:12:51] ...trap/engines/engine.py.325: elapsed: 0:01:46 | left: 0:00:52 [I 2022-10-28 10:12:51] ...trap/engines/engine.py.326: process: 0.28807 | load: 0.00014 [I 2022-10-28 10:12:54] ...trap/engines/engine.py.321: eval: epoch 119 | batch 340/510 [I 2022-10-28 10:12:54] ...trap/engines/engine.py.325: elapsed: 0:01:49 | left: 0:00:49 [I 2022-10-28 10:12:54] ...trap/engines/engine.py.326: process: 0.28940 | load: 0.00023 [I 2022-10-28 10:12:57] ...trap/engines/engine.py.321: eval: epoch 119 | batch 350/510 [I 2022-10-28 10:12:57] ...trap/engines/engine.py.325: elapsed: 0:01:52 | left: 0:00:46 [I 2022-10-28 10:12:57] ...trap/engines/engine.py.326: process: 0.29072 | load: 0.00019 [I 2022-10-28 10:13:00] ...trap/engines/engine.py.321: eval: epoch 119 | batch 360/510 [I 2022-10-28 10:13:00] ...trap/engines/engine.py.325: elapsed: 0:01:55 | left: 0:00:43 [I 2022-10-28 10:13:00] ...trap/engines/engine.py.326: process: 0.29512 | load: 0.00015 [I 2022-10-28 10:13:02] ...trap/engines/engine.py.321: eval: epoch 119 | batch 370/510 [I 2022-10-28 10:13:02] ...trap/engines/engine.py.325: elapsed: 0:01:58 | left: 0:00:40 [I 2022-10-28 10:13:02] ...trap/engines/engine.py.326: process: 0.28396 | load: 0.00015 [I 2022-10-28 10:13:05] ...trap/engines/engine.py.321: eval: epoch 119 | batch 380/510 [I 2022-10-28 10:13:05] ...trap/engines/engine.py.325: elapsed: 0:02:01 | left: 0:00:37 [I 2022-10-28 10:13:05] ...trap/engines/engine.py.326: process: 0.28533 | load: 0.00015 [I 2022-10-28 10:13:08] ...trap/engines/engine.py.321: eval: epoch 119 | batch 390/510 [I 2022-10-28 10:13:08] ...trap/engines/engine.py.325: elapsed: 0:02:04 | left: 0:00:34 [I 2022-10-28 10:13:08] ...trap/engines/engine.py.326: process: 0.28657 | load: 0.00016 [I 2022-10-28 10:13:11] ...trap/engines/engine.py.321: eval: epoch 119 | batch 400/510 [I 2022-10-28 10:13:11] ...trap/engines/engine.py.325: elapsed: 0:02:07 | left: 0:00:31 [I 2022-10-28 10:13:11] ...trap/engines/engine.py.326: process: 0.28545 | load: 0.00015 [I 2022-10-28 10:13:14] ...trap/engines/engine.py.321: eval: epoch 119 | batch 410/510 [I 2022-10-28 10:13:14] ...trap/engines/engine.py.325: elapsed: 0:02:10 | left: 0:00:28 [I 2022-10-28 10:13:14] ...trap/engines/engine.py.326: process: 0.29036 | load: 0.00015 [I 2022-10-28 10:13:17] ...trap/engines/engine.py.321: eval: epoch 119 | batch 420/510 [I 2022-10-28 10:13:17] ...trap/engines/engine.py.325: elapsed: 0:02:12 | left: 0:00:25 [I 2022-10-28 10:13:17] ...trap/engines/engine.py.326: process: 0.28071 | load: 0.00021 [I 2022-10-28 10:13:20] ...trap/engines/engine.py.321: eval: epoch 119 | batch 430/510 [I 2022-10-28 10:13:20] ...trap/engines/engine.py.325: elapsed: 0:02:15 | left: 0:00:23 [I 2022-10-28 10:13:20] ...trap/engines/engine.py.326: process: 0.28485 | load: 0.00023 [I 2022-10-28 10:13:23] ...trap/engines/engine.py.321: eval: epoch 119 | batch 440/510 [I 2022-10-28 10:13:23] ...trap/engines/engine.py.325: elapsed: 0:02:18 | left: 0:00:20 [I 2022-10-28 10:13:23] ...trap/engines/engine.py.326: process: 0.29327 | load: 0.00020 [I 2022-10-28 10:13:26] ...trap/engines/engine.py.321: eval: epoch 119 | batch 450/510 [I 2022-10-28 10:13:26] ...trap/engines/engine.py.325: elapsed: 0:02:21 | left: 0:00:17 [I 2022-10-28 10:13:26] ...trap/engines/engine.py.326: process: 0.28619 | load: 0.00026 [I 2022-10-28 10:13:29] ...trap/engines/engine.py.321: eval: epoch 119 | batch 460/510 [I 2022-10-28 10:13:29] ...trap/engines/engine.py.325: elapsed: 0:02:24 | left: 0:00:14 [I 2022-10-28 10:13:29] ...trap/engines/engine.py.326: process: 0.29117 | load: 0.00019 [I 2022-10-28 10:13:31] ...trap/engines/engine.py.321: eval: epoch 119 | batch 470/510 [I 2022-10-28 10:13:31] ...trap/engines/engine.py.325: elapsed: 0:02:27 | left: 0:00:11 [I 2022-10-28 10:13:31] ...trap/engines/engine.py.326: process: 0.28417 | load: 0.00026 [I 2022-10-28 10:13:34] ...trap/engines/engine.py.321: eval: epoch 119 | batch 480/510 [I 2022-10-28 10:13:34] ...trap/engines/engine.py.325: elapsed: 0:02:30 | left: 0:00:08 [I 2022-10-28 10:13:34] ...trap/engines/engine.py.326: process: 0.29277 | load: 0.00023 [I 2022-10-28 10:13:37] ...trap/engines/engine.py.321: eval: epoch 119 | batch 490/510 [I 2022-10-28 10:13:37] ...trap/engines/engine.py.325: elapsed: 0:02:33 | left: 0:00:05 [I 2022-10-28 10:13:37] ...trap/engines/engine.py.326: process: 0.28757 | load: 0.00019 [I 2022-10-28 10:13:40] ...trap/engines/engine.py.321: eval: epoch 119 | batch 500/510 [I 2022-10-28 10:13:40] ...trap/engines/engine.py.325: elapsed: 0:02:36 | left: 0:00:02 [I 2022-10-28 10:13:40] ...trap/engines/engine.py.326: process: 0.28818 | load: 0.00016 [I 2022-10-28 10:13:43] ...trap/engines/engine.py.321: eval: epoch 119 | batch 510/510 [I 2022-10-28 10:13:43] ...trap/engines/engine.py.325: elapsed: 0:02:38 | left: 0:00:00 [I 2022-10-28 10:13:43] ...trap/engines/engine.py.326: process: 0.28258 | load: 0.00018 [S 2022-10-28 10:13:44] ...trap/engines/engine.py.132: eval_epoch.epoch: 119 [I 2022-10-28 10:13:44] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.739000 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.903900 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.932600 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.734500 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.902800 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.932500 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011975 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.008264 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008321 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011994 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007332 [S 2022-10-28 10:13:48] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.009036 [I 2022-10-28 10:13:50] ...trap/engines/engine.py.141: Saving last checkpoint [I 2022-10-28 10:13:50] ...trap/engines/engine.py.470: Saving model... [W 2022-10-28 10:13:51] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json' [W 2022-10-28 10:13:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [W 2022-10-28 10:13:51] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']' [I 2022-10-28 10:13:56] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html [I 2022-10-28 10:13:57] ...trap/engines/engine.py.474: Saving optimizer... [I 2022-10-28 10:14:03] ...trap/engines/engine.py.478: Saving engine... [I 2022-10-28 10:14:03] ...trap/engines/engine.py.145: Ending training procedures